serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(codec): the waveform body is a record chain, not a tag stream

Browse Source 
Supersedes the segment-header model entirely, including the fixes made
earlier today.  Found via multi-agent structural analysis of the 25 files
that stalled the walker, then verified independently.

Records are self-delimiting: off+2 is a uint16 BE length, next_record =
off + 2 + len, and the chain ends on a record whose chan_id is 0x06.
off+8 carries a 3-valued mode enum:
  02 00  14-byte header, 2 anchors, then CUMULATIVE delta blocks
  01 00  10-byte header, no anchors, blocks are ABSOLUTE values
  00 03  10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute

`40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header.
Reading it as a 2*NN + 16 header is what made walks drift — the
"variable-prefix segment descriptors" reported earlier today were not a
format feature, just walker drift of exactly
4 - (old_stop - true_record_start), on all 25 affected files.

Measured on the production snapshot:
  all four channels equal length   156/1388 -> 1388/1388
  ASCII sample-count exact           72/75  ->   75/75
  ASCII fully exact                  70/75  ->   73/75
  device PPV waveform (live)       1288/1306 -> 1306/1306  (mean err 0.00000)
  device PPV histogram (live)      4434/4459 -> 4458/4459

Also eliminates the walker-over-read class: 24 of those 35 files were
histograms that read_blastware_file fed to the waveform codec first; the
old walker accepted them and returned garbage (one yielded 98,923
"intervals"), while the record-chain decoder returns None so they fall
through to histogram_codec.

00 03 records are DECODED, not skipped.  Skipping them silently shifts
the time base of everything after them on that channel — BE9558/
K558LOF2.820W had MicL displaced by exactly 512 samples with nothing
marking the gap.

Footer detection now prefers the 0e 08 candidate whose body yields a
chain terminating on 0x06; the signature can occur inside a sample
stream.  Blast radius 1 file of 1388.

The superseded model survives as decode_waveform_legacy, pinned by
micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes
candidates and keeps whichever yields the most samples, so the new
decoder returning None where the old returned garbage changes that
heuristic's winner.  Deferred until that search uses the record chain.

Tests: 253 passed (+11), failure list unchanged from baseline.  The 9
tests pinning the superseded model are retargeted at
decode_waveform_legacy, which still implements it.

NOTE: stored .h5 files need regenerating — nearly all get longer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
serversdown
2026-08-25 22:13:29 +00:00
co-authored by Claude Opus 5
parent 260bf0bc67
commit 9bb95003e9
8 changed files with 591 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/instantel_protocol_reference.md
+69 -25
View File
@@ -11,6 +11,7 @@
| Date | Section | Change |
|---|---|---|
| 2026-08-25 (3) | S7.6.1, S15 | **THE WAVEFORM BODY IS A RECORD CHAIN, NOT A TAG STREAM — supersedes the segment-header model entirely.** Records are self-delimiting: `off+2` is a uint16 BE length and `next = off + 2 + len`; the chain ends on a record whose chan_id is `0x06`. `off+8` holds a 3-valued mode enum - `02 00` (14-byte header, anchors, cumulative deltas), `01 00` (10-byte, no anchors, ABSOLUTE values), `00 03` (10-byte, no tags at all, raw 12-bit absolute). **`40 NN` is an ordinary int16 BE data block of length 2*NN+2**, never a header; reading it as a 2*NN+16 header is what made walks drift, and the 'variable prefix' of 0/2/4/6/8 bytes reported earlier the same day was walker drift, exactly `4 - (old_stop - true_record_start)`. Verified: chain terminates on `06` in 1387/1388 files; all four channels equal length in **1388/1388** (was 156/1388); ASCII sample-count exact 72/75 -> **75/75**, fully exact 70/75 -> **73/75**; device PPV on a live decode **1306/1306** waveform (mean abs ratio error 0.00000) and **4458/4459** histogram. Also eliminated the walker-over-read class: 24 of those files were histograms the waveform codec was wrongly accepting. The superseded model is retained as `decode_waveform_legacy` because `micromate/idf_file.py` pins it for Thor IDFW body-offset search. |
| 2026-08-25 (2) | S7.6.2, S15 | **HISTOGRAM BLOCK IS BIG-ENDIAN + terminal block tail.** The 32-byte histogram block's per-channel fields are uint16 **big-endian** (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], `V_halfperiod` [11:13], `L_peak` [13:15], `L_halfperiod` [15:17], `M_peak` [17:19], `M_halfperiod` [19:21]); only `block_ctr` [2:4] is little-endian. The marker is `block[4]` alone - the previous uint16 LE marker test at [4:6] forced `block[5] == 0` and thereby capped every geo peak at 255 counts (1.275 in/s), silently clipping larger peaks. The byte previously documented as a per-channel "annotation" is the high byte of the big-endian half-period, which is why it was non-zero exactly on sub-Hz intervals. Separately, the **final block of each stream carries tail `9c 06 00 42`** rather than `1e 0a 00 00` and holds arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram. Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly, plus 842,442 per-interval frequency comparisons with zero mismatches (previously 1 of 1196 files fully correct). |
| 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. |
| 2026-05-20 | §2, §3, §4.2, §5.1, §5.3, §6, §7.5b, §7.6.1, §7.6.3, §7.6.4, §7.7.2, §7.7.3, §7.7.5, §7.8.4, §7.8.7, §7.9, §8, §11, §12, §13, §14, §15, Appendix D | **DOC AUDIT PASS — accuracy sweep against `CLAUDE.md` + `minimateplus/` code.** Fixed: (1) S3 frames terminate on bare ETX, not DLE+ETX — §2/§3 rewritten. (2) §3 payload layout corrected — byte[1]=flags, byte[2]=SUB (was wrongly labelled DLE/ADDR). (3) §4.2 — probe responses do NOT carry data length; lengths are hardcoded `DATA_LENGTHS` constants. (4) §5.1 — removed stale duplicate "SUB 1C = TRIGGER CONFIG READ" row; SUB 0A lengths corrected from `0x30/0x26` to `0x46/0x2C` (real event / boundary marker). (5) §5.3 — added missing write-frame format (BW_CMD-only doubling, DLE-aware checksum, offset formula, ack format, SUB 71 chunk parameters). (6) §6 — fixed "SUB 06 → channel config read" → event storage range. (7) §7.5b / §8 — added the 10-byte `sub_code=0x03` continuous-mode timestamp variant alongside the 9-byte single-shot layout; peak vector sum location corrected from "fixed offset 87" to `tran_pos − 12` (label-relative). (8) §7.6 / §7.6.1 / §7.6.3 / §7.6.4 — switched compliance-anchor convention from the 10-byte form to the canonical 6-byte `\xbe\x80\x00\x00\x00\x00`; recording_mode confirmed at anchor−8 in BOTH read and write (was wrongly listed as anchor−3 write / anchor−4 read); sample_rate at anchor−6, histogram_interval at anchor−4, record_time at anchor+6; geo_range row added at channel_label+33. (9) §7.7.2 — token byte position corrected from `params[6]` to `params[7]`. (10) §7.8.4 — fi==9 skip marked FIXED (already removed from code); chunk-count totals updated. (11) §7.8.7 — TODO replaced with current state of `_decode_a5_metadata_into`. (12) §7.9 — Histogram Interval upgraded ❓ → ✅. (13) §11 — POLL example wire bytes corrected; SUB 5A row added to checksum table. (14) §13 — device-under-test updated for current primary unit (BE11529 / S338.17). (15) §14 — TCP Idle Timeout fixed (0→2 min); Data Forwarding Timeout units clarified. (16) §15 (renumbered from second §14) — open-question items already resolved in CLAUDE.md closed out. (17) Appendix D — extension taxonomy rewritten to reflect the AB0T timestamp encoding (D.5.2/D.5.3); EXTENSION REFUTED warning replaced with the resolved encoding. |
@@ -1268,36 +1269,79 @@ re-deriving the whole production store:
The series-4 figure is closer to correct but not exact — the Thor
per-count LSB is its own open question (see §15).
###### Unmapped: variable-prefix segment descriptors ❓ OPEN
###### SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN, not a tag stream
Three of 75 ground-truth production events still truncate. In each,
the walk reaches a segment header whose channel-id field is preceded by
a **variable-width prefix** — 2, 4 or 6 bytes have all been observed,
where the standard tagless form always has 4 (``field2`` + ``len``).
These records also carry the ``01 00`` marker rather than ``02 00``,
and appear packed back-to-back with little or no sample data between
them.
Everything above about ``40 NN`` segment headers, tagless headers, variable
header widths and channel rotation describes a model that is **wrong**. It
produced nearly-correct output because the block table happens to tile the
data sections correctly, but the framing is not what the device writes.
The ``01 00`` marker is *not* simply an anchor count: records carrying
it have been seen with both 2-byte and 4-byte anchor fields in the same
file, so the prefix width and the marker are not yet reconciled.
The decoder stops cleanly at these rather than emitting garbage.
Examples: ``BE12599/N599LPNB.JF0W`` at body offset 1155 (2-byte
prefix), ``BE12599/N599LPWJ.980W`` at 849 (6-byte prefix),
``BE9558/K558LOF2.820W`` at 1485.
Examples from event-c (1 sec single-shot):
The body is a chain of **self-delimiting per-channel records**:
```
Segment header 1 (offset 235):
40 02 | 00 00 00 00 | 0a 4b 01 1e | 47 00 00 00 | 02 00 00 01 | 00 01
^counter=0x47
Segment header 2 (offset 523):
40 02 | ff fe ff fe | 13 f5 01 06 | 48 00 00 00 | 02 00 00 01 | 00 02
^counter=0x48 (+1)
off+0 field2 uint16 purpose unknown (not a length, not a checksum)
off+2 len uint16 BE next_record = off + 2 + len <- authoritative
off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL
0x06 = END OF WAVEFORM STREAM
off+5 0x00
off+6 0x00
off+7 segment index
off+8 mode 2 bytes, a 3-valued enum
off+10 anchors 2 x int16 BE, ABSOLUTE -- present ONLY when mode == 02 00
```
**Mode enum**, all three ground-truth verified:
| mode | header | data section |
|---|---|---|
| ``02 00`` | 14 bytes | anchors emitted, then blocks are **cumulative deltas** |
| ``01 00`` | 10 bytes | no anchors, blocks carry **absolute** sample values |
| ``00 03`` | 10 bytes | **no tags at all** — raw 12-bit packed absolute samples |
Census over the 1,388 production series-3 waveform binaries:
``02 00`` x 32,617, ``01 00`` x 74, ``00 03`` x 71.
**``40 NN`` is an ordinary int16 BE data block** of length ``2*NN + 2``
(1 <= NN <= 8), never a header. The superseded model read it as a header of
length ``2*NN + 16``, which is exactly why walks drifted: the "variable prefix"
of 0/2/4/6/8 bytes reported earlier was walker drift, precisely
``4 - (old_stop - true_record_start)``, on all 25 affected files.
Block table for data sections (``NN = ((tag_hi & 0x0F) << 8) | tag_lo``):
| tag | length | samples |
|---|---|---|
| ``0X NN`` | 2 | NN (RLE hold — holds the previous value in BOTH delta and absolute modes) |
| ``1X NN`` | NN/2 + 2 | NN (4-bit nibble) |
| ``2X NN`` | NN + 2 | NN (int8) |
| ``30 NN`` | NN*1.5 + 2 | NN (12-bit packed) |
| ``40 NN`` | 2*NN + 2 | NN (int16 BE) |
The ``30 NN`` "trailer length = NN*4" fallback must NOT be applied inside a
record — it corrupts records whose ``30 NN`` sits near a boundary.
**The preamble is segment 0's implicit Tran record.** ``body[1:3]`` carries
the same mode pair: ``00 02 00`` (1,387 of 1,388 files) means two int16 BE
anchors at ``body[3:7]`` then delta blocks; ``00 00 03`` (1 file,
``BE13121/O121L4L1.KF0W``) means raw 12-bit absolute from ``body[3]``, which
cannot be block-walked — the first record must be located by scanning.
**Verification.** The length chain terminates on a ``0x06`` record in 1,387 of
1,388 files (the exception has an ambiguous footer signature inside its sample
stream). All four channels come out at identical length in **1,388/1,388**,
against 156/1,388 under the superseded model. Against the 75 events with a
preserved Blastware ASCII export: sample-count exact **72/75 -> 75/75**, fully
exact **70/75 -> 73/75** (the 2 remaining differ by exactly 1 LSB on samples
sitting at the +-10.000 in/s rail). Against device-reported PPVs on a live
decode: waveform **1306/1306** exact with mean absolute ratio error 0.00000;
histogram **4458/4459**.
This also eliminated the walker-over-read class entirely. 24 of those 35
"histogram" over-reads were histogram files that ``read_blastware_file`` fed to
the *waveform* codec first; the old walker accepted them and returned garbage
(one produced 98,923 "intervals"), while the record-chain decoder correctly
returns None so they fall through to ``histogram_codec``.
##### Trailer
The trailer (after the last segment's data) is a sequence of 32-byte
@@ -3037,7 +3081,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
| **ACH inbound server — RESOLVED.** `bridges/ach_server.py` implements full inbound ACH pipeline. `--clear-after-download` flag for delete-after-upload workflow. Post-erase key-reuse detection via `max_downloaded_key` high-water mark. | RESOLVED | 2026-04-11 | |
| **Sensor Check dropdown byte location** — byte offset in 1A compliance config payload for the "Sensor Check: Before monitoring / After each event / Disabled" setting is NOT YET LOCATED. Confirmed: unit always runs with "Before monitoring" set. Need a capture with "Disabled" to diff. | MEDIUM | 2026-04-08 | Still open |
| **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open |
| **Variable-prefix segment descriptors** — 3 of 75 ground-truth events still truncate. The walk reaches a segment header whose channel-id field is preceded by a *variable-width* prefix (2, 4 or 6 bytes observed; the standard tagless form always has 4), carrying an `01 00` marker instead of `02 00`. The marker is **not** simply an anchor count — `01 00` records appear with both 2- and 4-byte anchor fields in the same file, so prefix width and marker are not yet reconciled. Examples: `BE12599/N599LPNB.JF0W` @1155, `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485. | MEDIUM | 2026-08-25 | Still open |
| ~~**Variable-prefix segment descriptors**~~ - **RESOLVED 2026-08-25:** there is no variable prefix. The body is a chain of self-delimiting records (see S7.6.1); the 0/2/4/6/8-byte prefix was walker drift from reading `40 NN` as a segment header. All 25 affected files now chain cleanly to the `06` terminator. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
| ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
| **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open |
| **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open |