fix(codec): the waveform body is a record chain, not a tag stream
Supersedes the segment-header model entirely, including the fixes made earlier today. Found via multi-agent structural analysis of the 25 files that stalled the walker, then verified independently. Records are self-delimiting: off+2 is a uint16 BE length, next_record = off + 2 + len, and the chain ends on a record whose chan_id is 0x06. off+8 carries a 3-valued mode enum: 02 00 14-byte header, 2 anchors, then CUMULATIVE delta blocks 01 00 10-byte header, no anchors, blocks are ABSOLUTE values 00 03 10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute `40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header. Reading it as a 2*NN + 16 header is what made walks drift — the "variable-prefix segment descriptors" reported earlier today were not a format feature, just walker drift of exactly 4 - (old_stop - true_record_start), on all 25 affected files. Measured on the production snapshot: all four channels equal length 156/1388 -> 1388/1388 ASCII sample-count exact 72/75 -> 75/75 ASCII fully exact 70/75 -> 73/75 device PPV waveform (live) 1288/1306 -> 1306/1306 (mean err 0.00000) device PPV histogram (live) 4434/4459 -> 4458/4459 Also eliminates the walker-over-read class: 24 of those 35 files were histograms that read_blastware_file fed to the waveform codec first; the old walker accepted them and returned garbage (one yielded 98,923 "intervals"), while the record-chain decoder returns None so they fall through to histogram_codec. 00 03 records are DECODED, not skipped. Skipping them silently shifts the time base of everything after them on that channel — BE9558/ K558LOF2.820W had MicL displaced by exactly 512 samples with nothing marking the gap. Footer detection now prefers the 0e 08 candidate whose body yields a chain terminating on 0x06; the signature can occur inside a sample stream. Blast radius 1 file of 1388. The superseded model survives as decode_waveform_legacy, pinned by micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes candidates and keeps whichever yields the most samples, so the new decoder returning None where the old returned garbage changes that heuristic's winner. Deferred until that search uses the record chain. Tests: 253 passed (+11), failure list unchanged from baseline. The 9 tests pinning the superseded model are retargeted at decode_waveform_legacy, which still implements it. NOTE: stored .h5 files need regenerating — nearly all get longer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
@@ -8,6 +8,62 @@ All notable changes to seismo-relay are documented here.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The series-3 waveform body is a RECORD CHAIN, not a tag stream — this
|
||||
supersedes the segment-header model, including the fixes made earlier the
|
||||
same day.**
|
||||
|
||||
Records are self-delimiting. `off+2` is a `uint16 BE` length and
|
||||
`next_record = off + 2 + len`; the chain ends on a record whose `chan_id` is
|
||||
`0x06`. `off+8` carries a 3-valued mode enum:
|
||||
|
||||
| mode | header | data section |
|
||||
|---|---|---|
|
||||
| `02 00` | 14 B | anchors, then **cumulative deltas** |
|
||||
| `01 00` | 10 B | no anchors, **absolute** values |
|
||||
| `00 03` | 10 B | **no tags at all** — raw 12-bit packed absolute |
|
||||
|
||||
**`40 NN` is an ordinary int16 BE data block** (`2*NN + 2`), never a segment
|
||||
header. Reading it as a `2*NN + 16` header is what made walks drift — and the
|
||||
"variable-prefix segment descriptors" reported earlier today were not a format
|
||||
feature at all, just walker drift of exactly
|
||||
`4 - (old_stop - true_record_start)` on all 25 affected files.
|
||||
|
||||
Measured against the production snapshot:
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| all four channels equal length | 156 / 1388 | **1388 / 1388** |
|
||||
| ASCII sample-count exact | 72 / 75 | **75 / 75** |
|
||||
| ASCII fully exact | 70 / 75 | **73 / 75** |
|
||||
| device PPV, waveform (live decode) | 1288 / 1306 | **1306 / 1306** |
|
||||
| device PPV, histogram (live decode) | 4434 / 4459 | **4458 / 4459** |
|
||||
|
||||
Mean absolute PPV ratio error on waveforms is now 0.00000. The 2 remaining
|
||||
ASCII imperfections differ by exactly 1 LSB on samples sitting at the
|
||||
±10.000 in/s rail.
|
||||
|
||||
**This also eliminated the walker-over-read class.** 24 of those 35 files
|
||||
were histograms that `read_blastware_file` fed to the *waveform* codec first;
|
||||
the old walker accepted them and returned garbage (one yielded 98,923
|
||||
"intervals"), while the record-chain decoder correctly returns `None` so they
|
||||
fall through to `histogram_codec`.
|
||||
|
||||
`00 03` records are decoded rather than skipped. Skipping them does not merely
|
||||
lose samples — it silently shifts the time base of everything after them on
|
||||
that channel (observed on `BE9558/K558LOF2.820W`, MicL displaced by exactly
|
||||
512 samples with nothing marking the gap).
|
||||
|
||||
Footer detection now prefers whichever `0e 08` candidate yields a chain
|
||||
terminating on `0x06`, since the signature can occur inside a sample stream.
|
||||
Blast radius: 1 file of 1,388.
|
||||
|
||||
The superseded model is retained as `decode_waveform_legacy` and pinned by
|
||||
`micromate/idf_file.py`, whose Thor IDFW body-offset search trial-decodes
|
||||
candidate offsets and keeps whichever yields the most samples — the new
|
||||
decoder correctly returns `None` where the old one returned garbage, which
|
||||
changes that heuristic's winner. Switching Thor over is deferred until that
|
||||
search is reworked to use the record chain directly.
|
||||
|
||||
- **Series-3 histogram block is uniformly big-endian, and the stream's final
|
||||
block has its own tail — the codec was clipping large peaks and dropping the
|
||||
last interval of nearly every histogram.**
|
||||
|
||||
Reference in New Issue
Block a user