serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(codec): the waveform body is a record chain, not a tag stream

Browse Source 
Supersedes the segment-header model entirely, including the fixes made
earlier today.  Found via multi-agent structural analysis of the 25 files
that stalled the walker, then verified independently.

Records are self-delimiting: off+2 is a uint16 BE length, next_record =
off + 2 + len, and the chain ends on a record whose chan_id is 0x06.
off+8 carries a 3-valued mode enum:
  02 00  14-byte header, 2 anchors, then CUMULATIVE delta blocks
  01 00  10-byte header, no anchors, blocks are ABSOLUTE values
  00 03  10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute

`40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header.
Reading it as a 2*NN + 16 header is what made walks drift — the
"variable-prefix segment descriptors" reported earlier today were not a
format feature, just walker drift of exactly
4 - (old_stop - true_record_start), on all 25 affected files.

Measured on the production snapshot:
  all four channels equal length   156/1388 -> 1388/1388
  ASCII sample-count exact           72/75  ->   75/75
  ASCII fully exact                  70/75  ->   73/75
  device PPV waveform (live)       1288/1306 -> 1306/1306  (mean err 0.00000)
  device PPV histogram (live)      4434/4459 -> 4458/4459

Also eliminates the walker-over-read class: 24 of those 35 files were
histograms that read_blastware_file fed to the waveform codec first; the
old walker accepted them and returned garbage (one yielded 98,923
"intervals"), while the record-chain decoder returns None so they fall
through to histogram_codec.

00 03 records are DECODED, not skipped.  Skipping them silently shifts
the time base of everything after them on that channel — BE9558/
K558LOF2.820W had MicL displaced by exactly 512 samples with nothing
marking the gap.

Footer detection now prefers the 0e 08 candidate whose body yields a
chain terminating on 0x06; the signature can occur inside a sample
stream.  Blast radius 1 file of 1388.

The superseded model survives as decode_waveform_legacy, pinned by
micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes
candidates and keeps whichever yields the most samples, so the new
decoder returning None where the old returned garbage changes that
heuristic's winner.  Deferred until that search uses the record chain.

Tests: 253 passed (+11), failure list unchanged from baseline.  The 9
tests pinning the superseded model are retargeted at
decode_waveform_legacy, which still implements it.

NOTE: stored .h5 files need regenerating — nearly all get longer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
serversdown
2026-08-25 22:13:29 +00:00
co-authored by Claude Opus 5
parent 260bf0bc67
commit 9bb95003e9
8 changed files with 591 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+23
View File
@@ -223,6 +223,29 @@ custom delta + RLE + variable-width codec.
`NN + 2` for int8 blocks). Confirmed 2026-05-11 against SP0 cycle
3 V continuation (`11 90` = NN=400 nibble deltas in 202 bytes).
### ⚠ SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN
Everything in this section below about `40 NN` segment headers, tagless
headers, variable header widths and channel rotation describes a model that
is **wrong**. The body is a chain of self-delimiting per-channel records:
off+2 len uint16 BE -> next_record = off + 2 + len
off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL / 0x06 = END
off+8 mode 02 00 = deltas+anchors (14B hdr)
01 00 = ABSOLUTE values (10B hdr)
00 03 = raw 12-bit absolute, NO TAGS (10B hdr)
`40 NN` is an ordinary int16 BE data block (`2*NN + 2`), never a header. The
"variable prefix" of 0/2/4/6/8 bytes was walker drift, exactly
`4 - (old_stop - true_record_start)`.
All four channels now come out equal length in **1388/1388** files (was
156/1388); ASCII sample-count exact **75/75**, fully exact **73/75**; device
PPV on a live decode **1306/1306** waveform, **4458/4459** histogram.
The old model survives as `decode_waveform_legacy` because
`micromate/idf_file.py` pins it for Thor IDFW body-offset search.
### Framing cases added 2026-05-11 → 2026-08-25
Four more block-framing cases, each of which had been causing **silent