serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(codec): the waveform body is a record chain, not a tag stream

Browse Source 
Supersedes the segment-header model entirely, including the fixes made
earlier today.  Found via multi-agent structural analysis of the 25 files
that stalled the walker, then verified independently.

Records are self-delimiting: off+2 is a uint16 BE length, next_record =
off + 2 + len, and the chain ends on a record whose chan_id is 0x06.
off+8 carries a 3-valued mode enum:
  02 00  14-byte header, 2 anchors, then CUMULATIVE delta blocks
  01 00  10-byte header, no anchors, blocks are ABSOLUTE values
  00 03  10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute

`40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header.
Reading it as a 2*NN + 16 header is what made walks drift — the
"variable-prefix segment descriptors" reported earlier today were not a
format feature, just walker drift of exactly
4 - (old_stop - true_record_start), on all 25 affected files.

Measured on the production snapshot:
  all four channels equal length   156/1388 -> 1388/1388
  ASCII sample-count exact           72/75  ->   75/75
  ASCII fully exact                  70/75  ->   73/75
  device PPV waveform (live)       1288/1306 -> 1306/1306  (mean err 0.00000)
  device PPV histogram (live)      4434/4459 -> 4458/4459

Also eliminates the walker-over-read class: 24 of those 35 files were
histograms that read_blastware_file fed to the waveform codec first; the
old walker accepted them and returned garbage (one yielded 98,923
"intervals"), while the record-chain decoder returns None so they fall
through to histogram_codec.

00 03 records are DECODED, not skipped.  Skipping them silently shifts
the time base of everything after them on that channel — BE9558/
K558LOF2.820W had MicL displaced by exactly 512 samples with nothing
marking the gap.

Footer detection now prefers the 0e 08 candidate whose body yields a
chain terminating on 0x06; the signature can occur inside a sample
stream.  Blast radius 1 file of 1388.

The superseded model survives as decode_waveform_legacy, pinned by
micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes
candidates and keeps whichever yields the most samples, so the new
decoder returning None where the old returned garbage changes that
heuristic's winner.  Deferred until that search uses the record chain.

Tests: 253 passed (+11), failure list unchanged from baseline.  The 9
tests pinning the superseded model are retargeted at
decode_waveform_legacy, which still implements it.

NOTE: stored .h5 files need regenerating — nearly all get longer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
serversdown
2026-08-25 22:13:29 +00:00
co-authored by Claude Opus 5
parent 260bf0bc67
commit 9bb95003e9
8 changed files with 591 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
minimateplus/waveform_codec.py
+237 -1
View File
@@ -441,8 +441,16 @@ def decode_tran_initial(body: bytes) -> Optional[List[int]]:
return out
def decode_waveform_v2(body: bytes) -> Optional[dict]:
def decode_waveform_legacy(body: bytes) -> Optional[dict]:
"""
SUPERSEDED 2026-08-25 — the tag-dispatch / segment-header model.
Retained because ``micromate/idf_file.py`` trial-decodes Thor IDFW bodies
at many candidate offsets and keeps whichever yields the most samples;
the record-chain decoder returns None where this one returned garbage,
which shifts that heuristic's winner. Thor is pinned here until its own
body-offset search is reworked. Do not use for series-3.
Decode the body into per-channel sample arrays.
Status (2026-05-11 evening — channel-rotation hypothesis CONFIRMED):
@@ -673,3 +681,231 @@ def decode_a5_frames(a5_frames) -> Optional[dict]:
if decoded is None:
return None
return decoded_to_adc_counts(decoded)
# ── Record-chain body model (CONFIRMED 2026-08-25) ──────────────────────────
#
# The body is NOT a flat tag-dispatch stream with ``40 NN`` segment headers.
# It is a chain of self-delimiting per-channel RECORDS:
#
# off+0 field2 uint16 purpose unknown (not a length, not a checksum)
# off+2 len uint16 BE next_record = off + 2 + len <- authoritative
# off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL
# 0x06 = end of waveform stream
# off+5 0x00
# off+6 0x00
# off+7 segment index
# off+8 mode 2 bytes, a 3-valued enum (see below)
# off+10 anchors 2 x int16 BE, ABSOLUTE — present only when mode is 02 00
#
# Mode semantics, all ground-truth verified:
# 02 00 14-byte header; emit the 2 anchors, then blocks are CUMULATIVE deltas
# 01 00 10-byte header; no anchors; blocks carry ABSOLUTE sample values
# 00 03 10-byte header; NO TAGS AT ALL — the data section is raw 12-bit
# packed ABSOLUTE samples (6 bytes -> 4 samples)
#
# ``40 NN`` is an ordinary int16 BE DATA block (length 2*NN + 2), never a header.
# The previous model read it as a variable-width segment header of length
# 2*NN + 16, which is why walks drifted and channels came out unequal.
#
# Verified over the 1,388 series-3 waveform binaries in the production
# snapshot: the length chain terminates on a 0x06 record in 1,387 of them (the
# exception has an ambiguous footer, handled by the caller), and all four
# channels come out at identical length in 1,388/1,388 — against 156/1,388
# under the superseded model. Against the 75 events with a preserved
# Blastware ASCII export: sample-count exact 72/75 -> 75/75, fully exact
# 70/75 -> 73/75.
CHANNEL_IDS = {0x46: "Tran", 0x47: "Vert", 0x48: "Long", 0x49: "MicL"}
STREAM_END_ID = 0x06
MODE_DELTA = (0x02, 0x00)
MODE_ABSOLUTE = (0x01, 0x00)
MODE_RAW12 = (0x00, 0x03)
_MODES = (MODE_DELTA, MODE_ABSOLUTE, MODE_RAW12)
def _u16(b: bytes, p: int) -> int:
return (b[p] << 8) | b[p + 1]
def _i16(b: bytes, p: int) -> int:
v = _u16(b, p)
return v - 0x10000 if v >= 0x8000 else v
def data_block_len(body: bytes, p: int) -> Tuple[Optional[int], Optional[int]]:
"""``(byte_length, n_samples)`` of the data block at *p*, or ``(None, None)``.
Data-section blocks only — there is no segment-header tag in this model.
``30 NN`` has no trailer-length fallback here; that fallback corrupted
records whose ``30 NN`` sat near a record boundary.
"""
if p + 2 > len(body):
return None, None
t0, t1 = body[p], body[p + 1]
hi = t0 & 0xF0
nn = ((t0 & 0x0F) << 8) | t1
if hi == 0x40: # int16 BE data block
return (None, None) if (nn == 0 or nn > 0x08) else (2 * nn + 2, nn)
if nn == 0 or nn % 4:
return None, None
if hi == 0x00:
return 2, nn # RLE hold
if hi == 0x10:
return nn // 2 + 2, nn # 4-bit nibble
if hi == 0x20:
return nn + 2, nn # int8
if hi == 0x30:
return nn * 3 // 2 + 2, nn # 12-bit packed
return None, None
def unpack12(data: bytes) -> List[int]:
"""Raw 12-bit packed samples: 6 bytes -> 4 signed values."""
out: List[int] = []
for g in range(len(data) // 6):
hi = (data[6 * g] << 8) | data[6 * g + 1]
for k in range(4):
x = (((hi >> (12 - 4 * k)) & 0xF) << 8) | data[6 * g + 2 + k]
out.append(x - 0x1000 if x >= 0x800 else x)
return out
def is_record(body: bytes, p: int) -> bool:
"""True if a per-channel record header starts at *p*."""
return (p + 10 <= len(body)
and body[p + 4] in CHANNEL_IDS
and body[p + 5] == 0x00 and body[p + 6] == 0x00
and 8 <= _u16(body, p + 2) <= len(body) - p
and (body[p + 8], body[p + 9]) in _MODES)
def find_first_record(body: bytes) -> Optional[int]:
"""Offset of the first record, or None.
Under the normal ``00 02 00`` preamble the leading bytes are segment-0's
Tran blocks, so walk them. Under the ``00 00 03`` preamble that data is
raw 12-bit with no tags at all and cannot be block-walked — scan instead.
"""
if len(body) >= 3 and (body[1], body[2]) == MODE_RAW12:
scan_from = 3
else:
i = 7
while i < len(body):
if is_record(body, i):
nxt = i + 2 + _u16(body, i + 2)
if nxt + 5 <= len(body) and (is_record(body, nxt)
or body[nxt + 4] == STREAM_END_ID):
return i
length, _ = data_block_len(body, i)
if length is None:
return None
i += length
return None
for i in range(scan_from, max(scan_from, len(body) - 10)):
if is_record(body, i):
nxt = i + 2 + _u16(body, i + 2)
if nxt + 5 <= len(body) and (is_record(body, nxt)
or body[nxt + 4] == STREAM_END_ID):
return i
return None
def walk_records(body: bytes, first: Optional[int] = None) -> List[dict]:
"""Follow the length chain from *first* to the ``0x06`` terminator."""
if first is None:
first = find_first_record(body)
out: List[dict] = []
if first is None:
return out
p, seen = first, set()
while p is not None and p + 10 <= len(body):
if p in seen:
break
seen.add(p)
cid = body[p + 4]
if cid == STREAM_END_ID or cid not in CHANNEL_IDS:
break
length = _u16(body, p + 2)
if length < 8 or p + 2 + length > len(body):
break
out.append({"offset": p, "channel": CHANNEL_IDS[cid],
"segment_index": body[p + 7],
"mode": (body[p + 8], body[p + 9]),
"end": p + 2 + length})
p += 2 + length
return out
def decode_waveform_v2(body: bytes) -> Optional[dict]:
"""Decode a Blastware waveform body into per-channel sample arrays.
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}``
in 16-count units (LSB = 0.005 in/s at Normal range), or None if *body*
is not a decodable waveform body.
Implements the record-chain model documented above.
"""
if len(body) < 8 or body[0] != 0x00:
return None
preamble = (body[1], body[2])
if preamble not in (MODE_DELTA, MODE_RAW12):
return None
first = find_first_record(body)
if first is None:
return None
out: dict = {c: [] for c in ("Tran", "Vert", "Long", "MicL")}
def run(channel: str, start: int, end: int, absolute: bool) -> None:
cur = out[channel][-1] if out[channel] else 0
i = start
while i < end:
length, nn = data_block_len(body, i)
if length is None or i + length > end:
return # stop this record; the chain resyncs at end
hi = body[i] & 0xF0
if hi == 0x00:
vals = [None] * nn
elif hi == 0x10:
vals = []
for k in range(nn):
byte = body[i + 2 + k // 2]
v = (byte >> 4) if k % 2 == 0 else (byte & 0xF)
vals.append(v - 16 if v >= 8 else v)
elif hi == 0x20:
vals = [v - 256 if v >= 128 else v
for v in body[i + 2:i + 2 + nn]]
elif hi == 0x30:
vals = unpack12(body[i + 2:i + length])
else:
vals = [_i16(body, i + 2 + 2 * k) for k in range(nn)]
for v in vals:
if v is None:
pass # RLE hold, in delta AND absolute modes
elif absolute:
cur = v
else:
cur += v
out[channel].append(cur)
i += length
# Segment 0 is an implicit Tran record carried in the preamble.
if preamble == MODE_DELTA:
out["Tran"].extend([_i16(body, 3), _i16(body, 5)])
run("Tran", 7, first, absolute=False)
else:
out["Tran"].extend(unpack12(body[3:first]))
for rec in walk_records(body, first):
ch, off, mode, end = (rec["channel"], rec["offset"],
rec["mode"], rec["end"])
if mode == MODE_DELTA:
out[ch].extend([_i16(body, off + 10), _i16(body, off + 12)])
run(ch, off + 14, end, absolute=False)
elif mode == MODE_ABSOLUTE:
run(ch, off + 10, end, absolute=True)
elif mode == MODE_RAW12:
out[ch].extend(unpack12(body[off + 10:end]))
return out