fix(series4): Thor/Micromate decoder is now per-sample exact
Verified against Thor's own CSV exports, which carry a per-sample four-column block beside every binary (CSV/<name>.IDFW.csv). Those 1,012 paired files were in the corpus all along; the decoder had been pinned to a superseded walker on the stated grounds that "Thor has no ASCII ground truth in the corpus and its geo scaling is separately suspect". Both premises were false. IDFW per-sample exact 39.1% -> 100.000% (1,057,536/1,057,536) IDFW files fully exact 0/153 -> 153/153 IDFW PPV median error -3.32% -> -0.002% IDFH within 2% of Thor PPV 51.1% -> 100.0% (858/858) prod IDFW, 8 units -3.3% -> -0.001% Four independent root causes: - Geo LSB was 0.0003, the 4-dp *display rounding* of the real 0.000310308 mistaken for the LSB, so every series-4 geophone sample read 3.3% low. Pinned to +-6e-11 by intersecting 991,415 rounding constraints; corroborated by the +-full-scale seed (+-32226) left in unwritten IDFH slots. IDFH had a separate, also wrong, 10.0/32768. - IDFH histograms were capped at 250 intervals: the segment validator required the interval counter's high byte to be zero, but the counter is a uint16 cumulative index, so every segment past interval 255 was rejected. Runs over ~4 hours lost their tail, often the peak. 540/858 corpus files affected. - Record mode 00 00 (raw int16, 10-byte header) was unhandled and fell through the dispatch, silently dropping each channel's first 512 samples -- the long-standing "loud events truncate" symptom. MODE_ABSOLUTE is now also accepted as a segment-0 preamble. - The body-offset search matched 00 02 00 *inside* record headers, selecting a candidate part-way down the chain and decoding a rotation-shifted body. It now anchors on record headers and takes the chain head (6 ms/file). Also fixes the separately tracked "UM-series decodes ~1000x low" bug. Series-3 re-verified unchanged at 14,338/14,338 exact after the shared waveform_codec change. Known open: 41/575 prod IDFW files (7%, mostly UM12947/UM20147) decode with unequal channel lengths and also fail metadata extraction -- a different header variant with no Thor export in the store. NOTE: this is a codec change; the Thor store owes a regeneration via scripts/backfill_thor_events.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
@@ -722,7 +722,18 @@ STREAM_END_ID = 0x06
|
||||
MODE_DELTA = (0x02, 0x00)
|
||||
MODE_ABSOLUTE = (0x01, 0x00)
|
||||
MODE_RAW12 = (0x00, 0x03)
|
||||
_MODES = (MODE_DELTA, MODE_ABSOLUTE, MODE_RAW12)
|
||||
# Raw int16 BE absolute samples, 10-byte header, no tags — the same shape as
|
||||
# MODE_RAW12 but two bytes per sample instead of 1.5. Found on Thor/Micromate
|
||||
# segment-0 records (2026-09-10): a `len=1032` record carries exactly
|
||||
# (1032 - 8) / 2 = 512 samples and reproduces Thor's own export 512/512
|
||||
# exactly. Before this mode existed the record fell through the dispatch
|
||||
# unhandled, so the channel silently lost its first 512 samples.
|
||||
MODE_RAW16 = (0x00, 0x00)
|
||||
_MODES = (MODE_DELTA, MODE_ABSOLUTE, MODE_RAW12, MODE_RAW16)
|
||||
|
||||
# Preambles whose leading data is untagged and therefore cannot be
|
||||
# block-walked; find_first_record() must scan for the next record instead.
|
||||
_UNTAGGED_MODES = (MODE_RAW12, MODE_RAW16)
|
||||
|
||||
|
||||
def _u16(b: bytes, p: int) -> int:
|
||||
@@ -761,6 +772,11 @@ def data_block_len(body: bytes, p: int) -> Tuple[Optional[int], Optional[int]]:
|
||||
return None, None
|
||||
|
||||
|
||||
def unpack16(data: bytes) -> List[int]:
|
||||
"""Raw int16 BE absolute samples (MODE_RAW16)."""
|
||||
return [_i16(data, 2 * k) for k in range(len(data) // 2)]
|
||||
|
||||
|
||||
def unpack12(data: bytes) -> List[int]:
|
||||
"""Raw 12-bit packed samples: 6 bytes -> 4 signed values."""
|
||||
out: List[int] = []
|
||||
@@ -785,13 +801,17 @@ def find_first_record(body: bytes) -> Optional[int]:
|
||||
"""Offset of the first record, or None.
|
||||
|
||||
Under the normal ``00 02 00`` preamble the leading bytes are segment-0's
|
||||
Tran blocks, so walk them. Under the ``00 00 03`` preamble that data is
|
||||
raw 12-bit with no tags at all and cannot be block-walked — scan instead.
|
||||
Tran blocks, so walk them. Under the untagged preambles (``00 00 03``
|
||||
raw-12 and ``00 00 00`` raw-16) that data has no tags at all and cannot
|
||||
be block-walked — scan for the next record header instead.
|
||||
"""
|
||||
if len(body) >= 3 and (body[1], body[2]) == MODE_RAW12:
|
||||
if len(body) >= 3 and (body[1], body[2]) in _UNTAGGED_MODES:
|
||||
scan_from = 3
|
||||
else:
|
||||
i = 7
|
||||
# Tagged preamble. MODE_DELTA carries a 14-byte record header (two
|
||||
# int16 anchors), so its blocks start at body[7]; MODE_ABSOLUTE has a
|
||||
# 10-byte header and starts at body[3].
|
||||
i = 3 if (len(body) >= 3 and (body[1], body[2]) == MODE_ABSOLUTE) else 7
|
||||
while i < len(body):
|
||||
if is_record(body, i):
|
||||
nxt = i + 2 + _u16(body, i + 2)
|
||||
@@ -850,7 +870,7 @@ def decode_waveform_v2(body: bytes) -> Optional[dict]:
|
||||
if len(body) < 8 or body[0] != 0x00:
|
||||
return None
|
||||
preamble = (body[1], body[2])
|
||||
if preamble not in (MODE_DELTA, MODE_RAW12):
|
||||
if preamble not in (MODE_DELTA, MODE_ABSOLUTE, MODE_RAW12, MODE_RAW16):
|
||||
return None
|
||||
first = find_first_record(body)
|
||||
if first is None:
|
||||
@@ -895,6 +915,10 @@ def decode_waveform_v2(body: bytes) -> Optional[dict]:
|
||||
if preamble == MODE_DELTA:
|
||||
out["Tran"].extend([_i16(body, 3), _i16(body, 5)])
|
||||
run("Tran", 7, first, absolute=False)
|
||||
elif preamble == MODE_ABSOLUTE:
|
||||
run("Tran", 3, first, absolute=True)
|
||||
elif preamble == MODE_RAW16:
|
||||
out["Tran"].extend(unpack16(body[3:first]))
|
||||
else:
|
||||
out["Tran"].extend(unpack12(body[3:first]))
|
||||
|
||||
@@ -908,4 +932,6 @@ def decode_waveform_v2(body: bytes) -> Optional[dict]:
|
||||
run(ch, off + 10, end, absolute=True)
|
||||
elif mode == MODE_RAW12:
|
||||
out[ch].extend(unpack12(body[off + 10:end]))
|
||||
elif mode == MODE_RAW16:
|
||||
out[ch].extend(unpack16(body[off + 10:end]))
|
||||
return out
|
||||
|
||||
Reference in New Issue
Block a user