fix(series4): Thor/Micromate decoder is now per-sample exact
Verified against Thor's own CSV exports, which carry a per-sample four-column block beside every binary (CSV/<name>.IDFW.csv). Those 1,012 paired files were in the corpus all along; the decoder had been pinned to a superseded walker on the stated grounds that "Thor has no ASCII ground truth in the corpus and its geo scaling is separately suspect". Both premises were false. IDFW per-sample exact 39.1% -> 100.000% (1,057,536/1,057,536) IDFW files fully exact 0/153 -> 153/153 IDFW PPV median error -3.32% -> -0.002% IDFH within 2% of Thor PPV 51.1% -> 100.0% (858/858) prod IDFW, 8 units -3.3% -> -0.001% Four independent root causes: - Geo LSB was 0.0003, the 4-dp *display rounding* of the real 0.000310308 mistaken for the LSB, so every series-4 geophone sample read 3.3% low. Pinned to +-6e-11 by intersecting 991,415 rounding constraints; corroborated by the +-full-scale seed (+-32226) left in unwritten IDFH slots. IDFH had a separate, also wrong, 10.0/32768. - IDFH histograms were capped at 250 intervals: the segment validator required the interval counter's high byte to be zero, but the counter is a uint16 cumulative index, so every segment past interval 255 was rejected. Runs over ~4 hours lost their tail, often the peak. 540/858 corpus files affected. - Record mode 00 00 (raw int16, 10-byte header) was unhandled and fell through the dispatch, silently dropping each channel's first 512 samples -- the long-standing "loud events truncate" symptom. MODE_ABSOLUTE is now also accepted as a segment-0 preamble. - The body-offset search matched 00 02 00 *inside* record headers, selecting a candidate part-way down the chain and decoding a rotation-shifted body. It now anchors on record headers and takes the chain head (6 ms/file). Also fixes the separately tracked "UM-series decodes ~1000x low" bug. Series-3 re-verified unchanged at 14,338/14,338 exact after the shared waveform_codec change. Known open: 41/575 prod IDFW files (7%, mostly UM12947/UM20147) decode with unequal channel lengths and also fail metadata extraction -- a different header variant with no Thor export in the store. NOTE: this is a codec change; the Thor store owes a regeneration via scripts/backfill_thor_events.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
@@ -4,6 +4,73 @@ All notable changes to seismo-relay are documented here.
|
||||
|
||||
---
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Fixed — series-4 (Thor / Micromate) decoder is now per-sample exact
|
||||
|
||||
Verified against **Thor's own CSV exports**, which carry a per-sample
|
||||
four-column block beside every binary (`CSV/<name>.IDFW.csv`) — 1,012 paired
|
||||
files that had been sitting in the corpus unused. Previous notes asserted
|
||||
"Thor has no ASCII ground truth", which is why the decoder stayed pinned to a
|
||||
superseded walker with an unverifiable scale factor.
|
||||
|
||||
| metric | before | after |
|
||||
|---|---|---|
|
||||
| IDFW per-sample exact | 39.1% | **100.000%** (1,057,536/1,057,536) |
|
||||
| IDFW files fully exact | 0/153 | **153/153** |
|
||||
| IDFW PPV median error | −3.32% | **−0.002%** |
|
||||
| IDFH within 2% of Thor PPV | 51.1% | **100.0%** (858/858) |
|
||||
| prod IDFW PPV median error (8 units) | −3.3% | **−0.001%** |
|
||||
| decode cost | — | 6 ms/file |
|
||||
|
||||
Four independent root causes:
|
||||
|
||||
- **Geo LSB was `0.0003`, should be `0.000310308`** — the old value was Thor's
|
||||
4-decimal *display rounding* of the LSB mistaken for the LSB, so every
|
||||
series-4 geophone sample read **3.3% low**. Pinned to ±6e-11 by
|
||||
intersecting 991,415 rounding constraints; corroborated by the ±full-scale
|
||||
seed (`±32226`) in unwritten IDFH slots. Applies to IDFH too, which had a
|
||||
separate (also wrong) `10.0/32768`.
|
||||
- **IDFH histograms were capped at 250 intervals** — the segment validator
|
||||
required the interval counter's high byte to be zero, but the counter is a
|
||||
uint16 cumulative index, so every segment past interval 255 was rejected.
|
||||
Any run over ~4 hours lost its tail, often the part holding the peak.
|
||||
540/858 corpus files affected.
|
||||
- **Record mode `00 00` (raw int16, 10-byte header) was unhandled** — the
|
||||
record fell through the dispatch, silently dropping each channel's first
|
||||
512 samples. This produced the long-standing "loud events truncate"
|
||||
symptom. `MODE_ABSOLUTE` is now also accepted as a segment-0 preamble.
|
||||
- **Body-offset search matched `00 02 00` inside record headers** — picking a
|
||||
candidate part-way down the chain, which decodes a rotation-shifted body
|
||||
that drops each channel's segment 0. The search now anchors on record
|
||||
headers and takes the chain head.
|
||||
|
||||
Also fixes the separately-tracked "UM-series decodes ~1000× low" bug
|
||||
(`UM11402_20260406130113.IDFW` now matches its device report exactly).
|
||||
|
||||
Series-3 re-verified **unchanged at 14,338/14,338 exact** after the shared
|
||||
`waveform_codec` change.
|
||||
|
||||
⚠ **This is a codec change: the Thor store owes a regeneration.** Run
|
||||
`scripts/backfill_thor_events.py` (bump `TOOL_VERSION` first, or pass
|
||||
`--force`), DB backup first. All stored series-4 `.h5`/sidecar peaks are
|
||||
currently ~3.3% low, and histogram peaks for runs over ~4 hours may be
|
||||
badly low.
|
||||
|
||||
⚠ **Thor's histogram PPV has a 0.0050 in/s display floor** — 41.4% of prod
|
||||
IDFH sidecars report a component PPV larger than their own vector sum. On
|
||||
quiet files the decoder is now *more* accurate than that reference.
|
||||
|
||||
New: `scratch/verify_thor_against_csv.py`, `tests/test_idf_binary_codec.py`
|
||||
(10 tests, fixtures under `tests/fixtures/thor-idf/`).
|
||||
|
||||
**Known open:** 41/575 production IDFW files (7%, mostly UM12947/UM20147)
|
||||
still decode with unequal channel lengths and also fail metadata extraction —
|
||||
a different header variant with no Thor export in the store. Pull their CSV
|
||||
exports before attempting a fix.
|
||||
|
||||
---
|
||||
|
||||
## v0.29.0 — 2026-09-04
|
||||
|
||||
First release to reach prod since **v0.27.0**, so it ships **both** the
|
||||
|
||||
Reference in New Issue
Block a user