serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(series4): Thor/Micromate decoder is now per-sample exact

Browse Source 
Verified against Thor's own CSV exports, which carry a per-sample
four-column block beside every binary (CSV/<name>.IDFW.csv). Those 1,012
paired files were in the corpus all along; the decoder had been pinned to
a superseded walker on the stated grounds that "Thor has no ASCII ground
truth in the corpus and its geo scaling is separately suspect". Both
premises were false.

  IDFW per-sample exact      39.1%  -> 100.000% (1,057,536/1,057,536)
  IDFW files fully exact     0/153  -> 153/153
  IDFW PPV median error      -3.32% -> -0.002%
  IDFH within 2% of Thor PPV 51.1%  -> 100.0% (858/858)
  prod IDFW, 8 units         -3.3%  -> -0.001%

Four independent root causes:

- Geo LSB was 0.0003, the 4-dp *display rounding* of the real
  0.000310308 mistaken for the LSB, so every series-4 geophone sample
  read 3.3% low. Pinned to +-6e-11 by intersecting 991,415 rounding
  constraints; corroborated by the +-full-scale seed (+-32226) left in
  unwritten IDFH slots. IDFH had a separate, also wrong, 10.0/32768.

- IDFH histograms were capped at 250 intervals: the segment validator
  required the interval counter's high byte to be zero, but the counter
  is a uint16 cumulative index, so every segment past interval 255 was
  rejected. Runs over ~4 hours lost their tail, often the peak.
  540/858 corpus files affected.

- Record mode 00 00 (raw int16, 10-byte header) was unhandled and fell
  through the dispatch, silently dropping each channel's first 512
  samples -- the long-standing "loud events truncate" symptom.
  MODE_ABSOLUTE is now also accepted as a segment-0 preamble.

- The body-offset search matched 00 02 00 *inside* record headers,
  selecting a candidate part-way down the chain and decoding a
  rotation-shifted body. It now anchors on record headers and takes the
  chain head (6 ms/file).

Also fixes the separately tracked "UM-series decodes ~1000x low" bug.
Series-3 re-verified unchanged at 14,338/14,338 exact after the shared
waveform_codec change.

Known open: 41/575 prod IDFW files (7%, mostly UM12947/UM20147) decode
with unequal channel lengths and also fail metadata extraction -- a
different header variant with no Thor export in the store.

NOTE: this is a codec change; the Thor store owes a regeneration via
scripts/backfill_thor_events.py.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
serversdown
2026-09-10 18:06:14 +00:00
co-authored by Claude Opus 5
parent 91b9b4578c
commit 726c2ce1b5
8 changed files with 949 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/idf_protocol_reference.md
+155 -1
View File
@@ -6,7 +6,15 @@ Series IV event-file format. Sibling to
Series III "Rosetta Stone") — this doc holds what we know so far and
the open questions still to crack.
**Status (2026-05-28):** ASCII text sidecar fully decoded (1,014
> ⚠ **The "Status (2026-05-28)" block below is SUPERSEDED.** Its geo LSB
> (0.0003), its IDFH scale (`/32768 × 10`), its fixed body offset (`0x0f1f`)
> and its "87–99% byte-exact / loud events truncate" caveat were all wrong or
> incomplete. See **[Verified against Thor's own exports
> (2026-09-10)](#verified-against-thors-own-exports-2026-09-10)** — the
> decoder is now per-sample exact on 1,057,536/1,057,536 samples. The block
> is kept only for the reverse-engineering trail.
**Status (2026-05-28, SUPERSEDED):** ASCII text sidecar fully decoded (1,014
sample files round-trip). **Thor IDFW** binary now decodes via
`micromate.idf_file.read_idf_file()` — reuses the BW segment-rotated
block codec verbatim at fixed body offset `0x0f1f`; metadata (serial,
@@ -44,6 +52,152 @@ signature and raises `NotImplementedError` pointing callers at
time-of-peak); the two uint16 fields (probably PVS contributions);
8-byte interval tail (PVS data); mic dB(L) exact conversion constant.
## Verified against Thor's own exports (2026-09-10)
**The series-4 decoder is now per-sample exact.** 1,057,536 / 1,057,536
geophone samples across all 153 genuine Thor waveform files reproduce Thor's
own CSV export exactly; histogram peaks land within 2% on 858/858 files
(median error −0.004%).
### Ground truth — it was there all along
Thor writes `TXT/`, `CSV/`, `XML/` and `PDF/` exports beside every binary:
```
<serial dir>/UM13981_20220207084555.IDFW
<serial dir>/CSV/UM13981_20220207084555.IDFW.csv
```
The **CSV carries a per-sample block** — four columns (Tran, Vert, Long, Mic)
in in/s and psi, after the 2-column report header. That is the series-4
equivalent of Blastware's `_ASCII.TXT` exports, and it gives 1,012 paired
files (152 IDFW + 860 IDFH). Earlier notes in this file and in
`micromate/idf_file.py` asserted "Thor has no ASCII ground truth in the
corpus"; that was wrong, and it is why the decoder sat pinned to a
superseded walker with a scaling constant nobody could check.
Harness: `scratch/verify_thor_against_csv.py`.
### Geo LSB = 0.000310308 in/s per count (NOT 0.0003)
The old 0.0003 was read off the smallest non-zero sample in the exports —
but that is Thor's **4-decimal display rounding of the LSB, not the LSB**.
It read every series-4 geophone sample **3.3% low**. The quantisation
ladder gives it away: counts 1..6 export as 0.0003, 0.0006, 0.0009, 0.0012,
0.0016, 0.0019 — an LSB of exactly 0.0003 would end 0.0015, 0.0018.
Each exported sample constrains the LSB to the window that rounds to its
printed value. Intersecting 991,415 such constraints gives
```
LSB ∈ [0.000310307933, 0.000310308057] width 1.2e-10
```
so `_GEO_LSB_IPS = 0.000310308`, i.e. full scale 10.0 in/s = **32226.05
counts**. Corroboration: an IDFH interval that never recorded keeps its
min/max accumulator at its ±full-scale seed, and that seed is
`(min=+32226, max=-32226)`. ⚠ The tempting closed form `10.0/32226` is
very slightly wrong — it lands 4.5e-10 above the feasible window and loses
78 boundary samples while never winning one. **Series III uses 32000 counts
for the same 10.0 in/s, so the two generations do not share a scale.**
Independently confirmed on 8 production units (UM6047, UM11402, UM11719,
UM12947, UM13981, UM14133, UM20146, UM20147): every unit's median PPV error
against its device-reported peak moved from −3.3% to within ±0.03%. It is a
global constant, not a per-unit calibration.
### IDFH segment header: the counter is a uint16, and it is cumulative
```
[length_be 2B][0a 00 00 00][counter_be 2B][05 3f]
```
`counter` is the **0-based cumulative index of the last interval in the
segment** — 9, 19, 29, ... for the usual 10-intervals-per-segment layout
(`length` = 730).
The validator used to require `counter`'s high byte to be `0x00`. That
silently **capped every histogram at 250 intervals**: once the cumulative
counter passed 255 the high byte went non-zero and every later segment was
rejected. Any run longer than ~4 hours lost its tail — frequently the part
holding the event peak, so the file's PPV read low. **540 of 858 corpus
files were affected**; fixing it moved histogram peaks from 48.3% to 93.8%
within 0.5% of Thor's reported PPV.
### Unwritten interval slots carry a ±full-scale seed
An interval the device reserved but never wrote keeps `min = +32226`,
`max = -32226` on all four channels — `min > max`, impossible for real data.
Decoded naively it yields a 10.0 in/s peak on every channel and, being a
max-over-intervals, poisons the whole file's PPV. Rare but real: exactly 1
of 497,611 corpus intervals, and it inflated that file's Long PPV from
0.0081 to 10.0 in/s. The inversion is all-or-nothing across channels (0
partial cases), so requiring every channel to be inverted is a safe test.
### Record mode `00 00` — raw int16 absolute (MODE_RAW16)
The record chain's mode field at `off+8` takes a fourth value:
| mode | meaning | header |
|---|---|---|
| `02 00` | deltas + two int16 anchors | 14 B |
| `01 00` | absolute, tagged blocks | 10 B |
| `00 03` | raw 12-bit absolute, untagged | 10 B |
| **`00 00`** | **raw int16 BE absolute, untagged** | **10 B** |
A `MODE_RAW16` record with `length = 1032` carries exactly
`(1032 - 8) / 2 = 512` samples and reproduced Thor's export **512/512
exactly** on first test. Thor uses it for segment 0 (the pre-trigger
window) on some events. Before this mode existed the record fell through
the dispatch unhandled, so the channel silently lost its first 512 samples —
which is what produced the "loud events truncate" symptom.
`MODE_ABSOLUTE` is also valid as a **preamble** (the implicit segment-0 Tran
record); its tagged blocks start at `body[3]`, not `body[7]`, because its
header is 10 bytes rather than 14.
### Body offset is not fixed at 0x0f1f — and 0x0f1f is really a record + 7
A "body offset" is `<record start> + 7`, so that `body[0]` is the segment
index and `body[1:3]` is the mode. The canonical `0x0f1f` is simply the
record at `0x0f18`.
Searching for the literal preamble `00 02 00` finds only MODE_DELTA bodies,
and worse, it **matches the `[seg][mode]` bytes inside any record header**,
so the scan could pick a candidate part-way down the chain. That decodes a
plausible-looking but rotation-shifted body which drops each channel's
segment 0 — the real cause of the remaining truncations.
`_find_waveform_body_offset()` now anchors on record headers (the
`<channel_id> 00 00` signature at `+4`, validated with `is_record()`),
takes the **chain head** — a record no other record's length field points at
— and trial-decodes `head + 7`, preferring the candidate where all four
channels come out the same length.
⚠ Do **not** scan for candidate preambles instead: `MODE_RAW16` is
`00 00`, so every run of three zero bytes looks like a body start and each
costs a full trial decode (~0.5 s/file measured, vs 6 ms/file now).
### What is still open
- **41 of 575 production IDFW files (7%)** still decode with unequal channel
lengths — signature `Tran/Long 3072, Vert 2560, MicL 0`, and
`sample_rate`/`record_time` also fail to extract, so their header layout
differs. Concentrated in UM12947 (32) and UM20147 (8). No Thor export
exists for them in the production store, so **do not guess a fix** — pull
the paired CSV exports for those events first. Their PPV is mostly still
right (median error −0.001%, 74.8% within 1%).
- Mic → psi scale is still the rough `2.14e-6` regression, not derived.
- Per-channel `int16 field4` in the IDFH interval record (possibly
time-of-peak) and the 8-byte tail (PVS data) remain undecoded.
⚠ **Thor's histogram PPV has a display floor of 0.0050 in/s.** In the
production store 6,080 sidecar PPV values are exactly 0.0050 (next most
common value: 275 occurrences), and **41.4% of IDFH sidecars report a
component PPV larger than their own vector sum** — geometrically impossible.
On those quiet files the decoder's ~0.0025 in/s is *more* accurate than the
reference; do not "fix" the decoder to match it.
### Codec breakthroughs (2026-05-28)
- **Body offset is a fixed `0x0f1f`** across 151/154 corpus IDFW