Verified against Thor's own CSV exports, which carry a per-sample
four-column block beside every binary (CSV/<name>.IDFW.csv). Those 1,012
paired files were in the corpus all along; the decoder had been pinned to
a superseded walker on the stated grounds that "Thor has no ASCII ground
truth in the corpus and its geo scaling is separately suspect". Both
premises were false.
IDFW per-sample exact 39.1% -> 100.000% (1,057,536/1,057,536)
IDFW files fully exact 0/153 -> 153/153
IDFW PPV median error -3.32% -> -0.002%
IDFH within 2% of Thor PPV 51.1% -> 100.0% (858/858)
prod IDFW, 8 units -3.3% -> -0.001%
Four independent root causes:
- Geo LSB was 0.0003, the 4-dp *display rounding* of the real
0.000310308 mistaken for the LSB, so every series-4 geophone sample
read 3.3% low. Pinned to +-6e-11 by intersecting 991,415 rounding
constraints; corroborated by the +-full-scale seed (+-32226) left in
unwritten IDFH slots. IDFH had a separate, also wrong, 10.0/32768.
- IDFH histograms were capped at 250 intervals: the segment validator
required the interval counter's high byte to be zero, but the counter
is a uint16 cumulative index, so every segment past interval 255 was
rejected. Runs over ~4 hours lost their tail, often the peak.
540/858 corpus files affected.
- Record mode 00 00 (raw int16, 10-byte header) was unhandled and fell
through the dispatch, silently dropping each channel's first 512
samples -- the long-standing "loud events truncate" symptom.
MODE_ABSOLUTE is now also accepted as a segment-0 preamble.
- The body-offset search matched 00 02 00 *inside* record headers,
selecting a candidate part-way down the chain and decoding a
rotation-shifted body. It now anchors on record headers and takes the
chain head (6 ms/file).
Also fixes the separately tracked "UM-series decodes ~1000x low" bug.
Series-3 re-verified unchanged at 14,338/14,338 exact after the shared
waveform_codec change.
Known open: 41/575 prod IDFW files (7%, mostly UM12947/UM20147) decode
with unequal channel lengths and also fail metadata extraction -- a
different header variant with no Thor export in the store.
NOTE: this is a codec change; the Thor store owes a regeneration via
scripts/backfill_thor_events.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
Tighten the Series III / Series IV boundary so UI and storage dispatch
on a clean signal instead of sniffing filenames or applying magnitude
heuristics.
Phase 1 — events.device_family column ("series3" | "series4"):
self-applying migration with filename-based backfill of existing rows
(1,132 backfilled on prod 2026-05-20); plumbed through every import
path (BW endpoint, IDF endpoint, ACH server, BW CLI, sidecar
backfill); UPSERT preserves via COALESCE; UI dispatches on it.
Phase 2 — extract micromate/ package alongside minimateplus/:
native IdfEvent / IdfReport / IdfPeaks / IdfProjectInfo /
IdfSensorCheck (mic in dB(L), not pseudo-psi); moved
idf_ascii_report.py from sfm/ to micromate/; refactored
save_imported_idf to use IdfEvent and bridge to minimateplus.Event at
the SQL-insert boundary; idf_file.py stub for the future binary codec.
Phase 3 prep — docs/idf_protocol_reference.md captures the two
observed Thor binary header signatures (1,012 newer-firmware files vs
2 old files whose layout is byte-for-byte BW-STRT-compatible), file-size
hints suggesting int8 sample encoding, open questions in dependency
order, and a concrete first-session plan for cracking the codec.
Also rolled in the v0.18.1 hotfixes that motivated this work:
- idf_ascii_report parser now handles "<0.005 in/s" (below-threshold)
and "N/A" markers without leaving raw strings in numeric DB columns.
- sfm_webapp.html: defensive _ppvFmt / mic formatter so future
data-shape drift can't kill the whole events table render.
All 1,014 example-data sidecars round-trip through the new package.
See CHANGELOG.md for full notes.
serversdownandClaude Opus 5