fix(codec): 40 NN int16 blocks are not capped at NN=8

data_block_len() rejected any `40 NN` block with NN > 0x08. That guard had
no evidence behind it: every corpus available when it was written used only
NN in {1,2,3,4,8}, so it was never exercised. Loud UM12947 events use NN of
12, 16, 20 ... up to 196.

Because walk_body/run stop at the first unrecognised tag rather than
raising, rejecting those blocks surfaced as silently short channels -- e.g.
Tran 1812 / Vert 2132 / Long 2324 on a file whose export carries 2324 for
all three. The real bound is the buffer; the caller additionally clamps to
the record end.

Verified against Thor's own CSV exports for UM12947 (2025-07-14 .. 09-25,
167 waveforms, supplied as CSV.zip):

  length mismatches   22 -> 0
  per-sample exact    1,476,242 / 1,476,249

These are NOT truncated recordings, which was the competing hypothesis --
the exports carry the full sample count.

tests/test_waveform_codec.py asserted the cap as intended behaviour. That
assertion encoded an assumption, not a verified fact, and is replaced with
one pinning the opposite plus the evidence.

Across all three ground-truth corpora: 459 waveform files,
3,807,158 / 3,807,165 samples exact. Production IDFW is now 575/575 with
zero truncations and zero decode failures (median PPV error -0.0007% across
8 units). Series-3 re-verified unchanged at 14,338/14,338.

The 7 residual samples each differ by one 4th-decimal tick and are Thor's
own rounding: intersecting the per-sample rounding constraints over that
corpus is infeasible (binding pair contradict by 2.3e-11, 7e-5 relative),
so no single linear LSB reproduces every printed value. _GEO_LSB_IPS is
already pinned to ~1e-11; do not retune it to chase these.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
2026-09-11 05:05:46 +00:00
co-authored by Claude Opus 5
parent c07aaa552c
commit 904522a9c5
6 changed files with 143 additions and 40 deletions
+14 -5
View File
@@ -47,11 +47,20 @@ Read this first when picking the project back up.
was invisible and Vert came up exactly 512 short), and the histogram
interval record is **56 bytes, not 72** — `16 × n_channels + 8`, derived per
segment from the cumulative interval counter, never assumed.
**Still open:** 23/575 prod IDFW files (4%), all UM12947 2025-07-14..09-23,
stop the walker on tag `40 0c` — `data_block_len()` caps `40 NN` at NN>8 but
these use up to 196. **Diagnosed, NOT verified** — no Thor export exists for
those dates. Do not ship the cap change without one; see
`docs/idf_protocol_reference.md`.
- **`40 NN` blocks are not capped at NN=8 (2026-09-11).** `data_block_len()`
rejected `NN > 0x08`, a guard with no evidence behind it — the corpora
available when it was written only used NN ∈ {1,2,3,4,8}. Loud UM12947
events use NN up to 196, and since the walker stops at the first
unrecognised tag rather than raising, this surfaced as silently short
channels. Verified on 167 UM12947 waveforms: length mismatches 22 → 0,
1,476,242/1,476,249 samples exact.
- **Production IDFW is now 575/575** — zero truncations, zero decode
failures, median PPV error −0.0007% across 8 units (was 41 truncated + 1
failing, −3.3%). Across all three ground-truth corpora: **459 files,
3,807,158/3,807,165 samples exact**; the 7 stragglers differ by one
4th-decimal tick and are Thor's own rounding — no single linear LSB can
reproduce every printed value (the constraints are infeasible by 7e-5
relative), so do NOT retune `_GEO_LSB_IPS`.
- **Open, not blocking:** 14 sensitive-range files show an exact 8x
(= 10.0/1.25) units discrepancy; `scripts/backfill_sidecars.py --force` also
inserts DB rows for store files that have none (one-time per store) and the