fix(codec): 40 NN int16 blocks are not capped at NN=8

data_block_len() rejected any `40 NN` block with NN > 0x08. That guard had
no evidence behind it: every corpus available when it was written used only
NN in {1,2,3,4,8}, so it was never exercised. Loud UM12947 events use NN of
12, 16, 20 ... up to 196.

Because walk_body/run stop at the first unrecognised tag rather than
raising, rejecting those blocks surfaced as silently short channels -- e.g.
Tran 1812 / Vert 2132 / Long 2324 on a file whose export carries 2324 for
all three. The real bound is the buffer; the caller additionally clamps to
the record end.

Verified against Thor's own CSV exports for UM12947 (2025-07-14 .. 09-25,
167 waveforms, supplied as CSV.zip):

  length mismatches   22 -> 0
  per-sample exact    1,476,242 / 1,476,249

These are NOT truncated recordings, which was the competing hypothesis --
the exports carry the full sample count.

tests/test_waveform_codec.py asserted the cap as intended behaviour. That
assertion encoded an assumption, not a verified fact, and is replaced with
one pinning the opposite plus the evidence.

Across all three ground-truth corpora: 459 waveform files,
3,807,158 / 3,807,165 samples exact. Production IDFW is now 575/575 with
zero truncations and zero decode failures (median PPV error -0.0007% across
8 units). Series-3 re-verified unchanged at 14,338/14,338.

The 7 residual samples each differ by one 4th-decimal tick and are Thor's
own rounding: intersecting the per-sample rounding constraints over that
corpus is infeasible (binding pair contradict by 2.3e-11, 7e-5 relative),
so no single linear LSB reproduces every printed value. _GEO_LSB_IPS is
already pinned to ~1e-11; do not retune it to chase these.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
2026-09-11 05:05:46 +00:00
co-authored by Claude Opus 5
parent c07aaa552c
commit 904522a9c5
6 changed files with 143 additions and 40 deletions
+36 -25
View File
@@ -178,6 +178,39 @@ channels come out the same length.
`00 00`, so every run of three zero bytes looks like a body start and each
costs a full trial decode (~0.5 s/file measured, vs 6 ms/file now).
### `40 NN` is not capped at NN=8 (2026-09-11)
`data_block_len()` rejected any `40 NN` int16 block with `NN > 0x08`. The cap
had no evidence behind it — every corpus available when it was written used
only NN ∈ {1, 2, 3, 4, 8}, so it was never exercised. Loud events use much
wider blocks:
| corpus | `40 NN` values | walker stops |
|---|---|---|
| first + 3-channel corpora | 1, 2, 3, 4, 8 | none |
| UM12947 2025-07..09 | 2, 4, 8, **12, 16, 20 … 196** | every value > 8 |
Because `walk_body`/`run` stop at the first unrecognised tag rather than
raising, this surfaced as **silently short channels** — e.g. Tran 1812 /
Vert 2132 / Long 2324 on a file whose export has 2324 for all three. The
real bound is the buffer (and the caller's record end), not a magic constant.
Verified against Thor's exports for UM12947 (2025-07-14 … 2025-09-25, 167
waveforms): length mismatches **22 → 0**, and **1,476,242 / 1,476,249**
samples exact.
⚠ These events are **not** truncated recordings, which was the competing
hypothesis — the exports carry the full sample count.
**The 7 residual samples are Thor's rounding, not ours.** Each differs by
exactly one 4th-decimal tick (e.g. decoded 3.3551 vs export 3.3550).
Intersecting the per-sample rounding constraints over this corpus is
**infeasible** — the binding pair (count 2013 → 0.6247, count 4351 → 1.3501)
contradict by 2.3e-11, i.e. 7e-5 relative. No single linear LSB can
reproduce every printed value, so Thor is not doing plain round-half-up on
`count × LSB`. Do not retune `_GEO_LSB_IPS` to chase these; it is already
pinned to ~1e-11.
### Mic-disabled units are a distinct shape (2026-09-10, second corpus)
Some units run with the microphone disabled — **3 channels, not 4** — and that
@@ -218,32 +251,10 @@ files that previously decoded no intervals at all.
### What is still open
- **23 of 575 production IDFW files (4%)** still decode with unequal channel
lengths (22) or fail outright (1). All are **UM12947, 2025-07-14 to
2025-09-23**. The `Tran/Vert/Long 3072 / MicL 0` group was the 3-channel
shape above and is fixed; what remains is ragged, e.g.
`T1812 / V2132 / L2324 / M2324`.
- ~~23 of 575 production IDFW files~~ — **RESOLVED 2026-09-11.** Production
IDFW is now **575/575** with zero truncations and zero decode failures
(median PPV error −0.0007%). See "`40 NN` is not capped at NN=8" above.
**Diagnosed but NOT verified.** These files stop the block walker on tag
`40 0c`, because `data_block_len()` caps the `40 NN` int16 block at
`NN > 0x08`:
```python
if hi == 0x40: # int16 BE data block
return (None, None) if (nn == 0 or nn > 0x08) else (2 * nn + 2, nn)
```
The affected files use NN of 12, 16, 20 … up to 196 — every value above 8
halts the walk, which is why the channel comes up short. Both verified
corpora only ever use NN ∈ {1, 2, 3, 4, 8}, so the cap has never been
exercised, and lifting it leaves both at 100.000%.
⚠ **Do not ship the cap change on that evidence.** "Doesn't regress the
known-good corpus" is not "produces correct values here", and a too-loose
guard can accept a false `40 NN` inside data and emit plausible-but-wrong
samples — the exact failure mode this whole effort was about. It needs
Thor CSV exports for UM12947 events between **2025-07-14 and 2025-09-23**;
the `9-10-26-csv-req` upload starts at 2025-09-25 and misses them.
- Mic → psi scale is still the rough `2.14e-6` regression, not derived.
- Per-channel `int16 field4` in the IDFH interval record (possibly
time-of-peak) and the 8-byte tail (PVS data) remain undecoded.