fix(series4): support mic-disabled (3-channel) Thor units

Verified against a second Thor corpus (9-10-26-csv-req: UM11402, UM12947,
UM20147) with per-sample CSV exports: 139/139 waveforms exact
(1,273,380/1,273,380 samples) and 877/877 histograms within 2% of Thor's
reported PPV -- up from 66.9% and 56.6%.

Some units run with the microphone disabled, which changes two structural
things that were both hardcoded to the 4-channel shape:

- Waveform body head sat below the scan floor. A 3-channel unit has a
  shorter fixed header and puts its record chain head at 0x0dba, under the
  old _BODY_SCAN_FLOOR of 0x0E00. The scan could not see it and fell
  through to the Vert segment-0 record, decoding a body shifted one
  position around the channel rotation -- Vert came up exactly 512 samples
  short. Floor lowered to 0x0C00. The body-offset scoring also had to stop
  requiring four channels, or `equal` is permanently False for these events
  and the pick falls back to raw sample count.

- Histogram interval record is 56 bytes, not 72. It is
  16 * n_channels + 8, and is not inferable from the segment length alone.
  The interval count now comes from the segment's cumulative counter
  (n = counter - prev_counter) and the stride is derived from it. Assuming
  72 read 7 intervals out of every 10-interval segment, then walked off
  alignment into garbage that decoded as ~10 in/s peaks -- inflating some
  files' PPV by up to 191,000%. Also recovers 4 files that previously
  decoded no intervals at all.

Combined across both corpora: 292/292 waveform files,
2,330,916/2,330,916 samples exact. Production IDFW truncations 41 -> 22.
Series-3 unaffected (no shared-codec change in this commit; last full run
14,338/14,338).

Known open, diagnosed but NOT verified: the remaining 22 unequal + 1 failing
production IDFW files (all UM12947, 2025-07-14..09-23) stop the block walker
on tag 40 0c. data_block_len() caps the 40 NN int16 block at NN > 0x08 while
those files use NN up to 196. Both verified corpora only ever use
NN in {1,2,3,4,8}, so the cap is untested there and lifting it leaves both at
100.000% -- which is not evidence it decodes these correctly. Deliberately
not shipped; needs Thor CSV exports for UM12947 in that date range.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
2026-09-10 20:00:56 +00:00
co-authored by Claude Opus 5
parent 726c2ce1b5
commit c07aaa552c
5 changed files with 240 additions and 37 deletions
+64 -7
View File
@@ -178,15 +178,72 @@ channels come out the same length.
`00 00`, so every run of three zero bytes looks like a body start and each
costs a full trial decode (~0.5 s/file measured, vs 6 ms/file now).
### Mic-disabled units are a distinct shape (2026-09-10, second corpus)
Some units run with the microphone disabled — **3 channels, not 4** — and that
changes two structural things. Confirmed on the `9-10-26-csv-req` corpus
(UM11402, UM12947, UM20147): 139/139 waveforms and 877/877 histograms.
**Waveform: the body starts earlier.** A 3-channel unit has a shorter fixed
header and puts its record chain head at **`0x0dba`**, below the old
`_BODY_SCAN_FLOOR` of `0x0E00`. The head was therefore invisible to the scan,
which fell through to the *Vert* segment-0 record and decoded a body shifted
one position around the channel rotation. The signature is unmistakable:
```
Tran 3072 / Vert 2560 / Long 3072 / MicL 0 <- Vert exactly 512 short
```
46 of 139 files in that corpus were affected; all 46 became per-sample exact
once the floor dropped to `0x0C00`. Note the body-offset scoring also had to
stop requiring four channels — `len(lengths) >= 3`, not `== 4`, or `equal` is
permanently False for these events and the pick falls back to raw sample count.
**Histogram: the interval record is 56 bytes, not 72.**
```
interval_size = 16 × n_channels + 8 (72 for 4 channels, 56 for 3)
```
It is **not a constant**, and it cannot be inferred from `length` alone.
Derive the interval count from the segment counter — it is cumulative, so
`n = counter - previous_counter` — and then `stride = (length - 10) / n`.
`n_channels` follows from `(stride - 8) / 16`.
Assuming 72 read 7 intervals out of each 10-interval segment and then walked
off alignment into garbage that decoded as ~10 in/s peaks — inflating those
files' PPV by up to 191,000%. Fixing it moved the second corpus from 56.6% to
**100.0%** of histograms within 2% of Thor's reported PPV, and recovered 4
files that previously decoded no intervals at all.
### What is still open
- **41 of 575 production IDFW files (7%)** still decode with unequal channel
lengths — signature `Tran/Long 3072, Vert 2560, MicL 0`, and
`sample_rate`/`record_time` also fail to extract, so their header layout
differs. Concentrated in UM12947 (32) and UM20147 (8). No Thor export
exists for them in the production store, so **do not guess a fix** — pull
the paired CSV exports for those events first. Their PPV is mostly still
right (median error −0.001%, 74.8% within 1%).
- **23 of 575 production IDFW files (4%)** still decode with unequal channel
lengths (22) or fail outright (1). All are **UM12947, 2025-07-14 to
2025-09-23**. The `Tran/Vert/Long 3072 / MicL 0` group was the 3-channel
shape above and is fixed; what remains is ragged, e.g.
`T1812 / V2132 / L2324 / M2324`.
**Diagnosed but NOT verified.** These files stop the block walker on tag
`40 0c`, because `data_block_len()` caps the `40 NN` int16 block at
`NN > 0x08`:
```python
if hi == 0x40: # int16 BE data block
return (None, None) if (nn == 0 or nn > 0x08) else (2 * nn + 2, nn)
```
The affected files use NN of 12, 16, 20 … up to 196 — every value above 8
halts the walk, which is why the channel comes up short. Both verified
corpora only ever use NN ∈ {1, 2, 3, 4, 8}, so the cap has never been
exercised, and lifting it leaves both at 100.000%.
⚠ **Do not ship the cap change on that evidence.** "Doesn't regress the
known-good corpus" is not "produces correct values here", and a too-loose
guard can accept a false `40 NN` inside data and emit plausible-but-wrong
samples — the exact failure mode this whole effort was about. It needs
Thor CSV exports for UM12947 events between **2025-07-14 and 2025-09-23**;
the `9-10-26-csv-req` upload starts at 2025-09-25 and misses them.
- Mic → psi scale is still the rough `2.14e-6` regression, not derived.
- Per-channel `int16 field4` in the IDFH interval record (possibly
time-of-peak) and the 8-byte tail (PVS data) remain undecoded.