fix(series4): support mic-disabled (3-channel) Thor units
Verified against a second Thor corpus (9-10-26-csv-req: UM11402, UM12947,
UM20147) with per-sample CSV exports: 139/139 waveforms exact
(1,273,380/1,273,380 samples) and 877/877 histograms within 2% of Thor's
reported PPV -- up from 66.9% and 56.6%.
Some units run with the microphone disabled, which changes two structural
things that were both hardcoded to the 4-channel shape:
- Waveform body head sat below the scan floor. A 3-channel unit has a
shorter fixed header and puts its record chain head at 0x0dba, under the
old _BODY_SCAN_FLOOR of 0x0E00. The scan could not see it and fell
through to the Vert segment-0 record, decoding a body shifted one
position around the channel rotation -- Vert came up exactly 512 samples
short. Floor lowered to 0x0C00. The body-offset scoring also had to stop
requiring four channels, or `equal` is permanently False for these events
and the pick falls back to raw sample count.
- Histogram interval record is 56 bytes, not 72. It is
16 * n_channels + 8, and is not inferable from the segment length alone.
The interval count now comes from the segment's cumulative counter
(n = counter - prev_counter) and the stride is derived from it. Assuming
72 read 7 intervals out of every 10-interval segment, then walked off
alignment into garbage that decoded as ~10 in/s peaks -- inflating some
files' PPV by up to 191,000%. Also recovers 4 files that previously
decoded no intervals at all.
Combined across both corpora: 292/292 waveform files,
2,330,916/2,330,916 samples exact. Production IDFW truncations 41 -> 22.
Series-3 unaffected (no shared-codec change in this commit; last full run
14,338/14,338).
Known open, diagnosed but NOT verified: the remaining 22 unequal + 1 failing
production IDFW files (all UM12947, 2025-07-14..09-23) stop the block walker
on tag 40 0c. data_block_len() caps the 40 NN int16 block at NN > 0x08 while
those files use NN up to 196. Both verified corpora only ever use
NN in {1,2,3,4,8}, so the cap is untested there and lifting it leaves both at
100.000% -- which is not evidence it decodes these correctly. Deliberately
not shipped; needs Thor CSV exports for UM12947 in that date range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
@@ -178,15 +178,72 @@ channels come out the same length.
|
||||
`00 00`, so every run of three zero bytes looks like a body start and each
|
||||
costs a full trial decode (~0.5 s/file measured, vs 6 ms/file now).
|
||||
|
||||
### Mic-disabled units are a distinct shape (2026-09-10, second corpus)
|
||||
|
||||
Some units run with the microphone disabled — **3 channels, not 4** — and that
|
||||
changes two structural things. Confirmed on the `9-10-26-csv-req` corpus
|
||||
(UM11402, UM12947, UM20147): 139/139 waveforms and 877/877 histograms.
|
||||
|
||||
**Waveform: the body starts earlier.** A 3-channel unit has a shorter fixed
|
||||
header and puts its record chain head at **`0x0dba`**, below the old
|
||||
`_BODY_SCAN_FLOOR` of `0x0E00`. The head was therefore invisible to the scan,
|
||||
which fell through to the *Vert* segment-0 record and decoded a body shifted
|
||||
one position around the channel rotation. The signature is unmistakable:
|
||||
|
||||
```
|
||||
Tran 3072 / Vert 2560 / Long 3072 / MicL 0 <- Vert exactly 512 short
|
||||
```
|
||||
|
||||
46 of 139 files in that corpus were affected; all 46 became per-sample exact
|
||||
once the floor dropped to `0x0C00`. Note the body-offset scoring also had to
|
||||
stop requiring four channels — `len(lengths) >= 3`, not `== 4`, or `equal` is
|
||||
permanently False for these events and the pick falls back to raw sample count.
|
||||
|
||||
**Histogram: the interval record is 56 bytes, not 72.**
|
||||
|
||||
```
|
||||
interval_size = 16 × n_channels + 8 (72 for 4 channels, 56 for 3)
|
||||
```
|
||||
|
||||
It is **not a constant**, and it cannot be inferred from `length` alone.
|
||||
Derive the interval count from the segment counter — it is cumulative, so
|
||||
`n = counter - previous_counter` — and then `stride = (length - 10) / n`.
|
||||
`n_channels` follows from `(stride - 8) / 16`.
|
||||
|
||||
Assuming 72 read 7 intervals out of each 10-interval segment and then walked
|
||||
off alignment into garbage that decoded as ~10 in/s peaks — inflating those
|
||||
files' PPV by up to 191,000%. Fixing it moved the second corpus from 56.6% to
|
||||
**100.0%** of histograms within 2% of Thor's reported PPV, and recovered 4
|
||||
files that previously decoded no intervals at all.
|
||||
|
||||
### What is still open
|
||||
|
||||
- **41 of 575 production IDFW files (7%)** still decode with unequal channel
|
||||
lengths — signature `Tran/Long 3072, Vert 2560, MicL 0`, and
|
||||
`sample_rate`/`record_time` also fail to extract, so their header layout
|
||||
differs. Concentrated in UM12947 (32) and UM20147 (8). No Thor export
|
||||
exists for them in the production store, so **do not guess a fix** — pull
|
||||
the paired CSV exports for those events first. Their PPV is mostly still
|
||||
right (median error −0.001%, 74.8% within 1%).
|
||||
- **23 of 575 production IDFW files (4%)** still decode with unequal channel
|
||||
lengths (22) or fail outright (1). All are **UM12947, 2025-07-14 to
|
||||
2025-09-23**. The `Tran/Vert/Long 3072 / MicL 0` group was the 3-channel
|
||||
shape above and is fixed; what remains is ragged, e.g.
|
||||
`T1812 / V2132 / L2324 / M2324`.
|
||||
|
||||
**Diagnosed but NOT verified.** These files stop the block walker on tag
|
||||
`40 0c`, because `data_block_len()` caps the `40 NN` int16 block at
|
||||
`NN > 0x08`:
|
||||
|
||||
```python
|
||||
if hi == 0x40: # int16 BE data block
|
||||
return (None, None) if (nn == 0 or nn > 0x08) else (2 * nn + 2, nn)
|
||||
```
|
||||
|
||||
The affected files use NN of 12, 16, 20 … up to 196 — every value above 8
|
||||
halts the walk, which is why the channel comes up short. Both verified
|
||||
corpora only ever use NN ∈ {1, 2, 3, 4, 8}, so the cap has never been
|
||||
exercised, and lifting it leaves both at 100.000%.
|
||||
|
||||
⚠ **Do not ship the cap change on that evidence.** "Doesn't regress the
|
||||
known-good corpus" is not "produces correct values here", and a too-loose
|
||||
guard can accept a false `40 NN` inside data and emit plausible-but-wrong
|
||||
samples — the exact failure mode this whole effort was about. It needs
|
||||
Thor CSV exports for UM12947 events between **2025-07-14 and 2025-09-23**;
|
||||
the `9-10-26-csv-req` upload starts at 2025-09-25 and misses them.
|
||||
- Mic → psi scale is still the rough `2.14e-6` regression, not derived.
|
||||
- Per-channel `int16 field4` in the IDFH interval record (possibly
|
||||
time-of-peak) and the 8-byte tail (PVS data) remain undecoded.
|
||||
|
||||
Reference in New Issue
Block a user