fix(series4): support mic-disabled (3-channel) Thor units

Verified against a second Thor corpus (9-10-26-csv-req: UM11402, UM12947,
UM20147) with per-sample CSV exports: 139/139 waveforms exact
(1,273,380/1,273,380 samples) and 877/877 histograms within 2% of Thor's
reported PPV -- up from 66.9% and 56.6%.

Some units run with the microphone disabled, which changes two structural
things that were both hardcoded to the 4-channel shape:

- Waveform body head sat below the scan floor. A 3-channel unit has a
  shorter fixed header and puts its record chain head at 0x0dba, under the
  old _BODY_SCAN_FLOOR of 0x0E00. The scan could not see it and fell
  through to the Vert segment-0 record, decoding a body shifted one
  position around the channel rotation -- Vert came up exactly 512 samples
  short. Floor lowered to 0x0C00. The body-offset scoring also had to stop
  requiring four channels, or `equal` is permanently False for these events
  and the pick falls back to raw sample count.

- Histogram interval record is 56 bytes, not 72. It is
  16 * n_channels + 8, and is not inferable from the segment length alone.
  The interval count now comes from the segment's cumulative counter
  (n = counter - prev_counter) and the stride is derived from it. Assuming
  72 read 7 intervals out of every 10-interval segment, then walked off
  alignment into garbage that decoded as ~10 in/s peaks -- inflating some
  files' PPV by up to 191,000%. Also recovers 4 files that previously
  decoded no intervals at all.

Combined across both corpora: 292/292 waveform files,
2,330,916/2,330,916 samples exact. Production IDFW truncations 41 -> 22.
Series-3 unaffected (no shared-codec change in this commit; last full run
14,338/14,338).

Known open, diagnosed but NOT verified: the remaining 22 unequal + 1 failing
production IDFW files (all UM12947, 2025-07-14..09-23) stop the block walker
on tag 40 0c. data_block_len() caps the 40 NN int16 block at NN > 0x08 while
those files use NN up to 196. Both verified corpora only ever use
NN in {1,2,3,4,8}, so the cap is untested there and lifting it leaves both at
100.000% -- which is not evidence it decodes these correctly. Deliberately
not shipped; needs Thor CSV exports for UM12947 in that date range.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL

CHANGELOG.md

@@ -64,10 +64,36 @@ quiet files the decoder is now *more* accurate than that reference.
 New: `scratch/verify_thor_against_csv.py`, `tests/test_idf_binary_codec.py`
 (10 tests, fixtures under `tests/fixtures/thor-idf/`).
 
-**Known open:** 41/575 production IDFW files (7%, mostly UM12947/UM20147)
+### Fixed — mic-disabled (3-channel) units
-still decode with unequal channel lengths and also fail metadata extraction —
+
-a different header variant with no Thor export in the store.  Pull their CSV
+Verified on a second corpus (`9-10-26-csv-req`: UM11402, UM12947, UM20147) —
-exports before attempting a fix.
+**139/139 waveforms per-sample exact (1,273,380 samples), 877/877 histograms
+within 2%** (was 66.9% and 56.6%).
+
+- **Waveform body head sat below the scan floor.** A 3-channel unit's shorter
+  header puts the record chain head at `0x0dba`, under the old
+  `_BODY_SCAN_FLOOR` of `0x0E00`.  The scan couldn't see it and fell through
+  to the Vert segment-0 record, decoding a body shifted one position around
+  the channel rotation — Vert came up exactly 512 samples short.  Floor
+  lowered to `0x0C00`; body-offset scoring now accepts 3 channels as "equal"
+  instead of demanding 4.
+- **Histogram interval record is 56 bytes, not 72.**  It is
+  `16 × n_channels + 8`, so mic-disabled units pack 56.  Assuming 72 read 7
+  intervals out of every 10-interval segment then walked off alignment into
+  garbage decoding as ~10 in/s peaks (errors up to +191,000%).  The interval
+  count now comes from the segment's cumulative counter and the stride is
+  derived from it; also recovers 4 files that decoded no intervals at all.
+
+Combined across both corpora: **292/292 waveform files, 2,330,916/2,330,916
+samples exact.**  Production IDFW truncations 41 → 22.
+
+**Known open — diagnosed but NOT verified:** 23/575 production IDFW files
+(4%), all UM12947 between 2025-07-14 and 2025-09-23, stop the block walker on
+tag `40 0c`.  `data_block_len()` caps the `40 NN` int16 block at `NN > 0x08`,
+but these files use NN up to 196.  Both verified corpora only ever use
+NN ∈ {1,2,3,4,8}, so the cap is untested there and lifting it leaves both at
+100.000% — which is *not* evidence it decodes these correctly.  Needs Thor CSV
+exports for UM12947 in that date range (the 9-10-26 upload starts 2025-09-25).
 
 ---
 

CLAUDE.md

@@ -38,9 +38,20 @@ Read this first when picking the project back up.
   matched `00 02 00` *inside* record headers, decoding a rotation-shifted
   body.  IDFW is no longer pinned to `decode_waveform_legacy`.
   Series-3 re-verified unchanged at 14,338/14,338 after the shared-codec
-  change.  **Still open:** 41/575 prod IDFW files (7%, mostly UM12947) decode
+  change.
-  with unequal channel lengths and have no Thor export — pull their CSVs
+- **Mic-disabled (3-channel) units are a distinct shape (2026-09-10).**
-  before touching it.
+  Verified on a second corpus (`~/thor-csv-req`, UM11402/UM12947/UM20147):
+  **139/139** waveforms per-sample exact, **877/877** histograms within 2%.
+  Two structural differences: the shorter header puts the waveform record
+  chain head at `0x0dba` (below the old `_BODY_SCAN_FLOOR` of `0x0E00`, so it
+  was invisible and Vert came up exactly 512 short), and the histogram
+  interval record is **56 bytes, not 72** — `16 × n_channels + 8`, derived per
+  segment from the cumulative interval counter, never assumed.
+  **Still open:** 23/575 prod IDFW files (4%), all UM12947 2025-07-14..09-23,
+  stop the walker on tag `40 0c` — `data_block_len()` caps `40 NN` at NN>8 but
+  these use up to 196.  **Diagnosed, NOT verified** — no Thor export exists for
+  those dates.  Do not ship the cap change without one; see
+  `docs/idf_protocol_reference.md`.
 - **Open, not blocking:** 14 sensitive-range files show an exact 8x
   (= 10.0/1.25) units discrepancy; `scripts/backfill_sidecars.py --force` also
   inserts DB rows for store files that have none (one-time per store) and the

docs/idf_protocol_reference.md

@@ -178,15 +178,72 @@ channels come out the same length.
 `00 00`, so every run of three zero bytes looks like a body start and each
 costs a full trial decode (~0.5 s/file measured, vs 6 ms/file now).
 
+### Mic-disabled units are a distinct shape (2026-09-10, second corpus)
+
+Some units run with the microphone disabled — **3 channels, not 4** — and that
+changes two structural things.  Confirmed on the `9-10-26-csv-req` corpus
+(UM11402, UM12947, UM20147): 139/139 waveforms and 877/877 histograms.
+
+**Waveform: the body starts earlier.**  A 3-channel unit has a shorter fixed
+header and puts its record chain head at **`0x0dba`**, below the old
+`_BODY_SCAN_FLOOR` of `0x0E00`.  The head was therefore invisible to the scan,
+which fell through to the *Vert* segment-0 record and decoded a body shifted
+one position around the channel rotation.  The signature is unmistakable:
+
+```
+Tran 3072 / Vert 2560 / Long 3072 / MicL 0      <- Vert exactly 512 short
+```
+
+46 of 139 files in that corpus were affected; all 46 became per-sample exact
+once the floor dropped to `0x0C00`.  Note the body-offset scoring also had to
+stop requiring four channels — `len(lengths) >= 3`, not `== 4`, or `equal` is
+permanently False for these events and the pick falls back to raw sample count.
+
+**Histogram: the interval record is 56 bytes, not 72.**
+
+```
+interval_size = 16 × n_channels + 8        (72 for 4 channels, 56 for 3)
+```
+
+It is **not a constant**, and it cannot be inferred from `length` alone.
+Derive the interval count from the segment counter — it is cumulative, so
+`n = counter - previous_counter` — and then `stride = (length - 10) / n`.
+`n_channels` follows from `(stride - 8) / 16`.
+
+Assuming 72 read 7 intervals out of each 10-interval segment and then walked
+off alignment into garbage that decoded as ~10 in/s peaks — inflating those
+files' PPV by up to 191,000%.  Fixing it moved the second corpus from 56.6% to
+**100.0%** of histograms within 2% of Thor's reported PPV, and recovered 4
+files that previously decoded no intervals at all.
+
 ### What is still open
 
-- **41 of 575 production IDFW files (7%)** still decode with unequal channel
+- **23 of 575 production IDFW files (4%)** still decode with unequal channel
-  lengths — signature `Tran/Long 3072, Vert 2560, MicL 0`, and
+  lengths (22) or fail outright (1).  All are **UM12947, 2025-07-14 to
-  `sample_rate`/`record_time` also fail to extract, so their header layout
+  2025-09-23**.  The `Tran/Vert/Long 3072 / MicL 0` group was the 3-channel
-  differs.  Concentrated in UM12947 (32) and UM20147 (8).  No Thor export
+  shape above and is fixed; what remains is ragged, e.g.
-  exists for them in the production store, so **do not guess a fix** — pull
+  `T1812 / V2132 / L2324 / M2324`.
-  the paired CSV exports for those events first.  Their PPV is mostly still
+
-  right (median error −0.001%, 74.8% within 1%).
+  **Diagnosed but NOT verified.** These files stop the block walker on tag
+  `40 0c`, because `data_block_len()` caps the `40 NN` int16 block at
+  `NN > 0x08`:
+
+  ```python
+  if hi == 0x40:                     # int16 BE data block
+      return (None, None) if (nn == 0 or nn > 0x08) else (2 * nn + 2, nn)
+  ```
+
+  The affected files use NN of 12, 16, 20 … up to 196 — every value above 8
+  halts the walk, which is why the channel comes up short.  Both verified
+  corpora only ever use NN ∈ {1, 2, 3, 4, 8}, so the cap has never been
+  exercised, and lifting it leaves both at 100.000%.
+
+  ⚠ **Do not ship the cap change on that evidence.** "Doesn't regress the
+  known-good corpus" is not "produces correct values here", and a too-loose
+  guard can accept a false `40 NN` inside data and emit plausible-but-wrong
+  samples — the exact failure mode this whole effort was about.  It needs
+  Thor CSV exports for UM12947 events between **2025-07-14 and 2025-09-23**;
+  the `9-10-26-csv-req` upload starts at 2025-09-25 and misses them.
 - Mic → psi scale is still the rough `2.14e-6` regression, not derived.
 - Per-channel `int16 field4` in the IDFH interval record (possibly
   time-of-peak) and the 8-byte tail (PVS data) remain undecoded.

micromate/idf_file.py

@@ -94,7 +94,15 @@ _BODY_MAGIC = b"\x00\x02\x00"
 # fixed-header region where the same magic legitimately appears inside
 # channel-test records and the compliance block (offsets 0x015d, 0x091c,
 # 0x0ae2, 0x0d30 in observed events).
-_BODY_SCAN_FLOOR = 0x0E00
+# Lowered from 0x0E00 to 0x0C00 (2026-09-10).  Three-channel events -- mic
+# disabled -- have a shorter fixed header and put their record chain head at
+# 0x0dba, below the old floor.  The head was therefore invisible to the scan,
+# which fell through to the *Vert* segment-0 record and decoded a body shifted
+# one position around the channel rotation.  46 of 139 files in the
+# 9-10-26-csv-req corpus were affected; all 46 became per-sample exact once
+# the head was reachable.  The floor still skips the fixed-header region,
+# where `is_record()` can match channel-test records (0x015d, 0x091c, 0x0ae2).
+_BODY_SCAN_FLOOR = 0x0C00
 
 # Cap on trial decodes per file.  Chain-head detection normally yields one
 # or two candidates; the cap only bounds the worst case on a corrupt file.
@@ -140,7 +148,13 @@ _GEO_LSB_IPS = 0.000310308
 _MIC_LSB_PSI = 2.14e-6
 
 # IDFH histogram constants.
-_IDFH_INTERVAL_SIZE = 72        # bytes per per-interval record
+# Bytes per interval record = 16 per channel + an 8-byte tail, so a
+# 4-channel unit uses 72 and a mic-disabled 3-channel unit uses 56.  It is
+# NOT a constant: derive it per segment from the interval counter (see
+# decode_idfh_body).  This value survives only as the 4-channel default.
+_IDFH_INTERVAL_SIZE = 72        # bytes per per-interval record (4 channels)
+_IDFH_CHANNEL_BLOCK = 16        # bytes per channel inside an interval record
+_IDFH_INTERVAL_TAIL = 8         # bytes after the per-channel blocks
 _IDFH_SEGMENT_HEADER = 10       # bytes: [len_be 2B][0a 00 00 00 4B][00 NN 2B][05 3f 2B]
 _IDFH_SEGMENT_TAIL   = 2        # bytes after the interval data block, before next marker
 _IDFH_HALFP_FREQ_NUM = 512.0    # freq_hz = NUM / halfp; halfp ≤ 5 means ">100 Hz" sentinel
@@ -314,7 +328,10 @@ def _find_waveform_body_offset(buf: bytes) -> Optional[int]:
         # A "real" body has more than just the 2-sample preamble.
         if total <= 2:
             continue
-        equal = len(lengths) == 4 and len(set(lengths)) == 1
+        # >= 3 rather than == 4: a mic-disabled event has only the three geo
+        # channels, and demanding four made `equal` permanently False for
+        # them, leaving the pick to raw sample count alone.
+        equal = len(lengths) >= 3 and len(set(lengths)) == 1
         score = (equal, total)
         if best is None or score > best:
             best, best_off = score, j
@@ -375,6 +392,12 @@ class IdfhInterval:
     micl_min:    int
     micl_max:    int
     micl_halfp:  int
+    # 4 on a normal unit; 3 when the microphone is disabled, in which case the
+    # micl_* fields are absent from the record and read as zero.
+    n_channels:  int = 4
+
+    def has_channel(self, channel: str) -> bool:
+        return channel != "MicL" or self.n_channels >= 4
 
     def peak_count(self, channel: str) -> int:
         mn = getattr(self, f"{channel.lower()}_min")
@@ -412,22 +435,30 @@ def _is_unwritten_interval(interval: "IdfhInterval") -> bool:
     requiring every channel to be inverted keeps this from ever firing on
     genuine data.
     """
-    return all(
+    pairs = [
-        mn > mx
+        (interval.tran_min, interval.tran_max),
-        for mn, mx in (
+        (interval.vert_min, interval.vert_max),
-            (interval.tran_min, interval.tran_max),
+        (interval.long_min, interval.long_max),
-            (interval.vert_min, interval.vert_max),
+    ]
-            (interval.long_min, interval.long_max),
+    if interval.has_channel("MicL"):
-            (interval.micl_min, interval.micl_max),
+        pairs.append((interval.micl_min, interval.micl_max))
-        )
+    return all(mn > mx for mn, mx in pairs)
-    )
 
 
-def _decode_idfh_interval(buf72: bytes, offset: int) -> IdfhInterval:
+def _decode_idfh_interval(buf72: bytes, offset: int,
-    """Decode one 72-byte interval record into per-channel min/max/halfp."""
+                          n_channels: int = 4) -> IdfhInterval:
+    """Decode one interval record into per-channel min/max/halfp.
+
+    The record is ``n_channels`` × 16-byte blocks plus an 8-byte tail, so it
+    is 72 bytes on a normal unit and 56 when the microphone is disabled.
+    Missing channels read as zero.
+    """
     import struct
     fields = []
     for i in range(4):
+        if i >= n_channels:
+            fields.extend([0, 0, 0])
+            continue
         block = buf72[i * 16 : (i + 1) * 16]
         mn = struct.unpack_from(">h", block, 0)[0]
         mx = struct.unpack_from(">h", block, 2)[0]
@@ -443,6 +474,7 @@ def _decode_idfh_interval(buf72: bytes, offset: int) -> IdfhInterval:
         vert_min=fields[3], vert_max=fields[4], vert_halfp=fields[5],
         long_min=fields[6], long_max=fields[7], long_halfp=fields[8],
         micl_min=fields[9], micl_max=fields[10], micl_halfp=fields[11],
+        n_channels=n_channels,
     )
 
 
@@ -469,6 +501,7 @@ def decode_idfh_body(buf: bytes) -> list:
     """
     intervals: list = []
     i = 0
+    prev_counter = -1          # so the first segment's n = counter + 1
     while True:
         j = buf.find(b"\x0a\x00\x00\x00", i)
         if j < 0 or j < 2:
@@ -479,28 +512,43 @@ def decode_idfh_body(buf: bytes) -> list:
             i = j + 1
             continue
         length = int.from_bytes(buf[j - 2 : j], "big")
-        n = (length - _IDFH_SEGMENT_HEADER) // _IDFH_INTERVAL_SIZE
+        counter = int.from_bytes(buf[j + 4 : j + 6], "big")
+        header_start = j - 2
+        if length < _IDFH_SEGMENT_HEADER or header_start + length > len(buf):
+            # Truncated / bogus length — not a real segment header.
+            i = j + 1
+            continue
+        # The counter is the cumulative index of this segment's LAST interval,
+        # so the interval count is its delta from the previous segment.  That
+        # gives the record stride, which is NOT fixed: 16 bytes per channel
+        # plus an 8-byte tail, so 72 for a 4-channel unit and 56 for a
+        # mic-disabled 3-channel one.  Assuming 72 unconditionally made every
+        # 3-channel histogram read 7 intervals per 10-interval segment,
+        # walking off alignment into garbage that decoded as ~10 in/s peaks.
+        n = counter - prev_counter
         if n <= 0:
             i = j + 1
             continue
-        header_start = j - 2
+        stride = (length - _IDFH_SEGMENT_HEADER) // n
-        if header_start + length > len(buf):
+        n_channels, remainder = divmod(stride - _IDFH_INTERVAL_TAIL,
-            # Truncated / bogus length — not a real segment header.
+                                       _IDFH_CHANNEL_BLOCK)
+        if remainder or not (1 <= n_channels <= 4):
             i = j + 1
             continue
         interval_start = header_start + _IDFH_SEGMENT_HEADER
         for k in range(n):
-            off = interval_start + k * _IDFH_INTERVAL_SIZE
+            off = interval_start + k * stride
-            if off + _IDFH_INTERVAL_SIZE > len(buf):
+            if off + stride > len(buf):
                 break
-            chunk = buf[off : off + _IDFH_INTERVAL_SIZE]
+            chunk = buf[off : off + stride]
-            interval = _decode_idfh_interval(chunk, off)
+            interval = _decode_idfh_interval(chunk, off, n_channels)
             if _is_unwritten_interval(interval):
                 # Reserved-but-never-recorded slot: the min/max accumulators
                 # still hold their ±full-scale seed.  Counting it would
                 # fabricate a 10.0 in/s peak on every channel.
                 continue
             intervals.append(interval)
+        prev_counter = counter
         # Advance past this segment + the 2-byte tail.
         i = header_start + length + _IDFH_SEGMENT_TAIL
     return intervals
@@ -580,7 +628,12 @@ def read_idf_file(
         peak_long = max((iv.peak_ips("Long") for iv in intervals), default=0.0)
         # Mic peak in psi — Thor stores per-interval mic ADC counts in the
         # binary; convert the max count to psi via the per-count factor.
-        mic_peak_count = max((iv.peak_count("MicL") for iv in intervals), default=0)
+        # Skip on a mic-disabled (3-channel) unit: those records carry no mic
+        # block at all, so peak_count("MicL") would report a synthetic zero.
+        mic_peak_count = max(
+            (iv.peak_count("MicL") for iv in intervals if iv.has_channel("MicL")),
+            default=0,
+        )
         mic_peak_psi = mic_count_to_psi(mic_peak_count) if mic_peak_count else None
         rep = IdfReport(
             serial_number=md.serial,

tests/test_idf_binary_codec.py

@@ -235,3 +235,59 @@ def test_body_offset_search_is_not_quadratic():
         read_idf_file(IDFW_RAW16)
     elapsed = (time.perf_counter() - start) / 3
     assert elapsed < 0.15, f"body-offset search took {elapsed*1000:.0f} ms/file"
+
+
+# ─── Mic-disabled (3-channel) units, found 2026-09-10 ──────────────────────
+
+IDFW_3CH = FIXTURES / "UM20147_20250531135901.IDFW"   # body head below old floor
+IDFH_3CH = FIXTURES / "UM20147_20250330070110.IDFH"   # 56-byte interval records
+
+
+def test_three_channel_waveform_decodes_all_geo_channels():
+    """A mic-disabled unit's shorter header moves the record chain head.
+
+    Its head sits at 0x0dba, below the old ``_BODY_SCAN_FLOOR`` of 0x0E00, so
+    the scan could not see it and fell through to the *Vert* segment-0 record
+    — decoding a body shifted one position around the channel rotation, which
+    surfaced as Vert being exactly 512 samples short.
+    """
+    rows = _parse_export(IDFW_3CH.with_suffix(".IDFW.csv"))[1]
+    result = read_idf_file(IDFW_3CH)
+    for index, channel in enumerate(GEO_CHANNELS):
+        decoded = result.samples[channel]
+        assert len(decoded) == len(rows), (
+            f"{channel}: {len(decoded)} samples, export has {len(rows)}"
+        )
+        expected = [row[index] for row in rows]
+        bad = sum(
+            1 for c, v in zip(decoded, expected)
+            if abs(geo_count_to_ips(c) - v) >= 5e-5
+        )
+        assert bad == 0, f"{channel}: {bad} samples differ from Thor's export"
+    # Mic is genuinely absent on these units, not merely undecoded.
+    assert not result.samples.get("MicL")
+
+
+def test_three_channel_histogram_uses_56_byte_intervals():
+    """Interval stride is 16 bytes per channel + an 8-byte tail, not a constant.
+
+    A mic-disabled unit packs 56-byte records, so assuming 72 read 7 intervals
+    out of every 10-interval segment and then walked off alignment into
+    garbage, which decoded as ~10 in/s peaks.  The true count comes from the
+    segment's cumulative interval counter.
+    """
+    header, _rows = _parse_export(IDFH_3CH.with_suffix(".IDFH.csv"))
+    result = read_idf_file(IDFH_3CH)
+    expected_intervals = float(header["NumberOfIntervals"])
+    assert len(result.intervals) == pytest.approx(expected_intervals, abs=1.0)
+    assert {iv.n_channels for iv in result.intervals} == {3}
+    for channel, attr in (
+        ("Tran", "transverse_ips"),
+        ("Vert", "vertical_ips"),
+        ("Long", "longitudinal_ips"),
+    ):
+        decoded = getattr(result.event.peaks, attr)
+        assert decoded < 1.0, f"{channel} peak {decoded} looks like walked-off garbage"
+        assert decoded == pytest.approx(
+            _header_float(header, f"{channel}PPV"), rel=0.02
+        )