fix(histogram): block is big-endian + terminal block tail — 1/1196 to 1211/1211

Two errors in the series-3 histogram block model, both found by diffing
against the per-interval data table in the preserved Blastware ASCII
exports (1211 files in the prod snapshot — far stronger ground truth
than the header PPV used previously).

1. The block is uniformly BIG-ENDIAN.  Peaks and half-periods are uint16
   BE (T_peak [5:7], T_halfperiod [7:9], V_peak [9:11], V_halfperiod
   [11:13], L_peak [13:15], L_halfperiod [15:17], M_peak [17:19],
   M_halfperiod [19:21]); only block_ctr [2:4] is little-endian.

   The old uint8-peak model silently CLIPPED any peak above 1.275 in/s:
   the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's
   export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590.

   The byte documented as a per-channel "annotation" was never an
   annotation — it is the half-period's high byte, which is exactly why
   it was non-zero on the sub-Hz intervals BW renders as "<1.0".

   The marker is block[4] alone.  Testing [4:6] as a uint16 LE marker
   forced block[5] == 0, which is what capped the peak at one byte.

2. The final block of each stream carries tail 9c 06 00 42 instead of
   1e 0a 00 00, and holds arbitrary bytes at [21:23].  Rejecting it
   dropped the last interval of nearly every histogram — frequently the
   interval holding the event peak, so the file's PPV read low.

Verified end to end through the production path: 1211/1211 histograms
decode exactly (interval count + every per-interval peak), plus 842,442
per-interval frequency comparisons with zero mismatches.  Previously
1 of 1196 files was fully correct.

decode_histogram_body_full records expose `is_terminal` in place of the
removed `annotations` tuple.  +6 tests.  No regressions: full-suite
failure list unchanged from baseline.

NOTE: stored histogram .h5 files need regenerating to pick this up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
2026-08-25 18:59:25 +00:00
co-authored by Claude Opus 5
parent e449ac04af
commit ef1e99b0a0
6 changed files with 269 additions and 84 deletions
+28 -13
View File
@@ -261,6 +261,32 @@ rotation and corrupts every channel after it.
Corpus result, end to end through the production path:
**exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.**
### Histogram codec — corrected 2026-08-25
The histogram block is **uniformly big-endian**, and the stream's final
block has its own tail signature. Two long-standing errors:
- **Peaks and half-periods are `uint16` big-endian**, not `uint8` +
an "annotation" byte. `T_peak` is `[5:7]`, `T_halfperiod` `[7:9]`,
`V_peak` `[9:11]`, and so on; only `block_ctr` at `[2:4]` is LE.
The old model silently **clipped any peak above 1.275 in/s** — the
final interval of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in BW's
export and decoded as 0.590. The "annotation" byte was the
half-period's high byte, which is why it was non-zero exactly on the
sub-Hz intervals BW renders as `<1.0`.
- **The marker is `block[4]` alone.** Testing `[4:6]` as a uint16 LE
marker forced `block[5] == 0`, which is what capped the peak at one
byte in the first place.
- **The last block of the stream carries tail `9c 06 00 42`** instead of
`1e 0a 00 00`, with arbitrary bytes at `[21:23]`. Rejecting it
dropped the final interval of nearly every histogram — frequently the
one holding the event peak, so the file's PPV read low.
Verified against 1211 production histograms paired with their BW ASCII
exports: **1211/1211 decode exactly** (interval count plus every
per-interval peak), and 842,442 per-interval frequency comparisons match
with zero mismatches. Before: 1 of 1196.
### What's NOT solved
- **MicL channel conversion to dB(L)** — the codec emits MicL as
@@ -276,19 +302,8 @@ Corpus result, end to end through the production path:
anchor count — `01 00` records appear with both 2- and 4-byte anchor
fields in the same file. Examples: `BE12599/N599LPNB.JF0W` @1155,
`BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
- **Histogram codec drops trailing intervals (series-3)** — the
per-interval *values* are right: measured against the histogram ASCII
data table (Tran/Vert/Long peak + freq + PVS per interval), 1031 of
1196 files (86%) match within 1 LSB across the overlapping prefix.
The interval **count** is short in 1195 of 1196 files — median 1
missing; 1088 short by 1–2, 65 by 3–10, 39 by 11–100, 3 by >100 (max
205). When a dropped interval holds the event peak, the decoded max
falls below the device-reported PPV: 169/1196 files (14%). A
termination bug in `histogram_codec.decode_histogram_body` — same
family as the waveform-walker truncation fixed 2026-08-25, and
untouched by it. Note: the histogram ASCII carries a full
per-interval table, so this has proper ground truth available (1211
files in the prod snapshot).
(The series-3 histogram codec was fixed 2026-08-25 — see below.)
- **Micromate (UM-series) IDF decode is ~1000× low** — e.g.
`UM11402_20260406130113.IDFW` gives a Tran peak of 0.0009 in/s against
a device-reported 1.1168. The Thor IDF path decodes sanely, so this