fix(histogram): block is big-endian + terminal block tail — 1/1196 to 1211/1211
Two errors in the series-3 histogram block model, both found by diffing against the per-interval data table in the preserved Blastware ASCII exports (1211 files in the prod snapshot — far stronger ground truth than the header PPV used previously). 1. The block is uniformly BIG-ENDIAN. Peaks and half-periods are uint16 BE (T_peak [5:7], T_halfperiod [7:9], V_peak [9:11], V_halfperiod [11:13], L_peak [13:15], L_halfperiod [15:17], M_peak [17:19], M_halfperiod [19:21]); only block_ctr [2:4] is little-endian. The old uint8-peak model silently CLIPPED any peak above 1.275 in/s: the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590. The byte documented as a per-channel "annotation" was never an annotation — it is the half-period's high byte, which is exactly why it was non-zero on the sub-Hz intervals BW renders as "<1.0". The marker is block[4] alone. Testing [4:6] as a uint16 LE marker forced block[5] == 0, which is what capped the peak at one byte. 2. The final block of each stream carries tail 9c 06 00 42 instead of 1e 0a 00 00, and holds arbitrary bytes at [21:23]. Rejecting it dropped the last interval of nearly every histogram — frequently the interval holding the event peak, so the file's PPV read low. Verified end to end through the production path: 1211/1211 histograms decode exactly (interval count + every per-interval peak), plus 842,442 per-interval frequency comparisons with zero mismatches. Previously 1 of 1196 files was fully correct. decode_histogram_body_full records expose `is_terminal` in place of the removed `annotations` tuple. +6 tests. No regressions: full-suite failure list unchanged from baseline. NOTE: stored histogram .h5 files need regenerating to pick this up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
@@ -261,6 +261,32 @@ rotation and corrupts every channel after it.
|
||||
Corpus result, end to end through the production path:
|
||||
**exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.**
|
||||
|
||||
### Histogram codec — corrected 2026-08-25
|
||||
|
||||
The histogram block is **uniformly big-endian**, and the stream's final
|
||||
block has its own tail signature. Two long-standing errors:
|
||||
|
||||
- **Peaks and half-periods are `uint16` big-endian**, not `uint8` +
|
||||
an "annotation" byte. `T_peak` is `[5:7]`, `T_halfperiod` `[7:9]`,
|
||||
`V_peak` `[9:11]`, and so on; only `block_ctr` at `[2:4]` is LE.
|
||||
The old model silently **clipped any peak above 1.275 in/s** — the
|
||||
final interval of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in BW's
|
||||
export and decoded as 0.590. The "annotation" byte was the
|
||||
half-period's high byte, which is why it was non-zero exactly on the
|
||||
sub-Hz intervals BW renders as `<1.0`.
|
||||
- **The marker is `block[4]` alone.** Testing `[4:6]` as a uint16 LE
|
||||
marker forced `block[5] == 0`, which is what capped the peak at one
|
||||
byte in the first place.
|
||||
- **The last block of the stream carries tail `9c 06 00 42`** instead of
|
||||
`1e 0a 00 00`, with arbitrary bytes at `[21:23]`. Rejecting it
|
||||
dropped the final interval of nearly every histogram — frequently the
|
||||
one holding the event peak, so the file's PPV read low.
|
||||
|
||||
Verified against 1211 production histograms paired with their BW ASCII
|
||||
exports: **1211/1211 decode exactly** (interval count plus every
|
||||
per-interval peak), and 842,442 per-interval frequency comparisons match
|
||||
with zero mismatches. Before: 1 of 1196.
|
||||
|
||||
### What's NOT solved
|
||||
|
||||
- **MicL channel conversion to dB(L)** — the codec emits MicL as
|
||||
@@ -276,19 +302,8 @@ Corpus result, end to end through the production path:
|
||||
anchor count — `01 00` records appear with both 2- and 4-byte anchor
|
||||
fields in the same file. Examples: `BE12599/N599LPNB.JF0W` @1155,
|
||||
`BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
|
||||
- **Histogram codec drops trailing intervals (series-3)** — the
|
||||
per-interval *values* are right: measured against the histogram ASCII
|
||||
data table (Tran/Vert/Long peak + freq + PVS per interval), 1031 of
|
||||
1196 files (86%) match within 1 LSB across the overlapping prefix.
|
||||
The interval **count** is short in 1195 of 1196 files — median 1
|
||||
missing; 1088 short by 1–2, 65 by 3–10, 39 by 11–100, 3 by >100 (max
|
||||
205). When a dropped interval holds the event peak, the decoded max
|
||||
falls below the device-reported PPV: 169/1196 files (14%). A
|
||||
termination bug in `histogram_codec.decode_histogram_body` — same
|
||||
family as the waveform-walker truncation fixed 2026-08-25, and
|
||||
untouched by it. Note: the histogram ASCII carries a full
|
||||
per-interval table, so this has proper ground truth available (1211
|
||||
files in the prod snapshot).
|
||||
(The series-3 histogram codec was fixed 2026-08-25 — see below.)
|
||||
|
||||
- **Micromate (UM-series) IDF decode is ~1000× low** — e.g.
|
||||
`UM11402_20260406130113.IDFW` gives a Tran peak of 0.0009 in/s against
|
||||
a device-reported 1.1168. The Thor IDF path decodes sanely, so this
|
||||
|
||||
Reference in New Issue
Block a user