fix(histogram): block is big-endian + terminal block tail — 1/1196 to 1211/1211
Two errors in the series-3 histogram block model, both found by diffing against the per-interval data table in the preserved Blastware ASCII exports (1211 files in the prod snapshot — far stronger ground truth than the header PPV used previously). 1. The block is uniformly BIG-ENDIAN. Peaks and half-periods are uint16 BE (T_peak [5:7], T_halfperiod [7:9], V_peak [9:11], V_halfperiod [11:13], L_peak [13:15], L_halfperiod [15:17], M_peak [17:19], M_halfperiod [19:21]); only block_ctr [2:4] is little-endian. The old uint8-peak model silently CLIPPED any peak above 1.275 in/s: the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590. The byte documented as a per-channel "annotation" was never an annotation — it is the half-period's high byte, which is exactly why it was non-zero on the sub-Hz intervals BW renders as "<1.0". The marker is block[4] alone. Testing [4:6] as a uint16 LE marker forced block[5] == 0, which is what capped the peak at one byte. 2. The final block of each stream carries tail 9c 06 00 42 instead of 1e 0a 00 00, and holds arbitrary bytes at [21:23]. Rejecting it dropped the last interval of nearly every histogram — frequently the interval holding the event peak, so the file's PPV read low. Verified end to end through the production path: 1211/1211 histograms decode exactly (interval count + every per-interval peak), plus 842,442 per-interval frequency comparisons with zero mismatches. Previously 1 of 1196 files was fully correct. decode_histogram_body_full records expose `is_terminal` in place of the removed `annotations` tuple. +6 tests. No regressions: full-suite failure list unchanged from baseline. NOTE: stored histogram .h5 files need regenerating to pick this up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
Claude Opus 5
@@ -25,38 +25,60 @@ iterate 32-stride and stop before the tail.
|
||||
────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
[0] 0x00 always-zero tag
|
||||
[1] segment_id (uint8) 0x00..0x03 — 256 blocks per segment
|
||||
[2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, …)
|
||||
[4:6] 0x000a (uint16 LE) constant marker (= 10)
|
||||
[6] T_peak_count uint8 Tran peak (count × 0.005 → in/s, max 1.275 in/s)
|
||||
[7] T_annotation uint8 empirically non-zero on intervals with sub-Hz
|
||||
or unmeasurable Tran freq; meaning not fully RE'd
|
||||
[8:10] T_halfperiod uint16 LE Tran half-period in samples (freq = 512 / halfp Hz)
|
||||
[10] V_peak_count uint8
|
||||
[11] V_annotation uint8
|
||||
[12:14] V_halfperiod uint16 LE
|
||||
[14] L_peak_count uint8
|
||||
[15] L_annotation uint8
|
||||
[16:18] L_halfperiod uint16 LE
|
||||
[18] M_peak_count uint8 MicL peak (count → dB via mic_count_to_db)
|
||||
[19] M_annotation uint8
|
||||
[20:22] M_halfperiod uint16 LE MicL half-period in samples (freq = 512 / halfp Hz)
|
||||
[22:24] 0x00 0x00 constant
|
||||
[1] segment_id (uint8) 0x00..0x03 - 256 blocks per segment
|
||||
[2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, ...)
|
||||
[4] 0x0a (uint8) constant marker (= 10)
|
||||
[5:7] T_peak_count uint16 BE Tran peak (count x 0.005 -> in/s)
|
||||
[7:9] T_halfperiod uint16 BE Tran half-period in samples (freq = 512 / halfp)
|
||||
[9:11] V_peak_count uint16 BE
|
||||
[11:13] V_halfperiod uint16 BE
|
||||
[13:15] L_peak_count uint16 BE
|
||||
[15:17] L_halfperiod uint16 BE
|
||||
[17:19] M_peak_count uint16 BE MicL peak (count -> dB via mic_count_to_db)
|
||||
[19:21] M_halfperiod uint16 BE MicL half-period in samples
|
||||
[21:23] 0x00 0x00 constant on standard blocks
|
||||
[24:28] 4-byte variable purpose unknown (possibly CRC or timestamp delta)
|
||||
[28:32] 0x1e 0x0a 0x00 0x00 constant block-end signature
|
||||
[28:32] block-end signature see "Two block tails" below
|
||||
|
||||
NOTE on peak-count width: an earlier interpretation treated the peak
|
||||
fields as uint16 LE spanning [6:8] / [10:12] / [14:16] / [18:20].
|
||||
That happened to be byte-exact against the N844 fixture corpus only
|
||||
because every annotation byte in those fixtures was zero, making
|
||||
``uint16 LE == uint8``. Cross-correlating BE9558 (K558) Tran-drift
|
||||
and BE18003 (T003) Histogram+Continuous events against the BW ASCII
|
||||
export proved peak is uint8 alone — see test_histogram_codec.py
|
||||
and docs/histogram_codec_re_status.md.
|
||||
**Every per-channel field is uint16 BIG-endian** (confirmed 2026-08-25).
|
||||
Only ``block_ctr`` at [2:4] is little-endian.
|
||||
|
||||
Block-identification anchor: ``block[22:24] == b"\\x00\\x00"`` AND
|
||||
``block[28:32] == b"\\x1e\\x0a\\x00\\x00"``. This is the reliable
|
||||
distinguisher from non-block content in the file.
|
||||
HISTORY - two earlier readings of this block were wrong in ways that
|
||||
cancelled out on quiet data:
|
||||
|
||||
1. *peak as uint16 LE at [6:8]* - produced 268 in/s peaks on any
|
||||
interval whose next byte was non-zero.
|
||||
2. *peak as uint8 at [6] with an "annotation" byte at [7]* - correct
|
||||
for every peak below 256 counts (1.275 in/s), but it silently
|
||||
**clipped larger peaks**: the final interval of
|
||||
BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export
|
||||
(1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590 in/s.
|
||||
The "annotation" byte was never an annotation - it is the high
|
||||
byte of the big-endian half-period, which is why it was non-zero
|
||||
exactly on the sub-Hz intervals BW renders as "<1.0".
|
||||
|
||||
Both readings also forced ``block[5] == 0`` via a bogus ``uint16 LE``
|
||||
marker check at [4:6], which is what capped the peak at one byte.
|
||||
The marker is ``block[4]`` alone.
|
||||
|
||||
Verified 2026-08-25 against 1211 production histograms paired with
|
||||
their Blastware ASCII exports: **1211/1211 decode exactly** (interval
|
||||
count plus every per-interval peak), and 842,442 per-interval
|
||||
frequency comparisons match with **zero** mismatches.
|
||||
|
||||
Two block tails
|
||||
---------------
|
||||
Standard blocks end with ``1e 0a 00 00``. The **final block of the
|
||||
stream** ends with ``9c 06 00 42`` instead, and carries arbitrary bytes
|
||||
at [21:23]. Rejecting it dropped the last interval of nearly every
|
||||
histogram - and the last interval is frequently the one holding the
|
||||
event peak, so the file's reported PPV came out low. Observed in 1206
|
||||
of 1211 production histograms, always positioned after every
|
||||
standard-tail block.
|
||||
|
||||
Block-identification anchor: ``block[0] == 0x00`` AND
|
||||
``block[4] == 0x0A`` AND the tail is one of the two signatures above;
|
||||
standard-tail blocks additionally require ``block[22] == 0x00``.
|
||||
|
||||
────────────────────────────────────────────────────────────────────────────
|
||||
Per-channel encoding
|
||||
@@ -109,6 +131,12 @@ from typing import List, Optional, Tuple
|
||||
# real data block. More distinctive than the byte-22 `00 00` (which
|
||||
# matches many false positives), so we anchor on this.
|
||||
_BLOCK_TAIL = b"\x1e\x0a\x00\x00"
|
||||
|
||||
# The final block of a histogram stream ends with this instead. It is a
|
||||
# real data block - same layout - and holds the last interval. See the
|
||||
# module docstring, "Two block tails".
|
||||
_BLOCK_TAIL_TERMINAL = b"\x9c\x06\x00\x42"
|
||||
|
||||
_BLOCK_SIZE = 32
|
||||
|
||||
# Marker byte at block[4:6] of every histogram data block. Used as
|
||||
@@ -127,19 +155,25 @@ _FREQ_NUMERATOR = 512
|
||||
|
||||
|
||||
def _is_data_block(block: bytes) -> bool:
|
||||
"""Tight identification of a histogram data block."""
|
||||
"""Tight identification of a histogram data block.
|
||||
|
||||
Accepts both tail signatures. ``block[4]`` alone is the marker -
|
||||
``block[5]`` is the high byte of the Tran peak and is non-zero on any
|
||||
interval above 1.275 in/s, so it must not be part of the marker test.
|
||||
The ``block[22] == 0`` constraint is what keeps trailer content out,
|
||||
but it applies only to standard-tail blocks: terminal blocks carry
|
||||
arbitrary bytes there.
|
||||
"""
|
||||
if len(block) < _BLOCK_SIZE:
|
||||
return False
|
||||
if block[28:32] != _BLOCK_TAIL:
|
||||
return False
|
||||
if block[22:24] != b"\x00\x00":
|
||||
return False
|
||||
if block[0] != 0x00:
|
||||
return False
|
||||
marker = block[4] | (block[5] << 8)
|
||||
if marker != _BLOCK_MARKER:
|
||||
if block[4] != _BLOCK_MARKER:
|
||||
return False
|
||||
return True
|
||||
tail = block[28:32]
|
||||
if tail == _BLOCK_TAIL:
|
||||
return block[22] == 0x00
|
||||
return tail == _BLOCK_TAIL_TERMINAL
|
||||
|
||||
|
||||
def _decode_block(block: bytes) -> Optional[dict]:
|
||||
@@ -149,33 +183,23 @@ def _decode_block(block: bytes) -> Optional[dict]:
|
||||
Returns a record with per-channel peak counts (uint8) and
|
||||
half-periods (uint16 LE).
|
||||
"""
|
||||
# Peak counts are uint8 at bytes [6] / [10] / [14] / [18]. The
|
||||
# adjacent bytes [7] / [11] / [15] / [19] hold an annotation field
|
||||
# whose meaning isn't fully understood (empirically non-zero in
|
||||
# intervals with sub-Hz or unmeasurable geo frequencies, mostly
|
||||
# zero otherwise — see test fixtures from BE9558/BE18003 corpora).
|
||||
# Crucially, those annotation bytes are NOT the high byte of the
|
||||
# peak count: cross-correlating against BW's per-interval ASCII
|
||||
# export proves the peak is uint8 alone.
|
||||
#
|
||||
# Reading the peak as uint16 LE (the original interpretation) was
|
||||
# accidentally correct only because every block in the N844 fixture
|
||||
# corpus had a zero annotation byte; non-N844 events with non-zero
|
||||
# annotation bytes decoded to physically impossible peaks (e.g.
|
||||
# 268 in/s per channel) and produced 35× inflated PVS sums when
|
||||
# first run against prod data. See histogram_codec_re_status.md.
|
||||
t_peak = block[6]
|
||||
v_peak = block[10]
|
||||
l_peak = block[14]
|
||||
m_peak = block[18]
|
||||
t_halfp = block[8] | (block[9] << 8)
|
||||
v_halfp = block[12] | (block[13] << 8)
|
||||
l_halfp = block[16] | (block[17] << 8)
|
||||
m_halfp = block[20] | (block[21] << 8)
|
||||
# Every per-channel field is uint16 BIG-endian; only block_ctr is LE.
|
||||
# See the module docstring for the two superseded readings and why
|
||||
# each looked correct on quiet data.
|
||||
def _be16(i: int) -> int:
|
||||
return (block[i] << 8) | block[i + 1]
|
||||
|
||||
t_peak = _be16(5)
|
||||
t_halfp = _be16(7)
|
||||
v_peak = _be16(9)
|
||||
v_halfp = _be16(11)
|
||||
l_peak = _be16(13)
|
||||
l_halfp = _be16(15)
|
||||
m_peak = _be16(17)
|
||||
m_halfp = _be16(19)
|
||||
segment_id = block[1]
|
||||
block_ctr = block[2] | (block[3] << 8)
|
||||
var_meta = bytes(block[24:28])
|
||||
annotations = (block[7], block[11], block[15], block[19])
|
||||
return {
|
||||
"segment_id": segment_id,
|
||||
"block_ctr": block_ctr,
|
||||
@@ -188,7 +212,7 @@ def _decode_block(block: bytes) -> Optional[dict]:
|
||||
"m_peak": m_peak,
|
||||
"m_halfp": m_halfp,
|
||||
"meta_var": var_meta,
|
||||
"annotations": annotations,
|
||||
"is_terminal": block[28:32] == _BLOCK_TAIL_TERMINAL,
|
||||
}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user