histogram_codec: defensive bounds-check on peak counts

Discovered while running the backfill on prod: certain histogram
blocks contain an undocumented extension byte format whose naive
uint16 LE interpretation yields physically impossible peak values
(150+ in/s when the device max is 10).  Concrete example from
K558LKSG.3I0H block at body+7424:

  bytes [6:10] = 05 79 69 00
  current code: T_peak = uint16 LE = 0x7905 = 30981 → 154.9 in/s
  reality:     T_peak = byte[6] = 5 → 0.025 in/s (matches BW display)

The high byte (0x79 here) appears to be an extension field — possibly
"time of peak within interval" or a Histogram+Continuous sub-mode
marker.  Observed across BE9558 and BE18003 units in prod data; never
appeared in the BE12844 fixture corpus the codec was originally
verified against.

Effect on prod: 26 out of 1433 blocks in this one event had inflated
peaks, plus dozens of similar events across the fleet → sum(PVS)
inflated from baseline 988 to 34501 (35x).  Rolled back via the
pre-backfill snapshot before any UI exposure.

Defensive fix: bounds-check peak counts in `_decode_block`.  Any
field exceeding `_MAX_PEAK_COUNT` (4096 = ~20 in/s, well past the
device's 10 in/s Normal-range FS) causes the block to be skipped
entirely.  Other valid blocks in the same event still decode
correctly.

Trade-off: those skipped blocks lose their per-interval data
(peaks + frequencies).  Acceptable until the extension format is
reverse-engineered — better than propagating bogus values into PVS
computations downstream.

The 24 existing tests all still pass — the fixtures used during the
original codec development don't exercise the extension-byte case.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-21 02:17:33 +00:00
parent 88549bc659
commit bc5a2d3f19
+40 -5
View File
@@ -101,6 +101,23 @@ _BLOCK_SIZE = 32
# additional validation that we're looking at a real block.
_BLOCK_MARKER = 10
# Maximum plausible peak-count value. Normal-range geophone tops out
# at 10 in/s = 2000 counts at the 0.005 in/s per count scale; even
# Sensitive range (1.25 in/s FS) wouldn't exceed ~250. Mic counts run
# 0..~400 in observed data. 4096 leaves comfortable headroom for any
# legitimate value across all modes.
#
# Some prod blocks have been observed with peak-count fields whose
# HIGH byte is non-zero (block[7] != 0 etc.) — observed across BE9558
# and BE18003 units in Histogram-mode events. Reading these as
# uint16 LE produces values like 30981 / 41733 / 62469, which scale
# to physically impossible peaks (150+ in/s). Best guess: an
# undocumented "time-of-peak-within-interval" extension byte the
# device writes in some sub-mode (possibly Histogram+Continuous).
# Until reverse-engineered, blocks exceeding this bound are skipped
# rather than propagating bogus values into PVS computations.
_MAX_PEAK_COUNT = 4096
# Geo peak scaling: stored as "count × 0.005 in/s" where 1 count = one
# 0.005 in/s display quantum. Equivalent to the waveform codec's
# 16-count-unit output (1 unit = 0.005 in/s = 16 ADC counts).
@@ -128,14 +145,24 @@ def _is_data_block(block: bytes) -> bool:
return True
def _decode_block(block: bytes) -> dict:
def _decode_block(block: bytes) -> Optional[dict]:
"""Decode one 32-byte histogram block. Caller must have validated
with ``_is_data_block`` first."""
with ``_is_data_block`` first.
Returns ``None`` if any peak field exceeds ``_MAX_PEAK_COUNT`` —
those blocks contain an undocumented extension byte format whose
naive uint16 LE interpretation gives physically impossible peaks.
Skipping the block is safer than propagating bogus values into
PVS computations downstream.
"""
# All 16-bit fields are little-endian unsigned. Peak counts are
# always non-negative; half-periods are always positive when valid.
t_peak, t_halfp, v_peak, v_halfp, l_peak, l_halfp, m_peak, m_halfp = struct.unpack_from(
"<HHHHHHHH", block, 6
)
if (t_peak > _MAX_PEAK_COUNT or v_peak > _MAX_PEAK_COUNT
or l_peak > _MAX_PEAK_COUNT or m_peak > _MAX_PEAK_COUNT):
return None
segment_id = block[1]
block_ctr = block[2] | (block[3] << 8)
var_meta = bytes(block[24:28])
@@ -158,8 +185,10 @@ def walk_body(body: bytes) -> List[dict]:
"""Walk the body and return one dict per histogram interval.
Iterates 32-byte strides from offset 0. Yields a decoded record
for every block that passes ``_is_data_block`` validation. Stops
when the remaining bytes are too short to form a complete block.
for every block that passes ``_is_data_block`` validation AND has
plausible peak values (``_decode_block`` returns None for blocks
with out-of-bound peaks). Stops when the remaining bytes are too
short to form a complete block.
"""
records: List[dict] = []
for off in range(0, len(body) - _BLOCK_SIZE + 1, _BLOCK_SIZE):
@@ -169,7 +198,13 @@ def walk_body(body: bytes) -> List[dict]:
# Continue walking — block alignment is fixed at 32-stride
# from offset 0, so we don't lose alignment by skipping.
continue
records.append(_decode_block(blk))
decoded = _decode_block(blk)
if decoded is None:
# Block validated as a histogram block but had peak fields
# outside the plausible range — undocumented extension.
# Skip rather than propagating bogus PVS contributions.
continue
records.append(decoded)
return records