fix(histogram): partial final block no longer discards the correct stride

detect_multi_interval_stride() confirmed a candidate stride on a third block
header whenever the body was long enough to contain one. But a body can exceed
two strides and still hold only two real blocks: a partial final block leaves
trailing padding. BE18193 T193L0XM.CI0H — 51 intervals at 2 s, i.e. one full
30-interval block plus a 21-interval remainder in a 2787-byte body — had every
decisive check pass at stride 612 (header at 0, header at 612, block counter
256 -> 257) and was then rejected for the absent third header at 1224. It
decoded to nothing.

A missing third header now means end-of-stream rather than disqualification.
The block-counter check is untouched — that is the test that prevents the
false positives which once handed 9,082 standard-block files to the
multi-interval walker.

Found by running the full DL2 archive against its preserved Blastware ASCII
exports (14,340 paired files, 11x the previous ground-truth corpus).

Measured over 127,035 archive histogram binaries:
  recovered 8 files (strides 92, 252, 612; BE18193, BE18191, BE9557, BE9440)
  regressed 0 files
Full-corpus verification: 14,337 -> 14,338 exact of 14,338 decodable pairs
(the 2 excluded are series-4 IDF, a different codec).

Also adds scratch/verify_against_ascii.py (per-sample decoder verification
against BW exports, with a saturation carve-out — BW clamps clipped events to
the range max while the decoder reports true counts) and scratch/offset_scan.py.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
2026-08-28 20:19:30 +00:00
co-authored by Claude Opus 5
parent 75ac610c61
commit 14e997b20c
5 changed files with 453 additions and 4 deletions
+21
View File
@@ -7,6 +7,27 @@ All notable changes to seismo-relay are documented here.
## [Unreleased]
### Fixed
- **Sub-minute histograms with a partial final block decoded to nothing**
(`histogram_codec.detect_multi_interval_stride`). The stride search confirmed
itself on a third block header whenever the body was long enough to hold one —
but a body can exceed two strides and still contain only two real blocks, because
a *partial* final block leaves trailing padding. BE18193 `T193L0XM.CI0H` (51
intervals at 2 s = one full 30-interval block plus a 21-interval remainder, in a
2787-byte body) therefore had its correct stride of 612 discarded and produced an
empty decode. A missing third header now means end-of-stream rather than
disqualification; the block-counter check, which is what actually prevents the
false positives that once mis-dispatched 9,082 files, is unchanged.
Found by decoding the full DL2 archive against its preserved Blastware ASCII
exports. Across 127,035 histogram binaries the fix recovers **8 files** (strides
92, 252 and 612; units BE18193, BE18191, BE9557, BE9440) with **zero** files
regressed. Verification over all 14,340 archive pairs goes 14,337 → 14,338 exact,
the only remainder being two series-4 IDF files that belong to a different codec.
⚠ Prod stores hold `.h5` files generated before this fix. The 8 affected events
stay empty until `backfill_sidecars.py` is re-run — not urgent at 8 files, and
worth folding into the next backfill rather than doing one for this alone.
- **Histogram/waveform twin matching is now interval-based** (`find_twins`). A real
trigger is recorded twice — as a triggered waveform (stamped at the trigger instant)
and inside the scheduled histogram whose interval contains it (stamped at the 7am/7pm