fix(histogram): partial final block no longer discards the correct stride
detect_multi_interval_stride() confirmed a candidate stride on a third block header whenever the body was long enough to contain one. But a body can exceed two strides and still hold only two real blocks: a partial final block leaves trailing padding. BE18193 T193L0XM.CI0H — 51 intervals at 2 s, i.e. one full 30-interval block plus a 21-interval remainder in a 2787-byte body — had every decisive check pass at stride 612 (header at 0, header at 612, block counter 256 -> 257) and was then rejected for the absent third header at 1224. It decoded to nothing. A missing third header now means end-of-stream rather than disqualification. The block-counter check is untouched — that is the test that prevents the false positives which once handed 9,082 standard-block files to the multi-interval walker. Found by running the full DL2 archive against its preserved Blastware ASCII exports (14,340 paired files, 11x the previous ground-truth corpus). Measured over 127,035 archive histogram binaries: recovered 8 files (strides 92, 252, 612; BE18193, BE18191, BE9557, BE9440) regressed 0 files Full-corpus verification: 14,337 -> 14,338 exact of 14,338 decodable pairs (the 2 excluded are series-4 IDF, a different codec). Also adds scratch/verify_against_ascii.py (per-sample decoder verification against BW exports, with a saturation carve-out — BW clamps clipped events to the range max while the decoder reports true counts) and scratch/offset_scan.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
@@ -7,6 +7,27 @@ All notable changes to seismo-relay are documented here.
|
||||
## [Unreleased]
|
||||
|
||||
### Fixed
|
||||
- **Sub-minute histograms with a partial final block decoded to nothing**
|
||||
(`histogram_codec.detect_multi_interval_stride`). The stride search confirmed
|
||||
itself on a third block header whenever the body was long enough to hold one —
|
||||
but a body can exceed two strides and still contain only two real blocks, because
|
||||
a *partial* final block leaves trailing padding. BE18193 `T193L0XM.CI0H` (51
|
||||
intervals at 2 s = one full 30-interval block plus a 21-interval remainder, in a
|
||||
2787-byte body) therefore had its correct stride of 612 discarded and produced an
|
||||
empty decode. A missing third header now means end-of-stream rather than
|
||||
disqualification; the block-counter check, which is what actually prevents the
|
||||
false positives that once mis-dispatched 9,082 files, is unchanged.
|
||||
|
||||
Found by decoding the full DL2 archive against its preserved Blastware ASCII
|
||||
exports. Across 127,035 histogram binaries the fix recovers **8 files** (strides
|
||||
92, 252 and 612; units BE18193, BE18191, BE9557, BE9440) with **zero** files
|
||||
regressed. Verification over all 14,340 archive pairs goes 14,337 → 14,338 exact,
|
||||
the only remainder being two series-4 IDF files that belong to a different codec.
|
||||
|
||||
⚠ Prod stores hold `.h5` files generated before this fix. The 8 affected events
|
||||
stay empty until `backfill_sidecars.py` is re-run — not urgent at 8 files, and
|
||||
worth folding into the next backfill rather than doing one for this alone.
|
||||
|
||||
- **Histogram/waveform twin matching is now interval-based** (`find_twins`). A real
|
||||
trigger is recorded twice — as a triggered waveform (stamped at the trigger instant)
|
||||
and inside the scheduled histogram whose interval contains it (stamped at the 7am/7pm
|
||||
|
||||
Reference in New Issue
Block a user