fix(histogram): partial final block no longer discards the correct stride

detect_multi_interval_stride() confirmed a candidate stride on a third block
header whenever the body was long enough to contain one. But a body can exceed
two strides and still hold only two real blocks: a partial final block leaves
trailing padding. BE18193 T193L0XM.CI0H — 51 intervals at 2 s, i.e. one full
30-interval block plus a 21-interval remainder in a 2787-byte body — had every
decisive check pass at stride 612 (header at 0, header at 612, block counter
256 -> 257) and was then rejected for the absent third header at 1224. It
decoded to nothing.

A missing third header now means end-of-stream rather than disqualification.
The block-counter check is untouched — that is the test that prevents the
false positives which once handed 9,082 standard-block files to the
multi-interval walker.

Found by running the full DL2 archive against its preserved Blastware ASCII
exports (14,340 paired files, 11x the previous ground-truth corpus).

Measured over 127,035 archive histogram binaries:
  recovered 8 files (strides 92, 252, 612; BE18193, BE18191, BE9557, BE9440)
  regressed 0 files
Full-corpus verification: 14,337 -> 14,338 exact of 14,338 decodable pairs
(the 2 excluded are series-4 IDF, a different codec).

Also adds scratch/verify_against_ascii.py (per-sample decoder verification
against BW exports, with a saturation carve-out — BW clamps clipped events to
the range max while the decoder reports true counts) and scratch/offset_scan.py.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
2026-08-28 20:19:30 +00:00
co-authored by Claude Opus 5
parent 75ac610c61
commit 14e997b20c
5 changed files with 453 additions and 4 deletions
+12 -4
View File
@@ -413,10 +413,18 @@ def detect_multi_interval_stride(body: bytes) -> Optional[int]:
if (_ctr(stride) - _ctr(0)) & 0xFFFF != 1:
continue
# confirm on a third block when the body is long enough
if 2 * stride + _MULTI_HEADER_LEN <= len(body):
if not _is_multi_header(body, 2 * stride):
continue
# Confirm on a third block WHEN ONE IS ACTUALLY PRESENT. A body can
# be longer than two strides and still hold only two real blocks: a
# final *partial* block leaves trailing padding. E.g. 51 intervals at
# 2 s = one full 30-interval block + a 21-interval remainder, in a
# 2787-byte body — long enough to demand a third header at 1224 that
# does not exist. Requiring it unconditionally threw away the correct
# stride and the file decoded to nothing (BE18193 T193L0XM.CI0H).
# The block-counter check above is the decisive anti-false-positive
# test; this one is corroboration, so a missing third header means
# end-of-stream, not disqualification.
if (2 * stride + _MULTI_HEADER_LEN <= len(body)
and _is_multi_header(body, 2 * stride)):
if (_ctr(2 * stride) - _ctr(stride)) & 0xFFFF != 1:
continue
return stride