fix(scratch): dedupe the DL2 Sent/ mirror; correct the recovered-file count

The DL2 export keeps a byte-identical `Sent/` copy of its root, so walking it
counts every binary twice: 127,035 histogram paths are 63,535 distinct files,
and 13,077 waveform paths are 6,577. offset_scan.py now keeps the first
occurrence of each basename.

Corrects the previous commit's changelog claim of 8 recovered files — it is 4:
K440HJCN.3C0H and K557IF1U.8K0H (stride 252), T191HVNP.0S0H (92), T193L0XM.CI0H
(612). Still zero regressions. The per-unit breakdown reading exactly 2-2-2-2
should have given the doubling away.

The 14,338-exact verification result is unaffected: ASCII exports are not
mirrored (14,340 paths, 14,340 distinct names), and the harness enumerates
those rather than the binaries.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
2026-08-28 20:19:30 +00:00
co-authored by Claude Opus 5
parent 14e997b20c
commit 4839ddfa0e
2 changed files with 21 additions and 9 deletions
+12 -7
View File
@@ -19,14 +19,19 @@ All notable changes to seismo-relay are documented here.
false positives that once mis-dispatched 9,082 files, is unchanged. false positives that once mis-dispatched 9,082 files, is unchanged.
Found by decoding the full DL2 archive against its preserved Blastware ASCII Found by decoding the full DL2 archive against its preserved Blastware ASCII
exports. Across 127,035 histogram binaries the fix recovers **8 files** (strides exports. Across **63,535 unique** histogram binaries the fix recovers **4 files** —
92, 252 and 612; units BE18193, BE18191, BE9557, BE9440) with **zero** files `K440HJCN.3C0H` and `K557IF1U.8K0H` (stride 252), `T191HVNP.0S0H` (92) and
regressed. Verification over all 14,340 archive pairs goes 14,337 → 14,338 exact, `T193L0XM.CI0H` (612) — with **zero** files regressed. Verification over all
the only remainder being two series-4 IDF files that belong to a different codec. 14,340 archive pairs goes 14,337 → 14,338 exact, the only remainder being two
series-4 IDF files that belong to a different codec.
⚠ Prod stores hold `.h5` files generated before this fix. The 8 affected events (The DL2 export keeps a byte-identical `Sent/` mirror of its root, so a naive
stay empty until `backfill_sidecars.py` is re-run — not urgent at 8 files, and walk double-counts every binary — 127,035 paths are 63,535 distinct files. The
worth folding into the next backfill rather than doing one for this alone. ASCII exports are *not* mirrored, so the 14,340 pair count is already distinct.)
⚠ Prod stores hold `.h5` files generated before this fix. Those 4 events stay
empty until `backfill_sidecars.py` is re-run — not worth a two-hour prod backfill
on its own; fold it into the next one.
- **Histogram/waveform twin matching is now interval-based** (`find_twins`). A real - **Histogram/waveform twin matching is now interval-based** (`find_twins`). A real
trigger is recorded twice — as a triggered waveform (stamped at the trigger instant) trigger is recorded twice — as a triggered waveform (stamped at the trigger instant)
+9 -2
View File
@@ -107,8 +107,15 @@ def main():
ap.add_argument("--out", required=True) ap.add_argument("--out", required=True)
a = ap.parse_args() a = ap.parse_args()
files = [p for p in Path(a.dir).rglob("*") if p.is_file() and _WAVE_RE.search(p.name)] # The DL2 export keeps a byte-identical `Sent/` mirror of the root, so
files.sort() # enumerate paths but keep only the first occurrence of each basename —
# otherwise every event is counted twice.
seen = set()
files = []
for q in sorted(Path(a.dir).rglob("*")):
if q.is_file() and _WAVE_RE.search(q.name) and q.name not in seen:
seen.add(q.name)
files.append(q)
if a.limit: if a.limit:
files = files[: a.limit] files = files[: a.limit]
print(f"waveform binaries to scan: {len(files)}", flush=True) print(f"waveform binaries to scan: {len(files)}", flush=True)