serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(backfill): remove stale .h5 when nothing decodes; log the 415-file histogram variant

Browse Source 
backfill_sidecars.py skipped the .h5 write when a file produced no
samples, with the stated intent of not replacing it with an empty
placeholder.  That silently preserved output from a superseded decoder.

After the record-chain fix, 415 histogram files stopped decoding (216 on
BE18193, 199 on BE9440) but kept .h5 files whose peaks ran up to 400x
the device-reported PPV.  Those were feeding charts and the
false-trigger detector with nothing marking them.  The .h5 is now
removed in that case and the run reports stale_h5_removed.

Store-wide effect, series-3, decoded peak vs device-reported PPV:
  waveform   1307/1307 (100%), mean abs ratio error 0.00000
  histogram  4434/4435 (100%)
Both were 99% with a tail of 18 and 25 wrong files respectively.

The 415 files are a genuine unmapped format variant, not a regression:
their bodies open `00 00 00 01 0a 00` (valid block header, marker 0a at
[4], block_ctr 256) but block[28:32] matches neither known tail, and no
stride from 8 to 64 bytes places a marker at [4] consistently.  Bodies
are very large (one is 360,573 bytes).  They were previously being
decoded by the WAVEFORM codec, which accepted them and returned garbage
- so the gap pre-dates today's work; the fix only exposed it.  Logged as
an open question in the protocol reference.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
serversdown
2026-08-25 22:23:10 +00:00
co-authored by Claude Opus 5
parent 9bb95003e9
commit 4c58a532de
3 changed files with 29 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/backfill_sidecars.py
+19 -4
View File
@@ -136,6 +136,7 @@ def main(argv=None) -> int:
db = SeismoDb(db_path)
written = skipped = errors = 0
stale_h5_removed = 0
for serial_dir in sorted(p for p in store_root.iterdir() if p.is_dir()):
serial = serial_dir.name
for path in sorted(serial_dir.iterdir()):
@@ -388,9 +389,16 @@ def main(argv=None) -> int:
# waveform_codec.decode_waveform_v2 or histogram_codec.
# decode_histogram_body. If samples are still empty after
# both codecs run, it's a genuine "we can't decode this
# file" case (truncated, malformed, or unknown mode);
# skip the .h5 write so we don't replace whatever's
# there with an empty placeholder.
# file" case (truncated, malformed, or unknown mode).
#
# In that case we REMOVE any existing .h5 rather than leave
# it. Leaving it was the old behaviour and it silently
# preserved output from a superseded decoder: after the
# 2026-08-25 record-chain fix, 415 histogram files stopped
# decoding (an unmapped block variant on BE18193/BE9440) but
# kept .h5 files whose peaks were up to 400x the device's own
# reported PPV — garbage that fed the charts and the
# false-trigger detector with nothing marking it.
has_samples = bool(
ev.raw_samples and any(
ev.raw_samples.get(ch) for ch in ("Tran", "Vert", "Long", "MicL")
@@ -398,6 +406,13 @@ def main(argv=None) -> int:
)
hdf5_path = store.hdf5_path_for(serial, path.name)
hdf5_filename = hdf5_path.name if hdf5_path.exists() else None
if not has_samples and hdf5_path.exists():
log.warning("%s: no samples decode — removing stale %s",
path.name, hdf5_path.name)
if not args.dry_run:
hdf5_path.unlink()
hdf5_filename = None
stale_h5_removed += 1
hdf5_action = "kept"
need_h5 = (
not args.skip_hdf5
@@ -458,7 +473,7 @@ def main(argv=None) -> int:
log.error("backfill failed for %s: %s", path, exc, exc_info=args.verbose)
errors += 1
print(f"\nDone. written={written} skipped(uptodate)={skipped} errors={errors}")
print(f"\nDone. written={written} skipped(uptodate)={skipped} errors={errors} stale_h5_removed={stale_h5_removed}")
return 0 if errors == 0 else 1