fix(backfill): remove stale .h5 when nothing decodes; log the 415-file histogram variant
backfill_sidecars.py skipped the .h5 write when a file produced no samples, with the stated intent of not replacing it with an empty placeholder. That silently preserved output from a superseded decoder. After the record-chain fix, 415 histogram files stopped decoding (216 on BE18193, 199 on BE9440) but kept .h5 files whose peaks ran up to 400x the device-reported PPV. Those were feeding charts and the false-trigger detector with nothing marking them. The .h5 is now removed in that case and the run reports stale_h5_removed. Store-wide effect, series-3, decoded peak vs device-reported PPV: waveform 1307/1307 (100%), mean abs ratio error 0.00000 histogram 4434/4435 (100%) Both were 99% with a tail of 18 and 25 wrong files respectively. The 415 files are a genuine unmapped format variant, not a regression: their bodies open `00 00 00 01 0a 00` (valid block header, marker 0a at [4], block_ctr 256) but block[28:32] matches neither known tail, and no stride from 8 to 64 bytes places a marker at [4] consistently. Bodies are very large (one is 360,573 bytes). They were previously being decoded by the WAVEFORM codec, which accepted them and returned garbage - so the gap pre-dates today's work; the fix only exposed it. Logged as an open question in the protocol reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
Claude Opus 5
@@ -8,6 +8,15 @@ All notable changes to seismo-relay are documented here.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`backfill_sidecars.py` now removes a stale `.h5` when nothing decodes.**
|
||||
It previously skipped the write "so we don't replace whatever's there with an
|
||||
empty placeholder", which silently preserved output from a superseded
|
||||
decoder. After the record-chain fix, 415 histogram files stopped decoding (an
|
||||
unmapped block variant on BE18193 and BE9440) but kept `.h5` files whose peaks
|
||||
ran up to **400× the device's own reported PPV** — garbage feeding the charts
|
||||
and the false-trigger detector with nothing marking it. Reports a
|
||||
`stale_h5_removed` count.
|
||||
|
||||
- **The series-3 waveform body is a RECORD CHAIN, not a tag stream — this
|
||||
supersedes the segment-header model, including the fixes made earlier the
|
||||
same day.**
|
||||
|
||||
@@ -3083,6 +3083,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
|
||||
| **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open |
|
||||
| ~~**Variable-prefix segment descriptors**~~ - **RESOLVED 2026-08-25:** there is no variable prefix. The body is a chain of self-delimiting records (see S7.6.1); the 0/2/4/6/8-byte prefix was walker drift from reading `40 NN` as a segment header. All 25 affected files now chain cleanly to the `06` terminator. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
|
||||
| ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
|
||||
| **Unmapped histogram block variant (415 files, 2 units)** - after the 2026-08-25 record-chain fix these stop decoding entirely: 216 on BE18193 and 199 on BE9440, all histograms. Their bodies open `00 00 00 01 0a 00`, which is a valid block header (marker `0a` at [4], block_ctr 256 at [2:4] LE), but `block[28:32]` is neither the standard `1e 0a 00 00` nor the terminal `9c 06 00 42`, and no fixed stride between 8 and 64 bytes puts a marker at [4] consistently. Bodies are very large (one is 360,573 bytes). Previously these were being decoded by the WAVEFORM codec, which accepted them and returned garbage peaking up to 400x the device-reported PPV - so this is a pre-existing gap the fix merely exposed, not a regression. Their stale `.h5` files are now removed by `scripts/backfill_sidecars.py` rather than left behind. Example: `BE18193/T193L2X2.1J0H`. | MEDIUM | 2026-08-25 | Still open |
|
||||
| **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open |
|
||||
| **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open |
|
||||
|
||||
|
||||
@@ -136,6 +136,7 @@ def main(argv=None) -> int:
|
||||
db = SeismoDb(db_path)
|
||||
|
||||
written = skipped = errors = 0
|
||||
stale_h5_removed = 0
|
||||
for serial_dir in sorted(p for p in store_root.iterdir() if p.is_dir()):
|
||||
serial = serial_dir.name
|
||||
for path in sorted(serial_dir.iterdir()):
|
||||
@@ -388,9 +389,16 @@ def main(argv=None) -> int:
|
||||
# waveform_codec.decode_waveform_v2 or histogram_codec.
|
||||
# decode_histogram_body. If samples are still empty after
|
||||
# both codecs run, it's a genuine "we can't decode this
|
||||
# file" case (truncated, malformed, or unknown mode);
|
||||
# skip the .h5 write so we don't replace whatever's
|
||||
# there with an empty placeholder.
|
||||
# file" case (truncated, malformed, or unknown mode).
|
||||
#
|
||||
# In that case we REMOVE any existing .h5 rather than leave
|
||||
# it. Leaving it was the old behaviour and it silently
|
||||
# preserved output from a superseded decoder: after the
|
||||
# 2026-08-25 record-chain fix, 415 histogram files stopped
|
||||
# decoding (an unmapped block variant on BE18193/BE9440) but
|
||||
# kept .h5 files whose peaks were up to 400x the device's own
|
||||
# reported PPV — garbage that fed the charts and the
|
||||
# false-trigger detector with nothing marking it.
|
||||
has_samples = bool(
|
||||
ev.raw_samples and any(
|
||||
ev.raw_samples.get(ch) for ch in ("Tran", "Vert", "Long", "MicL")
|
||||
@@ -398,6 +406,13 @@ def main(argv=None) -> int:
|
||||
)
|
||||
hdf5_path = store.hdf5_path_for(serial, path.name)
|
||||
hdf5_filename = hdf5_path.name if hdf5_path.exists() else None
|
||||
if not has_samples and hdf5_path.exists():
|
||||
log.warning("%s: no samples decode — removing stale %s",
|
||||
path.name, hdf5_path.name)
|
||||
if not args.dry_run:
|
||||
hdf5_path.unlink()
|
||||
hdf5_filename = None
|
||||
stale_h5_removed += 1
|
||||
hdf5_action = "kept"
|
||||
need_h5 = (
|
||||
not args.skip_hdf5
|
||||
@@ -458,7 +473,7 @@ def main(argv=None) -> int:
|
||||
log.error("backfill failed for %s: %s", path, exc, exc_info=args.verbose)
|
||||
errors += 1
|
||||
|
||||
print(f"\nDone. written={written} skipped(uptodate)={skipped} errors={errors}")
|
||||
print(f"\nDone. written={written} skipped(uptodate)={skipped} errors={errors} stale_h5_removed={stale_h5_removed}")
|
||||
return 0 if errors == 0 else 1
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user