diff --git a/CHANGELOG.md b/CHANGELOG.md index 94bc028..cb7aa9c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,15 @@ All notable changes to seismo-relay are documented here. ### Fixed +- **`backfill_sidecars.py` now removes a stale `.h5` when nothing decodes.** + It previously skipped the write "so we don't replace whatever's there with an + empty placeholder", which silently preserved output from a superseded + decoder. After the record-chain fix, 415 histogram files stopped decoding (an + unmapped block variant on BE18193 and BE9440) but kept `.h5` files whose peaks + ran up to **400× the device's own reported PPV** — garbage feeding the charts + and the false-trigger detector with nothing marking it. Reports a + `stale_h5_removed` count. + - **The series-3 waveform body is a RECORD CHAIN, not a tag stream — this supersedes the segment-header model, including the fixes made earlier the same day.** diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index 1242d6a..371e800 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -3083,6 +3083,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger | **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open | | ~~**Variable-prefix segment descriptors**~~ - **RESOLVED 2026-08-25:** there is no variable prefix. The body is a chain of self-delimiting records (see S7.6.1); the 0/2/4/6/8-byte prefix was walker drift from reading `40 NN` as a segment header. All 25 affected files now chain cleanly to the `06` terminator. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 | | ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 | +| **Unmapped histogram block variant (415 files, 2 units)** - after the 2026-08-25 record-chain fix these stop decoding entirely: 216 on BE18193 and 199 on BE9440, all histograms. Their bodies open `00 00 00 01 0a 00`, which is a valid block header (marker `0a` at [4], block_ctr 256 at [2:4] LE), but `block[28:32]` is neither the standard `1e 0a 00 00` nor the terminal `9c 06 00 42`, and no fixed stride between 8 and 64 bytes puts a marker at [4] consistently. Bodies are very large (one is 360,573 bytes). Previously these were being decoded by the WAVEFORM codec, which accepted them and returned garbage peaking up to 400x the device-reported PPV - so this is a pre-existing gap the fix merely exposed, not a regression. Their stale `.h5` files are now removed by `scripts/backfill_sidecars.py` rather than left behind. Example: `BE18193/T193L2X2.1J0H`. | MEDIUM | 2026-08-25 | Still open | | **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open | | **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open | diff --git a/scripts/backfill_sidecars.py b/scripts/backfill_sidecars.py index 04789a5..366b854 100644 --- a/scripts/backfill_sidecars.py +++ b/scripts/backfill_sidecars.py @@ -136,6 +136,7 @@ def main(argv=None) -> int: db = SeismoDb(db_path) written = skipped = errors = 0 + stale_h5_removed = 0 for serial_dir in sorted(p for p in store_root.iterdir() if p.is_dir()): serial = serial_dir.name for path in sorted(serial_dir.iterdir()): @@ -388,9 +389,16 @@ def main(argv=None) -> int: # waveform_codec.decode_waveform_v2 or histogram_codec. # decode_histogram_body. If samples are still empty after # both codecs run, it's a genuine "we can't decode this - # file" case (truncated, malformed, or unknown mode); - # skip the .h5 write so we don't replace whatever's - # there with an empty placeholder. + # file" case (truncated, malformed, or unknown mode). + # + # In that case we REMOVE any existing .h5 rather than leave + # it. Leaving it was the old behaviour and it silently + # preserved output from a superseded decoder: after the + # 2026-08-25 record-chain fix, 415 histogram files stopped + # decoding (an unmapped block variant on BE18193/BE9440) but + # kept .h5 files whose peaks were up to 400x the device's own + # reported PPV — garbage that fed the charts and the + # false-trigger detector with nothing marking it. has_samples = bool( ev.raw_samples and any( ev.raw_samples.get(ch) for ch in ("Tran", "Vert", "Long", "MicL") @@ -398,6 +406,13 @@ def main(argv=None) -> int: ) hdf5_path = store.hdf5_path_for(serial, path.name) hdf5_filename = hdf5_path.name if hdf5_path.exists() else None + if not has_samples and hdf5_path.exists(): + log.warning("%s: no samples decode — removing stale %s", + path.name, hdf5_path.name) + if not args.dry_run: + hdf5_path.unlink() + hdf5_filename = None + stale_h5_removed += 1 hdf5_action = "kept" need_h5 = ( not args.skip_hdf5 @@ -458,7 +473,7 @@ def main(argv=None) -> int: log.error("backfill failed for %s: %s", path, exc, exc_info=args.verbose) errors += 1 - print(f"\nDone. written={written} skipped(uptodate)={skipped} errors={errors}") + print(f"\nDone. written={written} skipped(uptodate)={skipped} errors={errors} stale_h5_removed={stale_h5_removed}") return 0 if errors == 0 else 1