Release v0.31.0 — report parity + the inverted rescue (0.29.0 → 0.31.0) #40
Assignees
serversdown
Clear assignees
serversdown
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: serversdown/seismo-relay#40
Reference in New Issue
Block a user
Release v0.31.0 — report parity + the inverted rescue
Ships two releases' worth: prod is on v0.29.0, so this carries v0.30.0 and
v0.31.0 together. 31 commits.
v0.31.0 — report parity + the inverted rescue
Blastware Event/FFT-Report parity. The FFT, the USBM RI8507 compliance
chart and the sensor self-check now render on the event report,
reverse-engineered against BE12844 (MiniMate Plus) and UM (Thor) events. The
FFT matches Blastware to the exact bin on all 28 oracle channels. The sensor
check is decoded for both series and standardized into the
.h5(schema v2, a new
/sensor_checkgroup) so SFM serves itdevice-agnostically rather than decoding at report time.
Field rescue for a runaway unit.
bridges/ach_server.pygains--stop-monitoring/--disable-ach/--rescue, which invert thewedged-unit recovery: rather than racing a Stop into the gaps between a unit's
dial-outs, point the modem's Destination at our own ACH server and answer the
call. Proven in production on BE12599 — the stop landed on the first call-in
and held. Runbook rewritten around it.
Plus the Inspector (annotated hex reader for series-3 binaries), two
event-report layout fixes, and a written changelog/release convention in
CLAUDE.md.v0.30.0 — series-4 correctness (never shipped to prod)
The Thor / Micromate counterpart to v0.26.0's series-3 work. Decoder verified
per-sample against Thor's own CSV exports: 459 files, 3,807,158 / 3,807,165
samples exact across three ground-truth corpora; production IDFW 575/575,
zero truncations, zero decode failures. Series-3 re-verified unchanged at
14,338/14,338.
⚠ Deploy owes a backfill — budget ~2 hours
Both releases owe one, and they combine into a single pass.
.h5schema v1 → v2 — existing files carry no/sensor_checkgroup, sotheir reports show no sensor-check strip until regenerated.
TOOL_VERSIONis 0.31.0, so the standard backfill regenerates everyevent and picks the traces up with no
--force.scripts/backfill_thor_events.pyfor series-4 (it already owed thev0.30.0 Thor backfill — this rides along) and the series-3 sidecar/shape
backfill for MiniMate events.
Purely additive — no decoded value changes, and v1
.h5files read fine untilthen (empty strip).
Deploy
Build the image from
mainafter merging, push to the Gitea registry,recreate the
sfmservice on the NAS. See the Gitea Registry +Container-Manager Prod Deploy runbook.
No schema change to the DB. No Terra-View coupling — nothing here changes a
/db/*shape it reads.Verified against a second Thor corpus (9-10-26-csv-req: UM11402, UM12947, UM20147) with per-sample CSV exports: 139/139 waveforms exact (1,273,380/1,273,380 samples) and 877/877 histograms within 2% of Thor's reported PPV -- up from 66.9% and 56.6%. Some units run with the microphone disabled, which changes two structural things that were both hardcoded to the 4-channel shape: - Waveform body head sat below the scan floor. A 3-channel unit has a shorter fixed header and puts its record chain head at 0x0dba, under the old _BODY_SCAN_FLOOR of 0x0E00. The scan could not see it and fell through to the Vert segment-0 record, decoding a body shifted one position around the channel rotation -- Vert came up exactly 512 samples short. Floor lowered to 0x0C00. The body-offset scoring also had to stop requiring four channels, or `equal` is permanently False for these events and the pick falls back to raw sample count. - Histogram interval record is 56 bytes, not 72. It is 16 * n_channels + 8, and is not inferable from the segment length alone. The interval count now comes from the segment's cumulative counter (n = counter - prev_counter) and the stride is derived from it. Assuming 72 read 7 intervals out of every 10-interval segment, then walked off alignment into garbage that decoded as ~10 in/s peaks -- inflating some files' PPV by up to 191,000%. Also recovers 4 files that previously decoded no intervals at all. Combined across both corpora: 292/292 waveform files, 2,330,916/2,330,916 samples exact. Production IDFW truncations 41 -> 22. Series-3 unaffected (no shared-codec change in this commit; last full run 14,338/14,338). Known open, diagnosed but NOT verified: the remaining 22 unequal + 1 failing production IDFW files (all UM12947, 2025-07-14..09-23) stop the block walker on tag 40 0c. data_block_len() caps the 40 NN int16 block at NN > 0x08 while those files use NN up to 196. Both verified corpora only ever use NN in {1,2,3,4,8}, so the cap is untested there and lifting it leaves both at 100.000% -- which is not evidence it decodes these correctly. Deliberately not shipped; needs Thor CSV exports for UM12947 in that date range. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmLdata_block_len() rejected any `40 NN` block with NN > 0x08. That guard had no evidence behind it: every corpus available when it was written used only NN in {1,2,3,4,8}, so it was never exercised. Loud UM12947 events use NN of 12, 16, 20 ... up to 196. Because walk_body/run stop at the first unrecognised tag rather than raising, rejecting those blocks surfaced as silently short channels -- e.g. Tran 1812 / Vert 2132 / Long 2324 on a file whose export carries 2324 for all three. The real bound is the buffer; the caller additionally clamps to the record end. Verified against Thor's own CSV exports for UM12947 (2025-07-14 .. 09-25, 167 waveforms, supplied as CSV.zip): length mismatches 22 -> 0 per-sample exact 1,476,242 / 1,476,249 These are NOT truncated recordings, which was the competing hypothesis -- the exports carry the full sample count. tests/test_waveform_codec.py asserted the cap as intended behaviour. That assertion encoded an assumption, not a verified fact, and is replaced with one pinning the opposite plus the evidence. Across all three ground-truth corpora: 459 waveform files, 3,807,158 / 3,807,165 samples exact. Production IDFW is now 575/575 with zero truncations and zero decode failures (median PPV error -0.0007% across 8 units). Series-3 re-verified unchanged at 14,338/14,338. The 7 residual samples each differ by one 4th-decimal tick and are Thor's own rounding: intersecting the per-sample rounding constraints over that corpus is infeasible (binding pair contradict by 2.3e-11, 7e-5 relative), so no single linear LSB reproduces every printed value. _GEO_LSB_IPS is already pinned to ~1e-11; do not retune it to chase these. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmLThe compliance chart on the event-report PDF was correctly drawn but far too small ("it's tiny") — a ~2.6in square dropped between the mic and stats rows. Resize + reposition it to match Blastware's Event Report, measured directly off a BW reference PDF (n844lqhbzt0w) rasterized with fitz: the chart data box now spans figure fractions x[0.489,0.951] y[0.502,0.867] — a ~3.9in square running from just under the header down through the stats band, hard against the right page margin, exactly as BW draws it. Title updated to BW's "USBM RI8507 And OSMRE". To clear room for the BW-sized chart (waveform layout only): * _draw_stats_table gains bbox_width/col_widths/fontsize params; the waveform layout packs the Tran/Vert/Long table into the left ~0.42 so its columns no longer sit under the chart. Histogram layout keeps the wider defaults (byte-identical output; it has no compliance chart). * the mic block's long "Channel Test Passed (Freq … Amp … mv)" line gets a tighter indent + one-point-smaller font so it ends before the chart's left edge instead of running behind it (_kv gains a fontsize param). * the Peak Vector Sum line left-aligns under the compacted table (one pt smaller) so it clears the chart's bottom-left tick labels. Chart placement centralized in the _COMPLIANCE_BOX constant. No change to the compliance math, the scatter, or the histogram report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzfThe Blastware Event Report draws a "Sensor Check" strip on the right of the waveform panel — the little traces the unit records when it pulses each sensor before monitoring. Those live in the series-3 binary's trailing block, after the main waveform record-chain and the per-channel calibration records, as four length-prefixed records tagged 0x3c-0x3f (Tran/Vert/Long geophone ring-downs + MicL pulse train). Reverse-engineered against 7 BE12844 oracle events. New minimateplus/sensor_check.py: decode_sensor_check(raw) locates the record chain (validated by walking the ids 0x3c->0x3f via their length prefixes) and decodes each record's delta stream (payload[20:len-8]) with the same 10/20/30/00 delta-block tags as the main waveform codec, from an anchor of 0. Returns {Tran,Vert,Long,MicL: [samples]} in raw 16-count units, or {} when absent. Validated: mic pulse-train zero-crossing frequency = 20.1 Hz (exact match to BW's mic Channel Test freq); geophone ring-downs are consistent ~-990 raw deflections that damp to a ~-310 settle across all 7 events (a fixed calibration pulse, so near-identical every run). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzfWire the decoded sensor self-check waveforms (previous commit) onto the event report PDF, and fold in two related waveform-panel cleanups. Sensor-check strip (matches Blastware): * ReportData gains sensor_check_waveforms; gather_report_data decodes it from the retained raw BW binary (store.paths_for) at report time — no ingest or .h5 change, waveform events only. * _draw_waveform_subplot now draws a narrow right-hand strip of per-channel mini-plots (MicL pulse train + Long/Vert/Tran ring-downs) aligned to the lanes, captioned "Sensor Check". * stats table gains the "Frequency" / "Overswing Ratio" sub-rows under Sensor Check (7.5/7.7/7.3 Hz, 3.6/3.3/3.7), formatted to 1 decimal like BW; values come from the already-parsed sensor_check scalars. Cleanups (pre-existing, in the same panel): * fix the stacked-lane y-tick collision — adjacent lanes' -1.0 / 1.0 labels overprinted at the shared boundary; prune the extreme ticks (MaxNLocator prune="both") so each lane shows clean interior ticks only. * fix the header serial+firmware line running off the right page edge — tighter right-column indent + BW's slightly smaller 7.5pt header. Tests: sensor-check + compliance + geo-scale + fft all green (15). The test_bw_ascii_report failures are pre-existing (gitignored decode-re fixtures absent in this worktree), unrelated to this change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzfThe Thor/Micromate (series-4) IDFW binary carries the sensor self-check in its fixed-header region (before the waveform body), as up to four records tagged 01 0e 3c/3d/3e/3f — the SAME channel ids as series-3 (Tran/Vert/Long/MicL). Unlike series-3's delta-coded trailing block, series-4 stores each trace as a raw int16-BE array after an 18-byte record header (2-byte sample count at offset +8). Three-channel (mic-disabled) units carry only 3c/3d/3e. New micromate/sensor_check.py: decode_idf_sensor_check(raw) locates the record chain (id-ordered marker run, so a stray body match can't chain) and reads each trace's int16 samples → {Tran,Vert,Long[,MicL]: [counts]}, or {} when absent. Reverse-engineered + validated against 4 UM oracle events (added as fixtures): clean geophone ring-downs on all, mic pulse trains on the 4-channel units, correctly no MicL on the two 3-channel units. Validated by shape + cross-event consistency (no Thor report strip to exact-match, unlike series-3's BW reports). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzfMake the sensor self-check a first-class part of the standardized decoded event so SFM stops decoding it at report time — device-agnostic, per the store's decoder→standardized-.h5→SFM model. * Event gains a `sensor_check` field; both decoders attach the traces where they set raw_samples — series-3 in event_file_io.read_blastware_file (minimateplus.sensor_check), series-4 in waveform_store's IDF path (micromate.sensor_check). Covers ingest and backfill (both re-decode). * event_hdf5 bumps schema_version 1→2 and writes an optional /sensor_check group (raw counts, int32, per channel present). read_event_hdf5 returns it; plot_json_from_hdf5 carries it as a top-level key. Old v1 files still read cleanly (no group → None), so nothing breaks before the backfill. * gather_report_data reads sensor_check_waveforms from the .h5 and drops the report-time series-3 decode — the report no longer reaches into a decoder, and a series-4 event now lights up the same strip automatically. Stored as raw counts (a shape diagnostic, rendered fit-to-box): the per-series count scale differs and a physical mic unit is ill-defined, so conversion would add complexity for no display benefit — easy to add later if a numeric use appears. Tests: .h5 roundtrip + backward-compat + plot_json + real series-3 decode attaches to the Event. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzfComplete the sensor-check standardization: existing events need their .h5 regenerated to gain the v2 /sensor_check group. * backfill_thor_events.py attaches the decoded series-4 traces (micromate.sensor_check) on its own IDF decode path, mirroring save_imported_idf, so regenerated Thor .h5 files get the group. Series-3 backfill needs no change — it re-decodes via read_blastware_file, which now attaches the traces itself. * TOOL_VERSION 0.30.0 → 0.31.0 so the standard backfill regenerates every event (no --force): the tool now produces the /sensor_check group. Purely additive — no decoded value changes. * CHANGELOG (Unreleased): sensor-check now series-3 + series-4, standardized into the .h5 (schema v2), with the ⚠ backfill note; FFT + compliance stay no-backfill (they read existing .h5 samples). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzfBrian described the practice: Unreleased is the staging area for what is going into the next release, and the version bump happens when enough has accumulated to be worth shipping — not per commit, not per merge. The convention already implied it ("never touch the changelog at a merge boundary") but never said it outright. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN