Merge pull request 'Release v0.31.0 — report parity + the inverted rescue (0.29.0 → 0.31.0)' (#40) from dev into main

Reviewed-on: #40

CHANGELOG.md

@@ -4,17 +4,30 @@ All notable changes to seismo-relay are documented here.
 
 ---
 
-## Unreleased
+## v0.31.0 — 2026-09-18
 
-**Blastware Event/FFT-Report parity — the FFT, the USBM compliance chart, and
+**Report parity, and a second way to rescue a runaway unit.**  Two threads.
-the sensor self-check.** Three analyses Blastware derives from event data,
+
-reverse-engineered against BE12844 (MiniMate Plus) reports and reproduced in
+The first closes out Blastware Event/FFT-Report parity: the FFT, the USBM
-seismo-relay: the compliance chart and the sensor-check strip now render on
+RI8507 compliance chart and the sensor self-check now render on the event
-the event-report PDF, and the FFT reproduces Blastware's FFT Report. All three
+report, reverse-engineered against BE12844 (MiniMate Plus) and UM (Thor)
-are additive and read from data already on disk — the `.h5` samples and the
+events.  The sensor check is decoded for **both** series and standardized into
-retained raw BW binary — so there is **no `.h5`/DB change, no migration, and no
+the `.h5` (schema **v2**, a new `/sensor_check` group), so SFM serves it
-backfill**: a report regenerated for an existing event simply gains the new
+device-agnostically rather than decoding at report time.  The Inspector — an
-panels.
+annotated hex reader for series-3 binaries — is what made the trailing-block
+structure findable, and it earned its keep by *ruling out* a stored FFT block
+and proving Blastware computes it from the samples.
+
+The second came out of a field emergency.  BE12599's connector fault drove its
+Tran channel to its trigger level, so the unit recorded back-to-back and dialed
+the office ACH server every ~75 s, unreachable the whole time.
+`bridges/ach_server.py` gained `--stop-monitoring` / `--disable-ach` /
+`--rescue`, which **invert** the recovery: instead of racing a Stop into the
+gaps between dial-outs, point the modem's Destination at our own ACH server and
+answer the call.  Proven in production the same night — the stop landed on the
+first call-in and held.  See `docs/runbooks/wedged_unit_recovery.md`.
+
+⚠ **This release owes prod a backfill** — see Migration below.
 
 ### Added
 - **Rescue-on-connect for `bridges/ach_server.py`** — `--stop-monitoring`
@@ -56,18 +69,25 @@ panels.
   reference PDF. A technical breakdown of the curve is in
   `docs/ri8507_compliance_curve.md`.
 
-- **Sensor self-check waveforms decoded and drawn (`minimateplus.sensor_check`).**
+- **Sensor self-check waveforms decoded and drawn — both series.** The little
-  The "Sensor Check" traces Blastware shows to the right of the waveform panel
+  "Sensor Check" traces (geophone ring-downs — the transducer's damped impulse
-  live in the series-3 binary's trailing block as four length-prefixed records
+  response — plus a MicL pulse train, the mic's known-signal gain check) are the
-  (`0x3c`–`0x3f`) using the same delta-block codec as the main waveform:
+  unit's proof its sensors were healthy when it recorded the event.
-  Tran/Vert/Long geophone ring-downs (the transducer's damped impulse response —
+  - **Series-3** (`minimateplus.sensor_check`): four records (`0x3c`–`0x3f`) in
-  resonant frequency + overswing/damping) and a MicL pulse train (the mic's
+    the binary's trailing block, same delta-block codec as the main waveform.
-  known-signal gain check). `gather_report_data` decodes them from the retained
+    Verified against all 7 BE12844 reports (mic zero-crossing = 20.1 Hz exact;
-  BW binary at report time; the report renders them as a strip flush against the
+    geophone ring-downs ~7.5 Hz, overswing ~3.5).
-  waveform panel plus the **Sensor Check → Frequency / Overswing Ratio** sub-rows
+  - **Series-4** (`micromate.sensor_check`): the same self-test in the Thor IDFW
-  in the stats table. Verified against the reports on all 7 oracle events (mic
+    fixed header — four `01 0e 3c/3d/3e/3f` records (same channel ids) storing
-  zero-crossing frequency = 20.1 Hz exact; geophone ring-downs consistent
+    raw int16 traces; three-channel (mic-disabled) units carry only the three
-  ~7.5 Hz with overswing ~3.5). Tests in `tests/test_sensor_check.py`.
+    geophones. Validated by shape + cross-event consistency.
+  - **Standardized into the `.h5`** (`/sensor_check`, schema v2): each series'
+    decoder attaches the traces to the event at decode, the writer persists
+    them, and `gather_report_data` reads them back — so SFM renders the strip
+    (flush against the waveform panel) plus the **Sensor Check → Frequency /
+    Overswing Ratio** sub-rows without knowing the source instrument.
+  - Tests: `tests/test_sensor_check.py`, `tests/test_sensor_check_idf.py`,
+    `tests/test_event_hdf5_sensor_check.py`.
 
 - **Inspector tab in `seismo_lab.py` — annotated hex reader for series-3
   binaries (`minimateplus/binary_annotate.py`).** Tiles a raw Blastware file
@@ -85,6 +105,26 @@ panels.
 
 ---
 
+### Migration
+
+⚠ **The sensor-check needs a backfill.**  Existing `.h5` files are schema v1
+and carry no `/sensor_check` group, so their reports show no sensor-check strip
+until regenerated.  `TOOL_VERSION` is bumped to **0.31.0**, so the standard
+backfill regenerates every event and picks up the traces with **no `--force`**:
+`scripts/backfill_thor_events.py` for series-4 (it already owed a v0.30.0 Thor
+backfill — this rides along) and the series-3 sidecar/shape backfill for
+MiniMate events.  Purely additive — no decoded value changes, and v1 `.h5`
+files read fine until then (empty strip).  DB backup first, as always.
+
+⚠ Budget **~2 h on the NAS** — ~1.5 files/sec there versus ~85/sec on the dev
+box (gzip-4 in `sfm/event_hdf5.py` against a Synology CPU).
+
+Everything else in this release owes nothing: the FFT, the USBM compliance
+chart and the `ach_server` rescue flags are additive and read data already on
+disk — no schema change, no DB migration.
+
+---
+
 ## v0.30.0 — 2026-09-12
 
 **The series-4 correctness release** — the Thor / Micromate counterpart to

CLAUDE.md

@@ -2,7 +2,7 @@
 
 Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
 managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
-(Sierra Wireless RV50 / RV55). Current version: **v0.30.0**.
+(Sierra Wireless RV50 / RV55). Current version: **v0.31.0**.
 
 Stack-level context — which repo owns what, and how the three project versions
 pair — lives in `../terra-view/docs/tmi-stack.md`, which is also loaded as
@@ -89,6 +89,47 @@ When new information about the protocol is discovered, please update the instant
 
 ---
 
+## Changelog & release convention
+
+**Feature branches do NOT touch `CHANGELOG.md`.  Write the entry on `dev`, as
+part of finishing the merge, under `## Unreleased`.  Cut the version on `dev` in a
+dedicated release commit when you are ready to ship to `main`.**
+
+- **The changelog is written on `dev`, never on a feature branch.**  With
+  several branches in flight they all edit the same few lines at the top of
+  the file and conflict every time.  Writing it once, after the merge, also
+  lets it describe what actually *landed* — including anything that changed
+  during conflict resolution.
+- ⚠ **The merge is not finished until `## Unreleased` is updated.**  Same sitting,
+  not "later" — that is the one failure mode of writing it after the fact.
+  Reconstruct from the branch's own commit messages:
+  `git log --oneline dev..<branch>` before you merge, or
+  `git log --oneline <merge-base>..<branch>` after.
+- **No preamble under `## Unreleased`** — just the `### Added` / `### Changed` /
+  `### Fixed` lists.  The themed opening paragraph gets written at release
+  time, when the whole release is visible and can be named honestly.  A theme
+  written when the first item landed is stale by the third.
+- ⚠ **State the operational consequence** on any entry touching the codec, the
+  waveform store, or the DB — **including when it is "none."**  "requires
+  `backfill_sidecars.py` + `backfill_event_shape.py`, ~2 h on the NAS",
+  "`TOOL_VERSION` bumped", "no schema change, no migration".  Silence is
+  ambiguous; "none" is information.  This repo's changelog is how future-you
+  learns whether a deploy costs two hours.
+- **Releases are cut on judgement, not on a schedule or a merge.**  `Unreleased`
+  is the staging area for whatever is going into the next release; when enough
+  has accumulated to be worth shipping, it gets a number and a date.  Nothing
+  about a merge to `dev` triggers a release.
+- **Cutting a release** is its own `chore(release): vX.Y.Z — <theme>` commit on
+  `dev`, renaming `## Unreleased` → `## vX.Y.Z — YYYY-MM-DD` and touching:
+  `CHANGELOG.md`, `pyproject.toml`, the version line in `CLAUDE.md` and
+  `README.md`, and `minimateplus/event_file_io.py` (`TOOL_VERSION`) **when the
+  codec changed** — that constant gates `.h5` regeneration.
+- **`main` carries only released versions.**  No `## Unreleased` section there;
+  it lands via the `dev` → `main` PR.  `main` lagging `dev` by a version is
+  normal.
+
+---
+
 ## Architecture: three-tier conceptual model
 
 seismo-relay is a **suite of cooperating components**, not a single app.

README.md

@@ -1,4 +1,4 @@
-# seismo-relay  `v0.30.0`
+# seismo-relay  `v0.31.0`
 
 A ground-up replacement for **Blastware** — Instantel's aging Windows-only
 software for managing seismographs.  Supports both the **MiniMate Plus

docs/runbooks/wedged_unit_recovery.md

@@ -1,6 +1,7 @@
 # Runbook — Recovering a wedged unit stuck in a call-home loop
 
-**Original incident:** BE9558H at `166.246.130.1:9034`, recovered 2026-05-17.
+**Incidents:** BE9558H at `166.246.130.1:9034`, 2026-05-17 (Method B) ·
+BE12599 at `166.246.64.226:9034`, 2026-09-16 (Method A).
 
 A field unit with a stuck-triggered geophone (or any hardware fault causing
 constant event triggering) will record events back-to-back, and if Auto Call
@@ -14,24 +15,30 @@ This runbook describes how to break the loop and recover control.
 
 ---
 
-## ⚠ There are TWO failure modes under "wedged unit"
+## ⚠ Two cures for one disease — intercept first
 
-Same root cause — an offset/connector fault drives the geophone above trigger,
+Both incidents below are the **same failure**: a geophone offset crosses the
-the unit records back-to-back, ACH set to "after event recorded" dials
+trigger level, the unit records back-to-back, ACH set to "after event
-constantly — but the *recovery* differs, because the thing blocking you is
+recorded" dials continuously, and the unit becomes unreachable because its
-different.
+modem is in client mode almost all of the time.
 
-| | **BE9558H (2026-05)** | **BE12599 (2026-09)** |
+There are two ways to get a Stop Monitoring command into it.
+
+| | **A — intercept the call** (preferred) | **B — catch it between calls** (original) |
 |---|---|---|
-| What blocks you | Modem mode-flipping kills inbound TCP | Device never enters S3 mode at all |
+| Idea | Be the server it dials.  Point the modem's Destination at our own ACH server and answer it. | Clear the Destination so it stops dialing, then race a Stop into the gap. |
-| Device state | Alive, in S3 mode, responsive once reached | Stuck repeating an **AT modem-init** string, deaf to S3 |
+| Needs inbound? | **No — the unit calls us** | Yes: working inbound TCP to the modem |
-| Fix direction | **Inbound** — clear Destination, slow-drip a Stop | **Inverted** — point Destination at our own ACH server and let the unit call *us* |
+| Determinism | Deterministic — it dials every ~75 s, we only have to be listening | A race.  BE9558H took ~7 h of attempts before one landed. |
-| Winning tool | `scripts/slow_drip.sh` | `bridges/ach_server.py --stop-monitoring` |
+| Tool | `bridges/ach_server.py --stop-monitoring` | `scripts/slow_drip.sh` |
+| Proven on | BE12599, 2026-09-16 | BE9558H, 2026-05-17 |
 
-**Tell them apart with the ALEOS serial log** (see "Turn on ALEOS_SERIAL debug"
+**Method A is the standard procedure now.**  The unit won't answer us because
-below). If the device is emitting `ATQ1/ATE0/ATS0=2` every ~75 s, it is in the
+it is on the phone — so stop dialing it and be the one it calls.  It rings,
-BE12599 mode and **no amount of inbound work will reach it** — skip to
+we pick up, take its data, and tell it to stop calling here.
-"Second incident" below.
+
+Method B is kept because it is proven, and because A needs a listener the
+modem can actually reach (public IP + forwarded port).  When you have that,
+don't race it — intercept it.
 
 ---
 
@@ -52,9 +59,85 @@ If you see *all* of these, the unit is in this exact failure mode.
 
 ---
 
-## Quick reference — how to recover
+## Method A (preferred) — intercept the call
 
-You need **ACEmanager access** to the unit's modem.
+You need **ACEmanager access** and a host the modem can dial: public IP with
+the listener's port forwarded to it.
+
+### A1 — start the listener BEFORE touching the modem
+
+```bash
+cd /home/serversdown/seismo-relay
+tmux new -s rescue
+.venv/bin/python -u bridges/ach_server.py --port 12345 \
+  -o bridges/captures/<unit>-diag --stop-monitoring -v
+```
+
+⚠ **Listener first, always.**  A Destination pointed at a dead port is the
+worst state available — the device still dials, the modem still flips to
+client mode, inbound stays blocked, and nothing gets delivered.
+
+Do **not** add `--events-only` (it silently breaks dedup — see gotchas), and
+do **not** add `--disable-ach` yet (see A4).
+
+### A2 — point the modem at it
+
+ACEmanager → **Serial → Port Configuration**:
+
+| Field | Set to |
+|---|---|
+| **Destination Address** | the listener's public IP |
+| **Destination Port** | the listener's port (e.g. `12345`) |
+
+Apply.  The modem auto-dials its Destination whenever serial data arrives
+while the serial port is closed — so the unit's own retry cycle now lands on
+you instead of nowhere.
+
+### A3 — answer, and stop the bleeding
+
+Within ~75 s you should see a call-in.  `--stop-monitoring` fires SUB 0x97 at
+step 1.5 — after the handshake, **before** the event walk — so the recording
+halts at the earliest possible moment in the session.  Confirm via
+`rescue.json` in the session directory:
+
+```json
+{"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
+```
+
+That is the bleeding stopped.  Everything after this is cleanup.
+
+### A4 — drain the backlog, THEN disable ACH
+
+⚠ **Order matters, and it is counter-intuitive.**  Stopping monitoring also
+removes your call-in trigger: ACH fires on "after event recorded", so with
+recording stopped the unit has no reason to dial again.  The backlog sitting
+in its memory does **not** re-arm it.
+
+So if the stored events are worth keeping — and on a fault unit they usually
+are, they're the evidence — drain them across however many call-ins it takes
+*before* you silence it.  Only then add `--disable-ach` (or use
+`scripts/rescue_device.sh <host> <port> --no-erase`).
+
+If the unit has gone quiet and you still need it, cycling the modem produces
+a call-in, and a unit with a scheduled daily call will dial at its configured
+time regardless.
+
+### A5 — restore the Destination, and confirm you did
+
+Put `Destination Address` back to `0.0.0.0` (or the office Instantel ACH
+server) once you are finished, and only stop the listener after that is done.
+
+### A6 — do NOT re-enable ACH until the hardware fault is repaired
+
+Otherwise the loop restarts the moment monitoring resumes and you run this
+runbook again.
+
+---
+
+## Method B (fallback) — catch it between calls
+
+The original 2026-05 procedure.  Use when you cannot stand up a listener the
+modem can reach.  You need **ACEmanager access** to the unit's modem.
 
 ### Step 1: stop the modem's mode-flipping
 
@@ -287,9 +370,14 @@ recovery: ~7 hours of intermittent debugging across one evening.
 level.  Constant triggering → constant recording → ACH "after event recorded"
 → continuous dialing.  Same disease as BE9558H.
 
-**But the recovery was the opposite direction**, and none of the Step 1–4
+**Same disease, inverted cure.**  Method B's Step 1 *did* work — clearing the
-procedure above worked.  Total time ≈ 5 h, of which ~90 min was spent on two
+Destination stopped the dial-outs, confirmed in the ALEOS log.  It was Step 2
-red herrings documented below.
+that didn't land, and rather than keep racing we turned the rescue around:
+gave the unit a different server to call, and answered it.
+
+Total time ≈ 5 h, of which ~90 min went to two red herrings documented below.
+Much of the rest was rediscovering the May procedure, which is why the
+"two cures" table now sits at the top of this file.
 
 ---
 
@@ -302,7 +390,7 @@ DEBUG**, then view the serial log.
 It is the only thing that tells you what the *device* is actually saying.
 Everything before we did this was guesswork.
 
-## What the log showed — the device was never in S3 mode
+## What the log showed — the unit is on the phone
 
 Every ~75 seconds, verbatim:
 
@@ -326,9 +414,15 @@ Read that carefully:
 - The device therefore never receives `OK`, never progresses, and **retries
   the identical 29 bytes forever**.
 
-**Consequence: the device is not running the S3 protocol parser.**  You can
+**While it is in this state it is busy placing a call, not listening for
-land a byte-perfect S3 frame on it and it will be ignored.  This is why every
+us.**  This is almost certainly what BE9558H was doing too — we simply never
-inbound approach failed, and it is the structural difference from BE9558H.
+turned on ALEOS_SERIAL debug in May to look.  It is not a different disease;
+it is the same one, seen properly for the first time.
+
+It is also the argument for Method A in one picture: the unit is mid-dial
+every ~75 s, and our inbound Stop has to thread the gaps between those
+attempts.  Give it somewhere to dial and the problem inverts into a
+deterministic one.
 
 ### Why `slow_drip` lied
 
@@ -351,6 +445,12 @@ nobody was reading.
 ⚠ **`send_error: null` + full duration is NOT success.  Only
 `bytes_received > 0` is success.**
 
+⚠ **In fairness to slow_drip: it got exactly one attempt here**, run ~90 s
+after a modem reboot, with a dead session visible in the log at 20:19:17 in
+that same window.  BE9558H took hours of attempts before one landed.  Method B
+was not ruled out on BE12599 so much as abandoned in favour of something that
+doesn't need luck.
+
 ---
 
 ## ⚠ Two red herrings that cost ~90 minutes

micromate/sensor_check.py

@@ -0,0 +1,89 @@
+r"""Decode the Thor / Micromate (series-4) sensor self-check waveforms from an
+IDFW event binary.
+
+Reverse-engineered 2026-09-15 against 4 UM (Thor) oracle events.  The IDFW
+binary carries the sensor self-check in its fixed-header region (before the
+waveform body), as up to four records tagged ``01 0e 3c/3d/3e/3f`` — the SAME
+channel ids as the series-3 MiniMate Plus (Tran / Vert / Long / MicL), which is
+the physical self-test:
+
+  * 3c / 3d / 3e = Tran / Vert / Long geophone ring-downs (a damped impulse
+    response — resonant frequency + damping).
+  * 3f          = MicL pulse train (the mic's known-signal gain check).  Absent
+    on three-channel (mic-disabled) units.
+
+Record framing (per record)::
+
+    01 0e [id:1] [flags:3] [count:2 BE] [pad:10] [int16-BE samples × count]
+    \___ 18-byte header ___/
+
+Unlike series-3's delta-coded trailing block, series-4 stores each trace as a
+raw int16 big-endian array.  ``count`` (the 2-byte field at header offset +8)
+is the sample count; the record is padded to a fixed stride after that.
+"""
+from __future__ import annotations
+
+import struct
+from typing import Dict, List
+
+# Record id → channel.  Same ids/order as series-3 (minimateplus.sensor_check).
+_ID_TO_CHANNEL = {0x3C: "Tran", 0x3D: "Vert", 0x3E: "Long", 0x3F: "MicL"}
+_CHAIN_IDS = (0x3C, 0x3D, 0x3E, 0x3F)
+
+_MARKER = b"\x01\x0e"       # precedes the 1-byte channel id
+_HEADER_LEN = 18            # bytes from the marker start to the first sample
+_COUNT_OFF = 8              # 2-byte BE sample count, from the marker start
+_MAX_COUNT = 4000           # sanity cap (traces are ~70-200 samples)
+
+
+def _find_chain(raw: bytes):
+    """Locate the sensor-check record chain.  Returns a list of
+    ``(offset, id, count)`` for the first run of markers whose ids run
+    3c, 3d, 3e[, 3f] in order, or ``[]``.
+
+    Records are padded to a fixed stride, so the next marker is not at
+    ``header + count*2``; instead collect every ``01 0e [id]`` marker with a
+    sane count and take the first id-ordered run.  Validating the id sequence
+    (not a lone ``01 0e 3c``) keeps a stray marker in the waveform body from
+    matching — the real chain sits in the fixed header, ahead of the body.
+    """
+    n = len(raw)
+    markers = []
+    for p in range(n - _HEADER_LEN):
+        if raw[p:p + 2] == _MARKER and raw[p + 2] in _ID_TO_CHANNEL:
+            count = int.from_bytes(raw[p + _COUNT_OFF:p + _COUNT_OFF + 2], "big")
+            if 0 < count <= _MAX_COUNT:
+                markers.append((p, raw[p + 2], count))
+
+    for i, (off, rid, _c) in enumerate(markers):
+        if rid != 0x3C:
+            continue
+        run = [markers[i]]
+        for m in markers[i + 1:]:
+            if len(run) < len(_CHAIN_IDS) and m[1] == _CHAIN_IDS[len(run)]:
+                run.append(m)
+            else:
+                break
+        if len(run) >= 3:       # 3-channel (mic-disabled) units are valid
+            return run
+    return []
+
+
+def decode_idf_sensor_check(raw: bytes) -> Dict[str, List[int]]:
+    """Decode the sensor self-check traces from a Thor/Micromate IDFW binary.
+
+    Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}`` in
+    raw int16 ADC counts (MicL omitted on 3-channel units), or ``{}`` if the
+    binary carries no sensor-check chain (a non-IDF file, or an IDFH histogram).
+    """
+    chain = _find_chain(raw)
+    if not chain:
+        return {}
+    out: Dict[str, List[int]] = {}
+    for off, rid, count in chain:
+        start = off + _HEADER_LEN
+        blob = raw[start:start + count * 2]
+        if len(blob) < count * 2:
+            continue
+        out[_ID_TO_CHANNEL[rid]] = list(struct.unpack(">%dh" % count, blob))
+    return out

minimateplus/event_file_io.py

@@ -50,7 +50,7 @@ SIDECAR_KIND   = "sfm.event"
 # bumped without a `pip install` re-run — leading to confusing stale
 # version stamps in sidecars.  Bump this constant and CHANGELOG.md
 # together at release time.
-TOOL_VERSION = "0.30.0"
+TOOL_VERSION = "0.31.0"   # +/sensor_check group (schema v2); gates the backfill regen
 
 try:
     # Best-effort: prefer the installed metadata when it's NEWER than the
@@ -960,6 +960,11 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
         project=project, client=client, operator=user, sensor_location=seisloc,
     )
     ev.raw_samples = samples
+    # Sensor self-check traces from the binary's trailing block (waveform
+    # events only; returns {} for histograms / when absent).  Carried on the
+    # Event so the .h5 writer persists them device-agnostically.
+    from minimateplus.sensor_check import decode_sensor_check
+    ev.sensor_check = decode_sensor_check(raw) or None
     # Only compute peaks from samples when we actually have samples.
     # For events the codec couldn't decode (histogram-mode bodies, until
     # the §7.6.2 histogram codec is wired in), samples is an empty dict

minimateplus/models.py

@@ -544,6 +544,15 @@ class Event:
     pretrig_samples: Optional[int] = None   # from STRT record: pre-trigger sample count
     rectime_seconds: Optional[int] = None   # from STRT record: record duration (seconds)
 
+    # Sensor self-check traces keyed by channel label — the short diagnostic
+    # waveforms the unit records when it pulses each sensor before monitoring
+    # (geophone ring-downs + a mic pulse train).  Decoded from the binary by
+    # the per-series decoder (minimateplus.sensor_check / micromate.sensor_check)
+    # and carried here so the .h5 writer can persist them device-agnostically.
+    # Raw ADC counts; the source series' scale differs but the trace is a
+    # shape diagnostic (rendered fit-to-box).  None when absent.
+    sensor_check: Optional[dict] = None     # {"Tran": [...], ..., "MicL": [...]}
+
     # ── Debug / introspection ─────────────────────────────────────────────────
     # Raw 210-byte waveform record bytes, set when debug mode is active.
     # Exposed by the SFM server via ?debug=true so field layouts can be verified.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "seismo-relay"
-version = "0.30.0"
+version = "0.31.0"
 description = "Python client and REST server for MiniMate Plus seismographs"
 requires-python = ">=3.10"
 dependencies = [

scripts/backfill_thor_events.py

@@ -305,6 +305,11 @@ def main(argv=None) -> int:
                                 default=0,
                             )
                             ev.total_samples = ev.total_samples or n_samp
+                            # Sensor self-check traces from the IDFW fixed
+                            # header, so regenerated .h5 files gain the v2
+                            # /sensor_check group (mirrors save_imported_idf).
+                            from micromate.sensor_check import decode_idf_sensor_check
+                            ev.sensor_check = decode_idf_sensor_check(binary_bytes) or None
 
                         event_hdf5.write_event_hdf5(
                             hdf5_path, ev,

sfm/event_hdf5.py

@@ -12,8 +12,11 @@ Layout written to `<filename>.h5`:
   ├─ samples_int16/ (optional)
   │   ├─ Tran  (int16, raw ADC counts) shape: (N,)
   │   └─ ... per channel (only when present in the source)
+  ├─ sensor_check/ (optional, schema v2+)
+  │   ├─ Tran  (int32, raw counts)     shape: (M,)   M ≪ N
+  │   └─ ... per channel present in the source (MicL absent on 3-channel units)
   └─ root attrs (event metadata):
-       schema_version       int    = 1
+       schema_version       int    = 2
        kind                 str    = "sfm.event.hdf5"
        serial               str
        waveform_key         str    (8-hex)
@@ -64,7 +67,7 @@ from minimateplus.models import Event
 
 log = logging.getLogger(__name__)
 
-SCHEMA_VERSION = 1
+SCHEMA_VERSION = 2   # v2 adds the optional /sensor_check group
 HDF5_KIND = "sfm.event.hdf5"
 
 # Geophone full-scale velocity per range (in/s).  Confirmed in CLAUDE.md
@@ -270,6 +273,22 @@ def write_event_hdf5(
                     )
             igrp.attrs["mic_psi_per_count"] = float(mic_factor)
 
+        # /sensor_check — optional short diagnostic self-check traces (schema
+        # v2+).  Raw ADC counts (a shape diagnostic; the per-series count scale
+        # differs, and the renderer fits each trace to its box).  Only channels
+        # the decoder found are written — 3-channel units carry no MicL.
+        sc = event.sensor_check or {}
+        if sc:
+            scgrp = f.create_group("sensor_check")
+            for ch in ("Tran", "Vert", "Long", "MicL"):
+                vals = sc.get(ch)
+                if vals:
+                    scgrp.create_dataset(
+                        ch, data=np.asarray(vals, dtype=np.int32),
+                        compression="gzip", compression_opts=4, shuffle=True,
+                    )
+            scgrp.attrs["units"] = "raw_counts"
+
     import os
     os.replace(tmp, path)
 
@@ -334,6 +353,16 @@ def read_event_hdf5(path: Union[str, Path]) -> dict:
             if mic_attr is not None:
                 mic_psi = float(mic_attr)
 
+        # /sensor_check — optional (schema v2+); absent on older files.
+        sensor_check = None
+        scgrp = f.get("sensor_check")
+        if scgrp is not None:
+            sensor_check = {}
+            for ch in ("Tran", "Vert", "Long", "MicL"):
+                ds = scgrp.get(ch)
+                if ds is not None:
+                    sensor_check[ch] = np.asarray(ds[()])
+
     return {
         "schema_version":     sv,
         "kind":               attrs.get("kind"),
@@ -341,6 +370,7 @@ def read_event_hdf5(path: Union[str, Path]) -> dict:
         "samples":            samples,
         "samples_int16":      samples_int16,
         "mic_psi_per_count":  mic_psi,
+        "sensor_check":       sensor_check,
     }
 
 
@@ -431,11 +461,16 @@ def plot_json_from_hdf5(
     event_id: Optional[str] = None,
     index: Optional[int] = None,
 ) -> dict:
-    """Build a `sfm.plot.v1` JSON dict from a stored .h5 file."""
+    """Build a `sfm.plot.v1` JSON dict from a stored .h5 file.
+
+    The dict also carries a top-level ``sensor_check`` key (the raw self-check
+    traces as ``{ch: [int]}``, or None) beyond the plot schema, so report
+    generation can read the traces from the same single .h5 load.
+    """
     data = read_event_hdf5(path)
     a = data["attrs"]
     s = data["samples"]
-    return _build_plot_dict(
+    out = _build_plot_dict(
         n_samples=len(s["Tran"]) if "Tran" in s else 0,
         sample_rate=int(a.get("sample_rate", 1024) or 1024),
         pretrig_samples=int(a.get("pretrig_samples", 0) or 0),
@@ -463,6 +498,11 @@ def plot_json_from_hdf5(
         event_id=event_id,
         index=index,
     )
+    scd = data.get("sensor_check")
+    out["sensor_check"] = (
+        {ch: v.tolist() for ch, v in scd.items()} if scd else None
+    )
+    return out
 
 
 def _build_plot_dict(

sfm/report_pdf.py

@@ -121,9 +121,11 @@ class ReportData:
     t0_ms:                  Optional[float] = None
     dt_ms:                  Optional[float] = None
 
-    # Sensor self-check traces — {ch: [samples]} in raw decode units, decoded
+    # Sensor self-check traces — {ch: [samples]} in raw counts, read from the
-    # from the binary's trailing block (see minimateplus.sensor_check). The
+    # standardized .h5 (/sensor_check group, schema v2+) where the per-series
-    # little waveforms BW draws in its "Sensor Check" strip. Empty when absent.
+    # decoder stored them at ingest.  The little diagnostic waveforms BW draws
+    # in its "Sensor Check" strip.  Empty when absent (pre-v2 .h5, histogram,
+    # or 3-channel unit's MicL).
     sensor_check_waveforms: dict = field(default_factory=dict)
 
     # Record-type discriminator
@@ -294,22 +296,15 @@ def gather_report_data(
             rd.pretrig_samples  = ta.get("pretrig_samples")
             rd.t0_ms            = ta.get("t0_ms")
             rd.dt_ms            = ta.get("dt_ms")
+            # Sensor self-check traces — read from the standardized .h5 (schema
+            # v2+).  Device-agnostic: whichever decoder produced the event
+            # stored them at ingest, so SFM reads them here without knowing or
+            # caring about the source instrument series.  Empty on pre-v2 files
+            # (until backfilled) and on 3-channel / histogram events.
+            rd.sensor_check_waveforms = wf.get("sensor_check") or {}
         except Exception as exc:
             log.warning("gather_report_data: hdf5 read failed: %s", exc)
 
-    # ── Sensor self-check traces — decoded from the retained raw binary ──
-    # The .h5 holds only the main waveform; the sensor-check traces live in the
-    # binary's trailing block, so decode them straight from the kept BW file.
-    # Waveform events only (histograms have no sensor-check strip).
-    if not rd.is_histogram:
-        try:
-            from minimateplus.sensor_check import decode_sensor_check
-            bw_path, _a5 = store.paths_for(serial, filename)
-            if bw_path.exists():
-                rd.sensor_check_waveforms = decode_sensor_check(bw_path.read_bytes())
-        except Exception as exc:
-            log.warning("gather_report_data: sensor-check decode failed: %s", exc)
-
     # ── Histogram aggregation ──
     # Codec emits ~N per-block samples (typically 1/sec); BW reports
     # one bar per configured interval (1 min / 5 min / etc.).  When

sfm/waveform_store.py

@@ -662,6 +662,11 @@ class WaveformStore:
             ev.raw_samples = idf_samples
             n_samples = max((len(idf_samples.get(ch, [])) for ch in ("Tran", "Vert", "Long", "MicL")), default=0)
             ev.total_samples = ev.total_samples or n_samples
+            # Sensor self-check traces from the IDFW fixed header (waveform
+            # events only; {} on histograms / when absent).  Carried on the
+            # bridged Event so the .h5 writer persists them like series-3.
+            from micromate.sensor_check import decode_idf_sensor_check
+            ev.sensor_check = decode_idf_sensor_check(idf_bytes) or None
 
         # For IDFH histograms there are no per-sample waveform arrays — the
         # device stores one peak ADC count per interval per channel.  Synthesise

tests/test_event_hdf5_sensor_check.py

@@ -0,0 +1,71 @@
+"""The event .h5 carries the sensor self-check traces (schema v2).
+
+The sensor check is decoded by the per-series decoder and attached to the
+standardized Event, so the .h5 writer persists it device-agnostically and SFM
+reads it back without knowing which instrument produced it.  Old v1 files (no
+sensor_check group) must still read cleanly.
+"""
+import tempfile
+from pathlib import Path
+
+import numpy as np
+
+from minimateplus.models import Event
+from minimateplus.event_file_io import read_blastware_file
+from sfm import event_hdf5
+
+S3_FIX = Path(__file__).parent / "fixtures" / "fft-oracle-2026-09-14" / "N844LQHB.ZT0W"
+
+
+def _write(ev, **kw):
+    d = Path(tempfile.mkdtemp())
+    p = d / "e.h5"
+    event_hdf5.write_event_hdf5(p, ev, serial="BE12844", **kw)
+    return p
+
+
+def test_sensor_check_roundtrips_through_hdf5():
+    ev = Event(index=0)
+    ev.raw_samples = {"Tran": [1, 2, -3], "Vert": [0, 1], "Long": [2], "MicL": [5, -5]}
+    ev.sample_rate = 1024
+    sc = {"Tran": [0, -990, -500, -100], "Vert": [0, -980, -480],
+          "Long": [0, -986, -470], "MicL": [0, -1800, 1800, -1800]}
+    ev.sensor_check = sc
+
+    r = event_hdf5.read_event_hdf5(_write(ev))
+    assert r["schema_version"] == 2
+    assert set(r["sensor_check"]) == {"Tran", "Vert", "Long", "MicL"}
+    for ch, vals in sc.items():
+        assert r["sensor_check"][ch].tolist() == vals
+
+
+def test_plot_json_carries_sensor_check():
+    ev = Event(index=0)
+    ev.raw_samples = {"Tran": [1, 2, 3]}
+    ev.sample_rate = 1024
+    ev.sensor_check = {"Tran": [0, -990, -500], "Vert": [0, -980],
+                       "Long": [0, -986]}   # 3-channel: no MicL
+    pj = event_hdf5.plot_json_from_hdf5(_write(ev))
+    assert pj["sensor_check"] is not None
+    assert "MicL" not in pj["sensor_check"]
+    assert pj["sensor_check"]["Tran"] == [0, -990, -500]
+
+
+def test_event_without_sensor_check_still_reads_as_v2():
+    ev = Event(index=0)
+    ev.raw_samples = {"Tran": [1, 2, 3]}
+    ev.sample_rate = 1024
+    r = event_hdf5.read_event_hdf5(_write(ev))
+    assert r["schema_version"] == 2
+    assert r["sensor_check"] is None
+    assert event_hdf5.plot_json_from_hdf5(_write(ev))["sensor_check"] is None
+
+
+def test_series3_decode_populates_event_sensor_check():
+    # The real series-3 decoder attaches the traces to the Event, so the
+    # ingest/backfill .h5 write picks them up with no extra plumbing.
+    ev = read_blastware_file(S3_FIX)
+    assert ev.sensor_check is not None
+    assert set(ev.sensor_check) == {"Tran", "Vert", "Long", "MicL"}
+    tran = np.asarray(ev.sensor_check["Tran"], dtype=float)
+    assert tran.min() < -800   # the geophone ring-down deflection

tests/test_sensor_check_idf.py

@@ -0,0 +1,67 @@
+"""Series-4 (Thor / Micromate IDFW) sensor self-check waveform decode.
+
+Reverse-engineered 2026-09-15 against 4 UM (Thor) oracle events.  The IDFW
+binary carries the sensor self-check in its fixed-header region (before the
+waveform body) as up to four records tagged ``01 0e 3c/3d/3e/3f`` — the SAME
+channel ids as series-3 (Tran/Vert/Long/MicL).  Unlike series-3's delta-coded
+trailing block, series-4 stores each trace as a raw int16-BE array after an
+18-byte record header whose sample count is a 2-byte field at offset +8.
+
+Three-channel (mic-disabled) Thor units carry only 3c/3d/3e — no MicL record.
+
+Validated by shape (geophone ring-down / mic pulse train) and cross-event
+consistency, since there's no Thor Event-Report strip to exact-match against.
+"""
+from pathlib import Path
+
+import numpy as np
+
+from micromate.sensor_check import decode_idf_sensor_check
+
+FIXDIR = Path(__file__).parent / "fixtures" / "thor-idf-sc"
+EVENTS = sorted(p.name for p in FIXDIR.glob("*.IDFW"))
+
+
+def _decode(name):
+    return decode_idf_sensor_check((FIXDIR / name).read_bytes())
+
+
+def test_geo_channels_present_and_ringdown_shaped():
+    # Every IDFW event has the three geophone self-checks; each is a large
+    # one-sided deflection (~15000 raw counts) that rings back — the geophone's
+    # damped impulse response.
+    for name in EVENTS:
+        sc = _decode(name)
+        for ch in ("Tran", "Vert", "Long"):
+            assert ch in sc, f"{name} missing {ch}"
+            tr = np.asarray(sc[ch], dtype=float)
+            tr = tr - tr[:4].mean()          # reference to the pre-trigger baseline
+            assert 40 <= len(tr) <= 300, f"{name}:{ch} n={len(tr)}"
+            assert tr.min() < -8000, f"{name}:{ch} min {tr.min()}"
+            # deflects one way and rings back toward / past the baseline
+            assert tr.max() < abs(tr.min()), f"{name}:{ch} not one-sided"
+
+
+def test_mic_present_only_on_four_channel_units():
+    # UM11719 / UM12947 record a mic; UM13981 / UM20147 are 3-channel
+    # (mic-disabled) units and carry no MicL self-check.
+    got = {name: ("MicL" in _decode(name)) for name in EVENTS}
+    assert any(got.values()), "expected at least one 4-channel unit"
+    assert not all(got.values()), "expected at least one 3-channel unit"
+    for name, has_mic in got.items():
+        if has_mic:
+            tr = np.asarray(_decode(name)["MicL"], dtype=float)
+            tr = tr - tr[:4].mean()
+            # mic self-check is a bipolar pulse train — swings both ways, wide range
+            assert tr.max() > 5000 and tr.min() < -5000, f"{name} mic not bipolar"
+
+
+def test_channel_ids_and_order():
+    # ids decode to the canonical channel names, geo always in Tran/Vert/Long order
+    sc = _decode(EVENTS[0])
+    assert [c for c in ("Tran", "Vert", "Long") if c in sc] == ["Tran", "Vert", "Long"]
+
+
+def test_returns_empty_on_non_idf_input():
+    assert decode_idf_sensor_check(b"not an IDF file") == {}
+    assert decode_idf_sensor_check(b"") == {}