Merge pull request 'Release v0.31.0 — report parity + the inverted rescue (0.29.0 → 0.31.0)' (#40) from dev into main

Reviewed-on: #40

CHANGELOG.md

@@ -4,6 +4,127 @@ All notable changes to seismo-relay are documented here.
 
 ---
 
+## v0.31.0 — 2026-09-18
+
+**Report parity, and a second way to rescue a runaway unit.**  Two threads.
+
+The first closes out Blastware Event/FFT-Report parity: the FFT, the USBM
+RI8507 compliance chart and the sensor self-check now render on the event
+report, reverse-engineered against BE12844 (MiniMate Plus) and UM (Thor)
+events.  The sensor check is decoded for **both** series and standardized into
+the `.h5` (schema **v2**, a new `/sensor_check` group), so SFM serves it
+device-agnostically rather than decoding at report time.  The Inspector — an
+annotated hex reader for series-3 binaries — is what made the trailing-block
+structure findable, and it earned its keep by *ruling out* a stored FFT block
+and proving Blastware computes it from the samples.
+
+The second came out of a field emergency.  BE12599's connector fault drove its
+Tran channel to its trigger level, so the unit recorded back-to-back and dialed
+the office ACH server every ~75 s, unreachable the whole time.
+`bridges/ach_server.py` gained `--stop-monitoring` / `--disable-ach` /
+`--rescue`, which **invert** the recovery: instead of racing a Stop into the
+gaps between dial-outs, point the modem's Destination at our own ACH server and
+answer the call.  Proven in production the same night — the stop landed on the
+first call-in and held.  See `docs/runbooks/wedged_unit_recovery.md`.
+
+⚠ **This release owes prod a backfill** — see Migration below.
+
+### Added
+- **Rescue-on-connect for `bridges/ach_server.py`** — `--stop-monitoring`
+  (SUB 0x97), `--disable-ach` (SUB 0x2C read → 0x7E write → 0x7F confirm) and
+  `--rescue` (both).  They fire immediately after the startup handshake and
+  **before** the event walk, so a unit that is recording back-to-back on a
+  stuck-triggered geophone is quieted as early in the session as possible.
+  Each action is independently guarded — a failure does not abort the download
+  — and the outcome is written to `rescue.json` in the session directory.
+
+  This inverts the `docs/runbooks/wedged_unit_recovery.md` approach.  That
+  runbook reaches the unit *inbound* and clears the modem's Destination Address
+  to stop it dialing.  When the device is instead wedged mid-modem-init — ALEOS
+  logs `tcpmode trying to send to invalid socket` and re-runs `Initialize Auto
+  answer` every ~75 s, orphaning any held inbound session — inbound cannot win.
+  Pointing the modem's Destination at an `ach_server` and letting the unit call
+  *us* gives a device-initiated session the modem bridges properly.
+
+  ⚠ Prefer `--stop-monitoring` alone on first contact.  `--disable-ach` stops
+  the unit calling, which is the only channel to a unit in this state; stopping
+  the recording ends the call-home loop on its own when ACH is
+  "after event recorded".
+
+- **Blastware-compatible channel FFT (`waveform_fft`).** Reproduces Blastware's
+  FFT Report: DC-removed, no window, zero-padded to 4096 (0.25 Hz bins at
+  1024 sps), single-sided `2/N` amplitude. Matches Blastware's dominant
+  frequency to the exact bin and the amplitude to report precision across all
+  28 channels of the 7-event BE12844 oracle set. `channel_spectrum()` /
+  `dominant_frequency()`; tests in `tests/test_waveform_fft.py`.
+
+- **USBM RI8507 / OSMRE compliance chart on the event-report PDF
+  (`sfm/compliance.py`).** The velocity-vs-frequency blasting-compliance
+  scatter Blastware draws in the upper-right of its Event Report: each channel's
+  significant cycles as `(frequency, peak velocity)` points (zero-crossing
+  method, so each channel's cloud tops out at its PPV) plotted against the
+  RI8507 Drywall (0.75 in/s) and plaster (0.50 in/s) limit curves, drawn
+  continuous (constant-displacement bounds meeting the plateaus — no vertical
+  steps). Sized and positioned to match a Blastware report, measured off the
+  reference PDF. A technical breakdown of the curve is in
+  `docs/ri8507_compliance_curve.md`.
+
+- **Sensor self-check waveforms decoded and drawn — both series.** The little
+  "Sensor Check" traces (geophone ring-downs — the transducer's damped impulse
+  response — plus a MicL pulse train, the mic's known-signal gain check) are the
+  unit's proof its sensors were healthy when it recorded the event.
+  - **Series-3** (`minimateplus.sensor_check`): four records (`0x3c`–`0x3f`) in
+    the binary's trailing block, same delta-block codec as the main waveform.
+    Verified against all 7 BE12844 reports (mic zero-crossing = 20.1 Hz exact;
+    geophone ring-downs ~7.5 Hz, overswing ~3.5).
+  - **Series-4** (`micromate.sensor_check`): the same self-test in the Thor IDFW
+    fixed header — four `01 0e 3c/3d/3e/3f` records (same channel ids) storing
+    raw int16 traces; three-channel (mic-disabled) units carry only the three
+    geophones. Validated by shape + cross-event consistency.
+  - **Standardized into the `.h5`** (`/sensor_check`, schema v2): each series'
+    decoder attaches the traces to the event at decode, the writer persists
+    them, and `gather_report_data` reads them back — so SFM renders the strip
+    (flush against the waveform panel) plus the **Sensor Check → Frequency /
+    Overswing Ratio** sub-rows without knowing the source instrument.
+  - Tests: `tests/test_sensor_check.py`, `tests/test_sensor_check_idf.py`,
+    `tests/test_event_hdf5_sensor_check.py`.
+
+- **Inspector tab in `seismo_lab.py` — annotated hex reader for series-3
+  binaries (`minimateplus/binary_annotate.py`).** Tiles a raw Blastware file
+  into labeled spans (header / STRT / body record-chain / trailing metadata +
+  calibration + sensor-check records / footer) so a binary can be combed by eye.
+
+### Fixed
+- **Event-report waveform panel — stacked-lane y-tick collision.** The lanes
+  touch, so each lane's bottom `-1.0` overprinted the next lane's top `1.0` at
+  the shared boundary. Prune the extreme ticks so each lane shows clean interior
+  ticks only.
+- **Event-report header — serial+firmware line ran off the page.** The long
+  `BE##### V ##.##-#.## MiniMate Plus` string overflowed the right margin;
+  tighter right-column indent + BW's slightly smaller header size so it fits.
+
+---
+
+### Migration
+
+⚠ **The sensor-check needs a backfill.**  Existing `.h5` files are schema v1
+and carry no `/sensor_check` group, so their reports show no sensor-check strip
+until regenerated.  `TOOL_VERSION` is bumped to **0.31.0**, so the standard
+backfill regenerates every event and picks up the traces with **no `--force`**:
+`scripts/backfill_thor_events.py` for series-4 (it already owed a v0.30.0 Thor
+backfill — this rides along) and the series-3 sidecar/shape backfill for
+MiniMate events.  Purely additive — no decoded value changes, and v1 `.h5`
+files read fine until then (empty strip).  DB backup first, as always.
+
+⚠ Budget **~2 h on the NAS** — ~1.5 files/sec there versus ~85/sec on the dev
+box (gzip-4 in `sfm/event_hdf5.py` against a Synology CPU).
+
+Everything else in this release owes nothing: the FFT, the USBM compliance
+chart and the `ach_server` rescue flags are additive and read data already on
+disk — no schema change, no DB migration.
+
+---
+
 ## v0.30.0 — 2026-09-12
 
 **The series-4 correctness release** — the Thor / Micromate counterpart to

CLAUDE.md

@@ -2,7 +2,7 @@
 
 Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
 managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
-(Sierra Wireless RV50 / RV55). Current version: **v0.30.0**.
+(Sierra Wireless RV50 / RV55). Current version: **v0.31.0**.
 
 Stack-level context — which repo owns what, and how the three project versions
 pair — lives in `../terra-view/docs/tmi-stack.md`, which is also loaded as
@@ -89,6 +89,47 @@ When new information about the protocol is discovered, please update the instant
 
 ---
 
+## Changelog & release convention
+
+**Feature branches do NOT touch `CHANGELOG.md`.  Write the entry on `dev`, as
+part of finishing the merge, under `## Unreleased`.  Cut the version on `dev` in a
+dedicated release commit when you are ready to ship to `main`.**
+
+- **The changelog is written on `dev`, never on a feature branch.**  With
+  several branches in flight they all edit the same few lines at the top of
+  the file and conflict every time.  Writing it once, after the merge, also
+  lets it describe what actually *landed* — including anything that changed
+  during conflict resolution.
+- ⚠ **The merge is not finished until `## Unreleased` is updated.**  Same sitting,
+  not "later" — that is the one failure mode of writing it after the fact.
+  Reconstruct from the branch's own commit messages:
+  `git log --oneline dev..<branch>` before you merge, or
+  `git log --oneline <merge-base>..<branch>` after.
+- **No preamble under `## Unreleased`** — just the `### Added` / `### Changed` /
+  `### Fixed` lists.  The themed opening paragraph gets written at release
+  time, when the whole release is visible and can be named honestly.  A theme
+  written when the first item landed is stale by the third.
+- ⚠ **State the operational consequence** on any entry touching the codec, the
+  waveform store, or the DB — **including when it is "none."**  "requires
+  `backfill_sidecars.py` + `backfill_event_shape.py`, ~2 h on the NAS",
+  "`TOOL_VERSION` bumped", "no schema change, no migration".  Silence is
+  ambiguous; "none" is information.  This repo's changelog is how future-you
+  learns whether a deploy costs two hours.
+- **Releases are cut on judgement, not on a schedule or a merge.**  `Unreleased`
+  is the staging area for whatever is going into the next release; when enough
+  has accumulated to be worth shipping, it gets a number and a date.  Nothing
+  about a merge to `dev` triggers a release.
+- **Cutting a release** is its own `chore(release): vX.Y.Z — <theme>` commit on
+  `dev`, renaming `## Unreleased` → `## vX.Y.Z — YYYY-MM-DD` and touching:
+  `CHANGELOG.md`, `pyproject.toml`, the version line in `CLAUDE.md` and
+  `README.md`, and `minimateplus/event_file_io.py` (`TOOL_VERSION`) **when the
+  codec changed** — that constant gates `.h5` regeneration.
+- **`main` carries only released versions.**  No `## Unreleased` section there;
+  it lands via the `dev` → `main` PR.  `main` lagging `dev` by a version is
+  normal.
+
+---
+
 ## Architecture: three-tier conceptual model
 
 seismo-relay is a **suite of cooperating components**, not a single app.

README.md

@@ -1,4 +1,4 @@
-# seismo-relay  `v0.30.0`
+# seismo-relay  `v0.31.0`
 
 A ground-up replacement for **Blastware** — Instantel's aging Windows-only
 software for managing seismographs.  Supports both the **MiniMate Plus

bridges/ach_server.py

@@ -177,6 +177,8 @@ class AchSession:
         store: "WaveformStore",
         clear_after_download: bool = False,
         restart_monitoring: bool = False,
+        rescue_stop_monitoring: bool = False,
+        rescue_disable_ach: bool = False,
         force_redownload: bool = False,
     ) -> None:
         self.sock                 = sock
@@ -190,6 +192,9 @@ class AchSession:
         self.store                = store
         self.clear_after_download = clear_after_download
         self.restart_monitoring   = restart_monitoring
+        # Rescue actions for a runaway unit — fired before the event walk.
+        self.rescue_stop_monitoring = rescue_stop_monitoring
+        self.rescue_disable_ach     = rescue_disable_ach
         # `force_redownload` tells this session to ignore ach_state and
         # re-download every event currently on the device, regardless of any
         # (key, timestamp) match.  Useful as a manual override when state has
@@ -290,6 +295,41 @@ class AchSession:
         root_logger.addHandler(fh)
 
         try:
+            # ── Step 1.5: rescue actions ──────────────────────────────────────
+            # Fired BEFORE the event walk so a runaway unit is quieted as early
+            # in the session as possible.  A unit whose geophone sits above the
+            # trigger threshold records back-to-back and, with ACH set to "after
+            # event recorded", re-dials every time — saturating its own firmware
+            # so it never services inbound requests.  See
+            # docs/runbooks/wedged_unit_recovery.md.
+            #
+            # Each action is independently guarded: a failure here must not
+            # abort the download that follows.
+            if self.rescue_stop_monitoring or self.rescue_disable_ach:
+                rescue: dict = {"peer": self.peer, "ts": ts}
+
+                if self.rescue_stop_monitoring:
+                    log.info("Step 1.5: RESCUE — stop monitoring (SUB 0x97)")
+                    try:
+                        client.stop_monitoring()
+                        rescue["stop_monitoring"] = "ok"
+                        log.info("  stop monitoring OK — device should stop recording")
+                    except Exception as exc:
+                        rescue["stop_monitoring"] = f"failed: {exc}"
+                        log.error("  stop monitoring FAILED: %s", exc)
+
+                if self.rescue_disable_ach:
+                    log.info("Step 1.5: RESCUE — disable auto call home (SUB 0x2C/0x7E/0x7F)")
+                    try:
+                        client.set_call_home_config(auto_call_home_enabled=False)
+                        rescue["disable_ach"] = "ok"
+                        log.info("  disable ACH OK — unit should stop calling home")
+                    except Exception as exc:
+                        rescue["disable_ach"] = f"failed: {exc}"
+                        log.error("  disable ACH FAILED: %s", exc)
+
+                _save_json(session_dir / "rescue.json", rescue)
+
             # ── Step 2: device info ───────────────────────────────────────────
             device_info = None
             if not self.events_only:
@@ -747,6 +787,13 @@ def serve(args: argparse.Namespace) -> None:
     print(f"  Max events per session: {max_ev if max_ev else 'unlimited'}")
     print(f"  Clear device after download: {'YES' if args.clear_after_download else 'no'}")
     print(f"  Restart monitoring after download: {'YES' if args.restart_monitoring else 'no'}")
+    _stop_mon = args.stop_monitoring or args.rescue
+    _dis_ach  = args.disable_ach or args.rescue
+    print(f"  RESCUE stop monitoring on connect:  {'YES' if _stop_mon else 'no'}")
+    print(f"  RESCUE disable auto call home:      {'YES' if _dis_ach else 'no'}")
+    if _stop_mon and args.restart_monitoring:
+        print("  !! --restart-monitoring will re-start the unit after download,")
+        print("     undoing --stop-monitoring.  Drop one of them.")
     print(f"  Force re-download all (ignore state): {'YES' if args.force_redownload_all else 'no'}")
     print(f"{'='*60}")
     print(f"\n  Point your test unit's ACEmanager call-home settings to:")
@@ -788,6 +835,8 @@ def serve(args: argparse.Namespace) -> None:
                     store=store,
                     clear_after_download=args.clear_after_download,
                     restart_monitoring=args.restart_monitoring,
+                    rescue_stop_monitoring=args.stop_monitoring or args.rescue,
+                    rescue_disable_ach=args.disable_ach or args.rescue,
                     force_redownload=args.force_redownload_all,
                 )
                 t = threading.Thread(target=session.run, daemon=True, name=f"ach-{peer}")
@@ -862,6 +911,32 @@ def parse_args() -> argparse.Namespace:
             "DCD on disconnect — without this the unit stays idle after a call-home."
         ),
     )
+    p.add_argument(
+        "--stop-monitoring",
+        action="store_true",
+        default=False,
+        help=(
+            "RESCUE: send SUB 0x97 (stop monitoring) immediately after the "
+            "handshake, before any event download. Use on a unit that is "
+            "recording back-to-back because of a stuck-triggered geophone."
+        ),
+    )
+    p.add_argument(
+        "--disable-ach",
+        action="store_true",
+        default=False,
+        help=(
+            "RESCUE: disable Auto Call Home on the device (SUB 0x2C read → "
+            "0x7E write → 0x7F confirm) immediately after the handshake. The "
+            "unit stops dialing out until ACH is explicitly re-enabled."
+        ),
+    )
+    p.add_argument(
+        "--rescue",
+        action="store_true",
+        default=False,
+        help="Shorthand for --stop-monitoring --disable-ach.",
+    )
     p.add_argument(
         "--clear-after-download",
         action="store_true",

docs/runbooks/wedged_unit_recovery.md

@@ -1,6 +1,7 @@
 # Runbook — Recovering a wedged unit stuck in a call-home loop
 
-**Original incident:** BE9558H at `166.246.130.1:9034`, recovered 2026-05-17.
+**Incidents:** BE9558H at `166.246.130.1:9034`, 2026-05-17 (Method B) ·
+BE12599 at `166.246.64.226:9034`, 2026-09-16 (Method A).
 
 A field unit with a stuck-triggered geophone (or any hardware fault causing
 constant event triggering) will record events back-to-back, and if Auto Call
@@ -14,6 +15,33 @@ This runbook describes how to break the loop and recover control.
 
 ---
 
+## ⚠ Two cures for one disease — intercept first
+
+Both incidents below are the **same failure**: a geophone offset crosses the
+trigger level, the unit records back-to-back, ACH set to "after event
+recorded" dials continuously, and the unit becomes unreachable because its
+modem is in client mode almost all of the time.
+
+There are two ways to get a Stop Monitoring command into it.
+
+| | **A — intercept the call** (preferred) | **B — catch it between calls** (original) |
+|---|---|---|
+| Idea | Be the server it dials.  Point the modem's Destination at our own ACH server and answer it. | Clear the Destination so it stops dialing, then race a Stop into the gap. |
+| Needs inbound? | **No — the unit calls us** | Yes: working inbound TCP to the modem |
+| Determinism | Deterministic — it dials every ~75 s, we only have to be listening | A race.  BE9558H took ~7 h of attempts before one landed. |
+| Tool | `bridges/ach_server.py --stop-monitoring` | `scripts/slow_drip.sh` |
+| Proven on | BE12599, 2026-09-16 | BE9558H, 2026-05-17 |
+
+**Method A is the standard procedure now.**  The unit won't answer us because
+it is on the phone — so stop dialing it and be the one it calls.  It rings,
+we pick up, take its data, and tell it to stop calling here.
+
+Method B is kept because it is proven, and because A needs a listener the
+modem can actually reach (public IP + forwarded port).  When you have that,
+don't race it — intercept it.
+
+---
+
 ## Symptoms
 
 - Terra-View / SFM `/device/info` either hangs or fails on `count_events()`.
@@ -31,9 +59,85 @@ If you see *all* of these, the unit is in this exact failure mode.
 
 ---
 
-## Quick reference — how to recover
+## Method A (preferred) — intercept the call
 
-You need **ACEmanager access** to the unit's modem.
+You need **ACEmanager access** and a host the modem can dial: public IP with
+the listener's port forwarded to it.
+
+### A1 — start the listener BEFORE touching the modem
+
+```bash
+cd /home/serversdown/seismo-relay
+tmux new -s rescue
+.venv/bin/python -u bridges/ach_server.py --port 12345 \
+  -o bridges/captures/<unit>-diag --stop-monitoring -v
+```
+
+⚠ **Listener first, always.**  A Destination pointed at a dead port is the
+worst state available — the device still dials, the modem still flips to
+client mode, inbound stays blocked, and nothing gets delivered.
+
+Do **not** add `--events-only` (it silently breaks dedup — see gotchas), and
+do **not** add `--disable-ach` yet (see A4).
+
+### A2 — point the modem at it
+
+ACEmanager → **Serial → Port Configuration**:
+
+| Field | Set to |
+|---|---|
+| **Destination Address** | the listener's public IP |
+| **Destination Port** | the listener's port (e.g. `12345`) |
+
+Apply.  The modem auto-dials its Destination whenever serial data arrives
+while the serial port is closed — so the unit's own retry cycle now lands on
+you instead of nowhere.
+
+### A3 — answer, and stop the bleeding
+
+Within ~75 s you should see a call-in.  `--stop-monitoring` fires SUB 0x97 at
+step 1.5 — after the handshake, **before** the event walk — so the recording
+halts at the earliest possible moment in the session.  Confirm via
+`rescue.json` in the session directory:
+
+```json
+{"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
+```
+
+That is the bleeding stopped.  Everything after this is cleanup.
+
+### A4 — drain the backlog, THEN disable ACH
+
+⚠ **Order matters, and it is counter-intuitive.**  Stopping monitoring also
+removes your call-in trigger: ACH fires on "after event recorded", so with
+recording stopped the unit has no reason to dial again.  The backlog sitting
+in its memory does **not** re-arm it.
+
+So if the stored events are worth keeping — and on a fault unit they usually
+are, they're the evidence — drain them across however many call-ins it takes
+*before* you silence it.  Only then add `--disable-ach` (or use
+`scripts/rescue_device.sh <host> <port> --no-erase`).
+
+If the unit has gone quiet and you still need it, cycling the modem produces
+a call-in, and a unit with a scheduled daily call will dial at its configured
+time regardless.
+
+### A5 — restore the Destination, and confirm you did
+
+Put `Destination Address` back to `0.0.0.0` (or the office Instantel ACH
+server) once you are finished, and only stop the listener after that is done.
+
+### A6 — do NOT re-enable ACH until the hardware fault is repaired
+
+Otherwise the loop restarts the moment monitoring resumes and you run this
+runbook again.
+
+---
+
+## Method B (fallback) — catch it between calls
+
+The original 2026-05 procedure.  Use when you cannot stand up a listener the
+modem can reach.  You need **ACEmanager access** to the unit's modem.
 
 ### Step 1: stop the modem's mode-flipping
 
@@ -253,3 +357,223 @@ service).
 
 Total time from "i was wondering if its possible to" first attempt to
 recovery: ~7 hours of intermittent debugging across one evening.
+
+---
+
+# Second incident — BE12599, 2026-09-16/17
+
+**Unit:** BE12599 at `166.246.64.226:9034`, RV50, job *I-80 North Fork Bridge
+— Abut 1 West* (Fay Company).  Same job as BE9558H, which is a coincidence.
+
+**Fault:** the connector fault documented in `docs/offset_investigation.md`
+§8e progressed until the Tran pedestal reached **0.400 in/s** — its trigger
+level.  Constant triggering → constant recording → ACH "after event recorded"
+→ continuous dialing.  Same disease as BE9558H.
+
+**Same disease, inverted cure.**  Method B's Step 1 *did* work — clearing the
+Destination stopped the dial-outs, confirmed in the ALEOS log.  It was Step 2
+that didn't land, and rather than keep racing we turned the rescue around:
+gave the unit a different server to call, and answered it.
+
+Total time ≈ 5 h, of which ~90 min went to two red herrings documented below.
+Much of the rest was rediscovering the May procedure, which is why the
+"two cures" table now sits at the top of this file.
+
+---
+
+## Turn on ALEOS_SERIAL debug FIRST
+
+This is the single highest-value diagnostic and it should be step zero on any
+future incident.  ACEmanager → **Admin → Log → ALEOS_SERIAL log level →
+DEBUG**, then view the serial log.
+
+It is the only thing that tells you what the *device* is actually saying.
+Everything before we did this was guesswork.
+
+## What the log showed — the unit is on the phone
+
+Every ~75 seconds, verbatim:
+
+```
+ALEOS_SERIAL_HIF: 29 byte(s) in buffer: 'ATQ1^MATE0^MATS0=2^M^MRADIO RING^M'
+ALEOS_SERIAL_HMC: TCP recvhost fd 65535 len 29  state TCPMode::kClosed
+ALEOS_SERIAL_HMC: tcpmode trying to send to invalid socket
+ALEOS_SERIAL_HMC: Connect to IP: 0.0.0.0 Port 0
+ALEOS_SERIAL_HMC: Initialize Auto answer on port 9034
+ALEOS_SERIAL_HMC: Cannot connect to 0.0.0.0
+```
+
+Read that carefully:
+
+- `ATQ1` (quiet) / `ATE0` (echo off) / `ATS0=2` (auto-answer after 2 rings).
+  **There is no `ATD`.**  The device is not dialing — it is trying to
+  *configure* its modem.
+- The modem's serial port is in TCP data mode, so it never interprets these
+  as AT commands.  It treats them as payload and tries to ship them to a TCP
+  socket that does not exist.
+- The device therefore never receives `OK`, never progresses, and **retries
+  the identical 29 bytes forever**.
+
+**While it is in this state it is busy placing a call, not listening for
+us.**  This is almost certainly what BE9558H was doing too — we simply never
+turned on ALEOS_SERIAL debug in May to look.  It is not a different disease;
+it is the same one, seen properly for the first time.
+
+It is also the argument for Method A in one picture: the unit is mid-dial
+every ~75 s, and our inbound Stop has to thread the gaps between those
+attempts.  Give it somewhere to dial and the problem inverts into a
+deterministic one.
+
+### Why `slow_drip` lied
+
+`slow_drip` returned the *success* signature except for the one field that
+mattered:
+
+```json
+{"duration_s":120.0,"drips_sent":38,"bytes_sent":920,
+ "bytes_received":0,"send_error":null}
+```
+
+Full duration, no broken pipe — but zero bytes back.  Cause is in the log
+above: each 75 s cycle re-runs `Initialize Auto answer on port 9034`, which
+orphans the held session (`data in for unknown reason 3 removing from
+select`, `OnMsg recv error: 107 - Transport endpoint is not connected`).  Our
+local TCP stayed open so `sendall` never raised — but the modem stopped
+bridging after the first re-init, so every drip after that went into a socket
+nobody was reading.
+
+⚠ **`send_error: null` + full duration is NOT success.  Only
+`bytes_received > 0` is success.**
+
+⚠ **In fairness to slow_drip: it got exactly one attempt here**, run ~90 s
+after a modem reboot, with a dead session visible in the log at 20:19:17 in
+that same window.  BE9558H took hours of attempts before one landed.  Method B
+was not ruled out on BE12599 so much as abandoned in favour of something that
+doesn't need luck.
+
+---
+
+## ⚠ Two red herrings that cost ~90 minutes
+
+### 1. The trusted-IP whitelist (this was the real reason inbound never worked)
+
+The RV50s run with **Security → Trusted IPs (Friends List) enabled**.  A
+source IP that is not on the list is dropped **silently** — inbound presents
+as `Connection error: timed out`, never a refusal.
+
+Brian's dev-box public IP is **dynamic** and had changed, so `tmi-dev` was no
+longer whitelisted.  Every inbound attempt failed identically across four
+different modem and device states, which looked exactly like the BE9558H
+mode-flipping symptom and sent us chasing modem configuration for over an
+hour.
+
+**Check this before diagnosing anything else.**  Note that SFM in Docker
+egresses via the *host's public IP*, not its LAN IP.
+
+### 2. A 502 from SFM does not mean TCP connected
+
+`sfm/server.py` raises **502 for both** failure classes:
+
+```python
+raise HTTPException(status_code=502, detail=f"Protocol error: {exc}")
+raise HTTPException(status_code=502, detail=f"Connection error: {exc}")
+```
+
+We read an early 502 as "TCP connected, modem bridged, device mute" and built
+a whole theory on it.  It was almost certainly a connect timeout.
+**Always read the `detail` string** — "connect failed" and "device didn't
+answer" are completely different problems and the status code will not
+separate them.
+
+---
+
+## What actually worked — invert the direction
+
+The key observation is in the log above:
+
+> `TCP recvhost ... state TCPMode::kClosed` → `Connect to IP: 0.0.0.0 Port 0`
+
+**The modem auto-dials its Destination whenever serial data arrives while
+closed.**  So instead of fighting for inbound, give it somewhere to dial:
+point `Destination Address` at our own `ach_server` and the device's own
+75-second attempts become **device-initiated sessions the modem bridges
+correctly**.  No race, no contention, worst case a 75-second wait.
+
+### Procedure
+
+1. **Run the rescue server** on a host the modem can reach (public IP +
+   forwarded port):
+
+   ```bash
+   cd /home/serversdown/seismo-relay
+   .venv/bin/python -u bridges/ach_server.py --port 12345 \
+     -o bridges/captures/<unit>-diag --stop-monitoring -v
+   ```
+
+2. **Point the modem at it** — ACEmanager → Serial → Port Configuration →
+   `Destination Address` = your public IP, `Destination Port` = 12345.
+
+3. **Wait for the call-in.**  `--stop-monitoring` fires SUB 0x97 at step 1.5,
+   after the handshake and *before* the event walk.  Confirm via
+   `rescue.json` in the session directory:
+
+   ```json
+   {"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
+   ```
+
+4. **Restore the modem's Destination** once you are done, then finish the
+   device side (disable ACH, erase) through whichever channel works.
+
+On BE12599 the first call-in landed at 20:58:11 and reported
+`stop_monitoring: ok`; a second at 20:58:20 confirmed it.  `is_monitoring:
+false` was still true **6½ hours later** — the fix is durable.
+
+---
+
+## Hard-won gotchas (do not re-derive)
+
+- **Never leave the Destination pointed at a host with nothing listening.**
+  That is the worst state available: the device still dials, the modem still
+  flips, inbound stays blocked, and nothing is delivered.  An 8-minute gap
+  with the listener down produced a spurious inbound timeout that cost
+  another round of misdiagnosis.
+
+- **Stopping monitoring removes your call-in channel.**  ACH is "after event
+  recorded"; no new events means no new dials.  The backlog sitting in memory
+  does *not* re-arm it.  After a successful stop the unit goes quiet and you
+  need the modem cycled (works — produced a call-in), the scheduled daily call
+  (BE12599 calls at **05:00:14 device-local**, per §8e), or working inbound.
+  **Plan the order before you fire the stop.**
+
+- **`--events-only` silently breaks dedup.**  It skips the device-info step,
+  so the serial is never read; `ach_state.json` then keys on
+  `peer:ephemeral_port`, which is unique per connection.  Every session looks
+  like a new unit, starts from key 0, and re-downloads the same event.  Four
+  sessions on BE12599 downloaded the identical event four times and made zero
+  progress on the backlog.  Events also file as `serial=UNKNOWN` with a
+  `M000…` BW filename (serial_numeric 0) instead of `N599…`.
+  **Do not use `--events-only` when you intend to download anything.**
+
+- **`/device/events/index` reported `lifetime_count: 0`** on a unit with years
+  of history.  Suspected decode bug in the SUB 0x08 field offset — do not
+  trust that number.  The 88-byte payload is preserved in the `raw_hex` field
+  if someone wants to chase it.
+
+- **Memory used cross-checks the event keys exactly:**
+  `last_key − buffer_start = memory_total − memory_free`.  On BE12599:
+  `0x011230ec − 0x01110000 = 78,060` and `983,028 − 904,968 = 78,060`.
+  Useful sanity check that you are reading the keys right.
+
+---
+
+## Final state (2026-09-17 ~01:30 local)
+
+- `is_monitoring: false`, held 6½ hours
+- Battery 6.76 V
+- Memory 78,060 / 983,028 bytes used (8%)
+- `first_key 01121728`, `last_key 011230ec` — ~6.6 KB of addressable event
+  chain, roughly 3 events
+- ACH still **enabled** — to be disabled after the backlog is preserved
+- Modem Destination still pointed at tmi-dev — to be restored
+- ⚠ **Do not re-enable ACH until the connector is serviced.**  Tran is still
+  sitting at 0.400 and the loop restarts the moment monitoring resumes.

docs/superpowers/plans/2026-09-17-rescue-listener.md

@@ -0,0 +1,134 @@
+# Plan — "Rescue Listener": a first-class tool for the inverted rescue
+
+**Status:** proposal, not started.  Written 2026-09-17 ~01:40 local, straight
+off the BE12599 incident.  Open questions at the bottom need Brian's answer
+before anything is built.
+
+**Background:** `docs/runbooks/wedged_unit_recovery.md`, "Second incident —
+BE12599".  The manual version of this worked; this plan is about making it a
+tool instead of a sequence of remembered steps at 1 AM.
+
+---
+
+## The problem, stated plainly
+
+When a unit is wedged in the BE12599 mode — geophone offset above trigger,
+recording back-to-back, ACH dialing constantly, device stuck repeating an AT
+modem-init string and therefore **deaf to S3 over inbound** — the only channel
+that works is the one the *device* opens.
+
+Recovering it currently means:
+
+1. Remember that `bridges/ach_server.py` exists and takes the right flags
+2. Start it by hand on a box the modem can reach, with a public port forwarded
+3. Go into ACEmanager and repoint the modem's Destination
+4. Watch a terminal for a call-in
+5. Read `rescue.json` to find out whether it worked
+6. Go back into ACEmanager and repoint the modem to where it belongs
+7. **Not forget step 6**, because leaving the Destination pointed at a dead
+   listener is worse than never having started
+
+That is six manual steps and one landmine, executed under pressure while a
+unit floods the office server.
+
+## What the tool should be
+
+**A "rescue listener" an operator can start for one unit, which handles
+whatever that unit says when it calls in, and refuses to go away until the
+operator confirms the modem has been pointed back.**
+
+Lifecycle:
+
+1. **Start** — operator names the target unit and starts a rescue listener.
+   The tool reports the exact address/port to enter in ACEmanager, plus the
+   actions it will take.
+2. **Operator repoints the modem** to that address.
+3. **Wait** — listener sits there.  Live status: "waiting for call-in",
+   elapsed, last-seen.
+4. **Act** — on call-in, run the configured rescue actions automatically,
+   in a safe order, each independently guarded.  Report per-action outcome.
+5. **Hold** — the listener **stays up** and keeps reporting, because the
+   modem is still pointed at it.
+6. **Confirm & stop** — the operator explicitly confirms the Destination has
+   been restored (to `0.0.0.0`, or to the office Instantel ACH server).
+   Only then does the listener shut down.
+
+Step 6 is the whole point of making this a tool.  It is the step that is
+easiest to skip and most expensive to skip.
+
+## Default action set
+
+Ordered deliberately — see "order matters" below.
+
+| # | Action | Default | Why |
+|---|---|---|---|
+| 1 | **Stop monitoring** (SUB 0x97) | ✅ on | Halts recording; ends the trigger→record→dial loop at its source.  Already implemented as `--stop-monitoring`. |
+| 2 | **Drain events** to a diagnostics store | ⚙ configurable | The backlog is usually evidence, not garbage — see the BE12599 offset investigation.  Must NOT land in the prod SFM DB. |
+| 3 | **Disable ACH** (SUB 0x2C/0x7E/0x7F) | ❌ off by default | Stops the dialing — **and stops your only channel**.  Opt-in, and ideally gated on step 1 having succeeded. |
+| 4 | **Erase events** | ❌ off by default | Destructive.  Only after a verified drain. |
+
+### Order matters — the lesson from BE12599
+
+Stopping monitoring *removes the call-in trigger*.  ACH fires on "after event
+recorded"; with recording stopped, the unit has no reason to dial again, even
+though the backlog is still sitting in its memory.  So a naive
+"stop + disable + erase, all at once" rescue can silence the unit before
+you've collected anything, leaving you with no channel and a device full of
+evidence.
+
+The tool should either sequence around this or warn loudly about it.  My
+instinct is: **stop monitoring immediately** (it's the bleeding), then drain
+across however many call-ins it takes, and treat disable-ACH/erase as a
+separate, explicit "finish" action once the operator is satisfied.
+
+## Where it should live — open question, with a proposal
+
+The natural tier is **SFM** (device-side, per the three-tier model in
+CLAUDE.md).  But the rescue listener must be reachable *from the cellular
+network*, which is a deployment constraint SFM's usual profile doesn't have.
+
+**Proposal worth considering:** run it at the office, beside the real Instantel
+ACH server, on a **different port** (e.g. 12346 while Instantel holds 12345).
+Then the ACEmanager change is a **port change, not an IP change** — smaller,
+faster, less to get wrong, and trivially reversible.  It also means the office
+public IP (already stable and known) is the destination, rather than whatever
+Brian's dynamic home IP happens to be that week.
+
+The tmi-dev approach used on BE12599 worked, but required a router forward and
+ran into the dynamic-IP problem in the same session.
+
+## Open questions
+
+1. **Where does it run?**  Office beside Instantel ACH (port swap), SFM on the
+   NAS, or ad-hoc on tmi-dev?  Affects everything else.
+2. **What drives it?**  Terra-View admin page (fits "operator UI"), an SFM
+   endpoint pair (`POST /device/rescue_listener/start` + `/stop` + `/status`),
+   or a CLI wrapper?  A long-lived listener doesn't fit the request/response
+   endpoint shape well — probably needs a background task with a status poll.
+3. **How does it identify the unit?**  It can't know the serial until the
+   device calls in and the handshake reads it.  Allowlist by modem IP?  Accept
+   anything and report what showed up?
+4. **Where do drained events go?**  A per-incident diagnostics store
+   (`bridges/captures/<unit>-diag`) seems right — explicitly *not* the prod
+   SFM DB.  Does that store need to be a first-class thing with its own
+   retention, or is a directory fine?
+5. **How is "confirm the modem is repointed" verified?**  Operator attestation
+   (a button), or can we actually probe it?  If the listener stops seeing
+   call-ins that's weak evidence; if inbound to the unit starts working that's
+   stronger.
+6. **Multi-unit?**  One listener per incident, or one listener that handles any
+   unit that dials in?  Probably the former for safety.
+7. **Timeout / abandonment policy.**  If nobody ever confirms, does it run
+   forever?  Alert after N hours?
+
+## What already exists
+
+- `bridges/ach_server.py` — the listener itself, with `--stop-monitoring`,
+  `--disable-ach`, `--rescue` (added on `feat/ach-rescue-on-connect`, commit
+  `9f1050b`), `--clear-after-download`, `--max-events`, `--allow-ip`.
+- Per-session `rescue.json` recording per-action outcomes.
+- Isolated per-output-dir SQLite + waveform store, so a diagnostics capture is
+  already separate from prod by construction.
+
+So the gap is not protocol work — it's lifecycle, operator surface, and the
+confirmation gate.  Most of the risk is in questions 1 and 2.

micromate/sensor_check.py

@@ -0,0 +1,89 @@
+r"""Decode the Thor / Micromate (series-4) sensor self-check waveforms from an
+IDFW event binary.
+
+Reverse-engineered 2026-09-15 against 4 UM (Thor) oracle events.  The IDFW
+binary carries the sensor self-check in its fixed-header region (before the
+waveform body), as up to four records tagged ``01 0e 3c/3d/3e/3f`` — the SAME
+channel ids as the series-3 MiniMate Plus (Tran / Vert / Long / MicL), which is
+the physical self-test:
+
+  * 3c / 3d / 3e = Tran / Vert / Long geophone ring-downs (a damped impulse
+    response — resonant frequency + damping).
+  * 3f          = MicL pulse train (the mic's known-signal gain check).  Absent
+    on three-channel (mic-disabled) units.
+
+Record framing (per record)::
+
+    01 0e [id:1] [flags:3] [count:2 BE] [pad:10] [int16-BE samples × count]
+    \___ 18-byte header ___/
+
+Unlike series-3's delta-coded trailing block, series-4 stores each trace as a
+raw int16 big-endian array.  ``count`` (the 2-byte field at header offset +8)
+is the sample count; the record is padded to a fixed stride after that.
+"""
+from __future__ import annotations
+
+import struct
+from typing import Dict, List
+
+# Record id → channel.  Same ids/order as series-3 (minimateplus.sensor_check).
+_ID_TO_CHANNEL = {0x3C: "Tran", 0x3D: "Vert", 0x3E: "Long", 0x3F: "MicL"}
+_CHAIN_IDS = (0x3C, 0x3D, 0x3E, 0x3F)
+
+_MARKER = b"\x01\x0e"       # precedes the 1-byte channel id
+_HEADER_LEN = 18            # bytes from the marker start to the first sample
+_COUNT_OFF = 8              # 2-byte BE sample count, from the marker start
+_MAX_COUNT = 4000           # sanity cap (traces are ~70-200 samples)
+
+
+def _find_chain(raw: bytes):
+    """Locate the sensor-check record chain.  Returns a list of
+    ``(offset, id, count)`` for the first run of markers whose ids run
+    3c, 3d, 3e[, 3f] in order, or ``[]``.
+
+    Records are padded to a fixed stride, so the next marker is not at
+    ``header + count*2``; instead collect every ``01 0e [id]`` marker with a
+    sane count and take the first id-ordered run.  Validating the id sequence
+    (not a lone ``01 0e 3c``) keeps a stray marker in the waveform body from
+    matching — the real chain sits in the fixed header, ahead of the body.
+    """
+    n = len(raw)
+    markers = []
+    for p in range(n - _HEADER_LEN):
+        if raw[p:p + 2] == _MARKER and raw[p + 2] in _ID_TO_CHANNEL:
+            count = int.from_bytes(raw[p + _COUNT_OFF:p + _COUNT_OFF + 2], "big")
+            if 0 < count <= _MAX_COUNT:
+                markers.append((p, raw[p + 2], count))
+
+    for i, (off, rid, _c) in enumerate(markers):
+        if rid != 0x3C:
+            continue
+        run = [markers[i]]
+        for m in markers[i + 1:]:
+            if len(run) < len(_CHAIN_IDS) and m[1] == _CHAIN_IDS[len(run)]:
+                run.append(m)
+            else:
+                break
+        if len(run) >= 3:       # 3-channel (mic-disabled) units are valid
+            return run
+    return []
+
+
+def decode_idf_sensor_check(raw: bytes) -> Dict[str, List[int]]:
+    """Decode the sensor self-check traces from a Thor/Micromate IDFW binary.
+
+    Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}`` in
+    raw int16 ADC counts (MicL omitted on 3-channel units), or ``{}`` if the
+    binary carries no sensor-check chain (a non-IDF file, or an IDFH histogram).
+    """
+    chain = _find_chain(raw)
+    if not chain:
+        return {}
+    out: Dict[str, List[int]] = {}
+    for off, rid, count in chain:
+        start = off + _HEADER_LEN
+        blob = raw[start:start + count * 2]
+        if len(blob) < count * 2:
+            continue
+        out[_ID_TO_CHANNEL[rid]] = list(struct.unpack(">%dh" % count, blob))
+    return out

minimateplus/event_file_io.py

@@ -50,7 +50,7 @@ SIDECAR_KIND   = "sfm.event"
 # bumped without a `pip install` re-run — leading to confusing stale
 # version stamps in sidecars.  Bump this constant and CHANGELOG.md
 # together at release time.
-TOOL_VERSION = "0.30.0"
+TOOL_VERSION = "0.31.0"   # +/sensor_check group (schema v2); gates the backfill regen
 
 try:
     # Best-effort: prefer the installed metadata when it's NEWER than the
@@ -960,6 +960,11 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
         project=project, client=client, operator=user, sensor_location=seisloc,
     )
     ev.raw_samples = samples
+    # Sensor self-check traces from the binary's trailing block (waveform
+    # events only; returns {} for histograms / when absent).  Carried on the
+    # Event so the .h5 writer persists them device-agnostically.
+    from minimateplus.sensor_check import decode_sensor_check
+    ev.sensor_check = decode_sensor_check(raw) or None
     # Only compute peaks from samples when we actually have samples.
     # For events the codec couldn't decode (histogram-mode bodies, until
     # the §7.6.2 histogram codec is wired in), samples is an empty dict

minimateplus/models.py

@@ -544,6 +544,15 @@ class Event:
     pretrig_samples: Optional[int] = None   # from STRT record: pre-trigger sample count
     rectime_seconds: Optional[int] = None   # from STRT record: record duration (seconds)
 
+    # Sensor self-check traces keyed by channel label — the short diagnostic
+    # waveforms the unit records when it pulses each sensor before monitoring
+    # (geophone ring-downs + a mic pulse train).  Decoded from the binary by
+    # the per-series decoder (minimateplus.sensor_check / micromate.sensor_check)
+    # and carried here so the .h5 writer can persist them device-agnostically.
+    # Raw ADC counts; the source series' scale differs but the trace is a
+    # shape diagnostic (rendered fit-to-box).  None when absent.
+    sensor_check: Optional[dict] = None     # {"Tran": [...], ..., "MicL": [...]}
+
     # ── Debug / introspection ─────────────────────────────────────────────────
     # Raw 210-byte waveform record bytes, set when debug mode is active.
     # Exposed by the SFM server via ?debug=true so field layouts can be verified.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "seismo-relay"
-version = "0.30.0"
+version = "0.31.0"
 description = "Python client and REST server for MiniMate Plus seismographs"
 requires-python = ">=3.10"
 dependencies = [

scripts/backfill_thor_events.py

@@ -305,6 +305,11 @@ def main(argv=None) -> int:
                                 default=0,
                             )
                             ev.total_samples = ev.total_samples or n_samp
+                            # Sensor self-check traces from the IDFW fixed
+                            # header, so regenerated .h5 files gain the v2
+                            # /sensor_check group (mirrors save_imported_idf).
+                            from micromate.sensor_check import decode_idf_sensor_check
+                            ev.sensor_check = decode_idf_sensor_check(binary_bytes) or None
 
                         event_hdf5.write_event_hdf5(
                             hdf5_path, ev,

sfm/event_hdf5.py

@@ -12,8 +12,11 @@ Layout written to `<filename>.h5`:
   ├─ samples_int16/ (optional)
   │   ├─ Tran  (int16, raw ADC counts) shape: (N,)
   │   └─ ... per channel (only when present in the source)
+  ├─ sensor_check/ (optional, schema v2+)
+  │   ├─ Tran  (int32, raw counts)     shape: (M,)   M ≪ N
+  │   └─ ... per channel present in the source (MicL absent on 3-channel units)
   └─ root attrs (event metadata):
-       schema_version       int    = 1
+       schema_version       int    = 2
        kind                 str    = "sfm.event.hdf5"
        serial               str
        waveform_key         str    (8-hex)
@@ -64,7 +67,7 @@ from minimateplus.models import Event
 
 log = logging.getLogger(__name__)
 
-SCHEMA_VERSION = 1
+SCHEMA_VERSION = 2   # v2 adds the optional /sensor_check group
 HDF5_KIND = "sfm.event.hdf5"
 
 # Geophone full-scale velocity per range (in/s).  Confirmed in CLAUDE.md
@@ -270,6 +273,22 @@ def write_event_hdf5(
                     )
             igrp.attrs["mic_psi_per_count"] = float(mic_factor)
 
+        # /sensor_check — optional short diagnostic self-check traces (schema
+        # v2+).  Raw ADC counts (a shape diagnostic; the per-series count scale
+        # differs, and the renderer fits each trace to its box).  Only channels
+        # the decoder found are written — 3-channel units carry no MicL.
+        sc = event.sensor_check or {}
+        if sc:
+            scgrp = f.create_group("sensor_check")
+            for ch in ("Tran", "Vert", "Long", "MicL"):
+                vals = sc.get(ch)
+                if vals:
+                    scgrp.create_dataset(
+                        ch, data=np.asarray(vals, dtype=np.int32),
+                        compression="gzip", compression_opts=4, shuffle=True,
+                    )
+            scgrp.attrs["units"] = "raw_counts"
+
     import os
     os.replace(tmp, path)
 
@@ -334,6 +353,16 @@ def read_event_hdf5(path: Union[str, Path]) -> dict:
             if mic_attr is not None:
                 mic_psi = float(mic_attr)
 
+        # /sensor_check — optional (schema v2+); absent on older files.
+        sensor_check = None
+        scgrp = f.get("sensor_check")
+        if scgrp is not None:
+            sensor_check = {}
+            for ch in ("Tran", "Vert", "Long", "MicL"):
+                ds = scgrp.get(ch)
+                if ds is not None:
+                    sensor_check[ch] = np.asarray(ds[()])
+
     return {
         "schema_version":     sv,
         "kind":               attrs.get("kind"),
@@ -341,6 +370,7 @@ def read_event_hdf5(path: Union[str, Path]) -> dict:
         "samples":            samples,
         "samples_int16":      samples_int16,
         "mic_psi_per_count":  mic_psi,
+        "sensor_check":       sensor_check,
     }
 
 
@@ -431,11 +461,16 @@ def plot_json_from_hdf5(
     event_id: Optional[str] = None,
     index: Optional[int] = None,
 ) -> dict:
-    """Build a `sfm.plot.v1` JSON dict from a stored .h5 file."""
+    """Build a `sfm.plot.v1` JSON dict from a stored .h5 file.
+
+    The dict also carries a top-level ``sensor_check`` key (the raw self-check
+    traces as ``{ch: [int]}``, or None) beyond the plot schema, so report
+    generation can read the traces from the same single .h5 load.
+    """
     data = read_event_hdf5(path)
     a = data["attrs"]
     s = data["samples"]
-    return _build_plot_dict(
+    out = _build_plot_dict(
         n_samples=len(s["Tran"]) if "Tran" in s else 0,
         sample_rate=int(a.get("sample_rate", 1024) or 1024),
         pretrig_samples=int(a.get("pretrig_samples", 0) or 0),
@@ -463,6 +498,11 @@ def plot_json_from_hdf5(
         event_id=event_id,
         index=index,
     )
+    scd = data.get("sensor_check")
+    out["sensor_check"] = (
+        {ch: v.tolist() for ch, v in scd.items()} if scd else None
+    )
+    return out
 
 
 def _build_plot_dict(

sfm/report_pdf.py

@@ -121,9 +121,11 @@ class ReportData:
     t0_ms:                  Optional[float] = None
     dt_ms:                  Optional[float] = None
 
-    # Sensor self-check traces — {ch: [samples]} in raw decode units, decoded
+    # Sensor self-check traces — {ch: [samples]} in raw counts, read from the
-    # from the binary's trailing block (see minimateplus.sensor_check). The
+    # standardized .h5 (/sensor_check group, schema v2+) where the per-series
-    # little waveforms BW draws in its "Sensor Check" strip. Empty when absent.
+    # decoder stored them at ingest.  The little diagnostic waveforms BW draws
+    # in its "Sensor Check" strip.  Empty when absent (pre-v2 .h5, histogram,
+    # or 3-channel unit's MicL).
     sensor_check_waveforms: dict = field(default_factory=dict)
 
     # Record-type discriminator
@@ -294,22 +296,15 @@ def gather_report_data(
             rd.pretrig_samples  = ta.get("pretrig_samples")
             rd.t0_ms            = ta.get("t0_ms")
             rd.dt_ms            = ta.get("dt_ms")
+            # Sensor self-check traces — read from the standardized .h5 (schema
+            # v2+).  Device-agnostic: whichever decoder produced the event
+            # stored them at ingest, so SFM reads them here without knowing or
+            # caring about the source instrument series.  Empty on pre-v2 files
+            # (until backfilled) and on 3-channel / histogram events.
+            rd.sensor_check_waveforms = wf.get("sensor_check") or {}
         except Exception as exc:
             log.warning("gather_report_data: hdf5 read failed: %s", exc)
 
-    # ── Sensor self-check traces — decoded from the retained raw binary ──
-    # The .h5 holds only the main waveform; the sensor-check traces live in the
-    # binary's trailing block, so decode them straight from the kept BW file.
-    # Waveform events only (histograms have no sensor-check strip).
-    if not rd.is_histogram:
-        try:
-            from minimateplus.sensor_check import decode_sensor_check
-            bw_path, _a5 = store.paths_for(serial, filename)
-            if bw_path.exists():
-                rd.sensor_check_waveforms = decode_sensor_check(bw_path.read_bytes())
-        except Exception as exc:
-            log.warning("gather_report_data: sensor-check decode failed: %s", exc)
-
     # ── Histogram aggregation ──
     # Codec emits ~N per-block samples (typically 1/sec); BW reports
     # one bar per configured interval (1 min / 5 min / etc.).  When
@@ -856,9 +851,12 @@ def _draw_waveform_subplot(fig, gridspec_cell, rd: ReportData) -> None:
     order = ["MicL", "Long", "Vert", "Tran"]
     has_sc = bool(rd.sensor_check_waveforms)
     if has_sc:
-        # main lanes + a narrow sensor-check strip column on the right
+        # main lanes + a narrow sensor-check strip column, flush against the
-        inner = gridspec_cell.subgridspec(4, 2, width_ratios=[1.0, 0.15],
+        # main panel (BW shares the border — no gap), with the "0.0" baseline
-                                          wspace=0.04, hspace=0.0)
+        # labels moved to the right of the strip.  Proportions match BW's
+        # Event Report (main ~0.75 / strip ~0.10 of the panel width).
+        inner = gridspec_cell.subgridspec(4, 2, width_ratios=[1.0, 0.13],
+                                          wspace=0.0, hspace=0.0)
     else:
         inner = gridspec_cell.subgridspec(4, 1, hspace=0.0)
     sr = rd.sample_rate_sps or 1024
@@ -902,9 +900,12 @@ def _draw_waveform_subplot(fig, gridspec_cell, rd: ReportData) -> None:
         # Channel label on the LEFT (matches BW)
         ax.set_ylabel(ch, fontsize=8, rotation=0, ha="right", va="center",
                       color=_channel_axis_color(ch), weight="bold", labelpad=14)
-        # "0.0" on the RIGHT (BW convention)
+        # "0.0" baseline label on the RIGHT (BW convention).  With the sensor-
-        ax.text(1.005, 0.5, "0.0", transform=ax.transAxes,
+        # check strip attached, it goes to the right of the STRIP (drawn below);
-                fontsize=7, color="#555", va="center", ha="left")
+        # otherwise just outside the main lane.
+        if not has_sc:
+            ax.text(1.005, 0.5, "0.0", transform=ax.transAxes,
+                    fontsize=7, color="#555", va="center", ha="left")
 
         ax.grid(True, linestyle="--", linewidth=0.3, color="#bbb", alpha=0.6)
         # Vertical dashed trigger line at t=0
@@ -944,6 +945,9 @@ def _draw_waveform_subplot(fig, gridspec_cell, rd: ReportData) -> None:
             scx.set_xticks([]); scx.set_yticks([])
             for _s in scx.spines.values():
                 _s.set_linewidth(0.4); _s.set_color("#999")
+            # "0.0" baseline label to the RIGHT of the strip (BW convention)
+            scx.text(1.10, 0.5, "0.0", transform=scx.transAxes,
+                     fontsize=7, color="#555", va="center", ha="left")
 
     # Trigger triangle marker ▼ above the top channel at t=0
     top_ax = main_axes[0]  # MicL

sfm/waveform_store.py

@@ -662,6 +662,11 @@ class WaveformStore:
             ev.raw_samples = idf_samples
             n_samples = max((len(idf_samples.get(ch, [])) for ch in ("Tran", "Vert", "Long", "MicL")), default=0)
             ev.total_samples = ev.total_samples or n_samples
+            # Sensor self-check traces from the IDFW fixed header (waveform
+            # events only; {} on histograms / when absent).  Carried on the
+            # bridged Event so the .h5 writer persists them like series-3.
+            from micromate.sensor_check import decode_idf_sensor_check
+            ev.sensor_check = decode_idf_sensor_check(idf_bytes) or None
 
         # For IDFH histograms there are no per-sample waveform arrays — the
         # device stores one peak ADC count per interval per channel.  Synthesise

tests/test_event_hdf5_sensor_check.py

@@ -0,0 +1,71 @@
+"""The event .h5 carries the sensor self-check traces (schema v2).
+
+The sensor check is decoded by the per-series decoder and attached to the
+standardized Event, so the .h5 writer persists it device-agnostically and SFM
+reads it back without knowing which instrument produced it.  Old v1 files (no
+sensor_check group) must still read cleanly.
+"""
+import tempfile
+from pathlib import Path
+
+import numpy as np
+
+from minimateplus.models import Event
+from minimateplus.event_file_io import read_blastware_file
+from sfm import event_hdf5
+
+S3_FIX = Path(__file__).parent / "fixtures" / "fft-oracle-2026-09-14" / "N844LQHB.ZT0W"
+
+
+def _write(ev, **kw):
+    d = Path(tempfile.mkdtemp())
+    p = d / "e.h5"
+    event_hdf5.write_event_hdf5(p, ev, serial="BE12844", **kw)
+    return p
+
+
+def test_sensor_check_roundtrips_through_hdf5():
+    ev = Event(index=0)
+    ev.raw_samples = {"Tran": [1, 2, -3], "Vert": [0, 1], "Long": [2], "MicL": [5, -5]}
+    ev.sample_rate = 1024
+    sc = {"Tran": [0, -990, -500, -100], "Vert": [0, -980, -480],
+          "Long": [0, -986, -470], "MicL": [0, -1800, 1800, -1800]}
+    ev.sensor_check = sc
+
+    r = event_hdf5.read_event_hdf5(_write(ev))
+    assert r["schema_version"] == 2
+    assert set(r["sensor_check"]) == {"Tran", "Vert", "Long", "MicL"}
+    for ch, vals in sc.items():
+        assert r["sensor_check"][ch].tolist() == vals
+
+
+def test_plot_json_carries_sensor_check():
+    ev = Event(index=0)
+    ev.raw_samples = {"Tran": [1, 2, 3]}
+    ev.sample_rate = 1024
+    ev.sensor_check = {"Tran": [0, -990, -500], "Vert": [0, -980],
+                       "Long": [0, -986]}   # 3-channel: no MicL
+    pj = event_hdf5.plot_json_from_hdf5(_write(ev))
+    assert pj["sensor_check"] is not None
+    assert "MicL" not in pj["sensor_check"]
+    assert pj["sensor_check"]["Tran"] == [0, -990, -500]
+
+
+def test_event_without_sensor_check_still_reads_as_v2():
+    ev = Event(index=0)
+    ev.raw_samples = {"Tran": [1, 2, 3]}
+    ev.sample_rate = 1024
+    r = event_hdf5.read_event_hdf5(_write(ev))
+    assert r["schema_version"] == 2
+    assert r["sensor_check"] is None
+    assert event_hdf5.plot_json_from_hdf5(_write(ev))["sensor_check"] is None
+
+
+def test_series3_decode_populates_event_sensor_check():
+    # The real series-3 decoder attaches the traces to the Event, so the
+    # ingest/backfill .h5 write picks them up with no extra plumbing.
+    ev = read_blastware_file(S3_FIX)
+    assert ev.sensor_check is not None
+    assert set(ev.sensor_check) == {"Tran", "Vert", "Long", "MicL"}
+    tran = np.asarray(ev.sensor_check["Tran"], dtype=float)
+    assert tran.min() < -800   # the geophone ring-down deflection

tests/test_sensor_check_idf.py

@@ -0,0 +1,67 @@
+"""Series-4 (Thor / Micromate IDFW) sensor self-check waveform decode.
+
+Reverse-engineered 2026-09-15 against 4 UM (Thor) oracle events.  The IDFW
+binary carries the sensor self-check in its fixed-header region (before the
+waveform body) as up to four records tagged ``01 0e 3c/3d/3e/3f`` — the SAME
+channel ids as series-3 (Tran/Vert/Long/MicL).  Unlike series-3's delta-coded
+trailing block, series-4 stores each trace as a raw int16-BE array after an
+18-byte record header whose sample count is a 2-byte field at offset +8.
+
+Three-channel (mic-disabled) Thor units carry only 3c/3d/3e — no MicL record.
+
+Validated by shape (geophone ring-down / mic pulse train) and cross-event
+consistency, since there's no Thor Event-Report strip to exact-match against.
+"""
+from pathlib import Path
+
+import numpy as np
+
+from micromate.sensor_check import decode_idf_sensor_check
+
+FIXDIR = Path(__file__).parent / "fixtures" / "thor-idf-sc"
+EVENTS = sorted(p.name for p in FIXDIR.glob("*.IDFW"))
+
+
+def _decode(name):
+    return decode_idf_sensor_check((FIXDIR / name).read_bytes())
+
+
+def test_geo_channels_present_and_ringdown_shaped():
+    # Every IDFW event has the three geophone self-checks; each is a large
+    # one-sided deflection (~15000 raw counts) that rings back — the geophone's
+    # damped impulse response.
+    for name in EVENTS:
+        sc = _decode(name)
+        for ch in ("Tran", "Vert", "Long"):
+            assert ch in sc, f"{name} missing {ch}"
+            tr = np.asarray(sc[ch], dtype=float)
+            tr = tr - tr[:4].mean()          # reference to the pre-trigger baseline
+            assert 40 <= len(tr) <= 300, f"{name}:{ch} n={len(tr)}"
+            assert tr.min() < -8000, f"{name}:{ch} min {tr.min()}"
+            # deflects one way and rings back toward / past the baseline
+            assert tr.max() < abs(tr.min()), f"{name}:{ch} not one-sided"
+
+
+def test_mic_present_only_on_four_channel_units():
+    # UM11719 / UM12947 record a mic; UM13981 / UM20147 are 3-channel
+    # (mic-disabled) units and carry no MicL self-check.
+    got = {name: ("MicL" in _decode(name)) for name in EVENTS}
+    assert any(got.values()), "expected at least one 4-channel unit"
+    assert not all(got.values()), "expected at least one 3-channel unit"
+    for name, has_mic in got.items():
+        if has_mic:
+            tr = np.asarray(_decode(name)["MicL"], dtype=float)
+            tr = tr - tr[:4].mean()
+            # mic self-check is a bipolar pulse train — swings both ways, wide range
+            assert tr.max() > 5000 and tr.min() < -5000, f"{name} mic not bipolar"
+
+
+def test_channel_ids_and_order():
+    # ids decode to the canonical channel names, geo always in Tran/Vert/Long order
+    sc = _decode(EVENTS[0])
+    assert [c for c in ("Tran", "Vert", "Long") if c in sc] == ["Tran", "Vert", "Long"]
+
+
+def test_returns_empty_on_non_idf_input():
+    assert decode_idf_sensor_check(b"not an IDF file") == {}
+    assert decode_idf_sensor_check(b"") == {}