Release v0.31.0 — report parity + the inverted rescue (0.29.0 → 0.31.0) #40

Merged
serversdown merged 31 commits from dev into main 2026-09-18 16:46:45 -04:00
14 changed files with 341 additions and 38 deletions
Showing only changes of commit fa22bb9f59 - Show all commits
+34 -17
View File
@@ -46,18 +46,25 @@ All notable changes to seismo-relay are documented here.
reference PDF. A technical breakdown of the curve is in
`docs/ri8507_compliance_curve.md`.
- **Sensor self-check waveforms decoded and drawn (`minimateplus.sensor_check`).**
The "Sensor Check" traces Blastware shows to the right of the waveform panel
live in the series-3 binary's trailing block as four length-prefixed records
(`0x3c`–`0x3f`) using the same delta-block codec as the main waveform:
Tran/Vert/Long geophone ring-downs (the transducer's damped impulse response —
resonant frequency + overswing/damping) and a MicL pulse train (the mic's
known-signal gain check). `gather_report_data` decodes them from the retained
BW binary at report time; the report renders them as a strip flush against the
waveform panel plus the **Sensor Check → Frequency / Overswing Ratio** sub-rows
in the stats table. Verified against the reports on all 7 oracle events (mic
zero-crossing frequency = 20.1 Hz exact; geophone ring-downs consistent
~7.5 Hz with overswing ~3.5). Tests in `tests/test_sensor_check.py`.
- **Sensor self-check waveforms decoded and drawn — both series.** The little
"Sensor Check" traces (geophone ring-downs — the transducer's damped impulse
response — plus a MicL pulse train, the mic's known-signal gain check) are the
unit's proof its sensors were healthy when it recorded the event.
- **Series-3** (`minimateplus.sensor_check`): four records (`0x3c`–`0x3f`) in
the binary's trailing block, same delta-block codec as the main waveform.
Verified against all 7 BE12844 reports (mic zero-crossing = 20.1 Hz exact;
geophone ring-downs ~7.5 Hz, overswing ~3.5).
- **Series-4** (`micromate.sensor_check`): the same self-test in the Thor IDFW
fixed header — four `01 0e 3c/3d/3e/3f` records (same channel ids) storing
raw int16 traces; three-channel (mic-disabled) units carry only the three
geophones. Validated by shape + cross-event consistency.
- **Standardized into the `.h5`** (`/sensor_check`, schema v2): each series'
decoder attaches the traces to the event at decode, the writer persists
them, and `gather_report_data` reads them back — so SFM renders the strip
(flush against the waveform panel) plus the **Sensor Check → Frequency /
Overswing Ratio** sub-rows without knowing the source instrument.
- Tests: `tests/test_sensor_check.py`, `tests/test_sensor_check_idf.py`,
`tests/test_event_hdf5_sensor_check.py`.
- **Inspector tab in `seismo_lab.py` — annotated hex reader for series-3
binaries (`minimateplus/binary_annotate.py`).** Tiles a raw Blastware file
@@ -77,11 +84,21 @@ All notable changes to seismo-relay are documented here.
### Migration
**None.** Every change here is additive and reads from data already on disk —
the `.h5` samples and the retained raw BW binary. No `.h5`/DB change, no
schema change, no migration, no backfill, and **no `TOOL_VERSION` bump**: a
report regenerated for an existing event simply gains the new panels, and the
`ach_server` rescue flags don't touch the codec.
⚠ **The sensor-check needs a backfill.** Existing `.h5` files are schema v1
and carry no `/sensor_check` group, so their reports show no sensor-check strip
until regenerated. `TOOL_VERSION` is bumped to **0.31.0**, so the standard
backfill regenerates every event and picks up the traces with **no `--force`**:
`scripts/backfill_thor_events.py` for series-4 (it already owed a v0.30.0 Thor
backfill — this rides along) and the series-3 sidecar/shape backfill for
MiniMate events. Purely additive — no decoded value changes, and v1 `.h5`
files read fine until then (empty strip). DB backup first, as always.
⚠ Budget **~2 h on the NAS** — ~1.5 files/sec there versus ~85/sec on the dev
box (gzip-4 in `sfm/event_hdf5.py` against a Synology CPU).
Everything else in this release owes nothing: the FFT, the USBM compliance
chart and the `ach_server` rescue flags are additive and read data already on
disk — no schema change, no DB migration.
---
+89
View File
@@ -0,0 +1,89 @@
r"""Decode the Thor / Micromate (series-4) sensor self-check waveforms from an
IDFW event binary.
Reverse-engineered 2026-09-15 against 4 UM (Thor) oracle events. The IDFW
binary carries the sensor self-check in its fixed-header region (before the
waveform body), as up to four records tagged ``01 0e 3c/3d/3e/3f`` — the SAME
channel ids as the series-3 MiniMate Plus (Tran / Vert / Long / MicL), which is
the physical self-test:
* 3c / 3d / 3e = Tran / Vert / Long geophone ring-downs (a damped impulse
response — resonant frequency + damping).
* 3f = MicL pulse train (the mic's known-signal gain check). Absent
on three-channel (mic-disabled) units.
Record framing (per record)::
01 0e [id:1] [flags:3] [count:2 BE] [pad:10] [int16-BE samples × count]
\___ 18-byte header ___/
Unlike series-3's delta-coded trailing block, series-4 stores each trace as a
raw int16 big-endian array. ``count`` (the 2-byte field at header offset +8)
is the sample count; the record is padded to a fixed stride after that.
"""
from __future__ import annotations
import struct
from typing import Dict, List
# Record id → channel. Same ids/order as series-3 (minimateplus.sensor_check).
_ID_TO_CHANNEL = {0x3C: "Tran", 0x3D: "Vert", 0x3E: "Long", 0x3F: "MicL"}
_CHAIN_IDS = (0x3C, 0x3D, 0x3E, 0x3F)
_MARKER = b"\x01\x0e" # precedes the 1-byte channel id
_HEADER_LEN = 18 # bytes from the marker start to the first sample
_COUNT_OFF = 8 # 2-byte BE sample count, from the marker start
_MAX_COUNT = 4000 # sanity cap (traces are ~70-200 samples)
def _find_chain(raw: bytes):
"""Locate the sensor-check record chain. Returns a list of
``(offset, id, count)`` for the first run of markers whose ids run
3c, 3d, 3e[, 3f] in order, or ``[]``.
Records are padded to a fixed stride, so the next marker is not at
``header + count*2``; instead collect every ``01 0e [id]`` marker with a
sane count and take the first id-ordered run. Validating the id sequence
(not a lone ``01 0e 3c``) keeps a stray marker in the waveform body from
matching — the real chain sits in the fixed header, ahead of the body.
"""
n = len(raw)
markers = []
for p in range(n - _HEADER_LEN):
if raw[p:p + 2] == _MARKER and raw[p + 2] in _ID_TO_CHANNEL:
count = int.from_bytes(raw[p + _COUNT_OFF:p + _COUNT_OFF + 2], "big")
if 0 < count <= _MAX_COUNT:
markers.append((p, raw[p + 2], count))
for i, (off, rid, _c) in enumerate(markers):
if rid != 0x3C:
continue
run = [markers[i]]
for m in markers[i + 1:]:
if len(run) < len(_CHAIN_IDS) and m[1] == _CHAIN_IDS[len(run)]:
run.append(m)
else:
break
if len(run) >= 3: # 3-channel (mic-disabled) units are valid
return run
return []
def decode_idf_sensor_check(raw: bytes) -> Dict[str, List[int]]:
"""Decode the sensor self-check traces from a Thor/Micromate IDFW binary.
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}`` in
raw int16 ADC counts (MicL omitted on 3-channel units), or ``{}`` if the
binary carries no sensor-check chain (a non-IDF file, or an IDFH histogram).
"""
chain = _find_chain(raw)
if not chain:
return {}
out: Dict[str, List[int]] = {}
for off, rid, count in chain:
start = off + _HEADER_LEN
blob = raw[start:start + count * 2]
if len(blob) < count * 2:
continue
out[_ID_TO_CHANNEL[rid]] = list(struct.unpack(">%dh" % count, blob))
return out
+6 -1
View File
@@ -50,7 +50,7 @@ SIDECAR_KIND = "sfm.event"
# bumped without a `pip install` re-run — leading to confusing stale
# version stamps in sidecars. Bump this constant and CHANGELOG.md
# together at release time.
TOOL_VERSION = "0.30.0"
TOOL_VERSION = "0.31.0" # +/sensor_check group (schema v2); gates the backfill regen
try:
# Best-effort: prefer the installed metadata when it's NEWER than the
@@ -960,6 +960,11 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
project=project, client=client, operator=user, sensor_location=seisloc,
)
ev.raw_samples = samples
# Sensor self-check traces from the binary's trailing block (waveform
# events only; returns {} for histograms / when absent). Carried on the
# Event so the .h5 writer persists them device-agnostically.
from minimateplus.sensor_check import decode_sensor_check
ev.sensor_check = decode_sensor_check(raw) or None
# Only compute peaks from samples when we actually have samples.
# For events the codec couldn't decode (histogram-mode bodies, until
# the §7.6.2 histogram codec is wired in), samples is an empty dict
+9
View File
@@ -544,6 +544,15 @@ class Event:
pretrig_samples: Optional[int] = None # from STRT record: pre-trigger sample count
rectime_seconds: Optional[int] = None # from STRT record: record duration (seconds)
# Sensor self-check traces keyed by channel label — the short diagnostic
# waveforms the unit records when it pulses each sensor before monitoring
# (geophone ring-downs + a mic pulse train). Decoded from the binary by
# the per-series decoder (minimateplus.sensor_check / micromate.sensor_check)
# and carried here so the .h5 writer can persist them device-agnostically.
# Raw ADC counts; the source series' scale differs but the trace is a
# shape diagnostic (rendered fit-to-box). None when absent.
sensor_check: Optional[dict] = None # {"Tran": [...], ..., "MicL": [...]}
# ── Debug / introspection ─────────────────────────────────────────────────
# Raw 210-byte waveform record bytes, set when debug mode is active.
# Exposed by the SFM server via ?debug=true so field layouts can be verified.
+5
View File
@@ -305,6 +305,11 @@ def main(argv=None) -> int:
default=0,
)
ev.total_samples = ev.total_samples or n_samp
# Sensor self-check traces from the IDFW fixed
# header, so regenerated .h5 files gain the v2
# /sensor_check group (mirrors save_imported_idf).
from micromate.sensor_check import decode_idf_sensor_check
ev.sensor_check = decode_idf_sensor_check(binary_bytes) or None
event_hdf5.write_event_hdf5(
hdf5_path, ev,
+44 -4
View File
@@ -12,8 +12,11 @@ Layout written to `<filename>.h5`:
├─ samples_int16/ (optional)
│ ├─ Tran (int16, raw ADC counts) shape: (N,)
│ └─ ... per channel (only when present in the source)
├─ sensor_check/ (optional, schema v2+)
│ ├─ Tran (int32, raw counts) shape: (M,) M ≪ N
│ └─ ... per channel present in the source (MicL absent on 3-channel units)
└─ root attrs (event metadata):
schema_version int = 1
schema_version int = 2
kind str = "sfm.event.hdf5"
serial str
waveform_key str (8-hex)
@@ -64,7 +67,7 @@ from minimateplus.models import Event
log = logging.getLogger(__name__)
SCHEMA_VERSION = 1
SCHEMA_VERSION = 2 # v2 adds the optional /sensor_check group
HDF5_KIND = "sfm.event.hdf5"
# Geophone full-scale velocity per range (in/s). Confirmed in CLAUDE.md
@@ -270,6 +273,22 @@ def write_event_hdf5(
)
igrp.attrs["mic_psi_per_count"] = float(mic_factor)
# /sensor_check — optional short diagnostic self-check traces (schema
# v2+). Raw ADC counts (a shape diagnostic; the per-series count scale
# differs, and the renderer fits each trace to its box). Only channels
# the decoder found are written — 3-channel units carry no MicL.
sc = event.sensor_check or {}
if sc:
scgrp = f.create_group("sensor_check")
for ch in ("Tran", "Vert", "Long", "MicL"):
vals = sc.get(ch)
if vals:
scgrp.create_dataset(
ch, data=np.asarray(vals, dtype=np.int32),
compression="gzip", compression_opts=4, shuffle=True,
)
scgrp.attrs["units"] = "raw_counts"
import os
os.replace(tmp, path)
@@ -334,6 +353,16 @@ def read_event_hdf5(path: Union[str, Path]) -> dict:
if mic_attr is not None:
mic_psi = float(mic_attr)
# /sensor_check — optional (schema v2+); absent on older files.
sensor_check = None
scgrp = f.get("sensor_check")
if scgrp is not None:
sensor_check = {}
for ch in ("Tran", "Vert", "Long", "MicL"):
ds = scgrp.get(ch)
if ds is not None:
sensor_check[ch] = np.asarray(ds[()])
return {
"schema_version": sv,
"kind": attrs.get("kind"),
@@ -341,6 +370,7 @@ def read_event_hdf5(path: Union[str, Path]) -> dict:
"samples": samples,
"samples_int16": samples_int16,
"mic_psi_per_count": mic_psi,
"sensor_check": sensor_check,
}
@@ -431,11 +461,16 @@ def plot_json_from_hdf5(
event_id: Optional[str] = None,
index: Optional[int] = None,
) -> dict:
"""Build a `sfm.plot.v1` JSON dict from a stored .h5 file."""
"""Build a `sfm.plot.v1` JSON dict from a stored .h5 file.
The dict also carries a top-level ``sensor_check`` key (the raw self-check
traces as ``{ch: [int]}``, or None) beyond the plot schema, so report
generation can read the traces from the same single .h5 load.
"""
data = read_event_hdf5(path)
a = data["attrs"]
s = data["samples"]
return _build_plot_dict(
out = _build_plot_dict(
n_samples=len(s["Tran"]) if "Tran" in s else 0,
sample_rate=int(a.get("sample_rate", 1024) or 1024),
pretrig_samples=int(a.get("pretrig_samples", 0) or 0),
@@ -463,6 +498,11 @@ def plot_json_from_hdf5(
event_id=event_id,
index=index,
)
scd = data.get("sensor_check")
out["sensor_check"] = (
{ch: v.tolist() for ch, v in scd.items()} if scd else None
)
return out
def _build_plot_dict(
+11 -16
View File
@@ -121,9 +121,11 @@ class ReportData:
t0_ms: Optional[float] = None
dt_ms: Optional[float] = None
# Sensor self-check traces — {ch: [samples]} in raw decode units, decoded
# from the binary's trailing block (see minimateplus.sensor_check). The
# little waveforms BW draws in its "Sensor Check" strip. Empty when absent.
# Sensor self-check traces — {ch: [samples]} in raw counts, read from the
# standardized .h5 (/sensor_check group, schema v2+) where the per-series
# decoder stored them at ingest. The little diagnostic waveforms BW draws
# in its "Sensor Check" strip. Empty when absent (pre-v2 .h5, histogram,
# or 3-channel unit's MicL).
sensor_check_waveforms: dict = field(default_factory=dict)
# Record-type discriminator
@@ -294,22 +296,15 @@ def gather_report_data(
rd.pretrig_samples = ta.get("pretrig_samples")
rd.t0_ms = ta.get("t0_ms")
rd.dt_ms = ta.get("dt_ms")
# Sensor self-check traces — read from the standardized .h5 (schema
# v2+). Device-agnostic: whichever decoder produced the event
# stored them at ingest, so SFM reads them here without knowing or
# caring about the source instrument series. Empty on pre-v2 files
# (until backfilled) and on 3-channel / histogram events.
rd.sensor_check_waveforms = wf.get("sensor_check") or {}
except Exception as exc:
log.warning("gather_report_data: hdf5 read failed: %s", exc)
# ── Sensor self-check traces — decoded from the retained raw binary ──
# The .h5 holds only the main waveform; the sensor-check traces live in the
# binary's trailing block, so decode them straight from the kept BW file.
# Waveform events only (histograms have no sensor-check strip).
if not rd.is_histogram:
try:
from minimateplus.sensor_check import decode_sensor_check
bw_path, _a5 = store.paths_for(serial, filename)
if bw_path.exists():
rd.sensor_check_waveforms = decode_sensor_check(bw_path.read_bytes())
except Exception as exc:
log.warning("gather_report_data: sensor-check decode failed: %s", exc)
# ── Histogram aggregation ──
# Codec emits ~N per-block samples (typically 1/sec); BW reports
# one bar per configured interval (1 min / 5 min / etc.). When
+5
View File
@@ -662,6 +662,11 @@ class WaveformStore:
ev.raw_samples = idf_samples
n_samples = max((len(idf_samples.get(ch, [])) for ch in ("Tran", "Vert", "Long", "MicL")), default=0)
ev.total_samples = ev.total_samples or n_samples
# Sensor self-check traces from the IDFW fixed header (waveform
# events only; {} on histograms / when absent). Carried on the
# bridged Event so the .h5 writer persists them like series-3.
from micromate.sensor_check import decode_idf_sensor_check
ev.sensor_check = decode_idf_sensor_check(idf_bytes) or None
# For IDFH histograms there are no per-sample waveform arrays — the
# device stores one peak ADC count per interval per channel. Synthesise
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+71
View File
@@ -0,0 +1,71 @@
"""The event .h5 carries the sensor self-check traces (schema v2).
The sensor check is decoded by the per-series decoder and attached to the
standardized Event, so the .h5 writer persists it device-agnostically and SFM
reads it back without knowing which instrument produced it. Old v1 files (no
sensor_check group) must still read cleanly.
"""
import tempfile
from pathlib import Path
import numpy as np
from minimateplus.models import Event
from minimateplus.event_file_io import read_blastware_file
from sfm import event_hdf5
S3_FIX = Path(__file__).parent / "fixtures" / "fft-oracle-2026-09-14" / "N844LQHB.ZT0W"
def _write(ev, **kw):
d = Path(tempfile.mkdtemp())
p = d / "e.h5"
event_hdf5.write_event_hdf5(p, ev, serial="BE12844", **kw)
return p
def test_sensor_check_roundtrips_through_hdf5():
ev = Event(index=0)
ev.raw_samples = {"Tran": [1, 2, -3], "Vert": [0, 1], "Long": [2], "MicL": [5, -5]}
ev.sample_rate = 1024
sc = {"Tran": [0, -990, -500, -100], "Vert": [0, -980, -480],
"Long": [0, -986, -470], "MicL": [0, -1800, 1800, -1800]}
ev.sensor_check = sc
r = event_hdf5.read_event_hdf5(_write(ev))
assert r["schema_version"] == 2
assert set(r["sensor_check"]) == {"Tran", "Vert", "Long", "MicL"}
for ch, vals in sc.items():
assert r["sensor_check"][ch].tolist() == vals
def test_plot_json_carries_sensor_check():
ev = Event(index=0)
ev.raw_samples = {"Tran": [1, 2, 3]}
ev.sample_rate = 1024
ev.sensor_check = {"Tran": [0, -990, -500], "Vert": [0, -980],
"Long": [0, -986]} # 3-channel: no MicL
pj = event_hdf5.plot_json_from_hdf5(_write(ev))
assert pj["sensor_check"] is not None
assert "MicL" not in pj["sensor_check"]
assert pj["sensor_check"]["Tran"] == [0, -990, -500]
def test_event_without_sensor_check_still_reads_as_v2():
ev = Event(index=0)
ev.raw_samples = {"Tran": [1, 2, 3]}
ev.sample_rate = 1024
r = event_hdf5.read_event_hdf5(_write(ev))
assert r["schema_version"] == 2
assert r["sensor_check"] is None
assert event_hdf5.plot_json_from_hdf5(_write(ev))["sensor_check"] is None
def test_series3_decode_populates_event_sensor_check():
# The real series-3 decoder attaches the traces to the Event, so the
# ingest/backfill .h5 write picks them up with no extra plumbing.
ev = read_blastware_file(S3_FIX)
assert ev.sensor_check is not None
assert set(ev.sensor_check) == {"Tran", "Vert", "Long", "MicL"}
tran = np.asarray(ev.sensor_check["Tran"], dtype=float)
assert tran.min() < -800 # the geophone ring-down deflection
+67
View File
@@ -0,0 +1,67 @@
"""Series-4 (Thor / Micromate IDFW) sensor self-check waveform decode.
Reverse-engineered 2026-09-15 against 4 UM (Thor) oracle events. The IDFW
binary carries the sensor self-check in its fixed-header region (before the
waveform body) as up to four records tagged ``01 0e 3c/3d/3e/3f`` — the SAME
channel ids as series-3 (Tran/Vert/Long/MicL). Unlike series-3's delta-coded
trailing block, series-4 stores each trace as a raw int16-BE array after an
18-byte record header whose sample count is a 2-byte field at offset +8.
Three-channel (mic-disabled) Thor units carry only 3c/3d/3e — no MicL record.
Validated by shape (geophone ring-down / mic pulse train) and cross-event
consistency, since there's no Thor Event-Report strip to exact-match against.
"""
from pathlib import Path
import numpy as np
from micromate.sensor_check import decode_idf_sensor_check
FIXDIR = Path(__file__).parent / "fixtures" / "thor-idf-sc"
EVENTS = sorted(p.name for p in FIXDIR.glob("*.IDFW"))
def _decode(name):
return decode_idf_sensor_check((FIXDIR / name).read_bytes())
def test_geo_channels_present_and_ringdown_shaped():
# Every IDFW event has the three geophone self-checks; each is a large
# one-sided deflection (~15000 raw counts) that rings back — the geophone's
# damped impulse response.
for name in EVENTS:
sc = _decode(name)
for ch in ("Tran", "Vert", "Long"):
assert ch in sc, f"{name} missing {ch}"
tr = np.asarray(sc[ch], dtype=float)
tr = tr - tr[:4].mean() # reference to the pre-trigger baseline
assert 40 <= len(tr) <= 300, f"{name}:{ch} n={len(tr)}"
assert tr.min() < -8000, f"{name}:{ch} min {tr.min()}"
# deflects one way and rings back toward / past the baseline
assert tr.max() < abs(tr.min()), f"{name}:{ch} not one-sided"
def test_mic_present_only_on_four_channel_units():
# UM11719 / UM12947 record a mic; UM13981 / UM20147 are 3-channel
# (mic-disabled) units and carry no MicL self-check.
got = {name: ("MicL" in _decode(name)) for name in EVENTS}
assert any(got.values()), "expected at least one 4-channel unit"
assert not all(got.values()), "expected at least one 3-channel unit"
for name, has_mic in got.items():
if has_mic:
tr = np.asarray(_decode(name)["MicL"], dtype=float)
tr = tr - tr[:4].mean()
# mic self-check is a bipolar pulse train — swings both ways, wide range
assert tr.max() > 5000 and tr.min() < -5000, f"{name} mic not bipolar"
def test_channel_ids_and_order():
# ids decode to the canonical channel names, geo always in Tran/Vert/Long order
sc = _decode(EVENTS[0])
assert [c for c in ("Tran", "Vert", "Long") if c in sc] == ["Tran", "Vert", "Long"]
def test_returns_empty_on_non_idf_input():
assert decode_idf_sensor_check(b"not an IDF file") == {}
assert decode_idf_sensor_check(b"") == {}