7 Commits
9 changed files with 16 additions and 337 deletions
+3 -35
View File
@@ -8,30 +8,6 @@ All notable changes to seismo-relay are documented here.
---
## v0.28.0 — 2026-09-02
**Offset (DC-baseline) false-trigger detector.** Productionizes the validated
pre-trigger detector: a geophone event whose baseline sits off zero and stays
flat across the record (sensor bumped / settled / drifted) is now flagged and
surfaced in Terra-View as an `offset` false-trigger reason — catching offsets the
crest/near-peak spike rule misses (an offset is low-crest and flat).
### Added
- `shape_metrics.offset_from_samples` / `offset_from_h5`: per geophone channel,
`|median(pre-trigger)| ≥ 0.025 in/s` AND `pre/mid/end spread ≤ 0.02` → offset;
the consistency test rejects transients (a real event moves one third). Reads
the `.h5` samples + the `pretrig_samples` attr, range-aware via the in/s float
samples. Constants `OFFSET_FLOOR` / `OFFSET_MAX_SPREAD` are tunable.
- `events.shape_offset` / `shape_offset_axis` / `shape_offset_pre` /
`shape_offset_spread` columns (auto-migrated: `_SCHEMA` + the `_migrate`
ADD COLUMN loop), computed at all three ingest paths and by
`backfill_event_shape.py`, exposed via `/db/events`.
Requires the shape/offset backfill on the prod store to populate existing events:
`python scripts/backfill_event_shape.py --db-path … --store-root …`.
---
## v0.27.0 — 2026-08-28
**Per-sample decoder verification at scale, plus the offset investigation.**
@@ -63,17 +39,9 @@ carried, and it found one real codec bug (below).
walk double-counts every binary — 127,035 paths are 63,535 distinct files. The
ASCII exports are *not* mirrored, so the 14,340 pair count is already distinct.)
**No prod backfill is required for this.** Verified after the fact: all four
recovered files are archive-only — none exists in the production store or the
events DB — and re-running stride detection over the production store's
**10,215** histogram binaries shows **0 files whose decode changes**. The fix
matters for future ingests of sub-minute histograms with a partial final block,
not for anything already stored.
(`TOOL_VERSION` moves with the release, so whenever a backfill *is* next run for
some other reason it will regenerate the whole store rather than skipping. That
is harmless — the output is byte-identical for every currently-stored file — but
it means the run takes its full ~2 hours on the NAS.)
⚠ Prod stores hold `.h5` files generated before this fix. Those 4 events stay
empty until `backfill_sidecars.py` is re-run — not worth a two-hour prod backfill
on its own; fold it into the next one.
- **Histogram/waveform twin matching is now interval-based** (`find_twins`). A real
trigger is recorded twice — as a triggered waveform (stamped at the trigger instant)
+2 -7
View File
@@ -4,10 +4,6 @@ Ground-up Python replacement for **Blastware**, Instantel's Windows-only softwar
managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
(Sierra Wireless RV50 / RV55). Current version: **v0.27.0**.
Stack-level context — which repo owns what, and how the three project versions
pair — lives in `../terra-view/docs/tmi-stack.md`, which is also loaded as
`~/CLAUDE.md`.
---
## Where things stand (updated 2026-08-28)
@@ -37,9 +33,8 @@ Read this first when picking the project back up.
(it gates regeneration). ⚠ On the office NAS this takes **~2 hours**
(~1.5 files/sec vs 85/sec on the dev box — gzip-4 in `sfm/event_hdf5.py`
against a Synology CPU). Budget it up front.
**v0.27.0 does NOT owe prod a backfill** — verified: the partial-final-block
fix changes 0 of the 10,215 histograms in the prod store (the 4 recovered
files are archive-only and were never ingested).
**v0.27.0 owes prod a backfill:** the partial-final-block fix recovers 4
histograms that are still empty in the store.
- **The "offset" hardware fault has its own journal** --
`docs/offset_investigation.md`. **5 of 45 units (11%)**, and the fault is
**persistent** — it stays until the geophone is serviced. Detect it with
+1 -1
View File
@@ -50,7 +50,7 @@ SIDECAR_KIND = "sfm.event"
# bumped without a `pip install` re-run — leading to confusing stale
# version stamps in sidecars. Bump this constant and CHANGELOG.md
# together at release time.
TOOL_VERSION = "0.28.0"
TOOL_VERSION = "0.27.0"
try:
# Best-effort: prefer the installed metadata when it's NEWER than the
+6 -17
View File
@@ -1,12 +1,12 @@
#!/usr/bin/env python3
"""Backfill events.shape_* and shape_offset_* from each event's .h5 samples. Idempotent."""
"""Backfill events.shape_* from each event's .h5 waveform samples. Idempotent."""
from __future__ import annotations
import argparse, logging, sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from sfm.database import SeismoDb
from sfm.waveform_store import WaveformStore
from sfm.shape_metrics import shape_from_h5, offset_from_h5
from sfm.shape_metrics import shape_from_h5
log = logging.getLogger("backfill_event_shape")
@@ -21,7 +21,6 @@ def backfill_shape(db: SeismoDb, store: WaveformStore, *, dry_run: bool = False)
if not h5_path.exists():
counts["skipped_no_h5"] += 1; continue
shape = shape_from_h5(h5_path)
offset = offset_from_h5(h5_path)
if shape is None:
# The .h5 can no longer yield a shape (fewer than 2 samples, or a
# flat trace). Clear any previously stored value rather than
@@ -29,32 +28,22 @@ def backfill_shape(db: SeismoDb, store: WaveformStore, *, dry_run: bool = False)
# from and silently feeds the false-trigger detector. Seen after
# a decoder fix shrinks an event: 493 rows in the prod snapshot
# were carrying metrics from a superseded decode (2026-08-25).
if (row.get("shape_crest_factor") is not None
or row.get("shape_offset") is not None):
if row.get("shape_crest_factor") is not None:
if not dry_run:
with db._connect() as conn:
conn.execute(
"UPDATE events SET shape_crest_factor=NULL, "
"shape_near_peak_count=NULL, shape_sample_count=NULL, "
"shape_axis=NULL, shape_offset=NULL, shape_offset_axis=NULL, "
"shape_offset_pre=NULL, shape_offset_spread=NULL WHERE id=?",
(row["id"],))
"shape_axis=NULL WHERE id=?", (row["id"],))
counts["cleared_stale"] += 1
counts["skipped_no_samples"] += 1; continue
if not dry_run:
with db._connect() as conn:
conn.execute(
"UPDATE events SET shape_crest_factor=?, shape_near_peak_count=?, "
"shape_sample_count=?, shape_axis=?, shape_offset=?, "
"shape_offset_axis=?, shape_offset_pre=?, shape_offset_spread=? "
"WHERE id=?",
"shape_sample_count=?, shape_axis=? WHERE id=?",
(shape["crest_factor"], shape["near_peak_count"],
shape["sample_count"], shape["axis"],
(1 if offset["offset"] else 0) if offset else None,
offset["axis"] if offset else None,
offset["pre"] if offset else None,
offset["spread"] if offset else None,
row["id"]))
shape["sample_count"], shape["axis"], row["id"]))
counts["updated"] += 1
log.info("backfill_shape: %s", counts)
return counts
+3 -25
View File
@@ -99,10 +99,6 @@ CREATE TABLE IF NOT EXISTS events (
shape_near_peak_count INTEGER, -- samples >= 0.5 * peak (FT: few; real: many)
shape_sample_count INTEGER, -- total samples (to normalize near_peak_count)
shape_axis TEXT, -- geophone channel measured ("Tran"/"Vert"/"Long")
shape_offset INTEGER, -- 1 = DC-offset false trigger (pre-trigger baseline off zero + flat). Meaningful for waveforms only.
shape_offset_axis TEXT, -- geo channel the offset was measured on
shape_offset_pre REAL, -- pre-trigger baseline median (in/s)
shape_offset_spread REAL, -- max(pre,mid,end) - min(...) in in/s; small = constant/DC
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
UNIQUE(serial, timestamp)
);
@@ -229,10 +225,6 @@ class SeismoDb:
("shape_near_peak_count", "INTEGER"),
("shape_sample_count", "INTEGER"),
("shape_axis", "TEXT"),
("shape_offset", "INTEGER"),
("shape_offset_axis", "TEXT"),
("shape_offset_pre", "REAL"),
("shape_offset_spread", "REAL"),
("reviewed_real", "INTEGER NOT NULL DEFAULT 0"),
):
if col not in existing_cols:
@@ -438,11 +430,9 @@ class SeismoDb:
tran_zc_above_range, vert_zc_above_range,
long_zc_above_range, mic_zc_above_range,
shape_crest_factor, shape_near_peak_count,
shape_sample_count, shape_axis,
shape_offset, shape_offset_axis,
shape_offset_pre, shape_offset_spread)
shape_sample_count, shape_axis)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
self._new_id(), serial, key, session_id, ts,
@@ -474,10 +464,6 @@ class SeismoDb:
rec.get("shape_near_peak_count"),
rec.get("shape_sample_count"),
rec.get("shape_axis"),
rec.get("shape_offset"),
rec.get("shape_offset_axis"),
rec.get("shape_offset_pre"),
rec.get("shape_offset_spread"),
),
)
inserted += 1
@@ -531,11 +517,7 @@ class SeismoDb:
shape_crest_factor = COALESCE(?, shape_crest_factor),
shape_near_peak_count = COALESCE(?, shape_near_peak_count),
shape_sample_count = COALESCE(?, shape_sample_count),
shape_axis = COALESCE(?, shape_axis),
shape_offset = COALESCE(?, shape_offset),
shape_offset_axis = COALESCE(?, shape_offset_axis),
shape_offset_pre = COALESCE(?, shape_offset_pre),
shape_offset_spread = COALESCE(?, shape_offset_spread)
shape_axis = COALESCE(?, shape_axis)
WHERE serial = ? AND timestamp = ?
""",
(
@@ -567,10 +549,6 @@ class SeismoDb:
rec.get("shape_near_peak_count") if rec else None,
rec.get("shape_sample_count") if rec else None,
rec.get("shape_axis") if rec else None,
rec.get("shape_offset") if rec else None,
rec.get("shape_offset_axis") if rec else None,
rec.get("shape_offset_pre") if rec else None,
rec.get("shape_offset_spread") if rec else None,
serial,
ts,
),
-69
View File
@@ -47,60 +47,6 @@ def shape_from_samples(chans: dict) -> dict | None:
return s
# ── Offset (DC-baseline) detection ────────────────────────────────────────────
# A DC offset is a false trigger where the geophone baseline sits at a constant
# non-zero floor (sensor bumped / settled / drifted) instead of oscillating
# around zero. Brian's method (validated in scratch/offset_scan3.py): the
# pre-trigger window is definitionally quiet, so a true offset shows |pre| off
# zero AND stays flat across the record (pre ≈ mid ≈ end). A transient moves one
# third relative to the others and is rejected by the spread test.
# Thresholds are in in/s (the .h5 samples are already range-scaled); validated at
# Normal range (10 in/s) — the only range in the fleet.
OFFSET_FLOOR = 0.025 # |pre| at/above this reads as an off-zero baseline (5 A/D counts)
OFFSET_MAX_SPREAD = 0.02 # max(pre,mid,end) - min(...) at/below this reads as flat/constant
def _channel_offset(x, pretrig_n):
"""Return (pre, spread, is_offset) for one channel, or None if unusable."""
x = np.asarray(x, dtype=float)
n = x.size
if n < 3:
return None
t = n // 3
pre = x[:pretrig_n] if (pretrig_n and 0 < pretrig_n < n) else x[:t]
mid, end = x[t:2 * t], x[2 * t:]
if pre.size == 0 or mid.size == 0 or end.size == 0:
return None
vals = [float(np.median(seg)) for seg in (pre, mid, end)]
spread = max(vals) - min(vals)
is_offset = abs(vals[0]) >= OFFSET_FLOOR and spread <= OFFSET_MAX_SPREAD
return vals[0], spread, is_offset
def offset_from_samples(chans: dict, pretrig_n) -> dict | None:
"""Detect a DC-offset false trigger across the geophone channels.
An event is offset if ANY geo channel's pre-trigger baseline is off zero and
flat across the record. Reports the tripping axis (or, if none trips, the
most-offset-like axis) with its ``pre``/``spread`` for transparency + tuning.
Returns None when no geo channel is usable.
"""
results = []
for ax in _GEO_CHANNELS:
x = chans.get(ax)
if x is None:
continue
r = _channel_offset(x, pretrig_n)
if r is not None:
results.append((ax, r[0], r[1], r[2]))
if not results:
return None
offenders = [r for r in results if r[3]]
ax, pre, spread, _ = max(offenders or results, key=lambda r: abs(r[1]))
return {"offset": bool(offenders), "axis": ax,
"pre": round(pre, 6), "spread": round(spread, 6)}
def shape_from_h5(path) -> dict | None:
import h5py
try:
@@ -110,18 +56,3 @@ def shape_from_h5(path) -> dict | None:
except Exception:
return None
return shape_from_samples(chans)
def offset_from_h5(path) -> dict | None:
"""offset_from_samples fed from an event's .h5 (float32 in/s geo samples +
the pretrig_samples attribute)."""
import h5py
try:
with h5py.File(path, "r") as f:
chans = {ax: f[f"samples/{ax}"][:] for ax in _GEO_CHANNELS
if f"samples/{ax}" in f}
pretrig_n = f.attrs.get("pretrig_samples")
except Exception:
return None
pretrig_n = int(pretrig_n) if pretrig_n is not None else 0
return offset_from_samples(chans, pretrig_n)
+1 -25
View File
@@ -41,7 +41,7 @@ from minimateplus.blastware_file import blastware_filename, write_blastware_file
from minimateplus.framing import S3Frame
from minimateplus.models import Event
from sfm import event_hdf5
from sfm.shape_metrics import shape_from_h5, offset_from_h5
from sfm.shape_metrics import shape_from_h5
log = logging.getLogger("sfm.waveform_store")
@@ -270,13 +270,6 @@ class WaveformStore:
"shape_sample_count": _shape["sample_count"],
"shape_axis": _shape["axis"],
} if _shape else {}
_offset = offset_from_h5(hdf5_path) if hdf5_filename else None
_offset_rec = {
"shape_offset": 1 if _offset["offset"] else 0,
"shape_offset_axis": _offset["axis"],
"shape_offset_pre": _offset["pre"],
"shape_offset_spread": _offset["spread"],
} if _offset else {}
return {
"filename": filename,
"filesize": filesize,
@@ -285,7 +278,6 @@ class WaveformStore:
"hdf5_filename": hdf5_filename,
"sidecar_filename": sidecar_path.name,
**_shape_rec,
**_offset_rec,
}
def save_imported_bw(
@@ -469,13 +461,6 @@ class WaveformStore:
"shape_sample_count": _shape["sample_count"],
"shape_axis": _shape["axis"],
} if _shape else {}
_offset = offset_from_h5(hdf5_path) if hdf5_filename else None
_offset_rec = {
"shape_offset": 1 if _offset["offset"] else 0,
"shape_offset_axis": _offset["axis"],
"shape_offset_pre": _offset["pre"],
"shape_offset_spread": _offset["spread"],
} if _offset else {}
return ev, {
"filename": filename,
"filesize": filesize,
@@ -485,7 +470,6 @@ class WaveformStore:
"sidecar_filename": sidecar_path.name,
"serial": serial,
**_shape_rec,
**_offset_rec,
}
def save_imported_idf(
@@ -767,13 +751,6 @@ class WaveformStore:
"shape_sample_count": _shape["sample_count"],
"shape_axis": _shape["axis"],
} if _shape else {}
_offset = offset_from_h5(hdf5_path) if hdf5_filename else None
_offset_rec = {
"shape_offset": 1 if _offset["offset"] else 0,
"shape_offset_axis": _offset["axis"],
"shape_offset_pre": _offset["pre"],
"shape_offset_spread": _offset["spread"],
} if _offset else {}
return ev, {
"filename": filename,
"filesize": filesize,
@@ -783,7 +760,6 @@ class WaveformStore:
"sidecar_filename": sidecar_path.name,
"serial": serial,
**_shape_rec,
**_offset_rec,
}
def load_a5(self, serial: str, filename: str) -> Optional[list[S3Frame]]:
-94
View File
@@ -1,94 +0,0 @@
import numpy as np
import h5py
from sfm.shape_metrics import offset_from_samples, offset_from_h5
def test_flags_constant_dc_floor():
# A geophone channel sitting at a constant +0.05 in/s across the whole record
# is a DC offset: baseline off zero AND flat across pre/mid/end thirds.
n = 300
chans = {"Tran": np.full(n, 0.05), "Vert": np.zeros(n), "Long": np.zeros(n)}
r = offset_from_samples(chans, pretrig_n=50)
assert r["offset"] is True
assert r["axis"] == "Tran"
assert abs(r["pre"] - 0.05) < 1e-6
assert r["spread"] < 0.02
def test_transient_rejected_by_spread():
# Off-zero pre-trigger but the baseline SETTLES back over the record — a
# transient, not a constant offset. The spread test must reject it.
x = np.concatenate([np.full(100, 0.05), np.full(100, 0.025), np.zeros(100)])
chans = {"Tran": x, "Vert": np.zeros(300), "Long": np.zeros(300)}
r = offset_from_samples(chans, pretrig_n=100)
assert r["offset"] is False
def test_clean_oscillation_not_offset():
t = np.arange(300)
x = 0.4 * np.sin(2 * np.pi * t / 20) # oscillates around zero — baseline IS zero
chans = {"Tran": x, "Vert": np.zeros(300), "Long": np.zeros(300)}
r = offset_from_samples(chans, pretrig_n=50)
assert r["offset"] is False
def test_below_floor_not_offset_but_reports_pre():
# A flat baseline below the floor is not an offset; still report the axis/pre
# for tuning transparency.
n = 300
chans = {"Tran": np.full(n, 0.01), "Vert": np.zeros(n), "Long": np.zeros(n)}
r = offset_from_samples(chans, pretrig_n=50)
assert r["offset"] is False
assert r["axis"] == "Tran"
assert abs(r["pre"] - 0.01) < 1e-6
def test_none_when_no_geo_channels():
assert offset_from_samples({"MicL": np.full(300, 0.05)}, pretrig_n=50) is None
def test_pretrig_fallback_when_invalid():
# pretrig_n of 0 (missing/unusable) falls back to the first third.
n = 300
chans = {"Tran": np.full(n, 0.05), "Vert": np.zeros(n), "Long": np.zeros(n)}
r = offset_from_samples(chans, pretrig_n=0)
assert r["offset"] is True
def test_flags_offset_on_any_axis():
# Offset on Vert alone still flags the event, and Vert is reported.
n = 300
chans = {"Tran": np.zeros(n), "Vert": np.full(n, -0.06), "Long": np.zeros(n)}
r = offset_from_samples(chans, pretrig_n=50)
assert r["offset"] is True
assert r["axis"] == "Vert"
def _write_h5(path, chans, pretrig_n):
with h5py.File(path, "w") as f:
g = f.create_group("samples")
for k, v in chans.items():
g.create_dataset(k, data=np.asarray(v, dtype="float32"))
if pretrig_n is not None:
f.attrs["pretrig_samples"] = pretrig_n
def test_offset_from_h5_reads_pretrig_attr(tmp_path):
p = tmp_path / "ev.h5"
n = 300
_write_h5(p, {"Tran": np.full(n, 0.05), "Vert": np.zeros(n), "Long": np.zeros(n)},
pretrig_n=50)
r = offset_from_h5(str(p))
assert r["offset"] is True and r["axis"] == "Tran"
def test_offset_from_h5_missing_pretrig_attr_falls_back(tmp_path):
p = tmp_path / "noattr.h5"
n = 300
_write_h5(p, {"Tran": np.full(n, 0.05), "Vert": np.zeros(n), "Long": np.zeros(n)},
pretrig_n=None)
assert offset_from_h5(str(p))["offset"] is True # falls back to first-third
def test_offset_from_h5_missing_file_is_none(tmp_path):
assert offset_from_h5(str(tmp_path / "nope.h5")) is None
-64
View File
@@ -1,64 +0,0 @@
from __future__ import annotations
from pathlib import Path
import numpy as np, h5py
from sfm.database import SeismoDb
from sfm.waveform_store import WaveformStore
from scripts.backfill_event_shape import backfill_shape
from minimateplus.models import Event, Timestamp, PeakValues
_FIX = Path(__file__).parent / "fixtures/histogram-extension-re/events-5-21-26/K558LL8B.7I0W"
def _event(waveform_key="0111abcd"):
ev = Event(index=0)
ev._waveform_key = bytes.fromhex(waveform_key)
ev.timestamp = Timestamp(raw=b"", flag=0x10, year=2026, unknown_byte=0,
month=6, day=25, hour=8, minute=50, second=0)
ev.record_type = "Waveform"
ev.peak_values = PeakValues(tran=0.075, vert=0.220, long=0.045,
peak_vector_sum=0.231, micl=0.01)
return ev
def test_insert_stores_offset_from_record(tmp_path: Path):
db = SeismoDb(tmp_path / "s.db")
ev = _event()
rec = {ev._waveform_key.hex(): {
"filename": "F.CE0W", "filesize": 10,
"shape_offset": 1, "shape_offset_axis": "Tran",
"shape_offset_pre": 0.05, "shape_offset_spread": 0.001}}
db.insert_events([ev], serial="BE1", waveform_records=rec)
row = db.query_events(serial="BE1")[0]
assert row["shape_offset"] == 1
assert row["shape_offset_axis"] == "Tran"
assert abs(row["shape_offset_pre"] - 0.05) < 1e-6
assert abs(row["shape_offset_spread"] - 0.001) < 1e-6
def test_save_imported_bw_attaches_offset(tmp_path: Path):
store = WaveformStore(tmp_path / "waveforms")
ev, rec = store.save_imported_bw(_FIX.read_bytes(), source_path=_FIX, serial_hint="BE9558")
assert rec["shape_offset"] in (0, 1)
assert rec["shape_offset_axis"] in ("Tran", "Vert", "Long")
assert "shape_offset_pre" in rec and "shape_offset_spread" in rec
def test_backfill_updates_offset(tmp_path: Path):
db = SeismoDb(tmp_path / "s.db")
store = WaveformStore(tmp_path / "waveforms")
ev = Event(index=0); ev._waveform_key = bytes.fromhex("0111abcd")
db.insert_events([ev], serial="BE1",
waveform_records={ev._waveform_key.hex(): {"filename": "F.CE0W", "filesize": 10}})
p = store.hdf5_path_for("BE1", "F.CE0W")
with h5py.File(p, "w") as f:
g = f.create_group("samples")
g.create_dataset("Tran", data=np.full(300, 0.05, "float32"))
g.create_dataset("Vert", data=np.zeros(300, "float32"))
g.create_dataset("Long", data=np.zeros(300, "float32"))
f.attrs["pretrig_samples"] = 50
backfill_shape(db, store)
row = db.query_events(serial="BE1")[0]
assert row["shape_offset"] == 1
assert row["shape_offset_axis"] == "Tran"