19 Commits
Author SHA1 Message Date
serversdown d2258dad4e Merge pull request 'Release v0.31.0 — report parity + the inverted rescue (0.29.0 → 0.31.0)' (#40) from dev into main
Reviewed-on: #40
2026-09-18 16:46:41 -04:00
serversdownandClaude Opus 5 0b58415fe2 chore(release): v0.31.0 — report parity + the inverted rescue
Cuts Unreleased to v0.31.0 and writes the theme now that the whole release is
visible, per the convention adopted today.

Two threads landed. Blastware Event/FFT-Report parity — the FFT, the USBM
RI8507 compliance chart, and the sensor self-check decoded for both series and
standardized into the .h5 (schema v2, /sensor_check). And the ach_server rescue
flags out of the BE12599 field emergency, which invert the wedged-unit recovery:
answer the unit's call instead of racing a Stop into the gaps between its
dial-outs.

Version stamped in pyproject.toml, CLAUDE.md and README.md. TOOL_VERSION was
already at 0.31.0 — it came in with the sensor-check work, and it is what makes
the backfill pick up the new /sensor_check group without --force.

⚠ This release owes prod a backfill: .h5 schema v1 -> v2, ~2 h on the NAS.
Stated in the Migration block.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN
2026-09-18 20:40:48 +00:00
serversdownandClaude Opus 5 fa22bb9f59 Merge feat/sensor-check-h5 into dev
Sensor self-check standardized into the .h5 (schema v2, /sensor_check group),
decoded for both series-3 and series-4, plus the Thor backfill script.

CHANGELOG resolved per the convention adopted today: the incoming Unreleased
preamble was dropped rather than reconciled — no preamble under Unreleased, the
theme gets written at release time — and its load-bearing half was folded into
### Migration, which said "None" and is now false.

That block now states the real cost: .h5 schema v1 -> v2, TOOL_VERSION 0.31.0
so the standard backfill picks the traces up with no --force, and ~2 h on the
NAS. The FFT, the compliance chart and the ach_server rescue flags still owe
nothing.

The branch's rewritten "Sensor self-check — both series" entry merged cleanly
and is kept.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN
2026-09-18 20:16:13 +00:00
serversdown 27e9c56393 Merge pull request 'Feat/ach rescue on connect' (#38) from feat/ach-rescue-on-connect into dev
Reviewed-on: #38
2026-09-18 15:26:18 -04:00
serversdownandClaude Opus 5 0408c37866 docs: write the changelog on dev, not on feature branches
Reverses the "entry goes in with the work" rule from two commits ago. That was
wrong on the evidence: of the docs(changelog) commits in history, 3 of 4 in
seismo-relay and 2 of 4 in Terra-View were made directly on dev. The rule was
generalized from one unrepresentative commit rather than from the pattern.

It also caused the exact problem it was supposed to avoid. With four worktrees
in flight, every branch edits the same few lines at the top of CHANGELOG.md;
feat/ach-rescue-on-connect and feat/sensor-check-h5 collide on that file and
nothing else. Writing the entry once, on dev, after the merge removes the
whole conflict class.

The second benefit is accuracy: an entry written after the merge describes
what actually landed, including anything that changed during conflict
resolution. The sensor-check branch is a live example — its Unreleased
preamble describes a release that no longer looks like that.

The failure mode of writing it later is forgetting, so the merge is explicitly
not finished until Unreleased is updated — same sitting, reconstructed from the
branch commit messages.

Unchanged: no preamble under Unreleased, the mandatory operational consequence,
and cutting the version on dev when ready to ship to main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN
2026-09-18 18:52:17 +00:00
serversdownandClaude Opus 5 a42e8d3651 docs: make the release cadence explicit
Brian described the practice: Unreleased is the staging area for what is going
into the next release, and the version bump happens when enough has
accumulated to be worth shipping — not per commit, not per merge. The
convention already implied it ("never touch the changelog at a merge
boundary") but never said it outright.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN
2026-09-18 18:04:54 +00:00
serversdownandClaude Opus 4.8 8265e32ad5 feat(backfill): regenerate .h5 with /sensor_check; TOOL_VERSION 0.31.0
Complete the sensor-check standardization: existing events need their .h5
regenerated to gain the v2 /sensor_check group.

  * backfill_thor_events.py attaches the decoded series-4 traces
    (micromate.sensor_check) on its own IDF decode path, mirroring
    save_imported_idf, so regenerated Thor .h5 files get the group. Series-3
    backfill needs no change — it re-decodes via read_blastware_file, which now
    attaches the traces itself.
  * TOOL_VERSION 0.30.0 → 0.31.0 so the standard backfill regenerates every
    event (no --force): the tool now produces the /sensor_check group. Purely
    additive — no decoded value changes.
  * CHANGELOG (Unreleased): sensor-check now series-3 + series-4, standardized
    into the .h5 (schema v2), with the ⚠ backfill note; FFT + compliance stay
    no-backfill (they read existing .h5 samples).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
2026-09-16 06:53:33 +00:00
serversdownandClaude Opus 4.8 8d3cdba1b5 feat(h5): standardize sensor-check into the .h5 (schema v2); SFM reads it
Make the sensor self-check a first-class part of the standardized decoded event
so SFM stops decoding it at report time — device-agnostic, per the store's
decoder→standardized-.h5→SFM model.

  * Event gains a `sensor_check` field; both decoders attach the traces where
    they set raw_samples — series-3 in event_file_io.read_blastware_file
    (minimateplus.sensor_check), series-4 in waveform_store's IDF path
    (micromate.sensor_check).  Covers ingest and backfill (both re-decode).
  * event_hdf5 bumps schema_version 1→2 and writes an optional /sensor_check
    group (raw counts, int32, per channel present).  read_event_hdf5 returns
    it; plot_json_from_hdf5 carries it as a top-level key.  Old v1 files still
    read cleanly (no group → None), so nothing breaks before the backfill.
  * gather_report_data reads sensor_check_waveforms from the .h5 and drops the
    report-time series-3 decode — the report no longer reaches into a decoder,
    and a series-4 event now lights up the same strip automatically.

Stored as raw counts (a shape diagnostic, rendered fit-to-box): the per-series
count scale differs and a physical mic unit is ill-defined, so conversion would
add complexity for no display benefit — easy to add later if a numeric use
appears.

Tests: .h5 roundtrip + backward-compat + plot_json + real series-3 decode
attaches to the Event.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
2026-09-16 06:49:55 +00:00
serversdownandClaude Opus 4.8 685a17d180 feat(series4): decode sensor self-check waveforms from the IDFW binary
The Thor/Micromate (series-4) IDFW binary carries the sensor self-check in its
fixed-header region (before the waveform body), as up to four records tagged
01 0e 3c/3d/3e/3f — the SAME channel ids as series-3 (Tran/Vert/Long/MicL).
Unlike series-3's delta-coded trailing block, series-4 stores each trace as a
raw int16-BE array after an 18-byte record header (2-byte sample count at
offset +8). Three-channel (mic-disabled) units carry only 3c/3d/3e.

New micromate/sensor_check.py: decode_idf_sensor_check(raw) locates the record
chain (id-ordered marker run, so a stray body match can't chain) and reads each
trace's int16 samples → {Tran,Vert,Long[,MicL]: [counts]}, or {} when absent.

Reverse-engineered + validated against 4 UM oracle events (added as fixtures):
clean geophone ring-downs on all, mic pulse trains on the 4-channel units,
correctly no MicL on the two 3-channel units. Validated by shape + cross-event
consistency (no Thor report strip to exact-match, unlike series-3's BW reports).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
2026-09-15 20:01:39 +00:00
serversdownandClaude Opus 4.8 dcd9ad6f48 Merge feat/fft-series3: Blastware FFT, USBM compliance chart, sensor self-check
Reverse-engineered Blastware Event/FFT-Report parity, all additive (reads the
existing .h5 samples + retained raw binary, no DB/.h5 change or backfill):
 - Blastware-compatible channel FFT (waveform_fft)
 - USBM RI8507/OSMRE compliance chart on the event-report PDF (sfm/compliance)
 - sensor self-check strip decoded from the series-3 binary trailing block
   (minimateplus/sensor_check) + Frequency/Overswing sub-rows
 - seismo_lab Inspector hex reader (minimateplus/binary_annotate)
 - report fixes: stacked-lane y-tick collision, header serial fit

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
2026-09-15 14:37:44 +00:00
serversdown e050e602c2 Merge pull request 'Release v0.29.0 — offset detector + false_trigger_reason + BlastMate serials (0.27.0→0.29.0)' (#36) from dev into main
Reviewed-on: #36
2026-09-07 15:57:37 -04:00
serversdown fc1c7c7936 Merge pull request 'Docs/claude.md corrections' (#35) from dev into main
Reviewed-on: #35
2026-08-29 15:48:33 -04:00
serversdown 7d0d12079b Merge pull request 'v0.27.0 Decoder fixes, offset exploration and testing.' (#34) from dev into main
Reviewed-on: #34
2026-08-28 22:40:34 -04:00
serversdown 5203aab849 Merge pull request 'update to 0.26.0. Big chonking update including 0.23, 0.24, and 0.25 as well.' (#33) from dev into main
Reviewed-on: #33
2026-08-27 13:43:00 -04:00
serversdown 5b65718b72 Merge pull request 'v0.23.0 - ZC freq in events store' (#31) from dev into main
Reviewed-on: #31
2026-08-06 12:12:19 -04:00
serversdown 483762607e Merge pull request 'Update to v0.22.0' (#30) from dev into main
Reviewed-on: #30
2026-07-03 15:35:20 -04:00
serversdown d0b66368d5 Merge pull request 'update to v0.21.1, thor data import successful' (#29) from dev into main
Reviewed-on: #29
2026-06-01 16:54:23 -04:00
serversdown 2eb1d25028 Merge pull request 'v0.20.0 -- Full s3 event parse and PDF creation.' (#28) from dev into main
Reviewed-on: #28
2026-05-28 17:54:31 -04:00
claude cc821f9ee3 hotfix: fix dockerfile on main to fix import bug on prod 2026-05-21 20:42:15 +00:00
17 changed files with 385 additions and 48 deletions
+58 -18
View File
@@ -4,7 +4,30 @@ All notable changes to seismo-relay are documented here.
--- ---
## Unreleased ## v0.31.0 — 2026-09-18
**Report parity, and a second way to rescue a runaway unit.** Two threads.
The first closes out Blastware Event/FFT-Report parity: the FFT, the USBM
RI8507 compliance chart and the sensor self-check now render on the event
report, reverse-engineered against BE12844 (MiniMate Plus) and UM (Thor)
events. The sensor check is decoded for **both** series and standardized into
the `.h5` (schema **v2**, a new `/sensor_check` group), so SFM serves it
device-agnostically rather than decoding at report time. The Inspector — an
annotated hex reader for series-3 binaries — is what made the trailing-block
structure findable, and it earned its keep by *ruling out* a stored FFT block
and proving Blastware computes it from the samples.
The second came out of a field emergency. BE12599's connector fault drove its
Tran channel to its trigger level, so the unit recorded back-to-back and dialed
the office ACH server every ~75 s, unreachable the whole time.
`bridges/ach_server.py` gained `--stop-monitoring` / `--disable-ach` /
`--rescue`, which **invert** the recovery: instead of racing a Stop into the
gaps between dial-outs, point the modem's Destination at our own ACH server and
answer the call. Proven in production the same night — the stop landed on the
first call-in and held. See `docs/runbooks/wedged_unit_recovery.md`.
⚠ **This release owes prod a backfill** — see Migration below.
### Added ### Added
- **Rescue-on-connect for `bridges/ach_server.py`** — `--stop-monitoring` - **Rescue-on-connect for `bridges/ach_server.py`** — `--stop-monitoring`
@@ -46,18 +69,25 @@ All notable changes to seismo-relay are documented here.
reference PDF. A technical breakdown of the curve is in reference PDF. A technical breakdown of the curve is in
`docs/ri8507_compliance_curve.md`. `docs/ri8507_compliance_curve.md`.
- **Sensor self-check waveforms decoded and drawn (`minimateplus.sensor_check`).** - **Sensor self-check waveforms decoded and drawn — both series.** The little
The "Sensor Check" traces Blastware shows to the right of the waveform panel "Sensor Check" traces (geophone ring-downs — the transducer's damped impulse
live in the series-3 binary's trailing block as four length-prefixed records response — plus a MicL pulse train, the mic's known-signal gain check) are the
(`0x3c`–`0x3f`) using the same delta-block codec as the main waveform: unit's proof its sensors were healthy when it recorded the event.
Tran/Vert/Long geophone ring-downs (the transducer's damped impulse response — - **Series-3** (`minimateplus.sensor_check`): four records (`0x3c`–`0x3f`) in
resonant frequency + overswing/damping) and a MicL pulse train (the mic's the binary's trailing block, same delta-block codec as the main waveform.
known-signal gain check). `gather_report_data` decodes them from the retained Verified against all 7 BE12844 reports (mic zero-crossing = 20.1 Hz exact;
BW binary at report time; the report renders them as a strip flush against the geophone ring-downs ~7.5 Hz, overswing ~3.5).
waveform panel plus the **Sensor Check → Frequency / Overswing Ratio** sub-rows - **Series-4** (`micromate.sensor_check`): the same self-test in the Thor IDFW
in the stats table. Verified against the reports on all 7 oracle events (mic fixed header — four `01 0e 3c/3d/3e/3f` records (same channel ids) storing
zero-crossing frequency = 20.1 Hz exact; geophone ring-downs consistent raw int16 traces; three-channel (mic-disabled) units carry only the three
~7.5 Hz with overswing ~3.5). Tests in `tests/test_sensor_check.py`. geophones. Validated by shape + cross-event consistency.
- **Standardized into the `.h5`** (`/sensor_check`, schema v2): each series'
decoder attaches the traces to the event at decode, the writer persists
them, and `gather_report_data` reads them back — so SFM renders the strip
(flush against the waveform panel) plus the **Sensor Check → Frequency /
Overswing Ratio** sub-rows without knowing the source instrument.
- Tests: `tests/test_sensor_check.py`, `tests/test_sensor_check_idf.py`,
`tests/test_event_hdf5_sensor_check.py`.
- **Inspector tab in `seismo_lab.py` — annotated hex reader for series-3 - **Inspector tab in `seismo_lab.py` — annotated hex reader for series-3
binaries (`minimateplus/binary_annotate.py`).** Tiles a raw Blastware file binaries (`minimateplus/binary_annotate.py`).** Tiles a raw Blastware file
@@ -77,11 +107,21 @@ All notable changes to seismo-relay are documented here.
### Migration ### Migration
**None.** Every change here is additive and reads from data already on disk — ⚠ **The sensor-check needs a backfill.** Existing `.h5` files are schema v1
the `.h5` samples and the retained raw BW binary. No `.h5`/DB change, no and carry no `/sensor_check` group, so their reports show no sensor-check strip
schema change, no migration, no backfill, and **no `TOOL_VERSION` bump**: a until regenerated. `TOOL_VERSION` is bumped to **0.31.0**, so the standard
report regenerated for an existing event simply gains the new panels, and the backfill regenerates every event and picks up the traces with **no `--force`**:
`ach_server` rescue flags don't touch the codec. `scripts/backfill_thor_events.py` for series-4 (it already owed a v0.30.0 Thor
backfill — this rides along) and the series-3 sidecar/shape backfill for
MiniMate events. Purely additive — no decoded value changes, and v1 `.h5`
files read fine until then (empty strip). DB backup first, as always.
⚠ Budget **~2 h on the NAS** — ~1.5 files/sec there versus ~85/sec on the dev
box (gzip-4 in `sfm/event_hdf5.py` against a Synology CPU).
Everything else in this release owes nothing: the FFT, the USBM compliance
chart and the `ach_server` rescue flags are additive and read data already on
disk — no schema change, no DB migration.
--- ---
+18 -7
View File
@@ -2,7 +2,7 @@
Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
(Sierra Wireless RV50 / RV55). Current version: **v0.30.0**. (Sierra Wireless RV50 / RV55). Current version: **v0.31.0**.
Stack-level context — which repo owns what, and how the three project versions Stack-level context — which repo owns what, and how the three project versions
pair — lives in `../terra-view/docs/tmi-stack.md`, which is also loaded as pair — lives in `../terra-view/docs/tmi-stack.md`, which is also loaded as
@@ -91,13 +91,20 @@ When new information about the protocol is discovered, please update the instant
## Changelog & release convention ## Changelog & release convention
**Write the entry in the same commit as the work, under `## Unreleased`. Cut **Feature branches do NOT touch `CHANGELOG.md`. Write the entry on `dev`, as
the version on `dev` in a dedicated release commit. Never touch the changelog part of finishing the merge, under `## Unreleased`. Cut the version on `dev` in a
at a merge boundary.** dedicated release commit when you are ready to ship to `main`.**
- **Entry goes in with the change**, not at merge or release time — that is the - **The changelog is written on `dev`, never on a feature branch.** With
only moment you still know *why*. Feature branches edit `CHANGELOG.md` several branches in flight they all edit the same few lines at the top of
directly; the occasional conflict is two appended bullets and is trivial. the file and conflict every time. Writing it once, after the merge, also
lets it describe what actually *landed* — including anything that changed
during conflict resolution.
- ⚠ **The merge is not finished until `## Unreleased` is updated.** Same sitting,
not "later" — that is the one failure mode of writing it after the fact.
Reconstruct from the branch's own commit messages:
`git log --oneline dev..<branch>` before you merge, or
`git log --oneline <merge-base>..<branch>` after.
- **No preamble under `## Unreleased`** — just the `### Added` / `### Changed` / - **No preamble under `## Unreleased`** — just the `### Added` / `### Changed` /
`### Fixed` lists. The themed opening paragraph gets written at release `### Fixed` lists. The themed opening paragraph gets written at release
time, when the whole release is visible and can be named honestly. A theme time, when the whole release is visible and can be named honestly. A theme
@@ -108,6 +115,10 @@ at a merge boundary.**
"`TOOL_VERSION` bumped", "no schema change, no migration". Silence is "`TOOL_VERSION` bumped", "no schema change, no migration". Silence is
ambiguous; "none" is information. This repo's changelog is how future-you ambiguous; "none" is information. This repo's changelog is how future-you
learns whether a deploy costs two hours. learns whether a deploy costs two hours.
- **Releases are cut on judgement, not on a schedule or a merge.** `Unreleased`
is the staging area for whatever is going into the next release; when enough
has accumulated to be worth shipping, it gets a number and a date. Nothing
about a merge to `dev` triggers a release.
- **Cutting a release** is its own `chore(release): vX.Y.Z — <theme>` commit on - **Cutting a release** is its own `chore(release): vX.Y.Z — <theme>` commit on
`dev`, renaming `## Unreleased` → `## vX.Y.Z — YYYY-MM-DD` and touching: `dev`, renaming `## Unreleased` → `## vX.Y.Z — YYYY-MM-DD` and touching:
`CHANGELOG.md`, `pyproject.toml`, the version line in `CLAUDE.md` and `CHANGELOG.md`, `pyproject.toml`, the version line in `CLAUDE.md` and
+1 -1
View File
@@ -1,4 +1,4 @@
# seismo-relay `v0.30.0` # seismo-relay `v0.31.0`
A ground-up replacement for **Blastware** — Instantel's aging Windows-only A ground-up replacement for **Blastware** — Instantel's aging Windows-only
software for managing seismographs. Supports both the **MiniMate Plus software for managing seismographs. Supports both the **MiniMate Plus
+89
View File
@@ -0,0 +1,89 @@
r"""Decode the Thor / Micromate (series-4) sensor self-check waveforms from an
IDFW event binary.
Reverse-engineered 2026-09-15 against 4 UM (Thor) oracle events. The IDFW
binary carries the sensor self-check in its fixed-header region (before the
waveform body), as up to four records tagged ``01 0e 3c/3d/3e/3f`` — the SAME
channel ids as the series-3 MiniMate Plus (Tran / Vert / Long / MicL), which is
the physical self-test:
* 3c / 3d / 3e = Tran / Vert / Long geophone ring-downs (a damped impulse
response — resonant frequency + damping).
* 3f = MicL pulse train (the mic's known-signal gain check). Absent
on three-channel (mic-disabled) units.
Record framing (per record)::
01 0e [id:1] [flags:3] [count:2 BE] [pad:10] [int16-BE samples × count]
\___ 18-byte header ___/
Unlike series-3's delta-coded trailing block, series-4 stores each trace as a
raw int16 big-endian array. ``count`` (the 2-byte field at header offset +8)
is the sample count; the record is padded to a fixed stride after that.
"""
from __future__ import annotations
import struct
from typing import Dict, List
# Record id → channel. Same ids/order as series-3 (minimateplus.sensor_check).
_ID_TO_CHANNEL = {0x3C: "Tran", 0x3D: "Vert", 0x3E: "Long", 0x3F: "MicL"}
_CHAIN_IDS = (0x3C, 0x3D, 0x3E, 0x3F)
_MARKER = b"\x01\x0e" # precedes the 1-byte channel id
_HEADER_LEN = 18 # bytes from the marker start to the first sample
_COUNT_OFF = 8 # 2-byte BE sample count, from the marker start
_MAX_COUNT = 4000 # sanity cap (traces are ~70-200 samples)
def _find_chain(raw: bytes):
"""Locate the sensor-check record chain. Returns a list of
``(offset, id, count)`` for the first run of markers whose ids run
3c, 3d, 3e[, 3f] in order, or ``[]``.
Records are padded to a fixed stride, so the next marker is not at
``header + count*2``; instead collect every ``01 0e [id]`` marker with a
sane count and take the first id-ordered run. Validating the id sequence
(not a lone ``01 0e 3c``) keeps a stray marker in the waveform body from
matching — the real chain sits in the fixed header, ahead of the body.
"""
n = len(raw)
markers = []
for p in range(n - _HEADER_LEN):
if raw[p:p + 2] == _MARKER and raw[p + 2] in _ID_TO_CHANNEL:
count = int.from_bytes(raw[p + _COUNT_OFF:p + _COUNT_OFF + 2], "big")
if 0 < count <= _MAX_COUNT:
markers.append((p, raw[p + 2], count))
for i, (off, rid, _c) in enumerate(markers):
if rid != 0x3C:
continue
run = [markers[i]]
for m in markers[i + 1:]:
if len(run) < len(_CHAIN_IDS) and m[1] == _CHAIN_IDS[len(run)]:
run.append(m)
else:
break
if len(run) >= 3: # 3-channel (mic-disabled) units are valid
return run
return []
def decode_idf_sensor_check(raw: bytes) -> Dict[str, List[int]]:
"""Decode the sensor self-check traces from a Thor/Micromate IDFW binary.
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}`` in
raw int16 ADC counts (MicL omitted on 3-channel units), or ``{}`` if the
binary carries no sensor-check chain (a non-IDF file, or an IDFH histogram).
"""
chain = _find_chain(raw)
if not chain:
return {}
out: Dict[str, List[int]] = {}
for off, rid, count in chain:
start = off + _HEADER_LEN
blob = raw[start:start + count * 2]
if len(blob) < count * 2:
continue
out[_ID_TO_CHANNEL[rid]] = list(struct.unpack(">%dh" % count, blob))
return out
+6 -1
View File
@@ -50,7 +50,7 @@ SIDECAR_KIND = "sfm.event"
# bumped without a `pip install` re-run — leading to confusing stale # bumped without a `pip install` re-run — leading to confusing stale
# version stamps in sidecars. Bump this constant and CHANGELOG.md # version stamps in sidecars. Bump this constant and CHANGELOG.md
# together at release time. # together at release time.
TOOL_VERSION = "0.30.0" TOOL_VERSION = "0.31.0" # +/sensor_check group (schema v2); gates the backfill regen
try: try:
# Best-effort: prefer the installed metadata when it's NEWER than the # Best-effort: prefer the installed metadata when it's NEWER than the
@@ -960,6 +960,11 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
project=project, client=client, operator=user, sensor_location=seisloc, project=project, client=client, operator=user, sensor_location=seisloc,
) )
ev.raw_samples = samples ev.raw_samples = samples
# Sensor self-check traces from the binary's trailing block (waveform
# events only; returns {} for histograms / when absent). Carried on the
# Event so the .h5 writer persists them device-agnostically.
from minimateplus.sensor_check import decode_sensor_check
ev.sensor_check = decode_sensor_check(raw) or None
# Only compute peaks from samples when we actually have samples. # Only compute peaks from samples when we actually have samples.
# For events the codec couldn't decode (histogram-mode bodies, until # For events the codec couldn't decode (histogram-mode bodies, until
# the §7.6.2 histogram codec is wired in), samples is an empty dict # the §7.6.2 histogram codec is wired in), samples is an empty dict
+9
View File
@@ -544,6 +544,15 @@ class Event:
pretrig_samples: Optional[int] = None # from STRT record: pre-trigger sample count pretrig_samples: Optional[int] = None # from STRT record: pre-trigger sample count
rectime_seconds: Optional[int] = None # from STRT record: record duration (seconds) rectime_seconds: Optional[int] = None # from STRT record: record duration (seconds)
# Sensor self-check traces keyed by channel label — the short diagnostic
# waveforms the unit records when it pulses each sensor before monitoring
# (geophone ring-downs + a mic pulse train). Decoded from the binary by
# the per-series decoder (minimateplus.sensor_check / micromate.sensor_check)
# and carried here so the .h5 writer can persist them device-agnostically.
# Raw ADC counts; the source series' scale differs but the trace is a
# shape diagnostic (rendered fit-to-box). None when absent.
sensor_check: Optional[dict] = None # {"Tran": [...], ..., "MicL": [...]}
# ── Debug / introspection ───────────────────────────────────────────────── # ── Debug / introspection ─────────────────────────────────────────────────
# Raw 210-byte waveform record bytes, set when debug mode is active. # Raw 210-byte waveform record bytes, set when debug mode is active.
# Exposed by the SFM server via ?debug=true so field layouts can be verified. # Exposed by the SFM server via ?debug=true so field layouts can be verified.
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "seismo-relay" name = "seismo-relay"
version = "0.30.0" version = "0.31.0"
description = "Python client and REST server for MiniMate Plus seismographs" description = "Python client and REST server for MiniMate Plus seismographs"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [
+5
View File
@@ -305,6 +305,11 @@ def main(argv=None) -> int:
default=0, default=0,
) )
ev.total_samples = ev.total_samples or n_samp ev.total_samples = ev.total_samples or n_samp
# Sensor self-check traces from the IDFW fixed
# header, so regenerated .h5 files gain the v2
# /sensor_check group (mirrors save_imported_idf).
from micromate.sensor_check import decode_idf_sensor_check
ev.sensor_check = decode_idf_sensor_check(binary_bytes) or None
event_hdf5.write_event_hdf5( event_hdf5.write_event_hdf5(
hdf5_path, ev, hdf5_path, ev,
+44 -4
View File
@@ -12,8 +12,11 @@ Layout written to `<filename>.h5`:
├─ samples_int16/ (optional) ├─ samples_int16/ (optional)
│ ├─ Tran (int16, raw ADC counts) shape: (N,) │ ├─ Tran (int16, raw ADC counts) shape: (N,)
│ └─ ... per channel (only when present in the source) │ └─ ... per channel (only when present in the source)
├─ sensor_check/ (optional, schema v2+)
│ ├─ Tran (int32, raw counts) shape: (M,) M ≪ N
│ └─ ... per channel present in the source (MicL absent on 3-channel units)
└─ root attrs (event metadata): └─ root attrs (event metadata):
schema_version int = 1 schema_version int = 2
kind str = "sfm.event.hdf5" kind str = "sfm.event.hdf5"
serial str serial str
waveform_key str (8-hex) waveform_key str (8-hex)
@@ -64,7 +67,7 @@ from minimateplus.models import Event
log = logging.getLogger(__name__) log = logging.getLogger(__name__)
SCHEMA_VERSION = 1 SCHEMA_VERSION = 2 # v2 adds the optional /sensor_check group
HDF5_KIND = "sfm.event.hdf5" HDF5_KIND = "sfm.event.hdf5"
# Geophone full-scale velocity per range (in/s). Confirmed in CLAUDE.md # Geophone full-scale velocity per range (in/s). Confirmed in CLAUDE.md
@@ -270,6 +273,22 @@ def write_event_hdf5(
) )
igrp.attrs["mic_psi_per_count"] = float(mic_factor) igrp.attrs["mic_psi_per_count"] = float(mic_factor)
# /sensor_check — optional short diagnostic self-check traces (schema
# v2+). Raw ADC counts (a shape diagnostic; the per-series count scale
# differs, and the renderer fits each trace to its box). Only channels
# the decoder found are written — 3-channel units carry no MicL.
sc = event.sensor_check or {}
if sc:
scgrp = f.create_group("sensor_check")
for ch in ("Tran", "Vert", "Long", "MicL"):
vals = sc.get(ch)
if vals:
scgrp.create_dataset(
ch, data=np.asarray(vals, dtype=np.int32),
compression="gzip", compression_opts=4, shuffle=True,
)
scgrp.attrs["units"] = "raw_counts"
import os import os
os.replace(tmp, path) os.replace(tmp, path)
@@ -334,6 +353,16 @@ def read_event_hdf5(path: Union[str, Path]) -> dict:
if mic_attr is not None: if mic_attr is not None:
mic_psi = float(mic_attr) mic_psi = float(mic_attr)
# /sensor_check — optional (schema v2+); absent on older files.
sensor_check = None
scgrp = f.get("sensor_check")
if scgrp is not None:
sensor_check = {}
for ch in ("Tran", "Vert", "Long", "MicL"):
ds = scgrp.get(ch)
if ds is not None:
sensor_check[ch] = np.asarray(ds[()])
return { return {
"schema_version": sv, "schema_version": sv,
"kind": attrs.get("kind"), "kind": attrs.get("kind"),
@@ -341,6 +370,7 @@ def read_event_hdf5(path: Union[str, Path]) -> dict:
"samples": samples, "samples": samples,
"samples_int16": samples_int16, "samples_int16": samples_int16,
"mic_psi_per_count": mic_psi, "mic_psi_per_count": mic_psi,
"sensor_check": sensor_check,
} }
@@ -431,11 +461,16 @@ def plot_json_from_hdf5(
event_id: Optional[str] = None, event_id: Optional[str] = None,
index: Optional[int] = None, index: Optional[int] = None,
) -> dict: ) -> dict:
"""Build a `sfm.plot.v1` JSON dict from a stored .h5 file.""" """Build a `sfm.plot.v1` JSON dict from a stored .h5 file.
The dict also carries a top-level ``sensor_check`` key (the raw self-check
traces as ``{ch: [int]}``, or None) beyond the plot schema, so report
generation can read the traces from the same single .h5 load.
"""
data = read_event_hdf5(path) data = read_event_hdf5(path)
a = data["attrs"] a = data["attrs"]
s = data["samples"] s = data["samples"]
return _build_plot_dict( out = _build_plot_dict(
n_samples=len(s["Tran"]) if "Tran" in s else 0, n_samples=len(s["Tran"]) if "Tran" in s else 0,
sample_rate=int(a.get("sample_rate", 1024) or 1024), sample_rate=int(a.get("sample_rate", 1024) or 1024),
pretrig_samples=int(a.get("pretrig_samples", 0) or 0), pretrig_samples=int(a.get("pretrig_samples", 0) or 0),
@@ -463,6 +498,11 @@ def plot_json_from_hdf5(
event_id=event_id, event_id=event_id,
index=index, index=index,
) )
scd = data.get("sensor_check")
out["sensor_check"] = (
{ch: v.tolist() for ch, v in scd.items()} if scd else None
)
return out
def _build_plot_dict( def _build_plot_dict(
+11 -16
View File
@@ -121,9 +121,11 @@ class ReportData:
t0_ms: Optional[float] = None t0_ms: Optional[float] = None
dt_ms: Optional[float] = None dt_ms: Optional[float] = None
# Sensor self-check traces — {ch: [samples]} in raw decode units, decoded # Sensor self-check traces — {ch: [samples]} in raw counts, read from the
# from the binary's trailing block (see minimateplus.sensor_check). The # standardized .h5 (/sensor_check group, schema v2+) where the per-series
# little waveforms BW draws in its "Sensor Check" strip. Empty when absent. # decoder stored them at ingest. The little diagnostic waveforms BW draws
# in its "Sensor Check" strip. Empty when absent (pre-v2 .h5, histogram,
# or 3-channel unit's MicL).
sensor_check_waveforms: dict = field(default_factory=dict) sensor_check_waveforms: dict = field(default_factory=dict)
# Record-type discriminator # Record-type discriminator
@@ -294,22 +296,15 @@ def gather_report_data(
rd.pretrig_samples = ta.get("pretrig_samples") rd.pretrig_samples = ta.get("pretrig_samples")
rd.t0_ms = ta.get("t0_ms") rd.t0_ms = ta.get("t0_ms")
rd.dt_ms = ta.get("dt_ms") rd.dt_ms = ta.get("dt_ms")
# Sensor self-check traces — read from the standardized .h5 (schema
# v2+). Device-agnostic: whichever decoder produced the event
# stored them at ingest, so SFM reads them here without knowing or
# caring about the source instrument series. Empty on pre-v2 files
# (until backfilled) and on 3-channel / histogram events.
rd.sensor_check_waveforms = wf.get("sensor_check") or {}
except Exception as exc: except Exception as exc:
log.warning("gather_report_data: hdf5 read failed: %s", exc) log.warning("gather_report_data: hdf5 read failed: %s", exc)
# ── Sensor self-check traces — decoded from the retained raw binary ──
# The .h5 holds only the main waveform; the sensor-check traces live in the
# binary's trailing block, so decode them straight from the kept BW file.
# Waveform events only (histograms have no sensor-check strip).
if not rd.is_histogram:
try:
from minimateplus.sensor_check import decode_sensor_check
bw_path, _a5 = store.paths_for(serial, filename)
if bw_path.exists():
rd.sensor_check_waveforms = decode_sensor_check(bw_path.read_bytes())
except Exception as exc:
log.warning("gather_report_data: sensor-check decode failed: %s", exc)
# ── Histogram aggregation ── # ── Histogram aggregation ──
# Codec emits ~N per-block samples (typically 1/sec); BW reports # Codec emits ~N per-block samples (typically 1/sec); BW reports
# one bar per configured interval (1 min / 5 min / etc.). When # one bar per configured interval (1 min / 5 min / etc.). When
+5
View File
@@ -662,6 +662,11 @@ class WaveformStore:
ev.raw_samples = idf_samples ev.raw_samples = idf_samples
n_samples = max((len(idf_samples.get(ch, [])) for ch in ("Tran", "Vert", "Long", "MicL")), default=0) n_samples = max((len(idf_samples.get(ch, [])) for ch in ("Tran", "Vert", "Long", "MicL")), default=0)
ev.total_samples = ev.total_samples or n_samples ev.total_samples = ev.total_samples or n_samples
# Sensor self-check traces from the IDFW fixed header (waveform
# events only; {} on histograms / when absent). Carried on the
# bridged Event so the .h5 writer persists them like series-3.
from micromate.sensor_check import decode_idf_sensor_check
ev.sensor_check = decode_idf_sensor_check(idf_bytes) or None
# For IDFH histograms there are no per-sample waveform arrays — the # For IDFH histograms there are no per-sample waveform arrays — the
# device stores one peak ADC count per interval per channel. Synthesise # device stores one peak ADC count per interval per channel. Synthesise
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+71
View File
@@ -0,0 +1,71 @@
"""The event .h5 carries the sensor self-check traces (schema v2).
The sensor check is decoded by the per-series decoder and attached to the
standardized Event, so the .h5 writer persists it device-agnostically and SFM
reads it back without knowing which instrument produced it. Old v1 files (no
sensor_check group) must still read cleanly.
"""
import tempfile
from pathlib import Path
import numpy as np
from minimateplus.models import Event
from minimateplus.event_file_io import read_blastware_file
from sfm import event_hdf5
S3_FIX = Path(__file__).parent / "fixtures" / "fft-oracle-2026-09-14" / "N844LQHB.ZT0W"
def _write(ev, **kw):
d = Path(tempfile.mkdtemp())
p = d / "e.h5"
event_hdf5.write_event_hdf5(p, ev, serial="BE12844", **kw)
return p
def test_sensor_check_roundtrips_through_hdf5():
ev = Event(index=0)
ev.raw_samples = {"Tran": [1, 2, -3], "Vert": [0, 1], "Long": [2], "MicL": [5, -5]}
ev.sample_rate = 1024
sc = {"Tran": [0, -990, -500, -100], "Vert": [0, -980, -480],
"Long": [0, -986, -470], "MicL": [0, -1800, 1800, -1800]}
ev.sensor_check = sc
r = event_hdf5.read_event_hdf5(_write(ev))
assert r["schema_version"] == 2
assert set(r["sensor_check"]) == {"Tran", "Vert", "Long", "MicL"}
for ch, vals in sc.items():
assert r["sensor_check"][ch].tolist() == vals
def test_plot_json_carries_sensor_check():
ev = Event(index=0)
ev.raw_samples = {"Tran": [1, 2, 3]}
ev.sample_rate = 1024
ev.sensor_check = {"Tran": [0, -990, -500], "Vert": [0, -980],
"Long": [0, -986]} # 3-channel: no MicL
pj = event_hdf5.plot_json_from_hdf5(_write(ev))
assert pj["sensor_check"] is not None
assert "MicL" not in pj["sensor_check"]
assert pj["sensor_check"]["Tran"] == [0, -990, -500]
def test_event_without_sensor_check_still_reads_as_v2():
ev = Event(index=0)
ev.raw_samples = {"Tran": [1, 2, 3]}
ev.sample_rate = 1024
r = event_hdf5.read_event_hdf5(_write(ev))
assert r["schema_version"] == 2
assert r["sensor_check"] is None
assert event_hdf5.plot_json_from_hdf5(_write(ev))["sensor_check"] is None
def test_series3_decode_populates_event_sensor_check():
# The real series-3 decoder attaches the traces to the Event, so the
# ingest/backfill .h5 write picks them up with no extra plumbing.
ev = read_blastware_file(S3_FIX)
assert ev.sensor_check is not None
assert set(ev.sensor_check) == {"Tran", "Vert", "Long", "MicL"}
tran = np.asarray(ev.sensor_check["Tran"], dtype=float)
assert tran.min() < -800 # the geophone ring-down deflection
+67
View File
@@ -0,0 +1,67 @@
"""Series-4 (Thor / Micromate IDFW) sensor self-check waveform decode.
Reverse-engineered 2026-09-15 against 4 UM (Thor) oracle events. The IDFW
binary carries the sensor self-check in its fixed-header region (before the
waveform body) as up to four records tagged ``01 0e 3c/3d/3e/3f`` — the SAME
channel ids as series-3 (Tran/Vert/Long/MicL). Unlike series-3's delta-coded
trailing block, series-4 stores each trace as a raw int16-BE array after an
18-byte record header whose sample count is a 2-byte field at offset +8.
Three-channel (mic-disabled) Thor units carry only 3c/3d/3e — no MicL record.
Validated by shape (geophone ring-down / mic pulse train) and cross-event
consistency, since there's no Thor Event-Report strip to exact-match against.
"""
from pathlib import Path
import numpy as np
from micromate.sensor_check import decode_idf_sensor_check
FIXDIR = Path(__file__).parent / "fixtures" / "thor-idf-sc"
EVENTS = sorted(p.name for p in FIXDIR.glob("*.IDFW"))
def _decode(name):
return decode_idf_sensor_check((FIXDIR / name).read_bytes())
def test_geo_channels_present_and_ringdown_shaped():
# Every IDFW event has the three geophone self-checks; each is a large
# one-sided deflection (~15000 raw counts) that rings back — the geophone's
# damped impulse response.
for name in EVENTS:
sc = _decode(name)
for ch in ("Tran", "Vert", "Long"):
assert ch in sc, f"{name} missing {ch}"
tr = np.asarray(sc[ch], dtype=float)
tr = tr - tr[:4].mean() # reference to the pre-trigger baseline
assert 40 <= len(tr) <= 300, f"{name}:{ch} n={len(tr)}"
assert tr.min() < -8000, f"{name}:{ch} min {tr.min()}"
# deflects one way and rings back toward / past the baseline
assert tr.max() < abs(tr.min()), f"{name}:{ch} not one-sided"
def test_mic_present_only_on_four_channel_units():
# UM11719 / UM12947 record a mic; UM13981 / UM20147 are 3-channel
# (mic-disabled) units and carry no MicL self-check.
got = {name: ("MicL" in _decode(name)) for name in EVENTS}
assert any(got.values()), "expected at least one 4-channel unit"
assert not all(got.values()), "expected at least one 3-channel unit"
for name, has_mic in got.items():
if has_mic:
tr = np.asarray(_decode(name)["MicL"], dtype=float)
tr = tr - tr[:4].mean()
# mic self-check is a bipolar pulse train — swings both ways, wide range
assert tr.max() > 5000 and tr.min() < -5000, f"{name} mic not bipolar"
def test_channel_ids_and_order():
# ids decode to the canonical channel names, geo always in Tran/Vert/Long order
sc = _decode(EVENTS[0])
assert [c for c in ("Tran", "Vert", "Long") if c in sc] == ["Tran", "Vert", "Long"]
def test_returns_empty_on_non_idf_input():
assert decode_idf_sensor_check(b"not an IDF file") == {}
assert decode_idf_sensor_check(b"") == {}