feat(import): v0.16.0 - Fully implemented series 3 BW-ACH pipeline stablized. #19

Merged
serversdown merged 9 commits from ach-report-ingestion into main 2026-05-11 15:55:24 -04:00
4 changed files with 36 additions and 8 deletions
Showing only changes of commit 197c0630e2 - Show all commits
+27
View File
@@ -4,6 +4,33 @@ All notable changes to seismo-relay are documented here.
--- ---
## v0.16.0 — 2026-05-11
The "BW ACH ingestion" release. When paired with **series3-watcher v1.5.0**, every Blastware ACH event (binary + `_ASCII.TXT` report) lands in SeismoDb with device-authoritative peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data — without depending on the still-undecoded waveform body codec. This is the end-to-end product win discussed in v0.15.0's "out of scope" notes: sortable / filterable monthly-summary review of historical events, populated from the BW ASCII export rather than re-decoded samples.
### Added — `/db/import/blastware_file` rich-metadata ingestion
- **Paired BW ASCII reports.** The endpoint now accepts the `<binary>_<ext>_ASCII.TXT` partner BW writes alongside each event. Pairing handles both filename conventions: ACH (`M529LK44_AB0_ASCII.TXT`) and manual-export (`M529LK44.AB0.TXT`). When both present, ACH wins.
- **`minimateplus/bw_ascii_report.py`** (new) — parser + `BwAsciiReport` dataclass for BW's per-event ASCII export. Handles every field BW writes: identity, trigger config, per-channel PPV / ZC Freq / Time of Peak / Peak Acceleration / Peak Displacement, Peak Vector Sum + time, MicL PSPL / Time of Peak / ZC Freq, sensor self-check (Test Freq / Test Ratio / Test Amplitude / Pass-Fail per channel), monitor log, PC SW version.
- **Position-based user-notes parsing.** BW's Compliance Setup → Notes tab labels (Project / Client / User Name / Seis Loc) are *operator-editable* — an operator can rename them to "Building:", "Site Address:", etc. Rather than maintain a label-spelling map, the parser uses positional matching between the `Units :` and `Geo Range :` anchors in the ASCII output. The four canonical slots (project / client / operator / sensor_location) populate by position regardless of label; the original labels BW wrote are preserved in `report.user_note_labels` for downstream UIs (terra-view) to display verbatim.
- **`bw_report` sidecar block.** New top-level block in `.sfm.json` carrying the parsed BW report (trigger config, peaks with per-channel stats, mic block, sensor_check, monitor_log, PC SW version, operator-label labels).
- **`apply_report_to_event(event, report)` helper.** Overlays the report's device-authoritative fields onto an in-memory `Event` so `SeismoDb.insert_events()` writes correct DB columns instead of the broken-codec values from `_peaks_from_samples()`.
### Fixed — three compounding bugs that left forwarded events with garbage data
- **Import endpoint inserted under `serial="UNKNOWN"`.** `_serial_from_event(ev)` was a stub that always returned `None`; the BW-filename-decoded serial that `WaveformStore` had already resolved was never surfaced to `db.insert_events`. Now uses `rec["serial"]` as the authoritative source. `scripts/repair_unknown_serials.py` repairs existing DB rows.
- **`/db/units` ignored events from non-ACH ingest paths.** `query_units()` only aggregated from `ach_sessions` — events that arrived via `save_imported_bw()` were never visible in the fleet overview even though they populated `events` correctly. Now unions both tables.
- **Re-imports left stale DB rows.** The `IntegrityError` handler in `insert_events()` only refreshed filename / sidecar columns when a duplicate `(serial, timestamp)` arrived. Peak values, project info, sample_rate, record_type stayed locked at whatever the first (often broken-codec) insert wrote. Now the upsert path refreshes every device-authoritative column from the new data while preserving `false_trigger` and immutable fields (`id`, `created_at`).
- **Server-side TXT pairing only knew the legacy convention.** The endpoint stripped `.TXT` and looked up `<binary>` — which works for manual exports (`<binary>.TXT`) but not BW ACH (`<stem>_<ext>_ASCII.TXT`). Reports were arriving in the multipart but silently dropped. Now recognises both conventions and registers each report under all matching binary names.
### Migration
For existing deployments where events were forwarded by an older watcher (broken pairing) or imported during the UNKNOWN-bucketing window:
1. `python -m scripts.repair_unknown_serials --db <path> --apply` to re-attribute `serial="UNKNOWN"` rows.
2. Delete the watcher's `sfm_forwarded.json` state file and let it re-forward. The server's upsert path will refresh the existing DB rows with the report's authoritative values.
3. Operator review state (`false_trigger`, sidecar `review` block) is preserved across the re-import.
## v0.15.0 — 2026-05-07 ## v0.15.0 — 2026-05-07
### Added ### Added
+7 -6
View File
@@ -1,4 +1,4 @@
# seismo-relay `v0.15.0` # seismo-relay `v0.16.0`
A ground-up replacement for **Blastware** — Instantel's aging Windows-only A ground-up replacement for **Blastware** — Instantel's aging Windows-only
software for managing MiniMate Plus seismographs. software for managing MiniMate Plus seismographs.
@@ -14,11 +14,12 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
> byte-perfect against Blastware captures across 2-sec, 3-sec, and 10-sec > byte-perfect against Blastware captures across 2-sec, 3-sec, and 10-sec
> events.** Generated `.G10` / `.AB0` files open cleanly in Blastware with > events.** Generated `.G10` / `.AB0` files open cleanly in Blastware with
> full Event Reports, frequency analysis, and waveform plots. > full Event Reports, frequency analysis, and waveform plots.
> **v0.15.0 (2026-05-07)** adds layered per-event storage (BW binary + > **v0.16.0 (2026-05-11)** adds BW ASCII report ingestion to
> raw 5A pickle + HDF5 + `.sfm.json` sidecar), a plot-ready > `/db/import/blastware_file` — paired with **series3-watcher v1.5.0**,
> `sfm.plot.v1` JSON shape with server-side ADC-to-physical-units > every Blastware ACH event lands in SeismoDb with device-authoritative
> conversion, and a BW-file importer for ingesting externally-produced > peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data,
> events. See [CHANGELOG.md](CHANGELOG.md) for full version history. > without depending on the still-undecoded waveform body codec.
> See [CHANGELOG.md](CHANGELOG.md) for full version history.
--- ---
+1 -1
View File
@@ -47,7 +47,7 @@ SIDECAR_KIND = "sfm.event"
# bumped without a `pip install` re-run — leading to confusing stale # bumped without a `pip install` re-run — leading to confusing stale
# version stamps in sidecars. Bump this constant and CHANGELOG.md # version stamps in sidecars. Bump this constant and CHANGELOG.md
# together at release time. # together at release time.
TOOL_VERSION = "0.15.0" TOOL_VERSION = "0.16.0"
try: try:
# Best-effort: prefer the installed metadata when it's NEWER than the # Best-effort: prefer the installed metadata when it's NEWER than the
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "seismo-relay" name = "seismo-relay"
version = "0.15.0" version = "0.16.0"
description = "Python client and REST server for MiniMate Plus seismographs" description = "Python client and REST server for MiniMate Plus seismographs"
requires-python = ">=3.10" requires-python = ">=3.10"
dependencies = [ dependencies = [