serversdownandClaude Opus 4.8
1765b3300d
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
2094 lines
139 KiB
Markdown
2094 lines
139 KiB
Markdown
# Changelog
|
||
|
||
All notable changes to seismo-relay are documented here.
|
||
|
||
---
|
||
|
||
## Unreleased
|
||
|
||
### Fixed
|
||
|
||
- **Waveform event times were the monitoring-session start, not the trigger
|
||
(~hours off).** `read_blastware_file` stamped events with footer `ts1`, which
|
||
for a waveform is the session start a unit shares across every event that day
|
||
(a unit arming at 06:00 stamped 06:00 on all of them — the modal and PDF both
|
||
showed it, since it's the stored value). The event time is footer `ts2` (the
|
||
recording stop), and Blastware's trigger = `ts2 - record time`. The record
|
||
time is a big-endian float32 in the recording-setup config block (30 bytes
|
||
before the `Standard Recording Setup` marker), so the **exact trigger is now
|
||
recovered from the binary alone** — all 7 BE12844 oracle events decode to
|
||
their exact Blastware time (e.g. N844LQHB 10:33:29), no paired `.TXT` needed.
|
||
Histograms keep `ts1` (the ~24 h window start). A paired report's
|
||
`event_datetime` stays authoritative (unit-clock drift).
|
||
⚠ **Needs a re-decode backfill** to correct existing stored events' timestamps.
|
||
|
||
### Added
|
||
|
||
- **Diagnostics tab in the SFM standalone webapp.** Surfaces the device
|
||
endpoints that previously existed only as `curl`: `events/storage_range` and
|
||
`events/index` alongside `monitor/status`, then stop monitoring, disable ACH
|
||
(`rescue?erase=false`, so stored events survive), and erase. The wedged-unit
|
||
ladder — slow drip and blind stop — sits under its own heading pointing at
|
||
`docs/runbooks/wedged_unit_recovery.md`, with the reminder that `slow_drip`'s
|
||
success signal is `bytes_received > 0` and not a clean duration. Erase is
|
||
guarded by typing the unit's serial: auth answers *who*, not *did you mean
|
||
it*, and Swagger's try-it-out button on `/device/events/erase` is live on
|
||
`:8200/docs`.
|
||
|
||
- **`docs/sfm_tool_status.md`** — an honest per-capability maturity assessment:
|
||
what is production-grade (the codec library, the data side), what is
|
||
emergency-grade (the device side), what is a research artifact, the
|
||
known-issues table, and the gap to a real tool. Also records the **5A
|
||
page-boundary bug** as known: `parse_strt_end_offset()` discards the key's
|
||
page byte, so once a unit has recorded more than 64 KB since its last erase,
|
||
an event spanning the boundary reads an `end_offset` *behind* its own start —
|
||
the chunk loop fetches nothing and TERM packs a negative `offset_word`, which
|
||
500s. Reproduced on BE12599. Production is unaffected: it ingests complete
|
||
files via the watcher path and never runs this walk.
|
||
|
||
### Changed
|
||
|
||
- **Connecting to a unit no longer walks its event chain.** `/device/events`
|
||
reads every event header over the cellular link; on a unit with a large or
|
||
wrapped chain that takes minutes or fails outright, and it fired
|
||
automatically on every connect. Connect now uses only ~2 s probes —
|
||
`/device/info` (which already carried the compliance config the walk was
|
||
re-reading) plus `events/storage_range` — and the Device tab gains an Event
|
||
Chain card. The walk moved behind a **Load events** button in the Events
|
||
toolbar. Knowing whether a unit's ACH is on no longer requires reading every
|
||
event it has stored.
|
||
|
||
### Migration
|
||
|
||
**None.** Frontend and documentation only — no codec, waveform-store or DB
|
||
change, no schema change, and no `TOOL_VERSION` bump. The webapp is served
|
||
from the image, so the change appears after the next `sfm` rebuild.
|
||
|
||
---
|
||
|
||
## v0.31.0 — 2026-09-18
|
||
|
||
**Report parity, and a second way to rescue a runaway unit.** Two threads.
|
||
|
||
The first closes out Blastware Event/FFT-Report parity: the FFT, the USBM
|
||
RI8507 compliance chart and the sensor self-check now render on the event
|
||
report, reverse-engineered against BE12844 (MiniMate Plus) and UM (Thor)
|
||
events. The sensor check is decoded for **both** series and standardized into
|
||
the `.h5` (schema **v2**, a new `/sensor_check` group), so SFM serves it
|
||
device-agnostically rather than decoding at report time. The Inspector — an
|
||
annotated hex reader for series-3 binaries — is what made the trailing-block
|
||
structure findable, and it earned its keep by *ruling out* a stored FFT block
|
||
and proving Blastware computes it from the samples.
|
||
|
||
The second came out of a field emergency. BE12599's connector fault drove its
|
||
Tran channel to its trigger level, so the unit recorded back-to-back and dialed
|
||
the office ACH server every ~75 s, unreachable the whole time.
|
||
`bridges/ach_server.py` gained `--stop-monitoring` / `--disable-ach` /
|
||
`--rescue`, which **invert** the recovery: instead of racing a Stop into the
|
||
gaps between dial-outs, point the modem's Destination at our own ACH server and
|
||
answer the call. Proven in production the same night — the stop landed on the
|
||
first call-in and held. See `docs/runbooks/wedged_unit_recovery.md`.
|
||
|
||
⚠ **This release owes prod a backfill** — see Migration below.
|
||
|
||
### Added
|
||
- **Rescue-on-connect for `bridges/ach_server.py`** — `--stop-monitoring`
|
||
(SUB 0x97), `--disable-ach` (SUB 0x2C read → 0x7E write → 0x7F confirm) and
|
||
`--rescue` (both). They fire immediately after the startup handshake and
|
||
**before** the event walk, so a unit that is recording back-to-back on a
|
||
stuck-triggered geophone is quieted as early in the session as possible.
|
||
Each action is independently guarded — a failure does not abort the download
|
||
— and the outcome is written to `rescue.json` in the session directory.
|
||
|
||
This inverts the `docs/runbooks/wedged_unit_recovery.md` approach. That
|
||
runbook reaches the unit *inbound* and clears the modem's Destination Address
|
||
to stop it dialing. When the device is instead wedged mid-modem-init — ALEOS
|
||
logs `tcpmode trying to send to invalid socket` and re-runs `Initialize Auto
|
||
answer` every ~75 s, orphaning any held inbound session — inbound cannot win.
|
||
Pointing the modem's Destination at an `ach_server` and letting the unit call
|
||
*us* gives a device-initiated session the modem bridges properly.
|
||
|
||
⚠ Prefer `--stop-monitoring` alone on first contact. `--disable-ach` stops
|
||
the unit calling, which is the only channel to a unit in this state; stopping
|
||
the recording ends the call-home loop on its own when ACH is
|
||
"after event recorded".
|
||
|
||
- **Blastware-compatible channel FFT (`waveform_fft`).** Reproduces Blastware's
|
||
FFT Report: DC-removed, no window, zero-padded to 4096 (0.25 Hz bins at
|
||
1024 sps), single-sided `2/N` amplitude. Matches Blastware's dominant
|
||
frequency to the exact bin and the amplitude to report precision across all
|
||
28 channels of the 7-event BE12844 oracle set. `channel_spectrum()` /
|
||
`dominant_frequency()`; tests in `tests/test_waveform_fft.py`.
|
||
|
||
- **USBM RI8507 / OSMRE compliance chart on the event-report PDF
|
||
(`sfm/compliance.py`).** The velocity-vs-frequency blasting-compliance
|
||
scatter Blastware draws in the upper-right of its Event Report: each channel's
|
||
significant cycles as `(frequency, peak velocity)` points (zero-crossing
|
||
method, so each channel's cloud tops out at its PPV) plotted against the
|
||
RI8507 Drywall (0.75 in/s) and plaster (0.50 in/s) limit curves, drawn
|
||
continuous (constant-displacement bounds meeting the plateaus — no vertical
|
||
steps). Sized and positioned to match a Blastware report, measured off the
|
||
reference PDF. A technical breakdown of the curve is in
|
||
`docs/ri8507_compliance_curve.md`.
|
||
|
||
- **Sensor self-check waveforms decoded and drawn — both series.** The little
|
||
"Sensor Check" traces (geophone ring-downs — the transducer's damped impulse
|
||
response — plus a MicL pulse train, the mic's known-signal gain check) are the
|
||
unit's proof its sensors were healthy when it recorded the event.
|
||
- **Series-3** (`minimateplus.sensor_check`): four records (`0x3c`–`0x3f`) in
|
||
the binary's trailing block, same delta-block codec as the main waveform.
|
||
Verified against all 7 BE12844 reports (mic zero-crossing = 20.1 Hz exact;
|
||
geophone ring-downs ~7.5 Hz, overswing ~3.5).
|
||
- **Series-4** (`micromate.sensor_check`): the same self-test in the Thor IDFW
|
||
fixed header — four `01 0e 3c/3d/3e/3f` records (same channel ids) storing
|
||
raw int16 traces; three-channel (mic-disabled) units carry only the three
|
||
geophones. Validated by shape + cross-event consistency.
|
||
- **Standardized into the `.h5`** (`/sensor_check`, schema v2): each series'
|
||
decoder attaches the traces to the event at decode, the writer persists
|
||
them, and `gather_report_data` reads them back — so SFM renders the strip
|
||
(flush against the waveform panel) plus the **Sensor Check → Frequency /
|
||
Overswing Ratio** sub-rows without knowing the source instrument.
|
||
- Tests: `tests/test_sensor_check.py`, `tests/test_sensor_check_idf.py`,
|
||
`tests/test_event_hdf5_sensor_check.py`.
|
||
|
||
- **Inspector tab in `seismo_lab.py` — annotated hex reader for series-3
|
||
binaries (`minimateplus/binary_annotate.py`).** Tiles a raw Blastware file
|
||
into labeled spans (header / STRT / body record-chain / trailing metadata +
|
||
calibration + sensor-check records / footer) so a binary can be combed by eye.
|
||
|
||
### Fixed
|
||
- **Event-report waveform panel — stacked-lane y-tick collision.** The lanes
|
||
touch, so each lane's bottom `-1.0` overprinted the next lane's top `1.0` at
|
||
the shared boundary. Prune the extreme ticks so each lane shows clean interior
|
||
ticks only.
|
||
- **Event-report header — serial+firmware line ran off the page.** The long
|
||
`BE##### V ##.##-#.## MiniMate Plus` string overflowed the right margin;
|
||
tighter right-column indent + BW's slightly smaller header size so it fits.
|
||
|
||
---
|
||
|
||
### Migration
|
||
|
||
⚠ **The sensor-check needs a backfill.** Existing `.h5` files are schema v1
|
||
and carry no `/sensor_check` group, so their reports show no sensor-check strip
|
||
until regenerated. `TOOL_VERSION` is bumped to **0.31.0**, so the standard
|
||
backfill regenerates every event and picks up the traces with **no `--force`**:
|
||
`scripts/backfill_thor_events.py` for series-4 (it already owed a v0.30.0 Thor
|
||
backfill — this rides along) and the series-3 sidecar/shape backfill for
|
||
MiniMate events. Purely additive — no decoded value changes, and v1 `.h5`
|
||
files read fine until then (empty strip). DB backup first, as always.
|
||
|
||
⚠ Budget **~2 h on the NAS** — ~1.5 files/sec there versus ~85/sec on the dev
|
||
box (gzip-4 in `sfm/event_hdf5.py` against a Synology CPU).
|
||
|
||
Everything else in this release owes nothing: the FFT, the USBM compliance
|
||
chart and the `ach_server` rescue flags are additive and read data already on
|
||
disk — no schema change, no DB migration.
|
||
|
||
---
|
||
|
||
## v0.30.0 — 2026-09-12
|
||
|
||
**The series-4 correctness release** — the Thor / Micromate counterpart to
|
||
v0.26.0's series-3 work. The decoder is now verified per-sample against
|
||
Thor's own CSV exports: **459 waveform files, 3,807,158 / 3,807,165 samples
|
||
exact** across three independent ground-truth corpora, and production IDFW is
|
||
**575/575** with zero truncations and zero decode failures. Series-3
|
||
re-verified **unchanged at 14,338/14,338** after every shared-codec change.
|
||
|
||
⚠ **This release owes the prod store a Thor backfill.** Every stored
|
||
series-4 geophone value is **3.3% low**, and histogram peaks from monitoring
|
||
runs longer than ~4 hours can be far worse (the interval cap discarded the
|
||
tail, frequently the part holding the peak). Run
|
||
`scripts/backfill_thor_events.py` — `TOOL_VERSION` is bumped to `0.30.0`, so
|
||
regeneration is gated correctly and **no `--force` is needed**. DB backup
|
||
first. Series-3 events are untouched by this release and do not need
|
||
re-running.
|
||
|
||
⚠ **Terra-View displays these values.** Series-4 geophone readings will rise
|
||
~3.3% after the backfill, and some histogram PPVs will rise a great deal more.
|
||
That is a correction, not a regression.
|
||
|
||
|
||
### Fixed — event-report PDF used a per-trace geo Y scale
|
||
|
||
The waveform plot scaled each geo lane to its own peak, so a small channel
|
||
filled its lane and looked as large as a big one, and the `Geo: X in/s/div`
|
||
footer reflected only whichever channel was measured first — wrong for the
|
||
other two. All three geo lanes now share one symmetric scale (max |sample|
|
||
across them, padded, 0.05 in/s floor), matching the event modal and BW's
|
||
single amp/div; the footer reflects that shared scale. Mic keeps its own psi
|
||
scale. Large events are unchanged.
|
||
|
||
### Fixed — series-4 (Thor / Micromate) decoder is now per-sample exact
|
||
|
||
Verified against **Thor's own CSV exports**, which carry a per-sample
|
||
four-column block beside every binary (`CSV/<name>.IDFW.csv`) — 1,012 paired
|
||
files that had been sitting in the corpus unused. Previous notes asserted
|
||
"Thor has no ASCII ground truth", which is why the decoder stayed pinned to a
|
||
superseded walker with an unverifiable scale factor.
|
||
|
||
| metric | before | after |
|
||
|---|---|---|
|
||
| IDFW per-sample exact | 39.1% | **100.000%** (1,057,536/1,057,536) |
|
||
| IDFW files fully exact | 0/153 | **153/153** |
|
||
| IDFW PPV median error | −3.32% | **−0.002%** |
|
||
| IDFH within 2% of Thor PPV | 51.1% | **100.0%** (858/858) |
|
||
| prod IDFW PPV median error (8 units) | −3.3% | **−0.001%** |
|
||
| decode cost | — | 6 ms/file |
|
||
|
||
Four independent root causes:
|
||
|
||
- **Geo LSB was `0.0003`, should be `0.000310308`** — the old value was Thor's
|
||
4-decimal *display rounding* of the LSB mistaken for the LSB, so every
|
||
series-4 geophone sample read **3.3% low**. Pinned to ±6e-11 by
|
||
intersecting 991,415 rounding constraints; corroborated by the ±full-scale
|
||
seed (`±32226`) in unwritten IDFH slots. Applies to IDFH too, which had a
|
||
separate (also wrong) `10.0/32768`.
|
||
- **IDFH histograms were capped at 250 intervals** — the segment validator
|
||
required the interval counter's high byte to be zero, but the counter is a
|
||
uint16 cumulative index, so every segment past interval 255 was rejected.
|
||
Any run over ~4 hours lost its tail, often the part holding the peak.
|
||
540/858 corpus files affected.
|
||
- **Record mode `00 00` (raw int16, 10-byte header) was unhandled** — the
|
||
record fell through the dispatch, silently dropping each channel's first
|
||
512 samples. This produced the long-standing "loud events truncate"
|
||
symptom. `MODE_ABSOLUTE` is now also accepted as a segment-0 preamble.
|
||
- **Body-offset search matched `00 02 00` inside record headers** — picking a
|
||
candidate part-way down the chain, which decodes a rotation-shifted body
|
||
that drops each channel's segment 0. The search now anchors on record
|
||
headers and takes the chain head.
|
||
|
||
Also fixes the separately-tracked "UM-series decodes ~1000× low" bug
|
||
(`UM11402_20260406130113.IDFW` now matches its device report exactly).
|
||
|
||
Series-3 re-verified **unchanged at 14,338/14,338 exact** after the shared
|
||
`waveform_codec` change.
|
||
|
||
⚠ **This is a codec change: the Thor store owes a regeneration.** Run
|
||
`scripts/backfill_thor_events.py` (bump `TOOL_VERSION` first, or pass
|
||
`--force`), DB backup first. All stored series-4 `.h5`/sidecar peaks are
|
||
currently ~3.3% low, and histogram peaks for runs over ~4 hours may be
|
||
badly low.
|
||
|
||
⚠ **Thor's histogram PPV has a 0.0050 in/s display floor** — 41.4% of prod
|
||
IDFH sidecars report a component PPV larger than their own vector sum. On
|
||
quiet files the decoder is now *more* accurate than that reference.
|
||
|
||
New: `scratch/verify_thor_against_csv.py`, `tests/test_idf_binary_codec.py`
|
||
(10 tests, fixtures under `tests/fixtures/thor-idf/`).
|
||
|
||
### Fixed — mic-disabled (3-channel) units
|
||
|
||
Verified on a second corpus (`9-10-26-csv-req`: UM11402, UM12947, UM20147) —
|
||
**139/139 waveforms per-sample exact (1,273,380 samples), 877/877 histograms
|
||
within 2%** (was 66.9% and 56.6%).
|
||
|
||
- **Waveform body head sat below the scan floor.** A 3-channel unit's shorter
|
||
header puts the record chain head at `0x0dba`, under the old
|
||
`_BODY_SCAN_FLOOR` of `0x0E00`. The scan couldn't see it and fell through
|
||
to the Vert segment-0 record, decoding a body shifted one position around
|
||
the channel rotation — Vert came up exactly 512 samples short. Floor
|
||
lowered to `0x0C00`; body-offset scoring now accepts 3 channels as "equal"
|
||
instead of demanding 4.
|
||
- **Histogram interval record is 56 bytes, not 72.** It is
|
||
`16 × n_channels + 8`, so mic-disabled units pack 56. Assuming 72 read 7
|
||
intervals out of every 10-interval segment then walked off alignment into
|
||
garbage decoding as ~10 in/s peaks (errors up to +191,000%). The interval
|
||
count now comes from the segment's cumulative counter and the stride is
|
||
derived from it; also recovers 4 files that decoded no intervals at all.
|
||
|
||
Combined across both corpora: **292/292 waveform files, 2,330,916/2,330,916
|
||
samples exact.** Production IDFW truncations 41 → 22.
|
||
|
||
### Fixed — `40 NN` int16 blocks with NN > 8
|
||
|
||
`data_block_len()` rejected any `40 NN` block with `NN > 0x08`. The cap had
|
||
no evidence behind it: every corpus available when it was written used only
|
||
NN ∈ {1,2,3,4,8}, so it was never exercised. Loud UM12947 events use NN of
|
||
12, 16, 20 … up to 196, and because the block walker stops at the first
|
||
unrecognised tag rather than raising, rejecting them surfaced as **silently
|
||
short channels** (e.g. Tran 1812 / Vert 2132 / Long 2324 on a file whose
|
||
export has 2324 for all three). The bound is the buffer, not a constant.
|
||
|
||
Verified against Thor exports for UM12947 (2025-07-14 … 09-25, 167
|
||
waveforms): length mismatches **22 → 0**, **1,476,242/1,476,249** samples
|
||
exact. These are not truncated recordings — the exports carry full sample
|
||
counts.
|
||
|
||
`tests/test_waveform_codec.py` asserted the cap as intended behaviour; that
|
||
assertion was wrong and has been replaced with one pinning the opposite,
|
||
carrying the evidence.
|
||
|
||
### Result across all three ground-truth corpora
|
||
|
||
**459 waveform files, 3,807,158 / 3,807,165 samples exact.** Production
|
||
IDFW: **575/575**, zero truncations, zero decode failures, median PPV error
|
||
−0.0007% across 8 units. Series-3 re-verified **unchanged at 14,338/14,338**
|
||
after every shared-codec change.
|
||
|
||
The 7 residual samples each differ by one 4th-decimal tick and are **Thor's
|
||
own rounding**: intersecting the per-sample rounding constraints over that
|
||
corpus is infeasible (the binding pair contradict by 2.3e-11, 7e-5 relative),
|
||
so no single linear LSB reproduces every printed value. `_GEO_LSB_IPS` is
|
||
already pinned to ~1e-11 — do not retune it to chase these.
|
||
|
||
---
|
||
|
||
## v0.29.0 — 2026-09-04
|
||
|
||
First release to reach prod since **v0.27.0**, so it ships **both** the
|
||
`false_trigger_reason` column below *and* the v0.28.0 offset (DC-baseline)
|
||
detector: v0.28.0 was version-bumped in-tree (`TOOL_VERSION`, CHANGELOG) but
|
||
never tagged or deployed, so 0.29.0 is the first build to carry either to prod.
|
||
Pairs with Terra-View ≥ 0.24.0. The `false_trigger_reason` column auto-migrates
|
||
on startup; the offset detector still needs the shape backfill on the prod store
|
||
(`scripts/backfill_event_shape.py`) to populate `shape_offset*` on existing rows.
|
||
|
||
### Added
|
||
- **`events.false_trigger_reason` — optional FT cause.** A nullable `TEXT`
|
||
column recording *why* an event is a false trigger (e.g. `"offset"`), as a
|
||
subtype of the FT flag: setting a reason via the sidecar review PATCH implies
|
||
`false_trigger=1`, and the reason is cleared whenever FT ends up 0
|
||
(confirm-real, clear-FT, `set_false_trigger(false)`). `propagate_review_to_twins`
|
||
carries the reason to the histogram/waveform twin alongside the flag.
|
||
Auto-migrated (`_SCHEMA` + `_migrate` ADD COLUMN — not the Migration-1
|
||
rebuild); exposed via `/db/events`. Terra-View surfaces it as a manual
|
||
"Flag as offset" action + an `FT · offset` badge.
|
||
|
||
### Fixed
|
||
- **BlastMate serials — the family prefix is read from the file, not guessed.**
|
||
The Blastware filename encodes only the serial *number* (`L895…` → 10895);
|
||
the two-letter prefix is not in it. `waveform_store` synthesised `"BE"`, so
|
||
an imported **BlastMate** (serials `BA…`) was filed under a MiniMate Plus
|
||
serial that does not exist — silently, and Terra-View read it straight
|
||
through. `save_imported_bw` now resolves serial as hint → file body →
|
||
filename guess, via a new `_serial_from_bw_bytes` that accepts a candidate
|
||
only when its numeric part matches the filename. `client._decode_0a_partial_header`
|
||
likewise matched a literal `b"BE"` in monitor-log partial records; on a
|
||
BlastMate that returned −1 and skipped the whole block, losing the **geo
|
||
threshold** along with the serial. It now matches any two-letter prefix and
|
||
requires the NUL terminator — stricter than the search it replaces.
|
||
|
||
BlastMate is the MiniMate Plus's larger Series III sibling and its files are
|
||
byte-compatible: all 1,493 in the DL2 archive decode through the existing
|
||
codec at 100%, same four channels. **The serial string was the only thing
|
||
blocking BlastMate support in SFM.** Four archive units were affected —
|
||
BA9229, BA10060, BA10895, BA15957.
|
||
|
||
**No backfill and no `TOOL_VERSION` bump**: this changes which serial an
|
||
*import* is filed under, not any decoded value, so existing sidecars and
|
||
`.h5` files are untouched. **No migration either** — prod holds no BlastMate
|
||
events (the archive's BA units last recorded 2018-10 through 2023-11; the
|
||
prod backfill reaches back only to ~May 2025).
|
||
|
||
---
|
||
|
||
## v0.28.0 — 2026-09-02
|
||
|
||
**Offset (DC-baseline) false-trigger detector.** Productionizes the validated
|
||
pre-trigger detector: a geophone event whose baseline sits off zero and stays
|
||
flat across the record (sensor bumped / settled / drifted) is now flagged and
|
||
surfaced in Terra-View as an `offset` false-trigger reason — catching offsets the
|
||
crest/near-peak spike rule misses (an offset is low-crest and flat).
|
||
|
||
### Added
|
||
- `shape_metrics.offset_from_samples` / `offset_from_h5`: per geophone channel,
|
||
`|median(pre-trigger)| ≥ 0.025 in/s` AND `pre/mid/end spread ≤ 0.02` → offset;
|
||
the consistency test rejects transients (a real event moves one third). Reads
|
||
the `.h5` samples + the `pretrig_samples` attr, range-aware via the in/s float
|
||
samples. Constants `OFFSET_FLOOR` / `OFFSET_MAX_SPREAD` are tunable.
|
||
- `events.shape_offset` / `shape_offset_axis` / `shape_offset_pre` /
|
||
`shape_offset_spread` columns (auto-migrated: `_SCHEMA` + the `_migrate`
|
||
ADD COLUMN loop), computed at all three ingest paths and by
|
||
`backfill_event_shape.py`, exposed via `/db/events`.
|
||
|
||
Requires the shape/offset backfill on the prod store to populate existing events:
|
||
`python scripts/backfill_event_shape.py --db-path … --store-root …`.
|
||
|
||
---
|
||
|
||
## v0.27.0 — 2026-08-28
|
||
|
||
**Per-sample decoder verification at scale, plus the offset investigation.**
|
||
The series-3 codec is now verified sample-by-sample against **14,338** preserved
|
||
Blastware ASCII exports — 1,249 waveform and 13,089 histogram, spanning 45 units
|
||
and files back to 2018. That is 11x the ground truth the production store
|
||
carried, and it found one real codec bug (below).
|
||
|
||
### Fixed
|
||
- **Sub-minute histograms with a partial final block decoded to nothing**
|
||
(`histogram_codec.detect_multi_interval_stride`). The stride search confirmed
|
||
itself on a third block header whenever the body was long enough to hold one —
|
||
but a body can exceed two strides and still contain only two real blocks, because
|
||
a *partial* final block leaves trailing padding. BE18193 `T193L0XM.CI0H` (51
|
||
intervals at 2 s = one full 30-interval block plus a 21-interval remainder, in a
|
||
2787-byte body) therefore had its correct stride of 612 discarded and produced an
|
||
empty decode. A missing third header now means end-of-stream rather than
|
||
disqualification; the block-counter check, which is what actually prevents the
|
||
false positives that once mis-dispatched 9,082 files, is unchanged.
|
||
|
||
Found by decoding the full DL2 archive against its preserved Blastware ASCII
|
||
exports. Across **63,535 unique** histogram binaries the fix recovers **4 files** —
|
||
`K440HJCN.3C0H` and `K557IF1U.8K0H` (stride 252), `T191HVNP.0S0H` (92) and
|
||
`T193L0XM.CI0H` (612) — with **zero** files regressed. Verification over all
|
||
14,340 archive pairs goes 14,337 → 14,338 exact, the only remainder being two
|
||
series-4 IDF files that belong to a different codec.
|
||
|
||
(The DL2 export keeps a byte-identical `Sent/` mirror of its root, so a naive
|
||
walk double-counts every binary — 127,035 paths are 63,535 distinct files. The
|
||
ASCII exports are *not* mirrored, so the 14,340 pair count is already distinct.)
|
||
|
||
**No prod backfill is required for this.** Verified after the fact: all four
|
||
recovered files are archive-only — none exists in the production store or the
|
||
events DB — and re-running stride detection over the production store's
|
||
**10,215** histogram binaries shows **0 files whose decode changes**. The fix
|
||
matters for future ingests of sub-minute histograms with a partial final block,
|
||
not for anything already stored.
|
||
|
||
(`TOOL_VERSION` moves with the release, so whenever a backfill *is* next run for
|
||
some other reason it will regenerate the whole store rather than skipping. That
|
||
is harmless — the output is byte-identical for every currently-stored file — but
|
||
it means the run takes its full ~2 hours on the NAS.)
|
||
|
||
- **Histogram/waveform twin matching is now interval-based** (`find_twins`). A real
|
||
trigger is recorded twice — as a triggered waveform (stamped at the trigger instant)
|
||
and inside the scheduled histogram whose interval contains it (stamped at the 7am/7pm
|
||
interval start) — so the two twins can be **hours apart**. The old ±5-minute window
|
||
silently missed them, which broke review propagation (flagging one twin didn't flag its
|
||
twin). Twins are now matched by same serial + identical `peak_vector_sum` + opposite
|
||
record type + the waveform falling within the histogram's interval (bounded by the next
|
||
same-serial histogram). `window_seconds` is retained but ignored. Fixes terra-view #102
|
||
sub-task 2.
|
||
|
||
- **`/health` reported a hard-coded `0.1.0`** instead of the real service version.
|
||
`sfm/server.py` now derives its version from `minimateplus.event_file_io.TOOL_VERSION`,
|
||
making that constant the single source of truth for the service version and the
|
||
sidecar stamp alike — one place to bump at release.
|
||
|
||
- **`CLAUDE.md` had 793 NUL bytes appended** after its last line, which made `grep`
|
||
treat the file as binary and silently skip it. Present since at least v0.21.0.
|
||
Stripped.
|
||
|
||
### Added
|
||
- **`docs/offset_investigation.md`** — a dated journal of the "offset" hardware
|
||
fault: base rate, detector design, per-unit case files, ruled-out hypotheses
|
||
(each kept with the evidence that killed it), and Instantel's own autozero
|
||
procedure with its 2027–2069 acceptance window.
|
||
- **`scratch/verify_against_ascii.py`** — decodes a corpus of BW binaries and
|
||
diffs every sample against the paired `_ASCII.TXT`. Includes a saturation
|
||
carve-out: BW clamps clipped events to the range maximum and writes `OORANGE`,
|
||
while the decoder faithfully reports counts past nominal full scale.
|
||
- **`scratch/offset_scan3.py`** — offset detector. Measures the resting floor in
|
||
the *pre-trigger* window (definitionally quiet) and requires it to hold across
|
||
pre / middle / end. Result: **5 of 45 units (11%)**, stable across a 2x
|
||
threshold range. Supersedes `offset_scan.py` and `offset_scan2.py`, both kept
|
||
as the reasoning trail.
|
||
|
||
### Verified
|
||
- **19,244 healthy channel-events sit at a pre-trigger floor of exactly 0.000
|
||
(62.7%), 94.5% within ±1 quantisation unit, median +0.0000.** No systematic
|
||
zero-point bias in the decoder — an independent confirmation of the
|
||
32000-count geo full scale, arrived at from a different direction than the
|
||
ASCII sample comparisons.
|
||
|
||
---
|
||
|
||
## v0.26.0 — 2026-08-27
|
||
|
||
**Series-3 decode correctness.** Two body-model rewrites, a systematic
|
||
scale error affecting every geophone reading ever produced, a recovered
|
||
file format, and two artifact-hygiene bugs where stale files outlived the
|
||
decodes that made them. All 11,603 series-3 binaries in the production
|
||
snapshot now pass every check.
|
||
|
||
### Fixed
|
||
|
||
- **Series-3 health sweep: 11,603 / 11,603 binaries now clean on every check.**
|
||
Swept every series-3 file with the live decoder against five independent
|
||
checks — decode exceptions, zero samples, unequal geo channel lengths, peaks
|
||
above range full scale, decoded peak vs the device-reported PPV, and waveform
|
||
length vs the declared record time. Three real defects surfaced and were
|
||
fixed:
|
||
|
||
- **`block[22]` is not a constant and must not be tested.** It was documented
|
||
as always `0x00` but carries data on loud blocks, and rejecting those threw
|
||
away the interval holding the event peak.
|
||
`BE18350/T350L7HR.NL0H` block 92 has `block[22]=0x26` and a Tran peak of
|
||
`0x0563` = 1379 counts = **6.895 in/s** — exactly the device-reported PPV —
|
||
while the file as a whole decoded to 0.015 in/s. `block[0]==0x00`,
|
||
`block[4]==0x0A` and the 4-byte tail are six bytes of constraint, which is
|
||
what keeps trailer content out.
|
||
|
||
- **Block-model dispatch now goes on signature strength, not on whichever
|
||
decoder returns first.** A multi-interval body also yields scattered
|
||
standard-tail blocks by coincidence; dispatching on "first non-empty"
|
||
handed 193 BE18193 files to the standard walker and produced peaks of
|
||
149 in/s against a 10 in/s full scale.
|
||
|
||
- **Multi-interval stride detection requires the block counter to increment
|
||
by exactly 1.** Without it the detector false-positives on ordinary
|
||
standard-block bodies: those carry a header every 32 bytes, and
|
||
`192 = 12 + 20×9` and `512 = 12 + 20×25` are both multiples of 32, so a
|
||
stride "fits" while actually skipping 6 or 16 real blocks. That misrouted
|
||
9,082 files.
|
||
|
||
Partial-block garbage is now trimmed within the final block only, stopping at
|
||
the first slot with a non-zero tail word or a geo peak above full scale.
|
||
Trimming purely from the end left garbage stranded behind one slot that
|
||
happened to have a zero tail word; trimming on the tail word alone truncated
|
||
four BE9440 files by up to 2,800 intervals.
|
||
|
||
- **Sub-minute histogram intervals are packed several to a block — 415 files
|
||
recovered.** The device always writes one minute of data per block, so a
|
||
shorter interval just means more intervals in a longer block:
|
||
|
||
| interval | intervals/block | stride |
|
||
|---|---|---|
|
||
| 1 min | 1 | 32 (the standard block) |
|
||
| 15 s | 4 | 92 |
|
||
| 2 s | 30 | 612 |
|
||
|
||
`stride = 12 + n * 20`. Each 20-byte record carries 8 × uint16
|
||
**little**-endian values — peak and half-period per channel — plus a 2-word
|
||
tail whose first word is `0000` on every real interval (a session ending
|
||
mid-block leaves buffer garbage in the remaining slots, which decoded as
|
||
peaks thousands of times the real value until that check was added).
|
||
**The standard 32-byte block is big-endian; this variant is not.**
|
||
|
||
These 415 files (216 on BE18193 at 2 s intervals, 199 on BE9440 at 15 s)
|
||
previously decoded to nothing at all — and before that were being accepted
|
||
by the *waveform* codec, which returned garbage peaking up to 400× the
|
||
device-reported PPV.
|
||
|
||
Ground truth `BE9440/K440L3AQ.T70H` — 5,710 intervals — matches its
|
||
Blastware ASCII export on **17,130/17,130** geo peaks, **22,840/22,840**
|
||
frequencies and **5,710/5,710** mic dB(L) values. Across all 455 affected
|
||
files, **1,354/1,365 (99.2%)** channel peaks match the device-reported PPV;
|
||
the 11 that don't are under-reads on BE9440 where the walk stops early.
|
||
|
||
- **`backfill_sidecars.py` now removes a stale `.h5` when nothing decodes.**
|
||
It previously skipped the write "so we don't replace whatever's there with an
|
||
empty placeholder", which silently preserved output from a superseded
|
||
decoder. After the record-chain fix, 415 histogram files stopped decoding (an
|
||
unmapped block variant on BE18193 and BE9440) but kept `.h5` files whose peaks
|
||
ran up to **400× the device's own reported PPV** — garbage feeding the charts
|
||
and the false-trigger detector with nothing marking it. Reports a
|
||
`stale_h5_removed` count.
|
||
|
||
- **The series-3 waveform body is a RECORD CHAIN, not a tag stream — this
|
||
supersedes the segment-header model, including the fixes made earlier the
|
||
same day.**
|
||
|
||
Records are self-delimiting. `off+2` is a `uint16 BE` length and
|
||
`next_record = off + 2 + len`; the chain ends on a record whose `chan_id` is
|
||
`0x06`. `off+8` carries a 3-valued mode enum:
|
||
|
||
| mode | header | data section |
|
||
|---|---|---|
|
||
| `02 00` | 14 B | anchors, then **cumulative deltas** |
|
||
| `01 00` | 10 B | no anchors, **absolute** values |
|
||
| `00 03` | 10 B | **no tags at all** — raw 12-bit packed absolute |
|
||
|
||
**`40 NN` is an ordinary int16 BE data block** (`2*NN + 2`), never a segment
|
||
header. Reading it as a `2*NN + 16` header is what made walks drift — and the
|
||
"variable-prefix segment descriptors" reported earlier today were not a format
|
||
feature at all, just walker drift of exactly
|
||
`4 - (old_stop - true_record_start)` on all 25 affected files.
|
||
|
||
Measured against the production snapshot:
|
||
|
||
| | before | after |
|
||
|---|---|---|
|
||
| all four channels equal length | 156 / 1388 | **1388 / 1388** |
|
||
| ASCII sample-count exact | 72 / 75 | **75 / 75** |
|
||
| ASCII fully exact | 70 / 75 | **73 / 75** |
|
||
| device PPV, waveform (live decode) | 1288 / 1306 | **1306 / 1306** |
|
||
| device PPV, histogram (live decode) | 4434 / 4459 | **4458 / 4459** |
|
||
|
||
Mean absolute PPV ratio error on waveforms is now 0.00000. The 2 remaining
|
||
ASCII imperfections differ by exactly 1 LSB on samples sitting at the
|
||
±10.000 in/s rail.
|
||
|
||
**This also eliminated the walker-over-read class.** 24 of those 35 files
|
||
were histograms that `read_blastware_file` fed to the *waveform* codec first;
|
||
the old walker accepted them and returned garbage (one yielded 98,923
|
||
"intervals"), while the record-chain decoder correctly returns `None` so they
|
||
fall through to `histogram_codec`.
|
||
|
||
`00 03` records are decoded rather than skipped. Skipping them does not merely
|
||
lose samples — it silently shifts the time base of everything after them on
|
||
that channel (observed on `BE9558/K558LOF2.820W`, MicL displaced by exactly
|
||
512 samples with nothing marking the gap).
|
||
|
||
Footer detection now prefers whichever `0e 08` candidate yields a chain
|
||
terminating on `0x06`, since the signature can occur inside a sample stream.
|
||
Blast radius: 1 file of 1,388.
|
||
|
||
The superseded model is retained as `decode_waveform_legacy` and pinned by
|
||
`micromate/idf_file.py`, whose Thor IDFW body-offset search trial-decodes
|
||
candidate offsets and keeps whichever yields the most samples — the new
|
||
decoder correctly returns `None` where the old one returned garbage, which
|
||
changes that heuristic's winner. Switching Thor over is deferred until that
|
||
search is reworked to use the record chain directly.
|
||
|
||
- **Series-3 histogram block is uniformly big-endian, and the stream's final
|
||
block has its own tail — the codec was clipping large peaks and dropping the
|
||
last interval of nearly every histogram.**
|
||
|
||
- **Peaks and half-periods are `uint16` big-endian**, not `uint8` plus an
|
||
"annotation" byte: `T_peak` `[5:7]`, `T_halfperiod` `[7:9]`, `V_peak`
|
||
`[9:11]`, and so on. Only `block_ctr` `[2:4]` is little-endian. The old
|
||
model silently **clipped any peak above 1.275 in/s** — the final interval
|
||
of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in Blastware's own export and
|
||
decoded as 0.590. The "annotation" byte was the half-period's high byte,
|
||
which is why it was non-zero exactly on the sub-Hz intervals BW renders
|
||
as `<1.0`.
|
||
- **The marker is `block[4]` alone.** Testing `[4:6]` as a `uint16 LE`
|
||
marker forced `block[5] == 0` — which is what capped the peak at one byte
|
||
in the first place.
|
||
- **The last block of each stream carries tail `9c 06 00 42`** instead of
|
||
`1e 0a 00 00`, with arbitrary bytes at `[21:23]`. Rejecting it dropped
|
||
the final interval of nearly every histogram, and that interval is
|
||
frequently the one holding the event peak — so the file's reported PPV
|
||
came out low.
|
||
|
||
Verified against **1211 production histograms** paired with their preserved
|
||
Blastware ASCII exports, which carry a full per-interval data table:
|
||
**1211/1211 now decode exactly** (interval count plus every per-interval
|
||
peak), and 842,442 per-interval frequency comparisons match with zero
|
||
mismatches. Before this fix: **1 of 1196**.
|
||
|
||
`decode_histogram_body_full` records now expose `is_terminal` in place of
|
||
the removed `annotations` tuple.
|
||
|
||
- **Geophone full scale is 32000 ADC counts, not 32768 — every geo reading was
|
||
2.3% low.** The verified body codec emits geo samples in 16-count units whose
|
||
documented LSB is exactly 0.005 in/s, and `decoded_to_adc_counts` multiplies
|
||
by 16, so one ADC count is `0.005/16` in/s and Normal range (10.000 in/s) is
|
||
`10.0 / (0.005/16)` = **32000** counts. Both `sfm/event_hdf5.py` and
|
||
`minimateplus/event_file_io.py` divided by 32768, scaling every geophone
|
||
sample and every derived peak down by `1 - 32000/32768` = **2.34%**.
|
||
|
||
Measured against 216 per-channel comparisons with preserved Blastware ASCII
|
||
exports: **32768 → 151/216 exact** (worst error 0.238 in/s on a 10 in/s
|
||
event); **32000 → 216/216 exact**, worst error 0.005 in/s (exactly 1 LSB —
|
||
pure quantization). The error scales with amplitude, so it was invisible on
|
||
quiet events and worst on the loud ones that matter for compliance.
|
||
|
||
The mic path is unaffected — it back-solves its own per-count factor from the
|
||
device-reported peak.
|
||
|
||
**Scope:** the scale lives in `_samples_to_float`, which every event passes
|
||
through regardless of which codec produced the samples — so this affected
|
||
**waveforms, histograms and series-4 (Thor IDF) alike**, not just waveforms.
|
||
Verified after regeneration: series-3 histogram peaks vs their ASCII reports
|
||
now sit at a median ratio of 1.0000 across 1,137 comparisons (0.9766 under
|
||
32768); series-4 peaks vs device peaks moved from a median 0.960 to 0.983
|
||
across 1,468 comparisons. The four block-framing fixes below are
|
||
waveform-only — histograms decode via `histogram_codec.decode_histogram_body`,
|
||
which is untouched.
|
||
|
||
- **Series-3 waveform codec: four block-framing cases caused silent channel
|
||
truncation.** `walk_body` hit its unknown-tag `break` mid-stream and every
|
||
channel decoded after that point came out short — typically Vert/Long/MicL,
|
||
sometimes at a third of their true length, with no error raised.
|
||
|
||
- **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already handled for
|
||
`1X NN` / `2X NN` also applies to the `00 NN` RLE tag. Runs longer than
|
||
252 samples must use the wide form (e.g. `01 0c` = 268 repeats).
|
||
- **`30 NN` with NN > 0x10** — the `0 < NN <= 0x10` guard was arbitrary;
|
||
data-section `30 NN` blocks reach at least NN = 0x18. The length formula
|
||
(`NN × 1.5 + 2`) was already correct.
|
||
- **Variable-width `40 NN` segment headers** — NN is the *count of
|
||
previous-channel continuation deltas*, so the header is `2 × NN + 16`
|
||
bytes and every field after the deltas shifts by `2 × NN`. Only `40 02`
|
||
(20 bytes) was handled; `40 01` (18) and `40 03` (22) both occur.
|
||
- **Tagless segment headers** — a segment header can appear with no
|
||
`40 NN` tag at all: just the 14-byte tail
|
||
`[field2:2][len:2][channel_id:4][marker:2][anchors:4]`. This is the NN=0
|
||
case (no continuation deltas needed, so no tag and no delta bytes). It is
|
||
where the walk stopped in 7 of the 8 events still truncating after the
|
||
first three fixes.
|
||
|
||
### Changed
|
||
|
||
- **Segment channel now comes from the header's own channel-id byte** rather
|
||
than from rotation position. The field previously documented as a
|
||
"monotonic uint32 LE counter" is really `[channel][00][00][segment_index]`
|
||
with `0x46`=Tran `0x47`=Vert `0x48`=Long `0x49`=MicL — verified on
|
||
**1697 of 1697** segment headers across the ground-truth corpus with zero
|
||
disagreements. Rotation-by-position is kept only as a fallback for unknown
|
||
ids; it was fragile because a single missed or extra header (exactly what
|
||
tagless headers caused) desynced every channel after it.
|
||
|
||
- **`parse_segment_header` return shape** — now `n_prev_deltas`,
|
||
`prev_deltas`, `marker`, `anchors`, `channel`, `segment_index` in place of
|
||
the fixed-offset `anchor_bytes` / `fixed_pattern` / `tail` keys. The old
|
||
`fixed_pattern` (`02 00 00 01`) conflated the 2-byte constant marker with
|
||
the first anchor. `counter` is retained as the raw uint32 of the id field.
|
||
|
||
### Verification
|
||
|
||
Against the 75 ground-truth events (BW binary paired with its preserved
|
||
`_ASCII.TXT` export), decoding end-to-end through the production path:
|
||
|
||
| | before | after |
|
||
|---|---|---|
|
||
| exact (full length, within 1 LSB) | 37 | **72** |
|
||
| truncated | 23 | **3** |
|
||
| full length, value error > 2 LSB | 15 | **0** |
|
||
|
||
Worst remaining error among the 72: 0.0050 in/s = exactly 1 LSB.
|
||
|
||
No regressions — the byte-exact fixture suite still passes, and the full-suite
|
||
failure list is unchanged from baseline (16 pre-existing failures from
|
||
gitignored fixtures).
|
||
|
||
### Notes
|
||
|
||
- **The "DC offset" symptom is _not_ a decode bug.** Events whose geo trace
|
||
sits at a constant level instead of oscillating around zero
|
||
(dominant-axis `|mean| / peak` >> 0) reproduce *exactly* in Blastware's own
|
||
ASCII export — e.g. `BE12599/N599LQD7.8E0W` Tran reads mean +0.345,
|
||
min +0.335, max +0.355 in both. It is a known recurring hardware fault (the
|
||
operators call it an "offset"): the affected channel's baseline exceeds the
|
||
unit's own geo trigger level, so the unit retriggers continuously and floods
|
||
the ACH queue with garbage events. Store-wide it affects 2 units of 21 across
|
||
6 episodes; see `scratch/offset_candidates.csv` and the project memory notes.
|
||
|
||
- ~~**Still open:** 3 of 75 ground-truth events truncate at a segment-header
|
||
variant with a variable-width prefix.~~ **Resolved later the same day** — the
|
||
record-chain rewrite (above) showed there is no variable prefix; it was
|
||
walker drift. All 75 are now sample-count exact.
|
||
|
||
- **Still open after this release:**
|
||
- **Series-4 (Thor / Micromate) is not verified** — UM-series sits at ~48%
|
||
against device peaks with a ~1.7% systematic bias and a near-zero tail.
|
||
Thor IDFW is pinned to `decode_waveform_legacy` deliberately.
|
||
- **14 sensitive-range files** show a decoded/truth ratio of exactly 8.0
|
||
(= 10.0/1.25) — a units bug, not a codec one. Never chased.
|
||
- **`backfill_sidecars.py --force` also inserts DB rows** for store files
|
||
that have none (1,286 on the snapshot; one-time per store), and the
|
||
dry-run does not report that count before you commit to it.
|
||
- **Verification is uneven:** per-sample proof on the 11% of files with a
|
||
preserved `_ASCII.TXT`, peak-and-structure consistency on the other 89%.
|
||
|
||
---
|
||
|
||
## v0.25.0 — 2026-08-25
|
||
|
||
**reviewed_real 3-state review flag + twin review-propagation.** The
|
||
`events` table and the `/db/events` feed now carry `reviewed_real`, a
|
||
3-state review flag mutually exclusive with `false_trigger`, mirrored from
|
||
the sidecar review block — plus histogram/waveform twin review-propagation
|
||
(flagging one flags both, matched by serial + identical PVS + timestamp
|
||
window).
|
||
|
||
### Added
|
||
|
||
- **`events` column** `reviewed_real` — `INTEGER NOT NULL DEFAULT 0`, added
|
||
via the existing incremental `_migrate` ADD COLUMN pass (auto-migrates on
|
||
`SeismoDb()` construction, no manual migration). `query_events` /
|
||
`get_event` (and thus `/db/events`) return it automatically (`SELECT *`).
|
||
- **Mutual exclusivity with `false_trigger`** — setting `reviewed_real=1`
|
||
clears `false_trigger`, and vice versa, enforced on both review paths:
|
||
the sidecar review PATCH (`update_event_review`) and the quick
|
||
`PATCH /db/events/{id}/false_trigger` endpoint (`set_false_trigger`).
|
||
- **`find_twins`** — matches an event's histogram/waveform twins by serial +
|
||
identical peak-vector-sum + a timestamp window.
|
||
- **`propagate_review_to_twins`** — copies an event's `false_trigger`/
|
||
`reviewed_real` state onto its twins, wired into both the
|
||
`PATCH /db/events/{id}/sidecar` review path and the quick
|
||
`PATCH /db/events/{id}/false_trigger` path, so flagging one flags both
|
||
regardless of which endpoint made the change.
|
||
|
||
---
|
||
|
||
## v0.24.0 — 2026-08-22
|
||
|
||
**Waveform-shape metrics on events.** The `events` table and the `/db/events`
|
||
feed now carry per-event crest factor and points-near-peak, computed from the
|
||
decoded waveform samples at ingest — groundwork for Terra-View's
|
||
false-trigger detection (Phase B).
|
||
|
||
### Added
|
||
|
||
- **`events` columns** `shape_crest_factor`, `shape_near_peak_count`,
|
||
`shape_sample_count`, `shape_axis`. Added via the existing incremental
|
||
`_migrate` ADD COLUMN pass — **auto-migrates on `SeismoDb()` construction,
|
||
no manual migration**. `query_events` / `get_event` (and thus `/db/events`)
|
||
return them automatically (`SELECT *`).
|
||
- **Populated at ingest** — crest factor + near-peak-count are computed from
|
||
the decoded samples in every save path (`shape_from_h5`/
|
||
`shape_from_samples`), and `insert_events` persists them on INSERT and
|
||
UPSERT.
|
||
- **Backfill** `scripts/backfill_event_shape.py` — fills the columns for
|
||
existing events from their on-disk `.h5` waveform samples (idempotent,
|
||
UPDATE-only).
|
||
|
||
### Upgrade Notes
|
||
|
||
Run the backfill once after deploying, against the events DB + waveform store:
|
||
`python3 scripts/backfill_event_shape.py --db-path <seismo_relay.db> --store-root <waveforms/>`
|
||
Events with no decodable samples (or no waveform file) stay NULL and render
|
||
"—" downstream.
|
||
|
||
---
|
||
|
||
## v0.23.0 — 2026-08-04
|
||
|
||
**Per-channel ZC frequency in the events store.** The `events` table and the
|
||
`/db/events` feed now carry per-channel zero-crossing frequency, so consumers
|
||
(Terra-View's Vibration Summary) get frequency in bulk instead of one sidecar
|
||
fetch per event.
|
||
|
||
### Added
|
||
|
||
- **`events` columns** `tran_zc_freq`, `vert_zc_freq`, `long_zc_freq`, `mic_zc_freq`
|
||
(Hz) + `*_zc_above_range` flags (the Blastware `>N Hz` device-ceiling case). Added
|
||
via the existing incremental `_migrate` ADD COLUMN pass — **auto-migrates on
|
||
`SeismoDb()` construction, no manual migration**. `query_events` / `get_event`
|
||
(and thus `/db/events`) return them automatically (`SELECT *`).
|
||
- **Populated at ingest** — `PeakValues` gains the ZC-freq fields; both
|
||
`apply_report_to_event` and `apply_bw_report_dict_to_event` copy per-channel
|
||
`zc_freq_hz` / `zc_freq_above_range` from the parsed report onto them, and
|
||
`insert_events` persists them on INSERT and UPSERT.
|
||
- **Backfill** `scripts/backfill_event_zc_freq.py` — fills the columns for existing
|
||
events from their `.sfm.json` sidecars (idempotent, UPDATE-only).
|
||
|
||
### Fixed
|
||
|
||
- `sfm/server.py` app-version string was stale at `0.17.0`; now tracks the real version.
|
||
|
||
### Upgrade Notes
|
||
|
||
Run the backfill once after deploying, against the events DB + waveform store:
|
||
`python3 scripts/backfill_event_zc_freq.py --db-path <seismo_relay.db> --store-root <waveforms/>`
|
||
Events whose sidecar carries no frequency (or that have no sidecar) stay NULL and
|
||
render "—" downstream.
|
||
|
||
---
|
||
|
||
## v0.22.0 — 2026-07-03
|
||
|
||
Full-snapshot bundle support (SFM side). Adds the DB-snapshot, recent-waveform,
|
||
and gated-restore endpoints Terra-View drives to pull a complete prod→dev refresh
|
||
— the `seismo_relay.db` plus the waveform files behind recent events — in one
|
||
pass. Pairs with **Terra-View v0.17.0**'s "Create full snapshot (incl. SFM)"
|
||
control on Settings → Database. New logic lives in `sfm/db_snapshot.py`, kept
|
||
out of `server.py` so it's unit-testable without HTTP.
|
||
|
||
### Added
|
||
|
||
- **`GET /db/snapshot`** — streams a WAL-safe point-in-time copy of
|
||
`seismo_relay.db`, taken through the SQLite online backup API (not a raw
|
||
file grab), so the download stays consistent even while the ACH server is
|
||
writing under WAL.
|
||
- **`GET /db/waveforms/recent.zip?n=…`** — zips the on-disk files (event file
|
||
+ its `.h5` / sidecar siblings) for the *n* most-recent events, arcname
|
||
`<serial>/<filename>`, so a dev box can restore chart-openable events without
|
||
hauling the entire waveform store. Events with no resolvable files on disk
|
||
are skipped.
|
||
- **`POST /db/restore`** — gated by `SFM_DB_RESTORE_ENABLED` (dormant → 404
|
||
unless explicitly enabled; dev-only). Validates the uploaded DB first, takes
|
||
a WAL-safe `.pre-restore-<ts>` safety backup of the current DB, then swaps in
|
||
the restored DB + waveforms. Zip entries are path-traversal-guarded before
|
||
extraction.
|
||
|
||
### Fixed
|
||
|
||
- **Restore no longer blocks the event loop.** `/db/restore` runs its
|
||
synchronous DB/file work in a threadpool instead of `async def`, so a large
|
||
restore can't stall inbound ACH connections or other API requests.
|
||
|
||
---
|
||
|
||
## v0.21.1 — 2026-06-01
|
||
|
||
Bug fixes against v0.21.0 surfaced after the first prod redeploy. Three
|
||
production-visible symptoms — blank waveform charts on most Thor events,
|
||
blank histogram charts on all Thor events, and a mic chart that
|
||
auto-scaled against a dB(L) value treated as psi — all root-caused and
|
||
fixed.
|
||
|
||
### Fixed
|
||
|
||
- **Dynamic IDFW body offset.** The v0.21.0 codec hardcoded the body
|
||
at file offset `0x0f1f` based on the example corpus, but only ~52%
|
||
of production IDFW events use that offset; the rest sit at offsets
|
||
from `0x1033` up to `0x3082` depending on header padding. At
|
||
`0x0f1f` the codec would find a coincidentally-matching `00 02 00`
|
||
magic, read the 2-byte Tran preamble, and return empty V/L/M
|
||
arrays — producing near-empty .h5 files and blank charts.
|
||
`micromate.idf_file._find_waveform_body_offset()` now scans every
|
||
`00 02 00` magic position past `0x0E00`, trial-decodes each one,
|
||
and picks the offset with the most samples. Validated across 483
|
||
prod IDFW files: 0 preamble-only events (was ~50%), 355/483 fully
|
||
decode, 126/483 partial (BW codec walker-stops-early on loud
|
||
events — pre-existing limitation, samples reached are correct).
|
||
|
||
- **IDFH histograms now render bar charts.** Histograms previously
|
||
skipped the .h5 write because there are no per-sample arrays, but
|
||
the renderer drives the per-interval bar chart from .h5 channel
|
||
data + `bw_report.histogram.n_intervals`. `save_imported_idf` now
|
||
synthesizes a 1-sample-per-interval array from the decoded
|
||
`IdfhInterval` peak counts and writes an .h5 so the existing
|
||
renderer works unchanged — each "sample" is the per-interval peak
|
||
ADC count, so the writer's `count × geo_fs/32768` conversion
|
||
yields the right bar height.
|
||
|
||
- **Mic chart scaling on Thor events.** `PeakValues.micl` (consumed
|
||
by the h5 writer's per-count mic scale factor) expects psi, but
|
||
the Thor bridge was stuffing the dB(L) value (~99.4) into it,
|
||
producing a per-count factor 5+ orders of magnitude too large and
|
||
a flat-looking mic chart. Fixed by adding `IdfPeaks.mic_pspl_psi`
|
||
alongside `mic_pspl_dbl`; `read_idf_file()` computes it from
|
||
binary mic counts (`max(|MicL|) × 2.14e-6 psi/count`) for both
|
||
IDFW and IDFH paths; `save_imported_idf` merges it onto the typed
|
||
event after `IdfEvent.from_report`; the bridge feeds psi to
|
||
`PeakValues.micl` with a dB(L)→psi formula fallback when only the
|
||
dB(L) value is available. dB(L) for the report header still
|
||
flows through `bw_report.mic.pspl_dbl` unchanged.
|
||
|
||
### Operator
|
||
|
||
After deploy, run `python scripts/backfill_thor_events.py` to refresh
|
||
every existing Thor event's sidecar + .h5 with the corrected codec
|
||
output. The script auto-skips events already at the current
|
||
`TOOL_VERSION`, so the bump from `0.21.0` → `0.21.1` is what triggers
|
||
the refresh.
|
||
|
||
---
|
||
|
||
## v0.21.0 — 2026-05-29
|
||
|
||
The "Thor / Series IV codec" release. Two big pieces landed: (1) the IDF binary codec actually decodes now, both IDFW and IDFH, and (2) a Thor→BW adapter lets Thor events flow through the existing Series III Event Report PDF pipeline. Combined effect: a Thor event ingested via `/db/import/idf_file` now lands in the DB with the same fidelity as a Blastware event, gets a per-event PDF on demand, and renders in Terra-View's modal chart with the same plotting code as a BW event.
|
||
|
||
### Added — Thor IDF binary codec (`micromate/idf_file.read_idf_file`)
|
||
|
||
- **IDFW (waveform)** — body sits at fixed file offset `0x0f1f`; reuses the verified `decode_waveform_v2()` walker from `minimateplus.waveform_codec`. Sample fidelity is **87–99% byte-exact** against the ASCII-sidecar reference values on quiet events; loud events hit the same walker-stops-early limitation as the BW codec on `SP0/SS0/SV0`-style events.
|
||
- **IDFH (histogram)** — dedicated segment-based decoder for the Thor histogram body format: `[len_be][0a 00 00 00][00 NN][05 3f]` framing plus N × 72-byte interval records (4 × 16-byte per-channel min/max/halfp). **All 859 Thor IDFH corpus files decode**, totalling **181,071 intervals**; per-channel peaks match the sidecar within **~1.8% (ADC quantization)**.
|
||
- **BW-aliased binary detection** — a small number of corpus files (e.g. `BE9439_*.IDFW/IDFH`) are actually Series III Blastware binaries that share the IDF filename convention by accident. `read_idf_file()` detects them via their BW `STRT` signature and raises `NotImplementedError` pointing the caller at `read_blastware_file()` instead of trying to decode them as IDF.
|
||
- Full field layouts in `docs/idf_protocol_reference.md`; supporting analysis scripts in `analysis_idf/` (decode validators, per-file detail dumps, corpus accuracy reports).
|
||
|
||
### Added — Thor → BW report adapter (`micromate/idf_to_bw_report.py`)
|
||
|
||
- **`build_bw_report_from_idf(report_dict, binary_md=, intervals=, is_histogram=)`** projects a parsed Thor `IdfReport` plus binary-extracted metadata plus decoded IDFH intervals into the `bw_report`-shaped dict that `sfm.report_pdf.gather_report_data` consumes. No need to duplicate the renderer — Thor data is ~95% the same metric set as BW; the adapter handles the field-name mapping (`MicPSPL` → `pspl_dbl`, `>100` sentinel → `zc_freq_above_range`, free-form `Calibration : Nov 22, 2023 by Instantel` → `calibration_date` + `calibration_by`, etc.).
|
||
- For IDFH events the adapter derives `histogram.interval_times` by stepping `IntervalSize` from `HistogramStartTime`, matching what the BW pipeline expects from a histogram-mode event.
|
||
- **Wired into `WaveformStore.save_imported_idf`** — every Thor event ingested via `/db/import/idf_file` now gets a `bw_report` block in its sidecar in addition to the existing `extensions.idf_report` (the raw parsed Thor payload). Falls back gracefully (PDF renders from DB-only fields) if the adapter raises — logged as a warning rather than failing the ingest.
|
||
|
||
### Companion releases
|
||
|
||
- **Terra-View v0.13.0** ships in parallel — closes Phase 1 of the SFM integration. The shared event-detail modal now renders the SFM event story (Chart.js waveform/histogram chart, inline PDF preview, `.TXT` download, FT/reviewer/notes review form) without operators needing to bounce to the standalone SFM webapp on port 8200. Uses only existing seismo-relay endpoints — no API changes here, just better consumption.
|
||
|
||
### Migration / Operations
|
||
|
||
No DB migration needed. Existing Thor events already in the store don't automatically pick up the new `bw_report` block — they'd need a re-ingest (post the IDF binary + paired `.TXT` back to `/db/import/idf_file`) for the adapter to run. Alternatively, run `scripts/backfill_sidecars.py --reparse-txt` after a small adapter change (the script currently only re-runs the BW ASCII parser; extending it to handle Thor would be a small follow-up).
|
||
|
||
```bash
|
||
cd /home/serversdown/terra-view
|
||
docker compose build sfm && docker compose up -d sfm
|
||
```
|
||
|
||
The bumped `TOOL_VERSION = "0.21.0"` in `minimateplus/event_file_io.py` means any subsequent `backfill_sidecars.py --force` pass will re-write sidecars with the new version stamp; that's expected and harmless.
|
||
|
||
---
|
||
|
||
## v0.20.0 — 2026-05-28
|
||
|
||
The "PDF + parser polish" release. Closes out the Event-Report PDF iteration started in v0.17.x: histogram layouts now render correctly against BW reference PDFs, the ASCII parser handles the real-world edge cases production events were tripping over (OORANGE, `>100 Hz`, histogram timestamps), and the `.TXT` preservation rollout lets parser fixes be applied retroactively to ingested events. Adds server-wide timezone support so operator-visible timestamps no longer drift into UTC. Rolls up the substantial "pre-v0.20" body of work that had accumulated under `[Unreleased]` (PDF generation, histogram codec fix, histogram parser fields, `.TXT` preservation, backfill safety) — see the trailing "pre-v0.20.0 work" section below for the full list.
|
||
|
||
### Added (2026-05-28)
|
||
|
||
- **Server-wide display timezone via `TZ` env var.** Both seismo-relay and terra-view now respect a `TZ` environment variable (default `America/New_York` on prod). Affects server log timestamps, the PDF report renderer's UTC→local conversions on the "Created" footer line, matplotlib's datetime axes, and any other naïve-vs-aware datetime rendering. DB columns (`created_at`, etc.) stay UTC regardless — this is a display-side fix, not a storage-side one. Dockerfile now installs `tzdata` (required for the env var to take effect under `python:slim`). Override per-deployment via the `TZ` line in `docker-compose.yml`.
|
||
- **ZC Freq "above-range" handling — render `>100 Hz` instead of `—`.** BW writes `">100 Hz"` literally when the zero-crossing algorithm sees a peak too fast to count (device cuts off at 100 Hz on V10.72). Previously `_parse_number(">100")` returned None and the PDF stats table rendered `—`. Now the parser mirrors the OORANGE pattern: stores 100.0 on `zc_freq_hz` and sets a new `zc_freq_above_range` flag. Flag rides through the sidecar's `bw_report` block. Renders as `>100` in the PDF (per-channel + mic block), as `· >100 Hz` inline on the event modal's Peaks section, and as a dedicated column on the event-browser stats table. Verified against the real T190LD5Q.LK0W fixture from 2026-05-27 plus a synthetic test case.
|
||
- **Per-channel ZC Freq surfaced in event modals.** Neither the main webapp modal (`sfm_webapp.html`) nor the standalone event browser (`event_browser.html`) previously exposed ZC Freq. Now both do — webapp shows it inline alongside PPV (`0.04500 in/s · 47 Hz`); event-browser gets a dedicated column on its per-channel stats table. Required wiring a parallel sidecar fetch into the event-browser's `loadEvent()` (it was only fetching `waveform.json`). Falls back to `—` for events without a preserved `.TXT` (pre-2026-05-27 ingests).
|
||
- **`scripts/backfill_sidecars.py --reparse-txt` flag.** Before this, the backfill script preserved the `bw_report` block from existing sidecars verbatim — so parser-side fixes (like the `>100 Hz` addition above) couldn't reach old events. The new flag re-runs the current parser against the preserved `<serial>/<filename>_ASCII.TXT`, overwrites the bw_report block, and cascade-regenerates the sidecar. Implies sidecar regeneration on every event (bypasses the sha/version skip). No-op for events without a preserved .TXT (legacy ingests pre-2026-05-27 .TXT-preservation rollout). Idempotent. Run with `--skip-hdf5` to skip waveform regen — recommended when only the bw_report needs refreshing. Validated end-to-end on prod: 9,999 events refreshed cleanly, ZC Freq + OORANGE flags now populated where the original .TXT had them.
|
||
|
||
### Fixed (2026-05-28)
|
||
|
||
- **Histogram PDFs no longer 500 on the missing `histogram_interval_size_s` attribute.** The histogram-interval-times derivation block in `gather_report_data` referenced `rd.histogram_interval_size_s`, but the field was never declared on the `ReportData` dataclass nor read from the sidecar projection (it was inlined into `gather_report_data` without the seconds-numeric counterpart making it onto the dataclass). Every histogram PDF render raised `AttributeError → 500`. Waveform PDFs were unaffected. Fix: add the field, read it from the projection's existing `bw_report.histogram.interval_size_s` key.
|
||
- **Histogram PDF geo channels now share a single nice-quantized y-axis.** Previously each geo subplot auto-scaled independently — Tran, Vert, and Long all showed different per-channel maxes, so bar heights weren't directly comparable across channels. The footer "Amplitude Geo: X in/s/div" label was also computed as `max(first_geo_channel) / 5` with no LSB quantization, producing nonsense values like `0.003 in/s/div` when the geophone LSB is 0.005. Fix: compute a single shared geo y-axis range from `max(Tran, Vert, Long)`, quantize the per-division step to BW's 1-2-5 sequence rounded to the 0.005 in/s LSB (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, ...), apply the same `ylim` + ticks to all three subplots, and use that step for the footer label. MicL stays on its own auto-scale (different units). Matches BW's chart styling.
|
||
|
||
### Docs (2026-05-28)
|
||
|
||
- **Roadmap entry for a second undecoded histogram body sub-format.** BE17353 (S353) events observed on 2026-05-28 use a histogram body where `byte[5] = 0x00` (looks like a valid block header by every prior signal) but the walker finds zero data blocks. Different from the existing `byte[5] != 0` roadmap entry (T190 / O121). Operationally identical impact — ingestion succeeds, DB peaks come from the bw_report overlay, only the chart is empty. Sample events captured in the roadmap entry for future RE work.
|
||
|
||
### Migration / Operations
|
||
|
||
- **Re-parse existing events to pick up the new parser fields.** Run on whichever box hosts the live waveform store:
|
||
```bash
|
||
docker exec terra-view-sfm-1 python /app/scripts/backfill_sidecars.py \
|
||
--reparse-txt --skip-hdf5 --dry-run -v | tail
|
||
# Looks reasonable? Run for real:
|
||
docker exec terra-view-sfm-1 python /app/scripts/backfill_sidecars.py \
|
||
--reparse-txt --skip-hdf5 -v | tee /tmp/reparse.log | tail -30
|
||
```
|
||
Idempotent; safe to re-run. Only touches sidecars on disk — no DB writes.
|
||
- **terra-view docker-compose.yml**: add `TZ=America/New_York` (or your deployment's zone) to both the `terra-view` and `sfm` service `environment:` blocks. Without this, server-rendered timestamps stay in UTC even on the rebuilt SFM image.
|
||
|
||
### Pre-v0.20.0 work (rolled into this release)
|
||
|
||
The bullets below accumulated under `[Unreleased]` between v0.19.0 and v0.20.0; kept here so the historical narrative isn't lost.
|
||
|
||
#### Fixed
|
||
|
||
- **bw_ascii_report parser now handles `OORANGE` saturation marker.** BW writes `"OORANGE"` (truncation of "Out Of Range") in PPV / PVS / MicL PSPL fields when the underlying measurement exceeded the channel's full-scale. Previously our `_parse_number()` returned None → DB ended up with NULL peaks for legitimate high-amplitude events. Confirmed on real ASCII files pulled 2026-05-27 from the Windows watcher PC: T190LD5Q.LK0W (Vert saturated at Normal range 10 in/s), T438L713.RY0W (all three channels saturated at Sensitive range 1.25 in/s), K557L3YM.OE0W (Tran+Vert saturated + Mic PSPL OORANGE). New behavior:
|
||
- Per-channel PPV: substitute `geo_range_ips` as a conservative lower bound + set `ppv_saturated` flag
|
||
- Peak Vector Sum: substitute `sqrt(3) * geo_range_ips` (the theoretical max when all 3 channels are simultaneously at full-scale) + `peak_vector_sum_saturated` flag
|
||
- MicL PSPL: substitute 140 dB(L) (conservative NL-43 max) + `pspl_saturated` flag
|
||
- Saturation flags are propagated into the sidecar's `bw_report` block for downstream UI rendering (`> 10 in/s` or similar)
|
||
- Five events on prod (T190 / T438 / K557 + 2 others matching the same fault pattern) will pick up correct DB peaks + saturation flags once re-forwarded
|
||
- **bw_ascii_report parser handles `Peak Vector Sum TimeSum` typo'd label.** Real BW output uses this misspelled label (Sum appended twice instead of "Peak Vector Sum Time"). Now accepted as an alias. Confirmed against all three OORANGE example files — every one has the typo.
|
||
|
||
#### Added
|
||
|
||
- **Histogram per-interval aggregation in `waveform.json`.** Histogram events now render with one bar per BW-reported interval (matching the Blastware printout) instead of ~200 bars per event (the raw codec output). When the sidecar's `bw_report.histogram.n_intervals` is populated (events ingested with the new parser, see next bullet), the `/db/events/{id}/waveform.json` endpoint groups the codec samples into N intervals via max-per-group and returns the aggregated array. `time_axis` gains `histogram_aggregated: true`, `n_intervals`, `interval_size_s`, and `interval_times` (HH:MM:SS strings). Both the modal chart and the standalone event browser use those interval timestamps as x-axis labels when present. Defensive: no-op for events ingested before the parser extension landed (their sidecars lack `histogram.n_intervals`) — those continue to render with raw codec output.
|
||
- **`bw_ascii_report` parser now captures histogram-specific fields.** Previously the parser dropped these fields silently (Roadmap item closed):
|
||
- `Histogram Start Time` / `Histogram Start Date` (combined into `histogram_start: datetime`)
|
||
- `Histogram Stop Time` / `Histogram Stop Date` (combined into `histogram_stop: datetime`)
|
||
- `Number of Intervals` (`histogram_n_intervals: int`)
|
||
- `Interval Size` ("1 minute" string + parsed seconds: `histogram_interval_size_str`, `histogram_interval_size_s`)
|
||
- `<Channel> Peak Time` + `<Channel> Peak Date` for histogram events (combined into `channel_peak_when: dict`; waveforms continue to use `time_of_peak_s` relative)
|
||
- `Peak Vector Sum Date` (combined with PVS Time into `peak_vector_sum_when: datetime`; clears the previous bogus `peak_vector_sum_time_s` parse that interpreted "22:33:52" as 22.0 seconds)
|
||
- All new fields land in the sidecar's `bw_report.histogram` block via `_bw_report_to_dict`. Tested against synthetic K558LLB7.V20H-shaped input.
|
||
- **Raw BW ASCII report (.TXT) preservation.** `save_imported_bw` now writes the paired `_ASCII.TXT` to `<store>/<serial>/<filename>_ASCII.TXT` alongside the binary at ingest time. Previously the .TXT was parsed into the sidecar's `bw_report` projection and then discarded — meaning parser bug fixes couldn't be applied retroactively without re-forwarding from the watcher PC. Now the raw .TXT lives in the waveform store permanently (~15 KB per event; ~210 MB total for a 14k-event store; negligible). Sidecar's `source.txt_filename` field records the saved path; backfill_sidecars preserves it across regens. New `GET /db/events/{id}/ascii_report.txt` endpoint serves the raw .TXT for any event ingested after this change. Events ingested before today still return 404 from that endpoint until re-forwarded. Architectural rationale: with BW Mail / Forwarding Agent being phased out of the operator workflow, the XML/PDF/WMF that those tools produced are no longer available — the binary + .TXT (created by BW ACH itself) are our authoritative source for everything going forward.
|
||
|
||
- **Event Report PDF generation** — `GET /db/events/{id}/report.pdf` returns a single-page letter-portrait PDF for any event with waveform data on disk. Covers every field a Blastware Event Report includes: header metadata (date/time, trigger source, range, sample rate, project/client/operator/location, serial+firmware, battery, calibration, file name), microphone block (PSPL in dB(L) + psi, ZC freq, channel test), per-channel stats table (rows differ for waveform vs histogram), Peak Vector Sum, and the 4-channel plot. Iterated against real Blastware reference PDFs (uploaded to `example-events/pdfsnstuff/`):
|
||
- **Waveform layout**: header shows Date/Time, Trigger Source, Range, Sample Rate; stats table has PPV / ZC Freq / Time (Rel. to Trig) / Peak Accel / Peak Disp / Sensor Check; bottom plot is 4-channel line waveform (MicL top → Tran bottom), shared time axis in seconds, dashed trigger line + triangle marker at t=0, symmetric Y on geo channels, zero-anchored on mic, "0.0" baseline label on right per BW convention; footer shows `Time X sec/div Amplitude Geo: Y in/s/div Mic: 0.001 psi(L)/div` and the trigger window `▶━━◀` marker. USBM RI8507/OSMRE compliance chart placeholder upper-right.
|
||
- **Histogram layout**: header shows Start / Finish / Intervals At Size / Range / Sample Rate (no Trigger Source — histograms aren't triggered); NO USBM chart; stats table has PPV / ZC Freq / Date / Time / Sensor Check; bottom plot is per-interval bar chart, Y-axis 0-to-peak (never negative), 0.0 baseline at the bottom; footer shows `Time INTERVAL_SIZE /div Amplitude Geo: Y in/s/div Mic: 0.001 psi(L)/div`.
|
||
- Backed by matplotlib (vector PDF, no headless-browser dep). Adds matplotlib>=3.8 to deps.
|
||
- **Known gap**: histogram codec returns per-block granularity (~200 bars for a 4-interval event) instead of BW's per-interval aggregation. Visual difference vs BW's 4-bar display. XML-driven data source (parsing the structured `_XML.XML` files BW also exports) is the planned fix; that route also resolves the bw_ascii_report PPV-miss bug.
|
||
- **Stubbed**: USBM RI8507 / OSMRE compliance chart curves (separate work item; requires coding the regulatory piecewise functions).
|
||
- **"Download PDF" button** in the event modal's footer — triggers the new endpoint; opens in a new tab so the browser handles save-or-display + surfaces any 404 / server errors visibly.
|
||
|
||
- **SFM webapp now opens to Database view by default** and the History table is fully interactive. Click any column header to sort ascending / descending (timestamp, serial, per-channel PPV, PVS, mic dB(L), project, client, record type, key — all sortable). Click any event row to open the event modal, which now renders a **4-channel waveform plot inline** (MicL / Long / Vert / Tran stacked, Instantel-printout order) alongside the existing sidecar review fields. Headers are sticky so the columns stay visible while scrolling long event lists. No more "where is the viewer" — pick a unit from the filter dropdown, scan the table, click the event, see the waveform.
|
||
- **Stored-event browser** — new standalone HTML page at `GET /events` (`sfm/event_browser.html`). Pick a serial from the unit dropdown, scroll through that unit's events (newest-first), click any event to render its decoded waveform via the existing `/db/events/{id}/waveform.json` endpoint. Dark-themed Chart.js viewer, channels stacked vertically (MicL / Long / Vert / Tran — Instantel printout order, designed PDF-export-ready), trigger line at t=0, peak labels, search/filter, false-trigger flag honored. Companion to the existing live-device viewer at `/waveform`; the two routes are now clearly delineated in their docstrings. The webapp's inline plot at `/` is the primary path; `/events` remains a useful diagnostic when you want just a viewer.
|
||
- **Histogram body codec — uint8 peak count fix.** Per-channel peak fields at `block[6]/[10]/[14]/[18]` are `uint8`, not `uint16 LE` spanning `block[6:8]` etc. The original interpretation was byte-exact on the N844 fixture corpus only because every annotation byte (`block[7]/[11]/[15]/[19]`) in those fixtures was zero. On non-N844 events with non-zero annotation bytes (observed across BE9558 Tran-drift and BE18003 Histogram+Continuous units), the old interpretation produced peaks up to 268 in/s per channel and 35× inflated PVS sums when first deployed to prod (rolled back same day; properly fixed in this release). Cross-correlated against BW's per-interval ASCII export on K558 / T003 / N599 / N844 corpora — 100% byte-exact on T/V/L, 99%+ on M (sub-precision rounding). Annotation byte preserved on each record as `record["annotations"]` for future RE. Verified against ~3,500 blocks across 5 in-repo fixtures + a synthetic K558 interval-12 regression block.
|
||
- **`apply_bw_report_dict_to_event` helper** in `minimateplus.event_file_io`. Mirror of `apply_report_to_event` for the projected sidecar dict shape — used by the backfill path, which has the preserved `bw_report` block but not the original `.TXT` file. BW's reported peaks (and `sample_rate` / `record_time`) now win over codec output during `--force` backfill, matching ingest-path behavior.
|
||
- **`scripts/check_bw_report_preservation.py`** — two-step snapshot/diff tool to verify that `backfill_sidecars.py` doesn't wipe the `bw_report` block from existing sidecars. Classifies every sidecar as PRESERVED / CHANGED / WIPED / STILL_MISSING / NEW / ADDED / REMOVED. Exit code 1 if any WIPED or CHANGED entries are found, so it can gate a CI step or deploy script.
|
||
|
||
#### Fixed
|
||
|
||
- **`scripts/backfill_sidecars.py` no longer wipes `bw_report`.** Before this fix, `event_to_sidecar_dict` silently dropped the preserved `bw_report` block during every backfill, since the function only emits a `bw_report` when called with a live `BwAsciiReport` dataclass (which the backfill doesn't have — only the projected sidecar dict). Now we read the existing sidecar's `bw_report` and overlay it onto the regenerated sidecar, alongside the existing `review` and `extensions` preservation.
|
||
- **`scripts/backfill_sidecars.py --force` no longer overwrites BW-overlaid DB peaks with codec output.** The backfill path now calls `apply_bw_report_dict_to_event` before the DB upsert, mirroring what the ingest path does (`/db/import/blastware_file` parses the `.TXT` into a `BwAsciiReport`, calls `apply_report_to_event`, then upserts). Without this, events where the codec doesn't fully decode (waveform walker edge cases on SP0/SS0/SV0-style events, histogram `byte[5]!=0` sub-format) ended up with PVS=0 in the DB after a `--force` backfill; bit on prod 2026-05-22, rolled back the same day.
|
||
- **Thor IDF files no longer attempted as BW events in backfill.** `scripts/backfill_sidecars.py` now filters out `.IDFW` / `.IDFH` files in `_looks_like_event_file()`; they share the `.X0W` / `.X0H` suffix shape but use a separate ingest path (`WaveformStore.save_imported_idf`) and aren't decodable by `event_file_io.read_blastware_file`.
|
||
|
||
#### Docs
|
||
|
||
- **CLAUDE.md** — added a three-tier conceptual architecture model (SFM / SDM / shared codec library) near the top of the file, with a placement rule for where new code goes. Documents that what is conceptually SDM (database, waveform store, ingest, `/db/*` endpoints) still lives under `sfm/` for historical reasons; rename deferred until the codebase is quiet enough for a clean refactor.
|
||
- **README.md** — added a "Strategic direction" lead-in to the Roadmap that frames seismo-relay as a suite of cooperating components (not a single app), and an explicit "Terra-View ↔ SFM device control" roadmap section with a concrete implementation checklist (auth as hard prerequisite, embedded live-monitor view, action history, Series IV live-device support).
|
||
- **`docs/histogram_codec_re_status.md`** updated with the uint8 retraction and the annotation-byte status.
|
||
- Three known issues recorded in the Roadmap that were discovered during prod validation: (1) `bw_ascii_report` parser misses PPV / `vector_sum` on some `.TXT` formats (5 events on prod); (2) NULL-timestamp duplicate-row dedup needed (2 events on prod); (3) histogram body sub-format with `byte[5] != 0` not yet decoded (~3 events on prod with empty `.h5` plots).
|
||
|
||
---
|
||
|
||
## v0.19.0 — 2026-05-20
|
||
|
||
The "device-family separation" release. Tightens the boundary between Series III (MiniMate Plus / Blastware) and Series IV (Micromate / Thor) so the UI and storage layer dispatch deterministically by family instead of sniffing filename extensions or magnitude heuristics.
|
||
|
||
### Added — Phase 1: `device_family` column on `events`
|
||
|
||
- **`events.device_family TEXT`** — new column carrying `"series3"` or `"series4"`. Populated by every import path (`/db/import/blastware_file`, `/db/import/idf_file`, ACH server, BW CLI, sidecar backfill script). Returned through `/db/events` since `query_events` uses `SELECT *`.
|
||
- **Self-applying migration** — on startup, `ALTER TABLE ... ADD COLUMN` lands the new column; a follow-on `UPDATE` backfills existing rows from the binary filename extension (`.IDFH`/`.IDFW` → `series4`, everything else → `series3`). No manual SQL needed.
|
||
- **UPSERT preserves family** — re-imports without an explicit family don't blank existing rows (`COALESCE(?, device_family)`).
|
||
- **UI dispatches on the column** — `sfm_webapp.html` events-table mic formatter now branches on `ev.device_family === 'series4'` (Thor stores native dB(L); BW stores psi). Modal uses `source.kind === 'idf-import'` from the sidecar (sidecars don't carry the DB column). Source-files section labels changed from "BW filename / BW filesize / BW sha256" to format-neutral "Event file / File size / File sha256".
|
||
|
||
### Added — Phase 2: `micromate/` package alongside `minimateplus/`
|
||
|
||
- **`micromate/`** — new sibling package for the Thor / Micromate Series IV device. Currently scoped to offline-file ingest; live-device support (TCP transport, framing, protocol, client) will land here when reverse-engineering happens.
|
||
- `micromate/idf_ascii_report.py` — moved from `sfm/idf_ascii_report.py`. No behaviour change.
|
||
- `micromate/models.py` — typed `IdfReport`, `IdfEvent`, `IdfPeaks`, `IdfProjectInfo`, `IdfSensorCheck`. Stores mic in native `mic_pspl_dbl` (dB(L)) instead of the pseudo-psi shoehorn that the BW-shaped model uses. `IdfEvent.from_report()` constructs from a parsed dict + filename; `IdfEvent.to_minimateplus_event(waveform_key)` bridges to the existing sidecar / DB-insert machinery.
|
||
- `micromate/idf_file.py` — placeholder for the binary codec (`.IDFH` / `.IDFW`). Stubbed `read_idf_file()` raises `NotImplementedError`; documents the planned reverse-engineering path.
|
||
- **`WaveformStore.save_imported_idf`** refactored to use the native `IdfEvent` and bridge at the SQL-insert boundary. Cleaner separation of "parse a Thor event" (in `micromate/`) from "store it on disk + write a sidecar" (in `sfm/waveform_store.py`).
|
||
- **Tests** — `tests/test_idf_ascii_report.py` imports updated to `micromate.idf_ascii_report`. All 1,014 example-data sidecars round-trip through `IdfEvent.from_report()` without errors.
|
||
|
||
### Companion releases
|
||
|
||
- **thor-watcher** unaffected — it talks to the relay over HTTP only. No version bump needed.
|
||
- **terra-view** unaffected today; can use `device_family` in its event-detail rendering when convenient.
|
||
|
||
---
|
||
|
||
## v0.18.0 — 2026-05-19
|
||
|
||
The "Thor / Series IV ingest adapter" release. Seismo-relay can now accept event files from Instantel Micromate Series IV (Thor) units alongside the existing MiniMate Plus (Series III) Blastware pipeline.
|
||
|
||
### Added — Thor (Series IV) IDF ingest
|
||
|
||
- **`POST /db/import/idf_file`** (`sfm/server.py`) — multipart upload endpoint for `.IDFH` (histogram) and `.IDFW` (waveform) event files plus their `.IDFH.txt` / `.IDFW.txt` ASCII sidecars. Mirrors the shape of `/db/import/blastware_file`: pairing by filename, optional `serial` query hint, per-file outcome reporting.
|
||
- **`sfm/idf_ascii_report.py`** — parser for Thor's TXT sidecars (verified against 1,014 real-world samples). Extracts device-authoritative PPV, ZC Freq, Peak Vector Sum, Mic PSPL, calibration date, firmware version, sensor self-check results, and project/client/operator strings.
|
||
- **`WaveformStore.save_imported_idf()`** (`sfm/waveform_store.py`) — stores Thor binaries verbatim in `<root>/<serial>/<filename>`, writes a `.sfm.json` sidecar with `source.kind = "idf-import"` and the full parsed report under `extensions.idf_report`. Reuses the existing `events` table — Thor events dedupe on (serial, timestamp) and surface in `/db/events` alongside BW events.
|
||
- **`tests/test_idf_ascii_report.py`** — parser tests against the `thor-watcher/example-data/` corpus.
|
||
|
||
### Changed
|
||
|
||
- `event_to_sidecar_dict()` (`minimateplus/event_file_io.py`) allow-list for `source_kind` now includes `"idf-import"` so the existing sidecar machinery can carry Thor imports.
|
||
- Bumped `pyproject.toml` version to `0.18.0`.
|
||
|
||
### Companion release
|
||
|
||
This release ships alongside **thor-watcher v0.3.0**, which adds the SFM forwarder that targets the new `/db/import/idf_file` endpoint. Operators flip the switch in thor-watcher's new "SFM Forward" Settings tab; events POST to seismo-relay just like the series3-watcher BW forwarder does today.
|
||
|
||
---
|
||
|
||
## v0.17.0 — 2026-05-17
|
||
|
||
The "field rescue + DB management" release. Hardened against units that are stuck in a runaway call-home loop, and added an operator-facing path for purging bogus events that those same units dump into the DB before recovery. All work in this release was driven by the BE9558H incident (full incident log + recovery procedure at `docs/runbooks/wedged_unit_recovery.md`).
|
||
|
||
### Added — wedged-unit recovery toolkit
|
||
|
||
A toolkit for breaking the call-home loop on a misbehaving unit whose firmware is too busy to keep up with normal request/response handshakes. Tested in production against BE9558H (16 May 2026) — a unit with a stuck-triggered Long-axis geophone that had been call-homing the office BW ACH server every 30 seconds for hours. Endpoints layered from "single attempt" to "siege mode" to suit different contention levels:
|
||
|
||
- **`GET /device/events/storage_range`** — SUB 0x06 probe. POLL + one read; ~2s. Returns first/last event keys and an `is_empty` flag. Use to triage whether a unit has stored events without invoking the slow `count_events()` 1E/1F chain (which choked on BE9558H's corrupted event chain).
|
||
- **`GET /device/events/index`** — SUB 0x08 probe. POLL + one read; ~2s. Returns the lifetime event counter (does NOT decrement on erase — use `storage_range` for "right now" state).
|
||
- **`POST /device/events/erase`** — full erase sequence `0xA3 → 0x1C → 0x06 → 0xA2` (confirmed 2026-04-11, see the protocol reference). Resets event keys to `0x01110000`. Caller's responsibility to disable ACH first if the underlying trigger condition will re-fill the buffer.
|
||
- **`POST /device/rescue`** — one TCP session, short connect+recv timeouts: POLL → disable ACH (compliance config write) → erase events → close. Designed for race-loop usage when the device is busy in another session. 503 on connect-refused, 502 on protocol failure, 200 on full sequence success.
|
||
- **`POST /device/stop_monitoring_blind`** — fire-and-forget Stop Monitoring (SUB 0x97), TCP-only. Dumps `SESSION_RESET + POLL_PROBE + SESSION_RESET + POLL_DATA + 0x97 × repeat` and closes without reading any S3 response. The full POLL preamble is required — write commands without it are silently ignored by the device's protocol parser (false-positive surface area that bit the first version of this endpoint). Use when the device's firmware can't keep up with full request/response but might process inbound bytes at its own pace.
|
||
- **`POST /device/stop_monitoring_spam`** — server-side hammer loop, duration-bounded. Open TCP → write the same blind payload → close → repeat as fast as possible until `duration_s` elapses. Configurable `connect_timeout` (default 500ms) and `repeat` (frames per session). Reports `sent_ok`, `connect_failed`, `write_failed`, `rate_attempts_per_s`. Clamped to 5min duration.
|
||
- **`POST /device/stop_monitoring_slow_drip`** — opposite of spam. Open ONE TCP session, drip the wake handshake + stop frames at `interval_s` (default 3s) for `duration_s` (default 120s, max 10min). Each drip is ~23 bytes — well under any UART FIFO size. Opportunistically drains any inbound bytes the device sends back; `bytes_received > 0` in the response strongly suggests the device has started talking and the session is healthy. **This is the endpoint that saved BE9558H.** Spam mode had been overrunning the device's UART FIFO; slow drip stayed under it.
|
||
- **Six rescue scripts** under `scripts/` — thin bash wrappers around the endpoints, default `SFM_BASE_URL=http://localhost:8200` (direct, not via Terra-View proxy whose 60s timeout would cut off the longer endpoints):
|
||
- `rescue_device.sh` — race-loop wrapper for `/device/rescue`
|
||
- `blind_stop.sh` — race-loop wrapper for `/device/stop_monitoring_blind`
|
||
- `spam_stop.sh` — single-call burst hammer
|
||
- `slow_drip.sh` — single-call held-session drip
|
||
- `watch_unit.sh` — passive periodic reachability check (every N min, logs to file), useful for unattended overnight monitoring of a wedged unit
|
||
- **`docs/runbooks/wedged_unit_recovery.md`** — symptoms, quick-reference recovery procedure, the modem-layer mechanism (Sierra Wireless serial-port mode-flipping is the real failure mode — not the device firmware), and a table of "why simpler approaches don't work" so the next incident skips the dead ends.
|
||
|
||
### Added — operator event DB management
|
||
|
||
Endpoints powering Terra-View's new `/admin/events` page (v0.12.0). Designed for purging bogus events from a unit that's been forwarding them in bulk (e.g. a stuck-triggered seismograph dumping hundreds of junk events before it's recovered).
|
||
|
||
- **`DELETE /db/events/{event_id}`** — hard-delete one event row. Also unlinks the associated blastware binary (`.AB0*`), `.a5.pkl`, `.sfm.json` sidecar, and `.h5` clean-waveform files via the WaveformStore. Returns the per-file removal status. 404 if the event doesn't exist.
|
||
- **`POST /db/events/delete_bulk`** — filter-based or id-list-based bulk delete with safety rails:
|
||
- Filters (`serial`, `from_dt`, `to_dt`, `false_trigger`) combine with AND; same semantics as `GET /db/events`. `ids` is an additional inclusion list. Refuses to run with no filters (would wipe the whole table — raises 422).
|
||
- `confirm` must be `true` to actually delete. Otherwise returns a dry-run summary (`status: "dry_run"`, `matched: N`, `sample_serials: [...]`).
|
||
- `max_rows` (default 10,000) caps how many rows can be deleted by-filter in one call. If exceeded, returns `status: "too_many"` with a hint to narrow or raise the cap. Bypassed when only `ids` is supplied.
|
||
- **`_cleanup_event_files(row)`** helper in `sfm/server.py` — best-effort `unlink()` of all four sidecar paths derived from the row's `blastware_filename`. Logged at WARN if a path exists but unlink fails; the DB row deletion still proceeds.
|
||
- **`SeismoDb.delete_event(id)` and `SeismoDb.delete_events_bulk(...)`** in `sfm/database.py` — both return the deleted row dict(s) so callers can do file cleanup. `delete_events_bulk` raises `ValueError` if no filters are supplied.
|
||
|
||
### Changed
|
||
|
||
- **Default protocol recv timeout dropped from 30s → 10s** in `_build_client()`. The unit usually responds in well under a second over cellular; 10s leaves comfortable headroom for retransmits while failing reasonably fast when a unit is wedged. The two endpoints that perform full 5A waveform downloads still pass `timeout=120.0` explicitly so multi-minute event transfers are unaffected.
|
||
- **`_build_client()` now accepts an optional `connect_timeout`** (TCP-only) so rescue / race-loop endpoints can fail fast on busy modems without affecting the protocol-level recv timeout.
|
||
|
||
### Fixed
|
||
|
||
- **`GET /device/monitor/status` returned HTTP 500 + uncaught traceback when the device was unresponsive**. The retry-on-`Exception` inner block let the second `client.poll()`'s `ProtocolError` propagate out of the handler. Now wrapped in proper try/except — returns 502 with `{"detail": "Protocol error: No S3 frame received within 10.0s ..."}` on timeout, 502 on connection errors, 500 only for genuinely unexpected exceptions.
|
||
|
||
### Migration
|
||
|
||
No schema changes. No data migration required.
|
||
|
||
If you've been running a previous version against a wedged unit and accumulated bogus events, the new `/admin/events` page in Terra-View v0.12.0 (or direct `POST /db/events/delete_bulk` with `confirm: true`) is the cleanup tool. Watcher state on the upstream DL2 PC does NOT need separate cleaning — the watcher's `sfm_forwarded.json` keys on file sha256 and won't re-forward the same files.
|
||
|
||
### Pairing
|
||
|
||
This release pairs with **Terra-View v0.12.0**, which adds the `/admin/events` UI that consumes the new bulk-delete endpoints, the bulk false-trigger flagging on `/unit/{id}`, and the field-deployment workflow that uses the same `series3-watcher` → SFM ingest path as before.
|
||
|
||
---
|
||
|
||
## v0.16.1 — 2026-05-14
|
||
|
||
### Fixed
|
||
|
||
- **`record_type` always "Waveform" for forwarded events.** `read_blastware_file()` hardcoded `ev.record_type = "Waveform"` regardless of the file's actual type. The watcher-forward pipeline (the main BW ACH ingest path) compounds this by parsing files from a tmp path with a `.bw` suffix, so even a filename-based fallback inside the parser still wouldn't see the original extension. Now:
|
||
|
||
1. New `derive_record_type_from_filename(filename)` helper in `minimateplus/event_file_io.py` derives the type from the LAST character of the filename's extension (V10.72+ AB0T scheme: `H`=Histogram, `W`=Waveform, `M`=Manual, `E`=Event, `C`=Combo). Falls back to `"Waveform"` for old S338 firmware (3-char extensions ending in `0`) and any unrecognized suffix.
|
||
2. `read_blastware_file()` now calls the helper with its `path.name` so direct callers (the `--dry-run` path in `scripts/import_bw.py`, tests, ad-hoc scripts) get the right value automatically.
|
||
3. `WaveformStore.save_imported_bw()` overrides `ev.record_type` with the **original** filename's derived type after parsing (the tmp file inside the parser doesn't carry the original extension). This is the path the live watcher-forwarder hits, so the DB column now reflects the actual event type going forward.
|
||
|
||
Events ingested before this fix are stuck with `record_type="Waveform"` in the DB; a one-off backfill (`UPDATE events SET record_type = ... WHERE blastware_filename LIKE '%H'`) would fix them retroactively if desired. Terra-view's event modal also derives client-side from the filename, so the UI already shows the correct type for old events even without the backfill.
|
||
|
||
---
|
||
|
||
## v0.16.0 — 2026-05-11
|
||
|
||
The "BW ACH ingestion" release. When paired with **series3-watcher v1.5.0**, every Blastware ACH event (binary + `_ASCII.TXT` report) lands in SeismoDb with device-authoritative peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data — without depending on the still-undecoded waveform body codec. This is the end-to-end product win discussed in v0.15.0's "out of scope" notes: sortable / filterable monthly-summary review of historical events, populated from the BW ASCII export rather than re-decoded samples.
|
||
|
||
### Added — `/db/import/blastware_file` rich-metadata ingestion
|
||
|
||
- **Paired BW ASCII reports.** The endpoint now accepts the `<binary>_<ext>_ASCII.TXT` partner BW writes alongside each event. Pairing handles both filename conventions: ACH (`M529LK44_AB0_ASCII.TXT`) and manual-export (`M529LK44.AB0.TXT`). When both present, ACH wins.
|
||
- **`minimateplus/bw_ascii_report.py`** (new) — parser + `BwAsciiReport` dataclass for BW's per-event ASCII export. Handles every field BW writes: identity, trigger config, per-channel PPV / ZC Freq / Time of Peak / Peak Acceleration / Peak Displacement, Peak Vector Sum + time, MicL PSPL / Time of Peak / ZC Freq, sensor self-check (Test Freq / Test Ratio / Test Amplitude / Pass-Fail per channel), monitor log, PC SW version.
|
||
- **Position-based user-notes parsing.** BW's Compliance Setup → Notes tab labels (Project / Client / User Name / Seis Loc) are *operator-editable* — an operator can rename them to "Building:", "Site Address:", etc. Rather than maintain a label-spelling map, the parser uses positional matching between the `Units :` and `Geo Range :` anchors in the ASCII output. The four canonical slots (project / client / operator / sensor_location) populate by position regardless of label; the original labels BW wrote are preserved in `report.user_note_labels` for downstream UIs (terra-view) to display verbatim.
|
||
- **`bw_report` sidecar block.** New top-level block in `.sfm.json` carrying the parsed BW report (trigger config, peaks with per-channel stats, mic block, sensor_check, monitor_log, PC SW version, operator-label labels).
|
||
- **`apply_report_to_event(event, report)` helper.** Overlays the report's device-authoritative fields onto an in-memory `Event` so `SeismoDb.insert_events()` writes correct DB columns instead of the broken-codec values from `_peaks_from_samples()`.
|
||
|
||
### Fixed — three compounding bugs that left forwarded events with garbage data
|
||
|
||
- **Import endpoint inserted under `serial="UNKNOWN"`.** `_serial_from_event(ev)` was a stub that always returned `None`; the BW-filename-decoded serial that `WaveformStore` had already resolved was never surfaced to `db.insert_events`. Now uses `rec["serial"]` as the authoritative source. `scripts/repair_unknown_serials.py` repairs existing DB rows.
|
||
- **`/db/units` ignored events from non-ACH ingest paths.** `query_units()` only aggregated from `ach_sessions` — events that arrived via `save_imported_bw()` were never visible in the fleet overview even though they populated `events` correctly. Now unions both tables.
|
||
- **Re-imports left stale DB rows.** The `IntegrityError` handler in `insert_events()` only refreshed filename / sidecar columns when a duplicate `(serial, timestamp)` arrived. Peak values, project info, sample_rate, record_type stayed locked at whatever the first (often broken-codec) insert wrote. Now the upsert path refreshes every device-authoritative column from the new data while preserving `false_trigger` and immutable fields (`id`, `created_at`).
|
||
- **Server-side TXT pairing only knew the legacy convention.** The endpoint stripped `.TXT` and looked up `<binary>` — which works for manual exports (`<binary>.TXT`) but not BW ACH (`<stem>_<ext>_ASCII.TXT`). Reports were arriving in the multipart but silently dropped. Now recognises both conventions and registers each report under all matching binary names.
|
||
|
||
### Migration
|
||
|
||
For existing deployments where events were forwarded by an older watcher (broken pairing) or imported during the UNKNOWN-bucketing window:
|
||
|
||
1. `python -m scripts.repair_unknown_serials --db <path> --apply` to re-attribute `serial="UNKNOWN"` rows.
|
||
2. Delete the watcher's `sfm_forwarded.json` state file and let it re-forward. The server's upsert path will refresh the existing DB rows with the report's authoritative values.
|
||
3. Operator review state (`false_trigger`, sidecar `review` block) is preserved across the re-import.
|
||
|
||
## v0.15.0 — 2026-05-07
|
||
|
||
### Added
|
||
|
||
- **Layered event storage architecture.** Each event now lands as four
|
||
files in the per-serial waveform store, each with a clear role:
|
||
|
||
- `<filename>` — the Blastware-readable binary (BW file). Untouched.
|
||
- `<filename>.a5.pkl` — the raw 5A frames (regenerative source).
|
||
- `<filename>.h5` — clean per-channel waveform arrays in physical
|
||
units (in/s for geo, psi for mic) plus event metadata (HDF5 with
|
||
gzip compression). This is the canonical format for downstream
|
||
analysis tools.
|
||
- `<filename>.sfm.json` — the modern review/metadata sidecar (peaks,
|
||
project, source provenance, review state, extensions).
|
||
|
||
SQLite (`seismo_relay.db`) is the searchable index over all four.
|
||
|
||
- **Plot-ready waveform JSON (`sfm.plot.v1`).** The `/device/event/{idx}/waveform`
|
||
and `/db/events/{id}/waveform.json` endpoints now return samples in
|
||
physical units with explicit time-axis metadata, peak markers, and
|
||
per-channel unit hints — no more guessing the ADC-to-velocity scale
|
||
client-side. The webapp waveform viewer was rewritten to consume
|
||
this shape.
|
||
|
||
- **In-app waveform viewer accuracy fix.** The standalone SFM webapp
|
||
viewer was scaling geophone amplitudes by `geoAdcScale / 32767`
|
||
(≈ 6.206 / 32767), where `geoAdcScale = 6.206053` is the device's
|
||
*in/s per V* hardware constant — not the ADC-counts-to-velocity
|
||
factor. This silently scaled every plot ~38% too low for Normal-range
|
||
geophones (the correct full-scale is 10.0 in/s, or 1.25 in/s for
|
||
Sensitive). Conversion is now done server-side using the geo_range
|
||
from compliance config; the client just plots.
|
||
|
||
- New `sfm/event_hdf5.py` module: `write_event_hdf5()`,
|
||
`read_event_hdf5()`, plus a plot-JSON helper.
|
||
- Backfill script extended to also emit `.h5` for existing events.
|
||
|
||
### Dependencies
|
||
|
||
- Added `h5py>=3.10` and `numpy>=1.24` for the HDF5 storage layer.
|
||
- Added `python-multipart>=0.0.7` (required by FastAPI for the
|
||
`/db/import/blastware_file` endpoint introduced in this release).
|
||
|
||
---
|
||
|
||
## v0.14.3 — 2026-05-05
|
||
|
||
### Fixed
|
||
|
||
- **`build_5a_frame` — DLE-stuffing rule for 0x10 bytes in params (the
|
||
long-standing >1-sec event 0 "won't open in BW" bug).**
|
||
|
||
Previously `build_5a_frame` wrote params bytes RAW with no DLE stuffing,
|
||
based on the incorrect assumption that the device handled all `0x10`
|
||
bytes in params literally. It does not. The device's actual de-stuffing
|
||
rule for the params region is:
|
||
|
||
- `10 10` → de-stuffs to `10`
|
||
- `10 02/03/04` → kept literal (inner-frame markers)
|
||
- `10 X` for other X → de-stuffs to just `X` (drops the `0x10`)
|
||
|
||
When the counter passed in params has `0x10` in the high byte (e.g.
|
||
counter=`0x1000` produces params bytes `... 10 00 ...`), the device
|
||
silently corrupts the request to counter=`0x__00` and responds with
|
||
whatever lives at that wrong address. For counter=0x1000 the wrong
|
||
address was 0x0000, so the response was a copy of the file header +
|
||
STRT record. That STRT block then got embedded in the assembled body
|
||
at file offset `0x1016`, and Blastware refused to open the file
|
||
(interprets the second STRT as a malformed multi-event file).
|
||
|
||
This explains the entire >1-sec event-0 failure pattern:
|
||
|
||
- 1-sec events have `end_offset < 0x1000`, so the chunk walk never
|
||
requests counter `0x10__` and the bug never triggers.
|
||
- 2-sec / 3-sec / longer events all need a chunk at counter `0x1000`
|
||
(and longer events also need `0x1200`, `0x1400`, etc., none of which
|
||
have `0x10` in the high byte except `0x1000`). Just one corrupted
|
||
response is enough to embed STRT in the body and break the file.
|
||
|
||
Verified against BW 5-1-26 "copy 3sec" capture: all 17 5A request
|
||
frames (probe + 2 metadata pages + 13 sample chunks + TERM) now match
|
||
BW's wire output **byte-for-byte**, including the doubled `10 10 00`
|
||
for counter=0x1000.
|
||
|
||
### Notes
|
||
|
||
- `0x10` bytes in `offset_hi` (the standalone offset field at body[5])
|
||
are still written RAW — confirmed correct per the 1-2-26 capture.
|
||
- BW's actual encoding of `10 02` / `10 04` for meta pages 0x1002 /
|
||
0x1004 is *not* doubled — it relies on the device keeping `10 02`
|
||
and `10 04` as literal pairs. This is preserved by the fix.
|
||
|
||
---
|
||
|
||
## v0.14.2 — 2026-05-04
|
||
|
||
### Fixed
|
||
|
||
- **`blastware_file.py` — removed harmful "duplicate header+STRT" strip.**
|
||
The v0.13.x strip logic was matching the byte sequence `00 12 03 00 STRT`
|
||
in legitimate waveform data — sample chunks at counter `0x1000` and
|
||
beyond often contain those bytes coincidentally — and zeroing 25 bytes
|
||
of valid samples per match. This is why event 0 (event-1 case in the
|
||
protocol) downloads of >1-sec recordings always failed in BW: the strip
|
||
destroyed real data at body offset `0x1012..0x102B` and propagated
|
||
alignment differences through the rest of the body. Sub-1-sec events
|
||
worked because their `end_offset` was below `0x1002`, so no sample
|
||
chunks landed in the metadata-page region and the strip's needle never
|
||
matched. Verified fix by re-feeding the BW 5-1-26 "copy 3sec" capture's
|
||
A5 frames into the file builder: output is now byte-identical to BW's
|
||
saved `M529LKIQ.G10` reference (8708 bytes, 0 differences).
|
||
- BW already concatenates frame contributions in stream order without
|
||
any de-duplication; SFM now does the same.
|
||
|
||
---
|
||
|
||
## v0.14.0 — 2026-05-02
|
||
|
||
### Changed (major rewrite)
|
||
|
||
- **`read_bulk_waveform_stream` — STRT-bounded chunk walk.** Replaces the
|
||
earlier `0x0400`-step / `max(key4[2:4], 0x0400)` chunk-counter formula,
|
||
which over-read ~5× past the actual event end into post-event circular-
|
||
buffer garbage. The new walk:
|
||
|
||
1. Probe at `counter = start_offset` (event 1: `0x0000`; event N:
|
||
`cur_key[2:4]`).
|
||
2. Parse `end_offset` from the STRT record at `data[17]` of the probe
|
||
response (`end_key[2:4]` field).
|
||
3. For event 1 only, read the two fixed metadata pages at counter
|
||
`0x1002` and `0x1004` — these contain the global session-start
|
||
compliance setup (Project / Client / User Name / Seis Loc /
|
||
Extended Notes ASCII strings). Continuation events skip these
|
||
(BW caches them across the session).
|
||
4. Walk sample chunks at **`0x0200` increments (NOT `0x0400`)**, bounded
|
||
by `end_offset` — the loop exits when
|
||
`next_chunk_counter + 0x0200 > end_offset`.
|
||
5. Send the proper TERM frame (see new `bulk_waveform_term_v2()`) with
|
||
`offset_word = end_offset - next_boundary` and
|
||
`params[2:4] = next_boundary BE`. The TERM response carries the
|
||
partial last chunk + 26-byte file footer.
|
||
|
||
- **New helpers:** `bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`
|
||
and `parse_strt_end_offset(a5_data)` in `minimateplus.framing`.
|
||
|
||
- **`stop_after_metadata` / `extra_chunks_after_metadata` kwargs are now
|
||
no-ops** under the v0.14.x walk. They are retained on the
|
||
`read_bulk_waveform_stream` signature for backward compatibility but log a
|
||
DEBUG line when set. The old "scan for `b'Project:'` and stop one chunk
|
||
later" workaround is obsolete — the loop is deterministically bounded by
|
||
the STRT-derived `end_offset`.
|
||
|
||
- **Project / Client / User Name / Seis Loc string source corrected.**
|
||
These come from the dedicated metadata pages at counter `0x1002` /
|
||
`0x1004`, not from "A5 frame 7" of the sample-chunk stream. The
|
||
earlier "A5 frame 7" claim was an artifact of the broken `0x0400`-step
|
||
walk where the bad counter formula coincidentally landed sample-chunk
|
||
fi=7 on top of the 0x1002 metadata page.
|
||
|
||
### Verified
|
||
|
||
- Three independent BW MITM captures (4-27-26 + 5-1-26 + 5-4-26) confirm
|
||
the new walk matches BW's behaviour event-for-event.
|
||
- `end_offset` values verified across 3 events: `0x1ABE` (4-27-26 2-sec),
|
||
`0x21F2` (5-1-26 3-sec), `0x417E` (5-1-26 event-2).
|
||
|
||
### Notes
|
||
|
||
- Earlier v0.13.0 / v0.13.1 / v0.13.2 entries describe partial steps along
|
||
the way (some of the file builder fixes, filename bugs, etc.) that were
|
||
superseded by the full rewrite. Treat this v0.14.0 entry as the
|
||
definitive landing point for the corrected SUB 5A protocol.
|
||
|
||
---
|
||
|
||
## v0.14.1 — 2026-05-04
|
||
|
||
### Fixed
|
||
|
||
- **`read_bulk_waveform_stream` — event-N probe counter off-by-`0x46`.**
|
||
Continuation events (start_key[2:4] != 0) were being probed at counter
|
||
`start_offset + 0x0046` instead of just `start_offset`. In the iteration
|
||
walk, `cur_key` from 1F is already the off=0x46 WAVEHDR record key, so the
|
||
earlier formula effectively double-counted the WAVEHDR offset. The probe
|
||
landed one WAVEHDR past the actual event start, the response no longer
|
||
contained the STRT record at byte 17, `parse_strt_end_offset` returned
|
||
`None`, and the chunk loop fell back to the `max_chunks=128` cap — walking
|
||
~110 chunks of post-event circular-buffer garbage. Verified against the
|
||
5-1-26 "copy 2nd address" and 5-4-26 BW 2-sec event captures: BW probes
|
||
counter=`0x2238` with key=`01112238` and STRT is present at byte 17 of
|
||
the response (end_offset=`0x417E`).
|
||
- **CLAUDE.md / docs/instantel_protocol_reference.md** — corrected the
|
||
event-N section to clarify that `start_key` in those formulas is the
|
||
off=0x46 key, not the off=0x2C boundary key, and removed the spurious
|
||
`+0x46` from the chunk-walk pseudocode.
|
||
|
||
---
|
||
|
||
## v0.13.2 — 2026-05-01
|
||
|
||
### Fixed
|
||
|
||
- **`_extract_record_type` — third 0C-record header format ("short", 8 bytes).**
|
||
A live SFM download against BE11529 produced files named `M5290000.000`
|
||
(zero-stamped) because the 0C waveform record's first bytes were
|
||
`01 05 07 ea ...` — neither the 9-byte single-shot layout (`0x10` at byte 1)
|
||
nor the 10-byte continuous layout (`0x10` at bytes 0 and 2). Investigation
|
||
showed this is a third format observed in the wild: an 8-byte header with no
|
||
marker bytes at all (`[day][month][year_BE:2][unknown][hour][min][sec]`).
|
||
The detection logic now scans the year (uint16 BE) at byte 2 / byte 3 / byte
|
||
4 and picks whichever offset returns a sensible year (2015–2050) — each
|
||
format has the year at a unique position so this disambiguates cleanly.
|
||
- New format → `event.record_type = "Waveform (Short)"`,
|
||
`Timestamp.from_short_record()`.
|
||
- Existing single-shot and continuous parsers unchanged.
|
||
- The user's event from May 1, 2026 13:21:37 now correctly resolves to a
|
||
filename like `M529LKIQ.G10` instead of `M5290000.000`.
|
||
|
||
### Added
|
||
|
||
- `Timestamp.from_short_record(data)` — decodes the 8-byte header.
|
||
- `_detect_record_format(data)` — internal helper returning
|
||
`"single_shot" / "continuous" / "short" / None` via year-position scan.
|
||
|
||
---
|
||
|
||
## v0.13.1 — 2026-05-01
|
||
|
||
### Fixed
|
||
|
||
- **`_extract_record_type` — Continuous-mode record headers misclassified as Unknown.**
|
||
In single-shot mode the 0C waveform record's 9-byte header puts the sub_code
|
||
marker `0x10` at byte 1, with the day at byte 0. In Continuous mode the
|
||
header is 10 bytes with the marker at byte 0 *and* byte 2, and the day at
|
||
byte 1. Previous logic only inspected byte 1 and treated any value other
|
||
than `0x10` / `0x03` as `"Unknown"`, which prevented `event.timestamp` from
|
||
being populated for any continuous-mode event whose day-of-month wasn't
|
||
exactly 3 or 16. As a downstream effect, `blastware_filename()` saw
|
||
`event.timestamp == None`, fell back to `stem="0000"` / `ab="00"`, and
|
||
produced filenames like `M5290000.000`. Discovered from a live SFM run on
|
||
BE11529 in continuous mode (day-of-month = 5).
|
||
Now disambiguates by checking BOTH byte 0 and byte 2: if both are `0x10`,
|
||
it's the 10-byte continuous header; else if byte 1 is `0x10`, it's the
|
||
9-byte single-shot header. Day-of-month no longer matters.
|
||
|
||
*Superseded by v0.13.2 — the user's actual record uses a third 8-byte format
|
||
with no `0x10` markers, which v0.13.1 still misclassified.*
|
||
|
||
---
|
||
|
||
## v0.13.0 — 2026-05-01
|
||
|
||
### Fixed
|
||
|
||
- **SUB 5A bulk waveform stream — over-read bug for events ≥ 2 sec.**
|
||
`read_bulk_waveform_stream` was walking the chunk counter past the actual
|
||
end of the event, picking up post-event circular-buffer garbage that
|
||
corrupted reconstructed Blastware files for any waveform > ~1 sec. The
|
||
loop now extracts the event's `end_offset` from the STRT record at
|
||
`data[23:27]` of the probe response and stops the chunk walk when the next
|
||
counter would step past it. Verified against three BW MITM captures
|
||
(4-27-26 + 5-1-26): 2-sec event drops from 37 over-read chunks to 7
|
||
bounded chunks; 3-sec drops to 9; non-zero-start "event 2" drops to 9.
|
||
|
||
### Added
|
||
|
||
- `framing.bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)` —
|
||
computes the corrected SUB 5A TERM frame's `(offset_word, params)` per the
|
||
formula confirmed across all 3 BW captures. Not yet wired into
|
||
`read_bulk_waveform_stream` (the legacy TERM is still used to preserve the
|
||
existing `blastware_file.write_blastware_file` frame-structure expectations);
|
||
available for the next iteration that switches to BW's 0x0200 chunk step.
|
||
- `framing.parse_strt_end_offset(a5_data)` — extracts the event-end pointer
|
||
from the STRT record in an A5 response payload.
|
||
|
||
### Documentation
|
||
|
||
- **CLAUDE.md and `docs/instantel_protocol_reference.md` extensively
|
||
rewritten** to reflect the corrected SUB 5A protocol. See:
|
||
- CLAUDE.md "SUB 5A — chunk counter formula (REWRITTEN 2026-05-01)"
|
||
- CLAUDE.md "SUB 5A — STRT record encodes end_offset"
|
||
- CLAUDE.md "SUB 5A — TERM frame formula"
|
||
- CLAUDE.md "SUB 5A — fixed metadata pages 0x1002 and 0x1004"
|
||
- CLAUDE.md "SUB 0A — WAVEHDR response length distinguishes events from
|
||
boundaries" (0x46 = real event, 0x2C = boundary marker)
|
||
- protocol reference §7.8.5 / §7.8.6 / §7.8.7 / §7.8.8
|
||
- The previous chunk-counter formula (`max(key4[2:4], 0x0400) + (chunk-1) *
|
||
0x0400`) is now marked DEPRECATED and explicitly tagged WRONG with
|
||
pointers to the new sections, so future work doesn't re-derive it.
|
||
|
||
### Known minor diffs vs Blastware (deferred to a follow-up)
|
||
|
||
- We still use the OLD 0x0400 chunk step rather than BW's 0x0200; switching
|
||
also requires updating `blastware_file.write_blastware_file`'s skip values
|
||
and "extra chunk after metadata" logic, which depends on a fresh capture
|
||
to verify.
|
||
- We still use the legacy fixed `offset_word=0x005A` TERM frame rather than
|
||
BW's `end_offset - next_boundary` formula, for the same reason.
|
||
- Two fixed metadata pages at counter `0x1002` and `0x1004` are not yet
|
||
read explicitly; under the current 0x0400 walk their content is reachable
|
||
via the sample chunk that covers buffer addresses `[0x1000, 0x1400)`.
|
||
|
||
---
|
||
|
||
## v0.12.6 — 2026-05-01
|
||
|
||
### Fixed
|
||
|
||
- **`blastware_file.py` — waveform frame classification** — A5 frame classification for
|
||
waveform-only vs header-only frames now uses `frame.record_type` instead of frame index.
|
||
Only waveform frames (0x46) are written to the file body; metadata frames are skipped.
|
||
Fixes spurious data corruption from incorrectly classified frames.
|
||
|
||
- **`s3_analyzer.py` — A5/5A frame naming** — Bulk waveform stream frames (SUB 5A response)
|
||
are now correctly labeled "A5" in analyzer output instead of being conflated with other
|
||
multi-frame responses (SUB A4, E5, etc.).
|
||
|
||
- **`S3FrameParser` — frame terminator detection** — Corrected the bare ETX terminator
|
||
detection. Frame termination is now correctly identified by a standalone `ETX=0x03` byte,
|
||
not by the `DLE+ETX` sequence (which is part of the payload when it appears within a frame).
|
||
|
||
---
|
||
|
||
## v0.12.5 — 2026-04-21
|
||
|
||
### Added
|
||
|
||
- **`seismo_lab.py` — Download tab** — New fourth tab for live wire-byte capture during event
|
||
downloads. Captures both BW→device and device→S3 frames in real time, allowing inspection
|
||
of the 5A bulk stream chunk sequence and frame-by-frame analysis without needing a bridge
|
||
or MITM proxy. Files are saved with user-specified labels for easy tracking.
|
||
|
||
### Changed
|
||
|
||
- **`s3_bridge.py` — raw captures always-on by default** — `--raw-bw` and `--raw-s3` now
|
||
default to `"auto"` instead of `None`. Every bridge session automatically generates
|
||
timestamped `raw_bw_<ts>.bin` and `raw_s3_<ts>.bin` files alongside the `.bin`/`.log`
|
||
session files. Pass `--raw-bw ""` (explicit empty string) to disable if needed.
|
||
|
||
- **`gui_bridge.py` — raw capture checkboxes pre-checked** — Both "BW→S3 raw" and
|
||
"S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names
|
||
the files). Unchecking a box passes `--raw-bw ""` to explicitly disable capture.
|
||
|
||
- **`Bridge tab` — TCP mode added** — Serial/TCP radio toggle allows connection via cellular
|
||
modem (RV50/RV55) instead of direct RS-232. Supports multi-capture design (simultaneous
|
||
Bridge + Analyzer + Download sessions).
|
||
|
||
- **`ach_server.py` — TX capture added (`raw_tx_<ts>.bin`)** — Every ACH inbound session
|
||
now saves both directions: `raw_rx_<ts>.bin` (device → us, S3 side, as before) and
|
||
`raw_tx_<ts>.bin` (us → device, BW side). Both files are usable in the Analyzer.
|
||
TX bytes are buffered in memory until startup handshake succeeds (same as RX), preventing
|
||
scanner probes from creating empty files.
|
||
|
||
---
|
||
|
||
## v0.12.4 — 2026-04-21 (protocol analysis / docs only — no code changes)
|
||
|
||
### Discovered
|
||
|
||
- **compliance_raw is wire-encoded, not logical bytes** — `read_compliance_config()` returns
|
||
bytes that include DLE prefix bytes (`0x10`) before any `0x03` values (because S3FrameParser
|
||
preserves DLE+ETX inner-frame pairs as two literal bytes). The previous CLAUDE.md claim that
|
||
"S3FrameParser handles this transparently so compliance_raw contains logical bytes" was wrong.
|
||
|
||
- **anchor-9 behavior per recording mode** (confirmed from 4-20-26 BW write captures):
|
||
- Single Shot (0x00) / Continuous (0x01): anchor-9 = `0x00`
|
||
- Histogram (0x03): anchor-9 = `0x10` — the E5 DLE prefix for the `0x03` recording_mode byte
|
||
- Histogram+Continuous (0x04): anchor-9 = `0x10` — an actual stored config byte for this mode
|
||
Anchor position shifts by ±1 when recording_mode = `0x03` due to the extra DLE byte; the
|
||
dynamic anchor search (`buf.find(ANCHOR, 0, 150)`) handles this correctly without code changes.
|
||
|
||
- **Write frame ETX escaping** — BW escapes `0x03` bytes in write frame data as `0x10 0x03`
|
||
on the wire. Our `build_bw_write_frame` sends data bytes raw without ETX escaping. Device
|
||
accepts our raw writes for all tested modes. Hypothesis: device write parser uses the
|
||
offset/length field for frame boundaries, not ETX scanning, making ETX escaping optional.
|
||
Histogram mode (recording_mode = 0x03) write via SFM from a non-Histogram starting state
|
||
not yet tested.
|
||
|
||
- **BW write payload vs E5 read payload are byte-identical** around the anchor region (confirmed
|
||
by comparing 3-11-26 BW TX and S3 captures). BW does NOT strip DLE prefix bytes before writing;
|
||
it round-trips the wire-encoded bytes verbatim with only the modified fields changed.
|
||
|
||
- **Capture folder content catalogued** — see CLAUDE.md "BW capture reference" table for a
|
||
summary of all available protocol captures and their contents.
|
||
|
||
---
|
||
|
||
## v0.12.3 — 2026-04-20
|
||
|
||
### Added
|
||
|
||
- **Auto Call Home config protocol** — Full read/write/decode/encode pipeline for the
|
||
device's Remote Access → Setup Unit ACH settings, confirmed from 4-20-26 call home
|
||
settings captures.
|
||
|
||
**Protocol (new):**
|
||
- `SUB 0x2C` — Call Home Config READ (response `0xD3`); two-step read; data offset
|
||
`0x7C` = 124; raw payload 125 bytes (1-byte longer than DATA_LENGTH due to DLE-escaped
|
||
`\x10\x03` at raw[117:119] representing num_retries = 3)
|
||
- `SUB 0x7E` — Call Home Config WRITE (response `0x81`); 127-byte payload (125-byte read
|
||
payload + `\x00\x00`); offset = `data[1]+2 = 0x7E`; write format (DLE-aware checksum)
|
||
- `SUB 0x7F` — Call Home WRITE CONFIRM (response `0x80`); no data
|
||
|
||
**Field map (confirmed from 10-frame BW TX diff):**
|
||
- `raw[5]` — auto_call_home_enabled (bool)
|
||
- `raw[6:46]` — dial_string (40-byte null-padded ASCII)
|
||
- `raw[87]` — after_event_recorded (bool)
|
||
- `raw[91]` — at_specified_times (bool)
|
||
- `raw[93]` — time1_enabled / `raw[101]` — time1_hour / `raw[102]` — time1_min
|
||
- `raw[95]` — time2_enabled / `raw[105]` — time2_hour / `raw[106]` — time2_min
|
||
- `raw[117:119]` — `\x10\x03` (DLE-escaped 0x03 = num_retries value 3)
|
||
- `raw[120]` — time_between_retries_sec / `raw[122]` — wait_for_connection_sec / `raw[124]` — warm_up_time_sec
|
||
|
||
**Library (`minimateplus/`):**
|
||
- `models.py` — `CallHomeConfig` dataclass (14 fields; `raw` bytes preserved for
|
||
round-trip writes)
|
||
- `protocol.py` — `SUB_CALL_HOME = 0x2C`, `SUB_CALL_HOME_WRITE = 0x7E`,
|
||
`SUB_CALL_HOME_CONFIRM = 0x7F`; `read_call_home_config()`, `write_call_home_config()`
|
||
- `client.py` — `get_call_home_config()`, `set_call_home_config()`,
|
||
`_decode_call_home_config()` (handles DLE prefix at raw[117]),
|
||
`_encode_call_home_config()` (patches in-place; raises `ValueError` if hour/min = 3)
|
||
|
||
**REST API (`sfm/server.py`):**
|
||
- `GET /device/call_home` — reads and decodes call home config from device
|
||
- `POST /device/call_home` — reads, patches specified fields, writes back to device
|
||
- `CallHomeConfigBody` Pydantic model with 9 optional writable fields
|
||
|
||
**Web UI (`sfm/sfm_webapp.html`):**
|
||
- New "Call Home" tab with enable flag, dial string (read-only), after-event trigger,
|
||
at-specified-times flag, two time slots (enable + HH:MM each), and read-only retry
|
||
settings (num_retries, time_between_retries_sec, wait_for_connection_sec,
|
||
warm_up_time_sec)
|
||
- "Read from Device", "Write to Device", "Clear Form" action buttons
|
||
- Client-side guard: rejects hour or minute value equal to 3 with a clear message
|
||
explaining the DLE-encoding limitation
|
||
|
||
---
|
||
|
||
## v0.12.2 — 2026-04-20
|
||
|
||
### Added / Fixed
|
||
|
||
- **Geophone sensitivity / maximum range field confirmed** — 4-20-26 geo sensitivity
|
||
captures (1.25 in/s vs 10 in/s) diffed across all three SUB 71 write chunks and both
|
||
E5 read payloads. The `geo_range` uint8 field per channel is now fully confirmed:
|
||
- E5 read offset: `channel_label + 33`; SUB 71 write offset: `channel_label + 29`
|
||
- `0x00` = Normal 10.000 in/s (standard gain); `0x01` = Sensitive 1.250 in/s (high gain)
|
||
- **Correction:** previous hypothesis (`channel_label+20`, `0x01`=Normal) was wrong.
|
||
`channel_label+20` reads `0x01` on ALL captures regardless of range — not this field.
|
||
- `_decode_compliance_config_into`: read offset corrected from `tran_pos+20` → `tran_pos+33`
|
||
- `_encode_compliance_config`: added `geo_range` parameter; writes to Tran/Vert/Long at `+29`
|
||
- `apply_config`: added `geo_range` parameter
|
||
- `POST /device/config`: added `geo_range` to `DeviceConfigBody`
|
||
- Web UI Config tab: added "Maximum Range — Geo" select (Normal / Sensitive)
|
||
- Web UI Device tab: added "Max Range (geo)" row to compliance table
|
||
|
||
- **`recording_mode` + `histogram_interval_sec` confirmed and implemented** (4-20-26 captures)
|
||
- `recording_mode`: uint8 at anchor−8 (E5 read) / anchor−7 (write); enum: 0x00=Single Shot,
|
||
0x01=Continuous, 0x03=Histogram, 0x04=Histogram+Continuous
|
||
- `histogram_interval_sec`: uint16 BE seconds at anchor−4; same offset in read & write;
|
||
valid: 2, 5, 15, 60, 300, 900 (matching Blastware dropdown: 2s, 5s, 15s, 1m, 5m, 15m)
|
||
- Both fields added to `ComplianceConfig`, `_decode_compliance_config_into`,
|
||
`_encode_compliance_config`, `apply_config`, REST API body, and web UI
|
||
|
||
---
|
||
|
||
## v0.12.1 — 2026-04-16
|
||
|
||
### Added
|
||
|
||
- **`sfm/server.py` — `_LiveCache`** — in-memory live device cache that eliminates
|
||
redundant TCP round-trips between web requests. Plain Python dict +
|
||
`threading.Lock`, no extra dependencies.
|
||
|
||
Cache strategy per endpoint:
|
||
|
||
| Endpoint | Strategy |
|
||
|---|---|
|
||
| `GET /device/info` | Indefinite; invalidated by `POST /device/config` |
|
||
| `GET /device/events` | Count-probe fast path — `poll()+count_events()` (~2 s); returns cached data if event count is unchanged; full download only when new events are detected |
|
||
| `GET /device/monitor/status` | 30-second TTL; invalidated immediately on monitor start/stop |
|
||
| `GET /device/event/{idx}/waveform` | Permanent per-index (waveforms are immutable once recorded) |
|
||
|
||
- **`?force=true`** query param on all cached endpoints — bypasses cache and forces
|
||
a fresh read from the device.
|
||
|
||
- **Cache invalidation hooks** — `POST /device/config` marks device info and events
|
||
stale; `POST /device/monitor/start` and `/stop` evict the monitor status entry
|
||
immediately so the next status poll reflects the actual device state.
|
||
|
||
---
|
||
|
||
## v0.12.0 — 2026-04-13
|
||
|
||
### Added
|
||
|
||
- **`sfm/server.py` — `_LiveCache`** — in-memory live device cache, eliminating
|
||
redundant TCP round-trips between requests. No extra dependencies (plain Python
|
||
dict + threading.Lock). Replaces the SQLAlchemy-based `sfm/cache.py` experiment
|
||
from the `feature/intelligent-caching` branch.
|
||
|
||
Cache behaviour by endpoint:
|
||
|
||
| Endpoint | Cache strategy |
|
||
|---|---|
|
||
| `GET /device/info` | Indefinite; invalidated by `POST /device/config` |
|
||
| `GET /device/events` | Count-probe fast path: quick `poll()+count_events()` (~2s); return cache if count matches; full download only when new events detected |
|
||
| `GET /device/monitor/status` | 30-second TTL; invalidated by monitor start/stop |
|
||
| `GET /device/event/{idx}/waveform` | Permanent per-index (waveforms are immutable) |
|
||
|
||
- **`?force=true` param** on all four cached endpoints — bypasses cache and re-reads
|
||
from device.
|
||
|
||
- **`POST /device/config` cache invalidation** — marks device info + events dirty so
|
||
the next read reflects the new compliance config.
|
||
|
||
- **`POST /device/monitor/start` / `stop` cache invalidation** — evicts the monitor
|
||
status cache entry immediately so the next poll returns the updated state.
|
||
|
||
### Removed
|
||
|
||
- `sfm/cache.py` — SQLAlchemy-based cache from the experimental caching branch.
|
||
Its logic has been ported to the sqlite3-native `_LiveCache` class above.
|
||
`sqlalchemy` is no longer a dependency.
|
||
|
||
---
|
||
|
||
## v0.11.0 — 2026-04-13
|
||
|
||
### Added
|
||
|
||
- **`sfm/database.py` — SeismoDb** — SQLite persistence layer for all ACH data.
|
||
Three tables, all unit-keyed by serial number:
|
||
- `ach_sessions` — one row per inbound call-home: serial, timestamp, peer IP,
|
||
events_downloaded, monitor_entries, duration_seconds
|
||
- `events` — one row per triggered waveform event: serial, waveform_key (dedup),
|
||
timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location
|
||
strings, sample_rate, record_type, false_trigger flag
|
||
- `monitor_log` — one row per monitoring interval: serial, waveform_key (dedup),
|
||
start_time, stop_time, duration_seconds, geo_threshold_ips
|
||
- WAL mode, per-request connections — safe for the single-writer / occasional-reader
|
||
ACH server pattern
|
||
- Deduplication by `(serial, waveform_key)` UNIQUE constraint — re-runs and repeat
|
||
call-homes never produce duplicate rows
|
||
|
||
- **`ach_server.py` — DB integration** — after each successful call-home, writes new
|
||
events and monitor log entries to `seismo_relay.db` then records the session in
|
||
`ach_sessions`. DB write failures are logged as warnings and do not abort the session.
|
||
|
||
- **`sfm/server.py` — DB read endpoints**:
|
||
- `GET /db/units` — distinct serials with last_seen, total_events, total_monitor_entries
|
||
- `GET /db/events` — query events with serial / date range / false_trigger filters
|
||
- `GET /db/monitor_log` — query monitoring intervals
|
||
- `GET /db/sessions` — query ACH call-home sessions
|
||
- `PATCH /db/events/{id}/false_trigger` — flag/unflag false triggers (for review UI)
|
||
|
||
### Architecture
|
||
|
||
- seismo-relay DB is unit-keyed only — no project concepts. Project aggregation is
|
||
terra-view's responsibility via `UnitAssignment` / `DeploymentRecord` + date range
|
||
queries against the SFM DB endpoints.
|
||
- DB file lives at `bridges/captures/seismo_relay.db` by default.
|
||
|
||
---
|
||
|
||
## v0.10.0 — 2026-04-11
|
||
|
||
### Added
|
||
|
||
- **`MiniMateClient.get_monitor_log_entries(skip_keys=None)`** — browse-mode walk
|
||
(`1E → 0A → 1F`) that collects partial records (`0x2C` record type) from the device's
|
||
event list without triggering a full waveform download (no 0C or 5A). Returns
|
||
`list[MonitorLogEntry]`. Each entry represents one continuous monitoring interval where
|
||
no threshold was exceeded.
|
||
|
||
- **`_decode_0a_partial_header(raw_data, index, key4)`** in `client.py` — decodes a SUB
|
||
0x0A response payload whose record type is `0x2C`. Extracts:
|
||
- `start_time` / `stop_time` — two consecutive timestamps; auto-detects 9-byte
|
||
(sub_code=0x10, single-shot) vs 10-byte (sub_code=0x03, continuous) format from
|
||
`raw_data[11]`. Handles a 1-byte gap between the two timestamps that occurs when
|
||
ts1 and ts2 share the same minute:second.
|
||
- `serial` — device serial string found via `b"BE"` anchor scan.
|
||
- `geo_threshold_ips` — trigger level found via `b"Geo: "` anchor scan.
|
||
|
||
- **`MonitorLogEntry` dataclass** in `models.py` — new model for partial records:
|
||
`index`, `key`, `start_time`, `stop_time`, `serial`, `geo_threshold_ips`,
|
||
`raw_header`, and a `duration_seconds` property.
|
||
|
||
- **`read_waveform_header()` return value extended** — now returns `(data_rsp.data, length)`
|
||
(full payload) instead of `(data_rsp.data[11:11+length], length)`. Callers get the
|
||
complete payload including the record-type byte at position 0. Full records use
|
||
`raw_data[11:11+length]` as before; partial records are detected by `raw_data[0] == 0x2C`.
|
||
|
||
- **ACH server: monitor log collection** — after `get_events()`, calls
|
||
`get_monitor_log_entries(skip_keys=seen_keys)` and saves new entries to
|
||
`monitor_log.json` in the session directory. Monitor log keys are included in
|
||
`downloaded_keys` for state persistence (no re-processing on next call-home).
|
||
|
||
- **`_monitor_log_entry_to_dict()`** in `ach_server.py` — serialises a `MonitorLogEntry`
|
||
to a JSON-compatible dict with ISO-format timestamps.
|
||
|
||
### Protocol / Documentation
|
||
|
||
- **SUB 0x0A partial record (0x2C) format confirmed** (✅ 4-11-26 MITM capture, 12 frames):
|
||
- Record type `0x2C` at `raw_data[0]`; length < 64 bytes.
|
||
- Two timestamps at `raw_data[11:]` — start and stop of the monitoring interval.
|
||
- ASCII metadata region after timestamps: `BE<serial>\x00Geo: <float> in/s`.
|
||
- Edge case: 1-byte separator between timestamps when ts1 and ts2 share minute:second.
|
||
- 10-byte timestamp format (sub_code=0x03) signalled by `raw_data[11] == 0x10`.
|
||
|
||
- **Key reuse detection for monitor log entries** — monitor log keys are tracked alongside
|
||
event keys in `ach_state.json` so the ACH server does not re-process them after a
|
||
call-home cycle.
|
||
|
||
---
|
||
|
||
## v0.9.0 — 2026-04-11
|
||
|
||
### Added
|
||
|
||
- **`MiniMateClient.list_event_keys()`** — fast browse-mode walk (1E → 0A → 1F, no waveform
|
||
download) that returns the list of event key hex strings currently stored on the device.
|
||
Used by the ACH server as a cheap pre-check before deciding whether to call `get_events()`.
|
||
|
||
- **`get_events(skip_waveform_for_keys=set(...))`** — new optional parameter. For any key in
|
||
the set the function performs only 0A + 1F(browse) instead of the full
|
||
1E-arm → 0C → POLL×3 → 5A sequence. Eliminates redundant waveform downloads on repeat
|
||
call-homes when the device still holds previously downloaded events.
|
||
|
||
- **`MiniMateClient.delete_all_events()`** — erases all events from device memory using the
|
||
confirmed 4-step sequence:
|
||
- SUB 0xA3 `begin_erase_all` — initiate erase (token=0xFE) → ack 0x5C
|
||
- SUB 0x1C `read_monitor_status` — intermediate status read (Blastware-required)
|
||
- SUB 0x06 `read_event_storage_range` — verify storage state (token=0xFE) → 36-byte response
|
||
- SUB 0xA2 `confirm_erase_all` — commit erase (token=0xFE) → ack 0x5D
|
||
|
||
All four steps confirmed from 4-11-26 MITM capture of a live Blastware ACH session.
|
||
After a successful call, the device's event counter resets to `0x01110000`.
|
||
|
||
- **`MiniMateProtocol` erase methods**: `begin_erase_all()`, `confirm_erase_all()`,
|
||
`read_event_storage_range()` added to `protocol.py` with documented SUB constants
|
||
`SUB_ERASE_ALL_BEGIN = 0xA3` and `SUB_ERASE_ALL_CONFIRM = 0xA2`.
|
||
|
||
- **`bridges/ach_mitm.py`** — transparent TCP-to-TCP MITM proxy. Listens for inbound unit
|
||
connections, connects upstream to a real Blastware ACH server, and saves both directions
|
||
to `raw_bw_<ts>.bin` / `raw_s3_<ts>.bin` files matching the existing capture format.
|
||
Used to capture the 4-11-26 Blastware ACH session including event deletion.
|
||
Usage: `python bridges/ach_mitm.py --bw-host 127.0.0.1 --bw-port 9999 --listen-port 9998`
|
||
|
||
- **ACH server: key-based state tracking** — `ach_state.json` now stores
|
||
`downloaded_keys: [hex_strings]` and `max_downloaded_key: hex_string` per unit instead of
|
||
`event_count: N`. This correctly handles the standard workflow where events are deleted
|
||
from the device after upload — a count-based approach would see `count=0` on the next
|
||
call-home and silently skip new events.
|
||
|
||
- **ACH server: `--clear-after-download` flag** — after a successful download (at least one
|
||
new event saved), erases all events from the device using `delete_all_events()`. Mirrors
|
||
the standard Blastware ACH workflow. On success, `downloaded_keys` and
|
||
`max_downloaded_key` are reset to empty so the next session starts fresh.
|
||
|
||
- **ACH server: post-erase key-reuse detection** — after an external erase (Blastware or
|
||
manual), device keys restart from `0x01110000`, colliding with previously downloaded keys.
|
||
On each browse walk, if `max(device_keys) < max_downloaded_key` (device counter rolled
|
||
back), all device keys are treated as new regardless of `seen_keys`. This also catches
|
||
erases performed by Blastware between our sessions.
|
||
|
||
### Protocol / Documentation
|
||
|
||
- **SUB 0xA3 / SUB 0xA2 — erase-all sequence confirmed** (✅ 4-11-26 MITM capture):
|
||
Both frames use `token=0xFE` at `params[7]` and are standard `build_bw_frame` requests
|
||
(not write-format). Response SUBs follow the standard formula: 0x5C and 0x5D.
|
||
The intermediate 0x1C + 0x06 reads between them are required by Blastware.
|
||
|
||
- **SUB 0x06 — event storage range read confirmed** (✅ 4-11-26 MITM capture):
|
||
Two-step read, data offset = 0x24 (36 bytes). The last 8 bytes of the response contain
|
||
the first and last stored event keys (4 bytes each). After a successful erase, both keys
|
||
read as `01110000` (device-empty state).
|
||
|
||
- **Event key counter resets to `0x01110000` after erase** — confirmed by observing key
|
||
`01110000` on the device immediately after the MITM erase session.
|
||
|
||
---
|
||
|
||
## v0.8.0 — 2026-04-07
|
||
|
||
### Added
|
||
|
||
- **Write pipeline end-to-end** — `push_config_raw(event_index_data, compliance_data,
|
||
trigger_data, waveform_data)` on `MiniMateClient` orchestrates the full
|
||
`68→73 | 71×3→72 | 82→83 | 69→74→72` write sequence.
|
||
|
||
- **`build_bw_write_frame(sub, data, *, offset, params)`** in `framing.py` — dedicated frame
|
||
builder for write commands (SUBs 0x68–0x83). Doubles only the BW_CMD byte; all other
|
||
bytes including offset, params, data, and checksum are written raw. Uses the large-frame
|
||
DLE-aware checksum (`sum(b for b in payload[2:] if b != 0x10) + 0x10) & 0xFF`).
|
||
|
||
- **`MiniMateProtocol` write methods** — `write_event_index()`, `write_compliance()`,
|
||
`write_trigger_config()`, `write_waveform_data()`, `write_confirm()`,
|
||
`start_monitoring()`, `stop_monitoring()`.
|
||
|
||
- **`AchSession` inbound server** (`bridges/ach_server.py`) — accepts call-home TCP
|
||
connections, runs the full handshake + device-info + event-download sequence, saves
|
||
`device_info.json` + `events.json` per session.
|
||
|
||
### Protocol / Documentation
|
||
|
||
- **Write frame format confirmed** (✅ 3-11-26 BW TX capture, all 11 frames): only BW_CMD
|
||
byte `0x10` is doubled; all other bytes sent raw. Standard `build_bw_frame` DLE-stuffing
|
||
is incorrect for write commands.
|
||
- **Write ack responses** confirmed as 17-byte zero-data S3 frames.
|
||
- **Monitoring SUBs 0x96/0x97** confirmed from 4-8-26 capture.
|
||
- **SESSION_RESET signal** (`41 03`) required before POLL for monitoring units.
|
||
- **SUB 0x1C monitoring flag** at `section[1]`: `0x00` = idle, `0x10` = monitoring.
|
||
Confirmed by byte-diff of all 144 data frames in 4-8-26/2ndtry capture.
|
||
|
||
---
|
||
|
||
## v0.7.0 — 2026-04-03
|
||
|
||
### Added
|
||
- **Raw ADC waveform decode — `_decode_a5_waveform(frames_data, event)`** in `client.py`.
|
||
Parses the complete set of SUB 5A A5 response frames into per-channel time-series:
|
||
- Reads the STRT record from A5[0] (bytes 7+): extracts `total_samples` (BE uint16 at +8),
|
||
`pretrig_samples` (BE uint16 at +16), and `rectime_seconds` (uint8 at +18) into
|
||
`event.total_samples / pretrig_samples / rectime_seconds`.
|
||
- Skips the 6-byte preamble (`00 00 ff ff ff ff`) that follows the 21-byte STRT header;
|
||
waveform data begins at `strt_pos + 27`.
|
||
- Strips the 8-byte per-frame counter header from A5[1–6, 8] before appending waveform bytes.
|
||
- Skips A5[7] (metadata-only) and A5[9] (terminator).
|
||
- **Cross-frame alignment correction**: accumulates `running_offset % 8` across all frames
|
||
and discards `(8 − align) % 8` leading bytes per frame to re-align to a T/V/L/M boundary.
|
||
Required because individual frame waveform payloads are not always multiples of 8 bytes.
|
||
- Decodes as 4-channel interleaved signed 16-bit LE at 8 bytes per sample-set:
|
||
bytes 0–1 = Tran, 2–3 = Vert, 4–5 = Long, 6–7 = Mic.
|
||
- Stores result in `event.raw_samples = {"Tran": [...], "Vert": [...], "Long": [...], "Mic": [...]}`.
|
||
- **`download_waveform(event)` public method** on `MiniMateClient`.
|
||
Issues a full SUB 5A stream with `stop_after_metadata=False`, then calls
|
||
`_decode_a5_waveform()` to populate `event.raw_samples` and `event.total_samples /
|
||
pretrig_samples / rectime_seconds`. Previously only metadata frames were fetched during
|
||
`get_events()`; raw waveform data is now available on demand.
|
||
- **`Event` model new fields** (`models.py`): `total_samples`, `pretrig_samples`,
|
||
`rectime_seconds` (from STRT record), and `_waveform_key` (4-byte key stored during
|
||
`get_events()` for later use by `download_waveform()`).
|
||
|
||
### Protocol / Documentation
|
||
- **SUB 5A A5[0] STRT record layout confirmed** (✅ 2026-04-03, 4-2-26 blast capture):
|
||
- STRT header is 21 bytes: `b"STRT"` + length fields + `total_samples` (BE uint16 at +8) +
|
||
`pretrig_samples` (BE uint16 at +16) + `rectime_seconds` (uint8 at +18).
|
||
- Followed by 6-byte preamble: `00 00 ff ff ff ff`. Waveform begins at `strt_pos + 27`.
|
||
- Confirmed: 4-2-26 blast → `total_samples=9306`, `pretrig_samples=298`, `rectime_seconds=70`.
|
||
- **Blast/waveform mode A5 format confirmed** (✅ 2026-04-03, 4-2-26 blast capture):
|
||
4-channel interleaved int16 LE at 8 bytes per sample-set; cross-frame alignment correction
|
||
required. 948 of 9306 total sample-sets captured via `stop_after_metadata=True` (10 frames).
|
||
- **Noise/histogram mode A5 format — endianness corrected** (✅ 2026-04-03, 3-31-26 capture):
|
||
32-byte block samples are signed 16-bit **little-endian** (previously documented as BE).
|
||
`0a 00` → LE int16 = 10 (correct noise floor); BE would give 2560 (wrong).
|
||
- Protocol reference §7.6 rewritten — split into §7.6.1 (Blast/Waveform mode) and §7.6.2
|
||
(Noise/Histogram mode), each with confirmed field layouts and open questions noted.
|
||
|
||
---
|
||
|
||
## v0.6.0 — 2026-04-02
|
||
|
||
### Added
|
||
- **True event-time metadata via SUB 5A bulk waveform stream** — `get_events()` now issues a SUB 5A request after each SUB 0C download, reads the A5 response frames, and extracts the `Client:`, `User Name:`, and `Seis Loc:` fields as they existed at the moment the event was recorded. Previously these fields were backfilled from the current compliance config (SUB 1A), which reflects today's setup, not the setup active when the event triggered.
|
||
- `build_5a_frame(offset_word, raw_params)` in `framing.py` — reproduces Blastware's exact wire format for SUB 5A requests: raw (non-DLE-stuffed) `offset_hi`, DLE-stuffed params, and a DLE-aware checksum where `10 XX` pairs count only `XX`.
|
||
- `bulk_waveform_params()` returns 11 bytes (extra trailing `0x00` confirmed from 1-2-26 BW wire capture).
|
||
- `read_bulk_waveform_stream(key4, *, stop_after_metadata=True, max_chunks=32)` in `protocol.py` — loops sending chunk requests (counter increments `0x0400` per chunk), stops early when `b"Project:"` is found, then sends a termination frame.
|
||
- `_decode_a5_metadata_into(frames_data, event)` in `client.py` — needle-searches A5 frame data for `Project:`, `Client:`, `User Name:`, `Seis Loc:`, `Extended Notes` and overwrites `event.project_info`.
|
||
- **`get_events()` sequence extended** — now `1E → 0A → 0C → 5A → 1F` per event.
|
||
|
||
### Fixed
|
||
- **Compliance config (SUB 1A) channel block missing** — orphaned `self._send(build_bw_frame(SUB_COMPLIANCE, 0x2A, _DATA_PARAMS))` before the B/C/D receive loop had no corresponding `recv_one()`, shifting all subsequent receives one step behind and leaving frame D's channel-block data (trigger_level_geo, alarm_level_geo, max_range_geo) unread. Removed the orphaned send. Total config bytes received now correctly ~2126 (was ~1071).
|
||
- **Compliance config anchor search range** — `_decode_compliance_config_into()` searched `cfg[40:100]` for the sample-rate/record-time anchor. With the orphaned-send bug fixed the 44-byte padding it had been adding is gone, and the anchor now appears at `cfg[11]`. Search widened to `cfg[0:150]` to be robust to future layout shifts.
|
||
- Removed byte-content deduplication from `read_compliance_config()` — was masking the real receive-ordering bug.
|
||
|
||
### Protocol / Documentation
|
||
- **SUB 5A frame format confirmed** — `offset_hi` byte (`0x10`) must be sent raw (not DLE-stuffed); checksum is DLE-aware (only the second byte of a `10 XX` pair is summed). Standard `build_bw_frame` DLE-stuffs `0x10` incorrectly for 5A — a dedicated `build_5a_frame` is required.
|
||
- **Event-time metadata source confirmed** — `Client:`, `User Name:`, and `Seis Loc:` strings are present in A5 frame 7 of the bulk waveform stream (SUB 5A), not in the 210-byte SUB 0C waveform record. They reflect the compliance setup as it was when the event was stored on the device.
|
||
|
||
---
|
||
|
||
## v0.5.0 — 2026-03-31
|
||
|
||
### Added
|
||
- **Console tab in `seismo_lab.py`** — direct device connection without the bridge subprocess.
|
||
- Serial and TCP transport selectable via radio buttons.
|
||
- Four one-click commands: POLL, Serial #, Full Config, Event Index.
|
||
- Colour-coded scrolling output: TX (blue), RX raw hex (teal), parsed/decoded (green), errors (red).
|
||
- Save Log and Send to Analyzer buttons; logs auto-saved to `bridges/captures/console_<ts>.log`.
|
||
- Queue/`after(100)` pattern — no UI blocking or performance impact.
|
||
- **`minimateplus` package** — clean Python client library for the MiniMate Plus S3 protocol.
|
||
- `SerialTransport` and `TcpTransport` (for Sierra Wireless RV50/RV55 cellular modems).
|
||
- `MiniMateProtocol` — DLE frame parser/builder, two-step paged reads, checksum validation.
|
||
- `MiniMateClient` — high-level client: `connect()`, `get_serial()`, `get_config()`, `get_events()`.
|
||
- **TCP/cellular transport** (`TcpTransport`) — connect to field units via Sierra Wireless RV50/RV55 modems over cellular.
|
||
- `read_until_idle(idle_gap=1.5s)` to handle modem data-forwarding buffer delay.
|
||
- Confirmed working end-to-end: TCP → RV50/RV55 → RS-232 → MiniMate Plus.
|
||
- **`bridges/tcp_serial_bridge.py`** — local TCP-to-serial bridge for bench testing `TcpTransport` without a cellular modem.
|
||
- **SFM REST server** (`sfm/server.py`) — FastAPI server with device info, event list, and event record endpoints over both serial and TCP.
|
||
|
||
### Fixed
|
||
- `protocol.py` `startup()` was using a hardcoded `POLL_RECV_TIMEOUT = 10.0` constant, ignoring the configurable `self._recv_timeout`. Fixed to use `self._recv_timeout` throughout.
|
||
- `sfm/server.py` now retries once on `ProtocolError` for TCP connections to handle cold-boot timing on first connect.
|
||
|
||
### Protocol / Documentation
|
||
- **Sierra Wireless RV50/RV55 modem config** — confirmed required ACEmanager settings: Quiet Mode = Enable, Data Forwarding Timeout = 1, TCP Connect Response Delay = 0. Quiet Mode disabled causes modem to inject `RING\r\nCONNECT\r\n` onto the serial line, breaking the S3 handshake.
|
||
- **Calibration year** confirmed at SUB FE (Full Config) destuffed payload offset 0x56–0x57 (uint16 BE). `0x07E7` = 2023, `0x07E9` = 2025.
|
||
- **`"Operating System"` boot string** — 16-byte UART boot message captured on cold-start before unit enters DLE-framed mode. Parser handles correctly by scanning for DLE+STX.
|
||
- RV50/RV55 sends `RING`/`CONNECT` over TCP to the calling client even with Quiet Mode enabled — this is normal behaviour, parser discards it.
|
||
|
||
---
|
||
|
||
## v0.4.0 — 2026-03-12
|
||
|
||
### Added
|
||
- **`seismo_lab.py`** — combined Bridge + Analyzer GUI. Single window with two tabs; bridge start auto-wires live mode in the Analyzer.
|
||
- **`frame_db.py`** — SQLite frame database. Captures accumulate over time; Query DB tab searches across all sessions.
|
||
- **`bridges/s3-bridge/proxy.py`** — bridge proxy module.
|
||
- Large BW→S3 write frame checksum algorithm confirmed and implemented (`SUM8` of payload `[2:-1]` skipping `0x10` bytes, plus constant `0x10`, mod 256).
|
||
- SUB `A4` identified as composite container frame with embedded inner frames; `_extract_a4_inner_frames()` and `_diff_a4_payloads()` reduce diff noise from 2300 → 17 meaningful entries.
|
||
|
||
### Fixed
|
||
- BAD CHK false positives on BW POLL frames — BW frame terminator `03 41` was being included in the de-stuffed payload. Fixed to strip correctly.
|
||
- Aux Trigger read location confirmed at SUB FE offset `0x0109`.
|
||
|
||
---
|
||
|
||
## v0.3.0 — 2026-03-09
|
||
|
||
### Added
|
||
- Record time confirmed at SUB E5 page2 offset `+0x28` as float32 BE.
|
||
- Trigger Sample Width confirmed at BW→S3 write frame SUB `0x82`, destuffed payload offset `[22]`.
|
||
- Mode-gating documented: several settings only appear on the wire when the appropriate mode is active.
|
||
|
||
### Fixed
|
||
- `0x082A` mystery resolved — fixed-size E5 payload length (2090 bytes), not a record-time field.
|
||
|
||
---
|
||
|
||
## v0.2.0 — 2026-03-01
|
||
|
||
### Added
|
||
- Channel config float layout fully confirmed: trigger level, alarm level, and unit string per channel (IEEE 754 BE floats).
|
||
- Blastware `.set` file format decoded — little-endian binary struct mirroring the wire payload.
|
||
- Operator manual (716U0101 Rev 15) added as cross-reference source.
|
||
|
||
---
|
||
|
||
## v0.1.0 — 2026-02-26
|
||
|
||
### Added
|
||
- Initial `s3_bridge.py` serial bridge — transparent RS-232 tap between Blastware and MiniMate Plus.
|
||
- `s3_parser.py` — deterministic DLE state machine frame extractor.
|
||
- `s3_analyzer.py` — session parser, frame differ, Claude export.
|
||
- `gui_bridge.py` and `gui_analyzer.py` — Tkinter GUIs.
|
||
- DLE framing confirmed: `DLE+STX` / `DLE+ETX`, `0x41` = ACK (not STX), DLE stuffing rule.
|
||
- Response SUB rule confirmed: `response_SUB = 0xFF - request_SUB`.
|
||
- Year `0x07CB` = 1995 confirmed as MiniMate factory RTC default.
|
||
- Full write command family documented (SUBs `68`–`83`).
|