serversdown
3457ed0072
BW writes "OORANGE" (truncation of "Out Of Range") when a channel exceeds its full-scale, and uses a typo'd label "Peak Vector Sum TimeSum" for the PVS time field. Both confirmed against real ASCII files pulled from a Windows watcher PC 2026-05-27: T190LD5Q.LK0W Vert PPV = OORANGE (Normal range, 10 in/s exceeded) T438L713.RY0W All three PPVs OORANGE (Sensitive range, 1.25 in/s) K557L3YM.OE0W Tran+Vert PPV OORANGE + MicL PSPL OORANGE Previously our _parse_number() returned None for OORANGE → DB columns ended up NULL → events vanished from filters / sorts / dashboards despite being legitimate high-amplitude events. New behavior — substitute a conservative bound + set a saturation flag: - Channel PPV → geo_range_ips + ChannelStats.ppv_saturated - Peak Vector Sum → sqrt(3) * geo_range_ips + peak_vector_sum_saturated - MicL PSPL → 140 dB(L) + MicStats.pspl_saturated Flags propagate to the sidecar's bw_report block so the SFM UI can render "> 10 in/s" / "> 140 dBL" rather than treating the substituted value as exact. Same commit also accepts "Peak Vector Sum TimeSum" as an alias for "Peak Vector Sum Time" (BW always writes the typo on OORANGE PVS lines — every example file confirms it). Tests: new test_oorange_marker_treated_as_saturation (synthetic) + test_real_oorange_event_t190_parses (skips if real fixture absent). 177/177 tests pass; 16 pre-existing missing-fixture skips unchanged. Five events on prod (T190, T438, K557, plus 2 others matching the same fault pattern) will pick up correct peaks + saturation flags once watchers re-forward. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1068 lines
78 KiB
Markdown
1068 lines
78 KiB
Markdown
# Changelog
|
||
|
||
All notable changes to seismo-relay are documented here.
|
||
|
||
---
|
||
|
||
## [Unreleased]
|
||
|
||
### Fixed
|
||
|
||
- **bw_ascii_report parser now handles `OORANGE` saturation marker.** BW writes `"OORANGE"` (truncation of "Out Of Range") in PPV / PVS / MicL PSPL fields when the underlying measurement exceeded the channel's full-scale. Previously our `_parse_number()` returned None → DB ended up with NULL peaks for legitimate high-amplitude events. Confirmed on real ASCII files pulled 2026-05-27 from the Windows watcher PC: T190LD5Q.LK0W (Vert saturated at Normal range 10 in/s), T438L713.RY0W (all three channels saturated at Sensitive range 1.25 in/s), K557L3YM.OE0W (Tran+Vert saturated + Mic PSPL OORANGE). New behavior:
|
||
- Per-channel PPV: substitute `geo_range_ips` as a conservative lower bound + set `ppv_saturated` flag
|
||
- Peak Vector Sum: substitute `sqrt(3) * geo_range_ips` (the theoretical max when all 3 channels are simultaneously at full-scale) + `peak_vector_sum_saturated` flag
|
||
- MicL PSPL: substitute 140 dB(L) (conservative NL-43 max) + `pspl_saturated` flag
|
||
- Saturation flags are propagated into the sidecar's `bw_report` block for downstream UI rendering (`> 10 in/s` or similar)
|
||
- Five events on prod (T190 / T438 / K557 + 2 others matching the same fault pattern) will pick up correct DB peaks + saturation flags once re-forwarded
|
||
- **bw_ascii_report parser handles `Peak Vector Sum TimeSum` typo'd label.** Real BW output uses this misspelled label (Sum appended twice instead of "Peak Vector Sum Time"). Now accepted as an alias. Confirmed against all three OORANGE example files — every one has the typo.
|
||
|
||
### Added
|
||
|
||
- **Histogram per-interval aggregation in `waveform.json`.** Histogram events now render with one bar per BW-reported interval (matching the Blastware printout) instead of ~200 bars per event (the raw codec output). When the sidecar's `bw_report.histogram.n_intervals` is populated (events ingested with the new parser, see next bullet), the `/db/events/{id}/waveform.json` endpoint groups the codec samples into N intervals via max-per-group and returns the aggregated array. `time_axis` gains `histogram_aggregated: true`, `n_intervals`, `interval_size_s`, and `interval_times` (HH:MM:SS strings). Both the modal chart and the standalone event browser use those interval timestamps as x-axis labels when present. Defensive: no-op for events ingested before the parser extension landed (their sidecars lack `histogram.n_intervals`) — those continue to render with raw codec output.
|
||
- **`bw_ascii_report` parser now captures histogram-specific fields.** Previously the parser dropped these fields silently (Roadmap item closed):
|
||
- `Histogram Start Time` / `Histogram Start Date` (combined into `histogram_start: datetime`)
|
||
- `Histogram Stop Time` / `Histogram Stop Date` (combined into `histogram_stop: datetime`)
|
||
- `Number of Intervals` (`histogram_n_intervals: int`)
|
||
- `Interval Size` ("1 minute" string + parsed seconds: `histogram_interval_size_str`, `histogram_interval_size_s`)
|
||
- `<Channel> Peak Time` + `<Channel> Peak Date` for histogram events (combined into `channel_peak_when: dict`; waveforms continue to use `time_of_peak_s` relative)
|
||
- `Peak Vector Sum Date` (combined with PVS Time into `peak_vector_sum_when: datetime`; clears the previous bogus `peak_vector_sum_time_s` parse that interpreted "22:33:52" as 22.0 seconds)
|
||
- All new fields land in the sidecar's `bw_report.histogram` block via `_bw_report_to_dict`. Tested against synthetic K558LLB7.V20H-shaped input.
|
||
- **Raw BW ASCII report (.TXT) preservation.** `save_imported_bw` now writes the paired `_ASCII.TXT` to `<store>/<serial>/<filename>_ASCII.TXT` alongside the binary at ingest time. Previously the .TXT was parsed into the sidecar's `bw_report` projection and then discarded — meaning parser bug fixes couldn't be applied retroactively without re-forwarding from the watcher PC. Now the raw .TXT lives in the waveform store permanently (~15 KB per event; ~210 MB total for a 14k-event store; negligible). Sidecar's `source.txt_filename` field records the saved path; backfill_sidecars preserves it across regens. New `GET /db/events/{id}/ascii_report.txt` endpoint serves the raw .TXT for any event ingested after this change. Events ingested before today still return 404 from that endpoint until re-forwarded. Architectural rationale: with BW Mail / Forwarding Agent being phased out of the operator workflow, the XML/PDF/WMF that those tools produced are no longer available — the binary + .TXT (created by BW ACH itself) are our authoritative source for everything going forward.
|
||
|
||
- **Event Report PDF generation** — `GET /db/events/{id}/report.pdf` returns a single-page letter-portrait PDF for any event with waveform data on disk. Covers every field a Blastware Event Report includes: header metadata (date/time, trigger source, range, sample rate, project/client/operator/location, serial+firmware, battery, calibration, file name), microphone block (PSPL in dB(L) + psi, ZC freq, channel test), per-channel stats table (rows differ for waveform vs histogram), Peak Vector Sum, and the 4-channel plot. Iterated against real Blastware reference PDFs (uploaded to `example-events/pdfsnstuff/`):
|
||
- **Waveform layout**: header shows Date/Time, Trigger Source, Range, Sample Rate; stats table has PPV / ZC Freq / Time (Rel. to Trig) / Peak Accel / Peak Disp / Sensor Check; bottom plot is 4-channel line waveform (MicL top → Tran bottom), shared time axis in seconds, dashed trigger line + triangle marker at t=0, symmetric Y on geo channels, zero-anchored on mic, "0.0" baseline label on right per BW convention; footer shows `Time X sec/div Amplitude Geo: Y in/s/div Mic: 0.001 psi(L)/div` and the trigger window `▶━━◀` marker. USBM RI8507/OSMRE compliance chart placeholder upper-right.
|
||
- **Histogram layout**: header shows Start / Finish / Intervals At Size / Range / Sample Rate (no Trigger Source — histograms aren't triggered); NO USBM chart; stats table has PPV / ZC Freq / Date / Time / Sensor Check; bottom plot is per-interval bar chart, Y-axis 0-to-peak (never negative), 0.0 baseline at the bottom; footer shows `Time INTERVAL_SIZE /div Amplitude Geo: Y in/s/div Mic: 0.001 psi(L)/div`.
|
||
- Backed by matplotlib (vector PDF, no headless-browser dep). Adds matplotlib>=3.8 to deps.
|
||
- **Known gap**: histogram codec returns per-block granularity (~200 bars for a 4-interval event) instead of BW's per-interval aggregation. Visual difference vs BW's 4-bar display. XML-driven data source (parsing the structured `_XML.XML` files BW also exports) is the planned fix; that route also resolves the bw_ascii_report PPV-miss bug.
|
||
- **Stubbed**: USBM RI8507 / OSMRE compliance chart curves (separate work item; requires coding the regulatory piecewise functions).
|
||
- **"Download PDF" button** in the event modal's footer — triggers the new endpoint; opens in a new tab so the browser handles save-or-display + surfaces any 404 / server errors visibly.
|
||
|
||
- **SFM webapp now opens to Database view by default** and the History table is fully interactive. Click any column header to sort ascending / descending (timestamp, serial, per-channel PPV, PVS, mic dB(L), project, client, record type, key — all sortable). Click any event row to open the event modal, which now renders a **4-channel waveform plot inline** (MicL / Long / Vert / Tran stacked, Instantel-printout order) alongside the existing sidecar review fields. Headers are sticky so the columns stay visible while scrolling long event lists. No more "where is the viewer" — pick a unit from the filter dropdown, scan the table, click the event, see the waveform.
|
||
- **Stored-event browser** — new standalone HTML page at `GET /events` (`sfm/event_browser.html`). Pick a serial from the unit dropdown, scroll through that unit's events (newest-first), click any event to render its decoded waveform via the existing `/db/events/{id}/waveform.json` endpoint. Dark-themed Chart.js viewer, channels stacked vertically (MicL / Long / Vert / Tran — Instantel printout order, designed PDF-export-ready), trigger line at t=0, peak labels, search/filter, false-trigger flag honored. Companion to the existing live-device viewer at `/waveform`; the two routes are now clearly delineated in their docstrings. The webapp's inline plot at `/` is the primary path; `/events` remains a useful diagnostic when you want just a viewer.
|
||
- **Histogram body codec — uint8 peak count fix.** Per-channel peak fields at `block[6]/[10]/[14]/[18]` are `uint8`, not `uint16 LE` spanning `block[6:8]` etc. The original interpretation was byte-exact on the N844 fixture corpus only because every annotation byte (`block[7]/[11]/[15]/[19]`) in those fixtures was zero. On non-N844 events with non-zero annotation bytes (observed across BE9558 Tran-drift and BE18003 Histogram+Continuous units), the old interpretation produced peaks up to 268 in/s per channel and 35× inflated PVS sums when first deployed to prod (rolled back same day; properly fixed in this release). Cross-correlated against BW's per-interval ASCII export on K558 / T003 / N599 / N844 corpora — 100% byte-exact on T/V/L, 99%+ on M (sub-precision rounding). Annotation byte preserved on each record as `record["annotations"]` for future RE. Verified against ~3,500 blocks across 5 in-repo fixtures + a synthetic K558 interval-12 regression block.
|
||
- **`apply_bw_report_dict_to_event` helper** in `minimateplus.event_file_io`. Mirror of `apply_report_to_event` for the projected sidecar dict shape — used by the backfill path, which has the preserved `bw_report` block but not the original `.TXT` file. BW's reported peaks (and `sample_rate` / `record_time`) now win over codec output during `--force` backfill, matching ingest-path behavior.
|
||
- **`scripts/check_bw_report_preservation.py`** — two-step snapshot/diff tool to verify that `backfill_sidecars.py` doesn't wipe the `bw_report` block from existing sidecars. Classifies every sidecar as PRESERVED / CHANGED / WIPED / STILL_MISSING / NEW / ADDED / REMOVED. Exit code 1 if any WIPED or CHANGED entries are found, so it can gate a CI step or deploy script.
|
||
|
||
### Fixed
|
||
|
||
- **`scripts/backfill_sidecars.py` no longer wipes `bw_report`.** Before this fix, `event_to_sidecar_dict` silently dropped the preserved `bw_report` block during every backfill, since the function only emits a `bw_report` when called with a live `BwAsciiReport` dataclass (which the backfill doesn't have — only the projected sidecar dict). Now we read the existing sidecar's `bw_report` and overlay it onto the regenerated sidecar, alongside the existing `review` and `extensions` preservation.
|
||
- **`scripts/backfill_sidecars.py --force` no longer overwrites BW-overlaid DB peaks with codec output.** The backfill path now calls `apply_bw_report_dict_to_event` before the DB upsert, mirroring what the ingest path does (`/db/import/blastware_file` parses the `.TXT` into a `BwAsciiReport`, calls `apply_report_to_event`, then upserts). Without this, events where the codec doesn't fully decode (waveform walker edge cases on SP0/SS0/SV0-style events, histogram `byte[5]!=0` sub-format) ended up with PVS=0 in the DB after a `--force` backfill; bit on prod 2026-05-22, rolled back the same day.
|
||
- **Thor IDF files no longer attempted as BW events in backfill.** `scripts/backfill_sidecars.py` now filters out `.IDFW` / `.IDFH` files in `_looks_like_event_file()`; they share the `.X0W` / `.X0H` suffix shape but use a separate ingest path (`WaveformStore.save_imported_idf`) and aren't decodable by `event_file_io.read_blastware_file`.
|
||
|
||
### Docs
|
||
|
||
- **CLAUDE.md** — added a three-tier conceptual architecture model (SFM / SDM / shared codec library) near the top of the file, with a placement rule for where new code goes. Documents that what is conceptually SDM (database, waveform store, ingest, `/db/*` endpoints) still lives under `sfm/` for historical reasons; rename deferred until the codebase is quiet enough for a clean refactor.
|
||
- **README.md** — added a "Strategic direction" lead-in to the Roadmap that frames seismo-relay as a suite of cooperating components (not a single app), and an explicit "Terra-View ↔ SFM device control" roadmap section with a concrete implementation checklist (auth as hard prerequisite, embedded live-monitor view, action history, Series IV live-device support).
|
||
- **`docs/histogram_codec_re_status.md`** updated with the uint8 retraction and the annotation-byte status.
|
||
- Three known issues recorded in the Roadmap that were discovered during prod validation: (1) `bw_ascii_report` parser misses PPV / `vector_sum` on some `.TXT` formats (5 events on prod); (2) NULL-timestamp duplicate-row dedup needed (2 events on prod); (3) histogram body sub-format with `byte[5] != 0` not yet decoded (~3 events on prod with empty `.h5` plots).
|
||
|
||
---
|
||
|
||
## v0.19.0 — 2026-05-20
|
||
|
||
The "device-family separation" release. Tightens the boundary between Series III (MiniMate Plus / Blastware) and Series IV (Micromate / Thor) so the UI and storage layer dispatch deterministically by family instead of sniffing filename extensions or magnitude heuristics.
|
||
|
||
### Added — Phase 1: `device_family` column on `events`
|
||
|
||
- **`events.device_family TEXT`** — new column carrying `"series3"` or `"series4"`. Populated by every import path (`/db/import/blastware_file`, `/db/import/idf_file`, ACH server, BW CLI, sidecar backfill script). Returned through `/db/events` since `query_events` uses `SELECT *`.
|
||
- **Self-applying migration** — on startup, `ALTER TABLE ... ADD COLUMN` lands the new column; a follow-on `UPDATE` backfills existing rows from the binary filename extension (`.IDFH`/`.IDFW` → `series4`, everything else → `series3`). No manual SQL needed.
|
||
- **UPSERT preserves family** — re-imports without an explicit family don't blank existing rows (`COALESCE(?, device_family)`).
|
||
- **UI dispatches on the column** — `sfm_webapp.html` events-table mic formatter now branches on `ev.device_family === 'series4'` (Thor stores native dB(L); BW stores psi). Modal uses `source.kind === 'idf-import'` from the sidecar (sidecars don't carry the DB column). Source-files section labels changed from "BW filename / BW filesize / BW sha256" to format-neutral "Event file / File size / File sha256".
|
||
|
||
### Added — Phase 2: `micromate/` package alongside `minimateplus/`
|
||
|
||
- **`micromate/`** — new sibling package for the Thor / Micromate Series IV device. Currently scoped to offline-file ingest; live-device support (TCP transport, framing, protocol, client) will land here when reverse-engineering happens.
|
||
- `micromate/idf_ascii_report.py` — moved from `sfm/idf_ascii_report.py`. No behaviour change.
|
||
- `micromate/models.py` — typed `IdfReport`, `IdfEvent`, `IdfPeaks`, `IdfProjectInfo`, `IdfSensorCheck`. Stores mic in native `mic_pspl_dbl` (dB(L)) instead of the pseudo-psi shoehorn that the BW-shaped model uses. `IdfEvent.from_report()` constructs from a parsed dict + filename; `IdfEvent.to_minimateplus_event(waveform_key)` bridges to the existing sidecar / DB-insert machinery.
|
||
- `micromate/idf_file.py` — placeholder for the binary codec (`.IDFH` / `.IDFW`). Stubbed `read_idf_file()` raises `NotImplementedError`; documents the planned reverse-engineering path.
|
||
- **`WaveformStore.save_imported_idf`** refactored to use the native `IdfEvent` and bridge at the SQL-insert boundary. Cleaner separation of "parse a Thor event" (in `micromate/`) from "store it on disk + write a sidecar" (in `sfm/waveform_store.py`).
|
||
- **Tests** — `tests/test_idf_ascii_report.py` imports updated to `micromate.idf_ascii_report`. All 1,014 example-data sidecars round-trip through `IdfEvent.from_report()` without errors.
|
||
|
||
### Companion releases
|
||
|
||
- **thor-watcher** unaffected — it talks to the relay over HTTP only. No version bump needed.
|
||
- **terra-view** unaffected today; can use `device_family` in its event-detail rendering when convenient.
|
||
|
||
---
|
||
|
||
## v0.18.0 — 2026-05-19
|
||
|
||
The "Thor / Series IV ingest adapter" release. Seismo-relay can now accept event files from Instantel Micromate Series IV (Thor) units alongside the existing MiniMate Plus (Series III) Blastware pipeline.
|
||
|
||
### Added — Thor (Series IV) IDF ingest
|
||
|
||
- **`POST /db/import/idf_file`** (`sfm/server.py`) — multipart upload endpoint for `.IDFH` (histogram) and `.IDFW` (waveform) event files plus their `.IDFH.txt` / `.IDFW.txt` ASCII sidecars. Mirrors the shape of `/db/import/blastware_file`: pairing by filename, optional `serial` query hint, per-file outcome reporting.
|
||
- **`sfm/idf_ascii_report.py`** — parser for Thor's TXT sidecars (verified against 1,014 real-world samples). Extracts device-authoritative PPV, ZC Freq, Peak Vector Sum, Mic PSPL, calibration date, firmware version, sensor self-check results, and project/client/operator strings.
|
||
- **`WaveformStore.save_imported_idf()`** (`sfm/waveform_store.py`) — stores Thor binaries verbatim in `<root>/<serial>/<filename>`, writes a `.sfm.json` sidecar with `source.kind = "idf-import"` and the full parsed report under `extensions.idf_report`. Reuses the existing `events` table — Thor events dedupe on (serial, timestamp) and surface in `/db/events` alongside BW events.
|
||
- **`tests/test_idf_ascii_report.py`** — parser tests against the `thor-watcher/example-data/` corpus.
|
||
|
||
### Changed
|
||
|
||
- `event_to_sidecar_dict()` (`minimateplus/event_file_io.py`) allow-list for `source_kind` now includes `"idf-import"` so the existing sidecar machinery can carry Thor imports.
|
||
- Bumped `pyproject.toml` version to `0.18.0`.
|
||
|
||
### Companion release
|
||
|
||
This release ships alongside **thor-watcher v0.3.0**, which adds the SFM forwarder that targets the new `/db/import/idf_file` endpoint. Operators flip the switch in thor-watcher's new "SFM Forward" Settings tab; events POST to seismo-relay just like the series3-watcher BW forwarder does today.
|
||
|
||
---
|
||
|
||
## v0.17.0 — 2026-05-17
|
||
|
||
The "field rescue + DB management" release. Hardened against units that are stuck in a runaway call-home loop, and added an operator-facing path for purging bogus events that those same units dump into the DB before recovery. All work in this release was driven by the BE9558H incident (full incident log + recovery procedure at `docs/runbooks/wedged_unit_recovery.md`).
|
||
|
||
### Added — wedged-unit recovery toolkit
|
||
|
||
A toolkit for breaking the call-home loop on a misbehaving unit whose firmware is too busy to keep up with normal request/response handshakes. Tested in production against BE9558H (16 May 2026) — a unit with a stuck-triggered Long-axis geophone that had been call-homing the office BW ACH server every 30 seconds for hours. Endpoints layered from "single attempt" to "siege mode" to suit different contention levels:
|
||
|
||
- **`GET /device/events/storage_range`** — SUB 0x06 probe. POLL + one read; ~2s. Returns first/last event keys and an `is_empty` flag. Use to triage whether a unit has stored events without invoking the slow `count_events()` 1E/1F chain (which choked on BE9558H's corrupted event chain).
|
||
- **`GET /device/events/index`** — SUB 0x08 probe. POLL + one read; ~2s. Returns the lifetime event counter (does NOT decrement on erase — use `storage_range` for "right now" state).
|
||
- **`POST /device/events/erase`** — full erase sequence `0xA3 → 0x1C → 0x06 → 0xA2` (confirmed 2026-04-11, see the protocol reference). Resets event keys to `0x01110000`. Caller's responsibility to disable ACH first if the underlying trigger condition will re-fill the buffer.
|
||
- **`POST /device/rescue`** — one TCP session, short connect+recv timeouts: POLL → disable ACH (compliance config write) → erase events → close. Designed for race-loop usage when the device is busy in another session. 503 on connect-refused, 502 on protocol failure, 200 on full sequence success.
|
||
- **`POST /device/stop_monitoring_blind`** — fire-and-forget Stop Monitoring (SUB 0x97), TCP-only. Dumps `SESSION_RESET + POLL_PROBE + SESSION_RESET + POLL_DATA + 0x97 × repeat` and closes without reading any S3 response. The full POLL preamble is required — write commands without it are silently ignored by the device's protocol parser (false-positive surface area that bit the first version of this endpoint). Use when the device's firmware can't keep up with full request/response but might process inbound bytes at its own pace.
|
||
- **`POST /device/stop_monitoring_spam`** — server-side hammer loop, duration-bounded. Open TCP → write the same blind payload → close → repeat as fast as possible until `duration_s` elapses. Configurable `connect_timeout` (default 500ms) and `repeat` (frames per session). Reports `sent_ok`, `connect_failed`, `write_failed`, `rate_attempts_per_s`. Clamped to 5min duration.
|
||
- **`POST /device/stop_monitoring_slow_drip`** — opposite of spam. Open ONE TCP session, drip the wake handshake + stop frames at `interval_s` (default 3s) for `duration_s` (default 120s, max 10min). Each drip is ~23 bytes — well under any UART FIFO size. Opportunistically drains any inbound bytes the device sends back; `bytes_received > 0` in the response strongly suggests the device has started talking and the session is healthy. **This is the endpoint that saved BE9558H.** Spam mode had been overrunning the device's UART FIFO; slow drip stayed under it.
|
||
- **Six rescue scripts** under `scripts/` — thin bash wrappers around the endpoints, default `SFM_BASE_URL=http://localhost:8200` (direct, not via Terra-View proxy whose 60s timeout would cut off the longer endpoints):
|
||
- `rescue_device.sh` — race-loop wrapper for `/device/rescue`
|
||
- `blind_stop.sh` — race-loop wrapper for `/device/stop_monitoring_blind`
|
||
- `spam_stop.sh` — single-call burst hammer
|
||
- `slow_drip.sh` — single-call held-session drip
|
||
- `watch_unit.sh` — passive periodic reachability check (every N min, logs to file), useful for unattended overnight monitoring of a wedged unit
|
||
- **`docs/runbooks/wedged_unit_recovery.md`** — symptoms, quick-reference recovery procedure, the modem-layer mechanism (Sierra Wireless serial-port mode-flipping is the real failure mode — not the device firmware), and a table of "why simpler approaches don't work" so the next incident skips the dead ends.
|
||
|
||
### Added — operator event DB management
|
||
|
||
Endpoints powering Terra-View's new `/admin/events` page (v0.12.0). Designed for purging bogus events from a unit that's been forwarding them in bulk (e.g. a stuck-triggered seismograph dumping hundreds of junk events before it's recovered).
|
||
|
||
- **`DELETE /db/events/{event_id}`** — hard-delete one event row. Also unlinks the associated blastware binary (`.AB0*`), `.a5.pkl`, `.sfm.json` sidecar, and `.h5` clean-waveform files via the WaveformStore. Returns the per-file removal status. 404 if the event doesn't exist.
|
||
- **`POST /db/events/delete_bulk`** — filter-based or id-list-based bulk delete with safety rails:
|
||
- Filters (`serial`, `from_dt`, `to_dt`, `false_trigger`) combine with AND; same semantics as `GET /db/events`. `ids` is an additional inclusion list. Refuses to run with no filters (would wipe the whole table — raises 422).
|
||
- `confirm` must be `true` to actually delete. Otherwise returns a dry-run summary (`status: "dry_run"`, `matched: N`, `sample_serials: [...]`).
|
||
- `max_rows` (default 10,000) caps how many rows can be deleted by-filter in one call. If exceeded, returns `status: "too_many"` with a hint to narrow or raise the cap. Bypassed when only `ids` is supplied.
|
||
- **`_cleanup_event_files(row)`** helper in `sfm/server.py` — best-effort `unlink()` of all four sidecar paths derived from the row's `blastware_filename`. Logged at WARN if a path exists but unlink fails; the DB row deletion still proceeds.
|
||
- **`SeismoDb.delete_event(id)` and `SeismoDb.delete_events_bulk(...)`** in `sfm/database.py` — both return the deleted row dict(s) so callers can do file cleanup. `delete_events_bulk` raises `ValueError` if no filters are supplied.
|
||
|
||
### Changed
|
||
|
||
- **Default protocol recv timeout dropped from 30s → 10s** in `_build_client()`. The unit usually responds in well under a second over cellular; 10s leaves comfortable headroom for retransmits while failing reasonably fast when a unit is wedged. The two endpoints that perform full 5A waveform downloads still pass `timeout=120.0` explicitly so multi-minute event transfers are unaffected.
|
||
- **`_build_client()` now accepts an optional `connect_timeout`** (TCP-only) so rescue / race-loop endpoints can fail fast on busy modems without affecting the protocol-level recv timeout.
|
||
|
||
### Fixed
|
||
|
||
- **`GET /device/monitor/status` returned HTTP 500 + uncaught traceback when the device was unresponsive**. The retry-on-`Exception` inner block let the second `client.poll()`'s `ProtocolError` propagate out of the handler. Now wrapped in proper try/except — returns 502 with `{"detail": "Protocol error: No S3 frame received within 10.0s ..."}` on timeout, 502 on connection errors, 500 only for genuinely unexpected exceptions.
|
||
|
||
### Migration
|
||
|
||
No schema changes. No data migration required.
|
||
|
||
If you've been running a previous version against a wedged unit and accumulated bogus events, the new `/admin/events` page in Terra-View v0.12.0 (or direct `POST /db/events/delete_bulk` with `confirm: true`) is the cleanup tool. Watcher state on the upstream DL2 PC does NOT need separate cleaning — the watcher's `sfm_forwarded.json` keys on file sha256 and won't re-forward the same files.
|
||
|
||
### Pairing
|
||
|
||
This release pairs with **Terra-View v0.12.0**, which adds the `/admin/events` UI that consumes the new bulk-delete endpoints, the bulk false-trigger flagging on `/unit/{id}`, and the field-deployment workflow that uses the same `series3-watcher` → SFM ingest path as before.
|
||
|
||
---
|
||
|
||
## v0.16.1 — 2026-05-14
|
||
|
||
### Fixed
|
||
|
||
- **`record_type` always "Waveform" for forwarded events.** `read_blastware_file()` hardcoded `ev.record_type = "Waveform"` regardless of the file's actual type. The watcher-forward pipeline (the main BW ACH ingest path) compounds this by parsing files from a tmp path with a `.bw` suffix, so even a filename-based fallback inside the parser still wouldn't see the original extension. Now:
|
||
|
||
1. New `derive_record_type_from_filename(filename)` helper in `minimateplus/event_file_io.py` derives the type from the LAST character of the filename's extension (V10.72+ AB0T scheme: `H`=Histogram, `W`=Waveform, `M`=Manual, `E`=Event, `C`=Combo). Falls back to `"Waveform"` for old S338 firmware (3-char extensions ending in `0`) and any unrecognized suffix.
|
||
2. `read_blastware_file()` now calls the helper with its `path.name` so direct callers (the `--dry-run` path in `scripts/import_bw.py`, tests, ad-hoc scripts) get the right value automatically.
|
||
3. `WaveformStore.save_imported_bw()` overrides `ev.record_type` with the **original** filename's derived type after parsing (the tmp file inside the parser doesn't carry the original extension). This is the path the live watcher-forwarder hits, so the DB column now reflects the actual event type going forward.
|
||
|
||
Events ingested before this fix are stuck with `record_type="Waveform"` in the DB; a one-off backfill (`UPDATE events SET record_type = ... WHERE blastware_filename LIKE '%H'`) would fix them retroactively if desired. Terra-view's event modal also derives client-side from the filename, so the UI already shows the correct type for old events even without the backfill.
|
||
|
||
---
|
||
|
||
## v0.16.0 — 2026-05-11
|
||
|
||
The "BW ACH ingestion" release. When paired with **series3-watcher v1.5.0**, every Blastware ACH event (binary + `_ASCII.TXT` report) lands in SeismoDb with device-authoritative peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data — without depending on the still-undecoded waveform body codec. This is the end-to-end product win discussed in v0.15.0's "out of scope" notes: sortable / filterable monthly-summary review of historical events, populated from the BW ASCII export rather than re-decoded samples.
|
||
|
||
### Added — `/db/import/blastware_file` rich-metadata ingestion
|
||
|
||
- **Paired BW ASCII reports.** The endpoint now accepts the `<binary>_<ext>_ASCII.TXT` partner BW writes alongside each event. Pairing handles both filename conventions: ACH (`M529LK44_AB0_ASCII.TXT`) and manual-export (`M529LK44.AB0.TXT`). When both present, ACH wins.
|
||
- **`minimateplus/bw_ascii_report.py`** (new) — parser + `BwAsciiReport` dataclass for BW's per-event ASCII export. Handles every field BW writes: identity, trigger config, per-channel PPV / ZC Freq / Time of Peak / Peak Acceleration / Peak Displacement, Peak Vector Sum + time, MicL PSPL / Time of Peak / ZC Freq, sensor self-check (Test Freq / Test Ratio / Test Amplitude / Pass-Fail per channel), monitor log, PC SW version.
|
||
- **Position-based user-notes parsing.** BW's Compliance Setup → Notes tab labels (Project / Client / User Name / Seis Loc) are *operator-editable* — an operator can rename them to "Building:", "Site Address:", etc. Rather than maintain a label-spelling map, the parser uses positional matching between the `Units :` and `Geo Range :` anchors in the ASCII output. The four canonical slots (project / client / operator / sensor_location) populate by position regardless of label; the original labels BW wrote are preserved in `report.user_note_labels` for downstream UIs (terra-view) to display verbatim.
|
||
- **`bw_report` sidecar block.** New top-level block in `.sfm.json` carrying the parsed BW report (trigger config, peaks with per-channel stats, mic block, sensor_check, monitor_log, PC SW version, operator-label labels).
|
||
- **`apply_report_to_event(event, report)` helper.** Overlays the report's device-authoritative fields onto an in-memory `Event` so `SeismoDb.insert_events()` writes correct DB columns instead of the broken-codec values from `_peaks_from_samples()`.
|
||
|
||
### Fixed — three compounding bugs that left forwarded events with garbage data
|
||
|
||
- **Import endpoint inserted under `serial="UNKNOWN"`.** `_serial_from_event(ev)` was a stub that always returned `None`; the BW-filename-decoded serial that `WaveformStore` had already resolved was never surfaced to `db.insert_events`. Now uses `rec["serial"]` as the authoritative source. `scripts/repair_unknown_serials.py` repairs existing DB rows.
|
||
- **`/db/units` ignored events from non-ACH ingest paths.** `query_units()` only aggregated from `ach_sessions` — events that arrived via `save_imported_bw()` were never visible in the fleet overview even though they populated `events` correctly. Now unions both tables.
|
||
- **Re-imports left stale DB rows.** The `IntegrityError` handler in `insert_events()` only refreshed filename / sidecar columns when a duplicate `(serial, timestamp)` arrived. Peak values, project info, sample_rate, record_type stayed locked at whatever the first (often broken-codec) insert wrote. Now the upsert path refreshes every device-authoritative column from the new data while preserving `false_trigger` and immutable fields (`id`, `created_at`).
|
||
- **Server-side TXT pairing only knew the legacy convention.** The endpoint stripped `.TXT` and looked up `<binary>` — which works for manual exports (`<binary>.TXT`) but not BW ACH (`<stem>_<ext>_ASCII.TXT`). Reports were arriving in the multipart but silently dropped. Now recognises both conventions and registers each report under all matching binary names.
|
||
|
||
### Migration
|
||
|
||
For existing deployments where events were forwarded by an older watcher (broken pairing) or imported during the UNKNOWN-bucketing window:
|
||
|
||
1. `python -m scripts.repair_unknown_serials --db <path> --apply` to re-attribute `serial="UNKNOWN"` rows.
|
||
2. Delete the watcher's `sfm_forwarded.json` state file and let it re-forward. The server's upsert path will refresh the existing DB rows with the report's authoritative values.
|
||
3. Operator review state (`false_trigger`, sidecar `review` block) is preserved across the re-import.
|
||
|
||
## v0.15.0 — 2026-05-07
|
||
|
||
### Added
|
||
|
||
- **Layered event storage architecture.** Each event now lands as four
|
||
files in the per-serial waveform store, each with a clear role:
|
||
|
||
- `<filename>` — the Blastware-readable binary (BW file). Untouched.
|
||
- `<filename>.a5.pkl` — the raw 5A frames (regenerative source).
|
||
- `<filename>.h5` — clean per-channel waveform arrays in physical
|
||
units (in/s for geo, psi for mic) plus event metadata (HDF5 with
|
||
gzip compression). This is the canonical format for downstream
|
||
analysis tools.
|
||
- `<filename>.sfm.json` — the modern review/metadata sidecar (peaks,
|
||
project, source provenance, review state, extensions).
|
||
|
||
SQLite (`seismo_relay.db`) is the searchable index over all four.
|
||
|
||
- **Plot-ready waveform JSON (`sfm.plot.v1`).** The `/device/event/{idx}/waveform`
|
||
and `/db/events/{id}/waveform.json` endpoints now return samples in
|
||
physical units with explicit time-axis metadata, peak markers, and
|
||
per-channel unit hints — no more guessing the ADC-to-velocity scale
|
||
client-side. The webapp waveform viewer was rewritten to consume
|
||
this shape.
|
||
|
||
- **In-app waveform viewer accuracy fix.** The standalone SFM webapp
|
||
viewer was scaling geophone amplitudes by `geoAdcScale / 32767`
|
||
(≈ 6.206 / 32767), where `geoAdcScale = 6.206053` is the device's
|
||
*in/s per V* hardware constant — not the ADC-counts-to-velocity
|
||
factor. This silently scaled every plot ~38% too low for Normal-range
|
||
geophones (the correct full-scale is 10.0 in/s, or 1.25 in/s for
|
||
Sensitive). Conversion is now done server-side using the geo_range
|
||
from compliance config; the client just plots.
|
||
|
||
- New `sfm/event_hdf5.py` module: `write_event_hdf5()`,
|
||
`read_event_hdf5()`, plus a plot-JSON helper.
|
||
- Backfill script extended to also emit `.h5` for existing events.
|
||
|
||
### Dependencies
|
||
|
||
- Added `h5py>=3.10` and `numpy>=1.24` for the HDF5 storage layer.
|
||
- Added `python-multipart>=0.0.7` (required by FastAPI for the
|
||
`/db/import/blastware_file` endpoint introduced in this release).
|
||
|
||
---
|
||
|
||
## v0.14.3 — 2026-05-05
|
||
|
||
### Fixed
|
||
|
||
- **`build_5a_frame` — DLE-stuffing rule for 0x10 bytes in params (the
|
||
long-standing >1-sec event 0 "won't open in BW" bug).**
|
||
|
||
Previously `build_5a_frame` wrote params bytes RAW with no DLE stuffing,
|
||
based on the incorrect assumption that the device handled all `0x10`
|
||
bytes in params literally. It does not. The device's actual de-stuffing
|
||
rule for the params region is:
|
||
|
||
- `10 10` → de-stuffs to `10`
|
||
- `10 02/03/04` → kept literal (inner-frame markers)
|
||
- `10 X` for other X → de-stuffs to just `X` (drops the `0x10`)
|
||
|
||
When the counter passed in params has `0x10` in the high byte (e.g.
|
||
counter=`0x1000` produces params bytes `... 10 00 ...`), the device
|
||
silently corrupts the request to counter=`0x__00` and responds with
|
||
whatever lives at that wrong address. For counter=0x1000 the wrong
|
||
address was 0x0000, so the response was a copy of the file header +
|
||
STRT record. That STRT block then got embedded in the assembled body
|
||
at file offset `0x1016`, and Blastware refused to open the file
|
||
(interprets the second STRT as a malformed multi-event file).
|
||
|
||
This explains the entire >1-sec event-0 failure pattern:
|
||
|
||
- 1-sec events have `end_offset < 0x1000`, so the chunk walk never
|
||
requests counter `0x10__` and the bug never triggers.
|
||
- 2-sec / 3-sec / longer events all need a chunk at counter `0x1000`
|
||
(and longer events also need `0x1200`, `0x1400`, etc., none of which
|
||
have `0x10` in the high byte except `0x1000`). Just one corrupted
|
||
response is enough to embed STRT in the body and break the file.
|
||
|
||
Verified against BW 5-1-26 "copy 3sec" capture: all 17 5A request
|
||
frames (probe + 2 metadata pages + 13 sample chunks + TERM) now match
|
||
BW's wire output **byte-for-byte**, including the doubled `10 10 00`
|
||
for counter=0x1000.
|
||
|
||
### Notes
|
||
|
||
- `0x10` bytes in `offset_hi` (the standalone offset field at body[5])
|
||
are still written RAW — confirmed correct per the 1-2-26 capture.
|
||
- BW's actual encoding of `10 02` / `10 04` for meta pages 0x1002 /
|
||
0x1004 is *not* doubled — it relies on the device keeping `10 02`
|
||
and `10 04` as literal pairs. This is preserved by the fix.
|
||
|
||
---
|
||
|
||
## v0.14.2 — 2026-05-04
|
||
|
||
### Fixed
|
||
|
||
- **`blastware_file.py` — removed harmful "duplicate header+STRT" strip.**
|
||
The v0.13.x strip logic was matching the byte sequence `00 12 03 00 STRT`
|
||
in legitimate waveform data — sample chunks at counter `0x1000` and
|
||
beyond often contain those bytes coincidentally — and zeroing 25 bytes
|
||
of valid samples per match. This is why event 0 (event-1 case in the
|
||
protocol) downloads of >1-sec recordings always failed in BW: the strip
|
||
destroyed real data at body offset `0x1012..0x102B` and propagated
|
||
alignment differences through the rest of the body. Sub-1-sec events
|
||
worked because their `end_offset` was below `0x1002`, so no sample
|
||
chunks landed in the metadata-page region and the strip's needle never
|
||
matched. Verified fix by re-feeding the BW 5-1-26 "copy 3sec" capture's
|
||
A5 frames into the file builder: output is now byte-identical to BW's
|
||
saved `M529LKIQ.G10` reference (8708 bytes, 0 differences).
|
||
- BW already concatenates frame contributions in stream order without
|
||
any de-duplication; SFM now does the same.
|
||
|
||
---
|
||
|
||
## v0.14.0 — 2026-05-02
|
||
|
||
### Changed (major rewrite)
|
||
|
||
- **`read_bulk_waveform_stream` — STRT-bounded chunk walk.** Replaces the
|
||
earlier `0x0400`-step / `max(key4[2:4], 0x0400)` chunk-counter formula,
|
||
which over-read ~5× past the actual event end into post-event circular-
|
||
buffer garbage. The new walk:
|
||
|
||
1. Probe at `counter = start_offset` (event 1: `0x0000`; event N:
|
||
`cur_key[2:4]`).
|
||
2. Parse `end_offset` from the STRT record at `data[17]` of the probe
|
||
response (`end_key[2:4]` field).
|
||
3. For event 1 only, read the two fixed metadata pages at counter
|
||
`0x1002` and `0x1004` — these contain the global session-start
|
||
compliance setup (Project / Client / User Name / Seis Loc /
|
||
Extended Notes ASCII strings). Continuation events skip these
|
||
(BW caches them across the session).
|
||
4. Walk sample chunks at **`0x0200` increments (NOT `0x0400`)**, bounded
|
||
by `end_offset` — the loop exits when
|
||
`next_chunk_counter + 0x0200 > end_offset`.
|
||
5. Send the proper TERM frame (see new `bulk_waveform_term_v2()`) with
|
||
`offset_word = end_offset - next_boundary` and
|
||
`params[2:4] = next_boundary BE`. The TERM response carries the
|
||
partial last chunk + 26-byte file footer.
|
||
|
||
- **New helpers:** `bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`
|
||
and `parse_strt_end_offset(a5_data)` in `minimateplus.framing`.
|
||
|
||
- **`stop_after_metadata` / `extra_chunks_after_metadata` kwargs are now
|
||
no-ops** under the v0.14.x walk. They are retained on the
|
||
`read_bulk_waveform_stream` signature for backward compatibility but log a
|
||
DEBUG line when set. The old "scan for `b'Project:'` and stop one chunk
|
||
later" workaround is obsolete — the loop is deterministically bounded by
|
||
the STRT-derived `end_offset`.
|
||
|
||
- **Project / Client / User Name / Seis Loc string source corrected.**
|
||
These come from the dedicated metadata pages at counter `0x1002` /
|
||
`0x1004`, not from "A5 frame 7" of the sample-chunk stream. The
|
||
earlier "A5 frame 7" claim was an artifact of the broken `0x0400`-step
|
||
walk where the bad counter formula coincidentally landed sample-chunk
|
||
fi=7 on top of the 0x1002 metadata page.
|
||
|
||
### Verified
|
||
|
||
- Three independent BW MITM captures (4-27-26 + 5-1-26 + 5-4-26) confirm
|
||
the new walk matches BW's behaviour event-for-event.
|
||
- `end_offset` values verified across 3 events: `0x1ABE` (4-27-26 2-sec),
|
||
`0x21F2` (5-1-26 3-sec), `0x417E` (5-1-26 event-2).
|
||
|
||
### Notes
|
||
|
||
- Earlier v0.13.0 / v0.13.1 / v0.13.2 entries describe partial steps along
|
||
the way (some of the file builder fixes, filename bugs, etc.) that were
|
||
superseded by the full rewrite. Treat this v0.14.0 entry as the
|
||
definitive landing point for the corrected SUB 5A protocol.
|
||
|
||
---
|
||
|
||
## v0.14.1 — 2026-05-04
|
||
|
||
### Fixed
|
||
|
||
- **`read_bulk_waveform_stream` — event-N probe counter off-by-`0x46`.**
|
||
Continuation events (start_key[2:4] != 0) were being probed at counter
|
||
`start_offset + 0x0046` instead of just `start_offset`. In the iteration
|
||
walk, `cur_key` from 1F is already the off=0x46 WAVEHDR record key, so the
|
||
earlier formula effectively double-counted the WAVEHDR offset. The probe
|
||
landed one WAVEHDR past the actual event start, the response no longer
|
||
contained the STRT record at byte 17, `parse_strt_end_offset` returned
|
||
`None`, and the chunk loop fell back to the `max_chunks=128` cap — walking
|
||
~110 chunks of post-event circular-buffer garbage. Verified against the
|
||
5-1-26 "copy 2nd address" and 5-4-26 BW 2-sec event captures: BW probes
|
||
counter=`0x2238` with key=`01112238` and STRT is present at byte 17 of
|
||
the response (end_offset=`0x417E`).
|
||
- **CLAUDE.md / docs/instantel_protocol_reference.md** — corrected the
|
||
event-N section to clarify that `start_key` in those formulas is the
|
||
off=0x46 key, not the off=0x2C boundary key, and removed the spurious
|
||
`+0x46` from the chunk-walk pseudocode.
|
||
|
||
---
|
||
|
||
## v0.13.2 — 2026-05-01
|
||
|
||
### Fixed
|
||
|
||
- **`_extract_record_type` — third 0C-record header format ("short", 8 bytes).**
|
||
A live SFM download against BE11529 produced files named `M5290000.000`
|
||
(zero-stamped) because the 0C waveform record's first bytes were
|
||
`01 05 07 ea ...` — neither the 9-byte single-shot layout (`0x10` at byte 1)
|
||
nor the 10-byte continuous layout (`0x10` at bytes 0 and 2). Investigation
|
||
showed this is a third format observed in the wild: an 8-byte header with no
|
||
marker bytes at all (`[day][month][year_BE:2][unknown][hour][min][sec]`).
|
||
The detection logic now scans the year (uint16 BE) at byte 2 / byte 3 / byte
|
||
4 and picks whichever offset returns a sensible year (2015–2050) — each
|
||
format has the year at a unique position so this disambiguates cleanly.
|
||
- New format → `event.record_type = "Waveform (Short)"`,
|
||
`Timestamp.from_short_record()`.
|
||
- Existing single-shot and continuous parsers unchanged.
|
||
- The user's event from May 1, 2026 13:21:37 now correctly resolves to a
|
||
filename like `M529LKIQ.G10` instead of `M5290000.000`.
|
||
|
||
### Added
|
||
|
||
- `Timestamp.from_short_record(data)` — decodes the 8-byte header.
|
||
- `_detect_record_format(data)` — internal helper returning
|
||
`"single_shot" / "continuous" / "short" / None` via year-position scan.
|
||
|
||
---
|
||
|
||
## v0.13.1 — 2026-05-01
|
||
|
||
### Fixed
|
||
|
||
- **`_extract_record_type` — Continuous-mode record headers misclassified as Unknown.**
|
||
In single-shot mode the 0C waveform record's 9-byte header puts the sub_code
|
||
marker `0x10` at byte 1, with the day at byte 0. In Continuous mode the
|
||
header is 10 bytes with the marker at byte 0 *and* byte 2, and the day at
|
||
byte 1. Previous logic only inspected byte 1 and treated any value other
|
||
than `0x10` / `0x03` as `"Unknown"`, which prevented `event.timestamp` from
|
||
being populated for any continuous-mode event whose day-of-month wasn't
|
||
exactly 3 or 16. As a downstream effect, `blastware_filename()` saw
|
||
`event.timestamp == None`, fell back to `stem="0000"` / `ab="00"`, and
|
||
produced filenames like `M5290000.000`. Discovered from a live SFM run on
|
||
BE11529 in continuous mode (day-of-month = 5).
|
||
Now disambiguates by checking BOTH byte 0 and byte 2: if both are `0x10`,
|
||
it's the 10-byte continuous header; else if byte 1 is `0x10`, it's the
|
||
9-byte single-shot header. Day-of-month no longer matters.
|
||
|
||
*Superseded by v0.13.2 — the user's actual record uses a third 8-byte format
|
||
with no `0x10` markers, which v0.13.1 still misclassified.*
|
||
|
||
---
|
||
|
||
## v0.13.0 — 2026-05-01
|
||
|
||
### Fixed
|
||
|
||
- **SUB 5A bulk waveform stream — over-read bug for events ≥ 2 sec.**
|
||
`read_bulk_waveform_stream` was walking the chunk counter past the actual
|
||
end of the event, picking up post-event circular-buffer garbage that
|
||
corrupted reconstructed Blastware files for any waveform > ~1 sec. The
|
||
loop now extracts the event's `end_offset` from the STRT record at
|
||
`data[23:27]` of the probe response and stops the chunk walk when the next
|
||
counter would step past it. Verified against three BW MITM captures
|
||
(4-27-26 + 5-1-26): 2-sec event drops from 37 over-read chunks to 7
|
||
bounded chunks; 3-sec drops to 9; non-zero-start "event 2" drops to 9.
|
||
|
||
### Added
|
||
|
||
- `framing.bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)` —
|
||
computes the corrected SUB 5A TERM frame's `(offset_word, params)` per the
|
||
formula confirmed across all 3 BW captures. Not yet wired into
|
||
`read_bulk_waveform_stream` (the legacy TERM is still used to preserve the
|
||
existing `blastware_file.write_blastware_file` frame-structure expectations);
|
||
available for the next iteration that switches to BW's 0x0200 chunk step.
|
||
- `framing.parse_strt_end_offset(a5_data)` — extracts the event-end pointer
|
||
from the STRT record in an A5 response payload.
|
||
|
||
### Documentation
|
||
|
||
- **CLAUDE.md and `docs/instantel_protocol_reference.md` extensively
|
||
rewritten** to reflect the corrected SUB 5A protocol. See:
|
||
- CLAUDE.md "SUB 5A — chunk counter formula (REWRITTEN 2026-05-01)"
|
||
- CLAUDE.md "SUB 5A — STRT record encodes end_offset"
|
||
- CLAUDE.md "SUB 5A — TERM frame formula"
|
||
- CLAUDE.md "SUB 5A — fixed metadata pages 0x1002 and 0x1004"
|
||
- CLAUDE.md "SUB 0A — WAVEHDR response length distinguishes events from
|
||
boundaries" (0x46 = real event, 0x2C = boundary marker)
|
||
- protocol reference §7.8.5 / §7.8.6 / §7.8.7 / §7.8.8
|
||
- The previous chunk-counter formula (`max(key4[2:4], 0x0400) + (chunk-1) *
|
||
0x0400`) is now marked DEPRECATED and explicitly tagged WRONG with
|
||
pointers to the new sections, so future work doesn't re-derive it.
|
||
|
||
### Known minor diffs vs Blastware (deferred to a follow-up)
|
||
|
||
- We still use the OLD 0x0400 chunk step rather than BW's 0x0200; switching
|
||
also requires updating `blastware_file.write_blastware_file`'s skip values
|
||
and "extra chunk after metadata" logic, which depends on a fresh capture
|
||
to verify.
|
||
- We still use the legacy fixed `offset_word=0x005A` TERM frame rather than
|
||
BW's `end_offset - next_boundary` formula, for the same reason.
|
||
- Two fixed metadata pages at counter `0x1002` and `0x1004` are not yet
|
||
read explicitly; under the current 0x0400 walk their content is reachable
|
||
via the sample chunk that covers buffer addresses `[0x1000, 0x1400)`.
|
||
|
||
---
|
||
|
||
## v0.12.6 — 2026-05-01
|
||
|
||
### Fixed
|
||
|
||
- **`blastware_file.py` — waveform frame classification** — A5 frame classification for
|
||
waveform-only vs header-only frames now uses `frame.record_type` instead of frame index.
|
||
Only waveform frames (0x46) are written to the file body; metadata frames are skipped.
|
||
Fixes spurious data corruption from incorrectly classified frames.
|
||
|
||
- **`s3_analyzer.py` — A5/5A frame naming** — Bulk waveform stream frames (SUB 5A response)
|
||
are now correctly labeled "A5" in analyzer output instead of being conflated with other
|
||
multi-frame responses (SUB A4, E5, etc.).
|
||
|
||
- **`S3FrameParser` — frame terminator detection** — Corrected the bare ETX terminator
|
||
detection. Frame termination is now correctly identified by a standalone `ETX=0x03` byte,
|
||
not by the `DLE+ETX` sequence (which is part of the payload when it appears within a frame).
|
||
|
||
---
|
||
|
||
## v0.12.5 — 2026-04-21
|
||
|
||
### Added
|
||
|
||
- **`seismo_lab.py` — Download tab** — New fourth tab for live wire-byte capture during event
|
||
downloads. Captures both BW→device and device→S3 frames in real time, allowing inspection
|
||
of the 5A bulk stream chunk sequence and frame-by-frame analysis without needing a bridge
|
||
or MITM proxy. Files are saved with user-specified labels for easy tracking.
|
||
|
||
### Changed
|
||
|
||
- **`s3_bridge.py` — raw captures always-on by default** — `--raw-bw` and `--raw-s3` now
|
||
default to `"auto"` instead of `None`. Every bridge session automatically generates
|
||
timestamped `raw_bw_<ts>.bin` and `raw_s3_<ts>.bin` files alongside the `.bin`/`.log`
|
||
session files. Pass `--raw-bw ""` (explicit empty string) to disable if needed.
|
||
|
||
- **`gui_bridge.py` — raw capture checkboxes pre-checked** — Both "BW→S3 raw" and
|
||
"S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names
|
||
the files). Unchecking a box passes `--raw-bw ""` to explicitly disable capture.
|
||
|
||
- **`Bridge tab` — TCP mode added** — Serial/TCP radio toggle allows connection via cellular
|
||
modem (RV50/RV55) instead of direct RS-232. Supports multi-capture design (simultaneous
|
||
Bridge + Analyzer + Download sessions).
|
||
|
||
- **`ach_server.py` — TX capture added (`raw_tx_<ts>.bin`)** — Every ACH inbound session
|
||
now saves both directions: `raw_rx_<ts>.bin` (device → us, S3 side, as before) and
|
||
`raw_tx_<ts>.bin` (us → device, BW side). Both files are usable in the Analyzer.
|
||
TX bytes are buffered in memory until startup handshake succeeds (same as RX), preventing
|
||
scanner probes from creating empty files.
|
||
|
||
---
|
||
|
||
## v0.12.4 — 2026-04-21 (protocol analysis / docs only — no code changes)
|
||
|
||
### Discovered
|
||
|
||
- **compliance_raw is wire-encoded, not logical bytes** — `read_compliance_config()` returns
|
||
bytes that include DLE prefix bytes (`0x10`) before any `0x03` values (because S3FrameParser
|
||
preserves DLE+ETX inner-frame pairs as two literal bytes). The previous CLAUDE.md claim that
|
||
"S3FrameParser handles this transparently so compliance_raw contains logical bytes" was wrong.
|
||
|
||
- **anchor-9 behavior per recording mode** (confirmed from 4-20-26 BW write captures):
|
||
- Single Shot (0x00) / Continuous (0x01): anchor-9 = `0x00`
|
||
- Histogram (0x03): anchor-9 = `0x10` — the E5 DLE prefix for the `0x03` recording_mode byte
|
||
- Histogram+Continuous (0x04): anchor-9 = `0x10` — an actual stored config byte for this mode
|
||
Anchor position shifts by ±1 when recording_mode = `0x03` due to the extra DLE byte; the
|
||
dynamic anchor search (`buf.find(ANCHOR, 0, 150)`) handles this correctly without code changes.
|
||
|
||
- **Write frame ETX escaping** — BW escapes `0x03` bytes in write frame data as `0x10 0x03`
|
||
on the wire. Our `build_bw_write_frame` sends data bytes raw without ETX escaping. Device
|
||
accepts our raw writes for all tested modes. Hypothesis: device write parser uses the
|
||
offset/length field for frame boundaries, not ETX scanning, making ETX escaping optional.
|
||
Histogram mode (recording_mode = 0x03) write via SFM from a non-Histogram starting state
|
||
not yet tested.
|
||
|
||
- **BW write payload vs E5 read payload are byte-identical** around the anchor region (confirmed
|
||
by comparing 3-11-26 BW TX and S3 captures). BW does NOT strip DLE prefix bytes before writing;
|
||
it round-trips the wire-encoded bytes verbatim with only the modified fields changed.
|
||
|
||
- **Capture folder content catalogued** — see CLAUDE.md "BW capture reference" table for a
|
||
summary of all available protocol captures and their contents.
|
||
|
||
---
|
||
|
||
## v0.12.3 — 2026-04-20
|
||
|
||
### Added
|
||
|
||
- **Auto Call Home config protocol** — Full read/write/decode/encode pipeline for the
|
||
device's Remote Access → Setup Unit ACH settings, confirmed from 4-20-26 call home
|
||
settings captures.
|
||
|
||
**Protocol (new):**
|
||
- `SUB 0x2C` — Call Home Config READ (response `0xD3`); two-step read; data offset
|
||
`0x7C` = 124; raw payload 125 bytes (1-byte longer than DATA_LENGTH due to DLE-escaped
|
||
`\x10\x03` at raw[117:119] representing num_retries = 3)
|
||
- `SUB 0x7E` — Call Home Config WRITE (response `0x81`); 127-byte payload (125-byte read
|
||
payload + `\x00\x00`); offset = `data[1]+2 = 0x7E`; write format (DLE-aware checksum)
|
||
- `SUB 0x7F` — Call Home WRITE CONFIRM (response `0x80`); no data
|
||
|
||
**Field map (confirmed from 10-frame BW TX diff):**
|
||
- `raw[5]` — auto_call_home_enabled (bool)
|
||
- `raw[6:46]` — dial_string (40-byte null-padded ASCII)
|
||
- `raw[87]` — after_event_recorded (bool)
|
||
- `raw[91]` — at_specified_times (bool)
|
||
- `raw[93]` — time1_enabled / `raw[101]` — time1_hour / `raw[102]` — time1_min
|
||
- `raw[95]` — time2_enabled / `raw[105]` — time2_hour / `raw[106]` — time2_min
|
||
- `raw[117:119]` — `\x10\x03` (DLE-escaped 0x03 = num_retries value 3)
|
||
- `raw[120]` — time_between_retries_sec / `raw[122]` — wait_for_connection_sec / `raw[124]` — warm_up_time_sec
|
||
|
||
**Library (`minimateplus/`):**
|
||
- `models.py` — `CallHomeConfig` dataclass (14 fields; `raw` bytes preserved for
|
||
round-trip writes)
|
||
- `protocol.py` — `SUB_CALL_HOME = 0x2C`, `SUB_CALL_HOME_WRITE = 0x7E`,
|
||
`SUB_CALL_HOME_CONFIRM = 0x7F`; `read_call_home_config()`, `write_call_home_config()`
|
||
- `client.py` — `get_call_home_config()`, `set_call_home_config()`,
|
||
`_decode_call_home_config()` (handles DLE prefix at raw[117]),
|
||
`_encode_call_home_config()` (patches in-place; raises `ValueError` if hour/min = 3)
|
||
|
||
**REST API (`sfm/server.py`):**
|
||
- `GET /device/call_home` — reads and decodes call home config from device
|
||
- `POST /device/call_home` — reads, patches specified fields, writes back to device
|
||
- `CallHomeConfigBody` Pydantic model with 9 optional writable fields
|
||
|
||
**Web UI (`sfm/sfm_webapp.html`):**
|
||
- New "Call Home" tab with enable flag, dial string (read-only), after-event trigger,
|
||
at-specified-times flag, two time slots (enable + HH:MM each), and read-only retry
|
||
settings (num_retries, time_between_retries_sec, wait_for_connection_sec,
|
||
warm_up_time_sec)
|
||
- "Read from Device", "Write to Device", "Clear Form" action buttons
|
||
- Client-side guard: rejects hour or minute value equal to 3 with a clear message
|
||
explaining the DLE-encoding limitation
|
||
|
||
---
|
||
|
||
## v0.12.2 — 2026-04-20
|
||
|
||
### Added / Fixed
|
||
|
||
- **Geophone sensitivity / maximum range field confirmed** — 4-20-26 geo sensitivity
|
||
captures (1.25 in/s vs 10 in/s) diffed across all three SUB 71 write chunks and both
|
||
E5 read payloads. The `geo_range` uint8 field per channel is now fully confirmed:
|
||
- E5 read offset: `channel_label + 33`; SUB 71 write offset: `channel_label + 29`
|
||
- `0x00` = Normal 10.000 in/s (standard gain); `0x01` = Sensitive 1.250 in/s (high gain)
|
||
- **Correction:** previous hypothesis (`channel_label+20`, `0x01`=Normal) was wrong.
|
||
`channel_label+20` reads `0x01` on ALL captures regardless of range — not this field.
|
||
- `_decode_compliance_config_into`: read offset corrected from `tran_pos+20` → `tran_pos+33`
|
||
- `_encode_compliance_config`: added `geo_range` parameter; writes to Tran/Vert/Long at `+29`
|
||
- `apply_config`: added `geo_range` parameter
|
||
- `POST /device/config`: added `geo_range` to `DeviceConfigBody`
|
||
- Web UI Config tab: added "Maximum Range — Geo" select (Normal / Sensitive)
|
||
- Web UI Device tab: added "Max Range (geo)" row to compliance table
|
||
|
||
- **`recording_mode` + `histogram_interval_sec` confirmed and implemented** (4-20-26 captures)
|
||
- `recording_mode`: uint8 at anchor−8 (E5 read) / anchor−7 (write); enum: 0x00=Single Shot,
|
||
0x01=Continuous, 0x03=Histogram, 0x04=Histogram+Continuous
|
||
- `histogram_interval_sec`: uint16 BE seconds at anchor−4; same offset in read & write;
|
||
valid: 2, 5, 15, 60, 300, 900 (matching Blastware dropdown: 2s, 5s, 15s, 1m, 5m, 15m)
|
||
- Both fields added to `ComplianceConfig`, `_decode_compliance_config_into`,
|
||
`_encode_compliance_config`, `apply_config`, REST API body, and web UI
|
||
|
||
---
|
||
|
||
## v0.12.1 — 2026-04-16
|
||
|
||
### Added
|
||
|
||
- **`sfm/server.py` — `_LiveCache`** — in-memory live device cache that eliminates
|
||
redundant TCP round-trips between web requests. Plain Python dict +
|
||
`threading.Lock`, no extra dependencies.
|
||
|
||
Cache strategy per endpoint:
|
||
|
||
| Endpoint | Strategy |
|
||
|---|---|
|
||
| `GET /device/info` | Indefinite; invalidated by `POST /device/config` |
|
||
| `GET /device/events` | Count-probe fast path — `poll()+count_events()` (~2 s); returns cached data if event count is unchanged; full download only when new events are detected |
|
||
| `GET /device/monitor/status` | 30-second TTL; invalidated immediately on monitor start/stop |
|
||
| `GET /device/event/{idx}/waveform` | Permanent per-index (waveforms are immutable once recorded) |
|
||
|
||
- **`?force=true`** query param on all cached endpoints — bypasses cache and forces
|
||
a fresh read from the device.
|
||
|
||
- **Cache invalidation hooks** — `POST /device/config` marks device info and events
|
||
stale; `POST /device/monitor/start` and `/stop` evict the monitor status entry
|
||
immediately so the next status poll reflects the actual device state.
|
||
|
||
---
|
||
|
||
## v0.12.0 — 2026-04-13
|
||
|
||
### Added
|
||
|
||
- **`sfm/server.py` — `_LiveCache`** — in-memory live device cache, eliminating
|
||
redundant TCP round-trips between requests. No extra dependencies (plain Python
|
||
dict + threading.Lock). Replaces the SQLAlchemy-based `sfm/cache.py` experiment
|
||
from the `feature/intelligent-caching` branch.
|
||
|
||
Cache behaviour by endpoint:
|
||
|
||
| Endpoint | Cache strategy |
|
||
|---|---|
|
||
| `GET /device/info` | Indefinite; invalidated by `POST /device/config` |
|
||
| `GET /device/events` | Count-probe fast path: quick `poll()+count_events()` (~2s); return cache if count matches; full download only when new events detected |
|
||
| `GET /device/monitor/status` | 30-second TTL; invalidated by monitor start/stop |
|
||
| `GET /device/event/{idx}/waveform` | Permanent per-index (waveforms are immutable) |
|
||
|
||
- **`?force=true` param** on all four cached endpoints — bypasses cache and re-reads
|
||
from device.
|
||
|
||
- **`POST /device/config` cache invalidation** — marks device info + events dirty so
|
||
the next read reflects the new compliance config.
|
||
|
||
- **`POST /device/monitor/start` / `stop` cache invalidation** — evicts the monitor
|
||
status cache entry immediately so the next poll returns the updated state.
|
||
|
||
### Removed
|
||
|
||
- `sfm/cache.py` — SQLAlchemy-based cache from the experimental caching branch.
|
||
Its logic has been ported to the sqlite3-native `_LiveCache` class above.
|
||
`sqlalchemy` is no longer a dependency.
|
||
|
||
---
|
||
|
||
## v0.11.0 — 2026-04-13
|
||
|
||
### Added
|
||
|
||
- **`sfm/database.py` — SeismoDb** — SQLite persistence layer for all ACH data.
|
||
Three tables, all unit-keyed by serial number:
|
||
- `ach_sessions` — one row per inbound call-home: serial, timestamp, peer IP,
|
||
events_downloaded, monitor_entries, duration_seconds
|
||
- `events` — one row per triggered waveform event: serial, waveform_key (dedup),
|
||
timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location
|
||
strings, sample_rate, record_type, false_trigger flag
|
||
- `monitor_log` — one row per monitoring interval: serial, waveform_key (dedup),
|
||
start_time, stop_time, duration_seconds, geo_threshold_ips
|
||
- WAL mode, per-request connections — safe for the single-writer / occasional-reader
|
||
ACH server pattern
|
||
- Deduplication by `(serial, waveform_key)` UNIQUE constraint — re-runs and repeat
|
||
call-homes never produce duplicate rows
|
||
|
||
- **`ach_server.py` — DB integration** — after each successful call-home, writes new
|
||
events and monitor log entries to `seismo_relay.db` then records the session in
|
||
`ach_sessions`. DB write failures are logged as warnings and do not abort the session.
|
||
|
||
- **`sfm/server.py` — DB read endpoints**:
|
||
- `GET /db/units` — distinct serials with last_seen, total_events, total_monitor_entries
|
||
- `GET /db/events` — query events with serial / date range / false_trigger filters
|
||
- `GET /db/monitor_log` — query monitoring intervals
|
||
- `GET /db/sessions` — query ACH call-home sessions
|
||
- `PATCH /db/events/{id}/false_trigger` — flag/unflag false triggers (for review UI)
|
||
|
||
### Architecture
|
||
|
||
- seismo-relay DB is unit-keyed only — no project concepts. Project aggregation is
|
||
terra-view's responsibility via `UnitAssignment` / `DeploymentRecord` + date range
|
||
queries against the SFM DB endpoints.
|
||
- DB file lives at `bridges/captures/seismo_relay.db` by default.
|
||
|
||
---
|
||
|
||
## v0.10.0 — 2026-04-11
|
||
|
||
### Added
|
||
|
||
- **`MiniMateClient.get_monitor_log_entries(skip_keys=None)`** — browse-mode walk
|
||
(`1E → 0A → 1F`) that collects partial records (`0x2C` record type) from the device's
|
||
event list without triggering a full waveform download (no 0C or 5A). Returns
|
||
`list[MonitorLogEntry]`. Each entry represents one continuous monitoring interval where
|
||
no threshold was exceeded.
|
||
|
||
- **`_decode_0a_partial_header(raw_data, index, key4)`** in `client.py` — decodes a SUB
|
||
0x0A response payload whose record type is `0x2C`. Extracts:
|
||
- `start_time` / `stop_time` — two consecutive timestamps; auto-detects 9-byte
|
||
(sub_code=0x10, single-shot) vs 10-byte (sub_code=0x03, continuous) format from
|
||
`raw_data[11]`. Handles a 1-byte gap between the two timestamps that occurs when
|
||
ts1 and ts2 share the same minute:second.
|
||
- `serial` — device serial string found via `b"BE"` anchor scan.
|
||
- `geo_threshold_ips` — trigger level found via `b"Geo: "` anchor scan.
|
||
|
||
- **`MonitorLogEntry` dataclass** in `models.py` — new model for partial records:
|
||
`index`, `key`, `start_time`, `stop_time`, `serial`, `geo_threshold_ips`,
|
||
`raw_header`, and a `duration_seconds` property.
|
||
|
||
- **`read_waveform_header()` return value extended** — now returns `(data_rsp.data, length)`
|
||
(full payload) instead of `(data_rsp.data[11:11+length], length)`. Callers get the
|
||
complete payload including the record-type byte at position 0. Full records use
|
||
`raw_data[11:11+length]` as before; partial records are detected by `raw_data[0] == 0x2C`.
|
||
|
||
- **ACH server: monitor log collection** — after `get_events()`, calls
|
||
`get_monitor_log_entries(skip_keys=seen_keys)` and saves new entries to
|
||
`monitor_log.json` in the session directory. Monitor log keys are included in
|
||
`downloaded_keys` for state persistence (no re-processing on next call-home).
|
||
|
||
- **`_monitor_log_entry_to_dict()`** in `ach_server.py` — serialises a `MonitorLogEntry`
|
||
to a JSON-compatible dict with ISO-format timestamps.
|
||
|
||
### Protocol / Documentation
|
||
|
||
- **SUB 0x0A partial record (0x2C) format confirmed** (✅ 4-11-26 MITM capture, 12 frames):
|
||
- Record type `0x2C` at `raw_data[0]`; length < 64 bytes.
|
||
- Two timestamps at `raw_data[11:]` — start and stop of the monitoring interval.
|
||
- ASCII metadata region after timestamps: `BE<serial>\x00Geo: <float> in/s`.
|
||
- Edge case: 1-byte separator between timestamps when ts1 and ts2 share minute:second.
|
||
- 10-byte timestamp format (sub_code=0x03) signalled by `raw_data[11] == 0x10`.
|
||
|
||
- **Key reuse detection for monitor log entries** — monitor log keys are tracked alongside
|
||
event keys in `ach_state.json` so the ACH server does not re-process them after a
|
||
call-home cycle.
|
||
|
||
---
|
||
|
||
## v0.9.0 — 2026-04-11
|
||
|
||
### Added
|
||
|
||
- **`MiniMateClient.list_event_keys()`** — fast browse-mode walk (1E → 0A → 1F, no waveform
|
||
download) that returns the list of event key hex strings currently stored on the device.
|
||
Used by the ACH server as a cheap pre-check before deciding whether to call `get_events()`.
|
||
|
||
- **`get_events(skip_waveform_for_keys=set(...))`** — new optional parameter. For any key in
|
||
the set the function performs only 0A + 1F(browse) instead of the full
|
||
1E-arm → 0C → POLL×3 → 5A sequence. Eliminates redundant waveform downloads on repeat
|
||
call-homes when the device still holds previously downloaded events.
|
||
|
||
- **`MiniMateClient.delete_all_events()`** — erases all events from device memory using the
|
||
confirmed 4-step sequence:
|
||
- SUB 0xA3 `begin_erase_all` — initiate erase (token=0xFE) → ack 0x5C
|
||
- SUB 0x1C `read_monitor_status` — intermediate status read (Blastware-required)
|
||
- SUB 0x06 `read_event_storage_range` — verify storage state (token=0xFE) → 36-byte response
|
||
- SUB 0xA2 `confirm_erase_all` — commit erase (token=0xFE) → ack 0x5D
|
||
|
||
All four steps confirmed from 4-11-26 MITM capture of a live Blastware ACH session.
|
||
After a successful call, the device's event counter resets to `0x01110000`.
|
||
|
||
- **`MiniMateProtocol` erase methods**: `begin_erase_all()`, `confirm_erase_all()`,
|
||
`read_event_storage_range()` added to `protocol.py` with documented SUB constants
|
||
`SUB_ERASE_ALL_BEGIN = 0xA3` and `SUB_ERASE_ALL_CONFIRM = 0xA2`.
|
||
|
||
- **`bridges/ach_mitm.py`** — transparent TCP-to-TCP MITM proxy. Listens for inbound unit
|
||
connections, connects upstream to a real Blastware ACH server, and saves both directions
|
||
to `raw_bw_<ts>.bin` / `raw_s3_<ts>.bin` files matching the existing capture format.
|
||
Used to capture the 4-11-26 Blastware ACH session including event deletion.
|
||
Usage: `python bridges/ach_mitm.py --bw-host 127.0.0.1 --bw-port 9999 --listen-port 9998`
|
||
|
||
- **ACH server: key-based state tracking** — `ach_state.json` now stores
|
||
`downloaded_keys: [hex_strings]` and `max_downloaded_key: hex_string` per unit instead of
|
||
`event_count: N`. This correctly handles the standard workflow where events are deleted
|
||
from the device after upload — a count-based approach would see `count=0` on the next
|
||
call-home and silently skip new events.
|
||
|
||
- **ACH server: `--clear-after-download` flag** — after a successful download (at least one
|
||
new event saved), erases all events from the device using `delete_all_events()`. Mirrors
|
||
the standard Blastware ACH workflow. On success, `downloaded_keys` and
|
||
`max_downloaded_key` are reset to empty so the next session starts fresh.
|
||
|
||
- **ACH server: post-erase key-reuse detection** — after an external erase (Blastware or
|
||
manual), device keys restart from `0x01110000`, colliding with previously downloaded keys.
|
||
On each browse walk, if `max(device_keys) < max_downloaded_key` (device counter rolled
|
||
back), all device keys are treated as new regardless of `seen_keys`. This also catches
|
||
erases performed by Blastware between our sessions.
|
||
|
||
### Protocol / Documentation
|
||
|
||
- **SUB 0xA3 / SUB 0xA2 — erase-all sequence confirmed** (✅ 4-11-26 MITM capture):
|
||
Both frames use `token=0xFE` at `params[7]` and are standard `build_bw_frame` requests
|
||
(not write-format). Response SUBs follow the standard formula: 0x5C and 0x5D.
|
||
The intermediate 0x1C + 0x06 reads between them are required by Blastware.
|
||
|
||
- **SUB 0x06 — event storage range read confirmed** (✅ 4-11-26 MITM capture):
|
||
Two-step read, data offset = 0x24 (36 bytes). The last 8 bytes of the response contain
|
||
the first and last stored event keys (4 bytes each). After a successful erase, both keys
|
||
read as `01110000` (device-empty state).
|
||
|
||
- **Event key counter resets to `0x01110000` after erase** — confirmed by observing key
|
||
`01110000` on the device immediately after the MITM erase session.
|
||
|
||
---
|
||
|
||
## v0.8.0 — 2026-04-07
|
||
|
||
### Added
|
||
|
||
- **Write pipeline end-to-end** — `push_config_raw(event_index_data, compliance_data,
|
||
trigger_data, waveform_data)` on `MiniMateClient` orchestrates the full
|
||
`68→73 | 71×3→72 | 82→83 | 69→74→72` write sequence.
|
||
|
||
- **`build_bw_write_frame(sub, data, *, offset, params)`** in `framing.py` — dedicated frame
|
||
builder for write commands (SUBs 0x68–0x83). Doubles only the BW_CMD byte; all other
|
||
bytes including offset, params, data, and checksum are written raw. Uses the large-frame
|
||
DLE-aware checksum (`sum(b for b in payload[2:] if b != 0x10) + 0x10) & 0xFF`).
|
||
|
||
- **`MiniMateProtocol` write methods** — `write_event_index()`, `write_compliance()`,
|
||
`write_trigger_config()`, `write_waveform_data()`, `write_confirm()`,
|
||
`start_monitoring()`, `stop_monitoring()`.
|
||
|
||
- **`AchSession` inbound server** (`bridges/ach_server.py`) — accepts call-home TCP
|
||
connections, runs the full handshake + device-info + event-download sequence, saves
|
||
`device_info.json` + `events.json` per session.
|
||
|
||
### Protocol / Documentation
|
||
|
||
- **Write frame format confirmed** (✅ 3-11-26 BW TX capture, all 11 frames): only BW_CMD
|
||
byte `0x10` is doubled; all other bytes sent raw. Standard `build_bw_frame` DLE-stuffing
|
||
is incorrect for write commands.
|
||
- **Write ack responses** confirmed as 17-byte zero-data S3 frames.
|
||
- **Monitoring SUBs 0x96/0x97** confirmed from 4-8-26 capture.
|
||
- **SESSION_RESET signal** (`41 03`) required before POLL for monitoring units.
|
||
- **SUB 0x1C monitoring flag** at `section[1]`: `0x00` = idle, `0x10` = monitoring.
|
||
Confirmed by byte-diff of all 144 data frames in 4-8-26/2ndtry capture.
|
||
|
||
---
|
||
|
||
## v0.7.0 — 2026-04-03
|
||
|
||
### Added
|
||
- **Raw ADC waveform decode — `_decode_a5_waveform(frames_data, event)`** in `client.py`.
|
||
Parses the complete set of SUB 5A A5 response frames into per-channel time-series:
|
||
- Reads the STRT record from A5[0] (bytes 7+): extracts `total_samples` (BE uint16 at +8),
|
||
`pretrig_samples` (BE uint16 at +16), and `rectime_seconds` (uint8 at +18) into
|
||
`event.total_samples / pretrig_samples / rectime_seconds`.
|
||
- Skips the 6-byte preamble (`00 00 ff ff ff ff`) that follows the 21-byte STRT header;
|
||
waveform data begins at `strt_pos + 27`.
|
||
- Strips the 8-byte per-frame counter header from A5[1–6, 8] before appending waveform bytes.
|
||
- Skips A5[7] (metadata-only) and A5[9] (terminator).
|
||
- **Cross-frame alignment correction**: accumulates `running_offset % 8` across all frames
|
||
and discards `(8 − align) % 8` leading bytes per frame to re-align to a T/V/L/M boundary.
|
||
Required because individual frame waveform payloads are not always multiples of 8 bytes.
|
||
- Decodes as 4-channel interleaved signed 16-bit LE at 8 bytes per sample-set:
|
||
bytes 0–1 = Tran, 2–3 = Vert, 4–5 = Long, 6–7 = Mic.
|
||
- Stores result in `event.raw_samples = {"Tran": [...], "Vert": [...], "Long": [...], "Mic": [...]}`.
|
||
- **`download_waveform(event)` public method** on `MiniMateClient`.
|
||
Issues a full SUB 5A stream with `stop_after_metadata=False`, then calls
|
||
`_decode_a5_waveform()` to populate `event.raw_samples` and `event.total_samples /
|
||
pretrig_samples / rectime_seconds`. Previously only metadata frames were fetched during
|
||
`get_events()`; raw waveform data is now available on demand.
|
||
- **`Event` model new fields** (`models.py`): `total_samples`, `pretrig_samples`,
|
||
`rectime_seconds` (from STRT record), and `_waveform_key` (4-byte key stored during
|
||
`get_events()` for later use by `download_waveform()`).
|
||
|
||
### Protocol / Documentation
|
||
- **SUB 5A A5[0] STRT record layout confirmed** (✅ 2026-04-03, 4-2-26 blast capture):
|
||
- STRT header is 21 bytes: `b"STRT"` + length fields + `total_samples` (BE uint16 at +8) +
|
||
`pretrig_samples` (BE uint16 at +16) + `rectime_seconds` (uint8 at +18).
|
||
- Followed by 6-byte preamble: `00 00 ff ff ff ff`. Waveform begins at `strt_pos + 27`.
|
||
- Confirmed: 4-2-26 blast → `total_samples=9306`, `pretrig_samples=298`, `rectime_seconds=70`.
|
||
- **Blast/waveform mode A5 format confirmed** (✅ 2026-04-03, 4-2-26 blast capture):
|
||
4-channel interleaved int16 LE at 8 bytes per sample-set; cross-frame alignment correction
|
||
required. 948 of 9306 total sample-sets captured via `stop_after_metadata=True` (10 frames).
|
||
- **Noise/histogram mode A5 format — endianness corrected** (✅ 2026-04-03, 3-31-26 capture):
|
||
32-byte block samples are signed 16-bit **little-endian** (previously documented as BE).
|
||
`0a 00` → LE int16 = 10 (correct noise floor); BE would give 2560 (wrong).
|
||
- Protocol reference §7.6 rewritten — split into §7.6.1 (Blast/Waveform mode) and §7.6.2
|
||
(Noise/Histogram mode), each with confirmed field layouts and open questions noted.
|
||
|
||
---
|
||
|
||
## v0.6.0 — 2026-04-02
|
||
|
||
### Added
|
||
- **True event-time metadata via SUB 5A bulk waveform stream** — `get_events()` now issues a SUB 5A request after each SUB 0C download, reads the A5 response frames, and extracts the `Client:`, `User Name:`, and `Seis Loc:` fields as they existed at the moment the event was recorded. Previously these fields were backfilled from the current compliance config (SUB 1A), which reflects today's setup, not the setup active when the event triggered.
|
||
- `build_5a_frame(offset_word, raw_params)` in `framing.py` — reproduces Blastware's exact wire format for SUB 5A requests: raw (non-DLE-stuffed) `offset_hi`, DLE-stuffed params, and a DLE-aware checksum where `10 XX` pairs count only `XX`.
|
||
- `bulk_waveform_params()` returns 11 bytes (extra trailing `0x00` confirmed from 1-2-26 BW wire capture).
|
||
- `read_bulk_waveform_stream(key4, *, stop_after_metadata=True, max_chunks=32)` in `protocol.py` — loops sending chunk requests (counter increments `0x0400` per chunk), stops early when `b"Project:"` is found, then sends a termination frame.
|
||
- `_decode_a5_metadata_into(frames_data, event)` in `client.py` — needle-searches A5 frame data for `Project:`, `Client:`, `User Name:`, `Seis Loc:`, `Extended Notes` and overwrites `event.project_info`.
|
||
- **`get_events()` sequence extended** — now `1E → 0A → 0C → 5A → 1F` per event.
|
||
|
||
### Fixed
|
||
- **Compliance config (SUB 1A) channel block missing** — orphaned `self._send(build_bw_frame(SUB_COMPLIANCE, 0x2A, _DATA_PARAMS))` before the B/C/D receive loop had no corresponding `recv_one()`, shifting all subsequent receives one step behind and leaving frame D's channel-block data (trigger_level_geo, alarm_level_geo, max_range_geo) unread. Removed the orphaned send. Total config bytes received now correctly ~2126 (was ~1071).
|
||
- **Compliance config anchor search range** — `_decode_compliance_config_into()` searched `cfg[40:100]` for the sample-rate/record-time anchor. With the orphaned-send bug fixed the 44-byte padding it had been adding is gone, and the anchor now appears at `cfg[11]`. Search widened to `cfg[0:150]` to be robust to future layout shifts.
|
||
- Removed byte-content deduplication from `read_compliance_config()` — was masking the real receive-ordering bug.
|
||
|
||
### Protocol / Documentation
|
||
- **SUB 5A frame format confirmed** — `offset_hi` byte (`0x10`) must be sent raw (not DLE-stuffed); checksum is DLE-aware (only the second byte of a `10 XX` pair is summed). Standard `build_bw_frame` DLE-stuffs `0x10` incorrectly for 5A — a dedicated `build_5a_frame` is required.
|
||
- **Event-time metadata source confirmed** — `Client:`, `User Name:`, and `Seis Loc:` strings are present in A5 frame 7 of the bulk waveform stream (SUB 5A), not in the 210-byte SUB 0C waveform record. They reflect the compliance setup as it was when the event was stored on the device.
|
||
|
||
---
|
||
|
||
## v0.5.0 — 2026-03-31
|
||
|
||
### Added
|
||
- **Console tab in `seismo_lab.py`** — direct device connection without the bridge subprocess.
|
||
- Serial and TCP transport selectable via radio buttons.
|
||
- Four one-click commands: POLL, Serial #, Full Config, Event Index.
|
||
- Colour-coded scrolling output: TX (blue), RX raw hex (teal), parsed/decoded (green), errors (red).
|
||
- Save Log and Send to Analyzer buttons; logs auto-saved to `bridges/captures/console_<ts>.log`.
|
||
- Queue/`after(100)` pattern — no UI blocking or performance impact.
|
||
- **`minimateplus` package** — clean Python client library for the MiniMate Plus S3 protocol.
|
||
- `SerialTransport` and `TcpTransport` (for Sierra Wireless RV50/RV55 cellular modems).
|
||
- `MiniMateProtocol` — DLE frame parser/builder, two-step paged reads, checksum validation.
|
||
- `MiniMateClient` — high-level client: `connect()`, `get_serial()`, `get_config()`, `get_events()`.
|
||
- **TCP/cellular transport** (`TcpTransport`) — connect to field units via Sierra Wireless RV50/RV55 modems over cellular.
|
||
- `read_until_idle(idle_gap=1.5s)` to handle modem data-forwarding buffer delay.
|
||
- Confirmed working end-to-end: TCP → RV50/RV55 → RS-232 → MiniMate Plus.
|
||
- **`bridges/tcp_serial_bridge.py`** — local TCP-to-serial bridge for bench testing `TcpTransport` without a cellular modem.
|
||
- **SFM REST server** (`sfm/server.py`) — FastAPI server with device info, event list, and event record endpoints over both serial and TCP.
|
||
|
||
### Fixed
|
||
- `protocol.py` `startup()` was using a hardcoded `POLL_RECV_TIMEOUT = 10.0` constant, ignoring the configurable `self._recv_timeout`. Fixed to use `self._recv_timeout` throughout.
|
||
- `sfm/server.py` now retries once on `ProtocolError` for TCP connections to handle cold-boot timing on first connect.
|
||
|
||
### Protocol / Documentation
|
||
- **Sierra Wireless RV50/RV55 modem config** — confirmed required ACEmanager settings: Quiet Mode = Enable, Data Forwarding Timeout = 1, TCP Connect Response Delay = 0. Quiet Mode disabled causes modem to inject `RING\r\nCONNECT\r\n` onto the serial line, breaking the S3 handshake.
|
||
- **Calibration year** confirmed at SUB FE (Full Config) destuffed payload offset 0x56–0x57 (uint16 BE). `0x07E7` = 2023, `0x07E9` = 2025.
|
||
- **`"Operating System"` boot string** — 16-byte UART boot message captured on cold-start before unit enters DLE-framed mode. Parser handles correctly by scanning for DLE+STX.
|
||
- RV50/RV55 sends `RING`/`CONNECT` over TCP to the calling client even with Quiet Mode enabled — this is normal behaviour, parser discards it.
|
||
|
||
---
|
||
|
||
## v0.4.0 — 2026-03-12
|
||
|
||
### Added
|
||
- **`seismo_lab.py`** — combined Bridge + Analyzer GUI. Single window with two tabs; bridge start auto-wires live mode in the Analyzer.
|
||
- **`frame_db.py`** — SQLite frame database. Captures accumulate over time; Query DB tab searches across all sessions.
|
||
- **`bridges/s3-bridge/proxy.py`** — bridge proxy module.
|
||
- Large BW→S3 write frame checksum algorithm confirmed and implemented (`SUM8` of payload `[2:-1]` skipping `0x10` bytes, plus constant `0x10`, mod 256).
|
||
- SUB `A4` identified as composite container frame with embedded inner frames; `_extract_a4_inner_frames()` and `_diff_a4_payloads()` reduce diff noise from 2300 → 17 meaningful entries.
|
||
|
||
### Fixed
|
||
- BAD CHK false positives on BW POLL frames — BW frame terminator `03 41` was being included in the de-stuffed payload. Fixed to strip correctly.
|
||
- Aux Trigger read location confirmed at SUB FE offset `0x0109`.
|
||
|
||
---
|
||
|
||
## v0.3.0 — 2026-03-09
|
||
|
||
### Added
|
||
- Record time confirmed at SUB E5 page2 offset `+0x28` as float32 BE.
|
||
- Trigger Sample Width confirmed at BW→S3 write frame SUB `0x82`, destuffed payload offset `[22]`.
|
||
- Mode-gating documented: several settings only appear on the wire when the appropriate mode is active.
|
||
|
||
### Fixed
|
||
- `0x082A` mystery resolved — fixed-size E5 payload length (2090 bytes), not a record-time field.
|
||
|
||
---
|
||
|
||
## v0.2.0 — 2026-03-01
|
||
|
||
### Added
|
||
- Channel config float layout fully confirmed: trigger level, alarm level, and unit string per channel (IEEE 754 BE floats).
|
||
- Blastware `.set` file format decoded — little-endian binary struct mirroring the wire payload.
|
||
- Operator manual (716U0101 Rev 15) added as cross-reference source.
|
||
|
||
---
|
||
|
||
## v0.1.0 — 2026-02-26
|
||
|
||
### Added
|
||
- Initial `s3_bridge.py` serial bridge — transparent RS-232 tap between Blastware and MiniMate Plus.
|
||
- `s3_parser.py` — deterministic DLE state machine frame extractor.
|
||
- `s3_analyzer.py` — session parser, frame differ, Claude export.
|
||
- `gui_bridge.py` and `gui_analyzer.py` — Tkinter GUIs.
|
||
- DLE framing confirmed: `DLE+STX` / `DLE+ETX`, `0x41` = ACK (not STX), DLE stuffing rule.
|
||
- Response SUB rule confirmed: `response_SUB = 0xFF - request_SUB`.
|
||
- Year `0x07CB` = 1995 confirmed as MiniMate factory RTC default.
|
||
- Full write command family documented (SUBs `68`–`83`).
|