seismo-relay/CHANGELOG.md

# Changelog

All notable changes to seismo-relay are documented here.

---

## v0.11.0 — 2026-04-13

### Added

- **`sfm/database.py` — SeismoDb** — SQLite persistence layer for all ACH data.
  Three tables, all unit-keyed by serial number:
  - `ach_sessions` — one row per inbound call-home: serial, timestamp, peer IP,
    events_downloaded, monitor_entries, duration_seconds
  - `events` — one row per triggered waveform event: serial, waveform_key (dedup),
    timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location
    strings, sample_rate, record_type, false_trigger flag
  - `monitor_log` — one row per monitoring interval: serial, waveform_key (dedup),
    start_time, stop_time, duration_seconds, geo_threshold_ips
  - WAL mode, per-request connections — safe for the single-writer / occasional-reader
    ACH server pattern
  - Deduplication by `(serial, waveform_key)` UNIQUE constraint — re-runs and repeat
    call-homes never produce duplicate rows

- **`ach_server.py` — DB integration** — after each successful call-home, writes new
  events and monitor log entries to `seismo_relay.db` then records the session in
  `ach_sessions`.  DB write failures are logged as warnings and do not abort the session.

- **`sfm/server.py` — DB read endpoints**:
  - `GET /db/units` — distinct serials with last_seen, total_events, total_monitor_entries
  - `GET /db/events` — query events with serial / date range / false_trigger filters
  - `GET /db/monitor_log` — query monitoring intervals
  - `GET /db/sessions` — query ACH call-home sessions
  - `PATCH /db/events/{id}/false_trigger` — flag/unflag false triggers (for review UI)

### Architecture

- seismo-relay DB is unit-keyed only — no project concepts.  Project aggregation is
  terra-view's responsibility via `UnitAssignment` / `DeploymentRecord` + date range
  queries against the SFM DB endpoints.
- DB file lives at `bridges/captures/seismo_relay.db` by default.

---

## v0.10.0 — 2026-04-11

### Added

- **`MiniMateClient.get_monitor_log_entries(skip_keys=None)`** — browse-mode walk
  (`1E → 0A → 1F`) that collects partial records (`0x2C` record type) from the device's
  event list without triggering a full waveform download (no 0C or 5A).  Returns
  `list[MonitorLogEntry]`.  Each entry represents one continuous monitoring interval where
  no threshold was exceeded.

- **`_decode_0a_partial_header(raw_data, index, key4)`** in `client.py` — decodes a SUB
  0x0A response payload whose record type is `0x2C`.  Extracts:
  - `start_time` / `stop_time` — two consecutive timestamps; auto-detects 9-byte
    (sub_code=0x10, single-shot) vs 10-byte (sub_code=0x03, continuous) format from
    `raw_data[11]`.  Handles a 1-byte gap between the two timestamps that occurs when
    ts1 and ts2 share the same minute:second.
  - `serial` — device serial string found via `b"BE"` anchor scan.
  - `geo_threshold_ips` — trigger level found via `b"Geo: "` anchor scan.

- **`MonitorLogEntry` dataclass** in `models.py` — new model for partial records:
  `index`, `key`, `start_time`, `stop_time`, `serial`, `geo_threshold_ips`,
  `raw_header`, and a `duration_seconds` property.

- **`read_waveform_header()` return value extended** — now returns `(data_rsp.data, length)`
  (full payload) instead of `(data_rsp.data[11:11+length], length)`.  Callers get the
  complete payload including the record-type byte at position 0.  Full records use
  `raw_data[11:11+length]` as before; partial records are detected by `raw_data[0] == 0x2C`.

- **ACH server: monitor log collection** — after `get_events()`, calls
  `get_monitor_log_entries(skip_keys=seen_keys)` and saves new entries to
  `monitor_log.json` in the session directory.  Monitor log keys are included in
  `downloaded_keys` for state persistence (no re-processing on next call-home).

- **`_monitor_log_entry_to_dict()`** in `ach_server.py` — serialises a `MonitorLogEntry`
  to a JSON-compatible dict with ISO-format timestamps.

### Protocol / Documentation

- **SUB 0x0A partial record (0x2C) format confirmed** (✅ 4-11-26 MITM capture, 12 frames):
  - Record type `0x2C` at `raw_data[0]`; length < 64 bytes.
  - Two timestamps at `raw_data[11:]` — start and stop of the monitoring interval.
  - ASCII metadata region after timestamps: `BE<serial>\x00Geo: <float> in/s`.
  - Edge case: 1-byte separator between timestamps when ts1 and ts2 share minute:second.
  - 10-byte timestamp format (sub_code=0x03) signalled by `raw_data[11] == 0x10`.

- **Key reuse detection for monitor log entries** — monitor log keys are tracked alongside
  event keys in `ach_state.json` so the ACH server does not re-process them after a
  call-home cycle.

---

## v0.9.0 — 2026-04-11

### Added

- **`MiniMateClient.list_event_keys()`** — fast browse-mode walk (1E → 0A → 1F, no waveform
  download) that returns the list of event key hex strings currently stored on the device.
  Used by the ACH server as a cheap pre-check before deciding whether to call `get_events()`.

- **`get_events(skip_waveform_for_keys=set(...))`** — new optional parameter.  For any key in
  the set the function performs only 0A + 1F(browse) instead of the full
  1E-arm → 0C → POLL×3 → 5A sequence.  Eliminates redundant waveform downloads on repeat
  call-homes when the device still holds previously downloaded events.

- **`MiniMateClient.delete_all_events()`** — erases all events from device memory using the
  confirmed 4-step sequence:
  - SUB 0xA3 `begin_erase_all` — initiate erase (token=0xFE) → ack 0x5C
  - SUB 0x1C `read_monitor_status` — intermediate status read (Blastware-required)
  - SUB 0x06 `read_event_storage_range` — verify storage state (token=0xFE) → 36-byte response
  - SUB 0xA2 `confirm_erase_all` — commit erase (token=0xFE) → ack 0x5D

  All four steps confirmed from 4-11-26 MITM capture of a live Blastware ACH session.
  After a successful call, the device's event counter resets to `0x01110000`.

- **`MiniMateProtocol` erase methods**: `begin_erase_all()`, `confirm_erase_all()`,
  `read_event_storage_range()` added to `protocol.py` with documented SUB constants
  `SUB_ERASE_ALL_BEGIN = 0xA3` and `SUB_ERASE_ALL_CONFIRM = 0xA2`.

- **`bridges/ach_mitm.py`** — transparent TCP-to-TCP MITM proxy.  Listens for inbound unit
  connections, connects upstream to a real Blastware ACH server, and saves both directions
  to `raw_bw_<ts>.bin` / `raw_s3_<ts>.bin` files matching the existing capture format.
  Used to capture the 4-11-26 Blastware ACH session including event deletion.
  Usage: `python bridges/ach_mitm.py --bw-host 127.0.0.1 --bw-port 9999 --listen-port 9998`

- **ACH server: key-based state tracking** — `ach_state.json` now stores
  `downloaded_keys: [hex_strings]` and `max_downloaded_key: hex_string` per unit instead of
  `event_count: N`.  This correctly handles the standard workflow where events are deleted
  from the device after upload — a count-based approach would see `count=0` on the next
  call-home and silently skip new events.

- **ACH server: `--clear-after-download` flag** — after a successful download (at least one
  new event saved), erases all events from the device using `delete_all_events()`.  Mirrors
  the standard Blastware ACH workflow.  On success, `downloaded_keys` and
  `max_downloaded_key` are reset to empty so the next session starts fresh.

- **ACH server: post-erase key-reuse detection** — after an external erase (Blastware or
  manual), device keys restart from `0x01110000`, colliding with previously downloaded keys.
  On each browse walk, if `max(device_keys) < max_downloaded_key` (device counter rolled
  back), all device keys are treated as new regardless of `seen_keys`.  This also catches
  erases performed by Blastware between our sessions.

### Protocol / Documentation

- **SUB 0xA3 / SUB 0xA2 — erase-all sequence confirmed** (✅ 4-11-26 MITM capture):
  Both frames use `token=0xFE` at `params[7]` and are standard `build_bw_frame` requests
  (not write-format).  Response SUBs follow the standard formula: 0x5C and 0x5D.
  The intermediate 0x1C + 0x06 reads between them are required by Blastware.

- **SUB 0x06 — event storage range read confirmed** (✅ 4-11-26 MITM capture):
  Two-step read, data offset = 0x24 (36 bytes).  The last 8 bytes of the response contain
  the first and last stored event keys (4 bytes each).  After a successful erase, both keys
  read as `01110000` (device-empty state).

- **Event key counter resets to `0x01110000` after erase** — confirmed by observing key
  `01110000` on the device immediately after the MITM erase session.

---

## v0.8.0 — 2026-04-07

### Added

- **Write pipeline end-to-end** — `push_config_raw(event_index_data, compliance_data,
  trigger_data, waveform_data)` on `MiniMateClient` orchestrates the full
  `68→73 | 71×3→72 | 82→83 | 69→74→72` write sequence.

- **`build_bw_write_frame(sub, data, *, offset, params)`** in `framing.py` — dedicated frame
  builder for write commands (SUBs 0x68–0x83).  Doubles only the BW_CMD byte; all other
  bytes including offset, params, data, and checksum are written raw.  Uses the large-frame
  DLE-aware checksum (`sum(b for b in payload[2:] if b != 0x10) + 0x10) & 0xFF`).

- **`MiniMateProtocol` write methods** — `write_event_index()`, `write_compliance()`,
  `write_trigger_config()`, `write_waveform_data()`, `write_confirm()`,
  `start_monitoring()`, `stop_monitoring()`.

- **`AchSession` inbound server** (`bridges/ach_server.py`) — accepts call-home TCP
  connections, runs the full handshake + device-info + event-download sequence, saves
  `device_info.json` + `events.json` per session.

### Protocol / Documentation

- **Write frame format confirmed** (✅ 3-11-26 BW TX capture, all 11 frames): only BW_CMD
  byte `0x10` is doubled; all other bytes sent raw.  Standard `build_bw_frame` DLE-stuffing
  is incorrect for write commands.
- **Write ack responses** confirmed as 17-byte zero-data S3 frames.
- **Monitoring SUBs 0x96/0x97** confirmed from 4-8-26 capture.
- **SESSION_RESET signal** (`41 03`) required before POLL for monitoring units.
- **SUB 0x1C monitoring flag** at `section[1]`: `0x00` = idle, `0x10` = monitoring.
  Confirmed by byte-diff of all 144 data frames in 4-8-26/2ndtry capture.

---

## v0.7.0 — 2026-04-03

### Added
- **Raw ADC waveform decode — `_decode_a5_waveform(frames_data, event)`** in `client.py`.
  Parses the complete set of SUB 5A A5 response frames into per-channel time-series:
  - Reads the STRT record from A5[0] (bytes 7+): extracts `total_samples` (BE uint16 at +8),
    `pretrig_samples` (BE uint16 at +16), and `rectime_seconds` (uint8 at +18) into
    `event.total_samples / pretrig_samples / rectime_seconds`.
  - Skips the 6-byte preamble (`00 00 ff ff ff ff`) that follows the 21-byte STRT header;
    waveform data begins at `strt_pos + 27`.
  - Strips the 8-byte per-frame counter header from A5[1–6, 8] before appending waveform bytes.
  - Skips A5[7] (metadata-only) and A5[9] (terminator).
  - **Cross-frame alignment correction**: accumulates `running_offset % 8` across all frames
    and discards `(8 − align) % 8` leading bytes per frame to re-align to a T/V/L/M boundary.
    Required because individual frame waveform payloads are not always multiples of 8 bytes.
  - Decodes as 4-channel interleaved signed 16-bit LE at 8 bytes per sample-set:
    bytes 0–1 = Tran, 2–3 = Vert, 4–5 = Long, 6–7 = Mic.
  - Stores result in `event.raw_samples = {"Tran": [...], "Vert": [...], "Long": [...], "Mic": [...]}`.
- **`download_waveform(event)` public method** on `MiniMateClient`.
  Issues a full SUB 5A stream with `stop_after_metadata=False`, then calls
  `_decode_a5_waveform()` to populate `event.raw_samples` and `event.total_samples /
  pretrig_samples / rectime_seconds`. Previously only metadata frames were fetched during
  `get_events()`; raw waveform data is now available on demand.
- **`Event` model new fields** (`models.py`): `total_samples`, `pretrig_samples`,
  `rectime_seconds` (from STRT record), and `_waveform_key` (4-byte key stored during
  `get_events()` for later use by `download_waveform()`).

### Protocol / Documentation
- **SUB 5A A5[0] STRT record layout confirmed** (✅ 2026-04-03, 4-2-26 blast capture):
  - STRT header is 21 bytes: `b"STRT"` + length fields + `total_samples` (BE uint16 at +8) +
    `pretrig_samples` (BE uint16 at +16) + `rectime_seconds` (uint8 at +18).
  - Followed by 6-byte preamble: `00 00 ff ff ff ff`. Waveform begins at `strt_pos + 27`.
  - Confirmed: 4-2-26 blast → `total_samples=9306`, `pretrig_samples=298`, `rectime_seconds=70`.
- **Blast/waveform mode A5 format confirmed** (✅ 2026-04-03, 4-2-26 blast capture):
  4-channel interleaved int16 LE at 8 bytes per sample-set; cross-frame alignment correction
  required. 948 of 9306 total sample-sets captured via `stop_after_metadata=True` (10 frames).
- **Noise/histogram mode A5 format — endianness corrected** (✅ 2026-04-03, 3-31-26 capture):
  32-byte block samples are signed 16-bit **little-endian** (previously documented as BE).
  `0a 00` → LE int16 = 10 (correct noise floor); BE would give 2560 (wrong).
- Protocol reference §7.6 rewritten — split into §7.6.1 (Blast/Waveform mode) and §7.6.2
  (Noise/Histogram mode), each with confirmed field layouts and open questions noted.

---

## v0.6.0 — 2026-04-02

### Added
- **True event-time metadata via SUB 5A bulk waveform stream** — `get_events()` now issues a SUB 5A request after each SUB 0C download, reads the A5 response frames, and extracts the `Client:`, `User Name:`, and `Seis Loc:` fields as they existed at the moment the event was recorded. Previously these fields were backfilled from the current compliance config (SUB 1A), which reflects today's setup, not the setup active when the event triggered.
  - `build_5a_frame(offset_word, raw_params)` in `framing.py` — reproduces Blastware's exact wire format for SUB 5A requests: raw (non-DLE-stuffed) `offset_hi`, DLE-stuffed params, and a DLE-aware checksum where `10 XX` pairs count only `XX`.
  - `bulk_waveform_params()` returns 11 bytes (extra trailing `0x00` confirmed from 1-2-26 BW wire capture).
  - `read_bulk_waveform_stream(key4, *, stop_after_metadata=True, max_chunks=32)` in `protocol.py` — loops sending chunk requests (counter increments `0x0400` per chunk), stops early when `b"Project:"` is found, then sends a termination frame.
  - `_decode_a5_metadata_into(frames_data, event)` in `client.py` — needle-searches A5 frame data for `Project:`, `Client:`, `User Name:`, `Seis Loc:`, `Extended Notes` and overwrites `event.project_info`.
- **`get_events()` sequence extended** — now `1E → 0A → 0C → 5A → 1F` per event.

### Fixed
- **Compliance config (SUB 1A) channel block missing** — orphaned `self._send(build_bw_frame(SUB_COMPLIANCE, 0x2A, _DATA_PARAMS))` before the B/C/D receive loop had no corresponding `recv_one()`, shifting all subsequent receives one step behind and leaving frame D's channel-block data (trigger_level_geo, alarm_level_geo, max_range_geo) unread. Removed the orphaned send. Total config bytes received now correctly ~2126 (was ~1071).
- **Compliance config anchor search range** — `_decode_compliance_config_into()` searched `cfg[40:100]` for the sample-rate/record-time anchor. With the orphaned-send bug fixed the 44-byte padding it had been adding is gone, and the anchor now appears at `cfg[11]`. Search widened to `cfg[0:150]` to be robust to future layout shifts.
- Removed byte-content deduplication from `read_compliance_config()` — was masking the real receive-ordering bug.

### Protocol / Documentation
- **SUB 5A frame format confirmed** — `offset_hi` byte (`0x10`) must be sent raw (not DLE-stuffed); checksum is DLE-aware (only the second byte of a `10 XX` pair is summed). Standard `build_bw_frame` DLE-stuffs `0x10` incorrectly for 5A — a dedicated `build_5a_frame` is required.
- **Event-time metadata source confirmed** — `Client:`, `User Name:`, and `Seis Loc:` strings are present in A5 frame 7 of the bulk waveform stream (SUB 5A), not in the 210-byte SUB 0C waveform record. They reflect the compliance setup as it was when the event was stored on the device.

---

## v0.5.0 — 2026-03-31

### Added
- **Console tab in `seismo_lab.py`** — direct device connection without the bridge subprocess.
  - Serial and TCP transport selectable via radio buttons.
  - Four one-click commands: POLL, Serial #, Full Config, Event Index.
  - Colour-coded scrolling output: TX (blue), RX raw hex (teal), parsed/decoded (green), errors (red).
  - Save Log and Send to Analyzer buttons; logs auto-saved to `bridges/captures/console_<ts>.log`.
  - Queue/`after(100)` pattern — no UI blocking or performance impact.
- **`minimateplus` package** — clean Python client library for the MiniMate Plus S3 protocol.
  - `SerialTransport` and `TcpTransport` (for Sierra Wireless RV50/RV55 cellular modems).
  - `MiniMateProtocol` — DLE frame parser/builder, two-step paged reads, checksum validation.
  - `MiniMateClient` — high-level client: `connect()`, `get_serial()`, `get_config()`, `get_events()`.
- **TCP/cellular transport** (`TcpTransport`) — connect to field units via Sierra Wireless RV50/RV55 modems over cellular.
  - `read_until_idle(idle_gap=1.5s)` to handle modem data-forwarding buffer delay.
  - Confirmed working end-to-end: TCP → RV50/RV55 → RS-232 → MiniMate Plus.
- **`bridges/tcp_serial_bridge.py`** — local TCP-to-serial bridge for bench testing `TcpTransport` without a cellular modem.
- **SFM REST server** (`sfm/server.py`) — FastAPI server with device info, event list, and event record endpoints over both serial and TCP.

### Fixed
- `protocol.py` `startup()` was using a hardcoded `POLL_RECV_TIMEOUT = 10.0` constant, ignoring the configurable `self._recv_timeout`. Fixed to use `self._recv_timeout` throughout.
- `sfm/server.py` now retries once on `ProtocolError` for TCP connections to handle cold-boot timing on first connect.

### Protocol / Documentation
- **Sierra Wireless RV50/RV55 modem config** — confirmed required ACEmanager settings: Quiet Mode = Enable, Data Forwarding Timeout = 1, TCP Connect Response Delay = 0. Quiet Mode disabled causes modem to inject `RING\r\nCONNECT\r\n` onto the serial line, breaking the S3 handshake.
- **Calibration year** confirmed at SUB FE (Full Config) destuffed payload offset 0x56–0x57 (uint16 BE). `0x07E7` = 2023, `0x07E9` = 2025.
- **`"Operating System"` boot string** — 16-byte UART boot message captured on cold-start before unit enters DLE-framed mode. Parser handles correctly by scanning for DLE+STX.
- RV50/RV55 sends `RING`/`CONNECT` over TCP to the calling client even with Quiet Mode enabled — this is normal behaviour, parser discards it.

---

## v0.4.0 — 2026-03-12

### Added
- **`seismo_lab.py`** — combined Bridge + Analyzer GUI. Single window with two tabs; bridge start auto-wires live mode in the Analyzer.
- **`frame_db.py`** — SQLite frame database. Captures accumulate over time; Query DB tab searches across all sessions.
- **`bridges/s3-bridge/proxy.py`** — bridge proxy module.
- Large BW→S3 write frame checksum algorithm confirmed and implemented (`SUM8` of payload `[2:-1]` skipping `0x10` bytes, plus constant `0x10`, mod 256).
- SUB `A4` identified as composite container frame with embedded inner frames; `_extract_a4_inner_frames()` and `_diff_a4_payloads()` reduce diff noise from 2300 → 17 meaningful entries.

### Fixed
- BAD CHK false positives on BW POLL frames — BW frame terminator `03 41` was being included in the de-stuffed payload. Fixed to strip correctly.
- Aux Trigger read location confirmed at SUB FE offset `0x0109`.

---

## v0.3.0 — 2026-03-09

### Added
- Record time confirmed at SUB E5 page2 offset `+0x28` as float32 BE.
- Trigger Sample Width confirmed at BW→S3 write frame SUB `0x82`, destuffed payload offset `[22]`.
- Mode-gating documented: several settings only appear on the wire when the appropriate mode is active.

### Fixed
- `0x082A` mystery resolved — fixed-size E5 payload length (2090 bytes), not a record-time field.

---

## v0.2.0 — 2026-03-01

### Added
- Channel config float layout fully confirmed: trigger level, alarm level, and unit string per channel (IEEE 754 BE floats).
- Blastware `.set` file format decoded — little-endian binary struct mirroring the wire payload.
- Operator manual (716U0101 Rev 15) added as cross-reference source.

---

## v0.1.0 — 2026-02-26

### Added
- Initial `s3_bridge.py` serial bridge — transparent RS-232 tap between Blastware and MiniMate Plus.
- `s3_parser.py` — deterministic DLE state machine frame extractor.
- `s3_analyzer.py` — session parser, frame differ, Claude export.
- `gui_bridge.py` and `gui_analyzer.py` — Tkinter GUIs.
- DLE framing confirmed: `DLE+STX` / `DLE+ETX`, `0x41` = ACK (not STX), DLE stuffing rule.
- Response SUB rule confirmed: `response_SUB = 0xFF - request_SUB`.
- Year `0x07CB` = 1995 confirmed as MiniMate factory RTC default.
- Full write command family documented (SUBs `68`–`83`).