seismo-relay/CHANGELOG.md

# Changelog

All notable changes to seismo-relay are documented here.

---

## v0.15.0 — 2026-05-07

### Added

- **Layered event storage architecture.**  Each event now lands as four
  files in the per-serial waveform store, each with a clear role:

  - `<filename>` — the Blastware-readable binary (BW file).  Untouched.
  - `<filename>.a5.pkl` — the raw 5A frames (regenerative source).
  - `<filename>.h5` — clean per-channel waveform arrays in physical
    units (in/s for geo, psi for mic) plus event metadata (HDF5 with
    gzip compression).  This is the canonical format for downstream
    analysis tools.
  - `<filename>.sfm.json` — the modern review/metadata sidecar (peaks,
    project, source provenance, review state, extensions).

  SQLite (`seismo_relay.db`) is the searchable index over all four.

- **Plot-ready waveform JSON (`sfm.plot.v1`).**  The `/device/event/{idx}/waveform`
  and `/db/events/{id}/waveform.json` endpoints now return samples in
  physical units with explicit time-axis metadata, peak markers, and
  per-channel unit hints — no more guessing the ADC-to-velocity scale
  client-side.  The webapp waveform viewer was rewritten to consume
  this shape.

- **In-app waveform viewer accuracy fix.**  The standalone SFM webapp
  viewer was scaling geophone amplitudes by `geoAdcScale / 32767`
  (≈ 6.206 / 32767), where `geoAdcScale = 6.206053` is the device's
  *in/s per V* hardware constant — not the ADC-counts-to-velocity
  factor.  This silently scaled every plot ~38% too low for Normal-range
  geophones (the correct full-scale is 10.0 in/s, or 1.25 in/s for
  Sensitive).  Conversion is now done server-side using the geo_range
  from compliance config; the client just plots.

- New `sfm/event_hdf5.py` module: `write_event_hdf5()`,
  `read_event_hdf5()`, plus a plot-JSON helper.
- Backfill script extended to also emit `.h5` for existing events.

### Dependencies

- Added `h5py>=3.10` and `numpy>=1.24` for the HDF5 storage layer.
- Added `python-multipart>=0.0.7` (required by FastAPI for the
  `/db/import/blastware_file` endpoint introduced in this release).

---

## v0.14.3 — 2026-05-05

### Fixed

- **`build_5a_frame` — DLE-stuffing rule for 0x10 bytes in params (the
  long-standing >1-sec event 0 "won't open in BW" bug).**

  Previously `build_5a_frame` wrote params bytes RAW with no DLE stuffing,
  based on the incorrect assumption that the device handled all `0x10`
  bytes in params literally.  It does not.  The device's actual de-stuffing
  rule for the params region is:

    - `10 10`             → de-stuffs to `10`
    - `10 02/03/04`       → kept literal (inner-frame markers)
    - `10 X` for other X  → de-stuffs to just `X` (drops the `0x10`)

  When the counter passed in params has `0x10` in the high byte (e.g.
  counter=`0x1000` produces params bytes `... 10 00 ...`), the device
  silently corrupts the request to counter=`0x__00` and responds with
  whatever lives at that wrong address.  For counter=0x1000 the wrong
  address was 0x0000, so the response was a copy of the file header +
  STRT record.  That STRT block then got embedded in the assembled body
  at file offset `0x1016`, and Blastware refused to open the file
  (interprets the second STRT as a malformed multi-event file).

  This explains the entire >1-sec event-0 failure pattern:

    - 1-sec events have `end_offset < 0x1000`, so the chunk walk never
      requests counter `0x10__` and the bug never triggers.
    - 2-sec / 3-sec / longer events all need a chunk at counter `0x1000`
      (and longer events also need `0x1200`, `0x1400`, etc., none of which
      have `0x10` in the high byte except `0x1000`).  Just one corrupted
      response is enough to embed STRT in the body and break the file.

  Verified against BW 5-1-26 "copy 3sec" capture: all 17 5A request
  frames (probe + 2 metadata pages + 13 sample chunks + TERM) now match
  BW's wire output **byte-for-byte**, including the doubled `10 10 00`
  for counter=0x1000.

### Notes

- `0x10` bytes in `offset_hi` (the standalone offset field at body[5])
  are still written RAW — confirmed correct per the 1-2-26 capture.
- BW's actual encoding of `10 02` / `10 04` for meta pages 0x1002 /
  0x1004 is *not* doubled — it relies on the device keeping `10 02`
  and `10 04` as literal pairs.  This is preserved by the fix.

---

## v0.14.2 — 2026-05-04

### Fixed

- **`blastware_file.py` — removed harmful "duplicate header+STRT" strip.**
  The v0.13.x strip logic was matching the byte sequence `00 12 03 00 STRT`
  in legitimate waveform data — sample chunks at counter `0x1000` and
  beyond often contain those bytes coincidentally — and zeroing 25 bytes
  of valid samples per match.  This is why event 0 (event-1 case in the
  protocol) downloads of >1-sec recordings always failed in BW: the strip
  destroyed real data at body offset `0x1012..0x102B` and propagated
  alignment differences through the rest of the body.  Sub-1-sec events
  worked because their `end_offset` was below `0x1002`, so no sample
  chunks landed in the metadata-page region and the strip's needle never
  matched.  Verified fix by re-feeding the BW 5-1-26 "copy 3sec" capture's
  A5 frames into the file builder: output is now byte-identical to BW's
  saved `M529LKIQ.G10` reference (8708 bytes, 0 differences).
- BW already concatenates frame contributions in stream order without
  any de-duplication; SFM now does the same.

---

## v0.14.0 — 2026-05-02

### Changed (major rewrite)

- **`read_bulk_waveform_stream` — STRT-bounded chunk walk.**  Replaces the
  earlier `0x0400`-step / `max(key4[2:4], 0x0400)` chunk-counter formula,
  which over-read ~5× past the actual event end into post-event circular-
  buffer garbage.  The new walk:

    1. Probe at `counter = start_offset` (event 1: `0x0000`; event N:
       `cur_key[2:4]`).
    2. Parse `end_offset` from the STRT record at `data[17]` of the probe
       response (`end_key[2:4]` field).
    3. For event 1 only, read the two fixed metadata pages at counter
       `0x1002` and `0x1004` — these contain the global session-start
       compliance setup (Project / Client / User Name / Seis Loc /
       Extended Notes ASCII strings).  Continuation events skip these
       (BW caches them across the session).
    4. Walk sample chunks at **`0x0200` increments (NOT `0x0400`)**, bounded
       by `end_offset` — the loop exits when
       `next_chunk_counter + 0x0200 > end_offset`.
    5. Send the proper TERM frame (see new `bulk_waveform_term_v2()`) with
       `offset_word = end_offset - next_boundary` and
       `params[2:4] = next_boundary BE`.  The TERM response carries the
       partial last chunk + 26-byte file footer.

- **New helpers:** `bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`
  and `parse_strt_end_offset(a5_data)` in `minimateplus.framing`.

- **`stop_after_metadata` / `extra_chunks_after_metadata` kwargs are now
  no-ops** under the v0.14.x walk.  They are retained on the
  `read_bulk_waveform_stream` signature for backward compatibility but log a
  DEBUG line when set.  The old "scan for `b'Project:'` and stop one chunk
  later" workaround is obsolete — the loop is deterministically bounded by
  the STRT-derived `end_offset`.

- **Project / Client / User Name / Seis Loc string source corrected.**
  These come from the dedicated metadata pages at counter `0x1002` /
  `0x1004`, not from "A5 frame 7" of the sample-chunk stream.  The
  earlier "A5 frame 7" claim was an artifact of the broken `0x0400`-step
  walk where the bad counter formula coincidentally landed sample-chunk
  fi=7 on top of the 0x1002 metadata page.

### Verified

- Three independent BW MITM captures (4-27-26 + 5-1-26 + 5-4-26) confirm
  the new walk matches BW's behaviour event-for-event.
- `end_offset` values verified across 3 events: `0x1ABE` (4-27-26 2-sec),
  `0x21F2` (5-1-26 3-sec), `0x417E` (5-1-26 event-2).

### Notes

- Earlier v0.13.0 / v0.13.1 / v0.13.2 entries describe partial steps along
  the way (some of the file builder fixes, filename bugs, etc.) that were
  superseded by the full rewrite.  Treat this v0.14.0 entry as the
  definitive landing point for the corrected SUB 5A protocol.

---

## v0.14.1 — 2026-05-04

### Fixed

- **`read_bulk_waveform_stream` — event-N probe counter off-by-`0x46`.**
  Continuation events (start_key[2:4] != 0) were being probed at counter
  `start_offset + 0x0046` instead of just `start_offset`.  In the iteration
  walk, `cur_key` from 1F is already the off=0x46 WAVEHDR record key, so the
  earlier formula effectively double-counted the WAVEHDR offset.  The probe
  landed one WAVEHDR past the actual event start, the response no longer
  contained the STRT record at byte 17, `parse_strt_end_offset` returned
  `None`, and the chunk loop fell back to the `max_chunks=128` cap — walking
  ~110 chunks of post-event circular-buffer garbage.  Verified against the
  5-1-26 "copy 2nd address" and 5-4-26 BW 2-sec event captures: BW probes
  counter=`0x2238` with key=`01112238` and STRT is present at byte 17 of
  the response (end_offset=`0x417E`).
- **CLAUDE.md / docs/instantel_protocol_reference.md** — corrected the
  event-N section to clarify that `start_key` in those formulas is the
  off=0x46 key, not the off=0x2C boundary key, and removed the spurious
  `+0x46` from the chunk-walk pseudocode.

---

## v0.13.2 — 2026-05-01

### Fixed

- **`_extract_record_type` — third 0C-record header format ("short", 8 bytes).**
  A live SFM download against BE11529 produced files named `M5290000.000`
  (zero-stamped) because the 0C waveform record's first bytes were
  `01 05 07 ea ...` — neither the 9-byte single-shot layout (`0x10` at byte 1)
  nor the 10-byte continuous layout (`0x10` at bytes 0 and 2).  Investigation
  showed this is a third format observed in the wild: an 8-byte header with no
  marker bytes at all (`[day][month][year_BE:2][unknown][hour][min][sec]`).
  The detection logic now scans the year (uint16 BE) at byte 2 / byte 3 / byte
  4 and picks whichever offset returns a sensible year (2015–2050) — each
  format has the year at a unique position so this disambiguates cleanly.
  - New format → `event.record_type = "Waveform (Short)"`,
    `Timestamp.from_short_record()`.
  - Existing single-shot and continuous parsers unchanged.
  - The user's event from May 1, 2026 13:21:37 now correctly resolves to a
    filename like `M529LKIQ.G10` instead of `M5290000.000`.

### Added

- `Timestamp.from_short_record(data)` — decodes the 8-byte header.
- `_detect_record_format(data)` — internal helper returning
  `"single_shot" / "continuous" / "short" / None` via year-position scan.

---

## v0.13.1 — 2026-05-01

### Fixed

- **`_extract_record_type` — Continuous-mode record headers misclassified as Unknown.**
  In single-shot mode the 0C waveform record's 9-byte header puts the sub_code
  marker `0x10` at byte 1, with the day at byte 0.  In Continuous mode the
  header is 10 bytes with the marker at byte 0 *and* byte 2, and the day at
  byte 1.  Previous logic only inspected byte 1 and treated any value other
  than `0x10` / `0x03` as `"Unknown"`, which prevented `event.timestamp` from
  being populated for any continuous-mode event whose day-of-month wasn't
  exactly 3 or 16.  As a downstream effect, `blastware_filename()` saw
  `event.timestamp == None`, fell back to `stem="0000"` / `ab="00"`, and
  produced filenames like `M5290000.000`.  Discovered from a live SFM run on
  BE11529 in continuous mode (day-of-month = 5).
  Now disambiguates by checking BOTH byte 0 and byte 2: if both are `0x10`,
  it's the 10-byte continuous header; else if byte 1 is `0x10`, it's the
  9-byte single-shot header.  Day-of-month no longer matters.

  *Superseded by v0.13.2 — the user's actual record uses a third 8-byte format
  with no `0x10` markers, which v0.13.1 still misclassified.*

---

## v0.13.0 — 2026-05-01

### Fixed

- **SUB 5A bulk waveform stream — over-read bug for events ≥ 2 sec.**
  `read_bulk_waveform_stream` was walking the chunk counter past the actual
  end of the event, picking up post-event circular-buffer garbage that
  corrupted reconstructed Blastware files for any waveform > ~1 sec.  The
  loop now extracts the event's `end_offset` from the STRT record at
  `data[23:27]` of the probe response and stops the chunk walk when the next
  counter would step past it.  Verified against three BW MITM captures
  (4-27-26 + 5-1-26): 2-sec event drops from 37 over-read chunks to 7
  bounded chunks; 3-sec drops to 9; non-zero-start "event 2" drops to 9.

### Added

- `framing.bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)` —
  computes the corrected SUB 5A TERM frame's `(offset_word, params)` per the
  formula confirmed across all 3 BW captures.  Not yet wired into
  `read_bulk_waveform_stream` (the legacy TERM is still used to preserve the
  existing `blastware_file.write_blastware_file` frame-structure expectations);
  available for the next iteration that switches to BW's 0x0200 chunk step.
- `framing.parse_strt_end_offset(a5_data)` — extracts the event-end pointer
  from the STRT record in an A5 response payload.

### Documentation

- **CLAUDE.md and `docs/instantel_protocol_reference.md` extensively
  rewritten** to reflect the corrected SUB 5A protocol.  See:
  - CLAUDE.md "SUB 5A — chunk counter formula (REWRITTEN 2026-05-01)"
  - CLAUDE.md "SUB 5A — STRT record encodes end_offset"
  - CLAUDE.md "SUB 5A — TERM frame formula"
  - CLAUDE.md "SUB 5A — fixed metadata pages 0x1002 and 0x1004"
  - CLAUDE.md "SUB 0A — WAVEHDR response length distinguishes events from
    boundaries" (0x46 = real event, 0x2C = boundary marker)
  - protocol reference §7.8.5 / §7.8.6 / §7.8.7 / §7.8.8
- The previous chunk-counter formula (`max(key4[2:4], 0x0400) + (chunk-1) *
  0x0400`) is now marked DEPRECATED and explicitly tagged WRONG with
  pointers to the new sections, so future work doesn't re-derive it.

### Known minor diffs vs Blastware (deferred to a follow-up)

- We still use the OLD 0x0400 chunk step rather than BW's 0x0200; switching
  also requires updating `blastware_file.write_blastware_file`'s skip values
  and "extra chunk after metadata" logic, which depends on a fresh capture
  to verify.
- We still use the legacy fixed `offset_word=0x005A` TERM frame rather than
  BW's `end_offset - next_boundary` formula, for the same reason.
- Two fixed metadata pages at counter `0x1002` and `0x1004` are not yet
  read explicitly; under the current 0x0400 walk their content is reachable
  via the sample chunk that covers buffer addresses `[0x1000, 0x1400)`.

---

## v0.12.6 — 2026-05-01

### Fixed

- **`blastware_file.py` — waveform frame classification** — A5 frame classification for
  waveform-only vs header-only frames now uses `frame.record_type` instead of frame index.
  Only waveform frames (0x46) are written to the file body; metadata frames are skipped.
  Fixes spurious data corruption from incorrectly classified frames.

- **`s3_analyzer.py` — A5/5A frame naming** — Bulk waveform stream frames (SUB 5A response)
  are now correctly labeled "A5" in analyzer output instead of being conflated with other
  multi-frame responses (SUB A4, E5, etc.).

- **`S3FrameParser` — frame terminator detection** — Corrected the bare ETX terminator
  detection. Frame termination is now correctly identified by a standalone `ETX=0x03` byte,
  not by the `DLE+ETX` sequence (which is part of the payload when it appears within a frame).

---

## v0.12.5 — 2026-04-21

### Added

- **`seismo_lab.py` — Download tab** — New fourth tab for live wire-byte capture during event
  downloads. Captures both BW→device and device→S3 frames in real time, allowing inspection
  of the 5A bulk stream chunk sequence and frame-by-frame analysis without needing a bridge
  or MITM proxy. Files are saved with user-specified labels for easy tracking.

### Changed

- **`s3_bridge.py` — raw captures always-on by default** — `--raw-bw` and `--raw-s3` now
  default to `"auto"` instead of `None`. Every bridge session automatically generates
  timestamped `raw_bw_<ts>.bin` and `raw_s3_<ts>.bin` files alongside the `.bin`/`.log`
  session files. Pass `--raw-bw ""` (explicit empty string) to disable if needed.

- **`gui_bridge.py` — raw capture checkboxes pre-checked** — Both "BW→S3 raw" and
  "S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names
  the files). Unchecking a box passes `--raw-bw ""` to explicitly disable capture.

- **`Bridge tab` — TCP mode added** — Serial/TCP radio toggle allows connection via cellular
  modem (RV50/RV55) instead of direct RS-232. Supports multi-capture design (simultaneous
  Bridge + Analyzer + Download sessions).

- **`ach_server.py` — TX capture added (`raw_tx_<ts>.bin`)** — Every ACH inbound session
  now saves both directions: `raw_rx_<ts>.bin` (device → us, S3 side, as before) and
  `raw_tx_<ts>.bin` (us → device, BW side). Both files are usable in the Analyzer.
  TX bytes are buffered in memory until startup handshake succeeds (same as RX), preventing
  scanner probes from creating empty files.

---

## v0.12.4 — 2026-04-21 (protocol analysis / docs only — no code changes)

### Discovered

- **compliance_raw is wire-encoded, not logical bytes** — `read_compliance_config()` returns
  bytes that include DLE prefix bytes (`0x10`) before any `0x03` values (because S3FrameParser
  preserves DLE+ETX inner-frame pairs as two literal bytes). The previous CLAUDE.md claim that
  "S3FrameParser handles this transparently so compliance_raw contains logical bytes" was wrong.

- **anchor-9 behavior per recording mode** (confirmed from 4-20-26 BW write captures):
  - Single Shot (0x00) / Continuous (0x01): anchor-9 = `0x00`
  - Histogram (0x03): anchor-9 = `0x10` — the E5 DLE prefix for the `0x03` recording_mode byte
  - Histogram+Continuous (0x04): anchor-9 = `0x10` — an actual stored config byte for this mode
  Anchor position shifts by ±1 when recording_mode = `0x03` due to the extra DLE byte; the
  dynamic anchor search (`buf.find(ANCHOR, 0, 150)`) handles this correctly without code changes.

- **Write frame ETX escaping** — BW escapes `0x03` bytes in write frame data as `0x10 0x03`
  on the wire. Our `build_bw_write_frame` sends data bytes raw without ETX escaping. Device
  accepts our raw writes for all tested modes. Hypothesis: device write parser uses the
  offset/length field for frame boundaries, not ETX scanning, making ETX escaping optional.
  Histogram mode (recording_mode = 0x03) write via SFM from a non-Histogram starting state
  not yet tested.

- **BW write payload vs E5 read payload are byte-identical** around the anchor region (confirmed
  by comparing 3-11-26 BW TX and S3 captures). BW does NOT strip DLE prefix bytes before writing;
  it round-trips the wire-encoded bytes verbatim with only the modified fields changed.

- **Capture folder content catalogued** — see CLAUDE.md "BW capture reference" table for a
  summary of all available protocol captures and their contents.

---

## v0.12.3 — 2026-04-20

### Added

- **Auto Call Home config protocol** — Full read/write/decode/encode pipeline for the
  device's Remote Access → Setup Unit ACH settings, confirmed from 4-20-26 call home
  settings captures.

  **Protocol (new):**
  - `SUB 0x2C` — Call Home Config READ (response `0xD3`); two-step read; data offset
    `0x7C` = 124; raw payload 125 bytes (1-byte longer than DATA_LENGTH due to DLE-escaped
    `\x10\x03` at raw[117:119] representing num_retries = 3)
  - `SUB 0x7E` — Call Home Config WRITE (response `0x81`); 127-byte payload (125-byte read
    payload + `\x00\x00`); offset = `data[1]+2 = 0x7E`; write format (DLE-aware checksum)
  - `SUB 0x7F` — Call Home WRITE CONFIRM (response `0x80`); no data

  **Field map (confirmed from 10-frame BW TX diff):**
  - `raw[5]` — auto_call_home_enabled (bool)
  - `raw[6:46]` — dial_string (40-byte null-padded ASCII)
  - `raw[87]` — after_event_recorded (bool)
  - `raw[91]` — at_specified_times (bool)
  - `raw[93]` — time1_enabled / `raw[101]` — time1_hour / `raw[102]` — time1_min
  - `raw[95]` — time2_enabled / `raw[105]` — time2_hour / `raw[106]` — time2_min
  - `raw[117:119]` — `\x10\x03` (DLE-escaped 0x03 = num_retries value 3)
  - `raw[120]` — time_between_retries_sec / `raw[122]` — wait_for_connection_sec / `raw[124]` — warm_up_time_sec

  **Library (`minimateplus/`):**
  - `models.py` — `CallHomeConfig` dataclass (14 fields; `raw` bytes preserved for
    round-trip writes)
  - `protocol.py` — `SUB_CALL_HOME = 0x2C`, `SUB_CALL_HOME_WRITE = 0x7E`,
    `SUB_CALL_HOME_CONFIRM = 0x7F`; `read_call_home_config()`, `write_call_home_config()`
  - `client.py` — `get_call_home_config()`, `set_call_home_config()`,
    `_decode_call_home_config()` (handles DLE prefix at raw[117]),
    `_encode_call_home_config()` (patches in-place; raises `ValueError` if hour/min = 3)

  **REST API (`sfm/server.py`):**
  - `GET /device/call_home` — reads and decodes call home config from device
  - `POST /device/call_home` — reads, patches specified fields, writes back to device
  - `CallHomeConfigBody` Pydantic model with 9 optional writable fields

  **Web UI (`sfm/sfm_webapp.html`):**
  - New "Call Home" tab with enable flag, dial string (read-only), after-event trigger,
    at-specified-times flag, two time slots (enable + HH:MM each), and read-only retry
    settings (num_retries, time_between_retries_sec, wait_for_connection_sec,
    warm_up_time_sec)
  - "Read from Device", "Write to Device", "Clear Form" action buttons
  - Client-side guard: rejects hour or minute value equal to 3 with a clear message
    explaining the DLE-encoding limitation

---

## v0.12.2 — 2026-04-20

### Added / Fixed

- **Geophone sensitivity / maximum range field confirmed** — 4-20-26 geo sensitivity
  captures (1.25 in/s vs 10 in/s) diffed across all three SUB 71 write chunks and both
  E5 read payloads. The `geo_range` uint8 field per channel is now fully confirmed:
  - E5 read offset: `channel_label + 33`; SUB 71 write offset: `channel_label + 29`
  - `0x00` = Normal 10.000 in/s (standard gain); `0x01` = Sensitive 1.250 in/s (high gain)
  - **Correction:** previous hypothesis (`channel_label+20`, `0x01`=Normal) was wrong.
    `channel_label+20` reads `0x01` on ALL captures regardless of range — not this field.
  - `_decode_compliance_config_into`: read offset corrected from `tran_pos+20` → `tran_pos+33`
  - `_encode_compliance_config`: added `geo_range` parameter; writes to Tran/Vert/Long at `+29`
  - `apply_config`: added `geo_range` parameter
  - `POST /device/config`: added `geo_range` to `DeviceConfigBody`
  - Web UI Config tab: added "Maximum Range — Geo" select (Normal / Sensitive)
  - Web UI Device tab: added "Max Range (geo)" row to compliance table

- **`recording_mode` + `histogram_interval_sec` confirmed and implemented** (4-20-26 captures)
  - `recording_mode`: uint8 at anchor−8 (E5 read) / anchor−7 (write); enum: 0x00=Single Shot,
    0x01=Continuous, 0x03=Histogram, 0x04=Histogram+Continuous
  - `histogram_interval_sec`: uint16 BE seconds at anchor−4; same offset in read & write;
    valid: 2, 5, 15, 60, 300, 900 (matching Blastware dropdown: 2s, 5s, 15s, 1m, 5m, 15m)
  - Both fields added to `ComplianceConfig`, `_decode_compliance_config_into`,
    `_encode_compliance_config`, `apply_config`, REST API body, and web UI

---

## v0.12.1 — 2026-04-16

### Added

- **`sfm/server.py` — `_LiveCache`** — in-memory live device cache that eliminates
  redundant TCP round-trips between web requests.  Plain Python dict +
  `threading.Lock`, no extra dependencies.

  Cache strategy per endpoint:

  | Endpoint | Strategy |
  |---|---|
  | `GET /device/info` | Indefinite; invalidated by `POST /device/config` |
  | `GET /device/events` | Count-probe fast path — `poll()+count_events()` (~2 s); returns cached data if event count is unchanged; full download only when new events are detected |
  | `GET /device/monitor/status` | 30-second TTL; invalidated immediately on monitor start/stop |
  | `GET /device/event/{idx}/waveform` | Permanent per-index (waveforms are immutable once recorded) |

- **`?force=true`** query param on all cached endpoints — bypasses cache and forces
  a fresh read from the device.

- **Cache invalidation hooks** — `POST /device/config` marks device info and events
  stale; `POST /device/monitor/start` and `/stop` evict the monitor status entry
  immediately so the next status poll reflects the actual device state.

---

## v0.12.0 — 2026-04-13

### Added

- **`sfm/server.py` — `_LiveCache`** — in-memory live device cache, eliminating
  redundant TCP round-trips between requests.  No extra dependencies (plain Python
  dict + threading.Lock).  Replaces the SQLAlchemy-based `sfm/cache.py` experiment
  from the `feature/intelligent-caching` branch.

  Cache behaviour by endpoint:

  | Endpoint | Cache strategy |
  |---|---|
  | `GET /device/info` | Indefinite; invalidated by `POST /device/config` |
  | `GET /device/events` | Count-probe fast path: quick `poll()+count_events()` (~2s); return cache if count matches; full download only when new events detected |
  | `GET /device/monitor/status` | 30-second TTL; invalidated by monitor start/stop |
  | `GET /device/event/{idx}/waveform` | Permanent per-index (waveforms are immutable) |

- **`?force=true` param** on all four cached endpoints — bypasses cache and re-reads
  from device.

- **`POST /device/config` cache invalidation** — marks device info + events dirty so
  the next read reflects the new compliance config.

- **`POST /device/monitor/start` / `stop` cache invalidation** — evicts the monitor
  status cache entry immediately so the next poll returns the updated state.

### Removed

- `sfm/cache.py` — SQLAlchemy-based cache from the experimental caching branch.
  Its logic has been ported to the sqlite3-native `_LiveCache` class above.
  `sqlalchemy` is no longer a dependency.

---

## v0.11.0 — 2026-04-13

### Added

- **`sfm/database.py` — SeismoDb** — SQLite persistence layer for all ACH data.
  Three tables, all unit-keyed by serial number:
  - `ach_sessions` — one row per inbound call-home: serial, timestamp, peer IP,
    events_downloaded, monitor_entries, duration_seconds
  - `events` — one row per triggered waveform event: serial, waveform_key (dedup),
    timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location
    strings, sample_rate, record_type, false_trigger flag
  - `monitor_log` — one row per monitoring interval: serial, waveform_key (dedup),
    start_time, stop_time, duration_seconds, geo_threshold_ips
  - WAL mode, per-request connections — safe for the single-writer / occasional-reader
    ACH server pattern
  - Deduplication by `(serial, waveform_key)` UNIQUE constraint — re-runs and repeat
    call-homes never produce duplicate rows

- **`ach_server.py` — DB integration** — after each successful call-home, writes new
  events and monitor log entries to `seismo_relay.db` then records the session in
  `ach_sessions`.  DB write failures are logged as warnings and do not abort the session.

- **`sfm/server.py` — DB read endpoints**:
  - `GET /db/units` — distinct serials with last_seen, total_events, total_monitor_entries
  - `GET /db/events` — query events with serial / date range / false_trigger filters
  - `GET /db/monitor_log` — query monitoring intervals
  - `GET /db/sessions` — query ACH call-home sessions
  - `PATCH /db/events/{id}/false_trigger` — flag/unflag false triggers (for review UI)

### Architecture

- seismo-relay DB is unit-keyed only — no project concepts.  Project aggregation is
  terra-view's responsibility via `UnitAssignment` / `DeploymentRecord` + date range
  queries against the SFM DB endpoints.
- DB file lives at `bridges/captures/seismo_relay.db` by default.

---

## v0.10.0 — 2026-04-11

### Added

- **`MiniMateClient.get_monitor_log_entries(skip_keys=None)`** — browse-mode walk
  (`1E → 0A → 1F`) that collects partial records (`0x2C` record type) from the device's
  event list without triggering a full waveform download (no 0C or 5A).  Returns
  `list[MonitorLogEntry]`.  Each entry represents one continuous monitoring interval where
  no threshold was exceeded.

- **`_decode_0a_partial_header(raw_data, index, key4)`** in `client.py` — decodes a SUB
  0x0A response payload whose record type is `0x2C`.  Extracts:
  - `start_time` / `stop_time` — two consecutive timestamps; auto-detects 9-byte
    (sub_code=0x10, single-shot) vs 10-byte (sub_code=0x03, continuous) format from
    `raw_data[11]`.  Handles a 1-byte gap between the two timestamps that occurs when
    ts1 and ts2 share the same minute:second.
  - `serial` — device serial string found via `b"BE"` anchor scan.
  - `geo_threshold_ips` — trigger level found via `b"Geo: "` anchor scan.

- **`MonitorLogEntry` dataclass** in `models.py` — new model for partial records:
  `index`, `key`, `start_time`, `stop_time`, `serial`, `geo_threshold_ips`,
  `raw_header`, and a `duration_seconds` property.

- **`read_waveform_header()` return value extended** — now returns `(data_rsp.data, length)`
  (full payload) instead of `(data_rsp.data[11:11+length], length)`.  Callers get the
  complete payload including the record-type byte at position 0.  Full records use
  `raw_data[11:11+length]` as before; partial records are detected by `raw_data[0] == 0x2C`.

- **ACH server: monitor log collection** — after `get_events()`, calls
  `get_monitor_log_entries(skip_keys=seen_keys)` and saves new entries to
  `monitor_log.json` in the session directory.  Monitor log keys are included in
  `downloaded_keys` for state persistence (no re-processing on next call-home).

- **`_monitor_log_entry_to_dict()`** in `ach_server.py` — serialises a `MonitorLogEntry`
  to a JSON-compatible dict with ISO-format timestamps.

### Protocol / Documentation

- **SUB 0x0A partial record (0x2C) format confirmed** (✅ 4-11-26 MITM capture, 12 frames):
  - Record type `0x2C` at `raw_data[0]`; length < 64 bytes.
  - Two timestamps at `raw_data[11:]` — start and stop of the monitoring interval.
  - ASCII metadata region after timestamps: `BE<serial>\x00Geo: <float> in/s`.
  - Edge case: 1-byte separator between timestamps when ts1 and ts2 share minute:second.
  - 10-byte timestamp format (sub_code=0x03) signalled by `raw_data[11] == 0x10`.

- **Key reuse detection for monitor log entries** — monitor log keys are tracked alongside
  event keys in `ach_state.json` so the ACH server does not re-process them after a
  call-home cycle.

---

## v0.9.0 — 2026-04-11

### Added

- **`MiniMateClient.list_event_keys()`** — fast browse-mode walk (1E → 0A → 1F, no waveform
  download) that returns the list of event key hex strings currently stored on the device.
  Used by the ACH server as a cheap pre-check before deciding whether to call `get_events()`.

- **`get_events(skip_waveform_for_keys=set(...))`** — new optional parameter.  For any key in
  the set the function performs only 0A + 1F(browse) instead of the full
  1E-arm → 0C → POLL×3 → 5A sequence.  Eliminates redundant waveform downloads on repeat
  call-homes when the device still holds previously downloaded events.

- **`MiniMateClient.delete_all_events()`** — erases all events from device memory using the
  confirmed 4-step sequence:
  - SUB 0xA3 `begin_erase_all` — initiate erase (token=0xFE) → ack 0x5C
  - SUB 0x1C `read_monitor_status` — intermediate status read (Blastware-required)
  - SUB 0x06 `read_event_storage_range` — verify storage state (token=0xFE) → 36-byte response
  - SUB 0xA2 `confirm_erase_all` — commit erase (token=0xFE) → ack 0x5D

  All four steps confirmed from 4-11-26 MITM capture of a live Blastware ACH session.
  After a successful call, the device's event counter resets to `0x01110000`.

- **`MiniMateProtocol` erase methods**: `begin_erase_all()`, `confirm_erase_all()`,
  `read_event_storage_range()` added to `protocol.py` with documented SUB constants
  `SUB_ERASE_ALL_BEGIN = 0xA3` and `SUB_ERASE_ALL_CONFIRM = 0xA2`.

- **`bridges/ach_mitm.py`** — transparent TCP-to-TCP MITM proxy.  Listens for inbound unit
  connections, connects upstream to a real Blastware ACH server, and saves both directions
  to `raw_bw_<ts>.bin` / `raw_s3_<ts>.bin` files matching the existing capture format.
  Used to capture the 4-11-26 Blastware ACH session including event deletion.
  Usage: `python bridges/ach_mitm.py --bw-host 127.0.0.1 --bw-port 9999 --listen-port 9998`

- **ACH server: key-based state tracking** — `ach_state.json` now stores
  `downloaded_keys: [hex_strings]` and `max_downloaded_key: hex_string` per unit instead of
  `event_count: N`.  This correctly handles the standard workflow where events are deleted
  from the device after upload — a count-based approach would see `count=0` on the next
  call-home and silently skip new events.

- **ACH server: `--clear-after-download` flag** — after a successful download (at least one
  new event saved), erases all events from the device using `delete_all_events()`.  Mirrors
  the standard Blastware ACH workflow.  On success, `downloaded_keys` and
  `max_downloaded_key` are reset to empty so the next session starts fresh.

- **ACH server: post-erase key-reuse detection** — after an external erase (Blastware or
  manual), device keys restart from `0x01110000`, colliding with previously downloaded keys.
  On each browse walk, if `max(device_keys) < max_downloaded_key` (device counter rolled
  back), all device keys are treated as new regardless of `seen_keys`.  This also catches
  erases performed by Blastware between our sessions.

### Protocol / Documentation

- **SUB 0xA3 / SUB 0xA2 — erase-all sequence confirmed** (✅ 4-11-26 MITM capture):
  Both frames use `token=0xFE` at `params[7]` and are standard `build_bw_frame` requests
  (not write-format).  Response SUBs follow the standard formula: 0x5C and 0x5D.
  The intermediate 0x1C + 0x06 reads between them are required by Blastware.

- **SUB 0x06 — event storage range read confirmed** (✅ 4-11-26 MITM capture):
  Two-step read, data offset = 0x24 (36 bytes).  The last 8 bytes of the response contain
  the first and last stored event keys (4 bytes each).  After a successful erase, both keys
  read as `01110000` (device-empty state).

- **Event key counter resets to `0x01110000` after erase** — confirmed by observing key
  `01110000` on the device immediately after the MITM erase session.

---

## v0.8.0 — 2026-04-07

### Added

- **Write pipeline end-to-end** — `push_config_raw(event_index_data, compliance_data,
  trigger_data, waveform_data)` on `MiniMateClient` orchestrates the full
  `68→73 | 71×3→72 | 82→83 | 69→74→72` write sequence.

- **`build_bw_write_frame(sub, data, *, offset, params)`** in `framing.py` — dedicated frame
  builder for write commands (SUBs 0x68–0x83).  Doubles only the BW_CMD byte; all other
  bytes including offset, params, data, and checksum are written raw.  Uses the large-frame
  DLE-aware checksum (`sum(b for b in payload[2:] if b != 0x10) + 0x10) & 0xFF`).

- **`MiniMateProtocol` write methods** — `write_event_index()`, `write_compliance()`,
  `write_trigger_config()`, `write_waveform_data()`, `write_confirm()`,
  `start_monitoring()`, `stop_monitoring()`.

- **`AchSession` inbound server** (`bridges/ach_server.py`) — accepts call-home TCP
  connections, runs the full handshake + device-info + event-download sequence, saves
  `device_info.json` + `events.json` per session.

### Protocol / Documentation

- **Write frame format confirmed** (✅ 3-11-26 BW TX capture, all 11 frames): only BW_CMD
  byte `0x10` is doubled; all other bytes sent raw.  Standard `build_bw_frame` DLE-stuffing
  is incorrect for write commands.
- **Write ack responses** confirmed as 17-byte zero-data S3 frames.
- **Monitoring SUBs 0x96/0x97** confirmed from 4-8-26 capture.
- **SESSION_RESET signal** (`41 03`) required before POLL for monitoring units.
- **SUB 0x1C monitoring flag** at `section[1]`: `0x00` = idle, `0x10` = monitoring.
  Confirmed by byte-diff of all 144 data frames in 4-8-26/2ndtry capture.

---

## v0.7.0 — 2026-04-03

### Added
- **Raw ADC waveform decode — `_decode_a5_waveform(frames_data, event)`** in `client.py`.
  Parses the complete set of SUB 5A A5 response frames into per-channel time-series:
  - Reads the STRT record from A5[0] (bytes 7+): extracts `total_samples` (BE uint16 at +8),
    `pretrig_samples` (BE uint16 at +16), and `rectime_seconds` (uint8 at +18) into
    `event.total_samples / pretrig_samples / rectime_seconds`.
  - Skips the 6-byte preamble (`00 00 ff ff ff ff`) that follows the 21-byte STRT header;
    waveform data begins at `strt_pos + 27`.
  - Strips the 8-byte per-frame counter header from A5[1–6, 8] before appending waveform bytes.
  - Skips A5[7] (metadata-only) and A5[9] (terminator).
  - **Cross-frame alignment correction**: accumulates `running_offset % 8` across all frames
    and discards `(8 − align) % 8` leading bytes per frame to re-align to a T/V/L/M boundary.
    Required because individual frame waveform payloads are not always multiples of 8 bytes.
  - Decodes as 4-channel interleaved signed 16-bit LE at 8 bytes per sample-set:
    bytes 0–1 = Tran, 2–3 = Vert, 4–5 = Long, 6–7 = Mic.
  - Stores result in `event.raw_samples = {"Tran": [...], "Vert": [...], "Long": [...], "Mic": [...]}`.
- **`download_waveform(event)` public method** on `MiniMateClient`.
  Issues a full SUB 5A stream with `stop_after_metadata=False`, then calls
  `_decode_a5_waveform()` to populate `event.raw_samples` and `event.total_samples /
  pretrig_samples / rectime_seconds`. Previously only metadata frames were fetched during
  `get_events()`; raw waveform data is now available on demand.
- **`Event` model new fields** (`models.py`): `total_samples`, `pretrig_samples`,
  `rectime_seconds` (from STRT record), and `_waveform_key` (4-byte key stored during
  `get_events()` for later use by `download_waveform()`).

### Protocol / Documentation
- **SUB 5A A5[0] STRT record layout confirmed** (✅ 2026-04-03, 4-2-26 blast capture):
  - STRT header is 21 bytes: `b"STRT"` + length fields + `total_samples` (BE uint16 at +8) +
    `pretrig_samples` (BE uint16 at +16) + `rectime_seconds` (uint8 at +18).
  - Followed by 6-byte preamble: `00 00 ff ff ff ff`. Waveform begins at `strt_pos + 27`.
  - Confirmed: 4-2-26 blast → `total_samples=9306`, `pretrig_samples=298`, `rectime_seconds=70`.
- **Blast/waveform mode A5 format confirmed** (✅ 2026-04-03, 4-2-26 blast capture):
  4-channel interleaved int16 LE at 8 bytes per sample-set; cross-frame alignment correction
  required. 948 of 9306 total sample-sets captured via `stop_after_metadata=True` (10 frames).
- **Noise/histogram mode A5 format — endianness corrected** (✅ 2026-04-03, 3-31-26 capture):
  32-byte block samples are signed 16-bit **little-endian** (previously documented as BE).
  `0a 00` → LE int16 = 10 (correct noise floor); BE would give 2560 (wrong).
- Protocol reference §7.6 rewritten — split into §7.6.1 (Blast/Waveform mode) and §7.6.2
  (Noise/Histogram mode), each with confirmed field layouts and open questions noted.

---

## v0.6.0 — 2026-04-02

### Added
- **True event-time metadata via SUB 5A bulk waveform stream** — `get_events()` now issues a SUB 5A request after each SUB 0C download, reads the A5 response frames, and extracts the `Client:`, `User Name:`, and `Seis Loc:` fields as they existed at the moment the event was recorded. Previously these fields were backfilled from the current compliance config (SUB 1A), which reflects today's setup, not the setup active when the event triggered.
  - `build_5a_frame(offset_word, raw_params)` in `framing.py` — reproduces Blastware's exact wire format for SUB 5A requests: raw (non-DLE-stuffed) `offset_hi`, DLE-stuffed params, and a DLE-aware checksum where `10 XX` pairs count only `XX`.
  - `bulk_waveform_params()` returns 11 bytes (extra trailing `0x00` confirmed from 1-2-26 BW wire capture).
  - `read_bulk_waveform_stream(key4, *, stop_after_metadata=True, max_chunks=32)` in `protocol.py` — loops sending chunk requests (counter increments `0x0400` per chunk), stops early when `b"Project:"` is found, then sends a termination frame.
  - `_decode_a5_metadata_into(frames_data, event)` in `client.py` — needle-searches A5 frame data for `Project:`, `Client:`, `User Name:`, `Seis Loc:`, `Extended Notes` and overwrites `event.project_info`.
- **`get_events()` sequence extended** — now `1E → 0A → 0C → 5A → 1F` per event.

### Fixed
- **Compliance config (SUB 1A) channel block missing** — orphaned `self._send(build_bw_frame(SUB_COMPLIANCE, 0x2A, _DATA_PARAMS))` before the B/C/D receive loop had no corresponding `recv_one()`, shifting all subsequent receives one step behind and leaving frame D's channel-block data (trigger_level_geo, alarm_level_geo, max_range_geo) unread. Removed the orphaned send. Total config bytes received now correctly ~2126 (was ~1071).
- **Compliance config anchor search range** — `_decode_compliance_config_into()` searched `cfg[40:100]` for the sample-rate/record-time anchor. With the orphaned-send bug fixed the 44-byte padding it had been adding is gone, and the anchor now appears at `cfg[11]`. Search widened to `cfg[0:150]` to be robust to future layout shifts.
- Removed byte-content deduplication from `read_compliance_config()` — was masking the real receive-ordering bug.

### Protocol / Documentation
- **SUB 5A frame format confirmed** — `offset_hi` byte (`0x10`) must be sent raw (not DLE-stuffed); checksum is DLE-aware (only the second byte of a `10 XX` pair is summed). Standard `build_bw_frame` DLE-stuffs `0x10` incorrectly for 5A — a dedicated `build_5a_frame` is required.
- **Event-time metadata source confirmed** — `Client:`, `User Name:`, and `Seis Loc:` strings are present in A5 frame 7 of the bulk waveform stream (SUB 5A), not in the 210-byte SUB 0C waveform record. They reflect the compliance setup as it was when the event was stored on the device.

---

## v0.5.0 — 2026-03-31

### Added
- **Console tab in `seismo_lab.py`** — direct device connection without the bridge subprocess.
  - Serial and TCP transport selectable via radio buttons.
  - Four one-click commands: POLL, Serial #, Full Config, Event Index.
  - Colour-coded scrolling output: TX (blue), RX raw hex (teal), parsed/decoded (green), errors (red).
  - Save Log and Send to Analyzer buttons; logs auto-saved to `bridges/captures/console_<ts>.log`.
  - Queue/`after(100)` pattern — no UI blocking or performance impact.
- **`minimateplus` package** — clean Python client library for the MiniMate Plus S3 protocol.
  - `SerialTransport` and `TcpTransport` (for Sierra Wireless RV50/RV55 cellular modems).
  - `MiniMateProtocol` — DLE frame parser/builder, two-step paged reads, checksum validation.
  - `MiniMateClient` — high-level client: `connect()`, `get_serial()`, `get_config()`, `get_events()`.
- **TCP/cellular transport** (`TcpTransport`) — connect to field units via Sierra Wireless RV50/RV55 modems over cellular.
  - `read_until_idle(idle_gap=1.5s)` to handle modem data-forwarding buffer delay.
  - Confirmed working end-to-end: TCP → RV50/RV55 → RS-232 → MiniMate Plus.
- **`bridges/tcp_serial_bridge.py`** — local TCP-to-serial bridge for bench testing `TcpTransport` without a cellular modem.
- **SFM REST server** (`sfm/server.py`) — FastAPI server with device info, event list, and event record endpoints over both serial and TCP.

### Fixed
- `protocol.py` `startup()` was using a hardcoded `POLL_RECV_TIMEOUT = 10.0` constant, ignoring the configurable `self._recv_timeout`. Fixed to use `self._recv_timeout` throughout.
- `sfm/server.py` now retries once on `ProtocolError` for TCP connections to handle cold-boot timing on first connect.

### Protocol / Documentation
- **Sierra Wireless RV50/RV55 modem config** — confirmed required ACEmanager settings: Quiet Mode = Enable, Data Forwarding Timeout = 1, TCP Connect Response Delay = 0. Quiet Mode disabled causes modem to inject `RING\r\nCONNECT\r\n` onto the serial line, breaking the S3 handshake.
- **Calibration year** confirmed at SUB FE (Full Config) destuffed payload offset 0x56–0x57 (uint16 BE). `0x07E7` = 2023, `0x07E9` = 2025.
- **`"Operating System"` boot string** — 16-byte UART boot message captured on cold-start before unit enters DLE-framed mode. Parser handles correctly by scanning for DLE+STX.
- RV50/RV55 sends `RING`/`CONNECT` over TCP to the calling client even with Quiet Mode enabled — this is normal behaviour, parser discards it.

---

## v0.4.0 — 2026-03-12

### Added
- **`seismo_lab.py`** — combined Bridge + Analyzer GUI. Single window with two tabs; bridge start auto-wires live mode in the Analyzer.
- **`frame_db.py`** — SQLite frame database. Captures accumulate over time; Query DB tab searches across all sessions.
- **`bridges/s3-bridge/proxy.py`** — bridge proxy module.
- Large BW→S3 write frame checksum algorithm confirmed and implemented (`SUM8` of payload `[2:-1]` skipping `0x10` bytes, plus constant `0x10`, mod 256).
- SUB `A4` identified as composite container frame with embedded inner frames; `_extract_a4_inner_frames()` and `_diff_a4_payloads()` reduce diff noise from 2300 → 17 meaningful entries.

### Fixed
- BAD CHK false positives on BW POLL frames — BW frame terminator `03 41` was being included in the de-stuffed payload. Fixed to strip correctly.
- Aux Trigger read location confirmed at SUB FE offset `0x0109`.

---

## v0.3.0 — 2026-03-09

### Added
- Record time confirmed at SUB E5 page2 offset `+0x28` as float32 BE.
- Trigger Sample Width confirmed at BW→S3 write frame SUB `0x82`, destuffed payload offset `[22]`.
- Mode-gating documented: several settings only appear on the wire when the appropriate mode is active.

### Fixed
- `0x082A` mystery resolved — fixed-size E5 payload length (2090 bytes), not a record-time field.

---

## v0.2.0 — 2026-03-01

### Added
- Channel config float layout fully confirmed: trigger level, alarm level, and unit string per channel (IEEE 754 BE floats).
- Blastware `.set` file format decoded — little-endian binary struct mirroring the wire payload.
- Operator manual (716U0101 Rev 15) added as cross-reference source.

---

## v0.1.0 — 2026-02-26

### Added
- Initial `s3_bridge.py` serial bridge — transparent RS-232 tap between Blastware and MiniMate Plus.
- `s3_parser.py` — deterministic DLE state machine frame extractor.
- `s3_analyzer.py` — session parser, frame differ, Claude export.
- `gui_bridge.py` and `gui_analyzer.py` — Tkinter GUIs.
- DLE framing confirmed: `DLE+STX` / `DLE+ETX`, `0x41` = ACK (not STX), DLE stuffing rule.
- Response SUB rule confirmed: `response_SUB = 0xFF - request_SUB`.
- Year `0x07CB` = 1995 confirmed as MiniMate factory RTC default.
- Full write command family documented (SUBs `68`–`83`).