seismo-relay/README.md

# seismo-relay  `v0.19.0`

A ground-up replacement for **Blastware** — Instantel's aging Windows-only
software for managing seismographs.  Supports both the **MiniMate Plus
(Series III)** and the **Micromate (Series IV / "Thor")** families:
Series III via the live RS-232 / TCP wire protocol *and* Blastware ACH file
ingest; Series IV currently via Thor TXT-paired IDF file ingest, with the
binary codec on the roadmap.

Built in Python. Runs on Windows, Linux, or macOS. Connects to instruments
over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).

> **Status:** Active development. Full read + write + erase + monitoring
> pipeline working end-to-end over TCP/cellular. ACH Auto Call Home server
> handles inbound unit connections, downloads events, and persists everything
> to a SQLite database. SFM REST API exposes device control and DB queries.
> **As of v0.14.3 (2026-05-05): SUB 5A bulk waveform protocol is verified
> byte-perfect against Blastware captures across 2-sec, 3-sec, and 10-sec
> events.** Generated `.G10` / `.AB0` files open cleanly in Blastware with
> full Event Reports, frequency analysis, and waveform plots.
> **v0.16.0 (2026-05-11)** adds BW ASCII report ingestion to
> `/db/import/blastware_file` — paired with **series3-watcher v1.5.0**,
> every Blastware ACH event lands in SeismoDb with device-authoritative
> peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data,
> without depending on the still-undecoded waveform body codec.
> **v0.18.0 (2026-05-19)** adds Thor / Micromate Series IV ingest at
> `/db/import/idf_file` — paired with **thor-watcher v0.3.0**, every
> `.IDFH` / `.IDFW` event file (plus its `.txt` sidecar) lands in
> SeismoDb the same way BW events do.  See
> [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md) for
> the IDF format reference and reverse-engineering plan.
> **v0.19.0 (2026-05-20)** separates Series III and Series IV at the
> code level: new `micromate/` package alongside `minimateplus/`, new
> `events.device_family` DB column ("series3" / "series4") so the UI
> and storage layer dispatch deterministically instead of sniffing
> filenames.  Self-applying migration backfills existing rows from the
> binary filename extension.
> See [CHANGELOG.md](CHANGELOG.md) for full version history.

---

## What's in here

```
seismo-relay/
├── seismo_lab.py              ← Main GUI (Bridge + Analyzer + Download + Console tabs)
│
├── minimateplus/              ← Series III (MiniMate Plus) client library
│   ├── transport.py           ←   SerialTransport, TcpTransport, SocketTransport
│   ├── protocol.py            ←   DLE frame layer, SUB command dispatch
│   ├── client.py              ←   High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …)
│   ├── framing.py             ←   Frame builders, DLE codec, S3FrameParser
│   ├── models.py              ←   DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig, …
│   ├── bw_ascii_report.py     ←   Parse BW per-event ASCII reports (.TXT sidecars)
│   ├── event_file_io.py       ←   Read BW binaries, write .sfm.json sidecars
│   └── blastware_file.py      ←   Write events to Blastware-compatible .AB0 files
│
├── micromate/                 ← Series IV (Micromate / Thor) client library (NEW v0.19)
│   ├── models.py              ←   IdfEvent, IdfReport, IdfPeaks, IdfProjectInfo, IdfSensorCheck (mic in native dB(L))
│   ├── idf_ascii_report.py    ←   Parse Thor .IDFW.txt / .IDFH.txt event sidecars
│   └── idf_file.py            ←   Stub for the .IDFW / .IDFH binary codec (reverse-engineering pending)
│
├── sfm/                       ← SFM REST API server (FastAPI, port 8200)
│   ├── server.py              ←   Live device endpoints + DB query + ingest endpoints + caching
│   ├── database.py            ←   SeismoDb — SQLite persistence (events, monitor_log, ach_sessions)
│   ├── waveform_store.py      ←   On-disk store for BW + IDF event binaries + .sfm.json sidecars
│   └── sfm_webapp.html        ←   Embedded web UI with Call Home config tab
│
├── bridges/
│   ├── ach_server.py          ←   Inbound ACH call-home server (main production server)
│   ├── ach_mitm.py            ←   Transparent MITM proxy for capturing BW sessions
│   ├── s3-bridge/             ←   RS-232 serial bridge (capture tool)
│   ├── tcp_serial_bridge.py   ←   Local TCP↔serial bridge (bench testing)
│   ├── gui_bridge.py          ←   Standalone bridge GUI with raw capture checkboxes
│   └── raw_capture.py         ←   Simple raw capture tool
│
├── parsers/
│   ├── s3_analyzer.py         ←   Session parser, differ, Claude export
│   ├── gui_analyzer.py        ←   Standalone analyzer GUI
│   └── frame_db.py            ←   SQLite frame database
│
└── docs/
    ├── instantel_protocol_reference.md  ← Series III protocol spec (the Rosetta Stone)
    └── idf_protocol_reference.md         ← Series IV (Thor IDF) format reference + codec RE plan
```

---

## Quick start

### ACH inbound server (production)

Listens for inbound unit call-homes, downloads all new events and monitor log
entries, and writes everything to `bridges/captures/seismo_relay.db`.

```bash
python bridges/ach_server.py --port 12345 --output bridges/captures/
```

Point the unit's ACEmanager **Remote Host** to this machine's IP and **Remote Port** to `12345`.

Options:
```
--port N            Listen port (default 12345)
--output DIR        Capture directory (default bridges/captures/)
--allow-ip IP       Allowlist an IP (repeat for multiple; default: accept all)
--max-events N      Safety cap for first run (default: unlimited)
--clear-after-download  Erase device memory after successful download
--verbose           Debug logging
```

### SFM REST server

Exposes device control and DB queries as a REST API. Proxied by terra-view.

```bash
python sfm/server.py               # default: 0.0.0.0:8200
python -m uvicorn sfm.server:app --host 0.0.0.0 --port 8200 --reload
```

Open `http://localhost:8200` for the embedded web UI, or `http://localhost:8200/docs`
for the interactive API docs.

### Seismo Lab GUI

```bash
python seismo_lab.py
```

---

## SFM REST API

### Live device endpoints

Each call dials the device, does its work, and closes the connection. TCP
connections are retried once on `ProtocolError` to handle cold-boot timing.

**In-memory caching** — frequently-polled endpoints avoid redundant TCP round-trips
via a thread-safe `_LiveCache` (plain Python dict + `threading.Lock`):

| Method | URL | Cache Strategy |
|--------|-----|---|
| `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` |
| `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected |
| `GET` | `/device/event/{idx}/waveform` | Permanent per event index |
| `GET` | `/device/monitor/status` | 30-second TTL; invalidated by monitor start/stop |
| `GET` | `/device/call_home` | Fresh read from device (not cached) |
| `POST` | `/device/connect` | — |
| `POST` | `/device/config` | Writes compliance config; invalidates info + events cache |
| `POST` | `/device/config/project` | Patches project/client/operator/sensor_location strings |
| `POST` | `/device/monitor/start` | Sends SUB 0x96; immediately evicts status cache |
| `POST` | `/device/monitor/stop` | Sends SUB 0x97; immediately evicts status cache |
| `POST` | `/device/call_home` | Reads, patches specified fields, writes back to device |

**Cache bypass** — All cached endpoints accept `?force=true` to skip the cache and
force a fresh read from the device.

**Cache stats** — `GET /cache/stats` returns hit/miss counts and TTL info; `DELETE /cache/device`
clears the device cache immediately.

Transport query params (supply one set):
```
Serial:  ?port=COM5&baud=38400
TCP:     ?host=1.2.3.4&tcp_port=12345
```

### DB read endpoints

Query the SQLite database written by `ach_server.py`. All read-only except
`PATCH /db/events/{id}/false_trigger`.

| Method | URL | Description |
|--------|-----|-------------|
| `GET` | `/db/units` | All known serials with summary stats |
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger).  Response rows include `device_family` ("series3" / "series4") so clients dispatch on unit type without sniffing filenames. |
| `GET` | `/db/monitor_log` | Monitoring intervals |
| `GET` | `/db/sessions` | ACH call-home session history |
| `PATCH` | `/db/events/{id}/false_trigger?value=true` | Flag / unflag false triggers |

### File ingest endpoints

Used by watcher daemons to push field-collected event files into the SFM DB
+ waveform store.  Both accept multipart uploads of binary event files
optionally paired with their ASCII sidecar reports; both dedup by
`(serial, timestamp)` and UPSERT device-authoritative fields on re-import.

| Method | URL | Description |
|--------|-----|-------------|
| `POST` | `/db/import/blastware_file` | Series III: `.AB0*` / `.N00` binaries + paired `_ASCII.TXT`.  Source: `series3-watcher`. |
| `POST` | `/db/import/idf_file` | Series IV: `.IDFH` / `.IDFW` binaries + paired `.IDFW.txt` / `.IDFH.txt`.  Source: `thor-watcher`. |

---

## minimateplus library

```python
from minimateplus import MiniMateClient
from minimateplus.transport import TcpTransport

# Serial
client = MiniMateClient(port="COM5")

# TCP (cellular modem)
client = MiniMateClient(transport=TcpTransport("1.2.3.4", 12345), timeout=30.0)

with client:
    # Read
    info     = client.connect()               # DeviceInfo — serial, firmware, compliance config
    count    = client.count_events()          # Number of stored events
    keys     = client.list_event_keys()       # Fast browse walk — event keys only, no download
    events   = client.get_events()            # Full download: headers + peaks + metadata
    monitor  = client.get_monitor_status()    # Battery, memory, is_monitoring flag
    log      = client.get_monitor_log_entries()  # Monitoring intervals (partial 0x2C records)
    ach_cfg  = client.get_call_home_config()  # Auto Call Home settings (SUB 0x2C)

    # Write
    client.apply_config(
        sample_rate=1024,
        recording_mode="Continuous",         # Single Shot / Continuous / Histogram / Histogram+Continuous
        histogram_interval_sec=15,            # 2, 5, 15, 60, 300, 900
        trigger_level_geo=0.5,
        geo_range="Normal",                   # Normal (10.000 in/s) / Sensitive (1.25 in/s)
        project="Bridge Inspection 2026",
        client_name="City of Portland",
        operator="B. Harrison",
    )
    
    client.set_call_home_config(
        auto_call_home_enabled=True,
        after_event_recorded=True,
        at_specified_times=True,
        time1_hour=18, time1_min=30,          # 6:30 PM
        time2_hour=6, time2_min=0,            # 6:00 AM
    )

    # Control
    client.start_monitoring()                # SUB 0x96
    client.stop_monitoring()                 # SUB 0x97
    client.delete_all_events()              # Erase all (SUB 0xA3 → 0x1C → 0x06 → 0xA2)
```

`get_events()` runs the full per-event sequence:
`1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)`.
SUB 5A bulk stream walks chunks bounded by the `end_offset` extracted from
the STRT record at byte 17 of the probe response — no over-reading, no
chunk-count cap. Project / client / operator / sensor location strings come
from the dedicated metadata pages at counter `0x1002` and `0x1004`,
read once per session (they reflect the compliance setup at session start,
not per individual event).

---

## micromate library

Series IV / Thor support, sibling to `minimateplus`.  Currently scoped to
offline-file ingest from Thor's TXT exporter; live-device protocol is
deferred until the binary codec is cracked.

```python
from micromate import IdfEvent, parse_idf_report

# Parse a .IDFW.txt / .IDFH.txt sidecar (1014 example files round-trip cleanly)
text = open("UM11719_20231219162723.IDFW.txt").read()
report_dict = parse_idf_report(text)        # permissive dict

# Wrap into a typed event using the device-native binary filename
event = IdfEvent.from_report(report_dict, "UM11719_20231219162723.IDFW")

event.serial                     # "UM11719"
event.kind                       # "Waveform" or "Histogram"
event.peaks.transverse_ips       # 0.0251  (in/s, native unit)
event.peaks.mic_pspl_dbl         # 99.4    (dB(L), Thor's native mic unit — NOT psi)
event.project_info.project       # "UPMC Presby-Loc 3-Level1-1R Elevator Rm"
event.sensor_check.tran          # True (passed self-check)
event.firmware_version           # "Micromate ISEE 11.0AK"
event.calibration_text           # "November 22, 2023 by Instantel"

# Bridge to the existing minimateplus.Event shape for the DB / sidecar paths
# (waveform_key is a 16-byte sha256 prefix when ingesting from a binary file)
bridged_event = event.to_minimateplus_event(waveform_key=b"\x00" * 16)
```

The binary codec (`.IDFW` / `.IDFH` event files themselves) is on the
roadmap — see [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md)
for everything known so far, the two observed file signatures, and the
reverse-engineering plan.  The `micromate/idf_file.py` stub is where
`read_idf_file()` will land.

---

## Database

`ach_server.py` and the file-ingest endpoints write to
`bridges/captures/seismo_relay.db` (SQLite, WAL mode) via the `SeismoDb`
persistence layer.  Three tables, all unit-keyed by serial number:

| Table | Key | Contents |
|-------|-----|----------|
| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds |
| `events` | UUID, UNIQUE(serial, timestamp) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag, **`device_family`** ("series3" / "series4"), `blastware_filename` (binary at-rest in `waveforms/`), sidecar references |
| `monitor_log` | UUID, UNIQUE(serial, start_time) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |

**Deduplication is by `(serial, timestamp)`** — the device clock is the
stable natural key.  Repeat call-homes or re-runs UPSERT the row in place,
refreshing every device-authoritative field (peaks, project strings,
sample_rate, file references) so the latest writer wins.  `false_trigger`
and `device_family` are preserved across UPSERTs.  Earlier versions used
`(serial, waveform_key)` for dedup, but the device's event-key counter
resets to `0x01110000` after every erase, so timestamps are the correct
dedup field.  Migration handles the transition transparently on first
startup.

**`device_family` (added v0.19.0)** discriminates Series III from Series
IV at the SQL level.  Set by every import path; the UI dispatches on it
to render mic units correctly (Series III: psi → dBL conversion; Series
IV: native dBL passthrough).  Existing rows are backfilled at first
startup of v0.19.0+ by sniffing the binary filename extension.

The on-disk waveform store lives at `bridges/captures/waveforms/<serial>/`
and holds the original event binaries (BW `.AB0*` / `.N00` for Series III,
`.IDFH` / `.IDFW` for Series IV) plus their `.sfm.json` review/metadata
sidecars.  Series III events also produce `.a5.pkl` source-frame pickles
and `.h5` clean-waveform exports; Series IV doesn't yet (pending codec).

---

## Connecting over cellular (RV50 / RV55)

Field units connect via Sierra Wireless RV50 or RV55 cellular modems.

### Required ACEmanager settings

| Setting | Value | Why |
|---------|-------|-----|
| Configure Serial Port | `38400,8N1` | Must match MiniMate baud rate |
| Flow Control | `None` | Hardware FC blocks TX if pins unconnected |
| **Quiet Mode** | **Enable** | **Critical** — disabled injects `RING`/`CONNECT` onto serial, corrupting the S3 handshake |
| Data Forwarding Timeout | `1` (= 0.1 s) | Lower latency |
| TCP Connect Response Delay | `0` | Non-zero silently drops the first POLL frame |
| TCP Idle Timeout | `2` (minutes) | Prevents premature disconnect |
| DB9 Serial Echo | `Disable` | Echo corrupts the data stream |

---

## Protocol quick-reference

| Term | Value | Meaning |
|------|-------|---------|
| DLE | `0x10` | Data Link Escape |
| STX | `0x02` | Start of frame |
| ETX | `0x03` | End of frame |
| ACK | `0x41` | Frame-start marker sent before every BW frame |
| DLE stuffing | `10 10` on wire | Literal `0x10` in payload |

**Response SUB rule:** `response_SUB = 0xFF - request_SUB` (no exceptions)

Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/instantel_protocol_reference.md)

---

## Compliance Config Features

The REST API and web UI expose full control over device compliance settings:

- **Recording Mode** (Single Shot / Continuous / Histogram / Histogram+Continuous)
- **Sample Rate** (1024 / 2048 / 4096 sps)
- **Record Time** (float, seconds)
- **Histogram Interval** (2s, 5s, 15s, 1m, 5m, 15m) — when recording mode includes histogram
- **Geo Trigger Levels** (float, in/s per channel)
- **Geo Maximum Range** (Normal 10.000 in/s / Sensitive 1.250 in/s per channel)
- **Project / Client / Operator / Sensor Location** (ASCII strings)

Auto Call Home config:
- **Auto Call Home Enable** (bool)
- **Dial String** (read-only; 40-byte ASCII)
- **Trigger on Event** (bool)
- **Scheduled Call-Ins** (two time slots with HH:MM each)
- **Retry Settings** (count, delay, connection timeout, warm-up time)

---

## Requirements

```bash
pip install pyserial fastapi uvicorn
```

Python 3.10+. Tkinter is included with the standard Python installer on
Windows (check "tcl/tk and IDLE" during install).

---

## Virtual COM ports (bridge capture)

```
Blastware → COM4 (virtual) ↔ s3_bridge.py ↔ COM5 (physical) → MiniMate Plus
```

Use **com0com** or **VSPD** to create the virtual COM pair on Windows.

---

## Key Features

**Series III (MiniMate Plus) device support:**
- [x] Full read/write/erase pipelines over RS-232 or TCP/cellular
- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings)
- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries)
- [x] Monitor control (start/stop, status polling, battery/memory)
- [x] Monitor log entries (continuous monitoring intervals without full waveform download)
- [x] Blastware file ingest at `/db/import/blastware_file` (paired with `series3-watcher`)

**Series IV (Micromate / Thor) device support:**
- [x] Thor IDF file ingest at `/db/import/idf_file` (paired with `thor-watcher`, v0.18.0+)
- [x] Native `IdfEvent` / `IdfReport` typed models — mic in dB(L), full title strings, sensor self-check, calibration, firmware version
- [x] Parser verified against 1,014 paired `.txt` sidecars in `thor-watcher/example-data/`
- [ ] Binary `.IDFW` / `.IDFH` codec — pending (see Roadmap + [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md))
- [ ] Live-device protocol — pending codec

**Data persistence:**
- [x] SQLite database (`seismo_relay.db`) with `events`, `monitor_log`, `ach_sessions` tables
- [x] Per-row `device_family` column ("series3" / "series4") for clean UI / unit-of-measurement dispatch (v0.19.0+)
- [x] Deduplication by `(serial, timestamp)` — natural key handles post-erase counter resets
- [x] UPSERT on re-import refreshes every device-authoritative field (peaks, project, sample_rate); preserves operator review state (`false_trigger`)
- [x] Post-erase key-reuse detection (tracks high-water mark in `ach_state.json`)

**REST API:**
- [x] Live device endpoints with in-memory caching (`_LiveCache`)
- [x] Cache statistics (`/cache/stats`) and manual invalidation (`/cache/device`)
- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH)
- [x] Call Home config read/write endpoints
- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`)
- [x] Import endpoints for both device families (`/db/import/blastware_file`, `/db/import/idf_file`)

**File output (v0.7+, byte-perfect as of v0.14.3):**
- [x] Blastware-compatible `.AB0` / `.G10` file generation (waveform + metadata)
- [x] Multi-channel waveform decode from SUB 5A bulk stream
- [x] Second-resolution timestamp encoding in Blastware filename
- [x] **Byte-perfect against BW reference captures** (verified across 2-sec / 3-sec / 10-sec event durations, both event 0 and event N continuation events)
- [x] STRT-bounded chunk walk + correct event-N probe counter + partial DLE stuffing of `0x10` in 5A params (the four fixes that landed in v0.14.0–v0.14.3)

**Capture tools:**
- [x] Serial-to-TCP bridge with raw BW/S3 capture (s3_bridge.py, defaults to auto-capture)
- [x] GUI bridge with raw capture checkboxes (gui_bridge.py)
- [x] ACH inbound server with bidirectional capture (ach_server.py saves raw_tx + raw_rx)
- [x] Transparent TCP MITM proxy for live BW session capture (ach_mitm.py)

**Analysis tools:**
- [x] s3_analyzer.py — session parser, frame differ, Claude export
- [x] gui_analyzer.py — standalone analyzer GUI
- [x] frame_db.py — SQLite frame database for capture analysis

**seismo_lab.py GUI:**
- [x] Bridge tab — Serial/TCP mode selector with raw capture options
- [x] Analyzer tab — BW/S3 capture playback and differencing
- [x] Download tab — Live wire-byte capture during event download
- [x] Console tab — Logging and diagnostics

## Roadmap (Future)

### Strategic direction — where this is going

seismo-relay is being built as a **suite of cooperating components**
that together replace and improve on Blastware's role.  Three logical
tiers:

1. **SFM** (device-side) — owns the active connection to a physical
   unit.  Today: `minimateplus/`, `/device/*` HTTP endpoints,
   `seismo_lab.py`.  Future: live Thor / Micromate support.
2. **SDM** (data-side) — owns the database, waveform store, ingest
   pipelines, and the read-API that Terra-View consumes.  Today this
   code lives under `sfm/` for historical reasons; the role has
   migrated and the eventual rename is on the long-tail cleanup list.
3. **Codec library** — pure data-interpretation: `minimateplus/*_codec.py`,
   `bw_ascii_report.py`, `micromate/idf_*.py`.  Used by both SFM and
   SDM, depends on neither.

Terra-View is downstream of SDM for fleet listings, event detail, etc.
The long-term vision adds a **second link** from Terra-View → SFM for
direct device interaction (see below).

The codec work in this repo isn't trying to replace BW's network
layer — BW's ACH file forwarding and Thor's IDF call-home are
battle-tested.  The value is in the receiving and processing side: turn
the stream of binary+ASCII pairs into something users can search,
filter, alert on, and report from.

### Terra-View ↔ SFM device control (the long-term vision)

Today Terra-View only reads from SDM (event listings, dashboards,
project reports).  When a unit goes missing — operator notices in the
Terra-View dashboard — there's no way to *do* anything from the UI.
The path of least resistance is to RDP into a Windows box and open
Blastware, which defeats the purpose of having Terra-View.

Target experience:
- Operator notices a unit in Terra-View dashboard hasn't called in.
- Clicks unit detail → "Connect to Device" button.
- Terra-View opens an embedded view (modal or side-panel) that talks
  to SFM's `/device/*` endpoints over the network.
- Live view: device clock, battery, memory, current monitor status.
- Actions: start/stop monitoring, push compliance config changes, pull
  fresh events, run a sensor self-check, change call-home settings.
- Audit log: every connect / action recorded in SDM for the unit
  history.

Implementation steps (concrete):
- [ ] **SFM authentication & authorization layer.**  Today `/device/*`
      endpoints are unauthenticated — anyone on the network can call
      them.  Need at minimum a token-based auth, ideally with a "who
      can connect to which units" mapping.  Hard prerequisite for
      letting Terra-View users into the control surface.
- [ ] **Terra-View "Connect to Device" entry point** on the unit
      detail page.  Renders only when unit has connection info on file
      and the user has permission.
- [ ] **Embedded live-monitor view** in Terra-View — equivalent to
      `seismo_lab.py`'s Bridge tab, but in the browser.  Polls SFM's
      `/device/monitor/status` on an interval; sends start/stop via
      `/device/monitor/{start,stop}`.
- [ ] **Action history** — every connect / push / action call records
      a row in `unit_history`, viewable on the unit detail page.
- [ ] **Series IV live-device support in SFM** — currently `/device/*`
      only supports MiniMate Plus.  Blocks "Connect to Device" for
      Thor units until done.  Depends on Thor wire-protocol capture
      and a `micromate/` parallel of the `minimateplus/` modules.

### High-impact (unblocks product features)

- [ ] **Series III waveform body codec reverse-engineering.**  The 5A bulk-stream body is some kind of compressed/encoded format (not raw int16 LE as previously assumed — see §7.6.1 retraction in `docs/instantel_protocol_reference.md`).  Structural framing is ~50% decoded on branch `claude/codec-re-cBGNe` (tagged-block walker, segment counters); per-byte sample mapping is still open.  Until this lands, the in-app waveform viewer renders garbage and BW-import peak values fall back to `_peaks_from_samples()` saturation noise.  Workaround: pair every BW-imported event with its `_ASCII.TXT` so the device-authoritative peaks land in the DB regardless of codec.
- [ ] **Series IV (Thor IDF) binary codec reverse-engineering.**  `.IDFH` / `.IDFW` files are currently stored opaquely by `WaveformStore.save_imported_idf`, with all metadata sourced from the paired `.txt` sidecar.  This works because thor-watcher forwards both files together, but operators who haven't enabled Thor's TXT exporter get rows with NULL peaks.  Cracking the binary closes that gap and unlocks waveform display.  Starting-point reference at [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md) — two observed file signatures (1,012 newer-firmware files + 2 old files whose layout matches the Series III STRT-record format), suggested first-session plan (~2-4 hrs), 1,014 paired binary+txt files available as ground truth in `thor-watcher/example-data/`.  Code seam ready at `micromate/idf_file.py`.
- [ ] **In-app waveform viewer accuracy.**  Depends on Series III codec decode.  Plot.v1 JSON pipeline + viewer skeleton already exist; will start showing real waveforms automatically once `_decode_a5_waveform` produces correct samples.  Series IV waveforms come online when the IDF codec lands.
- [ ] **Series IV live-device support.**  Once the IDF binary is decoded, extend `micromate/` with `transport.py` / `framing.py` / `protocol.py` / `client.py` mirroring the `minimateplus/` package layout — depends on capturing Thor's wire protocol (TCP / RS-232 captures TBD).
- [ ] **Terra-view integration** — seismo-relay router, unit detail page, VISON-style event listing.
- [ ] **Vibration summary reports** — highest legit PPV per project → Word doc (false-trigger filtering first).

### BW ASCII report parser enhancements (built in v0.16.0)

- [ ] **PPV field misses on certain TXT formats.**  Discovered 2026-05-22 during the histogram-codec backfill validation: a handful of events (5 in prod) have a `bw_report` block where `peaks.{tran,vert,long}.ppv_ips` and `peaks.vector_sum.ips` are all `None`, despite the parser correctly extracting every OTHER field for the same channels (zc_freq_hz, time_of_peak_s, peak_accel_g, peak_disp_in).  Symptom on the DB side: `peak_vector_sum=0` after a `--force` backfill that overlays from the parsed bw_report dict.  Affected events on prod include `T190LD5Q.LK0W`, `T438L713.RY0W`, `K557L3YM.OE0W`.  Root cause likely a regex or format mismatch for the "PPV" header line in those specific firmware/event-type outputs.  Once fixed, re-forwarding the events from series3-watcher will re-populate the `bw_report` blocks correctly.
- [ ] **Histogram-specific structural fields.**  Current parser handles the shared fields (PPV, ZC Freq, sensor self-check, project) but silently drops histogram-only fields: `Histogram Start/Stop Time`, `Histogram Start/Stop Date`, `Number of Intervals`, `Interval Size`, per-channel `Peak Time` + `Peak Date` (absolute timestamps rather than the waveform's `Time of Peak` relative seconds).
- [ ] **Histogram interval bin-table parsing.**  Trailing 792-row table (per-interval Peak/Freq per channel + MicL) in histogram TXTs is unparsed.  Probably too big for the sidecar JSON; may want a separate `.histogram.h5` companion file.
- [ ] **`>100 Hz` value parsing.**  Histogram TXTs use `>100 Hz` for out-of-range ZC freq; current `_parse_number()` returns `None` for these (loses information).

### Ingestion gaps

- [ ] **MLG forwarding.**  `series3-watcher` forwards event binaries + their `_ASCII.TXT` reports, but skips `.MLG` per-unit monitor log files entirely.  Adding an `POST /db/import/mlg_file` endpoint + watcher scan path would populate `monitor_log` for non-ACH-routed units (coverage queries, "was this unit monitoring on date X" lookups).
- [ ] **0C-record raw bytes persistence in the sidecar.**  Currently on branch `claude/codec-re-cBGNe` as commit `a187124`; cherry-pick if useful as a standalone fix.  Preserves the 210-byte 0C record under `extensions.raw_records.waveform_record_b64` so future field-offset analysis (Peak Acceleration / Time of Peak / etc. — the fields BW computes client-side from samples) can run offline.

### Operational

- [ ] **`series3-watcher` file archive manager** — 90-day-old events moved to `<watch_folder>_archive/<year>/<month>/` subfolders.  Plan drafted in `claude/codec-re-cBGNe`'s plan-mode session; awaiting a 5-minute test on whether Blastware UI walks subfolders before any code lands (determines layout: in-place subfolders vs sibling archive).
- [ ] **Compliance config encoder** — build raw write payloads from a `ComplianceConfig` object.
- [ ] **Modem manager** — push RV50/RV55 configs via Sierra Wireless API.
- [ ] **Call Home dial_string write support** (requires DLE escaping for embedded control characters).
- [ ] **Histogram mode recording support** (5A stream analysis for mode 0x03 — separate from histogram ASCII parsing above).

### Test coverage

- [ ] Verify 30-sec event download — body may exceed `0xFFFF` and force the device into a different `end_key` encoding (none of the 2/3/10-sec test cases hit this boundary).
- [ ] Histogram mode (0x03) write via SFM — confirmed working for Single Shot / Continuous / Histogram+Continuous; Histogram (0x03) needs a live test from a non-Histogram starting state.

### Lower-priority cleanups

- [ ] Compliance write anchor-9 cleanup — when changing recording_mode via SFM, a spurious `0x10` may persist after Histogram→other mode transitions.  Doesn't affect device operation but differs from BW's byte-perfect output.
- [ ] Locate "Sensor Check" byte in compliance config (need capture with Disabled vs Before-monitoring).
- [ ] Call Home — map time slots 3/4 offsets; confirm `modem_power_relay_enabled`.
- [ ] RV55 DCD/DTR — newer RV55 firmware doesn't assert DCD by default; units don't resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred).
- [ ] **NULL-timestamp duplicate-row dedup.**  A small handful of events (2 known on prod as of 2026-05-22) have `events.timestamp IS NULL` because the codec couldn't extract a timestamp from the binary footer.  The `UNIQUE(serial, timestamp)` constraint doesn't fire on `NULL` (SQL semantics: `NULL ≠ NULL`), so every `--force` backfill INSERTs a new row instead of UPSERTing the existing one.  Cleanup: a one-shot SQL query that keeps only the newest row per `(serial, blastware_filename)` and deletes the rest.  Longer-term: extend the unique key to `(serial, COALESCE(timestamp, blastware_filename))` or reject inserts with NULL timestamp.
- [ ] **Histogram body sub-format with `byte[5] != 0`.**  ~3 events on prod (`T190LD5Q.LD0H`, `O121L4L1.GU0H`) use a histogram body my walker doesn't recognize — the first block has `byte[5] = 0x01` or `0x07` instead of `0x00`, and the entire body lacks the `1e 0a 00 00` tail signature.  Codec returns 0 valid blocks; their DB PVS comes from the bw_report ASCII overlay (which BW computed from the same binary, so the DB columns are correct).  Only the `.h5` waveform plot is empty.  Cracking the sub-format would unlock the plot.  Needs binary+ASCII pairs from a few `byte[5]!=0` events; same RE approach as the K558 case.