seismo-relay/README.md

# seismo-relay  `v0.16.1`

A ground-up replacement for **Blastware** — Instantel's aging Windows-only
software for managing MiniMate Plus seismographs.

Built in Python. Runs on Windows, Linux, or macOS. Connects to instruments
over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).

> **Status:** Active development. Full read + write + erase + monitoring
> pipeline working end-to-end over TCP/cellular. ACH Auto Call Home server
> handles inbound unit connections, downloads events, and persists everything
> to a SQLite database. SFM REST API exposes device control and DB queries.
> **As of v0.14.3 (2026-05-05): SUB 5A bulk waveform protocol is verified
> byte-perfect against Blastware captures across 2-sec, 3-sec, and 10-sec
> events.** Generated `.G10` / `.AB0` files open cleanly in Blastware with
> full Event Reports, frequency analysis, and waveform plots.
> **v0.16.0 (2026-05-11)** adds BW ASCII report ingestion to
> `/db/import/blastware_file` — paired with **series3-watcher v1.5.0**,
> every Blastware ACH event lands in SeismoDb with device-authoritative
> peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data,
> without depending on the still-undecoded waveform body codec.
> See [CHANGELOG.md](CHANGELOG.md) for full version history.

---

## What's in here

```
seismo-relay/
├── seismo_lab.py              ← Main GUI (Bridge + Analyzer + Download + Console tabs)
│
├── minimateplus/              ← MiniMate Plus client library
│   ├── transport.py           ←   SerialTransport, TcpTransport, SocketTransport
│   ├── protocol.py            ←   DLE frame layer, SUB command dispatch
│   ├── client.py              ←   High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …)
│   ├── framing.py             ←   Frame builders, DLE codec, S3FrameParser
│   ├── models.py              ←   DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig, …
│   └── blastware_file.py      ←   Write events to Blastware-compatible .AB0 files
│
├── sfm/                       ← SFM REST API server (FastAPI, port 8200)
│   ├── server.py              ←   Live device endpoints + DB query endpoints + caching
│   ├── database.py            ←   SeismoDb — SQLite persistence (events, monitor_log, ach_sessions, sessions table)
│   └── sfm_webapp.html        ←   Embedded web UI with Call Home config tab
│
├── bridges/
│   ├── ach_server.py          ←   Inbound ACH call-home server (main production server)
│   ├── ach_mitm.py            ←   Transparent MITM proxy for capturing BW sessions
│   ├── s3-bridge/             ←   RS-232 serial bridge (capture tool)
│   ├── tcp_serial_bridge.py   ←   Local TCP↔serial bridge (bench testing)
│   ├── gui_bridge.py          ←   Standalone bridge GUI with raw capture checkboxes
│   └── raw_capture.py         ←   Simple raw capture tool
│
├── parsers/
│   ├── s3_analyzer.py         ←   Session parser, differ, Claude export
│   ├── gui_analyzer.py        ←   Standalone analyzer GUI
│   └── frame_db.py            ←   SQLite frame database
│
└── docs/
    └── instantel_protocol_reference.md  ← Reverse-engineered protocol spec
```

---

## Quick start

### ACH inbound server (production)

Listens for inbound unit call-homes, downloads all new events and monitor log
entries, and writes everything to `bridges/captures/seismo_relay.db`.

```bash
python bridges/ach_server.py --port 12345 --output bridges/captures/
```

Point the unit's ACEmanager **Remote Host** to this machine's IP and **Remote Port** to `12345`.

Options:
```
--port N            Listen port (default 12345)
--output DIR        Capture directory (default bridges/captures/)
--allow-ip IP       Allowlist an IP (repeat for multiple; default: accept all)
--max-events N      Safety cap for first run (default: unlimited)
--clear-after-download  Erase device memory after successful download
--verbose           Debug logging
```

### SFM REST server

Exposes device control and DB queries as a REST API. Proxied by terra-view.

```bash
python sfm/server.py               # default: 0.0.0.0:8200
python -m uvicorn sfm.server:app --host 0.0.0.0 --port 8200 --reload
```

Open `http://localhost:8200` for the embedded web UI, or `http://localhost:8200/docs`
for the interactive API docs.

### Seismo Lab GUI

```bash
python seismo_lab.py
```

---

## SFM REST API

### Live device endpoints

Each call dials the device, does its work, and closes the connection. TCP
connections are retried once on `ProtocolError` to handle cold-boot timing.

**In-memory caching** — frequently-polled endpoints avoid redundant TCP round-trips
via a thread-safe `_LiveCache` (plain Python dict + `threading.Lock`):

| Method | URL | Cache Strategy |
|--------|-----|---|
| `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` |
| `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected |
| `GET` | `/device/event/{idx}/waveform` | Permanent per event index |
| `GET` | `/device/monitor/status` | 30-second TTL; invalidated by monitor start/stop |
| `GET` | `/device/call_home` | Fresh read from device (not cached) |
| `POST` | `/device/connect` | — |
| `POST` | `/device/config` | Writes compliance config; invalidates info + events cache |
| `POST` | `/device/config/project` | Patches project/client/operator/sensor_location strings |
| `POST` | `/device/monitor/start` | Sends SUB 0x96; immediately evicts status cache |
| `POST` | `/device/monitor/stop` | Sends SUB 0x97; immediately evicts status cache |
| `POST` | `/device/call_home` | Reads, patches specified fields, writes back to device |

**Cache bypass** — All cached endpoints accept `?force=true` to skip the cache and
force a fresh read from the device.

**Cache stats** — `GET /cache/stats` returns hit/miss counts and TTL info; `DELETE /cache/device`
clears the device cache immediately.

Transport query params (supply one set):
```
Serial:  ?port=COM5&baud=38400
TCP:     ?host=1.2.3.4&tcp_port=12345
```

### DB read endpoints

Query the SQLite database written by `ach_server.py`. All read-only except
`PATCH /db/events/{id}/false_trigger`.

| Method | URL | Description |
|--------|-----|-------------|
| `GET` | `/db/units` | All known serials with summary stats |
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger) |
| `GET` | `/db/monitor_log` | Monitoring intervals |
| `GET` | `/db/sessions` | ACH call-home session history |
| `PATCH` | `/db/events/{id}/false_trigger?value=true` | Flag / unflag false triggers |

---

## minimateplus library

```python
from minimateplus import MiniMateClient
from minimateplus.transport import TcpTransport

# Serial
client = MiniMateClient(port="COM5")

# TCP (cellular modem)
client = MiniMateClient(transport=TcpTransport("1.2.3.4", 12345), timeout=30.0)

with client:
    # Read
    info     = client.connect()               # DeviceInfo — serial, firmware, compliance config
    count    = client.count_events()          # Number of stored events
    keys     = client.list_event_keys()       # Fast browse walk — event keys only, no download
    events   = client.get_events()            # Full download: headers + peaks + metadata
    monitor  = client.get_monitor_status()    # Battery, memory, is_monitoring flag
    log      = client.get_monitor_log_entries()  # Monitoring intervals (partial 0x2C records)
    ach_cfg  = client.get_call_home_config()  # Auto Call Home settings (SUB 0x2C)

    # Write
    client.apply_config(
        sample_rate=1024,
        recording_mode="Continuous",         # Single Shot / Continuous / Histogram / Histogram+Continuous
        histogram_interval_sec=15,            # 2, 5, 15, 60, 300, 900
        trigger_level_geo=0.5,
        geo_range="Normal",                   # Normal (10.000 in/s) / Sensitive (1.25 in/s)
        project="Bridge Inspection 2026",
        client_name="City of Portland",
        operator="B. Harrison",
    )
    
    client.set_call_home_config(
        auto_call_home_enabled=True,
        after_event_recorded=True,
        at_specified_times=True,
        time1_hour=18, time1_min=30,          # 6:30 PM
        time2_hour=6, time2_min=0,            # 6:00 AM
    )

    # Control
    client.start_monitoring()                # SUB 0x96
    client.stop_monitoring()                 # SUB 0x97
    client.delete_all_events()              # Erase all (SUB 0xA3 → 0x1C → 0x06 → 0xA2)
```

`get_events()` runs the full per-event sequence:
`1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)`.
SUB 5A bulk stream walks chunks bounded by the `end_offset` extracted from
the STRT record at byte 17 of the probe response — no over-reading, no
chunk-count cap. Project / client / operator / sensor location strings come
from the dedicated metadata pages at counter `0x1002` and `0x1004`,
read once per session (they reflect the compliance setup at session start,
not per individual event).

---

## Database

`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode) using the
`SeismoDb` persistence layer. Four tables, all unit-keyed by serial number:

| Table | Key | Contents |
|-------|-----|----------|
| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds |
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag |
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
| `events.false_trigger` | Boolean flag | PATCH endpoint to mark/unmark false triggers for review |

Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs never
produce duplicate rows. Post-erase key reuse is handled automatically via the
high-water mark in `ach_state.json`. Key-based state tracking allows correct
handling of device erasures (external or post-download).

---

## Connecting over cellular (RV50 / RV55)

Field units connect via Sierra Wireless RV50 or RV55 cellular modems.

### Required ACEmanager settings

| Setting | Value | Why |
|---------|-------|-----|
| Configure Serial Port | `38400,8N1` | Must match MiniMate baud rate |
| Flow Control | `None` | Hardware FC blocks TX if pins unconnected |
| **Quiet Mode** | **Enable** | **Critical** — disabled injects `RING`/`CONNECT` onto serial, corrupting the S3 handshake |
| Data Forwarding Timeout | `1` (= 0.1 s) | Lower latency |
| TCP Connect Response Delay | `0` | Non-zero silently drops the first POLL frame |
| TCP Idle Timeout | `2` (minutes) | Prevents premature disconnect |
| DB9 Serial Echo | `Disable` | Echo corrupts the data stream |

---

## Protocol quick-reference

| Term | Value | Meaning |
|------|-------|---------|
| DLE | `0x10` | Data Link Escape |
| STX | `0x02` | Start of frame |
| ETX | `0x03` | End of frame |
| ACK | `0x41` | Frame-start marker sent before every BW frame |
| DLE stuffing | `10 10` on wire | Literal `0x10` in payload |

**Response SUB rule:** `response_SUB = 0xFF - request_SUB` (no exceptions)

Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/instantel_protocol_reference.md)

---

## Compliance Config Features

The REST API and web UI expose full control over device compliance settings:

- **Recording Mode** (Single Shot / Continuous / Histogram / Histogram+Continuous)
- **Sample Rate** (1024 / 2048 / 4096 sps)
- **Record Time** (float, seconds)
- **Histogram Interval** (2s, 5s, 15s, 1m, 5m, 15m) — when recording mode includes histogram
- **Geo Trigger Levels** (float, in/s per channel)
- **Geo Maximum Range** (Normal 10.000 in/s / Sensitive 1.250 in/s per channel)
- **Project / Client / Operator / Sensor Location** (ASCII strings)

Auto Call Home config:
- **Auto Call Home Enable** (bool)
- **Dial String** (read-only; 40-byte ASCII)
- **Trigger on Event** (bool)
- **Scheduled Call-Ins** (two time slots with HH:MM each)
- **Retry Settings** (count, delay, connection timeout, warm-up time)

---

## Requirements

```bash
pip install pyserial fastapi uvicorn
```

Python 3.10+. Tkinter is included with the standard Python installer on
Windows (check "tcl/tk and IDLE" during install).

---

## Virtual COM ports (bridge capture)

```
Blastware → COM4 (virtual) ↔ s3_bridge.py ↔ COM5 (physical) → MiniMate Plus
```

Use **com0com** or **VSPD** to create the virtual COM pair on Windows.

---

## Key Features

**Device support:**
- [x] Full read/write/erase pipelines
- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings)
- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries)
- [x] Monitor control (start/stop, status polling, battery/memory)
- [x] Monitor log entries (continuous monitoring intervals without full waveform download)

**Data persistence:**
- [x] SQLite database (`seismo_relay.db`) with 4 tables: ach_sessions, events, monitor_log, plus false_trigger flag
- [x] Deduplication by waveform key (handles re-runs and repeat call-homes)
- [x] Post-erase key-reuse detection (tracks high-water mark)
- [x] Session state (`ach_state.json`) with downloaded keys and max key

**REST API:**
- [x] Live device endpoints with in-memory caching (`_LiveCache`)
- [x] Cache statistics (`/cache/stats`) and manual invalidation (`/cache/device`)
- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH)
- [x] Call Home config read/write endpoints
- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`)

**File output (v0.7+, byte-perfect as of v0.14.3):**
- [x] Blastware-compatible `.AB0` / `.G10` file generation (waveform + metadata)
- [x] Multi-channel waveform decode from SUB 5A bulk stream
- [x] Second-resolution timestamp encoding in Blastware filename
- [x] **Byte-perfect against BW reference captures** (verified across 2-sec / 3-sec / 10-sec event durations, both event 0 and event N continuation events)
- [x] STRT-bounded chunk walk + correct event-N probe counter + partial DLE stuffing of `0x10` in 5A params (the four fixes that landed in v0.14.0–v0.14.3)

**Capture tools:**
- [x] Serial-to-TCP bridge with raw BW/S3 capture (s3_bridge.py, defaults to auto-capture)
- [x] GUI bridge with raw capture checkboxes (gui_bridge.py)
- [x] ACH inbound server with bidirectional capture (ach_server.py saves raw_tx + raw_rx)
- [x] Transparent TCP MITM proxy for live BW session capture (ach_mitm.py)

**Analysis tools:**
- [x] s3_analyzer.py — session parser, frame differ, Claude export
- [x] gui_analyzer.py — standalone analyzer GUI
- [x] frame_db.py — SQLite frame database for capture analysis

**seismo_lab.py GUI:**
- [x] Bridge tab — Serial/TCP mode selector with raw capture options
- [x] Analyzer tab — BW/S3 capture playback and differencing
- [x] Download tab — Live wire-byte capture during event download
- [x] Console tab — Logging and diagnostics

## Roadmap (Future)

### High-impact (unblocks product features)

- [ ] **Waveform body codec reverse-engineering.**  The 5A bulk-stream body is some kind of compressed/encoded format (not raw int16 LE as previously assumed — see §7.6.1 retraction in `docs/instantel_protocol_reference.md`).  Structural framing is ~50% decoded on branch `claude/codec-re-cBGNe` (tagged-block walker, segment counters); per-byte sample mapping is still open.  Until this lands, the in-app waveform viewer renders garbage and BW-import peak values fall back to `_peaks_from_samples()` saturation noise.  Workaround: pair every BW-imported event with its `_ASCII.TXT` so the device-authoritative peaks land in the DB regardless of codec.
- [ ] **In-app waveform viewer accuracy.**  Depends on codec decode.  Plot.v1 JSON pipeline + viewer skeleton already exist; will start showing real waveforms automatically once `_decode_a5_waveform` produces correct samples.
- [ ] **Terra-view integration** — seismo-relay router, unit detail page, VISON-style event listing.
- [ ] **Vibration summary reports** — highest legit PPV per project → Word doc (false-trigger filtering first).

### BW ASCII report parser enhancements (built in v0.16.0)

- [ ] **Histogram-specific structural fields.**  Current parser handles the shared fields (PPV, ZC Freq, sensor self-check, project) but silently drops histogram-only fields: `Histogram Start/Stop Time`, `Histogram Start/Stop Date`, `Number of Intervals`, `Interval Size`, per-channel `Peak Time` + `Peak Date` (absolute timestamps rather than the waveform's `Time of Peak` relative seconds).
- [ ] **Histogram interval bin-table parsing.**  Trailing 792-row table (per-interval Peak/Freq per channel + MicL) in histogram TXTs is unparsed.  Probably too big for the sidecar JSON; may want a separate `.histogram.h5` companion file.
- [ ] **`>100 Hz` value parsing.**  Histogram TXTs use `>100 Hz` for out-of-range ZC freq; current `_parse_number()` returns `None` for these (loses information).

### Ingestion gaps

- [ ] **MLG forwarding.**  `series3-watcher` forwards event binaries + their `_ASCII.TXT` reports, but skips `.MLG` per-unit monitor log files entirely.  Adding an `POST /db/import/mlg_file` endpoint + watcher scan path would populate `monitor_log` for non-ACH-routed units (coverage queries, "was this unit monitoring on date X" lookups).
- [ ] **0C-record raw bytes persistence in the sidecar.**  Currently on branch `claude/codec-re-cBGNe` as commit `a187124`; cherry-pick if useful as a standalone fix.  Preserves the 210-byte 0C record under `extensions.raw_records.waveform_record_b64` so future field-offset analysis (Peak Acceleration / Time of Peak / etc. — the fields BW computes client-side from samples) can run offline.

### Operational

- [ ] **`series3-watcher` file archive manager** — 90-day-old events moved to `<watch_folder>_archive/<year>/<month>/` subfolders.  Plan drafted in `claude/codec-re-cBGNe`'s plan-mode session; awaiting a 5-minute test on whether Blastware UI walks subfolders before any code lands (determines layout: in-place subfolders vs sibling archive).
- [ ] **Compliance config encoder** — build raw write payloads from a `ComplianceConfig` object.
- [ ] **Modem manager** — push RV50/RV55 configs via Sierra Wireless API.
- [ ] **Call Home dial_string write support** (requires DLE escaping for embedded control characters).
- [ ] **Histogram mode recording support** (5A stream analysis for mode 0x03 — separate from histogram ASCII parsing above).

### Test coverage

- [ ] Verify 30-sec event download — body may exceed `0xFFFF` and force the device into a different `end_key` encoding (none of the 2/3/10-sec test cases hit this boundary).
- [ ] Histogram mode (0x03) write via SFM — confirmed working for Single Shot / Continuous / Histogram+Continuous; Histogram (0x03) needs a live test from a non-Histogram starting state.

### Lower-priority cleanups

- [ ] Compliance write anchor-9 cleanup — when changing recording_mode via SFM, a spurious `0x10` may persist after Histogram→other mode transitions.  Doesn't affect device operation but differs from BW's byte-perfect output.
- [ ] Locate "Sensor Check" byte in compliance config (need capture with Disabled vs Before-monitoring).
- [ ] Call Home — map time slots 3/4 offsets; confirm `modem_power_relay_enabled`.
- [ ] RV55 DCD/DTR — newer RV55 firmware doesn't assert DCD by default; units don't resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred).