feat: implement raw ADC waveform decoding and download functionality
- Added `_decode_a5_waveform()` to parse SUB 5A frames into per-channel time-series data. - Introduced `download_waveform(event)` method in `MiniMateClient` to fetch full waveform data. - Updated `Event` model to include new fields: `total_samples`, `pretrig_samples`, `rectime_seconds`, and `_waveform_key`. - Enhanced documentation in `CHANGELOG.md` and `instantel_protocol_reference.md` to reflect new features and confirmed protocol details.
This commit is contained in:
Brian Harrison
260
CLAUDE.md
Normal file
260
CLAUDE.md
Normal file
@@ -0,0 +1,260 @@
|
||||
# CLAUDE.md — seismo-relay
|
||||
|
||||
Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
|
||||
managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
|
||||
(Sierra Wireless RV50 / RV55). Current version: **v0.6.0**.
|
||||
|
||||
---
|
||||
|
||||
## Project layout
|
||||
|
||||
```
|
||||
minimateplus/ ← Python client library (primary focus)
|
||||
transport.py ← SerialTransport, TcpTransport
|
||||
framing.py ← DLE codec, frame builders, S3FrameParser
|
||||
protocol.py ← MiniMateProtocol — wire-level read/write methods
|
||||
client.py ← MiniMateClient — high-level API (connect, get_events, …)
|
||||
models.py ← DeviceInfo, EventRecord, ComplianceConfig, …
|
||||
|
||||
sfm/server.py ← FastAPI REST server exposing device data over HTTP
|
||||
seismo_lab.py ← Tkinter GUI (Bridge + Analyzer + Console tabs)
|
||||
docs/
|
||||
instantel_protocol_reference.md ← reverse-engineered protocol spec ("the Rosetta Stone")
|
||||
CHANGELOG.md ← version history
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Current implementation state (v0.6.0)
|
||||
|
||||
Full read pipeline working end-to-end over TCP/cellular:
|
||||
|
||||
| Step | SUB | Status |
|
||||
|---|---|---|
|
||||
| POLL / startup handshake | 5B | ✅ |
|
||||
| Serial number | 15 | ✅ |
|
||||
| Full config (firmware, calibration date, etc.) | FE | ✅ |
|
||||
| Compliance config (record time, sample rate, geo thresholds) | 1A | ✅ |
|
||||
| Event index | 08 | ✅ |
|
||||
| Event header / first key | 1E | ✅ |
|
||||
| Waveform header | 0A | ✅ |
|
||||
| Waveform record (peaks, timestamp, project) | 0C | ✅ |
|
||||
| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ **new v0.6.0** |
|
||||
| Event advance / next key | 1F | ✅ |
|
||||
| Write commands (push config to device) | 68–83 | ❌ not yet implemented |
|
||||
|
||||
`get_events()` sequence per event: `1E → 0A → 0C → 5A → 1F`
|
||||
|
||||
---
|
||||
|
||||
## Protocol fundamentals
|
||||
|
||||
### DLE framing
|
||||
|
||||
```
|
||||
BW→S3 (our requests): [ACK=0x41] [STX=0x02] [stuffed payload+chk] [ETX=0x03]
|
||||
S3→BW (device replies): [DLE=0x10] [STX=0x02] [stuffed payload+chk] [bare ETX=0x03]
|
||||
```
|
||||
|
||||
- **DLE stuffing rule:** any literal `0x10` byte in the payload is doubled on the wire
|
||||
(`0x10` → `0x10 0x10`). This includes the checksum byte.
|
||||
- **Inner-frame terminators:** large S3 responses (A4, E5) contain embedded sub-frames
|
||||
using `DLE+ETX` as inner terminators. The outer parser treats `DLE+ETX` inside a frame
|
||||
as literal data — the bare ETX is the ONLY real frame terminator.
|
||||
- **Response SUB rule:** `response_SUB = 0xFF - request_SUB`
|
||||
(one known exception: SUB `1C` → response `6E`, not `0xE3`)
|
||||
- **Two-step read pattern:** every read command is sent twice — probe step (`offset=0x00`,
|
||||
get length) then data step (`offset=DATA_LENGTH`, get payload). All data lengths are
|
||||
hardcoded constants, not read from the probe response.
|
||||
|
||||
### De-stuffed payload header
|
||||
|
||||
```
|
||||
BW→S3 (request):
|
||||
[0] CMD 0x10
|
||||
[1] flags 0x00
|
||||
[2] SUB command byte
|
||||
[3] 0x00 always zero
|
||||
[4] 0x00 always zero
|
||||
[5] OFFSET 0x00 for probe step; DATA_LENGTH for data step
|
||||
[6-15] params (key, token, etc. — see helpers in framing.py)
|
||||
|
||||
S3→BW (response):
|
||||
[0] CMD 0x00
|
||||
[1] flags 0x10
|
||||
[2] SUB response sub byte
|
||||
[3] PAGE_HI
|
||||
[4] PAGE_LO
|
||||
[5+] data
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Critical protocol gotchas (hard-won — do not re-derive)
|
||||
|
||||
### SUB 5A — bulk waveform stream — NON-STANDARD frame format
|
||||
|
||||
**Always use `build_5a_frame()` for SUB 5A. Never use `build_bw_frame()` for SUB 5A.**
|
||||
|
||||
`build_bw_frame` produces WRONG output for 5A for two reasons:
|
||||
|
||||
1. **`offset_hi = 0x10` must NOT be DLE-stuffed.** Blastware sends the offset field raw.
|
||||
`build_bw_frame` would stuff it to `10 10` on the wire — the device silently ignores
|
||||
the frame. `build_5a_frame` writes it as a bare `10`.
|
||||
|
||||
2. **DLE-aware checksum.** When computing the checksum, `10 XX` pairs in the stuffed
|
||||
section contribute only `XX` to the running sum; lone bytes contribute normally. This
|
||||
differs from the standard SUM8-of-destuffed-payload that all other commands use.
|
||||
|
||||
Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26
|
||||
BW TX capture. All 10 frames verified.
|
||||
|
||||
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
|
||||
|
||||
`bulk_waveform_params()` returns 11 bytes (extra trailing `0x00`). The 11th byte was
|
||||
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
|
||||
Do not swap them.
|
||||
|
||||
### SUB 5A — event-time metadata lives in A5 frame 7
|
||||
|
||||
The bulk stream sends 9+ A5 response frames. Frame 7 (0-indexed) contains the compliance
|
||||
setup as it existed when the event was recorded:
|
||||
|
||||
```
|
||||
"Project:" → project description
|
||||
"Client:" → client name ← NOT in the 0C record
|
||||
"User Name:" → operator name ← NOT in the 0C record
|
||||
"Seis Loc:" → sensor location ← NOT in the 0C record
|
||||
"Extended Notes"→ notes
|
||||
```
|
||||
|
||||
These strings are **NOT** present in the 210-byte SUB 0C waveform record. They reflect
|
||||
the setup at record time, not the current device config — this is why we fetch them from
|
||||
5A instead of backfilling from the current compliance config.
|
||||
|
||||
`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears,
|
||||
then sends the termination frame.
|
||||
|
||||
### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce)
|
||||
|
||||
`read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where:
|
||||
- Frame A is a probe (no `recv_one` needed — device ACKs but returns no data page)
|
||||
- Frames B, C, D each need a `recv_one` to collect the response
|
||||
|
||||
**There must be NO extra `self._send(...)` call before the B/C/D recv loop without a
|
||||
matching `recv_one()`.** An orphaned send shifts all receives one step behind, leaving
|
||||
frame D's channel block (trigger_level_geo, alarm_level_geo, max_range_geo) unread and
|
||||
producing only ~1071 bytes instead of ~2126.
|
||||
|
||||
### SUB 1A — anchor search range
|
||||
|
||||
`_decode_compliance_config_into()` locates sample_rate and record_time via the anchor
|
||||
`b'\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00'`. Search range is `cfg[0:150]`.
|
||||
|
||||
Do not narrow this to `cfg[40:100]` — the old range was only accidentally correct because
|
||||
the orphaned-send bug was prepending a 44-byte spurious header, pushing the anchor from
|
||||
its real position (cfg[11]) into the 40–100 window.
|
||||
|
||||
### Sample rate and DLE jitter in cfg data
|
||||
|
||||
Sample rate 4096 (`0x1000`) causes DLE jitter: the frame carries `10 10 00` on the wire,
|
||||
which unstuffs to `10 00` — 2 bytes instead of 3. This makes frame C 1 byte shorter and
|
||||
shifts all subsequent absolute offsets by −1. The anchor approach is immune to this.
|
||||
Do NOT use fixed absolute offsets for sample_rate or record_time.
|
||||
|
||||
### TCP / cellular transport
|
||||
|
||||
- Protocol bytes over TCP are bit-for-bit identical to RS-232. No wrapping.
|
||||
- The modem (RV50/RV55) forwards bytes with up to ~1s buffering. `TcpTransport` uses
|
||||
`read_until_idle(idle_gap=1.5s)` to drain the buffer completely before parsing.
|
||||
- Cold-boot: unit sends the 16-byte ASCII string `"Operating System"` before entering
|
||||
DLE-framed mode. The parser discards it (scans for DLE+STX).
|
||||
- RV50/RV55 sends `\r\nRING\r\n\r\nCONNECT\r\n` over TCP to the caller even with
|
||||
Quiet Mode enabled. Parser handles this — do not strip it manually before feeding to
|
||||
`S3FrameParser`.
|
||||
|
||||
### Required ACEmanager settings (Sierra Wireless RV50/RV55)
|
||||
|
||||
| Setting | Value | Why |
|
||||
|---|---|---|
|
||||
| Configure Serial Port | `38400,8N1` | Must match MiniMate baud |
|
||||
| Flow Control | `None` | Hardware FC blocks TX if pins unconnected |
|
||||
| **Quiet Mode** | **Enable** | **Critical.** Disabled injects `RING`/`CONNECT` onto serial, corrupting S3 handshake |
|
||||
| Data Forwarding Timeout | `1` (= 0.1 s) | Lower latency |
|
||||
| TCP Connect Response Delay | `0` | Non-zero silently drops first POLL frame |
|
||||
| TCP Idle Timeout | `2` (minutes) | Prevents premature disconnect |
|
||||
| DB9 Serial Echo | `Disable` | Echo corrupts the data stream |
|
||||
|
||||
---
|
||||
|
||||
## Key confirmed field locations
|
||||
|
||||
### SUB FE — Full Config (166 destuffed bytes)
|
||||
|
||||
| Offset | Field | Type | Notes |
|
||||
|---|---|---|---|
|
||||
| 0x34 | firmware version string | ASCII | e.g. `"S338.17"` |
|
||||
| 0x56–0x57 | calibration year | uint16 BE | `0x07E9` = 2025 |
|
||||
| 0x0109 | aux trigger enabled | uint8 | `0x00` = off, `0x01` = on |
|
||||
|
||||
### SUB 1A — Compliance Config (~2126 bytes total after 4-frame sequence)
|
||||
|
||||
| Field | How to find it |
|
||||
|---|---|
|
||||
| sample_rate | uint16 BE at anchor − 2 |
|
||||
| record_time | float32 BE at anchor + 10 |
|
||||
| trigger_level_geo | float32 BE, located in channel block |
|
||||
| alarm_level_geo | float32 BE, adjacent to trigger_level_geo |
|
||||
| max_range_geo | float32 BE, adjacent to alarm_level_geo |
|
||||
| setup_name | ASCII, null-padded, in cfg body |
|
||||
| project / client / operator / sensor_location | ASCII, label-value pairs |
|
||||
|
||||
Anchor: `b'\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00'`, search `cfg[0:150]`
|
||||
|
||||
### SUB 0C — Waveform Record (210 bytes = data[11:11+0xD2])
|
||||
|
||||
| Offset | Field | Type |
|
||||
|---|---|---|
|
||||
| 0 | day | uint8 |
|
||||
| 1 | sub_code | uint8 (`0x10` = Waveform) |
|
||||
| 2 | month | uint8 |
|
||||
| 3–4 | year | uint16 BE |
|
||||
| 5 | unknown | uint8 (always 0) |
|
||||
| 6 | hour | uint8 |
|
||||
| 7 | minute | uint8 |
|
||||
| 8 | second | uint8 |
|
||||
| 87 | peak_vector_sum | float32 BE |
|
||||
| label+6 | PPV per channel | float32 BE (search for `"Tran"`, `"Vert"`, `"Long"`, `"MicL"`) |
|
||||
|
||||
PPV labels are NOT 4-byte aligned. The label-offset+6 approach is the only reliable method.
|
||||
|
||||
---
|
||||
|
||||
## SFM REST API (sfm/server.py)
|
||||
|
||||
```
|
||||
GET /device/info?port=COM5 ← serial
|
||||
GET /device/info?host=1.2.3.4&tcp_port=9034 ← cellular
|
||||
GET /device/events?host=1.2.3.4&tcp_port=9034&baud=38400
|
||||
GET /device/event?host=1.2.3.4&tcp_port=9034&index=0
|
||||
```
|
||||
|
||||
Server retries once on `ProtocolError` for TCP connections (handles cold-boot timing).
|
||||
|
||||
---
|
||||
|
||||
## Key wire captures (reference material)
|
||||
|
||||
| Capture | Location | Contents |
|
||||
|---|---|---|
|
||||
| 1-2-26 | `bridges/captures/1-2-26/` | SUB 5A BW TX frames — used to confirm 5A frame format, 11-byte params, DLE-aware checksum |
|
||||
| 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture |
|
||||
| 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence |
|
||||
|
||||
---
|
||||
|
||||
## What's next
|
||||
|
||||
- Write commands (SUBs 68–83) — push compliance config, channel config, trigger settings to device
|
||||
- ACH inbound server — accept call-home connections from field units
|
||||
- Modem manager — push RV50/RV55 configs via Sierra Wireless API
|
||||
Reference in New Issue
Block a user