diff --git a/CHANGELOG.md b/CHANGELOG.md index e98b2f2..42236a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,25 @@ All notable changes to seismo-relay are documented here. --- +## v0.12.6 — 2026-05-01 + +### Fixed + +- **`blastware_file.py` — waveform frame classification** — A5 frame classification for + waveform-only vs header-only frames now uses `frame.record_type` instead of frame index. + Only waveform frames (0x46) are written to the file body; metadata frames are skipped. + Fixes spurious data corruption from incorrectly classified frames. + +- **`s3_analyzer.py` — A5/5A frame naming** — Bulk waveform stream frames (SUB 5A response) + are now correctly labeled "A5" in analyzer output instead of being conflated with other + multi-frame responses (SUB A4, E5, etc.). + +- **`S3FrameParser` — frame terminator detection** — Corrected the bare ETX terminator + detection. Frame termination is now correctly identified by a standalone `ETX=0x03` byte, + not by the `DLE+ETX` sequence (which is part of the payload when it appears within a frame). + +--- + ## v0.13.2 — 2026-05-01 ### Fixed @@ -112,6 +131,13 @@ All notable changes to seismo-relay are documented here. ## v0.12.5 — 2026-04-21 +### Added + +- **`seismo_lab.py` — Download tab** — New fourth tab for live wire-byte capture during event + downloads. Captures both BW→device and device→S3 frames in real time, allowing inspection + of the 5A bulk stream chunk sequence and frame-by-frame analysis without needing a bridge + or MITM proxy. Files are saved with user-specified labels for easy tracking. + ### Changed - **`s3_bridge.py` — raw captures always-on by default** — `--raw-bw` and `--raw-s3` now @@ -123,6 +149,10 @@ All notable changes to seismo-relay are documented here. "S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names the files). Unchecking a box passes `--raw-bw ""` to explicitly disable capture. +- **`Bridge tab` — TCP mode added** — Serial/TCP radio toggle allows connection via cellular + modem (RV50/RV55) instead of direct RS-232. Supports multi-capture design (simultaneous + Bridge + Analyzer + Download sessions). + - **`ach_server.py` — TX capture added (`raw_tx_.bin`)** — Every ACH inbound session now saves both directions: `raw_rx_.bin` (device → us, S3 side, as before) and `raw_tx_.bin` (us → device, BW side). Both files are usable in the Analyzer. diff --git a/CLAUDE.md b/CLAUDE.md index f06d849..dbfa3c2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -118,29 +118,156 @@ S3→BW (response): Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26 BW TX capture. All 10 frames verified. -### SUB 5A — chunk counter formula (FINAL CORRECTION 2026-04-26) +### SUB 5A — chunk counter formula (REWRITTEN 2026-05-01 — see 5-1-26 captures) -**Chunk counter = `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` for ALL chunks.** +> ⚠️ **Everything that came before this rewrite was WRONG in important ways.** The previous +> formula `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` happened to *work* for events +> at start_key=0 because the device responds to whatever counter you ask for — but it caused +> a 5× over-read past the actual event, picking up post-event circular-buffer garbage that +> corrupts the reconstructed file for any event > ~1 sec of waveform. The captures in +> `bridges/captures/4-27-26/` and `5-1-26/comcheck/` show BW reads only ~12-16 chunks for +> the same events SFM was reading 37+ chunks for. See "TERM frame" and "STRT end_offset" +> sections below for the actual mechanism. -where `key4[2:4] = (key4[2] << 8) | key4[3]` is the event's circular-buffer base offset. +**Chunk addressing is just absolute device-buffer addresses.** -The `max(..., 0x0400)` guard is critical for events at the start of the circular buffer -(key4[2:4] == 0x0000, e.g. key `01110000`). Without it, chunk 1 gets counter=0x0000, which -is the same address as the probe frame — the device re-returns the STRT record data instead -of waveform payload. With the guard, chunk 1 gets counter=0x0400, which is confirmed correct -from the empirical live-device test 2026-04-06 (`counter=0x0400 → responds immediately and -streams all frames correctly`). +`params[0]=0x00`, `params[1:5]` is a 4-byte absolute device flash-buffer address (= the +"key" of that location), `params[5:11]` are zeros. The device returns 0x0200 (= 512) bytes +starting at that address. Increments between consecutive chunks are **0x0200 (NOT 0x0400)** +— this matches the chunk payload size. The previous "0x0400 step" worked by accident: BW +asks for half-size chunks; SFM was asking for double-size chunks, both with the same-named +"counter" field, but the value is just an address pointer the device honors as-is. -The 4-3-26 capture confirms the pattern for a second event (key `0111245a`, key4[2:4]=0x245a): -chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400). -`max(0x245a, 0x0400) = 0x245a` → formula works correctly for non-zero base offset too. +**The chunk pattern depends on whether the event sits at start_key=0 or not.** + +#### Event 1 case — start_key[2:4] == 0x0000 (first event after erase / wrap) + +``` +1. Probe at counter=0x0000 (params[1:5] = full key, returns STRT record) +2. Read 2 fixed metadata pages: counter=0x1002, counter=0x1004 + (these are GLOBAL session metadata — read ONCE per + Blastware session, not per event; contain the + Project/Client/User Name/Seis Loc strings) +3. Sample chunks: counter=0x0600, 0x0800, …, by 0x0200 increment, + up to but not including end_offset (rounded down to + 0x0200 boundary) +4. TERM frame (see TERM formula below) +``` + +The reason `0x0046..0x0600` is skipped for event 1 is unknown — likely some pre-event +firmware reserved area for the first slot in a freshly-erased buffer. Harmless to skip. + +#### Event 2+ case — start_key[2:4] != 0x0000 (continuation events) + +``` +1. First chunk at counter = start_key[2:4] + 0x0046 (this IS the probe — response + contains STRT) +2. Sample chunks: counter += 0x0200 each, up to but + not including end_offset +3. TERM frame +``` + +No metadata pages — those have already been read during event 1 in the same Blastware +session, and BW caches them. Note that the metadata-page reads happen ONCE per +Blastware-session-on-the-device, not once per event, so an SFM session that downloads +several events should read 0x1002/0x1004 only once at the start. + +#### History (do not re-derive) -**History:** - Original: `_CHUNK1_COUNTER = 0x1004` hardcoded (Blastware capture artifact — WRONG). -- 2026-04-06: Corrected to `chunk_num * 0x0400` (worked for key 01110000 only). -- 2026-04-24: Corrected to `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets, - but accidentally broke key 01110000 — counter=0x0000 sends probe address again). -- 2026-04-26: Final formula: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400`. +- 2026-04-06: `chunk_num * 0x0400` (worked for key 01110000 only). +- 2026-04-24: `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets, broke key 01110000). +- 2026-04-26: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` (broken — over-read past event end). +- 2026-05-01: Increments are 0x0200 not 0x0400; absolute addresses inside event range; bounded + by STRT end_key, not by `max_chunks` cap or device-side timeout. + +### SUB 5A — STRT record encodes end_offset (NEW 2026-05-01) + +The first A5 response (probe response, or the first chunk for event 2+) contains a STRT +record at byte offset 17 of the `data` field. Layout: + +``` +data[17:21] "STRT" magic +data[21:23] ff fe sentinel +data[23:27] end_key ← 4-byte key of where this event ENDS +data[27:31] start_key ← 4-byte key of where this event STARTS +data[31:33] uint16 BE ?? sample-count or total bytes (varies; not yet decoded) +data[33:35] uint16 BE ?? +data[35] 0x46 record type (waveform full record) +… +``` + +`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**. +SFM must extract this from the first A5 response and use it to bound the chunk loop and +encode the TERM frame. The device will happily respond to chunk requests past `end_offset` +(returning post-event circular-buffer contents) — that's the over-read bug. + +Verified across 3 events: + +| Capture | start_key | end_key | end_offset | event size | +|---|---|---|---|---| +| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B | +| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B | +| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | `0x417E` (event 2 span 0x1F8C = 8,076 B) | + +### SUB 5A — TERM frame formula (FINALIZED 2026-05-01) + +The TERM frame fetches the partial last chunk *and* the file footer. It is **not** a simple +"goodbye" frame — its response payload contains the bytes between the last full 0x0200-aligned +chunk and `end_offset`, and is required for reconstructing the Blastware file format. + +``` +last_chunk_counter = address of last full 0x0200-byte chunk read +next_boundary = last_chunk_counter + 0x0200 +TERM offset_word = end_offset - next_boundary +TERM params[0] = key[0] (= 0x01 on every observed device) +TERM params[1] = key[1] (= 0x11) +TERM params[2] = (next_boundary >> 8) & 0xFF +TERM params[3] = next_boundary & 0xFF +TERM params[4:10] = zeros +build_5a_frame(offset_word, params) (10-byte params, NOT 11) +``` + +The device reconstructs `requested_address = (params[2] << 8) | offset_word = end_offset` +and replies with `(end_offset - next_boundary)` bytes from `next_boundary` — the residual +between the last 0x0200 boundary and the actual event end. Append the TERM response data +to the chunk stream like any other A5 frame; it carries the final waveform tail + footer. + +Verified across 3 events: + +| end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] | +|---|---|---|---|---| +| `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ | +| `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ | +| `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ | + +The previous code's hard-coded `offset_word = 0x005A` and `term_counter = last + 0x0400` +are wrong; the device's response under that path is a tiny 101-byte device-side terminator +(arrived only after we walked the entire post-event buffer), not the proper file footer. + +### SUB 5A — fixed metadata pages 0x1002 and 0x1004 (NEW 2026-05-01) + +Two chunk addresses are GLOBAL device/session metadata, not event-specific: + +- `counter=0x1002` — first metadata page +- `counter=0x1004` — second metadata page + +These are at fixed absolute addresses in the device's flash buffer. They contain the +session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII +strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the +new walk these strings come from the dedicated metadata pages, not from the sample-chunk +stream. + +BW reads them ONCE per Blastware session (during event 1's download) and caches them. +For SFM, that means: +- Once per call-home / once per `MiniMateClient.connect()` is enough. +- Subsequent events in the same session don't need to re-fetch them. +- Their content does not change when iterating events; only when the user opens + Compliance Setup → Apply on the device or sends a SUB 71 compliance write. + +The contents have not been byte-for-byte decoded yet — first task on the implementation +side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the +strings we currently extract from A5[7]. ### SUB 5A — params are 11 bytes for chunk frames, 10 for termination @@ -148,10 +275,16 @@ chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400). confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes. Do not swap them. -### SUB 5A — event-time metadata lives in A5 frame 7 +### SUB 5A — event-time metadata source (UPDATED 2026-05-01) -The bulk stream sends 9+ A5 response frames. Frame 7 (0-indexed) contains the compliance -setup as it existed when the event was recorded: +> **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A +> bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at +> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because +> the broken counter formula was scanning the wrong region. +> +> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and +> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A +> sample-chunk stream itself does NOT contain these strings any more under the new walk. ``` "Project:" → project description @@ -171,26 +304,37 @@ used as the authoritative source. `_decode_a5_metadata_into` therefore only set "Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C record — 5A remains the sole source for those fields and they are set unconditionally. -`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears, -then sends the termination frame. +> ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and +> stops one chunk later) is a workaround for the missing end_offset bound — when the new +> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is +> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM +> frame. -### SUB 5A — end-of-stream signal (confirmed 2026-04-06) +### SUB 5A — end-of-stream — UPDATED 2026-05-01 -After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to -the next chunk request, then goes silent. This is the natural end-of-stream indicator — NOT -a complete A5 frame. `S3FrameParser.bytes_fed` will be 1; no frame is assembled. +> **Previous understanding (now known to be a symptom, not a feature):** "After streaming +> all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was +> not the device's natural end-of-event signal — it was the device's response when SFM had +> walked clean off the end of the addressable buffer region after over-reading by ~5×. +> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the +> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`, +> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser. -Handling: on `TimeoutError`, if `bytes_fed > 0` AND frames were already collected, treat as -graceful end-of-stream, break the loop, and proceed to the termination frame. If `bytes_fed -== 0` with no prior frames, it is a genuine transport failure — re-raise. +The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful +defence-in-depth fallback for malformed events or unexpected device states, but should not +be the primary loop-exit condition. **Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each. Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call in the chunk loop passes `timeout=10.0` explicitly. -**Typical chunk count (BE11529, 1024 sps):** A 9,306-sample event produces 35 chunks before -end-of-stream. Chunks with uniform 1,036-byte data are all-zero ADC samples (post-event -silence). Only the initial variable-size chunks contain actual signal. +**Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):** +A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames. +A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames. +An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames. + +Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks +17-37 containing post-event circular-buffer garbage that corrupted the file body. ### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06) @@ -303,6 +447,55 @@ sends token=0xFE and is NOT used by any caller. `advance_event()` returns `(key4, event_data8)`. Callers (`count_events`, `get_events`) loop while `data8[4:8] != b"\x00\x00\x00\x00"`. +### SUB 0A — WAVEHDR response length distinguishes events from boundaries (NEW 2026-05-01) + +When iterating events with the "Download All" pattern (1E → 0A → 1F → 0A → 1F → …), the +DATA_LENGTH at `data_rsp.data[5]` (= the byte BW echoes back as the offset for the data +fetch step) takes one of two values: + +| WAVEHDR offset | Meaning | +|---|---| +| `0x46` (= 70) | Real event start key — there is event data at this address | +| `0x2C` (= 44) | Boundary marker between events — this key is the END of the previous event AND the START key for the empty space after it (or is the next event's pre-header) | + +Confirmed from the 5-1-26 "Download All" capture: + +``` +0A(key=01110000) → off=0x46 ← event 1 real start +1F → key=011121F2 +0A(key=011121F2) → off=0x2C ← event 1 END / event 2 boundary +1F → key=01112238 +0A(key=01112238) → off=0x46 ← event 2 real start (= boundary + 0x46) +1F → key=0111417E +0A(key=0111417E) → off=0x2C ← event 2 END / next-empty marker +1F → null sentinel +``` + +This is why event 2's first 5A chunk is at `start_key + 0x46` — that's the address of the +"real start" 0x46-record, distinct from the `0x2C`-record at the raw boundary. Use the +`0x46` keys as the input to `read_bulk_waveform_stream`, not the `0x2C` keys. + +For event 1 only (start_key[2:4] = 0x0000) BW probes at counter=0x0000 directly, which is +the `0x46`-keyed start record. Subsequent events use `start_key + 0x46`. + +**Practical iteration pattern (replaces the old 1E/1F walk for downloads):** + +``` +Setup: SERIAL × 2 → CHCFG → 1E (token=0x00) → key0 +For each event: + 0A(cur_key) → DATA_LENGTH = 0x46 (real) or 0x2C (boundary) + 1F (token=0x00) → next_key + if length was 0x46: → cur_key is a real event; queue it for download + cur_key = next_key + if next_key all-zero null sentinel: stop + +Then for each queued real-event key: + download_event(key) → 5A bulk stream with STRT-bounded chunk walk +``` + +This is what BW does in the 5-1-26 "Download All" capture — it walks the full event chain +collecting `(key, length)` tuples first, *then* downloads each event using the `0x46` keys. + ### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce) `read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where: @@ -527,6 +720,8 @@ All DB endpoints are read-only except `PATCH /db/events/{id}/false_trigger`. | 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture | | 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence; only 1 event stored so token=0xFE appeared to work | | 4-3-26 | `bridges/captures/4-3-26/` | Browse-mode S3 capture with 2+ events — confirmed all-zero params for 1F, 1F response layout, null sentinel, 0A context requirement | +| 4-27-26 | `bridges/captures/4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof that SFM was over-reading 5× past event end. BW reads 14 chunks at 0x0200 increments + TERM at end_offset; SFM was reading 37 chunks at 0x0400 increments. STRT end_key field located. | +| 5-1-26 | `bridges/captures/5-1-26/comcheck/` | Three sub-captures: SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945`/`_171216`). Confirmed: TERM frame formula across 3 events; metadata pages 0x1002/0x1004 are global (read once per session); event-1 vs event-N chunk-pattern split; WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundaries. | --- diff --git a/README.md b/README.md index 9603039..e0aafb7 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# seismo-relay `v0.12.1` +# seismo-relay `v0.12.6` A ground-up replacement for **Blastware** — Instantel's aging Windows-only software for managing MiniMate Plus seismographs. @@ -18,26 +18,27 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55). ``` seismo-relay/ -├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Console tabs) +├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Download + Console tabs) │ ├── minimateplus/ ← MiniMate Plus client library │ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport │ ├── protocol.py ← DLE frame layer, SUB command dispatch -│ ├── client.py ← High-level client (connect, get_events, push_config, …) +│ ├── client.py ← High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …) │ ├── framing.py ← Frame builders, DLE codec, S3FrameParser -│ └── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, … +│ ├── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig, … +│ └── blastware_file.py ← Write events to Blastware-compatible .AB0 files │ ├── sfm/ ← SFM REST API server (FastAPI, port 8200) -│ ├── server.py ← All device + DB endpoints -│ ├── database.py ← SeismoDb — SQLite persistence layer -│ └── sfm_webapp.html ← Embedded web UI (served at /) +│ ├── server.py ← Live device endpoints + DB query endpoints + caching +│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions, sessions table) +│ └── sfm_webapp.html ← Embedded web UI with Call Home config tab │ ├── bridges/ │ ├── ach_server.py ← Inbound ACH call-home server (main production server) │ ├── ach_mitm.py ← Transparent MITM proxy for capturing BW sessions │ ├── s3-bridge/ ← RS-232 serial bridge (capture tool) │ ├── tcp_serial_bridge.py ← Local TCP↔serial bridge (bench testing) -│ ├── gui_bridge.py ← Standalone bridge GUI +│ ├── gui_bridge.py ← Standalone bridge GUI with raw capture checkboxes │ └── raw_capture.py ← Simple raw capture tool │ ├── parsers/ @@ -101,21 +102,28 @@ python seismo_lab.py Each call dials the device, does its work, and closes the connection. TCP connections are retried once on `ProtocolError` to handle cold-boot timing. -**Caching** — frequently-polled endpoints are cached in-process to avoid -redundant TCP round-trips: +**In-memory caching** — frequently-polled endpoints avoid redundant TCP round-trips +via a thread-safe `_LiveCache` (plain Python dict + `threading.Lock`): -| Method | URL | Cache | -|--------|-----|-------| +| Method | URL | Cache Strategy | +|--------|-----|---| | `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` | | `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected | | `GET` | `/device/event/{idx}/waveform` | Permanent per event index | -| `GET` | `/device/monitor/status` | 30-second TTL | +| `GET` | `/device/monitor/status` | 30-second TTL; invalidated by monitor start/stop | +| `GET` | `/device/call_home` | Fresh read from device (not cached) | | `POST` | `/device/connect` | — | -| `POST` | `/device/config` | Writes compliance config; invalidates cache | -| `POST` | `/device/monitor/start` | Sends SUB 0x96 | -| `POST` | `/device/monitor/stop` | Sends SUB 0x97 | +| `POST` | `/device/config` | Writes compliance config; invalidates info + events cache | +| `POST` | `/device/config/project` | Patches project/client/operator/sensor_location strings | +| `POST` | `/device/monitor/start` | Sends SUB 0x96; immediately evicts status cache | +| `POST` | `/device/monitor/stop` | Sends SUB 0x97; immediately evicts status cache | +| `POST` | `/device/call_home` | Reads, patches specified fields, writes back to device | -All cached endpoints accept `?force=true` to bypass the cache. +**Cache bypass** — All cached endpoints accept `?force=true` to skip the cache and +force a fresh read from the device. + +**Cache stats** — `GET /cache/stats` returns hit/miss counts and TTL info; `DELETE /cache/device` +clears the device cache immediately. Transport query params (supply one set): ``` @@ -152,21 +160,33 @@ client = MiniMateClient(transport=TcpTransport("1.2.3.4", 12345), timeout=30.0) with client: # Read - info = client.connect() # DeviceInfo — serial, firmware, compliance config - count = client.count_events() # Number of stored events - keys = client.list_event_keys() # Fast browse walk — event keys only, no download - events = client.get_events() # Full download: headers + peaks + metadata - monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag - log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records) + info = client.connect() # DeviceInfo — serial, firmware, compliance config + count = client.count_events() # Number of stored events + keys = client.list_event_keys() # Fast browse walk — event keys only, no download + events = client.get_events() # Full download: headers + peaks + metadata + monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag + log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records) + ach_cfg = client.get_call_home_config() # Auto Call Home settings (SUB 0x2C) # Write client.apply_config( sample_rate=1024, + recording_mode="Continuous", # Single Shot / Continuous / Histogram / Histogram+Continuous + histogram_interval_sec=15, # 2, 5, 15, 60, 300, 900 trigger_level_geo=0.5, + geo_range="Normal", # Normal (10.000 in/s) / Sensitive (1.25 in/s) project="Bridge Inspection 2026", client_name="City of Portland", operator="B. Harrison", ) + + client.set_call_home_config( + auto_call_home_enabled=True, + after_event_recorded=True, + at_specified_times=True, + time1_hour=18, time1_min=30, # 6:30 PM + time2_hour=6, time2_min=0, # 6:00 AM + ) # Control client.start_monitoring() # SUB 0x96 @@ -182,18 +202,20 @@ existed at record time — not backfilled from the current compliance config. ## Database -`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode). -Three tables, all unit-keyed by serial number: +`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode) using the +`SeismoDb` persistence layer. Four tables, all unit-keyed by serial number: | Table | Key | Contents | |-------|-----|----------| -| `ach_sessions` | UUID | Per-call-home audit record: serial, peer IP, events_downloaded, duration | -| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, PPV per channel, project/client/operator strings, false_trigger flag | -| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: start/stop time, duration, geo threshold | +| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds | +| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag | +| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips | +| `events.false_trigger` | Boolean flag | PATCH endpoint to mark/unmark false triggers for review | -Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs -never produce duplicate rows. Post-erase key reuse is handled automatically -via the high-water mark in `ach_state.json`. +Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs never +produce duplicate rows. Post-erase key reuse is handled automatically via the +high-water mark in `ach_state.json`. Key-based state tracking allows correct +handling of device erasures (external or post-download). --- @@ -231,6 +253,27 @@ Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/insta --- +## Compliance Config Features (v0.12.2–v0.12.3) + +The REST API and web UI expose full control over device compliance settings: + +- **Recording Mode** (Single Shot / Continuous / Histogram / Histogram+Continuous) +- **Sample Rate** (1024 / 2048 / 4096 sps) +- **Record Time** (float, seconds) +- **Histogram Interval** (2s, 5s, 15s, 1m, 5m, 15m) — when recording mode includes histogram +- **Geo Trigger Levels** (float, in/s per channel) +- **Geo Maximum Range** (Normal 10.000 in/s / Sensitive 1.250 in/s per channel) +- **Project / Client / Operator / Sensor Location** (ASCII strings) + +Auto Call Home config: +- **Auto Call Home Enable** (bool) +- **Dial String** (read-only; 40-byte ASCII) +- **Trigger on Event** (bool) +- **Scheduled Call-Ins** (two time slots with HH:MM each) +- **Retry Settings** (count, delay, connection timeout, warm-up time) + +--- + ## Requirements ```bash @@ -252,17 +295,55 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows. --- -## Roadmap +## Key Features (v0.10–v0.12) + +**Device support (v0.12.5):** +- [x] Full read/write/erase pipelines +- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings) +- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries) +- [x] Monitor control (start/stop, status polling, battery/memory) +- [x] Monitor log entries (continuous monitoring intervals without full waveform download) + +**Data persistence (v0.11):** +- [x] SQLite database (`seismo_relay.db`) with 4 tables: ach_sessions, events, monitor_log, plus false_trigger flag +- [x] Deduplication by waveform key (handles re-runs and repeat call-homes) +- [x] Post-erase key-reuse detection (tracks high-water mark) +- [x] Session state (`ach_state.json`) with downloaded keys and max key + +**REST API (v0.12.1):** +- [x] Live device endpoints with in-memory caching (`_LiveCache`) +- [x] Cache statistics (`/cache/stats`) and manual invalidation (`/cache/device`) +- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH) +- [x] Call Home config read/write endpoints +- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`) + +**File output (v0.7+):** +- [x] Blastware-compatible `.AB0` file generation (waveform + metadata) +- [x] Multi-channel waveform decode from SUB 5A bulk stream +- [x] Second-resolution timestamp encoding in Blastware filename + +**Capture tools (v0.12.5):** +- [x] Serial-to-TCP bridge with raw BW/S3 capture (s3_bridge.py, defaults to auto-capture) +- [x] GUI bridge with raw capture checkboxes (gui_bridge.py) +- [x] ACH inbound server with bidirectional capture (ach_server.py saves raw_tx + raw_rx) +- [x] Transparent TCP MITM proxy for live BW session capture (ach_mitm.py) + +**Analysis tools:** +- [x] s3_analyzer.py — session parser, frame differ, Claude export +- [x] gui_analyzer.py — standalone analyzer GUI +- [x] frame_db.py — SQLite frame database for capture analysis + +**seismo_lab.py GUI (v0.12.5):** +- [x] Bridge tab — Serial/TCP mode selector with raw capture options +- [x] Analyzer tab — BW/S3 capture playback and differencing +- [x] Download tab — Live wire-byte capture during event download (new v0.12.5) +- [x] Console tab — Logging and diagnostics + +## Roadmap (Future) -- [x] Full read pipeline — device info, compliance config, event download with true event-time metadata -- [x] Write commands — push compliance config, trigger thresholds, project strings to device -- [x] Erase all events — confirmed erase sequence from live MITM capture -- [x] Monitor control — start/stop monitoring, read battery/memory/status -- [x] Monitor log entries — decode partial 0x2C records (continuous monitoring intervals) -- [x] ACH inbound server — accept call-home connections, download events, dedup by key -- [x] SQLite persistence — events, monitor log, and session history in `seismo_relay.db` -- [x] SFM REST API — device control + DB query endpoints, live device cache - [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing - [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first) - [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object - [ ] Modem manager — push RV50/RV55 configs via Sierra Wireless API +- [ ] Histogram mode recording support (5A stream analysis for mode 0x03) +- [ ] Call Home dial_string write support (requires DLE escaping for embedded control characters) diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index 95950ec..0d90732 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -110,6 +110,7 @@ | 2026-04-21 | §7.6.2, §5.3 | **CORRECTED — compliance_raw contains wire-encoded bytes, NOT logical bytes.** S3FrameParser appends DLE+ETX inner-frame pairs as two literal bytes to the frame body. Any `0x03` values in the compliance config appear in `compliance_raw` as `\x10\x03` (two bytes), not as a single `0x03`. The previous claim "S3FrameParser handles this transparently so compliance_raw contains logical (destuffed) bytes" was wrong. Consequence: `compliance_raw` is the wire-encoded E5 payload; anchor-relative reads work correctly because the anchor position automatically accounts for any DLE-encoded bytes before it. For write-back, round-tripping `compliance_raw` verbatim sends the correct wire bytes to the device. **DLE ETX escaping in write frames:** Blastware escapes `0x03` bytes in write frame data as `\x10\x03` on wire; our `build_bw_write_frame` does not (writes data raw). Device is confirmed to accept raw writes for all tested modes — likely uses the offset/length field for write frame framing, not ETX scanning. | | 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. | | 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. | +| 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. | --- @@ -1226,7 +1227,24 @@ Two critical differences from `build_bw_frame`: 2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen, only `XX` is added to the running sum; lone bytes are added normally. -#### 7.8.2 Request Sequence +#### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol) + +> ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new +> BW MITM captures (4-27-26 + 5-1-26) prove the actual chunk increment is **0x0200**, the +> chunk loop is bounded by `end_offset` from the STRT record (not by chunk count or by a +> device-side timeout), and the TERM frame's `offset_word=0x005A` magic is incorrect — the +> real TERM offset_word is computed from `end_offset` and the last chunk address. Under the +> deprecated formula SFM over-reads roughly 5× past the actual event end into post-event +> circular-buffer garbage, corrupting reconstructed Blastware files for any waveform ≥ 2 sec. +> +> The whole "stop_after_metadata + one extra chunk + 0e 08 footer" workaround in this +> section was compensating for the missing end_offset bound. It is obsoleted by the +> STRT-bounded walk in §7.8.5. +> +> **Read this section for historical context only.** For the correct protocol, jump to: +> - §7.8.5 — chunk addressing and the STRT end_offset +> - §7.8.6 — TERM frame formula +> - §7.8.7 — fixed metadata pages 0x1002 and 0x1004 | Frame | offset_word | counter | params | Purpose | |---|---|---|---|---| @@ -1237,45 +1255,17 @@ Two critical differences from `build_bw_frame`: | … | … | … | … | … | | Termination | `0x005A` | `max(key4[2:4], 0x0400) + N * 0x0400` | 10 bytes | End transfer | -> ⚠️ **2026-04-06 CORRECTED — chunk counter is `key4[2:4] + (N-1) * 0x0400`.** -> The 4-2-26 BW TX capture showed counter=0x1004 for chunk 1 of key `01110000`, leading to -> an interim "monotonic n * 0x0400" formula. This was accidentally correct because -> `key4[2:4] == 0x0000` for that event. -> -> **2026-04-24 CORRECTION:** The counter is an absolute circular-buffer address. -> BW's true formula is `key4[2:4] + (chunk_num - 1) * 0x0400` where `key4[2:4]` is the -> event's storage base offset (`(key4[2]<<8) | key4[3]`). For keys where -> `key4[2:4] != 0x0000` (e.g. key `01111884`), using `n * 0x0400` sends requests into the -> wrong buffer region — the device returns data from a completely different event. -> -> **2026-04-26 FINAL CORRECTION:** The formula `key4[2:4] + (N-1) * 0x0400` is wrong when -> `key4[2:4] == 0x0000` (e.g. event key `01110000`, the very first event after a device erase). -> Counter=0x0000 for chunk 1 is the same address as the probe frame — the device re-returns -> the STRT record data instead of waveform payload (frame 1 has len=1097, same as probe, and -> contains `b"STRT\xff\xfe"`, contributing zero waveform bytes). -> Final formula: `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400`. -> For key `01110000`: chunk 1 = 0x0400 (confirmed working, empirical test 2026-04-06). -> For key `0111245a`: chunk 1 = 0x245a (unchanged, confirmed from 4-3-26 capture). +> Historical correction notes (left in place to deter re-derivation of the same wrong formula): +> the table above was the result of three iterative "corrections" between 2026-04-06 and +> 2026-04-26 that progressively narrowed in on the wrong answer because every test was on +> events with `key4[2:4]=0` and the device responds to whatever counter you ask for. The +> 5-1-26 captures with a non-zero start_key event (`01112238`) finally exposed the bug. -The `stop_after_metadata=True` flag causes the loop to stop as soon as `b"Project:"` is -found in the accumulated A5 frame data, typically after 4–9 chunks. A termination frame -is always sent before returning. - -**IMPORTANT — one extra chunk required after "Project:" for valid file footer (confirmed 2026-04-23):** -When writing a Blastware-compatible waveform file, stopping immediately at "Project:" and -sending termination produces an empty termination response with no footer bytes (`0e 08` -marker missing). Blastware downloads exactly **one more chunk** after finding "Project:" -before sending termination — that extra chunk primes the device to return valid footer -bytes (monitoring start/stop timestamps) in the termination response. - -`read_bulk_waveform_stream(stop_after_metadata=True)` implements this: after the "Project:" -chunk is received, one additional chunk is requested before breaking. The termination -response (`include_terminator=True`) then contains the correct `0e 08` footer. - -**do NOT use `full_waveform=True` for Blastware file writing** — for events with long -post-event silence (35 chunks), the silence chunks contain embedded device-internal -pointer structures that produce spurious STRT markers in the file body. Blastware only -downloads 4–5 chunks (metadata + one signal chunk) regardless of event length. +The `stop_after_metadata=True` flag (deprecated as a primary loop-exit) scanned for +`b"Project:"` in the chunk stream because the metadata strings happened to be reachable +when the broken 0x0400-step walk passed the global metadata pages at 0x1002/0x1004. Under +the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004, +not from the sample-chunk stream — see §7.8.7. #### 7.8.3 A5 Frame Layout @@ -1293,15 +1283,19 @@ for ASCII labels with a null-terminated value read: All five fields reflect the **setup at event-record time**, not the current device config. -#### 7.8.4 End-of-Stream Behaviour and Chunk Timing +#### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01 -> ✅ **Confirmed 2026-04-06** — empirical observation on BE11529 (S338.17) over TCP/cellular. +> The "1 raw byte then silence" pattern documented below was originally interpreted as +> "the device's natural end-of-event signal." The 5-1-26 captures show this is actually +> the device's response when the requester has walked **past** the addressable buffer +> region (i.e. ~5× past the actual event end under the deprecated 0x0400-step walk). +> Under the corrected STRT-bounded walk (§7.8.5), the stream ends cleanly with the TERM +> frame's response — no timeout, no 1-byte teaser. The fallback below remains useful as +> defensive handling for malformed events but should not be the primary loop-exit. -**End-of-stream signal:** After sending all waveform chunks, the device sends exactly **1 raw byte** in response to the next chunk request, then goes silent. This byte is not a complete DLE-framed A5 response — `S3FrameParser.bytes_fed` reports 1 and no frame is ever assembled. This is the device's natural end-of-stream indicator. - -Handling logic in `read_bulk_waveform_stream`: +**Defensive fallback handling in `read_bulk_waveform_stream`:** ``` -TimeoutError caught: +TimeoutError caught (rare under corrected walk): if bytes_fed > 0 AND frames already collected: → graceful end-of-stream; break loop; proceed to termination frame else (bytes_fed == 0, no prior frames): @@ -1313,14 +1307,15 @@ TimeoutError caught: | Metric | Observed value | |---|---| | Chunk response time | ~1 s per chunk | -| Chunks for a 9,306-sample event | 35 chunks | -| Data per chunk (active signal) | 1,036–1,123 bytes | -| Data per chunk (post-event silence) | 1,036 bytes (uniform) | +| Chunks for a 2-sec event (corrected walk) | 14 (12 sample chunks + 2 metadata pages) + TERM | +| Chunks for a 3-sec event (corrected walk) | 18 (16 sample chunks + 2 metadata pages) + TERM | +| Chunks for a continuation event (corrected walk) | ~15 sample chunks + TERM (no metadata reread) | +| Chunks under deprecated walk for 2-3 sec event | 37 (over-reads ~5×) | +| Data per chunk (corrected, 0x0200 size) | ~540–575 bytes wire (= 0x0200 payload + framing) | +| Data per chunk (deprecated 0x0400 step) | 1,036–1,123 bytes wire (= 0x0400 payload + framing) | | Safe recv timeout per chunk | **10 s** (10× typical) | | Default transport timeout | 120 s → ~2-min stall at end-of-stream | -Chunks with uniform 1,036-byte payload (chunks 17–35 in the observed event) contain all-zero ADC samples — the device continues recording silence until the configured record time expires before terminating the stream. - **ADC count-to-physical conversion — ✅ CONFIRMED 2026-04-17:** Raw samples are signed 16-bit integers (−32,768 to +32,767). Source: Interface Handbook §4.5. @@ -1339,6 +1334,186 @@ where `geo_range = 1.61133 V × 6.206053 = 10.000 in/s` is the Normal (Gain=1) f `_decode_a5_waveform()` contains `elif fi == 9: continue` from an earlier assumption that frame index 9 is always the device terminator. For streams with more than 9 frames, frame 9 is live waveform data. The skip discards ~1,070 bytes (~133 sample-sets) per event. Terminator detection should use `page_key == 0x0000`, not frame index. This skip should be removed. +#### 7.8.5 Chunk addressing and the STRT end_offset (NEW 2026-05-01) ✅ + +> ✅ Confirmed across 3 events (4-27-26 + 5-1-26 captures). + +`params[0]` is always `0x00`. `params[1:5]` is a 4-byte absolute device flash-buffer +address — equivalently, "the key of the page being requested." The device returns 0x0200 +(= 512) bytes starting at that address. Increments between consecutive sample chunks are +**0x0200, NOT 0x0400** (the previous 0x0400 figure was a Blastware-side artifact / our +implementation's bug — see §7.8.2). + +##### STRT record (data layout in the first A5 response) + +The first A5 response (the probe response, or the first chunk for continuation events) +contains a **STRT record** at byte offset 17 of `data`: + +``` +data[ 0:14] echoes request: [chunk_size_hi=0x02 / 0x04 ...] [00] [01 11] [counter_hi counter_lo] [00 × 8] [00 12] +data[14:17] 10 03 00 ← inner DLE+ETX frame separator (preserved literally) +data[17:21] "STRT" ← magic +data[21:23] ff fe ← sentinel +data[23:27] end_key ← 4-byte key of where this event ENDS +data[27:31] start_key ← 4-byte key of where this event STARTS +data[31:33] uint16 BE ← ?? sample count or byte count, varies (not yet decoded) +data[33:35] uint16 BE ← ?? +data[35] 0x46 ← record type marker (waveform full record) +data[36:] additional pointers / first sample bytes — content varies by event +``` + +`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**. +Use it to bound the chunk loop and to compute the TERM frame. + +##### Chunk pattern by event location in buffer + +**Event 1 / start_key[2:4] = 0x0000** (first event after erase or wrap): + +``` +1. Probe at counter = 0x0000 (params[1:5] = full key) +2. Read fixed metadata pages counter = 0x1002, then 0x1004 +3. Walk sample chunks counter = 0x0600, 0x0800, …, by 0x0200, + up to but not including end_offset & 0xFE00 +4. TERM (see §7.8.6) +``` + +The range `[0x0046, 0x0600)` is skipped — likely some pre-event firmware-reserved area for +the first slot in a freshly-erased buffer. Harmless to skip; BW does the same. + +**Event 2+ / start_key[2:4] != 0x0000** (continuation events in a populated buffer): + +``` +1. First chunk at counter = start_key[2:4] + 0x0046 ← acts as both probe and first + sample chunk; response carries STRT +2. Walk sample chunks counter += 0x0200 each +3. TERM +``` + +**No metadata-page reads.** Pages 0x1002/0x1004 are session-global and were already read +during event 1 in the same Blastware session. In SFM, treat metadata pages as a once- +per-`MiniMateClient.connect()` (or once-per-call-home) read, not per-event. + +##### Verified end_offset values + +| Capture | start_key | end_key | end_offset | event size | sample-chunk start | +|---|---|---|---|---|---| +| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B | 0x0600 (event-1 case) | +| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B | 0x0600 (event-1 case) | +| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | event 2 size = 0x1F8C = 8,076 B | 0x2238 (= 0x21F2 + 0x46) | + +#### 7.8.6 TERM Frame Formula (NEW 2026-05-01) ✅ + +> ✅ Confirmed across 3 events. Replaces the deprecated `offset_word=0x005A` / `params[2] = key4[2]` formula in §7.8.2. + +The TERM frame fetches the partial last chunk and the file footer. Its response payload +contains the bytes between the last full 0x0200-aligned chunk and `end_offset` — typically +20–520 B — and is **required for reconstructing the Blastware waveform file**. Append the +TERM response data to the chunk stream like any other A5 frame. + +``` +last_chunk_counter = address of last full 0x0200-byte chunk read +next_boundary = last_chunk_counter + 0x0200 +TERM offset_word = end_offset - next_boundary +TERM params[0] = key[0] (= 0x01 on every observed device) +TERM params[1] = key[1] (= 0x11) +TERM params[2] = (next_boundary >> 8) & 0xFF +TERM params[3] = next_boundary & 0xFF +TERM params[4:10] = zeros ← 10-byte params (not 11) + +Frame = build_5a_frame(offset_word, params) +``` + +The device receives `requested_address = (params[2] << 8) | offset_word` (where offset_word +contains both `offset_hi` and `offset_lo` of the 5A frame, with the high bit of offset_hi +being effectively `bit 0 of (end_offset >> 8)`). It reconstructs `end_offset` and replies +with `(end_offset - next_boundary)` bytes of waveform tail starting at `next_boundary`. + +##### Verification + +| Event | end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] | TERM response size | +|---|---|---|---|---|---|---| +| 2-sec | `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ | 208 B | +| 3-sec | `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ | 520 B | +| Event-2 | `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ | (not measured directly; same pattern) | + +Equivalent way to write the formula: +- `offset_word = end_offset & 0x01FF` — low 9 bits of end_offset +- `params[2:4] = (end_offset & 0xFE00) BE` — high 7 bits of end_offset, low byte zeroed + +(The two forms are arithmetically identical to `end_offset - next_boundary` and +`next_boundary` because `next_boundary = end_offset & 0xFE00` whenever the chunk loop +stopped at the last full 0x0200 boundary below end_offset.) + +#### 7.8.7 Fixed Metadata Pages 0x1002 / 0x1004 (NEW 2026-05-01) 🔶 + +> 🔶 Inferred — observed in BW captures but page contents not yet byte-decoded. + +Two chunk addresses are **GLOBAL** device/session metadata, not event-specific: + +- `counter = 0x1002` — first metadata page +- `counter = 0x1004` — second metadata page + +These are at fixed absolute addresses in the device's flash buffer. They contain the +session-start compliance-setup ASCII strings — **Project**, **Client**, **User Name**, +**Seis Loc**, **Extended Notes** — that under the deprecated 0x0400-step walk used to be +discoverable in the sample-chunk stream as "A5 frame 7" content. Under the corrected +0x0200-step walk these strings come exclusively from the dedicated metadata-page reads, +not from sample chunks. + +##### Caching strategy + +BW reads them ONCE per Blastware session, during event 1's download, and caches them. +For SFM: +- Read once per `MiniMateClient.connect()` / once per call-home session. +- Subsequent events in the same session don't need to re-fetch them. +- Their content does not change while iterating events. They DO change when the user + applies a new compliance setup (SUB 71 write) — invalidate the cache then. + +##### TODO — content layout + +The byte-for-byte layout of pages 0x1002 and 0x1004 has not been decoded. First task on +the implementation side: dump both pages from a fresh capture and verify they include all +the strings currently extracted from the deprecated A5 frame 7 of the chunk stream. +Compare to the existing `_decode_a5_metadata_into` parser — same string-search anchors +(`b"Project:"`, `b"Client:"`, `b"User Name:"`, `b"Seis Loc:"`, `b"Extended Notes"`) likely +apply directly. + +#### 7.8.8 "Download All" Sequence (NEW 2026-05-01) ✅ + +> ✅ Confirmed from 5-1-26 "Download All" capture (`raw_*_171216_download_all_2events.bin`). + +Before any 5A traffic, BW's "Download All" pre-walks the entire event chain to map keys +and event boundaries: + +``` +SERIAL × 2 → CHCFG → EVT_KEY (1E, all-zero) → key0 + → WAVEHDR (0A, key0) → off=0x46 (real event start) + → EVT_NEXT (1F, all-zero) → key1 + → WAVEHDR (0A, key1) → off=0x2C (boundary) + → EVT_NEXT → key2 + → WAVEHDR (0A, key2) → off=0x46 (real event start) + → EVT_NEXT → key3 + → WAVEHDR (0A, key3) → off=0x2C (boundary) + → EVT_NEXT → null sentinel +``` + +The DATA_LENGTH at `data_rsp.data[5]` (echoed BW offset for the data fetch step) +disambiguates real events from boundary markers: + +| WAVEHDR offset | Meaning | +|---|---| +| `0x46` (= 70) | Real event start key — this key has event data behind it | +| `0x2C` (= 44) | Boundary marker — this key is the END of the previous event AND the start of the empty/header gap before the next event | + +Pairs: each real event spans `[real_key, next_real_key)` in the buffer. In the example +above: event 1 = `[01110000, 011121F2)`, event 2 = `[01112238, 0111417E)`. Note that the +"end of event 1" key (`011121F2`) is also the "boundary key" that comes BEFORE event 2's +real start key (`01112238`) — they differ by exactly 0x46 bytes (the event header size). + +After the pre-walk completes, BW downloads each `0x46`-keyed event in turn using the 5A +bulk stream protocol from §7.8.5. Use the `0x46` keys, not the `0x2C` keys, as input to +`read_bulk_waveform_stream`. + --- ## 7.9 Compliance Config Field Inventory (Blastware UI, 2026-04-08) ✅ diff --git a/seismo_lab.py b/seismo_lab.py index 1e84d0e..1986127 100644 --- a/seismo_lab.py +++ b/seismo_lab.py @@ -214,7 +214,7 @@ class BridgePanel(tk.Frame): tk.Button(cfg, text="Browse", bg=BG3, fg=FG, relief="flat", cursor="hand2", font=MONO, command=self._choose_dir).grid(row=2, column=5, **pad) - # Row 2: buttons + status + # Row 3: buttons + status btn_row = tk.Frame(self, bg=BG2) btn_row.pack(side=tk.TOP, fill=tk.X, padx=4, pady=2)