diff --git a/CHANGELOG.md b/CHANGELOG.md index a0bf491..9833b70 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,89 @@ All notable changes to seismo-relay are documented here. --- +## v0.12.6 — 2026-05-01 + +### Fixed + +- **`blastware_file.py` — waveform frame classification** — A5 frame classification for + waveform-only vs header-only frames now uses `frame.record_type` instead of frame index. + Only waveform frames (0x46) are written to the file body; metadata frames are skipped. + Fixes spurious data corruption from incorrectly classified frames. + +- **`s3_analyzer.py` — A5/5A frame naming** — Bulk waveform stream frames (SUB 5A response) + are now correctly labeled "A5" in analyzer output instead of being conflated with other + multi-frame responses (SUB A4, E5, etc.). + +- **`S3FrameParser` — frame terminator detection** — Corrected the bare ETX terminator + detection. Frame termination is now correctly identified by a standalone `ETX=0x03` byte, + not by the `DLE+ETX` sequence (which is part of the payload when it appears within a frame). + +--- + +## v0.12.5 — 2026-04-21 + +### Added + +- **`seismo_lab.py` — Download tab** — New fourth tab for live wire-byte capture during event + downloads. Captures both BW→device and device→S3 frames in real time, allowing inspection + of the 5A bulk stream chunk sequence and frame-by-frame analysis without needing a bridge + or MITM proxy. Files are saved with user-specified labels for easy tracking. + +### Changed + +- **`s3_bridge.py` — raw captures always-on by default** — `--raw-bw` and `--raw-s3` now + default to `"auto"` instead of `None`. Every bridge session automatically generates + timestamped `raw_bw_.bin` and `raw_s3_.bin` files alongside the `.bin`/`.log` + session files. Pass `--raw-bw ""` (explicit empty string) to disable if needed. + +- **`gui_bridge.py` — raw capture checkboxes pre-checked** — Both "BW→S3 raw" and + "S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names + the files). Unchecking a box passes `--raw-bw ""` to explicitly disable capture. + +- **`Bridge tab` — TCP mode added** — Serial/TCP radio toggle allows connection via cellular + modem (RV50/RV55) instead of direct RS-232. Supports multi-capture design (simultaneous + Bridge + Analyzer + Download sessions). + +- **`ach_server.py` — TX capture added (`raw_tx_.bin`)** — Every ACH inbound session + now saves both directions: `raw_rx_.bin` (device → us, S3 side, as before) and + `raw_tx_.bin` (us → device, BW side). Both files are usable in the Analyzer. + TX bytes are buffered in memory until startup handshake succeeds (same as RX), preventing + scanner probes from creating empty files. + +--- + +## v0.12.4 — 2026-04-21 (protocol analysis / docs only — no code changes) + +### Discovered + +- **compliance_raw is wire-encoded, not logical bytes** — `read_compliance_config()` returns + bytes that include DLE prefix bytes (`0x10`) before any `0x03` values (because S3FrameParser + preserves DLE+ETX inner-frame pairs as two literal bytes). The previous CLAUDE.md claim that + "S3FrameParser handles this transparently so compliance_raw contains logical bytes" was wrong. + +- **anchor-9 behavior per recording mode** (confirmed from 4-20-26 BW write captures): + - Single Shot (0x00) / Continuous (0x01): anchor-9 = `0x00` + - Histogram (0x03): anchor-9 = `0x10` — the E5 DLE prefix for the `0x03` recording_mode byte + - Histogram+Continuous (0x04): anchor-9 = `0x10` — an actual stored config byte for this mode + Anchor position shifts by ±1 when recording_mode = `0x03` due to the extra DLE byte; the + dynamic anchor search (`buf.find(ANCHOR, 0, 150)`) handles this correctly without code changes. + +- **Write frame ETX escaping** — BW escapes `0x03` bytes in write frame data as `0x10 0x03` + on the wire. Our `build_bw_write_frame` sends data bytes raw without ETX escaping. Device + accepts our raw writes for all tested modes. Hypothesis: device write parser uses the + offset/length field for frame boundaries, not ETX scanning, making ETX escaping optional. + Histogram mode (recording_mode = 0x03) write via SFM from a non-Histogram starting state + not yet tested. + +- **BW write payload vs E5 read payload are byte-identical** around the anchor region (confirmed + by comparing 3-11-26 BW TX and S3 captures). BW does NOT strip DLE prefix bytes before writing; + it round-trips the wire-encoded bytes verbatim with only the modified fields changed. + +- **Capture folder content catalogued** — see CLAUDE.md "BW capture reference" table for a + summary of all available protocol captures and their contents. + +--- + ## v0.12.3 — 2026-04-20 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 5ab92f4..6180f69 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,7 +41,7 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c | Event header / first key | 1E | ✅ | | Waveform header | 0A | ✅ | | Waveform record (peaks, timestamp, project) | 0C | ✅ | -| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ new v0.6.0 | +| **Bulk waveform stream (event-time metadata)** | **5A** | ⚠️ partial — over-reads ~5× past event end for ≥2-sec events; corrected algorithm documented but not yet implemented (see "SUB 5A — chunk counter formula" section, dated 2026-05-01) | | Event advance / next key | 1F | ✅ | | **Write commands (push config to device)** | **68–83** | ✅ new v0.8.0 | | **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 | @@ -118,21 +118,156 @@ S3→BW (response): Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26 BW TX capture. All 10 frames verified. -### SUB 5A — chunk counter is monotonic (CORRECTED 2026-04-06) +### SUB 5A — chunk counter formula (REWRITTEN 2026-05-01 — see 5-1-26 captures) -**Chunk counters are `chunk_num * 0x0400` for ALL chunks including chunk 1.** +> ⚠️ **Everything that came before this rewrite was WRONG in important ways.** The previous +> formula `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` happened to *work* for events +> at start_key=0 because the device responds to whatever counter you ask for — but it caused +> a 5× over-read past the actual event, picking up post-event circular-buffer garbage that +> corrupts the reconstructed file for any event > ~1 sec of waveform. The captures in +> `bridges/captures/4-27-26/` and `5-1-26/comcheck/` show BW reads only ~12-16 chunks for +> the same events SFM was reading 37+ chunks for. See "TERM frame" and "STRT end_offset" +> sections below for the actual mechanism. -The 4-2-26 BW TX capture showed `counter=0x1004` for chunk 1 of event key `01110000`, which -led to `_CHUNK1_COUNTER = 0x1004` being hardcoded as a special case. This was a Blastware -artifact, not a protocol requirement. Empirical test 2026-04-06: with `counter=0x1004` for -chunk 1 the device times out (120 s); with `counter=0x0400` (= `1 * 0x0400`) it responds -immediately and streams all frames correctly. +**Chunk addressing is just absolute device-buffer addresses.** -The 4-3-26 capture confirms the pattern for a second event (key `0111245a`): -chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400). Blastware's -true formula is `key4[2:4] + n * 0x0400` — but since `key4[2:4]` of the first event is -`0x0000`, `n * 0x0400` produces the right result. The device does not strictly validate the -counter and streams data for any valid 5A request; using `chunk_num * 0x0400` is correct. +`params[0]=0x00`, `params[1:5]` is a 4-byte absolute device flash-buffer address (= the +"key" of that location), `params[5:11]` are zeros. The device returns 0x0200 (= 512) bytes +starting at that address. Increments between consecutive chunks are **0x0200 (NOT 0x0400)** +— this matches the chunk payload size. The previous "0x0400 step" worked by accident: BW +asks for half-size chunks; SFM was asking for double-size chunks, both with the same-named +"counter" field, but the value is just an address pointer the device honors as-is. + +**The chunk pattern depends on whether the event sits at start_key=0 or not.** + +#### Event 1 case — start_key[2:4] == 0x0000 (first event after erase / wrap) + +``` +1. Probe at counter=0x0000 (params[1:5] = full key, returns STRT record) +2. Read 2 fixed metadata pages: counter=0x1002, counter=0x1004 + (these are GLOBAL session metadata — read ONCE per + Blastware session, not per event; contain the + Project/Client/User Name/Seis Loc strings) +3. Sample chunks: counter=0x0600, 0x0800, …, by 0x0200 increment, + up to but not including end_offset (rounded down to + 0x0200 boundary) +4. TERM frame (see TERM formula below) +``` + +The reason `0x0046..0x0600` is skipped for event 1 is unknown — likely some pre-event +firmware reserved area for the first slot in a freshly-erased buffer. Harmless to skip. + +#### Event 2+ case — start_key[2:4] != 0x0000 (continuation events) + +``` +1. First chunk at counter = start_key[2:4] + 0x0046 (this IS the probe — response + contains STRT) +2. Sample chunks: counter += 0x0200 each, up to but + not including end_offset +3. TERM frame +``` + +No metadata pages — those have already been read during event 1 in the same Blastware +session, and BW caches them. Note that the metadata-page reads happen ONCE per +Blastware-session-on-the-device, not once per event, so an SFM session that downloads +several events should read 0x1002/0x1004 only once at the start. + +#### History (do not re-derive) + +- Original: `_CHUNK1_COUNTER = 0x1004` hardcoded (Blastware capture artifact — WRONG). +- 2026-04-06: `chunk_num * 0x0400` (worked for key 01110000 only). +- 2026-04-24: `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets, broke key 01110000). +- 2026-04-26: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` (broken — over-read past event end). +- 2026-05-01: Increments are 0x0200 not 0x0400; absolute addresses inside event range; bounded + by STRT end_key, not by `max_chunks` cap or device-side timeout. + +### SUB 5A — STRT record encodes end_offset (NEW 2026-05-01) + +The first A5 response (probe response, or the first chunk for event 2+) contains a STRT +record at byte offset 17 of the `data` field. Layout: + +``` +data[17:21] "STRT" magic +data[21:23] ff fe sentinel +data[23:27] end_key ← 4-byte key of where this event ENDS +data[27:31] start_key ← 4-byte key of where this event STARTS +data[31:33] uint16 BE ?? sample-count or total bytes (varies; not yet decoded) +data[33:35] uint16 BE ?? +data[35] 0x46 record type (waveform full record) +… +``` + +`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**. +SFM must extract this from the first A5 response and use it to bound the chunk loop and +encode the TERM frame. The device will happily respond to chunk requests past `end_offset` +(returning post-event circular-buffer contents) — that's the over-read bug. + +Verified across 3 events: + +| Capture | start_key | end_key | end_offset | event size | +|---|---|---|---|---| +| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B | +| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B | +| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | `0x417E` (event 2 span 0x1F8C = 8,076 B) | + +### SUB 5A — TERM frame formula (FINALIZED 2026-05-01) + +The TERM frame fetches the partial last chunk *and* the file footer. It is **not** a simple +"goodbye" frame — its response payload contains the bytes between the last full 0x0200-aligned +chunk and `end_offset`, and is required for reconstructing the Blastware file format. + +``` +last_chunk_counter = address of last full 0x0200-byte chunk read +next_boundary = last_chunk_counter + 0x0200 +TERM offset_word = end_offset - next_boundary +TERM params[0] = key[0] (= 0x01 on every observed device) +TERM params[1] = key[1] (= 0x11) +TERM params[2] = (next_boundary >> 8) & 0xFF +TERM params[3] = next_boundary & 0xFF +TERM params[4:10] = zeros +build_5a_frame(offset_word, params) (10-byte params, NOT 11) +``` + +The device reconstructs `requested_address = (params[2] << 8) | offset_word = end_offset` +and replies with `(end_offset - next_boundary)` bytes from `next_boundary` — the residual +between the last 0x0200 boundary and the actual event end. Append the TERM response data +to the chunk stream like any other A5 frame; it carries the final waveform tail + footer. + +Verified across 3 events: + +| end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] | +|---|---|---|---|---| +| `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ | +| `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ | +| `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ | + +The previous code's hard-coded `offset_word = 0x005A` and `term_counter = last + 0x0400` +are wrong; the device's response under that path is a tiny 101-byte device-side terminator +(arrived only after we walked the entire post-event buffer), not the proper file footer. + +### SUB 5A — fixed metadata pages 0x1002 and 0x1004 (NEW 2026-05-01) + +Two chunk addresses are GLOBAL device/session metadata, not event-specific: + +- `counter=0x1002` — first metadata page +- `counter=0x1004` — second metadata page + +These are at fixed absolute addresses in the device's flash buffer. They contain the +session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII +strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the +new walk these strings come from the dedicated metadata pages, not from the sample-chunk +stream. + +BW reads them ONCE per Blastware session (during event 1's download) and caches them. +For SFM, that means: +- Once per call-home / once per `MiniMateClient.connect()` is enough. +- Subsequent events in the same session don't need to re-fetch them. +- Their content does not change when iterating events; only when the user opens + Compliance Setup → Apply on the device or sends a SUB 71 compliance write. + +The contents have not been byte-for-byte decoded yet — first task on the implementation +side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the +strings we currently extract from A5[7]. ### SUB 5A — params are 11 bytes for chunk frames, 10 for termination @@ -140,10 +275,16 @@ counter and streams data for any valid 5A request; using `chunk_num * 0x0400` is confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes. Do not swap them. -### SUB 5A — event-time metadata lives in A5 frame 7 +### SUB 5A — event-time metadata source (UPDATED 2026-05-01) -The bulk stream sends 9+ A5 response frames. Frame 7 (0-indexed) contains the compliance -setup as it existed when the event was recorded: +> **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A +> bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at +> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because +> the broken counter formula was scanning the wrong region. +> +> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and +> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A +> sample-chunk stream itself does NOT contain these strings any more under the new walk. ``` "Project:" → project description @@ -163,26 +304,37 @@ used as the authoritative source. `_decode_a5_metadata_into` therefore only set "Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C record — 5A remains the sole source for those fields and they are set unconditionally. -`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears, -then sends the termination frame. +> ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and +> stops one chunk later) is a workaround for the missing end_offset bound — when the new +> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is +> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM +> frame. -### SUB 5A — end-of-stream signal (confirmed 2026-04-06) +### SUB 5A — end-of-stream — UPDATED 2026-05-01 -After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to -the next chunk request, then goes silent. This is the natural end-of-stream indicator — NOT -a complete A5 frame. `S3FrameParser.bytes_fed` will be 1; no frame is assembled. +> **Previous understanding (now known to be a symptom, not a feature):** "After streaming +> all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was +> not the device's natural end-of-event signal — it was the device's response when SFM had +> walked clean off the end of the addressable buffer region after over-reading by ~5×. +> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the +> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`, +> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser. -Handling: on `TimeoutError`, if `bytes_fed > 0` AND frames were already collected, treat as -graceful end-of-stream, break the loop, and proceed to the termination frame. If `bytes_fed -== 0` with no prior frames, it is a genuine transport failure — re-raise. +The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful +defence-in-depth fallback for malformed events or unexpected device states, but should not +be the primary loop-exit condition. **Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each. Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call in the chunk loop passes `timeout=10.0` explicitly. -**Typical chunk count (BE11529, 1024 sps):** A 9,306-sample event produces 35 chunks before -end-of-stream. Chunks with uniform 1,036-byte data are all-zero ADC samples (post-event -silence). Only the initial variable-size chunks contain actual signal. +**Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):** +A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames. +A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames. +An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames. + +Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks +17-37 containing post-event circular-buffer garbage that corrupted the file body. ### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06) @@ -295,6 +447,55 @@ sends token=0xFE and is NOT used by any caller. `advance_event()` returns `(key4, event_data8)`. Callers (`count_events`, `get_events`) loop while `data8[4:8] != b"\x00\x00\x00\x00"`. +### SUB 0A — WAVEHDR response length distinguishes events from boundaries (NEW 2026-05-01) + +When iterating events with the "Download All" pattern (1E → 0A → 1F → 0A → 1F → …), the +DATA_LENGTH at `data_rsp.data[5]` (= the byte BW echoes back as the offset for the data +fetch step) takes one of two values: + +| WAVEHDR offset | Meaning | +|---|---| +| `0x46` (= 70) | Real event start key — there is event data at this address | +| `0x2C` (= 44) | Boundary marker between events — this key is the END of the previous event AND the START key for the empty space after it (or is the next event's pre-header) | + +Confirmed from the 5-1-26 "Download All" capture: + +``` +0A(key=01110000) → off=0x46 ← event 1 real start +1F → key=011121F2 +0A(key=011121F2) → off=0x2C ← event 1 END / event 2 boundary +1F → key=01112238 +0A(key=01112238) → off=0x46 ← event 2 real start (= boundary + 0x46) +1F → key=0111417E +0A(key=0111417E) → off=0x2C ← event 2 END / next-empty marker +1F → null sentinel +``` + +This is why event 2's first 5A chunk is at `start_key + 0x46` — that's the address of the +"real start" 0x46-record, distinct from the `0x2C`-record at the raw boundary. Use the +`0x46` keys as the input to `read_bulk_waveform_stream`, not the `0x2C` keys. + +For event 1 only (start_key[2:4] = 0x0000) BW probes at counter=0x0000 directly, which is +the `0x46`-keyed start record. Subsequent events use `start_key + 0x46`. + +**Practical iteration pattern (replaces the old 1E/1F walk for downloads):** + +``` +Setup: SERIAL × 2 → CHCFG → 1E (token=0x00) → key0 +For each event: + 0A(cur_key) → DATA_LENGTH = 0x46 (real) or 0x2C (boundary) + 1F (token=0x00) → next_key + if length was 0x46: → cur_key is a real event; queue it for download + cur_key = next_key + if next_key all-zero null sentinel: stop + +Then for each queued real-event key: + download_event(key) → 5A bulk stream with STRT-bounded chunk walk +``` + +This is what BW does in the 5-1-26 "Download All" capture — it walks the full event chain +collecting `(key, length)` tuples first, *then* downloads each event using the `0x46` keys. + ### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce) `read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where: @@ -386,7 +587,9 @@ bytes `\x01\x2c` = 300 (5-minute default histogram interval); changes when inter | Offset | Field | Format | Notes | |---|---|---|---| -| anchor − 7 (write) / anchor − 8 (read) | recording_mode | uint8 | E5 read has extra `0x10` at anchor−7 | +| anchor − 9 | mode_prefix | uint8 | `0x00` for Single Shot / Continuous; `0x10` for Histogram (DLE prefix in E5 encoding) and Histogram+Continuous (actual config byte). See "compliance_raw DLE encoding" note below. | +| anchor − 8 | recording_mode | uint8 | **Same offset for both read and write** — confirmed 2026-04-21. `_encode_compliance_config` writes `buf[anc-8]`. NOTE: for Histogram (0x03), E5 encodes the value as `0x10 0x03` so compliance_raw[anc-9]=0x10, compliance_raw[anc-8]=0x03. | +| anchor − 7 | constant | `0x10` | Always `0x10` in both E5 read and BW write payloads (not a DLE marker — it is part of the sample_rate field area). Do NOT overwrite. | | anchor − 6 | sample_rate | uint16 BE | same in read & write | | anchor − 4 | histogram_interval_sec | uint16 BE | seconds; same in read & write ✅ 2026-04-20 | | anchor − 2 | `0x00 0x00` | padding | | @@ -395,15 +598,42 @@ bytes `\x01\x2c` = 300 (5-minute default histogram interval); changes when inter **recording_mode enum** (confirmed 2026-04-20 from 4-20-26 captures): -| Value | Mode | -|---|---| -| `0x00` | Single Shot | -| `0x01` | Continuous | -| `0x02` | ❓ not observed | -| `0x03` | Histogram | -| `0x04` | Histogram + Continuous | +| Value | Mode | anchor-9 in compliance_raw | +|---|---|---| +| `0x00` | Single Shot | `0x00` | +| `0x01` | Continuous | `0x00` | +| `0x02` | ❓ not observed | ❓ | +| `0x03` | Histogram | `0x10` (DLE prefix from E5 wire encoding of 0x03) | +| `0x04` | Histogram + Continuous | `0x10` (actual config byte for this mode) | -**DLE escaping in write frames — CONFIRMED 2026-04-20:** Write frame data payloads DO escape `0x03` (ETX) bytes with a `0x10` DLE prefix. For histogram_interval = 900 (0x0384), the wire carries `10 03 84` — the `0x03` high byte is preceded by a DLE escape. After DLE destuffing (`10 XX → XX`), the logical field value is correctly `03 84` = 900. The CLAUDE.md claim that write frame data is "written RAW" was incorrect; at minimum ETX (0x03) bytes are escaped. S3FrameParser handles this transparently so the decoded `compliance_raw` always contains logical (destuffed) bytes. +**compliance_raw DLE encoding — IMPORTANT (confirmed 2026-04-21 from 4-20-26 captures):** +`compliance_raw` (returned by `read_compliance_config()`) is NOT purely logical bytes — it is +the wire-encoded representation where `0x03` bytes in the config are preceded by a `0x10` DLE +prefix (because S3FrameParser preserves DLE+ETX inner-frame pairs as two literal bytes). + +Consequences: +- When recording_mode = `0x03` (Histogram), `compliance_raw[anc-9] = 0x10` (DLE prefix) and + `compliance_raw[anc-8] = 0x03` (the value). The anchor position is +1 compared to modes + without `0x03` bytes before the anchor. +- For Histogram+Continuous (`0x04`), `compliance_raw[anc-9] = 0x10` for a different reason: + it is an actual stored config byte, not a DLE prefix. +- The anchor search (`buf.find(b'\xbe\x80\x00\x00\x00\x00', 0, 150)`) correctly locates + the anchor regardless of these mode-dependent shifts. +- When SFM writes recording_mode and round-trips the rest verbatim, the byte at `anc-9` is + preserved from the previous read. This means transitioning Histogram→other modes via SFM + leaves a `0x10` at `anc-9`. The device stores it as a literal byte; it does not affect + recording mode operation (which is at `anc-8`), but differs from what BW writes. This is a + known minor discrepancy that does not impact device behavior. +- **Histogram recording mode (0x03) write via SFM**: untested. When starting from a mode with + `anc-9 = 0x00`, SFM writes bare `0x03` at anc-8. BW would write `0x10 0x03`. Device likely + accepts both (write frames probably use offset/length for framing, not ETX scanning). + +**DLE escaping in write frames — confirmed 2026-04-20:** Blastware escapes `0x03` bytes in +write frame data as `0x10 0x03` on the wire (defensive ETX escaping). Our `build_bw_write_frame` +does NOT do this escaping — it sends data bytes raw. Device acceptance of bare `0x03` bytes +in write frame data is confirmed for the tested modes (Single Shot, Continuous, Histogram+Continuous +where `0x10 0x03` already appears from round-tripping). Histogram mode (bare `0x03` write from +non-Histogram starting state) has not been directly tested. ### SUB 0C — Waveform Record (210 bytes = data[11:11+0xD2]) @@ -490,6 +720,8 @@ All DB endpoints are read-only except `PATCH /db/events/{id}/false_trigger`. | 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture | | 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence; only 1 event stored so token=0xFE appeared to work | | 4-3-26 | `bridges/captures/4-3-26/` | Browse-mode S3 capture with 2+ events — confirmed all-zero params for 1F, 1F response layout, null sentinel, 0A context requirement | +| 4-27-26 | `bridges/captures/4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof that SFM was over-reading 5× past event end. BW reads 14 chunks at 0x0200 increments + TERM at end_offset; SFM was reading 37 chunks at 0x0400 increments. STRT end_key field located. | +| 5-1-26 | `bridges/captures/5-1-26/comcheck/` | Three sub-captures: SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945`/`_171216`). Confirmed: TERM frame formula across 3 events; metadata pages 0x1002/0x1004 are global (read once per session); event-1 vs event-N chunk-pattern split; WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundaries. | --- @@ -1067,9 +1299,53 @@ body) because writing a dial string may require DLE escaping for embedded contro - **Database** — SQLite store for events + monitor log entries; dedup by key; queryable - **Histograms** — decode histogram-mode A5 data (noise floor tracking) +- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed working for Continuous mode events (2026-04-23):** SFM-generated file opens in Blastware, shows correct PPV/waveform/timestamp. File is ~200 bytes shorter than BW (missing last ADC tail slice) — all measurements correct. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that create spurious STRT markers in the body). Extension mapping: **CONFIRMED FALSE 2026-04-21** — extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<4-char-base36-stem>` + + **Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units). + + **Stem encoding (FULLY CONFIRMED 2026-04-22):** stem = 4-char base-36 of `floor(total_seconds / 1296)` where `total_seconds = (event_local_time − 1985-01-01T00:00:00_local)` in seconds. Epoch = `1985-01-01 00:00:00` device local time — confirmed against 3,248 files from 10-year production archive with zero errors. Decode: `event_time = datetime(1985,1,1) + timedelta(seconds=stem_int*1296 + ab_int)`. Example: P036L318.C80H → BE14036, 2025-05-26 15:00:08, Full Histogram. +- **Blastware filename extension — NEW FIRMWARE FULLY DECODED (confirmed 2026-04-21, further confirmed 2026-04-22 from 10-year production archive frequency analysis):** + + Extension format = `AB0T` (4 chars): + - `AB` = 2-char base-36 encoding of `total_seconds % 1296` (seconds within the 21.6-min window, 0–1295); `A = value // 36`, `B = value % 36` + - `0` = always literal digit zero (third character, invariant) + - `T` = event type: `W` = Full Waveform, `H` = Full Histogram + + Combined with the 4-char stem, the full filename encodes a complete second-resolution timestamp. Verified against three S353L4H0.{3M0W,8S0H,9X0W} events (all match to the second) plus large-scale frequency analysis of a 10-year archive. + + **3-day cycle property (confirmed 2026-04-22):** A unit recording at a fixed daily time cycles through exactly **3 extensions** with a 3-day period. Each calendar day shifts `total_seconds % 1296` by 864 (since `86400 % 1296 = 864`). The cycle repeats every 3 days because `gcd(1296, 864) = 432` and `1296 / 432 = 3`. The three extension values are spaced 432 seconds apart. Confirmed from 10-year archive: the top 3 extensions overall were `CE0H` (95 files), `0E0H` (93), `OE0H` (91) — all three are the 3-day cycle of a 06:00:14 daily call-in time (seconds-in-window = 14, 446, 878; all three have `E` as second character because `14 = E` in base-36 and adding 864 never changes `value % 36` since `864 = 24 × 36`). + + **B character invariance:** For a unit recording at a fixed time of day, the second character `B` of the extension (`value % 36`) **never changes** — only the first character `A` cycles through 3 values. This means same-time-of-day files from different dates all share the same `B` character. + + **Old firmware (S338, 3-char extensions ending in `0`):** encoding unknown. Extension is NOT recording mode. `blastware_filename()` returns `.N00` as a placeholder for old-firmware units. + + **Micromate Series 4** uses a different extension format entirely (observed: `IDFH`, `IDFW`). The `AB0T` formula applies only to MiniMate Plus / V10.72 firmware. - Compliance config encoder — build raw write payloads from a `ComplianceConfig` object +- **Test Histogram recording mode (0x03) write via SFM** — confirmed working for Single Shot / Continuous / Histogram+Continuous; Histogram (0x03) needs a live test from a non-Histogram starting state (bare 0x03 in write vs BW's DLE-escaped `10 03`) +- **Compliance write anchor-9 cleanup** — when changing recording_mode via SFM, the byte at anchor-9 is not explicitly managed. A spurious `0x10` may persist after Histogram→other mode transitions. Does not affect device operation but differs from BW's byte-perfect output. - Locate "Sensor Check" byte in compliance config (need capture with Disabled vs Before-monitoring) - Call Home — map time slots 3/4 offsets; add dial_string write support; confirm `modem_power_relay_enabled` - Modem manager — push RV50/RV55 configs via Sierra Wireless API - RV55 DCD/DTR issue — newer RV55 firmware doesn't assert DCD by default; units don't - resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred) \ No newline at end of file + resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred) + +## BW capture reference + +`bridges/captures/` contains the following BW TX + S3 response captures for protocol analysis: + +| Folder / File | Contents | +|---|---| +| `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102–112) | +| `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous | +| `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec | +| `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) | +| `4-20-26/call home settings/` | Call home config read/write captures | +| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check | +| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events (1E/0A/1F iteration confirmed) | +| `4-2-26/` | Download-mode BW TX capture (5A bulk stream, POLL×3 requirement confirmed) | +| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) | +| `mitm/ach_mitm_20260411_001912/` | Full ACH call-home MITM (erase protocol, 0xA3/0x06/0xA2 confirmed) | + +To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern +in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes +inside write frame data (the naive parser terminates early at the escaped `0x03`). \ No newline at end of file diff --git a/README.md b/README.md index 9603039..e0aafb7 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# seismo-relay `v0.12.1` +# seismo-relay `v0.12.6` A ground-up replacement for **Blastware** — Instantel's aging Windows-only software for managing MiniMate Plus seismographs. @@ -18,26 +18,27 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55). ``` seismo-relay/ -├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Console tabs) +├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Download + Console tabs) │ ├── minimateplus/ ← MiniMate Plus client library │ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport │ ├── protocol.py ← DLE frame layer, SUB command dispatch -│ ├── client.py ← High-level client (connect, get_events, push_config, …) +│ ├── client.py ← High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …) │ ├── framing.py ← Frame builders, DLE codec, S3FrameParser -│ └── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, … +│ ├── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig, … +│ └── blastware_file.py ← Write events to Blastware-compatible .AB0 files │ ├── sfm/ ← SFM REST API server (FastAPI, port 8200) -│ ├── server.py ← All device + DB endpoints -│ ├── database.py ← SeismoDb — SQLite persistence layer -│ └── sfm_webapp.html ← Embedded web UI (served at /) +│ ├── server.py ← Live device endpoints + DB query endpoints + caching +│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions, sessions table) +│ └── sfm_webapp.html ← Embedded web UI with Call Home config tab │ ├── bridges/ │ ├── ach_server.py ← Inbound ACH call-home server (main production server) │ ├── ach_mitm.py ← Transparent MITM proxy for capturing BW sessions │ ├── s3-bridge/ ← RS-232 serial bridge (capture tool) │ ├── tcp_serial_bridge.py ← Local TCP↔serial bridge (bench testing) -│ ├── gui_bridge.py ← Standalone bridge GUI +│ ├── gui_bridge.py ← Standalone bridge GUI with raw capture checkboxes │ └── raw_capture.py ← Simple raw capture tool │ ├── parsers/ @@ -101,21 +102,28 @@ python seismo_lab.py Each call dials the device, does its work, and closes the connection. TCP connections are retried once on `ProtocolError` to handle cold-boot timing. -**Caching** — frequently-polled endpoints are cached in-process to avoid -redundant TCP round-trips: +**In-memory caching** — frequently-polled endpoints avoid redundant TCP round-trips +via a thread-safe `_LiveCache` (plain Python dict + `threading.Lock`): -| Method | URL | Cache | -|--------|-----|-------| +| Method | URL | Cache Strategy | +|--------|-----|---| | `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` | | `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected | | `GET` | `/device/event/{idx}/waveform` | Permanent per event index | -| `GET` | `/device/monitor/status` | 30-second TTL | +| `GET` | `/device/monitor/status` | 30-second TTL; invalidated by monitor start/stop | +| `GET` | `/device/call_home` | Fresh read from device (not cached) | | `POST` | `/device/connect` | — | -| `POST` | `/device/config` | Writes compliance config; invalidates cache | -| `POST` | `/device/monitor/start` | Sends SUB 0x96 | -| `POST` | `/device/monitor/stop` | Sends SUB 0x97 | +| `POST` | `/device/config` | Writes compliance config; invalidates info + events cache | +| `POST` | `/device/config/project` | Patches project/client/operator/sensor_location strings | +| `POST` | `/device/monitor/start` | Sends SUB 0x96; immediately evicts status cache | +| `POST` | `/device/monitor/stop` | Sends SUB 0x97; immediately evicts status cache | +| `POST` | `/device/call_home` | Reads, patches specified fields, writes back to device | -All cached endpoints accept `?force=true` to bypass the cache. +**Cache bypass** — All cached endpoints accept `?force=true` to skip the cache and +force a fresh read from the device. + +**Cache stats** — `GET /cache/stats` returns hit/miss counts and TTL info; `DELETE /cache/device` +clears the device cache immediately. Transport query params (supply one set): ``` @@ -152,21 +160,33 @@ client = MiniMateClient(transport=TcpTransport("1.2.3.4", 12345), timeout=30.0) with client: # Read - info = client.connect() # DeviceInfo — serial, firmware, compliance config - count = client.count_events() # Number of stored events - keys = client.list_event_keys() # Fast browse walk — event keys only, no download - events = client.get_events() # Full download: headers + peaks + metadata - monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag - log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records) + info = client.connect() # DeviceInfo — serial, firmware, compliance config + count = client.count_events() # Number of stored events + keys = client.list_event_keys() # Fast browse walk — event keys only, no download + events = client.get_events() # Full download: headers + peaks + metadata + monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag + log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records) + ach_cfg = client.get_call_home_config() # Auto Call Home settings (SUB 0x2C) # Write client.apply_config( sample_rate=1024, + recording_mode="Continuous", # Single Shot / Continuous / Histogram / Histogram+Continuous + histogram_interval_sec=15, # 2, 5, 15, 60, 300, 900 trigger_level_geo=0.5, + geo_range="Normal", # Normal (10.000 in/s) / Sensitive (1.25 in/s) project="Bridge Inspection 2026", client_name="City of Portland", operator="B. Harrison", ) + + client.set_call_home_config( + auto_call_home_enabled=True, + after_event_recorded=True, + at_specified_times=True, + time1_hour=18, time1_min=30, # 6:30 PM + time2_hour=6, time2_min=0, # 6:00 AM + ) # Control client.start_monitoring() # SUB 0x96 @@ -182,18 +202,20 @@ existed at record time — not backfilled from the current compliance config. ## Database -`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode). -Three tables, all unit-keyed by serial number: +`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode) using the +`SeismoDb` persistence layer. Four tables, all unit-keyed by serial number: | Table | Key | Contents | |-------|-----|----------| -| `ach_sessions` | UUID | Per-call-home audit record: serial, peer IP, events_downloaded, duration | -| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, PPV per channel, project/client/operator strings, false_trigger flag | -| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: start/stop time, duration, geo threshold | +| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds | +| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag | +| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips | +| `events.false_trigger` | Boolean flag | PATCH endpoint to mark/unmark false triggers for review | -Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs -never produce duplicate rows. Post-erase key reuse is handled automatically -via the high-water mark in `ach_state.json`. +Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs never +produce duplicate rows. Post-erase key reuse is handled automatically via the +high-water mark in `ach_state.json`. Key-based state tracking allows correct +handling of device erasures (external or post-download). --- @@ -231,6 +253,27 @@ Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/insta --- +## Compliance Config Features (v0.12.2–v0.12.3) + +The REST API and web UI expose full control over device compliance settings: + +- **Recording Mode** (Single Shot / Continuous / Histogram / Histogram+Continuous) +- **Sample Rate** (1024 / 2048 / 4096 sps) +- **Record Time** (float, seconds) +- **Histogram Interval** (2s, 5s, 15s, 1m, 5m, 15m) — when recording mode includes histogram +- **Geo Trigger Levels** (float, in/s per channel) +- **Geo Maximum Range** (Normal 10.000 in/s / Sensitive 1.250 in/s per channel) +- **Project / Client / Operator / Sensor Location** (ASCII strings) + +Auto Call Home config: +- **Auto Call Home Enable** (bool) +- **Dial String** (read-only; 40-byte ASCII) +- **Trigger on Event** (bool) +- **Scheduled Call-Ins** (two time slots with HH:MM each) +- **Retry Settings** (count, delay, connection timeout, warm-up time) + +--- + ## Requirements ```bash @@ -252,17 +295,55 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows. --- -## Roadmap +## Key Features (v0.10–v0.12) + +**Device support (v0.12.5):** +- [x] Full read/write/erase pipelines +- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings) +- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries) +- [x] Monitor control (start/stop, status polling, battery/memory) +- [x] Monitor log entries (continuous monitoring intervals without full waveform download) + +**Data persistence (v0.11):** +- [x] SQLite database (`seismo_relay.db`) with 4 tables: ach_sessions, events, monitor_log, plus false_trigger flag +- [x] Deduplication by waveform key (handles re-runs and repeat call-homes) +- [x] Post-erase key-reuse detection (tracks high-water mark) +- [x] Session state (`ach_state.json`) with downloaded keys and max key + +**REST API (v0.12.1):** +- [x] Live device endpoints with in-memory caching (`_LiveCache`) +- [x] Cache statistics (`/cache/stats`) and manual invalidation (`/cache/device`) +- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH) +- [x] Call Home config read/write endpoints +- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`) + +**File output (v0.7+):** +- [x] Blastware-compatible `.AB0` file generation (waveform + metadata) +- [x] Multi-channel waveform decode from SUB 5A bulk stream +- [x] Second-resolution timestamp encoding in Blastware filename + +**Capture tools (v0.12.5):** +- [x] Serial-to-TCP bridge with raw BW/S3 capture (s3_bridge.py, defaults to auto-capture) +- [x] GUI bridge with raw capture checkboxes (gui_bridge.py) +- [x] ACH inbound server with bidirectional capture (ach_server.py saves raw_tx + raw_rx) +- [x] Transparent TCP MITM proxy for live BW session capture (ach_mitm.py) + +**Analysis tools:** +- [x] s3_analyzer.py — session parser, frame differ, Claude export +- [x] gui_analyzer.py — standalone analyzer GUI +- [x] frame_db.py — SQLite frame database for capture analysis + +**seismo_lab.py GUI (v0.12.5):** +- [x] Bridge tab — Serial/TCP mode selector with raw capture options +- [x] Analyzer tab — BW/S3 capture playback and differencing +- [x] Download tab — Live wire-byte capture during event download (new v0.12.5) +- [x] Console tab — Logging and diagnostics + +## Roadmap (Future) -- [x] Full read pipeline — device info, compliance config, event download with true event-time metadata -- [x] Write commands — push compliance config, trigger thresholds, project strings to device -- [x] Erase all events — confirmed erase sequence from live MITM capture -- [x] Monitor control — start/stop monitoring, read battery/memory/status -- [x] Monitor log entries — decode partial 0x2C records (continuous monitoring intervals) -- [x] ACH inbound server — accept call-home connections, download events, dedup by key -- [x] SQLite persistence — events, monitor log, and session history in `seismo_relay.db` -- [x] SFM REST API — device control + DB query endpoints, live device cache - [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing - [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first) - [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object - [ ] Modem manager — push RV50/RV55 configs via Sierra Wireless API +- [ ] Histogram mode recording support (5A stream analysis for mode 0x03) +- [ ] Call Home dial_string write support (requires DLE escaping for embedded control characters) diff --git a/bridges/ach_server.py b/bridges/ach_server.py index 65fb9e9..edd4f2e 100644 --- a/bridges/ach_server.py +++ b/bridges/ach_server.py @@ -35,6 +35,7 @@ Output per session device_info.json — serial number, firmware version, calibration date, etc. events.json — all events: timestamp, PPV per channel, peaks, metadata raw_rx_.bin — raw bytes from the device (S3 side) for Analyzer + raw_tx_.bin — raw bytes we sent to the device (BW side) for Analyzer session_.log — detailed protocol log What to look for @@ -172,16 +173,24 @@ class AchSession: transport = SocketTransport(self.sock, peer=self.peer) # Collect raw bytes in memory until startup succeeds, then flush to disk. - raw_buf: list[bytes] = [] - _orig_read = transport.read + raw_rx_buf: list[bytes] = [] # device → us (S3 side) + raw_tx_buf: list[bytes] = [] # us → device (BW side) + _orig_read = transport.read + _orig_write = transport.write def tapped_read(n: int) -> bytes: data = _orig_read(n) if data: - raw_buf.append(data) + raw_rx_buf.append(data) return data - transport.read = tapped_read # type: ignore[method-assign] + def tapped_write(data: bytes) -> None: + _orig_write(data) + if data: + raw_tx_buf.append(data) + + transport.read = tapped_read # type: ignore[method-assign] + transport.write = tapped_write # type: ignore[method-assign] serial: Optional[str] = None @@ -201,23 +210,35 @@ class AchSession: # Startup succeeded — this is a real unit. Create session dir now. session_dir = self.output_dir / f"ach_inbound_{ts}" session_dir.mkdir(parents=True, exist_ok=True) - log_path = session_dir / f"session_{ts}.log" - raw_path = session_dir / f"raw_rx_{ts}.bin" + log_path = session_dir / f"session_{ts}.log" + raw_rx_path = session_dir / f"raw_rx_{ts}.bin" # device → us (S3 side) + raw_tx_path = session_dir / f"raw_tx_{ts}.bin" # us → device (BW side) - # Flush buffered raw bytes to file and switch to direct file writes. - raw_fh = open(raw_path, "wb") - for chunk in raw_buf: - raw_fh.write(chunk) - raw_buf.clear() + # Flush buffered bytes to files and switch to direct file writes. + raw_rx_fh = open(raw_rx_path, "wb") + raw_tx_fh = open(raw_tx_path, "wb") + for chunk in raw_rx_buf: + raw_rx_fh.write(chunk) + for chunk in raw_tx_buf: + raw_tx_fh.write(chunk) + raw_rx_buf.clear() + raw_tx_buf.clear() def tapped_read_file(n: int) -> bytes: data = _orig_read(n) if data: - raw_fh.write(data) - raw_fh.flush() + raw_rx_fh.write(data) + raw_rx_fh.flush() return data - transport.read = tapped_read_file # type: ignore[method-assign] + def tapped_write_file(data: bytes) -> None: + _orig_write(data) + if data: + raw_tx_fh.write(data) + raw_tx_fh.flush() + + transport.read = tapped_read_file # type: ignore[method-assign] + transport.write = tapped_write_file # type: ignore[method-assign] # Wire up file handler now that the session dir exists. fh = logging.FileHandler(log_path, encoding="utf-8") @@ -530,7 +551,8 @@ class AchSession: log.warning(" [WARN] Failed to restart monitoring: %s", exc) finally: - raw_fh.close() + raw_rx_fh.close() + raw_tx_fh.close() client.close() # closes transport / socket cleanly root_logger.removeHandler(fh) fh.close() diff --git a/bridges/gui_bridge.py b/bridges/gui_bridge.py index c0ae686..7028baf 100644 --- a/bridges/gui_bridge.py +++ b/bridges/gui_bridge.py @@ -58,16 +58,24 @@ class BridgeGUI(tk.Tk): tk.Entry(self, textvariable=self.logdir_var, width=24).grid(row=1, column=3, sticky="we", **pad) tk.Button(self, text="Browse", command=self._choose_dir).grid(row=1, column=4, sticky="w", **pad) - # Row 2: Raw taps - self.raw_bw_var = tk.StringVar(value="") - self.raw_s3_var = tk.StringVar(value="") - tk.Checkbutton(self, text="Save BW->S3 raw", command=self._toggle_raw_bw, onvalue="1", offvalue="").grid(row=2, column=0, sticky="w", **pad) - tk.Entry(self, textvariable=self.raw_bw_var, width=28).grid(row=2, column=1, columnspan=3, sticky="we", **pad) - tk.Button(self, text="...", command=lambda: self._choose_file(self.raw_bw_var, "bw")).grid(row=2, column=4, **pad) + # Row 2: Raw taps — ON by default; "auto" = timestamped name; blank checkbox = disabled + self.raw_bw_enabled = tk.IntVar(value=1) + self.raw_s3_enabled = tk.IntVar(value=1) + # Path fields: empty means "auto" (bridge picks a timestamped name) + self.raw_bw_path_var = tk.StringVar(value="") + self.raw_s3_path_var = tk.StringVar(value="") - tk.Checkbutton(self, text="Save S3->BW raw", command=self._toggle_raw_s3, onvalue="1", offvalue="").grid(row=3, column=0, sticky="w", **pad) - tk.Entry(self, textvariable=self.raw_s3_var, width=28).grid(row=3, column=1, columnspan=3, sticky="we", **pad) - tk.Button(self, text="...", command=lambda: self._choose_file(self.raw_s3_var, "s3")).grid(row=3, column=4, **pad) + tk.Checkbutton(self, text="BW→S3 raw (auto)", variable=self.raw_bw_enabled, + command=self._toggle_raw_bw).grid(row=2, column=0, sticky="w", **pad) + tk.Entry(self, textvariable=self.raw_bw_path_var, width=28, + fg="grey").grid(row=2, column=1, columnspan=3, sticky="we", **pad) + tk.Button(self, text="...", command=lambda: self._choose_file(self.raw_bw_path_var, "bw")).grid(row=2, column=4, **pad) + + tk.Checkbutton(self, text="S3→BW raw (auto)", variable=self.raw_s3_enabled, + command=self._toggle_raw_s3).grid(row=3, column=0, sticky="w", **pad) + tk.Entry(self, textvariable=self.raw_s3_path_var, width=28, + fg="grey").grid(row=3, column=1, columnspan=3, sticky="we", **pad) + tk.Button(self, text="...", command=lambda: self._choose_file(self.raw_s3_path_var, "s3")).grid(row=3, column=4, **pad) # Row 4: Status + buttons self.status_var = tk.StringVar(value="Idle") @@ -102,13 +110,11 @@ class BridgeGUI(tk.Tk): var.set(filename) def _toggle_raw_bw(self) -> None: - if not self.raw_bw_var.get(): - # default name - self.raw_bw_var.set(os.path.join(self.logdir_var.get(), "raw_bw.bin")) + # Checkbox toggled — no path action needed; enabled state drives the flag. + pass def _toggle_raw_s3(self) -> None: - if not self.raw_s3_var.get(): - self.raw_s3_var.set(os.path.join(self.logdir_var.get(), "raw_s3.bin")) + pass def start_bridge(self) -> None: if self.process and self.process.poll() is None: @@ -126,23 +132,22 @@ class BridgeGUI(tk.Tk): args = [sys.executable, BRIDGE_PATH, "--bw", bw, "--s3", s3, "--baud", baud, "--logdir", logdir] - ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + # Raw tap flags. + # Checkbox on + empty path → pass "auto" (bridge generates timestamped name). + # Checkbox on + explicit path → pass that path. + # Checkbox off → pass "" to disable (overrides bridge's auto default). + raw_bw_explicit = self.raw_bw_path_var.get().strip() + raw_s3_explicit = self.raw_s3_path_var.get().strip() - raw_bw = self.raw_bw_var.get().strip() - raw_s3 = self.raw_s3_var.get().strip() + if self.raw_bw_enabled.get(): + args += ["--raw-bw", raw_bw_explicit if raw_bw_explicit else "auto"] + else: + args += ["--raw-bw", ""] # explicit disable - # If the user left the default generic name, replace with a timestamped one - # so each session gets its own file. - if raw_bw: - if os.path.basename(raw_bw) in ("raw_bw.bin", "raw_bw"): - raw_bw = os.path.join(os.path.dirname(raw_bw) or logdir, f"raw_bw_{ts}.bin") - self.raw_bw_var.set(raw_bw) - args += ["--raw-bw", raw_bw] - if raw_s3: - if os.path.basename(raw_s3) in ("raw_s3.bin", "raw_s3"): - raw_s3 = os.path.join(os.path.dirname(raw_s3) or logdir, f"raw_s3_{ts}.bin") - self.raw_s3_var.set(raw_s3) - args += ["--raw-s3", raw_s3] + if self.raw_s3_enabled.get(): + args += ["--raw-s3", raw_s3_explicit if raw_s3_explicit else "auto"] + else: + args += ["--raw-s3", ""] # explicit disable try: self.process = subprocess.Popen( diff --git a/bridges/s3-bridge/s3_bridge.py b/bridges/s3-bridge/s3_bridge.py index f3e1770..aa0ecac 100644 --- a/bridges/s3-bridge/s3_bridge.py +++ b/bridges/s3-bridge/s3_bridge.py @@ -390,8 +390,14 @@ def main() -> int: ap.add_argument("--s3", default="COM5", help="S3-side COM port (default: COM5)") ap.add_argument("--baud", type=int, default=38400, help="Baud rate (default: 38400)") ap.add_argument("--logdir", default=".", help="Directory to write session logs into (default: .)") - ap.add_argument("--raw-bw", default=None, help="Optional file to append raw bytes sent from BW->S3 (no headers)") - ap.add_argument("--raw-s3", default=None, help="Optional file to append raw bytes sent from S3->BW (no headers)") + ap.add_argument("--raw-bw", default="auto", + help="File to append raw bytes sent from BW->S3 (no headers). " + "Default 'auto' generates a timestamped name in --logdir. " + "Pass an empty string to disable.") + ap.add_argument("--raw-s3", default="auto", + help="File to append raw bytes sent from S3->BW (no headers). " + "Default 'auto' generates a timestamped name in --logdir. " + "Pass an empty string to disable.") ap.add_argument("--quiet", action="store_true", help="No console heartbeat output") ap.add_argument("--status-every", type=float, default=0.0, help="Seconds between console heartbeat lines (default: 0 = off)") args = ap.parse_args() @@ -414,12 +420,16 @@ def main() -> int: # If raw tap flags were passed without a path (bare --raw-bw / --raw-s3), # or if the sentinel value "auto" is used, generate a timestamped name. # If a specific path was provided, use it as-is (caller's responsibility). - raw_bw_path = args.raw_bw - raw_s3_path = args.raw_s3 - if raw_bw_path in (None, "", "auto"): - raw_bw_path = os.path.join(args.logdir, f"raw_bw_{ts}.bin") if args.raw_bw is not None else None - if raw_s3_path in (None, "", "auto"): - raw_s3_path = os.path.join(args.logdir, f"raw_s3_{ts}.bin") if args.raw_s3 is not None else None + # Resolve raw tap paths. + # "auto" (default) → timestamped file in logdir (always captured). + # Explicit path → use verbatim. + # None or "" → disabled (pass --raw-bw "" to suppress capture). + raw_bw_path: Optional[str] = args.raw_bw if args.raw_bw else None + raw_s3_path: Optional[str] = args.raw_s3 if args.raw_s3 else None + if raw_bw_path == "auto": + raw_bw_path = os.path.join(args.logdir, f"raw_bw_{ts}.bin") + if raw_s3_path == "auto": + raw_s3_path = os.path.join(args.logdir, f"raw_s3_{ts}.bin") logger = SessionLogger(log_path, bin_path, raw_bw_path=raw_bw_path, raw_s3_path=raw_s3_path) diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index 7b49bcd..0d90732 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -104,9 +104,13 @@ | 2026-04-11 | §7.11 (NEW) | **NEW — §7.11 Erase-All Protocol added.** Full wire sequence, SUB 0x06 storage range payload layout, post-erase key counter reset (resets to `0x01110000`). Confirmed from 4-11-26 MITM capture of live Blastware ACH session. | | 2026-04-11 | §14.6 | **RESOLVED — ACH Session Lifecycle is no longer "Future".** `bridges/ach_server.py` fully implements inbound ACH: POLL handshake, device info, event download. State tracked via `ach_state.json` (key-based, with `max_downloaded_key` for post-erase detection). `--clear-after-download` flag added for the standard delete-after-upload workflow. | | 2026-04-17 | §7.6.2, §14 | **RESOLVED — Float 6.206053 at channel_label+28 is the ADC-to-velocity scale factor.** Confirmed from Series III Interface Handbook §4.5 formula: `Range (×1) = 1.61133 V / Sensitivity (V/unit)`. For the standard Instantel geophone at Normal range (10.000 in/s): Sensitivity = 1.61133 / 10 = 0.161133 V/(in/s). The stored value is the **inverse sensitivity** = 1/0.161133 = **6.206053 (in/s)/V**. Cross-check: 1.61133 V × 6.206053 = 10.000 in/s ✅. The firmware uses it as: `PPV (in/s) = ADC_voltage (V) × 6.206053`. Value is identical on all Instantel standard geophones — it is a hardware/firmware constant, NOT a user-configurable setting. Do NOT write this field. Open question §14 item "Max Geo Range float 6.2061" is now **RESOLVED**. | -| 2026-04-20 | §7.6.4 (NEW), §7.9, Appendix B | **CONFIRMED — Recording Mode byte location.** Three targeted captures (4-20-26) confirmed `recording_mode` at **`cfg[5]`** in the SUB 71 write payload (3-chunk compliance write). Method: single Blastware session, one initial E5 config pull, then three sequential "Send to unit" writes changing Recording Mode only. Diff of SUB 71 chunk-1 payloads: only `cfg[5]` and `cfg[1024]` changed; `cfg[1024]` delta exactly equals `cfg[5]` delta (chunk running checksum). In the E5 read response (sub-frame 1, page=0x0010), the field is at **`data[17]`** (= **anchor − 4** from the 10-byte anchor), one position earlier than in the write payload due to an extra `0x10` byte at `data[18]` present only in the read format. Enum: `0x00`=Single Shot, `0x01`=Continuous, `0x03`=Histogram, `0x04`=Histogram+Continuous. `0x02` value not yet observed. See §7.6.4 for full details. | +| 2026-04-20 | §7.6.4 (NEW), §7.9, Appendix B | **CONFIRMED — Recording Mode byte location.** Three targeted captures (4-20-26) confirmed `recording_mode` at anchor−8 in both the E5 read payload and the BW write payload (6-byte anchor `\xbe\x80\x00\x00\x00\x00`). BW write payload and E5 read payload are **byte-identical** around the anchor region — Blastware round-trips the wire-encoded E5 bytes verbatim with only the target field modified. Anchor position varies by ±1 depending on whether recording_mode = 0x03 (Histogram), because E5 wire-encodes `0x03` as the inner DLE+ETX pair `\x10\x03` (2 bytes), which S3FrameParser preserves as two literal bytes in `compliance_raw`. Enum: `0x00`=Single Shot, `0x01`=Continuous, `0x03`=Histogram, `0x04`=Histogram+Continuous. `0x02` value not yet observed. The byte at anchor−9 is `0x00` for Single Shot / Continuous, and `0x10` for Histogram (DLE prefix from E5 encoding) and Histogram+Continuous (actual config byte). See §7.6.4 for full details. | +| 2026-04-21 | Appendix D (NEW) | **NEW — Blastware .N00 and .MLG file formats fully decoded.** `minimateplus/blastware_file.py` implements `write_n00()` and `write_mlg()`. N00 file format confirmed: 22B header + 21B STRT record + variable body + 26B footer. Body reconstructed from A5 bulk waveform stream frames with per-frame skip amounts (probe=7+strt_pos+21, A5[1]=13, A5[2+]=12, terminator=11) and DLE strip rule (strip `0x10` before `{0x02,0x03,0x04}`, keep following byte). Footer extracted verbatim from terminator frame's last 26 bytes. Split-pair edge case: when `frame.data[-1]==0x10` and `chk_byte∈{0x02,0x03,0x04}`, reunite both bytes before stripping and always remove trailing chk_byte (`stripped[:-1]`) — chk_byte is checksum, not payload. STRT record must be copied verbatim from A5[0]; bytes [10:20] are device-specific and cannot be reconstructed from Event fields. `write_n00` verified byte-perfect against `M529LIY6.N00` from 4-3-26-multi_event capture. MLG format: 308B header + N×292B records; CRC algorithm unknown (write as 0x0000). | +| 2026-04-21 | Appendix D §D.5 (NEW) | **NEW — Blastware filename encoding fully decoded.** Serial prefix: `chr(ord('B') + floor(serial/1000))` + last 3 digits zero-padded. Stem: 4-char base-36 of `floor(total_seconds/1296)`. Extension: `AB0` for manual/direct downloads (3 chars), `AB0W` or `AB0H` for ACH/call-home downloads (4 chars), where `AB` = 2-char base-36 of `total_seconds % 1296` and W/H = waveform/histogram. Epoch = 1985-01-01 00:00:00 device local time. Confirmed against 3,248 files from 10-year production archive with zero errors. 3-day cycle property: same daily recording time cycles through 3 extensions (864s/day shift, period=3 days). `blastware_filename(event, serial, ach=False)` implements full formula. | +| 2026-04-21 | §7.6.2, §5.3 | **CORRECTED — compliance_raw contains wire-encoded bytes, NOT logical bytes.** S3FrameParser appends DLE+ETX inner-frame pairs as two literal bytes to the frame body. Any `0x03` values in the compliance config appear in `compliance_raw` as `\x10\x03` (two bytes), not as a single `0x03`. The previous claim "S3FrameParser handles this transparently so compliance_raw contains logical (destuffed) bytes" was wrong. Consequence: `compliance_raw` is the wire-encoded E5 payload; anchor-relative reads work correctly because the anchor position automatically accounts for any DLE-encoded bytes before it. For write-back, round-tripping `compliance_raw` verbatim sends the correct wire bytes to the device. **DLE ETX escaping in write frames:** Blastware escapes `0x03` bytes in write frame data as `\x10\x03` on wire; our `build_bw_write_frame` does not (writes data raw). Device is confirmed to accept raw writes for all tested modes — likely uses the offset/length field for write frame framing, not ETX scanning. | | 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. | | 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. | +| 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. | --- @@ -1223,30 +1227,45 @@ Two critical differences from `build_bw_frame`: 2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen, only `XX` is added to the running sum; lone bytes are added normally. -#### 7.8.2 Request Sequence +#### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol) + +> ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new +> BW MITM captures (4-27-26 + 5-1-26) prove the actual chunk increment is **0x0200**, the +> chunk loop is bounded by `end_offset` from the STRT record (not by chunk count or by a +> device-side timeout), and the TERM frame's `offset_word=0x005A` magic is incorrect — the +> real TERM offset_word is computed from `end_offset` and the last chunk address. Under the +> deprecated formula SFM over-reads roughly 5× past the actual event end into post-event +> circular-buffer garbage, corrupting reconstructed Blastware files for any waveform ≥ 2 sec. +> +> The whole "stop_after_metadata + one extra chunk + 0e 08 footer" workaround in this +> section was compensating for the missing end_offset bound. It is obsoleted by the +> STRT-bounded walk in §7.8.5. +> +> **Read this section for historical context only.** For the correct protocol, jump to: +> - §7.8.5 — chunk addressing and the STRT end_offset +> - §7.8.6 — TERM frame formula +> - §7.8.7 — fixed metadata pages 0x1002 and 0x1004 | Frame | offset_word | counter | params | Purpose | |---|---|---|---|---| | Probe | `0x1004` | `0x0000` | 10 bytes (`bulk_waveform_params(0)`) | Initiate transfer | -| Chunk 1 | `0x1004` | `0x0400` | 11 bytes | First data chunk | -| Chunk 2 | `0x1004` | `0x0800` | 11 bytes | Second chunk | -| Chunk N | `0x1004` | `N * 0x0400` | 11 bytes | Nth chunk | +| Chunk 1 | `0x1004` | `max(key4[2:4], 0x0400)` | 11 bytes | First data chunk | +| Chunk 2 | `0x1004` | `max(key4[2:4], 0x0400) + 0x0400` | 11 bytes | Second chunk | +| Chunk N | `0x1004` | `max(key4[2:4], 0x0400) + (N-1) * 0x0400` | 11 bytes | Nth chunk | | … | … | … | … | … | -| Termination | `0x005A` | `last + 0x0400` | 10 bytes | End transfer | +| Termination | `0x005A` | `max(key4[2:4], 0x0400) + N * 0x0400` | 10 bytes | End transfer | -> ⚠️ **2026-04-06 CORRECTED — chunk counter is monotonic for ALL chunks.** -> The 4-2-26 BW TX capture showed counter=0x1004 for chunk 1, which was hardcoded as a -> special case. This was a Blastware artifact. Empirically confirmed: counter=0x0400 for -> chunk 1 works correctly; counter=0x1004 causes the device to time out. The device does -> NOT strictly validate the counter value — it streams data for any valid 5A request for -> the given key. Use `chunk_num * 0x0400` (monotonic) for all chunks. -> BW's true internal formula is `key4[2:4] + n * 0x0400`. For event 1 (key `01110000`) -> this equals `n * 0x0400` since `key4[2:4] = 0x0000`. The monotonic formula is correct -> for all keys encountered on this device. +> Historical correction notes (left in place to deter re-derivation of the same wrong formula): +> the table above was the result of three iterative "corrections" between 2026-04-06 and +> 2026-04-26 that progressively narrowed in on the wrong answer because every test was on +> events with `key4[2:4]=0` and the device responds to whatever counter you ask for. The +> 5-1-26 captures with a non-zero start_key event (`01112238`) finally exposed the bug. -The `stop_after_metadata=True` flag causes the loop to stop as soon as `b"Project:"` is -found in the accumulated A5 frame data, typically after 7–9 chunks. A termination frame -is always sent before returning. +The `stop_after_metadata=True` flag (deprecated as a primary loop-exit) scanned for +`b"Project:"` in the chunk stream because the metadata strings happened to be reachable +when the broken 0x0400-step walk passed the global metadata pages at 0x1002/0x1004. Under +the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004, +not from the sample-chunk stream — see §7.8.7. #### 7.8.3 A5 Frame Layout @@ -1264,15 +1283,19 @@ for ASCII labels with a null-terminated value read: All five fields reflect the **setup at event-record time**, not the current device config. -#### 7.8.4 End-of-Stream Behaviour and Chunk Timing +#### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01 -> ✅ **Confirmed 2026-04-06** — empirical observation on BE11529 (S338.17) over TCP/cellular. +> The "1 raw byte then silence" pattern documented below was originally interpreted as +> "the device's natural end-of-event signal." The 5-1-26 captures show this is actually +> the device's response when the requester has walked **past** the addressable buffer +> region (i.e. ~5× past the actual event end under the deprecated 0x0400-step walk). +> Under the corrected STRT-bounded walk (§7.8.5), the stream ends cleanly with the TERM +> frame's response — no timeout, no 1-byte teaser. The fallback below remains useful as +> defensive handling for malformed events but should not be the primary loop-exit. -**End-of-stream signal:** After sending all waveform chunks, the device sends exactly **1 raw byte** in response to the next chunk request, then goes silent. This byte is not a complete DLE-framed A5 response — `S3FrameParser.bytes_fed` reports 1 and no frame is ever assembled. This is the device's natural end-of-stream indicator. - -Handling logic in `read_bulk_waveform_stream`: +**Defensive fallback handling in `read_bulk_waveform_stream`:** ``` -TimeoutError caught: +TimeoutError caught (rare under corrected walk): if bytes_fed > 0 AND frames already collected: → graceful end-of-stream; break loop; proceed to termination frame else (bytes_fed == 0, no prior frames): @@ -1284,14 +1307,15 @@ TimeoutError caught: | Metric | Observed value | |---|---| | Chunk response time | ~1 s per chunk | -| Chunks for a 9,306-sample event | 35 chunks | -| Data per chunk (active signal) | 1,036–1,123 bytes | -| Data per chunk (post-event silence) | 1,036 bytes (uniform) | +| Chunks for a 2-sec event (corrected walk) | 14 (12 sample chunks + 2 metadata pages) + TERM | +| Chunks for a 3-sec event (corrected walk) | 18 (16 sample chunks + 2 metadata pages) + TERM | +| Chunks for a continuation event (corrected walk) | ~15 sample chunks + TERM (no metadata reread) | +| Chunks under deprecated walk for 2-3 sec event | 37 (over-reads ~5×) | +| Data per chunk (corrected, 0x0200 size) | ~540–575 bytes wire (= 0x0200 payload + framing) | +| Data per chunk (deprecated 0x0400 step) | 1,036–1,123 bytes wire (= 0x0400 payload + framing) | | Safe recv timeout per chunk | **10 s** (10× typical) | | Default transport timeout | 120 s → ~2-min stall at end-of-stream | -Chunks with uniform 1,036-byte payload (chunks 17–35 in the observed event) contain all-zero ADC samples — the device continues recording silence until the configured record time expires before terminating the stream. - **ADC count-to-physical conversion — ✅ CONFIRMED 2026-04-17:** Raw samples are signed 16-bit integers (−32,768 to +32,767). Source: Interface Handbook §4.5. @@ -1310,6 +1334,186 @@ where `geo_range = 1.61133 V × 6.206053 = 10.000 in/s` is the Normal (Gain=1) f `_decode_a5_waveform()` contains `elif fi == 9: continue` from an earlier assumption that frame index 9 is always the device terminator. For streams with more than 9 frames, frame 9 is live waveform data. The skip discards ~1,070 bytes (~133 sample-sets) per event. Terminator detection should use `page_key == 0x0000`, not frame index. This skip should be removed. +#### 7.8.5 Chunk addressing and the STRT end_offset (NEW 2026-05-01) ✅ + +> ✅ Confirmed across 3 events (4-27-26 + 5-1-26 captures). + +`params[0]` is always `0x00`. `params[1:5]` is a 4-byte absolute device flash-buffer +address — equivalently, "the key of the page being requested." The device returns 0x0200 +(= 512) bytes starting at that address. Increments between consecutive sample chunks are +**0x0200, NOT 0x0400** (the previous 0x0400 figure was a Blastware-side artifact / our +implementation's bug — see §7.8.2). + +##### STRT record (data layout in the first A5 response) + +The first A5 response (the probe response, or the first chunk for continuation events) +contains a **STRT record** at byte offset 17 of `data`: + +``` +data[ 0:14] echoes request: [chunk_size_hi=0x02 / 0x04 ...] [00] [01 11] [counter_hi counter_lo] [00 × 8] [00 12] +data[14:17] 10 03 00 ← inner DLE+ETX frame separator (preserved literally) +data[17:21] "STRT" ← magic +data[21:23] ff fe ← sentinel +data[23:27] end_key ← 4-byte key of where this event ENDS +data[27:31] start_key ← 4-byte key of where this event STARTS +data[31:33] uint16 BE ← ?? sample count or byte count, varies (not yet decoded) +data[33:35] uint16 BE ← ?? +data[35] 0x46 ← record type marker (waveform full record) +data[36:] additional pointers / first sample bytes — content varies by event +``` + +`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**. +Use it to bound the chunk loop and to compute the TERM frame. + +##### Chunk pattern by event location in buffer + +**Event 1 / start_key[2:4] = 0x0000** (first event after erase or wrap): + +``` +1. Probe at counter = 0x0000 (params[1:5] = full key) +2. Read fixed metadata pages counter = 0x1002, then 0x1004 +3. Walk sample chunks counter = 0x0600, 0x0800, …, by 0x0200, + up to but not including end_offset & 0xFE00 +4. TERM (see §7.8.6) +``` + +The range `[0x0046, 0x0600)` is skipped — likely some pre-event firmware-reserved area for +the first slot in a freshly-erased buffer. Harmless to skip; BW does the same. + +**Event 2+ / start_key[2:4] != 0x0000** (continuation events in a populated buffer): + +``` +1. First chunk at counter = start_key[2:4] + 0x0046 ← acts as both probe and first + sample chunk; response carries STRT +2. Walk sample chunks counter += 0x0200 each +3. TERM +``` + +**No metadata-page reads.** Pages 0x1002/0x1004 are session-global and were already read +during event 1 in the same Blastware session. In SFM, treat metadata pages as a once- +per-`MiniMateClient.connect()` (or once-per-call-home) read, not per-event. + +##### Verified end_offset values + +| Capture | start_key | end_key | end_offset | event size | sample-chunk start | +|---|---|---|---|---|---| +| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B | 0x0600 (event-1 case) | +| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B | 0x0600 (event-1 case) | +| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | event 2 size = 0x1F8C = 8,076 B | 0x2238 (= 0x21F2 + 0x46) | + +#### 7.8.6 TERM Frame Formula (NEW 2026-05-01) ✅ + +> ✅ Confirmed across 3 events. Replaces the deprecated `offset_word=0x005A` / `params[2] = key4[2]` formula in §7.8.2. + +The TERM frame fetches the partial last chunk and the file footer. Its response payload +contains the bytes between the last full 0x0200-aligned chunk and `end_offset` — typically +20–520 B — and is **required for reconstructing the Blastware waveform file**. Append the +TERM response data to the chunk stream like any other A5 frame. + +``` +last_chunk_counter = address of last full 0x0200-byte chunk read +next_boundary = last_chunk_counter + 0x0200 +TERM offset_word = end_offset - next_boundary +TERM params[0] = key[0] (= 0x01 on every observed device) +TERM params[1] = key[1] (= 0x11) +TERM params[2] = (next_boundary >> 8) & 0xFF +TERM params[3] = next_boundary & 0xFF +TERM params[4:10] = zeros ← 10-byte params (not 11) + +Frame = build_5a_frame(offset_word, params) +``` + +The device receives `requested_address = (params[2] << 8) | offset_word` (where offset_word +contains both `offset_hi` and `offset_lo` of the 5A frame, with the high bit of offset_hi +being effectively `bit 0 of (end_offset >> 8)`). It reconstructs `end_offset` and replies +with `(end_offset - next_boundary)` bytes of waveform tail starting at `next_boundary`. + +##### Verification + +| Event | end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] | TERM response size | +|---|---|---|---|---|---|---| +| 2-sec | `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ | 208 B | +| 3-sec | `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ | 520 B | +| Event-2 | `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ | (not measured directly; same pattern) | + +Equivalent way to write the formula: +- `offset_word = end_offset & 0x01FF` — low 9 bits of end_offset +- `params[2:4] = (end_offset & 0xFE00) BE` — high 7 bits of end_offset, low byte zeroed + +(The two forms are arithmetically identical to `end_offset - next_boundary` and +`next_boundary` because `next_boundary = end_offset & 0xFE00` whenever the chunk loop +stopped at the last full 0x0200 boundary below end_offset.) + +#### 7.8.7 Fixed Metadata Pages 0x1002 / 0x1004 (NEW 2026-05-01) 🔶 + +> 🔶 Inferred — observed in BW captures but page contents not yet byte-decoded. + +Two chunk addresses are **GLOBAL** device/session metadata, not event-specific: + +- `counter = 0x1002` — first metadata page +- `counter = 0x1004` — second metadata page + +These are at fixed absolute addresses in the device's flash buffer. They contain the +session-start compliance-setup ASCII strings — **Project**, **Client**, **User Name**, +**Seis Loc**, **Extended Notes** — that under the deprecated 0x0400-step walk used to be +discoverable in the sample-chunk stream as "A5 frame 7" content. Under the corrected +0x0200-step walk these strings come exclusively from the dedicated metadata-page reads, +not from sample chunks. + +##### Caching strategy + +BW reads them ONCE per Blastware session, during event 1's download, and caches them. +For SFM: +- Read once per `MiniMateClient.connect()` / once per call-home session. +- Subsequent events in the same session don't need to re-fetch them. +- Their content does not change while iterating events. They DO change when the user + applies a new compliance setup (SUB 71 write) — invalidate the cache then. + +##### TODO — content layout + +The byte-for-byte layout of pages 0x1002 and 0x1004 has not been decoded. First task on +the implementation side: dump both pages from a fresh capture and verify they include all +the strings currently extracted from the deprecated A5 frame 7 of the chunk stream. +Compare to the existing `_decode_a5_metadata_into` parser — same string-search anchors +(`b"Project:"`, `b"Client:"`, `b"User Name:"`, `b"Seis Loc:"`, `b"Extended Notes"`) likely +apply directly. + +#### 7.8.8 "Download All" Sequence (NEW 2026-05-01) ✅ + +> ✅ Confirmed from 5-1-26 "Download All" capture (`raw_*_171216_download_all_2events.bin`). + +Before any 5A traffic, BW's "Download All" pre-walks the entire event chain to map keys +and event boundaries: + +``` +SERIAL × 2 → CHCFG → EVT_KEY (1E, all-zero) → key0 + → WAVEHDR (0A, key0) → off=0x46 (real event start) + → EVT_NEXT (1F, all-zero) → key1 + → WAVEHDR (0A, key1) → off=0x2C (boundary) + → EVT_NEXT → key2 + → WAVEHDR (0A, key2) → off=0x46 (real event start) + → EVT_NEXT → key3 + → WAVEHDR (0A, key3) → off=0x2C (boundary) + → EVT_NEXT → null sentinel +``` + +The DATA_LENGTH at `data_rsp.data[5]` (echoed BW offset for the data fetch step) +disambiguates real events from boundary markers: + +| WAVEHDR offset | Meaning | +|---|---| +| `0x46` (= 70) | Real event start key — this key has event data behind it | +| `0x2C` (= 44) | Boundary marker — this key is the END of the previous event AND the start of the empty/header gap before the next event | + +Pairs: each real event spans `[real_key, next_real_key)` in the buffer. In the example +above: event 1 = `[01110000, 011121F2)`, event 2 = `[01112238, 0111417E)`. Note that the +"end of event 1" key (`011121F2`) is also the "boundary key" that comes BEFORE event 2's +real start key (`01112238`) — they differ by exactly 0x46 bytes (the event header size). + +After the pre-walk completes, BW downloads each `0x46`-keyed event in turn using the 5A +bulk stream protocol from §7.8.5. Use the `0x46` keys, not the `0x2C` keys, as input to +`read_bulk_waveform_stream`. + --- ## 7.9 Compliance Config Field Inventory (Blastware UI, 2026-04-08) ✅ @@ -2244,6 +2448,279 @@ Semantic Interpretation <- settings, events, responses --- +--- + +## Appendix D — Blastware Binary File Formats (.N00 / .MLG / others) + +> ✅ CONFIRMED 2026-04-21 — all fields verified by binary diff of reconstructed vs reference +> files from the 4-3-26-multi_event capture (M529LIY6.N00, BE11529.MLG). +> +> ⚠️ EXTENSION MAPPING REFUTED 2026-04-21 — earlier assumption that extension encodes +> recording mode is **FALSE**. A continuous-mode event produced `.EI0`, not `.9T0`. +> Extension encoding algorithm is unknown. Do not use extension to infer recording mode. + +### D.1 Common File Header (22 bytes) + +All Blastware files (regardless of type) share an 18-byte prefix followed by a 4-byte type tag. + +| Offset | Length | Value | Description | +|---|---|---|---| +| 0x00 | 6 | `10 00 01 80 00 00` | Fixed prefix | +| 0x06 | 10 | `Instantel\x00` | ASCII string | +| 0x10 | 2 | `07 2c` | Fixed suffix | +| 0x12 | 4 | varies | File type tag (see below) | + +**Total header: 22 bytes.** + +**Type tags:** + +| Extension | Type tag | Description | +|---|---|---| +| `.N00` | `00 12 03 00` | Waveform event (confirmed) | +| `.9T0` | `00 12 03 00` | Waveform event — same type tag as .N00 (assumed; not independently confirmed) | +| `.EI0` | `00 12 03 00` | Waveform event — same type tag (assumed; continuous-mode event observed 2026-04-21) | +| `.MLG` | `22 01 0e a0` | Monitor log | + +**Extension encoding — new firmware (V10.72+) FULLY DECODED (confirmed 2026-04-22):** + +The extension differs depending on how the file was saved: + +| Download method | Extension format | Example | +|---|---|---| +| Manual / direct (Blastware connected to unit) | `AB0` (3 chars) | `.CE0` | +| Call-home / ACH | `AB0W` or `AB0H` (4 chars) | `.CE0H` | + +Where: +- `AB` = 2-char base-36 of `total_seconds % 1296`; `A = value // 36`, `B = value % 36` +- `total_seconds = (event_local_time − 1985-01-01T00:00:00_local)` in seconds +- `0` = always literal digit zero +- `W` = Full Waveform, `H` = Full Histogram (ACH only) + +Base-36 alphabet: `0–9` = 0–9, `A–Z` = 10–35. + +The 10-year production archive contains only ACH files (all end in W or H). Manual Blastware downloads produce the same `AB0` prefix but without the trailing type character. + +**3-day cycle property (confirmed 2026-04-22):** A unit recording at a fixed daily time cycles through exactly **3 different extensions** with a 3-day period. Each calendar day shifts `total_seconds % 1296` by 864 (since `86400 % 1296 = 864`). The cycle repeats every 3 days because `gcd(1296, 864) = 432`. Confirmed from archive: top 3 extensions `CE0H` (95), `0E0H` (93), `OE0H` (91) are the 3-day cycle of a 06:00:14 daily call-in (seconds-in-window = 446, 14, 878). + +**B character invariance:** `864 = 24 × 36`, so adding one day never changes `value % 36` — the second extension character is invariant for a fixed daily recording time. Only the first character cycles through 3 values. + +**Old firmware (S338):** 3-char extensions observed (`.N00`, `.EI0`, etc.) — may simply be manual downloads under the same AB0 scheme, or a different encoding. Not yet confirmed. + +**Micromate Series 4** uses a different extension format (observed: `IDFH`, `IDFW`). This formula does NOT apply to Micromate units. + +All waveform files share the same `00 12 03 00` type tag regardless of extension. Blastware identifies file type by extension, not by type tag alone. + +### D.2 Timestamp Encoding (Blastware files) + +All timestamps in N00 and MLG files use an **8-byte big-endian format**: + +| Byte | Field | +|---|---| +| 0 | day (uint8) | +| 1 | month (uint8) | +| 2–3 | year (uint16 BE) | +| 4 | `0x00` (reserved) | +| 5 | hour (uint8) | +| 6 | minute (uint8) | +| 7 | second (uint8) | + +Example: `01 04 07 ea 00 00 1c 08` → April 1, 2026, 00:28:08. + +Note: this differs from the 8-byte protocol timestamp (`[day][sub_code][month][year_HI][year_LO][0x00][hour][min][sec]` = 9 bytes) used in the device's on-wire 0C waveform records. The file format uses a compact 8-byte layout without the `sub_code` byte. + +### D.3 N00 File Format — Single-Shot Waveform Event + +**File layout:** `[22B header] [21B STRT record] [body bytes] [26B footer]` + +#### D.3.1 STRT Record (21 bytes) + +The STRT record immediately follows the 22-byte header. + +| Offset | Length | Field | Notes | +|---|---|---|---| +| 0 | 4 | `STRT` | ASCII literal | +| 4 | 2 | `ff fe` | Fixed | +| 6 | 4 | event key (key4) | 4-byte waveform key | +| 10 | 4 | device-specific | NOT a repeat of key4 — device-internal field | +| 14 | 6 | device-specific | NOT zero-padded — device-internal fields | +| 20 | 1 | rectime | uint8 seconds | + +**Critical:** The STRT record must be copied verbatim from A5[0].data[7+strt_pos:] — bytes [10:20] contain device-specific values that cannot be reconstructed from protocol-level Event fields alone. + +#### D.3.2 Body Bytes (variable) + +The body is reconstructed from the raw A5 bulk waveform stream frames by stripping DLE framing markers and taking the appropriate slice of each frame's data section. + +**Per-frame contribution (from `frame.data`):** + +| Frame | Skip amount | Notes | +|---|---|---| +| A5[0] (probe) | `7 + strt_pos_in_w0 + 21` | Skip frame.data prefix + STRT record | +| A5[1] | 13 | 7-byte prefix + 6-byte first-chunk header | +| A5[2..N] | 12 | 7-byte prefix + 5-byte chunk header | +| Terminator (page_key=0x0000) | 11 | 7-byte prefix + 4-byte terminator header | + +**DLE strip rule:** For each frame's contribution (`frame.data[skip:]`), strip any `0x10` byte immediately followed by `0x02`, `0x03`, or `0x04`. Only the `0x10` is stripped; the following byte is kept as payload. + +**Split-pair edge case:** When `frame.data[-1] == 0x10` AND `frame.chk_byte ∈ {0x02, 0x03, 0x04}`, the S3FrameParser split a DLE+XX pair at the payload/checksum boundary. Reunite the bytes before stripping (`relevant + bytes([chk_byte])`), then always remove the trailing chk_byte from the result (`stripped[:-1]`) — chk_byte is the wire checksum, never payload. + +**Body/footer split:** Accumulate all frame contributions (data frames + terminator) into `all_bytes`. Then: +- `body = all_bytes[:-26]` (variable length) +- `footer = all_bytes[-26:]` (always 26 bytes — extracted from terminator content) + +#### D.3.3 Footer (26 bytes) + +The footer terminates the N00 file. Its bytes come directly from the terminator A5 frame's inner content — do NOT reconstruct from event metadata. + +| Offset | Length | Field | Notes | +|---|---|---|---| +| 0 | 2 | `0e 08` | Fixed marker | +| 2 | 8 | ts1 | Start timestamp (8B big-endian) | +| 10 | 8 | ts2 | Stop timestamp (8B big-endian) | +| 18 | 6 | `00 01 00 02 00 00` | Fixed | +| 24 | 2 | CRC | 2-byte CRC — algorithm unconfirmed | + +**CRC:** The 2-byte CRC at footer[24:26] has an unconfirmed algorithm. In M529LIY6.N00 it reads `fe da`. Attempts to match CRC-16/CCITT, CRC-16/IBM, CRC-32 (truncated), and 40+ polynomial/init combinations all failed. The writer copies it verbatim from the terminator frame. + +### D.4 MLG File Format — Monitor Log + +**File layout:** `[308B header] [N × 292B records]` + +#### D.4.1 MLG Header (308 bytes) + +| Offset | Length | Field | Notes | +|---|---|---|---| +| 0x00 | 22 | common header | prefix + `22 01 0e a0` type tag | +| 0x16 | 16 | unknown | observed as zeros in BE11529.MLG | +| 0x2A | 8 | serial number | null-padded ASCII (e.g. `"BE11529"`) | +| 0x32 | remainder | zero pad | pads to 308 bytes total | + +#### D.4.2 MLG Record (292 bytes each) + +| Offset | Length | Field | Notes | +|---|---|---|---| +| 0 | 2 | CRC | 2-byte CRC — algorithm unconfirmed; write as `00 00` | +| 2 | 4 | `22 01 0e 80` | Record marker | +| 6 | 8 | ts1 | Start timestamp (8B big-endian) | +| 14 | 8 | ts2 | Stop timestamp (8B big-endian); zeros if no stop | +| 22 | 4 | flags | Record type flags (see below) | +| 26 | 10 | serial | Null-padded ASCII serial number | +| 36 | variable | text | Type-dependent content | +| — | remainder | zero pad | pads to 292 bytes total | + +**Record flags:** + +| Value | Meaning | +|---|---| +| `ff ff 00 00` | Monitoring start with no stop recorded | +| `01 00 02 00` | Triggered event (has ts1 + ts2) | +| `02 00 00 00` | Monitoring interval (has ts1 + ts2) | + +**Text content for triggered events (`flags = 01 00 02 00`):** + +| Byte | Field | +|---|---| +| 0 | `0x08` | +| 1–8 | ts1 copy (8B big-endian) | +| 9+ | `"Geo: X.XXX in/s\x00"` ASCII geo threshold | + +#### D.4.3 MLG CRC + +The 2-byte CRC at record[0:2] uses an unconfirmed algorithm. Tested against CRC-16/CCITT, CRC-16/IBM, CRC-32 (truncated), word sums, XOR variants, and 40+ polynomial/init combinations — none matched. The writer emits `00 00`. Blastware may reject files with incorrect CRCs (impact on import unknown — TODO: test). + +### D.5 Filename Encoding ✅ PARTIALLY CONFIRMED 2026-04-22 + +Blastware assigns waveform filenames of the form ``, where: + +#### D.5.1 Serial Prefix ✅ CONFIRMED 2026-04-22 + +The first 4 characters of the filename encode the full device serial number: + +``` +prefix_letter = chr(ord('B') + floor(serial_numeric / 1000)) +serial3 = f"{serial_numeric % 1000:03d}" (last 3 digits, zero-padded) +``` + +Where `serial_numeric` is the integer after the "BE" device-type prefix. + +Examples (all confirmed from archive): + +| Serial | serial_numeric / 1000 | prefix_letter | serial3 | Filename prefix | +|--------|----------------------|---------------|---------|-----------------| +| BE6907 | 6 | H | 907 | H907 | +| BE7145 | 7 | I | 145 | I145 | +| BE11529 | 11 | M | 529 | M529 | +| BE14036 | 14 | P | 036 | P036 | +| BE17353 | 17 | S | 353 | S353 | +| BE18003 | 18 | T | 003 | T003 | +| BE18191 | 18 | T | 191 | T191 | +| BE18676 | 18 | T | 676 | T676 | + +**Interpretation:** The prefix letter encodes the production generation (batch of 1000 units). B=generation 0 (serials 0–999), C=generation 1 (1000–1999), etc. No units with prefix A have been observed — the earliest known units start around serial 2000+ (prefix D). + +**Note:** The "BE" device-type prefix is implicit. The filename only encodes the numeric part of the serial. Other Instantel device types (Micromate, Blastmate) may use a different scheme. + +#### D.5.2 Stem + Extension — full timestamp encoding ✅ FULLY CONFIRMED 2026-04-22 + +The stem (4 chars) and AB extension (2 chars) together form a 6-digit base-36 number encoding a complete second-resolution timestamp: + +```python +total_seconds = stem_int * 1296 + ab_int +event_local_time = datetime(1985, 1, 1) + timedelta(seconds=total_seconds) +``` + +- **Epoch:** `1985-01-01 00:00:00` **device local time** ✅ CONFIRMED — verified against 3,248 files from a 10-year production archive; zero errors (only 2 mismatches were Micromate `IDFH`/`IDFW` files which use a completely different naming scheme) +- **Unit:** 1296 seconds = 36² ≈ 21.6 minutes per stem increment +- **Alphabet:** `"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"` (digits then uppercase letters) +- **Collision:** Events within the same 21.6-minute window share a stem; extension distinguishes them + +**Decoding example — `P036L318.C80H` (BE14036, Full Histogram):** +``` +stem L318 = 21×36³ + 3×36² + 1×36 + 8 = 983,708 +AB C8 = 12×36 + 8 = 440 +total_sec = 983,708 × 1296 + 440 = 1,274,886,008 +event_time = 1985-01-01 + 1,274,886,008s = 2025-05-26 15:00:08 local +``` + +**Note on local time:** The device's onboard clock is set to the local timezone of the deployment site. The epoch and all timestamps are in that same local time — there is no UTC conversion. Files moved between timezones will decode to the original deployment timezone. + +#### D.5.3 Extension taxonomy + +Third character of extension is always `'0'`. File type is identified by extension, not by the type tag in the header (all waveform extensions share type tag `00 12 03 00`). + +| Extension | Recording mode | Sample rate | Status | +|---|---|---|---| +| `.N00` | Single Shot (0x00) | 1024 sps | ✅ CONFIRMED | +| `.9T0` | Continuous (0x01) | 1024 sps | ✅ CONFIRMED | +| `.490` | ? | ? | ❓ observed from M529LJ8V.490 | +| `.5K0` | ? | ? | ❓ observed from M529LJDY.5K0 | +| `.980` | ? | ? | ❓ observed from M529LJDY.980 | +| `.ML0` | ? | ? | ❓ observed from M529LJDY.ML0 (167s duration; possibly Histogram) | + +**Why 5 extensions for "Continuous"?** Binary analysis of all 6 example files shows that `.9T0`, `.490`, `.5K0`, `.980`, `.ML0` are byte-for-byte identical in all metadata regions (compliance anchor block, channel descriptor blocks `Tran/Vert/Long/MicL`). The A5 frame 7 body reflects the **session-start** compliance config, not the per-event capture config. All 5 files show recording_mode=0x01 and sample_rate=1024 in the body. The extension must therefore encode the **capture-time** compliance state — likely a combination of recording mode, sample rate, and possibly mic units or other options. This cannot be determined from file body alone without capture-time compliance data from the 0C record sub_code and the actual waveform sample count. + +**DLE-shift offset note for reading recording_mode from N00/9T0 body:** + +The compliance block in the file body has been through `_strip_inner_frame_dles`. The 0x10 constant at logical `anchor−7` (between recording_mode and sample_rate_HI) gets stripped when sample_rate_HI = `0x04` (1024 sps), because `0x10` precedes `0x04 ∈ {0x02,0x03,0x04}`. After stripping, the anchor shifts left by 1, so: + +| 1024 sps (strip occurs) | 2048 or 4096 sps (no strip) | +|---|---| +| `file[anc−7]` = recording_mode | `file[anc−8]` = recording_mode | +| `file[anc−6:anc−4]` = sample_rate | `file[anc−6:anc−4]` = sample_rate | + +For 1024 sps files, the expected file bytes around the anchor are: +``` +file[anc−9]: mode_prefix (0x00 for Single Shot/Continuous; 0x10 for Histogram) +file[anc−8]: 0x00 (was recording_mode, but shifted away — now reads 0x00 for mode_prefix) +file[anc−7]: recording_mode (0x00=Single Shot, 0x01=Continuous, etc.) +file[anc−6]: 0x04 (sample_rate_HI for 1024 sps) +file[anc−5]: 0x00 (sample_rate_LO) +file[anc−4]: histogram_interval_HI +file[anc−3]: histogram_interval_LO +``` + +--- + *All findings reverse-engineered from live RS-232 bridge captures.* *Cross-referenced from 2026-03-02 with Instantel MiniMate Plus Operator Manual (716U0101 Rev 15).* *This is a living document — append changelog entries and timestamps as new findings are confirmed or corrected.* \ No newline at end of file diff --git a/minimateplus/__init__.py b/minimateplus/__init__.py index 6c7be72..50e8d15 100644 --- a/minimateplus/__init__.py +++ b/minimateplus/__init__.py @@ -21,7 +21,15 @@ Typical usage (TCP / modem): from .client import MiniMateClient from .models import DeviceInfo, Event, MonitorLogEntry -from .transport import SerialTransport, TcpTransport +from .transport import CapturingTransport, SerialTransport, TcpTransport __version__ = "0.1.0" -__all__ = ["MiniMateClient", "DeviceInfo", "Event", "MonitorLogEntry", "SerialTransport", "TcpTransport"] +__all__ = [ + "MiniMateClient", + "DeviceInfo", + "Event", + "MonitorLogEntry", + "SerialTransport", + "TcpTransport", + "CapturingTransport", +] diff --git a/minimateplus/blastware_file.py b/minimateplus/blastware_file.py new file mode 100644 index 0000000..fedf229 --- /dev/null +++ b/minimateplus/blastware_file.py @@ -0,0 +1,979 @@ +""" +blastware_file.py — Blastware binary file codec for bidirectional interoperability. + +Reads and writes the proprietary Instantel/Blastware file formats: + Waveform events (.CE0W, .VM0H, .440, .7M0, etc.) (extension encoding UNKNOWN — see below) + .MLG — Monitor log (monitoring session history) + +All waveform formats share a common 22-byte file header prefix and identical +internal binary structure (same type tag 00 12 03 00, same STRT record layout). +Blastware identifies the file type by extension, not by a magic marker. + +EXTENSION ENCODING — V10.72 firmware FULLY CONFIRMED 2026-04-22: + + Direct / manual download: AB0 (3-char, no type character) + Call-home (ACH) download: AB0W or AB0H (4-char, W=waveform H=histogram) + + AB = 2-char base-36 of (total_seconds % 1296), where + total_seconds = (event_local_time − 1985-01-01T00:00:00_local). + 0 = always literal digit zero. + Verified against 3,248 call-home files from a 10-year production archive. + + The 10-year archive contains only ACH files (all end in W or H). + Manual Blastware downloads produce 3-char AB0 extensions — same encoding + but without the trailing type character. + + Old firmware (S338, 3-char extensions): encoding unknown / same as manual? + Micromate Series 4 uses a different scheme (literal datetime in filename). + +─── File structure overview ───────────────────────────────────────────────────── + +Waveform file structure (confirmed from example-events/4-3-26-multi/M529LIY6 (example event)): + + [22B header] [21B STRT record] [body bytes] [26B footer] + + Header (22 bytes): + 10 00 01 80 00 00 — fixed prefix + 49 6e 73 74 61 6e 74 65 6c 00 — b'Instantel\x00' + 07 2c — fixed + 00 12 03 00 — waveform file type tag (shared by all waveform extensions) + + STRT record (21 bytes, immediately follows header): + 53 54 52 54 — b'STRT' + ff fe — fixed (2 bytes) + [key4] — 4-byte waveform event key + [key4] — 4-byte waveform event key (repeated) + [zeros] — 7 bytes padding + [rectime] — uint8 record time in seconds + + Body (variable — reconstructed from A5 frame data): + The body bytes are derived from the raw A5 frame wire content, specifically + from the DLE-decoded representation of each frame's contribution. See the + _frame_body_bytes() helper for the exact algorithm. + + Footer (26 bytes): + 0e 08 + [ts1: 8B big-endian timestamp] — start timestamp + [ts2: 8B big-endian timestamp] — stop timestamp + 00 01 00 02 00 00 + [crc: 2B] — CRC (algorithm unconfirmed; written as 0x00 0x00 placeholder) + + Timestamp format (big-endian, 8 bytes): + [day] [month] [year_HI] [year_LO] [0x00] [hour] [min] [sec] + +MLG (monitor log, confirmed from example-events/4-3-26-multi/BE11529.MLG): + + [308B header] [N × 292B records] + + Header (308 bytes): + Offset 0x00: 10 00 01 80 00 00 Instantel\x00 07 2c 22 01 0e a0 — fixed (16B) + Offset 0x10: ... (unknown structure, written as zeros + serial) + Offset 0x2A: serial number (8 bytes, null-padded ASCII, e.g. "BE11529") + ... zero-padded to 308 bytes total + + Record (292 bytes each): + [2B CRC] — unknown algorithm; written as 0x00 0x00 + 22 01 0e 80 — record marker + [ts1: 8B big-endian timestamp] — start time + [ts2: 8B big-endian timestamp] — stop time (zeros if no stop) + [4B flags] — see MLG_FLAGS_* constants below + [10B serial] — null-padded serial number ASCII + [text] — for trigger records: [0x08][8B ts1_copy] then ASCII "Geo: X.XXX in/s" + for monitoring records: b'' (or minimal separator) + [zero-padded to 292 bytes] + +─── Critical implementation notes ────────────────────────────────────────────── + +Waveform body reconstruction algorithm (confirmed 2026-04-21 from verification against +M529LIY6 (example event) using raw_s3_20260403_153508.bin capture): + + The waveform body bytes come from the A5 frame content, stripped of DLE-framing + artifacts. Each A5 frame contributes a different slice of its data section, + with DLE+{0x02,0x03,0x04} byte pairs stripped. + + Skip amounts per frame index (offsets into frame.data): + A5[0] (probe): data[strt_pos + 21 + 7] (skip header + STRT record) + strt_pos found by searching frame.data[7:] for b'STRT'; + the contribution starts at strt_pos + 21 within data[7:] + which equals strt_pos + 21 + 7 within frame.data. + A5[1]: data[13] (skip 7-byte frame.data prefix + 6 header bytes) + A5[2..N]: data[12] (skip 7-byte frame.data prefix + 5 header bytes) + Terminator A5: data[11] (1 byte less than chunk frames; terminator inner header + is 4 bytes instead of 5 — confirmed 2026-04-21) + + DLE strip rule (applied AFTER slicing): + Strip any 0x10 byte that is immediately followed by 0x02, 0x03, or 0x04. + This undoes the DLE-escape that S3FrameParser preserves as literal pairs. + Applied to frame.data[skip:] + bytes([frame.chk_byte]) together, then + conditionally exclude the trailing chk_byte from the output. + + chk_byte absorption: + When frame.data[-1] == 0x10 AND frame.chk_byte ∈ {0x02, 0x03, 0x04}, + the last byte of frame.data is the DLE prefix of a split DLE+chk pair. + Including chk_byte in the strip buffer allows the pair to be stripped as + a unit. After stripping, the trailing chk_byte is ALWAYS removed — because + _strip_inner_frame_dles keeps the byte after the DLE (the chk_byte value), + and that value is the checksum, never payload. This applies to all three + cases (chk ∈ {0x02, 0x03, 0x04}) identically. + +MLG CRC: + The algorithm that produces the 2-byte CRC at the start of each MLG record + is unknown. All examined records use non-zero values that do not match + CRC-16/CCITT, CRC-16/IBM, CRC-32 (truncated), word sums, XOR variants, or + any of the 40+ polynomial/init combinations tested. The writer emits 0x0000. + This produces files that Blastware may reject or display without the CRC check — + the exact impact on BW import is unknown (TODO: test). + +─── Public API ────────────────────────────────────────────────────────────────── + + blastware_filename(event, serial) + Return the correct Blastware filename for an event (e.g. "M529LIY6.CE0W"). + Full AB0T extension encoding confirmed 2026-04-22 against 3,248 archive files. + Extension matches what Blastware itself would generate for the same event. + + write_blastware_file(event, a5_frames, path) + Create a Blastware waveform file from an Event and the full A5 frame list. + All waveform extensions share the same binary format — the extension is set + by blastware_filename() based on the event timestamp and type. + + read_blastware_file(path) → Event + Parse a Blastware waveform file into an Event object with waveform data populated. + (Not yet implemented — placeholder raises NotImplementedError.) + + write_mlg(entries, serial, path) + Create a .MLG file from a list of MonitorLogEntry objects. + + read_mlg(path) → list[MonitorLogEntry] + Parse a .MLG file into MonitorLogEntry objects. + (Not yet implemented — placeholder raises NotImplementedError.) +""" + +from __future__ import annotations + +import datetime +import logging +import struct +from pathlib import Path +from typing import Optional, Union + +from .framing import S3Frame +from .models import Event, MonitorLogEntry, Timestamp + +log = logging.getLogger(__name__) + +# ── File header constants ───────────────────────────────────────────────────── + +# Common 16-byte prefix shared by waveform files and MLG (confirmed from binary inspection). +_FILE_HEADER_PREFIX = bytes.fromhex("1000018000004973") + b"tantel\x00\x07\x2c" +# = 10 00 01 80 00 00 49 73 74 61 6e 74 65 6c 00 07 2c (17 bytes) +# Confirmed breakdown: 10 00 01 80 00 00 = fixed; "Instantel\x00" = 10B; 07 2c = fixed + +# Simpler construction: +_FILE_HEADER_PREFIX = b"\x10\x00\x01\x80\x00\x00Instantel\x00\x07\x2c" # 17 bytes + +# Waveform file type tag (4 bytes after common prefix) — shared by ALL waveform extensions +_WAVEFORM_TYPE_TAG = b"\x00\x12\x03\x00" # confirmed from M529LIY6 (example event) — same tag for .CE0W, .VM0H, etc. + +# MLG type tag (4 bytes after common prefix) +_MLG_TYPE_TAG = b"\x22\x01\x0e\xa0" # confirmed from BE11529.MLG offset 0x11..0x14 + +# Total header sizes +_WAVEFORM_HEADER_SIZE = 22 # 17 + 4 = 21... wait. Let me recalculate. +# From binary: first 22 bytes = header, then STRT at byte 22. +# 17-byte common prefix + 4-byte type tag = 21 bytes. But observed header is 22B. +# Checking: 6 fixed + 10 "Instantel\x00" + 2 "07 2c" = 18B prefix, then 4B type tag = 22B. +# Re-count: b"\x10\x00\x01\x80\x00\x00" = 6B + b"Instantel\x00" = 10B + b"\x07\x2c" = 2B = 18B prefix. +_FILE_HEADER_PREFIX = b"\x10\x00\x01\x80\x00\x00Instantel\x00\x07\x2c" # 18 bytes +_WAVEFORM_HEADER_SIZE = 22 # 18 + 4 = 22 bytes ✅ +_MLG_HEADER_SIZE = 308 # confirmed from BE11529.MLG + +# MLG record marker (4 bytes after 2-byte CRC at start of each record) +_MLG_RECORD_MARKER = b"\x22\x01\x0e\x80" +_MLG_RECORD_SIZE = 292 # bytes per record (confirmed from BE11529.MLG) + +# MLG record flags (4 bytes at record[22:26]) +# Confirmed from BE11529.MLG binary inspection: +MLG_FLAGS_START_ONLY = b"\xff\xff\x00\x00" # monitoring start with no stop +MLG_FLAGS_TRIGGER = b"\x01\x00\x02\x00" # triggered event (has ts1 + ts2) +MLG_FLAGS_INTERVAL = b"\x02\x00\x00\x00" # monitoring interval (has ts1 + ts2) + + +# ── Timestamp helpers ───────────────────────────────────────────────────────── + +def _encode_ts_be(ts: Optional[datetime.datetime]) -> bytes: + """ + Encode a datetime as an 8-byte big-endian Blastware timestamp. + + Format (waveform file and MLG record timestamps): + [day][month][year_HI][year_LO][0x00][hour][min][sec] + + Big-endian year confirmed from M529LIY6 (example event) footer: + footer bytes [2..9] = 01 04 07 ea 00 00 1c 08 + → day=1 month=4 year=0x07ea=2026 hour=0 min=28 sec=8 ✅ + + Returns 8 zero bytes if ts is None. + """ + if ts is None: + return bytes(8) + return bytes([ + ts.day, + ts.month, + (ts.year >> 8) & 0xFF, + ts.year & 0xFF, + 0x00, + ts.hour, + ts.minute, + ts.second, + ]) + + +def _decode_ts_be(raw: bytes) -> Optional[datetime.datetime]: + """ + Decode an 8-byte big-endian Blastware timestamp. + + Returns None if the bytes are all zero or structurally invalid. + """ + if len(raw) < 8 or raw == bytes(8): + return None + day = raw[0] + month = raw[1] + year = (raw[2] << 8) | raw[3] + hour = raw[5] + minute = raw[6] + sec = raw[7] + try: + return datetime.datetime(year, month, day, hour, minute, sec) + except ValueError: + return None + + +def _ts_from_model(ts: Optional[Timestamp]) -> Optional[datetime.datetime]: + """Convert a models.Timestamp to datetime.datetime, or None.""" + if ts is None: + return None + try: + return datetime.datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second) + except (ValueError, TypeError): + return None + + +# ── DLE strip helper ────────────────────────────────────────────────────────── + +def _strip_inner_frame_dles(data: bytes) -> bytes: + """ + Strip DLE (0x10) framing markers from A5 inner-frame content. + + The A5 (bulk waveform stream) response body contains DLE-encoded sub-frame + structure. S3FrameParser preserves DLE+XX pairs as two literal bytes in + frame.data. Only the DLE marker byte needs to be removed; the following + byte is actual payload content. + + Rule: when 0x10 is immediately followed by {0x02, 0x03, 0x04}, strip the + 0x10 (DLE marker) and keep the following byte as payload. + + Lone 0x10 bytes not followed by {0x02, 0x03, 0x04} are kept as-is. + + Confirmed correct by verifying reconstructed waveform body against M529LIY6 (example event): + - 0x10 0x02 in terminator → 0x02 kept ✓ + - 0x10 0x04 in terminator (month byte) → 0x04 kept ✓ + """ + out = bytearray() + i = 0 + while i < len(data): + b = data[i] + if b == 0x10 and i + 1 < len(data) and data[i + 1] in {0x02, 0x03, 0x04}: + # Strip the DLE marker; the next byte is payload and will be appended + # in the next loop iteration. + i += 1 + continue + out.append(b) + i += 1 + return bytes(out) + + +def _frame_body_bytes(frame: S3Frame, skip: int) -> bytes: + """ + Extract the waveform body contribution from one A5 S3Frame. + + The contribution is frame.data[skip:] with inner-frame DLE pairs stripped + per _strip_inner_frame_dles(). The chk_byte is temporarily appended before + stripping to handle the split-pair edge case where a DLE at the end of + frame.data is paired with chk_byte. + + Split-pair edge case (confirmed for A5[8] of M529LIY6 (example event), 2026-04-21): + + S3FrameParser appends DLE+XX pairs as two literal bytes when XX ∉ {DLE, ETX}. + When the LAST occurrence of such a pair straddles the payload/checksum boundary + (i.e., DLE is the last byte of raw_payload and XX is the checksum), the parser + splits them: + - DLE ends up as the last byte of frame.data (frame.data[-1] == 0x10) + - XX is stored as frame.chk_byte + + To strip the pair correctly, we reunite the bytes before calling the strip + function. Since chk_byte is the checksum (not payload data), it is excluded + from the final output regardless of whether it was part of a pair. + + Post-strip chk_byte removal (ALL cases): + _strip_inner_frame_dles strips the 0x10 and KEEPS chk_byte in all cases. + Chk_byte is always the checksum (not payload), so always strip it off. + + Args: + frame: S3Frame with frame.data and frame.chk_byte populated. + skip: Number of leading bytes in frame.data to exclude (frame header). + + Returns: + bytes — the waveform body contribution for this frame. + """ + if skip >= len(frame.data): + return b"" + + relevant = frame.data[skip:] + + # Detect split DLE+chk pair at the frame boundary. + has_split_pair = ( + len(relevant) > 0 + and relevant[-1] == 0x10 + and frame.chk_byte in {0x02, 0x03, 0x04} + ) + + if has_split_pair: + # Reunite the split pair so the strip function sees both bytes together. + buf = relevant + bytes([frame.chk_byte]) + stripped = _strip_inner_frame_dles(buf) + # _strip_inner_frame_dles strips the DLE (0x10) and KEEPS chk_byte. + # chk_byte is the received checksum — never payload — so remove it. + # This is correct for all values in {0x02, 0x03, 0x04}. + if stripped: + stripped = stripped[:-1] + return stripped + else: + return _strip_inner_frame_dles(relevant) + + +# ── Filename helper ─────────────────────────────────────────────────────────── + +_INSTANTEL_EPOCH = datetime.datetime(1985, 1, 1, 0, 0, 0) +""" +Instantel timestamp epoch — January 1, 1985, 00:00:00 local time. +Confirmed 2026-04-21: stem values for 6 independent events (April 1–9, 2026) +all converge to this epoch when decoded as floor(seconds_since_epoch / 1296). +1985 is the year Instantel was founded. +""" + +_STEM_UNIT_SEC = 1296 # = 36^2 seconds ≈ 21.6 minutes per stem unit + +_STEM_CHARS = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" + +# ── Waveform file extension encoding ───────────────────────────────────────── +# +# NEW FIRMWARE (V10.72+) — FULLY DECODED (confirmed 2026-04-21, 10-year archive): +# +# Extension format: AB0T (4 characters) +# AB = 2-char base-36 encoding of (seconds_since_epoch % 1296) +# i.e. the number of seconds into the current 21.6-minute stem window +# Range: 0 ("00") to 1295 ("ZZ") +# 0 = always literal '0' +# T = event type: 'W' = Full Waveform, 'H' = Full Histogram +# +# Combined with the 4-char stem (which encodes seconds_since_epoch // 1296), +# the FULL filename gives a second-resolution timestamp: +# total_seconds = stem_val * 1296 + ab_val +# timestamp = EPOCH + timedelta(seconds=total_seconds) +# +# Verified against three S353L4H0 events (all three match to the second): +# S353L4H0.3M0W Full Waveform 2025-06-23 13:57:22 AB=3M=130 ✓ +# S353L4H0.8S0H Full Histogram 2025-06-23 14:00:28 AB=8S=316 ✓ +# S353L4H0.9X0W Full Waveform 2025-06-23 14:01:09 AB=9X=357 ✓ +# +# OLD FIRMWARE (S338, 3-char extensions ending in '0') — UNKNOWN: +# Observed (old firmware / manual downloads): .440, .470, .7M0, .9T0, .EI0, etc. +# The V10.72 formula does NOT apply to these. +# Extension is NOT recording mode (refuted 2026-04-21: continuous → .EI0, not .9T0). +# blastware_filename() computes the correct AB0 extension for V10.72 firmware. +# +# WRONG earlier assumption (do not re-introduce): +# Extension was believed to encode recording mode × sample rate. +# Refuted by continuous-mode event producing .EI0 instead of .9T0. + + +def _make_stem(ts_local: datetime.datetime) -> str: + """ + Encode a local timestamp as a 4-character uppercase base-36 stem. + + Algorithm (confirmed 2026-04-21 from 6 known file/timestamp pairs): + stem_int = floor((ts_local - Jan_1_1985_midnight_local) / 1296_seconds) + stem = 4-char uppercase base-36 encoding of stem_int + + Unit = 36² = 1296 seconds ≈ 21.6 minutes. Events within the same 1296-second + window receive the same stem; their extension distinguishes them. + """ + delta_sec = int((ts_local - _INSTANTEL_EPOCH).total_seconds()) + n = delta_sec // _STEM_UNIT_SEC + s = "" + for _ in range(4): + s = _STEM_CHARS[n % 36] + s + n //= 36 + return s + + +def blastware_filename(event: Event, serial: str, ach: bool = False) -> str: + """ + Return the correct Blastware filename for an event. + + CONFIRMED 2026-04-22 — verified against 3,248 files from a 10-year archive. + + Filename format: 0[T] + where: + + prefix_letter = chr(ord('B') + floor(serial_numeric / 1000)) + — encodes the production generation (batch of 1000 units) + — e.g. BE6907→H, BE11529→M, BE14036→P, BE18003→T + + serial3 = f"{serial_numeric % 1000:03d}" + — last 3 digits of numeric serial, zero-padded + + stem = 4-char base-36 of floor(total_seconds / 1296) + — encodes which 21.6-minute window the event fell in + + AB = 2-char base-36 of (total_seconds % 1296) + — encodes seconds within the window (0–1295) + + 0 = always literal digit zero + + T = 'W' or 'H' — ONLY appended for call-home (ACH) downloads (ach=True). + Manual / direct downloads produce a 3-char extension (AB0) with no type char. + Call-home downloads produce a 4-char extension (AB0W or AB0H). + + total_seconds = (event_local_time − 1985-01-01T00:00:00_local) in seconds + + The 10-year production archive contains only call-home files (all end in W or H). + Manual Blastware downloads produce 3-char extensions — the same AB0 prefix but + without the trailing type character. + + Micromate Series 4 uses a completely different naming scheme (literal datetime + in filename); this function does not apply to Micromate units. + + Args: + event: Event object with timestamp set. + serial: Device serial number string (e.g. "BE11529"). + ach: If True, append W/H type character (call-home style). + If False (default), omit type character (direct download style). + + Returns: + Filename string, e.g. "M529LIY6.CE0" (direct) or "M529LIY6.CE0H" (ACH). + """ + # ── Serial prefix ────────────────────────────────────────────────────────── + serial_digits = "".join(c for c in serial if c.isdigit()) + if len(serial_digits) >= 1: + serial_numeric = int(serial_digits) + generation = serial_numeric // 1000 + prefix_letter = chr(ord('B') + generation) + serial3 = f"{serial_numeric % 1000:03d}" + else: + prefix_letter = "M" # fallback + serial3 = "000" + prefix = prefix_letter + serial3 + + # ── Stem + AB extension from timestamp ──────────────────────────────────── + if event.timestamp is not None: + try: + ts_local = datetime.datetime( + event.timestamp.year, event.timestamp.month, event.timestamp.day, + event.timestamp.hour, event.timestamp.minute, event.timestamp.second, + ) + delta_sec = int((ts_local - _INSTANTEL_EPOCH).total_seconds()) + stem = _make_stem(ts_local) + ab_val = delta_sec % _STEM_UNIT_SEC + ab_str = _STEM_CHARS[ab_val // 36] + _STEM_CHARS[ab_val % 36] + except (ValueError, TypeError, AttributeError): + stem = "0000" + ab_str = "00" + else: + stem = "0000" + ab_str = "00" + + # ── Type character (ACH only) ───────────────────────────────────────────── + if ach: + if getattr(event, 'recording_mode', None) in (3, 4): # Histogram / Hist+Cont + type_char = 'H' + else: + type_char = 'W' + ext = f".{ab_str}0{type_char}" + else: + ext = f".{ab_str}0" + + return prefix + stem + ext + + +# ── A5 frame classifier ─────────────────────────────────────────────────────────── + +# ASCII markers that identify a compliance-config / metadata frame. +# These strings appear in the A5 bulk stream as part of the device's +# compliance setup payload. They should NEVER appear in raw ADC waveform +# frames (which are binary-heavy, < 20 % printable ASCII). +_METADATA_FRAME_MARKERS = ( + b"Project:", + b"Client:", + b"Standard Recording Setup", + b"Extended Notes", + b"User Name:", + b"Seis Loc:", +) + + +def classify_frame(frame: S3Frame) -> str: + """ + Classify an A5 bulk waveform stream frame by its content. + + Returns one of: + "terminator" — page_key == 0x0000 + "probe_or_strt" — data contains b"STRT\xff\xfe" (the initial probe response) + "metadata" — data contains ASCII compliance-config markers + "waveform" — predominantly binary (< 20 % printable ASCII) + "unknown" — none of the above criteria matched + + Used by write_blastware_file() to filter non-waveform frames out of + the reconstructed body so that metadata blocks (Project:, Client:, …) + and spurious STRT records do not corrupt the output file. + """ + if frame.page_key == 0x0000: + return "terminator" + data = bytes(frame.data) + if b"STRT\xff\xfe" in data: + return "probe_or_strt" + if any(m in data for m in _METADATA_FRAME_MARKERS): + return "metadata" + if len(data) > 0: + printable = sum(1 for b in data if 32 <= b < 127) + if printable / len(data) < 0.20: + return "waveform" + return "unknown" + + +# ── Waveform file writer ─────────────────────────────────────────────────────────── + +def write_blastware_file( + event: Event, + a5_frames: list[S3Frame], + path: Union[str, Path], +) -> None: + """ + Write a Blastware waveform file from a downloaded event. + + Args: + event: Event object (populated by get_events() or download_waveform()). + Used for the STRT record (key, rectime) and footer timestamps. + a5_frames: Complete A5 frame list INCLUDING the terminator frame + (page_key=0x0000). Pass include_terminator=True to + read_bulk_waveform_stream() when collecting frames. + Must have at least 2 frames (probe + terminator). + path: Destination file path. Parent directory must exist. + Extension should be set via blastware_filename(). + + File layout: + [22B header] [21B STRT] [body bytes] [26B footer] + + Raises: + ValueError: if a5_frames is empty or has no terminator (page_key=0). + OSError: if the file cannot be written. + + Confirmed correct waveform body reconstruction against M529LIY6 (example event) (2026-04-21). + """ + if not a5_frames: + raise ValueError("a5_frames must not be empty") + + path = Path(path) + + # ── Extract STRT record from probe frame ──────────────────────────────── + # The STRT record (21 bytes) lives verbatim inside A5[0].data[7:]. + # It is stored as-is in the waveform file — do NOT reconstruct it from Event + # fields, as bytes [10:14] and [14:20] contain device-specific values + # (not simply key4 repeated or zero-padded). Confirmed 2026-04-21. + # + # STRT layout (21 bytes, observed in M529LIY6 files): + # [0:4] b'STRT' + # [4:6] 0xff 0xfe (fixed) + # [6:10] key4 (event key) + # [10:14] device-specific field (NOT a key4 repeat) + # [14:20] device-specific fields (NOT zeros) + # [20] rectime uint8 seconds + # Extract STRT from the DLE-stripped probe frame. + # + # frame.data[7:] is the raw wire representation; it may contain DLE+{02,03,04} + # inner-frame pairs that S3FrameParser preserves as two literal bytes. The + # Blastware file stores the stripped form, so we must strip before extracting. + # + # Example (M529LK0Y, 2026-04-21): STRT contains value 0x02 encoded as [10 02] + # on the wire. Without stripping, STRT is 22 raw bytes → write_blastware_file writes the + # DLE prefix into the file AND begins the body 1 byte too early (probe_skip off + # by 1). Stripping fixes both. + # + # probe_skip must be computed in the RAW frame.data domain (it is used as the + # `skip` argument to _frame_body_bytes which operates on raw frame.data). + # We walk the raw bytes counting stripped bytes until we have passed + # strt_pos + 21 stripped bytes, giving the raw offset of the first body byte. + w0_raw = bytes(a5_frames[0].data[7:]) + w0_stripped = _strip_inner_frame_dles(w0_raw) + strt_pos_stripped = w0_stripped.find(b"STRT") + + if strt_pos_stripped >= 0: + strt = bytes(w0_stripped[strt_pos_stripped : strt_pos_stripped + 21]) + + # Walk raw bytes to find the raw-domain end of the STRT (= body start). + target_stripped = strt_pos_stripped + 21 + stripped_so_far = 0 + raw_i = 0 + while stripped_so_far < target_stripped and raw_i < len(w0_raw): + if (w0_raw[raw_i] == 0x10 + and raw_i + 1 < len(w0_raw) + and w0_raw[raw_i + 1] in {0x02, 0x03, 0x04}): + raw_i += 2 # DLE pair → 1 stripped byte, 2 raw bytes + else: + raw_i += 1 # normal byte → 1 stripped byte, 1 raw byte + stripped_so_far += 1 + probe_skip = 7 + raw_i # raw bytes to skip: 7 header + raw STRT length + else: + # Fallback: construct a minimal STRT if probe frame lacks it + key4 = event._waveform_key if hasattr(event, '_waveform_key') and event._waveform_key else bytes(4) + rectime = event.rectime_seconds if event.rectime_seconds is not None else 0 + strt = b"STRT" + b"\xff\xfe" + key4 + bytes(14) + bytes([rectime & 0xFF]) + probe_skip = 7 + 21 + + log.warning( + "write_blastware_file: strt_pos_stripped=%d probe_skip=%d " + "probe_data_len=%d strt_hex=%s", + strt_pos_stripped if strt_pos_stripped >= 0 else -1, + probe_skip, + len(a5_frames[0].data), + strt.hex() if len(strt) >= 4 else "(short)", + ) + + if len(strt) != 21: + raise ValueError(f"STRT record must be 21 bytes, got {len(strt)}") + + # ── Build waveform file header ───────────────────────────────────────────────────── + header = _FILE_HEADER_PREFIX + _WAVEFORM_TYPE_TAG + assert len(header) == _WAVEFORM_HEADER_SIZE, f"Waveform header must be {_WAVEFORM_HEADER_SIZE} bytes" + + # ── Build body from A5 frames ──────────────────────────────────────────── + # The waveform body is reconstructed from ALL A5 frames (data + terminator). + # The terminator frame's contribution includes the 26-byte footer at its end. + # + # Reconstruction layout (confirmed from M529LIY6 captures, 2026-04-21): + # all_bytes = contributions from A5[0..N] + terminator_contribution + # body = all_bytes[:-26] (everything except the last 26 bytes) + # footer = all_bytes[-26:] (last 26 bytes = the waveform file footer) + # + # The footer bytes come directly from the terminator frame's inner content — + # using them verbatim ensures timestamps match the device's recorded values. + + # Separate terminator from data frames. + # Search from the FRONT for the first terminator (page_key == 0x0000). + # Do NOT use a5_frames[-1] — if _a5_frames contains stray frames from a + # subsequent event (a known get_events side-effect), the last frame will + # not be the terminator and the footer will be mis-identified. + term_idx: Optional[int] = None + for _i, _f in enumerate(a5_frames): + if _f.page_key == 0x0000: + term_idx = _i + break + + if term_idx is not None: + body_frames = a5_frames[:term_idx] + term_frame = a5_frames[term_idx] + else: + body_frames = a5_frames + term_frame = None + + # ── Identify first metadata frame and skip "extra chunks" ─────────────── + # When extra_chunks_after_metadata=1 in read_bulk_waveform_stream(), the + # frame list is: [probe, data..., metadata, extra_chunk, terminator]. + # The extra_chunk is downloaded to prime the TCP terminator response — its + # ADC data is NOT part of the Blastware file body. Skip it. + # + # Rule: any frame at index strictly between first_metadata_fi and last_fi + # (the final frame) is an extra chunk and must be excluded. + # + # If no metadata frame exists (e.g. full_waveform download), first_metadata_fi + # is None and no frames are skipped — all frames contribute normally. + first_metadata_fi: Optional[int] = None + for _fi_scan, _frame_scan in enumerate(body_frames): + if _fi_scan > 0 and any(m in bytes(_frame_scan.data) for m in _METADATA_FRAME_MARKERS): + first_metadata_fi = _fi_scan + break + last_fi = len(body_frames) - 1 + + log.warning( + "write_blastware_file: %d body_frames first_metadata_fi=%s last_fi=%d", + len(body_frames), + str(first_metadata_fi) if first_metadata_fi is not None else "None", + last_fi, + ) + + all_bytes = bytearray() + + for fi, frame in enumerate(body_frames): + # Skip "extra chunk" frames: frames after the first metadata frame but + # before the last frame (terminator). These prime the TCP terminator but + # their ADC data must NOT appear in the Blastware file body. + if (first_metadata_fi is not None + and fi > first_metadata_fi + and fi < last_fi): + log.warning( + "write_blastware_file: fi=%d SKIP (extra chunk after metadata fi=%d last_fi=%d)", + fi, first_metadata_fi, last_fi, + ) + continue + + if fi == 0: + # Probe frame: always process regardless of classification. + # It holds the STRT record; probe_skip positions us past it. + skip = probe_skip + else: + # ALL subsequent frames are included unconditionally — no filtering on + # frame type. In the A5 stream, frame 0 is always the probe response; + # frames 1+ are always data (waveform chunks, compliance config, or + # compliance continuation). Classification is for logging only. + # + # DO NOT gate on classify_frame() here: + # - "probe_or_strt" at fi>0 is always a false positive — ADC binary + # data can coincidentally contain b"STRT\xff\xfe" (confirmed from + # live capture: frames 1 and 5 matched on event key=01110000). + # - "metadata" frames must be included (compliance config body). + # - The compliance block spans 2 frames; skipping either produces a + # truncated file that Blastware rejects. + skip = 13 if fi == 1 else 12 + + contribution = _frame_body_bytes(frame, skip) + log.warning("write_blastware_file: fi=%d skip=%d raw_data=%d contribution=%d", + fi, skip, len(frame.data), len(contribution)) + all_bytes.extend(contribution) + + # Terminator contributes its content, which ends with the 26-byte footer. + # skip=11 (not 12) because the terminator's inner frame header is 4 bytes, + # one shorter than chunk frames' 5-byte inner header. Confirmed 2026-04-21. + if term_frame is not None: + term_contribution = _frame_body_bytes(term_frame, 11) + log.warning( + "write_blastware_file: term_frame data_len=%d skip=11 " + "contribution_len=%d first8=%s", + len(term_frame.data), + len(term_contribution), + term_contribution[:8].hex() if len(term_contribution) >= 8 else term_contribution.hex(), + ) + all_bytes.extend(term_contribution) + + log.warning( + "write_blastware_file: all_bytes total=%d last28=%s", + len(all_bytes), + bytes(all_bytes[-28:]).hex() if len(all_bytes) >= 28 else bytes(all_bytes).hex(), + ) + + if len(all_bytes) >= 26: + body = bytes(all_bytes[:-26]) + footer = bytes(all_bytes[-26:]) + else: + # Fallback: no terminator or very short stream → build footer from event metadata + body = bytes(all_bytes) + start_dt = _ts_from_model(event.timestamp) + stop_dt: Optional[datetime.datetime] = None + if start_dt is not None and event.rectime_seconds: + stop_dt = start_dt + datetime.timedelta(seconds=event.rectime_seconds) + footer = ( + b"\x0e\x08" + + _encode_ts_be(start_dt) + + _encode_ts_be(stop_dt) + + b"\x00\x01\x00\x02\x00\x00" + + b"\x00\x00" # CRC placeholder + ) + + # ── Write file ─────────────────────────────────────────────────────────── + with open(path, "wb") as f: + f.write(header) + f.write(strt) + f.write(body) + f.write(footer) + + +def read_blastware_file(path: Union[str, Path]) -> Event: + """ + Parse a Blastware waveform file into an Event object. + + NOT YET IMPLEMENTED. + + Args: + path: Path to the waveform file. + + Returns: + Event object with waveform data populated. + + Raises: + NotImplementedError: always (pending implementation). + """ + raise NotImplementedError("read_blastware_file() is not yet implemented") + + +# ── MLG file writer ─────────────────────────────────────────────────────────── + +def _build_mlg_header(serial: str) -> bytes: + """ + Build the 308-byte MLG file header. + + Header structure (confirmed from BE11529.MLG binary inspection): + Offset 0x00: 10 00 01 80 00 00 Instantel\x00 07 2c 22 01 0e a0 (22B) + Offset 0x16: ... (16B unknown — observed as zeros in BE11529.MLG) + Offset 0x2A: serial number (8 bytes, null-padded ASCII) + ... rest zero-padded to 308 bytes + + The serial string "BE11529" appears at offset 0x2A (42 decimal). + """ + buf = bytearray(_MLG_HEADER_SIZE) + + # Common prefix + MLG type tag + prefix = _FILE_HEADER_PREFIX + _MLG_TYPE_TAG # 22 bytes + buf[0:len(prefix)] = prefix + + # Serial number at offset 0x2A + serial_bytes = serial.encode("ascii", errors="replace")[:8] + serial_padded = serial_bytes.ljust(8, b"\x00") + buf[0x2A : 0x2A + 8] = serial_padded + + return bytes(buf) + + +def _build_mlg_record( + entry: MonitorLogEntry, + serial: str, +) -> bytes: + """ + Build one 292-byte MLG record from a MonitorLogEntry. + + Record layout (confirmed from BE11529.MLG binary inspection): + [0:2] CRC — 2-byte CRC (algorithm unknown; written as 0x0000) + [2:6] marker — 22 01 0e 80 + [6:14] ts1 — 8B big-endian start timestamp + [14:22] ts2 — 8B big-endian stop timestamp + [22:26] flags — 4B record flags (see MLG_FLAGS_* constants) + [26:36] serial — 10B null-padded serial number + [36:] text — for triggered events: [0x08][8B ts1_copy]["Geo: X.XXX in/s"] + for monitoring intervals: b"" or minimal separator + [... zero-padded to 292 bytes] + + Flags based on entry type: + - MonitorLogEntry with start_time only (no stop_time): MLG_FLAGS_START_ONLY + - MonitorLogEntry with both times and geo_threshold_ips set: MLG_FLAGS_TRIGGER + - MonitorLogEntry with both times (monitoring interval): MLG_FLAGS_INTERVAL + + The triggered-event text block (flags = MLG_FLAGS_TRIGGER): + [0x08] [ts1: 8B] [ASCII "Geo: X.XXX in/s\x00"] + Confirmed from BE11529.MLG records at offset 0x0134 and 0x0258. + """ + buf = bytearray(_MLG_RECORD_SIZE) + + start_dt = ( + datetime.datetime( + entry.start_time.year, entry.start_time.month, entry.start_time.day, + entry.start_time.hour, entry.start_time.minute, entry.start_time.second, + ) + if entry.start_time else None + ) + stop_dt = ( + datetime.datetime( + entry.stop_time.year, entry.stop_time.month, entry.stop_time.day, + entry.stop_time.hour, entry.stop_time.minute, entry.stop_time.second, + ) + if entry.stop_time else None + ) + + # [0:2] CRC placeholder + buf[0:2] = b"\x00\x00" + + # [2:6] Record marker + buf[2:6] = _MLG_RECORD_MARKER + + # [6:14] ts1 + buf[6:14] = _encode_ts_be(start_dt) + + # [14:22] ts2 + buf[14:22] = _encode_ts_be(stop_dt) + + # [22:26] flags + if stop_dt is None: + flags = MLG_FLAGS_START_ONLY + elif entry.geo_threshold_ips is not None: + flags = MLG_FLAGS_TRIGGER + else: + flags = MLG_FLAGS_INTERVAL + buf[22:26] = flags + + # [26:36] serial (10B null-padded) + serial_bytes = serial.encode("ascii", errors="replace")[:10] + buf[26 : 26 + len(serial_bytes)] = serial_bytes + + # [36:] text content + pos = 36 + if flags == MLG_FLAGS_TRIGGER: + # Extra ts1 copy: [0x08][ts1: 8B] + buf[pos] = 0x08 + pos += 1 + buf[pos : pos + 8] = _encode_ts_be(start_dt) + pos += 8 + + if entry.geo_threshold_ips is not None: + geo_text = f"Geo: {entry.geo_threshold_ips:.3f} in/s\x00".encode("ascii") + buf[pos : pos + len(geo_text)] = geo_text + pos += len(geo_text) + + return bytes(buf) + + +def write_mlg( + entries: list[MonitorLogEntry], + serial: str, + path: Union[str, Path], +) -> None: + """ + Write a Blastware .MLG monitor log file. + + Args: + entries: List of MonitorLogEntry objects (from get_monitor_log_entries()). + Each entry produces one 292-byte record in the file. + serial: Device serial number string (e.g. "BE11529"). + Written to the file header and each record. + path: Destination file path. Extension is not enforced — use ".MLG". + + File layout: + [308B header] [N × 292B records] + + Note: The 2-byte CRC at the start of each record is written as 0x0000. + The CRC algorithm is unknown (see module docstring). + + Raises: + OSError: if the file cannot be written. + """ + path = Path(path) + header = _build_mlg_header(serial) + + with open(path, "wb") as f: + f.write(header) + for entry in entries: + record = _build_mlg_record(entry, serial) + f.write(record) + + +def read_mlg(path: Union[str, Path]) -> list[MonitorLogEntry]: + """ + Parse a Blastware .MLG file into a list of MonitorLogEntry objects. + + NOT YET IMPLEMENTED. + + Args: + path: Path to the .MLG file. + + Returns: + List of MonitorLogEntry objects. + + Raises: + NotImplementedError: always (pending implementation). + """ + raise NotImplementedError("read_mlg() is not yet implemented") diff --git a/minimateplus/client.py b/minimateplus/client.py index 767d104..8f01f3d 100644 --- a/minimateplus/client.py +++ b/minimateplus/client.py @@ -449,7 +449,7 @@ class MiniMateClient: proto.confirm_erase_all() log.info("delete_all_events: erase confirmed — device memory cleared") - def get_events(self, full_waveform: bool = False, debug: bool = False, stop_after_index: Optional[int] = None, skip_waveform_for_keys: Optional[set] = None) -> list[Event]: + def get_events(self, full_waveform: bool = False, debug: bool = False, stop_after_index: Optional[int] = None, skip_waveform_for_keys: Optional[set] = None, extra_chunks_after_metadata: int = 1) -> list[Event]: """ Download all stored events from the device using the confirmed 1E → 0A → 0C → 5A → 1F event-iterator protocol. @@ -604,10 +604,12 @@ class MiniMateClient: "get_events: 5A full waveform download for key=%s", cur_key.hex() ) a5_frames = proto.read_bulk_waveform_stream( - cur_key, stop_after_metadata=False, max_chunks=128 + cur_key, stop_after_metadata=False, max_chunks=128, + include_terminator=True, ) if a5_frames: a5_ok = True + ev._a5_frames = a5_frames # store for write_blastware_file _decode_a5_metadata_into(a5_frames, ev) _decode_a5_waveform(a5_frames, ev) log.info( @@ -619,10 +621,14 @@ class MiniMateClient: "get_events: 5A metadata-only download for key=%s", cur_key.hex() ) a5_frames = proto.read_bulk_waveform_stream( - cur_key, stop_after_metadata=True + cur_key, stop_after_metadata=True, + include_terminator=True, + extra_chunks_after_metadata=extra_chunks_after_metadata, + max_chunks=128, ) if a5_frames: a5_ok = True + ev._a5_frames = a5_frames # store for write_blastware_file _decode_a5_metadata_into(a5_frames, ev) log.debug( "get_events: 5A metadata client=%r operator=%r", @@ -776,6 +782,39 @@ class MiniMateClient: else: log.warning("download_waveform: waveform decode produced no samples") + return a5_frames + + def save_blastware_file(self, event: "Event", path: "Union[str, Path]", serial: str) -> None: + """ + Download the full waveform for *event* and save it as a Blastware- + compatible Blastware waveform file at *path*. + + This is a convenience wrapper that calls download_waveform() (which + performs the complete SUB 5A BULK_WAVEFORM_STREAM download) and then + calls write_blastware_file() from blastware_file.py to encode the result. + + Args: + event: Event object with waveform key populated (from get_events()). + path: Destination file path. Caller should use blastware_filename() + to pick the correct extension via blastware_filename(). + serial: Device serial number (e.g. "BE11529") — passed to + blastware_filename() for reference, but the caller supplies + the final path. + """ + from pathlib import Path as _Path + from .blastware_file import write_blastware_file as _write_blastware_file + + a5_frames = self.download_waveform(event) + if not a5_frames: + raise RuntimeError( + f"save_blastware_file: no A5 frames received for event#{event.index}" + ) + _write_blastware_file(event, a5_frames, path) + log.info( + "save_blastware_file: wrote %s (%d A5 frames)", + path, len(a5_frames), + ) + # ── Write commands ──────────────────────────────────────────────────────── def push_config_raw( @@ -1324,7 +1363,7 @@ def _decode_waveform_record_into(data: bytes, event: Event) -> None: log.warning("waveform record project strings decode failed: %s", exc) -def _decode_a5_metadata_into(frames_data: list[bytes], event: Event) -> None: +def _decode_a5_metadata_into(frames_data: list[S3Frame], event: Event) -> None: """ Search A5 (BULK_WAVEFORM_STREAM) frame data for event-time metadata strings and populate event.project_info. @@ -1352,7 +1391,7 @@ def _decode_a5_metadata_into(frames_data: list[bytes], event: Event) -> None: Modifies event in-place. """ - combined = b"".join(frames_data) + combined = b"".join(f.data for f in frames_data) def _find_string_after(needle: bytes, max_len: int = 64) -> Optional[str]: pos = combined.find(needle) @@ -1376,7 +1415,7 @@ def _decode_a5_metadata_into(frames_data: list[bytes], event: Event) -> None: notes = _find_string_after(b"Extended Notes") if not any([project, client, operator, location, notes]): - log.debug("a5 metadata: no project strings found in %d frames", len(frames_data)) + log.debug("a5 metadata: no project strings found in %d frames (%d bytes)", len(frames_data), len(combined)) return if event.project_info is None: @@ -1402,7 +1441,7 @@ def _decode_a5_metadata_into(frames_data: list[bytes], event: Event) -> None: def _decode_a5_waveform( - frames_data: list[bytes], + frames_data: list[S3Frame], event: Event, ) -> None: """ @@ -1463,7 +1502,7 @@ def _decode_a5_waveform( return # ── Parse STRT record from A5[0] ──────────────────────────────────────── - w0 = frames_data[0][7:] # db[7:] for A5[0] + w0 = frames_data[0].data[7:] # frame.data[7:] for A5[0] strt_pos = w0.find(b"STRT") if strt_pos < 0: log.warning("_decode_a5_waveform: STRT record not found in A5[0]") @@ -1499,7 +1538,7 @@ def _decode_a5_waveform( global_offset = 0 for fi, db in enumerate(frames_data): - w = db[7:] + w = db.data[7:] # frame.data[7:] # A5[0]: waveform begins after the 21-byte STRT record and 6-byte preamble. # Layout: STRT(21B) + null-pad(2B) + 0xFF sentinel(4B) = 27 bytes total. @@ -1770,10 +1809,13 @@ def _encode_compliance_config( DLE-jitter shifts): Anchor: b'\\xbe\\x80\\x00\\x00\\x00\\x00' (confirmed stable, both BE11529 and BE18189) - recording_mode → uint8 at anchor_pos - 7 (write payload) + recording_mode → uint8 at anchor_pos - 8 (BOTH read and write) Values: 0x00=Single Shot, 0x01=Continuous, 0x03=Histogram, 0x04=Histogram+Continuous - NOTE: In the E5 read response (decode) field is at anchor_pos - 8 due to an - extra 0x10 byte at read anchor_pos - 7. Write payload has no extra byte. + NOTE: The byte at anchor_pos - 7 is always 0x10 (a DLE marker regenerated by + device firmware in every E5 response). It must NOT be overwritten during + write — doing so causes anchor drift (+1 per write cycle). + CORRECTION 2026-04-21: previous doc stated anchor-7 for write; empirically + confirmed wrong — writing to anchor-7 shifts the anchor by 1 on every cycle. sample_rate → uint16 BE at anchor_pos - 6 histogram_interval_sec → uint16 BE at anchor_pos - 4 (seconds; mode-gated to Histogram/Histogram+Continuous) Valid values: 2, 5, 15, 60, 300, 900 (= 2s, 5s, 15s, 1m, 5m, 15m) @@ -1833,13 +1875,40 @@ def _encode_compliance_config( _ANC = b'\xbe\x80\x00\x00\x00\x00' _anc = buf.find(_ANC, 0, 150) + # Log anchor position every time so we can detect unexpected shifts due to + # DLE jitter or firmware differences. Expected position is ~15. + if _anc < 0: + log.warning( + "_encode_compliance_config: anchor NOT FOUND in cfg[0:150] " + "(buf len=%d) — all anchor-relative writes will be skipped", + len(buf), + ) + else: + log.info( + "_encode_compliance_config: anchor at cfg[%d] buf_len=%d " + "(recording_mode@%d DLE_marker@%d sample_rate@%d:%d " + "histogram_interval@%d:%d record_time@%d:%d)", + _anc, len(buf), + _anc - 8, + _anc - 7, + _anc - 6, _anc - 4, + _anc - 4, _anc - 2, + _anc + 6, _anc + 10, + ) + if recording_mode is not None: - if _anc < 7: + if _anc < 8: log.warning("_encode_compliance_config: anchor not found — cannot write recording_mode") else: - buf[_anc - 7] = recording_mode & 0xFF + # Write to anchor-8, same physical position as the E5 read format. + # The byte at anchor-7 is a DLE marker (0x10) that the device firmware + # regenerates in every E5 response — it must NOT be overwritten. + # Writing to anchor-7 causes the device to add an extra byte on the + # next read-back, drifting the anchor by +1 on every write cycle. + # (CLAUDE.md "anchor-7 write" was incorrect — confirmed 2026-04-21) + buf[_anc - 8] = recording_mode & 0xFF log.debug("_encode_compliance_config: recording_mode=0x%02X -> offset %d", - recording_mode, _anc - 7) + recording_mode, _anc - 8) if sample_rate is not None: if _anc < 6: @@ -2001,6 +2070,27 @@ def _decode_compliance_config_into(data: bytes, info: DeviceInfo) -> None: # _anchor + 6 : record_time (float32 BE) _ANCHOR = b'\xbe\x80\x00\x00\x00\x00' _anchor = data.find(_ANCHOR, 0, 150) + + # Log anchor position on every decode so we can compare read vs write and + # catch unexpected shifts from DLE jitter or firmware differences. + # Expected position is ~15 for the E5 read payload (anchor - 8 = recording_mode). + if _anchor < 0: + log.warning( + "_decode_compliance_config_into: anchor NOT FOUND in data[0:150] (len=%d)", + len(data), + ) + else: + log.info( + "_decode_compliance_config_into: anchor at data[%d] data_len=%d " + "(expected ~15; recording_mode@%d sample_rate@%d:%d " + "histogram_interval@%d:%d record_time@%d:%d)", + _anchor, len(data), + _anchor - 8, + _anchor - 6, _anchor - 4, + _anchor - 4, _anchor - 2, + _anchor + 6, _anchor + 10, + ) + if _anchor >= 8 and _anchor + 10 <= len(data): try: config.recording_mode = data[_anchor - 8] diff --git a/minimateplus/framing.py b/minimateplus/framing.py index 31c1fba..3adf4ce 100644 --- a/minimateplus/framing.py +++ b/minimateplus/framing.py @@ -457,6 +457,11 @@ class S3Frame: page_lo: int # PAGE_LO from header data: bytes # payload data section (payload[5:], checksum already stripped) checksum_valid: bool + chk_byte: int = 0 # actual checksum byte received from wire (body[-1]) + # needed for waveform file reconstruction: when the last data byte + # is 0x10 and chk_byte ∈ {0x02, 0x03, 0x04}, the DLE+chk pair + # must be included in the DLE-strip operation to correctly + # reconstruct the Blastware binary body. @property def page_key(self) -> int: @@ -592,9 +597,10 @@ class S3FrameParser: return None return S3Frame( - sub = raw_payload[2], - page_hi = raw_payload[3], - page_lo = raw_payload[4], - data = raw_payload[5:], + sub = raw_payload[2], + page_hi = raw_payload[3], + page_lo = raw_payload[4], + data = raw_payload[5:], checksum_valid = (chk_received == chk_computed), + chk_byte = chk_received, ) diff --git a/minimateplus/models.py b/minimateplus/models.py index cdb74d1..47d4028 100644 --- a/minimateplus/models.py +++ b/minimateplus/models.py @@ -493,6 +493,10 @@ class Event: # Set by get_events(); required by download_waveform(). _waveform_key: Optional[bytes] = field(default=None, repr=False) + # Raw A5 frames from the full bulk waveform download (full_waveform=True). + # Populated by get_events() when full_waveform=True; used by write_blastware_file(). + _a5_frames: Optional[list] = field(default=None, repr=False) + def __str__(self) -> str: ts = str(self.timestamp) if self.timestamp else "no timestamp" ppv = "" diff --git a/minimateplus/protocol.py b/minimateplus/protocol.py index 0e2f048..0a69f93 100644 --- a/minimateplus/protocol.py +++ b/minimateplus/protocol.py @@ -126,10 +126,12 @@ DATA_LENGTHS: dict[int, int] = { _BULK_CHUNK_OFFSET = 0x1004 # offset field for probe + all regular chunk requests ✅ _BULK_TERM_OFFSET = 0x005A # offset field for termination request ✅ _BULK_COUNTER_STEP = 0x0400 # chunk counter increment per chunk ✅ -# Chunk counter formula: chunk_num * 0x0400 for ALL chunks including chunk 1. -# Earlier captures showed 0x1004 for chunk 1 — that was a Blastware artifact, not a -# protocol requirement. Confirmed 2026-04-06: 0x0400 for chunk 1 works; 0x1004 -# causes a 120-second device timeout. Formula n * 0x0400 is used for all chunks. +# Chunk counter formula: key4[2:4] + (chunk_num - 1) * 0x0400 +# where key4[2:4] is the event's circular-buffer base offset ((key4[2]<<8)|key4[3]). +# Earlier captures showed 0x1004 for chunk 1 of key 01110000 — that was a Blastware +# artifact. For keys where key4[2:4] != 0x0000 (e.g. key 01111884) the old +# "n * 0x0400" formula sends counters from the wrong buffer region and the device +# returns data from a different event. Confirmed correct 2026-04-24. # Default timeout values (seconds). # MiniMate Plus is a slow device — keep these generous. @@ -526,7 +528,9 @@ class MiniMateProtocol: *, stop_after_metadata: bool = True, max_chunks: int = 32, - ) -> list[bytes]: + include_terminator: bool = False, + extra_chunks_after_metadata: int = 1, + ) -> list[S3Frame]: """ Download the SUB 5A (BULK_WAVEFORM_STREAM) A5 frames for one event. @@ -542,7 +546,9 @@ class MiniMateProtocol: 4. Termination (offset=_BULK_TERM_OFFSET, counter=last+_BULK_COUNTER_STEP) Device responds with a final A5 frame (page_key=0x0000). - The termination frame (page_key=0x0000) is NOT included in the returned list. + By default the termination frame (page_key=0x0000) is NOT included in the + returned list. Pass include_terminator=True to append it; the blastware_file + writer needs the terminator frame's body to reconstruct the waveform file footer. Args: key4: 4-byte waveform key from EVENT_HEADER (1E). @@ -552,11 +558,16 @@ class MiniMateProtocol: hundred KB). Set False to download everything. max_chunks: Safety cap on the number of chunk requests sent (default 32; a typical event uses 9 large frames). + include_terminator: If True, append the terminator A5 frame + (page_key=0x0000) to the returned list. The + terminator carries the waveform file footer bytes. + Default False preserves existing caller behaviour. Returns: - List of raw data bytes from each A5 response frame (not including - the terminator frame). Frame indices match the request sequence: - index 0 = probe response, index 1 = first chunk, etc. + List of S3Frame objects from each A5 response frame. Frame indices + match the request sequence: index 0 = probe response, index 1 = first + chunk, etc. If include_terminator=True, the last element is the + terminator frame (page_key=0x0000). Raises: ProtocolError: on timeout, bad checksum, or unexpected SUB. @@ -571,11 +582,19 @@ class MiniMateProtocol: raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}") rsp_sub = _expected_rsp_sub(SUB_BULK_WAVEFORM) # 0xFF - 0x5A = 0xA5 - frames_data: list[bytes] = [] + frames_data: list[S3Frame] = [] counter = 0 + # BW counter formula (confirmed from 4-3-26 capture for key 0111245a, + # and empirical live-device test 2026-04-06 for key 01110000): + # counter for chunk n = max(key4[2:4], 0x0400) + (n - 1) * 0x0400 + # key4[2:4] is the event's circular-buffer base offset. The max() guard + # ensures chunk 1 never uses counter=0x0000 (which equals the probe address + # and causes the device to re-return STRT record data for the first chunk). + _key4_offset = (key4[2] << 8) | key4[3] + # ── Step 1: probe ──────────────────────────────────────────────────── - log.debug("5A probe key=%s", key4.hex()) + log.debug("5A probe key=%s key4_offset=0x%04X", key4.hex(), _key4_offset) params = bulk_waveform_params(key4, 0, is_probe=True) self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params)) self._parser.reset() # reset bytes_fed counter before probe recv @@ -588,17 +607,28 @@ class MiniMateProtocol: key4.hex(), self._parser.bytes_fed, ) raise - frames_data.append(rsp.data) + frames_data.append(rsp) log.debug("5A A5[0] page_key=0x%04X %d bytes", rsp.page_key, len(rsp.data)) # ── Step 2: chunk loop ─────────────────────────────────────────────── - # Chunk counters are monotonic: chunk_num * 0x0400 for all chunks. - # The 4-2-26 BW TX capture showed 0x1004 for chunk 1, but this is a - # Blastware artifact — the device accepts any counter value and streams - # data regardless. Empirically confirmed 2026-04-06: 0x0400 for chunk 1 - # works; 0x1004 causes the device to ignore the frame (timeout). + # Counter formula: _chunk_base + (chunk_num - 1) * 0x0400 + # where _chunk_base = max(key4[2:4], 0x0400). + # + # For events with key4[2:4] != 0 (e.g. key 0111245a, offset 0x245a): + # _chunk_base = 0x245a → chunk 1=0x245a, chunk 2=0x285a, ... + # Confirmed from 4-3-26 capture. + # + # For events with key4[2:4] == 0 (e.g. key 01110000): + # _chunk_base = max(0, 0x0400) = 0x0400 + # → chunk 1=0x0400, chunk 2=0x0800, ... (= old chunk_num*0x0400) + # CRITICAL: counter=0x0000 (same as the probe) causes the device to + # re-return the STRT record data for chunk 1, making frame 1 look like + # a second probe response (confirmed from server log: frame 1 len=1097, + # contains STRT\xff\xfe, contributes zero body bytes after DLE-strip). + # counter=0x0400 for chunk 1 confirmed working (empirical test 2026-04-06). + _chunk_base = max(_key4_offset, _BULK_COUNTER_STEP) for chunk_num in range(1, max_chunks + 1): - counter = chunk_num * _BULK_COUNTER_STEP + counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP params = bulk_waveform_params(key4, counter) log.debug("5A chunk %d counter=0x%04X", chunk_num, counter) self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params)) @@ -631,12 +661,43 @@ class MiniMateProtocol: if rsp.page_key == 0x0000: # Device unexpectedly terminated mid-stream (no termination needed). log.debug("5A A5[%d] page_key=0x0000 — device terminated early", chunk_num) + if include_terminator: + frames_data.append(rsp) return frames_data - frames_data.append(rsp.data) + frames_data.append(rsp) if stop_after_metadata and b"Project:" in rsp.data: - log.debug("5A A5[%d] metadata found — stopping early", chunk_num) + # Download exactly one more chunk after finding metadata — this is + # what Blastware does. The extra chunk contains the tail ADC data + # and primes the device to return a valid footer in the termination + # response. Without it, termination returns an empty ack with no + # footer bytes (confirmed 2026-04-23 from HxD comparison). + # Download extra_chunks_after_metadata more chunks past the + # metadata. The caller calculates this from record_time and + # sample_rate so we download exactly the right amount of ADC + # data — no more, no less — before terminating. + # The device returns the footer in the termination response only + # after the right amount of data has been consumed. + log.debug("5A A5[%d] metadata found — fetching %d more chunk(s)", + chunk_num, extra_chunks_after_metadata) + for _extra_n in range(extra_chunks_after_metadata): + chunk_num += 1 + counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP + params = bulk_waveform_params(key4, counter) + self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params)) + try: + extra = self._recv_one(expected_sub=rsp_sub, timeout=10.0) + log.debug("5A A5[%d] extra chunk page_key=0x%04X data_len=%d", + chunk_num, extra.page_key, len(extra.data)) + if extra.page_key == 0x0000: + if include_terminator: + frames_data.append(extra) + return frames_data + frames_data.append(extra) + except TimeoutError: + log.debug("5A extra chunk %d timed out — end of stream", _extra_n + 1) + break break else: log.warning( @@ -658,6 +719,8 @@ class MiniMateProtocol: "5A termination response page_key=0x%04X %d bytes", term_rsp.page_key, len(term_rsp.data), ) + if include_terminator: + frames_data.append(term_rsp) except TimeoutError: log.debug("5A no termination response — device may have already closed") diff --git a/minimateplus/transport.py b/minimateplus/transport.py index 65249d8..a0ee32b 100644 --- a/minimateplus/transport.py +++ b/minimateplus/transport.py @@ -454,3 +454,102 @@ class SocketTransport(TcpTransport): def __repr__(self) -> str: return f"SocketTransport(peer={self.host!r})" + + +# ── Capturing transport (MITM-style raw byte mirror) ────────────────────────── + +class CapturingTransport(BaseTransport): + """ + Wraps another BaseTransport and mirrors every byte to two raw capture files: + + raw_bw_<...>.bin — bytes WE wrote to the device (BW-side TX) + raw_s3_<...>.bin — bytes the device wrote back (S3-side TX) + + The file naming and on-wire byte layout are identical to the captures + produced by `bridges/ach_mitm.py`, so the resulting `.bin` files can be + loaded directly by the Analyzer (File > Open Capture) and parsed by the + same tooling used for genuine Blastware MITM captures. + + All BaseTransport methods are forwarded to the inner transport; the only + side-effect is that successful read/write byte streams are appended to the + two open binary files. + + Args: + inner: An already-built BaseTransport (SerialTransport / TcpTransport). + bw_path: File path for the "BW TX" stream (bytes we send). Opened "wb". + s3_path: File path for the "S3 TX" stream (bytes the device sends). + Opened "wb". + + Example: + with CapturingTransport(TcpTransport("1.2.3.4", 9034), + "raw_bw.bin", "raw_s3.bin") as t: + client = MiniMateClient(transport=t) + client.connect() + client.get_events() + # both .bin files now hold the full bidirectional capture. + """ + + def __init__(self, inner: BaseTransport, bw_path: str, s3_path: str) -> None: + self._inner = inner + self._bw_path = bw_path + self._s3_path = s3_path + self._bw_fh = None + self._s3_fh = None + # Forward inner attrs so callers can introspect (e.g. .host, .port). + self.host = getattr(inner, "host", None) + self.port = getattr(inner, "port", None) + + # ── BaseTransport interface ─────────────────────────────────────────────── + + def connect(self) -> None: + if self._bw_fh is None: + self._bw_fh = open(self._bw_path, "wb", buffering=0) + if self._s3_fh is None: + self._s3_fh = open(self._s3_path, "wb", buffering=0) + self._inner.connect() + + def disconnect(self) -> None: + try: + self._inner.disconnect() + finally: + for fh_attr in ("_bw_fh", "_s3_fh"): + fh = getattr(self, fh_attr) + if fh is not None: + try: + fh.flush() + fh.close() + except Exception: + pass + setattr(self, fh_attr, None) + + @property + def is_connected(self) -> bool: + return self._inner.is_connected + + def write(self, data: bytes) -> None: + self._inner.write(data) + if data and self._bw_fh is not None: + try: + self._bw_fh.write(data) + except Exception: + pass + + def read(self, n: int) -> bytes: + got = self._inner.read(n) + if got and self._s3_fh is not None: + try: + self._s3_fh.write(got) + except Exception: + pass + return got + + @property + def bw_path(self) -> str: + return self._bw_path + + @property + def s3_path(self) -> str: + return self._s3_path + + def __repr__(self) -> str: + return f"CapturingTransport({self._inner!r}, bw={self._bw_path!r}, s3={self._s3_path!r})" diff --git a/parsers/s3_analyzer.py b/parsers/s3_analyzer.py index c86477d..6feb32b 100644 --- a/parsers/s3_analyzer.py +++ b/parsers/s3_analyzer.py @@ -53,7 +53,9 @@ SUB_TABLE: dict[int, tuple[str, str, str]] = { 0x82: ("TRIGGER_CONFIG_WRITE", "BW→S3", "0x1C bytes; trigger config block; mirrors SUB 1C"), 0x83: ("TRIGGER_WRITE_CONFIRM", "BW→S3", "Short frame; commit step after 0x82"), # S3→BW responses + 0x5A: ("BULK_WAVEFORM_STREAM", "BW→S3", "Bulk waveform chunk request; response is A5 stream"), 0xA4: ("POLL_RESPONSE", "S3→BW", "Response to SUB 5B poll"), + 0xA5: ("BULK_WAVEFORM_RESPONSE", "S3→BW", "Response to SUB 5A; waveform chunks + metadata"), 0xFE: ("FULL_CONFIG_RESPONSE", "S3→BW", "Response to SUB 01"), 0xF9: ("CHANNEL_CONFIG_RESPONSE", "S3→BW", "Response to SUB 06"), 0xF7: ("EVENT_INDEX_RESPONSE", "S3→BW", "Response to SUB 08; contains backlight/power-save"), diff --git a/parsers/s3_parser.py b/parsers/s3_parser.py index 2f32933..2bb3de1 100644 --- a/parsers/s3_parser.py +++ b/parsers/s3_parser.py @@ -33,7 +33,7 @@ STX = 0x02 ETX = 0x03 ACK = 0x41 -__version__ = "0.2.3" +__version__ = "0.2.5" @dataclass @@ -184,9 +184,9 @@ def validate_bw_body_auto(body: bytes) -> Optional[Tuple[bytes, bytes, str]]: def parse_s3(blob: bytes, trailer_len: int) -> List[Frame]: frames: List[Frame] = [] - IDLE = 0 - IN_FRAME = 1 - AFTER_DLE = 2 + IDLE = 0 + IN_FRAME = 1 + IN_FRAME_DLE = 2 # saw DLE inside frame — waiting for next byte state = IDLE body = bytearray() @@ -206,66 +206,63 @@ def parse_s3(blob: bytes, trailer_len: int) -> List[Frame]: state = IN_FRAME i += 2 continue + # ACK bytes, boot strings, garbage — silently ignored elif state == IN_FRAME: if b == DLE: - state = AFTER_DLE + state = IN_FRAME_DLE i += 1 continue - body.append(b) - - else: # AFTER_DLE - if b == DLE: - body.append(DLE) - state = IN_FRAME - i += 1 - continue - if b == ETX: + # Bare ETX = real S3 frame terminator (confirmed from S3FrameParser) end_offset = i + 1 trailer_start = i + 1 trailer_end = trailer_start + trailer_len trailer = blob[trailer_start:trailer_end] - chk_valid = None - chk_type = None - chk_hex = None - payload = bytes(body) - - if len(body) >= 1: - received_chk = body[-1] - computed_chk = checksum8_sum(bytes(body[:-1])) - if computed_chk == received_chk: - chk_valid = True - chk_type = "SUM8" - chk_hex = f"{received_chk:02x}" - payload = bytes(body[:-1]) - else: - chk_valid = False - + # S3 checksums are deliberately not validated here. + # Large S3 responses (A5 bulk waveform, E5 compliance) embed + # inner DLE+ETX sub-frame terminators whose trailing 0x03 byte + # lands where the parser would expect the SUM8 checksum, causing + # false failures. The live protocol (protocol.py _validate_frame) + # also skips S3 checksum enforcement for the same reason. frames.append(Frame( index=idx, start_offset=start_offset, end_offset=end_offset, payload_raw=bytes(body), - payload=payload, + payload=bytes(body), trailer=trailer, - checksum_valid=chk_valid, - checksum_type=chk_type, - checksum_hex=chk_hex + checksum_valid=None, + checksum_type=None, + checksum_hex=None )) idx += 1 state = IDLE i = trailer_end continue + body.append(b) + else: # IN_FRAME_DLE + if b == DLE: + # DLE DLE → literal 0x10 in payload + body.append(DLE) + state = IN_FRAME + i += 1 + continue + if b == ETX: + # DLE+ETX inside a frame = inner-frame terminator (A4/E5 sub-frames). + # Treat as literal data, NOT the outer frame end. + body.append(DLE) + body.append(ETX) + state = IN_FRAME + i += 1 + continue # Unexpected DLE + byte → treat as literal data body.append(DLE) body.append(b) state = IN_FRAME - i += 1 - continue i += 1 diff --git a/seismo_lab.py b/seismo_lab.py index 899f85e..1986127 100644 --- a/seismo_lab.py +++ b/seismo_lab.py @@ -22,6 +22,7 @@ from __future__ import annotations import datetime import os import queue +import socket import subprocess import sys import threading @@ -96,10 +97,21 @@ class AnalyzerState: class BridgePanel(tk.Frame): """ - All bridge controls and live log output. - Calls on_bridge_started(struct_bin_path) when the bridge starts. - Calls on_capture_started(bw_path, s3_path, label) when a capture begins. - Calls on_capture_complete(bw_path, s3_path, label) when a capture ends. + Bridge controls and live log output. + + Two modes selectable at the top: + - Serial: wraps s3_bridge.py as a subprocess (two COM ports). + Single bridge session; use New Capture / Stop Capture to create + labelled raw-file segments on demand. + - TCP: MITM proxy — listens for Blastware on a local port, forwards to + the real device. Each incoming connection is a capture; segments + appear in the history list automatically. + + Callbacks (all optional except on_bridge_started / on_bridge_stopped): + on_bridge_started(struct_bin_path) — bridge is up + on_bridge_stopped() — bridge stopped + on_capture_started(bw_path, s3_path, label) — a capture segment began + on_capture_complete(bw_path, s3_path, label)— a capture segment finished """ def __init__(self, parent: tk.Widget, on_bridge_started, on_bridge_stopped, @@ -111,12 +123,25 @@ class BridgePanel(tk.Frame): self._on_cap_complete = on_capture_complete # (bw, s3, label) self.process: Optional[subprocess.Popen] = None self._stdout_q: queue.Queue[str] = queue.Queue() - # Capture state - self._capturing = False + # tcp state + self._server: Optional[socket.socket] = None + self._tcp_stop_event = threading.Event() + self._tcp_log_q: queue.Queue[str] = queue.Queue() + # tcp capture file handles — written only when capture is active + self._tcp_cap_lock = threading.Lock() + self._tcp_cap_bw_fh = None + self._tcp_cap_s3_fh = None + self._tcp_cap_bw_path: Optional[str] = None + self._tcp_cap_s3_path: Optional[str] = None + # shared capture state + self._capturing = False self._cap_label: Optional[str] = None self._cap_history: list[dict] = [] # {label, status, bw, s3} + # mode + self._mode = tk.StringVar(value="serial") self._build() self._poll_stdout() + self._poll_tcp_log() # ── build ───────────────────────────────────────────────────────────── @@ -126,35 +151,70 @@ class BridgePanel(tk.Frame): cfg = tk.Frame(self, bg=BG2) cfg.pack(side=tk.TOP, fill=tk.X, padx=4, pady=4) - # Row 0: ports - tk.Label(cfg, text="BW COM:", bg=BG2, fg=FG, font=MONO).grid(row=0, column=0, sticky="e", **pad) + # Row 0: mode selector + mode_row = tk.Frame(cfg, bg=BG2) + mode_row.grid(row=0, column=0, columnspan=6, sticky="w", padx=6, pady=(4, 0)) + tk.Label(mode_row, text="Mode:", bg=BG2, fg=FG, font=MONO).pack(side=tk.LEFT, padx=(0, 8)) + tk.Radiobutton(mode_row, text="Serial", variable=self._mode, value="serial", + bg=BG2, fg=FG, selectcolor=BG3, activebackground=BG2, + font=MONO, command=self._on_mode_change).pack(side=tk.LEFT, padx=4) + tk.Radiobutton(mode_row, text="TCP", variable=self._mode, value="tcp", + bg=BG2, fg=FG, selectcolor=BG3, activebackground=BG2, + font=MONO, command=self._on_mode_change).pack(side=tk.LEFT, padx=4) + + # Row 1a: serial connection fields (shown by default) + self._serial_frame = tk.Frame(cfg, bg=BG2) + self._serial_frame.grid(row=1, column=0, columnspan=6, sticky="w") + + tk.Label(self._serial_frame, text="BW COM:", bg=BG2, fg=FG, font=MONO).grid(row=0, column=0, sticky="e", **pad) self.bw_var = tk.StringVar(value="COM4") - tk.Entry(cfg, textvariable=self.bw_var, width=10, + tk.Entry(self._serial_frame, textvariable=self.bw_var, width=10, bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO).grid(row=0, column=1, sticky="w", **pad) - tk.Label(cfg, text="S3 COM:", bg=BG2, fg=FG, font=MONO).grid(row=0, column=2, sticky="e", **pad) + tk.Label(self._serial_frame, text="S3 COM:", bg=BG2, fg=FG, font=MONO).grid(row=0, column=2, sticky="e", **pad) self.s3_var = tk.StringVar(value="COM5") - tk.Entry(cfg, textvariable=self.s3_var, width=10, + tk.Entry(self._serial_frame, textvariable=self.s3_var, width=10, bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO).grid(row=0, column=3, sticky="w", **pad) - tk.Label(cfg, text="Baud:", bg=BG2, fg=FG, font=MONO).grid(row=0, column=4, sticky="e", **pad) + tk.Label(self._serial_frame, text="Baud:", bg=BG2, fg=FG, font=MONO).grid(row=0, column=4, sticky="e", **pad) self.baud_var = tk.StringVar(value="38400") - tk.Entry(cfg, textvariable=self.baud_var, width=8, + tk.Entry(self._serial_frame, textvariable=self.baud_var, width=8, bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO).grid(row=0, column=5, sticky="w", **pad) - # Row 1: log dir - tk.Label(cfg, text="Log dir:", bg=BG2, fg=FG, font=MONO).grid(row=1, column=0, sticky="e", **pad) + # Row 1b: TCP connection fields (hidden until TCP mode selected) + self._tcp_frame = tk.Frame(cfg, bg=BG2) + + tk.Label(self._tcp_frame, text="Listen port:", bg=BG2, fg=FG, font=MONO).grid(row=0, column=0, sticky="e", **pad) + self.listen_port_var = tk.StringVar(value="9034") + tk.Entry(self._tcp_frame, textvariable=self.listen_port_var, width=8, + bg=BG3, fg=FG, insertbackground=FG, relief="flat", + font=MONO).grid(row=0, column=1, sticky="w", **pad) + + tk.Label(self._tcp_frame, text="Device host:", bg=BG2, fg=FG, font=MONO).grid(row=0, column=2, sticky="e", **pad) + self.remote_host_var = tk.StringVar(value="63.43.212.232") + tk.Entry(self._tcp_frame, textvariable=self.remote_host_var, width=18, + bg=BG3, fg=FG, insertbackground=FG, relief="flat", + font=MONO).grid(row=0, column=3, sticky="w", **pad) + + tk.Label(self._tcp_frame, text="Port:", bg=BG2, fg=FG, font=MONO).grid(row=0, column=4, sticky="e", **pad) + self.remote_port_var = tk.StringVar(value="9034") + tk.Entry(self._tcp_frame, textvariable=self.remote_port_var, width=8, + bg=BG3, fg=FG, insertbackground=FG, relief="flat", + font=MONO).grid(row=0, column=5, sticky="w", **pad) + + # Row 2: log dir + tk.Label(cfg, text="Log dir:", bg=BG2, fg=FG, font=MONO).grid(row=2, column=0, sticky="e", **pad) self.logdir_var = tk.StringVar(value=str(SCRIPT_DIR / "bridges" / "captures")) tk.Entry(cfg, textvariable=self.logdir_var, width=40, bg=BG3, fg=FG, insertbackground=FG, relief="flat", - font=MONO).grid(row=1, column=1, columnspan=4, sticky="we", **pad) + font=MONO).grid(row=2, column=1, columnspan=4, sticky="we", **pad) tk.Button(cfg, text="Browse", bg=BG3, fg=FG, relief="flat", cursor="hand2", - font=MONO, command=self._choose_dir).grid(row=1, column=5, **pad) + font=MONO, command=self._choose_dir).grid(row=2, column=5, **pad) - # Row 2: buttons + status + # Row 3: buttons + status btn_row = tk.Frame(self, bg=BG2) btn_row.pack(side=tk.TOP, fill=tk.X, padx=4, pady=2) @@ -170,7 +230,7 @@ class BridgePanel(tk.Frame): tk.Frame(btn_row, bg=BG2, width=16).pack(side=tk.LEFT) # spacer - self.cap_btn = tk.Button(btn_row, text="⬤ New Capture", bg=ORANGE, fg="#000000", + self.cap_btn = tk.Button(btn_row, text="● New Capture", bg=ORANGE, fg="#000000", relief="flat", padx=10, cursor="hand2", font=MONO_B, command=self._start_capture, state="disabled") self.cap_btn.pack(side=tk.LEFT, padx=4) @@ -224,6 +284,14 @@ class BridgePanel(tk.Frame): # ── helpers ─────────────────────────────────────────────────────────── + def _on_mode_change(self) -> None: + if self._mode.get() == "serial": + self._tcp_frame.grid_remove() + self._serial_frame.grid(row=1, column=0, columnspan=6, sticky="w") + else: + self._serial_frame.grid_remove() + self._tcp_frame.grid(row=1, column=0, columnspan=6, sticky="w") + def _choose_dir(self) -> None: path = filedialog.askdirectory(initialdir=self.logdir_var.get()) if path: @@ -235,9 +303,79 @@ class BridgePanel(tk.Frame): self.log_view.see(tk.END) self.log_view.configure(state="disabled") - # ── bridge control ──────────────────────────────────────────────────── + def _refresh_hist(self) -> None: + self._hist_lb.delete(0, tk.END) + for entry in self._cap_history: + icon = "\U0001f534" if entry["status"] == "recording" else "✅" + self._hist_lb.insert(tk.END, f" {icon} {entry['label'] or '(unlabeled)'}") + if self._cap_history: + self._hist_lb.see(tk.END) + + def _on_hist_dblclick(self, _e=None) -> None: + sel = self._hist_lb.curselection() + if not sel: + return + entry = self._cap_history[sel[0]] + if entry["status"] == "done" and entry["bw"] and entry["s3"] and self._on_cap_complete: + self._on_cap_complete(entry["bw"], entry["s3"], entry["label"]) + + # ── bridge control (delegates to serial or TCP) ─────────────────────── def start_bridge(self) -> None: + if self._mode.get() == "tcp": + self._start_tcp() + else: + self._start_serial() + + def stop_bridge(self) -> None: + if self._mode.get() == "tcp": + self._stop_tcp() + else: + self._stop_serial() + + def _bridge_ended(self) -> None: + self.status_var.set("Stopped") + self.start_btn.configure(state="normal") + self.stop_btn.configure(state="disabled", bg=BG3) + self.cap_btn.configure(state="disabled") + self.stop_cap_btn.configure(state="disabled", bg=BG3) + self.mark_btn.configure(state="disabled") + self._capturing = False + self._cap_label = None + self._append_log("== Bridge stopped ==\n") + + # ── capture lifecycle (shared by serial and TCP) ────────────────────── + + def _on_cap_started_msg(self, bw_path: str, s3_path: str) -> None: + for entry in reversed(self._cap_history): + if entry["status"] == "recording" and entry["bw"] is None: + entry["bw"] = bw_path + entry["s3"] = s3_path + break + self._refresh_hist() + if self._on_cap_started: + self._on_cap_started(bw_path, s3_path, self._cap_label or "") + + def _on_cap_stopped_msg(self, bw_path: str, s3_path: str) -> None: + label = self._cap_label or "capture" + for entry in reversed(self._cap_history): + if entry["status"] == "recording": + entry["status"] = "done" + entry["bw"] = bw_path + entry["s3"] = s3_path + break + self._refresh_hist() + self._capturing = False + self._cap_label = None + self.cap_btn.configure(state="normal") + self.stop_cap_btn.configure(state="disabled", bg=BG3) + self._append_log(f"[CAPTURE] Done: {label!r} — ready in Analyzer\n") + if self._on_cap_complete: + self._on_cap_complete(bw_path, s3_path, label) + + # ── serial mode ─────────────────────────────────────────────────────── + + def _start_serial(self) -> None: if self.process and self.process.poll() is None: messagebox.showinfo("Bridge", "Bridge is already running.") return @@ -281,12 +419,11 @@ class BridgePanel(tk.Frame): self.stop_btn.configure(state="normal", bg=RED) self.cap_btn.configure(state="normal") self._append_log(f"== Bridge started [{ts}] ==\n") - self._append_log(" Click 'New Capture' when ready to record a setting change.\n") - + self._append_log(" Click 'New Capture' when ready to record.\n") # Notify parent — no raw files yet, just the structured bin path self._on_started(struct_bin_path) - def stop_bridge(self) -> None: + def _stop_serial(self) -> None: if self.process and self.process.poll() is None: self.process.terminate() try: @@ -296,17 +433,6 @@ class BridgePanel(tk.Frame): self._bridge_ended() self._on_stopped() - def _bridge_ended(self) -> None: - self.status_var.set("Stopped") - self.start_btn.configure(state="normal") - self.stop_btn.configure(state="disabled", bg=BG3) - self.cap_btn.configure(state="disabled") - self.stop_cap_btn.configure(state="disabled", bg=BG3) - self.mark_btn.configure(state="disabled") - self._capturing = False - self._cap_label = None - self._append_log("== Bridge stopped ==\n") - def _reader_thread(self) -> None: if not self.process or not self.process.stdout: return @@ -347,9 +473,7 @@ class BridgePanel(tk.Frame): # ── capture control ─────────────────────────────────────────────────── def _start_capture(self) -> None: - """Ask for a label and tell the bridge to start writing raw tap files.""" - if not self.process or self.process.poll() is not None: - return + """Ask for a label and start writing raw tap files (serial subprocess or TCP files).""" label = simpledialog.askstring( "New Capture", "Label for this capture\n(e.g. 'recording_mode_continuous').\nLeave blank for timestamp only:", @@ -358,25 +482,62 @@ class BridgePanel(tk.Frame): if label is None: return # user hit Cancel label = label.strip() - try: - self.process.stdin.write(f"CAP_START:{label}\n") - self.process.stdin.flush() - except Exception as e: - messagebox.showerror("Error", f"Failed to start capture:\n{e}") - return self._capturing = True self._cap_label = label or datetime.datetime.now().strftime("%H%M%S") + + if self._mode.get() == "tcp": + # TCP: open the capture files now; pipe threads write here while active + logdir = self.logdir_var.get().strip() or "." + os.makedirs(logdir, exist_ok=True) + ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + safe_label = self._cap_label.replace(" ", "_") if self._cap_label else "" + suffix = f"_{safe_label}" if safe_label else "" + bw_path = os.path.join(logdir, f"raw_bw_{ts}{suffix}.bin") + s3_path = os.path.join(logdir, f"raw_s3_{ts}{suffix}.bin") + with self._tcp_cap_lock: + self._tcp_cap_bw_fh = open(bw_path, "wb") + self._tcp_cap_s3_fh = open(s3_path, "wb") + self._tcp_cap_bw_path = bw_path + self._tcp_cap_s3_path = s3_path + self._cap_history.append({"label": self._cap_label, "status": "recording", + "bw": bw_path, "s3": s3_path}) + self._refresh_hist() + self._on_cap_started_msg(bw_path, s3_path) + else: + if not self.process or self.process.poll() is not None: + return + try: + self.process.stdin.write(f"CAP_START:{label}\n") + self.process.stdin.flush() + except Exception as e: + messagebox.showerror("Error", f"Failed to start capture:\n{e}") + return + self._cap_history.append({"label": self._cap_label, "status": "recording", + "bw": None, "s3": None}) + self._refresh_hist() + self.cap_btn.configure(state="disabled") self.stop_cap_btn.configure(state="normal", bg=RED) self.mark_btn.configure(state="normal") self._append_log(f"[CAPTURE] Starting: {self._cap_label!r}...\n") - # Add to history as recording (paths filled in when [CAP_START] arrives) - self._cap_history.append({"label": self._cap_label, "status": "recording", - "bw": None, "s3": None}) - self._refresh_hist() def _stop_capture(self) -> None: - """Tell the bridge to flush and close the current raw tap files.""" + """Flush and close the current raw tap files (TCP) or signal the bridge subprocess (serial).""" + if self._mode.get() == "tcp": + with self._tcp_cap_lock: + bw_path = self._tcp_cap_bw_path + s3_path = self._tcp_cap_s3_path + if self._tcp_cap_bw_fh: + self._tcp_cap_bw_fh.close() + self._tcp_cap_bw_fh = None + if self._tcp_cap_s3_fh: + self._tcp_cap_s3_fh.close() + self._tcp_cap_s3_fh = None + self._tcp_cap_bw_path = None + self._tcp_cap_s3_path = None + if bw_path and s3_path: + self._on_cap_stopped_msg(bw_path, s3_path) + return if not self.process or self.process.poll() is not None: return try: @@ -386,69 +547,167 @@ class BridgePanel(tk.Frame): messagebox.showerror("Error", f"Failed to stop capture:\n{e}") # UI is updated when [CAP_STOP] arrives in stdout - def _on_cap_started_msg(self, bw_path: str, s3_path: str) -> None: - """Called when bridge confirms capture has started (files are open).""" - # Fill in paths for the last 'recording' history entry - for entry in reversed(self._cap_history): - if entry["status"] == "recording" and entry["bw"] is None: - entry["bw"] = bw_path - entry["s3"] = s3_path - break - if self._on_cap_started: - self._on_cap_started(bw_path, s3_path, self._cap_label or "") + # ── TCP mode ────────────────────────────────────────────────────────── - def _on_cap_stopped_msg(self, bw_path: str, s3_path: str) -> None: - """Called when bridge confirms capture has stopped (files are closed).""" - label = self._cap_label or "capture" - # Mark history entry as done - for entry in reversed(self._cap_history): - if entry["status"] == "recording": - entry["status"] = "done" - entry["bw"] = bw_path - entry["s3"] = s3_path - break - self._refresh_hist() - self._capturing = False - self._cap_label = None - self.cap_btn.configure(state="normal") - self.stop_cap_btn.configure(state="disabled", bg=BG3) - self._append_log(f"[CAPTURE] Done: {label!r} — ready in Analyzer\n") - if self._on_cap_complete: - self._on_cap_complete(bw_path, s3_path, label) - - def _refresh_hist(self) -> None: - self._hist_lb.delete(0, tk.END) - for entry in self._cap_history: - icon = "🔴" if entry["status"] == "recording" else "✅" - label = entry["label"] or "(unlabeled)" - self._hist_lb.insert(tk.END, f" {icon} {label}") - if self._cap_history: - self._hist_lb.see(tk.END) - - def _on_hist_dblclick(self, _e=None) -> None: - sel = self._hist_lb.curselection() - if not sel: + def _start_tcp(self) -> None: + if self._server is not None: + messagebox.showinfo("Bridge", "TCP bridge is already listening.") return - entry = self._cap_history[sel[0]] - if entry["status"] == "done" and entry["bw"] and entry["s3"]: - if self._on_cap_complete: - self._on_cap_complete(entry["bw"], entry["s3"], entry["label"]) - # ── mark ────────────────────────────────────────────────────────────── + try: + listen_port = int(self.listen_port_var.get().strip()) + remote_host = self.remote_host_var.get().strip() + remote_port = int(self.remote_port_var.get().strip()) + except ValueError: + messagebox.showerror("Error", "Invalid port number.") + return + if not remote_host: + messagebox.showerror("Error", "Please enter the device host.") + return + + try: + srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + srv.bind(("0.0.0.0", listen_port)) + srv.listen(5) + srv.settimeout(1.0) + except OSError as e: + messagebox.showerror("Error", f"Cannot bind to port {listen_port}:\n{e}") + return + + self._server = srv + self._tcp_stop_event.clear() + self.start_btn.configure(state="disabled") + self.stop_btn.configure(state="normal", bg=RED) + self.cap_btn.configure(state="normal") + self.status_var.set(f"Listening on :{listen_port}") + + ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + self._append_log( + f"== TCP Bridge started [{ts}]\n" + f" Listening on 0.0.0.0:{listen_port}\n" + f" Forwarding to {remote_host}:{remote_port}\n" + f" Click 'New Capture' before the operation you want to record.\n==\n" + ) + self._on_started(None) + + threading.Thread( + target=self._accept_loop, + args=(srv, remote_host, remote_port), + daemon=True, + ).start() + + def _stop_tcp(self) -> None: + # Close any open capture files first + with self._tcp_cap_lock: + if self._tcp_cap_bw_fh: + self._tcp_cap_bw_fh.close() + self._tcp_cap_bw_fh = None + if self._tcp_cap_s3_fh: + self._tcp_cap_s3_fh.close() + self._tcp_cap_s3_fh = None + self._tcp_stop_event.set() + if self._server: + try: + self._server.close() + except OSError: + pass + self._server = None + self._bridge_ended() + self._on_stopped() + + def _accept_loop(self, srv: socket.socket, remote_host: str, remote_port: int) -> None: + while not self._tcp_stop_event.is_set(): + try: + client_sock, addr = srv.accept() + except socket.timeout: + continue + except OSError: + break + + peer = f"{addr[0]}:{addr[1]}" + self._tcp_log_q.put(f"[TCP] Blastware connected from {peer}\n") + + try: + dev_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + dev_sock.connect((remote_host, remote_port)) + except OSError as e: + self._tcp_log_q.put(f"[TCP] Cannot reach device {remote_host}:{remote_port}: {e}\n") + client_sock.close() + continue + + self._tcp_log_q.put(f"[TCP] Connected to device at {remote_host}:{remote_port}\n") + self._run_tcp_session(client_sock, dev_sock) + self._tcp_log_q.put(f"[TCP] Connection from {peer} closed\n") + + def _run_tcp_session(self, bw_sock: socket.socket, dev_sock: socket.socket) -> None: + """Forward bytes in both directions; write to capture files only when active.""" + bw_bytes = [0] + s3_bytes = [0] + + def _pipe(src, dst, get_fh, counter): + try: + while True: + data = src.recv(4096) + if not data: + break + dst.sendall(data) + with self._tcp_cap_lock: + fh = get_fh() + if fh: + fh.write(data) + fh.flush() + counter[0] += len(data) + except OSError: + pass + finally: + try: + dst.shutdown(socket.SHUT_WR) + except OSError: + pass + + t_bw = threading.Thread(target=_pipe, + args=(bw_sock, dev_sock, + lambda: self._tcp_cap_bw_fh, bw_bytes), daemon=True) + t_s3 = threading.Thread(target=_pipe, + args=(dev_sock, bw_sock, + lambda: self._tcp_cap_s3_fh, s3_bytes), daemon=True) + t_bw.start() + t_s3.start() + t_bw.join() + t_s3.join() + bw_sock.close() + dev_sock.close() + + def _poll_tcp_log(self) -> None: + try: + while True: + msg = self._tcp_log_q.get_nowait() + self._append_log(msg) + except queue.Empty: + pass + finally: + self.after(100, self._poll_tcp_log) + + # ── marks ───────────────────────────────────────────────────────────── def add_mark(self) -> None: - if not self.process or not self.process.stdin or self.process.poll() is not None: - return label = simpledialog.askstring("Mark", "Enter label for this mark:", parent=self) if not label or not label.strip(): return - try: - self.process.stdin.write("m\n") - self.process.stdin.write(label.strip() + "\n") - self.process.stdin.flush() - self._append_log(f"[MARK] {label.strip()}\n") - except Exception as e: - messagebox.showerror("Error", f"Failed to send mark:\n{e}") + if self._mode.get() == "tcp": + ts = datetime.datetime.now().strftime("%H:%M:%S") + self._append_log(f"[MARK {ts}] {label.strip()}\n") + else: + if not self.process or not self.process.stdin or self.process.poll() is not None: + return + try: + self.process.stdin.write("m\n") + self.process.stdin.write(label.strip() + "\n") + self.process.stdin.flush() + self._append_log(f"[MARK] {label.strip()}\n") + except Exception as e: + messagebox.showerror("Error", f"Failed to send mark:\n{e}") # ───────────────────────────────────────────────────────────────────────────── @@ -774,14 +1033,6 @@ class AnalyzerPanel(tk.Frame): self.state.bw_path = bwp self._do_analyze(s3p, bwp) - def _browse_bin(self) -> None: - path = filedialog.askopenfilename( - title="Select session .bin file", - filetypes=[("Binary", "*.bin"), ("All files", "*.*")], - ) - if path: - self.bin_var.set(path) - def _do_analyze(self, s3_path: Path, bw_path: Path) -> None: self.status_var.set("Parsing...") self.update_idletasks() @@ -1212,7 +1463,6 @@ class AnalyzerPanel(tk.Frame): w.configure(state="normal"); w.insert(tk.END, "\n"); w.configure(state="disabled") -# ───────────────────────────────────────────────────────────────────────────── # ───────────────────────────────────────────────────────────────────────────── # Serial Watch panel — tap the RS-232 line between device and modem # ───────────────────────────────────────────────────────────────────────────── @@ -1997,6 +2247,434 @@ class ConsolePanel(tk.Frame): self.after(100, self._poll_q) +# ───────────────────────────────────────────────────────────────────────────── +# Download panel — connect to a device, run get_events(), capture wire bytes +# ───────────────────────────────────────────────────────────────────────────── + +class DownloadPanel(tk.Frame): + """ + Connect directly to a MiniMate Plus and download events while transparently + saving every wire byte in the same format as a Blastware MITM capture. + + Each download produces a session directory containing: + + seismo_dl_/raw_bw_.bin — bytes WE sent (BW TX) + seismo_dl_/raw_s3_.bin — bytes the unit sent (S3 TX) + + These files are byte-for-byte compatible with the captures produced by + `bridges/ach_mitm.py` and load directly in the Analyzer tab. + + Use this when you want to reproduce or troubleshoot a flow that Blastware + is doing — any session captured here can be diffed against a real BW + capture to confirm wire-level parity. + """ + + TAG_TX = "tx" + TAG_RX_RAW = "rx_raw" + TAG_PARSED = "parsed" + TAG_ERROR = "error" + TAG_STATUS = "status" + TAG_HEAD = "head" + + MAX_LINES = 5000 + + def __init__(self, parent: tk.Widget, on_capture_ready=None, **kw): + """ + on_capture_ready(bw_path, s3_path, label) — invoked when a capture + completes successfully so the parent can hand the files to the Analyzer. + """ + super().__init__(parent, bg=BG2, **kw) + self._on_capture_ready = on_capture_ready + self._q: queue.Queue = queue.Queue() + self._running = False + self._cmd_btns: list[tk.Button] = [] + self._last_paths: Optional[tuple[str, str, str]] = None # (bw, s3, label) + self._build() + self._poll_q() + + # ── build ───────────────────────────────────────────────────────────── + + def _build(self) -> None: + pad = {"padx": 5, "pady": 3} + + cfg = tk.Frame(self, bg=BG2) + cfg.pack(side=tk.TOP, fill=tk.X, padx=6, pady=4) + + # Transport radio + self._transport_var = tk.StringVar(value="tcp") + tk.Radiobutton( + cfg, text="TCP", variable=self._transport_var, value="tcp", + bg=BG2, fg=FG, selectcolor=BG3, activebackground=BG2, + font=MONO, command=self._on_transport_change, + ).grid(row=0, column=0, padx=(0, 4)) + tk.Radiobutton( + cfg, text="Serial", variable=self._transport_var, value="serial", + bg=BG2, fg=FG, selectcolor=BG3, activebackground=BG2, + font=MONO, command=self._on_transport_change, + ).grid(row=0, column=1, padx=(0, 12)) + + # TCP fields + self._tcp_frame = tk.Frame(cfg, bg=BG2) + self._tcp_frame.grid(row=0, column=2, sticky="w") + tk.Label(self._tcp_frame, text="Host:", bg=BG2, fg=FG, font=MONO + ).pack(side=tk.LEFT, **pad) + self._host_var = tk.StringVar(value="127.0.0.1") + tk.Entry(self._tcp_frame, textvariable=self._host_var, width=18, + bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO, + ).pack(side=tk.LEFT, padx=2) + tk.Label(self._tcp_frame, text="Port:", bg=BG2, fg=FG, font=MONO + ).pack(side=tk.LEFT, padx=(10, 4)) + self._tcp_port_var = tk.StringVar(value="9034") + tk.Entry(self._tcp_frame, textvariable=self._tcp_port_var, width=6, + bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO, + ).pack(side=tk.LEFT, padx=2) + + # Serial fields (hidden by default) + self._serial_frame = tk.Frame(cfg, bg=BG2) + tk.Label(self._serial_frame, text="Port:", bg=BG2, fg=FG, font=MONO + ).pack(side=tk.LEFT, **pad) + self._port_var = tk.StringVar(value="COM5") + tk.Entry(self._serial_frame, textvariable=self._port_var, width=10, + bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO, + ).pack(side=tk.LEFT, padx=2) + tk.Label(self._serial_frame, text="Baud:", bg=BG2, fg=FG, font=MONO + ).pack(side=tk.LEFT, padx=(10, 4)) + self._baud_var = tk.StringVar(value="38400") + tk.Entry(self._serial_frame, textvariable=self._baud_var, width=8, + bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO, + ).pack(side=tk.LEFT, padx=2) + + # Timeout + tk.Label(cfg, text="Timeout:", bg=BG2, fg=FG, font=MONO + ).grid(row=0, column=3, padx=(18, 4)) + self._timeout_var = tk.StringVar(value="60") + tk.Entry(cfg, textvariable=self._timeout_var, width=5, + bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO, + ).grid(row=0, column=4, padx=2) + tk.Label(cfg, text="s", bg=BG2, fg=FG_DIM, font=MONO + ).grid(row=0, column=5) + + # Row 1 — output dir + label + tk.Label(cfg, text="Save to:", bg=BG2, fg=FG, font=MONO + ).grid(row=1, column=0, columnspan=2, sticky="e", padx=4, pady=4) + self._dir_var = tk.StringVar( + value=str(SCRIPT_DIR / "bridges" / "captures")) + tk.Entry(cfg, textvariable=self._dir_var, width=46, + bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO, + ).grid(row=1, column=2, columnspan=3, sticky="we", padx=4, pady=4) + tk.Button(cfg, text="Browse", bg=BG3, fg=FG, relief="flat", + cursor="hand2", font=MONO, command=self._choose_dir + ).grid(row=1, column=5, padx=4, pady=4) + + tk.Label(cfg, text="Label:", bg=BG2, fg=FG, font=MONO + ).grid(row=2, column=0, columnspan=2, sticky="e", padx=4, pady=4) + self._label_var = tk.StringVar(value="") + tk.Entry(cfg, textvariable=self._label_var, width=46, + bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO, + ).grid(row=2, column=2, columnspan=3, sticky="we", padx=4, pady=4) + tk.Label(cfg, text="(optional)", bg=BG2, fg=FG_DIM, font=MONO_SM + ).grid(row=2, column=5, sticky="w", padx=4) + + # Row 2 — full waveform toggle + opts = tk.Frame(self, bg=BG2) + opts.pack(side=tk.TOP, fill=tk.X, padx=6, pady=(0, 4)) + self._full_wf_var = tk.BooleanVar(value=False) + tk.Checkbutton( + opts, text="Full waveform (download raw ADC samples too)", + variable=self._full_wf_var, + bg=BG2, fg=FG, selectcolor=BG3, activebackground=BG2, font=MONO, + ).pack(side=tk.LEFT, padx=4) + + # Command button row + cmd_row = tk.Frame(self, bg=BG2) + cmd_row.pack(side=tk.TOP, fill=tk.X, padx=6, pady=(0, 4)) + + for label, cmd in [ + ("Connect Only", "connect"), + ("List Event Keys", "list_keys"), + ("Download Events", "download_events"), + ]: + btn = tk.Button( + cmd_row, text=label, bg=ACCENT, fg="#ffffff", + relief="flat", padx=10, cursor="hand2", font=MONO, + command=lambda c=cmd: self._run_command(c), + ) + btn.pack(side=tk.LEFT, padx=4) + self._cmd_btns.append(btn) + + self._open_btn = tk.Button( + cmd_row, text="Open in Analyzer", bg=BG3, fg=FG_DIM, + relief="flat", padx=10, cursor="hand2", font=MONO, + command=self._open_in_analyzer, state="disabled", + ) + self._open_btn.pack(side=tk.LEFT, padx=14) + + self._status_var = tk.StringVar(value="Ready") + tk.Label(cmd_row, textvariable=self._status_var, + bg=BG2, fg=FG_DIM, font=MONO).pack(side=tk.LEFT, padx=14) + + # Console output + self._console = scrolledtext.ScrolledText( + self, height=22, font=MONO_SM, + bg=BG, fg=FG, insertbackground=FG, + relief="flat", state="disabled", + ) + self._console.pack(fill=tk.BOTH, expand=True, padx=6, pady=4) + + self._console.tag_configure(self.TAG_TX, foreground=ACCENT) + self._console.tag_configure(self.TAG_RX_RAW, foreground=COL_S3) + self._console.tag_configure(self.TAG_PARSED, foreground=GREEN) + self._console.tag_configure(self.TAG_ERROR, foreground=RED) + self._console.tag_configure(self.TAG_STATUS, foreground=FG_DIM) + self._console.tag_configure(self.TAG_HEAD, foreground=YELLOW, font=MONO_B) + + # Bottom bar + bot = tk.Frame(self, bg=BG2) + bot.pack(side=tk.BOTTOM, fill=tk.X, padx=6, pady=4) + tk.Button(bot, text="Clear", bg=BG3, fg=FG, relief="flat", + padx=10, cursor="hand2", font=MONO, + command=self._clear_console).pack(side=tk.LEFT, padx=4) + + # ── transport toggle ────────────────────────────────────────────────── + + def _on_transport_change(self) -> None: + if self._transport_var.get() == "tcp": + self._serial_frame.grid_remove() + self._tcp_frame.grid(row=0, column=2, sticky="w") + else: + self._tcp_frame.grid_remove() + self._serial_frame.grid(row=0, column=2, sticky="w") + + def _choose_dir(self) -> None: + d = filedialog.askdirectory(initialdir=self._dir_var.get()) + if d: + self._dir_var.set(d) + + # ── console helpers ─────────────────────────────────────────────────── + + def _append(self, text: str, tag: str = "status") -> None: + self._console.configure(state="normal") + self._console.insert(tk.END, text, tag) + line_count = int(self._console.index("end-1c").split(".")[0]) + if line_count > self.MAX_LINES: + self._console.delete("1.0", f"{line_count - self.MAX_LINES}.0") + self._console.see(tk.END) + self._console.configure(state="disabled") + + def _clear_console(self) -> None: + self._console.configure(state="normal") + self._console.delete("1.0", tk.END) + self._console.configure(state="disabled") + + # ── command dispatch ────────────────────────────────────────────────── + + def _set_buttons_state(self, state: str) -> None: + for btn in self._cmd_btns: + btn.configure(state=state) + + def _run_command(self, cmd: str) -> None: + if self._running: + return + try: + tcp_port = int(self._tcp_port_var.get().strip() or "9034") + baud = int(self._baud_var.get().strip() or "38400") + timeout = float(self._timeout_var.get().strip() or "60") + except ValueError: + messagebox.showerror("Error", "Invalid numeric field.") + return + + cfg = { + "transport": self._transport_var.get(), + "host": self._host_var.get().strip(), + "tcp_port": tcp_port, + "port": self._port_var.get().strip(), + "baud": baud, + "timeout": timeout, + "cmd": cmd, + "out_dir": self._dir_var.get().strip() or ".", + "label": self._label_var.get().strip(), + "full_waveform": bool(self._full_wf_var.get()), + } + self._running = True + self._set_buttons_state("disabled") + self._status_var.set("Running…") + threading.Thread(target=self._worker, args=(cfg,), daemon=True).start() + + # ── worker thread ───────────────────────────────────────────────────── + + def _worker(self, cfg: dict) -> None: + q = self._q + + def post(kind: str, text: str) -> None: + q.put((kind, text)) + + try: + from minimateplus.transport import ( # noqa: WPS433 + CapturingTransport, + SerialTransport, + TcpTransport, + ) + from minimateplus.client import MiniMateClient # noqa: WPS433 + except ImportError as exc: + post("error", f"Import error: {exc}\n") + q.put(("done", None)) + return + + # Build session directory with timestamp + optional label + ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + sess_name = f"seismo_dl_{ts}" + if cfg["label"]: + safe_label = "".join( + ch if ch.isalnum() or ch in ("-", "_") else "_" + for ch in cfg["label"] + ) + sess_name = f"{sess_name}_{safe_label}" + + try: + session_dir = Path(cfg["out_dir"]) / sess_name + session_dir.mkdir(parents=True, exist_ok=True) + except OSError as exc: + post("error", f"Cannot create session dir: {exc}\n") + q.put(("done", None)) + return + + bw_path = str(session_dir / f"raw_bw_{ts}.bin") + s3_path = str(session_dir / f"raw_s3_{ts}.bin") + + # Build inner transport + if cfg["transport"] == "tcp": + host = cfg["host"] + tcp_port = cfg["tcp_port"] + post("status", f"Connecting {host}:{tcp_port} (TCP)…") + inner = TcpTransport(host, tcp_port, connect_timeout=cfg["timeout"]) + else: + port = cfg["port"] + baud = cfg["baud"] + post("status", f"Opening {port} @ {baud} baud…") + inner = SerialTransport(port, baud) + + transport = CapturingTransport(inner, bw_path, s3_path) + post("head", f"\n── Session {sess_name} ─────────────────────────────\n") + post("status", f"BW capture: {bw_path}") + post("status", f"S3 capture: {s3_path}") + + client = MiniMateClient(transport=transport, timeout=cfg["timeout"]) + + success = False + try: + with client: + post("head", "\n── connect() — POLL + serial + config + index ──\n") + info = client.connect() + post("parsed", f" serial: {info.serial!r}\n") + if getattr(info, "firmware", None): + post("parsed", f" firmware: {info.firmware!r}\n") + if getattr(info, "model", None): + post("parsed", f" model: {info.model!r}\n") + + if cfg["cmd"] == "connect": + success = True + + elif cfg["cmd"] == "list_keys": + post("head", "\n── list_event_keys() — browse 1E/0A/1F walk ──\n") + keys = client.list_event_keys() + if not keys: + post("parsed", " (no events stored)\n") + else: + post("parsed", f" {len(keys)} event(s):\n") + for i, k in enumerate(keys): + post("parsed", f" [{i}] {k}\n") + success = True + + elif cfg["cmd"] == "download_events": + post("head", "\n── get_events() — full download ──────────────\n") + if cfg["full_waveform"]: + post("status", "Full-waveform mode (raw ADC samples).") + events = client.get_events(full_waveform=cfg["full_waveform"]) + post("parsed", f" downloaded {len(events)} event(s)\n") + for ev in events: + ts_str = ( + ev.event_time.isoformat(sep=" ", timespec="seconds") + if getattr(ev, "event_time", None) else "?" + ) + ppv = getattr(ev, "peaks", None) + ppv_str = "" + if ppv is not None: + try: + ppv_str = f" PPV={ppv.peak_vector_sum:.4f} in/s" + except Exception: + pass + key = ( + ev._waveform_key.hex() + if getattr(ev, "_waveform_key", None) else "?" + ) + post("parsed", + f" [{ev.index:2d}] key={key} {ts_str}{ppv_str}\n") + success = True + + else: + post("error", f"Unknown command: {cfg['cmd']}\n") + + post("status", "Done.") + + except Exception as exc: + post("error", f"\nError: {exc}\n") + finally: + # Capture files are flushed on transport.disconnect() (via __exit__). + try: + bw_size = Path(bw_path).stat().st_size + s3_size = Path(s3_path).stat().st_size + post("status", + f"Capture closed. BW={bw_size}B S3={s3_size}B") + post("head", f"\n── Capture saved → {session_dir} ─────────\n") + if success: + q.put(("ready", (bw_path, s3_path, sess_name))) + except OSError: + pass + q.put(("done", None)) + + # ── queue poll ──────────────────────────────────────────────────────── + + def _poll_q(self) -> None: + try: + while True: + kind, payload = self._q.get_nowait() + + if kind == "tx": + self._append(payload, self.TAG_TX) + elif kind == "rx_raw": + self._append(payload, self.TAG_RX_RAW) + elif kind == "parsed": + self._append(payload, self.TAG_PARSED) + elif kind == "error": + self._append(payload, self.TAG_ERROR) + elif kind == "head": + self._append(payload, self.TAG_HEAD) + elif kind == "status": + self._status_var.set(str(payload)) + self._append(f" [{payload}]\n", self.TAG_STATUS) + elif kind == "ready": + bw_path, s3_path, label = payload + self._last_paths = (bw_path, s3_path, label) + self._open_btn.configure(state="normal", fg=FG) + elif kind == "done": + self._running = False + self._set_buttons_state("normal") + self._status_var.set("Ready") + + except queue.Empty: + pass + finally: + self.after(100, self._poll_q) + + # ── analyzer hand-off ───────────────────────────────────────────────── + + def _open_in_analyzer(self) -> None: + if not self._last_paths or not self._on_capture_ready: + return + bw_path, s3_path, label = self._last_paths + self._on_capture_ready(bw_path, s3_path, label) + + # ───────────────────────────────────────────────────────────────────────────── # Main application window # ───────────────────────────────────────────────────────────────────────────── @@ -2046,6 +2724,12 @@ class SeismoLab(tk.Tk): ) nb.add(self._serial_watch_panel, text=" Serial Watch ") + self._download_panel = DownloadPanel( + nb, + on_capture_ready=self._on_download_capture_ready, + ) + nb.add(self._download_panel, text=" Download ") + self._nb = nb self.protocol("WM_DELETE_WINDOW", self._on_close) @@ -2080,6 +2764,14 @@ class SeismoLab(tk.Tk): self._analyzer_panel.s3_var.set(raw_s3_path) self._nb.select(1) + def _on_download_capture_ready(self, bw_path: str, s3_path: str, label: str) -> None: + """Download capture done → load both BW + S3 files into Analyzer and run.""" + self._analyzer_panel.stop_live() + self._analyzer_panel.s3_var.set(s3_path) + self._analyzer_panel.bw_var.set(bw_path) + self._analyzer_panel._run_analyze() + self._nb.select(1) + def _on_close(self) -> None: self._bridge_panel.stop_bridge() self._serial_watch_panel._stop() diff --git a/sfm/server.py b/sfm/server.py index 407c680..3fc4bb2 100644 --- a/sfm/server.py +++ b/sfm/server.py @@ -61,6 +61,7 @@ from minimateplus import MiniMateClient from minimateplus.protocol import ProtocolError from minimateplus.models import CallHomeConfig, ComplianceConfig, DeviceInfo, Event, PeakValues, ProjectInfo, Timestamp from minimateplus.transport import TcpTransport, DEFAULT_TCP_PORT +from minimateplus.blastware_file import write_blastware_file, blastware_filename from sfm.cache import SFMCache, get_cache from sfm.database import SeismoDb @@ -848,6 +849,112 @@ def device_event_waveform( return result +@app.get("/device/event/{index}/blastware_file") +def device_event_blastware_file( + index: int, + port: Optional[str] = Query(None, description="Serial port (e.g. COM5)"), + baud: int = Query(38400, description="Serial baud rate"), + host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"), + tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"), +) -> FileResponse: + """ + Download the waveform for a single event (0-based index) and return it + as a Blastware-compatible binary file with a correct Blastware filename. + + Supply either *port* (serial) or *host* (TCP/modem). + + The file is written to /tmp and streamed back as a binary download. + Blastware can open it directly — filename encodes serial + timestamp. + + Filename format: 0 + - prefix letter = chr(ord('B') + floor(serial_numeric / 1000)) + - stem + AB = second-resolution timestamp since 1985-01-01 local + - W / H = Full Waveform / Full Histogram (defaults to W for + triggered events; histogram requires recording_mode + to be populated from compliance config) + + Performs: POLL startup → get_events(full_waveform=False, extra_chunks=1, + stop_after_index=index) → write_blastware_file() → FileResponse. + """ + log.info( + "GET /device/event/%d/blastware_file port=%s host=%s", + index, port, host, + ) + + try: + def _do(): + with _build_client(port, baud, host, tcp_port, timeout=120.0) as client: + info = client.connect() + # Use stop_after_metadata=True (full_waveform=False) with 1 extra + # chunk after "Project:". The extra chunk is required to prime the + # device over TCP: termination at term_counter=metadata_counter+0x0400 + # returns only ~90 bytes (no useful footer) over TCP/cellular, but + # termination at metadata_counter+0x0800 (one chunk later) returns + # the full 737-byte frame containing the footer. + # + # Confirmed from 4-26-26 BW RS-232 capture: BW terminates at 0x1800 + # without an extra chunk (works on RS-232 but not TCP). + # write_blastware_file() automatically skips the extra chunk's + # contribution — only the probe+ADC+metadata+terminator bytes appear + # in the output file. + # + # full_waveform=True (natural end-of-stream) downloads ALL chunks + # including post-event silence (35+ chunks for a 9-sec event at + # 1024 sps) — this produces 24KB+ files that Blastware rejects. + events = client.get_events( + full_waveform=False, + stop_after_index=index, + extra_chunks_after_metadata=1, + ) + matching = [ev for ev in events if ev.index == index] + return matching[0] if matching else None, info + ev, info = _run_with_retry(_do, is_tcp=_is_tcp(host)) + except HTTPException: + raise + except ProtocolError as exc: + log.error("blastware_file: protocol error: %s", exc, exc_info=True) + raise HTTPException(status_code=502, detail=f"Protocol error: {exc}") from exc + except OSError as exc: + log.error("blastware_file: connection error: %s", exc, exc_info=True) + raise HTTPException(status_code=502, detail=f"Connection error: {exc}") from exc + except Exception as exc: + log.error("blastware_file: unexpected error: %s", exc, exc_info=True) + raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc + + if ev is None: + raise HTTPException( + status_code=404, + detail=f"Event index {index} not found on device", + ) + + a5_frames = getattr(ev, "_a5_frames", None) + if not a5_frames: + raise HTTPException( + status_code=502, + detail=f"No waveform data received for event index {index} — 5A download failed", + ) + + # Determine serial number from device info + serial = getattr(info, "serial", None) or "UNKNOWN" + + # Build filename using the same algorithm Blastware uses + filename = blastware_filename(ev, serial) + + # Write to /tmp so FastAPI can stream it back + out_path = Path("/tmp") / filename + write_blastware_file(ev, a5_frames, out_path) + log.info( + "blastware_file: wrote %s (%d A5 frames, serial=%s)", + out_path, len(a5_frames), serial, + ) + + return FileResponse( + path=str(out_path), + filename=filename, + media_type="application/octet-stream", + ) + + # ── Write endpoints ─────────────────────────────────────────────────────────── class DeviceConfigBody(BaseModel):