22 Commits

Author SHA1 Message Date
claude 6b76934a04 Merge branch 'main' into protocol-fix 2026-05-04 14:43:05 -04:00
claude 7b62c790a9 fix(seismo-lab): remove duplicate capture history list 2026-05-04 14:30:46 -04:00
claude b66cc9d075 fix(blastware_file): update TERM detection logic and strip duplicate header blocks for accurate file writing 2026-05-04 14:28:11 -04:00
serversdown 4ab604eff1 Merge pull request 'v0.12.6' (#10) from seismo-lab-new into main
Reviewed-on: #10
2026-05-04 13:22:54 -04:00
serversdown e15f1567ef Doc: Update docs for 0.12.6 2026-05-04 17:18:28 +00:00
serversdown bb33ad3837 doc: update to v0.12.5 2026-05-04 17:13:37 +00:00
claude 45e61fbcaf big refactor of waveform protocol. 2026-05-03 01:20:21 -04:00
claude d758825c67 fix(protocol): correct continuous-mode record header classification for accurate timestamp extraction 2026-05-01 20:28:55 -04:00
claude 0fbb39c21a Big event bugfix. see details:
## v0.13.0 — 2026-05-01

### Fixed

- **SUB 5A bulk waveform stream — over-read bug for events ≥ 2 sec.**
  `read_bulk_waveform_stream` was walking the chunk counter past the actual
  end of the event, picking up post-event circular-buffer garbage that
  corrupted reconstructed Blastware files for any waveform > ~1 sec.  The
  loop now extracts the event's `end_offset` from the STRT record at
  `data[23:27]` of the probe response and stops the chunk walk when the next
  counter would step past it.  Verified against three BW MITM captures
  (4-27-26 + 5-1-26): 2-sec event drops from 37 over-read chunks to 7
  bounded chunks; 3-sec drops to 9; non-zero-start "event 2" drops to 9.

### Added

- `framing.bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)` —
  computes the corrected SUB 5A TERM frame's `(offset_word, params)` per the
  formula confirmed across all 3 BW captures.  Not yet wired into
  `read_bulk_waveform_stream` (the legacy TERM is still used to preserve the
  existing `blastware_file.write_blastware_file` frame-structure expectations);
  available for the next iteration that switches to BW's 0x0200 chunk step.
- `framing.parse_strt_end_offset(a5_data)` — extracts the event-end pointer
  from the STRT record in an A5 response payload.
2026-05-01 18:37:34 -04:00
claude 1ef55521b1 Fix: Removed duplicates from merge botch. Stable version of seismo_lab.py 2026-05-01 17:34:41 -04:00
claude 738b39f3cb Manually Merged seismo lab persistent connection branch into the new direct download branch, creating a new branch called seismo-lab-new 2026-05-01 15:13:50 -04:00
Claude 625b0a4dfc feat(seismo_lab): add Download tab that captures wire bytes during event download
Adds a new CapturingTransport wrapper in minimateplus.transport that mirrors
every TX/RX byte to two raw .bin files using the same on-wire format as
bridges/ach_mitm.py, so the resulting captures are byte-for-byte compatible
with the existing Blastware MITM captures and load directly in the Analyzer.

A new "Download" tab in seismo_lab.py lets the user connect to a device over
TCP or serial and run connect / list-keys / download-events while the wrapper
saves raw_bw_<ts>.bin (our TX) and raw_s3_<ts>.bin (device TX) into a
seismo_dl_<ts>[_<label>]/ session directory. On completion, the panel hands
both files to the Analyzer and switches tabs, mirroring the UX of the
existing Bridge capture flow.
2026-05-01 00:12:02 +00:00
Claude b14f31f3b0 Include capture label in TCP raw filename
Matches serial bridge naming: raw_bw_{ts}_{label}.bin / raw_s3_{ts}_{label}.bin

https://claude.ai/code/session_014NczSHUz9uTzCAf4cVASTJ
2026-04-27 20:48:10 +00:00
Claude b9ab368934 Fix TCP capture: write files only when capture is active
Previously every Blastware connection auto-created files.
Now TCP mode works the same as serial mode:
- Start Bridge: proxy listens and forwards silently, no files written
- New Capture: opens raw_bw/raw_s3 files; pipe threads write to them
- Stop Capture: flushes and closes files, fires Analyzer callback
- No connection = no file; multiple captures per bridge session work correctly

https://claude.ai/code/session_014NczSHUz9uTzCAf4cVASTJ
2026-04-27 20:26:31 +00:00
Claude 9004241846 Restore multi-capture Bridge design + TCP mode
Brings back the protocol-exp BridgePanel design:
- Single bridge session stays up; New Capture / Stop Capture create
  labelled raw-file segments on demand (no files created at bridge start)
- Capture history listbox shows all segments; double-click reloads in Analyzer
- On capture complete: Analyzer auto-populates and runs analysis

TCP mode integrated into same tab (Serial/TCP radio toggle):
- Each incoming Blastware connection is automatically a capture segment
- Session appears in history list; Analyzer wires up live on connect
- Stop Capture disconnects current TCP session

https://claude.ai/code/session_014NczSHUz9uTzCAf4cVASTJ
2026-04-27 20:20:43 +00:00
Claude 6861d9ed97 Merge TCP mode into Bridge tab (Serial/TCP radio toggle)
Removes the separate 'TCP Capture' tab and folds TCP MITM capture directly
into the existing Bridge tab.  A Serial/TCP radio selector at the top swaps
the connection fields (COM ports vs. listen port + device host:port) while
keeping the same Start Bridge / Stop Bridge / Add Mark buttons, capture
checkboxes, log dir, and live log — identical UX for both modes.

https://claude.ai/code/session_014NczSHUz9uTzCAf4cVASTJ
2026-04-26 23:01:45 +00:00
claude 5cd5652560 Merge branch 'seismo-lab' of https://github.com/serversdwn/seismo-relay into seismo-lab 2026-04-26 18:16:52 -04:00
Claude 897ac8a3f3 Add TCP MITM capture tab (TcpBridgePanel)
New 'TCP Capture' tab in seismo_lab.py: listens on a configurable local
port for an incoming Blastware connection, transparently forwards all
traffic to the real seismograph device, and saves both directions to
raw_bw_<ts>.bin / raw_s3_<ts>.bin in the same format the Analyzer already
understands.  Session start wires up Analyzer live mode automatically via
the same on_bridge_started callback as the COM-port bridge.

https://claude.ai/code/session_014NczSHUz9uTzCAf4cVASTJ
2026-04-26 22:10:48 +00:00
serversdown 310fc5986c Merge pull request 'seismo-lab2' (#7) from seismo-lab2 into seismo-lab
Reviewed-on: #7
2026-04-26 16:49:28 -04:00
Claude e1150b30aa fix(analyzer): name A5/5A frames; revert S3 checksum validation
Add 0x5A (BULK_WAVEFORM_STREAM) and 0xA5 (BULK_WAVEFORM_RESPONSE) to
SUB_TABLE so they display with real names instead of UNKNOWN_5A/A5.

Revert S3 checksum validation to checksum_valid=None (the original
intentional behavior). Large S3 frames (A5 bulk waveform, E5 compliance
config) embed inner DLE+ETX sub-frame delimiters; the trailing 0x03 of
the last inner delimiter can land where the parser expects the SUM8
checksum byte, causing false BAD CHK on every valid A5 frame.
protocol.py _validate_frame documents and ignores exactly this issue.

https://claude.ai/code/session_014NczSHUz9uTzCAf4cVASTJ
2026-04-26 20:40:45 +00:00
Claude 9bbecea70f fix(parser): correct S3 frame terminator — bare ETX, not DLE+ETX
parse_s3 had the S3 terminator logic inverted vs the real S3FrameParser
in framing.py. It was terminating on DLE+ETX and treating bare ETX as
payload, which caused every bare 0x03 to be swallowed — bundling multiple
real S3 frames into one giant body until a DLE+ETX sequence happened to
appear. Result: 583-byte POLL_RESPONSE 'frames' containing many real
frames concatenated, all showing BAD CHK.

Fix: mirror S3FrameParser exactly —
  - Bare ETX (0x03) = real frame terminator
  - DLE+ETX (0x10 0x03) = inner-frame literal data (A4/E5 sub-frames),
    appended to body and parsing continues

https://claude.ai/code/session_014NczSHUz9uTzCAf4cVASTJ
2026-04-26 20:23:18 +00:00
serversdown 4a0c9b6da5 Merge pull request 'merge protocol-exp 0.12.3 to main' (#5) from protocol-exp into main
Reviewed-on: #5
2026-04-21 00:22:24 -04:00
15 changed files with 2209 additions and 626 deletions
+136
View File
@@ -4,8 +4,140 @@ All notable changes to seismo-relay are documented here.
---
## v0.12.6 — 2026-05-01
### Fixed
- **`blastware_file.py` — waveform frame classification** — A5 frame classification for
waveform-only vs header-only frames now uses `frame.record_type` instead of frame index.
Only waveform frames (0x46) are written to the file body; metadata frames are skipped.
Fixes spurious data corruption from incorrectly classified frames.
- **`s3_analyzer.py` — A5/5A frame naming** — Bulk waveform stream frames (SUB 5A response)
are now correctly labeled "A5" in analyzer output instead of being conflated with other
multi-frame responses (SUB A4, E5, etc.).
- **`S3FrameParser` — frame terminator detection** — Corrected the bare ETX terminator
detection. Frame termination is now correctly identified by a standalone `ETX=0x03` byte,
not by the `DLE+ETX` sequence (which is part of the payload when it appears within a frame).
---
## v0.13.2 — 2026-05-01
### Fixed
- **`_extract_record_type` — third 0C-record header format ("short", 8 bytes).**
A live SFM download against BE11529 produced files named `M5290000.000`
(zero-stamped) because the 0C waveform record's first bytes were
`01 05 07 ea ...` — neither the 9-byte single-shot layout (`0x10` at byte 1)
nor the 10-byte continuous layout (`0x10` at bytes 0 and 2). Investigation
showed this is a third format observed in the wild: an 8-byte header with no
marker bytes at all (`[day][month][year_BE:2][unknown][hour][min][sec]`).
The detection logic now scans the year (uint16 BE) at byte 2 / byte 3 / byte
4 and picks whichever offset returns a sensible year (20152050) — each
format has the year at a unique position so this disambiguates cleanly.
- New format → `event.record_type = "Waveform (Short)"`,
`Timestamp.from_short_record()`.
- Existing single-shot and continuous parsers unchanged.
- The user's event from May 1, 2026 13:21:37 now correctly resolves to a
filename like `M529LKIQ.G10` instead of `M5290000.000`.
### Added
- `Timestamp.from_short_record(data)` — decodes the 8-byte header.
- `_detect_record_format(data)` — internal helper returning
`"single_shot" / "continuous" / "short" / None` via year-position scan.
---
## v0.13.1 — 2026-05-01
### Fixed
- **`_extract_record_type` — Continuous-mode record headers misclassified as Unknown.**
In single-shot mode the 0C waveform record's 9-byte header puts the sub_code
marker `0x10` at byte 1, with the day at byte 0. In Continuous mode the
header is 10 bytes with the marker at byte 0 *and* byte 2, and the day at
byte 1. Previous logic only inspected byte 1 and treated any value other
than `0x10` / `0x03` as `"Unknown"`, which prevented `event.timestamp` from
being populated for any continuous-mode event whose day-of-month wasn't
exactly 3 or 16. As a downstream effect, `blastware_filename()` saw
`event.timestamp == None`, fell back to `stem="0000"` / `ab="00"`, and
produced filenames like `M5290000.000`. Discovered from a live SFM run on
BE11529 in continuous mode (day-of-month = 5).
Now disambiguates by checking BOTH byte 0 and byte 2: if both are `0x10`,
it's the 10-byte continuous header; else if byte 1 is `0x10`, it's the
9-byte single-shot header. Day-of-month no longer matters.
*Superseded by v0.13.2 — the user's actual record uses a third 8-byte format
with no `0x10` markers, which v0.13.1 still misclassified.*
---
## v0.13.0 — 2026-05-01
### Fixed
- **SUB 5A bulk waveform stream — over-read bug for events ≥ 2 sec.**
`read_bulk_waveform_stream` was walking the chunk counter past the actual
end of the event, picking up post-event circular-buffer garbage that
corrupted reconstructed Blastware files for any waveform > ~1 sec. The
loop now extracts the event's `end_offset` from the STRT record at
`data[23:27]` of the probe response and stops the chunk walk when the next
counter would step past it. Verified against three BW MITM captures
(4-27-26 + 5-1-26): 2-sec event drops from 37 over-read chunks to 7
bounded chunks; 3-sec drops to 9; non-zero-start "event 2" drops to 9.
### Added
- `framing.bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`
computes the corrected SUB 5A TERM frame's `(offset_word, params)` per the
formula confirmed across all 3 BW captures. Not yet wired into
`read_bulk_waveform_stream` (the legacy TERM is still used to preserve the
existing `blastware_file.write_blastware_file` frame-structure expectations);
available for the next iteration that switches to BW's 0x0200 chunk step.
- `framing.parse_strt_end_offset(a5_data)` — extracts the event-end pointer
from the STRT record in an A5 response payload.
### Documentation
- **CLAUDE.md and `docs/instantel_protocol_reference.md` extensively
rewritten** to reflect the corrected SUB 5A protocol. See:
- CLAUDE.md "SUB 5A — chunk counter formula (REWRITTEN 2026-05-01)"
- CLAUDE.md "SUB 5A — STRT record encodes end_offset"
- CLAUDE.md "SUB 5A — TERM frame formula"
- CLAUDE.md "SUB 5A — fixed metadata pages 0x1002 and 0x1004"
- CLAUDE.md "SUB 0A — WAVEHDR response length distinguishes events from
boundaries" (0x46 = real event, 0x2C = boundary marker)
- protocol reference §7.8.5 / §7.8.6 / §7.8.7 / §7.8.8
- The previous chunk-counter formula (`max(key4[2:4], 0x0400) + (chunk-1) *
0x0400`) is now marked DEPRECATED and explicitly tagged WRONG with
pointers to the new sections, so future work doesn't re-derive it.
### Known minor diffs vs Blastware (deferred to a follow-up)
- We still use the OLD 0x0400 chunk step rather than BW's 0x0200; switching
also requires updating `blastware_file.write_blastware_file`'s skip values
and "extra chunk after metadata" logic, which depends on a fresh capture
to verify.
- We still use the legacy fixed `offset_word=0x005A` TERM frame rather than
BW's `end_offset - next_boundary` formula, for the same reason.
- Two fixed metadata pages at counter `0x1002` and `0x1004` are not yet
read explicitly; under the current 0x0400 walk their content is reachable
via the sample chunk that covers buffer addresses `[0x1000, 0x1400)`.
---
## v0.12.5 — 2026-04-21
### Added
- **`seismo_lab.py` — Download tab** — New fourth tab for live wire-byte capture during event
downloads. Captures both BW→device and device→S3 frames in real time, allowing inspection
of the 5A bulk stream chunk sequence and frame-by-frame analysis without needing a bridge
or MITM proxy. Files are saved with user-specified labels for easy tracking.
### Changed
- **`s3_bridge.py` — raw captures always-on by default** — `--raw-bw` and `--raw-s3` now
@@ -17,6 +149,10 @@ All notable changes to seismo-relay are documented here.
"S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names
the files). Unchecking a box passes `--raw-bw ""` to explicitly disable capture.
- **`Bridge tab` — TCP mode added** — Serial/TCP radio toggle allows connection via cellular
modem (RV50/RV55) instead of direct RS-232. Supports multi-capture design (simultaneous
Bridge + Analyzer + Download sessions).
- **`ach_server.py` — TX capture added (`raw_tx_<ts>.bin`)** — Every ACH inbound session
now saves both directions: `raw_rx_<ts>.bin` (device → us, S3 side, as before) and
`raw_tx_<ts>.bin` (us → device, BW side). Both files are usable in the Analyzer.
+229 -64
View File
@@ -2,7 +2,7 @@
Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
(Sierra Wireless RV50 / RV55). Current version: **v0.12.3**.
(Sierra Wireless RV50 / RV55). Current version: **v0.13.2**.
When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
@@ -41,7 +41,7 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c
| Event header / first key | 1E | ✅ |
| Waveform header | 0A | ✅ |
| Waveform record (peaks, timestamp, project) | 0C | ✅ |
| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ new v0.6.0 |
| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ over-read bug fixed v0.13.0 (chunk loop bounded by STRT end_offset); minor wire diffs vs BW deferred — see "SUB 5A — chunk counter formula" |
| Event advance / next key | 1F | ✅ |
| **Write commands (push config to device)** | **6883** | ✅ new v0.8.0 |
| **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 |
@@ -118,29 +118,156 @@ S3→BW (response):
Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26
BW TX capture. All 10 frames verified.
### SUB 5A — chunk counter formula (FINAL CORRECTION 2026-04-26)
### SUB 5A — chunk counter formula (REWRITTEN 2026-05-01 — see 5-1-26 captures)
**Chunk counter = `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` for ALL chunks.**
> ⚠️ **Everything that came before this rewrite was WRONG in important ways.** The previous
> formula `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` happened to *work* for events
> at start_key=0 because the device responds to whatever counter you ask for — but it caused
> a 5× over-read past the actual event, picking up post-event circular-buffer garbage that
> corrupts the reconstructed file for any event > ~1 sec of waveform. The captures in
> `bridges/captures/4-27-26/` and `5-1-26/comcheck/` show BW reads only ~12-16 chunks for
> the same events SFM was reading 37+ chunks for. See "TERM frame" and "STRT end_offset"
> sections below for the actual mechanism.
where `key4[2:4] = (key4[2] << 8) | key4[3]` is the event's circular-buffer base offset.
**Chunk addressing is just absolute device-buffer addresses.**
The `max(..., 0x0400)` guard is critical for events at the start of the circular buffer
(key4[2:4] == 0x0000, e.g. key `01110000`). Without it, chunk 1 gets counter=0x0000, which
is the same address as the probe frame — the device re-returns the STRT record data instead
of waveform payload. With the guard, chunk 1 gets counter=0x0400, which is confirmed correct
from the empirical live-device test 2026-04-06 (`counter=0x0400 → responds immediately and
streams all frames correctly`).
`params[0]=0x00`, `params[1:5]` is a 4-byte absolute device flash-buffer address (= the
"key" of that location), `params[5:11]` are zeros. The device returns 0x0200 (= 512) bytes
starting at that address. Increments between consecutive chunks are **0x0200 (NOT 0x0400)**
— this matches the chunk payload size. The previous "0x0400 step" worked by accident: BW
asks for half-size chunks; SFM was asking for double-size chunks, both with the same-named
"counter" field, but the value is just an address pointer the device honors as-is.
The 4-3-26 capture confirms the pattern for a second event (key `0111245a`, key4[2:4]=0x245a):
chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400).
`max(0x245a, 0x0400) = 0x245a` → formula works correctly for non-zero base offset too.
**The chunk pattern depends on whether the event sits at start_key=0 or not.**
#### Event 1 case — start_key[2:4] == 0x0000 (first event after erase / wrap)
```
1. Probe at counter=0x0000 (params[1:5] = full key, returns STRT record)
2. Read 2 fixed metadata pages: counter=0x1002, counter=0x1004
(these are GLOBAL session metadata — read ONCE per
Blastware session, not per event; contain the
Project/Client/User Name/Seis Loc strings)
3. Sample chunks: counter=0x0600, 0x0800, …, by 0x0200 increment,
up to but not including end_offset (rounded down to
0x0200 boundary)
4. TERM frame (see TERM formula below)
```
The reason `0x0046..0x0600` is skipped for event 1 is unknown — likely some pre-event
firmware reserved area for the first slot in a freshly-erased buffer. Harmless to skip.
#### Event 2+ case — start_key[2:4] != 0x0000 (continuation events)
```
1. First chunk at counter = start_key[2:4] + 0x0046 (this IS the probe — response
contains STRT)
2. Sample chunks: counter += 0x0200 each, up to but
not including end_offset
3. TERM frame
```
No metadata pages — those have already been read during event 1 in the same Blastware
session, and BW caches them. Note that the metadata-page reads happen ONCE per
Blastware-session-on-the-device, not once per event, so an SFM session that downloads
several events should read 0x1002/0x1004 only once at the start.
#### History (do not re-derive)
**History:**
- Original: `_CHUNK1_COUNTER = 0x1004` hardcoded (Blastware capture artifact — WRONG).
- 2026-04-06: Corrected to `chunk_num * 0x0400` (worked for key 01110000 only).
- 2026-04-24: Corrected to `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets,
but accidentally broke key 01110000 — counter=0x0000 sends probe address again).
- 2026-04-26: Final formula: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400`.
- 2026-04-06: `chunk_num * 0x0400` (worked for key 01110000 only).
- 2026-04-24: `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets, broke key 01110000).
- 2026-04-26: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` (broken — over-read past event end).
- 2026-05-01: Increments are 0x0200 not 0x0400; absolute addresses inside event range; bounded
by STRT end_key, not by `max_chunks` cap or device-side timeout.
### SUB 5A — STRT record encodes end_offset (NEW 2026-05-01)
The first A5 response (probe response, or the first chunk for event 2+) contains a STRT
record at byte offset 17 of the `data` field. Layout:
```
data[17:21] "STRT" magic
data[21:23] ff fe sentinel
data[23:27] end_key ← 4-byte key of where this event ENDS
data[27:31] start_key ← 4-byte key of where this event STARTS
data[31:33] uint16 BE ?? sample-count or total bytes (varies; not yet decoded)
data[33:35] uint16 BE ??
data[35] 0x46 record type (waveform full record)
```
`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**.
SFM must extract this from the first A5 response and use it to bound the chunk loop and
encode the TERM frame. The device will happily respond to chunk requests past `end_offset`
(returning post-event circular-buffer contents) — that's the over-read bug.
Verified across 3 events:
| Capture | start_key | end_key | end_offset | event size |
|---|---|---|---|---|
| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B |
| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B |
| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | `0x417E` (event 2 span 0x1F8C = 8,076 B) |
### SUB 5A — TERM frame formula (FINALIZED 2026-05-01)
The TERM frame fetches the partial last chunk *and* the file footer. It is **not** a simple
"goodbye" frame — its response payload contains the bytes between the last full 0x0200-aligned
chunk and `end_offset`, and is required for reconstructing the Blastware file format.
```
last_chunk_counter = address of last full 0x0200-byte chunk read
next_boundary = last_chunk_counter + 0x0200
TERM offset_word = end_offset - next_boundary
TERM params[0] = key[0] (= 0x01 on every observed device)
TERM params[1] = key[1] (= 0x11)
TERM params[2] = (next_boundary >> 8) & 0xFF
TERM params[3] = next_boundary & 0xFF
TERM params[4:10] = zeros
build_5a_frame(offset_word, params) (10-byte params, NOT 11)
```
The device reconstructs `requested_address = (params[2] << 8) | offset_word = end_offset`
and replies with `(end_offset - next_boundary)` bytes from `next_boundary` — the residual
between the last 0x0200 boundary and the actual event end. Append the TERM response data
to the chunk stream like any other A5 frame; it carries the final waveform tail + footer.
Verified across 3 events:
| end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] |
|---|---|---|---|---|
| `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ |
| `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ |
| `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ |
The previous code's hard-coded `offset_word = 0x005A` and `term_counter = last + 0x0400`
are wrong; the device's response under that path is a tiny 101-byte device-side terminator
(arrived only after we walked the entire post-event buffer), not the proper file footer.
### SUB 5A — fixed metadata pages 0x1002 and 0x1004 (NEW 2026-05-01)
Two chunk addresses are GLOBAL device/session metadata, not event-specific:
- `counter=0x1002` — first metadata page
- `counter=0x1004` — second metadata page
These are at fixed absolute addresses in the device's flash buffer. They contain the
session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII
strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the
new walk these strings come from the dedicated metadata pages, not from the sample-chunk
stream.
BW reads them ONCE per Blastware session (during event 1's download) and caches them.
For SFM, that means:
- Once per call-home / once per `MiniMateClient.connect()` is enough.
- Subsequent events in the same session don't need to re-fetch them.
- Their content does not change when iterating events; only when the user opens
Compliance Setup → Apply on the device or sends a SUB 71 compliance write.
The contents have not been byte-for-byte decoded yet — first task on the implementation
side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the
strings we currently extract from A5[7].
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
@@ -148,10 +275,16 @@ chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400).
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
Do not swap them.
### SUB 5A — event-time metadata lives in A5 frame 7
### SUB 5A — event-time metadata source (UPDATED 2026-05-01)
The bulk stream sends 9+ A5 response frames. Frame 7 (0-indexed) contains the compliance
setup as it existed when the event was recorded:
> **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A
> bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at
> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because
> the broken counter formula was scanning the wrong region.
>
> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and
> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A
> sample-chunk stream itself does NOT contain these strings any more under the new walk.
```
"Project:" → project description
@@ -171,26 +304,37 @@ used as the authoritative source. `_decode_a5_metadata_into` therefore only set
"Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C
record — 5A remains the sole source for those fields and they are set unconditionally.
`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears,
then sends the termination frame.
> ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and
> stops one chunk later) is a workaround for the missing end_offset bound — when the new
> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is
> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM
> frame.
### SUB 5A — end-of-stream signal (confirmed 2026-04-06)
### SUB 5A — end-of-stream — UPDATED 2026-05-01
After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to
the next chunk request, then goes silent. This is the natural end-of-stream indicator — NOT
a complete A5 frame. `S3FrameParser.bytes_fed` will be 1; no frame is assembled.
> **Previous understanding (now known to be a symptom, not a feature):** "After streaming
> all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was
> not the device's natural end-of-event signal — it was the device's response when SFM had
> walked clean off the end of the addressable buffer region after over-reading by ~5×.
> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the
> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`,
> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser.
Handling: on `TimeoutError`, if `bytes_fed > 0` AND frames were already collected, treat as
graceful end-of-stream, break the loop, and proceed to the termination frame. If `bytes_fed
== 0` with no prior frames, it is a genuine transport failure — re-raise.
The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful
defence-in-depth fallback for malformed events or unexpected device states, but should not
be the primary loop-exit condition.
**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call
in the chunk loop passes `timeout=10.0` explicitly.
**Typical chunk count (BE11529, 1024 sps):** A 9,306-sample event produces 35 chunks before
end-of-stream. Chunks with uniform 1,036-byte data are all-zero ADC samples (post-event
silence). Only the initial variable-size chunks contain actual signal.
**Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):**
A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames.
A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames.
An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames.
Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks
17-37 containing post-event circular-buffer garbage that corrupted the file body.
### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06)
@@ -303,6 +447,55 @@ sends token=0xFE and is NOT used by any caller.
`advance_event()` returns `(key4, event_data8)`.
Callers (`count_events`, `get_events`) loop while `data8[4:8] != b"\x00\x00\x00\x00"`.
### SUB 0A — WAVEHDR response length distinguishes events from boundaries (NEW 2026-05-01)
When iterating events with the "Download All" pattern (1E → 0A → 1F → 0A → 1F → …), the
DATA_LENGTH at `data_rsp.data[5]` (= the byte BW echoes back as the offset for the data
fetch step) takes one of two values:
| WAVEHDR offset | Meaning |
|---|---|
| `0x46` (= 70) | Real event start key — there is event data at this address |
| `0x2C` (= 44) | Boundary marker between events — this key is the END of the previous event AND the START key for the empty space after it (or is the next event's pre-header) |
Confirmed from the 5-1-26 "Download All" capture:
```
0A(key=01110000) → off=0x46 ← event 1 real start
1F → key=011121F2
0A(key=011121F2) → off=0x2C ← event 1 END / event 2 boundary
1F → key=01112238
0A(key=01112238) → off=0x46 ← event 2 real start (= boundary + 0x46)
1F → key=0111417E
0A(key=0111417E) → off=0x2C ← event 2 END / next-empty marker
1F → null sentinel
```
This is why event 2's first 5A chunk is at `start_key + 0x46` — that's the address of the
"real start" 0x46-record, distinct from the `0x2C`-record at the raw boundary. Use the
`0x46` keys as the input to `read_bulk_waveform_stream`, not the `0x2C` keys.
For event 1 only (start_key[2:4] = 0x0000) BW probes at counter=0x0000 directly, which is
the `0x46`-keyed start record. Subsequent events use `start_key + 0x46`.
**Practical iteration pattern (replaces the old 1E/1F walk for downloads):**
```
Setup: SERIAL × 2 → CHCFG → 1E (token=0x00) → key0
For each event:
0A(cur_key) → DATA_LENGTH = 0x46 (real) or 0x2C (boundary)
1F (token=0x00) → next_key
if length was 0x46: → cur_key is a real event; queue it for download
cur_key = next_key
if next_key all-zero null sentinel: stop
Then for each queued real-event key:
download_event(key) → 5A bulk stream with STRT-bounded chunk walk
```
This is what BW does in the 5-1-26 "Download All" capture — it walks the full event chain
collecting `(key, length)` tuples first, *then* downloads each event using the `0x46` keys.
### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce)
`read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where:
@@ -347,36 +540,6 @@ Do NOT use fixed absolute offsets for sample_rate or record_time.
Quiet Mode enabled. Parser handles this — do not strip it manually before feeding to
`S3FrameParser`.
**SUB 5A (bulk waveform) TCP frame splitting — confirmed 2026-04-27:**
Over TCP via cellular modem, each 5A chunk request that produces a single ~1100-byte
A5 response over direct RS-232 may arrive as **two separate, complete S3 frames** of
~550 bytes each ("2-frame mode"). The modem's Data Forwarding Timeout (~100-150 ms)
can split the RS-232 response into two TCP segments, each parsed as a complete S3 frame.
Under different modem/timing conditions the full ~1100-byte response arrives as **one
S3 frame** ("1-frame mode").
**Both modes require `extra_chunks_after_metadata=1`** (the extra chunk at metadata_counter
+ 0x0400). The device's waveform footer data lives at circular-buffer address 0x1C00 for
this event; the terminator frame must be sent at 0x1C00 (not 0x1800) to receive it.
Example for a 2-second Continuous event (BE11529, key=01110000) via TCP:
- **2-frame mode:** 1 probe frame (554 B) + 5 chunks × 2 frames (556-573 B) + 1 extra chunk × 2 frames + 1 terminator (208 B) = **14 A5 frames** → 6864-byte file
- **1-frame mode:** 1 probe frame (~1097 B) + 5 chunks × 1 frame (~1079-1113 B) + 1 extra chunk × 1 frame (smaller, tail of event) + 1 terminator → **8 A5 frames** → 6864-byte file
- All frames contribute body data; using all of them gives the correct file.
**Fix (confirmed 2026-04-27):** `_recv_5a_batch()` in `protocol.py` collects ALL
A5 frames per chunk request before the next request is sent, using a 0.5 s batch
timeout after the first frame to catch the ~150 ms delayed second frame. `write_blastware_file()`
includes ALL body frames without skipping — the extra chunk's frames are part of the
body data, NOT padding to be discarded.
**WRONG earlier hypothesis (do not re-introduce):** An attempt was made to auto-detect
1-frame vs 2-frame mode from the probe frame size and skip the extra chunk when
`probe_data_len >= 700`. This was wrong — the extra chunk is always needed to advance
the device's internal state to the footer address. The `_probe_is_large` branch was
removed 2026-04-27.
### Required ACEmanager settings (Sierra Wireless RV50/RV55)
| Setting | Value | Why |
@@ -557,6 +720,8 @@ All DB endpoints are read-only except `PATCH /db/events/{id}/false_trigger`.
| 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture |
| 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence; only 1 event stored so token=0xFE appeared to work |
| 4-3-26 | `bridges/captures/4-3-26/` | Browse-mode S3 capture with 2+ events — confirmed all-zero params for 1F, 1F response layout, null sentinel, 0A context requirement |
| 4-27-26 | `bridges/captures/4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof that SFM was over-reading 5× past event end. BW reads 14 chunks at 0x0200 increments + TERM at end_offset; SFM was reading 37 chunks at 0x0400 increments. STRT end_key field located. |
| 5-1-26 | `bridges/captures/5-1-26/comcheck/` | Three sub-captures: SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945`/`_171216`). Confirmed: TERM frame formula across 3 events; metadata pages 0x1002/0x1004 are global (read once per session); event-1 vs event-N chunk-pattern split; WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundaries. |
---
+121 -40
View File
@@ -1,4 +1,4 @@
# seismo-relay `v0.12.1`
# seismo-relay `v0.12.6`
A ground-up replacement for **Blastware** — Instantel's aging Windows-only
software for managing MiniMate Plus seismographs.
@@ -18,26 +18,27 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
```
seismo-relay/
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Console tabs)
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Download + Console tabs)
├── minimateplus/ ← MiniMate Plus client library
│ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport
│ ├── protocol.py ← DLE frame layer, SUB command dispatch
│ ├── client.py ← High-level client (connect, get_events, push_config, …)
│ ├── client.py ← High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …)
│ ├── framing.py ← Frame builders, DLE codec, S3FrameParser
── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, …
── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig,
│ └── blastware_file.py ← Write events to Blastware-compatible .AB0 files
├── sfm/ ← SFM REST API server (FastAPI, port 8200)
│ ├── server.py ← All device + DB endpoints
│ ├── database.py ← SeismoDb — SQLite persistence layer
│ └── sfm_webapp.html ← Embedded web UI (served at /)
│ ├── server.py ← Live device endpoints + DB query endpoints + caching
│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions, sessions table)
│ └── sfm_webapp.html ← Embedded web UI with Call Home config tab
├── bridges/
│ ├── ach_server.py ← Inbound ACH call-home server (main production server)
│ ├── ach_mitm.py ← Transparent MITM proxy for capturing BW sessions
│ ├── s3-bridge/ ← RS-232 serial bridge (capture tool)
│ ├── tcp_serial_bridge.py ← Local TCP↔serial bridge (bench testing)
│ ├── gui_bridge.py ← Standalone bridge GUI
│ ├── gui_bridge.py ← Standalone bridge GUI with raw capture checkboxes
│ └── raw_capture.py ← Simple raw capture tool
├── parsers/
@@ -101,21 +102,28 @@ python seismo_lab.py
Each call dials the device, does its work, and closes the connection. TCP
connections are retried once on `ProtocolError` to handle cold-boot timing.
**Caching** — frequently-polled endpoints are cached in-process to avoid
redundant TCP round-trips:
**In-memory caching** — frequently-polled endpoints avoid redundant TCP round-trips
via a thread-safe `_LiveCache` (plain Python dict + `threading.Lock`):
| Method | URL | Cache |
|--------|-----|-------|
| Method | URL | Cache Strategy |
|--------|-----|---|
| `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` |
| `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected |
| `GET` | `/device/event/{idx}/waveform` | Permanent per event index |
| `GET` | `/device/monitor/status` | 30-second TTL |
| `GET` | `/device/monitor/status` | 30-second TTL; invalidated by monitor start/stop |
| `GET` | `/device/call_home` | Fresh read from device (not cached) |
| `POST` | `/device/connect` | — |
| `POST` | `/device/config` | Writes compliance config; invalidates cache |
| `POST` | `/device/monitor/start` | Sends SUB 0x96 |
| `POST` | `/device/monitor/stop` | Sends SUB 0x97 |
| `POST` | `/device/config` | Writes compliance config; invalidates info + events cache |
| `POST` | `/device/config/project` | Patches project/client/operator/sensor_location strings |
| `POST` | `/device/monitor/start` | Sends SUB 0x96; immediately evicts status cache |
| `POST` | `/device/monitor/stop` | Sends SUB 0x97; immediately evicts status cache |
| `POST` | `/device/call_home` | Reads, patches specified fields, writes back to device |
All cached endpoints accept `?force=true` to bypass the cache.
**Cache bypass** — All cached endpoints accept `?force=true` to skip the cache and
force a fresh read from the device.
**Cache stats** — `GET /cache/stats` returns hit/miss counts and TTL info; `DELETE /cache/device`
clears the device cache immediately.
Transport query params (supply one set):
```
@@ -152,22 +160,34 @@ client = MiniMateClient(transport=TcpTransport("1.2.3.4", 12345), timeout=30.0)
with client:
# Read
info = client.connect() # DeviceInfo — serial, firmware, compliance config
count = client.count_events() # Number of stored events
keys = client.list_event_keys() # Fast browse walk — event keys only, no download
events = client.get_events() # Full download: headers + peaks + metadata
monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag
log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records)
info = client.connect() # DeviceInfo — serial, firmware, compliance config
count = client.count_events() # Number of stored events
keys = client.list_event_keys() # Fast browse walk — event keys only, no download
events = client.get_events() # Full download: headers + peaks + metadata
monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag
log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records)
ach_cfg = client.get_call_home_config() # Auto Call Home settings (SUB 0x2C)
# Write
client.apply_config(
sample_rate=1024,
recording_mode="Continuous", # Single Shot / Continuous / Histogram / Histogram+Continuous
histogram_interval_sec=15, # 2, 5, 15, 60, 300, 900
trigger_level_geo=0.5,
geo_range="Normal", # Normal (10.000 in/s) / Sensitive (1.25 in/s)
project="Bridge Inspection 2026",
client_name="City of Portland",
operator="B. Harrison",
)
client.set_call_home_config(
auto_call_home_enabled=True,
after_event_recorded=True,
at_specified_times=True,
time1_hour=18, time1_min=30, # 6:30 PM
time2_hour=6, time2_min=0, # 6:00 AM
)
# Control
client.start_monitoring() # SUB 0x96
client.stop_monitoring() # SUB 0x97
@@ -182,18 +202,20 @@ existed at record time — not backfilled from the current compliance config.
## Database
`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode).
Three tables, all unit-keyed by serial number:
`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode) using the
`SeismoDb` persistence layer. Four tables, all unit-keyed by serial number:
| Table | Key | Contents |
|-------|-----|----------|
| `ach_sessions` | UUID | Per-call-home audit record: serial, peer IP, events_downloaded, duration |
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, PPV per channel, project/client/operator strings, false_trigger flag |
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: start/stop time, duration, geo threshold |
| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds |
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag |
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
| `events.false_trigger` | Boolean flag | PATCH endpoint to mark/unmark false triggers for review |
Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs
never produce duplicate rows. Post-erase key reuse is handled automatically
via the high-water mark in `ach_state.json`.
Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs never
produce duplicate rows. Post-erase key reuse is handled automatically via the
high-water mark in `ach_state.json`. Key-based state tracking allows correct
handling of device erasures (external or post-download).
---
@@ -231,6 +253,27 @@ Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/insta
---
## Compliance Config Features (v0.12.2v0.12.3)
The REST API and web UI expose full control over device compliance settings:
- **Recording Mode** (Single Shot / Continuous / Histogram / Histogram+Continuous)
- **Sample Rate** (1024 / 2048 / 4096 sps)
- **Record Time** (float, seconds)
- **Histogram Interval** (2s, 5s, 15s, 1m, 5m, 15m) — when recording mode includes histogram
- **Geo Trigger Levels** (float, in/s per channel)
- **Geo Maximum Range** (Normal 10.000 in/s / Sensitive 1.250 in/s per channel)
- **Project / Client / Operator / Sensor Location** (ASCII strings)
Auto Call Home config:
- **Auto Call Home Enable** (bool)
- **Dial String** (read-only; 40-byte ASCII)
- **Trigger on Event** (bool)
- **Scheduled Call-Ins** (two time slots with HH:MM each)
- **Retry Settings** (count, delay, connection timeout, warm-up time)
---
## Requirements
```bash
@@ -252,17 +295,55 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
---
## Roadmap
## Key Features (v0.10v0.12)
**Device support (v0.12.5):**
- [x] Full read/write/erase pipelines
- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings)
- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries)
- [x] Monitor control (start/stop, status polling, battery/memory)
- [x] Monitor log entries (continuous monitoring intervals without full waveform download)
**Data persistence (v0.11):**
- [x] SQLite database (`seismo_relay.db`) with 4 tables: ach_sessions, events, monitor_log, plus false_trigger flag
- [x] Deduplication by waveform key (handles re-runs and repeat call-homes)
- [x] Post-erase key-reuse detection (tracks high-water mark)
- [x] Session state (`ach_state.json`) with downloaded keys and max key
**REST API (v0.12.1):**
- [x] Live device endpoints with in-memory caching (`_LiveCache`)
- [x] Cache statistics (`/cache/stats`) and manual invalidation (`/cache/device`)
- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH)
- [x] Call Home config read/write endpoints
- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`)
**File output (v0.7+):**
- [x] Blastware-compatible `.AB0` file generation (waveform + metadata)
- [x] Multi-channel waveform decode from SUB 5A bulk stream
- [x] Second-resolution timestamp encoding in Blastware filename
**Capture tools (v0.12.5):**
- [x] Serial-to-TCP bridge with raw BW/S3 capture (s3_bridge.py, defaults to auto-capture)
- [x] GUI bridge with raw capture checkboxes (gui_bridge.py)
- [x] ACH inbound server with bidirectional capture (ach_server.py saves raw_tx + raw_rx)
- [x] Transparent TCP MITM proxy for live BW session capture (ach_mitm.py)
**Analysis tools:**
- [x] s3_analyzer.py — session parser, frame differ, Claude export
- [x] gui_analyzer.py — standalone analyzer GUI
- [x] frame_db.py — SQLite frame database for capture analysis
**seismo_lab.py GUI (v0.12.5):**
- [x] Bridge tab — Serial/TCP mode selector with raw capture options
- [x] Analyzer tab — BW/S3 capture playback and differencing
- [x] Download tab — Live wire-byte capture during event download (new v0.12.5)
- [x] Console tab — Logging and diagnostics
## Roadmap (Future)
- [x] Full read pipeline — device info, compliance config, event download with true event-time metadata
- [x] Write commands — push compliance config, trigger thresholds, project strings to device
- [x] Erase all events — confirmed erase sequence from live MITM capture
- [x] Monitor control — start/stop monitoring, read battery/memory/status
- [x] Monitor log entries — decode partial 0x2C records (continuous monitoring intervals)
- [x] ACH inbound server — accept call-home connections, download events, dedup by key
- [x] SQLite persistence — events, monitor log, and session history in `seismo_relay.db`
- [x] SFM REST API — device control + DB query endpoints, live device cache
- [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing
- [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first)
- [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object
- [ ] Modem manager — push RV50/RV55 configs via Sierra Wireless API
- [ ] Histogram mode recording support (5A stream analysis for mode 0x03)
- [ ] Call Home dial_string write support (requires DLE escaping for embedded control characters)
+225 -50
View File
@@ -110,6 +110,7 @@
| 2026-04-21 | §7.6.2, §5.3 | **CORRECTED — compliance_raw contains wire-encoded bytes, NOT logical bytes.** S3FrameParser appends DLE+ETX inner-frame pairs as two literal bytes to the frame body. Any `0x03` values in the compliance config appear in `compliance_raw` as `\x10\x03` (two bytes), not as a single `0x03`. The previous claim "S3FrameParser handles this transparently so compliance_raw contains logical (destuffed) bytes" was wrong. Consequence: `compliance_raw` is the wire-encoded E5 payload; anchor-relative reads work correctly because the anchor position automatically accounts for any DLE-encoded bytes before it. For write-back, round-tripping `compliance_raw` verbatim sends the correct wire bytes to the device. **DLE ETX escaping in write frames:** Blastware escapes `0x03` bytes in write frame data as `\x10\x03` on wire; our `build_bw_write_frame` does not (writes data raw). Device is confirmed to accept raw writes for all tested modes — likely uses the offset/length field for write frame framing, not ETX scanning. |
| 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. |
| 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. |
| 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. |
---
@@ -1226,7 +1227,24 @@ Two critical differences from `build_bw_frame`:
2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen,
only `XX` is added to the running sum; lone bytes are added normally.
#### 7.8.2 Request Sequence
#### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol)
> ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new
> BW MITM captures (4-27-26 + 5-1-26) prove the actual chunk increment is **0x0200**, the
> chunk loop is bounded by `end_offset` from the STRT record (not by chunk count or by a
> device-side timeout), and the TERM frame's `offset_word=0x005A` magic is incorrect — the
> real TERM offset_word is computed from `end_offset` and the last chunk address. Under the
> deprecated formula SFM over-reads roughly 5× past the actual event end into post-event
> circular-buffer garbage, corrupting reconstructed Blastware files for any waveform ≥ 2 sec.
>
> The whole "stop_after_metadata + one extra chunk + 0e 08 footer" workaround in this
> section was compensating for the missing end_offset bound. It is obsoleted by the
> STRT-bounded walk in §7.8.5.
>
> **Read this section for historical context only.** For the correct protocol, jump to:
> - §7.8.5 — chunk addressing and the STRT end_offset
> - §7.8.6 — TERM frame formula
> - §7.8.7 — fixed metadata pages 0x1002 and 0x1004
| Frame | offset_word | counter | params | Purpose |
|---|---|---|---|---|
@@ -1237,45 +1255,17 @@ Two critical differences from `build_bw_frame`:
| … | … | … | … | … |
| Termination | `0x005A` | `max(key4[2:4], 0x0400) + N * 0x0400` | 10 bytes | End transfer |
> ⚠️ **2026-04-06 CORRECTED — chunk counter is `key4[2:4] + (N-1) * 0x0400`.**
> The 4-2-26 BW TX capture showed counter=0x1004 for chunk 1 of key `01110000`, leading to
> an interim "monotonic n * 0x0400" formula. This was accidentally correct because
> `key4[2:4] == 0x0000` for that event.
>
> **2026-04-24 CORRECTION:** The counter is an absolute circular-buffer address.
> BW's true formula is `key4[2:4] + (chunk_num - 1) * 0x0400` where `key4[2:4]` is the
> event's storage base offset (`(key4[2]<<8) | key4[3]`). For keys where
> `key4[2:4] != 0x0000` (e.g. key `01111884`), using `n * 0x0400` sends requests into the
> wrong buffer region — the device returns data from a completely different event.
>
> **2026-04-26 FINAL CORRECTION:** The formula `key4[2:4] + (N-1) * 0x0400` is wrong when
> `key4[2:4] == 0x0000` (e.g. event key `01110000`, the very first event after a device erase).
> Counter=0x0000 for chunk 1 is the same address as the probe frame — the device re-returns
> the STRT record data instead of waveform payload (frame 1 has len=1097, same as probe, and
> contains `b"STRT\xff\xfe"`, contributing zero waveform bytes).
> Final formula: `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400`.
> For key `01110000`: chunk 1 = 0x0400 (confirmed working, empirical test 2026-04-06).
> For key `0111245a`: chunk 1 = 0x245a (unchanged, confirmed from 4-3-26 capture).
> Historical correction notes (left in place to deter re-derivation of the same wrong formula):
> the table above was the result of three iterative "corrections" between 2026-04-06 and
> 2026-04-26 that progressively narrowed in on the wrong answer because every test was on
> events with `key4[2:4]=0` and the device responds to whatever counter you ask for. The
> 5-1-26 captures with a non-zero start_key event (`01112238`) finally exposed the bug.
The `stop_after_metadata=True` flag causes the loop to stop as soon as `b"Project:"` is
found in the accumulated A5 frame data, typically after 49 chunks. A termination frame
is always sent before returning.
**IMPORTANT — one extra chunk required after "Project:" for valid file footer (confirmed 2026-04-23):**
When writing a Blastware-compatible waveform file, stopping immediately at "Project:" and
sending termination produces an empty termination response with no footer bytes (`0e 08`
marker missing). Blastware downloads exactly **one more chunk** after finding "Project:"
before sending termination — that extra chunk primes the device to return valid footer
bytes (monitoring start/stop timestamps) in the termination response.
`read_bulk_waveform_stream(stop_after_metadata=True)` implements this: after the "Project:"
chunk is received, one additional chunk is requested before breaking. The termination
response (`include_terminator=True`) then contains the correct `0e 08` footer.
**do NOT use `full_waveform=True` for Blastware file writing** — for events with long
post-event silence (35 chunks), the silence chunks contain embedded device-internal
pointer structures that produce spurious STRT markers in the file body. Blastware only
downloads 45 chunks (metadata + one signal chunk) regardless of event length.
The `stop_after_metadata=True` flag (deprecated as a primary loop-exit) scanned for
`b"Project:"` in the chunk stream because the metadata strings happened to be reachable
when the broken 0x0400-step walk passed the global metadata pages at 0x1002/0x1004. Under
the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004,
not from the sample-chunk stream — see §7.8.7.
#### 7.8.3 A5 Frame Layout
@@ -1293,15 +1283,19 @@ for ASCII labels with a null-terminated value read:
All five fields reflect the **setup at event-record time**, not the current device config.
#### 7.8.4 End-of-Stream Behaviour and Chunk Timing
#### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01
> **Confirmed 2026-04-06** — empirical observation on BE11529 (S338.17) over TCP/cellular.
> The "1 raw byte then silence" pattern documented below was originally interpreted as
> "the device's natural end-of-event signal." The 5-1-26 captures show this is actually
> the device's response when the requester has walked **past** the addressable buffer
> region (i.e. ~5× past the actual event end under the deprecated 0x0400-step walk).
> Under the corrected STRT-bounded walk (§7.8.5), the stream ends cleanly with the TERM
> frame's response — no timeout, no 1-byte teaser. The fallback below remains useful as
> defensive handling for malformed events but should not be the primary loop-exit.
**End-of-stream signal:** After sending all waveform chunks, the device sends exactly **1 raw byte** in response to the next chunk request, then goes silent. This byte is not a complete DLE-framed A5 response — `S3FrameParser.bytes_fed` reports 1 and no frame is ever assembled. This is the device's natural end-of-stream indicator.
Handling logic in `read_bulk_waveform_stream`:
**Defensive fallback handling in `read_bulk_waveform_stream`:**
```
TimeoutError caught:
TimeoutError caught (rare under corrected walk):
if bytes_fed > 0 AND frames already collected:
→ graceful end-of-stream; break loop; proceed to termination frame
else (bytes_fed == 0, no prior frames):
@@ -1313,14 +1307,15 @@ TimeoutError caught:
| Metric | Observed value |
|---|---|
| Chunk response time | ~1 s per chunk |
| Chunks for a 9,306-sample event | 35 chunks |
| Data per chunk (active signal) | 1,0361,123 bytes |
| Data per chunk (post-event silence) | 1,036 bytes (uniform) |
| Chunks for a 2-sec event (corrected walk) | 14 (12 sample chunks + 2 metadata pages) + TERM |
| Chunks for a 3-sec event (corrected walk) | 18 (16 sample chunks + 2 metadata pages) + TERM |
| Chunks for a continuation event (corrected walk) | ~15 sample chunks + TERM (no metadata reread) |
| Chunks under deprecated walk for 2-3 sec event | 37 (over-reads ~5×) |
| Data per chunk (corrected, 0x0200 size) | ~540575 bytes wire (= 0x0200 payload + framing) |
| Data per chunk (deprecated 0x0400 step) | 1,0361,123 bytes wire (= 0x0400 payload + framing) |
| Safe recv timeout per chunk | **10 s** (10× typical) |
| Default transport timeout | 120 s → ~2-min stall at end-of-stream |
Chunks with uniform 1,036-byte payload (chunks 1735 in the observed event) contain all-zero ADC samples — the device continues recording silence until the configured record time expires before terminating the stream.
**ADC count-to-physical conversion — ✅ CONFIRMED 2026-04-17:**
Raw samples are signed 16-bit integers (32,768 to +32,767). Source: Interface Handbook §4.5.
@@ -1339,6 +1334,186 @@ where `geo_range = 1.61133 V × 6.206053 = 10.000 in/s` is the Normal (Gain=1) f
`_decode_a5_waveform()` contains `elif fi == 9: continue` from an earlier assumption that frame index 9 is always the device terminator. For streams with more than 9 frames, frame 9 is live waveform data. The skip discards ~1,070 bytes (~133 sample-sets) per event. Terminator detection should use `page_key == 0x0000`, not frame index. This skip should be removed.
#### 7.8.5 Chunk addressing and the STRT end_offset (NEW 2026-05-01) ✅
> ✅ Confirmed across 3 events (4-27-26 + 5-1-26 captures).
`params[0]` is always `0x00`. `params[1:5]` is a 4-byte absolute device flash-buffer
address — equivalently, "the key of the page being requested." The device returns 0x0200
(= 512) bytes starting at that address. Increments between consecutive sample chunks are
**0x0200, NOT 0x0400** (the previous 0x0400 figure was a Blastware-side artifact / our
implementation's bug — see §7.8.2).
##### STRT record (data layout in the first A5 response)
The first A5 response (the probe response, or the first chunk for continuation events)
contains a **STRT record** at byte offset 17 of `data`:
```
data[ 0:14] echoes request: [chunk_size_hi=0x02 / 0x04 ...] [00] [01 11] [counter_hi counter_lo] [00 × 8] [00 12]
data[14:17] 10 03 00 ← inner DLE+ETX frame separator (preserved literally)
data[17:21] "STRT" ← magic
data[21:23] ff fe ← sentinel
data[23:27] end_key ← 4-byte key of where this event ENDS
data[27:31] start_key ← 4-byte key of where this event STARTS
data[31:33] uint16 BE ← ?? sample count or byte count, varies (not yet decoded)
data[33:35] uint16 BE ← ??
data[35] 0x46 ← record type marker (waveform full record)
data[36:] additional pointers / first sample bytes — content varies by event
```
`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**.
Use it to bound the chunk loop and to compute the TERM frame.
##### Chunk pattern by event location in buffer
**Event 1 / start_key[2:4] = 0x0000** (first event after erase or wrap):
```
1. Probe at counter = 0x0000 (params[1:5] = full key)
2. Read fixed metadata pages counter = 0x1002, then 0x1004
3. Walk sample chunks counter = 0x0600, 0x0800, …, by 0x0200,
up to but not including end_offset & 0xFE00
4. TERM (see §7.8.6)
```
The range `[0x0046, 0x0600)` is skipped — likely some pre-event firmware-reserved area for
the first slot in a freshly-erased buffer. Harmless to skip; BW does the same.
**Event 2+ / start_key[2:4] != 0x0000** (continuation events in a populated buffer):
```
1. First chunk at counter = start_key[2:4] + 0x0046 ← acts as both probe and first
sample chunk; response carries STRT
2. Walk sample chunks counter += 0x0200 each
3. TERM
```
**No metadata-page reads.** Pages 0x1002/0x1004 are session-global and were already read
during event 1 in the same Blastware session. In SFM, treat metadata pages as a once-
per-`MiniMateClient.connect()` (or once-per-call-home) read, not per-event.
##### Verified end_offset values
| Capture | start_key | end_key | end_offset | event size | sample-chunk start |
|---|---|---|---|---|---|
| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B | 0x0600 (event-1 case) |
| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B | 0x0600 (event-1 case) |
| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | event 2 size = 0x1F8C = 8,076 B | 0x2238 (= 0x21F2 + 0x46) |
#### 7.8.6 TERM Frame Formula (NEW 2026-05-01) ✅
> ✅ Confirmed across 3 events. Replaces the deprecated `offset_word=0x005A` / `params[2] = key4[2]` formula in §7.8.2.
The TERM frame fetches the partial last chunk and the file footer. Its response payload
contains the bytes between the last full 0x0200-aligned chunk and `end_offset` — typically
20520 B — and is **required for reconstructing the Blastware waveform file**. Append the
TERM response data to the chunk stream like any other A5 frame.
```
last_chunk_counter = address of last full 0x0200-byte chunk read
next_boundary = last_chunk_counter + 0x0200
TERM offset_word = end_offset - next_boundary
TERM params[0] = key[0] (= 0x01 on every observed device)
TERM params[1] = key[1] (= 0x11)
TERM params[2] = (next_boundary >> 8) & 0xFF
TERM params[3] = next_boundary & 0xFF
TERM params[4:10] = zeros ← 10-byte params (not 11)
Frame = build_5a_frame(offset_word, params)
```
The device receives `requested_address = (params[2] << 8) | offset_word` (where offset_word
contains both `offset_hi` and `offset_lo` of the 5A frame, with the high bit of offset_hi
being effectively `bit 0 of (end_offset >> 8)`). It reconstructs `end_offset` and replies
with `(end_offset - next_boundary)` bytes of waveform tail starting at `next_boundary`.
##### Verification
| Event | end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] | TERM response size |
|---|---|---|---|---|---|---|
| 2-sec | `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ | 208 B |
| 3-sec | `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ | 520 B |
| Event-2 | `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ | (not measured directly; same pattern) |
Equivalent way to write the formula:
- `offset_word = end_offset & 0x01FF` — low 9 bits of end_offset
- `params[2:4] = (end_offset & 0xFE00) BE` — high 7 bits of end_offset, low byte zeroed
(The two forms are arithmetically identical to `end_offset - next_boundary` and
`next_boundary` because `next_boundary = end_offset & 0xFE00` whenever the chunk loop
stopped at the last full 0x0200 boundary below end_offset.)
#### 7.8.7 Fixed Metadata Pages 0x1002 / 0x1004 (NEW 2026-05-01) 🔶
> 🔶 Inferred — observed in BW captures but page contents not yet byte-decoded.
Two chunk addresses are **GLOBAL** device/session metadata, not event-specific:
- `counter = 0x1002` — first metadata page
- `counter = 0x1004` — second metadata page
These are at fixed absolute addresses in the device's flash buffer. They contain the
session-start compliance-setup ASCII strings — **Project**, **Client**, **User Name**,
**Seis Loc**, **Extended Notes** — that under the deprecated 0x0400-step walk used to be
discoverable in the sample-chunk stream as "A5 frame 7" content. Under the corrected
0x0200-step walk these strings come exclusively from the dedicated metadata-page reads,
not from sample chunks.
##### Caching strategy
BW reads them ONCE per Blastware session, during event 1's download, and caches them.
For SFM:
- Read once per `MiniMateClient.connect()` / once per call-home session.
- Subsequent events in the same session don't need to re-fetch them.
- Their content does not change while iterating events. They DO change when the user
applies a new compliance setup (SUB 71 write) — invalidate the cache then.
##### TODO — content layout
The byte-for-byte layout of pages 0x1002 and 0x1004 has not been decoded. First task on
the implementation side: dump both pages from a fresh capture and verify they include all
the strings currently extracted from the deprecated A5 frame 7 of the chunk stream.
Compare to the existing `_decode_a5_metadata_into` parser — same string-search anchors
(`b"Project:"`, `b"Client:"`, `b"User Name:"`, `b"Seis Loc:"`, `b"Extended Notes"`) likely
apply directly.
#### 7.8.8 "Download All" Sequence (NEW 2026-05-01) ✅
> ✅ Confirmed from 5-1-26 "Download All" capture (`raw_*_171216_download_all_2events.bin`).
Before any 5A traffic, BW's "Download All" pre-walks the entire event chain to map keys
and event boundaries:
```
SERIAL × 2 → CHCFG → EVT_KEY (1E, all-zero) → key0
→ WAVEHDR (0A, key0) → off=0x46 (real event start)
→ EVT_NEXT (1F, all-zero) → key1
→ WAVEHDR (0A, key1) → off=0x2C (boundary)
→ EVT_NEXT → key2
→ WAVEHDR (0A, key2) → off=0x46 (real event start)
→ EVT_NEXT → key3
→ WAVEHDR (0A, key3) → off=0x2C (boundary)
→ EVT_NEXT → null sentinel
```
The DATA_LENGTH at `data_rsp.data[5]` (echoed BW offset for the data fetch step)
disambiguates real events from boundary markers:
| WAVEHDR offset | Meaning |
|---|---|
| `0x46` (= 70) | Real event start key — this key has event data behind it |
| `0x2C` (= 44) | Boundary marker — this key is the END of the previous event AND the start of the empty/header gap before the next event |
Pairs: each real event spans `[real_key, next_real_key)` in the buffer. In the example
above: event 1 = `[01110000, 011121F2)`, event 2 = `[01112238, 0111417E)`. Note that the
"end of event 1" key (`011121F2`) is also the "boundary key" that comes BEFORE event 2's
real start key (`01112238`) — they differ by exactly 0x46 bytes (the event header size).
After the pre-walk completes, BW downloads each `0x46`-keyed event in turn using the 5A
bulk stream protocol from §7.8.5. Use the `0x46` keys, not the `0x2C` keys, as input to
`read_bulk_waveform_stream`.
---
## 7.9 Compliance Config Field Inventory (Blastware UI, 2026-04-08) ✅
+10 -2
View File
@@ -21,7 +21,15 @@ Typical usage (TCP / modem):
from .client import MiniMateClient
from .models import DeviceInfo, Event, MonitorLogEntry
from .transport import SerialTransport, TcpTransport
from .transport import CapturingTransport, SerialTransport, TcpTransport
__version__ = "0.1.0"
__all__ = ["MiniMateClient", "DeviceInfo", "Event", "MonitorLogEntry", "SerialTransport", "TcpTransport"]
__all__ = [
"MiniMateClient",
"DeviceInfo",
"Event",
"MonitorLogEntry",
"SerialTransport",
"TcpTransport",
"CapturingTransport",
]
+58 -27
View File
@@ -672,11 +672,10 @@ def write_blastware_file(
# Do NOT use a5_frames[-1] — if _a5_frames contains stray frames from a
# subsequent event (a known get_events side-effect), the last frame will
# not be the terminator and the footer will be mis-identified.
# TERM detection (v0.14.0): last frame if page_key != 0x0010 (sample marker)
term_idx: Optional[int] = None
for _i, _f in enumerate(a5_frames):
if _f.page_key == 0x0000:
term_idx = _i
break
if a5_frames and a5_frames[-1].page_key != 0x0010:
term_idx = len(a5_frames) - 1
if term_idx is not None:
body_frames = a5_frames[:term_idx]
@@ -685,34 +684,28 @@ def write_blastware_file(
body_frames = a5_frames
term_frame = None
log.warning(
"write_blastware_file: %d body_frames term_idx=%s",
len(body_frames),
str(term_idx) if term_idx is not None else "None",
# Frame contribution loop (v0.14.0 BW-exact walk).
# Skip values:
# probe (fi=0): probe_skip
# meta@0x1002 (fi=1): 13 (6-byte inner header)
# meta@0x1004 (fi=2): 13 (6-byte inner header)
# sample chunks (fi=3+): 12 (5-byte inner header)
last_fi = len(body_frames) - 1
log.debug(
"write_blastware_file: %d body_frames last_fi=%d",
len(body_frames), last_fi,
)
all_bytes = bytearray()
for fi, frame in enumerate(body_frames):
# All body frames contribute to the waveform body — no frames are skipped.
#
# Over TCP via cellular modem, _recv_5a_batch() correctly collects all
# A5 frames per chunk request (the device's ~1100-byte RS-232 response
# is forwarded as ~2 TCP segments of ~550 bytes each, each parsed as a
# separate S3 frame). ALL of these frames contain ADC body data and
# must be included in the file — confirmed from 4-27-26 TCP capture
# analysis: contributions from all 14 frames → 6821 bytes → file 6864 bytes.
#
# Skip amounts (offsets into frame.data):
# fi=0 (probe): probe_skip — skips the type_tag header + STRT record
# fi=1: 13 — 7-byte frame.data prefix + 6 inner header bytes
# fi>=2: 12 — 7-byte frame.data prefix + 5 inner header bytes
if fi == 0:
skip = probe_skip
elif fi == 1:
skip = 13
elif fi in (1, 2):
skip = 13 # metadata pages
else:
skip = 12
skip = 12 # sample chunks
contribution = _frame_body_bytes(frame, skip)
log.warning("write_blastware_file: fi=%d skip=%d raw_data=%d contribution=%d",
@@ -739,11 +732,49 @@ def write_blastware_file(
bytes(all_bytes[-28:]).hex() if len(all_bytes) >= 28 else bytes(all_bytes).hex(),
)
if len(all_bytes) >= 26:
# Strip embedded "duplicate header+STRT" blocks from body (v0.14.1).
# Chunk@0x1000 sometimes lands on the device's metadata-mirror page,
# whose response includes a 25-byte "00 12 03 00 STRT ..." block that
# mirrors the file's own header + STRT record. BW treats embedded STRT
# markers as second-event starts and rejects the file. Replace these
# blocks with zeros to preserve file size + alignment.
needle = b"\x00\x12\x03\x00STRT"
pos = bytes(all_bytes).find(needle)
while pos >= 0:
end = pos + 25
if end <= len(all_bytes):
all_bytes[pos:end] = b"\x00" * 25
log.warning(
"write_blastware_file: stripped duplicate header+STRT at "
"all_bytes[%d:%d] (replaced with 25 zero-bytes)",
pos, end,
)
pos = bytes(all_bytes).find(needle, end)
# Find the first valid 0e 08 footer marker (v0.14.0).
footer_pos = -1
pos = 0
while True:
pos = bytes(all_bytes).find(b"\x0e\x08", pos)
if pos < 0 or pos + 26 > len(all_bytes):
break
yr = (all_bytes[pos + 4] << 8) | all_bytes[pos + 5]
if 2015 <= yr <= 2050:
footer_pos = pos
break
pos += 1
if footer_pos >= 0:
body = bytes(all_bytes[:footer_pos])
footer = bytes(all_bytes[footer_pos:footer_pos + 26])
log.warning(
"write_blastware_file: real 0e 08 footer at all_bytes[%d]; "
"truncating %d post-footer bytes",
footer_pos, len(all_bytes) - footer_pos - 26,
)
elif len(all_bytes) >= 26:
body = bytes(all_bytes[:-26])
footer = bytes(all_bytes[-26:])
else:
# Fallback: no terminator or very short stream → build footer from event metadata
body = bytes(all_bytes)
start_dt = _ts_from_model(event.timestamp)
stop_dt: Optional[datetime.datetime] = None
@@ -754,7 +785,7 @@ def write_blastware_file(
+ _encode_ts_be(start_dt)
+ _encode_ts_be(stop_dt)
+ b"\x00\x01\x00\x02\x00\x00"
+ b"\x00\x00" # CRC placeholder
+ b"\x00\x00"
)
# ── Write file ───────────────────────────────────────────────────────────
+66 -22
View File
@@ -1345,6 +1345,11 @@ def _decode_waveform_record_into(data: bytes, event: Event) -> None:
event.timestamp = Timestamp.from_continuous_record(data)
except Exception as exc:
log.warning("continuous record timestamp decode failed: %s", exc)
elif event.record_type == "Waveform (Short)":
try:
event.timestamp = Timestamp.from_short_record(data)
except Exception as exc:
log.warning("short record timestamp decode failed: %s", exc)
# ── Peak values (per-channel PPV + Peak Vector Sum) ───────────────────────
try:
@@ -1636,34 +1641,73 @@ def _decode_a5_waveform(
}
def _detect_record_format(data: bytes) -> Optional[str]:
"""
Detect which timestamp-header format a 210-byte 0C waveform record uses.
THREE formats observed on BE11529 firmware S338.17:
"single_shot" 9-byte header:
[day] [0x10] [month] [year_BE:2] [unknown] [hour] [min] [sec]
sub_code=0x10 at byte [1]. Year at [3:5].
"continuous" 10-byte header:
[0x10] [day] [0x10] [month] [year_BE:2] [unknown] [hour] [min] [sec]
marker 0x10 at byte [0] AND byte [2]. Year at [4:6].
"short" 8-byte header (NEW 2026-05-01):
[day] [month] [year_BE:2] [unknown] [hour] [min] [sec]
No marker bytes. Year at [2:4].
Each format has the year (uint16 BE) at a UNIQUE byte position, so we can
disambiguate by scanning each candidate position and picking the one
where the year falls in a sane range (2015..2050).
Returns "single_shot" / "continuous" / "short" or None if no format matches.
"""
if len(data) < 8:
return None
def _sane_year(hi: int, lo: int) -> bool:
y = (hi << 8) | lo
return 2015 <= y <= 2050
# Order matters: prefer formats with stronger marker-byte evidence first.
if data[1] == 0x10 and len(data) >= 9 and _sane_year(data[3], data[4]):
return "single_shot"
if (data[0] == 0x10 and data[2] == 0x10
and len(data) >= 10 and _sane_year(data[4], data[5])):
return "continuous"
if _sane_year(data[2], data[3]):
return "short"
return None
def _extract_record_type(data: bytes) -> Optional[str]:
"""
Decode the recording mode from byte[1] of the 210-byte waveform record.
Return a human-readable name for the waveform record format detected
in the first bytes of a 210-byte 0C record.
Byte[1] is the sub-record code that immediately follows the day byte in the
9-byte timestamp header at the start of each waveform record:
[day:1] [sub_code:1] [month:1] [year:2 BE] ...
Confirmed codes ( 2026-04-01):
0x10 "Waveform" (continuous / single-shot mode)
Histogram mode code is not yet confirmed a histogram event must be
captured with debug=true to identify it. Returns None for unknown codes.
Maps to the format codes returned by _detect_record_format():
"single_shot" "Waveform"
"continuous" "Waveform (Continuous)"
"short" "Waveform (Short)"
None "Unknown(XX.YY.ZZ)"
"""
if len(data) < 2:
return None
code = data[1]
if code == 0x10:
fmt = _detect_record_format(data)
if fmt == "single_shot":
return "Waveform"
if code == 0x03:
# Continuous mode waveform record (confirmed by user — NOT a monitor log).
# The byte layout differs from 0x10 single-shot records: the timestamp
# fields decode as garbage under the 0x10 waveform layout.
# TODO: confirm correct timestamp layout for 0x03 records from a known-time event.
if fmt == "continuous":
return "Waveform (Continuous)"
log.warning("_extract_record_type: unknown sub_code=0x%02X", code)
return f"Unknown(0x{code:02X})"
if fmt == "short":
return "Waveform (Short)"
if len(data) >= 3:
log.warning(
"_extract_record_type: unrecognized header: data[0:3]=%02X %02X %02X",
data[0], data[1], data[2],
)
return f"Unknown({data[0]:02X}.{data[1]:02X}.{data[2]:02X})"
return None
def _extract_peak_floats(data: bytes) -> Optional[PeakValues]:
"""
+130 -18
View File
@@ -123,8 +123,11 @@ def build_5a_frame(offset_word: int, raw_params: bytes) -> bytes:
Returns:
Complete frame bytes: [ACK][STX][stuffed_section][chk][ETX]
"""
if len(raw_params) not in (10, 11):
raise ValueError(f"raw_params must be 10 or 11 bytes, got {len(raw_params)}")
if len(raw_params) not in (10, 11, 12):
# 10 = termination params; 11 = regular probe / chunk params;
# 12 = metadata-page params (extra trailing 0x00 — BW byte-perfect quirk
# for the two fixed metadata reads at counter=0x1002 and 0x1004).
raise ValueError(f"raw_params must be 10/11/12 bytes, got {len(raw_params)}")
# Build stuffed section between STX and checksum
s = bytearray()
@@ -398,28 +401,21 @@ def bulk_waveform_params(key4: bytes, counter: int, *, is_probe: bool = False) -
def bulk_waveform_term_params(key4: bytes, counter: int) -> bytes:
"""
Build the 10-byte params block for the SUB 5A termination request.
DEPRECATED 2026-05-01 see bulk_waveform_term_v2().
The termination request uses offset=0x005A and a DIFFERENT params layout
the leading 0x00 byte is dropped, key4[0:2] shifts to params[0:2], and the
counter high byte is at params[2]:
Build the 10-byte params block for the SUB 5A termination request, OLD layout
(used in conjunction with the fixed offset_word=0x005A). Kept for backward
compatibility produces a tiny ~100-byte device-side terminator response
rather than the proper partial-last-chunk + footer payload that BW gets.
params[0] = key4[0]
params[1] = key4[1]
params[2] = (counter >> 8) & 0xFF
params[3:] = zeros
Counter for the termination request = last_regular_counter + 0x0400.
Confirmed from 1-2-26 BW TX capture: final request (frame 83) uses
offset=0x005A, params[0:3] = key4[0:2] + term_counter_hi.
Args:
key4: 4-byte waveform key.
counter: Termination counter (= last regular counter + 0x0400).
Returns:
10-byte params block.
Use bulk_waveform_term_v2() for new code it computes the verified
offset_word + params from end_offset (extracted from STRT) and the last
chunk counter.
"""
if len(key4) != 4:
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
@@ -430,6 +426,123 @@ def bulk_waveform_term_params(key4: bytes, counter: int) -> bytes:
return bytes(p)
def bulk_waveform_term_v2(
key4: bytes,
end_offset: int,
last_chunk_counter: int,
) -> tuple[int, bytes]:
"""
Compute the SUB 5A TERM frame's offset_word and 10-byte params block.
Confirmed across 3 events (4-27-26 + 5-1-26 captures):
next_boundary = last_chunk_counter + 0x0200
offset_word = end_offset - next_boundary (residual byte count)
params[0] = key4[0] (= 0x01 on every observed device)
params[1] = key4[1] (= 0x11)
params[2] = (next_boundary >> 8) & 0xFF
params[3] = next_boundary & 0xFF
params[4:10] = zeros
Verification:
| end_offset | last_chunk | next_boundary | offset_word | params[2:4] |
| 0x1ABE | 0x1800 | 0x1A00 | 0x00BE | 1A 00 |
| 0x21F2 | 0x1E00 | 0x2000 | 0x01F2 | 20 00 |
| 0x417E | 0x3E38 | 0x4038 | 0x0146 | 40 38 |
The device receives `requested_address = (params[2] << 8) | offset_word`
and replies with `(end_offset - next_boundary)` bytes of waveform tail
starting at `next_boundary` including the 26-byte file footer.
Args:
key4: 4-byte waveform key for this event.
end_offset: Event-end pointer (= `(end_key[2] << 8) | end_key[3]`
from the STRT record at data[23:27] of A5[0]).
last_chunk_counter: Counter of the last full 0x0200-byte chunk fetched
(the chunk that covers [last_chunk_counter,
last_chunk_counter + 0x0200)).
Returns:
(offset_word, params10) tuple. Pass as
`build_5a_frame(offset_word, params)`.
Raises:
ValueError: on inconsistent inputs.
"""
if len(key4) != 4:
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
next_boundary = last_chunk_counter + 0x0200
if next_boundary > 0xFFFF:
raise ValueError(
f"next_boundary 0x{next_boundary:04X} exceeds uint16; check inputs"
)
if end_offset <= last_chunk_counter:
raise ValueError(
f"end_offset 0x{end_offset:04X} must be > "
f"last_chunk_counter 0x{last_chunk_counter:04X}"
)
offset_word = end_offset - next_boundary
if offset_word < 0:
# Last chunk overshot end_offset; caller should have stopped one chunk
# earlier. Treat as zero residual.
offset_word = 0
if offset_word > 0xFFFF:
raise ValueError(
f"offset_word 0x{offset_word:04X} exceeds uint16"
)
p = bytearray(10)
p[0] = key4[0]
p[1] = key4[1]
p[2] = (next_boundary >> 8) & 0xFF
p[3] = next_boundary & 0xFF
return offset_word, bytes(p)
# ── End-offset extraction from STRT record ────────────────────────────────────
STRT_MARKER = b"STRT"
def parse_strt_end_offset(a5_data: bytes) -> Optional[int]:
"""
Extract the event-end offset from the STRT record in an A5 response payload.
The first A5 response (the probe response, or the first chunk for events
with non-zero start_key[2:4]) contains a STRT record at byte offset 17 of
`data`. Layout:
data[17:21] "STRT"
data[21:23] ff fe sentinel
data[23:27] end_key 4-byte key of where this event ENDS
data[27:31] start_key
...
Returns `(end_key[2] << 8) | end_key[3]` the absolute device-buffer
address where the event ends. Use this to bound the chunk loop and to
compute the TERM frame.
Verified end_offset values:
| event start_key | end_key | end_offset |
| 01110000 | 01111ABE | 0x1ABE |
| 01110000 | 011121F2 | 0x21F2 |
| 011121F2 | 0111417E | 0x417E |
Args:
a5_data: The `data` field of an A5 response frame (frame.data).
Returns:
The end_offset (uint16) if STRT is found, else None.
"""
pos = a5_data.find(STRT_MARKER)
if pos < 0 or pos + 10 > len(a5_data):
return None
# data[pos+4:pos+6] is "ff fe"; data[pos+6:pos+10] is end_key.
end_key = a5_data[pos + 6 : pos + 10]
if len(end_key) < 4:
return None
return (end_key[2] << 8) | end_key[3]
# ── Pre-built POLL frames ─────────────────────────────────────────────────────
#
# POLL (SUB 0x5B) uses the same two-step pattern as all other reads — the
@@ -470,7 +583,6 @@ class S3Frame:
# ── Streaming S3 frame parser ─────────────────────────────────────────────────
class S3FrameParser:
"""
Incremental byte-stream parser for S3BW response frames.
+52
View File
@@ -201,6 +201,58 @@ class Timestamp:
second=second,
)
@classmethod
def from_short_record(cls, data: bytes) -> "Timestamp":
"""
Decode an 8-byte timestamp header from a 210-byte waveform record.
Wire layout ( CONFIRMED 2026-05-01 against live SFM run on BE11529 in
Continuous mode, day-of-month = 1 May, raw: 01 05 07 ea 00 0d 15 25):
byte[0]: day (uint8)
byte[1]: month (uint8)
bytes[2-3]: year (big-endian uint16)
byte[4]: unknown (0x00 in observed sample)
byte[5]: hour (uint8)
byte[6]: minute (uint8)
byte[7]: second (uint8)
This is a third format observed in the wild distinct from the 9-byte
(single-shot, sub_code=0x10 at [1]) and 10-byte (continuous, 0x10 at
[0] AND [2]) layouts. No marker bytes; disambiguated by where the
year lands when scanned at byte 2/3/4.
Args:
data: at least 8 bytes; only the first 8 are consumed.
Returns:
Decoded Timestamp.
Raises:
ValueError: if data is fewer than 8 bytes.
"""
if len(data) < 8:
raise ValueError(
f"Short record timestamp requires at least 8 bytes, got {len(data)}"
)
day = data[0]
month = data[1]
year = struct.unpack_from(">H", data, 2)[0]
unknown_byte = data[4]
hour = data[5]
minute = data[6]
second = data[7]
return cls(
raw=bytes(data[:8]),
flag=0,
year=year,
unknown_byte=unknown_byte,
month=month,
day=day,
hour=hour,
minute=minute,
second=second,
)
@property
def clock_set(self) -> bool:
"""False when year == 1995 (factory default / battery-lost state)."""
+225 -237
View File
@@ -35,6 +35,8 @@ from .framing import (
token_params,
bulk_waveform_params,
bulk_waveform_term_params,
bulk_waveform_term_v2,
parse_strt_end_offset,
POLL_PROBE,
POLL_DATA,
SESSION_RESET,
@@ -122,16 +124,22 @@ DATA_LENGTHS: dict[int, int] = {
}
# SUB 5A (BULK_WAVEFORM_STREAM) protocol constants.
# Confirmed from 1-2-26 BW TX capture analysis (2026-04-02).
_BULK_CHUNK_OFFSET = 0x1004 # offset field for probe + all regular chunk requests ✅
_BULK_TERM_OFFSET = 0x005A # offset field for termination request ✅
_BULK_COUNTER_STEP = 0x0400 # chunk counter increment per chunk ✅
# Chunk counter formula: key4[2:4] + (chunk_num - 1) * 0x0400
# where key4[2:4] is the event's circular-buffer base offset ((key4[2]<<8)|key4[3]).
# Earlier captures showed 0x1004 for chunk 1 of key 01110000 — that was a Blastware
# artifact. For keys where key4[2:4] != 0x0000 (e.g. key 01111884) the old
# "n * 0x0400" formula sends counters from the wrong buffer region and the device
# returns data from a different event. Confirmed correct 2026-04-24.
#
# 2026-05-01 minimal-fix: the chunk-counter walk is now bounded by the event's
# `end_offset` extracted from the STRT record at data[23:27] of the probe
# response. Without this bound the loop kept asking for chunks past the event
# end and the device responded with post-event circular-buffer garbage,
# corrupting reconstructed Blastware files for events ≥ 2 sec.
#
# We keep the OLD 0x0400 chunk step here (BW actually uses 0x0200 — see §7.8.5
# of the protocol reference for the corrected understanding) because the
# existing blastware_file.py builder relies on the 0x0400-step frame structure
# to produce valid files. Switching to BW's 0x0200 step is a separate task
# that also requires updating the file builder.
# BW-exact protocol values (v0.14.0). Verified against 4-27-26 + 5-1-26 captures.
_BULK_CHUNK_OFFSET = 0x1002 # offset_word for probe + all chunk requests
_BULK_TERM_OFFSET = 0x005A # offset_word for the legacy terminator (fallback only)
_BULK_COUNTER_STEP = 0x0200 # chunk counter increment (matches chunk payload size)
# Default timeout values (seconds).
# MiniMate Plus is a slow device — keep these generous.
@@ -526,223 +534,260 @@ class MiniMateProtocol:
self,
key4: bytes,
*,
stop_after_metadata: bool = True,
max_chunks: int = 32,
stop_after_metadata: bool = True, # DEPRECATED — no-op under BW-exact walk
max_chunks: int = 256, # safety cap only; loop is bounded by end_offset
include_terminator: bool = False,
extra_chunks_after_metadata: int = 1,
extra_chunks_after_metadata: int = 1, # DEPRECATED — no-op
) -> list[S3Frame]:
"""
Download the SUB 5A (BULK_WAVEFORM_STREAM) A5 frames for one event.
Download the SUB 5A (BULK_WAVEFORM_STREAM) A5 frames for one event using
Blastware's exact protocol. REWRITTEN 2026-05-02 (v0.14.0).
The bulk waveform stream carries both raw ADC samples (large) and
event-time metadata strings ("Project:", "Client:", "User Name:",
"Seis Loc:", "Extended Notes") embedded in one of the middle frames
(confirmed: A5[7] of 9 for 1-2-26 capture).
Algorithm (matches BW captures across 2-sec / 3-sec / event-2):
Protocol is request-per-chunk, NOT a continuous stream:
1. Probe (offset=_BULK_CHUNK_OFFSET, is_probe=True, counter=0x0000)
2. Chunks (offset=_BULK_CHUNK_OFFSET, is_probe=False, counter+=0x0400)
3. Loop until metadata found (stop_after_metadata=True) or max_chunks
4. Termination (offset=_BULK_TERM_OFFSET, counter=last+_BULK_COUNTER_STEP)
Device responds with a final A5 frame (page_key=0x0000).
1. Probe
- For events at start_key[2:4] = 0x0000 (first event after erase
/ wrap): probe at counter=0x0000 with full key in params.
- For continuation events (start_key[2:4] != 0): first chunk at
counter = start_key[2:4] + 0x0046; acts as both probe and
first sample chunk; response carries STRT.
By default the termination frame (page_key=0x0000) is NOT included in the
returned list. Pass include_terminator=True to append it; the blastware_file
writer needs the terminator frame's body to reconstruct the waveform file footer.
2. Parse end_offset from STRT record at data[23:27] of the probe response.
Args:
key4: 4-byte waveform key from EVENT_HEADER (1E).
stop_after_metadata: If True (default), send termination as soon as
b"Project:" is found in a frame's data — avoids
downloading the full ADC waveform payload (several
hundred KB). Set False to download everything.
max_chunks: Safety cap on the number of chunk requests sent
(default 32; a typical event uses 9 large frames).
include_terminator: If True, append the terminator A5 frame
(page_key=0x0000) to the returned list. The
terminator carries the waveform file footer bytes.
Default False preserves existing caller behaviour.
3. Read two fixed metadata pages at counter=0x1002 and counter=0x1004
global session metadata (Project / Client / User Name / Seis Loc
/ Extended Notes ASCII strings). Event 1 only; continuation
events skip these (BW caches them across the session).
4. Walk sample chunks at 0x0200 increments, starting from 0x0600 for
event 1 or `start + 0x0046 + 0x0200` for continuation events.
Stop when `next_chunk + 0x0200 > end_offset`.
5. Send TERM frame with offset_word and params computed by
`bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`.
The TERM response contains the partial last chunk (residual =
end_offset - next_boundary) including the 26-byte 0e 08 file
footer.
Returns:
List of S3Frame objects from each A5 response frame. Frame indices
match the request sequence: index 0 = probe response, index 1 = first
chunk, etc. If include_terminator=True, the last element is the
terminator frame (page_key=0x0000).
List of S3Frame objects from each A5 response (probe, metadata
pages, sample chunks, optional TERM response). Caller passes
`include_terminator=True` (e.g. write_blastware_file) to keep the
TERM response in the list it's required to reconstruct the
file footer.
Deprecated kwargs:
stop_after_metadata: legacy "Project:"-string-based stop condition.
No-op under the BW-exact walk; the loop is
deterministically bounded by end_offset from
STRT. Accepted for backward compat.
extra_chunks_after_metadata: same.
Raises:
ProtocolError: on timeout, bad checksum, or unexpected SUB.
Confirmed from 1-2-26 BW TX/RX captures (2026-04-02):
- probe + 8 regular chunks + 1 termination = 10 TX frames
- 9 large A5 responses + 1 terminator A5 = 10 RX frames
- page_key=0x0010 on large frames; page_key=0x0000 on terminator
- "Project:" metadata at A5[7].data[626]
ProtocolError: on timeout / bad checksum / unexpected SUB.
"""
if len(key4) != 4:
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
rsp_sub = _expected_rsp_sub(SUB_BULK_WAVEFORM) # 0xFF - 0x5A = 0xA5
# Quietly accept and warn on deprecated kwargs.
if not stop_after_metadata:
log.debug("5A: stop_after_metadata=False is no-op under BW-exact walk")
if extra_chunks_after_metadata not in (0, 1):
log.debug("5A: extra_chunks_after_metadata=%d is no-op under BW-exact walk",
extra_chunks_after_metadata)
rsp_sub = _expected_rsp_sub(SUB_BULK_WAVEFORM) # 0xA5
frames_data: list[S3Frame] = []
counter = 0
# BW counter formula (confirmed from 4-3-26 capture for key 0111245a,
# and empirical live-device test 2026-04-06 for key 01110000):
# counter for chunk n = max(key4[2:4], 0x0400) + (n - 1) * 0x0400
# key4[2:4] is the event's circular-buffer base offset. The max() guard
# ensures chunk 1 never uses counter=0x0000 (which equals the probe address
# and causes the device to re-return STRT record data for the first chunk).
_key4_offset = (key4[2] << 8) | key4[3]
start_offset = (key4[2] << 8) | key4[3]
is_event_1 = (start_offset == 0)
# ── Step 1: probe ────────────────────────────────────────────────────
log.debug("5A probe key=%s key4_offset=0x%04X", key4.hex(), _key4_offset)
params = bulk_waveform_params(key4, 0, is_probe=True)
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
self._parser.reset() # reset bytes_fed counter before probe recv
# ── Step 1: probe / first chunk ──────────────────────────────────────
if is_event_1:
probe_counter = 0
probe_params = bulk_waveform_params(key4, 0, is_probe=True)
log.debug("5A probe (event-1) key=%s counter=0x0000", key4.hex())
else:
# Continuation events: first 5A request lands at start+0x0046,
# acting as both probe and first sample chunk. Confirmed from
# 5-1-26 "copy 2nd address event" capture.
probe_counter = start_offset + 0x0046
probe_params = bulk_waveform_params(key4, probe_counter)
log.debug(
"5A probe (event-N) key=%s counter=0x%04X (start+0x46)",
key4.hex(), probe_counter,
)
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, probe_params))
self._parser.reset()
try:
probe_batch = self._recv_5a_batch(rsp_sub)
rsp = self._recv_one(expected_sub=rsp_sub, reset_parser=False)
except TimeoutError:
log.warning(
"5A probe TIMED OUT for key=%s"
"%d raw bytes received (no complete A5 frame assembled)",
"5A probe TIMED OUT for key=%s%d raw bytes received",
key4.hex(), self._parser.bytes_fed,
)
raise
frames_data.extend(probe_batch)
log.debug(
"5A probe: %d frame(s) page_keys=%s",
len(probe_batch),
[f"0x{f.page_key:04X}" for f in probe_batch],
)
# Log probe frame size for diagnostics.
# The device always needs extra_chunks_after_metadata chunks after the
# metadata frame before termination to prime the valid waveform footer.
# This holds regardless of TCP frame size (1-frame vs 2-frame mode).
_effective_extra_chunks = extra_chunks_after_metadata
log.warning(
"5A probe data_len=%d effective_extra_chunks=%d",
len(probe_batch[0].data),
_effective_extra_chunks,
)
frames_data.append(rsp)
log.debug("5A A5[0] (probe) page_key=0x%04X %d bytes",
rsp.page_key, len(rsp.data))
# ── Step 2: chunk loop ───────────────────────────────────────────────
# Counter formula: _chunk_base + (chunk_num - 1) * 0x0400
# where _chunk_base = max(key4[2:4], 0x0400).
#
# For events with key4[2:4] != 0 (e.g. key 0111245a, offset 0x245a):
# _chunk_base = 0x245a → chunk 1=0x245a, chunk 2=0x285a, ...
# Confirmed from 4-3-26 capture.
#
# For events with key4[2:4] == 0 (e.g. key 01110000):
# _chunk_base = max(0, 0x0400) = 0x0400
# → chunk 1=0x0400, chunk 2=0x0800, ... (= old chunk_num*0x0400)
# CRITICAL: counter=0x0000 (same as the probe) causes the device to
# re-return the STRT record data for chunk 1, making frame 1 look like
# a second probe response (confirmed from server log: frame 1 len=1097,
# contains STRT\xff\xfe, contributes zero body bytes after DLE-strip).
# counter=0x0400 for chunk 1 confirmed working (empirical test 2026-04-06).
_chunk_base = max(_key4_offset, _BULK_COUNTER_STEP)
for chunk_num in range(1, max_chunks + 1):
counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP
params = bulk_waveform_params(key4, counter)
log.debug("5A chunk %d counter=0x%04X", chunk_num, counter)
# ── Step 2: parse STRT end_offset from probe response ────────────────
end_offset = parse_strt_end_offset(rsp.data)
if end_offset is None:
log.warning(
"5A probe response did not contain a STRT record; "
"cannot bound chunk loop — falling back to max_chunks=%d cap",
max_chunks,
)
end_offset = 0xFFFF # impossible value → loop runs to max_chunks
else:
log.info(
"5A STRT start_offset=0x%04X end_offset=0x%04X size=0x%04X",
start_offset, end_offset, end_offset - start_offset,
)
# ── Step 3: metadata pages 0x1002 + 0x1004 (event 1 only) ────────────
# Confirmed from BW captures: BW reads these two fixed device-buffer
# pages immediately after the probe for events at start_key[2:4]=0.
# Continuation events skip them (BW caches across the session).
# Their content is global compliance-setup metadata: Project, Client,
# User Name, Seis Loc, Extended Notes.
if is_event_1:
for meta_counter in (0x1002, 0x1004):
# Metadata page params have an extra trailing 0x00 byte
# (12-byte params instead of 11) — empirical from BW captures.
# Checksum-neutral but matches BW byte-for-byte.
meta_params = bytes([
0x00,
key4[0], key4[1],
(meta_counter >> 8) & 0xFF,
meta_counter & 0xFF,
0, 0, 0, 0, 0, 0, 0,
])
log.debug("5A metadata page counter=0x%04X", meta_counter)
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, meta_params))
self._parser.reset()
try:
meta_rsp = self._recv_one(
expected_sub=rsp_sub, reset_parser=False, timeout=10.0,
)
except TimeoutError:
log.warning(
"5A metadata page 0x%04X TIMED OUT — continuing",
meta_counter,
)
continue
frames_data.append(meta_rsp)
log.debug(
"5A meta@0x%04X page_key=0x%04X %d bytes",
meta_counter, meta_rsp.page_key, len(meta_rsp.data),
)
# ── Step 4: sample chunk loop, bounded by end_offset ─────────────────
# Sample chunks start at:
# event 1: counter = 0x0600
# event N (>0): counter = probe_counter + 0x0200
# (probe was the first sample chunk)
if is_event_1:
counter = 0x0600
else:
counter = probe_counter + _BULK_COUNTER_STEP
last_chunk_counter: Optional[int] = (
probe_counter if not is_event_1 else None
)
chunks_fetched = 0
while chunks_fetched < max_chunks:
# Stop when next chunk would straddle the event end.
if counter + _BULK_COUNTER_STEP > end_offset:
log.debug(
"5A chunk loop done at counter=0x%04X (end=0x%04X); "
"%d chunks fetched",
counter, end_offset, chunks_fetched,
)
break
params = bulk_waveform_params(key4, counter)
log.debug("5A chunk #%d counter=0x%04X", chunks_fetched + 1, counter)
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
self._parser.reset() # reset bytes_fed for accurate per-chunk count
self._parser.reset()
try:
# Collect ALL frames from this chunk response.
# Over TCP via modem, a single large A5 device response (~1100 bytes
# RS-232) is split across ~2 TCP segments, each parsed as its own
# complete S3 frame. _recv_5a_batch gathers all of them so that
# every subsequent chunk request is paired with the correct response.
batch = self._recv_5a_batch(rsp_sub, first_timeout=10.0)
rsp = self._recv_one(
expected_sub=rsp_sub, reset_parser=False, timeout=10.0,
)
except TimeoutError:
raw = self._parser.bytes_fed
log.warning(
"5A TIMEOUT chunk=%d counter=0x%04X raw_bytes=%d",
chunk_num, counter, raw,
chunks_fetched + 1, counter, raw,
)
if raw > 0 and frames_data:
# Device sent a partial byte (likely a bare DLE/ETX end-of-stream
# signal) but never completed a full frame. Treat as graceful
# stream end and fall through to the termination step.
log.warning(
"5A end-of-stream detected at chunk=%d (raw_bytes=%d, "
"frames_collected=%d) — proceeding to termination",
chunk_num, raw, len(frames_data),
"5A unexpected end-of-stream — proceeding to TERM",
)
break
raise
# Process all frames from this batch.
metadata_found = False
for rsp in batch:
log.warning(
"5A RX chunk=%d page_key=0x%04X data_len=%d contains_Project=%s",
chunk_num, rsp.page_key, len(rsp.data), b"Project:" in rsp.data,
)
if rsp.page_key == 0x0000:
# Device unexpectedly terminated mid-stream.
log.debug("5A page_key=0x0000 — device terminated early")
if include_terminator:
frames_data.append(rsp)
return frames_data
frames_data.append(rsp)
if stop_after_metadata and b"Project:" in rsp.data:
metadata_found = True
if metadata_found:
# Download extra_chunks_after_metadata more chunks after metadata.
# This primes the device to return the valid waveform footer in the
# termination response — without it the terminator carries too few bytes
# (confirmed 2026-04-23). The extra chunk data also belongs in the
# file body (confirmed from TCP capture analysis 2026-04-27).
log.debug("5A metadata found — fetching %d more chunk(s)",
_effective_extra_chunks)
for _extra_n in range(_effective_extra_chunks):
chunk_num += 1
counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP
params = bulk_waveform_params(key4, counter)
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
try:
extra_batch = self._recv_5a_batch(rsp_sub, first_timeout=10.0)
for ef in extra_batch:
log.debug(
"5A extra chunk page_key=0x%04X data_len=%d",
ef.page_key, len(ef.data),
)
if ef.page_key == 0x0000:
if include_terminator:
frames_data.append(ef)
return frames_data
frames_data.append(ef)
except TimeoutError:
log.debug("5A extra chunk %d timed out — end of stream", _extra_n + 1)
break
break
else:
log.warning(
"5A reached max_chunks=%d without end-of-stream; sending termination",
max_chunks,
log.debug(
"5A RX chunk=%d page_key=0x%04X data_len=%d",
chunks_fetched + 1, rsp.page_key, len(rsp.data),
)
# ── Step 3: termination ──────────────────────────────────────────────
term_counter = counter + _BULK_COUNTER_STEP
term_params = bulk_waveform_term_params(key4, term_counter)
log.debug(
"5A termination term_counter=0x%04X offset=0x%04X",
term_counter, _BULK_TERM_OFFSET,
)
self._send(build_5a_frame(_BULK_TERM_OFFSET, term_params))
try:
term_rsp = self._recv_one(expected_sub=rsp_sub)
if rsp.page_key == 0x0000:
# Device terminated mid-stream unexpectedly.
log.warning(
"5A unexpected page_key=0x0000 mid-stream at counter=0x%04X",
counter,
)
if include_terminator:
frames_data.append(rsp)
return frames_data
frames_data.append(rsp)
last_chunk_counter = counter
counter += _BULK_COUNTER_STEP
chunks_fetched += 1
else:
log.warning(
"5A reached max_chunks=%d at counter=0x%04X (end=0x%04X)",
max_chunks, counter, end_offset,
)
# ── Step 5: TERM with proper end_offset-derived formula ──────────────
if last_chunk_counter is None or end_offset == 0xFFFF:
# No STRT or no chunks fetched — fall back to legacy TERM.
log.warning(
"5A using legacy TERM (offset_word=0x005A); "
"end_offset unavailable or no chunks fetched",
)
legacy_counter = (last_chunk_counter or probe_counter) + _BULK_COUNTER_STEP
term_offset_word = _BULK_TERM_OFFSET # 0x005A
term_params = bulk_waveform_term_params(key4, legacy_counter)
else:
term_offset_word, term_params = bulk_waveform_term_v2(
key4, end_offset, last_chunk_counter,
)
log.debug(
"5A termination response page_key=0x%04X %d bytes",
"5A TERM offset_word=0x%04X params[2:4]=%s end=0x%04X "
"last_chunk=0x%04X",
term_offset_word, term_params[2:4].hex(),
end_offset, last_chunk_counter,
)
self._send(build_5a_frame(term_offset_word, term_params))
try:
term_rsp = self._recv_one(expected_sub=rsp_sub, timeout=10.0)
log.info(
"5A TERM response page_key=0x%04X %d bytes",
term_rsp.page_key, len(term_rsp.data),
)
if include_terminator:
frames_data.append(term_rsp)
except TimeoutError:
log.debug("5A no termination response — device may have already closed")
log.warning("5A no TERM response (timeout)")
return frames_data
@@ -1403,63 +1448,6 @@ class MiniMateProtocol:
log.debug("TX %d bytes: %s", len(frame), frame.hex())
self._transport.write(frame)
def _recv_5a_batch(
self,
expected_sub: int,
first_timeout: float = 10.0,
batch_timeout: float = 0.5,
) -> list[S3Frame]:
"""
Collect all S3 frames that arrive as part of one device response.
Over TCP via cellular modem, a single device A5 response (~1100 bytes of
RS-232 data) is forwarded in multiple TCP segments due to the modem's
data-forwarding timeout (~100-150 ms per segment). Each TCP segment
contains a complete, valid S3 frame (~550 bytes). Calling _recv_one()
once returns only the first segment's frame and misses the rest, causing
the chunk request/response pairing to cascade out of alignment.
This helper collects ALL frames before returning, by trying additional
short-timeout receives after the first frame arrives.
The caller must call self._parser.reset() before this method to ensure
bytes_fed is accurate; this method always uses reset_parser=False.
Args:
expected_sub: Expected SUB byte for validation.
first_timeout: Timeout for the mandatory first frame. Should be
generous (default 10 s) since the device may be slow.
batch_timeout: Short timeout for subsequent frames. Default 0.5 s
comfortably longer than the modem forwarding gap
(~150 ms) but short enough to avoid stalling when
only one frame is expected (probe, terminator).
Returns:
List of S3Frame objects in arrival order (at least one).
Raises:
TimeoutError: If no frame arrives within first_timeout.
UnexpectedResponse: If any frame has the wrong SUB byte.
"""
frames: list[S3Frame] = []
first = self._recv_one(
expected_sub=expected_sub,
reset_parser=False,
timeout=first_timeout,
)
frames.append(first)
while True:
try:
extra = self._recv_one(
expected_sub=expected_sub,
reset_parser=False,
timeout=batch_timeout,
)
frames.append(extra)
except TimeoutError:
break
return frames
def _recv_one(
self,
expected_sub: Optional[int] = None,
+99
View File
@@ -454,3 +454,102 @@ class SocketTransport(TcpTransport):
def __repr__(self) -> str:
return f"SocketTransport(peer={self.host!r})"
# ── Capturing transport (MITM-style raw byte mirror) ──────────────────────────
class CapturingTransport(BaseTransport):
"""
Wraps another BaseTransport and mirrors every byte to two raw capture files:
raw_bw_<...>.bin bytes WE wrote to the device (BW-side TX)
raw_s3_<...>.bin bytes the device wrote back (S3-side TX)
The file naming and on-wire byte layout are identical to the captures
produced by `bridges/ach_mitm.py`, so the resulting `.bin` files can be
loaded directly by the Analyzer (File > Open Capture) and parsed by the
same tooling used for genuine Blastware MITM captures.
All BaseTransport methods are forwarded to the inner transport; the only
side-effect is that successful read/write byte streams are appended to the
two open binary files.
Args:
inner: An already-built BaseTransport (SerialTransport / TcpTransport).
bw_path: File path for the "BW TX" stream (bytes we send). Opened "wb".
s3_path: File path for the "S3 TX" stream (bytes the device sends).
Opened "wb".
Example:
with CapturingTransport(TcpTransport("1.2.3.4", 9034),
"raw_bw.bin", "raw_s3.bin") as t:
client = MiniMateClient(transport=t)
client.connect()
client.get_events()
# both .bin files now hold the full bidirectional capture.
"""
def __init__(self, inner: BaseTransport, bw_path: str, s3_path: str) -> None:
self._inner = inner
self._bw_path = bw_path
self._s3_path = s3_path
self._bw_fh = None
self._s3_fh = None
# Forward inner attrs so callers can introspect (e.g. .host, .port).
self.host = getattr(inner, "host", None)
self.port = getattr(inner, "port", None)
# ── BaseTransport interface ───────────────────────────────────────────────
def connect(self) -> None:
if self._bw_fh is None:
self._bw_fh = open(self._bw_path, "wb", buffering=0)
if self._s3_fh is None:
self._s3_fh = open(self._s3_path, "wb", buffering=0)
self._inner.connect()
def disconnect(self) -> None:
try:
self._inner.disconnect()
finally:
for fh_attr in ("_bw_fh", "_s3_fh"):
fh = getattr(self, fh_attr)
if fh is not None:
try:
fh.flush()
fh.close()
except Exception:
pass
setattr(self, fh_attr, None)
@property
def is_connected(self) -> bool:
return self._inner.is_connected
def write(self, data: bytes) -> None:
self._inner.write(data)
if data and self._bw_fh is not None:
try:
self._bw_fh.write(data)
except Exception:
pass
def read(self, n: int) -> bytes:
got = self._inner.read(n)
if got and self._s3_fh is not None:
try:
self._s3_fh.write(got)
except Exception:
pass
return got
@property
def bw_path(self) -> str:
return self._bw_path
@property
def s3_path(self) -> str:
return self._s3_path
def __repr__(self) -> str:
return f"CapturingTransport({self._inner!r}, bw={self._bw_path!r}, s3={self._s3_path!r})"
+2
View File
@@ -53,7 +53,9 @@ SUB_TABLE: dict[int, tuple[str, str, str]] = {
0x82: ("TRIGGER_CONFIG_WRITE", "BW→S3", "0x1C bytes; trigger config block; mirrors SUB 1C"),
0x83: ("TRIGGER_WRITE_CONFIRM", "BW→S3", "Short frame; commit step after 0x82"),
# S3→BW responses
0x5A: ("BULK_WAVEFORM_STREAM", "BW→S3", "Bulk waveform chunk request; response is A5 stream"),
0xA4: ("POLL_RESPONSE", "S3→BW", "Response to SUB 5B poll"),
0xA5: ("BULK_WAVEFORM_RESPONSE", "S3→BW", "Response to SUB 5A; waveform chunks + metadata"),
0xFE: ("FULL_CONFIG_RESPONSE", "S3→BW", "Response to SUB 01"),
0xF9: ("CHANNEL_CONFIG_RESPONSE", "S3→BW", "Response to SUB 06"),
0xF7: ("EVENT_INDEX_RESPONSE", "S3→BW", "Response to SUB 08; contains backlight/power-save"),
+33 -36
View File
@@ -33,7 +33,7 @@ STX = 0x02
ETX = 0x03
ACK = 0x41
__version__ = "0.2.3"
__version__ = "0.2.5"
@dataclass
@@ -184,9 +184,9 @@ def validate_bw_body_auto(body: bytes) -> Optional[Tuple[bytes, bytes, str]]:
def parse_s3(blob: bytes, trailer_len: int) -> List[Frame]:
frames: List[Frame] = []
IDLE = 0
IN_FRAME = 1
AFTER_DLE = 2
IDLE = 0
IN_FRAME = 1
IN_FRAME_DLE = 2 # saw DLE inside frame — waiting for next byte
state = IDLE
body = bytearray()
@@ -206,66 +206,63 @@ def parse_s3(blob: bytes, trailer_len: int) -> List[Frame]:
state = IN_FRAME
i += 2
continue
# ACK bytes, boot strings, garbage — silently ignored
elif state == IN_FRAME:
if b == DLE:
state = AFTER_DLE
state = IN_FRAME_DLE
i += 1
continue
body.append(b)
else: # AFTER_DLE
if b == DLE:
body.append(DLE)
state = IN_FRAME
i += 1
continue
if b == ETX:
# Bare ETX = real S3 frame terminator (confirmed from S3FrameParser)
end_offset = i + 1
trailer_start = i + 1
trailer_end = trailer_start + trailer_len
trailer = blob[trailer_start:trailer_end]
chk_valid = None
chk_type = None
chk_hex = None
payload = bytes(body)
if len(body) >= 1:
received_chk = body[-1]
computed_chk = checksum8_sum(bytes(body[:-1]))
if computed_chk == received_chk:
chk_valid = True
chk_type = "SUM8"
chk_hex = f"{received_chk:02x}"
payload = bytes(body[:-1])
else:
chk_valid = False
# S3 checksums are deliberately not validated here.
# Large S3 responses (A5 bulk waveform, E5 compliance) embed
# inner DLE+ETX sub-frame terminators whose trailing 0x03 byte
# lands where the parser would expect the SUM8 checksum, causing
# false failures. The live protocol (protocol.py _validate_frame)
# also skips S3 checksum enforcement for the same reason.
frames.append(Frame(
index=idx,
start_offset=start_offset,
end_offset=end_offset,
payload_raw=bytes(body),
payload=payload,
payload=bytes(body),
trailer=trailer,
checksum_valid=chk_valid,
checksum_type=chk_type,
checksum_hex=chk_hex
checksum_valid=None,
checksum_type=None,
checksum_hex=None
))
idx += 1
state = IDLE
i = trailer_end
continue
body.append(b)
else: # IN_FRAME_DLE
if b == DLE:
# DLE DLE → literal 0x10 in payload
body.append(DLE)
state = IN_FRAME
i += 1
continue
if b == ETX:
# DLE+ETX inside a frame = inner-frame terminator (A4/E5 sub-frames).
# Treat as literal data, NOT the outer frame end.
body.append(DLE)
body.append(ETX)
state = IN_FRAME
i += 1
continue
# Unexpected DLE + byte → treat as literal data
body.append(DLE)
body.append(b)
state = IN_FRAME
i += 1
continue
i += 1
+804 -112
View File
File diff suppressed because it is too large Load Diff
+19 -18
View File
@@ -37,6 +37,7 @@ from __future__ import annotations
import datetime
import logging
import sys
import tempfile
import threading
import time
from pathlib import Path
@@ -863,8 +864,8 @@ def device_event_blastware_file(
Supply either *port* (serial) or *host* (TCP/modem).
The file is written to /tmp and streamed back as a binary download.
Blastware can open it directly filename encodes serial + timestamp.
The file is written to the OS temp directory and streamed back as a binary
download. Blastware can open it directly filename encodes serial + timestamp.
Filename format: <prefix><serial3><stem><AB>0<W|H>
- prefix letter = chr(ord('B') + floor(serial_numeric / 1000))
@@ -885,23 +886,13 @@ def device_event_blastware_file(
def _do():
with _build_client(port, baud, host, tcp_port, timeout=120.0) as client:
info = client.connect()
# Use stop_after_metadata=True (full_waveform=False) with 1 extra
# chunk after "Project:". The extra chunk primes the device so that
# the termination response carries the full waveform footer bytes.
# Without it the terminator returns only ~90 bytes (no useful footer).
#
# The extra chunk's ADC data IS part of the Blastware file body —
# confirmed from 4-27-26 TCP capture: all 14 A5 frames (including the
# extra chunk's 2 TCP sub-frames) contribute to the correct 6864-byte
# output. write_blastware_file() includes all frames unconditionally.
#
# full_waveform=True (natural end-of-stream) downloads ALL chunks
# including post-event silence (35+ chunks for a 9-sec event at
# 1024 sps) — this produces 24KB+ files that Blastware rejects.
# Under v0.14.0 BW-exact 5A walk, the chunk loop is bounded by
# the event end_offset extracted from STRT. No more
# stop_after_metadata / extra_chunks gymnastics — these
# kwargs are now no-ops.
events = client.get_events(
full_waveform=False,
stop_after_index=index,
extra_chunks_after_metadata=1,
)
matching = [ev for ev in events if ev.index == index]
return matching[0] if matching else None, info
@@ -937,8 +928,18 @@ def device_event_blastware_file(
# Build filename using the same algorithm Blastware uses
filename = blastware_filename(ev, serial)
# Write to /tmp so FastAPI can stream it back
out_path = Path("/tmp") / filename
# Write to OS temp dir (cross-platform: /tmp on Linux/macOS,
# %TEMP% on Windows) so FastAPI can stream it back via FileResponse.
out_path = Path(tempfile.gettempdir()) / filename
# Delete any stale file at this path before writing. On Windows we have
# observed the new (smaller) file getting trailing zero-bytes from the
# previous (larger) file when filesystem semantics around open(...,"wb")
# don't truncate cleanly (e.g. through a synced folder). Explicit unlink
# eliminates that ambiguity.
try:
out_path.unlink()
except FileNotFoundError:
pass
write_blastware_file(ev, a5_frames, out_path)
log.info(
"blastware_file: wrote %s (%d A5 frames, serial=%s)",