Merge branch 'main' into protocol-fix

This commit was merged in pull request #11.
This commit is contained in:
2026-05-04 14:43:05 -04:00
5 changed files with 604 additions and 123 deletions
+30
View File
@@ -4,6 +4,25 @@ All notable changes to seismo-relay are documented here.
--- ---
## v0.12.6 — 2026-05-01
### Fixed
- **`blastware_file.py` — waveform frame classification** — A5 frame classification for
waveform-only vs header-only frames now uses `frame.record_type` instead of frame index.
Only waveform frames (0x46) are written to the file body; metadata frames are skipped.
Fixes spurious data corruption from incorrectly classified frames.
- **`s3_analyzer.py` — A5/5A frame naming** — Bulk waveform stream frames (SUB 5A response)
are now correctly labeled "A5" in analyzer output instead of being conflated with other
multi-frame responses (SUB A4, E5, etc.).
- **`S3FrameParser` — frame terminator detection** — Corrected the bare ETX terminator
detection. Frame termination is now correctly identified by a standalone `ETX=0x03` byte,
not by the `DLE+ETX` sequence (which is part of the payload when it appears within a frame).
---
## v0.13.2 — 2026-05-01 ## v0.13.2 — 2026-05-01
### Fixed ### Fixed
@@ -112,6 +131,13 @@ All notable changes to seismo-relay are documented here.
## v0.12.5 — 2026-04-21 ## v0.12.5 — 2026-04-21
### Added
- **`seismo_lab.py` — Download tab** — New fourth tab for live wire-byte capture during event
downloads. Captures both BW→device and device→S3 frames in real time, allowing inspection
of the 5A bulk stream chunk sequence and frame-by-frame analysis without needing a bridge
or MITM proxy. Files are saved with user-specified labels for easy tracking.
### Changed ### Changed
- **`s3_bridge.py` — raw captures always-on by default** — `--raw-bw` and `--raw-s3` now - **`s3_bridge.py` — raw captures always-on by default** — `--raw-bw` and `--raw-s3` now
@@ -123,6 +149,10 @@ All notable changes to seismo-relay are documented here.
"S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names "S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names
the files). Unchecking a box passes `--raw-bw ""` to explicitly disable capture. the files). Unchecking a box passes `--raw-bw ""` to explicitly disable capture.
- **`Bridge tab` — TCP mode added** — Serial/TCP radio toggle allows connection via cellular
modem (RV50/RV55) instead of direct RS-232. Supports multi-capture design (simultaneous
Bridge + Analyzer + Download sessions).
- **`ach_server.py` — TX capture added (`raw_tx_<ts>.bin`)** — Every ACH inbound session - **`ach_server.py` — TX capture added (`raw_tx_<ts>.bin`)** — Every ACH inbound session
now saves both directions: `raw_rx_<ts>.bin` (device → us, S3 side, as before) and now saves both directions: `raw_rx_<ts>.bin` (device → us, S3 side, as before) and
`raw_tx_<ts>.bin` (us → device, BW side). Both files are usable in the Analyzer. `raw_tx_<ts>.bin` (us → device, BW side). Both files are usable in the Analyzer.
+227 -32
View File
@@ -118,29 +118,156 @@ S3→BW (response):
Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26 Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26
BW TX capture. All 10 frames verified. BW TX capture. All 10 frames verified.
### SUB 5A — chunk counter formula (FINAL CORRECTION 2026-04-26) ### SUB 5A — chunk counter formula (REWRITTEN 2026-05-01 — see 5-1-26 captures)
**Chunk counter = `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` for ALL chunks.** > ⚠️ **Everything that came before this rewrite was WRONG in important ways.** The previous
> formula `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` happened to *work* for events
> at start_key=0 because the device responds to whatever counter you ask for — but it caused
> a 5× over-read past the actual event, picking up post-event circular-buffer garbage that
> corrupts the reconstructed file for any event > ~1 sec of waveform. The captures in
> `bridges/captures/4-27-26/` and `5-1-26/comcheck/` show BW reads only ~12-16 chunks for
> the same events SFM was reading 37+ chunks for. See "TERM frame" and "STRT end_offset"
> sections below for the actual mechanism.
where `key4[2:4] = (key4[2] << 8) | key4[3]` is the event's circular-buffer base offset. **Chunk addressing is just absolute device-buffer addresses.**
The `max(..., 0x0400)` guard is critical for events at the start of the circular buffer `params[0]=0x00`, `params[1:5]` is a 4-byte absolute device flash-buffer address (= the
(key4[2:4] == 0x0000, e.g. key `01110000`). Without it, chunk 1 gets counter=0x0000, which "key" of that location), `params[5:11]` are zeros. The device returns 0x0200 (= 512) bytes
is the same address as the probe frame — the device re-returns the STRT record data instead starting at that address. Increments between consecutive chunks are **0x0200 (NOT 0x0400)**
of waveform payload. With the guard, chunk 1 gets counter=0x0400, which is confirmed correct — this matches the chunk payload size. The previous "0x0400 step" worked by accident: BW
from the empirical live-device test 2026-04-06 (`counter=0x0400 → responds immediately and asks for half-size chunks; SFM was asking for double-size chunks, both with the same-named
streams all frames correctly`). "counter" field, but the value is just an address pointer the device honors as-is.
The 4-3-26 capture confirms the pattern for a second event (key `0111245a`, key4[2:4]=0x245a): **The chunk pattern depends on whether the event sits at start_key=0 or not.**
chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400).
`max(0x245a, 0x0400) = 0x245a` → formula works correctly for non-zero base offset too. #### Event 1 case — start_key[2:4] == 0x0000 (first event after erase / wrap)
```
1. Probe at counter=0x0000 (params[1:5] = full key, returns STRT record)
2. Read 2 fixed metadata pages: counter=0x1002, counter=0x1004
(these are GLOBAL session metadata — read ONCE per
Blastware session, not per event; contain the
Project/Client/User Name/Seis Loc strings)
3. Sample chunks: counter=0x0600, 0x0800, …, by 0x0200 increment,
up to but not including end_offset (rounded down to
0x0200 boundary)
4. TERM frame (see TERM formula below)
```
The reason `0x0046..0x0600` is skipped for event 1 is unknown — likely some pre-event
firmware reserved area for the first slot in a freshly-erased buffer. Harmless to skip.
#### Event 2+ case — start_key[2:4] != 0x0000 (continuation events)
```
1. First chunk at counter = start_key[2:4] + 0x0046 (this IS the probe — response
contains STRT)
2. Sample chunks: counter += 0x0200 each, up to but
not including end_offset
3. TERM frame
```
No metadata pages — those have already been read during event 1 in the same Blastware
session, and BW caches them. Note that the metadata-page reads happen ONCE per
Blastware-session-on-the-device, not once per event, so an SFM session that downloads
several events should read 0x1002/0x1004 only once at the start.
#### History (do not re-derive)
**History:**
- Original: `_CHUNK1_COUNTER = 0x1004` hardcoded (Blastware capture artifact — WRONG). - Original: `_CHUNK1_COUNTER = 0x1004` hardcoded (Blastware capture artifact — WRONG).
- 2026-04-06: Corrected to `chunk_num * 0x0400` (worked for key 01110000 only). - 2026-04-06: `chunk_num * 0x0400` (worked for key 01110000 only).
- 2026-04-24: Corrected to `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets, - 2026-04-24: `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets, broke key 01110000).
but accidentally broke key 01110000 — counter=0x0000 sends probe address again). - 2026-04-26: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` (broken — over-read past event end).
- 2026-04-26: Final formula: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400`. - 2026-05-01: Increments are 0x0200 not 0x0400; absolute addresses inside event range; bounded
by STRT end_key, not by `max_chunks` cap or device-side timeout.
### SUB 5A — STRT record encodes end_offset (NEW 2026-05-01)
The first A5 response (probe response, or the first chunk for event 2+) contains a STRT
record at byte offset 17 of the `data` field. Layout:
```
data[17:21] "STRT" magic
data[21:23] ff fe sentinel
data[23:27] end_key ← 4-byte key of where this event ENDS
data[27:31] start_key ← 4-byte key of where this event STARTS
data[31:33] uint16 BE ?? sample-count or total bytes (varies; not yet decoded)
data[33:35] uint16 BE ??
data[35] 0x46 record type (waveform full record)
```
`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**.
SFM must extract this from the first A5 response and use it to bound the chunk loop and
encode the TERM frame. The device will happily respond to chunk requests past `end_offset`
(returning post-event circular-buffer contents) — that's the over-read bug.
Verified across 3 events:
| Capture | start_key | end_key | end_offset | event size |
|---|---|---|---|---|
| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B |
| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B |
| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | `0x417E` (event 2 span 0x1F8C = 8,076 B) |
### SUB 5A — TERM frame formula (FINALIZED 2026-05-01)
The TERM frame fetches the partial last chunk *and* the file footer. It is **not** a simple
"goodbye" frame — its response payload contains the bytes between the last full 0x0200-aligned
chunk and `end_offset`, and is required for reconstructing the Blastware file format.
```
last_chunk_counter = address of last full 0x0200-byte chunk read
next_boundary = last_chunk_counter + 0x0200
TERM offset_word = end_offset - next_boundary
TERM params[0] = key[0] (= 0x01 on every observed device)
TERM params[1] = key[1] (= 0x11)
TERM params[2] = (next_boundary >> 8) & 0xFF
TERM params[3] = next_boundary & 0xFF
TERM params[4:10] = zeros
build_5a_frame(offset_word, params) (10-byte params, NOT 11)
```
The device reconstructs `requested_address = (params[2] << 8) | offset_word = end_offset`
and replies with `(end_offset - next_boundary)` bytes from `next_boundary` — the residual
between the last 0x0200 boundary and the actual event end. Append the TERM response data
to the chunk stream like any other A5 frame; it carries the final waveform tail + footer.
Verified across 3 events:
| end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] |
|---|---|---|---|---|
| `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ |
| `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ |
| `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ |
The previous code's hard-coded `offset_word = 0x005A` and `term_counter = last + 0x0400`
are wrong; the device's response under that path is a tiny 101-byte device-side terminator
(arrived only after we walked the entire post-event buffer), not the proper file footer.
### SUB 5A — fixed metadata pages 0x1002 and 0x1004 (NEW 2026-05-01)
Two chunk addresses are GLOBAL device/session metadata, not event-specific:
- `counter=0x1002` — first metadata page
- `counter=0x1004` — second metadata page
These are at fixed absolute addresses in the device's flash buffer. They contain the
session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII
strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the
new walk these strings come from the dedicated metadata pages, not from the sample-chunk
stream.
BW reads them ONCE per Blastware session (during event 1's download) and caches them.
For SFM, that means:
- Once per call-home / once per `MiniMateClient.connect()` is enough.
- Subsequent events in the same session don't need to re-fetch them.
- Their content does not change when iterating events; only when the user opens
Compliance Setup → Apply on the device or sends a SUB 71 compliance write.
The contents have not been byte-for-byte decoded yet — first task on the implementation
side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the
strings we currently extract from A5[7].
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination ### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
@@ -148,10 +275,16 @@ chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400).
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes. confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
Do not swap them. Do not swap them.
### SUB 5A — event-time metadata lives in A5 frame 7 ### SUB 5A — event-time metadata source (UPDATED 2026-05-01)
The bulk stream sends 9+ A5 response frames. Frame 7 (0-indexed) contains the compliance > **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A
setup as it existed when the event was recorded: > bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at
> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because
> the broken counter formula was scanning the wrong region.
>
> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and
> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A
> sample-chunk stream itself does NOT contain these strings any more under the new walk.
``` ```
"Project:" → project description "Project:" → project description
@@ -171,26 +304,37 @@ used as the authoritative source. `_decode_a5_metadata_into` therefore only set
"Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C "Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C
record — 5A remains the sole source for those fields and they are set unconditionally. record — 5A remains the sole source for those fields and they are set unconditionally.
`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears, > ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and
then sends the termination frame. > stops one chunk later) is a workaround for the missing end_offset bound — when the new
> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is
> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM
> frame.
### SUB 5A — end-of-stream signal (confirmed 2026-04-06) ### SUB 5A — end-of-stream — UPDATED 2026-05-01
After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to > **Previous understanding (now known to be a symptom, not a feature):** "After streaming
the next chunk request, then goes silent. This is the natural end-of-stream indicator — NOT > all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was
a complete A5 frame. `S3FrameParser.bytes_fed` will be 1; no frame is assembled. > not the device's natural end-of-event signal — it was the device's response when SFM had
> walked clean off the end of the addressable buffer region after over-reading by ~5×.
> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the
> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`,
> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser.
Handling: on `TimeoutError`, if `bytes_fed > 0` AND frames were already collected, treat as The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful
graceful end-of-stream, break the loop, and proceed to the termination frame. If `bytes_fed defence-in-depth fallback for malformed events or unexpected device states, but should not
== 0` with no prior frames, it is a genuine transport failure — re-raise. be the primary loop-exit condition.
**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each. **Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call
in the chunk loop passes `timeout=10.0` explicitly. in the chunk loop passes `timeout=10.0` explicitly.
**Typical chunk count (BE11529, 1024 sps):** A 9,306-sample event produces 35 chunks before **Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):**
end-of-stream. Chunks with uniform 1,036-byte data are all-zero ADC samples (post-event A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames.
silence). Only the initial variable-size chunks contain actual signal. A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames.
An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames.
Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks
17-37 containing post-event circular-buffer garbage that corrupted the file body.
### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06) ### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06)
@@ -303,6 +447,55 @@ sends token=0xFE and is NOT used by any caller.
`advance_event()` returns `(key4, event_data8)`. `advance_event()` returns `(key4, event_data8)`.
Callers (`count_events`, `get_events`) loop while `data8[4:8] != b"\x00\x00\x00\x00"`. Callers (`count_events`, `get_events`) loop while `data8[4:8] != b"\x00\x00\x00\x00"`.
### SUB 0A — WAVEHDR response length distinguishes events from boundaries (NEW 2026-05-01)
When iterating events with the "Download All" pattern (1E → 0A → 1F → 0A → 1F → …), the
DATA_LENGTH at `data_rsp.data[5]` (= the byte BW echoes back as the offset for the data
fetch step) takes one of two values:
| WAVEHDR offset | Meaning |
|---|---|
| `0x46` (= 70) | Real event start key — there is event data at this address |
| `0x2C` (= 44) | Boundary marker between events — this key is the END of the previous event AND the START key for the empty space after it (or is the next event's pre-header) |
Confirmed from the 5-1-26 "Download All" capture:
```
0A(key=01110000) → off=0x46 ← event 1 real start
1F → key=011121F2
0A(key=011121F2) → off=0x2C ← event 1 END / event 2 boundary
1F → key=01112238
0A(key=01112238) → off=0x46 ← event 2 real start (= boundary + 0x46)
1F → key=0111417E
0A(key=0111417E) → off=0x2C ← event 2 END / next-empty marker
1F → null sentinel
```
This is why event 2's first 5A chunk is at `start_key + 0x46` — that's the address of the
"real start" 0x46-record, distinct from the `0x2C`-record at the raw boundary. Use the
`0x46` keys as the input to `read_bulk_waveform_stream`, not the `0x2C` keys.
For event 1 only (start_key[2:4] = 0x0000) BW probes at counter=0x0000 directly, which is
the `0x46`-keyed start record. Subsequent events use `start_key + 0x46`.
**Practical iteration pattern (replaces the old 1E/1F walk for downloads):**
```
Setup: SERIAL × 2 → CHCFG → 1E (token=0x00) → key0
For each event:
0A(cur_key) → DATA_LENGTH = 0x46 (real) or 0x2C (boundary)
1F (token=0x00) → next_key
if length was 0x46: → cur_key is a real event; queue it for download
cur_key = next_key
if next_key all-zero null sentinel: stop
Then for each queued real-event key:
download_event(key) → 5A bulk stream with STRT-bounded chunk walk
```
This is what BW does in the 5-1-26 "Download All" capture — it walks the full event chain
collecting `(key, length)` tuples first, *then* downloads each event using the `0x46` keys.
### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce) ### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce)
`read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where: `read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where:
@@ -527,6 +720,8 @@ All DB endpoints are read-only except `PATCH /db/events/{id}/false_trigger`.
| 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture | | 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture |
| 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence; only 1 event stored so token=0xFE appeared to work | | 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence; only 1 event stored so token=0xFE appeared to work |
| 4-3-26 | `bridges/captures/4-3-26/` | Browse-mode S3 capture with 2+ events — confirmed all-zero params for 1F, 1F response layout, null sentinel, 0A context requirement | | 4-3-26 | `bridges/captures/4-3-26/` | Browse-mode S3 capture with 2+ events — confirmed all-zero params for 1F, 1F response layout, null sentinel, 0A context requirement |
| 4-27-26 | `bridges/captures/4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof that SFM was over-reading 5× past event end. BW reads 14 chunks at 0x0200 increments + TERM at end_offset; SFM was reading 37 chunks at 0x0400 increments. STRT end_key field located. |
| 5-1-26 | `bridges/captures/5-1-26/comcheck/` | Three sub-captures: SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945`/`_171216`). Confirmed: TERM frame formula across 3 events; metadata pages 0x1002/0x1004 are global (read once per session); event-1 vs event-N chunk-pattern split; WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundaries. |
--- ---
+121 -40
View File
@@ -1,4 +1,4 @@
# seismo-relay `v0.12.1` # seismo-relay `v0.12.6`
A ground-up replacement for **Blastware** — Instantel's aging Windows-only A ground-up replacement for **Blastware** — Instantel's aging Windows-only
software for managing MiniMate Plus seismographs. software for managing MiniMate Plus seismographs.
@@ -18,26 +18,27 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
``` ```
seismo-relay/ seismo-relay/
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Console tabs) ├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Download + Console tabs)
├── minimateplus/ ← MiniMate Plus client library ├── minimateplus/ ← MiniMate Plus client library
│ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport │ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport
│ ├── protocol.py ← DLE frame layer, SUB command dispatch │ ├── protocol.py ← DLE frame layer, SUB command dispatch
│ ├── client.py ← High-level client (connect, get_events, push_config, …) │ ├── client.py ← High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …)
│ ├── framing.py ← Frame builders, DLE codec, S3FrameParser │ ├── framing.py ← Frame builders, DLE codec, S3FrameParser
── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, … ── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig,
│ └── blastware_file.py ← Write events to Blastware-compatible .AB0 files
├── sfm/ ← SFM REST API server (FastAPI, port 8200) ├── sfm/ ← SFM REST API server (FastAPI, port 8200)
│ ├── server.py ← All device + DB endpoints │ ├── server.py ← Live device endpoints + DB query endpoints + caching
│ ├── database.py ← SeismoDb — SQLite persistence layer │ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions, sessions table)
│ └── sfm_webapp.html ← Embedded web UI (served at /) │ └── sfm_webapp.html ← Embedded web UI with Call Home config tab
├── bridges/ ├── bridges/
│ ├── ach_server.py ← Inbound ACH call-home server (main production server) │ ├── ach_server.py ← Inbound ACH call-home server (main production server)
│ ├── ach_mitm.py ← Transparent MITM proxy for capturing BW sessions │ ├── ach_mitm.py ← Transparent MITM proxy for capturing BW sessions
│ ├── s3-bridge/ ← RS-232 serial bridge (capture tool) │ ├── s3-bridge/ ← RS-232 serial bridge (capture tool)
│ ├── tcp_serial_bridge.py ← Local TCP↔serial bridge (bench testing) │ ├── tcp_serial_bridge.py ← Local TCP↔serial bridge (bench testing)
│ ├── gui_bridge.py ← Standalone bridge GUI │ ├── gui_bridge.py ← Standalone bridge GUI with raw capture checkboxes
│ └── raw_capture.py ← Simple raw capture tool │ └── raw_capture.py ← Simple raw capture tool
├── parsers/ ├── parsers/
@@ -101,21 +102,28 @@ python seismo_lab.py
Each call dials the device, does its work, and closes the connection. TCP Each call dials the device, does its work, and closes the connection. TCP
connections are retried once on `ProtocolError` to handle cold-boot timing. connections are retried once on `ProtocolError` to handle cold-boot timing.
**Caching** — frequently-polled endpoints are cached in-process to avoid **In-memory caching** — frequently-polled endpoints avoid redundant TCP round-trips
redundant TCP round-trips: via a thread-safe `_LiveCache` (plain Python dict + `threading.Lock`):
| Method | URL | Cache | | Method | URL | Cache Strategy |
|--------|-----|-------| |--------|-----|---|
| `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` | | `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` |
| `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected | | `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected |
| `GET` | `/device/event/{idx}/waveform` | Permanent per event index | | `GET` | `/device/event/{idx}/waveform` | Permanent per event index |
| `GET` | `/device/monitor/status` | 30-second TTL | | `GET` | `/device/monitor/status` | 30-second TTL; invalidated by monitor start/stop |
| `GET` | `/device/call_home` | Fresh read from device (not cached) |
| `POST` | `/device/connect` | — | | `POST` | `/device/connect` | — |
| `POST` | `/device/config` | Writes compliance config; invalidates cache | | `POST` | `/device/config` | Writes compliance config; invalidates info + events cache |
| `POST` | `/device/monitor/start` | Sends SUB 0x96 | | `POST` | `/device/config/project` | Patches project/client/operator/sensor_location strings |
| `POST` | `/device/monitor/stop` | Sends SUB 0x97 | | `POST` | `/device/monitor/start` | Sends SUB 0x96; immediately evicts status cache |
| `POST` | `/device/monitor/stop` | Sends SUB 0x97; immediately evicts status cache |
| `POST` | `/device/call_home` | Reads, patches specified fields, writes back to device |
All cached endpoints accept `?force=true` to bypass the cache. **Cache bypass**All cached endpoints accept `?force=true` to skip the cache and
force a fresh read from the device.
**Cache stats**`GET /cache/stats` returns hit/miss counts and TTL info; `DELETE /cache/device`
clears the device cache immediately.
Transport query params (supply one set): Transport query params (supply one set):
``` ```
@@ -152,22 +160,34 @@ client = MiniMateClient(transport=TcpTransport("1.2.3.4", 12345), timeout=30.0)
with client: with client:
# Read # Read
info = client.connect() # DeviceInfo — serial, firmware, compliance config info = client.connect() # DeviceInfo — serial, firmware, compliance config
count = client.count_events() # Number of stored events count = client.count_events() # Number of stored events
keys = client.list_event_keys() # Fast browse walk — event keys only, no download keys = client.list_event_keys() # Fast browse walk — event keys only, no download
events = client.get_events() # Full download: headers + peaks + metadata events = client.get_events() # Full download: headers + peaks + metadata
monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag
log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records) log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records)
ach_cfg = client.get_call_home_config() # Auto Call Home settings (SUB 0x2C)
# Write # Write
client.apply_config( client.apply_config(
sample_rate=1024, sample_rate=1024,
recording_mode="Continuous", # Single Shot / Continuous / Histogram / Histogram+Continuous
histogram_interval_sec=15, # 2, 5, 15, 60, 300, 900
trigger_level_geo=0.5, trigger_level_geo=0.5,
geo_range="Normal", # Normal (10.000 in/s) / Sensitive (1.25 in/s)
project="Bridge Inspection 2026", project="Bridge Inspection 2026",
client_name="City of Portland", client_name="City of Portland",
operator="B. Harrison", operator="B. Harrison",
) )
client.set_call_home_config(
auto_call_home_enabled=True,
after_event_recorded=True,
at_specified_times=True,
time1_hour=18, time1_min=30, # 6:30 PM
time2_hour=6, time2_min=0, # 6:00 AM
)
# Control # Control
client.start_monitoring() # SUB 0x96 client.start_monitoring() # SUB 0x96
client.stop_monitoring() # SUB 0x97 client.stop_monitoring() # SUB 0x97
@@ -182,18 +202,20 @@ existed at record time — not backfilled from the current compliance config.
## Database ## Database
`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode). `ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode) using the
Three tables, all unit-keyed by serial number: `SeismoDb` persistence layer. Four tables, all unit-keyed by serial number:
| Table | Key | Contents | | Table | Key | Contents |
|-------|-----|----------| |-------|-----|----------|
| `ach_sessions` | UUID | Per-call-home audit record: serial, peer IP, events_downloaded, duration | | `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds |
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, PPV per channel, project/client/operator strings, false_trigger flag | | `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag |
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: start/stop time, duration, geo threshold | | `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
| `events.false_trigger` | Boolean flag | PATCH endpoint to mark/unmark false triggers for review |
Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs never
never produce duplicate rows. Post-erase key reuse is handled automatically produce duplicate rows. Post-erase key reuse is handled automatically via the
via the high-water mark in `ach_state.json`. high-water mark in `ach_state.json`. Key-based state tracking allows correct
handling of device erasures (external or post-download).
--- ---
@@ -231,6 +253,27 @@ Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/insta
--- ---
## Compliance Config Features (v0.12.2v0.12.3)
The REST API and web UI expose full control over device compliance settings:
- **Recording Mode** (Single Shot / Continuous / Histogram / Histogram+Continuous)
- **Sample Rate** (1024 / 2048 / 4096 sps)
- **Record Time** (float, seconds)
- **Histogram Interval** (2s, 5s, 15s, 1m, 5m, 15m) — when recording mode includes histogram
- **Geo Trigger Levels** (float, in/s per channel)
- **Geo Maximum Range** (Normal 10.000 in/s / Sensitive 1.250 in/s per channel)
- **Project / Client / Operator / Sensor Location** (ASCII strings)
Auto Call Home config:
- **Auto Call Home Enable** (bool)
- **Dial String** (read-only; 40-byte ASCII)
- **Trigger on Event** (bool)
- **Scheduled Call-Ins** (two time slots with HH:MM each)
- **Retry Settings** (count, delay, connection timeout, warm-up time)
---
## Requirements ## Requirements
```bash ```bash
@@ -252,17 +295,55 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
--- ---
## Roadmap ## Key Features (v0.10v0.12)
**Device support (v0.12.5):**
- [x] Full read/write/erase pipelines
- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings)
- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries)
- [x] Monitor control (start/stop, status polling, battery/memory)
- [x] Monitor log entries (continuous monitoring intervals without full waveform download)
**Data persistence (v0.11):**
- [x] SQLite database (`seismo_relay.db`) with 4 tables: ach_sessions, events, monitor_log, plus false_trigger flag
- [x] Deduplication by waveform key (handles re-runs and repeat call-homes)
- [x] Post-erase key-reuse detection (tracks high-water mark)
- [x] Session state (`ach_state.json`) with downloaded keys and max key
**REST API (v0.12.1):**
- [x] Live device endpoints with in-memory caching (`_LiveCache`)
- [x] Cache statistics (`/cache/stats`) and manual invalidation (`/cache/device`)
- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH)
- [x] Call Home config read/write endpoints
- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`)
**File output (v0.7+):**
- [x] Blastware-compatible `.AB0` file generation (waveform + metadata)
- [x] Multi-channel waveform decode from SUB 5A bulk stream
- [x] Second-resolution timestamp encoding in Blastware filename
**Capture tools (v0.12.5):**
- [x] Serial-to-TCP bridge with raw BW/S3 capture (s3_bridge.py, defaults to auto-capture)
- [x] GUI bridge with raw capture checkboxes (gui_bridge.py)
- [x] ACH inbound server with bidirectional capture (ach_server.py saves raw_tx + raw_rx)
- [x] Transparent TCP MITM proxy for live BW session capture (ach_mitm.py)
**Analysis tools:**
- [x] s3_analyzer.py — session parser, frame differ, Claude export
- [x] gui_analyzer.py — standalone analyzer GUI
- [x] frame_db.py — SQLite frame database for capture analysis
**seismo_lab.py GUI (v0.12.5):**
- [x] Bridge tab — Serial/TCP mode selector with raw capture options
- [x] Analyzer tab — BW/S3 capture playback and differencing
- [x] Download tab — Live wire-byte capture during event download (new v0.12.5)
- [x] Console tab — Logging and diagnostics
## Roadmap (Future)
- [x] Full read pipeline — device info, compliance config, event download with true event-time metadata
- [x] Write commands — push compliance config, trigger thresholds, project strings to device
- [x] Erase all events — confirmed erase sequence from live MITM capture
- [x] Monitor control — start/stop monitoring, read battery/memory/status
- [x] Monitor log entries — decode partial 0x2C records (continuous monitoring intervals)
- [x] ACH inbound server — accept call-home connections, download events, dedup by key
- [x] SQLite persistence — events, monitor log, and session history in `seismo_relay.db`
- [x] SFM REST API — device control + DB query endpoints, live device cache
- [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing - [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing
- [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first) - [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first)
- [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object - [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object
- [ ] Modem manager — push RV50/RV55 configs via Sierra Wireless API - [ ] Modem manager — push RV50/RV55 configs via Sierra Wireless API
- [ ] Histogram mode recording support (5A stream analysis for mode 0x03)
- [ ] Call Home dial_string write support (requires DLE escaping for embedded control characters)
+225 -50
View File
@@ -110,6 +110,7 @@
| 2026-04-21 | §7.6.2, §5.3 | **CORRECTED — compliance_raw contains wire-encoded bytes, NOT logical bytes.** S3FrameParser appends DLE+ETX inner-frame pairs as two literal bytes to the frame body. Any `0x03` values in the compliance config appear in `compliance_raw` as `\x10\x03` (two bytes), not as a single `0x03`. The previous claim "S3FrameParser handles this transparently so compliance_raw contains logical (destuffed) bytes" was wrong. Consequence: `compliance_raw` is the wire-encoded E5 payload; anchor-relative reads work correctly because the anchor position automatically accounts for any DLE-encoded bytes before it. For write-back, round-tripping `compliance_raw` verbatim sends the correct wire bytes to the device. **DLE ETX escaping in write frames:** Blastware escapes `0x03` bytes in write frame data as `\x10\x03` on wire; our `build_bw_write_frame` does not (writes data raw). Device is confirmed to accept raw writes for all tested modes — likely uses the offset/length field for write frame framing, not ETX scanning. | | 2026-04-21 | §7.6.2, §5.3 | **CORRECTED — compliance_raw contains wire-encoded bytes, NOT logical bytes.** S3FrameParser appends DLE+ETX inner-frame pairs as two literal bytes to the frame body. Any `0x03` values in the compliance config appear in `compliance_raw` as `\x10\x03` (two bytes), not as a single `0x03`. The previous claim "S3FrameParser handles this transparently so compliance_raw contains logical (destuffed) bytes" was wrong. Consequence: `compliance_raw` is the wire-encoded E5 payload; anchor-relative reads work correctly because the anchor position automatically accounts for any DLE-encoded bytes before it. For write-back, round-tripping `compliance_raw` verbatim sends the correct wire bytes to the device. **DLE ETX escaping in write frames:** Blastware escapes `0x03` bytes in write frame data as `\x10\x03` on wire; our `build_bw_write_frame` does not (writes data raw). Device is confirmed to accept raw writes for all tested modes — likely uses the offset/length field for write frame framing, not ETX scanning. |
| 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. | | 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. |
| 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. | | 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. |
| 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. |
--- ---
@@ -1226,7 +1227,24 @@ Two critical differences from `build_bw_frame`:
2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen, 2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen,
only `XX` is added to the running sum; lone bytes are added normally. only `XX` is added to the running sum; lone bytes are added normally.
#### 7.8.2 Request Sequence #### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol)
> ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new
> BW MITM captures (4-27-26 + 5-1-26) prove the actual chunk increment is **0x0200**, the
> chunk loop is bounded by `end_offset` from the STRT record (not by chunk count or by a
> device-side timeout), and the TERM frame's `offset_word=0x005A` magic is incorrect — the
> real TERM offset_word is computed from `end_offset` and the last chunk address. Under the
> deprecated formula SFM over-reads roughly 5× past the actual event end into post-event
> circular-buffer garbage, corrupting reconstructed Blastware files for any waveform ≥ 2 sec.
>
> The whole "stop_after_metadata + one extra chunk + 0e 08 footer" workaround in this
> section was compensating for the missing end_offset bound. It is obsoleted by the
> STRT-bounded walk in §7.8.5.
>
> **Read this section for historical context only.** For the correct protocol, jump to:
> - §7.8.5 — chunk addressing and the STRT end_offset
> - §7.8.6 — TERM frame formula
> - §7.8.7 — fixed metadata pages 0x1002 and 0x1004
| Frame | offset_word | counter | params | Purpose | | Frame | offset_word | counter | params | Purpose |
|---|---|---|---|---| |---|---|---|---|---|
@@ -1237,45 +1255,17 @@ Two critical differences from `build_bw_frame`:
| … | … | … | … | … | | … | … | … | … | … |
| Termination | `0x005A` | `max(key4[2:4], 0x0400) + N * 0x0400` | 10 bytes | End transfer | | Termination | `0x005A` | `max(key4[2:4], 0x0400) + N * 0x0400` | 10 bytes | End transfer |
> ⚠️ **2026-04-06 CORRECTED — chunk counter is `key4[2:4] + (N-1) * 0x0400`.** > Historical correction notes (left in place to deter re-derivation of the same wrong formula):
> The 4-2-26 BW TX capture showed counter=0x1004 for chunk 1 of key `01110000`, leading to > the table above was the result of three iterative "corrections" between 2026-04-06 and
> an interim "monotonic n * 0x0400" formula. This was accidentally correct because > 2026-04-26 that progressively narrowed in on the wrong answer because every test was on
> `key4[2:4] == 0x0000` for that event. > events with `key4[2:4]=0` and the device responds to whatever counter you ask for. The
> > 5-1-26 captures with a non-zero start_key event (`01112238`) finally exposed the bug.
> **2026-04-24 CORRECTION:** The counter is an absolute circular-buffer address.
> BW's true formula is `key4[2:4] + (chunk_num - 1) * 0x0400` where `key4[2:4]` is the
> event's storage base offset (`(key4[2]<<8) | key4[3]`). For keys where
> `key4[2:4] != 0x0000` (e.g. key `01111884`), using `n * 0x0400` sends requests into the
> wrong buffer region — the device returns data from a completely different event.
>
> **2026-04-26 FINAL CORRECTION:** The formula `key4[2:4] + (N-1) * 0x0400` is wrong when
> `key4[2:4] == 0x0000` (e.g. event key `01110000`, the very first event after a device erase).
> Counter=0x0000 for chunk 1 is the same address as the probe frame — the device re-returns
> the STRT record data instead of waveform payload (frame 1 has len=1097, same as probe, and
> contains `b"STRT\xff\xfe"`, contributing zero waveform bytes).
> Final formula: `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400`.
> For key `01110000`: chunk 1 = 0x0400 (confirmed working, empirical test 2026-04-06).
> For key `0111245a`: chunk 1 = 0x245a (unchanged, confirmed from 4-3-26 capture).
The `stop_after_metadata=True` flag causes the loop to stop as soon as `b"Project:"` is The `stop_after_metadata=True` flag (deprecated as a primary loop-exit) scanned for
found in the accumulated A5 frame data, typically after 49 chunks. A termination frame `b"Project:"` in the chunk stream because the metadata strings happened to be reachable
is always sent before returning. when the broken 0x0400-step walk passed the global metadata pages at 0x1002/0x1004. Under
the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004,
**IMPORTANT — one extra chunk required after "Project:" for valid file footer (confirmed 2026-04-23):** not from the sample-chunk stream — see §7.8.7.
When writing a Blastware-compatible waveform file, stopping immediately at "Project:" and
sending termination produces an empty termination response with no footer bytes (`0e 08`
marker missing). Blastware downloads exactly **one more chunk** after finding "Project:"
before sending termination — that extra chunk primes the device to return valid footer
bytes (monitoring start/stop timestamps) in the termination response.
`read_bulk_waveform_stream(stop_after_metadata=True)` implements this: after the "Project:"
chunk is received, one additional chunk is requested before breaking. The termination
response (`include_terminator=True`) then contains the correct `0e 08` footer.
**do NOT use `full_waveform=True` for Blastware file writing** — for events with long
post-event silence (35 chunks), the silence chunks contain embedded device-internal
pointer structures that produce spurious STRT markers in the file body. Blastware only
downloads 45 chunks (metadata + one signal chunk) regardless of event length.
#### 7.8.3 A5 Frame Layout #### 7.8.3 A5 Frame Layout
@@ -1293,15 +1283,19 @@ for ASCII labels with a null-terminated value read:
All five fields reflect the **setup at event-record time**, not the current device config. All five fields reflect the **setup at event-record time**, not the current device config.
#### 7.8.4 End-of-Stream Behaviour and Chunk Timing #### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01
> ✅ **Confirmed 2026-04-06** — empirical observation on BE11529 (S338.17) over TCP/cellular. > The "1 raw byte then silence" pattern documented below was originally interpreted as
> "the device's natural end-of-event signal." The 5-1-26 captures show this is actually
> the device's response when the requester has walked **past** the addressable buffer
> region (i.e. ~5× past the actual event end under the deprecated 0x0400-step walk).
> Under the corrected STRT-bounded walk (§7.8.5), the stream ends cleanly with the TERM
> frame's response — no timeout, no 1-byte teaser. The fallback below remains useful as
> defensive handling for malformed events but should not be the primary loop-exit.
**End-of-stream signal:** After sending all waveform chunks, the device sends exactly **1 raw byte** in response to the next chunk request, then goes silent. This byte is not a complete DLE-framed A5 response — `S3FrameParser.bytes_fed` reports 1 and no frame is ever assembled. This is the device's natural end-of-stream indicator. **Defensive fallback handling in `read_bulk_waveform_stream`:**
Handling logic in `read_bulk_waveform_stream`:
``` ```
TimeoutError caught: TimeoutError caught (rare under corrected walk):
if bytes_fed > 0 AND frames already collected: if bytes_fed > 0 AND frames already collected:
→ graceful end-of-stream; break loop; proceed to termination frame → graceful end-of-stream; break loop; proceed to termination frame
else (bytes_fed == 0, no prior frames): else (bytes_fed == 0, no prior frames):
@@ -1313,14 +1307,15 @@ TimeoutError caught:
| Metric | Observed value | | Metric | Observed value |
|---|---| |---|---|
| Chunk response time | ~1 s per chunk | | Chunk response time | ~1 s per chunk |
| Chunks for a 9,306-sample event | 35 chunks | | Chunks for a 2-sec event (corrected walk) | 14 (12 sample chunks + 2 metadata pages) + TERM |
| Data per chunk (active signal) | 1,0361,123 bytes | | Chunks for a 3-sec event (corrected walk) | 18 (16 sample chunks + 2 metadata pages) + TERM |
| Data per chunk (post-event silence) | 1,036 bytes (uniform) | | Chunks for a continuation event (corrected walk) | ~15 sample chunks + TERM (no metadata reread) |
| Chunks under deprecated walk for 2-3 sec event | 37 (over-reads ~5×) |
| Data per chunk (corrected, 0x0200 size) | ~540575 bytes wire (= 0x0200 payload + framing) |
| Data per chunk (deprecated 0x0400 step) | 1,0361,123 bytes wire (= 0x0400 payload + framing) |
| Safe recv timeout per chunk | **10 s** (10× typical) | | Safe recv timeout per chunk | **10 s** (10× typical) |
| Default transport timeout | 120 s → ~2-min stall at end-of-stream | | Default transport timeout | 120 s → ~2-min stall at end-of-stream |
Chunks with uniform 1,036-byte payload (chunks 1735 in the observed event) contain all-zero ADC samples — the device continues recording silence until the configured record time expires before terminating the stream.
**ADC count-to-physical conversion — ✅ CONFIRMED 2026-04-17:** **ADC count-to-physical conversion — ✅ CONFIRMED 2026-04-17:**
Raw samples are signed 16-bit integers (32,768 to +32,767). Source: Interface Handbook §4.5. Raw samples are signed 16-bit integers (32,768 to +32,767). Source: Interface Handbook §4.5.
@@ -1339,6 +1334,186 @@ where `geo_range = 1.61133 V × 6.206053 = 10.000 in/s` is the Normal (Gain=1) f
`_decode_a5_waveform()` contains `elif fi == 9: continue` from an earlier assumption that frame index 9 is always the device terminator. For streams with more than 9 frames, frame 9 is live waveform data. The skip discards ~1,070 bytes (~133 sample-sets) per event. Terminator detection should use `page_key == 0x0000`, not frame index. This skip should be removed. `_decode_a5_waveform()` contains `elif fi == 9: continue` from an earlier assumption that frame index 9 is always the device terminator. For streams with more than 9 frames, frame 9 is live waveform data. The skip discards ~1,070 bytes (~133 sample-sets) per event. Terminator detection should use `page_key == 0x0000`, not frame index. This skip should be removed.
#### 7.8.5 Chunk addressing and the STRT end_offset (NEW 2026-05-01) ✅
> ✅ Confirmed across 3 events (4-27-26 + 5-1-26 captures).
`params[0]` is always `0x00`. `params[1:5]` is a 4-byte absolute device flash-buffer
address — equivalently, "the key of the page being requested." The device returns 0x0200
(= 512) bytes starting at that address. Increments between consecutive sample chunks are
**0x0200, NOT 0x0400** (the previous 0x0400 figure was a Blastware-side artifact / our
implementation's bug — see §7.8.2).
##### STRT record (data layout in the first A5 response)
The first A5 response (the probe response, or the first chunk for continuation events)
contains a **STRT record** at byte offset 17 of `data`:
```
data[ 0:14] echoes request: [chunk_size_hi=0x02 / 0x04 ...] [00] [01 11] [counter_hi counter_lo] [00 × 8] [00 12]
data[14:17] 10 03 00 ← inner DLE+ETX frame separator (preserved literally)
data[17:21] "STRT" ← magic
data[21:23] ff fe ← sentinel
data[23:27] end_key ← 4-byte key of where this event ENDS
data[27:31] start_key ← 4-byte key of where this event STARTS
data[31:33] uint16 BE ← ?? sample count or byte count, varies (not yet decoded)
data[33:35] uint16 BE ← ??
data[35] 0x46 ← record type marker (waveform full record)
data[36:] additional pointers / first sample bytes — content varies by event
```
`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**.
Use it to bound the chunk loop and to compute the TERM frame.
##### Chunk pattern by event location in buffer
**Event 1 / start_key[2:4] = 0x0000** (first event after erase or wrap):
```
1. Probe at counter = 0x0000 (params[1:5] = full key)
2. Read fixed metadata pages counter = 0x1002, then 0x1004
3. Walk sample chunks counter = 0x0600, 0x0800, …, by 0x0200,
up to but not including end_offset & 0xFE00
4. TERM (see §7.8.6)
```
The range `[0x0046, 0x0600)` is skipped — likely some pre-event firmware-reserved area for
the first slot in a freshly-erased buffer. Harmless to skip; BW does the same.
**Event 2+ / start_key[2:4] != 0x0000** (continuation events in a populated buffer):
```
1. First chunk at counter = start_key[2:4] + 0x0046 ← acts as both probe and first
sample chunk; response carries STRT
2. Walk sample chunks counter += 0x0200 each
3. TERM
```
**No metadata-page reads.** Pages 0x1002/0x1004 are session-global and were already read
during event 1 in the same Blastware session. In SFM, treat metadata pages as a once-
per-`MiniMateClient.connect()` (or once-per-call-home) read, not per-event.
##### Verified end_offset values
| Capture | start_key | end_key | end_offset | event size | sample-chunk start |
|---|---|---|---|---|---|
| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B | 0x0600 (event-1 case) |
| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B | 0x0600 (event-1 case) |
| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | event 2 size = 0x1F8C = 8,076 B | 0x2238 (= 0x21F2 + 0x46) |
#### 7.8.6 TERM Frame Formula (NEW 2026-05-01) ✅
> ✅ Confirmed across 3 events. Replaces the deprecated `offset_word=0x005A` / `params[2] = key4[2]` formula in §7.8.2.
The TERM frame fetches the partial last chunk and the file footer. Its response payload
contains the bytes between the last full 0x0200-aligned chunk and `end_offset` — typically
20520 B — and is **required for reconstructing the Blastware waveform file**. Append the
TERM response data to the chunk stream like any other A5 frame.
```
last_chunk_counter = address of last full 0x0200-byte chunk read
next_boundary = last_chunk_counter + 0x0200
TERM offset_word = end_offset - next_boundary
TERM params[0] = key[0] (= 0x01 on every observed device)
TERM params[1] = key[1] (= 0x11)
TERM params[2] = (next_boundary >> 8) & 0xFF
TERM params[3] = next_boundary & 0xFF
TERM params[4:10] = zeros ← 10-byte params (not 11)
Frame = build_5a_frame(offset_word, params)
```
The device receives `requested_address = (params[2] << 8) | offset_word` (where offset_word
contains both `offset_hi` and `offset_lo` of the 5A frame, with the high bit of offset_hi
being effectively `bit 0 of (end_offset >> 8)`). It reconstructs `end_offset` and replies
with `(end_offset - next_boundary)` bytes of waveform tail starting at `next_boundary`.
##### Verification
| Event | end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] | TERM response size |
|---|---|---|---|---|---|---|
| 2-sec | `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ | 208 B |
| 3-sec | `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ | 520 B |
| Event-2 | `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ | (not measured directly; same pattern) |
Equivalent way to write the formula:
- `offset_word = end_offset & 0x01FF` — low 9 bits of end_offset
- `params[2:4] = (end_offset & 0xFE00) BE` — high 7 bits of end_offset, low byte zeroed
(The two forms are arithmetically identical to `end_offset - next_boundary` and
`next_boundary` because `next_boundary = end_offset & 0xFE00` whenever the chunk loop
stopped at the last full 0x0200 boundary below end_offset.)
#### 7.8.7 Fixed Metadata Pages 0x1002 / 0x1004 (NEW 2026-05-01) 🔶
> 🔶 Inferred — observed in BW captures but page contents not yet byte-decoded.
Two chunk addresses are **GLOBAL** device/session metadata, not event-specific:
- `counter = 0x1002` — first metadata page
- `counter = 0x1004` — second metadata page
These are at fixed absolute addresses in the device's flash buffer. They contain the
session-start compliance-setup ASCII strings — **Project**, **Client**, **User Name**,
**Seis Loc**, **Extended Notes** — that under the deprecated 0x0400-step walk used to be
discoverable in the sample-chunk stream as "A5 frame 7" content. Under the corrected
0x0200-step walk these strings come exclusively from the dedicated metadata-page reads,
not from sample chunks.
##### Caching strategy
BW reads them ONCE per Blastware session, during event 1's download, and caches them.
For SFM:
- Read once per `MiniMateClient.connect()` / once per call-home session.
- Subsequent events in the same session don't need to re-fetch them.
- Their content does not change while iterating events. They DO change when the user
applies a new compliance setup (SUB 71 write) — invalidate the cache then.
##### TODO — content layout
The byte-for-byte layout of pages 0x1002 and 0x1004 has not been decoded. First task on
the implementation side: dump both pages from a fresh capture and verify they include all
the strings currently extracted from the deprecated A5 frame 7 of the chunk stream.
Compare to the existing `_decode_a5_metadata_into` parser — same string-search anchors
(`b"Project:"`, `b"Client:"`, `b"User Name:"`, `b"Seis Loc:"`, `b"Extended Notes"`) likely
apply directly.
#### 7.8.8 "Download All" Sequence (NEW 2026-05-01) ✅
> ✅ Confirmed from 5-1-26 "Download All" capture (`raw_*_171216_download_all_2events.bin`).
Before any 5A traffic, BW's "Download All" pre-walks the entire event chain to map keys
and event boundaries:
```
SERIAL × 2 → CHCFG → EVT_KEY (1E, all-zero) → key0
→ WAVEHDR (0A, key0) → off=0x46 (real event start)
→ EVT_NEXT (1F, all-zero) → key1
→ WAVEHDR (0A, key1) → off=0x2C (boundary)
→ EVT_NEXT → key2
→ WAVEHDR (0A, key2) → off=0x46 (real event start)
→ EVT_NEXT → key3
→ WAVEHDR (0A, key3) → off=0x2C (boundary)
→ EVT_NEXT → null sentinel
```
The DATA_LENGTH at `data_rsp.data[5]` (echoed BW offset for the data fetch step)
disambiguates real events from boundary markers:
| WAVEHDR offset | Meaning |
|---|---|
| `0x46` (= 70) | Real event start key — this key has event data behind it |
| `0x2C` (= 44) | Boundary marker — this key is the END of the previous event AND the start of the empty/header gap before the next event |
Pairs: each real event spans `[real_key, next_real_key)` in the buffer. In the example
above: event 1 = `[01110000, 011121F2)`, event 2 = `[01112238, 0111417E)`. Note that the
"end of event 1" key (`011121F2`) is also the "boundary key" that comes BEFORE event 2's
real start key (`01112238`) — they differ by exactly 0x46 bytes (the event header size).
After the pre-walk completes, BW downloads each `0x46`-keyed event in turn using the 5A
bulk stream protocol from §7.8.5. Use the `0x46` keys, not the `0x2C` keys, as input to
`read_bulk_waveform_stream`.
--- ---
## 7.9 Compliance Config Field Inventory (Blastware UI, 2026-04-08) ✅ ## 7.9 Compliance Config Field Inventory (Blastware UI, 2026-04-08) ✅
+1 -1
View File
@@ -214,7 +214,7 @@ class BridgePanel(tk.Frame):
tk.Button(cfg, text="Browse", bg=BG3, fg=FG, relief="flat", cursor="hand2", tk.Button(cfg, text="Browse", bg=BG3, fg=FG, relief="flat", cursor="hand2",
font=MONO, command=self._choose_dir).grid(row=2, column=5, **pad) font=MONO, command=self._choose_dir).grid(row=2, column=5, **pad)
# Row 2: buttons + status # Row 3: buttons + status
btn_row = tk.Frame(self, bg=BG2) btn_row = tk.Frame(self, bg=BG2)
btn_row.pack(side=tk.TOP, fill=tk.X, padx=4, pady=2) btn_row.pack(side=tk.TOP, fill=tk.X, padx=4, pady=2)