Compare commits
30 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 429c6ac87a | |||
| 52c6e7b618 | |||
| 29ebc75656 | |||
| ebfe9877fa | |||
| c914a15e12 | |||
| a27693242d | |||
| eefec0bd64 | |||
| 7444738883 | |||
| 6b76934a04 | |||
| 7b62c790a9 | |||
| b66cc9d075 | |||
| 4ab604eff1 | |||
| e15f1567ef | |||
| bb33ad3837 | |||
| 45e61fbcaf | |||
| d758825c67 | |||
| 0fbb39c21a | |||
| 1ef55521b1 | |||
| 738b39f3cb | |||
| 625b0a4dfc | |||
| b14f31f3b0 | |||
| b9ab368934 | |||
| 9004241846 | |||
| 6861d9ed97 | |||
| 5cd5652560 | |||
| 897ac8a3f3 | |||
| 310fc5986c | |||
| e1150b30aa | |||
| 9bbecea70f | |||
| 4a0c9b6da5 |
+289
@@ -4,8 +4,293 @@ All notable changes to seismo-relay are documented here.
|
||||
|
||||
---
|
||||
|
||||
## v0.14.3 — 2026-05-05
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`build_5a_frame` — DLE-stuffing rule for 0x10 bytes in params (the
|
||||
long-standing >1-sec event 0 "won't open in BW" bug).**
|
||||
|
||||
Previously `build_5a_frame` wrote params bytes RAW with no DLE stuffing,
|
||||
based on the incorrect assumption that the device handled all `0x10`
|
||||
bytes in params literally. It does not. The device's actual de-stuffing
|
||||
rule for the params region is:
|
||||
|
||||
- `10 10` → de-stuffs to `10`
|
||||
- `10 02/03/04` → kept literal (inner-frame markers)
|
||||
- `10 X` for other X → de-stuffs to just `X` (drops the `0x10`)
|
||||
|
||||
When the counter passed in params has `0x10` in the high byte (e.g.
|
||||
counter=`0x1000` produces params bytes `... 10 00 ...`), the device
|
||||
silently corrupts the request to counter=`0x__00` and responds with
|
||||
whatever lives at that wrong address. For counter=0x1000 the wrong
|
||||
address was 0x0000, so the response was a copy of the file header +
|
||||
STRT record. That STRT block then got embedded in the assembled body
|
||||
at file offset `0x1016`, and Blastware refused to open the file
|
||||
(interprets the second STRT as a malformed multi-event file).
|
||||
|
||||
This explains the entire >1-sec event-0 failure pattern:
|
||||
|
||||
- 1-sec events have `end_offset < 0x1000`, so the chunk walk never
|
||||
requests counter `0x10__` and the bug never triggers.
|
||||
- 2-sec / 3-sec / longer events all need a chunk at counter `0x1000`
|
||||
(and longer events also need `0x1200`, `0x1400`, etc., none of which
|
||||
have `0x10` in the high byte except `0x1000`). Just one corrupted
|
||||
response is enough to embed STRT in the body and break the file.
|
||||
|
||||
Verified against BW 5-1-26 "copy 3sec" capture: all 17 5A request
|
||||
frames (probe + 2 metadata pages + 13 sample chunks + TERM) now match
|
||||
BW's wire output **byte-for-byte**, including the doubled `10 10 00`
|
||||
for counter=0x1000.
|
||||
|
||||
### Notes
|
||||
|
||||
- `0x10` bytes in `offset_hi` (the standalone offset field at body[5])
|
||||
are still written RAW — confirmed correct per the 1-2-26 capture.
|
||||
- BW's actual encoding of `10 02` / `10 04` for meta pages 0x1002 /
|
||||
0x1004 is *not* doubled — it relies on the device keeping `10 02`
|
||||
and `10 04` as literal pairs. This is preserved by the fix.
|
||||
|
||||
---
|
||||
|
||||
## v0.14.2 — 2026-05-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`blastware_file.py` — removed harmful "duplicate header+STRT" strip.**
|
||||
The v0.13.x strip logic was matching the byte sequence `00 12 03 00 STRT`
|
||||
in legitimate waveform data — sample chunks at counter `0x1000` and
|
||||
beyond often contain those bytes coincidentally — and zeroing 25 bytes
|
||||
of valid samples per match. This is why event 0 (event-1 case in the
|
||||
protocol) downloads of >1-sec recordings always failed in BW: the strip
|
||||
destroyed real data at body offset `0x1012..0x102B` and propagated
|
||||
alignment differences through the rest of the body. Sub-1-sec events
|
||||
worked because their `end_offset` was below `0x1002`, so no sample
|
||||
chunks landed in the metadata-page region and the strip's needle never
|
||||
matched. Verified fix by re-feeding the BW 5-1-26 "copy 3sec" capture's
|
||||
A5 frames into the file builder: output is now byte-identical to BW's
|
||||
saved `M529LKIQ.G10` reference (8708 bytes, 0 differences).
|
||||
- BW already concatenates frame contributions in stream order without
|
||||
any de-duplication; SFM now does the same.
|
||||
|
||||
---
|
||||
|
||||
## v0.14.0 — 2026-05-02
|
||||
|
||||
### Changed (major rewrite)
|
||||
|
||||
- **`read_bulk_waveform_stream` — STRT-bounded chunk walk.** Replaces the
|
||||
earlier `0x0400`-step / `max(key4[2:4], 0x0400)` chunk-counter formula,
|
||||
which over-read ~5× past the actual event end into post-event circular-
|
||||
buffer garbage. The new walk:
|
||||
|
||||
1. Probe at `counter = start_offset` (event 1: `0x0000`; event N:
|
||||
`cur_key[2:4]`).
|
||||
2. Parse `end_offset` from the STRT record at `data[17]` of the probe
|
||||
response (`end_key[2:4]` field).
|
||||
3. For event 1 only, read the two fixed metadata pages at counter
|
||||
`0x1002` and `0x1004` — these contain the global session-start
|
||||
compliance setup (Project / Client / User Name / Seis Loc /
|
||||
Extended Notes ASCII strings). Continuation events skip these
|
||||
(BW caches them across the session).
|
||||
4. Walk sample chunks at **`0x0200` increments (NOT `0x0400`)**, bounded
|
||||
by `end_offset` — the loop exits when
|
||||
`next_chunk_counter + 0x0200 > end_offset`.
|
||||
5. Send the proper TERM frame (see new `bulk_waveform_term_v2()`) with
|
||||
`offset_word = end_offset - next_boundary` and
|
||||
`params[2:4] = next_boundary BE`. The TERM response carries the
|
||||
partial last chunk + 26-byte file footer.
|
||||
|
||||
- **New helpers:** `bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`
|
||||
and `parse_strt_end_offset(a5_data)` in `minimateplus.framing`.
|
||||
|
||||
- **`stop_after_metadata` / `extra_chunks_after_metadata` kwargs are now
|
||||
no-ops** under the v0.14.x walk. They are retained on the
|
||||
`read_bulk_waveform_stream` signature for backward compatibility but log a
|
||||
DEBUG line when set. The old "scan for `b'Project:'` and stop one chunk
|
||||
later" workaround is obsolete — the loop is deterministically bounded by
|
||||
the STRT-derived `end_offset`.
|
||||
|
||||
- **Project / Client / User Name / Seis Loc string source corrected.**
|
||||
These come from the dedicated metadata pages at counter `0x1002` /
|
||||
`0x1004`, not from "A5 frame 7" of the sample-chunk stream. The
|
||||
earlier "A5 frame 7" claim was an artifact of the broken `0x0400`-step
|
||||
walk where the bad counter formula coincidentally landed sample-chunk
|
||||
fi=7 on top of the 0x1002 metadata page.
|
||||
|
||||
### Verified
|
||||
|
||||
- Three independent BW MITM captures (4-27-26 + 5-1-26 + 5-4-26) confirm
|
||||
the new walk matches BW's behaviour event-for-event.
|
||||
- `end_offset` values verified across 3 events: `0x1ABE` (4-27-26 2-sec),
|
||||
`0x21F2` (5-1-26 3-sec), `0x417E` (5-1-26 event-2).
|
||||
|
||||
### Notes
|
||||
|
||||
- Earlier v0.13.0 / v0.13.1 / v0.13.2 entries describe partial steps along
|
||||
the way (some of the file builder fixes, filename bugs, etc.) that were
|
||||
superseded by the full rewrite. Treat this v0.14.0 entry as the
|
||||
definitive landing point for the corrected SUB 5A protocol.
|
||||
|
||||
---
|
||||
|
||||
## v0.14.1 — 2026-05-04
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`read_bulk_waveform_stream` — event-N probe counter off-by-`0x46`.**
|
||||
Continuation events (start_key[2:4] != 0) were being probed at counter
|
||||
`start_offset + 0x0046` instead of just `start_offset`. In the iteration
|
||||
walk, `cur_key` from 1F is already the off=0x46 WAVEHDR record key, so the
|
||||
earlier formula effectively double-counted the WAVEHDR offset. The probe
|
||||
landed one WAVEHDR past the actual event start, the response no longer
|
||||
contained the STRT record at byte 17, `parse_strt_end_offset` returned
|
||||
`None`, and the chunk loop fell back to the `max_chunks=128` cap — walking
|
||||
~110 chunks of post-event circular-buffer garbage. Verified against the
|
||||
5-1-26 "copy 2nd address" and 5-4-26 BW 2-sec event captures: BW probes
|
||||
counter=`0x2238` with key=`01112238` and STRT is present at byte 17 of
|
||||
the response (end_offset=`0x417E`).
|
||||
- **CLAUDE.md / docs/instantel_protocol_reference.md** — corrected the
|
||||
event-N section to clarify that `start_key` in those formulas is the
|
||||
off=0x46 key, not the off=0x2C boundary key, and removed the spurious
|
||||
`+0x46` from the chunk-walk pseudocode.
|
||||
|
||||
---
|
||||
|
||||
## v0.13.2 — 2026-05-01
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`_extract_record_type` — third 0C-record header format ("short", 8 bytes).**
|
||||
A live SFM download against BE11529 produced files named `M5290000.000`
|
||||
(zero-stamped) because the 0C waveform record's first bytes were
|
||||
`01 05 07 ea ...` — neither the 9-byte single-shot layout (`0x10` at byte 1)
|
||||
nor the 10-byte continuous layout (`0x10` at bytes 0 and 2). Investigation
|
||||
showed this is a third format observed in the wild: an 8-byte header with no
|
||||
marker bytes at all (`[day][month][year_BE:2][unknown][hour][min][sec]`).
|
||||
The detection logic now scans the year (uint16 BE) at byte 2 / byte 3 / byte
|
||||
4 and picks whichever offset returns a sensible year (2015–2050) — each
|
||||
format has the year at a unique position so this disambiguates cleanly.
|
||||
- New format → `event.record_type = "Waveform (Short)"`,
|
||||
`Timestamp.from_short_record()`.
|
||||
- Existing single-shot and continuous parsers unchanged.
|
||||
- The user's event from May 1, 2026 13:21:37 now correctly resolves to a
|
||||
filename like `M529LKIQ.G10` instead of `M5290000.000`.
|
||||
|
||||
### Added
|
||||
|
||||
- `Timestamp.from_short_record(data)` — decodes the 8-byte header.
|
||||
- `_detect_record_format(data)` — internal helper returning
|
||||
`"single_shot" / "continuous" / "short" / None` via year-position scan.
|
||||
|
||||
---
|
||||
|
||||
## v0.13.1 — 2026-05-01
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`_extract_record_type` — Continuous-mode record headers misclassified as Unknown.**
|
||||
In single-shot mode the 0C waveform record's 9-byte header puts the sub_code
|
||||
marker `0x10` at byte 1, with the day at byte 0. In Continuous mode the
|
||||
header is 10 bytes with the marker at byte 0 *and* byte 2, and the day at
|
||||
byte 1. Previous logic only inspected byte 1 and treated any value other
|
||||
than `0x10` / `0x03` as `"Unknown"`, which prevented `event.timestamp` from
|
||||
being populated for any continuous-mode event whose day-of-month wasn't
|
||||
exactly 3 or 16. As a downstream effect, `blastware_filename()` saw
|
||||
`event.timestamp == None`, fell back to `stem="0000"` / `ab="00"`, and
|
||||
produced filenames like `M5290000.000`. Discovered from a live SFM run on
|
||||
BE11529 in continuous mode (day-of-month = 5).
|
||||
Now disambiguates by checking BOTH byte 0 and byte 2: if both are `0x10`,
|
||||
it's the 10-byte continuous header; else if byte 1 is `0x10`, it's the
|
||||
9-byte single-shot header. Day-of-month no longer matters.
|
||||
|
||||
*Superseded by v0.13.2 — the user's actual record uses a third 8-byte format
|
||||
with no `0x10` markers, which v0.13.1 still misclassified.*
|
||||
|
||||
---
|
||||
|
||||
## v0.13.0 — 2026-05-01
|
||||
|
||||
### Fixed
|
||||
|
||||
- **SUB 5A bulk waveform stream — over-read bug for events ≥ 2 sec.**
|
||||
`read_bulk_waveform_stream` was walking the chunk counter past the actual
|
||||
end of the event, picking up post-event circular-buffer garbage that
|
||||
corrupted reconstructed Blastware files for any waveform > ~1 sec. The
|
||||
loop now extracts the event's `end_offset` from the STRT record at
|
||||
`data[23:27]` of the probe response and stops the chunk walk when the next
|
||||
counter would step past it. Verified against three BW MITM captures
|
||||
(4-27-26 + 5-1-26): 2-sec event drops from 37 over-read chunks to 7
|
||||
bounded chunks; 3-sec drops to 9; non-zero-start "event 2" drops to 9.
|
||||
|
||||
### Added
|
||||
|
||||
- `framing.bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)` —
|
||||
computes the corrected SUB 5A TERM frame's `(offset_word, params)` per the
|
||||
formula confirmed across all 3 BW captures. Not yet wired into
|
||||
`read_bulk_waveform_stream` (the legacy TERM is still used to preserve the
|
||||
existing `blastware_file.write_blastware_file` frame-structure expectations);
|
||||
available for the next iteration that switches to BW's 0x0200 chunk step.
|
||||
- `framing.parse_strt_end_offset(a5_data)` — extracts the event-end pointer
|
||||
from the STRT record in an A5 response payload.
|
||||
|
||||
### Documentation
|
||||
|
||||
- **CLAUDE.md and `docs/instantel_protocol_reference.md` extensively
|
||||
rewritten** to reflect the corrected SUB 5A protocol. See:
|
||||
- CLAUDE.md "SUB 5A — chunk counter formula (REWRITTEN 2026-05-01)"
|
||||
- CLAUDE.md "SUB 5A — STRT record encodes end_offset"
|
||||
- CLAUDE.md "SUB 5A — TERM frame formula"
|
||||
- CLAUDE.md "SUB 5A — fixed metadata pages 0x1002 and 0x1004"
|
||||
- CLAUDE.md "SUB 0A — WAVEHDR response length distinguishes events from
|
||||
boundaries" (0x46 = real event, 0x2C = boundary marker)
|
||||
- protocol reference §7.8.5 / §7.8.6 / §7.8.7 / §7.8.8
|
||||
- The previous chunk-counter formula (`max(key4[2:4], 0x0400) + (chunk-1) *
|
||||
0x0400`) is now marked DEPRECATED and explicitly tagged WRONG with
|
||||
pointers to the new sections, so future work doesn't re-derive it.
|
||||
|
||||
### Known minor diffs vs Blastware (deferred to a follow-up)
|
||||
|
||||
- We still use the OLD 0x0400 chunk step rather than BW's 0x0200; switching
|
||||
also requires updating `blastware_file.write_blastware_file`'s skip values
|
||||
and "extra chunk after metadata" logic, which depends on a fresh capture
|
||||
to verify.
|
||||
- We still use the legacy fixed `offset_word=0x005A` TERM frame rather than
|
||||
BW's `end_offset - next_boundary` formula, for the same reason.
|
||||
- Two fixed metadata pages at counter `0x1002` and `0x1004` are not yet
|
||||
read explicitly; under the current 0x0400 walk their content is reachable
|
||||
via the sample chunk that covers buffer addresses `[0x1000, 0x1400)`.
|
||||
|
||||
---
|
||||
|
||||
## v0.12.6 — 2026-05-01
|
||||
|
||||
### Fixed
|
||||
|
||||
- **`blastware_file.py` — waveform frame classification** — A5 frame classification for
|
||||
waveform-only vs header-only frames now uses `frame.record_type` instead of frame index.
|
||||
Only waveform frames (0x46) are written to the file body; metadata frames are skipped.
|
||||
Fixes spurious data corruption from incorrectly classified frames.
|
||||
|
||||
- **`s3_analyzer.py` — A5/5A frame naming** — Bulk waveform stream frames (SUB 5A response)
|
||||
are now correctly labeled "A5" in analyzer output instead of being conflated with other
|
||||
multi-frame responses (SUB A4, E5, etc.).
|
||||
|
||||
- **`S3FrameParser` — frame terminator detection** — Corrected the bare ETX terminator
|
||||
detection. Frame termination is now correctly identified by a standalone `ETX=0x03` byte,
|
||||
not by the `DLE+ETX` sequence (which is part of the payload when it appears within a frame).
|
||||
|
||||
---
|
||||
|
||||
## v0.12.5 — 2026-04-21
|
||||
|
||||
### Added
|
||||
|
||||
- **`seismo_lab.py` — Download tab** — New fourth tab for live wire-byte capture during event
|
||||
downloads. Captures both BW→device and device→S3 frames in real time, allowing inspection
|
||||
of the 5A bulk stream chunk sequence and frame-by-frame analysis without needing a bridge
|
||||
or MITM proxy. Files are saved with user-specified labels for easy tracking.
|
||||
|
||||
### Changed
|
||||
|
||||
- **`s3_bridge.py` — raw captures always-on by default** — `--raw-bw` and `--raw-s3` now
|
||||
@@ -17,6 +302,10 @@ All notable changes to seismo-relay are documented here.
|
||||
"S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names
|
||||
the files). Unchecking a box passes `--raw-bw ""` to explicitly disable capture.
|
||||
|
||||
- **`Bridge tab` — TCP mode added** — Serial/TCP radio toggle allows connection via cellular
|
||||
modem (RV50/RV55) instead of direct RS-232. Supports multi-capture design (simultaneous
|
||||
Bridge + Analyzer + Download sessions).
|
||||
|
||||
- **`ach_server.py` — TX capture added (`raw_tx_<ts>.bin`)** — Every ACH inbound session
|
||||
now saves both directions: `raw_rx_<ts>.bin` (device → us, S3 side, as before) and
|
||||
`raw_tx_<ts>.bin` (us → device, BW side). Both files are usable in the Analyzer.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
|
||||
managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
|
||||
(Sierra Wireless RV50 / RV55). Current version: **v0.12.3**.
|
||||
(Sierra Wireless RV50 / RV55). Current version: **v0.14.3**.
|
||||
|
||||
When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
|
||||
|
||||
@@ -27,7 +27,7 @@ CHANGELOG.md ← version history
|
||||
|
||||
---
|
||||
|
||||
## Current implementation state (v0.12.3)
|
||||
## Current implementation state (v0.14.3)
|
||||
|
||||
Full read pipeline + write pipeline + erase pipeline + monitor log + call home config working end-to-end over TCP/cellular:
|
||||
|
||||
@@ -41,14 +41,15 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c
|
||||
| Event header / first key | 1E | ✅ |
|
||||
| Waveform header | 0A | ✅ |
|
||||
| Waveform record (peaks, timestamp, project) | 0C | ✅ |
|
||||
| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ new v0.6.0 |
|
||||
| **Bulk waveform stream (event-time metadata + full waveform)** | **5A** | ✅ **byte-perfect against BW captures (v0.14.3, 2026-05-05)** — STRT-bounded chunk walk + correct event-N probe counter + DLE-stuffed `0x10` bytes in params + concatenate-only file body assembly. All 17 5A request frames in the 5-1-26 3-sec capture reproduce byte-for-byte. |
|
||||
| Event advance / next key | 1F | ✅ |
|
||||
| **Write commands (push config to device)** | **68–83** | ✅ new v0.8.0 |
|
||||
| **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 |
|
||||
| **Monitor log entries (partial 0x2C records)** | **0A browse** | ✅ new v0.10.0 |
|
||||
| **Auto Call Home config (read + write)** | **2C → 7E → 7F** | ✅ **new v0.12.3** |
|
||||
|
||||
`get_events()` sequence per event: `1E → 0A → 0C → 5A → 1F`
|
||||
`get_events()` sequence per event: `1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)`
|
||||
(see "Correct iteration pattern" section below for full detail)
|
||||
|
||||
`push_config_raw()` write sequence: `68→73 | 71×3→72 | 82→83 | 69→74→72`
|
||||
|
||||
@@ -115,32 +116,203 @@ S3→BW (response):
|
||||
section contribute only `XX` to the running sum; lone bytes contribute normally. This
|
||||
differs from the standard SUM8-of-destuffed-payload that all other commands use.
|
||||
|
||||
Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26
|
||||
BW TX capture. All 10 frames verified.
|
||||
3. **Params region uses partial DLE stuffing (CONFIRMED 2026-05-05).** The device's
|
||||
de-stuffing rule for bytes inside the params region is:
|
||||
|
||||
### SUB 5A — chunk counter formula (FINAL CORRECTION 2026-04-26)
|
||||
- `10 10` → de-stuffs to `10`
|
||||
- `10 02 / 03 / 04` → kept literal (these are inner-frame markers)
|
||||
- `10 X` for other X → de-stuffs to just `X` (drops the leading `0x10`)
|
||||
|
||||
**Chunk counter = `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` for ALL chunks.**
|
||||
Therefore any `0x10` byte in the *logical* params that is followed by a byte NOT in
|
||||
`{0x02, 0x03, 0x04, 0x10}` MUST be doubled on the wire (`10 X` → `10 10 X`) so the
|
||||
device's de-stuffer reproduces the original `10 X` pair. This applies most commonly
|
||||
to counters with `0x10` in the high byte (e.g. counter=`0x1000` produces logical
|
||||
params bytes `... 10 00 ...`, which BW encodes on the wire as `... 10 10 00 ...`).
|
||||
Without this stuffing the device interprets counter=`0x1000` as `0x0000` and returns
|
||||
the probe response (which contains a copy of the file header + STRT record). That
|
||||
STRT block then gets embedded in the assembled file body at offset `0x1016`, and
|
||||
Blastware refuses to open the file — see the v0.14.3 entry in `CHANGELOG.md`.
|
||||
|
||||
where `key4[2:4] = (key4[2] << 8) | key4[3]` is the event's circular-buffer base offset.
|
||||
`0x10` bytes in `offset_hi` (body[5]) are still written RAW — only the params region
|
||||
has this stuffing requirement. The metadata-page params for counter `0x1002` /
|
||||
`0x1004` survive without stuffing because `10 02` and `10 04` fall in the "kept
|
||||
literal" carve-out.
|
||||
|
||||
The `max(..., 0x0400)` guard is critical for events at the start of the circular buffer
|
||||
(key4[2:4] == 0x0000, e.g. key `01110000`). Without it, chunk 1 gets counter=0x0000, which
|
||||
is the same address as the probe frame — the device re-returns the STRT record data instead
|
||||
of waveform payload. With the guard, chunk 1 gets counter=0x0400, which is confirmed correct
|
||||
from the empirical live-device test 2026-04-06 (`counter=0x0400 → responds immediately and
|
||||
streams all frames correctly`).
|
||||
Both differences (1) and (2) confirmed by reproducing Blastware's exact wire bytes from
|
||||
the 1-2-26 BW TX capture (10 frames). Difference (3) confirmed against the 5-1-26
|
||||
"bwcap3sec" capture (17 frames, all match byte-for-byte after fix).
|
||||
|
||||
The 4-3-26 capture confirms the pattern for a second event (key `0111245a`, key4[2:4]=0x245a):
|
||||
chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400).
|
||||
`max(0x245a, 0x0400) = 0x245a` → formula works correctly for non-zero base offset too.
|
||||
### SUB 5A — chunk counter formula (REWRITTEN 2026-05-01 — see 5-1-26 captures)
|
||||
|
||||
> ⚠️ **Everything that came before this rewrite was WRONG in important ways.** The previous
|
||||
> formula `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` happened to *work* for events
|
||||
> at start_key=0 because the device responds to whatever counter you ask for — but it caused
|
||||
> a 5× over-read past the actual event, picking up post-event circular-buffer garbage that
|
||||
> corrupts the reconstructed file for any event > ~1 sec of waveform. The captures in
|
||||
> `bridges/captures/4-27-26/` and `5-1-26/comcheck/` show BW reads only ~12-16 chunks for
|
||||
> the same events SFM was reading 37+ chunks for. See "TERM frame" and "STRT end_offset"
|
||||
> sections below for the actual mechanism.
|
||||
|
||||
**Chunk addressing is just absolute device-buffer addresses.**
|
||||
|
||||
`params[0]=0x00`, `params[1:5]` is a 4-byte absolute device flash-buffer address (= the
|
||||
"key" of that location), `params[5:11]` are zeros. The device returns 0x0200 (= 512) bytes
|
||||
starting at that address. Increments between consecutive chunks are **0x0200 (NOT 0x0400)**
|
||||
— this matches the chunk payload size. The previous "0x0400 step" worked by accident: BW
|
||||
asks for half-size chunks; SFM was asking for double-size chunks, both with the same-named
|
||||
"counter" field, but the value is just an address pointer the device honors as-is.
|
||||
|
||||
**The chunk pattern depends on whether the event sits at start_key=0 or not.**
|
||||
|
||||
#### Event 1 case — start_key[2:4] == 0x0000 (first event after erase / wrap)
|
||||
|
||||
```
|
||||
1. Probe at counter=0x0000 (params[1:5] = full key, returns STRT record)
|
||||
2. Read 2 fixed metadata pages: counter=0x1002, counter=0x1004
|
||||
(these are GLOBAL session metadata — read ONCE per
|
||||
Blastware session, not per event; contain the
|
||||
Project/Client/User Name/Seis Loc strings)
|
||||
3. Sample chunks: counter=0x0600, 0x0800, …, by 0x0200 increment,
|
||||
up to but not including end_offset (rounded down to
|
||||
0x0200 boundary)
|
||||
4. TERM frame (see TERM formula below)
|
||||
```
|
||||
|
||||
The reason `0x0046..0x0600` is skipped for event 1 is unknown — likely some pre-event
|
||||
firmware reserved area for the first slot in a freshly-erased buffer. Harmless to skip.
|
||||
|
||||
#### Event 2+ case — start_key[2:4] != 0x0000 (continuation events)
|
||||
|
||||
```
|
||||
1. First chunk at counter = start_key[2:4] (this IS the probe — response
|
||||
contains STRT at byte 17)
|
||||
2. Sample chunks: counter += 0x0200 each, up to but
|
||||
not including end_offset
|
||||
3. TERM frame
|
||||
```
|
||||
|
||||
**`start_key` here is the off=0x46 WAVEHDR record key returned by 1F** (e.g. `01112238`),
|
||||
NOT the off=0x2C boundary key that immediately precedes it. An earlier draft of this
|
||||
doc described event-N as "probe at start + 0x46" — that formula came from naming the
|
||||
boundary key as `start_key`. In the iteration walk, `cur_key` passed to
|
||||
`read_bulk_waveform_stream` is always the off=0x46 key (the partial-record skip path in
|
||||
`get_events` re-runs 1F to advance past boundary records before invoking 5A), so the
|
||||
probe counter is just `cur_key[2:4]` with no extra offset. **Adding +0x46 caused the
|
||||
probe to overshoot, miss the STRT record at byte 17 of the response, fall back to the
|
||||
`max_chunks=128` cap, and walk ~110 chunks of post-event garbage** — observed in
|
||||
SFM 5-4-26 capture before the fix.
|
||||
|
||||
Confirmed across:
|
||||
- 5-1-26 "copy 2nd address" BW capture: probe counter=0x2238, key=01112238, STRT@17 end=0x417E.
|
||||
- 5-4-26 BW 2-sec event capture: probe counter=0x2238, key=01112238, TERM offset_word=0x0146 → end=0x417E.
|
||||
|
||||
No metadata pages — those have already been read during event 1 in the same Blastware
|
||||
session, and BW caches them. Note that the metadata-page reads happen ONCE per
|
||||
Blastware-session-on-the-device, not once per event, so an SFM session that downloads
|
||||
several events should read 0x1002/0x1004 only once at the start.
|
||||
|
||||
#### History (do not re-derive)
|
||||
|
||||
**History:**
|
||||
- Original: `_CHUNK1_COUNTER = 0x1004` hardcoded (Blastware capture artifact — WRONG).
|
||||
- 2026-04-06: Corrected to `chunk_num * 0x0400` (worked for key 01110000 only).
|
||||
- 2026-04-24: Corrected to `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets,
|
||||
but accidentally broke key 01110000 — counter=0x0000 sends probe address again).
|
||||
- 2026-04-26: Final formula: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400`.
|
||||
- 2026-04-06: `chunk_num * 0x0400` (worked for key 01110000 only).
|
||||
- 2026-04-24: `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets, broke key 01110000).
|
||||
- 2026-04-26: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` (broken — over-read past event end).
|
||||
- 2026-05-01: Increments are 0x0200 not 0x0400; absolute addresses inside event range; bounded
|
||||
by STRT end_key, not by `max_chunks` cap or device-side timeout.
|
||||
- 2026-05-04: Removed spurious `+0x0046` from event-N probe counter. `cur_key` from 1F
|
||||
is already the off=0x46 WAVEHDR key, so adding +0x46 would have placed the probe one
|
||||
WAVEHDR past the actual event start. This caused probe responses to lack a STRT
|
||||
record (no `end_offset` parsed → `0xFFFF` fallback → `max_chunks=128` cap), walking
|
||||
~110 chunks of post-event circular-buffer garbage. Fixed in protocol.py
|
||||
`read_bulk_waveform_stream`.
|
||||
|
||||
### SUB 5A — STRT record encodes end_offset (NEW 2026-05-01)
|
||||
|
||||
The first A5 response (probe response, or the first chunk for event 2+) contains a STRT
|
||||
record at byte offset 17 of the `data` field. Layout:
|
||||
|
||||
```
|
||||
data[17:21] "STRT" magic
|
||||
data[21:23] ff fe sentinel
|
||||
data[23:27] end_key ← 4-byte key of where this event ENDS
|
||||
data[27:31] start_key ← 4-byte key of where this event STARTS
|
||||
data[31:33] uint16 BE ?? sample-count or total bytes (varies; not yet decoded)
|
||||
data[33:35] uint16 BE ??
|
||||
data[35] 0x46 record type (waveform full record)
|
||||
…
|
||||
```
|
||||
|
||||
`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**.
|
||||
SFM must extract this from the first A5 response and use it to bound the chunk loop and
|
||||
encode the TERM frame. The device will happily respond to chunk requests past `end_offset`
|
||||
(returning post-event circular-buffer contents) — that's the over-read bug.
|
||||
|
||||
Verified across 3 events:
|
||||
|
||||
| Capture | start_key | end_key | end_offset | event size |
|
||||
|---|---|---|---|---|
|
||||
| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B |
|
||||
| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B |
|
||||
| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | `0x417E` (event 2 span 0x1F8C = 8,076 B) |
|
||||
|
||||
### SUB 5A — TERM frame formula (FINALIZED 2026-05-01)
|
||||
|
||||
The TERM frame fetches the partial last chunk *and* the file footer. It is **not** a simple
|
||||
"goodbye" frame — its response payload contains the bytes between the last full 0x0200-aligned
|
||||
chunk and `end_offset`, and is required for reconstructing the Blastware file format.
|
||||
|
||||
```
|
||||
last_chunk_counter = address of last full 0x0200-byte chunk read
|
||||
next_boundary = last_chunk_counter + 0x0200
|
||||
TERM offset_word = end_offset - next_boundary
|
||||
TERM params[0] = key[0] (= 0x01 on every observed device)
|
||||
TERM params[1] = key[1] (= 0x11)
|
||||
TERM params[2] = (next_boundary >> 8) & 0xFF
|
||||
TERM params[3] = next_boundary & 0xFF
|
||||
TERM params[4:10] = zeros
|
||||
build_5a_frame(offset_word, params) (10-byte params, NOT 11)
|
||||
```
|
||||
|
||||
The device reconstructs `requested_address = (params[2] << 8) | offset_word = end_offset`
|
||||
and replies with `(end_offset - next_boundary)` bytes from `next_boundary` — the residual
|
||||
between the last 0x0200 boundary and the actual event end. Append the TERM response data
|
||||
to the chunk stream like any other A5 frame; it carries the final waveform tail + footer.
|
||||
|
||||
Verified across 3 events:
|
||||
|
||||
| end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] |
|
||||
|---|---|---|---|---|
|
||||
| `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ |
|
||||
| `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ |
|
||||
| `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ |
|
||||
|
||||
The previous code's hard-coded `offset_word = 0x005A` and `term_counter = last + 0x0400`
|
||||
are wrong; the device's response under that path is a tiny 101-byte device-side terminator
|
||||
(arrived only after we walked the entire post-event buffer), not the proper file footer.
|
||||
|
||||
### SUB 5A — fixed metadata pages 0x1002 and 0x1004 (NEW 2026-05-01)
|
||||
|
||||
Two chunk addresses are GLOBAL device/session metadata, not event-specific:
|
||||
|
||||
- `counter=0x1002` — first metadata page
|
||||
- `counter=0x1004` — second metadata page
|
||||
|
||||
These are at fixed absolute addresses in the device's flash buffer. They contain the
|
||||
session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII
|
||||
strings). Under the v0.14.0+ walk these strings are read directly from the metadata
|
||||
pages, not from the sample-chunk stream.
|
||||
|
||||
BW reads them ONCE per Blastware session (during event 1's download) and caches them.
|
||||
For SFM, that means:
|
||||
- Once per call-home / once per `MiniMateClient.connect()` is enough.
|
||||
- Subsequent events in the same session don't need to re-fetch them.
|
||||
- Their content does not change when iterating events; only when the user opens
|
||||
Compliance Setup → Apply on the device or sends a SUB 71 compliance write.
|
||||
|
||||
The full byte-for-byte layout of the metadata pages has not been mapped — `_decode_a5_metadata_into`
|
||||
locates the ASCII strings via label scans (`Project:`, `Client:`, `User Name:`, `Seis Loc:`,
|
||||
`Extended Notes`) which works correctly across observed captures. Future work could
|
||||
dump the structural layout if more session-global fields need to be extracted.
|
||||
|
||||
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
|
||||
|
||||
@@ -148,10 +320,11 @@ chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400).
|
||||
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
|
||||
Do not swap them.
|
||||
|
||||
### SUB 5A — event-time metadata lives in A5 frame 7
|
||||
### SUB 5A — event-time metadata source (FINALIZED 2026-05-05)
|
||||
|
||||
The bulk stream sends 9+ A5 response frames. Frame 7 (0-indexed) contains the compliance
|
||||
setup as it existed when the event was recorded:
|
||||
The metadata strings come from the two fixed metadata pages at counter `0x1002` and
|
||||
`0x1004` (see "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above). These pages
|
||||
are GLOBAL session metadata — read once per Blastware/SFM session, not per event.
|
||||
|
||||
```
|
||||
"Project:" → project description
|
||||
@@ -161,44 +334,71 @@ setup as it existed when the event was recorded:
|
||||
"Extended Notes"→ notes
|
||||
```
|
||||
|
||||
**IMPORTANT — 5A "Project:" is session-start config, NOT per-event (confirmed 2026-04-05):**
|
||||
The "Project:" string in the A5 frame 7 payload reflects the compliance setup from when
|
||||
the *monitoring session first started*, not the individual event's project name. The per-
|
||||
event project name is correctly stored in the 210-byte 0C waveform record and must be
|
||||
used as the authoritative source. `_decode_a5_metadata_into` therefore only sets
|
||||
`project` from 5A when 0C didn't already supply one.
|
||||
**IMPORTANT — these strings are session-start config, NOT per-event:**
|
||||
Project / Client / User Name / Seis Loc reflect the compliance setup from when the
|
||||
*monitoring session first started*, not the individual event's per-event metadata. The
|
||||
authoritative per-event project name is stored in the 210-byte 0C waveform record.
|
||||
`_decode_a5_metadata_into` therefore only sets `project` from the 5A metadata pages
|
||||
when 0C didn't already supply one.
|
||||
|
||||
"Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C
|
||||
record — 5A remains the sole source for those fields and they are set unconditionally.
|
||||
record — the metadata pages are the sole source for those fields and they are set
|
||||
unconditionally.
|
||||
|
||||
`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears,
|
||||
then sends the termination frame.
|
||||
#### Deprecated knobs (do not re-introduce)
|
||||
|
||||
### SUB 5A — end-of-stream signal (confirmed 2026-04-06)
|
||||
The `read_bulk_waveform_stream()` function still accepts these legacy kwargs for
|
||||
backward compatibility, but they are **no-ops** under the v0.14.0+ walk:
|
||||
|
||||
After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to
|
||||
the next chunk request, then goes silent. This is the natural end-of-stream indicator — NOT
|
||||
a complete A5 frame. `S3FrameParser.bytes_fed` will be 1; no frame is assembled.
|
||||
- `stop_after_metadata=True` — used to scan the chunk stream for `b"Project:"` and stop
|
||||
one chunk later as a workaround for the missing end_offset bound. Obsolete: the loop
|
||||
is now deterministically bounded by `end_offset` parsed from the STRT record at
|
||||
data[17] of the probe response, with the partial tail fetched by the TERM frame.
|
||||
- `extra_chunks_after_metadata` — same era, same reason. No-op.
|
||||
|
||||
Handling: on `TimeoutError`, if `bytes_fed > 0` AND frames were already collected, treat as
|
||||
graceful end-of-stream, break the loop, and proceed to the termination frame. If `bytes_fed
|
||||
== 0` with no prior frames, it is a genuine transport failure — re-raise.
|
||||
If you find code or docs referencing "A5 frame 7" as the source of metadata strings,
|
||||
that's an old-walk artifact (the broken `0x0400`-step formula occasionally caught the
|
||||
0x1002 metadata page at sample-chunk fi=7). Update to reference the dedicated metadata
|
||||
pages instead.
|
||||
|
||||
**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
|
||||
Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call
|
||||
in the chunk loop passes `timeout=10.0` explicitly.
|
||||
### SUB 5A — end-of-stream (FINALIZED 2026-05-01)
|
||||
|
||||
**Typical chunk count (BE11529, 1024 sps):** A 9,306-sample event produces 35 chunks before
|
||||
end-of-stream. Chunks with uniform 1,036-byte data are all-zero ADC samples (post-event
|
||||
silence). Only the initial variable-size chunks contain actual signal.
|
||||
Under the v0.14.0+ STRT-bounded walk the stream ends cleanly:
|
||||
|
||||
```
|
||||
… last full chunk at counter < end_offset
|
||||
TERM request (offset_word = end_offset - next_boundary,
|
||||
params address (next_boundary))
|
||||
TERM response (page_key = 0x0000 or 0x0001, data = the residual
|
||||
end_offset - next_boundary bytes including the file footer)
|
||||
```
|
||||
|
||||
No timeout-based detection, no "1-byte teaser," no `max_chunks` cap. The chunk loop
|
||||
exits when `counter + 0x0200 > end_offset`; the TERM frame fetches the tail.
|
||||
|
||||
**Chunk recv timeout is 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
|
||||
Using 120 s would cause a ~2-minute stall on any unexpected timeout. The `_recv_one`
|
||||
call in the chunk loop passes `timeout=10.0` explicitly.
|
||||
|
||||
**Typical chunk count under the v0.14.0+ walk (BE11529, 1024 sps over TCP/cellular):**
|
||||
|
||||
| Event duration | Sample chunks | Metadata pages | TERM | Total A5 frames |
|
||||
|---|---|---|---|---|
|
||||
| 2-sec (event 1) | ~12 | 2 | 1 | ~15 |
|
||||
| 3-sec (event 1) | 13 | 2 | 1 | 16 |
|
||||
| 2-sec (continuation) | 15 | 0 | 1 | 16 |
|
||||
| 3-sec (continuation) | ~14 | 0 | 1 | ~15 |
|
||||
|
||||
For comparison, the deprecated `0x0400`-step walk produced ~37 chunks for a 2-sec
|
||||
event with chunks 17-37 containing post-event circular-buffer garbage. Do not
|
||||
re-introduce that walk under any circumstances.
|
||||
|
||||
### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06)
|
||||
|
||||
`_decode_a5_waveform()` previously had `elif fi == 9: continue` — a leftover from the
|
||||
9-frame original blast capture where frame 9 was assumed to be a terminator. For current
|
||||
35-frame streams, fi==9 is live waveform data (~133 sample-sets were being dropped).
|
||||
Removed. Terminator detection is via `page_key == 0x0000` in `read_bulk_waveform_stream`,
|
||||
not frame index.
|
||||
9-frame original blast capture where frame 9 was assumed to be a terminator. Removed.
|
||||
TERM detection in the file builder uses `frame.page_key != 0x0010` (sample marker),
|
||||
not frame index — see `blastware_file.py`.
|
||||
|
||||
### SUB 1E / 1F — event iteration null sentinel and token position (FIXED, do not re-introduce)
|
||||
|
||||
@@ -303,6 +503,55 @@ sends token=0xFE and is NOT used by any caller.
|
||||
`advance_event()` returns `(key4, event_data8)`.
|
||||
Callers (`count_events`, `get_events`) loop while `data8[4:8] != b"\x00\x00\x00\x00"`.
|
||||
|
||||
### SUB 0A — WAVEHDR response length distinguishes events from boundaries (NEW 2026-05-01)
|
||||
|
||||
When iterating events with the "Download All" pattern (1E → 0A → 1F → 0A → 1F → …), the
|
||||
DATA_LENGTH at `data_rsp.data[5]` (= the byte BW echoes back as the offset for the data
|
||||
fetch step) takes one of two values:
|
||||
|
||||
| WAVEHDR offset | Meaning |
|
||||
|---|---|
|
||||
| `0x46` (= 70) | Real event start key — there is event data at this address |
|
||||
| `0x2C` (= 44) | Boundary marker between events — this key is the END of the previous event AND the START key for the empty space after it (or is the next event's pre-header) |
|
||||
|
||||
Confirmed from the 5-1-26 "Download All" capture:
|
||||
|
||||
```
|
||||
0A(key=01110000) → off=0x46 ← event 1 real start
|
||||
1F → key=011121F2
|
||||
0A(key=011121F2) → off=0x2C ← event 1 END / event 2 boundary
|
||||
1F → key=01112238
|
||||
0A(key=01112238) → off=0x46 ← event 2 real start (= boundary + 0x46)
|
||||
1F → key=0111417E
|
||||
0A(key=0111417E) → off=0x2C ← event 2 END / next-empty marker
|
||||
1F → null sentinel
|
||||
```
|
||||
|
||||
This is why event 2's first 5A chunk is at `start_key + 0x46` — that's the address of the
|
||||
"real start" 0x46-record, distinct from the `0x2C`-record at the raw boundary. Use the
|
||||
`0x46` keys as the input to `read_bulk_waveform_stream`, not the `0x2C` keys.
|
||||
|
||||
For event 1 only (start_key[2:4] = 0x0000) BW probes at counter=0x0000 directly, which is
|
||||
the `0x46`-keyed start record. Subsequent events use `start_key + 0x46`.
|
||||
|
||||
**Practical iteration pattern (replaces the old 1E/1F walk for downloads):**
|
||||
|
||||
```
|
||||
Setup: SERIAL × 2 → CHCFG → 1E (token=0x00) → key0
|
||||
For each event:
|
||||
0A(cur_key) → DATA_LENGTH = 0x46 (real) or 0x2C (boundary)
|
||||
1F (token=0x00) → next_key
|
||||
if length was 0x46: → cur_key is a real event; queue it for download
|
||||
cur_key = next_key
|
||||
if next_key all-zero null sentinel: stop
|
||||
|
||||
Then for each queued real-event key:
|
||||
download_event(key) → 5A bulk stream with STRT-bounded chunk walk
|
||||
```
|
||||
|
||||
This is what BW does in the 5-1-26 "Download All" capture — it walks the full event chain
|
||||
collecting `(key, length)` tuples first, *then* downloads each event using the `0x46` keys.
|
||||
|
||||
### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce)
|
||||
|
||||
`read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where:
|
||||
@@ -527,6 +776,8 @@ All DB endpoints are read-only except `PATCH /db/events/{id}/false_trigger`.
|
||||
| 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture |
|
||||
| 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence; only 1 event stored so token=0xFE appeared to work |
|
||||
| 4-3-26 | `bridges/captures/4-3-26/` | Browse-mode S3 capture with 2+ events — confirmed all-zero params for 1F, 1F response layout, null sentinel, 0A context requirement |
|
||||
| 4-27-26 | `bridges/captures/4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof that SFM was over-reading 5× past event end. BW reads 14 chunks at 0x0200 increments + TERM at end_offset; SFM was reading 37 chunks at 0x0400 increments. STRT end_key field located. |
|
||||
| 5-1-26 | `bridges/captures/5-1-26/comcheck/` | Three sub-captures: SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945`/`_171216`). Confirmed: TERM frame formula across 3 events; metadata pages 0x1002/0x1004 are global (read once per session); event-1 vs event-N chunk-pattern split; WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundaries. |
|
||||
|
||||
---
|
||||
|
||||
@@ -790,7 +1041,7 @@ offsets in the raw 1A/E5 payload. Only fields with `✅` have confirmed offsets
|
||||
|
||||
**Notes tab:**
|
||||
- Enable User Notes (bool)
|
||||
- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from A5 frame 7 via 5A)
|
||||
- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from 5A metadata pages at counter 0x1002 / 0x1004 — see "SUB 5A — fixed metadata pages" section)
|
||||
- Enable Extended Notes (bool); Extended Notes text; Extended Notes Title
|
||||
- Enable Job Number (bool); Job Number (int)
|
||||
- Enable Scaled Distance (bool); Distance from Blast (float); Charge Weight (float) — Scaled Distance is derived
|
||||
@@ -1104,7 +1355,7 @@ body) because writing a dial string may require DLE escaping for embedded contro
|
||||
|
||||
- **Database** — SQLite store for events + monitor log entries; dedup by key; queryable
|
||||
- **Histograms** — decode histogram-mode A5 data (noise floor tracking)
|
||||
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed working for Continuous mode events (2026-04-23):** SFM-generated file opens in Blastware, shows correct PPV/waveform/timestamp. File is ~200 bytes shorter than BW (missing last ADC tail slice) — all measurements correct. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that create spurious STRT markers in the body). Extension mapping: **CONFIRMED FALSE 2026-04-21** — extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
|
||||
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed BYTE-PERFECT against BW reference (v0.14.3, 2026-05-05):** when fed the BW 5-1-26 3-sec capture's A5 frames, the SFM-built file matches BW's saved `M529LKIQ.G10` byte-for-byte (8708 bytes, 0 differences). Live SFM downloads of event 0 (3-sec) and event 1 (3-sec continuation) both open cleanly in Blastware with full Event Reports, frequency analysis, and waveform plots. Body assembly is just contiguous concatenation of frame contributions in stream order (probe → meta@0x1002 → meta@0x1004 → samples → TERM); no stripping, no overlay, no special handling. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that may need different handling — untested under v0.14.x). Extension mapping: extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
|
||||
|
||||
**Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units).
|
||||
|
||||
@@ -1140,16 +1391,21 @@ body) because writing a dial string may require DLE escaping for embedded contro
|
||||
|
||||
| Folder / File | Contents |
|
||||
|---|---|
|
||||
| `1-2-26/` | First SUB 5A BW TX capture — established 5A frame format (raw offset_hi, DLE-aware checksum). 10 frames verified. |
|
||||
| `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102–112) |
|
||||
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) — 1E/0A/0C/1F sequence confirmed (single event so token=0xFE appeared to work in either branch) |
|
||||
| `4-2-26/` | Download-mode BW TX capture — POLL×3 requirement confirmed (frames 68-73 between 1F and first 5A) |
|
||||
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events — all-zero params for 1F, null sentinel layout, 0A context requirement |
|
||||
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check |
|
||||
| `4-11-26 (mitm/ach_mitm_20260411_001912/)` | Full ACH call-home MITM — erase protocol (0xA3/0x06/0xA2), monitor log partial records confirmed |
|
||||
| `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous |
|
||||
| `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec |
|
||||
| `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) |
|
||||
| `4-20-26/call home settings/` | Call home config read/write captures |
|
||||
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check |
|
||||
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events (1E/0A/1F iteration confirmed) |
|
||||
| `4-2-26/` | Download-mode BW TX capture (5A bulk stream, POLL×3 requirement confirmed) |
|
||||
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) |
|
||||
| `mitm/ach_mitm_20260411_001912/` | Full ACH call-home MITM (erase protocol, 0xA3/0x06/0xA2 confirmed) |
|
||||
| `4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof of 5× SFM over-read. STRT end_key field located. |
|
||||
| **`5-1-26/comcheck/`** | **Triplet of captures that nailed the v0.14.0 walk:** SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945` / `_171216`). Confirmed: TERM frame formula across 3 events, metadata pages 0x1002/0x1004 are global session metadata, event-1 vs event-N chunk pattern split, WAVEHDR off=0x46 vs 0x2C disambiguates real events from boundaries. |
|
||||
| **`5-1-26/comcheck/bwcap3sec/`** | **The byte-perfect reference for v0.14.3.** All 17 BW 5A request frames (probe, 2 metadata, 13 samples, TERM) reproduce byte-for-byte from SFM's framing helpers — including the `10 10 00` DLE-stuffed counter for sample @ 0x1000 that was the long-standing failure mode. |
|
||||
| `5-4-26/` | BW MITM captures of "copy 3sec / 2sec / Download All" + paired SFM session (`seismo_dl_20260504_145701`) showing the +0x46 event-N probe bug producing 110-chunk runaway walk. Cross-references against 5-1-26 confirmed device behavior is identical. |
|
||||
|
||||
To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern
|
||||
in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# seismo-relay `v0.12.1`
|
||||
# seismo-relay `v0.14.3`
|
||||
|
||||
A ground-up replacement for **Blastware** — Instantel's aging Windows-only
|
||||
software for managing MiniMate Plus seismographs.
|
||||
@@ -10,6 +10,10 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
||||
> pipeline working end-to-end over TCP/cellular. ACH Auto Call Home server
|
||||
> handles inbound unit connections, downloads events, and persists everything
|
||||
> to a SQLite database. SFM REST API exposes device control and DB queries.
|
||||
> **As of v0.14.3 (2026-05-05): SUB 5A bulk waveform protocol is verified
|
||||
> byte-perfect against Blastware captures across 2-sec, 3-sec, and 10-sec
|
||||
> events.** Generated `.G10` / `.AB0` files open cleanly in Blastware with
|
||||
> full Event Reports, frequency analysis, and waveform plots.
|
||||
> See [CHANGELOG.md](CHANGELOG.md) for full version history.
|
||||
|
||||
---
|
||||
@@ -18,26 +22,27 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
||||
|
||||
```
|
||||
seismo-relay/
|
||||
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Console tabs)
|
||||
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Download + Console tabs)
|
||||
│
|
||||
├── minimateplus/ ← MiniMate Plus client library
|
||||
│ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport
|
||||
│ ├── protocol.py ← DLE frame layer, SUB command dispatch
|
||||
│ ├── client.py ← High-level client (connect, get_events, push_config, …)
|
||||
│ ├── client.py ← High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …)
|
||||
│ ├── framing.py ← Frame builders, DLE codec, S3FrameParser
|
||||
│ └── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, …
|
||||
│ ├── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig, …
|
||||
│ └── blastware_file.py ← Write events to Blastware-compatible .AB0 files
|
||||
│
|
||||
├── sfm/ ← SFM REST API server (FastAPI, port 8200)
|
||||
│ ├── server.py ← All device + DB endpoints
|
||||
│ ├── database.py ← SeismoDb — SQLite persistence layer
|
||||
│ └── sfm_webapp.html ← Embedded web UI (served at /)
|
||||
│ ├── server.py ← Live device endpoints + DB query endpoints + caching
|
||||
│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions, sessions table)
|
||||
│ └── sfm_webapp.html ← Embedded web UI with Call Home config tab
|
||||
│
|
||||
├── bridges/
|
||||
│ ├── ach_server.py ← Inbound ACH call-home server (main production server)
|
||||
│ ├── ach_mitm.py ← Transparent MITM proxy for capturing BW sessions
|
||||
│ ├── s3-bridge/ ← RS-232 serial bridge (capture tool)
|
||||
│ ├── tcp_serial_bridge.py ← Local TCP↔serial bridge (bench testing)
|
||||
│ ├── gui_bridge.py ← Standalone bridge GUI
|
||||
│ ├── gui_bridge.py ← Standalone bridge GUI with raw capture checkboxes
|
||||
│ └── raw_capture.py ← Simple raw capture tool
|
||||
│
|
||||
├── parsers/
|
||||
@@ -101,21 +106,28 @@ python seismo_lab.py
|
||||
Each call dials the device, does its work, and closes the connection. TCP
|
||||
connections are retried once on `ProtocolError` to handle cold-boot timing.
|
||||
|
||||
**Caching** — frequently-polled endpoints are cached in-process to avoid
|
||||
redundant TCP round-trips:
|
||||
**In-memory caching** — frequently-polled endpoints avoid redundant TCP round-trips
|
||||
via a thread-safe `_LiveCache` (plain Python dict + `threading.Lock`):
|
||||
|
||||
| Method | URL | Cache |
|
||||
|--------|-----|-------|
|
||||
| Method | URL | Cache Strategy |
|
||||
|--------|-----|---|
|
||||
| `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` |
|
||||
| `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected |
|
||||
| `GET` | `/device/event/{idx}/waveform` | Permanent per event index |
|
||||
| `GET` | `/device/monitor/status` | 30-second TTL |
|
||||
| `GET` | `/device/monitor/status` | 30-second TTL; invalidated by monitor start/stop |
|
||||
| `GET` | `/device/call_home` | Fresh read from device (not cached) |
|
||||
| `POST` | `/device/connect` | — |
|
||||
| `POST` | `/device/config` | Writes compliance config; invalidates cache |
|
||||
| `POST` | `/device/monitor/start` | Sends SUB 0x96 |
|
||||
| `POST` | `/device/monitor/stop` | Sends SUB 0x97 |
|
||||
| `POST` | `/device/config` | Writes compliance config; invalidates info + events cache |
|
||||
| `POST` | `/device/config/project` | Patches project/client/operator/sensor_location strings |
|
||||
| `POST` | `/device/monitor/start` | Sends SUB 0x96; immediately evicts status cache |
|
||||
| `POST` | `/device/monitor/stop` | Sends SUB 0x97; immediately evicts status cache |
|
||||
| `POST` | `/device/call_home` | Reads, patches specified fields, writes back to device |
|
||||
|
||||
All cached endpoints accept `?force=true` to bypass the cache.
|
||||
**Cache bypass** — All cached endpoints accept `?force=true` to skip the cache and
|
||||
force a fresh read from the device.
|
||||
|
||||
**Cache stats** — `GET /cache/stats` returns hit/miss counts and TTL info; `DELETE /cache/device`
|
||||
clears the device cache immediately.
|
||||
|
||||
Transport query params (supply one set):
|
||||
```
|
||||
@@ -152,21 +164,33 @@ client = MiniMateClient(transport=TcpTransport("1.2.3.4", 12345), timeout=30.0)
|
||||
|
||||
with client:
|
||||
# Read
|
||||
info = client.connect() # DeviceInfo — serial, firmware, compliance config
|
||||
count = client.count_events() # Number of stored events
|
||||
keys = client.list_event_keys() # Fast browse walk — event keys only, no download
|
||||
events = client.get_events() # Full download: headers + peaks + metadata
|
||||
monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag
|
||||
log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records)
|
||||
info = client.connect() # DeviceInfo — serial, firmware, compliance config
|
||||
count = client.count_events() # Number of stored events
|
||||
keys = client.list_event_keys() # Fast browse walk — event keys only, no download
|
||||
events = client.get_events() # Full download: headers + peaks + metadata
|
||||
monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag
|
||||
log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records)
|
||||
ach_cfg = client.get_call_home_config() # Auto Call Home settings (SUB 0x2C)
|
||||
|
||||
# Write
|
||||
client.apply_config(
|
||||
sample_rate=1024,
|
||||
recording_mode="Continuous", # Single Shot / Continuous / Histogram / Histogram+Continuous
|
||||
histogram_interval_sec=15, # 2, 5, 15, 60, 300, 900
|
||||
trigger_level_geo=0.5,
|
||||
geo_range="Normal", # Normal (10.000 in/s) / Sensitive (1.25 in/s)
|
||||
project="Bridge Inspection 2026",
|
||||
client_name="City of Portland",
|
||||
operator="B. Harrison",
|
||||
)
|
||||
|
||||
client.set_call_home_config(
|
||||
auto_call_home_enabled=True,
|
||||
after_event_recorded=True,
|
||||
at_specified_times=True,
|
||||
time1_hour=18, time1_min=30, # 6:30 PM
|
||||
time2_hour=6, time2_min=0, # 6:00 AM
|
||||
)
|
||||
|
||||
# Control
|
||||
client.start_monitoring() # SUB 0x96
|
||||
@@ -174,26 +198,33 @@ with client:
|
||||
client.delete_all_events() # Erase all (SUB 0xA3 → 0x1C → 0x06 → 0xA2)
|
||||
```
|
||||
|
||||
`get_events()` runs the full per-event sequence: `1E → 0A → 0C → 5A → 1F`.
|
||||
SUB 5A bulk stream provides `client`, `operator`, and `sensor_location` as they
|
||||
existed at record time — not backfilled from the current compliance config.
|
||||
`get_events()` runs the full per-event sequence:
|
||||
`1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)`.
|
||||
SUB 5A bulk stream walks chunks bounded by the `end_offset` extracted from
|
||||
the STRT record at byte 17 of the probe response — no over-reading, no
|
||||
chunk-count cap. Project / client / operator / sensor location strings come
|
||||
from the dedicated metadata pages at counter `0x1002` and `0x1004`,
|
||||
read once per session (they reflect the compliance setup at session start,
|
||||
not per individual event).
|
||||
|
||||
---
|
||||
|
||||
## Database
|
||||
|
||||
`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode).
|
||||
Three tables, all unit-keyed by serial number:
|
||||
`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode) using the
|
||||
`SeismoDb` persistence layer. Four tables, all unit-keyed by serial number:
|
||||
|
||||
| Table | Key | Contents |
|
||||
|-------|-----|----------|
|
||||
| `ach_sessions` | UUID | Per-call-home audit record: serial, peer IP, events_downloaded, duration |
|
||||
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, PPV per channel, project/client/operator strings, false_trigger flag |
|
||||
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: start/stop time, duration, geo threshold |
|
||||
| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds |
|
||||
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag |
|
||||
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
|
||||
| `events.false_trigger` | Boolean flag | PATCH endpoint to mark/unmark false triggers for review |
|
||||
|
||||
Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs
|
||||
never produce duplicate rows. Post-erase key reuse is handled automatically
|
||||
via the high-water mark in `ach_state.json`.
|
||||
Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs never
|
||||
produce duplicate rows. Post-erase key reuse is handled automatically via the
|
||||
high-water mark in `ach_state.json`. Key-based state tracking allows correct
|
||||
handling of device erasures (external or post-download).
|
||||
|
||||
---
|
||||
|
||||
@@ -231,6 +262,27 @@ Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/insta
|
||||
|
||||
---
|
||||
|
||||
## Compliance Config Features
|
||||
|
||||
The REST API and web UI expose full control over device compliance settings:
|
||||
|
||||
- **Recording Mode** (Single Shot / Continuous / Histogram / Histogram+Continuous)
|
||||
- **Sample Rate** (1024 / 2048 / 4096 sps)
|
||||
- **Record Time** (float, seconds)
|
||||
- **Histogram Interval** (2s, 5s, 15s, 1m, 5m, 15m) — when recording mode includes histogram
|
||||
- **Geo Trigger Levels** (float, in/s per channel)
|
||||
- **Geo Maximum Range** (Normal 10.000 in/s / Sensitive 1.250 in/s per channel)
|
||||
- **Project / Client / Operator / Sensor Location** (ASCII strings)
|
||||
|
||||
Auto Call Home config:
|
||||
- **Auto Call Home Enable** (bool)
|
||||
- **Dial String** (read-only; 40-byte ASCII)
|
||||
- **Trigger on Event** (bool)
|
||||
- **Scheduled Call-Ins** (two time slots with HH:MM each)
|
||||
- **Retry Settings** (count, delay, connection timeout, warm-up time)
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
|
||||
```bash
|
||||
@@ -252,17 +304,58 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
## Key Features
|
||||
|
||||
- [x] Full read pipeline — device info, compliance config, event download with true event-time metadata
|
||||
- [x] Write commands — push compliance config, trigger thresholds, project strings to device
|
||||
- [x] Erase all events — confirmed erase sequence from live MITM capture
|
||||
- [x] Monitor control — start/stop monitoring, read battery/memory/status
|
||||
- [x] Monitor log entries — decode partial 0x2C records (continuous monitoring intervals)
|
||||
- [x] ACH inbound server — accept call-home connections, download events, dedup by key
|
||||
- [x] SQLite persistence — events, monitor log, and session history in `seismo_relay.db`
|
||||
- [x] SFM REST API — device control + DB query endpoints, live device cache
|
||||
**Device support:**
|
||||
- [x] Full read/write/erase pipelines
|
||||
- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings)
|
||||
- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries)
|
||||
- [x] Monitor control (start/stop, status polling, battery/memory)
|
||||
- [x] Monitor log entries (continuous monitoring intervals without full waveform download)
|
||||
|
||||
**Data persistence:**
|
||||
- [x] SQLite database (`seismo_relay.db`) with 4 tables: ach_sessions, events, monitor_log, plus false_trigger flag
|
||||
- [x] Deduplication by waveform key (handles re-runs and repeat call-homes)
|
||||
- [x] Post-erase key-reuse detection (tracks high-water mark)
|
||||
- [x] Session state (`ach_state.json`) with downloaded keys and max key
|
||||
|
||||
**REST API:**
|
||||
- [x] Live device endpoints with in-memory caching (`_LiveCache`)
|
||||
- [x] Cache statistics (`/cache/stats`) and manual invalidation (`/cache/device`)
|
||||
- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH)
|
||||
- [x] Call Home config read/write endpoints
|
||||
- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`)
|
||||
|
||||
**File output (v0.7+, byte-perfect as of v0.14.3):**
|
||||
- [x] Blastware-compatible `.AB0` / `.G10` file generation (waveform + metadata)
|
||||
- [x] Multi-channel waveform decode from SUB 5A bulk stream
|
||||
- [x] Second-resolution timestamp encoding in Blastware filename
|
||||
- [x] **Byte-perfect against BW reference captures** (verified across 2-sec / 3-sec / 10-sec event durations, both event 0 and event N continuation events)
|
||||
- [x] STRT-bounded chunk walk + correct event-N probe counter + partial DLE stuffing of `0x10` in 5A params (the four fixes that landed in v0.14.0–v0.14.3)
|
||||
|
||||
**Capture tools:**
|
||||
- [x] Serial-to-TCP bridge with raw BW/S3 capture (s3_bridge.py, defaults to auto-capture)
|
||||
- [x] GUI bridge with raw capture checkboxes (gui_bridge.py)
|
||||
- [x] ACH inbound server with bidirectional capture (ach_server.py saves raw_tx + raw_rx)
|
||||
- [x] Transparent TCP MITM proxy for live BW session capture (ach_mitm.py)
|
||||
|
||||
**Analysis tools:**
|
||||
- [x] s3_analyzer.py — session parser, frame differ, Claude export
|
||||
- [x] gui_analyzer.py — standalone analyzer GUI
|
||||
- [x] frame_db.py — SQLite frame database for capture analysis
|
||||
|
||||
**seismo_lab.py GUI:**
|
||||
- [x] Bridge tab — Serial/TCP mode selector with raw capture options
|
||||
- [x] Analyzer tab — BW/S3 capture playback and differencing
|
||||
- [x] Download tab — Live wire-byte capture during event download
|
||||
- [x] Console tab — Logging and diagnostics
|
||||
|
||||
## Roadmap (Future)
|
||||
|
||||
- [ ] Verify 30-sec event download — body may exceed `0xFFFF` and force the device into a different `end_key` encoding (none of 2/3/10-sec test cases hit this boundary)
|
||||
- [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing
|
||||
- [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first)
|
||||
- [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object
|
||||
- [ ] Modem manager — push RV50/RV55 configs via Sierra Wireless API
|
||||
- [ ] Histogram mode recording support (5A stream analysis for mode 0x03)
|
||||
- [ ] Call Home dial_string write support (requires DLE escaping for embedded control characters)
|
||||
|
||||
@@ -110,6 +110,10 @@
|
||||
| 2026-04-21 | §7.6.2, §5.3 | **CORRECTED — compliance_raw contains wire-encoded bytes, NOT logical bytes.** S3FrameParser appends DLE+ETX inner-frame pairs as two literal bytes to the frame body. Any `0x03` values in the compliance config appear in `compliance_raw` as `\x10\x03` (two bytes), not as a single `0x03`. The previous claim "S3FrameParser handles this transparently so compliance_raw contains logical (destuffed) bytes" was wrong. Consequence: `compliance_raw` is the wire-encoded E5 payload; anchor-relative reads work correctly because the anchor position automatically accounts for any DLE-encoded bytes before it. For write-back, round-tripping `compliance_raw` verbatim sends the correct wire bytes to the device. **DLE ETX escaping in write frames:** Blastware escapes `0x03` bytes in write frame data as `\x10\x03` on wire; our `build_bw_write_frame` does not (writes data raw). Device is confirmed to accept raw writes for all tested modes — likely uses the offset/length field for write frame framing, not ETX scanning. |
|
||||
| 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. |
|
||||
| 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. |
|
||||
| 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. |
|
||||
| 2026-05-04 | §7.8.5, §7.8.8 | **CORRECTED — Event-N probe counter is just `start_offset`, NOT `start_offset + 0x0046`.** The `+0x46` formula in the original §7.8.5 was based on calling the off=0x2C boundary key the "start_key", but in the iteration walk `cur_key` passed into `read_bulk_waveform_stream` is always the off=0x46 WAVEHDR record key from 1F (the partial-record skip path in `get_events` re-runs 1F to advance past 0x2C boundary records). Adding +0x46 placed the probe one WAVEHDR past the actual event start; the response no longer contained STRT at byte 17, `parse_strt_end_offset` returned None, and the chunk loop fell back to the `max_chunks=128` cap, walking ~110 chunks of post-event circular-buffer garbage. Confirmed against both the 5-1-26 "copy 2nd address" capture (probe at counter=0x2238 with key=01112238) and the 5-4-26 BW 2-sec event capture. Fixed in protocol.py `read_bulk_waveform_stream` v0.14.1. |
|
||||
| 2026-05-05 | §7.8.1 (rule #3 added) | **CONFIRMED — Partial DLE stuffing of `0x10` bytes in 5A params region.** The device's de-stuffing rule for the SUB 5A params region is: `10 10` → `10`, `10 02/03/04` → kept literal (inner-frame markers), `10 X` for any other X → de-stuffs to just `X` (drops the `0x10`). Therefore any `0x10` byte in the logical params followed by a byte NOT in {0x02, 0x03, 0x04, 0x10} MUST be doubled on the wire. This affects counters with `0x10` in the high byte — most importantly counter=`0x1000`, where logical params bytes `... 10 00 ...` were being sent raw and the device de-stuffed `10 00` to just `00`, returning the response for counter=0x0000 (= the file header + STRT). That STRT block then ended up embedded in the assembled file body at file offset `0x1016` and Blastware refused to open the file. This was the root cause of the long-standing ">1-sec event 0 won't open in BW" pattern (1-sec events worked because their `end_offset < 0x1000`, so no chunk request ever needed counter `0x10__`). All 17 5A request frames in the 5-1-26 bwcap3sec capture (probe + 2 meta + 13 samples + TERM) now match BW byte-for-byte after the fix. Fixed in framing.py `build_5a_frame` v0.14.3. |
|
||||
| 2026-05-05 | §7.8 / Blastware file format | **CONFIRMED — File body assembly is contiguous concatenation, no de-duplication.** The "duplicate header+STRT strip" hack from v0.13.x was actively destroying valid waveform data — sample chunks at counter `0x1000` and beyond often coincidentally contain the byte sequence `00 12 03 00 STRT` in their delta-encoded ADC stream, and the strip was zeroing 25 bytes per match. Removed in v0.14.2. The correct file body is: probe contribution + meta@0x1002 + meta@0x1004 + sample contributions in stream order + TERM contribution. Verified byte-perfect against BW reference `M529LKIQ.G10` (8708 bytes, 0 differences) when fed the same A5 frames as the BW capture. |
|
||||
|
||||
---
|
||||
|
||||
@@ -260,7 +264,7 @@ Step 4 — Device sends actual data payload:
|
||||
| `0A` | **WAVEFORM HEADER READ** | Checks record type for a given waveform key. Variable DATA_LENGTH: 0x30=full bin, 0x26=partial bin. Key at params[4..7]. Required before every 1F call to establish device waveform context. | ✅ CONFIRMED 2026-03-31 |
|
||||
| `0C` | **FULL WAVEFORM RECORD** | Downloads 210-byte waveform/histogram record. Sub_code at byte[1]: 0x10=Waveform (9-byte timestamp hdr), 0x03=Waveform-continuous (10-byte hdr, 1-byte shift). PPV floats at label+6 (search "Tran"/"Vert"/"Long"/"MicL"). Peak Vector Sum at tran_label−12 (NOT fixed offset). Key at params[4..7], DATA_LENGTH=0xD2. | ✅ CONFIRMED 2026-04-03 |
|
||||
| `1F` | **EVENT ADVANCE** | Advances to next waveform key. Token byte at params[7] (⚠️ NOT params[6]): 0x00=browse (all-zero params), 0xFE=download (arm 5A state machine). Returns next key at data[11:15]; null sentinel when data[15:19]=0x00000000. Requires preceding 0A to establish context. Browse 1F must ONLY be called after successful 5A — calling it after a failed 5A disrupts device state for the next event's 5A probe. | ✅ CONFIRMED 2026-04-06 |
|
||||
| `5A` | **BULK WAVEFORM STREAM** | Bulk download of raw ADC sample data. Non-standard frame format: offset_hi=0x10 sent raw (not DLE-stuffed), DLE-aware checksum. Requires 1E-arm + 0C + 1F(0xFE) + POLL×3 before first probe. A5[7] contains event-time metadata (Project:/Client:/User Name:/Seis Loc:). 9+ A5 frames for full waveform; stop_after_metadata=True exits after A5[7]. | ✅ CONFIRMED 2026-04-06 |
|
||||
| `5A` | **BULK WAVEFORM STREAM** | Bulk download of raw ADC sample data. Non-standard frame format: offset_hi=0x10 sent raw (not DLE-stuffed), DLE-aware checksum, **partial DLE stuffing of 0x10 in params** (`10 X` where X∉{02,03,04,10} must be doubled to `10 10 X` — see §7.8). Requires 1E-arm + 0C + 1F(0xFE) + POLL×3 before first probe. Walk: probe at counter=`start_offset` (event 1: 0x0000) → metadata pages 0x1002 + 0x1004 (event 1 only) → sample chunks at 0x0600, 0x0800, …, step 0x0200, bounded by `end_offset` parsed from STRT@data[17] of probe response → TERM frame at residual offset_word. Project:/Client:/User Name:/Seis Loc: live in the metadata pages, NOT in the sample-chunk stream. | ✅ CONFIRMED 2026-05-05 (BYTE-PERFECT vs BW capture) |
|
||||
| `24` | **WAVEFORM PAGE A?** | Paged waveform read, possibly channel group A. | 🔶 INFERRED |
|
||||
| `25` | **WAVEFORM PAGE B?** | Paged waveform read, possibly channel group B. | 🔶 INFERRED |
|
||||
| `09` | **UNKNOWN READ A** | Read command, response (`F6`) returns 0xCA (202) bytes. Purpose unknown. | 🔶 INFERRED |
|
||||
@@ -836,6 +840,20 @@ MicL: 39 64 1D AA = 0.0000875 psi
|
||||
|
||||
### 7.6 Bulk Waveform Stream (SUB A5) — Raw ADC Sample Records
|
||||
|
||||
> ⛔ **§7.6 below describes the deprecated `0x0400`-step walk and is RETAINED FOR HISTORICAL CONTEXT ONLY.**
|
||||
> The "A5[7] is metadata", "A5[9] is terminator", and chunk-counter frame-index claims in this section
|
||||
> are all artifacts of the broken walk that was overrunning past event end by ~5×.
|
||||
>
|
||||
> **For the corrected protocol (v0.14.0+), use:**
|
||||
> - **§7.8.5** — chunk addressing (probe at `start_offset`, samples step 0x0200, bounded by STRT `end_offset`)
|
||||
> - **§7.8.6** — TERM frame formula
|
||||
> - **§7.8.7** — fixed metadata pages 0x1002 / 0x1004 (this is where Project / Client / User Name / Seis Loc
|
||||
> strings actually live — NOT in any sample-chunk frame)
|
||||
> - **§7.8.8** — multi-event "Download All" sequence
|
||||
>
|
||||
> The waveform sample encoding (4-channel interleaved s16 LE, 8 bytes per sample-set) described in §7.6.1
|
||||
> below is still correct. Only the frame-indexing claims and metadata-source claims are wrong.
|
||||
|
||||
**Two distinct formats exist depending on recording mode. Both confirmed from captures.**
|
||||
|
||||
---
|
||||
@@ -1118,20 +1136,26 @@ Near-ambient: 0x3C75C28F = 0.015 in/s (histogram event, near-zero ambient)
|
||||
|
||||
**Project strings** — ASCII label-value pairs (search for label, read null-terminated value):
|
||||
```
|
||||
"Project:" → project description (in 0C record ✅)
|
||||
"Client:" → client name (in SUB 5A / A5 frame 7 ✅ — NOT in 0C)
|
||||
"User Name:" → operator / user (in SUB 5A / A5 frame 7 ✅ — NOT in 0C)
|
||||
"Seis Loc:" → sensor location (in SUB 5A / A5 frame 7 ✅ — NOT in 0C)
|
||||
"Extended Notes"→ notes field (in SUB 5A / A5 frame 7 ✅)
|
||||
"Project:" → project description (in 0C record ✅, also mirrored in metadata pages)
|
||||
"Client:" → client name (in SUB 5A metadata pages ✅ — NOT in 0C)
|
||||
"User Name:" → operator / user (in SUB 5A metadata pages ✅ — NOT in 0C)
|
||||
"Seis Loc:" → sensor location (in SUB 5A metadata pages ✅ — NOT in 0C)
|
||||
"Extended Notes"→ notes field (in SUB 5A metadata pages ✅)
|
||||
```
|
||||
|
||||
> ✅ **2026-04-02 — CONFIRMED:** `Client:`, `User Name:`, and `Seis Loc:` are sourced from
|
||||
> **SUB 5A (bulk waveform stream)**, specifically A5 frame 7 of the multi-frame response.
|
||||
> They are NOT present in the 210-byte SUB 0C waveform record. The strings reflect the
|
||||
> compliance setup that was active when the event was recorded on the device — making SUB 5A
|
||||
> the authoritative source for true event-time metadata. The `get_events()` client method
|
||||
> now issues a SUB 5A request after each 0C download (`stop_after_metadata=True`) and
|
||||
> overwrites `event.project_info` with the decoded fields.
|
||||
> ✅ **UPDATED 2026-05-05:** `Client:`, `User Name:`, and `Seis Loc:` come from the
|
||||
> dedicated **SUB 5A metadata pages at counter `0x1002` and `0x1004`** — see §7.8.7.
|
||||
> They are NOT present in the 210-byte SUB 0C waveform record.
|
||||
>
|
||||
> An earlier draft of this doc claimed they came from "A5 frame 7" of the bulk waveform
|
||||
> stream — that was an artifact of the deprecated `0x0400`-step walk where the broken
|
||||
> chunk counter formula happened to land sample-chunk fi=7 on top of the 0x1002 metadata
|
||||
> page. Under the corrected v0.14.0+ walk (§7.8.5), sample chunks at `0x1000` / `0x1200`
|
||||
> contain ordinary waveform data, and the metadata pages are read separately.
|
||||
>
|
||||
> The strings reflect the compliance setup that was active when the *monitoring session*
|
||||
> first started (not per-event). `get_events()` reads the metadata pages once at the start
|
||||
> of the SFM session and the decoded values are stamped onto every event in that session.
|
||||
|
||||
---
|
||||
|
||||
@@ -1165,7 +1189,9 @@ return events
|
||||
|
||||
### 7.7.7 Updated Download Loop with SUB 5A Metadata
|
||||
|
||||
> ✅ **Added 2026-04-02.** Confirmed working on BE11529 over TCP/cellular.
|
||||
> ⛔ **The loop in this subsection is DEPRECATED — it uses the broken `stop_after_metadata=True`
|
||||
> hack and the wrong sequence ordering.** See §7.8.5–§7.8.8 for the corrected protocol.
|
||||
> The pseudocode below is preserved as historical record only.
|
||||
|
||||
```python
|
||||
key4, _ = proto.read_event_first() # SUB 1E
|
||||
@@ -1200,13 +1226,25 @@ return events
|
||||
|
||||
### 7.8 SUB 5A — Bulk Waveform Stream (event-time metadata)
|
||||
|
||||
> ✅ **Added 2026-04-02.** Frame format confirmed by reproducing Blastware wire bytes
|
||||
> byte-for-byte from the 1-2-26 BW capture.
|
||||
> ✅ **§7.8.1 (frame format) — added 2026-04-02; v0.14.3 partial DLE stuffing finalized 2026-05-05.**
|
||||
> Frame format confirmed by reproducing Blastware wire bytes byte-for-byte across the 1-2-26
|
||||
> capture (10 frames) and the 5-1-26 bwcap3sec capture (17 frames, all match including the
|
||||
> DLE-stuffed `10 10 00` for counter=0x1000).
|
||||
|
||||
SUB 5A initiates a bulk transfer of the raw sample data for a stored event. The response is a
|
||||
sequence of A5 frames. Frame 7 (0-indexed) contains the full compliance setup as it existed
|
||||
when the event was recorded — including `Client:`, `User Name:`, `Seis Loc:`, and
|
||||
`Extended Notes` ASCII label-value pairs.
|
||||
SUB 5A initiates a bulk transfer of the raw sample data for a stored event. The response is
|
||||
a sequence of A5 frames. Project-info ASCII strings (`Project:`, `Client:`, `User Name:`,
|
||||
`Seis Loc:`, `Extended Notes`) live in the dedicated metadata pages at counter `0x1002`
|
||||
and `0x1004` (see §7.8.7), not in the sample-chunk stream.
|
||||
|
||||
**For the corrected protocol read in order:**
|
||||
- §7.8.1 — frame format (raw `offset_hi`, DLE-aware checksum, partial DLE stuffing of params)
|
||||
- §7.8.5 — chunk addressing (probe → metadata pages → samples → TERM, all bounded by `end_offset`)
|
||||
- §7.8.6 — TERM frame formula
|
||||
- §7.8.7 — fixed metadata pages 0x1002 / 0x1004
|
||||
- §7.8.8 — multi-event "Download All" sequence
|
||||
|
||||
§7.8.2–§7.8.4 are retained as historical record of earlier (incorrect) understandings —
|
||||
do not implement against them.
|
||||
|
||||
#### 7.8.1 Frame Format
|
||||
|
||||
@@ -1217,7 +1255,7 @@ SUB 5A uses a **non-standard frame layout** that differs from all other BW→S3
|
||||
41 02 10 10 00 5A 00 ^^raw^^ ^^raw^^ ^^stuffed^^
|
||||
```
|
||||
|
||||
Two critical differences from `build_bw_frame`:
|
||||
Three critical differences from `build_bw_frame`:
|
||||
|
||||
1. **`offset_hi` is sent raw, not DLE-stuffed.** When `offset_hi = 0x10`, the wire carries
|
||||
a bare `0x10` — NOT the stuffed `10 10` that `build_bw_frame` would produce. The device
|
||||
@@ -1226,7 +1264,49 @@ Two critical differences from `build_bw_frame`:
|
||||
2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen,
|
||||
only `XX` is added to the running sum; lone bytes are added normally.
|
||||
|
||||
#### 7.8.2 Request Sequence
|
||||
3. **Partial DLE stuffing of `0x10` bytes in the params region** (CONFIRMED 2026-05-05).
|
||||
The device's de-stuffing rule for the params region is:
|
||||
|
||||
- `10 10` → de-stuffs to `10`
|
||||
- `10 02 / 03 / 04` → kept literal (these are inner-frame markers)
|
||||
- `10 X` for other X → de-stuffs to just `X` (drops the leading `0x10`)
|
||||
|
||||
Therefore any `0x10` byte in the *logical* params that is followed by a byte NOT in
|
||||
`{0x02, 0x03, 0x04, 0x10}` MUST be doubled on the wire (`10 X` → `10 10 X`) so the
|
||||
device's de-stuffer reproduces the original `10 X` pair. This applies most commonly
|
||||
to counters with `0x10` in the high byte (e.g. counter=`0x1000` produces logical
|
||||
params bytes `... 10 00 ...`, which BW encodes on the wire as `... 10 10 00 ...`).
|
||||
Without this stuffing the device interprets counter=`0x1000` as `0x0000` and returns
|
||||
the probe response (= a copy of the file header + STRT record); that STRT block then
|
||||
ends up embedded in the assembled file body and Blastware refuses to open the file.
|
||||
|
||||
`0x10` bytes in `offset_hi` are still written RAW per (1) above — only the params
|
||||
region has this stuffing requirement. Metadata-page params for counter `0x1002` /
|
||||
`0x1004` survive without stuffing because `10 02` / `10 04` fall in the "kept literal"
|
||||
carve-out.
|
||||
|
||||
Verified against BW 5-1-26 bwcap3sec frame 20: params logical bytes
|
||||
`00 01 11 10 00 00 00 00 00 00 00` (counter=0x1000) are encoded on the wire as
|
||||
`00 01 11 10 10 00 00 00 00 00 00 00` (12 wire bytes for 11 logical bytes).
|
||||
|
||||
#### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol)
|
||||
|
||||
> ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new
|
||||
> BW MITM captures (4-27-26 + 5-1-26) prove the actual chunk increment is **0x0200**, the
|
||||
> chunk loop is bounded by `end_offset` from the STRT record (not by chunk count or by a
|
||||
> device-side timeout), and the TERM frame's `offset_word=0x005A` magic is incorrect — the
|
||||
> real TERM offset_word is computed from `end_offset` and the last chunk address. Under the
|
||||
> deprecated formula SFM over-reads roughly 5× past the actual event end into post-event
|
||||
> circular-buffer garbage, corrupting reconstructed Blastware files for any waveform ≥ 2 sec.
|
||||
>
|
||||
> The whole "stop_after_metadata + one extra chunk + 0e 08 footer" workaround in this
|
||||
> section was compensating for the missing end_offset bound. It is obsoleted by the
|
||||
> STRT-bounded walk in §7.8.5.
|
||||
>
|
||||
> **Read this section for historical context only.** For the correct protocol, jump to:
|
||||
> - §7.8.5 — chunk addressing and the STRT end_offset
|
||||
> - §7.8.6 — TERM frame formula
|
||||
> - §7.8.7 — fixed metadata pages 0x1002 and 0x1004
|
||||
|
||||
| Frame | offset_word | counter | params | Purpose |
|
||||
|---|---|---|---|---|
|
||||
@@ -1237,51 +1317,32 @@ Two critical differences from `build_bw_frame`:
|
||||
| … | … | … | … | … |
|
||||
| Termination | `0x005A` | `max(key4[2:4], 0x0400) + N * 0x0400` | 10 bytes | End transfer |
|
||||
|
||||
> ⚠️ **2026-04-06 CORRECTED — chunk counter is `key4[2:4] + (N-1) * 0x0400`.**
|
||||
> The 4-2-26 BW TX capture showed counter=0x1004 for chunk 1 of key `01110000`, leading to
|
||||
> an interim "monotonic n * 0x0400" formula. This was accidentally correct because
|
||||
> `key4[2:4] == 0x0000` for that event.
|
||||
>
|
||||
> **2026-04-24 CORRECTION:** The counter is an absolute circular-buffer address.
|
||||
> BW's true formula is `key4[2:4] + (chunk_num - 1) * 0x0400` where `key4[2:4]` is the
|
||||
> event's storage base offset (`(key4[2]<<8) | key4[3]`). For keys where
|
||||
> `key4[2:4] != 0x0000` (e.g. key `01111884`), using `n * 0x0400` sends requests into the
|
||||
> wrong buffer region — the device returns data from a completely different event.
|
||||
>
|
||||
> **2026-04-26 FINAL CORRECTION:** The formula `key4[2:4] + (N-1) * 0x0400` is wrong when
|
||||
> `key4[2:4] == 0x0000` (e.g. event key `01110000`, the very first event after a device erase).
|
||||
> Counter=0x0000 for chunk 1 is the same address as the probe frame — the device re-returns
|
||||
> the STRT record data instead of waveform payload (frame 1 has len=1097, same as probe, and
|
||||
> contains `b"STRT\xff\xfe"`, contributing zero waveform bytes).
|
||||
> Final formula: `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400`.
|
||||
> For key `01110000`: chunk 1 = 0x0400 (confirmed working, empirical test 2026-04-06).
|
||||
> For key `0111245a`: chunk 1 = 0x245a (unchanged, confirmed from 4-3-26 capture).
|
||||
> Historical correction notes (left in place to deter re-derivation of the same wrong formula):
|
||||
> the table above was the result of three iterative "corrections" between 2026-04-06 and
|
||||
> 2026-04-26 that progressively narrowed in on the wrong answer because every test was on
|
||||
> events with `key4[2:4]=0` and the device responds to whatever counter you ask for. The
|
||||
> 5-1-26 captures with a non-zero start_key event (`01112238`) finally exposed the bug.
|
||||
|
||||
The `stop_after_metadata=True` flag causes the loop to stop as soon as `b"Project:"` is
|
||||
found in the accumulated A5 frame data, typically after 4–9 chunks. A termination frame
|
||||
is always sent before returning.
|
||||
The `stop_after_metadata=True` flag (deprecated as a primary loop-exit) scanned for
|
||||
`b"Project:"` in the chunk stream because the metadata strings happened to be reachable
|
||||
when the broken 0x0400-step walk passed the global metadata pages at 0x1002/0x1004. Under
|
||||
the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004,
|
||||
not from the sample-chunk stream — see §7.8.7.
|
||||
|
||||
**IMPORTANT — one extra chunk required after "Project:" for valid file footer (confirmed 2026-04-23):**
|
||||
When writing a Blastware-compatible waveform file, stopping immediately at "Project:" and
|
||||
sending termination produces an empty termination response with no footer bytes (`0e 08`
|
||||
marker missing). Blastware downloads exactly **one more chunk** after finding "Project:"
|
||||
before sending termination — that extra chunk primes the device to return valid footer
|
||||
bytes (monitoring start/stop timestamps) in the termination response.
|
||||
#### 7.8.3 A5 Frame Layout — DEPRECATED 2026-05-01
|
||||
|
||||
`read_bulk_waveform_stream(stop_after_metadata=True)` implements this: after the "Project:"
|
||||
chunk is received, one additional chunk is requested before breaking. The termination
|
||||
response (`include_terminator=True`) then contains the correct `0e 08` footer.
|
||||
> ⛔ **The "Frame 7 carries the compliance text block" claim below is WRONG.** It was
|
||||
> an artifact of the deprecated `0x0400`-step walk where the broken counter formula
|
||||
> happened to land sample-chunk fi=7 on top of the 0x1002 metadata page in flash.
|
||||
> Under the corrected v0.14.0+ walk (§7.8.5), Frame 7 of the sample-chunk sequence is
|
||||
> just sample-chunk #5 (counter=0x1000), and contains either ordinary waveform data or —
|
||||
> critically when DLE-stuffing of params is wrong (§7.8.1.3) — a duplicate file header +
|
||||
> STRT block when the device misinterprets counter=0x1000 as 0x0000. See §7.8.7 for the
|
||||
> actual source of these strings.
|
||||
|
||||
**do NOT use `full_waveform=True` for Blastware file writing** — for events with long
|
||||
post-event silence (35 chunks), the silence chunks contain embedded device-internal
|
||||
pointer structures that produce spurious STRT markers in the file body. Blastware only
|
||||
downloads 4–5 chunks (metadata + one signal chunk) regardless of event length.
|
||||
|
||||
#### 7.8.3 A5 Frame Layout
|
||||
|
||||
Each A5 response frame contains a chunk of raw bulk data. Frame 7 of the stream carries the
|
||||
compliance text block with all project-info label-value pairs. The `client` layer searches
|
||||
for ASCII labels with a null-terminated value read:
|
||||
Historical claim (NOT TO BE IMPLEMENTED): each A5 response frame contains a chunk of raw
|
||||
bulk data; Frame 7 of the stream carries the compliance text block with all project-info
|
||||
label-value pairs:
|
||||
|
||||
```
|
||||
"Project:" → null-terminated project name
|
||||
@@ -1291,17 +1352,23 @@ for ASCII labels with a null-terminated value read:
|
||||
"Extended Notes" → null-terminated notes
|
||||
```
|
||||
|
||||
All five fields reflect the **setup at event-record time**, not the current device config.
|
||||
All five fields do reflect the **setup at event-record time**, not the current device
|
||||
config. But the source is the metadata pages (§7.8.7), not "Frame 7" of the sample
|
||||
stream.
|
||||
|
||||
#### 7.8.4 End-of-Stream Behaviour and Chunk Timing
|
||||
#### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01
|
||||
|
||||
> ✅ **Confirmed 2026-04-06** — empirical observation on BE11529 (S338.17) over TCP/cellular.
|
||||
> The "1 raw byte then silence" pattern documented below was originally interpreted as
|
||||
> "the device's natural end-of-event signal." The 5-1-26 captures show this is actually
|
||||
> the device's response when the requester has walked **past** the addressable buffer
|
||||
> region (i.e. ~5× past the actual event end under the deprecated 0x0400-step walk).
|
||||
> Under the corrected STRT-bounded walk (§7.8.5), the stream ends cleanly with the TERM
|
||||
> frame's response — no timeout, no 1-byte teaser. The fallback below remains useful as
|
||||
> defensive handling for malformed events but should not be the primary loop-exit.
|
||||
|
||||
**End-of-stream signal:** After sending all waveform chunks, the device sends exactly **1 raw byte** in response to the next chunk request, then goes silent. This byte is not a complete DLE-framed A5 response — `S3FrameParser.bytes_fed` reports 1 and no frame is ever assembled. This is the device's natural end-of-stream indicator.
|
||||
|
||||
Handling logic in `read_bulk_waveform_stream`:
|
||||
**Defensive fallback handling in `read_bulk_waveform_stream`:**
|
||||
```
|
||||
TimeoutError caught:
|
||||
TimeoutError caught (rare under corrected walk):
|
||||
if bytes_fed > 0 AND frames already collected:
|
||||
→ graceful end-of-stream; break loop; proceed to termination frame
|
||||
else (bytes_fed == 0, no prior frames):
|
||||
@@ -1313,14 +1380,15 @@ TimeoutError caught:
|
||||
| Metric | Observed value |
|
||||
|---|---|
|
||||
| Chunk response time | ~1 s per chunk |
|
||||
| Chunks for a 9,306-sample event | 35 chunks |
|
||||
| Data per chunk (active signal) | 1,036–1,123 bytes |
|
||||
| Data per chunk (post-event silence) | 1,036 bytes (uniform) |
|
||||
| Chunks for a 2-sec event (corrected walk) | 14 (12 sample chunks + 2 metadata pages) + TERM |
|
||||
| Chunks for a 3-sec event (corrected walk) | 18 (16 sample chunks + 2 metadata pages) + TERM |
|
||||
| Chunks for a continuation event (corrected walk) | ~15 sample chunks + TERM (no metadata reread) |
|
||||
| Chunks under deprecated walk for 2-3 sec event | 37 (over-reads ~5×) |
|
||||
| Data per chunk (corrected, 0x0200 size) | ~540–575 bytes wire (= 0x0200 payload + framing) |
|
||||
| Data per chunk (deprecated 0x0400 step) | 1,036–1,123 bytes wire (= 0x0400 payload + framing) |
|
||||
| Safe recv timeout per chunk | **10 s** (10× typical) |
|
||||
| Default transport timeout | 120 s → ~2-min stall at end-of-stream |
|
||||
|
||||
Chunks with uniform 1,036-byte payload (chunks 17–35 in the observed event) contain all-zero ADC samples — the device continues recording silence until the configured record time expires before terminating the stream.
|
||||
|
||||
**ADC count-to-physical conversion — ✅ CONFIRMED 2026-04-17:**
|
||||
|
||||
Raw samples are signed 16-bit integers (−32,768 to +32,767). Source: Interface Handbook §4.5.
|
||||
@@ -1339,6 +1407,201 @@ where `geo_range = 1.61133 V × 6.206053 = 10.000 in/s` is the Normal (Gain=1) f
|
||||
|
||||
`_decode_a5_waveform()` contains `elif fi == 9: continue` from an earlier assumption that frame index 9 is always the device terminator. For streams with more than 9 frames, frame 9 is live waveform data. The skip discards ~1,070 bytes (~133 sample-sets) per event. Terminator detection should use `page_key == 0x0000`, not frame index. This skip should be removed.
|
||||
|
||||
#### 7.8.5 Chunk addressing and the STRT end_offset (NEW 2026-05-01) ✅
|
||||
|
||||
> ✅ Confirmed across 3 events (4-27-26 + 5-1-26 captures).
|
||||
|
||||
`params[0]` is always `0x00`. `params[1:5]` is a 4-byte absolute device flash-buffer
|
||||
address — equivalently, "the key of the page being requested." The device returns 0x0200
|
||||
(= 512) bytes starting at that address. Increments between consecutive sample chunks are
|
||||
**0x0200, NOT 0x0400** (the previous 0x0400 figure was a Blastware-side artifact / our
|
||||
implementation's bug — see §7.8.2).
|
||||
|
||||
##### STRT record (data layout in the first A5 response)
|
||||
|
||||
The first A5 response (the probe response, or the first chunk for continuation events)
|
||||
contains a **STRT record** at byte offset 17 of `data`:
|
||||
|
||||
```
|
||||
data[ 0:14] echoes request: [chunk_size_hi=0x02 / 0x04 ...] [00] [01 11] [counter_hi counter_lo] [00 × 8] [00 12]
|
||||
data[14:17] 10 03 00 ← inner DLE+ETX frame separator (preserved literally)
|
||||
data[17:21] "STRT" ← magic
|
||||
data[21:23] ff fe ← sentinel
|
||||
data[23:27] end_key ← 4-byte key of where this event ENDS
|
||||
data[27:31] start_key ← 4-byte key of where this event STARTS
|
||||
data[31:33] uint16 BE ← ?? sample count or byte count, varies (not yet decoded)
|
||||
data[33:35] uint16 BE ← ??
|
||||
data[35] 0x46 ← record type marker (waveform full record)
|
||||
data[36:] additional pointers / first sample bytes — content varies by event
|
||||
```
|
||||
|
||||
`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**.
|
||||
Use it to bound the chunk loop and to compute the TERM frame.
|
||||
|
||||
##### Chunk pattern by event location in buffer
|
||||
|
||||
**Event 1 / start_key[2:4] = 0x0000** (first event after erase or wrap):
|
||||
|
||||
```
|
||||
1. Probe at counter = 0x0000 (params[1:5] = full key)
|
||||
2. Read fixed metadata pages counter = 0x1002, then 0x1004
|
||||
3. Walk sample chunks counter = 0x0600, 0x0800, …, by 0x0200,
|
||||
up to but not including end_offset & 0xFE00
|
||||
4. TERM (see §7.8.6)
|
||||
```
|
||||
|
||||
The range `[0x0046, 0x0600)` is skipped — likely some pre-event firmware-reserved area for
|
||||
the first slot in a freshly-erased buffer. Harmless to skip; BW does the same.
|
||||
|
||||
**Event 2+ / start_key[2:4] != 0x0000** (continuation events in a populated buffer):
|
||||
|
||||
```
|
||||
1. First chunk at counter = start_key[2:4] ← acts as both probe and first
|
||||
sample chunk; response carries STRT at byte 17
|
||||
2. Walk sample chunks counter += 0x0200 each
|
||||
3. TERM
|
||||
```
|
||||
|
||||
**`start_key` here is the off=0x46 WAVEHDR record key returned by 1F** (e.g. `01112238`),
|
||||
NOT the off=0x2C boundary key that immediately precedes it. An earlier draft of this
|
||||
spec described event-N as "probe at start + 0x46" — that formula was correct only if
|
||||
"start" meant the boundary key (0x21F2 in the 5-1-26 event 2 case). In the iteration
|
||||
walk used by SFM and BW, `cur_key` passed into the 5A flow is always the off=0x46 key,
|
||||
so the probe counter equals `cur_key[2:4]` with no extra offset. Adding +0x46 places
|
||||
the probe one WAVEHDR past the actual event start, the response no longer contains
|
||||
STRT at byte 17, and the chunk loop falls back to the `max_chunks` cap.
|
||||
|
||||
Confirmed:
|
||||
- 5-1-26 "copy 2nd address" BW capture: probe counter=0x2238 with key=01112238; A5[0]
|
||||
has STRT@17 with end_offset=0x417E.
|
||||
- 5-4-26 BW 2-sec event capture: same probe counter=0x2238, same end_offset=0x417E.
|
||||
|
||||
**No metadata-page reads.** Pages 0x1002/0x1004 are session-global and were already read
|
||||
during event 1 in the same Blastware session. In SFM, treat metadata pages as a once-
|
||||
per-`MiniMateClient.connect()` (or once-per-call-home) read, not per-event.
|
||||
|
||||
##### Verified end_offset values
|
||||
|
||||
| Capture | start_key | end_key | end_offset | event size | sample-chunk start |
|
||||
|---|---|---|---|---|---|
|
||||
| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B | 0x0600 (event-1 case) |
|
||||
| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B | 0x0600 (event-1 case) |
|
||||
| 5-1-26 "copy 2nd address" / DA event 2 | `01112238` (= 1F result) | `0111417E` | `0x417E`, span 0x1F8C = 8,076 B | 0x2238 (= cur_key[2:4]) |
|
||||
| 5-4-26 BW 2-sec event | `01112238` | `0111417E` | `0x417E` | 0x2238 (= cur_key[2:4]) |
|
||||
|
||||
#### 7.8.6 TERM Frame Formula (NEW 2026-05-01) ✅
|
||||
|
||||
> ✅ Confirmed across 3 events. Replaces the deprecated `offset_word=0x005A` / `params[2] = key4[2]` formula in §7.8.2.
|
||||
|
||||
The TERM frame fetches the partial last chunk and the file footer. Its response payload
|
||||
contains the bytes between the last full 0x0200-aligned chunk and `end_offset` — typically
|
||||
20–520 B — and is **required for reconstructing the Blastware waveform file**. Append the
|
||||
TERM response data to the chunk stream like any other A5 frame.
|
||||
|
||||
```
|
||||
last_chunk_counter = address of last full 0x0200-byte chunk read
|
||||
next_boundary = last_chunk_counter + 0x0200
|
||||
TERM offset_word = end_offset - next_boundary
|
||||
TERM params[0] = key[0] (= 0x01 on every observed device)
|
||||
TERM params[1] = key[1] (= 0x11)
|
||||
TERM params[2] = (next_boundary >> 8) & 0xFF
|
||||
TERM params[3] = next_boundary & 0xFF
|
||||
TERM params[4:10] = zeros ← 10-byte params (not 11)
|
||||
|
||||
Frame = build_5a_frame(offset_word, params)
|
||||
```
|
||||
|
||||
The device receives `requested_address = (params[2] << 8) | offset_word` (where offset_word
|
||||
contains both `offset_hi` and `offset_lo` of the 5A frame, with the high bit of offset_hi
|
||||
being effectively `bit 0 of (end_offset >> 8)`). It reconstructs `end_offset` and replies
|
||||
with `(end_offset - next_boundary)` bytes of waveform tail starting at `next_boundary`.
|
||||
|
||||
##### Verification
|
||||
|
||||
| Event | end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] | TERM response size |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 2-sec | `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ | 208 B |
|
||||
| 3-sec | `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ | 520 B |
|
||||
| Event-2 | `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ | (not measured directly; same pattern) |
|
||||
|
||||
Equivalent way to write the formula:
|
||||
- `offset_word = end_offset & 0x01FF` — low 9 bits of end_offset
|
||||
- `params[2:4] = (end_offset & 0xFE00) BE` — high 7 bits of end_offset, low byte zeroed
|
||||
|
||||
(The two forms are arithmetically identical to `end_offset - next_boundary` and
|
||||
`next_boundary` because `next_boundary = end_offset & 0xFE00` whenever the chunk loop
|
||||
stopped at the last full 0x0200 boundary below end_offset.)
|
||||
|
||||
#### 7.8.7 Fixed Metadata Pages 0x1002 / 0x1004 (NEW 2026-05-01) 🔶
|
||||
|
||||
> 🔶 Inferred — observed in BW captures but page contents not yet byte-decoded.
|
||||
|
||||
Two chunk addresses are **GLOBAL** device/session metadata, not event-specific:
|
||||
|
||||
- `counter = 0x1002` — first metadata page
|
||||
- `counter = 0x1004` — second metadata page
|
||||
|
||||
These are at fixed absolute addresses in the device's flash buffer. They contain the
|
||||
session-start compliance-setup ASCII strings — **Project**, **Client**, **User Name**,
|
||||
**Seis Loc**, **Extended Notes** — that under the deprecated 0x0400-step walk used to be
|
||||
discoverable in the sample-chunk stream as "A5 frame 7" content. Under the corrected
|
||||
0x0200-step walk these strings come exclusively from the dedicated metadata-page reads,
|
||||
not from sample chunks.
|
||||
|
||||
##### Caching strategy
|
||||
|
||||
BW reads them ONCE per Blastware session, during event 1's download, and caches them.
|
||||
For SFM:
|
||||
- Read once per `MiniMateClient.connect()` / once per call-home session.
|
||||
- Subsequent events in the same session don't need to re-fetch them.
|
||||
- Their content does not change while iterating events. They DO change when the user
|
||||
applies a new compliance setup (SUB 71 write) — invalidate the cache then.
|
||||
|
||||
##### TODO — content layout
|
||||
|
||||
The byte-for-byte layout of pages 0x1002 and 0x1004 has not been decoded. First task on
|
||||
the implementation side: dump both pages from a fresh capture and verify they include all
|
||||
the strings currently extracted from the deprecated A5 frame 7 of the chunk stream.
|
||||
Compare to the existing `_decode_a5_metadata_into` parser — same string-search anchors
|
||||
(`b"Project:"`, `b"Client:"`, `b"User Name:"`, `b"Seis Loc:"`, `b"Extended Notes"`) likely
|
||||
apply directly.
|
||||
|
||||
#### 7.8.8 "Download All" Sequence (NEW 2026-05-01) ✅
|
||||
|
||||
> ✅ Confirmed from 5-1-26 "Download All" capture (`raw_*_171216_download_all_2events.bin`).
|
||||
|
||||
Before any 5A traffic, BW's "Download All" pre-walks the entire event chain to map keys
|
||||
and event boundaries:
|
||||
|
||||
```
|
||||
SERIAL × 2 → CHCFG → EVT_KEY (1E, all-zero) → key0
|
||||
→ WAVEHDR (0A, key0) → off=0x46 (real event start)
|
||||
→ EVT_NEXT (1F, all-zero) → key1
|
||||
→ WAVEHDR (0A, key1) → off=0x2C (boundary)
|
||||
→ EVT_NEXT → key2
|
||||
→ WAVEHDR (0A, key2) → off=0x46 (real event start)
|
||||
→ EVT_NEXT → key3
|
||||
→ WAVEHDR (0A, key3) → off=0x2C (boundary)
|
||||
→ EVT_NEXT → null sentinel
|
||||
```
|
||||
|
||||
The DATA_LENGTH at `data_rsp.data[5]` (echoed BW offset for the data fetch step)
|
||||
disambiguates real events from boundary markers:
|
||||
|
||||
| WAVEHDR offset | Meaning |
|
||||
|---|---|
|
||||
| `0x46` (= 70) | Real event start key — this key has event data behind it |
|
||||
| `0x2C` (= 44) | Boundary marker — this key is the END of the previous event AND the start of the empty/header gap before the next event |
|
||||
|
||||
Pairs: each real event spans `[real_key, next_real_key)` in the buffer. In the example
|
||||
above: event 1 = `[01110000, 011121F2)`, event 2 = `[01112238, 0111417E)`. Note that the
|
||||
"end of event 1" key (`011121F2`) is also the "boundary key" that comes BEFORE event 2's
|
||||
real start key (`01112238`) — they differ by exactly 0x46 bytes (the event header size).
|
||||
|
||||
After the pre-walk completes, BW downloads each `0x46`-keyed event in turn using the 5A
|
||||
bulk stream protocol from §7.8.5. Use the `0x46` keys, not the `0x2C` keys, as input to
|
||||
`read_bulk_waveform_stream`.
|
||||
|
||||
---
|
||||
|
||||
## 7.9 Compliance Config Field Inventory (Blastware UI, 2026-04-08) ✅
|
||||
@@ -1370,10 +1633,10 @@ Fields visible in the Blastware "Compliance Setup" dialog. ✅ = byte offset co
|
||||
| Field | Values / Type | Status |
|
||||
|---|---|---|
|
||||
| Enable User Notes | bool | ❓ |
|
||||
| Project | ASCII string | ✅ (sourced from A5 frame 7 via SUB 5A) |
|
||||
| Client | ASCII string | ✅ (sourced from A5 frame 7) |
|
||||
| User Name | ASCII string | ✅ (sourced from A5 frame 7) |
|
||||
| Seis Loc | ASCII string | ✅ (sourced from A5 frame 7) |
|
||||
| Project | ASCII string | ✅ (sourced from SUB 5A metadata pages at counter `0x1002` / `0x1004` — see §7.8.7) |
|
||||
| Client | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) |
|
||||
| User Name | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) |
|
||||
| Seis Loc | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) |
|
||||
| Enable Extended Notes | bool | ❓ |
|
||||
| Extended Notes | ASCII text | ❓ |
|
||||
| Extended Notes Title | ASCII string | ❓ |
|
||||
|
||||
@@ -21,7 +21,15 @@ Typical usage (TCP / modem):
|
||||
|
||||
from .client import MiniMateClient
|
||||
from .models import DeviceInfo, Event, MonitorLogEntry
|
||||
from .transport import SerialTransport, TcpTransport
|
||||
from .transport import CapturingTransport, SerialTransport, TcpTransport
|
||||
|
||||
__version__ = "0.1.0"
|
||||
__all__ = ["MiniMateClient", "DeviceInfo", "Event", "MonitorLogEntry", "SerialTransport", "TcpTransport"]
|
||||
__all__ = [
|
||||
"MiniMateClient",
|
||||
"DeviceInfo",
|
||||
"Event",
|
||||
"MonitorLogEntry",
|
||||
"SerialTransport",
|
||||
"TcpTransport",
|
||||
"CapturingTransport",
|
||||
]
|
||||
|
||||
@@ -639,7 +639,7 @@ def write_blastware_file(
|
||||
strt = b"STRT" + b"\xff\xfe" + key4 + bytes(14) + bytes([rectime & 0xFF])
|
||||
probe_skip = 7 + 21
|
||||
|
||||
log.warning(
|
||||
log.debug(
|
||||
"write_blastware_file: strt_pos_stripped=%d probe_skip=%d "
|
||||
"probe_data_len=%d strt_hex=%s",
|
||||
strt_pos_stripped if strt_pos_stripped >= 0 else -1,
|
||||
@@ -672,11 +672,10 @@ def write_blastware_file(
|
||||
# Do NOT use a5_frames[-1] — if _a5_frames contains stray frames from a
|
||||
# subsequent event (a known get_events side-effect), the last frame will
|
||||
# not be the terminator and the footer will be mis-identified.
|
||||
# TERM detection (v0.14.0): last frame if page_key != 0x0010 (sample marker)
|
||||
term_idx: Optional[int] = None
|
||||
for _i, _f in enumerate(a5_frames):
|
||||
if _f.page_key == 0x0000:
|
||||
term_idx = _i
|
||||
break
|
||||
if a5_frames and a5_frames[-1].page_key != 0x0010:
|
||||
term_idx = len(a5_frames) - 1
|
||||
|
||||
if term_idx is not None:
|
||||
body_frames = a5_frames[:term_idx]
|
||||
@@ -685,68 +684,32 @@ def write_blastware_file(
|
||||
body_frames = a5_frames
|
||||
term_frame = None
|
||||
|
||||
# ── Identify first metadata frame and skip "extra chunks" ───────────────
|
||||
# When extra_chunks_after_metadata=1 in read_bulk_waveform_stream(), the
|
||||
# frame list is: [probe, data..., metadata, extra_chunk, terminator].
|
||||
# The extra_chunk is downloaded to prime the TCP terminator response — its
|
||||
# ADC data is NOT part of the Blastware file body. Skip it.
|
||||
#
|
||||
# Rule: any frame at index strictly between first_metadata_fi and last_fi
|
||||
# (the final frame) is an extra chunk and must be excluded.
|
||||
#
|
||||
# If no metadata frame exists (e.g. full_waveform download), first_metadata_fi
|
||||
# is None and no frames are skipped — all frames contribute normally.
|
||||
first_metadata_fi: Optional[int] = None
|
||||
for _fi_scan, _frame_scan in enumerate(body_frames):
|
||||
if _fi_scan > 0 and any(m in bytes(_frame_scan.data) for m in _METADATA_FRAME_MARKERS):
|
||||
first_metadata_fi = _fi_scan
|
||||
break
|
||||
# Frame contribution loop (v0.14.0 BW-exact walk).
|
||||
# Skip values:
|
||||
# probe (fi=0): probe_skip
|
||||
# meta@0x1002 (fi=1): 13 (6-byte inner header)
|
||||
# meta@0x1004 (fi=2): 13 (6-byte inner header)
|
||||
# sample chunks (fi=3+): 12 (5-byte inner header)
|
||||
last_fi = len(body_frames) - 1
|
||||
|
||||
log.warning(
|
||||
"write_blastware_file: %d body_frames first_metadata_fi=%s last_fi=%d",
|
||||
len(body_frames),
|
||||
str(first_metadata_fi) if first_metadata_fi is not None else "None",
|
||||
last_fi,
|
||||
log.debug(
|
||||
"write_blastware_file: %d body_frames last_fi=%d",
|
||||
len(body_frames), last_fi,
|
||||
)
|
||||
|
||||
all_bytes = bytearray()
|
||||
|
||||
for fi, frame in enumerate(body_frames):
|
||||
# Skip "extra chunk" frames: frames after the first metadata frame but
|
||||
# before the last frame (terminator). These prime the TCP terminator but
|
||||
# their ADC data must NOT appear in the Blastware file body.
|
||||
if (first_metadata_fi is not None
|
||||
and fi > first_metadata_fi
|
||||
and fi < last_fi):
|
||||
log.warning(
|
||||
"write_blastware_file: fi=%d SKIP (extra chunk after metadata fi=%d last_fi=%d)",
|
||||
fi, first_metadata_fi, last_fi,
|
||||
)
|
||||
continue
|
||||
|
||||
if fi == 0:
|
||||
# Probe frame: always process regardless of classification.
|
||||
# It holds the STRT record; probe_skip positions us past it.
|
||||
skip = probe_skip
|
||||
elif fi in (1, 2):
|
||||
skip = 13 # metadata pages
|
||||
else:
|
||||
# ALL subsequent frames are included unconditionally — no filtering on
|
||||
# frame type. In the A5 stream, frame 0 is always the probe response;
|
||||
# frames 1+ are always data (waveform chunks, compliance config, or
|
||||
# compliance continuation). Classification is for logging only.
|
||||
#
|
||||
# DO NOT gate on classify_frame() here:
|
||||
# - "probe_or_strt" at fi>0 is always a false positive — ADC binary
|
||||
# data can coincidentally contain b"STRT\xff\xfe" (confirmed from
|
||||
# live capture: frames 1 and 5 matched on event key=01110000).
|
||||
# - "metadata" frames must be included (compliance config body).
|
||||
# - The compliance block spans 2 frames; skipping either produces a
|
||||
# truncated file that Blastware rejects.
|
||||
skip = 13 if fi == 1 else 12
|
||||
skip = 12 # sample chunks
|
||||
|
||||
contribution = _frame_body_bytes(frame, skip)
|
||||
log.warning("write_blastware_file: fi=%d skip=%d raw_data=%d contribution=%d",
|
||||
fi, skip, len(frame.data), len(contribution))
|
||||
log.debug("write_blastware_file: fi=%d skip=%d raw_data=%d contribution=%d",
|
||||
fi, skip, len(frame.data), len(contribution))
|
||||
all_bytes.extend(contribution)
|
||||
|
||||
# Terminator contributes its content, which ends with the 26-byte footer.
|
||||
@@ -754,7 +717,7 @@ def write_blastware_file(
|
||||
# one shorter than chunk frames' 5-byte inner header. Confirmed 2026-04-21.
|
||||
if term_frame is not None:
|
||||
term_contribution = _frame_body_bytes(term_frame, 11)
|
||||
log.warning(
|
||||
log.debug(
|
||||
"write_blastware_file: term_frame data_len=%d skip=11 "
|
||||
"contribution_len=%d first8=%s",
|
||||
len(term_frame.data),
|
||||
@@ -763,17 +726,49 @@ def write_blastware_file(
|
||||
)
|
||||
all_bytes.extend(term_contribution)
|
||||
|
||||
log.warning(
|
||||
log.debug(
|
||||
"write_blastware_file: all_bytes total=%d last28=%s",
|
||||
len(all_bytes),
|
||||
bytes(all_bytes[-28:]).hex() if len(all_bytes) >= 28 else bytes(all_bytes).hex(),
|
||||
)
|
||||
|
||||
if len(all_bytes) >= 26:
|
||||
# NOTE: The "duplicate header+STRT strip" logic from v0.13.x has been
|
||||
# REMOVED in v0.14.2. Under the v0.14.0 BW-exact 5A walk, body assembly
|
||||
# is just contiguous concatenation of frame contributions in stream order
|
||||
# (probe → meta@0x1002 → meta@0x1004 → samples → TERM), exactly as BW
|
||||
# writes its files. The previous strip was matching the `00 12 03 00 STRT`
|
||||
# byte sequence in legitimate waveform data — sample chunks at counter
|
||||
# 0x1000 and beyond often contain those bytes coincidentally — and
|
||||
# zeroing 25 bytes of valid samples per match. Compared to a known-good
|
||||
# BW reference for the same 3-sec event 0, the strip introduced 26 bytes
|
||||
# of zeros that BW did not have, then propagated alignment differences
|
||||
# through the rest of the body. See decode_test/5-1-26/bw vs SFM diff
|
||||
# at file[0x1012..0x102B] (2026-05-04 analysis).
|
||||
|
||||
# Find the first valid 0e 08 footer marker (v0.14.0).
|
||||
footer_pos = -1
|
||||
pos = 0
|
||||
while True:
|
||||
pos = bytes(all_bytes).find(b"\x0e\x08", pos)
|
||||
if pos < 0 or pos + 26 > len(all_bytes):
|
||||
break
|
||||
yr = (all_bytes[pos + 4] << 8) | all_bytes[pos + 5]
|
||||
if 2015 <= yr <= 2050:
|
||||
footer_pos = pos
|
||||
break
|
||||
pos += 1
|
||||
if footer_pos >= 0:
|
||||
body = bytes(all_bytes[:footer_pos])
|
||||
footer = bytes(all_bytes[footer_pos:footer_pos + 26])
|
||||
log.debug(
|
||||
"write_blastware_file: real 0e 08 footer at all_bytes[%d]; "
|
||||
"truncating %d post-footer bytes",
|
||||
footer_pos, len(all_bytes) - footer_pos - 26,
|
||||
)
|
||||
elif len(all_bytes) >= 26:
|
||||
body = bytes(all_bytes[:-26])
|
||||
footer = bytes(all_bytes[-26:])
|
||||
else:
|
||||
# Fallback: no terminator or very short stream → build footer from event metadata
|
||||
body = bytes(all_bytes)
|
||||
start_dt = _ts_from_model(event.timestamp)
|
||||
stop_dt: Optional[datetime.datetime] = None
|
||||
@@ -784,7 +779,7 @@ def write_blastware_file(
|
||||
+ _encode_ts_be(start_dt)
|
||||
+ _encode_ts_be(stop_dt)
|
||||
+ b"\x00\x01\x00\x02\x00\x00"
|
||||
+ b"\x00\x00" # CRC placeholder
|
||||
+ b"\x00\x00"
|
||||
)
|
||||
|
||||
# ── Write file ───────────────────────────────────────────────────────────
|
||||
|
||||
+66
-22
@@ -1345,6 +1345,11 @@ def _decode_waveform_record_into(data: bytes, event: Event) -> None:
|
||||
event.timestamp = Timestamp.from_continuous_record(data)
|
||||
except Exception as exc:
|
||||
log.warning("continuous record timestamp decode failed: %s", exc)
|
||||
elif event.record_type == "Waveform (Short)":
|
||||
try:
|
||||
event.timestamp = Timestamp.from_short_record(data)
|
||||
except Exception as exc:
|
||||
log.warning("short record timestamp decode failed: %s", exc)
|
||||
|
||||
# ── Peak values (per-channel PPV + Peak Vector Sum) ───────────────────────
|
||||
try:
|
||||
@@ -1636,34 +1641,73 @@ def _decode_a5_waveform(
|
||||
}
|
||||
|
||||
|
||||
def _detect_record_format(data: bytes) -> Optional[str]:
|
||||
"""
|
||||
Detect which timestamp-header format a 210-byte 0C waveform record uses.
|
||||
|
||||
THREE formats observed on BE11529 firmware S338.17:
|
||||
|
||||
"single_shot" — 9-byte header:
|
||||
[day] [0x10] [month] [year_BE:2] [unknown] [hour] [min] [sec]
|
||||
sub_code=0x10 at byte [1]. Year at [3:5].
|
||||
|
||||
"continuous" — 10-byte header:
|
||||
[0x10] [day] [0x10] [month] [year_BE:2] [unknown] [hour] [min] [sec]
|
||||
marker 0x10 at byte [0] AND byte [2]. Year at [4:6].
|
||||
|
||||
"short" — 8-byte header (NEW 2026-05-01):
|
||||
[day] [month] [year_BE:2] [unknown] [hour] [min] [sec]
|
||||
No marker bytes. Year at [2:4].
|
||||
|
||||
Each format has the year (uint16 BE) at a UNIQUE byte position, so we can
|
||||
disambiguate by scanning each candidate position and picking the one
|
||||
where the year falls in a sane range (2015..2050).
|
||||
|
||||
Returns "single_shot" / "continuous" / "short" or None if no format matches.
|
||||
"""
|
||||
if len(data) < 8:
|
||||
return None
|
||||
|
||||
def _sane_year(hi: int, lo: int) -> bool:
|
||||
y = (hi << 8) | lo
|
||||
return 2015 <= y <= 2050
|
||||
|
||||
# Order matters: prefer formats with stronger marker-byte evidence first.
|
||||
if data[1] == 0x10 and len(data) >= 9 and _sane_year(data[3], data[4]):
|
||||
return "single_shot"
|
||||
if (data[0] == 0x10 and data[2] == 0x10
|
||||
and len(data) >= 10 and _sane_year(data[4], data[5])):
|
||||
return "continuous"
|
||||
if _sane_year(data[2], data[3]):
|
||||
return "short"
|
||||
return None
|
||||
|
||||
|
||||
def _extract_record_type(data: bytes) -> Optional[str]:
|
||||
"""
|
||||
Decode the recording mode from byte[1] of the 210-byte waveform record.
|
||||
Return a human-readable name for the waveform record format detected
|
||||
in the first bytes of a 210-byte 0C record.
|
||||
|
||||
Byte[1] is the sub-record code that immediately follows the day byte in the
|
||||
9-byte timestamp header at the start of each waveform record:
|
||||
[day:1] [sub_code:1] [month:1] [year:2 BE] ...
|
||||
|
||||
Confirmed codes (✅ 2026-04-01):
|
||||
0x10 → "Waveform" (continuous / single-shot mode)
|
||||
|
||||
Histogram mode code is not yet confirmed — a histogram event must be
|
||||
captured with debug=true to identify it. Returns None for unknown codes.
|
||||
Maps to the format codes returned by _detect_record_format():
|
||||
"single_shot" → "Waveform"
|
||||
"continuous" → "Waveform (Continuous)"
|
||||
"short" → "Waveform (Short)"
|
||||
None → "Unknown(XX.YY.ZZ)"
|
||||
"""
|
||||
if len(data) < 2:
|
||||
return None
|
||||
code = data[1]
|
||||
if code == 0x10:
|
||||
fmt = _detect_record_format(data)
|
||||
if fmt == "single_shot":
|
||||
return "Waveform"
|
||||
if code == 0x03:
|
||||
# Continuous mode waveform record (confirmed by user — NOT a monitor log).
|
||||
# The byte layout differs from 0x10 single-shot records: the timestamp
|
||||
# fields decode as garbage under the 0x10 waveform layout.
|
||||
# TODO: confirm correct timestamp layout for 0x03 records from a known-time event.
|
||||
if fmt == "continuous":
|
||||
return "Waveform (Continuous)"
|
||||
log.warning("_extract_record_type: unknown sub_code=0x%02X", code)
|
||||
return f"Unknown(0x{code:02X})"
|
||||
|
||||
if fmt == "short":
|
||||
return "Waveform (Short)"
|
||||
if len(data) >= 3:
|
||||
log.warning(
|
||||
"_extract_record_type: unrecognized header: data[0:3]=%02X %02X %02X",
|
||||
data[0], data[1], data[2],
|
||||
)
|
||||
return f"Unknown({data[0]:02X}.{data[1]:02X}.{data[2]:02X})"
|
||||
return None
|
||||
|
||||
def _extract_peak_floats(data: bytes) -> Optional[PeakValues]:
|
||||
"""
|
||||
|
||||
+181
-31
@@ -111,20 +111,24 @@ def build_5a_frame(offset_word: int, raw_params: bytes) -> bytes:
|
||||
verified against this algorithm on 2026-04-02).
|
||||
|
||||
Args:
|
||||
offset_word: 16-bit offset (0x1004 for probe/chunks, 0x005A for term).
|
||||
raw_params: 10 or 11 params bytes (from bulk_waveform_params or
|
||||
bulk_waveform_term_params). 0x10 bytes in params are
|
||||
written RAW — NOT DLE-stuffed. Confirmed 2026-04-06 by
|
||||
comparing wire bytes: BW sends bare `10 04` for chunk 1
|
||||
(counter=0x1004), not stuffed `10 10 04`. Device reads
|
||||
params at fixed byte positions; stuffing shifts the bytes
|
||||
and corrupts the counter, causing device to ignore the frame.
|
||||
offset_word: 16-bit offset. For probe/chunks/metadata pages this is
|
||||
`0x1002`. For the proper TERM frame this is computed by
|
||||
`bulk_waveform_term_v2()` from the STRT-derived
|
||||
`end_offset`.
|
||||
raw_params: 10, 11, or 12 params bytes (from `bulk_waveform_params`
|
||||
for probes/samples, `bulk_waveform_term_v2` for TERM, or
|
||||
a manually-built 12-byte block for the metadata pages
|
||||
0x1002 / 0x1004). See gotcha #3 below — params region
|
||||
uses partial DLE stuffing of 0x10 bytes.
|
||||
|
||||
Returns:
|
||||
Complete frame bytes: [ACK][STX][stuffed_section][chk][ETX]
|
||||
"""
|
||||
if len(raw_params) not in (10, 11):
|
||||
raise ValueError(f"raw_params must be 10 or 11 bytes, got {len(raw_params)}")
|
||||
if len(raw_params) not in (10, 11, 12):
|
||||
# 10 = termination params; 11 = regular probe / chunk params;
|
||||
# 12 = metadata-page params (extra trailing 0x00 — BW byte-perfect quirk
|
||||
# for the two fixed metadata reads at counter=0x1002 and 0x1004).
|
||||
raise ValueError(f"raw_params must be 10/11/12 bytes, got {len(raw_params)}")
|
||||
|
||||
# Build stuffed section between STX and checksum
|
||||
s = bytearray()
|
||||
@@ -134,8 +138,40 @@ def build_5a_frame(offset_word: int, raw_params: bytes) -> bytes:
|
||||
s += b"\x00" # field3
|
||||
s += bytes([(offset_word >> 8) & 0xFF, # offset_hi — raw, NOT stuffed
|
||||
offset_word & 0xFF]) # offset_lo
|
||||
for b in raw_params: # params — NOT DLE-stuffed (raw bytes, match BW wire format)
|
||||
# Params — partial DLE stuffing of 0x10 bytes (CONFIRMED 2026-05-05).
|
||||
#
|
||||
# The device's de-stuffing rule for params is:
|
||||
# • `10 10` → de-stuffs to `10`
|
||||
# • `10 02/03/04` → kept literal (these are inner-frame markers)
|
||||
# • `10 X` other → de-stuffs to just `X` (drops the 0x10)
|
||||
#
|
||||
# So for any 0x10 byte in the *logical* params that is followed by a
|
||||
# byte NOT in {0x02, 0x03, 0x04, 0x10}, we must double the 0x10 on the
|
||||
# wire (`10 X` → `10 10 X`) so the device's de-stuffer reproduces the
|
||||
# original `10 X` pair. Without this, counter values with `0x10` in
|
||||
# the high byte (e.g. counter=0x1000 has params bytes `10 00`) are
|
||||
# silently corrupted to `0x__00` on the device side, and the device
|
||||
# responds for the wrong address — for counter=0x1000 it returns the
|
||||
# probe response (counter=0x0000), which contains the file header +
|
||||
# STRT. That STRT block then lands in the assembled file body and
|
||||
# Blastware rejects the file as malformed.
|
||||
#
|
||||
# Confirmed against BW capture 5-1-26 / bwcap3sec frame 20: params
|
||||
# logical bytes `00 01 11 10 00 00 00 00 00 00 00` (counter=0x1000)
|
||||
# are encoded on the wire as `00 01 11 10 10 00 00 00 00 00 00 00`.
|
||||
# BW frames 13/14 (meta @ 0x1002 / 0x1004) leave `10 02` and `10 04`
|
||||
# raw — the device handles those literal pairs correctly.
|
||||
i = 0
|
||||
while i < len(raw_params):
|
||||
b = raw_params[i]
|
||||
s.append(b)
|
||||
if (
|
||||
b == 0x10
|
||||
and i + 1 < len(raw_params)
|
||||
and raw_params[i + 1] not in (0x02, 0x03, 0x04, 0x10)
|
||||
):
|
||||
s.append(0x10) # double the 0x10 so it survives device de-stuffing
|
||||
i += 1
|
||||
|
||||
# DLE-aware checksum: for 0x10 XX pairs count XX; for lone bytes count them
|
||||
chk, i = 0, 0
|
||||
@@ -398,28 +434,26 @@ def bulk_waveform_params(key4: bytes, counter: int, *, is_probe: bool = False) -
|
||||
|
||||
def bulk_waveform_term_params(key4: bytes, counter: int) -> bytes:
|
||||
"""
|
||||
Build the 10-byte params block for the SUB 5A termination request.
|
||||
⛔ DEPRECATED — DO NOT USE IN NEW CODE.
|
||||
|
||||
The termination request uses offset=0x005A and a DIFFERENT params layout —
|
||||
the leading 0x00 byte is dropped, key4[0:2] shifts to params[0:2], and the
|
||||
counter high byte is at params[2]:
|
||||
This is the v1 termination params helper, paired with the broken
|
||||
`_BULK_TERM_OFFSET = 0x005A` magic offset_word. Together they produce a
|
||||
~100-byte device-side terminator response that does NOT contain the
|
||||
partial-last-chunk waveform tail or the 26-byte file footer. Files
|
||||
reconstructed using this terminator are missing their last ~512 bytes of
|
||||
waveform data and have a synthesized footer that disagrees with what BW
|
||||
would have written.
|
||||
|
||||
params[0] = key4[0]
|
||||
params[1] = key4[1]
|
||||
params[2] = (counter >> 8) & 0xFF
|
||||
params[3:] = zeros
|
||||
**For new code, use `bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`**
|
||||
which computes the correct offset_word + params from the STRT-derived
|
||||
`end_offset`. v2 produces wire bytes that match BW exactly across all
|
||||
tested events (4-27-26 / 5-1-26 / 5-4-26 captures).
|
||||
|
||||
Counter for the termination request = last_regular_counter + 0x0400.
|
||||
|
||||
Confirmed from 1-2-26 BW TX capture: final request (frame 83) uses
|
||||
offset=0x005A, params[0:3] = key4[0:2] + term_counter_hi.
|
||||
|
||||
Args:
|
||||
key4: 4-byte waveform key.
|
||||
counter: Termination counter (= last regular counter + 0x0400).
|
||||
|
||||
Returns:
|
||||
10-byte params block.
|
||||
This function is retained ONLY for the defensive fallback path in
|
||||
`read_bulk_waveform_stream()` that triggers when STRT parsing fails or no
|
||||
chunks are fetched (= a malformed event or an unexpected device state).
|
||||
The fallback already logs a WARNING when it activates; if you see that
|
||||
warning, the bug is upstream — STRT should have been parseable.
|
||||
"""
|
||||
if len(key4) != 4:
|
||||
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
|
||||
@@ -430,6 +464,123 @@ def bulk_waveform_term_params(key4: bytes, counter: int) -> bytes:
|
||||
return bytes(p)
|
||||
|
||||
|
||||
def bulk_waveform_term_v2(
|
||||
key4: bytes,
|
||||
end_offset: int,
|
||||
last_chunk_counter: int,
|
||||
) -> tuple[int, bytes]:
|
||||
"""
|
||||
Compute the SUB 5A TERM frame's offset_word and 10-byte params block.
|
||||
|
||||
Confirmed across 3 events (4-27-26 + 5-1-26 captures):
|
||||
|
||||
next_boundary = last_chunk_counter + 0x0200
|
||||
offset_word = end_offset - next_boundary (residual byte count)
|
||||
params[0] = key4[0] (= 0x01 on every observed device)
|
||||
params[1] = key4[1] (= 0x11)
|
||||
params[2] = (next_boundary >> 8) & 0xFF
|
||||
params[3] = next_boundary & 0xFF
|
||||
params[4:10] = zeros
|
||||
|
||||
Verification:
|
||||
| end_offset | last_chunk | next_boundary | offset_word | params[2:4] |
|
||||
| 0x1ABE | 0x1800 | 0x1A00 | 0x00BE | 1A 00 |
|
||||
| 0x21F2 | 0x1E00 | 0x2000 | 0x01F2 | 20 00 |
|
||||
| 0x417E | 0x3E38 | 0x4038 | 0x0146 | 40 38 |
|
||||
|
||||
The device receives `requested_address = (params[2] << 8) | offset_word`
|
||||
and replies with `(end_offset - next_boundary)` bytes of waveform tail
|
||||
starting at `next_boundary` — including the 26-byte file footer.
|
||||
|
||||
Args:
|
||||
key4: 4-byte waveform key for this event.
|
||||
end_offset: Event-end pointer (= `(end_key[2] << 8) | end_key[3]`
|
||||
from the STRT record at data[23:27] of A5[0]).
|
||||
last_chunk_counter: Counter of the last full 0x0200-byte chunk fetched
|
||||
(the chunk that covers [last_chunk_counter,
|
||||
last_chunk_counter + 0x0200)).
|
||||
|
||||
Returns:
|
||||
(offset_word, params10) tuple. Pass as
|
||||
`build_5a_frame(offset_word, params)`.
|
||||
|
||||
Raises:
|
||||
ValueError: on inconsistent inputs.
|
||||
"""
|
||||
if len(key4) != 4:
|
||||
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
|
||||
next_boundary = last_chunk_counter + 0x0200
|
||||
if next_boundary > 0xFFFF:
|
||||
raise ValueError(
|
||||
f"next_boundary 0x{next_boundary:04X} exceeds uint16; check inputs"
|
||||
)
|
||||
if end_offset <= last_chunk_counter:
|
||||
raise ValueError(
|
||||
f"end_offset 0x{end_offset:04X} must be > "
|
||||
f"last_chunk_counter 0x{last_chunk_counter:04X}"
|
||||
)
|
||||
offset_word = end_offset - next_boundary
|
||||
if offset_word < 0:
|
||||
# Last chunk overshot end_offset; caller should have stopped one chunk
|
||||
# earlier. Treat as zero residual.
|
||||
offset_word = 0
|
||||
if offset_word > 0xFFFF:
|
||||
raise ValueError(
|
||||
f"offset_word 0x{offset_word:04X} exceeds uint16"
|
||||
)
|
||||
p = bytearray(10)
|
||||
p[0] = key4[0]
|
||||
p[1] = key4[1]
|
||||
p[2] = (next_boundary >> 8) & 0xFF
|
||||
p[3] = next_boundary & 0xFF
|
||||
return offset_word, bytes(p)
|
||||
|
||||
|
||||
# ── End-offset extraction from STRT record ────────────────────────────────────
|
||||
|
||||
STRT_MARKER = b"STRT"
|
||||
|
||||
|
||||
def parse_strt_end_offset(a5_data: bytes) -> Optional[int]:
|
||||
"""
|
||||
Extract the event-end offset from the STRT record in an A5 response payload.
|
||||
|
||||
The first A5 response (the probe response, or the first chunk for events
|
||||
with non-zero start_key[2:4]) contains a STRT record at byte offset 17 of
|
||||
`data`. Layout:
|
||||
|
||||
data[17:21] "STRT"
|
||||
data[21:23] ff fe sentinel
|
||||
data[23:27] end_key ← 4-byte key of where this event ENDS
|
||||
data[27:31] start_key
|
||||
...
|
||||
|
||||
Returns `(end_key[2] << 8) | end_key[3]` — the absolute device-buffer
|
||||
address where the event ends. Use this to bound the chunk loop and to
|
||||
compute the TERM frame.
|
||||
|
||||
Verified end_offset values:
|
||||
| event start_key | end_key | end_offset |
|
||||
| 01110000 | 01111ABE | 0x1ABE |
|
||||
| 01110000 | 011121F2 | 0x21F2 |
|
||||
| 011121F2 | 0111417E | 0x417E |
|
||||
|
||||
Args:
|
||||
a5_data: The `data` field of an A5 response frame (frame.data).
|
||||
|
||||
Returns:
|
||||
The end_offset (uint16) if STRT is found, else None.
|
||||
"""
|
||||
pos = a5_data.find(STRT_MARKER)
|
||||
if pos < 0 or pos + 10 > len(a5_data):
|
||||
return None
|
||||
# data[pos+4:pos+6] is "ff fe"; data[pos+6:pos+10] is end_key.
|
||||
end_key = a5_data[pos + 6 : pos + 10]
|
||||
if len(end_key) < 4:
|
||||
return None
|
||||
return (end_key[2] << 8) | end_key[3]
|
||||
|
||||
|
||||
# ── Pre-built POLL frames ─────────────────────────────────────────────────────
|
||||
#
|
||||
# POLL (SUB 0x5B) uses the same two-step pattern as all other reads — the
|
||||
@@ -470,7 +621,6 @@ class S3Frame:
|
||||
|
||||
|
||||
# ── Streaming S3 frame parser ─────────────────────────────────────────────────
|
||||
|
||||
class S3FrameParser:
|
||||
"""
|
||||
Incremental byte-stream parser for S3→BW response frames.
|
||||
|
||||
@@ -201,6 +201,58 @@ class Timestamp:
|
||||
second=second,
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def from_short_record(cls, data: bytes) -> "Timestamp":
|
||||
"""
|
||||
Decode an 8-byte timestamp header from a 210-byte waveform record.
|
||||
|
||||
Wire layout (✅ CONFIRMED 2026-05-01 against live SFM run on BE11529 in
|
||||
Continuous mode, day-of-month = 1 May, raw: 01 05 07 ea 00 0d 15 25):
|
||||
byte[0]: day (uint8)
|
||||
byte[1]: month (uint8)
|
||||
bytes[2-3]: year (big-endian uint16)
|
||||
byte[4]: unknown (0x00 in observed sample)
|
||||
byte[5]: hour (uint8)
|
||||
byte[6]: minute (uint8)
|
||||
byte[7]: second (uint8)
|
||||
|
||||
This is a third format observed in the wild — distinct from the 9-byte
|
||||
(single-shot, sub_code=0x10 at [1]) and 10-byte (continuous, 0x10 at
|
||||
[0] AND [2]) layouts. No marker bytes; disambiguated by where the
|
||||
year lands when scanned at byte 2/3/4.
|
||||
|
||||
Args:
|
||||
data: at least 8 bytes; only the first 8 are consumed.
|
||||
|
||||
Returns:
|
||||
Decoded Timestamp.
|
||||
|
||||
Raises:
|
||||
ValueError: if data is fewer than 8 bytes.
|
||||
"""
|
||||
if len(data) < 8:
|
||||
raise ValueError(
|
||||
f"Short record timestamp requires at least 8 bytes, got {len(data)}"
|
||||
)
|
||||
day = data[0]
|
||||
month = data[1]
|
||||
year = struct.unpack_from(">H", data, 2)[0]
|
||||
unknown_byte = data[4]
|
||||
hour = data[5]
|
||||
minute = data[6]
|
||||
second = data[7]
|
||||
return cls(
|
||||
raw=bytes(data[:8]),
|
||||
flag=0,
|
||||
year=year,
|
||||
unknown_byte=unknown_byte,
|
||||
month=month,
|
||||
day=day,
|
||||
hour=hour,
|
||||
minute=minute,
|
||||
second=second,
|
||||
)
|
||||
|
||||
@property
|
||||
def clock_set(self) -> bool:
|
||||
"""False when year == 1995 (factory default / battery-lost state)."""
|
||||
|
||||
+234
-158
@@ -35,6 +35,8 @@ from .framing import (
|
||||
token_params,
|
||||
bulk_waveform_params,
|
||||
bulk_waveform_term_params,
|
||||
bulk_waveform_term_v2,
|
||||
parse_strt_end_offset,
|
||||
POLL_PROBE,
|
||||
POLL_DATA,
|
||||
SESSION_RESET,
|
||||
@@ -122,16 +124,22 @@ DATA_LENGTHS: dict[int, int] = {
|
||||
}
|
||||
|
||||
# SUB 5A (BULK_WAVEFORM_STREAM) protocol constants.
|
||||
# Confirmed from 1-2-26 BW TX capture analysis (2026-04-02).
|
||||
_BULK_CHUNK_OFFSET = 0x1004 # offset field for probe + all regular chunk requests ✅
|
||||
_BULK_TERM_OFFSET = 0x005A # offset field for termination request ✅
|
||||
_BULK_COUNTER_STEP = 0x0400 # chunk counter increment per chunk ✅
|
||||
# Chunk counter formula: key4[2:4] + (chunk_num - 1) * 0x0400
|
||||
# where key4[2:4] is the event's circular-buffer base offset ((key4[2]<<8)|key4[3]).
|
||||
# Earlier captures showed 0x1004 for chunk 1 of key 01110000 — that was a Blastware
|
||||
# artifact. For keys where key4[2:4] != 0x0000 (e.g. key 01111884) the old
|
||||
# "n * 0x0400" formula sends counters from the wrong buffer region and the device
|
||||
# returns data from a different event. Confirmed correct 2026-04-24.
|
||||
#
|
||||
# 2026-05-01 minimal-fix: the chunk-counter walk is now bounded by the event's
|
||||
# `end_offset` extracted from the STRT record at data[23:27] of the probe
|
||||
# response. Without this bound the loop kept asking for chunks past the event
|
||||
# end and the device responded with post-event circular-buffer garbage,
|
||||
# corrupting reconstructed Blastware files for events ≥ 2 sec.
|
||||
#
|
||||
# We keep the OLD 0x0400 chunk step here (BW actually uses 0x0200 — see §7.8.5
|
||||
# of the protocol reference for the corrected understanding) because the
|
||||
# existing blastware_file.py builder relies on the 0x0400-step frame structure
|
||||
# to produce valid files. Switching to BW's 0x0200 step is a separate task
|
||||
# that also requires updating the file builder.
|
||||
# BW-exact protocol values (v0.14.0). Verified against 4-27-26 + 5-1-26 captures.
|
||||
_BULK_CHUNK_OFFSET = 0x1002 # offset_word for probe + all chunk requests
|
||||
_BULK_TERM_OFFSET = 0x005A # offset_word for the legacy terminator (fallback only)
|
||||
_BULK_COUNTER_STEP = 0x0200 # chunk counter increment (matches chunk payload size)
|
||||
|
||||
# Default timeout values (seconds).
|
||||
# MiniMate Plus is a slow device — keep these generous.
|
||||
@@ -526,203 +534,270 @@ class MiniMateProtocol:
|
||||
self,
|
||||
key4: bytes,
|
||||
*,
|
||||
stop_after_metadata: bool = True,
|
||||
max_chunks: int = 32,
|
||||
stop_after_metadata: bool = True, # DEPRECATED — no-op under BW-exact walk
|
||||
max_chunks: int = 256, # safety cap only; loop is bounded by end_offset
|
||||
include_terminator: bool = False,
|
||||
extra_chunks_after_metadata: int = 1,
|
||||
extra_chunks_after_metadata: int = 1, # DEPRECATED — no-op
|
||||
) -> list[S3Frame]:
|
||||
"""
|
||||
Download the SUB 5A (BULK_WAVEFORM_STREAM) A5 frames for one event.
|
||||
Download the SUB 5A (BULK_WAVEFORM_STREAM) A5 frames for one event using
|
||||
Blastware's exact protocol. REWRITTEN 2026-05-02 (v0.14.0).
|
||||
|
||||
The bulk waveform stream carries both raw ADC samples (large) and
|
||||
event-time metadata strings ("Project:", "Client:", "User Name:",
|
||||
"Seis Loc:", "Extended Notes") embedded in one of the middle frames
|
||||
(confirmed: A5[7] of 9 for 1-2-26 capture).
|
||||
Algorithm (matches BW captures across 2-sec / 3-sec / event-2):
|
||||
|
||||
Protocol is request-per-chunk, NOT a continuous stream:
|
||||
1. Probe (offset=_BULK_CHUNK_OFFSET, is_probe=True, counter=0x0000)
|
||||
2. Chunks (offset=_BULK_CHUNK_OFFSET, is_probe=False, counter+=0x0400)
|
||||
3. Loop until metadata found (stop_after_metadata=True) or max_chunks
|
||||
4. Termination (offset=_BULK_TERM_OFFSET, counter=last+_BULK_COUNTER_STEP)
|
||||
Device responds with a final A5 frame (page_key=0x0000).
|
||||
1. Probe
|
||||
- For events at start_key[2:4] = 0x0000 (first event after erase
|
||||
/ wrap): probe at counter=0x0000 with full key in params.
|
||||
- For continuation events (start_key[2:4] != 0): first chunk at
|
||||
counter = start_key[2:4] + 0x0046; acts as both probe and
|
||||
first sample chunk; response carries STRT.
|
||||
|
||||
By default the termination frame (page_key=0x0000) is NOT included in the
|
||||
returned list. Pass include_terminator=True to append it; the blastware_file
|
||||
writer needs the terminator frame's body to reconstruct the waveform file footer.
|
||||
2. Parse end_offset from STRT record at data[23:27] of the probe response.
|
||||
|
||||
Args:
|
||||
key4: 4-byte waveform key from EVENT_HEADER (1E).
|
||||
stop_after_metadata: If True (default), send termination as soon as
|
||||
b"Project:" is found in a frame's data — avoids
|
||||
downloading the full ADC waveform payload (several
|
||||
hundred KB). Set False to download everything.
|
||||
max_chunks: Safety cap on the number of chunk requests sent
|
||||
(default 32; a typical event uses 9 large frames).
|
||||
include_terminator: If True, append the terminator A5 frame
|
||||
(page_key=0x0000) to the returned list. The
|
||||
terminator carries the waveform file footer bytes.
|
||||
Default False preserves existing caller behaviour.
|
||||
3. Read two fixed metadata pages at counter=0x1002 and counter=0x1004
|
||||
— global session metadata (Project / Client / User Name / Seis Loc
|
||||
/ Extended Notes ASCII strings). Event 1 only; continuation
|
||||
events skip these (BW caches them across the session).
|
||||
|
||||
4. Walk sample chunks at 0x0200 increments, starting from 0x0600 for
|
||||
event 1 or `start + 0x0046 + 0x0200` for continuation events.
|
||||
Stop when `next_chunk + 0x0200 > end_offset`.
|
||||
|
||||
5. Send TERM frame with offset_word and params computed by
|
||||
`bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`.
|
||||
The TERM response contains the partial last chunk (residual =
|
||||
end_offset - next_boundary) including the 26-byte 0e 08 file
|
||||
footer.
|
||||
|
||||
Returns:
|
||||
List of S3Frame objects from each A5 response frame. Frame indices
|
||||
match the request sequence: index 0 = probe response, index 1 = first
|
||||
chunk, etc. If include_terminator=True, the last element is the
|
||||
terminator frame (page_key=0x0000).
|
||||
List of S3Frame objects from each A5 response (probe, metadata
|
||||
pages, sample chunks, optional TERM response). Caller passes
|
||||
`include_terminator=True` (e.g. write_blastware_file) to keep the
|
||||
TERM response in the list — it's required to reconstruct the
|
||||
file footer.
|
||||
|
||||
Deprecated kwargs:
|
||||
stop_after_metadata: legacy "Project:"-string-based stop condition.
|
||||
No-op under the BW-exact walk; the loop is
|
||||
deterministically bounded by end_offset from
|
||||
STRT. Accepted for backward compat.
|
||||
extra_chunks_after_metadata: same.
|
||||
|
||||
Raises:
|
||||
ProtocolError: on timeout, bad checksum, or unexpected SUB.
|
||||
|
||||
Confirmed from 1-2-26 BW TX/RX captures (2026-04-02):
|
||||
- probe + 8 regular chunks + 1 termination = 10 TX frames
|
||||
- 9 large A5 responses + 1 terminator A5 = 10 RX frames
|
||||
- page_key=0x0010 on large frames; page_key=0x0000 on terminator ✅
|
||||
- "Project:" metadata at A5[7].data[626] ✅
|
||||
ProtocolError: on timeout / bad checksum / unexpected SUB.
|
||||
"""
|
||||
if len(key4) != 4:
|
||||
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
|
||||
|
||||
rsp_sub = _expected_rsp_sub(SUB_BULK_WAVEFORM) # 0xFF - 0x5A = 0xA5
|
||||
# Quietly accept and warn on deprecated kwargs.
|
||||
if not stop_after_metadata:
|
||||
log.debug("5A: stop_after_metadata=False is no-op under BW-exact walk")
|
||||
if extra_chunks_after_metadata not in (0, 1):
|
||||
log.debug("5A: extra_chunks_after_metadata=%d is no-op under BW-exact walk",
|
||||
extra_chunks_after_metadata)
|
||||
|
||||
rsp_sub = _expected_rsp_sub(SUB_BULK_WAVEFORM) # 0xA5
|
||||
frames_data: list[S3Frame] = []
|
||||
counter = 0
|
||||
|
||||
# BW counter formula (confirmed from 4-3-26 capture for key 0111245a,
|
||||
# and empirical live-device test 2026-04-06 for key 01110000):
|
||||
# counter for chunk n = max(key4[2:4], 0x0400) + (n - 1) * 0x0400
|
||||
# key4[2:4] is the event's circular-buffer base offset. The max() guard
|
||||
# ensures chunk 1 never uses counter=0x0000 (which equals the probe address
|
||||
# and causes the device to re-return STRT record data for the first chunk).
|
||||
_key4_offset = (key4[2] << 8) | key4[3]
|
||||
start_offset = (key4[2] << 8) | key4[3]
|
||||
is_event_1 = (start_offset == 0)
|
||||
|
||||
# ── Step 1: probe ────────────────────────────────────────────────────
|
||||
log.debug("5A probe key=%s key4_offset=0x%04X", key4.hex(), _key4_offset)
|
||||
params = bulk_waveform_params(key4, 0, is_probe=True)
|
||||
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
|
||||
self._parser.reset() # reset bytes_fed counter before probe recv
|
||||
# ── Step 1: probe / first chunk ──────────────────────────────────────
|
||||
if is_event_1:
|
||||
probe_counter = 0
|
||||
probe_params = bulk_waveform_params(key4, 0, is_probe=True)
|
||||
log.debug("5A probe (event-1) key=%s counter=0x0000", key4.hex())
|
||||
else:
|
||||
# Continuation events: first 5A request lands at counter = key[2:4]
|
||||
# (i.e. the address of the off=0x46 WAVEHDR record returned by 1F).
|
||||
# The probe response carries STRT at byte 17 with end_offset.
|
||||
#
|
||||
# Confirmed 2026-05-04 from 5-1-26 "copy 2nd address" capture
|
||||
# (BW probes counter=0x2238 with key=01112238, STRT@17 end=0x417E)
|
||||
# and 5-4-26 BW captures (2-sec event probes counter=0x2238).
|
||||
#
|
||||
# The earlier "+0x46" formula in the doc came from calling
|
||||
# start_key the BOUNDARY (off=0x2C) key, but the iteration walk
|
||||
# uses 1F's off=0x46 key as cur_key, which already incorporates
|
||||
# the +0x46 offset relative to the boundary. Adding it again
|
||||
# caused the probe to overshoot, miss STRT, and run uncapped.
|
||||
probe_counter = start_offset
|
||||
probe_params = bulk_waveform_params(key4, probe_counter)
|
||||
log.debug(
|
||||
"5A probe (event-N) key=%s counter=0x%04X",
|
||||
key4.hex(), probe_counter,
|
||||
)
|
||||
|
||||
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, probe_params))
|
||||
self._parser.reset()
|
||||
try:
|
||||
rsp = self._recv_one(expected_sub=rsp_sub, reset_parser=False)
|
||||
except TimeoutError:
|
||||
log.warning(
|
||||
"5A probe TIMED OUT for key=%s — "
|
||||
"%d raw bytes received (no complete A5 frame assembled)",
|
||||
"5A probe TIMED OUT for key=%s — %d raw bytes received",
|
||||
key4.hex(), self._parser.bytes_fed,
|
||||
)
|
||||
raise
|
||||
frames_data.append(rsp)
|
||||
log.debug("5A A5[0] page_key=0x%04X %d bytes", rsp.page_key, len(rsp.data))
|
||||
|
||||
# ── Step 2: chunk loop ───────────────────────────────────────────────
|
||||
# Counter formula: _chunk_base + (chunk_num - 1) * 0x0400
|
||||
# where _chunk_base = max(key4[2:4], 0x0400).
|
||||
#
|
||||
# For events with key4[2:4] != 0 (e.g. key 0111245a, offset 0x245a):
|
||||
# _chunk_base = 0x245a → chunk 1=0x245a, chunk 2=0x285a, ...
|
||||
# Confirmed from 4-3-26 capture.
|
||||
#
|
||||
# For events with key4[2:4] == 0 (e.g. key 01110000):
|
||||
# _chunk_base = max(0, 0x0400) = 0x0400
|
||||
# → chunk 1=0x0400, chunk 2=0x0800, ... (= old chunk_num*0x0400)
|
||||
# CRITICAL: counter=0x0000 (same as the probe) causes the device to
|
||||
# re-return the STRT record data for chunk 1, making frame 1 look like
|
||||
# a second probe response (confirmed from server log: frame 1 len=1097,
|
||||
# contains STRT\xff\xfe, contributes zero body bytes after DLE-strip).
|
||||
# counter=0x0400 for chunk 1 confirmed working (empirical test 2026-04-06).
|
||||
_chunk_base = max(_key4_offset, _BULK_COUNTER_STEP)
|
||||
for chunk_num in range(1, max_chunks + 1):
|
||||
counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP
|
||||
params = bulk_waveform_params(key4, counter)
|
||||
log.debug("5A chunk %d counter=0x%04X", chunk_num, counter)
|
||||
frames_data.append(rsp)
|
||||
log.debug("5A A5[0] (probe) page_key=0x%04X %d bytes",
|
||||
rsp.page_key, len(rsp.data))
|
||||
|
||||
# ── Step 2: parse STRT end_offset from probe response ────────────────
|
||||
end_offset = parse_strt_end_offset(rsp.data)
|
||||
if end_offset is None:
|
||||
log.warning(
|
||||
"5A probe response did not contain a STRT record; "
|
||||
"cannot bound chunk loop — falling back to max_chunks=%d cap",
|
||||
max_chunks,
|
||||
)
|
||||
end_offset = 0xFFFF # impossible value → loop runs to max_chunks
|
||||
else:
|
||||
log.info(
|
||||
"5A STRT start_offset=0x%04X end_offset=0x%04X size=0x%04X",
|
||||
start_offset, end_offset, end_offset - start_offset,
|
||||
)
|
||||
|
||||
# ── Step 3: metadata pages 0x1002 + 0x1004 (event 1 only) ────────────
|
||||
# Confirmed from BW captures: BW reads these two fixed device-buffer
|
||||
# pages immediately after the probe for events at start_key[2:4]=0.
|
||||
# Continuation events skip them (BW caches across the session).
|
||||
# Their content is global compliance-setup metadata: Project, Client,
|
||||
# User Name, Seis Loc, Extended Notes.
|
||||
if is_event_1:
|
||||
for meta_counter in (0x1002, 0x1004):
|
||||
# Metadata page params have an extra trailing 0x00 byte
|
||||
# (12-byte params instead of 11) — empirical from BW captures.
|
||||
# Checksum-neutral but matches BW byte-for-byte.
|
||||
meta_params = bytes([
|
||||
0x00,
|
||||
key4[0], key4[1],
|
||||
(meta_counter >> 8) & 0xFF,
|
||||
meta_counter & 0xFF,
|
||||
0, 0, 0, 0, 0, 0, 0,
|
||||
])
|
||||
log.debug("5A metadata page counter=0x%04X", meta_counter)
|
||||
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, meta_params))
|
||||
self._parser.reset()
|
||||
try:
|
||||
meta_rsp = self._recv_one(
|
||||
expected_sub=rsp_sub, reset_parser=False, timeout=10.0,
|
||||
)
|
||||
except TimeoutError:
|
||||
log.warning(
|
||||
"5A metadata page 0x%04X TIMED OUT — continuing",
|
||||
meta_counter,
|
||||
)
|
||||
continue
|
||||
frames_data.append(meta_rsp)
|
||||
log.debug(
|
||||
"5A meta@0x%04X page_key=0x%04X %d bytes",
|
||||
meta_counter, meta_rsp.page_key, len(meta_rsp.data),
|
||||
)
|
||||
|
||||
# ── Step 4: sample chunk loop, bounded by end_offset ─────────────────
|
||||
# Sample chunks start at:
|
||||
# event 1: counter = 0x0600
|
||||
# event N (>0): counter = probe_counter + 0x0200
|
||||
# (probe was the first sample chunk)
|
||||
if is_event_1:
|
||||
counter = 0x0600
|
||||
else:
|
||||
counter = probe_counter + _BULK_COUNTER_STEP
|
||||
|
||||
last_chunk_counter: Optional[int] = (
|
||||
probe_counter if not is_event_1 else None
|
||||
)
|
||||
chunks_fetched = 0
|
||||
|
||||
while chunks_fetched < max_chunks:
|
||||
# Stop when next chunk would straddle the event end.
|
||||
if counter + _BULK_COUNTER_STEP > end_offset:
|
||||
log.debug(
|
||||
"5A chunk loop done at counter=0x%04X (end=0x%04X); "
|
||||
"%d chunks fetched",
|
||||
counter, end_offset, chunks_fetched,
|
||||
)
|
||||
break
|
||||
|
||||
params = bulk_waveform_params(key4, counter)
|
||||
log.debug("5A chunk #%d counter=0x%04X", chunks_fetched + 1, counter)
|
||||
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
|
||||
self._parser.reset() # reset bytes_fed for accurate per-chunk count
|
||||
self._parser.reset()
|
||||
try:
|
||||
rsp = self._recv_one(expected_sub=rsp_sub, reset_parser=False, timeout=10.0)
|
||||
rsp = self._recv_one(
|
||||
expected_sub=rsp_sub, reset_parser=False, timeout=10.0,
|
||||
)
|
||||
except TimeoutError:
|
||||
raw = self._parser.bytes_fed
|
||||
log.warning(
|
||||
"5A TIMEOUT chunk=%d counter=0x%04X raw_bytes=%d",
|
||||
chunk_num, counter, raw,
|
||||
chunks_fetched + 1, counter, raw,
|
||||
)
|
||||
if raw > 0 and frames_data:
|
||||
# Device sent a partial byte (likely a bare DLE/ETX end-of-stream
|
||||
# signal) but never completed a full frame. Treat as graceful
|
||||
# stream end and fall through to the termination step.
|
||||
log.warning(
|
||||
"5A end-of-stream detected at chunk=%d (raw_bytes=%d, "
|
||||
"frames_collected=%d) — proceeding to termination",
|
||||
chunk_num, raw, len(frames_data),
|
||||
"5A unexpected end-of-stream — proceeding to TERM",
|
||||
)
|
||||
break
|
||||
raise
|
||||
|
||||
log.warning(
|
||||
"5A RX chunk=%d page_key=0x%04X data_len=%d contains_Project=%s",
|
||||
chunk_num, rsp.page_key, len(rsp.data), b"Project:" in rsp.data,
|
||||
log.debug(
|
||||
"5A RX chunk=%d page_key=0x%04X data_len=%d",
|
||||
chunks_fetched + 1, rsp.page_key, len(rsp.data),
|
||||
)
|
||||
|
||||
if rsp.page_key == 0x0000:
|
||||
# Device unexpectedly terminated mid-stream (no termination needed).
|
||||
log.debug("5A A5[%d] page_key=0x0000 — device terminated early", chunk_num)
|
||||
# Device terminated mid-stream unexpectedly.
|
||||
log.warning(
|
||||
"5A unexpected page_key=0x0000 mid-stream at counter=0x%04X",
|
||||
counter,
|
||||
)
|
||||
if include_terminator:
|
||||
frames_data.append(rsp)
|
||||
return frames_data
|
||||
|
||||
frames_data.append(rsp)
|
||||
|
||||
if stop_after_metadata and b"Project:" in rsp.data:
|
||||
# Download exactly one more chunk after finding metadata — this is
|
||||
# what Blastware does. The extra chunk contains the tail ADC data
|
||||
# and primes the device to return a valid footer in the termination
|
||||
# response. Without it, termination returns an empty ack with no
|
||||
# footer bytes (confirmed 2026-04-23 from HxD comparison).
|
||||
# Download extra_chunks_after_metadata more chunks past the
|
||||
# metadata. The caller calculates this from record_time and
|
||||
# sample_rate so we download exactly the right amount of ADC
|
||||
# data — no more, no less — before terminating.
|
||||
# The device returns the footer in the termination response only
|
||||
# after the right amount of data has been consumed.
|
||||
log.debug("5A A5[%d] metadata found — fetching %d more chunk(s)",
|
||||
chunk_num, extra_chunks_after_metadata)
|
||||
for _extra_n in range(extra_chunks_after_metadata):
|
||||
chunk_num += 1
|
||||
counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP
|
||||
params = bulk_waveform_params(key4, counter)
|
||||
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
|
||||
try:
|
||||
extra = self._recv_one(expected_sub=rsp_sub, timeout=10.0)
|
||||
log.debug("5A A5[%d] extra chunk page_key=0x%04X data_len=%d",
|
||||
chunk_num, extra.page_key, len(extra.data))
|
||||
if extra.page_key == 0x0000:
|
||||
if include_terminator:
|
||||
frames_data.append(extra)
|
||||
return frames_data
|
||||
frames_data.append(extra)
|
||||
except TimeoutError:
|
||||
log.debug("5A extra chunk %d timed out — end of stream", _extra_n + 1)
|
||||
break
|
||||
break
|
||||
last_chunk_counter = counter
|
||||
counter += _BULK_COUNTER_STEP
|
||||
chunks_fetched += 1
|
||||
else:
|
||||
log.warning(
|
||||
"5A reached max_chunks=%d without end-of-stream; sending termination",
|
||||
max_chunks,
|
||||
"5A reached max_chunks=%d at counter=0x%04X (end=0x%04X)",
|
||||
max_chunks, counter, end_offset,
|
||||
)
|
||||
|
||||
# ── Step 3: termination ──────────────────────────────────────────────
|
||||
term_counter = counter + _BULK_COUNTER_STEP
|
||||
term_params = bulk_waveform_term_params(key4, term_counter)
|
||||
log.debug(
|
||||
"5A termination term_counter=0x%04X offset=0x%04X",
|
||||
term_counter, _BULK_TERM_OFFSET,
|
||||
)
|
||||
self._send(build_5a_frame(_BULK_TERM_OFFSET, term_params))
|
||||
try:
|
||||
term_rsp = self._recv_one(expected_sub=rsp_sub)
|
||||
# ── Step 5: TERM with proper end_offset-derived formula ──────────────
|
||||
if last_chunk_counter is None or end_offset == 0xFFFF:
|
||||
# No STRT or no chunks fetched — fall back to legacy TERM.
|
||||
log.warning(
|
||||
"5A using legacy TERM (offset_word=0x005A); "
|
||||
"end_offset unavailable or no chunks fetched",
|
||||
)
|
||||
legacy_counter = (last_chunk_counter or probe_counter) + _BULK_COUNTER_STEP
|
||||
term_offset_word = _BULK_TERM_OFFSET # 0x005A
|
||||
term_params = bulk_waveform_term_params(key4, legacy_counter)
|
||||
else:
|
||||
term_offset_word, term_params = bulk_waveform_term_v2(
|
||||
key4, end_offset, last_chunk_counter,
|
||||
)
|
||||
log.debug(
|
||||
"5A termination response page_key=0x%04X %d bytes",
|
||||
"5A TERM offset_word=0x%04X params[2:4]=%s end=0x%04X "
|
||||
"last_chunk=0x%04X",
|
||||
term_offset_word, term_params[2:4].hex(),
|
||||
end_offset, last_chunk_counter,
|
||||
)
|
||||
|
||||
self._send(build_5a_frame(term_offset_word, term_params))
|
||||
try:
|
||||
term_rsp = self._recv_one(expected_sub=rsp_sub, timeout=10.0)
|
||||
log.info(
|
||||
"5A TERM response page_key=0x%04X %d bytes",
|
||||
term_rsp.page_key, len(term_rsp.data),
|
||||
)
|
||||
if include_terminator:
|
||||
frames_data.append(term_rsp)
|
||||
except TimeoutError:
|
||||
log.debug("5A no termination response — device may have already closed")
|
||||
log.warning("5A no TERM response (timeout)")
|
||||
|
||||
return frames_data
|
||||
|
||||
@@ -862,7 +937,7 @@ class MiniMateProtocol:
|
||||
continue
|
||||
|
||||
chunk = data_rsp.data[11:]
|
||||
log.warning(
|
||||
log.debug(
|
||||
"read_compliance_config: frame %s page=0x%04X data=%d cfg_chunk=%d running_total=%d",
|
||||
step_name, data_rsp.page_key, len(data_rsp.data),
|
||||
len(chunk), len(config) + len(chunk),
|
||||
@@ -882,17 +957,18 @@ class MiniMateProtocol:
|
||||
except TimeoutError:
|
||||
pass
|
||||
|
||||
log.warning(
|
||||
log.info(
|
||||
"read_compliance_config: done — %d cfg bytes total",
|
||||
len(config),
|
||||
)
|
||||
|
||||
# Hex dump first 128 bytes for field mapping
|
||||
for row in range(0, min(len(config), 128), 16):
|
||||
row_bytes = bytes(config[row:row + 16])
|
||||
hex_part = ' '.join(f'{b:02x}' for b in row_bytes)
|
||||
asc_part = ''.join(chr(b) if 32 <= b < 127 else '.' for b in row_bytes)
|
||||
log.warning(" cfg[%04x]: %-48s %s", row, hex_part, asc_part)
|
||||
# Hex dump first 128 bytes — useful only for field-mapping work, not normal operation.
|
||||
if log.isEnabledFor(logging.DEBUG):
|
||||
for row in range(0, min(len(config), 128), 16):
|
||||
row_bytes = bytes(config[row:row + 16])
|
||||
hex_part = ' '.join(f'{b:02x}' for b in row_bytes)
|
||||
asc_part = ''.join(chr(b) if 32 <= b < 127 else '.' for b in row_bytes)
|
||||
log.debug(" cfg[%04x]: %-48s %s", row, hex_part, asc_part)
|
||||
|
||||
return bytes(config)
|
||||
|
||||
|
||||
@@ -454,3 +454,102 @@ class SocketTransport(TcpTransport):
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"SocketTransport(peer={self.host!r})"
|
||||
|
||||
|
||||
# ── Capturing transport (MITM-style raw byte mirror) ──────────────────────────
|
||||
|
||||
class CapturingTransport(BaseTransport):
|
||||
"""
|
||||
Wraps another BaseTransport and mirrors every byte to two raw capture files:
|
||||
|
||||
raw_bw_<...>.bin — bytes WE wrote to the device (BW-side TX)
|
||||
raw_s3_<...>.bin — bytes the device wrote back (S3-side TX)
|
||||
|
||||
The file naming and on-wire byte layout are identical to the captures
|
||||
produced by `bridges/ach_mitm.py`, so the resulting `.bin` files can be
|
||||
loaded directly by the Analyzer (File > Open Capture) and parsed by the
|
||||
same tooling used for genuine Blastware MITM captures.
|
||||
|
||||
All BaseTransport methods are forwarded to the inner transport; the only
|
||||
side-effect is that successful read/write byte streams are appended to the
|
||||
two open binary files.
|
||||
|
||||
Args:
|
||||
inner: An already-built BaseTransport (SerialTransport / TcpTransport).
|
||||
bw_path: File path for the "BW TX" stream (bytes we send). Opened "wb".
|
||||
s3_path: File path for the "S3 TX" stream (bytes the device sends).
|
||||
Opened "wb".
|
||||
|
||||
Example:
|
||||
with CapturingTransport(TcpTransport("1.2.3.4", 9034),
|
||||
"raw_bw.bin", "raw_s3.bin") as t:
|
||||
client = MiniMateClient(transport=t)
|
||||
client.connect()
|
||||
client.get_events()
|
||||
# both .bin files now hold the full bidirectional capture.
|
||||
"""
|
||||
|
||||
def __init__(self, inner: BaseTransport, bw_path: str, s3_path: str) -> None:
|
||||
self._inner = inner
|
||||
self._bw_path = bw_path
|
||||
self._s3_path = s3_path
|
||||
self._bw_fh = None
|
||||
self._s3_fh = None
|
||||
# Forward inner attrs so callers can introspect (e.g. .host, .port).
|
||||
self.host = getattr(inner, "host", None)
|
||||
self.port = getattr(inner, "port", None)
|
||||
|
||||
# ── BaseTransport interface ───────────────────────────────────────────────
|
||||
|
||||
def connect(self) -> None:
|
||||
if self._bw_fh is None:
|
||||
self._bw_fh = open(self._bw_path, "wb", buffering=0)
|
||||
if self._s3_fh is None:
|
||||
self._s3_fh = open(self._s3_path, "wb", buffering=0)
|
||||
self._inner.connect()
|
||||
|
||||
def disconnect(self) -> None:
|
||||
try:
|
||||
self._inner.disconnect()
|
||||
finally:
|
||||
for fh_attr in ("_bw_fh", "_s3_fh"):
|
||||
fh = getattr(self, fh_attr)
|
||||
if fh is not None:
|
||||
try:
|
||||
fh.flush()
|
||||
fh.close()
|
||||
except Exception:
|
||||
pass
|
||||
setattr(self, fh_attr, None)
|
||||
|
||||
@property
|
||||
def is_connected(self) -> bool:
|
||||
return self._inner.is_connected
|
||||
|
||||
def write(self, data: bytes) -> None:
|
||||
self._inner.write(data)
|
||||
if data and self._bw_fh is not None:
|
||||
try:
|
||||
self._bw_fh.write(data)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def read(self, n: int) -> bytes:
|
||||
got = self._inner.read(n)
|
||||
if got and self._s3_fh is not None:
|
||||
try:
|
||||
self._s3_fh.write(got)
|
||||
except Exception:
|
||||
pass
|
||||
return got
|
||||
|
||||
@property
|
||||
def bw_path(self) -> str:
|
||||
return self._bw_path
|
||||
|
||||
@property
|
||||
def s3_path(self) -> str:
|
||||
return self._s3_path
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"CapturingTransport({self._inner!r}, bw={self._bw_path!r}, s3={self._s3_path!r})"
|
||||
|
||||
@@ -53,7 +53,9 @@ SUB_TABLE: dict[int, tuple[str, str, str]] = {
|
||||
0x82: ("TRIGGER_CONFIG_WRITE", "BW→S3", "0x1C bytes; trigger config block; mirrors SUB 1C"),
|
||||
0x83: ("TRIGGER_WRITE_CONFIRM", "BW→S3", "Short frame; commit step after 0x82"),
|
||||
# S3→BW responses
|
||||
0x5A: ("BULK_WAVEFORM_STREAM", "BW→S3", "Bulk waveform chunk request; response is A5 stream"),
|
||||
0xA4: ("POLL_RESPONSE", "S3→BW", "Response to SUB 5B poll"),
|
||||
0xA5: ("BULK_WAVEFORM_RESPONSE", "S3→BW", "Response to SUB 5A; waveform chunks + metadata"),
|
||||
0xFE: ("FULL_CONFIG_RESPONSE", "S3→BW", "Response to SUB 01"),
|
||||
0xF9: ("CHANNEL_CONFIG_RESPONSE", "S3→BW", "Response to SUB 06"),
|
||||
0xF7: ("EVENT_INDEX_RESPONSE", "S3→BW", "Response to SUB 08; contains backlight/power-save"),
|
||||
|
||||
+33
-36
@@ -33,7 +33,7 @@ STX = 0x02
|
||||
ETX = 0x03
|
||||
ACK = 0x41
|
||||
|
||||
__version__ = "0.2.3"
|
||||
__version__ = "0.2.5"
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -184,9 +184,9 @@ def validate_bw_body_auto(body: bytes) -> Optional[Tuple[bytes, bytes, str]]:
|
||||
def parse_s3(blob: bytes, trailer_len: int) -> List[Frame]:
|
||||
frames: List[Frame] = []
|
||||
|
||||
IDLE = 0
|
||||
IN_FRAME = 1
|
||||
AFTER_DLE = 2
|
||||
IDLE = 0
|
||||
IN_FRAME = 1
|
||||
IN_FRAME_DLE = 2 # saw DLE inside frame — waiting for next byte
|
||||
|
||||
state = IDLE
|
||||
body = bytearray()
|
||||
@@ -206,66 +206,63 @@ def parse_s3(blob: bytes, trailer_len: int) -> List[Frame]:
|
||||
state = IN_FRAME
|
||||
i += 2
|
||||
continue
|
||||
# ACK bytes, boot strings, garbage — silently ignored
|
||||
|
||||
elif state == IN_FRAME:
|
||||
if b == DLE:
|
||||
state = AFTER_DLE
|
||||
state = IN_FRAME_DLE
|
||||
i += 1
|
||||
continue
|
||||
body.append(b)
|
||||
|
||||
else: # AFTER_DLE
|
||||
if b == DLE:
|
||||
body.append(DLE)
|
||||
state = IN_FRAME
|
||||
i += 1
|
||||
continue
|
||||
|
||||
if b == ETX:
|
||||
# Bare ETX = real S3 frame terminator (confirmed from S3FrameParser)
|
||||
end_offset = i + 1
|
||||
trailer_start = i + 1
|
||||
trailer_end = trailer_start + trailer_len
|
||||
trailer = blob[trailer_start:trailer_end]
|
||||
|
||||
chk_valid = None
|
||||
chk_type = None
|
||||
chk_hex = None
|
||||
payload = bytes(body)
|
||||
|
||||
if len(body) >= 1:
|
||||
received_chk = body[-1]
|
||||
computed_chk = checksum8_sum(bytes(body[:-1]))
|
||||
if computed_chk == received_chk:
|
||||
chk_valid = True
|
||||
chk_type = "SUM8"
|
||||
chk_hex = f"{received_chk:02x}"
|
||||
payload = bytes(body[:-1])
|
||||
else:
|
||||
chk_valid = False
|
||||
|
||||
# S3 checksums are deliberately not validated here.
|
||||
# Large S3 responses (A5 bulk waveform, E5 compliance) embed
|
||||
# inner DLE+ETX sub-frame terminators whose trailing 0x03 byte
|
||||
# lands where the parser would expect the SUM8 checksum, causing
|
||||
# false failures. The live protocol (protocol.py _validate_frame)
|
||||
# also skips S3 checksum enforcement for the same reason.
|
||||
frames.append(Frame(
|
||||
index=idx,
|
||||
start_offset=start_offset,
|
||||
end_offset=end_offset,
|
||||
payload_raw=bytes(body),
|
||||
payload=payload,
|
||||
payload=bytes(body),
|
||||
trailer=trailer,
|
||||
checksum_valid=chk_valid,
|
||||
checksum_type=chk_type,
|
||||
checksum_hex=chk_hex
|
||||
checksum_valid=None,
|
||||
checksum_type=None,
|
||||
checksum_hex=None
|
||||
))
|
||||
|
||||
idx += 1
|
||||
state = IDLE
|
||||
i = trailer_end
|
||||
continue
|
||||
body.append(b)
|
||||
|
||||
else: # IN_FRAME_DLE
|
||||
if b == DLE:
|
||||
# DLE DLE → literal 0x10 in payload
|
||||
body.append(DLE)
|
||||
state = IN_FRAME
|
||||
i += 1
|
||||
continue
|
||||
if b == ETX:
|
||||
# DLE+ETX inside a frame = inner-frame terminator (A4/E5 sub-frames).
|
||||
# Treat as literal data, NOT the outer frame end.
|
||||
body.append(DLE)
|
||||
body.append(ETX)
|
||||
state = IN_FRAME
|
||||
i += 1
|
||||
continue
|
||||
# Unexpected DLE + byte → treat as literal data
|
||||
body.append(DLE)
|
||||
body.append(b)
|
||||
state = IN_FRAME
|
||||
i += 1
|
||||
continue
|
||||
|
||||
i += 1
|
||||
|
||||
|
||||
+804
-112
File diff suppressed because it is too large
Load Diff
+19
-21
@@ -37,6 +37,7 @@ from __future__ import annotations
|
||||
import datetime
|
||||
import logging
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
@@ -863,8 +864,8 @@ def device_event_blastware_file(
|
||||
|
||||
Supply either *port* (serial) or *host* (TCP/modem).
|
||||
|
||||
The file is written to /tmp and streamed back as a binary download.
|
||||
Blastware can open it directly — filename encodes serial + timestamp.
|
||||
The file is written to the OS temp directory and streamed back as a binary
|
||||
download. Blastware can open it directly — filename encodes serial + timestamp.
|
||||
|
||||
Filename format: <prefix><serial3><stem><AB>0<W|H>
|
||||
- prefix letter = chr(ord('B') + floor(serial_numeric / 1000))
|
||||
@@ -885,26 +886,13 @@ def device_event_blastware_file(
|
||||
def _do():
|
||||
with _build_client(port, baud, host, tcp_port, timeout=120.0) as client:
|
||||
info = client.connect()
|
||||
# Use stop_after_metadata=True (full_waveform=False) with 1 extra
|
||||
# chunk after "Project:". The extra chunk is required to prime the
|
||||
# device over TCP: termination at term_counter=metadata_counter+0x0400
|
||||
# returns only ~90 bytes (no useful footer) over TCP/cellular, but
|
||||
# termination at metadata_counter+0x0800 (one chunk later) returns
|
||||
# the full 737-byte frame containing the footer.
|
||||
#
|
||||
# Confirmed from 4-26-26 BW RS-232 capture: BW terminates at 0x1800
|
||||
# without an extra chunk (works on RS-232 but not TCP).
|
||||
# write_blastware_file() automatically skips the extra chunk's
|
||||
# contribution — only the probe+ADC+metadata+terminator bytes appear
|
||||
# in the output file.
|
||||
#
|
||||
# full_waveform=True (natural end-of-stream) downloads ALL chunks
|
||||
# including post-event silence (35+ chunks for a 9-sec event at
|
||||
# 1024 sps) — this produces 24KB+ files that Blastware rejects.
|
||||
# Under v0.14.0 BW-exact 5A walk, the chunk loop is bounded by
|
||||
# the event end_offset extracted from STRT. No more
|
||||
# stop_after_metadata / extra_chunks gymnastics — these
|
||||
# kwargs are now no-ops.
|
||||
events = client.get_events(
|
||||
full_waveform=False,
|
||||
stop_after_index=index,
|
||||
extra_chunks_after_metadata=1,
|
||||
)
|
||||
matching = [ev for ev in events if ev.index == index]
|
||||
return matching[0] if matching else None, info
|
||||
@@ -940,8 +928,18 @@ def device_event_blastware_file(
|
||||
# Build filename using the same algorithm Blastware uses
|
||||
filename = blastware_filename(ev, serial)
|
||||
|
||||
# Write to /tmp so FastAPI can stream it back
|
||||
out_path = Path("/tmp") / filename
|
||||
# Write to OS temp dir (cross-platform: /tmp on Linux/macOS,
|
||||
# %TEMP% on Windows) so FastAPI can stream it back via FileResponse.
|
||||
out_path = Path(tempfile.gettempdir()) / filename
|
||||
# Delete any stale file at this path before writing. On Windows we have
|
||||
# observed the new (smaller) file getting trailing zero-bytes from the
|
||||
# previous (larger) file when filesystem semantics around open(...,"wb")
|
||||
# don't truncate cleanly (e.g. through a synced folder). Explicit unlink
|
||||
# eliminates that ambiguity.
|
||||
try:
|
||||
out_path.unlink()
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
write_blastware_file(ev, a5_frames, out_path)
|
||||
log.info(
|
||||
"blastware_file: wrote %s (%d A5 frames, serial=%s)",
|
||||
|
||||
@@ -0,0 +1,252 @@
|
||||
"""
|
||||
test_5a_protocol.py — Regression test for the v0.14.x SUB 5A protocol fixes.
|
||||
|
||||
Verifies that SFM's framing helpers reproduce Blastware's exact wire bytes
|
||||
for every 5A request frame in the 5-1-26 "bwcap3sec" capture, AND that the
|
||||
file builder produces a byte-identical file when fed the BW capture's A5
|
||||
responses.
|
||||
|
||||
Together these two tests protect all four v0.14.x fixes:
|
||||
|
||||
v0.14.0 — STRT-bounded chunk walk (probe @ 0, metadata pages @ 0x1002 +
|
||||
0x1004, samples @ 0x0600..0x1E00 step 0x0200, TERM at residual)
|
||||
v0.14.1 — event-N probe counter is `start_offset`, not `start_offset+0x46`
|
||||
(covered by the multi-event captures, not this 3-sec event-1
|
||||
capture — but the helpers are the same code path)
|
||||
v0.14.2 — file body assembly is contiguous concatenation, no de-duplication
|
||||
v0.14.3 — partial DLE stuffing of `0x10` bytes in 5A params (counter=0x1000
|
||||
wire bytes are `10 10 00`, not `10 00`)
|
||||
|
||||
If any of these fixes regresses, this test fails immediately with a clear
|
||||
byte-level diff.
|
||||
|
||||
Run:
|
||||
python -m pytest tests/test_5a_protocol.py -v
|
||||
or:
|
||||
python tests/test_5a_protocol.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
# Allow running from the project root without installation
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from minimateplus.framing import (
|
||||
S3FrameParser,
|
||||
build_5a_frame,
|
||||
bulk_waveform_params,
|
||||
bulk_waveform_term_v2,
|
||||
)
|
||||
|
||||
|
||||
# ── Capture loading ────────────────────────────────────────────────────────────
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
# Reference BW MITM capture: BW saving a 3-sec event 0 (start_key=01110000,
|
||||
# end_offset=0x21F2). 17 5A frames: probe + 2 metadata pages + 13 samples + TERM.
|
||||
BW_TX_PATH = os.path.join(
|
||||
ROOT,
|
||||
"bridges/captures/5-1-26/comcheck/bwcap3sec/"
|
||||
"raw_bw_20260501_165723_copy_3sec_waveform_to_disk.bin",
|
||||
)
|
||||
BW_S3_PATH = os.path.join(
|
||||
ROOT,
|
||||
"bridges/captures/5-1-26/comcheck/bwcap3sec/"
|
||||
"raw_s3_20260501_165723_copy_3sec_waveform_to_disk.bin",
|
||||
)
|
||||
|
||||
# BW's saved Blastware file for the same event (used for file-builder verification).
|
||||
BW_SAVED_FILE = os.path.join(
|
||||
ROOT, "example-events/decode_test/5-1-26/bw/M529LKIQ.G10",
|
||||
)
|
||||
|
||||
|
||||
def _split_bw_frames(data: bytes) -> list[bytes]:
|
||||
"""Split BW TX bytes into individual frames (ACK STX … bare ETX)."""
|
||||
frames: list[bytes] = []
|
||||
i = 0
|
||||
while i < len(data):
|
||||
if data[i] != 0x41 or i + 1 >= len(data) or data[i + 1] != 0x02:
|
||||
i += 1
|
||||
continue
|
||||
j = i + 2
|
||||
while j < len(data):
|
||||
if data[j] == 0x03:
|
||||
break
|
||||
if data[j] == 0x10 and j + 1 < len(data):
|
||||
j += 2
|
||||
continue
|
||||
j += 1
|
||||
if j >= len(data):
|
||||
break
|
||||
frames.append(data[i : j + 1])
|
||||
i = j + 1
|
||||
return frames
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def bw_5a_frames() -> list[bytes]:
|
||||
"""All 5A frames from the BW TX capture, in wire order."""
|
||||
if not os.path.exists(BW_TX_PATH):
|
||||
pytest.skip(f"BW capture not found: {BW_TX_PATH}")
|
||||
raw = open(BW_TX_PATH, "rb").read()
|
||||
frames = [
|
||||
f for f in _split_bw_frames(raw)
|
||||
if len(f) >= 6 and f[5] == 0x5A # body[3] == 0x5A (SUB)
|
||||
]
|
||||
assert len(frames) == 17, f"expected 17 5A frames in capture, got {len(frames)}"
|
||||
return frames
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def bw_a5_frames():
|
||||
"""All A5 (response) frames from the matching S3 capture."""
|
||||
if not os.path.exists(BW_S3_PATH):
|
||||
pytest.skip(f"BW S3 capture not found: {BW_S3_PATH}")
|
||||
raw = open(BW_S3_PATH, "rb").read()
|
||||
p = S3FrameParser()
|
||||
p.feed(raw)
|
||||
a5 = [f for f in p.frames if f.sub == 0xA5]
|
||||
assert len(a5) == 17, f"expected 17 A5 frames in capture, got {len(a5)}"
|
||||
return a5
|
||||
|
||||
|
||||
# ── 5A request frame byte-perfect verification ────────────────────────────────
|
||||
|
||||
KEY4 = bytes.fromhex("01110000") # start_key for the 3-sec event 0
|
||||
END_OFFSET = 0x21F2 # parsed from STRT in the BW capture
|
||||
LAST_CHUNK_COUNTER = 0x1E00 # last full 0x0200-byte chunk before TERM
|
||||
|
||||
SAMPLE_COUNTERS = (
|
||||
0x0600, 0x0800, 0x0A00, 0x0C00, 0x0E00,
|
||||
0x1000, 0x1200, 0x1400, 0x1600, 0x1800,
|
||||
0x1A00, 0x1C00, 0x1E00,
|
||||
)
|
||||
|
||||
|
||||
def _meta_params(key: bytes, counter: int) -> bytes:
|
||||
"""Build the 12-byte metadata-page params block (matches BW for 0x1002 / 0x1004)."""
|
||||
return bytes(
|
||||
[
|
||||
0x00, key[0], key[1],
|
||||
(counter >> 8) & 0xFF, counter & 0xFF,
|
||||
0, 0, 0, 0, 0, 0, 0,
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def test_probe_frame_byte_perfect(bw_5a_frames):
|
||||
"""Probe @ counter=0x0000 (frame 0)."""
|
||||
sfm = build_5a_frame(0x1002, bulk_waveform_params(KEY4, 0, is_probe=True))
|
||||
assert sfm == bw_5a_frames[0], (
|
||||
f"\nSFM: {sfm.hex()}\nBW: {bw_5a_frames[0].hex()}"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("idx,counter", [(1, 0x1002), (2, 0x1004)])
|
||||
def test_metadata_page_frames_byte_perfect(bw_5a_frames, idx, counter):
|
||||
"""Metadata pages @ counter=0x1002 and 0x1004 (frames 1 and 2)."""
|
||||
sfm = build_5a_frame(0x1002, _meta_params(KEY4, counter))
|
||||
assert sfm == bw_5a_frames[idx], (
|
||||
f"\nSFM: {sfm.hex()}\nBW: {bw_5a_frames[idx].hex()}"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("i,counter", list(enumerate(SAMPLE_COUNTERS)))
|
||||
def test_sample_chunk_frames_byte_perfect(bw_5a_frames, i, counter):
|
||||
"""
|
||||
Sample chunks @ counter=0x0600..0x1E00, step 0x0200 (frames 3..15).
|
||||
|
||||
Critically, frame 8 (counter=0x1000) requires the v0.14.3 partial DLE
|
||||
stuffing fix — wire params include `10 10 00` for the counter, not `10 00`.
|
||||
"""
|
||||
sfm = build_5a_frame(0x1002, bulk_waveform_params(KEY4, counter))
|
||||
bw_idx = 3 + i
|
||||
assert sfm == bw_5a_frames[bw_idx], (
|
||||
f"\ncounter=0x{counter:04X}"
|
||||
f"\nSFM: {sfm.hex()}"
|
||||
f"\nBW: {bw_5a_frames[bw_idx].hex()}"
|
||||
)
|
||||
|
||||
|
||||
def test_term_frame_byte_perfect(bw_5a_frames):
|
||||
"""TERM frame at residual (frame 16)."""
|
||||
offset_word, params = bulk_waveform_term_v2(KEY4, END_OFFSET, LAST_CHUNK_COUNTER)
|
||||
sfm = build_5a_frame(offset_word, params)
|
||||
assert sfm == bw_5a_frames[16], (
|
||||
f"\nSFM: {sfm.hex()}\nBW: {bw_5a_frames[16].hex()}"
|
||||
)
|
||||
|
||||
|
||||
def test_strt_end_offset_parsing(bw_a5_frames):
|
||||
"""The probe response (A5[0]) carries STRT at byte 17 with end_offset=0x21F2."""
|
||||
from minimateplus.framing import parse_strt_end_offset
|
||||
|
||||
end_offset = parse_strt_end_offset(bw_a5_frames[0].data)
|
||||
assert end_offset == END_OFFSET, (
|
||||
f"expected end_offset=0x{END_OFFSET:04X}, got "
|
||||
f"{f'0x{end_offset:04X}' if end_offset is not None else 'None'}"
|
||||
)
|
||||
|
||||
|
||||
# ── File builder byte-perfect verification ────────────────────────────────────
|
||||
|
||||
def test_blastware_file_builder_byte_perfect(bw_a5_frames):
|
||||
"""
|
||||
Feed the BW capture's A5 frames into write_blastware_file() and verify the
|
||||
output is byte-identical to BW's saved M529LKIQ.G10 reference file.
|
||||
|
||||
This protects the v0.14.2 strip-removal fix and the file-builder skip
|
||||
values (probe=38, meta=13, samples=12, TERM=11).
|
||||
"""
|
||||
if not os.path.exists(BW_SAVED_FILE):
|
||||
pytest.skip(f"BW saved file not found: {BW_SAVED_FILE}")
|
||||
|
||||
import tempfile
|
||||
|
||||
from minimateplus.blastware_file import write_blastware_file
|
||||
from minimateplus.models import Event
|
||||
|
||||
ev = Event(index=0)
|
||||
ev._waveform_key = KEY4
|
||||
ev.rectime_seconds = 3
|
||||
ev.timestamp = None # let the builder pull the footer from the TERM frame
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".G10", delete=False) as tf:
|
||||
tmp_path = tf.name
|
||||
try:
|
||||
write_blastware_file(ev, bw_a5_frames, tmp_path)
|
||||
sfm_bytes = open(tmp_path, "rb").read()
|
||||
finally:
|
||||
os.unlink(tmp_path)
|
||||
|
||||
bw_bytes = open(BW_SAVED_FILE, "rb").read()
|
||||
|
||||
assert len(sfm_bytes) == len(bw_bytes), (
|
||||
f"file size mismatch: SFM={len(sfm_bytes)} BW={len(bw_bytes)}"
|
||||
)
|
||||
|
||||
if sfm_bytes != bw_bytes:
|
||||
# Find first diff for actionable error message
|
||||
for i in range(len(bw_bytes)):
|
||||
if bw_bytes[i] != sfm_bytes[i]:
|
||||
ctx_start = max(0, i - 8)
|
||||
ctx_end = min(len(bw_bytes), i + 16)
|
||||
pytest.fail(
|
||||
f"file diverges at byte 0x{i:04X}\n"
|
||||
f" BW : {bw_bytes[ctx_start:ctx_end].hex()}\n"
|
||||
f" SFM: {sfm_bytes[ctx_start:ctx_end].hex()}\n"
|
||||
f" {' ' * (i - ctx_start)}^^"
|
||||
)
|
||||
|
||||
|
||||
# ── Standalone runner ─────────────────────────────────────────────────────────
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(pytest.main([__file__, "-v"]))
|
||||
Reference in New Issue
Block a user