v0.14.3 - Full waveform DL pipeline tested and working. #15

Merged
serversdown merged 12 commits from protocol-fix into main 2026-05-05 20:49:48 -04:00
2 changed files with 181 additions and 91 deletions
Showing only changes of commit c914a15e12 - Show all commits
+78 -61
View File
@@ -27,7 +27,7 @@ CHANGELOG.md ← version history
--- ---
## Current implementation state (v0.12.3) ## Current implementation state (v0.14.3)
Full read pipeline + write pipeline + erase pipeline + monitor log + call home config working end-to-end over TCP/cellular: Full read pipeline + write pipeline + erase pipeline + monitor log + call home config working end-to-end over TCP/cellular:
@@ -41,14 +41,15 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c
| Event header / first key | 1E | ✅ | | Event header / first key | 1E | ✅ |
| Waveform header | 0A | ✅ | | Waveform header | 0A | ✅ |
| Waveform record (peaks, timestamp, project) | 0C | ✅ | | Waveform record (peaks, timestamp, project) | 0C | ✅ |
| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ over-read bug fixed v0.13.0 (chunk loop bounded by STRT end_offset); minor wire diffs vs BW deferred — see "SUB 5A — chunk counter formula" | | **Bulk waveform stream (event-time metadata + full waveform)** | **5A** | ✅ **byte-perfect against BW captures (v0.14.3, 2026-05-05)** — STRT-bounded chunk walk + correct event-N probe counter + DLE-stuffed `0x10` bytes in params + concatenate-only file body assembly. All 17 5A request frames in the 5-1-26 3-sec capture reproduce byte-for-byte. |
| Event advance / next key | 1F | ✅ | | Event advance / next key | 1F | ✅ |
| **Write commands (push config to device)** | **6883** | ✅ new v0.8.0 | | **Write commands (push config to device)** | **6883** | ✅ new v0.8.0 |
| **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 | | **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 |
| **Monitor log entries (partial 0x2C records)** | **0A browse** | ✅ new v0.10.0 | | **Monitor log entries (partial 0x2C records)** | **0A browse** | ✅ new v0.10.0 |
| **Auto Call Home config (read + write)** | **2C → 7E → 7F** | ✅ **new v0.12.3** | | **Auto Call Home config (read + write)** | **2C → 7E → 7F** | ✅ **new v0.12.3** |
`get_events()` sequence per event: `1E → 0A → 0C → 5A → 1F` `get_events()` sequence per event: `1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)`
(see "Correct iteration pattern" section below for full detail)
`push_config_raw()` write sequence: `68→73 | 71×3→72 | 82→83 | 69→74→72` `push_config_raw()` write sequence: `68→73 | 71×3→72 | 82→83 | 69→74→72`
@@ -298,9 +299,8 @@ Two chunk addresses are GLOBAL device/session metadata, not event-specific:
These are at fixed absolute addresses in the device's flash buffer. They contain the These are at fixed absolute addresses in the device's flash buffer. They contain the
session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII
strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the strings). Under the v0.14.0+ walk these strings are read directly from the metadata
new walk these strings come from the dedicated metadata pages, not from the sample-chunk pages, not from the sample-chunk stream.
stream.
BW reads them ONCE per Blastware session (during event 1's download) and caches them. BW reads them ONCE per Blastware session (during event 1's download) and caches them.
For SFM, that means: For SFM, that means:
@@ -309,9 +309,10 @@ For SFM, that means:
- Their content does not change when iterating events; only when the user opens - Their content does not change when iterating events; only when the user opens
Compliance Setup → Apply on the device or sends a SUB 71 compliance write. Compliance Setup → Apply on the device or sends a SUB 71 compliance write.
The contents have not been byte-for-byte decoded yet — first task on the implementation The full byte-for-byte layout of the metadata pages has not been mapped — `_decode_a5_metadata_into`
side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the locates the ASCII strings via label scans (`Project:`, `Client:`, `User Name:`, `Seis Loc:`,
strings we currently extract from A5[7]. `Extended Notes`) which works correctly across observed captures. Future work could
dump the structural layout if more session-global fields need to be extracted.
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination ### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
@@ -319,16 +320,11 @@ strings we currently extract from A5[7].
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes. confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
Do not swap them. Do not swap them.
### SUB 5A — event-time metadata source (UPDATED 2026-05-01) ### SUB 5A — event-time metadata source (FINALIZED 2026-05-05)
> **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A The metadata strings come from the two fixed metadata pages at counter `0x1002` and
> bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at `0x1004` (see "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above). These pages
> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because are GLOBAL session metadata — read once per Blastware/SFM session, not per event.
> the broken counter formula was scanning the wrong region.
>
> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and
> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A
> sample-chunk stream itself does NOT contain these strings any more under the new walk.
``` ```
"Project:" → project description "Project:" → project description
@@ -338,55 +334,71 @@ Do not swap them.
"Extended Notes"→ notes "Extended Notes"→ notes
``` ```
**IMPORTANT — 5A "Project:" is session-start config, NOT per-event (confirmed 2026-04-05):** **IMPORTANT — these strings are session-start config, NOT per-event:**
The "Project:" string in the A5 frame 7 payload reflects the compliance setup from when Project / Client / User Name / Seis Loc reflect the compliance setup from when the
the *monitoring session first started*, not the individual event's project name. The per- *monitoring session first started*, not the individual event's per-event metadata. The
event project name is correctly stored in the 210-byte 0C waveform record and must be authoritative per-event project name is stored in the 210-byte 0C waveform record.
used as the authoritative source. `_decode_a5_metadata_into` therefore only sets `_decode_a5_metadata_into` therefore only sets `project` from the 5A metadata pages
`project` from 5A when 0C didn't already supply one. when 0C didn't already supply one.
"Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C "Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C
record — 5A remains the sole source for those fields and they are set unconditionally. record — the metadata pages are the sole source for those fields and they are set
unconditionally.
> ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and #### Deprecated knobs (do not re-introduce)
> stops one chunk later) is a workaround for the missing end_offset bound — when the new
> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is
> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM
> frame.
### SUB 5A — end-of-stream — UPDATED 2026-05-01 The `read_bulk_waveform_stream()` function still accepts these legacy kwargs for
backward compatibility, but they are **no-ops** under the v0.14.0+ walk:
> **Previous understanding (now known to be a symptom, not a feature):** "After streaming - `stop_after_metadata=True` — used to scan the chunk stream for `b"Project:"` and stop
> all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was one chunk later as a workaround for the missing end_offset bound. Obsolete: the loop
> not the device's natural end-of-event signal — it was the device's response when SFM had is now deterministically bounded by `end_offset` parsed from the STRT record at
> walked clean off the end of the addressable buffer region after over-reading by ~5×. data[17] of the probe response, with the partial tail fetched by the TERM frame.
> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the - `extra_chunks_after_metadata` — same era, same reason. No-op.
> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`,
> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser.
The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful If you find code or docs referencing "A5 frame 7" as the source of metadata strings,
defence-in-depth fallback for malformed events or unexpected device states, but should not that's an old-walk artifact (the broken `0x0400`-step formula occasionally caught the
be the primary loop-exit condition. 0x1002 metadata page at sample-chunk fi=7). Update to reference the dedicated metadata
pages instead.
**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each. ### SUB 5A — end-of-stream (FINALIZED 2026-05-01)
Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call
in the chunk loop passes `timeout=10.0` explicitly.
**Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):** Under the v0.14.0+ STRT-bounded walk the stream ends cleanly:
A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames.
A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames.
An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames.
Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks ```
17-37 containing post-event circular-buffer garbage that corrupted the file body. … last full chunk at counter < end_offset
TERM request (offset_word = end_offset - next_boundary,
params address (next_boundary))
TERM response (page_key = 0x0000 or 0x0001, data = the residual
end_offset - next_boundary bytes including the file footer)
```
No timeout-based detection, no "1-byte teaser," no `max_chunks` cap. The chunk loop
exits when `counter + 0x0200 > end_offset`; the TERM frame fetches the tail.
**Chunk recv timeout is 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
Using 120 s would cause a ~2-minute stall on any unexpected timeout. The `_recv_one`
call in the chunk loop passes `timeout=10.0` explicitly.
**Typical chunk count under the v0.14.0+ walk (BE11529, 1024 sps over TCP/cellular):**
| Event duration | Sample chunks | Metadata pages | TERM | Total A5 frames |
|---|---|---|---|---|
| 2-sec (event 1) | ~12 | 2 | 1 | ~15 |
| 3-sec (event 1) | 13 | 2 | 1 | 16 |
| 2-sec (continuation) | 15 | 0 | 1 | 16 |
| 3-sec (continuation) | ~14 | 0 | 1 | ~15 |
For comparison, the deprecated `0x0400`-step walk produced ~37 chunks for a 2-sec
event with chunks 17-37 containing post-event circular-buffer garbage. Do not
re-introduce that walk under any circumstances.
### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06) ### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06)
`_decode_a5_waveform()` previously had `elif fi == 9: continue` — a leftover from the `_decode_a5_waveform()` previously had `elif fi == 9: continue` — a leftover from the
9-frame original blast capture where frame 9 was assumed to be a terminator. For current 9-frame original blast capture where frame 9 was assumed to be a terminator. Removed.
35-frame streams, fi==9 is live waveform data (~133 sample-sets were being dropped). TERM detection in the file builder uses `frame.page_key != 0x0010` (sample marker),
Removed. Terminator detection is via `page_key == 0x0000` in `read_bulk_waveform_stream`, not frame index — see `blastware_file.py`.
not frame index.
### SUB 1E / 1F — event iteration null sentinel and token position (FIXED, do not re-introduce) ### SUB 1E / 1F — event iteration null sentinel and token position (FIXED, do not re-introduce)
@@ -1029,7 +1041,7 @@ offsets in the raw 1A/E5 payload. Only fields with `✅` have confirmed offsets
**Notes tab:** **Notes tab:**
- Enable User Notes (bool) - Enable User Notes (bool)
- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from A5 frame 7 via 5A) - Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from 5A metadata pages at counter 0x1002 / 0x1004 — see "SUB 5A — fixed metadata pages" section)
- Enable Extended Notes (bool); Extended Notes text; Extended Notes Title - Enable Extended Notes (bool); Extended Notes text; Extended Notes Title
- Enable Job Number (bool); Job Number (int) - Enable Job Number (bool); Job Number (int)
- Enable Scaled Distance (bool); Distance from Blast (float); Charge Weight (float) — Scaled Distance is derived - Enable Scaled Distance (bool); Distance from Blast (float); Charge Weight (float) — Scaled Distance is derived
@@ -1343,7 +1355,7 @@ body) because writing a dial string may require DLE escaping for embedded contro
- **Database** — SQLite store for events + monitor log entries; dedup by key; queryable - **Database** — SQLite store for events + monitor log entries; dedup by key; queryable
- **Histograms** — decode histogram-mode A5 data (noise floor tracking) - **Histograms** — decode histogram-mode A5 data (noise floor tracking)
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed working for Continuous mode events (2026-04-23):** SFM-generated file opens in Blastware, shows correct PPV/waveform/timestamp. File is ~200 bytes shorter than BW (missing last ADC tail slice) — all measurements correct. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that create spurious STRT markers in the body). Extension mapping: **CONFIRMED FALSE 2026-04-21** extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>` - **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed BYTE-PERFECT against BW reference (v0.14.3, 2026-05-05):** when fed the BW 5-1-26 3-sec capture's A5 frames, the SFM-built file matches BW's saved `M529LKIQ.G10` byte-for-byte (8708 bytes, 0 differences). Live SFM downloads of event 0 (3-sec) and event 1 (3-sec continuation) both open cleanly in Blastware with full Event Reports, frequency analysis, and waveform plots. Body assembly is just contiguous concatenation of frame contributions in stream order (probe → meta@0x1002 → meta@0x1004 → samples → TERM); no stripping, no overlay, no special handling. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that may need different handling — untested under v0.14.x). Extension mapping: extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
**Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units). **Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units).
@@ -1379,16 +1391,21 @@ body) because writing a dial string may require DLE escaping for embedded contro
| Folder / File | Contents | | Folder / File | Contents |
|---|---| |---|---|
| `1-2-26/` | First SUB 5A BW TX capture — established 5A frame format (raw offset_hi, DLE-aware checksum). 10 frames verified. |
| `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102112) | | `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102112) |
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) — 1E/0A/0C/1F sequence confirmed (single event so token=0xFE appeared to work in either branch) |
| `4-2-26/` | Download-mode BW TX capture — POLL×3 requirement confirmed (frames 68-73 between 1F and first 5A) |
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events — all-zero params for 1F, null sentinel layout, 0A context requirement |
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check |
| `4-11-26 (mitm/ach_mitm_20260411_001912/)` | Full ACH call-home MITM — erase protocol (0xA3/0x06/0xA2), monitor log partial records confirmed |
| `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous | | `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous |
| `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec | | `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec |
| `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) | | `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) |
| `4-20-26/call home settings/` | Call home config read/write captures | | `4-20-26/call home settings/` | Call home config read/write captures |
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check | | `4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof of 5× SFM over-read. STRT end_key field located. |
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events (1E/0A/1F iteration confirmed) | | **`5-1-26/comcheck/`** | **Triplet of captures that nailed the v0.14.0 walk:** SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945` / `_171216`). Confirmed: TERM frame formula across 3 events, metadata pages 0x1002/0x1004 are global session metadata, event-1 vs event-N chunk pattern split, WAVEHDR off=0x46 vs 0x2C disambiguates real events from boundaries. |
| `4-2-26/` | Download-mode BW TX capture (5A bulk stream, POLL×3 requirement confirmed) | | **`5-1-26/comcheck/bwcap3sec/`** | **The byte-perfect reference for v0.14.3.** All 17 BW 5A request frames (probe, 2 metadata, 13 samples, TERM) reproduce byte-for-byte from SFM's framing helpers — including the `10 10 00` DLE-stuffed counter for sample @ 0x1000 that was the long-standing failure mode. |
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) | | `5-4-26/` | BW MITM captures of "copy 3sec / 2sec / Download All" + paired SFM session (`seismo_dl_20260504_145701`) showing the +0x46 event-N probe bug producing 110-chunk runaway walk. Cross-references against 5-1-26 confirmed device behavior is identical. |
| `mitm/ach_mitm_20260411_001912/` | Full ACH call-home MITM (erase protocol, 0xA3/0x06/0xA2 confirmed) |
To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern
in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes
+103 -30
View File
@@ -111,6 +111,9 @@
| 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. | | 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. |
| 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. | | 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. |
| 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. | | 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. |
| 2026-05-04 | §7.8.5, §7.8.8 | **CORRECTED — Event-N probe counter is just `start_offset`, NOT `start_offset + 0x0046`.** The `+0x46` formula in the original §7.8.5 was based on calling the off=0x2C boundary key the "start_key", but in the iteration walk `cur_key` passed into `read_bulk_waveform_stream` is always the off=0x46 WAVEHDR record key from 1F (the partial-record skip path in `get_events` re-runs 1F to advance past 0x2C boundary records). Adding +0x46 placed the probe one WAVEHDR past the actual event start; the response no longer contained STRT at byte 17, `parse_strt_end_offset` returned None, and the chunk loop fell back to the `max_chunks=128` cap, walking ~110 chunks of post-event circular-buffer garbage. Confirmed against both the 5-1-26 "copy 2nd address" capture (probe at counter=0x2238 with key=01112238) and the 5-4-26 BW 2-sec event capture. Fixed in protocol.py `read_bulk_waveform_stream` v0.14.1. |
| 2026-05-05 | §7.8.1 (rule #3 added) | **CONFIRMED — Partial DLE stuffing of `0x10` bytes in 5A params region.** The device's de-stuffing rule for the SUB 5A params region is: `10 10``10`, `10 02/03/04` → kept literal (inner-frame markers), `10 X` for any other X → de-stuffs to just `X` (drops the `0x10`). Therefore any `0x10` byte in the logical params followed by a byte NOT in {0x02, 0x03, 0x04, 0x10} MUST be doubled on the wire. This affects counters with `0x10` in the high byte — most importantly counter=`0x1000`, where logical params bytes `... 10 00 ...` were being sent raw and the device de-stuffed `10 00` to just `00`, returning the response for counter=0x0000 (= the file header + STRT). That STRT block then ended up embedded in the assembled file body at file offset `0x1016` and Blastware refused to open the file. This was the root cause of the long-standing ">1-sec event 0 won't open in BW" pattern (1-sec events worked because their `end_offset < 0x1000`, so no chunk request ever needed counter `0x10__`). All 17 5A request frames in the 5-1-26 bwcap3sec capture (probe + 2 meta + 13 samples + TERM) now match BW byte-for-byte after the fix. Fixed in framing.py `build_5a_frame` v0.14.3. |
| 2026-05-05 | §7.8 / Blastware file format | **CONFIRMED — File body assembly is contiguous concatenation, no de-duplication.** The "duplicate header+STRT strip" hack from v0.13.x was actively destroying valid waveform data — sample chunks at counter `0x1000` and beyond often coincidentally contain the byte sequence `00 12 03 00 STRT` in their delta-encoded ADC stream, and the strip was zeroing 25 bytes per match. Removed in v0.14.2. The correct file body is: probe contribution + meta@0x1002 + meta@0x1004 + sample contributions in stream order + TERM contribution. Verified byte-perfect against BW reference `M529LKIQ.G10` (8708 bytes, 0 differences) when fed the same A5 frames as the BW capture. |
--- ---
@@ -261,7 +264,7 @@ Step 4 — Device sends actual data payload:
| `0A` | **WAVEFORM HEADER READ** | Checks record type for a given waveform key. Variable DATA_LENGTH: 0x30=full bin, 0x26=partial bin. Key at params[4..7]. Required before every 1F call to establish device waveform context. | ✅ CONFIRMED 2026-03-31 | | `0A` | **WAVEFORM HEADER READ** | Checks record type for a given waveform key. Variable DATA_LENGTH: 0x30=full bin, 0x26=partial bin. Key at params[4..7]. Required before every 1F call to establish device waveform context. | ✅ CONFIRMED 2026-03-31 |
| `0C` | **FULL WAVEFORM RECORD** | Downloads 210-byte waveform/histogram record. Sub_code at byte[1]: 0x10=Waveform (9-byte timestamp hdr), 0x03=Waveform-continuous (10-byte hdr, 1-byte shift). PPV floats at label+6 (search "Tran"/"Vert"/"Long"/"MicL"). Peak Vector Sum at tran_label12 (NOT fixed offset). Key at params[4..7], DATA_LENGTH=0xD2. | ✅ CONFIRMED 2026-04-03 | | `0C` | **FULL WAVEFORM RECORD** | Downloads 210-byte waveform/histogram record. Sub_code at byte[1]: 0x10=Waveform (9-byte timestamp hdr), 0x03=Waveform-continuous (10-byte hdr, 1-byte shift). PPV floats at label+6 (search "Tran"/"Vert"/"Long"/"MicL"). Peak Vector Sum at tran_label12 (NOT fixed offset). Key at params[4..7], DATA_LENGTH=0xD2. | ✅ CONFIRMED 2026-04-03 |
| `1F` | **EVENT ADVANCE** | Advances to next waveform key. Token byte at params[7] (⚠️ NOT params[6]): 0x00=browse (all-zero params), 0xFE=download (arm 5A state machine). Returns next key at data[11:15]; null sentinel when data[15:19]=0x00000000. Requires preceding 0A to establish context. Browse 1F must ONLY be called after successful 5A — calling it after a failed 5A disrupts device state for the next event's 5A probe. | ✅ CONFIRMED 2026-04-06 | | `1F` | **EVENT ADVANCE** | Advances to next waveform key. Token byte at params[7] (⚠️ NOT params[6]): 0x00=browse (all-zero params), 0xFE=download (arm 5A state machine). Returns next key at data[11:15]; null sentinel when data[15:19]=0x00000000. Requires preceding 0A to establish context. Browse 1F must ONLY be called after successful 5A — calling it after a failed 5A disrupts device state for the next event's 5A probe. | ✅ CONFIRMED 2026-04-06 |
| `5A` | **BULK WAVEFORM STREAM** | Bulk download of raw ADC sample data. Non-standard frame format: offset_hi=0x10 sent raw (not DLE-stuffed), DLE-aware checksum. Requires 1E-arm + 0C + 1F(0xFE) + POLL×3 before first probe. A5[7] contains event-time metadata (Project:/Client:/User Name:/Seis Loc:). 9+ A5 frames for full waveform; stop_after_metadata=True exits after A5[7]. | ✅ CONFIRMED 2026-04-06 | | `5A` | **BULK WAVEFORM STREAM** | Bulk download of raw ADC sample data. Non-standard frame format: offset_hi=0x10 sent raw (not DLE-stuffed), DLE-aware checksum, **partial DLE stuffing of 0x10 in params** (`10 X` where X∉{02,03,04,10} must be doubled to `10 10 X` — see §7.8). Requires 1E-arm + 0C + 1F(0xFE) + POLL×3 before first probe. Walk: probe at counter=`start_offset` (event 1: 0x0000) → metadata pages 0x1002 + 0x1004 (event 1 only) → sample chunks at 0x0600, 0x0800, …, step 0x0200, bounded by `end_offset` parsed from STRT@data[17] of probe response → TERM frame at residual offset_word. Project:/Client:/User Name:/Seis Loc: live in the metadata pages, NOT in the sample-chunk stream. | ✅ CONFIRMED 2026-05-05 (BYTE-PERFECT vs BW capture) |
| `24` | **WAVEFORM PAGE A?** | Paged waveform read, possibly channel group A. | 🔶 INFERRED | | `24` | **WAVEFORM PAGE A?** | Paged waveform read, possibly channel group A. | 🔶 INFERRED |
| `25` | **WAVEFORM PAGE B?** | Paged waveform read, possibly channel group B. | 🔶 INFERRED | | `25` | **WAVEFORM PAGE B?** | Paged waveform read, possibly channel group B. | 🔶 INFERRED |
| `09` | **UNKNOWN READ A** | Read command, response (`F6`) returns 0xCA (202) bytes. Purpose unknown. | 🔶 INFERRED | | `09` | **UNKNOWN READ A** | Read command, response (`F6`) returns 0xCA (202) bytes. Purpose unknown. | 🔶 INFERRED |
@@ -837,6 +840,20 @@ MicL: 39 64 1D AA = 0.0000875 psi
### 7.6 Bulk Waveform Stream (SUB A5) — Raw ADC Sample Records ### 7.6 Bulk Waveform Stream (SUB A5) — Raw ADC Sample Records
> ⛔ **§7.6 below describes the deprecated `0x0400`-step walk and is RETAINED FOR HISTORICAL CONTEXT ONLY.**
> The "A5[7] is metadata", "A5[9] is terminator", and chunk-counter frame-index claims in this section
> are all artifacts of the broken walk that was overrunning past event end by ~5×.
>
> **For the corrected protocol (v0.14.0+), use:**
> - **§7.8.5** — chunk addressing (probe at `start_offset`, samples step 0x0200, bounded by STRT `end_offset`)
> - **§7.8.6** — TERM frame formula
> - **§7.8.7** — fixed metadata pages 0x1002 / 0x1004 (this is where Project / Client / User Name / Seis Loc
> strings actually live — NOT in any sample-chunk frame)
> - **§7.8.8** — multi-event "Download All" sequence
>
> The waveform sample encoding (4-channel interleaved s16 LE, 8 bytes per sample-set) described in §7.6.1
> below is still correct. Only the frame-indexing claims and metadata-source claims are wrong.
**Two distinct formats exist depending on recording mode. Both confirmed from captures.** **Two distinct formats exist depending on recording mode. Both confirmed from captures.**
--- ---
@@ -1119,20 +1136,26 @@ Near-ambient: 0x3C75C28F = 0.015 in/s (histogram event, near-zero ambient)
**Project strings** — ASCII label-value pairs (search for label, read null-terminated value): **Project strings** — ASCII label-value pairs (search for label, read null-terminated value):
``` ```
"Project:" → project description (in 0C record ✅) "Project:" → project description (in 0C record ✅, also mirrored in metadata pages)
"Client:" → client name (in SUB 5A / A5 frame 7 ✅ — NOT in 0C) "Client:" → client name (in SUB 5A metadata pages ✅ — NOT in 0C)
"User Name:" → operator / user (in SUB 5A / A5 frame 7 ✅ — NOT in 0C) "User Name:" → operator / user (in SUB 5A metadata pages ✅ — NOT in 0C)
"Seis Loc:" → sensor location (in SUB 5A / A5 frame 7 ✅ — NOT in 0C) "Seis Loc:" → sensor location (in SUB 5A metadata pages ✅ — NOT in 0C)
"Extended Notes"→ notes field (in SUB 5A / A5 frame 7 ✅) "Extended Notes"→ notes field (in SUB 5A metadata pages ✅)
``` ```
> ✅ **2026-04-02 — CONFIRMED:** `Client:`, `User Name:`, and `Seis Loc:` are sourced from > ✅ **UPDATED 2026-05-05:** `Client:`, `User Name:`, and `Seis Loc:` come from the
> **SUB 5A (bulk waveform stream)**, specifically A5 frame 7 of the multi-frame response. > dedicated **SUB 5A metadata pages at counter `0x1002` and `0x1004`** — see §7.8.7.
> They are NOT present in the 210-byte SUB 0C waveform record. The strings reflect the > They are NOT present in the 210-byte SUB 0C waveform record.
> compliance setup that was active when the event was recorded on the device — making SUB 5A >
> the authoritative source for true event-time metadata. The `get_events()` client method > An earlier draft of this doc claimed they came from "A5 frame 7" of the bulk waveform
> now issues a SUB 5A request after each 0C download (`stop_after_metadata=True`) and > stream — that was an artifact of the deprecated `0x0400`-step walk where the broken
> overwrites `event.project_info` with the decoded fields. > chunk counter formula happened to land sample-chunk fi=7 on top of the 0x1002 metadata
> page. Under the corrected v0.14.0+ walk (§7.8.5), sample chunks at `0x1000` / `0x1200`
> contain ordinary waveform data, and the metadata pages are read separately.
>
> The strings reflect the compliance setup that was active when the *monitoring session*
> first started (not per-event). `get_events()` reads the metadata pages once at the start
> of the SFM session and the decoded values are stamped onto every event in that session.
--- ---
@@ -1166,7 +1189,9 @@ return events
### 7.7.7 Updated Download Loop with SUB 5A Metadata ### 7.7.7 Updated Download Loop with SUB 5A Metadata
> **Added 2026-04-02.** Confirmed working on BE11529 over TCP/cellular. > ⛔ **The loop in this subsection is DEPRECATED — it uses the broken `stop_after_metadata=True`
> hack and the wrong sequence ordering.** See §7.8.5–§7.8.8 for the corrected protocol.
> The pseudocode below is preserved as historical record only.
```python ```python
key4, _ = proto.read_event_first() # SUB 1E key4, _ = proto.read_event_first() # SUB 1E
@@ -1201,13 +1226,25 @@ return events
### 7.8 SUB 5A — Bulk Waveform Stream (event-time metadata) ### 7.8 SUB 5A — Bulk Waveform Stream (event-time metadata)
> ✅ **Added 2026-04-02.** Frame format confirmed by reproducing Blastware wire bytes > ✅ **§7.8.1 (frame format) — added 2026-04-02; v0.14.3 partial DLE stuffing finalized 2026-05-05.**
> byte-for-byte from the 1-2-26 BW capture. > Frame format confirmed by reproducing Blastware wire bytes byte-for-byte across the 1-2-26
> capture (10 frames) and the 5-1-26 bwcap3sec capture (17 frames, all match including the
> DLE-stuffed `10 10 00` for counter=0x1000).
SUB 5A initiates a bulk transfer of the raw sample data for a stored event. The response is a SUB 5A initiates a bulk transfer of the raw sample data for a stored event. The response is
sequence of A5 frames. Frame 7 (0-indexed) contains the full compliance setup as it existed a sequence of A5 frames. Project-info ASCII strings (`Project:`, `Client:`, `User Name:`,
when the event was recorded — including `Client:`, `User Name:`, `Seis Loc:`, and `Seis Loc:`, `Extended Notes`) live in the dedicated metadata pages at counter `0x1002`
`Extended Notes` ASCII label-value pairs. and `0x1004` (see §7.8.7), not in the sample-chunk stream.
**For the corrected protocol read in order:**
- §7.8.1 — frame format (raw `offset_hi`, DLE-aware checksum, partial DLE stuffing of params)
- §7.8.5 — chunk addressing (probe → metadata pages → samples → TERM, all bounded by `end_offset`)
- §7.8.6 — TERM frame formula
- §7.8.7 — fixed metadata pages 0x1002 / 0x1004
- §7.8.8 — multi-event "Download All" sequence
§7.8.2–§7.8.4 are retained as historical record of earlier (incorrect) understandings —
do not implement against them.
#### 7.8.1 Frame Format #### 7.8.1 Frame Format
@@ -1218,7 +1255,7 @@ SUB 5A uses a **non-standard frame layout** that differs from all other BW→S3
41 02 10 10 00 5A 00 ^^raw^^ ^^raw^^ ^^stuffed^^ 41 02 10 10 00 5A 00 ^^raw^^ ^^raw^^ ^^stuffed^^
``` ```
Two critical differences from `build_bw_frame`: Three critical differences from `build_bw_frame`:
1. **`offset_hi` is sent raw, not DLE-stuffed.** When `offset_hi = 0x10`, the wire carries 1. **`offset_hi` is sent raw, not DLE-stuffed.** When `offset_hi = 0x10`, the wire carries
a bare `0x10` — NOT the stuffed `10 10` that `build_bw_frame` would produce. The device a bare `0x10` — NOT the stuffed `10 10` that `build_bw_frame` would produce. The device
@@ -1227,6 +1264,31 @@ Two critical differences from `build_bw_frame`:
2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen, 2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen,
only `XX` is added to the running sum; lone bytes are added normally. only `XX` is added to the running sum; lone bytes are added normally.
3. **Partial DLE stuffing of `0x10` bytes in the params region** (CONFIRMED 2026-05-05).
The device's de-stuffing rule for the params region is:
- `10 10` → de-stuffs to `10`
- `10 02 / 03 / 04` → kept literal (these are inner-frame markers)
- `10 X` for other X → de-stuffs to just `X` (drops the leading `0x10`)
Therefore any `0x10` byte in the *logical* params that is followed by a byte NOT in
`{0x02, 0x03, 0x04, 0x10}` MUST be doubled on the wire (`10 X``10 10 X`) so the
device's de-stuffer reproduces the original `10 X` pair. This applies most commonly
to counters with `0x10` in the high byte (e.g. counter=`0x1000` produces logical
params bytes `... 10 00 ...`, which BW encodes on the wire as `... 10 10 00 ...`).
Without this stuffing the device interprets counter=`0x1000` as `0x0000` and returns
the probe response (= a copy of the file header + STRT record); that STRT block then
ends up embedded in the assembled file body and Blastware refuses to open the file.
`0x10` bytes in `offset_hi` are still written RAW per (1) above — only the params
region has this stuffing requirement. Metadata-page params for counter `0x1002` /
`0x1004` survive without stuffing because `10 02` / `10 04` fall in the "kept literal"
carve-out.
Verified against BW 5-1-26 bwcap3sec frame 20: params logical bytes
`00 01 11 10 00 00 00 00 00 00 00` (counter=0x1000) are encoded on the wire as
`00 01 11 10 10 00 00 00 00 00 00 00` (12 wire bytes for 11 logical bytes).
#### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol) #### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol)
> ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new > ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new
@@ -1267,11 +1329,20 @@ when the broken 0x0400-step walk passed the global metadata pages at 0x1002/0x10
the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004, the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004,
not from the sample-chunk stream — see §7.8.7. not from the sample-chunk stream — see §7.8.7.
#### 7.8.3 A5 Frame Layout #### 7.8.3 A5 Frame Layout — DEPRECATED 2026-05-01
Each A5 response frame contains a chunk of raw bulk data. Frame 7 of the stream carries the > ⛔ **The "Frame 7 carries the compliance text block" claim below is WRONG.** It was
compliance text block with all project-info label-value pairs. The `client` layer searches > an artifact of the deprecated `0x0400`-step walk where the broken counter formula
for ASCII labels with a null-terminated value read: > happened to land sample-chunk fi=7 on top of the 0x1002 metadata page in flash.
> Under the corrected v0.14.0+ walk (§7.8.5), Frame 7 of the sample-chunk sequence is
> just sample-chunk #5 (counter=0x1000), and contains either ordinary waveform data or —
> critically when DLE-stuffing of params is wrong (§7.8.1.3) — a duplicate file header +
> STRT block when the device misinterprets counter=0x1000 as 0x0000. See §7.8.7 for the
> actual source of these strings.
Historical claim (NOT TO BE IMPLEMENTED): each A5 response frame contains a chunk of raw
bulk data; Frame 7 of the stream carries the compliance text block with all project-info
label-value pairs:
``` ```
"Project:" → null-terminated project name "Project:" → null-terminated project name
@@ -1281,7 +1352,9 @@ for ASCII labels with a null-terminated value read:
"Extended Notes" → null-terminated notes "Extended Notes" → null-terminated notes
``` ```
All five fields reflect the **setup at event-record time**, not the current device config. All five fields do reflect the **setup at event-record time**, not the current device
config. But the source is the metadata pages (§7.8.7), not "Frame 7" of the sample
stream.
#### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01 #### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01
@@ -1560,10 +1633,10 @@ Fields visible in the Blastware "Compliance Setup" dialog. ✅ = byte offset co
| Field | Values / Type | Status | | Field | Values / Type | Status |
|---|---|---| |---|---|---|
| Enable User Notes | bool | ❓ | | Enable User Notes | bool | ❓ |
| Project | ASCII string | ✅ (sourced from A5 frame 7 via SUB 5A) | | Project | ASCII string | ✅ (sourced from SUB 5A metadata pages at counter `0x1002` / `0x1004` — see §7.8.7) |
| Client | ASCII string | ✅ (sourced from A5 frame 7) | | Client | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) |
| User Name | ASCII string | ✅ (sourced from A5 frame 7) | | User Name | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) |
| Seis Loc | ASCII string | ✅ (sourced from A5 frame 7) | | Seis Loc | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) |
| Enable Extended Notes | bool | ❓ | | Enable Extended Notes | bool | ❓ |
| Extended Notes | ASCII text | ❓ | | Extended Notes | ASCII text | ❓ |
| Extended Notes Title | ASCII string | ❓ | | Extended Notes Title | ASCII string | ❓ |