diff --git a/CLAUDE.md b/CLAUDE.md index 0675bd7..627c0cb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ CHANGELOG.md ← version history --- -## Current implementation state (v0.12.3) +## Current implementation state (v0.14.3) Full read pipeline + write pipeline + erase pipeline + monitor log + call home config working end-to-end over TCP/cellular: @@ -41,14 +41,15 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c | Event header / first key | 1E | ✅ | | Waveform header | 0A | ✅ | | Waveform record (peaks, timestamp, project) | 0C | ✅ | -| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ over-read bug fixed v0.13.0 (chunk loop bounded by STRT end_offset); minor wire diffs vs BW deferred — see "SUB 5A — chunk counter formula" | +| **Bulk waveform stream (event-time metadata + full waveform)** | **5A** | ✅ **byte-perfect against BW captures (v0.14.3, 2026-05-05)** — STRT-bounded chunk walk + correct event-N probe counter + DLE-stuffed `0x10` bytes in params + concatenate-only file body assembly. All 17 5A request frames in the 5-1-26 3-sec capture reproduce byte-for-byte. | | Event advance / next key | 1F | ✅ | | **Write commands (push config to device)** | **68–83** | ✅ new v0.8.0 | | **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 | | **Monitor log entries (partial 0x2C records)** | **0A browse** | ✅ new v0.10.0 | | **Auto Call Home config (read + write)** | **2C → 7E → 7F** | ✅ **new v0.12.3** | -`get_events()` sequence per event: `1E → 0A → 0C → 5A → 1F` +`get_events()` sequence per event: `1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)` +(see "Correct iteration pattern" section below for full detail) `push_config_raw()` write sequence: `68→73 | 71×3→72 | 82→83 | 69→74→72` @@ -298,9 +299,8 @@ Two chunk addresses are GLOBAL device/session metadata, not event-specific: These are at fixed absolute addresses in the device's flash buffer. They contain the session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII -strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the -new walk these strings come from the dedicated metadata pages, not from the sample-chunk -stream. +strings). Under the v0.14.0+ walk these strings are read directly from the metadata +pages, not from the sample-chunk stream. BW reads them ONCE per Blastware session (during event 1's download) and caches them. For SFM, that means: @@ -309,9 +309,10 @@ For SFM, that means: - Their content does not change when iterating events; only when the user opens Compliance Setup → Apply on the device or sends a SUB 71 compliance write. -The contents have not been byte-for-byte decoded yet — first task on the implementation -side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the -strings we currently extract from A5[7]. +The full byte-for-byte layout of the metadata pages has not been mapped — `_decode_a5_metadata_into` +locates the ASCII strings via label scans (`Project:`, `Client:`, `User Name:`, `Seis Loc:`, +`Extended Notes`) which works correctly across observed captures. Future work could +dump the structural layout if more session-global fields need to be extracted. ### SUB 5A — params are 11 bytes for chunk frames, 10 for termination @@ -319,16 +320,11 @@ strings we currently extract from A5[7]. confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes. Do not swap them. -### SUB 5A — event-time metadata source (UPDATED 2026-05-01) +### SUB 5A — event-time metadata source (FINALIZED 2026-05-05) -> **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A -> bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at -> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because -> the broken counter formula was scanning the wrong region. -> -> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and -> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A -> sample-chunk stream itself does NOT contain these strings any more under the new walk. +The metadata strings come from the two fixed metadata pages at counter `0x1002` and +`0x1004` (see "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above). These pages +are GLOBAL session metadata — read once per Blastware/SFM session, not per event. ``` "Project:" → project description @@ -338,55 +334,71 @@ Do not swap them. "Extended Notes"→ notes ``` -**IMPORTANT — 5A "Project:" is session-start config, NOT per-event (confirmed 2026-04-05):** -The "Project:" string in the A5 frame 7 payload reflects the compliance setup from when -the *monitoring session first started*, not the individual event's project name. The per- -event project name is correctly stored in the 210-byte 0C waveform record and must be -used as the authoritative source. `_decode_a5_metadata_into` therefore only sets -`project` from 5A when 0C didn't already supply one. +**IMPORTANT — these strings are session-start config, NOT per-event:** +Project / Client / User Name / Seis Loc reflect the compliance setup from when the +*monitoring session first started*, not the individual event's per-event metadata. The +authoritative per-event project name is stored in the 210-byte 0C waveform record. +`_decode_a5_metadata_into` therefore only sets `project` from the 5A metadata pages +when 0C didn't already supply one. "Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C -record — 5A remains the sole source for those fields and they are set unconditionally. +record — the metadata pages are the sole source for those fields and they are set +unconditionally. -> ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and -> stops one chunk later) is a workaround for the missing end_offset bound — when the new -> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is -> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM -> frame. +#### Deprecated knobs (do not re-introduce) -### SUB 5A — end-of-stream — UPDATED 2026-05-01 +The `read_bulk_waveform_stream()` function still accepts these legacy kwargs for +backward compatibility, but they are **no-ops** under the v0.14.0+ walk: -> **Previous understanding (now known to be a symptom, not a feature):** "After streaming -> all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was -> not the device's natural end-of-event signal — it was the device's response when SFM had -> walked clean off the end of the addressable buffer region after over-reading by ~5×. -> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the -> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`, -> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser. +- `stop_after_metadata=True` — used to scan the chunk stream for `b"Project:"` and stop + one chunk later as a workaround for the missing end_offset bound. Obsolete: the loop + is now deterministically bounded by `end_offset` parsed from the STRT record at + data[17] of the probe response, with the partial tail fetched by the TERM frame. +- `extra_chunks_after_metadata` — same era, same reason. No-op. -The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful -defence-in-depth fallback for malformed events or unexpected device states, but should not -be the primary loop-exit condition. +If you find code or docs referencing "A5 frame 7" as the source of metadata strings, +that's an old-walk artifact (the broken `0x0400`-step formula occasionally caught the +0x1002 metadata page at sample-chunk fi=7). Update to reference the dedicated metadata +pages instead. -**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each. -Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call -in the chunk loop passes `timeout=10.0` explicitly. +### SUB 5A — end-of-stream (FINALIZED 2026-05-01) -**Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):** -A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames. -A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames. -An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames. +Under the v0.14.0+ STRT-bounded walk the stream ends cleanly: -Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks -17-37 containing post-event circular-buffer garbage that corrupted the file body. +``` +… last full chunk at counter < end_offset +TERM request (offset_word = end_offset - next_boundary, + params address (next_boundary)) +TERM response (page_key = 0x0000 or 0x0001, data = the residual + end_offset - next_boundary bytes including the file footer) +``` + +No timeout-based detection, no "1-byte teaser," no `max_chunks` cap. The chunk loop +exits when `counter + 0x0200 > end_offset`; the TERM frame fetches the tail. + +**Chunk recv timeout is 10 s, not the default 120 s.** Chunks arrive within ~1 s each. +Using 120 s would cause a ~2-minute stall on any unexpected timeout. The `_recv_one` +call in the chunk loop passes `timeout=10.0` explicitly. + +**Typical chunk count under the v0.14.0+ walk (BE11529, 1024 sps over TCP/cellular):** + +| Event duration | Sample chunks | Metadata pages | TERM | Total A5 frames | +|---|---|---|---|---| +| 2-sec (event 1) | ~12 | 2 | 1 | ~15 | +| 3-sec (event 1) | 13 | 2 | 1 | 16 | +| 2-sec (continuation) | 15 | 0 | 1 | 16 | +| 3-sec (continuation) | ~14 | 0 | 1 | ~15 | + +For comparison, the deprecated `0x0400`-step walk produced ~37 chunks for a 2-sec +event with chunks 17-37 containing post-event circular-buffer garbage. Do not +re-introduce that walk under any circumstances. ### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06) `_decode_a5_waveform()` previously had `elif fi == 9: continue` — a leftover from the -9-frame original blast capture where frame 9 was assumed to be a terminator. For current -35-frame streams, fi==9 is live waveform data (~133 sample-sets were being dropped). -Removed. Terminator detection is via `page_key == 0x0000` in `read_bulk_waveform_stream`, -not frame index. +9-frame original blast capture where frame 9 was assumed to be a terminator. Removed. +TERM detection in the file builder uses `frame.page_key != 0x0010` (sample marker), +not frame index — see `blastware_file.py`. ### SUB 1E / 1F — event iteration null sentinel and token position (FIXED, do not re-introduce) @@ -1029,7 +1041,7 @@ offsets in the raw 1A/E5 payload. Only fields with `✅` have confirmed offsets **Notes tab:** - Enable User Notes (bool) -- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from A5 frame 7 via 5A) +- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from 5A metadata pages at counter 0x1002 / 0x1004 — see "SUB 5A — fixed metadata pages" section) - Enable Extended Notes (bool); Extended Notes text; Extended Notes Title - Enable Job Number (bool); Job Number (int) - Enable Scaled Distance (bool); Distance from Blast (float); Charge Weight (float) — Scaled Distance is derived @@ -1343,7 +1355,7 @@ body) because writing a dial string may require DLE escaping for embedded contro - **Database** — SQLite store for events + monitor log entries; dedup by key; queryable - **Histograms** — decode histogram-mode A5 data (noise floor tracking) -- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed working for Continuous mode events (2026-04-23):** SFM-generated file opens in Blastware, shows correct PPV/waveform/timestamp. File is ~200 bytes shorter than BW (missing last ADC tail slice) — all measurements correct. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that create spurious STRT markers in the body). Extension mapping: **CONFIRMED FALSE 2026-04-21** — extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<4-char-base36-stem>` +- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed BYTE-PERFECT against BW reference (v0.14.3, 2026-05-05):** when fed the BW 5-1-26 3-sec capture's A5 frames, the SFM-built file matches BW's saved `M529LKIQ.G10` byte-for-byte (8708 bytes, 0 differences). Live SFM downloads of event 0 (3-sec) and event 1 (3-sec continuation) both open cleanly in Blastware with full Event Reports, frequency analysis, and waveform plots. Body assembly is just contiguous concatenation of frame contributions in stream order (probe → meta@0x1002 → meta@0x1004 → samples → TERM); no stripping, no overlay, no special handling. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that may need different handling — untested under v0.14.x). Extension mapping: extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<4-char-base36-stem>` **Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units). @@ -1379,16 +1391,21 @@ body) because writing a dial string may require DLE escaping for embedded contro | Folder / File | Contents | |---|---| +| `1-2-26/` | First SUB 5A BW TX capture — established 5A frame format (raw offset_hi, DLE-aware checksum). 10 frames verified. | | `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102–112) | +| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) — 1E/0A/0C/1F sequence confirmed (single event so token=0xFE appeared to work in either branch) | +| `4-2-26/` | Download-mode BW TX capture — POLL×3 requirement confirmed (frames 68-73 between 1F and first 5A) | +| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events — all-zero params for 1F, null sentinel layout, 0A context requirement | +| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check | +| `4-11-26 (mitm/ach_mitm_20260411_001912/)` | Full ACH call-home MITM — erase protocol (0xA3/0x06/0xA2), monitor log partial records confirmed | | `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous | | `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec | | `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) | | `4-20-26/call home settings/` | Call home config read/write captures | -| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check | -| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events (1E/0A/1F iteration confirmed) | -| `4-2-26/` | Download-mode BW TX capture (5A bulk stream, POLL×3 requirement confirmed) | -| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) | -| `mitm/ach_mitm_20260411_001912/` | Full ACH call-home MITM (erase protocol, 0xA3/0x06/0xA2 confirmed) | +| `4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof of 5× SFM over-read. STRT end_key field located. | +| **`5-1-26/comcheck/`** | **Triplet of captures that nailed the v0.14.0 walk:** SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945` / `_171216`). Confirmed: TERM frame formula across 3 events, metadata pages 0x1002/0x1004 are global session metadata, event-1 vs event-N chunk pattern split, WAVEHDR off=0x46 vs 0x2C disambiguates real events from boundaries. | +| **`5-1-26/comcheck/bwcap3sec/`** | **The byte-perfect reference for v0.14.3.** All 17 BW 5A request frames (probe, 2 metadata, 13 samples, TERM) reproduce byte-for-byte from SFM's framing helpers — including the `10 10 00` DLE-stuffed counter for sample @ 0x1000 that was the long-standing failure mode. | +| `5-4-26/` | BW MITM captures of "copy 3sec / 2sec / Download All" + paired SFM session (`seismo_dl_20260504_145701`) showing the +0x46 event-N probe bug producing 110-chunk runaway walk. Cross-references against 5-1-26 confirmed device behavior is identical. | To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index af8d1fc..ad2d841 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -111,6 +111,9 @@ | 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. | | 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. | | 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. | +| 2026-05-04 | §7.8.5, §7.8.8 | **CORRECTED — Event-N probe counter is just `start_offset`, NOT `start_offset + 0x0046`.** The `+0x46` formula in the original §7.8.5 was based on calling the off=0x2C boundary key the "start_key", but in the iteration walk `cur_key` passed into `read_bulk_waveform_stream` is always the off=0x46 WAVEHDR record key from 1F (the partial-record skip path in `get_events` re-runs 1F to advance past 0x2C boundary records). Adding +0x46 placed the probe one WAVEHDR past the actual event start; the response no longer contained STRT at byte 17, `parse_strt_end_offset` returned None, and the chunk loop fell back to the `max_chunks=128` cap, walking ~110 chunks of post-event circular-buffer garbage. Confirmed against both the 5-1-26 "copy 2nd address" capture (probe at counter=0x2238 with key=01112238) and the 5-4-26 BW 2-sec event capture. Fixed in protocol.py `read_bulk_waveform_stream` v0.14.1. | +| 2026-05-05 | §7.8.1 (rule #3 added) | **CONFIRMED — Partial DLE stuffing of `0x10` bytes in 5A params region.** The device's de-stuffing rule for the SUB 5A params region is: `10 10` → `10`, `10 02/03/04` → kept literal (inner-frame markers), `10 X` for any other X → de-stuffs to just `X` (drops the `0x10`). Therefore any `0x10` byte in the logical params followed by a byte NOT in {0x02, 0x03, 0x04, 0x10} MUST be doubled on the wire. This affects counters with `0x10` in the high byte — most importantly counter=`0x1000`, where logical params bytes `... 10 00 ...` were being sent raw and the device de-stuffed `10 00` to just `00`, returning the response for counter=0x0000 (= the file header + STRT). That STRT block then ended up embedded in the assembled file body at file offset `0x1016` and Blastware refused to open the file. This was the root cause of the long-standing ">1-sec event 0 won't open in BW" pattern (1-sec events worked because their `end_offset < 0x1000`, so no chunk request ever needed counter `0x10__`). All 17 5A request frames in the 5-1-26 bwcap3sec capture (probe + 2 meta + 13 samples + TERM) now match BW byte-for-byte after the fix. Fixed in framing.py `build_5a_frame` v0.14.3. | +| 2026-05-05 | §7.8 / Blastware file format | **CONFIRMED — File body assembly is contiguous concatenation, no de-duplication.** The "duplicate header+STRT strip" hack from v0.13.x was actively destroying valid waveform data — sample chunks at counter `0x1000` and beyond often coincidentally contain the byte sequence `00 12 03 00 STRT` in their delta-encoded ADC stream, and the strip was zeroing 25 bytes per match. Removed in v0.14.2. The correct file body is: probe contribution + meta@0x1002 + meta@0x1004 + sample contributions in stream order + TERM contribution. Verified byte-perfect against BW reference `M529LKIQ.G10` (8708 bytes, 0 differences) when fed the same A5 frames as the BW capture. | --- @@ -261,7 +264,7 @@ Step 4 — Device sends actual data payload: | `0A` | **WAVEFORM HEADER READ** | Checks record type for a given waveform key. Variable DATA_LENGTH: 0x30=full bin, 0x26=partial bin. Key at params[4..7]. Required before every 1F call to establish device waveform context. | ✅ CONFIRMED 2026-03-31 | | `0C` | **FULL WAVEFORM RECORD** | Downloads 210-byte waveform/histogram record. Sub_code at byte[1]: 0x10=Waveform (9-byte timestamp hdr), 0x03=Waveform-continuous (10-byte hdr, 1-byte shift). PPV floats at label+6 (search "Tran"/"Vert"/"Long"/"MicL"). Peak Vector Sum at tran_label−12 (NOT fixed offset). Key at params[4..7], DATA_LENGTH=0xD2. | ✅ CONFIRMED 2026-04-03 | | `1F` | **EVENT ADVANCE** | Advances to next waveform key. Token byte at params[7] (⚠️ NOT params[6]): 0x00=browse (all-zero params), 0xFE=download (arm 5A state machine). Returns next key at data[11:15]; null sentinel when data[15:19]=0x00000000. Requires preceding 0A to establish context. Browse 1F must ONLY be called after successful 5A — calling it after a failed 5A disrupts device state for the next event's 5A probe. | ✅ CONFIRMED 2026-04-06 | -| `5A` | **BULK WAVEFORM STREAM** | Bulk download of raw ADC sample data. Non-standard frame format: offset_hi=0x10 sent raw (not DLE-stuffed), DLE-aware checksum. Requires 1E-arm + 0C + 1F(0xFE) + POLL×3 before first probe. A5[7] contains event-time metadata (Project:/Client:/User Name:/Seis Loc:). 9+ A5 frames for full waveform; stop_after_metadata=True exits after A5[7]. | ✅ CONFIRMED 2026-04-06 | +| `5A` | **BULK WAVEFORM STREAM** | Bulk download of raw ADC sample data. Non-standard frame format: offset_hi=0x10 sent raw (not DLE-stuffed), DLE-aware checksum, **partial DLE stuffing of 0x10 in params** (`10 X` where X∉{02,03,04,10} must be doubled to `10 10 X` — see §7.8). Requires 1E-arm + 0C + 1F(0xFE) + POLL×3 before first probe. Walk: probe at counter=`start_offset` (event 1: 0x0000) → metadata pages 0x1002 + 0x1004 (event 1 only) → sample chunks at 0x0600, 0x0800, …, step 0x0200, bounded by `end_offset` parsed from STRT@data[17] of probe response → TERM frame at residual offset_word. Project:/Client:/User Name:/Seis Loc: live in the metadata pages, NOT in the sample-chunk stream. | ✅ CONFIRMED 2026-05-05 (BYTE-PERFECT vs BW capture) | | `24` | **WAVEFORM PAGE A?** | Paged waveform read, possibly channel group A. | 🔶 INFERRED | | `25` | **WAVEFORM PAGE B?** | Paged waveform read, possibly channel group B. | 🔶 INFERRED | | `09` | **UNKNOWN READ A** | Read command, response (`F6`) returns 0xCA (202) bytes. Purpose unknown. | 🔶 INFERRED | @@ -837,6 +840,20 @@ MicL: 39 64 1D AA = 0.0000875 psi ### 7.6 Bulk Waveform Stream (SUB A5) — Raw ADC Sample Records +> ⛔ **§7.6 below describes the deprecated `0x0400`-step walk and is RETAINED FOR HISTORICAL CONTEXT ONLY.** +> The "A5[7] is metadata", "A5[9] is terminator", and chunk-counter frame-index claims in this section +> are all artifacts of the broken walk that was overrunning past event end by ~5×. +> +> **For the corrected protocol (v0.14.0+), use:** +> - **§7.8.5** — chunk addressing (probe at `start_offset`, samples step 0x0200, bounded by STRT `end_offset`) +> - **§7.8.6** — TERM frame formula +> - **§7.8.7** — fixed metadata pages 0x1002 / 0x1004 (this is where Project / Client / User Name / Seis Loc +> strings actually live — NOT in any sample-chunk frame) +> - **§7.8.8** — multi-event "Download All" sequence +> +> The waveform sample encoding (4-channel interleaved s16 LE, 8 bytes per sample-set) described in §7.6.1 +> below is still correct. Only the frame-indexing claims and metadata-source claims are wrong. + **Two distinct formats exist depending on recording mode. Both confirmed from captures.** --- @@ -1119,20 +1136,26 @@ Near-ambient: 0x3C75C28F = 0.015 in/s (histogram event, near-zero ambient) **Project strings** — ASCII label-value pairs (search for label, read null-terminated value): ``` -"Project:" → project description (in 0C record ✅) -"Client:" → client name (in SUB 5A / A5 frame 7 ✅ — NOT in 0C) -"User Name:" → operator / user (in SUB 5A / A5 frame 7 ✅ — NOT in 0C) -"Seis Loc:" → sensor location (in SUB 5A / A5 frame 7 ✅ — NOT in 0C) -"Extended Notes"→ notes field (in SUB 5A / A5 frame 7 ✅) +"Project:" → project description (in 0C record ✅, also mirrored in metadata pages) +"Client:" → client name (in SUB 5A metadata pages ✅ — NOT in 0C) +"User Name:" → operator / user (in SUB 5A metadata pages ✅ — NOT in 0C) +"Seis Loc:" → sensor location (in SUB 5A metadata pages ✅ — NOT in 0C) +"Extended Notes"→ notes field (in SUB 5A metadata pages ✅) ``` -> ✅ **2026-04-02 — CONFIRMED:** `Client:`, `User Name:`, and `Seis Loc:` are sourced from -> **SUB 5A (bulk waveform stream)**, specifically A5 frame 7 of the multi-frame response. -> They are NOT present in the 210-byte SUB 0C waveform record. The strings reflect the -> compliance setup that was active when the event was recorded on the device — making SUB 5A -> the authoritative source for true event-time metadata. The `get_events()` client method -> now issues a SUB 5A request after each 0C download (`stop_after_metadata=True`) and -> overwrites `event.project_info` with the decoded fields. +> ✅ **UPDATED 2026-05-05:** `Client:`, `User Name:`, and `Seis Loc:` come from the +> dedicated **SUB 5A metadata pages at counter `0x1002` and `0x1004`** — see §7.8.7. +> They are NOT present in the 210-byte SUB 0C waveform record. +> +> An earlier draft of this doc claimed they came from "A5 frame 7" of the bulk waveform +> stream — that was an artifact of the deprecated `0x0400`-step walk where the broken +> chunk counter formula happened to land sample-chunk fi=7 on top of the 0x1002 metadata +> page. Under the corrected v0.14.0+ walk (§7.8.5), sample chunks at `0x1000` / `0x1200` +> contain ordinary waveform data, and the metadata pages are read separately. +> +> The strings reflect the compliance setup that was active when the *monitoring session* +> first started (not per-event). `get_events()` reads the metadata pages once at the start +> of the SFM session and the decoded values are stamped onto every event in that session. --- @@ -1166,7 +1189,9 @@ return events ### 7.7.7 Updated Download Loop with SUB 5A Metadata -> ✅ **Added 2026-04-02.** Confirmed working on BE11529 over TCP/cellular. +> ⛔ **The loop in this subsection is DEPRECATED — it uses the broken `stop_after_metadata=True` +> hack and the wrong sequence ordering.** See §7.8.5–§7.8.8 for the corrected protocol. +> The pseudocode below is preserved as historical record only. ```python key4, _ = proto.read_event_first() # SUB 1E @@ -1201,13 +1226,25 @@ return events ### 7.8 SUB 5A — Bulk Waveform Stream (event-time metadata) -> ✅ **Added 2026-04-02.** Frame format confirmed by reproducing Blastware wire bytes -> byte-for-byte from the 1-2-26 BW capture. +> ✅ **§7.8.1 (frame format) — added 2026-04-02; v0.14.3 partial DLE stuffing finalized 2026-05-05.** +> Frame format confirmed by reproducing Blastware wire bytes byte-for-byte across the 1-2-26 +> capture (10 frames) and the 5-1-26 bwcap3sec capture (17 frames, all match including the +> DLE-stuffed `10 10 00` for counter=0x1000). -SUB 5A initiates a bulk transfer of the raw sample data for a stored event. The response is a -sequence of A5 frames. Frame 7 (0-indexed) contains the full compliance setup as it existed -when the event was recorded — including `Client:`, `User Name:`, `Seis Loc:`, and -`Extended Notes` ASCII label-value pairs. +SUB 5A initiates a bulk transfer of the raw sample data for a stored event. The response is +a sequence of A5 frames. Project-info ASCII strings (`Project:`, `Client:`, `User Name:`, +`Seis Loc:`, `Extended Notes`) live in the dedicated metadata pages at counter `0x1002` +and `0x1004` (see §7.8.7), not in the sample-chunk stream. + +**For the corrected protocol read in order:** +- §7.8.1 — frame format (raw `offset_hi`, DLE-aware checksum, partial DLE stuffing of params) +- §7.8.5 — chunk addressing (probe → metadata pages → samples → TERM, all bounded by `end_offset`) +- §7.8.6 — TERM frame formula +- §7.8.7 — fixed metadata pages 0x1002 / 0x1004 +- §7.8.8 — multi-event "Download All" sequence + +§7.8.2–§7.8.4 are retained as historical record of earlier (incorrect) understandings — +do not implement against them. #### 7.8.1 Frame Format @@ -1218,7 +1255,7 @@ SUB 5A uses a **non-standard frame layout** that differs from all other BW→S3 41 02 10 10 00 5A 00 ^^raw^^ ^^raw^^ ^^stuffed^^ ``` -Two critical differences from `build_bw_frame`: +Three critical differences from `build_bw_frame`: 1. **`offset_hi` is sent raw, not DLE-stuffed.** When `offset_hi = 0x10`, the wire carries a bare `0x10` — NOT the stuffed `10 10` that `build_bw_frame` would produce. The device @@ -1227,6 +1264,31 @@ Two critical differences from `build_bw_frame`: 2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen, only `XX` is added to the running sum; lone bytes are added normally. +3. **Partial DLE stuffing of `0x10` bytes in the params region** (CONFIRMED 2026-05-05). + The device's de-stuffing rule for the params region is: + + - `10 10` → de-stuffs to `10` + - `10 02 / 03 / 04` → kept literal (these are inner-frame markers) + - `10 X` for other X → de-stuffs to just `X` (drops the leading `0x10`) + + Therefore any `0x10` byte in the *logical* params that is followed by a byte NOT in + `{0x02, 0x03, 0x04, 0x10}` MUST be doubled on the wire (`10 X` → `10 10 X`) so the + device's de-stuffer reproduces the original `10 X` pair. This applies most commonly + to counters with `0x10` in the high byte (e.g. counter=`0x1000` produces logical + params bytes `... 10 00 ...`, which BW encodes on the wire as `... 10 10 00 ...`). + Without this stuffing the device interprets counter=`0x1000` as `0x0000` and returns + the probe response (= a copy of the file header + STRT record); that STRT block then + ends up embedded in the assembled file body and Blastware refuses to open the file. + + `0x10` bytes in `offset_hi` are still written RAW per (1) above — only the params + region has this stuffing requirement. Metadata-page params for counter `0x1002` / + `0x1004` survive without stuffing because `10 02` / `10 04` fall in the "kept literal" + carve-out. + + Verified against BW 5-1-26 bwcap3sec frame 20: params logical bytes + `00 01 11 10 00 00 00 00 00 00 00` (counter=0x1000) are encoded on the wire as + `00 01 11 10 10 00 00 00 00 00 00 00` (12 wire bytes for 11 logical bytes). + #### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol) > ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new @@ -1267,11 +1329,20 @@ when the broken 0x0400-step walk passed the global metadata pages at 0x1002/0x10 the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004, not from the sample-chunk stream — see §7.8.7. -#### 7.8.3 A5 Frame Layout +#### 7.8.3 A5 Frame Layout — DEPRECATED 2026-05-01 -Each A5 response frame contains a chunk of raw bulk data. Frame 7 of the stream carries the -compliance text block with all project-info label-value pairs. The `client` layer searches -for ASCII labels with a null-terminated value read: +> ⛔ **The "Frame 7 carries the compliance text block" claim below is WRONG.** It was +> an artifact of the deprecated `0x0400`-step walk where the broken counter formula +> happened to land sample-chunk fi=7 on top of the 0x1002 metadata page in flash. +> Under the corrected v0.14.0+ walk (§7.8.5), Frame 7 of the sample-chunk sequence is +> just sample-chunk #5 (counter=0x1000), and contains either ordinary waveform data or — +> critically when DLE-stuffing of params is wrong (§7.8.1.3) — a duplicate file header + +> STRT block when the device misinterprets counter=0x1000 as 0x0000. See §7.8.7 for the +> actual source of these strings. + +Historical claim (NOT TO BE IMPLEMENTED): each A5 response frame contains a chunk of raw +bulk data; Frame 7 of the stream carries the compliance text block with all project-info +label-value pairs: ``` "Project:" → null-terminated project name @@ -1281,7 +1352,9 @@ for ASCII labels with a null-terminated value read: "Extended Notes" → null-terminated notes ``` -All five fields reflect the **setup at event-record time**, not the current device config. +All five fields do reflect the **setup at event-record time**, not the current device +config. But the source is the metadata pages (§7.8.7), not "Frame 7" of the sample +stream. #### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01 @@ -1560,10 +1633,10 @@ Fields visible in the Blastware "Compliance Setup" dialog. ✅ = byte offset co | Field | Values / Type | Status | |---|---|---| | Enable User Notes | bool | ❓ | -| Project | ASCII string | ✅ (sourced from A5 frame 7 via SUB 5A) | -| Client | ASCII string | ✅ (sourced from A5 frame 7) | -| User Name | ASCII string | ✅ (sourced from A5 frame 7) | -| Seis Loc | ASCII string | ✅ (sourced from A5 frame 7) | +| Project | ASCII string | ✅ (sourced from SUB 5A metadata pages at counter `0x1002` / `0x1004` — see §7.8.7) | +| Client | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) | +| User Name | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) | +| Seis Loc | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) | | Enable Extended Notes | bool | ❓ | | Extended Notes | ASCII text | ❓ | | Extended Notes Title | ASCII string | ❓ |