v0.14.3 - Full waveform DL pipeline tested and working. #15
@@ -27,7 +27,7 @@ CHANGELOG.md ← version history
|
||||
|
||||
---
|
||||
|
||||
## Current implementation state (v0.12.3)
|
||||
## Current implementation state (v0.14.3)
|
||||
|
||||
Full read pipeline + write pipeline + erase pipeline + monitor log + call home config working end-to-end over TCP/cellular:
|
||||
|
||||
@@ -41,14 +41,15 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c
|
||||
| Event header / first key | 1E | ✅ |
|
||||
| Waveform header | 0A | ✅ |
|
||||
| Waveform record (peaks, timestamp, project) | 0C | ✅ |
|
||||
| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ over-read bug fixed v0.13.0 (chunk loop bounded by STRT end_offset); minor wire diffs vs BW deferred — see "SUB 5A — chunk counter formula" |
|
||||
| **Bulk waveform stream (event-time metadata + full waveform)** | **5A** | ✅ **byte-perfect against BW captures (v0.14.3, 2026-05-05)** — STRT-bounded chunk walk + correct event-N probe counter + DLE-stuffed `0x10` bytes in params + concatenate-only file body assembly. All 17 5A request frames in the 5-1-26 3-sec capture reproduce byte-for-byte. |
|
||||
| Event advance / next key | 1F | ✅ |
|
||||
| **Write commands (push config to device)** | **68–83** | ✅ new v0.8.0 |
|
||||
| **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 |
|
||||
| **Monitor log entries (partial 0x2C records)** | **0A browse** | ✅ new v0.10.0 |
|
||||
| **Auto Call Home config (read + write)** | **2C → 7E → 7F** | ✅ **new v0.12.3** |
|
||||
|
||||
`get_events()` sequence per event: `1E → 0A → 0C → 5A → 1F`
|
||||
`get_events()` sequence per event: `1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)`
|
||||
(see "Correct iteration pattern" section below for full detail)
|
||||
|
||||
`push_config_raw()` write sequence: `68→73 | 71×3→72 | 82→83 | 69→74→72`
|
||||
|
||||
@@ -298,9 +299,8 @@ Two chunk addresses are GLOBAL device/session metadata, not event-specific:
|
||||
|
||||
These are at fixed absolute addresses in the device's flash buffer. They contain the
|
||||
session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII
|
||||
strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the
|
||||
new walk these strings come from the dedicated metadata pages, not from the sample-chunk
|
||||
stream.
|
||||
strings). Under the v0.14.0+ walk these strings are read directly from the metadata
|
||||
pages, not from the sample-chunk stream.
|
||||
|
||||
BW reads them ONCE per Blastware session (during event 1's download) and caches them.
|
||||
For SFM, that means:
|
||||
@@ -309,9 +309,10 @@ For SFM, that means:
|
||||
- Their content does not change when iterating events; only when the user opens
|
||||
Compliance Setup → Apply on the device or sends a SUB 71 compliance write.
|
||||
|
||||
The contents have not been byte-for-byte decoded yet — first task on the implementation
|
||||
side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the
|
||||
strings we currently extract from A5[7].
|
||||
The full byte-for-byte layout of the metadata pages has not been mapped — `_decode_a5_metadata_into`
|
||||
locates the ASCII strings via label scans (`Project:`, `Client:`, `User Name:`, `Seis Loc:`,
|
||||
`Extended Notes`) which works correctly across observed captures. Future work could
|
||||
dump the structural layout if more session-global fields need to be extracted.
|
||||
|
||||
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
|
||||
|
||||
@@ -319,16 +320,11 @@ strings we currently extract from A5[7].
|
||||
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
|
||||
Do not swap them.
|
||||
|
||||
### SUB 5A — event-time metadata source (UPDATED 2026-05-01)
|
||||
### SUB 5A — event-time metadata source (FINALIZED 2026-05-05)
|
||||
|
||||
> **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A
|
||||
> bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at
|
||||
> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because
|
||||
> the broken counter formula was scanning the wrong region.
|
||||
>
|
||||
> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and
|
||||
> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A
|
||||
> sample-chunk stream itself does NOT contain these strings any more under the new walk.
|
||||
The metadata strings come from the two fixed metadata pages at counter `0x1002` and
|
||||
`0x1004` (see "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above). These pages
|
||||
are GLOBAL session metadata — read once per Blastware/SFM session, not per event.
|
||||
|
||||
```
|
||||
"Project:" → project description
|
||||
@@ -338,55 +334,71 @@ Do not swap them.
|
||||
"Extended Notes"→ notes
|
||||
```
|
||||
|
||||
**IMPORTANT — 5A "Project:" is session-start config, NOT per-event (confirmed 2026-04-05):**
|
||||
The "Project:" string in the A5 frame 7 payload reflects the compliance setup from when
|
||||
the *monitoring session first started*, not the individual event's project name. The per-
|
||||
event project name is correctly stored in the 210-byte 0C waveform record and must be
|
||||
used as the authoritative source. `_decode_a5_metadata_into` therefore only sets
|
||||
`project` from 5A when 0C didn't already supply one.
|
||||
**IMPORTANT — these strings are session-start config, NOT per-event:**
|
||||
Project / Client / User Name / Seis Loc reflect the compliance setup from when the
|
||||
*monitoring session first started*, not the individual event's per-event metadata. The
|
||||
authoritative per-event project name is stored in the 210-byte 0C waveform record.
|
||||
`_decode_a5_metadata_into` therefore only sets `project` from the 5A metadata pages
|
||||
when 0C didn't already supply one.
|
||||
|
||||
"Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C
|
||||
record — 5A remains the sole source for those fields and they are set unconditionally.
|
||||
record — the metadata pages are the sole source for those fields and they are set
|
||||
unconditionally.
|
||||
|
||||
> ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and
|
||||
> stops one chunk later) is a workaround for the missing end_offset bound — when the new
|
||||
> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is
|
||||
> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM
|
||||
> frame.
|
||||
#### Deprecated knobs (do not re-introduce)
|
||||
|
||||
### SUB 5A — end-of-stream — UPDATED 2026-05-01
|
||||
The `read_bulk_waveform_stream()` function still accepts these legacy kwargs for
|
||||
backward compatibility, but they are **no-ops** under the v0.14.0+ walk:
|
||||
|
||||
> **Previous understanding (now known to be a symptom, not a feature):** "After streaming
|
||||
> all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was
|
||||
> not the device's natural end-of-event signal — it was the device's response when SFM had
|
||||
> walked clean off the end of the addressable buffer region after over-reading by ~5×.
|
||||
> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the
|
||||
> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`,
|
||||
> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser.
|
||||
- `stop_after_metadata=True` — used to scan the chunk stream for `b"Project:"` and stop
|
||||
one chunk later as a workaround for the missing end_offset bound. Obsolete: the loop
|
||||
is now deterministically bounded by `end_offset` parsed from the STRT record at
|
||||
data[17] of the probe response, with the partial tail fetched by the TERM frame.
|
||||
- `extra_chunks_after_metadata` — same era, same reason. No-op.
|
||||
|
||||
The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful
|
||||
defence-in-depth fallback for malformed events or unexpected device states, but should not
|
||||
be the primary loop-exit condition.
|
||||
If you find code or docs referencing "A5 frame 7" as the source of metadata strings,
|
||||
that's an old-walk artifact (the broken `0x0400`-step formula occasionally caught the
|
||||
0x1002 metadata page at sample-chunk fi=7). Update to reference the dedicated metadata
|
||||
pages instead.
|
||||
|
||||
**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
|
||||
Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call
|
||||
in the chunk loop passes `timeout=10.0` explicitly.
|
||||
### SUB 5A — end-of-stream (FINALIZED 2026-05-01)
|
||||
|
||||
**Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):**
|
||||
A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames.
|
||||
A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames.
|
||||
An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames.
|
||||
Under the v0.14.0+ STRT-bounded walk the stream ends cleanly:
|
||||
|
||||
Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks
|
||||
17-37 containing post-event circular-buffer garbage that corrupted the file body.
|
||||
```
|
||||
… last full chunk at counter < end_offset
|
||||
TERM request (offset_word = end_offset - next_boundary,
|
||||
params address (next_boundary))
|
||||
TERM response (page_key = 0x0000 or 0x0001, data = the residual
|
||||
end_offset - next_boundary bytes including the file footer)
|
||||
```
|
||||
|
||||
No timeout-based detection, no "1-byte teaser," no `max_chunks` cap. The chunk loop
|
||||
exits when `counter + 0x0200 > end_offset`; the TERM frame fetches the tail.
|
||||
|
||||
**Chunk recv timeout is 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
|
||||
Using 120 s would cause a ~2-minute stall on any unexpected timeout. The `_recv_one`
|
||||
call in the chunk loop passes `timeout=10.0` explicitly.
|
||||
|
||||
**Typical chunk count under the v0.14.0+ walk (BE11529, 1024 sps over TCP/cellular):**
|
||||
|
||||
| Event duration | Sample chunks | Metadata pages | TERM | Total A5 frames |
|
||||
|---|---|---|---|---|
|
||||
| 2-sec (event 1) | ~12 | 2 | 1 | ~15 |
|
||||
| 3-sec (event 1) | 13 | 2 | 1 | 16 |
|
||||
| 2-sec (continuation) | 15 | 0 | 1 | 16 |
|
||||
| 3-sec (continuation) | ~14 | 0 | 1 | ~15 |
|
||||
|
||||
For comparison, the deprecated `0x0400`-step walk produced ~37 chunks for a 2-sec
|
||||
event with chunks 17-37 containing post-event circular-buffer garbage. Do not
|
||||
re-introduce that walk under any circumstances.
|
||||
|
||||
### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06)
|
||||
|
||||
`_decode_a5_waveform()` previously had `elif fi == 9: continue` — a leftover from the
|
||||
9-frame original blast capture where frame 9 was assumed to be a terminator. For current
|
||||
35-frame streams, fi==9 is live waveform data (~133 sample-sets were being dropped).
|
||||
Removed. Terminator detection is via `page_key == 0x0000` in `read_bulk_waveform_stream`,
|
||||
not frame index.
|
||||
9-frame original blast capture where frame 9 was assumed to be a terminator. Removed.
|
||||
TERM detection in the file builder uses `frame.page_key != 0x0010` (sample marker),
|
||||
not frame index — see `blastware_file.py`.
|
||||
|
||||
### SUB 1E / 1F — event iteration null sentinel and token position (FIXED, do not re-introduce)
|
||||
|
||||
@@ -1029,7 +1041,7 @@ offsets in the raw 1A/E5 payload. Only fields with `✅` have confirmed offsets
|
||||
|
||||
**Notes tab:**
|
||||
- Enable User Notes (bool)
|
||||
- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from A5 frame 7 via 5A)
|
||||
- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from 5A metadata pages at counter 0x1002 / 0x1004 — see "SUB 5A — fixed metadata pages" section)
|
||||
- Enable Extended Notes (bool); Extended Notes text; Extended Notes Title
|
||||
- Enable Job Number (bool); Job Number (int)
|
||||
- Enable Scaled Distance (bool); Distance from Blast (float); Charge Weight (float) — Scaled Distance is derived
|
||||
@@ -1343,7 +1355,7 @@ body) because writing a dial string may require DLE escaping for embedded contro
|
||||
|
||||
- **Database** — SQLite store for events + monitor log entries; dedup by key; queryable
|
||||
- **Histograms** — decode histogram-mode A5 data (noise floor tracking)
|
||||
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed working for Continuous mode events (2026-04-23):** SFM-generated file opens in Blastware, shows correct PPV/waveform/timestamp. File is ~200 bytes shorter than BW (missing last ADC tail slice) — all measurements correct. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that create spurious STRT markers in the body). Extension mapping: **CONFIRMED FALSE 2026-04-21** — extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
|
||||
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed BYTE-PERFECT against BW reference (v0.14.3, 2026-05-05):** when fed the BW 5-1-26 3-sec capture's A5 frames, the SFM-built file matches BW's saved `M529LKIQ.G10` byte-for-byte (8708 bytes, 0 differences). Live SFM downloads of event 0 (3-sec) and event 1 (3-sec continuation) both open cleanly in Blastware with full Event Reports, frequency analysis, and waveform plots. Body assembly is just contiguous concatenation of frame contributions in stream order (probe → meta@0x1002 → meta@0x1004 → samples → TERM); no stripping, no overlay, no special handling. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that may need different handling — untested under v0.14.x). Extension mapping: extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
|
||||
|
||||
**Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units).
|
||||
|
||||
@@ -1379,16 +1391,21 @@ body) because writing a dial string may require DLE escaping for embedded contro
|
||||
|
||||
| Folder / File | Contents |
|
||||
|---|---|
|
||||
| `1-2-26/` | First SUB 5A BW TX capture — established 5A frame format (raw offset_hi, DLE-aware checksum). 10 frames verified. |
|
||||
| `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102–112) |
|
||||
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) — 1E/0A/0C/1F sequence confirmed (single event so token=0xFE appeared to work in either branch) |
|
||||
| `4-2-26/` | Download-mode BW TX capture — POLL×3 requirement confirmed (frames 68-73 between 1F and first 5A) |
|
||||
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events — all-zero params for 1F, null sentinel layout, 0A context requirement |
|
||||
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check |
|
||||
| `4-11-26 (mitm/ach_mitm_20260411_001912/)` | Full ACH call-home MITM — erase protocol (0xA3/0x06/0xA2), monitor log partial records confirmed |
|
||||
| `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous |
|
||||
| `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec |
|
||||
| `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) |
|
||||
| `4-20-26/call home settings/` | Call home config read/write captures |
|
||||
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check |
|
||||
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events (1E/0A/1F iteration confirmed) |
|
||||
| `4-2-26/` | Download-mode BW TX capture (5A bulk stream, POLL×3 requirement confirmed) |
|
||||
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) |
|
||||
| `mitm/ach_mitm_20260411_001912/` | Full ACH call-home MITM (erase protocol, 0xA3/0x06/0xA2 confirmed) |
|
||||
| `4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof of 5× SFM over-read. STRT end_key field located. |
|
||||
| **`5-1-26/comcheck/`** | **Triplet of captures that nailed the v0.14.0 walk:** SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945` / `_171216`). Confirmed: TERM frame formula across 3 events, metadata pages 0x1002/0x1004 are global session metadata, event-1 vs event-N chunk pattern split, WAVEHDR off=0x46 vs 0x2C disambiguates real events from boundaries. |
|
||||
| **`5-1-26/comcheck/bwcap3sec/`** | **The byte-perfect reference for v0.14.3.** All 17 BW 5A request frames (probe, 2 metadata, 13 samples, TERM) reproduce byte-for-byte from SFM's framing helpers — including the `10 10 00` DLE-stuffed counter for sample @ 0x1000 that was the long-standing failure mode. |
|
||||
| `5-4-26/` | BW MITM captures of "copy 3sec / 2sec / Download All" + paired SFM session (`seismo_dl_20260504_145701`) showing the +0x46 event-N probe bug producing 110-chunk runaway walk. Cross-references against 5-1-26 confirmed device behavior is identical. |
|
||||
|
||||
To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern
|
||||
in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes
|
||||
|
||||
@@ -111,6 +111,9 @@
|
||||
| 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. |
|
||||
| 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. |
|
||||
| 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. |
|
||||
| 2026-05-04 | §7.8.5, §7.8.8 | **CORRECTED — Event-N probe counter is just `start_offset`, NOT `start_offset + 0x0046`.** The `+0x46` formula in the original §7.8.5 was based on calling the off=0x2C boundary key the "start_key", but in the iteration walk `cur_key` passed into `read_bulk_waveform_stream` is always the off=0x46 WAVEHDR record key from 1F (the partial-record skip path in `get_events` re-runs 1F to advance past 0x2C boundary records). Adding +0x46 placed the probe one WAVEHDR past the actual event start; the response no longer contained STRT at byte 17, `parse_strt_end_offset` returned None, and the chunk loop fell back to the `max_chunks=128` cap, walking ~110 chunks of post-event circular-buffer garbage. Confirmed against both the 5-1-26 "copy 2nd address" capture (probe at counter=0x2238 with key=01112238) and the 5-4-26 BW 2-sec event capture. Fixed in protocol.py `read_bulk_waveform_stream` v0.14.1. |
|
||||
| 2026-05-05 | §7.8.1 (rule #3 added) | **CONFIRMED — Partial DLE stuffing of `0x10` bytes in 5A params region.** The device's de-stuffing rule for the SUB 5A params region is: `10 10` → `10`, `10 02/03/04` → kept literal (inner-frame markers), `10 X` for any other X → de-stuffs to just `X` (drops the `0x10`). Therefore any `0x10` byte in the logical params followed by a byte NOT in {0x02, 0x03, 0x04, 0x10} MUST be doubled on the wire. This affects counters with `0x10` in the high byte — most importantly counter=`0x1000`, where logical params bytes `... 10 00 ...` were being sent raw and the device de-stuffed `10 00` to just `00`, returning the response for counter=0x0000 (= the file header + STRT). That STRT block then ended up embedded in the assembled file body at file offset `0x1016` and Blastware refused to open the file. This was the root cause of the long-standing ">1-sec event 0 won't open in BW" pattern (1-sec events worked because their `end_offset < 0x1000`, so no chunk request ever needed counter `0x10__`). All 17 5A request frames in the 5-1-26 bwcap3sec capture (probe + 2 meta + 13 samples + TERM) now match BW byte-for-byte after the fix. Fixed in framing.py `build_5a_frame` v0.14.3. |
|
||||
| 2026-05-05 | §7.8 / Blastware file format | **CONFIRMED — File body assembly is contiguous concatenation, no de-duplication.** The "duplicate header+STRT strip" hack from v0.13.x was actively destroying valid waveform data — sample chunks at counter `0x1000` and beyond often coincidentally contain the byte sequence `00 12 03 00 STRT` in their delta-encoded ADC stream, and the strip was zeroing 25 bytes per match. Removed in v0.14.2. The correct file body is: probe contribution + meta@0x1002 + meta@0x1004 + sample contributions in stream order + TERM contribution. Verified byte-perfect against BW reference `M529LKIQ.G10` (8708 bytes, 0 differences) when fed the same A5 frames as the BW capture. |
|
||||
|
||||
---
|
||||
|
||||
@@ -261,7 +264,7 @@ Step 4 — Device sends actual data payload:
|
||||
| `0A` | **WAVEFORM HEADER READ** | Checks record type for a given waveform key. Variable DATA_LENGTH: 0x30=full bin, 0x26=partial bin. Key at params[4..7]. Required before every 1F call to establish device waveform context. | ✅ CONFIRMED 2026-03-31 |
|
||||
| `0C` | **FULL WAVEFORM RECORD** | Downloads 210-byte waveform/histogram record. Sub_code at byte[1]: 0x10=Waveform (9-byte timestamp hdr), 0x03=Waveform-continuous (10-byte hdr, 1-byte shift). PPV floats at label+6 (search "Tran"/"Vert"/"Long"/"MicL"). Peak Vector Sum at tran_label−12 (NOT fixed offset). Key at params[4..7], DATA_LENGTH=0xD2. | ✅ CONFIRMED 2026-04-03 |
|
||||
| `1F` | **EVENT ADVANCE** | Advances to next waveform key. Token byte at params[7] (⚠️ NOT params[6]): 0x00=browse (all-zero params), 0xFE=download (arm 5A state machine). Returns next key at data[11:15]; null sentinel when data[15:19]=0x00000000. Requires preceding 0A to establish context. Browse 1F must ONLY be called after successful 5A — calling it after a failed 5A disrupts device state for the next event's 5A probe. | ✅ CONFIRMED 2026-04-06 |
|
||||
| `5A` | **BULK WAVEFORM STREAM** | Bulk download of raw ADC sample data. Non-standard frame format: offset_hi=0x10 sent raw (not DLE-stuffed), DLE-aware checksum. Requires 1E-arm + 0C + 1F(0xFE) + POLL×3 before first probe. A5[7] contains event-time metadata (Project:/Client:/User Name:/Seis Loc:). 9+ A5 frames for full waveform; stop_after_metadata=True exits after A5[7]. | ✅ CONFIRMED 2026-04-06 |
|
||||
| `5A` | **BULK WAVEFORM STREAM** | Bulk download of raw ADC sample data. Non-standard frame format: offset_hi=0x10 sent raw (not DLE-stuffed), DLE-aware checksum, **partial DLE stuffing of 0x10 in params** (`10 X` where X∉{02,03,04,10} must be doubled to `10 10 X` — see §7.8). Requires 1E-arm + 0C + 1F(0xFE) + POLL×3 before first probe. Walk: probe at counter=`start_offset` (event 1: 0x0000) → metadata pages 0x1002 + 0x1004 (event 1 only) → sample chunks at 0x0600, 0x0800, …, step 0x0200, bounded by `end_offset` parsed from STRT@data[17] of probe response → TERM frame at residual offset_word. Project:/Client:/User Name:/Seis Loc: live in the metadata pages, NOT in the sample-chunk stream. | ✅ CONFIRMED 2026-05-05 (BYTE-PERFECT vs BW capture) |
|
||||
| `24` | **WAVEFORM PAGE A?** | Paged waveform read, possibly channel group A. | 🔶 INFERRED |
|
||||
| `25` | **WAVEFORM PAGE B?** | Paged waveform read, possibly channel group B. | 🔶 INFERRED |
|
||||
| `09` | **UNKNOWN READ A** | Read command, response (`F6`) returns 0xCA (202) bytes. Purpose unknown. | 🔶 INFERRED |
|
||||
@@ -837,6 +840,20 @@ MicL: 39 64 1D AA = 0.0000875 psi
|
||||
|
||||
### 7.6 Bulk Waveform Stream (SUB A5) — Raw ADC Sample Records
|
||||
|
||||
> ⛔ **§7.6 below describes the deprecated `0x0400`-step walk and is RETAINED FOR HISTORICAL CONTEXT ONLY.**
|
||||
> The "A5[7] is metadata", "A5[9] is terminator", and chunk-counter frame-index claims in this section
|
||||
> are all artifacts of the broken walk that was overrunning past event end by ~5×.
|
||||
>
|
||||
> **For the corrected protocol (v0.14.0+), use:**
|
||||
> - **§7.8.5** — chunk addressing (probe at `start_offset`, samples step 0x0200, bounded by STRT `end_offset`)
|
||||
> - **§7.8.6** — TERM frame formula
|
||||
> - **§7.8.7** — fixed metadata pages 0x1002 / 0x1004 (this is where Project / Client / User Name / Seis Loc
|
||||
> strings actually live — NOT in any sample-chunk frame)
|
||||
> - **§7.8.8** — multi-event "Download All" sequence
|
||||
>
|
||||
> The waveform sample encoding (4-channel interleaved s16 LE, 8 bytes per sample-set) described in §7.6.1
|
||||
> below is still correct. Only the frame-indexing claims and metadata-source claims are wrong.
|
||||
|
||||
**Two distinct formats exist depending on recording mode. Both confirmed from captures.**
|
||||
|
||||
---
|
||||
@@ -1119,20 +1136,26 @@ Near-ambient: 0x3C75C28F = 0.015 in/s (histogram event, near-zero ambient)
|
||||
|
||||
**Project strings** — ASCII label-value pairs (search for label, read null-terminated value):
|
||||
```
|
||||
"Project:" → project description (in 0C record ✅)
|
||||
"Client:" → client name (in SUB 5A / A5 frame 7 ✅ — NOT in 0C)
|
||||
"User Name:" → operator / user (in SUB 5A / A5 frame 7 ✅ — NOT in 0C)
|
||||
"Seis Loc:" → sensor location (in SUB 5A / A5 frame 7 ✅ — NOT in 0C)
|
||||
"Extended Notes"→ notes field (in SUB 5A / A5 frame 7 ✅)
|
||||
"Project:" → project description (in 0C record ✅, also mirrored in metadata pages)
|
||||
"Client:" → client name (in SUB 5A metadata pages ✅ — NOT in 0C)
|
||||
"User Name:" → operator / user (in SUB 5A metadata pages ✅ — NOT in 0C)
|
||||
"Seis Loc:" → sensor location (in SUB 5A metadata pages ✅ — NOT in 0C)
|
||||
"Extended Notes"→ notes field (in SUB 5A metadata pages ✅)
|
||||
```
|
||||
|
||||
> ✅ **2026-04-02 — CONFIRMED:** `Client:`, `User Name:`, and `Seis Loc:` are sourced from
|
||||
> **SUB 5A (bulk waveform stream)**, specifically A5 frame 7 of the multi-frame response.
|
||||
> They are NOT present in the 210-byte SUB 0C waveform record. The strings reflect the
|
||||
> compliance setup that was active when the event was recorded on the device — making SUB 5A
|
||||
> the authoritative source for true event-time metadata. The `get_events()` client method
|
||||
> now issues a SUB 5A request after each 0C download (`stop_after_metadata=True`) and
|
||||
> overwrites `event.project_info` with the decoded fields.
|
||||
> ✅ **UPDATED 2026-05-05:** `Client:`, `User Name:`, and `Seis Loc:` come from the
|
||||
> dedicated **SUB 5A metadata pages at counter `0x1002` and `0x1004`** — see §7.8.7.
|
||||
> They are NOT present in the 210-byte SUB 0C waveform record.
|
||||
>
|
||||
> An earlier draft of this doc claimed they came from "A5 frame 7" of the bulk waveform
|
||||
> stream — that was an artifact of the deprecated `0x0400`-step walk where the broken
|
||||
> chunk counter formula happened to land sample-chunk fi=7 on top of the 0x1002 metadata
|
||||
> page. Under the corrected v0.14.0+ walk (§7.8.5), sample chunks at `0x1000` / `0x1200`
|
||||
> contain ordinary waveform data, and the metadata pages are read separately.
|
||||
>
|
||||
> The strings reflect the compliance setup that was active when the *monitoring session*
|
||||
> first started (not per-event). `get_events()` reads the metadata pages once at the start
|
||||
> of the SFM session and the decoded values are stamped onto every event in that session.
|
||||
|
||||
---
|
||||
|
||||
@@ -1166,7 +1189,9 @@ return events
|
||||
|
||||
### 7.7.7 Updated Download Loop with SUB 5A Metadata
|
||||
|
||||
> ✅ **Added 2026-04-02.** Confirmed working on BE11529 over TCP/cellular.
|
||||
> ⛔ **The loop in this subsection is DEPRECATED — it uses the broken `stop_after_metadata=True`
|
||||
> hack and the wrong sequence ordering.** See §7.8.5–§7.8.8 for the corrected protocol.
|
||||
> The pseudocode below is preserved as historical record only.
|
||||
|
||||
```python
|
||||
key4, _ = proto.read_event_first() # SUB 1E
|
||||
@@ -1201,13 +1226,25 @@ return events
|
||||
|
||||
### 7.8 SUB 5A — Bulk Waveform Stream (event-time metadata)
|
||||
|
||||
> ✅ **Added 2026-04-02.** Frame format confirmed by reproducing Blastware wire bytes
|
||||
> byte-for-byte from the 1-2-26 BW capture.
|
||||
> ✅ **§7.8.1 (frame format) — added 2026-04-02; v0.14.3 partial DLE stuffing finalized 2026-05-05.**
|
||||
> Frame format confirmed by reproducing Blastware wire bytes byte-for-byte across the 1-2-26
|
||||
> capture (10 frames) and the 5-1-26 bwcap3sec capture (17 frames, all match including the
|
||||
> DLE-stuffed `10 10 00` for counter=0x1000).
|
||||
|
||||
SUB 5A initiates a bulk transfer of the raw sample data for a stored event. The response is a
|
||||
sequence of A5 frames. Frame 7 (0-indexed) contains the full compliance setup as it existed
|
||||
when the event was recorded — including `Client:`, `User Name:`, `Seis Loc:`, and
|
||||
`Extended Notes` ASCII label-value pairs.
|
||||
SUB 5A initiates a bulk transfer of the raw sample data for a stored event. The response is
|
||||
a sequence of A5 frames. Project-info ASCII strings (`Project:`, `Client:`, `User Name:`,
|
||||
`Seis Loc:`, `Extended Notes`) live in the dedicated metadata pages at counter `0x1002`
|
||||
and `0x1004` (see §7.8.7), not in the sample-chunk stream.
|
||||
|
||||
**For the corrected protocol read in order:**
|
||||
- §7.8.1 — frame format (raw `offset_hi`, DLE-aware checksum, partial DLE stuffing of params)
|
||||
- §7.8.5 — chunk addressing (probe → metadata pages → samples → TERM, all bounded by `end_offset`)
|
||||
- §7.8.6 — TERM frame formula
|
||||
- §7.8.7 — fixed metadata pages 0x1002 / 0x1004
|
||||
- §7.8.8 — multi-event "Download All" sequence
|
||||
|
||||
§7.8.2–§7.8.4 are retained as historical record of earlier (incorrect) understandings —
|
||||
do not implement against them.
|
||||
|
||||
#### 7.8.1 Frame Format
|
||||
|
||||
@@ -1218,7 +1255,7 @@ SUB 5A uses a **non-standard frame layout** that differs from all other BW→S3
|
||||
41 02 10 10 00 5A 00 ^^raw^^ ^^raw^^ ^^stuffed^^
|
||||
```
|
||||
|
||||
Two critical differences from `build_bw_frame`:
|
||||
Three critical differences from `build_bw_frame`:
|
||||
|
||||
1. **`offset_hi` is sent raw, not DLE-stuffed.** When `offset_hi = 0x10`, the wire carries
|
||||
a bare `0x10` — NOT the stuffed `10 10` that `build_bw_frame` would produce. The device
|
||||
@@ -1227,6 +1264,31 @@ Two critical differences from `build_bw_frame`:
|
||||
2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen,
|
||||
only `XX` is added to the running sum; lone bytes are added normally.
|
||||
|
||||
3. **Partial DLE stuffing of `0x10` bytes in the params region** (CONFIRMED 2026-05-05).
|
||||
The device's de-stuffing rule for the params region is:
|
||||
|
||||
- `10 10` → de-stuffs to `10`
|
||||
- `10 02 / 03 / 04` → kept literal (these are inner-frame markers)
|
||||
- `10 X` for other X → de-stuffs to just `X` (drops the leading `0x10`)
|
||||
|
||||
Therefore any `0x10` byte in the *logical* params that is followed by a byte NOT in
|
||||
`{0x02, 0x03, 0x04, 0x10}` MUST be doubled on the wire (`10 X` → `10 10 X`) so the
|
||||
device's de-stuffer reproduces the original `10 X` pair. This applies most commonly
|
||||
to counters with `0x10` in the high byte (e.g. counter=`0x1000` produces logical
|
||||
params bytes `... 10 00 ...`, which BW encodes on the wire as `... 10 10 00 ...`).
|
||||
Without this stuffing the device interprets counter=`0x1000` as `0x0000` and returns
|
||||
the probe response (= a copy of the file header + STRT record); that STRT block then
|
||||
ends up embedded in the assembled file body and Blastware refuses to open the file.
|
||||
|
||||
`0x10` bytes in `offset_hi` are still written RAW per (1) above — only the params
|
||||
region has this stuffing requirement. Metadata-page params for counter `0x1002` /
|
||||
`0x1004` survive without stuffing because `10 02` / `10 04` fall in the "kept literal"
|
||||
carve-out.
|
||||
|
||||
Verified against BW 5-1-26 bwcap3sec frame 20: params logical bytes
|
||||
`00 01 11 10 00 00 00 00 00 00 00` (counter=0x1000) are encoded on the wire as
|
||||
`00 01 11 10 10 00 00 00 00 00 00 00` (12 wire bytes for 11 logical bytes).
|
||||
|
||||
#### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol)
|
||||
|
||||
> ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new
|
||||
@@ -1267,11 +1329,20 @@ when the broken 0x0400-step walk passed the global metadata pages at 0x1002/0x10
|
||||
the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004,
|
||||
not from the sample-chunk stream — see §7.8.7.
|
||||
|
||||
#### 7.8.3 A5 Frame Layout
|
||||
#### 7.8.3 A5 Frame Layout — DEPRECATED 2026-05-01
|
||||
|
||||
Each A5 response frame contains a chunk of raw bulk data. Frame 7 of the stream carries the
|
||||
compliance text block with all project-info label-value pairs. The `client` layer searches
|
||||
for ASCII labels with a null-terminated value read:
|
||||
> ⛔ **The "Frame 7 carries the compliance text block" claim below is WRONG.** It was
|
||||
> an artifact of the deprecated `0x0400`-step walk where the broken counter formula
|
||||
> happened to land sample-chunk fi=7 on top of the 0x1002 metadata page in flash.
|
||||
> Under the corrected v0.14.0+ walk (§7.8.5), Frame 7 of the sample-chunk sequence is
|
||||
> just sample-chunk #5 (counter=0x1000), and contains either ordinary waveform data or —
|
||||
> critically when DLE-stuffing of params is wrong (§7.8.1.3) — a duplicate file header +
|
||||
> STRT block when the device misinterprets counter=0x1000 as 0x0000. See §7.8.7 for the
|
||||
> actual source of these strings.
|
||||
|
||||
Historical claim (NOT TO BE IMPLEMENTED): each A5 response frame contains a chunk of raw
|
||||
bulk data; Frame 7 of the stream carries the compliance text block with all project-info
|
||||
label-value pairs:
|
||||
|
||||
```
|
||||
"Project:" → null-terminated project name
|
||||
@@ -1281,7 +1352,9 @@ for ASCII labels with a null-terminated value read:
|
||||
"Extended Notes" → null-terminated notes
|
||||
```
|
||||
|
||||
All five fields reflect the **setup at event-record time**, not the current device config.
|
||||
All five fields do reflect the **setup at event-record time**, not the current device
|
||||
config. But the source is the metadata pages (§7.8.7), not "Frame 7" of the sample
|
||||
stream.
|
||||
|
||||
#### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01
|
||||
|
||||
@@ -1560,10 +1633,10 @@ Fields visible in the Blastware "Compliance Setup" dialog. ✅ = byte offset co
|
||||
| Field | Values / Type | Status |
|
||||
|---|---|---|
|
||||
| Enable User Notes | bool | ❓ |
|
||||
| Project | ASCII string | ✅ (sourced from A5 frame 7 via SUB 5A) |
|
||||
| Client | ASCII string | ✅ (sourced from A5 frame 7) |
|
||||
| User Name | ASCII string | ✅ (sourced from A5 frame 7) |
|
||||
| Seis Loc | ASCII string | ✅ (sourced from A5 frame 7) |
|
||||
| Project | ASCII string | ✅ (sourced from SUB 5A metadata pages at counter `0x1002` / `0x1004` — see §7.8.7) |
|
||||
| Client | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) |
|
||||
| User Name | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) |
|
||||
| Seis Loc | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) |
|
||||
| Enable Extended Notes | bool | ❓ |
|
||||
| Extended Notes | ASCII text | ❓ |
|
||||
| Extended Notes Title | ASCII string | ❓ |
|
||||
|
||||
Reference in New Issue
Block a user