docs: update for v0.14.3 - Full continuous waveform download successful!
This commit is contained in:
@@ -27,7 +27,7 @@ CHANGELOG.md ← version history
|
||||
|
||||
---
|
||||
|
||||
## Current implementation state (v0.12.3)
|
||||
## Current implementation state (v0.14.3)
|
||||
|
||||
Full read pipeline + write pipeline + erase pipeline + monitor log + call home config working end-to-end over TCP/cellular:
|
||||
|
||||
@@ -41,14 +41,15 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c
|
||||
| Event header / first key | 1E | ✅ |
|
||||
| Waveform header | 0A | ✅ |
|
||||
| Waveform record (peaks, timestamp, project) | 0C | ✅ |
|
||||
| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ over-read bug fixed v0.13.0 (chunk loop bounded by STRT end_offset); minor wire diffs vs BW deferred — see "SUB 5A — chunk counter formula" |
|
||||
| **Bulk waveform stream (event-time metadata + full waveform)** | **5A** | ✅ **byte-perfect against BW captures (v0.14.3, 2026-05-05)** — STRT-bounded chunk walk + correct event-N probe counter + DLE-stuffed `0x10` bytes in params + concatenate-only file body assembly. All 17 5A request frames in the 5-1-26 3-sec capture reproduce byte-for-byte. |
|
||||
| Event advance / next key | 1F | ✅ |
|
||||
| **Write commands (push config to device)** | **68–83** | ✅ new v0.8.0 |
|
||||
| **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 |
|
||||
| **Monitor log entries (partial 0x2C records)** | **0A browse** | ✅ new v0.10.0 |
|
||||
| **Auto Call Home config (read + write)** | **2C → 7E → 7F** | ✅ **new v0.12.3** |
|
||||
|
||||
`get_events()` sequence per event: `1E → 0A → 0C → 5A → 1F`
|
||||
`get_events()` sequence per event: `1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)`
|
||||
(see "Correct iteration pattern" section below for full detail)
|
||||
|
||||
`push_config_raw()` write sequence: `68→73 | 71×3→72 | 82→83 | 69→74→72`
|
||||
|
||||
@@ -298,9 +299,8 @@ Two chunk addresses are GLOBAL device/session metadata, not event-specific:
|
||||
|
||||
These are at fixed absolute addresses in the device's flash buffer. They contain the
|
||||
session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII
|
||||
strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the
|
||||
new walk these strings come from the dedicated metadata pages, not from the sample-chunk
|
||||
stream.
|
||||
strings). Under the v0.14.0+ walk these strings are read directly from the metadata
|
||||
pages, not from the sample-chunk stream.
|
||||
|
||||
BW reads them ONCE per Blastware session (during event 1's download) and caches them.
|
||||
For SFM, that means:
|
||||
@@ -309,9 +309,10 @@ For SFM, that means:
|
||||
- Their content does not change when iterating events; only when the user opens
|
||||
Compliance Setup → Apply on the device or sends a SUB 71 compliance write.
|
||||
|
||||
The contents have not been byte-for-byte decoded yet — first task on the implementation
|
||||
side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the
|
||||
strings we currently extract from A5[7].
|
||||
The full byte-for-byte layout of the metadata pages has not been mapped — `_decode_a5_metadata_into`
|
||||
locates the ASCII strings via label scans (`Project:`, `Client:`, `User Name:`, `Seis Loc:`,
|
||||
`Extended Notes`) which works correctly across observed captures. Future work could
|
||||
dump the structural layout if more session-global fields need to be extracted.
|
||||
|
||||
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
|
||||
|
||||
@@ -319,16 +320,11 @@ strings we currently extract from A5[7].
|
||||
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
|
||||
Do not swap them.
|
||||
|
||||
### SUB 5A — event-time metadata source (UPDATED 2026-05-01)
|
||||
### SUB 5A — event-time metadata source (FINALIZED 2026-05-05)
|
||||
|
||||
> **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A
|
||||
> bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at
|
||||
> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because
|
||||
> the broken counter formula was scanning the wrong region.
|
||||
>
|
||||
> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and
|
||||
> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A
|
||||
> sample-chunk stream itself does NOT contain these strings any more under the new walk.
|
||||
The metadata strings come from the two fixed metadata pages at counter `0x1002` and
|
||||
`0x1004` (see "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above). These pages
|
||||
are GLOBAL session metadata — read once per Blastware/SFM session, not per event.
|
||||
|
||||
```
|
||||
"Project:" → project description
|
||||
@@ -338,55 +334,71 @@ Do not swap them.
|
||||
"Extended Notes"→ notes
|
||||
```
|
||||
|
||||
**IMPORTANT — 5A "Project:" is session-start config, NOT per-event (confirmed 2026-04-05):**
|
||||
The "Project:" string in the A5 frame 7 payload reflects the compliance setup from when
|
||||
the *monitoring session first started*, not the individual event's project name. The per-
|
||||
event project name is correctly stored in the 210-byte 0C waveform record and must be
|
||||
used as the authoritative source. `_decode_a5_metadata_into` therefore only sets
|
||||
`project` from 5A when 0C didn't already supply one.
|
||||
**IMPORTANT — these strings are session-start config, NOT per-event:**
|
||||
Project / Client / User Name / Seis Loc reflect the compliance setup from when the
|
||||
*monitoring session first started*, not the individual event's per-event metadata. The
|
||||
authoritative per-event project name is stored in the 210-byte 0C waveform record.
|
||||
`_decode_a5_metadata_into` therefore only sets `project` from the 5A metadata pages
|
||||
when 0C didn't already supply one.
|
||||
|
||||
"Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C
|
||||
record — 5A remains the sole source for those fields and they are set unconditionally.
|
||||
record — the metadata pages are the sole source for those fields and they are set
|
||||
unconditionally.
|
||||
|
||||
> ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and
|
||||
> stops one chunk later) is a workaround for the missing end_offset bound — when the new
|
||||
> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is
|
||||
> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM
|
||||
> frame.
|
||||
#### Deprecated knobs (do not re-introduce)
|
||||
|
||||
### SUB 5A — end-of-stream — UPDATED 2026-05-01
|
||||
The `read_bulk_waveform_stream()` function still accepts these legacy kwargs for
|
||||
backward compatibility, but they are **no-ops** under the v0.14.0+ walk:
|
||||
|
||||
> **Previous understanding (now known to be a symptom, not a feature):** "After streaming
|
||||
> all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was
|
||||
> not the device's natural end-of-event signal — it was the device's response when SFM had
|
||||
> walked clean off the end of the addressable buffer region after over-reading by ~5×.
|
||||
> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the
|
||||
> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`,
|
||||
> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser.
|
||||
- `stop_after_metadata=True` — used to scan the chunk stream for `b"Project:"` and stop
|
||||
one chunk later as a workaround for the missing end_offset bound. Obsolete: the loop
|
||||
is now deterministically bounded by `end_offset` parsed from the STRT record at
|
||||
data[17] of the probe response, with the partial tail fetched by the TERM frame.
|
||||
- `extra_chunks_after_metadata` — same era, same reason. No-op.
|
||||
|
||||
The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful
|
||||
defence-in-depth fallback for malformed events or unexpected device states, but should not
|
||||
be the primary loop-exit condition.
|
||||
If you find code or docs referencing "A5 frame 7" as the source of metadata strings,
|
||||
that's an old-walk artifact (the broken `0x0400`-step formula occasionally caught the
|
||||
0x1002 metadata page at sample-chunk fi=7). Update to reference the dedicated metadata
|
||||
pages instead.
|
||||
|
||||
**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
|
||||
Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call
|
||||
in the chunk loop passes `timeout=10.0` explicitly.
|
||||
### SUB 5A — end-of-stream (FINALIZED 2026-05-01)
|
||||
|
||||
**Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):**
|
||||
A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames.
|
||||
A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames.
|
||||
An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames.
|
||||
Under the v0.14.0+ STRT-bounded walk the stream ends cleanly:
|
||||
|
||||
Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks
|
||||
17-37 containing post-event circular-buffer garbage that corrupted the file body.
|
||||
```
|
||||
… last full chunk at counter < end_offset
|
||||
TERM request (offset_word = end_offset - next_boundary,
|
||||
params address (next_boundary))
|
||||
TERM response (page_key = 0x0000 or 0x0001, data = the residual
|
||||
end_offset - next_boundary bytes including the file footer)
|
||||
```
|
||||
|
||||
No timeout-based detection, no "1-byte teaser," no `max_chunks` cap. The chunk loop
|
||||
exits when `counter + 0x0200 > end_offset`; the TERM frame fetches the tail.
|
||||
|
||||
**Chunk recv timeout is 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
|
||||
Using 120 s would cause a ~2-minute stall on any unexpected timeout. The `_recv_one`
|
||||
call in the chunk loop passes `timeout=10.0` explicitly.
|
||||
|
||||
**Typical chunk count under the v0.14.0+ walk (BE11529, 1024 sps over TCP/cellular):**
|
||||
|
||||
| Event duration | Sample chunks | Metadata pages | TERM | Total A5 frames |
|
||||
|---|---|---|---|---|
|
||||
| 2-sec (event 1) | ~12 | 2 | 1 | ~15 |
|
||||
| 3-sec (event 1) | 13 | 2 | 1 | 16 |
|
||||
| 2-sec (continuation) | 15 | 0 | 1 | 16 |
|
||||
| 3-sec (continuation) | ~14 | 0 | 1 | ~15 |
|
||||
|
||||
For comparison, the deprecated `0x0400`-step walk produced ~37 chunks for a 2-sec
|
||||
event with chunks 17-37 containing post-event circular-buffer garbage. Do not
|
||||
re-introduce that walk under any circumstances.
|
||||
|
||||
### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06)
|
||||
|
||||
`_decode_a5_waveform()` previously had `elif fi == 9: continue` — a leftover from the
|
||||
9-frame original blast capture where frame 9 was assumed to be a terminator. For current
|
||||
35-frame streams, fi==9 is live waveform data (~133 sample-sets were being dropped).
|
||||
Removed. Terminator detection is via `page_key == 0x0000` in `read_bulk_waveform_stream`,
|
||||
not frame index.
|
||||
9-frame original blast capture where frame 9 was assumed to be a terminator. Removed.
|
||||
TERM detection in the file builder uses `frame.page_key != 0x0010` (sample marker),
|
||||
not frame index — see `blastware_file.py`.
|
||||
|
||||
### SUB 1E / 1F — event iteration null sentinel and token position (FIXED, do not re-introduce)
|
||||
|
||||
@@ -1029,7 +1041,7 @@ offsets in the raw 1A/E5 payload. Only fields with `✅` have confirmed offsets
|
||||
|
||||
**Notes tab:**
|
||||
- Enable User Notes (bool)
|
||||
- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from A5 frame 7 via 5A)
|
||||
- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from 5A metadata pages at counter 0x1002 / 0x1004 — see "SUB 5A — fixed metadata pages" section)
|
||||
- Enable Extended Notes (bool); Extended Notes text; Extended Notes Title
|
||||
- Enable Job Number (bool); Job Number (int)
|
||||
- Enable Scaled Distance (bool); Distance from Blast (float); Charge Weight (float) — Scaled Distance is derived
|
||||
@@ -1343,7 +1355,7 @@ body) because writing a dial string may require DLE escaping for embedded contro
|
||||
|
||||
- **Database** — SQLite store for events + monitor log entries; dedup by key; queryable
|
||||
- **Histograms** — decode histogram-mode A5 data (noise floor tracking)
|
||||
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed working for Continuous mode events (2026-04-23):** SFM-generated file opens in Blastware, shows correct PPV/waveform/timestamp. File is ~200 bytes shorter than BW (missing last ADC tail slice) — all measurements correct. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that create spurious STRT markers in the body). Extension mapping: **CONFIRMED FALSE 2026-04-21** — extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
|
||||
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed BYTE-PERFECT against BW reference (v0.14.3, 2026-05-05):** when fed the BW 5-1-26 3-sec capture's A5 frames, the SFM-built file matches BW's saved `M529LKIQ.G10` byte-for-byte (8708 bytes, 0 differences). Live SFM downloads of event 0 (3-sec) and event 1 (3-sec continuation) both open cleanly in Blastware with full Event Reports, frequency analysis, and waveform plots. Body assembly is just contiguous concatenation of frame contributions in stream order (probe → meta@0x1002 → meta@0x1004 → samples → TERM); no stripping, no overlay, no special handling. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that may need different handling — untested under v0.14.x). Extension mapping: extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
|
||||
|
||||
**Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units).
|
||||
|
||||
@@ -1379,16 +1391,21 @@ body) because writing a dial string may require DLE escaping for embedded contro
|
||||
|
||||
| Folder / File | Contents |
|
||||
|---|---|
|
||||
| `1-2-26/` | First SUB 5A BW TX capture — established 5A frame format (raw offset_hi, DLE-aware checksum). 10 frames verified. |
|
||||
| `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102–112) |
|
||||
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) — 1E/0A/0C/1F sequence confirmed (single event so token=0xFE appeared to work in either branch) |
|
||||
| `4-2-26/` | Download-mode BW TX capture — POLL×3 requirement confirmed (frames 68-73 between 1F and first 5A) |
|
||||
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events — all-zero params for 1F, null sentinel layout, 0A context requirement |
|
||||
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check |
|
||||
| `4-11-26 (mitm/ach_mitm_20260411_001912/)` | Full ACH call-home MITM — erase protocol (0xA3/0x06/0xA2), monitor log partial records confirmed |
|
||||
| `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous |
|
||||
| `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec |
|
||||
| `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) |
|
||||
| `4-20-26/call home settings/` | Call home config read/write captures |
|
||||
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check |
|
||||
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events (1E/0A/1F iteration confirmed) |
|
||||
| `4-2-26/` | Download-mode BW TX capture (5A bulk stream, POLL×3 requirement confirmed) |
|
||||
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) |
|
||||
| `mitm/ach_mitm_20260411_001912/` | Full ACH call-home MITM (erase protocol, 0xA3/0x06/0xA2 confirmed) |
|
||||
| `4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof of 5× SFM over-read. STRT end_key field located. |
|
||||
| **`5-1-26/comcheck/`** | **Triplet of captures that nailed the v0.14.0 walk:** SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945` / `_171216`). Confirmed: TERM frame formula across 3 events, metadata pages 0x1002/0x1004 are global session metadata, event-1 vs event-N chunk pattern split, WAVEHDR off=0x46 vs 0x2C disambiguates real events from boundaries. |
|
||||
| **`5-1-26/comcheck/bwcap3sec/`** | **The byte-perfect reference for v0.14.3.** All 17 BW 5A request frames (probe, 2 metadata, 13 samples, TERM) reproduce byte-for-byte from SFM's framing helpers — including the `10 10 00` DLE-stuffed counter for sample @ 0x1000 that was the long-standing failure mode. |
|
||||
| `5-4-26/` | BW MITM captures of "copy 3sec / 2sec / Download All" + paired SFM session (`seismo_dl_20260504_145701`) showing the +0x46 event-N probe bug producing 110-chunk runaway walk. Cross-references against 5-1-26 confirmed device behavior is identical. |
|
||||
|
||||
To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern
|
||||
in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes
|
||||
|
||||
Reference in New Issue
Block a user