strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the
new walk these strings come from the dedicated metadata pages, not from the sample-chunk
stream.
BW reads them ONCE per Blastware session (during event 1's download) and caches them.
For SFM, that means:
- Once per call-home / once per `MiniMateClient.connect()` is enough.
- Subsequent events in the same session don't need to re-fetch them.
- Their content does not change when iterating events; only when the user opens
Compliance Setup → Apply on the device or sends a SUB 71 compliance write.
The contents have not been byte-for-byte decoded yet — first task on the implementation
side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the
strings we currently extract from A5[7].
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
@@ -140,10 +275,16 @@ counter and streams data for any valid 5A request; using `chunk_num * 0x0400` is
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
Do not swap them.
### SUB 5A — event-time metadata lives in A5 frame 7
### SUB 5A — event-time metadata source (UPDATED 2026-05-01)
The bulk stream sends 9+ A5 response frames. Frame 7 (0-indexed) contains the compliance
setup as it existed when the event was recorded:
> **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A
> bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at
> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because
> the broken counter formula was scanning the wrong region.
>
> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and
> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A
> sample-chunk stream itself does NOT contain these strings any more under the new walk.
```
"Project:" → project description
@@ -163,26 +304,37 @@ used as the authoritative source. `_decode_a5_metadata_into` therefore only set
"Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C
record — 5A remains the sole source for those fields and they are set unconditionally.
`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears,
then sends the termination frame.
> ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and
> stops one chunk later) is a workaround for the missing end_offset bound — when the new
> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is
> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM
> frame.
### SUB 5A — end-of-stream signal (confirmed 2026-04-06)
### SUB 5A — end-of-stream — UPDATED 2026-05-01
Afterstreaming all waveform chunks, the device sends exactly **1 raw byte** in response to
the next chunk request, then goes silent. This is the natural end-of-stream indicator — NOT
a complete A5 frame. `S3FrameParser.bytes_fed` will be 1; no frame is assembled.
> **Previous understanding (now known to be a symptom, not a feature):** "After streaming
> all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was
> not the device's natural end-of-event signal — it was the device's response when SFM had
> walked clean off the end of the addressable buffer region after over-reading by ~5×.
> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the
> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`,
> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser.
Handling: on `TimeoutError`, if `bytes_fed > 0` AND frames were already collected, treat as
graceful end-of-stream, break the loop, and proceed to the termination frame. If `bytes_fed
== 0` with no prior frames, it is a genuine transport failure — re-raise.
The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful
defence-in-depth fallback for malformed events or unexpected device states, but should not
be the primary loop-exit condition.
**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call
in the chunk loop passes `timeout=10.0` explicitly.
**Typical chunk count (BE11529, 1024 sps):** A 9,306-sample event produces 35 chunks before
end-of-stream. Chunks with uniform 1,036-byte data are all-zero ADC samples (post-event
silence). Only the initial variable-size chunks contain actual signal.
**Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):**
A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames.
A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames.
An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames.
Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks
17-37 containing post-event circular-buffer garbage that corrupted the file body.
### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06)
@@ -295,6 +447,55 @@ sends token=0xFE and is NOT used by any caller.
`advance_event()` returns `(key4, event_data8)`.
Callers (`count_events`, `get_events`) loop while `data8[4:8] != b"\x00\x00\x00\x00"`.
### SUB 0A — WAVEHDR response length distinguishes events from boundaries (NEW 2026-05-01)
When iterating events with the "Download All" pattern (1E → 0A → 1F → 0A → 1F → …), the
DATA_LENGTH at `data_rsp.data[5]` (= the byte BW echoes back as the offset for the data
fetch step) takes one of two values:
| WAVEHDR offset | Meaning |
|---|---|
| `0x46` (= 70) | Real event start key — there is event data at this address |
| `0x2C` (= 44) | Boundary marker between events — this key is the END of the previous event AND the START key for the empty space after it (or is the next event's pre-header) |
0A(key=0111417E) → off=0x2C ← event 2 END / next-empty marker
1F → null sentinel
```
This is why event 2's first 5A chunk is at `start_key + 0x46` — that's the address of the
"real start" 0x46-record, distinct from the `0x2C`-record at the raw boundary. Use the
`0x46` keys as the input to `read_bulk_waveform_stream`, not the `0x2C` keys.
For event 1 only (start_key[2:4] = 0x0000) BW probes at counter=0x0000 directly, which is
the `0x46`-keyed start record. Subsequent events use `start_key + 0x46`.
**Practical iteration pattern (replaces the old 1E/1F walk for downloads):**
```
Setup: SERIAL × 2 → CHCFG → 1E (token=0x00) → key0
For each event:
0A(cur_key) → DATA_LENGTH = 0x46 (real) or 0x2C (boundary)
1F (token=0x00) → next_key
if length was 0x46: → cur_key is a real event; queue it for download
cur_key = next_key
if next_key all-zero null sentinel: stop
Then for each queued real-event key:
download_event(key) → 5A bulk stream with STRT-bounded chunk walk
```
This is what BW does in the 5-1-26 "Download All" capture — it walks the full event chain
collecting `(key, length)` tuples first, *then* downloads each event using the `0x46` keys.
### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce)
`read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where:
@@ -386,7 +587,9 @@ bytes `\x01\x2c` = 300 (5-minute default histogram interval); changes when inter
| Offset | Field | Format | Notes |
|---|---|---|---|
| anchor −7 (write) / anchor − 8 (read) | recording_mode | uint8 | E5 read has extra `0x10` at anchor−7 |
| anchor −9 | mode_prefix | uint8 | `0x00` for Single Shot / Continuous; `0x10` for Histogram (DLE prefix in E5 encoding) and Histogram+Continuous (actual config byte). See "compliance_raw DLE encoding" note below. |
| anchor − 8 | recording_mode | uint8 | **Same offset for both read and write** — confirmed 2026-04-21. `_encode_compliance_config` writes `buf[anc-8]`. NOTE: for Histogram (0x03), E5 encodes the value as `0x10 0x03` so compliance_raw[anc-9]=0x10, compliance_raw[anc-8]=0x03. |
| anchor − 7 | constant | `0x10` | Always `0x10` in both E5 read and BW write payloads (not a DLE marker — it is part of the sample_rate field area). Do NOT overwrite. |
| anchor − 6 | sample_rate | uint16 BE | same in read & write |
| anchor − 4 | histogram_interval_sec | uint16 BE | seconds; same in read & write ✅ 2026-04-20 |
| anchor − 2 | `0x00 0x00` | padding | |
@@ -395,15 +598,42 @@ bytes `\x01\x2c` = 300 (5-minute default histogram interval); changes when inter
**recording_mode enum** (confirmed 2026-04-20 from 4-20-26 captures):
| Value | Mode |
|---|---|
| `0x00` | Single Shot |
| `0x01` | Continuous |
| `0x02` | ❓ not observed |
| `0x03` | Histogram |
| `0x04` | Histogram + Continuous |
| Value | Mode | anchor-9 in compliance_raw |
|---|---|---|
| `0x00` | Single Shot |`0x00` |
| `0x01` | Continuous |`0x00` |
| `0x02` | ❓ not observed | ❓ |
| `0x03` | Histogram |`0x10` (DLE prefix from E5 wire encoding of 0x03) |
| `0x04` | Histogram + Continuous |`0x10` (actual config byte for this mode) |
**DLE escaping in write frames — CONFIRMED 2026-04-20:** Write frame data payloads DO escape `0x03` (ETX) bytes with a `0x10` DLE prefix. For histogram_interval = 900 (0x0384), the wire carries `10 03 84` — the `0x03` high byte is preceded by a DLE escape. After DLE destuffing (`10 XX → XX`), the logical field value is correctly `03 84` = 900. The CLAUDE.md claim that write frame data is "written RAW" was incorrect; at minimum ETX (0x03) bytes are escaped. S3FrameParser handles this transparently so the decoded `compliance_raw` always contains logical (destuffed) bytes.
**compliance_raw DLE encoding — IMPORTANT (confirmed 2026-04-21 from 4-20-26 captures):**
`compliance_raw` (returned by `read_compliance_config()`) is NOT purely logical bytes — it is
the wire-encoded representation where `0x03` bytes in the config are preceded by a `0x10` DLE
prefix (because S3FrameParser preserves DLE+ETX inner-frame pairs as two literal bytes).
Consequences:
- When recording_mode = `0x03` (Histogram), `compliance_raw[anc-9] = 0x10` (DLE prefix) and
`compliance_raw[anc-8] = 0x03` (the value). The anchor position is +1 compared to modes
without `0x03` bytes before the anchor.
- For Histogram+Continuous (`0x04`), `compliance_raw[anc-9] = 0x10` for a different reason:
it is an actual stored config byte, not a DLE prefix.
- The anchor search (`buf.find(b'\xbe\x80\x00\x00\x00\x00', 0, 150)`) correctly locates
the anchor regardless of these mode-dependent shifts.
- When SFM writes recording_mode and round-trips the rest verbatim, the byte at `anc-9` is
preserved from the previous read. This means transitioning Histogram→other modes via SFM
leaves a `0x10` at `anc-9`. The device stores it as a literal byte; it does not affect
recording mode operation (which is at `anc-8`), but differs from what BW writes. This is a
known minor discrepancy that does not impact device behavior.
- **Histogram recording mode (0x03) write via SFM**: untested. When starting from a mode with
`anc-9 = 0x00`, SFM writes bare `0x03` at anc-8. BW would write `0x10 0x03`. Device likely
accepts both (write frames probably use offset/length for framing, not ETX scanning).
**DLE escaping in write frames — confirmed 2026-04-20:** Blastware escapes `0x03` bytes in
write frame data as `0x10 0x03` on the wire (defensive ETX escaping). Our `build_bw_write_frame`
does NOT do this escaping — it sends data bytes raw. Device acceptance of bare `0x03` bytes
in write frame data is confirmed for the tested modes (Single Shot, Continuous, Histogram+Continuous
where `0x10 0x03` already appears from round-tripping). Histogram mode (bare `0x03` write from
non-Histogram starting state) has not been directly tested.
### SUB 0C — Waveform Record (210 bytes = data[11:11+0xD2])
@@ -490,6 +720,8 @@ All DB endpoints are read-only except `PATCH /db/events/{id}/false_trigger`.
| 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture |
| 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence; only 1 event stored so token=0xFE appeared to work |
| 4-27-26 | `bridges/captures/4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof that SFM was over-reading 5× past event end. BW reads 14 chunks at 0x0200 increments + TERM at end_offset; SFM was reading 37 chunks at 0x0400 increments. STRT end_key field located. |
| 5-1-26 | `bridges/captures/5-1-26/comcheck/` | Three sub-captures: SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945`/`_171216`). Confirmed: TERM frame formula across 3 events; metadata pages 0x1002/0x1004 are global (read once per session); event-1 vs event-N chunk-pattern split; WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundaries. |
---
@@ -1067,9 +1299,53 @@ body) because writing a dial string may require DLE escaping for embedded contro
- **Database** — SQLite store for events + monitor log entries; dedup by key; queryable
- **Histograms** — decode histogram-mode A5 data (noise floor tracking)
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed working for Continuous mode events (2026-04-23):** SFM-generated file opens in Blastware, shows correct PPV/waveform/timestamp. File is ~200 bytes shorter than BW (missing last ADC tail slice) — all measurements correct. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that create spurious STRT markers in the body). Extension mapping: **CONFIRMED FALSE 2026-04-21** — extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
**Serial encoding (CONFIRMED 2026-04-22):**`prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units).
**Stem encoding (FULLY CONFIRMED 2026-04-22):** stem = 4-char base-36 of `floor(total_seconds / 1296)` where `total_seconds = (event_local_time − 1985-01-01T00:00:00_local)` in seconds. Epoch = `1985-01-01 00:00:00` device local time — confirmed against 3,248 files from 10-year production archive with zero errors. Decode: `event_time = datetime(1985,1,1) + timedelta(seconds=stem_int*1296 + ab_int)`. Example: P036L318.C80H → BE14036, 2025-05-26 15:00:08, Full Histogram.
- **Blastware filename extension — NEW FIRMWARE FULLY DECODED (confirmed 2026-04-21, further confirmed 2026-04-22 from 10-year production archive frequency analysis):**
Extension format = `AB0T` (4 chars):
-`AB` = 2-char base-36 encoding of `total_seconds % 1296` (seconds within the 21.6-min window, 0–1295); `A = value // 36`, `B = value % 36`
-`0` = always literal digit zero (third character, invariant)
-`T` = event type: `W` = Full Waveform, `H` = Full Histogram
Combined with the 4-char stem, the full filename encodes a complete second-resolution timestamp. Verified against three S353L4H0.{3M0W,8S0H,9X0W} events (all match to the second) plus large-scale frequency analysis of a 10-year archive.
**3-day cycle property (confirmed 2026-04-22):** A unit recording at a fixed daily time cycles through exactly **3 extensions** with a 3-day period. Each calendar day shifts `total_seconds % 1296` by 864 (since `86400 % 1296 = 864`). The cycle repeats every 3 days because `gcd(1296, 864) = 432` and `1296 / 432 = 3`. The three extension values are spaced 432 seconds apart. Confirmed from 10-year archive: the top 3 extensions overall were `CE0H` (95 files), `0E0H` (93), `OE0H` (91) — all three are the 3-day cycle of a 06:00:14 daily call-in time (seconds-in-window = 14, 446, 878; all three have `E` as second character because `14 = E` in base-36 and adding 864 never changes `value % 36` since `864 = 24 × 36`).
**B character invariance:** For a unit recording at a fixed time of day, the second character `B` of the extension (`value % 36`) **never changes** — only the first character `A` cycles through 3 values. This means same-time-of-day files from different dates all share the same `B` character.
**Old firmware (S338, 3-char extensions ending in `0`):** encoding unknown. Extension is NOT recording mode. `blastware_filename()` returns `.N00` as a placeholder for old-firmware units.
**Micromate Series 4** uses a different extension format entirely (observed: `IDFH`, `IDFW`). The `AB0T` formula applies only to MiniMate Plus / V10.72 firmware.
- Compliance config encoder — build raw write payloads from a `ComplianceConfig` object
- **Test Histogram recording mode (0x03) write via SFM** — confirmed working for Single Shot / Continuous / Histogram+Continuous; Histogram (0x03) needs a live test from a non-Histogram starting state (bare 0x03 in write vs BW's DLE-escaped `10 03`)
- **Compliance write anchor-9 cleanup** — when changing recording_mode via SFM, the byte at anchor-9 is not explicitly managed. A spurious `0x10` may persist after Histogram→other mode transitions. Does not affect device operation but differs from BW's byte-perfect output.
- Locate "Sensor Check" byte in compliance config (need capture with Disabled vs Before-monitoring)
- Call Home — map time slots 3/4 offsets; add dial_string write support; confirm `modem_power_relay_enabled`
- Modem manager — push RV50/RV55 configs via Sierra Wireless API
- RV55 DCD/DTR issue — newer RV55 firmware doesn't assert DCD by default; units don't
resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred)