feat: add full event download pipeline

2026-03-31 20:48:03 -04:00
parent 6a0422a6fc
commit 9f52745bb4
4 changed files with 595 additions and 129 deletions
--- a/docs/instantel_protocol_reference.md
+++ b/docs/instantel_protocol_reference.md
@@ -209,13 +209,13 @@ Step 4 — Device sends actual data payload:
 | `08` | **EVENT INDEX READ** | Requests the event record index (0x58 bytes). Event count and record pointers. | ✅ CONFIRMED |
 | `06` | **CHANNEL CONFIG READ** | Requests channel configuration block (0x24 bytes). | ✅ CONFIRMED |
 | `1C` | **TRIGGER CONFIG READ** | Requests trigger settings block (0x2C bytes). | ✅ CONFIRMED |
-| `1E` | **EVENT HEADER READ** | Reads event header by index. Contains timestamp and sample rate. | ✅ CONFIRMED |
-| `0A` | **WAVEFORM HEADER READ** | Reads waveform header keyed by timestamp (0x30 bytes/page). | ✅ CONFIRMED |
-| `0C` | **FULL WAVEFORM RECORD** | Downloads complete waveform record (0xD2 bytes/page, 2 pages). Project strings, PPV floats, channel labels. | ✅ CONFIRMED |
-| `5A` | **BULK WAVEFORM STREAM** | Initiates bulk download of raw ADC sample data, keyed by timestamp. Large multi-page transfer. | ✅ CONFIRMED |
+| `1E` | **EVENT HEADER READ** | Gets the first waveform key (4-byte opaque record address). All-zero params; key returned at data[11:15]. | ✅ CONFIRMED 2026-03-31 |
+| `0A` | **WAVEFORM HEADER READ** | Checks record type for a given waveform key. Variable DATA_LENGTH: 0x30=full bin, 0x26=partial bin. Key at params[4..7]. | ✅ CONFIRMED 2026-03-31 |
+| `0C` | **FULL WAVEFORM RECORD** | Downloads 210-byte waveform/histogram record. Contains record type, PPV floats (at channel label+6), project strings, 7-byte timestamp. Key at params[4..7], DATA_LENGTH=0xD2. | ✅ CONFIRMED 2026-03-31 |
+| `1F` | **EVENT ADVANCE** | Advances to next waveform key. Token byte at params[6]: 0x00=browse (one step), 0xFE=download (skip partial bins). Returns next key at data[11:15]; zeros = no more events. | ✅ CONFIRMED 2026-03-31 |
+| `5A` | **BULK WAVEFORM STREAM** | Initiates bulk download of raw ADC sample data, keyed by waveform key. Large multi-page transfer. | ✅ CONFIRMED |
 | `24` | **WAVEFORM PAGE A?** | Paged waveform read, possibly channel group A. | 🔶 INFERRED |
 | `25` | **WAVEFORM PAGE B?** | Paged waveform read, possibly channel group B. | 🔶 INFERRED |
-| `1F` | **EVENT ADVANCE / CLOSE** | Sent after waveform download completes. Likely advances internal record pointer. | 🔶 INFERRED |
 | `09` | **UNKNOWN READ A** | Read command, response (`F6`) returns 0xCA (202) bytes. Purpose unknown. | 🔶 INFERRED |
 | `1A` | **CHANNEL SCALING / COMPLIANCE CONFIG READ** | Read command, response (`E5`) returns large block containing IEEE 754 floats including trigger level, alarm level, max range, and unit strings. Contains `0x082A` — purpose unknown, possibly alarm threshold or record config. | 🔶 INFERRED |
 | `2E` | **UNKNOWN READ B** | Read command, response (`D1`) returns 0x1A (26) bytes. Purpose unknown. | 🔶 INFERRED |
@@ -238,7 +238,7 @@ All requests use CMD byte `0x02`. All responses use CMD byte `0x10 0x02` (which,
 | `0A` | `F5` | ✅ CONFIRMED |
 | `0C` | `F3` | ✅ CONFIRMED |
 | `5A` | `A5` | ✅ CONFIRMED |
-| `1F` | `E0` | 🔶 INFERRED |
+| `1F` | `E0` | ✅ CONFIRMED 2026-03-31 |
 | `09` | `F6` | ✅ CONFIRMED |
 | `1A` | `E5` | ✅ CONFIRMED |
 | `2E` | `D1` | ✅ CONFIRMED |
@@ -624,25 +624,36 @@ Several settings are **mode-gated**: the device only transmits (reads) or accept

 ---

-### 7.5 Full Waveform Record (SUB F3) — 0xD2 bytes × 2 pages
+### 7.5 Full Waveform Record (SUB F3) — 0xD2 bytes (210 bytes)

-Peak values as IEEE 754 big-endian floats (restored section header):
+> ✅ **Updated 2026-03-31** — Full layout confirmed. See §7.7.5 for the
+> complete record structure including timestamp, record type, PPV float
+> positions, and project strings.

+Peak values are found by searching for channel label strings `"Tran"`,
+`"Vert"`, `"Long"`, `"MicL"` and reading `float32 BE` at `label_offset + 6`.
+The floats are **not 4-byte aligned** — confirmed from 3-31-26 capture.
+
+Example peak values (event 1 from 3-31-26):
 ```
-Tran:  3D BB 45 7A  =  0.0916  (in/s — unit config dependent)
-Vert:  3D B9 56 E1  =  0.0907
-Long:  3D 75 C2 7C  =  0.0605
-MicL:  39 BE 18 B8  =  0.000145  (PSI or dB linear — ❓ units unconfirmed)
+Tran:  3D BB 45 7A  =  0.0916 in/s
+Vert:  3D B9 56 E1  =  0.0907 in/s
+Long:  3D 75 C2 7C  =  0.0605 in/s
+MicL:  39 BE 18 B8  =  0.000145 psi  ✅ units confirmed
 ```

-Peak values — event 2:
+Example peak values (event 2 from earlier capture):
 ```
-Tran:  3D 56 CB B9  =  0.0521
-Vert:  3C F5 C2 7C  =  0.0300
-Long:  3C F5 C2 7C  =  0.0300
-MicL:  39 64 1D AA  =  0.0000875
+Tran:  3D 56 CB B9  =  0.0521 in/s
+Vert:  3C F5 C2 7C  =  0.0300 in/s
+Long:  3C F5 C2 7C  =  0.0300 in/s
+MicL:  39 64 1D AA  =  0.0000875 psi
 ```

+> ⚠️ The record is delivered as `data_rsp.data[11:11+0xD2]` — the outer
+> data section header (LENGTH_ECHO, KEY_ECHO) occupies data[0..10].
+> Callers of `read_waveform_record()` receive the 210-byte record directly.
+
 ### 7.6 Bulk Waveform Stream (SUB A5) — Raw ADC Sample Records

 Each repeating record (🔶 INFERRED structure):
@@ -662,6 +673,186 @@ Each repeating record (🔶 INFERRED structure):

 ---

+### 7.7 Event Download Protocol — Confirmed from 3-31-26 Capture ✅
+
+> **Added 2026-03-31.** All findings confirmed from live bridge capture
+> `bridges/captures/3-31-26/raw_bw_20260331_200245.bin` +
+> `raw_s3_20260331_200245.bin` (148 BW frames / 147 S3 frames).
+> Analysis scripts: `parsers/analyze_3_31_26.py`.
+
+#### Overview
+
+Event download uses four SUBs in a key-driven iterator loop.  The
+"waveform key" is a 4-byte opaque record address that uniquely identifies
+one histogram bin or waveform record on the device's internal storage.
+
+| Step | BW SUB | S3 Response | Purpose |
+|---|---|---|---|
+| 1 (once) | `1E` — EVENT_HEADER | `E1` | Get the first waveform key |
+| 2 | `0A` — WAVEFORM_HEADER | `F5` | Check record type / confirm full bin |
+| 3 | `0C` — WAVEFORM_RECORD | `F3` | Download 210-byte record (peaks, project, timestamp) |
+| 4 | `1F` — EVENT_ADVANCE | `E0` | Advance iterator, get next key |
+| ↑ repeat steps 2–4 until key == `00 00 00 00` | | | |
+
+**Blastware optimisation (confirmed):** Step 2 (0A) is only called for the
+_first_ key.  Subsequent keys come from `1F` with token `0xFE` (download
+mode), which guarantees they are full records — so Blastware skips 0A and
+jumps directly to 0C.  Our implementation follows the same pattern.
+
+---
+
+#### 7.7.1 Waveform Key
+
+The waveform key is a 4-byte opaque record address (`uint32`, likely
+a flash sector offset or circular-buffer pointer internal to the S3 DSP).
+
+- First key: returned by `1E` at `data[11:15]`
+- Subsequent keys: returned by `1F` at `data[11:15]`
+- Terminator: `00 00 00 00` signals no more events
+
+Example keys from 3-31-26 capture (one Blastware "event" / 4 histogram bins):
+```
+01 11 00 16   ← first bin  (full, 0x30 length)
+01 11 11 B6   ← second bin (partial, 0x26 length — skipped by 1F/0xFE)
+01 11 11 F6   ← third bin  (partial, 0x26 length — skipped)
+01 11 12 36   ← fourth bin (full, 0x30 length — returned by 1F/0xFE)
+00 00 00 00   ← terminator
+```
+
+---
+
+#### 7.7.2 Token Byte (SUB 1E / 1F)
+
+A token byte at `payload[12]` (= `params[6]` in `build_bw_frame`) controls
+the 1F advance behaviour:
+
+| Token | Mode | Behaviour |
+|---|---|---|
+| `0x00` | Browse | Advance one record, including partial histogram bins |
+| `0xFE` | Download | Skip partial bins, advance to the next full record |
+
+**We always use `0xFE`** — it minimises round trips and avoids needing to
+handle partial-bin `0C` calls.
+
+---
+
+#### 7.7.3 Variable DATA_LENGTH for SUB 0A (WAVEFORM_HEADER)
+
+Unlike all other SUBs, `0A` does NOT have a fixed data length.  The length
+is returned in the probe response at `data[4]`:
+
+| Length | Meaning |
+|---|---|
+| `0x30` | Full histogram bin — has a waveform record to download |
+| `0x26` | Partial histogram bin — no waveform record |
+
+Both the probe and data-request frames carry the same key in `params[4..7]`.
+The `read_waveform_header()` method in `protocol.py` reads `probe.data[4]`
+and uses that value as the data-request offset.
+
+---
+
+#### 7.7.4 Response Data Section Layout
+
+**All S3 event download responses** share this data section prefix:
+
+```
+data[0]    LENGTH_ECHO  — echoes the request DATA_LENGTH byte
+data[1..4] 00 00 00 00  — four zero bytes
+data[5..8] KEY_ECHO     — echoes the 4-byte waveform key from the request
+data[9..10] 00 00       — two zero bytes
+data[11..] ACTUAL_DATA  — real payload starts here
+```
+
+Actual data lengths:
+- `1E` response (`E1`): `data[11:19]` — 8 bytes (`data[11:15]` = key4)
+- `0A` probe response (`F5`): `data[4]` = variable length (0x30 or 0x26)
+- `0A` data response (`F5`): `data[11:11+length]` — waveform header bytes
+- `0C` data response (`F3`): `data[11:11+0xD2]` — 210-byte waveform record
+- `1F` response (`E0`): `data[11:15]` = next key4; `data[8]` = token echo
+
+---
+
+#### 7.7.5 Waveform Record Layout (210 bytes, SUB F3 → response F3)
+
+The 210-byte record (`data_rsp.data[11:11+0xD2]`) contains:
+
+**Record type string** (search at variable offset):
+- `"Histogram"` — histogram mode recording
+- `"Waveform"` — single-shot waveform recording
+
+**Timestamp** (7-byte format, confirmed from 3-31-26 capture):
+```
+byte 0:    0x09 (magic/type marker)
+bytes 1–2: year (uint16 big-endian)
+byte 3:    0x00
+byte 4:    hour
+byte 5:    minute
+byte 6:    second
+```
+> ❓ Month and day are not present in the waveform record timestamp.
+> Month/day may appear in the event index (SUB F7) or a separate header
+> field not yet confirmed.
+
+**Peak particle velocity floats** (✅ CONFIRMED 2026-03-31):
+
+Channel labels `"Tran"`, `"Vert"`, `"Long"`, `"MicL"` are embedded as
+ASCII strings at variable offsets within the record.  The PPV float for
+each channel is at `label_offset + 6` (IEEE 754 big-endian float32).
+
+The floats are **NOT 4-byte aligned** — Tran, Long, and MicL all fall at
+non-aligned offsets.  The previous heuristic step-4 scanner missed all three.
+
+Example from 3-31-26 capture:
+```
+"Tran" at offset N  →  float at N+6  =  0.0916 in/s
+"Vert" at offset M  →  float at M+6  =  0.0907 in/s
+"Long" at offset P  →  float at P+6  =  0.0605 in/s
+"MicL" at offset Q  →  float at Q+6  =  0.000145 psi
+```
+
+Channel labels are separated by inner-frame bytes `10 03` (DLE ETX),
+preserved as literal data by `S3FrameParser`.
+
+**Project strings** — ASCII label-value pairs (search for label, read null-terminated value):
+```
+"Project:"      → project description
+"Client:"       → client name  ✅ offset confirmed
+"User Name:"    → operator / user
+"Seis Loc:"     → sensor location
+"Extended Notes"→ notes field
+```
+
+---
+
+#### 7.7.6 Complete Download Loop (Python pseudocode)
+
+```python
+key4, _ = proto.read_event_first()            # SUB 1E
+if key4 == b'\x00\x00\x00\x00':
+    return []  # no events
+
+events = []
+is_first = True
+
+while key4 != b'\x00\x00\x00\x00':
+    if is_first:
+        _header, rec_len = proto.read_waveform_header(key4)  # SUB 0A
+        is_first = False
+        if rec_len < 0x30:
+            key4 = proto.advance_event()   # skip partial first bin
+            continue
+
+    record = proto.read_waveform_record(key4)   # SUB 0C (0xD2 bytes)
+    events.append(decode(record))
+
+    key4 = proto.advance_event()                # SUB 1F (token=0xFE)
+
+return events
+```
+
+---
+
 ## 8. Timestamp Format
 > 🔶 **Updated 2026-02-26** — Year field resolved. Confidence upgraded.