diff --git a/bridges/s3-bridge/s3_bridge.py b/bridges/s3-bridge/s3_bridge.py index 8c69c19..306297c 100644 --- a/bridges/s3-bridge/s3_bridge.py +++ b/bridges/s3-bridge/s3_bridge.py @@ -35,7 +35,7 @@ from typing import Optional import serial -VERSION = "v0.5.0" +VERSION = "v0.5.1" DLE = 0x10 STX = 0x02 diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index c65d41e..678301d 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -40,6 +40,9 @@ | 2026-03-02 | §14 Open Questions | `0x082A` hypothesis refined: 2090 decimal. At 1024 sps, 2 sec record = 2048 samples. Possible that 0x082A = total samples including 0.25s pre-trigger (256 samples) at some adjusted rate. Needs capture with different record time. | | 2026-03-02 | §14 Open Questions | **NEW items added:** Trigger sample width (default=2), Auto Window (1-9 sec), Aux Trigger (enabled/disabled) — all confirmed settings from operator manual not yet mapped in protocol. | | 2026-03-02 | §14 Open Questions | Monitoring LCD Cycle resolved — removed from open questions. | +| 2026-03-03 | §2 Frame Structure | **UPDATED:** Documented BW/S3 framing asymmetry. BW uses bare STX (`0x02`); S3 uses DLE+STX (`0x10 0x02`). ETX initially believed symmetric — see correction below. | +| 2026-03-03 | §2 Frame Structure | **CORRECTED:** ETX is also asymmetric. BW uses bare ETX (`0x03`); S3 uses DLE+ETX (`0x10 0x03`). Confirmed via checksum validation: 91/98 BW frames pass with bare `0x03` as terminator. All `10 03` sequences in `raw_bw.bin` are in-payload data, never followed by `41 02` (next frame start). Full confirmed grammar: BW=`02`...`03`, S3=`10 02`...`10 03`. Both sides stuff literal `0x10` as `10 10`. This is the formally confirmed link-layer grammar. | +| 2026-03-03 | Appendix A | **CORRECTED:** Previous entry stated logger strips DLE from ETX. This was wrong — applied to older logger only. `s3_bridge v0.5.0+` is lossless. Section rewritten to reflect current flat raw dump format. | | 2026-03-02 | Appendix A | **CORRECTED:** Previous entry stated logger strips DLE from ETX. This was wrong — it applied to an older logger version. `s3_bridge v0.5.0` confirmed to preserve raw wire bytes including `0x10 0x03` intact. HxD inspection of new capture confirmed `10 03` present in S3→BW record payloads. | | 2026-03-02 | Appendix A | **UPDATED:** New capture architecture: two flat raw wire dumps per session (`raw_s3.bin`, `raw_bw.bin`), one per direction, no record wrapper. Replaces structured `.bin` format for parser input. | | 2026-03-02 | Appendix A | **PARSER:** Deterministic DLE state machine implemented (`s3_parser.py`). Three states: `IDLE → IN_FRAME → AFTER_DLE`. Replaces heuristic global scanning. Properly handles DLE stuffing (`10 10` → literal `10`). Only complete STX→ETX pairs counted as frames. | @@ -62,38 +65,70 @@ ## 2. Frame Structure > ⚠️ **2026-02-26 — CORRECTED:** Previous version incorrectly identified `0x41` as STX and `0x02`/`0x03` as bare frame delimiters. The protocol uses proper **DLE framing**. See below. +> ⚠️ **2026-03-03 — UPDATED:** Frame start AND end are both asymmetric by direction. See confirmed grammar below. -Every message follows this structure: +### Confirmed Link-Layer Grammar ✅ CONFIRMED — 2026-03-03 +The two sides of the connection use **fully asymmetric framing**. DLE stuffing applies on both sides. + +| Direction | STX (frame start) | ETX (frame end) | Stuffing | Notes | +|---|---|---|---|---| +| S3 → BW (device) | `0x10 0x02` (DLE+STX) | `0x10 0x03` (DLE+ETX) | `0x10` → `0x10 0x10` | Full DLE framing | +| BW → S3 (Blastware) | `0x02` (bare STX) | `0x03` (bare ETX) | `0x10` → `0x10 0x10` | Bare delimiters, DLE stuffing only | + +**Evidence:** +- 91/98 BW frames validate checksum when parsed with bare `0x03` as ETX +- All `10 03` sequences in `raw_bw.bin` are in-payload data — none are followed by `41 02` (next frame start) +- `10 03` appearing in BW payload is always `10 10 03` origin (stuffed DLE + literal `03`) — the S3 device correctly parses this via its own state machine without false ETX detection +- S3 captures consistently terminate with `10 03` confirmed via HxD + +**Practical impact for parsers:** +- Parser on `raw_s3.bin`: trigger on `10 02`, terminate on `10 03` +- Parser on `raw_bw.bin`: trigger on bare `02`, terminate on bare `03` +- Both parsers must handle `10 10` → literal `10` unstuffing +- ETX detection must be state-machine-aware (not raw byte search) to avoid false matches on stuffed sequences + +### Frame Structure by Direction + +**S3 → BW (device responses):** ``` [ACK] [DLE+STX] [PAYLOAD...] [CHECKSUM] [DLE+ETX] 0x41 0x10 0x02 N bytes 1 byte 0x10 0x03 ``` +**BW → S3 (Blastware commands):** +``` +[ACK] [STX] [PAYLOAD...] [CHECKSUM] [ETX] + 0x41 0x02 N bytes 1 byte 0x03 +``` + ### Special Byte Definitions | Token | Raw Bytes | Meaning | Certainty | |---|---|---|---| | ACK | `0x41` (ASCII `'A'`) | Acknowledgment / ready token. Standalone single byte. Sent before every frame by both sides. | ✅ CONFIRMED | -| DLE | `0x10` | Data Link Escape. Prefixes the next byte to give it special meaning. | ✅ CONFIRMED — 2026-02-26 | -| STX | `0x10 0x02` | DLE+STX = Start of frame (two-byte sequence) | ✅ CONFIRMED — 2026-02-26 | -| ETX | `0x10 0x03` | DLE+ETX = End of frame (two-byte sequence) | ✅ CONFIRMED — 2026-02-26 | -| CHECKSUM | 1 byte | 8-bit sum of de-stuffed payload bytes, modulo 256. Sits between payload and DLE+ETX. | ✅ CONFIRMED | +| DLE | `0x10` | Data Link Escape. Used for stuffing on both sides; also prefixes STX/ETX on S3 side only. | ✅ CONFIRMED — 2026-02-26 | +| STX (S3) | `0x10 0x02` | DLE+STX = Start of frame sent by device | ✅ CONFIRMED — 2026-02-26 | +| STX (BW) | `0x02` | Bare STX = Start of frame sent by Blastware | ✅ CONFIRMED — 2026-03-03 | +| ETX (S3) | `0x10 0x03` | DLE+ETX = End of frame sent by device | ✅ CONFIRMED — 2026-02-26 | +| ETX (BW) | `0x03` | Bare ETX = End of frame sent by Blastware | ✅ CONFIRMED — 2026-03-03 | +| CHECKSUM | 1 byte | 8-bit sum of de-stuffed payload bytes, modulo 256. Sits between payload and ETX. | ✅ CONFIRMED | ### DLE Byte Stuffing Rule > ✅ CONFIRMED — 2026-02-26 -Any `0x10` byte appearing **naturally in the payload data** is escaped by doubling it: `0x10` → `0x10 0x10`. This prevents the parser from confusing real data with frame control sequences. +Any `0x10` byte appearing **naturally in the payload data** is escaped by doubling it: `0x10` → `0x10 0x10`. This applies on **both sides** of the connection. - **Transmit:** Replace every `0x10` in payload with `0x10 0x10` - **Receive:** Replace every `0x10 0x10` in the frame body with a single `0x10` -| Sequence on wire | Meaning | -|---|---| -| `0x10 0x02` | Frame START — only valid at beginning | -| `0x10 0x03` | Frame END | -| `0x10 0x10` | Escaped literal `0x10` byte in payload data | -| Any other `0x10 0xXX` | Protocol error / undefined | +| Sequence on wire | S3 context | BW context | +|---|---|---| +| `0x10 0x02` | Frame START | Stuffed `0x10` + payload `0x02` | +| `0x10 0x03` | Frame END | Stuffed `0x10` + payload `0x03` | +| `0x10 0x10` | Escaped literal `0x10` | Escaped literal `0x10` | +| `0x02` | Payload byte | Frame START | +| `0x03` | Payload byte | Frame END | ### Frame Parser Notes @@ -595,14 +630,17 @@ Timestamps are 6-byte sequences appearing in event headers and waveform keys. ## 10. DLE Byte Stuffing > ✅ **CONFIRMED — 2026-02-26** (previously ❓ SPECULATIVE) -This protocol uses standard **DLE (Data Link Escape) byte stuffing**, a classical technique used in protocols like IBM BISYNC dating to the 1970s. +This protocol uses standard **DLE (Data Link Escape) byte stuffing**, a classical technique used in protocols like IBM BISYNC dating to the 1970s. Both sides stuff literal `0x10` bytes as `0x10 0x10`. The framing delimiters differ by direction — see §2. -### Parser State Machine +### Parser State Machine — S3→BW direction (device responses) + +Trigger on DLE+STX, terminate on DLE+ETX. ``` IDLE: receive 0x41 → emit ACK event, stay IDLE receive 0x10 → goto WAIT_STX + receive anything → discard, stay IDLE WAIT_STX: receive 0x02 → frame started, goto IN_FRAME @@ -615,8 +653,27 @@ IN_FRAME: ESCAPE: receive 0x03 → frame complete — validate checksum, process buffer, goto IDLE receive 0x10 → append single 0x10 to buffer, goto IN_FRAME (stuffed literal) - receive 0x02 → error (nested STX), goto IDLE - receive anything → error, goto IDLE + receive anything → append DLE + byte to buffer (recovery), goto IN_FRAME +``` + +### Parser State Machine — BW→S3 direction (Blastware commands) + +Trigger on bare STX, terminate on bare ETX. DLE only appears in stuffing context. + +``` +IDLE: + receive 0x41 → emit ACK event, stay IDLE + receive 0x02 → frame started, goto IN_FRAME + receive anything → discard, stay IDLE + +IN_FRAME: + receive 0x10 → goto ESCAPE + receive 0x03 → frame complete — validate checksum, process buffer, goto IDLE + receive any byte → append to buffer, stay IN_FRAME + +ESCAPE: + receive 0x10 → append single 0x10 to buffer, goto IN_FRAME (stuffed literal) + receive anything → append DLE + byte to buffer (recovery), goto IN_FRAME ``` --- @@ -757,23 +814,25 @@ Build in this order — each step is independently testable: --- ## Appendix A — s3_bridge Capture Format -> ✅ **CONFIRMED — 2026-02-26** +> ⚠️ **This section describes tooling behavior, not protocol semantics.** +> **2026-03-03 — CORRECTED:** Previous version of this section incorrectly stated that `s3_bridge` strips DLE from ETX. This applied to an older logger version only. `s3_bridge v0.5.0+` is confirmed lossless. See Appendix C for full validation details. -> ⚠️ **This behavior is not part of the Instantel protocol. It is an artifact of the bridge logger implementation.** +### Current Format (v0.5.0+) ✅ CONFIRMED — 2026-03-03 -The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger makes one modification: +As of `s3_bridge v0.5.0`, captures are produced as **two flat raw wire dump files per session**: -| Wire sequence | In .bin file | Notes | -|---|---|---| -| `0x10 0x03` (DLE+ETX) | `0x03` | DLE stripped from end-of-frame marker | -| All other bytes | Unchanged | ACK, DLE+STX, stuffed payload, checksum all preserved verbatim | +| File | Contents | +|---|---| +| `raw_s3.bin` | All bytes transmitted by S3 (device → Blastware), in order | +| `raw_bw.bin` | All bytes transmitted by BW (Blastware → device), in order | -**Practical impact for parsing `.bin` files:** -- Frame end: scan for bare `0x03` (not `0x10 0x03`) -- Checksum: the byte immediately before the bare `0x03` is the checksum -- Everything else (ACK detection, DLE+STX, payload de-stuffing) works as documented in §10 +Every byte on the wire is written verbatim — no modification, no record headers, no timestamps. `0x10 0x03` (DLE+ETX) is preserved intact. -> ⚠️ This means checksums cannot be verified on frames where the stuffed payload ends in `0x10` — that trailing `0x10` would normally be the DLE prefix of ETX, but the logger strips it, making the frame boundary ambiguous in that edge case. In practice this has not been observed in captured data. +**Practical impact for parsing:** +- `raw_s3.bin`: trigger on `0x10 0x02`, terminate on `0x10 0x03` (DLE+ETX) +- `raw_bw.bin`: trigger on bare `0x02`, terminate on bare `0x03` +- Both: handle `0x10 0x10` → literal `0x10` unstuffing +- ETX detection must be state-machine-aware on both sides to avoid false matches on stuffed sequences --- @@ -846,9 +905,9 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger The earlier stripping behavior applied to a previous logger version. v0.5.0 is confirmed lossless with respect to wire bytes. **Confirmed wire framing:** -- Frame start: `0x10 0x02` (DLE STX) ✅ -- Frame end: `0x10 0x03` (DLE ETX) ✅ -- DLE stuffing: `0x10 0x10` in payload = literal `0x10` ✅ +- S3→BW: frame start `0x10 0x02`, frame end `0x10 0x03` ✅ +- BW→S3: frame start `0x02`, frame end `0x03` ✅ +- Both sides: DLE stuffing `0x10 0x10` = literal `0x10` ✅ ### C.2 Capture Architecture (Current) @@ -875,6 +934,8 @@ STATE_AFTER_DLE — last byte was 0x10, awaiting qualifier **Transitions:** +**S3→BW parser states:** + | Current State | Byte | Action | Next State | |---|---|---|---| | IDLE | `10 02` | Begin new frame | IN_FRAME | @@ -883,7 +944,19 @@ STATE_AFTER_DLE — last byte was 0x10, awaiting qualifier | IN_FRAME | `10` | — | AFTER_DLE | | AFTER_DLE | `10` | Append literal `0x10` | IN_FRAME | | AFTER_DLE | `03` | Frame complete, emit | IDLE | -| AFTER_DLE | other | Treat as payload (recovery) | IN_FRAME | +| AFTER_DLE | other | Append DLE + byte (recovery) | IN_FRAME | + +**BW→S3 parser states:** + +| Current State | Byte | Action | Next State | +|---|---|---|---| +| IDLE | `02` | Begin new frame | IN_FRAME | +| IDLE | any | Discard | IDLE | +| IN_FRAME | `03` | Frame complete, emit | IDLE | +| IN_FRAME | `10` | — | AFTER_DLE | +| IN_FRAME | other | Append to payload | IN_FRAME | +| AFTER_DLE | `10` | Append literal `0x10` | IN_FRAME | +| AFTER_DLE | other | Append DLE + byte (recovery) | IN_FRAME | **Properties:** - Does not scan globally for `10 02` @@ -894,9 +967,9 @@ STATE_AFTER_DLE — last byte was 0x10, awaiting qualifier ### C.4 Observed Traffic (Validation Captures) **`raw_bw.bin`** (Blastware → S3): -- 7 complete frames via state machine -- Mostly small command/control frames, several zero-length payloads -- Bare `0x02` used as STX (asymmetric — BW does not use DLE STX) +- 98 complete frames via state machine (bare STX + bare ETX mode) +- 91/98 checksums validate; 7 failures are large frames containing in-payload `10 03` sequences that a naive scanner misreads as ETX +- Bare `0x02` STX and bare `0x03` ETX confirmed; DLE used for stuffing only - Contains project metadata strings: `"Standard Recording Setup.set"`, `"Claude test2"`, `"Location #1 - Brians House"` **`raw_s3.bin`** (S3 → Blastware):