feat(micromate): framing layer -- and three spec rules the bytes refuted
Step 1 of docs/micromate_client_spec.md: micromate/framing.py plus 31 offline tests. Every rule was checked against the captures BEFORE being written, which is the only reason this commit is not a bug. Three things the spec asserted are wrong, all of which fail silently: 1. Requests are NOT plain Series III frames. A Micromate escapes four byte values -- 0x02 0x03 0x04 0x10 -- where Series III escapes one. minimateplus.build_bw_frame reproduces 161 of Thor's 218 captured read frames; build_request() reproduces 218/218. The 57 it missed include EVERY 0x5A download (offset 0x0400 puts a literal 0x04 in offset_hi) and the scheduler enable. An unescaped 0x03/0x04 terminates the frame, so the unit never answers -- indistinguishable from a dead unit, and event download would have hit it on the first request ever sent. 2. The checksum is plain SUM8 of the destuffed payload, not the DLE-aware variant. 251/251 both directions. The DLE-aware form is correct paired with Series III destuffing, which leaves an escaped byte as two bytes; after uniform destuffing it subtracts the correction twice and disagrees with the wire on 55 of 251 responses. scratch/mm_frame_parse.py shipped with exactly that pairing and looked clean only because it accepts either rule -- so it labelled those 55 "SUM8" and never flagged one bad. "Zero bad checksums" was true and carried no information. A tool that tries N candidate rules cannot falsify any of them. Fixed to validate against SUM8 alone. 3. SUB 0x5A is a 1024-byte chunk loop, not one request per event. Thor's form, verified on all six bench events (4,076 -> 13,424 B): chunks = ceil(size/1024), offset = min(1024, size - 1024*i) as a byte count, params[2:4] = the byte offset, response data = offset + 11. sum(offsets) == size exactly, every time. This does not retract the earlier single-request observation -- that used offset_hi = 0x10, which in Series III is the bulk-stream marker, so it is plausibly a distinct streaming mode returning several frames. Those captures never landed in the repo, so it cannot be re-derived. Implement Thor's form; the other is worth one bench test. Also: a 0x10 inside request params needs no special handling (settled -- Thor sends it, the wire doubles it), so the planned NotImplementedError guard is gone. declared_length -> probe_length, because it is only meaningful in a probe reply and Thor never probes. Synthesised test frames are marked and each says what it stands in for. The flags=0x03 case is the only coverage of the Thor firmware line -- it wants a real 11.0BD capture next time UM20147 is on a bench. No writes. Read-path framing only; nothing here can originate a command. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
+114
-44
@@ -57,32 +57,37 @@ branches until neither case is readable.
|
||||
|
||||
---
|
||||
|
||||
## `micromate/framing.py`
|
||||
## `micromate/framing.py` — ✅ BUILT 2026-09-27
|
||||
|
||||
Implemented, with `tests/test_micromate_framing.py` (31 tests, all passing).
|
||||
**Two things in this section as originally drafted were wrong**, and both were
|
||||
caught by measuring against the captures before writing code rather than after.
|
||||
They are left in place below, struck through, because both are the kind of
|
||||
mistake that would be made again.
|
||||
|
||||
### Requests
|
||||
|
||||
Series IV accepts Series III request frames unmodified. The simplest correct
|
||||
implementation re-exports the builder rather than duplicating it:
|
||||
~~Series IV accepts Series III request frames unmodified. The simplest correct
|
||||
implementation re-exports the builder rather than duplicating it:~~
|
||||
|
||||
```python
|
||||
from minimateplus.framing import build_bw_frame # requests are identical
|
||||
from minimateplus.framing import build_bw_frame # ✗ WRONG — 161/218
|
||||
```
|
||||
|
||||
⚠ One open question, flagged in the protocol reference and **not** settled:
|
||||
whether `0x10` bytes inside request *params* need stuffing. No probe we sent
|
||||
carried one. Until it is settled, assert on it rather than guessing:
|
||||
⚠ **`build_bw_frame` reproduces only 161 of Thor's 218 captured read frames.**
|
||||
The payload layout is identical; the stuffing is not. A Micromate escapes
|
||||
**four** byte values — `0x02`, `0x03`, `0x04`, `0x10` — where Series III
|
||||
escapes one. The frames this breaks are **every `SUB 0x5A` download**
|
||||
(`offset = 0x0400` → a literal `0x04` in `offset_hi`) and the scheduler enable.
|
||||
An unescaped `0x03`/`0x04` terminates the frame early, so the unit just does not
|
||||
answer.
|
||||
|
||||
```python
|
||||
def build_request(sub: int, offset: int = 0, params: bytes = bytes(10)) -> bytes:
|
||||
if 0x10 in params:
|
||||
raise NotImplementedError(
|
||||
"params containing 0x10 — stuffing rule unconfirmed; see "
|
||||
"micromate_protocol_reference.md, 'Untested and unsafe-until-agreed'"
|
||||
)
|
||||
return build_bw_frame(sub, offset, params)
|
||||
```
|
||||
`build_request()` with the correct escape set reproduces **218/218**.
|
||||
|
||||
That turns an unknown into a loud failure instead of a corrupt frame.
|
||||
✅ **Settled: `0x10` inside params needs no special handling.** The planned
|
||||
`NotImplementedError` guard is unnecessary — Thor sends
|
||||
`params = 00 00 10 00 …` on `SUB 0x5A` in five captured frames and the wire
|
||||
carries an ordinary doubled `10 10`. One rule covers the whole payload.
|
||||
|
||||
### Responses — where Series III's parser cannot follow
|
||||
|
||||
@@ -104,11 +109,22 @@ validate.
|
||||
|
||||
```python
|
||||
def checksum(payload: bytes) -> int:
|
||||
return sum(b for b in payload if b != 0x10) & 0xFF
|
||||
return sum(payload) & 0xFF # payload already de-stuffed
|
||||
```
|
||||
|
||||
The DLE-aware variant, same as Series III's `5A` and write frames — not the plain
|
||||
SUM8 of ordinary Series III reads.
|
||||
~~The DLE-aware variant, same as Series III's `5A` and write frames.~~
|
||||
|
||||
⚠ **Plain SUM8, not the DLE-aware variant** — 251/251 both directions. The
|
||||
DLE-aware form is the right answer paired with *Series III* de-stuffing, which
|
||||
leaves an escaped byte in the payload as two bytes. De-stuffing `10 XX → XX`
|
||||
already removes the `0x10`, so excluding it again subtracts the correction
|
||||
twice, and the result disagrees with the wire on **55 of 251** captured
|
||||
responses — every frame holding a literal `0x10`.
|
||||
|
||||
`scratch/mm_frame_parse.py` shipped with exactly that pairing. It looked clean
|
||||
only because it accepts a frame matching *either* rule, so it labelled those 55
|
||||
`SUM8` and never flagged one bad. "Zero bad checksums" was true and carried no
|
||||
information. Fixed there too.
|
||||
|
||||
### ⚠ The SUB byte can be escaped
|
||||
|
||||
@@ -129,16 +145,27 @@ class MicromateFrame:
|
||||
data: bytes # payload[5:], checksum stripped
|
||||
checksum_valid: bool
|
||||
|
||||
@property
|
||||
def request_sub(self) -> int: # 0xFF - sub
|
||||
@property
|
||||
def page_key(self) -> int: # uint16 BE at payload[3:5]
|
||||
@property
|
||||
def firmware_line(self) -> str: # "blastware" | "thor" | "unknown"
|
||||
@property
|
||||
def declared_length(self) -> int: # uint16 BE at data[3:5] (= payload[8:10])
|
||||
def probe_length(self) -> int | None: # uint16 BE at data[3:5] (= payload[8:10])
|
||||
```
|
||||
|
||||
⚠ **`declared_length` is a uint16 BE.** Read as a single byte it under-reads
|
||||
⚠ **`probe_length` is a uint16 BE.** Read as a single byte it under-reads
|
||||
`SUB 0x1A` by 47x — 44 against a true 2092. This is the single most expensive
|
||||
mistake available in this protocol and it has already been made once.
|
||||
|
||||
Renamed from `declared_length`, because it is **only meaningful in the reply to
|
||||
an `offset = 0` probe** — and Thor never probes. Across all 251 captured
|
||||
responses the field reads 0 or a page count, never a length, precisely because
|
||||
that session is single-step reads throughout. `page_key` is the field that
|
||||
carries meaning there. The only genuine probe reply we hold is the POLL one
|
||||
preserved in `scratch/fake_unit.py`.
|
||||
|
||||
`MicromateFrameParser` mirrors `S3FrameParser`: `feed(bytes) -> list[frame]`,
|
||||
accumulates in `.frames`, `reset()`, and keeps the `bytes_fed` counter (it is
|
||||
what distinguishes "no bytes at all" from "bytes but no complete frame" on a
|
||||
@@ -183,21 +210,41 @@ unit yields a battery voltage of **577.92 V**.
|
||||
⚠ **Test the monitoring flag for non-zero**, never against a constant. It has
|
||||
read both `0x0E` and `0x0C` while monitoring.
|
||||
|
||||
### `0x5A` — simpler than Series III, deliberately
|
||||
### `0x5A` — a bounded chunk loop, and much simpler than Series III
|
||||
|
||||
No arming ritual, no chunk loop, no `STRT` end-offset parsing, no `TERM` frame.
|
||||
One request returns the whole event:
|
||||
⚠ **Corrected 2026-09-27.** This section said "no chunk loop — one request
|
||||
returns the whole event", with `offset_word = 0x1000 + 2 * ceil(size / 512)`.
|
||||
That describes our own 2026-09-23 probes, which set `offset_hi = 0x10`. **Thor
|
||||
chunks**, and Thor's form is the one verified from bytes on disk:
|
||||
|
||||
```python
|
||||
offset_word = 0x1000 + 2 * ceil(size / 512) # size from the chain walk
|
||||
n = ceil(size / 1024) # size from the chain walk
|
||||
for i in range(n):
|
||||
offset = min(1024, size - 1024 * i) # a BYTE COUNT
|
||||
params = key4 + bytes(6) if i == 0 else bytes(2) + pack(">H", 1024*i) + bytes(6)
|
||||
file_bytes += response.data[11:] # response data is exactly offset + 11
|
||||
```
|
||||
|
||||
The payload **is** the `.IDFW` file, byte for byte — so it feeds
|
||||
Verified on all six bench events (4,076 → 13,424 B): `sum(offsets) == size`
|
||||
exactly, with the predicted chunk count and final offset every time.
|
||||
|
||||
Still no arming ritual for `0x5A` itself, no `STRT` end-offset parsing and no
|
||||
`TERM` frame — the simplification the original claim celebrated is real, it just
|
||||
is not single-shot. (`SUB 0x93` arms the *chain walk*, before `1E`/`1F`, not
|
||||
the download.)
|
||||
|
||||
The concatenated payload **is** the `.IDFW` file, byte for byte — so it feeds
|
||||
`micromate.idf_file.read_idf_file()` and `/db/import/idf_file` unchanged.
|
||||
|
||||
⚠ Do not port the Series III `5A` walk. Its address arithmetic caused a 5x
|
||||
over-read and a `> 64 KB` page-boundary bug that is *still open* on the Series
|
||||
III side. None of that applies here.
|
||||
III side. None of that applies here: the chunk index is a byte offset into the
|
||||
file, bounded by a size the device told us, and it cannot run past the event.
|
||||
|
||||
⚠ **`assert sum(len(chunk) - 11 for chunk in chunks) == size`.** A silently
|
||||
short event is the failure mode this project has been bitten by repeatedly on
|
||||
the Series III side, and here the check is free because the size is known up
|
||||
front.
|
||||
|
||||
---
|
||||
|
||||
@@ -242,22 +289,34 @@ tests/test_micromate_framing.py
|
||||
|
||||
⚠ `bridges/captures/` and `tests/fixtures/` are both gitignored, so tests must
|
||||
not depend on files being present. **Embed the frames as hex constants** — they
|
||||
are 16–68 bytes each, and a handful covers every case:
|
||||
are 19–138 bytes each. ✅ Done; what actually landed:
|
||||
|
||||
| case | why |
|
||||
|---|---|
|
||||
| POLL probe reply, 19 B | shortest valid frame |
|
||||
| POLL data reply, 68 B | contains a literal `0x10` — only the DLE-aware checksum matches |
|
||||
| `0x1A` response, 2108 B | exercises `declared_length` as a true uint16 (`0x082C`) |
|
||||
| a Thor-line reply, `flags = 0x03` | `0x03` is ETX; proves destuffing before framing |
|
||||
| a frame whose SUB is `0x02` | arrives as `10 02`; proves destuff-then-index |
|
||||
| a truncated frame | parser must return nothing, not a bad frame |
|
||||
| a corrupted checksum | `checksum_valid == False`, frame still returned |
|
||||
| case | source | why |
|
||||
|---|---|---|
|
||||
| POLL probe reply, 19 B | captured (via `fake_unit.py`) | shortest valid frame; the only real probe reply we hold |
|
||||
| `0x5A` chunk, 138 B | captured | holds literal `0x10` **and** literal `0x41` — the checksum case, and proves ACK is not escaped |
|
||||
| `0x49` state reply, 25 B | captured | a literal `0x02`, escaped |
|
||||
| `0x48` file reply, 24 B | captured | escaped `0x04` in `data[0]` — one byte late without destuffing |
|
||||
| 8 Thor request frames | captured | byte-for-byte against `build_request()`, incl. both `0x5A` forms and the scheduler enable |
|
||||
| `probe_length = 0x082C` | **synthesised** | no probe reply for `0x1A` exists on disk — the 9-24-26 session never probes |
|
||||
| Thor-line reply, `flags = 0x03` | **synthesised** | no 11.0BD capture is in the repo; built by flipping one byte of the real POLL reply |
|
||||
| escaped checksum byte | **synthesised** | the shortest real one is 1,070 B, too long to embed for one assertion |
|
||||
| truncated / corrupt / split-across-feeds | derived | parser must return nothing, flag rather than swallow, and survive any split point |
|
||||
|
||||
Then a round-trip assertion: feed a whole captured session through the parser and
|
||||
assert the frame count and every SUB, against `scratch/mm_frame_parse.py`'s
|
||||
output — which is already known good, having parsed 24, 38 and 40-frame sessions
|
||||
with zero bad checksums.
|
||||
⚠ Synthesised frames are marked `SYNTH_`-style in the test and each says what it
|
||||
stands in for and why no capture was available. Do not let that set grow
|
||||
quietly: the `flags = 0x03` case in particular is the **only** coverage of half
|
||||
the fleet, and it deserves a real 11.0BD capture the next time UM20147 is on a
|
||||
bench.
|
||||
|
||||
Two corpus-backed tests run when the captures happen to be on the box and skip
|
||||
cleanly otherwise: **251 response frames parse with zero bad checksums**, and
|
||||
**`build_request()` reproduces 218/218 read frames**. The second is the test
|
||||
that would have caught the escape-set error, so it is worth the skip marker.
|
||||
|
||||
⚠ Do **not** assert against `scratch/mm_frame_parse.py`'s output as the original
|
||||
plan proposed. That script accepts either checksum rule and is wrong about
|
||||
which one is right — using it as an oracle would have pinned the bug.
|
||||
|
||||
**Live, second:** against the bench unit on mint-mac via `mm_link.py`.
|
||||
`connect()`, `list_setups()` (should return the 23 known names), `list_events()`,
|
||||
@@ -268,7 +327,7 @@ then `download_event()` and assert the bytes decode and match a
|
||||
|
||||
## Order of work
|
||||
|
||||
1. `framing.py` + its tests — offline, verifiable immediately
|
||||
1. ✅ `framing.py` + its tests — **done 2026-09-27**, 31 tests, offline
|
||||
2. `protocol.py` — reads only, one method per row of the table above
|
||||
3. `client.py` — `connect()`, `get_state()`, `list_setups()`
|
||||
4. the event chain and `download_event()`
|
||||
@@ -276,11 +335,22 @@ then `download_event()` and assert the bytes decode and match a
|
||||
|
||||
Steps 1–2 need no hardware at all.
|
||||
|
||||
**Worth carrying forward from step 1:** every rule got checked against the
|
||||
captures *before* being written, and two of the three the spec asserted turned
|
||||
out wrong — the escape set (26% of frames) and the checksum (22%). Both fail
|
||||
silently. The captures are on disk and a re-stuff-and-compare loop takes about
|
||||
two minutes per rule, so do that for `protocol.py`'s per-command offsets too
|
||||
rather than trusting the table above.
|
||||
|
||||
---
|
||||
|
||||
## Open questions to settle while implementing
|
||||
|
||||
- **Request param stuffing** — raise `NotImplementedError` rather than guess.
|
||||
- ~~**Request param stuffing**~~ — ✅ settled 2026-09-27; no special handling.
|
||||
- **Is the single-request `0x5A` form real?** Our 2026-09-23 probes set
|
||||
`offset_hi = 0x10` and appeared to get a whole 11 KB event back, where Thor
|
||||
chunks at 1024 B. Plausibly a distinct streaming mode that returns several
|
||||
frames. One bench test settles it; implement Thor's form regardless.
|
||||
- **Is THOR's preamble required?** Try one command cold and find out; it is a
|
||||
two-minute test with the bench unit and it removes a ritual if unnecessary.
|
||||
- **`Event` model fit** — Series IV carries fields Series III lacks (setup file
|
||||
|
||||
@@ -150,12 +150,38 @@ Every frame in this session was produced by `build_bw_frame(sub, offset)` with
|
||||
no Series IV changes, and the device accepted all of them:
|
||||
|
||||
```
|
||||
[ACK 0x41] [STX 0x02] [10 10] [flags 00] [SUB] [00] [00] [offset] [params×10] [chk] [ETX 0x03]
|
||||
[ACK 0x41] [STX 0x02] [10 10] [flags 00] [SUB] [00] [offset_hi] [offset_lo] [params×10] [chk] [ETX 0x03]
|
||||
```
|
||||
|
||||
⚠ Only the doubled `BW_CMD` (`10 10`) form has been exercised. Whether other
|
||||
literal `0x10` bytes inside params require stuffing is **untested** — none of
|
||||
the probes sent carried one.
|
||||
The *payload layout* is identical to Series III. The *stuffing* is not.
|
||||
|
||||
> #### ⚠ Correction, 2026-09-27 — `build_bw_frame()` is NOT sufficient
|
||||
>
|
||||
> This section used to say requests were Series III frames "unmodified", and
|
||||
> the client spec accordingly planned to re-export
|
||||
> `minimateplus.framing.build_bw_frame`. Measured against Thor's own frames,
|
||||
> that builder reproduces **161 of 218** captured read frames. It escapes only
|
||||
> `0x10`; a Micromate escapes four bytes (see *The escape set* below).
|
||||
>
|
||||
> The 57 it gets wrong are not obscure:
|
||||
>
|
||||
> | frame | why it breaks |
|
||||
> |---|---|
|
||||
> | **every `SUB 0x5A` download** | `offset = 0x0400` puts a literal `0x04` in `offset_hi`, which must go out as `10 04` |
|
||||
> | `SUB 0x47` scheduler enable | `params[7] = 0x03` → `10 03` |
|
||||
>
|
||||
> An unescaped `0x03` or `0x04` *is* a frame terminator, so the unit sees a
|
||||
> short frame and does not answer — indistinguishable from a dead unit, and
|
||||
> event download would have hit it on the first request ever sent.
|
||||
>
|
||||
> `micromate/framing.py:build_request()` reproduces **218/218**. Pinned per
|
||||
> frame in `tests/test_micromate_framing.py`.
|
||||
|
||||
✅ **Settled 2026-09-27: a `0x10` inside params needs no special handling.**
|
||||
Previously flagged untested. Thor sends `SUB 0x5A` with
|
||||
`params = 00 00 10 00 …` in five captured frames and the wire carries the
|
||||
ordinary doubled `10 10`. There is no Series-III-style partial-stuffing
|
||||
carve-out here — one rule covers the whole payload including the checksum.
|
||||
|
||||
### Responses — Series III *minus the DLE prefix*
|
||||
|
||||
@@ -181,18 +207,75 @@ mismatch.
|
||||
[5+] data
|
||||
```
|
||||
|
||||
### Checksum — the DLE-aware variant
|
||||
### Checksum — plain SUM8 of the de-stuffed payload
|
||||
|
||||
```python
|
||||
chk = sum(b for b in payload if b != 0x10) & 0xFF
|
||||
chk = sum(payload) & 0xFF # payload already de-stuffed
|
||||
```
|
||||
|
||||
Confirmed on every frame captured. The `POLL` probe response contains no
|
||||
`0x10` and so cannot distinguish plain SUM8 from the DLE-aware form; the
|
||||
`POLL` **data** response contains a `0x10` at payload offset 42, and only the
|
||||
DLE-aware rule matches there. This is the same checksum Series III uses for
|
||||
its `5A` bulk-stream and write frames — not the plain SUM8 of ordinary
|
||||
Series III reads.
|
||||
**251/251 responses and 251/251 requests**, across every capture in
|
||||
`bridges/captures/9-24-26 - micromate2/`.
|
||||
|
||||
> #### ⚠ Correction, 2026-09-27 — "the DLE-aware variant"
|
||||
>
|
||||
> This section previously specified
|
||||
> `chk = sum(b for b in payload if b != 0x10) & 0xFF`, and that is wrong **as
|
||||
> paired with the uniform de-stuffing rule below.**
|
||||
>
|
||||
> The earlier claim was not a misreading; it was a correct rule attached to the
|
||||
> wrong convention. The DLE-aware form belongs with *Series III* de-stuffing,
|
||||
> which leaves an escaped byte in the payload as **two** bytes — there,
|
||||
> skipping the `0x10` is the necessary correction. De-stuffing `10 XX → XX`
|
||||
> already removes it, so excluding `0x10` as well **subtracts the correction
|
||||
> twice**.
|
||||
>
|
||||
> Cost of the pairing, measured: it disagrees with the wire on **55 of 251**
|
||||
> response frames — every frame whose payload holds a literal `0x10`. A clean
|
||||
> example, from the `0x5A` chunk at offset `0x0070` of
|
||||
> `raw_s3_…_Download_events_then_delete_1_event.bin`: the wire says `0xC1`,
|
||||
> plain SUM8 says `0xC1`, the DLE-aware form says `0x91`.
|
||||
>
|
||||
> **Why nobody noticed:** `scratch/mm_frame_parse.py` accepts a frame matching
|
||||
> *either* rule and labels which one hit. It reported those 55 frames as
|
||||
> `SUM8` and never flagged one bad, so "zero bad checksums across 24, 38 and
|
||||
> 40-frame sessions" was true and told us nothing about which rule was right.
|
||||
> A tool that tries every candidate cannot falsify any of them — if it is going
|
||||
> to stay permissive, it has to *report the split*, not just the pass.
|
||||
>
|
||||
> Pinned by `tests/test_micromate_framing.py`.
|
||||
|
||||
### The escape set — exactly four bytes
|
||||
|
||||
A Micromate escapes `0x02`, `0x03`, `0x04` and `0x10`, each prefixed with a
|
||||
`DLE`, **and nothing else** — in both directions.
|
||||
|
||||
Established by re-stuffing every captured frame and comparing to the wire:
|
||||
|
||||
| candidate escape set | responses reproduced | requests reproduced |
|
||||
|---|---|---|
|
||||
| `{0x10}` — the Series III rule | 130/251 | 177/251 |
|
||||
| `{0x10, 0x03}` | 135/251 | 184/251 |
|
||||
| `{0x10, 0x02, 0x03}` | 178/251 | 192/251 |
|
||||
| **`{0x10, 0x02, 0x03, 0x04}`** | **251/251** | **251/251** |
|
||||
| `{0x10, 0x02, 0x03, 0x04, 0x41}` | 196/251 | — |
|
||||
|
||||
Corroborated independently by the byte that follows a wire `DLE`: across all
|
||||
251 responses it is only ever `0x02` (933×), `0x03` (510×), `0x04` (655×) or
|
||||
`0x10` (1179×). Nothing else ever appears there.
|
||||
|
||||
The last row matters because ACK looks like it ought to be escaped and is not —
|
||||
a literal `0x41` in the data goes out bare.
|
||||
|
||||
**The checksum byte is escaped too.** Three captured responses have a checksum
|
||||
of `0x02`/`0x03`/`0x04` and all three arrive as `10 XX` immediately before the
|
||||
terminating `ETX`. No captured *request* happened to land on one, so Thor's
|
||||
behaviour there is unobserved — but a device and its host share one framing
|
||||
routine, and the alternative is a frame the far end truncates, so escape it.
|
||||
|
||||
This supersedes nothing: the de-stuffing rule `10 XX → XX` stays exactly as
|
||||
documented. Knowing only four values are ever escaped is what makes that
|
||||
uniform rule *exact* rather than merely convenient — and it is what an
|
||||
**encoder** needs, which the write path will.
|
||||
|
||||
### The probe response carries the data length
|
||||
|
||||
@@ -573,8 +656,63 @@ Series III ignores a `5A` probe unless preceded by
|
||||
answers a **bare `5A` request** with nothing before it. That whole ritual is
|
||||
gone.
|
||||
|
||||
### ⚠ Correction, 2026-09-27 — THOR uses a 1024-byte chunk loop
|
||||
|
||||
The section below ("One request returns the entire event; there is no chunk
|
||||
loop") describes **our own probes**, and it is not how THOR downloads an event.
|
||||
Read from the wire, THOR's sequence per event is:
|
||||
|
||||
```
|
||||
0x93 (arm) → 0x1E / 0x1F → key + size
|
||||
0x0C → the 210-byte record
|
||||
0x5A × n → n = ceil(size / 1024)
|
||||
```
|
||||
|
||||
and the `0x5A` frames are a plain bounded chunk walk:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| chunk `i` offset | `min(1024, size − 1024·i)` — a **byte count** |
|
||||
| chunk 0 params | `[key4][6 × 0x00]` — the key means "from the start" |
|
||||
| chunk `i>0` params | `[00 00][uint16 BE of 1024·i][6 × 0x00]` |
|
||||
| response data | exactly `offset + 11` bytes; the file bytes are `data[11:]` |
|
||||
| response `page_key` | `offset // 256` — a page count, not an address |
|
||||
|
||||
**Verified on all six bench events**, sizes 4,076 → 13,424 bytes:
|
||||
`sum(offsets) == size` **exactly** in every case, with the predicted chunk
|
||||
count and predicted final offset. No `STRT` parsing, no `TERM` frame, no
|
||||
over-read — that part of the original claim holds, and it is still far simpler
|
||||
than the Series III walk.
|
||||
|
||||
```
|
||||
key size(1E) chunks sum(offsets) last offset
|
||||
055d4a81 4076 4 4076 0x03ec
|
||||
055d4a82 11032 11 11032 0x0318
|
||||
055d4a83 11502 12 11502 0x00ee
|
||||
055d4a84 13424 14 13424 0x0070
|
||||
055d4a85 8746 9 8746 0x022a
|
||||
055d4a86 6092 6 6092 0x03cc
|
||||
```
|
||||
|
||||
**These are probably two different modes, not a contradiction.** THOR's
|
||||
`offset_hi` is the chunk length (`0x04`, `0x03`, `0x00` …). Our single-request
|
||||
probes set `offset_hi = 0x10` — which in Series III is precisely the
|
||||
bulk-stream marker `build_5a_frame()` writes raw. So `0x10XX` plausibly means
|
||||
"stream until done" and returns **several** frames, which the parser of the day
|
||||
concatenated into the 11,049 bytes recorded below. That reconciles both
|
||||
observations, but it is a hypothesis: those 2026-09-23 captures never landed in
|
||||
the repo, so it cannot be re-derived from bytes on disk.
|
||||
|
||||
**Implement THOR's chunked form.** It is verified byte-exact across six events
|
||||
and five distinct sizes, and it is what the firmware runs every day. The
|
||||
single-request form is worth one bench test as an optimisation — `offset_hi =
|
||||
0x10` and count the frames — but not worth depending on first.
|
||||
|
||||
### The offset word is a LENGTH, not a position
|
||||
|
||||
⚠ Superseded as the implementation path by the correction above; the *reading*
|
||||
of the field is right and is what makes THOR's chunk walk make sense.
|
||||
|
||||
This is the key divergence. Series III walks chunks by absolute flash
|
||||
address, stepping `0x0200` per request. On the Micromate the offset word
|
||||
requests *how much to send*:
|
||||
@@ -591,10 +729,11 @@ offset_word = 0x1000 + 2 × pages pages = ceil(event_size / 512)
|
||||
| `0x102C` | 22 | **11,033 — the whole event** |
|
||||
|
||||
`event_size` comes from the chain walk (the 4 bytes after the key in
|
||||
`1E`/`1F`). **One request returns the entire event**; there is no chunk loop,
|
||||
no `STRT` end-offset parsing, and no `TERM` frame. Over-requesting is safe —
|
||||
`0x1030` (24 pages) returned exactly the same bytes as `0x102C`, so the device
|
||||
caps at the real size.
|
||||
`1E`/`1F`). **One request returns the entire event** ⚠ *— true of our
|
||||
`offset_hi = 0x10` probes; see the 2026-09-27 correction above. THOR chunks* —
|
||||
there is no `STRT` end-offset parsing and no `TERM` frame in either form.
|
||||
Over-requesting is safe — `0x1030` (24 pages) returned exactly the same bytes
|
||||
as `0x102C`, so the device caps at the real size.
|
||||
|
||||
Params are the Series III *probe* form: `[0x00][key4][6 × 0x00]`.
|
||||
|
||||
@@ -636,10 +775,11 @@ per-sample-exact applies.
|
||||
### What a full read now looks like
|
||||
|
||||
```
|
||||
0x93 → arm (THOR sends this before every 1E/1F)
|
||||
1E → first key + size
|
||||
0C(key) → project/client/operator, timestamp, peaks
|
||||
5A(key, 0x1000+2×ceil(size/512)) → the whole .IDFW
|
||||
1F → next key + size (until null sentinel)
|
||||
5A × ceil(size/1024) → the .IDFW, 1024 bytes at a time
|
||||
0x93 → 1F → next key + size (until null sentinel)
|
||||
```
|
||||
|
||||
## Setups are FILES, not a config block
|
||||
|
||||
@@ -0,0 +1,304 @@
|
||||
"""
|
||||
framing.py — frame codec for the Instantel Micromate (Series IV) wire protocol.
|
||||
|
||||
A Micromate answers Series III *command* frames, so the request side looks
|
||||
familiar. The framing underneath is not the same, and the differences are all
|
||||
of the kind that produce a silently-ignored frame rather than an error:
|
||||
|
||||
Series III response: [DLE 0x10] [STX 0x02] … [chk] [ETX 0x03]
|
||||
Micromate response: [STX 0x02] … [chk] [ETX 0x03]
|
||||
^ no leading DLE
|
||||
|
||||
That missing byte is why `minimateplus.framing.S3FrameParser` returns *nothing*
|
||||
on Micromate traffic — it locates frames by scanning for `DLE STX`, which never
|
||||
occurs. A capture holding 12 acknowledged writes reads as 12 unanswered
|
||||
requests.
|
||||
|
||||
De-stuffed payload layout (both directions):
|
||||
|
||||
request response
|
||||
[0] CMD 0x10 [0] CMD 0x00
|
||||
[1] flags 0x00 [1] flags 0xC5 / 0x03 ← firmware line
|
||||
[2] SUB [2] SUB 0xFF − request_SUB
|
||||
[3] 0x00 [3] PAGE_HI
|
||||
[4] offset_hi [4] PAGE_LO
|
||||
[5] offset_lo [5+] data
|
||||
[6:16] params (10 bytes)
|
||||
|
||||
Everything below was established against the 251 request and 251 response
|
||||
frames in `bridges/captures/9-24-26 - micromate2/` (UM12947, firmware 11.0CB).
|
||||
Where a rule is asserted, the number of frames it was checked on is given — the
|
||||
two rules that look like small details cost 26% and 22% of frames respectively
|
||||
when guessed wrong, so the counts are the point.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
# ── Protocol byte constants ───────────────────────────────────────────────────
|
||||
|
||||
DLE = 0x10 # Data Link Escape
|
||||
STX = 0x02 # Start of text — begins a frame
|
||||
ETX = 0x03 # End of text — ends a frame
|
||||
ACK = 0x41 # Frame-start marker on the request side
|
||||
|
||||
MM_CMD = 0x10 # payload[0] in a request
|
||||
MM_RSP_CMD = 0x00 # payload[0] in a response
|
||||
|
||||
# payload[1] of a response identifies the firmware line it came from.
|
||||
# ⚠ Two units, one of each — a strong hypothesis, not a proven encoding.
|
||||
FLAGS_BLASTWARE = 0xC5 # the 11.0CB line (UM12947)
|
||||
FLAGS_THOR = 0x03 # the 11.0BD line (UM20147)
|
||||
|
||||
# ⚠ THE ESCAPE SET. A Micromate escapes exactly these four byte values,
|
||||
# prefixing each with a DLE — and nothing else. Established by re-stuffing
|
||||
# every captured frame and comparing to the wire: 251/251 responses and 251/251
|
||||
# requests reproduce byte-for-byte with this set, and no other candidate set
|
||||
# reproduces even 200 of either.
|
||||
#
|
||||
# The two near-misses are worth naming, because both look plausible:
|
||||
# * `{0x10}` alone — the Series III rule — reproduces 130/251 responses and
|
||||
# 177/251 requests.
|
||||
# * adding ACK (0x41) reproduces only 196/251 responses: a literal 0x41 in
|
||||
# the data is NOT escaped.
|
||||
_ESCAPED = frozenset({STX, ETX, 0x04, DLE})
|
||||
|
||||
# A response header is 5 bytes; a frame must also carry its checksum.
|
||||
_MIN_PAYLOAD = 5
|
||||
_REQUEST_PAYLOAD_SIZE = 16
|
||||
|
||||
|
||||
# ── Stuffing ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def stuff(data: bytes) -> bytes:
|
||||
"""Escape every byte the Micromate escapes: `XX` → `10 XX` for the four."""
|
||||
out = bytearray()
|
||||
for b in data:
|
||||
if b in _ESCAPED:
|
||||
out.append(DLE)
|
||||
out.append(b)
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def unstuff(data: bytes) -> bytes:
|
||||
"""Reverse `stuff()`: `10 XX` → `XX`, for any XX.
|
||||
|
||||
Uniform, with no inner-frame carve-out — which is a real simplification
|
||||
over Series III, where `DLE+ETX` inside a frame is literal data that must
|
||||
survive de-stuffing. Since only four byte values are ever escaped, taking
|
||||
*any* `10 XX` as `XX` is exact rather than merely convenient.
|
||||
"""
|
||||
out = bytearray()
|
||||
i = 0
|
||||
while i < len(data):
|
||||
if data[i] == DLE and i + 1 < len(data):
|
||||
out.append(data[i + 1])
|
||||
i += 2
|
||||
else:
|
||||
out.append(data[i])
|
||||
i += 1
|
||||
return bytes(out)
|
||||
|
||||
|
||||
# ── Checksum ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def checksum(payload: bytes) -> int:
|
||||
"""SUM8 of the **de-stuffed** payload, mod 256. 251/251 both directions.
|
||||
|
||||
⚠ Do NOT exclude `0x10` bytes from this sum. The DLE-aware checksum that
|
||||
Series III uses for its `5A` and write frames is the right answer to a
|
||||
*different* question: it pairs with Series III de-stuffing, which leaves
|
||||
escaped bytes in the payload as two bytes. De-stuffing uniformly already
|
||||
removes the DLE, so excluding `0x10` as well subtracts the correction
|
||||
twice.
|
||||
|
||||
That combination — uniform de-stuffing *and* an exclusive sum — is what
|
||||
`scratch/mm_frame_parse.py` shipped with. It disagrees with the wire on
|
||||
**55 of 251** captured response frames, all of them frames whose payload
|
||||
holds a literal `0x10`. The script only ever looked correct because it
|
||||
accepts a frame that matches *either* rule, so it reported those 55 as
|
||||
plain SUM8 and never flagged one bad.
|
||||
"""
|
||||
return sum(payload) & 0xFF
|
||||
|
||||
|
||||
# ── Request builder ───────────────────────────────────────────────────────────
|
||||
|
||||
def build_request(sub: int, offset: int = 0, params: bytes = bytes(10)) -> bytes:
|
||||
"""Build a host→unit command frame.
|
||||
|
||||
⚠ Do **not** substitute `minimateplus.framing.build_bw_frame()` here, even
|
||||
though the payload layout is identical. That builder escapes only `0x10`,
|
||||
so it reproduces just **161 of Thor's 218** captured read frames. The 57 it
|
||||
gets wrong are not edge cases:
|
||||
|
||||
* every `SUB 0x5A` bulk download — `offset = 0x0400` puts a literal
|
||||
`0x04` in `offset_hi`, which must go out as `10 04`
|
||||
* `SUB 0x47` (scheduler enable), whose params carry a `0x03`
|
||||
|
||||
An unescaped `0x03` or `0x04` reads as a frame terminator, so the unit sees
|
||||
a truncated frame and simply does not answer. That is indistinguishable
|
||||
from a dead unit, and event download would have hit it on the first try.
|
||||
|
||||
With the correct escape set this builder reproduces **218/218**.
|
||||
|
||||
Args:
|
||||
sub: command SUB byte.
|
||||
offset: uint16 at payload[4:5]. Micromate reads are single-step —
|
||||
Thor asks for `0xFFFF` and gets the whole block — so this is
|
||||
usually `0xFFFF`, not Series III's probe-then-data pair.
|
||||
params: exactly 10 bytes at payload[6:16].
|
||||
|
||||
A `0x10` inside `params` is fine and needs no special handling: Thor sends
|
||||
`SUB 0x5A` with `params = 00 00 10 00 …` and the wire carries `10 10`.
|
||||
(This was the spec's one open question; five captured frames settle it.)
|
||||
"""
|
||||
if len(params) != 10:
|
||||
raise ValueError(f"params must be exactly 10 bytes, got {len(params)}")
|
||||
if not 0 <= offset <= 0xFFFF:
|
||||
raise ValueError(f"offset must fit in uint16, got {offset:#x}")
|
||||
if not 0 <= sub <= 0xFF:
|
||||
raise ValueError(f"sub must be a single byte, got {sub:#x}")
|
||||
|
||||
payload = bytes([MM_CMD, 0x00, sub, 0x00, (offset >> 8) & 0xFF, offset & 0xFF]) + params
|
||||
body = payload + bytes([checksum(payload)])
|
||||
return bytes([ACK, STX]) + stuff(body) + bytes([ETX])
|
||||
|
||||
|
||||
# ── Response frame ────────────────────────────────────────────────────────────
|
||||
|
||||
@dataclass
|
||||
class MicromateFrame:
|
||||
"""A parsed, de-stuffed unit→host response frame."""
|
||||
|
||||
sub: int # response SUB; the request was 0xFF − this
|
||||
flags: int # payload[1] — 0xC5 Blastware line, 0x03 Thor line
|
||||
page_hi: int
|
||||
page_lo: int
|
||||
data: bytes # payload[5:], checksum stripped
|
||||
checksum_valid: bool
|
||||
chk_byte: int = 0 # the checksum byte as received
|
||||
|
||||
@property
|
||||
def request_sub(self) -> int:
|
||||
"""The SUB this is answering. No known exception to `0xFF − SUB`."""
|
||||
return 0xFF - self.sub
|
||||
|
||||
@property
|
||||
def page_key(self) -> int:
|
||||
"""payload[3:5] as a uint16 BE — a page/address on `0x5A` responses."""
|
||||
return (self.page_hi << 8) | self.page_lo
|
||||
|
||||
@property
|
||||
def firmware_line(self) -> str:
|
||||
return {FLAGS_BLASTWARE: "blastware", FLAGS_THOR: "thor"}.get(self.flags, "unknown")
|
||||
|
||||
@property
|
||||
def probe_length(self) -> Optional[int]:
|
||||
"""Data length declared by a **probe** response: uint16 BE at data[3:5].
|
||||
|
||||
⚠ Only meaningful in the reply to an `offset = 0` probe. Series III
|
||||
hardcodes a `DATA_LENGTHS` table; a Micromate will tell you instead,
|
||||
which already caught one divergence (call-home config is `0x7E`, where
|
||||
Series III has `0x7C`).
|
||||
|
||||
⚠ It is a **uint16 BE**, not a byte. Read as `data[3]` alone it is
|
||||
right only while the high byte is zero, and wrong by 47x for
|
||||
`SUB 0x1A`: a true `0x082C` (2092) reads as 44.
|
||||
|
||||
Returns None on a frame too short to hold the field. Note this reads
|
||||
as 0 on the single-step reads Thor actually uses — those are not probes,
|
||||
and `page_key` is the meaningful field there.
|
||||
"""
|
||||
if len(self.data) < 5:
|
||||
return None
|
||||
return (self.data[3] << 8) | self.data[4]
|
||||
|
||||
|
||||
# ── Streaming parser ──────────────────────────────────────────────────────────
|
||||
|
||||
class MicromateFrameParser:
|
||||
"""Incremental parser for unit→host frames. Mirrors `S3FrameParser`.
|
||||
|
||||
Feed bytes with `feed()`; completed frames are returned and also collected
|
||||
in `.frames`.
|
||||
|
||||
IDLE — scanning for a bare STX
|
||||
IN_FRAME — collecting; bare ETX terminates
|
||||
AFTER_DLE — the next byte is literal, whatever it is
|
||||
|
||||
Request frames are rejected rather than parsed: a frame whose `payload[0]`
|
||||
is not `0x00` is dropped, so feeding a bidirectional capture yields only
|
||||
the responses.
|
||||
"""
|
||||
|
||||
_IDLE, _IN_FRAME, _AFTER_DLE = 0, 1, 2
|
||||
|
||||
def __init__(self) -> None:
|
||||
self._state = self._IDLE
|
||||
self._body = bytearray()
|
||||
self.frames: list[MicromateFrame] = []
|
||||
# Distinguishes "no bytes at all" from "bytes but no complete frame" on
|
||||
# a timeout. That distinction earned its keep during the Series III
|
||||
# work and costs one integer here.
|
||||
self.bytes_fed: int = 0
|
||||
|
||||
def reset(self) -> None:
|
||||
self._state = self._IDLE
|
||||
self._body.clear()
|
||||
self.bytes_fed = 0
|
||||
|
||||
def feed(self, data: bytes) -> list[MicromateFrame]:
|
||||
self.bytes_fed += len(data)
|
||||
completed: list[MicromateFrame] = []
|
||||
for b in data:
|
||||
frame = self._step(b)
|
||||
if frame is not None:
|
||||
completed.append(frame)
|
||||
self.frames.append(frame)
|
||||
return completed
|
||||
|
||||
def _step(self, b: int) -> Optional[MicromateFrame]:
|
||||
if self._state == self._IDLE:
|
||||
if b == STX:
|
||||
self._body.clear()
|
||||
self._state = self._IN_FRAME
|
||||
# Boot strings, modem RING/CONNECT chatter and stray ACKs land here
|
||||
# and are discarded.
|
||||
|
||||
elif self._state == self._IN_FRAME:
|
||||
if b == DLE:
|
||||
self._state = self._AFTER_DLE
|
||||
elif b == ETX:
|
||||
self._state = self._IDLE
|
||||
return self._finalise()
|
||||
else:
|
||||
self._body.append(b)
|
||||
|
||||
elif self._state == self._AFTER_DLE:
|
||||
# Uniform rule: the escaped byte is itself, including 0x03.
|
||||
self._body.append(b)
|
||||
self._state = self._IN_FRAME
|
||||
|
||||
return None
|
||||
|
||||
def _finalise(self) -> Optional[MicromateFrame]:
|
||||
body = bytes(self._body)
|
||||
if len(body) < _MIN_PAYLOAD + 1:
|
||||
return None
|
||||
|
||||
payload, chk_received = body[:-1], body[-1]
|
||||
if payload[0] != MM_RSP_CMD:
|
||||
return None # a request frame, or garbage that framed by accident
|
||||
|
||||
return MicromateFrame(
|
||||
sub = payload[2],
|
||||
flags = payload[1],
|
||||
page_hi = payload[3],
|
||||
page_lo = payload[4],
|
||||
data = payload[5:],
|
||||
checksum_valid = (chk_received == checksum(payload)),
|
||||
chk_byte = chk_received,
|
||||
)
|
||||
+31
-10
@@ -22,10 +22,30 @@ One rule covers both directions: after the leading doubled `BW_CMD`, every
|
||||
escapes literal `0x03` bytes in write data so they are not mistaken for ETX,
|
||||
exactly as Blastware does.
|
||||
|
||||
That rule was chosen by evidence, not assumption: of the four candidates tried
|
||||
against the 9-24-26 capture's four data-carrying write frames, it is the only
|
||||
one under which all four checksums validate. See
|
||||
`docs/micromate_protocol_reference.md` → *The write path*.
|
||||
A Micromate escapes exactly four byte values — `0x02 0x03 0x04 0x10` — and
|
||||
nothing else, which is what makes the uniform rule exact rather than merely
|
||||
convenient. Established by re-stuffing all 502 captured frames and comparing
|
||||
to the wire: 251/251 each direction, where `{0x10}` alone gets 130 and 177.
|
||||
|
||||
⚠ Checksum, corrected 2026-09-27
|
||||
--------------------------------
|
||||
With uniform destuffing the checksum is **plain SUM8 of the destuffed
|
||||
payload**. This script used to try SUM8 *and* a "DLE-aware" variant that
|
||||
excludes `0x10` bytes, and report whichever matched — which is why it never
|
||||
flagged a bad frame and why the protocol reference carried the wrong rule for
|
||||
two days. The DLE-aware form belongs with *Series III* destuffing, where an
|
||||
escaped byte survives as two bytes; applying it after uniform destuffing
|
||||
subtracts the correction twice and disagrees with the wire on 55 of 251
|
||||
responses.
|
||||
|
||||
The lesson generalises: a tool that accepts any of N candidate rules cannot
|
||||
falsify any of them. It now validates against SUM8 alone, and reports
|
||||
`DLE-aware` only to name what a mismatch *would* have been — never as a pass.
|
||||
See `docs/micromate_protocol_reference.md` → *Checksum*, and
|
||||
`tests/test_micromate_framing.py`, which pins it.
|
||||
|
||||
`micromate/framing.py` is the production implementation; this stays as the
|
||||
one-shot capture-inspection tool.
|
||||
|
||||
Usage
|
||||
-----
|
||||
@@ -119,12 +139,13 @@ def frames(blob: bytes, *, is_request: bool):
|
||||
except ValueError:
|
||||
i += 1
|
||||
continue
|
||||
sum8 = sum(payload) & 0xFF
|
||||
dle_aware = (sum(b for b in payload if b != DLE) & 0xFF)
|
||||
if sum8 == chk:
|
||||
kind = "SUM8"
|
||||
elif dle_aware == chk:
|
||||
kind = "DLE-aware"
|
||||
# SUM8 of the destuffed payload is THE rule -- 502/502 captured frames.
|
||||
# The DLE-aware variant is reported only to name a near-miss; it is
|
||||
# never a pass. See the module docstring.
|
||||
if (sum(payload) & 0xFF) == chk:
|
||||
kind = "ok"
|
||||
elif (sum(b for b in payload if b != DLE) & 0xFF) == chk:
|
||||
kind = "BAD(dle-aware)"
|
||||
else:
|
||||
kind = "BAD"
|
||||
yield i, payload, chk, kind
|
||||
|
||||
@@ -0,0 +1,399 @@
|
||||
"""Framing tests for the Micromate (series-4) live protocol.
|
||||
|
||||
Every constant below is a **real frame**, lifted from
|
||||
``bridges/captures/9-24-26 - micromate2/`` (UM12947, firmware 11.0CB) or from
|
||||
``scratch/fake_unit.py``, which preserves a POLL probe reply. Frames are
|
||||
embedded as hex rather than read from disk because both ``bridges/captures/``
|
||||
and ``tests/fixtures/`` are gitignored -- these tests must pass on a fresh
|
||||
clone.
|
||||
|
||||
The few synthesised frames are marked ``SYNTH_`` and each says what it stands
|
||||
in for and why a captured frame was not available.
|
||||
|
||||
Two of these tests exist because the first draft of
|
||||
``docs/micromate_client_spec.md`` got the rule wrong, and both wrong rules
|
||||
fail quietly -- a frame the unit ignores, or a checksum that reads as bad:
|
||||
|
||||
* ``test_builder_matches_thor_byte_for_byte`` -- the escape set. Escaping
|
||||
only 0x10 (the series-3 rule) reproduces 161 of Thor's 218 read frames.
|
||||
* ``test_checksum_is_plain_sum8_over_destuffed_payload`` -- the checksum.
|
||||
The DLE-aware variant disagrees with the wire on 55 of 251 responses.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from micromate.framing import (
|
||||
ACK,
|
||||
DLE,
|
||||
ETX,
|
||||
FLAGS_BLASTWARE,
|
||||
FLAGS_THOR,
|
||||
STX,
|
||||
MicromateFrame,
|
||||
MicromateFrameParser,
|
||||
build_request,
|
||||
checksum,
|
||||
stuff,
|
||||
unstuff,
|
||||
)
|
||||
|
||||
# ── Captured response frames ──────────────────────────────────────────────────
|
||||
|
||||
# POLL probe reply, 19 B on the wire -- the shortest valid frame there is.
|
||||
# Preserved in scratch/fake_unit.py, captured from UM12947 on 2026-09-24.
|
||||
# Its payload[8:10] is 0x0030, the data length POLL then asks for.
|
||||
RSP_POLL_PROBE = bytes.fromhex("0200c5a400000000000030000000000000009903")
|
||||
|
||||
# SUB 0x49 -> 0xB6, the cheap state read. Carries a literal 0x02, escaped.
|
||||
RSP_STATE = bytes.fromhex("0200c5b6000005000000000000000000001002e8000b007503")
|
||||
|
||||
# SUB 0x48 -> 0xB7, a path-addressed file read. offset_hi 0x04 arrives escaped,
|
||||
# so page_hi is only correct if the parser destuffs before indexing.
|
||||
RSP_FILE_READ = bytes.fromhex("0200c5b70000100400000000000000010000000000018203")
|
||||
|
||||
# An 0x5A download chunk, 138 B on the wire. THE checksum case: its payload
|
||||
# holds literal 0x10 bytes, so plain SUM8 (0xC1, correct) and the DLE-aware
|
||||
# variant (0x91) disagree. Also holds a literal 0x41, which is NOT escaped.
|
||||
RSP_CHUNK_WITH_DLE = bytes.fromhex(
|
||||
"0200c5a500007000003400000000000000e3fd1f10020f0f0e2e1e1fd4f2d4c3d2f00f000"
|
||||
"e003fe03fe200d6e2d12d2e1c4e101011d101f3d0f3e0f2efe23d2b2e3c0d3e0e2c4101e0"
|
||||
"d4d3e21f1d311e202fe2e2f1e3f100210d201002ee2e22d2f2f0f30fd3f0101000f0111f1"
|
||||
"ff01e011010011f0f1002b4d200e0f0302c4010020001fffedb2dc103"
|
||||
)
|
||||
|
||||
# ── Captured request frames (Thor -> unit) ────────────────────────────────────
|
||||
|
||||
REQ_POLL = bytes.fromhex("41021010005b000030000000000000000000009b03")
|
||||
REQ_SERIAL = bytes.fromhex("41021010001500000a000000000000000000002f03")
|
||||
REQ_STATE = bytes.fromhex("41021010004900ffff000000000000000000005703")
|
||||
REQ_COMPLIANCE = bytes.fromhex("41021010001a00ffff000000000000000000002803")
|
||||
REQ_ARM_EVENT = bytes.fromhex("41021010009300ffff00000000000000000000a103")
|
||||
|
||||
# ⚠ The two frames that break a 0x10-only escaper.
|
||||
# Scheduler enable: params[7] = 0x03, on the wire as `10 03`.
|
||||
REQ_SCHED_ON = bytes.fromhex("41021010004700ffff00000000000000100300005803")
|
||||
# Bulk download, first chunk of event 055d4a81: offset 0x0400 puts a literal
|
||||
# 0x04 in offset_hi, on the wire as `10 04`. EVERY download frame needs this.
|
||||
REQ_DOWNLOAD_CHUNK0 = bytes.fromhex(
|
||||
"41021010005a00100400055d4a810000000000009b03"
|
||||
)
|
||||
# A later chunk of the same event: params[2:4] = 0x1000, doubled to `10 10`.
|
||||
REQ_DOWNLOAD_CHUNK4 = bytes.fromhex(
|
||||
"41021010005a0010040000001010000000000000007e03"
|
||||
)
|
||||
|
||||
|
||||
# ── Stuffing ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def test_escape_set_is_exactly_four_bytes():
|
||||
"""0x02, 0x03, 0x04 and 0x10 -- and nothing else.
|
||||
|
||||
ACK (0x41) in particular is NOT escaped; assuming it was reproduced only
|
||||
196 of 251 captured responses.
|
||||
"""
|
||||
assert stuff(bytes([0x02, 0x03, 0x04, 0x10])) == bytes(
|
||||
[DLE, 0x02, DLE, 0x03, DLE, 0x04, DLE, 0x10]
|
||||
)
|
||||
for b in (0x00, 0x01, 0x05, 0x41, 0xC5, 0xFF):
|
||||
assert stuff(bytes([b])) == bytes([b]), f"0x{b:02x} must not be escaped"
|
||||
|
||||
|
||||
def test_unstuff_is_uniform_with_no_inner_frame_carve_out():
|
||||
"""`10 XX` -> `XX` for any XX -- the series-3 DLE+ETX exception is absent."""
|
||||
assert unstuff(bytes.fromhex("1003")) == b"\x03"
|
||||
assert unstuff(bytes.fromhex("1010")) == b"\x10"
|
||||
assert unstuff(bytes.fromhex("001002ff")) == bytes.fromhex("0002ff")
|
||||
|
||||
|
||||
def test_stuff_unstuff_round_trips_over_every_byte_value():
|
||||
data = bytes(range(256))
|
||||
assert unstuff(stuff(data)) == data
|
||||
|
||||
|
||||
def test_a_trailing_dle_is_held_not_dropped():
|
||||
"""A DLE as the last byte of a chunk must not consume nothing and vanish."""
|
||||
assert unstuff(b"\xff\x10") == b"\xff\x10"
|
||||
|
||||
|
||||
# ── Checksum ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def test_checksum_is_plain_sum8_over_destuffed_payload():
|
||||
"""⚠ Plain SUM8 -- do NOT exclude 0x10 bytes.
|
||||
|
||||
RSP_CHUNK_WITH_DLE is a real frame whose payload holds literal 0x10 bytes.
|
||||
The wire says 0xC1; plain SUM8 gives 0xC1 and the DLE-aware variant used by
|
||||
series-3 `5A`/write frames gives 0x91. 55 of 251 captured responses
|
||||
disagree the same way.
|
||||
"""
|
||||
payload = unstuff(RSP_CHUNK_WITH_DLE[1:-1])[:-1]
|
||||
chk_on_wire = unstuff(RSP_CHUNK_WITH_DLE[1:-1])[-1]
|
||||
|
||||
assert DLE in payload, "this frame is only interesting if it holds a 0x10"
|
||||
assert checksum(payload) == chk_on_wire == 0xC1
|
||||
assert (sum(b for b in payload if b != DLE) & 0xFF) == 0x91 # the wrong rule
|
||||
|
||||
|
||||
def test_every_captured_frame_validates():
|
||||
parser = MicromateFrameParser()
|
||||
frames = parser.feed(
|
||||
RSP_POLL_PROBE + RSP_STATE + RSP_FILE_READ + RSP_CHUNK_WITH_DLE
|
||||
)
|
||||
assert len(frames) == 4
|
||||
assert all(f.checksum_valid for f in frames)
|
||||
|
||||
|
||||
# ── Request builder ───────────────────────────────────────────────────────────
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"wire, sub, offset, params",
|
||||
[
|
||||
(REQ_POLL, 0x5B, 0x0030, bytes(10)),
|
||||
(REQ_SERIAL, 0x15, 0x000A, bytes(10)),
|
||||
(REQ_STATE, 0x49, 0xFFFF, bytes(10)),
|
||||
(REQ_COMPLIANCE, 0x1A, 0xFFFF, bytes(10)),
|
||||
(REQ_ARM_EVENT, 0x93, 0xFFFF, bytes(10)),
|
||||
(REQ_SCHED_ON, 0x47, 0xFFFF, bytes.fromhex("00000000000000030000")),
|
||||
(REQ_DOWNLOAD_CHUNK0, 0x5A, 0x0400, bytes.fromhex("055d4a81000000000000")),
|
||||
(REQ_DOWNLOAD_CHUNK4, 0x5A, 0x0400, bytes.fromhex("00001000000000000000")),
|
||||
],
|
||||
ids="poll serial state compliance arm sched_on dl_chunk0 dl_chunk4".split(),
|
||||
)
|
||||
def test_builder_matches_thor_byte_for_byte(wire, sub, offset, params):
|
||||
assert build_request(sub, offset, params) == wire
|
||||
|
||||
|
||||
def test_a_0x10_in_params_needs_no_special_handling():
|
||||
"""The spec's one open question. Thor sends it; the wire doubles it."""
|
||||
frame = build_request(0x5A, 0x0400, bytes.fromhex("00001000000000000000"))
|
||||
assert bytes([DLE, DLE]) in frame
|
||||
assert frame == REQ_DOWNLOAD_CHUNK4
|
||||
|
||||
|
||||
def test_builder_rejects_malformed_arguments():
|
||||
with pytest.raises(ValueError):
|
||||
build_request(0x5B, 0, bytes(9))
|
||||
with pytest.raises(ValueError):
|
||||
build_request(0x5B, 0x10000)
|
||||
with pytest.raises(ValueError):
|
||||
build_request(0x100)
|
||||
|
||||
|
||||
# ── Parsing ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def test_poll_probe_reply_fields():
|
||||
(f,) = MicromateFrameParser().feed(RSP_POLL_PROBE)
|
||||
assert f.sub == 0xA4
|
||||
assert f.request_sub == 0x5B
|
||||
assert f.flags == FLAGS_BLASTWARE
|
||||
assert f.firmware_line == "blastware"
|
||||
assert f.checksum_valid
|
||||
assert f.probe_length == 0x0030
|
||||
|
||||
|
||||
def test_probe_length_is_a_uint16_not_a_byte():
|
||||
"""⚠ Read as data[3] alone, SUB 0x1A's 0x082C (2092) reads as 44 -- 47x low.
|
||||
|
||||
SYNTHESISED: no probe response survives in the captures on disk (the
|
||||
9-24-26 session uses single-step reads at offset 0xFFFF throughout, so it
|
||||
contains no probes at all). The field position is taken from the captured
|
||||
POLL probe reply above, which does exercise it for real.
|
||||
"""
|
||||
payload = bytes([0x00, FLAGS_BLASTWARE, 0xE5, 0x00, 0x00]) + bytes(
|
||||
[0x00, 0x00, 0x00, 0x08, 0x2C]
|
||||
)
|
||||
synth = bytes([STX]) + stuff(payload + bytes([checksum(payload)])) + bytes([ETX])
|
||||
|
||||
(f,) = MicromateFrameParser().feed(synth)
|
||||
assert f.probe_length == 0x082C == 2092
|
||||
assert f.data[3] == 0x08, "the high byte is where a byte-wide read loses 2048"
|
||||
|
||||
|
||||
def test_escaped_bytes_land_in_the_right_field():
|
||||
"""RSP_FILE_READ's first data byte is 0x04, which arrives as `10 04`.
|
||||
|
||||
Without destuffing it reads as 0x10 and every field after it is one byte
|
||||
late -- the failure mode that put `SUB 0x02` in the log as `SUB_10` for an
|
||||
afternoon.
|
||||
"""
|
||||
(f,) = MicromateFrameParser().feed(RSP_FILE_READ)
|
||||
assert f.sub == 0xB7
|
||||
assert f.request_sub == 0x48
|
||||
assert (f.page_hi, f.page_lo) == (0x00, 0x00)
|
||||
assert f.data[0] == 0x04
|
||||
assert len(f.data) == 15, "one byte shorter than the wire suggests"
|
||||
assert f.checksum_valid
|
||||
|
||||
|
||||
def test_thor_firmware_line_survives_destuffing():
|
||||
"""⚠ flags = 0x03 is ETX, so it arrives as `10 03`.
|
||||
|
||||
A parser that does not destuff ends the frame at byte 2 on half the fleet.
|
||||
|
||||
SYNTHESISED: no 11.0BD capture is on disk -- UM20147's sweep was recorded
|
||||
on 2026-09-23 and those bins never landed in the repo. Built by re-stuffing
|
||||
the captured POLL probe reply's payload with flags flipped to 0x03, so the
|
||||
only difference from a real frame is the one byte under test.
|
||||
"""
|
||||
real = unstuff(RSP_POLL_PROBE[1:-1])[:-1]
|
||||
payload = bytes([real[0], FLAGS_THOR]) + real[2:]
|
||||
synth = bytes([STX]) + stuff(payload + bytes([checksum(payload)])) + bytes([ETX])
|
||||
|
||||
assert bytes([DLE, ETX]) in synth, "flags must be escaped on the wire"
|
||||
(f,) = MicromateFrameParser().feed(synth)
|
||||
assert f.flags == FLAGS_THOR
|
||||
assert f.firmware_line == "thor"
|
||||
assert f.sub == 0xA4
|
||||
assert f.checksum_valid
|
||||
|
||||
|
||||
def test_an_escaped_checksum_byte_is_read_correctly():
|
||||
"""SYNTHESISED, but the behaviour is real: three captured responses have a
|
||||
checksum of 0x02/0x03/0x04 and all three escape it on the wire. The
|
||||
shortest is 1,070 B (an 0x5A chunk in
|
||||
raw_s3_20260925_011403_Download_events_then_delete_1_event.bin, chk = 0x03),
|
||||
too long to embed for one byte's worth of assertion.
|
||||
"""
|
||||
payload = bytes([0x00, FLAGS_BLASTWARE, 0xA4, 0x00, 0x00, 0x03])
|
||||
assert checksum(payload) == 0x03 + FLAGS_BLASTWARE + 0xA4 & 0xFF
|
||||
body = payload + bytes([checksum(payload)])
|
||||
synth = bytes([STX]) + stuff(body) + bytes([ETX])
|
||||
|
||||
(f,) = MicromateFrameParser().feed(synth)
|
||||
assert f.chk_byte == checksum(payload)
|
||||
assert f.checksum_valid
|
||||
|
||||
|
||||
def test_a_corrupted_checksum_still_yields_a_frame():
|
||||
"""Flag it, do not swallow it -- a dropped frame looks like a dead unit."""
|
||||
broken = bytearray(RSP_POLL_PROBE)
|
||||
broken[-2] ^= 0xFF
|
||||
(f,) = MicromateFrameParser().feed(bytes(broken))
|
||||
assert f.sub == 0xA4
|
||||
assert not f.checksum_valid
|
||||
|
||||
|
||||
def test_a_truncated_frame_yields_nothing():
|
||||
parser = MicromateFrameParser()
|
||||
assert parser.feed(RSP_POLL_PROBE[:-1]) == []
|
||||
assert parser.frames == []
|
||||
assert parser.bytes_fed == len(RSP_POLL_PROBE) - 1
|
||||
|
||||
|
||||
def test_a_frame_too_short_to_hold_a_header_is_rejected():
|
||||
assert MicromateFrameParser().feed(bytes([STX, 0x00, 0xC5, 0xA4, ETX])) == []
|
||||
|
||||
|
||||
def test_request_frames_are_not_mistaken_for_responses():
|
||||
"""Feeding a bidirectional capture must yield only the unit's side."""
|
||||
parser = MicromateFrameParser()
|
||||
frames = parser.feed(REQ_POLL + RSP_POLL_PROBE + REQ_SERIAL)
|
||||
assert len(frames) == 1
|
||||
assert frames[0].request_sub == 0x5B
|
||||
|
||||
|
||||
def test_leading_noise_is_discarded():
|
||||
"""Cold-boot banners and the RV55's RING/CONNECT chatter precede frames."""
|
||||
noise = b"\r\nRING\r\n\r\nCONNECT\r\n" + bytes([ACK])
|
||||
(f,) = MicromateFrameParser().feed(noise + RSP_POLL_PROBE)
|
||||
assert f.sub == 0xA4
|
||||
assert f.checksum_valid
|
||||
|
||||
|
||||
def test_frames_split_across_feeds_reassemble():
|
||||
"""The transport hands over whatever the socket returned, DLE pairs and all."""
|
||||
whole = RSP_STATE + RSP_CHUNK_WITH_DLE
|
||||
for cut in (1, 2, 5, 17, 24, 25, 40, len(RSP_STATE), len(whole) - 1):
|
||||
parser = MicromateFrameParser()
|
||||
got = parser.feed(whole[:cut]) + parser.feed(whole[cut:])
|
||||
assert len(got) == 2, f"split at {cut} lost a frame"
|
||||
assert all(f.checksum_valid for f in got), f"split at {cut} broke a checksum"
|
||||
|
||||
|
||||
def test_reset_clears_partial_state():
|
||||
parser = MicromateFrameParser()
|
||||
parser.feed(RSP_POLL_PROBE[:6])
|
||||
parser.reset()
|
||||
assert parser.bytes_fed == 0
|
||||
(f,) = parser.feed(RSP_POLL_PROBE)
|
||||
assert f.checksum_valid
|
||||
|
||||
|
||||
def test_firmware_line_of_an_unknown_flags_byte():
|
||||
f = MicromateFrame(sub=0xA4, flags=0x99, page_hi=0, page_lo=0,
|
||||
data=b"", checksum_valid=True)
|
||||
assert f.firmware_line == "unknown"
|
||||
assert f.probe_length is None
|
||||
|
||||
|
||||
# ── Whole-session round trip ──────────────────────────────────────────────────
|
||||
|
||||
_CAPTURES = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "bridges" / "captures" / "9-24-26 - micromate2"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
not _CAPTURES.is_dir(),
|
||||
reason="capture directory is gitignored; present only on a dev box",
|
||||
)
|
||||
def test_whole_captured_sessions_parse_with_no_bad_checksums():
|
||||
"""Belt-and-braces against the real bytes when they happen to be here.
|
||||
|
||||
scratch/mm_frame_parse.py reported zero bad checksums on these sessions
|
||||
only because it accepts a frame matching *either* checksum rule. This
|
||||
asserts the single rule holds across all of them.
|
||||
"""
|
||||
total = 0
|
||||
for path in sorted(_CAPTURES.rglob("raw_s3_*.bin")):
|
||||
parser = MicromateFrameParser()
|
||||
frames = parser.feed(path.read_bytes())
|
||||
assert frames, f"{path.name}: no frames parsed"
|
||||
bad = [f for f in frames if not f.checksum_valid]
|
||||
assert not bad, f"{path.name}: {len(bad)} bad checksums"
|
||||
total += len(frames)
|
||||
assert total == 251, f"expected 251 response frames across the corpus, got {total}"
|
||||
|
||||
|
||||
@pytest.mark.skipif(
|
||||
not _CAPTURES.is_dir(),
|
||||
reason="capture directory is gitignored; present only on a dev box",
|
||||
)
|
||||
def test_builder_reproduces_every_captured_read_frame():
|
||||
"""218/218. This is the test that would have caught the escape-set error."""
|
||||
checked = 0
|
||||
for path in sorted(_CAPTURES.rglob("raw_bw_*.bin")):
|
||||
blob = path.read_bytes()
|
||||
i = 0
|
||||
while i < len(blob):
|
||||
if not (blob[i] == ACK and i + 1 < len(blob) and blob[i + 1] == STX):
|
||||
i += 1
|
||||
continue
|
||||
j = i + 2
|
||||
body = bytearray()
|
||||
while j < len(blob):
|
||||
if blob[j] == DLE and j + 1 < len(blob):
|
||||
body.append(blob[j + 1])
|
||||
j += 2
|
||||
continue
|
||||
if blob[j] == ETX:
|
||||
break
|
||||
body.append(blob[j])
|
||||
j += 1
|
||||
payload = bytes(body[:-1])
|
||||
if len(payload) == 16: # a read frame; writes carry a data section
|
||||
sub = payload[2]
|
||||
offset = (payload[4] << 8) | payload[5]
|
||||
assert build_request(sub, offset, payload[6:16]) == blob[i:j + 1], (
|
||||
f"{path.name} @0x{i:04x} SUB=0x{sub:02x} offset=0x{offset:04x}"
|
||||
)
|
||||
checked += 1
|
||||
i = j + 1
|
||||
assert checked == 218, f"expected 218 read frames, checked {checked}"
|
||||
Reference in New Issue
Block a user