diff --git a/docs/micromate_client_spec.md b/docs/micromate_client_spec.md index ff5ca38..b6a619f 100644 --- a/docs/micromate_client_spec.md +++ b/docs/micromate_client_spec.md @@ -173,7 +173,16 @@ timeout, and that distinction earned its keep during the Series III work). --- -## `micromate/protocol.py` +## `micromate/protocol.py` — ✅ BUILT 2026-09-27 + +Implemented, with `tests/test_micromate_protocol.py` (35 tests). Reads only; +nothing here writes, erases or changes monitoring state. + +The tests replay Thor's captured responses through a scripted transport and +assert **the bytes we emit are the bytes Thor emits** — including a full replay +of the six-event download session, all 56 `0x5A` frames byte-for-byte. That is +a stronger guarantee than "our parser understands the device": a passing test +means a real unit has already answered exactly that frame. One method per command, returning raw payload bytes. No interpretation — that belongs in `client.py`. @@ -198,9 +207,29 @@ taking its data length. Per-command offsets, all observed: | arm event | `0x93` | `0x6C` | — | before every event | | first event | `0x1E` | `0xE1` | `0xFFFF` | key + size | | next event | `0x1F` | `0xE0` | `0xFFFF` | key + size | -| event record | `0x0C` | `0xF3` | `0xFFFF` | 210 B — project, location, peaks | -| event header | `0x0A` | `0xF5` | `0xFFFF` | 30 B list record | -| bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim** | +| event record | `0x0C` | `0xF3` | `0xFFFF` | 221 B — project, location, peaks | +| ~~event header~~ **monitor log** | `0x0A` | `0xF5` | `0xFFFF` | ⚠ 297 B, a **walk** — see below | +| bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim**, 1024 B at a time | + +⚠ **Corrected 2026-09-27, from Thor's frames.** Three rows of the table above +were wrong or incomplete, and the last one is a different command than labelled: + +- **`0x0A` is the monitor-log walk**, not a keyed "30 B list record" read. The + *same request repeated* returns successive 297-byte records — serial, mode, + thresholds — until an 11-byte ack ends the list. The device holds the cursor; + nothing in the request selects a record. Series III reaches this data through + a record-type discriminator on its event chain; here it has its own cursor and + the event chain never sees it. +- **`0x1E`/`0x1F` carry token `0xFE` at `params[7]`.** The reference documents + all-zero params (our own probing, which also worked). Thor's form is the one + with mileage. +- **`0x01` has no Thor frame behind it** — it is never read in any captured + session. Its `0xFFFF` comes from our probes. + +And one useful negative: **no `SESSION_RESET` (`41 03`)**. Series III needs that +2-byte signal or a monitoring unit will not answer `POLL` over TCP. Thor never +sends it — zero occurrences across 8 sessions, including 40 frames exchanged +with a unit that *was* monitoring. ⚠ **`SUB 0x1C` is 4 bytes longer on the Thor firmware line** (`0x30` vs `0x2C`). Parse **forward** from `declared_length`, never backward from the end — Series @@ -328,19 +357,21 @@ then `download_event()` and assert the bytes decode and match a ## Order of work 1. ✅ `framing.py` + its tests — **done 2026-09-27**, 31 tests, offline -2. `protocol.py` — reads only, one method per row of the table above +2. ✅ `protocol.py` + its tests — **done 2026-09-27**, 35 tests, offline 3. `client.py` — `connect()`, `get_state()`, `list_setups()` 4. the event chain and `download_event()` 5. decode end-to-end and compare against a store event Steps 1–2 need no hardware at all. -**Worth carrying forward from step 1:** every rule got checked against the -captures *before* being written, and two of the three the spec asserted turned -out wrong — the escape set (26% of frames) and the checksum (22%). Both fail -silently. The captures are on disk and a re-stuff-and-compare loop takes about -two minutes per rule, so do that for `protocol.py`'s per-command offsets too -rather than trusting the table above. +**Worth carrying forward.** Both steps began by measuring against the captures +rather than trusting this document, and both found errors in it — three in the +framing rules (the escape set, 26% of frames; the checksum, 22%; the `0x5A` +chunk model) and three more in the command table (`0x0A`'s meaning, the +`1E`/`1F` token, `0x01`'s provenance). All six fail quietly. The captures are on +disk and a measure-then-write loop costs about two minutes per rule, so keep +doing it for `client.py`'s field offsets — and treat this spec as a plan, not a +source. --- diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 4b87096..f444e0f 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1216,6 +1216,96 @@ the protocol's requirements*. Thor sends it before trivial reads too, so it may be habit rather than handshake. Do not assume it is mandatory. Note `POLL` here carries `offset = 0x0030` (its data length), not `0xFFFF` — `POLL` is the one read Thor still addresses by length. +> #### ⚠ Narrowed 2026-09-27 — there is no *universal* preamble +> +> Across all 8 captured sessions, the only invariant is that **the session opens +> with `POLL`**. What follows depends on the operation: +> +> | opening sequence | sessions | operation | +> |---|---|---| +> | `5b 15 49 5b …` | 3 | status refresh / monitoring / ACH change | +> | `5b 41 08 2e 1a da …` | 3 | setup push | +> | `5b 94 48 48 48 …` | 2 | scheduler read | +> +> So `POLL → SERIAL → 0x49 → POLL` is Thor's **connection check**, not a +> handshake the protocol demands — it appears where Thor wants to refresh what +> it displays. Treat `POLL` as the one thing to send first. + +### 🔑 No `SESSION_RESET` — the Series III requirement does not carry over + +Series III needs a bare `41 03` (ACK + ETX, no STX) to wake a unit that is +actively monitoring; without it the unit will not answer `POLL` over TCP, and +`protocol.startup()` sends it before and between the POLL frames. + +**Thor never sends it to a Micromate.** Zero occurrences across all 8 sessions +— including `raw_bw_20260924_191214_turn_on_monitormode_…`, which exchanges 40 +frames with a unit that *was* monitoring at the time. + +### Measured offsets and response lengths (all read off Thor's frames) + +`offset = 0xFFFF` for everything except two commands. Data lengths are from +UM12947 (`11.0CB`) and are **orientation, not assertions** — `0x1C` is 4 bytes +longer on the Thor line. + +| SUB | rsp | offset | data | notes | +|---|---|---|---|---| +| `0x5B` POLL | `0xA4` | **`0x0030`** | 59 | the one length-addressed read | +| `0x15` serial | `0xEA` | **`0x000A`** | 21 | | +| `0x49` state | `0xB6` | `0xFFFF` | 16 | | +| `0x1C` monitor status | `0xE3` | `0xFFFF` | 55 | +4 on `11.0BD` | +| `0x06` storage range | `0xF9` | `0xFFFF` | 47 | | +| `0x08` event index | `0xF7` | `0xFFFF` | 101 | contents unmapped | +| `0x2E` trigger config | `0xD1` | `0xFFFF` | 39 | | +| `0x1A` compliance | `0xE5` | `0xFFFF` | 2103 | one frame, not Series III's four | +| `0x2C` call-home | `0xD3` | `0xFFFF` | 137 | | +| `0x3F`/`0x40`/`0x41` setups | `0xC0`/`0xBF`/`0xBE` | `0xFFFF` | 266 | | +| `0x93` arm | `0x6C` | `0xFFFF` | 11 | ack only | +| `0x1E`/`0x1F` chain | `0xE1`/`0xE0` | `0xFFFF` | 19 | ⚠ **token `0xFE` at `params[7]`** | +| `0x0C` event record | `0xF3` | `0xFFFF` | 221 | full key at `params[4:8]` | +| `0x0A` monitor log | `0xF5` | `0xFFFF` | 297 | ⚠ a **walk** — see below | +| `0x5A` download | `0xA5` | computed | offset+11 | 1024-byte chunk loop | + +An acknowledgement is an **11-byte data section**, and that doubles as the +end-of-list signal on the walks. + +#### ⚠ `0x1E`/`0x1F` carry token `0xFE` + +`params = 00 00 00 00 00 00 00 fe 00 00` on all 7 captured chain reads — the +same `token_params(0xFE)` form Series III uses to arm its bulk stream. The +event-chain section above documents **all-zero params**; that was our own browse +probing, which also worked. Both evidently do, but Thor's form is the one with +mileage on it, and it is sent on browse and download alike. + +#### 🔑 `SUB 0x0A` is the monitor-log walk, not a keyed read + +The command table long described `0x0A` as a keyed "waveform header / partial +record" read, by analogy with Series III. What the bytes show is a **cursor +walk**: the *same request repeated*, the device advancing its own position. + +``` +0x93 → 1E → 0x0A ×8 (297 B each: "UM12947", "Histo…", "Ver…", " 0.49", " 28.4") + 0x0A (11 B ack = end of list) +``` + +All nine frames carry identical params (`…00 00 4a 81 00 00`), so nothing in the +request selects the record. Terminate on a response of `ACK_DATA_LEN` (11). + +⚠ The `4a 81` is the **low two bytes** of the event key then in play +(`055d4a81`). One key cannot distinguish "the key's low half" from "a cursor +handle that happened to equal it" — both produce those bytes. It does not +matter operationally, since the walk works with the params held constant. + +This is a genuine structural divergence: Series III reaches the same data +through a record-type discriminator (`0x2C` partial vs `0x46` full) *on its +event walk*, so partial records and events share one chain. Here the monitor +log has its own cursor and the event chain never sees it. + +#### `SUB 0x01` has no Thor frame behind it + +Thor never reads device info in any captured session. `0xFFFF` for `0x01` comes +from our own 2026-09-23 probes — it answered correctly on both firmware lines, +but it is the only read in the table with no Thor precedent. + ### `SUB 0x96` / `0x97` — start and stop monitoring ✅ Identical to Series III, including the acks: diff --git a/micromate/protocol.py b/micromate/protocol.py new file mode 100644 index 0000000..6e35a2f --- /dev/null +++ b/micromate/protocol.py @@ -0,0 +1,501 @@ +""" +protocol.py — one method per Micromate (Series IV) wire command. + +Returns raw payload bytes. Interpretation belongs in ``client.py``; this layer +knows frames, offsets and sequencing, and nothing about what a field means. + +Scope: **reads only.** Nothing here writes, erases, or changes monitoring +state. That is deliberate and worth keeping — no command has ever been +originated against a unit by this project; every write in +``docs/micromate_protocol_reference.md`` was performed by THOR while we +recorded. The first thing this codebase ever sends to a customer's instrument +should be a decision someone made on purpose, not a side effect of a client +that grew a method. + +Every offset and params layout below was read off THOR's own frames in +``bridges/captures/9-24-26 - micromate2/`` rather than taken from the spec +table, because the same exercise during the framing work found three of that +table's rules wrong. It found three more here: + + * ``0x0A`` is the **monitor-log walk** — the same request repeated, the + device advancing its own cursor, terminated by a short response — not the + keyed "event header, 30 B list record" the spec describes. + * ``0x1E``/``0x1F`` carry **token 0xFE** at ``params[7]``. The protocol + reference documents all-zero params for the browse walk; that was our own + probing, and THOR does not do it that way. + * There is **no fixed preamble**. Sessions open with ``POLL`` and go + straight to the operation. ``POLL → SERIAL → 0x49 → POLL`` appears in 3 of + 8 captured sessions and is THOR's *connection check*, not a handshake. + +And one useful negative: **no ``SESSION_RESET`` (``41 03``).** Series III +needs that 2-byte signal to wake a monitoring unit or it will not answer POLL +over TCP. THOR never sends it — 0 occurrences across all 8 sessions, including +40 frames exchanged with a unit that *was* monitoring. +""" + +from __future__ import annotations + +import logging +import math +import struct +import time +from typing import Optional + +from minimateplus.transport import BaseTransport + +from .framing import MicromateFrame, MicromateFrameParser, build_request + +log = logging.getLogger(__name__) + +DEFAULT_RECV_TIMEOUT = 10.0 + +# An acknowledgement carries an 11-byte data section and nothing else. It is +# also how the monitor-log walk says "no more records". +ACK_DATA_LEN = 11 + + +# ── Command SUBs ────────────────────────────────────────────────────────────── + +SUB_DEVICE_INFO = 0x01 +SUB_STORAGE_RANGE = 0x06 +SUB_EVENT_INDEX = 0x08 +SUB_MONITOR_LOG = 0x0A +SUB_EVENT_RECORD = 0x0C +SUB_SERIAL = 0x15 +SUB_COMPLIANCE_CONFIG = 0x1A +SUB_MONITOR_STATUS = 0x1C +SUB_EVENT_FIRST = 0x1E +SUB_EVENT_NEXT = 0x1F +SUB_CALL_HOME_CONFIG = 0x2C +SUB_TRIGGER_CONFIG = 0x2E +SUB_SETUP_FIRST = 0x3F +SUB_SETUP_NEXT = 0x40 +SUB_SETUP_ACTIVE = 0x41 +SUB_STATE = 0x49 +SUB_BULK_DOWNLOAD = 0x5A +SUB_POLL = 0x5B +SUB_ARM_EVENT = 0x93 + +# ⚠ Reads are SINGLE-STEP. Series III probes at offset 0 to learn the length, +# then reads again at that length; a Micromate returns the whole block when +# asked for 0xFFFF. THOR never probes, which is why `MicromateFrame.probe_length` +# reads 0 on live traffic. +READ_ALL = 0xFFFF + +# The two commands that do NOT use READ_ALL, and the data length each returned +# on UM12947 (firmware 11.0CB). +_OFFSETS = { + SUB_POLL: 0x0030, # 59 B — the one offset THOR treats as a constant + SUB_SERIAL: 0x000A, # 21 B +} + +# Data-section lengths observed, for orientation only — deliberately NOT +# asserted. `SUB 0x1C` is 4 bytes longer on the Thor firmware line (0x30 vs +# 0x2C declared), so a length check here would fire spuriously on half the +# fleet. See the protocol reference, "A/B: Blastware build vs Thor build". +OBSERVED_DATA_LEN = { + SUB_STORAGE_RANGE: 47, SUB_EVENT_INDEX: 101, SUB_MONITOR_LOG: 297, + SUB_EVENT_RECORD: 221, SUB_SERIAL: 21, SUB_COMPLIANCE_CONFIG: 2103, + SUB_MONITOR_STATUS: 55, SUB_EVENT_FIRST: 19, SUB_EVENT_NEXT: 19, + SUB_CALL_HOME_CONFIG: 137, SUB_TRIGGER_CONFIG: 39, + SUB_SETUP_FIRST: 266, SUB_SETUP_NEXT: 266, SUB_SETUP_ACTIVE: 266, + SUB_STATE: 16, SUB_POLL: 59, SUB_ARM_EVENT: ACK_DATA_LEN, +} + +# `1E`/`1F` carry this at params[7]. Series III uses the same value to arm its +# bulk stream; here THOR sends it on every chain read, browse or download. +EVENT_TOKEN = 0xFE + +# `SUB 0x5A` chunk size, in bytes of file payload per response. +CHUNK_SIZE = 1024 + +# Every `0x5A` response prefixes the file bytes with 11 bytes of header. +_CHUNK_PREFIX = 11 + + +# ── Exceptions ──────────────────────────────────────────────────────────────── + +class ProtocolError(Exception): + """The device violated the expected protocol.""" + + +class TimeoutError(ProtocolError): + """No response arrived within the allowed time.""" + + +class ChecksumError(ProtocolError): + """A received frame failed its checksum.""" + + +class UnexpectedResponse(ProtocolError): + """The response SUB did not match the request.""" + + +class ShortRead(ProtocolError): + """A bulk download returned fewer bytes than the device promised.""" + + +# ── Params builders ─────────────────────────────────────────────────────────── + +def token_params(token: int = EVENT_TOKEN) -> bytes: + """`1E`/`1F`: the token sits at params[7].""" + return bytes(7) + bytes([token]) + bytes(2) + + +def key_params(key4: bytes) -> bytes: + """`0x0C`: the full 4-byte event key at params[4:8].""" + if len(key4) != 4: + raise ValueError(f"key4 must be 4 bytes, got {len(key4)}") + return bytes(4) + key4 + bytes(2) + + +def key_lo_params(key4: bytes) -> bytes: + """`0x0A`: only the key's **low two bytes**, at params[6:8]. + + ⚠ Inferred from a single key value. All nine captured `0x0A` frames carry + `4a 81`, and the event key in play was `055d4a81` — so this is consistent + with "the low half of the current key" and equally consistent with "a + cursor handle that happened to equal it". Both readings produce the same + bytes for that key, so one event cannot separate them. + + It does not matter much in practice: the walk works with the same params + repeated, so whichever it is, passing the current key is right. + """ + if len(key4) != 4: + raise ValueError(f"key4 must be 4 bytes, got {len(key4)}") + return bytes(6) + key4[2:4] + bytes(2) + + +def chunk_params(key4: bytes, byte_offset: int) -> bytes: + """`0x5A`: the key opens the file, then a byte offset walks it. + + Chunk 0 carries the event key at params[0:4] — that is what says "from the + beginning". Later chunks carry a uint16 BE byte offset at params[2:4]. + """ + if byte_offset == 0: + if len(key4) != 4: + raise ValueError(f"key4 must be 4 bytes, got {len(key4)}") + return key4 + bytes(6) + if not 0 <= byte_offset <= 0xFFFF: + raise ValueError(f"byte_offset must fit in uint16, got {byte_offset}") + return bytes(2) + struct.pack(">H", byte_offset) + bytes(6) + + +# ── Protocol ────────────────────────────────────────────────────────────────── + +class MicromateProtocol: + """Wire-level command set for one open connection to a Micromate. + + Does not own the transport; lifetime belongs to the client. + + proto = MicromateProtocol(transport) + proto.poll() + serial = proto.read_serial() + """ + + def __init__( + self, + transport: BaseTransport, + recv_timeout: float = DEFAULT_RECV_TIMEOUT, + strict_checksums: bool = True, + ) -> None: + """ + Args: + strict_checksums: raise on a bad checksum. **Defaults to True, + unlike the Series III sibling**, which logs and continues + because its parser cannot reliably tell an inner-frame + delimiter from a checksum byte. That excuse does not apply + here: the Micromate rule is plain SUM8 over the de-stuffed + payload and it holds on 251 of 251 captured frames, so a + mismatch means something real — line noise, a desync, or a rule + we have wrong — and all three are worth hearing about. + + The lenient Series III default is instructive: it hid the fact + that the documented checksum rule was wrong for two days. Set + False only to get a field diagnosis unstuck. + """ + self._transport = transport + self._recv_timeout = recv_timeout + self._strict = strict_checksums + self._parser = MicromateFrameParser() + self._pending: list[MicromateFrame] = [] + + # ── Identity and state ──────────────────────────────────────────────────── + + def poll(self) -> MicromateFrame: + """`0x5B` → `0xA4`. Handshake; carries the ID block and model string. + + Every captured session opens with this and nothing before it. + """ + return self._exchange(SUB_POLL) + + def read_serial(self) -> bytes: + """`0x15` → `0xEA`. ASCII, null-terminated — e.g. `UM12947`.""" + return self._read(SUB_SERIAL) + + def read_device_info(self) -> bytes: + """`0x01` → `0xFE`. Firmware, calibration, per-channel float block. + + ⚠ THOR never sends this in any captured session, so the `0xFFFF` offset + is from our own 2026-09-23 probes rather than from THOR's behaviour. It + answered correctly on both firmware lines, but it is the one read here + with no THOR frame behind it. + """ + return self._read(SUB_DEVICE_INFO) + + def read_state(self) -> bytes: + """`0x49` → `0xB6`. A cheap monitoring check; 16 B. + + ⚠ Test `data[11]` for **non-zero**, never against a constant — it has + read both `0x0E` and `0x0C` while monitoring. + """ + return self._read(SUB_STATE) + + def read_monitor_status(self) -> bytes: + """`0x1C` → `0xE3`. Flag, **device clock**, battery, memory. + + ⚠ Parse **forward** from the declared length, never backward from the + end. This block is 4 bytes longer on the Thor firmware line, and Series + III's relative-to-end offsets yield a battery voltage of 577.92 V on a + `11.0BD` unit. + """ + return self._read(SUB_MONITOR_STATUS) + + def read_storage_range(self) -> bytes: + """`0x06` → `0xF9`. Event storage extent; 47 B.""" + return self._read(SUB_STORAGE_RANGE) + + def read_event_index(self) -> bytes: + """`0x08` → `0xF7`. 101 B. Contents not yet mapped.""" + return self._read(SUB_EVENT_INDEX) + + def read_trigger_config(self) -> bytes: + """`0x2E` → `0xD1`. 39 B. Series IV only; no Series III equivalent.""" + return self._read(SUB_TRIGGER_CONFIG) + + def read_compliance_config(self) -> bytes: + """`0x1A` → `0xE5`. The whole active setup — 2103 B on UM12947. + + One response. Series III needs a 4-frame sequence for the same thing. + """ + return self._read(SUB_COMPLIANCE_CONFIG) + + def read_call_home_config(self) -> bytes: + """`0x2C` → `0xD3`. 137 B — Series III's is 124, so do not reuse its map.""" + return self._read(SUB_CALL_HOME_CONFIG) + + # ── Setups ──────────────────────────────────────────────────────────────── + + def read_active_setup_name(self) -> bytes: + """`0x41` → `0xBE`. 266 B; carries the active `.MMB` name.""" + return self._read(SUB_SETUP_ACTIVE) + + def read_first_setup(self) -> bytes: + """`0x3F` → `0xC0`. Head of the setup-file list.""" + return self._read(SUB_SETUP_FIRST) + + def read_next_setup(self) -> bytes: + """`0x40` → `0xBF`. Repeat until the record carries an empty name. + + Stateful: the device holds the cursor, so the same request walks the + list. 22 of these appear back to back in one captured session. + """ + return self._read(SUB_SETUP_NEXT) + + # ── Event chain ─────────────────────────────────────────────────────────── + + def arm_event(self) -> MicromateFrame: + """`0x93` → `0x6C`. THOR sends this before **every** `1E`/`1F`. + + It replaces Series III's `1E(token=0xFE)` arming step. No params, no + offset payload — an 11-byte ack. + + ⚠ Whether a unit actually requires it is untested. Do it because it is + known-good, not because it is known-necessary. + """ + return self._exchange(SUB_ARM_EVENT, offset=READ_ALL) + + def read_event_first(self) -> bytes: + """`0x1E` → `0xE1`. First event key + size; 19 B.""" + return self._read(SUB_EVENT_FIRST, params=token_params()) + + def read_event_next(self) -> bytes: + """`0x1F` → `0xE0`. Next key + size, or the all-zero null sentinel.""" + return self._read(SUB_EVENT_NEXT, params=token_params()) + + def read_event_record(self, key4: bytes) -> bytes: + """`0x0C` → `0xF3`. 221 B — project, client, operator, timestamp, peaks. + + ⚠ The peak float in here runs 2–5% above `max(T,V,L)` and is **not** the + vector sum; its offset was inferred, not established. Prefer decoded + samples. + """ + return self._read(SUB_EVENT_RECORD, params=key_params(key4)) + + def read_monitor_log_next(self, key4: bytes) -> Optional[bytes]: + """`0x0A` → `0xF5`. One monitor-log record, or None at end of list. + + ⚠ Not the keyed single read the spec describes. This is a **walk**: + the same request repeated, the device advancing its own cursor, each + response a 297-byte record carrying serial, mode and thresholds. The + list ends with a bare 11-byte ack — nine captured frames, eight records + then the terminator. + + Series III reaches the same data through a record-type discriminator on + its event walk (`0x2C` partial vs `0x46` full). Here it is a separate + cursor and the event chain does not see it at all. + """ + data = self._read(SUB_MONITOR_LOG, params=key_lo_params(key4)) + return None if len(data) <= ACK_DATA_LEN else data + + # ── Bulk download ───────────────────────────────────────────────────────── + + def read_event_file(self, key4: bytes, size: int) -> bytes: + """`0x5A` → `0xA5`. The `.IDFW`/`.IDFH` file, byte for byte. + + `size` is the 4 bytes after the key in the `1E`/`1F` response. Returns + exactly that many bytes, or raises `ShortRead`. + + A bounded chunk walk — `ceil(size / 1024)` requests, each asking for + `min(1024, remaining)` bytes: + + offset = the byte count wanted (NOT an address) + params = the key on chunk 0, then a uint16 BE byte offset + response = exactly `offset + 11` bytes; file bytes are data[11:] + + Verified against THOR on all six bench events (4,076 → 13,424 B): + `sum(offsets) == size` exactly, every time. + + ⚠ Do not port the Series III `5A` walk. Its address arithmetic caused a + 5x over-read and a `>64 KB` page-boundary bug that is *still open* on + that side. Neither applies here — the cursor is a byte offset into the + file, bounded by a size the device supplied, so it cannot run past the + event. + + The result feeds `micromate.idf_file.read_idf_file()` and + `/db/import/idf_file` unchanged; no new codec work is needed. + """ + if size <= 0: + raise ValueError(f"size must be positive, got {size}") + + out = bytearray() + n_chunks = math.ceil(size / CHUNK_SIZE) + for i in range(n_chunks): + want = min(CHUNK_SIZE, size - i * CHUNK_SIZE) + data = self._read( + SUB_BULK_DOWNLOAD, + offset=want, + params=chunk_params(key4, i * CHUNK_SIZE), + ) + if len(data) < _CHUNK_PREFIX: + raise ShortRead( + f"chunk {i + 1}/{n_chunks} of {key4.hex()}: " + f"{len(data)} B is too short to hold a chunk header" + ) + body = data[_CHUNK_PREFIX:] + if len(body) != want: + # Worth being loud: a silently short event is the failure mode + # this project has been bitten by repeatedly on the Series III + # side, and here the expected length is known up front. + raise ShortRead( + f"chunk {i + 1}/{n_chunks} of {key4.hex()}: asked for " + f"{want} B, got {len(body)}" + ) + out += body + + if len(out) != size: + raise ShortRead( + f"{key4.hex()}: assembled {len(out)} B, device promised {size}" + ) + log.debug("downloaded %s: %d B in %d chunks", key4.hex(), len(out), n_chunks) + return bytes(out) + + # ── Plumbing ────────────────────────────────────────────────────────────── + + def _read( + self, + sub: int, + *, + params: bytes = bytes(10), + offset: Optional[int] = None, + timeout: Optional[float] = None, + ) -> bytes: + """Send one read command, return the response's data section.""" + return self._exchange(sub, params=params, offset=offset, timeout=timeout).data + + def _exchange( + self, + sub: int, + *, + params: bytes = bytes(10), + offset: Optional[int] = None, + timeout: Optional[float] = None, + ) -> MicromateFrame: + if offset is None: + offset = _OFFSETS.get(sub, READ_ALL) + # Start every exchange clean: drop any half-frame and any frame left + # stashed by the last one. This is a strict request/response protocol, + # so anything already buffered when we send is by definition stale, and + # `expected_sub` would reject it anyway — better to discard it here than + # to raise a confusing UnexpectedResponse one command later. + self._parser.reset() + self._pending.clear() + self._send(build_request(sub, offset, params)) + return self._recv_one(expected_sub=0xFF - sub, timeout=timeout, + reset_parser=False) + + def _send(self, frame: bytes) -> None: + log.debug("TX %d bytes: %s", len(frame), frame.hex()) + self._transport.write(frame) + + def _recv_one( + self, + expected_sub: Optional[int] = None, + timeout: Optional[float] = None, + reset_parser: bool = True, + ) -> MicromateFrame: + """Read until one complete frame is parsed.""" + deadline = time.monotonic() + (timeout or self._recv_timeout) + if reset_parser: + self._parser.reset() + self._pending.clear() + + if self._pending: + return self._validate(self._pending.pop(0), expected_sub) + + while time.monotonic() < deadline: + chunk = self._transport.read(4096) + if not chunk: + time.sleep(0.005) + continue + log.debug("RX %d bytes", len(chunk)) + frames = self._parser.feed(chunk) + if frames: + self._pending.extend(frames[1:]) + return self._validate(frames[0], expected_sub) + + raise TimeoutError( + f"no frame in {timeout or self._recv_timeout:.1f}s" + + (f" (expected SUB 0x{expected_sub:02X})" if expected_sub is not None else "") + + f"; {self._parser.bytes_fed} bytes were received" + # That byte count is the whole point: it separates "nothing came + # back at all" from "bytes arrived but never framed", and those have + # completely different causes. It earned its keep on Series III. + ) + + def _validate( + self, frame: MicromateFrame, expected_sub: Optional[int] + ) -> MicromateFrame: + if not frame.checksum_valid: + msg = ( + f"SUB 0x{frame.sub:02X}: checksum mismatch " + f"(got 0x{frame.chk_byte:02X}, {len(frame.data)} B data)" + ) + if self._strict: + raise ChecksumError(msg) + log.warning("%s — continuing (strict_checksums=False)", msg) + if expected_sub is not None and frame.sub != expected_sub: + raise UnexpectedResponse( + f"expected SUB 0x{expected_sub:02X}, got 0x{frame.sub:02X}" + ) + return frame diff --git a/tests/test_micromate_protocol.py b/tests/test_micromate_protocol.py new file mode 100644 index 0000000..ccb0214 --- /dev/null +++ b/tests/test_micromate_protocol.py @@ -0,0 +1,402 @@ +"""Protocol-layer tests for the Micromate (series-4) live client. + +The load-bearing assertion in here is not "our parser understands the device" — +it is **"the bytes we put on the wire are the bytes THOR puts on the wire."** +Every request constant below is lifted from +``bridges/captures/9-24-26 - micromate2/`` (UM12947, firmware 11.0CB), so a +passing test means a real unit has already answered exactly that frame. + +Responses are replayed through a scripted transport. No hardware, no network. +""" +from __future__ import annotations + +import os +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from micromate import protocol as P +from micromate.framing import ETX, STX, checksum, stuff +from micromate.protocol import ( + ACK_DATA_LEN, + ChecksumError, + MicromateProtocol, + ShortRead, + UnexpectedResponse, + chunk_params, + key_lo_params, + key_params, + token_params, +) + +FLAGS_CB = 0xC5 + + +# ── Test doubles ────────────────────────────────────────────────────────────── + +class ScriptedTransport: + """Hands back queued responses; records every byte written.""" + + def __init__(self, responses: list[bytes] | None = None) -> None: + self.queue = list(responses or []) + self.written: list[bytes] = [] + self._connected = True + + # BaseTransport surface actually used by MicromateProtocol + def connect(self) -> None: + self._connected = True + + def disconnect(self) -> None: + self._connected = False + + def is_connected(self) -> bool: + return self._connected + + def write(self, data: bytes) -> None: + self.written.append(data) + + def read(self, n: int) -> bytes: + return self.queue.pop(0) if self.queue else b"" + + +def frame(rsp_sub: int, data: bytes, *, flags: int = FLAGS_CB, page: int = 0) -> bytes: + """Build a response frame the way a unit would.""" + payload = bytes([0x00, flags, rsp_sub, (page >> 8) & 0xFF, page & 0xFF]) + data + return bytes([STX]) + stuff(payload + bytes([checksum(payload)])) + bytes([ETX]) + + +def ack(rsp_sub: int) -> bytes: + return frame(rsp_sub, bytes(ACK_DATA_LEN)) + + +def proto(responses: list[bytes], **kw) -> tuple[MicromateProtocol, ScriptedTransport]: + t = ScriptedTransport(responses) + return MicromateProtocol(t, recv_timeout=0.5, **kw), t + + +# ── Captured THOR request frames ────────────────────────────────────────────── + +REQ = { + "poll": bytes.fromhex("41021010005b000030000000000000000000009b03"), + "serial": bytes.fromhex("41021010001500000a000000000000000000002f03"), + "state": bytes.fromhex("41021010004900ffff000000000000000000005703"), + "compliance": bytes.fromhex("41021010001a00ffff000000000000000000002803"), + "arm": bytes.fromhex("41021010009300ffff00000000000000000000a103"), + "setup_first": bytes.fromhex("41021010003f00ffff000000000000000000004d03"), + "setup_next": bytes.fromhex("41021010004000ffff000000000000000000004e03"), +} + +# The complete 0x5A sequence for event 055d4a81 (4,076 bytes → 4 chunks), as +# THOR sent it. Chunk 1's params hold a literal 0x04 and chunk 3's offset is +# the exact remainder. +REQ_CHUNKS_4A81 = [ + bytes.fromhex("41021010005a00100400055d4a810000000000009b03"), + bytes.fromhex("41021010005a0010040000001004000000000000007203"), + bytes.fromhex("41021010005a00100400000008000000000000007603"), + bytes.fromhex("41021010005a001003ec00000c000000000000006503"), +] +SIZE_4A81 = 4076 + + +# ── Params builders ─────────────────────────────────────────────────────────── + +def test_event_token_sits_at_params_7(): + """⚠ THOR sends 0xFE here; the protocol reference documents all-zero params. + + That reference entry describes our own browse probing, not THOR's. + """ + assert token_params() == bytes.fromhex("00000000000000fe0000") + + +def test_event_record_takes_the_full_key_at_params_4(): + assert key_params(bytes.fromhex("055d4a81")) == bytes.fromhex("00000000055d4a810000") + + +def test_monitor_log_takes_only_the_low_half_of_the_key(): + """⚠ Inferred from one key value -- see key_lo_params' docstring.""" + assert key_lo_params(bytes.fromhex("055d4a81")) == bytes.fromhex("0000000000004a810000") + + +def test_chunk_params_switch_from_key_to_byte_offset(): + key = bytes.fromhex("055d4a81") + assert chunk_params(key, 0) == bytes.fromhex("055d4a81000000000000") + assert chunk_params(key, 1024) == bytes.fromhex("00000400000000000000") + assert chunk_params(key, 13312) == bytes.fromhex("00003400000000000000") + + +@pytest.mark.parametrize("bad", [b"", b"\x01\x02\x03", b"\x01\x02\x03\x04\x05"]) +def test_params_builders_reject_a_wrong_length_key(bad): + for fn in (key_params, key_lo_params): + with pytest.raises(ValueError): + fn(bad) + + +# ── Each read emits the frame THOR emits ────────────────────────────────────── + +@pytest.mark.parametrize( + "name, method, rsp_sub, data_len", + [ + ("poll", "poll", 0xA4, 59), + ("serial", "read_serial", 0xEA, 21), + ("state", "read_state", 0xB6, 16), + ("compliance", "read_compliance_config", 0xE5, 2103), + ("setup_first", "read_first_setup", 0xC0, 266), + ("setup_next", "read_next_setup", 0xBF, 266), + ], +) +def test_reads_match_thors_wire_bytes(name, method, rsp_sub, data_len): + p, t = proto([frame(rsp_sub, bytes(data_len))]) + getattr(p, method)() + assert t.written == [REQ[name]] + + +def test_arm_event_matches_thors_wire_bytes(): + p, t = proto([ack(0x6C)]) + p.arm_event() + assert t.written == [REQ["arm"]] + + +def test_poll_is_the_only_read_with_a_non_ffff_offset_besides_serial(): + """Reads are single-step at 0xFFFF; POLL and SERIAL are the exceptions.""" + assert set(P._OFFSETS) == {P.SUB_POLL, P.SUB_SERIAL} + assert P._OFFSETS[P.SUB_POLL] == 0x0030 + assert P._OFFSETS[P.SUB_SERIAL] == 0x000A + + +# ── The chunk walk ──────────────────────────────────────────────────────────── + +def test_download_reproduces_thors_chunk_sequence_byte_for_byte(): + """The whole point of step 2. Four chunks, 4,076 bytes, THOR's exact frames.""" + payload = bytes(range(256)) * 16 # 4096 B, we use the first 4076 + payload = payload[:SIZE_4A81] + responses = [] + for i in range(4): + want = min(P.CHUNK_SIZE, SIZE_4A81 - i * P.CHUNK_SIZE) + body = payload[i * P.CHUNK_SIZE: i * P.CHUNK_SIZE + want] + responses.append(frame(0xA5, bytes(11) + body, page=want // 256)) + + p, t = proto(responses) + got = p.read_event_file(bytes.fromhex("055d4a81"), SIZE_4A81) + + assert t.written == REQ_CHUNKS_4A81 + assert got == payload + assert len(got) == SIZE_4A81 + + +@pytest.mark.parametrize( + "size, n_chunks, last_offset", + [ + (4076, 4, 0x03EC), (11032, 11, 0x0318), (11502, 12, 0x00EE), + (13424, 14, 0x0070), (8746, 9, 0x022A), (6092, 6, 0x03CC), + (1024, 1, 0x0400), (1, 1, 0x0001), (1025, 2, 0x0001), + ], +) +def test_chunk_count_and_final_offset(size, n_chunks, last_offset): + """The first six rows are the six bench events, with THOR's real offsets.""" + responses = [] + for i in range(n_chunks): + want = min(P.CHUNK_SIZE, size - i * P.CHUNK_SIZE) + responses.append(frame(0xA5, bytes(11) + bytes(want))) + + p, t = proto(responses) + p.read_event_file(bytes.fromhex("055d4a81"), size) + + assert len(t.written) == n_chunks + # offset is payload[4:5] of the request; recover it from the built frame + final = t.written[-1] + assert final[7:9] in ( + bytes([last_offset >> 8, last_offset & 0xFF]), + # a 0x02/0x03/0x04/0x10 high byte arrives escaped, shifting the pair + bytes([0x10, last_offset >> 8]), + ) + + +def test_a_short_chunk_raises_rather_than_truncating(): + """A silently short event is the failure mode this codebase keeps hitting.""" + p, _ = proto([frame(0xA5, bytes(11) + bytes(900))]) # asked for 1024 + with pytest.raises(ShortRead, match="asked for 1024 B, got 900"): + p.read_event_file(bytes.fromhex("055d4a81"), 1024) + + +def test_a_chunk_too_short_to_hold_its_header_raises(): + p, _ = proto([frame(0xA5, bytes(4))]) + with pytest.raises(ShortRead, match="too short to hold a chunk header"): + p.read_event_file(bytes.fromhex("055d4a81"), 1024) + + +def test_download_rejects_a_nonsense_size(): + p, _ = proto([]) + with pytest.raises(ValueError): + p.read_event_file(bytes.fromhex("055d4a81"), 0) + + +# ── The monitor-log walk ────────────────────────────────────────────────────── + +def test_monitor_log_walk_ends_on_a_short_response(): + """⚠ Not a keyed read -- the same request repeated, device-side cursor. + + Eight records then an 11-byte ack, which is what the capture shows. + """ + key = bytes.fromhex("055d4a81") + responses = [frame(0xF5, bytes(297)) for _ in range(8)] + [ack(0xF5)] + p, t = proto(responses) + + records = [] + while (rec := p.read_monitor_log_next(key)) is not None: + records.append(rec) + + assert len(records) == 8 + assert len(t.written) == 9 + assert len(set(t.written)) == 1, "every request in the walk is identical" + + +# ── Error handling ──────────────────────────────────────────────────────────── + +def test_a_bad_checksum_raises_by_default(): + """⚠ Deliberately stricter than the Series III sibling. + + That one logs and continues because its parser cannot always tell an + inner-frame delimiter from a checksum byte. The Micromate rule is exact on + 251/251 captured frames, so a mismatch here means something real. + """ + bad = bytearray(frame(0xA4, bytes(59))) + bad[-2] ^= 0xFF + p, _ = proto([bytes(bad)]) + with pytest.raises(ChecksumError, match="checksum mismatch"): + p.poll() + + +def test_a_bad_checksum_can_be_downgraded_for_field_diagnosis(): + bad = bytearray(frame(0xA4, bytes(59))) + bad[-2] ^= 0xFF + p, _ = proto([bytes(bad)], strict_checksums=False) + assert p.poll().sub == 0xA4 + + +def test_the_wrong_response_sub_raises(): + p, _ = proto([frame(0xE0, bytes(19))]) # 0xE0 answers 0x1F, not 0x1E + with pytest.raises(UnexpectedResponse, match="expected SUB 0xE1"): + p.read_event_first() + + +def test_a_timeout_reports_how_many_bytes_arrived(): + """Separates "nothing came back" from "bytes arrived but never framed". + + Those have completely different causes -- and on a Micromate the second one + is the signature of a modem forwarding a session it should not be. + """ + p, _ = proto([]) + with pytest.raises(P.TimeoutError, match="0 bytes were received"): + p.poll() + + unframed = b"\x02\x00\xc5\xa4garbage-no-terminator" + p2, _ = proto([unframed]) + with pytest.raises(P.TimeoutError, match=f"{len(unframed)} bytes were received"): + p2.poll() + + +def test_a_leftover_frame_is_discarded_rather_than_answered_with(): + """If a read returns two frames, the extra must not answer the NEXT request. + + Every exchange resets the parser before sending, so anything already + buffered is treated as stale. Delivering it would be the worse failure: + `expected_sub` happens to catch a mismatched SUB, but a same-SUB leftover + would sail through and return data for the wrong key. + """ + # Both frames arrive while answering arm_event(); the 0xE1 is left over. + p, _ = proto([ack(0x6C) + frame(0xE1, b"\xaa" * 19)]) + p.arm_event() + + # The next request gets no bytes of its own, so it must time out rather + # than hand back the stale 0xE1. + with pytest.raises(P.TimeoutError): + p.read_event_first() + + +# ── Against the real capture, when it happens to be present ─────────────────── + +_CAPTURES = ( + Path(__file__).resolve().parents[1] + / "bridges" / "captures" / "9-24-26 - micromate2" +) +_DOWNLOAD = "raw_bw_20260925_011403_Download_events_then_delete_1_event.bin" + + +@pytest.mark.skipif( + not (_CAPTURES / _DOWNLOAD).is_file(), + reason="capture is gitignored; present only on a dev box", +) +def test_every_captured_download_frame_is_one_we_would_have_sent(): + """Replay the real session: for each event, assert our chunk walk emits + exactly the frames THOR emitted -- all 50-odd of them, six events.""" + from micromate.framing import ACK, DLE + + def destuffed_frames(blob: bytes, is_req: bool): + i, n = 0, len(blob) + while i < n: + if is_req: + if not (blob[i] == ACK and i + 1 < n and blob[i + 1] == STX): + i += 1 + continue + j = i + 2 + else: + if blob[i] != STX: + i += 1 + continue + j = i + 1 + out = bytearray() + while j < n: + if blob[j] == DLE and j + 1 < n: + out.append(blob[j + 1]) + j += 2 + continue + if blob[j] == ETX: + break + out.append(blob[j]) + j += 1 + if len(out) >= 6: + yield blob[i:j + 1], bytes(out[:-1]) + i = j + 1 + + bw = list(destuffed_frames((_CAPTURES / _DOWNLOAD).read_bytes(), True)) + s3 = list( + destuffed_frames( + (_CAPTURES / _DOWNLOAD.replace("raw_bw", "raw_s3")).read_bytes(), False + ) + ) + + # Group THOR's 0x5A frames per event, taking each event's key+size from the + # 1E/1F that preceded them. + events, cur = [], None + for (wire, req), (_, rsp) in zip(bw, s3): + sub, data = req[2], rsp[5:] + if sub in (0x1E, 0x1F) and len(data) >= 19: + cur = {"key": data[11:15], "size": int.from_bytes(data[15:19], "big"), + "reqs": [], "rsps": []} + if cur["size"]: + events.append(cur) + elif sub == 0x5A and cur is not None: + cur["reqs"].append(wire) + cur["rsps"].append(rsp) + + # The capture walks the chain twice (it deletes an event on the second + # pass), so some 1E/1F hits carry a size but no download behind them. + events = [e for e in events if e["reqs"]] + assert len(events) == 6, f"expected 6 downloaded events, found {len(events)}" + + total = 0 + for e in events: + p, t = proto([bytes([STX]) + stuff(r + bytes([checksum(r)])) + bytes([ETX]) + for r in e["rsps"]]) + got = p.read_event_file(e["key"], e["size"]) + assert t.written == e["reqs"], ( + f"event {e['key'].hex()}: our {len(t.written)} frames differ from " + f"THOR's {len(e['reqs'])}" + ) + assert len(got) == e["size"] + total += len(e["reqs"]) + + assert total == 56, f"expected 56 download frames across the 6 events, saw {total}"