# Spec — a live client for Series IV (Micromate) Drafted 2026-09-26, ahead of implementation. The protocol work is finished; this is the plan for turning `docs/micromate_protocol_reference.md` into code SFM can run. **Read that document first.** Everything here assumes it, and every constant below is sourced from it rather than restated with justification. --- ## Goal and scope `micromate/` is codec-only today — `idf_file.py`, `models.py`, the report writers. There is no way to talk to a unit. This adds the live half, mirroring `minimateplus/`. **In scope, first pass:** - connect over TCP (a field modem) or serial/USB (a bench unit) - identify a unit, read its state, clock, memory and setups - walk the event chain and download events - return `Event` objects the existing codec already understands **Explicitly out of scope, first pass:** - ⚠ **Any write.** Setups, schedules, call-home config, monitoring start/stop, and per-event delete are all mapped, and none of them will be implemented here. **No command has ever been originated against a unit by this project** — every write observed was performed by THOR while we recorded. Keeping that true through the read client is deliberate: it means the first thing we ever send to a customer's instrument is a decision someone made on purpose, not a side effect of a client that happened to grow a method. - the inbound call-home session — still the one protocol unknown --- ## Layout ``` micromate/ framing.py NEW frame building, response parsing, checksum protocol.py NEW one method per wire command, returns raw payloads client.py NEW high-level API, returns models idf_file.py (existing — decodes what 0x5A returns, unchanged) models.py (existing — extend, do not fork) ``` **Transport is reused, not rewritten.** `minimateplus/transport.py` is byte-level and protocol-agnostic — `BaseTransport`, `SerialTransport`, `TcpTransport`, plus `read_until_idle()` which already handles the RV50/RV55 habit of emitting `\r\nRING\r\n\r\nCONNECT\r\n` to a caller. Import it. ⚠ Do **not** import `minimateplus.framing`. The two framings differ in ways that look small and are not, and a shared module would accumulate `if series ==` branches until neither case is readable. --- ## `micromate/framing.py` — ✅ BUILT 2026-09-27 Implemented, with `tests/test_micromate_framing.py` (31 tests, all passing). **Two things in this section as originally drafted were wrong**, and both were caught by measuring against the captures before writing code rather than after. They are left in place below, struck through, because both are the kind of mistake that would be made again. ### Requests ~~Series IV accepts Series III request frames unmodified. The simplest correct implementation re-exports the builder rather than duplicating it:~~ ```python from minimateplus.framing import build_bw_frame # ✗ WRONG — 161/218 ``` ⚠ **`build_bw_frame` reproduces only 161 of Thor's 218 captured read frames.** The payload layout is identical; the stuffing is not. A Micromate escapes **four** byte values — `0x02`, `0x03`, `0x04`, `0x10` — where Series III escapes one. The frames this breaks are **every `SUB 0x5A` download** (`offset = 0x0400` → a literal `0x04` in `offset_hi`) and the scheduler enable. An unescaped `0x03`/`0x04` terminates the frame early, so the unit just does not answer. `build_request()` with the correct escape set reproduces **218/218**. ✅ **Settled: `0x10` inside params needs no special handling.** The planned `NotImplementedError` guard is unnecessary — Thor sends `params = 00 00 10 00 …` on `SUB 0x5A` in five captured frames and the wire carries an ordinary doubled `10 10`. One rule covers the whole payload. ### Responses — where Series III's parser cannot follow | | Series III | Micromate | |---|---|---| | frame start | `DLE STX` | **bare `STX`** | | `payload[1]` | `0x10` | `0xC5` (Blastware fw) / `0x03` (Thor fw) | | destuffing | `DLE+ETX` kept as literal inner-frame data | **`10 XX` → `XX`, uniformly** | The first row is why `S3FrameParser` returns nothing at all on Series IV traffic: it scans for `DLE+STX`, which never appears. The third is a genuine **simplification** — no inner-frame carve-out. Validated by checksum across every capture in `bridges/captures/9-24-26 - micromate2/`: four candidate destuffing rules were tried, and only this one makes all frames validate. ### Checksum ```python def checksum(payload: bytes) -> int: return sum(payload) & 0xFF # payload already de-stuffed ``` ~~The DLE-aware variant, same as Series III's `5A` and write frames.~~ ⚠ **Plain SUM8, not the DLE-aware variant** — 251/251 both directions. The DLE-aware form is the right answer paired with *Series III* de-stuffing, which leaves an escaped byte in the payload as two bytes. De-stuffing `10 XX → XX` already removes the `0x10`, so excluding it again subtracts the correction twice, and the result disagrees with the wire on **55 of 251** captured responses — every frame holding a literal `0x10`. `scratch/mm_frame_parse.py` shipped with exactly that pairing. It looked clean only because it accepts a frame matching *either* rule, so it labelled those 55 `SUM8` and never flagged one bad. "Zero bad checksums" was true and carried no information. Fixed there too. ### ⚠ The SUB byte can be escaped When a SUB's value is `0x02`, `0x03`, `0x04` or `0x10` it arrives as `10 XX`. Reading it positionally without destuffing reports `0x10`. This bit once already — `SUB 0x02` was logged as `SUB_10` for an afternoon. Destuff first, then index. ### Response shape ```python @dataclass class MicromateFrame: sub: int # response SUB; request = 0xFF - sub flags: int # 0xC5 Blastware line, 0x03 Thor line page_hi: int page_lo: int data: bytes # payload[5:], checksum stripped checksum_valid: bool @property def request_sub(self) -> int: # 0xFF - sub @property def page_key(self) -> int: # uint16 BE at payload[3:5] @property def firmware_line(self) -> str: # "blastware" | "thor" | "unknown" @property def probe_length(self) -> int | None: # uint16 BE at data[3:5] (= payload[8:10]) ``` ⚠ **`probe_length` is a uint16 BE.** Read as a single byte it under-reads `SUB 0x1A` by 47x — 44 against a true 2092. This is the single most expensive mistake available in this protocol and it has already been made once. Renamed from `declared_length`, because it is **only meaningful in the reply to an `offset = 0` probe** — and Thor never probes. Across all 251 captured responses the field reads 0 or a page count, never a length, precisely because that session is single-step reads throughout. `page_key` is the field that carries meaning there. The only genuine probe reply we hold is the POLL one preserved in `scratch/fake_unit.py`. `MicromateFrameParser` mirrors `S3FrameParser`: `feed(bytes) -> list[frame]`, accumulates in `.frames`, `reset()`, and keeps the `bytes_fed` counter (it is what distinguishes "no bytes at all" from "bytes but no complete frame" on a timeout, and that distinction earned its keep during the Series III work). --- ## `micromate/protocol.py` — ✅ BUILT 2026-09-27 Implemented, with `tests/test_micromate_protocol.py` (35 tests). Reads only; nothing here writes, erases or changes monitoring state. The tests replay Thor's captured responses through a scripted transport and assert **the bytes we emit are the bytes Thor emits** — including a full replay of the six-event download session, all 56 `0x5A` frames byte-for-byte. That is a stronger guarantee than "our parser understands the device": a passing test means a real unit has already answered exactly that frame. One method per command, returning raw payload bytes. No interpretation — that belongs in `client.py`. **Reads use `offset = 0xFFFF`** and return the whole block in one response; Series III's two-step probe/data dance is unnecessary. `POLL` is the exception, taking its data length. Per-command offsets, all observed: | command | SUB | rsp | offset | returns | |---|---|---|---|---| | poll | `0x5B` | `0xA4` | `0x0030` | device string, model | | serial | `0x15` | `0xEA` | `0x000A` | `UM12947` | | device info | `0x01` | `0xFE` | `0xFFFF` | firmware, calibration | | state | `0x49` | `0xB6` | `0xFFFF` | `data[11]`: non-zero = monitoring | | monitor status | `0x1C` | `0xE3` | `0xFFFF` | flag, **device clock**, battery, memory | | storage range | `0x06` | `0xF9` | `0xFFFF` | event storage extent | | active setup name | `0x41` | `0xBE` | `0xFFFF` | `TEST1.mmb` | | first setup | `0x3F` | `0xC0` | `0xFFFF` | setup-list walk head | | next setup | `0x40` | `0xBF` | `0xFFFF` | …until an empty name | | compliance config | `0x1A` | `0xE5` | `0xFFFF` | ~2103 B setup block | | call-home config | `0x2C` | `0xD3` | `0xFFFF` | 137 B | | arm event | `0x93` | `0x6C` | — | before every event | | first event | `0x1E` | `0xE1` | `0xFFFF` | key + size | | next event | `0x1F` | `0xE0` | `0xFFFF` | key + size | | event record | `0x0C` | `0xF3` | `0xFFFF` | 221 B — project, location, peaks | | ~~event header~~ **monitor log** | `0x0A` | `0xF5` | `0xFFFF` | ⚠ 297 B, a **walk** — see below | | bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim**, 1024 B at a time | ⚠ **Corrected 2026-09-27, from Thor's frames.** Three rows of the table above were wrong or incomplete, and the last one is a different command than labelled: - **`0x0A` is the monitor-log walk**, not a keyed "30 B list record" read. The *same request repeated* returns successive 297-byte records — serial, mode, thresholds — until an 11-byte ack ends the list. The device holds the cursor; nothing in the request selects a record. Series III reaches this data through a record-type discriminator on its event chain; here it has its own cursor and the event chain never sees it. - **`0x1E`/`0x1F` carry token `0xFE` at `params[7]`.** The reference documents all-zero params (our own probing, which also worked). Thor's form is the one with mileage. - **`0x01` has no Thor frame behind it** — it is never read in any captured session. Its `0xFFFF` comes from our probes. And one useful negative: **no `SESSION_RESET` (`41 03`)**. Series III needs that 2-byte signal or a monitoring unit will not answer `POLL` over TCP. Thor never sends it — zero occurrences across 8 sessions, including 40 frames exchanged with a unit that *was* monitoring. ⚠ **`SUB 0x1C` is 4 bytes longer on the Thor firmware line** (`0x30` vs `0x2C`). Parse **forward** from `declared_length`, never backward from the end — Series III reads battery and memory from the end of that block, and doing so on a BD unit yields a battery voltage of **577.92 V**. ⚠ **Test the monitoring flag for non-zero**, never against a constant. It has read both `0x0E` and `0x0C` while monitoring. ### `0x5A` — a bounded chunk loop, and much simpler than Series III ⚠ **Corrected 2026-09-27.** This section said "no chunk loop — one request returns the whole event", with `offset_word = 0x1000 + 2 * ceil(size / 512)`. That describes our own 2026-09-23 probes, which set `offset_hi = 0x10`. **Thor chunks**, and Thor's form is the one verified from bytes on disk: ```python n = ceil(size / 1024) # size from the chain walk for i in range(n): offset = min(1024, size - 1024 * i) # a BYTE COUNT params = key4 + bytes(6) if i == 0 else bytes(2) + pack(">H", 1024*i) + bytes(6) file_bytes += response.data[11:] # response data is exactly offset + 11 ``` Verified on all six bench events (4,076 → 13,424 B): `sum(offsets) == size` exactly, with the predicted chunk count and final offset every time. Still no arming ritual for `0x5A` itself, no `STRT` end-offset parsing and no `TERM` frame — the simplification the original claim celebrated is real, it just is not single-shot. (`SUB 0x93` arms the *chain walk*, before `1E`/`1F`, not the download.) The concatenated payload **is** the `.IDFW` file, byte for byte — so it feeds `micromate.idf_file.read_idf_file()` and `/db/import/idf_file` unchanged. ⚠ Do not port the Series III `5A` walk. Its address arithmetic caused a 5x over-read and a `> 64 KB` page-boundary bug that is *still open* on the Series III side. None of that applies here: the chunk index is a byte offset into the file, bounded by a size the device told us, and it cannot run past the event. ⚠ **`assert sum(len(chunk) - 11 for chunk in chunks) == size`.** A silently short event is the failure mode this project has been bitten by repeatedly on the Series III side, and here the check is free because the size is known up front. --- ## `micromate/client.py` ```python class MicromateClient: def __init__(self, transport: BaseTransport): ... def open(self) / close(self) / is_open(self) # identity and state def connect(self) -> DeviceInfo # poll → serial → device info → state def get_state(self) -> UnitState # monitoring?, clock, battery, memory def get_active_setup(self) -> str def list_setups(self) -> list[str] # 0x3F → 0x40… until empty # events def list_events(self) -> list[EventRef] # 0x93 → 0x1E → 0x1F… (key + size) def download_event(self, ref) -> bytes # raw .IDFW/.IDFH def get_event(self, ref) -> Event # download + decode via idf_file ``` `connect()` should mirror THOR's preamble (`POLL → SERIAL → 0x49 → POLL`) — ⚠ but note the reference records that **whether the unit requires it is untested**. Do it because it is known-good, not because it is known-necessary, and say so in the docstring. `list_events()` returns the key *and* the size, because `download_event()` needs the size to compute its offset word. --- ## Tests **Offline, from captured bytes — no hardware.** This is the part worth doing first, because it can be fully verified tonight's-captures-style before any unit is involved. ``` tests/test_micromate_framing.py ``` ⚠ `bridges/captures/` and `tests/fixtures/` are both gitignored, so tests must not depend on files being present. **Embed the frames as hex constants** — they are 19–138 bytes each. ✅ Done; what actually landed: | case | source | why | |---|---|---| | POLL probe reply, 19 B | captured (via `fake_unit.py`) | shortest valid frame; the only real probe reply we hold | | `0x5A` chunk, 138 B | captured | holds literal `0x10` **and** literal `0x41` — the checksum case, and proves ACK is not escaped | | `0x49` state reply, 25 B | captured | a literal `0x02`, escaped | | `0x48` file reply, 24 B | captured | escaped `0x04` in `data[0]` — one byte late without destuffing | | 8 Thor request frames | captured | byte-for-byte against `build_request()`, incl. both `0x5A` forms and the scheduler enable | | `probe_length = 0x082C` | **synthesised** | no probe reply for `0x1A` exists on disk — the 9-24-26 session never probes | | Thor-line reply, `flags = 0x03` | **synthesised** | no 11.0BD capture is in the repo; built by flipping one byte of the real POLL reply | | escaped checksum byte | **synthesised** | the shortest real one is 1,070 B, too long to embed for one assertion | | truncated / corrupt / split-across-feeds | derived | parser must return nothing, flag rather than swallow, and survive any split point | ⚠ Synthesised frames are marked `SYNTH_`-style in the test and each says what it stands in for and why no capture was available. Do not let that set grow quietly: the `flags = 0x03` case in particular is the **only** coverage of half the fleet, and it deserves a real 11.0BD capture the next time UM20147 is on a bench. Two corpus-backed tests run when the captures happen to be on the box and skip cleanly otherwise: **251 response frames parse with zero bad checksums**, and **`build_request()` reproduces 218/218 read frames**. The second is the test that would have caught the escape-set error, so it is worth the skip marker. ⚠ Do **not** assert against `scratch/mm_frame_parse.py`'s output as the original plan proposed. That script accepts either checksum rule and is wrong about which one is right — using it as an oracle would have pinned the bug. **Live, second:** against the bench unit on mint-mac via `mm_link.py`. `connect()`, `list_setups()` (should return the 23 known names), `list_events()`, then `download_event()` and assert the bytes decode and match a `/db/import/idf_file` ingest of the same event. --- ## Order of work 1. ✅ `framing.py` + its tests — **done 2026-09-27**, 31 tests, offline 2. ✅ `protocol.py` + its tests — **done 2026-09-27**, 35 tests, offline 3. `client.py` — `connect()`, `get_state()`, `list_setups()` 4. the event chain and `download_event()` 5. decode end-to-end and compare against a store event Steps 1–2 need no hardware at all. **Worth carrying forward.** Both steps began by measuring against the captures rather than trusting this document, and both found errors in it — three in the framing rules (the escape set, 26% of frames; the checksum, 22%; the `0x5A` chunk model) and three more in the command table (`0x0A`'s meaning, the `1E`/`1F` token, `0x01`'s provenance). All six fail quietly. The captures are on disk and a measure-then-write loop costs about two minutes per rule, so keep doing it for `client.py`'s field offsets — and treat this spec as a plan, not a source. --- ## Open questions to settle while implementing - ~~**Request param stuffing**~~ — ✅ settled 2026-09-27; no special handling. - **Is the single-request `0x5A` form real?** Our 2026-09-23 probes set `offset_hi = 0x10` and appeared to get a whole 11 KB event back, where Thor chunks at 1024 B. Plausibly a distinct streaming mode that returns several frames. One bench test settles it; implement Thor's form regardless. - **Is THOR's preamble required?** Try one command cold and find out; it is a two-minute test with the bench unit and it removes a ritual if unnecessary. - **`Event` model fit** — Series IV carries fields Series III lacks (setup file name, `LMic`/`SMic` channels). Extend `micromate/models.py`; do not fork the shared `Event`. - **Which `0x0C` fields to trust.** The peak float there runs 2–5% above `max(T,V,L)` and is **not** the vector sum; its offset was inferred, not established. The reference marks it do-not-rely-on — prefer decoded samples.