diff --git a/docs/micromate_client_spec.md b/docs/micromate_client_spec.md new file mode 100644 index 0000000..7d07959 --- /dev/null +++ b/docs/micromate_client_spec.md @@ -0,0 +1,291 @@ +# Spec — a live client for Series IV (Micromate) + +Drafted 2026-09-26, ahead of implementation. The protocol work is finished; this +is the plan for turning `docs/micromate_protocol_reference.md` into code SFM can +run. + +**Read that document first.** Everything here assumes it, and every constant +below is sourced from it rather than restated with justification. + +--- + +## Goal and scope + +`micromate/` is codec-only today — `idf_file.py`, `models.py`, the report +writers. There is no way to talk to a unit. This adds the live half, mirroring +`minimateplus/`. + +**In scope, first pass:** + +- connect over TCP (a field modem) or serial/USB (a bench unit) +- identify a unit, read its state, clock, memory and setups +- walk the event chain and download events +- return `Event` objects the existing codec already understands + +**Explicitly out of scope, first pass:** + +- ⚠ **Any write.** Setups, schedules, call-home config, monitoring start/stop, + and per-event delete are all mapped, and none of them will be implemented + here. **No command has ever been originated against a unit by this project** + — every write observed was performed by THOR while we recorded. Keeping that + true through the read client is deliberate: it means the first thing we ever + send to a customer's instrument is a decision someone made on purpose, not a + side effect of a client that happened to grow a method. +- the inbound call-home session — still the one protocol unknown + +--- + +## Layout + +``` +micromate/ + framing.py NEW frame building, response parsing, checksum + protocol.py NEW one method per wire command, returns raw payloads + client.py NEW high-level API, returns models + idf_file.py (existing — decodes what 0x5A returns, unchanged) + models.py (existing — extend, do not fork) +``` + +**Transport is reused, not rewritten.** `minimateplus/transport.py` is +byte-level and protocol-agnostic — `BaseTransport`, `SerialTransport`, +`TcpTransport`, plus `read_until_idle()` which already handles the RV50/RV55 +habit of emitting `\r\nRING\r\n\r\nCONNECT\r\n` to a caller. Import it. + +⚠ Do **not** import `minimateplus.framing`. The two framings differ in ways +that look small and are not, and a shared module would accumulate `if series ==` +branches until neither case is readable. + +--- + +## `micromate/framing.py` + +### Requests + +Series IV accepts Series III request frames unmodified. The simplest correct +implementation re-exports the builder rather than duplicating it: + +```python +from minimateplus.framing import build_bw_frame # requests are identical +``` + +⚠ One open question, flagged in the protocol reference and **not** settled: +whether `0x10` bytes inside request *params* need stuffing. No probe we sent +carried one. Until it is settled, assert on it rather than guessing: + +```python +def build_request(sub: int, offset: int = 0, params: bytes = bytes(10)) -> bytes: + if 0x10 in params: + raise NotImplementedError( + "params containing 0x10 — stuffing rule unconfirmed; see " + "micromate_protocol_reference.md, 'Untested and unsafe-until-agreed'" + ) + return build_bw_frame(sub, offset, params) +``` + +That turns an unknown into a loud failure instead of a corrupt frame. + +### Responses — where Series III's parser cannot follow + +| | Series III | Micromate | +|---|---|---| +| frame start | `DLE STX` | **bare `STX`** | +| `payload[1]` | `0x10` | `0xC5` (Blastware fw) / `0x03` (Thor fw) | +| destuffing | `DLE+ETX` kept as literal inner-frame data | **`10 XX` → `XX`, uniformly** | + +The first row is why `S3FrameParser` returns nothing at all on Series IV traffic: +it scans for `DLE+STX`, which never appears. + +The third is a genuine **simplification** — no inner-frame carve-out. Validated +by checksum across every capture in `bridges/captures/9-24-26 - micromate2/`: +four candidate destuffing rules were tried, and only this one makes all frames +validate. + +### Checksum + +```python +def checksum(payload: bytes) -> int: + return sum(b for b in payload if b != 0x10) & 0xFF +``` + +The DLE-aware variant, same as Series III's `5A` and write frames — not the plain +SUM8 of ordinary Series III reads. + +### ⚠ The SUB byte can be escaped + +When a SUB's value is `0x02`, `0x03`, `0x04` or `0x10` it arrives as `10 XX`. +Reading it positionally without destuffing reports `0x10`. This bit once +already — `SUB 0x02` was logged as `SUB_10` for an afternoon. Destuff first, +then index. + +### Response shape + +```python +@dataclass +class MicromateFrame: + sub: int # response SUB; request = 0xFF - sub + flags: int # 0xC5 Blastware line, 0x03 Thor line + page_hi: int + page_lo: int + data: bytes # payload[5:], checksum stripped + checksum_valid: bool + + @property + def firmware_line(self) -> str: # "blastware" | "thor" | "unknown" + @property + def declared_length(self) -> int: # uint16 BE at data[3:5] (= payload[8:10]) +``` + +⚠ **`declared_length` is a uint16 BE.** Read as a single byte it under-reads +`SUB 0x1A` by 47x — 44 against a true 2092. This is the single most expensive +mistake available in this protocol and it has already been made once. + +`MicromateFrameParser` mirrors `S3FrameParser`: `feed(bytes) -> list[frame]`, +accumulates in `.frames`, `reset()`, and keeps the `bytes_fed` counter (it is +what distinguishes "no bytes at all" from "bytes but no complete frame" on a +timeout, and that distinction earned its keep during the Series III work). + +--- + +## `micromate/protocol.py` + +One method per command, returning raw payload bytes. No interpretation — that +belongs in `client.py`. + +**Reads use `offset = 0xFFFF`** and return the whole block in one response; +Series III's two-step probe/data dance is unnecessary. `POLL` is the exception, +taking its data length. Per-command offsets, all observed: + +| command | SUB | rsp | offset | returns | +|---|---|---|---|---| +| poll | `0x5B` | `0xA4` | `0x0030` | device string, model | +| serial | `0x15` | `0xEA` | `0x000A` | `UM12947` | +| device info | `0x01` | `0xFE` | `0xFFFF` | firmware, calibration | +| state | `0x49` | `0xB6` | `0xFFFF` | `data[11]`: non-zero = monitoring | +| monitor status | `0x1C` | `0xE3` | `0xFFFF` | flag, **device clock**, battery, memory | +| storage range | `0x06` | `0xF9` | `0xFFFF` | event storage extent | +| active setup name | `0x41` | `0xBE` | `0xFFFF` | `TEST1.mmb` | +| first setup | `0x3F` | `0xC0` | `0xFFFF` | setup-list walk head | +| next setup | `0x40` | `0xBF` | `0xFFFF` | …until an empty name | +| compliance config | `0x1A` | `0xE5` | `0xFFFF` | ~2103 B setup block | +| call-home config | `0x2C` | `0xD3` | `0xFFFF` | 137 B | +| arm event | `0x93` | `0x6C` | — | before every event | +| first event | `0x1E` | `0xE1` | `0xFFFF` | key + size | +| next event | `0x1F` | `0xE0` | `0xFFFF` | key + size | +| event record | `0x0C` | `0xF3` | `0xFFFF` | 210 B — project, location, peaks | +| event header | `0x0A` | `0xF5` | `0xFFFF` | 30 B list record | +| bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim** | + +⚠ **`SUB 0x1C` is 4 bytes longer on the Thor firmware line** (`0x30` vs `0x2C`). +Parse **forward** from `declared_length`, never backward from the end — Series +III reads battery and memory from the end of that block, and doing so on a BD +unit yields a battery voltage of **577.92 V**. + +⚠ **Test the monitoring flag for non-zero**, never against a constant. It has +read both `0x0E` and `0x0C` while monitoring. + +### `0x5A` — simpler than Series III, deliberately + +No arming ritual, no chunk loop, no `STRT` end-offset parsing, no `TERM` frame. +One request returns the whole event: + +```python +offset_word = 0x1000 + 2 * ceil(size / 512) # size from the chain walk +``` + +The payload **is** the `.IDFW` file, byte for byte — so it feeds +`micromate.idf_file.read_idf_file()` and `/db/import/idf_file` unchanged. + +⚠ Do not port the Series III `5A` walk. Its address arithmetic caused a 5x +over-read and a `> 64 KB` page-boundary bug that is *still open* on the Series +III side. None of that applies here. + +--- + +## `micromate/client.py` + +```python +class MicromateClient: + def __init__(self, transport: BaseTransport): ... + def open(self) / close(self) / is_open(self) + + # identity and state + def connect(self) -> DeviceInfo # poll → serial → device info → state + def get_state(self) -> UnitState # monitoring?, clock, battery, memory + def get_active_setup(self) -> str + def list_setups(self) -> list[str] # 0x3F → 0x40… until empty + + # events + def list_events(self) -> list[EventRef] # 0x93 → 0x1E → 0x1F… (key + size) + def download_event(self, ref) -> bytes # raw .IDFW/.IDFH + def get_event(self, ref) -> Event # download + decode via idf_file +``` + +`connect()` should mirror THOR's preamble (`POLL → SERIAL → 0x49 → POLL`) — +⚠ but note the reference records that **whether the unit requires it is +untested**. Do it because it is known-good, not because it is known-necessary, +and say so in the docstring. + +`list_events()` returns the key *and* the size, because `download_event()` needs +the size to compute its offset word. + +--- + +## Tests + +**Offline, from captured bytes — no hardware.** This is the part worth doing +first, because it can be fully verified tonight's-captures-style before any unit +is involved. + +``` +tests/test_micromate_framing.py +``` + +⚠ `bridges/captures/` and `tests/fixtures/` are both gitignored, so tests must +not depend on files being present. **Embed the frames as hex constants** — they +are 16–68 bytes each, and a handful covers every case: + +| case | why | +|---|---| +| POLL probe reply, 19 B | shortest valid frame | +| POLL data reply, 68 B | contains a literal `0x10` — only the DLE-aware checksum matches | +| `0x1A` response, 2108 B | exercises `declared_length` as a true uint16 (`0x082C`) | +| a Thor-line reply, `flags = 0x03` | `0x03` is ETX; proves destuffing before framing | +| a frame whose SUB is `0x02` | arrives as `10 02`; proves destuff-then-index | +| a truncated frame | parser must return nothing, not a bad frame | +| a corrupted checksum | `checksum_valid == False`, frame still returned | + +Then a round-trip assertion: feed a whole captured session through the parser and +assert the frame count and every SUB, against `scratch/mm_frame_parse.py`'s +output — which is already known good, having parsed 24, 38 and 40-frame sessions +with zero bad checksums. + +**Live, second:** against the bench unit on mint-mac via `mm_link.py`. +`connect()`, `list_setups()` (should return the 23 known names), `list_events()`, +then `download_event()` and assert the bytes decode and match a +`/db/import/idf_file` ingest of the same event. + +--- + +## Order of work + +1. `framing.py` + its tests — offline, verifiable immediately +2. `protocol.py` — reads only, one method per row of the table above +3. `client.py` — `connect()`, `get_state()`, `list_setups()` +4. the event chain and `download_event()` +5. decode end-to-end and compare against a store event + +Steps 1–2 need no hardware at all. + +--- + +## Open questions to settle while implementing + +- **Request param stuffing** — raise `NotImplementedError` rather than guess. +- **Is THOR's preamble required?** Try one command cold and find out; it is a + two-minute test with the bench unit and it removes a ritual if unnecessary. +- **`Event` model fit** — Series IV carries fields Series III lacks (setup file + name, `LMic`/`SMic` channels). Extend `micromate/models.py`; do not fork the + shared `Event`. +- **Which `0x0C` fields to trust.** The peak float there runs 2–5% above + `max(T,V,L)` and is **not** the vector sum; its offset was inferred, not + established. The reference marks it do-not-rely-on — prefer decoded samples.