From 0659543a156edb8b114284b702202c97e05be0eb Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 23:27:21 -0400 Subject: [PATCH 01/16] docs(series4): the 2026-09-22 failure is accounted for -- the timeline was wrong An earlier draft said the Prolific-cable finding did not fit the timeline, since the unit was described as working and then degrading, which a cable that never enumerates cannot do. The timeline was what needed correcting. From the operator: the initial setup was ONE modem and ONE cable, and it worked. Two Micromates then had to be deployed, so a second cable was fetched to run both at once -- and only one of the two ever worked. The other never talked over its modem at any point, was never deployed, and a MiniMate Plus went out in its place. That Micromate came back to the bench, and the cable on it is a Benfei (Prolific PL2303). So nothing degraded. The "worked at first" was the single-cable setup before the split. One unit working and one not, configured side by side, is the signature of a mixed cable supply rather than a unit fault. Still worth confirming whether that cable travelled home with the unit or was picked up at the bench; if it travelled with it there is no inference left. The FTDI cable arriving 2026-09-26 settles it either way -- same unit, same modem, one variable. Records a corollary: "just deploy a Series III instead" works partly because a MiniMate Plus has a DB-9 port directly on the unit, with no USB-to-serial adapter in the path and therefore no chipset to get wrong. That workaround has been routing around this exact failure mode. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 42 ++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 11 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index b9e8610..396dd70 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -2119,20 +2119,40 @@ FTDI is VID `0403`, Prolific is `067b`. Note also that counterfeit FTDI chips are common in cheap cables — they carry FTDI's VID but may not behave like one, and an embedded host with a single driver is far less forgiving than Linux. -### Does this explain the 2026-09-22 field outage? +### ✅ This accounts for the 2026-09-22 failure -⚠ **Not on the timeline as reported.** A Prolific cable does not fail -*gradually* — it never enumerates, so the modem port never works at all. The -unit was described as working at first and degrading. +An earlier draft of this section said the timeline did not fit, because the unit +was described as *working at first and then degrading* — which a Prolific cable +cannot do, since it never enumerates at all. -There is one story where it fits: if the initial success was over a **different -path** — the USB **PC** port, or a bench test before deployment — that would work -regardless of which serial cable was attached. The modem path would then have -been broken from the moment it was deployed, and "it worked and then stopped" -would be a recollection conflating two connection types days after the fact. +**The timeline was the thing that needed correcting, not the finding.** From the +operator, 2026-09-25: -Plausible, unverified, and recorded as such. Identifying that unit's cable would -settle it. +1. Initial setup — **one** modem, **one** cable. Worked: configs sent, status + read, no trouble. +2. Two Micromates then had to be deployed, so a **second cable** was fetched to + run both at once. +3. **Only one of the two ever worked.** The other would not talk over its modem + at any point. +4. That unit was never deployed — a **MiniMate Plus was put out in its place**, + and the Micromate came back to the bench. +5. The cable on that bench unit is a **Benfei (Prolific PL2303)**. + +So nothing degraded. The "worked at first" was the original single-cable setup; +the failure began when the pair was split across two cables and one of them was +the wrong chipset. **One working and one not, set up side by side, is the +signature of a mixed cable supply** — not of a unit fault. + +⚠ Still worth confirming: whether that cable travelled home *with* the unit, or +was picked up separately at the bench. If it travelled with it, there is no +inference left. Either way the FTDI cable arriving 2026-09-26 settles it — same +unit, same modem, one variable. + +**Corollary worth knowing.** The standing workaround — *"just deploy a Series III +instead"* — works partly because a **MiniMate Plus has a DB-9 port directly on the +unit.** No USB-to-serial adapter anywhere in the path, so there is no chipset to +get wrong. That workaround has been quietly routing around this exact failure +mode. ### How this was isolated — the method is reusable From 5fa181fb7901a56a9eece0b4f3faeea054476fec Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 23:27:39 -0400 Subject: [PATCH 02/16] docs(series4): confirmed -- the cable came home with the failed unit Removes the last caveat. The Benfei (Prolific PL2303) on the bench is physically the cable from the failed office setup, not a substitute picked up later. The chain has no inference left: that cable is a PL2303; the Micromate USB host has no Prolific driver in either firmware line; a laptop on the same cable and modem round-trips perfectly; the Micromate on it never answered. The FTDI cable arriving 2026-09-26 is now a confirmation rather than a test. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 396dd70..2ceb32d 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -2143,10 +2143,18 @@ the failure began when the pair was split across two cables and one of them was the wrong chipset. **One working and one not, set up side by side, is the signature of a mixed cable supply** — not of a unit fault. -⚠ Still worth confirming: whether that cable travelled home *with* the unit, or -was picked up separately at the bench. If it travelled with it, there is no -inference left. Either way the FTDI cable arriving 2026-09-26 settles it — same -unit, same modem, one variable. +✅ **The cable travelled home with the unit** (confirmed 2026-09-25). So the +Benfei on the bench is physically the cable from the failed office setup — not a +substitute picked up later. There is no inference left in the chain: + +| | | +|---|---| +| the cable from the failed unit | **is** a Benfei / Prolific PL2303 | +| the Micromate's USB host | has **no** Prolific driver, in either firmware line | +| a laptop on that same cable + modem | round-trips perfectly | +| the Micromate on it | never answered | + +The FTDI cable arriving 2026-09-26 is now a confirmation rather than a test. **Corollary worth knowing.** The standing workaround — *"just deploy a Series III instead"* — works partly because a **MiniMate Plus has a DB-9 port directly on the From 27b0b25f159d50a4ab000e8ac84c4689c85c06ef Mon Sep 17 00:00:00 2001 From: serversdown Date: Sat, 26 Sep 2026 00:09:55 -0400 Subject: [PATCH 03/16] docs(claude): refresh "Where things stand" -- Series-4 live protocol and tooling The orientation block was dated 2026-08-28 and predated every finding from the Series-4 live-protocol work, which is exactly the content a fresh session needs and cannot discover on its own. Adds, at the top where it will actually be read: * The Series-4 live wire protocol is mapped end to end, with the three framing differences that break a Series III parser (no leading DLE, 0xC5/0x03 flags, uint16 length at payload[8:10]) and the note that the inbound call-home session is the only protocol unknown left. * That NO command has ever been originated against a unit by this project -- every write was performed by THOR while we recorded. Worth stating plainly so the next session does not casually break it. * That micromate/ is still codec-only with no live client, and that minimateplus.transport is protocol-agnostic and reusable when one is built. * The bench tooling, including why mm_frame_parse.py has to exist at all: S3FrameParser scans for DLE+STX and therefore cannot see Micromate responses. * The FTDI/CDC-ACM-only USB host constraint and the Sabrent/Benfei cable trap, since a PL2303 cable leaves a unit with no working modem port and the cables are indistinguishable by eye. Corrects a stale operational claim: the v0.30.0 Series-4 backfill HAS now been run on prod, so the ~3.3%-low geophone values are fixed and the job does not need repeating. Also generalises the "update the protocol reference" instruction from the single Series III document to a table of all three, and adds bridges/ plus the two newer docs to the project layout. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- CLAUDE.md | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 3 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a8faa16..cc80b6f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,10 +10,38 @@ pair — lives in `../terra-view/docs/tmi-stack.md`, which is also loaded as --- -## Where things stand (updated 2026-08-28) +## Where things stand (updated 2026-09-26) Read this first when picking the project back up. +- **The Series-4 LIVE wire protocol is reverse-engineered end to end + (2026-09-25).** `docs/micromate_protocol_reference.md` is the Series-4 + Rosetta Stone, sibling to `instantel_protocol_reference.md`. **A Micromate + answers Series III command frames** — three framing differences: responses + have **no leading `DLE`** (a bare `STX`), `payload[1]` is `0xC5` (Blastware + firmware) or `0x03` (Thor firmware) rather than `0x10`, and the data length + is a **uint16 BE at `payload[8:10]`** (as a byte it under-reads `SUB 0x1A` + by 47x). Read path, event chain, setups, scheduler, monitoring control and + per-event delete are all mapped; **the inbound call-home session is the only + protocol unknown left.** + ⚠ **No command has ever been originated against a unit by this project.** + Every write was performed by THOR while we recorded. That line is worth + keeping. + ⚠ `micromate/` still has **no live client** — it is codec-only. The + `minimateplus/` stack (transport/framing/protocol/client) has no Series-4 + counterpart yet. `minimateplus.transport` is protocol-agnostic and reusable. +- **Bench tooling for device diagnosis (2026-09-25).** `bridges/mm_probe.py` + distinguishes the four faults THOR reports identically as "disconnected" + (refused / connect timeout / **connected but no reply** / replied) and names + what to try next. `bridges/mm_link.py` is a stand-in for a cellular modem + with a decoded log and fault injection. `scratch/mm_frame_parse.py` exists + because **`S3FrameParser` cannot see Micromate responses at all** — it scans + for `DLE+STX`, which never appears in Series-4 traffic. +- **A Micromate's USB-A host port drives FTDI and CDC-ACM only** — no Prolific, + in either firmware line. TMI buys both Sabrent (FTDI) and Benfei (PL2303) + cables and they are indistinguishable by eye. A PL2303 cable leaves a unit + with **no working modem port at all**; identify by `lsusb` VID, `0403` vs + `067b`. This accounted for a unit that could not be deployed. - **Series-3 decode is verified per-sample at scale (v0.27.0).** The full DL2 archive decodes **14,338 / 14,338** paired files exactly against their preserved Blastware ASCII exports — 1,249 waveform + 13,089 histogram, 45 @@ -92,6 +120,9 @@ Read this first when picking the project back up. **v0.27.0 does NOT owe prod a backfill** — verified: the partial-final-block fix changes 0 of the 10,215 histograms in the prod store (the 4 recovered files are archive-only and were never ingested). + ✅ **The v0.30.0 Series-4 backfill HAS been run on prod (2026-09-25).** Every + stored Series-4 geophone value was ~3.3% low until then; that is corrected and + the job does not need repeating. - **The "offset" hardware fault has its own journal** -- `docs/offset_investigation.md`. **5 of 45 units (11%)**, and the fault is **persistent** — it stays until the geophone is serviced. Detect it with @@ -103,7 +134,18 @@ Read this first when picking the project back up. `SUB 0x0E` (unimplemented), which may carry those very numbers. -When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document +When new information about a protocol is discovered, record it in the matching +reference **in addition to** this document: + +| series | document | +|---|---| +| Series III (MiniMate Plus / BlastMate) | `docs/instantel_protocol_reference.md` | +| **Series IV (Micromate / THOR)** | **`docs/micromate_protocol_reference.md`** | +| Thor IDF file format | `docs/idf_protocol_reference.md` | + +Both protocol references carry retractions in place rather than deleting what +turned out to be wrong — that convention has already saved re-deriving the same +mistakes twice, so keep it. --- @@ -276,8 +318,15 @@ minimateplus/ ← Python client library (primary focus) sfm/server.py ← FastAPI REST server exposing device data over HTTP seismo_lab.py ← Tkinter GUI (Bridge + Analyzer + Console tabs) +bridges/ + mm_probe.py ← name the fault behind a dead unit (4 verdicts, read-only) + mm_link.py ← bench stand-in for a cellular modem, with fault injection + ach_mitm.py ← TCP relay for recording a Series-3 ACH session + docs/ - instantel_protocol_reference.md ← reverse-engineered protocol spec ("the Rosetta Stone") + instantel_protocol_reference.md ← Series III protocol spec ("the Rosetta Stone") + micromate_protocol_reference.md ← Series IV protocol spec + THOR's measured behaviour + idf_protocol_reference.md ← Thor IDF file format CHANGELOG.md ← version history ``` From 8a1d1c3f176ec5e3760958328c849f5585324b37 Mon Sep 17 00:00:00 2001 From: serversdown Date: Sat, 26 Sep 2026 01:59:35 -0400 Subject: [PATCH 04/16] docs(series4): spec the live client, so tomorrow is implementation not design Turns docs/micromate_protocol_reference.md into a build plan for the missing half of micromate/ -- it is codec-only today with no way to talk to a unit. Layout mirrors minimateplus/: framing, protocol, client. Transport is REUSED -- minimateplus/transport.py is byte-level and protocol-agnostic, and already handles the RV50/RV55 habit of emitting RING/CONNECT to a caller. But minimateplus/framing is deliberately NOT shared: the two framings differ in ways that look small and are not, and a shared module would accumulate `if series ==` branches until neither case is readable. Records the three things that make S3FrameParser useless on Series IV (bare STX, 0xC5/0x03 flags, uniform 10 XX -> XX destuffing with no inner-frame carve-out), and the traps that have already cost time: the declared length is a uint16 BE and reading it as a byte under-reads SUB 0x1A by 47x; SUB 0x1C is four bytes longer on the Thor line so parse forward not backward, or a BD unit reports 577.92 V; the monitoring flag must be tested non-zero; and the SUB byte itself can arrive DLE-escaped, so destuff before indexing. Scope is READ-ONLY, stated with the reasoning: no command has ever been originated against a unit by this project, and keeping that true through the read client means the first thing we ever send to a customer's instrument is a deliberate decision rather than a side effect of a client that grew a method. Tests are specified to embed frames as hex constants rather than read fixtures, since bridges/captures/ and tests/fixtures/ are both gitignored -- with a table of which cases matter and why, and a round-trip assertion against scratch/mm_frame_parse.py, which is already known good across three sessions at zero bad checksums. Steps 1-2 need no hardware. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_client_spec.md | 291 ++++++++++++++++++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100644 docs/micromate_client_spec.md diff --git a/docs/micromate_client_spec.md b/docs/micromate_client_spec.md new file mode 100644 index 0000000..7d07959 --- /dev/null +++ b/docs/micromate_client_spec.md @@ -0,0 +1,291 @@ +# Spec — a live client for Series IV (Micromate) + +Drafted 2026-09-26, ahead of implementation. The protocol work is finished; this +is the plan for turning `docs/micromate_protocol_reference.md` into code SFM can +run. + +**Read that document first.** Everything here assumes it, and every constant +below is sourced from it rather than restated with justification. + +--- + +## Goal and scope + +`micromate/` is codec-only today — `idf_file.py`, `models.py`, the report +writers. There is no way to talk to a unit. This adds the live half, mirroring +`minimateplus/`. + +**In scope, first pass:** + +- connect over TCP (a field modem) or serial/USB (a bench unit) +- identify a unit, read its state, clock, memory and setups +- walk the event chain and download events +- return `Event` objects the existing codec already understands + +**Explicitly out of scope, first pass:** + +- ⚠ **Any write.** Setups, schedules, call-home config, monitoring start/stop, + and per-event delete are all mapped, and none of them will be implemented + here. **No command has ever been originated against a unit by this project** + — every write observed was performed by THOR while we recorded. Keeping that + true through the read client is deliberate: it means the first thing we ever + send to a customer's instrument is a decision someone made on purpose, not a + side effect of a client that happened to grow a method. +- the inbound call-home session — still the one protocol unknown + +--- + +## Layout + +``` +micromate/ + framing.py NEW frame building, response parsing, checksum + protocol.py NEW one method per wire command, returns raw payloads + client.py NEW high-level API, returns models + idf_file.py (existing — decodes what 0x5A returns, unchanged) + models.py (existing — extend, do not fork) +``` + +**Transport is reused, not rewritten.** `minimateplus/transport.py` is +byte-level and protocol-agnostic — `BaseTransport`, `SerialTransport`, +`TcpTransport`, plus `read_until_idle()` which already handles the RV50/RV55 +habit of emitting `\r\nRING\r\n\r\nCONNECT\r\n` to a caller. Import it. + +⚠ Do **not** import `minimateplus.framing`. The two framings differ in ways +that look small and are not, and a shared module would accumulate `if series ==` +branches until neither case is readable. + +--- + +## `micromate/framing.py` + +### Requests + +Series IV accepts Series III request frames unmodified. The simplest correct +implementation re-exports the builder rather than duplicating it: + +```python +from minimateplus.framing import build_bw_frame # requests are identical +``` + +⚠ One open question, flagged in the protocol reference and **not** settled: +whether `0x10` bytes inside request *params* need stuffing. No probe we sent +carried one. Until it is settled, assert on it rather than guessing: + +```python +def build_request(sub: int, offset: int = 0, params: bytes = bytes(10)) -> bytes: + if 0x10 in params: + raise NotImplementedError( + "params containing 0x10 — stuffing rule unconfirmed; see " + "micromate_protocol_reference.md, 'Untested and unsafe-until-agreed'" + ) + return build_bw_frame(sub, offset, params) +``` + +That turns an unknown into a loud failure instead of a corrupt frame. + +### Responses — where Series III's parser cannot follow + +| | Series III | Micromate | +|---|---|---| +| frame start | `DLE STX` | **bare `STX`** | +| `payload[1]` | `0x10` | `0xC5` (Blastware fw) / `0x03` (Thor fw) | +| destuffing | `DLE+ETX` kept as literal inner-frame data | **`10 XX` → `XX`, uniformly** | + +The first row is why `S3FrameParser` returns nothing at all on Series IV traffic: +it scans for `DLE+STX`, which never appears. + +The third is a genuine **simplification** — no inner-frame carve-out. Validated +by checksum across every capture in `bridges/captures/9-24-26 - micromate2/`: +four candidate destuffing rules were tried, and only this one makes all frames +validate. + +### Checksum + +```python +def checksum(payload: bytes) -> int: + return sum(b for b in payload if b != 0x10) & 0xFF +``` + +The DLE-aware variant, same as Series III's `5A` and write frames — not the plain +SUM8 of ordinary Series III reads. + +### ⚠ The SUB byte can be escaped + +When a SUB's value is `0x02`, `0x03`, `0x04` or `0x10` it arrives as `10 XX`. +Reading it positionally without destuffing reports `0x10`. This bit once +already — `SUB 0x02` was logged as `SUB_10` for an afternoon. Destuff first, +then index. + +### Response shape + +```python +@dataclass +class MicromateFrame: + sub: int # response SUB; request = 0xFF - sub + flags: int # 0xC5 Blastware line, 0x03 Thor line + page_hi: int + page_lo: int + data: bytes # payload[5:], checksum stripped + checksum_valid: bool + + @property + def firmware_line(self) -> str: # "blastware" | "thor" | "unknown" + @property + def declared_length(self) -> int: # uint16 BE at data[3:5] (= payload[8:10]) +``` + +⚠ **`declared_length` is a uint16 BE.** Read as a single byte it under-reads +`SUB 0x1A` by 47x — 44 against a true 2092. This is the single most expensive +mistake available in this protocol and it has already been made once. + +`MicromateFrameParser` mirrors `S3FrameParser`: `feed(bytes) -> list[frame]`, +accumulates in `.frames`, `reset()`, and keeps the `bytes_fed` counter (it is +what distinguishes "no bytes at all" from "bytes but no complete frame" on a +timeout, and that distinction earned its keep during the Series III work). + +--- + +## `micromate/protocol.py` + +One method per command, returning raw payload bytes. No interpretation — that +belongs in `client.py`. + +**Reads use `offset = 0xFFFF`** and return the whole block in one response; +Series III's two-step probe/data dance is unnecessary. `POLL` is the exception, +taking its data length. Per-command offsets, all observed: + +| command | SUB | rsp | offset | returns | +|---|---|---|---|---| +| poll | `0x5B` | `0xA4` | `0x0030` | device string, model | +| serial | `0x15` | `0xEA` | `0x000A` | `UM12947` | +| device info | `0x01` | `0xFE` | `0xFFFF` | firmware, calibration | +| state | `0x49` | `0xB6` | `0xFFFF` | `data[11]`: non-zero = monitoring | +| monitor status | `0x1C` | `0xE3` | `0xFFFF` | flag, **device clock**, battery, memory | +| storage range | `0x06` | `0xF9` | `0xFFFF` | event storage extent | +| active setup name | `0x41` | `0xBE` | `0xFFFF` | `TEST1.mmb` | +| first setup | `0x3F` | `0xC0` | `0xFFFF` | setup-list walk head | +| next setup | `0x40` | `0xBF` | `0xFFFF` | …until an empty name | +| compliance config | `0x1A` | `0xE5` | `0xFFFF` | ~2103 B setup block | +| call-home config | `0x2C` | `0xD3` | `0xFFFF` | 137 B | +| arm event | `0x93` | `0x6C` | — | before every event | +| first event | `0x1E` | `0xE1` | `0xFFFF` | key + size | +| next event | `0x1F` | `0xE0` | `0xFFFF` | key + size | +| event record | `0x0C` | `0xF3` | `0xFFFF` | 210 B — project, location, peaks | +| event header | `0x0A` | `0xF5` | `0xFFFF` | 30 B list record | +| bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim** | + +⚠ **`SUB 0x1C` is 4 bytes longer on the Thor firmware line** (`0x30` vs `0x2C`). +Parse **forward** from `declared_length`, never backward from the end — Series +III reads battery and memory from the end of that block, and doing so on a BD +unit yields a battery voltage of **577.92 V**. + +⚠ **Test the monitoring flag for non-zero**, never against a constant. It has +read both `0x0E` and `0x0C` while monitoring. + +### `0x5A` — simpler than Series III, deliberately + +No arming ritual, no chunk loop, no `STRT` end-offset parsing, no `TERM` frame. +One request returns the whole event: + +```python +offset_word = 0x1000 + 2 * ceil(size / 512) # size from the chain walk +``` + +The payload **is** the `.IDFW` file, byte for byte — so it feeds +`micromate.idf_file.read_idf_file()` and `/db/import/idf_file` unchanged. + +⚠ Do not port the Series III `5A` walk. Its address arithmetic caused a 5x +over-read and a `> 64 KB` page-boundary bug that is *still open* on the Series +III side. None of that applies here. + +--- + +## `micromate/client.py` + +```python +class MicromateClient: + def __init__(self, transport: BaseTransport): ... + def open(self) / close(self) / is_open(self) + + # identity and state + def connect(self) -> DeviceInfo # poll → serial → device info → state + def get_state(self) -> UnitState # monitoring?, clock, battery, memory + def get_active_setup(self) -> str + def list_setups(self) -> list[str] # 0x3F → 0x40… until empty + + # events + def list_events(self) -> list[EventRef] # 0x93 → 0x1E → 0x1F… (key + size) + def download_event(self, ref) -> bytes # raw .IDFW/.IDFH + def get_event(self, ref) -> Event # download + decode via idf_file +``` + +`connect()` should mirror THOR's preamble (`POLL → SERIAL → 0x49 → POLL`) — +⚠ but note the reference records that **whether the unit requires it is +untested**. Do it because it is known-good, not because it is known-necessary, +and say so in the docstring. + +`list_events()` returns the key *and* the size, because `download_event()` needs +the size to compute its offset word. + +--- + +## Tests + +**Offline, from captured bytes — no hardware.** This is the part worth doing +first, because it can be fully verified tonight's-captures-style before any unit +is involved. + +``` +tests/test_micromate_framing.py +``` + +⚠ `bridges/captures/` and `tests/fixtures/` are both gitignored, so tests must +not depend on files being present. **Embed the frames as hex constants** — they +are 16–68 bytes each, and a handful covers every case: + +| case | why | +|---|---| +| POLL probe reply, 19 B | shortest valid frame | +| POLL data reply, 68 B | contains a literal `0x10` — only the DLE-aware checksum matches | +| `0x1A` response, 2108 B | exercises `declared_length` as a true uint16 (`0x082C`) | +| a Thor-line reply, `flags = 0x03` | `0x03` is ETX; proves destuffing before framing | +| a frame whose SUB is `0x02` | arrives as `10 02`; proves destuff-then-index | +| a truncated frame | parser must return nothing, not a bad frame | +| a corrupted checksum | `checksum_valid == False`, frame still returned | + +Then a round-trip assertion: feed a whole captured session through the parser and +assert the frame count and every SUB, against `scratch/mm_frame_parse.py`'s +output — which is already known good, having parsed 24, 38 and 40-frame sessions +with zero bad checksums. + +**Live, second:** against the bench unit on mint-mac via `mm_link.py`. +`connect()`, `list_setups()` (should return the 23 known names), `list_events()`, +then `download_event()` and assert the bytes decode and match a +`/db/import/idf_file` ingest of the same event. + +--- + +## Order of work + +1. `framing.py` + its tests — offline, verifiable immediately +2. `protocol.py` — reads only, one method per row of the table above +3. `client.py` — `connect()`, `get_state()`, `list_setups()` +4. the event chain and `download_event()` +5. decode end-to-end and compare against a store event + +Steps 1–2 need no hardware at all. + +--- + +## Open questions to settle while implementing + +- **Request param stuffing** — raise `NotImplementedError` rather than guess. +- **Is THOR's preamble required?** Try one command cold and find out; it is a + two-minute test with the bench unit and it removes a ritual if unnecessary. +- **`Event` model fit** — Series IV carries fields Series III lacks (setup file + name, `LMic`/`SMic` channels). Extend `micromate/models.py`; do not fork the + shared `Event`. +- **Which `0x0C` fields to trust.** The peak float there runs 2–5% above + `max(T,V,L)` and is **not** the vector sum; its offset was inferred, not + established. The reference marks it do-not-rely-on — prefer decoded samples. From a7631a81796e8caff22b9c825e47807db6d58fee Mon Sep 17 00:00:00 2001 From: serversdown Date: Sat, 26 Sep 2026 18:11:16 -0400 Subject: [PATCH 05/16] docs(series4): RESOLVED -- FTDI cable AND a cold boot; modems bridge one session The cable swap alone did not fix it. A probe immediately after fitting a genuine FTDI cable (lsusb 0403:6001, FT232) returned the same "connected, no reply". A COLD BOOT of the unit was also required -- power button held five seconds through the two-stage battery-disconnect prompt, powered back up with the cable already attached. THOR connected the moment it finished booting, and an independent probe returned UM12947, idle, over cellular. So the Micromate's USB host does not rescan on hot-swap: it enumerates that port at boot and, having once failed to identify a device, does not try again. Changing the cable is a power-cycle operation -- and anyone swapping one in the field who probes immediately will conclude the new cable is faulty too. Full chain: an FTDI or CDC-ACM cable, AND a cold boot with it attached. SEPARATELY CONFIRMED, by accident: these modems bridge ONE TCP session to serial at a time. With THOR connected, the probe got "TCP ok, NO REPLY"; with THOR disconnected and nothing else changed, the same probe returned the serial number. The PAD accepts a second connection and then never forwards it -- it does not refuse and does not close. That makes "connected but no reply" ambiguous between a broken serial path and somebody else already holding the unit, and mm_probe reported only the former -- which would push a diagnosis in exactly the wrong direction, on the very fault we spent two days chasing. Its verdict now names both causes, tells you to eliminate contention first, and includes the cable-chipset and cold-boot steps. Recorded as an SFM design constraint: our receiver and THOR cannot both hold a unit, and during any migration both will exist. "Another client holds this unit" has to be a distinct visible state. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- bridges/mm_probe.py | 33 ++++++++++++++------ docs/micromate_protocol_reference.md | 46 ++++++++++++++++++++++++++++ 2 files changed, 69 insertions(+), 10 deletions(-) diff --git a/bridges/mm_probe.py b/bridges/mm_probe.py index 8ba7992..e020a03 100644 --- a/bridges/mm_probe.py +++ b/bridges/mm_probe.py @@ -134,17 +134,30 @@ def probe(host: str, port: int, timeout: float) -> int: step(2, "POLL", f"NO REPLY in {timeout:.1f} s") print("\nverdict: the MODEM answered but the unit did not.") print(" TCP is fine end to end — something accepted the connection.") - print(" What is missing is the serial side. Most likely the modem is") - print(" not forwarding to its serial port, which is what a wedged") - print(" transparent-TCP session looks like: the slot is held by a") - print(" connection that never closed.") + print(" What is missing is the serial side. Two quite different") + print(" causes produce this, and they are NOT distinguishable from") + print(" here:") + print("\n 1. SOMEONE ELSE HOLDS THE SESSION. These modems bridge ONE") + print(" TCP session to serial at a time. A second connection is") + print(" accepted and then simply not forwarded. Confirmed 2026-09-26:") + print(" with THOR connected this probe saw exactly this; the moment") + print(" THOR disconnected the same probe returned the serial number.") + print(" ** Check whether THOR (or anything else) has the unit first. **") + print("\n 2. The serial path is genuinely broken — a stale session the") + print(" modem never released, a cable the unit cannot enumerate, or") + print(" a unit that is off.") print("\n Try, in order:") - print(" 1. ACEmanager -> TCP Idle Timeout. If 0/disabled, a stale") - print(" session holds the slot forever. 2 minutes is the value") - print(" this project standardised on.") - print(" 2. Reboot the modem. If that fixes it, the modem was") - print(" holding state and the timeout is the permanent fix.") - print(" 3. Check the unit's own screen — serial cable, power.") + print(" 1. Disconnect any other client and re-probe. If it answers,") + print(" it was contention, not a fault.") + print(" 2. The cable's chipset. A Micromate drives FTDI and CDC-ACM") + print(" only — a Prolific PL2303 gives it no serial port at all.") + print(" lsusb: FTDI is 0403, Prolific 067b.") + print(" 3. Power-cycle the UNIT with the cable attached (hold power") + print(" 5 s, through the two-stage prompt). Its USB host rescans") + print(" on cold boot; it may not on hot-swap.") + print(" 4. AirLink OS -> TCP Idle Timeout. If 0/disabled, a stale") + print(" session holds the slot indefinitely. 2 minutes is the") + print(" value this project standardised on.") sock.close() return 5 diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 2ceb32d..452704c 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -2162,6 +2162,52 @@ unit.** No USB-to-serial adapter anywhere in the path, so there is no chipset t get wrong. That workaround has been quietly routing around this exact failure mode. +### ✅ RESOLVED (2026-09-26) — and it took a cold boot as well as the cable + +Swapping to a genuine FTDI cable (`lsusb`: `0403:6001`, FT232) **did not work on +its own.** A probe immediately after the swap returned the same *connected, no +reply*. + +**A cold boot of the unit was also required.** Holding the power button for five +seconds — through the two-stage prompt that disconnects the internal battery — +and powering back up **with the cable already attached**, brought it straight up. +THOR at the office connected the moment it finished booting, and an independent +probe returned `UM12947, idle` over cellular. + +⚠ **The USB host does not rescan on hot-swap.** The Micromate enumerates its +USB-A host port at boot; once it has failed to identify a device there it does +not appear to try again. Changing that cable is a power-cycle operation, and +anyone swapping one in the field who probes immediately will conclude the new +cable is faulty too. + +So the full chain for a Micromate on a modem is: + +1. an **FTDI** or CDC-ACM cable — a Prolific PL2303 gives the unit no serial port +2. a **cold boot** with that cable attached + +### ✅ These modems bridge ONE session at a time — confirmed + +Long suspected, never demonstrated. Demonstrated now, by accident: + +| | probe result | +|---|---| +| THOR connected to the unit | `TCP connect ok` … **`NO REPLY`** | +| THOR disconnected, nothing else changed | `reply 68 B`, **`UM12947`**, idle | + +The PAD **accepts** a second TCP connection and then does not forward it. It +does not refuse, and it does not close — it simply never bridges. + +⚠ **This means "connected but no reply" has two causes that are indistinguishable +from the client side:** a genuinely broken serial path, and *somebody else +already has the unit*. `bridges/mm_probe.py` originally reported only the first, +which would send a diagnosis in exactly the wrong direction; its verdict now +names both and tells you to eliminate contention first. + +**For SFM this is a design constraint, not a footnote.** Our receiver and THOR +cannot both hold a unit, and during any migration both will exist. "Another +client holds this unit" needs to be a distinct, visible state — not folded into +a failure, and certainly not into a green tick. + ### How this was isolated — the method is reusable A Micromate on an RX55 was unreachable from THOR. Rather than guess, each layer From 5fe99568a20cf5f299feb0749a3687901cdfee8a Mon Sep 17 00:00:00 2001 From: serversdown Date: Sun, 27 Sep 2026 05:24:24 -0400 Subject: [PATCH 06/16] feat(micromate): framing layer -- and three spec rules the bytes refuted Step 1 of docs/micromate_client_spec.md: micromate/framing.py plus 31 offline tests. Every rule was checked against the captures BEFORE being written, which is the only reason this commit is not a bug. Three things the spec asserted are wrong, all of which fail silently: 1. Requests are NOT plain Series III frames. A Micromate escapes four byte values -- 0x02 0x03 0x04 0x10 -- where Series III escapes one. minimateplus.build_bw_frame reproduces 161 of Thor's 218 captured read frames; build_request() reproduces 218/218. The 57 it missed include EVERY 0x5A download (offset 0x0400 puts a literal 0x04 in offset_hi) and the scheduler enable. An unescaped 0x03/0x04 terminates the frame, so the unit never answers -- indistinguishable from a dead unit, and event download would have hit it on the first request ever sent. 2. The checksum is plain SUM8 of the destuffed payload, not the DLE-aware variant. 251/251 both directions. The DLE-aware form is correct paired with Series III destuffing, which leaves an escaped byte as two bytes; after uniform destuffing it subtracts the correction twice and disagrees with the wire on 55 of 251 responses. scratch/mm_frame_parse.py shipped with exactly that pairing and looked clean only because it accepts either rule -- so it labelled those 55 "SUM8" and never flagged one bad. "Zero bad checksums" was true and carried no information. A tool that tries N candidate rules cannot falsify any of them. Fixed to validate against SUM8 alone. 3. SUB 0x5A is a 1024-byte chunk loop, not one request per event. Thor's form, verified on all six bench events (4,076 -> 13,424 B): chunks = ceil(size/1024), offset = min(1024, size - 1024*i) as a byte count, params[2:4] = the byte offset, response data = offset + 11. sum(offsets) == size exactly, every time. This does not retract the earlier single-request observation -- that used offset_hi = 0x10, which in Series III is the bulk-stream marker, so it is plausibly a distinct streaming mode returning several frames. Those captures never landed in the repo, so it cannot be re-derived. Implement Thor's form; the other is worth one bench test. Also: a 0x10 inside request params needs no special handling (settled -- Thor sends it, the wire doubles it), so the planned NotImplementedError guard is gone. declared_length -> probe_length, because it is only meaningful in a probe reply and Thor never probes. Synthesised test frames are marked and each says what it stands in for. The flags=0x03 case is the only coverage of the Thor firmware line -- it wants a real 11.0BD capture next time UM20147 is on a bench. No writes. Read-path framing only; nothing here can originate a command. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_client_spec.md | 158 ++++++++--- docs/micromate_protocol_reference.md | 176 ++++++++++-- micromate/framing.py | 304 ++++++++++++++++++++ scratch/mm_frame_parse.py | 41 ++- tests/test_micromate_framing.py | 399 +++++++++++++++++++++++++++ 5 files changed, 1006 insertions(+), 72 deletions(-) create mode 100644 micromate/framing.py create mode 100644 tests/test_micromate_framing.py diff --git a/docs/micromate_client_spec.md b/docs/micromate_client_spec.md index 7d07959..ff5ca38 100644 --- a/docs/micromate_client_spec.md +++ b/docs/micromate_client_spec.md @@ -57,32 +57,37 @@ branches until neither case is readable. --- -## `micromate/framing.py` +## `micromate/framing.py` — ✅ BUILT 2026-09-27 + +Implemented, with `tests/test_micromate_framing.py` (31 tests, all passing). +**Two things in this section as originally drafted were wrong**, and both were +caught by measuring against the captures before writing code rather than after. +They are left in place below, struck through, because both are the kind of +mistake that would be made again. ### Requests -Series IV accepts Series III request frames unmodified. The simplest correct -implementation re-exports the builder rather than duplicating it: +~~Series IV accepts Series III request frames unmodified. The simplest correct +implementation re-exports the builder rather than duplicating it:~~ ```python -from minimateplus.framing import build_bw_frame # requests are identical +from minimateplus.framing import build_bw_frame # ✗ WRONG — 161/218 ``` -⚠ One open question, flagged in the protocol reference and **not** settled: -whether `0x10` bytes inside request *params* need stuffing. No probe we sent -carried one. Until it is settled, assert on it rather than guessing: +⚠ **`build_bw_frame` reproduces only 161 of Thor's 218 captured read frames.** +The payload layout is identical; the stuffing is not. A Micromate escapes +**four** byte values — `0x02`, `0x03`, `0x04`, `0x10` — where Series III +escapes one. The frames this breaks are **every `SUB 0x5A` download** +(`offset = 0x0400` → a literal `0x04` in `offset_hi`) and the scheduler enable. +An unescaped `0x03`/`0x04` terminates the frame early, so the unit just does not +answer. -```python -def build_request(sub: int, offset: int = 0, params: bytes = bytes(10)) -> bytes: - if 0x10 in params: - raise NotImplementedError( - "params containing 0x10 — stuffing rule unconfirmed; see " - "micromate_protocol_reference.md, 'Untested and unsafe-until-agreed'" - ) - return build_bw_frame(sub, offset, params) -``` +`build_request()` with the correct escape set reproduces **218/218**. -That turns an unknown into a loud failure instead of a corrupt frame. +✅ **Settled: `0x10` inside params needs no special handling.** The planned +`NotImplementedError` guard is unnecessary — Thor sends +`params = 00 00 10 00 …` on `SUB 0x5A` in five captured frames and the wire +carries an ordinary doubled `10 10`. One rule covers the whole payload. ### Responses — where Series III's parser cannot follow @@ -104,11 +109,22 @@ validate. ```python def checksum(payload: bytes) -> int: - return sum(b for b in payload if b != 0x10) & 0xFF + return sum(payload) & 0xFF # payload already de-stuffed ``` -The DLE-aware variant, same as Series III's `5A` and write frames — not the plain -SUM8 of ordinary Series III reads. +~~The DLE-aware variant, same as Series III's `5A` and write frames.~~ + +⚠ **Plain SUM8, not the DLE-aware variant** — 251/251 both directions. The +DLE-aware form is the right answer paired with *Series III* de-stuffing, which +leaves an escaped byte in the payload as two bytes. De-stuffing `10 XX → XX` +already removes the `0x10`, so excluding it again subtracts the correction +twice, and the result disagrees with the wire on **55 of 251** captured +responses — every frame holding a literal `0x10`. + +`scratch/mm_frame_parse.py` shipped with exactly that pairing. It looked clean +only because it accepts a frame matching *either* rule, so it labelled those 55 +`SUM8` and never flagged one bad. "Zero bad checksums" was true and carried no +information. Fixed there too. ### ⚠ The SUB byte can be escaped @@ -129,16 +145,27 @@ class MicromateFrame: data: bytes # payload[5:], checksum stripped checksum_valid: bool + @property + def request_sub(self) -> int: # 0xFF - sub + @property + def page_key(self) -> int: # uint16 BE at payload[3:5] @property def firmware_line(self) -> str: # "blastware" | "thor" | "unknown" @property - def declared_length(self) -> int: # uint16 BE at data[3:5] (= payload[8:10]) + def probe_length(self) -> int | None: # uint16 BE at data[3:5] (= payload[8:10]) ``` -⚠ **`declared_length` is a uint16 BE.** Read as a single byte it under-reads +⚠ **`probe_length` is a uint16 BE.** Read as a single byte it under-reads `SUB 0x1A` by 47x — 44 against a true 2092. This is the single most expensive mistake available in this protocol and it has already been made once. +Renamed from `declared_length`, because it is **only meaningful in the reply to +an `offset = 0` probe** — and Thor never probes. Across all 251 captured +responses the field reads 0 or a page count, never a length, precisely because +that session is single-step reads throughout. `page_key` is the field that +carries meaning there. The only genuine probe reply we hold is the POLL one +preserved in `scratch/fake_unit.py`. + `MicromateFrameParser` mirrors `S3FrameParser`: `feed(bytes) -> list[frame]`, accumulates in `.frames`, `reset()`, and keeps the `bytes_fed` counter (it is what distinguishes "no bytes at all" from "bytes but no complete frame" on a @@ -183,21 +210,41 @@ unit yields a battery voltage of **577.92 V**. ⚠ **Test the monitoring flag for non-zero**, never against a constant. It has read both `0x0E` and `0x0C` while monitoring. -### `0x5A` — simpler than Series III, deliberately +### `0x5A` — a bounded chunk loop, and much simpler than Series III -No arming ritual, no chunk loop, no `STRT` end-offset parsing, no `TERM` frame. -One request returns the whole event: +⚠ **Corrected 2026-09-27.** This section said "no chunk loop — one request +returns the whole event", with `offset_word = 0x1000 + 2 * ceil(size / 512)`. +That describes our own 2026-09-23 probes, which set `offset_hi = 0x10`. **Thor +chunks**, and Thor's form is the one verified from bytes on disk: ```python -offset_word = 0x1000 + 2 * ceil(size / 512) # size from the chain walk +n = ceil(size / 1024) # size from the chain walk +for i in range(n): + offset = min(1024, size - 1024 * i) # a BYTE COUNT + params = key4 + bytes(6) if i == 0 else bytes(2) + pack(">H", 1024*i) + bytes(6) + file_bytes += response.data[11:] # response data is exactly offset + 11 ``` -The payload **is** the `.IDFW` file, byte for byte — so it feeds +Verified on all six bench events (4,076 → 13,424 B): `sum(offsets) == size` +exactly, with the predicted chunk count and final offset every time. + +Still no arming ritual for `0x5A` itself, no `STRT` end-offset parsing and no +`TERM` frame — the simplification the original claim celebrated is real, it just +is not single-shot. (`SUB 0x93` arms the *chain walk*, before `1E`/`1F`, not +the download.) + +The concatenated payload **is** the `.IDFW` file, byte for byte — so it feeds `micromate.idf_file.read_idf_file()` and `/db/import/idf_file` unchanged. ⚠ Do not port the Series III `5A` walk. Its address arithmetic caused a 5x over-read and a `> 64 KB` page-boundary bug that is *still open* on the Series -III side. None of that applies here. +III side. None of that applies here: the chunk index is a byte offset into the +file, bounded by a size the device told us, and it cannot run past the event. + +⚠ **`assert sum(len(chunk) - 11 for chunk in chunks) == size`.** A silently +short event is the failure mode this project has been bitten by repeatedly on +the Series III side, and here the check is free because the size is known up +front. --- @@ -242,22 +289,34 @@ tests/test_micromate_framing.py ⚠ `bridges/captures/` and `tests/fixtures/` are both gitignored, so tests must not depend on files being present. **Embed the frames as hex constants** — they -are 16–68 bytes each, and a handful covers every case: +are 19–138 bytes each. ✅ Done; what actually landed: -| case | why | -|---|---| -| POLL probe reply, 19 B | shortest valid frame | -| POLL data reply, 68 B | contains a literal `0x10` — only the DLE-aware checksum matches | -| `0x1A` response, 2108 B | exercises `declared_length` as a true uint16 (`0x082C`) | -| a Thor-line reply, `flags = 0x03` | `0x03` is ETX; proves destuffing before framing | -| a frame whose SUB is `0x02` | arrives as `10 02`; proves destuff-then-index | -| a truncated frame | parser must return nothing, not a bad frame | -| a corrupted checksum | `checksum_valid == False`, frame still returned | +| case | source | why | +|---|---|---| +| POLL probe reply, 19 B | captured (via `fake_unit.py`) | shortest valid frame; the only real probe reply we hold | +| `0x5A` chunk, 138 B | captured | holds literal `0x10` **and** literal `0x41` — the checksum case, and proves ACK is not escaped | +| `0x49` state reply, 25 B | captured | a literal `0x02`, escaped | +| `0x48` file reply, 24 B | captured | escaped `0x04` in `data[0]` — one byte late without destuffing | +| 8 Thor request frames | captured | byte-for-byte against `build_request()`, incl. both `0x5A` forms and the scheduler enable | +| `probe_length = 0x082C` | **synthesised** | no probe reply for `0x1A` exists on disk — the 9-24-26 session never probes | +| Thor-line reply, `flags = 0x03` | **synthesised** | no 11.0BD capture is in the repo; built by flipping one byte of the real POLL reply | +| escaped checksum byte | **synthesised** | the shortest real one is 1,070 B, too long to embed for one assertion | +| truncated / corrupt / split-across-feeds | derived | parser must return nothing, flag rather than swallow, and survive any split point | -Then a round-trip assertion: feed a whole captured session through the parser and -assert the frame count and every SUB, against `scratch/mm_frame_parse.py`'s -output — which is already known good, having parsed 24, 38 and 40-frame sessions -with zero bad checksums. +⚠ Synthesised frames are marked `SYNTH_`-style in the test and each says what it +stands in for and why no capture was available. Do not let that set grow +quietly: the `flags = 0x03` case in particular is the **only** coverage of half +the fleet, and it deserves a real 11.0BD capture the next time UM20147 is on a +bench. + +Two corpus-backed tests run when the captures happen to be on the box and skip +cleanly otherwise: **251 response frames parse with zero bad checksums**, and +**`build_request()` reproduces 218/218 read frames**. The second is the test +that would have caught the escape-set error, so it is worth the skip marker. + +⚠ Do **not** assert against `scratch/mm_frame_parse.py`'s output as the original +plan proposed. That script accepts either checksum rule and is wrong about +which one is right — using it as an oracle would have pinned the bug. **Live, second:** against the bench unit on mint-mac via `mm_link.py`. `connect()`, `list_setups()` (should return the 23 known names), `list_events()`, @@ -268,7 +327,7 @@ then `download_event()` and assert the bytes decode and match a ## Order of work -1. `framing.py` + its tests — offline, verifiable immediately +1. ✅ `framing.py` + its tests — **done 2026-09-27**, 31 tests, offline 2. `protocol.py` — reads only, one method per row of the table above 3. `client.py` — `connect()`, `get_state()`, `list_setups()` 4. the event chain and `download_event()` @@ -276,11 +335,22 @@ then `download_event()` and assert the bytes decode and match a Steps 1–2 need no hardware at all. +**Worth carrying forward from step 1:** every rule got checked against the +captures *before* being written, and two of the three the spec asserted turned +out wrong — the escape set (26% of frames) and the checksum (22%). Both fail +silently. The captures are on disk and a re-stuff-and-compare loop takes about +two minutes per rule, so do that for `protocol.py`'s per-command offsets too +rather than trusting the table above. + --- ## Open questions to settle while implementing -- **Request param stuffing** — raise `NotImplementedError` rather than guess. +- ~~**Request param stuffing**~~ — ✅ settled 2026-09-27; no special handling. +- **Is the single-request `0x5A` form real?** Our 2026-09-23 probes set + `offset_hi = 0x10` and appeared to get a whole 11 KB event back, where Thor + chunks at 1024 B. Plausibly a distinct streaming mode that returns several + frames. One bench test settles it; implement Thor's form regardless. - **Is THOR's preamble required?** Try one command cold and find out; it is a two-minute test with the bench unit and it removes a ritual if unnecessary. - **`Event` model fit** — Series IV carries fields Series III lacks (setup file diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 452704c..4b87096 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -150,12 +150,38 @@ Every frame in this session was produced by `build_bw_frame(sub, offset)` with no Series IV changes, and the device accepted all of them: ``` -[ACK 0x41] [STX 0x02] [10 10] [flags 00] [SUB] [00] [00] [offset] [params×10] [chk] [ETX 0x03] +[ACK 0x41] [STX 0x02] [10 10] [flags 00] [SUB] [00] [offset_hi] [offset_lo] [params×10] [chk] [ETX 0x03] ``` -⚠ Only the doubled `BW_CMD` (`10 10`) form has been exercised. Whether other -literal `0x10` bytes inside params require stuffing is **untested** — none of -the probes sent carried one. +The *payload layout* is identical to Series III. The *stuffing* is not. + +> #### ⚠ Correction, 2026-09-27 — `build_bw_frame()` is NOT sufficient +> +> This section used to say requests were Series III frames "unmodified", and +> the client spec accordingly planned to re-export +> `minimateplus.framing.build_bw_frame`. Measured against Thor's own frames, +> that builder reproduces **161 of 218** captured read frames. It escapes only +> `0x10`; a Micromate escapes four bytes (see *The escape set* below). +> +> The 57 it gets wrong are not obscure: +> +> | frame | why it breaks | +> |---|---| +> | **every `SUB 0x5A` download** | `offset = 0x0400` puts a literal `0x04` in `offset_hi`, which must go out as `10 04` | +> | `SUB 0x47` scheduler enable | `params[7] = 0x03` → `10 03` | +> +> An unescaped `0x03` or `0x04` *is* a frame terminator, so the unit sees a +> short frame and does not answer — indistinguishable from a dead unit, and +> event download would have hit it on the first request ever sent. +> +> `micromate/framing.py:build_request()` reproduces **218/218**. Pinned per +> frame in `tests/test_micromate_framing.py`. + +✅ **Settled 2026-09-27: a `0x10` inside params needs no special handling.** +Previously flagged untested. Thor sends `SUB 0x5A` with +`params = 00 00 10 00 …` in five captured frames and the wire carries the +ordinary doubled `10 10`. There is no Series-III-style partial-stuffing +carve-out here — one rule covers the whole payload including the checksum. ### Responses — Series III *minus the DLE prefix* @@ -181,18 +207,75 @@ mismatch. [5+] data ``` -### Checksum — the DLE-aware variant +### Checksum — plain SUM8 of the de-stuffed payload ```python -chk = sum(b for b in payload if b != 0x10) & 0xFF +chk = sum(payload) & 0xFF # payload already de-stuffed ``` -Confirmed on every frame captured. The `POLL` probe response contains no -`0x10` and so cannot distinguish plain SUM8 from the DLE-aware form; the -`POLL` **data** response contains a `0x10` at payload offset 42, and only the -DLE-aware rule matches there. This is the same checksum Series III uses for -its `5A` bulk-stream and write frames — not the plain SUM8 of ordinary -Series III reads. +**251/251 responses and 251/251 requests**, across every capture in +`bridges/captures/9-24-26 - micromate2/`. + +> #### ⚠ Correction, 2026-09-27 — "the DLE-aware variant" +> +> This section previously specified +> `chk = sum(b for b in payload if b != 0x10) & 0xFF`, and that is wrong **as +> paired with the uniform de-stuffing rule below.** +> +> The earlier claim was not a misreading; it was a correct rule attached to the +> wrong convention. The DLE-aware form belongs with *Series III* de-stuffing, +> which leaves an escaped byte in the payload as **two** bytes — there, +> skipping the `0x10` is the necessary correction. De-stuffing `10 XX → XX` +> already removes it, so excluding `0x10` as well **subtracts the correction +> twice**. +> +> Cost of the pairing, measured: it disagrees with the wire on **55 of 251** +> response frames — every frame whose payload holds a literal `0x10`. A clean +> example, from the `0x5A` chunk at offset `0x0070` of +> `raw_s3_…_Download_events_then_delete_1_event.bin`: the wire says `0xC1`, +> plain SUM8 says `0xC1`, the DLE-aware form says `0x91`. +> +> **Why nobody noticed:** `scratch/mm_frame_parse.py` accepts a frame matching +> *either* rule and labels which one hit. It reported those 55 frames as +> `SUM8` and never flagged one bad, so "zero bad checksums across 24, 38 and +> 40-frame sessions" was true and told us nothing about which rule was right. +> A tool that tries every candidate cannot falsify any of them — if it is going +> to stay permissive, it has to *report the split*, not just the pass. +> +> Pinned by `tests/test_micromate_framing.py`. + +### The escape set — exactly four bytes + +A Micromate escapes `0x02`, `0x03`, `0x04` and `0x10`, each prefixed with a +`DLE`, **and nothing else** — in both directions. + +Established by re-stuffing every captured frame and comparing to the wire: + +| candidate escape set | responses reproduced | requests reproduced | +|---|---|---| +| `{0x10}` — the Series III rule | 130/251 | 177/251 | +| `{0x10, 0x03}` | 135/251 | 184/251 | +| `{0x10, 0x02, 0x03}` | 178/251 | 192/251 | +| **`{0x10, 0x02, 0x03, 0x04}`** | **251/251** | **251/251** | +| `{0x10, 0x02, 0x03, 0x04, 0x41}` | 196/251 | — | + +Corroborated independently by the byte that follows a wire `DLE`: across all +251 responses it is only ever `0x02` (933×), `0x03` (510×), `0x04` (655×) or +`0x10` (1179×). Nothing else ever appears there. + +The last row matters because ACK looks like it ought to be escaped and is not — +a literal `0x41` in the data goes out bare. + +**The checksum byte is escaped too.** Three captured responses have a checksum +of `0x02`/`0x03`/`0x04` and all three arrive as `10 XX` immediately before the +terminating `ETX`. No captured *request* happened to land on one, so Thor's +behaviour there is unobserved — but a device and its host share one framing +routine, and the alternative is a frame the far end truncates, so escape it. + +This supersedes nothing: the de-stuffing rule `10 XX → XX` stays exactly as +documented. Knowing only four values are ever escaped is what makes that +uniform rule *exact* rather than merely convenient — and it is what an +**encoder** needs, which the write path will. ### The probe response carries the data length @@ -573,8 +656,63 @@ Series III ignores a `5A` probe unless preceded by answers a **bare `5A` request** with nothing before it. That whole ritual is gone. +### ⚠ Correction, 2026-09-27 — THOR uses a 1024-byte chunk loop + +The section below ("One request returns the entire event; there is no chunk +loop") describes **our own probes**, and it is not how THOR downloads an event. +Read from the wire, THOR's sequence per event is: + +``` +0x93 (arm) → 0x1E / 0x1F → key + size +0x0C → the 210-byte record +0x5A × n → n = ceil(size / 1024) +``` + +and the `0x5A` frames are a plain bounded chunk walk: + +| | | +|---|---| +| chunk `i` offset | `min(1024, size − 1024·i)` — a **byte count** | +| chunk 0 params | `[key4][6 × 0x00]` — the key means "from the start" | +| chunk `i>0` params | `[00 00][uint16 BE of 1024·i][6 × 0x00]` | +| response data | exactly `offset + 11` bytes; the file bytes are `data[11:]` | +| response `page_key` | `offset // 256` — a page count, not an address | + +**Verified on all six bench events**, sizes 4,076 → 13,424 bytes: +`sum(offsets) == size` **exactly** in every case, with the predicted chunk +count and predicted final offset. No `STRT` parsing, no `TERM` frame, no +over-read — that part of the original claim holds, and it is still far simpler +than the Series III walk. + +``` +key size(1E) chunks sum(offsets) last offset +055d4a81 4076 4 4076 0x03ec +055d4a82 11032 11 11032 0x0318 +055d4a83 11502 12 11502 0x00ee +055d4a84 13424 14 13424 0x0070 +055d4a85 8746 9 8746 0x022a +055d4a86 6092 6 6092 0x03cc +``` + +**These are probably two different modes, not a contradiction.** THOR's +`offset_hi` is the chunk length (`0x04`, `0x03`, `0x00` …). Our single-request +probes set `offset_hi = 0x10` — which in Series III is precisely the +bulk-stream marker `build_5a_frame()` writes raw. So `0x10XX` plausibly means +"stream until done" and returns **several** frames, which the parser of the day +concatenated into the 11,049 bytes recorded below. That reconciles both +observations, but it is a hypothesis: those 2026-09-23 captures never landed in +the repo, so it cannot be re-derived from bytes on disk. + +**Implement THOR's chunked form.** It is verified byte-exact across six events +and five distinct sizes, and it is what the firmware runs every day. The +single-request form is worth one bench test as an optimisation — `offset_hi = +0x10` and count the frames — but not worth depending on first. + ### The offset word is a LENGTH, not a position +⚠ Superseded as the implementation path by the correction above; the *reading* +of the field is right and is what makes THOR's chunk walk make sense. + This is the key divergence. Series III walks chunks by absolute flash address, stepping `0x0200` per request. On the Micromate the offset word requests *how much to send*: @@ -591,10 +729,11 @@ offset_word = 0x1000 + 2 × pages pages = ceil(event_size / 512) | `0x102C` | 22 | **11,033 — the whole event** | `event_size` comes from the chain walk (the 4 bytes after the key in -`1E`/`1F`). **One request returns the entire event**; there is no chunk loop, -no `STRT` end-offset parsing, and no `TERM` frame. Over-requesting is safe — -`0x1030` (24 pages) returned exactly the same bytes as `0x102C`, so the device -caps at the real size. +`1E`/`1F`). **One request returns the entire event** ⚠ *— true of our +`offset_hi = 0x10` probes; see the 2026-09-27 correction above. THOR chunks* — +there is no `STRT` end-offset parsing and no `TERM` frame in either form. +Over-requesting is safe — `0x1030` (24 pages) returned exactly the same bytes +as `0x102C`, so the device caps at the real size. Params are the Series III *probe* form: `[0x00][key4][6 × 0x00]`. @@ -636,10 +775,11 @@ per-sample-exact applies. ### What a full read now looks like ``` +0x93 → arm (THOR sends this before every 1E/1F) 1E → first key + size 0C(key) → project/client/operator, timestamp, peaks - 5A(key, 0x1000+2×ceil(size/512)) → the whole .IDFW -1F → next key + size (until null sentinel) + 5A × ceil(size/1024) → the .IDFW, 1024 bytes at a time +0x93 → 1F → next key + size (until null sentinel) ``` ## Setups are FILES, not a config block diff --git a/micromate/framing.py b/micromate/framing.py new file mode 100644 index 0000000..abf16a6 --- /dev/null +++ b/micromate/framing.py @@ -0,0 +1,304 @@ +""" +framing.py — frame codec for the Instantel Micromate (Series IV) wire protocol. + +A Micromate answers Series III *command* frames, so the request side looks +familiar. The framing underneath is not the same, and the differences are all +of the kind that produce a silently-ignored frame rather than an error: + + Series III response: [DLE 0x10] [STX 0x02] … [chk] [ETX 0x03] + Micromate response: [STX 0x02] … [chk] [ETX 0x03] + ^ no leading DLE + +That missing byte is why `minimateplus.framing.S3FrameParser` returns *nothing* +on Micromate traffic — it locates frames by scanning for `DLE STX`, which never +occurs. A capture holding 12 acknowledged writes reads as 12 unanswered +requests. + +De-stuffed payload layout (both directions): + + request response + [0] CMD 0x10 [0] CMD 0x00 + [1] flags 0x00 [1] flags 0xC5 / 0x03 ← firmware line + [2] SUB [2] SUB 0xFF − request_SUB + [3] 0x00 [3] PAGE_HI + [4] offset_hi [4] PAGE_LO + [5] offset_lo [5+] data + [6:16] params (10 bytes) + +Everything below was established against the 251 request and 251 response +frames in `bridges/captures/9-24-26 - micromate2/` (UM12947, firmware 11.0CB). +Where a rule is asserted, the number of frames it was checked on is given — the +two rules that look like small details cost 26% and 22% of frames respectively +when guessed wrong, so the counts are the point. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Optional + +# ── Protocol byte constants ─────────────────────────────────────────────────── + +DLE = 0x10 # Data Link Escape +STX = 0x02 # Start of text — begins a frame +ETX = 0x03 # End of text — ends a frame +ACK = 0x41 # Frame-start marker on the request side + +MM_CMD = 0x10 # payload[0] in a request +MM_RSP_CMD = 0x00 # payload[0] in a response + +# payload[1] of a response identifies the firmware line it came from. +# ⚠ Two units, one of each — a strong hypothesis, not a proven encoding. +FLAGS_BLASTWARE = 0xC5 # the 11.0CB line (UM12947) +FLAGS_THOR = 0x03 # the 11.0BD line (UM20147) + +# ⚠ THE ESCAPE SET. A Micromate escapes exactly these four byte values, +# prefixing each with a DLE — and nothing else. Established by re-stuffing +# every captured frame and comparing to the wire: 251/251 responses and 251/251 +# requests reproduce byte-for-byte with this set, and no other candidate set +# reproduces even 200 of either. +# +# The two near-misses are worth naming, because both look plausible: +# * `{0x10}` alone — the Series III rule — reproduces 130/251 responses and +# 177/251 requests. +# * adding ACK (0x41) reproduces only 196/251 responses: a literal 0x41 in +# the data is NOT escaped. +_ESCAPED = frozenset({STX, ETX, 0x04, DLE}) + +# A response header is 5 bytes; a frame must also carry its checksum. +_MIN_PAYLOAD = 5 +_REQUEST_PAYLOAD_SIZE = 16 + + +# ── Stuffing ────────────────────────────────────────────────────────────────── + +def stuff(data: bytes) -> bytes: + """Escape every byte the Micromate escapes: `XX` → `10 XX` for the four.""" + out = bytearray() + for b in data: + if b in _ESCAPED: + out.append(DLE) + out.append(b) + return bytes(out) + + +def unstuff(data: bytes) -> bytes: + """Reverse `stuff()`: `10 XX` → `XX`, for any XX. + + Uniform, with no inner-frame carve-out — which is a real simplification + over Series III, where `DLE+ETX` inside a frame is literal data that must + survive de-stuffing. Since only four byte values are ever escaped, taking + *any* `10 XX` as `XX` is exact rather than merely convenient. + """ + out = bytearray() + i = 0 + while i < len(data): + if data[i] == DLE and i + 1 < len(data): + out.append(data[i + 1]) + i += 2 + else: + out.append(data[i]) + i += 1 + return bytes(out) + + +# ── Checksum ────────────────────────────────────────────────────────────────── + +def checksum(payload: bytes) -> int: + """SUM8 of the **de-stuffed** payload, mod 256. 251/251 both directions. + + ⚠ Do NOT exclude `0x10` bytes from this sum. The DLE-aware checksum that + Series III uses for its `5A` and write frames is the right answer to a + *different* question: it pairs with Series III de-stuffing, which leaves + escaped bytes in the payload as two bytes. De-stuffing uniformly already + removes the DLE, so excluding `0x10` as well subtracts the correction + twice. + + That combination — uniform de-stuffing *and* an exclusive sum — is what + `scratch/mm_frame_parse.py` shipped with. It disagrees with the wire on + **55 of 251** captured response frames, all of them frames whose payload + holds a literal `0x10`. The script only ever looked correct because it + accepts a frame that matches *either* rule, so it reported those 55 as + plain SUM8 and never flagged one bad. + """ + return sum(payload) & 0xFF + + +# ── Request builder ─────────────────────────────────────────────────────────── + +def build_request(sub: int, offset: int = 0, params: bytes = bytes(10)) -> bytes: + """Build a host→unit command frame. + + ⚠ Do **not** substitute `minimateplus.framing.build_bw_frame()` here, even + though the payload layout is identical. That builder escapes only `0x10`, + so it reproduces just **161 of Thor's 218** captured read frames. The 57 it + gets wrong are not edge cases: + + * every `SUB 0x5A` bulk download — `offset = 0x0400` puts a literal + `0x04` in `offset_hi`, which must go out as `10 04` + * `SUB 0x47` (scheduler enable), whose params carry a `0x03` + + An unescaped `0x03` or `0x04` reads as a frame terminator, so the unit sees + a truncated frame and simply does not answer. That is indistinguishable + from a dead unit, and event download would have hit it on the first try. + + With the correct escape set this builder reproduces **218/218**. + + Args: + sub: command SUB byte. + offset: uint16 at payload[4:5]. Micromate reads are single-step — + Thor asks for `0xFFFF` and gets the whole block — so this is + usually `0xFFFF`, not Series III's probe-then-data pair. + params: exactly 10 bytes at payload[6:16]. + + A `0x10` inside `params` is fine and needs no special handling: Thor sends + `SUB 0x5A` with `params = 00 00 10 00 …` and the wire carries `10 10`. + (This was the spec's one open question; five captured frames settle it.) + """ + if len(params) != 10: + raise ValueError(f"params must be exactly 10 bytes, got {len(params)}") + if not 0 <= offset <= 0xFFFF: + raise ValueError(f"offset must fit in uint16, got {offset:#x}") + if not 0 <= sub <= 0xFF: + raise ValueError(f"sub must be a single byte, got {sub:#x}") + + payload = bytes([MM_CMD, 0x00, sub, 0x00, (offset >> 8) & 0xFF, offset & 0xFF]) + params + body = payload + bytes([checksum(payload)]) + return bytes([ACK, STX]) + stuff(body) + bytes([ETX]) + + +# ── Response frame ──────────────────────────────────────────────────────────── + +@dataclass +class MicromateFrame: + """A parsed, de-stuffed unit→host response frame.""" + + sub: int # response SUB; the request was 0xFF − this + flags: int # payload[1] — 0xC5 Blastware line, 0x03 Thor line + page_hi: int + page_lo: int + data: bytes # payload[5:], checksum stripped + checksum_valid: bool + chk_byte: int = 0 # the checksum byte as received + + @property + def request_sub(self) -> int: + """The SUB this is answering. No known exception to `0xFF − SUB`.""" + return 0xFF - self.sub + + @property + def page_key(self) -> int: + """payload[3:5] as a uint16 BE — a page/address on `0x5A` responses.""" + return (self.page_hi << 8) | self.page_lo + + @property + def firmware_line(self) -> str: + return {FLAGS_BLASTWARE: "blastware", FLAGS_THOR: "thor"}.get(self.flags, "unknown") + + @property + def probe_length(self) -> Optional[int]: + """Data length declared by a **probe** response: uint16 BE at data[3:5]. + + ⚠ Only meaningful in the reply to an `offset = 0` probe. Series III + hardcodes a `DATA_LENGTHS` table; a Micromate will tell you instead, + which already caught one divergence (call-home config is `0x7E`, where + Series III has `0x7C`). + + ⚠ It is a **uint16 BE**, not a byte. Read as `data[3]` alone it is + right only while the high byte is zero, and wrong by 47x for + `SUB 0x1A`: a true `0x082C` (2092) reads as 44. + + Returns None on a frame too short to hold the field. Note this reads + as 0 on the single-step reads Thor actually uses — those are not probes, + and `page_key` is the meaningful field there. + """ + if len(self.data) < 5: + return None + return (self.data[3] << 8) | self.data[4] + + +# ── Streaming parser ────────────────────────────────────────────────────────── + +class MicromateFrameParser: + """Incremental parser for unit→host frames. Mirrors `S3FrameParser`. + + Feed bytes with `feed()`; completed frames are returned and also collected + in `.frames`. + + IDLE — scanning for a bare STX + IN_FRAME — collecting; bare ETX terminates + AFTER_DLE — the next byte is literal, whatever it is + + Request frames are rejected rather than parsed: a frame whose `payload[0]` + is not `0x00` is dropped, so feeding a bidirectional capture yields only + the responses. + """ + + _IDLE, _IN_FRAME, _AFTER_DLE = 0, 1, 2 + + def __init__(self) -> None: + self._state = self._IDLE + self._body = bytearray() + self.frames: list[MicromateFrame] = [] + # Distinguishes "no bytes at all" from "bytes but no complete frame" on + # a timeout. That distinction earned its keep during the Series III + # work and costs one integer here. + self.bytes_fed: int = 0 + + def reset(self) -> None: + self._state = self._IDLE + self._body.clear() + self.bytes_fed = 0 + + def feed(self, data: bytes) -> list[MicromateFrame]: + self.bytes_fed += len(data) + completed: list[MicromateFrame] = [] + for b in data: + frame = self._step(b) + if frame is not None: + completed.append(frame) + self.frames.append(frame) + return completed + + def _step(self, b: int) -> Optional[MicromateFrame]: + if self._state == self._IDLE: + if b == STX: + self._body.clear() + self._state = self._IN_FRAME + # Boot strings, modem RING/CONNECT chatter and stray ACKs land here + # and are discarded. + + elif self._state == self._IN_FRAME: + if b == DLE: + self._state = self._AFTER_DLE + elif b == ETX: + self._state = self._IDLE + return self._finalise() + else: + self._body.append(b) + + elif self._state == self._AFTER_DLE: + # Uniform rule: the escaped byte is itself, including 0x03. + self._body.append(b) + self._state = self._IN_FRAME + + return None + + def _finalise(self) -> Optional[MicromateFrame]: + body = bytes(self._body) + if len(body) < _MIN_PAYLOAD + 1: + return None + + payload, chk_received = body[:-1], body[-1] + if payload[0] != MM_RSP_CMD: + return None # a request frame, or garbage that framed by accident + + return MicromateFrame( + sub = payload[2], + flags = payload[1], + page_hi = payload[3], + page_lo = payload[4], + data = payload[5:], + checksum_valid = (chk_received == checksum(payload)), + chk_byte = chk_received, + ) diff --git a/scratch/mm_frame_parse.py b/scratch/mm_frame_parse.py index c0e6ebc..6bdd101 100644 --- a/scratch/mm_frame_parse.py +++ b/scratch/mm_frame_parse.py @@ -22,10 +22,30 @@ One rule covers both directions: after the leading doubled `BW_CMD`, every escapes literal `0x03` bytes in write data so they are not mistaken for ETX, exactly as Blastware does. -That rule was chosen by evidence, not assumption: of the four candidates tried -against the 9-24-26 capture's four data-carrying write frames, it is the only -one under which all four checksums validate. See -`docs/micromate_protocol_reference.md` → *The write path*. +A Micromate escapes exactly four byte values — `0x02 0x03 0x04 0x10` — and +nothing else, which is what makes the uniform rule exact rather than merely +convenient. Established by re-stuffing all 502 captured frames and comparing +to the wire: 251/251 each direction, where `{0x10}` alone gets 130 and 177. + +⚠ Checksum, corrected 2026-09-27 +-------------------------------- +With uniform destuffing the checksum is **plain SUM8 of the destuffed +payload**. This script used to try SUM8 *and* a "DLE-aware" variant that +excludes `0x10` bytes, and report whichever matched — which is why it never +flagged a bad frame and why the protocol reference carried the wrong rule for +two days. The DLE-aware form belongs with *Series III* destuffing, where an +escaped byte survives as two bytes; applying it after uniform destuffing +subtracts the correction twice and disagrees with the wire on 55 of 251 +responses. + +The lesson generalises: a tool that accepts any of N candidate rules cannot +falsify any of them. It now validates against SUM8 alone, and reports +`DLE-aware` only to name what a mismatch *would* have been — never as a pass. +See `docs/micromate_protocol_reference.md` → *Checksum*, and +`tests/test_micromate_framing.py`, which pins it. + +`micromate/framing.py` is the production implementation; this stays as the +one-shot capture-inspection tool. Usage ----- @@ -119,12 +139,13 @@ def frames(blob: bytes, *, is_request: bool): except ValueError: i += 1 continue - sum8 = sum(payload) & 0xFF - dle_aware = (sum(b for b in payload if b != DLE) & 0xFF) - if sum8 == chk: - kind = "SUM8" - elif dle_aware == chk: - kind = "DLE-aware" + # SUM8 of the destuffed payload is THE rule -- 502/502 captured frames. + # The DLE-aware variant is reported only to name a near-miss; it is + # never a pass. See the module docstring. + if (sum(payload) & 0xFF) == chk: + kind = "ok" + elif (sum(b for b in payload if b != DLE) & 0xFF) == chk: + kind = "BAD(dle-aware)" else: kind = "BAD" yield i, payload, chk, kind diff --git a/tests/test_micromate_framing.py b/tests/test_micromate_framing.py new file mode 100644 index 0000000..d2c15b2 --- /dev/null +++ b/tests/test_micromate_framing.py @@ -0,0 +1,399 @@ +"""Framing tests for the Micromate (series-4) live protocol. + +Every constant below is a **real frame**, lifted from +``bridges/captures/9-24-26 - micromate2/`` (UM12947, firmware 11.0CB) or from +``scratch/fake_unit.py``, which preserves a POLL probe reply. Frames are +embedded as hex rather than read from disk because both ``bridges/captures/`` +and ``tests/fixtures/`` are gitignored -- these tests must pass on a fresh +clone. + +The few synthesised frames are marked ``SYNTH_`` and each says what it stands +in for and why a captured frame was not available. + +Two of these tests exist because the first draft of +``docs/micromate_client_spec.md`` got the rule wrong, and both wrong rules +fail quietly -- a frame the unit ignores, or a checksum that reads as bad: + + * ``test_builder_matches_thor_byte_for_byte`` -- the escape set. Escaping + only 0x10 (the series-3 rule) reproduces 161 of Thor's 218 read frames. + * ``test_checksum_is_plain_sum8_over_destuffed_payload`` -- the checksum. + The DLE-aware variant disagrees with the wire on 55 of 251 responses. +""" +from __future__ import annotations + +import os +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from micromate.framing import ( + ACK, + DLE, + ETX, + FLAGS_BLASTWARE, + FLAGS_THOR, + STX, + MicromateFrame, + MicromateFrameParser, + build_request, + checksum, + stuff, + unstuff, +) + +# ── Captured response frames ────────────────────────────────────────────────── + +# POLL probe reply, 19 B on the wire -- the shortest valid frame there is. +# Preserved in scratch/fake_unit.py, captured from UM12947 on 2026-09-24. +# Its payload[8:10] is 0x0030, the data length POLL then asks for. +RSP_POLL_PROBE = bytes.fromhex("0200c5a400000000000030000000000000009903") + +# SUB 0x49 -> 0xB6, the cheap state read. Carries a literal 0x02, escaped. +RSP_STATE = bytes.fromhex("0200c5b6000005000000000000000000001002e8000b007503") + +# SUB 0x48 -> 0xB7, a path-addressed file read. offset_hi 0x04 arrives escaped, +# so page_hi is only correct if the parser destuffs before indexing. +RSP_FILE_READ = bytes.fromhex("0200c5b70000100400000000000000010000000000018203") + +# An 0x5A download chunk, 138 B on the wire. THE checksum case: its payload +# holds literal 0x10 bytes, so plain SUM8 (0xC1, correct) and the DLE-aware +# variant (0x91) disagree. Also holds a literal 0x41, which is NOT escaped. +RSP_CHUNK_WITH_DLE = bytes.fromhex( + "0200c5a500007000003400000000000000e3fd1f10020f0f0e2e1e1fd4f2d4c3d2f00f000" + "e003fe03fe200d6e2d12d2e1c4e101011d101f3d0f3e0f2efe23d2b2e3c0d3e0e2c4101e0" + "d4d3e21f1d311e202fe2e2f1e3f100210d201002ee2e22d2f2f0f30fd3f0101000f0111f1" + "ff01e011010011f0f1002b4d200e0f0302c4010020001fffedb2dc103" +) + +# ── Captured request frames (Thor -> unit) ──────────────────────────────────── + +REQ_POLL = bytes.fromhex("41021010005b000030000000000000000000009b03") +REQ_SERIAL = bytes.fromhex("41021010001500000a000000000000000000002f03") +REQ_STATE = bytes.fromhex("41021010004900ffff000000000000000000005703") +REQ_COMPLIANCE = bytes.fromhex("41021010001a00ffff000000000000000000002803") +REQ_ARM_EVENT = bytes.fromhex("41021010009300ffff00000000000000000000a103") + +# ⚠ The two frames that break a 0x10-only escaper. +# Scheduler enable: params[7] = 0x03, on the wire as `10 03`. +REQ_SCHED_ON = bytes.fromhex("41021010004700ffff00000000000000100300005803") +# Bulk download, first chunk of event 055d4a81: offset 0x0400 puts a literal +# 0x04 in offset_hi, on the wire as `10 04`. EVERY download frame needs this. +REQ_DOWNLOAD_CHUNK0 = bytes.fromhex( + "41021010005a00100400055d4a810000000000009b03" +) +# A later chunk of the same event: params[2:4] = 0x1000, doubled to `10 10`. +REQ_DOWNLOAD_CHUNK4 = bytes.fromhex( + "41021010005a0010040000001010000000000000007e03" +) + + +# ── Stuffing ────────────────────────────────────────────────────────────────── + +def test_escape_set_is_exactly_four_bytes(): + """0x02, 0x03, 0x04 and 0x10 -- and nothing else. + + ACK (0x41) in particular is NOT escaped; assuming it was reproduced only + 196 of 251 captured responses. + """ + assert stuff(bytes([0x02, 0x03, 0x04, 0x10])) == bytes( + [DLE, 0x02, DLE, 0x03, DLE, 0x04, DLE, 0x10] + ) + for b in (0x00, 0x01, 0x05, 0x41, 0xC5, 0xFF): + assert stuff(bytes([b])) == bytes([b]), f"0x{b:02x} must not be escaped" + + +def test_unstuff_is_uniform_with_no_inner_frame_carve_out(): + """`10 XX` -> `XX` for any XX -- the series-3 DLE+ETX exception is absent.""" + assert unstuff(bytes.fromhex("1003")) == b"\x03" + assert unstuff(bytes.fromhex("1010")) == b"\x10" + assert unstuff(bytes.fromhex("001002ff")) == bytes.fromhex("0002ff") + + +def test_stuff_unstuff_round_trips_over_every_byte_value(): + data = bytes(range(256)) + assert unstuff(stuff(data)) == data + + +def test_a_trailing_dle_is_held_not_dropped(): + """A DLE as the last byte of a chunk must not consume nothing and vanish.""" + assert unstuff(b"\xff\x10") == b"\xff\x10" + + +# ── Checksum ────────────────────────────────────────────────────────────────── + +def test_checksum_is_plain_sum8_over_destuffed_payload(): + """⚠ Plain SUM8 -- do NOT exclude 0x10 bytes. + + RSP_CHUNK_WITH_DLE is a real frame whose payload holds literal 0x10 bytes. + The wire says 0xC1; plain SUM8 gives 0xC1 and the DLE-aware variant used by + series-3 `5A`/write frames gives 0x91. 55 of 251 captured responses + disagree the same way. + """ + payload = unstuff(RSP_CHUNK_WITH_DLE[1:-1])[:-1] + chk_on_wire = unstuff(RSP_CHUNK_WITH_DLE[1:-1])[-1] + + assert DLE in payload, "this frame is only interesting if it holds a 0x10" + assert checksum(payload) == chk_on_wire == 0xC1 + assert (sum(b for b in payload if b != DLE) & 0xFF) == 0x91 # the wrong rule + + +def test_every_captured_frame_validates(): + parser = MicromateFrameParser() + frames = parser.feed( + RSP_POLL_PROBE + RSP_STATE + RSP_FILE_READ + RSP_CHUNK_WITH_DLE + ) + assert len(frames) == 4 + assert all(f.checksum_valid for f in frames) + + +# ── Request builder ─────────────────────────────────────────────────────────── + +@pytest.mark.parametrize( + "wire, sub, offset, params", + [ + (REQ_POLL, 0x5B, 0x0030, bytes(10)), + (REQ_SERIAL, 0x15, 0x000A, bytes(10)), + (REQ_STATE, 0x49, 0xFFFF, bytes(10)), + (REQ_COMPLIANCE, 0x1A, 0xFFFF, bytes(10)), + (REQ_ARM_EVENT, 0x93, 0xFFFF, bytes(10)), + (REQ_SCHED_ON, 0x47, 0xFFFF, bytes.fromhex("00000000000000030000")), + (REQ_DOWNLOAD_CHUNK0, 0x5A, 0x0400, bytes.fromhex("055d4a81000000000000")), + (REQ_DOWNLOAD_CHUNK4, 0x5A, 0x0400, bytes.fromhex("00001000000000000000")), + ], + ids="poll serial state compliance arm sched_on dl_chunk0 dl_chunk4".split(), +) +def test_builder_matches_thor_byte_for_byte(wire, sub, offset, params): + assert build_request(sub, offset, params) == wire + + +def test_a_0x10_in_params_needs_no_special_handling(): + """The spec's one open question. Thor sends it; the wire doubles it.""" + frame = build_request(0x5A, 0x0400, bytes.fromhex("00001000000000000000")) + assert bytes([DLE, DLE]) in frame + assert frame == REQ_DOWNLOAD_CHUNK4 + + +def test_builder_rejects_malformed_arguments(): + with pytest.raises(ValueError): + build_request(0x5B, 0, bytes(9)) + with pytest.raises(ValueError): + build_request(0x5B, 0x10000) + with pytest.raises(ValueError): + build_request(0x100) + + +# ── Parsing ─────────────────────────────────────────────────────────────────── + +def test_poll_probe_reply_fields(): + (f,) = MicromateFrameParser().feed(RSP_POLL_PROBE) + assert f.sub == 0xA4 + assert f.request_sub == 0x5B + assert f.flags == FLAGS_BLASTWARE + assert f.firmware_line == "blastware" + assert f.checksum_valid + assert f.probe_length == 0x0030 + + +def test_probe_length_is_a_uint16_not_a_byte(): + """⚠ Read as data[3] alone, SUB 0x1A's 0x082C (2092) reads as 44 -- 47x low. + + SYNTHESISED: no probe response survives in the captures on disk (the + 9-24-26 session uses single-step reads at offset 0xFFFF throughout, so it + contains no probes at all). The field position is taken from the captured + POLL probe reply above, which does exercise it for real. + """ + payload = bytes([0x00, FLAGS_BLASTWARE, 0xE5, 0x00, 0x00]) + bytes( + [0x00, 0x00, 0x00, 0x08, 0x2C] + ) + synth = bytes([STX]) + stuff(payload + bytes([checksum(payload)])) + bytes([ETX]) + + (f,) = MicromateFrameParser().feed(synth) + assert f.probe_length == 0x082C == 2092 + assert f.data[3] == 0x08, "the high byte is where a byte-wide read loses 2048" + + +def test_escaped_bytes_land_in_the_right_field(): + """RSP_FILE_READ's first data byte is 0x04, which arrives as `10 04`. + + Without destuffing it reads as 0x10 and every field after it is one byte + late -- the failure mode that put `SUB 0x02` in the log as `SUB_10` for an + afternoon. + """ + (f,) = MicromateFrameParser().feed(RSP_FILE_READ) + assert f.sub == 0xB7 + assert f.request_sub == 0x48 + assert (f.page_hi, f.page_lo) == (0x00, 0x00) + assert f.data[0] == 0x04 + assert len(f.data) == 15, "one byte shorter than the wire suggests" + assert f.checksum_valid + + +def test_thor_firmware_line_survives_destuffing(): + """⚠ flags = 0x03 is ETX, so it arrives as `10 03`. + + A parser that does not destuff ends the frame at byte 2 on half the fleet. + + SYNTHESISED: no 11.0BD capture is on disk -- UM20147's sweep was recorded + on 2026-09-23 and those bins never landed in the repo. Built by re-stuffing + the captured POLL probe reply's payload with flags flipped to 0x03, so the + only difference from a real frame is the one byte under test. + """ + real = unstuff(RSP_POLL_PROBE[1:-1])[:-1] + payload = bytes([real[0], FLAGS_THOR]) + real[2:] + synth = bytes([STX]) + stuff(payload + bytes([checksum(payload)])) + bytes([ETX]) + + assert bytes([DLE, ETX]) in synth, "flags must be escaped on the wire" + (f,) = MicromateFrameParser().feed(synth) + assert f.flags == FLAGS_THOR + assert f.firmware_line == "thor" + assert f.sub == 0xA4 + assert f.checksum_valid + + +def test_an_escaped_checksum_byte_is_read_correctly(): + """SYNTHESISED, but the behaviour is real: three captured responses have a + checksum of 0x02/0x03/0x04 and all three escape it on the wire. The + shortest is 1,070 B (an 0x5A chunk in + raw_s3_20260925_011403_Download_events_then_delete_1_event.bin, chk = 0x03), + too long to embed for one byte's worth of assertion. + """ + payload = bytes([0x00, FLAGS_BLASTWARE, 0xA4, 0x00, 0x00, 0x03]) + assert checksum(payload) == 0x03 + FLAGS_BLASTWARE + 0xA4 & 0xFF + body = payload + bytes([checksum(payload)]) + synth = bytes([STX]) + stuff(body) + bytes([ETX]) + + (f,) = MicromateFrameParser().feed(synth) + assert f.chk_byte == checksum(payload) + assert f.checksum_valid + + +def test_a_corrupted_checksum_still_yields_a_frame(): + """Flag it, do not swallow it -- a dropped frame looks like a dead unit.""" + broken = bytearray(RSP_POLL_PROBE) + broken[-2] ^= 0xFF + (f,) = MicromateFrameParser().feed(bytes(broken)) + assert f.sub == 0xA4 + assert not f.checksum_valid + + +def test_a_truncated_frame_yields_nothing(): + parser = MicromateFrameParser() + assert parser.feed(RSP_POLL_PROBE[:-1]) == [] + assert parser.frames == [] + assert parser.bytes_fed == len(RSP_POLL_PROBE) - 1 + + +def test_a_frame_too_short_to_hold_a_header_is_rejected(): + assert MicromateFrameParser().feed(bytes([STX, 0x00, 0xC5, 0xA4, ETX])) == [] + + +def test_request_frames_are_not_mistaken_for_responses(): + """Feeding a bidirectional capture must yield only the unit's side.""" + parser = MicromateFrameParser() + frames = parser.feed(REQ_POLL + RSP_POLL_PROBE + REQ_SERIAL) + assert len(frames) == 1 + assert frames[0].request_sub == 0x5B + + +def test_leading_noise_is_discarded(): + """Cold-boot banners and the RV55's RING/CONNECT chatter precede frames.""" + noise = b"\r\nRING\r\n\r\nCONNECT\r\n" + bytes([ACK]) + (f,) = MicromateFrameParser().feed(noise + RSP_POLL_PROBE) + assert f.sub == 0xA4 + assert f.checksum_valid + + +def test_frames_split_across_feeds_reassemble(): + """The transport hands over whatever the socket returned, DLE pairs and all.""" + whole = RSP_STATE + RSP_CHUNK_WITH_DLE + for cut in (1, 2, 5, 17, 24, 25, 40, len(RSP_STATE), len(whole) - 1): + parser = MicromateFrameParser() + got = parser.feed(whole[:cut]) + parser.feed(whole[cut:]) + assert len(got) == 2, f"split at {cut} lost a frame" + assert all(f.checksum_valid for f in got), f"split at {cut} broke a checksum" + + +def test_reset_clears_partial_state(): + parser = MicromateFrameParser() + parser.feed(RSP_POLL_PROBE[:6]) + parser.reset() + assert parser.bytes_fed == 0 + (f,) = parser.feed(RSP_POLL_PROBE) + assert f.checksum_valid + + +def test_firmware_line_of_an_unknown_flags_byte(): + f = MicromateFrame(sub=0xA4, flags=0x99, page_hi=0, page_lo=0, + data=b"", checksum_valid=True) + assert f.firmware_line == "unknown" + assert f.probe_length is None + + +# ── Whole-session round trip ────────────────────────────────────────────────── + +_CAPTURES = ( + Path(__file__).resolve().parents[1] + / "bridges" / "captures" / "9-24-26 - micromate2" +) + + +@pytest.mark.skipif( + not _CAPTURES.is_dir(), + reason="capture directory is gitignored; present only on a dev box", +) +def test_whole_captured_sessions_parse_with_no_bad_checksums(): + """Belt-and-braces against the real bytes when they happen to be here. + + scratch/mm_frame_parse.py reported zero bad checksums on these sessions + only because it accepts a frame matching *either* checksum rule. This + asserts the single rule holds across all of them. + """ + total = 0 + for path in sorted(_CAPTURES.rglob("raw_s3_*.bin")): + parser = MicromateFrameParser() + frames = parser.feed(path.read_bytes()) + assert frames, f"{path.name}: no frames parsed" + bad = [f for f in frames if not f.checksum_valid] + assert not bad, f"{path.name}: {len(bad)} bad checksums" + total += len(frames) + assert total == 251, f"expected 251 response frames across the corpus, got {total}" + + +@pytest.mark.skipif( + not _CAPTURES.is_dir(), + reason="capture directory is gitignored; present only on a dev box", +) +def test_builder_reproduces_every_captured_read_frame(): + """218/218. This is the test that would have caught the escape-set error.""" + checked = 0 + for path in sorted(_CAPTURES.rglob("raw_bw_*.bin")): + blob = path.read_bytes() + i = 0 + while i < len(blob): + if not (blob[i] == ACK and i + 1 < len(blob) and blob[i + 1] == STX): + i += 1 + continue + j = i + 2 + body = bytearray() + while j < len(blob): + if blob[j] == DLE and j + 1 < len(blob): + body.append(blob[j + 1]) + j += 2 + continue + if blob[j] == ETX: + break + body.append(blob[j]) + j += 1 + payload = bytes(body[:-1]) + if len(payload) == 16: # a read frame; writes carry a data section + sub = payload[2] + offset = (payload[4] << 8) | payload[5] + assert build_request(sub, offset, payload[6:16]) == blob[i:j + 1], ( + f"{path.name} @0x{i:04x} SUB=0x{sub:02x} offset=0x{offset:04x}" + ) + checked += 1 + i = j + 1 + assert checked == 218, f"expected 218 read frames, checked {checked}" From a7e3a8f20a9c18182a89a92fa81d9c12c158fc08 Mon Sep 17 00:00:00 2001 From: serversdown Date: Mon, 28 Sep 2026 19:58:22 -0400 Subject: [PATCH 07/16] feat(micromate): protocol layer -- reads only, verified against Thor's frames Step 2 of docs/micromate_client_spec.md: micromate/protocol.py plus 35 offline tests. Reads only; nothing here writes, erases or changes monitoring state. The tests replay Thor's captured responses through a scripted transport and assert the bytes we emit are the bytes Thor emits -- including a full replay of the six-event download session, all 56 0x5A frames byte-for-byte. A passing test therefore means a real unit has already answered exactly that frame. Measuring the spec's command table against the captures found three more errors in it, on top of the three the framing work found: 1. SUB 0x0A is the MONITOR-LOG WALK, not a keyed "event header, 30 B list record" read. The same request repeated returns successive 297-byte records -- serial, mode, thresholds -- until an 11-byte ack ends the list. The device holds the cursor; nothing in the request selects a record, and all nine captured frames carry identical params. Structural divergence worth noting: Series III reaches the same data via a record-type discriminator on its event chain, so partials and events share one walk. Here the monitor log has its own cursor and the event chain never sees it. 2. 0x1E/0x1F carry token 0xFE at params[7]. The protocol reference documents all-zero params -- that was our own browse probing, which also worked. Thor sends 0xFE on browse and download alike. 3. SUB 0x01 (device info) is never read by Thor in any captured session. Its 0xFFFF offset comes from our own probes, so it is the one read in the table with no Thor precedent. Flagged in the docstring. Two useful negatives, both from absence rather than presence: - No SESSION_RESET (41 03). Series III needs that 2-byte signal or a monitoring unit will not answer POLL over TCP. Zero occurrences across all 8 sessions, including 40 frames exchanged with a unit that WAS monitoring. - No universal preamble. The only invariant is that a session opens with POLL; POLL -> SERIAL -> 0x49 -> POLL is Thor's connection check and appears in 3 of 8 sessions. Setup pushes and scheduler reads open differently. Two deliberate divergences from the Series III sibling: - strict_checksums defaults True and RAISES. minimateplus logs and continues because its parser cannot always tell an inner-frame delimiter from a checksum byte; that does not apply here, where the rule is exact on 251/251 frames. The lenient default is instructive -- it hid a wrong checksum rule for two days. - read_event_file() raises ShortRead rather than returning a truncated event. The expected length is known up front, so the check is free, and a silently short event is the failure mode this codebase keeps hitting. Every exchange resets the parser before sending, so a leftover frame is discarded rather than answered with -- expected_sub catches a mismatched SUB, but a same-SUB leftover would sail through with data for the wrong key. File transfer (0x94/0x48) is deliberately out of scope: it needs a data-carrying request frame, which is the frame type writes use, and that boundary is worth keeping crisp in a read-only pass. Full suite unchanged at 16 pre-existing failures (missing gitignored fixtures); 419 passed, up 35. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_client_spec.md | 53 ++- docs/micromate_protocol_reference.md | 90 +++++ micromate/protocol.py | 501 +++++++++++++++++++++++++++ tests/test_micromate_protocol.py | 402 +++++++++++++++++++++ 4 files changed, 1035 insertions(+), 11 deletions(-) create mode 100644 micromate/protocol.py create mode 100644 tests/test_micromate_protocol.py diff --git a/docs/micromate_client_spec.md b/docs/micromate_client_spec.md index ff5ca38..b6a619f 100644 --- a/docs/micromate_client_spec.md +++ b/docs/micromate_client_spec.md @@ -173,7 +173,16 @@ timeout, and that distinction earned its keep during the Series III work). --- -## `micromate/protocol.py` +## `micromate/protocol.py` — ✅ BUILT 2026-09-27 + +Implemented, with `tests/test_micromate_protocol.py` (35 tests). Reads only; +nothing here writes, erases or changes monitoring state. + +The tests replay Thor's captured responses through a scripted transport and +assert **the bytes we emit are the bytes Thor emits** — including a full replay +of the six-event download session, all 56 `0x5A` frames byte-for-byte. That is +a stronger guarantee than "our parser understands the device": a passing test +means a real unit has already answered exactly that frame. One method per command, returning raw payload bytes. No interpretation — that belongs in `client.py`. @@ -198,9 +207,29 @@ taking its data length. Per-command offsets, all observed: | arm event | `0x93` | `0x6C` | — | before every event | | first event | `0x1E` | `0xE1` | `0xFFFF` | key + size | | next event | `0x1F` | `0xE0` | `0xFFFF` | key + size | -| event record | `0x0C` | `0xF3` | `0xFFFF` | 210 B — project, location, peaks | -| event header | `0x0A` | `0xF5` | `0xFFFF` | 30 B list record | -| bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim** | +| event record | `0x0C` | `0xF3` | `0xFFFF` | 221 B — project, location, peaks | +| ~~event header~~ **monitor log** | `0x0A` | `0xF5` | `0xFFFF` | ⚠ 297 B, a **walk** — see below | +| bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim**, 1024 B at a time | + +⚠ **Corrected 2026-09-27, from Thor's frames.** Three rows of the table above +were wrong or incomplete, and the last one is a different command than labelled: + +- **`0x0A` is the monitor-log walk**, not a keyed "30 B list record" read. The + *same request repeated* returns successive 297-byte records — serial, mode, + thresholds — until an 11-byte ack ends the list. The device holds the cursor; + nothing in the request selects a record. Series III reaches this data through + a record-type discriminator on its event chain; here it has its own cursor and + the event chain never sees it. +- **`0x1E`/`0x1F` carry token `0xFE` at `params[7]`.** The reference documents + all-zero params (our own probing, which also worked). Thor's form is the one + with mileage. +- **`0x01` has no Thor frame behind it** — it is never read in any captured + session. Its `0xFFFF` comes from our probes. + +And one useful negative: **no `SESSION_RESET` (`41 03`)**. Series III needs that +2-byte signal or a monitoring unit will not answer `POLL` over TCP. Thor never +sends it — zero occurrences across 8 sessions, including 40 frames exchanged +with a unit that *was* monitoring. ⚠ **`SUB 0x1C` is 4 bytes longer on the Thor firmware line** (`0x30` vs `0x2C`). Parse **forward** from `declared_length`, never backward from the end — Series @@ -328,19 +357,21 @@ then `download_event()` and assert the bytes decode and match a ## Order of work 1. ✅ `framing.py` + its tests — **done 2026-09-27**, 31 tests, offline -2. `protocol.py` — reads only, one method per row of the table above +2. ✅ `protocol.py` + its tests — **done 2026-09-27**, 35 tests, offline 3. `client.py` — `connect()`, `get_state()`, `list_setups()` 4. the event chain and `download_event()` 5. decode end-to-end and compare against a store event Steps 1–2 need no hardware at all. -**Worth carrying forward from step 1:** every rule got checked against the -captures *before* being written, and two of the three the spec asserted turned -out wrong — the escape set (26% of frames) and the checksum (22%). Both fail -silently. The captures are on disk and a re-stuff-and-compare loop takes about -two minutes per rule, so do that for `protocol.py`'s per-command offsets too -rather than trusting the table above. +**Worth carrying forward.** Both steps began by measuring against the captures +rather than trusting this document, and both found errors in it — three in the +framing rules (the escape set, 26% of frames; the checksum, 22%; the `0x5A` +chunk model) and three more in the command table (`0x0A`'s meaning, the +`1E`/`1F` token, `0x01`'s provenance). All six fail quietly. The captures are on +disk and a measure-then-write loop costs about two minutes per rule, so keep +doing it for `client.py`'s field offsets — and treat this spec as a plan, not a +source. --- diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 4b87096..f444e0f 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1216,6 +1216,96 @@ the protocol's requirements*. Thor sends it before trivial reads too, so it may be habit rather than handshake. Do not assume it is mandatory. Note `POLL` here carries `offset = 0x0030` (its data length), not `0xFFFF` — `POLL` is the one read Thor still addresses by length. +> #### ⚠ Narrowed 2026-09-27 — there is no *universal* preamble +> +> Across all 8 captured sessions, the only invariant is that **the session opens +> with `POLL`**. What follows depends on the operation: +> +> | opening sequence | sessions | operation | +> |---|---|---| +> | `5b 15 49 5b …` | 3 | status refresh / monitoring / ACH change | +> | `5b 41 08 2e 1a da …` | 3 | setup push | +> | `5b 94 48 48 48 …` | 2 | scheduler read | +> +> So `POLL → SERIAL → 0x49 → POLL` is Thor's **connection check**, not a +> handshake the protocol demands — it appears where Thor wants to refresh what +> it displays. Treat `POLL` as the one thing to send first. + +### 🔑 No `SESSION_RESET` — the Series III requirement does not carry over + +Series III needs a bare `41 03` (ACK + ETX, no STX) to wake a unit that is +actively monitoring; without it the unit will not answer `POLL` over TCP, and +`protocol.startup()` sends it before and between the POLL frames. + +**Thor never sends it to a Micromate.** Zero occurrences across all 8 sessions +— including `raw_bw_20260924_191214_turn_on_monitormode_…`, which exchanges 40 +frames with a unit that *was* monitoring at the time. + +### Measured offsets and response lengths (all read off Thor's frames) + +`offset = 0xFFFF` for everything except two commands. Data lengths are from +UM12947 (`11.0CB`) and are **orientation, not assertions** — `0x1C` is 4 bytes +longer on the Thor line. + +| SUB | rsp | offset | data | notes | +|---|---|---|---|---| +| `0x5B` POLL | `0xA4` | **`0x0030`** | 59 | the one length-addressed read | +| `0x15` serial | `0xEA` | **`0x000A`** | 21 | | +| `0x49` state | `0xB6` | `0xFFFF` | 16 | | +| `0x1C` monitor status | `0xE3` | `0xFFFF` | 55 | +4 on `11.0BD` | +| `0x06` storage range | `0xF9` | `0xFFFF` | 47 | | +| `0x08` event index | `0xF7` | `0xFFFF` | 101 | contents unmapped | +| `0x2E` trigger config | `0xD1` | `0xFFFF` | 39 | | +| `0x1A` compliance | `0xE5` | `0xFFFF` | 2103 | one frame, not Series III's four | +| `0x2C` call-home | `0xD3` | `0xFFFF` | 137 | | +| `0x3F`/`0x40`/`0x41` setups | `0xC0`/`0xBF`/`0xBE` | `0xFFFF` | 266 | | +| `0x93` arm | `0x6C` | `0xFFFF` | 11 | ack only | +| `0x1E`/`0x1F` chain | `0xE1`/`0xE0` | `0xFFFF` | 19 | ⚠ **token `0xFE` at `params[7]`** | +| `0x0C` event record | `0xF3` | `0xFFFF` | 221 | full key at `params[4:8]` | +| `0x0A` monitor log | `0xF5` | `0xFFFF` | 297 | ⚠ a **walk** — see below | +| `0x5A` download | `0xA5` | computed | offset+11 | 1024-byte chunk loop | + +An acknowledgement is an **11-byte data section**, and that doubles as the +end-of-list signal on the walks. + +#### ⚠ `0x1E`/`0x1F` carry token `0xFE` + +`params = 00 00 00 00 00 00 00 fe 00 00` on all 7 captured chain reads — the +same `token_params(0xFE)` form Series III uses to arm its bulk stream. The +event-chain section above documents **all-zero params**; that was our own browse +probing, which also worked. Both evidently do, but Thor's form is the one with +mileage on it, and it is sent on browse and download alike. + +#### 🔑 `SUB 0x0A` is the monitor-log walk, not a keyed read + +The command table long described `0x0A` as a keyed "waveform header / partial +record" read, by analogy with Series III. What the bytes show is a **cursor +walk**: the *same request repeated*, the device advancing its own position. + +``` +0x93 → 1E → 0x0A ×8 (297 B each: "UM12947", "Histo…", "Ver…", " 0.49", " 28.4") + 0x0A (11 B ack = end of list) +``` + +All nine frames carry identical params (`…00 00 4a 81 00 00`), so nothing in the +request selects the record. Terminate on a response of `ACK_DATA_LEN` (11). + +⚠ The `4a 81` is the **low two bytes** of the event key then in play +(`055d4a81`). One key cannot distinguish "the key's low half" from "a cursor +handle that happened to equal it" — both produce those bytes. It does not +matter operationally, since the walk works with the params held constant. + +This is a genuine structural divergence: Series III reaches the same data +through a record-type discriminator (`0x2C` partial vs `0x46` full) *on its +event walk*, so partial records and events share one chain. Here the monitor +log has its own cursor and the event chain never sees it. + +#### `SUB 0x01` has no Thor frame behind it + +Thor never reads device info in any captured session. `0xFFFF` for `0x01` comes +from our own 2026-09-23 probes — it answered correctly on both firmware lines, +but it is the only read in the table with no Thor precedent. + ### `SUB 0x96` / `0x97` — start and stop monitoring ✅ Identical to Series III, including the acks: diff --git a/micromate/protocol.py b/micromate/protocol.py new file mode 100644 index 0000000..6e35a2f --- /dev/null +++ b/micromate/protocol.py @@ -0,0 +1,501 @@ +""" +protocol.py — one method per Micromate (Series IV) wire command. + +Returns raw payload bytes. Interpretation belongs in ``client.py``; this layer +knows frames, offsets and sequencing, and nothing about what a field means. + +Scope: **reads only.** Nothing here writes, erases, or changes monitoring +state. That is deliberate and worth keeping — no command has ever been +originated against a unit by this project; every write in +``docs/micromate_protocol_reference.md`` was performed by THOR while we +recorded. The first thing this codebase ever sends to a customer's instrument +should be a decision someone made on purpose, not a side effect of a client +that grew a method. + +Every offset and params layout below was read off THOR's own frames in +``bridges/captures/9-24-26 - micromate2/`` rather than taken from the spec +table, because the same exercise during the framing work found three of that +table's rules wrong. It found three more here: + + * ``0x0A`` is the **monitor-log walk** — the same request repeated, the + device advancing its own cursor, terminated by a short response — not the + keyed "event header, 30 B list record" the spec describes. + * ``0x1E``/``0x1F`` carry **token 0xFE** at ``params[7]``. The protocol + reference documents all-zero params for the browse walk; that was our own + probing, and THOR does not do it that way. + * There is **no fixed preamble**. Sessions open with ``POLL`` and go + straight to the operation. ``POLL → SERIAL → 0x49 → POLL`` appears in 3 of + 8 captured sessions and is THOR's *connection check*, not a handshake. + +And one useful negative: **no ``SESSION_RESET`` (``41 03``).** Series III +needs that 2-byte signal to wake a monitoring unit or it will not answer POLL +over TCP. THOR never sends it — 0 occurrences across all 8 sessions, including +40 frames exchanged with a unit that *was* monitoring. +""" + +from __future__ import annotations + +import logging +import math +import struct +import time +from typing import Optional + +from minimateplus.transport import BaseTransport + +from .framing import MicromateFrame, MicromateFrameParser, build_request + +log = logging.getLogger(__name__) + +DEFAULT_RECV_TIMEOUT = 10.0 + +# An acknowledgement carries an 11-byte data section and nothing else. It is +# also how the monitor-log walk says "no more records". +ACK_DATA_LEN = 11 + + +# ── Command SUBs ────────────────────────────────────────────────────────────── + +SUB_DEVICE_INFO = 0x01 +SUB_STORAGE_RANGE = 0x06 +SUB_EVENT_INDEX = 0x08 +SUB_MONITOR_LOG = 0x0A +SUB_EVENT_RECORD = 0x0C +SUB_SERIAL = 0x15 +SUB_COMPLIANCE_CONFIG = 0x1A +SUB_MONITOR_STATUS = 0x1C +SUB_EVENT_FIRST = 0x1E +SUB_EVENT_NEXT = 0x1F +SUB_CALL_HOME_CONFIG = 0x2C +SUB_TRIGGER_CONFIG = 0x2E +SUB_SETUP_FIRST = 0x3F +SUB_SETUP_NEXT = 0x40 +SUB_SETUP_ACTIVE = 0x41 +SUB_STATE = 0x49 +SUB_BULK_DOWNLOAD = 0x5A +SUB_POLL = 0x5B +SUB_ARM_EVENT = 0x93 + +# ⚠ Reads are SINGLE-STEP. Series III probes at offset 0 to learn the length, +# then reads again at that length; a Micromate returns the whole block when +# asked for 0xFFFF. THOR never probes, which is why `MicromateFrame.probe_length` +# reads 0 on live traffic. +READ_ALL = 0xFFFF + +# The two commands that do NOT use READ_ALL, and the data length each returned +# on UM12947 (firmware 11.0CB). +_OFFSETS = { + SUB_POLL: 0x0030, # 59 B — the one offset THOR treats as a constant + SUB_SERIAL: 0x000A, # 21 B +} + +# Data-section lengths observed, for orientation only — deliberately NOT +# asserted. `SUB 0x1C` is 4 bytes longer on the Thor firmware line (0x30 vs +# 0x2C declared), so a length check here would fire spuriously on half the +# fleet. See the protocol reference, "A/B: Blastware build vs Thor build". +OBSERVED_DATA_LEN = { + SUB_STORAGE_RANGE: 47, SUB_EVENT_INDEX: 101, SUB_MONITOR_LOG: 297, + SUB_EVENT_RECORD: 221, SUB_SERIAL: 21, SUB_COMPLIANCE_CONFIG: 2103, + SUB_MONITOR_STATUS: 55, SUB_EVENT_FIRST: 19, SUB_EVENT_NEXT: 19, + SUB_CALL_HOME_CONFIG: 137, SUB_TRIGGER_CONFIG: 39, + SUB_SETUP_FIRST: 266, SUB_SETUP_NEXT: 266, SUB_SETUP_ACTIVE: 266, + SUB_STATE: 16, SUB_POLL: 59, SUB_ARM_EVENT: ACK_DATA_LEN, +} + +# `1E`/`1F` carry this at params[7]. Series III uses the same value to arm its +# bulk stream; here THOR sends it on every chain read, browse or download. +EVENT_TOKEN = 0xFE + +# `SUB 0x5A` chunk size, in bytes of file payload per response. +CHUNK_SIZE = 1024 + +# Every `0x5A` response prefixes the file bytes with 11 bytes of header. +_CHUNK_PREFIX = 11 + + +# ── Exceptions ──────────────────────────────────────────────────────────────── + +class ProtocolError(Exception): + """The device violated the expected protocol.""" + + +class TimeoutError(ProtocolError): + """No response arrived within the allowed time.""" + + +class ChecksumError(ProtocolError): + """A received frame failed its checksum.""" + + +class UnexpectedResponse(ProtocolError): + """The response SUB did not match the request.""" + + +class ShortRead(ProtocolError): + """A bulk download returned fewer bytes than the device promised.""" + + +# ── Params builders ─────────────────────────────────────────────────────────── + +def token_params(token: int = EVENT_TOKEN) -> bytes: + """`1E`/`1F`: the token sits at params[7].""" + return bytes(7) + bytes([token]) + bytes(2) + + +def key_params(key4: bytes) -> bytes: + """`0x0C`: the full 4-byte event key at params[4:8].""" + if len(key4) != 4: + raise ValueError(f"key4 must be 4 bytes, got {len(key4)}") + return bytes(4) + key4 + bytes(2) + + +def key_lo_params(key4: bytes) -> bytes: + """`0x0A`: only the key's **low two bytes**, at params[6:8]. + + ⚠ Inferred from a single key value. All nine captured `0x0A` frames carry + `4a 81`, and the event key in play was `055d4a81` — so this is consistent + with "the low half of the current key" and equally consistent with "a + cursor handle that happened to equal it". Both readings produce the same + bytes for that key, so one event cannot separate them. + + It does not matter much in practice: the walk works with the same params + repeated, so whichever it is, passing the current key is right. + """ + if len(key4) != 4: + raise ValueError(f"key4 must be 4 bytes, got {len(key4)}") + return bytes(6) + key4[2:4] + bytes(2) + + +def chunk_params(key4: bytes, byte_offset: int) -> bytes: + """`0x5A`: the key opens the file, then a byte offset walks it. + + Chunk 0 carries the event key at params[0:4] — that is what says "from the + beginning". Later chunks carry a uint16 BE byte offset at params[2:4]. + """ + if byte_offset == 0: + if len(key4) != 4: + raise ValueError(f"key4 must be 4 bytes, got {len(key4)}") + return key4 + bytes(6) + if not 0 <= byte_offset <= 0xFFFF: + raise ValueError(f"byte_offset must fit in uint16, got {byte_offset}") + return bytes(2) + struct.pack(">H", byte_offset) + bytes(6) + + +# ── Protocol ────────────────────────────────────────────────────────────────── + +class MicromateProtocol: + """Wire-level command set for one open connection to a Micromate. + + Does not own the transport; lifetime belongs to the client. + + proto = MicromateProtocol(transport) + proto.poll() + serial = proto.read_serial() + """ + + def __init__( + self, + transport: BaseTransport, + recv_timeout: float = DEFAULT_RECV_TIMEOUT, + strict_checksums: bool = True, + ) -> None: + """ + Args: + strict_checksums: raise on a bad checksum. **Defaults to True, + unlike the Series III sibling**, which logs and continues + because its parser cannot reliably tell an inner-frame + delimiter from a checksum byte. That excuse does not apply + here: the Micromate rule is plain SUM8 over the de-stuffed + payload and it holds on 251 of 251 captured frames, so a + mismatch means something real — line noise, a desync, or a rule + we have wrong — and all three are worth hearing about. + + The lenient Series III default is instructive: it hid the fact + that the documented checksum rule was wrong for two days. Set + False only to get a field diagnosis unstuck. + """ + self._transport = transport + self._recv_timeout = recv_timeout + self._strict = strict_checksums + self._parser = MicromateFrameParser() + self._pending: list[MicromateFrame] = [] + + # ── Identity and state ──────────────────────────────────────────────────── + + def poll(self) -> MicromateFrame: + """`0x5B` → `0xA4`. Handshake; carries the ID block and model string. + + Every captured session opens with this and nothing before it. + """ + return self._exchange(SUB_POLL) + + def read_serial(self) -> bytes: + """`0x15` → `0xEA`. ASCII, null-terminated — e.g. `UM12947`.""" + return self._read(SUB_SERIAL) + + def read_device_info(self) -> bytes: + """`0x01` → `0xFE`. Firmware, calibration, per-channel float block. + + ⚠ THOR never sends this in any captured session, so the `0xFFFF` offset + is from our own 2026-09-23 probes rather than from THOR's behaviour. It + answered correctly on both firmware lines, but it is the one read here + with no THOR frame behind it. + """ + return self._read(SUB_DEVICE_INFO) + + def read_state(self) -> bytes: + """`0x49` → `0xB6`. A cheap monitoring check; 16 B. + + ⚠ Test `data[11]` for **non-zero**, never against a constant — it has + read both `0x0E` and `0x0C` while monitoring. + """ + return self._read(SUB_STATE) + + def read_monitor_status(self) -> bytes: + """`0x1C` → `0xE3`. Flag, **device clock**, battery, memory. + + ⚠ Parse **forward** from the declared length, never backward from the + end. This block is 4 bytes longer on the Thor firmware line, and Series + III's relative-to-end offsets yield a battery voltage of 577.92 V on a + `11.0BD` unit. + """ + return self._read(SUB_MONITOR_STATUS) + + def read_storage_range(self) -> bytes: + """`0x06` → `0xF9`. Event storage extent; 47 B.""" + return self._read(SUB_STORAGE_RANGE) + + def read_event_index(self) -> bytes: + """`0x08` → `0xF7`. 101 B. Contents not yet mapped.""" + return self._read(SUB_EVENT_INDEX) + + def read_trigger_config(self) -> bytes: + """`0x2E` → `0xD1`. 39 B. Series IV only; no Series III equivalent.""" + return self._read(SUB_TRIGGER_CONFIG) + + def read_compliance_config(self) -> bytes: + """`0x1A` → `0xE5`. The whole active setup — 2103 B on UM12947. + + One response. Series III needs a 4-frame sequence for the same thing. + """ + return self._read(SUB_COMPLIANCE_CONFIG) + + def read_call_home_config(self) -> bytes: + """`0x2C` → `0xD3`. 137 B — Series III's is 124, so do not reuse its map.""" + return self._read(SUB_CALL_HOME_CONFIG) + + # ── Setups ──────────────────────────────────────────────────────────────── + + def read_active_setup_name(self) -> bytes: + """`0x41` → `0xBE`. 266 B; carries the active `.MMB` name.""" + return self._read(SUB_SETUP_ACTIVE) + + def read_first_setup(self) -> bytes: + """`0x3F` → `0xC0`. Head of the setup-file list.""" + return self._read(SUB_SETUP_FIRST) + + def read_next_setup(self) -> bytes: + """`0x40` → `0xBF`. Repeat until the record carries an empty name. + + Stateful: the device holds the cursor, so the same request walks the + list. 22 of these appear back to back in one captured session. + """ + return self._read(SUB_SETUP_NEXT) + + # ── Event chain ─────────────────────────────────────────────────────────── + + def arm_event(self) -> MicromateFrame: + """`0x93` → `0x6C`. THOR sends this before **every** `1E`/`1F`. + + It replaces Series III's `1E(token=0xFE)` arming step. No params, no + offset payload — an 11-byte ack. + + ⚠ Whether a unit actually requires it is untested. Do it because it is + known-good, not because it is known-necessary. + """ + return self._exchange(SUB_ARM_EVENT, offset=READ_ALL) + + def read_event_first(self) -> bytes: + """`0x1E` → `0xE1`. First event key + size; 19 B.""" + return self._read(SUB_EVENT_FIRST, params=token_params()) + + def read_event_next(self) -> bytes: + """`0x1F` → `0xE0`. Next key + size, or the all-zero null sentinel.""" + return self._read(SUB_EVENT_NEXT, params=token_params()) + + def read_event_record(self, key4: bytes) -> bytes: + """`0x0C` → `0xF3`. 221 B — project, client, operator, timestamp, peaks. + + ⚠ The peak float in here runs 2–5% above `max(T,V,L)` and is **not** the + vector sum; its offset was inferred, not established. Prefer decoded + samples. + """ + return self._read(SUB_EVENT_RECORD, params=key_params(key4)) + + def read_monitor_log_next(self, key4: bytes) -> Optional[bytes]: + """`0x0A` → `0xF5`. One monitor-log record, or None at end of list. + + ⚠ Not the keyed single read the spec describes. This is a **walk**: + the same request repeated, the device advancing its own cursor, each + response a 297-byte record carrying serial, mode and thresholds. The + list ends with a bare 11-byte ack — nine captured frames, eight records + then the terminator. + + Series III reaches the same data through a record-type discriminator on + its event walk (`0x2C` partial vs `0x46` full). Here it is a separate + cursor and the event chain does not see it at all. + """ + data = self._read(SUB_MONITOR_LOG, params=key_lo_params(key4)) + return None if len(data) <= ACK_DATA_LEN else data + + # ── Bulk download ───────────────────────────────────────────────────────── + + def read_event_file(self, key4: bytes, size: int) -> bytes: + """`0x5A` → `0xA5`. The `.IDFW`/`.IDFH` file, byte for byte. + + `size` is the 4 bytes after the key in the `1E`/`1F` response. Returns + exactly that many bytes, or raises `ShortRead`. + + A bounded chunk walk — `ceil(size / 1024)` requests, each asking for + `min(1024, remaining)` bytes: + + offset = the byte count wanted (NOT an address) + params = the key on chunk 0, then a uint16 BE byte offset + response = exactly `offset + 11` bytes; file bytes are data[11:] + + Verified against THOR on all six bench events (4,076 → 13,424 B): + `sum(offsets) == size` exactly, every time. + + ⚠ Do not port the Series III `5A` walk. Its address arithmetic caused a + 5x over-read and a `>64 KB` page-boundary bug that is *still open* on + that side. Neither applies here — the cursor is a byte offset into the + file, bounded by a size the device supplied, so it cannot run past the + event. + + The result feeds `micromate.idf_file.read_idf_file()` and + `/db/import/idf_file` unchanged; no new codec work is needed. + """ + if size <= 0: + raise ValueError(f"size must be positive, got {size}") + + out = bytearray() + n_chunks = math.ceil(size / CHUNK_SIZE) + for i in range(n_chunks): + want = min(CHUNK_SIZE, size - i * CHUNK_SIZE) + data = self._read( + SUB_BULK_DOWNLOAD, + offset=want, + params=chunk_params(key4, i * CHUNK_SIZE), + ) + if len(data) < _CHUNK_PREFIX: + raise ShortRead( + f"chunk {i + 1}/{n_chunks} of {key4.hex()}: " + f"{len(data)} B is too short to hold a chunk header" + ) + body = data[_CHUNK_PREFIX:] + if len(body) != want: + # Worth being loud: a silently short event is the failure mode + # this project has been bitten by repeatedly on the Series III + # side, and here the expected length is known up front. + raise ShortRead( + f"chunk {i + 1}/{n_chunks} of {key4.hex()}: asked for " + f"{want} B, got {len(body)}" + ) + out += body + + if len(out) != size: + raise ShortRead( + f"{key4.hex()}: assembled {len(out)} B, device promised {size}" + ) + log.debug("downloaded %s: %d B in %d chunks", key4.hex(), len(out), n_chunks) + return bytes(out) + + # ── Plumbing ────────────────────────────────────────────────────────────── + + def _read( + self, + sub: int, + *, + params: bytes = bytes(10), + offset: Optional[int] = None, + timeout: Optional[float] = None, + ) -> bytes: + """Send one read command, return the response's data section.""" + return self._exchange(sub, params=params, offset=offset, timeout=timeout).data + + def _exchange( + self, + sub: int, + *, + params: bytes = bytes(10), + offset: Optional[int] = None, + timeout: Optional[float] = None, + ) -> MicromateFrame: + if offset is None: + offset = _OFFSETS.get(sub, READ_ALL) + # Start every exchange clean: drop any half-frame and any frame left + # stashed by the last one. This is a strict request/response protocol, + # so anything already buffered when we send is by definition stale, and + # `expected_sub` would reject it anyway — better to discard it here than + # to raise a confusing UnexpectedResponse one command later. + self._parser.reset() + self._pending.clear() + self._send(build_request(sub, offset, params)) + return self._recv_one(expected_sub=0xFF - sub, timeout=timeout, + reset_parser=False) + + def _send(self, frame: bytes) -> None: + log.debug("TX %d bytes: %s", len(frame), frame.hex()) + self._transport.write(frame) + + def _recv_one( + self, + expected_sub: Optional[int] = None, + timeout: Optional[float] = None, + reset_parser: bool = True, + ) -> MicromateFrame: + """Read until one complete frame is parsed.""" + deadline = time.monotonic() + (timeout or self._recv_timeout) + if reset_parser: + self._parser.reset() + self._pending.clear() + + if self._pending: + return self._validate(self._pending.pop(0), expected_sub) + + while time.monotonic() < deadline: + chunk = self._transport.read(4096) + if not chunk: + time.sleep(0.005) + continue + log.debug("RX %d bytes", len(chunk)) + frames = self._parser.feed(chunk) + if frames: + self._pending.extend(frames[1:]) + return self._validate(frames[0], expected_sub) + + raise TimeoutError( + f"no frame in {timeout or self._recv_timeout:.1f}s" + + (f" (expected SUB 0x{expected_sub:02X})" if expected_sub is not None else "") + + f"; {self._parser.bytes_fed} bytes were received" + # That byte count is the whole point: it separates "nothing came + # back at all" from "bytes arrived but never framed", and those have + # completely different causes. It earned its keep on Series III. + ) + + def _validate( + self, frame: MicromateFrame, expected_sub: Optional[int] + ) -> MicromateFrame: + if not frame.checksum_valid: + msg = ( + f"SUB 0x{frame.sub:02X}: checksum mismatch " + f"(got 0x{frame.chk_byte:02X}, {len(frame.data)} B data)" + ) + if self._strict: + raise ChecksumError(msg) + log.warning("%s — continuing (strict_checksums=False)", msg) + if expected_sub is not None and frame.sub != expected_sub: + raise UnexpectedResponse( + f"expected SUB 0x{expected_sub:02X}, got 0x{frame.sub:02X}" + ) + return frame diff --git a/tests/test_micromate_protocol.py b/tests/test_micromate_protocol.py new file mode 100644 index 0000000..ccb0214 --- /dev/null +++ b/tests/test_micromate_protocol.py @@ -0,0 +1,402 @@ +"""Protocol-layer tests for the Micromate (series-4) live client. + +The load-bearing assertion in here is not "our parser understands the device" — +it is **"the bytes we put on the wire are the bytes THOR puts on the wire."** +Every request constant below is lifted from +``bridges/captures/9-24-26 - micromate2/`` (UM12947, firmware 11.0CB), so a +passing test means a real unit has already answered exactly that frame. + +Responses are replayed through a scripted transport. No hardware, no network. +""" +from __future__ import annotations + +import os +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from micromate import protocol as P +from micromate.framing import ETX, STX, checksum, stuff +from micromate.protocol import ( + ACK_DATA_LEN, + ChecksumError, + MicromateProtocol, + ShortRead, + UnexpectedResponse, + chunk_params, + key_lo_params, + key_params, + token_params, +) + +FLAGS_CB = 0xC5 + + +# ── Test doubles ────────────────────────────────────────────────────────────── + +class ScriptedTransport: + """Hands back queued responses; records every byte written.""" + + def __init__(self, responses: list[bytes] | None = None) -> None: + self.queue = list(responses or []) + self.written: list[bytes] = [] + self._connected = True + + # BaseTransport surface actually used by MicromateProtocol + def connect(self) -> None: + self._connected = True + + def disconnect(self) -> None: + self._connected = False + + def is_connected(self) -> bool: + return self._connected + + def write(self, data: bytes) -> None: + self.written.append(data) + + def read(self, n: int) -> bytes: + return self.queue.pop(0) if self.queue else b"" + + +def frame(rsp_sub: int, data: bytes, *, flags: int = FLAGS_CB, page: int = 0) -> bytes: + """Build a response frame the way a unit would.""" + payload = bytes([0x00, flags, rsp_sub, (page >> 8) & 0xFF, page & 0xFF]) + data + return bytes([STX]) + stuff(payload + bytes([checksum(payload)])) + bytes([ETX]) + + +def ack(rsp_sub: int) -> bytes: + return frame(rsp_sub, bytes(ACK_DATA_LEN)) + + +def proto(responses: list[bytes], **kw) -> tuple[MicromateProtocol, ScriptedTransport]: + t = ScriptedTransport(responses) + return MicromateProtocol(t, recv_timeout=0.5, **kw), t + + +# ── Captured THOR request frames ────────────────────────────────────────────── + +REQ = { + "poll": bytes.fromhex("41021010005b000030000000000000000000009b03"), + "serial": bytes.fromhex("41021010001500000a000000000000000000002f03"), + "state": bytes.fromhex("41021010004900ffff000000000000000000005703"), + "compliance": bytes.fromhex("41021010001a00ffff000000000000000000002803"), + "arm": bytes.fromhex("41021010009300ffff00000000000000000000a103"), + "setup_first": bytes.fromhex("41021010003f00ffff000000000000000000004d03"), + "setup_next": bytes.fromhex("41021010004000ffff000000000000000000004e03"), +} + +# The complete 0x5A sequence for event 055d4a81 (4,076 bytes → 4 chunks), as +# THOR sent it. Chunk 1's params hold a literal 0x04 and chunk 3's offset is +# the exact remainder. +REQ_CHUNKS_4A81 = [ + bytes.fromhex("41021010005a00100400055d4a810000000000009b03"), + bytes.fromhex("41021010005a0010040000001004000000000000007203"), + bytes.fromhex("41021010005a00100400000008000000000000007603"), + bytes.fromhex("41021010005a001003ec00000c000000000000006503"), +] +SIZE_4A81 = 4076 + + +# ── Params builders ─────────────────────────────────────────────────────────── + +def test_event_token_sits_at_params_7(): + """⚠ THOR sends 0xFE here; the protocol reference documents all-zero params. + + That reference entry describes our own browse probing, not THOR's. + """ + assert token_params() == bytes.fromhex("00000000000000fe0000") + + +def test_event_record_takes_the_full_key_at_params_4(): + assert key_params(bytes.fromhex("055d4a81")) == bytes.fromhex("00000000055d4a810000") + + +def test_monitor_log_takes_only_the_low_half_of_the_key(): + """⚠ Inferred from one key value -- see key_lo_params' docstring.""" + assert key_lo_params(bytes.fromhex("055d4a81")) == bytes.fromhex("0000000000004a810000") + + +def test_chunk_params_switch_from_key_to_byte_offset(): + key = bytes.fromhex("055d4a81") + assert chunk_params(key, 0) == bytes.fromhex("055d4a81000000000000") + assert chunk_params(key, 1024) == bytes.fromhex("00000400000000000000") + assert chunk_params(key, 13312) == bytes.fromhex("00003400000000000000") + + +@pytest.mark.parametrize("bad", [b"", b"\x01\x02\x03", b"\x01\x02\x03\x04\x05"]) +def test_params_builders_reject_a_wrong_length_key(bad): + for fn in (key_params, key_lo_params): + with pytest.raises(ValueError): + fn(bad) + + +# ── Each read emits the frame THOR emits ────────────────────────────────────── + +@pytest.mark.parametrize( + "name, method, rsp_sub, data_len", + [ + ("poll", "poll", 0xA4, 59), + ("serial", "read_serial", 0xEA, 21), + ("state", "read_state", 0xB6, 16), + ("compliance", "read_compliance_config", 0xE5, 2103), + ("setup_first", "read_first_setup", 0xC0, 266), + ("setup_next", "read_next_setup", 0xBF, 266), + ], +) +def test_reads_match_thors_wire_bytes(name, method, rsp_sub, data_len): + p, t = proto([frame(rsp_sub, bytes(data_len))]) + getattr(p, method)() + assert t.written == [REQ[name]] + + +def test_arm_event_matches_thors_wire_bytes(): + p, t = proto([ack(0x6C)]) + p.arm_event() + assert t.written == [REQ["arm"]] + + +def test_poll_is_the_only_read_with_a_non_ffff_offset_besides_serial(): + """Reads are single-step at 0xFFFF; POLL and SERIAL are the exceptions.""" + assert set(P._OFFSETS) == {P.SUB_POLL, P.SUB_SERIAL} + assert P._OFFSETS[P.SUB_POLL] == 0x0030 + assert P._OFFSETS[P.SUB_SERIAL] == 0x000A + + +# ── The chunk walk ──────────────────────────────────────────────────────────── + +def test_download_reproduces_thors_chunk_sequence_byte_for_byte(): + """The whole point of step 2. Four chunks, 4,076 bytes, THOR's exact frames.""" + payload = bytes(range(256)) * 16 # 4096 B, we use the first 4076 + payload = payload[:SIZE_4A81] + responses = [] + for i in range(4): + want = min(P.CHUNK_SIZE, SIZE_4A81 - i * P.CHUNK_SIZE) + body = payload[i * P.CHUNK_SIZE: i * P.CHUNK_SIZE + want] + responses.append(frame(0xA5, bytes(11) + body, page=want // 256)) + + p, t = proto(responses) + got = p.read_event_file(bytes.fromhex("055d4a81"), SIZE_4A81) + + assert t.written == REQ_CHUNKS_4A81 + assert got == payload + assert len(got) == SIZE_4A81 + + +@pytest.mark.parametrize( + "size, n_chunks, last_offset", + [ + (4076, 4, 0x03EC), (11032, 11, 0x0318), (11502, 12, 0x00EE), + (13424, 14, 0x0070), (8746, 9, 0x022A), (6092, 6, 0x03CC), + (1024, 1, 0x0400), (1, 1, 0x0001), (1025, 2, 0x0001), + ], +) +def test_chunk_count_and_final_offset(size, n_chunks, last_offset): + """The first six rows are the six bench events, with THOR's real offsets.""" + responses = [] + for i in range(n_chunks): + want = min(P.CHUNK_SIZE, size - i * P.CHUNK_SIZE) + responses.append(frame(0xA5, bytes(11) + bytes(want))) + + p, t = proto(responses) + p.read_event_file(bytes.fromhex("055d4a81"), size) + + assert len(t.written) == n_chunks + # offset is payload[4:5] of the request; recover it from the built frame + final = t.written[-1] + assert final[7:9] in ( + bytes([last_offset >> 8, last_offset & 0xFF]), + # a 0x02/0x03/0x04/0x10 high byte arrives escaped, shifting the pair + bytes([0x10, last_offset >> 8]), + ) + + +def test_a_short_chunk_raises_rather_than_truncating(): + """A silently short event is the failure mode this codebase keeps hitting.""" + p, _ = proto([frame(0xA5, bytes(11) + bytes(900))]) # asked for 1024 + with pytest.raises(ShortRead, match="asked for 1024 B, got 900"): + p.read_event_file(bytes.fromhex("055d4a81"), 1024) + + +def test_a_chunk_too_short_to_hold_its_header_raises(): + p, _ = proto([frame(0xA5, bytes(4))]) + with pytest.raises(ShortRead, match="too short to hold a chunk header"): + p.read_event_file(bytes.fromhex("055d4a81"), 1024) + + +def test_download_rejects_a_nonsense_size(): + p, _ = proto([]) + with pytest.raises(ValueError): + p.read_event_file(bytes.fromhex("055d4a81"), 0) + + +# ── The monitor-log walk ────────────────────────────────────────────────────── + +def test_monitor_log_walk_ends_on_a_short_response(): + """⚠ Not a keyed read -- the same request repeated, device-side cursor. + + Eight records then an 11-byte ack, which is what the capture shows. + """ + key = bytes.fromhex("055d4a81") + responses = [frame(0xF5, bytes(297)) for _ in range(8)] + [ack(0xF5)] + p, t = proto(responses) + + records = [] + while (rec := p.read_monitor_log_next(key)) is not None: + records.append(rec) + + assert len(records) == 8 + assert len(t.written) == 9 + assert len(set(t.written)) == 1, "every request in the walk is identical" + + +# ── Error handling ──────────────────────────────────────────────────────────── + +def test_a_bad_checksum_raises_by_default(): + """⚠ Deliberately stricter than the Series III sibling. + + That one logs and continues because its parser cannot always tell an + inner-frame delimiter from a checksum byte. The Micromate rule is exact on + 251/251 captured frames, so a mismatch here means something real. + """ + bad = bytearray(frame(0xA4, bytes(59))) + bad[-2] ^= 0xFF + p, _ = proto([bytes(bad)]) + with pytest.raises(ChecksumError, match="checksum mismatch"): + p.poll() + + +def test_a_bad_checksum_can_be_downgraded_for_field_diagnosis(): + bad = bytearray(frame(0xA4, bytes(59))) + bad[-2] ^= 0xFF + p, _ = proto([bytes(bad)], strict_checksums=False) + assert p.poll().sub == 0xA4 + + +def test_the_wrong_response_sub_raises(): + p, _ = proto([frame(0xE0, bytes(19))]) # 0xE0 answers 0x1F, not 0x1E + with pytest.raises(UnexpectedResponse, match="expected SUB 0xE1"): + p.read_event_first() + + +def test_a_timeout_reports_how_many_bytes_arrived(): + """Separates "nothing came back" from "bytes arrived but never framed". + + Those have completely different causes -- and on a Micromate the second one + is the signature of a modem forwarding a session it should not be. + """ + p, _ = proto([]) + with pytest.raises(P.TimeoutError, match="0 bytes were received"): + p.poll() + + unframed = b"\x02\x00\xc5\xa4garbage-no-terminator" + p2, _ = proto([unframed]) + with pytest.raises(P.TimeoutError, match=f"{len(unframed)} bytes were received"): + p2.poll() + + +def test_a_leftover_frame_is_discarded_rather_than_answered_with(): + """If a read returns two frames, the extra must not answer the NEXT request. + + Every exchange resets the parser before sending, so anything already + buffered is treated as stale. Delivering it would be the worse failure: + `expected_sub` happens to catch a mismatched SUB, but a same-SUB leftover + would sail through and return data for the wrong key. + """ + # Both frames arrive while answering arm_event(); the 0xE1 is left over. + p, _ = proto([ack(0x6C) + frame(0xE1, b"\xaa" * 19)]) + p.arm_event() + + # The next request gets no bytes of its own, so it must time out rather + # than hand back the stale 0xE1. + with pytest.raises(P.TimeoutError): + p.read_event_first() + + +# ── Against the real capture, when it happens to be present ─────────────────── + +_CAPTURES = ( + Path(__file__).resolve().parents[1] + / "bridges" / "captures" / "9-24-26 - micromate2" +) +_DOWNLOAD = "raw_bw_20260925_011403_Download_events_then_delete_1_event.bin" + + +@pytest.mark.skipif( + not (_CAPTURES / _DOWNLOAD).is_file(), + reason="capture is gitignored; present only on a dev box", +) +def test_every_captured_download_frame_is_one_we_would_have_sent(): + """Replay the real session: for each event, assert our chunk walk emits + exactly the frames THOR emitted -- all 50-odd of them, six events.""" + from micromate.framing import ACK, DLE + + def destuffed_frames(blob: bytes, is_req: bool): + i, n = 0, len(blob) + while i < n: + if is_req: + if not (blob[i] == ACK and i + 1 < n and blob[i + 1] == STX): + i += 1 + continue + j = i + 2 + else: + if blob[i] != STX: + i += 1 + continue + j = i + 1 + out = bytearray() + while j < n: + if blob[j] == DLE and j + 1 < n: + out.append(blob[j + 1]) + j += 2 + continue + if blob[j] == ETX: + break + out.append(blob[j]) + j += 1 + if len(out) >= 6: + yield blob[i:j + 1], bytes(out[:-1]) + i = j + 1 + + bw = list(destuffed_frames((_CAPTURES / _DOWNLOAD).read_bytes(), True)) + s3 = list( + destuffed_frames( + (_CAPTURES / _DOWNLOAD.replace("raw_bw", "raw_s3")).read_bytes(), False + ) + ) + + # Group THOR's 0x5A frames per event, taking each event's key+size from the + # 1E/1F that preceded them. + events, cur = [], None + for (wire, req), (_, rsp) in zip(bw, s3): + sub, data = req[2], rsp[5:] + if sub in (0x1E, 0x1F) and len(data) >= 19: + cur = {"key": data[11:15], "size": int.from_bytes(data[15:19], "big"), + "reqs": [], "rsps": []} + if cur["size"]: + events.append(cur) + elif sub == 0x5A and cur is not None: + cur["reqs"].append(wire) + cur["rsps"].append(rsp) + + # The capture walks the chain twice (it deletes an event on the second + # pass), so some 1E/1F hits carry a size but no download behind them. + events = [e for e in events if e["reqs"]] + assert len(events) == 6, f"expected 6 downloaded events, found {len(events)}" + + total = 0 + for e in events: + p, t = proto([bytes([STX]) + stuff(r + bytes([checksum(r)])) + bytes([ETX]) + for r in e["rsps"]]) + got = p.read_event_file(e["key"], e["size"]) + assert t.written == e["reqs"], ( + f"event {e['key'].hex()}: our {len(t.written)} frames differ from " + f"THOR's {len(e['reqs'])}" + ) + assert len(got) == e["size"] + total += len(e["reqs"]) + + assert total == 56, f"expected 56 download frames across the 6 events, saw {total}" From f09b7dcaafc65091e10454826939451142cdd1cc Mon Sep 17 00:00:00 2001 From: serversdown Date: Mon, 28 Sep 2026 20:25:40 -0400 Subject: [PATCH 08/16] feat(micromate): client layer -- connect, state, setups; read-only Step 3 of docs/micromate_client_spec.md: micromate/client.py, two models in micromate/models.py, 26 offline tests. Every response constant in the tests is a real captured data section from UM12947. Field offsets were measured rather than taken from the spec, which turned up one general rule and one trap: THE RESPONSE SHAPE. Every response carries an 11-byte prefix and content starts at data[11]. One rule, every command. THE TRAP: data[0] is the content length & 0xFF, with no high byte anywhere in the prefix. It is therefore correct for every response under 256 bytes -- most of them -- and then reports 44 for a 2,092-byte setup block, 30 for a 286-byte monitor-log record, and 0 for a 1,024-byte download chunk. 182 of 251 captured responses agree with a naive read; the 69 that disagree are exactly the ones >= 256 bytes. That is the third length in this protocol read too narrow, after payload[9]-vs-payload[8:10] in the probe response. The client takes content as data[11:] and lets the frame's own length bound it -- nothing needs the declared length, since the frame already knows how long it is. Also measured: - POLL content[3] is 0x50, printable as "P", immediately before "Instantel". A generic printable-run scan therefore returns "PInstantel" -- it caught a test, not a unit. Vendor comes from a fixed offset; the model is found by searching for "MM/", which is structural rather than positional and so survives the Thor line's shorter "MM/ISEE/S". - The setup walk terminates on an EMPTY NAME, not an error: 23 responses, 22 names, factory.MMB first through TEST1.mmb last. - The 0x1C clock has an unidentified byte at content[6]; the hour is at content[7]. The protocol reference's 0x1C section already had this right and names the byte -- its one-line summary in the divergences list reads as six contiguous fields and is the version not to trust. Re-verified against three captures: 19:12:25, 19:13:34 and 01:14:05 against filenames stamped 19:12:14, 19:12:14 and 01:14:03. - Battery and memory are read FORWARD from content start, never backward from the end. This block is 4 bytes longer on the Thor line; the Series III from-the-end offsets give a 11.0BD unit 577.92 V. A test appends the four trailing bytes and asserts the forward offsets survive. connect() is narrower than the spec asked. The spec said to mirror Thor's POLL -> SERIAL -> 0x49 -> POLL "because it is known-good"; measurement showed that is Thor's connection check (3 of 8 sessions) and its fourth frame repeats its first. So connect() sends the three reads that gather something, and 0x01 is not read at all -- Thor never reads it, its layout is unmapped, and firmware_line comes free from any response's flags byte. If a unit ever refuses the next command after a cold connect, put the fourth POLL back and record it. A dead clock battery yields device_time=None rather than failing the whole state read; an unreadable active setup yields active_setup=None rather than failing connect. Both are real device states. Still verified only against 11.0CB and only over USB. The BD offsets follow from the extra bytes being trailing, which is documented but not something this code has seen. Full suite unchanged at 16 pre-existing failures; 445 passed, up 26. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_client_spec.md | 20 +- docs/micromate_protocol_reference.md | 47 ++++ micromate/client.py | 294 ++++++++++++++++++++ micromate/models.py | 82 ++++++ tests/test_micromate_client.py | 396 +++++++++++++++++++++++++++ 5 files changed, 837 insertions(+), 2 deletions(-) create mode 100644 micromate/client.py create mode 100644 tests/test_micromate_client.py diff --git a/docs/micromate_client_spec.md b/docs/micromate_client_spec.md index b6a619f..6c57b57 100644 --- a/docs/micromate_client_spec.md +++ b/docs/micromate_client_spec.md @@ -277,7 +277,23 @@ front. --- -## `micromate/client.py` +## `micromate/client.py` — ✅ BUILT (read half) 2026-09-27 + +`connect()`, `get_state()`, `get_active_setup()`, `list_setups()` plus +`MicromateDeviceInfo` / `MicromateState` in `models.py`. 26 tests, every +response constant a real captured data section. + +⚠ **`connect()` is deliberately narrower than this spec asked for.** The spec +said to mirror Thor's `POLL → SERIAL → 0x49 → POLL` "because it is known-good". +Measurement showed the four-command form is Thor's *connection check*, present +in 3 of 8 sessions, and its fourth frame repeats its first — so `connect()` +sends the three reads that gather something. `0x01` is not read at all: Thor +never reads it, its layout is unmapped, and `firmware_line` comes free from any +response's flags byte. + +Event-chain methods (`list_events`, `download_event`, `get_event`) are step 4 +and not yet written; `MicromateProtocol.read_event_file()` already does the +download. ```python class MicromateClient: @@ -358,7 +374,7 @@ then `download_event()` and assert the bytes decode and match a 1. ✅ `framing.py` + its tests — **done 2026-09-27**, 31 tests, offline 2. ✅ `protocol.py` + its tests — **done 2026-09-27**, 35 tests, offline -3. `client.py` — `connect()`, `get_state()`, `list_setups()` +3. ✅ `client.py` + its tests — **done 2026-09-27**, 26 tests, offline 4. the event chain and `download_event()` 5. decode end-to-end and compare against a store event diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index f444e0f..7d46fed 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1300,6 +1300,53 @@ through a record-type discriminator (`0x2C` partial vs `0x46` full) *on its event walk*, so partial records and events share one chain. Here the monitor log has its own cursor and the event chain never sees it. +#### 🔑 Every response has an 11-byte prefix — and its length byte lies + +**Content starts at `data[11]`.** One rule, every command. + +⚠ **`data[0]` is the content length `& 0xFF`, and there is no high byte +anywhere in the prefix.** `data[1]` is zero on every frame examined. So it +reads as a perfectly good length field for any response under 256 bytes — which +is most of them — and then: + +| command | true content | `data[0]` says | +|---|---|---| +| `0x1A` compliance | 2,092 | **44** | +| `0x0A` monitor log | 286 | **30** | +| `0x5A` download chunk | 1,024 | **0** | +| `0x41` setup name | 255 | 255 ✓ (only just) | + +182 of 251 captured responses agree with a naive uint16 LE read; the 69 that do +not are exactly the ones ≥ 256 bytes. + +**This is the third time a length in this protocol has been read too narrow** — +after `payload[9]` vs `payload[8:10]` in the probe response, and after +`data[0]` here. The pattern is worth naming rather than fixing case by case: +*take the content as `data[11:]` and let the frame's own length bound it.* +Nothing needs the declared length; the frame already knows how long it is. + +#### `SUB 0x5B` POLL content — where the strings actually sit + +``` +content[3] 0x50 ⚠ printable as "P", immediately before the vendor string +content[4] "Instantel\0" +content[17:26] binary — 06 00 c3 f0 4a 00 e4 19 4f 00 74 02 +content[26] "MM/ISEE/S/IO\0" ← "MM/ISEE/S" on the Thor line +``` + +⚠ **Do not parse this by scanning for printable runs.** That `0x50` at +content[3] is printable, so a run scan returns `PInstantel`. Nothing +distinguishes a length or tag byte from text by inspection. Read the vendor +from the fixed offset and find the model by searching for `MM/` — structural +rather than positional, which is what survives the model string's +firmware-line difference. + +#### `SUB 0x3F`/`0x40` setup walk — the terminator is an empty name + +23 responses on the bench unit: 22 names, then a record whose name field is all +zeros. The empty name **is** the end of list — not an error, not a setup. +`factory.MMB` first, `TEST1.mmb` last. + #### `SUB 0x01` has no Thor frame behind it Thor never reads device info in any captured session. `0xFFFF` for `0x01` comes diff --git a/micromate/client.py b/micromate/client.py new file mode 100644 index 0000000..997faca --- /dev/null +++ b/micromate/client.py @@ -0,0 +1,294 @@ +""" +client.py — high-level API for a live Micromate (Series IV). + +Owns the transport, turns raw payloads into models. Read-only, like the layer +below it: nothing here writes, erases, or changes monitoring state. + + with MicromateClient(TcpTransport("63.45.161.30", 9034)) as mm: + info = mm.connect() + print(info) # UM12947 MM/ISEE/S/IO blastware fw idle + print(mm.get_state()) # idle 2026-09-25 01:14:05 3.80 V memory 0.4% used + for name in mm.list_setups(): + print(name) + +The response layout, measured rather than assumed +------------------------------------------------- +**Every response carries an 11-byte prefix, and the content starts at +``data[11]``.** That one rule covers every command. + +⚠ **``data[0]`` looks like the content length and is only its low byte.** A +2,092-byte setup block (`SUB 0x1A`) reports 44, and a 1,024-byte download chunk +reports 0. It happens to be right for every response shorter than 256 bytes, +which is most of them — so it reads as a working length field right up until it +silently loses 2,048 bytes. There is no high byte anywhere in the prefix; it is +``length & 0xFF`` and nothing more. + +This is the same trap as ``MicromateFrame.probe_length``, in a different place, +and it is now the third time a length in this protocol has been read too narrow. +**Take the content as ``data[11:]`` and let the frame's own length bound it.** +""" + +from __future__ import annotations + +import datetime +import logging +from typing import Optional + +from minimateplus.transport import BaseTransport + +from .models import MicromateDeviceInfo, MicromateState +from .protocol import MicromateProtocol, ProtocolError + +log = logging.getLogger(__name__) + +# The content of every response begins here; the first 11 bytes are a prefix +# whose only decoded field is an unreliable low-byte length (see module docstring). +CONTENT = 11 + +# Field offsets, relative to the start of content. Sources are named because +# two of them disagree with docs/micromate_protocol_reference.md. +_STATE_FLAG = 0 # 0x49: 0x00 idle, 0x02 monitoring + +_MS_FLAG = 1 # 0x1C: monitoring flag — test NON-ZERO +_MS_DAY, _MS_MONTH, _MS_YEAR = 2, 3, slice(4, 6) +_MS_UNKNOWN_6 = 6 # ⚠ NOT the hour — see read note below +_MS_HOUR, _MS_MIN, _MS_SEC = 7, 8, 9 +_MS_BATTERY = slice(34, 36) # uint16 BE, volts × 100 +_MS_MEM_TOTAL = slice(36, 40) # uint32 BE +_MS_MEM_FREE = slice(40, 44) # uint32 BE + +# A setup-list walk that does not terminate is a bug, not a big fleet. The +# bench unit holds 22 setups; this is a generous ceiling, not a limit. +_MAX_SETUPS = 512 + +def _content(data: bytes) -> bytes: + """Strip the 11-byte response prefix.""" + return data[CONTENT:] if len(data) > CONTENT else b"" + + +def _cstring(buf: bytes, offset: int = 0) -> str: + """A null-terminated ASCII run, stripped.""" + return buf[offset:].split(b"\x00")[0].decode("ascii", "replace").strip() + + +class MicromateClient: + """High-level read-only client for one Micromate. + + Owns the transport, unlike ``MicromateProtocol``, which borrows it. + """ + + def __init__( + self, + transport: BaseTransport, + recv_timeout: float = 10.0, + strict_checksums: bool = True, + ) -> None: + self._transport = transport + self._proto = MicromateProtocol( + transport, recv_timeout=recv_timeout, strict_checksums=strict_checksums + ) + self._firmware_line: Optional[str] = None + + # ── Lifecycle ───────────────────────────────────────────────────────────── + + def open(self) -> None: + self._transport.connect() + + def close(self) -> None: + self._transport.disconnect() + + def is_open(self) -> bool: + return self._transport.is_connected() + + def __enter__(self) -> "MicromateClient": + self.open() + return self + + def __exit__(self, *_) -> None: + self.close() + + @property + def protocol(self) -> MicromateProtocol: + """The wire layer, for anything this class does not wrap yet.""" + return self._proto + + # ── Identity ────────────────────────────────────────────────────────────── + + def connect(self, *, with_active_setup: bool = True) -> MicromateDeviceInfo: + """`POLL → SERIAL → state`, plus the active setup name. + + ⚠ **This is deliberately not Thor's full preamble.** Thor sends + `POLL → SERIAL → 0x49 → POLL` and the client spec said to copy it + verbatim on the grounds that it is known-good. Measuring all 8 captured + sessions showed the only invariant is that a session **opens with + POLL** — the four-command form appears in 3 of 8 and is Thor's + *connection check*, run where it wants to refresh what it displays. The + trailing POLL is a repeat of the first. + + So this sends the three reads that actually gather something. Dropping + the fourth is a judgement call on measured evidence, not a proof that + nothing depends on it; if a unit ever refuses the next command after a + cold connect, put it back and say so in the protocol reference. + + `SUB 0x01` (device info) is **not** read. Thor never reads it in any + captured session, its field layout is unmapped beyond eight `1.0f` + floats, and `firmware_line` — the one thing we would want from it — comes + free from the flags byte of any response. + """ + poll = self._proto.poll() + self._firmware_line = poll.firmware_line + + manufacturer, model = self._parse_poll(poll.data) + serial = _cstring(_content(self._proto.read_serial())) + monitoring = self._parse_state(self._proto.read_state()) + + info = MicromateDeviceInfo( + serial=serial, + manufacturer=manufacturer, + model=model, + firmware_line=poll.firmware_line, + monitoring=monitoring, + ) + if with_active_setup: + try: + info.active_setup = self.get_active_setup() + except ProtocolError as e: + # Not worth failing a connect over: a unit with no setup loaded + # is a real state, and the caller can still read everything else. + log.warning("active setup unreadable: %s", e) + log.info("connected: %s", info) + return info + + @staticmethod + def _parse_poll(data: bytes) -> tuple[Optional[str], Optional[str]]: + """Manufacturer and model out of the POLL block. + + `Instantel` sits at content[4] and the model at content[26], with 13 + binary bytes between them. + + ⚠ A generic "find the printable runs" scan does **not** work here, which + cost a test failure before it cost anything worse. content[3] is `0x50` + — printable as `P` — sitting immediately before `Instantel`, so a run + scan returns `PInstantel`. Nothing distinguishes a length or tag byte + from text by inspection. + + So: the manufacturer comes from a fixed offset, and the model is found by + searching for `MM/`. That anchor is structural rather than positional, + which matters because the model string **differs by firmware line** — + `MM/ISEE/S/IO` on the Blastware build, `MM/ISEE/S` on the Thor build — + and only its tail changes. + """ + c = _content(data) + manufacturer = _cstring(c, 4) or None + + idx = c.find(b"MM/") + model = _cstring(c, idx) if idx >= 0 else None + return manufacturer, model + + @staticmethod + def _parse_state(data: bytes) -> Optional[bool]: + """`SUB 0x49` content[0]: 0x00 idle, 0x02 monitoring. + + ⚠ Tested for non-zero, never against `0x02`. The sibling flag in + `SUB 0x1C` has read both `0x0E` and `0x0C` while monitoring, so this + family of flags is not a stable enum. + """ + c = _content(data) + return bool(c[_STATE_FLAG]) if c else None + + # ── State ───────────────────────────────────────────────────────────────── + + def get_state(self) -> MicromateState: + """`SUB 0x1C` — monitoring, device clock, battery, memory. + + ⚠ Every offset here is **forward from the start of content**, never + backward from the end. Series III reads battery and memory from the end + of this block, and this block is **4 bytes longer on the Thor firmware + line** — applying from-the-end offsets to a `11.0BD` unit yields a + battery voltage of 577.92 V. The four extra bytes are trailing, so + from-the-start offsets hold for both lines. + + ⚠ Verified on `11.0CB` only. That the same offsets hold on `11.0BD` + follows from the extra bytes being trailing, which is documented but not + something this code has seen. + """ + data = self._proto.read_monitor_status() + c = _content(data) + if len(c) < 44: + raise ProtocolError( + f"monitor status content is {len(c)} B, need at least 44" + ) + + battery = int.from_bytes(c[_MS_BATTERY], "big") / 100.0 + return MicromateState( + monitoring=bool(c[_MS_FLAG]), + device_time=self._parse_clock(c), + battery_volts=battery, + memory_total_bytes=int.from_bytes(c[_MS_MEM_TOTAL], "big"), + memory_free_bytes=int.from_bytes(c[_MS_MEM_FREE], "big"), + raw=data, + ) + + @staticmethod + def _parse_clock(c: bytes) -> Optional[datetime.datetime]: + """The unit's own clock, in its own local time. + + ⚠ **content[6] is not part of the time.** The layout is day, month, + year, *one unidentified byte*, then h/m/s — so the hour is at content[7]. + The protocol reference's `SUB 0x1C` section has this right and names + `data[17]` as unidentified; its one-line summary in the divergences list + ("day/month/year/h/m/s at `data[13:21]`") reads as six contiguous fields + and is the version worth not trusting. + + Re-measured here across three captures: content[6] read 32, 100 and 116, + none a valid hour, while content[7:10] gave 19:12:25, 19:13:34 and + 01:14:05 against capture filenames stamped 19:12:14, 19:12:14 and + 01:14:03 — each seconds to a minute after its session opened, which is + what a device clock should do. + + content[6] is undecoded and deliberately not exposed. + """ + try: + return datetime.datetime( + year=int.from_bytes(c[_MS_YEAR], "big"), + month=c[_MS_MONTH], + day=c[_MS_DAY], + hour=c[_MS_HOUR], + minute=c[_MS_MIN], + second=c[_MS_SEC], + ) + except ValueError as e: + # A unit with a dead clock battery reports an impossible date. That + # is information, not a reason to fail the whole state read. + log.warning("device clock unreadable (%s): %s", e, c[2:10].hex(" ")) + return None + + # ── Setups ──────────────────────────────────────────────────────────────── + + def get_active_setup(self) -> str: + """`SUB 0x41` — the loaded `.MMB` file name, e.g. `TEST1.mmb`.""" + return _cstring(_content(self._proto.read_active_setup_name())) + + def list_setups(self) -> list[str]: + """`0x3F` then `0x40`… — every setup file stored on the unit. + + A cursor walk: the device holds the position, so the same `0x40` request + returns the next name. **An empty name terminates the list** — it is + not an error and not a real setup. + + Measured on the bench unit: 23 responses, 22 names then the empty one, + `factory.MMB` first through `TEST1.mmb` last. + """ + names: list[str] = [] + raw = self._proto.read_first_setup() + for _ in range(_MAX_SETUPS): + name = _cstring(_content(raw)) + if not name: + return names + names.append(name) + raw = self._proto.read_next_setup() + + raise ProtocolError( + f"setup list did not terminate after {_MAX_SETUPS} entries — the " + f"device cursor is not advancing" + ) diff --git a/micromate/models.py b/micromate/models.py index 68a91a7..49e4c25 100644 --- a/micromate/models.py +++ b/micromate/models.py @@ -396,3 +396,85 @@ class IdfEvent: ) ev._waveform_key = waveform_key return ev + + +# ── Live-device models (2026-09-27) ─────────────────────────────────────────── +# +# These describe what a unit reports over the wire, not what Thor wrote to a +# file. Everything above this line came out of Thor's exports; everything below +# came out of Thor's *traffic*. Field offsets are recorded in +# ``micromate/client.py`` next to the code that reads them. + + +@dataclass +class MicromateDeviceInfo: + """Identity gathered by ``MicromateClient.connect()``. + + Sourced from three reads: + ``0x5B`` POLL → manufacturer, model + ``0x15`` SERIAL → serial + ``0x49`` STATE → monitoring + plus ``firmware_line``, which comes free from the flags byte of any + response and needs no read of its own. + """ + + serial: str + manufacturer: Optional[str] = None # "Instantel" + model: Optional[str] = None # "MM/ISEE/S/IO" (CB) / "MM/ISEE/S" (BD) + firmware_line: Optional[str] = None # "blastware" | "thor" | "unknown" + monitoring: Optional[bool] = None + active_setup: Optional[str] = None # e.g. "TEST1.mmb" + + def __str__(self) -> str: + bits = [self.serial] + if self.model: + bits.append(self.model) + if self.firmware_line: + bits.append(f"{self.firmware_line} fw") + if self.monitoring is not None: + bits.append("MONITORING" if self.monitoring else "idle") + if self.active_setup: + bits.append(f"setup={self.active_setup}") + return " ".join(bits) + + +@dataclass +class MicromateState: + """A unit's live state, from ``SUB 0x1C``. + + ``device_time`` is the unit's own clock, in its own local timezone — it is + NOT converted. Nothing else this protocol exposes reports the unit's time, + which makes it the only way to detect a drifted clock before it lands in + event timestamps. + """ + + monitoring: bool + device_time: Optional[datetime.datetime] = None + battery_volts: Optional[float] = None + memory_total_bytes: Optional[int] = None + memory_free_bytes: Optional[int] = None + raw: Optional[bytes] = field(default=None, repr=False) + + @property + def memory_used_bytes(self) -> Optional[int]: + if self.memory_total_bytes is None or self.memory_free_bytes is None: + return None + return self.memory_total_bytes - self.memory_free_bytes + + @property + def memory_used_fraction(self) -> Optional[float]: + used = self.memory_used_bytes + if used is None or not self.memory_total_bytes: + return None + return used / self.memory_total_bytes + + def __str__(self) -> str: + bits = ["MONITORING" if self.monitoring else "idle"] + if self.device_time: + bits.append(self.device_time.strftime("%Y-%m-%d %H:%M:%S")) + if self.battery_volts is not None: + bits.append(f"{self.battery_volts:.2f} V") + frac = self.memory_used_fraction + if frac is not None: + bits.append(f"memory {frac * 100:.1f}% used") + return " ".join(bits) diff --git a/tests/test_micromate_client.py b/tests/test_micromate_client.py new file mode 100644 index 0000000..ce0964c --- /dev/null +++ b/tests/test_micromate_client.py @@ -0,0 +1,396 @@ +"""Client-layer tests for the Micromate (series-4) live client. + +Every response constant below is a **real data section**, captured from UM12947 +(firmware 11.0CB) in ``bridges/captures/9-24-26 - micromate2/``. They are +embedded as hex because the captures are gitignored. + +Where a decoded value can be checked against something outside the bytes, it is: +the device clock against the capture's own filename timestamp, the battery +against Thor's event reports (3.8 V), the setup list against what the unit +displays. +""" +from __future__ import annotations + +import datetime +import os +import sys + +import pytest + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from micromate.client import CONTENT, MicromateClient, _content, _cstring +from micromate.framing import ETX, STX, checksum, stuff +from micromate.protocol import ProtocolError + +FLAGS_CB, FLAGS_THOR = 0xC5, 0x03 + + +# ── Captured response data sections ─────────────────────────────────────────── + +# Generated from the captures by hand-free extraction -- the hex below is +# verbatim response data, not reconstructed. The trailing comment on each +# names the capture it came from, which is what lets the clock assertions be +# checked against a wall-clock timestamp. + +POLL = bytes.fromhex( # 59 B, from_20260924_185113_ + "300000000000000000000000000050496e7374616e74656c" + "000600c3f04a00e4194f0074024d4d2f495345452f532f49" + "4f00001f603e7657603e76" +) +SERIAL = bytes.fromhex( # 21 B, from_20260924_191214_ + "0a00000000000000000000554d3132393437003100" +) +STATE_IDLE = bytes.fromhex( # 16 B, from_20260924_191214_ + "050000000000000000000000e8000b00" +) +STATE_MONITORING = bytes.fromhex( # 16 B, from_20260924_191214_ + "050000000000000000000002e8000b00" +) +MS_MONITORING = bytes.fromhex( # 55 B, from_20260924_191214_ + "2c00000000000000000000000e180907ea20130c19000000" + "000001000000000000000000000000000000000000017c00" + "e4e1c000e3f1c0" +) +MS_IDLE = bytes.fromhex( # 55 B, from_20260924_191214_ + "2c000000000000000000000000180907ea64130d22000000" + "000001000000000000000000000000000000000000017c00" + "e4e1c000e3e1c0" +) +MS_LATE = bytes.fromhex( # 55 B, from_20260925_011403_ + "2c000000000000000000000000190907ea74010e05000000" + "000001000000000000000000000000000000000000017c00" + "e4e1c000e3e1c0" +) + +_SETUP_PAD = 266 - CONTENT + + +def setup_response(name: str) -> bytes: + """A 0x41/0x3F/0x40 response: 11-byte prefix then a null-padded name.""" + body = name.encode("ascii").ljust(_SETUP_PAD, b"\x00") + return bytes([0xFF]) + bytes(10) + body + + +# The real 22 names, in the order the unit walked them. +SETUP_NAMES = [ + "factory.MMB", "TEST.MMB", "BUS TEST.MMB", "Walsh JV 241.mmb", + "Walsh JV 008.mmb", "Hawbaker 322.mmb", "Hawbaker 322 blasting.mmb", + "min.mmb", "Playhouse Loc 1.mmb", "Valley Rock Solution.MMB", + "RecordingSetup.mmb", "UPMC.mmb", "UPMC Loc 3.mmb", "Residence Inn.mmb", + "Micromate ext trigger.mmb", "Micromate remort alarm.mmb", + "Tree of Life - Loc 1 - 5861 Solway.mmb", "Mele-PWSA-Carroll -Loc 4.mmb", + "Micromate min trigger mmb.mmb", "Fay - Layton Bridge Project.mmb", + "Default Micromate ISEE.mmb", "TEST1.mmb", +] + + +# ── Test doubles ────────────────────────────────────────────────────────────── + +class ScriptedTransport: + def __init__(self, responses: list[bytes]) -> None: + self.queue = list(responses) + self.written: list[bytes] = [] + self._connected = False + + def connect(self) -> None: + self._connected = True + + def disconnect(self) -> None: + self._connected = False + + def is_connected(self) -> bool: + return self._connected + + def write(self, data: bytes) -> None: + self.written.append(data) + + def read(self, n: int) -> bytes: + return self.queue.pop(0) if self.queue else b"" + + +def frame(rsp_sub: int, data: bytes, *, flags: int = FLAGS_CB) -> bytes: + payload = bytes([0x00, flags, rsp_sub, 0x00, 0x00]) + data + return bytes([STX]) + stuff(payload + bytes([checksum(payload)])) + bytes([ETX]) + + +def client(responses: list[bytes], **kw) -> tuple[MicromateClient, ScriptedTransport]: + t = ScriptedTransport(responses) + return MicromateClient(t, recv_timeout=0.5, **kw), t + + +# ── The captured constants are what we think they are ───────────────────────── + +def test_captured_constants_have_the_expected_lengths(): + assert len(POLL) == 59 + assert len(SERIAL) == 21 + assert len(STATE_IDLE) == len(STATE_MONITORING) == 16 + assert len(MS_MONITORING) == len(MS_IDLE) == len(MS_LATE) == 55 + + +def test_the_prefix_length_byte_is_only_the_low_byte(): + """⚠ data[0] is `content_length & 0xFF`, with no high byte anywhere. + + True for every response under 256 bytes, which is why it reads as a working + length field -- and then loses 2,048 bytes on a setup block. The client + takes content as data[11:] for exactly this reason. + """ + for data in (POLL, SERIAL, STATE_IDLE, MS_MONITORING): + assert data[0] == (len(data) - CONTENT) & 0xFF + assert data[1] == 0x00, "no high byte is stored" + + # The two that prove it is not a real length: a 2,092-byte setup block + # reports 44, and a 1,024-byte download chunk reports 0. + assert (2092 & 0xFF) == 44 + assert (1024 & 0xFF) == 0 + + +# ── Helpers ─────────────────────────────────────────────────────────────────── + +def test_a_printable_byte_precedes_the_manufacturer_string(): + """content[2] is 0x50 -- "P". This is why the POLL parse cannot be a scan.""" + c = _content(POLL) + assert c[3] == 0x50 and chr(c[3]) == "P" + assert c[4:13] == b"Instantel" + + +def test_content_strips_exactly_eleven_bytes(): + assert _content(SERIAL) == bytes.fromhex("554d3132393437003100") + assert _content(b"short") == b"" + + +def test_cstring_stops_at_the_null(): + assert _cstring(bytes.fromhex("554d3132393437003100")) == "UM12947" + assert _cstring(b"\x00rest") == "" + + +# ── connect() ───────────────────────────────────────────────────────────────── + +def test_connect_decodes_identity(): + mm, t = client([ + frame(0xA4, POLL), frame(0xEA, SERIAL), frame(0xB6, STATE_IDLE), + frame(0xBE, setup_response("TEST1.mmb")), + ]) + info = mm.connect() + + assert info.serial == "UM12947" + assert info.manufacturer == "Instantel" + assert info.model == "MM/ISEE/S/IO" + assert info.firmware_line == "blastware" + assert info.monitoring is False + assert info.active_setup == "TEST1.mmb" + assert "UM12947" in str(info) and "idle" in str(info) + + +def test_connect_sends_three_reads_not_thors_four(): + """⚠ Deliberately narrower than Thor's POLL -> SERIAL -> 0x49 -> POLL. + + The trailing POLL repeats the first; measuring all 8 captured sessions + showed the four-command form is Thor's connection check (3 of 8 sessions), + not a handshake. Only "opens with POLL" is invariant. + """ + mm, t = client([ + frame(0xA4, POLL), frame(0xEA, SERIAL), frame(0xB6, STATE_IDLE), + frame(0xBE, setup_response("TEST1.mmb")), + ]) + mm.connect() + subs = [w[5] for w in t.written] # payload[2] lands at wire[5] + assert subs == [0x5B, 0x15, 0x49, 0x41] + assert 0x01 not in subs, "device info has no Thor precedent; do not read it" + + +def test_connect_can_skip_the_active_setup(): + mm, t = client([frame(0xA4, POLL), frame(0xEA, SERIAL), frame(0xB6, STATE_IDLE)]) + info = mm.connect(with_active_setup=False) + assert info.active_setup is None + assert len(t.written) == 3 + + +def test_connect_survives_an_unreadable_active_setup(): + """A unit with no setup loaded is a real state, not a failed connect.""" + mm, _ = client([ + frame(0xA4, POLL), frame(0xEA, SERIAL), frame(0xB6, STATE_IDLE), + frame(0x00, bytes(20)), # wrong SUB -> UnexpectedResponse + ]) + info = mm.connect() + assert info.serial == "UM12947" + assert info.active_setup is None + + +def test_connect_reports_a_monitoring_unit(): + mm, _ = client([ + frame(0xA4, POLL), frame(0xEA, SERIAL), frame(0xB6, STATE_MONITORING), + frame(0xBE, setup_response("TEST1.mmb")), + ]) + assert mm.connect().monitoring is True + + +def test_the_state_flag_is_tested_for_non_zero(): + """⚠ Never compared against 0x02 -- this flag family is not a stable enum. + + Its sibling in SUB 0x1C has read both 0x0E and 0x0C while monitoring. + """ + for value in (0x01, 0x02, 0x0C, 0x0E, 0xFF): + data = bytearray(STATE_IDLE) + data[CONTENT] = value + mm, _ = client([ + frame(0xA4, POLL), frame(0xEA, SERIAL), frame(0xB6, bytes(data)), + frame(0xBE, setup_response("x.mmb")), + ]) + assert mm.connect().monitoring is True, f"0x{value:02x} should read as monitoring" + + +def test_the_model_string_is_anchored_on_MM_not_on_an_offset(): + """The Thor firmware line reports a SHORTER model string, "MM/ISEE/S". + + Anchoring on b"MM/" survives that; a fixed end offset would not. A generic + printable-run scan fails for a different reason -- see _parse_poll. + """ + bd = bytearray(POLL) + assert bd[CONTENT + 26:CONTENT + 38] == b"MM/ISEE/S/IO" + bd[CONTENT + 26:CONTENT + 38] = b"MM/ISEE/S\x00\x00\x00" + mm, _ = client([ + frame(0xA4, bytes(bd), flags=FLAGS_THOR), frame(0xEA, SERIAL), + frame(0xB6, STATE_IDLE), frame(0xBE, setup_response("x.mmb")), + ]) + info = mm.connect() + assert info.model == "MM/ISEE/S" + assert info.manufacturer == "Instantel" + assert info.firmware_line == "thor" + + +# ── get_state() ─────────────────────────────────────────────────────────────── + +@pytest.mark.parametrize( + "data, monitoring, when, free", + [ + (MS_MONITORING, True, datetime.datetime(2026, 9, 24, 19, 12, 25), 0x00E3F1C0), + (MS_IDLE, False, datetime.datetime(2026, 9, 24, 19, 13, 34), 0x00E3E1C0), + (MS_LATE, False, datetime.datetime(2026, 9, 25, 1, 14, 5), 0x00E3E1C0), + ], + ids=["monitoring", "idle", "after-midnight"], +) +def test_get_state_decodes_the_real_reads(data, monitoring, when, free): + """⚠ There is an unidentified byte at content[6]; the hour is at content[7]. + + The protocol reference's 0x1C section has this right. Its one-line summary + in the divergences list reads as six contiguous fields and does not. + + Each expected time is checked against the capture filename that produced the + bytes: 19:12:14, 19:12:14 and 01:14:03. All three decode to seconds-to-a- + minute after their session opened, which is what a device clock should do. + Reading content[6] as the hour gives 32, 100 and 116. + """ + mm, _ = client([frame(0xE3, data)]) + st = mm.get_state() + + assert st.monitoring is monitoring + assert st.device_time == when + assert st.battery_volts == 3.80 # Thor's reports print 3.8 V + assert st.memory_total_bytes == 15_000_000 + assert st.memory_free_bytes == free + assert st.raw == data + + +def test_content_6_is_not_the_hour(): + """The byte the reference implies is the hour reads 32, 100 and 116.""" + for data in (MS_MONITORING, MS_IDLE, MS_LATE): + assert _content(data)[6] not in range(24) + + +def test_memory_derivations(): + mm, _ = client([frame(0xE3, MS_MONITORING)]) + st = mm.get_state() + assert st.memory_used_bytes == 15_000_000 - 0x00E3F1C0 + assert 0 < st.memory_used_fraction < 0.02 + assert "3.80 V" in str(st) + + +def test_battery_and_memory_are_read_forward_from_content_start(): + """⚠ NOT backward from the end. + + This block is 4 bytes longer on the Thor firmware line, and Series III's + from-the-end offsets give a 11.0BD unit a battery reading of 577.92 V. The + extra bytes are trailing, so appending four does not move anything. + """ + bd = MS_MONITORING + bytes.fromhex("0fa00000") + bd = bytes([0x30]) + bd[1:] # low-byte length becomes 48 + mm, _ = client([frame(0xE3, bd)]) + st = mm.get_state() + + assert st.battery_volts == 3.80, "forward offsets must survive the 4 extra bytes" + assert st.memory_total_bytes == 15_000_000 + # What the Series III from-the-end offsets would have produced: + assert int.from_bytes(bd[-10:-8], "big") / 100 == pytest.approx(577.92, abs=0.01) + + +def test_a_dead_clock_battery_does_not_fail_the_whole_read(): + """An impossible date is information; the rest of the block is still good.""" + broken = bytearray(MS_MONITORING) + broken[CONTENT + 3] = 0xFF # month 255 + mm, _ = client([frame(0xE3, bytes(broken))]) + st = mm.get_state() + assert st.device_time is None + assert st.battery_volts == 3.80 + + +def test_a_truncated_state_block_raises(): + mm, _ = client([frame(0xE3, bytes(20))]) + with pytest.raises(ProtocolError, match="need at least 44"): + mm.get_state() + + +# ── Setups ──────────────────────────────────────────────────────────────────── + +def test_list_setups_walks_to_the_empty_terminator(): + """22 real names then an empty one, exactly as the unit walked them.""" + responses = [frame(0xC0, setup_response(SETUP_NAMES[0]))] + responses += [frame(0xBF, setup_response(n)) for n in SETUP_NAMES[1:]] + responses += [frame(0xBF, setup_response(""))] + + mm, t = client(responses) + assert mm.list_setups() == SETUP_NAMES + assert len(t.written) == 23, "22 names plus the terminator" + assert t.written[0][5] == 0x3F + assert {w[5] for w in t.written[1:]} == {0x40} + + +def test_list_setups_handles_an_empty_unit(): + mm, _ = client([frame(0xC0, setup_response(""))]) + assert mm.list_setups() == [] + + +def test_list_setups_refuses_to_loop_forever(): + """A cursor that never advances is a bug, and must not hang the caller.""" + from micromate import client as C + + mm, _ = client([frame(0xC0, setup_response("a.mmb"))] + + [frame(0xBF, setup_response("a.mmb"))] * (C._MAX_SETUPS + 5)) + with pytest.raises(ProtocolError, match="not advancing"): + mm.list_setups() + + +def test_get_active_setup_handles_a_long_name(): + long_name = "Tree of Life - Loc 1 - 5861 Solway.mmb" + mm, _ = client([frame(0xBE, setup_response(long_name))]) + assert mm.get_active_setup() == long_name + + +# ── Lifecycle ───────────────────────────────────────────────────────────────── + +def test_the_client_owns_the_transport(): + mm, t = client([]) + assert not mm.is_open() + mm.open() + assert mm.is_open() and t.is_connected() + mm.close() + assert not mm.is_open() + + +def test_context_manager_opens_and_closes(): + t = ScriptedTransport([frame(0xA4, POLL), frame(0xEA, SERIAL), + frame(0xB6, STATE_IDLE), frame(0xBE, setup_response("x.mmb"))]) + with MicromateClient(t, recv_timeout=0.5) as mm: + assert t.is_connected() + assert mm.connect().serial == "UM12947" + assert not t.is_connected() From c7ffdab5708b999a0084e2c1861d924348e1d73b Mon Sep 17 00:00:00 2001 From: serversdown Date: Mon, 28 Sep 2026 20:30:11 -0400 Subject: [PATCH 09/16] tooling(micromate): mm_client_check -- exercise the read client on real hardware Read-only: POLL, SERIAL, state, monitor status, the setup walk, and optionally one event download. Never writes, erases, or changes monitoring state. Exists because everything in micromate/{framing,protocol,client}.py is verified against captures taken over USB on one firmware line (11.0CB). Two things that cannot be verified that way: - The modem path. An RX55/RV55 buffers up to ~1 s before forwarding, so one logical response arrives as many small reads. The client reads to frame completion rather than using read_until_idle's idle-gap detection, which should be strictly more robust for this -- but that needed proving. - The 11.0BD firmware line, which reports flags=0x03, a shorter model string, and a 0x1C block 4 bytes longer. All inference from one 2026-09-23 sweep whose captures never landed in the repo. The tool says so loudly when it meets one, and flags an impossible battery voltage as the signature of the from-the-end offset bug. Run it over both paths and diff; anything differing beyond timings is a finding. It counts reads and bytes per transport, because a higher read count for the same bytes IS the buffering, made visible. Smoke-tested against a scripted TCP unit replaying captured response bytes in 37-byte dribbles: 289 reads to carry 10,958 B, every frame reassembled, a 4,076 B event downloaded across 4 chunks with the assembled length exact. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- bridges/mm_client_check.py | 209 +++++++++++++++++++++++++++++++++++++ 1 file changed, 209 insertions(+) create mode 100644 bridges/mm_client_check.py diff --git a/bridges/mm_client_check.py b/bridges/mm_client_check.py new file mode 100644 index 0000000..8bd86cb --- /dev/null +++ b/bridges/mm_client_check.py @@ -0,0 +1,209 @@ +#!/usr/bin/env python3 +""" +mm_client_check.py — exercise the Micromate read client against a real unit. + +**Read-only.** It sends POLL, SERIAL, state, monitor status, the setup walk and +(optionally) one event download. It never writes, never erases, never starts or +stops monitoring. + +Why it exists +------------- +`micromate/{framing,protocol,client}.py` are verified against captures taken +**over USB**, on **one firmware line** (`11.0CB`). Two things that cannot be +verified that way: + + * **the modem path.** An RX55/RV55 bridges serial to TCP transparently, but + it buffers up to ~1 s before forwarding, so a single logical response can + arrive as many small reads. The client reads to frame completion rather + than using idle-gap detection, which should be strictly more robust — but + "should be" is the point of this script. + * **the other firmware line.** `11.0BD` reports `flags = 0x03`, a shorter + model string, and a `SUB 0x1C` block 4 bytes longer. Everything about that + is currently inference from one 2026-09-23 sweep whose captures never + landed in the repo. + +Run it over both paths and diff the two reports. Anything that differs beyond +timings is a finding. + +Usage +----- + # over the modem + python3 bridges/mm_client_check.py 63.45.161.30:9034 + + # over USB / direct serial + python3 bridges/mm_client_check.py /dev/ttyACM0 --baud 115200 + + # include one event download (still read-only) + python3 bridges/mm_client_check.py --download + +⚠ These modems bridge ONE TCP session to serial at a time. If THOR holds the +unit, this will connect and then see nothing — that is contention, not a fault. +`bridges/mm_probe.py` explains that case; disconnect THOR first. +""" + +from __future__ import annotations + +import argparse +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from micromate.client import MicromateClient, _content # noqa: E402 +from micromate.protocol import ProtocolError # noqa: E402 +from minimateplus.transport import SerialTransport, TcpTransport # noqa: E402 + + +class _Timed: + """Count bytes and time each read, so the two transports can be compared.""" + + def __init__(self, inner) -> None: + self._inner = inner + self.reads = 0 + self.bytes_in = 0 + self.bytes_out = 0 + + def connect(self): + return self._inner.connect() + + def disconnect(self): + return self._inner.disconnect() + + def is_connected(self): + return self._inner.is_connected() + + def write(self, data: bytes): + self.bytes_out += len(data) + return self._inner.write(data) + + def read(self, n: int) -> bytes: + chunk = self._inner.read(n) + if chunk: + self.reads += 1 + self.bytes_in += len(chunk) + return chunk + + +def step(label: str, fn): + """Run one read, report how long it took and what it returned.""" + t0 = time.monotonic() + try: + value = fn() + except Exception as e: + print(f" {label:.<26} FAILED {type(e).__name__}: {e}") + return None + ms = 1000 * (time.monotonic() - t0) + shown = value if isinstance(value, str) else repr(value) + if isinstance(value, list): + shown = f"{len(value)} entries" + print(f" {label:.<26} {ms:7.0f} ms {shown}") + return value + + +def main() -> int: + ap = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + ap.add_argument("target", help="host:port for TCP, or a serial device path") + ap.add_argument("--baud", type=int, default=115200, + help="serial only; a Micromate's modem port runs at 115200") + ap.add_argument("--timeout", type=float, default=10.0) + ap.add_argument("--download", action="store_true", + help="also download the first stored event (read-only)") + ap.add_argument("--lenient", action="store_true", + help="do not raise on a bad checksum — for diagnosis only") + a = ap.parse_args() + + if ":" in a.target and not Path(a.target).exists(): + host, _, port = a.target.rpartition(":") + inner = TcpTransport(host, int(port), connect_timeout=a.timeout) + path = f"TCP {host}:{port}" + else: + inner = SerialTransport(a.target, baud=a.baud) + path = f"serial {a.target} @ {a.baud}" + + transport = _Timed(inner) + mm = MicromateClient(transport, recv_timeout=a.timeout, + strict_checksums=not a.lenient) + + print(f"\n{path} (read-only: POLL, SERIAL, state, status, setups)\n") + t0 = time.monotonic() + try: + mm.open() + except OSError as e: + print(f" connect.................... FAILED {e}") + return 2 + print(f" {'connect':.<26} {1000*(time.monotonic()-t0):7.0f} ms") + + try: + info = step("connect() identity", mm.connect) + if info: + print(f" serial={info.serial} model={info.model} " + f"fw={info.firmware_line} monitoring={info.monitoring}") + print(f" active setup={info.active_setup!r}") + if info.firmware_line == "thor": + print(" *** 11.0BD unit — the FIRST one this code has met. ***") + print(" *** Check the battery and clock below carefully: ***") + print(" *** its 0x1C block is 4 bytes longer. ***") + + state = step("get_state()", mm.get_state) + if state: + print(f" {state}") + if state.battery_volts and not 2.5 < state.battery_volts < 9.0: + print(f" *** battery {state.battery_volts} V is impossible — " + f"this is the from-the-end offset bug. ***") + if state.device_time is None: + print(" *** device clock did not decode — dump raw below. ***") + print(f" raw 0x1C content: {_content(state.raw).hex(' ')}") + + setups = step("list_setups()", mm.list_setups) + if setups: + print(f" first={setups[0]!r} last={setups[-1]!r}") + + if a.download: + print("\n event chain (read-only):") + proto = mm.protocol + proto.arm_event() + hdr = _content(proto.read_event_first()) + key, size = hdr[0:4], int.from_bytes(hdr[4:8], "big") + if not size: + print(" no events stored") + else: + print(f" first event key={key.hex()} size={size} B") + t1 = time.monotonic() + blob = proto.read_event_file(key, size) + dt = time.monotonic() - t1 + print(f" downloaded {len(blob)} B in {dt:.1f} s " + f"({len(blob)/dt/1024:.1f} KiB/s)") + assert len(blob) == size + # Decode it with the existing codec to prove the bytes are real. + try: + from micromate.idf_file import read_idf_file + import tempfile + with tempfile.NamedTemporaryFile(suffix=".IDFW", delete=False) as f: + f.write(blob) + tmp = f.name + ev = read_idf_file(tmp) + print(f" decoded OK: {ev}") + except Exception as e: + print(f" decode failed: {type(e).__name__}: {e}") + out = Path(f"./{key.hex()}.IDFW") + out.write_bytes(blob) + print(f" saved to {out} for offline analysis") + + except ProtocolError as e: + print(f"\n ABORTED {type(e).__name__}: {e}") + return 3 + finally: + mm.close() + + print(f"\n transport: {transport.reads} reads, " + f"{transport.bytes_in} B in, {transport.bytes_out} B out") + print(" A modem path should show MORE reads for the same bytes than USB —") + print(" that is the buffering, and it is exactly what needed proving.\n") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From f7d9a1d9cbd82c26d1328c4d520260cfed406623 Mon Sep 17 00:00:00 2001 From: serversdown Date: Tue, 29 Sep 2026 14:06:55 -0400 Subject: [PATCH 10/16] tooling(micromate): drop the pyserial dependency from mm_client_check Hit on the bench: mint-mac has no pyserial, and `pip install pyserial` is refused outright by PEP 668 (externally-managed-environment) on Mint 22 / Ubuntu 24.04 / Debian 12. A field diagnostic that needs a pip install on a locked-down host is one you cannot run at the moment you need it -- which is exactly when this tool is for. Replaced minimateplus.SerialTransport with a ~30-line stdlib `termios` port, the same approach bridges/mm_link.py and scratch/fake_unit.py already take in this repo. Runs on a stock Python 3 anywhere. Not a general SerialTransport replacement: no flow control, no parity options, Linux/macOS only. Enough for a Micromate, which is 8N1 with no handshaking. Verified end to end against a fake unit on a pty, dribbling responses in 64-byte pieces: 194 reads for 10,958 B, identity/state/setup-walk all correct, and a 4,076 B event downloaded across 4 chunks at the exact expected length. Also clarified --baud in the help: it applies to the USB-A/FTDI path, and is ignored by the USB-B "PC" port, which is CDC-ACM and negotiates its own rate. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- bridges/mm_client_check.py | 77 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 74 insertions(+), 3 deletions(-) diff --git a/bridges/mm_client_check.py b/bridges/mm_client_check.py index 8bd86cb..76d971f 100644 --- a/bridges/mm_client_check.py +++ b/bridges/mm_client_check.py @@ -44,7 +44,11 @@ unit, this will connect and then see nothing — that is contention, not a fault from __future__ import annotations import argparse +import errno +import os +import select import sys +import termios import time from pathlib import Path @@ -52,7 +56,72 @@ sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) from micromate.client import MicromateClient, _content # noqa: E402 from micromate.protocol import ProtocolError # noqa: E402 -from minimateplus.transport import SerialTransport, TcpTransport # noqa: E402 +from minimateplus.transport import TcpTransport # noqa: E402 + + +class StdlibSerial: + """Raw serial on stdlib `termios` — no pyserial. + + `minimateplus.SerialTransport` needs pyserial, and a bench host is whatever + is to hand. On a PEP 668 distro (Mint 22, Ubuntu 24.04, Debian 12) a plain + `pip install pyserial` is refused outright, so a diagnostic that depends on + it is one you cannot run at the moment you need it. `bridges/mm_link.py` + and `scratch/fake_unit.py` already take this approach; this is the same + ~30 lines, and it means the tool runs on a stock Python 3 anywhere. + + Not a general replacement for SerialTransport — no flow control, no + parity options, Linux/macOS only. Enough for a Micromate, which is 8N1 + with no handshaking. + """ + + _BAUD = {9600: termios.B9600, 19200: termios.B19200, 38400: termios.B38400, + 57600: termios.B57600, 115200: termios.B115200} + + def __init__(self, path: str, baud: int = 115200) -> None: + if baud not in self._BAUD: + raise ValueError(f"unsupported baud {baud}; pick from {sorted(self._BAUD)}") + self.path, self.baud, self.fd = path, baud, None + + def connect(self) -> None: + if self.fd is not None: + return + self.fd = os.open(self.path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) + a = termios.tcgetattr(self.fd) + a[0] = a[1] = a[3] = 0 # raw in/out, non-canonical + a[2] = termios.CS8 | termios.CREAD | termios.CLOCAL # 8N1, ignore modem lines + a[4] = a[5] = self._BAUD[self.baud] + a[6] = list(a[6]) + a[6][termios.VMIN] = 0 + a[6][termios.VTIME] = 0 + termios.tcsetattr(self.fd, termios.TCSANOW, a) + termios.tcflush(self.fd, termios.TCIOFLUSH) + + def disconnect(self) -> None: + if self.fd is not None: + os.close(self.fd) + self.fd = None + + def is_connected(self) -> bool: + return self.fd is not None + + def read(self, n: int) -> bytes: + if self.fd is None: + return b"" + r, _, _ = select.select([self.fd], [], [], 0.05) + if not r: + return b"" + try: + return os.read(self.fd, n) + except OSError as e: + if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK): + return b"" + raise + + def write(self, data: bytes) -> None: + if self.fd is None: + raise OSError("port is not open") + while data: + data = data[os.write(self.fd, data):] class _Timed: @@ -107,7 +176,9 @@ def main() -> int: ) ap.add_argument("target", help="host:port for TCP, or a serial device path") ap.add_argument("--baud", type=int, default=115200, - help="serial only; a Micromate's modem port runs at 115200") + help="serial only; the USB-A/FTDI path runs at 115200. " + "Ignored by the USB-B 'PC' port, which is CDC-ACM " + "and negotiates its own rate.") ap.add_argument("--timeout", type=float, default=10.0) ap.add_argument("--download", action="store_true", help="also download the first stored event (read-only)") @@ -120,7 +191,7 @@ def main() -> int: inner = TcpTransport(host, int(port), connect_timeout=a.timeout) path = f"TCP {host}:{port}" else: - inner = SerialTransport(a.target, baud=a.baud) + inner = StdlibSerial(a.target, baud=a.baud) path = f"serial {a.target} @ {a.baud}" transport = _Timed(inner) From 0d6eb1621ff34129dd0c95cb516f0a1426b9a1e4 Mon Sep 17 00:00:00 2001 From: serversdown Date: Tue, 29 Sep 2026 14:13:36 -0400 Subject: [PATCH 11/16] verify(micromate): read client works on real hardware, both transports Run against UM12947 by bridges/mm_client_check.py, read-only, over the USB-B CDC-ACM port and over the RX55 to TCP :9034. THE WIRE BYTES ARE IDENTICAL ON BOTH PATHS -- 11,580 B in and 761 B out, to the byte, with matching identity, state and a 24-entry setup walk. The protocol does not care which physical layer it runs over, which is the thing the modem path needed to prove. Three findings, one of them a correction to my own prediction: 1. THE MODEM NEEDS FEWER READS, NOT MORE. 36 against USB's 83. The tool's banner claimed the opposite. The modem coalesces -- it buffers ~1 s then forwards one large TCP segment, where CDC-ACM delivers many small chunks. That is reassuring rather than alarming: the risk was a frame arriving split across reads, and the modem splits LESS than USB does. Banner fixed to state the measured numbers instead of a guess. 2. ~0.65 s PER ROUND TRIP over cellular, independent of payload size. A 1,024 B chunk and a 16 B state read cost the same. list_setups() takes 16.05 s over the modem against 0.46 s over USB, for 24 commands. This is the number that matters for SFM's design: over cellular, minimise round trips, not bytes. Enumerating setups costs 16 s -- cache it, never refresh it on a timer. A 13 KB event is 14 chunks ~ 8.4 s of latency against ~0.03 s of data, which makes the unconfirmed single-request 0x5A streaming mode worth its two-minute bench test on its own. 3. THERE IS A RECORD-TYPE FIELD: 0x0C content[11], 0x07 waveform, 0x08 histogram. The reference says no type field is known and that the type must be carried out of the chain walk. Found by diffing the six bench events' 0x0C records against their known types -- exactly one byte separates the groups and is constant within each -- then confirmed by predicting the right suffix for 6 of 6 on a blind re-run. Six events split 4/2 is thin evidence for a byte that could be a counter or a channel count, so it is recorded as a strong candidate, not settled, and mm_client_check keeps a fallback: it tries the other suffix on failure and says when the guess was wrong. Also flagged: the reference's claim that the type comes from SUB 0x0A's length is not visible in the download capture, where 0x0A is a standalone monitor-log walk after the last chain entry, not a per-event probe. END TO END: all six bench events assembled by read_event_file() from captured 0x5A responses decode with the existing codec -- 4 waveforms at 12,288 / 12,288 / 12,288 / 8,192 samples and 2 histograms. No new codec work needed; /db/import/idf_file ingests a directly downloaded event unchanged. Not covered: 11.0BD (still pure inference), a monitoring unit, a nearly-full buffer, and the inbound call-home session. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- bridges/mm_client_check.py | 86 ++++++++++++++++----- docs/micromate_protocol_reference.md | 109 +++++++++++++++++++++++++++ 2 files changed, 177 insertions(+), 18 deletions(-) diff --git a/bridges/mm_client_check.py b/bridges/mm_client_check.py index 76d971f..712ad81 100644 --- a/bridges/mm_client_check.py +++ b/bridges/mm_client_check.py @@ -154,6 +154,60 @@ class _Timed: return chunk +# ⚠ HYPOTHESIS, 6 events. content[11] of the 0x0C record separated 4 waveforms +# from 2 histograms cleanly and was constant within each group. A 4/2 split is +# thin evidence for a byte that could be anything, so _decode() below does NOT +# trust it -- it tries the other suffix on failure and says when the guess was +# wrong. The protocol reference states no type field is known; this may be it. +_TYPE_BYTE = 11 +_TYPES = {0x07: ".IDFW", 0x08: ".IDFH"} + + +def _event_type(record: bytes) -> str: + if len(record) <= _TYPE_BYTE: + return "?" + b = record[_TYPE_BYTE] + return {0x07: "waveform", 0x08: "histogram"}.get(b, f"unknown(0x{b:02x})") + + +def _decode(blob: bytes, key: bytes, record: bytes) -> None: + """Decode the downloaded bytes, proving they are a real event file. + + read_idf_file() picks waveform vs histogram from the FILENAME SUFFIX, and a + wire download has no filename -- so the suffix has to come from somewhere. + This tries the 0x0C type byte first and the other suffix second; getting a + decode either way proves the chunk assembly, and which one worked is itself + the finding. + """ + import tempfile + from micromate.idf_file import read_idf_file + + guess = _TYPES.get(record[_TYPE_BYTE] if len(record) > _TYPE_BYTE else -1, ".IDFW") + order = [guess] + [e for e in (".IDFW", ".IDFH") if e != guess] + + for n, ext in enumerate(order): + with tempfile.NamedTemporaryFile(suffix=ext, delete=False) as f: + f.write(blob) + tmp = f.name + try: + res = read_idf_file(tmp) + samples = sum(len(v) for v in getattr(res, "samples", {}).values()) + note = "" if n == 0 else f" *** the 0x0C type byte guessed {guess} — WRONG ***" + print(f" decoded OK as {ext}: {samples} samples{note}") + os.unlink(tmp) + return + except Exception as e: + last = f"{ext}: {type(e).__name__}: {e}" + finally: + if os.path.exists(tmp): + os.unlink(tmp) + + print(f" decode failed BOTH ways — last: {last}") + out = Path(f"./{key.hex()}.bin") + out.write_bytes(blob) + print(f" saved to {out} for offline analysis") + + def step(label: str, fn): """Run one read, report how long it took and what it returned.""" t0 = time.monotonic() @@ -241,27 +295,16 @@ def main() -> int: if not size: print(" no events stored") else: - print(f" first event key={key.hex()} size={size} B") + rec = _content(proto.read_event_record(key)) + print(f" first event key={key.hex()} size={size} B " + f"type={_event_type(rec)}") t1 = time.monotonic() blob = proto.read_event_file(key, size) dt = time.monotonic() - t1 print(f" downloaded {len(blob)} B in {dt:.1f} s " f"({len(blob)/dt/1024:.1f} KiB/s)") assert len(blob) == size - # Decode it with the existing codec to prove the bytes are real. - try: - from micromate.idf_file import read_idf_file - import tempfile - with tempfile.NamedTemporaryFile(suffix=".IDFW", delete=False) as f: - f.write(blob) - tmp = f.name - ev = read_idf_file(tmp) - print(f" decoded OK: {ev}") - except Exception as e: - print(f" decode failed: {type(e).__name__}: {e}") - out = Path(f"./{key.hex()}.IDFW") - out.write_bytes(blob) - print(f" saved to {out} for offline analysis") + _decode(blob, key, rec) except ProtocolError as e: print(f"\n ABORTED {type(e).__name__}: {e}") @@ -269,10 +312,17 @@ def main() -> int: finally: mm.close() + elapsed = time.monotonic() - t0 print(f"\n transport: {transport.reads} reads, " - f"{transport.bytes_in} B in, {transport.bytes_out} B out") - print(" A modem path should show MORE reads for the same bytes than USB —") - print(" that is the buffering, and it is exactly what needed proving.\n") + f"{transport.bytes_in} B in, {transport.bytes_out} B out, " + f"{elapsed:.1f} s total") + print(" Measured 2026-09-29, UM12947, same unit both ways:") + print(" USB-B (CDC-ACM) 83 reads list_setups 0.46 s download 394 KiB/s") + print(" RX55 (TCP) 36 reads list_setups 16.05 s download 1.6 KiB/s") + print(" The modem needs FEWER reads, not more -- it buffers ~1 s and then") + print(" forwards one large segment, where CDC-ACM delivers many small ones.") + print(" Cost is ~0.65 s PER ROUND TRIP regardless of payload size, so what") + print(" matters over cellular is the number of commands, not the bytes.\n") return 0 diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 7d46fed..d11e450 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -782,6 +782,88 @@ per-sample-exact applies. 0x93 → 1F → next key + size (until null sentinel) ``` +## ✅ Verified against real hardware, both transports (2026-09-29) + +`micromate/{framing,protocol,client}.py` driven against **UM12947** by +`bridges/mm_client_check.py`, read-only, over both physical paths: + +| | USB-B "PC" (CDC-ACM) | RX55 → TCP :9034 | +|---|---|---| +| identity, state, setups | identical | identical | +| bytes in / out | **11,580 / 761** | **11,580 / 761** | +| `list_setups()` (24 entries) | 0.46 s | **16.05 s** | +| 4,076 B download | 394 KiB/s | 1.6 KiB/s | +| transport reads | **83** | **36** | + +**The wire bytes are identical on both paths** — same counts to the byte. The +protocol does not care which physical layer it runs over, which is what the +modem path needed to prove. + +### ⚠ The modem needs FEWER reads, not more — the prediction was backwards + +`mm_client_check.py` originally printed "a modem path should show MORE reads +for the same bytes". It shows **36 against USB's 83**. The modem *coalesces*: +it buffers for up to ~1 s and then forwards one large TCP segment, where +CDC-ACM delivers many small chunks as they arrive. + +That is reassuring rather than alarming — the risk was a frame arriving split +across reads, and the modem splits **less** than USB does. The client reads to +frame completion rather than using `read_until_idle`'s idle-gap detection, and +handled both without a retry. + +### 🔑 ~0.65 s per round trip over cellular, independent of payload + +This is the number that matters for SFM's design. + +``` +list_setups() 24 commands 16.05 s ≈ 0.67 s each +get_state() 1 command 0.64 s +connect() 4 commands 2.53 s ≈ 0.63 s each +download 4 chunks 2.40 s ≈ 0.60 s each (1024 B per chunk) +``` + +The cost is per *command*, not per byte: a 1,024-byte chunk and a 16-byte +state read cost the same. **Over cellular, minimise round trips, not bytes.** + +Concrete consequences: + +- Enumerating setups costs **16 seconds** on a unit with 24 of them. Cache it; + do not refresh it on a timer. +- A 13 KB event is 14 chunks ≈ 8.4 s of latency against ~0.03 s of data. If + the single-request `offset_hi = 0x10` streaming mode is real (see + *`SUB 0x5A`*), it would cut a download to **one** round trip — that is worth + the two-minute bench test on its own. +- THOR's measured polling cost should be re-read in this light: its per-unit + status sweep is round trips, and round trips are what cellular charges for. + +### End to end: downloaded events decode with the existing codec + +All six bench events, assembled by `MicromateProtocol.read_event_file()` from +captured `0x5A` responses and fed to `micromate.idf_file.read_idf_file()`: + +| key | size | type | decode | +|---|---|---|---| +| `055d4a81` | 4,076 | histogram | ✅ | +| `055d4a82` | 11,032 | waveform | ✅ 12,288 samples | +| `055d4a83` | 11,502 | waveform | ✅ 12,288 samples | +| `055d4a84` | 13,424 | waveform | ✅ 12,288 samples | +| `055d4a85` | 8,746 | waveform | ✅ 8,192 samples | +| `055d4a86` | 6,092 | histogram | ✅ | + +**No new codec work is needed.** The bytes off the wire are the bytes +`thor-watcher` forwards today, so `/db/import/idf_file` ingests a directly +downloaded event unchanged. + +### Still not covered + +- **`11.0BD`** — UM12947 is a `11.0CB` unit. The Thor firmware line is still + entirely inference: `flags = 0x03`, a shorter model string, and a `0x1C` + block 4 bytes longer. `mm_client_check.py` says so loudly when it meets one. +- **A unit that is monitoring**, and a unit with a nearly-full event buffer. +- **The inbound call-home session** — still the one protocol unknown. + +--- + ## Setups are FILES, not a config block Series III has one compliance config you overwrite. Series IV keeps **named @@ -1526,6 +1608,33 @@ length, and the timestamps are sequential across the recording session. ### Record type + filename: generate it, don't detect it +> #### 🔑 Update 2026-09-29 — there IS a type field, in the `0x0C` record +> +> **`0x0C` content[11]: `0x07` = waveform, `0x08` = histogram.** +> +> Found by diffing the six bench events' `0x0C` records against their known +> types: exactly one byte separates the two groups and is constant within each. +> It then predicted the right suffix for **6 of 6** on a blind re-run. +> +> ⚠ **Six events, split 4/2.** That is thin evidence for a byte that could be +> a counter, a channel count or a mode. Treat it as a strong candidate, not a +> settled field, and **keep a fallback**: `bridges/mm_client_check.py` tries the +> other suffix on failure and reports when the guess was wrong, which is how +> this would be caught rather than silently mis-filing an event. +> +> This does not overturn the section below — the *filename* still has to be +> generated, and the type still comes out of the protocol rather than the +> payload. It just means the type is available from a command we already send +> for every event, instead of needing to be carried out of the chain walk +> separately. +> +> ⚠ The claim below that the type comes from "`SUB 0x0A` length `0x1E` = +> histogram, `0x00` = waveform" **is not visible in the 2026-09-24 download +> capture**, where `0x0A` appears once as a standalone monitor-log walk after +> the last chain entry, not per event. Either it was observed in a session not +> in the repo, or the two are being conflated. The `0x0C` byte is reproducible +> from bytes on disk; prefer it. + `read_idf_file()` decides waveform vs histogram from the **filename suffix** — and there is no filename when downloading over the wire. From b55e4e946ab55f599ba4d23630536cde9457635f Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 30 Sep 2026 13:17:53 -0400 Subject: [PATCH 12/16] verify(micromate): 11.0BD confirmed on hardware -- every inference held UM20147 read over USB by mm_client_check.py. The Thor firmware line was entirely inference until now, and two of its three differences were covered only by SYNTHESISED test frames. All three held: - flags = 0x03 identifies the line -> reported firmware_line="thor". That is only reachable if `10 03` in the flags position destuffs before indexing, since 0x03 is ETX -- so the escaped-flags case is confirmed too. - the model string is shorter -> reported "MM/ISEE/S" exactly. Anchoring the search on b"MM/" rather than a fixed span is what made this work. - the 0x1C block is 4 bytes longer with the extras TRAILING, so from-the-start offsets survive -> battery 3.55 V and a clock correct to the second. That last one is the one that mattered. Series III's from-the-end offsets would have given this unit 577.92 V, which is why mm_client_check watches for an impossible voltage: it is a free self-check on exactly the inference most likely to be wrong. Also clean: serial UM20147, 5 setups (factory.MMB -> test2.mmb), active setup test2.mmb, 15,000,000 B total and free, 21 reads / 2,165 B in. "One protocol stack drives the whole fleet regardless of firmware line" -- the headline finding of 2026-09-23 -- is now demonstrated by a working client rather than by matching response SUBs. Test and docstring claims downgraded from inference to confirmed where the hardware settled them, and left as synthesised-frame notes where it did not: the BEHAVIOUR is confirmed but raw BD bytes are still not in the repo. Added --capture DIR to mm_client_check: writes a raw_bw_*/raw_s3_* pair in the layout scratch/mm_frame_parse.py already reads, so a run on an unfamiliar unit becomes a test fixture without setting up a relay. Verified by round-tripping its own output through that parser: 28 frames, 0 bad checksums. Still not covered: a download from a BD unit (UM20147 had no events stored, so the chunk walk remains CB-only), raw BD fixture bytes, a monitoring unit, a nearly-full buffer, and the inbound call-home session. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- bridges/mm_client_check.py | 39 +++++++++++++++++++++++++--- docs/micromate_protocol_reference.md | 36 ++++++++++++++++++++++--- micromate/client.py | 7 ++--- tests/test_micromate_client.py | 4 +++ tests/test_micromate_framing.py | 11 +++++--- 5 files changed, 84 insertions(+), 13 deletions(-) diff --git a/bridges/mm_client_check.py b/bridges/mm_client_check.py index 712ad81..bce8921 100644 --- a/bridges/mm_client_check.py +++ b/bridges/mm_client_check.py @@ -125,13 +125,33 @@ class StdlibSerial: class _Timed: - """Count bytes and time each read, so the two transports can be compared.""" + """Count bytes and time each read, so the two transports can be compared. - def __init__(self, inner) -> None: + With `capture`, also writes the raw byte streams to a `raw_bw_*` / + `raw_s3_*` pair in the layout `scratch/mm_frame_parse.py` already reads -- + so a run on an unfamiliar unit can be turned into test fixtures without + setting up a relay. + """ + + def __init__(self, inner, capture: str | None = None) -> None: self._inner = inner self.reads = 0 self.bytes_in = 0 self.bytes_out = 0 + self._bw = self._s3 = None + if capture: + stamp = time.strftime("%Y%m%d_%H%M%S") + d = Path(capture) + d.mkdir(parents=True, exist_ok=True) + self.bw_path = d / f"raw_bw_{stamp}_mm_client_check.bin" + self.s3_path = d / f"raw_s3_{stamp}_mm_client_check.bin" + self._bw = open(self.bw_path, "wb") + self._s3 = open(self.s3_path, "wb") + + def close_capture(self) -> None: + for f in (self._bw, self._s3): + if f: + f.close() def connect(self): return self._inner.connect() @@ -144,6 +164,8 @@ class _Timed: def write(self, data: bytes): self.bytes_out += len(data) + if self._bw: + self._bw.write(data); self._bw.flush() return self._inner.write(data) def read(self, n: int) -> bytes: @@ -151,6 +173,8 @@ class _Timed: if chunk: self.reads += 1 self.bytes_in += len(chunk) + if self._s3: + self._s3.write(chunk); self._s3.flush() return chunk @@ -236,6 +260,10 @@ def main() -> int: ap.add_argument("--timeout", type=float, default=10.0) ap.add_argument("--download", action="store_true", help="also download the first stored event (read-only)") + ap.add_argument("--capture", metavar="DIR", + help="also write a raw_bw_*/raw_s3_*.bin pair to DIR, so " + "this run can become a test fixture. Worth doing on " + "any unit whose firmware line is new to us.") ap.add_argument("--lenient", action="store_true", help="do not raise on a bad checksum — for diagnosis only") a = ap.parse_args() @@ -248,7 +276,7 @@ def main() -> int: inner = StdlibSerial(a.target, baud=a.baud) path = f"serial {a.target} @ {a.baud}" - transport = _Timed(inner) + transport = _Timed(inner, capture=a.capture) mm = MicromateClient(transport, recv_timeout=a.timeout, strict_checksums=not a.lenient) @@ -311,6 +339,11 @@ def main() -> int: return 3 finally: mm.close() + transport.close_capture() + if a.capture: + print(f"\n capture written:\n {transport.bw_path}\n {transport.s3_path}") + print(" parse it with: python3 scratch/mm_frame_parse.py " + f"{transport.bw_path} {transport.s3_path}") elapsed = time.monotonic() - t0 print(f"\n transport: {transport.reads} reads, " diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index d11e450..819aa09 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -854,11 +854,41 @@ captured `0x5A` responses and fed to `micromate.idf_file.read_idf_file()`: `thor-watcher` forwards today, so `/db/import/idf_file` ingests a directly downloaded event unchanged. +### ✅ `11.0BD` confirmed on real hardware (2026-09-30) + +UM20147 read over USB. **Every inference about the Thor firmware line held**, +and two of the three were covered only by *synthesised* test frames until now: + +| inference | predicted | UM20147 reported | +|---|---|---| +| `flags = 0x03` identifies the line | `thor` | ✅ `thor` | +| `0x03` is ETX, so it arrives as `10 03` | destuffs before indexing | ✅ implied — the line could not be identified otherwise | +| model string is shorter | `MM/ISEE/S` | ✅ `MM/ISEE/S` | +| `0x1C` is 4 bytes longer, extras **trailing** | forward offsets survive | ✅ battery **3.55 V**, clock correct to the second | + +That last row is the one that mattered. Series III's from-the-end offsets would +have produced **577.92 V** on this unit — a number so obviously wrong it is a +cheap self-check, which is why `mm_client_check.py` watches for it. Reading a +plausible 3.55 V and a correct clock confirms the four extra bytes really are +trailing. + +Also read cleanly: serial `UM20147`, 5 setups (`factory.MMB` → `test2.mmb`), +active setup `test2.mmb`, 15,000,000 bytes total and free. 21 reads, 2,165 B in. + +**One protocol stack drives the whole fleet regardless of firmware line** — the +headline finding of 2026-09-23 — is now demonstrated by a working client, not +just by matching response SUBs. + ### Still not covered -- **`11.0BD`** — UM12947 is a `11.0CB` unit. The Thor firmware line is still - entirely inference: `flags = 0x03`, a shorter model string, and a `0x1C` - block 4 bytes longer. `mm_client_check.py` says so loudly when it meets one. +- **A download from a BD unit.** UM20147 had no events stored, so the chunk + walk is verified on `11.0CB` only. Nothing suggests it differs — the `0x5A` + prefix and the offset arithmetic are not firmware-line dependent in anything + observed — but it is untested. +- **Raw BD bytes are still not in the repo.** The behaviour is confirmed; the + test fixtures for the Thor line remain synthesised. `mm_client_check.py + --capture DIR` now writes a `raw_bw_*`/`raw_s3_*` pair in the layout + `scratch/mm_frame_parse.py` reads, so one more run on UM20147 would close it. - **A unit that is monitoring**, and a unit with a nearly-full event buffer. - **The inbound call-home session** — still the one protocol unknown. diff --git a/micromate/client.py b/micromate/client.py index 997faca..8149863 100644 --- a/micromate/client.py +++ b/micromate/client.py @@ -208,9 +208,10 @@ class MicromateClient: battery voltage of 577.92 V. The four extra bytes are trailing, so from-the-start offsets hold for both lines. - ⚠ Verified on `11.0CB` only. That the same offsets hold on `11.0BD` - follows from the extra bytes being trailing, which is documented but not - something this code has seen. + ✅ **Confirmed on `11.0BD` 2026-09-30.** UM20147 read back 3.55 V and + a clock correct to the second over USB, so the from-the-start offsets do + survive the four extra trailing bytes. Had they not, the battery would + have read 577.92 V — which is what makes this cheap to check. """ data = self._proto.read_monitor_status() c = _content(data) diff --git a/tests/test_micromate_client.py b/tests/test_micromate_client.py index ce0964c..6f1f914 100644 --- a/tests/test_micromate_client.py +++ b/tests/test_micromate_client.py @@ -245,6 +245,10 @@ def test_the_model_string_is_anchored_on_MM_not_on_an_offset(): Anchoring on b"MM/" survives that; a fixed end offset would not. A generic printable-run scan fails for a different reason -- see _parse_poll. + + ✅ CONFIRMED on real hardware 2026-09-30: UM20147 (11.0BD) read back + model="MM/ISEE/S" over USB. The frame below is still synthesised because + no BD capture is in the repo, but the string it asserts is the real one. """ bd = bytearray(POLL) assert bd[CONTENT + 26:CONTENT + 38] == b"MM/ISEE/S/IO" diff --git a/tests/test_micromate_framing.py b/tests/test_micromate_framing.py index d2c15b2..15966c1 100644 --- a/tests/test_micromate_framing.py +++ b/tests/test_micromate_framing.py @@ -236,10 +236,13 @@ def test_thor_firmware_line_survives_destuffing(): A parser that does not destuff ends the frame at byte 2 on half the fleet. - SYNTHESISED: no 11.0BD capture is on disk -- UM20147's sweep was recorded - on 2026-09-23 and those bins never landed in the repo. Built by re-stuffing - the captured POLL probe reply's payload with flags flipped to 0x03, so the - only difference from a real frame is the one byte under test. + SYNTHESISED frame, but the BEHAVIOUR IS CONFIRMED on real hardware: + UM20147 (11.0BD) was read over USB on 2026-09-30 and reported + firmware_line="thor", which is only reachable if `10 03` in the flags + position destuffed correctly. Raw BD bytes are still not in the repo, so + this frame stays synthesised -- built by re-stuffing the captured POLL + probe reply with flags flipped to 0x03, so the only difference from a real + frame is the one byte under test. """ real = unstuff(RSP_POLL_PROBE[1:-1])[:-1] payload = bytes([real[0], FLAGS_THOR]) + real[2:] From ea229ed19d5cfc58ca008c97434b1af2650ce7ee Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 30 Sep 2026 14:02:32 -0400 Subject: [PATCH 13/16] docs(series4): monitoring reads confirmed, and setup-name case is inconsistent Two findings from a UM20147 run that was only meant to capture fixture bytes. 1. A MONITORING UNIT ANSWERS READS NORMALLY. UM20147 was read while actively monitoring -- monitoring=True on both indicators, which agree -- and served POLL, serial, state, monitor status and a 5-entry setup walk with no special handling. This confirms the SESSION_RESET finding with our own client. Series III REQUIRES a bare 41 03 before POLL or a monitoring unit will not answer over TCP. That was inferred from its absence in THOR's captures; it is now demonstrated directly, on the other firmware line, by a client that never sends one. (THOR's refusal to send a SETUP while monitoring is a write restriction -- reads are unaffected.) Monitoring moves two values, so neither is stable to compare against: battery drifted 3.55 -> 3.50 V and memory free 15,000,000 -> 14,848,448. 2. SETUP-NAME CASE IS NOT CONSISTENT BETWEEN COMMANDS. Same unit, same session, same file: 0x41 reported "test2.MMB" and 0x40 reported "test2.mmb". A run 45 minutes earlier reported "test2.mmb" from both, so it is not fixed per command either; what changed between is that the unit started monitoring. Cause is a guess from two samples. The consequence does not depend on it: compare setup file names CASE-INSENSITIVELY. An exact-string test of "is the active setup one I know about?" answers no on this unit. Worth fixing before it is a field bug, since name matching is on the path for anything that replaces THOR's setup/scheduler coupling. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 40 ++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 819aa09..a6371ab 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -879,6 +879,46 @@ active setup `test2.mmb`, 15,000,000 bytes total and free. 21 reads, 2,165 B in. headline finding of 2026-09-23 — is now demonstrated by a working client, not just by matching response SUBs. +### ✅ A MONITORING unit answers reads normally (2026-09-30) + +UM20147 was read while actively monitoring — `monitoring=True` on **both** +indicators (`0x49` content[0] and `0x1C` content[1], which agree) — and +answered POLL, serial, state, monitor status and a 5-entry setup walk with no +special handling. + +**This confirms the `SESSION_RESET` finding with our own client.** Series III +*requires* a bare `41 03` before POLL or a monitoring unit will not answer over +TCP. That was previously inferred from its absence in THOR's captures; it is now +demonstrated directly, on the other firmware line, by a client that never sends +one. + +Note THOR *refuses to send a setup* to a monitoring unit — that restriction is +about writes. Reads are unaffected. + +Monitoring also moves two values, so neither is a stable thing to compare +against: battery drifted 3.55 → 3.50 V, and memory free dropped 15,000,000 → +14,848,448 (monitoring allocates as it runs). + +### ⚠ Setup-name case is NOT consistent between `0x41` and `0x40` + +Same unit, same session, same file: + +``` +0x41 active setup name -> test2.MMB +0x40 setup-list entry -> test2.mmb +``` + +An earlier run 45 minutes before reported `test2.mmb` from *both*, so the case +is not fixed per command either. What changed in between is that the unit +started monitoring — plausibly it rewrites the active-setup name in a canonical +form when loading a setup, but that is a guess from two samples. + +**Practical consequence, which does not depend on the cause: compare setup file +names case-insensitively.** An exact-string test of "is the active setup one I +know about?" answers *no* on this unit. Worth fixing before it is a field bug — +Thor's own UI couples the scheduler to a setup name, so name matching is on the +path for anything that replaces it. + ### Still not covered - **A download from a BD unit.** UM20147 had no events stored, so the chunk From 080fb92ba2782355b76f37f9158716cf3953ccc8 Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 30 Sep 2026 14:08:27 -0400 Subject: [PATCH 14/16] test(micromate): real 11.0BD fixtures replace the synthesised ones UM20147 captured over USB with mm_client_check.py --capture, read out with scratch/mm_frame_parse.py: 11 frames each direction, 0 bad checksums. The two BD-specific response data sections are now verbatim fixtures; only the frame wrapper is rebuilt, which is sound because the framing is verified 251/251 elsewhere in the same file, and the reconstruction is pinned against the payload length the parser reported. Every field lands where the forward-offset model says it should: data[0] = 0x30 = 48 = 59 - 11 (CB declares 0x2C = 44; the +4 holds) content[1] = 0x0e monitoring content[2:10] = 30 Sep 2026 14:00:49 matches the wall clock content[34:36] = 3.50 V content[36:44] = 15,000,000 total / 14,848,448 free AND THE WARNING IS NOW DEMONSTRATED ON THE UNIT IT PREDICTED. Series III reads battery from data[-10:-8], which on this block is e1 c0 -> 577.92 V. That is the exact figure the A/B section named, and there is now a test asserting it, so the reason the offsets are forward cannot be quietly refactored away. Two corrections to the reference: 1. The four extra BD bytes are `0f a0 00 04`, not `0f a0 00 00`. Only the first two look fixed. Nothing reads them, but the last is not padding. 2. POLL content[3] VARIES BETWEEN UNITS -- 0x50 on UM12947, 0x56 on UM20147. It is printable in both cases ("P" and "V"), which is a second, independent reason the POLL block cannot be parsed by scanning for printable runs: a scan returns "PInstantel" on one unit and "VInstantel" on the other. The fixed offset content[4] was the right call for a reason I had not seen. The model string sits at content[26] on both firmware lines; only its tail differs, so anchoring on "MM/" remains correct. 98 micromate tests pass. Full suite unchanged at 16 pre-existing failures. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 49 +++++++++++++-- tests/test_micromate_client.py | 93 ++++++++++++++++++++++++++++ tests/test_micromate_framing.py | 35 +++++++---- 3 files changed, 161 insertions(+), 16 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index a6371ab..326e6bc 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -919,16 +919,57 @@ know about?" answers *no* on this unit. Worth fixing before it is a field bug Thor's own UI couples the scheduler to a setup name, so name matching is on the path for anything that replaces it. +### ✅ Real BD bytes, and the exact numbers (2026-09-30) + +Captured with `mm_client_check.py --capture` and read out with +`scratch/mm_frame_parse.py`: 11 frames each direction, 0 bad checksums. Both +BD-specific responses are now real fixtures in `tests/`, replacing synthesised +ones. + +**`SUB 0x1C`, BD, 59 B data — the block the from-the-end offsets break on:** + +``` +content[ 1] 0x0e monitoring +content[ 2:6] 1e 09 07ea 30 Sep 2026 +content[ 6] 0x60 = 96 ⚠ still unidentified (32/100/116 on CB) +content[ 7:10] 0e 00 31 14:00:49 ✓ matches the wall clock +content[34:36] 01 5e 3.50 V +content[36:40] 00 e4 e1 c0 15,000,000 +content[40:44] 00 e2 91 c0 14,848,448 +content[44:48] 0f a0 00 04 ⚠ the four extra bytes +``` + +`data[0] = 0x30` = 48 = 59 − 11, against CB's `0x2C` = 44. The low-byte length +rule and the +4 both hold. + +⚠ **The extra four bytes are `0f a0 00 04`, not `0f a0 00 00`** as recorded +earlier. Only the first two look fixed. Nothing reads them, but do not treat +the last as padding. + +**And the warning is now demonstrated on the unit it predicted:** +`data[-10:-8]` on this block is `e1 c0` → **577.92 V**. Exactly the figure in +the A/B section. Forward offsets give 3.50 V. + +**`SUB 0x5B` POLL, BD:** + +``` +content[ 3] 0x56 ⚠ 0x50 on UM12947 — THIS BYTE VARIES BETWEEN UNITS +content[ 4] "Instantel\0" +content[26] "MM/ISEE/S\0" (CB: "MM/ISEE/S/IO\0", same offset) +``` + +⚠ **content[3] is not a constant.** It is printable on both units — `P` and `V` +— which is the second reason a printable-run scan is the wrong way to read this +block: it would return `PInstantel` on one unit and `VInstantel` on the other. +Read the vendor at the fixed offset content[4]; the model is at content[26] on +both lines, but anchor on `MM/` since only its tail varies. + ### Still not covered - **A download from a BD unit.** UM20147 had no events stored, so the chunk walk is verified on `11.0CB` only. Nothing suggests it differs — the `0x5A` prefix and the offset arithmetic are not firmware-line dependent in anything observed — but it is untested. -- **Raw BD bytes are still not in the repo.** The behaviour is confirmed; the - test fixtures for the Thor line remain synthesised. `mm_client_check.py - --capture DIR` now writes a `raw_bw_*`/`raw_s3_*` pair in the layout - `scratch/mm_frame_parse.py` reads, so one more run on UM20147 would close it. - **A unit that is monitoring**, and a unit with a nearly-full event buffer. - **The inbound call-home session** — still the one protocol unknown. diff --git a/tests/test_micromate_client.py b/tests/test_micromate_client.py index 6f1f914..2baac1d 100644 --- a/tests/test_micromate_client.py +++ b/tests/test_micromate_client.py @@ -63,6 +63,27 @@ MS_LATE = bytes.fromhex( # 55 B, from_20260925_011403_ "e4e1c000e3e1c0" ) +# ── Real 11.0BD bytes (UM20147, captured over USB 2026-09-30) ───────────────── +# +# The Thor firmware line, which was pure inference until this capture. These +# are the two responses where BD differs from CB. Captured with +# `mm_client_check.py --capture` and read out of the resulting pair with +# scratch/mm_frame_parse.py, so the data sections are verbatim; only the frame +# wrapper is rebuilt, and the framing is independently verified 251/251. + +POLL_BD = bytes.fromhex( # 59 B -- flags 0x03, and the SHORTER model string + "30000000000000000000000000005649" + "6e7374616e74656c000600c3f04a00e4" + "194f0052024d4d2f495345452f530058" + "f406001f60c0755760c075" +) +MS_BD_MONITORING = bytes.fromhex( # 59 B -- FOUR BYTES LONGER than CB's 55 + "3000000000000000000000000e1e0907" + "ea600e00310000000000000000000000" + "00000000000000000000000000015e00" + "e4e1c000e291c00fa00004" +) + _SETUP_PAD = 266 - CONTENT @@ -398,3 +419,75 @@ def test_context_manager_opens_and_closes(): assert t.is_connected() assert mm.connect().serial == "UM12947" assert not t.is_connected() + + +# ── The Thor firmware line, on real bytes ───────────────────────────────────── + +def test_bd_constants_have_the_lengths_the_parser_reported(): + assert len(POLL_BD) == 59 + assert len(MS_BD_MONITORING) == 59, "CB is 55; BD is four bytes longer" + assert MS_BD_MONITORING[0] == 0x30, "low-byte length 48 = 59 - 11" + assert MS_IDLE[0] == 0x2C, "the CB equivalent declares 44" + + +def test_connect_on_a_thor_line_unit(): + """UM20147, real POLL bytes. flags 0x03 is ETX, so it arrives as `10 03`.""" + mm, _ = client([ + frame(0xA4, POLL_BD, flags=FLAGS_THOR), frame(0xEA, SERIAL), + frame(0xB6, STATE_MONITORING), frame(0xBE, setup_response("test2.MMB")), + ]) + info = mm.connect() + assert info.firmware_line == "thor" + assert info.model == "MM/ISEE/S", "the BD model string is shorter than CB's" + assert info.manufacturer == "Instantel" + assert info.monitoring is True + + +def test_poll_content_3_is_not_a_constant(): + """⚠ 0x50 on UM12947, 0x56 on UM20147 -- it VARIES between units. + + This is why the manufacturer is read at the fixed offset content[4] and not + by scanning: content[3] is printable in both cases ("P" and "V"), so a + printable-run scan would return "PInstantel" on one unit and "VInstantel" on + the other. Whatever the byte is, it is not a stable marker to anchor on. + """ + assert _content(POLL)[3] == 0x50 + assert _content(POLL_BD)[3] == 0x56 + assert chr(_content(POLL_BD)[3]) == "V" + + +def test_get_state_on_a_thor_line_unit(): + """⚠ THE test for the forward-offset decision. Real UM20147 bytes. + + The 0x1C block is four bytes longer here, and the extras are TRAILING, so + offsets measured from the start of content are unmoved. Series III reads + battery and memory from the END of this block; test_series_iii_offsets_... + below shows what that produces. + """ + mm, _ = client([frame(0xE3, MS_BD_MONITORING, flags=FLAGS_THOR)]) + st = mm.get_state() + + assert st.monitoring is True + assert st.device_time == datetime.datetime(2026, 9, 30, 14, 0, 49) + assert st.battery_volts == 3.50 + assert st.memory_total_bytes == 15_000_000 + assert st.memory_free_bytes == 14_848_448 + + +def test_series_iii_from_the_end_offsets_give_577_volts_on_a_bd_unit(): + """The exact number the protocol reference warned about, now demonstrated. + + 577.92 V is not a plausible battery reading for anything, which is what + makes it a free self-check -- bridges/mm_client_check.py watches for it. + """ + assert int.from_bytes(MS_BD_MONITORING[-10:-8], "big") / 100 == 577.92 + + +def test_the_four_extra_bd_bytes_are_not_all_zero(): + """⚠ The reference records them as `0f a0 00 00`; UM20147 sent `0f a0 00 04`. + + Only the first two bytes look fixed. Nothing reads them, but a future + decoder must not treat the last one as padding. + """ + assert _content(MS_BD_MONITORING)[44:] == bytes.fromhex("0fa00004") + assert _content(MS_IDLE)[44:] == b"", "the CB block has no such tail" diff --git a/tests/test_micromate_framing.py b/tests/test_micromate_framing.py index 15966c1..3392154 100644 --- a/tests/test_micromate_framing.py +++ b/tests/test_micromate_framing.py @@ -231,29 +231,40 @@ def test_escaped_bytes_land_in_the_right_field(): assert f.checksum_valid +# The real 11.0BD POLL data section, UM20147 over USB 2026-09-30. This +# replaces a synthesised frame -- the Thor firmware line was inference-only +# until this capture. +_POLL_BD_DATA = bytes.fromhex( + "30000000000000000000000000005649" + "6e7374616e74656c000600c3f04a00e4" + "194f0052024d4d2f495345452f530058" + "f406001f60c0755760c075" +) + + def test_thor_firmware_line_survives_destuffing(): """⚠ flags = 0x03 is ETX, so it arrives as `10 03`. A parser that does not destuff ends the frame at byte 2 on half the fleet. - SYNTHESISED frame, but the BEHAVIOUR IS CONFIRMED on real hardware: - UM20147 (11.0BD) was read over USB on 2026-09-30 and reported - firmware_line="thor", which is only reachable if `10 03` in the flags - position destuffed correctly. Raw BD bytes are still not in the repo, so - this frame stays synthesised -- built by re-stuffing the captured POLL - probe reply with flags flipped to 0x03, so the only difference from a real - frame is the one byte under test. + The data section is REAL (UM20147, 11.0BD); the frame wrapper is rebuilt, + which is sound because the framing is verified 251/251 elsewhere in this + file. scratch/mm_frame_parse.py reported payload=64 for this frame, and + the assertion below pins that, so the reconstruction is checked rather than + assumed. """ - real = unstuff(RSP_POLL_PROBE[1:-1])[:-1] - payload = bytes([real[0], FLAGS_THOR]) + real[2:] - synth = bytes([STX]) + stuff(payload + bytes([checksum(payload)])) + bytes([ETX]) + payload = bytes([0x00, FLAGS_THOR, 0xA4, 0x00, 0x00]) + _POLL_BD_DATA + assert len(payload) == 64, "the parser reported payload=64 for this frame" + wire = bytes([STX]) + stuff(payload + bytes([checksum(payload)])) + bytes([ETX]) - assert bytes([DLE, ETX]) in synth, "flags must be escaped on the wire" - (f,) = MicromateFrameParser().feed(synth) + assert bytes([DLE, ETX]) in wire, "flags 0x03 must be escaped on the wire" + (f,) = MicromateFrameParser().feed(wire) assert f.flags == FLAGS_THOR assert f.firmware_line == "thor" assert f.sub == 0xA4 + assert f.request_sub == 0x5B assert f.checksum_valid + assert b"MM/ISEE/S\x00" in f.data, "the shorter BD model string" def test_an_escaped_checksum_byte_is_read_correctly(): From 7d1b033a05ddd0f832fb7ecfcc46d964b2d2299c Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 30 Sep 2026 14:24:03 -0400 Subject: [PATCH 15/16] docs(series4): BD download works -- and event keys COLLIDE across units UM20147, event 055d4a81, 4,796 B, type read as histogram from the 0x0C byte and decoded as .IDFH on the first try. The chunk walk, the offset arithmetic, the 11-byte chunk prefix and the record-type byte all hold on the Thor firmware line unchanged. That closes the last BD gap. The type byte is now 7 events across both firmware lines -- 4 waveform, 3 histogram -- correct every time. Still not a large sample; the fallback stays. THE FINDING THAT MATTERS: UM20147's first event key is 055d4a81. So is UM12947's. Different units, different sizes (4,796 vs 4,076 B), different contents, identical key. This confirms keys are a sequential counter, and adds the consequence: the counter starts from the same value on every unit, so a key is MEANINGLESS WITHOUT ITS SERIAL. Anything that stores, deduplicates or addresses Series IV events must key on (serial, event_key). A store keyed on the event key alone silently treats one unit's event as a duplicate of another's, and the failure is invisible -- the second event is simply never ingested. Series III has a related hazard the ACH server already handles: its counter resets after an erase, so keys are reused WITHIN a unit, which is why ach_state.json tracks max_downloaded_key per serial. Series IV inherits that and adds cross-unit collision on top. Incidentally resolved: memory free is a usable signal after all. Run 1 showed 15,000,000 free and no events; run 3 showed 14,799,296 free, idle, with events stored. The earlier ambiguity was monitoring overhead confounding it, not memory being meaningless. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 35 ++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 5 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 326e6bc..ad2020b 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -964,12 +964,37 @@ block: it would return `PInstantel` on one unit and `VInstantel` on the other. Read the vendor at the fixed offset content[4]; the model is at content[26] on both lines, but anchor on `MM/` since only its tail varies. -### Still not covered +### ✅ Download from a BD unit (2026-09-30) -- **A download from a BD unit.** UM20147 had no events stored, so the chunk - walk is verified on `11.0CB` only. Nothing suggests it differs — the `0x5A` - prefix and the offset arithmetic are not firmware-line dependent in anything - observed — but it is untested. +UM20147, event `055d4a81`, 4,796 B, **type read as histogram from the `0x0C` +byte and decoded as `.IDFH` on the first try.** The chunk walk, the offset +arithmetic, the 11-byte chunk prefix and the record-type byte all hold on the +Thor firmware line unchanged. + +The type byte is now **7 events across both firmware lines** — 4 waveform, +3 histogram — and has predicted correctly every time. Still not a large +sample, and the fallback stays. + +### 🔑 EVENT KEYS COLLIDE ACROSS UNITS — dedup must be (serial, key) + +UM20147's first event is `055d4a81`. So is UM12947's. **Different units, +different sizes (4,796 B vs 4,076 B), different contents, identical key.** + +This confirms *keys are a sequential counter* (already recorded) and adds the +consequence: **the counter starts from the same value on every unit, so a key is +meaningless without its serial.** + +⚠ **Anything that stores, deduplicates or addresses Series IV events must key on +`(serial, event_key)`.** A store keyed on the event key alone will silently +treat one unit's event as a duplicate of another's — and the failure is invisible, +because the second event is simply never ingested. + +Series III has a related hazard the ACH server already handles: after an erase +its counter resets, so keys are reused *within* a unit, which is why +`ach_state.json` tracks `max_downloaded_key` per serial. Series IV inherits that +and adds cross-unit collision on top. + +### Still not covered - **A unit that is monitoring**, and a unit with a nearly-full event buffer. - **The inbound call-home session** — still the one protocol unknown. From a5237bb3b4bf013eda438d33d686191bd2d85b27 Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 30 Sep 2026 14:27:31 -0400 Subject: [PATCH 16/16] docs(series4): RESOLVED -- the 0x0C float is the PER-SAMPLE peak vector sum Exact to 0.000% on all four bench waveforms, against a PVS recomputed from the decoded samples. The offset (Tran label - 12) is now established rather than inferred: four independent confirmations plus a fifth on the other firmware line. RETRACTION. The previous entry observed the float running 2-5% above max(T,V,L) and concluded it was NOT the peak vector sum, "because computed PVS runs higher than both". That computed PVS was sqrt(T^2+V^2+L^2) from the three REPORTED CHANNEL PEAKS, which is an upper bound on the real quantity rather than the quantity -- the channel maxima do not occur at the same instant, so combining them overstates the true peak. Every row of the old table is in fact consistent with max(T,V,L) <= stored <= sqrt(sum of squares). I also got this wrong in the other direction today and want it recorded. On UM20147's event the stored float matched sqrt(sum of peak squares) to 0.048%, and I called it the vector sum on that basis. It is not -- that event's peaks merely happen to be nearly coincident in time. The six CB events separated the two formulas immediately, with errors to -9.5%. ONE SAMPLE AGREEING WITH A FORMULA IS NOT EVIDENCE WHEN A SECOND FORMULA FITS IT EQUALLY WELL. Why it is worth having: it is a FREE SELF-CHECK ON THE DECODER. The device computed this number from the same samples we decode, independently of our codec, so a mismatch means the decode is wrong. Given this codebase's history of silent channel truncation -- walker bugs that shorten a channel and raise nothing -- a per-event invariant costing one float comparison is cheap insurance, and worth wiring into the download path. Also added the full 0x0C field map from real UM20147 bytes: timestamp, the record-type byte, sensor-location label, setup name, serial, the PVS float and the four channel labels with their offsets. Noted in passing: the setup name appears as "test2" in 0x0C, "test2.MMB" in 0x41 and "test2.mmb" in 0x40 -- three commands, three spellings of one file, in one session. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 77 ++++++++++++++++++++++++++-- 1 file changed, 74 insertions(+), 3 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index ad2020b..d93d0ba 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1811,7 +1811,55 @@ given it, and `/db/import/idf_file` needs no change at all. must carry it out of the chain walk. Losing it means losing the ability to name the file correctly. -### ⚠ Unresolved: the `0x0C` peak float +### ✅ RESOLVED 2026-09-30: the `0x0C` float is the PER-SAMPLE peak vector sum + +**`0x0C` content[`Tran` label − 12], float32 BE, = the peak vector sum computed +sample by sample.** Exact to 0.000% on all four bench waveforms, against a PVS +recomputed from the decoded samples: + +| key | `0x0C` float | per-sample PVS | err | `sqrt(Σpeak²)` | `max(T,V,L)` | +|---|---|---|---|---|---| +| `…82` | 1.37198 | 1.37198 | **+0.000%** | 1.41746 | 1.37063 | +| `…83` | 2.35414 | 2.35414 | **+0.000%** | 2.37824 | 2.22274 | +| `…84` | 3.51515 | 3.51515 | **−0.000%** | 3.78731 | 3.44194 | +| `…85` | 0.42267 | 0.42267 | **−0.000%** | 0.46678 | 0.41985 | + +> #### ⚠ Retraction — why this looked unresolved +> +> The previous entry (below, kept) observed the float running 2–5% above +> `max(T,V,L)` and concluded it was **not** the peak vector sum, because +> "computed PVS runs *higher* than both". +> +> That computed PVS was `sqrt(T² + V² + L²)` **from the three reported channel +> peaks** — and that is an *upper bound* on the real thing, not the real thing. +> The channel maxima do not occur at the same instant, so combining them +> overstates the true peak. Every measurement in the old table is consistent +> with the correct answer: `max(T,V,L) ≤ stored ≤ sqrt(Σpeak²)`, in all five rows. +> +> A near-miss worth recording: on UM20147's event the stored float matched +> `sqrt(Σpeak²)` to **0.048%**, which briefly looked like proof it *was* that +> quantity. It is not — the peaks on that event simply happen to be nearly +> coincident in time. **One sample agreeing with a formula is not evidence when +> a second formula fits it equally well;** the six CB events separated them +> immediately (errors to −9.5%). + +**The offset is now established, not inferred** — four independent confirmations +at 0.000%, plus a fifth on the other firmware line. + +**Why this is worth having: it is a free self-check on the decoder.** The device +computed this number from the same samples we decode, independently of our +codec. If a decoded event's per-sample PVS does not match the stored float, the +decode is wrong. Given this codebase's history of *silent* channel truncation — +walker bugs that shorten a channel and raise nothing — a per-event invariant +that costs one float comparison is cheap insurance. Worth wiring into the +download path. + +⚠ It remains true that `max(T,V,L)` and `sqrt(Σpeak²)` are **not** +interchangeable with it, so a consumer wanting the vector sum must use this +field or decode the samples — it cannot be reconstructed from the three +per-channel peaks. + +#### Superseded: the original "unresolved" entry The float32 extracted from `0x0C` runs 2–5% above `max(Tran, Vert, Long)` from the decoded samples: @@ -1824,15 +1872,38 @@ the decoded samples: | `…84` | 3.5152 | 3.4419 | | `…85` | 0.4227 | 0.4198 | -It is not peak vector sum either (computed PVS runs *higher* than both). The +~~It is not peak vector sum either (computed PVS runs *higher* than both). The field may not be the peak at all — its offset was inferred from a byte marker, -not established. **Do not rely on it** until it is pinned properly. +not established. **Do not rely on it** until it is pinned properly.~~ Worth noting the histogram (`…81`) and the loudest waveform (`…84`) report *identical* peaks to four decimals, in both measures. That is self-consistent: the histogram's single 1-minute interval spans the whole thumping session, so its maximum should equal the loudest event in it. +### `0x0C` field map, from UM20147 (11.0BD, 2026-09-30) + +Real bytes, 221 B response → 210 B content declared at `data[0] = 0xd2`: + +``` +content[ 0] day 0x1e = 30 +content[ 1] month 0x09 +content[ 2:4] year BE 0x07ea = 2026 +content[ 4] ⚠ 0xb3 = 179 — unidentified, same slot as 0x1C's content[6] +content[ 5:8] h:m:s 13:27:33 +content[11] RECORD TYPE 0x08 histogram / 0x07 waveform +content[12] "Location\0" sensor-location label +content[34] "test2\0" the setup file name, without its extension +content[76] "UM20147\0" serial +content[86] float32 BE the per-sample PEAK VECTOR SUM (= Tran − 12) +content[98] "Tran\0\0" + float32 BE at +6 +content[112] "Vert" … content[126] "Long" … content[140] "Mic" … +``` + +⚠ The setup name here is `test2` — no extension — where `0x41` reported +`test2.MMB` and `0x40` reported `test2.mmb` in the same session. Three commands, +three spellings of one file. See *Setup-name case*. + ## Event download, per-event delete, and ACH config (2026-09-25) Three captures with operator-supplied ground truth, including Thor screenshots of