serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(micromate): protocol layer -- reads only, verified against Thor's frames

Browse Source 
Step 2 of docs/micromate_client_spec.md: micromate/protocol.py plus 35
offline tests.  Reads only; nothing here writes, erases or changes
monitoring state.

The tests replay Thor's captured responses through a scripted transport and
assert the bytes we emit are the bytes Thor emits -- including a full replay
of the six-event download session, all 56 0x5A frames byte-for-byte.  A
passing test therefore means a real unit has already answered exactly that
frame.

Measuring the spec's command table against the captures found three more
errors in it, on top of the three the framing work found:

1. SUB 0x0A is the MONITOR-LOG WALK, not a keyed "event header, 30 B list
   record" read.  The same request repeated returns successive 297-byte
   records -- serial, mode, thresholds -- until an 11-byte ack ends the
   list.  The device holds the cursor; nothing in the request selects a
   record, and all nine captured frames carry identical params.

   Structural divergence worth noting: Series III reaches the same data via
   a record-type discriminator on its event chain, so partials and events
   share one walk.  Here the monitor log has its own cursor and the event
   chain never sees it.

2. 0x1E/0x1F carry token 0xFE at params[7].  The protocol reference
   documents all-zero params -- that was our own browse probing, which also
   worked.  Thor sends 0xFE on browse and download alike.

3. SUB 0x01 (device info) is never read by Thor in any captured session.
   Its 0xFFFF offset comes from our own probes, so it is the one read in
   the table with no Thor precedent.  Flagged in the docstring.

Two useful negatives, both from absence rather than presence:

- No SESSION_RESET (41 03).  Series III needs that 2-byte signal or a
  monitoring unit will not answer POLL over TCP.  Zero occurrences across
  all 8 sessions, including 40 frames exchanged with a unit that WAS
  monitoring.
- No universal preamble.  The only invariant is that a session opens with
  POLL; POLL -> SERIAL -> 0x49 -> POLL is Thor's connection check and
  appears in 3 of 8 sessions.  Setup pushes and scheduler reads open
  differently.

Two deliberate divergences from the Series III sibling:

- strict_checksums defaults True and RAISES.  minimateplus logs and
  continues because its parser cannot always tell an inner-frame delimiter
  from a checksum byte; that does not apply here, where the rule is exact
  on 251/251 frames.  The lenient default is instructive -- it hid a wrong
  checksum rule for two days.
- read_event_file() raises ShortRead rather than returning a truncated
  event.  The expected length is known up front, so the check is free, and
  a silently short event is the failure mode this codebase keeps hitting.

Every exchange resets the parser before sending, so a leftover frame is
discarded rather than answered with -- expected_sub catches a mismatched
SUB, but a same-SUB leftover would sail through with data for the wrong key.

File transfer (0x94/0x48) is deliberately out of scope: it needs a
data-carrying request frame, which is the frame type writes use, and that
boundary is worth keeping crisp in a read-only pass.

Full suite unchanged at 16 pre-existing failures (missing gitignored
fixtures); 419 passed, up 35.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
serversdown
2026-09-28 19:58:22 -04:00
co-authored by Claude Opus 5
parent 5fe99568a2
commit a7e3a8f20a
4 changed files with 1035 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/micromate_client_spec.md
+42 -11
View File
@@ -173,7 +173,16 @@ timeout, and that distinction earned its keep during the Series III work).
---
## `micromate/protocol.py`
## `micromate/protocol.py` — ✅ BUILT 2026-09-27
Implemented, with `tests/test_micromate_protocol.py` (35 tests). Reads only;
nothing here writes, erases or changes monitoring state.
The tests replay Thor's captured responses through a scripted transport and
assert **the bytes we emit are the bytes Thor emits** — including a full replay
of the six-event download session, all 56 `0x5A` frames byte-for-byte. That is
a stronger guarantee than "our parser understands the device": a passing test
means a real unit has already answered exactly that frame.
One method per command, returning raw payload bytes. No interpretation — that
belongs in `client.py`.
@@ -198,9 +207,29 @@ taking its data length. Per-command offsets, all observed:
| arm event | `0x93` | `0x6C` | — | before every event |
| first event | `0x1E` | `0xE1` | `0xFFFF` | key + size |
| next event | `0x1F` | `0xE0` | `0xFFFF` | key + size |
| event record | `0x0C` | `0xF3` | `0xFFFF` | 210 B — project, location, peaks |
| event header | `0x0A` | `0xF5` | `0xFFFF` | 30 B list record |
| bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim** |
| event record | `0x0C` | `0xF3` | `0xFFFF` | 221 B — project, location, peaks |
| ~~event header~~ **monitor log** | `0x0A` | `0xF5` | `0xFFFF` | ⚠ 297 B, a **walk** — see below |
| bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim**, 1024 B at a time |
⚠ **Corrected 2026-09-27, from Thor's frames.** Three rows of the table above
were wrong or incomplete, and the last one is a different command than labelled:
- **`0x0A` is the monitor-log walk**, not a keyed "30 B list record" read. The
*same request repeated* returns successive 297-byte records — serial, mode,
thresholds — until an 11-byte ack ends the list. The device holds the cursor;
nothing in the request selects a record. Series III reaches this data through
a record-type discriminator on its event chain; here it has its own cursor and
the event chain never sees it.
- **`0x1E`/`0x1F` carry token `0xFE` at `params[7]`.** The reference documents
all-zero params (our own probing, which also worked). Thor's form is the one
with mileage.
- **`0x01` has no Thor frame behind it** — it is never read in any captured
session. Its `0xFFFF` comes from our probes.
And one useful negative: **no `SESSION_RESET` (`41 03`)**. Series III needs that
2-byte signal or a monitoring unit will not answer `POLL` over TCP. Thor never
sends it — zero occurrences across 8 sessions, including 40 frames exchanged
with a unit that *was* monitoring.
⚠ **`SUB 0x1C` is 4 bytes longer on the Thor firmware line** (`0x30` vs `0x2C`).
Parse **forward** from `declared_length`, never backward from the end — Series
@@ -328,19 +357,21 @@ then `download_event()` and assert the bytes decode and match a
## Order of work
1. ✅ `framing.py` + its tests — **done 2026-09-27**, 31 tests, offline
2. `protocol.py` — reads only, one method per row of the table above
2. ✅ `protocol.py` + its tests — **done 2026-09-27**, 35 tests, offline
3. `client.py` — `connect()`, `get_state()`, `list_setups()`
4. the event chain and `download_event()`
5. decode end-to-end and compare against a store event
Steps 1–2 need no hardware at all.
**Worth carrying forward from step 1:** every rule got checked against the
captures *before* being written, and two of the three the spec asserted turned
out wrong — the escape set (26% of frames) and the checksum (22%). Both fail
silently. The captures are on disk and a re-stuff-and-compare loop takes about
two minutes per rule, so do that for `protocol.py`'s per-command offsets too
rather than trusting the table above.
**Worth carrying forward.** Both steps began by measuring against the captures
rather than trusting this document, and both found errors in it — three in the
framing rules (the escape set, 26% of frames; the checksum, 22%; the `0x5A`
chunk model) and three more in the command table (`0x0A`'s meaning, the
`1E`/`1F` token, `0x01`'s provenance). All six fail quietly. The captures are on
disk and a measure-then-write loop costs about two minutes per rule, so keep
doing it for `client.py`'s field offsets — and treat this spec as a plan, not a
source.
---
docs/micromate_protocol_reference.md
+90
View File
@@ -1216,6 +1216,96 @@ the protocol's requirements*. Thor sends it before trivial reads too, so it may
be habit rather than handshake. Do not assume it is mandatory. Note `POLL` here carries `offset = 0x0030` (its data
length), not `0xFFFF` — `POLL` is the one read Thor still addresses by length.
> #### ⚠ Narrowed 2026-09-27 — there is no *universal* preamble
>
> Across all 8 captured sessions, the only invariant is that **the session opens
> with `POLL`**. What follows depends on the operation:
>
> | opening sequence | sessions | operation |
> |---|---|---|
> | `5b 15 49 5b …` | 3 | status refresh / monitoring / ACH change |
> | `5b 41 08 2e 1a da …` | 3 | setup push |
> | `5b 94 48 48 48 …` | 2 | scheduler read |
>
> So `POLL → SERIAL → 0x49 → POLL` is Thor's **connection check**, not a
> handshake the protocol demands — it appears where Thor wants to refresh what
> it displays. Treat `POLL` as the one thing to send first.
### 🔑 No `SESSION_RESET` — the Series III requirement does not carry over
Series III needs a bare `41 03` (ACK + ETX, no STX) to wake a unit that is
actively monitoring; without it the unit will not answer `POLL` over TCP, and
`protocol.startup()` sends it before and between the POLL frames.
**Thor never sends it to a Micromate.** Zero occurrences across all 8 sessions
— including `raw_bw_20260924_191214_turn_on_monitormode_…`, which exchanges 40
frames with a unit that *was* monitoring at the time.
### Measured offsets and response lengths (all read off Thor's frames)
`offset = 0xFFFF` for everything except two commands. Data lengths are from
UM12947 (`11.0CB`) and are **orientation, not assertions** — `0x1C` is 4 bytes
longer on the Thor line.
| SUB | rsp | offset | data | notes |
|---|---|---|---|---|
| `0x5B` POLL | `0xA4` | **`0x0030`** | 59 | the one length-addressed read |
| `0x15` serial | `0xEA` | **`0x000A`** | 21 | |
| `0x49` state | `0xB6` | `0xFFFF` | 16 | |
| `0x1C` monitor status | `0xE3` | `0xFFFF` | 55 | +4 on `11.0BD` |
| `0x06` storage range | `0xF9` | `0xFFFF` | 47 | |
| `0x08` event index | `0xF7` | `0xFFFF` | 101 | contents unmapped |
| `0x2E` trigger config | `0xD1` | `0xFFFF` | 39 | |
| `0x1A` compliance | `0xE5` | `0xFFFF` | 2103 | one frame, not Series III's four |
| `0x2C` call-home | `0xD3` | `0xFFFF` | 137 | |
| `0x3F`/`0x40`/`0x41` setups | `0xC0`/`0xBF`/`0xBE` | `0xFFFF` | 266 | |
| `0x93` arm | `0x6C` | `0xFFFF` | 11 | ack only |
| `0x1E`/`0x1F` chain | `0xE1`/`0xE0` | `0xFFFF` | 19 | ⚠ **token `0xFE` at `params[7]`** |
| `0x0C` event record | `0xF3` | `0xFFFF` | 221 | full key at `params[4:8]` |
| `0x0A` monitor log | `0xF5` | `0xFFFF` | 297 | ⚠ a **walk** — see below |
| `0x5A` download | `0xA5` | computed | offset+11 | 1024-byte chunk loop |
An acknowledgement is an **11-byte data section**, and that doubles as the
end-of-list signal on the walks.
#### ⚠ `0x1E`/`0x1F` carry token `0xFE`
`params = 00 00 00 00 00 00 00 fe 00 00` on all 7 captured chain reads — the
same `token_params(0xFE)` form Series III uses to arm its bulk stream. The
event-chain section above documents **all-zero params**; that was our own browse
probing, which also worked. Both evidently do, but Thor's form is the one with
mileage on it, and it is sent on browse and download alike.
#### 🔑 `SUB 0x0A` is the monitor-log walk, not a keyed read
The command table long described `0x0A` as a keyed "waveform header / partial
record" read, by analogy with Series III. What the bytes show is a **cursor
walk**: the *same request repeated*, the device advancing its own position.
```
0x93 → 1E → 0x0A ×8 (297 B each: "UM12947", "Histo…", "Ver…", " 0.49", " 28.4")
0x0A (11 B ack = end of list)
```
All nine frames carry identical params (`…00 00 4a 81 00 00`), so nothing in the
request selects the record. Terminate on a response of `ACK_DATA_LEN` (11).
⚠ The `4a 81` is the **low two bytes** of the event key then in play
(`055d4a81`). One key cannot distinguish "the key's low half" from "a cursor
handle that happened to equal it" — both produce those bytes. It does not
matter operationally, since the walk works with the params held constant.
This is a genuine structural divergence: Series III reaches the same data
through a record-type discriminator (`0x2C` partial vs `0x46` full) *on its
event walk*, so partial records and events share one chain. Here the monitor
log has its own cursor and the event chain never sees it.
#### `SUB 0x01` has no Thor frame behind it
Thor never reads device info in any captured session. `0xFFFF` for `0x01` comes
from our own 2026-09-23 probes — it answered correctly on both firmware lines,
but it is the only read in the table with no Thor precedent.
### `SUB 0x96` / `0x97` — start and stop monitoring ✅
Identical to Series III, including the acks:
micromate/protocol.py
+501
View File
@@ -0,0 +1,501 @@
"""
protocol.py — one method per Micromate (Series IV) wire command.
Returns raw payload bytes. Interpretation belongs in ``client.py``; this layer
knows frames, offsets and sequencing, and nothing about what a field means.
Scope: **reads only.** Nothing here writes, erases, or changes monitoring
state. That is deliberate and worth keeping — no command has ever been
originated against a unit by this project; every write in
``docs/micromate_protocol_reference.md`` was performed by THOR while we
recorded. The first thing this codebase ever sends to a customer's instrument
should be a decision someone made on purpose, not a side effect of a client
that grew a method.
Every offset and params layout below was read off THOR's own frames in
``bridges/captures/9-24-26 - micromate2/`` rather than taken from the spec
table, because the same exercise during the framing work found three of that
table's rules wrong. It found three more here:
* ``0x0A`` is the **monitor-log walk** — the same request repeated, the
device advancing its own cursor, terminated by a short response — not the
keyed "event header, 30 B list record" the spec describes.
* ``0x1E``/``0x1F`` carry **token 0xFE** at ``params[7]``. The protocol
reference documents all-zero params for the browse walk; that was our own
probing, and THOR does not do it that way.
* There is **no fixed preamble**. Sessions open with ``POLL`` and go
straight to the operation. ``POLL → SERIAL → 0x49 → POLL`` appears in 3 of
8 captured sessions and is THOR's *connection check*, not a handshake.
And one useful negative: **no ``SESSION_RESET`` (``41 03``).** Series III
needs that 2-byte signal to wake a monitoring unit or it will not answer POLL
over TCP. THOR never sends it — 0 occurrences across all 8 sessions, including
40 frames exchanged with a unit that *was* monitoring.
"""
from __future__ import annotations
import logging
import math
import struct
import time
from typing import Optional
from minimateplus.transport import BaseTransport
from .framing import MicromateFrame, MicromateFrameParser, build_request
log = logging.getLogger(__name__)
DEFAULT_RECV_TIMEOUT = 10.0
# An acknowledgement carries an 11-byte data section and nothing else. It is
# also how the monitor-log walk says "no more records".
ACK_DATA_LEN = 11
# ── Command SUBs ──────────────────────────────────────────────────────────────
SUB_DEVICE_INFO = 0x01
SUB_STORAGE_RANGE = 0x06
SUB_EVENT_INDEX = 0x08
SUB_MONITOR_LOG = 0x0A
SUB_EVENT_RECORD = 0x0C
SUB_SERIAL = 0x15
SUB_COMPLIANCE_CONFIG = 0x1A
SUB_MONITOR_STATUS = 0x1C
SUB_EVENT_FIRST = 0x1E
SUB_EVENT_NEXT = 0x1F
SUB_CALL_HOME_CONFIG = 0x2C
SUB_TRIGGER_CONFIG = 0x2E
SUB_SETUP_FIRST = 0x3F
SUB_SETUP_NEXT = 0x40
SUB_SETUP_ACTIVE = 0x41
SUB_STATE = 0x49
SUB_BULK_DOWNLOAD = 0x5A
SUB_POLL = 0x5B
SUB_ARM_EVENT = 0x93
# ⚠ Reads are SINGLE-STEP. Series III probes at offset 0 to learn the length,
# then reads again at that length; a Micromate returns the whole block when
# asked for 0xFFFF. THOR never probes, which is why `MicromateFrame.probe_length`
# reads 0 on live traffic.
READ_ALL = 0xFFFF
# The two commands that do NOT use READ_ALL, and the data length each returned
# on UM12947 (firmware 11.0CB).
_OFFSETS = {
SUB_POLL: 0x0030, # 59 B — the one offset THOR treats as a constant
SUB_SERIAL: 0x000A, # 21 B
}
# Data-section lengths observed, for orientation only — deliberately NOT
# asserted. `SUB 0x1C` is 4 bytes longer on the Thor firmware line (0x30 vs
# 0x2C declared), so a length check here would fire spuriously on half the
# fleet. See the protocol reference, "A/B: Blastware build vs Thor build".
OBSERVED_DATA_LEN = {
SUB_STORAGE_RANGE: 47, SUB_EVENT_INDEX: 101, SUB_MONITOR_LOG: 297,
SUB_EVENT_RECORD: 221, SUB_SERIAL: 21, SUB_COMPLIANCE_CONFIG: 2103,
SUB_MONITOR_STATUS: 55, SUB_EVENT_FIRST: 19, SUB_EVENT_NEXT: 19,
SUB_CALL_HOME_CONFIG: 137, SUB_TRIGGER_CONFIG: 39,
SUB_SETUP_FIRST: 266, SUB_SETUP_NEXT: 266, SUB_SETUP_ACTIVE: 266,
SUB_STATE: 16, SUB_POLL: 59, SUB_ARM_EVENT: ACK_DATA_LEN,
}
# `1E`/`1F` carry this at params[7]. Series III uses the same value to arm its
# bulk stream; here THOR sends it on every chain read, browse or download.
EVENT_TOKEN = 0xFE
# `SUB 0x5A` chunk size, in bytes of file payload per response.
CHUNK_SIZE = 1024
# Every `0x5A` response prefixes the file bytes with 11 bytes of header.
_CHUNK_PREFIX = 11
# ── Exceptions ────────────────────────────────────────────────────────────────
class ProtocolError(Exception):
"""The device violated the expected protocol."""
class TimeoutError(ProtocolError):
"""No response arrived within the allowed time."""
class ChecksumError(ProtocolError):
"""A received frame failed its checksum."""
class UnexpectedResponse(ProtocolError):
"""The response SUB did not match the request."""
class ShortRead(ProtocolError):
"""A bulk download returned fewer bytes than the device promised."""
# ── Params builders ───────────────────────────────────────────────────────────
def token_params(token: int = EVENT_TOKEN) -> bytes:
"""`1E`/`1F`: the token sits at params[7]."""
return bytes(7) + bytes([token]) + bytes(2)
def key_params(key4: bytes) -> bytes:
"""`0x0C`: the full 4-byte event key at params[4:8]."""
if len(key4) != 4:
raise ValueError(f"key4 must be 4 bytes, got {len(key4)}")
return bytes(4) + key4 + bytes(2)
def key_lo_params(key4: bytes) -> bytes:
"""`0x0A`: only the key's **low two bytes**, at params[6:8].
⚠ Inferred from a single key value. All nine captured `0x0A` frames carry
`4a 81`, and the event key in play was `055d4a81` — so this is consistent
with "the low half of the current key" and equally consistent with "a
cursor handle that happened to equal it". Both readings produce the same
bytes for that key, so one event cannot separate them.
It does not matter much in practice: the walk works with the same params
repeated, so whichever it is, passing the current key is right.
"""
if len(key4) != 4:
raise ValueError(f"key4 must be 4 bytes, got {len(key4)}")
return bytes(6) + key4[2:4] + bytes(2)
def chunk_params(key4: bytes, byte_offset: int) -> bytes:
"""`0x5A`: the key opens the file, then a byte offset walks it.
Chunk 0 carries the event key at params[0:4] — that is what says "from the
beginning". Later chunks carry a uint16 BE byte offset at params[2:4].
"""
if byte_offset == 0:
if len(key4) != 4:
raise ValueError(f"key4 must be 4 bytes, got {len(key4)}")
return key4 + bytes(6)
if not 0 <= byte_offset <= 0xFFFF:
raise ValueError(f"byte_offset must fit in uint16, got {byte_offset}")
return bytes(2) + struct.pack(">H", byte_offset) + bytes(6)
# ── Protocol ──────────────────────────────────────────────────────────────────
class MicromateProtocol:
"""Wire-level command set for one open connection to a Micromate.
Does not own the transport; lifetime belongs to the client.
proto = MicromateProtocol(transport)
proto.poll()
serial = proto.read_serial()
"""
def __init__(
self,
transport: BaseTransport,
recv_timeout: float = DEFAULT_RECV_TIMEOUT,
strict_checksums: bool = True,
) -> None:
"""
Args:
strict_checksums: raise on a bad checksum. **Defaults to True,
unlike the Series III sibling**, which logs and continues
because its parser cannot reliably tell an inner-frame
delimiter from a checksum byte. That excuse does not apply
here: the Micromate rule is plain SUM8 over the de-stuffed
payload and it holds on 251 of 251 captured frames, so a
mismatch means something real — line noise, a desync, or a rule
we have wrong — and all three are worth hearing about.
The lenient Series III default is instructive: it hid the fact
that the documented checksum rule was wrong for two days. Set
False only to get a field diagnosis unstuck.
"""
self._transport = transport
self._recv_timeout = recv_timeout
self._strict = strict_checksums
self._parser = MicromateFrameParser()
self._pending: list[MicromateFrame] = []
# ── Identity and state ────────────────────────────────────────────────────
def poll(self) -> MicromateFrame:
"""`0x5B` → `0xA4`. Handshake; carries the ID block and model string.
Every captured session opens with this and nothing before it.
"""
return self._exchange(SUB_POLL)
def read_serial(self) -> bytes:
"""`0x15` → `0xEA`. ASCII, null-terminated — e.g. `UM12947`."""
return self._read(SUB_SERIAL)
def read_device_info(self) -> bytes:
"""`0x01` → `0xFE`. Firmware, calibration, per-channel float block.
⚠ THOR never sends this in any captured session, so the `0xFFFF` offset
is from our own 2026-09-23 probes rather than from THOR's behaviour. It
answered correctly on both firmware lines, but it is the one read here
with no THOR frame behind it.
"""
return self._read(SUB_DEVICE_INFO)
def read_state(self) -> bytes:
"""`0x49` → `0xB6`. A cheap monitoring check; 16 B.
⚠ Test `data[11]` for **non-zero**, never against a constant — it has
read both `0x0E` and `0x0C` while monitoring.
"""
return self._read(SUB_STATE)
def read_monitor_status(self) -> bytes:
"""`0x1C` → `0xE3`. Flag, **device clock**, battery, memory.
⚠ Parse **forward** from the declared length, never backward from the
end. This block is 4 bytes longer on the Thor firmware line, and Series
III's relative-to-end offsets yield a battery voltage of 577.92 V on a
`11.0BD` unit.
"""
return self._read(SUB_MONITOR_STATUS)
def read_storage_range(self) -> bytes:
"""`0x06` → `0xF9`. Event storage extent; 47 B."""
return self._read(SUB_STORAGE_RANGE)
def read_event_index(self) -> bytes:
"""`0x08` → `0xF7`. 101 B. Contents not yet mapped."""
return self._read(SUB_EVENT_INDEX)
def read_trigger_config(self) -> bytes:
"""`0x2E` → `0xD1`. 39 B. Series IV only; no Series III equivalent."""
return self._read(SUB_TRIGGER_CONFIG)
def read_compliance_config(self) -> bytes:
"""`0x1A` → `0xE5`. The whole active setup — 2103 B on UM12947.
One response. Series III needs a 4-frame sequence for the same thing.
"""
return self._read(SUB_COMPLIANCE_CONFIG)
def read_call_home_config(self) -> bytes:
"""`0x2C` → `0xD3`. 137 B — Series III's is 124, so do not reuse its map."""
return self._read(SUB_CALL_HOME_CONFIG)
# ── Setups ────────────────────────────────────────────────────────────────
def read_active_setup_name(self) -> bytes:
"""`0x41` → `0xBE`. 266 B; carries the active `.MMB` name."""
return self._read(SUB_SETUP_ACTIVE)
def read_first_setup(self) -> bytes:
"""`0x3F` → `0xC0`. Head of the setup-file list."""
return self._read(SUB_SETUP_FIRST)
def read_next_setup(self) -> bytes:
"""`0x40` → `0xBF`. Repeat until the record carries an empty name.
Stateful: the device holds the cursor, so the same request walks the
list. 22 of these appear back to back in one captured session.
"""
return self._read(SUB_SETUP_NEXT)
# ── Event chain ───────────────────────────────────────────────────────────
def arm_event(self) -> MicromateFrame:
"""`0x93` → `0x6C`. THOR sends this before **every** `1E`/`1F`.
It replaces Series III's `1E(token=0xFE)` arming step. No params, no
offset payload — an 11-byte ack.
⚠ Whether a unit actually requires it is untested. Do it because it is
known-good, not because it is known-necessary.
"""
return self._exchange(SUB_ARM_EVENT, offset=READ_ALL)
def read_event_first(self) -> bytes:
"""`0x1E` → `0xE1`. First event key + size; 19 B."""
return self._read(SUB_EVENT_FIRST, params=token_params())
def read_event_next(self) -> bytes:
"""`0x1F` → `0xE0`. Next key + size, or the all-zero null sentinel."""
return self._read(SUB_EVENT_NEXT, params=token_params())
def read_event_record(self, key4: bytes) -> bytes:
"""`0x0C` → `0xF3`. 221 B — project, client, operator, timestamp, peaks.
⚠ The peak float in here runs 2–5% above `max(T,V,L)` and is **not** the
vector sum; its offset was inferred, not established. Prefer decoded
samples.
"""
return self._read(SUB_EVENT_RECORD, params=key_params(key4))
def read_monitor_log_next(self, key4: bytes) -> Optional[bytes]:
"""`0x0A` → `0xF5`. One monitor-log record, or None at end of list.
⚠ Not the keyed single read the spec describes. This is a **walk**:
the same request repeated, the device advancing its own cursor, each
response a 297-byte record carrying serial, mode and thresholds. The
list ends with a bare 11-byte ack — nine captured frames, eight records
then the terminator.
Series III reaches the same data through a record-type discriminator on
its event walk (`0x2C` partial vs `0x46` full). Here it is a separate
cursor and the event chain does not see it at all.
"""
data = self._read(SUB_MONITOR_LOG, params=key_lo_params(key4))
return None if len(data) <= ACK_DATA_LEN else data
# ── Bulk download ─────────────────────────────────────────────────────────
def read_event_file(self, key4: bytes, size: int) -> bytes:
"""`0x5A` → `0xA5`. The `.IDFW`/`.IDFH` file, byte for byte.
`size` is the 4 bytes after the key in the `1E`/`1F` response. Returns
exactly that many bytes, or raises `ShortRead`.
A bounded chunk walk — `ceil(size / 1024)` requests, each asking for
`min(1024, remaining)` bytes:
offset = the byte count wanted (NOT an address)
params = the key on chunk 0, then a uint16 BE byte offset
response = exactly `offset + 11` bytes; file bytes are data[11:]
Verified against THOR on all six bench events (4,076 → 13,424 B):
`sum(offsets) == size` exactly, every time.
⚠ Do not port the Series III `5A` walk. Its address arithmetic caused a
5x over-read and a `>64 KB` page-boundary bug that is *still open* on
that side. Neither applies here — the cursor is a byte offset into the
file, bounded by a size the device supplied, so it cannot run past the
event.
The result feeds `micromate.idf_file.read_idf_file()` and
`/db/import/idf_file` unchanged; no new codec work is needed.
"""
if size <= 0:
raise ValueError(f"size must be positive, got {size}")
out = bytearray()
n_chunks = math.ceil(size / CHUNK_SIZE)
for i in range(n_chunks):
want = min(CHUNK_SIZE, size - i * CHUNK_SIZE)
data = self._read(
SUB_BULK_DOWNLOAD,
offset=want,
params=chunk_params(key4, i * CHUNK_SIZE),
)
if len(data) < _CHUNK_PREFIX:
raise ShortRead(
f"chunk {i + 1}/{n_chunks} of {key4.hex()}: "
f"{len(data)} B is too short to hold a chunk header"
)
body = data[_CHUNK_PREFIX:]
if len(body) != want:
# Worth being loud: a silently short event is the failure mode
# this project has been bitten by repeatedly on the Series III
# side, and here the expected length is known up front.
raise ShortRead(
f"chunk {i + 1}/{n_chunks} of {key4.hex()}: asked for "
f"{want} B, got {len(body)}"
)
out += body
if len(out) != size:
raise ShortRead(
f"{key4.hex()}: assembled {len(out)} B, device promised {size}"
)
log.debug("downloaded %s: %d B in %d chunks", key4.hex(), len(out), n_chunks)
return bytes(out)
# ── Plumbing ──────────────────────────────────────────────────────────────
def _read(
self,
sub: int,
*,
params: bytes = bytes(10),
offset: Optional[int] = None,
timeout: Optional[float] = None,
) -> bytes:
"""Send one read command, return the response's data section."""
return self._exchange(sub, params=params, offset=offset, timeout=timeout).data
def _exchange(
self,
sub: int,
*,
params: bytes = bytes(10),
offset: Optional[int] = None,
timeout: Optional[float] = None,
) -> MicromateFrame:
if offset is None:
offset = _OFFSETS.get(sub, READ_ALL)
# Start every exchange clean: drop any half-frame and any frame left
# stashed by the last one. This is a strict request/response protocol,
# so anything already buffered when we send is by definition stale, and
# `expected_sub` would reject it anyway — better to discard it here than
# to raise a confusing UnexpectedResponse one command later.
self._parser.reset()
self._pending.clear()
self._send(build_request(sub, offset, params))
return self._recv_one(expected_sub=0xFF - sub, timeout=timeout,
reset_parser=False)
def _send(self, frame: bytes) -> None:
log.debug("TX %d bytes: %s", len(frame), frame.hex())
self._transport.write(frame)
def _recv_one(
self,
expected_sub: Optional[int] = None,
timeout: Optional[float] = None,
reset_parser: bool = True,
) -> MicromateFrame:
"""Read until one complete frame is parsed."""
deadline = time.monotonic() + (timeout or self._recv_timeout)
if reset_parser:
self._parser.reset()
self._pending.clear()
if self._pending:
return self._validate(self._pending.pop(0), expected_sub)
while time.monotonic() < deadline:
chunk = self._transport.read(4096)
if not chunk:
time.sleep(0.005)
continue
log.debug("RX %d bytes", len(chunk))
frames = self._parser.feed(chunk)
if frames:
self._pending.extend(frames[1:])
return self._validate(frames[0], expected_sub)
raise TimeoutError(
f"no frame in {timeout or self._recv_timeout:.1f}s"
+ (f" (expected SUB 0x{expected_sub:02X})" if expected_sub is not None else "")
+ f"; {self._parser.bytes_fed} bytes were received"
# That byte count is the whole point: it separates "nothing came
# back at all" from "bytes arrived but never framed", and those have
# completely different causes. It earned its keep on Series III.
)
def _validate(
self, frame: MicromateFrame, expected_sub: Optional[int]
) -> MicromateFrame:
if not frame.checksum_valid:
msg = (
f"SUB 0x{frame.sub:02X}: checksum mismatch "
f"(got 0x{frame.chk_byte:02X}, {len(frame.data)} B data)"
)
if self._strict:
raise ChecksumError(msg)
log.warning("%s — continuing (strict_checksums=False)", msg)
if expected_sub is not None and frame.sub != expected_sub:
raise UnexpectedResponse(
f"expected SUB 0x{expected_sub:02X}, got 0x{frame.sub:02X}"
)
return frame
tests/test_micromate_protocol.py
+402
View File
@@ -0,0 +1,402 @@
"""Protocol-layer tests for the Micromate (series-4) live client.
The load-bearing assertion in here is not "our parser understands the device" —
it is **"the bytes we put on the wire are the bytes THOR puts on the wire."**
Every request constant below is lifted from
``bridges/captures/9-24-26 - micromate2/`` (UM12947, firmware 11.0CB), so a
passing test means a real unit has already answered exactly that frame.
Responses are replayed through a scripted transport. No hardware, no network.
"""
from __future__ import annotations
import os
import sys
from pathlib import Path
import pytest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from micromate import protocol as P
from micromate.framing import ETX, STX, checksum, stuff
from micromate.protocol import (
ACK_DATA_LEN,
ChecksumError,
MicromateProtocol,
ShortRead,
UnexpectedResponse,
chunk_params,
key_lo_params,
key_params,
token_params,
)
FLAGS_CB = 0xC5
# ── Test doubles ──────────────────────────────────────────────────────────────
class ScriptedTransport:
"""Hands back queued responses; records every byte written."""
def __init__(self, responses: list[bytes] | None = None) -> None:
self.queue = list(responses or [])
self.written: list[bytes] = []
self._connected = True
# BaseTransport surface actually used by MicromateProtocol
def connect(self) -> None:
self._connected = True
def disconnect(self) -> None:
self._connected = False
def is_connected(self) -> bool:
return self._connected
def write(self, data: bytes) -> None:
self.written.append(data)
def read(self, n: int) -> bytes:
return self.queue.pop(0) if self.queue else b""
def frame(rsp_sub: int, data: bytes, *, flags: int = FLAGS_CB, page: int = 0) -> bytes:
"""Build a response frame the way a unit would."""
payload = bytes([0x00, flags, rsp_sub, (page >> 8) & 0xFF, page & 0xFF]) + data
return bytes([STX]) + stuff(payload + bytes([checksum(payload)])) + bytes([ETX])
def ack(rsp_sub: int) -> bytes:
return frame(rsp_sub, bytes(ACK_DATA_LEN))
def proto(responses: list[bytes], **kw) -> tuple[MicromateProtocol, ScriptedTransport]:
t = ScriptedTransport(responses)
return MicromateProtocol(t, recv_timeout=0.5, **kw), t
# ── Captured THOR request frames ──────────────────────────────────────────────
REQ = {
"poll": bytes.fromhex("41021010005b000030000000000000000000009b03"),
"serial": bytes.fromhex("41021010001500000a000000000000000000002f03"),
"state": bytes.fromhex("41021010004900ffff000000000000000000005703"),
"compliance": bytes.fromhex("41021010001a00ffff000000000000000000002803"),
"arm": bytes.fromhex("41021010009300ffff00000000000000000000a103"),
"setup_first": bytes.fromhex("41021010003f00ffff000000000000000000004d03"),
"setup_next": bytes.fromhex("41021010004000ffff000000000000000000004e03"),
}
# The complete 0x5A sequence for event 055d4a81 (4,076 bytes → 4 chunks), as
# THOR sent it. Chunk 1's params hold a literal 0x04 and chunk 3's offset is
# the exact remainder.
REQ_CHUNKS_4A81 = [
bytes.fromhex("41021010005a00100400055d4a810000000000009b03"),
bytes.fromhex("41021010005a0010040000001004000000000000007203"),
bytes.fromhex("41021010005a00100400000008000000000000007603"),
bytes.fromhex("41021010005a001003ec00000c000000000000006503"),
]
SIZE_4A81 = 4076
# ── Params builders ───────────────────────────────────────────────────────────
def test_event_token_sits_at_params_7():
"""⚠ THOR sends 0xFE here; the protocol reference documents all-zero params.
That reference entry describes our own browse probing, not THOR's.
"""
assert token_params() == bytes.fromhex("00000000000000fe0000")
def test_event_record_takes_the_full_key_at_params_4():
assert key_params(bytes.fromhex("055d4a81")) == bytes.fromhex("00000000055d4a810000")
def test_monitor_log_takes_only_the_low_half_of_the_key():
"""⚠ Inferred from one key value -- see key_lo_params' docstring."""
assert key_lo_params(bytes.fromhex("055d4a81")) == bytes.fromhex("0000000000004a810000")
def test_chunk_params_switch_from_key_to_byte_offset():
key = bytes.fromhex("055d4a81")
assert chunk_params(key, 0) == bytes.fromhex("055d4a81000000000000")
assert chunk_params(key, 1024) == bytes.fromhex("00000400000000000000")
assert chunk_params(key, 13312) == bytes.fromhex("00003400000000000000")
@pytest.mark.parametrize("bad", [b"", b"\x01\x02\x03", b"\x01\x02\x03\x04\x05"])
def test_params_builders_reject_a_wrong_length_key(bad):
for fn in (key_params, key_lo_params):
with pytest.raises(ValueError):
fn(bad)
# ── Each read emits the frame THOR emits ──────────────────────────────────────
@pytest.mark.parametrize(
"name, method, rsp_sub, data_len",
[
("poll", "poll", 0xA4, 59),
("serial", "read_serial", 0xEA, 21),
("state", "read_state", 0xB6, 16),
("compliance", "read_compliance_config", 0xE5, 2103),
("setup_first", "read_first_setup", 0xC0, 266),
("setup_next", "read_next_setup", 0xBF, 266),
],
)
def test_reads_match_thors_wire_bytes(name, method, rsp_sub, data_len):
p, t = proto([frame(rsp_sub, bytes(data_len))])
getattr(p, method)()
assert t.written == [REQ[name]]
def test_arm_event_matches_thors_wire_bytes():
p, t = proto([ack(0x6C)])
p.arm_event()
assert t.written == [REQ["arm"]]
def test_poll_is_the_only_read_with_a_non_ffff_offset_besides_serial():
"""Reads are single-step at 0xFFFF; POLL and SERIAL are the exceptions."""
assert set(P._OFFSETS) == {P.SUB_POLL, P.SUB_SERIAL}
assert P._OFFSETS[P.SUB_POLL] == 0x0030
assert P._OFFSETS[P.SUB_SERIAL] == 0x000A
# ── The chunk walk ────────────────────────────────────────────────────────────
def test_download_reproduces_thors_chunk_sequence_byte_for_byte():
"""The whole point of step 2. Four chunks, 4,076 bytes, THOR's exact frames."""
payload = bytes(range(256)) * 16 # 4096 B, we use the first 4076
payload = payload[:SIZE_4A81]
responses = []
for i in range(4):
want = min(P.CHUNK_SIZE, SIZE_4A81 - i * P.CHUNK_SIZE)
body = payload[i * P.CHUNK_SIZE: i * P.CHUNK_SIZE + want]
responses.append(frame(0xA5, bytes(11) + body, page=want // 256))
p, t = proto(responses)
got = p.read_event_file(bytes.fromhex("055d4a81"), SIZE_4A81)
assert t.written == REQ_CHUNKS_4A81
assert got == payload
assert len(got) == SIZE_4A81
@pytest.mark.parametrize(
"size, n_chunks, last_offset",
[
(4076, 4, 0x03EC), (11032, 11, 0x0318), (11502, 12, 0x00EE),
(13424, 14, 0x0070), (8746, 9, 0x022A), (6092, 6, 0x03CC),
(1024, 1, 0x0400), (1, 1, 0x0001), (1025, 2, 0x0001),
],
)
def test_chunk_count_and_final_offset(size, n_chunks, last_offset):
"""The first six rows are the six bench events, with THOR's real offsets."""
responses = []
for i in range(n_chunks):
want = min(P.CHUNK_SIZE, size - i * P.CHUNK_SIZE)
responses.append(frame(0xA5, bytes(11) + bytes(want)))
p, t = proto(responses)
p.read_event_file(bytes.fromhex("055d4a81"), size)
assert len(t.written) == n_chunks
# offset is payload[4:5] of the request; recover it from the built frame
final = t.written[-1]
assert final[7:9] in (
bytes([last_offset >> 8, last_offset & 0xFF]),
# a 0x02/0x03/0x04/0x10 high byte arrives escaped, shifting the pair
bytes([0x10, last_offset >> 8]),
)
def test_a_short_chunk_raises_rather_than_truncating():
"""A silently short event is the failure mode this codebase keeps hitting."""
p, _ = proto([frame(0xA5, bytes(11) + bytes(900))]) # asked for 1024
with pytest.raises(ShortRead, match="asked for 1024 B, got 900"):
p.read_event_file(bytes.fromhex("055d4a81"), 1024)
def test_a_chunk_too_short_to_hold_its_header_raises():
p, _ = proto([frame(0xA5, bytes(4))])
with pytest.raises(ShortRead, match="too short to hold a chunk header"):
p.read_event_file(bytes.fromhex("055d4a81"), 1024)
def test_download_rejects_a_nonsense_size():
p, _ = proto([])
with pytest.raises(ValueError):
p.read_event_file(bytes.fromhex("055d4a81"), 0)
# ── The monitor-log walk ──────────────────────────────────────────────────────
def test_monitor_log_walk_ends_on_a_short_response():
"""⚠ Not a keyed read -- the same request repeated, device-side cursor.
Eight records then an 11-byte ack, which is what the capture shows.
"""
key = bytes.fromhex("055d4a81")
responses = [frame(0xF5, bytes(297)) for _ in range(8)] + [ack(0xF5)]
p, t = proto(responses)
records = []
while (rec := p.read_monitor_log_next(key)) is not None:
records.append(rec)
assert len(records) == 8
assert len(t.written) == 9
assert len(set(t.written)) == 1, "every request in the walk is identical"
# ── Error handling ────────────────────────────────────────────────────────────
def test_a_bad_checksum_raises_by_default():
"""⚠ Deliberately stricter than the Series III sibling.
That one logs and continues because its parser cannot always tell an
inner-frame delimiter from a checksum byte. The Micromate rule is exact on
251/251 captured frames, so a mismatch here means something real.
"""
bad = bytearray(frame(0xA4, bytes(59)))
bad[-2] ^= 0xFF
p, _ = proto([bytes(bad)])
with pytest.raises(ChecksumError, match="checksum mismatch"):
p.poll()
def test_a_bad_checksum_can_be_downgraded_for_field_diagnosis():
bad = bytearray(frame(0xA4, bytes(59)))
bad[-2] ^= 0xFF
p, _ = proto([bytes(bad)], strict_checksums=False)
assert p.poll().sub == 0xA4
def test_the_wrong_response_sub_raises():
p, _ = proto([frame(0xE0, bytes(19))]) # 0xE0 answers 0x1F, not 0x1E
with pytest.raises(UnexpectedResponse, match="expected SUB 0xE1"):
p.read_event_first()
def test_a_timeout_reports_how_many_bytes_arrived():
"""Separates "nothing came back" from "bytes arrived but never framed".
Those have completely different causes -- and on a Micromate the second one
is the signature of a modem forwarding a session it should not be.
"""
p, _ = proto([])
with pytest.raises(P.TimeoutError, match="0 bytes were received"):
p.poll()
unframed = b"\x02\x00\xc5\xa4garbage-no-terminator"
p2, _ = proto([unframed])
with pytest.raises(P.TimeoutError, match=f"{len(unframed)} bytes were received"):
p2.poll()
def test_a_leftover_frame_is_discarded_rather_than_answered_with():
"""If a read returns two frames, the extra must not answer the NEXT request.
Every exchange resets the parser before sending, so anything already
buffered is treated as stale. Delivering it would be the worse failure:
`expected_sub` happens to catch a mismatched SUB, but a same-SUB leftover
would sail through and return data for the wrong key.
"""
# Both frames arrive while answering arm_event(); the 0xE1 is left over.
p, _ = proto([ack(0x6C) + frame(0xE1, b"\xaa" * 19)])
p.arm_event()
# The next request gets no bytes of its own, so it must time out rather
# than hand back the stale 0xE1.
with pytest.raises(P.TimeoutError):
p.read_event_first()
# ── Against the real capture, when it happens to be present ───────────────────
_CAPTURES = (
Path(__file__).resolve().parents[1]
/ "bridges" / "captures" / "9-24-26 - micromate2"
)
_DOWNLOAD = "raw_bw_20260925_011403_Download_events_then_delete_1_event.bin"
@pytest.mark.skipif(
not (_CAPTURES / _DOWNLOAD).is_file(),
reason="capture is gitignored; present only on a dev box",
)
def test_every_captured_download_frame_is_one_we_would_have_sent():
"""Replay the real session: for each event, assert our chunk walk emits
exactly the frames THOR emitted -- all 50-odd of them, six events."""
from micromate.framing import ACK, DLE
def destuffed_frames(blob: bytes, is_req: bool):
i, n = 0, len(blob)
while i < n:
if is_req:
if not (blob[i] == ACK and i + 1 < n and blob[i + 1] == STX):
i += 1
continue
j = i + 2
else:
if blob[i] != STX:
i += 1
continue
j = i + 1
out = bytearray()
while j < n:
if blob[j] == DLE and j + 1 < n:
out.append(blob[j + 1])
j += 2
continue
if blob[j] == ETX:
break
out.append(blob[j])
j += 1
if len(out) >= 6:
yield blob[i:j + 1], bytes(out[:-1])
i = j + 1
bw = list(destuffed_frames((_CAPTURES / _DOWNLOAD).read_bytes(), True))
s3 = list(
destuffed_frames(
(_CAPTURES / _DOWNLOAD.replace("raw_bw", "raw_s3")).read_bytes(), False
)
)
# Group THOR's 0x5A frames per event, taking each event's key+size from the
# 1E/1F that preceded them.
events, cur = [], None
for (wire, req), (_, rsp) in zip(bw, s3):
sub, data = req[2], rsp[5:]
if sub in (0x1E, 0x1F) and len(data) >= 19:
cur = {"key": data[11:15], "size": int.from_bytes(data[15:19], "big"),
"reqs": [], "rsps": []}
if cur["size"]:
events.append(cur)
elif sub == 0x5A and cur is not None:
cur["reqs"].append(wire)
cur["rsps"].append(rsp)
# The capture walks the chain twice (it deletes an event on the second
# pass), so some 1E/1F hits carry a size but no download behind them.
events = [e for e in events if e["reqs"]]
assert len(events) == 6, f"expected 6 downloaded events, found {len(events)}"
total = 0
for e in events:
p, t = proto([bytes([STX]) + stuff(r + bytes([checksum(r)])) + bytes([ETX])
for r in e["rsps"]])
got = p.read_event_file(e["key"], e["size"])
assert t.written == e["reqs"], (
f"event {e['key'].hex()}: our {len(t.written)} frames differ from "
f"THOR's {len(e['reqs'])}"
)
assert len(got) == e["size"]
total += len(e["reqs"])
assert total == 56, f"expected 56 download frames across the 6 events, saw {total}"