diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 8de4a36..82089db 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -428,6 +428,20 @@ memory 15,000,000 total and free (no events stored). Blastware, `0x03` Thor — where Series III has a constant `0x10`. 5. **Modem serial rate is 115200**, not 38400 (per TMI provisioning practice; not independently verified here). +6. **Reads are single-step** — Thor asks for `offset = 0xFFFF` and gets the + whole block; Series III always probes first. (2026-09-24) +7. **Setup writes are preceded by `SUB 0xDA`**, which names the target `.MMB` + file. Series III has no equivalent — it has one config, not named files. +8. **The compliance block is written in ONE `0x71` frame**, not Series III's + three chunks. +9. **`0x69` / `0x74` (waveform data write) are absent** from a setup push. +10. **Six channel blocks** (`Tran`/`Vert`/`Long`/`Mic`/`LMic`/`SMic`) on a + 48-byte stride, against Series III's four. +11. **The notes block is four fixed-width `[label:22][value:42]` entries** on a + 64-byte stride, with different labels — not Series III's `label: value` + scan targets. +12. **The geophone scale factor is `3.10308`**, exactly half Series III's + `6.206053`, with an ADC of 10,000 counts per volt. --- @@ -649,8 +663,192 @@ Note `LMic` / `SMic` — linear and sound-level microphone variants that Series III does not have. This is the **read half of setup management**, and it means a setup can be -round-tripped: read the active config, modify, write it back. The write half -is not yet attempted. +round-tripped: read the active config, modify, write it back. ✅ **The write +half was observed on 2026-09-24** — see *The write path* below, which confirms +the round-trip: the written block is the read block, 91% byte-identical. + +## The write path — observed end to end (2026-09-24) + +⚠ **We have still never sent a write command to a unit.** Thor did every write +below; seismo_lab's TCP bridge sat between Thor and the unit and recorded both +directions. Capture: `bridges/captures/be12599-diag/9-24-26 - micromate2/` +(`raw_bw_*` = Thor, `raw_s3_*` = unit), UM12947 on the USB **PC** port, relayed +to TCP by `socat` on mint-mac. Operation: push a setup named `TEST1.mmb`. + +**All 12 device responses checksum-validate and every write is acked.** + +### The sequence + +``` +Thor: 5B │ 41 │ 08 │ 2E │ 1A │ DA │ 68 → 73 │ 82 → 83 │ 71 → 72 +unit: A4 │ BE │ F7 │ D1 │ E5 │ 25 │ 97 8C │ 7D 7C │ 8E 8D + └──────── reads ────────┘ └──────────── writes ────────────┘ +``` + +Series III, for comparison: `68→73 │ 71×3→72 │ 82→83 │ 69→74→72`. + +Same write SUBs, and **every response SUB matches the Series III table +exactly** (`68`→`97`, `73`→`8C`, `82`→`7D`, `83`→`7C`, `71`→`8E`, `72`→`8D`). +Three differences: + +- **`SUB 0xDA` is new** and comes first — see below. +- **`0x71` is a single write, not three chunks.** Series III splits the + compliance block into 1027 + 1055 + remainder; the Micromate takes all + 2,090 bytes in one frame. +- **`0x69` / `0x74` (waveform data write) do not appear at all.** + +Every write ack is a 16-byte zero-data frame, same shape as Series III's. + +### Three new read commands + +| SUB | rsp | payload | what it carries | +|---|---|---|---| +| `0x41` | `0xBE` | 271 B | **the active setup's file name**, e.g. `TEST1.mmb` | +| `0x2E` | `0xD1` | 44 B | trigger-config block; mirrors what `0x82` writes | +| `0xDA` | `0x25` | 272 B (write) | **declares the target setup file** — see below | + +### `SUB 0xDA` — name the setup file you are about to write + +Data is exactly **256 bytes: the file name, null-padded, nothing else.** + +``` +54 45 53 54 31 2e 6d 6d 62 00 00 … "TEST1.mmb" + 247 × 0x00 +``` + +This is the missing link in *setups are files*: there is no generic +file-transfer command because there does not need to be one. Thor names the +target file, then writes the ordinary config block into it. The unit acks with +`0x25` before any config bytes are sent. + +### Reads are single-step — no probe + +Series III sends every read twice (probe at `offset=0x00` to learn the length, +then a data step). Thor **skips the probe** and asks for +**`offset = 0xFFFF`**, getting the whole block in one response: + +``` +41 02 10 10 00 1a 00 ff ff 00 … SUB 0x1A, offset 0xFFFF +→ 2,103-byte response +``` + +`POLL` is the exception — it uses `offset = 0x0030` (48), its data length. + +This does not contradict the probe response documented above; the probe still +works and still reports its length at `payload[8:10]`. Thor simply does not +need it. + +### Write-frame destuffing — `10 XX` → `XX`, uniformly + +Only the leading `BW_CMD` is doubled (`10 10`); after that **every `10 XX` pair +on the wire destuffs to `XX`**, including `10 03`. + +This was settled by checksum, not by assumption. Four candidate rules were +tested against all four data-carrying write frames; **only this one makes all +four checksums validate**: + +| rule | `0xDA` | `0x68` | `0x82` | `0x71` | +|---|---|---|---|---| +| **`10 XX` → `XX`** | ok | **ok** | ok | **ok** | +| only `10 03` → `03` | ok | BAD | ok | BAD | +| `10 10`→`10`, `10 03`→`03` | ok | BAD | ok | BAD | +| nothing collapses | ok | BAD | ok | BAD | + +`0x71`'s data contains **4 literal `0x03` bytes**, escaped as `10 03` on the +wire. A writer that does not escape `0x03` will emit a frame the device +terminates early — this is the same defensive ETX escaping Blastware does, and +it is mandatory, not optional. + +Checksum is plain SUM8 of the destuffed payload; the DLE-aware form gives the +same answer once destuffing is correct, so it does not discriminate. + +### The `offset` field is a per-command constant + +| SUB | data bytes | `offset` | note | +|---|---|---|---| +| `0xDA` | 256 | `0x0100` = 256 | = the name-field size | +| `0x68` | 88 | `0x005A` = 90 | **identical to Series III's documented value** | +| `0x82` | 28 | `0x001C` = 28 | **identical to Series III's documented value** | +| `0x71` | 2090 | `0x082C` = 2092 | = the length `SUB 0x1A` reports | + +⚠ **There is no single length formula** — two are `len`, two are `len + 2`, and +Series III's `data[1] + 2` rule does not reproduce either. Treat these as +observed constants the device accepted. Pinning the actual rule needs a second +capture whose payloads differ in size. (This document has already been wrong +once by inferring a length field; do not infer this one.) + +### The write body **is** the read body + +The `0x71` write payload and the `0xE5` read response align at a fixed +**11-byte shift** with **1902/2090 bytes equal (91.0%)** — the remaining 9% is +exactly what was edited. + +**So a setup is read-modify-write**, the same shape as Series III, and a write +client does not need to synthesise a config block from scratch. + +### Field map, from the read/write diff + +Twelve differing regions, all accounted for. Offsets are into the `0x71` +**data** section (= the read response's data + 11). + +| offset | size | field | this capture | +|---|---|---|---| +| `0x0000` | 3 | block length echo | `08 2a` (=2090) → zeroed on write | +| `0x0009` | 1 | unidentified | `3c` → `02` | +| `0x0013` | 1 | unidentified | `00` → `40` | +| `0x002A` | 44 | **setup file name** | → `TEST1.mmb` | +| `0x0090` | 42 | note 1 value — `Location` | → `Test Location 1 - 1234 electric boogaloo` | +| `0x00D0` | 42 | note 2 value — `Client` | → `TMI` | +| `0x0110` | 42 | note 3 value — `Company` | → `ServersDownLabs` | +| `0x0150` | 42 | note 4 value — `General Notes` | → `Hopefully this works!` | +| `0x0428` | 16 | sensor location | → `Test Config Send` | +| `0x06CA` | 4 | **Tran trigger level** float32 BE | `0.3` → `0.5` in/s | +| `0x06FA` | 4 | **Vert trigger level** | `0.3` → `0.5` in/s | +| `0x072A` | 4 | **Long trigger level** | `0.3` → `0.5` in/s | + +**Notes block: four entries on a 64-byte stride**, each `[label: 22][value: 42]`, +labels at `0x008A + 64n`. The labels are `Location`, `Client`, `Company`, +`General Notes` — **not** Series III's `Project:` / `Client:` / `User Name:` / +`Seis Loc:`, and they are fixed-width fields, not `label: value` pairs. + +**Channel blocks: six, 48 bytes each**, from `0x06BC` — `Tran`, `Vert`, `Long`, +`Mic`, `LMic`, `SMic`. Trigger level sits at **label + 30**. + +### The geophone scale factor is in the config block — and it confirms our LSB + +Each geo channel block carries a float32 BE at **label + 24**: + +``` +40 46 98 dd = 3.10308003… +``` + +Which closes a loop from the file-decode work: + +``` +3.10308003… / 10000 = 0.000310308 = _GEO_LSB_IPS, to 8 figures +10.0 / 3.10308003… × 10000 = 32226.046 = the 32226.05 full scale +``` + +`_GEO_LSB_IPS` was derived statistically — intersecting 991,415 rounding +constraints from Thor's own CSV exports (see `idf_protocol_reference.md`). +**The unit reports the constant directly**, and it agrees. That upgrades the +value from a fit to a reading, and explains the odd full-scale count: the +Micromate's ADC is **10,000 counts per volt**, and 3.10308 in/s per volt is +**exactly half** Series III's documented `6.206053` (ratio 1.99997). + +Do **not** retune `_GEO_LSB_IPS` — this is corroboration, not a correction. + +### Still unknown on the write path + +- **What `0x68` and `0x82` actually contain.** Both were written with + near-zero payloads here and nothing in them changed, so no field is located. + Series III maps backlight/power-save/LCD-cycle into `0x68`; unverified here. +- **The three header bytes** at data `0x0000`/`0x0009`/`0x0013`. +- **Whether a *new* setup file can be created**, or only an existing one + overwritten. `0xDA` named a file that did not previously exist and the unit + accepted it — but we did not confirm on the unit's screen that `TEST1` now + exists as a selectable setup. Worth checking on the device. +- **Scheduler / call-home writes.** `callhome.MMB` is a file too, so it may go + through the same `0xDA` + block-write shape with a different target name. ## Static analysis of the firmware (2026-09-23, solo session) @@ -789,10 +987,17 @@ its maximum should equal the loudest event in it. ## ⚠ Untested and unsafe-until-agreed -Nothing below has been sent to a unit, and nothing should be without an -explicit decision: +**Nothing below has been sent to a unit by us, and nothing should be without an +explicit decision.** -- **Writes** (`0x68`–`0x83`), **call-home write** (`0x7E`/`0x7F`) +Note the distinction introduced on 2026-09-24: the setup-write sequence +(`0xDA`, `0x68`/`0x73`, `0x82`/`0x83`, `0x71`/`0x72`) has now been **observed**, +because Thor performed it while we recorded. Observed is not the same as +exercised — **we have still never originated a write frame.** The wire format +is known; our encoder is unwritten and unproven. + +- **Writes** (`0x68`–`0x83`) — format now known, never sent by us +- **Call-home write** (`0x7E` / `0x7F`) — not observed at all - **Erase** (`0xA3` / `0xA2`) - **Start / stop monitoring** (`0x96` / `0x97`) - `0x1F` (advance event pointer) — non-destructive on Series III but it does @@ -800,7 +1005,8 @@ explicit decision: Also unknown: -- Whether `0x10` bytes inside request params need stuffing +- Whether `0x10` bytes inside **request params** need stuffing. (Write-frame + **data** stuffing *is* now settled — `10 XX` → `XX`; see *The write path*.) - Everything about the **call-home session** — the device-initiated direction has not been observed at all. Specifically: how a unit announces itself, and **how it learns an event was accepted so it stops re-sending it.** @@ -812,14 +1018,25 @@ Also unknown: ## Session provenance Unit **UM12947**, firmware **`11.0CB`** (Blastware line), on the bench via -USB, **zero events stored** (memory free == total). Read commands only. -Every response in this document was checksum-validated. +USB. Every response in this document was checksum-validated. + +Three sittings, all on the same unit: + +| date | state | what was exercised | +|---|---|---| +| 2026-09-23 | zero events stored | read commands, direct from Python | +| 2026-09-23 | 5 events (4 waveform, 1 histogram) | the event chain + `0x5A` | +| 2026-09-24 | 5 events | **Thor pushing a setup**, via a recording relay | + +The 2026-09-24 sitting used a different topology worth noting, because it is +reusable: `socat` on mint-mac shares `/dev/ttyACM0` on TCP, seismo_lab's TCP +bridge relays Thor to it and records both directions. Thor is configured with +the unit at `127.0.0.1:` exactly as if it were a field modem. No +modem, no SIM, and **nothing on the production Thor box is touched.** ⚠ Firmware is the single biggest caveat on this document. Every finding here is from one unit on the Blastware build. A `11.0BD` unit has not been touched. -An empty unit is a real limitation: `0x08`, `0x1E`, `0x0A` and `0x06` all have -event-dependent payloads that could not be exercised. Recording a couple of -events on the bench unit would unlock the entire event-walk half of the -protocol. +✅ The original "empty unit" limitation is gone — `0x08`, `0x1E`, `0x0A` and +`0x06` were all exercised against 5 stored events on 2026-09-23. diff --git a/scratch/mm_frame_parse.py b/scratch/mm_frame_parse.py new file mode 100644 index 0000000..c0e6ebc --- /dev/null +++ b/scratch/mm_frame_parse.py @@ -0,0 +1,202 @@ +#!/usr/bin/env python3 +""" +mm_frame_parse.py — parse Micromate (Series IV) frames out of a seismo_lab +raw capture pair. + +Why this exists +--------------- +`minimateplus.framing.S3FrameParser` cannot see Micromate traffic. It locates +frames by scanning for `DLE STX`, and a Micromate response has **no leading +DLE** — it starts at a bare `STX`. It also expects `payload[1] == 0x10`, where +the Micromate sends `0xC5` (Blastware firmware) or `0x03` (Thor firmware). + +The practical consequence, seen on the 9-24-26 setup-push capture: the +Blastware-side requests parse fine (Thor emits Series III request frames), but +**every device response is silently dropped or mis-framed** — so a capture that +actually contains 12 acked writes looks like 12 unanswered requests. + +Destuffing +---------- +One rule covers both directions: after the leading doubled `BW_CMD`, every +`10 XX` pair on the wire destuffs to `XX`. That includes `10 03` — Thor +escapes literal `0x03` bytes in write data so they are not mistaken for ETX, +exactly as Blastware does. + +That rule was chosen by evidence, not assumption: of the four candidates tried +against the 9-24-26 capture's four data-carrying write frames, it is the only +one under which all four checksums validate. See +`docs/micromate_protocol_reference.md` → *The write path*. + +Usage +----- + python scratch/mm_frame_parse.py + python scratch/mm_frame_parse.py + python scratch/mm_frame_parse.py --dump 0x71 +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +DLE, STX, ETX, ACK = 0x10, 0x02, 0x03, 0x41 + +# Request SUB -> short name. Series III names where they carry over; the +# Series IV additions are marked. +SUBNAME = { + 0x01: "DEVICE_INFO", + 0x06: "STORAGE_RANGE", + 0x08: "EVENT_INDEX", + 0x0A: "WAVEFORM_HDR", + 0x0C: "WAVEFORM_REC", + 0x15: "SERIAL", + 0x1A: "COMPLIANCE_CFG", + 0x1C: "MONITOR_STATUS", + 0x1E: "EVENT_HDR", + 0x2C: "CALLHOME_CFG", + 0x2E: "TRIGGER_CFG_READ", # Series IV + 0x3E: "OPERATOR", + 0x41: "SETUP_NAME_READ", # Series IV + 0x5A: "BULK_DOWNLOAD", + 0x5B: "POLL", + 0x68: "EVENT_INDEX_WRITE", + 0x69: "WAVEFORM_WRITE", + 0x71: "COMPLIANCE_WRITE", + 0x72: "CONFIRM_A", + 0x73: "CONFIRM_B", + 0x74: "CONFIRM_C", + 0x82: "TRIGGER_WRITE", + 0x83: "TRIGGER_CONFIRM", + 0xDA: "SETUP_FILE_DECL", # Series IV — names the target .MMB + 0xFE: "FULL_CFG", +} + + +def destuff(blob: bytes, start: int, *, is_request: bool) -> tuple[bytes, int, int]: + """Destuff one frame starting at `start`. + + Returns (payload, checksum, index_of_terminating_ETX). `payload` excludes + the trailing checksum byte. A request frame opens `ACK STX 10 10`; a + response opens with a bare `STX`. + """ + i = start + (2 if is_request else 1) + out = bytearray() + if is_request: + # The doubled BW_CMD is the one guaranteed stuffed byte. + if blob[i : i + 2] != bytes([DLE, DLE]): + raise ValueError(f"@0x{start:04x}: request does not open with 10 10") + out.append(DLE) + i += 2 + while i < len(blob): + b = blob[i] + if b == DLE and i + 1 < len(blob): + out.append(blob[i + 1]) + i += 2 + continue + if b == ETX: + break + out.append(b) + i += 1 + if len(out) < 2: + raise ValueError(f"@0x{start:04x}: frame too short") + return bytes(out[:-1]), out[-1], i + + +def frames(blob: bytes, *, is_request: bool): + """Yield (offset, payload, chk, checksum_kind).""" + i, n = 0, len(blob) + while i < n: + if is_request: + if not (blob[i] == ACK and i + 1 < n and blob[i + 1] == STX): + i += 1 + continue + elif blob[i] != STX: + i += 1 + continue + try: + payload, chk, end = destuff(blob, i, is_request=is_request) + except ValueError: + i += 1 + continue + sum8 = sum(payload) & 0xFF + dle_aware = (sum(b for b in payload if b != DLE) & 0xFF) + if sum8 == chk: + kind = "SUM8" + elif dle_aware == chk: + kind = "DLE-aware" + else: + kind = "BAD" + yield i, payload, chk, kind + i = end + 1 + + +def describe(payload: bytes, is_request: bool) -> str: + if len(payload) < 3: + return "??" + sub = payload[2] + if is_request: + return SUBNAME.get(sub, f"SUB_{sub:02X}") + req = 0xFF - sub + return "rsp<-" + SUBNAME.get(req, f"SUB_{req:02X}") + + +def report(path: Path, *, is_request: bool, dump_sub: int | None) -> None: + blob = path.read_bytes() + side = "Thor" if is_request else "unit" + print(f"== {side:4} {path.name} ({len(blob)} bytes)") + n_bad = 0 + for idx, (off, p, chk, kind) in enumerate(frames(blob, is_request=is_request)): + if kind == "BAD": + n_bad += 1 + sub = p[2] if len(p) > 2 else -1 + flags = p[1] if len(p) > 1 else -1 + # Requests carry offset at payload[4:6]; responses page at [3:5]. + word = int.from_bytes(p[4:6] if is_request else p[3:5], "big") + data = len(p) - 16 if is_request else max(len(p) - 5, 0) + print( + f" [{idx:2}] @0x{off:04x} payload={len(p):5} data={data:5} " + f"flags=0x{flags:02x} SUB=0x{sub:02x} {describe(p, is_request):18} " + f"{'offset' if is_request else 'page'}=0x{word:04x} chk={kind}" + ) + if dump_sub is not None and sub == dump_sub: + body = p[16:] if is_request else p[5:] + print(f" ---- data ({len(body)} bytes) ----") + for o in range(0, len(body), 16): + chunk = body[o : o + 16] + txt = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk) + print(f" {o:06x} {chunk.hex(' '):<47} |{txt}|") + print(f" -- {idx + 1} frames, {n_bad} bad checksum\n") + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("paths", nargs="+", + help="a capture directory, or raw_bw.bin and raw_s3.bin") + ap.add_argument("--dump", default=None, + help="hex-dump the data section of this SUB (e.g. 0x71)") + args = ap.parse_args() + + dump_sub = int(args.dump, 0) if args.dump else None + + if len(args.paths) == 1 and Path(args.paths[0]).is_dir(): + d = Path(args.paths[0]) + bw = sorted(d.glob("raw_bw_*.bin")) + s3 = sorted(d.glob("raw_s3_*.bin")) + if not bw or not s3: + print(f"{d}: need one raw_bw_*.bin and one raw_s3_*.bin", file=sys.stderr) + return 2 + pairs = [(bw[0], True), (s3[0], False)] + elif len(args.paths) == 2: + pairs = [(Path(args.paths[0]), True), (Path(args.paths[1]), False)] + else: + ap.error("pass a capture directory, or exactly two .bin files") + + for path, is_request in pairs: + report(path, is_request=is_request, dump_sub=dump_sub) + return 0 + + +if __name__ == "__main__": + sys.exit(main())