From 095834183e170ae7bf70faf6716058cabce7a9a7 Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 15:59:17 -0400 Subject: [PATCH 01/43] docs(series4): open the Micromate live-protocol reference First bench session against a Micromate over USB. The headline: the unit answers Series III command frames unmodified. An untouched Series III POLL (SUB 0x5B), built by build_bw_frame with no changes, completed a full two-step probe/data cycle. Ten Series III read commands were then tried and all ten answered, every one obeying the response_SUB = 0xFF - request_SUB rule. Confirmed this session: - Transport is a plain USB CDC-ACM port (2504:0300, "MICROMATE COM PORT"). No vendor driver, no Thor, no Windows box needed. Baud is ignored over USB (identical responses at 38400 and 115200). - Device never speaks first -- 20 s idle listen produced nothing. - Responses are Series III framing MINUS the leading DLE: bare [STX][payload][chk][ETX]. This alone means Blastware can never find a frame boundary in Micromate traffic, since its parser scans for DLE+STX. - Response flags byte is 0xC5, not Series III's 0x10. - Checksum is the DLE-aware variant (SUM8 excluding 0x10 bytes) -- the same one Series III uses for 5A and write frames, not the plain SUM8 of its ordinary reads. Disambiguated by the POLL data frame, which contains a 0x10. - The probe response carries the data length at payload[9]. Four of four known Series III lengths match; call-home config differs (0x7E vs 0x7C). - Series III monitor-status field offsets apply unchanged: battery 3.81 V (Thor's own reports say 3.8), memory 15,000,000 total and free, date 23 Sep 2026. - SUB 0x2C carries the string "RADIO RING" -- the same string seen in the RV50 ALEOS debug during the BE12599 incident. That block holds the modem dial/answer strings and is the most relevant command to the call-home goal. Read commands only. Nothing that writes, erases, or changes monitoring state has been sent to a unit; those are listed as unsafe-until-agreed. Caveat recorded in the doc: one unit, over USB, with zero events stored, so the event-walk commands (0x08, 0x1E, 0x0A, 0x06) could only be probed, not exercised. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 231 +++++++++++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 docs/micromate_protocol_reference.md diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md new file mode 100644 index 0000000..ee0eb0f --- /dev/null +++ b/docs/micromate_protocol_reference.md @@ -0,0 +1,231 @@ +# Micromate Protocol Reference — Thor / Micromate Series IV, live wire protocol + +Sibling to [instantel_protocol_reference.md](instantel_protocol_reference.md) +(Series III, "the Rosetta Stone") and +[idf_protocol_reference.md](idf_protocol_reference.md) (Series IV *file* +format). This document covers the Series IV **live device protocol** — what +the unit says over the wire, as opposed to what it writes into a `.IDFW`. + +**Status (2026-09-23): opening session.** Everything below was established in +a single bench session against one unit. Treat it as a strong start, not a +settled spec — in particular, everything here comes from **one unit, over USB, +with no events stored**. + +--- + +## The headline + +**The Micromate answers Series III command frames.** + +An unmodified Series III `POLL` (`SUB 0x5B`), built by +`minimateplus.framing.build_bw_frame` with no changes at all, produced a +complete two-step probe/data cycle from a Micromate. Ten Series III read +commands were then tried and **all ten answered**, every one obeying the +Series III response-SUB rule. + +This is not a new protocol. It is close enough to Series III that large parts +of `minimateplus/protocol.py` may apply more or less directly, and the work +ahead is mapping *divergence* rather than mapping from scratch. + +--- + +## How this was obtained + +No Thor, no modem, no Windows machine. The Micromate exposes its protocol on +a **USB CDC-ACM virtual serial port**: + +``` +ID 2504:0300 Instantel Inc. MICROMATE COM PORT +driver: cdc_acm ATTRS{serial}=="V1.00" +→ /dev/ttyACM0 +``` + +Plain CDC ACM, so no vendor driver and no proprietary USB layer — any host +that can open a serial port can talk to the unit. + +**Baud is irrelevant over USB.** Identical byte-for-byte responses at 38400 +and 115200; CDC-ACM ignores the line rate. TMI provisions Micromate *modem* +links at **115200** (Series III uses 38400) — that matters for the cellular +path, not for USB. + +**The device never speaks first.** 20 s of passive listening on an idle open +port produced zero bytes. It is strictly request/response. + +--- + +## Physical + framing layer + +### Requests — Series III format, unmodified + +Every frame in this session was produced by `build_bw_frame(sub, offset)` with +no Series IV changes, and the device accepted all of them: + +``` +[ACK 0x41] [STX 0x02] [10 10] [flags 00] [SUB] [00] [00] [offset] [params×10] [chk] [ETX 0x03] +``` + +⚠ Only the doubled `BW_CMD` (`10 10`) form has been exercised. Whether other +literal `0x10` bytes inside params require stuffing is **untested** — none of +the probes sent carried one. + +### Responses — Series III *minus the DLE prefix* + +``` +Series III: [DLE 0x10] [STX 0x02] … [chk] [ETX 0x03] +Micromate: [STX 0x02] … [chk] [ETX 0x03] ← no leading DLE +``` + +This single byte matters operationally: Blastware's parser locates frames by +scanning for `DLE+STX`, so it will **never find a frame boundary** in Micromate +traffic no matter what else is correct. That is a structural reason a +Micromate cannot call into a Blastware ACH server, independent of any baud +mismatch. + +### Response payload header + +``` +[0] CMD 0x00 same as Series III +[1] flags 0xC5 ← Series III uses 0x10. Constant across all 10 SUBs. +[2] SUB 0xFF − request_SUB +[3] PAGE_HI +[4] PAGE_LO +[5+] data +``` + +### Checksum — the DLE-aware variant + +```python +chk = sum(b for b in payload if b != 0x10) & 0xFF +``` + +Confirmed on every frame captured. The `POLL` probe response contains no +`0x10` and so cannot distinguish plain SUM8 from the DLE-aware form; the +`POLL` **data** response contains a `0x10` at payload offset 42, and only the +DLE-aware rule matches there. This is the same checksum Series III uses for +its `5A` bulk-stream and write frames — not the plain SUM8 of ordinary +Series III reads. + +### The probe response carries the data length + +Series III hardcodes `DATA_LENGTHS` per SUB. On the Micromate the **probe +response tells you**, at `payload[9]`: + +| SUB | command | `payload[9]` | Series III constant | +|---|---|---|---| +| `0x15` | serial number | `0x0A` | `0x0A` ✓ | +| `0x01` | device info | `0x98` | `0x98` ✓ | +| `0x1C` | monitor status | `0x2C` | `0x2C` ✓ | +| `0x06` | storage range | `0x24` | `0x24` ✓ | +| `0x2C` | call-home config | `0x7E` | `0x7C` ✗ **differs by 2** | +| `0x08` | event index | `0x5A` | — | +| `0x1E` | event header | `0x08` | — | +| `0x1A` | compliance config | `0x2C` | — | +| `0x0A` | waveform header | `0x00` | — (no event context) | +| `0xFE` | full config | `0x00` | — (see note) | + +Four of four known Series III lengths match exactly. **Read the length from +the probe rather than hardcoding it** — it is free, and it already caught the +call-home divergence. + +--- + +## Confirmed commands (read-only) + +All ten below answered with a correct `0xFF − SUB` response. Nothing that +writes, erases, or changes monitoring state has been sent to a unit. + +| SUB | RSP | Command | Data proven | +|---|---|---|---| +| `0x5B` | `0xA4` | POLL / handshake | yes — ID block | +| `0x15` | `0xEA` | Serial number | yes — `UM12947` | +| `0x01` | `0xFE` | Device info | yes — 152 B | +| `0x2C` | `0xD3` | Call-home config | yes — 126 B | +| `0x1C` | `0xE3` | Monitor status | yes — 44 B | +| `0x06` | `0xF9` | Event storage range | yes — 36 B | +| `0x08` | `0xF7` | Event index | probe only | +| `0x1E` | `0xE1` | Event header / first key | probe only | +| `0x0A` | `0xF5` | Waveform header | probe only | +| `0x1A` | `0xE5` | Compliance config | probe only | + +### Decoded so far + +**`SUB 0x15` — serial.** ASCII, null-terminated: `UM12947`. + +**`SUB 0x5B` / `0x01` — identification strings.** +`Instantel\0` and `MM/ISEE/S/IO` (MicroMate / ISEE standard). `0x01` also +carries eight consecutive `3f 80 00 00` float32 values (= `1.0f`) — almost +certainly per-channel calibration/scale factors, by analogy with Series III's +`geo_hardware_constant`. **Unverified.** + +**`SUB 0x1C` — monitor status. Series III field offsets apply unchanged:** + +| field | offset | read | +|---|---|---| +| battery × 100 | `payload[-10:-8]` uint16 BE | `0x017D` → **3.81 V** | +| memory total | `payload[-8:-4]` uint32 BE | 15,000,000 | +| memory free | `payload[-4:]` uint32 BE | 15,000,000 (empty) | +| date | `payload[18:22]` | day 23, month 9, year `0x07EA` = 2026 | + +The battery reading independently corroborates: Thor's own event reports for +these units print `BatteryLevel : 3.8 volts`. + +**`SUB 0x2C` — call-home config.** Contains the ASCII string `RADIO RING`. +Worth flagging: that is the *exact* string seen in the RV50 `ALEOS_SERIAL` +debug during the BE12599 incident — +`'ATQ1^MATE0^MATS0=2^M^MRADIO RING^M'`. So this block holds the modem dial / +answer strings, and it is the most directly relevant command to the +call-home-receiver goal. Field layout **not yet mapped**; Series III's map +(`raw[5]` enabled, `raw[6:46]` dial string) is a starting hypothesis only, and +the length already differs (`0x7E` vs `0x7C`). + +**`SUB 0x06` — storage range.** All zeros on this unit, consistent with +`memory free == memory total`. Series III reads first/last event keys from +the final 8 bytes; untestable until the unit holds events. + +--- + +## Divergences from Series III (running list) + +1. **No `DLE` prefix on responses** — bare `STX`. +2. **Response flags byte is `0xC5`**, not `0x10`. +3. **Call-home config is 126 bytes**, not 124. +4. **Data lengths are discoverable** from the probe response at `payload[9]`. +5. **Modem serial rate is 115200**, not 38400 (per TMI provisioning practice; + not independently verified here). + +--- + +## ⚠ Untested and unsafe-until-agreed + +Nothing below has been sent to a unit, and nothing should be without an +explicit decision: + +- **Writes** (`0x68`–`0x83`), **call-home write** (`0x7E`/`0x7F`) +- **Erase** (`0xA3` / `0xA2`) +- **Start / stop monitoring** (`0x96` / `0x97`) +- `0x1F` (advance event pointer) — non-destructive on Series III but it does + move device state, so it is parked with the rest + +Also unknown: + +- Whether `0x10` bytes inside request params need stuffing +- Whether the bulk waveform stream (`5A` on Series III) exists here, and + whether it is the transport for `.IDFW` bodies we already decode +- Everything about the **call-home session** — the device-initiated direction + has not been observed at all. Specifically: how a unit announces itself, + and **how it learns an event was accepted so it stops re-sending it.** + That last question gates any homebrew receiver and cannot be answered over + USB. + +--- + +## Session provenance + +Unit **UM12947**, on the bench via USB, **zero events stored** (memory free == +total). Read commands only. Every response in this document was checksum- +validated. + +An empty unit is a real limitation: `0x08`, `0x1E`, `0x0A` and `0x06` all have +event-dependent payloads that could not be exercised. Recording a couple of +events on the bench unit would unlock the entire event-walk half of the +protocol. From b9c52442a7f87a9892b565bdddce86643cbad4bd Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 16:04:42 -0400 Subject: [PATCH 02/43] docs(series4): the Series III behaviour is firmware-conditional The bench unit reports 11.0CB -- Instantel's *Blastware* firmware line. It almost certainly answers Series III commands because it is in Blastware mode, not because the Micromate natively speaks Series III. Instantel ships two lines: 11.0CB (Blastware) and 11.0BD (THOR, Vision, Vision II). That also explains the two-ACH-server problem as designed behaviour rather than misconfiguration. Corpus firmware audit: 932 event files from 11.0AK, 83 from 10.90GC. UM12947 itself produced 10.90GC files in production last year and reports 11.0CB now, so units get reflashed and firmware is not stable per-unit over time. Records the resulting strategic fork (standardise on the Blastware line vs reverse-engineer the Thor line) with the four unknowns that decide it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 81 ++++++++++++++++++++++++---- 1 file changed, 71 insertions(+), 10 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index ee0eb0f..c9f653a 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -15,20 +15,77 @@ with no events stored**. ## The headline -**The Micromate answers Series III command frames.** +**A Micromate running the *Blastware* firmware answers Series III command +frames.** An unmodified Series III `POLL` (`SUB 0x5B`), built by `minimateplus.framing.build_bw_frame` with no changes at all, produced a -complete two-step probe/data cycle from a Micromate. Ten Series III read -commands were then tried and **all ten answered**, every one obeying the -Series III response-SUB rule. +complete two-step probe/data cycle. Ten Series III read commands were then +tried and **all ten answered**, every one obeying the Series III response-SUB +rule. -This is not a new protocol. It is close enough to Series III that large parts -of `minimateplus/protocol.py` may apply more or less directly, and the work -ahead is mapping *divergence* rather than mapping from scratch. +⚠ **That qualifier is load-bearing, and it was discovered after the fact.** +Instantel ships the Micromate in two firmware lines: + +| firmware | Instantel's own description | +|---|---| +| `11.0CB` | "Utilize with **Blastware**" | +| `11.0BD` | "Utilize with **THOR, Vision, Vision II**" | + +The bench unit reports **`11.0CB`** — the Blastware build. So the clean +Series III behaviour above is very likely *because the unit is in Blastware +mode*, not because the Micromate natively speaks Series III. **Nothing here +should be assumed to hold on a `11.0BD` unit until tested.** + +This reframes the project. The question is no longer only "what is the +Series IV protocol" but **"which firmware line do we target, and does one of +them let the existing Series III stack drive the whole fleet?"** --- +## Firmware — the variable nobody knew was a variable + +This explains a production problem TMI has lived with: two ACH servers, two +machines, and units that will not cross over. It is not a misconfiguration. +**Instantel ships different firmware for different host software**, and the +wire protocol differs with it. + +Versions observed so far: + +| where | version | notes | +|---|---|---| +| bench unit UM12947, today | **`11.0CB`** | Blastware line — answers Series III | +| corpus, 932 event files | `11.0AK` | produced Thor-collected `.IDFW`/`.IDFH` | +| corpus, 83 event files | `10.90GC` | older; UM12947's own files from Sept 2025 | + +Note UM12947 produced `10.90GC` files in production last year and reports +`11.0CB` on the bench now — so **it has been reflashed at some point**, and +firmware is not stable per-unit over time. Any fleet-wide claim needs a +per-unit firmware audit first. + +The version is readable over the wire from `SUB 0x01` as two separate ASCII +runs — `"0CB"` then `"11"` — with no single concatenated string. + +### The strategic fork + +- **Option A — standardise the fleet on the Blastware line.** Every unit, + MiniMate and Micromate alike, then speaks Series III, and the existing + `minimateplus/` stack drives all of it. One protocol, one call-home + receiver. Dramatically cheaper *if* it holds up. +- **Option B — reverse-engineer the Thor line (`11.0BD`) and support both.** + +Option A is the shortcut, but it is unproven and carries real unknowns, all +of which are cheap to answer on the bench and expensive to discover later: + +1. **What file format does a `11.0CB` unit produce?** If it emits Blastware + binaries rather than `.IDFW`/`.IDFH`, the (now exact) Series III decoder + applies and the IDF codec becomes a legacy path. Not a loss — we have + both — but it changes what the ingest pipeline sees. +2. **Does Thor still work with a `11.0CB` unit?** If not, flashing a + production unit breaks data collection until the replacement path exists. +3. **Are any Micromate-specific capabilities lost** on the Blastware line? +4. **Is the flash reversible in the field**, and what does it cost in downtime? + ## How this was obtained No Thor, no modem, no Windows machine. The Micromate exposes its protocol on @@ -221,9 +278,13 @@ Also unknown: ## Session provenance -Unit **UM12947**, on the bench via USB, **zero events stored** (memory free == -total). Read commands only. Every response in this document was checksum- -validated. +Unit **UM12947**, firmware **`11.0CB`** (Blastware line), on the bench via +USB, **zero events stored** (memory free == total). Read commands only. +Every response in this document was checksum-validated. + +⚠ Firmware is the single biggest caveat on this document. Every finding here +is from one unit on the Blastware build. A `11.0BD` unit has not been +touched. An empty unit is a real limitation: `0x08`, `0x1E`, `0x0A` and `0x06` all have event-dependent payloads that could not be exercised. Recording a couple of From 73eaa0a6acacfb1e54d9b62ed1a5acc6ea4a66c5 Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 16:39:11 -0400 Subject: [PATCH 03/43] docs(series4): the event chain, walked end to end Five events on the bench unit (4 waveform + 1 histogram). The Series III browse walk -- 1E, then 0A/0C per key, then 1F to advance -- works unmodified, and the null sentinel terminated correctly after exactly 5. Findings: - Event keys are a sequential counter (055d4a81..85), NOT flash-buffer addresses. Series III key arithmetic does not carry over; its 5A chunk walk assumes addresses and must not be ported blindly. - The 4 bytes after the key in 1E/1F are the event's SIZE in bytes, where Series III puts an offset to the next key. 4,076 for the histogram and 8.7-13.4 KB for the waveforms, matching real .IDFH/.IDFW file sizes. - SUB 0x0C returns a 210-byte (0xD2) waveform record -- the same length as Series III -- carrying the event key, date/time, the title note "Location", the PROJECT STRING, the serial, channel labels Tran/Vert/Long/Mic and float32 peaks. That last point closes the biggest open question for the call-home receiver: the job identity strings that today arrive only via Thor's .txt sidecar, and which no amount of sample decoding can reconstruct, are readable over the wire. Direct-to-SFM events need not arrive with blank metadata. - SUB 0x0A returns len 0x1E for the histogram and 0x00 for every waveform. The histogram payload holds two timestamps plus a "Vert: 0.300 in/s" trigger string -- structurally the Series III monitor-log partial record. So 0A describes interval records and 0C describes triggered events; Series III's 0x46-vs-0x2C length discriminator does not apply. - DLE stuffing in responses is now confirmed (previously marked untested): the 0C timestamp contains 10 10, which destuffs to one 0x10 and yields a clock reading of 16:33 on 23 Sep 2026 -- matching when the events were recorded. Read-only throughout. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 93 ++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index c9f653a..0fe5332 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -241,12 +241,105 @@ the final 8 bytes; untestable until the unit holds events. --- +## The event chain — walked end to end (2026-09-23, 5 events) + +With 5 events on the bench unit (4 waveform + 1 histogram), the Series III +browse walk works unmodified: + +``` +1E (all-zero params) -> first key + size + 0A (key) -> partial record, histogram only + 0C (key) -> 210-byte waveform record +1F (all-zero params/browse) -> next key + size + ... repeat ... +1F -> all-zero key = NULL SENTINEL, chain ends +``` + +The sentinel terminated correctly after exactly 5 events. + +### Event keys are sequential, not addresses + +``` +055d4a81 055d4a82 055d4a83 055d4a84 055d4a85 +``` + +**This is a real divergence.** Series III keys are flash-buffer *addresses* +(`01110000`, `011121F2`, …) that advance by the event's byte length, which is +why its 5A chunk walk is address-arithmetic. Micromate keys are a plain +incrementing counter. Any port of the Series III download walk must not +assume key arithmetic means anything. + +### The 4 bytes after the key are the event's size + +`1E`/`1F` return `[key 4B][size 4B]`. Series III uses that slot as an offset +to the next key; here it is a byte count: + +| key | size | kind | +|---|---|---| +| `055d4a81` | 4,076 | histogram | +| `055d4a82` | 11,032 | waveform | +| `055d4a83` | 11,502 | waveform | +| `055d4a84` | 13,424 | waveform | +| `055d4a85` | 8,746 | waveform | + +Consistent with real file sizes (corpus `.IDFH` ≈ 3.7–25 KB, `.IDFW` ≈ +8.6–15.8 KB), and the histogram is unmistakably the small one. ⚠ Inferred, +not proven: the sizes sum to 48,780 while monitor status reports 57,344 bytes +used, so ~8.5 KB of overhead is unaccounted for. + +### `SUB 0x0C` — waveform record, and it carries the job metadata + +**Length `0xD2` = 210 bytes — identical to Series III.** Contents confirmed +across all 5 events: + +- the event key, echoed +- date + time (`17 09 07 ea` → 23 Sep 2026, then `10 21` → 16:33 — matching + the actual bench recording time) +- title note `"Location"` +- **the project string** — `"Univ of Pitt-1st Yr Housing-Loc1 Ruskin"` +- serial `"UM12947"` +- channel labels `Tran` / `Vert` / `Long` / `Mic` — the same labels Series III + uses, and the same label-relative float32 layout +- per-event float32 peaks: 3.5152, 1.3720, 2.3542, 3.5152, 0.4227 in/s + across the five events (varied deliberately during recording) + +**This closes the biggest open question for the call-home-receiver goal.** +The job identity strings (`project` / `client` / `operator` / `setup`) that +today arrive only via Thor's `.txt` sidecar — and which no amount of sample +decoding can reconstruct — are **available over the wire from `0x0C`**. A +direct-to-SFM event need not arrive with blank metadata. + +### `SUB 0x0A` — partial record, histogram only + +`0x0A` returned `len = 0x1E` (30 B) for the histogram and `len = 0x00` for all +four waveforms. The histogram payload carries **two timestamps** and the +ASCII string `"\r Vert: 0.300 in/s"` — structurally the Series III +**monitor-log partial record** (`0x2C` type), which likewise holds a start/stop +pair and a `"Geo: in/s"` trigger string. + +So on the Micromate the division of labour is: `0x0A` describes interval-style +records, `0x0C` describes triggered events. Series III uses `0x0A`'s +*response length* (`0x46` vs `0x2C`) to tell real events from boundaries; +that discriminator does not apply here. + +### DLE stuffing in responses — confirmed present + +Earlier marked untested. The `0x0C` timestamp field contains `10 10`, which +destuffs to a single `0x10` and yields a sensible clock reading. **Responses +are DLE-stuffed**, so a parser must destuff before applying field offsets. + +--- + ## Divergences from Series III (running list) 1. **No `DLE` prefix on responses** — bare `STX`. 2. **Response flags byte is `0xC5`**, not `0x10`. 3. **Call-home config is 126 bytes**, not 124. 4. **Data lengths are discoverable** from the probe response at `payload[9]`. +4b. **Event keys are a sequential counter**, not flash addresses. +4c. **`1E`/`1F` return the event size**, where Series III returns an offset. +4d. **`0x0A` vs `0x0C` split by record type**, not by the `0x46`/`0x2C` + length discriminator Series III uses. 5. **Modem serial rate is 115200**, not 38400 (per TMI provisioning practice; not independently verified here). From 492b6683a44c397328bdf20744b7be3739aae7a7 Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 17:08:47 -0400 Subject: [PATCH 04/43] docs(series4): fleet firmware audit, and retract the Thor-compatibility claim Physical audit of all nine Micromates: 4 on the Blastware line (11.0CB), 2 on the Thor line (11.0BD), 3 pre-split (11.0AK x2, 10.90GC). The Blastware line is already the plurality, which makes "standardise on Blastware" less disruptive than it first looked. Cross-checked against a store-derived audit (firmware is recorded in every .sfm.json as extensions.idf_report.version): 7 of 9 agree. The two that differ, UM6047 and UM14133, are the most recently deployed and were reflashed after their last stored event -- so the store reconstructs firmware history without touching a unit, but lags reality by one deployment. RETRACTION: an earlier draft suggested UM12947's trouble with Thor was explained by its Blastware firmware. Not supported. Ped Bridge runs UM11402 (11.0BD) and UM11719 (11.0CB) side by side from the same deploy date and both call Thor fine -- UM11719 has 331 Thor-collected events while on 11.0CB. A Blastware-line unit does feed Thor, so the CB/BD split is not "which host can collect from it", and UM12947's problem remains unexplained. What is actually established is narrower: a 11.0CB unit answers Series III command frames. Whether a 11.0BD unit does is untested -- and UM20147 (11.0BD) is on the bench, so that is one A/B away. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 52 ++++++++++++++++++++++------ 1 file changed, 42 insertions(+), 10 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 0fe5332..88e45f5 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -50,18 +50,50 @@ machines, and units that will not cross over. It is not a misconfiguration. **Instantel ships different firmware for different host software**, and the wire protocol differs with it. -Versions observed so far: +### Fleet audit (physical, 2026-09-22/23) -| where | version | notes | -|---|---|---| -| bench unit UM12947, today | **`11.0CB`** | Blastware line — answers Series III | -| corpus, 932 event files | `11.0AK` | produced Thor-collected `.IDFW`/`.IDFH` | -| corpus, 83 event files | `10.90GC` | older; UM12947's own files from Sept 2025 | +| unit | firmware | line | location | +|---|---|---|---| +| UM11719 | `11.0CB` | Blastware | Ped Bridge Loc 2 | +| UM6047 | `11.0CB` | Blastware | Brookville Loc 9 | +| UM12947 | `11.0CB` | Blastware | **bench** | +| UM14133 | `11.0CB` | Blastware | Pitt-Music Bldg Loc 1 | +| UM11402 | `11.0BD` | Thor | Ped Bridge Loc 1 | +| UM20147 | `11.0BD` | Thor | **bench** | +| UM13981 | `11.0AK` | pre-split | RKM Loc 1 | +| UM20146 | `11.0AK` | pre-split | Karns Loc 2 | +| UM12420 | `10.90GC` | pre-split | RKM Loc 2 | -Note UM12947 produced `10.90GC` files in production last year and reports -`11.0CB` on the bench now — so **it has been reflashed at some point**, and -firmware is not stable per-unit over time. Any fleet-wide claim needs a -per-unit firmware audit first. +**4 Blastware / 2 Thor / 3 pre-split.** The Blastware line is already the +plurality, which makes "standardise on Blastware" less disruptive than it +first appeared. + +A store-derived audit (firmware is recorded in every `.sfm.json` sidecar as +`extensions.idf_report.version`) agreed with the physical audit on **7 of 9**. +The two that differed — UM6047 and UM14133 — are the most recently deployed +units, reflashed after their last stored event. Useful technique: the fleet's +firmware history is reconstructable from the store without touching a unit, +but it lags reality by one deployment. + +Firmware is **not stable per-unit over time** — five of nine have been +reflashed at least once. Any fleet-wide claim needs a fresh audit. + +### ⚠ Retraction: firmware line does NOT determine Thor compatibility + +An earlier draft of this document suggested that UM12947's trouble with Thor +was explained by its being on the Blastware build. **That is not supported.** + +Ped Bridge runs UM11402 (`11.0BD`) and UM11719 (`11.0CB`) side by side, both +deployed 2026-04-20, and **both call Thor successfully** — UM11719 has 331 +Thor-collected events in the store while on `11.0CB`, through 2026-08-23. + +So a Blastware-line unit does feed Thor. Whatever the CB/BD split changes, it +is not "which host software can collect from it", and UM12947's specific +problem remains unexplained. + +**What is actually established:** a `11.0CB` unit answers Series III command +frames. Whether a `11.0BD` unit does is **untested** — and UM20147 (`11.0BD`) +is on the bench, which makes that a direct A/B away. The version is readable over the wire from `SUB 0x01` as two separate ASCII runs — `"0CB"` then `"11"` — with no single concatenated string. From 38ad58d4a4c9308d4a583a86cdb8c0f45f521896 Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 18:20:11 -0400 Subject: [PATCH 05/43] docs(series4): A/B the two firmware lines -- the protocol is the same UM12947 (11.0CB, Blastware line) and UM20147 (11.0BD, Thor line) each given the identical read-only sweep on the bench. Both answer Series III command frames: all ten read SUBs, correct response-SUB rule, valid DLE-aware checksums, working two-step probe/data reads. The firmware line does not change the wire protocol. One protocol stack can drive the whole fleet regardless of build, which downgrades "standardise the fleet on one firmware" from a prerequisite to an optional convenience. Two differences do exist: 1. Response payload[1] (flags) is 0xC5 on the Blastware line and 0x03 on the Thor line, constant across all ten SUBs on both units -- so the build is detectable from any response without reading device info. Two units, one each, so this is a strong hypothesis rather than a proven encoding. Note 0x03 is ETX, so it arrives DLE-escaped as 10 03 on Thor-line units. A parser that does not destuff will mis-locate every field by one byte on half the fleet. 2. SUB 0x1C (monitor status) is 4 bytes longer on the Thor line, 0x30 vs 0x2C, with four extra trailing bytes (0f a0 00 00, purpose unknown). That second one breaks relative-to-end parsing: Series III reads battery and memory from the end of the 0x1C block, and those offsets yield a battery reading of 577.92 V on UM20147. Parse forward from the declared length, not backward from the end. With the shift applied, UM20147 reads 3.81 V and 15,000,000 bytes total/free. Also noted: ID string is MM/ISEE/S/IO on the Blastware unit and MM/ISEE/S on the Thor one. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 49 ++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 88e45f5..be516a2 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -362,6 +362,53 @@ are DLE-stuffed**, so a parser must destuff before applying field offsets. --- +## A/B: Blastware build vs Thor build (2026-09-23) + +UM12947 (`11.0CB`) and UM20147 (`11.0BD`) were each put on the bench and given +the identical read-only sweep. **Both answer Series III command frames.** + +| | UM12947 `11.0CB` | UM20147 `11.0BD` | +|---|---|---| +| POLL answers | ✓ | ✓ | +| All 10 read SUBs answer | ✓ | ✓ | +| `response_SUB = 0xFF − req` | ✓ | ✓ | +| DLE-aware checksum valid | ✓ | ✓ | +| Two-step probe/data read | ✓ | ✓ | +| ID string | `MM/ISEE/S/IO` | `MM/ISEE/S` | +| **flags byte** | **`0xC5`** | **`0x03`** | +| **`0x1C` length** | **`0x2C`** | **`0x30`** | + +**The firmware line does not change the wire protocol.** One protocol stack +can drive the whole fleet regardless of which build a unit is on. This is the +single most consequential finding so far: "standardise the fleet on one +firmware" becomes an *optional* convenience rather than a prerequisite for +building a call-home receiver. + +### The two differences that do exist + +**1. The flags byte identifies the build.** Response `payload[1]` is `0xC5` +on the Blastware line and `0x03` on the Thor line, constant across all ten +SUBs on both units. That makes firmware line detectable from *any* response, +without reading device info. ⚠ Two units, one each — treat as a strong +hypothesis, not a proven encoding. + +Note `0x03` is ETX, so on Thor-line units it arrives DLE-escaped as `10 03`. +A parser that fails to destuff will mis-locate every field by one byte on +exactly half your fleet. + +**2. `SUB 0x1C` (monitor status) is 4 bytes longer on the Thor line** — +`0x30` vs `0x2C` — with four extra trailing bytes (`0f a0 00 00`, purpose +unknown). + +⚠ **This breaks relative-to-end parsing.** Series III reads battery and +memory from the *end* of the `0x1C` block (`[-10:-8]`, `[-8:-4]`, `[-4:]`). +Those offsets are correct on `11.0CB` and wrong on `11.0BD` — applying them +blindly to UM20147 yields a battery reading of **577.92 V**. Parse forward +from the declared length instead of backward from the end. + +With the offsets shifted by 4, UM20147 reads correctly: battery **3.81 V**, +memory 15,000,000 total and free (no events stored). + ## Divergences from Series III (running list) 1. **No `DLE` prefix on responses** — bare `STX`. @@ -372,6 +419,8 @@ are DLE-stuffed**, so a parser must destuff before applying field offsets. 4c. **`1E`/`1F` return the event size**, where Series III returns an offset. 4d. **`0x0A` vs `0x0C` split by record type**, not by the `0x46`/`0x2C` length discriminator Series III uses. +4e. **Response `payload[1]` (flags) encodes the firmware line** — `0xC5` + Blastware, `0x03` Thor — where Series III has a constant `0x10`. 5. **Modem serial rate is 115200**, not 38400 (per TMI provisioning practice; not independently verified here). From 45007e12d809abf90baa9a011dd3b092be73be4e Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 18:28:40 -0400 Subject: [PATCH 06/43] docs(series4): the firmware images are unencrypted and self-documenting Both MICROMATE(CB).BIN and MICROMATE(BD).BIN are plain code and data -- entropy 6.08 bits/byte, big-endian vector table at 0x4010_30xx, ~16,700 extractable strings including the developers' own debug printf formats with function names intact. This answers, from strings alone, questions I had scoped as needing a live modem capture. The call-home state machine, verbatim: ACH_NOT_STARTED -> ACH_IDLE -> ACH_INITIALIZING -> ACH_CONNECTING -> ACH_CONNECTED -> ACH_TRANSFER_DATA -> ACH_RETRY / ACH_QUITTING And with it: - Retry limit is three ("three attempts and it's over"). - ExpectedCommunicationsDetected() gates the session: if the host does not say something the unit recognises, the call is cancelled and rescheduled after TimeBetweenRetries. A homebrew receiver must satisfy this check or units retry forever -- exactly the BE12599 failure mode. - The unit stops monitoring to call home and restarts after (Send CMD_STOP_MONITOR / CMD_START_MONITOR), so monitoring state around a call is the device's own doing. - Calls are not re-entrant. - CMD_CALLHOME_CONNECTION_CONFIRMED exists as a state distinct from CONNECTION_COMPLETE, implying a handshake the host must complete before data flows. Event delivery, inferred not confirmed: "All Events Uploaded" plus "Mark/Unmark File" / "Delete Marked Events" / CMD_PURGE_EVENT_FLASH suggest events are marked as transferred rather than deleted on send, with purging a separate explicit act. If so, a receiver that fails to mark would see the same events re-offered every call. Needs a live capture or disassembly to confirm. The firmware also embeds its own HTML user manual, documenting modem mode (Generic vs USB to PC), the modem baud options (9600-230400, confirming 115200 is a setting not a fixed rate), modem relay/warmup, record modes, and a scheduler downloaded from THOR that pairs with CMD_CALLHOME_SET_SCHEDULE. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 98 ++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index be516a2..d3a61c6 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -426,6 +426,104 @@ memory 15,000,000 total and free (no events stored). --- +## The firmware images are unencrypted — and they document themselves + +`ref-stuff/micromate-firmware/MICROMATE(CB).BIN` and `MICROMATE(BD).BIN`, +~2.77 MB each and within 192 bytes of one another. + +- **Entropy 6.08 bits/byte** — neither encrypted nor compressed. Plain code + and data. +- Header is a **big-endian vector table**, handlers at `0x4010_30xx`. +- **~16,700 extractable strings**, including the developers' own debug + `printf` format strings with function names intact. + +This is a legitimate interoperability reference for hardware TMI owns, and it +short-circuits work I had scoped as "only answerable from a live modem +capture". + +### The call-home state machine, verbatim + +``` +ACH_NOT_STARTED → ACH_IDLE → ACH_INITIALIZING → ACH_CONNECTING + → ACH_CONNECTED → ACH_TRANSFER_DATA + → ACH_RETRY / ACH_QUITTING (also ACH_STARTED) +``` + +Supporting strings: + +``` +ACH: Entry StartCallHome() +ACH: CallHome_task ; CheckAliveTime CANCEL ; TimeBetweenRetries = %d +ACH: CallHome_task ; !ExpectedCommunicationsDetected() CANCEL ; TimeBetweenRetries = %d +ACH: CallHomeCommectionCompleteProcessing() ACH=%s EAMWC=%s +ACH: %s() three attempts and it's over +ACH: %s() Send CMD_START_MONITOR +ACH: %s() Send CMD_STOP_MONITOR +ACH: Start Ignore request, Call Home is in progress +``` + +What this tells us without a single captured packet: + +1. **Retry limit is three** — "three attempts and it's over". +2. **`ExpectedCommunicationsDetected()` gates the session.** If the host does + not say something the unit recognises, the call is *cancelled* and + rescheduled after `TimeBetweenRetries`. A homebrew receiver must satisfy + this check or units will retry forever — which is exactly the failure mode + seen on BE12599. +3. **The unit stops monitoring to call home and restarts afterwards** + (`Send CMD_STOP_MONITOR` / `CMD_START_MONITOR`). Relevant to any + wedged-unit rescue: the monitoring state around a call is the device's own + doing, not ours. +4. **Calls are not re-entrant** — "Call Home is in progress" is ignored. + +### Internal command table + +`CMD_CALLHOME`, `CMD_CALLHOME_CANCEL`, **`CMD_CALLHOME_CONNECTION_CONFIRMED`**, +`CMD_CALLHOME_CONNECTION_COMPLETE`, `CMD_CALLHOME_SET_SCHEDULE`, +`CMD_CALLHOME_CLEAR_SCHEDULE`, `CMD_STOP_CALLHOME_FILETRANSFER`, +`CMD_DUTYCYCLE_AUTOCALLHOME`, `CMD_PURGE_EVENT_FLASH`. + +`CONNECTION_CONFIRMED` as a distinct state from `CONNECTION_COMPLETE` implies +a **handshake the host must complete before data flows** — the concrete shape +of `ExpectedCommunicationsDetected()`. + +### Event delivery — the mechanism, probably + +``` +All Events Uploaded +Mark/Unmark File Delete Marked Events Marked Events were Deleted +MONITOR::MESG PURGE_EVENT_FLASH BEGIN / END +``` + +A **marking** mechanism exists, alongside a distinct "all uploaded" terminal +state. 🔶 **Inferred:** events are *marked* as transferred rather than +deleted on send, and purging is a separate explicit act. If so, a receiver +that fails to mark would see the same events re-offered every call — the +question that gates a safe homebrew receiver. **Not yet confirmed**; needs +either a live call-home capture or disassembly around these strings. + +### A full user manual is embedded + +The firmware carries its own HTML help, which documents configuration we would +otherwise have to infer: + +- **Modem mode**: `Generic` (through a modem) vs `USB to PC`. +- **Modem baud**: 9600 / 19200 / 38400 / 57600 / 115200 / 230400 — "must match + the expected rate of the PC or modem". Confirms 115200 is a *setting*, not + a fixed rate. +- **Modem relay + warmup** (0–300 s), auxiliary mode, warning/alarm hold. +- **Record modes**: Waveform, Waveform Manual, Histogram, Histogram-Combo; + sample rates 1024 / 2048 / 4096. +- **A scheduler downloaded from THOR** that can start/stop monitoring, change + record mode, trigger a call home, or run a self check on a daily/weekly + schedule. Pairs with `CMD_CALLHOME_SET_SCHEDULE`. + +### Still worth doing + +Diffing the two images should isolate exactly what the CB/BD split changes — +we know the wire protocol is not it, and the flags byte (`0xC5` vs `0x03`) +gives a concrete anchor to search for. + ## ⚠ Untested and unsafe-until-agreed Nothing below has been sent to a unit, and nothing should be without an From 71f19c90d13338e052e1ac3f6da005c2a64536b8 Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 18:40:54 -0400 Subject: [PATCH 07/43] docs(series4): SUB 5A streams the .IDFW file verbatim -- read path complete The complete read path now works with no Instantel software in the loop. Three divergences from Series III, all simplifications: - No arming sequence. Series III ignores a 5A probe unless preceded by 1E / 0A / 1E(0xFE) / 0C / 1F(0xFE) / POLL x3. The Micromate answers a bare 5A request with nothing before it. - The offset word is a LENGTH, not a position: 0x1000 + 2*pages, where pages = ceil(event_size / 512), and event_size comes from the chain walk. ONE request returns the entire event -- no chunk loop, no STRT end-offset parsing, no TERM frame. Over-requesting is safe; the device caps at the real size. - Params are the Series III probe form: [0x00][key4][6 x 0x00]. The payload is the .IDFW file byte for byte. It begins 00 12 01 00 00 00 "Instantel\0" -- _THOR_PREFIX + _INSTANTEL_TAG from micromate/idf_file.py -- and the first 32 bytes are identical to a production .IDFW from the store. Responses are DLE-stuffed, so destuff before locating the file (11,781 raw -> 11,049 destuffed for an 11,032-byte event). End-to-end: event 055d4a82 downloaded over USB and fed straight to read_idf_file() yields serial UM12947, timestamp 2026-09-23 16:33:19, and 3072 samples on all four channels. Cross-check: the 0C record reports a stored Vert peak of 1.3720 for this event; the decoded samples give 1.3706 -- two unrelated paths agreeing to 0.1%. Consequence: no new codec work is needed. The bytes off the wire are the same bytes thor-watcher forwards today, so /db/import/idf_file ingests a directly downloaded event unchanged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 82 +++++++++++++++++++++++++++- 1 file changed, 80 insertions(+), 2 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index d3a61c6..72d7f9b 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -524,6 +524,86 @@ Diffing the two images should isolate exactly what the CB/BD split changes — we know the wire protocol is not it, and the flags byte (`0xC5` vs `0x03`) gives a concrete anchor to search for. +## `SUB 0x5A` — bulk download. It streams the `.IDFW` file verbatim. + +**The complete read path works with no Instantel software in the loop.** + +### It needs no arming sequence + +Series III ignores a `5A` probe unless preceded by +`1E → 0A → 1E(token 0xFE) → 0C → 1F(token 0xFE) → POLL × 3`. The Micromate +answers a **bare `5A` request** with nothing before it. That whole ritual is +gone. + +### The offset word is a LENGTH, not a position + +This is the key divergence. Series III walks chunks by absolute flash +address, stepping `0x0200` per request. On the Micromate the offset word +requests *how much to send*: + +``` +offset_word = 0x1000 + 2 × pages pages = ceil(event_size / 512) +``` + +| `offset_word` | pages | destuffed file bytes | +|---|---|---| +| `0x1002` | 1 | 518 | +| `0x1004` | 2 | 1,030 | +| `0x1006` | 3 | 1,541 | +| `0x102C` | 22 | **11,033 — the whole event** | + +`event_size` comes from the chain walk (the 4 bytes after the key in +`1E`/`1F`). **One request returns the entire event**; there is no chunk loop, +no `STRT` end-offset parsing, and no `TERM` frame. Over-requesting is safe — +`0x1030` (24 pages) returned exactly the same bytes as `0x102C`, so the device +caps at the real size. + +Params are the Series III *probe* form: `[0x00][key4][6 × 0x00]`. + +### The payload is the `.IDFW` file, byte for byte + +``` +[18-byte frame header] [ .IDFW file ] [chk] [ETX] ← raw wire + ^ destuffed offset 16 +``` + +The file begins `00 12 01 00 00 00 "Instantel\0"` — `_THOR_PREFIX` + +`_INSTANTEL_TAG` from `micromate/idf_file.py`. The first 32 bytes are +**identical to a production `.IDFW`** pulled from the store. + +⚠ Responses are DLE-stuffed. Destuff before locating the file, or the raw +byte count overshoots (11,781 raw → 11,049 destuffed for an 11,032-byte event). + +### End-to-end proof + +Event `055d4a82` downloaded over USB and fed straight to `read_idf_file()`: + +``` +serial UM12947 +timestamp 2026-09-23 16:33:19 +samples Tran 3072 Vert 3072 Long 3072 MicL 3072 +peaks Tran 0.2433 Vert 1.3706 Long 0.2672 in/s +``` + +All four channels equal length, and the timestamp matches the `0x0C` record +for the same key. **Independent cross-check:** `0x0C` reports a stored peak +of **1.3720** for this event; the decoded samples give **1.3706** — two +unrelated paths agreeing to 0.1%. + +**Consequence:** no new codec work is needed. The bytes off the wire are the +same bytes `thor-watcher` forwards today, so `/db/import/idf_file` ingests a +directly-downloaded event unchanged. Everything the IDF decoder already does +per-sample-exact applies. + +### What a full read now looks like + +``` +1E → first key + size + 0C(key) → project/client/operator, timestamp, peaks + 5A(key, 0x1000+2×ceil(size/512)) → the whole .IDFW +1F → next key + size (until null sentinel) +``` + ## ⚠ Untested and unsafe-until-agreed Nothing below has been sent to a unit, and nothing should be without an @@ -538,8 +618,6 @@ explicit decision: Also unknown: - Whether `0x10` bytes inside request params need stuffing -- Whether the bulk waveform stream (`5A` on Series III) exists here, and - whether it is the transport for `.IDFW` bodies we already decode - Everything about the **call-home session** — the device-initiated direction has not been observed at all. Specifically: how a unit announces itself, and **how it learns an event was accepted so it stops re-sending it.** From 23cdbef737af2f75a4fa0db61719b8970699891a Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 18:55:20 -0400 Subject: [PATCH 08/43] docs(series4): setups are files; and the length field is a uint16 Two findings and one correction. CORRECTION: the probe response's data length is a uint16 BE at payload[8:10], not a single byte at payload[9] as an earlier draft claimed. That reading is right only while the high byte is zero. For SUB 0x1A the real length is 0x082C = 2092; read as a byte it gives 44, a 47x under-read. Setups are FILES, not a config block. Series III has one compliance config you overwrite; Series IV keeps named .MMB setup files on an on-device filesystem with a current-selection pointer -- csetup.MMB, factory.MMB, and callhome.MMB for the call-home config. Names up to 20 chars. Filesystem primitives exist internally (NS_ReadFile_internal / NS_WriteFile_internal / NS_SeekFile_internal) but no generic file-transfer command is exposed on the wire, so setups are unlikely to be pushed as raw .MMB blobs over the protocol. SUB 0x1A reads the whole active setup in 2,092 bytes -- structurally close to Series III's ~2,126-byte compliance block -- carrying the setup FILE NAME, all four title note/value pairs (Location, Client, Company, General Notes), the sensor location, and per-channel labels with units. Note LMic and SMic (linear and sound-level microphone variants) which Series III does not have. That is the read half of setup management, so a setup can in principle be round-tripped. The write half has not been attempted. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 52 ++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 72d7f9b..5b69bcc 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -197,7 +197,12 @@ Series III reads. ### The probe response carries the data length Series III hardcodes `DATA_LENGTHS` per SUB. On the Micromate the **probe -response tells you**, at `payload[9]`: +response tells you**, as a **uint16 BE at `payload[8:10]`**: + +⚠ **Corrected 2026-09-23.** An earlier draft read this as a single byte at +`payload[9]`. That is right only while the high byte is zero, and it is +catastrophically wrong for `SUB 0x1A`, whose real length is `0x082C` = **2092** +— read as a byte it gives **44**, a 47× under-read. Always read the pair. | SUB | command | `payload[9]` | Series III constant | |---|---|---|---| @@ -208,7 +213,7 @@ response tells you**, at `payload[9]`: | `0x2C` | call-home config | `0x7E` | `0x7C` ✗ **differs by 2** | | `0x08` | event index | `0x5A` | — | | `0x1E` | event header | `0x08` | — | -| `0x1A` | compliance config | `0x2C` | — | +| `0x1A` | compliance config | `0x082C` (2092) | — | | `0x0A` | waveform header | `0x00` | — (no event context) | | `0xFE` | full config | `0x00` | — (see note) | @@ -604,6 +609,49 @@ per-sample-exact applies. 1F → next key + size (until null sentinel) ``` +## Setups are FILES, not a config block + +Series III has one compliance config you overwrite. Series IV keeps **named +setup files on an on-device filesystem**, with a pointer to the current one. +From the firmware: + +``` +csetup.MMB the current setup +factory.MMB Factory Default Setup File +callhome.MMB call-home config is a file too +"Current Setup File: " "Can Not Delete Active Setup File" +GetSelectedSetupFilePathName() CSelectSetupFiles CSaveSetupFile +``` + +Names are up to 20 characters and may contain spaces, hyphens, underscores. +The unit's help text describes selecting, renaming and deleting them, and the +event list records which setup file produced each event. + +Filesystem primitives exist internally (`NS_ReadFile_internal`, +`NS_WriteFile_internal`, `NS_SeekFile_internal`), but **no generic +file-transfer command is exposed on the wire** — the only file-transfer string +is `CMD_STOP_CALLHOME_FILETRANSFER`. So setups are unlikely to be pushed as +raw `.MMB` blobs over the protocol. + +### `SUB 0x1A` reads the whole active setup — 2,092 bytes + +Structurally close to Series III's ~2,126-byte compliance block, and it +carries everything a setup consists of: + +- **the setup file name** — `Univ of Pitt-1st Yr. Housing-Loc1 Ruskin.MMB` +- all four title note/value pairs — `Location`, `Client`, `Company`, + `General Notes`, with their strings +- the sensor location string (`Loc 1`) +- per-channel labels *and units*: `Tran in./s.`, `Vert in./s.`, + `Long in./s.`, `Mic psi (L)`, `LMic psi (L)`, `SMic (A)` + +Note `LMic` / `SMic` — linear and sound-level microphone variants that +Series III does not have. + +This is the **read half of setup management**, and it means a setup can be +round-tripped: read the active config, modify, write it back. The write half +is not yet attempted. + ## ⚠ Untested and unsafe-until-agreed Nothing below has been sent to a unit, and nothing should be without an From 02ed22f561d81dc4fcb8c91e267f64ae5130a39d Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 19:14:26 -0400 Subject: [PATCH 09/43] docs(series4): firmware static analysis, and all five bench events decoded Solo session while the bench was unattended. Read-only throughout. Architecture: ColdFire/68K, big-endian, Freescale MQX RTOS -- not ARM as the vector table first suggested. The tell is 4E 5E 4E 75 4E 56 (UNLK A6 / RTS / LINK A6) throughout both images, plus an MQX_OK assertion. CB vs BD: a byte diff is useless (68% of bytes differ -- separately linked builds, everything relocated). A string-set diff is position-independent and shows 17,128 strings shared, with almost every "unique" string being the same message at a different source line: CB: MONITOR[3268]: STATUS_BATTERY_LOW BD: MONITOR[3258]: STATUS_BATTERY_LOW Consistently 10 lines apart across five different MONITOR messages, so one ~10-line block differs in the monitor module and essentially nothing else. The only functional string unique to either build is CITIZEN (a printer brand) in BD. This corroborates the bench A/B from the other direction: the split is a tiny code delta, not two protocol stacks. The SUB dispatch is a 68K switch jump table, so byte-pattern hunting will not isolate the write opcodes -- that needs a disassembler. Call-home config field names recovered from the firmware's own debug dump: Enable, DialString, Retries, SessionTimeout, WaitForConnection, WarmupTime, PowerSave -- seven fields for the 126-byte SUB 0x2C block. SessionTimeout and PowerSave have no Series III equivalent, and Series III's scheduled-time fields are absent, consistent with scheduling moving into the THOR-downloaded scheduler. AT+CSQ is present, so the firmware speaks AT to the modem directly. All five bench events downloaded and decoded over USB: each arrived at exactly its declared size, every channel equal length, timestamps sequential. Two gaps recorded: - No content-based record-type discriminator. read_idf_file() dispatches on the .IDFH/.IDFW filename suffix, which does not exist over the wire, and the first 64 bytes of a histogram and a waveform are byte-identical. The protocol supplies one instead: SUB 0x0A returns 0x1E for a histogram and 0x00 for a waveform, so the type must be carried from the chain walk. - The 0x0C peak float runs 2-5% above max(Tran,Vert,Long) and is not the vector sum either. Its offset was inferred from a byte marker rather than established, so it may not be the peak at all. Marked do-not-rely-on. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 106 +++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 5b69bcc..9f86180 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -652,6 +652,112 @@ This is the **read half of setup management**, and it means a setup can be round-tripped: read the active config, modify, write it back. The write half is not yet attempted. +## Static analysis of the firmware (2026-09-23, solo session) + +### Architecture + +**ColdFire / 68K, big-endian, Freescale MQX RTOS** — not ARM as the vector +table first suggested. The giveaway is the function epilogue/prologue +`4E 5E 4E 75 4E 56` = `UNLK A6` / `RTS` / `LINK A6`, littered through both +images, plus an `MQX_OK` assertion string. + +### CB vs BD: the same source, ~10 lines apart + +A byte diff is useless — **68% of bytes differ** because the two are separately +linked builds with everything relocated. A *string-set* diff is +position-independent and tells the real story: **17,128 strings shared**, and +almost every "unique" string is the same message with a different source line +number: + +``` +CB: MONITOR[3268]: STATUS_BATTERY_LOW +BD: MONITOR[3258]: STATUS_BATTERY_LOW ← consistently 10 lines apart +``` + +The offset is exactly 10 across `STATUS_BATTERY_LOW`, `STATUS_BATTERY_CRITICAL`, +`Battery Critical Exit Monitor`, `histogram interval size of 0` and +`Offsets by channel` — so one ~10-line block differs in the monitor module and +essentially nothing else. The only functional string unique to either build is +`CITIZEN` (a receipt-printer brand) in BD. + +**This corroborates the bench A/B from the other direction:** the CB/BD split +is a tiny code delta, not two protocol stacks. Whatever drives Instantel to +ship two downloads, it is not a different wire protocol. + +⚠ The `SUB` dispatch is a 68K switch jump table (`CMPI.L` bounds check → +`MOVE.W (table,PC,Dn)` → `JMP (d8,PC,Xn)`). Byte-pattern hunting will not +isolate the write opcodes — that needs a real disassembler. + +### Call-home config field names, from the firmware's own debug dump + +``` +CallHome.Enable = %s +CallHome.DialString = "%s" +CallHome.Retries = %d +CallHome.SessionTimeout = %d +CallHome.WaitForConnection = %d +CallHome.WarmupTime = %d +CallHome.PowerSave = %s +``` + +Seven fields, which is what the 126-byte `SUB 0x2C` block has to encode. Note +`SessionTimeout` and `PowerSave` have no Series III equivalent, and Series III's +scheduled-time fields (`time1/time2 hour/min`) are absent here — consistent +with Series IV moving scheduling into the THOR-downloaded scheduler instead. + +Also present: `AT+CSQ` (signal quality), so the firmware talks AT to the modem +directly. + +## All five bench events, downloaded and decoded + +Read-only, over USB, no Instantel software: + +| key | declared size | got | decoded | +|---|---|---|---| +| `055d4a81` | 4,076 | 4,076 | histogram, 1 interval, 16:33:16 | +| `055d4a82` | 11,032 | 11,032 | waveform, 3072 × 4 ch, 16:33:19 | +| `055d4a83` | 11,502 | 11,502 | waveform, 3072 × 4 ch, 16:33:27 | +| `055d4a84` | 13,424 | 13,424 | waveform, 3072 × 4 ch, 16:33:34 | +| `055d4a85` | 8,746 | 8,746 | waveform, 2048 × 4 ch, 16:33:36 | + +Every event arrived at exactly its declared size, every channel came out equal +length, and the timestamps are sequential across the recording session. + +### ⚠ Gap: no content-based record-type discriminator + +`read_idf_file()` decides waveform vs histogram from the **filename suffix** +(`.IDFH` / `.IDFW`) — and there is no filename when downloading over the wire. +The first 64 bytes of a histogram and a waveform are **byte-identical**, so the +header does not obviously carry the type either. + +The protocol does supply one: **`SUB 0x0A` returns length `0x1E` for a +histogram and `0x00` for a waveform** (see the event-chain section). Any +direct-download implementation must carry the type from the chain walk rather +than inferring it from the payload, or find the type field inside the IDF +header. + +### ⚠ Unresolved: the `0x0C` peak float + +The float32 extracted from `0x0C` runs 2–5% above `max(Tran, Vert, Long)` from +the decoded samples: + +| key | `0x0C` float | max channel | +|---|---|---| +| `…81` | 3.5152 | 3.4419 | +| `…82` | 1.3720 | 1.3706 | +| `…83` | 2.3542 | 2.2227 | +| `…84` | 3.5152 | 3.4419 | +| `…85` | 0.4227 | 0.4198 | + +It is not peak vector sum either (computed PVS runs *higher* than both). The +field may not be the peak at all — its offset was inferred from a byte marker, +not established. **Do not rely on it** until it is pinned properly. + +Worth noting the histogram (`…81`) and the loudest waveform (`…84`) report +*identical* peaks to four decimals, in both measures. That is self-consistent: +the histogram's single 1-minute interval spans the whole thumping session, so +its maximum should equal the loudest event in it. + ## ⚠ Untested and unsafe-until-agreed Nothing below has been sent to a unit, and nothing should be without an From 701af47170174554f268b3aa61b16288abe79456 Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 19:50:45 -0400 Subject: [PATCH 10/43] docs(series4): generate IDF filenames rather than detecting record type Closes the record-type gap flagged earlier, and corrects the premise behind it. Series III does NOT detect record type from file content -- event_file_io.derive_record_type_from_filename() reads the last character of the extension (M529LKIQ.G10H -> H -> Histogram). Nothing in the codebase infers record type from content, for either family. Nor is there an obvious type field to find in an IDF: the first 64 bytes of a histogram and a waveform are byte-identical, and they diverge at ~0x0947 into wholly different structures rather than differing by a flag. The answer is the Series III pattern -- generate the name. Series III has blastware_filename(); Series IV needs the same, and its convention is far simpler: _.IDF{W,H} e.g. UM12947_20260923163319.IDFW against Series III's . All three inputs are already available on a direct download: serial and timestamp from extract_binary_metadata(), and type from the chain walk (SUB 0x0A returns 0x1E for a histogram, 0x00 for a waveform). Verified on all five bench events -- generated names match real production-store filenames byte for byte, so a directly downloaded event can be filed under exactly the name Thor would have given it and /db/import/idf_file needs no change. The type still comes from the protocol rather than the payload, so a downloader must carry it out of the chain walk; losing it means losing the ability to name the file. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 49 ++++++++++++++++++++++------ 1 file changed, 39 insertions(+), 10 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 9f86180..8de4a36 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -723,18 +723,47 @@ Read-only, over USB, no Instantel software: Every event arrived at exactly its declared size, every channel came out equal length, and the timestamps are sequential across the recording session. -### ⚠ Gap: no content-based record-type discriminator +### Record type + filename: generate it, don't detect it -`read_idf_file()` decides waveform vs histogram from the **filename suffix** -(`.IDFH` / `.IDFW`) — and there is no filename when downloading over the wire. -The first 64 bytes of a histogram and a waveform are **byte-identical**, so the -header does not obviously carry the type either. +`read_idf_file()` decides waveform vs histogram from the **filename suffix** — +and there is no filename when downloading over the wire. -The protocol does supply one: **`SUB 0x0A` returns length `0x1E` for a -histogram and `0x00` for a waveform** (see the event-chain section). Any -direct-download implementation must carry the type from the chain walk rather -than inferring it from the payload, or find the type field inside the IDF -header. +⚠ Worth correcting a natural assumption: **Series III does not detect this from +content either.** `event_file_io.derive_record_type_from_filename()` reads the +last character of the extension (`M529LKIQ.G10H` → `H` → Histogram). Nothing +in the codebase infers record type from file content, for either family. + +And there is no obvious type field to find. The first 64 bytes of a histogram +and a waveform are byte-identical; they diverge at ~`0x0947` into wholly +different structures rather than differing by a flag. + +**The answer is the Series III pattern — generate the name.** Series III has +`blastware_filename()`, which builds a name from serial + timestamp + type. +Series IV needs the same thing, and its convention is far simpler: + +``` +_.IDF{W,H} e.g. UM12947_20260923163319.IDFW +``` + +versus Series III's `<4-char base-36 stem>`, where +the stem is base-36 of seconds-since-1985 ÷ 1296. + +All three inputs are already available on a direct download: + +| input | source | +|---|---| +| serial | `extract_binary_metadata()` — decoded from the IDF header | +| timestamp | `extract_binary_metadata()` — same | +| **type** | **the chain walk** — `SUB 0x0A` length `0x1E` = histogram, `0x00` = waveform | + +Verified against all five bench events: the generated names match the +convention of real files in the production store byte for byte. A directly +downloaded event can therefore be filed under exactly the name Thor would have +given it, and `/db/import/idf_file` needs no change at all. + +⚠ The type still comes from the *protocol*, not the payload — so a downloader +must carry it out of the chain walk. Losing it means losing the ability to +name the file correctly. ### ⚠ Unresolved: the `0x0C` peak float From c8d972f6856c843f9774f0aa1f9366c28b8c831f Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 18:35:58 -0400 Subject: [PATCH 11/43] docs(series4): the setup-write path, observed end to end Thor pushed a setup named TEST1.mmb to UM12947 while seismo_lab's TCP bridge recorded both directions. We still have not originated a write frame -- the wire format is now known, our encoder is not written. Topology worth reusing: socat shares /dev/ttyACM0 on TCP from mint-mac, seismo_lab relays Thor to it. Thor is pointed at 127.0.0.1 as if the unit were a field modem. No modem, no SIM, production Thor box untouched. The sequence is Series III's, plus one command: Thor: 5B | 41 | 08 | 2E | 1A | DA | 68->73 | 82->83 | 71->72 unit: A4 | BE | F7 | D1 | E5 | 25 | 97 8C | 7D 7C | 8E 8D All 12 device responses checksum-validate and every write is acked. Every write response SUB matches the Series III table exactly. New: * SUB 0xDA names the target .MMB file -- 256 bytes, filename null-padded, nothing else. This is why no generic file-transfer command exists: Thor names the file, then writes the ordinary config block into it. * SUB 0x41 reads the active setup's filename; SUB 0x2E reads trigger config. * Reads are single-step -- Thor asks offset=0xFFFF and skips the probe. * 0x71 writes the whole 2090-byte block in ONE frame, not Series III's three chunks. 0x69/0x74 are absent. Write-frame destuffing is `10 XX` -> `XX` uniformly, including `10 03`. Chosen by checksum, not assumption: of four candidate rules, only this one makes all four data-carrying write frames validate. 0x71's data holds 4 literal 0x03 bytes escaped as `10 03`, so escaping is mandatory for any writer. The write body IS the read body -- 0x71 and the 0xE5 response align at a fixed 11-byte shift with 1902/2090 bytes equal (91.0%). Setups are read-modify-write. The 12 differing regions are fully mapped: setup name, four 64-byte [label:22][value:42] note entries, sensor location, and the three geo trigger levels (0.3 -> 0.5 in/s) on a 48-byte channel stride. Independent confirmation of the geo LSB: each channel block carries float32BE 3.10308 at label+24. 3.10308/10000 = 0.000310308 = _GEO_LSB_IPS to 8 figures, and 10.0/3.10308*10000 = 32226.046 = the 32226.05 full scale. That value was derived statistically from 991,415 rounding constraints in v0.30.0; the unit reports it directly. It is exactly half Series III's 6.206053, so the ADC runs 10,000 counts per volt. Do NOT retune _GEO_LSB_IPS -- this corroborates it. The `offset` field is NOT a single length formula: two frames are len, two are len+2, and Series III's data[1]+2 reproduces neither. Recorded as observed constants the device accepted; pinning the rule needs a capture with differently-sized payloads. This doc has been wrong once by inferring a length field -- not inferring this one. Also adds scratch/mm_frame_parse.py, because S3FrameParser cannot see Micromate responses at all (it scans for DLE+STX; Micromate responses start at a bare STX). That is why the first pass at this capture looked like 12 unanswered requests. 24/24 frames parse with 0 bad checksums. Stale claims corrected: the "write half is not yet attempted" note, the "empty unit" limitation (5 events since 2026-09-23), and the unsafe-until-agreed list, which now distinguishes observed from exercised. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 241 +++++++++++++++++++++++++-- scratch/mm_frame_parse.py | 202 ++++++++++++++++++++++ 2 files changed, 431 insertions(+), 12 deletions(-) create mode 100644 scratch/mm_frame_parse.py diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 8de4a36..82089db 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -428,6 +428,20 @@ memory 15,000,000 total and free (no events stored). Blastware, `0x03` Thor — where Series III has a constant `0x10`. 5. **Modem serial rate is 115200**, not 38400 (per TMI provisioning practice; not independently verified here). +6. **Reads are single-step** — Thor asks for `offset = 0xFFFF` and gets the + whole block; Series III always probes first. (2026-09-24) +7. **Setup writes are preceded by `SUB 0xDA`**, which names the target `.MMB` + file. Series III has no equivalent — it has one config, not named files. +8. **The compliance block is written in ONE `0x71` frame**, not Series III's + three chunks. +9. **`0x69` / `0x74` (waveform data write) are absent** from a setup push. +10. **Six channel blocks** (`Tran`/`Vert`/`Long`/`Mic`/`LMic`/`SMic`) on a + 48-byte stride, against Series III's four. +11. **The notes block is four fixed-width `[label:22][value:42]` entries** on a + 64-byte stride, with different labels — not Series III's `label: value` + scan targets. +12. **The geophone scale factor is `3.10308`**, exactly half Series III's + `6.206053`, with an ADC of 10,000 counts per volt. --- @@ -649,8 +663,192 @@ Note `LMic` / `SMic` — linear and sound-level microphone variants that Series III does not have. This is the **read half of setup management**, and it means a setup can be -round-tripped: read the active config, modify, write it back. The write half -is not yet attempted. +round-tripped: read the active config, modify, write it back. ✅ **The write +half was observed on 2026-09-24** — see *The write path* below, which confirms +the round-trip: the written block is the read block, 91% byte-identical. + +## The write path — observed end to end (2026-09-24) + +⚠ **We have still never sent a write command to a unit.** Thor did every write +below; seismo_lab's TCP bridge sat between Thor and the unit and recorded both +directions. Capture: `bridges/captures/be12599-diag/9-24-26 - micromate2/` +(`raw_bw_*` = Thor, `raw_s3_*` = unit), UM12947 on the USB **PC** port, relayed +to TCP by `socat` on mint-mac. Operation: push a setup named `TEST1.mmb`. + +**All 12 device responses checksum-validate and every write is acked.** + +### The sequence + +``` +Thor: 5B │ 41 │ 08 │ 2E │ 1A │ DA │ 68 → 73 │ 82 → 83 │ 71 → 72 +unit: A4 │ BE │ F7 │ D1 │ E5 │ 25 │ 97 8C │ 7D 7C │ 8E 8D + └──────── reads ────────┘ └──────────── writes ────────────┘ +``` + +Series III, for comparison: `68→73 │ 71×3→72 │ 82→83 │ 69→74→72`. + +Same write SUBs, and **every response SUB matches the Series III table +exactly** (`68`→`97`, `73`→`8C`, `82`→`7D`, `83`→`7C`, `71`→`8E`, `72`→`8D`). +Three differences: + +- **`SUB 0xDA` is new** and comes first — see below. +- **`0x71` is a single write, not three chunks.** Series III splits the + compliance block into 1027 + 1055 + remainder; the Micromate takes all + 2,090 bytes in one frame. +- **`0x69` / `0x74` (waveform data write) do not appear at all.** + +Every write ack is a 16-byte zero-data frame, same shape as Series III's. + +### Three new read commands + +| SUB | rsp | payload | what it carries | +|---|---|---|---| +| `0x41` | `0xBE` | 271 B | **the active setup's file name**, e.g. `TEST1.mmb` | +| `0x2E` | `0xD1` | 44 B | trigger-config block; mirrors what `0x82` writes | +| `0xDA` | `0x25` | 272 B (write) | **declares the target setup file** — see below | + +### `SUB 0xDA` — name the setup file you are about to write + +Data is exactly **256 bytes: the file name, null-padded, nothing else.** + +``` +54 45 53 54 31 2e 6d 6d 62 00 00 … "TEST1.mmb" + 247 × 0x00 +``` + +This is the missing link in *setups are files*: there is no generic +file-transfer command because there does not need to be one. Thor names the +target file, then writes the ordinary config block into it. The unit acks with +`0x25` before any config bytes are sent. + +### Reads are single-step — no probe + +Series III sends every read twice (probe at `offset=0x00` to learn the length, +then a data step). Thor **skips the probe** and asks for +**`offset = 0xFFFF`**, getting the whole block in one response: + +``` +41 02 10 10 00 1a 00 ff ff 00 … SUB 0x1A, offset 0xFFFF +→ 2,103-byte response +``` + +`POLL` is the exception — it uses `offset = 0x0030` (48), its data length. + +This does not contradict the probe response documented above; the probe still +works and still reports its length at `payload[8:10]`. Thor simply does not +need it. + +### Write-frame destuffing — `10 XX` → `XX`, uniformly + +Only the leading `BW_CMD` is doubled (`10 10`); after that **every `10 XX` pair +on the wire destuffs to `XX`**, including `10 03`. + +This was settled by checksum, not by assumption. Four candidate rules were +tested against all four data-carrying write frames; **only this one makes all +four checksums validate**: + +| rule | `0xDA` | `0x68` | `0x82` | `0x71` | +|---|---|---|---|---| +| **`10 XX` → `XX`** | ok | **ok** | ok | **ok** | +| only `10 03` → `03` | ok | BAD | ok | BAD | +| `10 10`→`10`, `10 03`→`03` | ok | BAD | ok | BAD | +| nothing collapses | ok | BAD | ok | BAD | + +`0x71`'s data contains **4 literal `0x03` bytes**, escaped as `10 03` on the +wire. A writer that does not escape `0x03` will emit a frame the device +terminates early — this is the same defensive ETX escaping Blastware does, and +it is mandatory, not optional. + +Checksum is plain SUM8 of the destuffed payload; the DLE-aware form gives the +same answer once destuffing is correct, so it does not discriminate. + +### The `offset` field is a per-command constant + +| SUB | data bytes | `offset` | note | +|---|---|---|---| +| `0xDA` | 256 | `0x0100` = 256 | = the name-field size | +| `0x68` | 88 | `0x005A` = 90 | **identical to Series III's documented value** | +| `0x82` | 28 | `0x001C` = 28 | **identical to Series III's documented value** | +| `0x71` | 2090 | `0x082C` = 2092 | = the length `SUB 0x1A` reports | + +⚠ **There is no single length formula** — two are `len`, two are `len + 2`, and +Series III's `data[1] + 2` rule does not reproduce either. Treat these as +observed constants the device accepted. Pinning the actual rule needs a second +capture whose payloads differ in size. (This document has already been wrong +once by inferring a length field; do not infer this one.) + +### The write body **is** the read body + +The `0x71` write payload and the `0xE5` read response align at a fixed +**11-byte shift** with **1902/2090 bytes equal (91.0%)** — the remaining 9% is +exactly what was edited. + +**So a setup is read-modify-write**, the same shape as Series III, and a write +client does not need to synthesise a config block from scratch. + +### Field map, from the read/write diff + +Twelve differing regions, all accounted for. Offsets are into the `0x71` +**data** section (= the read response's data + 11). + +| offset | size | field | this capture | +|---|---|---|---| +| `0x0000` | 3 | block length echo | `08 2a` (=2090) → zeroed on write | +| `0x0009` | 1 | unidentified | `3c` → `02` | +| `0x0013` | 1 | unidentified | `00` → `40` | +| `0x002A` | 44 | **setup file name** | → `TEST1.mmb` | +| `0x0090` | 42 | note 1 value — `Location` | → `Test Location 1 - 1234 electric boogaloo` | +| `0x00D0` | 42 | note 2 value — `Client` | → `TMI` | +| `0x0110` | 42 | note 3 value — `Company` | → `ServersDownLabs` | +| `0x0150` | 42 | note 4 value — `General Notes` | → `Hopefully this works!` | +| `0x0428` | 16 | sensor location | → `Test Config Send` | +| `0x06CA` | 4 | **Tran trigger level** float32 BE | `0.3` → `0.5` in/s | +| `0x06FA` | 4 | **Vert trigger level** | `0.3` → `0.5` in/s | +| `0x072A` | 4 | **Long trigger level** | `0.3` → `0.5` in/s | + +**Notes block: four entries on a 64-byte stride**, each `[label: 22][value: 42]`, +labels at `0x008A + 64n`. The labels are `Location`, `Client`, `Company`, +`General Notes` — **not** Series III's `Project:` / `Client:` / `User Name:` / +`Seis Loc:`, and they are fixed-width fields, not `label: value` pairs. + +**Channel blocks: six, 48 bytes each**, from `0x06BC` — `Tran`, `Vert`, `Long`, +`Mic`, `LMic`, `SMic`. Trigger level sits at **label + 30**. + +### The geophone scale factor is in the config block — and it confirms our LSB + +Each geo channel block carries a float32 BE at **label + 24**: + +``` +40 46 98 dd = 3.10308003… +``` + +Which closes a loop from the file-decode work: + +``` +3.10308003… / 10000 = 0.000310308 = _GEO_LSB_IPS, to 8 figures +10.0 / 3.10308003… × 10000 = 32226.046 = the 32226.05 full scale +``` + +`_GEO_LSB_IPS` was derived statistically — intersecting 991,415 rounding +constraints from Thor's own CSV exports (see `idf_protocol_reference.md`). +**The unit reports the constant directly**, and it agrees. That upgrades the +value from a fit to a reading, and explains the odd full-scale count: the +Micromate's ADC is **10,000 counts per volt**, and 3.10308 in/s per volt is +**exactly half** Series III's documented `6.206053` (ratio 1.99997). + +Do **not** retune `_GEO_LSB_IPS` — this is corroboration, not a correction. + +### Still unknown on the write path + +- **What `0x68` and `0x82` actually contain.** Both were written with + near-zero payloads here and nothing in them changed, so no field is located. + Series III maps backlight/power-save/LCD-cycle into `0x68`; unverified here. +- **The three header bytes** at data `0x0000`/`0x0009`/`0x0013`. +- **Whether a *new* setup file can be created**, or only an existing one + overwritten. `0xDA` named a file that did not previously exist and the unit + accepted it — but we did not confirm on the unit's screen that `TEST1` now + exists as a selectable setup. Worth checking on the device. +- **Scheduler / call-home writes.** `callhome.MMB` is a file too, so it may go + through the same `0xDA` + block-write shape with a different target name. ## Static analysis of the firmware (2026-09-23, solo session) @@ -789,10 +987,17 @@ its maximum should equal the loudest event in it. ## ⚠ Untested and unsafe-until-agreed -Nothing below has been sent to a unit, and nothing should be without an -explicit decision: +**Nothing below has been sent to a unit by us, and nothing should be without an +explicit decision.** -- **Writes** (`0x68`–`0x83`), **call-home write** (`0x7E`/`0x7F`) +Note the distinction introduced on 2026-09-24: the setup-write sequence +(`0xDA`, `0x68`/`0x73`, `0x82`/`0x83`, `0x71`/`0x72`) has now been **observed**, +because Thor performed it while we recorded. Observed is not the same as +exercised — **we have still never originated a write frame.** The wire format +is known; our encoder is unwritten and unproven. + +- **Writes** (`0x68`–`0x83`) — format now known, never sent by us +- **Call-home write** (`0x7E` / `0x7F`) — not observed at all - **Erase** (`0xA3` / `0xA2`) - **Start / stop monitoring** (`0x96` / `0x97`) - `0x1F` (advance event pointer) — non-destructive on Series III but it does @@ -800,7 +1005,8 @@ explicit decision: Also unknown: -- Whether `0x10` bytes inside request params need stuffing +- Whether `0x10` bytes inside **request params** need stuffing. (Write-frame + **data** stuffing *is* now settled — `10 XX` → `XX`; see *The write path*.) - Everything about the **call-home session** — the device-initiated direction has not been observed at all. Specifically: how a unit announces itself, and **how it learns an event was accepted so it stops re-sending it.** @@ -812,14 +1018,25 @@ Also unknown: ## Session provenance Unit **UM12947**, firmware **`11.0CB`** (Blastware line), on the bench via -USB, **zero events stored** (memory free == total). Read commands only. -Every response in this document was checksum-validated. +USB. Every response in this document was checksum-validated. + +Three sittings, all on the same unit: + +| date | state | what was exercised | +|---|---|---| +| 2026-09-23 | zero events stored | read commands, direct from Python | +| 2026-09-23 | 5 events (4 waveform, 1 histogram) | the event chain + `0x5A` | +| 2026-09-24 | 5 events | **Thor pushing a setup**, via a recording relay | + +The 2026-09-24 sitting used a different topology worth noting, because it is +reusable: `socat` on mint-mac shares `/dev/ttyACM0` on TCP, seismo_lab's TCP +bridge relays Thor to it and records both directions. Thor is configured with +the unit at `127.0.0.1:` exactly as if it were a field modem. No +modem, no SIM, and **nothing on the production Thor box is touched.** ⚠ Firmware is the single biggest caveat on this document. Every finding here is from one unit on the Blastware build. A `11.0BD` unit has not been touched. -An empty unit is a real limitation: `0x08`, `0x1E`, `0x0A` and `0x06` all have -event-dependent payloads that could not be exercised. Recording a couple of -events on the bench unit would unlock the entire event-walk half of the -protocol. +✅ The original "empty unit" limitation is gone — `0x08`, `0x1E`, `0x0A` and +`0x06` were all exercised against 5 stored events on 2026-09-23. diff --git a/scratch/mm_frame_parse.py b/scratch/mm_frame_parse.py new file mode 100644 index 0000000..c0e6ebc --- /dev/null +++ b/scratch/mm_frame_parse.py @@ -0,0 +1,202 @@ +#!/usr/bin/env python3 +""" +mm_frame_parse.py — parse Micromate (Series IV) frames out of a seismo_lab +raw capture pair. + +Why this exists +--------------- +`minimateplus.framing.S3FrameParser` cannot see Micromate traffic. It locates +frames by scanning for `DLE STX`, and a Micromate response has **no leading +DLE** — it starts at a bare `STX`. It also expects `payload[1] == 0x10`, where +the Micromate sends `0xC5` (Blastware firmware) or `0x03` (Thor firmware). + +The practical consequence, seen on the 9-24-26 setup-push capture: the +Blastware-side requests parse fine (Thor emits Series III request frames), but +**every device response is silently dropped or mis-framed** — so a capture that +actually contains 12 acked writes looks like 12 unanswered requests. + +Destuffing +---------- +One rule covers both directions: after the leading doubled `BW_CMD`, every +`10 XX` pair on the wire destuffs to `XX`. That includes `10 03` — Thor +escapes literal `0x03` bytes in write data so they are not mistaken for ETX, +exactly as Blastware does. + +That rule was chosen by evidence, not assumption: of the four candidates tried +against the 9-24-26 capture's four data-carrying write frames, it is the only +one under which all four checksums validate. See +`docs/micromate_protocol_reference.md` → *The write path*. + +Usage +----- + python scratch/mm_frame_parse.py + python scratch/mm_frame_parse.py + python scratch/mm_frame_parse.py --dump 0x71 +""" + +from __future__ import annotations + +import argparse +import sys +from pathlib import Path + +DLE, STX, ETX, ACK = 0x10, 0x02, 0x03, 0x41 + +# Request SUB -> short name. Series III names where they carry over; the +# Series IV additions are marked. +SUBNAME = { + 0x01: "DEVICE_INFO", + 0x06: "STORAGE_RANGE", + 0x08: "EVENT_INDEX", + 0x0A: "WAVEFORM_HDR", + 0x0C: "WAVEFORM_REC", + 0x15: "SERIAL", + 0x1A: "COMPLIANCE_CFG", + 0x1C: "MONITOR_STATUS", + 0x1E: "EVENT_HDR", + 0x2C: "CALLHOME_CFG", + 0x2E: "TRIGGER_CFG_READ", # Series IV + 0x3E: "OPERATOR", + 0x41: "SETUP_NAME_READ", # Series IV + 0x5A: "BULK_DOWNLOAD", + 0x5B: "POLL", + 0x68: "EVENT_INDEX_WRITE", + 0x69: "WAVEFORM_WRITE", + 0x71: "COMPLIANCE_WRITE", + 0x72: "CONFIRM_A", + 0x73: "CONFIRM_B", + 0x74: "CONFIRM_C", + 0x82: "TRIGGER_WRITE", + 0x83: "TRIGGER_CONFIRM", + 0xDA: "SETUP_FILE_DECL", # Series IV — names the target .MMB + 0xFE: "FULL_CFG", +} + + +def destuff(blob: bytes, start: int, *, is_request: bool) -> tuple[bytes, int, int]: + """Destuff one frame starting at `start`. + + Returns (payload, checksum, index_of_terminating_ETX). `payload` excludes + the trailing checksum byte. A request frame opens `ACK STX 10 10`; a + response opens with a bare `STX`. + """ + i = start + (2 if is_request else 1) + out = bytearray() + if is_request: + # The doubled BW_CMD is the one guaranteed stuffed byte. + if blob[i : i + 2] != bytes([DLE, DLE]): + raise ValueError(f"@0x{start:04x}: request does not open with 10 10") + out.append(DLE) + i += 2 + while i < len(blob): + b = blob[i] + if b == DLE and i + 1 < len(blob): + out.append(blob[i + 1]) + i += 2 + continue + if b == ETX: + break + out.append(b) + i += 1 + if len(out) < 2: + raise ValueError(f"@0x{start:04x}: frame too short") + return bytes(out[:-1]), out[-1], i + + +def frames(blob: bytes, *, is_request: bool): + """Yield (offset, payload, chk, checksum_kind).""" + i, n = 0, len(blob) + while i < n: + if is_request: + if not (blob[i] == ACK and i + 1 < n and blob[i + 1] == STX): + i += 1 + continue + elif blob[i] != STX: + i += 1 + continue + try: + payload, chk, end = destuff(blob, i, is_request=is_request) + except ValueError: + i += 1 + continue + sum8 = sum(payload) & 0xFF + dle_aware = (sum(b for b in payload if b != DLE) & 0xFF) + if sum8 == chk: + kind = "SUM8" + elif dle_aware == chk: + kind = "DLE-aware" + else: + kind = "BAD" + yield i, payload, chk, kind + i = end + 1 + + +def describe(payload: bytes, is_request: bool) -> str: + if len(payload) < 3: + return "??" + sub = payload[2] + if is_request: + return SUBNAME.get(sub, f"SUB_{sub:02X}") + req = 0xFF - sub + return "rsp<-" + SUBNAME.get(req, f"SUB_{req:02X}") + + +def report(path: Path, *, is_request: bool, dump_sub: int | None) -> None: + blob = path.read_bytes() + side = "Thor" if is_request else "unit" + print(f"== {side:4} {path.name} ({len(blob)} bytes)") + n_bad = 0 + for idx, (off, p, chk, kind) in enumerate(frames(blob, is_request=is_request)): + if kind == "BAD": + n_bad += 1 + sub = p[2] if len(p) > 2 else -1 + flags = p[1] if len(p) > 1 else -1 + # Requests carry offset at payload[4:6]; responses page at [3:5]. + word = int.from_bytes(p[4:6] if is_request else p[3:5], "big") + data = len(p) - 16 if is_request else max(len(p) - 5, 0) + print( + f" [{idx:2}] @0x{off:04x} payload={len(p):5} data={data:5} " + f"flags=0x{flags:02x} SUB=0x{sub:02x} {describe(p, is_request):18} " + f"{'offset' if is_request else 'page'}=0x{word:04x} chk={kind}" + ) + if dump_sub is not None and sub == dump_sub: + body = p[16:] if is_request else p[5:] + print(f" ---- data ({len(body)} bytes) ----") + for o in range(0, len(body), 16): + chunk = body[o : o + 16] + txt = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk) + print(f" {o:06x} {chunk.hex(' '):<47} |{txt}|") + print(f" -- {idx + 1} frames, {n_bad} bad checksum\n") + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("paths", nargs="+", + help="a capture directory, or raw_bw.bin and raw_s3.bin") + ap.add_argument("--dump", default=None, + help="hex-dump the data section of this SUB (e.g. 0x71)") + args = ap.parse_args() + + dump_sub = int(args.dump, 0) if args.dump else None + + if len(args.paths) == 1 and Path(args.paths[0]).is_dir(): + d = Path(args.paths[0]) + bw = sorted(d.glob("raw_bw_*.bin")) + s3 = sorted(d.glob("raw_s3_*.bin")) + if not bw or not s3: + print(f"{d}: need one raw_bw_*.bin and one raw_s3_*.bin", file=sys.stderr) + return 2 + pairs = [(bw[0], True), (s3[0], False)] + elif len(args.paths) == 2: + pairs = [(Path(args.paths[0]), True), (Path(args.paths[1]), False)] + else: + ap.error("pass a capture directory, or exactly two .bin files") + + for path, is_request in pairs: + report(path, is_request=is_request, dump_sub=dump_sub) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From d33e2d85be50a20df86ed8f31f59868e9b858401 Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 18:48:17 -0400 Subject: [PATCH 12/43] docs(series4): 0xDA creates setup files -- confirmed on the device TEST1.mmb did not exist on UM12947 before the push. After it, the setup is present in the unit's own setup list and selected as active -- verified on the Micromate's screen, not inferred from the ack. This was the last open question about whether Series IV setup management is reachable without Thor. It is: 0x41 read name, 0x1A read block, 0xDA name the target, 0x71 -> 0x72 write it back. No file-transfer primitive is needed and the target file does not have to exist. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 23 +++++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 82089db..b389601 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -843,13 +843,28 @@ Do **not** retune `_GEO_LSB_IPS` — this is corroboration, not a correction. near-zero payloads here and nothing in them changed, so no field is located. Series III maps backlight/power-save/LCD-cycle into `0x68`; unverified here. - **The three header bytes** at data `0x0000`/`0x0009`/`0x0013`. -- **Whether a *new* setup file can be created**, or only an existing one - overwritten. `0xDA` named a file that did not previously exist and the unit - accepted it — but we did not confirm on the unit's screen that `TEST1` now - exists as a selectable setup. Worth checking on the device. - **Scheduler / call-home writes.** `callhome.MMB` is a file too, so it may go through the same `0xDA` + block-write shape with a different target name. +### ✅ `0xDA` CREATES setup files (confirmed on the device, 2026-09-24) + +`TEST1.mmb` did not exist on the unit before the push. Afterwards it is +**present in the unit's setup list and selected as the active config** — +verified on the Micromate's own screen, not inferred from the ack. + +So the four commands below are the complete setup-management path, and a +homebrew client needs no file-transfer primitive and no pre-existing target: + +``` +0x41 read the active setup's name +0x1A read its config block ─┐ read +0xDA name the target .MMB ─┘ modify +0x71 → 0x72 write the block back write (creates the file if absent) +``` + +That closes the last open question about whether Series IV setup management is +reachable from outside Thor. It is. + ## Static analysis of the firmware (2026-09-23, solo session) ### Architecture From b5e34ce8aec078d1761ba206e5b6507a4301c854 Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 18:56:20 -0400 Subject: [PATCH 13/43] docs(series4): overwrite is protocol-identical to create -- no handshake The firmware carries `Overwrite File`, `MFS FILE EXISTS` and `Cannot be Overwritten`, which suggested the wire path might negotiate an overwrite. It does not. Those strings belong to the on-device Save screen (CSaveSetupFile), not the protocol. A second Thor push to TEST1.mmb -- a name that now existed, and which SUB 0x41 confirmed was the ACTIVE setup -- produced an identical sequence: * same 12 SUBs in the same order, same offset fields * 0xDA / 0x68 / 0x82 data byte-identical * 0x71 differs in exactly 18 bytes = the one edited note string * all seven write acks identical and still all-zero * no dialog on Thor Verified on the unit: the edited General Notes string is present in the setup on the device. The write applied silently and in place, and being the active setup bought it no protection. Two consequences recorded: * A writer needs no exists-check and no overwrite negotiation. * We have never seen this protocol report a FAILED write -- acks are all-zero across create and overwrite alike. Do not treat a zero ack as proof a write applied; read back with 0x41 + 0x1A and compare. And a remote push to a monitoring unit's active setup changes what it is recording with, unprompted -- gating that belongs in SFM, because the device will not do it. Still untested: overwriting a non-active setup, and factory.MMB. Neither blocks a writer. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 48 ++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index b389601..e55a682 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -865,6 +865,54 @@ homebrew client needs no file-transfer primitive and no pre-existing target: That closes the last open question about whether Series IV setup management is reachable from outside Thor. It is. +### ✅ Overwriting is protocol-identical to creating (2026-09-24) + +The firmware carries strings that suggest an overwrite handshake: + +``` +Overwrite File MFS FILE EXISTS +Cannot be Overwritten Can Not Delete Active Setup File +``` + +**There is no such handshake on the wire.** A second push to `TEST1.mmb` — a +name that now existed, and which `SUB 0x41` confirmed was the *active* setup — +produced a byte-for-byte identical command sequence: + +| | create | overwrite | +|---|---|---| +| SUBs, and their order | `5B 41 08 2E 1A DA 68 73 82 83 71 72` | **identical** | +| every `offset` field | — | **identical** | +| `0xDA` / `0x68` / `0x82` data | — | **0 bytes differ** | +| `0x71` data | — | 18 bytes differ = the one edited note | +| all seven write acks | 11 zero bytes | **identical, still all-zero** | + +No extra command, no confirm step, no error status, and **no dialog on Thor**. +Those firmware strings belong to the on-device Save screen (the +`CSaveSetupFile` UI class), not to the protocol. + +**The overwrite was verified on the unit itself** — the edited `General Notes` +string is present in the setup on the device, so the write applied, silently and +in place. + +This is the case that mattered most, and it landed the right way round: the +target was the **active** setup, which is what a real remote config push would +hit. Being active bought it **no protection** — it was overwritten directly. +A writer therefore needs no exists-check and no overwrite negotiation. + +⚠ That cuts both ways. A remote push to the active setup of a **monitoring** +unit changes the config it is recording with, with no prompt, no warning and no +distinguishable ack. Whatever SFM eventually exposes should gate this on the +operator, not on the protocol — the device will not stop anyone. + +⚠ Two overwrite cases remain untested: a **non-active** setup file, and +`factory.MMB`, which `Cannot be Overwritten` probably guards. Neither blocks a +writer — the active setup is the one worth pushing to. + +⚠ Note also that the write acks are **all-zero in every case observed**, across +a create and an overwrite. We have never seen this protocol report a *failed* +write, so **do not treat a zero ack as proof a write was applied.** Read the +config back and compare; `SUB 0x41` plus `SUB 0x1A` make that cheap. + ## Static analysis of the firmware (2026-09-23, solo session) ### Architecture From 5400f1bef7afbd47156c58ed3bddb852cc42e087 Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 19:18:58 -0400 Subject: [PATCH 14/43] docs(series4): monitoring control, the setup-list walk, and the device clock Thor started monitoring, listed the unit's setups and stopped monitoring while seismo_lab recorded. 40 requests, 40 responses, every checksum valid. As before, Thor did all of it -- we have still never originated any of these. Confirmed identical to Series III: * SUB 0x96 start monitoring -> ack 0x69 * SUB 0x97 stop monitoring -> ack 0x68 Both bare frames, no params, no data. These were on the unsafe-until-agreed list as entirely unobserved; they are now observed but still never sent by us. Erase (0xA3/0xA2) is now the only genuinely untouched destructive path. NOT identical to Series III, and worth not reusing constants for: * The monitoring flag is SUB 0x1C data[12] = 0x0E monitoring / 0x00 idle. Series III uses 0x10. * SUB 0x49 -> 0xB6 is a second, cheaper monitoring indicator at data[11] (0x02 monitoring / 0x00 idle) in a 21-byte response rather than 60. Thor puts it in its preamble before every operation, so it is the routine check. New this capture: * SUB 0x1C carries the DEVICE CLOCK at data[13:21] -- day, month, year (u16 BE), hour, minute, second. Verified against the capture's own wall time. Nothing else read so far reports the unit's time. data[17] remains unidentified (32 monitoring, 100 idle) -- not claimed as anything. * Memory total is exactly 15,000,000 bytes; free dropped 4,096 bytes across a ~70s monitoring session, so free memory is not stable to compare against. * SUB 0x3F/0x40 walk the setup-file list, the same first/next shape as Series III's 1E/1F event walk. 0x3F -> 0xC0 first, 0x40 -> 0xBF next, terminating on an empty name. 23 setups on this unit. * Setup records carry ONLY the name -- 11-byte header, null-terminated name, zero padding. There is no active-setup flag; the header is byte-identical on every record including the terminator. The active setup is identified solely by SUB 0x41, which uses the same record format. The asterisk on the unit's screen is UI decoration, not a field. * TEST1.mmb, created over the wire earlier today, appears in the list and is what 0x41 reports as active -- a written setup becomes a real enumerable file. Also: Thor greys out send-to-unit while a unit is monitoring, and transmits nothing (this capture contains no 0xDA or 0x71). That is Thor policy, not a device refusal -- nothing suggests the Micromate would reject it, and a push to the active setup overwrites silently. Thor is guarding the footgun the protocol leaves open, and any client we write should do the same. Checking 0x49 data[11] first makes that cheap. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 124 ++++++++++++++++++++++++++- 1 file changed, 122 insertions(+), 2 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index e55a682..6350d69 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -442,6 +442,13 @@ memory 15,000,000 total and free (no events stored). scan targets. 12. **The geophone scale factor is `3.10308`**, exactly half Series III's `6.206053`, with an ADC of 10,000 counts per volt. +13. **The monitoring flag is `0x0E`**, where Series III uses `0x10` — and it + sits at `SUB 0x1C` `data[12]`. A second indicator lives at `SUB 0x49` + `data[11]` (`0x02` monitoring), which Series III has no equivalent of. +14. **Setup files are enumerated** with a `0x3F`/`0x40` first/next walk. + Series III has one config and nothing to enumerate. +15. **`SUB 0x1C` carries the device clock** (day/month/year/h/m/s at + `data[13:21]`). Nothing else read so far reports the unit's own time. --- @@ -913,6 +920,118 @@ a create and an overwrite. We have never seen this protocol report a *failed* write, so **do not treat a zero ack as proof a write was applied.** Read the config back and compare; `SUB 0x41` plus `SUB 0x1A` make that cheap. +## Monitoring control and the setup list (2026-09-24) + +Capture: `bridges/captures/9-24-26 - micromate2/*_turn_on_monitormode_*`. +Thor started monitoring, listed the unit's setups, and stopped monitoring. +**40 request frames and 40 responses, every checksum valid.** + +### Thor's per-operation preamble + +Thor re-runs this before *every* operation — three times in this one capture: + +``` +POLL (0x5B) → SERIAL (0x15) → 0x49 → POLL (0x5B) → +``` + +A client should mirror it. Note `POLL` here carries `offset = 0x0030` (its data +length), not `0xFFFF` — `POLL` is the one read Thor still addresses by length. + +### `SUB 0x96` / `0x97` — start and stop monitoring ✅ + +Identical to Series III, including the acks: + +| request | ack | effect | +|---|---|---| +| `0x96` | `0x69` | **start monitoring** | +| `0x97` | `0x68` | **stop monitoring** | + +Both are bare frames — no params, no data — and both ack with the usual 16-byte +zero-data response. Thor follows each with a `SUB 0x1C` status read to confirm. + +### `SUB 0x1C` — monitor status, 55-byte data + +``` +data[12] monitoring flag 0x0E monitoring / 0x00 idle +data[13] day +data[14] month +data[15:17] year, uint16 BE +data[17] ⚠ unidentified — 32 while monitoring, 100 when idle +data[18] hour ─┐ device clock, verified against the capture's +data[19] minute │ own wall time (19:12:25 → 19:13:34 EDT) +data[20] second ─┘ +data[-8:-4] memory total, uint32 BE = 15,000,000 bytes exactly +data[-4:] memory free, uint32 BE +``` + +⚠ **The monitoring flag is `0x0E`, not Series III's `0x10`.** Do not reuse the +Series III constant. Only five bytes differ between the monitoring and idle +responses: the flag, `data[17]`, the clock, and the memory-free field. + +Memory free dropped by exactly **4,096 bytes** across the ~70-second monitoring +session — monitoring allocates as it runs, so free memory is not a stable value +to compare against. + +The device clock is free here, on a command Thor already sends. That is worth +having: nothing else read so far reports the unit's own time. + +### `SUB 0x49` → `0xB6` — a cheap state check + +16-byte data, in Thor's preamble before every operation: + +``` +05 00 00 00 00 00 00 00 00 00 00 [ST] e8 00 0b 00 + ↑ data[11]: 0x02 monitoring, 0x00 idle +``` + +**A second monitoring indicator, in a 21-byte response instead of `0x1C`'s 60.** +Different encoding from `0x1C`'s flag (`0x02` vs `0x0E`), so they are separate +fields, not the same byte read twice. For a polling client this is the cheaper +of the two, and Thor evidently treats it as the routine one. + +### `SUB 0x3F` / `0x40` — walking the setup-file list ✅ + +The same first/next shape as Series III's `1E`/`1F` event walk, applied to +setup files: + +| request | ack | meaning | +|---|---|---| +| `0x3F` | `0xC0` | **first** setup record | +| `0x40` | `0xBF` | **next** setup record; repeat until the name is empty | + +Every record is 266 bytes of data and carries **nothing but the name**: + +``` +ff 00 00 00 00 00 00 00 00 00 00 00 … +└──────────── 11-byte header ─────┘ +``` + +The walk ends on a record whose name is empty — 24 records for 23 setups. +`factory.MMB` comes first, from `0x3F`. + +**There is no active-setup flag in the list.** The header is byte-identical on +every record including the terminator, and the tail is all zeros. The active +setup is identified only by `SUB 0x41`, whose response uses this exact record +format. (On the unit's own screen the active setup is marked with a trailing +asterisk — that is a UI decoration, not a field.) + +`TEST1.mmb`, created over the wire earlier the same day, appears in the list and +is what `0x41` reports as active — independent confirmation that a written setup +becomes a real, enumerable file. + +### Thor refuses to send a setup while a unit is monitoring + +Send-to-unit is **greyed out in Thor's UI** when the unit is monitoring, and +nothing is transmitted — this capture contains no `0xDA` or `0x71` at all. + +That is a Thor-side policy, not a device refusal: nothing observed suggests the +Micromate would reject the write. It squares with the earlier finding that a +push to the **active** setup overwrites silently — Thor is preventing exactly +the footgun the protocol leaves open. + +**Any client we write should adopt the same rule**, and it is now cheap to +enforce: check `0x49` data[11] (or `0x1C` data[12]) before offering a write. + ## Static analysis of the firmware (2026-09-23, solo session) ### Architecture @@ -1061,8 +1180,9 @@ is known; our encoder is unwritten and unproven. - **Writes** (`0x68`–`0x83`) — format now known, never sent by us - **Call-home write** (`0x7E` / `0x7F`) — not observed at all -- **Erase** (`0xA3` / `0xA2`) -- **Start / stop monitoring** (`0x96` / `0x97`) +- **Erase** (`0xA3` / `0xA2`) — **the only genuinely untouched destructive path** +- **Start / stop monitoring** (`0x96` / `0x97`) — observed via Thor 2026-09-24, + acks `0x69` / `0x68`; still never originated by us - `0x1F` (advance event pointer) — non-destructive on Series III but it does move device state, so it is parked with the rest From c5eed6fa46b884680978be9a0b7f34e090d815b0 Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 19:26:39 -0400 Subject: [PATCH 15/43] docs(series4): RETRACT "no file transfer" -- 0x94/0x48/0x8D/0x8E read and write by path The scheduler capture caught a generic file transfer in the open, and it invalidates a claim made earlier today. RETRACTION. The "Setups are FILES" section concluded "no generic file-transfer command is exposed on the wire", reasoning from the absence of firmware strings. Wrong. The commands exist, they carry a full filesystem path in plain ASCII, and they were the first thing Thor did when asked for the schedule: 0x94 -> 0x6B open for read 0x48 -> 0xB7 read next page, until an all-zero response = EOF 0x8D -> 0x72 open for write 0x8E -> 0x71 write the body Path is unpadded with offset = its exact length; 0x94 and 0x8D sent byte- identical payloads for "\system\schedule\schedule.dat". The 0x48 read is paged with the page number in the response header at payload[3:5]. The lesson: absence of a firmware string is not absence of a command. Dispatch is a 68K jump table and these carry no strings. The setups half of the original claim survives -- setups go via 0xDA plus the config block, not via this. Why it matters beyond the scheduler: callhome.MMB is a file too, and call-home is the last unsolved goal. Reading it may be a matter of pointing 0x94 at the right path. Recorded as a lead -- no path but schedule.dat has been tried. The schedule file: an entry carries a length-prefixed SETUP FILE NAME (0x28=40 for the name read off the unit, 0x09=9 for TEST1.mmb written back -- confirmed both directions). So a schedule entry says "at this time, load this setup", which is how the help text's "change the record mode" works, and it couples the scheduler to the setup list. Entry internals are NOT decoded and are recorded as observed bytes only -- one entry, no variation to diff. SUB 0x47 is the scheduler enable, probably: two bare frames differing only in params[7] (0x01, 0x03). Whether it sets or reads is genuinely undetermined -- both returned the same value and there is no disabled reading to compare. Flagged do-not-implement until a disable-then-enable capture settles it. A prediction made before the capture -- that the Scheduler On/Off switch would show up as a byte in the 0x71 block, since the unit's help text lists it beside Record Mode -- did not hold. 0x71, 0x68 and 0x82 are byte-identical to the previous capture. Noted that the test is weak (the operator re-sent the same config), but enabling the scheduler required no config write either way. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 146 +++++++++++++++++++++++++-- 1 file changed, 138 insertions(+), 8 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 6350d69..1e82239 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -449,6 +449,11 @@ memory 15,000,000 total and free (no events stored). Series III has one config and nothing to enumerate. 15. **`SUB 0x1C` carries the device clock** (day/month/year/h/m/s at `data[13:21]`). Nothing else read so far reports the unit's own time. +16. **There is a generic file transfer addressed by full path** — `0x94`/`0x48` + read, `0x8D`/`0x8E` write. Series III has nothing comparable; its config is + reachable only through dedicated commands. +17. **The scheduler is a separate file**, `\system\schedule\schedule.dat`, + and a schedule entry names a **setup file** to load. --- @@ -649,10 +654,21 @@ The unit's help text describes selecting, renaming and deleting them, and the event list records which setup file produced each event. Filesystem primitives exist internally (`NS_ReadFile_internal`, -`NS_WriteFile_internal`, `NS_SeekFile_internal`), but **no generic -file-transfer command is exposed on the wire** — the only file-transfer string -is `CMD_STOP_CALLHOME_FILETRANSFER`. So setups are unlikely to be pushed as -raw `.MMB` blobs over the protocol. +`NS_WriteFile_internal`, `NS_SeekFile_internal`). + +> ⚠ **RETRACTED 2026-09-24.** This section originally concluded "**no generic +> file-transfer command is exposed on the wire**", reasoning from the absence of +> firmware strings. **That was wrong.** `SUB 0x94` / `0x48` / `0x8D` / `0x8E` +> are exactly that — a read/write file transfer addressed by **full filesystem +> path** — and they were caught in the open on the first capture that touched +> the scheduler. See *The scheduler, and a generic file transfer* below. +> +> The lesson is the usual one: absence of a firmware string is not absence of a +> command. The dispatch is a 68K jump table and the commands carry no strings +> of their own. +> +> The *setups* half of the original claim survives: setups are **not** pushed as +> raw `.MMB` blobs. They go through `0xDA` + the ordinary config block. ### `SUB 0x1A` reads the whole active setup — 2,092 bytes @@ -722,10 +738,14 @@ Data is exactly **256 bytes: the file name, null-padded, nothing else.** 54 45 53 54 31 2e 6d 6d 62 00 00 … "TEST1.mmb" + 247 × 0x00 ``` -This is the missing link in *setups are files*: there is no generic -file-transfer command because there does not need to be one. Thor names the -target file, then writes the ordinary config block into it. The unit acks with -`0x25` before any config bytes are sent. +This is the missing link in *setups are files*: Thor names the target file, then +writes the ordinary config block into it. The unit acks with `0x25` before any +config bytes are sent. + +Note `0xDA` takes a **bare filename** null-padded to 256 bytes. That is a +different mechanism from the path-addressed file transfer (`0x94` / `0x8D`, +which carry `\system\schedule\schedule.dat` unpadded) — setups do not go through +the file transfer, and the file transfer is not how setups are written. ### Reads are single-step — no probe @@ -920,6 +940,116 @@ a create and an overwrite. We have never seen this protocol report a *failed* write, so **do not treat a zero ack as proof a write was applied.** Read the config back and compare; `SUB 0x41` plus `SUB 0x1A` make that cheap. +## The scheduler, and a generic file transfer (2026-09-24) + +Capture: `bridges/captures/9-24-26 - micromate2/*_read_scheduler.bin`. +25 request frames, 25 responses, every checksum valid. + +Four operations in one capture, segmented by Thor's `POLL` preamble (marks are +not written into the raw `.bin` — in TCP mode seismo_lab logs them to the +on-screen log only): + +| frames | operation | +|---|---| +| 0–4 | read the schedule off the unit | +| 5–18 | push a setup — the sequence already documented above, unchanged | +| 19–21 | write the schedule back | +| 22–24 | enable the scheduler | + +### 🔑 `SUB 0x94` / `0x48` / `0x8D` / `0x8E` — file transfer by path + +**The unit will read and write files addressed by full filesystem path.** + +``` +0x94 → 0x6B open for READ +0x48 → 0xB7 read next page; repeat until an all-zero response +0x8D → 0x72 open for WRITE +0x8E → 0x71 write the file body +``` + +The path is plain ASCII, **unpadded**, with `offset` = its exact length: + +``` +5c 73 79 73 74 65 6d 5c 73 63 68 65 64 75 6c 65 5c 73 63 68 65 64 75 6c 65 2e 64 61 74 +\system\schedule\schedule.dat 29 bytes, offset = 0x001D +``` + +`0x94` and `0x8D` sent **byte-identical** 29-byte payloads — the same path, only +the command distinguishing read from write. + +The `0x48` read is paged, and the page number comes back in the response header +at `payload[3:5]`: + +| call | page | payload | content | +|---|---|---|---| +| 1st | `0x0000` | 15 B | descriptor — `04 00 … 01 00 00 00 00 00 01` | +| 2nd | `0x0002` | 535 B | the file body | +| 3rd | `0x0000` | 11 B | all zeros = **end of file** | + +⚠ This is the single most consequential find of the session, because it is not +specific to the scheduler. **`callhome.MMB` is a file too**, and the call-home +config is the last unsolved piece of the project. Reading it should be a +matter of pointing `0x94` at the right path. That is a *lead*, not a result — +no path other than `schedule.dat` has been tried. + +### The schedule file itself + +The write body is the read body minus an 11-byte response prefix (524 vs 535), +so both describe the same structure: + +``` +03 00 00 00 0f 03 02 [namelen] [setup-file name] …zeros… 27 03 10 +``` + +- **`[namelen]` is a length prefix**: `0x28` = 40 for the 40-character name read + off the unit, `0x09` = 9 for `TEST1.mmb` written back. Confirmed both ways. +- The trailing `27 03 10` sits at `0x108` in the write body and `0x113` in the + read — the same offset once the 11-byte prefix is accounted for. + +**A schedule entry names a setup file.** That is how the help text's "change +the record mode" works: an entry says *at this time, load this setup*. It also +means the scheduler and the setup list are coupled — deleting a setup that a +schedule references is a foot-gun worth checking before we ever expose either. + +⚠ **The entry internals are NOT decoded.** One entry, one capture, no +variation to diff against. `03 00 00 00 0f 03 02` and `27 03 10` are recorded +as observed bytes, nothing more. Decoding needs schedules that differ in a +known way — two entries, or one entry at a different time of day. + +### `SUB 0x47` — the scheduler enable, probably + +Step 4 is two bare `0x47` frames with no data, differing only in `params[7]`: + +``` +req params = 00 00 00 00 00 00 00 [01] 00 00 → rsp 04 00 … 00 [01] 00 00 00 00 00 01 +req params = 00 00 00 00 00 00 00 [03] 00 00 → rsp 04 00 … 00 [03] 00 00 00 00 00 01 +``` + +The response echoes the selector and ends `01`. Its 15-byte shape is identical +to the descriptor `0x48` returns on page 0. + +⚠ **Whether `0x47` sets or merely reads is genuinely undetermined.** Both calls +returned the same trailing `01`, and there is no before/after to compare — the +scheduler was enabled in this same capture, so no "disabled" reading exists. +`params[7]` is also the token position Series III uses, which is suggestive but +not evidence. **Do not implement an enable against this until a +disable-then-enable capture settles it.** + +### The scheduler enable is NOT in the setup config block + +A prediction made before this capture — that the `Scheduler On/Off` switch would +appear as a byte in the `0x71` block, because the unit's help text lists it in +the same edit screen as Record Mode and Sample Rate — **did not hold**. + +The `0x71`, `0x68` and `0x82` payloads are **byte-identical** to the previous +capture's, all three, zero differences. The scheduler was enabled without any +of them changing. + +⚠ The test is weaker than it looks: the operator re-sent the *same* config, so +a byte-identical block is also what "nothing changed" looks like. What it does +establish is that enabling the scheduler did **not** require a config write — +whatever `0x47` does, it does alone. + ## Monitoring control and the setup list (2026-09-24) Capture: `bridges/captures/9-24-26 - micromate2/*_turn_on_monitormode_*`. From bd876b6752d460453b09c83815c8ab6ca16bd06a Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 19:44:05 -0400 Subject: [PATCH 16/43] docs(series4): separate Thor's conventions from the protocol's requirements SFM is not meant to reimplement Thor. Thor is the only available teacher of the wire protocol, but almost nothing about how it sequences its work has been shown to be required by the device, and this document was starting to blur the two -- it said "a client should mirror it" where the honest claim is "Thor does this and we have not checked whether the unit cares." Adds a table separating the two, with required / not-required / unknown marked honestly, and corrects the two places that gave Thor-copying advice. The consequential unknowns, all testable: * Thor's POLL -> 0x15 -> 0x49 -> POLL preamble before EVERY operation. Plausibly required (Series III needed POLL x3 before 5A) but Thor sends it before trivial reads too. * 0x68 and 0x82 appear in every setup push carrying near-zero payloads that changed nothing in either capture. If optional, our setup write is 3 frames instead of 7 with less to get wrong. Worth settling BEFORE building the writer. * Whether a narrower write than the full 2,090-byte block is accepted. One place Thor's shortcut is probably worse than the alternative: it reads with offset=0xFFFF and skips the probe, but the probe works and reports the length rather than making us trust a fixed one. Two reliability problems to design against, both observed rather than assumed: a zero ack does not mean a write applied (no failing write has ever been seen), and nothing warns before clobbering a monitoring unit's active setup. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 50 ++++++++++++++++++++++++++-- 1 file changed, 47 insertions(+), 3 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 1e82239..55cf545 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1064,7 +1064,9 @@ Thor re-runs this before *every* operation — three times in this one capture: POLL (0x5B) → SERIAL (0x15) → 0x49 → POLL (0x5B) → ``` -A client should mirror it. Note `POLL` here carries `offset = 0x0030` (its data +⚠ **Whether the unit requires this is untested** — see *Thor's conventions vs +the protocol's requirements*. Thor sends it before trivial reads too, so it may +be habit rather than handshake. Do not assume it is mandatory. Note `POLL` here carries `offset = 0x0030` (its data length), not `0xFFFF` — `POLL` is the one read Thor still addresses by length. ### `SUB 0x96` / `0x97` — start and stop monitoring ✅ @@ -1159,8 +1161,10 @@ Micromate would reject the write. It squares with the earlier finding that a push to the **active** setup overwrites silently — Thor is preventing exactly the footgun the protocol leaves open. -**Any client we write should adopt the same rule**, and it is now cheap to -enforce: check `0x49` data[11] (or `0x1C` data[12]) before offering a write. +The *reason* behind the rule is sound and SFM should honour it — but not +necessarily by copying the greyed-out button. Stop → push → restart as a single +operation is what an operator usually wants, and `0x49` data[11] (or `0x1C` +data[12]) makes the state check cheap either way. ## Static analysis of the firmware (2026-09-23, solo session) @@ -1297,6 +1301,46 @@ Worth noting the histogram (`…81`) and the loudest waveform (`…84`) report the histogram's single 1-minute interval spans the whole thumping session, so its maximum should equal the loudest event in it. +## Thor's conventions vs the protocol's requirements + +**SFM is not meant to reimplement Thor.** Thor is the only available teacher of +the wire protocol, but almost nothing about *how* it sequences its work has been +shown to be required by the device. Those are two different things and this +document should not blur them. + +The distinction matters because several observations above were written as +"a client should do X" when the honest statement is "Thor does X, and we have not +checked whether the unit cares." + +| Thor does this | required? | what it means for SFM | +|---|---|---| +| `POLL → 0x15 → 0x49 → POLL` before **every** operation | **unknown** | Series III needed `POLL`×3 before `5A` *specifically*, so a preamble requirement is plausible — but Thor sends this before trivial reads too. **Testable:** issue one operation cold and see if it answers. | +| Reads with `offset = 0xFFFF`, skipping the probe | **no — the probe works** | We have a genuine choice, and the probe is arguably better: it reports the length instead of making us trust a fixed one. This is the one place Thor's shortcut is probably worse. | +| Sends `0x68` + `0x82` in every setup push | **unknown** | In both setup captures these carried near-zero payloads and **changed nothing**. If they are optional, a setup write is 3 frames instead of 7, with less to get wrong. **Worth testing before we build the writer.** | +| Rewrites the whole 2,090-byte config for a one-field change | **unknown** | No narrower write has been observed. Read-modify-write is safe and known; a targeted write would be nicer but is unevidenced. | +| Reads `0x41` twice in a row (scheduler capture, frames 6 and 8) | **no** | Plainly redundant. A reminder that Thor's sequence is not a minimal one. | +| Greys out send-to-unit while monitoring | **Thor policy** | The *reason* is real — a push to the active setup overwrites silently. But "the button is grey and you figure it out" is a UX choice, not the only answer. SFM could offer stop → push → restart as one operation, which is what an operator actually wants. | +| Names the target with `0xDA` before a config write | **almost certainly required** | The unit has to know which file to write. Closest thing here to a genuine protocol requirement. | + +### What this implies for the build order + +The read path is fully known and needs no Thor-shaped decisions, so a read-only +client can be written now with confidence. **The write path should not be built +by transcribing Thor's sequence** — the `0x68`/`0x82` question above decides +whether our setup write is 3 frames or 7, and it is answerable with one capture +plus one careful experiment. + +Two reliability problems worth designing *against*, both observed rather than +assumed: + +1. **A zero ack does not mean a write applied.** Every ack seen is 11 zero + bytes, across creates and overwrites alike, and no failing write has ever been + observed. Whatever SFM does, it should read back and compare rather than + trust the ack. `0x41` + `0x1A` makes that cheap. +2. **Nothing warns before clobbering a monitoring unit's active setup.** The + device will not stop it and the ack will not distinguish it. That guard has + to live in SFM. + ## ⚠ Untested and unsafe-until-agreed **Nothing below has been sent to a unit by us, and nothing should be without an From ebcc55e2ecc0ffd4046b038aed0b4162b1075641 Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 19:46:09 -0400 Subject: [PATCH 17/43] docs(series4): the schedule<->config coupling is Thor's, not the protocol's In Thor, a schedule entry that starts monitoring forces you to attach a setup, and sending the schedule pushes that setup too, overwriting anything with the same name. The protocol requires none of it. Evidence from the scheduler capture: * The two writes are separate operations, not one transaction. The config write ends at frame 18, Thor sends a fresh POLL preamble, and only then opens the schedule at frame 20. Different commands, different paths: config 0xDA -> 0x68/0x73 -> 0x82/0x83 -> 0x71/0x72 schedule 0x8D -> 0x8E * The schedule stores a length-prefixed NAME, not a config blob. It is a reference, and a reference does not require rewriting its referent. * The config Thor pushed was already on the unit unchanged -- its 2,090-byte 0x71 payload is byte-identical to the previous capture's, zero differences. Thor spent a whole block write re-sending a setup the device already had. So SFM can, with today's protocol: enumerate setups with 0x3F/0x40, write ONLY the schedule when the referenced setup already exists, and push a config only when it is genuinely missing or deliberately edited. Common case drops from 524 + 2090 bytes to 524, and the write disappears entirely. It also removes a real hazard. Because the reference is by name, and a same-name write overwrites silently with an indistinguishable ack, Thor's pattern means scheduling something can quietly rewrite a setup that other schedules or the operator's own work depend on. Validating the reference instead of rewriting the referent avoids the class of problem. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 49 ++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 55cf545..a0d9c53 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1322,6 +1322,55 @@ checked whether the unit cares." | Greys out send-to-unit while monitoring | **Thor policy** | The *reason* is real — a push to the active setup overwrites silently. But "the button is grey and you figure it out" is a UX choice, not the only answer. SFM could offer stop → push → restart as one operation, which is what an operator actually wants. | | Names the target with `0xDA` before a config write | **almost certainly required** | The unit has to know which file to write. Closest thing here to a genuine protocol requirement. | +### The schedule↔config coupling is Thor's, not the protocol's + +In Thor, putting "start monitoring" in a schedule forces you to attach a setup, +and sending the schedule pushes that setup at the same time — overwriting +whatever already has that name. **The protocol does not require any of this.** + +The evidence, from the scheduler capture: + +- **The two writes are separate operations**, not one transaction. The config + write ends at frame 18 (`0x72`), then Thor sends a fresh `POLL` preamble, and + only then opens the schedule for writing at frame 20. Different commands, + different paths, no shared state: + + ``` + config 0xDA → 0x68/0x73 → 0x82/0x83 → 0x71/0x72 + schedule 0x8D → 0x8E + ``` + +- **The schedule stores a NAME, not a config.** A length-prefixed filename is + all an entry carries. It is a *reference*, and references do not require the + referent to be rewritten. + +- **The config Thor pushed was already on the unit, unchanged.** Its 2,090-byte + `0x71` payload is **byte-identical** to the previous capture's — zero + differences. Thor spent a full block write re-sending a setup the device + already had, purely to satisfy its own coupling. + +So the sequencing is a Thor UI decision, and a costly one. + +**What SFM can do instead, with today's protocol and nothing new:** + +1. Enumerate the unit's setups with `0x3F` / `0x40` — cheap, read-only, and it + yields the exact names a schedule may legally reference. +2. Write **only** the schedule (`0x8D` / `0x8E`) when the referenced setup is + already present. +3. Push a config only when it is actually missing, or when the operator + deliberately edited it — and say so explicitly rather than silently. + +That removes the write entirely from the common case: scheduling against a setup +that already exists becomes a 524-byte schedule write instead of 524 bytes plus +a 2,090-byte config overwrite. + +⚠ **And it removes a real hazard.** Because the reference is by name, and +because a same-name write overwrites silently with an indistinguishable ack, +Thor's pattern means *scheduling* something can quietly rewrite a setup that +other schedules — or the operator's own saved work — depend on. Nothing in the +protocol or the ack reports that this happened. Validating the reference +instead of rewriting the referent avoids the whole class of problem. + ### What this implies for the build order The read path is fully known and needs no Thor-shaped decisions, so a read-only From 8c2dacf0355c049393dbf67d0c8994ef96106913 Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 19:49:59 -0400 Subject: [PATCH 18/43] docs(series4): the schedule has a setup-less start action -- Thor never uses it The schedule<->config coupling looked like it might be a workaround for the unit crashing on a missing setup. The firmware says otherwise: there are two distinct start-monitoring actions in the scheduler's duty-cycle dispatch, and only one involves a setup file. _PSA() case DUTYCYCLE_START_MONITOR _PSA() case DUTYCYCLE_START_MONITOR_WITH_SETUP Full action set: START_MONITOR, START_MONITOR_WITH_SETUP, STOP_MONITOR, CALLHOME, SELF_CHECK, plus ON/OFF/NEXT for scheduler state. So the coupling is not a crash workaround -- Thor picks the more demanding of two available actions every time. SFM can emit START_MONITOR and skip the config. On whether a missing setup would actually break the unit: the firmware suggests graceful degradation (`Setup File Not Found`, and `Invalid parameters reset to factory default - please review setup`, a deliberate fallback). Untested, and recorded as untested. Hypothesis, flagged as such: the schedule entry's leading byte may be the action code -- the one captured entry reads `03 00 00 00 0f 03 02 [namelen][name]` and 03 would fit START_MONITOR_WITH_SETUP. One entry, nothing to diff, unverified. Names the capture that would settle it, and which is worth more than the 0x47 disable/enable test: a schedule entry with a setup-less action (stop monitoring, or call home). It would confirm or kill the action-code hypothesis and prove from the other direction that a schedule needs no config push. Also noted: DUTYCYCLE_CALLHOME -> CMD_SCHEDULE_CALL_HOME means a scheduled call-home can make a unit dial out on demand -- the one remaining lever on the unsolved call-home direction. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 47 ++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index a0d9c53..5a0cd30 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1371,6 +1371,53 @@ other schedules — or the operator's own saved work — depend on. Nothing in protocol or the ack reports that this happened. Validating the reference instead of rewriting the referent avoids the whole class of problem. +#### The schedule has a setup-less start action — Thor just never uses it + +The coupling above looked like it might be a workaround for the unit crashing on +a missing setup. The firmware says otherwise: **there are two distinct +start-monitoring actions**, and only one of them involves a setup file. + +``` +_PSA() case DUTYCYCLE_START_MONITOR +_PSA() case DUTYCYCLE_START_MONITOR_WITH_SETUP +``` + +The full duty-cycle action set, from the scheduler task's own dispatch strings: + +| action | notes | +|---|---| +| `DUTYCYCLE_START_MONITOR` | **no setup involved** — uses whatever config is loaded | +| `DUTYCYCLE_START_MONITOR_WITH_SETUP` | the variant Thor always emits | +| `DUTYCYCLE_STOP_MONITOR` | | +| `DUTYCYCLE_CALLHOME` | `send ==> CMD_SCHEDULE_CALL_HOME` | +| `DUTYCYCLE_SELF_CHECK` | `call DailySelfCheck()` | +| `DUTYCYCLE_ON` / `OFF` / `NEXT` | scheduler state, not per-entry actions | + +So the schedule↔config coupling is not a crash workaround — **Thor picks the more +demanding of two available actions, every time.** SFM can emit +`START_MONITOR` and skip the config entirely. + +On whether a missing setup *would* break the unit: the firmware suggests +graceful degradation rather than a crash — `Setup File Not Found` exists, and so +does `Invalid parameters reset to factory default - please review setup`, which +is a deliberate fallback. ⚠ **Untested**, and not worth relying on until it is. + +⚠ **Hypothesis, not a finding:** the schedule entry's leading byte may be the +action code — the one captured entry reads `03 00 00 00 0f 03 02 [namelen] +[name]`, and `03` would fit `START_MONITOR_WITH_SETUP` in an enum of the above. +Unverified; there is one entry and nothing to diff against. + +**The capture that would settle it** (more valuable than the `0x47` +disable/enable test): build a schedule whose entry is **stop monitoring** or +**call home** — an action with no setup attached. That would (a) confirm or kill +the action-code hypothesis, since a setup-less action should change the leading +byte and drop the name, and (b) prove from the other direction that a schedule +can be sent with no config push at all. + +Note also `DUTYCYCLE_CALLHOME` → `CMD_SCHEDULE_CALL_HOME`: **a scheduled +call-home is a way to make a unit dial out on demand**, which is the one +remaining lever on the unsolved call-home direction. + ### What this implies for the build order The read path is fully known and needs no Thor-shaped decisions, so a read-only From 3f58402085054517fe49d959356f04779f868dc8 Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 20:02:30 -0400 Subject: [PATCH 19/43] docs(series4): the ACH session, from the THOR manual -- and I was asking the wrong question Brian uploaded the Instantel manuals (gitignored, manuals/). The THOR Operator Manual Rev 08 settles the thing this document called the blocker for a homebrew receiver. I had written that what gates a receiver is "how it learns an event was accepted so it stops re-sending it." There is no such mechanism to find, because the unit does not track it. Per THOR manual 6.2.2.2 an ACH session is a list of SERVER-chosen actions -- Copy events, Copy monitor log, Delete events and Logs from Unit, Set Date/Time -- and "Only applies to events not previously downloaded" is computer-side bookkeeping. The manual's own warning proves copy and delete are decoupled: enable delete but disable copy and "the events and logs will be deleted without being uploaded." That is exactly the model our Series III ACH server already implements (ach_state.json high-water mark, erase as a deliberate separate step). No new mechanism is needed for Series IV. A receiver needs: accept, identify, walk the events (already solved), keep our own high-water mark, optionally erase. The ERASE OPCODES are now the only genuinely missing piece and stay on the unsafe list. Other things the manual settles: * The session is server-driven, matching the firmware state machine. Scheduled and event-triggered ACH differ: with Monitoring While Calling Home enabled, an event-triggered session will NOT delete events or sync time. So a receiver that relies on erase to avoid re-reading would silently never erase on those units -- the high-water mark has to be primary, erase an optimisation. * Session Time Out is unit-side only, which places it in callhome.MMB -- another reason to read that file with 0x94. * Units are routed by serial number with wildcards, so the serial is presented early enough for a server to dispatch on it. * THOR requires Idle for ACH setup too, confirming the greyed-out send is deliberate policy rather than a device refusal. * THOR exposes four schedule actions; the firmware has five. 6.3.2 step 8 ("A Unit Setup must exist") is THOR's own requirement, while the same section says the unit "will execute any actions in a schedule using its current settings" -- the two pull opposite ways, consistent with START_MONITOR existing and THOR never emitting it. * Schedule fields to look for when the entry is decoded: action, setup name, time, day-or-week, day selection, repeat. The captured entry has seven bytes before the name, the right order of magnitude for that list. Flagged: the manual's filter example contradicts its own table (it has UM* and MP* backwards). The table is right. Marked throughout as vendor documentation rather than observed bytes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 134 ++++++++++++++++++++++++++- 1 file changed, 129 insertions(+), 5 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 5a0cd30..08ef338 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1301,6 +1301,125 @@ Worth noting the histogram (`…81`) and the loudest waveform (`…84`) report the histogram's single 1-minute interval spans the whole thumping session, so its maximum should equal the loudest event in it. +## What the THOR manual settles about the ACH session (2026-09-24) + +Source: `manuals/723U0201 THOR Operator Manual Rev 08.pdf` §6.2, §3.6. +⚠ **This is vendor documentation, not observed bytes.** It tells us the shape of +the session and the vocabulary; it does not give opcodes. Treated as a strong +prior, not as confirmed protocol. + +### 🔑 A unit does NOT learn that an event was accepted — the server decides + +This document previously listed, as the thing *gating any homebrew receiver*: + +> how a unit announces itself, and **how it learns an event was accepted so it +> stops re-sending it.** + +**That was the wrong question.** There is no acknowledgement mechanism to +discover, because the unit is not tracking what has been collected. Per §6.2.2.2 +the ACH session is a list of **server-chosen actions**, and two of them are +independent: + +| ACH action | what it does | +|---|---| +| **Copy events** | downloads events — *"Only applies to events not previously downloaded"* | +| **Copy monitor log** | downloads the monitor log | +| **Delete events and Logs from Unit** | **explicitly** deletes them from the unit | +| **Set Date/Time** | unit synchronises its clock **from the computer** | +| Send Events / Schedule to Vision | Instantel cloud, not relevant to us | + +"Not previously downloaded" is **computer-side bookkeeping**. The unit keeps its +events until a server tells it to erase them, and the manual's own warning proves +the two are decoupled: + +> ⚠ *"If you enable 'Delete Events and Logs from Unit' but disable 'Copy Events'. +> The events and logs will be deleted without being uploaded."* + +A server that never issues the delete simply re-reads the same events forever. + +**This is exactly the model our Series III ACH server already implements** — +`ach_state.json` with `downloaded_keys` / `max_downloaded_key`, and erase as a +separate deliberate step. No new mechanism is needed for Series IV. + +So a homebrew receiver needs: accept the connection, identify the unit, walk the +events (already solved — `0x08`/`0x1E`/`0x0A`/`0x5A`), keep our own high-water +mark, and *optionally* erase. **The erase opcodes are the only genuinely missing +piece**, and they remain on the unsafe list. + +### The session is server-driven + +Which matches the firmware's state machine and its +`CMD_EXIT_CALL_HOME_DELETE_EVENTS_START_MONITORING` / +`Call Home Deleting Events` strings: the unit dials in, then waits to be told +what to do, and returns to monitoring when the server is finished. + +Scheduled ACH and event-triggered ACH behave differently (§3.6.1): + +- **Event-triggered** (ACH enabled on the unit, an event occurred) — *"Records + events + transfer data, no stopping to monitor."* Governed by **Monitoring + While Calling Home**; with MWCH enabled *"the monitor log will not be copied, + events will not be deleted, and time will not synchronize"*. +- **Scheduled** — *"Stop monitoring + transfer data (or delete events, or + synchronize)."* Monitoring stops and in-progress events complete first. + +⚠ Worth noting for SFM: with MWCH on, an event-triggered session **cannot** +delete or sync. A receiver that relies on erase to avoid re-reading will +silently never erase on those units. Our high-water mark must be the primary +mechanism, with erase as an optimisation — which is how Series III already does it. + +### Session Time Out is unit-side only + +§6.2 is explicit that THOR configures ACH on both ends *"with one exception; the +Session Time Out. This must be configured directly on the unit."* That matches +the firmware's `CallHome.SessionTimeout` field, and means it lives in +`callhome.MMB` rather than anywhere THOR reaches — one more reason to read that +file with `0x94`. + +### Unit identification is by serial number, with wildcards + +§6.2.2.2: filters match against the serial number the unit presents, `*` +wildcards allowed, best-match wins — `UM*` all Micromates, `MP*` all Minimate +Pros, `BE`/`BC` Minimate Plus variants. + +So **the unit announces its serial early enough for the server to route on it.** +`SUB 0x15` returns the serial and is in Thor's standard preamble, which is +consistent, though the inbound direction has still never been observed. + +⚠ The manual's worked example contradicts its own table — it says *"To filter +Micromate units type MP*"* and *"To filter Minimate Pro units type UM*"*, which is +backwards. The table is right. Noted so nobody copies the error. + +### Thor requires Idle for configuration + +§6.2.1 step 3: *"The Monitoring Mode must be 'Idle'."* Independent confirmation +that the greyed-out send observed on the bench is deliberate Thor policy, applied +to ACH setup as well as compliance setup — and still not evidence the device +refuses. + +### The schedule's actions, from the UI side + +§3.6 lists exactly **four** actions THOR exposes: Start monitoring, Stop +monitoring, Self-check, Auto Call Home. The firmware has **five** — it also has +a setup-less `DUTYCYCLE_START_MONITOR`. + +§3.6.2 step 8 confirms the coupling is Thor's own requirement: *"Select the +appropriate Action and the Unit Setup. (A Unit Setup must exist…)"* — while §3.6.2 +step 1 notes *"The unit will execute any actions in a schedule using its current +settings."* The two statements sit a paragraph apart and pull in opposite +directions, which is consistent with `START_MONITOR` existing and THOR never +emitting it. + +Also from §3.6: schedules are **Day or Week** ("Select Day… Select Week"), have a +**Repeat Daily / Repeat Weekly** flag, carry a Name, Description and Unit Type, +and *"Saving a schedule will only store it on the computer, it must still be sent +to the unit and enabled"* — which is why the capture shows `0x8E` (send) and +`0x47` (enable) as separate steps. + +Those are the fields to look for when the schedule entry is finally decoded: +**action, setup name, time, day-or-week, day selection, repeat.** The one +captured entry (`03 00 00 00 0f 03 02 …`) has seven bytes before the name, which +is the right order of magnitude for that field list. + ## Thor's conventions vs the protocol's requirements **SFM is not meant to reimplement Thor.** Thor is the only available teacher of @@ -1460,11 +1579,16 @@ Also unknown: - Whether `0x10` bytes inside **request params** need stuffing. (Write-frame **data** stuffing *is* now settled — `10 XX` → `XX`; see *The write path*.) -- Everything about the **call-home session** — the device-initiated direction - has not been observed at all. Specifically: how a unit announces itself, - and **how it learns an event was accepted so it stops re-sending it.** - That last question gates any homebrew receiver and cannot be answered over - USB. +- The **call-home session's bytes** — the device-initiated direction has still + not been observed at all, and cannot be over USB. + + ✅ **The second half of this item is resolved** (2026-09-24, from the THOR + manual): "how it learns an event was accepted so it stops re-sending it" was + the wrong question — **it does not learn.** Collection state is server-side + bookkeeping and erase is a separate, optional, server-issued action. See + *What the THOR manual settles about the ACH session*. What remains unknown is + the framing of the inbound session and the **erase opcodes**, not any + acknowledgement mechanism. --- From 508448e2dd8d10b5316537b47cd6a4308d1a043a Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 20:29:51 -0400 Subject: [PATCH 20/43] docs(series4): the schedule record format, named by the firmware itself A debug printf in the scheduler names the record's fields outright: SCHEDULER : _ReadRecord(%d) -> %s (Action=%u, 1/2h=%u, Day=%u, Setup="%s") [WDAY=%d] They fit the captured record, and the name-length byte anchors the alignment -- it reads 40 for the 40-character name pulled off the unit and 9 for TEST1.mmb written back, same position, both directions: [0] Action = 3 [1:4] zero (padding, or Action is a uint32) [4] 1/2h = 15 -> 30-minute resolution, 48 slots/day [5] Day = 3 [6] unidentified (WDAY?) [7] name length -- 0x28=40 read, 0x09=9 written <-- confirms the layout [8:] setup name [264] trailer 27 03 10 Slot 15 would be 07:30 counted from midnight; flagged unconfirmed because the schedule's actual time was not recorded with the capture. Six duty-cycle actions, not the five previously recorded -- there is also DUTYCYCLE_START_MONITOR_WITH_SETUP_STOP_COMPLETE. THOR exposes four. Two start variants it never offers, one needing no setup file. Strengthened the setup-less-action finding and ruled out an alternative explanation I had not considered: the Micromate has a separate Timer Mode (MODE_TIMER, Monitor Once Only, under Special Setup), so START_MONITOR could have belonged to that path. It does not -- it is a case in _PSA(), the scheduler's own dispatcher for _ReadRecord's Action field, and `_PSA() send ->> CMD_DUTYCYCLE_ START_MONITOR` shows it is live code sending a real message, not a dead case. Also: SysPref.bMonitorScheduler places the scheduler enable in system preferences, which confirms from the other side why the 0x71 block was byte-identical when the scheduler was switched on -- the flag was never going to be in the compliance config. It also suggests 0x47 is a SysPref get/set rather than anything scheduler-specific, which would explain its params[7] selector and its response shape matching 0x48's page-0 descriptor. Still a hypothesis. Names the one capture that would settle the rest: a schedule with TWO entries at different times with different actions. That yields the record stride, the action code values, and the time encoding at once. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 82 ++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 08ef338..f600869 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1490,6 +1490,88 @@ other schedules — or the operator's own saved work — depend on. Nothing in protocol or the ack reports that this happened. Validating the reference instead of rewriting the referent avoids the whole class of problem. +### The schedule record format, from the firmware's own debug printf + +``` +SCHEDULER : _ReadRecord(%d) -> %s (Action=%u, 1/2h=%u, Day=%u, Setup="%s") [WDAY=%d] +``` + +That names the fields outright, and they fit the captured record: + +| offset | field | read capture | write capture | +|---|---|---|---| +| `[0]` | **Action** | `03` | `03` | +| `[1:4]` | zero — padding, or `Action` is a uint32 | `00 00 00` | `00 00 00` | +| `[4]` | **1/2h** — half-hour slot | `0f` = 15 | `0f` = 15 | +| `[5]` | **Day** | `03` | `03` | +| `[6]` | unidentified (`WDAY`?) | `02` | `02` | +| `[7]` | **name length** | `0x28` = **40** | `0x09` = **9** | +| `[8:]` | **Setup** name | 40-char name | `TEST1.mmb` | +| `[264]` | trailer | `27 03 10` | `27 03 10` | + +**The length byte is what anchors this** — it reads 40 for the 40-character name +and 9 for `TEST1.mmb`, in the same position, in both directions. The layout is +not a guess. + +`1/2h` implies **30-minute resolution, 48 slots per day**. Slot 15 would be +07:30 if counted from midnight — plausible but ⚠ **unconfirmed**, since the +schedule's actual time was not recorded alongside the capture. One question to +the operator settles it. + +⚠ Still unknown, and all answerable with **one schedule containing two entries**: + +- the **record stride** and how many records the file holds (one record was + captured; `27 03 10` at `[264]` may be a trailer or a second record's head) +- whether `Action` is one byte or a uint32 +- the **action code values** — `03` is presumably + `DUTYCYCLE_START_MONITOR_WITH_SETUP` but the enum's base is unknown +- `[6]`, and how `Day` and `WDAY` divide the work + +### Six duty-cycle actions, not five + +The full `_PSA()` dispatch — the scheduler's own action processor, reading records +via `_ReadRecord`: + +``` +DUTYCYCLE_START_MONITOR ← no setup +DUTYCYCLE_START_MONITOR_WITH_SETUP +DUTYCYCLE_START_MONITOR_WITH_SETUP_STOP_COMPLETE ← a third start variant +DUTYCYCLE_STOP_MONITOR +DUTYCYCLE_CALLHOME +DUTYCYCLE_SELF_CHECK +``` + +**THOR exposes four.** Two start variants it never offers, one of which needs no +setup file at all. + +`_PSA() send ->> CMD_DUTYCYCLE_START_MONITOR` shows the setup-less action is live +code that the scheduler genuinely dispatches, not a dead case — it sends a real +message. And since every one of these cases sits in the function that consumes +`_ReadRecord`'s `Action` field, **a schedule record can carry it.** + +An alternative explanation was checked and ruled out: the Micromate does have a +separate *Timer Mode* (`MODE_TIMER`, `MODE_MONITOR_TIMER`, `Monitor Once Only`, +under Special Setup), so `START_MONITOR` could have belonged to that path +instead. It does not — it is in `_PSA()`, the scheduler's dispatcher. + +### The scheduler enable lives in SysPref, not in the setup + +``` +SysPref.bMonitorScheduler = %s +_Task() CMD_SET_SCHEDULE - MonitorScheduler ENABLED +_Task() CMD_SET_SCHEDULE - ! MonitorScheduler NOT Enabled +``` + +This **confirms the failed prediction above from the other side**: the +`Scheduler On/Off` switch is a *system preference*, which is why the `0x71` +compliance block was byte-identical when the scheduler was turned on. It was +never going to be in there. + +It also suggests `0x47` — the two bare frames at the end of the capture — is a +SysPref get/set rather than anything scheduler-specific, which would explain its +`params[7]` selector and its 15-byte response shape being shared with `0x48`'s +page-0 descriptor. ⚠ Still a hypothesis; the disable/enable capture settles it. + #### The schedule has a setup-less start action — Thor just never uses it The coupling above looked like it might be a workaround for the unit crashing on From 4a1cccf3f585a492facc1473fb49fdf68c74fb58 Mon Sep 17 00:00:00 2001 From: serversdown Date: Thu, 24 Sep 2026 21:40:05 -0400 Subject: [PATCH 21/43] docs(series4): the schedule file is DECODED -- two records, five confirmations The operator supplied the ground truth: entry 1 is "start monitoring TEST1" at 07:30, entry 2 is "Auto Call Home" at 19:30. That decodes the file completely. The body is two 260-byte records plus four zero bytes = 524 exactly, and every non-zero byte falls inside them: [0] Action (0 = Auto Call Home, 3 = start monitoring with setup) [1:4] padding [4] 1/2h half-hour slot, 0-47 [5] Day [6] ?? the one unidentified field [7] name length [8:260] setup name, null-padded record @ 0: Action=3 1/2h=15 -> 07:30 Day=3 [6]=2 namelen=9 "TEST1.mmb" record @260: Action=0 1/2h=39 -> 19:30 Day=3 [6]=16 namelen=0 (no setup) Five independent confirmations, no fitting: 1. Slots 15 and 39 match the stated 07:30 and 19:30 on a 30-minute grid, and 39-15 = 24 slots = exactly 12 hours. 2. The length byte reads 9 for TEST1.mmb, 40 for the long name in the read capture, and 0 for the Auto Call Home entry. 3. Auto Call Home carries NO setup name -- direct proof that a schedule entry can exist with no setup attached, which is what the earlier DUTYCYCLE_START_MONITOR finding predicted from the firmware side. 4. The 260-byte stride lands record 2's 1/2h exactly at [264]. 5. 2 x 260 + 4 = 524, the whole body, nothing left over. CORRECTION: `27 03 10` at [264] was recorded in the previous commit as a possible trailer. It is record 2's 1/2h, Day and [6] fields. I had assumed the file held one record and read the second one as padding -- the non-zero bytes were sitting there the whole time. Still open: [6] (2 on the start entry, 16 on the ACH entry -- a Repeat or Day/Week capture would isolate it), the four remaining action codes, and whether the file can hold unused record slots. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 87 ++++++++++++++++++---------- 1 file changed, 57 insertions(+), 30 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index f600869..112015f 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1011,10 +1011,10 @@ the record mode" works: an entry says *at this time, load this setup*. It also means the scheduler and the setup list are coupled — deleting a setup that a schedule references is a foot-gun worth checking before we ever expose either. -⚠ **The entry internals are NOT decoded.** One entry, one capture, no -variation to diff against. `03 00 00 00 0f 03 02` and `27 03 10` are recorded -as observed bytes, nothing more. Decoding needs schedules that differ in a -known way — two entries, or one entry at a different time of day. +✅ **The entry internals ARE decoded** — see *The schedule record format* below. +The file turned out to hold **two** 260-byte records, not one: `27 03 10` is the +second record's time/day fields, not a trailer. `1/2h` is a half-hour slot and +both times check out against what the operator entered. ### `SUB 0x47` — the scheduler enable, probably @@ -1490,42 +1490,69 @@ other schedules — or the operator's own saved work — depend on. Nothing in protocol or the ack reports that this happened. Validating the reference instead of rewriting the referent avoids the whole class of problem. -### The schedule record format, from the firmware's own debug printf +### ✅ The schedule record format — DECODED (2026-09-24) + +A debug printf in the scheduler names the fields outright: ``` SCHEDULER : _ReadRecord(%d) -> %s (Action=%u, 1/2h=%u, Day=%u, Setup="%s") [WDAY=%d] ``` -That names the fields outright, and they fit the captured record: +The captured file is **two 260-byte records plus four zero bytes** — 524 bytes +exactly. Every non-zero byte in the file falls inside those two records. -| offset | field | read capture | write capture | -|---|---|---|---| -| `[0]` | **Action** | `03` | `03` | -| `[1:4]` | zero — padding, or `Action` is a uint32 | `00 00 00` | `00 00 00` | -| `[4]` | **1/2h** — half-hour slot | `0f` = 15 | `0f` = 15 | -| `[5]` | **Day** | `03` | `03` | -| `[6]` | unidentified (`WDAY`?) | `02` | `02` | -| `[7]` | **name length** | `0x28` = **40** | `0x09` = **9** | -| `[8:]` | **Setup** name | 40-char name | `TEST1.mmb` | -| `[264]` | trailer | `27 03 10` | `27 03 10` | +``` +offset field size notes +[0] Action 1 (or uint32 LE at [0:4] — both are zero-extended here) +[1:4] padding 3 zero in both records +[4] 1/2h 1 half-hour slot, 0–47 +[5] Day 1 3 in both records +[6] ?? 1 the one unidentified field +[7] name length 1 +[8:260] Setup name 252 null-padded +``` -**The length byte is what anchors this** — it reads 40 for the 40-character name -and 9 for `TEST1.mmb`, in the same position, in both directions. The layout is -not a guess. +Decoded against the operator's own description of what they entered — +*"start monitoring TEST1 at 7:30 AM, Auto Call Home at 7:30 PM"*: -`1/2h` implies **30-minute resolution, 48 slots per day**. Slot 15 would be -07:30 if counted from midnight — plausible but ⚠ **unconfirmed**, since the -schedule's actual time was not recorded alongside the capture. One question to -the operator settles it. +| | record @0 | record @260 | +|---|---|---| +| `Action` | **3** | **0** | +| `1/2h` | 15 → **07:30** ✅ | 39 → **19:30** ✅ | +| `Day` | 3 | 3 | +| `[6]` | 2 | 16 | +| name length | 9 | **0** | +| Setup | `TEST1.mmb` | *(none)* ✅ | -⚠ Still unknown, and all answerable with **one schedule containing two entries**: +**Five independent confirmations**, no fitting required: -- the **record stride** and how many records the file holds (one record was - captured; `27 03 10` at `[264]` may be a trailer or a second record's head) -- whether `Action` is one byte or a uint32 -- the **action code values** — `03` is presumably - `DUTYCYCLE_START_MONITOR_WITH_SETUP` but the enum's base is unknown -- `[6]`, and how `Day` and `WDAY` divide the work +1. Slot 15 = 07:30 and slot 39 = 19:30, both matching the operator's stated + times, on a 30-minute grid — and 39 − 15 = 24 slots = exactly 12 hours. +2. The name-length byte reads 9 for `TEST1.mmb`, 40 for the 40-character name in + the read capture, and **0** for the Auto Call Home entry. +3. **Auto Call Home carries no setup name** — which is what it should do, and + direct proof that a schedule entry can exist with no setup attached. +4. The 260-byte stride lands the second record's `1/2h` exactly at `[264]`. +5. `2 × 260 + 4 = 524`, the whole body, with nothing left over. + +⚠ **Correction:** the `27 03 10` bytes at `[264]` were previously recorded here as +a possible trailer. They are **record 2's `1/2h`, `Day` and `[6]` fields**. Not +a trailer — a second record hiding behind an assumption that the file held one. + +Action codes, so far: + +| value | action | +|---|---| +| `0` | **Auto Call Home** | +| `3` | **Start monitoring, with setup** | + +⚠ `[6]` is the one field still unidentified — `2` on the start entry, `16` on the +Auto Call Home entry. It is not the name length and not the time. A capture +that changes only the **Repeat** checkbox, or switches Day↔Week, would isolate it. + +⚠ Also still unknown: whether the file can hold more than the records in use +(both captures had every used record contiguous from offset 0), and the remaining +four action codes. ### Six duty-cycle actions, not five From 98cbdad489353bce7203e32a35aa26bcc509104e Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 00:35:09 -0400 Subject: [PATCH 22/43] docs(series4): Repeat is folded into Action -- and [6] is not the repeat flag A capture that changed ONLY the Repeat Daily checkbox (pull schedule, disable repeat, push schedule) moved exactly one byte in the entire session: record @0 Action 0x03 -> 0x02 Record 2 unchanged, [6] unchanged, and 0xDA / 0x68 / 0x82 / 0x71 / 0x94 / 0x8D all byte-identical. Two corrections to the previous commit: * [6] is NOT the repeat flag. That was the field I predicted this capture would isolate; it stayed 2 and 16. Still unidentified. * The label "Action" on [0] was too simple -- it carries repeat behaviour too. Leading hypothesis: 2 and 3 are the two setup-bearing start actions the firmware names, with repeat selecting between them -- 0 = DUTYCYCLE_CALLHOME 2 = DUTYCYCLE_START_MONITOR_WITH_SETUP (repeat off) 3 = DUTYCYCLE_START_MONITOR_WITH_SETUP_STOP_COMPLETE (repeat on) Supported independently by the THOR manual, which says a repeating schedule hitting a Start Monitoring event while already monitoring will "stop the current monitoring session, run any Auto Call Home actions, load the compliance setup and continue monitoring" -- exactly what _STOP_COMPLETE should mean. A repeating start must terminate the in-progress session; a one-shot start need not. The firmware name, the manual's behaviour and the single moved byte all agree. Explicitly NOT claiming the enum ordering: the action names were recovered with `strings | sort`, so source order is lost. Do not infer 1 = START_MONITOR just because it falls between the two known values. Three of six codes observed. Also noted: Thor re-pushed the whole 2,090-byte config for a one-byte schedule change -- a third instance of the schedule<->config coupling. Capture provenance: the seismo_lab bins did not reach the dev box, but the socat relay on mint-mac keeps its own timestamped -x log, and the session was reconstructed from it byte-for-byte (25 frames each way, 0 bad checksums). That backup log is worth keeping in the loop -- it has now saved a capture once. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 49 ++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 112015f..fb280af 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1546,9 +1546,52 @@ Action codes, so far: | `0` | **Auto Call Home** | | `3` | **Start monitoring, with setup** | -⚠ `[6]` is the one field still unidentified — `2` on the start entry, `16` on the -Auto Call Home entry. It is not the name length and not the time. A capture -that changes only the **Repeat** checkbox, or switches Day↔Week, would isolate it. +### The Repeat flag is folded into `Action` (2026-09-25) + +A capture that changed **only** the *Repeat Daily* checkbox — schedule pulled, +repeat disabled, schedule pushed — moved **exactly one byte in the whole +session**: + +``` +record @0 Action 0x03 → 0x02 (repeat ON → repeat OFF) +``` + +Nothing else moved. Record 2 unchanged, `[6]` unchanged, and `0xDA`, `0x68`, +`0x82`, `0x71`, `0x94`, `0x8D` all byte-identical. + +⚠ So **`[6]` is NOT the repeat flag** — the field predicted for that test is still +unidentified. And the earlier label on `[0]` was too simple: it is not a plain +action index, it carries the repeat behaviour too. + +**Leading hypothesis** — `2` and `3` are the two *setup-bearing start* actions the +firmware names, and repeat selects between them: + +| value | action | why | +|---|---|---| +| `0` | `DUTYCYCLE_CALLHOME` | the 19:30 entry, no setup name | +| `2` | `DUTYCYCLE_START_MONITOR_WITH_SETUP` | repeat **off** | +| `3` | `DUTYCYCLE_START_MONITOR_WITH_SETUP_STOP_COMPLETE` | repeat **on** | + +That reading is supported by the THOR manual's description of what a *repeating* +schedule does on hitting a Start Monitoring event while already monitoring: + +> *"the unit will stop the current monitoring session, run any Auto Call Home +> actions, load the compliance setup and continue monitoring."* + +Which is precisely what a name ending `_STOP_COMPLETE` would mean: a repeating +start has to terminate the in-progress session first, a one-shot start does not. +The firmware action name, the manual's behaviour, and the single byte that moved +all agree. + +⚠ **The enum's ordering is NOT known.** The action names were recovered with +`strings | sort`, so their source order is lost — do **not** infer that `1` is +`DUTYCYCLE_START_MONITOR` merely because it sits between the two known values, +tempting as that is. Three of six codes are observed (`0`, `2`, `3`); the +remaining three need entries that use those actions. + +⚠ Note also that Thor re-pushed the **entire 2,090-byte config block** for a +one-byte schedule change — a third instance of the coupling described in +*The schedule↔config coupling is Thor's, not the protocol's*. ⚠ Also still unknown: whether the file can hold more than the records in use (both captures had every used record contiguous from offset 0), and the remaining From cc1388df655f49ced9f496ebf7aba9e0ed4692f5 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 00:37:10 -0400 Subject: [PATCH 23/43] feat(scratch): recover captures from the socat relay log -- validated byte-exact The bench relay runs socat with -x, which hex-dumps every forwarded byte in both directions. That makes its log a complete second copy of every capture taken through it, independent of whether seismo_lab was recording. On 2026-09-25 a capture's .bin files never left the Windows machine and the session was rebuilt from the relay log instead. When the real bins turned up afterwards, the reconstruction was byte-for-byte IDENTICAL in both directions (3,595 and 4,004 bytes) -- verified again through the committed script, not just the ad-hoc version used at the time. So this is a validated fallback, not a lossy approximation. Adds scratch/socat_log_split.py, with --from-line/--to-line for picking one session out of a log that spans several (split on the "accepting connection" markers, or the frame walk runs sessions together). Also documents the -x flag and the fallback in the session-provenance section, so the next person runs the relay in a way that keeps the safety net. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 19 ++++- scratch/socat_log_split.py | 119 +++++++++++++++++++++++++++ 2 files changed, 137 insertions(+), 1 deletion(-) create mode 100644 scratch/socat_log_split.py diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index fb280af..1111523 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1757,12 +1757,29 @@ Three sittings, all on the same unit: | 2026-09-23 | 5 events (4 waveform, 1 histogram) | the event chain + `0x5A` | | 2026-09-24 | 5 events | **Thor pushing a setup**, via a recording relay | -The 2026-09-24 sitting used a different topology worth noting, because it is +The 2026-09-24/25 sittings used a topology worth keeping, because it is reusable: `socat` on mint-mac shares `/dev/ttyACM0` on TCP, seismo_lab's TCP bridge relays Thor to it and records both directions. Thor is configured with the unit at `127.0.0.1:` exactly as if it were a field modem. No modem, no SIM, and **nothing on the production Thor box is touched.** +**Run the relay with `-x` and keep its log.** It hex-dumps every forwarded byte +in both directions, which makes it a complete second copy of every capture, +independent of whether seismo_lab was recording: + +``` +socat -d -d -x TCP-LISTEN:12345,reuseaddr,fork /dev/ttyACM0,raw,echo=0,b115200 \ + > ~/mm-captures/socat_.log 2>&1 +``` + +That paid off on 2026-09-25, when a capture's `.bin` files never left the Windows +machine — the session was rebuilt from the relay log with +`scratch/socat_log_split.py`. When the real bins arrived later the +reconstruction was **byte-for-byte identical in both directions** (3,595 and +4,004 bytes), so the log is a validated fallback rather than an approximation. +Sessions concatenate in one log; split them on the `accepting connection` +markers. + ⚠ Firmware is the single biggest caveat on this document. Every finding here is from one unit on the Blastware build. A `11.0BD` unit has not been touched. diff --git a/scratch/socat_log_split.py b/scratch/socat_log_split.py new file mode 100644 index 0000000..9dd3427 --- /dev/null +++ b/scratch/socat_log_split.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +""" +socat_log_split.py — recover a capture pair from a `socat -x` relay log. + +Why this exists +--------------- +The bench relay that puts Thor in front of a USB-attached Micromate is: + + socat -d -d -x TCP-LISTEN:12345,reuseaddr,fork /dev/ttyACM0,raw,echo=0,b115200 \ + > ~/mm-captures/socat_.log 2>&1 + +`-x` makes socat hex-dump every byte it forwards, in both directions, with +timestamps. That log is therefore a **complete second copy of every capture** +taken through the relay — independent of whether seismo_lab was recording. + +On 2026-09-25 that mattered: a capture's `.bin` files never made it off the +Windows machine, and the session was rebuilt from this log instead. When the +real bins turned up later, the reconstruction was **byte-for-byte identical in +both directions** (3,595 and 4,004 bytes). So this is a validated fallback, not +a lossy approximation. + +Log format +---------- +``` +> 2026/09/25 00:30:35.000276659 length=21 from=0 to=20 + 41 02 10 10 00 5b 00 00 30 00 ... +2026/09/25 00:30:35 socat[32190] N write(5, 0x..., 21) completed +< 2026/09/25 00:30:35.000384100 length=64 from=0 to=63 + 02 00 c5 a4 00 00 30 00 ... +``` + +`>` is data heading toward the serial device (Thor → unit). `<` is data coming +back (unit → Thor). Hex lines are space-separated and indented; socat's own +status lines start with a date and carry no payload. + +Usage +----- + # whole log + python scratch/socat_log_split.py socat_20260924_181248.log --out-dir ./recovered + + # one session — line numbers from the "accepting connection" markers + grep -n "accepting connection" socat_*.log + python scratch/socat_log_split.py socat_*.log --from-line 919 --out-dir ./recovered + +Then parse the result as usual: + + python scratch/mm_frame_parse.py recovered/raw_bw.bin recovered/raw_s3.bin + +⚠ A log spanning several sessions concatenates them. Split by line number using +the `accepting connection` markers, or the frame walk will run sessions together. +""" + +from __future__ import annotations + +import argparse +import re +from pathlib import Path + +_HEX = re.compile(r"\A[0-9a-f]{2}\Z") +_SOCAT_STATUS = re.compile(r"\A\d{4}/\d{2}/\d{2}") + + +def split(lines) -> tuple[bytes, bytes]: + """Return (to_device, from_device) byte streams.""" + to_dev, from_dev = bytearray(), bytearray() + cur = None + for line in lines: + if line.startswith(">"): + cur = to_dev + continue + if line.startswith("<"): + cur = from_dev + continue + if _SOCAT_STATUS.match(line): + # socat's own status line ends the current dump block. + cur = None + continue + if cur is None or not line.startswith(" "): + continue + toks = line.split() + if toks and all(_HEX.match(t) for t in toks): + cur.extend(int(t, 16) for t in toks) + return bytes(to_dev), bytes(from_dev) + + +def main() -> int: + ap = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + ap.add_argument("log", help="a socat -x log file") + ap.add_argument("--out-dir", default=".", help="where to write the .bin pair") + ap.add_argument("--from-line", type=int, default=1, + help="first log line to read (1-based) — use the " + "'accepting connection' marker of the session you want") + ap.add_argument("--to-line", type=int, default=None, + help="last log line to read (1-based, inclusive)") + ap.add_argument("--prefix", default="raw", help="output basename prefix") + args = ap.parse_args() + + lines = Path(args.log).read_text(errors="replace").splitlines() + lo = max(args.from_line - 1, 0) + hi = args.to_line if args.to_line is not None else len(lines) + to_dev, from_dev = split(lines[lo:hi]) + + out = Path(args.out_dir) + out.mkdir(parents=True, exist_ok=True) + bw = out / f"{args.prefix}_bw.bin" + s3 = out / f"{args.prefix}_s3.bin" + bw.write_bytes(to_dev) + s3.write_bytes(from_dev) + print(f"Thor -> unit {len(to_dev):>7} bytes {bw}") + print(f"unit -> Thor {len(from_dev):>7} bytes {s3}") + if not to_dev or not from_dev: + print("⚠ one direction is empty — check --from-line / --to-line") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From f839541d0843d90adc13df806f125ebea59bcd9a Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 00:55:54 -0400 Subject: [PATCH 24/43] docs(series4): CORRECT the schedule record -- [6] is the Action bitmask, not [0] A five-entry schedule (start/stop/self-check/start/ACH) overturns the two previous readings of this record, and the operator supplied a Thor screenshot of the schedule as ground truth. [6] is the Action, and the values are powers of two: 2 = Start Monitoring 4 = Stop Monitoring 8 = Self Check 16 = Auto Call Home Bits 1-4 of a bitmask; bit 0 (value 1) is unobserved -- a natural home for the setup-less DUTYCYCLE_START_MONITOR, but that is a guess. Two retractions: * [6] was recorded as "the one unidentified field" and predicted to be the Repeat flag. It is the Action. * [0] was labelled Action, then "Action with repeat folded in". Both wrong. [0] is non-zero only on record 0 -- records 0 and 3 here are the SAME action with different [0] values. It is a schedule-level field carried in the first record, holding Repeat: 3 on, 2 off, matching "Repeat Daily: Disabled" on the Thor screen. The earlier repeat capture was consistent with both readings because it had one start entry and moved one byte. A single-variable test is not always enough; it took four distinct actions to separate the fields. SUB 0x47 is confirmed as the scheduler enable. Previously recorded as "genuinely undetermined" whether it sets or reads -- Thor's notification pane timestamps it: schedule write completes 00:51:29, "successfully SENT" 00:51:31, the 0x47 pair at 00:51:31 and 00:51:33, "successfully ENABLED" 00:51:35. Nothing else sits between the two notifications. params[7] in {1,3} is still open and is more likely a selector than a value, since a lone params[7]=3 also appears at session start. It must be DLE-escaped -- a bare 0x03 truncates the frame. SECOND RETRACTION: setups ARE written as raw .MMB files. This document twice said they are not. Thor used both paths in one session, choosing by whether the setup is active: TEST1.mmb (active) 0xDA -> 0x68/0x73 -> 0x82/0x83 -> 0x71/0x72 test2.mmb (not active) 0x8D \system\setups\test2.mmb -> 0x8E (2192 B) The .MMB file is nearly the compliance block -- 1968/2086 bytes equal (94.3%) at a 4-byte shift, 102 bytes longer, name at offset 38 vs 42. Same structure, different framing. Writing a setup as a file is the cleaner path for SFM: two frames, no 0xDA/0x68/0x82 ritual, and it does not disturb the active setup. Also: file writes are chunked. The schedule's 1,304 bytes went as 1,024 + 280 with each offset = that chunk's length, but the 2,192-byte setup went in one frame, so 1,024 is not a hard ceiling. Rule unexplained, recorded as observed. Capture provenance: the seismo_lab bins were empty (capture not stopped), and the session was recovered from the socat relay log again -- 38 frames each way, 0 bad checksums. That fallback has now saved two captures. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 145 +++++++++++++++++++-------- 1 file changed, 101 insertions(+), 44 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 1111523..f2bda66 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1502,16 +1502,19 @@ The captured file is **two 260-byte records plus four zero bytes** — 524 bytes exactly. Every non-zero byte in the file falls inside those two records. ``` -offset field size notes -[0] Action 1 (or uint32 LE at [0:4] — both are zero-extended here) -[1:4] padding 3 zero in both records -[4] 1/2h 1 half-hour slot, 0–47 -[5] Day 1 3 in both records -[6] ?? 1 the one unidentified field -[7] name length 1 -[8:260] Setup name 252 null-padded +offset field size notes +[0] schedule flags 1 record 0 ONLY — 3 = repeat on, 2 = repeat off +[1:4] padding 3 zero in every record observed +[4] 1/2h 1 half-hour slot, 0–47 +[5] Day 1 3 in every record observed — undetermined +[6] Action 1 bitmask: 2 start, 4 stop, 8 self-check, 16 ACH +[7] name length 1 +[8:260] Setup name 252 null-padded; empty for actions that take no setup ``` +⚠ `[0]` and `[6]` were **mislabelled twice** before a five-entry schedule +separated them — see *CORRECTED* below. + Decoded against the operator's own description of what they entered — *"start monitoring TEST1 at 7:30 AM, Auto Call Home at 7:30 PM"*: @@ -1546,56 +1549,110 @@ Action codes, so far: | `0` | **Auto Call Home** | | `3` | **Start monitoring, with setup** | -### The Repeat flag is folded into `Action` (2026-09-25) +### ⚠ CORRECTED — `[6]` is the Action, and it is a bitmask (2026-09-25) -A capture that changed **only** the *Repeat Daily* checkbox — schedule pulled, -repeat disabled, schedule pushed — moved **exactly one byte in the whole -session**: +A five-entry schedule settled this, and it **overturns the two previous +readings of this record.** The operator's Thor screen, captured alongside: ``` -record @0 Action 0x03 → 0x02 (repeat ON → repeat OFF) +actioncheck — Micromate — Repeat Daily: Disabled + 7:30 AM Start Monitoring TEST1 + 8:00 AM Stop Monitoring + 8:30 AM Self Check + 9:00 AM Start Monitoring test2 + 7:30 PM Auto Call Home ``` -Nothing else moved. Record 2 unchanged, `[6]` unchanged, and `0xDA`, `0x68`, -`0x82`, `0x71`, `0x94`, `0x8D` all byte-identical. +Five entries, and the file holds exactly five 260-byte records: -⚠ So **`[6]` is NOT the repeat flag** — the field predicted for that test is still -unidentified. And the earlier label on `[0]` was too simple: it is not a plain -action index, it carries the repeat behaviour too. +| rec | `[0]` | `1/2h` | `Day` | **`[6]`** | len | name | +|---|---|---|---|---|---|---| +| @0 | **2** | 15 = 07:30 | 3 | **2** | 9 | `TEST1.mmb` | +| @260 | 0 | 16 = 08:00 | 3 | **4** | 0 | | +| @520 | 0 | 17 = 08:30 | 3 | **8** | 0 | | +| @780 | 0 | 18 = 09:00 | 3 | **2** | 9 | `test2.mmb` | +| @1040 | 0 | 39 = 19:30 | 3 | **16** | 0 | | -**Leading hypothesis** — `2` and `3` are the two *setup-bearing start* actions the -firmware names, and repeat selects between them: +**`[6]` tracks the action exactly, and the values are powers of two:** -| value | action | why | -|---|---|---| -| `0` | `DUTYCYCLE_CALLHOME` | the 19:30 entry, no setup name | -| `2` | `DUTYCYCLE_START_MONITOR_WITH_SETUP` | repeat **off** | -| `3` | `DUTYCYCLE_START_MONITOR_WITH_SETUP_STOP_COMPLETE` | repeat **on** | +| `[6]` | action | +|---|---| +| `2` | Start Monitoring | +| `4` | Stop Monitoring | +| `8` | Self Check | +| `16` | Auto Call Home | -That reading is supported by the THOR manual's description of what a *repeating* -schedule does on hitting a Start Monitoring event while already monitoring: +Bits 1–4 of a bitmask. **Bit 0 (value `1`) is unobserved** — a natural home for +the setup-less `DUTYCYCLE_START_MONITOR`, but that is a guess, not a finding. -> *"the unit will stop the current monitoring session, run any Auto Call Home -> actions, load the compliance setup and continue monitoring."* +**Two retractions:** -Which is precisely what a name ending `_STOP_COMPLETE` would mean: a repeating -start has to terminate the in-progress session first, a one-shot start does not. -The firmware action name, the manual's behaviour, and the single byte that moved -all agree. +1. `[6]` was recorded as *"the one unidentified field"* and predicted to be the + Repeat flag. It is the **Action**. +2. `[0]` was labelled **Action**, then *"Action with repeat folded in"*. Both + wrong. `[0]` is **non-zero only on record 0** — including here, where record + 0 and record 3 are the same action with different `[0]` values. It is a + **schedule-level field carried in the first record**, and it holds the Repeat + flag: `3` repeat on, `2` repeat off, matching *"Repeat Daily: Disabled"* on + the screen. -⚠ **The enum's ordering is NOT known.** The action names were recovered with -`strings | sort`, so their source order is lost — do **not** infer that `1` is -`DUTYCYCLE_START_MONITOR` merely because it sits between the two known values, -tempting as that is. Three of six codes are observed (`0`, `2`, `3`); the -remaining three need entries that use those actions. +The earlier repeat capture was consistent with both readings because it had one +start entry and changed one byte. A single-variable test is not always enough — +it took a schedule with four distinct actions to separate the two fields. -⚠ Note also that Thor re-pushed the **entire 2,090-byte config block** for a -one-byte schedule change — a third instance of the coupling described in -*The schedule↔config coupling is Thor's, not the protocol's*. +`Day` is `3` on all five records across every capture. Still undetermined. -⚠ Also still unknown: whether the file can hold more than the records in use -(both captures had every used record contiguous from offset 0), and the remaining -four action codes. +### ✅ `SUB 0x47` is the scheduler enable — confirmed by Thor's own notifications + +Previously recorded as *"genuinely undetermined"* whether `0x47` sets or reads. +Thor's notification pane timestamps it: + +``` +00:51:29 schedule write (0x8E) completes +00:51:31 Thor: "actioncheck has been successfully SENT to ... UM12947" +00:51:31 → 0x47 params[7] = 0x01 +00:51:33 → 0x47 params[7] = 0x03 (on the wire as 10 03 — DLE-escaped) +00:51:35 Thor: "actioncheck was successfully ENABLED on ... UM12947" +``` + +The only frames between *sent* and *enabled* are the `0x47` pair. **`0x47` +performs the enable.** + +⚠ The meaning of `params[7]` ∈ {1, 3} is still open — a single `0x47` with +`params[7] = 3` also appears at *session start*, where nothing is being enabled, +so the byte is more likely a selector than a value. Note it must be +**DLE-escaped**: a bare `0x03` in params truncates the frame at what the device +reads as ETX. + +### 🔑 Setups ARE written as raw `.MMB` files — second retraction + +This document twice stated that setups are *not* pushed as raw `.MMB` blobs and +go only through `0xDA` + the config block. **Wrong.** In this session Thor did +both, choosing by whether the setup is the active one: + +| setup | how it was written | +|---|---| +| `TEST1.mmb` (**active**) | `0xDA` → `0x68`/`0x73` → `0x82`/`0x83` → `0x71`/`0x72` | +| `test2.mmb` (not active) | `0x8D` `\system\setups\test2.mmb` → `0x8E` (2,192 B) | + +So **`\system\setups\.mmb`** is the setup directory, and the generic file +transfer writes there directly. + +The `.MMB` file is *nearly* the compliance block: aligning the two gives +**1,968 / 2,086 bytes equal (94.3%) at a 4-byte shift**. The file is 102 bytes +longer (2,192 vs 2,090) and puts the setup name at offset 38 where the `0x71` +block has it at 42. So the on-disk setup and the wire compliance block are the +same structure with different framing — not two formats. + +**This is the cleaner path for SFM.** Writing a setup as a file needs two frames +and no `0xDA`/`0x68`/`0x82` ritual, and it does not disturb the active setup. + +### File writes are chunked + +The schedule's 1,304 bytes went as **two `0x8E` frames, 1,024 + 280**, each with +`offset` = that chunk's own length. The 2,192-byte setup file went in **one** +frame, so 1,024 is not a hard ceiling — the chunking rule is unexplained and +recorded as observed. ### Six duty-cycle actions, not five From 8e37803d4093dfc3e8aa12948e17850557560b70 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 01:26:42 -0400 Subject: [PATCH 25/43] docs(series4): event download, PER-EVENT delete, and the ACH config write Three captures with operator ground truth (Thor screenshots of the event list and the weekly schedule). All Thor-originated. DAY AND SCHEDULE TYPE, both settled by a weekly schedule. Thor's screen showed "Start Monitoring 8:00 AM every day, alternating TEST1/test2, Repeat Weekly disabled". The file is 7 x 260 + 4 = 1824 bytes and every record matches row for row: Day = 0 Sunday .. 6 Saturday and [0] on record 0 read 4, against 2 and 3 in the daily schedules, so it carries schedule TYPE as well as Repeat: 2 daily, 3 daily+repeat, 4 weekly, 5 weekly+repeat (predicted, unobserved). Bit 0 is Repeat; 2 and 4 are the bases. Same bitmask style as [6]. In daily schedules Day reads 3 everywhere and is presumably ignored -- inference, and the value 3 is unexplained. EVENT DOWNLOAD. SUB 0x93 -> 0x6C arms each event before 1E/1F, with empty params and an all-zero ack -- the Series IV analogue of Series III's 1E(token=0xFE), and simpler. Event keys are a plain sequential counter (055D4A81..86 for six events) at data[11:15], with the event size at data[17:19]. SUB 0x0A walks the list as 30-byte timestamped records; the dates match Thor's event list exactly. DELETE IS PER-EVENT -- and this is the last piece a homebrew ACH receiver was missing: 0xA8 params[0:4] = -> ack 0x57 0xAA params = zeros -> ack 0x55 The operator deleted the top row of Thor's list (the newest event) and 0xA8 carried 055D4A86, the highest key from the walk. Confirmed end to end. Strictly safer than Series III, which can only erase everything: a receiver can delete exactly what it has confirmed it stored. Different opcodes -- do not reach for 0xA3/0xA2. Noted that SUB 0x06 read identically before and after, so it is not a way to confirm a deletion landed. ACH CONFIG. 0x2C / 0x7E / 0x7F with acks 0xD3 / 0x81 / 0x80 -- identical to Series III. 126-byte write payload, offset 0x007E; the 0x2C read returns the same bytes behind an 11-byte prefix. The enable flag is write[5]: 0x05 enabled, 0x04 disabled. Bit 0 is the flag, bit 2 set in both states -- do NOT test for 0x01/0x00 as Series III does. Dial string at write[6:] ("RADIO RING"). Flagged: write[118:120] changed on its own between the two sessions (43 23 -> 2e 5e) with nothing touched, and Thor writes back whatever it read. Round-trip that field, never synthesise it. Beyond the enable byte and dial string the field map is NOT established -- only one setting was varied, and Series III's offsets are a hypothesis, not a transfer. Every command on the unsafe list is now observed. None has been originated by us, which is the line that still matters. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 158 ++++++++++++++++++++++++++- 1 file changed, 154 insertions(+), 4 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index f2bda66..124adde 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -449,6 +449,13 @@ memory 15,000,000 total and free (no events stored). Series III has one config and nothing to enumerate. 15. **`SUB 0x1C` carries the device clock** (day/month/year/h/m/s at `data[13:21]`). Nothing else read so far reports the unit's own time. +18. **Deletion is per-event** (`0xA8` + a key, then `0xAA`), where Series III can + only erase everything (`0xA3`/`0xA2`). +19. **`SUB 0x93` arms each event** before `1E`/`1F`, replacing Series III's + `1E(token=0xFE)` — no token, no params. +20. **Event keys are a sequential counter** (`055D4A81…86`), not flash addresses. +21. **The ACH enable is `0x05`/`0x04`**, not Series III's `0x01`/`0x00` — bit 0 + is the flag, bit 2 is set in both states. 16. **There is a generic file transfer addressed by full path** — `0x94`/`0x48` read, `0x8D`/`0x8E` write. Series III has nothing comparable; its config is reachable only through dedicated commands. @@ -1301,6 +1308,141 @@ Worth noting the histogram (`…81`) and the loudest waveform (`…84`) report the histogram's single 1-minute interval spans the whole thumping session, so its maximum should equal the loudest event in it. +## Event download, per-event delete, and ACH config (2026-09-25) + +Three captures with operator-supplied ground truth, including Thor screenshots of +the event list and the schedule. All Thor-originated; we still send nothing. + +### ✅ `Day` is the day of week — and `[0]` is the schedule type + +A **weekly** schedule settles both fields. Thor's screen: *"multiday, Repeat +Weekly: Disabled, Start Monitoring at 8:00 AM every day, alternating +TEST1 / test2."* The file is **7 × 260 + 4 = 1,824 bytes**, and every record +matches: + +| rec | `1/2h` | time | `Day` | | `[6]` | name | +|---|---|---|---|---|---|---| +| @0 | 16 | 08:00 | **0** | Sun | 2 | `TEST1.mmb` | +| @260 | 16 | 08:00 | **1** | Mon | 2 | `test2.mmb` | +| @520 | 16 | 08:00 | **2** | Tue | 2 | `TEST1.mmb` | +| @780 | 16 | 08:00 | **3** | Wed | 2 | `test2.mmb` | +| @1040 | 16 | 08:00 | **4** | Thu | 2 | `TEST1.mmb` | +| @1300 | 16 | 08:00 | **5** | Fri | 2 | `test2.mmb` | +| @1560 | 16 | 08:00 | **6** | Sat | 2 | `TEST1.mmb` | + +**`Day` = 0 Sunday … 6 Saturday**, and the alternating setup names line up with +the screen row for row. + +`[0]` on record 0 read **4** here, against `2` and `3` in the daily schedules — +so it carries the schedule *type* as well as Repeat: + +| `[0]` | meaning | +|---|---| +| `2` | Daily, repeat off | +| `3` | Daily, repeat on | +| `4` | **Weekly, repeat off** | +| `5` | Weekly, repeat on — ⚠ predicted, not observed | + +Bit 0 is Repeat; `2` and `4` are the Day/Week bases. Same bitmask style as +`[6]`. + +⚠ In *daily* schedules `Day` reads `3` in every record of every capture. With +`[0]` already saying "daily", the field is presumably ignored there — but that is +inference, and `3` is unexplained. + +### Event download — `SUB 0x93` arms each event + +Thor's download of six events: + +``` +POLL 0x15 0x49 POLL 0x1C 0x15 0x49 0x06 + 0x93 0x1E 0x0C 0x5A×4 ← event 1 + 0x93 0x1F 0x0C 0x5A×11 ← event 2 + 0x93 0x1F 0x0C 0x5A×12 ← event 3 + 0x93 0x1F 0x0C 0x5A×14 ← event 4 + 0x93 0x1F 0x0C 0x5A×9 ← event 5 + 0x93 0x1F 0x0C 0x5A×6 ← event 6 + 0x93 0x06 0x93 0x1E 0x0A×9 ← re-walk the list +``` + +**`SUB 0x93` → ack `0x6C`** is sent before every event, with empty params and an +all-zero ack. It is the Series IV analogue of Series III's `1E(token=0xFE)` arm +step, and it is simpler: no token, no params. + +**Event keys are a plain sequential counter.** The six events walked as +`055D4A81 … 055D4A86`, consecutive, at `data[11:15]` of the `0x1E`/`0x1F` +response, with the event's byte size at `data[17:19]`. Nothing like Series III's +flash addresses. + +`SUB 0x0A` walks the event list returning **30-byte records** carrying start and +stop timestamps (`[11]` day, `[12]` month, `[13:15]` year BE …), terminating on +an all-zero record. The dates match Thor's event list exactly (five events on +09/23/2026, one on 09/24/2026). + +### 🔑 Delete is PER-EVENT — `SUB 0xA8` + `0xAA` + +``` +0xA8 params[0:4] = → ack 0x57 +0xAA params = zeros → ack 0x55 +``` + +The operator deleted the **top row of Thor's list** — the newest event — and +`0xA8` carried `05 5d 4a 86`, the **highest** key from the walk. Thor lists +newest-first, so newest = highest key. Confirmed end to end. + +**This is the last piece a homebrew ACH receiver was missing.** The manual +established that collection state is server-side bookkeeping and that "delete +events from unit" is a separate server-issued action; this is that action's +opcodes. + +Two things worth noting against Series III: + +- Series III erases **everything** (`0xA3 → 0x1C → 0x06 → 0xA2`). Series IV + deletes **one named event**, which is strictly safer — a receiver can delete + exactly what it has confirmed it stored. +- The opcodes are different. Do not reach for `0xA3`/`0xA2` here. + +⚠ `SUB 0x06` (storage range) read **identically before and after** the delete in +this capture, so it is not a quick way to confirm a deletion landed. Re-walk the +event list instead. + +### ✅ ACH config — `0x2C` / `0x7E` / `0x7F`, exactly Series III + +Thor enabling then disabling Auto Call Home, twice through the same sequence: + +``` +POLL → 0x15 → 0x49 → POLL → 0x2C (read) → 0x7E (write, 126 B) → 0x7F (confirm) +``` + +Same SUBs and the same acks as Series III — `0x2C`→`0xD3`, `0x7E`→`0x81`, +`0x7F`→`0x80`. The write payload is **126 bytes** with `offset = 0x007E`, and +the `0x2C` read returns those same 126 bytes behind an **11-byte prefix**, so +`read[n + 11]` is `write[n]`. + +**The enable flag is `write[5]`** (= `read[16]`): + +| value | state | +|---|---| +| `0x05` | Auto Call Home **enabled** | +| `0x04` | **disabled** | + +Bit 0 is the enable — the same bit position Series III uses at `raw[5]`, but with +bit 2 set as well in both states. Do not test for `0x01`/`0x00`. + +`write[6:]` holds the dial string, null-padded — `"RADIO RING"` on this unit, +matching Series III's `raw[6:46]`. + +⚠ **`write[118:120]` changed on its own between the two sessions** (`43 23` → +`2e 5e`) without the operator touching anything. Thor reads it and writes back +whatever it read. Its meaning is unknown — plausibly a counter, timer or +session value. **Round-trip this field; never synthesise it.** `write[117]` is +`0x03`, which lines up with Series III's `num_retries` of 3, and `write[120:122]` +is `00 3c` = 60. + +⚠ Beyond the enable byte and the dial string, **the field map is not +established** — only one setting was varied. Series III's offsets are a starting +hypothesis, not a transfer. + ## What the THOR manual settles about the ACH session (2026-09-24) Source: `manuals/723U0201 THOR Operator Manual Rev 08.pdf` §6.2, §3.6. @@ -1776,11 +1918,19 @@ because Thor performed it while we recorded. Observed is not the same as exercised — **we have still never originated a write frame.** The wire format is known; our encoder is unwritten and unproven. -- **Writes** (`0x68`–`0x83`) — format now known, never sent by us -- **Call-home write** (`0x7E` / `0x7F`) — not observed at all -- **Erase** (`0xA3` / `0xA2`) — **the only genuinely untouched destructive path** +- **Writes** (`0x68`–`0x83`) — format known, never sent by us +- **Call-home write** (`0x7E` / `0x7F`) — ✅ observed 2026-09-25, acks `0x81` / + `0x80`; enable flag isolated +- **Per-event delete** (`0xA8` / `0xAA`) — ✅ observed 2026-09-25, acks `0x57` / + `0x55`. **Destructive.** Series III's `0xA3`/`0xA2` erase-all has *not* been + seen on Series IV and may not exist - **Start / stop monitoring** (`0x96` / `0x97`) — observed via Thor 2026-09-24, - acks `0x69` / `0x68`; still never originated by us + acks `0x69` / `0x68` +- **Scheduler enable** (`0x47`) — observed; `params[7]` semantics still unclear + +Every one of these is now *observed*. **None has ever been originated by us**, +and that is the line that still matters: the wire format is known, our encoder is +unwritten and unproven. - `0x1F` (advance event pointer) — non-destructive on Series III but it does move device state, so it is parked with the rest From 96a8831472323d7d6eba4d7ab91cd913609047f4 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 01:56:21 -0400 Subject: [PATCH 26/43] docs(series4): pin down what the ACH config's volatile field is NOT Followed up the write[118:120] field flagged in the previous commit. A third sample plus a brute-force sweep rules out most of the obvious explanations. Samples: 43 23 at 00:50 (idle), still 43 23 at 01:16 twenty-six minutes later, then 2e 5e after a config write. Thor echoes back whatever it last read -- including a value that no longer matches the config it is sending -- and the write is accepted regardless. Ruled out: * a clock or timer -- identical across 26 minutes of idle; only a write moved it * a counter -- it decreased, 17187 -> 11870 * computed by Thor -- Thor demonstrably sends a stale value * a standard CRC16 -- swept all 65,536 polynomials x init {0x0000,0xFFFF} x all four reflection combinations over four candidate regions. No match. Recorded so the sweep is not repeated. It behaves like a unit-computed hash: a one-byte input change scattered the output (XOR 0x6D7D) where a sum would move by 1. But two samples cannot separate that from a nonce regenerated per write. Operationally it does not matter, which is the point: the unit does not validate the field on input, so read-modify-write with the rest of the block echoed verbatim is provably safe. Never synthesise or zero it. Notes what would resolve it -- several config writes with times recorded, cheap to collect during any future ACH capture. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 45 ++++++++++++++++++++++++---- 1 file changed, 39 insertions(+), 6 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 124adde..fb827b2 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1432,12 +1432,45 @@ bit 2 set as well in both states. Do not test for `0x01`/`0x00`. `write[6:]` holds the dial string, null-padded — `"RADIO RING"` on this unit, matching Series III's `raw[6:46]`. -⚠ **`write[118:120]` changed on its own between the two sessions** (`43 23` → -`2e 5e`) without the operator touching anything. Thor reads it and writes back -whatever it read. Its meaning is unknown — plausibly a counter, timer or -session value. **Round-trip this field; never synthesise it.** `write[117]` is -`0x03`, which lines up with Series III's `num_retries` of 3, and `write[120:122]` -is `00 3c` = 60. +#### ⚠ `write[118:120]` — a field the UNIT maintains + +Three samples: + +| when | enable | field | +|---|---|---| +| 00:50, idle (schedule session) | `0x04` | `43 23` | +| 01:16, ACH read #1 | `0x04` | `43 23` — **unchanged after 26 min** | +| 01:16, ACH read #2, after a config write | `0x05` | `2e 5e` | + +Thor **echoes back whatever it last read**, including a value that no longer +matches the config it is sending — and the write is accepted anyway. + +What that rules out: + +- **Not a clock or timer** — identical across 26 minutes of idle. Only a config + write moved it. +- **Not a counter** — it went *down*, 17187 → 11870. +- **Not computed by Thor** — Thor demonstrably sends a stale value. +- **Not a standard CRC16.** Brute-forced all 65,536 polynomials × init + {`0x0000`, `0xFFFF`} × all four reflection combinations, over four candidate + regions (`[0:118]`, `[0:118]+[120:]`, `[5:118]`, and the payload with the field + zeroed). **No match.** Recorded so nobody repeats the sweep. + +It behaves like a hash or checksum the unit computes over the stored config: a +one-byte input change (`0x04`→`0x05`) scattered the output completely, XOR +`0x6D7D`, where a sum would have moved by 1. But two samples cannot separate +that from a nonce regenerated on each write. + +✅ **Operationally this does not matter.** The unit does not validate the field +on input — Thor proved that twice in one capture. **Read the config, change your +field, send everything else back byte-for-byte.** Never synthesise or zero it. +Same read-modify-write shape as the setup block. + +Resolving it properly needs more samples: several config writes with the times +noted, which is cheap to collect during any future ACH capture. + +Neighbouring bytes: `write[117]` is `0x03`, lining up with Series III's +`num_retries` of 3, and `write[120:122]` is `00 3c` = 60. ⚠ Beyond the enable byte and the dial string, **the field map is not established** — only one setting was varied. Series III's offsets are a starting From 45f2997a5b912b6d38cc978e000a6bb48ab75fbc Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 11:29:44 -0400 Subject: [PATCH 27/43] feat(bridges): mm_link -- a bench "modem" with a readable log and fault injection THOR gives almost no visibility into a connection: a refresh button, two poll intervals, and no way to see whether a check succeeded, timed out, or was never attempted. When a unit "won't stay connected" there is nothing to look at. This sits where the cellular modem would and answers that directly. Over socat -x it adds the two things that were missing: * A READABLE LOG. Frames are decoded and timestamped as they pass -- "THOR->unit POLL (21 B)" rather than hex -- so THOR's polling cadence, and its silences, are visible. Raw .bin pairs are still written alongside and load straight into scratch/mm_frame_parse.py. * FAULT INJECTION, via a control file read on the fly: pass normal relay blackhole TCP stays up, bytes are swallowed drop close the connection abruptly delay:N forward N seconds late, both directions onewaydev THOR->unit passes, unit->THOR is swallowed `blackhole` is the point of the exercise. It reproduces the classic cellular failure -- socket open at both ends, nothing crossing -- which a real cell link will not do on cue. THOR was observed last night holding one TCP connection for 17 minutes (00:30 to 00:47), so if the link dies silently the OS will not tell it for roughly the default keepalive, ~2 hours. That is a candidate explanation for "refresh does nothing and only a restart helps", and this makes it testable rather than speculative. No pyserial: the port is driven through stdlib termios. The bench hosts are whatever is to hand and requiring a pip install on someone else's machine is a poor trade for ~30 lines. Deployed and verified on mint-mac (Python 3.12, no third-party modules) against UM12947 -- a POLL round-trips and decodes. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- bridges/mm_link.py | 324 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 324 insertions(+) create mode 100644 bridges/mm_link.py diff --git a/bridges/mm_link.py b/bridges/mm_link.py new file mode 100644 index 0000000..2cefce7 --- /dev/null +++ b/bridges/mm_link.py @@ -0,0 +1,324 @@ +#!/usr/bin/env python3 +""" +mm_link.py — a "perfect modem" between THOR and a Micromate, with a readable +log and deliberate fault injection. + +Why +--- +THOR gives almost no visibility into a connection: a refresh button, two poll +intervals, and no way to see whether a check succeeded, timed out, or was never +sent. When a unit "won't stay connected" there is nothing to look at. + +This sits where the cellular modem would sit and answers the question directly: + + * **What is THOR actually doing?** Every frame is decoded and timestamped — + `POLL`, `MONITOR_STATUS`, `SETUP_NAME_READ` — not a hex dump. + * **Is it even trying?** Silence is visible: the log shows gaps. + * **How does it behave when the link misbehaves?** Faults can be injected on + demand, which a real cell link will not do on cue. + +Point THOR at this host and port exactly as if it were a modem (Communication: +TCP, IP: , Port: <--listen>). + +Fault injection +--------------- +Write a mode into the control file (default `mm_link.ctl`) and it takes effect +on the next byte: + + echo pass > mm_link.ctl # normal relay + echo blackhole > mm_link.ctl # TCP stays up, bytes are swallowed + echo drop > mm_link.ctl # close the connection abruptly (RST-ish) + echo delay:2.0 > mm_link.ctl # forward, but 2 s late in both directions + echo onewaydev > mm_link.ctl # THOR->unit passes, unit->THOR is swallowed + +**`blackhole` is the one that matters.** It reproduces the classic cellular +failure: the socket is still open as far as both ends are concerned, but nothing +crosses. A client that relies on TCP to tell it the peer is gone will sit there +until the OS keepalive fires — which by default is about two hours. + +Usage +----- + python3 bridges/mm_link.py --serial /dev/ttyACM0 --baud 115200 \\ + --listen 12345 --logdir ~/mm-captures + +Writes, per session: + /mmlink_/session.log decoded, timestamped, human-readable + /mmlink_/raw_bw.bin THOR -> unit, raw + /mmlink_/raw_s3.bin unit -> THOR, raw + +The raw pair loads straight into `scratch/mm_frame_parse.py`. +""" + +from __future__ import annotations + +import argparse +import datetime +import os +import socket +import sys +import threading +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "scratch")) +try: + from mm_frame_parse import SUBNAME, destuff # noqa: F401 +except Exception: # pragma: no cover + SUBNAME = {} + +import errno +import select +import termios + +DLE, STX, ETX, ACK = 0x10, 0x02, 0x03, 0x41 + +_BAUD = {9600: termios.B9600, 19200: termios.B19200, 38400: termios.B38400, + 57600: termios.B57600, 115200: termios.B115200} + + +class SerialPort: + """Minimal raw serial port on stdlib termios — no pyserial dependency. + + The bench hosts are whatever is to hand; requiring a pip install on someone + else's machine is a poor trade for the ~30 lines this saves. + """ + + def __init__(self, path: str, baud: int): + if baud not in _BAUD: + raise ValueError(f"unsupported baud {baud}; pick one of {sorted(_BAUD)}") + self.fd = os.open(path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) + a = termios.tcgetattr(self.fd) + a[0] = 0 # iflag: no translation + a[1] = 0 # oflag: raw + a[2] = termios.CS8 | termios.CREAD | termios.CLOCAL # cflag: 8N1, ignore modem lines + a[3] = 0 # lflag: non-canonical, no echo + a[4] = a[5] = _BAUD[baud] + a[6] = list(a[6]) + a[6][termios.VMIN] = 0 + a[6][termios.VTIME] = 0 + termios.tcsetattr(self.fd, termios.TCSANOW, a) + termios.tcflush(self.fd, termios.TCIOFLUSH) + + def read(self, n: int) -> bytes: + r, _, _ = select.select([self.fd], [], [], 0.2) + if not r: + return b"" + try: + return os.read(self.fd, n) + except OSError as e: + if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK): + return b"" + raise + + def write(self, data: bytes) -> None: + while data: + try: + data = data[os.write(self.fd, data):] + except OSError as e: + if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK): + select.select([], [self.fd], [], 0.2) + continue + raise + + def close(self) -> None: + try: + os.close(self.fd) + except OSError: + pass + + +def name_of(sub: int, is_request: bool) -> str: + if is_request: + return SUBNAME.get(sub, f"SUB_{sub:02X}") + return "rsp " + SUBNAME.get(0xFF - sub, f"SUB_{0xFF - sub:02X}") + + +class FrameSniffer: + """Accumulate bytes and report complete frames, without altering the stream.""" + + def __init__(self, is_request: bool): + self.is_request = is_request + self.buf = bytearray() + + def feed(self, data: bytes): + """Yield (sub, payload_len) for each complete frame seen.""" + self.buf.extend(data) + while True: + start = -1 + for i, b in enumerate(self.buf): + if self.is_request and b == ACK and i + 1 < len(self.buf) and self.buf[i + 1] == STX: + start = i + break + if not self.is_request and b == STX: + start = i + break + if start < 0: + if len(self.buf) > 8192: + del self.buf[:-16] + return + j = start + (2 if self.is_request else 1) + end = -1 + while j < len(self.buf): + if self.buf[j] == DLE and j + 1 < len(self.buf): + j += 2 + continue + if self.buf[j] == ETX: + end = j + break + j += 1 + if end < 0: + return # wait for more bytes + body = self.buf[start:end + 1] + del self.buf[:end + 1] + # sub sits at a fixed spot once the leading framing is skipped + off = 5 if self.is_request else 3 + if len(body) > off: + yield body[off], len(body) + + +class Link: + def __init__(self, args): + self.args = args + self.mode = "pass" + self.delay = 0.0 + self.ctl = Path(args.control) + self.session: Path | None = None + self.log_fh = None + self.raw = {} + self.t0 = time.time() + self.counts = {} + + # ── logging ──────────────────────────────────────────────────────────── + def open_session(self): + ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S") + self.session = Path(self.args.logdir) / f"mmlink_{ts}" + self.session.mkdir(parents=True, exist_ok=True) + self.log_fh = open(self.session / "session.log", "a", buffering=1) + self.raw = { + "bw": open(self.session / "raw_bw.bin", "ab"), + "s3": open(self.session / "raw_s3.bin", "ab"), + } + self.say(f"=== session {ts} — serial {self.args.serial} @ {self.args.baud} ===") + + def say(self, text: str): + line = f"{datetime.datetime.now().strftime('%H:%M:%S.%f')[:-3]} {text}" + print(line, flush=True) + if self.log_fh: + self.log_fh.write(line + "\n") + + # ── control file ─────────────────────────────────────────────────────── + def poll_control(self): + while True: + try: + if self.ctl.exists(): + want = self.ctl.read_text().strip().lower() + if want.startswith("delay:"): + d = float(want.split(":", 1)[1]) + if ("delay", d) != (self.mode, self.delay): + self.mode, self.delay = "delay", d + self.say(f"*** MODE -> delay {d}s ***") + elif want and want != self.mode: + self.mode, self.delay = want, 0.0 + self.say(f"*** MODE -> {want} ***") + except Exception: + pass + time.sleep(0.25) + + # ── the relay ────────────────────────────────────────────────────────── + def pump(self, src, dst, tag: str, is_request: bool, stop: threading.Event): + sniff = FrameSniffer(is_request) + arrow = "THOR->unit" if is_request else "unit->THOR" + last = time.time() + while not stop.is_set(): + try: + data = src.recv(4096) if isinstance(src, socket.socket) else src.read(4096) + except OSError: + break + if isinstance(src, socket.socket) and data == b"": + self.say(f"{arrow}: peer closed the connection") + break + if not data: + if time.time() - last > self.args.quiet_after and self.counts: + self.say(f"--- {self.args.quiet_after:.0f}s with no traffic ---") + last = time.time() + continue + last = time.time() + + self.raw[tag].write(data) + self.raw[tag].flush() + for sub, ln in sniff.feed(data): + label = name_of(sub, is_request) + self.counts[label] = self.counts.get(label, 0) + 1 + self.say(f"{arrow} {label:<20} ({ln} B)" + + ("" if self.mode == "pass" else f" [mode={self.mode}]")) + + mode = self.mode + if mode == "drop": + self.say(f"{arrow}: DROPPING the connection (fault injection)") + stop.set() + break + if mode == "blackhole": + continue # swallow, keep the socket open + if mode == "onewaydev" and not is_request: + continue # unit's replies never reach THOR + if mode == "delay" and self.delay: + time.sleep(self.delay) + try: + if isinstance(dst, socket.socket): + dst.sendall(data) + else: + dst.write(data) + except OSError: + break + stop.set() + + def serve(self): + srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + srv.bind(("0.0.0.0", self.args.listen)) + srv.listen(5) + self.open_session() + self.say(f"listening on 0.0.0.0:{self.args.listen} control file: {self.ctl}") + self.say("point THOR at this host/port as Communication=TCP") + threading.Thread(target=self.poll_control, daemon=True).start() + + while True: + conn, addr = srv.accept() + conn.settimeout(0.2) + self.say(f"+++ THOR connected from {addr[0]}:{addr[1]} +++") + try: + ser = SerialPort(self.args.serial, self.args.baud) + except OSError as e: + self.say(f"!!! cannot open {self.args.serial}: {e}") + conn.close() + continue + stop = threading.Event() + ts = [ + threading.Thread(target=self.pump, args=(conn, ser, "bw", True, stop), daemon=True), + threading.Thread(target=self.pump, args=(ser, conn, "s3", False, stop), daemon=True), + ] + for t in ts: + t.start() + for t in ts: + t.join() + conn.close() + ser.close() + summary = ", ".join(f"{k}x{v}" for k, v in sorted(self.counts.items())) + self.say(f"--- connection closed. frames this session: {summary or 'none'} ---") + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("--serial", default="/dev/ttyACM0") + ap.add_argument("--baud", type=int, default=115200) + ap.add_argument("--listen", type=int, default=12345) + ap.add_argument("--logdir", default=os.path.expanduser("~/mm-captures")) + ap.add_argument("--control", default="mm_link.ctl") + ap.add_argument("--quiet-after", type=float, default=30.0, + help="log a marker after this many seconds of silence") + Link(ap.parse_args()).serve() + + +if __name__ == "__main__": + main() From d522d31d631ddf27815d568d80744aa4ca50f093 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 11:34:06 -0400 Subject: [PATCH 28/43] docs(series4): what THOR's "status check" actually is -- 11 commands, 563 MB/month First capture through bridges/mm_link.py, with THOR polling a unit over the bench link at "check connection every 5 s / check status every 5 s". A status check is ELEVEN commands, not one: POLL -> DEVICE_INFO -> 0x49 -> 0x5C -> MONITOR_STATUS -> SETUP_NAME_READ -> STORAGE_RANGE -> 0x02 -> OPERATOR -> 0x47 -> CALLHOME_CFG Each check opens a NEW TCP connection, runs all eleven exchanges in ~300 ms and closes it. Measured over 21 consecutive checks: 236 B out, 936 B back, plus a full handshake each time -- about 2.2 KB per check. At the observed cadence that is 18.8 MB/day, 563 MB/month, per unit. On a metered cellular plan that is real money, and most of it is waste: the check re-reads the call-home config, operator name, active setup name and full device info every ten seconds, none of which changes. SETUP_NAME_READ alone returns 274 bytes a time. POLL + MONITOR_STATUS answers "alive?" and "monitoring?" in two commands and 131 bytes. Both intervals set to 5 s yields one combined pass every 10.1 s, steady across eight measured connections. So the two settings are not independent 5-second timers, which is a plausible reason changing them appears to do nothing. REVISES an earlier hypothesis. Because idle polling reconnects every cycle, a silently-dead link is LESS dangerous while idle than I assumed -- a dead socket fails at connect and the next cycle retries. The exposure is during OPERATIONS: THOR held one connection from 00:30 to 00:47 last night while downloading events and pushing setups. A link dying mid-operation leaves it waiting on a socket the OS will not fail for ~2 h. The blackhole test should therefore be run during a download, not while idle. Also fixes a mislabel in mm_link.py: the SUB byte is DLE-escaped when its value is 0x02/0x03/0x04/0x10, so reading it raw reported SUB 0x02 as "SUB_10". Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- bridges/mm_link.py | 10 +++- docs/micromate_protocol_reference.md | 83 ++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+), 2 deletions(-) diff --git a/bridges/mm_link.py b/bridges/mm_link.py index 2cefce7..a1724d2 100644 --- a/bridges/mm_link.py +++ b/bridges/mm_link.py @@ -170,10 +170,16 @@ class FrameSniffer: return # wait for more bytes body = self.buf[start:end + 1] del self.buf[:end + 1] - # sub sits at a fixed spot once the leading framing is skipped + # SUB sits at a fixed spot past the leading framing -- but it is + # DLE-escaped when its own value is 0x02/0x03/0x04/0x10, so a raw + # read reports 0x10 for those. SUB 0x02 was being logged as + # "SUB_10" until this was handled. off = 5 if self.is_request else 3 if len(body) > off: - yield body[off], len(body) + sub = body[off] + if sub == DLE and len(body) > off + 1: + sub = body[off + 1] + yield sub, len(body) class Link: diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index fb827b2..fc9bdf0 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1595,6 +1595,89 @@ Those are the fields to look for when the schedule entry is finally decoded: captured entry (`03 00 00 00 0f 03 02 …`) has seven bytes before the name, which is the right order of magnitude for that field list. +## What THOR's "status check" actually is (2026-09-25) + +Captured with `bridges/mm_link.py` standing in for the modem, THOR configured +`Communication: TCP`, **Check connection every 5 s**, **Check status every 5 s**. + +### It is eleven commands, not one + +``` +POLL → DEVICE_INFO → 0x49 → 0x5C → MONITOR_STATUS → SETUP_NAME_READ + → STORAGE_RANGE → 0x02 → OPERATOR → 0x47 → CALLHOME_CFG +``` + +Each check opens a **new TCP connection**, runs all eleven exchanges in ~300 ms, +and closes it. Measured over 21 consecutive checks: + +| | | +|---|---| +| request payload | **236 B** | +| response payload | **936 B** | +| TCP connect + teardown | ~1,000 B (fresh handshake every time) | +| **per check** | **~2.2 KB** | + +### The cost + +| cadence | checks/day | per unit | +|---|---|---| +| **every 10 s (as observed)** | 8,640 | **18.8 MB/day — 563 MB/month** | +| every 60 s | 1,440 | 3.1 MB/day — 94 MB/month | +| every 15 min | 96 | 0.2 MB/day — 6 MB/month | + +⚠ **On a metered cellular plan this is real money, and most of it is waste.** +A status check re-reads the call-home config, the operator name, the active setup +name and the full device info block **every ten seconds** — none of which changes +between checks. `SETUP_NAME_READ` alone returns 274 bytes each time. + +A genuine liveness + state check is **`POLL` + `MONITOR_STATUS`**: two commands, +131 bytes of response. Same information about whether the unit is alive and +whether it is monitoring, for a ninth of the traffic. + +### Both intervals at 5 s produces one check every ~10 s + +Connection timestamps over three minutes: `11:31:48.7`, `11:31:58.9`, +`11:32:08.9`, `11:32:19.0`, `11:32:29.1`, `11:32:39.2`, `11:32:49.3`, +`11:32:59.4` — a **10.1 s** period, steady. + +So *"Check connection every 5 s"* and *"Check status every 5 s"* do not describe +two independent 5-second timers; together they yield one combined pass every ten +seconds. That is a plausible reason changing those fields appears to do nothing: +the relationship between the setting and the observed cadence is not what the UI +implies. + +### ⚠ What this means for the "won't stay connected" failure + +Because idle polling **opens a fresh connection each time**, a silently-dead link +is *less* dangerous here than expected — a dead socket fails at connect and the +next cycle simply tries again. The earlier hypothesis (THOR wedged on a +half-open socket during polling) is weakened by this. + +The exposure is during **operations**, not polling: THOR was observed holding one +connection from **00:30 to 00:47 — seventeen minutes** — while downloading events +and pushing setups. A link that dies silently mid-operation leaves THOR waiting +on a socket the OS will not fail for roughly the default keepalive (~2 h). That +remains the best candidate for a unit that will not come back and where refresh +does nothing. + +**Testable with `mm_link.py`:** set `blackhole` *during* a download or config +push rather than while idle, and watch whether THOR ever gives up, whether +refresh emits any bytes, and whether it recovers when `pass` is restored. + +### Design notes for SFM + +Three things this argues for, all cheap: + +1. **Separate liveness from inventory.** Poll `POLL` + `MONITOR_STATUS` + frequently; read config, setup name and call-home settings only when + something says they changed, or on demand. +2. **Show the connection attempt.** Every check here has a visible outcome — + connected, frames exchanged, closed. THOR surfaces none of it, which is why + a failing unit is undiagnosable from the UI. The log this section is built + from took one afternoon to produce and answers questions THOR cannot. +3. **Bound every operation with its own timeout**, independent of TCP's. Do not + rely on the socket to report a dead peer. + ## Thor's conventions vs the protocol's requirements **SFM is not meant to reimplement Thor.** Thor is the only available teacher of From 61e8a5e2209b73afcee2e4baee14619c35c56224 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 11:46:11 -0400 Subject: [PATCH 29/43] docs(series4): two poll checks, and only the status dial is honoured Corrects the previous commit. It claimed both intervals were set to 5 s, taken from a screenshot that turned out to predate the operator's change -- they were at 10 s. The "setting + 5 s" pattern I inferred from that does not survive, and is removed. What a longer capture with asymmetric intervals (connection 10 s, status 30 s) actually shows: There are TWO distinct checks, not one. The eleven-command sequence is the status check; there is also a three-command connection check, POLL -> SERIAL -> 0x49. With both dials EQUAL the connection check never runs separately at all -- every connection observed was the full eleven commands. It only appears once the intervals differ. That alone explains much of "changing the settings does nothing": at equal values you only ever get the expensive one. Steady state over 14 consecutive cycles: FULL at T short at T+10.1 short at T+25.3 FULL at T+30.4 status check set 30 s -> observed 30.4 s honoured connection check set 10 s -> observed 15.2 s 52% slow 27 consecutive connection-check gaps, all 15.1-15.3 s. Systematic, not jitter. 15.2 s is exactly half of 30.4 s and the two are phase-locked 2:1, suggesting the connection check runs at half the STATUS period rather than on its own setting. Flagged as a hypothesis with a sharp prediction: at status 60 s the connection check should land at 30 s whatever its dial says. Worth settling before SFM offers a similar control -- a dial that silently does nothing is worse than no dial. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 50 +++++++++++++++++++++++----- 1 file changed, 41 insertions(+), 9 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index fc9bdf0..fa9e91d 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1634,17 +1634,49 @@ A genuine liveness + state check is **`POLL` + `MONITOR_STATUS`**: two commands, 131 bytes of response. Same information about whether the unit is alive and whether it is monitoring, for a ninth of the traffic. -### Both intervals at 5 s produces one check every ~10 s +### There are TWO checks, and only one of the dials is honoured -Connection timestamps over three minutes: `11:31:48.7`, `11:31:58.9`, -`11:32:08.9`, `11:32:19.0`, `11:32:29.1`, `11:32:39.2`, `11:32:49.3`, -`11:32:59.4` — a **10.1 s** period, steady. +The eleven-command sequence is the **status check**. There is also a much +cheaper **connection check** — three commands: -So *"Check connection every 5 s"* and *"Check status every 5 s"* do not describe -two independent 5-second timers; together they yield one combined pass every ten -seconds. That is a plausible reason changing those fields appears to do nothing: -the relationship between the setting and the observed cadence is not what the UI -implies. +``` +connection check POLL → SERIAL → 0x49 +status check POLL → DEVICE_INFO → 0x49 → 0x5C → MONITOR_STATUS + → SETUP_NAME_READ → STORAGE_RANGE → 0x02 → OPERATOR + → 0x47 → CALLHOME_CFG +``` + +⚠ **With both dials equal, the connection check never runs separately at all** — +every connection observed was the full eleven. It only appears once the two +intervals differ. That alone explains a lot of "changing the settings does +nothing": at equal values you only ever get the expensive one. + +With *connection check = 10 s* and *status check = 30 s*, the steady state over +14 consecutive cycles is: + +``` +FULL at T +short at T + 10.1 +short at T + 25.3 (+15.2) +FULL at T + 30.4 (+5.1) +``` + +| dial | set to | observed | | +|---|---|---|---| +| status check | 30 s | **30.4 s** | ✅ honoured | +| connection check | 10 s | **15.2 s** | ❌ 52% slow | + +27 consecutive connection-check gaps, all 15.1–15.3 s. That is systematic, not +jitter. + +**15.2 s is exactly half of 30.4 s**, and the two are phase-locked 2:1 — which +suggests the connection check is not running on its own setting at all, but at +half the *status* period. + +⚠ **Hypothesis, not established.** It predicts sharply: with status at 60 s the +connection check should land at 30 s **whatever its dial says**. One capture +settles it, and it is worth settling before SFM offers any similar control — a +dial that silently does nothing is worse than no dial. ### ⚠ What this means for the "won't stay connected" failure From f1215ef9d90053dbcf82b3567dc69e13cadba373 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 12:50:36 -0400 Subject: [PATCH 30/43] docs(series4): REFUTE the half-period theory -- the connection dial sets a count The prediction was sharp and it was wrong. I hypothesised the connection check ran at half the status period, predicting 30 s when status is set to 60 s. Measured: 60.5 s. Recorded rather than quietly deleted. Three configurations now measured, each over many cycles: status 10 / conn 10 -> status 10.1 s conn: none ever ran 0 per cycle status 30 / conn 10 -> status 30.4 s conn 15.2 s 2 per cycle status 60 / conn 30 -> status 60.5 s conn 60.5 s 1 per cycle The status dial is honoured in all three, within ~1%. The connection dial is honoured in none. What holds across all three is a count, not a period: separate connection checks per status cycle = (status / connection) - 1 Consequences: setting the two dials equal yields ZERO connection checks, so every connection is the expensive eleven-command status read -- and that is the configuration that looks like the default. "Every 30 s" with status at 60 s gives one check per minute, half the advertised rate. No simple scale factor describes the observed cadences either (10 -> 15.2, 30 -> 60.5). Still unexplained: the phase within a cycle. At status 30 / conn 10 the two short checks landed at T+10.1 and T+25.3 where an evenly divided cycle would put them at T+10 and T+20. The count rule holds; the phase does not follow from it. Also adds a traffic table across the three configurations: 563 MB/month at 10 s/10 s, 147 MB/month at the current 60 s/30 s, against 9 MB/month for a POLL + MONITOR_STATUS check at 60 s. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 62 ++++++++++++++++++++++++---- 1 file changed, 55 insertions(+), 7 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index fa9e91d..5901d1f 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1669,14 +1669,62 @@ FULL at T + 30.4 (+5.1) 27 consecutive connection-check gaps, all 15.1–15.3 s. That is systematic, not jitter. -**15.2 s is exactly half of 30.4 s**, and the two are phase-locked 2:1 — which -suggests the connection check is not running on its own setting at all, but at -half the *status* period. +> ⚠ **A "half the status period" hypothesis was raised here and REFUTED.** It +> predicted that with status at 60 s the connection check would run at 30 s. +> Measured: **60.5 s**. Recorded rather than deleted — the prediction was sharp +> and it was wrong. -⚠ **Hypothesis, not established.** It predicts sharply: with status at 60 s the -connection check should land at 30 s **whatever its dial says**. One capture -settles it, and it is worth settling before SFM offers any similar control — a -dial that silently does nothing is worse than no dial. +### The connection dial sets a COUNT, not a period + +Three configurations, each measured over many cycles: + +| status | connection | status observed | connection observed | separate conn checks per status cycle | +|---|---|---|---|---| +| 10 s | 10 s | 10.1 s ✅ | — | **0** (none ever ran) | +| 30 s | 10 s | 30.4 s ✅ | 15.2 s ❌ | **2** | +| 60 s | 30 s | 60.5 s ✅ | 60.5 s ❌ | **1** | + +**The status dial is honoured in all three** — within about 1%. + +**The connection dial is not honoured in any of them.** What actually holds is: + +``` +separate connection checks per status cycle = (status ÷ connection) − 1 +``` + +1 → 0 checks, 3 → 2 checks, 2 → 1 check. So the field does not set a period; it +sets how many checks fit inside a status cycle, and it is off by one. + +Consequences worth naming: + +- **Setting the two dials equal yields zero connection checks.** Every + connection is then the expensive eleven-command status check. That is the + default-looking configuration, and it is the worst one. +- **"Every 30 s" with status at 60 s gives one check per minute** — half the + advertised rate. +- The observed cadences are not multiples of the connection setting at all + (10 s → 15.2 s; 30 s → 60.5 s), so no simple scale factor describes it either. + +⚠ The *timing within* a cycle is still not fully explained. At status 30 / +connection 10 the two short checks landed at T+10.1 and T+25.3, where an evenly +divided cycle would put them at T+10 and T+20. The count rule holds across all +three configurations; the phase does not follow from it. + +**For SFM:** this is the concrete argument against exposing a dial whose +behaviour is not what it says. If a control sets an interval, it should set that +interval — and if the effective rate is derived from something else, the UI +should show the derived value, not the input. + +### Traffic at each configuration + +| status / connection | per minute | per day | per month | +|---|---|---|---| +| 10 s / 10 s | ~13 KB | **18.8 MB** | **563 MB** | +| 60 s / 30 s (current) | ~3.4 KB | **4.9 MB** | **147 MB** | +| a minimal check at 60 s | ~0.2 KB | **0.3 MB** | **9 MB** | + +The last row is `POLL` + `MONITOR_STATUS` only — enough to answer "alive?" and +"monitoring?". ### ⚠ What this means for the "won't stay connected" failure From 04b1ef3e04dcdd603ebc89a6c811e24deb0b2f25 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 13:07:12 -0400 Subject: [PATCH 31/43] docs(series4): REPRODUCED -- THOR stops polling after a drop mid-download The field failure on UM12947 ("wouldn't stay connected, refresh did nothing, no way to view a connection attempt") reproduced on the bench, with timestamps. THOR was mid-bulk-download when the link was faulted. Sequence: 13:02:50 connection dies mid-transfer (165 BULK_DOWNLOAD frames in) 13:03:10 THOR reconnects once after 20 s, sends SUB_1F to resume, fails 13:03:33 link fully restored and healthy 13:04:21 operator clicks Refresh -> full 11-command check, all correct 13:06:27 still nothing. That refresh is the ONLY connection since 13:03:10. Before the fault THOR had connected every 30 s without a miss for over an hour. Establishes four things: * one retry then give up -- no backoff, no further attempts * the automatic poll loop dies too, not just the download * it does not recover when the link returns (3 min of healthy link, nothing) * Refresh works but only once -- it does NOT restart the automatic loop The fourth is the dangerous one: Refresh makes the UI report a healthy unit while nothing is watching it. Silent failure that looks like success. It also explains why the field symptom resists characterisation -- the unit is reachable the whole time; THOR has simply stopped asking and says nothing about it. CAVEAT, recorded prominently: what THOR experienced was a TCP close mid-download, not the silent link intended. mm_link.py mistook socket.timeout (which subclasses OSError) for a closed socket, so 200 ms of quiet closed the connection -- the relay killed the link it was meant to be faking a fault on. Fixed in this commit. The run stands as a drop-mid-download test, arguably the more realistic case. Single trial; true blackhole and clean drop not yet tested. Adds four design consequences for SFM: unbounded retry with backoff; a manual check must restart the automatic loop or the UI must say it is stopped; surface the poll loop's own state (last success, last attempt, next attempt, consecutive failures -- all four invisible here); and distinguish "unit unreachable" from "we stopped checking", which present identically in THOR. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- bridges/mm_link.py | 10 +++- docs/micromate_protocol_reference.md | 80 ++++++++++++++++++++++++++++ 2 files changed, 89 insertions(+), 1 deletion(-) diff --git a/bridges/mm_link.py b/bridges/mm_link.py index a1724d2..8a72bfc 100644 --- a/bridges/mm_link.py +++ b/bridges/mm_link.py @@ -236,11 +236,19 @@ class Link: arrow = "THOR->unit" if is_request else "unit->THOR" last = time.time() while not stop.is_set(): + timed_out = False try: data = src.recv(4096) if isinstance(src, socket.socket) else src.read(4096) + except TimeoutError: + timed_out = True + # socket.timeout subclasses OSError, so it MUST be caught first. + # Treating it as a dead socket closes the connection after 200 ms + # of quiet -- which is exactly what `blackhole` produces, so the + # relay killed the link it was supposed to be faking a fault on. + data = b"" except OSError: break - if isinstance(src, socket.socket) and data == b"": + if isinstance(src, socket.socket) and data == b"" and not timed_out: self.say(f"{arrow}: peer closed the connection") break if not data: diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 5901d1f..f058996 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1758,6 +1758,86 @@ Three things this argues for, all cheap: 3. **Bound every operation with its own timeout**, independent of TCP's. Do not rely on the socket to report a dead peer. +## Reproduced: THOR stops polling after a connection drops mid-download + +**2026-09-25, one trial, on the bench link.** This is the failure the operator +described in the field on UM12947 — *"just wouldn't stay connected… clicking the +little refresh button did nothing… no way to view the status of a connection +attempt."* + +### What was done + +THOR was downloading events. Partway through the bulk transfer the link was +faulted, the connection died, and the link was then fully restored. + +⚠ **What THOR actually experienced was a TCP close mid-download**, not the silent +link that was intended — `mm_link.py` had a bug (`socket.timeout` subclasses +`OSError`, so 200 ms of quiet was mistaken for a closed socket). Fixed, but the +run stands as a **drop-mid-download** test, which is arguably the more realistic +case: a modem losing its tower often does produce a real close. + +### What happened + +``` +13:02:49.0 THOR connects, bulk download running (165 BULK_DOWNLOAD frames in) +13:02:50.0 link faulted -> connection dies mid-transfer +13:03:10.4 THOR reconnects once, 20 s later, sends SUB_1F (advance event + pointer) — an attempt to resume +13:03:10.8 that attempt dies too +13:03:33.3 link fully restored and healthy + ... 48 s of complete silence ... +13:04:21.4 operator clicks Refresh -> full 11-command check, every response + correct, unit perfectly healthy + ... 2 min 6 s of complete silence ... +13:06:27 still nothing. That refresh is the ONLY connection since 13:03:10. +``` + +Before the fault THOR was connecting every 30 s without a miss for over an hour. + +### What this establishes + +1. **One retry, then give up.** THOR retried once after 20 s and stopped. No + backoff, no further attempts. +2. **The automatic poll loop dies with it.** Not just the download — the 30 s + connection check and 60 s status check both stopped entirely. +3. **It does not recover when the link returns.** Three minutes of healthy link + produced zero connection attempts. +4. **Refresh works, but only once.** The manual check succeeded completely — + all eleven commands, correct responses. It did **not** restart the automatic + loop; two minutes later there was still nothing. + +⚠ **Point 4 is the dangerous one.** Refresh makes the UI report a healthy unit +while nothing is actually watching it. The operator sees a successful check and +moves on; the background loop stays dead. That is worse than a refresh button +that plainly fails, because the failure is *silent and looks like success*. + +It also explains why the field symptom is hard to characterise: the unit is +perfectly reachable the whole time. Poke it and it answers. Nothing is wrong +with the unit, the modem, or the link — THOR has simply stopped asking, and says +nothing about it. + +### Design consequences for SFM + +1. **Retry must be unbounded with backoff**, never one-and-done. A transient + link fault should not take a unit out of service until a human notices. +2. **A manual check must restart the automatic loop**, or the UI must state + clearly that automatic checking is stopped. A green tick that means "it + worked when you asked" is a trap. +3. **Surface the poll loop's own state** — last successful check, last attempt, + next scheduled attempt, consecutive failures. All four were invisible here, + and every one of them would have made this diagnosable in seconds. +4. **Distinguish "unit unreachable" from "we stopped checking."** They present + identically in THOR and have completely different causes and fixes. + +### Caveats + +- **One trial.** Not yet repeated, and not yet tried with a true silent link + (`blackhole` with the fixed tool) or with a clean `drop`. +- THOR's UI state during the dead window was not recorded — worth capturing on a + repeat, since what the operator *sees* is half the problem. +- Whether a THOR restart is required, or whether it eventually recovers on a + longer timescale, was not tested. + ## Thor's conventions vs the protocol's requirements **SFM is not meant to reimplement Thor.** Thor is the only available teacher of From eccd28ea0d4f1dfa3d05b01194c531f3bbad22b0 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 13:11:18 -0400 Subject: [PATCH 32/43] docs(series4): "not polling" and "unreachable" are different states Follow-up to the reproduced wedge, and the sharpest point to come out of it -- the operator's observation, demonstrated directly. At 13:10:44, six and a half minutes after THOR's last connection of any kind: THOR connections since 13:04:22 : 0 independent POLL through the same relay, same moment : succeeds The unit is reachable. The link is fine. THOR is simply not asking. So whatever THOR's UI reports in that window is wrong. "Connected/OK" is false because nothing has been checked for minutes. "Disconnected/unreachable" is also false because the unit answers on demand. The true state -- "I have given up checking this unit" -- is not one THOR can display. An operator therefore cannot separate "the unit is down" from "the poller is asleep", and those demand completely different responses: a site visit versus a mouse click. This settles a question the previous commit left open. It does not matter much whether stopping after one retry is intentional or a defect: the reporting is wrong either way, since both plausible displays misrepresent reality. Sharpens design consequence 4 accordingly -- a unit's displayed state should be one of: checks passing, checks failing (with attempt count and last error), or not being checked (with why, and a way to resume). Collapsing the last two into a single indicator is the root of this failure being undiagnosable. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 44 ++++++++++++++++++++++++---- 1 file changed, 38 insertions(+), 6 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index f058996..9aaf771 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1811,10 +1811,38 @@ while nothing is actually watching it. The operator sees a successful check and moves on; the background loop stays dead. That is worse than a refresh button that plainly fails, because the failure is *silent and looks like success*. -It also explains why the field symptom is hard to characterise: the unit is -perfectly reachable the whole time. Poke it and it answers. Nothing is wrong -with the unit, the modem, or the link — THOR has simply stopped asking, and says -nothing about it. +### 🔑 "Not polling" and "unreachable" are different states — and THOR conflates them + +Demonstrated directly. At **13:10:44**, six and a half minutes after THOR's last +connection of any kind: + +``` +THOR connections since 13:04:22 : 0 +independent POLL through the same relay, same moment : succeeds, correct response +``` + +**The unit is reachable. The link is fine. THOR is simply not asking.** + +Whatever THOR's UI reports in that window, it is wrong: + +| if it shows | why it is false | +|---|---| +| connected / OK | nothing has been checked for minutes | +| disconnected / unreachable | it **is** reachable — THOR stopped trying | + +The true state is *"I have given up checking this unit"*, and no display in THOR +expresses it. So an operator cannot separate **the unit is down** from **the +poller is asleep** — and those demand completely different responses: a site +visit versus a mouse click. + +This also settles a question the previous section left open. It does not much +matter whether stopping after one retry is an intentional design choice or a +defect: **the reporting is wrong either way**, because both plausible displays +misrepresent reality. + +It explains, too, why the field symptom resisted characterisation. The unit is +perfectly reachable throughout. Poke it and it answers. Nothing is wrong with +the unit, the modem, or the link. ### Design consequences for SFM @@ -1826,8 +1854,12 @@ nothing about it. 3. **Surface the poll loop's own state** — last successful check, last attempt, next scheduled attempt, consecutive failures. All four were invisible here, and every one of them would have made this diagnosable in seconds. -4. **Distinguish "unit unreachable" from "we stopped checking."** They present - identically in THOR and have completely different causes and fixes. +4. **Distinguish "unit unreachable" from "we stopped checking."** Proven above + to be different states that THOR cannot tell apart. A unit's displayed state + should be one of: *checks passing*, *checks failing* (with the attempt count + and last error), or *not being checked* (with why, and a way to resume). + Collapsing the last two into one indicator is the root of this whole failure + being undiagnosable. ### Caveats From cfc715c02f5d94652ae348fbd8a653b3e8b50e8f Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 13:15:39 -0400 Subject: [PATCH 33/43] docs(series4): THOR shows "Connected" for 11 minutes after it stopped checking Closes the question the last two commits left open -- what the UI actually shows during the dead window. It is the bad case. Captured at 13:15, eleven minutes after THOR's last contact with the unit: Connection Status : Connected <- false Last Updated : 09/25/2026 01:04:22 PM <- true, and that is the REFRESH Notification : "Unable to download event(s) ... Did not receive response from unit." (01:03:31 PM) Three separable points: * The green tile is false -- it reports a live connection not exercised for eleven minutes. * Last Updated is TRUE, and is the only honest field on the screen. THOR knows when it last succeeded; it renders that as small grey text under the unit name, unhighlighted and unmarked as stale, beneath a large green Connected tile. The operator must read a timestamp and do arithmetic to find out the headline is wrong. * The failure THOR did report was the DOWNLOAD, not the poller stopping. The two are treated as unrelated; nothing states that automatic checking ceased. So the state is not merely undisplayed: THOR holds the data that would reveal it and presents a contradicting summary instead. Adds design consequence 0, ahead of the others because it is the highest-value fix and the cheapest: connection status must EXPIRE. If the last successful check is older than a small multiple of the interval, the state is stale/unknown, never Connected. THOR already has the timestamp; it just does not let it invalidate the summary. Same screen independently corroborates four of our decodes: memory 14.94/15.00 MB against the exact 15,000,000-byte total from SUB 0x1C; Unit Date/Time 01:04:20 PM against the device clock at 0x1C data[13:21]; Scheduler Enabled against 0x47; and Auto Call Home Disabled against the write[5]=0x04 observed in the 0x7E capture. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 35 ++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 7 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 9aaf771..222a595 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1823,15 +1823,32 @@ independent POLL through the same relay, same moment : succeeds, correct respons **The unit is reachable. The link is fine. THOR is simply not asking.** -Whatever THOR's UI reports in that window, it is wrong: +**THOR reports `Connection Status: Connected`.** Captured at 13:15, eleven +minutes after its last contact with the unit: -| if it shows | why it is false | -|---|---| -| connected / OK | nothing has been checked for minutes | -| disconnected / unreachable | it **is** reachable — THOR stopped trying | +``` +Connection Status : Connected <- false; nothing checked in 11 min +Last Updated : 09/25/2026 01:04:22 PM <- true, and that is the REFRESH +Notification : "Unable to download event(s) from UM12947 due to: + Did not receive response from unit." (01:03:31 PM) +``` -The true state is *"I have given up checking this unit"*, and no display in THOR -expresses it. So an operator cannot separate **the unit is down** from **the +Three things worth separating: + +1. **The green tile is false.** It reports a live connection that has not been + exercised for eleven minutes. +2. **`Last Updated` is true** — and it is the only honest field on the screen. + THOR *knows* when it last succeeded. It renders that as small grey text under + the unit name, unhighlighted, with nothing marking it stale, beneath a large + green **Connected** tile. The operator must read a timestamp and do mental + arithmetic to discover the headline is wrong. +3. **The failure THOR did report was the download**, not the poller stopping. + Those are treated as unrelated events; nothing says "automatic checking has + ceased." + +So the state *"I have given up checking this unit"* is not merely undisplayed — +THOR holds the data that would reveal it and presents a contradicting summary +instead. So an operator cannot separate **the unit is down** from **the poller is asleep** — and those demand completely different responses: a site visit versus a mouse click. @@ -1846,6 +1863,10 @@ the unit, the modem, or the link. ### Design consequences for SFM +0. **Connection status must EXPIRE.** The single highest-value fix here. If the + last successful check is older than a small multiple of the check interval, + the state is *stale / unknown* — never *Connected*. THOR already has the + timestamp; it simply does not let it invalidate the summary. 1. **Retry must be unbounded with backoff**, never one-and-done. A transient link fault should not take a unit out of service until a human notices. 2. **A manual check must restart the automatic loop**, or the UI must state From d93ebe2322c93f37d946e074b06879072186815d Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 13:20:52 -0400 Subject: [PATCH 34/43] docs(series4): correct a sloppy count -- the conclusion holds, the evidence did not The claim "THOR connections since 13:04:22 : 0" was an artifact of a time filter (13:0[4-9]) that silently dropped everything from 13:10 onward. THOR had in fact connected four times, at 13:10:03-13:10:37. Recorded rather than quietly fixed: the number was stated as proven and it was wrong. Inspecting those four connections shows the conclusion survives, for a better reason than the one originally given. They were user-initiated commands, not polling: 13:10:03 POLL -> SUB_96 (start monitoring) 13:10:05 MONITOR_STATUS 13:10:35 POLL -> SUB_97 (stop monitoring) 13:10:37 MONITOR_STATUS Single commands a human clicked, each followed by one status read. Neither the eleven-command status check nor the three-command connection check appears anywhere in the window. So automatic polling still has not resumed since 13:03:10 -- 16 minutes by 13:19:53 -- and every THOR connection in that window was operator-initiated: the refresh at 13:04:21, then start and stop monitoring at 13:10. Updates finding 3 from "three minutes of healthy link produced zero connection attempts" to the stronger and now properly-evidenced "sixteen minutes produced no AUTOMATIC attempts at all, only ones initiated by hand". Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 222a595..324faf9 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1800,8 +1800,9 @@ Before the fault THOR was connecting every 30 s without a miss for over an hour. backoff, no further attempts. 2. **The automatic poll loop dies with it.** Not just the download — the 30 s connection check and 60 s status check both stopped entirely. -3. **It does not recover when the link returns.** Three minutes of healthy link - produced zero connection attempts. +3. **It does not recover when the link returns.** Sixteen minutes of healthy + link produced **no automatic connection attempts at all** — only ones an + operator initiated by hand. 4. **Refresh works, but only once.** The manual check succeeded completely — all eleven commands, correct responses. It did **not** restart the automatic loop; two minutes later there was still nothing. @@ -1813,14 +1814,29 @@ that plainly fails, because the failure is *silent and looks like success*. ### 🔑 "Not polling" and "unreachable" are different states — and THOR conflates them -Demonstrated directly. At **13:10:44**, six and a half minutes after THOR's last -connection of any kind: +Demonstrated directly. At **13:10:44** an independent `POLL` through the same +relay succeeded with a correct response while THOR's automatic polling was dead. + +⚠ **A correction, recorded because the first version of this claim was sloppy.** +It originally read "THOR connections since 13:04:22 : 0". That count was an +artifact of a time filter (`13:0[4-9]`) that silently dropped everything from +13:10 onward. THOR *had* connected four times, at 13:10:03–13:10:37. + +Inspecting those four shows the conclusion survives — but for a better reason +than the one first given. They were **user-initiated commands, not polling**: ``` -THOR connections since 13:04:22 : 0 -independent POLL through the same relay, same moment : succeeds, correct response +13:10:03 POLL → SUB_96 (start monitoring) 13:10:05 MONITOR_STATUS +13:10:35 POLL → SUB_97 (stop monitoring) 13:10:37 MONITOR_STATUS ``` +Single commands a human clicked, each followed by one status read. Neither the +eleven-command status check nor the three-command connection check appears. + +**Automatic polling did not resume**, and by 13:19:53 had been absent for +**16 minutes**. Every THOR connection in that window was something an operator +clicked: the refresh at 13:04:21, then start and stop monitoring at 13:10. + **The unit is reachable. The link is fine. THOR is simply not asking.** **THOR reports `Connection Status: Connected`.** Captured at 13:15, eleven From 8b6c89da42f1a4ae616dd7fb58cc757918574c66 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 13:21:49 -0400 Subject: [PATCH 35/43] docs(series4): a live monitoring unit displayed as Idle -- and a flag I got wrong The operator's test: start monitoring from the unit's keypad and see whether THOR notices. It does not. At 13:21:15, verified by an independent probe through the same relay: unit on the wire 0x49 data[11]=0x02, 0x1C data[12]=0x0c -> MONITORING THOR last contact 13:10:37 (a stop-monitoring command it issued itself) THOR display Connected . Monitoring Mode: Idle . Last Updated 1:04:22 A unit is actively recording and THOR shows it as Idle. Had an event triggered in that window THOR would not have known and would not have collected it. That is the operational consequence of the wedge -- not a wrong indicator, but a monitoring system that has silently stopped monitoring its monitor. Further detail: THOR DID contact the unit at 13:10:37 and read MONITOR_STATUS, yet Last Updated still reads 1:04:22. A successful exchange does not refresh that timestamp; only the full status check does. The one honest field on the screen is honest about the wrong thing, which makes it useless as a staleness indicator exactly when staleness is the problem. CORRECTS a documented constant. SUB 0x1C data[12] was recorded as "0x0E monitoring / 0x00 idle". It read 0x0E on 2026-09-24 and 0x0C on 2026-09-25, both while monitoring, so it carries sub-state in its low bits and is not a flag. Test for NON-ZERO, never against a constant -- an implementation comparing to 0x0E would have reported this unit idle. Same caution noted for 0x49 data[11], which has only ever been seen as 0x02. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 36 +++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 324faf9..e948a44 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1091,7 +1091,7 @@ zero-data response. Thor follows each with a `SUB 0x1C` status read to confirm. ### `SUB 0x1C` — monitor status, 55-byte data ``` -data[12] monitoring flag 0x0E monitoring / 0x00 idle +data[12] monitoring flag NON-ZERO = monitoring, 0x00 = idle data[13] day data[14] month data[15:17] year, uint16 BE @@ -1103,8 +1103,11 @@ data[-8:-4] memory total, uint32 BE = 15,000,000 bytes exactly data[-4:] memory free, uint32 BE ``` -⚠ **The monitoring flag is `0x0E`, not Series III's `0x10`.** Do not reuse the -Series III constant. Only five bytes differ between the monitoring and idle +⚠ **Test `data[12]` for non-zero — do NOT compare against a constant.** It read +`0x0E` on 2026-09-24 and `0x0C` on 2026-09-25, both while monitoring, so it +carries sub-state in its low bits rather than being a flag. Series III's `0x10` +does not apply either. `SUB 0x49` `data[11]` read `0x02` in every monitoring +sample so far, but the same caution applies until more values are seen. Only five bytes differ between the monitoring and idle responses: the flag, `data[17]`, the clock, and the memory-free field. Memory free dropped by exactly **4,096 bytes** across the ~70-second monitoring @@ -1877,6 +1880,33 @@ It explains, too, why the field symptom resisted characterisation. The unit is perfectly reachable throughout. Poke it and it answers. Nothing is wrong with the unit, the modem, or the link. +### The stale display extends to monitoring state — a live unit shown as Idle + +The operator's own test: start monitoring **from the unit's keypad**, and see +whether THOR ever notices. + +At **13:21:15**, verified by an independent probe through the same relay: + +``` +unit, on the wire SUB 0x49 data[11] = 0x02 MONITORING + SUB 0x1C data[12] = 0x0c MONITORING + device clock 13:21:14 +THOR's last contact 13:10:37 (a stop-monitoring command it issued itself) +THOR's display Connected · Monitoring Mode: Idle · Last Updated 1:04:22 PM +``` + +**A unit is actively recording and THOR shows it as Idle.** Had an event +triggered in that window, THOR would not have known and would not have collected +it. This is the operational consequence of the wedge: not merely a wrong +indicator, but a monitoring system that has silently stopped monitoring its +monitor. + +One further detail. THOR **did** contact the unit at 13:10:37 and read +`MONITOR_STATUS` — yet `Last Updated` still reads 1:04:22. So a successful +exchange does not refresh that timestamp; only the full status check does. The +single honest field on the screen is honest about the wrong thing, which makes it +useless as a staleness indicator precisely when staleness is the problem. + ### Design consequences for SFM 0. **Connection status must EXPIRE.** The single highest-value fix here. If the From 992b84df51a769b4b980212a3051c47eab35a445 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 13:47:42 -0400 Subject: [PATCH 36/43] docs(series4): modem settings live in SysParm.cfg, not the setup -- deliberately Settled by a clean experiment. The call-modem setting was changed on the keypad from 'generic' to 'USB to PC', then the ACTIVE SETUP was switched from TEST1 to test2 on the device. The setting did not change. So it is device-global, in the SysParm.cfg the firmware strings already hinted at alongside SysPref.bMonitorScheduler -- not a per-setup field. That closes a hypothesis this document was chasing: the ~102 bytes a .MMB file carries beyond the 2,090-byte config block are NOT where this lives. Those bytes remain unexplained but are no longer a candidate. CORRECTS an over-reading in an earlier commit. "The call-home block (0x2C) was byte-identical across 218 samples today" does not bear on this question: the last 0x2C sample was at 13:30:23 and the keypad change came around 13:35, so no sample exists on the far side of it. Why the split is right, and the operator's reading of it: you do not want modem settings reachable remotely, because getting them wrong over the air destroys the connection you would need to put them back, and the unit must then be visited. So SUB 0x2C is not an incomplete view of the modem configuration -- it is the deliberately-chosen subset that is SAFE to change remotely (enable, dial string, retries, timings), and the unreachable remainder is unreachable on purpose. Records the design principle for SFM: for settings whose misconfiguration destroys the channel you would use to fix them, either do not expose them for remote write, or require commit/confirm with automatic rollback (apply, require a call-back within N minutes, revert otherwise). Always allow READING them, so an operator can diagnose a unit they cannot reconfigure. Instantel chose the first option and given the failure mode that is worth copying rather than improving on. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 52 ++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index e948a44..53e12b8 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1677,6 +1677,58 @@ jitter. > Measured: **60.5 s**. Recorded rather than deleted — the prediction was sharp > and it was wrong. +### Modem settings live in `SysParm.cfg`, not in the setup — and that is deliberate + +**Settled by experiment (2026-09-25).** The unit's *call modem* setting was +changed on the keypad from `generic` to `USB to PC`. Then the **active setup was +switched from `TEST1` to `test2` on the device**, and the setting was checked +again: **unchanged.** + +So it does not travel with the setup. It is device-global, in the `SysParm.cfg` +the firmware strings already hinted at, alongside `SysPref.bMonitorScheduler`. + +That closes a hypothesis this document was pursuing: the ~102 bytes a `.MMB` +**file** carries beyond the 2,090-byte config block are **not** where this lives. +Those bytes remain unexplained, but they are no longer a candidate for it. + +⚠ It also means the earlier observation "the call-home block (`0x2C`) was +byte-identical across 218 samples today" **does not bear on this** — the last +`0x2C` sample was at 13:30:23 and the keypad change came around 13:35. No sample +exists on the far side of it. + +#### Why the split is the right design + +The operator's reading, and it is convincing: **you do not want modem settings +reachable remotely.** Get them wrong over the air and you lose the connection +you would need to put them back — the unit must then be visited physically. + +That explains the shape of the protocol rather than treating it as an oddity: + +| | reachable over the wire? | +|---|---| +| Call Home **enable**, dial string, retries, timings (`0x2C` / `0x7E`) | **yes** — recoverable if wrong | +| **Modem type** and the rest of `SysParm.cfg` | **no** — a wrong value is unrecoverable remotely | + +`SUB 0x2C` is not an incomplete view of the modem configuration. It is the +deliberately-chosen subset that is **safe to change remotely**, and the +unreachable remainder is unreachable on purpose. + +#### Design principle for SFM + +There is a class of setting whose misconfiguration **destroys the channel you +would use to fix it** — modem type, baud, serial parameters, the call-home +destination itself. For those: + +1. **Do not expose them for remote write**, or gate them behind an explicit + confirmation that names the risk. +2. If they must be writable, use a **commit/confirm with automatic rollback** — + apply, require the unit to call back within N minutes, revert if it does not. +3. **Always allow reading them**, so an operator can diagnose a unit they cannot + reconfigure. + +Instantel chose option 1. Given the failure mode, that is defensible, and worth +copying rather than improving on. + ### The connection dial sets a COUNT, not a period Three configurations, each measured over many cycles: From a404d7079115126d29ca88850410fc1b030ff996 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 14:13:36 -0400 Subject: [PATCH 37/43] docs(series4): THOR leaks event subscriptions -- evidence from its own log The office THOR PC's log for 2026-09-22 contains a textbook WPF defect, visible without any instrumentation. 85% of a day's logging is one message: "UnitOperatingModeViewModel - Start/stop monitoring request timed out: False", 178 of 209 lines. Only 12 lines describe an actual operation. Grouping that message by exact timestamp to the millisecond -- so each group is ONE logical event -- the count per event grows over the day: 11:48 1-4 13:36-13:40 2-4 15:16 3 16:03-16:07 6 22:00-22:06 12 1 -> 12 over ten hours of uptime. Twelve identical lines sharing a single millisecond is not twelve events; it is one event dispatched to twelve handlers. That is a subscription leak, and the class name identifies it. THOR is .NET/WPF ("App thread", ViewModel naming), where a view model subscribing on view-open and never unsubscribing on view-close is the archetypal case. Consequences that follow directly: N grows without bound with uptime and usage; every notification does N times the work; and a restart resets N to 1 -- matching the operator's report that only restarting recovers a degraded session. The only five "timed out: True" entries in the file sit at the very top, an episode caught just before rotation. Claim discipline stated explicitly in the doc. ESTABLISHED: the handler count grows. STRONG INFERENCE: it is a subscription leak. NOT ESTABLISHED: that it caused the 2026-09-22 field outage -- this log does not cover that window and the link between leaked handlers and a dead TCP path is unproven. Includes a ten-minute confirmation procedure: restart THOR, note the burst size, open and close a unit detail view ten times, re-count. Three lessons for SFM: unsubscribe on teardown or use weak events; log once per event rather than once per handler; and log what CHANGED -- 178 "timed out: False" lines are noise that buried the five that mattered, which is plausibly why this went unnoticed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 78 ++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 53e12b8..ce34348 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1989,6 +1989,84 @@ useless as a staleness indicator precisely when staleness is the problem. - Whether a THOR restart is required, or whether it eventually recovers on a longer timescale, was not tested. +## THOR leaks event subscriptions — evidence from its own log + +Source: `thor.log.2026-09-22` from the office THOR PC, supplied 2026-09-25. +209 lines covering roughly ten hours. + +### What the log contains + +``` +178x UnitOperatingModeViewModel - Start/stop monitoring request timed out: False + 6x UnitOperatingModeViewModel - Callback of start/stop monitor request received + 5x UnitOperatingModeViewModel - Start/stop monitoring request timed out: True + 3x SchedulerViewModel - Retrieving unit setup ... + 2x BWLicenseManager - License State: IsKeyValid 0, IsActivated 0, ... +``` + +85% of a day's logging is one message. Only **12** lines describe an actual +operation. + +### The message is emitted N times for ONE event, and N grows + +Grouping the repeated message by **exact timestamp to the millisecond** — so +each group is one logical event, not repeated events: + +| time | identical lines | +|---|---| +| 11:48 | 1–4 | +| 13:05 – 13:31 | 1–2 | +| 13:36 – 13:40 | 2–4 | +| 15:16 | 3 | +| 16:03 – 16:07 | **6** | +| 16:14 – 16:16 | 4–6 | +| **22:00 – 22:06** | **12** | + +**1 → 12 over ten hours of uptime.** Twelve identical lines sharing one +millisecond timestamp is not twelve events; it is one event dispatched to twelve +handlers. + +### What it is + +A **subscription leak**, and the class name identifies it: +`UnitOperatingModeViewModel`. THOR is a .NET/WPF application ("App thread", +ViewModel naming), where this is the archetypal leak — a view model subscribes to +an event when its view opens and never unsubscribes when the view closes. Each +open adds a live handler; every later notification runs them all. + +Consequences that follow directly: + +- N grows with **uptime and usage**, without bound. Ten hours of moderate use + reached 12×; a THOR left running for days across a fleet will be far worse. +- Every notification performs N× the work — N timers, N callbacks, N state + updates racing one another. +- **A restart resets N to 1**, which is consistent with the operator's report + that only restarting THOR recovers a degraded session. +- The only `timed out: True` entries in the file — five of them — sit at the very + top, an episode captured just before log rotation. + +⚠ **Claim discipline.** *Established* from this log: the handler count grows. +*Strong inference*: it is a subscription leak. **Not established**: that it +caused the specific field outage on 2026-09-22. This log does not cover that +window, and the connection between leaked handlers and a dead TCP path is +unproven. + +### How to confirm it in ten minutes + +Restart THOR, note the burst size for one event, open and close a unit's detail +view ten times, then trigger another event and re-count. If the burst grows, the +leak is reproducible and attributable to a specific UI action. + +### For SFM + +1. **Unsubscribe on teardown**, or use weak event patterns. This is the single + most common long-running-UI defect and it is entirely preventable. +2. **Log once per event, not once per handler.** The duplication here is what + made the leak visible, which is lucky — but a log that is 85% one repeated + line is also a log nobody reads, which is why this went unnoticed for a year. +3. **Log what changed, not what did not.** `timed out: False` 178 times is + noise; the five `True` entries are the signal, and they are buried. + ## Thor's conventions vs the protocol's requirements **SFM is not meant to reimplement Thor.** Thor is the only available teacher of From 704cd7b111690a85a91dc80cc49943d950945826 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 14:50:44 -0400 Subject: [PATCH 38/43] feat(bridges): mm_probe -- tell apart the four faults THOR calls "disconnected" THOR reports every failed connection as "disconnected" and nothing more. That one word covers at least four distinct faults with four different fixes, and telling them apart is the difference between a modem reboot and a site visit. Nobody had a way to do that during the 2026-09-22 outage, which is the actual gap that incident exposed -- not a missing THOR feature, but a missing tool. connection refused something answered and said no -- wrong port, or the modem refusing a further session connect timed out nothing answered -- trusted-IP whitelist, firewall, or the modem is off the network. A whitelist DISCARDS rather than refuses, so this is its signature connected, no reply the MODEM answered but the unit did not. TCP is fine; the modem is not forwarding to serial. This is what a wedged transparent-TCP session looks like, and it is the case THOR cannot distinguish from the others replied the unit is alive; the fault is upstream software Each verdict prints what to try next. The no-reply case points at ACEmanager's TCP Idle Timeout first, since a stale session holds a single-slot modem's only connection until that timeout frees it. --slots N opens N simultaneous connections and reports how many the far end accepts, which directly tests the single-session hypothesis against a real modem. Read-only throughout: POLL, SERIAL and the 0x49 state read -- the same three commands THOR's own connection check uses. Sends the correct per-SUB data offsets (POLL 0x0030, SERIAL 0x000A, 0x49 0xFFFF); offset 0 returns only the short probe reply. Works for both series and says which answered: a Series III reply opens DLE STX, a Micromate reply opens with a bare STX. Verified against UM12947 through the bench relay, and against a closed port for the refused path. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- bridges/mm_probe.py | 226 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 bridges/mm_probe.py diff --git a/bridges/mm_probe.py b/bridges/mm_probe.py new file mode 100644 index 0000000..8ba7992 --- /dev/null +++ b/bridges/mm_probe.py @@ -0,0 +1,226 @@ +#!/usr/bin/env python3 +""" +mm_probe.py — answer "why can't we reach this unit?" in one command. + +THOR reports a failed connection as "disconnected" and nothing else. That single +word covers at least four completely different faults with four different fixes, +and telling them apart is the difference between a modem reboot and a site visit: + + * **connection refused** something answered and said no — wrong port, or the + modem is refusing a further session + * **connect timed out** nothing answered at all — trusted-IP whitelist, + firewall, or the modem is off the network + * **connected, no reply** the MODEM answered but the unit did not. The TCP + path is fine; the modem is not forwarding to serial. + This is the signature of a wedged transparent-TCP + session, and it is the one THOR cannot distinguish + from any of the others + * **replied** the unit is alive; the problem is upstream software + +Read-only. It sends `POLL`, then optionally `SERIAL` and the state read — the +same three commands THOR's own connection check uses — and never writes. + +Usage +----- + python3 bridges/mm_probe.py 63.45.161.30:9034 + python3 bridges/mm_probe.py 10.0.0.8:12345 --timeout 5 + python3 bridges/mm_probe.py --slots 3 + +`--slots N` opens N connections at once and reports how many the far end accepts. +A transparent-TCP modem typically serves **one** session; if the first succeeds +and the rest are refused or hang, that confirms the single-slot behaviour and +explains why a leaked session takes a unit offline until the slot frees. + +Works for both series: a Series III reply opens `DLE STX`, a Micromate reply +opens with a bare `STX`, so the probe also tells you which one answered. +""" + +from __future__ import annotations + +import argparse +import socket +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from minimateplus.framing import build_bw_frame # noqa: E402 + +DLE, STX, ETX = 0x10, 0x02, 0x03 + + +def destuff(raw: bytes) -> bytes: + """Strip framing and DLE escapes; return the payload without its checksum.""" + i = 1 if raw and raw[0] == STX else (2 if len(raw) > 1 and raw[1] == STX else 0) + out = bytearray() + while i < len(raw): + b = raw[i] + if b == DLE and i + 1 < len(raw): + out.append(raw[i + 1]) + i += 2 + continue + if b == ETX: + break + out.append(b) + i += 1 + return bytes(out[:-1]) if len(out) > 1 else b"" + + +# Reads are two-step on Series III: a probe at offset 0, then a data read at the +# block's length. THOR sends these offsets, and they also work on a Micromate. +OFFSETS = {0x5B: 0x0030, 0x15: 0x000A, 0x49: 0xFFFF} + + +def exchange(sock: socket.socket, sub: int, timeout: float) -> tuple[bytes, float]: + sock.sendall(build_bw_frame(sub, OFFSETS.get(sub, 0))) + t0 = time.time() + buf, deadline = b"", t0 + timeout + sock.settimeout(0.3) + while time.time() < deadline: + try: + chunk = sock.recv(4096) + if not chunk: + break + buf += chunk + if buf.endswith(bytes([ETX])) and len(buf) > 8: + break + except TimeoutError: + continue + except OSError: + break + return buf, time.time() - t0 + + +def step(n: int, label: str, result: str) -> None: + print(f" [{n}] {label:.<28} {result}") + + +def probe(host: str, port: int, timeout: float) -> int: + print(f"\ntarget {host}:{port} (read-only: POLL, SERIAL, state)\n") + + # ── 1. TCP ──────────────────────────────────────────────────────────── + t0 = time.time() + try: + sock = socket.create_connection((host, port), timeout=timeout) + except ConnectionRefusedError: + step(1, "TCP connect", f"REFUSED after {1000*(time.time()-t0):.0f} ms") + print("\nverdict: something answered and actively refused.") + print(" Not a silent firewall drop — the host is reachable.") + print(" Wrong port, the service is down, or the modem is refusing") + print(" an additional session because its one slot is in use.") + return 2 + except (TimeoutError, socket.timeout): + step(1, "TCP connect", f"TIMED OUT after {time.time()-t0:.1f} s") + print("\nverdict: nothing answered at all.") + print(" A silent drop, which is what a trusted-IP whitelist looks") + print(" like — it discards rather than refuses. Check the modem's") + print(" Trusted IPs (and note a VPN changes the IP you arrive from),") + print(" the firewall, and whether the modem is on the network.") + return 3 + except OSError as e: + step(1, "TCP connect", f"FAILED: {e}") + return 4 + step(1, "TCP connect", f"ok ({1000*(time.time()-t0):.0f} ms)") + + # ── 2. POLL ─────────────────────────────────────────────────────────── + try: + raw, dt = exchange(sock, 0x5B, timeout) + except OSError as e: + step(2, "POLL", f"send failed: {e}") + sock.close() + return 4 + + if not raw: + step(2, "POLL", f"NO REPLY in {timeout:.1f} s") + print("\nverdict: the MODEM answered but the unit did not.") + print(" TCP is fine end to end — something accepted the connection.") + print(" What is missing is the serial side. Most likely the modem is") + print(" not forwarding to its serial port, which is what a wedged") + print(" transparent-TCP session looks like: the slot is held by a") + print(" connection that never closed.") + print("\n Try, in order:") + print(" 1. ACEmanager -> TCP Idle Timeout. If 0/disabled, a stale") + print(" session holds the slot forever. 2 minutes is the value") + print(" this project standardised on.") + print(" 2. Reboot the modem. If that fixes it, the modem was") + print(" holding state and the timeout is the permanent fix.") + print(" 3. Check the unit's own screen — serial cable, power.") + sock.close() + return 5 + + series = "Series III (DLE STX)" if raw[0] == DLE else "Micromate (bare STX)" + step(2, "POLL", f"reply {len(raw)} B in {1000*dt:.0f} ms") + p = destuff(raw) + ok = len(p) > 3 and p[2] == 0xFF - 0x5B + step(3, "frame", f"{'valid' if ok else 'MALFORMED'}, {series}") + if not ok: + print("\nverdict: something replied, but not a seismograph.") + print(" Another service is on this port, or the modem is in a mode") + print(" that injects its own text (check Quiet Mode / AT echo).") + print(f" first bytes: {raw[:16].hex(' ')}") + sock.close() + return 6 + + # ── 3. identity + state ─────────────────────────────────────────────── + for n, (sub, label) in enumerate(((0x15, "serial"), (0x49, "state")), start=4): + try: + r, dt = exchange(sock, sub, timeout) + d = destuff(r)[5:] + if sub == 0x15: + # serial is a null-terminated run; a further field follows it + serial = bytes(d[11:]).split(b"\x00")[0] + step(n, label, serial.decode("ascii", "replace") or "(empty)") + else: + step(n, label, "MONITORING" if len(d) > 11 and d[11] else "idle") + except OSError: + step(n, label, "no reply") + + sock.close() + print("\nverdict: the unit is alive and answering.") + print(" If THOR still shows it disconnected, the fault is in THOR, not") + print(" the network or the device.") + return 0 + + +def slots(host: str, port: int, n: int, timeout: float) -> None: + print(f"\nopening {n} simultaneous connections to {host}:{port}\n") + held = [] + for i in range(n): + try: + s = socket.create_connection((host, port), timeout=timeout) + held.append(s) + step(i + 1, f"connection {i+1}", "accepted") + except ConnectionRefusedError: + step(i + 1, f"connection {i+1}", "REFUSED") + except (TimeoutError, socket.timeout): + step(i + 1, f"connection {i+1}", "timed out") + except OSError as e: + step(i + 1, f"connection {i+1}", f"failed: {e}") + print(f"\n{len(held)} of {n} accepted.") + if len(held) == 1: + print(" Single-slot behaviour confirmed — this far end serves ONE") + print(" session at a time. A connection that is never closed takes") + print(" the unit offline until the idle timeout frees the slot.") + for s in held: + s.close() + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("target", help="host:port, e.g. 63.45.161.30:9034") + ap.add_argument("--timeout", type=float, default=10.0) + ap.add_argument("--slots", type=int, metavar="N", + help="open N simultaneous connections to test single-slot behaviour") + a = ap.parse_args() + host, _, port = a.target.rpartition(":") + if not host: + ap.error("target must be host:port") + if a.slots: + slots(host, int(port), a.slots, a.timeout) + return 0 + return probe(host, int(port), a.timeout) + + +if __name__ == "__main__": + raise SystemExit(main()) From 7b2aa871bdf7d3ff8e01619946f1f778ccce290d Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 15:01:46 -0400 Subject: [PATCH 39/43] docs(series4): record a known-good RV55 config, and the office topology it reveals Read off a Micromate modem deployed and working for months. Recorded because this knowledge currently lives only in a modem's web UI, and TMI has seven working units to diff a misbehaving one against -- the fastest diagnostic available. Serial: PAD mode, 115200 8N1, flow control None. (Series III units use 38400 -- a modem moved between series needs this changed and nobody would think to.) PAD: TCP, auto-answer ON, listening port 9034, destination 50.197.32.91:12345, idle timeout 2 min, data forwarding 500 ms, MTU 1304, TCP keepalive OFF. Cellular: APN mw01.VZWSTATIC -- Verizon's static-IP APN, which confirms the fleet has fixed addresses and rules out any "the IP moved" explanation. TOPOLOGY this answers: the office ACH listener is on port 12345 at 50.197.32.91, and field modems listen on 9034 for THOR to dial in. Both were open questions. Two things worth questioning in the config, flagged as hypothesis not finding: * TCP keepalive is Off, so nothing detects a half-open session from the modem side -- the 2-minute idle timeout is the only reaper. * An idle timer can be held open indefinitely by a client that keeps writing. THOR polls every 30 s (measured). If it holds a session the unit stopped answering on and keeps writing into it, each write plausibly resets the idle timer, the session never ages out, the PAD's single slot stays occupied, and all new inbound fails -- while ACEmanager answers on its own service. Whether one-directional traffic resets that timer is NOT confirmed, and the single-slot behaviour is untested on a real modem. mm_probe --slots tests the latter directly. Adds an ordered diff checklist for a misbehaving modem, noting that "listen for connections" and "destination address" fail in opposite directions -- inbound broken with call-home working points at the listener; call-home broken with inbound working points at the destination. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 74 ++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index ce34348..ff07130 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -2067,6 +2067,80 @@ leak is reproducible and attributable to a specific UI action. 3. **Log what changed, not what did not.** `timed out: False` 178 times is noise; the five `True` entries are the signal, and they are buried. +## Known-good RV55 config for a Micromate (2026-09-25) + +Read off a unit **deployed and working for months**. This is the reference to +diff a misbehaving modem against — the fastest diagnostic available, since TMI +has seven working units and only one that misbehaves. + +### Serial interface — Port 1 + +| field | value | note | +|---|---|---| +| Mode | **PAD** | | +| **Baud rate** | **115200** | Micromate rate. Series III units use **38400** — a modem moved between series needs this changed | +| Data bits / Parity / Stop bits | 8 / None / 1 | | +| Flow control | **None** | hardware FC blocks TX when the pins are not wired | + +### PAD settings — Port 1 + +| field | value | note | +|---|---|---| +| PAD mode | TCP | | +| **Listen for connections (auto-answer)** | **On** | this is what lets THOR dial *in*. Off, and only call-home works — which would present exactly as "unreachable but it still calls home" | +| **Listening port** | **9034** | the port THOR dials | +| **Destination address** | **50.197.32.91** | the office public IP — where the unit calls home *to* | +| **Destination port** | **12345** | the office ACH listener | +| MTU | 1304 | | +| Data forwarding timeout | 500 ms | | +| **Idle timeout** | **2 min** | matches the value this project settled on for Series III | +| Connect timeout | ≈1.08 min | | +| Start only when WAN available | Off | | +| TCP reconnect on disconnect | Off (locked) | | +| **TCP keepalive** | **Off** | ⚠ see below | +| Keepalive idle / interval / count | 30 s / 10 s / 3 | configured but inactive while keepalive is Off | + +### Cellular + +APN `mw01.VZWSTATIC` — Verizon's **static-IP** APN, which confirms the fleet's +addresses are fixed and rules out any "the IP moved" explanation. IPv4 only, +MTU 1428, radio firmware VERIZON. + +### ⚠ Two things worth questioning in this config + +**TCP keepalive is Off.** Nothing detects a half-open TCP session from the +modem's side; the 2-minute **idle** timeout is the only reaper. + +**And an idle timer can be held open indefinitely by a client that keeps +writing.** THOR polls every 30 s (measured — see the polling section). If it +holds a session the unit has stopped answering on, and keeps writing into it, +each write plausibly resets the modem's idle timer. The session then never ages +out, the PAD's single slot stays occupied, and every new inbound connection +fails — while ACEmanager keeps answering on its own separate service. + +⚠ **Hypothesis.** Whether traffic in one direction resets that timer is not +confirmed, and the single-slot behaviour has not been tested on a real modem. +`bridges/mm_probe.py --slots N` tests the second part directly. + +Turning **keepalive On** would give the modem an independent way to notice a +dead peer, rather than depending on an idle timer a chatty client can hold open. +That is a change to make deliberately and one modem at a time. + +### What to diff on a misbehaving modem + +In rough order of likelihood for "worked at first, then stopped": + +1. **Idle timeout** — 0/disabled lets a stale session hold the slot forever +2. **Listen for connections** — Off means inbound never works, but call-home still does +3. **Baud rate** — 115200 for Micromate, 38400 for Series III +4. **Listening port** — must match what THOR dials +5. **Destination address/port** — wrong here breaks call-home but *not* inbound +6. **Data forwarding timeout** and **MTU** — affect framing, not reachability + +Note 2 and 5 fail in opposite directions, which is diagnostic in itself: inbound +broken with call-home working points at the listener; call-home broken with +inbound working points at the destination. + ## Thor's conventions vs the protocol's requirements **SFM is not meant to reimplement Thor.** Thor is the only available teacher of From 84ca10eb3cfb60dd9e92c864d722da1f76050e6b Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 17:48:36 -0400 Subject: [PATCH 40/43] docs(series4): the Micromate USB host is FTDI + CDC-ACM only -- not Prolific A Micromate on an RX55 was unreachable from THOR. Isolated layer by layer with bridges/mm_probe.py, and the answer turned out to be the cable. The USB-A port is a HOST port with a fixed driver set, identical in both firmware lines: CDCACM FTDI MFS (mass storage) PRINTER HUB HC USBH 11 FTDISER strings and 22 CDCACM strings in each image, and ZERO matches for prolific / pl2303 / cp210 / ch34 / silabs in either. So a Prolific PL2303 cable (VID 067b) cannot work with a Micromate on any firmware -- no driver, no enumeration, no serial path. It needs an FTDI cable (VID 0403) or CDC-ACM. The isolation method is the part worth keeping. Eliminated in turn: public IP (static APN), firewall (probe got TCP connect in 268 ms from a whitelisted source), network path, baud, the unit's modem-type setting, and THOR itself (the probe bypasses it). Then the decisive pair -- a Linux box was swapped in for the unit on the SAME cable and modem: * POLL arrived at the serial side byte-perfect * a canned reply came back over TCP, full round trip in 700 ms So the modem, PAD config, firewall and network are all clean, and the only element that changed is what sits at the end of the cable. Substituting a known-good device converts "the unit is not answering" into "the unit is not receiving" -- very different problems. Adds scratch/fake_unit.py for that. Flagged: this explains the BENCH setup conclusively. It does NOT establish the cause of the 2026-09-22 field outage, which reportedly worked at first and degraded -- a wrong cable would not do that. Kept separate until that unit's cable is identified. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 54 ++++++++++++++++++++++++++++ scratch/fake_unit.py | 33 +++++++++++++++++ 2 files changed, 87 insertions(+) create mode 100644 scratch/fake_unit.py diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index ff07130..c124e4a 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -2067,6 +2067,60 @@ leak is reproducible and attributable to a specific UI action. 3. **Log what changed, not what did not.** `timed out: False` 178 times is noise; the five `True` entries are the signal, and they are buried. +## The Micromate's USB host supports FTDI and CDC-ACM only — not Prolific + +**Established 2026-09-25**, by isolating a bench modem link layer by layer and +then confirming against both firmware images. + +### The USB-A port is a HOST port with a fixed driver set + +``` +usbHostDelete_CDCACM CDC-ACM — the standard USB-serial class +usbHostDelete_FTDI FTDI chips (FTDISER handlers, set_line_coding, etc.) +usbHostDelete_MFS mass storage (thumb drives) +usbHostDelete_PRINTER printers +usbHostDelete_HUB / HC / USBH +``` + +**Identical in `11.0CB` and `11.0BD`** — 11 `FTDISER` strings and 22 `CDCACM` +strings in each, and **zero** matches for `prolific`, `pl2303`, `cp210`, `ch34` +or `silabs` in either. The firmware line makes no difference here. + +⚠ **So a Prolific PL2303 cable (VID `067b`) cannot work with a Micromate.** The +unit has no driver, never enumerates the device, and the serial path simply does +not exist. Use an **FTDI** cable (VID `0403`) or a CDC-ACM one. + +### How this was isolated — the method is reusable + +A Micromate on an RX55 was unreachable from THOR. Rather than guess, each layer +was eliminated in turn with `bridges/mm_probe.py`: + +| layer | test | result | +|---|---|---| +| unit's public IP | static APN `mw01.VZWSTATIC` | ruled out | +| firewall / trusted IPs | probe from a whitelisted source | **TCP connect ok, 268 ms** | +| network path | same | ruled out | +| baud | 115200 both ends | ruled out | +| unit modem-type setting | set back to `generic` | ruled out | +| THOR | probe bypasses it entirely and still failed | ruled out | +| **modem TCP→serial** | swapped the unit for a Linux box on the same cable | **POLL arrived byte-perfect** | +| **modem serial→TCP** | that box answered with a canned POLL reply | **round trip in 700 ms** | + +The last two are the decisive pair. With a laptop standing in for the unit on +the *same cable and modem*, the full round trip works — so the modem, the PAD +configuration, the firewall and the network are all clean, and the only element +that changed is what sits at the end of the cable. + +`scratch/fake_unit.py`-style substitution is worth remembering: **replace the +device with something known-good and re-run the same test.** It converts "the +unit is not answering" into "the unit is not receiving", which are very +different problems. + +⚠ This explains the **bench** setup conclusively. Whether it explains the +2026-09-22 field outage is **not** established — that unit reportedly worked at +first and degraded, which a wrong cable would not do. Treat them as separate +until the field unit's cable is identified. + ## Known-good RV55 config for a Micromate (2026-09-25) Read off a unit **deployed and working for months**. This is the reference to diff --git a/scratch/fake_unit.py b/scratch/fake_unit.py new file mode 100644 index 0000000..ec0c04d --- /dev/null +++ b/scratch/fake_unit.py @@ -0,0 +1,33 @@ +"""Pretend to be a Micromate on a serial port: log what arrives, reply to POLL. + +Proves the modem's return path (serial -> TCP) independently of the real unit. +""" +import os, select, sys, termios, time + +path, baud = sys.argv[1], int(sys.argv[2]) if len(sys.argv) > 2 else 115200 +B = {9600: termios.B9600, 38400: termios.B38400, 115200: termios.B115200}[baud] +fd = os.open(path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK) +a = termios.tcgetattr(fd) +a[0] = a[1] = a[3] = 0 +a[2] = termios.CS8 | termios.CREAD | termios.CLOCAL +a[4] = a[5] = B +a[6] = list(a[6]); a[6][termios.VMIN] = 0; a[6][termios.VTIME] = 0 +termios.tcsetattr(fd, termios.TCSANOW, a) +termios.tcflush(fd, termios.TCIOFLUSH) + +# A real POLL probe reply, captured from UM12947 on 2026-09-24. +REPLY = bytes.fromhex("0200c5a4000000000000300000000000000099") + b"\x03" + +print(f"fake unit on {path} @ {baud}; will answer any inbound frame", flush=True) +while True: + r, _, _ = select.select([fd], [], [], 1.0) + if not r: + continue + data = os.read(fd, 4096) + if not data: + continue + ts = time.strftime("%H:%M:%S") + print(f"{ts} IN {len(data):3} B {data.hex(' ')}", flush=True) + time.sleep(0.02) + os.write(fd, REPLY) + print(f"{ts} OUT {len(REPLY):3} B {REPLY.hex(' ')} <- canned POLL reply", flush=True) From 8a0134070b3448fd2962acf97f13d2efe15579b2 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 20:35:13 -0400 Subject: [PATCH 41/43] docs(series4): soften the cable finding -- well-supported, not proven The operator pushed back on calling this solved, correctly. Adjusting the wording to match what the evidence actually carries. What IS supported: on this bench setup, the modem and cable work bidirectionally (laptop round trip, 700 ms), the same cable/modem pairing answers for a laptop and not for the Micromate, and neither firmware image contains a Prolific driver. Observation plus mechanism. What is NOT: that this explains the field failures. Those cables cannot be inspected, and the timeline does not fit -- a wrong cable fails from the first packet, and that unit reportedly worked before degrading. A gap in the bench claim itself, now stated: "no driver" is inferred from ABSENCE of strings. usbHostDelete_* reads like a complete per-class list, which is good evidence, but driver code can exist without a matching string and the USB enumeration path has not been disassembled. Names the test that removes the inference entirely: try an FTDI cable (VID 0403) on the same modem. Either it works or it does not. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index c124e4a..c3154be 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -2086,10 +2086,20 @@ usbHostDelete_HUB / HC / USBH strings in each, and **zero** matches for `prolific`, `pl2303`, `cp210`, `ch34` or `silabs` in either. The firmware line makes no difference here. -⚠ **So a Prolific PL2303 cable (VID `067b`) cannot work with a Micromate.** The -unit has no driver, never enumerates the device, and the serial path simply does +⚠ **So a Prolific PL2303 cable (VID `067b`) very probably cannot work with a +Micromate** — no driver, so the device never enumerates and the serial path does not exist. Use an **FTDI** cable (VID `0403`) or a CDC-ACM one. +**Calibration.** This rests on *absence of strings*, plus the observation that a +laptop answers on the same cable where the unit does not. `usbHostDelete_*` reads +like a complete per-class list, which is good evidence — but driver code can exist +without a matching string, and the USB enumeration path has not been +disassembled. + +**The test that would remove the inference: try an FTDI cable on the same modem.** +Either it works or it does not, and nothing is left to argue about. Until then +this is well-supported, not proven. + ### How this was isolated — the method is reusable A Micromate on an RX55 was unreachable from THOR. Rather than guess, each layer @@ -2116,7 +2126,7 @@ device with something known-good and re-run the same test.** It converts "the unit is not answering" into "the unit is not receiving", which are very different problems. -⚠ This explains the **bench** setup conclusively. Whether it explains the +⚠ This is the best explanation for the **bench** setup. Whether it explains the 2026-09-22 field outage is **not** established — that unit reportedly worked at first and degraded, which a wrong cable would not do. Treat them as separate until the field unit's cable is identified. From 062d65ba6911b29e9f842413cf190ef29cbfc6f7 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 20:54:56 -0400 Subject: [PATCH 42/43] docs(series4): purchasing history corroborates the cable finding TMI's order history shows TWO kinds of USB-to-serial cable in circulation: Sabrent USB 2.0 to serial FTDI chipset -> Micromate has a driver Benfei USB to RS-232 Prolific PL2303 -> no driver So both types are in the supply and indistinguishable by eye. A unit handed the wrong one has no working modem port and nothing about the cable says so. That moves the PL2303 explanation from "a theory about one odd cable" to a known mix. Adds the identification note: check chipset with lsusb, not appearance -- FTDI is VID 0403, Prolific 067b -- and flags that counterfeit FTDI chips are common in cheap cables, carrying FTDI's VID without behaving like one. An embedded host with a single driver is far less forgiving than Linux. Also records honestly that this does NOT fit the 2026-09-22 field outage as reported: a Prolific cable never enumerates, so it cannot fail gradually, and that unit was described as working before degrading. Notes the one story where it would fit -- if the initial success was over the USB PC port or a bench test before deployment, the modem path would have been broken from deployment onward and the recollection would be conflating two connection types. Plausible, unverified, recorded as such. Identifying that unit's cable settles it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 35 ++++++++++++++++++++++++++-- 1 file changed, 33 insertions(+), 2 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index c3154be..f2c5bb3 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -2097,8 +2097,39 @@ without a matching string, and the USB enumeration path has not been disassembled. **The test that would remove the inference: try an FTDI cable on the same modem.** -Either it works or it does not, and nothing is left to argue about. Until then -this is well-supported, not proven. +Either it works or it does not, and nothing is left to argue about. + +✅ **Corroborated by purchasing history (2026-09-25).** TMI has bought **two** +kinds of USB-to-serial cable: + +| cable | chipset | works with a Micromate? | +|---|---|---| +| **Sabrent** USB 2.0 to serial | **FTDI** | ✅ driver present | +| **Benfei** USB to RS-232 | **Prolific PL2303** | ❌ no driver | + +So both types are in circulation and indistinguishable by eye. A unit handed the +wrong one has no working modem port, and nothing about the cable's appearance +says so. + +⚠ **Identify cables by chipset, not by looks.** `lsusb` on any Linux box: +FTDI is VID `0403`, Prolific is `067b`. Note also that counterfeit FTDI chips +are common in cheap cables — they carry FTDI's VID but may not behave like one, +and an embedded host with a single driver is far less forgiving than Linux. + +### Does this explain the 2026-09-22 field outage? + +⚠ **Not on the timeline as reported.** A Prolific cable does not fail +*gradually* — it never enumerates, so the modem port never works at all. The +unit was described as working at first and degrading. + +There is one story where it fits: if the initial success was over a **different +path** — the USB **PC** port, or a bench test before deployment — that would work +regardless of which serial cable was attached. The modem path would then have +been broken from the moment it was deployed, and "it worked and then stopped" +would be a recollection conflating two connection types days after the fact. + +Plausible, unverified, and recorded as such. Identifying that unit's cable would +settle it. ### How this was isolated — the method is reusable From 4a96e1a2b274460f0222cc62d219f6e75bca4932 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 20:55:18 -0400 Subject: [PATCH 43/43] docs(series4): the bench cable was confirmed a Benfei (Prolific) Closes the identification chain: firmware has no Prolific driver, a laptop answers on that cable where the Micromate does not, lsusb reported PL2303, purchasing history shows both FTDI and Prolific cables in circulation, and the physical cable on the bench is a Benfei. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index f2c5bb3..b9e8610 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -2111,6 +2111,9 @@ So both types are in circulation and indistinguishable by eye. A unit handed th wrong one has no working modem port, and nothing about the cable's appearance says so. +**The cable on the bench during this investigation was confirmed to be a Benfei** +— the Prolific one. + ⚠ **Identify cables by chipset, not by looks.** `lsusb` on any Linux box: FTDI is VID `0403`, Prolific is `067b`. Note also that counterfeit FTDI chips are common in cheap cables — they carry FTDI's VID but may not behave like one,