From d522d31d631ddf27815d568d80744aa4ca50f093 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 25 Sep 2026 11:34:06 -0400 Subject: [PATCH] docs(series4): what THOR's "status check" actually is -- 11 commands, 563 MB/month First capture through bridges/mm_link.py, with THOR polling a unit over the bench link at "check connection every 5 s / check status every 5 s". A status check is ELEVEN commands, not one: POLL -> DEVICE_INFO -> 0x49 -> 0x5C -> MONITOR_STATUS -> SETUP_NAME_READ -> STORAGE_RANGE -> 0x02 -> OPERATOR -> 0x47 -> CALLHOME_CFG Each check opens a NEW TCP connection, runs all eleven exchanges in ~300 ms and closes it. Measured over 21 consecutive checks: 236 B out, 936 B back, plus a full handshake each time -- about 2.2 KB per check. At the observed cadence that is 18.8 MB/day, 563 MB/month, per unit. On a metered cellular plan that is real money, and most of it is waste: the check re-reads the call-home config, operator name, active setup name and full device info every ten seconds, none of which changes. SETUP_NAME_READ alone returns 274 bytes a time. POLL + MONITOR_STATUS answers "alive?" and "monitoring?" in two commands and 131 bytes. Both intervals set to 5 s yields one combined pass every 10.1 s, steady across eight measured connections. So the two settings are not independent 5-second timers, which is a plausible reason changing them appears to do nothing. REVISES an earlier hypothesis. Because idle polling reconnects every cycle, a silently-dead link is LESS dangerous while idle than I assumed -- a dead socket fails at connect and the next cycle retries. The exposure is during OPERATIONS: THOR held one connection from 00:30 to 00:47 last night while downloading events and pushing setups. A link dying mid-operation leaves it waiting on a socket the OS will not fail for ~2 h. The blackhole test should therefore be run during a download, not while idle. Also fixes a mislabel in mm_link.py: the SUB byte is DLE-escaped when its value is 0x02/0x03/0x04/0x10, so reading it raw reported SUB 0x02 as "SUB_10". Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- bridges/mm_link.py | 10 +++- docs/micromate_protocol_reference.md | 83 ++++++++++++++++++++++++++++ 2 files changed, 91 insertions(+), 2 deletions(-) diff --git a/bridges/mm_link.py b/bridges/mm_link.py index 2cefce7..a1724d2 100644 --- a/bridges/mm_link.py +++ b/bridges/mm_link.py @@ -170,10 +170,16 @@ class FrameSniffer: return # wait for more bytes body = self.buf[start:end + 1] del self.buf[:end + 1] - # sub sits at a fixed spot once the leading framing is skipped + # SUB sits at a fixed spot past the leading framing -- but it is + # DLE-escaped when its own value is 0x02/0x03/0x04/0x10, so a raw + # read reports 0x10 for those. SUB 0x02 was being logged as + # "SUB_10" until this was handled. off = 5 if self.is_request else 3 if len(body) > off: - yield body[off], len(body) + sub = body[off] + if sub == DLE and len(body) > off + 1: + sub = body[off + 1] + yield sub, len(body) class Link: diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index fb827b2..fc9bdf0 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -1595,6 +1595,89 @@ Those are the fields to look for when the schedule entry is finally decoded: captured entry (`03 00 00 00 0f 03 02 …`) has seven bytes before the name, which is the right order of magnitude for that field list. +## What THOR's "status check" actually is (2026-09-25) + +Captured with `bridges/mm_link.py` standing in for the modem, THOR configured +`Communication: TCP`, **Check connection every 5 s**, **Check status every 5 s**. + +### It is eleven commands, not one + +``` +POLL → DEVICE_INFO → 0x49 → 0x5C → MONITOR_STATUS → SETUP_NAME_READ + → STORAGE_RANGE → 0x02 → OPERATOR → 0x47 → CALLHOME_CFG +``` + +Each check opens a **new TCP connection**, runs all eleven exchanges in ~300 ms, +and closes it. Measured over 21 consecutive checks: + +| | | +|---|---| +| request payload | **236 B** | +| response payload | **936 B** | +| TCP connect + teardown | ~1,000 B (fresh handshake every time) | +| **per check** | **~2.2 KB** | + +### The cost + +| cadence | checks/day | per unit | +|---|---|---| +| **every 10 s (as observed)** | 8,640 | **18.8 MB/day — 563 MB/month** | +| every 60 s | 1,440 | 3.1 MB/day — 94 MB/month | +| every 15 min | 96 | 0.2 MB/day — 6 MB/month | + +⚠ **On a metered cellular plan this is real money, and most of it is waste.** +A status check re-reads the call-home config, the operator name, the active setup +name and the full device info block **every ten seconds** — none of which changes +between checks. `SETUP_NAME_READ` alone returns 274 bytes each time. + +A genuine liveness + state check is **`POLL` + `MONITOR_STATUS`**: two commands, +131 bytes of response. Same information about whether the unit is alive and +whether it is monitoring, for a ninth of the traffic. + +### Both intervals at 5 s produces one check every ~10 s + +Connection timestamps over three minutes: `11:31:48.7`, `11:31:58.9`, +`11:32:08.9`, `11:32:19.0`, `11:32:29.1`, `11:32:39.2`, `11:32:49.3`, +`11:32:59.4` — a **10.1 s** period, steady. + +So *"Check connection every 5 s"* and *"Check status every 5 s"* do not describe +two independent 5-second timers; together they yield one combined pass every ten +seconds. That is a plausible reason changing those fields appears to do nothing: +the relationship between the setting and the observed cadence is not what the UI +implies. + +### ⚠ What this means for the "won't stay connected" failure + +Because idle polling **opens a fresh connection each time**, a silently-dead link +is *less* dangerous here than expected — a dead socket fails at connect and the +next cycle simply tries again. The earlier hypothesis (THOR wedged on a +half-open socket during polling) is weakened by this. + +The exposure is during **operations**, not polling: THOR was observed holding one +connection from **00:30 to 00:47 — seventeen minutes** — while downloading events +and pushing setups. A link that dies silently mid-operation leaves THOR waiting +on a socket the OS will not fail for roughly the default keepalive (~2 h). That +remains the best candidate for a unit that will not come back and where refresh +does nothing. + +**Testable with `mm_link.py`:** set `blackhole` *during* a download or config +push rather than while idle, and watch whether THOR ever gives up, whether +refresh emits any bytes, and whether it recovers when `pass` is restored. + +### Design notes for SFM + +Three things this argues for, all cheap: + +1. **Separate liveness from inventory.** Poll `POLL` + `MONITOR_STATUS` + frequently; read config, setup name and call-home settings only when + something says they changed, or on demand. +2. **Show the connection attempt.** Every check here has a visible outcome — + connected, frames exchanged, closed. THOR surfaces none of it, which is why + a failing unit is undiagnosable from the UI. The log this section is built + from took one afternoon to produce and answers questions THOR cannot. +3. **Bound every operation with its own timeout**, independent of TCP's. Do not + rely on the socket to report a dead peer. + ## Thor's conventions vs the protocol's requirements **SFM is not meant to reimplement Thor.** Thor is the only available teacher of