First capture through bridges/mm_link.py, with THOR polling a unit over the bench
link at "check connection every 5 s / check status every 5 s".
A status check is ELEVEN commands, not one:
POLL -> DEVICE_INFO -> 0x49 -> 0x5C -> MONITOR_STATUS -> SETUP_NAME_READ
-> STORAGE_RANGE -> 0x02 -> OPERATOR -> 0x47 -> CALLHOME_CFG
Each check opens a NEW TCP connection, runs all eleven exchanges in ~300 ms and
closes it. Measured over 21 consecutive checks: 236 B out, 936 B back, plus a
full handshake each time -- about 2.2 KB per check.
At the observed cadence that is 18.8 MB/day, 563 MB/month, per unit. On a
metered cellular plan that is real money, and most of it is waste: the check
re-reads the call-home config, operator name, active setup name and full device
info every ten seconds, none of which changes. SETUP_NAME_READ alone returns 274
bytes a time. POLL + MONITOR_STATUS answers "alive?" and "monitoring?" in two
commands and 131 bytes.
Both intervals set to 5 s yields one combined pass every 10.1 s, steady across
eight measured connections. So the two settings are not independent 5-second
timers, which is a plausible reason changing them appears to do nothing.
REVISES an earlier hypothesis. Because idle polling reconnects every cycle, a
silently-dead link is LESS dangerous while idle than I assumed -- a dead socket
fails at connect and the next cycle retries. The exposure is during OPERATIONS:
THOR held one connection from 00:30 to 00:47 last night while downloading events
and pushing setups. A link dying mid-operation leaves it waiting on a socket the
OS will not fail for ~2 h. The blackhole test should therefore be run during a
download, not while idle.
Also fixes a mislabel in mm_link.py: the SUB byte is DLE-escaped when its value is
0x02/0x03/0x04/0x10, so reading it raw reported SUB 0x02 as "SUB_10".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
THOR gives almost no visibility into a connection: a refresh button, two poll
intervals, and no way to see whether a check succeeded, timed out, or was never
attempted. When a unit "won't stay connected" there is nothing to look at. This
sits where the cellular modem would and answers that directly.
Over socat -x it adds the two things that were missing:
* A READABLE LOG. Frames are decoded and timestamped as they pass --
"THOR->unit POLL (21 B)" rather than hex -- so THOR's polling cadence, and
its silences, are visible. Raw .bin pairs are still written alongside and
load straight into scratch/mm_frame_parse.py.
* FAULT INJECTION, via a control file read on the fly:
pass normal relay
blackhole TCP stays up, bytes are swallowed
drop close the connection abruptly
delay:N forward N seconds late, both directions
onewaydev THOR->unit passes, unit->THOR is swallowed
`blackhole` is the point of the exercise. It reproduces the classic cellular
failure -- socket open at both ends, nothing crossing -- which a real cell link
will not do on cue. THOR was observed last night holding one TCP connection for
17 minutes (00:30 to 00:47), so if the link dies silently the OS will not tell it
for roughly the default keepalive, ~2 hours. That is a candidate explanation for
"refresh does nothing and only a restart helps", and this makes it testable
rather than speculative.
No pyserial: the port is driven through stdlib termios. The bench hosts are
whatever is to hand and requiring a pip install on someone else's machine is a
poor trade for ~30 lines. Deployed and verified on mint-mac (Python 3.12, no
third-party modules) against UM12947 -- a POLL round-trips and decodes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
serversdownandClaude Opus 5