The wedged_unit_recovery runbook covered exactly one failure mode. BE12599
turned out to be a second one wearing the same symptoms, and the existing
procedure did not work on it.
Adds a "TWO failure modes" table up front so the next incident branches
correctly, and a full second-incident section covering what the ALEOS serial
debug log revealed: the device repeating a 29-byte AT modem-init string
(ATQ1/ATE0/ATS0=2, no ATD) every 75 s, never getting an OK because the modem
is in TCP data mode, and therefore never entering S3 mode at all. Inbound
cannot win against that, no matter how well framed.
Also records the two red herrings, since together they cost ~90 minutes:
the RV50 trusted-IP whitelist drops non-listed sources silently (presents as
a connect timeout, and Brian's dynamic dev IP had rotated off the list), and
sfm/server.py returns 502 for BOTH "Protocol error:" and "Connection error:",
so a 502 was misread as "TCP connected, device mute" and a theory built on it.
And the gotchas worth never re-deriving: slow_drip's send_error=null plus a
full duration is not success (only bytes_received > 0 is); stopping monitoring
removes the call-in trigger, so it costs you the channel; --events-only skips
the device-info step, so the serial is never read and ach_state keys on
peer:ephemeral_port, silently breaking dedup and re-downloading the same event
every session.
The plan doc captures the tool Brian wants built out of this — a rescue
listener with a real lifecycle and, critically, a confirmation gate before
shutdown, because leaving the modem's Destination pointed at a dead listener
is worse than never having started. Open questions are listed rather than
guessed at.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN
serversdownandClaude Opus 5
c6fc3d0241