docs: BE12599 incident — the inverted rescue, plus a rescue-listener plan
The wedged_unit_recovery runbook covered exactly one failure mode. BE12599 turned out to be a second one wearing the same symptoms, and the existing procedure did not work on it. Adds a "TWO failure modes" table up front so the next incident branches correctly, and a full second-incident section covering what the ALEOS serial debug log revealed: the device repeating a 29-byte AT modem-init string (ATQ1/ATE0/ATS0=2, no ATD) every 75 s, never getting an OK because the modem is in TCP data mode, and therefore never entering S3 mode at all. Inbound cannot win against that, no matter how well framed. Also records the two red herrings, since together they cost ~90 minutes: the RV50 trusted-IP whitelist drops non-listed sources silently (presents as a connect timeout, and Brian's dynamic dev IP had rotated off the list), and sfm/server.py returns 502 for BOTH "Protocol error:" and "Connection error:", so a 502 was misread as "TCP connected, device mute" and a theory built on it. And the gotchas worth never re-deriving: slow_drip's send_error=null plus a full duration is not success (only bytes_received > 0 is); stopping monitoring removes the call-in trigger, so it costs you the channel; --events-only skips the device-info step, so the serial is never read and ach_state keys on peer:ephemeral_port, silently breaking dedup and re-downloading the same event every session. The plan doc captures the tool Brian wants built out of this — a rescue listener with a real lifecycle and, critically, a confirmation gate before shutdown, because leaving the modem's Destination pointed at a dead listener is worse than never having started. Open questions are listed rather than guessed at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN
This commit is contained in:
Claude Opus 5
@@ -0,0 +1,134 @@
|
||||
# Plan — "Rescue Listener": a first-class tool for the inverted rescue
|
||||
|
||||
**Status:** proposal, not started. Written 2026-09-17 ~01:40 local, straight
|
||||
off the BE12599 incident. Open questions at the bottom need Brian's answer
|
||||
before anything is built.
|
||||
|
||||
**Background:** `docs/runbooks/wedged_unit_recovery.md`, "Second incident —
|
||||
BE12599". The manual version of this worked; this plan is about making it a
|
||||
tool instead of a sequence of remembered steps at 1 AM.
|
||||
|
||||
---
|
||||
|
||||
## The problem, stated plainly
|
||||
|
||||
When a unit is wedged in the BE12599 mode — geophone offset above trigger,
|
||||
recording back-to-back, ACH dialing constantly, device stuck repeating an AT
|
||||
modem-init string and therefore **deaf to S3 over inbound** — the only channel
|
||||
that works is the one the *device* opens.
|
||||
|
||||
Recovering it currently means:
|
||||
|
||||
1. Remember that `bridges/ach_server.py` exists and takes the right flags
|
||||
2. Start it by hand on a box the modem can reach, with a public port forwarded
|
||||
3. Go into ACEmanager and repoint the modem's Destination
|
||||
4. Watch a terminal for a call-in
|
||||
5. Read `rescue.json` to find out whether it worked
|
||||
6. Go back into ACEmanager and repoint the modem to where it belongs
|
||||
7. **Not forget step 6**, because leaving the Destination pointed at a dead
|
||||
listener is worse than never having started
|
||||
|
||||
That is six manual steps and one landmine, executed under pressure while a
|
||||
unit floods the office server.
|
||||
|
||||
## What the tool should be
|
||||
|
||||
**A "rescue listener" an operator can start for one unit, which handles
|
||||
whatever that unit says when it calls in, and refuses to go away until the
|
||||
operator confirms the modem has been pointed back.**
|
||||
|
||||
Lifecycle:
|
||||
|
||||
1. **Start** — operator names the target unit and starts a rescue listener.
|
||||
The tool reports the exact address/port to enter in ACEmanager, plus the
|
||||
actions it will take.
|
||||
2. **Operator repoints the modem** to that address.
|
||||
3. **Wait** — listener sits there. Live status: "waiting for call-in",
|
||||
elapsed, last-seen.
|
||||
4. **Act** — on call-in, run the configured rescue actions automatically,
|
||||
in a safe order, each independently guarded. Report per-action outcome.
|
||||
5. **Hold** — the listener **stays up** and keeps reporting, because the
|
||||
modem is still pointed at it.
|
||||
6. **Confirm & stop** — the operator explicitly confirms the Destination has
|
||||
been restored (to `0.0.0.0`, or to the office Instantel ACH server).
|
||||
Only then does the listener shut down.
|
||||
|
||||
Step 6 is the whole point of making this a tool. It is the step that is
|
||||
easiest to skip and most expensive to skip.
|
||||
|
||||
## Default action set
|
||||
|
||||
Ordered deliberately — see "order matters" below.
|
||||
|
||||
| # | Action | Default | Why |
|
||||
|---|---|---|---|
|
||||
| 1 | **Stop monitoring** (SUB 0x97) | ✅ on | Halts recording; ends the trigger→record→dial loop at its source. Already implemented as `--stop-monitoring`. |
|
||||
| 2 | **Drain events** to a diagnostics store | ⚙ configurable | The backlog is usually evidence, not garbage — see the BE12599 offset investigation. Must NOT land in the prod SFM DB. |
|
||||
| 3 | **Disable ACH** (SUB 0x2C/0x7E/0x7F) | ❌ off by default | Stops the dialing — **and stops your only channel**. Opt-in, and ideally gated on step 1 having succeeded. |
|
||||
| 4 | **Erase events** | ❌ off by default | Destructive. Only after a verified drain. |
|
||||
|
||||
### Order matters — the lesson from BE12599
|
||||
|
||||
Stopping monitoring *removes the call-in trigger*. ACH fires on "after event
|
||||
recorded"; with recording stopped, the unit has no reason to dial again, even
|
||||
though the backlog is still sitting in its memory. So a naive
|
||||
"stop + disable + erase, all at once" rescue can silence the unit before
|
||||
you've collected anything, leaving you with no channel and a device full of
|
||||
evidence.
|
||||
|
||||
The tool should either sequence around this or warn loudly about it. My
|
||||
instinct is: **stop monitoring immediately** (it's the bleeding), then drain
|
||||
across however many call-ins it takes, and treat disable-ACH/erase as a
|
||||
separate, explicit "finish" action once the operator is satisfied.
|
||||
|
||||
## Where it should live — open question, with a proposal
|
||||
|
||||
The natural tier is **SFM** (device-side, per the three-tier model in
|
||||
CLAUDE.md). But the rescue listener must be reachable *from the cellular
|
||||
network*, which is a deployment constraint SFM's usual profile doesn't have.
|
||||
|
||||
**Proposal worth considering:** run it at the office, beside the real Instantel
|
||||
ACH server, on a **different port** (e.g. 12346 while Instantel holds 12345).
|
||||
Then the ACEmanager change is a **port change, not an IP change** — smaller,
|
||||
faster, less to get wrong, and trivially reversible. It also means the office
|
||||
public IP (already stable and known) is the destination, rather than whatever
|
||||
Brian's dynamic home IP happens to be that week.
|
||||
|
||||
The tmi-dev approach used on BE12599 worked, but required a router forward and
|
||||
ran into the dynamic-IP problem in the same session.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Where does it run?** Office beside Instantel ACH (port swap), SFM on the
|
||||
NAS, or ad-hoc on tmi-dev? Affects everything else.
|
||||
2. **What drives it?** Terra-View admin page (fits "operator UI"), an SFM
|
||||
endpoint pair (`POST /device/rescue_listener/start` + `/stop` + `/status`),
|
||||
or a CLI wrapper? A long-lived listener doesn't fit the request/response
|
||||
endpoint shape well — probably needs a background task with a status poll.
|
||||
3. **How does it identify the unit?** It can't know the serial until the
|
||||
device calls in and the handshake reads it. Allowlist by modem IP? Accept
|
||||
anything and report what showed up?
|
||||
4. **Where do drained events go?** A per-incident diagnostics store
|
||||
(`bridges/captures/<unit>-diag`) seems right — explicitly *not* the prod
|
||||
SFM DB. Does that store need to be a first-class thing with its own
|
||||
retention, or is a directory fine?
|
||||
5. **How is "confirm the modem is repointed" verified?** Operator attestation
|
||||
(a button), or can we actually probe it? If the listener stops seeing
|
||||
call-ins that's weak evidence; if inbound to the unit starts working that's
|
||||
stronger.
|
||||
6. **Multi-unit?** One listener per incident, or one listener that handles any
|
||||
unit that dials in? Probably the former for safety.
|
||||
7. **Timeout / abandonment policy.** If nobody ever confirms, does it run
|
||||
forever? Alert after N hours?
|
||||
|
||||
## What already exists
|
||||
|
||||
- `bridges/ach_server.py` — the listener itself, with `--stop-monitoring`,
|
||||
`--disable-ach`, `--rescue` (added on `feat/ach-rescue-on-connect`, commit
|
||||
`9f1050b`), `--clear-after-download`, `--max-events`, `--allow-ip`.
|
||||
- Per-session `rescue.json` recording per-action outcomes.
|
||||
- Isolated per-output-dir SQLite + waveform store, so a diagnostics capture is
|
||||
already separate from prod by construction.
|
||||
|
||||
So the gap is not protocol work — it's lifecycle, operator surface, and the
|
||||
confirmation gate. Most of the risk is in questions 1 and 2.
|
||||
Reference in New Issue
Block a user