docs/runbooks/wedged_unit_recovery.md

@@ -1,6 +1,7 @@
 # Runbook — Recovering a wedged unit stuck in a call-home loop
 
-**Original incident:** BE9558H at `166.246.130.1:9034`, recovered 2026-05-17.
+**Incidents:** BE9558H at `166.246.130.1:9034`, 2026-05-17 (Method B) ·
+BE12599 at `166.246.64.226:9034`, 2026-09-16 (Method A).
 
 A field unit with a stuck-triggered geophone (or any hardware fault causing
 constant event triggering) will record events back-to-back, and if Auto Call
@@ -14,24 +15,30 @@ This runbook describes how to break the loop and recover control.
 
 ---
 
-## ⚠ There are TWO failure modes under "wedged unit"
+## ⚠ Two cures for one disease — intercept first
 
-Same root cause — an offset/connector fault drives the geophone above trigger,
-the unit records back-to-back, ACH set to "after event recorded" dials
-constantly — but the *recovery* differs, because the thing blocking you is
-different.
+Both incidents below are the **same failure**: a geophone offset crosses the
+trigger level, the unit records back-to-back, ACH set to "after event
+recorded" dials continuously, and the unit becomes unreachable because its
+modem is in client mode almost all of the time.
 
-| | **BE9558H (2026-05)** | **BE12599 (2026-09)** |
+There are two ways to get a Stop Monitoring command into it.
+
+| | **A — intercept the call** (preferred) | **B — catch it between calls** (original) |
 |---|---|---|
-| What blocks you | Modem mode-flipping kills inbound TCP | Device never enters S3 mode at all |
-| Device state | Alive, in S3 mode, responsive once reached | Stuck repeating an **AT modem-init** string, deaf to S3 |
-| Fix direction | **Inbound** — clear Destination, slow-drip a Stop | **Inverted** — point Destination at our own ACH server and let the unit call *us* |
-| Winning tool | `scripts/slow_drip.sh` | `bridges/ach_server.py --stop-monitoring` |
+| Idea | Be the server it dials.  Point the modem's Destination at our own ACH server and answer it. | Clear the Destination so it stops dialing, then race a Stop into the gap. |
+| Needs inbound? | **No — the unit calls us** | Yes: working inbound TCP to the modem |
+| Determinism | Deterministic — it dials every ~75 s, we only have to be listening | A race.  BE9558H took ~7 h of attempts before one landed. |
+| Tool | `bridges/ach_server.py --stop-monitoring` | `scripts/slow_drip.sh` |
+| Proven on | BE12599, 2026-09-16 | BE9558H, 2026-05-17 |
 
-**Tell them apart with the ALEOS serial log** (see "Turn on ALEOS_SERIAL debug"
-below). If the device is emitting `ATQ1/ATE0/ATS0=2` every ~75 s, it is in the
-BE12599 mode and **no amount of inbound work will reach it** — skip to
-"Second incident" below.
+**Method A is the standard procedure now.**  The unit won't answer us because
+it is on the phone — so stop dialing it and be the one it calls.  It rings,
+we pick up, take its data, and tell it to stop calling here.
+
+Method B is kept because it is proven, and because A needs a listener the
+modem can actually reach (public IP + forwarded port).  When you have that,
+don't race it — intercept it.
 
 ---
 
@@ -52,9 +59,85 @@ If you see *all* of these, the unit is in this exact failure mode.
 
 ---
 
-## Quick reference — how to recover
+## Method A (preferred) — intercept the call
 
-You need **ACEmanager access** to the unit's modem.
+You need **ACEmanager access** and a host the modem can dial: public IP with
+the listener's port forwarded to it.
+
+### A1 — start the listener BEFORE touching the modem
+
+```bash
+cd /home/serversdown/seismo-relay
+tmux new -s rescue
+.venv/bin/python -u bridges/ach_server.py --port 12345 \
+  -o bridges/captures/<unit>-diag --stop-monitoring -v
+```
+
+⚠ **Listener first, always.**  A Destination pointed at a dead port is the
+worst state available — the device still dials, the modem still flips to
+client mode, inbound stays blocked, and nothing gets delivered.
+
+Do **not** add `--events-only` (it silently breaks dedup — see gotchas), and
+do **not** add `--disable-ach` yet (see A4).
+
+### A2 — point the modem at it
+
+ACEmanager → **Serial → Port Configuration**:
+
+| Field | Set to |
+|---|---|
+| **Destination Address** | the listener's public IP |
+| **Destination Port** | the listener's port (e.g. `12345`) |
+
+Apply.  The modem auto-dials its Destination whenever serial data arrives
+while the serial port is closed — so the unit's own retry cycle now lands on
+you instead of nowhere.
+
+### A3 — answer, and stop the bleeding
+
+Within ~75 s you should see a call-in.  `--stop-monitoring` fires SUB 0x97 at
+step 1.5 — after the handshake, **before** the event walk — so the recording
+halts at the earliest possible moment in the session.  Confirm via
+`rescue.json` in the session directory:
+
+```json
+{"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
+```
+
+That is the bleeding stopped.  Everything after this is cleanup.
+
+### A4 — drain the backlog, THEN disable ACH
+
+⚠ **Order matters, and it is counter-intuitive.**  Stopping monitoring also
+removes your call-in trigger: ACH fires on "after event recorded", so with
+recording stopped the unit has no reason to dial again.  The backlog sitting
+in its memory does **not** re-arm it.
+
+So if the stored events are worth keeping — and on a fault unit they usually
+are, they're the evidence — drain them across however many call-ins it takes
+*before* you silence it.  Only then add `--disable-ach` (or use
+`scripts/rescue_device.sh <host> <port> --no-erase`).
+
+If the unit has gone quiet and you still need it, cycling the modem produces
+a call-in, and a unit with a scheduled daily call will dial at its configured
+time regardless.
+
+### A5 — restore the Destination, and confirm you did
+
+Put `Destination Address` back to `0.0.0.0` (or the office Instantel ACH
+server) once you are finished, and only stop the listener after that is done.
+
+### A6 — do NOT re-enable ACH until the hardware fault is repaired
+
+Otherwise the loop restarts the moment monitoring resumes and you run this
+runbook again.
+
+---
+
+## Method B (fallback) — catch it between calls
+
+The original 2026-05 procedure.  Use when you cannot stand up a listener the
+modem can reach.  You need **ACEmanager access** to the unit's modem.
 
 ### Step 1: stop the modem's mode-flipping
 
@@ -287,9 +370,14 @@ recovery: ~7 hours of intermittent debugging across one evening.
 level.  Constant triggering → constant recording → ACH "after event recorded"
 → continuous dialing.  Same disease as BE9558H.
 
-**But the recovery was the opposite direction**, and none of the Step 1–4
-procedure above worked.  Total time ≈ 5 h, of which ~90 min was spent on two
-red herrings documented below.
+**Same disease, inverted cure.**  Method B's Step 1 *did* work — clearing the
+Destination stopped the dial-outs, confirmed in the ALEOS log.  It was Step 2
+that didn't land, and rather than keep racing we turned the rescue around:
+gave the unit a different server to call, and answered it.
+
+Total time ≈ 5 h, of which ~90 min went to two red herrings documented below.
+Much of the rest was rediscovering the May procedure, which is why the
+"two cures" table now sits at the top of this file.
 
 ---
 
@@ -302,7 +390,7 @@ DEBUG**, then view the serial log.
 It is the only thing that tells you what the *device* is actually saying.
 Everything before we did this was guesswork.
 
-## What the log showed — the device was never in S3 mode
+## What the log showed — the unit is on the phone
 
 Every ~75 seconds, verbatim:
 
@@ -326,9 +414,15 @@ Read that carefully:
 - The device therefore never receives `OK`, never progresses, and **retries
   the identical 29 bytes forever**.
 
-**Consequence: the device is not running the S3 protocol parser.**  You can
-land a byte-perfect S3 frame on it and it will be ignored.  This is why every
-inbound approach failed, and it is the structural difference from BE9558H.
+**While it is in this state it is busy placing a call, not listening for
+us.**  This is almost certainly what BE9558H was doing too — we simply never
+turned on ALEOS_SERIAL debug in May to look.  It is not a different disease;
+it is the same one, seen properly for the first time.
+
+It is also the argument for Method A in one picture: the unit is mid-dial
+every ~75 s, and our inbound Stop has to thread the gaps between those
+attempts.  Give it somewhere to dial and the problem inverts into a
+deterministic one.
 
 ### Why `slow_drip` lied
 
@@ -351,6 +445,12 @@ nobody was reading.
 ⚠ **`send_error: null` + full duration is NOT success.  Only
 `bytes_received > 0` is success.**
 
+⚠ **In fairness to slow_drip: it got exactly one attempt here**, run ~90 s
+after a modem reboot, with a dead session visible in the log at 20:19:17 in
+that same window.  BE9558H took hours of attempts before one landed.  Method B
+was not ruled out on BE12599 so much as abandoned in favour of something that
+doesn't need luck.
+
 ---
 
 ## ⚠ Two red herrings that cost ~90 minutes