diff --git a/docs/runbooks/wedged_unit_recovery.md b/docs/runbooks/wedged_unit_recovery.md index 3536195..bf427a0 100644 --- a/docs/runbooks/wedged_unit_recovery.md +++ b/docs/runbooks/wedged_unit_recovery.md @@ -1,6 +1,7 @@ # Runbook — Recovering a wedged unit stuck in a call-home loop -**Original incident:** BE9558H at `166.246.130.1:9034`, recovered 2026-05-17. +**Incidents:** BE9558H at `166.246.130.1:9034`, 2026-05-17 (Method B) · +BE12599 at `166.246.64.226:9034`, 2026-09-16 (Method A). A field unit with a stuck-triggered geophone (or any hardware fault causing constant event triggering) will record events back-to-back, and if Auto Call @@ -14,24 +15,30 @@ This runbook describes how to break the loop and recover control. --- -## ⚠ There are TWO failure modes under "wedged unit" +## ⚠ Two cures for one disease — intercept first -Same root cause — an offset/connector fault drives the geophone above trigger, -the unit records back-to-back, ACH set to "after event recorded" dials -constantly — but the *recovery* differs, because the thing blocking you is -different. +Both incidents below are the **same failure**: a geophone offset crosses the +trigger level, the unit records back-to-back, ACH set to "after event +recorded" dials continuously, and the unit becomes unreachable because its +modem is in client mode almost all of the time. -| | **BE9558H (2026-05)** | **BE12599 (2026-09)** | +There are two ways to get a Stop Monitoring command into it. + +| | **A — intercept the call** (preferred) | **B — catch it between calls** (original) | |---|---|---| -| What blocks you | Modem mode-flipping kills inbound TCP | Device never enters S3 mode at all | -| Device state | Alive, in S3 mode, responsive once reached | Stuck repeating an **AT modem-init** string, deaf to S3 | -| Fix direction | **Inbound** — clear Destination, slow-drip a Stop | **Inverted** — point Destination at our own ACH server and let the unit call *us* | -| Winning tool | `scripts/slow_drip.sh` | `bridges/ach_server.py --stop-monitoring` | +| Idea | Be the server it dials. Point the modem's Destination at our own ACH server and answer it. | Clear the Destination so it stops dialing, then race a Stop into the gap. | +| Needs inbound? | **No — the unit calls us** | Yes: working inbound TCP to the modem | +| Determinism | Deterministic — it dials every ~75 s, we only have to be listening | A race. BE9558H took ~7 h of attempts before one landed. | +| Tool | `bridges/ach_server.py --stop-monitoring` | `scripts/slow_drip.sh` | +| Proven on | BE12599, 2026-09-16 | BE9558H, 2026-05-17 | -**Tell them apart with the ALEOS serial log** (see "Turn on ALEOS_SERIAL debug" -below). If the device is emitting `ATQ1/ATE0/ATS0=2` every ~75 s, it is in the -BE12599 mode and **no amount of inbound work will reach it** — skip to -"Second incident" below. +**Method A is the standard procedure now.** The unit won't answer us because +it is on the phone — so stop dialing it and be the one it calls. It rings, +we pick up, take its data, and tell it to stop calling here. + +Method B is kept because it is proven, and because A needs a listener the +modem can actually reach (public IP + forwarded port). When you have that, +don't race it — intercept it. --- @@ -52,9 +59,85 @@ If you see *all* of these, the unit is in this exact failure mode. --- -## Quick reference — how to recover +## Method A (preferred) — intercept the call -You need **ACEmanager access** to the unit's modem. +You need **ACEmanager access** and a host the modem can dial: public IP with +the listener's port forwarded to it. + +### A1 — start the listener BEFORE touching the modem + +```bash +cd /home/serversdown/seismo-relay +tmux new -s rescue +.venv/bin/python -u bridges/ach_server.py --port 12345 \ + -o bridges/captures/-diag --stop-monitoring -v +``` + +⚠ **Listener first, always.** A Destination pointed at a dead port is the +worst state available — the device still dials, the modem still flips to +client mode, inbound stays blocked, and nothing gets delivered. + +Do **not** add `--events-only` (it silently breaks dedup — see gotchas), and +do **not** add `--disable-ach` yet (see A4). + +### A2 — point the modem at it + +ACEmanager → **Serial → Port Configuration**: + +| Field | Set to | +|---|---| +| **Destination Address** | the listener's public IP | +| **Destination Port** | the listener's port (e.g. `12345`) | + +Apply. The modem auto-dials its Destination whenever serial data arrives +while the serial port is closed — so the unit's own retry cycle now lands on +you instead of nowhere. + +### A3 — answer, and stop the bleeding + +Within ~75 s you should see a call-in. `--stop-monitoring` fires SUB 0x97 at +step 1.5 — after the handshake, **before** the event walk — so the recording +halts at the earliest possible moment in the session. Confirm via +`rescue.json` in the session directory: + +```json +{"peer": "166.246.64.226:60921", "stop_monitoring": "ok"} +``` + +That is the bleeding stopped. Everything after this is cleanup. + +### A4 — drain the backlog, THEN disable ACH + +⚠ **Order matters, and it is counter-intuitive.** Stopping monitoring also +removes your call-in trigger: ACH fires on "after event recorded", so with +recording stopped the unit has no reason to dial again. The backlog sitting +in its memory does **not** re-arm it. + +So if the stored events are worth keeping — and on a fault unit they usually +are, they're the evidence — drain them across however many call-ins it takes +*before* you silence it. Only then add `--disable-ach` (or use +`scripts/rescue_device.sh --no-erase`). + +If the unit has gone quiet and you still need it, cycling the modem produces +a call-in, and a unit with a scheduled daily call will dial at its configured +time regardless. + +### A5 — restore the Destination, and confirm you did + +Put `Destination Address` back to `0.0.0.0` (or the office Instantel ACH +server) once you are finished, and only stop the listener after that is done. + +### A6 — do NOT re-enable ACH until the hardware fault is repaired + +Otherwise the loop restarts the moment monitoring resumes and you run this +runbook again. + +--- + +## Method B (fallback) — catch it between calls + +The original 2026-05 procedure. Use when you cannot stand up a listener the +modem can reach. You need **ACEmanager access** to the unit's modem. ### Step 1: stop the modem's mode-flipping @@ -287,9 +370,14 @@ recovery: ~7 hours of intermittent debugging across one evening. level. Constant triggering → constant recording → ACH "after event recorded" → continuous dialing. Same disease as BE9558H. -**But the recovery was the opposite direction**, and none of the Step 1–4 -procedure above worked. Total time ≈ 5 h, of which ~90 min was spent on two -red herrings documented below. +**Same disease, inverted cure.** Method B's Step 1 *did* work — clearing the +Destination stopped the dial-outs, confirmed in the ALEOS log. It was Step 2 +that didn't land, and rather than keep racing we turned the rescue around: +gave the unit a different server to call, and answered it. + +Total time ≈ 5 h, of which ~90 min went to two red herrings documented below. +Much of the rest was rediscovering the May procedure, which is why the +"two cures" table now sits at the top of this file. --- @@ -302,7 +390,7 @@ DEBUG**, then view the serial log. It is the only thing that tells you what the *device* is actually saying. Everything before we did this was guesswork. -## What the log showed — the device was never in S3 mode +## What the log showed — the unit is on the phone Every ~75 seconds, verbatim: @@ -326,9 +414,15 @@ Read that carefully: - The device therefore never receives `OK`, never progresses, and **retries the identical 29 bytes forever**. -**Consequence: the device is not running the S3 protocol parser.** You can -land a byte-perfect S3 frame on it and it will be ignored. This is why every -inbound approach failed, and it is the structural difference from BE9558H. +**While it is in this state it is busy placing a call, not listening for +us.** This is almost certainly what BE9558H was doing too — we simply never +turned on ALEOS_SERIAL debug in May to look. It is not a different disease; +it is the same one, seen properly for the first time. + +It is also the argument for Method A in one picture: the unit is mid-dial +every ~75 s, and our inbound Stop has to thread the gaps between those +attempts. Give it somewhere to dial and the problem inverts into a +deterministic one. ### Why `slow_drip` lied @@ -351,6 +445,12 @@ nobody was reading. ⚠ **`send_error: null` + full duration is NOT success. Only `bytes_received > 0` is success.** +⚠ **In fairness to slow_drip: it got exactly one attempt here**, run ~90 s +after a modem reboot, with a dead session visible in the log at 20:19:17 in +that same window. BE9558H took hours of attempts before one landed. Method B +was not ruled out on BE12599 so much as abandoned in favour of something that +doesn't need luck. + --- ## ⚠ Two red herrings that cost ~90 minutes