Feat/ach rescue on connect #38
@@ -1,6 +1,7 @@
|
||||
# Runbook — Recovering a wedged unit stuck in a call-home loop
|
||||
|
||||
**Original incident:** BE9558H at `166.246.130.1:9034`, recovered 2026-05-17.
|
||||
**Incidents:** BE9558H at `166.246.130.1:9034`, 2026-05-17 (Method B) ·
|
||||
BE12599 at `166.246.64.226:9034`, 2026-09-16 (Method A).
|
||||
|
||||
A field unit with a stuck-triggered geophone (or any hardware fault causing
|
||||
constant event triggering) will record events back-to-back, and if Auto Call
|
||||
@@ -14,24 +15,30 @@ This runbook describes how to break the loop and recover control.
|
||||
|
||||
---
|
||||
|
||||
## ⚠ There are TWO failure modes under "wedged unit"
|
||||
## ⚠ Two cures for one disease — intercept first
|
||||
|
||||
Same root cause — an offset/connector fault drives the geophone above trigger,
|
||||
the unit records back-to-back, ACH set to "after event recorded" dials
|
||||
constantly — but the *recovery* differs, because the thing blocking you is
|
||||
different.
|
||||
Both incidents below are the **same failure**: a geophone offset crosses the
|
||||
trigger level, the unit records back-to-back, ACH set to "after event
|
||||
recorded" dials continuously, and the unit becomes unreachable because its
|
||||
modem is in client mode almost all of the time.
|
||||
|
||||
| | **BE9558H (2026-05)** | **BE12599 (2026-09)** |
|
||||
There are two ways to get a Stop Monitoring command into it.
|
||||
|
||||
| | **A — intercept the call** (preferred) | **B — catch it between calls** (original) |
|
||||
|---|---|---|
|
||||
| What blocks you | Modem mode-flipping kills inbound TCP | Device never enters S3 mode at all |
|
||||
| Device state | Alive, in S3 mode, responsive once reached | Stuck repeating an **AT modem-init** string, deaf to S3 |
|
||||
| Fix direction | **Inbound** — clear Destination, slow-drip a Stop | **Inverted** — point Destination at our own ACH server and let the unit call *us* |
|
||||
| Winning tool | `scripts/slow_drip.sh` | `bridges/ach_server.py --stop-monitoring` |
|
||||
| Idea | Be the server it dials. Point the modem's Destination at our own ACH server and answer it. | Clear the Destination so it stops dialing, then race a Stop into the gap. |
|
||||
| Needs inbound? | **No — the unit calls us** | Yes: working inbound TCP to the modem |
|
||||
| Determinism | Deterministic — it dials every ~75 s, we only have to be listening | A race. BE9558H took ~7 h of attempts before one landed. |
|
||||
| Tool | `bridges/ach_server.py --stop-monitoring` | `scripts/slow_drip.sh` |
|
||||
| Proven on | BE12599, 2026-09-16 | BE9558H, 2026-05-17 |
|
||||
|
||||
**Tell them apart with the ALEOS serial log** (see "Turn on ALEOS_SERIAL debug"
|
||||
below). If the device is emitting `ATQ1/ATE0/ATS0=2` every ~75 s, it is in the
|
||||
BE12599 mode and **no amount of inbound work will reach it** — skip to
|
||||
"Second incident" below.
|
||||
**Method A is the standard procedure now.** The unit won't answer us because
|
||||
it is on the phone — so stop dialing it and be the one it calls. It rings,
|
||||
we pick up, take its data, and tell it to stop calling here.
|
||||
|
||||
Method B is kept because it is proven, and because A needs a listener the
|
||||
modem can actually reach (public IP + forwarded port). When you have that,
|
||||
don't race it — intercept it.
|
||||
|
||||
---
|
||||
|
||||
@@ -52,9 +59,85 @@ If you see *all* of these, the unit is in this exact failure mode.
|
||||
|
||||
---
|
||||
|
||||
## Quick reference — how to recover
|
||||
## Method A (preferred) — intercept the call
|
||||
|
||||
You need **ACEmanager access** to the unit's modem.
|
||||
You need **ACEmanager access** and a host the modem can dial: public IP with
|
||||
the listener's port forwarded to it.
|
||||
|
||||
### A1 — start the listener BEFORE touching the modem
|
||||
|
||||
```bash
|
||||
cd /home/serversdown/seismo-relay
|
||||
tmux new -s rescue
|
||||
.venv/bin/python -u bridges/ach_server.py --port 12345 \
|
||||
-o bridges/captures/<unit>-diag --stop-monitoring -v
|
||||
```
|
||||
|
||||
⚠ **Listener first, always.** A Destination pointed at a dead port is the
|
||||
worst state available — the device still dials, the modem still flips to
|
||||
client mode, inbound stays blocked, and nothing gets delivered.
|
||||
|
||||
Do **not** add `--events-only` (it silently breaks dedup — see gotchas), and
|
||||
do **not** add `--disable-ach` yet (see A4).
|
||||
|
||||
### A2 — point the modem at it
|
||||
|
||||
ACEmanager → **Serial → Port Configuration**:
|
||||
|
||||
| Field | Set to |
|
||||
|---|---|
|
||||
| **Destination Address** | the listener's public IP |
|
||||
| **Destination Port** | the listener's port (e.g. `12345`) |
|
||||
|
||||
Apply. The modem auto-dials its Destination whenever serial data arrives
|
||||
while the serial port is closed — so the unit's own retry cycle now lands on
|
||||
you instead of nowhere.
|
||||
|
||||
### A3 — answer, and stop the bleeding
|
||||
|
||||
Within ~75 s you should see a call-in. `--stop-monitoring` fires SUB 0x97 at
|
||||
step 1.5 — after the handshake, **before** the event walk — so the recording
|
||||
halts at the earliest possible moment in the session. Confirm via
|
||||
`rescue.json` in the session directory:
|
||||
|
||||
```json
|
||||
{"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
|
||||
```
|
||||
|
||||
That is the bleeding stopped. Everything after this is cleanup.
|
||||
|
||||
### A4 — drain the backlog, THEN disable ACH
|
||||
|
||||
⚠ **Order matters, and it is counter-intuitive.** Stopping monitoring also
|
||||
removes your call-in trigger: ACH fires on "after event recorded", so with
|
||||
recording stopped the unit has no reason to dial again. The backlog sitting
|
||||
in its memory does **not** re-arm it.
|
||||
|
||||
So if the stored events are worth keeping — and on a fault unit they usually
|
||||
are, they're the evidence — drain them across however many call-ins it takes
|
||||
*before* you silence it. Only then add `--disable-ach` (or use
|
||||
`scripts/rescue_device.sh <host> <port> --no-erase`).
|
||||
|
||||
If the unit has gone quiet and you still need it, cycling the modem produces
|
||||
a call-in, and a unit with a scheduled daily call will dial at its configured
|
||||
time regardless.
|
||||
|
||||
### A5 — restore the Destination, and confirm you did
|
||||
|
||||
Put `Destination Address` back to `0.0.0.0` (or the office Instantel ACH
|
||||
server) once you are finished, and only stop the listener after that is done.
|
||||
|
||||
### A6 — do NOT re-enable ACH until the hardware fault is repaired
|
||||
|
||||
Otherwise the loop restarts the moment monitoring resumes and you run this
|
||||
runbook again.
|
||||
|
||||
---
|
||||
|
||||
## Method B (fallback) — catch it between calls
|
||||
|
||||
The original 2026-05 procedure. Use when you cannot stand up a listener the
|
||||
modem can reach. You need **ACEmanager access** to the unit's modem.
|
||||
|
||||
### Step 1: stop the modem's mode-flipping
|
||||
|
||||
@@ -287,9 +370,14 @@ recovery: ~7 hours of intermittent debugging across one evening.
|
||||
level. Constant triggering → constant recording → ACH "after event recorded"
|
||||
→ continuous dialing. Same disease as BE9558H.
|
||||
|
||||
**But the recovery was the opposite direction**, and none of the Step 1–4
|
||||
procedure above worked. Total time ≈ 5 h, of which ~90 min was spent on two
|
||||
red herrings documented below.
|
||||
**Same disease, inverted cure.** Method B's Step 1 *did* work — clearing the
|
||||
Destination stopped the dial-outs, confirmed in the ALEOS log. It was Step 2
|
||||
that didn't land, and rather than keep racing we turned the rescue around:
|
||||
gave the unit a different server to call, and answered it.
|
||||
|
||||
Total time ≈ 5 h, of which ~90 min went to two red herrings documented below.
|
||||
Much of the rest was rediscovering the May procedure, which is why the
|
||||
"two cures" table now sits at the top of this file.
|
||||
|
||||
---
|
||||
|
||||
@@ -302,7 +390,7 @@ DEBUG**, then view the serial log.
|
||||
It is the only thing that tells you what the *device* is actually saying.
|
||||
Everything before we did this was guesswork.
|
||||
|
||||
## What the log showed — the device was never in S3 mode
|
||||
## What the log showed — the unit is on the phone
|
||||
|
||||
Every ~75 seconds, verbatim:
|
||||
|
||||
@@ -326,9 +414,15 @@ Read that carefully:
|
||||
- The device therefore never receives `OK`, never progresses, and **retries
|
||||
the identical 29 bytes forever**.
|
||||
|
||||
**Consequence: the device is not running the S3 protocol parser.** You can
|
||||
land a byte-perfect S3 frame on it and it will be ignored. This is why every
|
||||
inbound approach failed, and it is the structural difference from BE9558H.
|
||||
**While it is in this state it is busy placing a call, not listening for
|
||||
us.** This is almost certainly what BE9558H was doing too — we simply never
|
||||
turned on ALEOS_SERIAL debug in May to look. It is not a different disease;
|
||||
it is the same one, seen properly for the first time.
|
||||
|
||||
It is also the argument for Method A in one picture: the unit is mid-dial
|
||||
every ~75 s, and our inbound Stop has to thread the gaps between those
|
||||
attempts. Give it somewhere to dial and the problem inverts into a
|
||||
deterministic one.
|
||||
|
||||
### Why `slow_drip` lied
|
||||
|
||||
@@ -351,6 +445,12 @@ nobody was reading.
|
||||
⚠ **`send_error: null` + full duration is NOT success. Only
|
||||
`bytes_received > 0` is success.**
|
||||
|
||||
⚠ **In fairness to slow_drip: it got exactly one attempt here**, run ~90 s
|
||||
after a modem reboot, with a dead session visible in the log at 20:19:17 in
|
||||
that same window. BE9558H took hours of attempts before one landed. Method B
|
||||
was not ruled out on BE12599 so much as abandoned in favour of something that
|
||||
doesn't need luck.
|
||||
|
||||
---
|
||||
|
||||
## ⚠ Two red herrings that cost ~90 minutes
|
||||
|
||||
Reference in New Issue
Block a user