seismo-relay/docs/runbooks/wedged_unit_recovery.md

# Runbook — Recovering a wedged unit stuck in a call-home loop

**Incidents:** BE9558H at `166.246.130.1:9034`, 2026-05-17 (Method B) ·
BE12599 at `166.246.64.226:9034`, 2026-09-16 (Method A).

A field unit with a stuck-triggered geophone (or any hardware fault causing
constant event triggering) will record events back-to-back, and if Auto Call
Home is set to "After Event Recorded" the device will dial the office BW
ACH server in a tight loop. Combined with a Sierra Wireless modem in
bidirectional serial-TCP mode, this makes the unit effectively unreachable
from SFM — every TCP connection we open gets killed when the modem flips
from server-mode to client-mode to honor the device's next AT dial command.

This runbook describes how to break the loop and recover control.

---

## ⚠ Two cures for one disease — intercept first

Both incidents below are the **same failure**: a geophone offset crosses the
trigger level, the unit records back-to-back, ACH set to "after event
recorded" dials continuously, and the unit becomes unreachable because its
modem is in client mode almost all of the time.

There are two ways to get a Stop Monitoring command into it.

| | **A — intercept the call** (preferred) | **B — catch it between calls** (original) |
|---|---|---|
| Idea | Be the server it dials.  Point the modem's Destination at our own ACH server and answer it. | Clear the Destination so it stops dialing, then race a Stop into the gap. |
| Needs inbound? | **No — the unit calls us** | Yes: working inbound TCP to the modem |
| Determinism | Deterministic — it dials every ~75 s, we only have to be listening | A race.  BE9558H took ~7 h of attempts before one landed. |
| Tool | `bridges/ach_server.py --stop-monitoring` | `scripts/slow_drip.sh` |
| Proven on | BE12599, 2026-09-16 | BE9558H, 2026-05-17 |

**Method A is the standard procedure now.**  The unit won't answer us because
it is on the phone — so stop dialing it and be the one it calls.  It rings,
we pick up, take its data, and tell it to stop calling here.

Method B is kept because it is proven, and because A needs a listener the
modem can actually reach (public IP + forwarded port).  When you have that,
don't race it — intercept it.

---

## Symptoms

- Terra-View / SFM `/device/info` either hangs or fails on `count_events()`.
- `/device/monitor/status` and `/device/rescue` return 502 (protocol timeout
  waiting for POLL response) or 503 (TCP connect refused).
- ACEmanager serial log shows repeating
  `Connect to IP: <BW_IP> Port: <BW_PORT>` → `Shutdown TCP socket` cycles
  every 30-60 seconds.
- Spam-mode endpoints (`/device/stop_monitoring_spam`) report many
  `sent_ok` but the device's monitoring state never changes.
- `slow_drip` reports `[Errno 32] Broken pipe` after sending the preamble
  but before completing the drip loop.

If you see *all* of these, the unit is in this exact failure mode.

---

## Method A (preferred) — intercept the call

You need **ACEmanager access** and a host the modem can dial: public IP with
the listener's port forwarded to it.

### A1 — start the listener BEFORE touching the modem

```bash
cd /home/serversdown/seismo-relay
tmux new -s rescue
.venv/bin/python -u bridges/ach_server.py --port 12345 \
  -o bridges/captures/<unit>-diag --stop-monitoring -v
```

⚠ **Listener first, always.**  A Destination pointed at a dead port is the
worst state available — the device still dials, the modem still flips to
client mode, inbound stays blocked, and nothing gets delivered.

Do **not** add `--events-only` (it silently breaks dedup — see gotchas), and
do **not** add `--disable-ach` yet (see A4).

### A2 — point the modem at it

ACEmanager → **Serial → Port Configuration**:

| Field | Set to |
|---|---|
| **Destination Address** | the listener's public IP |
| **Destination Port** | the listener's port (e.g. `12345`) |

Apply.  The modem auto-dials its Destination whenever serial data arrives
while the serial port is closed — so the unit's own retry cycle now lands on
you instead of nowhere.

### A3 — answer, and stop the bleeding

Within ~75 s you should see a call-in.  `--stop-monitoring` fires SUB 0x97 at
step 1.5 — after the handshake, **before** the event walk — so the recording
halts at the earliest possible moment in the session.  Confirm via
`rescue.json` in the session directory:

```json
{"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
```

That is the bleeding stopped.  Everything after this is cleanup.

### A4 — drain the backlog, THEN disable ACH

⚠ **Order matters, and it is counter-intuitive.**  Stopping monitoring also
removes your call-in trigger: ACH fires on "after event recorded", so with
recording stopped the unit has no reason to dial again.  The backlog sitting
in its memory does **not** re-arm it.

So if the stored events are worth keeping — and on a fault unit they usually
are, they're the evidence — drain them across however many call-ins it takes
*before* you silence it.  Only then add `--disable-ach` (or use
`scripts/rescue_device.sh <host> <port> --no-erase`).

If the unit has gone quiet and you still need it, cycling the modem produces
a call-in, and a unit with a scheduled daily call will dial at its configured
time regardless.

### A5 — restore the Destination, and confirm you did

Put `Destination Address` back to `0.0.0.0` (or the office Instantel ACH
server) once you are finished, and only stop the listener after that is done.

### A6 — do NOT re-enable ACH until the hardware fault is repaired

Otherwise the loop restarts the moment monitoring resumes and you run this
runbook again.

---

## Method B (fallback) — catch it between calls

The original 2026-05 procedure.  Use when you cannot stand up a listener the
modem can reach.  You need **ACEmanager access** to the unit's modem.

### Step 1: stop the modem's mode-flipping

In ACEmanager → **Serial → Port Configuration**:

| Field | Set to |
|---|---|
| **Destination Address** | clear (blank) |
| **Destination Port** | `0` |

Click **Apply**. This removes the modem's auto-dial-out target. The device's
AT dial commands now error back at the modem instead of triggering a
mode-flip, so the modem stays in TCP-server mode permanently and our inbound
TCP sessions stay alive.

*(Optional belt-and-suspenders: also add the BW server's port to
**Security → Port Filtering - Outbound** as a blocked port, with
Outbound Port Filtering Mode = Blocked Ports.)*

### Step 2: stop monitoring on the device (slow drip)

From the SFM host:

```bash
/home/serversdown/seismo-relay/scripts/slow_drip.sh <DEVICE_IP> <PORT>
```

Defaults are 120s duration with a drip every 3s. Watch the response:

- `duration_s ≈ 120` and `drips_sent ≈ 40` → session held the full duration ✓
- `bytes_received > 0` → device is responding ✓ (this is the success signal)

If `duration_s` is small or `send_error: "Broken pipe"`, Step 1 didn't take
hold — re-check ACEmanager, may need to reboot the modem after Apply.

### Step 3: confirm monitoring stopped

```bash
curl 'http://localhost:8200/device/monitor/status?host=<DEVICE_IP>&tcp_port=<PORT>&force=true'
# expect: {"is_monitoring": false, ...}
```

### Step 4: disable ACH at the device level + erase corrupted events

Either fire the rescue endpoint:

```bash
/home/serversdown/seismo-relay/scripts/rescue_device.sh <DEVICE_IP> <PORT>
```

Or do the two steps manually:

```bash
# Disable ACH in the device's compliance config
curl -X POST 'http://localhost:8200/device/call_home?host=<DEVICE_IP>&tcp_port=<PORT>' \
  -H 'Content-Type: application/json' \
  -d '{"auto_call_home_enabled": false}'

# Erase corrupted event chain
curl -X POST 'http://localhost:8200/device/events/erase?host=<DEVICE_IP>&tcp_port=<PORT>'
```

You can also do this via the SFM standalone UI → **Call Home** tab → set
`Enable Auto Call Home` to `Disabled` → **Write to Device**.

### Step 5: restore modem config (housekeeping)

Once the device-side ACH is disabled, restore the modem's Destination
Address and Port to the original values (e.g. `50.197.32.92` / `12345`) in
ACEmanager. The modem will resume normal bidirectional behavior, but the
unit won't issue any dial commands until ACH is explicitly re-enabled on
the device.

### Step 6: do NOT re-enable ACH on this unit until the underlying hardware
fault is repaired. If you do, the call-home loop starts again immediately
and you'll be running this runbook a second time.

---

## Why this works — the failure mode explained

The Sierra Wireless RV50/RV55 serial port operates in one of two TCP modes
at any moment:

- **Server mode** — listens on `Device Port` (e.g. 9034), bridges inbound
  TCP to the device's serial port. This is what we need to interact with
  the device.
- **Client mode** — when the device sends an AT dial command on its serial
  TX line, the modem opens an outbound TCP to `Destination Address:Port`
  and bridges that to serial.

A serial port in this configuration is **bidirectional**: the modem flips
between server and client modes on demand. When the device's firmware is
healthy and only dials occasionally, this works fine.

When the unit is constantly triggering events and ACH is set to "After
Event Recorded", the device sends an AT dial command every few seconds.
Each one causes the modem to:

1. Drop any active inbound TCP session
2. Flip to client mode
3. Attempt outbound TCP to `Destination Address:Port`
4. Hang for up to a minute waiting for it to succeed/fail
5. Drop back to server mode

**During the entire hang, no inbound TCP can establish.** Even between
hangs, the modem closes any existing inbound session before flipping. So
any tool that needs more than a few seconds of held TCP (e.g. POLL +
config read + write) gets repeatedly kicked off.

Clearing `Destination Address` removes step 3-4 from the cycle: the modem
has nowhere to dial, so it doesn't flip modes when it receives an AT dial
command. The serial port effectively becomes server-only, and inbound TCP
sessions can stay open as long as needed.

**This is a modem-layer issue, not a device firmware issue.** The device
is alive and responsive the whole time — confirmed in the BE9558H
recovery by 990 bytes of S3 responses received over a 120s slow-drip
session once the modem was no longer mode-flipping.

---

## Why simpler approaches don't work

| Approach | Why it fails |
|---|---|
| Standard `/device/info` | Triggers `count_events()` 1E/1F walk, takes 90s+ and hits corrupted event chain in this scenario |
| `/device/rescue` race loop | Gets 502 (protocol timeout) because the modem closes the TCP before the POLL handshake can complete |
| `/device/stop_monitoring_blind` (single frame) | Even if the bytes leave the wire, the device's protocol parser ignores write commands without a preceding POLL handshake (early-version bug, now fixed by including POLL preamble in blind sends) |
| `/device/stop_monitoring_spam` (sub-second cadence) | Each session is killed by the modem's mode-flip before the device can drain its UART RX buffer; high-rate spam also risks UART FIFO overrun on the device side |
| Outbound port firewall block alone | Stops the outbound TCP from succeeding, but doesn't stop the modem from *trying* and mode-flipping. Reduces but doesn't eliminate the contention. |
| Modem reboot | Temporary — as soon as the device starts triggering again, the loop resumes within seconds |

The combination of `slow_drip` + cleared `Destination Address` works because:

1. The modem stops mode-flipping → TCP session stays open for the full
   drip duration
2. Slow drip rate → device's UART RX FIFO never overflows even if
   firmware is busy with event recording
3. The drip is `SESSION_RESET + STOP_MONITORING` every 3s → many
   independent chances for the parser to land one valid frame
4. Once one Stop Monitoring is parsed, event recording halts → firmware
   has CPU to spare → subsequent operations are trivially easy

---

## Tooling reference

All endpoints live in `seismo-relay/sfm/server.py`. All scripts live in
`seismo-relay/scripts/` and default to SFM direct (`http://localhost:8200`),
overridable via `SFM_BASE_URL`.

### Endpoints added during BE9558H recovery

| Endpoint | Purpose |
|---|---|
| `GET /device/events/storage_range` | SUB 0x06 — first/last event keys, `is_empty` flag. ~2s, no event walk. |
| `GET /device/events/index` | SUB 0x08 — lifetime event counter (does NOT decrement on erase). ~2s. |
| `POST /device/events/erase` | Full erase sequence 0xA3 → 0x1C → 0x06 → 0xA2. |
| `POST /device/rescue` | Disable ACH + erase in one TCP session. Short timeouts for race-loop usage. |
| `POST /device/stop_monitoring_blind` | Fire-and-forget Stop with full POLL preamble (single attempt). |
| `POST /device/stop_monitoring_spam` | Server-side tight retry loop, sub-second cadence, duration-bounded. |
| `POST /device/stop_monitoring_slow_drip` | One held TCP session, slow trickle of stop frames. **The endpoint that saved BE9558H.** |

Also changed: default protocol recv timeout dropped from 30s → 10s in
`_build_client`. Added `connect_timeout` knob to same. Cleaned up
unhandled-exception path in `/device/monitor/status` so it returns 502
instead of 500 on protocol timeouts.

### Scripts

| Script | Purpose |
|---|---|
| `scripts/rescue_device.sh` | Race-loop wrapper around `/device/rescue` |
| `scripts/blind_stop.sh` | Race-loop wrapper around `/device/stop_monitoring_blind` |
| `scripts/spam_stop.sh` | Single-call burst hammer (`/device/stop_monitoring_spam`) |
| `scripts/slow_drip.sh` | Single-call held-session drip (`/device/stop_monitoring_slow_drip`) |
| `scripts/watch_unit.sh` | Passive periodic reachability check, logs to file |

---

## Incident log — BE9558H, 2026-05-16/17

What was wrong: Long-axis geophone developed an offset, constantly above
trigger threshold → constant event recording → after-event ACH set →
modem dialing office BW server (`50.197.32.92:12345`) every 30-60s.
Local event chain corrupted (`next_boundary 0x100EE exceeds uint16`).

Diagnostic path:

1. `/device/info` slow, choked on event walk
2. Built lightweight probe endpoints (`storage_range`, `index`) — useful
   but didn't reach the wedged unit
3. Built `/device/rescue` with short timeouts — got 502 (POLL no response)
4. Built `/device/stop_monitoring_blind` — first version was a false
   positive (no POLL preamble); fixed by including
   `SESSION_RESET+POLL_PROBE+SESSION_RESET+POLL_DATA` in the dump
5. Verified blind stop works on bench unit
6. Built `/device/stop_monitoring_spam` — 420 successful sends over
   5 min, zero behavior change on field unit
7. Inspected ACEmanager logs → saw outbound dial-out attempts every ~30s,
   confirmed device was not fully locked up
8. Added outbound port-12345 firewall block → outbound attempts now fail
   instantly but contention persisted
9. Built `/device/stop_monitoring_slow_drip` — session died at 3s with
   broken pipe (modem closing on us)
10. Looked at full ACEmanager Port Configuration → **found
    `Destination Address: 50.197.32.92` configured**, realized every AT
    dial command was triggering a modem mode-flip that killed our inbound
11. Cleared Destination Address + Port → slow_drip held 120s, device
    responded with 990 bytes, 39 stop commands acked
12. Disabled ACH at device level via `/device/call_home`, erased events

Final state: device IDLE, memory 958.1 / 960 KB free, ACH disabled at
device level, modem destination cleared (to be restored after physical
service).

Total time from "i was wondering if its possible to" first attempt to
recovery: ~7 hours of intermittent debugging across one evening.

---

# Second incident — BE12599, 2026-09-16/17

**Unit:** BE12599 at `166.246.64.226:9034`, RV50, job *I-80 North Fork Bridge
— Abut 1 West* (Fay Company).  Same job as BE9558H, which is a coincidence.

**Fault:** the connector fault documented in `docs/offset_investigation.md`
§8e progressed until the Tran pedestal reached **0.400 in/s** — its trigger
level.  Constant triggering → constant recording → ACH "after event recorded"
→ continuous dialing.  Same disease as BE9558H.

**Same disease, inverted cure.**  Method B's Step 1 *did* work — clearing the
Destination stopped the dial-outs, confirmed in the ALEOS log.  It was Step 2
that didn't land, and rather than keep racing we turned the rescue around:
gave the unit a different server to call, and answered it.

Total time ≈ 5 h, of which ~90 min went to two red herrings documented below.
Much of the rest was rediscovering the May procedure, which is why the
"two cures" table now sits at the top of this file.

---

## Turn on ALEOS_SERIAL debug FIRST

This is the single highest-value diagnostic and it should be step zero on any
future incident.  ACEmanager → **Admin → Log → ALEOS_SERIAL log level →
DEBUG**, then view the serial log.

It is the only thing that tells you what the *device* is actually saying.
Everything before we did this was guesswork.

## What the log showed — the unit is on the phone

Every ~75 seconds, verbatim:

```
ALEOS_SERIAL_HIF: 29 byte(s) in buffer: 'ATQ1^MATE0^MATS0=2^M^MRADIO RING^M'
ALEOS_SERIAL_HMC: TCP recvhost fd 65535 len 29  state TCPMode::kClosed
ALEOS_SERIAL_HMC: tcpmode trying to send to invalid socket
ALEOS_SERIAL_HMC: Connect to IP: 0.0.0.0 Port 0
ALEOS_SERIAL_HMC: Initialize Auto answer on port 9034
ALEOS_SERIAL_HMC: Cannot connect to 0.0.0.0
```

Read that carefully:

- `ATQ1` (quiet) / `ATE0` (echo off) / `ATS0=2` (auto-answer after 2 rings).
  **There is no `ATD`.**  The device is not dialing — it is trying to
  *configure* its modem.
- The modem's serial port is in TCP data mode, so it never interprets these
  as AT commands.  It treats them as payload and tries to ship them to a TCP
  socket that does not exist.
- The device therefore never receives `OK`, never progresses, and **retries
  the identical 29 bytes forever**.

**While it is in this state it is busy placing a call, not listening for
us.**  This is almost certainly what BE9558H was doing too — we simply never
turned on ALEOS_SERIAL debug in May to look.  It is not a different disease;
it is the same one, seen properly for the first time.

It is also the argument for Method A in one picture: the unit is mid-dial
every ~75 s, and our inbound Stop has to thread the gaps between those
attempts.  Give it somewhere to dial and the problem inverts into a
deterministic one.

### Why `slow_drip` lied

`slow_drip` returned the *success* signature except for the one field that
mattered:

```json
{"duration_s":120.0,"drips_sent":38,"bytes_sent":920,
 "bytes_received":0,"send_error":null}
```

Full duration, no broken pipe — but zero bytes back.  Cause is in the log
above: each 75 s cycle re-runs `Initialize Auto answer on port 9034`, which
orphans the held session (`data in for unknown reason 3 removing from
select`, `OnMsg recv error: 107 - Transport endpoint is not connected`).  Our
local TCP stayed open so `sendall` never raised — but the modem stopped
bridging after the first re-init, so every drip after that went into a socket
nobody was reading.

⚠ **`send_error: null` + full duration is NOT success.  Only
`bytes_received > 0` is success.**

⚠ **In fairness to slow_drip: it got exactly one attempt here**, run ~90 s
after a modem reboot, with a dead session visible in the log at 20:19:17 in
that same window.  BE9558H took hours of attempts before one landed.  Method B
was not ruled out on BE12599 so much as abandoned in favour of something that
doesn't need luck.

---

## ⚠ Two red herrings that cost ~90 minutes

### 1. The trusted-IP whitelist (this was the real reason inbound never worked)

The RV50s run with **Security → Trusted IPs (Friends List) enabled**.  A
source IP that is not on the list is dropped **silently** — inbound presents
as `Connection error: timed out`, never a refusal.

Brian's dev-box public IP is **dynamic** and had changed, so `tmi-dev` was no
longer whitelisted.  Every inbound attempt failed identically across four
different modem and device states, which looked exactly like the BE9558H
mode-flipping symptom and sent us chasing modem configuration for over an
hour.

**Check this before diagnosing anything else.**  Note that SFM in Docker
egresses via the *host's public IP*, not its LAN IP.

### 2. A 502 from SFM does not mean TCP connected

`sfm/server.py` raises **502 for both** failure classes:

```python
raise HTTPException(status_code=502, detail=f"Protocol error: {exc}")
raise HTTPException(status_code=502, detail=f"Connection error: {exc}")
```

We read an early 502 as "TCP connected, modem bridged, device mute" and built
a whole theory on it.  It was almost certainly a connect timeout.
**Always read the `detail` string** — "connect failed" and "device didn't
answer" are completely different problems and the status code will not
separate them.

---

## What actually worked — invert the direction

The key observation is in the log above:

> `TCP recvhost ... state TCPMode::kClosed` → `Connect to IP: 0.0.0.0 Port 0`

**The modem auto-dials its Destination whenever serial data arrives while
closed.**  So instead of fighting for inbound, give it somewhere to dial:
point `Destination Address` at our own `ach_server` and the device's own
75-second attempts become **device-initiated sessions the modem bridges
correctly**.  No race, no contention, worst case a 75-second wait.

### Procedure

1. **Run the rescue server** on a host the modem can reach (public IP +
   forwarded port):

   ```bash
   cd /home/serversdown/seismo-relay
   .venv/bin/python -u bridges/ach_server.py --port 12345 \
     -o bridges/captures/<unit>-diag --stop-monitoring -v
   ```

2. **Point the modem at it** — ACEmanager → Serial → Port Configuration →
   `Destination Address` = your public IP, `Destination Port` = 12345.

3. **Wait for the call-in.**  `--stop-monitoring` fires SUB 0x97 at step 1.5,
   after the handshake and *before* the event walk.  Confirm via
   `rescue.json` in the session directory:

   ```json
   {"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
   ```

4. **Restore the modem's Destination** once you are done, then finish the
   device side (disable ACH, erase) through whichever channel works.

On BE12599 the first call-in landed at 20:58:11 and reported
`stop_monitoring: ok`; a second at 20:58:20 confirmed it.  `is_monitoring:
false` was still true **6½ hours later** — the fix is durable.

---

## Hard-won gotchas (do not re-derive)

- **Never leave the Destination pointed at a host with nothing listening.**
  That is the worst state available: the device still dials, the modem still
  flips, inbound stays blocked, and nothing is delivered.  An 8-minute gap
  with the listener down produced a spurious inbound timeout that cost
  another round of misdiagnosis.

- **Stopping monitoring removes your call-in channel.**  ACH is "after event
  recorded"; no new events means no new dials.  The backlog sitting in memory
  does *not* re-arm it.  After a successful stop the unit goes quiet and you
  need the modem cycled (works — produced a call-in), the scheduled daily call
  (BE12599 calls at **05:00:14 device-local**, per §8e), or working inbound.
  **Plan the order before you fire the stop.**

- **`--events-only` silently breaks dedup.**  It skips the device-info step,
  so the serial is never read; `ach_state.json` then keys on
  `peer:ephemeral_port`, which is unique per connection.  Every session looks
  like a new unit, starts from key 0, and re-downloads the same event.  Four
  sessions on BE12599 downloaded the identical event four times and made zero
  progress on the backlog.  Events also file as `serial=UNKNOWN` with a
  `M000…` BW filename (serial_numeric 0) instead of `N599…`.
  **Do not use `--events-only` when you intend to download anything.**

- **`/device/events/index` reported `lifetime_count: 0`** on a unit with years
  of history.  Suspected decode bug in the SUB 0x08 field offset — do not
  trust that number.  The 88-byte payload is preserved in the `raw_hex` field
  if someone wants to chase it.

- **Memory used cross-checks the event keys exactly:**
  `last_key − buffer_start = memory_total − memory_free`.  On BE12599:
  `0x011230ec − 0x01110000 = 78,060` and `983,028 − 904,968 = 78,060`.
  Useful sanity check that you are reading the keys right.

---

## Final state (2026-09-17 ~01:30 local)

- `is_monitoring: false`, held 6½ hours
- Battery 6.76 V
- Memory 78,060 / 983,028 bytes used (8%)
- `first_key 01121728`, `last_key 011230ec` — ~6.6 KB of addressable event
  chain, roughly 3 events
- ACH still **enabled** — to be disabled after the backlog is preserved
- Modem Destination still pointed at tmi-dev — to be restored
- ⚠ **Do not re-enable ACH until the connector is serviced.**  Tran is still
  sitting at 0.400 and the loop restarts the moment monitoring resumes.