seismo-relay/docs/runbooks/wedged_unit_recovery.md

Runbook — Recovering a wedged unit stuck in a call-home loop

Original incident: BE9558H at 166.246.130.1:9034, recovered 2026-05-17.

A field unit with a stuck-triggered geophone (or any hardware fault causing constant event triggering) will record events back-to-back, and if Auto Call Home is set to "After Event Recorded" the device will dial the office BW ACH server in a tight loop. Combined with a Sierra Wireless modem in bidirectional serial-TCP mode, this makes the unit effectively unreachable from SFM — every TCP connection we open gets killed when the modem flips from server-mode to client-mode to honor the device's next AT dial command.

This runbook describes how to break the loop and recover control.

---

⚠ There are TWO failure modes under "wedged unit"

Same root cause — an offset/connector fault drives the geophone above trigger, the unit records back-to-back, ACH set to "after event recorded" dials constantly — but the recovery differs, because the thing blocking you is different.

	BE9558H (2026-05)	BE12599 (2026-09)
What blocks you	Modem mode-flipping kills inbound TCP	Device never enters S3 mode at all
Device state	Alive, in S3 mode, responsive once reached	Stuck repeating an AT modem-init string, deaf to S3
Fix direction	Inbound — clear Destination, slow-drip a Stop	Inverted — point Destination at our own ACH server and let the unit call us
Winning tool	scripts/slow_drip.sh	bridges/ach_server.py --stop-monitoring

Tell them apart with the ALEOS serial log (see "Turn on ALEOS_SERIAL debug" below). If the device is emitting ATQ1/ATE0/ATS0=2 every ~75 s, it is in the BE12599 mode and no amount of inbound work will reach it — skip to "Second incident" below.

---

Symptoms

• Terra-View / SFM /device/info either hangs or fails on count_events().
• /device/monitor/status and /device/rescue return 502 (protocol timeout waiting for POLL response) or 503 (TCP connect refused).
• ACEmanager serial log shows repeating Connect to IP: <BW_IP> Port: <BW_PORT> → Shutdown TCP socket cycles every 30-60 seconds.
• Spam-mode endpoints (/device/stop_monitoring_spam) report many sent_ok but the device's monitoring state never changes.
• slow_drip reports [Errno 32] Broken pipe after sending the preamble but before completing the drip loop.

If you see all of these, the unit is in this exact failure mode.

---

Quick reference — how to recover

You need ACEmanager access to the unit's modem.

Step 1: stop the modem's mode-flipping

In ACEmanager → Serial → Port Configuration:

Field	Set to
Destination Address	clear (blank)
Destination Port	0

Click Apply. This removes the modem's auto-dial-out target. The device's AT dial commands now error back at the modem instead of triggering a mode-flip, so the modem stays in TCP-server mode permanently and our inbound TCP sessions stay alive.

(Optional belt-and-suspenders: also add the BW server's port to Security → Port Filtering - Outbound as a blocked port, with Outbound Port Filtering Mode = Blocked Ports.)

Step 2: stop monitoring on the device (slow drip)

From the SFM host:

/home/serversdown/seismo-relay/scripts/slow_drip.sh <DEVICE_IP> <PORT>

Defaults are 120s duration with a drip every 3s. Watch the response:

• duration_s ≈ 120 and drips_sent ≈ 40 → session held the full duration ✓
• bytes_received > 0 → device is responding ✓ (this is the success signal)

If duration_s is small or send_error: "Broken pipe", Step 1 didn't take hold — re-check ACEmanager, may need to reboot the modem after Apply.

Step 3: confirm monitoring stopped

curl 'http://localhost:8200/device/monitor/status?host=<DEVICE_IP>&tcp_port=<PORT>&force=true'
# expect: {"is_monitoring": false, ...}

Step 4: disable ACH at the device level + erase corrupted events

Either fire the rescue endpoint:

/home/serversdown/seismo-relay/scripts/rescue_device.sh <DEVICE_IP> <PORT>

Or do the two steps manually:

# Disable ACH in the device's compliance config
curl -X POST 'http://localhost:8200/device/call_home?host=<DEVICE_IP>&tcp_port=<PORT>' \
  -H 'Content-Type: application/json' \
  -d '{"auto_call_home_enabled": false}'

# Erase corrupted event chain
curl -X POST 'http://localhost:8200/device/events/erase?host=<DEVICE_IP>&tcp_port=<PORT>'

You can also do this via the SFM standalone UI → Call Home tab → set Enable Auto Call Home to Disabled → Write to Device.

Step 5: restore modem config (housekeeping)

Once the device-side ACH is disabled, restore the modem's Destination Address and Port to the original values (e.g. 50.197.32.92 / 12345) in ACEmanager. The modem will resume normal bidirectional behavior, but the unit won't issue any dial commands until ACH is explicitly re-enabled on the device.

Step 6: do NOT re-enable ACH on this unit until the underlying hardware

fault is repaired. If you do, the call-home loop starts again immediately and you'll be running this runbook a second time.

---

Why this works — the failure mode explained

The Sierra Wireless RV50/RV55 serial port operates in one of two TCP modes at any moment:

• Server mode — listens on Device Port (e.g. 9034), bridges inbound TCP to the device's serial port. This is what we need to interact with the device.
• Client mode — when the device sends an AT dial command on its serial TX line, the modem opens an outbound TCP to Destination Address:Port and bridges that to serial.

A serial port in this configuration is bidirectional: the modem flips between server and client modes on demand. When the device's firmware is healthy and only dials occasionally, this works fine.

When the unit is constantly triggering events and ACH is set to "After Event Recorded", the device sends an AT dial command every few seconds. Each one causes the modem to:

1. Drop any active inbound TCP session
2. Flip to client mode
3. Attempt outbound TCP to Destination Address:Port
4. Hang for up to a minute waiting for it to succeed/fail
5. Drop back to server mode

During the entire hang, no inbound TCP can establish. Even between hangs, the modem closes any existing inbound session before flipping. So any tool that needs more than a few seconds of held TCP (e.g. POLL + config read + write) gets repeatedly kicked off.

Clearing Destination Address removes step 3-4 from the cycle: the modem has nowhere to dial, so it doesn't flip modes when it receives an AT dial command. The serial port effectively becomes server-only, and inbound TCP sessions can stay open as long as needed.

This is a modem-layer issue, not a device firmware issue. The device is alive and responsive the whole time — confirmed in the BE9558H recovery by 990 bytes of S3 responses received over a 120s slow-drip session once the modem was no longer mode-flipping.

---

Why simpler approaches don't work

Approach	Why it fails
Standard /device/info	Triggers count_events() 1E/1F walk, takes 90s+ and hits corrupted event chain in this scenario
/device/rescue race loop	Gets 502 (protocol timeout) because the modem closes the TCP before the POLL handshake can complete
/device/stop_monitoring_blind (single frame)	Even if the bytes leave the wire, the device's protocol parser ignores write commands without a preceding POLL handshake (early-version bug, now fixed by including POLL preamble in blind sends)
/device/stop_monitoring_spam (sub-second cadence)	Each session is killed by the modem's mode-flip before the device can drain its UART RX buffer; high-rate spam also risks UART FIFO overrun on the device side
Outbound port firewall block alone	Stops the outbound TCP from succeeding, but doesn't stop the modem from trying and mode-flipping. Reduces but doesn't eliminate the contention.
Modem reboot	Temporary — as soon as the device starts triggering again, the loop resumes within seconds

The combination of slow_drip + cleared Destination Address works because:

1. The modem stops mode-flipping → TCP session stays open for the full drip duration
2. Slow drip rate → device's UART RX FIFO never overflows even if firmware is busy with event recording
3. The drip is SESSION_RESET + STOP_MONITORING every 3s → many independent chances for the parser to land one valid frame
4. Once one Stop Monitoring is parsed, event recording halts → firmware has CPU to spare → subsequent operations are trivially easy

---

Tooling reference

All endpoints live in seismo-relay/sfm/server.py. All scripts live in seismo-relay/scripts/ and default to SFM direct (http://localhost:8200), overridable via SFM_BASE_URL.

Endpoints added during BE9558H recovery

Endpoint	Purpose
GET /device/events/storage_range	SUB 0x06 — first/last event keys, is_empty flag. ~2s, no event walk.
GET /device/events/index	SUB 0x08 — lifetime event counter (does NOT decrement on erase). ~2s.
POST /device/events/erase	Full erase sequence 0xA3 → 0x1C → 0x06 → 0xA2.
POST /device/rescue	Disable ACH + erase in one TCP session. Short timeouts for race-loop usage.
POST /device/stop_monitoring_blind	Fire-and-forget Stop with full POLL preamble (single attempt).
POST /device/stop_monitoring_spam	Server-side tight retry loop, sub-second cadence, duration-bounded.
POST /device/stop_monitoring_slow_drip	One held TCP session, slow trickle of stop frames. The endpoint that saved BE9558H.

Also changed: default protocol recv timeout dropped from 30s → 10s in _build_client. Added connect_timeout knob to same. Cleaned up unhandled-exception path in /device/monitor/status so it returns 502 instead of 500 on protocol timeouts.

Scripts

Script	Purpose
scripts/rescue_device.sh	Race-loop wrapper around /device/rescue
scripts/blind_stop.sh	Race-loop wrapper around /device/stop_monitoring_blind
scripts/spam_stop.sh	Single-call burst hammer (/device/stop_monitoring_spam)
scripts/slow_drip.sh	Single-call held-session drip (/device/stop_monitoring_slow_drip)
scripts/watch_unit.sh	Passive periodic reachability check, logs to file

---

Incident log — BE9558H, 2026-05-16/17

What was wrong: Long-axis geophone developed an offset, constantly above trigger threshold → constant event recording → after-event ACH set → modem dialing office BW server (50.197.32.92:12345) every 30-60s. Local event chain corrupted (next_boundary 0x100EE exceeds uint16).

Diagnostic path:

1. /device/info slow, choked on event walk
2. Built lightweight probe endpoints (storage_range, index) — useful but didn't reach the wedged unit
3. Built /device/rescue with short timeouts — got 502 (POLL no response)
4. Built /device/stop_monitoring_blind — first version was a false positive (no POLL preamble); fixed by including SESSION_RESET+POLL_PROBE+SESSION_RESET+POLL_DATA in the dump
5. Verified blind stop works on bench unit
6. Built /device/stop_monitoring_spam — 420 successful sends over 5 min, zero behavior change on field unit
7. Inspected ACEmanager logs → saw outbound dial-out attempts every ~30s, confirmed device was not fully locked up
8. Added outbound port-12345 firewall block → outbound attempts now fail instantly but contention persisted
9. Built /device/stop_monitoring_slow_drip — session died at 3s with broken pipe (modem closing on us)
10. Looked at full ACEmanager Port Configuration → found Destination Address: 50.197.32.92 configured, realized every AT dial command was triggering a modem mode-flip that killed our inbound
11. Cleared Destination Address + Port → slow_drip held 120s, device responded with 990 bytes, 39 stop commands acked
12. Disabled ACH at device level via /device/call_home, erased events

Final state: device IDLE, memory 958.1 / 960 KB free, ACH disabled at device level, modem destination cleared (to be restored after physical service).

Total time from "i was wondering if its possible to" first attempt to recovery: ~7 hours of intermittent debugging across one evening.

---

Second incident — BE12599, 2026-09-16/17

Unit: BE12599 at 166.246.64.226:9034, RV50, job I-80 North Fork Bridge — Abut 1 West (Fay Company). Same job as BE9558H, which is a coincidence.

Fault: the connector fault documented in docs/offset_investigation.md §8e progressed until the Tran pedestal reached 0.400 in/s — its trigger level. Constant triggering → constant recording → ACH "after event recorded" → continuous dialing. Same disease as BE9558H.

But the recovery was the opposite direction, and none of the Step 1–4 procedure above worked. Total time ≈ 5 h, of which ~90 min was spent on two red herrings documented below.

---

Turn on ALEOS_SERIAL debug FIRST

This is the single highest-value diagnostic and it should be step zero on any future incident. ACEmanager → Admin → Log → ALEOS_SERIAL log level → DEBUG, then view the serial log.

It is the only thing that tells you what the device is actually saying. Everything before we did this was guesswork.

What the log showed — the device was never in S3 mode

Every ~75 seconds, verbatim:

ALEOS_SERIAL_HIF: 29 byte(s) in buffer: 'ATQ1^MATE0^MATS0=2^M^MRADIO RING^M'
ALEOS_SERIAL_HMC: TCP recvhost fd 65535 len 29  state TCPMode::kClosed
ALEOS_SERIAL_HMC: tcpmode trying to send to invalid socket
ALEOS_SERIAL_HMC: Connect to IP: 0.0.0.0 Port 0
ALEOS_SERIAL_HMC: Initialize Auto answer on port 9034
ALEOS_SERIAL_HMC: Cannot connect to 0.0.0.0

Read that carefully:

• ATQ1 (quiet) / ATE0 (echo off) / ATS0=2 (auto-answer after 2 rings). There is no ATD. The device is not dialing — it is trying to configure its modem.
• The modem's serial port is in TCP data mode, so it never interprets these as AT commands. It treats them as payload and tries to ship them to a TCP socket that does not exist.
• The device therefore never receives OK, never progresses, and retries the identical 29 bytes forever.

Consequence: the device is not running the S3 protocol parser. You can land a byte-perfect S3 frame on it and it will be ignored. This is why every inbound approach failed, and it is the structural difference from BE9558H.

Why slow_drip lied

slow_drip returned the success signature except for the one field that mattered:

{"duration_s":120.0,"drips_sent":38,"bytes_sent":920,
 "bytes_received":0,"send_error":null}

Full duration, no broken pipe — but zero bytes back. Cause is in the log above: each 75 s cycle re-runs Initialize Auto answer on port 9034, which orphans the held session (data in for unknown reason 3 removing from select, OnMsg recv error: 107 - Transport endpoint is not connected). Our local TCP stayed open so sendall never raised — but the modem stopped bridging after the first re-init, so every drip after that went into a socket nobody was reading.

⚠ send_error: null + full duration is NOT success. Only bytes_received > 0 is success.

---

⚠ Two red herrings that cost ~90 minutes

1. The trusted-IP whitelist (this was the real reason inbound never worked)

The RV50s run with Security → Trusted IPs (Friends List) enabled. A source IP that is not on the list is dropped silently — inbound presents as Connection error: timed out, never a refusal.

Brian's dev-box public IP is dynamic and had changed, so tmi-dev was no longer whitelisted. Every inbound attempt failed identically across four different modem and device states, which looked exactly like the BE9558H mode-flipping symptom and sent us chasing modem configuration for over an hour.

Check this before diagnosing anything else. Note that SFM in Docker egresses via the host's public IP, not its LAN IP.

2. A 502 from SFM does not mean TCP connected

sfm/server.py raises 502 for both failure classes:

raise HTTPException(status_code=502, detail=f"Protocol error: {exc}")
raise HTTPException(status_code=502, detail=f"Connection error: {exc}")

We read an early 502 as "TCP connected, modem bridged, device mute" and built a whole theory on it. It was almost certainly a connect timeout. Always read the detail string — "connect failed" and "device didn't answer" are completely different problems and the status code will not separate them.

---

What actually worked — invert the direction

The key observation is in the log above:

TCP recvhost ... state TCPMode::kClosed → Connect to IP: 0.0.0.0 Port 0

The modem auto-dials its Destination whenever serial data arrives while closed. So instead of fighting for inbound, give it somewhere to dial: point Destination Address at our own ach_server and the device's own 75-second attempts become device-initiated sessions the modem bridges correctly. No race, no contention, worst case a 75-second wait.

Procedure

1. Run the rescue server on a host the modem can reach (public IP + forwarded port):

   cd /home/serversdown/seismo-relay
   .venv/bin/python -u bridges/ach_server.py --port 12345 \
     -o bridges/captures/<unit>-diag --stop-monitoring -v
   

2. Point the modem at it — ACEmanager → Serial → Port Configuration → Destination Address = your public IP, Destination Port = 12345.

3. Wait for the call-in. --stop-monitoring fires SUB 0x97 at step 1.5, after the handshake and before the event walk. Confirm via rescue.json in the session directory:

   {"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
   

4. Restore the modem's Destination once you are done, then finish the device side (disable ACH, erase) through whichever channel works.

On BE12599 the first call-in landed at 20:58:11 and reported stop_monitoring: ok; a second at 20:58:20 confirmed it. is_monitoring: false was still true 6½ hours later — the fix is durable.

---

Hard-won gotchas (do not re-derive)

• Never leave the Destination pointed at a host with nothing listening. That is the worst state available: the device still dials, the modem still flips, inbound stays blocked, and nothing is delivered. An 8-minute gap with the listener down produced a spurious inbound timeout that cost another round of misdiagnosis.

• Stopping monitoring removes your call-in channel. ACH is "after event recorded"; no new events means no new dials. The backlog sitting in memory does not re-arm it. After a successful stop the unit goes quiet and you need the modem cycled (works — produced a call-in), the scheduled daily call (BE12599 calls at 05:00:14 device-local, per §8e), or working inbound. Plan the order before you fire the stop.

• --events-only silently breaks dedup. It skips the device-info step, so the serial is never read; ach_state.json then keys on peer:ephemeral_port, which is unique per connection. Every session looks like a new unit, starts from key 0, and re-downloads the same event. Four sessions on BE12599 downloaded the identical event four times and made zero progress on the backlog. Events also file as serial=UNKNOWN with a M000… BW filename (serial_numeric 0) instead of N599…. Do not use --events-only when you intend to download anything.

• /device/events/index reported lifetime_count: 0 on a unit with years of history. Suspected decode bug in the SUB 0x08 field offset — do not trust that number. The 88-byte payload is preserved in the raw_hex field if someone wants to chase it.

• Memory used cross-checks the event keys exactly: last_key − buffer_start = memory_total − memory_free. On BE12599: 0x011230ec − 0x01110000 = 78,060 and 983,028 − 904,968 = 78,060. Useful sanity check that you are reading the keys right.

---

Final state (2026-09-17 ~01:30 local)

• is_monitoring: false, held 6½ hours
• Battery 6.76 V
• Memory 78,060 / 983,028 bytes used (8%)
• first_key 01121728, last_key 011230ec — ~6.6 KB of addressable event chain, roughly 3 events
• ACH still enabled — to be disabled after the backlog is preserved
• Modem Destination still pointed at tmi-dev — to be restored
• ⚠ Do not re-enable ACH until the connector is serviced. Tran is still sitting at 0.400 and the loop restarts the moment monitoring resumes.