docs: BE12599 incident — the inverted rescue, plus a rescue-listener plan
The wedged_unit_recovery runbook covered exactly one failure mode. BE12599 turned out to be a second one wearing the same symptoms, and the existing procedure did not work on it. Adds a "TWO failure modes" table up front so the next incident branches correctly, and a full second-incident section covering what the ALEOS serial debug log revealed: the device repeating a 29-byte AT modem-init string (ATQ1/ATE0/ATS0=2, no ATD) every 75 s, never getting an OK because the modem is in TCP data mode, and therefore never entering S3 mode at all. Inbound cannot win against that, no matter how well framed. Also records the two red herrings, since together they cost ~90 minutes: the RV50 trusted-IP whitelist drops non-listed sources silently (presents as a connect timeout, and Brian's dynamic dev IP had rotated off the list), and sfm/server.py returns 502 for BOTH "Protocol error:" and "Connection error:", so a 502 was misread as "TCP connected, device mute" and a theory built on it. And the gotchas worth never re-deriving: slow_drip's send_error=null plus a full duration is not success (only bytes_received > 0 is); stopping monitoring removes the call-in trigger, so it costs you the channel; --events-only skips the device-info step, so the serial is never read and ach_state keys on peer:ephemeral_port, silently breaking dedup and re-downloading the same event every session. The plan doc captures the tool Brian wants built out of this — a rescue listener with a real lifecycle and, critically, a confirmation gate before shutdown, because leaving the modem's Destination pointed at a dead listener is worse than never having started. Open questions are listed rather than guessed at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN
This commit is contained in:
Claude Opus 5
@@ -14,6 +14,27 @@ This runbook describes how to break the loop and recover control.
|
||||
|
||||
---
|
||||
|
||||
## ⚠ There are TWO failure modes under "wedged unit"
|
||||
|
||||
Same root cause — an offset/connector fault drives the geophone above trigger,
|
||||
the unit records back-to-back, ACH set to "after event recorded" dials
|
||||
constantly — but the *recovery* differs, because the thing blocking you is
|
||||
different.
|
||||
|
||||
| | **BE9558H (2026-05)** | **BE12599 (2026-09)** |
|
||||
|---|---|---|
|
||||
| What blocks you | Modem mode-flipping kills inbound TCP | Device never enters S3 mode at all |
|
||||
| Device state | Alive, in S3 mode, responsive once reached | Stuck repeating an **AT modem-init** string, deaf to S3 |
|
||||
| Fix direction | **Inbound** — clear Destination, slow-drip a Stop | **Inverted** — point Destination at our own ACH server and let the unit call *us* |
|
||||
| Winning tool | `scripts/slow_drip.sh` | `bridges/ach_server.py --stop-monitoring` |
|
||||
|
||||
**Tell them apart with the ALEOS serial log** (see "Turn on ALEOS_SERIAL debug"
|
||||
below). If the device is emitting `ATQ1/ATE0/ATS0=2` every ~75 s, it is in the
|
||||
BE12599 mode and **no amount of inbound work will reach it** — skip to
|
||||
"Second incident" below.
|
||||
|
||||
---
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Terra-View / SFM `/device/info` either hangs or fails on `count_events()`.
|
||||
@@ -253,3 +274,206 @@ service).
|
||||
|
||||
Total time from "i was wondering if its possible to" first attempt to
|
||||
recovery: ~7 hours of intermittent debugging across one evening.
|
||||
|
||||
---
|
||||
|
||||
# Second incident — BE12599, 2026-09-16/17
|
||||
|
||||
**Unit:** BE12599 at `166.246.64.226:9034`, RV50, job *I-80 North Fork Bridge
|
||||
— Abut 1 West* (Fay Company). Same job as BE9558H, which is a coincidence.
|
||||
|
||||
**Fault:** the connector fault documented in `docs/offset_investigation.md`
|
||||
§8e progressed until the Tran pedestal reached **0.400 in/s** — its trigger
|
||||
level. Constant triggering → constant recording → ACH "after event recorded"
|
||||
→ continuous dialing. Same disease as BE9558H.
|
||||
|
||||
**But the recovery was the opposite direction**, and none of the Step 1–4
|
||||
procedure above worked. Total time ≈ 5 h, of which ~90 min was spent on two
|
||||
red herrings documented below.
|
||||
|
||||
---
|
||||
|
||||
## Turn on ALEOS_SERIAL debug FIRST
|
||||
|
||||
This is the single highest-value diagnostic and it should be step zero on any
|
||||
future incident. ACEmanager → **Admin → Log → ALEOS_SERIAL log level →
|
||||
DEBUG**, then view the serial log.
|
||||
|
||||
It is the only thing that tells you what the *device* is actually saying.
|
||||
Everything before we did this was guesswork.
|
||||
|
||||
## What the log showed — the device was never in S3 mode
|
||||
|
||||
Every ~75 seconds, verbatim:
|
||||
|
||||
```
|
||||
ALEOS_SERIAL_HIF: 29 byte(s) in buffer: 'ATQ1^MATE0^MATS0=2^M^MRADIO RING^M'
|
||||
ALEOS_SERIAL_HMC: TCP recvhost fd 65535 len 29 state TCPMode::kClosed
|
||||
ALEOS_SERIAL_HMC: tcpmode trying to send to invalid socket
|
||||
ALEOS_SERIAL_HMC: Connect to IP: 0.0.0.0 Port 0
|
||||
ALEOS_SERIAL_HMC: Initialize Auto answer on port 9034
|
||||
ALEOS_SERIAL_HMC: Cannot connect to 0.0.0.0
|
||||
```
|
||||
|
||||
Read that carefully:
|
||||
|
||||
- `ATQ1` (quiet) / `ATE0` (echo off) / `ATS0=2` (auto-answer after 2 rings).
|
||||
**There is no `ATD`.** The device is not dialing — it is trying to
|
||||
*configure* its modem.
|
||||
- The modem's serial port is in TCP data mode, so it never interprets these
|
||||
as AT commands. It treats them as payload and tries to ship them to a TCP
|
||||
socket that does not exist.
|
||||
- The device therefore never receives `OK`, never progresses, and **retries
|
||||
the identical 29 bytes forever**.
|
||||
|
||||
**Consequence: the device is not running the S3 protocol parser.** You can
|
||||
land a byte-perfect S3 frame on it and it will be ignored. This is why every
|
||||
inbound approach failed, and it is the structural difference from BE9558H.
|
||||
|
||||
### Why `slow_drip` lied
|
||||
|
||||
`slow_drip` returned the *success* signature except for the one field that
|
||||
mattered:
|
||||
|
||||
```json
|
||||
{"duration_s":120.0,"drips_sent":38,"bytes_sent":920,
|
||||
"bytes_received":0,"send_error":null}
|
||||
```
|
||||
|
||||
Full duration, no broken pipe — but zero bytes back. Cause is in the log
|
||||
above: each 75 s cycle re-runs `Initialize Auto answer on port 9034`, which
|
||||
orphans the held session (`data in for unknown reason 3 removing from
|
||||
select`, `OnMsg recv error: 107 - Transport endpoint is not connected`). Our
|
||||
local TCP stayed open so `sendall` never raised — but the modem stopped
|
||||
bridging after the first re-init, so every drip after that went into a socket
|
||||
nobody was reading.
|
||||
|
||||
⚠ **`send_error: null` + full duration is NOT success. Only
|
||||
`bytes_received > 0` is success.**
|
||||
|
||||
---
|
||||
|
||||
## ⚠ Two red herrings that cost ~90 minutes
|
||||
|
||||
### 1. The trusted-IP whitelist (this was the real reason inbound never worked)
|
||||
|
||||
The RV50s run with **Security → Trusted IPs (Friends List) enabled**. A
|
||||
source IP that is not on the list is dropped **silently** — inbound presents
|
||||
as `Connection error: timed out`, never a refusal.
|
||||
|
||||
Brian's dev-box public IP is **dynamic** and had changed, so `tmi-dev` was no
|
||||
longer whitelisted. Every inbound attempt failed identically across four
|
||||
different modem and device states, which looked exactly like the BE9558H
|
||||
mode-flipping symptom and sent us chasing modem configuration for over an
|
||||
hour.
|
||||
|
||||
**Check this before diagnosing anything else.** Note that SFM in Docker
|
||||
egresses via the *host's public IP*, not its LAN IP.
|
||||
|
||||
### 2. A 502 from SFM does not mean TCP connected
|
||||
|
||||
`sfm/server.py` raises **502 for both** failure classes:
|
||||
|
||||
```python
|
||||
raise HTTPException(status_code=502, detail=f"Protocol error: {exc}")
|
||||
raise HTTPException(status_code=502, detail=f"Connection error: {exc}")
|
||||
```
|
||||
|
||||
We read an early 502 as "TCP connected, modem bridged, device mute" and built
|
||||
a whole theory on it. It was almost certainly a connect timeout.
|
||||
**Always read the `detail` string** — "connect failed" and "device didn't
|
||||
answer" are completely different problems and the status code will not
|
||||
separate them.
|
||||
|
||||
---
|
||||
|
||||
## What actually worked — invert the direction
|
||||
|
||||
The key observation is in the log above:
|
||||
|
||||
> `TCP recvhost ... state TCPMode::kClosed` → `Connect to IP: 0.0.0.0 Port 0`
|
||||
|
||||
**The modem auto-dials its Destination whenever serial data arrives while
|
||||
closed.** So instead of fighting for inbound, give it somewhere to dial:
|
||||
point `Destination Address` at our own `ach_server` and the device's own
|
||||
75-second attempts become **device-initiated sessions the modem bridges
|
||||
correctly**. No race, no contention, worst case a 75-second wait.
|
||||
|
||||
### Procedure
|
||||
|
||||
1. **Run the rescue server** on a host the modem can reach (public IP +
|
||||
forwarded port):
|
||||
|
||||
```bash
|
||||
cd /home/serversdown/seismo-relay
|
||||
.venv/bin/python -u bridges/ach_server.py --port 12345 \
|
||||
-o bridges/captures/<unit>-diag --stop-monitoring -v
|
||||
```
|
||||
|
||||
2. **Point the modem at it** — ACEmanager → Serial → Port Configuration →
|
||||
`Destination Address` = your public IP, `Destination Port` = 12345.
|
||||
|
||||
3. **Wait for the call-in.** `--stop-monitoring` fires SUB 0x97 at step 1.5,
|
||||
after the handshake and *before* the event walk. Confirm via
|
||||
`rescue.json` in the session directory:
|
||||
|
||||
```json
|
||||
{"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
|
||||
```
|
||||
|
||||
4. **Restore the modem's Destination** once you are done, then finish the
|
||||
device side (disable ACH, erase) through whichever channel works.
|
||||
|
||||
On BE12599 the first call-in landed at 20:58:11 and reported
|
||||
`stop_monitoring: ok`; a second at 20:58:20 confirmed it. `is_monitoring:
|
||||
false` was still true **6½ hours later** — the fix is durable.
|
||||
|
||||
---
|
||||
|
||||
## Hard-won gotchas (do not re-derive)
|
||||
|
||||
- **Never leave the Destination pointed at a host with nothing listening.**
|
||||
That is the worst state available: the device still dials, the modem still
|
||||
flips, inbound stays blocked, and nothing is delivered. An 8-minute gap
|
||||
with the listener down produced a spurious inbound timeout that cost
|
||||
another round of misdiagnosis.
|
||||
|
||||
- **Stopping monitoring removes your call-in channel.** ACH is "after event
|
||||
recorded"; no new events means no new dials. The backlog sitting in memory
|
||||
does *not* re-arm it. After a successful stop the unit goes quiet and you
|
||||
need the modem cycled (works — produced a call-in), the scheduled daily call
|
||||
(BE12599 calls at **05:00:14 device-local**, per §8e), or working inbound.
|
||||
**Plan the order before you fire the stop.**
|
||||
|
||||
- **`--events-only` silently breaks dedup.** It skips the device-info step,
|
||||
so the serial is never read; `ach_state.json` then keys on
|
||||
`peer:ephemeral_port`, which is unique per connection. Every session looks
|
||||
like a new unit, starts from key 0, and re-downloads the same event. Four
|
||||
sessions on BE12599 downloaded the identical event four times and made zero
|
||||
progress on the backlog. Events also file as `serial=UNKNOWN` with a
|
||||
`M000…` BW filename (serial_numeric 0) instead of `N599…`.
|
||||
**Do not use `--events-only` when you intend to download anything.**
|
||||
|
||||
- **`/device/events/index` reported `lifetime_count: 0`** on a unit with years
|
||||
of history. Suspected decode bug in the SUB 0x08 field offset — do not
|
||||
trust that number. The 88-byte payload is preserved in the `raw_hex` field
|
||||
if someone wants to chase it.
|
||||
|
||||
- **Memory used cross-checks the event keys exactly:**
|
||||
`last_key − buffer_start = memory_total − memory_free`. On BE12599:
|
||||
`0x011230ec − 0x01110000 = 78,060` and `983,028 − 904,968 = 78,060`.
|
||||
Useful sanity check that you are reading the keys right.
|
||||
|
||||
---
|
||||
|
||||
## Final state (2026-09-17 ~01:30 local)
|
||||
|
||||
- `is_monitoring: false`, held 6½ hours
|
||||
- Battery 6.76 V
|
||||
- Memory 78,060 / 983,028 bytes used (8%)
|
||||
- `first_key 01121728`, `last_key 011230ec` — ~6.6 KB of addressable event
|
||||
chain, roughly 3 events
|
||||
- ACH still **enabled** — to be disabled after the backlog is preserved
|
||||
- Modem Destination still pointed at tmi-dev — to be restored
|
||||
- ⚠ **Do not re-enable ACH until the connector is serviced.** Tran is still
|
||||
sitting at 0.400 and the loop restarts the moment monitoring resumes.
|
||||
|
||||
Reference in New Issue
Block a user