Feat/ach rescue on connect #38

2026-09-18T15:25:57-04:00

serversdown commented

2026-09-18 15:25:57 -04:00

We redid the ACH rescue method. This is a tool meant to stop a machine that for what ever reason is perpetually triggering and as a result is calling the main ACH server repeadedly. Our old attempt was to try to sneak a command in in-between call-in attempts, but I realized its easier to instead just answer the call-ins with our own server.

So the process is as such:

Identify the unit in question, get its paired modems IP address, and connected to the modem via ACEmanager (or whatever).
Set up an ACH rescue server at a different IP and port combination than the main ACH. Open the appropriate port (in this case 12345 forwards to tmi-dev:12345) (in this case i hosted it from my home network on the tmi-dev VM, but it can be done anywhere that isn't the same address and port as the main ACH server. (so we cant do :12345)).
Run the ach_server.py program.
Go to Serial > Destination address. Point the destination address to the rescue server. (NOTE: if you point it at the rescue server and the rescue server is not up, it can do some wonky stuff, so its best to just make sure the rescue server is listening before this step. After that reboot the rv50 modem.
The unit should then dial into the rescue server and the program will stop the unit from monitoring.

See the notes in the programs for more info.

We redid the ACH rescue method. This is a tool meant to stop a machine that for what ever reason is perpetually triggering and as a result is calling the main ACH server repeadedly. Our old attempt was to try to sneak a command in in-between call-in attempts, but I realized its easier to instead just answer the call-ins with our own server. So the process is as such: 1. Identify the unit in question, get its paired modems IP address, and connected to the modem via ACEmanager (or whatever). 2. Set up an ACH rescue server at a different IP and port combination than the main ACH. Open the appropriate port (in this case 12345 forwards to tmi-dev:12345) (in this case i hosted it from my home network on the tmi-dev VM, but it can be done anywhere that isn't the same address and port as the main ACH server. (so we cant do <officeip>:12345)). 3. Run the ach_server.py program. 4. Go to Serial > Destination address. Point the destination address to the rescue server. (NOTE: if you point it at the rescue server and the rescue server is not up, it can do some wonky stuff, so its best to just make sure the rescue server is listening before this step. After that reboot the rv50 modem. 5. The unit should then dial into the rescue server and the program will stop the unit from monitoring. See the notes in the programs for more info.

serversdown added 6 commits 2026-09-18 15:25:58 -04:00

feat(ach): rescue-on-connect — stop monitoring / disable ACH from the server side 9f1050b5e7

A unit whose geophone offset has grown past its trigger level records
back-to-back and, with ACH set to "after event recorded", re-dials every
time.  The wedged_unit_recovery runbook handles that by reaching the unit
inbound and clearing the modem's Destination Address so it stops dialing.

That fails when the device is wedged mid-modem-init.  BE12599 (2026-09-16)
sat repeating a 29-byte AT setup string — ATQ1/ATE0/ATS0=2, no ATD — every
75 s.  The modem is in TCP data mode, never interprets it, never answers OK,
so the device never progresses into S3 mode and ignores every frame we send.
Worse, each attempt makes ALEOS log "tcpmode trying to send to invalid
socket" and re-run "Initialize Auto answer on port 9034", which orphans any
held inbound session — slow_drip reports a clean 120 s hold with
bytes_received=0 because the modem stopped bridging after the first re-init.

Inbound cannot win that race.  But the modem auto-dials its Destination
whenever serial data arrives while closed, so pointing Destination at an
ach_server turns those 75 s attempts into a device-initiated session that
the modem bridges correctly.

Adds --stop-monitoring, --disable-ach and --rescue.  They run as step 1.5,
after the handshake and before the event walk, each independently guarded so
a failure does not abort the download.  Outcome is written to rescue.json.
Startup banner reports both, and warns when --restart-monitoring would undo
--stop-monitoring.

Prefer --stop-monitoring alone on first contact: --disable-ach stops the unit
calling, which is the only channel to a unit in this state, and halting the
recording ends the loop on its own.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN

docs: BE12599 incident — the inverted rescue, plus a rescue-listener plan c6fc3d0241

The wedged_unit_recovery runbook covered exactly one failure mode.  BE12599
turned out to be a second one wearing the same symptoms, and the existing
procedure did not work on it.

Adds a "TWO failure modes" table up front so the next incident branches
correctly, and a full second-incident section covering what the ALEOS serial
debug log revealed: the device repeating a 29-byte AT modem-init string
(ATQ1/ATE0/ATS0=2, no ATD) every 75 s, never getting an OK because the modem
is in TCP data mode, and therefore never entering S3 mode at all.  Inbound
cannot win against that, no matter how well framed.

Also records the two red herrings, since together they cost ~90 minutes:
the RV50 trusted-IP whitelist drops non-listed sources silently (presents as
a connect timeout, and Brian's dynamic dev IP had rotated off the list), and
sfm/server.py returns 502 for BOTH "Protocol error:" and "Connection error:",
so a 502 was misread as "TCP connected, device mute" and a theory built on it.

And the gotchas worth never re-deriving: slow_drip's send_error=null plus a
full duration is not success (only bytes_received > 0 is); stopping monitoring
removes the call-in trigger, so it costs you the channel; --events-only skips
the device-info step, so the serial is never read and ach_state keys on
peer:ephemeral_port, silently breaking dedup and re-downloading the same event
every session.

The plan doc captures the tool Brian wants built out of this — a rescue
listener with a real lifecycle and, critically, a confirmation gate before
shutdown, because leaving the modem's Destination pointed at a dead listener
is worse than never having started.  Open questions are listed rather than
guessed at.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN

docs(runbook): reframe as one disease with two cures, intercept first 402bf30e37

The previous commit called BE12599 a second failure mode and claimed the
device "never enters S3 mode at all" and that no inbound work could reach it.
That was an overclaim built on a single slow_drip attempt, and Brian was right
to push back.

It is the same disease.  Method B's step 1 worked fine on BE12599 — clearing
the Destination did stop the dial-outs.  It was step 2 that did not land, on
one attempt, run ~90 s after a modem reboot with a dead session visible in the
log in that same window; BE9558H needed hours of attempts before one landed.
And the AT-init loop the ALEOS log revealed is almost certainly what BE9558H
was doing too — we just never turned on serial debug in May to look.  The
device speaks S3 fine; it handshook cleanly the moment it had a session.

What is genuinely new is the cure, and it deserves to be the default rather
than a footnote.  Racing a Stop into the gaps between dial-outs is a coin
flip.  Intercepting is deterministic: the unit dials every ~75 s, so give it
somewhere to dial and answer it.  It will not answer us because it is on the
phone — so be the one it calls.

Restructures accordingly: a "two cures" table up top, the intercept promoted
to Method A with its own procedure (listener before modem, stop at step 1.5,
drain before disabling ACH, restore the Destination and confirm it), and the
original inbound procedure kept intact as Method B for when there is no
listener the modem can reach.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN

docs: adopt a changelog convention, and make Unreleased follow it 2fabf84d4d

Brian asked what the standard is; there wasn't a written one, only a de facto
pattern in the history. This writes it down in CLAUDE.md and fixes the one
place the repo already diverged from it.

The rule: write the entry in the same commit as the work, under ## Unreleased;
cut the version on dev in a dedicated chore(release) commit; never touch the
changelog at a merge boundary. The entry goes in with the change because that
is the only moment you still know why.

Two additions beyond what the history already did:

No preamble under ## Unreleased. The themed opening paragraph gets written at
release time, when the whole release is visible and can be named honestly. The
current one proved the point — "Blastware Event/FFT-Report parity: the FFT,
the USBM compliance chart, and the sensor self-check" was accurate when the
first item landed and stopped being accurate once rescue-on-connect landed
under the same heading. Removed here; the release commit writes a new one
covering everything actually in the release.

And the operational consequence is now mandatory on any entry touching the
codec, the waveform store, or the DB — including when it is "none". This
repo's changelog is how future-you learns whether a deploy costs two hours on
the NAS, so silence is ambiguous and "none" is information. The old preamble's
load-bearing half is preserved as an explicit ### Migration block rather than
dropped with the prose around it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN

docs: make the release cadence explicit a42e8d3651

Brian described the practice: Unreleased is the staging area for what is going
into the next release, and the version bump happens when enough has
accumulated to be worth shipping — not per commit, not per merge. The
convention already implied it ("never touch the changelog at a merge
boundary") but never said it outright.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN

docs: write the changelog on dev, not on feature branches 0408c37866

Reverses the "entry goes in with the work" rule from two commits ago. That was
wrong on the evidence: of the docs(changelog) commits in history, 3 of 4 in
seismo-relay and 2 of 4 in Terra-View were made directly on dev. The rule was
generalized from one unrepresentative commit rather than from the pattern.

It also caused the exact problem it was supposed to avoid. With four worktrees
in flight, every branch edits the same few lines at the top of CHANGELOG.md;
feat/ach-rescue-on-connect and feat/sensor-check-h5 collide on that file and
nothing else. Writing the entry once, on dev, after the merge removes the
whole conflict class.

The second benefit is accuracy: an entry written after the merge describes
what actually landed, including anything that changed during conflict
resolution. The sensor-check branch is a live example — its Unreleased
preamble describes a release that no longer looks like that.

The failure mode of writing it later is forgetting, so the merge is explicitly
not finished until Unreleased is updated — same sitting, reconstructed from the
branch commit messages.

Unchanged: no preamble under Unreleased, the mandatory operational consequence,
and cutting the version on dev when ready to ship to main.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN

serversdown merged commit 27e9c56393 into dev

2026-09-18 15:26:19 -04:00

serversdown deleted branch feat/ach-rescue-on-connect

2026-09-18 15:26:28 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: serversdown/seismo-relay#38