THOR reports every failed connection as "disconnected" and nothing more. That
one word covers at least four distinct faults with four different fixes, and
telling them apart is the difference between a modem reboot and a site visit.
Nobody had a way to do that during the 2026-09-22 outage, which is the actual gap
that incident exposed -- not a missing THOR feature, but a missing tool.
connection refused something answered and said no -- wrong port, or the
modem refusing a further session
connect timed out nothing answered -- trusted-IP whitelist, firewall, or
the modem is off the network. A whitelist DISCARDS
rather than refuses, so this is its signature
connected, no reply the MODEM answered but the unit did not. TCP is fine;
the modem is not forwarding to serial. This is what a
wedged transparent-TCP session looks like, and it is
the case THOR cannot distinguish from the others
replied the unit is alive; the fault is upstream software
Each verdict prints what to try next. The no-reply case points at ACEmanager's
TCP Idle Timeout first, since a stale session holds a single-slot modem's only
connection until that timeout frees it.
--slots N opens N simultaneous connections and reports how many the far end
accepts, which directly tests the single-session hypothesis against a real modem.
Read-only throughout: POLL, SERIAL and the 0x49 state read -- the same three
commands THOR's own connection check uses. Sends the correct per-SUB data
offsets (POLL 0x0030, SERIAL 0x000A, 0x49 0xFFFF); offset 0 returns only the
short probe reply.
Works for both series and says which answered: a Series III reply opens DLE STX,
a Micromate reply opens with a bare STX.
Verified against UM12947 through the bench relay, and against a closed port for
the refused path.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
The field failure on UM12947 ("wouldn't stay connected, refresh did nothing, no
way to view a connection attempt") reproduced on the bench, with timestamps.
THOR was mid-bulk-download when the link was faulted. Sequence:
13:02:50 connection dies mid-transfer (165 BULK_DOWNLOAD frames in)
13:03:10 THOR reconnects once after 20 s, sends SUB_1F to resume, fails
13:03:33 link fully restored and healthy
13:04:21 operator clicks Refresh -> full 11-command check, all correct
13:06:27 still nothing. That refresh is the ONLY connection since 13:03:10.
Before the fault THOR had connected every 30 s without a miss for over an hour.
Establishes four things:
* one retry then give up -- no backoff, no further attempts
* the automatic poll loop dies too, not just the download
* it does not recover when the link returns (3 min of healthy link, nothing)
* Refresh works but only once -- it does NOT restart the automatic loop
The fourth is the dangerous one: Refresh makes the UI report a healthy unit while
nothing is watching it. Silent failure that looks like success. It also explains
why the field symptom resists characterisation -- the unit is reachable the whole
time; THOR has simply stopped asking and says nothing about it.
CAVEAT, recorded prominently: what THOR experienced was a TCP close mid-download,
not the silent link intended. mm_link.py mistook socket.timeout (which subclasses
OSError) for a closed socket, so 200 ms of quiet closed the connection -- the
relay killed the link it was meant to be faking a fault on. Fixed in this commit.
The run stands as a drop-mid-download test, arguably the more realistic case.
Single trial; true blackhole and clean drop not yet tested.
Adds four design consequences for SFM: unbounded retry with backoff; a manual
check must restart the automatic loop or the UI must say it is stopped; surface
the poll loop's own state (last success, last attempt, next attempt, consecutive
failures -- all four invisible here); and distinguish "unit unreachable" from "we
stopped checking", which present identically in THOR.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
First capture through bridges/mm_link.py, with THOR polling a unit over the bench
link at "check connection every 5 s / check status every 5 s".
A status check is ELEVEN commands, not one:
POLL -> DEVICE_INFO -> 0x49 -> 0x5C -> MONITOR_STATUS -> SETUP_NAME_READ
-> STORAGE_RANGE -> 0x02 -> OPERATOR -> 0x47 -> CALLHOME_CFG
Each check opens a NEW TCP connection, runs all eleven exchanges in ~300 ms and
closes it. Measured over 21 consecutive checks: 236 B out, 936 B back, plus a
full handshake each time -- about 2.2 KB per check.
At the observed cadence that is 18.8 MB/day, 563 MB/month, per unit. On a
metered cellular plan that is real money, and most of it is waste: the check
re-reads the call-home config, operator name, active setup name and full device
info every ten seconds, none of which changes. SETUP_NAME_READ alone returns 274
bytes a time. POLL + MONITOR_STATUS answers "alive?" and "monitoring?" in two
commands and 131 bytes.
Both intervals set to 5 s yields one combined pass every 10.1 s, steady across
eight measured connections. So the two settings are not independent 5-second
timers, which is a plausible reason changing them appears to do nothing.
REVISES an earlier hypothesis. Because idle polling reconnects every cycle, a
silently-dead link is LESS dangerous while idle than I assumed -- a dead socket
fails at connect and the next cycle retries. The exposure is during OPERATIONS:
THOR held one connection from 00:30 to 00:47 last night while downloading events
and pushing setups. A link dying mid-operation leaves it waiting on a socket the
OS will not fail for ~2 h. The blackhole test should therefore be run during a
download, not while idle.
Also fixes a mislabel in mm_link.py: the SUB byte is DLE-escaped when its value is
0x02/0x03/0x04/0x10, so reading it raw reported SUB 0x02 as "SUB_10".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
THOR gives almost no visibility into a connection: a refresh button, two poll
intervals, and no way to see whether a check succeeded, timed out, or was never
attempted. When a unit "won't stay connected" there is nothing to look at. This
sits where the cellular modem would and answers that directly.
Over socat -x it adds the two things that were missing:
* A READABLE LOG. Frames are decoded and timestamped as they pass --
"THOR->unit POLL (21 B)" rather than hex -- so THOR's polling cadence, and
its silences, are visible. Raw .bin pairs are still written alongside and
load straight into scratch/mm_frame_parse.py.
* FAULT INJECTION, via a control file read on the fly:
pass normal relay
blackhole TCP stays up, bytes are swallowed
drop close the connection abruptly
delay:N forward N seconds late, both directions
onewaydev THOR->unit passes, unit->THOR is swallowed
`blackhole` is the point of the exercise. It reproduces the classic cellular
failure -- socket open at both ends, nothing crossing -- which a real cell link
will not do on cue. THOR was observed last night holding one TCP connection for
17 minutes (00:30 to 00:47), so if the link dies silently the OS will not tell it
for roughly the default keepalive, ~2 hours. That is a candidate explanation for
"refresh does nothing and only a restart helps", and this makes it testable
rather than speculative.
No pyserial: the port is driven through stdlib termios. The bench hosts are
whatever is to hand and requiring a pip install on someone else's machine is a
poor trade for ~30 lines. Deployed and verified on mint-mac (Python 3.12, no
third-party modules) against UM12947 -- a POLL round-trips and decodes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
A unit whose geophone offset has grown past its trigger level records
back-to-back and, with ACH set to "after event recorded", re-dials every
time. The wedged_unit_recovery runbook handles that by reaching the unit
inbound and clearing the modem's Destination Address so it stops dialing.
That fails when the device is wedged mid-modem-init. BE12599 (2026-09-16)
sat repeating a 29-byte AT setup string — ATQ1/ATE0/ATS0=2, no ATD — every
75 s. The modem is in TCP data mode, never interprets it, never answers OK,
so the device never progresses into S3 mode and ignores every frame we send.
Worse, each attempt makes ALEOS log "tcpmode trying to send to invalid
socket" and re-run "Initialize Auto answer on port 9034", which orphans any
held inbound session — slow_drip reports a clean 120 s hold with
bytes_received=0 because the modem stopped bridging after the first re-init.
Inbound cannot win that race. But the modem auto-dials its Destination
whenever serial data arrives while closed, so pointing Destination at an
ach_server turns those 75 s attempts into a device-initiated session that
the modem bridges correctly.
Adds --stop-monitoring, --disable-ach and --rescue. They run as step 1.5,
after the handshake and before the event walk, each independently guarded so
a failure does not abort the download. Outcome is written to rescue.json.
Startup banner reports both, and warns when --restart-monitoring would undo
--stop-monitoring.
Prefer --stop-monitoring alone on first contact: --disable-ach stops the unit
calling, which is the only channel to a unit in this state, and halting the
recording ends the loop on its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN
- Added `waveform_key` and `event_timestamp` columns to `CachedEvent` and `CachedWaveform` for integrity verification.
- Implemented logic to flush the cache when a mismatch in (waveform_key, event_timestamp) is detected during event and waveform updates.
- Enhanced `set_events` and `set_waveform` methods to check for mismatches and trigger cache eviction as necessary.
- Introduced a new `LiveCache` class to manage in-memory caching of live device data, separating it from the server logic for better testability.
- Added tests to verify the correctness of cache invalidation logic, particularly for post-erase key reuse scenarios.
- Updated web application to include a "Force refresh" toggle, allowing users to bypass the cache and re-fetch data from the device.
- Update `s3_bridge.py` to default raw capture file paths to "auto" for timestamped naming.
- Modify `gui_bridge.py` to pre-check raw capture options and streamline path handling.
- Extend `ach_server.py` to save both incoming and outgoing raw bytes for analysis.
- Revise `CHANGELOG.md` and `instantel_protocol_reference.md` to reflect changes in recording mode handling and compliance data encoding.
sfm/database.py (new)
- SeismoDb class: three tables keyed by unit serial number
- ach_sessions: one row per ACH call-home
- events: one row per triggered event, deduped by (serial, waveform_key)
- monitor_log: one row per monitoring interval, deduped by (serial, waveform_key)
- WAL mode, per-request connections, silent dedup via UNIQUE constraint
- Query helpers: query_events(), query_monitor_log(), get_sessions(), query_units()
- false_trigger flag on events for future review UI / report filtering
bridges/ach_server.py
- Import SeismoDb; create shared instance at startup pointed at
bridges/captures/seismo_relay.db
- After each call-home: insert_events() + insert_monitor_log() + insert_ach_session()
- DB failures logged as warnings, never abort the session
sfm/server.py
- Import SeismoDb; lazy singleton via _get_db()
- New DB read endpoints: GET /db/units, /db/events, /db/monitor_log, /db/sessions
- PATCH /db/events/{id}/false_trigger for manual review flagging
CLAUDE.md / CHANGELOG.md
- Document DB schema, SFM DB endpoints, architecture decision (unit-keyed only)
- Version bump to v0.11.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add full decode pipeline for 0x2C partial records from the device's event
list, representing continuous monitoring intervals where no threshold was
crossed. These records appear interleaved with full triggered events in the
browse walk and were previously ignored.
minimateplus/models.py
- Add MonitorLogEntry dataclass: key, start_time, stop_time, serial,
geo_threshold_ips, raw_header, duration_seconds property
minimateplus/protocol.py
- read_waveform_header() now returns (data_rsp.data, length) — full payload
including the record-type byte at position 0 — instead of the sliced header.
Callers that need the old slice use raw_data[11:11+length] as before.
minimateplus/client.py
- Add _decode_0a_partial_header(): auto-detects 9-byte (sub_code=0x10) vs
10-byte (sub_code=0x03) timestamp format, handles 1-byte inter-timestamp
gap, extracts serial via BE anchor and geo threshold via Geo: anchor.
- Add get_monitor_log_entries(skip_keys=None): browse walk (1E → 0A → 1F),
decodes partial records, skips full records and already-seen keys.
minimateplus/__init__.py
- Export MonitorLogEntry
bridges/ach_server.py
- After get_events(), call get_monitor_log_entries(skip_keys=seen_keys) and
save new entries to monitor_log.json in the session directory.
- Add _monitor_log_entry_to_dict() helper.
- Include monitor log keys in downloaded_keys for state persistence.
CLAUDE.md / CHANGELOG.md
- Document 0x2C partial record layout (timestamp format, ASCII metadata
region, 1-byte gap edge case) confirmed from 4-11-26 MITM capture.
- Version bump to v0.10.0; update What's next.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Listens for inbound unit connections, connects upstream to a real Blastware
ACH server, and forwards bytes bidirectionally while saving both directions to
raw_bw_<ts>.bin and raw_s3_<ts>.bin in the existing capture format.
Used to capture the 4-11-26 Blastware ACH session that confirmed the erase-all
protocol (SUBs 0xA3/0x1C/0x06/0xA2) and the event deletion wire sequence.
Usage:
python bridges/ach_mitm.py --bw-host 127.0.0.1 --bw-port 9999 --listen-port 9998
Point the unit's call-home destination at this machine:9998.
Point this proxy's --bw-host/port at the upstream Blastware ACH server.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
State format (ach_state.json):
- Replace event_count with downloaded_keys (set of hex strings) + max_downloaded_key
- Key-based tracking correctly handles delete-then-re-record: after device erase the
count drops to 0, but new events have new (or recycled) keys
Browse pre-check:
- list_event_keys() walk before get_events() to bail early when nothing is new
- get_events() called with skip_waveform_for_keys= for already-seen keys, so repeat
call-homes only download waveforms for genuinely new events
--clear-after-download flag:
- After saving new events, calls client.delete_all_events() (0xA3→0x1C→0x06→0xA2)
- On success: resets downloaded_keys=[] and max_downloaded_key="00000000" so the
next session starts fresh (device counter resets to 0x01110000 after erase)
Post-erase key-reuse detection:
- Device counter resets to 0x01110000 after any erase; new events reuse old keys
- If max(device_keys) < max_downloaded_key, the device was wiped externally
(Blastware, manual) — seen_keys is discarded and all device keys treated as new
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Event model uses peak_values (not peaks) and project_info (not direct fields).
PeakValues fields are tran/vert/long/micl/peak_vector_sum (not transverse etc).
ProjectInfo fields accessed via ev.project_info.project etc.
Also fix ev.timestamp serialization: use str() instead of .isoformat() since
Timestamp is a custom dataclass, not datetime.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
calibration_date, aux_trigger, setup_name etc. don't exist directly on
DeviceInfo — they live in DeviceInfo.compliance_config (ComplianceConfig).
_device_info_to_dict now accesses them via cc = d.compliance_config.
Log line updated to show serial/firmware/model/event_count instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
socket.accept() on Windows blocks indefinitely and ignores KeyboardInterrupt.
Setting a 1-second timeout on the server socket causes the accept loop to wake
up every second and re-check, so Ctrl-C is handled within ~1 second.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Session directory and log file are now created ONLY after startup() succeeds.
Internet scanners and dropped connections no longer litter the output folder.
Raw bytes are buffered in memory until startup succeeds, then flushed to disk.
- Add --allow-ip IP flag (repeatable) to allowlist specific source IPs.
Connections from un-listed IPs are rejected immediately (socket closed, no log).
If no --allow-ip flags are given, all IPs are still accepted (original behavior).
Usage: --allow-ip 63.43.212.232 --allow-ip 152.1.2.3
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
_decode_event_count: read uint16 BE at offset 10 (confirmed 2026-04-10 from
live BE11529 event index — data[10:12]=0x0006=6, matches device LCD).
Previous uint32 at offset 3 always returned 1 regardless of event count.
ach_server.py: use device_info.event_count (already fetched during connect())
instead of calling count_events() separately. This saves 2*N round-trips and
avoids the 1F linked-list walk which was overcounting on some devices.
count_events() kept as fallback when connect() is skipped (--events-only).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace all Unicode arrows/checkmarks (-> [OK] [FAIL]) in ach_server.py
and client.py log calls — Windows cp1252 console can't encode them
- Fix DeviceInfo attribute: serial_number -> serial
- Fix _device_info_to_dict key: serial_number -> serial
- Demote count_events 1E/1F per-key log lines from WARNING to DEBUG
(they were flooding the console on devices with many stored events)
- FileHandler now opens with encoding='utf-8' so session log files
can hold any characters without codec errors
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ach_server.py:
- Add ach_state.json per-unit state tracking (keyed by serial number)
- count_events() before any download; skip session if no new events since last call-home
- Download only events beyond the previous high-water mark (all_events[last_count:])
- --max-events N safety cap for first-run units with many stored events
- state_path and max_events wired through AchSession constructor and serve()
client.py (_decode_monitor_status):
- Revert monitoring flag to section[1] == 0x10 (was incorrectly changed to section[6])
- Fix battery/memory offsets to section[-10:-8], [-8:-4], [-4:] (no trailing checksum byte)
- Both confirmed by full byte diff of all 144 0xE3 data frames in 4-8-26/2ndtry capture
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
minimateplus/transport.py:
- Add SocketTransport(TcpTransport) — wraps an already-accepted inbound
socket; connect() is a no-op; everything else inherited from TcpTransport.
Enables the ACH server to reuse all existing protocol/client code without
any changes.
bridges/ach_server.py:
- Minimal inbound ACH server — listens on port 12345, accepts call-home
connections from MiniMate Plus units, runs the full BW protocol:
startup handshake → get_device_info → get_events(full_waveform=True)
- Saves device_info.json + events.json + raw_rx_<ts>.bin + session log
per connection to bridges/captures/ach_inbound_<ts>/
- raw_rx.bin is byte-compatible with existing Analyzer tooling
- Taps transport.read() to capture raw S3 bytes alongside parsed output
- Each connection runs in its own daemon thread
- Clearly distinguishes push vs pull protocol in the startup log
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a production-safe headphone-splitter mode:
- Device bytes tee'd to both --upstream (primary/prod) and --mirror (new server)
- Only primary server responses are returned to the device
- Mirror connect/write failures are non-fatal and logged; prod is unaffected
- New raw_mirror_<ts>.bin capture file alongside raw_client/raw_server
Three modes: standalone (capture only), bridge (one upstream), splitter (two).
Default listen port changed to 12345 to match project ACH setup.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- bridges/ach_bridge.py: transparent TCP bridge that MITMs the MiniMate Plus
call-home connection — forwards to real ACH server while logging all frames
to raw_client/raw_server .bin files compatible with parse_capture.py;
standalone capture mode for lab use without a real server
- bridges/serial_watch.py: RS-232 serial monitor with live S3 frame parsing;
taps the line between MiniMate and modem (RV50/RV55); captures raw bytes,
.log and .jsonl; --ack-ok mode auto-replies to AT commands; fixed fatal
indentation bug in the original that silently prevented any data capture
- seismo_lab.py: new "Serial Watch" fourth tab (SerialWatchPanel) wrapping
serial_watch.py functionality; COM port picker with refresh, baud config,
ack-ok toggle, colour-coded live frame log (teal frames / yellow ctrl /
blue AT), raw .bin capture auto-fed into Analyzer tab on stop
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- bridges/tcp_serial_bridge.py: increase default boot_delay 2s → 8s to
cover MiniMate Plus cold-start time (unit wakes from RS-232 line
assertion but takes 5-10s to be ready for POLL_PROBE).
- sfm/server.py: add _run_with_retry() — on TCP connections only, retries
once on ProtocolError. Serial timeouts are not retried (usually a real
fault). Confirmed behaviour: unit wakes purely from RS-232 line voltage,
no software wake-up frame needed.
serversdownandClaude Opus 5
claude