docs(series4): REPRODUCED -- THOR stops polling after a drop mid-download
The field failure on UM12947 ("wouldn't stay connected, refresh did nothing, no
way to view a connection attempt") reproduced on the bench, with timestamps.
THOR was mid-bulk-download when the link was faulted. Sequence:
13:02:50 connection dies mid-transfer (165 BULK_DOWNLOAD frames in)
13:03:10 THOR reconnects once after 20 s, sends SUB_1F to resume, fails
13:03:33 link fully restored and healthy
13:04:21 operator clicks Refresh -> full 11-command check, all correct
13:06:27 still nothing. That refresh is the ONLY connection since 13:03:10.
Before the fault THOR had connected every 30 s without a miss for over an hour.
Establishes four things:
* one retry then give up -- no backoff, no further attempts
* the automatic poll loop dies too, not just the download
* it does not recover when the link returns (3 min of healthy link, nothing)
* Refresh works but only once -- it does NOT restart the automatic loop
The fourth is the dangerous one: Refresh makes the UI report a healthy unit while
nothing is watching it. Silent failure that looks like success. It also explains
why the field symptom resists characterisation -- the unit is reachable the whole
time; THOR has simply stopped asking and says nothing about it.
CAVEAT, recorded prominently: what THOR experienced was a TCP close mid-download,
not the silent link intended. mm_link.py mistook socket.timeout (which subclasses
OSError) for a closed socket, so 200 ms of quiet closed the connection -- the
relay killed the link it was meant to be faking a fault on. Fixed in this commit.
The run stands as a drop-mid-download test, arguably the more realistic case.
Single trial; true blackhole and clean drop not yet tested.
Adds four design consequences for SFM: unbounded retry with backoff; a manual
check must restart the automatic loop or the UI must say it is stopped; surface
the poll loop's own state (last success, last attempt, next attempt, consecutive
failures -- all four invisible here); and distinguish "unit unreachable" from "we
stopped checking", which present identically in THOR.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
+9
-1
@@ -236,11 +236,19 @@ class Link:
|
||||
arrow = "THOR->unit" if is_request else "unit->THOR"
|
||||
last = time.time()
|
||||
while not stop.is_set():
|
||||
timed_out = False
|
||||
try:
|
||||
data = src.recv(4096) if isinstance(src, socket.socket) else src.read(4096)
|
||||
except TimeoutError:
|
||||
timed_out = True
|
||||
# socket.timeout subclasses OSError, so it MUST be caught first.
|
||||
# Treating it as a dead socket closes the connection after 200 ms
|
||||
# of quiet -- which is exactly what `blackhole` produces, so the
|
||||
# relay killed the link it was supposed to be faking a fault on.
|
||||
data = b""
|
||||
except OSError:
|
||||
break
|
||||
if isinstance(src, socket.socket) and data == b"":
|
||||
if isinstance(src, socket.socket) and data == b"" and not timed_out:
|
||||
self.say(f"{arrow}: peer closed the connection")
|
||||
break
|
||||
if not data:
|
||||
|
||||
Reference in New Issue
Block a user