Author SHA1 Message Date
serversdownandClaude Opus 5 0a96a34b02 docs(offset): CONFIRMED — BE12599 is a geophone-assembly fault, not the recorder
The first mechanism this investigation has established physically rather than
inferred. Brian pulled BE12599 and bench-tested it; a geophone swap settles it.

Same recorder, twenty minutes apart: Vert overswing 2026.0 with its own
geophone, 3.4 with BE9888's known-good one. Frequency 2.2 -> 7.4 Hz. All three
channels Passed, no offsets, no self-triggering, and a clean damped impulse with
broadband content. The recorder is functional and merely out of calibration.

A bench control -- trigger forced by slapping the microphone, geophone untouched
-- shows Vert producing 0.19 in/s of bipolar wander with no mechanical input at
all. The fault is continuous, not episodic. All three channels are affected:
Tran and Long sit on a small DC while their coils still pass the swing test,
Vert has lost damping entirely. That is leakage on several contacts plus one
gone open, not three elements failing in sequence.

The consequence that reaches past this unit: an autozero adjusts the RECORDER's
zero reference and cannot fix a geophone. If most offsets are geophone-assembly
faults, a re-zero succeeds only on the minority where the recorder really is at
fault -- which is exactly the ~10% rate recorded in section 5 and unexplained
since August. Procedure change: swap the geophone first.

Corrects 8e's title, which called it a connector rather than a geophone: the
swap narrows the fault to the assembly, and element / cable / connector remain
open within it. And reweights the cause -- the geophone is buried, not in the
enclosure with the nest, and the unit sat ~6 ft from I-80 WB, so road salt now
outranks the mice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
2026-09-23 02:57:34 +00:00
9 changed files with 121 additions and 4020 deletions
+3 -52
View File
@@ -10,38 +10,10 @@ pair — lives in `../terra-view/docs/tmi-stack.md`, which is also loaded as
---
## Where things stand (updated 2026-09-26)
## Where things stand (updated 2026-08-28)
Read this first when picking the project back up.
- **The Series-4 LIVE wire protocol is reverse-engineered end to end
(2026-09-25).** `docs/micromate_protocol_reference.md` is the Series-4
Rosetta Stone, sibling to `instantel_protocol_reference.md`. **A Micromate
answers Series III command frames** — three framing differences: responses
have **no leading `DLE`** (a bare `STX`), `payload[1]` is `0xC5` (Blastware
firmware) or `0x03` (Thor firmware) rather than `0x10`, and the data length
is a **uint16 BE at `payload[8:10]`** (as a byte it under-reads `SUB 0x1A`
by 47x). Read path, event chain, setups, scheduler, monitoring control and
per-event delete are all mapped; **the inbound call-home session is the only
protocol unknown left.**
⚠ **No command has ever been originated against a unit by this project.**
Every write was performed by THOR while we recorded. That line is worth
keeping.
⚠ `micromate/` still has **no live client** — it is codec-only. The
`minimateplus/` stack (transport/framing/protocol/client) has no Series-4
counterpart yet. `minimateplus.transport` is protocol-agnostic and reusable.
- **Bench tooling for device diagnosis (2026-09-25).** `bridges/mm_probe.py`
distinguishes the four faults THOR reports identically as "disconnected"
(refused / connect timeout / **connected but no reply** / replied) and names
what to try next. `bridges/mm_link.py` is a stand-in for a cellular modem
with a decoded log and fault injection. `scratch/mm_frame_parse.py` exists
because **`S3FrameParser` cannot see Micromate responses at all** — it scans
for `DLE+STX`, which never appears in Series-4 traffic.
- **A Micromate's USB-A host port drives FTDI and CDC-ACM only** — no Prolific,
in either firmware line. TMI buys both Sabrent (FTDI) and Benfei (PL2303)
cables and they are indistinguishable by eye. A PL2303 cable leaves a unit
with **no working modem port at all**; identify by `lsusb` VID, `0403` vs
`067b`. This accounted for a unit that could not be deployed.
- **Series-3 decode is verified per-sample at scale (v0.27.0).** The full DL2
archive decodes **14,338 / 14,338** paired files exactly against their
preserved Blastware ASCII exports — 1,249 waveform + 13,089 histogram, 45
@@ -120,9 +92,6 @@ Read this first when picking the project back up.
**v0.27.0 does NOT owe prod a backfill** — verified: the partial-final-block
fix changes 0 of the 10,215 histograms in the prod store (the 4 recovered
files are archive-only and were never ingested).
✅ **The v0.30.0 Series-4 backfill HAS been run on prod (2026-09-25).** Every
stored Series-4 geophone value was ~3.3% low until then; that is corrected and
the job does not need repeating.
- **The "offset" hardware fault has its own journal** --
`docs/offset_investigation.md`. **5 of 45 units (11%)**, and the fault is
**persistent** — it stays until the geophone is serviced. Detect it with
@@ -134,18 +103,7 @@ Read this first when picking the project back up.
`SUB 0x0E` (unimplemented), which may carry those very numbers.
When new information about a protocol is discovered, record it in the matching
reference **in addition to** this document:
| series | document |
|---|---|
| Series III (MiniMate Plus / BlastMate) | `docs/instantel_protocol_reference.md` |
| **Series IV (Micromate / THOR)** | **`docs/micromate_protocol_reference.md`** |
| Thor IDF file format | `docs/idf_protocol_reference.md` |
Both protocol references carry retractions in place rather than deleting what
turned out to be wrong — that convention has already saved re-deriving the same
mistakes twice, so keep it.
When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
---
@@ -318,15 +276,8 @@ minimateplus/ ← Python client library (primary focus)
sfm/server.py ← FastAPI REST server exposing device data over HTTP
seismo_lab.py ← Tkinter GUI (Bridge + Analyzer + Console tabs)
bridges/
mm_probe.py ← name the fault behind a dead unit (4 verdicts, read-only)
mm_link.py ← bench stand-in for a cellular modem, with fault injection
ach_mitm.py ← TCP relay for recording a Series-3 ACH session
docs/
instantel_protocol_reference.md ← Series III protocol spec ("the Rosetta Stone")
micromate_protocol_reference.md ← Series IV protocol spec + THOR's measured behaviour
idf_protocol_reference.md ← Thor IDF file format
instantel_protocol_reference.md ← reverse-engineered protocol spec ("the Rosetta Stone")
CHANGELOG.md ← version history
```
-338
View File
@@ -1,338 +0,0 @@
#!/usr/bin/env python3
"""
mm_link.py — a "perfect modem" between THOR and a Micromate, with a readable
log and deliberate fault injection.
Why
---
THOR gives almost no visibility into a connection: a refresh button, two poll
intervals, and no way to see whether a check succeeded, timed out, or was never
sent. When a unit "won't stay connected" there is nothing to look at.
This sits where the cellular modem would sit and answers the question directly:
* **What is THOR actually doing?** Every frame is decoded and timestamped —
`POLL`, `MONITOR_STATUS`, `SETUP_NAME_READ` — not a hex dump.
* **Is it even trying?** Silence is visible: the log shows gaps.
* **How does it behave when the link misbehaves?** Faults can be injected on
demand, which a real cell link will not do on cue.
Point THOR at this host and port exactly as if it were a modem (Communication:
TCP, IP: <this host>, Port: <--listen>).
Fault injection
---------------
Write a mode into the control file (default `mm_link.ctl`) and it takes effect
on the next byte:
echo pass > mm_link.ctl # normal relay
echo blackhole > mm_link.ctl # TCP stays up, bytes are swallowed
echo drop > mm_link.ctl # close the connection abruptly (RST-ish)
echo delay:2.0 > mm_link.ctl # forward, but 2 s late in both directions
echo onewaydev > mm_link.ctl # THOR->unit passes, unit->THOR is swallowed
**`blackhole` is the one that matters.** It reproduces the classic cellular
failure: the socket is still open as far as both ends are concerned, but nothing
crosses. A client that relies on TCP to tell it the peer is gone will sit there
until the OS keepalive fires — which by default is about two hours.
Usage
-----
python3 bridges/mm_link.py --serial /dev/ttyACM0 --baud 115200 \\
--listen 12345 --logdir ~/mm-captures
Writes, per session:
<logdir>/mmlink_<ts>/session.log decoded, timestamped, human-readable
<logdir>/mmlink_<ts>/raw_bw.bin THOR -> unit, raw
<logdir>/mmlink_<ts>/raw_s3.bin unit -> THOR, raw
The raw pair loads straight into `scratch/mm_frame_parse.py`.
"""
from __future__ import annotations
import argparse
import datetime
import os
import socket
import sys
import threading
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "scratch"))
try:
from mm_frame_parse import SUBNAME, destuff # noqa: F401
except Exception: # pragma: no cover
SUBNAME = {}
import errno
import select
import termios
DLE, STX, ETX, ACK = 0x10, 0x02, 0x03, 0x41
_BAUD = {9600: termios.B9600, 19200: termios.B19200, 38400: termios.B38400,
57600: termios.B57600, 115200: termios.B115200}
class SerialPort:
"""Minimal raw serial port on stdlib termios — no pyserial dependency.
The bench hosts are whatever is to hand; requiring a pip install on someone
else's machine is a poor trade for the ~30 lines this saves.
"""
def __init__(self, path: str, baud: int):
if baud not in _BAUD:
raise ValueError(f"unsupported baud {baud}; pick one of {sorted(_BAUD)}")
self.fd = os.open(path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
a = termios.tcgetattr(self.fd)
a[0] = 0 # iflag: no translation
a[1] = 0 # oflag: raw
a[2] = termios.CS8 | termios.CREAD | termios.CLOCAL # cflag: 8N1, ignore modem lines
a[3] = 0 # lflag: non-canonical, no echo
a[4] = a[5] = _BAUD[baud]
a[6] = list(a[6])
a[6][termios.VMIN] = 0
a[6][termios.VTIME] = 0
termios.tcsetattr(self.fd, termios.TCSANOW, a)
termios.tcflush(self.fd, termios.TCIOFLUSH)
def read(self, n: int) -> bytes:
r, _, _ = select.select([self.fd], [], [], 0.2)
if not r:
return b""
try:
return os.read(self.fd, n)
except OSError as e:
if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK):
return b""
raise
def write(self, data: bytes) -> None:
while data:
try:
data = data[os.write(self.fd, data):]
except OSError as e:
if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK):
select.select([], [self.fd], [], 0.2)
continue
raise
def close(self) -> None:
try:
os.close(self.fd)
except OSError:
pass
def name_of(sub: int, is_request: bool) -> str:
if is_request:
return SUBNAME.get(sub, f"SUB_{sub:02X}")
return "rsp " + SUBNAME.get(0xFF - sub, f"SUB_{0xFF - sub:02X}")
class FrameSniffer:
"""Accumulate bytes and report complete frames, without altering the stream."""
def __init__(self, is_request: bool):
self.is_request = is_request
self.buf = bytearray()
def feed(self, data: bytes):
"""Yield (sub, payload_len) for each complete frame seen."""
self.buf.extend(data)
while True:
start = -1
for i, b in enumerate(self.buf):
if self.is_request and b == ACK and i + 1 < len(self.buf) and self.buf[i + 1] == STX:
start = i
break
if not self.is_request and b == STX:
start = i
break
if start < 0:
if len(self.buf) > 8192:
del self.buf[:-16]
return
j = start + (2 if self.is_request else 1)
end = -1
while j < len(self.buf):
if self.buf[j] == DLE and j + 1 < len(self.buf):
j += 2
continue
if self.buf[j] == ETX:
end = j
break
j += 1
if end < 0:
return # wait for more bytes
body = self.buf[start:end + 1]
del self.buf[:end + 1]
# SUB sits at a fixed spot past the leading framing -- but it is
# DLE-escaped when its own value is 0x02/0x03/0x04/0x10, so a raw
# read reports 0x10 for those. SUB 0x02 was being logged as
# "SUB_10" until this was handled.
off = 5 if self.is_request else 3
if len(body) > off:
sub = body[off]
if sub == DLE and len(body) > off + 1:
sub = body[off + 1]
yield sub, len(body)
class Link:
def __init__(self, args):
self.args = args
self.mode = "pass"
self.delay = 0.0
self.ctl = Path(args.control)
self.session: Path | None = None
self.log_fh = None
self.raw = {}
self.t0 = time.time()
self.counts = {}
# ── logging ────────────────────────────────────────────────────────────
def open_session(self):
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
self.session = Path(self.args.logdir) / f"mmlink_{ts}"
self.session.mkdir(parents=True, exist_ok=True)
self.log_fh = open(self.session / "session.log", "a", buffering=1)
self.raw = {
"bw": open(self.session / "raw_bw.bin", "ab"),
"s3": open(self.session / "raw_s3.bin", "ab"),
}
self.say(f"=== session {ts} — serial {self.args.serial} @ {self.args.baud} ===")
def say(self, text: str):
line = f"{datetime.datetime.now().strftime('%H:%M:%S.%f')[:-3]} {text}"
print(line, flush=True)
if self.log_fh:
self.log_fh.write(line + "\n")
# ── control file ───────────────────────────────────────────────────────
def poll_control(self):
while True:
try:
if self.ctl.exists():
want = self.ctl.read_text().strip().lower()
if want.startswith("delay:"):
d = float(want.split(":", 1)[1])
if ("delay", d) != (self.mode, self.delay):
self.mode, self.delay = "delay", d
self.say(f"*** MODE -> delay {d}s ***")
elif want and want != self.mode:
self.mode, self.delay = want, 0.0
self.say(f"*** MODE -> {want} ***")
except Exception:
pass
time.sleep(0.25)
# ── the relay ──────────────────────────────────────────────────────────
def pump(self, src, dst, tag: str, is_request: bool, stop: threading.Event):
sniff = FrameSniffer(is_request)
arrow = "THOR->unit" if is_request else "unit->THOR"
last = time.time()
while not stop.is_set():
timed_out = False
try:
data = src.recv(4096) if isinstance(src, socket.socket) else src.read(4096)
except TimeoutError:
timed_out = True
# socket.timeout subclasses OSError, so it MUST be caught first.
# Treating it as a dead socket closes the connection after 200 ms
# of quiet -- which is exactly what `blackhole` produces, so the
# relay killed the link it was supposed to be faking a fault on.
data = b""
except OSError:
break
if isinstance(src, socket.socket) and data == b"" and not timed_out:
self.say(f"{arrow}: peer closed the connection")
break
if not data:
if time.time() - last > self.args.quiet_after and self.counts:
self.say(f"--- {self.args.quiet_after:.0f}s with no traffic ---")
last = time.time()
continue
last = time.time()
self.raw[tag].write(data)
self.raw[tag].flush()
for sub, ln in sniff.feed(data):
label = name_of(sub, is_request)
self.counts[label] = self.counts.get(label, 0) + 1
self.say(f"{arrow} {label:<20} ({ln} B)"
+ ("" if self.mode == "pass" else f" [mode={self.mode}]"))
mode = self.mode
if mode == "drop":
self.say(f"{arrow}: DROPPING the connection (fault injection)")
stop.set()
break
if mode == "blackhole":
continue # swallow, keep the socket open
if mode == "onewaydev" and not is_request:
continue # unit's replies never reach THOR
if mode == "delay" and self.delay:
time.sleep(self.delay)
try:
if isinstance(dst, socket.socket):
dst.sendall(data)
else:
dst.write(data)
except OSError:
break
stop.set()
def serve(self):
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
srv.bind(("0.0.0.0", self.args.listen))
srv.listen(5)
self.open_session()
self.say(f"listening on 0.0.0.0:{self.args.listen} control file: {self.ctl}")
self.say("point THOR at this host/port as Communication=TCP")
threading.Thread(target=self.poll_control, daemon=True).start()
while True:
conn, addr = srv.accept()
conn.settimeout(0.2)
self.say(f"+++ THOR connected from {addr[0]}:{addr[1]} +++")
try:
ser = SerialPort(self.args.serial, self.args.baud)
except OSError as e:
self.say(f"!!! cannot open {self.args.serial}: {e}")
conn.close()
continue
stop = threading.Event()
ts = [
threading.Thread(target=self.pump, args=(conn, ser, "bw", True, stop), daemon=True),
threading.Thread(target=self.pump, args=(ser, conn, "s3", False, stop), daemon=True),
]
for t in ts:
t.start()
for t in ts:
t.join()
conn.close()
ser.close()
summary = ", ".join(f"{k}x{v}" for k, v in sorted(self.counts.items()))
self.say(f"--- connection closed. frames this session: {summary or 'none'} ---")
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--serial", default="/dev/ttyACM0")
ap.add_argument("--baud", type=int, default=115200)
ap.add_argument("--listen", type=int, default=12345)
ap.add_argument("--logdir", default=os.path.expanduser("~/mm-captures"))
ap.add_argument("--control", default="mm_link.ctl")
ap.add_argument("--quiet-after", type=float, default=30.0,
help="log a marker after this many seconds of silence")
Link(ap.parse_args()).serve()
if __name__ == "__main__":
main()
-239
View File
@@ -1,239 +0,0 @@
#!/usr/bin/env python3
"""
mm_probe.py — answer "why can't we reach this unit?" in one command.
THOR reports a failed connection as "disconnected" and nothing else. That single
word covers at least four completely different faults with four different fixes,
and telling them apart is the difference between a modem reboot and a site visit:
* **connection refused** something answered and said no — wrong port, or the
modem is refusing a further session
* **connect timed out** nothing answered at all — trusted-IP whitelist,
firewall, or the modem is off the network
* **connected, no reply** the MODEM answered but the unit did not. The TCP
path is fine; the modem is not forwarding to serial.
This is the signature of a wedged transparent-TCP
session, and it is the one THOR cannot distinguish
from any of the others
* **replied** the unit is alive; the problem is upstream software
Read-only. It sends `POLL`, then optionally `SERIAL` and the state read — the
same three commands THOR's own connection check uses — and never writes.
Usage
-----
python3 bridges/mm_probe.py 63.45.161.30:9034
python3 bridges/mm_probe.py 10.0.0.8:12345 --timeout 5
python3 bridges/mm_probe.py <host:port> --slots 3
`--slots N` opens N connections at once and reports how many the far end accepts.
A transparent-TCP modem typically serves **one** session; if the first succeeds
and the rest are refused or hang, that confirms the single-slot behaviour and
explains why a leaked session takes a unit offline until the slot frees.
Works for both series: a Series III reply opens `DLE STX`, a Micromate reply
opens with a bare `STX`, so the probe also tells you which one answered.
"""
from __future__ import annotations
import argparse
import socket
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from minimateplus.framing import build_bw_frame # noqa: E402
DLE, STX, ETX = 0x10, 0x02, 0x03
def destuff(raw: bytes) -> bytes:
"""Strip framing and DLE escapes; return the payload without its checksum."""
i = 1 if raw and raw[0] == STX else (2 if len(raw) > 1 and raw[1] == STX else 0)
out = bytearray()
while i < len(raw):
b = raw[i]
if b == DLE and i + 1 < len(raw):
out.append(raw[i + 1])
i += 2
continue
if b == ETX:
break
out.append(b)
i += 1
return bytes(out[:-1]) if len(out) > 1 else b""
# Reads are two-step on Series III: a probe at offset 0, then a data read at the
# block's length. THOR sends these offsets, and they also work on a Micromate.
OFFSETS = {0x5B: 0x0030, 0x15: 0x000A, 0x49: 0xFFFF}
def exchange(sock: socket.socket, sub: int, timeout: float) -> tuple[bytes, float]:
sock.sendall(build_bw_frame(sub, OFFSETS.get(sub, 0)))
t0 = time.time()
buf, deadline = b"", t0 + timeout
sock.settimeout(0.3)
while time.time() < deadline:
try:
chunk = sock.recv(4096)
if not chunk:
break
buf += chunk
if buf.endswith(bytes([ETX])) and len(buf) > 8:
break
except TimeoutError:
continue
except OSError:
break
return buf, time.time() - t0
def step(n: int, label: str, result: str) -> None:
print(f" [{n}] {label:.<28} {result}")
def probe(host: str, port: int, timeout: float) -> int:
print(f"\ntarget {host}:{port} (read-only: POLL, SERIAL, state)\n")
# ── 1. TCP ────────────────────────────────────────────────────────────
t0 = time.time()
try:
sock = socket.create_connection((host, port), timeout=timeout)
except ConnectionRefusedError:
step(1, "TCP connect", f"REFUSED after {1000*(time.time()-t0):.0f} ms")
print("\nverdict: something answered and actively refused.")
print(" Not a silent firewall drop — the host is reachable.")
print(" Wrong port, the service is down, or the modem is refusing")
print(" an additional session because its one slot is in use.")
return 2
except (TimeoutError, socket.timeout):
step(1, "TCP connect", f"TIMED OUT after {time.time()-t0:.1f} s")
print("\nverdict: nothing answered at all.")
print(" A silent drop, which is what a trusted-IP whitelist looks")
print(" like — it discards rather than refuses. Check the modem's")
print(" Trusted IPs (and note a VPN changes the IP you arrive from),")
print(" the firewall, and whether the modem is on the network.")
return 3
except OSError as e:
step(1, "TCP connect", f"FAILED: {e}")
return 4
step(1, "TCP connect", f"ok ({1000*(time.time()-t0):.0f} ms)")
# ── 2. POLL ───────────────────────────────────────────────────────────
try:
raw, dt = exchange(sock, 0x5B, timeout)
except OSError as e:
step(2, "POLL", f"send failed: {e}")
sock.close()
return 4
if not raw:
step(2, "POLL", f"NO REPLY in {timeout:.1f} s")
print("\nverdict: the MODEM answered but the unit did not.")
print(" TCP is fine end to end — something accepted the connection.")
print(" What is missing is the serial side. Two quite different")
print(" causes produce this, and they are NOT distinguishable from")
print(" here:")
print("\n 1. SOMEONE ELSE HOLDS THE SESSION. These modems bridge ONE")
print(" TCP session to serial at a time. A second connection is")
print(" accepted and then simply not forwarded. Confirmed 2026-09-26:")
print(" with THOR connected this probe saw exactly this; the moment")
print(" THOR disconnected the same probe returned the serial number.")
print(" ** Check whether THOR (or anything else) has the unit first. **")
print("\n 2. The serial path is genuinely broken — a stale session the")
print(" modem never released, a cable the unit cannot enumerate, or")
print(" a unit that is off.")
print("\n Try, in order:")
print(" 1. Disconnect any other client and re-probe. If it answers,")
print(" it was contention, not a fault.")
print(" 2. The cable's chipset. A Micromate drives FTDI and CDC-ACM")
print(" only — a Prolific PL2303 gives it no serial port at all.")
print(" lsusb: FTDI is 0403, Prolific 067b.")
print(" 3. Power-cycle the UNIT with the cable attached (hold power")
print(" 5 s, through the two-stage prompt). Its USB host rescans")
print(" on cold boot; it may not on hot-swap.")
print(" 4. AirLink OS -> TCP Idle Timeout. If 0/disabled, a stale")
print(" session holds the slot indefinitely. 2 minutes is the")
print(" value this project standardised on.")
sock.close()
return 5
series = "Series III (DLE STX)" if raw[0] == DLE else "Micromate (bare STX)"
step(2, "POLL", f"reply {len(raw)} B in {1000*dt:.0f} ms")
p = destuff(raw)
ok = len(p) > 3 and p[2] == 0xFF - 0x5B
step(3, "frame", f"{'valid' if ok else 'MALFORMED'}, {series}")
if not ok:
print("\nverdict: something replied, but not a seismograph.")
print(" Another service is on this port, or the modem is in a mode")
print(" that injects its own text (check Quiet Mode / AT echo).")
print(f" first bytes: {raw[:16].hex(' ')}")
sock.close()
return 6
# ── 3. identity + state ───────────────────────────────────────────────
for n, (sub, label) in enumerate(((0x15, "serial"), (0x49, "state")), start=4):
try:
r, dt = exchange(sock, sub, timeout)
d = destuff(r)[5:]
if sub == 0x15:
# serial is a null-terminated run; a further field follows it
serial = bytes(d[11:]).split(b"\x00")[0]
step(n, label, serial.decode("ascii", "replace") or "(empty)")
else:
step(n, label, "MONITORING" if len(d) > 11 and d[11] else "idle")
except OSError:
step(n, label, "no reply")
sock.close()
print("\nverdict: the unit is alive and answering.")
print(" If THOR still shows it disconnected, the fault is in THOR, not")
print(" the network or the device.")
return 0
def slots(host: str, port: int, n: int, timeout: float) -> None:
print(f"\nopening {n} simultaneous connections to {host}:{port}\n")
held = []
for i in range(n):
try:
s = socket.create_connection((host, port), timeout=timeout)
held.append(s)
step(i + 1, f"connection {i+1}", "accepted")
except ConnectionRefusedError:
step(i + 1, f"connection {i+1}", "REFUSED")
except (TimeoutError, socket.timeout):
step(i + 1, f"connection {i+1}", "timed out")
except OSError as e:
step(i + 1, f"connection {i+1}", f"failed: {e}")
print(f"\n{len(held)} of {n} accepted.")
if len(held) == 1:
print(" Single-slot behaviour confirmed — this far end serves ONE")
print(" session at a time. A connection that is never closed takes")
print(" the unit offline until the idle timeout frees the slot.")
for s in held:
s.close()
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("target", help="host:port, e.g. 63.45.161.30:9034")
ap.add_argument("--timeout", type=float, default=10.0)
ap.add_argument("--slots", type=int, metavar="N",
help="open N simultaneous connections to test single-slot behaviour")
a = ap.parse_args()
host, _, port = a.target.rpartition(":")
if not host:
ap.error("target must be host:port")
if a.slots:
slots(host, int(port), a.slots, a.timeout)
return 0
return probe(host, int(port), a.timeout)
if __name__ == "__main__":
raise SystemExit(main())
-291
View File
@@ -1,291 +0,0 @@
# Spec — a live client for Series IV (Micromate)
Drafted 2026-09-26, ahead of implementation. The protocol work is finished; this
is the plan for turning `docs/micromate_protocol_reference.md` into code SFM can
run.
**Read that document first.** Everything here assumes it, and every constant
below is sourced from it rather than restated with justification.
---
## Goal and scope
`micromate/` is codec-only today — `idf_file.py`, `models.py`, the report
writers. There is no way to talk to a unit. This adds the live half, mirroring
`minimateplus/`.
**In scope, first pass:**
- connect over TCP (a field modem) or serial/USB (a bench unit)
- identify a unit, read its state, clock, memory and setups
- walk the event chain and download events
- return `Event` objects the existing codec already understands
**Explicitly out of scope, first pass:**
- ⚠ **Any write.** Setups, schedules, call-home config, monitoring start/stop,
and per-event delete are all mapped, and none of them will be implemented
here. **No command has ever been originated against a unit by this project**
— every write observed was performed by THOR while we recorded. Keeping that
true through the read client is deliberate: it means the first thing we ever
send to a customer's instrument is a decision someone made on purpose, not a
side effect of a client that happened to grow a method.
- the inbound call-home session — still the one protocol unknown
---
## Layout
```
micromate/
framing.py NEW frame building, response parsing, checksum
protocol.py NEW one method per wire command, returns raw payloads
client.py NEW high-level API, returns models
idf_file.py (existing — decodes what 0x5A returns, unchanged)
models.py (existing — extend, do not fork)
```
**Transport is reused, not rewritten.** `minimateplus/transport.py` is
byte-level and protocol-agnostic — `BaseTransport`, `SerialTransport`,
`TcpTransport`, plus `read_until_idle()` which already handles the RV50/RV55
habit of emitting `\r\nRING\r\n\r\nCONNECT\r\n` to a caller. Import it.
⚠ Do **not** import `minimateplus.framing`. The two framings differ in ways
that look small and are not, and a shared module would accumulate `if series ==`
branches until neither case is readable.
---
## `micromate/framing.py`
### Requests
Series IV accepts Series III request frames unmodified. The simplest correct
implementation re-exports the builder rather than duplicating it:
```python
from minimateplus.framing import build_bw_frame # requests are identical
```
⚠ One open question, flagged in the protocol reference and **not** settled:
whether `0x10` bytes inside request *params* need stuffing. No probe we sent
carried one. Until it is settled, assert on it rather than guessing:
```python
def build_request(sub: int, offset: int = 0, params: bytes = bytes(10)) -> bytes:
if 0x10 in params:
raise NotImplementedError(
"params containing 0x10 — stuffing rule unconfirmed; see "
"micromate_protocol_reference.md, 'Untested and unsafe-until-agreed'"
)
return build_bw_frame(sub, offset, params)
```
That turns an unknown into a loud failure instead of a corrupt frame.
### Responses — where Series III's parser cannot follow
| | Series III | Micromate |
|---|---|---|
| frame start | `DLE STX` | **bare `STX`** |
| `payload[1]` | `0x10` | `0xC5` (Blastware fw) / `0x03` (Thor fw) |
| destuffing | `DLE+ETX` kept as literal inner-frame data | **`10 XX` → `XX`, uniformly** |
The first row is why `S3FrameParser` returns nothing at all on Series IV traffic:
it scans for `DLE+STX`, which never appears.
The third is a genuine **simplification** — no inner-frame carve-out. Validated
by checksum across every capture in `bridges/captures/9-24-26 - micromate2/`:
four candidate destuffing rules were tried, and only this one makes all frames
validate.
### Checksum
```python
def checksum(payload: bytes) -> int:
return sum(b for b in payload if b != 0x10) & 0xFF
```
The DLE-aware variant, same as Series III's `5A` and write frames — not the plain
SUM8 of ordinary Series III reads.
### ⚠ The SUB byte can be escaped
When a SUB's value is `0x02`, `0x03`, `0x04` or `0x10` it arrives as `10 XX`.
Reading it positionally without destuffing reports `0x10`. This bit once
already — `SUB 0x02` was logged as `SUB_10` for an afternoon. Destuff first,
then index.
### Response shape
```python
@dataclass
class MicromateFrame:
sub: int # response SUB; request = 0xFF - sub
flags: int # 0xC5 Blastware line, 0x03 Thor line
page_hi: int
page_lo: int
data: bytes # payload[5:], checksum stripped
checksum_valid: bool
@property
def firmware_line(self) -> str: # "blastware" | "thor" | "unknown"
@property
def declared_length(self) -> int: # uint16 BE at data[3:5] (= payload[8:10])
```
⚠ **`declared_length` is a uint16 BE.** Read as a single byte it under-reads
`SUB 0x1A` by 47x — 44 against a true 2092. This is the single most expensive
mistake available in this protocol and it has already been made once.
`MicromateFrameParser` mirrors `S3FrameParser`: `feed(bytes) -> list[frame]`,
accumulates in `.frames`, `reset()`, and keeps the `bytes_fed` counter (it is
what distinguishes "no bytes at all" from "bytes but no complete frame" on a
timeout, and that distinction earned its keep during the Series III work).
---
## `micromate/protocol.py`
One method per command, returning raw payload bytes. No interpretation — that
belongs in `client.py`.
**Reads use `offset = 0xFFFF`** and return the whole block in one response;
Series III's two-step probe/data dance is unnecessary. `POLL` is the exception,
taking its data length. Per-command offsets, all observed:
| command | SUB | rsp | offset | returns |
|---|---|---|---|---|
| poll | `0x5B` | `0xA4` | `0x0030` | device string, model |
| serial | `0x15` | `0xEA` | `0x000A` | `UM12947` |
| device info | `0x01` | `0xFE` | `0xFFFF` | firmware, calibration |
| state | `0x49` | `0xB6` | `0xFFFF` | `data[11]`: non-zero = monitoring |
| monitor status | `0x1C` | `0xE3` | `0xFFFF` | flag, **device clock**, battery, memory |
| storage range | `0x06` | `0xF9` | `0xFFFF` | event storage extent |
| active setup name | `0x41` | `0xBE` | `0xFFFF` | `TEST1.mmb` |
| first setup | `0x3F` | `0xC0` | `0xFFFF` | setup-list walk head |
| next setup | `0x40` | `0xBF` | `0xFFFF` | …until an empty name |
| compliance config | `0x1A` | `0xE5` | `0xFFFF` | ~2103 B setup block |
| call-home config | `0x2C` | `0xD3` | `0xFFFF` | 137 B |
| arm event | `0x93` | `0x6C` | — | before every event |
| first event | `0x1E` | `0xE1` | `0xFFFF` | key + size |
| next event | `0x1F` | `0xE0` | `0xFFFF` | key + size |
| event record | `0x0C` | `0xF3` | `0xFFFF` | 210 B — project, location, peaks |
| event header | `0x0A` | `0xF5` | `0xFFFF` | 30 B list record |
| bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim** |
⚠ **`SUB 0x1C` is 4 bytes longer on the Thor firmware line** (`0x30` vs `0x2C`).
Parse **forward** from `declared_length`, never backward from the end — Series
III reads battery and memory from the end of that block, and doing so on a BD
unit yields a battery voltage of **577.92 V**.
⚠ **Test the monitoring flag for non-zero**, never against a constant. It has
read both `0x0E` and `0x0C` while monitoring.
### `0x5A` — simpler than Series III, deliberately
No arming ritual, no chunk loop, no `STRT` end-offset parsing, no `TERM` frame.
One request returns the whole event:
```python
offset_word = 0x1000 + 2 * ceil(size / 512) # size from the chain walk
```
The payload **is** the `.IDFW` file, byte for byte — so it feeds
`micromate.idf_file.read_idf_file()` and `/db/import/idf_file` unchanged.
⚠ Do not port the Series III `5A` walk. Its address arithmetic caused a 5x
over-read and a `> 64 KB` page-boundary bug that is *still open* on the Series
III side. None of that applies here.
---
## `micromate/client.py`
```python
class MicromateClient:
def __init__(self, transport: BaseTransport): ...
def open(self) / close(self) / is_open(self)
# identity and state
def connect(self) -> DeviceInfo # poll → serial → device info → state
def get_state(self) -> UnitState # monitoring?, clock, battery, memory
def get_active_setup(self) -> str
def list_setups(self) -> list[str] # 0x3F → 0x40… until empty
# events
def list_events(self) -> list[EventRef] # 0x93 → 0x1E → 0x1F… (key + size)
def download_event(self, ref) -> bytes # raw .IDFW/.IDFH
def get_event(self, ref) -> Event # download + decode via idf_file
```
`connect()` should mirror THOR's preamble (`POLL → SERIAL → 0x49 → POLL`) —
⚠ but note the reference records that **whether the unit requires it is
untested**. Do it because it is known-good, not because it is known-necessary,
and say so in the docstring.
`list_events()` returns the key *and* the size, because `download_event()` needs
the size to compute its offset word.
---
## Tests
**Offline, from captured bytes — no hardware.** This is the part worth doing
first, because it can be fully verified tonight's-captures-style before any unit
is involved.
```
tests/test_micromate_framing.py
```
⚠ `bridges/captures/` and `tests/fixtures/` are both gitignored, so tests must
not depend on files being present. **Embed the frames as hex constants** — they
are 16–68 bytes each, and a handful covers every case:
| case | why |
|---|---|
| POLL probe reply, 19 B | shortest valid frame |
| POLL data reply, 68 B | contains a literal `0x10` — only the DLE-aware checksum matches |
| `0x1A` response, 2108 B | exercises `declared_length` as a true uint16 (`0x082C`) |
| a Thor-line reply, `flags = 0x03` | `0x03` is ETX; proves destuffing before framing |
| a frame whose SUB is `0x02` | arrives as `10 02`; proves destuff-then-index |
| a truncated frame | parser must return nothing, not a bad frame |
| a corrupted checksum | `checksum_valid == False`, frame still returned |
Then a round-trip assertion: feed a whole captured session through the parser and
assert the frame count and every SUB, against `scratch/mm_frame_parse.py`'s
output — which is already known good, having parsed 24, 38 and 40-frame sessions
with zero bad checksums.
**Live, second:** against the bench unit on mint-mac via `mm_link.py`.
`connect()`, `list_setups()` (should return the 23 known names), `list_events()`,
then `download_event()` and assert the bytes decode and match a
`/db/import/idf_file` ingest of the same event.
---
## Order of work
1. `framing.py` + its tests — offline, verifiable immediately
2. `protocol.py` — reads only, one method per row of the table above
3. `client.py` — `connect()`, `get_state()`, `list_setups()`
4. the event chain and `download_event()`
5. decode end-to-end and compare against a store event
Steps 1–2 need no hardware at all.
---
## Open questions to settle while implementing
- **Request param stuffing** — raise `NotImplementedError` rather than guess.
- **Is THOR's preamble required?** Try one command cold and find out; it is a
two-minute test with the bench unit and it removes a ritual if unnecessary.
- **`Event` model fit** — Series IV carries fields Series III lacks (setup file
name, `LMic`/`SMic` channels). Extend `micromate/models.py`; do not fork the
shared `Event`.
- **Which `0x0C` fields to trust.** The peak float there runs 2–5% above
`max(T,V,L)` and is **not** the vector sum; its offset was inferred, not
established. The reference marks it do-not-rely-on — prefer decoded samples.
File diff suppressed because it is too large Load Diff
+118
View File
@@ -934,6 +934,120 @@ which looks like a different failure mode wearing the same signature.
---
### 8f. BE12599 — CONFIRMED on the bench: the geophone assembly, not the recorder (2026-09-23)
**The first mechanism in this investigation established physically rather than
inferred.** Brian pulled BE12599 from I-80 Abut 1 West on 2026-09-21 and bench-
tested it. A geophone swap settles it.
#### The swap test
Same recorder, twenty minutes apart, sensor self-check:
| geophone fitted | Tran | **Vert** | Long |
|---|---|---|---|
| BE12599's own (2026-09-22 21:10) | 4.2 | **2026.0** | 4.8 |
| BE9888's known-good (2026-09-22 22:48) | 3.6 | **3.4** | 3.6 |
Vert frequency likewise **2.2 Hz -> 7.4 Hz**. Fleet reference: overswing 3.3–5.1
(median 3.8, max 5.1 across 42,979 passing records), frequency 7.4–7.8 Hz.
With the good geophone the unit shows **no offsets, no self-triggering, all three
channels Passed**, and records a clean damped impulse with broadband 5–100 Hz
content. **BE12599 the recorder is functional** — it is only out of calibration
(due 2025-08-12).
**The fault is in the geophone assembly: element, cable, or its connector.**
The swap does not separate those three, because the cable is permanently glanded
into the geophone.
#### The bench evidence, with ground motion removed
17 events recorded on a workbench with nothing touching the sensor.
**The control** — `N599LRY0.4B0`, trigger forced by slapping the *microphone*,
geophone untouched:
| channel | peak in/s | frac < 0 | mean/peak |
|---|---|---|---|
| Tran | 0.070 | **0.00** | 0.87 |
| **Vert** | **0.190** | **0.35** | 0.13 |
| Long | 0.070 | **0.00** | 0.84 |
Vert produces 0.19 in/s of bipolar wander with **no mechanical input at all**.
The fault is *continuous*, not episodic — every field event was simply the
moment it crossed the trigger.
**All three channels are affected**, consistently across all 17 events:
Tran and Long parked at a small positive DC (`frac<0 = 0.00`, mp 0.64–0.95);
Vert wandering at 0.89–1.78 Hz, peaks 0.14–0.54. Yet the swing test calls Tran
and Long **Passed** (4.2 / 4.8). So their *coils* are healthy and merely sitting
on an offset, while Vert's circuit has lost damping entirely.
That reads as **leakage on several contacts plus one gone open** — which is what
progressive corrosion at a shared connection does. Three independent geophone
elements do not fail in sequence.
**It is still progressing.** Tran's DC offset was ~0.011 in/s on 2026-09-14 and
~0.064 on 2026-09-22 — 6x in eight days. Vert's overswing went 7.5 -> 2026.0 over
the same window.
#### ⚠ This explains the autozero's 10% success rate
The journal has framed this fault around the **recorder** throughout: Instantel's
autozero, the X1/X8 gains, the 2027–2069 window, `SUB 0x0E` as the best lead.
All of that adjusts the recorder's zero reference.
**An autozero cannot fix a geophone.** If most offsets are geophone-assembly
faults, a re-zero would succeed only on the minority where the recorder really is
at fault — which is exactly the ~10% Brian's father reported and which has sat
unexplained in §5 since 2026-08-28.
**Procedure change: swap the geophone first.** It is a five-minute field test
costing nothing. On this unit it would have answered the question in August.
⚠ Scoped to one unit. The other four confirmed units are the obvious check.
#### Correction to §8e
§8e is titled "a connector, not a geophone" and concluded the element was fine
because the fault hopped channels. The channel-hopping observation stands and the
swap test supports the general shape, but **that title overstated what was known**:
the swap narrows the fault to the *assembly*, and element / cable / connector are
still open within it. §8e's reasoning was sound; its confidence was not.
#### Cause — road salt now outranks the mice
The enclosure held a mouse nest on the battery ([[§8e]]), and that framed the
2026-09-22 analysis. But **the geophone is buried in the ground, not in the
enclosure** — they are two independent contamination paths, and the fault is in
the buried half.
The unit sat **~6 feet from I-80 westbound**. Winter road salt spray and highway
runoff put a buried geophone in chloride-rich water for months a year. Chloride
drives exactly this failure: corrosion at terminations, leakage across
insulation, eventually an open circuit.
The mice remain a real finding for the *enclosure* and the recorder end. They are
probably not the cause of this fault.
**Testable consequence:** road salt is a property of the **site**, not the unit.
Units deployed near salted highways should carry higher geophone-assembly failure
rates than units on quiet sites. Nothing in the archive records proximity to a
road, so this needs the deployment records rather than the event data.
#### Open — to be settled by opening the geophone
1. **Meter at the geophone plug** before disassembly: three coil pairs, and each
to shell. Prediction: **Vert open or very high** (that is the 2026 overswing);
Tran and Long matched and normal; **insulation to shell finite on all three**,
Vert worst (that is the leakage putting DC on healthy coils).
2. **Then open it.** Look for a water line, white/green corrosion at the coil
terminations, the state of the damping shunt, and the cable gland from the
inside. Measure the coils again **at the internal terminals** — if Vert reads
fine there, the fault is in the internal wiring or the gland, not the element.
---
### ⚠ Serial prefixes — four of these units are BlastMates, not MiniMates
Corrected 2026-09-06, after Brian queried "BA10895?" against a report that
@@ -985,3 +1099,7 @@ decoder lost the geo threshold along with the serial. See commit `9ceff65`.
| 2026-09-06 | The spread gate is **blind to onsets by construction** — it rejects a moving floor, which is what an onset is. It rejected the very record in which the ramp is visible. |
| 2026-09-07 | **The non-motion test** (Brian: "it doesn't cross zero"). `\|mean\|/peak` is bimodal with a 1.09% dead zone and returns exactly the 5 confirmed units from physics, not a threshold. Independent corroboration of the unit list. **BE11007 settled as NOT an offset** — a one-sided 9 in/s blast. |
| 2026-09-07 | **BE12599 is a connector fault, not a geophone fault.** Unipolar spikes with a 26→118 ms RC tail progressing to a flat pedestal, and the fault MOVES between Long and Tran while the sensor self-check passes on every event. An element cannot hop channels; a connector can. Inspect before reseating. |
| 2026-09-23 | **CONFIRMED: BE12599's fault is the geophone assembly, not the recorder.** Geophone swap on the same unit: Vert overswing **2026.0 -> 3.4**, frequency 2.2 -> 7.4 Hz, all three channels Passed, no self-triggering. First physically established mechanism in this investigation. |
| 2026-09-23 | **This explains the autozero's ~10% success rate** (§5): an autozero adjusts the *recorder's* zero reference and cannot fix a geophone. **Procedure change: swap the geophone first.** |
| 2026-09-23 | Bench control event (mic-slap trigger, geophone untouched) shows Vert wandering **0.19 in/s with no mechanical input** — the fault is continuous, not episodic. All three channels affected; Tran/Long parked on DC while their coils pass the swing test. |
| 2026-09-23 | **Cause: road salt now outranks the mice.** The geophone is buried, not in the enclosure — two independent contamination paths, and the fault is in the buried half. The unit sat ~6 ft from I-80 WB. Salt is a *site* property, so this predicts higher failure rates near salted highways. |
-33
View File
@@ -1,33 +0,0 @@
"""Pretend to be a Micromate on a serial port: log what arrives, reply to POLL.
Proves the modem's return path (serial -> TCP) independently of the real unit.
"""
import os, select, sys, termios, time
path, baud = sys.argv[1], int(sys.argv[2]) if len(sys.argv) > 2 else 115200
B = {9600: termios.B9600, 38400: termios.B38400, 115200: termios.B115200}[baud]
fd = os.open(path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
a = termios.tcgetattr(fd)
a[0] = a[1] = a[3] = 0
a[2] = termios.CS8 | termios.CREAD | termios.CLOCAL
a[4] = a[5] = B
a[6] = list(a[6]); a[6][termios.VMIN] = 0; a[6][termios.VTIME] = 0
termios.tcsetattr(fd, termios.TCSANOW, a)
termios.tcflush(fd, termios.TCIOFLUSH)
# A real POLL probe reply, captured from UM12947 on 2026-09-24.
REPLY = bytes.fromhex("0200c5a4000000000000300000000000000099") + b"\x03"
print(f"fake unit on {path} @ {baud}; will answer any inbound frame", flush=True)
while True:
r, _, _ = select.select([fd], [], [], 1.0)
if not r:
continue
data = os.read(fd, 4096)
if not data:
continue
ts = time.strftime("%H:%M:%S")
print(f"{ts} IN {len(data):3} B {data.hex(' ')}", flush=True)
time.sleep(0.02)
os.write(fd, REPLY)
print(f"{ts} OUT {len(REPLY):3} B {REPLY.hex(' ')} <- canned POLL reply", flush=True)
-202
View File
@@ -1,202 +0,0 @@
#!/usr/bin/env python3
"""
mm_frame_parse.py — parse Micromate (Series IV) frames out of a seismo_lab
raw capture pair.
Why this exists
---------------
`minimateplus.framing.S3FrameParser` cannot see Micromate traffic. It locates
frames by scanning for `DLE STX`, and a Micromate response has **no leading
DLE** — it starts at a bare `STX`. It also expects `payload[1] == 0x10`, where
the Micromate sends `0xC5` (Blastware firmware) or `0x03` (Thor firmware).
The practical consequence, seen on the 9-24-26 setup-push capture: the
Blastware-side requests parse fine (Thor emits Series III request frames), but
**every device response is silently dropped or mis-framed** — so a capture that
actually contains 12 acked writes looks like 12 unanswered requests.
Destuffing
----------
One rule covers both directions: after the leading doubled `BW_CMD`, every
`10 XX` pair on the wire destuffs to `XX`. That includes `10 03` — Thor
escapes literal `0x03` bytes in write data so they are not mistaken for ETX,
exactly as Blastware does.
That rule was chosen by evidence, not assumption: of the four candidates tried
against the 9-24-26 capture's four data-carrying write frames, it is the only
one under which all four checksums validate. See
`docs/micromate_protocol_reference.md` → *The write path*.
Usage
-----
python scratch/mm_frame_parse.py <capture-dir>
python scratch/mm_frame_parse.py <raw_bw.bin> <raw_s3.bin>
python scratch/mm_frame_parse.py <capture-dir> --dump 0x71
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
DLE, STX, ETX, ACK = 0x10, 0x02, 0x03, 0x41
# Request SUB -> short name. Series III names where they carry over; the
# Series IV additions are marked.
SUBNAME = {
0x01: "DEVICE_INFO",
0x06: "STORAGE_RANGE",
0x08: "EVENT_INDEX",
0x0A: "WAVEFORM_HDR",
0x0C: "WAVEFORM_REC",
0x15: "SERIAL",
0x1A: "COMPLIANCE_CFG",
0x1C: "MONITOR_STATUS",
0x1E: "EVENT_HDR",
0x2C: "CALLHOME_CFG",
0x2E: "TRIGGER_CFG_READ", # Series IV
0x3E: "OPERATOR",
0x41: "SETUP_NAME_READ", # Series IV
0x5A: "BULK_DOWNLOAD",
0x5B: "POLL",
0x68: "EVENT_INDEX_WRITE",
0x69: "WAVEFORM_WRITE",
0x71: "COMPLIANCE_WRITE",
0x72: "CONFIRM_A",
0x73: "CONFIRM_B",
0x74: "CONFIRM_C",
0x82: "TRIGGER_WRITE",
0x83: "TRIGGER_CONFIRM",
0xDA: "SETUP_FILE_DECL", # Series IV — names the target .MMB
0xFE: "FULL_CFG",
}
def destuff(blob: bytes, start: int, *, is_request: bool) -> tuple[bytes, int, int]:
"""Destuff one frame starting at `start`.
Returns (payload, checksum, index_of_terminating_ETX). `payload` excludes
the trailing checksum byte. A request frame opens `ACK STX 10 10`; a
response opens with a bare `STX`.
"""
i = start + (2 if is_request else 1)
out = bytearray()
if is_request:
# The doubled BW_CMD is the one guaranteed stuffed byte.
if blob[i : i + 2] != bytes([DLE, DLE]):
raise ValueError(f"@0x{start:04x}: request does not open with 10 10")
out.append(DLE)
i += 2
while i < len(blob):
b = blob[i]
if b == DLE and i + 1 < len(blob):
out.append(blob[i + 1])
i += 2
continue
if b == ETX:
break
out.append(b)
i += 1
if len(out) < 2:
raise ValueError(f"@0x{start:04x}: frame too short")
return bytes(out[:-1]), out[-1], i
def frames(blob: bytes, *, is_request: bool):
"""Yield (offset, payload, chk, checksum_kind)."""
i, n = 0, len(blob)
while i < n:
if is_request:
if not (blob[i] == ACK and i + 1 < n and blob[i + 1] == STX):
i += 1
continue
elif blob[i] != STX:
i += 1
continue
try:
payload, chk, end = destuff(blob, i, is_request=is_request)
except ValueError:
i += 1
continue
sum8 = sum(payload) & 0xFF
dle_aware = (sum(b for b in payload if b != DLE) & 0xFF)
if sum8 == chk:
kind = "SUM8"
elif dle_aware == chk:
kind = "DLE-aware"
else:
kind = "BAD"
yield i, payload, chk, kind
i = end + 1
def describe(payload: bytes, is_request: bool) -> str:
if len(payload) < 3:
return "??"
sub = payload[2]
if is_request:
return SUBNAME.get(sub, f"SUB_{sub:02X}")
req = 0xFF - sub
return "rsp<-" + SUBNAME.get(req, f"SUB_{req:02X}")
def report(path: Path, *, is_request: bool, dump_sub: int | None) -> None:
blob = path.read_bytes()
side = "Thor" if is_request else "unit"
print(f"== {side:4} {path.name} ({len(blob)} bytes)")
n_bad = 0
for idx, (off, p, chk, kind) in enumerate(frames(blob, is_request=is_request)):
if kind == "BAD":
n_bad += 1
sub = p[2] if len(p) > 2 else -1
flags = p[1] if len(p) > 1 else -1
# Requests carry offset at payload[4:6]; responses page at [3:5].
word = int.from_bytes(p[4:6] if is_request else p[3:5], "big")
data = len(p) - 16 if is_request else max(len(p) - 5, 0)
print(
f" [{idx:2}] @0x{off:04x} payload={len(p):5} data={data:5} "
f"flags=0x{flags:02x} SUB=0x{sub:02x} {describe(p, is_request):18} "
f"{'offset' if is_request else 'page'}=0x{word:04x} chk={kind}"
)
if dump_sub is not None and sub == dump_sub:
body = p[16:] if is_request else p[5:]
print(f" ---- data ({len(body)} bytes) ----")
for o in range(0, len(body), 16):
chunk = body[o : o + 16]
txt = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk)
print(f" {o:06x} {chunk.hex(' '):<47} |{txt}|")
print(f" -- {idx + 1} frames, {n_bad} bad checksum\n")
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("paths", nargs="+",
help="a capture directory, or raw_bw.bin and raw_s3.bin")
ap.add_argument("--dump", default=None,
help="hex-dump the data section of this SUB (e.g. 0x71)")
args = ap.parse_args()
dump_sub = int(args.dump, 0) if args.dump else None
if len(args.paths) == 1 and Path(args.paths[0]).is_dir():
d = Path(args.paths[0])
bw = sorted(d.glob("raw_bw_*.bin"))
s3 = sorted(d.glob("raw_s3_*.bin"))
if not bw or not s3:
print(f"{d}: need one raw_bw_*.bin and one raw_s3_*.bin", file=sys.stderr)
return 2
pairs = [(bw[0], True), (s3[0], False)]
elif len(args.paths) == 2:
pairs = [(Path(args.paths[0]), True), (Path(args.paths[1]), False)]
else:
ap.error("pass a capture directory, or exactly two .bin files")
for path, is_request in pairs:
report(path, is_request=is_request, dump_sub=dump_sub)
return 0
if __name__ == "__main__":
sys.exit(main())
-119
View File
@@ -1,119 +0,0 @@
#!/usr/bin/env python3
"""
socat_log_split.py — recover a capture pair from a `socat -x` relay log.
Why this exists
---------------
The bench relay that puts Thor in front of a USB-attached Micromate is:
socat -d -d -x TCP-LISTEN:12345,reuseaddr,fork /dev/ttyACM0,raw,echo=0,b115200 \
> ~/mm-captures/socat_<ts>.log 2>&1
`-x` makes socat hex-dump every byte it forwards, in both directions, with
timestamps. That log is therefore a **complete second copy of every capture**
taken through the relay — independent of whether seismo_lab was recording.
On 2026-09-25 that mattered: a capture's `.bin` files never made it off the
Windows machine, and the session was rebuilt from this log instead. When the
real bins turned up later, the reconstruction was **byte-for-byte identical in
both directions** (3,595 and 4,004 bytes). So this is a validated fallback, not
a lossy approximation.
Log format
----------
```
> 2026/09/25 00:30:35.000276659 length=21 from=0 to=20
41 02 10 10 00 5b 00 00 30 00 ...
2026/09/25 00:30:35 socat[32190] N write(5, 0x..., 21) completed
< 2026/09/25 00:30:35.000384100 length=64 from=0 to=63
02 00 c5 a4 00 00 30 00 ...
```
`>` is data heading toward the serial device (Thor → unit). `<` is data coming
back (unit → Thor). Hex lines are space-separated and indented; socat's own
status lines start with a date and carry no payload.
Usage
-----
# whole log
python scratch/socat_log_split.py socat_20260924_181248.log --out-dir ./recovered
# one session — line numbers from the "accepting connection" markers
grep -n "accepting connection" socat_*.log
python scratch/socat_log_split.py socat_*.log --from-line 919 --out-dir ./recovered
Then parse the result as usual:
python scratch/mm_frame_parse.py recovered/raw_bw.bin recovered/raw_s3.bin
⚠ A log spanning several sessions concatenates them. Split by line number using
the `accepting connection` markers, or the frame walk will run sessions together.
"""
from __future__ import annotations
import argparse
import re
from pathlib import Path
_HEX = re.compile(r"\A[0-9a-f]{2}\Z")
_SOCAT_STATUS = re.compile(r"\A\d{4}/\d{2}/\d{2}")
def split(lines) -> tuple[bytes, bytes]:
"""Return (to_device, from_device) byte streams."""
to_dev, from_dev = bytearray(), bytearray()
cur = None
for line in lines:
if line.startswith(">"):
cur = to_dev
continue
if line.startswith("<"):
cur = from_dev
continue
if _SOCAT_STATUS.match(line):
# socat's own status line ends the current dump block.
cur = None
continue
if cur is None or not line.startswith(" "):
continue
toks = line.split()
if toks and all(_HEX.match(t) for t in toks):
cur.extend(int(t, 16) for t in toks)
return bytes(to_dev), bytes(from_dev)
def main() -> int:
ap = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
ap.add_argument("log", help="a socat -x log file")
ap.add_argument("--out-dir", default=".", help="where to write the .bin pair")
ap.add_argument("--from-line", type=int, default=1,
help="first log line to read (1-based) — use the "
"'accepting connection' marker of the session you want")
ap.add_argument("--to-line", type=int, default=None,
help="last log line to read (1-based, inclusive)")
ap.add_argument("--prefix", default="raw", help="output basename prefix")
args = ap.parse_args()
lines = Path(args.log).read_text(errors="replace").splitlines()
lo = max(args.from_line - 1, 0)
hi = args.to_line if args.to_line is not None else len(lines)
to_dev, from_dev = split(lines[lo:hi])
out = Path(args.out_dir)
out.mkdir(parents=True, exist_ok=True)
bw = out / f"{args.prefix}_bw.bin"
s3 = out / f"{args.prefix}_s3.bin"
bw.write_bytes(to_dev)
s3.write_bytes(from_dev)
print(f"Thor -> unit {len(to_dev):>7} bytes {bw}")
print(f"unit -> Thor {len(from_dev):>7} bytes {s3}")
if not to_dev or not from_dev:
print("⚠ one direction is empty — check --from-line / --to-line")
return 0
if __name__ == "__main__":
raise SystemExit(main())