Merge pull request 'Release v0.31.0 — report parity + the inverted rescue (0.29.0 → 0.31.0)' (#40 ) from dev into main

Reviewed-on: #40
Merge pull request 'Release v0.29.0 — offset detector + false_trigger_reason + BlastMate serials (0.27.0→0.29.0)' (#36 ) from dev into main
2026-09-18 16:46:41 -04:00 · 2026-09-07 15:57:37 -04:00 · 2026-08-29 15:48:33 -04:00 · 2026-08-28 22:40:34 -04:00 · 2026-08-27 13:43:00 -04:00 · 2026-08-06 12:12:19 -04:00
15 changed files with 24 additions and 4667 deletions
@@ -4,68 +4,6 @@ All notable changes to seismo-relay are documented here.
 ---
 ## Unreleased
 ### Fixed
 - **Waveform event times were the monitoring-session start, not the trigger
  (~hours off).**  `read_blastware_file` stamped events with footer `ts1`, which
  for a waveform is the session start a unit shares across every event that day
  (a unit arming at 06:00 stamped 06:00 on all of them — the modal and PDF both
  showed it, since it's the stored value).  The event time is footer `ts2` (the
  recording stop), and Blastware's trigger = `ts2 - record time`.  The record
  time is a big-endian float32 in the recording-setup config block (30 bytes
  before the `Standard Recording Setup` marker), so the **exact trigger is now
  recovered from the binary alone** — all 7 BE12844 oracle events decode to
  their exact Blastware time (e.g. N844LQHB 10:33:29), no paired `.TXT` needed.
  Histograms keep `ts1` (the ~24 h window start).  A paired report's
  `event_datetime` stays authoritative (unit-clock drift).
  ⚠ **Needs a re-decode backfill** to correct existing stored events' timestamps.
 ### Added
 - **Diagnostics tab in the SFM standalone webapp.**  Surfaces the device
  endpoints that previously existed only as `curl`: `events/storage_range` and
  `events/index` alongside `monitor/status`, then stop monitoring, disable ACH
  (`rescue?erase=false`, so stored events survive), and erase.  The wedged-unit
  ladder — slow drip and blind stop — sits under its own heading pointing at
  `docs/runbooks/wedged_unit_recovery.md`, with the reminder that `slow_drip`'s
  success signal is `bytes_received > 0` and not a clean duration.  Erase is
  guarded by typing the unit's serial: auth answers *who*, not *did you mean
  it*, and Swagger's try-it-out button on `/device/events/erase` is live on
  `:8200/docs`.
 - **`docs/sfm_tool_status.md`** — an honest per-capability maturity assessment:
  what is production-grade (the codec library, the data side), what is
  emergency-grade (the device side), what is a research artifact, the
  known-issues table, and the gap to a real tool.  Also records the **5A
  page-boundary bug** as known: `parse_strt_end_offset()` discards the key's
  page byte, so once a unit has recorded more than 64 KB since its last erase,
  an event spanning the boundary reads an `end_offset` *behind* its own start —
  the chunk loop fetches nothing and TERM packs a negative `offset_word`, which
  500s.  Reproduced on BE12599.  Production is unaffected: it ingests complete
  files via the watcher path and never runs this walk.
 ### Changed
 - **Connecting to a unit no longer walks its event chain.**  `/device/events`
  reads every event header over the cellular link; on a unit with a large or
  wrapped chain that takes minutes or fails outright, and it fired
  automatically on every connect.  Connect now uses only ~2 s probes —
  `/device/info` (which already carried the compliance config the walk was
  re-reading) plus `events/storage_range` — and the Device tab gains an Event
  Chain card.  The walk moved behind a **Load events** button in the Events
  toolbar.  Knowing whether a unit's ACH is on no longer requires reading every
  event it has stored.
 ### Migration
 **None.**  Frontend and documentation only — no codec, waveform-store or DB
 change, no schema change, and no `TOOL_VERSION` bump.  The webapp is served
 from the image, so the change appears after the next `sfm` rebuild.
 ---
 ## v0.31.0 — 2026-09-18
 **Report parity, and a second way to rescue a runaway unit.**  Two threads.
@@ -10,38 +10,10 @@ pair — lives in `../terra-view/docs/tmi-stack.md`, which is also loaded as
 ---
-## Where things stand (updated 2026-09-26)
+## Where things stand (updated 2026-08-28)
 Read this first when picking the project back up.
 - **The Series-4 LIVE wire protocol is reverse-engineered end to end
  (2026-09-25).** `docs/micromate_protocol_reference.md` is the Series-4
  Rosetta Stone, sibling to `instantel_protocol_reference.md`. **A Micromate
  answers Series III command frames** — three framing differences: responses
  have **no leading `DLE`** (a bare `STX`), `payload[1]` is `0xC5` (Blastware
  firmware) or `0x03` (Thor firmware) rather than `0x10`, and the data length
  is a **uint16 BE at `payload[8:10]`** (as a byte it under-reads `SUB 0x1A`
  by 47x). Read path, event chain, setups, scheduler, monitoring control and
  per-event delete are all mapped; **the inbound call-home session is the only
  protocol unknown left.**
  ⚠ **No command has ever been originated against a unit by this project.**
  Every write was performed by THOR while we recorded. That line is worth
  keeping.
  ⚠ `micromate/` still has **no live client** — it is codec-only. The
  `minimateplus/` stack (transport/framing/protocol/client) has no Series-4
  counterpart yet. `minimateplus.transport` is protocol-agnostic and reusable.
 - **Bench tooling for device diagnosis (2026-09-25).** `bridges/mm_probe.py`
  distinguishes the four faults THOR reports identically as "disconnected"
  (refused / connect timeout / **connected but no reply** / replied) and names
  what to try next. `bridges/mm_link.py` is a stand-in for a cellular modem
  with a decoded log and fault injection. `scratch/mm_frame_parse.py` exists
  because **`S3FrameParser` cannot see Micromate responses at all** — it scans
  for `DLE+STX`, which never appears in Series-4 traffic.
 - **A Micromate's USB-A host port drives FTDI and CDC-ACM only** — no Prolific,
  in either firmware line. TMI buys both Sabrent (FTDI) and Benfei (PL2303)
  cables and they are indistinguishable by eye. A PL2303 cable leaves a unit
  with **no working modem port at all**; identify by `lsusb` VID, `0403` vs
  `067b`. This accounted for a unit that could not be deployed.
 - **Series-3 decode is verified per-sample at scale (v0.27.0).** The full DL2
  archive decodes **14,338 / 14,338** paired files exactly against their
  preserved Blastware ASCII exports — 1,249 waveform + 13,089 histogram, 45
@@ -89,24 +61,6 @@ Read this first when picking the project back up.
  4th-decimal tick and are Thor's own rounding — no single linear LSB can
  reproduce every printed value (the constraints are infeasible by 7e-5
  relative), so do NOT retune `_GEO_LSB_IPS`.
 - **⚠ KNOWN BUG — the 5A walk breaks once a unit's buffer crosses 64 KB.**
  `parse_strt_end_offset()` returns only `(end_key[2] << 8) | end_key[3]`,
  discarding the key's page byte.  An event starting at `0x0111F2A2` and ending
  at `0x0112_1010` therefore reads `end_offset = 0x1010` — *behind* its own
  start.  The chunk loop then exits before fetching anything and TERM computes
  a negative `offset_word`, which `struct.pack(">H", ...)` rejects: the
  `/device/events` walk 500s.  Reproduced on BE12599 (2026-09-19), which had
  78 KB stored and had rolled into page `0x12`.
  **Why it hid so long:** every 5A capture the walk was verified against came
  from a freshly-erased BE11529 — all three confirmed TERM examples in
  `framing.py` (`0x1ABE`, `0x21F2`, `0x417E`) sit inside page `0x11`.  Prod is
  unaffected: it ingests complete files via BW ACH, never this walk.
  **Fixing it has two layers** — the arithmetic (`if end < start: end +=
  0x10000`) stops the crash and bounds the loop correctly; carrying the page
  byte through the chunk requests (`params[1]` 0x11 -> 0x12, counter rolling
  over) needs a BW capture of a spanning event first.  Do not ship layer one
  alone without a loud truncation warning — a silently short event is the
  failure mode this codec has been bitten by repeatedly.
 - **Open, not blocking:** 14 sensitive-range files show an exact 8x
  (= 10.0/1.25) units discrepancy; `scripts/backfill_sidecars.py --force` also
  inserts DB rows for store files that have none (one-time per store) and the
@@ -120,9 +74,6 @@ Read this first when picking the project back up.
  **v0.27.0 does NOT owe prod a backfill** — verified: the partial-final-block
  fix changes 0 of the 10,215 histograms in the prod store (the 4 recovered
  files are archive-only and were never ingested).
  ✅ **The v0.30.0 Series-4 backfill HAS been run on prod (2026-09-25).** Every
  stored Series-4 geophone value was ~3.3% low until then; that is corrected and
  the job does not need repeating.
 - **The "offset" hardware fault has its own journal** --
  `docs/offset_investigation.md`. **5 of 45 units (11%)**, and the fault is
  **persistent** — it stays until the geophone is serviced. Detect it with
@@ -134,18 +85,7 @@ Read this first when picking the project back up.
  `SUB 0x0E` (unimplemented), which may carry those very numbers.
-When new information about a protocol is discovered, record it in the matching
+When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
 reference **in addition to** this document:
 | series | document |
 |---|---|
 | Series III (MiniMate Plus / BlastMate) | `docs/instantel_protocol_reference.md` |
 | **Series IV (Micromate / THOR)** | **`docs/micromate_protocol_reference.md`** |
 | Thor IDF file format | `docs/idf_protocol_reference.md` |
 Both protocol references carry retractions in place rather than deleting what
 turned out to be wrong — that convention has already saved re-deriving the same
 mistakes twice, so keep it.
 ---
@@ -318,15 +258,8 @@ minimateplus/         ← Python client library (primary focus)
 sfm/server.py         ← FastAPI REST server exposing device data over HTTP
 seismo_lab.py         ← Tkinter GUI (Bridge + Analyzer + Console tabs)
 bridges/
  mm_probe.py         ←   name the fault behind a dead unit (4 verdicts, read-only)
  mm_link.py          ←   bench stand-in for a cellular modem, with fault injection
  ach_mitm.py         ←   TCP relay for recording a Series-3 ACH session
 docs/
-  instantel_protocol_reference.md  ← Series III protocol spec ("the Rosetta Stone")
+  instantel_protocol_reference.md  ← reverse-engineered protocol spec ("the Rosetta Stone")
  micromate_protocol_reference.md  ← Series IV protocol spec + THOR's measured behaviour
  idf_protocol_reference.md        ← Thor IDF file format
 CHANGELOG.md          ← version history
 ```
@@ -496,11 +496,6 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
 ## Roadmap (Future)
 > **Where it stands *today*** — an honest per-capability maturity assessment,
 > what to rely on, known issues, and the gap to a real tool:
 > [`docs/sfm_tool_status.md`](docs/sfm_tool_status.md).  This section covers
 > where it is *going*.
 ### Strategic direction — where this is going
 seismo-relay is being built as a **suite of cooperating components**
@@ -1,338 +0,0 @@
 #!/usr/bin/env python3
 """
 mm_link.py — a "perfect modem" between THOR and a Micromate, with a readable
 log and deliberate fault injection.
 Why
 ---
 THOR gives almost no visibility into a connection: a refresh button, two poll
 intervals, and no way to see whether a check succeeded, timed out, or was never
 sent.  When a unit "won't stay connected" there is nothing to look at.
 This sits where the cellular modem would sit and answers the question directly:
  * **What is THOR actually doing?**  Every frame is decoded and timestamped —
    `POLL`, `MONITOR_STATUS`, `SETUP_NAME_READ` — not a hex dump.
  * **Is it even trying?**  Silence is visible: the log shows gaps.
  * **How does it behave when the link misbehaves?**  Faults can be injected on
    demand, which a real cell link will not do on cue.
 Point THOR at this host and port exactly as if it were a modem (Communication:
 TCP, IP: <this host>, Port: <--listen>).
 Fault injection
 ---------------
 Write a mode into the control file (default `mm_link.ctl`) and it takes effect
 on the next byte:
    echo pass      > mm_link.ctl    # normal relay
    echo blackhole > mm_link.ctl    # TCP stays up, bytes are swallowed
    echo drop      > mm_link.ctl    # close the connection abruptly (RST-ish)
    echo delay:2.0 > mm_link.ctl    # forward, but 2 s late in both directions
    echo onewaydev > mm_link.ctl    # THOR->unit passes, unit->THOR is swallowed
 **`blackhole` is the one that matters.**  It reproduces the classic cellular
 failure: the socket is still open as far as both ends are concerned, but nothing
 crosses.  A client that relies on TCP to tell it the peer is gone will sit there
 until the OS keepalive fires — which by default is about two hours.
 Usage
 -----
    python3 bridges/mm_link.py --serial /dev/ttyACM0 --baud 115200 \\
        --listen 12345 --logdir ~/mm-captures
 Writes, per session:
    <logdir>/mmlink_<ts>/session.log   decoded, timestamped, human-readable
    <logdir>/mmlink_<ts>/raw_bw.bin    THOR -> unit, raw
    <logdir>/mmlink_<ts>/raw_s3.bin    unit -> THOR, raw
 The raw pair loads straight into `scratch/mm_frame_parse.py`.
 """
 from __future__ import annotations
 import argparse
 import datetime
 import os
 import socket
 import sys
 import threading
 import time
 from pathlib import Path
 sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "scratch"))
 try:
    from mm_frame_parse import SUBNAME, destuff  # noqa: F401
 except Exception:                                 # pragma: no cover
    SUBNAME = {}
 import errno
 import select
 import termios
 DLE, STX, ETX, ACK = 0x10, 0x02, 0x03, 0x41
 _BAUD = {9600: termios.B9600, 19200: termios.B19200, 38400: termios.B38400,
         57600: termios.B57600, 115200: termios.B115200}
 class SerialPort:
    """Minimal raw serial port on stdlib termios — no pyserial dependency.
    The bench hosts are whatever is to hand; requiring a pip install on someone
    else's machine is a poor trade for the ~30 lines this saves.
    """
    def __init__(self, path: str, baud: int):
        if baud not in _BAUD:
            raise ValueError(f"unsupported baud {baud}; pick one of {sorted(_BAUD)}")
        self.fd = os.open(path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
        a = termios.tcgetattr(self.fd)
        a[0] = 0                                             # iflag: no translation
        a[1] = 0                                             # oflag: raw
        a[2] = termios.CS8 | termios.CREAD | termios.CLOCAL  # cflag: 8N1, ignore modem lines
        a[3] = 0                                             # lflag: non-canonical, no echo
        a[4] = a[5] = _BAUD[baud]
        a[6] = list(a[6])
        a[6][termios.VMIN] = 0
        a[6][termios.VTIME] = 0
        termios.tcsetattr(self.fd, termios.TCSANOW, a)
        termios.tcflush(self.fd, termios.TCIOFLUSH)
    def read(self, n: int) -> bytes:
        r, _, _ = select.select([self.fd], [], [], 0.2)
        if not r:
            return b""
        try:
            return os.read(self.fd, n)
        except OSError as e:
            if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK):
                return b""
            raise
    def write(self, data: bytes) -> None:
        while data:
            try:
                data = data[os.write(self.fd, data):]
            except OSError as e:
                if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK):
                    select.select([], [self.fd], [], 0.2)
                    continue
                raise
    def close(self) -> None:
        try:
            os.close(self.fd)
        except OSError:
            pass
 def name_of(sub: int, is_request: bool) -> str:
    if is_request:
        return SUBNAME.get(sub, f"SUB_{sub:02X}")
    return "rsp " + SUBNAME.get(0xFF - sub, f"SUB_{0xFF - sub:02X}")
 class FrameSniffer:
    """Accumulate bytes and report complete frames, without altering the stream."""
    def __init__(self, is_request: bool):
        self.is_request = is_request
        self.buf = bytearray()
    def feed(self, data: bytes):
        """Yield (sub, payload_len) for each complete frame seen."""
        self.buf.extend(data)
        while True:
            start = -1
            for i, b in enumerate(self.buf):
                if self.is_request and b == ACK and i + 1 < len(self.buf) and self.buf[i + 1] == STX:
                    start = i
                    break
                if not self.is_request and b == STX:
                    start = i
                    break
            if start < 0:
                if len(self.buf) > 8192:
                    del self.buf[:-16]
                return
            j = start + (2 if self.is_request else 1)
            end = -1
            while j < len(self.buf):
                if self.buf[j] == DLE and j + 1 < len(self.buf):
                    j += 2
                    continue
                if self.buf[j] == ETX:
                    end = j
                    break
                j += 1
            if end < 0:
                return                      # wait for more bytes
            body = self.buf[start:end + 1]
            del self.buf[:end + 1]
            # SUB sits at a fixed spot past the leading framing -- but it is
            # DLE-escaped when its own value is 0x02/0x03/0x04/0x10, so a raw
            # read reports 0x10 for those.  SUB 0x02 was being logged as
            # "SUB_10" until this was handled.
            off = 5 if self.is_request else 3
            if len(body) > off:
                sub = body[off]
                if sub == DLE and len(body) > off + 1:
                    sub = body[off + 1]
                yield sub, len(body)
 class Link:
    def __init__(self, args):
        self.args = args
        self.mode = "pass"
        self.delay = 0.0
        self.ctl = Path(args.control)
        self.session: Path | None = None
        self.log_fh = None
        self.raw = {}
        self.t0 = time.time()
        self.counts = {}
    # ── logging ────────────────────────────────────────────────────────────
    def open_session(self):
        ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
        self.session = Path(self.args.logdir) / f"mmlink_{ts}"
        self.session.mkdir(parents=True, exist_ok=True)
        self.log_fh = open(self.session / "session.log", "a", buffering=1)
        self.raw = {
            "bw": open(self.session / "raw_bw.bin", "ab"),
            "s3": open(self.session / "raw_s3.bin", "ab"),
        }
        self.say(f"=== session {ts} — serial {self.args.serial} @ {self.args.baud} ===")
    def say(self, text: str):
        line = f"{datetime.datetime.now().strftime('%H:%M:%S.%f')[:-3]}  {text}"
        print(line, flush=True)
        if self.log_fh:
            self.log_fh.write(line + "\n")
    # ── control file ───────────────────────────────────────────────────────
    def poll_control(self):
        while True:
            try:
                if self.ctl.exists():
                    want = self.ctl.read_text().strip().lower()
                    if want.startswith("delay:"):
                        d = float(want.split(":", 1)[1])
                        if ("delay", d) != (self.mode, self.delay):
                            self.mode, self.delay = "delay", d
                            self.say(f"*** MODE -> delay {d}s ***")
                    elif want and want != self.mode:
                        self.mode, self.delay = want, 0.0
                        self.say(f"*** MODE -> {want} ***")
            except Exception:
                pass
            time.sleep(0.25)
    # ── the relay ──────────────────────────────────────────────────────────
    def pump(self, src, dst, tag: str, is_request: bool, stop: threading.Event):
        sniff = FrameSniffer(is_request)
        arrow = "THOR->unit" if is_request else "unit->THOR"
        last = time.time()
        while not stop.is_set():
            timed_out = False
            try:
                data = src.recv(4096) if isinstance(src, socket.socket) else src.read(4096)
            except TimeoutError:
                timed_out = True
                # socket.timeout subclasses OSError, so it MUST be caught first.
                # Treating it as a dead socket closes the connection after 200 ms
                # of quiet -- which is exactly what `blackhole` produces, so the
                # relay killed the link it was supposed to be faking a fault on.
                data = b""
            except OSError:
                break
            if isinstance(src, socket.socket) and data == b"" and not timed_out:
                self.say(f"{arrow}: peer closed the connection")
                break
            if not data:
                if time.time() - last > self.args.quiet_after and self.counts:
                    self.say(f"--- {self.args.quiet_after:.0f}s with no traffic ---")
                    last = time.time()
                continue
            last = time.time()
            self.raw[tag].write(data)
            self.raw[tag].flush()
            for sub, ln in sniff.feed(data):
                label = name_of(sub, is_request)
                self.counts[label] = self.counts.get(label, 0) + 1
                self.say(f"{arrow}  {label:<20} ({ln} B)"
                         + ("" if self.mode == "pass" else f"   [mode={self.mode}]"))
            mode = self.mode
            if mode == "drop":
                self.say(f"{arrow}: DROPPING the connection (fault injection)")
                stop.set()
                break
            if mode == "blackhole":
                continue                      # swallow, keep the socket open
            if mode == "onewaydev" and not is_request:
                continue                      # unit's replies never reach THOR
            if mode == "delay" and self.delay:
                time.sleep(self.delay)
            try:
                if isinstance(dst, socket.socket):
                    dst.sendall(data)
                else:
                    dst.write(data)
            except OSError:
                break
        stop.set()
    def serve(self):
        srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
        srv.bind(("0.0.0.0", self.args.listen))
        srv.listen(5)
        self.open_session()
        self.say(f"listening on 0.0.0.0:{self.args.listen}   control file: {self.ctl}")
        self.say("point THOR at this host/port as Communication=TCP")
        threading.Thread(target=self.poll_control, daemon=True).start()
        while True:
            conn, addr = srv.accept()
            conn.settimeout(0.2)
            self.say(f"+++ THOR connected from {addr[0]}:{addr[1]} +++")
            try:
                ser = SerialPort(self.args.serial, self.args.baud)
            except OSError as e:
                self.say(f"!!! cannot open {self.args.serial}: {e}")
                conn.close()
                continue
            stop = threading.Event()
            ts = [
                threading.Thread(target=self.pump, args=(conn, ser, "bw", True, stop), daemon=True),
                threading.Thread(target=self.pump, args=(ser, conn, "s3", False, stop), daemon=True),
            ]
            for t in ts:
                t.start()
            for t in ts:
                t.join()
            conn.close()
            ser.close()
            summary = ", ".join(f"{k}x{v}" for k, v in sorted(self.counts.items()))
            self.say(f"--- connection closed.  frames this session: {summary or 'none'} ---")
 def main():
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("--serial", default="/dev/ttyACM0")
    ap.add_argument("--baud", type=int, default=115200)
    ap.add_argument("--listen", type=int, default=12345)
    ap.add_argument("--logdir", default=os.path.expanduser("~/mm-captures"))
    ap.add_argument("--control", default="mm_link.ctl")
    ap.add_argument("--quiet-after", type=float, default=30.0,
                    help="log a marker after this many seconds of silence")
    Link(ap.parse_args()).serve()
 if __name__ == "__main__":
    main()
@@ -1,239 +0,0 @@
 #!/usr/bin/env python3
 """
 mm_probe.py — answer "why can't we reach this unit?" in one command.
 THOR reports a failed connection as "disconnected" and nothing else.  That single
 word covers at least four completely different faults with four different fixes,
 and telling them apart is the difference between a modem reboot and a site visit:
  * **connection refused**   something answered and said no — wrong port, or the
                             modem is refusing a further session
  * **connect timed out**    nothing answered at all — trusted-IP whitelist,
                             firewall, or the modem is off the network
  * **connected, no reply**  the MODEM answered but the unit did not.  The TCP
                             path is fine; the modem is not forwarding to serial.
                             This is the signature of a wedged transparent-TCP
                             session, and it is the one THOR cannot distinguish
                             from any of the others
  * **replied**              the unit is alive; the problem is upstream software
 Read-only.  It sends `POLL`, then optionally `SERIAL` and the state read — the
 same three commands THOR's own connection check uses — and never writes.
 Usage
 -----
    python3 bridges/mm_probe.py 63.45.161.30:9034
    python3 bridges/mm_probe.py 10.0.0.8:12345 --timeout 5
    python3 bridges/mm_probe.py <host:port> --slots 3
 `--slots N` opens N connections at once and reports how many the far end accepts.
 A transparent-TCP modem typically serves **one** session; if the first succeeds
 and the rest are refused or hang, that confirms the single-slot behaviour and
 explains why a leaked session takes a unit offline until the slot frees.
 Works for both series: a Series III reply opens `DLE STX`, a Micromate reply
 opens with a bare `STX`, so the probe also tells you which one answered.
 """
 from __future__ import annotations
 import argparse
 import socket
 import sys
 import time
 from pathlib import Path
 sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
 from minimateplus.framing import build_bw_frame  # noqa: E402
 DLE, STX, ETX = 0x10, 0x02, 0x03
 def destuff(raw: bytes) -> bytes:
    """Strip framing and DLE escapes; return the payload without its checksum."""
    i = 1 if raw and raw[0] == STX else (2 if len(raw) > 1 and raw[1] == STX else 0)
    out = bytearray()
    while i < len(raw):
        b = raw[i]
        if b == DLE and i + 1 < len(raw):
            out.append(raw[i + 1])
            i += 2
            continue
        if b == ETX:
            break
        out.append(b)
        i += 1
    return bytes(out[:-1]) if len(out) > 1 else b""
 # Reads are two-step on Series III: a probe at offset 0, then a data read at the
 # block's length.  THOR sends these offsets, and they also work on a Micromate.
 OFFSETS = {0x5B: 0x0030, 0x15: 0x000A, 0x49: 0xFFFF}
 def exchange(sock: socket.socket, sub: int, timeout: float) -> tuple[bytes, float]:
    sock.sendall(build_bw_frame(sub, OFFSETS.get(sub, 0)))
    t0 = time.time()
    buf, deadline = b"", t0 + timeout
    sock.settimeout(0.3)
    while time.time() < deadline:
        try:
            chunk = sock.recv(4096)
            if not chunk:
                break
            buf += chunk
            if buf.endswith(bytes([ETX])) and len(buf) > 8:
                break
        except TimeoutError:
            continue
        except OSError:
            break
    return buf, time.time() - t0
 def step(n: int, label: str, result: str) -> None:
    print(f"  [{n}] {label:.<28} {result}")
 def probe(host: str, port: int, timeout: float) -> int:
    print(f"\ntarget {host}:{port}   (read-only: POLL, SERIAL, state)\n")
    # ── 1. TCP ────────────────────────────────────────────────────────────
    t0 = time.time()
    try:
        sock = socket.create_connection((host, port), timeout=timeout)
    except ConnectionRefusedError:
        step(1, "TCP connect", f"REFUSED after {1000*(time.time()-t0):.0f} ms")
        print("\nverdict: something answered and actively refused.")
        print("  Not a silent firewall drop — the host is reachable.")
        print("  Wrong port, the service is down, or the modem is refusing")
        print("  an additional session because its one slot is in use.")
        return 2
    except (TimeoutError, socket.timeout):
        step(1, "TCP connect", f"TIMED OUT after {time.time()-t0:.1f} s")
        print("\nverdict: nothing answered at all.")
        print("  A silent drop, which is what a trusted-IP whitelist looks")
        print("  like — it discards rather than refuses.  Check the modem's")
        print("  Trusted IPs (and note a VPN changes the IP you arrive from),")
        print("  the firewall, and whether the modem is on the network.")
        return 3
    except OSError as e:
        step(1, "TCP connect", f"FAILED: {e}")
        return 4
    step(1, "TCP connect", f"ok ({1000*(time.time()-t0):.0f} ms)")
    # ── 2. POLL ───────────────────────────────────────────────────────────
    try:
        raw, dt = exchange(sock, 0x5B, timeout)
    except OSError as e:
        step(2, "POLL", f"send failed: {e}")
        sock.close()
        return 4
    if not raw:
        step(2, "POLL", f"NO REPLY in {timeout:.1f} s")
        print("\nverdict: the MODEM answered but the unit did not.")
        print("  TCP is fine end to end — something accepted the connection.")
        print("  What is missing is the serial side.  Two quite different")
        print("  causes produce this, and they are NOT distinguishable from")
        print("  here:")
        print("\n  1. SOMEONE ELSE HOLDS THE SESSION.  These modems bridge ONE")
        print("     TCP session to serial at a time.  A second connection is")
        print("     accepted and then simply not forwarded.  Confirmed 2026-09-26:")
        print("     with THOR connected this probe saw exactly this; the moment")
        print("     THOR disconnected the same probe returned the serial number.")
        print("     ** Check whether THOR (or anything else) has the unit first. **")
        print("\n  2. The serial path is genuinely broken — a stale session the")
        print("     modem never released, a cable the unit cannot enumerate, or")
        print("     a unit that is off.")
        print("\n  Try, in order:")
        print("    1. Disconnect any other client and re-probe.  If it answers,")
        print("       it was contention, not a fault.")
        print("    2. The cable's chipset.  A Micromate drives FTDI and CDC-ACM")
        print("       only — a Prolific PL2303 gives it no serial port at all.")
        print("       lsusb: FTDI is 0403, Prolific 067b.")
        print("    3. Power-cycle the UNIT with the cable attached (hold power")
        print("       5 s, through the two-stage prompt).  Its USB host rescans")
        print("       on cold boot; it may not on hot-swap.")
        print("    4. AirLink OS -> TCP Idle Timeout.  If 0/disabled, a stale")
        print("       session holds the slot indefinitely.  2 minutes is the")
        print("       value this project standardised on.")
        sock.close()
        return 5
    series = "Series III (DLE STX)" if raw[0] == DLE else "Micromate (bare STX)"
    step(2, "POLL", f"reply {len(raw)} B in {1000*dt:.0f} ms")
    p = destuff(raw)
    ok = len(p) > 3 and p[2] == 0xFF - 0x5B
    step(3, "frame", f"{'valid' if ok else 'MALFORMED'}, {series}")
    if not ok:
        print("\nverdict: something replied, but not a seismograph.")
        print("  Another service is on this port, or the modem is in a mode")
        print("  that injects its own text (check Quiet Mode / AT echo).")
        print(f"  first bytes: {raw[:16].hex(' ')}")
        sock.close()
        return 6
    # ── 3. identity + state ───────────────────────────────────────────────
    for n, (sub, label) in enumerate(((0x15, "serial"), (0x49, "state")), start=4):
        try:
            r, dt = exchange(sock, sub, timeout)
            d = destuff(r)[5:]
            if sub == 0x15:
                # serial is a null-terminated run; a further field follows it
                serial = bytes(d[11:]).split(b"\x00")[0]
                step(n, label, serial.decode("ascii", "replace") or "(empty)")
            else:
                step(n, label, "MONITORING" if len(d) > 11 and d[11] else "idle")
        except OSError:
            step(n, label, "no reply")
    sock.close()
    print("\nverdict: the unit is alive and answering.")
    print("  If THOR still shows it disconnected, the fault is in THOR, not")
    print("  the network or the device.")
    return 0
 def slots(host: str, port: int, n: int, timeout: float) -> None:
    print(f"\nopening {n} simultaneous connections to {host}:{port}\n")
    held = []
    for i in range(n):
        try:
            s = socket.create_connection((host, port), timeout=timeout)
            held.append(s)
            step(i + 1, f"connection {i+1}", "accepted")
        except ConnectionRefusedError:
            step(i + 1, f"connection {i+1}", "REFUSED")
        except (TimeoutError, socket.timeout):
            step(i + 1, f"connection {i+1}", "timed out")
        except OSError as e:
            step(i + 1, f"connection {i+1}", f"failed: {e}")
    print(f"\n{len(held)} of {n} accepted.")
    if len(held) == 1:
        print("  Single-slot behaviour confirmed — this far end serves ONE")
        print("  session at a time.  A connection that is never closed takes")
        print("  the unit offline until the idle timeout frees the slot.")
    for s in held:
        s.close()
 def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("target", help="host:port, e.g. 63.45.161.30:9034")
    ap.add_argument("--timeout", type=float, default=10.0)
    ap.add_argument("--slots", type=int, metavar="N",
                    help="open N simultaneous connections to test single-slot behaviour")
    a = ap.parse_args()
    host, _, port = a.target.rpartition(":")
    if not host:
        ap.error("target must be host:port")
    if a.slots:
        slots(host, int(port), a.slots, a.timeout)
        return 0
    return probe(host, int(port), a.timeout)
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -1,291 +0,0 @@
 # Spec — a live client for Series IV (Micromate)
 Drafted 2026-09-26, ahead of implementation.  The protocol work is finished; this
 is the plan for turning `docs/micromate_protocol_reference.md` into code SFM can
 run.
 **Read that document first.**  Everything here assumes it, and every constant
 below is sourced from it rather than restated with justification.
 ---
 ## Goal and scope
 `micromate/` is codec-only today — `idf_file.py`, `models.py`, the report
 writers.  There is no way to talk to a unit.  This adds the live half, mirroring
 `minimateplus/`.
 **In scope, first pass:**
 - connect over TCP (a field modem) or serial/USB (a bench unit)
 - identify a unit, read its state, clock, memory and setups
 - walk the event chain and download events
 - return `Event` objects the existing codec already understands
 **Explicitly out of scope, first pass:**
 - ⚠ **Any write.**  Setups, schedules, call-home config, monitoring start/stop,
  and per-event delete are all mapped, and none of them will be implemented
  here.  **No command has ever been originated against a unit by this project**
  — every write observed was performed by THOR while we recorded.  Keeping that
  true through the read client is deliberate: it means the first thing we ever
  send to a customer's instrument is a decision someone made on purpose, not a
  side effect of a client that happened to grow a method.
 - the inbound call-home session — still the one protocol unknown
 ---
 ## Layout
 ```
 micromate/
  framing.py     NEW   frame building, response parsing, checksum
  protocol.py    NEW   one method per wire command, returns raw payloads
  client.py      NEW   high-level API, returns models
  idf_file.py    (existing — decodes what 0x5A returns, unchanged)
  models.py      (existing — extend, do not fork)
 ```
 **Transport is reused, not rewritten.**  `minimateplus/transport.py` is
 byte-level and protocol-agnostic — `BaseTransport`, `SerialTransport`,
 `TcpTransport`, plus `read_until_idle()` which already handles the RV50/RV55
 habit of emitting `\r\nRING\r\n\r\nCONNECT\r\n` to a caller.  Import it.
 ⚠ Do **not** import `minimateplus.framing`.  The two framings differ in ways
 that look small and are not, and a shared module would accumulate `if series ==`
 branches until neither case is readable.
 ---
 ## `micromate/framing.py`
 ### Requests
 Series IV accepts Series III request frames unmodified.  The simplest correct
 implementation re-exports the builder rather than duplicating it:
 ```python
 from minimateplus.framing import build_bw_frame   # requests are identical
 ```
 ⚠ One open question, flagged in the protocol reference and **not** settled:
 whether `0x10` bytes inside request *params* need stuffing.  No probe we sent
 carried one.  Until it is settled, assert on it rather than guessing:
 ```python
 def build_request(sub: int, offset: int = 0, params: bytes = bytes(10)) -> bytes:
    if 0x10 in params:
        raise NotImplementedError(
            "params containing 0x10 — stuffing rule unconfirmed; see "
            "micromate_protocol_reference.md, 'Untested and unsafe-until-agreed'"
        )
    return build_bw_frame(sub, offset, params)
 ```
 That turns an unknown into a loud failure instead of a corrupt frame.
 ### Responses — where Series III's parser cannot follow
 | | Series III | Micromate |
 |---|---|---|
 | frame start | `DLE STX` | **bare `STX`** |
 | `payload[1]` | `0x10` | `0xC5` (Blastware fw) / `0x03` (Thor fw) |
 | destuffing | `DLE+ETX` kept as literal inner-frame data | **`10 XX` → `XX`, uniformly** |
 The first row is why `S3FrameParser` returns nothing at all on Series IV traffic:
 it scans for `DLE+STX`, which never appears.
 The third is a genuine **simplification** — no inner-frame carve-out.  Validated
 by checksum across every capture in `bridges/captures/9-24-26 - micromate2/`:
 four candidate destuffing rules were tried, and only this one makes all frames
 validate.
 ### Checksum
 ```python
 def checksum(payload: bytes) -> int:
    return sum(b for b in payload if b != 0x10) & 0xFF
 ```
 The DLE-aware variant, same as Series III's `5A` and write frames — not the plain
 SUM8 of ordinary Series III reads.
 ### ⚠ The SUB byte can be escaped
 When a SUB's value is `0x02`, `0x03`, `0x04` or `0x10` it arrives as `10 XX`.
 Reading it positionally without destuffing reports `0x10`.  This bit once
 already — `SUB 0x02` was logged as `SUB_10` for an afternoon.  Destuff first,
 then index.
 ### Response shape
 ```python
@dataclass
 class MicromateFrame:
    sub: int              # response SUB;  request = 0xFF - sub
    flags: int            # 0xC5 Blastware line, 0x03 Thor line
    page_hi: int
    page_lo: int
    data: bytes           # payload[5:], checksum stripped
    checksum_valid: bool
    @property
    def firmware_line(self) -> str:      # "blastware" | "thor" | "unknown"
    @property
    def declared_length(self) -> int:    # uint16 BE at data[3:5]  (= payload[8:10])
 ```
 ⚠ **`declared_length` is a uint16 BE.**  Read as a single byte it under-reads
 `SUB 0x1A` by 47x — 44 against a true 2092.  This is the single most expensive
 mistake available in this protocol and it has already been made once.
 `MicromateFrameParser` mirrors `S3FrameParser`: `feed(bytes) -> list[frame]`,
 accumulates in `.frames`, `reset()`, and keeps the `bytes_fed` counter (it is
 what distinguishes "no bytes at all" from "bytes but no complete frame" on a
 timeout, and that distinction earned its keep during the Series III work).
 ---
 ## `micromate/protocol.py`
 One method per command, returning raw payload bytes.  No interpretation — that
 belongs in `client.py`.
 **Reads use `offset = 0xFFFF`** and return the whole block in one response;
 Series III's two-step probe/data dance is unnecessary.  `POLL` is the exception,
 taking its data length.  Per-command offsets, all observed:
 | command | SUB | rsp | offset | returns |
 |---|---|---|---|---|
 | poll | `0x5B` | `0xA4` | `0x0030` | device string, model |
 | serial | `0x15` | `0xEA` | `0x000A` | `UM12947` |
 | device info | `0x01` | `0xFE` | `0xFFFF` | firmware, calibration |
 | state | `0x49` | `0xB6` | `0xFFFF` | `data[11]`: non-zero = monitoring |
 | monitor status | `0x1C` | `0xE3` | `0xFFFF` | flag, **device clock**, battery, memory |
 | storage range | `0x06` | `0xF9` | `0xFFFF` | event storage extent |
 | active setup name | `0x41` | `0xBE` | `0xFFFF` | `TEST1.mmb` |
 | first setup | `0x3F` | `0xC0` | `0xFFFF` | setup-list walk head |
 | next setup | `0x40` | `0xBF` | `0xFFFF` | …until an empty name |
 | compliance config | `0x1A` | `0xE5` | `0xFFFF` | ~2103 B setup block |
 | call-home config | `0x2C` | `0xD3` | `0xFFFF` | 137 B |
 | arm event | `0x93` | `0x6C` | — | before every event |
 | first event | `0x1E` | `0xE1` | `0xFFFF` | key + size |
 | next event | `0x1F` | `0xE0` | `0xFFFF` | key + size |
 | event record | `0x0C` | `0xF3` | `0xFFFF` | 210 B — project, location, peaks |
 | event header | `0x0A` | `0xF5` | `0xFFFF` | 30 B list record |
 | bulk download | `0x5A` | `0xA5` | computed | **the `.IDFW` verbatim** |
 ⚠ **`SUB 0x1C` is 4 bytes longer on the Thor firmware line** (`0x30` vs `0x2C`).
 Parse **forward** from `declared_length`, never backward from the end — Series
 III reads battery and memory from the end of that block, and doing so on a BD
 unit yields a battery voltage of **577.92 V**.
 ⚠ **Test the monitoring flag for non-zero**, never against a constant.  It has
 read both `0x0E` and `0x0C` while monitoring.
 ### `0x5A` — simpler than Series III, deliberately
 No arming ritual, no chunk loop, no `STRT` end-offset parsing, no `TERM` frame.
 One request returns the whole event:
 ```python
 offset_word = 0x1000 + 2 * ceil(size / 512)     # size from the chain walk
 ```
 The payload **is** the `.IDFW` file, byte for byte — so it feeds
 `micromate.idf_file.read_idf_file()` and `/db/import/idf_file` unchanged.
 ⚠ Do not port the Series III `5A` walk.  Its address arithmetic caused a 5x
 over-read and a `> 64 KB` page-boundary bug that is *still open* on the Series
 III side.  None of that applies here.
 ---
 ## `micromate/client.py`
 ```python
 class MicromateClient:
    def __init__(self, transport: BaseTransport): ...
    def open(self) / close(self) / is_open(self)
    # identity and state
    def connect(self) -> DeviceInfo          # poll → serial → device info → state
    def get_state(self) -> UnitState         # monitoring?, clock, battery, memory
    def get_active_setup(self) -> str
    def list_setups(self) -> list[str]       # 0x3F → 0x40… until empty
    # events
    def list_events(self) -> list[EventRef]  # 0x93 → 0x1E → 0x1F… (key + size)
    def download_event(self, ref) -> bytes   # raw .IDFW/.IDFH
    def get_event(self, ref) -> Event        # download + decode via idf_file
 ```
 `connect()` should mirror THOR's preamble (`POLL → SERIAL → 0x49 → POLL`) —
 ⚠ but note the reference records that **whether the unit requires it is
 untested**.  Do it because it is known-good, not because it is known-necessary,
 and say so in the docstring.
 `list_events()` returns the key *and* the size, because `download_event()` needs
 the size to compute its offset word.
 ---
 ## Tests
 **Offline, from captured bytes — no hardware.**  This is the part worth doing
 first, because it can be fully verified tonight's-captures-style before any unit
 is involved.
 ```
 tests/test_micromate_framing.py
 ```
 ⚠ `bridges/captures/` and `tests/fixtures/` are both gitignored, so tests must
 not depend on files being present.  **Embed the frames as hex constants** — they
 are 16–68 bytes each, and a handful covers every case:
 | case | why |
 |---|---|
 | POLL probe reply, 19 B | shortest valid frame |
 | POLL data reply, 68 B | contains a literal `0x10` — only the DLE-aware checksum matches |
 | `0x1A` response, 2108 B | exercises `declared_length` as a true uint16 (`0x082C`) |
 | a Thor-line reply, `flags = 0x03` | `0x03` is ETX; proves destuffing before framing |
 | a frame whose SUB is `0x02` | arrives as `10 02`; proves destuff-then-index |
 | a truncated frame | parser must return nothing, not a bad frame |
 | a corrupted checksum | `checksum_valid == False`, frame still returned |
 Then a round-trip assertion: feed a whole captured session through the parser and
 assert the frame count and every SUB, against `scratch/mm_frame_parse.py`'s
 output — which is already known good, having parsed 24, 38 and 40-frame sessions
 with zero bad checksums.
 **Live, second:** against the bench unit on mint-mac via `mm_link.py`.
 `connect()`, `list_setups()` (should return the 23 known names), `list_events()`,
 then `download_event()` and assert the bytes decode and match a
 `/db/import/idf_file` ingest of the same event.
 ---
 ## Order of work
 1. `framing.py` + its tests — offline, verifiable immediately
 2. `protocol.py` — reads only, one method per row of the table above
 3. `client.py` — `connect()`, `get_state()`, `list_setups()`
 4. the event chain and `download_event()`
 5. decode end-to-end and compare against a store event
 Steps 1–2 need no hardware at all.
 ---
 ## Open questions to settle while implementing
 - **Request param stuffing** — raise `NotImplementedError` rather than guess.
 - **Is THOR's preamble required?**  Try one command cold and find out; it is a
  two-minute test with the bench unit and it removes a ritual if unnecessary.
 - **`Event` model fit** — Series IV carries fields Series III lacks (setup file
  name, `LMic`/`SMic` channels).  Extend `micromate/models.py`; do not fork the
  shared `Event`.
 - **Which `0x0C` fields to trust.**  The peak float there runs 2–5% above
  `max(T,V,L)` and is **not** the vector sum; its offset was inferred, not
  established.  The reference marks it do-not-rely-on — prefer decoded samples.
@@ -1,150 +0,0 @@
 # SFM — where it actually stands as a tool
 **Status as of 2026-09-20 (v0.31.0).**  This is the honest assessment, not the
 roadmap — `README.md § Roadmap` covers where it is *going*.  Expect this file to
 go stale; re-date it when you revise it.
 ---
 ## The framing
 SFM is **three different things wearing one name**, at three very different
 levels of maturity:
 | | what it is | maturity |
 |---|---|---|
 | **The codec library** | `minimateplus/`, `micromate/` — bytes in, `Event` out | **Production.** Verified per-sample at scale. |
 | **SDM — the data side** | the DB, waveform store, `/db/*`, ingest | **Production.** Terra-View depends on it daily. |
 | **SFM — the device side** | `/device/*`, live connections to units | **Emergency-grade.** Works, but manual, unauthenticated, and thinly tested. |
 | **The lab** | `seismo_lab.py`, `scratch/`, the Inspector | **Research artifacts.** Useful, not products. |
 Brian's own description — *"right now it's an emergency tool and a research
 project"* — is accurate, and it applies specifically to the **device side**.
 The data side is not an emergency tool; it has been carrying production for
 months.
 Most confusion about "is SFM reliable?" comes from answering for the wrong
 tier.
 ---
 ## 1. What you can rely on
 ### Production-grade — trust it
 - **Series-3 decode.** 14,338 / 14,338 files decode per-sample exact against
  preserved Blastware ASCII exports, 45 units, files back to 2018.
 - **Series-4 (Thor) decode.** 1,057,536 / 1,057,536 geo samples exact against
  Thor's own CSV exports; production IDFW 575/575 with zero truncations.
 - **Histogram decode.** 1,211 / 1,211 production histograms exact, including
  842,442 per-interval frequency comparisons with zero mismatches.
 - **The ingest path.** `/db/import/blastware_file` and `/db/import/idf_file`
  fed by the watchers — this is how prod actually gets its data, and it has
  been running unattended for months.
 - **`/db/*` read API.** Always-on, consumed by Terra-View for every fleet
  listing, event detail and report.
 - **The waveform store** — `.h5` + `.sfm.json` sidecars + retained raw
  binaries, with operator review state preserved across regeneration.
 - **`bridges/ach_server.py`** — speaks the full BW protocol to calling units.
  Proven in the field, including as a rescue tool (see the runbook).
 ### Emergency-grade — works, but you are the error handling
 - **`/device/*` live endpoints.**  They do what they say.  But they are
  synchronous, unauthenticated, and a single cellular download can exceed the
  60 s timeouts that sit in front of them.
 - **The rescue ladder** (`rescue`, `stop_monitoring_*`, `events/erase`).
  Each has worked in a real incident — but each has been used a handful of
  times, by one person, with the runbook open.
 - **The standalone webapp.**  Perfectly usable, and as of v0.31.0 the cheap
  probes and rescue actions are reachable without curl.  No auth of any kind.
 ### Research artifacts — useful, not products
 - **`seismo_lab.py`** — 2,789 lines of Tkinter (Bridge / Analyzer / Query DB /
  Inspector).  Desktop-only, single-user, no tests.
 - **`scratch/`** — the verification harnesses (`verify_against_ascii.py`,
  `verify_thor_against_csv.py`) and the offset detector (`offset_scan3.py`).
  These produced the numbers the production claims rest on, so they matter —
  but they are analysis scripts, not maintained code.
 - **`docs/offset_investigation.md`** — an open investigation, not a feature.
 ---
 ## 2. What to use when
 | you want to… | use | notes |
 |---|---|---|
 | Know if a unit is monitoring / its battery / memory | `GET /device/monitor/status?force=true` | ~2 s |
 | Know whether ACH is on | `GET /device/call_home` | ~2 s.  **Not** `/device/events`. |
 | See how full a unit's buffer is | `GET /device/events/storage_range` | ~2 s, no chain walk |
 | Stop a runaway unit | Diagnostics tab → Stop Monitoring | see the runbook first |
 | Reach a unit that will not answer | **point its modem at an `ach_server` and answer its call** | runbook Method A — do not race it |
 | List a unit's stored events | Events tab → Load events | **slow**, and broken past 64 KB (below) |
 | Get event data into the DB | the watcher → `/db/import/*` path | not the live walk |
 The single most useful habit: **the cheap probes are cheap and the event walk
 is not.**  Reaching for `/device/events` to answer a yes/no question about a
 unit is the mistake that motivated the v0.31.0 webapp changes.
 ---
 ## 3. Known issues
 | issue | impact | status |
 |---|---|---|
 | **5A walk dies once a unit's buffer crosses 64 KB** | `/device/events` 500s; event body never downloads | Known, documented in `CLAUDE.md`.  Needs a BW capture of a spanning event to fix properly. |
 | **No auth on SFM at all** | 21 `/device/*` endpoints, including destructive ones, open to anything that reaches the port | Design agreed (Terra-View as authenticated jump host); not built. |
 | **Swagger try-it-out is live on destructive endpoints** | `POST /device/events/erase` is one click away at `:8200/docs` | Partially mitigated: the webapp's erase now requires typing the serial.  `/docs` itself is unguarded. |
 | **`SUB 0x08` lifetime counter reads 0** | `/device/events/index` returns a meaningless number | Suspected field-offset bug.  Surfaced in the UI as "unreliable". |
 | **Long device operations are synchronous** | 60 s timeouts in `routers/sfm.py` and the reverse proxy; a full download exceeds both | Known design constraint.  Must be POST-starts-job / GET-polls before any remote lab. |
 | **`backfill_sidecars.py --force` silently inserts DB rows** | store files with no DB row get one; the dry-run does not report the count | Known.  Avoid `--force` — `TOOL_VERSION` gates regeneration anyway. |
 | **14 sensitive-range files show an exact 8× discrepancy** | 10.0 / 1.25 — a units problem, not a decode problem | Open, not blocking. |
 | **16 failing tests on `dev`** | 15 need gitignored fixture bundles; 1 is real (`sc["peak_values"]["transverse"]` returns `None` where `0.0` is expected) | The real one shipped in v0.31.0. |
 ---
 ## 4. What stands between this and a real tool
 Roughly in dependency order — each unblocks the ones below it.
 **1. Authentication.**  Everything else is gated on this.  SFM has none, and
 the modem IP whitelist gives zero protection because SFM *is* the whitelisted
 origin.  The agreed design delegates rather than builds: Terra-View becomes the
 authenticated jump host (`/api/sfm/*` already inherits deny-by-default operator
 auth), and the `8200:8200` publish is dropped so Terra-View is the only door.
 **2. Async long operations.**  POST starts a job, GET polls.  Retrofitting this
 after building a remote lab on top of synchronous endpoints would be far worse
 than designing for it now.
 **3. Confirm-guards on the remaining destructive endpoints.**  Auth answers
 *who*, not *did you mean it*.  The webapp's erase is guarded; the other seven
 destructive POSTs and `/docs` are not.
 **4. The 5A page-boundary fix.**  Until this lands, live event download is
 unreliable on exactly the units most likely to need attention — the ones that
 have been recording heavily.  Wants a Blastware capture of an event spanning a
 page boundary before the chunk-addressing half is trustworthy.
 **5. A live Thor / Micromate client.**  The device side is MiniMate-only.
 Series-4 units can only be read from forwarded files, so half the fleet has no
 live path at all.
 **6. Test coverage that runs from a clean checkout.**  15 of 16 current
 failures are missing fixture bundles.  A test suite that cannot go green on a
 fresh clone cannot gate anything.
 **7. The SDM rename.**  Cosmetic relative to the above, but the longer `sfm/`
 holds the data-side code the more the tiers blur.  ~30–50 files here, ~10–15 in
 Terra-View, plus a Docker volume migration.  Do it when the codebase is quiet.
 ---
 ## The short version
 The **data side is a real tool already**.  The **device side is a set of sharp
 instruments** that work in the hands of the person who wrote them, with the
 runbook open.  The gap between those two states is mostly **auth, async, and
 guardrails** — not protocol work.  The protocol is the part that is actually
 finished.
@@ -296,16 +296,6 @@ def apply_report_to_event(event: Event, report: BwAsciiReport) -> None:
        event.sample_rate = report.sample_rate_sps
    if report.record_time_s is not None:
        event.rectime_seconds = report.record_time_s
    # The report's event_datetime is Blastware's exact trigger time (parsed
    # from Event Time + Event Date).  Prefer it over the binary footer's stop
    # time so a report-paired import matches BW to the second.
    edt = report.event_datetime
    if edt is not None:
        event.timestamp = Timestamp(
            raw=b"", flag=0x10,
            year=edt.year, unknown_byte=0, month=edt.month, day=edt.day,
            hour=edt.hour, minute=edt.minute, second=edt.second,
        )
 def apply_bw_report_dict_to_event(event: Event, bw_report: dict) -> None:
@@ -818,30 +808,6 @@ def derive_record_type_from_filename(filename, default: str = "Waveform") -> str
    return _RECORD_TYPE_BY_EXT_SUFFIX.get(ext[-1].upper(), default)
 # Marker for the recording-setup config block, and the offset of the record-time
 # float32 within it.  The configured post-trigger record time (seconds) is a
 # big-endian float32 exactly 30 bytes before the "Standard Recording Setup"
 # label.  Verified across the corpus reading 1.0 / 2.0 / 3.0 s on different
 # setups — and ts2 - record_time reproduces Blastware's trigger to the second
 # (N844LQHB: stop 10:33:32 - 3.0 = 10:33:29).
 _RECSETUP_MARKER = b"Standard Recording Setup"
 _RECTIME_OFFSET_BEFORE_MARKER = 30
 def _parse_record_time_seconds(raw: bytes) -> Optional[float]:
    """The configured post-trigger record time in seconds, from the recording-
    setup config block, or None when absent / implausible."""
    a = raw.find(_RECSETUP_MARKER)
    if a < _RECTIME_OFFSET_BEFORE_MARKER:
        return None
    off = a - _RECTIME_OFFSET_BEFORE_MARKER
    try:
        rt = struct.unpack(">f", raw[off:off + 4])[0]
    except struct.error:
        return None
    return rt if 0.05 <= rt <= 600.0 else None
 def read_blastware_file(path: Union[str, Path]) -> Event:
    """
    Parse a Blastware waveform file into an Event.
@@ -951,10 +917,6 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
    # rest of the event (timestamp, waveform_key, project strings) is
    # still recoverable and useful.
    decoded = decode_waveform_v2(body)
    # Discriminator for the timestamp logic below: a waveform (trigger) event
    # vs a histogram window.  Keyed on the codec, not the filename — the
    # save_imported_bw path passes a tmp ".bw" name whose extension lies.
    is_waveform_body = decoded is not None
    if decoded is None:
        decoded = decode_histogram_body(body)
    if decoded is None:
@@ -986,31 +948,7 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
    ev.total_samples   = strt_fields.get("total_samples")
    ev.pretrig_samples = strt_fields.get("pretrig_samples")
-    # Event timestamp.  The footer's two timestamps mean different things by
+    if ts1 is not None:
    # record type:
    #   * Waveform: ts1 = the monitoring-SESSION start (shared across every
    #     event that day — a unit arming at 06:00 stamps 06:00 on all of them),
    #     ts2 = THIS event's recording STOP.  Blastware's Date/Time is the
    #     TRIGGER = ts2 - record time, and the record time is a float32 in the
    #     recording-setup config block (see _parse_record_time_seconds), so the
    #     exact trigger is recoverable from the binary alone.  Falls back to ts2
    #     (the stop, within the record duration) if the config block is absent.
    #     (Stamping ts1 showed the session start, hours off.)
    #   * Histogram / undecodable: ts1 = the window start, which IS the event
    #     time — keep it.
    # Discriminate by ``is_waveform_body`` (the codec), not the filename.
    if is_waveform_body and ts2 is not None:
        _stop = datetime.datetime(ts2.year, ts2.month, ts2.day,
                                  ts2.hour, ts2.minute, ts2.second)
        _rt = _parse_record_time_seconds(raw)
        _trig = _stop - datetime.timedelta(seconds=_rt) if _rt is not None else _stop
        ev.timestamp = Timestamp(
            raw=footer[10:18],
            flag=0x10,
            year=_trig.year, unknown_byte=0, month=_trig.month, day=_trig.day,
            hour=_trig.hour, minute=_trig.minute, second=_trig.second,
        )
    elif ts1 is not None:
        ev.timestamp = Timestamp(
            raw=footer[2:10],
            flag=0x10,
@@ -1,33 +0,0 @@
 """Pretend to be a Micromate on a serial port: log what arrives, reply to POLL.
 Proves the modem's return path (serial -> TCP) independently of the real unit.
 """
 import os, select, sys, termios, time
 path, baud = sys.argv[1], int(sys.argv[2]) if len(sys.argv) > 2 else 115200
 B = {9600: termios.B9600, 38400: termios.B38400, 115200: termios.B115200}[baud]
 fd = os.open(path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
 a = termios.tcgetattr(fd)
 a[0] = a[1] = a[3] = 0
 a[2] = termios.CS8 | termios.CREAD | termios.CLOCAL
 a[4] = a[5] = B
 a[6] = list(a[6]); a[6][termios.VMIN] = 0; a[6][termios.VTIME] = 0
 termios.tcsetattr(fd, termios.TCSANOW, a)
 termios.tcflush(fd, termios.TCIOFLUSH)
 # A real POLL probe reply, captured from UM12947 on 2026-09-24.
 REPLY = bytes.fromhex("0200c5a4000000000000300000000000000099") + b"\x03"
 print(f"fake unit on {path} @ {baud}; will answer any inbound frame", flush=True)
 while True:
    r, _, _ = select.select([fd], [], [], 1.0)
    if not r:
        continue
    data = os.read(fd, 4096)
    if not data:
        continue
    ts = time.strftime("%H:%M:%S")
    print(f"{ts}  IN  {len(data):3} B  {data.hex(' ')}", flush=True)
    time.sleep(0.02)
    os.write(fd, REPLY)
    print(f"{ts}  OUT {len(REPLY):3} B  {REPLY.hex(' ')}  <- canned POLL reply", flush=True)
@@ -1,202 +0,0 @@
 #!/usr/bin/env python3
 """
 mm_frame_parse.py — parse Micromate (Series IV) frames out of a seismo_lab
 raw capture pair.
 Why this exists
 ---------------
 `minimateplus.framing.S3FrameParser` cannot see Micromate traffic.  It locates
 frames by scanning for `DLE STX`, and a Micromate response has **no leading
 DLE** — it starts at a bare `STX`.  It also expects `payload[1] == 0x10`, where
 the Micromate sends `0xC5` (Blastware firmware) or `0x03` (Thor firmware).
 The practical consequence, seen on the 9-24-26 setup-push capture: the
 Blastware-side requests parse fine (Thor emits Series III request frames), but
 **every device response is silently dropped or mis-framed** — so a capture that
 actually contains 12 acked writes looks like 12 unanswered requests.
 Destuffing
 ----------
 One rule covers both directions: after the leading doubled `BW_CMD`, every
 `10 XX` pair on the wire destuffs to `XX`.  That includes `10 03` — Thor
 escapes literal `0x03` bytes in write data so they are not mistaken for ETX,
 exactly as Blastware does.
 That rule was chosen by evidence, not assumption: of the four candidates tried
 against the 9-24-26 capture's four data-carrying write frames, it is the only
 one under which all four checksums validate.  See
 `docs/micromate_protocol_reference.md` → *The write path*.
 Usage
 -----
    python scratch/mm_frame_parse.py <capture-dir>
    python scratch/mm_frame_parse.py <raw_bw.bin> <raw_s3.bin>
    python scratch/mm_frame_parse.py <capture-dir> --dump 0x71
 """
 from __future__ import annotations
 import argparse
 import sys
 from pathlib import Path
 DLE, STX, ETX, ACK = 0x10, 0x02, 0x03, 0x41
 # Request SUB -> short name.  Series III names where they carry over; the
 # Series IV additions are marked.
 SUBNAME = {
    0x01: "DEVICE_INFO",
    0x06: "STORAGE_RANGE",
    0x08: "EVENT_INDEX",
    0x0A: "WAVEFORM_HDR",
    0x0C: "WAVEFORM_REC",
    0x15: "SERIAL",
    0x1A: "COMPLIANCE_CFG",
    0x1C: "MONITOR_STATUS",
    0x1E: "EVENT_HDR",
    0x2C: "CALLHOME_CFG",
    0x2E: "TRIGGER_CFG_READ",     # Series IV
    0x3E: "OPERATOR",
    0x41: "SETUP_NAME_READ",      # Series IV
    0x5A: "BULK_DOWNLOAD",
    0x5B: "POLL",
    0x68: "EVENT_INDEX_WRITE",
    0x69: "WAVEFORM_WRITE",
    0x71: "COMPLIANCE_WRITE",
    0x72: "CONFIRM_A",
    0x73: "CONFIRM_B",
    0x74: "CONFIRM_C",
    0x82: "TRIGGER_WRITE",
    0x83: "TRIGGER_CONFIRM",
    0xDA: "SETUP_FILE_DECL",      # Series IV — names the target .MMB
    0xFE: "FULL_CFG",
 }
 def destuff(blob: bytes, start: int, *, is_request: bool) -> tuple[bytes, int, int]:
    """Destuff one frame starting at `start`.
    Returns (payload, checksum, index_of_terminating_ETX).  `payload` excludes
    the trailing checksum byte.  A request frame opens `ACK STX 10 10`; a
    response opens with a bare `STX`.
    """
    i = start + (2 if is_request else 1)
    out = bytearray()
    if is_request:
        # The doubled BW_CMD is the one guaranteed stuffed byte.
        if blob[i : i + 2] != bytes([DLE, DLE]):
            raise ValueError(f"@0x{start:04x}: request does not open with 10 10")
        out.append(DLE)
        i += 2
    while i < len(blob):
        b = blob[i]
        if b == DLE and i + 1 < len(blob):
            out.append(blob[i + 1])
            i += 2
            continue
        if b == ETX:
            break
        out.append(b)
        i += 1
    if len(out) < 2:
        raise ValueError(f"@0x{start:04x}: frame too short")
    return bytes(out[:-1]), out[-1], i
 def frames(blob: bytes, *, is_request: bool):
    """Yield (offset, payload, chk, checksum_kind)."""
    i, n = 0, len(blob)
    while i < n:
        if is_request:
            if not (blob[i] == ACK and i + 1 < n and blob[i + 1] == STX):
                i += 1
                continue
        elif blob[i] != STX:
            i += 1
            continue
        try:
            payload, chk, end = destuff(blob, i, is_request=is_request)
        except ValueError:
            i += 1
            continue
        sum8 = sum(payload) & 0xFF
        dle_aware = (sum(b for b in payload if b != DLE) & 0xFF)
        if sum8 == chk:
            kind = "SUM8"
        elif dle_aware == chk:
            kind = "DLE-aware"
        else:
            kind = "BAD"
        yield i, payload, chk, kind
        i = end + 1
 def describe(payload: bytes, is_request: bool) -> str:
    if len(payload) < 3:
        return "??"
    sub = payload[2]
    if is_request:
        return SUBNAME.get(sub, f"SUB_{sub:02X}")
    req = 0xFF - sub
    return "rsp<-" + SUBNAME.get(req, f"SUB_{req:02X}")
 def report(path: Path, *, is_request: bool, dump_sub: int | None) -> None:
    blob = path.read_bytes()
    side = "Thor" if is_request else "unit"
    print(f"== {side:4}  {path.name}  ({len(blob)} bytes)")
    n_bad = 0
    for idx, (off, p, chk, kind) in enumerate(frames(blob, is_request=is_request)):
        if kind == "BAD":
            n_bad += 1
        sub = p[2] if len(p) > 2 else -1
        flags = p[1] if len(p) > 1 else -1
        # Requests carry offset at payload[4:6]; responses page at [3:5].
        word = int.from_bytes(p[4:6] if is_request else p[3:5], "big")
        data = len(p) - 16 if is_request else max(len(p) - 5, 0)
        print(
            f"  [{idx:2}] @0x{off:04x} payload={len(p):5} data={data:5} "
            f"flags=0x{flags:02x} SUB=0x{sub:02x} {describe(p, is_request):18} "
            f"{'offset' if is_request else 'page'}=0x{word:04x} chk={kind}"
        )
        if dump_sub is not None and sub == dump_sub:
            body = p[16:] if is_request else p[5:]
            print(f"       ---- data ({len(body)} bytes) ----")
            for o in range(0, len(body), 16):
                chunk = body[o : o + 16]
                txt = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk)
                print(f"       {o:06x}  {chunk.hex(' '):<47}  |{txt}|")
    print(f"  -- {idx + 1} frames, {n_bad} bad checksum\n")
 def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("paths", nargs="+",
                    help="a capture directory, or raw_bw.bin and raw_s3.bin")
    ap.add_argument("--dump", default=None,
                    help="hex-dump the data section of this SUB (e.g. 0x71)")
    args = ap.parse_args()
    dump_sub = int(args.dump, 0) if args.dump else None
    if len(args.paths) == 1 and Path(args.paths[0]).is_dir():
        d = Path(args.paths[0])
        bw = sorted(d.glob("raw_bw_*.bin"))
        s3 = sorted(d.glob("raw_s3_*.bin"))
        if not bw or not s3:
            print(f"{d}: need one raw_bw_*.bin and one raw_s3_*.bin", file=sys.stderr)
            return 2
        pairs = [(bw[0], True), (s3[0], False)]
    elif len(args.paths) == 2:
        pairs = [(Path(args.paths[0]), True), (Path(args.paths[1]), False)]
    else:
        ap.error("pass a capture directory, or exactly two .bin files")
    for path, is_request in pairs:
        report(path, is_request=is_request, dump_sub=dump_sub)
    return 0
 if __name__ == "__main__":
    sys.exit(main())
@@ -1,119 +0,0 @@
 #!/usr/bin/env python3
 """
 socat_log_split.py — recover a capture pair from a `socat -x` relay log.
 Why this exists
 ---------------
 The bench relay that puts Thor in front of a USB-attached Micromate is:
    socat -d -d -x TCP-LISTEN:12345,reuseaddr,fork /dev/ttyACM0,raw,echo=0,b115200 \
        > ~/mm-captures/socat_<ts>.log 2>&1
 `-x` makes socat hex-dump every byte it forwards, in both directions, with
 timestamps.  That log is therefore a **complete second copy of every capture**
 taken through the relay — independent of whether seismo_lab was recording.
 On 2026-09-25 that mattered: a capture's `.bin` files never made it off the
 Windows machine, and the session was rebuilt from this log instead.  When the
 real bins turned up later, the reconstruction was **byte-for-byte identical in
 both directions** (3,595 and 4,004 bytes).  So this is a validated fallback, not
 a lossy approximation.
 Log format
 ----------
 ```
 > 2026/09/25 00:30:35.000276659  length=21 from=0 to=20
 41 02 10 10 00 5b 00 00 30 00 ...
 2026/09/25 00:30:35 socat[32190] N write(5, 0x..., 21) completed
 < 2026/09/25 00:30:35.000384100  length=64 from=0 to=63
 02 00 c5 a4 00 00 30 00 ...
 ```
 `>` is data heading toward the serial device (Thor → unit).  `<` is data coming
 back (unit → Thor).  Hex lines are space-separated and indented; socat's own
 status lines start with a date and carry no payload.
 Usage
 -----
    # whole log
    python scratch/socat_log_split.py socat_20260924_181248.log --out-dir ./recovered
    # one session — line numbers from the "accepting connection" markers
    grep -n "accepting connection" socat_*.log
    python scratch/socat_log_split.py socat_*.log --from-line 919 --out-dir ./recovered
 Then parse the result as usual:
    python scratch/mm_frame_parse.py recovered/raw_bw.bin recovered/raw_s3.bin
 ⚠ A log spanning several sessions concatenates them.  Split by line number using
 the `accepting connection` markers, or the frame walk will run sessions together.
 """
 from __future__ import annotations
 import argparse
 import re
 from pathlib import Path
 _HEX = re.compile(r"\A[0-9a-f]{2}\Z")
 _SOCAT_STATUS = re.compile(r"\A\d{4}/\d{2}/\d{2}")
 def split(lines) -> tuple[bytes, bytes]:
    """Return (to_device, from_device) byte streams."""
    to_dev, from_dev = bytearray(), bytearray()
    cur = None
    for line in lines:
        if line.startswith(">"):
            cur = to_dev
            continue
        if line.startswith("<"):
            cur = from_dev
            continue
        if _SOCAT_STATUS.match(line):
            # socat's own status line ends the current dump block.
            cur = None
            continue
        if cur is None or not line.startswith(" "):
            continue
        toks = line.split()
        if toks and all(_HEX.match(t) for t in toks):
            cur.extend(int(t, 16) for t in toks)
    return bytes(to_dev), bytes(from_dev)
 def main() -> int:
    ap = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    ap.add_argument("log", help="a socat -x log file")
    ap.add_argument("--out-dir", default=".", help="where to write the .bin pair")
    ap.add_argument("--from-line", type=int, default=1,
                    help="first log line to read (1-based) — use the "
                         "'accepting connection' marker of the session you want")
    ap.add_argument("--to-line", type=int, default=None,
                    help="last log line to read (1-based, inclusive)")
    ap.add_argument("--prefix", default="raw", help="output basename prefix")
    args = ap.parse_args()
    lines = Path(args.log).read_text(errors="replace").splitlines()
    lo = max(args.from_line - 1, 0)
    hi = args.to_line if args.to_line is not None else len(lines)
    to_dev, from_dev = split(lines[lo:hi])
    out = Path(args.out_dir)
    out.mkdir(parents=True, exist_ok=True)
    bw = out / f"{args.prefix}_bw.bin"
    s3 = out / f"{args.prefix}_s3.bin"
    bw.write_bytes(to_dev)
    s3.write_bytes(from_dev)
    print(f"Thor -> unit  {len(to_dev):>7} bytes  {bw}")
    print(f"unit -> Thor  {len(from_dev):>7} bytes  {s3}")
    if not to_dev or not from_dev:
        print("⚠ one direction is empty — check --from-line / --to-line")
    return 0
 if __name__ == "__main__":
    raise SystemExit(main())
@@ -108,12 +108,6 @@
      color: var(--text);
    }
    .btn-ghost:hover { border-color: var(--blue-lt); color: var(--blue-lt); }
    .btn-danger   { background: var(--red);    color: #fff; }
    .btn-danger:hover:not(:disabled) { filter: brightness(1.15); }
    .diag-result  { display:block; margin-top:6px; font-size:12px; opacity:.85;
                    white-space:pre-wrap; word-break:break-word; }
    .diag-result.ok    { color: var(--green); }
    .diag-result.error { color: var(--red); }
    .btn:disabled { background: var(--surface2) !important; color: var(--text-mute) !important; cursor: not-allowed; border-color: var(--border2) !important; }
    /* #connect-btn styles moved to #live-connect-bar block */
@@ -916,7 +910,6 @@
  <button class="tab-btn"        data-tab="events"    onclick="switchTab('events')">Events</button>
  <button class="tab-btn"        data-tab="config"    onclick="switchTab('config')">Config</button>
  <button class="tab-btn"        data-tab="call-home" onclick="switchTab('call-home')">Call Home</button>
  <button class="tab-btn"        data-tab="diagnostics" onclick="switchTab('diagnostics')">Diagnostics</button>
 </div>
 <!-- ════════════════════════════════════════════════════════════════
@@ -945,10 +938,6 @@
 <div id="tab-events" class="tab-pane" style="display:flex; flex-direction:column; overflow:hidden;">
  <div class="event-toolbar">
    <button class="btn btn-ghost" id="load-events-btn" onclick="loadEventList()" disabled
            title="Walk the device's event chain and list its stored events. This is the slow one — it reads every event header over the cellular link.">
      ⟳ Load events
    </button>
    <button class="btn btn-ghost" id="load-btn" onclick="loadWaveform()" disabled>Load Waveform</button>
    <button class="btn btn-ghost" id="save-btn" onclick="saveEventToDb()" disabled
            title="Download the full waveform from the device and save it to the SFM database + waveform store. Honors the Force refresh toggle.">
@@ -1216,77 +1205,6 @@
 </div><!-- end #tab-call-home -->
 <!-- ════════════════════════════════════════════════════════════════
     TAB: Diagnostics
 ═══════════════════════════════════════════════════════════════════ -->
 <div id="tab-diagnostics" class="tab-pane">
  <div class="cfg-grid">
    <div class="cfg-section">
      <div class="cfg-section-title">Device State</div>
      <div class="hint" style="margin-bottom:10px">
        Fast probes — POLL plus one read each, about 2 s. None of these walk the event chain.
      </div>
      <div class="dev-table" id="diag-table"></div>
      <div class="cfg-actions" style="margin-top:12px">
        <button class="btn btn-ghost" id="diag-refresh-btn" onclick="refreshDiagnostics()" disabled>Refresh</button>
        <span id="diag-status"></span>
      </div>
    </div>
    <div class="cfg-section">
      <div class="cfg-section-title">Actions</div>
      <div class="cfg-field">
        <label>Stop Monitoring</label>
        <button class="btn btn-ghost" id="diag-stop-btn" onclick="diagStopMonitoring()" disabled>Send Stop (SUB 0x97)</button>
        <div class="hint">Halts recording. On a unit triggering continuously, this is what breaks the call-home loop.</div>
        <span class="diag-result" id="diag-stop-result"></span>
      </div>
      <div class="cfg-field">
        <label>Disable Auto Call Home</label>
        <button class="btn btn-ghost" id="diag-ach-btn" onclick="diagDisableAch()" disabled>Disable ACH</button>
        <div class="hint">Stored events are left untouched (<code>rescue?erase=false</code>). The unit stops dialing out until ACH is re-enabled.</div>
        <span class="diag-result" id="diag-ach-result"></span>
      </div>
      <div class="cfg-field">
        <label>Erase All Events</label>
        <input type="text" id="diag-erase-confirm" placeholder="Type the serial to enable"
               oninput="diagCheckEraseConfirm()" autocomplete="off" />
        <button class="btn btn-danger" id="diag-erase-btn" onclick="diagEraseEvents()" disabled>Erase Events</button>
        <div class="hint">⚠ Permanent, and resets the event chain to key <code>0x01110000</code>. Download anything worth keeping first.</div>
        <span class="diag-result" id="diag-erase-result"></span>
      </div>
    </div>
    <div class="cfg-section">
      <div class="cfg-section-title">Unresponsive Unit</div>
      <div class="hint" style="margin-bottom:10px">
        The escalation ladder from <code>docs/runbooks/wedged_unit_recovery.md</code>, for a unit too busy
        to answer normal request/response. Prefer <b>Method A</b> — point the modem at an
        <code>ach_server</code> and answer its call — before racing it with these.
      </div>
      <div class="cfg-field">
        <label>Slow drip <span class="hint" style="display:inline">(one held session, a stop every 3 s)</span></label>
        <button class="btn btn-ghost" id="diag-drip-btn" onclick="diagSlowDrip()" disabled>Run 120 s drip</button>
        <div class="hint">Success is <code>bytes_received &gt; 0</code>. A full duration with <code>send_error: null</code> is <b>not</b> success on its own.</div>
        <span class="diag-result" id="diag-drip-result"></span>
      </div>
      <div class="cfg-field">
        <label>Blind stop <span class="hint" style="display:inline">(fire-and-forget, one attempt)</span></label>
        <button class="btn btn-ghost" id="diag-blind-btn" onclick="diagBlindStop()" disabled>Send blind stop</button>
        <span class="diag-result" id="diag-blind-result"></span>
      </div>
    </div>
  </div>
 </div><!-- end #tab-diagnostics -->
 </div><!-- end #section-live -->
 <!-- ════════════════════════════════════════════════════════════════
@@ -1443,8 +1361,6 @@
 // ── State ──────────────────────────────────────────────────────────────────────
 let unitInfo      = null;
 let eventList     = [];
 let storageInfo   = null;   // /device/events/storage_range — cheap, read on connect
 let eventsLoaded  = false;  // the event chain walk is opt-in; see loadEventList()
 let currentEvent  = 0;
 let charts        = {};
 let geoAdcScale      = 6.206;
@@ -1542,7 +1458,6 @@ function switchTab(name) {
  if (name === 'units')    { if (!unitsLoaded)  loadUnits(); }
  if (name === 'monlog')   { if (!monlogLoaded) loadMonitorLog(); }
  if (name === 'sessions') { if (!sessLoaded)   loadSessions(); }
  if (name === 'diagnostics' && devHost() && unitInfo) refreshDiagnostics();
 }
 // ── Connect ────────────────────────────────────────────────────────────────────
@@ -1563,13 +1478,18 @@ async function connectUnit() {
    btn.disabled = false; btn.textContent = 'Connect'; return;
  }
-  // Connecting deliberately does NOT walk the event chain.  That walk reads
+  setStatus('Fetching event list…', 'loading');
-  // every event header over the cellular link and can take minutes — or fail
+  try {
-  // outright on a unit whose buffer has wrapped past 0xFFFF.  Use the ~2 s
+    const r = await fetch(`${api()}/device/events?${deviceParams()}`);
-  // probes instead; the event list is opt-in via loadEventList().
+    if (!r.ok) { const e = await r.json().catch(() => ({})); throw new Error(e.detail || r.statusText); }
-  eventList = []; eventsLoaded = false;
+    const evData = await r.json();
-  setStatus('Reading device state…', 'loading');
+    eventList = evData.events || [];
-  storageInfo = await fetchJson(`/device/events/storage_range`).catch(() => null);
+    // Merge compliance from /device/events response (it re-reads it)
    if (evData.device) unitInfo = { ...unitInfo, ...evData.device };
  } catch (e) {
    setStatus(`Event fetch failed: ${e.message}`, 'error');
    btn.disabled = false; btn.textContent = 'Reconnect'; return;
  }
  populateDeviceBar();
  populateDeviceTab();
@@ -1578,9 +1498,11 @@ async function connectUnit() {
  document.getElementById('device-bar').style.display    = 'flex';
  document.getElementById('monitor-panel').style.display = 'flex';
-  setEventButtonsEnabled();
+  document.getElementById('load-btn').disabled     = eventList.length === 0;
-  document.getElementById('load-events-btn').disabled = false;
+  document.getElementById('save-btn').disabled     = eventList.length === 0;
-  setDiagButtonsEnabled(true);
+  document.getElementById('download-btn').disabled = eventList.length === 0;
  document.getElementById('prev-btn').disabled     = true;
  document.getElementById('next-btn').disabled     = eventList.length <= 1;
  document.getElementById('cfg-read-btn').disabled  = false;
  document.getElementById('cfg-write-btn').disabled = false;
  document.getElementById('ch-read-btn').disabled   = false;
@@ -1588,9 +1510,7 @@ async function connectUnit() {
  btn.disabled = false; btn.textContent = 'Reconnect';
-  setStatus(storageInfo && storageInfo.is_empty
+  setStatus(`Connected — ${eventList.length} event${eventList.length !== 1 ? 's' : ''} stored.`, 'ok');
    ? 'Connected — no events stored.'
    : 'Connected. Event list not loaded (Events → Load events).', 'ok');
  // Fetch monitor status in background (non-blocking)
  refreshMonitorStatus().catch(() => {});
@@ -1602,48 +1522,6 @@ async function connectUnit() {
  }
 }
 // ── Shared fetch helper ────────────────────────────────────────────────────────
 async function fetchJson(path, opts) {
  const sep = path.includes('?') ? '&' : '?';
  const r = await fetch(`${api()}${path}${sep}${deviceParams()}`, opts);
  const body = await r.json().catch(() => ({}));
  if (!r.ok) throw new Error(body.detail || r.statusText);
  return body;
 }
 function setEventButtonsEnabled() {
  const n = eventList.length;
  document.getElementById('load-btn').disabled     = n === 0;
  document.getElementById('save-btn').disabled     = n === 0;
  document.getElementById('download-btn').disabled = n === 0;
  document.getElementById('prev-btn').disabled     = true;
  document.getElementById('next-btn').disabled     = n <= 1;
 }
 // ── Event list (opt-in — this is the slow chain walk) ──────────────────────────
 async function loadEventList() {
  if (!devHost()) { setStatus('Connect to a device first.', 'error'); return; }
  const btn = document.getElementById('load-events-btn');
  btn.disabled = true;
  setStatus('Walking the event chain — this can take a while…', 'loading');
  try {
    const evData = await fetchJson('/device/events');
    eventList = evData.events || [];
    eventsLoaded = true;
    // /device/events re-reads compliance; fold it in.
    if (evData.device) unitInfo = { ...unitInfo, ...evData.device };
  } catch (e) {
    setStatus(`Event fetch failed: ${e.message}`, 'error');
    btn.disabled = false; return;
  }
  populateDeviceBar();
  populateDeviceTab();
  populateEventChips();
  setEventButtonsEnabled();
  btn.disabled = false;
  setStatus(`${eventList.length} event${eventList.length !== 1 ? 's' : ''} stored.`, 'ok');
 }
 // ── Device bar ─────────────────────────────────────────────────────────────────
 function populateDeviceBar() {
  qs('di-serial').textContent   = unitInfo.serial || '—';
@@ -1652,7 +1530,7 @@ function populateDeviceBar() {
  qs('di-sr').textContent       = cc.sample_rate ? `${cc.sample_rate} sps` : '—';
  qs('di-rt').textContent       = cc.record_time != null ? `${cc.record_time.toFixed(1)} s` : '—';
  qs('di-trig').textContent     = cc.trigger_level_geo != null ? `${cc.trigger_level_geo.toFixed(3)} in/s` : '—';
-  qs('di-count').textContent    = eventsLoaded ? eventList.length : '—';
+  qs('di-count').textContent    = eventList.length;
  qs('di-project').textContent  = cc.project || '—';
  qs('di-client').textContent   = cc.client  || '—';
  qs('di-operator').textContent = cc.operator || '—';
@@ -1782,8 +1660,7 @@ function populateDeviceTab() {
    { label:'DSP',            value: unitInfo.dsp_version || '—' },
    { label:'Model',          value: unitInfo.model || '—' },
    { label:'Manufacturer',   value: unitInfo.manufacturer || '—' },
-    { label:'Stored Events',  value: eventsLoaded ? eventList.length : 'not loaded' },
+    { label:'Stored Events',  value: eventList.length },
    { label:'Storage Used',   value: storageUsedLabel() },
  ];
  for (const {label, value} of cardData) {
    const c = document.createElement('div');
@@ -1830,158 +1707,6 @@ function renderTable(id, rows) {
  }
 }
 // ── Diagnostics ────────────────────────────────────────────────────────────────
 // Everything here is a cheap probe (POLL + one read) or a single write.  None of
 // it walks the event chain.  See docs/runbooks/wedged_unit_recovery.md.
 function storageUsedLabel() {
  if (!storageInfo) return '—';
  if (storageInfo.is_empty) return 'empty';
  const f = storageInfo.first_key, l = storageInfo.last_key;
  return (f && l) ? `${f} → ${l}` : '—';
 }
 function setDiagButtonsEnabled(on) {
  for (const id of ['diag-refresh-btn','diag-stop-btn','diag-ach-btn',
                    'diag-drip-btn','diag-blind-btn']) {
    const el = document.getElementById(id);
    if (el) el.disabled = !on;
  }
  diagCheckEraseConfirm();
 }
 // Erase is guarded by typing the serial — auth answers "who", not "did you mean it".
 function diagCheckEraseConfirm() {
  const box = document.getElementById('diag-erase-confirm');
  const btn = document.getElementById('diag-erase-btn');
  if (!box || !btn) return;
  const serial = (unitInfo && unitInfo.serial) || '';
  btn.disabled = !serial || box.value.trim().toUpperCase() !== serial.toUpperCase();
 }
 function diagResult(id, text, cls) {
  const el = document.getElementById(id);
  if (!el) return;
  el.textContent = text;
  el.className = 'diag-result' + (cls ? ' ' + cls : '');
 }
 async function refreshDiagnostics() {
  if (!devHost()) return;
  const st = document.getElementById('diag-status');
  if (st) { st.textContent = 'Reading…'; st.className = 'loading'; }
  const [mon, store, idx] = await Promise.all([
    fetchJson('/device/monitor/status?force=true').catch(e => ({ _err: e.message })),
    fetchJson('/device/events/storage_range').catch(e => ({ _err: e.message })),
    fetchJson('/device/events/index').catch(e => ({ _err: e.message })),
  ]);
  if (!store._err) storageInfo = store;
  const err = v => `<span style="color:var(--red)">${v}</span>`;
  const rows = [];
  rows.push(['Monitoring', mon._err ? err(mon._err)
    : (mon.is_monitoring ? '<b>MONITORING</b>' : 'idle')]);
  if (!mon._err) {
    rows.push(['Battery', mon.battery_v != null ? `${mon.battery_v.toFixed(2)} V` : '—']);
    if (mon.memory_total_bytes) {
      const used = mon.memory_total_bytes - (mon.memory_free_bytes ?? 0);
      const pct  = (used / mon.memory_total_bytes * 100).toFixed(1);
      rows.push(['Memory used', `${used.toLocaleString()} / ${mon.memory_total_bytes.toLocaleString()} bytes (${pct}%)`]);
    }
  }
  rows.push(['Event chain', store._err ? err(store._err) : storageUsedLabel()]);
  if (!store._err) rows.push(['Chain empty', store.is_empty ? 'yes' : 'no']);
  // SUB 0x08.  Known to report 0 on units with years of history — suspected
  // field-offset bug in the decode, so show it but do not trust it.
  rows.push(['Lifetime events', idx._err ? err(idx._err)
    : `${idx.lifetime_count} <span class="hint" style="display:inline">(unreliable — see CHANGELOG)</span>`]);
  renderTable('diag-table', rows);
  populateDeviceTab();
  if (st) { st.textContent = ''; st.className = ''; }
 }
 async function diagStopMonitoring() {
  const btn = document.getElementById('diag-stop-btn');
  btn.disabled = true; diagResult('diag-stop-result', 'Sending…');
  try {
    await fetchJson('/device/monitor/stop', { method: 'POST' });
    diagResult('diag-stop-result', 'Stop acknowledged — recording halted.', 'ok');
    refreshDiagnostics();
  } catch (e) {
    diagResult('diag-stop-result', `Failed: ${e.message}`, 'error');
  }
  btn.disabled = false;
 }
 async function diagDisableAch() {
  const btn = document.getElementById('diag-ach-btn');
  btn.disabled = true; diagResult('diag-ach-result', 'Writing call-home config…');
  try {
    const r = await fetchJson('/device/rescue?erase=false', { method: 'POST' });
    const steps = (r.steps || []).map(s => s.step).join(' → ') || 'done';
    diagResult('diag-ach-result', `ACH disabled (${steps}). Events untouched.`, 'ok');
  } catch (e) {
    diagResult('diag-ach-result', `Failed: ${e.message}`, 'error');
  }
  btn.disabled = false;
 }
 async function diagEraseEvents() {
  const serial = (unitInfo && unitInfo.serial) || 'this unit';
  if (!confirm(`Permanently erase ALL events on ${serial}?\n\nThis cannot be undone.`)) return;
  const btn = document.getElementById('diag-erase-btn');
  btn.disabled = true; diagResult('diag-erase-result', 'Erasing…');
  try {
    await fetchJson('/device/events/erase', { method: 'POST' });
    diagResult('diag-erase-result', 'Events erased — chain reset to 0x01110000.', 'ok');
    document.getElementById('diag-erase-confirm').value = '';
    eventList = []; eventsLoaded = false;
    setEventButtonsEnabled(); populateEventChips();
    refreshDiagnostics();
  } catch (e) {
    diagResult('diag-erase-result', `Failed: ${e.message}`, 'error');
  }
  diagCheckEraseConfirm();
 }
 async function diagSlowDrip() {
  const btn = document.getElementById('diag-drip-btn');
  btn.disabled = true;
  diagResult('diag-drip-result', 'Holding a session for 120 s…');
  try {
    const r = await fetchJson('/device/stop_monitoring_slow_drip?duration_s=120&interval_s=3',
                              { method: 'POST' });
    const good = (r.bytes_received || 0) > 0;
    diagResult('diag-drip-result',
      `drips ${r.drips_sent}  ·  held ${r.duration_s}s  ·  bytes back ${r.bytes_received}` +
      (r.send_error ? `  ·  ${r.send_error}` : '') +
      (good ? '  → device responded' : '  → no response; the modem may not be bridging'),
      good ? 'ok' : 'error');
  } catch (e) {
    diagResult('diag-drip-result', `Failed: ${e.message}`, 'error');
  }
  btn.disabled = false;
 }
 async function diagBlindStop() {
  const btn = document.getElementById('diag-blind-btn');
  btn.disabled = true; diagResult('diag-blind-result', 'Sending…');
  try {
    const r = await fetchJson('/device/stop_monitoring_blind', { method: 'POST' });
    diagResult('diag-blind-result',
      `Sent ${r.bytes_sent ?? '?'} bytes, no response read (fire-and-forget).`, 'ok');
  } catch (e) {
    diagResult('diag-blind-result', `Failed: ${e.message}`, 'error');
  }
  btn.disabled = false;
 }
 // ── Config form ────────────────────────────────────────────────────────────────
 function populateConfigFromDeviceInfo() {
  if (!unitInfo) return;
@@ -1,54 +0,0 @@
 """Event timestamp decode — waveform trigger/stop vs histogram window start.
 The Blastware footer holds two timestamps: ts1 = footer[2:10], ts2 = footer[10:18].
 Their meaning depends on record type:
  * Waveform: ts1 is the monitoring-SESSION start (e.g. 06:00 for a unit that
    arms at 06:00 daily — shared across every event that day), and ts2 is THIS
    event's recording STOP.  read_blastware_file used to stamp events with ts1 →
    every waveform showed the session start (~4.5 h off).  Binary-only, the best
    estimate is ts2 (the stop); the exact trigger BW displays (= ts2 - record
    duration) comes from the paired report's event_datetime, since the binary
    STRT record-time byte is a misparsed record-type marker.
  * Histogram: ts1/ts2 are the ~24 h window [start, stop]; the event time is the
    window start = ts1 (unchanged).
 """
 import datetime
 from pathlib import Path
 from minimateplus.event_file_io import read_blastware_file, apply_report_to_event
 from minimateplus.bw_ascii_report import BwAsciiReport
 from minimateplus.models import Event
 FIX = Path(__file__).parent / "fixtures"
 WAVEFORM = FIX / "fft-oracle-2026-09-14" / "N844LQHB.ZT0W"   # footer ts2 = 2026-08-25 10:33:32
 HISTOGRAM = FIX / "ts-fix" / "K441LKZU.C30H"                 # window start 2026-05-10 19:04:50
 def _tuple(ts):
    return (ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second)
 def test_waveform_timestamp_is_exact_trigger_from_binary():
    ev = read_blastware_file(WAVEFORM)
    # The EXACT Blastware trigger, from the binary alone: ts2 (stop 10:33:32)
    # minus the config record time (3.0 s) = 10:33:29 — NOT the 06:00:13
    # monitoring-session start the old decode used.
    assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 33, 29), _tuple(ev.timestamp)
 def test_histogram_timestamp_is_window_start_unchanged():
    ev = read_blastware_file(HISTOGRAM)
    # Histogram event time = the window start (ts1); must NOT get the waveform
    # ts2 treatment (that would land ~24 h off).
    assert _tuple(ev.timestamp) == (2026, 5, 10, 19, 4, 50), _tuple(ev.timestamp)
 def test_report_event_datetime_is_authoritative_over_binary():
    # The binary already yields the exact trigger, but a paired report stays
    # authoritative (e.g. if the unit clock had drifted) — applying it wins.
    ev = read_blastware_file(WAVEFORM)
    assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 33, 29)   # exact, from binary
    apply_report_to_event(ev, BwAsciiReport(
        event_datetime=datetime.datetime(2026, 8, 25, 10, 35, 0)))
    assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 35, 0)    # report wins
Author	SHA1	Message	Date
serversdown	d2258dad4e	Merge pull request 'Release v0.31.0 — report parity + the inverted rescue (0.29.0 → 0.31.0)' (#40 ) from dev into main Reviewed-on: #40	2026-09-18 16:46:41 -04:00
serversdown	e050e602c2	Merge pull request 'Release v0.29.0 — offset detector + false_trigger_reason + BlastMate serials (0.27.0→0.29.0)' (#36 ) from dev into main Reviewed-on: #36	2026-09-07 15:57:37 -04:00
serversdown	fc1c7c7936	Merge pull request 'Docs/claude.md corrections' (#35 ) from dev into main Reviewed-on: #35	2026-08-29 15:48:33 -04:00
serversdown	7d0d12079b	Merge pull request 'v0.27.0 Decoder fixes, offset exploration and testing.' (#34 ) from dev into main Reviewed-on: #34	2026-08-28 22:40:34 -04:00
serversdown	5203aab849	Merge pull request 'update to 0.26.0. Big chonking update including 0.23, 0.24, and 0.25 as well.' (#33 ) from dev into main Reviewed-on: #33	2026-08-27 13:43:00 -04:00
serversdown	5b65718b72	Merge pull request 'v0.23.0 - ZC freq in events store' (#31 ) from dev into main Reviewed-on: #31	2026-08-06 12:12:19 -04:00
serversdown	483762607e	Merge pull request 'Update to v0.22.0' (#30 ) from dev into main Reviewed-on: #30	2026-07-03 15:35:20 -04:00
serversdown	d0b66368d5	Merge pull request 'update to v0.21.1, thor data import successful' (#29 ) from dev into main Reviewed-on: #29	2026-06-01 16:54:23 -04:00
serversdown	2eb1d25028	Merge pull request 'v0.20.0 -- Full s3 event parse and PDF creation.' (#28 ) from dev into main Reviewed-on: #28	2026-05-28 17:54:31 -04:00
claude	cc821f9ee3	hotfix: fix dockerfile on main to fix import bug on prod	2026-05-21 20:42:15 +00:00