Merge pull request 'Release v0.31.0 — report parity + the inverted rescue (0.29.0 → 0.31.0)' (#40) from dev into main

Reviewed-on: #40

CHANGELOG.md

@@ -4,68 +4,6 @@ All notable changes to seismo-relay are documented here.
 
 ---
 
-## Unreleased
-
-### Fixed
-
-- **Waveform event times were the monitoring-session start, not the trigger
-  (~hours off).**  `read_blastware_file` stamped events with footer `ts1`, which
-  for a waveform is the session start a unit shares across every event that day
-  (a unit arming at 06:00 stamped 06:00 on all of them — the modal and PDF both
-  showed it, since it's the stored value).  The event time is footer `ts2` (the
-  recording stop), and Blastware's trigger = `ts2 - record time`.  The record
-  time is a big-endian float32 in the recording-setup config block (30 bytes
-  before the `Standard Recording Setup` marker), so the **exact trigger is now
-  recovered from the binary alone** — all 7 BE12844 oracle events decode to
-  their exact Blastware time (e.g. N844LQHB 10:33:29), no paired `.TXT` needed.
-  Histograms keep `ts1` (the ~24 h window start).  A paired report's
-  `event_datetime` stays authoritative (unit-clock drift).
-  ⚠ **Needs a re-decode backfill** to correct existing stored events' timestamps.
-
-### Added
-
-- **Diagnostics tab in the SFM standalone webapp.**  Surfaces the device
-  endpoints that previously existed only as `curl`: `events/storage_range` and
-  `events/index` alongside `monitor/status`, then stop monitoring, disable ACH
-  (`rescue?erase=false`, so stored events survive), and erase.  The wedged-unit
-  ladder — slow drip and blind stop — sits under its own heading pointing at
-  `docs/runbooks/wedged_unit_recovery.md`, with the reminder that `slow_drip`'s
-  success signal is `bytes_received > 0` and not a clean duration.  Erase is
-  guarded by typing the unit's serial: auth answers *who*, not *did you mean
-  it*, and Swagger's try-it-out button on `/device/events/erase` is live on
-  `:8200/docs`.
-
-- **`docs/sfm_tool_status.md`** — an honest per-capability maturity assessment:
-  what is production-grade (the codec library, the data side), what is
-  emergency-grade (the device side), what is a research artifact, the
-  known-issues table, and the gap to a real tool.  Also records the **5A
-  page-boundary bug** as known: `parse_strt_end_offset()` discards the key's
-  page byte, so once a unit has recorded more than 64 KB since its last erase,
-  an event spanning the boundary reads an `end_offset` *behind* its own start —
-  the chunk loop fetches nothing and TERM packs a negative `offset_word`, which
-  500s.  Reproduced on BE12599.  Production is unaffected: it ingests complete
-  files via the watcher path and never runs this walk.
-
-### Changed
-
-- **Connecting to a unit no longer walks its event chain.**  `/device/events`
-  reads every event header over the cellular link; on a unit with a large or
-  wrapped chain that takes minutes or fails outright, and it fired
-  automatically on every connect.  Connect now uses only ~2 s probes —
-  `/device/info` (which already carried the compliance config the walk was
-  re-reading) plus `events/storage_range` — and the Device tab gains an Event
-  Chain card.  The walk moved behind a **Load events** button in the Events
-  toolbar.  Knowing whether a unit's ACH is on no longer requires reading every
-  event it has stored.
-
-### Migration
-
-**None.**  Frontend and documentation only — no codec, waveform-store or DB
-change, no schema change, and no `TOOL_VERSION` bump.  The webapp is served
-from the image, so the change appears after the next `sfm` rebuild.
-
----
-
 ## v0.31.0 — 2026-09-18
 
 **Report parity, and a second way to rescue a runaway unit.**  Two threads.

CLAUDE.md

@@ -61,24 +61,6 @@ Read this first when picking the project back up.
   4th-decimal tick and are Thor's own rounding — no single linear LSB can
   reproduce every printed value (the constraints are infeasible by 7e-5
   relative), so do NOT retune `_GEO_LSB_IPS`.
-- **⚠ KNOWN BUG — the 5A walk breaks once a unit's buffer crosses 64 KB.**
-  `parse_strt_end_offset()` returns only `(end_key[2] << 8) | end_key[3]`,
-  discarding the key's page byte.  An event starting at `0x0111F2A2` and ending
-  at `0x0112_1010` therefore reads `end_offset = 0x1010` — *behind* its own
-  start.  The chunk loop then exits before fetching anything and TERM computes
-  a negative `offset_word`, which `struct.pack(">H", ...)` rejects: the
-  `/device/events` walk 500s.  Reproduced on BE12599 (2026-09-19), which had
-  78 KB stored and had rolled into page `0x12`.
-  **Why it hid so long:** every 5A capture the walk was verified against came
-  from a freshly-erased BE11529 — all three confirmed TERM examples in
-  `framing.py` (`0x1ABE`, `0x21F2`, `0x417E`) sit inside page `0x11`.  Prod is
-  unaffected: it ingests complete files via BW ACH, never this walk.
-  **Fixing it has two layers** — the arithmetic (`if end < start: end +=
-  0x10000`) stops the crash and bounds the loop correctly; carrying the page
-  byte through the chunk requests (`params[1]` 0x11 -> 0x12, counter rolling
-  over) needs a BW capture of a spanning event first.  Do not ship layer one
-  alone without a loud truncation warning — a silently short event is the
-  failure mode this codec has been bitten by repeatedly.
 - **Open, not blocking:** 14 sensitive-range files show an exact 8x
   (= 10.0/1.25) units discrepancy; `scripts/backfill_sidecars.py --force` also
   inserts DB rows for store files that have none (one-time per store) and the

README.md

@@ -496,11 +496,6 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
 
 ## Roadmap (Future)
 
-> **Where it stands *today*** — an honest per-capability maturity assessment,
-> what to rely on, known issues, and the gap to a real tool:
-> [`docs/sfm_tool_status.md`](docs/sfm_tool_status.md).  This section covers
-> where it is *going*.
-
 ### Strategic direction — where this is going
 
 seismo-relay is being built as a **suite of cooperating components**

docs/micromate_protocol_reference.md

@@ -1,564 +0,0 @@
-# Micromate Protocol Reference — Thor / Micromate Series IV, live wire protocol
-
-Sibling to [instantel_protocol_reference.md](instantel_protocol_reference.md)
-(Series III, "the Rosetta Stone") and
-[idf_protocol_reference.md](idf_protocol_reference.md) (Series IV *file*
-format).  This document covers the Series IV **live device protocol** — what
-the unit says over the wire, as opposed to what it writes into a `.IDFW`.
-
-**Status (2026-09-23): opening session.** Everything below was established in
-a single bench session against one unit.  Treat it as a strong start, not a
-settled spec — in particular, everything here comes from **one unit, over USB,
-with no events stored**.
-
----
-
-## The headline
-
-**A Micromate running the *Blastware* firmware answers Series III command
-frames.**
-
-An unmodified Series III `POLL` (`SUB 0x5B`), built by
-`minimateplus.framing.build_bw_frame` with no changes at all, produced a
-complete two-step probe/data cycle.  Ten Series III read commands were then
-tried and **all ten answered**, every one obeying the Series III response-SUB
-rule.
-
-⚠ **That qualifier is load-bearing, and it was discovered after the fact.**
-Instantel ships the Micromate in two firmware lines:
-
-| firmware | Instantel's own description |
-|---|---|
-| `11.0CB` | "Utilize with **Blastware**" |
-| `11.0BD` | "Utilize with **THOR, Vision, Vision II**" |
-
-The bench unit reports **`11.0CB`** — the Blastware build.  So the clean
-Series III behaviour above is very likely *because the unit is in Blastware
-mode*, not because the Micromate natively speaks Series III.  **Nothing here
-should be assumed to hold on a `11.0BD` unit until tested.**
-
-This reframes the project.  The question is no longer only "what is the
-Series IV protocol" but **"which firmware line do we target, and does one of
-them let the existing Series III stack drive the whole fleet?"**
-
----
-
-## Firmware — the variable nobody knew was a variable
-
-This explains a production problem TMI has lived with: two ACH servers, two
-machines, and units that will not cross over.  It is not a misconfiguration.
-**Instantel ships different firmware for different host software**, and the
-wire protocol differs with it.
-
-### Fleet audit (physical, 2026-09-22/23)
-
-| unit | firmware | line | location |
-|---|---|---|---|
-| UM11719 | `11.0CB` | Blastware | Ped Bridge Loc 2 |
-| UM6047 | `11.0CB` | Blastware | Brookville Loc 9 |
-| UM12947 | `11.0CB` | Blastware | **bench** |
-| UM14133 | `11.0CB` | Blastware | Pitt-Music Bldg Loc 1 |
-| UM11402 | `11.0BD` | Thor | Ped Bridge Loc 1 |
-| UM20147 | `11.0BD` | Thor | **bench** |
-| UM13981 | `11.0AK` | pre-split | RKM Loc 1 |
-| UM20146 | `11.0AK` | pre-split | Karns Loc 2 |
-| UM12420 | `10.90GC` | pre-split | RKM Loc 2 |
-
-**4 Blastware / 2 Thor / 3 pre-split.**  The Blastware line is already the
-plurality, which makes "standardise on Blastware" less disruptive than it
-first appeared.
-
-A store-derived audit (firmware is recorded in every `.sfm.json` sidecar as
-`extensions.idf_report.version`) agreed with the physical audit on **7 of 9**.
-The two that differed — UM6047 and UM14133 — are the most recently deployed
-units, reflashed after their last stored event.  Useful technique: the fleet's
-firmware history is reconstructable from the store without touching a unit,
-but it lags reality by one deployment.
-
-Firmware is **not stable per-unit over time** — five of nine have been
-reflashed at least once.  Any fleet-wide claim needs a fresh audit.
-
-### ⚠ Retraction: firmware line does NOT determine Thor compatibility
-
-An earlier draft of this document suggested that UM12947's trouble with Thor
-was explained by its being on the Blastware build.  **That is not supported.**
-
-Ped Bridge runs UM11402 (`11.0BD`) and UM11719 (`11.0CB`) side by side, both
-deployed 2026-04-20, and **both call Thor successfully** — UM11719 has 331
-Thor-collected events in the store while on `11.0CB`, through 2026-08-23.
-
-So a Blastware-line unit does feed Thor.  Whatever the CB/BD split changes, it
-is not "which host software can collect from it", and UM12947's specific
-problem remains unexplained.
-
-**What is actually established:** a `11.0CB` unit answers Series III command
-frames.  Whether a `11.0BD` unit does is **untested** — and UM20147 (`11.0BD`)
-is on the bench, which makes that a direct A/B away.
-
-The version is readable over the wire from `SUB 0x01` as two separate ASCII
-runs — `"0CB"` then `"11"` — with no single concatenated string.
-
-### The strategic fork
-
-- **Option A — standardise the fleet on the Blastware line.**  Every unit,
-  MiniMate and Micromate alike, then speaks Series III, and the existing
-  `minimateplus/` stack drives all of it.  One protocol, one call-home
-  receiver.  Dramatically cheaper *if* it holds up.
-- **Option B — reverse-engineer the Thor line (`11.0BD`) and support both.**
-
-Option A is the shortcut, but it is unproven and carries real unknowns, all
-of which are cheap to answer on the bench and expensive to discover later:
-
-1. **What file format does a `11.0CB` unit produce?**  If it emits Blastware
-   binaries rather than `.IDFW`/`.IDFH`, the (now exact) Series III decoder
-   applies and the IDF codec becomes a legacy path.  Not a loss — we have
-   both — but it changes what the ingest pipeline sees.
-2. **Does Thor still work with a `11.0CB` unit?**  If not, flashing a
-   production unit breaks data collection until the replacement path exists.
-3. **Are any Micromate-specific capabilities lost** on the Blastware line?
-4. **Is the flash reversible in the field**, and what does it cost in downtime?
-
-## How this was obtained
-
-No Thor, no modem, no Windows machine.  The Micromate exposes its protocol on
-a **USB CDC-ACM virtual serial port**:
-
-```
-ID 2504:0300 Instantel Inc. MICROMATE COM PORT
-driver: cdc_acm          ATTRS{serial}=="V1.00"
-→ /dev/ttyACM0
-```
-
-Plain CDC ACM, so no vendor driver and no proprietary USB layer — any host
-that can open a serial port can talk to the unit.
-
-**Baud is irrelevant over USB.**  Identical byte-for-byte responses at 38400
-and 115200; CDC-ACM ignores the line rate.  TMI provisions Micromate *modem*
-links at **115200** (Series III uses 38400) — that matters for the cellular
-path, not for USB.
-
-**The device never speaks first.**  20 s of passive listening on an idle open
-port produced zero bytes.  It is strictly request/response.
-
----
-
-## Physical + framing layer
-
-### Requests — Series III format, unmodified
-
-Every frame in this session was produced by `build_bw_frame(sub, offset)` with
-no Series IV changes, and the device accepted all of them:
-
-```
-[ACK 0x41] [STX 0x02] [10 10] [flags 00] [SUB] [00] [00] [offset] [params×10] [chk] [ETX 0x03]
-```
-
-⚠ Only the doubled `BW_CMD` (`10 10`) form has been exercised.  Whether other
-literal `0x10` bytes inside params require stuffing is **untested** — none of
-the probes sent carried one.
-
-### Responses — Series III *minus the DLE prefix*
-
-```
-Series III:   [DLE 0x10] [STX 0x02] … [chk] [ETX 0x03]
-Micromate:              [STX 0x02] … [chk] [ETX 0x03]      ← no leading DLE
-```
-
-This single byte matters operationally: Blastware's parser locates frames by
-scanning for `DLE+STX`, so it will **never find a frame boundary** in Micromate
-traffic no matter what else is correct.  That is a structural reason a
-Micromate cannot call into a Blastware ACH server, independent of any baud
-mismatch.
-
-### Response payload header
-
-```
-[0] CMD    0x00          same as Series III
-[1] flags  0xC5          ← Series III uses 0x10.  Constant across all 10 SUBs.
-[2] SUB    0xFF − request_SUB
-[3] PAGE_HI
-[4] PAGE_LO
-[5+] data
-```
-
-### Checksum — the DLE-aware variant
-
-```python
-chk = sum(b for b in payload if b != 0x10) & 0xFF
-```
-
-Confirmed on every frame captured.  The `POLL` probe response contains no
-`0x10` and so cannot distinguish plain SUM8 from the DLE-aware form; the
-`POLL` **data** response contains a `0x10` at payload offset 42, and only the
-DLE-aware rule matches there.  This is the same checksum Series III uses for
-its `5A` bulk-stream and write frames — not the plain SUM8 of ordinary
-Series III reads.
-
-### The probe response carries the data length
-
-Series III hardcodes `DATA_LENGTHS` per SUB.  On the Micromate the **probe
-response tells you**, at `payload[9]`:
-
-| SUB | command | `payload[9]` | Series III constant |
-|---|---|---|---|
-| `0x15` | serial number | `0x0A` | `0x0A` ✓ |
-| `0x01` | device info | `0x98` | `0x98` ✓ |
-| `0x1C` | monitor status | `0x2C` | `0x2C` ✓ |
-| `0x06` | storage range | `0x24` | `0x24` ✓ |
-| `0x2C` | call-home config | `0x7E` | `0x7C` ✗ **differs by 2** |
-| `0x08` | event index | `0x5A` | — |
-| `0x1E` | event header | `0x08` | — |
-| `0x1A` | compliance config | `0x2C` | — |
-| `0x0A` | waveform header | `0x00` | — (no event context) |
-| `0xFE` | full config | `0x00` | — (see note) |
-
-Four of four known Series III lengths match exactly.  **Read the length from
-the probe rather than hardcoding it** — it is free, and it already caught the
-call-home divergence.
-
----
-
-## Confirmed commands (read-only)
-
-All ten below answered with a correct `0xFF − SUB` response.  Nothing that
-writes, erases, or changes monitoring state has been sent to a unit.
-
-| SUB | RSP | Command | Data proven |
-|---|---|---|---|
-| `0x5B` | `0xA4` | POLL / handshake | yes — ID block |
-| `0x15` | `0xEA` | Serial number | yes — `UM12947` |
-| `0x01` | `0xFE` | Device info | yes — 152 B |
-| `0x2C` | `0xD3` | Call-home config | yes — 126 B |
-| `0x1C` | `0xE3` | Monitor status | yes — 44 B |
-| `0x06` | `0xF9` | Event storage range | yes — 36 B |
-| `0x08` | `0xF7` | Event index | probe only |
-| `0x1E` | `0xE1` | Event header / first key | probe only |
-| `0x0A` | `0xF5` | Waveform header | probe only |
-| `0x1A` | `0xE5` | Compliance config | probe only |
-
-### Decoded so far
-
-**`SUB 0x15` — serial.** ASCII, null-terminated: `UM12947`.
-
-**`SUB 0x5B` / `0x01` — identification strings.**
-`Instantel\0` and `MM/ISEE/S/IO` (MicroMate / ISEE standard).  `0x01` also
-carries eight consecutive `3f 80 00 00` float32 values (= `1.0f`) — almost
-certainly per-channel calibration/scale factors, by analogy with Series III's
-`geo_hardware_constant`.  **Unverified.**
-
-**`SUB 0x1C` — monitor status.  Series III field offsets apply unchanged:**
-
-| field | offset | read |
-|---|---|---|
-| battery × 100 | `payload[-10:-8]` uint16 BE | `0x017D` → **3.81 V** |
-| memory total | `payload[-8:-4]` uint32 BE | 15,000,000 |
-| memory free | `payload[-4:]` uint32 BE | 15,000,000 (empty) |
-| date | `payload[18:22]` | day 23, month 9, year `0x07EA` = 2026 |
-
-The battery reading independently corroborates: Thor's own event reports for
-these units print `BatteryLevel : 3.8 volts`.
-
-**`SUB 0x2C` — call-home config.** Contains the ASCII string `RADIO RING`.
-Worth flagging: that is the *exact* string seen in the RV50 `ALEOS_SERIAL`
-debug during the BE12599 incident —
-`'ATQ1^MATE0^MATS0=2^M^MRADIO RING^M'`.  So this block holds the modem dial /
-answer strings, and it is the most directly relevant command to the
-call-home-receiver goal.  Field layout **not yet mapped**; Series III's map
-(`raw[5]` enabled, `raw[6:46]` dial string) is a starting hypothesis only, and
-the length already differs (`0x7E` vs `0x7C`).
-
-**`SUB 0x06` — storage range.** All zeros on this unit, consistent with
-`memory free == memory total`.  Series III reads first/last event keys from
-the final 8 bytes; untestable until the unit holds events.
-
----
-
-## The event chain — walked end to end (2026-09-23, 5 events)
-
-With 5 events on the bench unit (4 waveform + 1 histogram), the Series III
-browse walk works unmodified:
-
-```
-1E (all-zero params)        -> first key + size
-  0A (key)                  -> partial record, histogram only
-  0C (key)                  -> 210-byte waveform record
-1F (all-zero params/browse) -> next key + size
-  ... repeat ...
-1F                          -> all-zero key = NULL SENTINEL, chain ends
-```
-
-The sentinel terminated correctly after exactly 5 events.
-
-### Event keys are sequential, not addresses
-
-```
-055d4a81  055d4a82  055d4a83  055d4a84  055d4a85
-```
-
-**This is a real divergence.** Series III keys are flash-buffer *addresses*
-(`01110000`, `011121F2`, …) that advance by the event's byte length, which is
-why its 5A chunk walk is address-arithmetic. Micromate keys are a plain
-incrementing counter. Any port of the Series III download walk must not
-assume key arithmetic means anything.
-
-### The 4 bytes after the key are the event's size
-
-`1E`/`1F` return `[key 4B][size 4B]`. Series III uses that slot as an offset
-to the next key; here it is a byte count:
-
-| key | size | kind |
-|---|---|---|
-| `055d4a81` | 4,076 | histogram |
-| `055d4a82` | 11,032 | waveform |
-| `055d4a83` | 11,502 | waveform |
-| `055d4a84` | 13,424 | waveform |
-| `055d4a85` | 8,746 | waveform |
-
-Consistent with real file sizes (corpus `.IDFH` ≈ 3.7–25 KB, `.IDFW` ≈
-8.6–15.8 KB), and the histogram is unmistakably the small one.  ⚠ Inferred,
-not proven: the sizes sum to 48,780 while monitor status reports 57,344 bytes
-used, so ~8.5 KB of overhead is unaccounted for.
-
-### `SUB 0x0C` — waveform record, and it carries the job metadata
-
-**Length `0xD2` = 210 bytes — identical to Series III.**  Contents confirmed
-across all 5 events:
-
-- the event key, echoed
-- date + time (`17 09 07 ea` → 23 Sep 2026, then `10 21` → 16:33 — matching
-  the actual bench recording time)
-- title note `"Location"`
-- **the project string** — `"Univ of Pitt-1st Yr Housing-Loc1 Ruskin"`
-- serial `"UM12947"`
-- channel labels `Tran` / `Vert` / `Long` / `Mic` — the same labels Series III
-  uses, and the same label-relative float32 layout
-- per-event float32 peaks: 3.5152, 1.3720, 2.3542, 3.5152, 0.4227 in/s
-  across the five events (varied deliberately during recording)
-
-**This closes the biggest open question for the call-home-receiver goal.**
-The job identity strings (`project` / `client` / `operator` / `setup`) that
-today arrive only via Thor's `.txt` sidecar — and which no amount of sample
-decoding can reconstruct — are **available over the wire from `0x0C`**.  A
-direct-to-SFM event need not arrive with blank metadata.
-
-### `SUB 0x0A` — partial record, histogram only
-
-`0x0A` returned `len = 0x1E` (30 B) for the histogram and `len = 0x00` for all
-four waveforms.  The histogram payload carries **two timestamps** and the
-ASCII string `"\r Vert: 0.300 in/s"` — structurally the Series III
-**monitor-log partial record** (`0x2C` type), which likewise holds a start/stop
-pair and a `"Geo: <float> in/s"` trigger string.
-
-So on the Micromate the division of labour is: `0x0A` describes interval-style
-records, `0x0C` describes triggered events.  Series III uses `0x0A`'s
-*response length* (`0x46` vs `0x2C`) to tell real events from boundaries;
-that discriminator does not apply here.
-
-### DLE stuffing in responses — confirmed present
-
-Earlier marked untested.  The `0x0C` timestamp field contains `10 10`, which
-destuffs to a single `0x10` and yields a sensible clock reading.  **Responses
-are DLE-stuffed**, so a parser must destuff before applying field offsets.
-
----
-
-## A/B: Blastware build vs Thor build (2026-09-23)
-
-UM12947 (`11.0CB`) and UM20147 (`11.0BD`) were each put on the bench and given
-the identical read-only sweep.  **Both answer Series III command frames.**
-
-| | UM12947 `11.0CB` | UM20147 `11.0BD` |
-|---|---|---|
-| POLL answers | ✓ | ✓ |
-| All 10 read SUBs answer | ✓ | ✓ |
-| `response_SUB = 0xFF − req` | ✓ | ✓ |
-| DLE-aware checksum valid | ✓ | ✓ |
-| Two-step probe/data read | ✓ | ✓ |
-| ID string | `MM/ISEE/S/IO` | `MM/ISEE/S` |
-| **flags byte** | **`0xC5`** | **`0x03`** |
-| **`0x1C` length** | **`0x2C`** | **`0x30`** |
-
-**The firmware line does not change the wire protocol.**  One protocol stack
-can drive the whole fleet regardless of which build a unit is on.  This is the
-single most consequential finding so far: "standardise the fleet on one
-firmware" becomes an *optional* convenience rather than a prerequisite for
-building a call-home receiver.
-
-### The two differences that do exist
-
-**1. The flags byte identifies the build.**  Response `payload[1]` is `0xC5`
-on the Blastware line and `0x03` on the Thor line, constant across all ten
-SUBs on both units.  That makes firmware line detectable from *any* response,
-without reading device info.  ⚠ Two units, one each — treat as a strong
-hypothesis, not a proven encoding.
-
-Note `0x03` is ETX, so on Thor-line units it arrives DLE-escaped as `10 03`.
-A parser that fails to destuff will mis-locate every field by one byte on
-exactly half your fleet.
-
-**2. `SUB 0x1C` (monitor status) is 4 bytes longer on the Thor line** —
-`0x30` vs `0x2C` — with four extra trailing bytes (`0f a0 00 00`, purpose
-unknown).
-
-⚠ **This breaks relative-to-end parsing.**  Series III reads battery and
-memory from the *end* of the `0x1C` block (`[-10:-8]`, `[-8:-4]`, `[-4:]`).
-Those offsets are correct on `11.0CB` and wrong on `11.0BD` — applying them
-blindly to UM20147 yields a battery reading of **577.92 V**.  Parse forward
-from the declared length instead of backward from the end.
-
-With the offsets shifted by 4, UM20147 reads correctly: battery **3.81 V**,
-memory 15,000,000 total and free (no events stored).
-
-## Divergences from Series III (running list)
-
-1. **No `DLE` prefix on responses** — bare `STX`.
-2. **Response flags byte is `0xC5`**, not `0x10`.
-3. **Call-home config is 126 bytes**, not 124.
-4. **Data lengths are discoverable** from the probe response at `payload[9]`.
-4b. **Event keys are a sequential counter**, not flash addresses.
-4c. **`1E`/`1F` return the event size**, where Series III returns an offset.
-4d. **`0x0A` vs `0x0C` split by record type**, not by the `0x46`/`0x2C`
-    length discriminator Series III uses.
-4e. **Response `payload[1]` (flags) encodes the firmware line** — `0xC5`
-    Blastware, `0x03` Thor — where Series III has a constant `0x10`.
-5. **Modem serial rate is 115200**, not 38400 (per TMI provisioning practice;
-   not independently verified here).
-
----
-
-## The firmware images are unencrypted — and they document themselves
-
-`ref-stuff/micromate-firmware/MICROMATE(CB).BIN` and `MICROMATE(BD).BIN`,
-~2.77 MB each and within 192 bytes of one another.
-
-- **Entropy 6.08 bits/byte** — neither encrypted nor compressed.  Plain code
-  and data.
-- Header is a **big-endian vector table**, handlers at `0x4010_30xx`.
-- **~16,700 extractable strings**, including the developers' own debug
-  `printf` format strings with function names intact.
-
-This is a legitimate interoperability reference for hardware TMI owns, and it
-short-circuits work I had scoped as "only answerable from a live modem
-capture".
-
-### The call-home state machine, verbatim
-
-```
-ACH_NOT_STARTED → ACH_IDLE → ACH_INITIALIZING → ACH_CONNECTING
-                → ACH_CONNECTED → ACH_TRANSFER_DATA
-                → ACH_RETRY / ACH_QUITTING          (also ACH_STARTED)
-```
-
-Supporting strings:
-
-```
-ACH: Entry StartCallHome()
-ACH: CallHome_task ; CheckAliveTime  CANCEL ; TimeBetweenRetries = %d
-ACH: CallHome_task ;  !ExpectedCommunicationsDetected() CANCEL ; TimeBetweenRetries = %d
-ACH: CallHomeCommectionCompleteProcessing() ACH=%s EAMWC=%s
-ACH: %s() three attempts and it's over
-ACH: %s() Send CMD_START_MONITOR
-ACH: %s() Send CMD_STOP_MONITOR
-ACH: Start Ignore request, Call Home is in progress
-```
-
-What this tells us without a single captured packet:
-
-1. **Retry limit is three** — "three attempts and it's over".
-2. **`ExpectedCommunicationsDetected()` gates the session.**  If the host does
-   not say something the unit recognises, the call is *cancelled* and
-   rescheduled after `TimeBetweenRetries`.  A homebrew receiver must satisfy
-   this check or units will retry forever — which is exactly the failure mode
-   seen on BE12599.
-3. **The unit stops monitoring to call home and restarts afterwards**
-   (`Send CMD_STOP_MONITOR` / `CMD_START_MONITOR`).  Relevant to any
-   wedged-unit rescue: the monitoring state around a call is the device's own
-   doing, not ours.
-4. **Calls are not re-entrant** — "Call Home is in progress" is ignored.
-
-### Internal command table
-
-`CMD_CALLHOME`, `CMD_CALLHOME_CANCEL`, **`CMD_CALLHOME_CONNECTION_CONFIRMED`**,
-`CMD_CALLHOME_CONNECTION_COMPLETE`, `CMD_CALLHOME_SET_SCHEDULE`,
-`CMD_CALLHOME_CLEAR_SCHEDULE`, `CMD_STOP_CALLHOME_FILETRANSFER`,
-`CMD_DUTYCYCLE_AUTOCALLHOME`, `CMD_PURGE_EVENT_FLASH`.
-
-`CONNECTION_CONFIRMED` as a distinct state from `CONNECTION_COMPLETE` implies
-a **handshake the host must complete before data flows** — the concrete shape
-of `ExpectedCommunicationsDetected()`.
-
-### Event delivery — the mechanism, probably
-
-```
-All Events Uploaded
-Mark/Unmark File          Delete Marked Events        Marked Events were Deleted
-MONITOR::MESG PURGE_EVENT_FLASH BEGIN / END
-```
-
-A **marking** mechanism exists, alongside a distinct "all uploaded" terminal
-state.  🔶 **Inferred:** events are *marked* as transferred rather than
-deleted on send, and purging is a separate explicit act.  If so, a receiver
-that fails to mark would see the same events re-offered every call — the
-question that gates a safe homebrew receiver.  **Not yet confirmed**; needs
-either a live call-home capture or disassembly around these strings.
-
-### A full user manual is embedded
-
-The firmware carries its own HTML help, which documents configuration we would
-otherwise have to infer:
-
-- **Modem mode**: `Generic` (through a modem) vs `USB to PC`.
-- **Modem baud**: 9600 / 19200 / 38400 / 57600 / 115200 / 230400 — "must match
-  the expected rate of the PC or modem".  Confirms 115200 is a *setting*, not
-  a fixed rate.
-- **Modem relay + warmup** (0–300 s), auxiliary mode, warning/alarm hold.
-- **Record modes**: Waveform, Waveform Manual, Histogram, Histogram-Combo;
-  sample rates 1024 / 2048 / 4096.
-- **A scheduler downloaded from THOR** that can start/stop monitoring, change
-  record mode, trigger a call home, or run a self check on a daily/weekly
-  schedule.  Pairs with `CMD_CALLHOME_SET_SCHEDULE`.
-
-### Still worth doing
-
-Diffing the two images should isolate exactly what the CB/BD split changes —
-we know the wire protocol is not it, and the flags byte (`0xC5` vs `0x03`)
-gives a concrete anchor to search for.
-
-## ⚠ Untested and unsafe-until-agreed
-
-Nothing below has been sent to a unit, and nothing should be without an
-explicit decision:
-
-- **Writes** (`0x68`–`0x83`), **call-home write** (`0x7E`/`0x7F`)
-- **Erase** (`0xA3` / `0xA2`)
-- **Start / stop monitoring** (`0x96` / `0x97`)
-- `0x1F` (advance event pointer) — non-destructive on Series III but it does
-  move device state, so it is parked with the rest
-
-Also unknown:
-
-- Whether `0x10` bytes inside request params need stuffing
-- Whether the bulk waveform stream (`5A` on Series III) exists here, and
-  whether it is the transport for `.IDFW` bodies we already decode
-- Everything about the **call-home session** — the device-initiated direction
-  has not been observed at all.  Specifically: how a unit announces itself,
-  and **how it learns an event was accepted so it stops re-sending it.**
-  That last question gates any homebrew receiver and cannot be answered over
-  USB.
-
----
-
-## Session provenance
-
-Unit **UM12947**, firmware **`11.0CB`** (Blastware line), on the bench via
-USB, **zero events stored** (memory free == total).  Read commands only.
-Every response in this document was checksum-validated.
-
-⚠ Firmware is the single biggest caveat on this document.  Every finding here
-is from one unit on the Blastware build.  A `11.0BD` unit has not been
-touched.
-
-An empty unit is a real limitation: `0x08`, `0x1E`, `0x0A` and `0x06` all have
-event-dependent payloads that could not be exercised.  Recording a couple of
-events on the bench unit would unlock the entire event-walk half of the
-protocol.

docs/sfm_tool_status.md

@@ -1,150 +0,0 @@
-# SFM — where it actually stands as a tool
-
-**Status as of 2026-09-20 (v0.31.0).**  This is the honest assessment, not the
-roadmap — `README.md § Roadmap` covers where it is *going*.  Expect this file to
-go stale; re-date it when you revise it.
-
----
-
-## The framing
-
-SFM is **three different things wearing one name**, at three very different
-levels of maturity:
-
-| | what it is | maturity |
-|---|---|---|
-| **The codec library** | `minimateplus/`, `micromate/` — bytes in, `Event` out | **Production.** Verified per-sample at scale. |
-| **SDM — the data side** | the DB, waveform store, `/db/*`, ingest | **Production.** Terra-View depends on it daily. |
-| **SFM — the device side** | `/device/*`, live connections to units | **Emergency-grade.** Works, but manual, unauthenticated, and thinly tested. |
-| **The lab** | `seismo_lab.py`, `scratch/`, the Inspector | **Research artifacts.** Useful, not products. |
-
-Brian's own description — *"right now it's an emergency tool and a research
-project"* — is accurate, and it applies specifically to the **device side**.
-The data side is not an emergency tool; it has been carrying production for
-months.
-
-Most confusion about "is SFM reliable?" comes from answering for the wrong
-tier.
-
----
-
-## 1. What you can rely on
-
-### Production-grade — trust it
-
-- **Series-3 decode.** 14,338 / 14,338 files decode per-sample exact against
-  preserved Blastware ASCII exports, 45 units, files back to 2018.
-- **Series-4 (Thor) decode.** 1,057,536 / 1,057,536 geo samples exact against
-  Thor's own CSV exports; production IDFW 575/575 with zero truncations.
-- **Histogram decode.** 1,211 / 1,211 production histograms exact, including
-  842,442 per-interval frequency comparisons with zero mismatches.
-- **The ingest path.** `/db/import/blastware_file` and `/db/import/idf_file`
-  fed by the watchers — this is how prod actually gets its data, and it has
-  been running unattended for months.
-- **`/db/*` read API.** Always-on, consumed by Terra-View for every fleet
-  listing, event detail and report.
-- **The waveform store** — `.h5` + `.sfm.json` sidecars + retained raw
-  binaries, with operator review state preserved across regeneration.
-- **`bridges/ach_server.py`** — speaks the full BW protocol to calling units.
-  Proven in the field, including as a rescue tool (see the runbook).
-
-### Emergency-grade — works, but you are the error handling
-
-- **`/device/*` live endpoints.**  They do what they say.  But they are
-  synchronous, unauthenticated, and a single cellular download can exceed the
-  60 s timeouts that sit in front of them.
-- **The rescue ladder** (`rescue`, `stop_monitoring_*`, `events/erase`).
-  Each has worked in a real incident — but each has been used a handful of
-  times, by one person, with the runbook open.
-- **The standalone webapp.**  Perfectly usable, and as of v0.31.0 the cheap
-  probes and rescue actions are reachable without curl.  No auth of any kind.
-
-### Research artifacts — useful, not products
-
-- **`seismo_lab.py`** — 2,789 lines of Tkinter (Bridge / Analyzer / Query DB /
-  Inspector).  Desktop-only, single-user, no tests.
-- **`scratch/`** — the verification harnesses (`verify_against_ascii.py`,
-  `verify_thor_against_csv.py`) and the offset detector (`offset_scan3.py`).
-  These produced the numbers the production claims rest on, so they matter —
-  but they are analysis scripts, not maintained code.
-- **`docs/offset_investigation.md`** — an open investigation, not a feature.
-
----
-
-## 2. What to use when
-
-| you want to… | use | notes |
-|---|---|---|
-| Know if a unit is monitoring / its battery / memory | `GET /device/monitor/status?force=true` | ~2 s |
-| Know whether ACH is on | `GET /device/call_home` | ~2 s.  **Not** `/device/events`. |
-| See how full a unit's buffer is | `GET /device/events/storage_range` | ~2 s, no chain walk |
-| Stop a runaway unit | Diagnostics tab → Stop Monitoring | see the runbook first |
-| Reach a unit that will not answer | **point its modem at an `ach_server` and answer its call** | runbook Method A — do not race it |
-| List a unit's stored events | Events tab → Load events | **slow**, and broken past 64 KB (below) |
-| Get event data into the DB | the watcher → `/db/import/*` path | not the live walk |
-
-The single most useful habit: **the cheap probes are cheap and the event walk
-is not.**  Reaching for `/device/events` to answer a yes/no question about a
-unit is the mistake that motivated the v0.31.0 webapp changes.
-
----
-
-## 3. Known issues
-
-| issue | impact | status |
-|---|---|---|
-| **5A walk dies once a unit's buffer crosses 64 KB** | `/device/events` 500s; event body never downloads | Known, documented in `CLAUDE.md`.  Needs a BW capture of a spanning event to fix properly. |
-| **No auth on SFM at all** | 21 `/device/*` endpoints, including destructive ones, open to anything that reaches the port | Design agreed (Terra-View as authenticated jump host); not built. |
-| **Swagger try-it-out is live on destructive endpoints** | `POST /device/events/erase` is one click away at `:8200/docs` | Partially mitigated: the webapp's erase now requires typing the serial.  `/docs` itself is unguarded. |
-| **`SUB 0x08` lifetime counter reads 0** | `/device/events/index` returns a meaningless number | Suspected field-offset bug.  Surfaced in the UI as "unreliable". |
-| **Long device operations are synchronous** | 60 s timeouts in `routers/sfm.py` and the reverse proxy; a full download exceeds both | Known design constraint.  Must be POST-starts-job / GET-polls before any remote lab. |
-| **`backfill_sidecars.py --force` silently inserts DB rows** | store files with no DB row get one; the dry-run does not report the count | Known.  Avoid `--force` — `TOOL_VERSION` gates regeneration anyway. |
-| **14 sensitive-range files show an exact 8× discrepancy** | 10.0 / 1.25 — a units problem, not a decode problem | Open, not blocking. |
-| **16 failing tests on `dev`** | 15 need gitignored fixture bundles; 1 is real (`sc["peak_values"]["transverse"]` returns `None` where `0.0` is expected) | The real one shipped in v0.31.0. |
-
----
-
-## 4. What stands between this and a real tool
-
-Roughly in dependency order — each unblocks the ones below it.
-
-**1. Authentication.**  Everything else is gated on this.  SFM has none, and
-the modem IP whitelist gives zero protection because SFM *is* the whitelisted
-origin.  The agreed design delegates rather than builds: Terra-View becomes the
-authenticated jump host (`/api/sfm/*` already inherits deny-by-default operator
-auth), and the `8200:8200` publish is dropped so Terra-View is the only door.
-
-**2. Async long operations.**  POST starts a job, GET polls.  Retrofitting this
-after building a remote lab on top of synchronous endpoints would be far worse
-than designing for it now.
-
-**3. Confirm-guards on the remaining destructive endpoints.**  Auth answers
-*who*, not *did you mean it*.  The webapp's erase is guarded; the other seven
-destructive POSTs and `/docs` are not.
-
-**4. The 5A page-boundary fix.**  Until this lands, live event download is
-unreliable on exactly the units most likely to need attention — the ones that
-have been recording heavily.  Wants a Blastware capture of an event spanning a
-page boundary before the chunk-addressing half is trustworthy.
-
-**5. A live Thor / Micromate client.**  The device side is MiniMate-only.
-Series-4 units can only be read from forwarded files, so half the fleet has no
-live path at all.
-
-**6. Test coverage that runs from a clean checkout.**  15 of 16 current
-failures are missing fixture bundles.  A test suite that cannot go green on a
-fresh clone cannot gate anything.
-
-**7. The SDM rename.**  Cosmetic relative to the above, but the longer `sfm/`
-holds the data-side code the more the tiers blur.  ~30–50 files here, ~10–15 in
-Terra-View, plus a Docker volume migration.  Do it when the codebase is quiet.
-
----
-
-## The short version
-
-The **data side is a real tool already**.  The **device side is a set of sharp
-instruments** that work in the hands of the person who wrote them, with the
-runbook open.  The gap between those two states is mostly **auth, async, and
-guardrails** — not protocol work.  The protocol is the part that is actually
-finished.

minimateplus/event_file_io.py

@@ -296,16 +296,6 @@ def apply_report_to_event(event: Event, report: BwAsciiReport) -> None:
         event.sample_rate = report.sample_rate_sps
     if report.record_time_s is not None:
         event.rectime_seconds = report.record_time_s
-    # The report's event_datetime is Blastware's exact trigger time (parsed
-    # from Event Time + Event Date).  Prefer it over the binary footer's stop
-    # time so a report-paired import matches BW to the second.
-    edt = report.event_datetime
-    if edt is not None:
-        event.timestamp = Timestamp(
-            raw=b"", flag=0x10,
-            year=edt.year, unknown_byte=0, month=edt.month, day=edt.day,
-            hour=edt.hour, minute=edt.minute, second=edt.second,
-        )
 
 
 def apply_bw_report_dict_to_event(event: Event, bw_report: dict) -> None:
@@ -818,30 +808,6 @@ def derive_record_type_from_filename(filename, default: str = "Waveform") -> str
     return _RECORD_TYPE_BY_EXT_SUFFIX.get(ext[-1].upper(), default)
 
 
-# Marker for the recording-setup config block, and the offset of the record-time
-# float32 within it.  The configured post-trigger record time (seconds) is a
-# big-endian float32 exactly 30 bytes before the "Standard Recording Setup"
-# label.  Verified across the corpus reading 1.0 / 2.0 / 3.0 s on different
-# setups — and ts2 - record_time reproduces Blastware's trigger to the second
-# (N844LQHB: stop 10:33:32 - 3.0 = 10:33:29).
-_RECSETUP_MARKER = b"Standard Recording Setup"
-_RECTIME_OFFSET_BEFORE_MARKER = 30
-
-
-def _parse_record_time_seconds(raw: bytes) -> Optional[float]:
-    """The configured post-trigger record time in seconds, from the recording-
-    setup config block, or None when absent / implausible."""
-    a = raw.find(_RECSETUP_MARKER)
-    if a < _RECTIME_OFFSET_BEFORE_MARKER:
-        return None
-    off = a - _RECTIME_OFFSET_BEFORE_MARKER
-    try:
-        rt = struct.unpack(">f", raw[off:off + 4])[0]
-    except struct.error:
-        return None
-    return rt if 0.05 <= rt <= 600.0 else None
-
-
 def read_blastware_file(path: Union[str, Path]) -> Event:
     """
     Parse a Blastware waveform file into an Event.
@@ -951,10 +917,6 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
     # rest of the event (timestamp, waveform_key, project strings) is
     # still recoverable and useful.
     decoded = decode_waveform_v2(body)
-    # Discriminator for the timestamp logic below: a waveform (trigger) event
-    # vs a histogram window.  Keyed on the codec, not the filename — the
-    # save_imported_bw path passes a tmp ".bw" name whose extension lies.
-    is_waveform_body = decoded is not None
     if decoded is None:
         decoded = decode_histogram_body(body)
     if decoded is None:
@@ -986,31 +948,7 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
     ev.total_samples   = strt_fields.get("total_samples")
     ev.pretrig_samples = strt_fields.get("pretrig_samples")
 
-    # Event timestamp.  The footer's two timestamps mean different things by
-    # record type:
-    #   * Waveform: ts1 = the monitoring-SESSION start (shared across every
-    #     event that day — a unit arming at 06:00 stamps 06:00 on all of them),
-    #     ts2 = THIS event's recording STOP.  Blastware's Date/Time is the
-    #     TRIGGER = ts2 - record time, and the record time is a float32 in the
-    #     recording-setup config block (see _parse_record_time_seconds), so the
-    #     exact trigger is recoverable from the binary alone.  Falls back to ts2
-    #     (the stop, within the record duration) if the config block is absent.
-    #     (Stamping ts1 showed the session start, hours off.)
-    #   * Histogram / undecodable: ts1 = the window start, which IS the event
-    #     time — keep it.
-    # Discriminate by ``is_waveform_body`` (the codec), not the filename.
-    if is_waveform_body and ts2 is not None:
-        _stop = datetime.datetime(ts2.year, ts2.month, ts2.day,
-                                  ts2.hour, ts2.minute, ts2.second)
-        _rt = _parse_record_time_seconds(raw)
-        _trig = _stop - datetime.timedelta(seconds=_rt) if _rt is not None else _stop
-        ev.timestamp = Timestamp(
-            raw=footer[10:18],
-            flag=0x10,
-            year=_trig.year, unknown_byte=0, month=_trig.month, day=_trig.day,
-            hour=_trig.hour, minute=_trig.minute, second=_trig.second,
-        )
-    elif ts1 is not None:
+    if ts1 is not None:
         ev.timestamp = Timestamp(
             raw=footer[2:10],
             flag=0x10,

sfm/sfm_webapp.html

@@ -108,12 +108,6 @@
       color: var(--text);
     }
     .btn-ghost:hover { border-color: var(--blue-lt); color: var(--blue-lt); }
-    .btn-danger   { background: var(--red);    color: #fff; }
-    .btn-danger:hover:not(:disabled) { filter: brightness(1.15); }
-    .diag-result  { display:block; margin-top:6px; font-size:12px; opacity:.85;
-                    white-space:pre-wrap; word-break:break-word; }
-    .diag-result.ok    { color: var(--green); }
-    .diag-result.error { color: var(--red); }
     .btn:disabled { background: var(--surface2) !important; color: var(--text-mute) !important; cursor: not-allowed; border-color: var(--border2) !important; }
 
     /* #connect-btn styles moved to #live-connect-bar block */
@@ -916,7 +910,6 @@
   <button class="tab-btn"        data-tab="events"    onclick="switchTab('events')">Events</button>
   <button class="tab-btn"        data-tab="config"    onclick="switchTab('config')">Config</button>
   <button class="tab-btn"        data-tab="call-home" onclick="switchTab('call-home')">Call Home</button>
-  <button class="tab-btn"        data-tab="diagnostics" onclick="switchTab('diagnostics')">Diagnostics</button>
 </div>
 
 <!-- ════════════════════════════════════════════════════════════════
@@ -945,10 +938,6 @@
 <div id="tab-events" class="tab-pane" style="display:flex; flex-direction:column; overflow:hidden;">
 
   <div class="event-toolbar">
-    <button class="btn btn-ghost" id="load-events-btn" onclick="loadEventList()" disabled
-            title="Walk the device's event chain and list its stored events. This is the slow one — it reads every event header over the cellular link.">
-      ⟳ Load events
-    </button>
     <button class="btn btn-ghost" id="load-btn" onclick="loadWaveform()" disabled>Load Waveform</button>
     <button class="btn btn-ghost" id="save-btn" onclick="saveEventToDb()" disabled
             title="Download the full waveform from the device and save it to the SFM database + waveform store. Honors the Force refresh toggle.">
@@ -1216,77 +1205,6 @@
 
 </div><!-- end #tab-call-home -->
 
-<!-- ════════════════════════════════════════════════════════════════
-     TAB: Diagnostics
-═══════════════════════════════════════════════════════════════════ -->
-<div id="tab-diagnostics" class="tab-pane">
-
-  <div class="cfg-grid">
-
-    <div class="cfg-section">
-      <div class="cfg-section-title">Device State</div>
-      <div class="hint" style="margin-bottom:10px">
-        Fast probes — POLL plus one read each, about 2 s. None of these walk the event chain.
-      </div>
-      <div class="dev-table" id="diag-table"></div>
-      <div class="cfg-actions" style="margin-top:12px">
-        <button class="btn btn-ghost" id="diag-refresh-btn" onclick="refreshDiagnostics()" disabled>Refresh</button>
-        <span id="diag-status"></span>
-      </div>
-    </div>
-
-    <div class="cfg-section">
-      <div class="cfg-section-title">Actions</div>
-
-      <div class="cfg-field">
-        <label>Stop Monitoring</label>
-        <button class="btn btn-ghost" id="diag-stop-btn" onclick="diagStopMonitoring()" disabled>Send Stop (SUB 0x97)</button>
-        <div class="hint">Halts recording. On a unit triggering continuously, this is what breaks the call-home loop.</div>
-        <span class="diag-result" id="diag-stop-result"></span>
-      </div>
-
-      <div class="cfg-field">
-        <label>Disable Auto Call Home</label>
-        <button class="btn btn-ghost" id="diag-ach-btn" onclick="diagDisableAch()" disabled>Disable ACH</button>
-        <div class="hint">Stored events are left untouched (<code>rescue?erase=false</code>). The unit stops dialing out until ACH is re-enabled.</div>
-        <span class="diag-result" id="diag-ach-result"></span>
-      </div>
-
-      <div class="cfg-field">
-        <label>Erase All Events</label>
-        <input type="text" id="diag-erase-confirm" placeholder="Type the serial to enable"
-               oninput="diagCheckEraseConfirm()" autocomplete="off" />
-        <button class="btn btn-danger" id="diag-erase-btn" onclick="diagEraseEvents()" disabled>Erase Events</button>
-        <div class="hint">⚠ Permanent, and resets the event chain to key <code>0x01110000</code>. Download anything worth keeping first.</div>
-        <span class="diag-result" id="diag-erase-result"></span>
-      </div>
-    </div>
-
-    <div class="cfg-section">
-      <div class="cfg-section-title">Unresponsive Unit</div>
-      <div class="hint" style="margin-bottom:10px">
-        The escalation ladder from <code>docs/runbooks/wedged_unit_recovery.md</code>, for a unit too busy
-        to answer normal request/response. Prefer <b>Method A</b> — point the modem at an
-        <code>ach_server</code> and answer its call — before racing it with these.
-      </div>
-
-      <div class="cfg-field">
-        <label>Slow drip <span class="hint" style="display:inline">(one held session, a stop every 3 s)</span></label>
-        <button class="btn btn-ghost" id="diag-drip-btn" onclick="diagSlowDrip()" disabled>Run 120 s drip</button>
-        <div class="hint">Success is <code>bytes_received &gt; 0</code>. A full duration with <code>send_error: null</code> is <b>not</b> success on its own.</div>
-        <span class="diag-result" id="diag-drip-result"></span>
-      </div>
-
-      <div class="cfg-field">
-        <label>Blind stop <span class="hint" style="display:inline">(fire-and-forget, one attempt)</span></label>
-        <button class="btn btn-ghost" id="diag-blind-btn" onclick="diagBlindStop()" disabled>Send blind stop</button>
-        <span class="diag-result" id="diag-blind-result"></span>
-      </div>
-    </div>
-
-  </div>
-</div><!-- end #tab-diagnostics -->
-
 </div><!-- end #section-live -->
 
 <!-- ════════════════════════════════════════════════════════════════
@@ -1443,8 +1361,6 @@
 // ── State ──────────────────────────────────────────────────────────────────────
 let unitInfo      = null;
 let eventList     = [];
-let storageInfo   = null;   // /device/events/storage_range — cheap, read on connect
-let eventsLoaded  = false;  // the event chain walk is opt-in; see loadEventList()
 let currentEvent  = 0;
 let charts        = {};
 let geoAdcScale      = 6.206;
@@ -1542,7 +1458,6 @@ function switchTab(name) {
   if (name === 'units')    { if (!unitsLoaded)  loadUnits(); }
   if (name === 'monlog')   { if (!monlogLoaded) loadMonitorLog(); }
   if (name === 'sessions') { if (!sessLoaded)   loadSessions(); }
-  if (name === 'diagnostics' && devHost() && unitInfo) refreshDiagnostics();
 }
 
 // ── Connect ────────────────────────────────────────────────────────────────────
@@ -1563,13 +1478,18 @@ async function connectUnit() {
     btn.disabled = false; btn.textContent = 'Connect'; return;
   }
 
-  // Connecting deliberately does NOT walk the event chain.  That walk reads
-  // every event header over the cellular link and can take minutes — or fail
-  // outright on a unit whose buffer has wrapped past 0xFFFF.  Use the ~2 s
-  // probes instead; the event list is opt-in via loadEventList().
-  eventList = []; eventsLoaded = false;
-  setStatus('Reading device state…', 'loading');
-  storageInfo = await fetchJson(`/device/events/storage_range`).catch(() => null);
+  setStatus('Fetching event list…', 'loading');
+  try {
+    const r = await fetch(`${api()}/device/events?${deviceParams()}`);
+    if (!r.ok) { const e = await r.json().catch(() => ({})); throw new Error(e.detail || r.statusText); }
+    const evData = await r.json();
+    eventList = evData.events || [];
+    // Merge compliance from /device/events response (it re-reads it)
+    if (evData.device) unitInfo = { ...unitInfo, ...evData.device };
+  } catch (e) {
+    setStatus(`Event fetch failed: ${e.message}`, 'error');
+    btn.disabled = false; btn.textContent = 'Reconnect'; return;
+  }
 
   populateDeviceBar();
   populateDeviceTab();
@@ -1578,9 +1498,11 @@ async function connectUnit() {
 
   document.getElementById('device-bar').style.display    = 'flex';
   document.getElementById('monitor-panel').style.display = 'flex';
-  setEventButtonsEnabled();
-  document.getElementById('load-events-btn').disabled = false;
-  setDiagButtonsEnabled(true);
+  document.getElementById('load-btn').disabled     = eventList.length === 0;
+  document.getElementById('save-btn').disabled     = eventList.length === 0;
+  document.getElementById('download-btn').disabled = eventList.length === 0;
+  document.getElementById('prev-btn').disabled     = true;
+  document.getElementById('next-btn').disabled     = eventList.length <= 1;
   document.getElementById('cfg-read-btn').disabled  = false;
   document.getElementById('cfg-write-btn').disabled = false;
   document.getElementById('ch-read-btn').disabled   = false;
@@ -1588,9 +1510,7 @@ async function connectUnit() {
 
   btn.disabled = false; btn.textContent = 'Reconnect';
 
-  setStatus(storageInfo && storageInfo.is_empty
-    ? 'Connected — no events stored.'
-    : 'Connected. Event list not loaded (Events → Load events).', 'ok');
+  setStatus(`Connected — ${eventList.length} event${eventList.length !== 1 ? 's' : ''} stored.`, 'ok');
 
   // Fetch monitor status in background (non-blocking)
   refreshMonitorStatus().catch(() => {});
@@ -1602,48 +1522,6 @@ async function connectUnit() {
   }
 }
 
-// ── Shared fetch helper ────────────────────────────────────────────────────────
-async function fetchJson(path, opts) {
-  const sep = path.includes('?') ? '&' : '?';
-  const r = await fetch(`${api()}${path}${sep}${deviceParams()}`, opts);
-  const body = await r.json().catch(() => ({}));
-  if (!r.ok) throw new Error(body.detail || r.statusText);
-  return body;
-}
-
-function setEventButtonsEnabled() {
-  const n = eventList.length;
-  document.getElementById('load-btn').disabled     = n === 0;
-  document.getElementById('save-btn').disabled     = n === 0;
-  document.getElementById('download-btn').disabled = n === 0;
-  document.getElementById('prev-btn').disabled     = true;
-  document.getElementById('next-btn').disabled     = n <= 1;
-}
-
-// ── Event list (opt-in — this is the slow chain walk) ──────────────────────────
-async function loadEventList() {
-  if (!devHost()) { setStatus('Connect to a device first.', 'error'); return; }
-  const btn = document.getElementById('load-events-btn');
-  btn.disabled = true;
-  setStatus('Walking the event chain — this can take a while…', 'loading');
-  try {
-    const evData = await fetchJson('/device/events');
-    eventList = evData.events || [];
-    eventsLoaded = true;
-    // /device/events re-reads compliance; fold it in.
-    if (evData.device) unitInfo = { ...unitInfo, ...evData.device };
-  } catch (e) {
-    setStatus(`Event fetch failed: ${e.message}`, 'error');
-    btn.disabled = false; return;
-  }
-  populateDeviceBar();
-  populateDeviceTab();
-  populateEventChips();
-  setEventButtonsEnabled();
-  btn.disabled = false;
-  setStatus(`${eventList.length} event${eventList.length !== 1 ? 's' : ''} stored.`, 'ok');
-}
-
 // ── Device bar ─────────────────────────────────────────────────────────────────
 function populateDeviceBar() {
   qs('di-serial').textContent   = unitInfo.serial || '—';
@@ -1652,7 +1530,7 @@ function populateDeviceBar() {
   qs('di-sr').textContent       = cc.sample_rate ? `${cc.sample_rate} sps` : '—';
   qs('di-rt').textContent       = cc.record_time != null ? `${cc.record_time.toFixed(1)} s` : '—';
   qs('di-trig').textContent     = cc.trigger_level_geo != null ? `${cc.trigger_level_geo.toFixed(3)} in/s` : '—';
-  qs('di-count').textContent    = eventsLoaded ? eventList.length : '—';
+  qs('di-count').textContent    = eventList.length;
   qs('di-project').textContent  = cc.project || '—';
   qs('di-client').textContent   = cc.client  || '—';
   qs('di-operator').textContent = cc.operator || '—';
@@ -1782,8 +1660,7 @@ function populateDeviceTab() {
     { label:'DSP',            value: unitInfo.dsp_version || '—' },
     { label:'Model',          value: unitInfo.model || '—' },
     { label:'Manufacturer',   value: unitInfo.manufacturer || '—' },
-    { label:'Stored Events',  value: eventsLoaded ? eventList.length : 'not loaded' },
-    { label:'Storage Used',   value: storageUsedLabel() },
+    { label:'Stored Events',  value: eventList.length },
   ];
   for (const {label, value} of cardData) {
     const c = document.createElement('div');
@@ -1830,158 +1707,6 @@ function renderTable(id, rows) {
   }
 }
 
-// ── Diagnostics ────────────────────────────────────────────────────────────────
-// Everything here is a cheap probe (POLL + one read) or a single write.  None of
-// it walks the event chain.  See docs/runbooks/wedged_unit_recovery.md.
-
-function storageUsedLabel() {
-  if (!storageInfo) return '—';
-  if (storageInfo.is_empty) return 'empty';
-  const f = storageInfo.first_key, l = storageInfo.last_key;
-  return (f && l) ? `${f} → ${l}` : '—';
-}
-
-function setDiagButtonsEnabled(on) {
-  for (const id of ['diag-refresh-btn','diag-stop-btn','diag-ach-btn',
-                    'diag-drip-btn','diag-blind-btn']) {
-    const el = document.getElementById(id);
-    if (el) el.disabled = !on;
-  }
-  diagCheckEraseConfirm();
-}
-
-// Erase is guarded by typing the serial — auth answers "who", not "did you mean it".
-function diagCheckEraseConfirm() {
-  const box = document.getElementById('diag-erase-confirm');
-  const btn = document.getElementById('diag-erase-btn');
-  if (!box || !btn) return;
-  const serial = (unitInfo && unitInfo.serial) || '';
-  btn.disabled = !serial || box.value.trim().toUpperCase() !== serial.toUpperCase();
-}
-
-function diagResult(id, text, cls) {
-  const el = document.getElementById(id);
-  if (!el) return;
-  el.textContent = text;
-  el.className = 'diag-result' + (cls ? ' ' + cls : '');
-}
-
-async function refreshDiagnostics() {
-  if (!devHost()) return;
-  const st = document.getElementById('diag-status');
-  if (st) { st.textContent = 'Reading…'; st.className = 'loading'; }
-
-  const [mon, store, idx] = await Promise.all([
-    fetchJson('/device/monitor/status?force=true').catch(e => ({ _err: e.message })),
-    fetchJson('/device/events/storage_range').catch(e => ({ _err: e.message })),
-    fetchJson('/device/events/index').catch(e => ({ _err: e.message })),
-  ]);
-
-  if (!store._err) storageInfo = store;
-
-  const err = v => `<span style="color:var(--red)">${v}</span>`;
-  const rows = [];
-
-  rows.push(['Monitoring', mon._err ? err(mon._err)
-    : (mon.is_monitoring ? '<b>MONITORING</b>' : 'idle')]);
-  if (!mon._err) {
-    rows.push(['Battery', mon.battery_v != null ? `${mon.battery_v.toFixed(2)} V` : '—']);
-    if (mon.memory_total_bytes) {
-      const used = mon.memory_total_bytes - (mon.memory_free_bytes ?? 0);
-      const pct  = (used / mon.memory_total_bytes * 100).toFixed(1);
-      rows.push(['Memory used', `${used.toLocaleString()} / ${mon.memory_total_bytes.toLocaleString()} bytes (${pct}%)`]);
-    }
-  }
-
-  rows.push(['Event chain', store._err ? err(store._err) : storageUsedLabel()]);
-  if (!store._err) rows.push(['Chain empty', store.is_empty ? 'yes' : 'no']);
-
-  // SUB 0x08.  Known to report 0 on units with years of history — suspected
-  // field-offset bug in the decode, so show it but do not trust it.
-  rows.push(['Lifetime events', idx._err ? err(idx._err)
-    : `${idx.lifetime_count} <span class="hint" style="display:inline">(unreliable — see CHANGELOG)</span>`]);
-
-  renderTable('diag-table', rows);
-  populateDeviceTab();
-  if (st) { st.textContent = ''; st.className = ''; }
-}
-
-async function diagStopMonitoring() {
-  const btn = document.getElementById('diag-stop-btn');
-  btn.disabled = true; diagResult('diag-stop-result', 'Sending…');
-  try {
-    await fetchJson('/device/monitor/stop', { method: 'POST' });
-    diagResult('diag-stop-result', 'Stop acknowledged — recording halted.', 'ok');
-    refreshDiagnostics();
-  } catch (e) {
-    diagResult('diag-stop-result', `Failed: ${e.message}`, 'error');
-  }
-  btn.disabled = false;
-}
-
-async function diagDisableAch() {
-  const btn = document.getElementById('diag-ach-btn');
-  btn.disabled = true; diagResult('diag-ach-result', 'Writing call-home config…');
-  try {
-    const r = await fetchJson('/device/rescue?erase=false', { method: 'POST' });
-    const steps = (r.steps || []).map(s => s.step).join(' → ') || 'done';
-    diagResult('diag-ach-result', `ACH disabled (${steps}). Events untouched.`, 'ok');
-  } catch (e) {
-    diagResult('diag-ach-result', `Failed: ${e.message}`, 'error');
-  }
-  btn.disabled = false;
-}
-
-async function diagEraseEvents() {
-  const serial = (unitInfo && unitInfo.serial) || 'this unit';
-  if (!confirm(`Permanently erase ALL events on ${serial}?\n\nThis cannot be undone.`)) return;
-  const btn = document.getElementById('diag-erase-btn');
-  btn.disabled = true; diagResult('diag-erase-result', 'Erasing…');
-  try {
-    await fetchJson('/device/events/erase', { method: 'POST' });
-    diagResult('diag-erase-result', 'Events erased — chain reset to 0x01110000.', 'ok');
-    document.getElementById('diag-erase-confirm').value = '';
-    eventList = []; eventsLoaded = false;
-    setEventButtonsEnabled(); populateEventChips();
-    refreshDiagnostics();
-  } catch (e) {
-    diagResult('diag-erase-result', `Failed: ${e.message}`, 'error');
-  }
-  diagCheckEraseConfirm();
-}
-
-async function diagSlowDrip() {
-  const btn = document.getElementById('diag-drip-btn');
-  btn.disabled = true;
-  diagResult('diag-drip-result', 'Holding a session for 120 s…');
-  try {
-    const r = await fetchJson('/device/stop_monitoring_slow_drip?duration_s=120&interval_s=3',
-                              { method: 'POST' });
-    const good = (r.bytes_received || 0) > 0;
-    diagResult('diag-drip-result',
-      `drips ${r.drips_sent}  ·  held ${r.duration_s}s  ·  bytes back ${r.bytes_received}` +
-      (r.send_error ? `  ·  ${r.send_error}` : '') +
-      (good ? '  → device responded' : '  → no response; the modem may not be bridging'),
-      good ? 'ok' : 'error');
-  } catch (e) {
-    diagResult('diag-drip-result', `Failed: ${e.message}`, 'error');
-  }
-  btn.disabled = false;
-}
-
-async function diagBlindStop() {
-  const btn = document.getElementById('diag-blind-btn');
-  btn.disabled = true; diagResult('diag-blind-result', 'Sending…');
-  try {
-    const r = await fetchJson('/device/stop_monitoring_blind', { method: 'POST' });
-    diagResult('diag-blind-result',
-      `Sent ${r.bytes_sent ?? '?'} bytes, no response read (fire-and-forget).`, 'ok');
-  } catch (e) {
-    diagResult('diag-blind-result', `Failed: ${e.message}`, 'error');
-  }
-  btn.disabled = false;
-}
-
 // ── Config form ────────────────────────────────────────────────────────────────
 function populateConfigFromDeviceInfo() {
   if (!unitInfo) return;

tests/test_event_timestamp.py

@@ -1,54 +0,0 @@
-"""Event timestamp decode — waveform trigger/stop vs histogram window start.
-
-The Blastware footer holds two timestamps: ts1 = footer[2:10], ts2 = footer[10:18].
-Their meaning depends on record type:
-
-  * Waveform: ts1 is the monitoring-SESSION start (e.g. 06:00 for a unit that
-    arms at 06:00 daily — shared across every event that day), and ts2 is THIS
-    event's recording STOP.  read_blastware_file used to stamp events with ts1 →
-    every waveform showed the session start (~4.5 h off).  Binary-only, the best
-    estimate is ts2 (the stop); the exact trigger BW displays (= ts2 - record
-    duration) comes from the paired report's event_datetime, since the binary
-    STRT record-time byte is a misparsed record-type marker.
-  * Histogram: ts1/ts2 are the ~24 h window [start, stop]; the event time is the
-    window start = ts1 (unchanged).
-"""
-import datetime
-from pathlib import Path
-
-from minimateplus.event_file_io import read_blastware_file, apply_report_to_event
-from minimateplus.bw_ascii_report import BwAsciiReport
-from minimateplus.models import Event
-
-FIX = Path(__file__).parent / "fixtures"
-WAVEFORM = FIX / "fft-oracle-2026-09-14" / "N844LQHB.ZT0W"   # footer ts2 = 2026-08-25 10:33:32
-HISTOGRAM = FIX / "ts-fix" / "K441LKZU.C30H"                 # window start 2026-05-10 19:04:50
-
-
-def _tuple(ts):
-    return (ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second)
-
-
-def test_waveform_timestamp_is_exact_trigger_from_binary():
-    ev = read_blastware_file(WAVEFORM)
-    # The EXACT Blastware trigger, from the binary alone: ts2 (stop 10:33:32)
-    # minus the config record time (3.0 s) = 10:33:29 — NOT the 06:00:13
-    # monitoring-session start the old decode used.
-    assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 33, 29), _tuple(ev.timestamp)
-
-
-def test_histogram_timestamp_is_window_start_unchanged():
-    ev = read_blastware_file(HISTOGRAM)
-    # Histogram event time = the window start (ts1); must NOT get the waveform
-    # ts2 treatment (that would land ~24 h off).
-    assert _tuple(ev.timestamp) == (2026, 5, 10, 19, 4, 50), _tuple(ev.timestamp)
-
-
-def test_report_event_datetime_is_authoritative_over_binary():
-    # The binary already yields the exact trigger, but a paired report stays
-    # authoritative (e.g. if the unit clock had drifted) — applying it wins.
-    ev = read_blastware_file(WAVEFORM)
-    assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 33, 29)   # exact, from binary
-    apply_report_to_event(ev, BwAsciiReport(
-        event_datetime=datetime.datetime(2026, 8, 25, 10, 35, 0)))
-    assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 35, 0)    # report wins