diff --git a/CLAUDE.md b/CLAUDE.md index 67dee54..a8faa16 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -61,6 +61,24 @@ Read this first when picking the project back up. 4th-decimal tick and are Thor's own rounding — no single linear LSB can reproduce every printed value (the constraints are infeasible by 7e-5 relative), so do NOT retune `_GEO_LSB_IPS`. +- **⚠ KNOWN BUG — the 5A walk breaks once a unit's buffer crosses 64 KB.** + `parse_strt_end_offset()` returns only `(end_key[2] << 8) | end_key[3]`, + discarding the key's page byte. An event starting at `0x0111F2A2` and ending + at `0x0112_1010` therefore reads `end_offset = 0x1010` — *behind* its own + start. The chunk loop then exits before fetching anything and TERM computes + a negative `offset_word`, which `struct.pack(">H", ...)` rejects: the + `/device/events` walk 500s. Reproduced on BE12599 (2026-09-19), which had + 78 KB stored and had rolled into page `0x12`. + **Why it hid so long:** every 5A capture the walk was verified against came + from a freshly-erased BE11529 — all three confirmed TERM examples in + `framing.py` (`0x1ABE`, `0x21F2`, `0x417E`) sit inside page `0x11`. Prod is + unaffected: it ingests complete files via BW ACH, never this walk. + **Fixing it has two layers** — the arithmetic (`if end < start: end += + 0x10000`) stops the crash and bounds the loop correctly; carrying the page + byte through the chunk requests (`params[1]` 0x11 -> 0x12, counter rolling + over) needs a BW capture of a spanning event first. Do not ship layer one + alone without a loud truncation warning — a silently short event is the + failure mode this codec has been bitten by repeatedly. - **Open, not blocking:** 14 sensitive-range files show an exact 8x (= 10.0/1.25) units discrepancy; `scripts/backfill_sidecars.py --force` also inserts DB rows for store files that have none (one-time per store) and the diff --git a/README.md b/README.md index 7f384f2..83a897c 100644 --- a/README.md +++ b/README.md @@ -496,6 +496,11 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows. ## Roadmap (Future) +> **Where it stands *today*** — an honest per-capability maturity assessment, +> what to rely on, known issues, and the gap to a real tool: +> [`docs/sfm_tool_status.md`](docs/sfm_tool_status.md). This section covers +> where it is *going*. + ### Strategic direction — where this is going seismo-relay is being built as a **suite of cooperating components** diff --git a/docs/sfm_tool_status.md b/docs/sfm_tool_status.md new file mode 100644 index 0000000..4924e6e --- /dev/null +++ b/docs/sfm_tool_status.md @@ -0,0 +1,150 @@ +# SFM — where it actually stands as a tool + +**Status as of 2026-09-20 (v0.31.0).** This is the honest assessment, not the +roadmap — `README.md § Roadmap` covers where it is *going*. Expect this file to +go stale; re-date it when you revise it. + +--- + +## The framing + +SFM is **three different things wearing one name**, at three very different +levels of maturity: + +| | what it is | maturity | +|---|---|---| +| **The codec library** | `minimateplus/`, `micromate/` — bytes in, `Event` out | **Production.** Verified per-sample at scale. | +| **SDM — the data side** | the DB, waveform store, `/db/*`, ingest | **Production.** Terra-View depends on it daily. | +| **SFM — the device side** | `/device/*`, live connections to units | **Emergency-grade.** Works, but manual, unauthenticated, and thinly tested. | +| **The lab** | `seismo_lab.py`, `scratch/`, the Inspector | **Research artifacts.** Useful, not products. | + +Brian's own description — *"right now it's an emergency tool and a research +project"* — is accurate, and it applies specifically to the **device side**. +The data side is not an emergency tool; it has been carrying production for +months. + +Most confusion about "is SFM reliable?" comes from answering for the wrong +tier. + +--- + +## 1. What you can rely on + +### Production-grade — trust it + +- **Series-3 decode.** 14,338 / 14,338 files decode per-sample exact against + preserved Blastware ASCII exports, 45 units, files back to 2018. +- **Series-4 (Thor) decode.** 1,057,536 / 1,057,536 geo samples exact against + Thor's own CSV exports; production IDFW 575/575 with zero truncations. +- **Histogram decode.** 1,211 / 1,211 production histograms exact, including + 842,442 per-interval frequency comparisons with zero mismatches. +- **The ingest path.** `/db/import/blastware_file` and `/db/import/idf_file` + fed by the watchers — this is how prod actually gets its data, and it has + been running unattended for months. +- **`/db/*` read API.** Always-on, consumed by Terra-View for every fleet + listing, event detail and report. +- **The waveform store** — `.h5` + `.sfm.json` sidecars + retained raw + binaries, with operator review state preserved across regeneration. +- **`bridges/ach_server.py`** — speaks the full BW protocol to calling units. + Proven in the field, including as a rescue tool (see the runbook). + +### Emergency-grade — works, but you are the error handling + +- **`/device/*` live endpoints.** They do what they say. But they are + synchronous, unauthenticated, and a single cellular download can exceed the + 60 s timeouts that sit in front of them. +- **The rescue ladder** (`rescue`, `stop_monitoring_*`, `events/erase`). + Each has worked in a real incident — but each has been used a handful of + times, by one person, with the runbook open. +- **The standalone webapp.** Perfectly usable, and as of v0.31.0 the cheap + probes and rescue actions are reachable without curl. No auth of any kind. + +### Research artifacts — useful, not products + +- **`seismo_lab.py`** — 2,789 lines of Tkinter (Bridge / Analyzer / Query DB / + Inspector). Desktop-only, single-user, no tests. +- **`scratch/`** — the verification harnesses (`verify_against_ascii.py`, + `verify_thor_against_csv.py`) and the offset detector (`offset_scan3.py`). + These produced the numbers the production claims rest on, so they matter — + but they are analysis scripts, not maintained code. +- **`docs/offset_investigation.md`** — an open investigation, not a feature. + +--- + +## 2. What to use when + +| you want to… | use | notes | +|---|---|---| +| Know if a unit is monitoring / its battery / memory | `GET /device/monitor/status?force=true` | ~2 s | +| Know whether ACH is on | `GET /device/call_home` | ~2 s. **Not** `/device/events`. | +| See how full a unit's buffer is | `GET /device/events/storage_range` | ~2 s, no chain walk | +| Stop a runaway unit | Diagnostics tab → Stop Monitoring | see the runbook first | +| Reach a unit that will not answer | **point its modem at an `ach_server` and answer its call** | runbook Method A — do not race it | +| List a unit's stored events | Events tab → Load events | **slow**, and broken past 64 KB (below) | +| Get event data into the DB | the watcher → `/db/import/*` path | not the live walk | + +The single most useful habit: **the cheap probes are cheap and the event walk +is not.** Reaching for `/device/events` to answer a yes/no question about a +unit is the mistake that motivated the v0.31.0 webapp changes. + +--- + +## 3. Known issues + +| issue | impact | status | +|---|---|---| +| **5A walk dies once a unit's buffer crosses 64 KB** | `/device/events` 500s; event body never downloads | Known, documented in `CLAUDE.md`. Needs a BW capture of a spanning event to fix properly. | +| **No auth on SFM at all** | 21 `/device/*` endpoints, including destructive ones, open to anything that reaches the port | Design agreed (Terra-View as authenticated jump host); not built. | +| **Swagger try-it-out is live on destructive endpoints** | `POST /device/events/erase` is one click away at `:8200/docs` | Partially mitigated: the webapp's erase now requires typing the serial. `/docs` itself is unguarded. | +| **`SUB 0x08` lifetime counter reads 0** | `/device/events/index` returns a meaningless number | Suspected field-offset bug. Surfaced in the UI as "unreliable". | +| **Long device operations are synchronous** | 60 s timeouts in `routers/sfm.py` and the reverse proxy; a full download exceeds both | Known design constraint. Must be POST-starts-job / GET-polls before any remote lab. | +| **`backfill_sidecars.py --force` silently inserts DB rows** | store files with no DB row get one; the dry-run does not report the count | Known. Avoid `--force` — `TOOL_VERSION` gates regeneration anyway. | +| **14 sensitive-range files show an exact 8× discrepancy** | 10.0 / 1.25 — a units problem, not a decode problem | Open, not blocking. | +| **16 failing tests on `dev`** | 15 need gitignored fixture bundles; 1 is real (`sc["peak_values"]["transverse"]` returns `None` where `0.0` is expected) | The real one shipped in v0.31.0. | + +--- + +## 4. What stands between this and a real tool + +Roughly in dependency order — each unblocks the ones below it. + +**1. Authentication.** Everything else is gated on this. SFM has none, and +the modem IP whitelist gives zero protection because SFM *is* the whitelisted +origin. The agreed design delegates rather than builds: Terra-View becomes the +authenticated jump host (`/api/sfm/*` already inherits deny-by-default operator +auth), and the `8200:8200` publish is dropped so Terra-View is the only door. + +**2. Async long operations.** POST starts a job, GET polls. Retrofitting this +after building a remote lab on top of synchronous endpoints would be far worse +than designing for it now. + +**3. Confirm-guards on the remaining destructive endpoints.** Auth answers +*who*, not *did you mean it*. The webapp's erase is guarded; the other seven +destructive POSTs and `/docs` are not. + +**4. The 5A page-boundary fix.** Until this lands, live event download is +unreliable on exactly the units most likely to need attention — the ones that +have been recording heavily. Wants a Blastware capture of an event spanning a +page boundary before the chunk-addressing half is trustworthy. + +**5. A live Thor / Micromate client.** The device side is MiniMate-only. +Series-4 units can only be read from forwarded files, so half the fleet has no +live path at all. + +**6. Test coverage that runs from a clean checkout.** 15 of 16 current +failures are missing fixture bundles. A test suite that cannot go green on a +fresh clone cannot gate anything. + +**7. The SDM rename.** Cosmetic relative to the above, but the longer `sfm/` +holds the data-side code the more the tiers blur. ~30–50 files here, ~10–15 in +Terra-View, plus a Docker volume migration. Do it when the codebase is quiet. + +--- + +## The short version + +The **data side is a real tool already**. The **device side is a set of sharp +instruments** that work in the hands of the person who wrote them, with the +runbook open. The gap between those two states is mostly **auth, async, and +guardrails** — not protocol work. The protocol is the part that is actually +finished.