# SFM — where it actually stands as a tool **Status as of 2026-09-20 (v0.31.0).** This is the honest assessment, not the roadmap — `README.md § Roadmap` covers where it is *going*. Expect this file to go stale; re-date it when you revise it. --- ## The framing SFM is **three different things wearing one name**, at three very different levels of maturity: | | what it is | maturity | |---|---|---| | **The codec library** | `minimateplus/`, `micromate/` — bytes in, `Event` out | **Production.** Verified per-sample at scale. | | **SDM — the data side** | the DB, waveform store, `/db/*`, ingest | **Production.** Terra-View depends on it daily. | | **SFM — the device side** | `/device/*`, live connections to units | **Emergency-grade.** Works, but manual, unauthenticated, and thinly tested. | | **The lab** | `seismo_lab.py`, `scratch/`, the Inspector | **Research artifacts.** Useful, not products. | Brian's own description — *"right now it's an emergency tool and a research project"* — is accurate, and it applies specifically to the **device side**. The data side is not an emergency tool; it has been carrying production for months. Most confusion about "is SFM reliable?" comes from answering for the wrong tier. --- ## 1. What you can rely on ### Production-grade — trust it - **Series-3 decode.** 14,338 / 14,338 files decode per-sample exact against preserved Blastware ASCII exports, 45 units, files back to 2018. - **Series-4 (Thor) decode.** 1,057,536 / 1,057,536 geo samples exact against Thor's own CSV exports; production IDFW 575/575 with zero truncations. - **Histogram decode.** 1,211 / 1,211 production histograms exact, including 842,442 per-interval frequency comparisons with zero mismatches. - **The ingest path.** `/db/import/blastware_file` and `/db/import/idf_file` fed by the watchers — this is how prod actually gets its data, and it has been running unattended for months. - **`/db/*` read API.** Always-on, consumed by Terra-View for every fleet listing, event detail and report. - **The waveform store** — `.h5` + `.sfm.json` sidecars + retained raw binaries, with operator review state preserved across regeneration. - **`bridges/ach_server.py`** — speaks the full BW protocol to calling units. Proven in the field, including as a rescue tool (see the runbook). ### Emergency-grade — works, but you are the error handling - **`/device/*` live endpoints.** They do what they say. But they are synchronous, unauthenticated, and a single cellular download can exceed the 60 s timeouts that sit in front of them. - **The rescue ladder** (`rescue`, `stop_monitoring_*`, `events/erase`). Each has worked in a real incident — but each has been used a handful of times, by one person, with the runbook open. - **The standalone webapp.** Perfectly usable, and as of v0.31.0 the cheap probes and rescue actions are reachable without curl. No auth of any kind. ### Research artifacts — useful, not products - **`seismo_lab.py`** — 2,789 lines of Tkinter (Bridge / Analyzer / Query DB / Inspector). Desktop-only, single-user, no tests. - **`scratch/`** — the verification harnesses (`verify_against_ascii.py`, `verify_thor_against_csv.py`) and the offset detector (`offset_scan3.py`). These produced the numbers the production claims rest on, so they matter — but they are analysis scripts, not maintained code. - **`docs/offset_investigation.md`** — an open investigation, not a feature. --- ## 2. What to use when | you want to… | use | notes | |---|---|---| | Know if a unit is monitoring / its battery / memory | `GET /device/monitor/status?force=true` | ~2 s | | Know whether ACH is on | `GET /device/call_home` | ~2 s. **Not** `/device/events`. | | See how full a unit's buffer is | `GET /device/events/storage_range` | ~2 s, no chain walk | | Stop a runaway unit | Diagnostics tab → Stop Monitoring | see the runbook first | | Reach a unit that will not answer | **point its modem at an `ach_server` and answer its call** | runbook Method A — do not race it | | List a unit's stored events | Events tab → Load events | **slow**, and broken past 64 KB (below) | | Get event data into the DB | the watcher → `/db/import/*` path | not the live walk | The single most useful habit: **the cheap probes are cheap and the event walk is not.** Reaching for `/device/events` to answer a yes/no question about a unit is the mistake that motivated the v0.31.0 webapp changes. --- ## 3. Known issues | issue | impact | status | |---|---|---| | **5A walk dies once a unit's buffer crosses 64 KB** | `/device/events` 500s; event body never downloads | Known, documented in `CLAUDE.md`. Needs a BW capture of a spanning event to fix properly. | | **No auth on SFM at all** | 21 `/device/*` endpoints, including destructive ones, open to anything that reaches the port | Design agreed (Terra-View as authenticated jump host); not built. | | **Swagger try-it-out is live on destructive endpoints** | `POST /device/events/erase` is one click away at `:8200/docs` | Partially mitigated: the webapp's erase now requires typing the serial. `/docs` itself is unguarded. | | **`SUB 0x08` lifetime counter reads 0** | `/device/events/index` returns a meaningless number | Suspected field-offset bug. Surfaced in the UI as "unreliable". | | **Long device operations are synchronous** | 60 s timeouts in `routers/sfm.py` and the reverse proxy; a full download exceeds both | Known design constraint. Must be POST-starts-job / GET-polls before any remote lab. | | **`backfill_sidecars.py --force` silently inserts DB rows** | store files with no DB row get one; the dry-run does not report the count | Known. Avoid `--force` — `TOOL_VERSION` gates regeneration anyway. | | **14 sensitive-range files show an exact 8× discrepancy** | 10.0 / 1.25 — a units problem, not a decode problem | Open, not blocking. | | **16 failing tests on `dev`** | 15 need gitignored fixture bundles; 1 is real (`sc["peak_values"]["transverse"]` returns `None` where `0.0` is expected) | The real one shipped in v0.31.0. | --- ## 4. What stands between this and a real tool Roughly in dependency order — each unblocks the ones below it. **1. Authentication.** Everything else is gated on this. SFM has none, and the modem IP whitelist gives zero protection because SFM *is* the whitelisted origin. The agreed design delegates rather than builds: Terra-View becomes the authenticated jump host (`/api/sfm/*` already inherits deny-by-default operator auth), and the `8200:8200` publish is dropped so Terra-View is the only door. **2. Async long operations.** POST starts a job, GET polls. Retrofitting this after building a remote lab on top of synchronous endpoints would be far worse than designing for it now. **3. Confirm-guards on the remaining destructive endpoints.** Auth answers *who*, not *did you mean it*. The webapp's erase is guarded; the other seven destructive POSTs and `/docs` are not. **4. The 5A page-boundary fix.** Until this lands, live event download is unreliable on exactly the units most likely to need attention — the ones that have been recording heavily. Wants a Blastware capture of an event spanning a page boundary before the chunk-addressing half is trustworthy. **5. A live Thor / Micromate client.** The device side is MiniMate-only. Series-4 units can only be read from forwarded files, so half the fleet has no live path at all. **6. Test coverage that runs from a clean checkout.** 15 of 16 current failures are missing fixture bundles. A test suite that cannot go green on a fresh clone cannot gate anything. **7. The SDM rename.** Cosmetic relative to the above, but the longer `sfm/` holds the data-side code the more the tiers blur. ~30–50 files here, ~10–15 in Terra-View, plus a Docker volume migration. Do it when the codebase is quiet. --- ## The short version The **data side is a real tool already**. The **device side is a set of sharp instruments** that work in the hands of the person who wrote them, with the runbook open. The gap between those two states is mostly **auth, async, and guardrails** — not protocol work. The protocol is the part that is actually finished.