docs: record the 5A page-boundary bug, and assess SFM as a tool

Two things Brian asked for after the BE12599 work.

The known bug: the 5A walk discards the key's page byte, so once a unit has
recorded more than 64 KB since its last erase, an event spanning the boundary
reads an end_offset behind its own start. The chunk loop then fetches nothing
and TERM packs a negative offset_word, which is the 500. Reproduced on BE12599.
It hid this long because every capture the walk was verified against came from
a freshly-erased BE11529 — all three confirmed TERM examples sit inside page
0x11. Prod is unaffected; it ingests complete files and never runs this walk.

The status doc exists because "is SFM reliable?" has three different answers
depending on which tier is meant. The codec library and the data side are
production — verified per-sample at scale, carrying Terra-View daily. The
device side is emergency-grade: it works, but it is synchronous,
unauthenticated, and thinly tested. The lab is research artifacts. Most
confusion comes from answering for the wrong tier.

It covers all three of what Brian asked for: maturity per capability, an
operator-facing "what to use when" (the cheap probes are cheap and the event
walk is not), the known-issues table, and the gap analysis. That gap is mostly
auth, async and guardrails — not protocol work. The protocol is the finished
part.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN
This commit is contained in:
2026-09-20 01:30:14 +00:00
co-authored by Claude Opus 5
parent 6589da445b
commit f1ab5b1e9d
3 changed files with 173 additions and 0 deletions
+18
View File
@@ -61,6 +61,24 @@ Read this first when picking the project back up.
4th-decimal tick and are Thor's own rounding — no single linear LSB can
reproduce every printed value (the constraints are infeasible by 7e-5
relative), so do NOT retune `_GEO_LSB_IPS`.
- **⚠ KNOWN BUG — the 5A walk breaks once a unit's buffer crosses 64 KB.**
`parse_strt_end_offset()` returns only `(end_key[2] << 8) | end_key[3]`,
discarding the key's page byte. An event starting at `0x0111F2A2` and ending
at `0x0112_1010` therefore reads `end_offset = 0x1010` — *behind* its own
start. The chunk loop then exits before fetching anything and TERM computes
a negative `offset_word`, which `struct.pack(">H", ...)` rejects: the
`/device/events` walk 500s. Reproduced on BE12599 (2026-09-19), which had
78 KB stored and had rolled into page `0x12`.
**Why it hid so long:** every 5A capture the walk was verified against came
from a freshly-erased BE11529 — all three confirmed TERM examples in
`framing.py` (`0x1ABE`, `0x21F2`, `0x417E`) sit inside page `0x11`. Prod is
unaffected: it ingests complete files via BW ACH, never this walk.
**Fixing it has two layers** — the arithmetic (`if end < start: end +=
0x10000`) stops the crash and bounds the loop correctly; carrying the page
byte through the chunk requests (`params[1]` 0x11 -> 0x12, counter rolling
over) needs a BW capture of a spanning event first. Do not ship layer one
alone without a loud truncation warning — a silently short event is the
failure mode this codec has been bitten by repeatedly.
- **Open, not blocking:** 14 sensitive-range files show an exact 8x
(= 10.0/1.25) units discrepancy; `scripts/backfill_sidecars.py --force` also
inserts DB rows for store files that have none (one-time per store) and the