docs(series4): SUB 5A streams the .IDFW file verbatim -- read path complete

The complete read path now works with no Instantel software in the loop. Three divergences from Series III, all simplifications: - No arming sequence. Series III ignores a 5A probe unless preceded by 1E / 0A / 1E(0xFE) / 0C / 1F(0xFE) / POLL x3. The Micromate answers a bare 5A request with nothing before it. - The offset word is a LENGTH, not a position: 0x1000 + 2*pages, where pages = ceil(event_size / 512), and event_size comes from the chain walk. ONE request returns the entire event -- no chunk loop, no STRT end-offset parsing, no TERM frame. Over-requesting is safe; the device caps at the real size. - Params are the Series III probe form: [0x00][key4][6 x 0x00]. The payload is the .IDFW file byte for byte. It begins 00 12 01 00 00 00 "Instantel\0" -- _THOR_PREFIX + _INSTANTEL_TAG from micromate/idf_file.py -- and the first 32 bytes are identical to a production .IDFW from the store. Responses are DLE-stuffed, so destuff before locating the file (11,781 raw -> 11,049 destuffed for an 11,032-byte event). End-to-end: event 055d4a82 downloaded over USB and fed straight to read_idf_file() yields serial UM12947, timestamp 2026-09-23 16:33:19, and 3072 samples on all four channels. Cross-check: the 0C record reports a stored Vert peak of 1.3720 for this event; the decoded samples give 1.3706 -- two unrelated paths agreeing to 0.1%. Consequence: no new codec work is needed. The bytes off the wire are the same bytes thor-watcher forwards today, so /db/import/idf_file ingests a directly downloaded event unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
2026-09-23 18:40:54 -04:00
parent 45007e12d8
commit 71f19c90d1
1 changed files with 80 additions and 2 deletions
@@ -524,6 +524,86 @@ Diffing the two images should isolate exactly what the CB/BD split changes —
 we know the wire protocol is not it, and the flags byte (`0xC5` vs `0x03`)
 gives a concrete anchor to search for.

+## `SUB 0x5A` — bulk download.  It streams the `.IDFW` file verbatim.
+
+**The complete read path works with no Instantel software in the loop.**
+
+### It needs no arming sequence
+
+Series III ignores a `5A` probe unless preceded by
+`1E → 0A → 1E(token 0xFE) → 0C → 1F(token 0xFE) → POLL × 3`.  The Micromate
+answers a **bare `5A` request** with nothing before it.  That whole ritual is
+gone.
+
+### The offset word is a LENGTH, not a position
+
+This is the key divergence.  Series III walks chunks by absolute flash
+address, stepping `0x0200` per request.  On the Micromate the offset word
+requests *how much to send*:
+
+```
+offset_word = 0x1000 + 2 × pages        pages = ceil(event_size / 512)
+```
+
+| `offset_word` | pages | destuffed file bytes |
+|---|---|---|
+| `0x1002` | 1 | 518 |
+| `0x1004` | 2 | 1,030 |
+| `0x1006` | 3 | 1,541 |
+| `0x102C` | 22 | **11,033 — the whole event** |
+
+`event_size` comes from the chain walk (the 4 bytes after the key in
+`1E`/`1F`).  **One request returns the entire event**; there is no chunk loop,
+no `STRT` end-offset parsing, and no `TERM` frame.  Over-requesting is safe —
+`0x1030` (24 pages) returned exactly the same bytes as `0x102C`, so the device
+caps at the real size.
+
+Params are the Series III *probe* form: `[0x00][key4][6 × 0x00]`.
+
+### The payload is the `.IDFW` file, byte for byte
+
+```
+[18-byte frame header] [ .IDFW file ] [chk] [ETX]     ← raw wire
+                        ^ destuffed offset 16
+```
+
+The file begins `00 12 01 00 00 00 "Instantel\0"` — `_THOR_PREFIX` +
+`_INSTANTEL_TAG` from `micromate/idf_file.py`.  The first 32 bytes are
+**identical to a production `.IDFW`** pulled from the store.
+
+⚠ Responses are DLE-stuffed.  Destuff before locating the file, or the raw
+byte count overshoots (11,781 raw → 11,049 destuffed for an 11,032-byte event).
+
+### End-to-end proof
+
+Event `055d4a82` downloaded over USB and fed straight to `read_idf_file()`:
+
+```
+serial     UM12947
+timestamp  2026-09-23 16:33:19
+samples    Tran 3072   Vert 3072   Long 3072   MicL 3072
+peaks      Tran 0.2433   Vert 1.3706   Long 0.2672  in/s
+```
+
+All four channels equal length, and the timestamp matches the `0x0C` record
+for the same key.  **Independent cross-check:** `0x0C` reports a stored peak
+of **1.3720** for this event; the decoded samples give **1.3706** — two
+unrelated paths agreeing to 0.1%.
+
+**Consequence:** no new codec work is needed.  The bytes off the wire are the
+same bytes `thor-watcher` forwards today, so `/db/import/idf_file` ingests a
+directly-downloaded event unchanged.  Everything the IDF decoder already does
+per-sample-exact applies.
+
+### What a full read now looks like
+
+```
+1E                     → first key + size
+  0C(key)              → project/client/operator, timestamp, peaks
+  5A(key, 0x1000+2×ceil(size/512))  → the whole .IDFW
+1F                     → next key + size   (until null sentinel)
+```
+
 ## ⚠ Untested and unsafe-until-agreed

 Nothing below has been sent to a unit, and nothing should be without an
@@ -538,8 +618,6 @@ explicit decision:
 Also unknown:

 - Whether `0x10` bytes inside request params need stuffing
- Whether the bulk waveform stream (`5A` on Series III) exists here, and
-  whether it is the transport for `.IDFW` bodies we already decode
 - Everything about the **call-home session** — the device-initiated direction
  has not been observed at all.  Specifically: how a unit announces itself,
  and **how it learns an event was accepted so it stops re-sending it.**