From 71f19c90d13338e052e1ac3f6da005c2a64536b8 Mon Sep 17 00:00:00 2001 From: serversdown Date: Wed, 23 Sep 2026 18:40:54 -0400 Subject: [PATCH] docs(series4): SUB 5A streams the .IDFW file verbatim -- read path complete The complete read path now works with no Instantel software in the loop. Three divergences from Series III, all simplifications: - No arming sequence. Series III ignores a 5A probe unless preceded by 1E / 0A / 1E(0xFE) / 0C / 1F(0xFE) / POLL x3. The Micromate answers a bare 5A request with nothing before it. - The offset word is a LENGTH, not a position: 0x1000 + 2*pages, where pages = ceil(event_size / 512), and event_size comes from the chain walk. ONE request returns the entire event -- no chunk loop, no STRT end-offset parsing, no TERM frame. Over-requesting is safe; the device caps at the real size. - Params are the Series III probe form: [0x00][key4][6 x 0x00]. The payload is the .IDFW file byte for byte. It begins 00 12 01 00 00 00 "Instantel\0" -- _THOR_PREFIX + _INSTANTEL_TAG from micromate/idf_file.py -- and the first 32 bytes are identical to a production .IDFW from the store. Responses are DLE-stuffed, so destuff before locating the file (11,781 raw -> 11,049 destuffed for an 11,032-byte event). End-to-end: event 055d4a82 downloaded over USB and fed straight to read_idf_file() yields serial UM12947, timestamp 2026-09-23 16:33:19, and 3072 samples on all four channels. Cross-check: the 0C record reports a stored Vert peak of 1.3720 for this event; the decoded samples give 1.3706 -- two unrelated paths agreeing to 0.1%. Consequence: no new codec work is needed. The bytes off the wire are the same bytes thor-watcher forwards today, so /db/import/idf_file ingests a directly downloaded event unchanged. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL --- docs/micromate_protocol_reference.md | 82 +++++++++++++++++++++++++++- 1 file changed, 80 insertions(+), 2 deletions(-) diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index d3a61c6..72d7f9b 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -524,6 +524,86 @@ Diffing the two images should isolate exactly what the CB/BD split changes — we know the wire protocol is not it, and the flags byte (`0xC5` vs `0x03`) gives a concrete anchor to search for. +## `SUB 0x5A` — bulk download. It streams the `.IDFW` file verbatim. + +**The complete read path works with no Instantel software in the loop.** + +### It needs no arming sequence + +Series III ignores a `5A` probe unless preceded by +`1E → 0A → 1E(token 0xFE) → 0C → 1F(token 0xFE) → POLL × 3`. The Micromate +answers a **bare `5A` request** with nothing before it. That whole ritual is +gone. + +### The offset word is a LENGTH, not a position + +This is the key divergence. Series III walks chunks by absolute flash +address, stepping `0x0200` per request. On the Micromate the offset word +requests *how much to send*: + +``` +offset_word = 0x1000 + 2 × pages pages = ceil(event_size / 512) +``` + +| `offset_word` | pages | destuffed file bytes | +|---|---|---| +| `0x1002` | 1 | 518 | +| `0x1004` | 2 | 1,030 | +| `0x1006` | 3 | 1,541 | +| `0x102C` | 22 | **11,033 — the whole event** | + +`event_size` comes from the chain walk (the 4 bytes after the key in +`1E`/`1F`). **One request returns the entire event**; there is no chunk loop, +no `STRT` end-offset parsing, and no `TERM` frame. Over-requesting is safe — +`0x1030` (24 pages) returned exactly the same bytes as `0x102C`, so the device +caps at the real size. + +Params are the Series III *probe* form: `[0x00][key4][6 × 0x00]`. + +### The payload is the `.IDFW` file, byte for byte + +``` +[18-byte frame header] [ .IDFW file ] [chk] [ETX] ← raw wire + ^ destuffed offset 16 +``` + +The file begins `00 12 01 00 00 00 "Instantel\0"` — `_THOR_PREFIX` + +`_INSTANTEL_TAG` from `micromate/idf_file.py`. The first 32 bytes are +**identical to a production `.IDFW`** pulled from the store. + +⚠ Responses are DLE-stuffed. Destuff before locating the file, or the raw +byte count overshoots (11,781 raw → 11,049 destuffed for an 11,032-byte event). + +### End-to-end proof + +Event `055d4a82` downloaded over USB and fed straight to `read_idf_file()`: + +``` +serial UM12947 +timestamp 2026-09-23 16:33:19 +samples Tran 3072 Vert 3072 Long 3072 MicL 3072 +peaks Tran 0.2433 Vert 1.3706 Long 0.2672 in/s +``` + +All four channels equal length, and the timestamp matches the `0x0C` record +for the same key. **Independent cross-check:** `0x0C` reports a stored peak +of **1.3720** for this event; the decoded samples give **1.3706** — two +unrelated paths agreeing to 0.1%. + +**Consequence:** no new codec work is needed. The bytes off the wire are the +same bytes `thor-watcher` forwards today, so `/db/import/idf_file` ingests a +directly-downloaded event unchanged. Everything the IDF decoder already does +per-sample-exact applies. + +### What a full read now looks like + +``` +1E → first key + size + 0C(key) → project/client/operator, timestamp, peaks + 5A(key, 0x1000+2×ceil(size/512)) → the whole .IDFW +1F → next key + size (until null sentinel) +``` + ## ⚠ Untested and unsafe-until-agreed Nothing below has been sent to a unit, and nothing should be without an @@ -538,8 +618,6 @@ explicit decision: Also unknown: - Whether `0x10` bytes inside request params need stuffing -- Whether the bulk waveform stream (`5A` on Series III) exists here, and - whether it is the transport for `.IDFW` bodies we already decode - Everything about the **call-home session** — the device-initiated direction has not been observed at all. Specifically: how a unit announces itself, and **how it learns an event was accepted so it stops re-sending it.**