diff --git a/docs/micromate_protocol_reference.md b/docs/micromate_protocol_reference.md index 2016afb..7363e38 100644 --- a/docs/micromate_protocol_reference.md +++ b/docs/micromate_protocol_reference.md @@ -694,19 +694,55 @@ key size(1E) chunks sum(offsets) last offset 055d4a86 6092 6 6092 0x03cc ``` -**These are probably two different modes, not a contradiction.** THOR's -`offset_hi` is the chunk length (`0x04`, `0x03`, `0x00` …). Our single-request -probes set `offset_hi = 0x10` — which in Series III is precisely the -bulk-stream marker `build_5a_frame()` writes raw. So `0x10XX` plausibly means -"stream until done" and returns **several** frames, which the parser of the day -concatenated into the 11,049 bytes recorded below. That reconciles both -observations, but it is a hypothesis: those 2026-09-23 captures never landed in -the repo, so it cannot be re-derived from bytes on disk. +> #### ⚠ RETRACTED 2026-10-02 — there is no streaming mode +> +> This section previously hypothesised that `offset_hi = 0x10` meant "stream +> until done" and returned several frames, reconciling THOR's chunk loop with +> our 2026-09-23 single-request observation. **Measured directly on UM20147, +> it does not:** +> +> | request | one frame returns | file bytes | +> |---|---|---| +> | `offset = 0x1014` (4,116) | 4,127 B data | **4,116** | +> | `offset = 0x111c` (4,380) | 4,391 B data | **4,380** | +> +> **`offset` is simply a byte count.** The device returns exactly `offset + 11` +> bytes in one frame, and `page_key` is `offset // 256`. `0x1000` is not a +> marker — it is part of the number. One model covers every `0x5A` request ever +> observed, THOR's and ours. +> +> So the 2026-09-23 note that `offset_word = 0x1000 + 2 × pages` returned an +> entire 11 KB event is **wrong**: `0x102C` is 4,140, and 4,140 bytes is what it +> would have returned. Most likely the 11,049 figure was the whole capture +> rather than one frame's payload. Those captures never landed in the repo, so +> the error cannot be traced further — but it does not need to be, because the +> live measurement is unambiguous. +> +> 🔑 **And the negative result turned up something better. See below.** + +### 🔑 1024 bytes per request is THOR's choice, not the device's limit + +The retraction above has a payoff. In establishing that `offset` is a plain byte +count, UM20147 served **4,380 bytes in a single frame** without being asked +twice. THOR uses 1024; nothing about the device requires it. + +That matters because of the round-trip cost: ~0.65 s each over cellular, +regardless of payload. A 72,560-byte event is **71 requests ≈ 46 seconds** at +THOR's chunk size. The offset field is a **uint16**, so the structural ceiling is +**65,535 bytes per request** — which would make that same event **2 requests, +~1.3 seconds**. + +⚠ The *device's* ceiling is not yet known, only that it is at least 4,380. It may +be bounded by an internal buffer well below 65,535. `scratch/mm_stream_probe.py` +walks ascending sizes and checks each against a known-good chunk-loop download, +so a pass means byte-identical output rather than a plausible length. + +⚠ `micromate/protocol.py` still uses **1024** — THOR's value, the one with +captures behind it. Raising it is a one-constant change once the ceiling is +measured, and it should not be raised on inference. **Implement THOR's chunked form.** It is verified byte-exact across six events -and five distinct sizes, and it is what the firmware runs every day. The -single-request form is worth one bench test as an optimisation — `offset_hi = -0x10` and count the frames — but not worth depending on first. +and five distinct sizes, and it is what the firmware runs every day. ### The offset word is a LENGTH, not a position diff --git a/scratch/mm_stream_probe.py b/scratch/mm_stream_probe.py index a233d82..b664b90 100644 --- a/scratch/mm_stream_probe.py +++ b/scratch/mm_stream_probe.py @@ -1,50 +1,42 @@ #!/usr/bin/env python3 """ -mm_stream_probe.py — is there a one-request streaming mode for `SUB 0x5A`? +mm_stream_probe.py — how many bytes will `SUB 0x5A` serve in one request? -The question ------------- -THOR downloads an event as a **chunk loop**: `ceil(size / 1024)` requests, each -asking for `min(1024, remaining)` bytes. Verified byte-for-byte against its own -frames, and confirmed on hardware up to 71 chunks. +Settled 2026-10-02: there is NO streaming mode +---------------------------------------------- +This script started out asking whether `offset_hi = 0x10` meant "stream until +done", because our 2026-09-23 notes recorded a single request appearing to return +an entire 11 KB event. Measured directly on UM20147, it does not: -But our own 2026-09-23 probes recorded something different — a **single** request -with `offset_hi = 0x10` that appeared to return an entire 11 KB event: + offset 0x1014 (4116) -> one frame, 4127 B data, 4116 B of file + offset 0x111c (4380) -> one frame, 4391 B data, 4380 B of file - offset_word = 0x1000 + 2 * ceil(size / 512) +**`offset` is simply a byte count**, and the device returns exactly `offset + 11` +bytes in one frame. `0x1000` is not a marker; it is part of the number. The +2026-09-23 reading was wrong, and the protocol reference now says so. -`0x10` in `offset_hi` is exactly the bulk-stream marker Series III's -`build_5a_frame()` writes raw, so "`0x10XX` means stream until done, and the -device sends several frames" is a plausible reading. Those captures never landed -in the repo, so it cannot be re-derived from bytes on disk. +The useful question it turned up +------------------------------- +**1024 bytes per request is THOR's choice, not the device's limit.** The unit +served 4,380 bytes in a single frame without being asked twice. Since a round +trip over cellular costs ~0.65 s regardless of payload, and UM20147's +72,560-byte event is 71 chunks ≈ 46 seconds, the ceiling on one request is worth +knowing precisely: every doubling halves the dominant cost. -Why it matters --------------- -A round trip over cellular costs ~0.65 s regardless of payload. UM20147's -72,560-byte event is **71 chunks ≈ 46 seconds**. If one request can fetch it, -that becomes under a second. On a fleet of units called daily, that is the -difference between a workable receiver and an unworkable one. +So this now walks ascending request sizes against one event and **checks each +against a known-good chunk-loop download** — a pass means byte-identical output, +not merely a plausible length. The offset field is a uint16, so 65,535 is the +structural maximum. -What this does --------------- -Downloads **the same event twice** — once with the known-good chunk loop, once -with a single `0x10XX` request — and **diffs the bytes**. A differential test -rather than a suggestive one: if the streaming form returns byte-identical -output, it is safe to adopt; if it returns anything else, we learn exactly what. - -Then it re-POLLs, because the honest risk here is leaving the session in an odd -state, and the script should say so rather than leave you guessing. - -⚠ **Read-only.** `0x5A` is a read we have already sent thousands of times; the -only thing new is the value in its offset field. Nothing here writes, erases or -changes monitoring state. The worst realistic outcome is an unanswered frame or -a session that needs reconnecting. +⚠ **Read-only.** `0x5A` is a read we have sent thousands of times; the only new +thing is a larger value in its offset field. Nothing here writes, erases or +changes monitoring state. It re-POLLs at the end, because the honest risk is +leaving the session in an odd state and the script should say so. Usage ----- python3 scratch/mm_stream_probe.py /dev/ttyACM1 - python3 scratch/mm_stream_probe.py /dev/ttyACM1 --event 055d4a82 - python3 scratch/mm_stream_probe.py 63.45.161.30:9034 --event smallest + python3 scratch/mm_stream_probe.py /dev/ttyACM1 --event largest """ from __future__ import annotations @@ -142,104 +134,58 @@ def main() -> int: print(f"\n [1] chunk loop ...... {len(chunked)} B in {dt_chunked:.2f} s " f"({n_chunks} requests)") - # ── 2. one request, offset_hi = 0x10 ────────────────────────────────── - # The form our 2026-09-23 probes recorded. `pages` is 512-byte pages; - # the +0x1000 is the bulk-stream marker. - pages = math.ceil(ref.size / 512) - offset = 0x1000 + 2 * pages - params = ref.key + bytes(6) + # ── 2. how many bytes will it serve in ONE frame? ───────────────── + # The streaming hypothesis is dead (see the module docstring): `offset` + # is simply a BYTE COUNT, and the device returns `offset + 11` bytes in + # one frame. So the real question is the ceiling -- because 1024 is + # THOR's choice, not the device's limit, and every doubling halves the + # round trips that dominate a cellular download. + print("\n [2] chunk-size ceiling — ascending single requests") + print(" each asks for N bytes from offset 0 and is checked against") + print(" the known-good download, so a pass means identical bytes.\n") - # ⚠ TRY BOTH ESCAPINGS, or a null result means nothing. - # - # On Series III, `offset_hi = 0x10` in a 5A frame must be written RAW -- - # doubled to `10 10`, the device SILENTLY IGNORES the frame. That is a - # documented, hard-won rule for this exact command. - # - # The Micromate escapes `0x04` in offset_hi (218/218 captured THOR - # frames), which argues the uniform escape set applies to 0x10 too. But - # Series III is a direct counter-example in the same command, so testing - # only one form risks concluding "no streaming mode" when the real - # finding is "that frame was malformed". - escaped = build_request(SUB_BULK_DOWNLOAD, offset, params) - candidates = [("escaped offset_hi (uniform rule)", escaped)] - if (offset >> 8) == 0x10: - # Hand-build the raw form: the uniform builder cannot express it. - payload = bytes([0x10, 0x00, SUB_BULK_DOWNLOAD, 0x00]) + \ - bytes([(offset >> 8) & 0xFF, offset & 0xFF]) + params - body = payload + bytes([sum(payload) & 0xFF]) - out = bytearray([0x41, 0x02]) - for i, b in enumerate(body): - # escape everything EXCEPT the offset_hi at payload index 4 - if b in (0x02, 0x03, 0x04, 0x10) and i != 4: - out.append(0x10) - out.append(b) - out.append(0x03) - candidates.append(("RAW offset_hi (series-III rule)", bytes(out))) - else: - print(f"\n note: offset_hi is 0x{offset >> 8:02x}, not 0x10, so the " - f"escaping question does not arise for this event") + candidates = [1024, 2048, 4096, 8192, 16384, 32768, 65535] + candidates = [n for n in candidates if n <= ref.size] or [ref.size] + if ref.size not in candidates and ref.size < 65536: + candidates.append(ref.size) # the whole event in one request - print(f"\n [2] streaming ....... one request, offset=0x{offset:04x} " - f"(0x1000 + 2 x {pages} pages)") - - frames, dt_stream, used = [], 0.0, None - for name, frame in candidates: - print(f"\n trying {name}") - print(f" wire: {frame.hex(' ')}") + best = None + for n in candidates: + frame = build_request(SUB_BULK_DOWNLOAD, n, ref.key + bytes(6)) parser = MicromateFrameParser() t0 = time.monotonic() mm.protocol._send(frame) got = collect(inner, parser, idle_gap=a.idle_gap, deadline=t0 + a.timeout) dt = time.monotonic() - t0 - print(f" -> {len(got)} frame(s), {parser.bytes_fed} raw bytes, " - f"{dt:.2f} s") - if got: - frames, dt_stream, used = got, dt, name - break - if used: - print(f"\n answered by: {used}") - if not frames: - print("\n VERDICT: no answer to EITHER escaping. The 0x10XX form") - print(" is not a streaming mode — or not with these params. Since") - print(" both escapings were tried, this is not a framing artefact.") - print(" The chunk loop stands as the only way to download an event,") - print(" and the 2026-09-23 note should be retracted.") + if not got: + print(f" {n:6} B no answer ({dt:.2f} s)") + continue + body = b"".join(f.data[_CHUNK_PREFIX:] for f in got) + ok = body == chunked[:n] + flag = "OK " if ok else "MISMATCH" + print(f" {n:6} B {len(got)} frame(s) {len(body):6} B back " + f"{dt:5.2f} s {flag}" + + ("" if ok or not body else + f" (first diff at {next((i for i in range(min(len(body), n)) if body[i] != chunked[i]), None)})")) + if ok and len(body) == n: + best = n + + print() + if best is None: + print(" VERDICT: nothing above the current chunk size verified.") else: - bad = [f for f in frames if not f.checksum_valid] - subs = sorted({f"0x{f.sub:02x}" for f in frames}) - print(f" SUBs {subs}, {len(bad)} bad checksum") - - # Assemble the same way a chunk response is assembled. - streamed = b"".join(f.data[_CHUNK_PREFIX:] for f in frames) - print(f" assembled {len(streamed)} B " - f"(event is {ref.size} B)") - - print() - if streamed == chunked: - print(f" VERDICT: ** BYTE-IDENTICAL ** in {len(frames)} frame(s) " - f"against {n_chunks}.") - print(f" {dt_chunked:.2f} s -> {dt_stream:.2f} s here; over " - f"cellular that is ~{n_chunks * 0.65:.0f} s -> ~0.7 s.") - print(" The streaming mode is real. Worth adopting.") - elif len(streamed) == ref.size: - print(" VERDICT: right LENGTH, wrong BYTES. So it streams, but") - print(" the assembly differs — likely a different per-frame") - print(" prefix than the chunk form's 11 bytes. Compare below.") - for i in range(min(len(streamed), len(chunked))): - if streamed[i] != chunked[i]: - print(f" first difference at byte {i}") - print(f" chunked {chunked[max(0,i-4):i+8].hex(' ')}") - print(f" streamed {streamed[max(0,i-4):i+8].hex(' ')}") - break - else: - print(f" VERDICT: answered, but {len(streamed)} B against " - f"{ref.size} B expected.") - print(" Inconclusive — dump the frame sizes and look again:") - for i, f in enumerate(frames[:12]): - print(f" frame {i}: data {len(f.data)} B, " - f"page=0x{f.page_key:04x}") + now = math.ceil(ref.size / 1024) + then = math.ceil(ref.size / best) + print(f" VERDICT: the device serves at least {best} B per request,") + print(f" verified byte-identical. For this {ref.size} B event that") + print(f" is {then} request(s) instead of {now}.") + if best > 1024: + print(f" Over cellular at ~0.65 s per round trip: " + f"~{now * 0.65:.0f} s -> ~{then * 0.65:.1f} s.") + if best >= ref.size: + print(" The WHOLE EVENT fits in one request.") # ── 3. is the unit still healthy? ───────────────────────────────────── # The real risk of this experiment is leaving the session wedged, so