serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs(series4): RETRACTED -- there is no 0x5A streaming mode; offset is a byte count

Browse Source 
Measured directly on UM20147.  offset 0x1014 (4,116) returned one frame of 4,127
B data = 4,116 B of file; offset 0x111c (4,380) returned 4,391 B = 4,380 B.  So
`offset` is simply a BYTE COUNT, the device returns exactly offset + 11 bytes in
one frame, and page_key is offset // 256.  `0x1000` is not a bulk-stream marker;
it is part of the number.  One model now covers every 0x5A request ever observed,
THOR's and ours.

The 2026-09-23 note that offset_word = 0x1000 + 2*pages returned an entire 11 KB
event is therefore wrong -- 0x102C is 4,140, and 4,140 bytes is what it would
have returned.  Most likely the 11,049 figure was the whole capture rather than
one frame's payload.  Those captures never landed in the repo so the error cannot
be traced further, and does not need to be: the live measurement is unambiguous.

The probe tried BOTH escapings of offset_hi, which is why this negative counts --
a malformed frame would have produced the same silence.  The escaped form
answered, so the uniform escape set holds for 0x10 in offset_hi too, and the
Series III exception does not carry over.

AND THE NEGATIVE TURNED UP SOMETHING BETTER.  In establishing that offset is a
byte count, the unit served 4,380 bytes in a SINGLE frame.  1024 is THOR's
choice, not the device's limit.  Since a round trip costs ~0.65 s over cellular
regardless of payload, and a 72,560-byte event is 71 requests ~ 46 s at THOR's
chunk size, the ceiling is worth knowing precisely: the offset field is a uint16,
so 65,535 B per request would make that same event 2 requests, ~1.3 s.

mm_stream_probe.py repurposed to walk ascending request sizes, checking each
against a known-good chunk-loop download so a pass means byte-identical output
rather than a plausible length.

micromate/protocol.py still uses 1024 -- THOR's value, the one with captures
behind it.  Raising it is a one-constant change once the ceiling is MEASURED, and
it should not be raised on inference.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
serversdown
2026-10-02 01:28:43 -04:00
co-authored by Claude Opus 5
parent 18ed1a27be
commit 248ddc9cda
2 changed files with 116 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/micromate_protocol_reference.md
+47 -11
View File
@@ -694,19 +694,55 @@ key size(1E) chunks sum(offsets) last offset
055d4a86 6092 6 6092 0x03cc
```
**These are probably two different modes, not a contradiction.** THOR's
`offset_hi` is the chunk length (`0x04`, `0x03`, `0x00` …). Our single-request
probes set `offset_hi = 0x10` — which in Series III is precisely the
bulk-stream marker `build_5a_frame()` writes raw. So `0x10XX` plausibly means
"stream until done" and returns **several** frames, which the parser of the day
concatenated into the 11,049 bytes recorded below. That reconciles both
observations, but it is a hypothesis: those 2026-09-23 captures never landed in
the repo, so it cannot be re-derived from bytes on disk.
> #### ⚠ RETRACTED 2026-10-02 — there is no streaming mode
>
> This section previously hypothesised that `offset_hi = 0x10` meant "stream
> until done" and returned several frames, reconciling THOR's chunk loop with
> our 2026-09-23 single-request observation. **Measured directly on UM20147,
> it does not:**
>
> | request | one frame returns | file bytes |
> |---|---|---|
> | `offset = 0x1014` (4,116) | 4,127 B data | **4,116** |
> | `offset = 0x111c` (4,380) | 4,391 B data | **4,380** |
>
> **`offset` is simply a byte count.** The device returns exactly `offset + 11`
> bytes in one frame, and `page_key` is `offset // 256`. `0x1000` is not a
> marker — it is part of the number. One model covers every `0x5A` request ever
> observed, THOR's and ours.
>
> So the 2026-09-23 note that `offset_word = 0x1000 + 2 × pages` returned an
> entire 11 KB event is **wrong**: `0x102C` is 4,140, and 4,140 bytes is what it
> would have returned. Most likely the 11,049 figure was the whole capture
> rather than one frame's payload. Those captures never landed in the repo, so
> the error cannot be traced further — but it does not need to be, because the
> live measurement is unambiguous.
>
> 🔑 **And the negative result turned up something better. See below.**
### 🔑 1024 bytes per request is THOR's choice, not the device's limit
The retraction above has a payoff. In establishing that `offset` is a plain byte
count, UM20147 served **4,380 bytes in a single frame** without being asked
twice. THOR uses 1024; nothing about the device requires it.
That matters because of the round-trip cost: ~0.65 s each over cellular,
regardless of payload. A 72,560-byte event is **71 requests ≈ 46 seconds** at
THOR's chunk size. The offset field is a **uint16**, so the structural ceiling is
**65,535 bytes per request** — which would make that same event **2 requests,
~1.3 seconds**.
⚠ The *device's* ceiling is not yet known, only that it is at least 4,380. It may
be bounded by an internal buffer well below 65,535. `scratch/mm_stream_probe.py`
walks ascending sizes and checks each against a known-good chunk-loop download,
so a pass means byte-identical output rather than a plausible length.
⚠ `micromate/protocol.py` still uses **1024** — THOR's value, the one with
captures behind it. Raising it is a one-constant change once the ceiling is
measured, and it should not be raised on inference.
**Implement THOR's chunked form.** It is verified byte-exact across six events
and five distinct sizes, and it is what the firmware runs every day. The
single-request form is worth one bench test as an optimisation — `offset_hi =
0x10` and count the frames — but not worth depending on first.
and five distinct sizes, and it is what the firmware runs every day.
### The offset word is a LENGTH, not a position
scratch/mm_stream_probe.py
+67 -121
View File
@@ -1,50 +1,42 @@
#!/usr/bin/env python3
"""
mm_stream_probe.py — is there a one-request streaming mode for `SUB 0x5A`?
mm_stream_probe.py — how many bytes will `SUB 0x5A` serve in one request?
The question
------------
THOR downloads an event as a **chunk loop**: `ceil(size / 1024)` requests, each
asking for `min(1024, remaining)` bytes. Verified byte-for-byte against its own
frames, and confirmed on hardware up to 71 chunks.
Settled 2026-10-02: there is NO streaming mode
----------------------------------------------
This script started out asking whether `offset_hi = 0x10` meant "stream until
done", because our 2026-09-23 notes recorded a single request appearing to return
an entire 11 KB event. Measured directly on UM20147, it does not:
But our own 2026-09-23 probes recorded something different — a **single** request
with `offset_hi = 0x10` that appeared to return an entire 11 KB event:
offset 0x1014 (4116) -> one frame, 4127 B data, 4116 B of file
offset 0x111c (4380) -> one frame, 4391 B data, 4380 B of file
offset_word = 0x1000 + 2 * ceil(size / 512)
**`offset` is simply a byte count**, and the device returns exactly `offset + 11`
bytes in one frame. `0x1000` is not a marker; it is part of the number. The
2026-09-23 reading was wrong, and the protocol reference now says so.
`0x10` in `offset_hi` is exactly the bulk-stream marker Series III's
`build_5a_frame()` writes raw, so "`0x10XX` means stream until done, and the
device sends several frames" is a plausible reading. Those captures never landed
in the repo, so it cannot be re-derived from bytes on disk.
The useful question it turned up
-------------------------------
**1024 bytes per request is THOR's choice, not the device's limit.** The unit
served 4,380 bytes in a single frame without being asked twice. Since a round
trip over cellular costs ~0.65 s regardless of payload, and UM20147's
72,560-byte event is 71 chunks ≈ 46 seconds, the ceiling on one request is worth
knowing precisely: every doubling halves the dominant cost.
Why it matters
--------------
A round trip over cellular costs ~0.65 s regardless of payload. UM20147's
72,560-byte event is **71 chunks ≈ 46 seconds**. If one request can fetch it,
that becomes under a second. On a fleet of units called daily, that is the
difference between a workable receiver and an unworkable one.
So this now walks ascending request sizes against one event and **checks each
against a known-good chunk-loop download** — a pass means byte-identical output,
not merely a plausible length. The offset field is a uint16, so 65,535 is the
structural maximum.
What this does
--------------
Downloads **the same event twice** — once with the known-good chunk loop, once
with a single `0x10XX` request — and **diffs the bytes**. A differential test
rather than a suggestive one: if the streaming form returns byte-identical
output, it is safe to adopt; if it returns anything else, we learn exactly what.
Then it re-POLLs, because the honest risk here is leaving the session in an odd
state, and the script should say so rather than leave you guessing.
⚠ **Read-only.** `0x5A` is a read we have already sent thousands of times; the
only thing new is the value in its offset field. Nothing here writes, erases or
changes monitoring state. The worst realistic outcome is an unanswered frame or
a session that needs reconnecting.
⚠ **Read-only.** `0x5A` is a read we have sent thousands of times; the only new
thing is a larger value in its offset field. Nothing here writes, erases or
changes monitoring state. It re-POLLs at the end, because the honest risk is
leaving the session in an odd state and the script should say so.
Usage
-----
python3 scratch/mm_stream_probe.py /dev/ttyACM1
python3 scratch/mm_stream_probe.py /dev/ttyACM1 --event 055d4a82
python3 scratch/mm_stream_probe.py 63.45.161.30:9034 --event smallest
python3 scratch/mm_stream_probe.py /dev/ttyACM1 --event largest
"""
from __future__ import annotations
@@ -142,104 +134,58 @@ def main() -> int:
print(f"\n [1] chunk loop ...... {len(chunked)} B in {dt_chunked:.2f} s "
f"({n_chunks} requests)")
# ── 2. one request, offset_hi = 0x10 ──────────────────────────────────
# The form our 2026-09-23 probes recorded. `pages` is 512-byte pages;
# the +0x1000 is the bulk-stream marker.
pages = math.ceil(ref.size / 512)
offset = 0x1000 + 2 * pages
params = ref.key + bytes(6)
# ── 2. how many bytes will it serve in ONE frame? ─────────────────
# The streaming hypothesis is dead (see the module docstring): `offset`
# is simply a BYTE COUNT, and the device returns `offset + 11` bytes in
# one frame. So the real question is the ceiling -- because 1024 is
# THOR's choice, not the device's limit, and every doubling halves the
# round trips that dominate a cellular download.
print("\n [2] chunk-size ceiling — ascending single requests")
print(" each asks for N bytes from offset 0 and is checked against")
print(" the known-good download, so a pass means identical bytes.\n")
# ⚠ TRY BOTH ESCAPINGS, or a null result means nothing.
#
# On Series III, `offset_hi = 0x10` in a 5A frame must be written RAW --
# doubled to `10 10`, the device SILENTLY IGNORES the frame. That is a
# documented, hard-won rule for this exact command.
#
# The Micromate escapes `0x04` in offset_hi (218/218 captured THOR
# frames), which argues the uniform escape set applies to 0x10 too. But
# Series III is a direct counter-example in the same command, so testing
# only one form risks concluding "no streaming mode" when the real
# finding is "that frame was malformed".
escaped = build_request(SUB_BULK_DOWNLOAD, offset, params)
candidates = [("escaped offset_hi (uniform rule)", escaped)]
if (offset >> 8) == 0x10:
# Hand-build the raw form: the uniform builder cannot express it.
payload = bytes([0x10, 0x00, SUB_BULK_DOWNLOAD, 0x00]) + \
bytes([(offset >> 8) & 0xFF, offset & 0xFF]) + params
body = payload + bytes([sum(payload) & 0xFF])
out = bytearray([0x41, 0x02])
for i, b in enumerate(body):
# escape everything EXCEPT the offset_hi at payload index 4
if b in (0x02, 0x03, 0x04, 0x10) and i != 4:
out.append(0x10)
out.append(b)
out.append(0x03)
candidates.append(("RAW offset_hi (series-III rule)", bytes(out)))
else:
print(f"\n note: offset_hi is 0x{offset >> 8:02x}, not 0x10, so the "
f"escaping question does not arise for this event")
candidates = [1024, 2048, 4096, 8192, 16384, 32768, 65535]
candidates = [n for n in candidates if n <= ref.size] or [ref.size]
if ref.size not in candidates and ref.size < 65536:
candidates.append(ref.size) # the whole event in one request
print(f"\n [2] streaming ....... one request, offset=0x{offset:04x} "
f"(0x1000 + 2 x {pages} pages)")
frames, dt_stream, used = [], 0.0, None
for name, frame in candidates:
print(f"\n trying {name}")
print(f" wire: {frame.hex(' ')}")
best = None
for n in candidates:
frame = build_request(SUB_BULK_DOWNLOAD, n, ref.key + bytes(6))
parser = MicromateFrameParser()
t0 = time.monotonic()
mm.protocol._send(frame)
got = collect(inner, parser, idle_gap=a.idle_gap,
deadline=t0 + a.timeout)
dt = time.monotonic() - t0
print(f" -> {len(got)} frame(s), {parser.bytes_fed} raw bytes, "
f"{dt:.2f} s")
if got:
frames, dt_stream, used = got, dt, name
break
if used:
print(f"\n answered by: {used}")
if not frames:
print("\n VERDICT: no answer to EITHER escaping. The 0x10XX form")
print(" is not a streaming mode — or not with these params. Since")
print(" both escapings were tried, this is not a framing artefact.")
print(" The chunk loop stands as the only way to download an event,")
print(" and the 2026-09-23 note should be retracted.")
else:
bad = [f for f in frames if not f.checksum_valid]
subs = sorted({f"0x{f.sub:02x}" for f in frames})
print(f" SUBs {subs}, {len(bad)} bad checksum")
# Assemble the same way a chunk response is assembled.
streamed = b"".join(f.data[_CHUNK_PREFIX:] for f in frames)
print(f" assembled {len(streamed)} B "
f"(event is {ref.size} B)")
if not got:
print(f" {n:6} B no answer ({dt:.2f} s)")
continue
body = b"".join(f.data[_CHUNK_PREFIX:] for f in got)
ok = body == chunked[:n]
flag = "OK " if ok else "MISMATCH"
print(f" {n:6} B {len(got)} frame(s) {len(body):6} B back "
f"{dt:5.2f} s {flag}"
+ ("" if ok or not body else
f" (first diff at {next((i for i in range(min(len(body), n)) if body[i] != chunked[i]), None)})"))
if ok and len(body) == n:
best = n
print()
if streamed == chunked:
print(f" VERDICT: ** BYTE-IDENTICAL ** in {len(frames)} frame(s) "
f"against {n_chunks}.")
print(f" {dt_chunked:.2f} s -> {dt_stream:.2f} s here; over "
f"cellular that is ~{n_chunks * 0.65:.0f} s -> ~0.7 s.")
print(" The streaming mode is real. Worth adopting.")
elif len(streamed) == ref.size:
print(" VERDICT: right LENGTH, wrong BYTES. So it streams, but")
print(" the assembly differs — likely a different per-frame")
print(" prefix than the chunk form's 11 bytes. Compare below.")
for i in range(min(len(streamed), len(chunked))):
if streamed[i] != chunked[i]:
print(f" first difference at byte {i}")
print(f" chunked {chunked[max(0,i-4):i+8].hex(' ')}")
print(f" streamed {streamed[max(0,i-4):i+8].hex(' ')}")
break
if best is None:
print(" VERDICT: nothing above the current chunk size verified.")
else:
print(f" VERDICT: answered, but {len(streamed)} B against "
f"{ref.size} B expected.")
print(" Inconclusive — dump the frame sizes and look again:")
for i, f in enumerate(frames[:12]):
print(f" frame {i}: data {len(f.data)} B, "
f"page=0x{f.page_key:04x}")
now = math.ceil(ref.size / 1024)
then = math.ceil(ref.size / best)
print(f" VERDICT: the device serves at least {best} B per request,")
print(f" verified byte-identical. For this {ref.size} B event that")
print(f" is {then} request(s) instead of {now}.")
if best > 1024:
print(f" Over cellular at ~0.65 s per round trip: "
f"~{now * 0.65:.0f} s -> ~{then * 0.65:.1f} s.")
if best >= ref.size:
print(" The WHOLE EVENT fits in one request.")
# ── 3. is the unit still healthy? ─────────────────────────────────────
# The real risk of this experiment is leaving the session wedged, so