serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs(series4): RETRACTED -- there is no 0x5A streaming mode; offset is a byte count

Browse Source 
Measured directly on UM20147.  offset 0x1014 (4,116) returned one frame of 4,127
B data = 4,116 B of file; offset 0x111c (4,380) returned 4,391 B = 4,380 B.  So
`offset` is simply a BYTE COUNT, the device returns exactly offset + 11 bytes in
one frame, and page_key is offset // 256.  `0x1000` is not a bulk-stream marker;
it is part of the number.  One model now covers every 0x5A request ever observed,
THOR's and ours.

The 2026-09-23 note that offset_word = 0x1000 + 2*pages returned an entire 11 KB
event is therefore wrong -- 0x102C is 4,140, and 4,140 bytes is what it would
have returned.  Most likely the 11,049 figure was the whole capture rather than
one frame's payload.  Those captures never landed in the repo so the error cannot
be traced further, and does not need to be: the live measurement is unambiguous.

The probe tried BOTH escapings of offset_hi, which is why this negative counts --
a malformed frame would have produced the same silence.  The escaped form
answered, so the uniform escape set holds for 0x10 in offset_hi too, and the
Series III exception does not carry over.

AND THE NEGATIVE TURNED UP SOMETHING BETTER.  In establishing that offset is a
byte count, the unit served 4,380 bytes in a SINGLE frame.  1024 is THOR's
choice, not the device's limit.  Since a round trip costs ~0.65 s over cellular
regardless of payload, and a 72,560-byte event is 71 requests ~ 46 s at THOR's
chunk size, the ceiling is worth knowing precisely: the offset field is a uint16,
so 65,535 B per request would make that same event 2 requests, ~1.3 s.

mm_stream_probe.py repurposed to walk ascending request sizes, checking each
against a known-good chunk-loop download so a pass means byte-identical output
rather than a plausible length.

micromate/protocol.py still uses 1024 -- THOR's value, the one with captures
behind it.  Raising it is a one-constant change once the ceiling is MEASURED, and
it should not be raised on inference.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
serversdown
2026-10-02 01:28:43 -04:00
co-authored by Claude Opus 5
parent 18ed1a27be
commit 248ddc9cda
2 changed files with 116 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/micromate_protocol_reference.md
+47 -11
View File
@@ -694,19 +694,55 @@ key size(1E) chunks sum(offsets) last offset
055d4a86 6092 6 6092 0x03cc 055d4a86 6092 6 6092 0x03cc
``` ```
**These are probably two different modes, not a contradiction.** THOR's > #### ⚠ RETRACTED 2026-10-02 — there is no streaming mode
`offset_hi` is the chunk length (`0x04`, `0x03`, `0x00` …). Our single-request >
probes set `offset_hi = 0x10` — which in Series III is precisely the > This section previously hypothesised that `offset_hi = 0x10` meant "stream
bulk-stream marker `build_5a_frame()` writes raw. So `0x10XX` plausibly means > until done" and returned several frames, reconciling THOR's chunk loop with
"stream until done" and returns **several** frames, which the parser of the day > our 2026-09-23 single-request observation. **Measured directly on UM20147,
concatenated into the 11,049 bytes recorded below. That reconciles both > it does not:**
observations, but it is a hypothesis: those 2026-09-23 captures never landed in >
the repo, so it cannot be re-derived from bytes on disk. > | request | one frame returns | file bytes |
> |---|---|---|
> | `offset = 0x1014` (4,116) | 4,127 B data | **4,116** |
> | `offset = 0x111c` (4,380) | 4,391 B data | **4,380** |
>
> **`offset` is simply a byte count.** The device returns exactly `offset + 11`
> bytes in one frame, and `page_key` is `offset // 256`. `0x1000` is not a
> marker — it is part of the number. One model covers every `0x5A` request ever
> observed, THOR's and ours.
>
> So the 2026-09-23 note that `offset_word = 0x1000 + 2 × pages` returned an
> entire 11 KB event is **wrong**: `0x102C` is 4,140, and 4,140 bytes is what it
> would have returned. Most likely the 11,049 figure was the whole capture
> rather than one frame's payload. Those captures never landed in the repo, so
> the error cannot be traced further — but it does not need to be, because the
> live measurement is unambiguous.
>
> 🔑 **And the negative result turned up something better. See below.**
### 🔑 1024 bytes per request is THOR's choice, not the device's limit
The retraction above has a payoff. In establishing that `offset` is a plain byte
count, UM20147 served **4,380 bytes in a single frame** without being asked
twice. THOR uses 1024; nothing about the device requires it.
That matters because of the round-trip cost: ~0.65 s each over cellular,
regardless of payload. A 72,560-byte event is **71 requests ≈ 46 seconds** at
THOR's chunk size. The offset field is a **uint16**, so the structural ceiling is
**65,535 bytes per request** — which would make that same event **2 requests,
~1.3 seconds**.
⚠ The *device's* ceiling is not yet known, only that it is at least 4,380. It may
be bounded by an internal buffer well below 65,535. `scratch/mm_stream_probe.py`
walks ascending sizes and checks each against a known-good chunk-loop download,
so a pass means byte-identical output rather than a plausible length.
⚠ `micromate/protocol.py` still uses **1024** — THOR's value, the one with
captures behind it. Raising it is a one-constant change once the ceiling is
measured, and it should not be raised on inference.
**Implement THOR's chunked form.** It is verified byte-exact across six events **Implement THOR's chunked form.** It is verified byte-exact across six events
and five distinct sizes, and it is what the firmware runs every day. The and five distinct sizes, and it is what the firmware runs every day.
single-request form is worth one bench test as an optimisation — `offset_hi =
0x10` and count the frames — but not worth depending on first.
### The offset word is a LENGTH, not a position ### The offset word is a LENGTH, not a position
scratch/mm_stream_probe.py
+67 -121
View File
@@ -1,50 +1,42 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
""" """
mm_stream_probe.py — is there a one-request streaming mode for `SUB 0x5A`? mm_stream_probe.py — how many bytes will `SUB 0x5A` serve in one request?
The question Settled 2026-10-02: there is NO streaming mode
------------ ----------------------------------------------
THOR downloads an event as a **chunk loop**: `ceil(size / 1024)` requests, each This script started out asking whether `offset_hi = 0x10` meant "stream until
asking for `min(1024, remaining)` bytes. Verified byte-for-byte against its own done", because our 2026-09-23 notes recorded a single request appearing to return
frames, and confirmed on hardware up to 71 chunks. an entire 11 KB event. Measured directly on UM20147, it does not:
But our own 2026-09-23 probes recorded something different — a **single** request offset 0x1014 (4116) -> one frame, 4127 B data, 4116 B of file
with `offset_hi = 0x10` that appeared to return an entire 11 KB event: offset 0x111c (4380) -> one frame, 4391 B data, 4380 B of file
offset_word = 0x1000 + 2 * ceil(size / 512) **`offset` is simply a byte count**, and the device returns exactly `offset + 11`
bytes in one frame. `0x1000` is not a marker; it is part of the number. The
2026-09-23 reading was wrong, and the protocol reference now says so.
`0x10` in `offset_hi` is exactly the bulk-stream marker Series III's The useful question it turned up
`build_5a_frame()` writes raw, so "`0x10XX` means stream until done, and the -------------------------------
device sends several frames" is a plausible reading. Those captures never landed **1024 bytes per request is THOR's choice, not the device's limit.** The unit
in the repo, so it cannot be re-derived from bytes on disk. served 4,380 bytes in a single frame without being asked twice. Since a round
trip over cellular costs ~0.65 s regardless of payload, and UM20147's
72,560-byte event is 71 chunks ≈ 46 seconds, the ceiling on one request is worth
knowing precisely: every doubling halves the dominant cost.
Why it matters So this now walks ascending request sizes against one event and **checks each
-------------- against a known-good chunk-loop download** — a pass means byte-identical output,
A round trip over cellular costs ~0.65 s regardless of payload. UM20147's not merely a plausible length. The offset field is a uint16, so 65,535 is the
72,560-byte event is **71 chunks ≈ 46 seconds**. If one request can fetch it, structural maximum.
that becomes under a second. On a fleet of units called daily, that is the
difference between a workable receiver and an unworkable one.
What this does ⚠ **Read-only.** `0x5A` is a read we have sent thousands of times; the only new
-------------- thing is a larger value in its offset field. Nothing here writes, erases or
Downloads **the same event twice** — once with the known-good chunk loop, once changes monitoring state. It re-POLLs at the end, because the honest risk is
with a single `0x10XX` request — and **diffs the bytes**. A differential test leaving the session in an odd state and the script should say so.
rather than a suggestive one: if the streaming form returns byte-identical
output, it is safe to adopt; if it returns anything else, we learn exactly what.
Then it re-POLLs, because the honest risk here is leaving the session in an odd
state, and the script should say so rather than leave you guessing.
⚠ **Read-only.** `0x5A` is a read we have already sent thousands of times; the
only thing new is the value in its offset field. Nothing here writes, erases or
changes monitoring state. The worst realistic outcome is an unanswered frame or
a session that needs reconnecting.
Usage Usage
----- -----
python3 scratch/mm_stream_probe.py /dev/ttyACM1 python3 scratch/mm_stream_probe.py /dev/ttyACM1
python3 scratch/mm_stream_probe.py /dev/ttyACM1 --event 055d4a82 python3 scratch/mm_stream_probe.py /dev/ttyACM1 --event largest
python3 scratch/mm_stream_probe.py 63.45.161.30:9034 --event smallest
""" """
from __future__ import annotations from __future__ import annotations
@@ -142,104 +134,58 @@ def main() -> int:
print(f"\n [1] chunk loop ...... {len(chunked)} B in {dt_chunked:.2f} s " print(f"\n [1] chunk loop ...... {len(chunked)} B in {dt_chunked:.2f} s "
f"({n_chunks} requests)") f"({n_chunks} requests)")
# ── 2. one request, offset_hi = 0x10 ────────────────────────────────── # ── 2. how many bytes will it serve in ONE frame? ─────────────────
# The form our 2026-09-23 probes recorded. `pages` is 512-byte pages; # The streaming hypothesis is dead (see the module docstring): `offset`
# the +0x1000 is the bulk-stream marker. # is simply a BYTE COUNT, and the device returns `offset + 11` bytes in
pages = math.ceil(ref.size / 512) # one frame. So the real question is the ceiling -- because 1024 is
offset = 0x1000 + 2 * pages # THOR's choice, not the device's limit, and every doubling halves the
params = ref.key + bytes(6) # round trips that dominate a cellular download.
print("\n [2] chunk-size ceiling — ascending single requests")
print(" each asks for N bytes from offset 0 and is checked against")
print(" the known-good download, so a pass means identical bytes.\n")
# ⚠ TRY BOTH ESCAPINGS, or a null result means nothing. candidates = [1024, 2048, 4096, 8192, 16384, 32768, 65535]
# candidates = [n for n in candidates if n <= ref.size] or [ref.size]
# On Series III, `offset_hi = 0x10` in a 5A frame must be written RAW -- if ref.size not in candidates and ref.size < 65536:
# doubled to `10 10`, the device SILENTLY IGNORES the frame. That is a candidates.append(ref.size) # the whole event in one request
# documented, hard-won rule for this exact command.
#
# The Micromate escapes `0x04` in offset_hi (218/218 captured THOR
# frames), which argues the uniform escape set applies to 0x10 too. But
# Series III is a direct counter-example in the same command, so testing
# only one form risks concluding "no streaming mode" when the real
# finding is "that frame was malformed".
escaped = build_request(SUB_BULK_DOWNLOAD, offset, params)
candidates = [("escaped offset_hi (uniform rule)", escaped)]
if (offset >> 8) == 0x10:
# Hand-build the raw form: the uniform builder cannot express it.
payload = bytes([0x10, 0x00, SUB_BULK_DOWNLOAD, 0x00]) + \
bytes([(offset >> 8) & 0xFF, offset & 0xFF]) + params
body = payload + bytes([sum(payload) & 0xFF])
out = bytearray([0x41, 0x02])
for i, b in enumerate(body):
# escape everything EXCEPT the offset_hi at payload index 4
if b in (0x02, 0x03, 0x04, 0x10) and i != 4:
out.append(0x10)
out.append(b)
out.append(0x03)
candidates.append(("RAW offset_hi (series-III rule)", bytes(out)))
else:
print(f"\n note: offset_hi is 0x{offset >> 8:02x}, not 0x10, so the "
f"escaping question does not arise for this event")
print(f"\n [2] streaming ....... one request, offset=0x{offset:04x} " best = None
f"(0x1000 + 2 x {pages} pages)") for n in candidates:
frame = build_request(SUB_BULK_DOWNLOAD, n, ref.key + bytes(6))
frames, dt_stream, used = [], 0.0, None
for name, frame in candidates:
print(f"\n trying {name}")
print(f" wire: {frame.hex(' ')}")
parser = MicromateFrameParser() parser = MicromateFrameParser()
t0 = time.monotonic() t0 = time.monotonic()
mm.protocol._send(frame) mm.protocol._send(frame)
got = collect(inner, parser, idle_gap=a.idle_gap, got = collect(inner, parser, idle_gap=a.idle_gap,
deadline=t0 + a.timeout) deadline=t0 + a.timeout)
dt = time.monotonic() - t0 dt = time.monotonic() - t0
print(f" -> {len(got)} frame(s), {parser.bytes_fed} raw bytes, "
f"{dt:.2f} s")
if got:
frames, dt_stream, used = got, dt, name
break
if used: if not got:
print(f"\n answered by: {used}") print(f" {n:6} B no answer ({dt:.2f} s)")
if not frames: continue
print("\n VERDICT: no answer to EITHER escaping. The 0x10XX form") body = b"".join(f.data[_CHUNK_PREFIX:] for f in got)
print(" is not a streaming mode — or not with these params. Since") ok = body == chunked[:n]
print(" both escapings were tried, this is not a framing artefact.") flag = "OK " if ok else "MISMATCH"
print(" The chunk loop stands as the only way to download an event,") print(f" {n:6} B {len(got)} frame(s) {len(body):6} B back "
print(" and the 2026-09-23 note should be retracted.") f"{dt:5.2f} s {flag}"
else: + ("" if ok or not body else
bad = [f for f in frames if not f.checksum_valid] f" (first diff at {next((i for i in range(min(len(body), n)) if body[i] != chunked[i]), None)})"))
subs = sorted({f"0x{f.sub:02x}" for f in frames}) if ok and len(body) == n:
print(f" SUBs {subs}, {len(bad)} bad checksum") best = n
# Assemble the same way a chunk response is assembled.
streamed = b"".join(f.data[_CHUNK_PREFIX:] for f in frames)
print(f" assembled {len(streamed)} B "
f"(event is {ref.size} B)")
print() print()
if streamed == chunked: if best is None:
print(f" VERDICT: ** BYTE-IDENTICAL ** in {len(frames)} frame(s) " print(" VERDICT: nothing above the current chunk size verified.")
f"against {n_chunks}.")
print(f" {dt_chunked:.2f} s -> {dt_stream:.2f} s here; over "
f"cellular that is ~{n_chunks * 0.65:.0f} s -> ~0.7 s.")
print(" The streaming mode is real. Worth adopting.")
elif len(streamed) == ref.size:
print(" VERDICT: right LENGTH, wrong BYTES. So it streams, but")
print(" the assembly differs — likely a different per-frame")
print(" prefix than the chunk form's 11 bytes. Compare below.")
for i in range(min(len(streamed), len(chunked))):
if streamed[i] != chunked[i]:
print(f" first difference at byte {i}")
print(f" chunked {chunked[max(0,i-4):i+8].hex(' ')}")
print(f" streamed {streamed[max(0,i-4):i+8].hex(' ')}")
break
else: else:
print(f" VERDICT: answered, but {len(streamed)} B against " now = math.ceil(ref.size / 1024)
f"{ref.size} B expected.") then = math.ceil(ref.size / best)
print(" Inconclusive — dump the frame sizes and look again:") print(f" VERDICT: the device serves at least {best} B per request,")
for i, f in enumerate(frames[:12]): print(f" verified byte-identical. For this {ref.size} B event that")
print(f" frame {i}: data {len(f.data)} B, " print(f" is {then} request(s) instead of {now}.")
f"page=0x{f.page_key:04x}") if best > 1024:
print(f" Over cellular at ~0.65 s per round trip: "
f"~{now * 0.65:.0f} s -> ~{then * 0.65:.1f} s.")
if best >= ref.size:
print(" The WHOLE EVENT fits in one request.")
# ── 3. is the unit still healthy? ───────────────────────────────────── # ── 3. is the unit still healthy? ─────────────────────────────────────
# The real risk of this experiment is leaving the session wedged, so # The real risk of this experiment is leaving the session wedged, so