serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs(series4): RETRACTED -- there is no 0x5A streaming mode; offset is a byte count

Browse Source 
Measured directly on UM20147.  offset 0x1014 (4,116) returned one frame of 4,127
B data = 4,116 B of file; offset 0x111c (4,380) returned 4,391 B = 4,380 B.  So
`offset` is simply a BYTE COUNT, the device returns exactly offset + 11 bytes in
one frame, and page_key is offset // 256.  `0x1000` is not a bulk-stream marker;
it is part of the number.  One model now covers every 0x5A request ever observed,
THOR's and ours.

The 2026-09-23 note that offset_word = 0x1000 + 2*pages returned an entire 11 KB
event is therefore wrong -- 0x102C is 4,140, and 4,140 bytes is what it would
have returned.  Most likely the 11,049 figure was the whole capture rather than
one frame's payload.  Those captures never landed in the repo so the error cannot
be traced further, and does not need to be: the live measurement is unambiguous.

The probe tried BOTH escapings of offset_hi, which is why this negative counts --
a malformed frame would have produced the same silence.  The escaped form
answered, so the uniform escape set holds for 0x10 in offset_hi too, and the
Series III exception does not carry over.

AND THE NEGATIVE TURNED UP SOMETHING BETTER.  In establishing that offset is a
byte count, the unit served 4,380 bytes in a SINGLE frame.  1024 is THOR's
choice, not the device's limit.  Since a round trip costs ~0.65 s over cellular
regardless of payload, and a 72,560-byte event is 71 requests ~ 46 s at THOR's
chunk size, the ceiling is worth knowing precisely: the offset field is a uint16,
so 65,535 B per request would make that same event 2 requests, ~1.3 s.

mm_stream_probe.py repurposed to walk ascending request sizes, checking each
against a known-good chunk-loop download so a pass means byte-identical output
rather than a plausible length.

micromate/protocol.py still uses 1024 -- THOR's value, the one with captures
behind it.  Raising it is a one-constant change once the ceiling is MEASURED, and
it should not be raised on inference.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
serversdown
2026-10-02 01:28:43 -04:00
co-authored by Claude Opus 5
parent 18ed1a27be
commit 248ddc9cda
2 changed files with 116 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scratch/mm_stream_probe.py
+69 -123
View File
@@ -1,50 +1,42 @@
#!/usr/bin/env python3
"""
mm_stream_probe.py — is there a one-request streaming mode for `SUB 0x5A`?
mm_stream_probe.py — how many bytes will `SUB 0x5A` serve in one request?
The question
------------
THOR downloads an event as a **chunk loop**: `ceil(size / 1024)` requests, each
asking for `min(1024, remaining)` bytes. Verified byte-for-byte against its own
frames, and confirmed on hardware up to 71 chunks.
Settled 2026-10-02: there is NO streaming mode
----------------------------------------------
This script started out asking whether `offset_hi = 0x10` meant "stream until
done", because our 2026-09-23 notes recorded a single request appearing to return
an entire 11 KB event. Measured directly on UM20147, it does not:
But our own 2026-09-23 probes recorded something different — a **single** request
with `offset_hi = 0x10` that appeared to return an entire 11 KB event:
offset 0x1014 (4116) -> one frame, 4127 B data, 4116 B of file
offset 0x111c (4380) -> one frame, 4391 B data, 4380 B of file
offset_word = 0x1000 + 2 * ceil(size / 512)
**`offset` is simply a byte count**, and the device returns exactly `offset + 11`
bytes in one frame. `0x1000` is not a marker; it is part of the number. The
2026-09-23 reading was wrong, and the protocol reference now says so.
`0x10` in `offset_hi` is exactly the bulk-stream marker Series III's
`build_5a_frame()` writes raw, so "`0x10XX` means stream until done, and the
device sends several frames" is a plausible reading. Those captures never landed
in the repo, so it cannot be re-derived from bytes on disk.
The useful question it turned up
-------------------------------
**1024 bytes per request is THOR's choice, not the device's limit.** The unit
served 4,380 bytes in a single frame without being asked twice. Since a round
trip over cellular costs ~0.65 s regardless of payload, and UM20147's
72,560-byte event is 71 chunks ≈ 46 seconds, the ceiling on one request is worth
knowing precisely: every doubling halves the dominant cost.
Why it matters
--------------
A round trip over cellular costs ~0.65 s regardless of payload. UM20147's
72,560-byte event is **71 chunks ≈ 46 seconds**. If one request can fetch it,
that becomes under a second. On a fleet of units called daily, that is the
difference between a workable receiver and an unworkable one.
So this now walks ascending request sizes against one event and **checks each
against a known-good chunk-loop download** — a pass means byte-identical output,
not merely a plausible length. The offset field is a uint16, so 65,535 is the
structural maximum.
What this does
--------------
Downloads **the same event twice** — once with the known-good chunk loop, once
with a single `0x10XX` request — and **diffs the bytes**. A differential test
rather than a suggestive one: if the streaming form returns byte-identical
output, it is safe to adopt; if it returns anything else, we learn exactly what.
Then it re-POLLs, because the honest risk here is leaving the session in an odd
state, and the script should say so rather than leave you guessing.
⚠ **Read-only.** `0x5A` is a read we have already sent thousands of times; the
only thing new is the value in its offset field. Nothing here writes, erases or
changes monitoring state. The worst realistic outcome is an unanswered frame or
a session that needs reconnecting.
⚠ **Read-only.** `0x5A` is a read we have sent thousands of times; the only new
thing is a larger value in its offset field. Nothing here writes, erases or
changes monitoring state. It re-POLLs at the end, because the honest risk is
leaving the session in an odd state and the script should say so.
Usage
-----
python3 scratch/mm_stream_probe.py /dev/ttyACM1
python3 scratch/mm_stream_probe.py /dev/ttyACM1 --event 055d4a82
python3 scratch/mm_stream_probe.py 63.45.161.30:9034 --event smallest
python3 scratch/mm_stream_probe.py /dev/ttyACM1 --event largest
"""
from __future__ import annotations
@@ -142,104 +134,58 @@ def main() -> int:
print(f"\n [1] chunk loop ...... {len(chunked)} B in {dt_chunked:.2f} s "
f"({n_chunks} requests)")
# ── 2. one request, offset_hi = 0x10 ──────────────────────────────────
# The form our 2026-09-23 probes recorded. `pages` is 512-byte pages;
# the +0x1000 is the bulk-stream marker.
pages = math.ceil(ref.size / 512)
offset = 0x1000 + 2 * pages
params = ref.key + bytes(6)
# ── 2. how many bytes will it serve in ONE frame? ─────────────────
# The streaming hypothesis is dead (see the module docstring): `offset`
# is simply a BYTE COUNT, and the device returns `offset + 11` bytes in
# one frame. So the real question is the ceiling -- because 1024 is
# THOR's choice, not the device's limit, and every doubling halves the
# round trips that dominate a cellular download.
print("\n [2] chunk-size ceiling — ascending single requests")
print(" each asks for N bytes from offset 0 and is checked against")
print(" the known-good download, so a pass means identical bytes.\n")
# ⚠ TRY BOTH ESCAPINGS, or a null result means nothing.
#
# On Series III, `offset_hi = 0x10` in a 5A frame must be written RAW --
# doubled to `10 10`, the device SILENTLY IGNORES the frame. That is a
# documented, hard-won rule for this exact command.
#
# The Micromate escapes `0x04` in offset_hi (218/218 captured THOR
# frames), which argues the uniform escape set applies to 0x10 too. But
# Series III is a direct counter-example in the same command, so testing
# only one form risks concluding "no streaming mode" when the real
# finding is "that frame was malformed".
escaped = build_request(SUB_BULK_DOWNLOAD, offset, params)
candidates = [("escaped offset_hi (uniform rule)", escaped)]
if (offset >> 8) == 0x10:
# Hand-build the raw form: the uniform builder cannot express it.
payload = bytes([0x10, 0x00, SUB_BULK_DOWNLOAD, 0x00]) + \
bytes([(offset >> 8) & 0xFF, offset & 0xFF]) + params
body = payload + bytes([sum(payload) & 0xFF])
out = bytearray([0x41, 0x02])
for i, b in enumerate(body):
# escape everything EXCEPT the offset_hi at payload index 4
if b in (0x02, 0x03, 0x04, 0x10) and i != 4:
out.append(0x10)
out.append(b)
out.append(0x03)
candidates.append(("RAW offset_hi (series-III rule)", bytes(out)))
else:
print(f"\n note: offset_hi is 0x{offset >> 8:02x}, not 0x10, so the "
f"escaping question does not arise for this event")
candidates = [1024, 2048, 4096, 8192, 16384, 32768, 65535]
candidates = [n for n in candidates if n <= ref.size] or [ref.size]
if ref.size not in candidates and ref.size < 65536:
candidates.append(ref.size) # the whole event in one request
print(f"\n [2] streaming ....... one request, offset=0x{offset:04x} "
f"(0x1000 + 2 x {pages} pages)")
frames, dt_stream, used = [], 0.0, None
for name, frame in candidates:
print(f"\n trying {name}")
print(f" wire: {frame.hex(' ')}")
best = None
for n in candidates:
frame = build_request(SUB_BULK_DOWNLOAD, n, ref.key + bytes(6))
parser = MicromateFrameParser()
t0 = time.monotonic()
mm.protocol._send(frame)
got = collect(inner, parser, idle_gap=a.idle_gap,
deadline=t0 + a.timeout)
dt = time.monotonic() - t0
print(f" -> {len(got)} frame(s), {parser.bytes_fed} raw bytes, "
f"{dt:.2f} s")
if got:
frames, dt_stream, used = got, dt, name
break
if used:
print(f"\n answered by: {used}")
if not frames:
print("\n VERDICT: no answer to EITHER escaping. The 0x10XX form")
print(" is not a streaming mode — or not with these params. Since")
print(" both escapings were tried, this is not a framing artefact.")
print(" The chunk loop stands as the only way to download an event,")
print(" and the 2026-09-23 note should be retracted.")
if not got:
print(f" {n:6} B no answer ({dt:.2f} s)")
continue
body = b"".join(f.data[_CHUNK_PREFIX:] for f in got)
ok = body == chunked[:n]
flag = "OK " if ok else "MISMATCH"
print(f" {n:6} B {len(got)} frame(s) {len(body):6} B back "
f"{dt:5.2f} s {flag}"
+ ("" if ok or not body else
f" (first diff at {next((i for i in range(min(len(body), n)) if body[i] != chunked[i]), None)})"))
if ok and len(body) == n:
best = n
print()
if best is None:
print(" VERDICT: nothing above the current chunk size verified.")
else:
bad = [f for f in frames if not f.checksum_valid]
subs = sorted({f"0x{f.sub:02x}" for f in frames})
print(f" SUBs {subs}, {len(bad)} bad checksum")
# Assemble the same way a chunk response is assembled.
streamed = b"".join(f.data[_CHUNK_PREFIX:] for f in frames)
print(f" assembled {len(streamed)} B "
f"(event is {ref.size} B)")
print()
if streamed == chunked:
print(f" VERDICT: ** BYTE-IDENTICAL ** in {len(frames)} frame(s) "
f"against {n_chunks}.")
print(f" {dt_chunked:.2f} s -> {dt_stream:.2f} s here; over "
f"cellular that is ~{n_chunks * 0.65:.0f} s -> ~0.7 s.")
print(" The streaming mode is real. Worth adopting.")
elif len(streamed) == ref.size:
print(" VERDICT: right LENGTH, wrong BYTES. So it streams, but")
print(" the assembly differs — likely a different per-frame")
print(" prefix than the chunk form's 11 bytes. Compare below.")
for i in range(min(len(streamed), len(chunked))):
if streamed[i] != chunked[i]:
print(f" first difference at byte {i}")
print(f" chunked {chunked[max(0,i-4):i+8].hex(' ')}")
print(f" streamed {streamed[max(0,i-4):i+8].hex(' ')}")
break
else:
print(f" VERDICT: answered, but {len(streamed)} B against "
f"{ref.size} B expected.")
print(" Inconclusive — dump the frame sizes and look again:")
for i, f in enumerate(frames[:12]):
print(f" frame {i}: data {len(f.data)} B, "
f"page=0x{f.page_key:04x}")
now = math.ceil(ref.size / 1024)
then = math.ceil(ref.size / best)
print(f" VERDICT: the device serves at least {best} B per request,")
print(f" verified byte-identical. For this {ref.size} B event that")
print(f" is {then} request(s) instead of {now}.")
if best > 1024:
print(f" Over cellular at ~0.65 s per round trip: "
f"~{now * 0.65:.0f} s -> ~{then * 0.65:.1f} s.")
if best >= ref.size:
print(" The WHOLE EVENT fits in one request.")
# ── 3. is the unit still healthy? ─────────────────────────────────────
# The real risk of this experiment is leaving the session wedged, so