serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

perf(micromate): 16 KB per request, not 1024 -- 14x fewer round trips

Browse Source 
The chunk-size ceiling, measured on UM20147 with every size checked
byte-for-byte against a known-good download:

  1024 / 2048 / 4096 / 8192 / 16384  ->  served in full
  32768 / 65535                      ->  SILENTLY CLAMPED to 16384

The clamp is the important part: a 32,768 B request returns 16,384 B of perfectly
good data and no error.  Nothing in the response says it was truncated; the
length is the only signal.

THAT DICTATES HOW THE LOOP MUST BE WRITTEN.  read_event_file() now tracks its
offset by BYTES RECEIVED rather than striding by chunk index.  A fixed stride
would either fail on a clamp or skip the bytes it never collected; tracking what
actually arrived makes a clamp cost one extra request, and makes the loop
self-correcting against any short response -- precisely the failure mode that has
bitten the Series III side repeatedly.

Consequence of that change, recorded because it reverses an earlier decision: a
short chunk is NO LONGER AN ERROR.  It used to raise, on the principle that a
silently short event is this codebase's recurring bug.  But the device returns
short legitimately, and the real protection is the offset arithmetic plus the
final total-length check -- which still raises on a genuinely truncated event.
The test was rewritten rather than deleted, and says why.

CHUNK_SIZE = 16384.  For UM20147's events: 4,796 B goes 5 requests -> 1,
30,230 B goes 30 -> 2, and 72,560 B goes 71 -> 5.  Over cellular at ~0.65 s per
round trip that is ~46 s -> ~3.2 s on the large one.

Measured on one unit over USB, so THOR_CHUNK_SIZE = 1024 stays available and the
replay tests pin it -- reproducing THOR's exact traffic is one argument away, and
client.download_event()/get_event() take chunk_size for a link where large
responses are not surviving.

Full suite: 472 passed, 16 pre-existing failures unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
serversdown
2026-10-02 14:14:02 -04:00
co-authored by Claude Opus 5
parent 248ddc9cda
commit 423608ccd1
5 changed files with 194 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
micromate/client.py
+11 -4
View File
@@ -474,16 +474,23 @@ class MicromateClient:
ref.peak_vector_sum_ips = struct.unpack(
">f", c[tran - _REC_PVS_BACK: tran - _REC_PVS_BACK + 4])[0]
def download_event(self, ref: MicromateEventRef) -> bytes:
def download_event(self, ref: MicromateEventRef, *,
chunk_size: Optional[int] = None) -> bytes:
"""The raw `.IDFW`/`.IDFH` bytes, exactly as THOR would have stored them.
Feeds `micromate.idf_file.read_idf_file()` and `/db/import/idf_file`
unchanged — no new codec work is needed for a directly downloaded event.
`chunk_size` defaults to the measured device ceiling (16,384 B), which is
16x THOR's 1024 and therefore ~14x fewer round trips on a large event.
Pass `THOR_CHUNK_SIZE` to reproduce THOR's wire traffic exactly, or a
smaller value on a link where big responses are not surviving.
"""
return self._proto.read_event_file(ref.key, ref.size)
kw = {} if chunk_size is None else {"chunk_size": chunk_size}
return self._proto.read_event_file(ref.key, ref.size, **kw)
def get_event(self, ref: MicromateEventRef, *, verify: bool = True,
tolerance: float = 0.01):
tolerance: float = 0.01, chunk_size: Optional[int] = None):
"""Download and decode one event.
Returns the codec's `IdfReadResult`. Needs `ref.record_type`, since
@@ -510,7 +517,7 @@ class MicromateClient:
f"event {ref.key_hex}: record_type is unknown, so the codec "
f"cannot be dispatched. Use list_events(with_records=True)."
)
blob = self.download_event(ref)
blob = self.download_event(ref, chunk_size=chunk_size)
import tempfile
from .idf_file import read_idf_file
micromate/protocol.py
+54 -18
View File
@@ -106,8 +106,25 @@ OBSERVED_DATA_LEN = {
# bulk stream; here THOR sends it on every chain read, browse or download.
EVENT_TOKEN = 0xFE
# `SUB 0x5A` chunk size, in bytes of file payload per response.
CHUNK_SIZE = 1024
# `SUB 0x5A` request size, in bytes of file payload per response.
#
# ⚠ **16,384, not THOR's 1024.** Measured on UM20147 (2026-10-02): requests of
# 1024 / 2048 / 4096 / 8192 / 16384 all returned byte-identical data against a
# known-good download, and anything larger is **silently clamped to 16,384** —
# correct bytes, short length, no error.
#
# This matters because round trips dominate a cellular download at ~0.65 s each
# regardless of payload. A 72,560-byte event is 71 requests (~46 s) at THOR's
# size and **5 requests (~3.2 s)** at this one.
#
# ⚠ Measured on ONE unit, over USB. The loop below is driven by bytes received
# rather than chunk index, so a unit that clamps lower simply takes more
# requests instead of failing — which is what makes raising this safe.
CHUNK_SIZE = 16384
# THOR's value. Pass `chunk_size=THOR_CHUNK_SIZE` to reproduce its wire traffic
# exactly; the replay test does.
THOR_CHUNK_SIZE = 1024
# Every `0x5A` response prefixes the file bytes with 11 bytes of header.
_CHUNK_PREFIX = 11
@@ -373,14 +390,15 @@ class MicromateProtocol:
# ── Bulk download ─────────────────────────────────────────────────────────
def read_event_file(self, key4: bytes, size: int) -> bytes:
def read_event_file(self, key4: bytes, size: int, *,
chunk_size: int = CHUNK_SIZE) -> bytes:
"""`0x5A` → `0xA5`. The `.IDFW`/`.IDFH` file, byte for byte.
`size` is the 4 bytes after the key in the `1E`/`1F` response. Returns
exactly that many bytes, or raises `ShortRead`.
A bounded chunk walk — `ceil(size / 1024)` requests, each asking for
`min(1024, remaining)` bytes:
A bounded walk — `ceil(size / chunk_size)` requests, each asking for
`min(chunk_size, remaining)` bytes:
offset = the byte count wanted (NOT an address)
params = the key on chunk 0, then a uint16 BE byte offset
@@ -401,36 +419,54 @@ class MicromateProtocol:
if size <= 0:
raise ValueError(f"size must be positive, got {size}")
if chunk_size < 1:
raise ValueError(f"chunk_size must be positive, got {chunk_size}")
# ⚠ Driven by BYTES RECEIVED, not by chunk index.
#
# The device silently clamps an over-large request: ask for 32,768 and it
# returns exactly 16,384 — correct bytes, short length, no error. A loop
# that strides by a fixed chunk size would either fail on that or, worse,
# skip the bytes it never collected. Tracking the offset by what actually
# arrived makes the clamp a non-event: it just takes another request.
#
# That also makes this self-correcting against any short response, which
# is the failure mode this codebase has been bitten by repeatedly on the
# Series III side — there, a short read surfaced as a silently truncated
# channel.
out = bytearray()
n_chunks = math.ceil(size / CHUNK_SIZE)
for i in range(n_chunks):
want = min(CHUNK_SIZE, size - i * CHUNK_SIZE)
requests = 0
while len(out) < size:
want = min(chunk_size, size - len(out))
data = self._read(
SUB_BULK_DOWNLOAD,
offset=want,
params=chunk_params(key4, i * CHUNK_SIZE),
params=chunk_params(key4, len(out)),
)
requests += 1
if len(data) < _CHUNK_PREFIX:
raise ShortRead(
f"chunk {i + 1}/{n_chunks} of {key4.hex()}: "
f"{len(data)} B is too short to hold a chunk header"
f"{key4.hex()} at offset {len(out)}: {len(data)} B is too "
f"short to hold a chunk header"
)
body = data[_CHUNK_PREFIX:]
if len(body) != want:
# Worth being loud: a silently short event is the failure mode
# this project has been bitten by repeatedly on the Series III
# side, and here the expected length is known up front.
if not body:
# No progress at all — continuing would spin forever.
raise ShortRead(
f"chunk {i + 1}/{n_chunks} of {key4.hex()}: asked for "
f"{want} B, got {len(body)}"
f"{key4.hex()} at offset {len(out)}: asked for {want} B and "
f"got none; {len(out)} of {size} B assembled"
)
if len(body) < want:
log.debug("%s: asked %d B at offset %d, served %d — clamped",
key4.hex(), want, len(out), len(body))
out += body
if len(out) != size:
raise ShortRead(
f"{key4.hex()}: assembled {len(out)} B, device promised {size}"
)
log.debug("downloaded %s: %d B in %d chunks", key4.hex(), len(out), n_chunks)
log.debug("downloaded %s: %d B in %d request(s)",
key4.hex(), len(out), requests)
return bytes(out)
# ── Plumbing ──────────────────────────────────────────────────────────────