perf(micromate): 16 KB per request, not 1024 -- 14x fewer round trips

The chunk-size ceiling, measured on UM20147 with every size checked
byte-for-byte against a known-good download:

  1024 / 2048 / 4096 / 8192 / 16384  ->  served in full
  32768 / 65535                      ->  SILENTLY CLAMPED to 16384

The clamp is the important part: a 32,768 B request returns 16,384 B of perfectly
good data and no error.  Nothing in the response says it was truncated; the
length is the only signal.

THAT DICTATES HOW THE LOOP MUST BE WRITTEN.  read_event_file() now tracks its
offset by BYTES RECEIVED rather than striding by chunk index.  A fixed stride
would either fail on a clamp or skip the bytes it never collected; tracking what
actually arrived makes a clamp cost one extra request, and makes the loop
self-correcting against any short response -- precisely the failure mode that has
bitten the Series III side repeatedly.

Consequence of that change, recorded because it reverses an earlier decision: a
short chunk is NO LONGER AN ERROR.  It used to raise, on the principle that a
silently short event is this codebase's recurring bug.  But the device returns
short legitimately, and the real protection is the offset arithmetic plus the
final total-length check -- which still raises on a genuinely truncated event.
The test was rewritten rather than deleted, and says why.

CHUNK_SIZE = 16384.  For UM20147's events: 4,796 B goes 5 requests -> 1,
30,230 B goes 30 -> 2, and 72,560 B goes 71 -> 5.  Over cellular at ~0.65 s per
round trip that is ~46 s -> ~3.2 s on the large one.

Measured on one unit over USB, so THOR_CHUNK_SIZE = 1024 stays available and the
replay tests pin it -- reproducing THOR's exact traffic is one argument away, and
client.download_event()/get_event() take chunk_size for a link where large
responses are not surviving.

Full suite: 472 passed, 16 pre-existing failures unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
2026-10-02 14:14:02 -04:00
co-authored by Claude Opus 5
parent 248ddc9cda
commit 423608ccd1
5 changed files with 194 additions and 46 deletions
+11 -4
View File
@@ -474,16 +474,23 @@ class MicromateClient:
ref.peak_vector_sum_ips = struct.unpack(
">f", c[tran - _REC_PVS_BACK: tran - _REC_PVS_BACK + 4])[0]
def download_event(self, ref: MicromateEventRef) -> bytes:
def download_event(self, ref: MicromateEventRef, *,
chunk_size: Optional[int] = None) -> bytes:
"""The raw `.IDFW`/`.IDFH` bytes, exactly as THOR would have stored them.
Feeds `micromate.idf_file.read_idf_file()` and `/db/import/idf_file`
unchanged — no new codec work is needed for a directly downloaded event.
`chunk_size` defaults to the measured device ceiling (16,384 B), which is
16x THOR's 1024 and therefore ~14x fewer round trips on a large event.
Pass `THOR_CHUNK_SIZE` to reproduce THOR's wire traffic exactly, or a
smaller value on a link where big responses are not surviving.
"""
return self._proto.read_event_file(ref.key, ref.size)
kw = {} if chunk_size is None else {"chunk_size": chunk_size}
return self._proto.read_event_file(ref.key, ref.size, **kw)
def get_event(self, ref: MicromateEventRef, *, verify: bool = True,
tolerance: float = 0.01):
tolerance: float = 0.01, chunk_size: Optional[int] = None):
"""Download and decode one event.
Returns the codec's `IdfReadResult`. Needs `ref.record_type`, since
@@ -510,7 +517,7 @@ class MicromateClient:
f"event {ref.key_hex}: record_type is unknown, so the codec "
f"cannot be dispatched. Use list_events(with_records=True)."
)
blob = self.download_event(ref)
blob = self.download_event(ref, chunk_size=chunk_size)
import tempfile
from .idf_file import read_idf_file