Step 3 of docs/micromate_client_spec.md: micromate/client.py, two models in
micromate/models.py, 26 offline tests. Every response constant in the tests
is a real captured data section from UM12947.
Field offsets were measured rather than taken from the spec, which turned up
one general rule and one trap:
THE RESPONSE SHAPE. Every response carries an 11-byte prefix and content
starts at data[11]. One rule, every command.
THE TRAP: data[0] is the content length & 0xFF, with no high byte anywhere in
the prefix. It is therefore correct for every response under 256 bytes --
most of them -- and then reports 44 for a 2,092-byte setup block, 30 for a
286-byte monitor-log record, and 0 for a 1,024-byte download chunk. 182 of
251 captured responses agree with a naive read; the 69 that disagree are
exactly the ones >= 256 bytes.
That is the third length in this protocol read too narrow, after
payload[9]-vs-payload[8:10] in the probe response. The client takes content
as data[11:] and lets the frame's own length bound it -- nothing needs the
declared length, since the frame already knows how long it is.
Also measured:
- POLL content[3] is 0x50, printable as "P", immediately before "Instantel".
A generic printable-run scan therefore returns "PInstantel" -- it caught a
test, not a unit. Vendor comes from a fixed offset; the model is found by
searching for "MM/", which is structural rather than positional and so
survives the Thor line's shorter "MM/ISEE/S".
- The setup walk terminates on an EMPTY NAME, not an error: 23 responses, 22
names, factory.MMB first through TEST1.mmb last.
- The 0x1C clock has an unidentified byte at content[6]; the hour is at
content[7]. The protocol reference's 0x1C section already had this right
and names the byte -- its one-line summary in the divergences list reads as
six contiguous fields and is the version not to trust. Re-verified against
three captures: 19:12:25, 19:13:34 and 01:14:05 against filenames stamped
19:12:14, 19:12:14 and 01:14:03.
- Battery and memory are read FORWARD from content start, never backward from
the end. This block is 4 bytes longer on the Thor line; the Series III
from-the-end offsets give a 11.0BD unit 577.92 V. A test appends the four
trailing bytes and asserts the forward offsets survive.
connect() is narrower than the spec asked. The spec said to mirror Thor's
POLL -> SERIAL -> 0x49 -> POLL "because it is known-good"; measurement showed
that is Thor's connection check (3 of 8 sessions) and its fourth frame repeats
its first. So connect() sends the three reads that gather something, and
0x01 is not read at all -- Thor never reads it, its layout is unmapped, and
firmware_line comes free from any response's flags byte. If a unit ever
refuses the next command after a cold connect, put the fourth POLL back and
record it.
A dead clock battery yields device_time=None rather than failing the whole
state read; an unreadable active setup yields active_setup=None rather than
failing connect. Both are real device states.
Still verified only against 11.0CB and only over USB. The BD offsets follow
from the extra bytes being trailing, which is documented but not something
this code has seen.
Full suite unchanged at 16 pre-existing failures; 445 passed, up 26.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
Step 2 of docs/micromate_client_spec.md: micromate/protocol.py plus 35
offline tests. Reads only; nothing here writes, erases or changes
monitoring state.
The tests replay Thor's captured responses through a scripted transport and
assert the bytes we emit are the bytes Thor emits -- including a full replay
of the six-event download session, all 56 0x5A frames byte-for-byte. A
passing test therefore means a real unit has already answered exactly that
frame.
Measuring the spec's command table against the captures found three more
errors in it, on top of the three the framing work found:
1. SUB 0x0A is the MONITOR-LOG WALK, not a keyed "event header, 30 B list
record" read. The same request repeated returns successive 297-byte
records -- serial, mode, thresholds -- until an 11-byte ack ends the
list. The device holds the cursor; nothing in the request selects a
record, and all nine captured frames carry identical params.
Structural divergence worth noting: Series III reaches the same data via
a record-type discriminator on its event chain, so partials and events
share one walk. Here the monitor log has its own cursor and the event
chain never sees it.
2. 0x1E/0x1F carry token 0xFE at params[7]. The protocol reference
documents all-zero params -- that was our own browse probing, which also
worked. Thor sends 0xFE on browse and download alike.
3. SUB 0x01 (device info) is never read by Thor in any captured session.
Its 0xFFFF offset comes from our own probes, so it is the one read in
the table with no Thor precedent. Flagged in the docstring.
Two useful negatives, both from absence rather than presence:
- No SESSION_RESET (41 03). Series III needs that 2-byte signal or a
monitoring unit will not answer POLL over TCP. Zero occurrences across
all 8 sessions, including 40 frames exchanged with a unit that WAS
monitoring.
- No universal preamble. The only invariant is that a session opens with
POLL; POLL -> SERIAL -> 0x49 -> POLL is Thor's connection check and
appears in 3 of 8 sessions. Setup pushes and scheduler reads open
differently.
Two deliberate divergences from the Series III sibling:
- strict_checksums defaults True and RAISES. minimateplus logs and
continues because its parser cannot always tell an inner-frame delimiter
from a checksum byte; that does not apply here, where the rule is exact
on 251/251 frames. The lenient default is instructive -- it hid a wrong
checksum rule for two days.
- read_event_file() raises ShortRead rather than returning a truncated
event. The expected length is known up front, so the check is free, and
a silently short event is the failure mode this codebase keeps hitting.
Every exchange resets the parser before sending, so a leftover frame is
discarded rather than answered with -- expected_sub catches a mismatched
SUB, but a same-SUB leftover would sail through with data for the wrong key.
File transfer (0x94/0x48) is deliberately out of scope: it needs a
data-carrying request frame, which is the frame type writes use, and that
boundary is worth keeping crisp in a read-only pass.
Full suite unchanged at 16 pre-existing failures (missing gitignored
fixtures); 419 passed, up 35.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
Step 1 of docs/micromate_client_spec.md: micromate/framing.py plus 31
offline tests. Every rule was checked against the captures BEFORE being
written, which is the only reason this commit is not a bug.
Three things the spec asserted are wrong, all of which fail silently:
1. Requests are NOT plain Series III frames. A Micromate escapes four
byte values -- 0x02 0x03 0x04 0x10 -- where Series III escapes one.
minimateplus.build_bw_frame reproduces 161 of Thor's 218 captured read
frames; build_request() reproduces 218/218. The 57 it missed include
EVERY 0x5A download (offset 0x0400 puts a literal 0x04 in offset_hi)
and the scheduler enable. An unescaped 0x03/0x04 terminates the frame,
so the unit never answers -- indistinguishable from a dead unit, and
event download would have hit it on the first request ever sent.
2. The checksum is plain SUM8 of the destuffed payload, not the DLE-aware
variant. 251/251 both directions. The DLE-aware form is correct
paired with Series III destuffing, which leaves an escaped byte as two
bytes; after uniform destuffing it subtracts the correction twice and
disagrees with the wire on 55 of 251 responses.
scratch/mm_frame_parse.py shipped with exactly that pairing and looked
clean only because it accepts either rule -- so it labelled those 55
"SUM8" and never flagged one bad. "Zero bad checksums" was true and
carried no information. A tool that tries N candidate rules cannot
falsify any of them. Fixed to validate against SUM8 alone.
3. SUB 0x5A is a 1024-byte chunk loop, not one request per event. Thor's
form, verified on all six bench events (4,076 -> 13,424 B): chunks =
ceil(size/1024), offset = min(1024, size - 1024*i) as a byte count,
params[2:4] = the byte offset, response data = offset + 11.
sum(offsets) == size exactly, every time.
This does not retract the earlier single-request observation -- that
used offset_hi = 0x10, which in Series III is the bulk-stream marker,
so it is plausibly a distinct streaming mode returning several frames.
Those captures never landed in the repo, so it cannot be re-derived.
Implement Thor's form; the other is worth one bench test.
Also: a 0x10 inside request params needs no special handling (settled --
Thor sends it, the wire doubles it), so the planned NotImplementedError
guard is gone. declared_length -> probe_length, because it is only
meaningful in a probe reply and Thor never probes.
Synthesised test frames are marked and each says what it stands in for.
The flags=0x03 case is the only coverage of the Thor firmware line -- it
wants a real 11.0BD capture next time UM20147 is on a bench.
No writes. Read-path framing only; nothing here can originate a command.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
Follow-up to the ts1→ts2 fix: get the trigger to the second from the binary
alone, instead of falling back to the stop time (~record-duration late) for
no-report events.
The configured post-trigger record time is a big-endian float32 in the
recording-setup config block, exactly 30 bytes before the "Standard Recording
Setup" marker. _parse_record_time_seconds reads it; the waveform branch now
stamps trigger = ts2 - record_time. Verified: the field reads 1.0 / 2.0 / 3.0 s
across different setups in the corpus, and all 7 BE12844 oracle events now
decode to their exact Blastware trigger (N844LQHB 10:33:29) from the binary,
no paired .TXT needed. Falls back to ts2 (the stop) if the config block is
absent. A paired report's event_datetime stays authoritative (clock drift).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
read_blastware_file built ev.timestamp from footer ts1, which for a WAVEFORM is
the monitoring-session start (a unit arming at 06:00 stamps 06:00 on every event
that day) — so every waveform's time was hours off (vomit-list #3, "~4.5 h off").
The event time is footer ts2 (the recording stop); BW's displayed Date/Time is
the trigger = ts2 - record duration.
Root cause proven against the BE12844 oracle set: 5 of 7 events decoded to the
identical 06:00:13 (the shared session start); ts2 gives distinct plausible
event times (N844LQHB ts2 = 10:33:32, BW trigger 10:33:29 = ts2 - 3.0 s rectime).
* read_blastware_file now uses ts2 for waveforms (discriminated by which codec
decoded the body, not the filename — save_imported_bw passes a tmp name).
Histograms keep ts1 (the ~24 h window start, which IS the event time).
* Binary-only decode can't get the exact trigger: the STRT record-time byte is
a misparsed record-type marker (0x46=70), so ts2 (the stop, ~record duration
after the trigger) is the best estimate. A paired BW report carries the exact
trigger — apply_report_to_event now overlays event.timestamp from
report.event_datetime, matching the existing build-path override (line ~441).
Tests: waveform → ts2, histogram → ts1 unchanged, report → exact trigger.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
Make the sensor self-check a first-class part of the standardized decoded event
so SFM stops decoding it at report time — device-agnostic, per the store's
decoder→standardized-.h5→SFM model.
* Event gains a `sensor_check` field; both decoders attach the traces where
they set raw_samples — series-3 in event_file_io.read_blastware_file
(minimateplus.sensor_check), series-4 in waveform_store's IDF path
(micromate.sensor_check). Covers ingest and backfill (both re-decode).
* event_hdf5 bumps schema_version 1→2 and writes an optional /sensor_check
group (raw counts, int32, per channel present). read_event_hdf5 returns
it; plot_json_from_hdf5 carries it as a top-level key. Old v1 files still
read cleanly (no group → None), so nothing breaks before the backfill.
* gather_report_data reads sensor_check_waveforms from the .h5 and drops the
report-time series-3 decode — the report no longer reaches into a decoder,
and a series-4 event now lights up the same strip automatically.
Stored as raw counts (a shape diagnostic, rendered fit-to-box): the per-series
count scale differs and a physical mic unit is ill-defined, so conversion would
add complexity for no display benefit — easy to add later if a numeric use
appears.
Tests: .h5 roundtrip + backward-compat + plot_json + real series-3 decode
attaches to the Event.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
The Thor/Micromate (series-4) IDFW binary carries the sensor self-check in its
fixed-header region (before the waveform body), as up to four records tagged
01 0e 3c/3d/3e/3f — the SAME channel ids as series-3 (Tran/Vert/Long/MicL).
Unlike series-3's delta-coded trailing block, series-4 stores each trace as a
raw int16-BE array after an 18-byte record header (2-byte sample count at
offset +8). Three-channel (mic-disabled) units carry only 3c/3d/3e.
New micromate/sensor_check.py: decode_idf_sensor_check(raw) locates the record
chain (id-ordered marker run, so a stray body match can't chain) and reads each
trace's int16 samples → {Tran,Vert,Long[,MicL]: [counts]}, or {} when absent.
Reverse-engineered + validated against 4 UM oracle events (added as fixtures):
clean geophone ring-downs on all, mic pulse trains on the 4-channel units,
correctly no MicL on the two 3-channel units. Validated by shape + cross-event
consistency (no Thor report strip to exact-match, unlike series-3's BW reports).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
The Blastware Event Report draws a "Sensor Check" strip on the right of the
waveform panel — the little traces the unit records when it pulses each sensor
before monitoring. Those live in the series-3 binary's trailing block, after
the main waveform record-chain and the per-channel calibration records, as four
length-prefixed records tagged 0x3c-0x3f (Tran/Vert/Long geophone ring-downs +
MicL pulse train). Reverse-engineered against 7 BE12844 oracle events.
New minimateplus/sensor_check.py: decode_sensor_check(raw) locates the record
chain (validated by walking the ids 0x3c->0x3f via their length prefixes) and
decodes each record's delta stream (payload[20:len-8]) with the same 10/20/30/00
delta-block tags as the main waveform codec, from an anchor of 0. Returns
{Tran,Vert,Long,MicL: [samples]} in raw 16-count units, or {} when absent.
Validated: mic pulse-train zero-crossing frequency = 20.1 Hz (exact match to
BW's mic Channel Test freq); geophone ring-downs are consistent ~-990 raw
deflections that damp to a ~-310 settle across all 7 events (a fixed
calibration pulse, so near-identical every run).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
sfm/compliance.py renders the velocity-vs-frequency blasting compliance chart
Blastware draws on its Event Report:
- limit_at()/limit_curve() — the RI8507 Fig B-1 / 30 CFR 816.67 curve as data
(Drywall 0.75 + plaster 0.50 lines): 0.030in low-freq bound, plateau, 0.008in
rising diagonal to a 2.0 in/s cap at ~40 Hz, drawn continuous.
- channel_compliance_points() — the per-cycle (freq, peak-velocity) scatter by
the zero-crossing method (matches Blastware; cloud ceiling = channel PPV).
- draw_compliance_chart() — matplotlib rendering (both lines + scatter, BW tick
scales + channel markers).
Verified against 7 BE12844 Blastware reports. docs/ri8507_compliance_curve.md
captures the curve construction, the SHM basis, and the scatter method.
Not yet wired into report_pdf.py — that placeholder is the next step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
channel_spectrum(samples, sps) → the single-sided amplitude spectrum Blastware's
FFT Report draws, and dominant_frequency() picks its peak in the 2–250 Hz band.
Reverse-engineered against 7 BE12844 (MiniMate Plus) events with Blastware FFT
reports as ground truth. Recipe: DC-remove, NO window (a window smears the peak
and worsens the match), zero-pad to 4096 (→ 0.25 Hz bins at 1024 sps — the
resolution every reported dominant frequency lands on), single-sided 2/N
amplitude. Reproduces Blastware's dominant frequency to the exact bin on all
28 channels and the amplitude to report precision.
This is the missing piece for both the USBM RI8507 compliance chart (its scatter
is these (freq, amp) points vs the limit curve) and the FFT view.
Pure numpy, series-agnostic (feed it in/s samples from either decoder). The 7
events land in tests/fixtures as the oracle (force-added past the fixtures
gitignore, matching 5-11-26 / decode-re-5-8-26).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
annotate_blastware_binary(raw) → a gap-free tiling of labelled Spans
(header / STRT / per-channel sample records / footer / unknown) for a hex
viewer to paint. Every byte is covered; anything the decoder can't account
for is a first-class `unknown` span, so undecoded regions stand out.
Composes the existing waveform_codec.walk_records over the body between the
STRT record and the 26-byte footer. On the cracking fixtures this already
surfaces a ~1700-byte undecoded trailing region (stream-end marker + serial +
…) per file — a candidate home for stored spectral/FFT data.
TDD: tests assert the spans tile the whole file, STRT is located, the geo
sample records are labelled, and the footer is last.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
data_block_len() rejected any `40 NN` block with NN > 0x08. That guard had
no evidence behind it: every corpus available when it was written used only
NN in {1,2,3,4,8}, so it was never exercised. Loud UM12947 events use NN of
12, 16, 20 ... up to 196.
Because walk_body/run stop at the first unrecognised tag rather than
raising, rejecting those blocks surfaced as silently short channels -- e.g.
Tran 1812 / Vert 2132 / Long 2324 on a file whose export carries 2324 for
all three. The real bound is the buffer; the caller additionally clamps to
the record end.
Verified against Thor's own CSV exports for UM12947 (2025-07-14 .. 09-25,
167 waveforms, supplied as CSV.zip):
length mismatches 22 -> 0
per-sample exact 1,476,242 / 1,476,249
These are NOT truncated recordings, which was the competing hypothesis --
the exports carry the full sample count.
tests/test_waveform_codec.py asserted the cap as intended behaviour. That
assertion encoded an assumption, not a verified fact, and is replaced with
one pinning the opposite plus the evidence.
Across all three ground-truth corpora: 459 waveform files,
3,807,158 / 3,807,165 samples exact. Production IDFW is now 575/575 with
zero truncations and zero decode failures (median PPV error -0.0007% across
8 units). Series-3 re-verified unchanged at 14,338/14,338.
The 7 residual samples each differ by one 4th-decimal tick and are Thor's
own rounding: intersecting the per-sample rounding constraints over that
corpus is infeasible (binding pair contradict by 2.3e-11, 7e-5 relative),
so no single linear LSB reproduces every printed value. _GEO_LSB_IPS is
already pinned to ~1e-11; do not retune it to chase these.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
Verified against a second Thor corpus (9-10-26-csv-req: UM11402, UM12947,
UM20147) with per-sample CSV exports: 139/139 waveforms exact
(1,273,380/1,273,380 samples) and 877/877 histograms within 2% of Thor's
reported PPV -- up from 66.9% and 56.6%.
Some units run with the microphone disabled, which changes two structural
things that were both hardcoded to the 4-channel shape:
- Waveform body head sat below the scan floor. A 3-channel unit has a
shorter fixed header and puts its record chain head at 0x0dba, under the
old _BODY_SCAN_FLOOR of 0x0E00. The scan could not see it and fell
through to the Vert segment-0 record, decoding a body shifted one
position around the channel rotation -- Vert came up exactly 512 samples
short. Floor lowered to 0x0C00. The body-offset scoring also had to stop
requiring four channels, or `equal` is permanently False for these events
and the pick falls back to raw sample count.
- Histogram interval record is 56 bytes, not 72. It is
16 * n_channels + 8, and is not inferable from the segment length alone.
The interval count now comes from the segment's cumulative counter
(n = counter - prev_counter) and the stride is derived from it. Assuming
72 read 7 intervals out of every 10-interval segment, then walked off
alignment into garbage that decoded as ~10 in/s peaks -- inflating some
files' PPV by up to 191,000%. Also recovers 4 files that previously
decoded no intervals at all.
Combined across both corpora: 292/292 waveform files,
2,330,916/2,330,916 samples exact. Production IDFW truncations 41 -> 22.
Series-3 unaffected (no shared-codec change in this commit; last full run
14,338/14,338).
Known open, diagnosed but NOT verified: the remaining 22 unequal + 1 failing
production IDFW files (all UM12947, 2025-07-14..09-23) stop the block walker
on tag 40 0c. data_block_len() caps the 40 NN int16 block at NN > 0x08 while
those files use NN up to 196. Both verified corpora only ever use
NN in {1,2,3,4,8}, so the cap is untested there and lifting it leaves both at
100.000% -- which is not evidence it decodes these correctly. Deliberately
not shipped; needs Thor CSV exports for UM12947 in that date range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
Verified against Thor's own CSV exports, which carry a per-sample
four-column block beside every binary (CSV/<name>.IDFW.csv). Those 1,012
paired files were in the corpus all along; the decoder had been pinned to
a superseded walker on the stated grounds that "Thor has no ASCII ground
truth in the corpus and its geo scaling is separately suspect". Both
premises were false.
IDFW per-sample exact 39.1% -> 100.000% (1,057,536/1,057,536)
IDFW files fully exact 0/153 -> 153/153
IDFW PPV median error -3.32% -> -0.002%
IDFH within 2% of Thor PPV 51.1% -> 100.0% (858/858)
prod IDFW, 8 units -3.3% -> -0.001%
Four independent root causes:
- Geo LSB was 0.0003, the 4-dp *display rounding* of the real
0.000310308 mistaken for the LSB, so every series-4 geophone sample
read 3.3% low. Pinned to +-6e-11 by intersecting 991,415 rounding
constraints; corroborated by the +-full-scale seed (+-32226) left in
unwritten IDFH slots. IDFH had a separate, also wrong, 10.0/32768.
- IDFH histograms were capped at 250 intervals: the segment validator
required the interval counter's high byte to be zero, but the counter
is a uint16 cumulative index, so every segment past interval 255 was
rejected. Runs over ~4 hours lost their tail, often the peak.
540/858 corpus files affected.
- Record mode 00 00 (raw int16, 10-byte header) was unhandled and fell
through the dispatch, silently dropping each channel's first 512
samples -- the long-standing "loud events truncate" symptom.
MODE_ABSOLUTE is now also accepted as a segment-0 preamble.
- The body-offset search matched 00 02 00 *inside* record headers,
selecting a candidate part-way down the chain and decoding a
rotation-shifted body. It now anchors on record headers and takes the
chain head (6 ms/file).
Also fixes the separately tracked "UM-series decodes ~1000x low" bug.
Series-3 re-verified unchanged at 14,338/14,338 exact after the shared
waveform_codec change.
Known open: 41/575 prod IDFW files (7%, mostly UM12947/UM20147) decode
with unequal channel lengths and also fail metadata extraction -- a
different header variant with no Thor export in the store.
NOTE: this is a codec change; the Thor store owes a regeneration via
scripts/backfill_thor_events.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
The event-report waveform plot scaled each geo lane to its own peak, so a small
channel filled its lane looking as big as a large one — and the "Geo: X in/s/div"
footer only reflected whichever channel was checked first, so its div value was
wrong for the other two. Now all three geo lanes share ONE symmetric scale =
max |sample| across them (padded, 0.05 in/s floor), matching the event modal and
BW's single amp/div; the footer reflects that shared scale. Mic keeps its own psi
scale. Big events are unchanged (e.g. BE12844 stays 0.185 in/s/div).
Test-first: tests/test_report_pdf_geo_scale.py (shared scale + floor), 2 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
The BW filename encodes only the serial NUMBER — `<letter><3 digits>`, so
`L895…` is 10895 and nothing more. The two-letter family prefix is not in it:
"BE" is a MiniMate Plus, "BA" a BlastMate. Both are Series III and their
files are byte-identical in every way that matters — all 1,493 BlastMate
binaries in the DL2 archive decode through the existing codec at 100%, same
four channels — so the serial string was the only thing standing between SFM
and BlastMate support.
Two sites synthesised the prefix and got it wrong:
- waveform_store `_serial_from_bw_filename` returned f"BE{num}" on import, so
a BlastMate event was filed under a unit that does not exist, silently, and
Terra-View read it straight through. Split into
`_serial_number_from_bw_filename` (the number, which the filename really
does carry) and a new `_serial_from_bw_bytes` that reads the serial out of
the body and accepts it only when its numeric part agrees with the
filename. save_imported_bw now prefers hint -> body -> filename guess.
Verified against real archive bytes for BA9229, BA10060, BA10895, BA15957
and BE9558/BE11529/BE18003.
- client `_decode_0a_partial_header` searched for a literal b"BE" in the
monitor-log partial record. On a BlastMate that returns -1 and skips the
whole block, so the geo threshold went missing along with the serial. Now
matches any two-letter prefix, and requires the NUL terminator — stricter
than the bare two-byte search it replaces.
Nothing to migrate: no BlastMate events are in prod. The archive's BA units
last recorded 2018-10 (BA9229, BA15957), 2023-08 (BA10895) and 2023-11
(BA10060), and the prod backfill only reaches back to ~May 2025.
21 tests. Suite: 309 passed, same 16 pre-existing failures as at HEAD
(15 missing ASCII fixtures + one peak_values assertion, all untouched here).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
A reason records *why* an event is a false trigger. It is optional (plain
FT flags still record no reason) and is a subtype of the FT flag: setting a
reason implies false_trigger=1, and the reason is cleared whenever FT ends
up 0 (confirm-real, clear-FT, set_false_trigger(false)). Twin propagation
carries the reason to the histogram/waveform twin alongside the FT flag.
New nullable `false_trigger_reason TEXT` column (schema + _migrate ADD
COLUMN only — not the Migration-1 rebuild). 7 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
Productionizes the validated scratch/offset_scan3.py: a DC offset (baseline
shifted off zero — sensor bumped/settled/drifted) is |median(pre-trigger)| >= 5
counts (0.025 in/s) AND flat across pre/mid/end thirds (spread <= 0.02); a
transient moves one third and is rejected by the spread test.
- shape_metrics: offset_from_samples / offset_from_h5 (reads .h5 samples +
pretrig_samples attr; range-aware via the .h5's in/s float samples)
- events schema: shape_offset / _axis / _pre / _spread (via _SCHEMA + the
_migrate ADD COLUMN loop only; NOT the Migration-1 rebuild), threaded through
insert + upsert mirroring shape_*
- ingest: computed at all three waveform_store save paths alongside shape
- backfill_event_shape: also computes + stores (and stale-clears) offset
- exposed via /db/events automatically (SELECT *)
Gating to waveforms is done downstream in terra-view ft_suspicion (mirrors how
shape is ignored for histograms), not at the SFM call sites. 13 new tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
terra-view's SFM Admin page (/admin/sfm) displays whatever /health returns for
`version`. That was hardcoded to "0.1.0" and never bumped, so the page showed
0.1.0 while the service was actually 0.26.0. Point both /health and the FastAPI
OpenAPI version at the release-bumped TOOL_VERSION (single source of truth), so
they can't drift again. Adds httpx-free regression tests (call health() directly).
Note: minimateplus.__version__ is separately stale at 0.1.0 — left as-is here
(nothing user-facing reads it; touching the package __init__ risks import order).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
detect_multi_interval_stride() confirmed a candidate stride on a third block
header whenever the body was long enough to contain one. But a body can exceed
two strides and still hold only two real blocks: a partial final block leaves
trailing padding. BE18193 T193L0XM.CI0H — 51 intervals at 2 s, i.e. one full
30-interval block plus a 21-interval remainder in a 2787-byte body — had every
decisive check pass at stride 612 (header at 0, header at 612, block counter
256 -> 257) and was then rejected for the absent third header at 1224. It
decoded to nothing.
A missing third header now means end-of-stream rather than disqualification.
The block-counter check is untouched — that is the test that prevents the
false positives which once handed 9,082 standard-block files to the
multi-interval walker.
Found by running the full DL2 archive against its preserved Blastware ASCII
exports (14,340 paired files, 11x the previous ground-truth corpus).
Measured over 127,035 archive histogram binaries:
recovered 8 files (strides 92, 252, 612; BE18193, BE18191, BE9557, BE9440)
regressed 0 files
Full-corpus verification: 14,337 -> 14,338 exact of 14,338 decodable pairs
(the 2 excluded are series-4 IDF, a different codec).
Also adds scratch/verify_against_ascii.py (per-sample decoder verification
against BW exports, with a saturation carve-out — BW clamps clipped events to
the range max while the decoder reports true counts) and scratch/offset_scan.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
A real trigger is recorded twice — as a triggered waveform (stamped at the
trigger instant) and inside the scheduled histogram whose interval contains it
(stamped at the 7am/7pm interval start). The two twins routinely differ by
HOURS, so the old ±5-minute window in find_twins silently missed them — which
broke review propagation (flagging one twin left its twin unflagged).
Twins are now matched by: same serial + identical peak_vector_sum + OPPOSITE
record type + the waveform's timestamp falling within the histogram's interval
(bounded by the next same-serial histogram). Matching keys off record timestamps
(not call-in/received times, which drift with field connectivity). window_seconds
is retained but ignored.
Rewrote test_find_twins + test_twin_propagation for the new contract (incl. the
75-min-apart UM12947 case, cross-type exclusion, containing-interval selection,
open-ended latest interval). Full suite: 264 passed; the 16 failures are
pre-existing (missing gitignored fixtures + a v0.26.0 codec case), unchanged
from baseline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
Swept every series-3 binary with the live decoder against five
independent checks: decode exceptions, zero samples, unequal geo channel
lengths, peaks above range full scale, decoded peak vs device-reported
PPV, and waveform length vs declared record time.
1. block[22] is NOT a constant and must not be tested. Documented as
always 0x00, it carries data on loud blocks, and rejecting those threw
away the interval holding the event peak. BE18350/T350L7HR.NL0H
block 92 has block[22]=0x26 and a Tran peak of 0x0563 = 1379 counts =
6.895 in/s — exactly the device-reported PPV — while the file decoded
to 0.015 in/s. block[0]==0, block[4]==0x0A and the 4-byte tail are
six bytes of constraint, which is what keeps trailer content out.
2. Block-model dispatch now goes on signature strength rather than on
whichever decoder returns first. A multi-interval body also yields
scattered standard-tail blocks by coincidence, so "first non-empty"
handed 193 BE18193 files to the standard walker and produced peaks of
149 in/s against a 10 in/s full scale.
3. Multi-interval stride detection requires the block counter to
increment by exactly 1. Without it the detector false-positives on
ordinary standard-block bodies: they carry a header every 32 bytes,
and 192 = 12 + 20*9 and 512 = 12 + 20*25 are both multiples of 32, so
a stride "fits" while skipping 6 or 16 real blocks. That misrouted
9,082 files.
Partial-block garbage is trimmed within the final block only, stopping
at the first slot with a non-zero tail word or a geo peak above full
scale (2000 counts in 16-count units). Trimming purely from the end
left garbage stranded behind a slot that happened to have a zero tail
word; trimming on the tail word alone truncated four BE9440 files by up
to 2,800 intervals.
Result: 11,603 / 11,603 series-3 binaries clean on every check.
Ground truth unchanged: 1211/1211 histograms exact per-interval, 75/75
waveform sample counts exact (73/75 fully exact, the 2 differ by 1 LSB
on rail samples), and the multi-interval fixture still matches its BW
ASCII export on all 45,680 values.
Tests: 259 passed, failure list unchanged from baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Sub-minute histogram intervals are packed several to a block so that
every block still covers exactly one minute of data:
interval intervals/block stride
1 minute 1 32 <- the standard big-endian block
15 s 4 92
2 s 30 612
stride = 12 + n * 20
Block = [00][segment][ctr uint16 LE][0a][00], then n x 20-byte records of
8 x uint16 LITTLE-endian values (T_peak, T_halfp, V_peak, V_halfp,
L_peak, L_halfp, M_peak, M_halfp) plus a 2-word tail whose first word is
0000 on every real interval, then a 6-byte block trailer.
The standard 32-byte block is BIG-endian; this variant is LITTLE-endian.
The tail-word check matters: a session ending mid-block leaves buffer
garbage in the remaining interval slots, which decoded as peaks
thousands of times the real value. Stride detection also requires at
least 2 records, since a 1-record block would have stride 32 and
collide with the standard block.
Recovers 415 files that decoded to nothing: 216 on BE18193 (2 s
intervals) and 199 on BE9440 (15 s). Before decoding to nothing they
were being accepted by the WAVEFORM codec, which returned garbage
peaking up to 400x the device-reported PPV.
Ground truth BE9440/K440L3AQ.T70H (5,710 intervals) matches its
Blastware ASCII export exactly: 17,130/17,130 geo peaks, 22,840/22,840
frequencies, 5,710/5,710 mic dB(L). Across all 455 affected files,
1,354/1,365 channel peaks (99.2%) match the device-reported PPV; the 11
that don't are under-reads on BE9440 where the walk stops early.
Fixture (binary + ASCII) saved under tests/fixtures/, which is
gitignored per repo practice — the ground-truth test skips when absent.
Tests: 258 passed, failure list unchanged from baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Supersedes the segment-header model entirely, including the fixes made
earlier today. Found via multi-agent structural analysis of the 25 files
that stalled the walker, then verified independently.
Records are self-delimiting: off+2 is a uint16 BE length, next_record =
off + 2 + len, and the chain ends on a record whose chan_id is 0x06.
off+8 carries a 3-valued mode enum:
02 00 14-byte header, 2 anchors, then CUMULATIVE delta blocks
01 00 10-byte header, no anchors, blocks are ABSOLUTE values
00 03 10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute
`40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header.
Reading it as a 2*NN + 16 header is what made walks drift — the
"variable-prefix segment descriptors" reported earlier today were not a
format feature, just walker drift of exactly
4 - (old_stop - true_record_start), on all 25 affected files.
Measured on the production snapshot:
all four channels equal length 156/1388 -> 1388/1388
ASCII sample-count exact 72/75 -> 75/75
ASCII fully exact 70/75 -> 73/75
device PPV waveform (live) 1288/1306 -> 1306/1306 (mean err 0.00000)
device PPV histogram (live) 4434/4459 -> 4458/4459
Also eliminates the walker-over-read class: 24 of those 35 files were
histograms that read_blastware_file fed to the waveform codec first; the
old walker accepted them and returned garbage (one yielded 98,923
"intervals"), while the record-chain decoder returns None so they fall
through to histogram_codec.
00 03 records are DECODED, not skipped. Skipping them silently shifts
the time base of everything after them on that channel — BE9558/
K558LOF2.820W had MicL displaced by exactly 512 samples with nothing
marking the gap.
Footer detection now prefers the 0e 08 candidate whose body yields a
chain terminating on 0x06; the signature can occur inside a sample
stream. Blast radius 1 file of 1388.
The superseded model survives as decode_waveform_legacy, pinned by
micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes
candidates and keeps whichever yields the most samples, so the new
decoder returning None where the old returned garbage changes that
heuristic's winner. Deferred until that search uses the record chain.
Tests: 253 passed (+11), failure list unchanged from baseline. The 9
tests pinning the superseded model are retargeted at
decode_waveform_legacy, which still implements it.
NOTE: stored .h5 files need regenerating — nearly all get longer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Two errors in the series-3 histogram block model, both found by diffing
against the per-interval data table in the preserved Blastware ASCII
exports (1211 files in the prod snapshot — far stronger ground truth
than the header PPV used previously).
1. The block is uniformly BIG-ENDIAN. Peaks and half-periods are uint16
BE (T_peak [5:7], T_halfperiod [7:9], V_peak [9:11], V_halfperiod
[11:13], L_peak [13:15], L_halfperiod [15:17], M_peak [17:19],
M_halfperiod [19:21]); only block_ctr [2:4] is little-endian.
The old uint8-peak model silently CLIPPED any peak above 1.275 in/s:
the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's
export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590.
The byte documented as a per-channel "annotation" was never an
annotation — it is the half-period's high byte, which is exactly why
it was non-zero on the sub-Hz intervals BW renders as "<1.0".
The marker is block[4] alone. Testing [4:6] as a uint16 LE marker
forced block[5] == 0, which is what capped the peak at one byte.
2. The final block of each stream carries tail 9c 06 00 42 instead of
1e 0a 00 00, and holds arbitrary bytes at [21:23]. Rejecting it
dropped the last interval of nearly every histogram — frequently the
interval holding the event peak, so the file's PPV read low.
Verified end to end through the production path: 1211/1211 histograms
decode exactly (interval count + every per-interval peak), plus 842,442
per-interval frequency comparisons with zero mismatches. Previously
1 of 1196 files was fully correct.
decode_histogram_body_full records expose `is_terminal` in place of the
removed `annotations` tuple. +6 tests. No regressions: full-suite
failure list unchanged from baseline.
NOTE: stored histogram .h5 files need regenerating to pick this up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Two independent bugs, both found by diffing 75 production events against
their preserved Blastware ASCII exports (<store>/<serial>/<file>_ASCII.TXT).
1. Geo full scale was wrong — every geophone reading was 2.34% low.
The codec emits geo samples in 16-count units with a documented LSB of
exactly 0.005 in/s, and decoded_to_adc_counts multiplies by 16, so one
ADC count is 0.005/16 in/s and 10.000 in/s is 10.0/(0.005/16) = 32000
counts. sfm/event_hdf5.py and minimateplus/event_file_io.py both
divided by 32768 (2^15), scaling every sample and derived peak down by
1 - 32000/32768. The error scales with amplitude, so it was invisible
on quiet events and worst on the loud ones that matter for compliance.
Mic is unaffected (it back-solves its scale from the device peak).
216 per-channel comparisons: 32768 -> 151/216 exact, worst error 0.238
in/s on a 10 in/s event; 32000 -> 216/216 exact, worst 0.005 = 1 LSB.
2. walk_body silently truncated channels on four unhandled framing cases.
An unrecognised tag ends the walk and decode_waveform_v2 returns
whatever it got, so this surfaced as short channels, never an error:
- wide-NN RLE `0X NN` (runs longer than 252 samples)
- `30 NN` with NN > 0x10 (the old cap was arbitrary)
- variable-width `40 NN` headers: NN counts previous-channel
continuation deltas, so the header is 2*NN + 16 bytes; `40 01`
and `40 03` occur alongside `40 02`
- tagless segment headers: no `40 NN` tag at all, just the 14-byte
tail [field2:2][len:2][channel_id:4][marker:2][anchors:4]
Also: the header field documented as a "monotonic uint32 LE counter" is
really [channel_id][00][00][segment_index], with 0x46=Tran 0x47=Vert
0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero
disagreements. decode_waveform_v2 now takes the channel from that field
instead of rotation position, which was fragile: one missed header
desynced every channel after it.
parse_segment_header now returns n_prev_deltas/prev_deltas/marker/
anchors/channel/segment_index; the old fixed_pattern (02 00 00 01)
conflated the 2-byte marker with the first anchor.
Ground-truth corpus, end to end through the production path:
exact 37 -> 72, truncated 23 -> 3, full-length value errors 15 -> 0.
Store-wide, 729 of 1388 series-3 waveform events decode differently and
728 gain samples; the scale fix changes float values on all of them, so
stored .h5 files need regenerating.
Still open: 3 events truncate at a header variant with a variable-width
prefix (2/4/6 bytes) before the channel id and an `01 00` marker.
Documented in docs/instantel_protocol_reference.md with byte offsets.
+20 tests. No regressions: the byte-exact fixture suite still passes and
the full-suite failure list is unchanged from baseline (16 pre-existing
failures from gitignored fixtures).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
set_false_trigger now clears reviewed_real when flagging false_trigger=True
(mirrors update_event_review's exclusivity), and the quick
PATCH /db/events/{id}/false_trigger endpoint now calls
propagate_review_to_twins after the flag write, matching the sidecar PATCH
path's try/except-with-log.warning pattern. Previously the quick path could
leave both flags set and never touched twins.
Also corrects the v0.25.0 CHANGELOG bullet (exclusivity was not actually
enforced on the quick path until this commit) and adds a caveat comment on
find_twins about rare clamped/saturated-PVS false-positive twin matches.
BW writes ">100 Hz" for ZC Freq when the zero-crossing algorithm sees a
peak too fast to count — the device's reporting ceiling is 100 Hz on
V10.72. Our parser fell back to None via _parse_number (which requires
a leading digit), so the PDF rendered "—" where BW shows ">100".
Mirrors the OORANGE/saturated pattern already used for PPV and PSPL:
parser stores the threshold (100.0) on zc_freq_hz + sets a new
zc_freq_above_range flag. Projection carries the flag through to the
sidecar; PDF renderer prepends ">" when set.
Affects both per-channel stats tables (waveform + histogram variants)
and the mic block's ZC Freq row.
Verified on the real T190LD5Q.LK0W fixture: Tran zc_freq_hz=100.0
above_range=True; Vert/Long (normal values) above_range=False; "N/A"
still produces zc_freq_hz=None which renders as "—" (unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
serversdownandClaude Opus 5