update to 0.26.0. Big chonking update including 0.23, 0.24, and 0.25 as well. #33
Assignees
serversdown
Clear assignees
serversdown
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: serversdown/seismo-relay#33
Reference in New Issue
Block a user
v0.26.0 — 2026-08-27
Series-3 decode correctness. Two body-model rewrites, a systematic
scale error affecting every geophone reading ever produced, a recovered
file format, and two artifact-hygiene bugs where stale files outlived the
decodes that made them. All 11,603 series-3 binaries in the production
snapshot now pass every check.
Fixed
Series-3 health sweep: 11,603 / 11,603 binaries now clean on every check.
Swept every series-3 file with the live decoder against five independent
checks — decode exceptions, zero samples, unequal geo channel lengths, peaks
above range full scale, decoded peak vs the device-reported PPV, and waveform
length vs the declared record time. Three real defects surfaced and were
fixed:
block[22]is not a constant and must not be tested. It was documentedas always
0x00but carries data on loud blocks, and rejecting those threwaway the interval holding the event peak.
BE18350/T350L7HR.NL0Hblock 92 hasblock[22]=0x26and a Tran peak of0x0563= 1379 counts = 6.895 in/s — exactly the device-reported PPV —while the file as a whole decoded to 0.015 in/s.
block[0]==0x00,block[4]==0x0Aand the 4-byte tail are six bytes of constraint, which iswhat keeps trailer content out.
Block-model dispatch now goes on signature strength, not on whichever
decoder returns first. A multi-interval body also yields scattered
standard-tail blocks by coincidence; dispatching on "first non-empty"
handed 193 BE18193 files to the standard walker and produced peaks of
149 in/s against a 10 in/s full scale.
Multi-interval stride detection requires the block counter to increment
by exactly 1. Without it the detector false-positives on ordinary
standard-block bodies: those carry a header every 32 bytes, and
192 = 12 + 20×9and512 = 12 + 20×25are both multiples of 32, so astride "fits" while actually skipping 6 or 16 real blocks. That misrouted
9,082 files.
Partial-block garbage is now trimmed within the final block only, stopping at
the first slot with a non-zero tail word or a geo peak above full scale.
Trimming purely from the end left garbage stranded behind one slot that
happened to have a zero tail word; trimming on the tail word alone truncated
four BE9440 files by up to 2,800 intervals.
Sub-minute histogram intervals are packed several to a block — 415 files
recovered. The device always writes one minute of data per block, so a
shorter interval just means more intervals in a longer block:
stride = 12 + n * 20. Each 20-byte record carries 8 × uint16little-endian values — peak and half-period per channel — plus a 2-word
tail whose first word is
0000on every real interval (a session endingmid-block leaves buffer garbage in the remaining slots, which decoded as
peaks thousands of times the real value until that check was added).
The standard 32-byte block is big-endian; this variant is not.
These 415 files (216 on BE18193 at 2 s intervals, 199 on BE9440 at 15 s)
previously decoded to nothing at all — and before that were being accepted
by the waveform codec, which returned garbage peaking up to 400× the
device-reported PPV.
Ground truth
BE9440/K440L3AQ.T70H— 5,710 intervals — matches itsBlastware ASCII export on 17,130/17,130 geo peaks, 22,840/22,840
frequencies and 5,710/5,710 mic dB(L) values. Across all 455 affected
files, 1,354/1,365 (99.2%) channel peaks match the device-reported PPV;
the 11 that don't are under-reads on BE9440 where the walk stops early.
backfill_sidecars.pynow removes a stale.h5when nothing decodes.It previously skipped the write "so we don't replace whatever's there with an
empty placeholder", which silently preserved output from a superseded
decoder. After the record-chain fix, 415 histogram files stopped decoding (an
unmapped block variant on BE18193 and BE9440) but kept
.h5files whose peaksran up to 400× the device's own reported PPV — garbage feeding the charts
and the false-trigger detector with nothing marking it. Reports a
stale_h5_removedcount.The series-3 waveform body is a RECORD CHAIN, not a tag stream — this
supersedes the segment-header model, including the fixes made earlier the
same day.
Records are self-delimiting.
off+2is auint16 BElength andnext_record = off + 2 + len; the chain ends on a record whosechan_idis0x06.off+8carries a 3-valued mode enum:02 0001 0000 0340 NNis an ordinary int16 BE data block (2*NN + 2), never a segmentheader. Reading it as a
2*NN + 16header is what made walks drift — and the"variable-prefix segment descriptors" reported earlier today were not a format
feature at all, just walker drift of exactly
4 - (old_stop - true_record_start)on all 25 affected files.Measured against the production snapshot:
Mean absolute PPV ratio error on waveforms is now 0.00000. The 2 remaining
ASCII imperfections differ by exactly 1 LSB on samples sitting at the
±10.000 in/s rail.
This also eliminated the walker-over-read class. 24 of those 35 files
were histograms that
read_blastware_filefed to the waveform codec first;the old walker accepted them and returned garbage (one yielded 98,923
"intervals"), while the record-chain decoder correctly returns
Noneso theyfall through to
histogram_codec.00 03records are decoded rather than skipped. Skipping them does not merelylose samples — it silently shifts the time base of everything after them on
that channel (observed on
BE9558/K558LOF2.820W, MicL displaced by exactly512 samples with nothing marking the gap).
Footer detection now prefers whichever
0e 08candidate yields a chainterminating on
0x06, since the signature can occur inside a sample stream.Blast radius: 1 file of 1,388.
The superseded model is retained as
decode_waveform_legacyand pinned bymicromate/idf_file.py, whose Thor IDFW body-offset search trial-decodescandidate offsets and keeps whichever yields the most samples — the new
decoder correctly returns
Nonewhere the old one returned garbage, whichchanges that heuristic's winner. Switching Thor over is deferred until that
search is reworked to use the record chain directly.
Series-3 histogram block is uniformly big-endian, and the stream's final
block has its own tail — the codec was clipping large peaks and dropping the
last interval of nearly every histogram.
uint16big-endian, notuint8plus an"annotation" byte:
T_peak[5:7],T_halfperiod[7:9],V_peak[9:11], and so on. Onlyblock_ctr[2:4]is little-endian. The oldmodel silently clipped any peak above 1.275 in/s — the final interval
of
BE18193/T193LQ9K.OE0Hreads 8.270 in/s in Blastware's own export anddecoded as 0.590. The "annotation" byte was the half-period's high byte,
which is why it was non-zero exactly on the sub-Hz intervals BW renders
as
<1.0.block[4]alone. Testing[4:6]as auint16 LEmarker forced
block[5] == 0— which is what capped the peak at one bytein the first place.
9c 06 00 42instead of1e 0a 00 00, with arbitrary bytes at[21:23]. Rejecting it droppedthe final interval of nearly every histogram, and that interval is
frequently the one holding the event peak — so the file's reported PPV
came out low.
Verified against 1211 production histograms paired with their preserved
Blastware ASCII exports, which carry a full per-interval data table:
1211/1211 now decode exactly (interval count plus every per-interval
peak), and 842,442 per-interval frequency comparisons match with zero
mismatches. Before this fix: 1 of 1196.
decode_histogram_body_fullrecords now exposeis_terminalin place ofthe removed
annotationstuple.Geophone full scale is 32000 ADC counts, not 32768 — every geo reading was
2.3% low. The verified body codec emits geo samples in 16-count units whose
documented LSB is exactly 0.005 in/s, and
decoded_to_adc_countsmultipliesby 16, so one ADC count is
0.005/16in/s and Normal range (10.000 in/s) is10.0 / (0.005/16)= 32000 counts. Bothsfm/event_hdf5.pyandminimateplus/event_file_io.pydivided by 32768, scaling every geophonesample and every derived peak down by
1 - 32000/32768= 2.34%.Measured against 216 per-channel comparisons with preserved Blastware ASCII
exports: 32768 → 151/216 exact (worst error 0.238 in/s on a 10 in/s
event); 32000 → 216/216 exact, worst error 0.005 in/s (exactly 1 LSB —
pure quantization). The error scales with amplitude, so it was invisible on
quiet events and worst on the loud ones that matter for compliance.
The mic path is unaffected — it back-solves its own per-count factor from the
device-reported peak.
Scope: the scale lives in
_samples_to_float, which every event passesthrough regardless of which codec produced the samples — so this affected
waveforms, histograms and series-4 (Thor IDF) alike, not just waveforms.
Verified after regeneration: series-3 histogram peaks vs their ASCII reports
now sit at a median ratio of 1.0000 across 1,137 comparisons (0.9766 under
32768); series-4 peaks vs device peaks moved from a median 0.960 to 0.983
across 1,468 comparisons. The four block-framing fixes below are
waveform-only — histograms decode via
histogram_codec.decode_histogram_body,which is untouched.
Series-3 waveform codec: four block-framing cases caused silent channel
truncation.
walk_bodyhit its unknown-tagbreakmid-stream and everychannel decoded after that point came out short — typically Vert/Long/MicL,
sometimes at a third of their true length, with no error raised.
0X NN— the 12-bit NN encoding already handled for1X NN/2X NNalso applies to the00 NNRLE tag. Runs longer than252 samples must use the wide form (e.g.
01 0c= 268 repeats).30 NNwith NN > 0x10 — the0 < NN <= 0x10guard was arbitrary;data-section
30 NNblocks reach at least NN = 0x18. The length formula(
NN × 1.5 + 2) was already correct.40 NNsegment headers — NN is the count ofprevious-channel continuation deltas, so the header is
2 × NN + 16bytes and every field after the deltas shifts by
2 × NN. Only40 02(20 bytes) was handled;
40 01(18) and40 03(22) both occur.40 NNtag at all: just the 14-byte tail[field2:2][len:2][channel_id:4][marker:2][anchors:4]. This is the NN=0case (no continuation deltas needed, so no tag and no delta bytes). It is
where the walk stopped in 7 of the 8 events still truncating after the
first three fixes.
Changed
Segment channel now comes from the header's own channel-id byte rather
than from rotation position. The field previously documented as a
"monotonic uint32 LE counter" is really
[channel][00][00][segment_index]with
0x46=Tran0x47=Vert0x48=Long0x49=MicL — verified on1697 of 1697 segment headers across the ground-truth corpus with zero
disagreements. Rotation-by-position is kept only as a fallback for unknown
ids; it was fragile because a single missed or extra header (exactly what
tagless headers caused) desynced every channel after it.
parse_segment_headerreturn shape — nown_prev_deltas,prev_deltas,marker,anchors,channel,segment_indexin place ofthe fixed-offset
anchor_bytes/fixed_pattern/tailkeys. The oldfixed_pattern(02 00 00 01) conflated the 2-byte constant marker withthe first anchor.
counteris retained as the raw uint32 of the id field.Verification
Against the 75 ground-truth events (BW binary paired with its preserved
_ASCII.TXTexport), decoding end-to-end through the production path:Worst remaining error among the 72: 0.0050 in/s = exactly 1 LSB.
No regressions — the byte-exact fixture suite still passes, and the full-suite
failure list is unchanged from baseline (16 pre-existing failures from
gitignored fixtures).
Notes
The "DC offset" symptom is not a decode bug. Events whose geo trace
sits at a constant level instead of oscillating around zero
(dominant-axis
|mean| / peak>> 0) reproduce exactly in Blastware's ownASCII export — e.g.
BE12599/N599LQD7.8E0WTran reads mean +0.345,min +0.335, max +0.355 in both. It is a known recurring hardware fault (the
operators call it an "offset"): the affected channel's baseline exceeds the
unit's own geo trigger level, so the unit retriggers continuously and floods
the ACH queue with garbage events. Store-wide it affects 2 units of 21 across
6 episodes; see
scratch/offset_candidates.csvand the project memory notes.Still open: 3 of 75 ground-truth events truncate at a segment-headerResolved later the same day — thevariant with a variable-width prefix.
record-chain rewrite (above) showed there is no variable prefix; it was
walker drift. All 75 are now sample-count exact.
Still open after this release:
against device peaks with a ~1.7% systematic bias and a near-zero tail.
Thor IDFW is pinned to
decode_waveform_legacydeliberately.(= 10.0/1.25) — a units bug, not a codec one. Never chased.
backfill_sidecars.py --forcealso inserts DB rows for store filesthat have none (1,286 on the snapshot; one-time per store), and the
dry-run does not report that count before you commit to it.
preserved
_ASCII.TXT, peak-and-structure consistency on the other 89%.v0.25.0 — 2026-08-25
reviewed_real 3-state review flag + twin review-propagation. The
eventstable and the/db/eventsfeed now carryreviewed_real, a3-state review flag mutually exclusive with
false_trigger, mirrored fromthe sidecar review block — plus histogram/waveform twin review-propagation
(flagging one flags both, matched by serial + identical PVS + timestamp
window).
Added
eventscolumnreviewed_real—INTEGER NOT NULL DEFAULT 0, addedvia the existing incremental
_migrateADD COLUMN pass (auto-migrates onSeismoDb()construction, no manual migration).query_events/get_event(and thus/db/events) return it automatically (SELECT *).false_trigger— settingreviewed_real=1clears
false_trigger, and vice versa, enforced on both review paths:the sidecar review PATCH (
update_event_review) and the quickPATCH /db/events/{id}/false_triggerendpoint (set_false_trigger).find_twins— matches an event's histogram/waveform twins by serial +identical peak-vector-sum + a timestamp window.
propagate_review_to_twins— copies an event'sfalse_trigger/reviewed_realstate onto its twins, wired into both thePATCH /db/events/{id}/sidecarreview path and the quickPATCH /db/events/{id}/false_triggerpath, so flagging one flags bothregardless of which endpoint made the change.
v0.24.0 — 2026-08-22
Waveform-shape metrics on events. The
eventstable and the/db/eventsfeed now carry per-event crest factor and points-near-peak, computed from the
decoded waveform samples at ingest — groundwork for Terra-View's
false-trigger detection (Phase B).
Added
eventscolumnsshape_crest_factor,shape_near_peak_count,shape_sample_count,shape_axis. Added via the existing incremental_migrateADD COLUMN pass — auto-migrates onSeismoDb()construction,no manual migration.
query_events/get_event(and thus/db/events)return them automatically (
SELECT *).the decoded samples in every save path (
shape_from_h5/shape_from_samples), andinsert_eventspersists them on INSERT andUPSERT.
scripts/backfill_event_shape.py— fills the columns forexisting events from their on-disk
.h5waveform samples (idempotent,UPDATE-only).
Upgrade Notes
Run the backfill once after deploying, against the events DB + waveform store:
python3 scripts/backfill_event_shape.py --db-path <seismo_relay.db> --store-root <waveforms/>Events with no decodable samples (or no waveform file) stay NULL and render
"—" downstream.
set_false_trigger now clears reviewed_real when flagging false_trigger=True (mirrors update_event_review's exclusivity), and the quick PATCH /db/events/{id}/false_trigger endpoint now calls propagate_review_to_twins after the flag write, matching the sidecar PATCH path's try/except-with-log.warning pattern. Previously the quick path could leave both flags set and never touched twins. Also corrects the v0.25.0 CHANGELOG bullet (exclusivity was not actually enforced on the quick path until this commit) and adds a caveat comment on find_twins about rare clamped/saturated-PVS false-positive twin matches.Two independent bugs, both found by diffing 75 production events against their preserved Blastware ASCII exports (<store>/<serial>/<file>_ASCII.TXT). 1. Geo full scale was wrong — every geophone reading was 2.34% low. The codec emits geo samples in 16-count units with a documented LSB of exactly 0.005 in/s, and decoded_to_adc_counts multiplies by 16, so one ADC count is 0.005/16 in/s and 10.000 in/s is 10.0/(0.005/16) = 32000 counts. sfm/event_hdf5.py and minimateplus/event_file_io.py both divided by 32768 (2^15), scaling every sample and derived peak down by 1 - 32000/32768. The error scales with amplitude, so it was invisible on quiet events and worst on the loud ones that matter for compliance. Mic is unaffected (it back-solves its scale from the device peak). 216 per-channel comparisons: 32768 -> 151/216 exact, worst error 0.238 in/s on a 10 in/s event; 32000 -> 216/216 exact, worst 0.005 = 1 LSB. 2. walk_body silently truncated channels on four unhandled framing cases. An unrecognised tag ends the walk and decode_waveform_v2 returns whatever it got, so this surfaced as short channels, never an error: - wide-NN RLE `0X NN` (runs longer than 252 samples) - `30 NN` with NN > 0x10 (the old cap was arbitrary) - variable-width `40 NN` headers: NN counts previous-channel continuation deltas, so the header is 2*NN + 16 bytes; `40 01` and `40 03` occur alongside `40 02` - tagless segment headers: no `40 NN` tag at all, just the 14-byte tail [field2:2][len:2][channel_id:4][marker:2][anchors:4] Also: the header field documented as a "monotonic uint32 LE counter" is really [channel_id][00][00][segment_index], with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. decode_waveform_v2 now takes the channel from that field instead of rotation position, which was fragile: one missed header desynced every channel after it. parse_segment_header now returns n_prev_deltas/prev_deltas/marker/ anchors/channel/segment_index; the old fixed_pattern (02 00 00 01) conflated the 2-byte marker with the first anchor. Ground-truth corpus, end to end through the production path: exact 37 -> 72, truncated 23 -> 3, full-length value errors 15 -> 0. Store-wide, 729 of 1388 series-3 waveform events decode differently and 728 gain samples; the scale fix changes float values on all of them, so stored .h5 files need regenerating. Still open: 3 events truncate at a header variant with a variable-width prefix (2/4/6 bytes) before the channel id and an `01 00` marker. Documented in docs/instantel_protocol_reference.md with byte offsets. +20 tests. No regressions: the byte-exact fixture suite still passes and the full-suite failure list is unchanged from baseline (16 pre-existing failures from gitignored fixtures). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcogBrings the protocol reference, CLAUDE.md and the codec RE status doc up to date with everything confirmed in this pass. instantel_protocol_reference.md - Changelog row for the five findings. - S7.6.1: scope table showing the 32000 scale correction applies to series-3 waveform, series-3 histogram and series-4 Thor alike, with the measured before/after ratios for each. - S15: closed "Full channel ID mapping in SUB 5A stream" — resolved by the segment-header channel id ([channel][00][00][segment], 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL, 1697/1697 verified). Four new open questions: variable-prefix segment descriptors, the histogram codec missing peak intervals (26% of channels), UM-series IDF decoding ~1000x low, and the Thor per-count LSB residual. - NEW Appendix E — Known Device Faults. Documents the field-observed "offset" fault: symptom, why it floods the ACH queue (pedestal exceeds the unit's own geo trigger level), the episode table, the detection rule that works, what the data rules out (not the geophone, not the battery, not environmental, not condensation), and the two remaining candidate mechanisms with the test that separates them. Explicitly flags that it is NOT a decode artifact, since that mistake has already been made once. CLAUDE.md - Body-codec section: the four framing cases and the channel-id finding, with the corpus result. - "What's NOT solved": replaced the stale walker-edge-cases bullet with the four genuinely open items. waveform_codec_re_status.md - Scale scope table matching the protocol reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcogRe-measured properly. The first pass compared h5 max against the ASCII header PPV and reported "26% of channels miss the peak". The histogram ASCII actually carries a full per-interval data table (Tran/Vert/Long peak + freq + PVS per interval), which is real ground truth, so the comparison should have been per-interval from the start. Per-interval result, n=1196 series-3 histograms: - decoded VALUES are right: 1031/1196 (86%) match within 1 LSB across the overlapping prefix - the interval COUNT is short in 1195 of 1196 files: median 1 missing, 1088 short by 1-2, 65 by 3-10, 39 by 11-100, 3 by >100 (max 205) - decoded max falls below the device PPV in 169/1196 files (14%), not 26% — that happens when a dropped interval held the peak So it is a termination bug in histogram_codec.decode_histogram_body, the same family as the waveform-walker truncation fixed earlier today, rather than mis-decoded interval values. Series-3 only; there is no preserved series-4 ASCII in the snapshot to compare against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcogSub-minute histogram intervals are packed several to a block so that every block still covers exactly one minute of data: interval intervals/block stride 1 minute 1 32 <- the standard big-endian block 15 s 4 92 2 s 30 612 stride = 12 + n * 20 Block = [00][segment][ctr uint16 LE][0a][00], then n x 20-byte records of 8 x uint16 LITTLE-endian values (T_peak, T_halfp, V_peak, V_halfp, L_peak, L_halfp, M_peak, M_halfp) plus a 2-word tail whose first word is 0000 on every real interval, then a 6-byte block trailer. The standard 32-byte block is BIG-endian; this variant is LITTLE-endian. The tail-word check matters: a session ending mid-block leaves buffer garbage in the remaining interval slots, which decoded as peaks thousands of times the real value. Stride detection also requires at least 2 records, since a 1-record block would have stride 32 and collide with the standard block. Recovers 415 files that decoded to nothing: 216 on BE18193 (2 s intervals) and 199 on BE9440 (15 s). Before decoding to nothing they were being accepted by the WAVEFORM codec, which returned garbage peaking up to 400x the device-reported PPV. Ground truth BE9440/K440L3AQ.T70H (5,710 intervals) matches its Blastware ASCII export exactly: 17,130/17,130 geo peaks, 22,840/22,840 frequencies, 5,710/5,710 mic dB(L). Across all 455 affected files, 1,354/1,365 channel peaks (99.2%) match the device-reported PPV; the 11 that don't are under-reads on BE9440 where the walk stops early. Fixture (binary + ASCII) saved under tests/fixtures/, which is gitignored per repo practice — the ground-truth test skips when absent. Tests: 258 passed, failure list unchanged from baseline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcogTOOL_VERSION had been frozen at 0.21.1 for four releases despite its own comment saying "Bump this constant and CHANGELOG.md together at release time". It is not cosmetic: backfill_sidecars.py decides whether to regenerate with ver_ok = sidecar.source.tool_version >= event_file_io.TOOL_VERSION so with the constant stuck at 0.21.1 and every sidecar stamped 0.21.1, a backfill WITHOUT --force skipped the entire store. That is precisely the failure the check exists to prevent, and it means every sidecar regenerated during the 0.26.0 decode work is stamped 0.21.1 while having been produced by 0.26.0 code. Verified: a non-force dry-run over the snapshot now reports written=11603 skipped(uptodate)=0, where before it would have skipped all 11,603. Prod therefore does not need --force to pick up the decode corrections — the version difference alone is enough. Note the installed dist metadata reads 0.12.0, older than the constant, so the best-effort "prefer installed metadata when newer" path correctly defers to TOOL_VERSION. Also bumps the README header, which still read v0.22.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog