Brian's method, and better than v2's whole-record median: the pre-trigger
window is definitionally quiet (the buffer captured before the trigger fired),
whereas a median is merely robust to the event. Requiring the floor to hold
across pre-trigger / middle / end rejects transients that a median cannot.
per channel: pre/mid/end medians, spread = max - min
offset when |pre| >= floor AND spread <= 0.02 in/s
real fault >= 3 consecutive flagged events on that channel
The empirical noise floor justifies the threshold and validates the decoder:
across 19,244 non-flagged channel-events the pre-trigger floor is 62.7% exactly
0.000, 94.5% within +/-1 quantisation unit, median +0.0000, mean -0.0008. There
is no systematic zero-point bias — an independent confirmation of the
32000-count geo scale.
The result is threshold-insensitive across a 2x range (0.020 to 0.040 in/s),
which is what separates a real signal from a tuned one:
FINAL: 5 of 45 units (11%) — BE9558, BE11529, BE12599, BE13117, BE18438
BE11007 and BE10895 drop out; the spread test identifies them as transients
rather than pedestals. v1's 11% headline was right by luck — it included
BE11007 and named the wrong channel on most units.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Brian challenged the v1 finding that offsets "come and go", against field
experience that a unit which develops one stays broken until the geophone is
replaced. He was right; v1 had two flaws, both of which manufactured false
recoveries:
1. It scored only the axis with the largest peak, so a real event on one axis
hid a persistent pedestal on another. BE12599 on 2026-08-21 read "clean"
because Long had a 1.065 in/s event, while Tran sat at +0.4732 in/s and was
never examined.
2. It used the mean, which a real transient perturbs. The median is the resting
baseline and a blast does not move it. Same event, Long channel:
mean +0.0783 vs median -0.0050.
offset_scan2.py flags a CHANNEL when |median| >= 0.025 in/s (5 A/D counts,
Instantel's own criterion) and treats >=3 consecutive flagged events as the
real signal. No m/p ratio guard is needed — that existed only to compensate for
the mean.
Corrected results:
units with any flagged event 6 -> 19 of 45
units with a sustained pedestal 8 of 45 (18%)
runs >=3 consecutive 29; 1-2 event runs (noise) 69
Also corrected: the affected channel is most often Vert, not Tran (v1 named
whichever axis had the largest peak, so it was frequently wrong). BE10895 and
BE18003 were invisible to v1. BE12599's fault began 2026-08-14, not 08-17.
The decode itself was never in question and is confirmed against Blastware's
own ASCII export: on K558LJN3.BK0W, BW shows Tran parked at +0.265..+0.375 in/s
for the entire record while Vert and Long sit at ~0.005 — Instantel's "parallel
lines above or below the zero line".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
terra-view's SFM Admin page (/admin/sfm) displays whatever /health returns for
`version`. That was hardcoded to "0.1.0" and never bumped, so the page showed
0.1.0 while the service was actually 0.26.0. Point both /health and the FastAPI
OpenAPI version at the release-bumped TOOL_VERSION (single source of truth), so
they can't drift again. Adds httpx-free regression tests (call health() directly).
Note: minimateplus.__version__ is separately stale at 0.1.0 — left as-is here
(nothing user-facing reads it; touching the package __init__ risks import order).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
Adds docs/offset_investigation.md — a dated journal of the "offset" hardware
fault, in the style of the codec status docs: findings with provenance, dead
ends kept with the reason they died, and per-unit case files.
Contents:
- base rate 5-6 of 45 units (11-13%) over the DL2 archive, 2018-2026,
confirming rather than overturning the earlier 2-of-21 estimate
- the detector, with the rationale for each term and its known blind spot
(event traces carry real motion, so only trace-dominating offsets show)
- the bimodality result: relaxing the amplitude floor 11x adds no new units
- Instantel's own procedure and thresholds from their FAQs 13-0-21 / 12-0-10:
A/D-mode ">5 counts", the autozero key sequence, and the 2027-2069 X1/X8
acceptance window that explains the ~10% field success rate of a re-zero
- four ruled-out hypotheses, each with the evidence that killed it:
condensation, clipping, the sensor check as a predictor (102 offset events,
zero failures — a grossly offset unit passes its own self-check), and the
calibration-timing correlation (confounded, one unit per time bucket)
- open questions, chiefly whether SUB 0x0E carries the autozero numbers
Cross-referenced from CLAUDE.md and Appendix E of the protocol reference
(whose CRLF line endings are preserved).
Separately: CLAUDE.md had 793 NUL bytes appended after its last line. They
predate this work (present at least as far back as e42956a / v0.21.0) and made
grep treat the file as binary, silently skipping it. Stripped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
The DL2 export keeps a byte-identical `Sent/` copy of its root, so walking it
counts every binary twice: 127,035 histogram paths are 63,535 distinct files,
and 13,077 waveform paths are 6,577. offset_scan.py now keeps the first
occurrence of each basename.
Corrects the previous commit's changelog claim of 8 recovered files — it is 4:
K440HJCN.3C0H and K557IF1U.8K0H (stride 252), T191HVNP.0S0H (92), T193L0XM.CI0H
(612). Still zero regressions. The per-unit breakdown reading exactly 2-2-2-2
should have given the doubling away.
The 14,338-exact verification result is unaffected: ASCII exports are not
mirrored (14,340 paths, 14,340 distinct names), and the harness enumerates
those rather than the binaries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
detect_multi_interval_stride() confirmed a candidate stride on a third block
header whenever the body was long enough to contain one. But a body can exceed
two strides and still hold only two real blocks: a partial final block leaves
trailing padding. BE18193 T193L0XM.CI0H — 51 intervals at 2 s, i.e. one full
30-interval block plus a 21-interval remainder in a 2787-byte body — had every
decisive check pass at stride 612 (header at 0, header at 612, block counter
256 -> 257) and was then rejected for the absent third header at 1224. It
decoded to nothing.
A missing third header now means end-of-stream rather than disqualification.
The block-counter check is untouched — that is the test that prevents the
false positives which once handed 9,082 standard-block files to the
multi-interval walker.
Found by running the full DL2 archive against its preserved Blastware ASCII
exports (14,340 paired files, 11x the previous ground-truth corpus).
Measured over 127,035 archive histogram binaries:
recovered 8 files (strides 92, 252, 612; BE18193, BE18191, BE9557, BE9440)
regressed 0 files
Full-corpus verification: 14,337 -> 14,338 exact of 14,338 decodable pairs
(the 2 excluded are series-4 IDF, a different codec).
Also adds scratch/verify_against_ascii.py (per-sample decoder verification
against BW exports, with a saturation carve-out — BW clamps clipped events to
the range max while the decoder reports true counts) and scratch/offset_scan.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
A real trigger is recorded twice — as a triggered waveform (stamped at the
trigger instant) and inside the scheduled histogram whose interval contains it
(stamped at the 7am/7pm interval start). The two twins routinely differ by
HOURS, so the old ±5-minute window in find_twins silently missed them — which
broke review propagation (flagging one twin left its twin unflagged).
Twins are now matched by: same serial + identical peak_vector_sum + OPPOSITE
record type + the waveform's timestamp falling within the histogram's interval
(bounded by the next same-serial histogram). Matching keys off record timestamps
(not call-in/received times, which drift with field connectivity). window_seconds
is retained but ignored.
Rewrote test_find_twins + test_twin_propagation for the new contract (incl. the
75-min-apart UM12947 case, cross-type exclusion, containing-interval selection,
open-ended latest interval). Full suite: 264 passed; the 16 failures are
pre-existing (missing gitignored fixtures + a v0.26.0 codec case), unchanged
from baseline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
TOOL_VERSION had been frozen at 0.21.1 for four releases despite its own
comment saying "Bump this constant and CHANGELOG.md together at release
time". It is not cosmetic: backfill_sidecars.py decides whether to
regenerate with
ver_ok = sidecar.source.tool_version >= event_file_io.TOOL_VERSION
so with the constant stuck at 0.21.1 and every sidecar stamped 0.21.1,
a backfill WITHOUT --force skipped the entire store. That is precisely
the failure the check exists to prevent, and it means every sidecar
regenerated during the 0.26.0 decode work is stamped 0.21.1 while having
been produced by 0.26.0 code.
Verified: a non-force dry-run over the snapshot now reports
written=11603 skipped(uptodate)=0, where before it would have skipped
all 11,603. Prod therefore does not need --force to pick up the decode
corrections — the version difference alone is enough.
Note the installed dist metadata reads 0.12.0, older than the constant,
so the best-effort "prefer installed metadata when newer" path correctly
defers to TOOL_VERSION.
Also bumps the README header, which still read v0.22.0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Two body-model rewrites, a systematic scale error affecting every
geophone reading the system ever produced, a recovered file format, and
two artifact-hygiene bugs where stale files outlived the decodes that
made them.
- geo full scale is 32000 ADC counts, not 32768 (every reading 2.34% low)
- the waveform body is a record chain, not a tag stream
- the histogram block is big-endian, with a terminal tail
- sub-minute intervals pack several per block (415 files recovered)
- three more defects found by a full-corpus sweep, each masking the next
- stale .h5 files and stale shape_* columns are now cleared, not left
All 11,603 series-3 binaries in the production snapshot pass every check.
Ground truth: 1,211/1,211 histograms exact per-interval, 75/75 waveform
sample counts exact, multi-interval fixture exact on all 45,680 values.
Also corrects a changelog note that went stale within the same day: the
"3 of 75 events still truncate" item was resolved by the record-chain
rewrite, and the remaining open items are now listed explicitly.
CLAUDE.md gains a "Where things stand" block at the top — the header had
been reading v0.21.0, four releases behind, which is the first thing you
see when picking the project back up.
Tests: 259 passed; the 16 failures are pre-existing (gitignored
fixtures) and unchanged from baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Swept every series-3 binary with the live decoder against five
independent checks: decode exceptions, zero samples, unequal geo channel
lengths, peaks above range full scale, decoded peak vs device-reported
PPV, and waveform length vs declared record time.
1. block[22] is NOT a constant and must not be tested. Documented as
always 0x00, it carries data on loud blocks, and rejecting those threw
away the interval holding the event peak. BE18350/T350L7HR.NL0H
block 92 has block[22]=0x26 and a Tran peak of 0x0563 = 1379 counts =
6.895 in/s — exactly the device-reported PPV — while the file decoded
to 0.015 in/s. block[0]==0, block[4]==0x0A and the 4-byte tail are
six bytes of constraint, which is what keeps trailer content out.
2. Block-model dispatch now goes on signature strength rather than on
whichever decoder returns first. A multi-interval body also yields
scattered standard-tail blocks by coincidence, so "first non-empty"
handed 193 BE18193 files to the standard walker and produced peaks of
149 in/s against a 10 in/s full scale.
3. Multi-interval stride detection requires the block counter to
increment by exactly 1. Without it the detector false-positives on
ordinary standard-block bodies: they carry a header every 32 bytes,
and 192 = 12 + 20*9 and 512 = 12 + 20*25 are both multiples of 32, so
a stride "fits" while skipping 6 or 16 real blocks. That misrouted
9,082 files.
Partial-block garbage is trimmed within the final block only, stopping
at the first slot with a non-zero tail word or a geo peak above full
scale (2000 counts in 16-count units). Trimming purely from the end
left garbage stranded behind a slot that happened to have a zero tail
word; trimming on the tail word alone truncated four BE9440 files by up
to 2,800 intervals.
Result: 11,603 / 11,603 series-3 binaries clean on every check.
Ground truth unchanged: 1211/1211 histograms exact per-interval, 75/75
waveform sample counts exact (73/75 fully exact, the 2 differ by 1 LSB
on rail samples), and the multi-interval fixture still matches its BW
ASCII export on all 45,680 values.
Tests: 259 passed, failure list unchanged from baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Sub-minute histogram intervals are packed several to a block so that
every block still covers exactly one minute of data:
interval intervals/block stride
1 minute 1 32 <- the standard big-endian block
15 s 4 92
2 s 30 612
stride = 12 + n * 20
Block = [00][segment][ctr uint16 LE][0a][00], then n x 20-byte records of
8 x uint16 LITTLE-endian values (T_peak, T_halfp, V_peak, V_halfp,
L_peak, L_halfp, M_peak, M_halfp) plus a 2-word tail whose first word is
0000 on every real interval, then a 6-byte block trailer.
The standard 32-byte block is BIG-endian; this variant is LITTLE-endian.
The tail-word check matters: a session ending mid-block leaves buffer
garbage in the remaining interval slots, which decoded as peaks
thousands of times the real value. Stride detection also requires at
least 2 records, since a 1-record block would have stride 32 and
collide with the standard block.
Recovers 415 files that decoded to nothing: 216 on BE18193 (2 s
intervals) and 199 on BE9440 (15 s). Before decoding to nothing they
were being accepted by the WAVEFORM codec, which returned garbage
peaking up to 400x the device-reported PPV.
Ground truth BE9440/K440L3AQ.T70H (5,710 intervals) matches its
Blastware ASCII export exactly: 17,130/17,130 geo peaks, 22,840/22,840
frequencies, 5,710/5,710 mic dB(L). Across all 455 affected files,
1,354/1,365 channel peaks (99.2%) match the device-reported PPV; the 11
that don't are under-reads on BE9440 where the walk stops early.
Fixture (binary + ASCII) saved under tests/fixtures/, which is
gitignored per repo practice — the ground-truth test skips when absent.
Tests: 258 passed, failure list unchanged from baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
backfill_sidecars.py skipped the .h5 write when a file produced no
samples, with the stated intent of not replacing it with an empty
placeholder. That silently preserved output from a superseded decoder.
After the record-chain fix, 415 histogram files stopped decoding (216 on
BE18193, 199 on BE9440) but kept .h5 files whose peaks ran up to 400x
the device-reported PPV. Those were feeding charts and the
false-trigger detector with nothing marking them. The .h5 is now
removed in that case and the run reports stale_h5_removed.
Store-wide effect, series-3, decoded peak vs device-reported PPV:
waveform 1307/1307 (100%), mean abs ratio error 0.00000
histogram 4434/4435 (100%)
Both were 99% with a tail of 18 and 25 wrong files respectively.
The 415 files are a genuine unmapped format variant, not a regression:
their bodies open `00 00 00 01 0a 00` (valid block header, marker 0a at
[4], block_ctr 256) but block[28:32] matches neither known tail, and no
stride from 8 to 64 bytes places a marker at [4] consistently. Bodies
are very large (one is 360,573 bytes). They were previously being
decoded by the WAVEFORM codec, which accepted them and returned garbage
- so the gap pre-dates today's work; the fix only exposed it. Logged as
an open question in the protocol reference.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Supersedes the segment-header model entirely, including the fixes made
earlier today. Found via multi-agent structural analysis of the 25 files
that stalled the walker, then verified independently.
Records are self-delimiting: off+2 is a uint16 BE length, next_record =
off + 2 + len, and the chain ends on a record whose chan_id is 0x06.
off+8 carries a 3-valued mode enum:
02 00 14-byte header, 2 anchors, then CUMULATIVE delta blocks
01 00 10-byte header, no anchors, blocks are ABSOLUTE values
00 03 10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute
`40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header.
Reading it as a 2*NN + 16 header is what made walks drift — the
"variable-prefix segment descriptors" reported earlier today were not a
format feature, just walker drift of exactly
4 - (old_stop - true_record_start), on all 25 affected files.
Measured on the production snapshot:
all four channels equal length 156/1388 -> 1388/1388
ASCII sample-count exact 72/75 -> 75/75
ASCII fully exact 70/75 -> 73/75
device PPV waveform (live) 1288/1306 -> 1306/1306 (mean err 0.00000)
device PPV histogram (live) 4434/4459 -> 4458/4459
Also eliminates the walker-over-read class: 24 of those 35 files were
histograms that read_blastware_file fed to the waveform codec first; the
old walker accepted them and returned garbage (one yielded 98,923
"intervals"), while the record-chain decoder returns None so they fall
through to histogram_codec.
00 03 records are DECODED, not skipped. Skipping them silently shifts
the time base of everything after them on that channel — BE9558/
K558LOF2.820W had MicL displaced by exactly 512 samples with nothing
marking the gap.
Footer detection now prefers the 0e 08 candidate whose body yields a
chain terminating on 0x06; the signature can occur inside a sample
stream. Blast radius 1 file of 1388.
The superseded model survives as decode_waveform_legacy, pinned by
micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes
candidates and keeps whichever yields the most samples, so the new
decoder returning None where the old returned garbage changes that
heuristic's winner. Deferred until that search uses the record chain.
Tests: 253 passed (+11), failure list unchanged from baseline. The 9
tests pinning the superseded model are retargeted at
decode_waveform_legacy, which still implements it.
NOTE: stored .h5 files need regenerating — nearly all get longer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
backfill_event_shape.py skipped rows whose .h5 produced no shape and left
the previously stored value in place. A stale shape outlives the decode
it came from and silently feeds the false-trigger detector.
Found while re-running the backfill after the histogram codec fix: 493
rows in the prod snapshot were carrying shape metrics that no longer
matched their .h5 — e.g. BE17353/S353LDOK.XZ0H held crest_factor from a
223-sample decode while its .h5 holds a single interval. These predate
today's work (present in the pre-32000 snapshot), so this is pre-existing
behaviour rather than fallout from the codec fixes.
Now NULLs shape_crest_factor / shape_near_peak_count / shape_sample_count
/ shape_axis in that case and reports a `cleared_stale` count. Verified
on the snapshot: 493 cleared, 0 stale rows remaining, 11570 rows matching
their .h5 exactly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Two errors in the series-3 histogram block model, both found by diffing
against the per-interval data table in the preserved Blastware ASCII
exports (1211 files in the prod snapshot — far stronger ground truth
than the header PPV used previously).
1. The block is uniformly BIG-ENDIAN. Peaks and half-periods are uint16
BE (T_peak [5:7], T_halfperiod [7:9], V_peak [9:11], V_halfperiod
[11:13], L_peak [13:15], L_halfperiod [15:17], M_peak [17:19],
M_halfperiod [19:21]); only block_ctr [2:4] is little-endian.
The old uint8-peak model silently CLIPPED any peak above 1.275 in/s:
the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's
export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590.
The byte documented as a per-channel "annotation" was never an
annotation — it is the half-period's high byte, which is exactly why
it was non-zero on the sub-Hz intervals BW renders as "<1.0".
The marker is block[4] alone. Testing [4:6] as a uint16 LE marker
forced block[5] == 0, which is what capped the peak at one byte.
2. The final block of each stream carries tail 9c 06 00 42 instead of
1e 0a 00 00, and holds arbitrary bytes at [21:23]. Rejecting it
dropped the last interval of nearly every histogram — frequently the
interval holding the event peak, so the file's PPV read low.
Verified end to end through the production path: 1211/1211 histograms
decode exactly (interval count + every per-interval peak), plus 842,442
per-interval frequency comparisons with zero mismatches. Previously
1 of 1196 files was fully correct.
decode_histogram_body_full records expose `is_terminal` in place of the
removed `annotations` tuple. +6 tests. No regressions: full-suite
failure list unchanged from baseline.
NOTE: stored histogram .h5 files need regenerating to pick this up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Re-measured properly. The first pass compared h5 max against the ASCII
header PPV and reported "26% of channels miss the peak". The histogram
ASCII actually carries a full per-interval data table (Tran/Vert/Long
peak + freq + PVS per interval), which is real ground truth, so the
comparison should have been per-interval from the start.
Per-interval result, n=1196 series-3 histograms:
- decoded VALUES are right: 1031/1196 (86%) match within 1 LSB across
the overlapping prefix
- the interval COUNT is short in 1195 of 1196 files: median 1 missing,
1088 short by 1-2, 65 by 3-10, 39 by 11-100, 3 by >100 (max 205)
- decoded max falls below the device PPV in 169/1196 files (14%), not
26% — that happens when a dropped interval held the peak
So it is a termination bug in histogram_codec.decode_histogram_body,
the same family as the waveform-walker truncation fixed earlier today,
rather than mis-decoded interval values. Series-3 only; there is no
preserved series-4 ASCII in the snapshot to compare against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Operator report: attaching a different geophone to an affected unit
makes the offset go away. That rules out the unit's analog front-end
and any stored per-channel zero constant (a constant lives in the unit
and would survive a sensor swap).
The stored data agrees — MicL, a separate transducer on its own cable,
shows no offset during either episode (|mean|/peak 0.17 and 0.02) while
the geo channels on the same unit at the same moment are pinned.
Two distinct sensor-side patterns recorded:
BE18438 Vert 0.97, Tran 0.16, Long 0.18 -> one conductor pair
BE9558 Long 0.99, Tran 0.90, Vert 0.81 -> shared return / ground
Candidate mechanisms narrowed to three, since a geophone coil is passive
and cannot generate sustained DC: galvanic corrosion at a connector or
splice (matches the ~46 mV referred to the ADC input), a leakage path to
shield, or changed coil DC resistance interacting with the amplifier's
input bias current.
Also records the confound: swapping a sensor requires a monitoring
restart, and these units run Sensor Check "Before monitoring", so the
restart re-zeros too. The swap does not cleanly separate "new sensor"
from "the restart re-zeroed it". Controls and the single best
measurement (open-circuit DC across the suspect connector) documented.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Brings the protocol reference, CLAUDE.md and the codec RE status doc up
to date with everything confirmed in this pass.
instantel_protocol_reference.md
- Changelog row for the five findings.
- S7.6.1: scope table showing the 32000 scale correction applies to
series-3 waveform, series-3 histogram and series-4 Thor alike, with
the measured before/after ratios for each.
- S15: closed "Full channel ID mapping in SUB 5A stream" — resolved by
the segment-header channel id ([channel][00][00][segment], 0x46=Tran
0x47=Vert 0x48=Long 0x49=MicL, 1697/1697 verified). Four new open
questions: variable-prefix segment descriptors, the histogram codec
missing peak intervals (26% of channels), UM-series IDF decoding
~1000x low, and the Thor per-count LSB residual.
- NEW Appendix E — Known Device Faults. Documents the field-observed
"offset" fault: symptom, why it floods the ACH queue (pedestal
exceeds the unit's own geo trigger level), the episode table, the
detection rule that works, what the data rules out (not the
geophone, not the battery, not environmental, not condensation),
and the two remaining candidate mechanisms with the test that
separates them. Explicitly flags that it is NOT a decode artifact,
since that mistake has already been made once.
CLAUDE.md
- Body-codec section: the four framing cases and the channel-id
finding, with the corpus result.
- "What's NOT solved": replaced the stale walker-edge-cases bullet
with the four genuinely open items.
waveform_codec_re_status.md
- Scale scope table matching the protocol reference.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
274 series-3 waveform events across 6 episodes on 2 units (BE9558,
BE18438) whose dominant geo axis sits pinned at a DC offset above that
unit's own geo trigger level — the "offset" hardware fault that makes a
unit retrigger continuously and flood the ACH queue.
Detection rule: dominant-axis |mean|/peak > 0.7 AND |mean| >= 0.9 x the
unit's geo trigger level. Bare |mean|/peak is useless on quiet events —
a trace at the 0.010 in/s noise floor clears any ratio threshold.
Not a decode artifact: these reproduce exactly in Blastware's own ASCII
export. Kept as the starting point for the archive-wide analysis.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
The scale lives in _samples_to_float, which every event passes through
regardless of source codec, so waveforms, histograms and Thor IDF events
were all 2.34% low — not just waveforms. Verified after regeneration:
series-3 histogram peaks vs ASCII reports now median 1.0000 across 1137
comparisons (0.9766 under 32768); series-4 peaks vs device peaks moved
from median 0.960 to 0.983 across 1468.
The four block-framing fixes remain waveform-only; histogram_codec is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Two independent bugs, both found by diffing 75 production events against
their preserved Blastware ASCII exports (<store>/<serial>/<file>_ASCII.TXT).
1. Geo full scale was wrong — every geophone reading was 2.34% low.
The codec emits geo samples in 16-count units with a documented LSB of
exactly 0.005 in/s, and decoded_to_adc_counts multiplies by 16, so one
ADC count is 0.005/16 in/s and 10.000 in/s is 10.0/(0.005/16) = 32000
counts. sfm/event_hdf5.py and minimateplus/event_file_io.py both
divided by 32768 (2^15), scaling every sample and derived peak down by
1 - 32000/32768. The error scales with amplitude, so it was invisible
on quiet events and worst on the loud ones that matter for compliance.
Mic is unaffected (it back-solves its scale from the device peak).
216 per-channel comparisons: 32768 -> 151/216 exact, worst error 0.238
in/s on a 10 in/s event; 32000 -> 216/216 exact, worst 0.005 = 1 LSB.
2. walk_body silently truncated channels on four unhandled framing cases.
An unrecognised tag ends the walk and decode_waveform_v2 returns
whatever it got, so this surfaced as short channels, never an error:
- wide-NN RLE `0X NN` (runs longer than 252 samples)
- `30 NN` with NN > 0x10 (the old cap was arbitrary)
- variable-width `40 NN` headers: NN counts previous-channel
continuation deltas, so the header is 2*NN + 16 bytes; `40 01`
and `40 03` occur alongside `40 02`
- tagless segment headers: no `40 NN` tag at all, just the 14-byte
tail [field2:2][len:2][channel_id:4][marker:2][anchors:4]
Also: the header field documented as a "monotonic uint32 LE counter" is
really [channel_id][00][00][segment_index], with 0x46=Tran 0x47=Vert
0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero
disagreements. decode_waveform_v2 now takes the channel from that field
instead of rotation position, which was fragile: one missed header
desynced every channel after it.
parse_segment_header now returns n_prev_deltas/prev_deltas/marker/
anchors/channel/segment_index; the old fixed_pattern (02 00 00 01)
conflated the 2-byte marker with the first anchor.
Ground-truth corpus, end to end through the production path:
exact 37 -> 72, truncated 23 -> 3, full-length value errors 15 -> 0.
Store-wide, 729 of 1388 series-3 waveform events decode differently and
728 gain samples; the scale fix changes float values on all of them, so
stored .h5 files need regenerating.
Still open: 3 events truncate at a header variant with a variable-width
prefix (2/4/6 bytes) before the channel id and an `01 00` marker.
Documented in docs/instantel_protocol_reference.md with byte offsets.
+20 tests. No regressions: the byte-exact fixture suite still passes and
the full-suite failure list is unchanged from baseline (16 pre-existing
failures from gitignored fixtures).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
set_false_trigger now clears reviewed_real when flagging false_trigger=True
(mirrors update_event_review's exclusivity), and the quick
PATCH /db/events/{id}/false_trigger endpoint now calls
propagate_review_to_twins after the flag write, matching the sidecar PATCH
path's try/except-with-log.warning pattern. Previously the quick path could
leave both flags set and never touched twins.
Also corrects the v0.25.0 CHANGELOG bullet (exclusivity was not actually
enforced on the quick path until this commit) and adds a caveat comment on
find_twins about rare clamped/saturated-PVS false-positive twin matches.
7-task TDD plan: shape DSP module (crest factor + points-near-peak), shape_*
columns + auto-migrate, insert_events persistence, ingest population in the
save paths, backfill script, /db/events exposure + v0.24.0 bump. Phase B
(terra-view scoring/UI/review) gets its own plan once this feed is live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
Full-snapshot bundle (SFM side): document the /db/snapshot,
/db/waveforms/recent.zip and gated /db/restore endpoints. Bump package
version 0.21.1 -> 0.22.0 (pyproject), README version badge + history, and
CHANGELOG. TOOL_VERSION (codec/output provenance) left at 0.21.1 on purpose
— the event codec is unchanged, so existing sidecars must not be flagged
stale for re-backfill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WyT5m9xppYmoVGmCCVVV6X
Two gaps in backfill_thor_events.py that left old Thor events showing
stale charts after a v0.21.1 backfill pass:
1. IDFH events were skipped from .h5 regeneration (the "have decoded
samples" gate was IDFW-only). Histograms kept their pre-v0.21.1
.h5 — written from raw_samples = None, which the renderer turned
into a near-empty bar chart, or for older events the dB(L)-as-pseudo-
psi mic scale that produced "107.7 psi" peaks (atomic-bomb level
instead of footstep level). Fix: synthesise the same 1-sample-per-
interval array save_imported_idf v0.21.1 uses (peak ADC count per
channel per interval) so the renderer's bar-chart grouping has
data to work with.
2. The IDFW h5 path didn't merge binary_peaks.mic_pspl_psi onto the
IdfEvent before to_minimateplus_event(). The live save_imported_idf
does this merge — without it, IdfEvent.from_report() only sees the
.txt's dB(L) value, the bridge falls back to the dBL→psi formula
(instead of the binary-accurate 2.14e-6 psi/count value), and the
h5 writer's per-count mic factor lands on a less-correct value.
Fix: same merge the live ingest does (lift res.event.peaks.mic_pspl_psi
onto idf_event.peaks before the bridge call).
Verified against UM6047_20250804190047.IDFH (250-interval prod
histogram): 250 intervals decode, mic_pspl_psi = 2.78e-5 (was being
treated as dB(L)=107.7 in the old h5).
Operator: re-run after deploy. `docker compose exec sfm python
scripts/backfill_thor_events.py` is idempotent — the existing version
check still skips events already at the new TOOL_VERSION, and review
state + captured_at are preserved on the second pass.
serversdownandClaude Opus 5