The BW filename encodes only the serial NUMBER — `<letter><3 digits>` where
letter = chr(ord('B') + serial // 1000), so `L895…` decodes to 10895. The
two-letter family prefix is not in the filename at all, and every offset
scanner synthesized it as f"BE{num}".
Four of the 43 archive units are BA, not BE. Their binaries say so plainly:
BA9229, BA10060, BA10895, BA15957. Brian caught BA10895 by recognising that
no such unit as BE10895 exists.
serial_of() now reads the serial string out of the file body and falls back
to the old synthesis only when no matching string is found. No analysis
changes: grouping was by the numeric part, which was always correct, and no
unit number maps to more than one serial (checked across all 43).
The same assumption is live in two production sites and is NOT touched here,
because fixing ingest renames rows a running store and Terra-View already
reads them:
- sfm/waveform_store.py:870 `return f"BE{serial_num}"` on import
- minimateplus/client.py:2538 `raw_data.find(b"BE")` in the monitor-log
partial-record decode, which yields serial=None on a BA unit
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
offset_scan3.py covers only waveforms (6,577 unique binaries). The archive
also holds 63,535 unique histograms, which the pre-trigger method cannot
touch: a histogram carries no samples, only a per-interval per-channel peak.
scratch/offset_hist_scan.py scans them — 63,505/63,535 decoded (99.95%),
43 units, 77.9M intervals. It emits every candidate floor statistic per
(file, channel) rather than deciding anything, so thresholds get calibrated
against the waveform ground truth instead of guessed.
Journal §8b records the outcome. What survives is a site-quiet-gated
cross-channel differential that independently confirms BE18438|Vert and
BE9558|Tran+Long with a clean 2.5x separation gap and 0.037% day-level false
alarm, threshold-insensitive across a 2.3x span — the first operating point
in this investigation to pass that test cleanly.
What it does not do, recorded just as plainly: it finds 2 of the 5 confirmed
units, not 5. DC leakage into the interval peak is bimodal (0.9 on BE18438,
0.02 on BE12599), so a negative histogram result is not evidence of health.
Per-channel attribution is not established (channel-scramble p = 0.769) and
timing resolves to ~a month, not a day.
Two dead ends buried for good: the absolute floor is retired (66% of its
discrimination is a day/site confound), and zero-fraction is structurally
impossible — the device clamps every interval peak at >= 1 A/D count.
Two findings independent of the histograms:
- offset_scan3's spread<=0.02 gate discards 18.8% of rows with |pre|>=0.025,
concentrated on 41 unit-channels currently labelled clean; 4 would be
sustained positives without it. The fleet label is three-state, not two.
- The waveform corpus observes ~7% of the days a unit was deployed.
BE10895 is reclassified from transient to a genuine Vert fault of a different
subtype: 49.4% single-axis-dominant events, the highest in the fleet, all on
Vert. The other six marginal units are clean.
Not done: the 11 thin-coverage units were not screened, and no completeness
audit was run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
A reason records *why* an event is a false trigger. It is optional (plain
FT flags still record no reason) and is a subtype of the FT flag: setting a
reason implies false_trigger=1, and the reason is cleared whenever FT ends
up 0 (confirm-real, clear-FT, set_false_trigger(false)). Twin propagation
carries the reason to the histogram/waveform twin alongside the FT flag.
New nullable `false_trigger_reason TEXT` column (schema + _migrate ADD
COLUMN only — not the Migration-1 rebuild). 7 tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
Productionizes the validated scratch/offset_scan3.py: a DC offset (baseline
shifted off zero — sensor bumped/settled/drifted) is |median(pre-trigger)| >= 5
counts (0.025 in/s) AND flat across pre/mid/end thirds (spread <= 0.02); a
transient moves one third and is rejected by the spread test.
- shape_metrics: offset_from_samples / offset_from_h5 (reads .h5 samples +
pretrig_samples attr; range-aware via the .h5's in/s float samples)
- events schema: shape_offset / _axis / _pre / _spread (via _SCHEMA + the
_migrate ADD COLUMN loop only; NOT the Migration-1 rebuild), threaded through
insert + upsert mirroring shape_*
- ingest: computed at all three waveform_store save paths alongside shape
- backfill_event_shape: also computes + stores (and stale-clears) offset
- exposed via /db/events automatically (SELECT *)
Gating to waveforms is done downstream in terra-view ft_suspicion (mirrors how
shape is ignored for histograms), not at the SFM call sites. 13 new tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
The stack-level context (version pairing across seismo-relay / Terra-View /
SLMM, and which repo a change belongs in) now lives version-controlled at
terra-view/docs/tmi-stack.md, symlinked as ~/CLAUDE.md. Reference it here so
the three project docs are symmetric.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
The v0.27.0 notes said prod held 4 histograms that would stay empty until a
backfill. That was wrong, and asserted without checking.
Verified: all four recovered files (K440HJCN.3C0H, K557IF1U.8K0H,
T191HVNP.0S0H, T193L0XM.CI0H) are archive-only — none appears in the production
store or the events DB. Re-running stride detection over the prod store's
10,215 histogram binaries under both the old and new code shows 0 files whose
decode changes.
So the partial-final-block fix is forward-looking: it matters for future
ingests of sub-minute histograms with a partial final block, not for anything
already stored.
TOOL_VERSION still moves with the release, so a future backfill run will
regenerate the whole store instead of skipping. Harmless — byte-identical
output for every stored file — but it costs the full ~2 hours on the NAS, so
it should not be started casually.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Bumps pyproject, TOOL_VERSION, README and CLAUDE.md to 0.27.0. sfm/server.py
now derives its version from TOOL_VERSION (c8c4ec2), so that constant is the
single source of truth for the service version and the sidecar stamp alike.
What ships:
- histogram partial-final-block fix (4 files recovered, 0 regressed)
- interval-based find_twins matching (terra-view #102 sub-task 2)
- /health no longer reports a hard-coded 0.1.0
- 793 NUL bytes stripped from CLAUDE.md (made grep skip it as binary)
- docs/offset_investigation.md, and the offset detectors
- scratch/verify_against_ascii.py
Verification: the series-3 codec now decodes 14,338 / 14,338 archive pairs
exactly against their Blastware ASCII exports (1,249 waveform + 13,089
histogram, 45 units, back to 2018) — 11x the ground truth the prod store
carried, and it supersedes the old "per-sample on 11% of files" caveat.
Independent check on the scale: 19,244 healthy channel-events sit at a
pre-trigger floor of exactly 0.000 (62.7%), 94.5% within one quantisation
unit, median +0.0000. No zero-point bias in the decoder.
⚠ TOOL_VERSION moved, so the next prod backfill regenerates the whole store
(~2 hours on the NAS). That is intended — it is what publishes the 4 recovered
histograms — but it is not a no-op; schedule it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Brian's method, and better than v2's whole-record median: the pre-trigger
window is definitionally quiet (the buffer captured before the trigger fired),
whereas a median is merely robust to the event. Requiring the floor to hold
across pre-trigger / middle / end rejects transients that a median cannot.
per channel: pre/mid/end medians, spread = max - min
offset when |pre| >= floor AND spread <= 0.02 in/s
real fault >= 3 consecutive flagged events on that channel
The empirical noise floor justifies the threshold and validates the decoder:
across 19,244 non-flagged channel-events the pre-trigger floor is 62.7% exactly
0.000, 94.5% within +/-1 quantisation unit, median +0.0000, mean -0.0008. There
is no systematic zero-point bias — an independent confirmation of the
32000-count geo scale.
The result is threshold-insensitive across a 2x range (0.020 to 0.040 in/s),
which is what separates a real signal from a tuned one:
FINAL: 5 of 45 units (11%) — BE9558, BE11529, BE12599, BE13117, BE18438
BE11007 and BE10895 drop out; the spread test identifies them as transients
rather than pedestals. v1's 11% headline was right by luck — it included
BE11007 and named the wrong channel on most units.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Brian challenged the v1 finding that offsets "come and go", against field
experience that a unit which develops one stays broken until the geophone is
replaced. He was right; v1 had two flaws, both of which manufactured false
recoveries:
1. It scored only the axis with the largest peak, so a real event on one axis
hid a persistent pedestal on another. BE12599 on 2026-08-21 read "clean"
because Long had a 1.065 in/s event, while Tran sat at +0.4732 in/s and was
never examined.
2. It used the mean, which a real transient perturbs. The median is the resting
baseline and a blast does not move it. Same event, Long channel:
mean +0.0783 vs median -0.0050.
offset_scan2.py flags a CHANNEL when |median| >= 0.025 in/s (5 A/D counts,
Instantel's own criterion) and treats >=3 consecutive flagged events as the
real signal. No m/p ratio guard is needed — that existed only to compensate for
the mean.
Corrected results:
units with any flagged event 6 -> 19 of 45
units with a sustained pedestal 8 of 45 (18%)
runs >=3 consecutive 29; 1-2 event runs (noise) 69
Also corrected: the affected channel is most often Vert, not Tran (v1 named
whichever axis had the largest peak, so it was frequently wrong). BE10895 and
BE18003 were invisible to v1. BE12599's fault began 2026-08-14, not 08-17.
The decode itself was never in question and is confirmed against Blastware's
own ASCII export: on K558LJN3.BK0W, BW shows Tran parked at +0.265..+0.375 in/s
for the entire record while Vert and Long sit at ~0.005 — Instantel's "parallel
lines above or below the zero line".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
terra-view's SFM Admin page (/admin/sfm) displays whatever /health returns for
`version`. That was hardcoded to "0.1.0" and never bumped, so the page showed
0.1.0 while the service was actually 0.26.0. Point both /health and the FastAPI
OpenAPI version at the release-bumped TOOL_VERSION (single source of truth), so
they can't drift again. Adds httpx-free regression tests (call health() directly).
Note: minimateplus.__version__ is separately stale at 0.1.0 — left as-is here
(nothing user-facing reads it; touching the package __init__ risks import order).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
Adds docs/offset_investigation.md — a dated journal of the "offset" hardware
fault, in the style of the codec status docs: findings with provenance, dead
ends kept with the reason they died, and per-unit case files.
Contents:
- base rate 5-6 of 45 units (11-13%) over the DL2 archive, 2018-2026,
confirming rather than overturning the earlier 2-of-21 estimate
- the detector, with the rationale for each term and its known blind spot
(event traces carry real motion, so only trace-dominating offsets show)
- the bimodality result: relaxing the amplitude floor 11x adds no new units
- Instantel's own procedure and thresholds from their FAQs 13-0-21 / 12-0-10:
A/D-mode ">5 counts", the autozero key sequence, and the 2027-2069 X1/X8
acceptance window that explains the ~10% field success rate of a re-zero
- four ruled-out hypotheses, each with the evidence that killed it:
condensation, clipping, the sensor check as a predictor (102 offset events,
zero failures — a grossly offset unit passes its own self-check), and the
calibration-timing correlation (confounded, one unit per time bucket)
- open questions, chiefly whether SUB 0x0E carries the autozero numbers
Cross-referenced from CLAUDE.md and Appendix E of the protocol reference
(whose CRLF line endings are preserved).
Separately: CLAUDE.md had 793 NUL bytes appended after its last line. They
predate this work (present at least as far back as e42956a / v0.21.0) and made
grep treat the file as binary, silently skipping it. Stripped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
The DL2 export keeps a byte-identical `Sent/` copy of its root, so walking it
counts every binary twice: 127,035 histogram paths are 63,535 distinct files,
and 13,077 waveform paths are 6,577. offset_scan.py now keeps the first
occurrence of each basename.
Corrects the previous commit's changelog claim of 8 recovered files — it is 4:
K440HJCN.3C0H and K557IF1U.8K0H (stride 252), T191HVNP.0S0H (92), T193L0XM.CI0H
(612). Still zero regressions. The per-unit breakdown reading exactly 2-2-2-2
should have given the doubling away.
The 14,338-exact verification result is unaffected: ASCII exports are not
mirrored (14,340 paths, 14,340 distinct names), and the harness enumerates
those rather than the binaries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
detect_multi_interval_stride() confirmed a candidate stride on a third block
header whenever the body was long enough to contain one. But a body can exceed
two strides and still hold only two real blocks: a partial final block leaves
trailing padding. BE18193 T193L0XM.CI0H — 51 intervals at 2 s, i.e. one full
30-interval block plus a 21-interval remainder in a 2787-byte body — had every
decisive check pass at stride 612 (header at 0, header at 612, block counter
256 -> 257) and was then rejected for the absent third header at 1224. It
decoded to nothing.
A missing third header now means end-of-stream rather than disqualification.
The block-counter check is untouched — that is the test that prevents the
false positives which once handed 9,082 standard-block files to the
multi-interval walker.
Found by running the full DL2 archive against its preserved Blastware ASCII
exports (14,340 paired files, 11x the previous ground-truth corpus).
Measured over 127,035 archive histogram binaries:
recovered 8 files (strides 92, 252, 612; BE18193, BE18191, BE9557, BE9440)
regressed 0 files
Full-corpus verification: 14,337 -> 14,338 exact of 14,338 decodable pairs
(the 2 excluded are series-4 IDF, a different codec).
Also adds scratch/verify_against_ascii.py (per-sample decoder verification
against BW exports, with a saturation carve-out — BW clamps clipped events to
the range max while the decoder reports true counts) and scratch/offset_scan.py.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
A real trigger is recorded twice — as a triggered waveform (stamped at the
trigger instant) and inside the scheduled histogram whose interval contains it
(stamped at the 7am/7pm interval start). The two twins routinely differ by
HOURS, so the old ±5-minute window in find_twins silently missed them — which
broke review propagation (flagging one twin left its twin unflagged).
Twins are now matched by: same serial + identical peak_vector_sum + OPPOSITE
record type + the waveform's timestamp falling within the histogram's interval
(bounded by the next same-serial histogram). Matching keys off record timestamps
(not call-in/received times, which drift with field connectivity). window_seconds
is retained but ignored.
Rewrote test_find_twins + test_twin_propagation for the new contract (incl. the
75-min-apart UM12947 case, cross-type exclusion, containing-interval selection,
open-ended latest interval). Full suite: 264 passed; the 16 failures are
pre-existing (missing gitignored fixtures + a v0.26.0 codec case), unchanged
from baseline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
TOOL_VERSION had been frozen at 0.21.1 for four releases despite its own
comment saying "Bump this constant and CHANGELOG.md together at release
time". It is not cosmetic: backfill_sidecars.py decides whether to
regenerate with
ver_ok = sidecar.source.tool_version >= event_file_io.TOOL_VERSION
so with the constant stuck at 0.21.1 and every sidecar stamped 0.21.1,
a backfill WITHOUT --force skipped the entire store. That is precisely
the failure the check exists to prevent, and it means every sidecar
regenerated during the 0.26.0 decode work is stamped 0.21.1 while having
been produced by 0.26.0 code.
Verified: a non-force dry-run over the snapshot now reports
written=11603 skipped(uptodate)=0, where before it would have skipped
all 11,603. Prod therefore does not need --force to pick up the decode
corrections — the version difference alone is enough.
Note the installed dist metadata reads 0.12.0, older than the constant,
so the best-effort "prefer installed metadata when newer" path correctly
defers to TOOL_VERSION.
Also bumps the README header, which still read v0.22.0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Two body-model rewrites, a systematic scale error affecting every
geophone reading the system ever produced, a recovered file format, and
two artifact-hygiene bugs where stale files outlived the decodes that
made them.
- geo full scale is 32000 ADC counts, not 32768 (every reading 2.34% low)
- the waveform body is a record chain, not a tag stream
- the histogram block is big-endian, with a terminal tail
- sub-minute intervals pack several per block (415 files recovered)
- three more defects found by a full-corpus sweep, each masking the next
- stale .h5 files and stale shape_* columns are now cleared, not left
All 11,603 series-3 binaries in the production snapshot pass every check.
Ground truth: 1,211/1,211 histograms exact per-interval, 75/75 waveform
sample counts exact, multi-interval fixture exact on all 45,680 values.
Also corrects a changelog note that went stale within the same day: the
"3 of 75 events still truncate" item was resolved by the record-chain
rewrite, and the remaining open items are now listed explicitly.
CLAUDE.md gains a "Where things stand" block at the top — the header had
been reading v0.21.0, four releases behind, which is the first thing you
see when picking the project back up.
Tests: 259 passed; the 16 failures are pre-existing (gitignored
fixtures) and unchanged from baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Swept every series-3 binary with the live decoder against five
independent checks: decode exceptions, zero samples, unequal geo channel
lengths, peaks above range full scale, decoded peak vs device-reported
PPV, and waveform length vs declared record time.
1. block[22] is NOT a constant and must not be tested. Documented as
always 0x00, it carries data on loud blocks, and rejecting those threw
away the interval holding the event peak. BE18350/T350L7HR.NL0H
block 92 has block[22]=0x26 and a Tran peak of 0x0563 = 1379 counts =
6.895 in/s — exactly the device-reported PPV — while the file decoded
to 0.015 in/s. block[0]==0, block[4]==0x0A and the 4-byte tail are
six bytes of constraint, which is what keeps trailer content out.
2. Block-model dispatch now goes on signature strength rather than on
whichever decoder returns first. A multi-interval body also yields
scattered standard-tail blocks by coincidence, so "first non-empty"
handed 193 BE18193 files to the standard walker and produced peaks of
149 in/s against a 10 in/s full scale.
3. Multi-interval stride detection requires the block counter to
increment by exactly 1. Without it the detector false-positives on
ordinary standard-block bodies: they carry a header every 32 bytes,
and 192 = 12 + 20*9 and 512 = 12 + 20*25 are both multiples of 32, so
a stride "fits" while skipping 6 or 16 real blocks. That misrouted
9,082 files.
Partial-block garbage is trimmed within the final block only, stopping
at the first slot with a non-zero tail word or a geo peak above full
scale (2000 counts in 16-count units). Trimming purely from the end
left garbage stranded behind a slot that happened to have a zero tail
word; trimming on the tail word alone truncated four BE9440 files by up
to 2,800 intervals.
Result: 11,603 / 11,603 series-3 binaries clean on every check.
Ground truth unchanged: 1211/1211 histograms exact per-interval, 75/75
waveform sample counts exact (73/75 fully exact, the 2 differ by 1 LSB
on rail samples), and the multi-interval fixture still matches its BW
ASCII export on all 45,680 values.
Tests: 259 passed, failure list unchanged from baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Sub-minute histogram intervals are packed several to a block so that
every block still covers exactly one minute of data:
interval intervals/block stride
1 minute 1 32 <- the standard big-endian block
15 s 4 92
2 s 30 612
stride = 12 + n * 20
Block = [00][segment][ctr uint16 LE][0a][00], then n x 20-byte records of
8 x uint16 LITTLE-endian values (T_peak, T_halfp, V_peak, V_halfp,
L_peak, L_halfp, M_peak, M_halfp) plus a 2-word tail whose first word is
0000 on every real interval, then a 6-byte block trailer.
The standard 32-byte block is BIG-endian; this variant is LITTLE-endian.
The tail-word check matters: a session ending mid-block leaves buffer
garbage in the remaining interval slots, which decoded as peaks
thousands of times the real value. Stride detection also requires at
least 2 records, since a 1-record block would have stride 32 and
collide with the standard block.
Recovers 415 files that decoded to nothing: 216 on BE18193 (2 s
intervals) and 199 on BE9440 (15 s). Before decoding to nothing they
were being accepted by the WAVEFORM codec, which returned garbage
peaking up to 400x the device-reported PPV.
Ground truth BE9440/K440L3AQ.T70H (5,710 intervals) matches its
Blastware ASCII export exactly: 17,130/17,130 geo peaks, 22,840/22,840
frequencies, 5,710/5,710 mic dB(L). Across all 455 affected files,
1,354/1,365 channel peaks (99.2%) match the device-reported PPV; the 11
that don't are under-reads on BE9440 where the walk stops early.
Fixture (binary + ASCII) saved under tests/fixtures/, which is
gitignored per repo practice — the ground-truth test skips when absent.
Tests: 258 passed, failure list unchanged from baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
backfill_sidecars.py skipped the .h5 write when a file produced no
samples, with the stated intent of not replacing it with an empty
placeholder. That silently preserved output from a superseded decoder.
After the record-chain fix, 415 histogram files stopped decoding (216 on
BE18193, 199 on BE9440) but kept .h5 files whose peaks ran up to 400x
the device-reported PPV. Those were feeding charts and the
false-trigger detector with nothing marking them. The .h5 is now
removed in that case and the run reports stale_h5_removed.
Store-wide effect, series-3, decoded peak vs device-reported PPV:
waveform 1307/1307 (100%), mean abs ratio error 0.00000
histogram 4434/4435 (100%)
Both were 99% with a tail of 18 and 25 wrong files respectively.
The 415 files are a genuine unmapped format variant, not a regression:
their bodies open `00 00 00 01 0a 00` (valid block header, marker 0a at
[4], block_ctr 256) but block[28:32] matches neither known tail, and no
stride from 8 to 64 bytes places a marker at [4] consistently. Bodies
are very large (one is 360,573 bytes). They were previously being
decoded by the WAVEFORM codec, which accepted them and returned garbage
- so the gap pre-dates today's work; the fix only exposed it. Logged as
an open question in the protocol reference.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Supersedes the segment-header model entirely, including the fixes made
earlier today. Found via multi-agent structural analysis of the 25 files
that stalled the walker, then verified independently.
Records are self-delimiting: off+2 is a uint16 BE length, next_record =
off + 2 + len, and the chain ends on a record whose chan_id is 0x06.
off+8 carries a 3-valued mode enum:
02 00 14-byte header, 2 anchors, then CUMULATIVE delta blocks
01 00 10-byte header, no anchors, blocks are ABSOLUTE values
00 03 10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute
`40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header.
Reading it as a 2*NN + 16 header is what made walks drift — the
"variable-prefix segment descriptors" reported earlier today were not a
format feature, just walker drift of exactly
4 - (old_stop - true_record_start), on all 25 affected files.
Measured on the production snapshot:
all four channels equal length 156/1388 -> 1388/1388
ASCII sample-count exact 72/75 -> 75/75
ASCII fully exact 70/75 -> 73/75
device PPV waveform (live) 1288/1306 -> 1306/1306 (mean err 0.00000)
device PPV histogram (live) 4434/4459 -> 4458/4459
Also eliminates the walker-over-read class: 24 of those 35 files were
histograms that read_blastware_file fed to the waveform codec first; the
old walker accepted them and returned garbage (one yielded 98,923
"intervals"), while the record-chain decoder returns None so they fall
through to histogram_codec.
00 03 records are DECODED, not skipped. Skipping them silently shifts
the time base of everything after them on that channel — BE9558/
K558LOF2.820W had MicL displaced by exactly 512 samples with nothing
marking the gap.
Footer detection now prefers the 0e 08 candidate whose body yields a
chain terminating on 0x06; the signature can occur inside a sample
stream. Blast radius 1 file of 1388.
The superseded model survives as decode_waveform_legacy, pinned by
micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes
candidates and keeps whichever yields the most samples, so the new
decoder returning None where the old returned garbage changes that
heuristic's winner. Deferred until that search uses the record chain.
Tests: 253 passed (+11), failure list unchanged from baseline. The 9
tests pinning the superseded model are retargeted at
decode_waveform_legacy, which still implements it.
NOTE: stored .h5 files need regenerating — nearly all get longer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
backfill_event_shape.py skipped rows whose .h5 produced no shape and left
the previously stored value in place. A stale shape outlives the decode
it came from and silently feeds the false-trigger detector.
Found while re-running the backfill after the histogram codec fix: 493
rows in the prod snapshot were carrying shape metrics that no longer
matched their .h5 — e.g. BE17353/S353LDOK.XZ0H held crest_factor from a
223-sample decode while its .h5 holds a single interval. These predate
today's work (present in the pre-32000 snapshot), so this is pre-existing
behaviour rather than fallout from the codec fixes.
Now NULLs shape_crest_factor / shape_near_peak_count / shape_sample_count
/ shape_axis in that case and reports a `cleared_stale` count. Verified
on the snapshot: 493 cleared, 0 stale rows remaining, 11570 rows matching
their .h5 exactly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Two errors in the series-3 histogram block model, both found by diffing
against the per-interval data table in the preserved Blastware ASCII
exports (1211 files in the prod snapshot — far stronger ground truth
than the header PPV used previously).
1. The block is uniformly BIG-ENDIAN. Peaks and half-periods are uint16
BE (T_peak [5:7], T_halfperiod [7:9], V_peak [9:11], V_halfperiod
[11:13], L_peak [13:15], L_halfperiod [15:17], M_peak [17:19],
M_halfperiod [19:21]); only block_ctr [2:4] is little-endian.
The old uint8-peak model silently CLIPPED any peak above 1.275 in/s:
the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's
export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590.
The byte documented as a per-channel "annotation" was never an
annotation — it is the half-period's high byte, which is exactly why
it was non-zero on the sub-Hz intervals BW renders as "<1.0".
The marker is block[4] alone. Testing [4:6] as a uint16 LE marker
forced block[5] == 0, which is what capped the peak at one byte.
2. The final block of each stream carries tail 9c 06 00 42 instead of
1e 0a 00 00, and holds arbitrary bytes at [21:23]. Rejecting it
dropped the last interval of nearly every histogram — frequently the
interval holding the event peak, so the file's PPV read low.
Verified end to end through the production path: 1211/1211 histograms
decode exactly (interval count + every per-interval peak), plus 842,442
per-interval frequency comparisons with zero mismatches. Previously
1 of 1196 files was fully correct.
decode_histogram_body_full records expose `is_terminal` in place of the
removed `annotations` tuple. +6 tests. No regressions: full-suite
failure list unchanged from baseline.
NOTE: stored histogram .h5 files need regenerating to pick this up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Re-measured properly. The first pass compared h5 max against the ASCII
header PPV and reported "26% of channels miss the peak". The histogram
ASCII actually carries a full per-interval data table (Tran/Vert/Long
peak + freq + PVS per interval), which is real ground truth, so the
comparison should have been per-interval from the start.
Per-interval result, n=1196 series-3 histograms:
- decoded VALUES are right: 1031/1196 (86%) match within 1 LSB across
the overlapping prefix
- the interval COUNT is short in 1195 of 1196 files: median 1 missing,
1088 short by 1-2, 65 by 3-10, 39 by 11-100, 3 by >100 (max 205)
- decoded max falls below the device PPV in 169/1196 files (14%), not
26% — that happens when a dropped interval held the peak
So it is a termination bug in histogram_codec.decode_histogram_body,
the same family as the waveform-walker truncation fixed earlier today,
rather than mis-decoded interval values. Series-3 only; there is no
preserved series-4 ASCII in the snapshot to compare against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Operator report: attaching a different geophone to an affected unit
makes the offset go away. That rules out the unit's analog front-end
and any stored per-channel zero constant (a constant lives in the unit
and would survive a sensor swap).
The stored data agrees — MicL, a separate transducer on its own cable,
shows no offset during either episode (|mean|/peak 0.17 and 0.02) while
the geo channels on the same unit at the same moment are pinned.
Two distinct sensor-side patterns recorded:
BE18438 Vert 0.97, Tran 0.16, Long 0.18 -> one conductor pair
BE9558 Long 0.99, Tran 0.90, Vert 0.81 -> shared return / ground
Candidate mechanisms narrowed to three, since a geophone coil is passive
and cannot generate sustained DC: galvanic corrosion at a connector or
splice (matches the ~46 mV referred to the ADC input), a leakage path to
shield, or changed coil DC resistance interacting with the amplifier's
input bias current.
Also records the confound: swapping a sensor requires a monitoring
restart, and these units run Sensor Check "Before monitoring", so the
restart re-zeros too. The swap does not cleanly separate "new sensor"
from "the restart re-zeroed it". Controls and the single best
measurement (open-circuit DC across the suspect connector) documented.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Brings the protocol reference, CLAUDE.md and the codec RE status doc up
to date with everything confirmed in this pass.
instantel_protocol_reference.md
- Changelog row for the five findings.
- S7.6.1: scope table showing the 32000 scale correction applies to
series-3 waveform, series-3 histogram and series-4 Thor alike, with
the measured before/after ratios for each.
- S15: closed "Full channel ID mapping in SUB 5A stream" — resolved by
the segment-header channel id ([channel][00][00][segment], 0x46=Tran
0x47=Vert 0x48=Long 0x49=MicL, 1697/1697 verified). Four new open
questions: variable-prefix segment descriptors, the histogram codec
missing peak intervals (26% of channels), UM-series IDF decoding
~1000x low, and the Thor per-count LSB residual.
- NEW Appendix E — Known Device Faults. Documents the field-observed
"offset" fault: symptom, why it floods the ACH queue (pedestal
exceeds the unit's own geo trigger level), the episode table, the
detection rule that works, what the data rules out (not the
geophone, not the battery, not environmental, not condensation),
and the two remaining candidate mechanisms with the test that
separates them. Explicitly flags that it is NOT a decode artifact,
since that mistake has already been made once.
CLAUDE.md
- Body-codec section: the four framing cases and the channel-id
finding, with the corpus result.
- "What's NOT solved": replaced the stale walker-edge-cases bullet
with the four genuinely open items.
waveform_codec_re_status.md
- Scale scope table matching the protocol reference.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
274 series-3 waveform events across 6 episodes on 2 units (BE9558,
BE18438) whose dominant geo axis sits pinned at a DC offset above that
unit's own geo trigger level — the "offset" hardware fault that makes a
unit retrigger continuously and flood the ACH queue.
Detection rule: dominant-axis |mean|/peak > 0.7 AND |mean| >= 0.9 x the
unit's geo trigger level. Bare |mean|/peak is useless on quiet events —
a trace at the 0.010 in/s noise floor clears any ratio threshold.
Not a decode artifact: these reproduce exactly in Blastware's own ASCII
export. Kept as the starting point for the archive-wide analysis.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
The scale lives in _samples_to_float, which every event passes through
regardless of source codec, so waveforms, histograms and Thor IDF events
were all 2.34% low — not just waveforms. Verified after regeneration:
series-3 histogram peaks vs ASCII reports now median 1.0000 across 1137
comparisons (0.9766 under 32768); series-4 peaks vs device peaks moved
from median 0.960 to 0.983 across 1468.
The four block-framing fixes remain waveform-only; histogram_codec is
untouched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
Two independent bugs, both found by diffing 75 production events against
their preserved Blastware ASCII exports (<store>/<serial>/<file>_ASCII.TXT).
1. Geo full scale was wrong — every geophone reading was 2.34% low.
The codec emits geo samples in 16-count units with a documented LSB of
exactly 0.005 in/s, and decoded_to_adc_counts multiplies by 16, so one
ADC count is 0.005/16 in/s and 10.000 in/s is 10.0/(0.005/16) = 32000
counts. sfm/event_hdf5.py and minimateplus/event_file_io.py both
divided by 32768 (2^15), scaling every sample and derived peak down by
1 - 32000/32768. The error scales with amplitude, so it was invisible
on quiet events and worst on the loud ones that matter for compliance.
Mic is unaffected (it back-solves its scale from the device peak).
216 per-channel comparisons: 32768 -> 151/216 exact, worst error 0.238
in/s on a 10 in/s event; 32000 -> 216/216 exact, worst 0.005 = 1 LSB.
2. walk_body silently truncated channels on four unhandled framing cases.
An unrecognised tag ends the walk and decode_waveform_v2 returns
whatever it got, so this surfaced as short channels, never an error:
- wide-NN RLE `0X NN` (runs longer than 252 samples)
- `30 NN` with NN > 0x10 (the old cap was arbitrary)
- variable-width `40 NN` headers: NN counts previous-channel
continuation deltas, so the header is 2*NN + 16 bytes; `40 01`
and `40 03` occur alongside `40 02`
- tagless segment headers: no `40 NN` tag at all, just the 14-byte
tail [field2:2][len:2][channel_id:4][marker:2][anchors:4]
Also: the header field documented as a "monotonic uint32 LE counter" is
really [channel_id][00][00][segment_index], with 0x46=Tran 0x47=Vert
0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero
disagreements. decode_waveform_v2 now takes the channel from that field
instead of rotation position, which was fragile: one missed header
desynced every channel after it.
parse_segment_header now returns n_prev_deltas/prev_deltas/marker/
anchors/channel/segment_index; the old fixed_pattern (02 00 00 01)
conflated the 2-byte marker with the first anchor.
Ground-truth corpus, end to end through the production path:
exact 37 -> 72, truncated 23 -> 3, full-length value errors 15 -> 0.
Store-wide, 729 of 1388 series-3 waveform events decode differently and
728 gain samples; the scale fix changes float values on all of them, so
stored .h5 files need regenerating.
Still open: 3 events truncate at a header variant with a variable-width
prefix (2/4/6 bytes) before the channel id and an `01 00` marker.
Documented in docs/instantel_protocol_reference.md with byte offsets.
+20 tests. No regressions: the byte-exact fixture suite still passes and
the full-suite failure list is unchanged from baseline (16 pre-existing
failures from gitignored fixtures).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
set_false_trigger now clears reviewed_real when flagging false_trigger=True
(mirrors update_event_review's exclusivity), and the quick
PATCH /db/events/{id}/false_trigger endpoint now calls
propagate_review_to_twins after the flag write, matching the sidecar PATCH
path's try/except-with-log.warning pattern. Previously the quick path could
leave both flags set and never touched twins.
Also corrects the v0.25.0 CHANGELOG bullet (exclusivity was not actually
enforced on the quick path until this commit) and adds a caveat comment on
find_twins about rare clamped/saturated-PVS false-positive twin matches.
7-task TDD plan: shape DSP module (crest factor + points-near-peak), shape_*
columns + auto-migrate, insert_events persistence, ingest population in the
save paths, backfill script, /db/events exposure + v0.24.0 bump. Phase B
(terra-view scoring/UI/review) gets its own plan once this feed is live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
Full-snapshot bundle (SFM side): document the /db/snapshot,
/db/waveforms/recent.zip and gated /db/restore endpoints. Bump package
version 0.21.1 -> 0.22.0 (pyproject), README version badge + history, and
CHANGELOG. TOOL_VERSION (codec/output provenance) left at 0.21.1 on purpose
— the event codec is unchanged, so existing sidecars must not be flagged
stale for re-backfill.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WyT5m9xppYmoVGmCCVVV6X
Two gaps in backfill_thor_events.py that left old Thor events showing
stale charts after a v0.21.1 backfill pass:
1. IDFH events were skipped from .h5 regeneration (the "have decoded
samples" gate was IDFW-only). Histograms kept their pre-v0.21.1
.h5 — written from raw_samples = None, which the renderer turned
into a near-empty bar chart, or for older events the dB(L)-as-pseudo-
psi mic scale that produced "107.7 psi" peaks (atomic-bomb level
instead of footstep level). Fix: synthesise the same 1-sample-per-
interval array save_imported_idf v0.21.1 uses (peak ADC count per
channel per interval) so the renderer's bar-chart grouping has
data to work with.
2. The IDFW h5 path didn't merge binary_peaks.mic_pspl_psi onto the
IdfEvent before to_minimateplus_event(). The live save_imported_idf
does this merge — without it, IdfEvent.from_report() only sees the
.txt's dB(L) value, the bridge falls back to the dBL→psi formula
(instead of the binary-accurate 2.14e-6 psi/count value), and the
h5 writer's per-count mic factor lands on a less-correct value.
Fix: same merge the live ingest does (lift res.event.peaks.mic_pspl_psi
onto idf_event.peaks before the bridge call).
Verified against UM6047_20250804190047.IDFH (250-interval prod
histogram): 250 intervals decode, mic_pspl_psi = 2.78e-5 (was being
treated as dB(L)=107.7 in the old h5).
Operator: re-run after deploy. `docker compose exec sfm python
scripts/backfill_thor_events.py` is idempotent — the existing version
check still skips events already at the new TOOL_VERSION, and review
state + captured_at are preserved on the second pass.
IDFH now synthesises a 1-sample-per-interval array from the binary intervals and writes an .h5 so the existing renderer works unchanged. Each "sample" is the per-interval peak ADC count → h5_value = count × geo_fs/32768 yields the right bar height.
Refreshes the bw_report sidecar block + .h5 waveform files for Thor
events ingested before the v0.21.0 adapter wiring + the bee1185 codec
fix. Those events landed with extensions.idf_report only (no
bw_report, no .h5 for IDFW) — symptom on the UI side: the modal chart
404'd on /waveform.json and the PDF rendered from DB-only fields
without sensor self-check, full per-channel breakdown, or mic dB(L).
Walks <store>/<serial>/<filename>:
- Reads the existing sidecar (preserves review state + captured_at)
- Re-runs read_idf_file() on the binary bytes (passes data=
kwarg so codec doesn't try the broken bare-path Path.read_bytes)
- Reads extensions.idf_report from the existing sidecar
- Runs build_bw_report_from_idf adapter
- Writes refreshed sidecar with bw_report + bumped tool_version,
preserving review block and original captured_at
- For IDFW: regenerates .h5 by bridging IdfEvent.from_report ->
to_minimateplus_event -> write_event_hdf5 (mirrors save_imported_idf
steps 4-7)
- IDFH events skip .h5 (histograms have no per-sample data)
Skips events already at current TOOL_VERSION with bw_report present.
--force overrides. --skip-hdf5 limits to sidecar-only refresh.
--dry-run for preview.
Validated against the prod-snap waveform store: 3,815 Thor sidecars
refreshed cleanly with 0 errors, 462 IDFW .h5 files written, 2 skipped
(binaries with no sidecar — backfill doesn't conjure events from
nothing). Verified one originally-broken IDFW event now serves
waveform.json (200, 168KB) and a fully populated PDF (119KB vs the
previous 56KB sparse output).
Operator workflow on prod:
docker exec <sfm-container> python3 /app/scripts/backfill_thor_events.py --dry-run
# Inspect counts, then for real:
docker exec <sfm-container> python3 /app/scripts/backfill_thor_events.py
Idempotent — re-running it is a no-op once everything's at the current
TOOL_VERSION.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related fixes to the per-channel stats block:
1. Pin the stats table's position via an explicit bbox= on
ax.table() so the bottom edge is at a known axes-fraction Y.
The previous loc="upper left" + tbl.scale(1, 1.4) combo let
matplotlib choose row heights based on text size, which made the
table extend further below the axes than the hard-coded PVS line
at y=-0.08 expected. Result was the "Peak Vector Sum X in/s"
string landing horizontally inside the Peak Displacement row.
With bbox=[0, 1-N*0.12, 0.80, N*0.12] the table is pinned to a
precise rectangle (12% axes-fraction per row × N rows tall).
_draw_stats_table now stashes the bottom Y on the axes for the
PVS helper to reference, so the geometry stays in sync.
2. Center PVS horizontally (ha="center" at x=0.5 instead of ha="left"
at x=0). The previous left-edge alignment put PVS at the same
X as the label column, which read as "off-center" once the rest
of the stats data was column-aligned further right.
3. Drop the "NA: Not Applicable" caption. It existed to explain
"—" placeholder cells, but "—" is universally understood and the
caption was always visually squished against the PVS line below.
Less cruft on the page; one fewer position to manage.
Verified against a real BE12599 histogram event (5 data rows) and
a real UM12947 IDFW waveform event (6 data rows) — both layouts
clear the table cleanly with no overlap.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bug shipped in v0.21.0: save_imported_idf called read_idf_file()
with `source_path` (a bare filename like "UM12947_….IDFW") BEFORE
writing the binary to disk. The codec did Path(path).read_bytes()
which resolved relative to /app and hit FileNotFoundError. The
error was caught + logged as a warning, and ingest fell back to
.txt-only — events still landed in the DB but lost the bw_report
block + .h5 waveform that the codec was supposed to produce.
Observed during a full re-forward from thor-watcher on 2026-05-29:
every Thor event logged "binary codec failed for X: [Errno 2] No
such file or directory" and got binary_decoded=False.
Fix:
- read_idf_file() gains a `data: Optional[bytes]` kwarg. When
supplied, skips the disk read and decodes the provided bytes
directly. `path` stays required (used for filename in error
messages + .IDFH vs .IDFW suffix detection); only the read is
conditional. Backward compatible — existing positional callers
(CLI scripts, tests) continue to work unchanged.
- save_imported_idf passes `data=idf_bytes` since the bytes are
already in memory from the multipart upload. Filesystem write
still happens at step 5 of the existing flow; codec just no
longer depends on it.
Verified end-to-end against UM11719_20231219162723.IDFW from the
example-data corpus: ingest endpoint returns inserted=1, log line
shows binary_decoded=True + h5=...IDFW.h5, no warnings.
Re-forward existing Thor events from thor-watcher after deploy to
backfill the bw_report block — UPSERT preserves review state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matches Terra-View's event-modal relabel from the same iteration.
Wording was already clearer here than in Terra-View's "Captured at",
but using identical text across both surfaces means operators see the
same label whether they're in the native modal or the standalone
webapp.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents two commits that landed on dev since v0.20.0:
9b71ead series 4 codec work, initial decode success
micromate/idf_file.read_idf_file() decodes both IDFW
(waveform; 87-99% sample fidelity reusing
decode_waveform_v2 at offset 0x0f1f) and IDFH (histogram;
dedicated segment-based decoder, all 859 corpus files
decode, 181,071 intervals total).
9fd52dd feat: add thor report generation, pdf generation
micromate/idf_to_bw_report.py adapter projects parsed
Thor data into the bw_report sidecar shape so Thor
events flow through sfm/report_pdf.py without a
separate renderer. Wired into save_imported_idf.
Net effect: a Thor event ingested via /db/import/idf_file now
lands with the same fidelity as a BW event, gets a per-event PDF
on demand, and renders in Terra-View's modal chart using the same
plotting code as a BW event.
Roadmap items closed:
- Binary .IDFW / .IDFH codec (was pending)
- Series IV (Thor IDF) binary codec reverse-engineering
Companion: Terra-View v0.13.0 ships in parallel and closes Phase 1
of the SFM integration. No API changes in seismo-relay for that
piece — Terra-View just consumes existing endpoints better.
Bumps:
- pyproject.toml 0.20.0 → 0.21.0
- minimateplus.event_file_io.TOOL_VERSION 0.20.0 → 0.21.0
(any subsequent backfill_sidecars.py --force will re-stamp
existing sidecars; expected + harmless)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes out the Event-Report PDF iteration started in v0.17.x and ships
the parser fixes the real-world events were tripping over.
Today's additions on top of the pre-v0.20 unreleased body:
- Server-wide display TZ via the TZ env var (default America/New_York
on prod). Affects server logs, the PDF report's "Created" footer,
matplotlib datetime axes. DB columns stay UTC. Dockerfile now
installs tzdata.
- ZC Freq "above-range" handling — parser stores 100.0 +
zc_freq_above_range flag for BW's ">100 Hz" marker. Renders as
>100 in the PDF stats table, both modals (inline on webapp Peaks,
new column on event-browser table).
- scripts/backfill_sidecars.py --reparse-txt — re-runs the current
parser against the preserved _ASCII.TXT and overwrites the
sidecar's bw_report block. Lets parser fixes reach old events
without re-forwarding. Validated end-to-end against ~10k prod
events.
Fixes shipped today:
- histogram_interval_size_s missing from ReportData → every
histogram PDF render 500'd.
- Histogram PDF geo channels now share a nice-quantized y-axis
(0.005-LSB-aware 1-2-5 step sequence) instead of auto-scaling
per channel + inventing sub-LSB "0.003 in/s/div" footer labels.
Roadmap delta: closes the BW ASCII parser "PPV-miss on some TXT
formats", "histogram-specific structural fields", and ">100 Hz value
parsing" items. Adds a new entry for the byte[5]==0 histogram body
sub-format observed on S353 events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The existing backfill_sidecars.py PRESERVES the bw_report block across
regenerations — it's treated as the source of truth from the original
ingest pass (the .TXT isn't reachable from the script's normal data
path, so it can't be re-derived).
That means parser-side fixes (like the 2026-05-28 ">100 Hz" ZC Freq
addition) won't reach old events even with --force. The new
--reparse-txt flag fixes that: when the sidecar's source.txt_filename
points at a preserved <serial>/<filename>_ASCII.TXT, the script re-runs
the current parser against it and overwrites the bw_report block.
Implies sidecar regeneration on every event (bypasses the
sha-up-to-date / version-up-to-date skip), so that the .h5 cascade-
regenerates alongside. No-op for events without a preserved .TXT
(legacy ingests pre-2026-05-27). Idempotent — re-running it produces
the same sidecar bytes when the parser hasn't changed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The PDF report shows per-channel ZC Freq alongside PPV in the stats
block, but neither modal exposed it. Now that the sidecar projection
carries zc_freq_hz + zc_freq_above_range, plumb them through:
- sfm_webapp.html: inline suffix on existing Peaks cells, e.g.
"Tran 0.04500 in/s · >100 Hz". Empty suffix when no ZC is
available (legacy events without a preserved .TXT).
- event_browser.html: new ZC Freq column on the per-channel stats
table. Required adding a parallel sidecar fetch in loadEvent()
(waveform.json alone doesn't carry bw_report). Fetch failure is
non-fatal — falls back to "—" in the new column.
Above-range ZC peaks (BW ">100 Hz") render with a literal ">"
prefix mirroring the PDF, so operators don't have to generate the
PDF to see when a channel hit the zero-crossing ceiling.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BW writes ">100 Hz" for ZC Freq when the zero-crossing algorithm sees a
peak too fast to count — the device's reporting ceiling is 100 Hz on
V10.72. Our parser fell back to None via _parse_number (which requires
a leading digit), so the PDF rendered "—" where BW shows ">100".
Mirrors the OORANGE/saturated pattern already used for PPV and PSPL:
parser stores the threshold (100.0) on zc_freq_hz + sets a new
zc_freq_above_range flag. Projection carries the flag through to the
sidecar; PDF renderer prepends ">" when set.
Affects both per-channel stats tables (waveform + histogram variants)
and the mic block's ZC Freq row.
Verified on the real T190LD5Q.LK0W fixture: Tran zc_freq_hz=100.0
above_range=True; Vert/Long (normal values) above_range=False; "N/A"
still produces zc_freq_hz=None which renders as "—" (unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related visual bugs on histogram PDFs:
1. Per-channel auto-scale meant Tran/Vert/Long had different y-axes
(e.g. 0-0.015, 0-0.025, 0-0.020) — bars looked taller on the
channel that happened to be quietest. Not directly comparable.
2. Footer "Amplitude Geo: X in/s/div" was just amax/5 of the FIRST
geo channel with data, with no LSB quantization — producing
nonsense like 0.003 in/s/div when the geophone LSB is 0.005.
Fix: compute a single shared geo y-axis range from max(Tran,Vert,Long),
quantize the per-division step to BW's 1-2-5 sequence rounded to the
0.005 LSB (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, ...), apply the same
ylim + ticks to all three geo subplots, and use that same step for the
footer label. MicL stays on its own auto-scale (different units).
Verified across edge cases including the reported event
(geo max 0.025 → 0.005/div, top 0.025), small PVS events, and large
blast amplitudes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The histogram-interval-times derivation block at line 314 references
rd.histogram_interval_size_s, but the field wasn't declared on the
ReportData dataclass — only the string form histogram_interval_size
was. Result: every PDF render of a histogram event raised
AttributeError → 500 from /db/events/{id}/report.pdf.
Cause: when the histogram aggregation block was inlined into
gather_report_data, the seconds-numeric counterpart that the
projection already carries (bw_report.histogram.interval_size_s) was
never wired into the dataclass. Waveform PDFs weren't affected
because the offending line is gated on is_histogram.
Fix: add the field, read it from the projection alongside the other
histogram keys. No-op for waveform events (the field stays None and
the gate skips it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Observed in fresh ingest logs on 2026-05-28: BE17353 events
(S353L4H2.FZ0H, S353L4H2.P00H, etc.) cause "body codec failed to
decode" warnings. Different from the byte[5]!=0 case already tracked
(T190 / O121) — these have byte[5]==0x00 with what looks like a
valid block header, but the walker finds zero data blocks anyway.
Operational impact identical to the existing case: ingestion
succeeds, DB peaks come from bw_report overlay, only the chart is
empty. No data loss.
Pinning so it doesn't get lost — needs a hex dump of one body to
work out what's different about these.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
User-reported issue: server logs were timestamped in UTC ("05:36:20"
when local was ~01:36 EDT), and the PDF report's "Created" footer
similarly showed raw UTC. Inconsistent with the modal which already
converts to browser local via toLocaleString.
Solution: standard Linux TZ env var. Set once in the container, and:
- Python's datetime.now() uses local
- Logging module's timestamps use local
- matplotlib renderers + report_pdf formatters use local
- astimezone() conversions resolve to the configured TZ
DB columns stay UTC (created_at uses SQLite's strftime('%Y-...Z', 'now')
which is always UTC, regardless of TZ env var — proper "store UTC,
display local" pattern).
Changes:
- Dockerfile: install tzdata (python:3.11-slim omits the timezone
database), set default TZ=America/New_York
- sfm/report_pdf.py: _fmt_iso_to_bw and _split_iso_to_date_time now
convert UTC inputs (Z-suffixed) to local via astimezone(); naïve
inputs (BW recorded-at, already unit-local) returned as-is.
New _to_display_local helper centralizes the logic.
- "Created" line in the PDF page footer now uses the converted
timestamp.
Override per-deployment via the TZ env var in docker-compose
(separate commit on terra-view side).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
_cleanup_event_files() removes the on-disk artifacts when an event is
hard-deleted (binary, a5_pickle, sidecar, h5). Today's .TXT
preservation feature added a new on-disk file (_ASCII.TXT next to the
binary) but the cleanup didn't know about it — so any event deleted
via /db/events/{id} (single) or /db/events/delete_bulk (or the
Terra-View "SFM Event DB Manager" UI which proxies through to those
endpoints) was leaving orphan .TXT files in the store.
Added "txt" to the cleanup list using the new
WaveformStore.txt_path_for(). Safe for old events without a .TXT —
the exists() check skips the unlink.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues spotted on a histogram event PDF:
1. Footer scale ("Time — /div Amplitude Geo: X in/s/div Mic: Y
psi(L)/div") was overlapping horizontally with the x-axis tick
labels (0, 20, 40, 60...). Both rendered on the same Y row.
Fix: bumped gridspec bottom margin from 0.06 → 0.12, moved the
footer text from y=0.045 → y=0.030 (below the tick labels), moved
the page-bottom Created/Event line from y=0.015 → y=0.005.
Trigger legend on waveforms moved 0.030 → 0.018. Everything
stacks cleanly now without collision.
2. PDF was showing the raw codec output (~150+ bars per histogram)
instead of BW's per-interval aggregation. Why: the aggregation
I'd added to /db/events/{id}/waveform.json wasn't replicated in
the PDF gather path. Now: gather_report_data does the same
max-per-group aggregation when bw_report.histogram.n_intervals is
populated, AND derives per-interval HH:MM:SS labels from the
start time + interval_size_s. Result: histogram PDFs now match
BW's display (one bar per BW interval, x-axis labeled with actual
times) — same fix as the modal chart, applied to the PDF.
For events ingested BEFORE the parser extension (no histogram block
in their sidecar), aggregation is a no-op — they still render with
per-block bars + interval-index x-axis (but the overlap fix applies
to them too). Re-forwarding repopulates the histogram block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Quiet histogram events were filling the chart panel even though the
peak was tiny (0.005 in/s rendered as 90% of chart height because
Chart.js auto-scaled to peak * 1.1). Made everything look uniformly
loud regardless of actual amplitude.
BW's solution: a near-fixed scale per channel ("Geo: 0.002 in/s/div"
from the footer). Quiet events render small, loud events render
proportionally tall.
Match the intent without copying BW's "no Y-axis labels at all"
convention. For histogram channels:
Geo (in/s): min Y range 0.05 in/s
Mic in psi: min Y range 0.001 psi
Mic in dBL: unchanged (the 60 dBL floor + peak+5 top already
gives quiet events a sensible baseline)
So a 0.005 in/s geo event renders as ~10% of chart height; a 0.05
event fills it; a 5.0 event still fills it (max(peak*1.1, 0.05) ==
peak*1.1 for any peak > 0.045).
Waveform charts unchanged — they should zoom for shape detail.
Applied to both the modal in sfm_webapp.html and the standalone
/events page in event_browser.html.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BW's Event Report PDFs include a per-channel sensor-check response
waveform on the right side of the bottom plot (damped sinusoid for
geo channels, sawtooth-at-test-freq for mic). Looks like real
per-sample data extracted from the binary, not synthesized.
Our parser captures the test results (freq, ratio, amplitude,
pass/fail) but not the waveform samples — so the report shows text
only for sensor check. Pinning a roadmap entry to investigate the
binary for the sample data (path a) or fall back to synthesized
visualization (path b).
Current text-only display is operationally sufficient.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues spotted in the modal:
1. Mic dBL chart looked spikey/discontinuous — isolated bars at 80-95
with gaps in between. Cause: _psiToDbl() returns null for zero or
negative samples, and most mic samples on a quiet event sit at the
digitization noise floor where they're effectively zero. Result:
the chart only renders the moments when instantaneous SPL exceeded
the Y-axis bottom — looks like a sound trigger gate.
Fix: new _psiToDblForChart() rectifies the AC waveform (abs), then
converts to dBL, then floors at MIC_DBL_FLOOR=60 dBL. Chart now
has a continuous 60 dBL baseline with peaks above it — matches how
acoustic engineers expect SPL-vs-time. Y-axis bottom pinned to
MIC_DBL_FLOOR, top to peak + 5 dB headroom. Peak label still uses
the unrectified _psiToDbl so the displayed peak value is exact.
2. Filename in Source/Files block was unlinked. Endpoint exists
(/db/events/{id}/blastware_file) — just wasn't wired to the modal.
Made it a clickable download link. Same treatment for the
preserved .TXT — added "(download .TXT)" link next to source kind
when source.txt_filename is populated (events ingested after the
.TXT preservation feature landed; older events show no link).
Applied to both the inline modal in sfm_webapp.html and the
standalone /events page in event_browser.html.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spotted comparing our PDF to BW's reference for T003LLUB.CE0H:
- Finish blank
- Per-channel Date / Time rows all dashes
- MicL PSPL line missing "on May 27, 2026 at 06:19:14"
- Peak Vector Sum missing "on May 27, 2026 At 06:06:14"
Root cause: I'd added these fields to the projection (write side) in
_bw_report_to_dict but never wired them into gather_report_data
(read side). Plus the projection used keys "start"/"stop" while
gather was reading "start_str"/"stop_str" — typo'd lookup.
Fixes:
- gather_report_data now reads bw_report.histogram.start /
.stop / .channel_peak_when (correct keys, matching the projection)
- Per-channel "peak_date" / "peak_time" populated from
channel_peak_when[<channel>] for the histogram stats table
- MicL PSPL line formats as "PSPL 125.7 dB(L) on May 27, 2026
at 06:19:14" (BW style) when channel_peak_when["MicL"] is present;
falls back to the waveform-relative "at 0.012 sec" otherwise
- PVS line formats as "Peak Vector Sum 0.091 in/s on May 27, 2026
At 06:06:14" (BW style) when bw_report.peaks.vector_sum.when is
populated; falls back to the relative time_s for waveforms
- New _split_iso_to_date_time() helper splits ISO timestamps into
BW-formatted ("May 27 /26", "06:06:14") date+time pairs for the
stats table's separate Date and Time rows
Events ingested BEFORE the parser extension landed (most of the
existing prod corpus) still show dashes — their sidecars lack the
histogram block. Re-forwarding repopulates.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spotted on the SFM webapp event modal — "Received by server at" was
showing the raw ISO string "2026-05-27T21:59:57.213043Z" because we
were assigning ev.timestamp / src.captured_at directly to the
textContent of the modal fields, bypassing the existing _fmtTs()
helper that wraps them in toLocaleString().
Net effect for operators: confusing "21:59 vs it's 6 PM" mismatch
when the displayed UTC timestamp didn't match wall-clock time. The
values were always correct; the display was just ambiguous.
After this fix:
- "Recorded at" (naive ISO from BW = unit local time) renders
cleanly as the unit wrote it: "5/27/2026, 6:00:13 AM"
- "Received by server at" (UTC with Z suffix) converts to browser
local: "5/27/2026, 5:59:57 PM"
- Timestamp column in the history table already used _fmtTs —
unchanged
- Same fix applied to the standalone /events page (sidebar event
list + meta header) via a new _fmtTsLocal helper
Note: did NOT add file-mtime-on-watcher-PC tracking as a separate
"Called in at" column — discussed and decided created_at is close
enough for schedule-compliance monitoring (worst case lag = watcher
poll interval ~60s, indistinguishable from BW write time at the
operationally-relevant resolution).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BW writes "OORANGE" (truncation of "Out Of Range") when a channel
exceeds its full-scale, and uses a typo'd label "Peak Vector Sum
TimeSum" for the PVS time field. Both confirmed against real ASCII
files pulled from a Windows watcher PC 2026-05-27:
T190LD5Q.LK0W Vert PPV = OORANGE (Normal range, 10 in/s exceeded)
T438L713.RY0W All three PPVs OORANGE (Sensitive range, 1.25 in/s)
K557L3YM.OE0W Tran+Vert PPV OORANGE + MicL PSPL OORANGE
Previously our _parse_number() returned None for OORANGE → DB columns
ended up NULL → events vanished from filters / sorts / dashboards
despite being legitimate high-amplitude events.
New behavior — substitute a conservative bound + set a saturation flag:
- Channel PPV → geo_range_ips + ChannelStats.ppv_saturated
- Peak Vector Sum → sqrt(3) * geo_range_ips + peak_vector_sum_saturated
- MicL PSPL → 140 dB(L) + MicStats.pspl_saturated
Flags propagate to the sidecar's bw_report block so the SFM UI can
render "> 10 in/s" / "> 140 dBL" rather than treating the substituted
value as exact.
Same commit also accepts "Peak Vector Sum TimeSum" as an alias for
"Peak Vector Sum Time" (BW always writes the typo on OORANGE PVS
lines — every example file confirms it).
Tests: new test_oorange_marker_treated_as_saturation (synthetic) +
test_real_oorange_event_t190_parses (skips if real fixture absent).
177/177 tests pass; 16 pre-existing missing-fixture skips unchanged.
Five events on prod (T190, T438, K557, plus 2 others matching the
same fault pattern) will pick up correct peaks + saturation flags
once watchers re-forward.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three layered changes that together make histogram charts visually
match BW's printout (one bar per interval, not per codec block):
1. bw_ascii_report parser captures histogram fields it previously
dropped:
- Histogram Start/Stop Time + Date → datetime
- Number of Intervals + Interval Size (string + parsed seconds)
- <Channel> Peak Time + Peak Date → datetime (per-channel)
- Peak Vector Sum Date (combined with PVS Time → datetime;
clears the bogus seconds parse that interpreted "22:33:52"
as 22.0)
New _parse_iso_date() handles BW's ISO format for histograms
(waveforms use "May 8, 2026" long form). New _parse_interval_size()
handles "1 minute" / "5 minutes" / "15 seconds" etc.
2. _bw_report_to_dict() projects the new fields into a new
bw_report.histogram block in the sidecar.
3. /db/events/{id}/waveform.json wraps the existing path 1 (HDF5)
output with _maybe_aggregate_histogram(): when the event is a
histogram AND the sidecar has bw_report.histogram.n_intervals,
group the codec's per-block samples into N intervals via
max-per-group and return the aggregated array. time_axis gains
histogram_aggregated / n_intervals / interval_size_s / interval_times
fields.
Frontend (both modal chart in sfm_webapp.html + standalone event
browser) uses interval_times as x-axis labels when provided (BW-style
HH:MM:SS), falls back to interval index.
Defensive: aggregation is no-op when the sidecar lacks the histogram
block (events ingested before this change). Activates automatically
on prod once a watcher re-forward populates new sidecars.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the .TXT was parsed into the sidecar's bw_report projection
and then discarded at ingest time. Now save_imported_bw() writes it
to <store>/<serial>/<filename>_ASCII.TXT permanently.
Rationale: with BW Mail / Forwarding Agent being phased out of the
operator workflow, the XML/PDF/WMF those tools produce won't be
available — the binary + .TXT (created by BW ACH itself) are our
only authoritative inputs going forward. Keeping the raw .TXT
unlocks:
- Parser bug fixes can be applied RETROACTIVELY by re-parsing the
stored .TXT, instead of requiring a re-forward from the watcher
PC (which lost the .TXT after BW ACH cleanup).
- Audit trail of what BW actually sent us, for debugging.
- The five known parser-PPV-miss events will be re-parseable once
the regex fix lands (instead of staying broken indefinitely).
Storage cost: ~15 KB per event × 14k events = ~210 MB on the
existing prod corpus. Negligible.
Implementation:
- WaveformStore gains txt_path_for() + open_txt()
- save_imported_bw() writes the .TXT when bw_report_text is supplied
- sidecar source block records the txt_filename
- backfill_sidecars.py preserves txt_filename across regens
- New GET /db/events/{id}/ascii_report.txt endpoint serves it
- Returns 404 for events ingested before this change (no .TXT in
the store yet) — re-forward to populate
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reviewed against real Blastware Event Report PDFs (uploaded to
example-events/pdfsnstuff/) for K558LLB7.V20H (histogram) and
K558LLB8.0E0W (waveform). Each event type has its own layout because
BW's printouts genuinely differ:
Waveform header: Date/Time, Trigger Source, Range, Sample Rate
Histogram header: Start, Finish, Intervals At Size, Range, Sample Rate
(no trigger field — histograms aren't triggered)
Waveform stats: PPV, ZC Freq, Time (Rel. to Trig),
Peak Acceleration, Peak Displacement, Sensor Check
Histogram stats: PPV, ZC Freq, Date, Time (of peak), Sensor Check
Waveform plot: 4-channel stacked line, x-axis in SECONDS,
trigger triangle + window markers, symmetric Y
for geo, zero-anchored mic, "0.0" baseline label
on right edge per BW convention
Histogram plot: 4-channel stacked bars, Y-axis 0-to-peak only
(never negative — peaks are magnitudes), 0.0
baseline at the bottom
Waveform footer: USBM chart placeholder upper-right;
"Time X sec/div Amplitude Geo: Y in/s/div Mic: 0.001 psi(L)/div"
"Trigger = ▶━━◀"
Histogram footer: No USBM chart; same scale-info footer with
interval-size as the time unit
Other fixes from the first-pass screenshot review:
- Channel labels (MicL/Long/Vert/Tran) no longer cut off (wider
left margin)
- Histogram bars rise from zero baseline (abs of any signed values)
- ISO timestamp "2026-05-16T22:33:50" → "22:33:50 May 16, 2026"
matching BW's display format
Known gaps (separate work):
- Histogram codec returns per-block granularity (~200 bars for
BW's 4-interval display). XML-driven data source is the planned
fix; the structured BW XML has the per-interval aggregates.
- USBM RI8507 / OSMRE compliance chart still placeholder
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New endpoint GET /db/events/{id}/report.pdf returns a single-page
letter-portrait PDF for any event with waveform data on disk.
Architecture:
sfm/report_pdf.py — gather_report_data() assembles fields from
SeismoDb row + .sfm.json sidecar (bw_report block) + .h5 samples;
render_event_report_pdf() turns that into PDF bytes via matplotlib.
sfm/server.py — new endpoint wires them together, streams PDF back
with Content-Disposition: inline so the browser displays it.
sfm_webapp.html — new "Download PDF" button in the event modal
footer that opens the endpoint in a new tab.
Fields surfaced — same coverage as a Blastware Event Report:
Header metadata (date/time, trigger source, range, sample rate,
project, client, operator, location, serial+firmware,
battery, calibration, file name)
Microphone block (PSPL in dB(L) + psi, ZC freq, channel test)
Per-channel stats (PPV, ZC Freq, Time of Peak, Peak Accel,
Peak Disp, Sensor Check) for Tran/Vert/Long
Peak Vector Sum
Waveform plot (MicL/Long/Vert/Tran stacked, shared time axis,
trigger marker, symmetric Y for geo, zero-anchored
mic) — OR per-interval bar chart for histograms.
Rendering pipeline = matplotlib only (vector PDF, no headless-browser
dep). Adds matplotlib>=3.8 to deps.
Visual layout is approximate until reference PDFs from Instantel land
at docs/reference/instantel/ for iteration. USBM RI8507 / OSMRE
compliance chart is stubbed (placeholder rectangle) — separate work
item.
Smoke-tested on a K558 waveform event: 77 KB valid PDF, all fields
populated correctly from the snapshot DB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The sidecar-modal waveform plot was rendering mic in raw psi, while the
rest of SFM (history table column, peaks block, live-device chart,
event detail modal mic field) had already converted to dB(L) — matching
the BW Event Report convention. Unifying.
Both viewers now:
- Default mic chart values + axis title + peak label to dB(L)
- Provide a header toggle ("Mic: dBL" pill) to flip to psi
- Persist the preference via localStorage (sfm_mic_unit)
- Re-render the open chart immediately on toggle
Conversion: dBL = 20 * log10(psi / 2.9e-9), where 2.9e-9 psi is the
20 µPa reference pressure already defined for the rest of the webapp.
Non-positive psi samples (log undefined) render as null; Chart.js
handles them as gaps in line mode and missing bars in histogram mode.
Also fixes event_browser.html's stats table — the MicL row was
hard-coding "<value> psi"; now honors the same toggle.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two fixes from the second screenshot review:
1. Geophone waveform Y-axis now renders SYMMETRIC around zero — zero
line sits in the middle of the chart, signal goes both above and
below. Standard seismograph display convention; matches the
Instantel printout look. Previously Chart.js auto-scaled to the
data range so e.g. Vert showing values from -0.005 to -0.015 had
the zero line completely off-screen.
Mic channel (sound pressure, always positive) keeps the default
auto-scale anchored at zero. Histograms (per-interval peaks, also
always positive) likewise keep bars rising from a zero baseline.
2. Modal labels clarified to remove the 'Timestamp' vs 'Captured at'
ambiguity:
'Timestamp' → 'Recorded at' (when the seismograph
recorded the event —
from BW report's Event
Time field)
'Captured at' → 'Received by server at' (when our sfm-db
inserted the row)
Both have tooltips explaining the distinction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three polish fixes spotted in the first prod screenshot of the inline
event-modal waveform plot:
1. Peak labels were rendering as "PEAK 2.500E-2 IN/S" because of a
blanket toExponential(3) call. New _fmtPeak() formatter picks
decimal with adaptive precision for normal-range values (0.0001 to
10000) and falls back to scientific only for truly extreme
magnitudes. Same value now reads "peak 0.0250 in/s".
2. Histogram events were being plotted as connected line charts, but
histograms are per-INTERVAL peaks (one bar per minute, typically),
not per-sample waveforms. Now: detect histogram via record_type,
render as a tight bar graph (bars touch), suppress the trigger line
+ zero baseline overlays (no trigger event on a histogram), and
label the x-axis with interval number instead of milliseconds.
3. X-axis tick labels were displaying as "11.7187040000000002 ms"
because the callback used the raw float, not the formatted label.
Snap to 1 decimal place (or integer for whole-number values like
histogram intervals).
Applied to both the inline modal plot in sfm_webapp.html and the
standalone /events viewer in event_browser.html — they share the same
data shape and presentation conventions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /db/events/{id}/waveform.json endpoint returns `time_axis` as a
metadata object — {sample_rate, pretrig_samples, t0_ms, dt_ms,
n_samples, total_samples, rectime_seconds} — not a per-sample times
array. Both viewers (sfm_webapp.html sidecar modal + event_browser.html)
were treating it as an array, silently falling back to a derived path
that ignored pretrig entirely and started the time axis at 0.
Symptom: trigger line drawn at the very left edge of every chart, no
visible "leading up to the event" samples even though they're in the
decoded data.
Fix: read time_axis.t0_ms (negative when pretrig samples exist),
time_axis.dt_ms, build per-sample times as `t0_ms + i * dt_ms`. Trigger
line lands at sample where t crosses 0; pretrig samples render at
negative t to the left of it.
Confirmed on a K558 event with 208 pretrig samples + 2 sec rectime at
1024 sps — time axis now spans -203 ms to +2046 ms, trigger line at
~9% from the left edge as expected.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three UX upgrades to the main SFM webapp at /, all reinforcing the
'browse stored events' flow as the primary entry point:
1. Default section is now Database, not Live Device. Most users land
here to look at stored events; Live Device is opt-in (click the tab
to talk to a unit). Initial history + units fetch fires on first
paint so the table is populated when the page loads.
2. History table columns are sortable. Click any header to sort:
timestamp, serial, per-channel PPV (Tran/Vert/Long), PVS, mic dB(L),
project, client, type, key. Default direction varies by column type
(desc for numbers + timestamps, asc for text). Sort arrows appear
in the active column header. Headers are sticky so they stay
visible while scrolling.
3. Click-event-to-see-waveform. The existing sidecar review modal now
renders the 4-channel waveform plot inline at the top, fetched from
/db/events/{id}/waveform.json in parallel with the sidecar fetch.
Channels stacked MicL / Long / Vert / Tran (Instantel printout
order), shared bottom time axis, dashed trigger line + triangle
markers at t=0, zero baseline with "0.0" label on the right edge,
peak callouts per channel. Charts cleaned up on modal close.
Resolves the "where is the viewer" surprise — operators no longer need
to know about the /events route to see waveforms.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Apply the cheap visual wins from the BW Event Report layout:
1. Channel order reversed → MicL (top), Long, Vert, Tran (bottom)
to match the Instantel printout.
2. Shared bottom time axis — x-axis ticks only render on the
bottom-most data channel; other channels hide ticks so all four
visually share one time scale.
3. Triangle trigger markers above and below the t=0 dashed line.
4. Horizontal zero-baseline (dotted) per channel with "0.0" label
on the right edge — Instantel convention.
5. "Print view" toggle that flips dark→light theme (white panels,
light grids, dark text) so the viewer can render usefully on
paper-style output / @media print.
6. Per-channel PPV stats table in the metadata header, with Peak
Vector Sum displayed prominently.
7. Colors adjusted to approximate BW trace colors (magenta MicL,
blue Long, green Vert, red Tran).
Future PDF-export work will reproduce the same layout server-side
once you upload a real example PDF and we pick a rendering pipeline
(weasyprint / chromium --print-to-pdf / etc.).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New standalone HTML page (sfm/event_browser.html, ~470 lines, Chart.js)
that lets you browse persisted events from the SeismoDb + WaveformStore.
Companion to the existing live-device viewer at /waveform:
/waveform — connect to a unit and pull events in real time
/events — browse events already stored in the DB
Flow:
1. Page loads → GET /db/units → populate serial dropdown
2. Select serial → GET /db/events?serial=X&limit=500 → event list
3. Click event → GET /db/events/{id}/waveform.json → render
Layout is Instantel-printout-ready: channels stacked vertically in
Tran / Vert / Long / MicL order, trigger line at t=0, peak labels,
clean dark theme. Frames the future PDF-export feature without
needing extra layout work.
Smoke-tested against the dev prod-snapshot — 4 channels render with
correct peaks for K558 events (L=0.3 in/s = the offset-fault peak
we've been chasing all week).
CHANGELOG entry added under [Unreleased] per the v0.20.0 release plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1. bw_ascii_report parser misses PPV/vector_sum fields on certain TXT
formats (5 events in prod). Parser extracts every OTHER field for
the same channels — likely a regex / format mismatch specific to
some firmware-or-event-type combination.
2. NULL-timestamp duplicate rows. events.timestamp can come back as
NULL when the codec can't extract a footer timestamp; UNIQUE(serial,
timestamp) doesn't fire on NULL, so backfills create new rows
instead of upserting. 2 affected events on prod, easy SQL cleanup.
3. Histogram body sub-format with byte[5] != 0. ~3 events on prod
(T190LD5Q, O121L4L1) use a histogram body the walker doesn't
recognize. Codec returns 0 valid blocks; DB peaks come from the
bw_report ASCII overlay so DB columns are correct, only the .h5
plot is empty. Cracking the sub-format unlocks the plot.
All three are pre-existing issues that today's deployment surfaced
during validation; none are regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror what the ingest path does: BW's reported peaks (and sample_rate
/ record_time) take precedence over codec output where present.
Without this, --force backfill silently overwrites bw_report-overlaid
DB columns with codec-derived peaks. Wrong for events where the codec
doesn't fully decode (waveform walker edge cases on SP0/SS0/SV0-style
events, histogram byte[5]!=0 sub-format that isn't yet RE'd), producing
PVS=0 on real high-amplitude events. Bit on prod 2026-05-22 with
three top-10 waveform events ending up at PVS=0 (rolled back same day,
this fix is the proper resolution).
New helper minimateplus.event_file_io.apply_bw_report_dict_to_event
operates on the projected sidecar dict shape (the structure
_bw_report_to_dict produces, which is what gets preserved in the
sidecar). Mirrors apply_report_to_event's semantics: only writes
fields where bw_report has a non-None value, no-ops cleanly on
empty / None input.
Dev validation against prod snapshot:
pre : 1839.7315 pvs_sum 356 events with DB PVS ≠ sidecar bw_report
post : 2016.4902 pvs_sum 2 events still mismatched (both have NULL
timestamp + duplicate rows, edge case)
Both edge-case events DO get the correct value written by the new
backfill — their stale rows from prior backfills remain because
UNIQUE(serial, timestamp) doesn't fire on NULL. Separate dedup
cleanup needed for those 2 events (0.014% of corpus); not blocking.
Backfill remains idempotent + bw_report preservation still passes
(0 WIPED, 0 CHANGED on the 3rd consecutive run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
serversdownandClaude Opus 5