Merge pull request 'v0.20.0 -- Full s3 event parse and PDF creation.' (#28) from dev into main

Reviewed-on: #28

CLAUDE.md

@@ -2,152 +2,12 @@
 
 Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
 managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
-(Sierra Wireless RV50 / RV55). Current version: **v0.31.0**.
+(Sierra Wireless RV50 / RV55). Current version: **v0.20.0**.
-
-Stack-level context — which repo owns what, and how the three project versions
-pair — lives in `../terra-view/docs/tmi-stack.md`, which is also loaded as
-`~/CLAUDE.md`.
-
----
-
-## Where things stand (updated 2026-08-28)
-
-Read this first when picking the project back up.
-
-- **Series-3 decode is verified per-sample at scale (v0.27.0).** The full DL2
-  archive decodes **14,338 / 14,338** paired files exactly against their
-  preserved Blastware ASCII exports — 1,249 waveform + 13,089 histogram, 45
-  units, files back to 2018. That is 11x the ground truth the prod store
-  carried, and it supersedes the old "per-sample on 11%, peak-only on 89%"
-  caveat. Harness: `scratch/verify_against_ascii.py` (note its saturation
-  carve-out — BW clamps clipped events, the decoder reports true counts).
-  Independent corroboration of the 32000-count scale: 19,244 healthy
-  channel-events sit at a pre-trigger floor of exactly 0.000 (62.7%), 94.5%
-  within ±1 quantisation unit, median +0.0000 — no zero-point bias.
-- **Series-4 (Thor / Micromate) is now verified per-sample (2026-09-10).**
-  **1,057,536 / 1,057,536** geo samples across all 153 genuine Thor waveform
-  files reproduce Thor's own CSV export exactly; IDFH peaks are within 2% on
-  858/858 (median -0.004%).  The ground truth was in the corpus all along —
-  Thor writes `CSV/<name>.IDFW.csv` beside each binary with a **per-sample**
-  four-column block.  Harness: `scratch/verify_thor_against_csv.py`.
-  Four bugs, all fixed: geo LSB was `0.0003` (display rounding of the real
-  `0.000310308`, so every sample read **3.3% low**); the IDFH segment
-  validator required a zero counter high byte, **capping every histogram at
-  250 intervals**; record mode `00 00` (raw int16) was unhandled, silently
-  dropping each channel's first 512 samples; and the body-offset search
-  matched `00 02 00` *inside* record headers, decoding a rotation-shifted
-  body.  IDFW is no longer pinned to `decode_waveform_legacy`.
-  Series-3 re-verified unchanged at 14,338/14,338 after the shared-codec
-  change.
-- **Mic-disabled (3-channel) units are a distinct shape (2026-09-10).**
-  Verified on a second corpus (`~/thor-csv-req`, UM11402/UM12947/UM20147):
-  **139/139** waveforms per-sample exact, **877/877** histograms within 2%.
-  Two structural differences: the shorter header puts the waveform record
-  chain head at `0x0dba` (below the old `_BODY_SCAN_FLOOR` of `0x0E00`, so it
-  was invisible and Vert came up exactly 512 short), and the histogram
-  interval record is **56 bytes, not 72** — `16 × n_channels + 8`, derived per
-  segment from the cumulative interval counter, never assumed.
-- **`40 NN` blocks are not capped at NN=8 (2026-09-11).**  `data_block_len()`
-  rejected `NN > 0x08`, a guard with no evidence behind it — the corpora
-  available when it was written only used NN ∈ {1,2,3,4,8}.  Loud UM12947
-  events use NN up to 196, and since the walker stops at the first
-  unrecognised tag rather than raising, this surfaced as silently short
-  channels.  Verified on 167 UM12947 waveforms: length mismatches 22 → 0,
-  1,476,242/1,476,249 samples exact.
-- **Production IDFW is now 575/575** — zero truncations, zero decode
-  failures, median PPV error −0.0007% across 8 units (was 41 truncated + 1
-  failing, −3.3%).  Across all three ground-truth corpora: **459 files,
-  3,807,158/3,807,165 samples exact**; the 7 stragglers differ by one
-  4th-decimal tick and are Thor's own rounding — no single linear LSB can
-  reproduce every printed value (the constraints are infeasible by 7e-5
-  relative), so do NOT retune `_GEO_LSB_IPS`.
-- **⚠ KNOWN BUG — the 5A walk breaks once a unit's buffer crosses 64 KB.**
-  `parse_strt_end_offset()` returns only `(end_key[2] << 8) | end_key[3]`,
-  discarding the key's page byte.  An event starting at `0x0111F2A2` and ending
-  at `0x0112_1010` therefore reads `end_offset = 0x1010` — *behind* its own
-  start.  The chunk loop then exits before fetching anything and TERM computes
-  a negative `offset_word`, which `struct.pack(">H", ...)` rejects: the
-  `/device/events` walk 500s.  Reproduced on BE12599 (2026-09-19), which had
-  78 KB stored and had rolled into page `0x12`.
-  **Why it hid so long:** every 5A capture the walk was verified against came
-  from a freshly-erased BE11529 — all three confirmed TERM examples in
-  `framing.py` (`0x1ABE`, `0x21F2`, `0x417E`) sit inside page `0x11`.  Prod is
-  unaffected: it ingests complete files via BW ACH, never this walk.
-  **Fixing it has two layers** — the arithmetic (`if end < start: end +=
-  0x10000`) stops the crash and bounds the loop correctly; carrying the page
-  byte through the chunk requests (`params[1]` 0x11 -> 0x12, counter rolling
-  over) needs a BW capture of a spanning event first.  Do not ship layer one
-  alone without a loud truncation warning — a silently short event is the
-  failure mode this codec has been bitten by repeatedly.
-- **Open, not blocking:** 14 sensitive-range files show an exact 8x
-  (= 10.0/1.25) units discrepancy; `scripts/backfill_sidecars.py --force` also
-  inserts DB rows for store files that have none (one-time per store) and the
-  dry-run does not report that count.
-- **After any codec change, regenerate the store** — `backfill_sidecars.py`
-  then `backfill_event_shape.py`, DB backup first. Stored `.h5` files do not
-  update themselves. No `--force` needed as long as `TOOL_VERSION` was bumped
-  (it gates regeneration). ⚠ On the office NAS this takes **~2 hours**
-  (~1.5 files/sec vs 85/sec on the dev box — gzip-4 in `sfm/event_hdf5.py`
-  against a Synology CPU). Budget it up front.
-  **v0.27.0 does NOT owe prod a backfill** — verified: the partial-final-block
-  fix changes 0 of the 10,215 histograms in the prod store (the 4 recovered
-  files are archive-only and were never ingested).
-- **The "offset" hardware fault has its own journal** --
-  `docs/offset_investigation.md`. **5 of 45 units (11%)**, and the fault is
-  **persistent** — it stays until the geophone is serviced. Detect it with
-  `scratch/offset_scan3.py`: the resting floor in the **pre-trigger** window,
-  required to hold across pre/middle/end. Never score only the dominant-peak
-  axis and never use the mean — both produce false recoveries (see the
-  retraction banner in the journal). Instantel's autozero procedure and its
-  2027-2069 acceptance window are recorded there too. Best open lead is
-  `SUB 0x0E` (unimplemented), which may carry those very numbers.
-
 
 When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
 
 ---
 
-## Changelog & release convention
-
-**Feature branches do NOT touch `CHANGELOG.md`.  Write the entry on `dev`, as
-part of finishing the merge, under `## Unreleased`.  Cut the version on `dev` in a
-dedicated release commit when you are ready to ship to `main`.**
-
-- **The changelog is written on `dev`, never on a feature branch.**  With
-  several branches in flight they all edit the same few lines at the top of
-  the file and conflict every time.  Writing it once, after the merge, also
-  lets it describe what actually *landed* — including anything that changed
-  during conflict resolution.
-- ⚠ **The merge is not finished until `## Unreleased` is updated.**  Same sitting,
-  not "later" — that is the one failure mode of writing it after the fact.
-  Reconstruct from the branch's own commit messages:
-  `git log --oneline dev..<branch>` before you merge, or
-  `git log --oneline <merge-base>..<branch>` after.
-- **No preamble under `## Unreleased`** — just the `### Added` / `### Changed` /
-  `### Fixed` lists.  The themed opening paragraph gets written at release
-  time, when the whole release is visible and can be named honestly.  A theme
-  written when the first item landed is stale by the third.
-- ⚠ **State the operational consequence** on any entry touching the codec, the
-  waveform store, or the DB — **including when it is "none."**  "requires
-  `backfill_sidecars.py` + `backfill_event_shape.py`, ~2 h on the NAS",
-  "`TOOL_VERSION` bumped", "no schema change, no migration".  Silence is
-  ambiguous; "none" is information.  This repo's changelog is how future-you
-  learns whether a deploy costs two hours.
-- **Releases are cut on judgement, not on a schedule or a merge.**  `Unreleased`
-  is the staging area for whatever is going into the next release; when enough
-  has accumulated to be worth shipping, it gets a number and a date.  Nothing
-  about a merge to `dev` triggers a release.
-- **Cutting a release** is its own `chore(release): vX.Y.Z — <theme>` commit on
-  `dev`, renaming `## Unreleased` → `## vX.Y.Z — YYYY-MM-DD` and touching:
-  `CHANGELOG.md`, `pyproject.toml`, the version line in `CLAUDE.md` and
-  `README.md`, and `minimateplus/event_file_io.py` (`TOOL_VERSION`) **when the
-  codec changed** — that constant gates `.h5` regeneration.
-- **`main` carries only released versions.**  No `## Unreleased` section there;
-  it lands via the `dev` → `main` PR.  `main` lagging `dev` by a version is
-  normal.
-
----
-
 ## Architecture: three-tier conceptual model
 
 seismo-relay is a **suite of cooperating components**, not a single app.
@@ -213,42 +73,6 @@ should not import from `sfm/`, must not touch a DB, and have no I/O
 beyond reading files passed as arguments.  Keep them pure — both
 tiers can then depend on them without circularity.
 
-#### Thor IDF binary codec (updated 2026-09-10)
-
-`micromate/idf_file.read_idf_file()` decodes both Thor IDFW
-(waveform) and IDFH (histogram) binaries.  **Verified per-sample
-against Thor's own CSV exports** — see
-`scratch/verify_thor_against_csv.py`.
-
-- **IDFW** uses the series-3 record-chain `decode_waveform_v2()`.  The
-  body offset is **not** fixed: it is `<chain-head record> + 7`, found
-  by `_find_waveform_body_offset()` anchoring on record headers.  All
-  **153/153** genuine Thor waveform files decode per-sample exact
-  (1,057,536/1,057,536 samples).
-- **IDFH** segment header is `[len_be][0a 00 00 00][counter_be][05 3f]`,
-  where `counter` is a **uint16 cumulative interval index** — it must
-  not be constrained to a zero high byte (that capped histograms at 250
-  intervals).  Intervals whose `min > max` on all channels are unwritten
-  slots carrying a ±full-scale seed and are skipped.  858/858 files land
-  within 2% of Thor's PPV (median -0.004%).
-- **Geo LSB is `0.000310308` in/s per count** (full scale 10.0 in/s =
-  32226.05 counts).  Series-3's 32000-count scale does NOT apply.
-- **Record modes** are `02 00` deltas (14 B header), `01 00` absolute,
-  `00 03` raw 12-bit, and `00 00` **raw int16** (all 10 B headers).
-  `01 00` and `00 00` are also valid as the implicit segment-0 preamble.
-
-⚠ **Thor's histogram PPV has a 0.0050 in/s display floor.**  41.4% of
-prod IDFH sidecars report a component PPV exceeding their own vector
-sum — impossible.  On quiet files our decode is *more* accurate than
-the reference; do not "fix" the decoder to match it.
-
-The two outlier `BE9439_*` files in the Thor example corpus are
-actually Series III Blastware binaries that share the `.IDFW`/`.IDFH`
-filename convention by accident.  `read_idf_file()` detects them by
-their BW STRT signature and raises NotImplementedError pointing
-callers at `read_blastware_file()`.  See
-`docs/idf_protocol_reference.md` for full field layouts.
-
 ### Practical consequences
 
 When deciding where new code goes, ask:
@@ -377,118 +201,6 @@ custom delta + RLE + variable-width codec.
   `NN + 2` for int8 blocks).  Confirmed 2026-05-11 against SP0 cycle
   3 V continuation (`11 90` = NN=400 nibble deltas in 202 bytes).
 
-### ⚠ SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN
-
-Everything in this section below about `40 NN` segment headers, tagless
-headers, variable header widths and channel rotation describes a model that
-is **wrong**.  The body is a chain of self-delimiting per-channel records:
-
-    off+2  len  uint16 BE  ->  next_record = off + 2 + len
-    off+4  chan_id  0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL / 0x06 = END
-    off+8  mode  02 00 = deltas+anchors (14B hdr)
-                 01 00 = ABSOLUTE values (10B hdr)
-                 00 03 = raw 12-bit absolute, NO TAGS (10B hdr)
-
-`40 NN` is an ordinary int16 BE data block (`2*NN + 2`), never a header.  The
-"variable prefix" of 0/2/4/6/8 bytes was walker drift, exactly
-`4 - (old_stop - true_record_start)`.
-
-All four channels now come out equal length in **1388/1388** files (was
-156/1388); ASCII sample-count exact **75/75**, fully exact **73/75**; device
-PPV on a live decode **1306/1306** waveform, **4458/4459** histogram.
-
-The old model survives as `decode_waveform_legacy` because
-`micromate/idf_file.py` pins it for Thor IDFW body-offset search.
-
-### Framing cases added 2026-05-11 → 2026-08-25
-
-Four more block-framing cases, each of which had been causing **silent
-channel truncation** — `walk_body` ends its loop on an unrecognised tag
-and `decode_waveform_v2` returns whatever channels it got, so an
-unhandled tag surfaces as short channels with no error raised.  Found by
-diffing 75 production events against their preserved Blastware ASCII
-exports (`<store>/<serial>/<file>_ASCII.TXT`).
-
-- **Wide-NN RLE `0X NN`** — the 12-bit NN encoding documented above for
-  `1X`/`2X` **also applies to the `00 NN` RLE tag**.  A narrow run maxes
-  out at NN=0xFC, so a quiet stretch longer than 252 samples must use
-  the wide form (e.g. `01 0c` = 268 repeats).
-- **`30 NN` is not capped at NN=0x10** — data-section blocks reach at
-  least NN=0x18.  The `NN × 1.5 + 2` length formula was already right;
-  only the guard was wrong.
-- **`40 NN` segment headers are variable width** — NN is the *count of
-  int16 BE continuation deltas for the PREVIOUS channel*, so the header
-  is `2*NN + 16` bytes and every field after the deltas shifts by
-  `2*NN`.  `40 01` (18 B) and `40 03` (22 B) both occur alongside the
-  common `40 02` (20 B).
-- **Tagless segment headers** — a header can appear with **no `40 NN`
-  tag at all**: just the 14-byte tail
-  `[field2:2][len:2][channel_id:4][marker:2][anchors:4]`.  This is the
-  NN=0 case (previous channel needed no continuation deltas).
-
-**The header "counter" is really a channel id.**  The 4-byte field long
-documented as a "monotonic uint32 LE counter" is
-`[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert
-`0x48`=Long `0x49`=MicL — verified on **1697/1697** segment headers
-across the corpus, zero disagreements.  `decode_waveform_v2` now takes
-the channel from this field rather than from rotation position; a single
-missed or extra header (exactly what tagless headers caused) desyncs
-rotation and corrupts every channel after it.
-
-Corpus result, end to end through the production path:
-**exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.**
-
-### Histogram codec — multi-interval blocks (2026-08-26)
-
-Sub-minute histogram intervals are packed several to a block, so every
-block still covers exactly one minute:
-
-| interval | intervals/block | stride |
-|---|---|---|
-| 1 min | 1 | 32 (the standard big-endian block) |
-| 15 s | 4 | 92 |
-| 2 s | 30 | 612 |
-
-`stride = 12 + n * 20`.  Block = `[00][segment][ctr uint16 LE][0a][00]`,
-then n x 20-byte records of 8 x uint16 **LITTLE**-endian values
-(`T_peak, T_halfp, V_peak, V_halfp, L_peak, L_halfp, M_peak, M_halfp`)
-plus a 2-word tail whose first word is `0000` on every real interval,
-then a 6-byte block trailer.
-
-⚠ The standard 32-byte block is BIG-endian; this variant is LITTLE-endian.
-
-Recovers **415 files** (216 on BE18193, 199 on BE9440) that decoded to
-nothing.  Ground truth `BE9440/K440L3AQ.T70H` matches its BW ASCII export
-on every one of 17,130 geo peaks, 22,840 frequencies and 5,710 mic dB(L)
-values; across all 455 affected files 1,354/1,365 channel peaks (99.2%)
-match the device-reported PPV.
-
-### Histogram codec — corrected 2026-08-25
-
-The histogram block is **uniformly big-endian**, and the stream's final
-block has its own tail signature.  Two long-standing errors:
-
-- **Peaks and half-periods are `uint16` big-endian**, not `uint8` +
-  an "annotation" byte.  `T_peak` is `[5:7]`, `T_halfperiod` `[7:9]`,
-  `V_peak` `[9:11]`, and so on; only `block_ctr` at `[2:4]` is LE.
-  The old model silently **clipped any peak above 1.275 in/s** — the
-  final interval of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in BW's
-  export and decoded as 0.590.  The "annotation" byte was the
-  half-period's high byte, which is why it was non-zero exactly on the
-  sub-Hz intervals BW renders as `<1.0`.
-- **The marker is `block[4]` alone.**  Testing `[4:6]` as a uint16 LE
-  marker forced `block[5] == 0`, which is what capped the peak at one
-  byte in the first place.
-- **The last block of the stream carries tail `9c 06 00 42`** instead of
-  `1e 0a 00 00`, with arbitrary bytes at `[21:23]`.  Rejecting it
-  dropped the final interval of nearly every histogram — frequently the
-  one holding the event peak, so the file's PPV read low.
-
-Verified against 1211 production histograms paired with their BW ASCII
-exports: **1211/1211 decode exactly** (interval count plus every
-per-interval peak), and 842,442 per-interval frequency comparisons match
-with zero mismatches.  Before: 1 of 1196.
-
 ### What's NOT solved
 
 - **MicL channel conversion to dB(L)** — the codec emits MicL as
@@ -496,29 +208,9 @@ with zero mismatches.  Before: 1 of 1196.
   shows mic in dB(L) with ~6 dB quantization steps.  Need to map
   ADC counts → dB(L) for direct comparison; likely
   `dB = 20*log10(|counts|) + offset` or similar.
-- **Variable-prefix segment descriptors** — 3 of the 75 ground-truth
+- **Walker edge cases** — SP0/SS0/SV0 don't walk the full event due
-  production events still truncate.  The walk reaches a segment header
+  to block-length quirks past the first few segments.  Every sample
-  whose channel-id field is preceded by a *variable-width* prefix (2, 4
+  reached is correct; the walker just needs robustness improvements.
-  or 6 bytes observed; the standard tagless form always has 4), carrying
-  an `01 00` marker instead of `02 00`.  The marker is **not** simply an
-  anchor count — `01 00` records appear with both 2- and 4-byte anchor
-  fields in the same file.  Examples: `BE12599/N599LPNB.JF0W` @1155,
-  `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
-(The series-3 histogram codec was fixed 2026-08-25 — see below.)
-
-- ~~**Micromate (UM-series) IDF decode is ~1000× low**~~ — FIXED 2026-09-10.
-  `UM11402_20260406130113.IDFW` now decodes Tran 1.1168 / Vert 4.3220 /
-  Long 0.9135, matching the device report exactly.  Root cause was the
-  body-offset search landing inside a record header plus the unhandled
-  `00 00` record mode, not anything UM-specific.
-- ~~**Thor IDF per-count LSB**~~ — RESOLVED 2026-09-10.  The 0.983 ratio was
-  exactly `0.0003 / 0.000310308`.  Thor's geo LSB is **0.000310308 in/s per
-  count** (full scale 10.0 in/s = 32226.05 counts), pinned to ±6e-11 by
-  intersecting 991,415 rounding constraints from Thor's own exports and
-  corroborated by the ±full-scale seed (`±32226`) left in unwritten IDFH
-  interval slots.  Series-3's 32000-count scale does **not** carry over.
-  Note `10.0/32226` is very slightly wrong — see
-  `docs/idf_protocol_reference.md`.
 
 ### Decoded sample counts (across the fixture bundle)
 
@@ -550,14 +242,6 @@ then `decoded_to_adc_counts()` to scale to int16 ADC counts (geos × 16;
 mic pass-through).  The `.h5` sidecars SFM produces now contain
 correct samples for any event without walker edge cases.
 
-**Geo full scale is 32000 ADC counts, NOT 32768** (fixed 2026-08-25).
-One decoder unit = 16 ADC counts = exactly 0.005 in/s, so
-`10.000 in/s / (0.005/16)` = 32000.  Consumers must use
-`sfm.event_hdf5._GEO_INT16_FS` / `event_file_io._GEO_INT16_FS` (both
-32000).  Dividing by 32768 reads every geophone sample 2.34% low —
-that was a live bug in both modules until 2026-08-25.  Mic is
-unaffected (it back-solves its scale from the device-reported peak).
-
 The original int16 LE decoder is preserved as
 `_decode_a5_waveform_LEGACY` for reference but is not called.
 
@@ -1934,4 +1618,4 @@ body) because writing a dial string may require DLE escaping for embedded contro
 
 To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern
 in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes
-inside write frame data (the naive parser terminates early at the escaped `0x03`).
+inside write frame data (the naive parser terminates early at the escaped `0x03`).

README.md

@@ -1,4 +1,4 @@
-# seismo-relay  `v0.31.0`
+# seismo-relay  `v0.20.0`
 
 A ground-up replacement for **Blastware** — Instantel's aging Windows-only
 software for managing seismographs.  Supports both the **MiniMate Plus
@@ -45,20 +45,6 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
 > `scripts/backfill_sidecars.py --reparse-txt` lets parser fixes be
 > applied retroactively to existing events without re-forwarding,
 > using the `.TXT` files preserved at ingest time.
-> **v0.21.0 (2026-05-29)** is the Thor / Series IV decoder release —
-> `micromate/idf_file.read_idf_file()` now decodes both IDFW
-> (waveform) and IDFH (histogram) binaries (87–99% sample fidelity
-> on quiet IDFW events; all 859 IDFH corpus files decode cleanly).
-> A new `micromate/idf_to_bw_report.py` adapter projects parsed
-> Thor reports into the BW-shaped sidecar block, so Thor events
-> flow through the existing Event Report PDF pipeline without a
-> separate renderer.  Terra-View v0.13.0 ships in parallel and
-> closes Phase 1 of the SFM integration — see its CHANGELOG.
-> **v0.22.0 (2026-07-03)** adds the SFM side of the full-snapshot
-> bundle: `GET /db/snapshot` (WAL-safe DB copy), `GET /db/waveforms/
-> recent.zip` (recent events' waveform files), and a gated
-> `POST /db/restore` (`SFM_DB_RESTORE_ENABLED`, dev-only).  Terra-View
-> v0.17.0 drives them to pull a one-pass prod→dev refresh.
 > See [CHANGELOG.md](CHANGELOG.md) for full version history.
 
 ---
@@ -82,8 +68,7 @@ seismo-relay/
 ├── micromate/                 ← Series IV (Micromate / Thor) client library (NEW v0.19)
 │   ├── models.py              ←   IdfEvent, IdfReport, IdfPeaks, IdfProjectInfo, IdfSensorCheck (mic in native dB(L))
 │   ├── idf_ascii_report.py    ←   Parse Thor .IDFW.txt / .IDFH.txt event sidecars
-│   ├── idf_file.py            ←   Binary codec for .IDFW + .IDFH (v0.21.0+)
+│   └── idf_file.py            ←   Stub for the .IDFW / .IDFH binary codec (reverse-engineering pending)
-│   └── idf_to_bw_report.py    ←   Adapter projecting Thor IDF into the BW report shape (v0.21.0+)
 │
 ├── sfm/                       ← SFM REST API server (FastAPI, port 8200)
 │   ├── server.py              ←   Live device endpoints + DB query + ingest endpoints + caching
@@ -215,18 +200,6 @@ optionally paired with their ASCII sidecar reports; both dedup by
 | `POST` | `/db/import/blastware_file` | Series III: `.AB0*` / `.N00` binaries + paired `_ASCII.TXT`.  Source: `series3-watcher`. |
 | `POST` | `/db/import/idf_file` | Series IV: `.IDFH` / `.IDFW` binaries + paired `.IDFW.txt` / `.IDFH.txt`.  Source: `thor-watcher`. |
 
-### DB snapshot / restore endpoints
-
-Back the "full snapshot bundle" prod→dev refresh Terra-View orchestrates
-(Settings → Database).  Snapshot/zip are read-only; restore is gated and
-dev-only.
-
-| Method | URL | Description |
-|--------|-----|-------------|
-| `GET` | `/db/snapshot` | WAL-safe point-in-time copy of `seismo_relay.db` via the SQLite online backup API. |
-| `GET` | `/db/waveforms/recent.zip?n=…` | Zips the on-disk files (event file + `.h5` / sidecars) for the *n* most-recent events. |
-| `POST` | `/db/restore` | Validate-first restore of an uploaded DB + waveforms; takes a WAL-safe pre-restore safety backup, path-traversal-guards the zip.  Gated by `SFM_DB_RESTORE_ENABLED` (dormant → 404). |
-
 ---
 
 ## minimateplus library
@@ -452,7 +425,7 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
 - [x] Thor IDF file ingest at `/db/import/idf_file` (paired with `thor-watcher`, v0.18.0+)
 - [x] Native `IdfEvent` / `IdfReport` typed models — mic in dB(L), full title strings, sensor self-check, calibration, firmware version
 - [x] Parser verified against 1,014 paired `.txt` sidecars in `thor-watcher/example-data/`
-- [x] Binary `.IDFW` / `.IDFH` codec — ✅ v0.21.0.  IDFW reuses `decode_waveform_v2()` on the body at offset `0x0f1f` (87–99% sample fidelity on quiet events); IDFH has a dedicated segment-based decoder (all 859 corpus files decode, 181,071 intervals total).  See `micromate/idf_file.py` + `docs/idf_protocol_reference.md`.
+- [ ] Binary `.IDFW` / `.IDFH` codec — pending (see Roadmap + [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md))
 - [ ] Live-device protocol — pending codec
 
 **Data persistence:**
@@ -496,11 +469,6 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
 
 ## Roadmap (Future)
 
-> **Where it stands *today*** — an honest per-capability maturity assessment,
-> what to rely on, known issues, and the gap to a real tool:
-> [`docs/sfm_tool_status.md`](docs/sfm_tool_status.md).  This section covers
-> where it is *going*.
-
 ### Strategic direction — where this is going
 
 seismo-relay is being built as a **suite of cooperating components**
@@ -570,7 +538,7 @@ Implementation steps (concrete):
 ### High-impact (unblocks product features)
 
 - [ ] **Series III waveform body codec reverse-engineering.**  The 5A bulk-stream body is some kind of compressed/encoded format (not raw int16 LE as previously assumed — see §7.6.1 retraction in `docs/instantel_protocol_reference.md`).  Structural framing is ~50% decoded on branch `claude/codec-re-cBGNe` (tagged-block walker, segment counters); per-byte sample mapping is still open.  Until this lands, the in-app waveform viewer renders garbage and BW-import peak values fall back to `_peaks_from_samples()` saturation noise.  Workaround: pair every BW-imported event with its `_ASCII.TXT` so the device-authoritative peaks land in the DB regardless of codec.
-- [x] **Series IV (Thor IDF) binary codec reverse-engineering.** ✅ v0.21.0 — `micromate/idf_file.read_idf_file()` decodes both IDFW (waveform body at offset `0x0f1f`, reusing `decode_waveform_v2()`; 87–99% sample fidelity on quiet events) and IDFH (dedicated segment-based decoder: all 859 corpus files decode, 181,071 intervals, peaks within ~1.8% of sidecar values).  `WaveformStore.save_imported_idf` now also projects parsed Thor data into a `bw_report` block via `micromate/idf_to_bw_report.py` so Thor events render in the existing Event Report PDF pipeline without a separate renderer.
+- [ ] **Series IV (Thor IDF) binary codec reverse-engineering.**  `.IDFH` / `.IDFW` files are currently stored opaquely by `WaveformStore.save_imported_idf`, with all metadata sourced from the paired `.txt` sidecar.  This works because thor-watcher forwards both files together, but operators who haven't enabled Thor's TXT exporter get rows with NULL peaks.  Cracking the binary closes that gap and unlocks waveform display.  Starting-point reference at [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md) — two observed file signatures (1,012 newer-firmware files + 2 old files whose layout matches the Series III STRT-record format), suggested first-session plan (~2-4 hrs), 1,014 paired binary+txt files available as ground truth in `thor-watcher/example-data/`.  Code seam ready at `micromate/idf_file.py`.
 - [ ] **In-app waveform viewer accuracy.**  Depends on Series III codec decode.  Plot.v1 JSON pipeline + viewer skeleton already exist; will start showing real waveforms automatically once `_decode_a5_waveform` produces correct samples.  Series IV waveforms come online when the IDF codec lands.
 - [ ] **Series IV live-device support.**  Once the IDF binary is decoded, extend `micromate/` with `transport.py` / `framing.py` / `protocol.py` / `client.py` mirroring the `minimateplus/` package layout — depends on capturing Thor's wire protocol (TCP / RS-232 captures TBD).
 - [ ] **Terra-view integration** — seismo-relay router, unit detail page, VISON-style event listing.

analysis_idf/corpus_accuracy.py

@@ -1,65 +0,0 @@
-"""Run read_idf_file across the corpus and report per-channel accuracy vs sidecars."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from micromate.idf_file import read_idf_file
-from analysis_idf.recon import load_sidecar_samples
-
-
-def sidecar_path(idfw: Path) -> Path:
-    return idfw.parent / "TXT" / f"{idfw.name}.txt"
-
-
-def main():
-    root = REPO / "tests/fixtures/THORDATA_example"
-    files = [f for f in root.rglob("*.IDFW") if not str(f).endswith(".CDB")]
-    files.sort()
-    GEO_LSB = 0.0003
-
-    n_ok = n_skip = 0
-    overall = {"Tran": [], "Vert": [], "Long": []}
-
-    for f in files:
-        try:
-            res = read_idf_file(f)
-        except Exception:
-            n_skip += 1
-            continue
-        sc_path = sidecar_path(f)
-        if not sc_path.exists():
-            n_skip += 1
-            continue
-        try:
-            sc = load_sidecar_samples(sc_path)
-        except Exception:
-            n_skip += 1
-            continue
-
-        per_file = {}
-        for ch in ("Tran", "Vert", "Long"):
-            sc_counts = [int(round(v / GEO_LSB)) for v in sc[ch]]
-            dec = res.samples.get(ch, [])
-            n = min(len(sc_counts), len(dec))
-            if n == 0:
-                per_file[ch] = 0.0
-                continue
-            exact = sum(1 for i in range(n) if sc_counts[i] == dec[i])
-            pct = 100.0 * exact / n
-            per_file[ch] = pct
-            overall[ch].append(pct)
-        n_ok += 1
-
-    print(f"Processed {n_ok} files (skipped {n_skip})")
-    print("Per-channel exact-match % (mean / min / max):")
-    for ch, vals in overall.items():
-        if vals:
-            avg = sum(vals) / len(vals)
-            print(f"  {ch}: mean={avg:.2f}%  min={min(vals):.2f}%  max={max(vals):.2f}%  n={len(vals)}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/diff_trail.py

@@ -1,49 +0,0 @@
-"""Find where decoded-vs-sidecar diverges for each channel."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from minimateplus.waveform_codec import decode_waveform_v2
-from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
-
-
-def main():
-    buf = TARGET.read_bytes()
-    sc = load_sidecar_samples(TXT)
-    decoded = decode_waveform_v2(buf[0x0f1f:])
-    GEO_LSB = 0.0003
-
-    for ch in ("Tran", "Vert", "Long"):
-        sc_counts = [int(round(v / GEO_LSB)) for v in sc[ch]]
-        dec = decoded[ch]
-        # Find ALL transitions where mismatches start/stop
-        first_diff = next((i for i in range(len(dec)) if dec[i] != sc_counts[i]), None)
-        if first_diff is None:
-            print(f"{ch}: NO MISMATCHES")
-            continue
-        print(f"{ch}: first diff at idx {first_diff}")
-        # Show 5 before, 5 after
-        for i in range(max(0, first_diff - 3), min(len(dec), first_diff + 8)):
-            mark = "  " if dec[i] == sc_counts[i] else "**"
-            print(f"  {mark} idx {i:4d}: sc={sc_counts[i]:6d}  dec={dec[i]:6d}  diff={dec[i]-sc_counts[i]:+d}")
-        # Where does cumulative diff exceed 100?
-        cum_match_run = 0
-        max_match_run = 0
-        match_run_start = 0
-        diff_count = 0
-        for i in range(len(dec)):
-            if dec[i] == sc_counts[i]:
-                cum_match_run += 1
-                max_match_run = max(max_match_run, cum_match_run)
-            else:
-                cum_match_run = 0
-                diff_count += 1
-        print(f"  total mismatches: {diff_count}/{len(dec)}, longest run of matches: {max_match_run}")
-        print()
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/e2e_idfh.py

@@ -1,48 +0,0 @@
-"""End-to-end IDFH ingest verification."""
-from __future__ import annotations
-import sys
-import tempfile
-import json
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from sfm.waveform_store import WaveformStore
-
-
-def main():
-    idfh = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM13981/UM13981_20220805075441.IDFH"
-    txt  = idfh.parent / "TXT" / f"{idfh.name}.txt"
-
-    with tempfile.TemporaryDirectory() as td:
-        store = WaveformStore(Path(td))
-        ev, rec = store.save_imported_idf(
-            idfh.read_bytes(),
-            idfh,
-            idf_report_text=txt.read_text(errors="replace"),
-        )
-        print("=== save_imported_idf (IDFH) ===")
-        print(f"  serial:        {rec['serial']}")
-        print(f"  filename:      {rec['filename']}")
-        print(f"  filesize:      {rec['filesize']}")
-        print(f"  h5:            {rec['hdf5_filename']}")  # expect None for histogram
-        print(f"  sidecar:       {rec['sidecar_filename']}")
-        print()
-        print("=== Event ===")
-        print(f"  timestamp:     {ev.timestamp}")
-        print(f"  record_type:   {ev.record_type}")
-        print(f"  sample_rate:   {ev.sample_rate}")
-        print()
-        # Inspect sidecar to confirm intervals were stashed
-        sc_path = Path(td) / "UM13981" / f"{idfh.name}.sfm.json"
-        sc = json.loads(sc_path.read_text())
-        intervals = sc.get("extensions", {}).get("idf_intervals", [])
-        print(f"  sidecar intervals: {len(intervals)}")
-        if intervals:
-            print(f"  first interval:    {intervals[0]}")
-            print(f"  last interval:     {intervals[-1]}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/e2e_no_txt.py

@@ -1,40 +0,0 @@
-"""Verify the had_report=False path: ingest IDFW with no .txt."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-import tempfile
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from sfm.waveform_store import WaveformStore
-
-
-def main():
-    idfw = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162723.IDFW"
-    with tempfile.TemporaryDirectory() as td:
-        store = WaveformStore(Path(td))
-        ev, rec = store.save_imported_idf(
-            idfw.read_bytes(),
-            idfw,
-            serial_hint=None,
-            idf_report_text=None,        # ← no .txt!
-        )
-        print("=== IDFW without .txt ingest ===")
-        print(f"  serial:        {rec['serial']}")
-        print(f"  timestamp:     {ev.timestamp}")
-        print(f"  sample_rate:   {ev.sample_rate}")
-        print(f"  record_type:   {ev.record_type}")
-        print(f"  rectime_sec:   {ev.rectime_seconds}")
-        nT = len(ev.raw_samples.get('Tran', [])) if ev.raw_samples else 0
-        nV = len(ev.raw_samples.get('Vert', [])) if ev.raw_samples else 0
-        nL = len(ev.raw_samples.get('Long', [])) if ev.raw_samples else 0
-        nM = len(ev.raw_samples.get('MicL', [])) if ev.raw_samples else 0
-        print(f"  raw_samples:   Tran={nT} Vert={nV} Long={nL} MicL={nM}")
-        if ev.peak_values:
-            print(f"  peak_values:   tran={ev.peak_values.tran} vert={ev.peak_values.vert} long={ev.peak_values.long}")
-        print(f"  h5 written:    {rec['hdf5_filename']}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/e2e_report.py

@@ -1,102 +0,0 @@
-"""End-to-end Thor report PDF rendering.
-
-Ingests an IDFW + .txt via save_imported_idf, runs gather_report_data
-(faking a minimal DB row), and renders the PDF to disk.
-"""
-from __future__ import annotations
-import sys
-import tempfile
-import json
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from sfm.waveform_store import WaveformStore
-from sfm import report_pdf
-
-
-class FakeDb:
-    """Stand-in for SeismoDb.get_event(); the renderer only needs a few cols."""
-    def __init__(self, event):
-        self.event = event
-
-    def get_event(self, _id):
-        return self.event
-
-
-def main():
-    base = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719"
-    idfw = base / "UM11719_20231219162723.IDFW"
-    txt  = base / "TXT" / f"{idfw.name}.txt"
-
-    with tempfile.TemporaryDirectory() as td:
-        store = WaveformStore(Path(td))
-        ev, rec = store.save_imported_idf(
-            idfw.read_bytes(),
-            idfw,
-            idf_report_text=txt.read_text(errors="replace"),
-        )
-        print(f"save_imported_idf: h5={rec['hdf5_filename']}, sidecar={rec['sidecar_filename']}")
-
-        # Verify sidecar has bw_report block
-        sc_path = Path(td) / "UM11719" / f"{idfw.name}.sfm.json"
-        sc = json.loads(sc_path.read_text())
-        bw = sc.get("bw_report", {})
-        print(f"  bw_report.available: {bw.get('available')}")
-        print(f"  bw_report.peaks.tran.ppv_ips: {bw.get('peaks', {}).get('tran', {}).get('ppv_ips')}")
-        print(f"  bw_report.mic.pspl_dbl: {bw.get('mic', {}).get('pspl_dbl')}")
-        print(f"  bw_report.histogram.n_intervals: {bw.get('histogram', {}).get('n_intervals')}")
-
-        # Build a DB-row-shaped dict from the Event for gather_report_data
-        import datetime
-        ts = ev.timestamp
-        ts_iso = None
-        if ts is not None:
-            try:
-                ts_iso = datetime.datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second).isoformat()
-            except Exception:
-                pass
-        fake_row = {
-            "serial":              "UM11719",
-            "blastware_filename":  rec["filename"],
-            "record_type":         "Waveform",
-            "timestamp":           ts_iso,
-            "sample_rate":         ev.sample_rate,
-            "project":             ev.project_info.project if ev.project_info else None,
-            "client":              ev.project_info.client  if ev.project_info else None,
-            "operator":            ev.project_info.operator if ev.project_info else None,
-            "sensor_location":     ev.project_info.sensor_location if ev.project_info else None,
-            "created_at":          None,
-        }
-
-        rd = report_pdf.gather_report_data(FakeDb(fake_row), store, event_id="test-1")
-        print()
-        print(f"=== ReportData ===")
-        print(f"  event_id:           {rd.event_id}")
-        print(f"  serial:             {rd.serial}")
-        print(f"  record_type:        {rd.record_type}")
-        print(f"  event_datetime:     {rd.event_datetime_str}")
-        print(f"  trigger:            {rd.trigger_source}")
-        print(f"  geo_range:          {rd.geo_range_str}")
-        print(f"  sample_rate:        {rd.sample_rate_str}")
-        print(f"  firmware:           {rd.firmware}")
-        print(f"  calibration:        {rd.calibration_date} by {rd.calibration_by}")
-        print(f"  battery:            {rd.battery_volts}")
-        print(f"  PVS:                {rd.peak_vector_sum_ips} in/s at {rd.peak_vector_sum_time_s} sec")
-        print(f"  mic_pspl_dbl:       {rd.mic_pspl_dbl}")
-        print(f"  mic_zc_freq_hz:     {rd.mic_zc_freq_hz}")
-        print(f"  channel_stats:      {len(rd.channel_stats)} rows")
-        for cs in rd.channel_stats:
-            print(f"    {cs['name']}: PPV={cs['ppv_ips']} ZC={cs['zc_freq_hz']} ToP={cs['time_of_peak_s']} Acc={cs['peak_accel_g']} Disp={cs['peak_disp_in']} Test={cs['sensor_check']}")
-
-        # Render the PDF
-        out_path = REPO / "analysis_idf" / "thor_report.pdf"
-        pdf_bytes = report_pdf.render_event_report_pdf(rd)
-        out_path.write_bytes(pdf_bytes)
-        print()
-        print(f"  PDF written: {out_path} ({len(pdf_bytes)} bytes)")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/e2e_report_idfh.py

@@ -1,91 +0,0 @@
-"""End-to-end Thor IDFH histogram report PDF rendering."""
-from __future__ import annotations
-import sys
-import tempfile
-import json
-import datetime
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from sfm.waveform_store import WaveformStore
-from sfm import report_pdf
-
-
-class FakeDb:
-    def __init__(self, event):
-        self.event = event
-
-    def get_event(self, _id):
-        return self.event
-
-
-def main():
-    # Use the multi-interval IDFH (81 + trigger row)
-    idfh = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM13981/UM13981_20220805075441.IDFH"
-    txt  = idfh.parent / "TXT" / f"{idfh.name}.txt"
-
-    with tempfile.TemporaryDirectory() as td:
-        store = WaveformStore(Path(td))
-        ev, rec = store.save_imported_idf(
-            idfh.read_bytes(),
-            idfh,
-            idf_report_text=txt.read_text(errors="replace"),
-        )
-        print(f"save_imported_idf: h5={rec['hdf5_filename']}, sidecar={rec['sidecar_filename']}")
-
-        sc_path = Path(td) / "UM13981" / f"{idfh.name}.sfm.json"
-        sc = json.loads(sc_path.read_text())
-        bw = sc.get("bw_report", {})
-        hist = bw.get("histogram", {})
-        print(f"  bw_report.histogram.start:           {hist.get('start')}")
-        print(f"  bw_report.histogram.stop:            {hist.get('stop')}")
-        print(f"  bw_report.histogram.n_intervals:     {hist.get('n_intervals')}")
-        print(f"  bw_report.histogram.interval_size:   {hist.get('interval_size')}")
-        print(f"  bw_report.histogram.interval_size_s: {hist.get('interval_size_s')}")
-        print(f"  bw_report.peaks.tran.ppv_ips:        {bw.get('peaks', {}).get('tran', {}).get('ppv_ips')}")
-
-        ts = ev.timestamp
-        ts_iso = None
-        if ts is not None:
-            try:
-                ts_iso = datetime.datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second).isoformat()
-            except Exception:
-                pass
-        fake_row = {
-            "serial":              "UM13981",
-            "blastware_filename":  rec["filename"],
-            "record_type":         "Histogram",
-            "timestamp":           ts_iso,
-            "sample_rate":         ev.sample_rate,
-            "project":             ev.project_info.project if ev.project_info else None,
-            "client":              ev.project_info.client  if ev.project_info else None,
-            "operator":            ev.project_info.operator if ev.project_info else None,
-            "sensor_location":     ev.project_info.sensor_location if ev.project_info else None,
-            "created_at":          None,
-        }
-        rd = report_pdf.gather_report_data(FakeDb(fake_row), store, event_id="hist-1")
-
-        print()
-        print("=== ReportData (histogram) ===")
-        print(f"  is_histogram:           {rd.is_histogram}")
-        print(f"  histogram_start:        {rd.histogram_start_str}")
-        print(f"  histogram_stop:         {rd.histogram_stop_str}")
-        print(f"  histogram_n_intervals:  {rd.histogram_n_intervals}")
-        print(f"  histogram_interval_size:{rd.histogram_interval_size}")
-        print(f"  histogram_interval_times[:3]: {rd.histogram_interval_times[:3]}")
-        print(f"  histogram_interval_times[-2:]: {rd.histogram_interval_times[-2:]}")
-        print(f"  channel_stats: {len(rd.channel_stats)} rows")
-        for cs in rd.channel_stats:
-            print(f"    {cs['name']}: PPV={cs['ppv_ips']} ZC={cs['zc_freq_hz']} peak_date={cs['peak_date']} peak_time={cs['peak_time']}")
-
-        pdf_bytes = report_pdf.render_event_report_pdf(rd)
-        out_path = REPO / "analysis_idf" / "thor_report_idfh.pdf"
-        out_path.write_bytes(pdf_bytes)
-        print()
-        print(f"  PDF written: {out_path} ({len(pdf_bytes)} bytes)")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/e2e_save_idf.py

@@ -1,52 +0,0 @@
-"""End-to-end ingest test: feed an IDFW + .txt to save_imported_idf in a tmp store."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-import tempfile
-import shutil
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from sfm.waveform_store import WaveformStore
-
-
-def main():
-    idfw = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162723.IDFW"
-    txt  = idfw.parent / "TXT" / f"{idfw.name}.txt"
-
-    with tempfile.TemporaryDirectory() as td:
-        store = WaveformStore(Path(td))
-        ev, rec = store.save_imported_idf(
-            idfw.read_bytes(),
-            idfw,
-            serial_hint=None,
-            idf_report_text=txt.read_text(errors="replace"),
-        )
-        print("=== Save result ===")
-        print(f"  serial:    {rec['serial']}")
-        print(f"  filename:  {rec['filename']}")
-        print(f"  filesize:  {rec['filesize']}")
-        print(f"  h5:        {rec['hdf5_filename']}")
-        print(f"  sidecar:   {rec['sidecar_filename']}")
-        print()
-        print("=== Event ===")
-        print(f"  serial:        {ev.serial if hasattr(ev,'serial') else '(n/a)'}")
-        print(f"  timestamp:     {ev.timestamp}")
-        print(f"  sample_rate:   {ev.sample_rate}")
-        print(f"  record_type:   {ev.record_type}")
-        print(f"  rectime_sec:   {ev.rectime_seconds}")
-        print(f"  raw_samples:   Tran={len(ev.raw_samples.get('Tran', [])) if ev.raw_samples else 0}, Vert={len(ev.raw_samples.get('Vert', [])) if ev.raw_samples else 0}, Long={len(ev.raw_samples.get('Long', [])) if ev.raw_samples else 0}, MicL={len(ev.raw_samples.get('MicL', [])) if ev.raw_samples else 0}")
-        if ev.peak_values:
-            print(f"  peaks (txt):   Tran={ev.peak_values.tran} Vert={ev.peak_values.vert} Long={ev.peak_values.long}")
-        print()
-
-        # Verify the h5 file actually got written
-        h5path = Path(td) / "UM11719" / f"{idfw.name}.h5"
-        print(f"  h5 exists:     {h5path.exists()}  size={h5path.stat().st_size if h5path.exists() else 0}")
-        sidecar = Path(td) / "UM11719" / f"{idfw.name}.sfm.json"
-        print(f"  sidecar exists:{sidecar.exists()}  size={sidecar.stat().st_size if sidecar.exists() else 0}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/idfh_decode.py

@@ -1,137 +0,0 @@
-"""Decode IDFH histogram intervals + verify against sidecar."""
-from __future__ import annotations
-import sys
-import struct
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-
-SEGMENT_MAGIC = b"\x02\xda\x0a\x00\x00\x00"
-SEGMENT_SIZE = 732   # = 10-byte header + 10 × 72-byte intervals + 2-byte tail
-INTERVAL_SIZE = 72
-CHANNELS = ("Tran", "Vert", "Long", "MicL")
-
-
-def decode_interval(buf72: bytes) -> dict:
-    """Decode one 72-byte interval into per-channel min/max/halfp."""
-    out = {}
-    for i, ch in enumerate(CHANNELS):
-        block = buf72[i*16 : (i+1)*16]
-        mn = struct.unpack_from(">h", block, 0)[0]
-        mx = struct.unpack_from(">h", block, 2)[0]
-        sb = struct.unpack_from(">h", block, 4)[0]
-        halfp = struct.unpack_from(">H", block, 6)[0]
-        f10 = struct.unpack_from(">H", block, 10)[0]
-        f14 = struct.unpack_from(">H", block, 14)[0]
-        peak_count = max(abs(mn), abs(mx))
-        out[ch] = {
-            "min":     mn,
-            "max":     mx,
-            "field4":  sb,
-            "halfp":   halfp,
-            "field10": f10,
-            "field14": f14,
-            "peak":    peak_count,
-            "freq_hz": (512.0 / halfp) if halfp > 5 else None,
-        }
-    out["_tail"] = buf72[64:].hex(" ")
-    return out
-
-
-def walk_idfh(buf: bytes) -> list:
-    """Walk all interval records in an IDFH file."""
-    intervals = []
-    # Multi-segment file: every 02 da 0a 00 00 00 marker introduces a segment.
-    # Single-interval file: just one body header at 0xf96 of form ?? ?? 0a 00 00 00.
-    # Find them all.
-    i = 0
-    while True:
-        j = buf.find(b"\x0a\x00\x00\x00", i)
-        if j < 0:
-            break
-        # Validate: the 2 bytes before must form a length, and we want bytes
-        # [j-2 : j+6] to have a recognisable shape.  Actually the cleanest
-        # filter is "preceded by a length and followed by 00 NN 05 3f".
-        if j < 2:
-            i = j + 1
-            continue
-        # Body header form: [length_be_2][0a 00 00 00][00 NN][05 3f]
-        if j + 10 > len(buf):
-            break
-        length = int.from_bytes(buf[j-2:j], "big")
-        # Verify the segment-marker shape: [length_be][0a 00 00 00][00 NN][05 3f]
-        if buf[j+4] != 0x00:
-            i = j + 1
-            continue
-        if buf[j+6:j+8] != b"\x05\x3f":
-            i = j + 1
-            continue
-        # Header layout (10 bytes): [length_be 2B][0a 00 00 00 4B][00 NN 2B][05 3f 2B]
-        # Followed by N interval records of 72 bytes each, then 2 tail bytes.
-        # length value = (N × 72) + 10  (counts bytes from 0x0a... through interval data).
-        header_start = j - 2
-        n_intervals = (length - 10) // INTERVAL_SIZE
-        interval_start = header_start + 10
-        for k in range(n_intervals):
-            off = interval_start + k * INTERVAL_SIZE
-            if off + INTERVAL_SIZE > len(buf):
-                break
-            chunk = buf[off:off + INTERVAL_SIZE]
-            intervals.append({"offset": off, **decode_interval(chunk)})
-        i = header_start + length + 2
-    return intervals
-
-
-def main():
-    # Test against multi-segment IDFH
-    target = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM13981/UM13981_20220805075441.IDFH"
-    sc_path = target.parent / "TXT" / f"{target.name}.txt"
-    buf = target.read_bytes()
-    intervals = walk_idfh(buf)
-    print(f"=== {target.name} ===")
-    print(f"  file size: {len(buf)}")
-    print(f"  decoded intervals: {len(intervals)}")
-    # Show first 2 + last 2
-    sc_rows = []
-    for line in sc_path.read_text(errors="replace").splitlines():
-        if line.startswith("2022-") or line.startswith("2023-"):
-            sc_rows.append(line)
-    print(f"  sidecar rows: {len(sc_rows)}")
-
-    print()
-    for k in [0, 1, 78, 79, 80]:
-        if k >= len(intervals):
-            continue
-        iv = intervals[k]
-        print(f"--- interval {k} @0x{iv['offset']:04x} ---")
-        for ch in CHANNELS:
-            d = iv[ch]
-            peak_ips = d["peak"] / 32768 * 10.0
-            print(f"  {ch}: peak={d['peak']:5d} ({peak_ips:.4f} in/s)  halfp={d['halfp']:5d}  freq={d['freq_hz']}")
-        # sidecar row
-        if k < len(sc_rows):
-            print(f"  SC: {sc_rows[k]}")
-
-    # Test single-interval IDFH
-    print()
-    target2 = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162648.IDFH"
-    sc2 = target2.parent / "TXT" / f"{target2.name}.txt"
-    buf2 = target2.read_bytes()
-    intervals2 = walk_idfh(buf2)
-    print(f"=== {target2.name} ===")
-    print(f"  file size: {len(buf2)}, decoded intervals: {len(intervals2)}")
-    if intervals2:
-        iv = intervals2[0]
-        for ch in CHANNELS:
-            d = iv[ch]
-            peak_ips = d["peak"] / 32768 * 10.0
-            print(f"  {ch}: peak={d['peak']:5d} ({peak_ips:.4f} in/s)  halfp={d['halfp']:5d}  freq={d['freq_hz']}")
-        sc_rows2 = [l for l in sc2.read_text(errors='replace').splitlines() if l.startswith("2023-")]
-        if sc_rows2:
-            print(f"  SC: {sc_rows2[0]}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/idfh_period.py

@@ -1,41 +0,0 @@
-"""Find IDFH interval period via auto-correlation of structural patterns."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-from collections import Counter
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-
-def main():
-    target = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM13981/UM13981_20220805075441.IDFH"
-    buf = target.read_bytes()
-    body_start = 0xF96
-    body_end   = 0x270C
-    body = buf[body_start:body_end]
-    print(f"body size: {len(body)} bytes (file {len(buf)} bytes)")
-
-    # For each candidate interval size, count how many bytes at fixed offsets within
-    # each interval are zero (consistent column-zero pattern indicates correct size).
-    print()
-    print("=== zero-column score by interval size (higher = more likely) ===")
-    best = []
-    for sz in range(16, 100):
-        n = len(body) // sz
-        if n < 30:
-            continue
-        # For each column position within an interval, count how many of n intervals have zero
-        score = 0
-        for col in range(sz):
-            zeros = sum(1 for i in range(n) if body[i*sz + col] == 0)
-            if zeros >= n * 0.9:
-                score += 1
-        best.append((score, sz, n))
-    best.sort(reverse=True)
-    for score, sz, n in best[:10]:
-        print(f"  size={sz:3d}  n_intervals={n}  consistently-zero-cols={score}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/per_file_detail.py

@@ -1,40 +0,0 @@
-"""Per-file accuracy + sample-count details."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from micromate.idf_file import read_idf_file
-from analysis_idf.recon import load_sidecar_samples
-
-
-def main():
-    root = REPO / "tests/fixtures/THORDATA_example"
-    files = sorted([f for f in root.rglob("*.IDFW") if not str(f).endswith(".CDB")])
-    GEO_LSB = 0.0003
-    # Limit to first 15 successful files for detail.
-    shown = 0
-    for f in files:
-        try:
-            res = read_idf_file(f)
-        except Exception:
-            continue
-        sc_path = f.parent / "TXT" / f"{f.name}.txt"
-        if not sc_path.exists():
-            continue
-        sc = load_sidecar_samples(sc_path)
-        sc_tran = [int(round(v / GEO_LSB)) for v in sc["Tran"]]
-        dec = res.samples.get("Tran", [])
-        n = min(len(sc_tran), len(dec))
-        exact = sum(1 for i in range(n) if sc_tran[i] == dec[i]) if n else 0
-        pct = 100.0 * exact / n if n else 0.0
-        print(f"{f.name:40s}  size={f.stat().st_size:6d}  sc_n={len(sc_tran):4d}  dec_n={len(dec):4d}  exact={pct:.1f}%")
-        shown += 1
-        if shown >= 20:
-            break
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/probe_boundary.py

@@ -1,64 +0,0 @@
-"""Look at what's at the divergence boundary."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from minimateplus.waveform_codec import walk_body, find_data_start, parse_segment_header
-from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
-
-
-def main():
-    buf = TARGET.read_bytes()
-    body = buf[0x0f1f:]
-    start = find_data_start(body)
-    print(f"data_start: {start}  (= file offset 0x{0x0f1f + start:04x})")
-
-    blocks = walk_body(body, start)
-    print(f"{len(blocks)} blocks total")
-    print()
-
-    # First 25 blocks
-    print("=== first 30 blocks ===")
-    for i, b in enumerate(blocks[:30]):
-        body_off = 0x0f1f + b.offset
-        if b.tag_hi == 0x40:
-            hdr = parse_segment_header(b)
-            print(f"  [{i:3d}] @0x{body_off:04x}  {b.kind}  (segment header)  counter={hdr['counter'] if hdr else '?'}  field2={hdr['field2'].hex() if hdr else '?'}  anchor={hdr['anchor_bytes'].hex() if hdr else '?'}  tail={hdr['tail'].hex() if hdr else '?'}")
-        else:
-            print(f"  [{i:3d}] @0x{body_off:04x}  {b.kind}  len={b.length}  data={b.data[:16].hex()}")
-    print()
-
-    # Cumulative sample counts per block to find which block contains sample 254
-    print("=== cumulative samples through blocks ===")
-    cur_ch = "Tran"
-    rotation = ["Vert", "Long", "MicL", "Tran"]
-    seg_count = 0
-    samples_in_curseg = 2  # preamble Tran[0], Tran[1]
-    for i, b in enumerate(blocks[:30]):
-        if b.tag_hi == 0x40:
-            seg_count += 1
-            prev_ch = cur_ch
-            cur_ch = rotation[(seg_count - 1) % 4]
-            print(f"  [{i:3d}] 40 02 -> end of {prev_ch} segment, start {cur_ch} (segment {seg_count})")
-            samples_in_curseg = 2  # anchors
-        elif (b.tag_hi & 0xF0) == 0x10:
-            nn = ((b.tag_hi & 0x0F) << 8) | b.tag_lo
-            samples_in_curseg += nn
-            print(f"  [{i:3d}] {b.kind} nibble: +{nn} samples, ch={cur_ch}, ch_total~{samples_in_curseg}")
-        elif (b.tag_hi & 0xF0) == 0x20:
-            nn = ((b.tag_hi & 0x0F) << 8) | b.tag_lo
-            samples_in_curseg += nn
-            print(f"  [{i:3d}] {b.kind} int8: +{nn} samples, ch={cur_ch}, ch_total~{samples_in_curseg}")
-        elif b.tag_hi == 0x00:
-            samples_in_curseg += b.tag_lo
-            print(f"  [{i:3d}] {b.kind} RLE: +{b.tag_lo}, ch={cur_ch}, ch_total~{samples_in_curseg}")
-        elif b.tag_hi == 0x30:
-            samples_in_curseg += b.tag_lo
-            print(f"  [{i:3d}] {b.kind} packed12: +{b.tag_lo} samples, ch={cur_ch}, ch_total~{samples_in_curseg}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/recon.py

@@ -1,89 +0,0 @@
-"""Reconnaissance helpers for cracking the Thor IDFW binary."""
-from __future__ import annotations
-
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-TARGET = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162723.IDFW"
-TXT = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/TXT/UM11719_20231219162723.IDFW.txt"
-
-
-def hex_at(buf: bytes, off: int, n: int = 32) -> str:
-    chunk = buf[off : off + n]
-    hexs = " ".join(f"{b:02x}" for b in chunk)
-    asc = "".join(chr(b) if 32 <= b < 127 else "." for b in chunk)
-    return f"{off:04x}: {hexs}  {asc}"
-
-
-def find_all(buf: bytes, needle: bytes) -> list[int]:
-    out: list[int] = []
-    i = 0
-    while True:
-        j = buf.find(needle, i)
-        if j < 0:
-            break
-        out.append(j)
-        i = j + 1
-    return out
-
-
-def load_sidecar_samples(path: Path) -> dict[str, list[float]]:
-    """Parse the txt sample table — Tran/Vert/Long/MicL."""
-    out = {"Tran": [], "Vert": [], "Long": [], "MicL": []}
-    in_block = False
-    for line in path.read_text(errors="replace").splitlines():
-        if not in_block:
-            if line.strip() == "Waveform Data Channels":
-                in_block = True
-            continue
-        if line.startswith("Waveform Data USB Channels"):
-            break
-        parts = line.split("\t")
-        # First row is the header "\tTran\tVert\tLong\tMicL"
-        if len(parts) >= 5 and parts[1] == "Tran":
-            continue
-        if len(parts) < 5:
-            continue
-        try:
-            out["Tran"].append(float(parts[1]))
-            out["Vert"].append(float(parts[2]))
-            out["Long"].append(float(parts[3]))
-            out["MicL"].append(float(parts[4]))
-        except ValueError:
-            continue
-    return out
-
-
-def main():
-    buf = TARGET.read_bytes()
-    samples = load_sidecar_samples(TXT)
-    print(f"file size: {len(buf)} bytes")
-    print(f"sample rows: Tran={len(samples['Tran'])} Vert={len(samples['Vert'])} Long={len(samples['Long'])} MicL={len(samples['MicL'])}")
-    print(f"first 6 Tran samples: {samples['Tran'][:6]}")
-    print(f"first 6 Vert samples: {samples['Vert'][:6]}")
-    print(f"first 6 Long samples: {samples['Long'][:6]}")
-    print(f"first 6 MicL samples: {samples['MicL'][:6]}")
-
-    print()
-    print("=== BW magic '00 02 00' positions ===")
-    hits = find_all(buf, b"\x00\x02\x00")
-    print(f"{len(hits)} hits")
-    for h in hits[:20]:
-        print(hex_at(buf, h, 24))
-
-    print()
-    print("=== '40 02' segment-header positions ===")
-    hits = find_all(buf, b"\x40\x02")
-    print(f"{len(hits)} hits")
-    for h in hits:
-        ctx_pre = buf[max(0, h - 4): h].hex()
-        ctx_post = buf[h: h + 20].hex()
-        # Show byte preceding to help identify real headers vs casual occurrences
-        print(f"  0x{h:04x}  pre={ctx_pre}  post={ctx_post}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/seg_resync.py

@@ -1,40 +0,0 @@
-"""Find each segment boundary in the channel and check if errors reset there."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from minimateplus.waveform_codec import decode_waveform_v2
-from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
-
-
-def main():
-    buf = TARGET.read_bytes()
-    sc = load_sidecar_samples(TXT)
-    decoded = decode_waveform_v2(buf[0x0f1f:])
-    GEO_LSB = 0.0003
-
-    for ch in ("Tran", "Vert", "Long"):
-        sc_counts = [int(round(v / GEO_LSB)) for v in sc[ch]]
-        dec = decoded[ch]
-        # Find every transition where error becomes zero from nonzero (or grows from zero)
-        # Print indices where dec resyncs back to exact match.
-        n = min(len(sc_counts), len(dec))
-        events = []
-        prev_match = True
-        for i in range(n):
-            match = sc_counts[i] == dec[i]
-            if match != prev_match:
-                kind = "RESYNC" if match else "DIVERGE"
-                events.append((i, kind, sc_counts[i], dec[i]))
-                prev_match = match
-        print(f"{ch}: {len(events)} transitions")
-        for i, kind, sc_v, dec_v in events[:20]:
-            print(f"  idx {i:4d}  {kind:8s}  sc={sc_v:6d}  dec={dec_v:6d}  diff={dec_v-sc_v:+d}")
-        print()
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/smoke_idfh.py

@@ -1,46 +0,0 @@
-"""Smoke-test read_idf_file on IDFH across the corpus."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from micromate.idf_file import read_idf_file
-
-
-def main():
-    target = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162648.IDFH"
-    result = read_idf_file(target)
-    ev = result.event
-    print(f"=== {target.name} ===")
-    print(f"  signature:   {result.signature}")
-    print(f"  serial:      {ev.serial}")
-    print(f"  timestamp:   {ev.timestamp}")
-    print(f"  sample_rate: {ev.sample_rate}")
-    print(f"  kind:        {ev.kind}")
-    print(f"  intervals:   {len(result.intervals or [])}")
-    print(f"  peaks:       T={ev.peaks.transverse_ips:.4f} V={ev.peaks.vertical_ips:.4f} L={ev.peaks.longitudinal_ips:.4f}")
-    print()
-
-    root = REPO / "tests/fixtures/THORDATA_example"
-    files = list(root.rglob("*.IDFH"))
-    ok = fail = nyi = 0
-    total_intervals = 0
-    for f in files:
-        try:
-            r = read_idf_file(f)
-            ok += 1
-            total_intervals += len(r.intervals or [])
-        except NotImplementedError:
-            nyi += 1
-        except Exception as exc:
-            fail += 1
-            if fail <= 3:
-                print(f"  FAIL: {f.name}: {type(exc).__name__}: {exc}")
-    print(f"Corpus: {len(files)} IDFH files | ok={ok} fail={fail} nyi={nyi}")
-    print(f"Total intervals decoded: {total_intervals}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/smoke_test.py

@@ -1,48 +0,0 @@
-"""Smoke-test read_idf_file across the sample corpus."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from micromate.idf_file import read_idf_file, geo_count_to_ips, mic_count_to_psi
-
-
-def main():
-    target = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162723.IDFW"
-    result = read_idf_file(target)
-    ev = result.event
-    print(f"=== {target.name} ===")
-    print(f"  signature: {result.signature}")
-    print(f"  serial:    {ev.serial}")
-    print(f"  timestamp: {ev.timestamp}")
-    print(f"  sample_rate: {ev.sample_rate}")
-    print(f"  record_time: {ev.record_time_sec}")
-    print(f"  calibration: {result.binary_metadata.calibration_date}")
-    print(f"  Tran samples: {len(result.samples['Tran'])}, peak_ips={ev.peaks.transverse_ips:.4f}")
-    print(f"  Vert samples: {len(result.samples['Vert'])}, peak_ips={ev.peaks.vertical_ips:.4f}")
-    print(f"  Long samples: {len(result.samples['Long'])}, peak_ips={ev.peaks.longitudinal_ips:.4f}")
-    print(f"  MicL samples: {len(result.samples['MicL'])}")
-    print()
-
-    # Corpus sweep
-    root = REPO / "tests/fixtures/THORDATA_example"
-    files = [f for f in root.rglob("*.IDFW") if not str(f).endswith(".CDB")]
-    ok = fail = nyi = 0
-    for f in files:
-        try:
-            r = read_idf_file(f)
-            ok += 1
-        except NotImplementedError:
-            nyi += 1
-        except Exception as exc:
-            fail += 1
-            if fail <= 5:
-                print(f"  FAIL: {f.name}: {type(exc).__name__}: {exc}")
-    print()
-    print(f"Corpus: {len(files)} IDFW files | ok={ok} fail={fail} not-implemented={nyi}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/test_adapter.py

@@ -1,47 +0,0 @@
-"""Verify build_bw_report_from_idf against a known sidecar."""
-from __future__ import annotations
-import json
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from micromate.idf_ascii_report import parse_idf_report
-from micromate.idf_to_bw_report import build_bw_report_from_idf
-from micromate.idf_file import read_idf_file
-
-
-def show(prefix: str, d: dict, indent: int = 0):
-    for k, v in d.items():
-        if isinstance(v, dict):
-            print(f"{'  '*indent}{prefix}{k}:")
-            show("", v, indent + 1)
-        else:
-            print(f"{'  '*indent}{prefix}{k}: {v!r}")
-
-
-def main():
-    base = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719"
-    idfw = base / "UM11719_20231219162723.IDFW"
-    txt  = base / "TXT" / f"{idfw.name}.txt"
-
-    report_dict = parse_idf_report(txt.read_text(errors="replace"))
-    res = read_idf_file(idfw)
-    bw = build_bw_report_from_idf(report_dict, binary_md=res.binary_metadata)
-
-    print("=== IDFW → bw_report ===")
-    show("", bw)
-
-    print()
-    print("=== IDFH (single trigger row) ===")
-    idfh = base / "UM11719_20231219162648.IDFH"
-    txt_h = base / "TXT" / f"{idfh.name}.txt"
-    rh = parse_idf_report(txt_h.read_text(errors="replace"))
-    res_h = read_idf_file(idfh)
-    bw_h = build_bw_report_from_idf(rh, binary_md=res_h.binary_metadata, intervals=res_h.intervals)
-    show("", bw_h)
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/trace_path.py

@@ -1,73 +0,0 @@
-"""Trace Tran sample-by-sample to find exactly where the codec drifts."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
-
-
-def s4(n: int) -> int:
-    return n if n < 8 else n - 16
-
-
-def i8(b: int) -> int:
-    return b if b < 128 else b - 256
-
-
-def main():
-    buf = TARGET.read_bytes()
-    sc = load_sidecar_samples(TXT)
-    GEO_LSB = 0.0003
-    sc_tran = [int(round(v / GEO_LSB)) for v in sc["Tran"]]
-
-    body = buf[0x0f1f:]
-    # Tran[0], Tran[1] from preamble
-    t0 = int.from_bytes(body[3:5], "big", signed=True)
-    t1 = int.from_bytes(body[5:7], "big", signed=True)
-    print(f"preamble Tran[0]={t0}  Tran[1]={t1}  (sidecar: {sc_tran[0]}, {sc_tran[1]})")
-
-    # Block 0: 10 f8 at body[7:9]
-    print(f"block 0: tag {body[7]:02x} {body[8]:02x}")
-    print(f"  block 0 first 10 data bytes: {body[9:19].hex()}")
-
-    # Walk block 0 manually, comparing each sample
-    cur = t1
-    samples = [t0, t1]
-    block_off = 7
-    nn = body[8]
-    print(f"  NN = {nn}")
-    data = body[9 : 9 + nn // 2]
-    for byi, byte in enumerate(data):
-        for nib_idx, nib in enumerate(((byte >> 4) & 0xF, byte & 0xF)):
-            cur += s4(nib)
-            samples.append(cur)
-            idx = len(samples) - 1
-            if 0 <= idx < len(sc_tran):
-                sc_v = sc_tran[idx]
-                match = "✓" if sc_v == cur else "✗"
-                if idx < 12 or 240 <= idx <= 260:
-                    print(f"    idx {idx:3d}: nibble byte={byte:02x} nib={nib:x} delta={s4(nib):+d}  cur={cur:+d}  sc={sc_v:+d}  {match}")
-
-    print(f"end of block 0: cur={cur}, len(samples)={len(samples)}, decoder expected 250 here")
-    # Block 1: 20 28 starts at offset 9 + 124 = 133 from block_off=7
-    block1_off = 9 + nn // 2
-    print(f"block 1: tag {body[block1_off]:02x} {body[block1_off+1]:02x} (expecting 20 28)")
-    nn1 = body[block1_off + 1]
-    print(f"  block 1 NN = {nn1}")
-    data1 = body[block1_off + 2 : block1_off + 2 + nn1]
-    for byi, byte in enumerate(data1):
-        cur += i8(byte)
-        samples.append(cur)
-        idx = len(samples) - 1
-        if idx < len(sc_tran):
-            sc_v = sc_tran[idx]
-            match = "✓" if sc_v == cur else "✗"
-            if 248 <= idx <= 295:
-                print(f"    idx {idx:3d}: int8 byte={byte:02x} delta={i8(byte):+d}  cur={cur:+d}  sc={sc_v:+d}  {match}")
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/try_codec.py

@@ -1,42 +0,0 @@
-"""Feed candidate body offsets to the BW codec and compare with sidecar."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from minimateplus.waveform_codec import decode_waveform_v2, walk_body, find_data_start
-from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
-
-
-def main():
-    buf = TARGET.read_bytes()
-    sc = load_sidecar_samples(TXT)
-    # Sidecar samples in 0.0003 counts (Thor geo LSB).
-    sc_tran = [int(round(v / 0.0003)) for v in sc["Tran"][:30]]
-    sc_vert = [int(round(v / 0.0003)) for v in sc["Vert"][:30]]
-    sc_long = [int(round(v / 0.0003)) for v in sc["Long"][:30]]
-    sc_micl = [int(round(v / 1e-6)) for v in sc["MicL"][:30]]  # 1 µ unit for mic? Will iterate.
-    print(f"sidecar Tran (counts): {sc_tran}")
-    print(f"sidecar Vert (counts): {sc_vert}")
-    print(f"sidecar Long (counts): {sc_long}")
-    print(f"sidecar MicL (×1e-6):  {sc_micl}")
-    print()
-
-    # Try candidate body start offsets.
-    for off in (0x0f1f, 0x1057, 0x11f1, 0x1333, 0x1bde, 0x0d30):
-        print(f"=== body @ 0x{off:04x} ===")
-        body = buf[off:]
-        decoded = decode_waveform_v2(body)
-        if not decoded:
-            print("  decode_waveform_v2 returned None")
-            continue
-        for ch in ("Tran", "Vert", "Long", "MicL"):
-            arr = decoded.get(ch, [])
-            print(f"  {ch}[{len(arr)}]: {arr[:20]}")
-        print()
-
-
-if __name__ == "__main__":
-    main()

analysis_idf/verify_full.py

@@ -1,51 +0,0 @@
-"""Verify decode_waveform_v2 against sidecar across all 2304 samples per channel."""
-from __future__ import annotations
-import sys
-from pathlib import Path
-
-REPO = Path(__file__).resolve().parents[1]
-sys.path.insert(0, str(REPO))
-
-from minimateplus.waveform_codec import decode_waveform_v2
-from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
-
-
-def main():
-    buf = TARGET.read_bytes()
-    sc = load_sidecar_samples(TXT)
-    body = buf[0x0f1f:]
-    decoded = decode_waveform_v2(body)
-
-    print(f"Sidecar lengths: Tran={len(sc['Tran'])} Vert={len(sc['Vert'])} Long={len(sc['Long'])} MicL={len(sc['MicL'])}")
-    print(f"Decoded lengths: Tran={len(decoded['Tran'])} Vert={len(decoded['Vert'])} Long={len(decoded['Long'])} MicL={len(decoded['MicL'])}")
-    print()
-
-    GEO_LSB = 0.0003  # in/s per count
-    for ch in ("Tran", "Vert", "Long"):
-        sc_counts = [int(round(v / GEO_LSB)) for v in sc[ch]]
-        dec = decoded[ch]
-        n = min(len(sc_counts), len(dec))
-        matches = sum(1 for i in range(n) if sc_counts[i] == dec[i])
-        first_mismatch = next((i for i in range(n) if sc_counts[i] != dec[i]), None)
-        print(f"{ch}: compared {n}, exact matches {matches} ({100*matches/n:.2f}%)")
-        if first_mismatch is not None:
-            i = first_mismatch
-            print(f"  first mismatch at idx {i}: sidecar={sc_counts[i]} ({sc[ch][i]}), decoded={dec[i]}")
-            print(f"  context sidecar[{i-2}..{i+5}]: {sc_counts[max(0,i-2):i+5]}")
-            print(f"  context decoded[{i-2}..{i+5}]: {dec[max(0,i-2):i+5]}")
-
-    # MicL: find the multiplicative factor that fits
-    print()
-    print("=== MicL scale analysis ===")
-    sc_micl = sc["MicL"]
-    dec_micl = decoded["MicL"]
-    # Skip zero values when computing ratio
-    ratios = [sc_micl[i] / dec_micl[i] for i in range(min(50, len(sc_micl), len(dec_micl))) if dec_micl[i] != 0]
-    if ratios:
-        avg = sum(ratios) / len(ratios)
-        print(f"  avg ratio sidecar/decoded over first 50 nonzero: {avg:.4e} (n={len(ratios)})")
-        print(f"  ratios sample: {[f'{r:.4e}' for r in ratios[:6]]}")
-
-
-if __name__ == "__main__":
-    main()

bridges/ach_server.py

@@ -177,8 +177,6 @@ class AchSession:
         store: "WaveformStore",
         clear_after_download: bool = False,
         restart_monitoring: bool = False,
-        rescue_stop_monitoring: bool = False,
-        rescue_disable_ach: bool = False,
         force_redownload: bool = False,
     ) -> None:
         self.sock                 = sock
@@ -192,9 +190,6 @@ class AchSession:
         self.store                = store
         self.clear_after_download = clear_after_download
         self.restart_monitoring   = restart_monitoring
-        # Rescue actions for a runaway unit — fired before the event walk.
-        self.rescue_stop_monitoring = rescue_stop_monitoring
-        self.rescue_disable_ach     = rescue_disable_ach
         # `force_redownload` tells this session to ignore ach_state and
         # re-download every event currently on the device, regardless of any
         # (key, timestamp) match.  Useful as a manual override when state has
@@ -295,41 +290,6 @@ class AchSession:
         root_logger.addHandler(fh)
 
         try:
-            # ── Step 1.5: rescue actions ──────────────────────────────────────
-            # Fired BEFORE the event walk so a runaway unit is quieted as early
-            # in the session as possible.  A unit whose geophone sits above the
-            # trigger threshold records back-to-back and, with ACH set to "after
-            # event recorded", re-dials every time — saturating its own firmware
-            # so it never services inbound requests.  See
-            # docs/runbooks/wedged_unit_recovery.md.
-            #
-            # Each action is independently guarded: a failure here must not
-            # abort the download that follows.
-            if self.rescue_stop_monitoring or self.rescue_disable_ach:
-                rescue: dict = {"peer": self.peer, "ts": ts}
-
-                if self.rescue_stop_monitoring:
-                    log.info("Step 1.5: RESCUE — stop monitoring (SUB 0x97)")
-                    try:
-                        client.stop_monitoring()
-                        rescue["stop_monitoring"] = "ok"
-                        log.info("  stop monitoring OK — device should stop recording")
-                    except Exception as exc:
-                        rescue["stop_monitoring"] = f"failed: {exc}"
-                        log.error("  stop monitoring FAILED: %s", exc)
-
-                if self.rescue_disable_ach:
-                    log.info("Step 1.5: RESCUE — disable auto call home (SUB 0x2C/0x7E/0x7F)")
-                    try:
-                        client.set_call_home_config(auto_call_home_enabled=False)
-                        rescue["disable_ach"] = "ok"
-                        log.info("  disable ACH OK — unit should stop calling home")
-                    except Exception as exc:
-                        rescue["disable_ach"] = f"failed: {exc}"
-                        log.error("  disable ACH FAILED: %s", exc)
-
-                _save_json(session_dir / "rescue.json", rescue)
-
             # ── Step 2: device info ───────────────────────────────────────────
             device_info = None
             if not self.events_only:
@@ -787,13 +747,6 @@ def serve(args: argparse.Namespace) -> None:
     print(f"  Max events per session: {max_ev if max_ev else 'unlimited'}")
     print(f"  Clear device after download: {'YES' if args.clear_after_download else 'no'}")
     print(f"  Restart monitoring after download: {'YES' if args.restart_monitoring else 'no'}")
-    _stop_mon = args.stop_monitoring or args.rescue
-    _dis_ach  = args.disable_ach or args.rescue
-    print(f"  RESCUE stop monitoring on connect:  {'YES' if _stop_mon else 'no'}")
-    print(f"  RESCUE disable auto call home:      {'YES' if _dis_ach else 'no'}")
-    if _stop_mon and args.restart_monitoring:
-        print("  !! --restart-monitoring will re-start the unit after download,")
-        print("     undoing --stop-monitoring.  Drop one of them.")
     print(f"  Force re-download all (ignore state): {'YES' if args.force_redownload_all else 'no'}")
     print(f"{'='*60}")
     print(f"\n  Point your test unit's ACEmanager call-home settings to:")
@@ -835,8 +788,6 @@ def serve(args: argparse.Namespace) -> None:
                     store=store,
                     clear_after_download=args.clear_after_download,
                     restart_monitoring=args.restart_monitoring,
-                    rescue_stop_monitoring=args.stop_monitoring or args.rescue,
-                    rescue_disable_ach=args.disable_ach or args.rescue,
                     force_redownload=args.force_redownload_all,
                 )
                 t = threading.Thread(target=session.run, daemon=True, name=f"ach-{peer}")
@@ -911,32 +862,6 @@ def parse_args() -> argparse.Namespace:
             "DCD on disconnect — without this the unit stays idle after a call-home."
         ),
     )
-    p.add_argument(
-        "--stop-monitoring",
-        action="store_true",
-        default=False,
-        help=(
-            "RESCUE: send SUB 0x97 (stop monitoring) immediately after the "
-            "handshake, before any event download. Use on a unit that is "
-            "recording back-to-back because of a stuck-triggered geophone."
-        ),
-    )
-    p.add_argument(
-        "--disable-ach",
-        action="store_true",
-        default=False,
-        help=(
-            "RESCUE: disable Auto Call Home on the device (SUB 0x2C read → "
-            "0x7E write → 0x7F confirm) immediately after the handshake. The "
-            "unit stops dialing out until ACH is explicitly re-enabled."
-        ),
-    )
-    p.add_argument(
-        "--rescue",
-        action="store_true",
-        default=False,
-        help="Shorthand for --stop-monitoring --disable-ach.",
-    )
     p.add_argument(
         "--clear-after-download",
         action="store_true",

bridges/mm_link.py

@@ -1,338 +0,0 @@
-#!/usr/bin/env python3
-"""
-mm_link.py — a "perfect modem" between THOR and a Micromate, with a readable
-log and deliberate fault injection.
-
-Why
----
-THOR gives almost no visibility into a connection: a refresh button, two poll
-intervals, and no way to see whether a check succeeded, timed out, or was never
-sent.  When a unit "won't stay connected" there is nothing to look at.
-
-This sits where the cellular modem would sit and answers the question directly:
-
-  * **What is THOR actually doing?**  Every frame is decoded and timestamped —
-    `POLL`, `MONITOR_STATUS`, `SETUP_NAME_READ` — not a hex dump.
-  * **Is it even trying?**  Silence is visible: the log shows gaps.
-  * **How does it behave when the link misbehaves?**  Faults can be injected on
-    demand, which a real cell link will not do on cue.
-
-Point THOR at this host and port exactly as if it were a modem (Communication:
-TCP, IP: <this host>, Port: <--listen>).
-
-Fault injection
----------------
-Write a mode into the control file (default `mm_link.ctl`) and it takes effect
-on the next byte:
-
-    echo pass      > mm_link.ctl    # normal relay
-    echo blackhole > mm_link.ctl    # TCP stays up, bytes are swallowed
-    echo drop      > mm_link.ctl    # close the connection abruptly (RST-ish)
-    echo delay:2.0 > mm_link.ctl    # forward, but 2 s late in both directions
-    echo onewaydev > mm_link.ctl    # THOR->unit passes, unit->THOR is swallowed
-
-**`blackhole` is the one that matters.**  It reproduces the classic cellular
-failure: the socket is still open as far as both ends are concerned, but nothing
-crosses.  A client that relies on TCP to tell it the peer is gone will sit there
-until the OS keepalive fires — which by default is about two hours.
-
-Usage
------
-    python3 bridges/mm_link.py --serial /dev/ttyACM0 --baud 115200 \\
-        --listen 12345 --logdir ~/mm-captures
-
-Writes, per session:
-    <logdir>/mmlink_<ts>/session.log   decoded, timestamped, human-readable
-    <logdir>/mmlink_<ts>/raw_bw.bin    THOR -> unit, raw
-    <logdir>/mmlink_<ts>/raw_s3.bin    unit -> THOR, raw
-
-The raw pair loads straight into `scratch/mm_frame_parse.py`.
-"""
-
-from __future__ import annotations
-
-import argparse
-import datetime
-import os
-import socket
-import sys
-import threading
-import time
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "scratch"))
-try:
-    from mm_frame_parse import SUBNAME, destuff  # noqa: F401
-except Exception:                                 # pragma: no cover
-    SUBNAME = {}
-
-import errno
-import select
-import termios
-
-DLE, STX, ETX, ACK = 0x10, 0x02, 0x03, 0x41
-
-_BAUD = {9600: termios.B9600, 19200: termios.B19200, 38400: termios.B38400,
-         57600: termios.B57600, 115200: termios.B115200}
-
-
-class SerialPort:
-    """Minimal raw serial port on stdlib termios — no pyserial dependency.
-
-    The bench hosts are whatever is to hand; requiring a pip install on someone
-    else's machine is a poor trade for the ~30 lines this saves.
-    """
-
-    def __init__(self, path: str, baud: int):
-        if baud not in _BAUD:
-            raise ValueError(f"unsupported baud {baud}; pick one of {sorted(_BAUD)}")
-        self.fd = os.open(path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
-        a = termios.tcgetattr(self.fd)
-        a[0] = 0                                             # iflag: no translation
-        a[1] = 0                                             # oflag: raw
-        a[2] = termios.CS8 | termios.CREAD | termios.CLOCAL  # cflag: 8N1, ignore modem lines
-        a[3] = 0                                             # lflag: non-canonical, no echo
-        a[4] = a[5] = _BAUD[baud]
-        a[6] = list(a[6])
-        a[6][termios.VMIN] = 0
-        a[6][termios.VTIME] = 0
-        termios.tcsetattr(self.fd, termios.TCSANOW, a)
-        termios.tcflush(self.fd, termios.TCIOFLUSH)
-
-    def read(self, n: int) -> bytes:
-        r, _, _ = select.select([self.fd], [], [], 0.2)
-        if not r:
-            return b""
-        try:
-            return os.read(self.fd, n)
-        except OSError as e:
-            if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK):
-                return b""
-            raise
-
-    def write(self, data: bytes) -> None:
-        while data:
-            try:
-                data = data[os.write(self.fd, data):]
-            except OSError as e:
-                if e.errno in (errno.EAGAIN, errno.EWOULDBLOCK):
-                    select.select([], [self.fd], [], 0.2)
-                    continue
-                raise
-
-    def close(self) -> None:
-        try:
-            os.close(self.fd)
-        except OSError:
-            pass
-
-
-def name_of(sub: int, is_request: bool) -> str:
-    if is_request:
-        return SUBNAME.get(sub, f"SUB_{sub:02X}")
-    return "rsp " + SUBNAME.get(0xFF - sub, f"SUB_{0xFF - sub:02X}")
-
-
-class FrameSniffer:
-    """Accumulate bytes and report complete frames, without altering the stream."""
-
-    def __init__(self, is_request: bool):
-        self.is_request = is_request
-        self.buf = bytearray()
-
-    def feed(self, data: bytes):
-        """Yield (sub, payload_len) for each complete frame seen."""
-        self.buf.extend(data)
-        while True:
-            start = -1
-            for i, b in enumerate(self.buf):
-                if self.is_request and b == ACK and i + 1 < len(self.buf) and self.buf[i + 1] == STX:
-                    start = i
-                    break
-                if not self.is_request and b == STX:
-                    start = i
-                    break
-            if start < 0:
-                if len(self.buf) > 8192:
-                    del self.buf[:-16]
-                return
-            j = start + (2 if self.is_request else 1)
-            end = -1
-            while j < len(self.buf):
-                if self.buf[j] == DLE and j + 1 < len(self.buf):
-                    j += 2
-                    continue
-                if self.buf[j] == ETX:
-                    end = j
-                    break
-                j += 1
-            if end < 0:
-                return                      # wait for more bytes
-            body = self.buf[start:end + 1]
-            del self.buf[:end + 1]
-            # SUB sits at a fixed spot past the leading framing -- but it is
-            # DLE-escaped when its own value is 0x02/0x03/0x04/0x10, so a raw
-            # read reports 0x10 for those.  SUB 0x02 was being logged as
-            # "SUB_10" until this was handled.
-            off = 5 if self.is_request else 3
-            if len(body) > off:
-                sub = body[off]
-                if sub == DLE and len(body) > off + 1:
-                    sub = body[off + 1]
-                yield sub, len(body)
-
-
-class Link:
-    def __init__(self, args):
-        self.args = args
-        self.mode = "pass"
-        self.delay = 0.0
-        self.ctl = Path(args.control)
-        self.session: Path | None = None
-        self.log_fh = None
-        self.raw = {}
-        self.t0 = time.time()
-        self.counts = {}
-
-    # ── logging ────────────────────────────────────────────────────────────
-    def open_session(self):
-        ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
-        self.session = Path(self.args.logdir) / f"mmlink_{ts}"
-        self.session.mkdir(parents=True, exist_ok=True)
-        self.log_fh = open(self.session / "session.log", "a", buffering=1)
-        self.raw = {
-            "bw": open(self.session / "raw_bw.bin", "ab"),
-            "s3": open(self.session / "raw_s3.bin", "ab"),
-        }
-        self.say(f"=== session {ts} — serial {self.args.serial} @ {self.args.baud} ===")
-
-    def say(self, text: str):
-        line = f"{datetime.datetime.now().strftime('%H:%M:%S.%f')[:-3]}  {text}"
-        print(line, flush=True)
-        if self.log_fh:
-            self.log_fh.write(line + "\n")
-
-    # ── control file ───────────────────────────────────────────────────────
-    def poll_control(self):
-        while True:
-            try:
-                if self.ctl.exists():
-                    want = self.ctl.read_text().strip().lower()
-                    if want.startswith("delay:"):
-                        d = float(want.split(":", 1)[1])
-                        if ("delay", d) != (self.mode, self.delay):
-                            self.mode, self.delay = "delay", d
-                            self.say(f"*** MODE -> delay {d}s ***")
-                    elif want and want != self.mode:
-                        self.mode, self.delay = want, 0.0
-                        self.say(f"*** MODE -> {want} ***")
-            except Exception:
-                pass
-            time.sleep(0.25)
-
-    # ── the relay ──────────────────────────────────────────────────────────
-    def pump(self, src, dst, tag: str, is_request: bool, stop: threading.Event):
-        sniff = FrameSniffer(is_request)
-        arrow = "THOR->unit" if is_request else "unit->THOR"
-        last = time.time()
-        while not stop.is_set():
-            timed_out = False
-            try:
-                data = src.recv(4096) if isinstance(src, socket.socket) else src.read(4096)
-            except TimeoutError:
-                timed_out = True
-                # socket.timeout subclasses OSError, so it MUST be caught first.
-                # Treating it as a dead socket closes the connection after 200 ms
-                # of quiet -- which is exactly what `blackhole` produces, so the
-                # relay killed the link it was supposed to be faking a fault on.
-                data = b""
-            except OSError:
-                break
-            if isinstance(src, socket.socket) and data == b"" and not timed_out:
-                self.say(f"{arrow}: peer closed the connection")
-                break
-            if not data:
-                if time.time() - last > self.args.quiet_after and self.counts:
-                    self.say(f"--- {self.args.quiet_after:.0f}s with no traffic ---")
-                    last = time.time()
-                continue
-            last = time.time()
-
-            self.raw[tag].write(data)
-            self.raw[tag].flush()
-            for sub, ln in sniff.feed(data):
-                label = name_of(sub, is_request)
-                self.counts[label] = self.counts.get(label, 0) + 1
-                self.say(f"{arrow}  {label:<20} ({ln} B)"
-                         + ("" if self.mode == "pass" else f"   [mode={self.mode}]"))
-
-            mode = self.mode
-            if mode == "drop":
-                self.say(f"{arrow}: DROPPING the connection (fault injection)")
-                stop.set()
-                break
-            if mode == "blackhole":
-                continue                      # swallow, keep the socket open
-            if mode == "onewaydev" and not is_request:
-                continue                      # unit's replies never reach THOR
-            if mode == "delay" and self.delay:
-                time.sleep(self.delay)
-            try:
-                if isinstance(dst, socket.socket):
-                    dst.sendall(data)
-                else:
-                    dst.write(data)
-            except OSError:
-                break
-        stop.set()
-
-    def serve(self):
-        srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
-        srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
-        srv.bind(("0.0.0.0", self.args.listen))
-        srv.listen(5)
-        self.open_session()
-        self.say(f"listening on 0.0.0.0:{self.args.listen}   control file: {self.ctl}")
-        self.say("point THOR at this host/port as Communication=TCP")
-        threading.Thread(target=self.poll_control, daemon=True).start()
-
-        while True:
-            conn, addr = srv.accept()
-            conn.settimeout(0.2)
-            self.say(f"+++ THOR connected from {addr[0]}:{addr[1]} +++")
-            try:
-                ser = SerialPort(self.args.serial, self.args.baud)
-            except OSError as e:
-                self.say(f"!!! cannot open {self.args.serial}: {e}")
-                conn.close()
-                continue
-            stop = threading.Event()
-            ts = [
-                threading.Thread(target=self.pump, args=(conn, ser, "bw", True, stop), daemon=True),
-                threading.Thread(target=self.pump, args=(ser, conn, "s3", False, stop), daemon=True),
-            ]
-            for t in ts:
-                t.start()
-            for t in ts:
-                t.join()
-            conn.close()
-            ser.close()
-            summary = ", ".join(f"{k}x{v}" for k, v in sorted(self.counts.items()))
-            self.say(f"--- connection closed.  frames this session: {summary or 'none'} ---")
-
-
-def main():
-    ap = argparse.ArgumentParser(description=__doc__,
-                                 formatter_class=argparse.RawDescriptionHelpFormatter)
-    ap.add_argument("--serial", default="/dev/ttyACM0")
-    ap.add_argument("--baud", type=int, default=115200)
-    ap.add_argument("--listen", type=int, default=12345)
-    ap.add_argument("--logdir", default=os.path.expanduser("~/mm-captures"))
-    ap.add_argument("--control", default="mm_link.ctl")
-    ap.add_argument("--quiet-after", type=float, default=30.0,
-                    help="log a marker after this many seconds of silence")
-    Link(ap.parse_args()).serve()
-
-
-if __name__ == "__main__":
-    main()

bridges/mm_probe.py

@@ -1,226 +0,0 @@
-#!/usr/bin/env python3
-"""
-mm_probe.py — answer "why can't we reach this unit?" in one command.
-
-THOR reports a failed connection as "disconnected" and nothing else.  That single
-word covers at least four completely different faults with four different fixes,
-and telling them apart is the difference between a modem reboot and a site visit:
-
-  * **connection refused**   something answered and said no — wrong port, or the
-                             modem is refusing a further session
-  * **connect timed out**    nothing answered at all — trusted-IP whitelist,
-                             firewall, or the modem is off the network
-  * **connected, no reply**  the MODEM answered but the unit did not.  The TCP
-                             path is fine; the modem is not forwarding to serial.
-                             This is the signature of a wedged transparent-TCP
-                             session, and it is the one THOR cannot distinguish
-                             from any of the others
-  * **replied**              the unit is alive; the problem is upstream software
-
-Read-only.  It sends `POLL`, then optionally `SERIAL` and the state read — the
-same three commands THOR's own connection check uses — and never writes.
-
-Usage
------
-    python3 bridges/mm_probe.py 63.45.161.30:9034
-    python3 bridges/mm_probe.py 10.0.0.8:12345 --timeout 5
-    python3 bridges/mm_probe.py <host:port> --slots 3
-
-`--slots N` opens N connections at once and reports how many the far end accepts.
-A transparent-TCP modem typically serves **one** session; if the first succeeds
-and the rest are refused or hang, that confirms the single-slot behaviour and
-explains why a leaked session takes a unit offline until the slot frees.
-
-Works for both series: a Series III reply opens `DLE STX`, a Micromate reply
-opens with a bare `STX`, so the probe also tells you which one answered.
-"""
-
-from __future__ import annotations
-
-import argparse
-import socket
-import sys
-import time
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-from minimateplus.framing import build_bw_frame  # noqa: E402
-
-DLE, STX, ETX = 0x10, 0x02, 0x03
-
-
-def destuff(raw: bytes) -> bytes:
-    """Strip framing and DLE escapes; return the payload without its checksum."""
-    i = 1 if raw and raw[0] == STX else (2 if len(raw) > 1 and raw[1] == STX else 0)
-    out = bytearray()
-    while i < len(raw):
-        b = raw[i]
-        if b == DLE and i + 1 < len(raw):
-            out.append(raw[i + 1])
-            i += 2
-            continue
-        if b == ETX:
-            break
-        out.append(b)
-        i += 1
-    return bytes(out[:-1]) if len(out) > 1 else b""
-
-
-# Reads are two-step on Series III: a probe at offset 0, then a data read at the
-# block's length.  THOR sends these offsets, and they also work on a Micromate.
-OFFSETS = {0x5B: 0x0030, 0x15: 0x000A, 0x49: 0xFFFF}
-
-
-def exchange(sock: socket.socket, sub: int, timeout: float) -> tuple[bytes, float]:
-    sock.sendall(build_bw_frame(sub, OFFSETS.get(sub, 0)))
-    t0 = time.time()
-    buf, deadline = b"", t0 + timeout
-    sock.settimeout(0.3)
-    while time.time() < deadline:
-        try:
-            chunk = sock.recv(4096)
-            if not chunk:
-                break
-            buf += chunk
-            if buf.endswith(bytes([ETX])) and len(buf) > 8:
-                break
-        except TimeoutError:
-            continue
-        except OSError:
-            break
-    return buf, time.time() - t0
-
-
-def step(n: int, label: str, result: str) -> None:
-    print(f"  [{n}] {label:.<28} {result}")
-
-
-def probe(host: str, port: int, timeout: float) -> int:
-    print(f"\ntarget {host}:{port}   (read-only: POLL, SERIAL, state)\n")
-
-    # ── 1. TCP ────────────────────────────────────────────────────────────
-    t0 = time.time()
-    try:
-        sock = socket.create_connection((host, port), timeout=timeout)
-    except ConnectionRefusedError:
-        step(1, "TCP connect", f"REFUSED after {1000*(time.time()-t0):.0f} ms")
-        print("\nverdict: something answered and actively refused.")
-        print("  Not a silent firewall drop — the host is reachable.")
-        print("  Wrong port, the service is down, or the modem is refusing")
-        print("  an additional session because its one slot is in use.")
-        return 2
-    except (TimeoutError, socket.timeout):
-        step(1, "TCP connect", f"TIMED OUT after {time.time()-t0:.1f} s")
-        print("\nverdict: nothing answered at all.")
-        print("  A silent drop, which is what a trusted-IP whitelist looks")
-        print("  like — it discards rather than refuses.  Check the modem's")
-        print("  Trusted IPs (and note a VPN changes the IP you arrive from),")
-        print("  the firewall, and whether the modem is on the network.")
-        return 3
-    except OSError as e:
-        step(1, "TCP connect", f"FAILED: {e}")
-        return 4
-    step(1, "TCP connect", f"ok ({1000*(time.time()-t0):.0f} ms)")
-
-    # ── 2. POLL ───────────────────────────────────────────────────────────
-    try:
-        raw, dt = exchange(sock, 0x5B, timeout)
-    except OSError as e:
-        step(2, "POLL", f"send failed: {e}")
-        sock.close()
-        return 4
-
-    if not raw:
-        step(2, "POLL", f"NO REPLY in {timeout:.1f} s")
-        print("\nverdict: the MODEM answered but the unit did not.")
-        print("  TCP is fine end to end — something accepted the connection.")
-        print("  What is missing is the serial side.  Most likely the modem is")
-        print("  not forwarding to its serial port, which is what a wedged")
-        print("  transparent-TCP session looks like: the slot is held by a")
-        print("  connection that never closed.")
-        print("\n  Try, in order:")
-        print("    1. ACEmanager -> TCP Idle Timeout.  If 0/disabled, a stale")
-        print("       session holds the slot forever.  2 minutes is the value")
-        print("       this project standardised on.")
-        print("    2. Reboot the modem.  If that fixes it, the modem was")
-        print("       holding state and the timeout is the permanent fix.")
-        print("    3. Check the unit's own screen — serial cable, power.")
-        sock.close()
-        return 5
-
-    series = "Series III (DLE STX)" if raw[0] == DLE else "Micromate (bare STX)"
-    step(2, "POLL", f"reply {len(raw)} B in {1000*dt:.0f} ms")
-    p = destuff(raw)
-    ok = len(p) > 3 and p[2] == 0xFF - 0x5B
-    step(3, "frame", f"{'valid' if ok else 'MALFORMED'}, {series}")
-    if not ok:
-        print("\nverdict: something replied, but not a seismograph.")
-        print("  Another service is on this port, or the modem is in a mode")
-        print("  that injects its own text (check Quiet Mode / AT echo).")
-        print(f"  first bytes: {raw[:16].hex(' ')}")
-        sock.close()
-        return 6
-
-    # ── 3. identity + state ───────────────────────────────────────────────
-    for n, (sub, label) in enumerate(((0x15, "serial"), (0x49, "state")), start=4):
-        try:
-            r, dt = exchange(sock, sub, timeout)
-            d = destuff(r)[5:]
-            if sub == 0x15:
-                # serial is a null-terminated run; a further field follows it
-                serial = bytes(d[11:]).split(b"\x00")[0]
-                step(n, label, serial.decode("ascii", "replace") or "(empty)")
-            else:
-                step(n, label, "MONITORING" if len(d) > 11 and d[11] else "idle")
-        except OSError:
-            step(n, label, "no reply")
-
-    sock.close()
-    print("\nverdict: the unit is alive and answering.")
-    print("  If THOR still shows it disconnected, the fault is in THOR, not")
-    print("  the network or the device.")
-    return 0
-
-
-def slots(host: str, port: int, n: int, timeout: float) -> None:
-    print(f"\nopening {n} simultaneous connections to {host}:{port}\n")
-    held = []
-    for i in range(n):
-        try:
-            s = socket.create_connection((host, port), timeout=timeout)
-            held.append(s)
-            step(i + 1, f"connection {i+1}", "accepted")
-        except ConnectionRefusedError:
-            step(i + 1, f"connection {i+1}", "REFUSED")
-        except (TimeoutError, socket.timeout):
-            step(i + 1, f"connection {i+1}", "timed out")
-        except OSError as e:
-            step(i + 1, f"connection {i+1}", f"failed: {e}")
-    print(f"\n{len(held)} of {n} accepted.")
-    if len(held) == 1:
-        print("  Single-slot behaviour confirmed — this far end serves ONE")
-        print("  session at a time.  A connection that is never closed takes")
-        print("  the unit offline until the idle timeout frees the slot.")
-    for s in held:
-        s.close()
-
-
-def main() -> int:
-    ap = argparse.ArgumentParser(description=__doc__,
-                                 formatter_class=argparse.RawDescriptionHelpFormatter)
-    ap.add_argument("target", help="host:port, e.g. 63.45.161.30:9034")
-    ap.add_argument("--timeout", type=float, default=10.0)
-    ap.add_argument("--slots", type=int, metavar="N",
-                    help="open N simultaneous connections to test single-slot behaviour")
-    a = ap.parse_args()
-    host, _, port = a.target.rpartition(":")
-    if not host:
-        ap.error("target must be host:port")
-    if a.slots:
-        slots(host, int(port), a.slots, a.timeout)
-        return 0
-    return probe(host, int(port), a.timeout)
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

docs/histogram_codec_re_status.md

@@ -1,30 +1,3 @@
-> ## SUPERSEDED 2026-08-25 — the block is uniformly BIG-ENDIAN
->
-> The `uint8` peak / `annotation` byte model described below is wrong,
-> though it decoded quiet data correctly.  The real layout:
->
-> - **Every per-channel field is `uint16` big-endian.**  `T_peak` is
->   `[5:7]`, `T_halfperiod` `[7:9]`, `V_peak` `[9:11]`, and so on.
->   Only `block_ctr` at `[2:4]` is little-endian.
-> - The **marker is `block[4]` alone**, not a `uint16 LE` at `[4:6]`.
->   Testing `[4:6] == 10` forced `block[5] == 0`, which is exactly what
->   capped every geo peak at one byte (255 counts = 1.275 in/s).
-> - The **"annotation" byte was never an annotation** — it is the high
->   byte of the big-endian half-period.  That is why it was non-zero
->   precisely on the sub-Hz intervals Blastware renders as `<1.0`.
-> - The **final block of the stream carries tail `9c 06 00 42`** instead
->   of `1e 0a 00 00`, and arbitrary bytes at `[21:23]`.  Rejecting it
->   dropped the last interval of nearly every histogram — often the one
->   holding the event peak, so the file's PPV read low.
->
-> Verified against 1211 production histograms paired with their Blastware
-> ASCII exports: **1211/1211 decode exactly** (interval count plus every
-> per-interval peak), and 842,442 per-interval frequency comparisons match
-> with zero mismatches.  The uint8 model scored 1204/1211 — the seven
-> failures are exactly the files containing a peak above 1.275 in/s.
->
-> The section below is retained as the reasoning trail.
-
 # Histogram body codec — FULLY DECODED (2026-05-20)
 
 Clean working status doc for the MiniMate Plus histogram-mode event

docs/idf_protocol_reference.md

@@ -6,290 +6,11 @@ Series IV event-file format.  Sibling to
 Series III "Rosetta Stone") — this doc holds what we know so far and
 the open questions still to crack.
 
-> ⚠ **The "Status (2026-05-28)" block below is SUPERSEDED.**  Its geo LSB
+**Status (2026-05-20):** ASCII text sidecar fully decoded (1,014
-> (0.0003), its IDFH scale (`/32768 × 10`), its fixed body offset (`0x0f1f`)
+sample files round-trip).  Binary `.IDFH` / `.IDFW` codec
-> and its "87–99% byte-exact / loud events truncate" caveat were all wrong or
+**not yet implemented** — binaries are stored opaquely by
-> incomplete.  See **[Verified against Thor's own exports
+`WaveformStore.save_imported_idf`, with metadata sourced from the
-> (2026-09-10)](#verified-against-thors-own-exports-2026-09-10)** — the
+paired `.txt` sidecar.
-> decoder is now per-sample exact on 1,057,536/1,057,536 samples.  The block
-> is kept only for the reverse-engineering trail.
-
-**Status (2026-05-28, SUPERSEDED):** ASCII text sidecar fully decoded (1,014
-sample files round-trip).  **Thor IDFW** binary now decodes via
-`micromate.idf_file.read_idf_file()` — reuses the BW segment-rotated
-block codec verbatim at fixed body offset `0x0f1f`; metadata (serial,
-timestamp, sample_rate, record_time, calibration_date) extracted from
-the binary header.  Sample fidelity is 87–99% byte-exact on quiet
-events; loud events hit the BW codec's known walker-stops-early
-limitation.  Residual ~3% drift on per-sample deltas (likely a
-Thor-specific 12-bit delta refinement not yet modelled).
-
-**Thor IDFH histograms also decoded.**  Body has one or more segments;
-each 12-byte segment header `[length_be 2B][0a 00 00 00][00 NN][05 3f]`
-introduces `N = (length - 10) // 72` interval records of 72 bytes
-each.  Each interval = 4 × 16-byte per-channel records:
-`[int16 min][int16 max][int16 ??][uint16 halfp][2B 00][uint16 ??][2B 00][uint16 ??]`.
-Geo peak `= max(|min|, |max|) / 32768 × 10` in/s (matches sidecar
-~1.8%); freq `= 512 / halfp` Hz (None for halfp ≤ 5 → ">100"
-sentinel).  Corpus: **all 859 Thor IDFH files decode, 181,071
-intervals**.  Wired through `read_idf_file()` →
-`save_imported_idf()` → sidecar's `extensions.idf_intervals`.
-
-**Note on the BE9439 outliers in the example corpus:** Two files
-(`BE9439_20200713131747.IDFW` and `BE9439_20200713124251.IDFH`) are
-**Series III Blastware** binaries, not Thor.  Provenance: TMI tried
-to use Thor to manage auto-call-homes for Series III units; the
-experiment didn't work out, but it did leave a few BW event files
-in Thor's per-serial directory structure with `.IDFW`/`.IDFH`
-extensions — Thor's forwarder applied its own naming convention to
-the BW bodies it was relaying.  Their header `10 00 01 80 00 00
-Instantel STRT ff fe <end_key> <start_key>` is the BW SUB 5A STRT
-record, not a Thor body preamble.  The reader detects them by
-signature and raises `NotImplementedError` pointing callers at
-`read_blastware_file()`, which extracts BW-format peaks from them.
-
-**Still NYI for Thor IDFH:** per-channel `int16 field4` (possibly
-time-of-peak); the two uint16 fields (probably PVS contributions);
-8-byte interval tail (PVS data); mic dB(L) exact conversion constant.
-
-## Verified against Thor's own exports (2026-09-10)
-
-**The series-4 decoder is now per-sample exact.**  1,057,536 / 1,057,536
-geophone samples across all 153 genuine Thor waveform files reproduce Thor's
-own CSV export exactly; histogram peaks land within 2% on 858/858 files
-(median error −0.004%).
-
-### Ground truth — it was there all along
-
-Thor writes `TXT/`, `CSV/`, `XML/` and `PDF/` exports beside every binary:
-
-```
-<serial dir>/UM13981_20220207084555.IDFW
-<serial dir>/CSV/UM13981_20220207084555.IDFW.csv
-```
-
-The **CSV carries a per-sample block** — four columns (Tran, Vert, Long, Mic)
-in in/s and psi, after the 2-column report header.  That is the series-4
-equivalent of Blastware's `_ASCII.TXT` exports, and it gives 1,012 paired
-files (152 IDFW + 860 IDFH).  Earlier notes in this file and in
-`micromate/idf_file.py` asserted "Thor has no ASCII ground truth in the
-corpus"; that was wrong, and it is why the decoder sat pinned to a
-superseded walker with a scaling constant nobody could check.
-
-Harness: `scratch/verify_thor_against_csv.py`.
-
-### Geo LSB = 0.000310308 in/s per count (NOT 0.0003)
-
-The old 0.0003 was read off the smallest non-zero sample in the exports —
-but that is Thor's **4-decimal display rounding of the LSB, not the LSB**.
-It read every series-4 geophone sample **3.3% low**.  The quantisation
-ladder gives it away: counts 1..6 export as 0.0003, 0.0006, 0.0009, 0.0012,
-0.0016, 0.0019 — an LSB of exactly 0.0003 would end 0.0015, 0.0018.
-
-Each exported sample constrains the LSB to the window that rounds to its
-printed value.  Intersecting 991,415 such constraints gives
-
-```
-LSB ∈ [0.000310307933, 0.000310308057]      width 1.2e-10
-```
-
-so `_GEO_LSB_IPS = 0.000310308`, i.e. full scale 10.0 in/s = **32226.05
-counts**.  Corroboration: an IDFH interval that never recorded keeps its
-min/max accumulator at its ±full-scale seed, and that seed is
-`(min=+32226, max=-32226)`.  ⚠ The tempting closed form `10.0/32226` is
-very slightly wrong — it lands 4.5e-10 above the feasible window and loses
-78 boundary samples while never winning one.  **Series III uses 32000 counts
-for the same 10.0 in/s, so the two generations do not share a scale.**
-
-Independently confirmed on 8 production units (UM6047, UM11402, UM11719,
-UM12947, UM13981, UM14133, UM20146, UM20147): every unit's median PPV error
-against its device-reported peak moved from −3.3% to within ±0.03%.  It is a
-global constant, not a per-unit calibration.
-
-### IDFH segment header: the counter is a uint16, and it is cumulative
-
-```
-[length_be 2B][0a 00 00 00][counter_be 2B][05 3f]
-```
-
-`counter` is the **0-based cumulative index of the last interval in the
-segment** — 9, 19, 29, ... for the usual 10-intervals-per-segment layout
-(`length` = 730).
-
-The validator used to require `counter`'s high byte to be `0x00`.  That
-silently **capped every histogram at 250 intervals**: once the cumulative
-counter passed 255 the high byte went non-zero and every later segment was
-rejected.  Any run longer than ~4 hours lost its tail — frequently the part
-holding the event peak, so the file's PPV read low.  **540 of 858 corpus
-files were affected**; fixing it moved histogram peaks from 48.3% to 93.8%
-within 0.5% of Thor's reported PPV.
-
-### Unwritten interval slots carry a ±full-scale seed
-
-An interval the device reserved but never wrote keeps `min = +32226`,
-`max = -32226` on all four channels — `min > max`, impossible for real data.
-Decoded naively it yields a 10.0 in/s peak on every channel and, being a
-max-over-intervals, poisons the whole file's PPV.  Rare but real: exactly 1
-of 497,611 corpus intervals, and it inflated that file's Long PPV from
-0.0081 to 10.0 in/s.  The inversion is all-or-nothing across channels (0
-partial cases), so requiring every channel to be inverted is a safe test.
-
-### Record mode `00 00` — raw int16 absolute (MODE_RAW16)
-
-The record chain's mode field at `off+8` takes a fourth value:
-
-| mode | meaning | header |
-|---|---|---|
-| `02 00` | deltas + two int16 anchors | 14 B |
-| `01 00` | absolute, tagged blocks | 10 B |
-| `00 03` | raw 12-bit absolute, untagged | 10 B |
-| **`00 00`** | **raw int16 BE absolute, untagged** | **10 B** |
-
-A `MODE_RAW16` record with `length = 1032` carries exactly
-`(1032 - 8) / 2 = 512` samples and reproduced Thor's export **512/512
-exactly** on first test.  Thor uses it for segment 0 (the pre-trigger
-window) on some events.  Before this mode existed the record fell through
-the dispatch unhandled, so the channel silently lost its first 512 samples —
-which is what produced the "loud events truncate" symptom.
-
-`MODE_ABSOLUTE` is also valid as a **preamble** (the implicit segment-0 Tran
-record); its tagged blocks start at `body[3]`, not `body[7]`, because its
-header is 10 bytes rather than 14.
-
-### Body offset is not fixed at 0x0f1f — and 0x0f1f is really a record + 7
-
-A "body offset" is `<record start> + 7`, so that `body[0]` is the segment
-index and `body[1:3]` is the mode.  The canonical `0x0f1f` is simply the
-record at `0x0f18`.
-
-Searching for the literal preamble `00 02 00` finds only MODE_DELTA bodies,
-and worse, it **matches the `[seg][mode]` bytes inside any record header**,
-so the scan could pick a candidate part-way down the chain.  That decodes a
-plausible-looking but rotation-shifted body which drops each channel's
-segment 0 — the real cause of the remaining truncations.
-
-`_find_waveform_body_offset()` now anchors on record headers (the
-`<channel_id> 00 00` signature at `+4`, validated with `is_record()`),
-takes the **chain head** — a record no other record's length field points at
-— and trial-decodes `head + 7`, preferring the candidate where all four
-channels come out the same length.
-
-⚠ Do **not** scan for candidate preambles instead: `MODE_RAW16` is
-`00 00`, so every run of three zero bytes looks like a body start and each
-costs a full trial decode (~0.5 s/file measured, vs 6 ms/file now).
-
-### `40 NN` is not capped at NN=8 (2026-09-11)
-
-`data_block_len()` rejected any `40 NN` int16 block with `NN > 0x08`.  The cap
-had no evidence behind it — every corpus available when it was written used
-only NN ∈ {1, 2, 3, 4, 8}, so it was never exercised.  Loud events use much
-wider blocks:
-
-| corpus | `40 NN` values | walker stops |
-|---|---|---|
-| first + 3-channel corpora | 1, 2, 3, 4, 8 | none |
-| UM12947 2025-07..09 | 2, 4, 8, **12, 16, 20 … 196** | every value > 8 |
-
-Because `walk_body`/`run` stop at the first unrecognised tag rather than
-raising, this surfaced as **silently short channels** — e.g. Tran 1812 /
-Vert 2132 / Long 2324 on a file whose export has 2324 for all three.  The
-real bound is the buffer (and the caller's record end), not a magic constant.
-
-Verified against Thor's exports for UM12947 (2025-07-14 … 2025-09-25, 167
-waveforms): length mismatches **22 → 0**, and **1,476,242 / 1,476,249**
-samples exact.
-
-⚠ These events are **not** truncated recordings, which was the competing
-hypothesis — the exports carry the full sample count.
-
-**The 7 residual samples are Thor's rounding, not ours.**  Each differs by
-exactly one 4th-decimal tick (e.g. decoded 3.3551 vs export 3.3550).
-Intersecting the per-sample rounding constraints over this corpus is
-**infeasible** — the binding pair (count 2013 → 0.6247, count 4351 → 1.3501)
-contradict by 2.3e-11, i.e. 7e-5 relative.  No single linear LSB can
-reproduce every printed value, so Thor is not doing plain round-half-up on
-`count × LSB`.  Do not retune `_GEO_LSB_IPS` to chase these; it is already
-pinned to ~1e-11.
-
-### Mic-disabled units are a distinct shape (2026-09-10, second corpus)
-
-Some units run with the microphone disabled — **3 channels, not 4** — and that
-changes two structural things.  Confirmed on the `9-10-26-csv-req` corpus
-(UM11402, UM12947, UM20147): 139/139 waveforms and 877/877 histograms.
-
-**Waveform: the body starts earlier.**  A 3-channel unit has a shorter fixed
-header and puts its record chain head at **`0x0dba`**, below the old
-`_BODY_SCAN_FLOOR` of `0x0E00`.  The head was therefore invisible to the scan,
-which fell through to the *Vert* segment-0 record and decoded a body shifted
-one position around the channel rotation.  The signature is unmistakable:
-
-```
-Tran 3072 / Vert 2560 / Long 3072 / MicL 0      <- Vert exactly 512 short
-```
-
-46 of 139 files in that corpus were affected; all 46 became per-sample exact
-once the floor dropped to `0x0C00`.  Note the body-offset scoring also had to
-stop requiring four channels — `len(lengths) >= 3`, not `== 4`, or `equal` is
-permanently False for these events and the pick falls back to raw sample count.
-
-**Histogram: the interval record is 56 bytes, not 72.**
-
-```
-interval_size = 16 × n_channels + 8        (72 for 4 channels, 56 for 3)
-```
-
-It is **not a constant**, and it cannot be inferred from `length` alone.
-Derive the interval count from the segment counter — it is cumulative, so
-`n = counter - previous_counter` — and then `stride = (length - 10) / n`.
-`n_channels` follows from `(stride - 8) / 16`.
-
-Assuming 72 read 7 intervals out of each 10-interval segment and then walked
-off alignment into garbage that decoded as ~10 in/s peaks — inflating those
-files' PPV by up to 191,000%.  Fixing it moved the second corpus from 56.6% to
-**100.0%** of histograms within 2% of Thor's reported PPV, and recovered 4
-files that previously decoded no intervals at all.
-
-### What is still open
-
-- ~~23 of 575 production IDFW files~~ — **RESOLVED 2026-09-11.** Production
-  IDFW is now **575/575** with zero truncations and zero decode failures
-  (median PPV error −0.0007%).  See "`40 NN` is not capped at NN=8" above.
-
-- Mic → psi scale is still the rough `2.14e-6` regression, not derived.
-- Per-channel `int16 field4` in the IDFH interval record (possibly
-  time-of-peak) and the 8-byte tail (PVS data) remain undecoded.
-
-⚠ **Thor's histogram PPV has a display floor of 0.0050 in/s.**  In the
-production store 6,080 sidecar PPV values are exactly 0.0050 (next most
-common value: 275 occurrences), and **41.4% of IDFH sidecars report a
-component PPV larger than their own vector sum** — geometrically impossible.
-On those quiet files the decoder's ~0.0025 in/s is *more* accurate than the
-reference; do not "fix" the decoder to match it.
-
-### Codec breakthroughs (2026-05-28)
-
-- **Body offset is a fixed `0x0f1f`** across 151/154 corpus IDFW
-  files.  Preceded by a 4-byte record-type marker (`46 00 00 00`)
-  + magic preamble `00 02 00 [Tran[0] BE] [Tran[1] BE]`.
-- **Sample stream is BW's segment-rotated block codec verbatim.**
-  Thor reuses `10 NN` (nibble), `20 NN` (int8), `00 NN` (RLE),
-  `30 NN` (packed12), `40 02` (segment header) tags with the same
-  semantics.  Channel rotation Tran→Vert→Long→MicL.
-- **Geo LSB = 0.0003 in/s** (not BW's 0.005), because Thor's 16-bit
-  ADC range maps to 10 in/s without the 16-count BW quantization step.
-- **Mic ≈ 2.14×10⁻⁶ psi/count** (rough scale; refine after channel
-  block calibration constants are decoded).
-- **BW compliance anchor `\xbe\x80\x00\x00\x00\x00` reappears at
-  IDFW offset 0x952** — sample_rate at anchor−6 (uint16 BE),
-  record_time at anchor+6 (float32 BE), same layout as BW.
-- **Event timestamp at offset 0x97A** — 8 bytes `[day][month]
-  [year_be][unk][hour][min][sec]`.  Stop-time mirrors at 0x982.
-- **Serial as null-terminated ASCII at 0x14E**.
-- **Calibration date** at 0x194–0x197 (day, month, year_be).
-- Per-sample residual drift of ~3% suggests Thor encodes int8/nibble
-  deltas with an extra refinement bit that BW doesn't carry —
-  unsolved; errors resync within a few samples so cumulative impact
-  is small.
 
 ---
 

docs/instantel_protocol_reference.md

@@ -11,10 +11,6 @@
 
 | Date | Section | Change |
 |---|---|---|
-| 2026-08-26 | S7.6.2, S15 | **MULTI-INTERVAL HISTOGRAM BLOCKS - sub-minute intervals pack several per block.** The device always writes one minute of data per block, so a shorter histogram interval means more intervals packed into a longer block: 1 min -> 1 (the standard 32-byte block), 15 s -> 4 (stride 92), 2 s -> 30 (stride 612), with `stride = 12 + n * 20`. Each 20-byte record holds 8 x uint16 **little**-endian values (peak/half-period per channel) plus a 2-word tail whose first word is `0000` on every real interval. The standard block is big-endian - the variant is not. This recovers **415 files** (216 on BE18193 at 2 s, 199 on BE9440 at 15 s) that previously decoded to nothing, and before that were being accepted by the WAVEFORM codec and returning garbage up to 400x the device-reported PPV. |
-| 2026-08-25 (3) | S7.6.1, S15 | **THE WAVEFORM BODY IS A RECORD CHAIN, NOT A TAG STREAM — supersedes the segment-header model entirely.** Records are self-delimiting: `off+2` is a uint16 BE length and `next = off + 2 + len`; the chain ends on a record whose chan_id is `0x06`. `off+8` holds a 3-valued mode enum - `02 00` (14-byte header, anchors, cumulative deltas), `01 00` (10-byte, no anchors, ABSOLUTE values), `00 03` (10-byte, no tags at all, raw 12-bit absolute). **`40 NN` is an ordinary int16 BE data block of length 2*NN+2**, never a header; reading it as a 2*NN+16 header is what made walks drift, and the 'variable prefix' of 0/2/4/6/8 bytes reported earlier the same day was walker drift, exactly `4 - (old_stop - true_record_start)`. Verified: chain terminates on `06` in 1387/1388 files; all four channels equal length in **1388/1388** (was 156/1388); ASCII sample-count exact 72/75 -> **75/75**, fully exact 70/75 -> **73/75**; device PPV on a live decode **1306/1306** waveform (mean abs ratio error 0.00000) and **4458/4459** histogram. Also eliminated the walker-over-read class: 24 of those files were histograms the waveform codec was wrongly accepting. The superseded model is retained as `decode_waveform_legacy` because `micromate/idf_file.py` pins it for Thor IDFW body-offset search. |
-| 2026-08-25 (2) | S7.6.2, S15 | **HISTOGRAM BLOCK IS BIG-ENDIAN + terminal block tail.** The 32-byte histogram block's per-channel fields are uint16 **big-endian** (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], `V_halfperiod` [11:13], `L_peak` [13:15], `L_halfperiod` [15:17], `M_peak` [17:19], `M_halfperiod` [19:21]); only `block_ctr` [2:4] is little-endian. The marker is `block[4]` alone - the previous uint16 LE marker test at [4:6] forced `block[5] == 0` and thereby capped every geo peak at 255 counts (1.275 in/s), silently clipping larger peaks. The byte previously documented as a per-channel "annotation" is the high byte of the big-endian half-period, which is why it was non-zero exactly on sub-Hz intervals. Separately, the **final block of each stream carries tail `9c 06 00 42`** rather than `1e 0a 00 00` and holds arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram. Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly, plus 842,442 per-interval frequency comparisons with zero mismatches (previously 1 of 1196 files fully correct). |
-| 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. |
 | 2026-05-20 | §2, §3, §4.2, §5.1, §5.3, §6, §7.5b, §7.6.1, §7.6.3, §7.6.4, §7.7.2, §7.7.3, §7.7.5, §7.8.4, §7.8.7, §7.9, §8, §11, §12, §13, §14, §15, Appendix D | **DOC AUDIT PASS — accuracy sweep against `CLAUDE.md` + `minimateplus/` code.** Fixed: (1) S3 frames terminate on bare ETX, not DLE+ETX — §2/§3 rewritten. (2) §3 payload layout corrected — byte[1]=flags, byte[2]=SUB (was wrongly labelled DLE/ADDR). (3) §4.2 — probe responses do NOT carry data length; lengths are hardcoded `DATA_LENGTHS` constants. (4) §5.1 — removed stale duplicate "SUB 1C = TRIGGER CONFIG READ" row; SUB 0A lengths corrected from `0x30/0x26` to `0x46/0x2C` (real event / boundary marker). (5) §5.3 — added missing write-frame format (BW_CMD-only doubling, DLE-aware checksum, offset formula, ack format, SUB 71 chunk parameters). (6) §6 — fixed "SUB 06 → channel config read" → event storage range. (7) §7.5b / §8 — added the 10-byte `sub_code=0x03` continuous-mode timestamp variant alongside the 9-byte single-shot layout; peak vector sum location corrected from "fixed offset 87" to `tran_pos − 12` (label-relative). (8) §7.6 / §7.6.1 / §7.6.3 / §7.6.4 — switched compliance-anchor convention from the 10-byte form to the canonical 6-byte `\xbe\x80\x00\x00\x00\x00`; recording_mode confirmed at anchor−8 in BOTH read and write (was wrongly listed as anchor−3 write / anchor−4 read); sample_rate at anchor−6, histogram_interval at anchor−4, record_time at anchor+6; geo_range row added at channel_label+33. (9) §7.7.2 — token byte position corrected from `params[6]` to `params[7]`. (10) §7.8.4 — fi==9 skip marked FIXED (already removed from code); chunk-count totals updated. (11) §7.8.7 — TODO replaced with current state of `_decode_a5_metadata_into`. (12) §7.9 — Histogram Interval upgraded ❓ → ✅. (13) §11 — POLL example wire bytes corrected; SUB 5A row added to checksum table. (14) §13 — device-under-test updated for current primary unit (BE11529 / S338.17). (15) §14 — TCP Idle Timeout fixed (0→2 min); Data Forwarding Timeout units clarified. (16) §15 (renumbered from second §14) — open-question items already resolved in CLAUDE.md closed out. (17) Appendix D — extension taxonomy rewritten to reflect the AB0T timestamp encoding (D.5.2/D.5.3); EXTENSION REFUTED warning replaced with the resolved encoding. |
 | 2026-05-08 | §7.6.1 (RETRACTION) | **❌ RETRACTED — "raw int16 LE 8 bytes/sample-set" body codec was never validated.** The original 4-2-26 confirmation was based on misreading broken-decoder output (full-scale ±32K noise) as evidence the signal had saturated. BW's own 0C peaks for that capture (Tran=0.420 / Vert=3.870 / Long=0.495 in/s) prove the signal was NOT saturated — none of those exceed 13K ADC counts. No event in the project's archive has ever come close to saturation, yet the decoder consistently produces ±32K noise on every event. Conclusion: the body codec is not raw int16 LE; the actual encoding is open. Body byte distribution is heavily skewed (24% `0x00`, 10.5% `0x10`, lots of `10 XX` pairs) — likely a delta encoding with `0x10` as escape, but unverified. Retraction box added at top of §7.6.1; "fully-saturating event" claim removed from channel-identification note. The histogram codec in §7.6.2 IS verified and decoded correctly (different recording mode, 32-byte blocks); use it as a structural hint when reverse-engineering the waveform codec. |
 | 2026-02-26 | Initial | Document created from first hex dump analysis |
@@ -1105,51 +1101,14 @@ Every block starts with a 2-byte tag.  Five tag types are confirmed:
 |-----------|-------------------------------------|-----------------------|
 | ``10 NN`` | Small-delta data block              | NN/2 + 2 bytes        |
 | ``20 NN`` | Literal data block (int8-shaped)    | NN + 2 bytes          |
-| ``00 NN`` | RLE zero-delta run                  | 2 bytes               |
+| ``00 NN`` | 2-byte marker between data blocks   | 2 bytes               |
 | ``30 NN`` | Trailer summary block               | NN × 4 bytes          |
-| ``40 NN`` | Segment header                      | 2 × NN + 16 bytes     |
+| ``40 02`` | Segment header                      | 20 bytes (fixed)      |
 
 NN is always a multiple of 4.  ``10 NN`` and ``20 NN`` data blocks
 alternate with ``00 NN`` markers — every ``10/20 NN`` block is
 followed by a ``00 NN`` marker before the next data block.
 
-###### Wide-NN forms — ``0X NN`` (CONFIRMED 2026-08-25)
-
-The 12-bit wide-NN encoding already documented for ``1X NN`` /
-``2X NN`` (low nibble of the tag byte carries the high nibble of NN,
-so effective ``NN = ((tag & 0x0F) << 8) | NN``) **also applies to the
-``00 NN`` RLE tag.**  A narrow RLE run maxes out at NN = 0xFC, so a
-quiet stretch longer than 252 samples must use the wide form.
-
-Confirmed against six production events, e.g. ``01 0c`` (NN = 268) in
-``BE9558/K558LKOF.460W``.  Before this was handled, the walker hit its
-unknown-tag break at the first long quiet run and silently truncated
-every channel decoded after that point.
-
-###### ``30 NN`` is not capped at NN = 0x10 (CONFIRMED 2026-08-25)
-
-Data-section ``30 NN`` blocks occur with NN up to at least 0x18 (24),
-e.g. ``30 18`` in ``BE18193/T193LQ45.NN0W`` and ``30 14`` in
-``BE18193/T193LQ9W.AF0W``.  The data-section length formula
-(``NN × 1.5 + 2``) holds for these; only the earlier ``NN ≤ 0x10``
-guard was wrong.
-
-###### ``40 NN`` segment headers are variable width (CONFIRMED 2026-08-25)
-
-``40 02`` is the common case, but **NN is the count of int16 BE
-continuation deltas the header carries for the *previous* channel**, so
-the header grows with NN and every field after the deltas shifts by
-``2 × NN``:
-
-```
-length = 2 (tag) + 2 × NN (prev-channel deltas) + 14 (fixed tail)
-```
-
-``40 01`` (18 bytes) and ``40 03`` (22 bytes) both occur in production
-files — see ``BE12599/N599LP1S.UO0W`` and ``BE18438/T438LO30.GA0W``.
-In each case the constant ``02 00`` marker sits at ``data[2×NN+8]`` and
-the following tag lands exactly on a valid block boundary.
-
 ##### Segments
 
 The body is divided into segments separated by ``40 02`` segment headers.
@@ -1170,179 +1129,32 @@ fit fewer.  Observed first-segment sizes in the bundled fixtures:
 based on incomplete walks; that figure is wrong.  Segments are
 flash-page-sized in bytes, not sample-count-sized.
 
-The ``40 NN`` payload structure (offsets shown for the common NN=2 /
+The 18-byte ``40 02`` payload structure:
-18-byte-payload case; add ``2 × (NN − 2)`` to every offset from ``[4:6]``
-onward for other widths):
 
-| Offset (NN=2) | Generic          | Field                                  | Status      |
+| Offset    | Field                                       | Status      |
-|---------------|------------------|----------------------------------------|-------------|
+|-----------|---------------------------------------------|-------------|
-| [0:4]         | [0 : 2NN]        | NN × int16 BE continuation deltas for  | ✅ confirmed|
+| [0:2]     | T_delta at first sample of new segment      | ✅ confirmed|
-|               |                  | the PREVIOUS channel (16-count units)  |             |
+|           | (int16 BE, in 16-count units)               |             |
-| [4:6]         | [2NN : 2NN+2]    | Unknown (varies; possibly a checksum)  | ❓ open     |
+| [2:4]     | Likely T_delta at sample seg_start+1        | 🟡 likely   |
-| [6:8]         | [2NN+2 : 2NN+4]  | Byte length to next segment header − 2 | 🟡 likely   |
+| [4:6]     | Unknown (varies; possibly a checksum)       | ❓ open     |
-|               |                  | (uint16 BE; off by ±4 on some files)   |             |
+| [6:8]     | Byte length to next segment header − 2      | ✅ confirmed|
-| [8:12]        | [2NN+4 : 2NN+8]  | Monotonic uint32 LE counter            | ✅ confirmed|
+|           | (uint16 BE; useful for walker pre-scan)     |             |
-|               |                  | (starts ~0x47, +1 per segment)         |             |
+| [8:12]    | Monotonic uint32 LE counter                 | ✅ confirmed|
-| [12:14]       | [2NN+8 : 2NN+10] | Constant ``02 00``                     | ✅ confirmed|
+|           | (starts ~0x47, increments by 1 per segment) |             |
-| [14:18]       | [2NN+10 : 2NN+14]| THIS channel's 2-sample anchor pair    | ✅ confirmed|
+| [12:14]   | Constant ``02 00``                          | ✅ confirmed|
-|               |                  | (2 × int16 BE)                         |             |
+| [14:18]   | Unknown 4-byte field                        | ❓ open     |
 
-⚠️ An earlier draft listed ``[14:18]`` as an "unknown 4-byte field" and
+Examples from event-c (1 sec single-shot):
-``[12:16]`` as a constant ``02 00 00 01``.  Both were wrong: the constant
-is only the 2-byte ``02 00``, and the four bytes after it are the anchor
-pair the decoder needs.  Corrected 2026-08-25.
-
-###### Tagless segment headers (CONFIRMED 2026-08-25)
-
-A segment header can appear with **no ``40 NN`` tag at all** — just the
-14-byte tail:
 
 ```
-[field2:2][len_to_next:2][channel_id:4][marker:2][anchors:4]
+Segment header 1 (offset 235):
+  40 02 | 00 00 00 00 | 0a 4b 01 1e | 47 00 00 00 | 02 00 00 01 | 00 01
+                                                  ^counter=0x47
+Segment header 2 (offset 523):
+  40 02 | ff fe ff fe | 13 f5 01 06 | 48 00 00 00 | 02 00 00 01 | 00 02
+                                                  ^counter=0x48 (+1)
 ```
 
-This is the NN=0 case: the previous channel needed no continuation
-deltas, so there is no tag and no delta bytes.  Detect it by the six
-bytes at ``[4:10]`` — a known channel id, two zero bytes, a small
-segment index, then the ``01 00`` / ``02 00`` marker.
-
-It is where the walk stopped in 7 of the 8 events that still truncated
-after the wide-RLE / ``30 NN`` / variable-width-``40 NN`` fixes.
-
-###### The header "counter" is really a channel id (CONFIRMED 2026-08-25)
-
-The 4-byte field previously documented as a *"monotonic uint32 LE
-counter (starts ~0x47, increments by 1 per segment)"* is actually:
-
-```
-[channel_id:1][00][00][segment_index:1]
-```
-
-| channel_id | channel |
-|---|---|
-| ``0x46`` | Tran |
-| ``0x47`` | Vert |
-| ``0x48`` | Long |
-| ``0x49`` | MicL |
-
-Verified on **1697 of 1697** segment headers across the ground-truth
-corpus — every one agrees with the channel the rotation would assign,
-zero disagreements, no other id values observed.  The old reading was
-plausible because the id byte cycles 0x46→0x47→0x48→0x49 and the
-segment index increments, which *looks* monotonic in LE.
-
-Decoders should take the channel from this field rather than from
-rotation position: one missed or extra header (exactly what tagless
-headers used to cause) desyncs rotation and corrupts every channel
-after it.
-
-###### Geophone full scale is 32000 counts, not 32768 (CONFIRMED 2026-08-25)
-
-The body codec emits geo samples in 16-count units whose LSB is exactly
-**0.005 in/s**.  With the consumer-side ``×16`` to ADC counts, one ADC
-count is ``0.005 / 16`` in/s, so Normal range (10.000 in/s) is
-
-```
-10.0 / (0.005 / 16) = 32000 counts
-```
-
-Dividing by 32768 scales every geophone sample and every derived peak
-down by ``1 - 32000/32768`` = **2.34%**.  Measured on 216 per-channel
-comparisons against preserved Blastware ASCII exports: 32768 gave
-151/216 exact (worst error 0.238 in/s on a 10 in/s event); 32000 gives
-216/216 exact with a worst error of 0.005 in/s — exactly 1 LSB, i.e.
-pure quantization.
-
-This also explains why Blastware reports geo peaks slightly above
-nominal full scale (e.g. 10.14 in/s): the ADC has headroom past 32000.
-
-**Scope — this is not waveform-specific.**  The scale is applied where
-ADC counts become physical units, which every event passes through
-regardless of which codec produced the samples.  Verified after
-re-deriving the whole production store:
-
-| source | median ratio ours/device, 32768 | with 32000 |
-|---|---|---|
-| series-3 waveform (vs ASCII sample table) | 0.9766 | **1.0000** |
-| series-3 histogram (vs ASCII PPV, n=1137) | 0.9766 | **1.0000** |
-| series-4 Thor IDF (vs device peak, n=1468) | 0.960 | **0.983** |
-
-The series-4 figure is closer to correct but not exact — the Thor
-per-count LSB is its own open question (see §15).
-
-###### SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN, not a tag stream
-
-Everything above about ``40 NN`` segment headers, tagless headers, variable
-header widths and channel rotation describes a model that is **wrong**.  It
-produced nearly-correct output because the block table happens to tile the
-data sections correctly, but the framing is not what the device writes.
-
-The body is a chain of **self-delimiting per-channel records**:
-
-```
-off+0   field2    uint16     purpose unknown (not a length, not a checksum)
-off+2   len       uint16 BE  next_record = off + 2 + len   <- authoritative
-off+4   chan_id   0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL
-                  0x06 = END OF WAVEFORM STREAM
-off+5   0x00
-off+6   0x00
-off+7   segment index
-off+8   mode      2 bytes, a 3-valued enum
-off+10  anchors   2 x int16 BE, ABSOLUTE  -- present ONLY when mode == 02 00
-```
-
-**Mode enum**, all three ground-truth verified:
-
-| mode | header | data section |
-|---|---|---|
-| ``02 00`` | 14 bytes | anchors emitted, then blocks are **cumulative deltas** |
-| ``01 00`` | 10 bytes | no anchors, blocks carry **absolute** sample values |
-| ``00 03`` | 10 bytes | **no tags at all** — raw 12-bit packed absolute samples |
-
-Census over the 1,388 production series-3 waveform binaries:
-``02 00`` x 32,617, ``01 00`` x 74, ``00 03`` x 71.
-
-**``40 NN`` is an ordinary int16 BE data block** of length ``2*NN + 2``
-(1 <= NN <= 8), never a header.  The superseded model read it as a header of
-length ``2*NN + 16``, which is exactly why walks drifted: the "variable prefix"
-of 0/2/4/6/8 bytes reported earlier was walker drift, precisely
-``4 - (old_stop - true_record_start)``, on all 25 affected files.
-
-Block table for data sections (``NN = ((tag_hi & 0x0F) << 8) | tag_lo``):
-
-| tag | length | samples |
-|---|---|---|
-| ``0X NN`` | 2 | NN (RLE hold — holds the previous value in BOTH delta and absolute modes) |
-| ``1X NN`` | NN/2 + 2 | NN (4-bit nibble) |
-| ``2X NN`` | NN + 2 | NN (int8) |
-| ``30 NN`` | NN*1.5 + 2 | NN (12-bit packed) |
-| ``40 NN`` | 2*NN + 2 | NN (int16 BE) |
-
-The ``30 NN`` "trailer length = NN*4" fallback must NOT be applied inside a
-record — it corrupts records whose ``30 NN`` sits near a boundary.
-
-**The preamble is segment 0's implicit Tran record.**  ``body[1:3]`` carries
-the same mode pair: ``00 02 00`` (1,387 of 1,388 files) means two int16 BE
-anchors at ``body[3:7]`` then delta blocks; ``00 00 03`` (1 file,
-``BE13121/O121L4L1.KF0W``) means raw 12-bit absolute from ``body[3]``, which
-cannot be block-walked — the first record must be located by scanning.
-
-**Verification.**  The length chain terminates on a ``0x06`` record in 1,387 of
-1,388 files (the exception has an ambiguous footer signature inside its sample
-stream).  All four channels come out at identical length in **1,388/1,388**,
-against 156/1,388 under the superseded model.  Against the 75 events with a
-preserved Blastware ASCII export: sample-count exact **72/75 -> 75/75**, fully
-exact **70/75 -> 73/75** (the 2 remaining differ by exactly 1 LSB on samples
-sitting at the +-10.000 in/s rail).  Against device-reported PPVs on a live
-decode: waveform **1306/1306** exact with mean absolute ratio error 0.00000;
-histogram **4458/4459**.
-
-This also eliminated the walker-over-read class entirely.  24 of those 35
-"histogram" over-reads were histogram files that ``read_blastware_file`` fed to
-the *waveform* codec first; the old walker accepted them and returned garbage
-(one produced 98,923 "intervals"), while the record-chain decoder correctly
-returns None so they fall through to ``histogram_codec``.
-
 ##### Trailer
 
 The trailer (after the last segment's data) is a sequence of 32-byte
@@ -3055,7 +2867,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
 |---|---|---|---|
 | Timestamp 6-byte format byte[3] — purpose of the separator `0x00` byte | LOW | 2026-02-26 | Not blocking; 9-byte waveform record format (§8.2) fully confirmed without this byte. |
 | `trail[0]` in serial number response — unit-specific byte, derivation unknown. `trail[1]` resolved as firmware minor version. | MEDIUM | 2026-02-26 | |
-| ~~Full channel ID mapping in SUB `5A` stream~~ — **RESOLVED 2026-08-25 for the waveform body:** every `40 NN` segment header carries `[channel_id][00][00][segment_index]` at `data[2*NN+4]`, with `0x46`=Tran `0x47`=Vert `0x48`=Long `0x49`=MicL. Verified on 1697/1697 segment headers with zero disagreements against the decoded channel rotation. See §7.6.1. | RESOLVED | 2026-02-26 | Resolved 2026-08-25 |
+| Full channel ID mapping in SUB `5A` stream (01/02/03/04 → which sensor?) | MEDIUM | 2026-02-26 | |
 | ~~Exact byte boundaries of project string fields in SUB `71` write frame~~ — **RESOLVED 2026-05-05:** project/client/operator/seis-loc/extended-notes come from SUB 5A metadata pages at counter `0x1002` / `0x1004` (§7.8.7), NOT from the SUB 71 write payload. `_decode_a5_metadata_into` locates them via ASCII label scans. | RESOLVED | 2026-02-26 | Resolved 2026-05-05 |
 | Purpose of SUB `09` / response `F6` — 202-byte read block | MEDIUM | 2026-02-26 | |
 | Purpose of SUB `2E` / response `D1` — 26-byte read block | MEDIUM | 2026-02-26 | |
@@ -3082,11 +2894,6 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
 | **ACH inbound server — RESOLVED.** `bridges/ach_server.py` implements full inbound ACH pipeline. `--clear-after-download` flag for delete-after-upload workflow. Post-erase key-reuse detection via `max_downloaded_key` high-water mark. | RESOLVED | 2026-04-11 | |
 | **Sensor Check dropdown byte location** — byte offset in 1A compliance config payload for the "Sensor Check: Before monitoring / After each event / Disabled" setting is NOT YET LOCATED. Confirmed: unit always runs with "Before monitoring" set. Need a capture with "Disabled" to diff. | MEDIUM | 2026-04-08 | Still open |
 | **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open |
-| ~~**Variable-prefix segment descriptors**~~ - **RESOLVED 2026-08-25:** there is no variable prefix. The body is a chain of self-delimiting records (see S7.6.1); the 0/2/4/6/8-byte prefix was walker drift from reading `40 NN` as a segment header. All 25 affected files now chain cleanly to the `06` terminator. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
-| ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
-| ~~**Unmapped histogram block variant (415 files, 2 units)**~~ - **RESOLVED 2026-08-26.** When the histogram interval is SHORTER than one minute the device packs several intervals into a single block so each block still covers exactly one minute: 1 min -> 1 interval (the standard 32-byte block), 15 s -> 4 (stride 92), 2 s -> 30 (stride 612). `stride = 12 + n * 20`. Block is `[00][segment][ctr uint16 LE][0a][00]` then n x 20-byte records of 8 x uint16 **LITTLE**-endian `T_peak,T_halfp,V_peak,V_halfp,L_peak,L_halfp,M_peak,M_halfp` plus a 2-word tail whose first word is `0000` on every real interval (a session ending mid-block leaves buffer garbage in the remaining slots), then a 6-byte block trailer. **Note the endianness flip** - the standard 32-byte block is big-endian. Ground truth `BE9440/K440L3AQ.T70H` (5,710 intervals at 15 s) decodes with 17,130/17,130 geo peaks, 22,840/22,840 frequencies and 5,710/5,710 mic dB(L) matching its Blastware ASCII export exactly; across all 455 affected files 1,354/1,365 channel peaks (99.2%) match the device-reported PPV. | RESOLVED | 2026-08-25 | Resolved 2026-08-26 |
-| **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open |
-| **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open |
 
 ---
 
@@ -3513,141 +3320,3 @@ file[anc−3]: histogram_interval_LO
 *All findings reverse-engineered from live RS-232 bridge captures.*  
 *Cross-referenced from 2026-03-02 with Instantel MiniMate Plus Operator Manual (716U0101 Rev 15).*  
 *This is a living document — append changelog entries and timestamps as new findings are confirmed or corrected.*
-
----
-
-## Appendix E — Known Device Faults (field-observed, 2026-08-25)
-
-This appendix records *device* behaviour, not protocol.  It exists
-because these signatures are easy to mistake for decoder bugs — and one
-of them was, for a while.
-
-### E.1 The "offset" fault
-
-> **The full investigation now lives in `docs/offset_investigation.md`** --
-> base rate, detector definition, per-unit case files, ruled-out hypotheses,
-> and Instantel's own autozero procedure with its 2027-2069 acceptance
-> window.  This appendix is kept as the protocol-side summary.
-
-**Symptom.**  One geophone channel's baseline steps away from zero and
-stays there.  The trace still carries the real AC signal, but it rides
-on a DC pedestal of a few tenths of an in/s.  Operators call this an
-"offset"; it is a known recurring hardware fault, historically resolved
-by returning the unit to Instantel, shelving it until calibration, or a
-per-channel re-zero in an advanced/professional Blastware build.
-
-**Why it floods the queue.**  The pedestal exceeds the unit's own geo
-trigger level, so the channel sits permanently above threshold and the
-unit retriggers as fast as it can rearm — one event every 3–7 minutes
-for as long as the fault persists.  In the surveyed snapshot the worst
-episode produced 193 events in 38 hours.
-
-| unit | window | events | channel | pedestal | that unit's trigger level |
-|---|---|---|---|---|---|
-| BE9558 | 2026-05-15 → 05-16 | 193 | Long | +0.335 | 0.200 |
-| BE18438 | 2026-02-25 → 02-26 | 64 | Vert | +0.327 | 0.200 |
-| BE18438 | 2025-11-15 | 12 | Vert | +0.312 | 0.200 |
-| BE9558 | 2026-04-14 / 04-29 / 05-04 | 5 total | Tran | +0.29…+0.44 | 0.200 |
-
-**It is NOT a decode artifact.**  These events reproduce *exactly* in
-Blastware's own ASCII export — e.g. `BE12599/N599LQD7.8E0W` Tran reads
-mean +0.345, min +0.335, max +0.355 in both our decode and BW's.  Any
-shape-based false-trigger detector must treat this as a real device
-condition, not corrupt data.
-
-**Detection rule that works:**
-
-```
-dominant geo axis:  |mean| / peak  > 0.7
-              AND:  |mean| >= 0.9 x that unit's geo trigger level
-group hits into episodes by serial with a 12 h gap
-```
-
-A bare `|mean|/peak` threshold is useless — a quiet trace sitting at the
-0.010 in/s noise floor clears any ratio test.  On the surveyed store the
-bare rule flagged 7,184 events; the rule above flags 274, all real.
-Candidate list: `scratch/offset_candidates.csv`.
-
-**What the data rules out.**
-
-- *Not the geophone.*  The on-device sensor check passes on every unit,
-  offset or not — test frequency 7.2–8.1 Hz, damping ratio 3.3–4.7,
-  zero failures fleet-wide, including mid-episode.  The coil and its
-  mechanical response are healthy.
-- *Not the battery.*  6.6–6.8 V on the affected units, same as the rest.
-- *Not environmental.*  Only 2 units of 21 ever show it.  BE17353 logged
-  395 waveform events over nine months with zero occurrences; BE7145,
-  295 events, zero.  Weather acts on all of them equally.
-- *Not condensation.*  BE18438's Vert held 0.179 → 0.189 for 34 hours
-  **including a 10-hour overnight gap**, then stepped to 0.33 at 13:24
-  in the afternoon.  Dew would peak overnight; it was flat overnight.
-
-**What the data shows.**  The pedestal is *piecewise constant* — it
-holds rock-steady, survives power-off gaps, and changes only in discrete
-steps.  A small common-mode diurnal wobble rides on top of it, but that
-wobble is present on the healthy channels too (it is the unit's normal
-thermal breathing) and is not the fault.
-
-**The fault is on the geophone side.**  Operator report (2026-08-25):
-*attaching a different geophone to an affected unit makes the offset go
-away.*  That is the channel-swap test, already run in the field many
-times, and it rules out the unit's analog front-end and any stored
-per-channel zero constant — a constant lives in the unit and would
-survive a sensor swap.
-
-The stored data agrees.  MicL — a separate transducer on its own cable —
-shows no offset during either episode (`|mean|/peak` = 0.17 and 0.02),
-while the geo channels on the same unit at the same moment are pinned.
-
-**Two distinct failure patterns**, both geophone-side:
-
-| unit | Tran | Vert | Long | MicL | reading |
-|---|---|---|---|---|---|
-| BE18438 | 0.16 | **0.97** | 0.18 | 0.02 | one element only |
-| BE9558 | 0.90 | 0.81 | **0.99** | 0.17 | all three, Long dominant |
-
-(values are `\|mean\|/peak` on the dominant axis)
-
-BE18438 has a single faulted conductor pair with the other two clean.
-BE9558 has all three geo channels offset by different amounts
-(Long +0.340, Tran +0.037, Vert −0.020) — that points at a shared return
-or ground path in the housing rather than one element.
-
-**Candidate mechanisms.**  A geophone coil is passive and cannot
-generate sustained DC on its own, so an offset originating on the sensor
-side has to come from one of:
-
-1. *Galvanic corrosion at a connector or splice* — dissimilar metals
-   plus moisture form a literal cell.  This generates DC directly, in
-   the tens-of-millivolts range that matches the measured offset
-   (0.29 in/s ÷ 6.206053 in/s per volt ≈ 46 mV referred to the ADC
-   input).  Fits the discrete steps, surviving power cycles, and being
-   cured by swapping the sensor.
-2. *A leakage path to shield or another conductor*, letting the unit's
-   own bias/reference divide into the signal pair.  Best fit for the
-   BE9558 all-three-channels pattern.
-3. *Changed coil DC resistance* (partial short or degraded winding)
-   interacting with the amplifier's input bias current —
-   `V_offset = I_bias × R_source`.  Best fit for the BE18438
-   single-element pattern.  Note the on-device sensor check would not
-   catch this: it measures ring-down frequency and damping, not DC.
-
-**Caveat on the swap test.**  Swapping a geophone requires stopping and
-restarting monitoring, and these units run Sensor Check "Before
-monitoring" — so the restart also re-zeros the channels.  The swap
-therefore does not cleanly separate *"new sensor"* from *"the restart
-re-zeroed it"*.  Two cheap controls settle it: restart monitoring
-**without** swapping (if the offset clears, it was the re-zero), or swap
-the **original** sensor back (if the offset returns, it is the sensor).
-The operator's remedy being a swap rather than a restart is weak
-evidence for the sensor, since a restart is the easier thing to try
-first.
-
-**Best single measurement when it recurs:** DC voltage across the
-suspect geophone's connector pins with the sensor **disconnected from
-the unit**.  A galvanic cell (mechanism 1) shows an open-circuit
-voltage; mechanisms 2 and 3 do not.
-
-**Note:** the surveyed population is subject to survivorship bias —
-flooded events were routinely deleted before this snapshot, so units
-that look clean here may have had episodes that were scrubbed.

docs/offset_investigation.md

@@ -1,987 +0,0 @@
-# The "offset" fault — investigation journal
-
-> ## ⚠ CORRECTED 2026-08-28 (same day) — the v1 detector was wrong
->
-> Brian pushed back on the finding that offsets "come and go": in the field,
-> once a unit develops one it stays broken until the geophone is replaced.
-> He was right, and the challenge exposed **two real flaws** in the v1 detector:
->
-> 1. **It scored only the axis with the largest peak.** A real event on one axis
->    hid a persistent pedestal on another. BE12599 on 2026-08-21 read "clean"
->    solely because Long had a 1.065 in/s event — Tran was sitting at
->    **+0.4732 in/s** at that moment and was never examined.
-> 2. **It used the MEAN**, which a real transient perturbs. The **median** is the
->    resting baseline — most samples sit at it, so a blast does not move it.
->    Same event, Long channel: mean **+0.0783** vs median **-0.0050**.
->
-> Both flaws manufactured false recoveries. The corrected detector
-> (`scratch/offset_scan2.py`, per-channel median) shows the pedestal is
-> **persistent**, exactly as the field experience says. See §2b and §3b.
->
-> **Then Brian proposed a better detector still** — measure the floor during
-> the *pre-trigger* window, and require it to hold across pre/middle/end.
-> That is now the detector of record (§2c). Final answer: **5 of 45 units
-> (11%)**, stable across a 2x threshold range.
->
-> Sections below that were written against v1 are marked; v1 numbers are kept
-> for the reasoning trail, not as current fact.
-
-A running record of the **offset** hardware fault on Instantel Series III
-seismographs: a geophone channel whose trace sits displaced from zero rather
-than centred on it.
-
-This is a *journal*, not a spec. Findings are dated, dead ends are kept with
-the reason they died, and every number says where it came from. When something
-here is superseded, strike it and say why rather than deleting it — the point
-is that a future session can tell what was actually established from what was
-merely believed at the time.
-
-Companion material:
-- `scratch/offset_scan.py` — the detector
-- `scratch/verify_against_ascii.py` — decoder verification harness
-- `docs/instantel_protocol_reference.md` — wire protocol, incl. the
-  unimplemented `SUB 0x0E` this investigation now wants
-
----
-
-## TL;DR (current state, 2026-08-28)
-
-- **It is real device data, not a decode bug.** Settled early and confirmed
-  against Blastware's own ASCII exports.
-- **Base rate: 5–6 of 45 units (11–13%)** across the full DL2 archive,
-  2018–2026. This *confirms* the earlier 2-of-21 (9.5%) estimate from the much
-  smaller Terra-View DB — survivorship bias from deleted events had **not**
-  concealed a wave of cases.
-- **The fault is bimodal, not a drift continuum.** A unit is either clean or
-  grossly off. Loosening the amplitude threshold 11× adds no new units.
-- **The unit's own sensor check cannot see it.** 102 offset events, zero
-  sensor-check failures. Do not try to use it as a screen.
-- **Cause is still unsettled.** Instantel's autozero fixes the minority of
-  cases; the rest are hardware. We cannot yet tell which is which remotely.
-- **The histogram corpus (63,535 files, 9.7x the waveforms) is now scanned too** —
-  see §8b. It independently confirms BE18438 and BE9558 with a clean 2.5x
-  separation, but detects only **2 of the 5** confirmed units, cannot attribute a
-  channel, and resolves time to ~a month. **A negative histogram result is not
-  evidence of health** — DC leakage into the interval peak varies 45x between units.
-- **`offset_scan3.py` has a label defect** (§8b): its spread gate discards 18.8% of
-  high-|pre| rows onto units currently counted as clean. Re-cut before quoting any
-  precision number again.
-- **Best open lead:** `SUB 0x0E` (channel sensor data, 8 channels × 10 bytes,
-  unimplemented) may carry the very numbers Instantel says to check against
-  **2027–2069**. Untested.
-
----
-
-## 1. What the fault looks like
-
-A healthy geophone trace is centred on zero. An offset channel is parked away
-from zero, so the channel **mean approaches its own peak**. In Blastware the
-signature is "parallel lines above or below the zero line" (Instantel's own
-wording).
-
-Consequences observed in the field:
-- The unit can **self-trigger on its own offset** when the displacement exceeds
-  the geo trigger level, producing streams of junk events with no ground
-  motion. Instantel has a separate FAQ for this symptom (13-0-22, *"Unit
-  triggers continuously without activity"*).
-- Recorded PPV for that channel is meaningless while the fault persists.
-
----
-
-## 2. The detector
-
-Implemented in `scratch/offset_scan.py`. Operates on raw BW binaries only — no
-DB, no sidecars.
-
-```
-for each series-3 waveform binary:
-    decode -> per-channel ADC counts
-    dominant axis = channel with the largest |peak|
-    flag when   |mean| / peak  >  0.70
-          and   |mean|         >= 0.90 x the unit's geo trigger level
-episodes = per-serial runs of flagged events, split on a >12 h gap
-```
-
-Why each term:
-
-| term | purpose |
-|---|---|
-| `\|mean\|/peak > 0.7` | the discriminator. A DC-parked trace has mean ≈ peak. |
-| `\|mean\| >= 0.9 × trigger` | amplitude floor — suppresses quiet traces where mean and peak are both tiny and the ratio is meaningless. |
-| dominant axis only | the fault is per-channel; scoring all three dilutes it. |
-| 12 h episode gap | separates deployments/visits rather than counting events. |
-
-Trigger level comes from a paired `_ASCII.TXT` when one exists, else the
-per-serial median learned from that unit's ASCII files, else 0.2 in/s.
-
-**Known limitation.** Event traces contain real ground motion, so this can only
-see offsets large enough to *dominate* the trace. A mild offset on a real blast
-is invisible. Instantel's A/D-mode check (§5) is the only thing that sees the
-mild end. Our base rate is therefore a **gross-offset** rate.
-
-### 2b. Detector v2 — per-channel median (CURRENT)
-
-`scratch/offset_scan2.py`. Supersedes the above.
-
-```
-for each series-3 waveform binary:
-    for each geo channel independently:
-        pedestal = median(samples)          # resting baseline, robust to blasts
-        flag the CHANNEL when |pedestal| >= 0.025 in/s   (5 A/D counts)
-a unit has a real fault when a channel is flagged on >=3 CONSECUTIVE events
-```
-
-Why median: a DC pedestal shifts every sample, so it moves the median. A real
-event moves only a minority of samples, so it does not. This removes the need
-for the `m/p` ratio guard entirely — that guard existed only to compensate for
-using the mean.
-
-Why per-channel: the fault is on one geophone axis. Scoring only the dominant
-axis means any event with motion elsewhere hides it.
-
-Why "3 consecutive": the 0.025 in/s floor is only ~2x a healthy channel's
-resting median (observed 0.010-0.015), so isolated flags are noise. Persistence
-is the discriminator — and it is what the field experience predicts.
-
----
-
-## 3b. Archive results, corrected (v2)
-
-| | v1 (dominant axis, mean) | **v2 (per-channel median)** |
-|---|---|---|
-| units with any flagged event | 6 of 45 | 19 of 45 |
-| **units with a sustained pedestal (>=3 consecutive)** | — | **8 of 45 (18%)** |
-| runs of >=3 consecutive | — | 29 |
-| runs of 1-2 events (noise) | — | 69 |
-
-Units with a sustained pedestal: **BE9558, BA10895, BE11007, BE11529, BE12599,
-BE13117, BE18003, BE18438**. BA10895 and BE18003 were invisible to v1.
-
-**The affected channel is most often Vert**, which v1 got wrong — it named
-whichever axis had the largest peak. BE13117 and BE18438 are both Vert faults.
-
-Longest / clearest runs:
-
-| unit | ch | span | events | median in/s |
-|---|---|---|---|---|
-| BE13117 | Vert | 2023-05-03 → 05-04 | 194 | 0.035 → **1.915** |
-| BE18438 | Vert | 2026-02-25 → 02-26 | 75 | 0.180 → 0.370 |
-| BE9558 | Vert | 2020-02-11 (6 h) | 33 | 0.065 → 0.090 |
-| BE12599 | Tran | 2026-08-14 → 08-23 | 8 | 0.030 → **0.565** |
-| BE18003 | Vert | 2021-03-17 → 06-11 | 3 | 0.040 → 0.060 |
-
-BE12599 began **2026-08-14**, not 08-17 as v1 reported, and was still faulting
-at the last event in the archive.
-
-### 2c. Detector v3 — PRE-TRIGGER floor + constant-floor test (CURRENT)
-
-`scratch/offset_scan3.py`. Brian's method, and better than v2 for a reason
-worth naming: **the pre-trigger window is definitionally quiet** — it is the
-buffer captured before the trigger fired — whereas a whole-record median is
-merely *robust* to the event. `pretrig_samples` comes from the STRT record.
-
-```
-per channel:
-    pre    = median of the first pretrig_samples samples
-    mid    = median of the middle third
-    end    = median of the final third
-    spread = max(pre,mid,end) - min(pre,mid,end)
-
-    offset when  |pre| >= floor  AND  spread <= 0.02 in/s
-real fault when a channel is flagged on >=3 CONSECUTIVE events
-```
-
-A DC offset is a **constant floor** — present before the trigger, during, and
-after. The spread test rejects transients (settling, handling, a long event
-tail) that move one segment relative to the others, which is what v2's
-whole-record median could not do.
-
-**The empirical noise floor justifies the threshold.** Across 19,244
-non-flagged channel-events the pre-trigger floor distributes as:
-
-| floor | share |
-|---|---|
-| −1 unit (−0.005) | 18.4% |
-| **0.000** | **62.7%** |
-| +1 unit (+0.005) | 13.4% |
-
-**94.5% within ±1 quantisation unit; median exactly +0.0000, mean −0.0008.**
-So there is **no systematic zero-point bias in the decoder** — an independent
-confirmation of the 32000-count scale. A healthy channel really does read
-0.000, and "any constant floor that is not 0.000" is the right signal, with
-±1 unit of slack for quantisation.
-
-**The result is threshold-insensitive**, which is what distinguishes a real
-signal from a tuned one:
-
-| floor | units flagged | sustained units |
-|---|---|---|
-| 2 units (0.010) | 34 | 15 ← into the noise |
-| 3 units (0.015) | 26 | 8 |
-| **4 units (0.020)** | 17 | **5** |
-| **5 units (0.025)** — Instantel's | 12 | **5** |
-| **8 units (0.040)** | 8 | **5** |
-
-### FINAL RESULT: 5 of 45 units (11%)
-
-**BE9558, BE11529, BE12599, BE13117, BE18438.**
-
-Unchanged across a 2x threshold range. BE11007 and BA10895 drop out — the
-spread test identifies them as transients, not pedestals.
-
-The 11% headline happens to match v1's, but the reasoning and the unit list
-differ: v1 included BE11007 and named the wrong *channel* on most units.
-
----
-
-## 3. Archive results (2026-08-28)
-
-Source: DL2 event export, 6,577 **unique** series-3 waveforms, 45 units.
-See [`dl2-archive`](#8-data-and-tooling) for the `Sent/` mirror trap.
-
-**283 suspect events, 15 episodes, 6 of 45 units (13.3%).**
-Excluding BE11007 (§4, likely not an offset at all): **5 of 45 = 11.1%**.
-
-### Threshold sensitivity — the bimodality result
-
-Re-scoring the same corpus at a range of amplitude floors, with two
-ratio cut-offs (1 A/D count = 0.005 in/s, see §5):
-
-| \|offset\| floor | m/p > 0.7 | m/p > 0.9 |
-|---|---|---|
-| 5 cts (0.025 in/s) — *Instantel's own* | 333 ev / 6 units | 279 ev / **5 units** |
-| 10 cts (0.050) | 294 / 6 | 274 / 5 |
-| 20 cts (0.100) | 250 / 5 | 244 / 4 |
-| 40 cts (0.200) | 209 / 5 | 203 / 4 |
-| 80 cts (0.400) | 152 / 4 | 148 / 2 |
-| 160 cts (0.800) | 144 / 2 | 141 / 1 |
-
-Relaxing the floor by 11× (0.27 → 0.025 in/s) adds ~14% more events and **no
-new units**. There is no population of mild offsets hiding below our threshold
-*in event data*. Either a unit is clean or it is grossly off.
-
----
-
-## 4. Per-unit case files
-
-Ordered by severity. `m/p` medians are on the offending channel.
-
-### BE13117 — one violent day, never again
-`145 / 454 events (32%)`, **1 episode**, 2023-05-04, 6.8 h.
-Offset climbed **0.393 → 1.875 in/s within the episode**. `m/p` median
-**0.996** — the trace is almost pure DC. No recurrence in the rest of its 454
-events. No ASCII files in the archive, so no calibration history.
-
-### BE18438 — recurring, months apart
-`87 / 293 (30%)`, **2 episodes**: 2025-11-15 (1.2 h, n=12, 0.279 → 0.369) and
-2026-02-25 (**28.8 h**, n=75, 0.183 → 0.366). `m/p` median 0.967.
-Clean across all 196 events preceding its 2025-08-12 calibration.
-
-### BE9558 — six years apart
-`38 / 196 (19%)`, **4 episodes**: 2020-02-11 (6.3 h, n=33, but only
-0.068 → 0.086 — very mild), then 2026-04-14, 2026-04-29, 2026-05-04
-(0.28–0.45). `m/p` median 0.919. Calibrated 2026-06-26; 0/7 events flagged
-after, but n=7 is far too small to call it fixed.
-
-### BE12599 — the live case ⚠
-`6 / 77 (8%)`, **6 single-event episodes, one per day at exactly 05:00**,
-2026-08-17 → 2026-08-23. Offset rose 0.383 → 0.565 then fell back to 0.345.
-`m/p` ≈ 0.965, geo trigger 0.3 in/s — **the offset exceeds the trigger level,
-so the unit is triggering on its own fault**. Last calibrated 2025-08-12.
-
-This is the most recent and the most useful: a currently-faulting unit is the
-natural experiment for the re-zero-vs-repair question (§7).
-
-### BE11529 — marginal
-`4 / 99 (4%)`, 1 episode 2025-07-08, 0.4 h, offsets only 0.051 → 0.058 in/s.
-`m/p` median 0.959, so DC-dominated, but the magnitude is near the noise of
-this method. Treat as unconfirmed.
-
-### BE11007 — probably NOT an offset
-`3 / 70 (4%)`, 1 episode 2022-01-17, offsets 5.500 → 6.904 in/s — by far the
-largest. But `m/p` is only **0.719–0.738** against ≥0.9 for every other unit,
-and the peaks are 7.6–9.4 in/s on a 10 in/s range. That reads as a **large
-real blast with asymmetric ground motion**, not a parked trace. Excluded from
-the headline base rate.
-
----
-
-## 5. Instantel's own procedure and thresholds
-
-From two Instantel technical-support FAQs supplied 2026-08-28
-(answers **13-0-21** *"How to determine offsets"* and **12-0-10** *"Removing
-offsets on an Instantel Series III monitor"*; created 2008/2007, last updated
-2009-03-06).
-
-### Identifying (13-0-21)
-
-1. Create or use an event with the **manual minimum trigger** set for the
-   connected geophone and microphone — i.e. an event that recorded no real data.
-2. Save it and open in Blastware.
-3. An offset shows as **parallel lines above or below the zero line**.
-4. Put the unit in **A/D mode** — on Series III, press and hold `OPTION`, then
-   press `START MONITOR`.
-5. **Display counts higher than 5**, with no vibration or overpressure present,
-   indicate an offset.
-
-### Removing — the autozero (12-0-10)
-
-1. Be in a **quiet area with low vibration**.
-2. Power on the Blastmate III / Minimate Plus.
-3. Connect the geophone and microphone — **LINEAR mic only**.
-   ⚠ *Do not connect an "A" weight microphone, regardless of what the monitor
-   displays.*
-4. Press `Test`.
-5. Wait for the **Sensor Check** results to appear.
-6. Press `OPTION` and `START MONITOR` **simultaneously**.
-7. `Performing Autozero` appears; press `Enter`.
-8. Confirm the sensors are properly connected; press `Enter`.
-9. Wait for the autozero to complete.
-10. Press `Enter` twice → Main Menu, *Ready To Monitor*, offset corrected.
-
-### The go/no-go number — 2027 to 2069
-
-> When you perform an Autozero on any Series III unit, the lists of numbers in
-> the **X1 and X8 gains should all be between 2027 and 2069**. If not, repeat
-> the Autozero. **If the numbers are extremely out of the specified range, then
-> the unit should be sent in for repair.**
->
-> If this process does not remove the offset problem, return the unit **and
-> sensors** to Instantel for repair.
-
-This is the documented explanation for the field experience (Brian's dad,
-2026-08-28) that **a re-zero works maybe 10% of the time** — the autozero only
-recovers units whose zero reference is still near-correct.
-
-### Scale derivation (inference, well-supported — not proven)
-
-2048 is 12-bit midscale. Our codec's geo full scale is 32000 internal counts =
-10 in/s, with 1 decoder unit = 16 counts = exactly 0.005 in/s
-(`geo-full-scale-is-32000-counts`). ±2000 A/D counts about 2048 therefore maps
-to ±10 in/s at **0.005 in/s per A/D count**. That makes:
-
-- Instantel's ">5 counts" threshold ≈ **0.025 in/s**
-- the 2027–2069 window = **±21 counts = ±0.105 in/s** of tolerated zero error
-
-Consistent and mutually corroborating, but we have not confirmed the A/D-count
-scale directly from a device reading.
-
----
-
-## 6. Ruled out — keep these dead
-
-### Condensation / humidity — DEAD (2026-08-25)
-Proposed, then killed by its own controls: BE18438 stayed flat across a 10-hour
-overnight gap, and only 2 of 21 units showed the fault while 19 sat in the same
-weather. The apparent "diurnal cycle" was an artifact of binning by hour-of-day
-across two days. See `waveform-dc-offset-is-real-device-data`.
-
-### Clipping as a false-positive source — RULED OUT (2026-08-28)
-A rail-hitting trace would fake an offset (mean → peak). It isn't happening:
-median suspect peak is only **10% of full scale**, p90 is 18.6%. Only BE11007's
-3 events exceed 50% FS, and none reach 98%.
-
-### The sensor check as a predictor — DOES NOT WORK (2026-08-28)
-Tested on 102 offset events across 4 units:
-
-| unit | state | n | failed | median ratio | median freq |
-|---|---|---|---|---|---|
-| BE11529 | offset | 4 | **0** | 3.90 | 7.6 |
-| BE11529 | clean | 14 | 0 | 3.80 | 7.5 |
-| BE12599 | offset | 6 | **0** | 4.00 | 7.4 |
-| BE12599 | clean | 13 | 0 | 4.00 | 7.6 |
-| BE18438 | offset | 87 | **0** | 3.70 | 7.6 |
-| BE18438 | clean | 25 | 0 | 3.80 | 7.5 |
-| BE9558 | offset | 5 | **0** | 3.90 | 7.8 |
-| BE9558 | clean | 44 | 0 | 3.80 | 7.5 |
-
-Zero failures on either side and indistinguishable ratios/frequencies. The
-swing test measures geophone frequency response and damping — it never examines
-DC zero. **A grossly offset unit passes its own self-check.** This is why the
-fault goes unnoticed until somebody looks at waveforms.
-
-### "Offsets are transient / come and go on their own" — RETRACTED 2026-08-28
-v1 reported episodes lasting hours that ended spontaneously. **This was an
-artifact of the v1 detector** (see the banner at the top). With the per-channel
-median, the pedestal persists. Every clear case reads clean again only after a
-multi-day-to-multi-month gap consistent with service: BE13117 6 days, BE18438
-24 days, BE9558 63 days **with a confirmed Instantel calibration inside the
-gap**. BE12599 never reads clean — it is still faulting at the end of the
-archive. This matches the operational experience: once a unit develops an
-offset it stays broken until the geophone is replaced.
-
-### "Offsets develop N months after calibration" — CONFOUNDED, NOT A FINDING
-Tempting, and it looked strong:
-
-| unit | suspect before latest cal | after |
-|---|---|---|
-| BE18438 | 0 / 196 | 87 / 97 |
-| BE12599 | 0 / 62 | 6 / 15 |
-| BE11529 | 0 / 82 | 4 / 17 |
-| BE9558 | 38 / 189 | 0 / 7 |
-
-But bucketing suspects by months-since-calibration gives **one unit per bucket**:
-`0–3mo={BE11529}`, `3–6 & 6–9mo={BE18438}`, `9–12mo={BE9558}`,
-`12–15mo={BE12599}`. The apparent "51% failure rate at 6–9 months" is entirely
-BE18438's single February 2026 episode. Five units with roughly one episode
-each cannot support a population trend. **Do not re-derive this.**
-
-Also note: all affected units are calibrated on a **~12–13 month cadence**, so
-"sent to Instantel" is the routine annual schedule, not evidence of a
-fault-driven return.
-
----
-
-## 7. Open questions
-
-### Q1 — Is it a latched bad zero or analog degradation?
-The question that decides everything. A latched zero is correctable (possibly
-over the wire); degradation means a repair. Instantel's 2027–2069 rule implies
-*both* populations exist, with the split roughly 10/90 in the field.
-
-**BE12599 is the natural experiment** — faulting as of 2026-08-23. Read its
-values, run the autozero, read them again.
-
-### Q2 — Can we read the autozero numbers over the wire? (best lead)
-Instantel says to check *"the lists of numbers in the **X1 and X8 gains**"* —
-4 sensors × 2 gains = **8 channels**. The protocol reference already documents
-an unimplemented command with exactly that shape:
-
-```
-SUB 0x0E  ->  RSP 0xF1     "channel sensor data"
-  2-step read; channel selector in params[6:8] = 0x0000..0x0007
-  data length 0x0A (10 bytes) per channel
-```
-
-Blastware's *Unit Channel Test* sequence:
-`POLL×N → 0x15 → 0x01 → 0x08 → 0x01 → 0x0E×8 → 0x98×2 → 0x0E×8`
-— note the **second `0x0E` pass carries live ADC readings**.
-
-**Hypothesis (untested):** `0x0E` returns the numbers Instantel wants compared
-against 2027–2069. If true, SFM could diagnose an offset remotely *and* predict
-whether a re-zero will succeed — converting a 10%/90% shipping gamble into a
-decision made before packing a box.
-
-**How to test.** `bridges/ach_mitm.py` is a generic TCP proxy:
-
-```bash
-python bridges/ach_mitm.py --bw-host <MODEM_IP> --bw-port 9034 --listen-port 9999
-```
-
-Point Blastware at the proxy and run **Unit Channel Test**.
-⚠ In this topology the output filenames are reversed — the tool labels the
-*connecting* side "unit", so `raw_s3_*.bin` holds Blastware's bytes and
-`raw_bw_*.bin` the unit's.
-
-Capture priority: (1) BE12599 while faulting, (2) a known-good unit as control,
-(3) before/after an autozero on the same unit. Eight 10-byte payloads with an
-expected value near 2048 is a very constrained puzzle.
-
-### Q3 — What is the mild-offset rate?
-Unmeasurable from event files (§2). Only the A/D-mode check sees it. Would
-need a fleet sweep in A/D mode, or Q2 to succeed.
-
-### Q4 — Does an offset recur on the same unit after service?
-BE9558 shows episodes in 2020 and 2026; BE18438 twice in four months. Suggestive
-of recurrence, but service records aren't in the data — only calibration dates.
-
----
-
-## 8. Data and tooling
-
-| what | where |
-|---|---|
-| detector | `scratch/offset_scan.py` |
-| current results | `/home/serversdown/dl2-archive/offset_archive.csv` |
-| earlier candidate list (Terra-View DB, 274 events) | `scratch/offset_candidates.csv` |
-| archive working copy | `/home/serversdown/dl2-archive/files/` |
-| archive source | NAS `DeathStar` 10.0.0.2, `/volume1/Uploads/TMI/DL2-Event-backup-8-25-26/Event/autocall home/` |
-
-⚠ **The DL2 export keeps a byte-identical `Sent/` mirror of its root.** 13,077
-waveform paths are 6,577 distinct files. Always dedupe by basename — this
-doubled two reported figures before it was caught.
-
----
-
-## 8b. The histogram corpus — the other 90% of the archive (2026-09-04)
-
-Every result above §8 comes from **waveform** files. `offset_scan3.py` filters on
-`\.[A-Za-z0-9]{2}0[Ww]$`, so the corpus it scanned is 6,577 unique binaries. The
-archive also holds **63,535 unique histograms** — 9.7x more files — which the
-pre-trigger method cannot touch, because a histogram carries no samples: only a
-per-interval, per-channel peak and half-period.
-
-`scratch/offset_hist_scan.py` scans them. **63,505 of 63,535 decoded (99.95%),
-43 units, 77.9M intervals.** Two of the 45 units have no histograms at all.
-Output: `/home/serversdown/dl2-archive/offset_hist.csv` (190,515 channel-rows).
-
-### The premise, and how far it actually holds
-
-A histogram file is hours of continuous monitoring, so most of its intervals are
-definitionally quiet, and a channel parked off zero cannot report a peak below
-its own displacement. The signal is real — two within-unit contrasts, siblings
-unmoved in both:
-
-| unit | channel | in-episode floor | outside | waveform \|pre\| same window |
-|---|---|---|---|---|
-| BE18438 | Vert | 0.0350 | 0.0050 | +0.18 .. +0.37 |
-| BE12599 | Tran | 0.0250 | 0.0050 | +0.03 .. +0.49 |
-
-But the **leakage from a waveform pedestal into the histogram floor is bimodal,
-not merely partial**: measured ratio ~0.9 on BE18438 Vert, ~0.7 on BE9558,
-**~0.02 on BE12599** — two orders of magnitude on one instrument. The device
-evidently measures each interval peak against a running baseline, and how much
-DC survives that varies per unit. **Consequence: a negative histogram result
-carries almost no information.** Do not read "clean in the histograms" as clean.
-
-### The detector that survived
-
-    dmin(file, ch) = min[ch] - min over the other two geo channels, SAME file
-    gates (both hard):  n_intervals >= 60   AND   mic_p5 <= 5 raw counts
-    day statistic:      median of dmin over that day's qualifying files
-    flag day at        dmin >= 0.020 in/s (4 A/D counts)
-    episode at         >= 3 CONSECUTIVE observed days
-
-**Result: BE18438|Vert, BE9558|Tran, BE9558|Long.** Threshold-insensitive —
-the journal's own test for a real signal against a tuned one — and this is the
-first operating point in the investigation that passes it cleanly. The identical
-answer holds across: statistic `min` or `p5`; length gate 10/30/60/120/300; mic
-gate 3/5/8/10; threshold 0.015–0.035 (a 2.3x span); persistence K = 2,3,4,5,7.
-
-Separation, ranked by highest floor sustained over 3 consecutive gated days
-across all 135 unit-channels:
-
-| unit-channel | best3 |
-|---|---|
-| BE18438 Vert | 0.1650 |
-| BE9558 Long | 0.0350 |
-| BE9558 Tran | 0.0250 |
-| *(2.5x gap)* | |
-| BE7145 Tran | 0.0100 |
-| entire rest of fleet | <= 0.0050 (one quantisation count) |
-
-Day-level false alarm: **37 of 99,432 gated unit-channel-days = 0.037%.**
-
-### What it does NOT do — read this before trusting it
-
-- **It finds 2 of the 5 confirmed units, not 5.** The site-quiet gate is what
-  makes it work and it is also what costs BE11529 and BE12599. BE11529's
-  four-day single-axis ramp (Tran 0.025 -> 0.055, both siblings pinned at 0.005)
-  is the most offset-shaped thing in the corpus outside the two detections, and
-  the gate discards it.
-- **The positive class is two units.** Every threshold here is fitted to
-  BE18438 and BE9558, which contribute 22 of the 37 flagged days in the entire
-  corpus. No cross-validation is possible at n=2.
-- **Per-channel attribution is NOT established.** Rotating the three geo channel
-  labels within each file — preserving every value, file and day, destroying
-  only channel identity — reproduces the episode *count* with p = 0.769 and the
-  label agreement at p = 0.038–0.077. Report a **unit and a window**; do not
-  name a geophone axis on the strength of this detector alone.
-- **Timing resolution is ~1 month, not ~1 day.** A 30-day label shift still
-  scores 2 of 9 episode hits; the signal dies only past ~60 days. The day-level
-  series look far crisper than they are.
-- **Ground truth here is a sibling detector, not a service record.** Agreement
-  between the two corpora is corroboration of a shared method. Nothing in this
-  section has been checked against an actual repair, calibration or RMA.
-
-### Dead ends — keep these dead
-
-- **Absolute floor (min / p1 / p5 / p10 / p25, thresholded alone) — RETIRED.**
-  Not fleet-comparable and mostly not about the channel. Scoring each cell using
-  *only the other two channels* — a statistic containing zero information about
-  the suspect channel — reaches AUC 0.746 against the same labels, versus 0.872
-  for the absolute floor itself. **66% of its apparent discrimination is "that
-  day was noisy at that site."** Interval size alone moves its p99 7x (0.0350 at
-  1 min vs 0.0050 at 2 s). And of all files with any channel above 0.025, 56.5%
-  have **all three** channels above it — common-mode, i.e. the wrong physics.
-- **Zero-fraction — STRUCTURALLY IMPOSSIBLE, not merely weak.** The device never
-  reports a zero histogram interval peak. The value is a max over hundreds of
-  samples of a channel that always carries at least 1 count of noise, so it is
-  clamped at 1 A/D count (0.005 in/s). There is no zero to count.
-- **Interval size, sample rate, geo range, firmware — refuted as confounds for
-  the differential.** All four are *file-level scalars*: they move all three geo
-  channels together, so they cannot produce a single-channel lift and the
-  within-file differential is immune to them by construction. Geo range is
-  identical across the three geo channels in **63,535 of 63,535** binaries.
-  (Interval size remains fatal to the *absolute*-floor version, above.)
-
-### Two findings that are independent of the histogram detector
-
-**1. `offset_scan3.py`'s `spread <= 0.02` gate is discarding real signal.**
-It rejects **113 of the 600 channel-rows with |pre| >= 0.025 (18.8%)**, and the
-rejections are not random — 92 of them fall across 41 unit-channels currently
-labelled NEGATIVE. Four would become sustained positives under an
-amplitude-only >=3-consecutive rule: **BE12599|Long (run of 8), BE18003|Vert
-(4), BA10895|Vert (3), BE12844|Tran (3).** Until this is re-cut, the fleet label
-is **three-state — POSITIVE / NEGATIVE / SPREAD-REJECTED(unknown)** — and the
-third state should be excluded from both TP and FP counts rather than silently
-scored as healthy. Every precision figure computed against the two-state label,
-in this section and in §3, is affected.
-
-**2. The waveform corpus sees ~7% of the days a unit was deployed.** 2,627
-(unit, day) observations against the histogram corpus's 35,105 — 13.4x — with a
-per-unit median ratio of 0.070. BE12599, a confirmed unit, is waveform-observed
-on 39 of its 1,666 histogram-observed days (**2.3%**). Any statement of the form
-"the fault was absent before date X" that rests on waveform coverage alone is
-much weaker than its event count suggests.
-
-### BA10895 — reclassified (see also §4)
-
-Previously dismissed as a transient. The histogram record shows its **Vert**
-quiet-minute floor at 0.005 on 62/62 qualifying files from 2023-07-07, then
-0.010–0.015 on 48/58 files from 2023-08-03 to 08-27, while Tran moves on 2/58
-and Long on 9/58 and the site mic floor never leaves 1–3 counts. Independently,
-**42 of its 85 waveform events (49.4%) are single-axis-dominant** — one geo peak
->= 10x both siblings and >= 0.05 in/s — the **highest rate in the 45-unit
-fleet** (BE13117 36.1%, BE18438 29.4%), and **100% of it on Vert**. Vert
-excursions of 0.1–1.5 in/s with Tran/Long at 0.005–0.035 are not ground motion.
-
-This is a genuine Vert-channel hardware fault, but **not the classic pedestal** —
-the differential is only one A/D count. Caveat: its entire histogram record is a
-single 52-day deployment ending 2023-08-27, so nothing says whether it
-persisted, was serviced, or resolved.
-
-The other six marginal units — BE11007, BE17354, BE18004, BE18104, BE9557,
-BE18003 — are **clean**. All seven cap at +0.005 to +0.007 (one A/D count)
-lifetime under the quiet-site gate, against +0.175 for BE18438 Vert and +0.062
-for BE9558 Long. Three individual waveform flags fall in windows with **zero**
-histogram coverage and are NO-DATA, not clean: BE18004|Tran 2024-10-16,
-BE9557|Tran 2021-06-28, BE9557|Vert 2025-06-12.
-
-### Still open in this section
-
-- **The 11 thin-coverage units were not screened** (BE10202, BE11462, BE13779,
-  BE15760, BA15957, BE16754, BE16758, BE8081, BE8626, BA9229, BE9887 — each
-  under 20 waveform events, several with hundreds of histograms). This is the
-  population most likely to hold a previously unknown offset, and it is the one
-  slice of the plan that did not run. BE11462 was incidentally scored clean by
-  the full-archive pass; BE10202 has no histogram files at all.
-- **No completeness audit was run** over the above.
-- Re-cutting the ground truth three-state (finding 1) and re-scoring everything
-  against it.
-
----
-
-## 8c. Mechanism — five hypotheses tested, all dead (2026-09-06)
-
-**The mechanism is still unknown.** Five campaigns, ~105 effectively independent
-tests, seven nominally significant results against **5.2 expected by chance**
-under a global null. Every one died to its own confound analysis. What the
-campaign bought is a set of *shape constraints* and a long list of dead ends.
-
-### ⚠ Two things retracted from this journal
-
-**1. "Polarity is perfectly consistent — 11 of 11, zero mixed cases."** That is
-a **tautology of the spread gate**, not a property of the fault. `spread <= 0.02`
-requires pre/mid/end to agree, which forces one sign. Amplitude-only at the same
-0.025 threshold: **12 of 53 unit-channels are mixed**, including BE18438|Vert
-(88+/1−) and BE9558|Vert (1+/35−). Withdrawn.
-
-**2. "5 of 45 units, unchanged across a 2x threshold range."** The
-threshold-insensitivity is also a property of the gate. Amplitude-only gives
-**9 units at 0.020, 8 at 0.025** (adding BA10895, BE12844, BE18003), 5 at 0.040.
-The fleet is **8–9 units, not 5**.
-
-**3. "Persistent — it stays until the geophone is serviced."** Weakened, not
-withdrawn. There are **23 recoveries after runs of >=3 flagged events, median
-gap 6.03 days**, three inside ten minutes. BE18438|Vert reads `pre=mid=end=
-+0.0000` on 2026-02-10, +0.185→+0.370 across 02-25/26, and `+0.0000` again on
-2026-03-22 — identical Project, Seis Loc, calibration date, geo range and
-trigger throughout. The one thing that cannot be excluded is a **field
-autozero**: it is a button sequence at the unit and writes nothing into the
-event header. So "persistent" may be "persistent unless somebody pressed the
-buttons," and the archive cannot tell those apart.
-
-### The one positive finding: onset is a RAMP, minutes to hours
-
-Both onsets resolvable at minute cadence are ramps. **BE18438|Vert,
-2026-02-20** — the histogram corpus collapses a 14 d 21 h waveform bracket to
-**one minute**:
-
-```
-~14,200 consecutive quiet minutes at 0.000–0.005   (ten full daily files)
-09:32 +0.005   09:39 +0.045   10:20 +0.125   16:00 +0.165
-09:33 +0.010   09:42 +0.070   13:13 +0.150   20:17 +0.185  plateau
-```
-
-**50% of the excursion in 7 minutes**, the rest asymptotic over ~10 h, **>=25
-distinct one-minute intermediates**. Validated **75/75** against Blastware's own
-ASCII export. Its 2025-11-15 onset is the same shape over 2.7 h. BE13117 stage B
-is a 91-minute monotone rise, +0.035 → +1.745 in/s over ~40 samples.
-
-**This kills both poles of the original dichotomy** (journal Q1): not an
-instantaneous latched step (a bad autozero, a stuck trim-DAC), and not slow
-component degradation over days or weeks.
-
-⚠ It rests on **2 of 45 instruments**. Clopper-Pearson on 4/4 resolved onsets
-gives 95% CI [0.40, 1.00] — a mixed population with up to 60% true steps is not
-excluded. BE13117 has zero paired ASCII, so its ramp rests on our decoder alone.
-
-### The methodological corollary — more important than the finding
-
-**A waveform-only bracket manufactures the appearance of a step, and the spread
-gate is blind to onsets by construction.**
-
-The offset is what fires the trigger, so no waveform event can exist until the
-ramp has nearly reached the trigger level. BE18438's first event of each episode
-sits at 0.280 against a 0.300 trigger, and 0.185 against 0.200. At daily cadence
-against a 3 h ramp, P(catching an intermediate) = **0.125**.
-
-And `spread <= 0.02` rejects any record in which the floor is *moving* — which
-is exactly what an onset is. **The gate rejected the very BE18438 record where
-the ramp is visible.** If the operational goal is catching a fault early, before
-the unit floods the store with junk events, the current detector is the wrong
-shape for the job.
-
-### The surviving shape
-
-An **electrical, reversible, two-time-constant settling process** (~10 min and
-~hours), saturating at a ceiling, with occasional sub-3-minute discrete jumps
-superposed (BE18438 2026-02-26: 13:24 pre +0.180 / mid +0.240 / end +0.255 →
-13:27 +0.325, identical metadata). That is the signature of a **bias or leakage
-path charging a high-impedance node** — the class of fault Instantel's autozero
-recovers ~10% of the time, and what the X1/X8 gains measure.
-
-**It is a shape constraint, not a mechanism. Do not write it up as one.**
-
-### Dead — with the evidence, so none of this is re-derived
-
-| Killed | Evidence |
-|---|---|
-| **Latched step at onset** | >=25 one-minute intermediates over ~10 h, ASCII-validated. Direct observation, not a test. |
-| **Slow degradation over days/weeks** | Same observation — bulk of the excursion in 7 min to 2.7 h. |
-| **Thermal driving of pedestal magnitude** | BE13117, 365-count pedestal, n=128: full-day modulation **−0.42% ± 0.42%**, 95% CI [−1.25%, +0.40%]. Healthy-fleet seasonal zero drift totals **~0.3 A/D counts** — 15x to 1200x too small. Best-powered result in the campaign. |
-| **Ground-motion shock** | 30-day window-max percentile ranks 0.03/0.98/0.15/0.01/0.68/0.24, median **0.194** against a null of 0.5. **0 of 7 events >=9 in/s** was followed by an onset within 30 d. BE12599 hit 10.220 in/s (2023-11) and 10.005 (2025-04) and did not onset until 2026-08-14. |
-| **Handling / redeployment** | **0 of 9** onsets had a Project/Client/Seis Loc change. Widened to 30 d: 2 observed vs 4.90 expected, P(X>=2)=0.995 — *depleted*, the wrong direction. The apparent gap effect (p=0.035) died on histogram coverage: BE18438's "59.7-day gap" contains 122 histogram files; true silence 0.52 d. |
-| **Mechanical resonance / damping change** | BE18438|Vert at a 64-count pedestal (3x outside Instantel's ±21): ΔTest-Freq **CI [−0.090, +0.021]** against 0.127 Hz for a real calibration. Block permutation p=0.658. |
-| **Accumulated-duty threshold** | ~4 clean units logged more monitoring than the largest positive onset dose; BE18193 logged **13.45M intervals, 6.2x**. A counterexample — no power argument weakens it. |
-| **Firmware** | **14,338 of 14,340** exports read `V 10.72-8.17`. A constant cannot explain a variable. |
-| **Unit age** | Serial rank-sum 118.0 vs null 115.0, p=0.549; unchanged on the 8-unit re-cut (p=0.586). Serial is a poor age proxy anyway (Spearman +0.113 against archive entry). |
-| **Strong seasonal clustering** | 25 onsets, exposure-weighted permutation **p=0.59**. Excludes >=75%-in-one-season only; a 2x seasonal hazard is *not* excluded. |
-
-Also retire two overstated bounds. H6's dose-response exclusion "|r| > 0.03" is
-a **10x overstatement** once clustering is corrected — the honest bound is
-|r| > 0.1–0.3, so a real r=0.2 is not excluded. And **any statistic quoted
-per-event**: 512 flagged channel-events collapse to **4.9 effective independent
-observations** (unequal-cluster design effect 104.6 at ICC=1), and **55% of the
-flagged corpus is one instrument on two calendar days** (BE13117, 2023-05-03/04).
-
-### Power — read every negative in this section as bounded
-
-Fisher exact, 5 positives of 45, one-sided α=0.05, exposure a third of the fleet:
-
-| relative risk | power |
-|---|---|
-| 1.5 | 0.059 |
-| 2 | 0.112 |
-| 3 | 0.231 |
-| 6 | 0.497 |
-| 15 | 0.753 |
-
-80% power needs **RR ≈ 13–20**. Even a *perfect* split reaches p<0.05 only if
-the exposed group is <=25 of 45 units. **This archive can detect only
-near-deterministic unit-level causes.** Every negative above excludes a strong
-effect, not a real one.
-
-### What this archive can NEVER answer
-
-- **The A/D zero and the X1/X8 gains.** The 2027–2069 numbers appear in no file,
-  header or decoded record. They exist only on a live device behind `SUB 0x0E`.
-  Q1 is structurally unanswerable from data.
-- **Unit-level vs component-level cause.** **Zero of 14,340** exports carry a
-  geophone or sensor serial. Q4 is dead — there is no way to know whether the
-  same physical geophone came back after service.
-- **Service history.** The only service-adjacent field is `Calibration: <date>`
-  — 30 distinct dates fleet-wide, none before 2023, ASCII corpus entirely
-  2025–26. BE9558's 2020 and BE13117's 2023 episodes have no calibration record.
-- **Temperature.** Zero exports carry it. Battery Level is a verified coarse
-  thermometer (+0.204 V winter over summer, 20/20 unit-years, p=9.5e−7, matching
-  lead-acid tempco) but quantised at 0.1 V ≈ 10 °C — useless within a day. The
-  archive can *bound* thermal; it can never *test* it.
-- **BE13117 specifically** — 55% of the flagged corpus, the largest pedestal at
-  1.92 in/s, **zero** ASCII exports, histogram record ending eight months before
-  its episode. The most informative case in the archive is permanently outside
-  every metadata test.
-- **The mild-offset rate**, and therefore the base rate's denominator. Event
-  files only see offsets large enough to dominate the trace.
-
-### The experiment to run — `SUB 0x0E`, one afternoon
-
-Point Blastware at `bridges/ach_mitm.py` and run **Unit Channel Test** against
-(1) a faulting unit, (2) a known-good control, (3) the same unit before and
-after an autozero. BW's sequence is `0x0E x8 → 0x98 x2 → 0x0E x8`, the second
-pass carrying live ADC. Eight 10-byte payloads with expected values near 2048 is
-a very constrained puzzle.
-
-- **Proves:** whether the X1/X8 gains are readable over the wire, and whether
-  the fault sits at or upstream of the ADC zero reference. Gains walk out of
-  2027–2069 with the pedestal → the fault *is* the zero reference, Q1 answered.
-  Gains hold while the trace moves → the fault is downstream, look at the front
-  end.
-- **§8c hands it a falsifiable time course:** poll at ~1-minute cadence and the
-  numbers should **ramp over minutes-to-hours, not step**. If they step while
-  the trace ramps, the two are decoupled.
-- **Payoff:** converts the 10%/90% ship-it-or-not gamble into a decision made
-  before packing a box, remotely, for the whole fleet.
-- ⚠ In the MITM topology filenames are reversed — `raw_s3_*.bin` holds
-  Blastware's bytes.
-
-**Second: swap the geophone** between a faulted base and a healthy one. Fault
-follows the sensor → element or cable. Fault stays with the base → front-end
-board. One afternoon, zero code, and it settles the one question the archive is
-permanently blind to.
-
-**Third: log a faulting unit for 72 h untouched.** Every recovery we have is
-confounded by a possible field autozero. A shelf and a logger settles whether
-the fault genuinely self-reverses.
-
-**Fourth, free: re-cut the fleet label** — drop the spread gate, re-score
-amplitude-only, screen the 11 unscreened thin-coverage units. Might reach 9–10
-positives. Be honest about the gain: power against "older half carries 3x the
-hazard" rises only 0.23 → 0.30.
-
-**Highest-value item overall, and not an experiment: the RMA/repair records.**
-Which unit went back, when, what was done (autozero vs geophone replaced vs
-board), and the geophone serial fitted. "Same channel after a documented
-geophone *replacement*" is component-level-negative in one observation.
-
----
-
-### 8d. The non-motion test — Brian's "it doesn't cross zero" (2026-09-07)
-
-Looking at BE12599's 2026-08-09 event, Brian noted it reports no ZC frequency
-**because the trace never crosses zero**. That observation is the best detector
-in this investigation, and it comes from physics rather than a threshold.
-
-A geophone is a velocity sensor with no DC response, so its output over a record
-must integrate to ~zero — the ground does not relocate. Real motion therefore
-sits roughly half below zero. Anything electrical is one-sided.
-
-    mp = |mean| / peak        ~0 for motion, ~1 for a fault
-    frac_neg = share of samples < 0
-
-`scratch/nonmotion_scan.py`, all 6,577 waveforms, 19,731 channel-rows.
-Restricted to peak >= 0.05 in/s (n = 12,068), the distribution is **bimodal
-with an empty middle**:
-
-| mp band | channel-events |
-|---|---|
-| 0.0–0.1 | 11,384 |
-| 0.1–0.2 | 293 |
-| **0.15–0.85 (dead zone)** | **131 = 1.09%** |
-| 0.9–1.0 | 278 |
-
-At `mp >= 0.8` with >=3 events it returns **exactly the five confirmed units** —
-BE9558, BE11529, BE12599, BE13117, BE18438 — stable from 0.5 to 0.9. Two
-detectors on entirely different principles agreeing on the unit list is the
-strongest corroboration that list has.
-
-**BE11007 is settled: NOT an offset.** It reaches mp 0.75–0.89, but with
-`frac_neg = 0.99` at peaks of **7.4–9.4 in/s** — parked *negative* during a
-near-full-scale blast. §4's guess was right. `mp` alone cannot separate a
-pedestal from a large one-sided blast; pair it with a peak ceiling or with
-sign-consistency across events.
-
-⚠ **Not a rediscovery of the retracted v1 detector.** v1 scored only the
-largest-peak axis and used the mean as a *baseline estimator* where the median
-was required. Here the mean is the signal itself, per channel — that is what the
-physics licenses.
-
-**Correction to §8c.** That section says the spread gate is "blind to onsets by
-construction." Too strong: of 87 BE18438|Vert events at mp >= 0.5 the gate
-rejected **one** — the transitional record. It does not lose onsets
-systematically; it loses the transition specifically.
-
-### 8e. BE12599 — a connector, not a geophone (2026-09-07)
-
-Waveform shapes across its August episode, measured rather than eyeballed:
-
-| date | channel | shape |
-|---|---|---|
-| Aug 09 05:29 | Long | **unipolar +**, 0/2304 samples below zero, decay tau **26 ms** |
-| Aug 09 05:35 | Long | unipolar +, 3 spikes at irregular gaps (744, 1032 ms), tau **38 ms** |
-| Aug 14 05:00 | Long | single lobe, bipolar, tau **118 ms** |
-| Aug 17–23 | Tran | **flat DC pedestal**, sd/level 0.015–0.020, 0 zero crossings |
-
-**Unipolar impulses with an RC tail are not mechanical.** Fast rise, exponential
-decay, one polarity, irregular timing — that is charge dumped into a
-capacitively-coupled input and draining through the input resistance. The
-progression 26 ms -> 118 ms -> never recovers, over 14 days, is a leakage path
-worsening.
-
-**And the fault moved channels** — Long on Aug 9/14, Tran on Aug 17–23, Long
-again on Aug 21 (1.065 in/s) while Tran held its pedestal. Vert stayed clean
-throughout. **A failing geophone element cannot hop channels. A connector can.**
-
-That single fact explains what had been puzzling:
-- **The sensor self-check keeps passing** (7.4/7.5/7.6 Hz, ratios 3.6–4.2, all
-  four channels Passed, on the very events where Long throws 0.5 in/s spikes).
-  The swing test drives the element; the element is fine. The fault is in the
-  wiring to it.
-- **Why Instantel's autozero fixes only ~10%** — it cannot fix a connector.
-- **Why onset "ramps" over minutes to hours** — contact resistance drifting.
-
-All seven Aug 17–23 events are stamped **05:00:14**, the same second, and their
-filename extensions run `8E → WE → KE → 8E → WE → KE → 8E` — the documented
-3-day cycle for a fixed daily time. Clock-scheduled, not physically triggered:
-the modem powers up, draws a surge, and a marginal connection responds.
-
-**Field action: inspect and photograph the geophone connector BEFORE reseating
-anything** — an intermittent contact clears the moment it is disturbed.
-
-⚠ Scoped to BE12599. BE18438's onset was a smooth 7-minute ramp with no spikes,
-which looks like a different failure mode wearing the same signature.
-
----
-
-### ⚠ Serial prefixes — four of these units are BlastMates, not MiniMates
-
-Corrected 2026-09-06, after Brian queried "BA10895?" against a report that
-said BE10895. He was right. The BW filename encodes the serial **number
-only** — `L895` -> 10895 — and every offset scanner synthesised the family
-prefix as `"BE"`. Four of the 43 archive units are **BA** (BlastMate, the
-MiniMate Plus's bigger sibling; same Series III, byte-identical data):
-
-**BA9229, BA10060, BA10895, BA15957.**
-
-Read off the file bodies, which carry the serial verbatim. No analysis
-changed — grouping was always on the numeric part, and no unit number maps
-to two serials — but every earlier reference to "BE10895" and the other
-three is a label error and has been corrected throughout this document.
-
-The same assumption was live in two production sites and is fixed
-(`sfm/waveform_store.py`, `minimateplus/client.py`): the store would have
-filed a BlastMate under a unit that does not exist, and the monitor-log
-decoder lost the geo threshold along with the serial. See commit `9ceff65`.
-
----
-
-## 9. Chronology
-
-| date | event |
-|---|---|
-| 2026-08-25 | Reported as a *waveform decode bug* — traces with a DC offset. Investigation shows the offset is **real device data**; the decoder is correct. |
-| 2026-08-25 | Brian relays his dad's description: a known hardware fault called an "offset"; usually sent to Instantel. |
-| 2026-08-25 | First detection pass over the Terra-View DB: **2 of 21 units**, 274 events, 11 months. Flagged as vulnerable to survivorship bias — flooded events were routinely deleted. |
-| 2026-08-25 | Condensation hypothesis proposed, then **killed by its own controls**. |
-| 2026-08-25 | Parked pending the multi-year archive. |
-| 2026-08-28 | DL2 archive pulled (33 GB, 546k files; 6.6 GB working set). |
-| 2026-08-28 | Archive scan: **6 of 45 units**, 283 events, 15 episodes. Prior base rate **confirmed**, not overturned. |
-| 2026-08-28 | Clipping ruled out; `m/p` established as the discriminator; BE11007 reclassified as probably a real blast. |
-| 2026-08-28 | Calibration-timing correlation attempted and **rejected as confounded**. |
-| 2026-08-28 | Instantel FAQs supplied: autozero procedure, the **2027–2069** window, the **>5 counts** threshold. Explains the ~10% re-zero success rate. |
-| 2026-08-28 | Bimodality established; sensor check proven **blind** to offsets; `SUB 0x0E` identified as the best open lead. |
-| 2026-08-28 | **v1 detector retracted.** Brian challenged the "come and go" finding against field experience. Two flaws found: dominant-axis-only scoring and mean-instead-of-median. Corrected detector shows persistent pedestals on **8 of 45 units**, and the gaps are service windows. |
-| 2026-08-28 | **Detector v3 (Brian's method):** pre-trigger floor + pre/mid/end consistency. Healthy channels proven to sit at 0.000 +/-1 unit (94.5%), confirming no decoder zero-point bias. Final: **5 of 45 units (11%)**, threshold-insensitive. |
-| 2026-09-04 | **Histogram corpus scanned** — 63,505 of 63,535 files, 43 units, 77.9M intervals (9.7x the waveform corpus). `scratch/offset_hist_scan.py`. |
-| 2026-09-04 | Absolute-floor statistic **retired**: 66% of its discrimination is a day/site confound (other-channels-only AUC 0.746 vs 0.872). Zero-fraction shown **structurally impossible** — the device clamps every interval peak at >= 1 count. |
-| 2026-09-04 | Site-quiet-gated cross-channel differential established: **BE18438 Vert, BE9558 Tran+Long**, threshold-insensitive over a 2.3x span. Finds only **2 of the 5** confirmed units — leakage into the histogram floor is bimodal (0.9 to 0.02), so a negative result carries almost no information. Per-channel attribution **not** established (channel-scramble p = 0.769). |
-| 2026-09-04 | **BA10895 reclassified** from transient to a genuine Vert fault of a different subtype — 49.4% single-axis-dominant events, the highest in the fleet, 100% on Vert. The other six marginal units are clean. |
-| 2026-09-04 | **Defect found in `offset_scan3.py`**: its `spread <= 0.02` gate discards 18.8% of rows with \|pre\| >= 0.025, concentrated on 41 negative unit-channels; 4 would be sustained positives without it. The fleet label is three-state, not two. |
-| 2026-09-06 | **Four units relabelled BA, not BE** — BA9229, BA10060, BA10895, BA15957 are BlastMates. The BW filename carries only the serial number; the family prefix must be read from the file body. Fixed in the scanners and in two production sites. |
-| 2026-09-06 | **Mechanism campaign — five hypotheses, all dead.** Thermal, ground-motion shock, handling/redeployment, accumulated duty, unit age, firmware and a mechanical element fault are each refuted or bounded. 7 nominally significant results against 5.2 expected by chance. |
-| 2026-09-06 | **Onset is a RAMP of minutes-to-hours, not a step** — BE18438 Vert resolved to one-minute cadence, 50% of the excursion in 7 min, >=25 intermediates, ASCII-validated 75/75. Kills both a latched digital step AND slow component degradation. Surviving shape: a reversible two-time-constant settling process — a bias/leakage path charging a high-impedance node. |
-| 2026-09-06 | **Polarity consistency RETRACTED** (a tautology of the spread gate; amplitude-only gives 12 of 53 unit-channels mixed) and the fleet **re-cut to 8–9 units, not 5**. "Persistent until serviced" weakened: 23 recoveries, median gap 6 days — though a field autozero cannot be excluded. |
-| 2026-09-06 | The spread gate is **blind to onsets by construction** — it rejects a moving floor, which is what an onset is. It rejected the very record in which the ramp is visible. |
-| 2026-09-07 | **The non-motion test** (Brian: "it doesn't cross zero"). `\|mean\|/peak` is bimodal with a 1.09% dead zone and returns exactly the 5 confirmed units from physics, not a threshold. Independent corroboration of the unit list. **BE11007 settled as NOT an offset** — a one-sided 9 in/s blast. |
-| 2026-09-07 | **BE12599 is a connector fault, not a geophone fault.** Unipolar spikes with a 26→118 ms RC tail progressing to a flat pedestal, and the fault MOVES between Long and Tran while the sensor self-check passes on every event. An element cannot hop channels; a connector can. Inspect before reseating. |

docs/ri8507_compliance_curve.md

@@ -1,135 +0,0 @@
-# USBM RI8507 / OSMRE Blasting Compliance Curve — Reference
-
-Reference for the **velocity-vs-frequency blasting compliance chart** Blastware
-draws on its Event Report ("USBM RI8507 And OSMRE"), and how seismo-relay
-reproduces it. Implemented in [`sfm/compliance.py`](../sfm/compliance.py); the
-spectral (FFT) side lives in [`waveform_fft.py`](../waveform_fft.py).
-
-Reverse-engineered 2026-09-14 against 7 BE12844 (MiniMate Plus) events, each
-with a Blastware Event Report + FFT Report as ground truth. Curve values from
-USBM RI8507 Appendix B and 30 CFR 816.67.
-
----
-
-## What it is
-
-Two closely-related sources for the same limit curve:
-
-- **USBM RI8507** — Bureau of Mines *Report of Investigations 8507* (Siskind
-  et al., 1980), *"Structure Response and Damage Produced by Ground Vibration
-  From Surface Mine Blasting."* The curve is **Figure B-1**, Appendix B
-  ("Alternative Blasting Level Criteria"), p.73–74.
-- **OSMRE / OSM** — the Office of Surface Mining Reclamation and Enforcement
-  codified it as **30 CFR 816.67, Figure 1**. "CFR" = the U.S. Code of Federal
-  Regulations. Same curve, regulatory force.
-
-The chart plots each geophone channel's significant vibration cycles as
-`(frequency, peak velocity)` points against this limit. A point **below** the
-line passes; **above** fails.
-
----
-
-## The limit curve
-
-A structure has a resonance band (~4–12 Hz for whole structures) where it is
-most vulnerable, so the safe velocity is **lower** at those frequencies and
-**higher** away from them. The curve captures this by alternating two kinds of
-bound:
-
-- **Constant-velocity** segments — a flat horizontal line at a fixed PPV.
-- **Constant-displacement** segments — a fixed peak *displacement* `d`. For
-  simple harmonic motion, peak velocity `v = 2πf·d`, so on a velocity-vs-
-  frequency **log-log** plot this is a straight line of slope +1 (velocity rises
-  with frequency). This is why the low- and high-frequency bounds are sloped.
-
-### Two lines — structure type
-
-RI8507 gives two lines for two interior-wall constructions (Table 13, p.67):
-
-| line | construction | plateau PPV |
-|---|---|---|
-| **Drywall** (solid) | modern gypsum wallboard | **0.75 in/s** |
-| **Plaster** (dashed) | older plaster on wood lath | **0.50 in/s** |
-
-Plaster-on-lath is more damage-prone, hence the lower limit. You apply **one**
-line depending on the monitored structure.
-
-### The four segments (Figure B-1, p.74)
-
-Going low → high frequency, each line is:
-
-1. **Ultimate low-frequency bound** — constant displacement **0.030 in**
-   (`v = 2πf·0.030`). Only relevant below ~4 Hz.
-2. **Plateau** — constant velocity **0.75** (Drywall) / **0.50** (plaster) in/s.
-3. **Rising diagonal** — constant displacement **0.008 in** (`v = 2πf·0.008`),
-   climbing from the plateau up to the high-frequency cap.
-4. **High-frequency cap** — constant velocity **2.0 in/s** above ~40 Hz.
-
-The segments are drawn **continuous**: each bound is used over the frequency
-range where it is the binding (lowest) limit, and consecutive bounds meet where
-they are equal — so there are no vertical steps. Transition frequencies come
-straight from the values (`f = V / (2π·d)`):
-
-| transition | formula | Drywall | Plaster |
-|---|---|---|---|
-| 0.030 in → plateau | `V_mid / (2π·0.030)` | 3.98 Hz | 2.65 Hz |
-| plateau → 0.008 in | `V_mid / (2π·0.008)` | 14.92 Hz | 9.95 Hz |
-| 0.008 in → 2.0 in/s | `2.0 / (2π·0.008)` | 39.79 Hz | 39.79 Hz |
-
-Because both lines share the same **0.008 in** rising diagonal, above ~15 Hz
-they lie on the *same* line (both reach 2.0 in/s at ~40 Hz) — RI8507's literal
-construction merges them there. Blastware renders the dashed line as a separate
-parallel diagonal, but that is cosmetic: above ~15 Hz both structure types carry
-the identical limit, so compliance is unaffected.
-
-> ⚠ RI8507's *Table 13* is a simpler two-range criterion with a **sharp
-> discontinuity at 40 Hz** (flat plateau, then a jump to 2.0). Figure B-1 is the
-> **smoothed** version that adds the 0.008 in transition — that is the one drawn
-> on reports and implemented here.
-
----
-
-## The compliance scatter (the points)
-
-The cloud is **not** the FFT spectrum. It is a per-cycle, time-domain measure by
-the **zero-crossing method** (`channel_compliance_points`):
-
-- Split the channel's waveform at its zero crossings.
-- Each half-cycle contributes one point: **frequency** `= 1 / (2 · half-period)`
-  (from the samples between the two crossings), **velocity** `= peak |amplitude|`
-  in that half-cycle.
-
-This yields ~90–110 points per channel, and — by construction — each channel's
-**highest** point equals that channel's PPV. Verified against Blastware: the
-cloud shape, density, and ceiling all match.
-
-### Why not the FFT?
-
-A broadband blast spreads its energy across many FFT bins, so no single bin
-reaches the time-domain peak — the FFT amplitudes come out ~10× below the
-compliance-chart velocities. The compliance chart is a *per-cycle peak* view;
-the **FFT** is a separate analysis (Blastware's *FFT Report*), reproduced by
-[`waveform_fft.py`](../waveform_fft.py) and used for the dominant-frequency
-readout and the #10 FFT view — not for this scatter.
-
----
-
-## Implementation
-
-- `sfm/compliance.py`
-  - `limit_at(freq, curve)` — the limit PPV at a frequency (`curve` = `"Drywall"`
-    or `"Plaster"`); curves are data in `_CURVES`, so more standards can be added.
-  - `channel_compliance_points(samples, sps)` — the zero-crossing scatter.
-  - `draw_compliance_chart(ax, channels, sps)` — matplotlib rendering (both
-    limit lines + per-channel scatter, Blastware's tick scales and channel
-    markers: Tran `+` red, Vert `×` green, Long `o` blue).
-- Tests: `tests/test_compliance.py`.
-
----
-
-## Sources
-
-- USBM **RI8507** (Siskind, Stagg, Kopp, Dowding, 1980), Appendix B / Figure B-1,
-  p.73–74; Table 13, p.67. (`ref-stuff/usbm-ri8507-ground_vibration.pdf`.)
-- **30 CFR 816.67**, "Use of explosives: Control of adverse effects," Figure 1 —
-  <https://www.ecfr.gov/current/title-30/chapter-VII/subchapter-K/part-816/section-816.67>

docs/runbooks/wedged_unit_recovery.md

@@ -1,7 +1,6 @@
 # Runbook — Recovering a wedged unit stuck in a call-home loop
 
-**Incidents:** BE9558H at `166.246.130.1:9034`, 2026-05-17 (Method B) ·
+**Original incident:** BE9558H at `166.246.130.1:9034`, recovered 2026-05-17.
-BE12599 at `166.246.64.226:9034`, 2026-09-16 (Method A).
 
 A field unit with a stuck-triggered geophone (or any hardware fault causing
 constant event triggering) will record events back-to-back, and if Auto Call
@@ -15,33 +14,6 @@ This runbook describes how to break the loop and recover control.
 
 ---
 
-## ⚠ Two cures for one disease — intercept first
-
-Both incidents below are the **same failure**: a geophone offset crosses the
-trigger level, the unit records back-to-back, ACH set to "after event
-recorded" dials continuously, and the unit becomes unreachable because its
-modem is in client mode almost all of the time.
-
-There are two ways to get a Stop Monitoring command into it.
-
-| | **A — intercept the call** (preferred) | **B — catch it between calls** (original) |
-|---|---|---|
-| Idea | Be the server it dials.  Point the modem's Destination at our own ACH server and answer it. | Clear the Destination so it stops dialing, then race a Stop into the gap. |
-| Needs inbound? | **No — the unit calls us** | Yes: working inbound TCP to the modem |
-| Determinism | Deterministic — it dials every ~75 s, we only have to be listening | A race.  BE9558H took ~7 h of attempts before one landed. |
-| Tool | `bridges/ach_server.py --stop-monitoring` | `scripts/slow_drip.sh` |
-| Proven on | BE12599, 2026-09-16 | BE9558H, 2026-05-17 |
-
-**Method A is the standard procedure now.**  The unit won't answer us because
-it is on the phone — so stop dialing it and be the one it calls.  It rings,
-we pick up, take its data, and tell it to stop calling here.
-
-Method B is kept because it is proven, and because A needs a listener the
-modem can actually reach (public IP + forwarded port).  When you have that,
-don't race it — intercept it.
-
----
-
 ## Symptoms
 
 - Terra-View / SFM `/device/info` either hangs or fails on `count_events()`.
@@ -59,85 +31,9 @@ If you see *all* of these, the unit is in this exact failure mode.
 
 ---
 
-## Method A (preferred) — intercept the call
+## Quick reference — how to recover
 
-You need **ACEmanager access** and a host the modem can dial: public IP with
+You need **ACEmanager access** to the unit's modem.
-the listener's port forwarded to it.
-
-### A1 — start the listener BEFORE touching the modem
-
-```bash
-cd /home/serversdown/seismo-relay
-tmux new -s rescue
-.venv/bin/python -u bridges/ach_server.py --port 12345 \
-  -o bridges/captures/<unit>-diag --stop-monitoring -v
-```
-
-⚠ **Listener first, always.**  A Destination pointed at a dead port is the
-worst state available — the device still dials, the modem still flips to
-client mode, inbound stays blocked, and nothing gets delivered.
-
-Do **not** add `--events-only` (it silently breaks dedup — see gotchas), and
-do **not** add `--disable-ach` yet (see A4).
-
-### A2 — point the modem at it
-
-ACEmanager → **Serial → Port Configuration**:
-
-| Field | Set to |
-|---|---|
-| **Destination Address** | the listener's public IP |
-| **Destination Port** | the listener's port (e.g. `12345`) |
-
-Apply.  The modem auto-dials its Destination whenever serial data arrives
-while the serial port is closed — so the unit's own retry cycle now lands on
-you instead of nowhere.
-
-### A3 — answer, and stop the bleeding
-
-Within ~75 s you should see a call-in.  `--stop-monitoring` fires SUB 0x97 at
-step 1.5 — after the handshake, **before** the event walk — so the recording
-halts at the earliest possible moment in the session.  Confirm via
-`rescue.json` in the session directory:
-
-```json
-{"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
-```
-
-That is the bleeding stopped.  Everything after this is cleanup.
-
-### A4 — drain the backlog, THEN disable ACH
-
-⚠ **Order matters, and it is counter-intuitive.**  Stopping monitoring also
-removes your call-in trigger: ACH fires on "after event recorded", so with
-recording stopped the unit has no reason to dial again.  The backlog sitting
-in its memory does **not** re-arm it.
-
-So if the stored events are worth keeping — and on a fault unit they usually
-are, they're the evidence — drain them across however many call-ins it takes
-*before* you silence it.  Only then add `--disable-ach` (or use
-`scripts/rescue_device.sh <host> <port> --no-erase`).
-
-If the unit has gone quiet and you still need it, cycling the modem produces
-a call-in, and a unit with a scheduled daily call will dial at its configured
-time regardless.
-
-### A5 — restore the Destination, and confirm you did
-
-Put `Destination Address` back to `0.0.0.0` (or the office Instantel ACH
-server) once you are finished, and only stop the listener after that is done.
-
-### A6 — do NOT re-enable ACH until the hardware fault is repaired
-
-Otherwise the loop restarts the moment monitoring resumes and you run this
-runbook again.
-
----
-
-## Method B (fallback) — catch it between calls
-
-The original 2026-05 procedure.  Use when you cannot stand up a listener the
-modem can reach.  You need **ACEmanager access** to the unit's modem.
 
 ### Step 1: stop the modem's mode-flipping
 
@@ -357,223 +253,3 @@ service).
 
 Total time from "i was wondering if its possible to" first attempt to
 recovery: ~7 hours of intermittent debugging across one evening.
-
----
-
-# Second incident — BE12599, 2026-09-16/17
-
-**Unit:** BE12599 at `166.246.64.226:9034`, RV50, job *I-80 North Fork Bridge
-— Abut 1 West* (Fay Company).  Same job as BE9558H, which is a coincidence.
-
-**Fault:** the connector fault documented in `docs/offset_investigation.md`
-§8e progressed until the Tran pedestal reached **0.400 in/s** — its trigger
-level.  Constant triggering → constant recording → ACH "after event recorded"
-→ continuous dialing.  Same disease as BE9558H.
-
-**Same disease, inverted cure.**  Method B's Step 1 *did* work — clearing the
-Destination stopped the dial-outs, confirmed in the ALEOS log.  It was Step 2
-that didn't land, and rather than keep racing we turned the rescue around:
-gave the unit a different server to call, and answered it.
-
-Total time ≈ 5 h, of which ~90 min went to two red herrings documented below.
-Much of the rest was rediscovering the May procedure, which is why the
-"two cures" table now sits at the top of this file.
-
----
-
-## Turn on ALEOS_SERIAL debug FIRST
-
-This is the single highest-value diagnostic and it should be step zero on any
-future incident.  ACEmanager → **Admin → Log → ALEOS_SERIAL log level →
-DEBUG**, then view the serial log.
-
-It is the only thing that tells you what the *device* is actually saying.
-Everything before we did this was guesswork.
-
-## What the log showed — the unit is on the phone
-
-Every ~75 seconds, verbatim:
-
-```
-ALEOS_SERIAL_HIF: 29 byte(s) in buffer: 'ATQ1^MATE0^MATS0=2^M^MRADIO RING^M'
-ALEOS_SERIAL_HMC: TCP recvhost fd 65535 len 29  state TCPMode::kClosed
-ALEOS_SERIAL_HMC: tcpmode trying to send to invalid socket
-ALEOS_SERIAL_HMC: Connect to IP: 0.0.0.0 Port 0
-ALEOS_SERIAL_HMC: Initialize Auto answer on port 9034
-ALEOS_SERIAL_HMC: Cannot connect to 0.0.0.0
-```
-
-Read that carefully:
-
-- `ATQ1` (quiet) / `ATE0` (echo off) / `ATS0=2` (auto-answer after 2 rings).
-  **There is no `ATD`.**  The device is not dialing — it is trying to
-  *configure* its modem.
-- The modem's serial port is in TCP data mode, so it never interprets these
-  as AT commands.  It treats them as payload and tries to ship them to a TCP
-  socket that does not exist.
-- The device therefore never receives `OK`, never progresses, and **retries
-  the identical 29 bytes forever**.
-
-**While it is in this state it is busy placing a call, not listening for
-us.**  This is almost certainly what BE9558H was doing too — we simply never
-turned on ALEOS_SERIAL debug in May to look.  It is not a different disease;
-it is the same one, seen properly for the first time.
-
-It is also the argument for Method A in one picture: the unit is mid-dial
-every ~75 s, and our inbound Stop has to thread the gaps between those
-attempts.  Give it somewhere to dial and the problem inverts into a
-deterministic one.
-
-### Why `slow_drip` lied
-
-`slow_drip` returned the *success* signature except for the one field that
-mattered:
-
-```json
-{"duration_s":120.0,"drips_sent":38,"bytes_sent":920,
- "bytes_received":0,"send_error":null}
-```
-
-Full duration, no broken pipe — but zero bytes back.  Cause is in the log
-above: each 75 s cycle re-runs `Initialize Auto answer on port 9034`, which
-orphans the held session (`data in for unknown reason 3 removing from
-select`, `OnMsg recv error: 107 - Transport endpoint is not connected`).  Our
-local TCP stayed open so `sendall` never raised — but the modem stopped
-bridging after the first re-init, so every drip after that went into a socket
-nobody was reading.
-
-⚠ **`send_error: null` + full duration is NOT success.  Only
-`bytes_received > 0` is success.**
-
-⚠ **In fairness to slow_drip: it got exactly one attempt here**, run ~90 s
-after a modem reboot, with a dead session visible in the log at 20:19:17 in
-that same window.  BE9558H took hours of attempts before one landed.  Method B
-was not ruled out on BE12599 so much as abandoned in favour of something that
-doesn't need luck.
-
----
-
-## ⚠ Two red herrings that cost ~90 minutes
-
-### 1. The trusted-IP whitelist (this was the real reason inbound never worked)
-
-The RV50s run with **Security → Trusted IPs (Friends List) enabled**.  A
-source IP that is not on the list is dropped **silently** — inbound presents
-as `Connection error: timed out`, never a refusal.
-
-Brian's dev-box public IP is **dynamic** and had changed, so `tmi-dev` was no
-longer whitelisted.  Every inbound attempt failed identically across four
-different modem and device states, which looked exactly like the BE9558H
-mode-flipping symptom and sent us chasing modem configuration for over an
-hour.
-
-**Check this before diagnosing anything else.**  Note that SFM in Docker
-egresses via the *host's public IP*, not its LAN IP.
-
-### 2. A 502 from SFM does not mean TCP connected
-
-`sfm/server.py` raises **502 for both** failure classes:
-
-```python
-raise HTTPException(status_code=502, detail=f"Protocol error: {exc}")
-raise HTTPException(status_code=502, detail=f"Connection error: {exc}")
-```
-
-We read an early 502 as "TCP connected, modem bridged, device mute" and built
-a whole theory on it.  It was almost certainly a connect timeout.
-**Always read the `detail` string** — "connect failed" and "device didn't
-answer" are completely different problems and the status code will not
-separate them.
-
----
-
-## What actually worked — invert the direction
-
-The key observation is in the log above:
-
-> `TCP recvhost ... state TCPMode::kClosed` → `Connect to IP: 0.0.0.0 Port 0`
-
-**The modem auto-dials its Destination whenever serial data arrives while
-closed.**  So instead of fighting for inbound, give it somewhere to dial:
-point `Destination Address` at our own `ach_server` and the device's own
-75-second attempts become **device-initiated sessions the modem bridges
-correctly**.  No race, no contention, worst case a 75-second wait.
-
-### Procedure
-
-1. **Run the rescue server** on a host the modem can reach (public IP +
-   forwarded port):
-
-   ```bash
-   cd /home/serversdown/seismo-relay
-   .venv/bin/python -u bridges/ach_server.py --port 12345 \
-     -o bridges/captures/<unit>-diag --stop-monitoring -v
-   ```
-
-2. **Point the modem at it** — ACEmanager → Serial → Port Configuration →
-   `Destination Address` = your public IP, `Destination Port` = 12345.
-
-3. **Wait for the call-in.**  `--stop-monitoring` fires SUB 0x97 at step 1.5,
-   after the handshake and *before* the event walk.  Confirm via
-   `rescue.json` in the session directory:
-
-   ```json
-   {"peer": "166.246.64.226:60921", "stop_monitoring": "ok"}
-   ```
-
-4. **Restore the modem's Destination** once you are done, then finish the
-   device side (disable ACH, erase) through whichever channel works.
-
-On BE12599 the first call-in landed at 20:58:11 and reported
-`stop_monitoring: ok`; a second at 20:58:20 confirmed it.  `is_monitoring:
-false` was still true **6½ hours later** — the fix is durable.
-
----
-
-## Hard-won gotchas (do not re-derive)
-
-- **Never leave the Destination pointed at a host with nothing listening.**
-  That is the worst state available: the device still dials, the modem still
-  flips, inbound stays blocked, and nothing is delivered.  An 8-minute gap
-  with the listener down produced a spurious inbound timeout that cost
-  another round of misdiagnosis.
-
-- **Stopping monitoring removes your call-in channel.**  ACH is "after event
-  recorded"; no new events means no new dials.  The backlog sitting in memory
-  does *not* re-arm it.  After a successful stop the unit goes quiet and you
-  need the modem cycled (works — produced a call-in), the scheduled daily call
-  (BE12599 calls at **05:00:14 device-local**, per §8e), or working inbound.
-  **Plan the order before you fire the stop.**
-
-- **`--events-only` silently breaks dedup.**  It skips the device-info step,
-  so the serial is never read; `ach_state.json` then keys on
-  `peer:ephemeral_port`, which is unique per connection.  Every session looks
-  like a new unit, starts from key 0, and re-downloads the same event.  Four
-  sessions on BE12599 downloaded the identical event four times and made zero
-  progress on the backlog.  Events also file as `serial=UNKNOWN` with a
-  `M000…` BW filename (serial_numeric 0) instead of `N599…`.
-  **Do not use `--events-only` when you intend to download anything.**
-
-- **`/device/events/index` reported `lifetime_count: 0`** on a unit with years
-  of history.  Suspected decode bug in the SUB 0x08 field offset — do not
-  trust that number.  The 88-byte payload is preserved in the `raw_hex` field
-  if someone wants to chase it.
-
-- **Memory used cross-checks the event keys exactly:**
-  `last_key − buffer_start = memory_total − memory_free`.  On BE12599:
-  `0x011230ec − 0x01110000 = 78,060` and `983,028 − 904,968 = 78,060`.
-  Useful sanity check that you are reading the keys right.
-
----
-
-## Final state (2026-09-17 ~01:30 local)
-
-- `is_monitoring: false`, held 6½ hours
-- Battery 6.76 V
-- Memory 78,060 / 983,028 bytes used (8%)
-- `first_key 01121728`, `last_key 011230ec` — ~6.6 KB of addressable event
-  chain, roughly 3 events
-- ACH still **enabled** — to be disabled after the backlog is preserved
-- Modem Destination still pointed at tmi-dev — to be restored
-- ⚠ **Do not re-enable ACH until the connector is serviced.**  Tran is still
-  sitting at 0.400 and the loop restarts the moment monitoring resumes.

docs/sfm_tool_status.md

@@ -1,150 +0,0 @@
-# SFM — where it actually stands as a tool
-
-**Status as of 2026-09-20 (v0.31.0).**  This is the honest assessment, not the
-roadmap — `README.md § Roadmap` covers where it is *going*.  Expect this file to
-go stale; re-date it when you revise it.
-
----
-
-## The framing
-
-SFM is **three different things wearing one name**, at three very different
-levels of maturity:
-
-| | what it is | maturity |
-|---|---|---|
-| **The codec library** | `minimateplus/`, `micromate/` — bytes in, `Event` out | **Production.** Verified per-sample at scale. |
-| **SDM — the data side** | the DB, waveform store, `/db/*`, ingest | **Production.** Terra-View depends on it daily. |
-| **SFM — the device side** | `/device/*`, live connections to units | **Emergency-grade.** Works, but manual, unauthenticated, and thinly tested. |
-| **The lab** | `seismo_lab.py`, `scratch/`, the Inspector | **Research artifacts.** Useful, not products. |
-
-Brian's own description — *"right now it's an emergency tool and a research
-project"* — is accurate, and it applies specifically to the **device side**.
-The data side is not an emergency tool; it has been carrying production for
-months.
-
-Most confusion about "is SFM reliable?" comes from answering for the wrong
-tier.
-
----
-
-## 1. What you can rely on
-
-### Production-grade — trust it
-
-- **Series-3 decode.** 14,338 / 14,338 files decode per-sample exact against
-  preserved Blastware ASCII exports, 45 units, files back to 2018.
-- **Series-4 (Thor) decode.** 1,057,536 / 1,057,536 geo samples exact against
-  Thor's own CSV exports; production IDFW 575/575 with zero truncations.
-- **Histogram decode.** 1,211 / 1,211 production histograms exact, including
-  842,442 per-interval frequency comparisons with zero mismatches.
-- **The ingest path.** `/db/import/blastware_file` and `/db/import/idf_file`
-  fed by the watchers — this is how prod actually gets its data, and it has
-  been running unattended for months.
-- **`/db/*` read API.** Always-on, consumed by Terra-View for every fleet
-  listing, event detail and report.
-- **The waveform store** — `.h5` + `.sfm.json` sidecars + retained raw
-  binaries, with operator review state preserved across regeneration.
-- **`bridges/ach_server.py`** — speaks the full BW protocol to calling units.
-  Proven in the field, including as a rescue tool (see the runbook).
-
-### Emergency-grade — works, but you are the error handling
-
-- **`/device/*` live endpoints.**  They do what they say.  But they are
-  synchronous, unauthenticated, and a single cellular download can exceed the
-  60 s timeouts that sit in front of them.
-- **The rescue ladder** (`rescue`, `stop_monitoring_*`, `events/erase`).
-  Each has worked in a real incident — but each has been used a handful of
-  times, by one person, with the runbook open.
-- **The standalone webapp.**  Perfectly usable, and as of v0.31.0 the cheap
-  probes and rescue actions are reachable without curl.  No auth of any kind.
-
-### Research artifacts — useful, not products
-
-- **`seismo_lab.py`** — 2,789 lines of Tkinter (Bridge / Analyzer / Query DB /
-  Inspector).  Desktop-only, single-user, no tests.
-- **`scratch/`** — the verification harnesses (`verify_against_ascii.py`,
-  `verify_thor_against_csv.py`) and the offset detector (`offset_scan3.py`).
-  These produced the numbers the production claims rest on, so they matter —
-  but they are analysis scripts, not maintained code.
-- **`docs/offset_investigation.md`** — an open investigation, not a feature.
-
----
-
-## 2. What to use when
-
-| you want to… | use | notes |
-|---|---|---|
-| Know if a unit is monitoring / its battery / memory | `GET /device/monitor/status?force=true` | ~2 s |
-| Know whether ACH is on | `GET /device/call_home` | ~2 s.  **Not** `/device/events`. |
-| See how full a unit's buffer is | `GET /device/events/storage_range` | ~2 s, no chain walk |
-| Stop a runaway unit | Diagnostics tab → Stop Monitoring | see the runbook first |
-| Reach a unit that will not answer | **point its modem at an `ach_server` and answer its call** | runbook Method A — do not race it |
-| List a unit's stored events | Events tab → Load events | **slow**, and broken past 64 KB (below) |
-| Get event data into the DB | the watcher → `/db/import/*` path | not the live walk |
-
-The single most useful habit: **the cheap probes are cheap and the event walk
-is not.**  Reaching for `/device/events` to answer a yes/no question about a
-unit is the mistake that motivated the v0.31.0 webapp changes.
-
----
-
-## 3. Known issues
-
-| issue | impact | status |
-|---|---|---|
-| **5A walk dies once a unit's buffer crosses 64 KB** | `/device/events` 500s; event body never downloads | Known, documented in `CLAUDE.md`.  Needs a BW capture of a spanning event to fix properly. |
-| **No auth on SFM at all** | 21 `/device/*` endpoints, including destructive ones, open to anything that reaches the port | Design agreed (Terra-View as authenticated jump host); not built. |
-| **Swagger try-it-out is live on destructive endpoints** | `POST /device/events/erase` is one click away at `:8200/docs` | Partially mitigated: the webapp's erase now requires typing the serial.  `/docs` itself is unguarded. |
-| **`SUB 0x08` lifetime counter reads 0** | `/device/events/index` returns a meaningless number | Suspected field-offset bug.  Surfaced in the UI as "unreliable". |
-| **Long device operations are synchronous** | 60 s timeouts in `routers/sfm.py` and the reverse proxy; a full download exceeds both | Known design constraint.  Must be POST-starts-job / GET-polls before any remote lab. |
-| **`backfill_sidecars.py --force` silently inserts DB rows** | store files with no DB row get one; the dry-run does not report the count | Known.  Avoid `--force` — `TOOL_VERSION` gates regeneration anyway. |
-| **14 sensitive-range files show an exact 8× discrepancy** | 10.0 / 1.25 — a units problem, not a decode problem | Open, not blocking. |
-| **16 failing tests on `dev`** | 15 need gitignored fixture bundles; 1 is real (`sc["peak_values"]["transverse"]` returns `None` where `0.0` is expected) | The real one shipped in v0.31.0. |
-
----
-
-## 4. What stands between this and a real tool
-
-Roughly in dependency order — each unblocks the ones below it.
-
-**1. Authentication.**  Everything else is gated on this.  SFM has none, and
-the modem IP whitelist gives zero protection because SFM *is* the whitelisted
-origin.  The agreed design delegates rather than builds: Terra-View becomes the
-authenticated jump host (`/api/sfm/*` already inherits deny-by-default operator
-auth), and the `8200:8200` publish is dropped so Terra-View is the only door.
-
-**2. Async long operations.**  POST starts a job, GET polls.  Retrofitting this
-after building a remote lab on top of synchronous endpoints would be far worse
-than designing for it now.
-
-**3. Confirm-guards on the remaining destructive endpoints.**  Auth answers
-*who*, not *did you mean it*.  The webapp's erase is guarded; the other seven
-destructive POSTs and `/docs` are not.
-
-**4. The 5A page-boundary fix.**  Until this lands, live event download is
-unreliable on exactly the units most likely to need attention — the ones that
-have been recording heavily.  Wants a Blastware capture of an event spanning a
-page boundary before the chunk-addressing half is trustworthy.
-
-**5. A live Thor / Micromate client.**  The device side is MiniMate-only.
-Series-4 units can only be read from forwarded files, so half the fleet has no
-live path at all.
-
-**6. Test coverage that runs from a clean checkout.**  15 of 16 current
-failures are missing fixture bundles.  A test suite that cannot go green on a
-fresh clone cannot gate anything.
-
-**7. The SDM rename.**  Cosmetic relative to the above, but the longer `sfm/`
-holds the data-side code the more the tiers blur.  ~30–50 files here, ~10–15 in
-Terra-View, plus a Docker volume migration.  Do it when the codebase is quiet.
-
----
-
-## The short version
-
-The **data side is a real tool already**.  The **device side is a set of sharp
-instruments** that work in the hands of the person who wrote them, with the
-runbook open.  The gap between those two states is mostly **auth, async, and
-guardrails** — not protocol work.  The protocol is the part that is actually
-finished.

docs/superpowers/plans/2026-08-21-waveform-ft-detection-phase-a.md

@@ -1,628 +0,0 @@
-# Waveform-shape FT detection — Phase A (seismo-relay) Implementation Plan
-
-> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
-
-**Goal:** Compute per-event waveform-shape metrics (crest factor + points-near-peak) in SFM and store them as `events` columns, populated at ingest and by a backfill script, so Terra-View can read them from `/db/events`.
-
-**Architecture:** A pure DSP module (`sfm/shape_metrics.py`) turns decoded `.h5` samples into shape metrics. New nullable `events.shape_*` columns are added via the existing `_migrate` ADD COLUMN loop. The three `WaveformStore.save*` paths compute shape from the just-written `.h5` and hand it to `insert_events`; a backfill script does the same over existing events. Mirrors exactly how per-channel ZC frequency was added.
-
-**Tech Stack:** Python 3.10, numpy, h5py, sqlite3 (raw), pytest. seismo-relay venv: `/home/serversdown/seismo-relay/.venv/bin/python3`.
-
-## Global Constraints
-
-- Metrics are read from the `.h5` `samples/{Tran,Vert,Long}` float32 arrays (physical in/s). The measured channel is the max-|peak| geophone channel.
-- All new columns are nullable; histogram records and events without usable samples store NULL (Terra-View falls back to cheap signals). Legacy rows stay valid.
-- Crest factor = `max(|x|) / rms(x)`; near-peak count = number of samples with `|x| ≥ 0.5·peak`. Threshold `0.5` is a module constant so calibration can tune it.
-- No manual migration: columns are added in `SeismoDb._migrate`, run at `SeismoDb()` construction.
-- `/db/events` needs no change — it returns all columns via `SELECT *` (verify with a test).
-- Run tests with `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest`.
-
----
-
-### Task 1: Shape DSP module
-
-**Files:**
-- Create: `sfm/shape_metrics.py`
-- Test: `tests/test_shape_metrics.py`
-
-**Interfaces:**
-- Produces:
-  - `channel_shape(x) -> dict | None` — `{"crest_factor": float, "near_peak_count": int, "sample_count": int}` or None for unusable input (size < 2, flat, all-zero).
-  - `shape_from_samples(chans: dict[str, ArrayLike]) -> dict | None` — picks the max-peak geophone channel; returns `{"crest_factor","near_peak_count","sample_count","axis"}` or None.
-  - `shape_from_h5(path) -> dict | None` — reads `samples/{Tran,Vert,Long}` and delegates to `shape_from_samples`; None on any read error.
-  - Constant `NEAR_PEAK_FRACTION = 0.5`.
-
-- [ ] **Step 1: Write the failing test**
-
-```python
-# tests/test_shape_metrics.py
-import numpy as np
-from sfm.shape_metrics import channel_shape, shape_from_samples
-
-def test_needle_spike_high_crest_few_near_peak():
-    x = np.zeros(1024); x[500] = 1.0          # one isolated spike
-    s = channel_shape(x)
-    assert s["sample_count"] == 1024
-    assert s["crest_factor"] > 15             # peak towers over rms
-    assert s["near_peak_count"] <= 3          # almost nothing near the peak
-
-def test_ringing_low_crest_many_near_peak():
-    t = np.arange(1024)
-    x = np.sin(2*np.pi*t/32) * np.exp(-t/4000)  # decaying oscillation
-    s = channel_shape(x)
-    assert s["crest_factor"] < 6
-    assert s["near_peak_count"] > 30          # many samples near the peak
-
-def test_channel_shape_none_for_unusable():
-    assert channel_shape(np.zeros(1024)) is None   # flat / all-zero
-    assert channel_shape(np.array([1.0])) is None  # too short
-
-def test_shape_from_samples_picks_max_peak_axis():
-    chans = {"Tran": np.zeros(1024), "Vert": np.zeros(1024), "Long": np.zeros(1024)}
-    chans["Long"][10] = 0.5
-    chans["Vert"] = np.sin(np.arange(1024)/5) * 0.01
-    s = shape_from_samples(chans)
-    assert s["axis"] == "Long"                # Long has the biggest peak
-    assert s["near_peak_count"] <= 3
-
-def test_shape_from_samples_none_when_no_geo():
-    assert shape_from_samples({"MicL": np.ones(1024)}) is None
-```
-
-- [ ] **Step 2: Run test to verify it fails**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_metrics.py -q`
-Expected: FAIL — `ModuleNotFoundError: sfm.shape_metrics`.
-
-- [ ] **Step 3: Write minimal implementation**
-
-```python
-# sfm/shape_metrics.py
-"""Waveform-shape metrics for false-trigger detection.
-
-A false trigger is an isolated impulse (quiet → spike → quiet); a real event
-rings for many cycles. Two numbers separate them: crest factor (how far the
-peak stands above the typical sample) and how many samples sit near the peak.
-"""
-from __future__ import annotations
-import numpy as np
-
-_GEO_CHANNELS = ("Tran", "Vert", "Long")
-NEAR_PEAK_FRACTION = 0.5   # a sample "near the peak" is >= this * peak amplitude
-
-
-def channel_shape(x) -> dict | None:
-    x = np.asarray(x, dtype=float)
-    if x.size < 2:
-        return None
-    peak = float(np.max(np.abs(x)))
-    if peak <= 0:
-        return None
-    rms = float(np.sqrt(np.mean(x ** 2)))
-    if rms <= 0:
-        return None
-    near = int(np.sum(np.abs(x) >= NEAR_PEAK_FRACTION * peak))
-    return {"crest_factor": peak / rms, "near_peak_count": near,
-            "sample_count": int(x.size)}
-
-
-def shape_from_samples(chans: dict) -> dict | None:
-    best_axis, best_peak, best_x = None, -1.0, None
-    for ax in _GEO_CHANNELS:
-        x = chans.get(ax)
-        if x is None:
-            continue
-        x = np.asarray(x, dtype=float)
-        if x.size < 2:
-            continue
-        p = float(np.max(np.abs(x)))
-        if p > best_peak:
-            best_axis, best_peak, best_x = ax, p, x
-    if best_axis is None:
-        return None
-    s = channel_shape(best_x)
-    if s is None:
-        return None
-    s["axis"] = best_axis
-    return s
-
-
-def shape_from_h5(path) -> dict | None:
-    import h5py
-    try:
-        with h5py.File(path, "r") as f:
-            chans = {ax: f[f"samples/{ax}"][:] for ax in _GEO_CHANNELS
-                     if f"samples/{ax}" in f}
-    except Exception:
-        return None
-    return shape_from_samples(chans)
-```
-
-- [ ] **Step 4: Run test to verify it passes**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_metrics.py -q`
-Expected: PASS (5 tests).
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add sfm/shape_metrics.py tests/test_shape_metrics.py
-git commit -m "feat(shape): crest-factor + points-near-peak waveform metrics"
-```
-
----
-
-### Task 2: shape_from_h5 round-trips a real .h5
-
-**Files:**
-- Test: `tests/test_shape_metrics_h5.py`
-
-**Interfaces:**
-- Consumes: `sfm.shape_metrics.shape_from_h5`; `h5py`.
-
-- [ ] **Step 1: Write the failing test** (writes a tiny .h5 the same shape SFM writes, then reads it back)
-
-```python
-# tests/test_shape_metrics_h5.py
-import numpy as np, h5py
-from sfm.shape_metrics import shape_from_h5
-
-def _write_h5(path, chans):
-    with h5py.File(path, "w") as f:
-        g = f.create_group("samples")
-        for k, v in chans.items():
-            g.create_dataset(k, data=np.asarray(v, dtype="float32"))
-
-def test_shape_from_h5_reads_dominant_axis(tmp_path):
-    p = tmp_path / "ev.h5"
-    long = np.zeros(1024, dtype="float32"); long[100] = 0.48
-    _write_h5(p, {"Tran": np.zeros(1024), "Vert": np.zeros(1024), "Long": long,
-                  "MicL": np.ones(1024)})
-    s = shape_from_h5(str(p))
-    assert s["axis"] == "Long" and s["near_peak_count"] <= 3
-
-def test_shape_from_h5_none_on_missing_or_degenerate(tmp_path):
-    assert shape_from_h5(str(tmp_path / "nope.h5")) is None
-    p = tmp_path / "degen.h5"
-    _write_h5(p, {"Tran": np.zeros(1), "Vert": np.zeros(1), "Long": np.zeros(1)})
-    assert shape_from_h5(str(p)) is None
-```
-
-- [ ] **Step 2: Run to verify it fails**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_metrics_h5.py -q`
-Expected: FAIL (assertion or, if Task 1 incomplete, import error).
-
-- [ ] **Step 3: Implementation** — none needed; `shape_from_h5` already exists from Task 1. If a test fails, fix `shape_from_h5` (not the test).
-
-- [ ] **Step 4: Run to verify it passes**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_metrics_h5.py -q`
-Expected: PASS (2 tests).
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add tests/test_shape_metrics_h5.py
-git commit -m "test(shape): shape_from_h5 round-trips a real .h5"
-```
-
----
-
-### Task 3: Add shape columns to the events schema + migration
-
-**Files:**
-- Modify: `sfm/database.py` — `_SCHEMA` CREATE TABLE `events` (after `mic_zc_above_range`); `_migrate` ADD COLUMN loop (the tuple around line 205-218).
-- Test: `tests/test_shape_columns.py`
-
-**Interfaces:**
-- Produces: `events` columns `shape_crest_factor REAL`, `shape_near_peak_count INTEGER`, `shape_sample_count INTEGER`, `shape_axis TEXT`.
-
-- [ ] **Step 1: Write the failing test**
-
-```python
-# tests/test_shape_columns.py
-import sqlite3
-from sfm.database import SeismoDb
-
-_SHAPE_COLS = {"shape_crest_factor", "shape_near_peak_count",
-               "shape_sample_count", "shape_axis"}
-
-def _cols(db):
-    with sqlite3.connect(db.db_path) as c:
-        return {r[1] for r in c.execute("PRAGMA table_info(events)")}
-
-def test_fresh_db_has_shape_columns(tmp_path):
-    db = SeismoDb(tmp_path / "s.db")
-    assert _SHAPE_COLS <= _cols(db)
-
-def test_existing_db_migrates_shape_columns(tmp_path):
-    p = tmp_path / "s.db"
-    db = SeismoDb(p)
-    with sqlite3.connect(p) as c:      # simulate an older DB missing the columns
-        for col in _SHAPE_COLS:
-            c.execute(f"ALTER TABLE events DROP COLUMN {col}")
-    assert not (_SHAPE_COLS <= _cols(SeismoDb(p)))  # sanity: dropped
-    SeismoDb(p)                        # re-open triggers _migrate
-    assert _SHAPE_COLS <= _cols(SeismoDb(p))
-```
-
-> Note: sqlite `DROP COLUMN` needs sqlite ≥ 3.35 (bundled py3.10 has it). If the runner's sqlite lacks it, replace the "simulate older DB" block with building a table without the columns; keep the assertion that re-open adds them.
-
-- [ ] **Step 2: Run to verify it fails**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_columns.py -q`
-Expected: FAIL — columns absent.
-
-- [ ] **Step 3: Implementation**
-
-In `_SCHEMA`, after the `mic_zc_above_range      INTEGER,` line in the `events` CREATE TABLE, add:
-
-```
-    shape_crest_factor      REAL,                   -- peak / rms of the triggering channel
-    shape_near_peak_count   INTEGER,                -- samples >= 0.5 * peak (FT: few; real: many)
-    shape_sample_count      INTEGER,                -- total samples (to normalize near_peak_count)
-    shape_axis              TEXT,                   -- geophone channel measured ("Tran"/"Vert"/"Long")
-```
-
-In `_migrate`, extend the ADD COLUMN tuple (the `for col, ddl in (...)` list) with:
-
-```python
-            ("shape_crest_factor",    "REAL"),
-            ("shape_near_peak_count", "INTEGER"),
-            ("shape_sample_count",    "INTEGER"),
-            ("shape_axis",            "TEXT"),
-```
-
-- [ ] **Step 4: Run to verify it passes**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_columns.py -q`
-Expected: PASS.
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add sfm/database.py tests/test_shape_columns.py
-git commit -m "feat(db): shape_* columns on events (+ auto-migrate)"
-```
-
----
-
-### Task 4: insert_events persists shape from the waveform record
-
-**Files:**
-- Modify: `sfm/database.py` — `insert_events` INSERT (column list + placeholders + values) and the UPSERT `UPDATE` block.
-- Test: `tests/test_insert_events_shape.py`
-
-**Interfaces:**
-- Consumes: a `waveform_records` rec dict that may carry `shape_crest_factor`, `shape_near_peak_count`, `shape_sample_count`, `shape_axis`.
-- Produces: those four values stored on the row; refreshed on UPSERT.
-
-- [ ] **Step 1: Write the failing test**
-
-```python
-# tests/test_insert_events_shape.py
-from sfm.database import SeismoDb
-from tests.helpers_events import make_event  # existing helper used by other insert tests
-
-def test_insert_stores_shape_from_record(tmp_path):
-    db = SeismoDb(tmp_path / "s.db")
-    ev = make_event(serial="BE1", key="0111abcd")
-    rec = {ev._waveform_key.hex(): {
-        "filename": "F.CE0W", "filesize": 10,
-        "shape_crest_factor": 34.0, "shape_near_peak_count": 3,
-        "shape_sample_count": 1024, "shape_axis": "Long"}}
-    db.insert_events([ev], serial="BE1", waveform_records=rec)
-    row = db.query_events(serial="BE1")[0]
-    assert row["shape_crest_factor"] == 34.0
-    assert row["shape_near_peak_count"] == 3
-    assert row["shape_axis"] == "Long"
-```
-
-> If `tests/helpers_events.make_event` doesn't exist, build the `Event` inline the way `tests/test_zc_freq_columns.py` does (copy its event-construction helper). Keep the assertion identical.
-
-- [ ] **Step 2: Run to verify it fails**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_insert_events_shape.py -q`
-Expected: FAIL — `KeyError`/`sqlite3.OperationalError` (columns not in INSERT) or values are NULL.
-
-- [ ] **Step 3: Implementation**
-
-In `insert_events` INSERT: add `shape_crest_factor, shape_near_peak_count, shape_sample_count, shape_axis` to the column list, add four `?` placeholders, and add these to the VALUES tuple (after the `mic_zc_above_range` value):
-
-```python
-                            rec.get("shape_crest_factor"),
-                            rec.get("shape_near_peak_count"),
-                            rec.get("shape_sample_count"),
-                            rec.get("shape_axis"),
-```
-
-In the UPSERT `UPDATE ... SET`: add
-
-```sql
-                               shape_crest_factor    = COALESCE(?, shape_crest_factor),
-                               shape_near_peak_count = COALESCE(?, shape_near_peak_count),
-                               shape_sample_count    = COALESCE(?, shape_sample_count),
-                               shape_axis            = COALESCE(?, shape_axis),
-```
-
-and the matching params (before `serial, ts`):
-
-```python
-                            rec.get("shape_crest_factor") if rec else None,
-                            rec.get("shape_near_peak_count") if rec else None,
-                            rec.get("shape_sample_count") if rec else None,
-                            rec.get("shape_axis") if rec else None,
-```
-
-(`COALESCE` on UPSERT so a re-import that lacks samples doesn't wipe a previously-computed shape.)
-
-- [ ] **Step 4: Run to verify it passes**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_insert_events_shape.py -q`
-Expected: PASS.
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add sfm/database.py tests/test_insert_events_shape.py
-git commit -m "feat(db): insert_events persists shape_* from waveform record"
-```
-
----
-
-### Task 5: Populate shape at ingest (the three save paths)
-
-**Files:**
-- Modify: `sfm/waveform_store.py` — in `save`, `save_imported_bw`, `save_imported_idf`, after the `.h5` is written, add its shape to the returned `rec` dict.
-- Test: `tests/test_save_shape.py`
-
-**Interfaces:**
-- Consumes: `sfm.shape_metrics.shape_from_h5`.
-- Produces: `save*` return dicts carry `shape_crest_factor / shape_near_peak_count / shape_sample_count / shape_axis` (present only when the `.h5` had usable samples).
-
-- [ ] **Step 1: Write the failing test** (drives the BW-import path, which the existing suite already exercises)
-
-```python
-# tests/test_save_shape.py
-from sfm.waveform_store import WaveformStore
-from tests.helpers_bw import sample_bw_bytes, sample_serial  # reuse existing import-test fixtures
-
-def test_save_imported_bw_attaches_shape(tmp_path):
-    store = WaveformStore(tmp_path / "waveforms")
-    ev, rec = store.save_imported_bw(sample_bw_bytes(), serial=sample_serial())
-    # A real BW waveform → shape present with a geo axis.
-    assert rec.get("shape_axis") in ("Tran", "Vert", "Long")
-    assert rec["shape_crest_factor"] > 0
-    assert rec["shape_sample_count"] > 200
-```
-
-> Reuse whatever fixture `tests/test_save_imported_bw*.py` already uses for BW bytes; match its import. If the existing BW fixture produces a degenerate/short waveform, use the fixture from the test that asserts a full h5.
-
-- [ ] **Step 2: Run to verify it fails**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_save_shape.py -q`
-Expected: FAIL — `rec` has no `shape_*` keys.
-
-- [ ] **Step 3: Implementation**
-
-Add a helper near the top of `WaveformStore` methods (module-level import `from sfm.shape_metrics import shape_from_h5`). In each of `save`, `save_imported_bw`, `save_imported_idf`, immediately before building/returning the `rec` dict — and only when the `.h5` was written (i.e. `hdf5_filename`/`hdf5_path` is set) — compute and merge:
-
-```python
-        shape = shape_from_h5(hdf5_path) if hdf5_filename else None
-        # ... in the returned rec dict literal, add:
-        #   **(shape and {
-        #       "shape_crest_factor":    shape["crest_factor"],
-        #       "shape_near_peak_count": shape["near_peak_count"],
-        #       "shape_sample_count":    shape["sample_count"],
-        #       "shape_axis":            shape["axis"],
-        #   } or {}),
-```
-
-Concretely, after each method computes `hdf5_filename`, add before its `return {...}`:
-
-```python
-        _shape = shape_from_h5(hdf5_path) if hdf5_filename else None
-        _shape_rec = {
-            "shape_crest_factor":    _shape["crest_factor"],
-            "shape_near_peak_count": _shape["near_peak_count"],
-            "shape_sample_count":    _shape["sample_count"],
-            "shape_axis":            _shape["axis"],
-        } if _shape else {}
-```
-
-and spread `**_shape_rec` into the returned dict. (`save_imported_bw`/`save_imported_idf` use their own hdf5 path variable names — use whichever local holds the written `.h5` path in each method.)
-
-- [ ] **Step 4: Run to verify it passes**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_save_shape.py -q`
-Expected: PASS.
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add sfm/waveform_store.py tests/test_save_shape.py
-git commit -m "feat(ingest): compute shape from the written .h5 in all save paths"
-```
-
----
-
-### Task 6: Backfill script for existing events
-
-**Files:**
-- Create: `scripts/backfill_event_shape.py` (mirror `scripts/backfill_event_zc_freq.py`, but read the `.h5` for samples instead of the sidecar).
-- Test: `tests/test_backfill_event_shape.py`
-
-**Interfaces:**
-- Produces: `backfill_shape(db, store, *, dry_run=False) -> dict` with counts `{"updated","skipped_no_h5","skipped_no_samples"}`; `main(argv)` CLI mirroring the zc-freq script's args (`--db-path`, `--store-root`, `--dry-run`).
-
-- [ ] **Step 1: Write the failing test**
-
-```python
-# tests/test_backfill_event_shape.py
-import numpy as np, h5py
-from sfm.database import SeismoDb
-from sfm.waveform_store import WaveformStore
-from scripts.backfill_event_shape import backfill_shape
-from tests.helpers_events import make_event   # or inline as in test_zc_freq_columns
-
-def _h5(path, long):
-    with h5py.File(path, "w") as f:
-        g = f.create_group("samples")
-        for k in ("Tran", "Vert"): g.create_dataset(k, data=np.zeros(1024, "float32"))
-        g.create_dataset("Long", data=np.asarray(long, "float32"))
-
-def test_backfill_updates_shape_and_is_idempotent(tmp_path):
-    db = SeismoDb(tmp_path / "s.db")
-    store = WaveformStore(tmp_path / "waveforms")
-    ev = make_event(serial="BE1", key="0111abcd")
-    db.insert_events([ev], serial="BE1",
-                     waveform_records={ev._waveform_key.hex():
-                                       {"filename": "F.CE0W", "filesize": 10}})
-    # place the .h5 where store.paths_for expects it
-    long = np.zeros(1024); long[100] = 0.48
-    _h5(store.hdf5_path_for("BE1", "F.CE0W"), long)
-
-    c1 = backfill_shape(db, store)
-    assert c1["updated"] == 1
-    row = db.query_events(serial="BE1")[0]
-    assert row["shape_axis"] == "Long" and row["shape_near_peak_count"] <= 3
-    c2 = backfill_shape(db, store)                  # idempotent: re-run overwrites same values
-    assert db.query_events(serial="BE1")[0]["shape_crest_factor"] == row["shape_crest_factor"]
-```
-
-> Match `make_event` / column names to whatever `tests/test_zc_freq_columns.py` uses. `store.hdf5_path_for(serial, filename)` is the existing helper that returns the `.h5` path.
-
-- [ ] **Step 2: Run to verify it fails**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_backfill_event_shape.py -q`
-Expected: FAIL — `ModuleNotFoundError: scripts.backfill_event_shape`.
-
-- [ ] **Step 3: Implementation** (mirror the zc-freq script structure)
-
-```python
-#!/usr/bin/env python3
-"""Backfill events.shape_* from each event's .h5 waveform samples. Idempotent."""
-from __future__ import annotations
-import argparse, logging, sys
-from pathlib import Path
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-from sfm.database import SeismoDb
-from sfm.waveform_store import WaveformStore
-from sfm.shape_metrics import shape_from_h5
-
-log = logging.getLogger("backfill_event_shape")
-
-def backfill_shape(db: SeismoDb, store: WaveformStore, *, dry_run: bool = False) -> dict:
-    counts = {"updated": 0, "skipped_no_h5": 0, "skipped_no_samples": 0}
-    for row in db.query_events(limit=1_000_000):
-        serial, filename = row.get("serial"), row.get("blastware_filename")
-        if not serial or not filename:
-            counts["skipped_no_h5"] += 1; continue
-        h5_path = store.hdf5_path_for(serial, filename)
-        if not h5_path.exists():
-            counts["skipped_no_h5"] += 1; continue
-        shape = shape_from_h5(h5_path)
-        if shape is None:
-            counts["skipped_no_samples"] += 1; continue
-        if not dry_run:
-            with db._connect() as conn:
-                conn.execute(
-                    "UPDATE events SET shape_crest_factor=?, shape_near_peak_count=?, "
-                    "shape_sample_count=?, shape_axis=? WHERE id=?",
-                    (shape["crest_factor"], shape["near_peak_count"],
-                     shape["sample_count"], shape["axis"], row["id"]))
-        counts["updated"] += 1
-    log.info("backfill_shape: %s", counts)
-    return counts
-
-def main(argv=None) -> int:
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--db-path", required=True)
-    ap.add_argument("--store-root", required=True)
-    ap.add_argument("--dry-run", action="store_true")
-    a = ap.parse_args(argv)
-    logging.basicConfig(level=logging.INFO)
-    counts = backfill_shape(SeismoDb(a.db_path), WaveformStore(a.store_root), dry_run=a.dry_run)
-    print(counts)
-    return 0
-
-if __name__ == "__main__":
-    raise SystemExit(main())
-```
-
-> `store.hdf5_path_for` and `db._connect` are existing internals used the same way by other scripts. If `hdf5_path_for` isn't public, use `store.paths_for(...)` sibling `.h5` path exactly as `save()` derives `hdf5_path`.
-
-- [ ] **Step 4: Run to verify it passes**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_backfill_event_shape.py -q`
-Expected: PASS.
-
-- [ ] **Step 5: Run the full suite + commit**
-
-```bash
-/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest -q
-git add scripts/backfill_event_shape.py tests/test_backfill_event_shape.py
-git commit -m "feat(scripts): backfill events.shape_* from .h5 samples"
-```
-
----
-
-### Task 7: Confirm /db/events carries shape + version bump
-
-**Files:**
-- Test: `tests/test_db_events_exposes_shape.py`
-- Modify: `pyproject.toml` version; `sfm/server.py` version string; `CHANGELOG.md`.
-
-**Interfaces:**
-- Consumes: the running `/db/events` route (already returns `SELECT *`).
-
-- [ ] **Step 1: Write the failing test** (guards that the feed dict includes the new keys)
-
-```python
-# tests/test_db_events_exposes_shape.py
-from sfm.database import SeismoDb
-def test_query_events_row_includes_shape_keys(tmp_path):
-    db = SeismoDb(tmp_path / "s.db")
-    # query_events returns dict(row); a fresh insert has the keys (values may be None)
-    from tests.helpers_events import make_event
-    db.insert_events([make_event(serial="BE1", key="0111abcd")], serial="BE1")
-    row = db.query_events(serial="BE1")[0]
-    for k in ("shape_crest_factor","shape_near_peak_count","shape_sample_count","shape_axis"):
-        assert k in row
-```
-
-- [ ] **Step 2: Run to verify it fails / passes**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_db_events_exposes_shape.py -q`
-Expected: PASS immediately if Task 3 landed (columns present in `SELECT *`). If it fails, the columns weren't added — fix Task 3. (This task is the guard, not new behavior.)
-
-- [ ] **Step 3: Version bump**
-
-Bump `pyproject.toml` `version` 0.23.0 → 0.24.0; set `sfm/server.py` `version="0.24.0"`; add a `## v0.24.0` CHANGELOG entry ("waveform-shape metrics on events: crest factor + points-near-peak, ingest + backfill").
-
-- [ ] **Step 4: Full suite**
-
-Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest -q`
-Expected: PASS (no regressions).
-
-- [ ] **Step 5: Commit**
-
-```bash
-git add tests/test_db_events_exposes_shape.py pyproject.toml sfm/server.py CHANGELOG.md
-git commit -m "chore(release): v0.24.0 — waveform-shape metrics on events"
-```
-
----
-
-## Self-Review
-
-**Spec coverage:** SFM shape columns (Tasks 3–4) ✓; DSP crest + near-peak (Task 1) ✓; ingest population (Task 5) ✓; backfill (Task 6) ✓; `/db/events` exposure (Task 7) ✓; NULL for histogram/no-sample events (Tasks 1/5/6 return None → NULL) ✓; 94%-coverage / series-4 fallback handled by NULL-then-Terra-View-fallback (Phase B) ✓. Terra-View scoring, 3-state review, twin flagging, export Notes column → **Phase B plan** (separate, depends on this feed). Calibration → Phase C.
-
-**Placeholder scan:** No TBD/TODO; every code step has real code. The two "reuse existing fixture" notes point at concrete existing tests (`test_zc_freq_columns.py`, `test_save_imported_bw*.py`) rather than leaving blanks.
-
-**Type consistency:** `shape_from_h5`/`shape_from_samples`/`channel_shape` return the same dict keys (`crest_factor`, `near_peak_count`, `sample_count`, `axis`) throughout; the DB columns (`shape_crest_factor`, `shape_near_peak_count`, `shape_sample_count`, `shape_axis`) and rec keys match across Tasks 4–6.
-
-## Deferred to Phase B (terra-view, separate plan)
-
-Scoring service combining shape + cheap signals; suspicion column + reason chips; `reviewed_real` mirror + 3-state review; twin-aware flag propagation; Notes-column export + Maximums "(excludes N flagged)". Written once this feed is live so column names/values are real.

docs/superpowers/plans/2026-08-25-b2a-reviewed-real-twin-propagation.md

@@ -1,264 +0,0 @@
-# Phase B2-A (seismo-relay) — reviewed_real + 3-state mirror + twin propagation
-
-> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
-
-**Goal:** Give SFM a persisted, queryable `reviewed_real` flag alongside `false_trigger` (mirrored from the sidecar review block, mutually exclusive), and propagate a review to an event's histogram/waveform twin so flagging one flags both.
-
-**Architecture:** Mirror the existing `false_trigger` mechanism. New `events.reviewed_real` column (auto-migrate). `update_event_review` mirrors BOTH flags from the sidecar review block and enforces mutual exclusivity on the columns. A `find_twins` matcher (same serial + identical peak_vector_sum + timestamp within a window) drives `propagate_review_to_twins`, which the sidecar-PATCH endpoint calls after mirroring the primary. Terra-View reads `reviewed_real` from `/db/events` (SELECT *).
-
-**Tech Stack:** Python 3.10, raw sqlite3, FastAPI, pytest. Runner: `/home/serversdown/seismo-relay/.venv/bin/python3`.
-
-## Global Constraints
-
-- 3 review states are **mutually exclusive**: an event is `false_trigger=1` XOR `reviewed_real=1` XOR neither. Setting one true forces the other's column to 0.
-- The sidecar JSON stays the source of truth for full review state; the `false_trigger`/`reviewed_real` columns are derived indexes (like today). B2-A propagates twins at the **column** level (what the feed/peak/export read); twin sidecars are not rewritten (known limitation — noted).
-- Twin match = **same serial AND identical `peak_vector_sum` (exact equality) AND `timestamp` within ± window (default 300 s), excluding the event itself.** Identical PVS is the safety anchor.
-- Known pre-existing test failures (~16, missing gitignored fixtures under `tests/fixtures/`) are unrelated — confirm zero NEW failures, don't try to fix them.
-- Run tests with `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest`.
-
----
-
-### Task 1: `reviewed_real` column on events
-
-**Files:** Modify `sfm/database.py` (`_SCHEMA` CREATE TABLE `events` + the Migration-1 rebuild `CREATE TABLE` + the `_migrate` ADD COLUMN loop). Test: `tests/test_reviewed_real_column.py`.
-
-**Interfaces:** Produces `events.reviewed_real INTEGER NOT NULL DEFAULT 0`.
-
-- [ ] **Step 1: Failing test**
-```python
-# tests/test_reviewed_real_column.py
-import sqlite3
-from sfm.database import SeismoDb
-def _cols(db):
-    with sqlite3.connect(db.db_path) as c:
-        return {r[1] for r in c.execute("PRAGMA table_info(events)")}
-def test_fresh_db_has_reviewed_real(tmp_path):
-    assert "reviewed_real" in _cols(SeismoDb(tmp_path/"s.db"))
-def test_existing_db_migrates_reviewed_real(tmp_path):
-    p = tmp_path/"s.db"; db = SeismoDb(p)
-    with sqlite3.connect(p) as c:
-        c.execute("ALTER TABLE events DROP COLUMN reviewed_real")
-    assert "reviewed_real" not in _cols(db)      # dropped (read via existing handle/connection)
-    SeismoDb(p)                                   # re-open migrates
-    assert "reviewed_real" in _cols(SeismoDb(p))
-```
-> If sqlite < 3.35 lacks DROP COLUMN, fall back to building a table without the column and asserting re-open adds it (same as the shape-columns test did).
-
-- [ ] **Step 2: Run → FAIL** (`pytest tests/test_reviewed_real_column.py -q`).
-- [ ] **Step 3: Implement**
-  - In `_SCHEMA` `events` CREATE TABLE, after the `false_trigger ... DEFAULT 0,` line add: `    reviewed_real           INTEGER NOT NULL DEFAULT 0,  -- 0=no, 1=operator-confirmed real (mutually exclusive with false_trigger)`
-  - In the `_migrate` ADD COLUMN loop tuple add: `("reviewed_real", "INTEGER NOT NULL DEFAULT 0"),`
-  - **Do NOT** add it to the Migration-1 rebuild `CREATE TABLE events (...)` block — that block uses a positional `INSERT ... SELECT * FROM events_old` and, by convention, contains only the columns that existed when Migration 1 was written; every later column is added by the ADD COLUMN loop only. Adding it there crashes `_migrate` on genuinely legacy (pre-Migration-1) DBs.
-- [ ] **Step 4: Run → PASS.**
-- [ ] **Step 5: Commit** `feat(db): reviewed_real column on events (+ auto-migrate)`
-
----
-
-### Task 2: `update_event_review` mirrors both flags + mutual exclusivity
-
-**Files:** Modify `sfm/database.py` `update_event_review`. Test: `tests/test_update_event_review_reviewed_real.py`.
-
-**Interfaces:** Consumes a `review` dict that may carry `false_trigger` and/or `reviewed_real` (bools). Produces mutually-exclusive column state.
-
-- [ ] **Step 1: Failing test**
-```python
-# tests/test_update_event_review_reviewed_real.py
-from sfm.database import SeismoDb
-from minimateplus.models import Event
-def _ins(db, eid_key="01110000", serial="BE1"):
-    ev = Event(index=0); ev._waveform_key = bytes.fromhex(eid_key)
-    db.insert_events([ev], serial=serial)
-    return db.query_events(serial=serial)[0]["id"]
-def test_confirm_real_sets_and_clears_ft(tmp_path):
-    db = SeismoDb(tmp_path/"s.db"); eid = _ins(db)
-    db.update_event_review(eid, {"false_trigger": True})
-    assert db.get_event(eid)["false_trigger"] == 1
-    db.update_event_review(eid, {"reviewed_real": True})   # confirming real clears FT
-    row = db.get_event(eid)
-    assert row["reviewed_real"] == 1 and row["false_trigger"] == 0
-def test_flag_ft_clears_reviewed_real(tmp_path):
-    db = SeismoDb(tmp_path/"s.db"); eid = _ins(db)
-    db.update_event_review(eid, {"reviewed_real": True})
-    db.update_event_review(eid, {"false_trigger": True})
-    row = db.get_event(eid)
-    assert row["false_trigger"] == 1 and row["reviewed_real"] == 0
-```
-> Build the Event inline like `tests/test_zc_freq_columns.py` if the import differs.
-
-- [ ] **Step 2: Run → FAIL.**
-- [ ] **Step 3: Implement** — replace the body of `update_event_review` so it handles both keys:
-```python
-    if not isinstance(review, dict):
-        return False
-    has_ft = "false_trigger" in review
-    has_real = "reviewed_real" in review
-    if not has_ft and not has_real:
-        with self._connect() as conn:
-            row = conn.execute("SELECT 1 FROM events WHERE id=?", (event_id,)).fetchone()
-        return row is not None
-    sets = {}
-    if has_ft:
-        sets["false_trigger"] = 1 if review.get("false_trigger") else 0
-    if has_real:
-        sets["reviewed_real"] = 1 if review.get("reviewed_real") else 0
-    # mutual exclusivity: a true in one forces the other column to 0
-    if sets.get("false_trigger") == 1:
-        sets["reviewed_real"] = 0
-    if sets.get("reviewed_real") == 1:
-        sets["false_trigger"] = 0
-    assign = ", ".join(f"{k}=?" for k in sets)
-    params = list(sets.values()) + [event_id]
-    with self._connect() as conn:
-        cur = conn.execute(f"UPDATE events SET {assign} WHERE id=?", params)
-    return cur.rowcount > 0
-```
-- [ ] **Step 4: Run → PASS** (+ run `tests/test_*false_trigger*`/existing review tests to confirm no regression).
-- [ ] **Step 5: Commit** `feat(db): update_event_review mirrors reviewed_real + enforces 3-state exclusivity`
-
----
-
-### Task 3: `find_twins` matcher
-
-**Files:** Modify `sfm/database.py` (add `find_twins`). Test: `tests/test_find_twins.py`.
-
-**Interfaces:** Produces `find_twins(event_id, *, window_seconds=300) -> list[dict]` — same serial, identical peak_vector_sum, timestamp within ±window, excluding self.
-
-- [ ] **Step 1: Failing test**
-```python
-# tests/test_find_twins.py
-import datetime
-from sfm.database import SeismoDb
-from minimateplus.models import Event, Timestamp
-def _ins(db, key, serial, pvs, ts):
-    ev = Event(index=0); ev._waveform_key = bytes.fromhex(key)
-    ev.timestamp = ts
-    # peak_vector_sum comes from peak_values; simplest: insert then UPDATE pvs directly
-    db.insert_events([ev], serial=serial)
-    row = [r for r in db.query_events(serial=serial) if r["waveform_key"] == key][0]
-    import sqlite3
-    with sqlite3.connect(db.db_path) as c:
-        c.execute("UPDATE events SET peak_vector_sum=? WHERE id=?", (pvs, row["id"]))
-    return row["id"]
-def test_find_twins_matches_same_serial_pvs_near_time(tmp_path):
-    db = SeismoDb(tmp_path/"s.db")
-    base = Timestamp(raw=b"", flag=0x10, year=2026, unknown_byte=0, month=2, day=25, hour=20, minute=19, second=5)
-    twin = Timestamp(raw=b"", flag=0x10, year=2026, unknown_byte=0, month=2, day=25, hour=20, minute=19, second=45)
-    far  = Timestamp(raw=b"", flag=0x10, year=2026, unknown_byte=0, month=2, day=25, hour=21, minute=0,  second=0)
-    a = _ins(db, "01110001", "BE1", 0.4763, base)
-    b = _ins(db, "01110002", "BE1", 0.4763, twin)   # twin: same serial+pvs, 40s apart
-    c = _ins(db, "01110003", "BE1", 0.4763, far)    # same pvs but >window away
-    d = _ins(db, "01110004", "BE1", 0.9999, twin)   # near time but different pvs
-    ids = {r["id"] for r in db.find_twins(a, window_seconds=300)}
-    assert ids == {b}
-```
-> Adjust the Event/Timestamp construction to match how `tests/test_waveform_store.py::_make_synthetic_event` builds them if fields differ.
-
-- [ ] **Step 2: Run → FAIL.**
-- [ ] **Step 3: Implement**
-```python
-    def find_twins(self, event_id: str, *, window_seconds: int = 300) -> list[dict]:
-        row = self.get_event(event_id)
-        if not row:
-            return []
-        serial = row.get("serial"); pvs = row.get("peak_vector_sum"); ts = row.get("timestamp")
-        if serial is None or pvs is None or not ts:
-            return []
-        try:
-            t = datetime.datetime.fromisoformat(ts.replace(" ", "T"))
-        except ValueError:
-            return []
-        lo = (t - datetime.timedelta(seconds=window_seconds)).isoformat()
-        hi = (t + datetime.timedelta(seconds=window_seconds)).isoformat()
-        with self._connect() as conn:
-            rows = conn.execute(
-                "SELECT * FROM events WHERE serial=? AND id!=? AND peak_vector_sum=? "
-                "AND timestamp BETWEEN ? AND ?",
-                (serial, event_id, pvs, lo, hi),
-            ).fetchall()
-        return [dict(r) for r in rows]
-```
-- [ ] **Step 4: Run → PASS.**
-- [ ] **Step 5: Commit** `feat(db): find_twins (serial + identical PVS + time window)`
-
----
-
-### Task 4: propagate a review to twins + wire into the sidecar-PATCH endpoint
-
-**Files:** Modify `sfm/database.py` (add `propagate_review_to_twins`); `sfm/server.py` (`db_event_sidecar_patch`). Test: `tests/test_twin_propagation.py`.
-
-**Interfaces:** `propagate_review_to_twins(event_id, *, window_seconds=300) -> list[str]` copies the event's `false_trigger`/`reviewed_real` columns onto each twin; returns twin ids.
-
-- [ ] **Step 1: Failing test** (DB-level)
-```python
-# tests/test_twin_propagation.py — reuse the _ins helper pattern from test_find_twins
-def test_propagate_copies_flags_to_twins(tmp_path):
-    db = SeismoDb(tmp_path/"s.db")
-    # (build primary + twin via the _ins helper as in test_find_twins)
-    # flag the primary FT, then propagate:
-    db.update_event_review(primary_id, {"false_trigger": True})
-    moved = db.propagate_review_to_twins(primary_id)
-    assert twin_id in moved
-    assert db.get_event(twin_id)["false_trigger"] == 1
-```
-- [ ] **Step 2: Run → FAIL.**
-- [ ] **Step 3: Implement**
-  - `sfm/database.py`:
-```python
-    def propagate_review_to_twins(self, event_id: str, *, window_seconds: int = 300) -> list[str]:
-        row = self.get_event(event_id)
-        if not row:
-            return []
-        ft = 1 if row.get("false_trigger") else 0
-        real = 1 if row.get("reviewed_real") else 0
-        twins = self.find_twins(event_id, window_seconds=window_seconds)
-        moved = []
-        with self._connect() as conn:
-            for tw in twins:
-                conn.execute("UPDATE events SET false_trigger=?, reviewed_real=? WHERE id=?",
-                             (ft, real, tw["id"]))
-                moved.append(tw["id"])
-        return moved
-```
-  - `sfm/server.py` `db_event_sidecar_patch`: after the existing `_get_db().update_event_review(event_id, new_sidecar.get("review", {}))`, add:
-```python
-        # Propagate the review to the event's histogram/waveform twin(s) so
-        # flagging one flags both (column-level; twins share serial+PVS+near time).
-        try:
-            _get_db().propagate_review_to_twins(event_id)
-        except Exception as exc:
-            log.warning("twin review-propagation failed for %s: %s", event_id, exc)
-```
-  (Guard the `if body.review is not None:` block so propagation only runs when review changed.)
-- [ ] **Step 4: Run → PASS.**
-- [ ] **Step 5: Commit** `feat(review): propagate false_trigger/reviewed_real to twins on sidecar PATCH`
-
----
-
-### Task 5: expose in feed guard + version bump
-
-**Files:** Test `tests/test_reviewed_real_in_feed.py`; `pyproject.toml`, `sfm/server.py` version, `CHANGELOG.md`.
-
-- [ ] **Step 1: Guard test** — a `query_events` row dict includes `reviewed_real` (SELECT * returns it).
-```python
-from sfm.database import SeismoDb
-from minimateplus.models import Event
-def test_query_events_includes_reviewed_real(tmp_path):
-    db = SeismoDb(tmp_path/"s.db")
-    ev = Event(index=0); ev._waveform_key = bytes.fromhex("01110000")
-    db.insert_events([ev], serial="BE1")
-    assert "reviewed_real" in db.query_events(serial="BE1")[0]
-```
-- [ ] **Step 2: Run → PASS** (columns already present from Task 1).
-- [ ] **Step 3: Bump** `pyproject.toml` 0.24.0 → 0.25.0; `sfm/server.py` version="0.25.0"; add `## v0.25.0` CHANGELOG entry ("reviewed_real 3-state review flag + histogram/waveform twin review-propagation").
-- [ ] **Step 4: Full suite** — confirm zero NEW failures beyond the ~16 pre-existing.
-- [ ] **Step 5: Commit** `chore(release): v0.25.0 — reviewed_real + twin review-propagation`
-
----
-
-## Self-Review
-
-**Spec coverage:** reviewed_real column (T1) ✓; mirror + mutual exclusivity (T2) ✓; twin match serial+identical-PVS+window (T3) ✓; twin propagation wired into the review path (T4) ✓; feed exposure + version (T5) ✓. Twin **sidecar** rewrite intentionally deferred (column-level only — documented in Global Constraints); this is the SFM half — the 3-state UI + confirm-real PATCH is the separate **B2-B (terra-view)** plan.
-
-**Placeholder scan:** none — every step has real code (the two "adjust Event construction to match test_waveform_store" notes point at a concrete existing helper).
-
-**Type consistency:** `false_trigger`/`reviewed_real` INTEGER columns used identically across T1/T2/T4; `find_twins`→`propagate_review_to_twins` both key on the same match; server calls the DB methods by the exact names T2–T4 define.

docs/superpowers/plans/2026-09-17-rescue-listener.md

@@ -1,134 +0,0 @@
-# Plan — "Rescue Listener": a first-class tool for the inverted rescue
-
-**Status:** proposal, not started.  Written 2026-09-17 ~01:40 local, straight
-off the BE12599 incident.  Open questions at the bottom need Brian's answer
-before anything is built.
-
-**Background:** `docs/runbooks/wedged_unit_recovery.md`, "Second incident —
-BE12599".  The manual version of this worked; this plan is about making it a
-tool instead of a sequence of remembered steps at 1 AM.
-
----
-
-## The problem, stated plainly
-
-When a unit is wedged in the BE12599 mode — geophone offset above trigger,
-recording back-to-back, ACH dialing constantly, device stuck repeating an AT
-modem-init string and therefore **deaf to S3 over inbound** — the only channel
-that works is the one the *device* opens.
-
-Recovering it currently means:
-
-1. Remember that `bridges/ach_server.py` exists and takes the right flags
-2. Start it by hand on a box the modem can reach, with a public port forwarded
-3. Go into ACEmanager and repoint the modem's Destination
-4. Watch a terminal for a call-in
-5. Read `rescue.json` to find out whether it worked
-6. Go back into ACEmanager and repoint the modem to where it belongs
-7. **Not forget step 6**, because leaving the Destination pointed at a dead
-   listener is worse than never having started
-
-That is six manual steps and one landmine, executed under pressure while a
-unit floods the office server.
-
-## What the tool should be
-
-**A "rescue listener" an operator can start for one unit, which handles
-whatever that unit says when it calls in, and refuses to go away until the
-operator confirms the modem has been pointed back.**
-
-Lifecycle:
-
-1. **Start** — operator names the target unit and starts a rescue listener.
-   The tool reports the exact address/port to enter in ACEmanager, plus the
-   actions it will take.
-2. **Operator repoints the modem** to that address.
-3. **Wait** — listener sits there.  Live status: "waiting for call-in",
-   elapsed, last-seen.
-4. **Act** — on call-in, run the configured rescue actions automatically,
-   in a safe order, each independently guarded.  Report per-action outcome.
-5. **Hold** — the listener **stays up** and keeps reporting, because the
-   modem is still pointed at it.
-6. **Confirm & stop** — the operator explicitly confirms the Destination has
-   been restored (to `0.0.0.0`, or to the office Instantel ACH server).
-   Only then does the listener shut down.
-
-Step 6 is the whole point of making this a tool.  It is the step that is
-easiest to skip and most expensive to skip.
-
-## Default action set
-
-Ordered deliberately — see "order matters" below.
-
-| # | Action | Default | Why |
-|---|---|---|---|
-| 1 | **Stop monitoring** (SUB 0x97) | ✅ on | Halts recording; ends the trigger→record→dial loop at its source.  Already implemented as `--stop-monitoring`. |
-| 2 | **Drain events** to a diagnostics store | ⚙ configurable | The backlog is usually evidence, not garbage — see the BE12599 offset investigation.  Must NOT land in the prod SFM DB. |
-| 3 | **Disable ACH** (SUB 0x2C/0x7E/0x7F) | ❌ off by default | Stops the dialing — **and stops your only channel**.  Opt-in, and ideally gated on step 1 having succeeded. |
-| 4 | **Erase events** | ❌ off by default | Destructive.  Only after a verified drain. |
-
-### Order matters — the lesson from BE12599
-
-Stopping monitoring *removes the call-in trigger*.  ACH fires on "after event
-recorded"; with recording stopped, the unit has no reason to dial again, even
-though the backlog is still sitting in its memory.  So a naive
-"stop + disable + erase, all at once" rescue can silence the unit before
-you've collected anything, leaving you with no channel and a device full of
-evidence.
-
-The tool should either sequence around this or warn loudly about it.  My
-instinct is: **stop monitoring immediately** (it's the bleeding), then drain
-across however many call-ins it takes, and treat disable-ACH/erase as a
-separate, explicit "finish" action once the operator is satisfied.
-
-## Where it should live — open question, with a proposal
-
-The natural tier is **SFM** (device-side, per the three-tier model in
-CLAUDE.md).  But the rescue listener must be reachable *from the cellular
-network*, which is a deployment constraint SFM's usual profile doesn't have.
-
-**Proposal worth considering:** run it at the office, beside the real Instantel
-ACH server, on a **different port** (e.g. 12346 while Instantel holds 12345).
-Then the ACEmanager change is a **port change, not an IP change** — smaller,
-faster, less to get wrong, and trivially reversible.  It also means the office
-public IP (already stable and known) is the destination, rather than whatever
-Brian's dynamic home IP happens to be that week.
-
-The tmi-dev approach used on BE12599 worked, but required a router forward and
-ran into the dynamic-IP problem in the same session.
-
-## Open questions
-
-1. **Where does it run?**  Office beside Instantel ACH (port swap), SFM on the
-   NAS, or ad-hoc on tmi-dev?  Affects everything else.
-2. **What drives it?**  Terra-View admin page (fits "operator UI"), an SFM
-   endpoint pair (`POST /device/rescue_listener/start` + `/stop` + `/status`),
-   or a CLI wrapper?  A long-lived listener doesn't fit the request/response
-   endpoint shape well — probably needs a background task with a status poll.
-3. **How does it identify the unit?**  It can't know the serial until the
-   device calls in and the handshake reads it.  Allowlist by modem IP?  Accept
-   anything and report what showed up?
-4. **Where do drained events go?**  A per-incident diagnostics store
-   (`bridges/captures/<unit>-diag`) seems right — explicitly *not* the prod
-   SFM DB.  Does that store need to be a first-class thing with its own
-   retention, or is a directory fine?
-5. **How is "confirm the modem is repointed" verified?**  Operator attestation
-   (a button), or can we actually probe it?  If the listener stops seeing
-   call-ins that's weak evidence; if inbound to the unit starts working that's
-   stronger.
-6. **Multi-unit?**  One listener per incident, or one listener that handles any
-   unit that dials in?  Probably the former for safety.
-7. **Timeout / abandonment policy.**  If nobody ever confirms, does it run
-   forever?  Alert after N hours?
-
-## What already exists
-
-- `bridges/ach_server.py` — the listener itself, with `--stop-monitoring`,
-  `--disable-ach`, `--rescue` (added on `feat/ach-rescue-on-connect`, commit
-  `9f1050b`), `--clear-after-download`, `--max-events`, `--allow-ip`.
-- Per-session `rescue.json` recording per-action outcomes.
-- Isolated per-output-dir SQLite + waveform store, so a diagnostics capture is
-  already separate from prod by construction.
-
-So the gap is not protocol work — it's lifecycle, operator surface, and the
-confirmation gate.  Most of the risk is in questions 1 and 2.

docs/waveform_codec_re_status.md

@@ -1,22 +1,3 @@
-> ## SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN
->
-> The tag-dispatch model described in this document — `40 NN` segment headers,
-> tagless headers, channel rotation — is **wrong**.  It produced nearly-correct
-> output only because the block table happens to tile the data sections.
->
-> The body is a chain of self-delimiting per-channel records: `off+2` is a
-> uint16 BE length, `next = off + 2 + len`, and the chain ends on a record whose
-> chan_id is `0x06`.  A 3-valued mode enum at `off+8` selects delta / absolute /
-> raw-12-bit semantics.  `40 NN` is an ordinary int16 BE data block.
->
-> See the record-chain section of `docs/instantel_protocol_reference.md` §7.6.1
-> and the implementation in `minimateplus/waveform_codec.py`.
->
-> Result: all four channels equal length in 1388/1388 files (was 156/1388);
-> ASCII sample-count exact 75/75, fully exact 73/75; device PPV 1306/1306.
->
-> This document is retained as the reasoning trail.
-
 # Waveform body codec — FULLY DECODED (2026-05-11)
 
 This is the **clean working note** for the body-codec reverse-engineering
@@ -121,24 +102,12 @@ correct.
 |          |                       | nibble first; signed 0..7 / 8..F = -8..-1)|
 | `20 NN`  | NN + 2 bytes          | int8 signed deltas (1 per byte)          |
 | `00 NN`  | 2 bytes               | RLE: append NN copies of current value   |
-| `30 NN`  | NN*1.5 + 2 in data    | 12-bit signed deltas (see below).        |
+| `30 NN`  | NN*2 in data section, | Unknown content.  Only in loud-from-     |
-|          | section, NN*4 trailer |                                          |
+|          | NN*4 in trailer       | start events.                            |
-| `40 NN`  | 2*NN + 16 bytes       | Segment header (NN = prev-channel deltas)|
+| `40 02`  | 20 bytes (fixed)      | Segment header                           |
 
 NN is always a multiple of 4.
 
-**Wide-NN forms.**  `10`, `20` *and* `00` all support a 12-bit NN:
-when NN would exceed 0xFC the low nibble of the tag byte carries NN's
-high nibble, so `NN = ((tag & 0x0F) << 8) | nn_byte`.  Confirmed for
-`1X`/`2X` in 2026-05-11 and for `0X` (RLE) in 2026-08-25 — e.g.
-`01 0c` = a 268-sample zero-delta run.
-
-**`40 NN` is variable width.**  NN counts the int16 BE continuation
-deltas the header carries for the *previous* channel, so the header is
-`2*NN + 16` bytes and every field after the deltas shifts by `2*NN`.
-`40 01` (18 B) and `40 03` (22 B) both occur alongside the common
-`40 02` (20 B).  Confirmed 2026-08-25.
-
 Implementation: `walk_body()` in `minimateplus/waveform_codec.py`.
 
 ### 7-byte preamble
@@ -238,61 +207,6 @@ TL;DR table above are now locked in by pytest regression tests.
    still bails out partway through.  Lower priority since the other
    7 events walk cleanly.
 
-4. **Variable-prefix segment descriptors** (found 2026-08-25).
-   3 of 75 ground-truth production events still truncate.  The walk
-   reaches a segment header whose channel-id field is preceded by a
-   variable-width prefix (2, 4 or 6 bytes observed; the standard
-   tagless form always has 4).  These also carry an `01 00` marker
-   instead of `02 00`.  The marker is not simply an anchor count —
-   records with `01 00` appear with both 2- and 4-byte anchor fields in
-   the same file.  Examples: `BE12599/N599LPNB.JF0W` @1155,
-   `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
-
-## Segment header: channel id and tagless form — 2026-08-25
-
-The 4-byte field previously read as a "monotonic uint32 LE counter" is
-`[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert
-`0x48`=Long `0x49`=MicL.  Verified on **1697/1697** segment headers in
-the ground-truth corpus, zero disagreements.  `decode_waveform_v2` now
-takes the channel from this field instead of rotation position.
-
-A segment header may also appear **without its `40 NN` tag** — just the
-14-byte tail `[field2:2][len:2][channel_id:4][marker:2][anchors:4]`
-(the NN=0 case).  `is_tagless_segment_header()` detects it from the six
-bytes at `[4:10]`.
-
-## Geo scale: full scale is 32000 counts — 2026-08-25
-
-One decoder unit (16 ADC counts) is exactly 0.005 in/s, so Normal range
-(10.000 in/s) is `10.0 / (0.005/16)` = **32000** ADC counts.  Consumers
-that divided by 32768 read every geophone sample 2.34% low.  Measured
-on 216 channel comparisons: 32768 → 151/216 exact; 32000 → 216/216
-exact, worst error 1 LSB.
-
-**Scope — not waveform-specific.**  The scale is applied where ADC
-counts become physical units, which every event passes through
-regardless of source codec:
-
-| source | median ratio ours/device, 32768 | with 32000 |
-|---|---|---|
-| series-3 waveform (vs ASCII sample table) | 0.9766 | **1.0000** |
-| series-3 histogram (vs ASCII PPV, n=1137) | 0.9766 | **1.0000** |
-| series-4 Thor IDF (vs device peak, n=1468) | 0.960 | **0.983** |
-
-The four block-framing fixes are waveform-only — `histogram_codec` is
-untouched by them.
-
-## Ground-truth corpus (2026-08-25)
-
-Beyond the bundled fixtures, the production waveform store keeps each
-event's original Blastware ASCII export at
-`<store>/<serial>/<filename>_ASCII.TXT`.  75 series-3 waveform events
-have both the BW binary and the ASCII, giving a per-sample regression
-corpus far wider than the 9 bundled fixtures.  Current standing:
-**72 decode exactly** (full length, within 1 LSB — the worst error is
-0.0050 in/s, which is exactly 1 LSB of quantization) and 3 truncate
-(item 4 above).  Zero events have full-length value errors.
-
 ## `30 NN` block format — CRACKED 2026-05-11 late
 
 The `30 NN` block carries `NN` 12-bit signed deltas, packed as `NN/4`

micromate/idf_ascii_report.py

@@ -210,7 +210,8 @@ def parse_idf_report(text: Union[str, bytes]) -> Dict[str, Any]:
         "long_peak_acceleration",
         "tran_peak_displacement", "vert_peak_displacement",
         "long_peak_displacement",
-        "mic_zc_freq",
+        "tran_time_of_peak", "vert_time_of_peak", "long_time_of_peak",
+        "mic_time_of_peak", "mic_zc_freq",
     )
     for key in float_fields:
         v = raw.get(key)
@@ -222,22 +223,6 @@ def parse_idf_report(text: Union[str, bytes]) -> Dict[str, Any]:
         else:
             out.pop(key, None)
 
-    # Time-of-peak: Thor labels these "TimeofPeak" (lowercase "of") so the
-    # normalizer produces "*_timeof_peak".  Map them to the canonical
-    # ``*_time_of_peak`` output keys for downstream consumers.
-    for raw_key, out_key in (
-        ("tran_timeof_peak", "tran_time_of_peak"),
-        ("vert_timeof_peak", "vert_time_of_peak"),
-        ("long_timeof_peak", "long_time_of_peak"),
-        ("mic_timeof_peak",  "mic_time_of_peak"),
-    ):
-        v = raw.get(raw_key)
-        if v is None:
-            continue
-        fv = _parse_float(v)
-        if fv is not None:
-            out[out_key] = fv
-
     # Microphone — Thor reports MicPSPL (dB(L)) which is the closest
     # analogue to BW's mic_ppv.  The raw "99.4 dB(L)" string stays in
     # `out` under the original `mic_pspl` key for display; the parsed

micromate/idf_file.py

@@ -1,723 +1,64 @@
 """
-micromate/idf_file.py — Thor IDF binary codec.
+micromate/idf_file.py — placeholder for the Thor IDF binary codec.
 
-Decodes the Instantel Micromate Series IV ``.IDFW`` (waveform) and
+Thor's ``.IDFH`` (histogram) and ``.IDFW`` (waveform) event files are an
-``.IDFH`` (histogram) binary on-disk format.  Sister module to
+Instantel proprietary binary format that has not yet been reverse-
-``minimateplus/event_file_io.py``.
+engineered.  Today seismo-relay treats them as opaque blobs:
+``WaveformStore.save_imported_idf`` stores the bytes verbatim and reads
+all device-authoritative metadata from the paired ``.IDFW.txt`` /
+``.IDFH.txt`` ASCII sidecar (parsed by ``idf_ascii_report.py``).
 
-Status (2026-05-28):
+When we crack the binary codec — same reverse-engineering playbook we
+used to byte-perfect-parse Series III BW files (see
+``docs/instantel_protocol_reference.md`` and ``minimateplus/event_file_io.py``)
+— this module will grow:
 
-- **Genuine Series IV / Thor binaries** are all signed
+  - ``read_idf_file(path) -> IdfEvent``
-  ``00 12 01 00 00 00 Instantel\\0`` (sig-A in earlier notes).  Two
+        Parse a ``.IDFW``/``.IDFH`` binary and return a fully populated
-  Series III (Blastware) binaries appear in the example corpus
+        ``IdfEvent`` whose waveform-sample arrays come from the binary
-  (``BE9439_*``) — they share the ``.IDFW``/``.IDFH`` extension by
+        (the .txt sidecar's tabular sample block being a best-effort
-  filing convention but carry a BW STRT header (``10 00 01 80 00 00
+        check).  Lets us ingest Thor events even when the operator
-  Instantel STRT...``) and are NOT Thor data.  The reader detects
+        hasn't enabled the .txt exporter — closing the
-  them by signature and raises NotImplementedError pointing callers
+        ``had_report=False`` gap that the thor-watcher forwarder
-  at ``minimateplus.event_file_io.read_blastware_file()``.
+        currently tolerates as a known limitation.
-- **IDFW waveform body** reuses the BW segment-rotated block codec
-  verbatim.  Body always starts at file offset ``0x0f1f``.  Samples
-  decoded via ``minimateplus.waveform_codec.decode_waveform_v2``
-  with 87–99% byte-exact match against ``.IDFW.txt`` sidecar (quiet
-  events).  Loud events hit the BW codec's known walker-stops-early
-  limit.  Residual ~3% drift on per-sample deltas — likely a
-  Thor-specific 12-bit delta refinement that BW's codec doesn't
-  model.  Geo LSB = 0.0003 in/s; mic factor ~2.14e-6 psi/count.
-- **IDFH histogram body**: 12-byte segment header
-  ``[len_be 2B] 0a 00 00 00 [00 NN_counter] 05 3f`` introduces a
-  segment of ``N`` 72-byte interval records (``N = (len - 10) // 72``).
-  Each record holds 4 × 16-byte per-channel min/max/halfp + 8-byte
-  tail.  Geo peaks via ``max(|min|, |max|) / 32768 × 10`` in/s
-  (matches sidecar within ~1.8%), freq via ``512 / halfp`` Hz.
-  **All 859 Thor IDFH files in the corpus decode (181,071 intervals).**
-- Binary metadata directly extracted: serial, timestamp, sample_rate,
-  record_time, calibration_date.  Other fields fall back to the paired
-  ``.IDFW.txt`` / ``.IDFH.txt`` sidecar (consumed by
-  ``WaveformStore.save_imported_idf``).
 
-The full reverse-engineering writeup lives in
+  - ``write_idf_file(path, event)`` (eventually)
-``docs/idf_protocol_reference.md``.
+        Round-trip event reconstruction, used for verifying the codec
+        against captured device files the way ``write_blastware_file``
+        verifies the Series III codec.
+
+  - Helpers for decoding the binary's per-channel sample arrays into
+    physical units, the per-event flash buffer's monitor-log records,
+    etc.
+
+The reverse-engineering path: pair every ``.IDFW`` binary in
+``thor-watcher/example-data/`` with its sibling ``.IDFW.txt``, treating
+the txt's "Waveform Data Channels" block as ground-truth, and align
+the binary's per-channel int16-or-similar arrays against it.  Header
+fields (sample rate, channel count, record time, timestamps) sit before
+the sample block — same approach as the BW codec where ASCII strings
+inside the binary (``Project:``, ``Client:``, etc.) anchored field
+discovery.
 """
 
 from __future__ import annotations
 
-import datetime
-import struct
-from dataclasses import dataclass
 from pathlib import Path
-from typing import Optional, Union
+from typing import Union
 
-# Thor IDFW bodies use the series-3 record-chain decoder.
+from .models import IdfEvent
-#
-# This was previously pinned to the SUPERSEDED tag-dispatch walker
-# (`decode_waveform_legacy`) on the stated grounds that "Thor has no ASCII
-# ground truth in the corpus and its geo scaling is separately suspect".
-# Both premises were false: Thor writes a per-sample CSV export next to every
-# binary (see scratch/verify_thor_against_csv.py), and the scaling is now
-# resolved (see _GEO_LSB_IPS).  Measured against that ground truth on
-# 2026-09-10, the record chain beats the legacy walker outright:
-#
-#     channel truncation   55/153 files -> 3/153
-#     files exact           98/153      -> 150/153
-#     per-sample exact      99.781%     -> 99.854%
-#
-# The legacy walker stops at the first unrecognised tag and returns whatever
-# channels it had, so its failure mode is silent short channels rather than an
-# error.  Do not re-pin it.
-from minimateplus.waveform_codec import _MODES, decode_waveform_v2, is_record
-
-from .models import IdfEvent, IdfPeaks, IdfReport
 
 
-# Genuine Series IV / Thor IDF binary signature: 6 bytes, then ASCII "Instantel".
+def read_idf_file(path: Union[str, Path]) -> "IdfEvent":
-_THOR_PREFIX = b"\x00\x12\x01\x00\x00\x00"
+    """Parse a Thor ``.IDFW``/``.IDFH`` binary into an ``IdfEvent``.
-# Stray Series III (Blastware) binaries that occasionally turn up in Thor
-# corpus directories renamed to the .IDFW/.IDFH convention.  Their header
-# (`10 00 01 80 00 00 Instantel STRT ...`) is byte-for-byte a BW SUB 5A
-# STRT record, not a Thor binary.  Detected so we can refuse-and-route
-# rather than mis-parse.
-_BW_STRAY_PREFIX = b"\x10\x00\x01\x80\x00\x00"
-_INSTANTEL_TAG = b"Instantel"
 
-# Most common body offset for sig-A IDFW files (~50% of prod events;
+    Not yet implemented.  When implemented, this will be the canonical
-# 151/154 in the original tests/fixtures/THORDATA_example corpus).  The
+    entry point for reading Thor binaries — the ASCII sidecar parser
-# body is the segment-rotated block stream consumed by decode_waveform_v2;
+    becomes an optional fast-path metadata supplement rather than the
-# bytes [0:3] are the magic ``00 02 00`` preamble.  Production events
+    sole source of device-authoritative data.
-# routinely use other offsets — see :func:`_find_waveform_body_offset`
-# for the dynamic scan.  This constant survives only as the priority hint.
-_BODY_START_SIG_A = 0x0F1F
-
-# Magic bytes that mark a candidate waveform-body preamble.
-_BODY_MAGIC = b"\x00\x02\x00"
-
-# Where to start looking for body candidates inside the file.  Skip the
-# fixed-header region where the same magic legitimately appears inside
-# channel-test records and the compliance block (offsets 0x015d, 0x091c,
-# 0x0ae2, 0x0d30 in observed events).
-# Lowered from 0x0E00 to 0x0C00 (2026-09-10).  Three-channel events -- mic
-# disabled -- have a shorter fixed header and put their record chain head at
-# 0x0dba, below the old floor.  The head was therefore invisible to the scan,
-# which fell through to the *Vert* segment-0 record and decoded a body shifted
-# one position around the channel rotation.  46 of 139 files in the
-# 9-10-26-csv-req corpus were affected; all 46 became per-sample exact once
-# the head was reachable.  The floor still skips the fixed-header region,
-# where `is_record()` can match channel-test records (0x015d, 0x091c, 0x0ae2).
-_BODY_SCAN_FLOOR = 0x0C00
-
-# Cap on trial decodes per file.  Chain-head detection normally yields one
-# or two candidates; the cap only bounds the worst case on a corrupt file.
-_MAX_BODY_CANDIDATES = 16
-
-# Geophone count → in/s.
-#
-# The old value 0.0003 was read off the smallest non-zero sample in the
-# sidecar corpus, but that sample is Thor's *4-decimal display rounding* of
-# the true LSB, not the LSB itself.  It read every series-4 geophone sample
-# 3.3% low.  The quantisation ladder gives it away: counts 1..6 export as
-# 0.0003, 0.0006, 0.0009, 0.0012, 0.0016, 0.0019 — an LSB of exactly 0.0003
-# would end 0.0015, 0.0018.
-#
-# The value below maximises exact 4-dp agreement over 1,046,016 paired
-# samples (454 channel-events, 2 units) at 99.854%, versus 50.7% for 0.0003.
-# It is a global constant, not a per-unit calibration: all 8 UM units in the
-# production store independently agree to within ±0.07% on their
-# device-reported PPV.  1/LSB = 3222.6 counts per in/s.
-#
-# The value is pinned, not guessed.  Each exported sample constrains the LSB
-# to the window that rounds to the printed 4-dp figure; intersecting 991,415
-# such constraints (clean channel-events only) gives
-#
-#     LSB in [0.000310307933, 0.000310308057]     width 1.2e-10
-#
-# 0.000310308 sits at the centre of that window.  Equivalent full scale is
-# 10.0 in/s / 0.000310308 = 32226.05 counts.
-#
-# Corroboration from the device: an IDFH interval that never recorded keeps
-# its min/max accumulator at its ±full-scale seed, and that seed is
-# (min=+32226, max=-32226) — the same magnitude, independently.  Note the
-# tempting closed form 10.0/32226 is very slightly WRONG: it lands 4.5e-10
-# above the feasible window and loses 78 boundary samples to the literal
-# value while never winning one.  Series-3 uses 32000 counts for the same
-# 10.0 in/s, so the two generations do NOT share a scale.
-#
-# Ground truth + harness: scratch/verify_thor_against_csv.py
-_GEO_LSB_IPS = 0.000310308
-
-# Microphone count → psi, derived from sidecar regression on 50 sample
-# pairs from UM11719_20231219162723.IDFW (mic-heavy event).
-_MIC_LSB_PSI = 2.14e-6
-
-# IDFH histogram constants.
-# Bytes per interval record = 16 per channel + an 8-byte tail, so a
-# 4-channel unit uses 72 and a mic-disabled 3-channel unit uses 56.  It is
-# NOT a constant: derive it per segment from the interval counter (see
-# decode_idfh_body).  This value survives only as the 4-channel default.
-_IDFH_INTERVAL_SIZE = 72        # bytes per per-interval record (4 channels)
-_IDFH_CHANNEL_BLOCK = 16        # bytes per channel inside an interval record
-_IDFH_INTERVAL_TAIL = 8         # bytes after the per-channel blocks
-_IDFH_SEGMENT_HEADER = 10       # bytes: [len_be 2B][0a 00 00 00 4B][00 NN 2B][05 3f 2B]
-_IDFH_SEGMENT_TAIL   = 2        # bytes after the interval data block, before next marker
-_IDFH_HALFP_FREQ_NUM = 512.0    # freq_hz = NUM / halfp; halfp ≤ 5 means ">100 Hz" sentinel
-_IDFH_CHANNELS = ("Tran", "Vert", "Long", "MicL")
-
-
-# ─── Binary metadata extraction ─────────────────────────────────────────────
-
-
-@dataclass
-class IdfBinaryMetadata:
-    """Fields recoverable from the sig-A binary header (no .txt needed)."""
-    serial:           Optional[str] = None
-    event_datetime:   Optional[datetime.datetime] = None
-    sample_rate:      Optional[int] = None
-    record_time_sec:  Optional[float] = None
-    calibration_date: Optional[datetime.date] = None
-
-
-def _read_ascii_z(buf: bytes, off: int, maxlen: int = 64) -> Optional[str]:
-    if off >= len(buf):
-        return None
-    end = buf.find(b"\x00", off, off + maxlen)
-    if end < 0:
-        end = min(off + maxlen, len(buf))
-    s = buf[off:end].decode("ascii", errors="replace").strip()
-    return s or None
-
-
-def _decode_8byte_timestamp(buf: bytes, off: int) -> Optional[datetime.datetime]:
-    """Layout: ``[day][month][year_hi][year_lo][unknown][hour][min][sec]``."""
-    if off + 8 > len(buf):
-        return None
-    day, mon, yh, yl, _unk, hr, mn, sc = buf[off : off + 8]
-    year = (yh << 8) | yl
-    if not (2015 <= year <= 2050 and 1 <= mon <= 12 and 1 <= day <= 31
-            and 0 <= hr < 24 and 0 <= mn < 60 and 0 <= sc < 60):
-        return None
-    try:
-        return datetime.datetime(year, mon, day, hr, mn, sc)
-    except ValueError:
-        return None
-
-
-def extract_binary_metadata(buf: bytes) -> IdfBinaryMetadata:
-    """Pull serial/timestamp/sample_rate/record_time/calibration from the
-    sig-A binary header.
-
-    Field positions confirmed against UM11719_20231219162723.IDFW; stable
-    across the 151-file sig-A corpus.
     """
-    md = IdfBinaryMetadata()
-
-    # Serial: null-terminated ASCII at 0x14E.
-    md.serial = _read_ascii_z(buf, 0x14E, maxlen=16)
-
-    # Sample rate + record time live in a BW-compatible compliance block.
-    # Locate the 6-byte anchor `be 80 00 00 00 00` and read offsets relative
-    # to it: anchor-6 = sample_rate uint16 BE; anchor+6 = record_time float32 BE.
-    anchor = buf.find(b"\xbe\x80\x00\x00\x00\x00", 0x800, 0xA00)
-    if anchor > 0:
-        sr_bytes = buf[anchor - 6 : anchor - 4]
-        if len(sr_bytes) == 2:
-            sr = int.from_bytes(sr_bytes, "big")
-            if sr in (256, 512, 1024, 2048, 4096):
-                md.sample_rate = sr
-        rt_bytes = buf[anchor + 6 : anchor + 10]
-        if len(rt_bytes) == 4:
-            try:
-                rt = struct.unpack(">f", rt_bytes)[0]
-                if 0.1 <= rt <= 600.0:
-                    md.record_time_sec = float(rt)
-            except struct.error:
-                pass
-
-    # Event timestamp: 8 bytes.  Position differs between IDFW (0x97A) and
-    # IDFH (0x9F8); scan a small range and accept the first valid decode.
-    for off in (0x97A, 0x9F8):
-        ts = _decode_8byte_timestamp(buf, off)
-        if ts is not None:
-            md.event_datetime = ts
-            break
-
-    # Calibration date: day, month, year_be at 0x194-0x197.
-    if len(buf) > 0x197:
-        day, mon = buf[0x194], buf[0x195]
-        year = int.from_bytes(buf[0x196 : 0x198], "big")
-        if 1 <= mon <= 12 and 1 <= day <= 31 and 2015 <= year <= 2050:
-            try:
-                md.calibration_date = datetime.date(year, mon, day)
-            except ValueError:
-                pass
-
-    return md
-
-
-# ─── Sample decoder + unit conversion ───────────────────────────────────────
-
-
-def _find_waveform_body_offset(buf: bytes) -> Optional[int]:
-    """Pick the file offset of the waveform body by trial-decoding every
-    ``00 02 00`` magic position past the fixed-header region.
-
-    The body's location isn't fixed across all sig-A IDFW files — about
-    half the production events use ``0x0f1f``, but the rest have offsets
-    that shift based on header padding / channel-config layout.  We
-    auto-detect by:
-
-      1. Find every ``00 02 00`` occurrence past ``_BODY_SCAN_FLOOR``.
-      2. Try ``decode_waveform_v2()`` on each candidate.
-      3. Pick the offset whose decoded sample count is largest.
-
-    Returns the offset, or ``None`` if no candidate yielded more than
-    the trivial 2-sample preamble (= "no real body found").
-
-    Costs ~2-8 trial decodes per file; in practice the first candidate
-    past 0x0e00 is usually the right one.
-    """
-    if len(buf) < _BODY_SCAN_FLOOR + 8:
-        return None
-
-    # 1. Locate every plausible per-channel record header.  A header carries
-    #    [len 2B][channel_id][00][00] at +2..+6, so anchor the search on the
-    #    three-byte ``<cid> 00 00`` signature and validate with is_record().
-    #    Scanning candidate *preambles* instead is not viable: MODE_RAW16 is
-    #    ``00 00``, so every run of three zero bytes would look like a body
-    #    start and each would cost a full trial decode (~0.5 s/file measured).
-    floor = max(0, _BODY_SCAN_FLOOR - 7)
-    starts: list = []
-    for cid in (0x46, 0x47, 0x48, 0x49):
-        sig = bytes((cid, 0x00, 0x00))
-        i = floor
-        while True:
-            j = buf.find(sig, i)
-            if j < 0:
-                break
-            i = j + 1
-            q = j - 4
-            if q >= floor and is_record(buf, q):
-                starts.append(q)
-    if not starts:
-        return None
-    starts.sort()
-
-    # 2. A body begins at the head of a record chain -- a record that no other
-    #    record's length field points at.  The head's own payload is the
-    #    implicit segment-0 Tran record, and the body offset is head + 7 (past
-    #    [len 2B][cid][00][00][seg]) so that body[1:3] lands on the mode.
-    ends = {q + 2 + int.from_bytes(buf[q + 2 : q + 4], "big") for q in starts}
-    heads = [q for q in starts if q not in ends] or starts[:1]
-
-    # 3. Trial-decode each head and keep the best.  Prefer a candidate where
-    #    all four channels come out the same length: scoring on raw sample
-    #    count alone picks false positives sitting *inside* a record header,
-    #    which decode a plausible-looking but rotation-shifted body that
-    #    silently drops each channel's segment 0.
-    best = None
-    best_off = None
-    for head in heads[:_MAX_BODY_CANDIDATES]:
-        j = head + 7
-        if j + 3 > len(buf) or (buf[j + 1], buf[j + 2]) not in _MODES:
-            continue
-        try:
-            decoded = decode_waveform_v2(buf[j:])
-        except Exception:
-            continue
-        if not decoded:
-            continue
-        lengths = [len(v) for v in decoded.values() if v]
-        total = sum(len(v) for v in decoded.values())
-        # A "real" body has more than just the 2-sample preamble.
-        if total <= 2:
-            continue
-        # >= 3 rather than == 4: a mic-disabled event has only the three geo
-        # channels, and demanding four made `equal` permanently False for
-        # them, leaving the pick to raw sample count alone.
-        equal = len(lengths) >= 3 and len(set(lengths)) == 1
-        score = (equal, total)
-        if best is None or score > best:
-            best, best_off = score, j
-    return best_off
-
-
-def _decode_waveform_samples(buf: bytes) -> Optional[dict]:
-    """Decode samples from the sig-A waveform body.
-
-    Returns the raw decoder counts dict — geo LSB = 0.0003 in/s, mic in
-    its own count unit (see :func:`mic_count_to_psi`).  Returns None if
-    no usable body is found.
-
-    Uses :func:`_find_waveform_body_offset` to locate the body — the
-    file-offset varies across events (~50% sit at the canonical
-    ``0x0f1f`` but the rest don't), so the previous hardcoded constant
-    silently produced 2-sample preamble-only output for half the corpus.
-    """
-    off = _find_waveform_body_offset(buf)
-    if off is None:
-        return None
-    return decode_waveform_v2(buf[off:])
-
-
-def geo_count_to_ips(count: int) -> float:
-    """Convert a Thor geo decoder count to in/s.  LSB = 0.0003 in/s."""
-    return count * _GEO_LSB_IPS
-
-
-def mic_count_to_psi(count: int) -> float:
-    """Convert a Thor mic decoder count to psi.  Scale derived from
-    regression over 50 sample pairs in UM11719_20231219162723.IDFW;
-    consistent to ~5%.  Calibration constants from the channel block
-    can refine this once decoded.
-    """
-    return count * _MIC_LSB_PSI
-
-
-# ─── IDFH histogram decoder ─────────────────────────────────────────────────
-
-
-@dataclass
-class IdfhInterval:
-    """One decoded histogram interval (typically one minute of monitoring)."""
-    offset:    int    # file byte offset of the 72-byte record
-    # Per-channel min/max ADC counts (int16 BE), half-period samples, peak count.
-    # Peak = max(|min|, |max|).  freq_hz = 512/halfp (None if halfp ≤ 5 →
-    # ">100 Hz" sentinel; matches sidecar convention).
-    tran_min:    int
-    tran_max:    int
-    tran_halfp:  int
-    vert_min:    int
-    vert_max:    int
-    vert_halfp:  int
-    long_min:    int
-    long_max:    int
-    long_halfp:  int
-    micl_min:    int
-    micl_max:    int
-    micl_halfp:  int
-    # 4 on a normal unit; 3 when the microphone is disabled, in which case the
-    # micl_* fields are absent from the record and read as zero.
-    n_channels:  int = 4
-
-    def has_channel(self, channel: str) -> bool:
-        return channel != "MicL" or self.n_channels >= 4
-
-    def peak_count(self, channel: str) -> int:
-        mn = getattr(self, f"{channel.lower()}_min")
-        mx = getattr(self, f"{channel.lower()}_max")
-        return max(abs(mn), abs(mx))
-
-    def peak_ips(self, channel: str) -> float:
-        """Convert peak count to in/s (geo channels only)."""
-        # Same geo LSB as the waveform path — verified independently against
-        # the IDFH exports: as peak magnitude rises (and 4-dp quantisation
-        # noise falls) the implied LSB converges on 0.0003103, matching
-        # _GEO_LSB_IPS.  The old 10.0/32768 read histogram peaks 1.7% low.
-        return self.peak_count(channel) * _GEO_LSB_IPS
-
-    def freq_hz(self, channel: str) -> Optional[float]:
-        halfp = getattr(self, f"{channel.lower()}_halfp")
-        if halfp <= 5:
-            return None
-        return _IDFH_HALFP_FREQ_NUM / halfp
-
-
-def _is_unwritten_interval(interval: "IdfhInterval") -> bool:
-    """True for an interval slot the device reserved but never wrote.
-
-    Thor seeds each interval's per-channel accumulators at ``min = +full
-    scale`` and ``max = -full scale`` and then narrows them as samples
-    arrive.  A slot that never recorded keeps that seed, so ``min > max`` —
-    impossible for real data.  Such a record decodes to a full-scale
-    10.0 in/s peak on every channel and, being a max-over-intervals, poisons
-    the whole file's PPV.
-
-    Rare but real: exactly 1 of 497,611 corpus intervals, and it inflated
-    that file's Long PPV from 0.0081 to 10.0 in/s.  The inversion is always
-    all-or-nothing across channels (0 partial cases in the corpus), so
-    requiring every channel to be inverted keeps this from ever firing on
-    genuine data.
-    """
-    pairs = [
-        (interval.tran_min, interval.tran_max),
-        (interval.vert_min, interval.vert_max),
-        (interval.long_min, interval.long_max),
-    ]
-    if interval.has_channel("MicL"):
-        pairs.append((interval.micl_min, interval.micl_max))
-    return all(mn > mx for mn, mx in pairs)
-
-
-def _decode_idfh_interval(buf72: bytes, offset: int,
-                          n_channels: int = 4) -> IdfhInterval:
-    """Decode one interval record into per-channel min/max/halfp.
-
-    The record is ``n_channels`` × 16-byte blocks plus an 8-byte tail, so it
-    is 72 bytes on a normal unit and 56 when the microphone is disabled.
-    Missing channels read as zero.
-    """
-    import struct
-    fields = []
-    for i in range(4):
-        if i >= n_channels:
-            fields.extend([0, 0, 0])
-            continue
-        block = buf72[i * 16 : (i + 1) * 16]
-        mn = struct.unpack_from(">h", block, 0)[0]
-        mx = struct.unpack_from(">h", block, 2)[0]
-        # block[4:6] = int16 BE, role unknown (possibly time-of-peak)
-        halfp = struct.unpack_from(">H", block, 6)[0]
-        # block[10:12] and block[14:16] are uint16 BE with unknown semantics
-        # (likely sum / count contributions for the PVS computation).
-        fields.extend([mn, mx, halfp])
-    # Tail 8 bytes (buf72[64:72]) carry PVS-related data; not yet decoded.
-    return IdfhInterval(
-        offset=offset,
-        tran_min=fields[0], tran_max=fields[1], tran_halfp=fields[2],
-        vert_min=fields[3], vert_max=fields[4], vert_halfp=fields[5],
-        long_min=fields[6], long_max=fields[7], long_halfp=fields[8],
-        micl_min=fields[9], micl_max=fields[10], micl_halfp=fields[11],
-        n_channels=n_channels,
-    )
-
-
-def decode_idfh_body(buf: bytes) -> list:
-    """Walk an IDFH file and decode every interval record.
-
-    The body has one or more segments; each segment header is 12 bytes:
-    ``[length_be 2B][0a 00 00 00][counter_be 2B][05 3f]`` where ``length``
-    is bytes from the magic through the end of the interval block
-    (= 10 + 72 × n_intervals).  Segments are separated by a 2-byte tail
-    + next-segment 2-byte prefix (the bytes before the next length field).
-
-    ``counter`` is a **uint16 BE cumulative interval index** — the 0-based
-    index of the LAST interval in this segment.  Segments carry 10
-    intervals each, so it runs 9, 19, 29, ... across the file.
-
-    ⚠ This validator used to require ``buf[j + 4] == 0x00``, i.e. that the
-    counter's high byte was zero.  That silently capped every histogram at
-    **250 intervals**: the moment the cumulative counter passed 255 the high
-    byte went non-zero and every later segment was rejected, so any
-    monitoring run longer than ~4 hours lost its tail — frequently the part
-    holding the event peak, which is why those files' PPV read low.  540 of
-    858 corpus files were affected.  Do not reinstate that check.
-    """
-    intervals: list = []
-    i = 0
-    prev_counter = -1          # so the first segment's n = counter + 1
-    while True:
-        j = buf.find(b"\x0a\x00\x00\x00", i)
-        if j < 0 or j < 2:
-            break
-        # Validate: [length_be][0a 00 00 00][counter_be][05 3f].  The counter
-        # is deliberately NOT constrained — see the note above.
-        if buf[j + 6 : j + 8] != b"\x05\x3f":
-            i = j + 1
-            continue
-        length = int.from_bytes(buf[j - 2 : j], "big")
-        counter = int.from_bytes(buf[j + 4 : j + 6], "big")
-        header_start = j - 2
-        if length < _IDFH_SEGMENT_HEADER or header_start + length > len(buf):
-            # Truncated / bogus length — not a real segment header.
-            i = j + 1
-            continue
-        # The counter is the cumulative index of this segment's LAST interval,
-        # so the interval count is its delta from the previous segment.  That
-        # gives the record stride, which is NOT fixed: 16 bytes per channel
-        # plus an 8-byte tail, so 72 for a 4-channel unit and 56 for a
-        # mic-disabled 3-channel one.  Assuming 72 unconditionally made every
-        # 3-channel histogram read 7 intervals per 10-interval segment,
-        # walking off alignment into garbage that decoded as ~10 in/s peaks.
-        n = counter - prev_counter
-        if n <= 0:
-            i = j + 1
-            continue
-        stride = (length - _IDFH_SEGMENT_HEADER) // n
-        n_channels, remainder = divmod(stride - _IDFH_INTERVAL_TAIL,
-                                       _IDFH_CHANNEL_BLOCK)
-        if remainder or not (1 <= n_channels <= 4):
-            i = j + 1
-            continue
-        interval_start = header_start + _IDFH_SEGMENT_HEADER
-        for k in range(n):
-            off = interval_start + k * stride
-            if off + stride > len(buf):
-                break
-            chunk = buf[off : off + stride]
-            interval = _decode_idfh_interval(chunk, off, n_channels)
-            if _is_unwritten_interval(interval):
-                # Reserved-but-never-recorded slot: the min/max accumulators
-                # still hold their ±full-scale seed.  Counting it would
-                # fabricate a 10.0 in/s peak on every channel.
-                continue
-            intervals.append(interval)
-        prev_counter = counter
-        # Advance past this segment + the 2-byte tail.
-        i = header_start + length + _IDFH_SEGMENT_TAIL
-    return intervals
-
-
-# ─── Top-level reader ───────────────────────────────────────────────────────
-
-
-@dataclass
-class IdfReadResult:
-    """Return type for :func:`read_idf_file`.
-
-    For waveforms (``.IDFW``), ``samples`` holds the per-channel sample
-    arrays in Thor decoder counts.  For histograms (``.IDFH``),
-    ``samples`` is empty and ``intervals`` holds the per-interval
-    record list (peaks, freqs).
-    """
-    event:           IdfEvent
-    samples:         dict   # {"Tran": [...], ...} for IDFW; empty for IDFH
-    binary_metadata: IdfBinaryMetadata
-    signature:       str    # always "thor" for now (sig-A genuine Thor)
-    intervals:       Optional[list] = None  # list[IdfhInterval] for IDFH; None for IDFW
-
-
-def read_idf_file(
-    path: Union[str, Path],
-    *,
-    data: Optional[bytes] = None,
-) -> IdfReadResult:
-    """Parse a Thor ``.IDFW`` binary into an ``IdfEvent`` + decoded samples.
-
-    Currently implements signature-A waveforms only.  Signature-B
-    (old-firmware) and ``.IDFH`` histograms raise NotImplementedError;
-    use the paired ``.IDFW.txt`` / ``.IDFH.txt`` sidecar for those via
-    ``parse_idf_report()``.
-
-    Returns an :class:`IdfReadResult`.  The caller converts int sample
-    counts to physical units via :func:`geo_count_to_ips` /
-    :func:`mic_count_to_psi`.
-
-    ``path`` is used for filename in error messages and ``.IDFH`` vs
-    ``.IDFW`` suffix detection.  When ``data`` is supplied the disk
-    read is skipped — useful for ingest paths that already have the
-    bytes in memory and where the file may not exist on disk yet.
-    """
-    p = Path(path)
-    buf = data if data is not None else p.read_bytes()
-
-    if len(buf) < 16 or buf[6:16] != _INSTANTEL_TAG + b"\x00":
-        raise ValueError(f"{p.name}: not an IDF file (missing Instantel magic)")
-
-    sig_prefix = buf[:6]
-    if sig_prefix == _THOR_PREFIX:
-        signature = "thor"
-    elif sig_prefix == _BW_STRAY_PREFIX:
     raise NotImplementedError(
-            f"{p.name}: file has a Series III (Blastware) STRT header in "
+        "IDF binary codec not yet implemented; the .IDFW/.IDFH binary format "
-            "an IDF-named container — not a Thor binary.  Route through "
+        "is undecoded.  Use parse_idf_report() on the paired .txt sidecar "
-            "minimateplus.event_file_io.read_blastware_file() instead "
+        "for device-authoritative metadata."
-            "(peaks decode; samples & full metadata don't, but it's not "
-            "Thor data so the Thor codec doesn't apply)."
-        )
-    else:
-        raise ValueError(f"{p.name}: unknown IDF signature {sig_prefix.hex()}")
-
-    is_histogram = p.suffix.upper() == ".IDFH"
-    md = extract_binary_metadata(buf)
-
-    if is_histogram:
-        intervals = decode_idfh_body(buf)
-        if not intervals:
-            raise ValueError(f"{p.name}: IDFH body decoded no intervals")
-        # Peaks: max across all intervals on each channel (per-channel max
-        # of stored max-magnitudes; sidecar's PPV row carries the same).
-        peak_tran = max((iv.peak_ips("Tran") for iv in intervals), default=0.0)
-        peak_vert = max((iv.peak_ips("Vert") for iv in intervals), default=0.0)
-        peak_long = max((iv.peak_ips("Long") for iv in intervals), default=0.0)
-        # Mic peak in psi — Thor stores per-interval mic ADC counts in the
-        # binary; convert the max count to psi via the per-count factor.
-        # Skip on a mic-disabled (3-channel) unit: those records carry no mic
-        # block at all, so peak_count("MicL") would report a synthetic zero.
-        mic_peak_count = max(
-            (iv.peak_count("MicL") for iv in intervals if iv.has_channel("MicL")),
-            default=0,
-        )
-        mic_peak_psi = mic_count_to_psi(mic_peak_count) if mic_peak_count else None
-        rep = IdfReport(
-            serial_number=md.serial,
-            event_type="Full Histogram",
-            event_datetime=md.event_datetime,
-            filename=p.name,
-            sample_rate=md.sample_rate,
-            record_time_sec=md.record_time_sec,
-        )
-        peaks = IdfPeaks(
-            transverse_ips=peak_tran,
-            vertical_ips=peak_vert,
-            longitudinal_ips=peak_long,
-            peak_vector_sum_ips=None,
-            mic_pspl_dbl=None,         # IDFH binary doesn't carry the dB(L) value
-            mic_pspl_psi=mic_peak_psi,
-        )
-        event = IdfEvent(
-            serial=md.serial or "UNKNOWN",
-            timestamp=md.event_datetime or datetime.datetime(1970, 1, 1),
-            kind="Histogram",
-            filename=p.name,
-            sample_rate=md.sample_rate,
-            record_time_sec=md.record_time_sec,
-            peaks=peaks,
-            report=rep,
-        )
-        return IdfReadResult(
-            event=event,
-            samples={},
-            binary_metadata=md,
-            signature=signature,
-            intervals=intervals,
-        )
-
-    # Waveform path.
-    decoded = _decode_waveform_samples(buf)
-    if decoded is None:
-        raise ValueError(f"{p.name}: waveform body codec failed")
-
-    rep = IdfReport(
-        serial_number=md.serial,
-        event_type="Full Waveform",
-        event_datetime=md.event_datetime,
-        filename=p.name,
-        sample_rate=md.sample_rate,
-        record_time_sec=md.record_time_sec,
-    )
-
-    def _peak_ips(ch: str) -> float:
-        arr = decoded.get(ch, [])
-        return geo_count_to_ips(max((abs(v) for v in arr), default=0))
-
-    # Mic peak psi from binary: max absolute MicL ADC count × 2.14e-6 psi/count.
-    mic_arr = decoded.get("MicL", [])
-    mic_peak_count = max((abs(v) for v in mic_arr), default=0)
-    mic_peak_psi = mic_count_to_psi(mic_peak_count) if mic_peak_count else None
-
-    peaks = IdfPeaks(
-        transverse_ips=_peak_ips("Tran"),
-        vertical_ips=_peak_ips("Vert"),
-        longitudinal_ips=_peak_ips("Long"),
-        # PVS requires aligned per-sample √(T²+V²+L²); leave None — the
-        # sidecar carries it and the bridge picks it up if present.
-        peak_vector_sum_ips=None,
-        mic_pspl_dbl=None,             # binary IDFW doesn't carry the dB(L) value;
-                                       # sidecar .txt fills it via IdfReport.from_dict
-        mic_pspl_psi=mic_peak_psi,
-    )
-
-    event = IdfEvent(
-        serial=md.serial or "UNKNOWN",
-        timestamp=md.event_datetime or datetime.datetime(1970, 1, 1),
-        kind="Waveform",
-        filename=p.name,
-        sample_rate=md.sample_rate,
-        record_time_sec=md.record_time_sec,
-        peaks=peaks,
-        report=rep,
-    )
-
-    return IdfReadResult(
-        event=event,
-        samples=decoded,
-        binary_metadata=md,
-        signature=signature,
     )

micromate/idf_to_bw_report.py

@@ -1,323 +0,0 @@
-"""
-micromate/idf_to_bw_report.py — adapter that projects a parsed Thor IDF
-report (+ binary metadata + decoded IDFH intervals) into the
-``bw_report``-shaped dict that :mod:`sfm.report_pdf.gather_report_data`
-consumes.
-
-Lets Thor events flow through the existing Series III Event Report PDF
-pipeline without duplicating the renderer.  Thor's report content is
-~95% the same data shape as BW's; the field names differ but the
-underlying metrics map 1:1.
-
-Caveats
-───────
-
-- **Mic units** — Thor records ``MicPSPL`` natively in dB(L).  This
-  adapter sets ``bw_report.mic.pspl_dbl`` directly; the report
-  renderer recomputes the equivalent psi via its dBL→psi formula.
-- **Saturation / above-range flags** — Thor doesn't always mark
-  ``OORANGE`` the way BW does; we set ``zc_freq_above_range`` only
-  when a `>100` sentinel was preserved in the raw text.
-- **Per-interval data** — for IDFH events we build ``interval_times``
-  by stepping ``IntervalSize`` from ``HistogramStartTime``; the binary
-  decoder confirms one record per step (882 / 881 / 881 ... across
-  the corpus).
-- **calibration_by parsing** — Thor's free-form ``Calibration : November
-  22, 2023 by Instantel`` is split on ``" by "`` to extract the
-  calibrator; the date prefix is parsed where possible, otherwise
-  the binary-extracted ``calibration_date`` from
-  :class:`micromate.idf_file.IdfBinaryMetadata` wins.
-"""
-
-from __future__ import annotations
-
-import datetime
-import re
-from typing import Any, Dict, List, Optional
-
-
-# ─── Helpers ────────────────────────────────────────────────────────────────
-
-
-_NUM_RE = re.compile(r"-?\d+(?:\.\d+)?")
-
-
-def _parse_first_number(s: Optional[str]) -> Optional[float]:
-    """Pull the first numeric token from a string like ``"0.1500 in/s"``."""
-    if s is None:
-        return None
-    m = _NUM_RE.search(str(s))
-    if not m:
-        return None
-    try:
-        return float(m.group(0))
-    except ValueError:
-        return None
-
-
-def _parse_interval_size_s(s: Optional[str]) -> Optional[float]:
-    """``"60 sec"`` → 60.0, ``"5 min"`` → 300.0, ``"1 hour"`` → 3600."""
-    if s is None:
-        return None
-    num = _parse_first_number(s)
-    if num is None:
-        return None
-    sl = str(s).lower()
-    if "hour" in sl or "hr" in sl:
-        return num * 3600.0
-    if "min" in sl:
-        return num * 60.0
-    return num   # default to seconds
-
-
-def _parse_calibration(text: Optional[str]) -> tuple[Optional[str], Optional[str]]:
-    """Split ``"November 22, 2023 by Instantel"`` → (ISO date, calibrator).
-
-    Returns ``(None, None)`` if neither half parses.
-    """
-    if not text:
-        return None, None
-    parts = str(text).split(" by ", 1)
-    date_part = parts[0].strip() if parts else None
-    by_part = parts[1].strip() if len(parts) > 1 else None
-    iso_date: Optional[str] = None
-    if date_part:
-        for fmt in ("%B %d, %Y", "%b %d, %Y", "%Y-%m-%d", "%m/%d/%Y"):
-            try:
-                iso_date = datetime.datetime.strptime(date_part, fmt).date().isoformat()
-                break
-            except ValueError:
-                continue
-    return iso_date, by_part
-
-
-def _channel_peaks(idf: Dict[str, Any], ch_lc: str) -> Dict[str, Any]:
-    """Map ``tran_ppv`` / ``tran_zc_freq`` / ... → bw_report.peaks.tran shape."""
-    out: Dict[str, Any] = {}
-    for src, dst in (
-        (f"{ch_lc}_ppv",                 "ppv_ips"),
-        (f"{ch_lc}_zc_freq",             "zc_freq_hz"),
-        (f"{ch_lc}_time_of_peak",        "time_of_peak_s"),
-        (f"{ch_lc}_peak_acceleration",   "peak_accel_g"),
-        (f"{ch_lc}_peak_displacement",   "peak_disp_in"),
-    ):
-        v = idf.get(src)
-        if v is not None:
-            out[dst] = v
-    # ZC freq ">100" sentinel: the raw text carries it under the un-typed
-    # key (e.g. ``raw["tran_zc_freq"]`` would be ``">100"``), and our parser
-    # dropped the typed entry.  Detect that case and flag.
-    raw_zc = idf.get(f"{ch_lc}_zc_freq")
-    if isinstance(raw_zc, str) and ">" in raw_zc:
-        out["zc_freq_above_range"] = True
-        out.pop("zc_freq_hz", None)
-    return out
-
-
-def _sensor_check(idf: Dict[str, Any], ch_lc: str) -> Dict[str, Any]:
-    out: Dict[str, Any] = {}
-    fr = idf.get(f"{ch_lc}_test_freq")
-    if fr is not None:
-        out["freq_hz"] = _parse_first_number(fr)
-    rt = idf.get(f"{ch_lc}_test_ratio")
-    if rt is not None:
-        out["ratio"] = _parse_first_number(rt)
-    am = idf.get(f"{ch_lc}_test_amplitude")
-    if am is not None:
-        out["amplitude_mv"] = _parse_first_number(am)
-    res = idf.get(f"{ch_lc}_test_results")
-    if res is not None:
-        out["result"] = str(res).strip()
-    return {k: v for k, v in out.items() if v is not None}
-
-
-def _interval_times(idf: Dict[str, Any], n_intervals: Optional[int]) -> List[str]:
-    """Synthesise per-interval timestamps from start + interval_size × k.
-
-    Returns ``[]`` when start time or interval size is unknown.
-    """
-    if not n_intervals:
-        return []
-    start_date = idf.get("histogram_start_date") or idf.get("event_date")
-    start_time = idf.get("histogram_start_time") or idf.get("event_time")
-    iv_str = idf.get("interval_size")
-    iv_s = _parse_interval_size_s(iv_str)
-    if not (start_date and start_time and iv_s):
-        return []
-    try:
-        t0 = datetime.datetime.strptime(f"{start_date} {start_time}", "%Y-%m-%d %H:%M:%S")
-    except ValueError:
-        return []
-    out = []
-    for k in range(int(n_intervals)):
-        t = t0 + datetime.timedelta(seconds=iv_s * (k + 1))
-        out.append(t.isoformat())
-    return out
-
-
-# ─── Top-level adapter ──────────────────────────────────────────────────────
-
-
-def build_bw_report_from_idf(
-    idf_report: Dict[str, Any],
-    *,
-    binary_md=None,
-    intervals: Optional[list] = None,
-    is_histogram: Optional[bool] = None,
-) -> Dict[str, Any]:
-    """Project a parsed IDF report dict (and optional binary metadata +
-    decoded IDFH intervals) into the BW report sidecar shape.
-
-    The returned dict is structurally identical to what
-    ``minimateplus.event_file_io._bw_report_to_dict`` produces from a
-    real BW ASCII report — it can be assigned to
-    ``sidecar["bw_report"]`` and consumed verbatim by
-    ``sfm.report_pdf.gather_report_data``.
-
-    ``intervals`` is the list of :class:`micromate.idf_file.IdfhInterval`
-    objects from :func:`micromate.idf_file.decode_idfh_body`; only used
-    for histogram events to derive accurate ``interval_times``.
-    """
-    if is_histogram is None:
-        et = str(idf_report.get("event_type", ""))
-        is_histogram = et.lower().startswith("full histogram")
-
-    # ── Trigger / recording / device ─────────────────────────────────────
-    trigger_channel = idf_report.get("trigger")
-    trigger_level   = _parse_first_number(idf_report.get("geo_trigger_level"))
-    geo_range_ips   = _parse_first_number(idf_report.get("geo_range"))
-
-    cal_iso, cal_by = _parse_calibration(idf_report.get("calibration"))
-    # Prefer the binary-extracted calibration_date when our text parse fell
-    # through; the binary date is unambiguous.
-    if cal_iso is None and binary_md is not None and binary_md.calibration_date:
-        cal_iso = binary_md.calibration_date.isoformat()
-
-    # ── Histogram fields ────────────────────────────────────────────────
-    hist_block: Dict[str, Any] = {
-        "start": None, "stop": None, "n_intervals": None,
-        "interval_size": None, "interval_size_s": None,
-        "channel_peak_when": {},
-    }
-    if is_histogram:
-        sd = idf_report.get("histogram_start_date")
-        st = idf_report.get("histogram_start_time")
-        if sd and st:
-            try:
-                hist_block["start"] = datetime.datetime.strptime(
-                    f"{sd} {st}", "%Y-%m-%d %H:%M:%S"
-                ).isoformat()
-            except ValueError:
-                pass
-        ed = idf_report.get("histogram_stop_date")
-        et_ = idf_report.get("histogram_stop_time")
-        if ed and et_:
-            try:
-                hist_block["stop"] = datetime.datetime.strptime(
-                    f"{ed} {et_}", "%Y-%m-%d %H:%M:%S"
-                ).isoformat()
-            except ValueError:
-                pass
-        n_raw = idf_report.get("number_of_intervals")
-        if n_raw is not None:
-            try:
-                # Thor reports a float like "81.04"; round to int (the BW
-                # report uses an int for the column).
-                hist_block["n_intervals"] = int(float(str(n_raw)))
-            except ValueError:
-                pass
-        # When the binary decoder gave us the actual interval count, prefer it.
-        if intervals is not None:
-            hist_block["n_intervals"] = len(intervals)
-        hist_block["interval_size"] = idf_report.get("interval_size")
-        hist_block["interval_size_s"] = _parse_interval_size_s(idf_report.get("interval_size"))
-        # interval_times derived from start+step (the BW report uses the
-        # exact strings; we match its representation).
-        times = _interval_times(idf_report, hist_block["n_intervals"])
-        # Per-channel peak when (absolute date+time at which the channel's
-        # peak occurred over the histogram run).  Thor splits this into
-        # ``TranPeakDate`` / ``TranPeakTime`` etc.
-        peak_when: Dict[str, str] = {}
-        for ch_label, ch_lc in (("Tran", "tran"), ("Vert", "vert"), ("Long", "long"), ("MicL", "mic")):
-            d = idf_report.get(f"{ch_lc}_peak_date")
-            t = idf_report.get(f"{ch_lc}_peak_time")
-            if d and t:
-                try:
-                    peak_when[ch_label] = datetime.datetime.strptime(
-                        f"{d} {t}", "%Y-%m-%d %H:%M:%S"
-                    ).isoformat()
-                except ValueError:
-                    continue
-        if peak_when:
-            hist_block["channel_peak_when"] = peak_when
-
-    # ── Mic block ────────────────────────────────────────────────────────
-    mic_block = {
-        "weighting":           "L",                   # Thor mic is ISEE Linear
-        "pspl_dbl":            idf_report.get("mic_ppv"),  # the dB(L) float
-        "pspl_saturated":      False,
-        "zc_freq_hz":          idf_report.get("mic_zc_freq"),
-        "zc_freq_above_range": isinstance(idf_report.get("mic_zc_freq"), str)
-                               and ">" in str(idf_report.get("mic_zc_freq")),
-        "time_of_peak_s":      idf_report.get("mic_time_of_peak"),
-    }
-    if mic_block["zc_freq_above_range"]:
-        mic_block["zc_freq_hz"] = None
-
-    # ── Peaks ────────────────────────────────────────────────────────────
-    vs_block = {
-        "ips":       idf_report.get("peak_vector_sum"),
-        "time_s":    _parse_first_number(idf_report.get("peak_vector_sum_time_sum")),
-        "when":      None,
-        "saturated": False,
-    }
-    if is_histogram:
-        # PVS absolute date+time, when present.
-        vs_d = idf_report.get("peak_vector_sum_date")
-        vs_t = idf_report.get("peak_vector_sum_time")
-        if vs_d and vs_t:
-            try:
-                vs_block["when"] = datetime.datetime.strptime(
-                    f"{vs_d} {vs_t}", "%Y-%m-%d %H:%M:%S"
-                ).isoformat()
-            except ValueError:
-                pass
-
-    return {
-        "available":  True,
-        "event_type": idf_report.get("event_type"),
-        "version":    idf_report.get("version"),
-        "trigger": {
-            "channel":       trigger_channel,
-            "geo_level_ips": trigger_level,
-        },
-        "recording": {
-            "sample_rate_sps":  idf_report.get("sample_rate"),
-            "record_time_s":    idf_report.get("record_time_sec"),
-            "pretrig_s":        idf_report.get("pre_trigger_sec"),
-            "stop_mode":        idf_report.get("record_stop_mode"),
-            "geo_range_ips":    geo_range_ips,
-            "units":            idf_report.get("units"),
-        },
-        "device": {
-            "battery_volts":    idf_report.get("battery_volts"),
-            "calibration_date": cal_iso,
-            "calibration_by":   cal_by,
-        },
-        "peaks": {
-            "tran":       _channel_peaks(idf_report, "tran"),
-            "vert":       _channel_peaks(idf_report, "vert"),
-            "long":       _channel_peaks(idf_report, "long"),
-            "vector_sum": vs_block,
-        },
-        "mic":          mic_block,
-        "sensor_check": {
-            "tran": _sensor_check(idf_report, "tran"),
-            "vert": _sensor_check(idf_report, "vert"),
-            "long": _sensor_check(idf_report, "long"),
-            "mic":  _sensor_check(idf_report, "mic"),
-        },
-        "histogram":    hist_block,
-        "monitor_log":  [],
-        "pc_sw_version": None,
-    }

micromate/models.py

@@ -159,23 +159,12 @@ class IdfReport:
 
 @dataclass
 class IdfPeaks:
-    """Geophone + mic peak values for one Thor event.  Native Thor units.
+    """Geophone + mic peak values for one Thor event.  Native Thor units."""
-
-    Thor stores the mic peak in two parallel forms — ``mic_pspl_dbl`` is
-    what the sidecar's top-level ``MicPSPL`` header field carries (dB(L)),
-    used in the report header.  ``mic_pspl_psi`` is the psi value derived
-    either from the IDFW sample table / IDFH interval column 9, or from
-    the binary mic counts (~2.14e-6 psi/count).  Needed because the
-    BW-shaped ``PeakValues.micl`` consumed by ``event_hdf5.write_event_hdf5``
-    expects psi — feeding it dB(L) makes the h5 mic-chart scale factor
-    blow up.
-    """
     transverse_ips:    Optional[float] = None    # in/s
     vertical_ips:      Optional[float] = None    # in/s
     longitudinal_ips:  Optional[float] = None    # in/s
     peak_vector_sum_ips: Optional[float] = None  # in/s
     mic_pspl_dbl:      Optional[float] = None    # dB(L)
-    mic_pspl_psi:      Optional[float] = None    # psi
 
 
 @dataclass
@@ -335,14 +324,10 @@ class IdfEvent:
         machinery without those code paths needing to know about Thor.
 
         Caveats of the bridge:
-          - ``PeakValues.micl`` carries the mic peak in **psi** (matching
+          - ``mic_ppv`` on the produced Event carries Thor's dB(L) value
-            BW's convention) — set from :attr:`IdfPeaks.mic_pspl_psi`,
+            verbatim — the UI distinguishes via the ``device_family``
-            with a dB(L)→psi fallback when only the dB(L) value is
+            column (Phase 1).  Don't run the BW psi→dBL converter on
-            available.  This is what the h5 writer's mic-scale-factor
+            Series IV rows.
-            logic needs.  The dB(L) value still flows through
-            ``bw_report.mic.pspl_dbl`` (set by the
-            ``idf_to_bw_report`` adapter) and the renderer reads it
-            from there for the report header.
           - Many Thor-specific fields (Peak Acceleration / Displacement,
             sensor self-check, calibration) don't have a slot in
             ``Event``.  The full IdfReport is preserved on the
@@ -364,17 +349,11 @@ class IdfEvent:
             minute=self.timestamp.minute,
             second=self.timestamp.second,
         )
-        # Resolve mic peak as psi.  Priority: binary-derived mic_pspl_psi
-        # (set by read_idf_file) > dB(L)→psi fallback via standard formula
-        # (psi = 2.9e-9 × 10^(dBL/20)) > None.
-        mic_psi = self.peaks.mic_pspl_psi
-        if mic_psi is None and self.peaks.mic_pspl_dbl is not None:
-            mic_psi = 2.9e-9 * (10.0 ** (self.peaks.mic_pspl_dbl / 20.0))
         pv = PeakValues(
             tran=self.peaks.transverse_ips,
             vert=self.peaks.vertical_ips,
             long=self.peaks.longitudinal_ips,
-            micl=mic_psi,   # psi, matching BW's convention (h5 scaling depends on this)
+            micl=self.peaks.mic_pspl_dbl,   # dB(L) — see caveat above
             peak_vector_sum=self.peaks.peak_vector_sum_ips,
         )
         pi = ProjectInfo(

micromate/sensor_check.py

@@ -1,89 +0,0 @@
-r"""Decode the Thor / Micromate (series-4) sensor self-check waveforms from an
-IDFW event binary.
-
-Reverse-engineered 2026-09-15 against 4 UM (Thor) oracle events.  The IDFW
-binary carries the sensor self-check in its fixed-header region (before the
-waveform body), as up to four records tagged ``01 0e 3c/3d/3e/3f`` — the SAME
-channel ids as the series-3 MiniMate Plus (Tran / Vert / Long / MicL), which is
-the physical self-test:
-
-  * 3c / 3d / 3e = Tran / Vert / Long geophone ring-downs (a damped impulse
-    response — resonant frequency + damping).
-  * 3f          = MicL pulse train (the mic's known-signal gain check).  Absent
-    on three-channel (mic-disabled) units.
-
-Record framing (per record)::
-
-    01 0e [id:1] [flags:3] [count:2 BE] [pad:10] [int16-BE samples × count]
-    \___ 18-byte header ___/
-
-Unlike series-3's delta-coded trailing block, series-4 stores each trace as a
-raw int16 big-endian array.  ``count`` (the 2-byte field at header offset +8)
-is the sample count; the record is padded to a fixed stride after that.
-"""
-from __future__ import annotations
-
-import struct
-from typing import Dict, List
-
-# Record id → channel.  Same ids/order as series-3 (minimateplus.sensor_check).
-_ID_TO_CHANNEL = {0x3C: "Tran", 0x3D: "Vert", 0x3E: "Long", 0x3F: "MicL"}
-_CHAIN_IDS = (0x3C, 0x3D, 0x3E, 0x3F)
-
-_MARKER = b"\x01\x0e"       # precedes the 1-byte channel id
-_HEADER_LEN = 18            # bytes from the marker start to the first sample
-_COUNT_OFF = 8              # 2-byte BE sample count, from the marker start
-_MAX_COUNT = 4000           # sanity cap (traces are ~70-200 samples)
-
-
-def _find_chain(raw: bytes):
-    """Locate the sensor-check record chain.  Returns a list of
-    ``(offset, id, count)`` for the first run of markers whose ids run
-    3c, 3d, 3e[, 3f] in order, or ``[]``.
-
-    Records are padded to a fixed stride, so the next marker is not at
-    ``header + count*2``; instead collect every ``01 0e [id]`` marker with a
-    sane count and take the first id-ordered run.  Validating the id sequence
-    (not a lone ``01 0e 3c``) keeps a stray marker in the waveform body from
-    matching — the real chain sits in the fixed header, ahead of the body.
-    """
-    n = len(raw)
-    markers = []
-    for p in range(n - _HEADER_LEN):
-        if raw[p:p + 2] == _MARKER and raw[p + 2] in _ID_TO_CHANNEL:
-            count = int.from_bytes(raw[p + _COUNT_OFF:p + _COUNT_OFF + 2], "big")
-            if 0 < count <= _MAX_COUNT:
-                markers.append((p, raw[p + 2], count))
-
-    for i, (off, rid, _c) in enumerate(markers):
-        if rid != 0x3C:
-            continue
-        run = [markers[i]]
-        for m in markers[i + 1:]:
-            if len(run) < len(_CHAIN_IDS) and m[1] == _CHAIN_IDS[len(run)]:
-                run.append(m)
-            else:
-                break
-        if len(run) >= 3:       # 3-channel (mic-disabled) units are valid
-            return run
-    return []
-
-
-def decode_idf_sensor_check(raw: bytes) -> Dict[str, List[int]]:
-    """Decode the sensor self-check traces from a Thor/Micromate IDFW binary.
-
-    Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}`` in
-    raw int16 ADC counts (MicL omitted on 3-channel units), or ``{}`` if the
-    binary carries no sensor-check chain (a non-IDF file, or an IDFH histogram).
-    """
-    chain = _find_chain(raw)
-    if not chain:
-        return {}
-    out: Dict[str, List[int]] = {}
-    for off, rid, count in chain:
-        start = off + _HEADER_LEN
-        blob = raw[start:start + count * 2]
-        if len(blob) < count * 2:
-            continue
-        out[_ID_TO_CHANNEL[rid]] = list(struct.unpack(">%dh" % count, blob))
-    return out

minimateplus/binary_annotate.py

@@ -1,75 +0,0 @@
-"""Structural annotation of a Series-3 Blastware waveform binary.
-
-Pure, no I/O: takes the raw file bytes and returns a flat, gap-free tiling of
-labelled :class:`Span` regions for a hex viewer to paint.  Every byte is
-covered — anything the decoder can't account for becomes an ``unknown`` span,
-so undecoded regions (e.g. a stored spectral/FFT block, if one exists) stand
-out instead of hiding.
-
-File layout (see ``blastware_file.py``):  ``[header][21B STRT][body][26B footer]``.
-The body is the record chain walked by :func:`waveform_codec.walk_records`.
-"""
-from __future__ import annotations
-
-from dataclasses import dataclass
-from typing import List
-
-from .waveform_codec import walk_records
-
-_STRT_LEN = 21
-_FOOTER_LEN = 26
-
-
-@dataclass
-class Span:
-    start: int          # inclusive byte offset
-    end: int            # exclusive byte offset
-    label: str          # human-readable description
-    kind: str           # 'header' | 'strt' | 'sample' | 'footer' | 'unknown'
-
-
-def _tile(known: List[Span], total: int) -> List[Span]:
-    """Sort *known* spans and fill every gap with an ``unknown`` span, so the
-    result is a contiguous, non-overlapping tiling of ``[0, total)``.  Overlaps
-    are resolved by clamping to the running position (first writer wins)."""
-    out: List[Span] = []
-    pos = 0
-    for s in sorted(known, key=lambda x: (x.start, x.end)):
-        if s.end <= pos:
-            continue                       # fully behind — dropped overlap
-        start = max(s.start, pos)
-        if start > pos:
-            out.append(Span(pos, start, "unknown", "unknown"))
-        out.append(s if start == s.start else Span(start, s.end, s.label, s.kind))
-        pos = s.end
-    if pos < total:
-        out.append(Span(pos, total, "unknown", "unknown"))
-    return out
-
-
-def annotate_blastware_binary(raw: bytes) -> List[Span]:
-    """Annotate a Series-3 waveform binary into a gap-free list of spans."""
-    total = len(raw)
-    strt_pos = raw.find(b"STRT")
-    if strt_pos < 0:
-        return [Span(0, total, "unrecognized — no STRT record", "unknown")]
-
-    known: List[Span] = []
-    if strt_pos > 0:
-        known.append(Span(0, strt_pos, "File header", "header"))
-    known.append(Span(strt_pos, strt_pos + _STRT_LEN, "STRT record", "strt"))
-
-    body_start = strt_pos + _STRT_LEN
-    footer_start = total - _FOOTER_LEN
-    if footer_start >= body_start:
-        known.append(Span(footer_start, total, "File footer", "footer"))
-    else:
-        footer_start = total          # file too short for a footer
-
-    body = raw[body_start:footer_start]
-    for rec in walk_records(body):
-        hi, lo = rec["mode"]
-        label = f"{rec['channel']} record (seg {rec['segment_index']}, mode {hi:02x} {lo:02x})"
-        known.append(Span(body_start + rec["offset"], body_start + rec["end"], label, "sample"))
-
-    return _tile(known, total)

minimateplus/client.py

@@ -30,7 +30,6 @@ from __future__ import annotations
 
 import datetime
 import logging
-import re
 import struct
 from typing import Optional
 
@@ -2533,17 +2532,10 @@ def _decode_0a_partial_header(raw_data: bytes, index: int, key4: bytes) -> Optio
         ts2 = try_ts(raw_data[ts1_end + 1:ts1_end + 1 + ts_size])
 
     # Extract serial and geo threshold from "BE11529\0" and "Geo: X.XXX in/s\0".
-    #
-    # Match any two-letter family prefix, not a literal "BE" — a BlastMate
-    # reports "BA10895", and the old `find(b"BE")` returned -1 on one.  That
-    # skipped this whole block, so the geo threshold went missing along with
-    # the serial.  Requiring the NUL terminator in the pattern also makes the
-    # match stricter than the bare two-byte search it replaces.
     serial: Optional[str] = None
     geo_ips: Optional[float] = None
 
-    serial_match = re.search(rb"[A-Z]{2}\d{3,6}(?=\x00)", raw_data)
+    serial_pos = raw_data.find(b"BE")
-    serial_pos = serial_match.start() if serial_match else -1
     if serial_pos >= 0:
         # Read null-terminated serial starting at serial_pos.
         null_pos = raw_data.find(b"\x00", serial_pos)

minimateplus/event_file_io.py

@@ -27,7 +27,6 @@ from typing import Optional, Union
 from .models import Event, PeakValues, ProjectInfo, Timestamp
 from . import blastware_file as _bw  # avoid circular reference at module load
 from .bw_ascii_report import BwAsciiReport
-from . import waveform_codec as _wc
 from .waveform_codec import decode_waveform_v2, decoded_to_adc_counts
 from .histogram_codec import decode_histogram_body
 
@@ -50,7 +49,7 @@ SIDECAR_KIND   = "sfm.event"
 # bumped without a `pip install` re-run — leading to confusing stale
 # version stamps in sidecars.  Bump this constant and CHANGELOG.md
 # together at release time.
-TOOL_VERSION = "0.31.0"   # +/sensor_check group (schema v2); gates the backfill regen
+TOOL_VERSION = "0.20.0"
 
 try:
     # Best-effort: prefer the installed metadata when it's NEWER than the
@@ -271,19 +270,6 @@ def apply_report_to_event(event: Event, report: BwAsciiReport) -> None:
     if report.mic.pspl_dbl is not None and report.mic.pspl_dbl > 0:
         pv.micl = _dbl_to_psi(report.mic.pspl_dbl)
 
-    if (t := ch.get("Tran")):
-        pv.tran_zc_freq = t.zc_freq_hz
-        pv.tran_zc_above_range = bool(getattr(t, "zc_freq_above_range", False))
-    if (v := ch.get("Vert")):
-        pv.vert_zc_freq = v.zc_freq_hz
-        pv.vert_zc_above_range = bool(getattr(v, "zc_freq_above_range", False))
-    if (l := ch.get("Long")):
-        pv.long_zc_freq = l.zc_freq_hz
-        pv.long_zc_above_range = bool(getattr(l, "zc_freq_above_range", False))
-    if report.mic is not None:
-        pv.mic_zc_freq = report.mic.zc_freq_hz
-        pv.mic_zc_above_range = bool(getattr(report.mic, "zc_freq_above_range", False))
-
     if event.project_info is None:
         event.project_info = ProjectInfo()
     pi = event.project_info
@@ -296,16 +282,6 @@ def apply_report_to_event(event: Event, report: BwAsciiReport) -> None:
         event.sample_rate = report.sample_rate_sps
     if report.record_time_s is not None:
         event.rectime_seconds = report.record_time_s
-    # The report's event_datetime is Blastware's exact trigger time (parsed
-    # from Event Time + Event Date).  Prefer it over the binary footer's stop
-    # time so a report-paired import matches BW to the second.
-    edt = report.event_datetime
-    if edt is not None:
-        event.timestamp = Timestamp(
-            raw=b"", flag=0x10,
-            year=edt.year, unknown_byte=0, month=edt.month, day=edt.day,
-            hour=edt.hour, minute=edt.minute, second=edt.second,
-        )
 
 
 def apply_bw_report_dict_to_event(event: Event, bw_report: dict) -> None:
@@ -353,17 +329,6 @@ def apply_bw_report_dict_to_event(event: Event, bw_report: dict) -> None:
     if pspl is not None and pspl > 0:
         pv.micl = _dbl_to_psi(pspl)
 
-    for axis, freq_attr, above_attr in (
-        ("tran", "tran_zc_freq", "tran_zc_above_range"),
-        ("vert", "vert_zc_freq", "vert_zc_above_range"),
-        ("long", "long_zc_freq", "long_zc_above_range"),
-    ):
-        chd = peaks.get(axis) or {}
-        setattr(pv, freq_attr, chd.get("zc_freq_hz"))
-        setattr(pv, above_attr, bool(chd.get("zc_freq_above_range", False)))
-    pv.mic_zc_freq = mic.get("zc_freq_hz")
-    pv.mic_zc_above_range = bool(mic.get("zc_freq_above_range", False))
-
     rec = bw_report.get("recording") or {}
     sr = rec.get("sample_rate_sps")
     if sr:
@@ -670,11 +635,6 @@ def file_sha256(path: Union[str, Path], chunk_size: int = 65536) -> str:
 _GEO_NORMAL_FS_INS  = 10.0
 _GEO_SENSITIVE_FS_INS = 1.250
 _INT16_FS = 32768.0
-# Geophone full-scale count — 32000, not 32768.  One decoder unit (16 ADC
-# counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts.  Must match
-# sfm.event_hdf5._GEO_INT16_FS or sidecar peaks disagree with the plotted
-# waveform by 2.3%.  Confirmed 2026-08-25 against the BW ASCII corpus.
-_GEO_INT16_FS = 32000.0
 
 # Microphone scale factor, psi per ADC count.  Approximate — exact factor
 # depends on the geophone-vs-mic ADC scaling and the firmware reference.
@@ -744,7 +704,7 @@ def _peaks_from_samples(samples: dict[str, list[int]]) -> PeakValues:
         if not ch:
             return 0.0
         m = max(abs(int(v)) for v in ch)
-        return m / _GEO_INT16_FS * _GEO_NORMAL_FS_INS
+        return m / _INT16_FS * _GEO_NORMAL_FS_INS
 
     tran = _peak_ins(samples.get("Tran", []))
     vert = _peak_ins(samples.get("Vert", []))
@@ -758,7 +718,7 @@ def _peaks_from_samples(samples: dict[str, list[int]]) -> PeakValues:
     pvs = 0.0
     n = min(len(samples.get("Tran", [])), len(samples.get("Vert", [])), len(samples.get("Long", [])))
     if n:
-        scale = _GEO_NORMAL_FS_INS / _GEO_INT16_FS
+        scale = _GEO_NORMAL_FS_INS / _INT16_FS
         T = samples["Tran"]; V = samples["Vert"]; L = samples["Long"]
         for i in range(n):
             t = T[i] * scale
@@ -818,30 +778,6 @@ def derive_record_type_from_filename(filename, default: str = "Waveform") -> str
     return _RECORD_TYPE_BY_EXT_SUFFIX.get(ext[-1].upper(), default)
 
 
-# Marker for the recording-setup config block, and the offset of the record-time
-# float32 within it.  The configured post-trigger record time (seconds) is a
-# big-endian float32 exactly 30 bytes before the "Standard Recording Setup"
-# label.  Verified across the corpus reading 1.0 / 2.0 / 3.0 s on different
-# setups — and ts2 - record_time reproduces Blastware's trigger to the second
-# (N844LQHB: stop 10:33:32 - 3.0 = 10:33:29).
-_RECSETUP_MARKER = b"Standard Recording Setup"
-_RECTIME_OFFSET_BEFORE_MARKER = 30
-
-
-def _parse_record_time_seconds(raw: bytes) -> Optional[float]:
-    """The configured post-trigger record time in seconds, from the recording-
-    setup config block, or None when absent / implausible."""
-    a = raw.find(_RECSETUP_MARKER)
-    if a < _RECTIME_OFFSET_BEFORE_MARKER:
-        return None
-    off = a - _RECTIME_OFFSET_BEFORE_MARKER
-    try:
-        rt = struct.unpack(">f", raw[off:off + 4])[0]
-    except struct.error:
-        return None
-    return rt if 0.05 <= rt <= 600.0 else None
-
-
 def read_blastware_file(path: Union[str, Path]) -> Event:
     """
     Parse a Blastware waveform file into an Event.
@@ -878,13 +814,7 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
 
     # Footer: locate the 0e 08 marker, validating the year is in a sane range.
     body_start = _bw._WAVEFORM_HEADER_SIZE + 21
-    # The 0e 08 + plausible-year footer signature can occur inside the sample
+    footer_pos = -1
-    # stream.  Collect every candidate and prefer the first whose body yields a
-    # waveform record chain terminating on the 0x06 marker; fall back to the
-    # first candidate otherwise.  Blast radius measured 2026-08-25: changes the
-    # chosen footer on exactly 1 of 1,388 series-3 waveform files
-    # (BE17353/S353L4O5.OX0W, false positive at 3800, real footer at 8576).
-    footer_candidates = []
     pos = body_start
     while True:
         pos = raw.find(b"\x0e\x08", pos)
@@ -892,23 +822,9 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
             break
         yr = (raw[pos + 4] << 8) | raw[pos + 5]
         if 2015 <= yr <= 2050:
-            footer_candidates.append(pos)
+            footer_pos = pos
-        pos += 1
-
-    footer_pos = -1
-    for cand in footer_candidates:
-        cand_body = raw[body_start:cand]
-        try:
-            recs = _wc.walk_records(cand_body)
-        except Exception:
-            recs = []
-        if recs:
-            tail = recs[-1]["end"]
-            if tail + 5 <= len(cand_body) and cand_body[tail + 4] == _wc.STREAM_END_ID:
-                footer_pos = cand
             break
-    if footer_pos < 0 and footer_candidates:
+        pos += 1
-        footer_pos = footer_candidates[0]
 
     if footer_pos < 0 and len(raw) >= 26:
         footer_pos = len(raw) - 26
@@ -951,10 +867,6 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
     # rest of the event (timestamp, waveform_key, project strings) is
     # still recoverable and useful.
     decoded = decode_waveform_v2(body)
-    # Discriminator for the timestamp logic below: a waveform (trigger) event
-    # vs a histogram window.  Keyed on the codec, not the filename — the
-    # save_imported_bw path passes a tmp ".bw" name whose extension lies.
-    is_waveform_body = decoded is not None
     if decoded is None:
         decoded = decode_histogram_body(body)
     if decoded is None:
@@ -986,31 +898,7 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
     ev.total_samples   = strt_fields.get("total_samples")
     ev.pretrig_samples = strt_fields.get("pretrig_samples")
 
-    # Event timestamp.  The footer's two timestamps mean different things by
+    if ts1 is not None:
-    # record type:
-    #   * Waveform: ts1 = the monitoring-SESSION start (shared across every
-    #     event that day — a unit arming at 06:00 stamps 06:00 on all of them),
-    #     ts2 = THIS event's recording STOP.  Blastware's Date/Time is the
-    #     TRIGGER = ts2 - record time, and the record time is a float32 in the
-    #     recording-setup config block (see _parse_record_time_seconds), so the
-    #     exact trigger is recoverable from the binary alone.  Falls back to ts2
-    #     (the stop, within the record duration) if the config block is absent.
-    #     (Stamping ts1 showed the session start, hours off.)
-    #   * Histogram / undecodable: ts1 = the window start, which IS the event
-    #     time — keep it.
-    # Discriminate by ``is_waveform_body`` (the codec), not the filename.
-    if is_waveform_body and ts2 is not None:
-        _stop = datetime.datetime(ts2.year, ts2.month, ts2.day,
-                                  ts2.hour, ts2.minute, ts2.second)
-        _rt = _parse_record_time_seconds(raw)
-        _trig = _stop - datetime.timedelta(seconds=_rt) if _rt is not None else _stop
-        ev.timestamp = Timestamp(
-            raw=footer[10:18],
-            flag=0x10,
-            year=_trig.year, unknown_byte=0, month=_trig.month, day=_trig.day,
-            hour=_trig.hour, minute=_trig.minute, second=_trig.second,
-        )
-    elif ts1 is not None:
         ev.timestamp = Timestamp(
             raw=footer[2:10],
             flag=0x10,
@@ -1022,11 +910,6 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
         project=project, client=client, operator=user, sensor_location=seisloc,
     )
     ev.raw_samples = samples
-    # Sensor self-check traces from the binary's trailing block (waveform
-    # events only; returns {} for histograms / when absent).  Carried on the
-    # Event so the .h5 writer persists them device-agnostically.
-    from minimateplus.sensor_check import decode_sensor_check
-    ev.sensor_check = decode_sensor_check(raw) or None
     # Only compute peaks from samples when we actually have samples.
     # For events the codec couldn't decode (histogram-mode bodies, until
     # the §7.6.2 histogram codec is wired in), samples is an empty dict

minimateplus/histogram_codec.py

@@ -25,60 +25,38 @@ iterate 32-stride and stop before the tail.
 ────────────────────────────────────────────────────────────────────────────
 
     [0]    0x00                      always-zero tag
-    [1]    segment_id  (uint8)       0x00..0x03 - 256 blocks per segment
+    [1]    segment_id  (uint8)       0x00..0x03 — 256 blocks per segment
-    [2:4]  block_ctr  (uint16 LE)    resets each segment (0x0100, 0x0101, ...)
+    [2:4]  block_ctr  (uint16 LE)    resets each segment (0x0100, 0x0101, …)
-    [4]    0x0a (uint8)              constant marker (= 10)
+    [4:6]  0x000a (uint16 LE)        constant marker (= 10)
-    [5:7]  T_peak_count   uint16 BE  Tran peak (count x 0.005 -> in/s)
+    [6]    T_peak_count   uint8      Tran peak (count × 0.005 → in/s, max 1.275 in/s)
-    [7:9]  T_halfperiod   uint16 BE  Tran half-period in samples (freq = 512 / halfp)
+    [7]    T_annotation   uint8      empirically non-zero on intervals with sub-Hz
-    [9:11] V_peak_count   uint16 BE
+                                     or unmeasurable Tran freq; meaning not fully RE'd
-    [11:13] V_halfperiod  uint16 BE
+    [8:10] T_halfperiod   uint16 LE  Tran half-period in samples (freq = 512 / halfp Hz)
-    [13:15] L_peak_count  uint16 BE
+    [10]   V_peak_count   uint8
-    [15:17] L_halfperiod  uint16 BE
+    [11]   V_annotation   uint8
-    [17:19] M_peak_count  uint16 BE  MicL peak (count -> dB via mic_count_to_db)
+    [12:14] V_halfperiod  uint16 LE
-    [19:21] M_halfperiod  uint16 BE  MicL half-period in samples
+    [14]   L_peak_count   uint8
-    [21:23] 0x00 0x00                constant on standard blocks
+    [15]   L_annotation   uint8
+    [16:18] L_halfperiod  uint16 LE
+    [18]   M_peak_count   uint8      MicL peak (count → dB via mic_count_to_db)
+    [19]   M_annotation   uint8
+    [20:22] M_halfperiod  uint16 LE  MicL half-period in samples (freq = 512 / halfp Hz)
+    [22:24] 0x00 0x00                constant
     [24:28] 4-byte variable          purpose unknown (possibly CRC or timestamp delta)
-    [28:32] block-end signature      see "Two block tails" below
+    [28:32] 0x1e 0x0a 0x00 0x00      constant block-end signature
 
-**Every per-channel field is uint16 BIG-endian** (confirmed 2026-08-25).
+NOTE on peak-count width: an earlier interpretation treated the peak
-Only ``block_ctr`` at [2:4] is little-endian.
+fields as uint16 LE spanning [6:8] / [10:12] / [14:16] / [18:20].
+That happened to be byte-exact against the N844 fixture corpus only
+because every annotation byte in those fixtures was zero, making
+``uint16 LE == uint8``.  Cross-correlating BE9558 (K558) Tran-drift
+and BE18003 (T003) Histogram+Continuous events against the BW ASCII
+export proved peak is uint8 alone — see test_histogram_codec.py
+and docs/histogram_codec_re_status.md.
 
-HISTORY - two earlier readings of this block were wrong in ways that
+Block-identification anchor: ``block[22:24] == b"\\x00\\x00"`` AND
-cancelled out on quiet data:
+``block[28:32] == b"\\x1e\\x0a\\x00\\x00"``.  This is the reliable
-
+distinguisher from non-block content in the file.
-  1. *peak as uint16 LE at [6:8]* - produced 268 in/s peaks on any
-     interval whose next byte was non-zero.
-  2. *peak as uint8 at [6] with an "annotation" byte at [7]* - correct
-     for every peak below 256 counts (1.275 in/s), but it silently
-     **clipped larger peaks**: the final interval of
-     BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export
-     (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590 in/s.
-     The "annotation" byte was never an annotation - it is the high
-     byte of the big-endian half-period, which is why it was non-zero
-     exactly on the sub-Hz intervals BW renders as "<1.0".
-
-Both readings also forced ``block[5] == 0`` via a bogus ``uint16 LE``
-marker check at [4:6], which is what capped the peak at one byte.
-The marker is ``block[4]`` alone.
-
-Verified 2026-08-25 against 1211 production histograms paired with
-their Blastware ASCII exports: **1211/1211 decode exactly** (interval
-count plus every per-interval peak), and 842,442 per-interval
-frequency comparisons match with **zero** mismatches.
-
-Two block tails
----------------
-Standard blocks end with ``1e 0a 00 00``.  The **final block of the
-stream** ends with ``9c 06 00 42`` instead, and carries arbitrary bytes
-at [21:23].  Rejecting it dropped the last interval of nearly every
-histogram - and the last interval is frequently the one holding the
-event peak, so the file's reported PPV came out low.  Observed in 1206
-of 1211 production histograms, always positioned after every
-standard-tail block.
-
-Block-identification anchor: ``block[0] == 0x00`` AND
-``block[4] == 0x0A`` AND the tail is one of the two signatures above;
-standard-tail blocks additionally require ``block[22] == 0x00``.
 
 ────────────────────────────────────────────────────────────────────────────
 Per-channel encoding
@@ -131,12 +109,6 @@ from typing import List, Optional, Tuple
 # real data block.  More distinctive than the byte-22 `00 00` (which
 # matches many false positives), so we anchor on this.
 _BLOCK_TAIL = b"\x1e\x0a\x00\x00"
-
-# The final block of a histogram stream ends with this instead.  It is a
-# real data block - same layout - and holds the last interval.  See the
-# module docstring, "Two block tails".
-_BLOCK_TAIL_TERMINAL = b"\x9c\x06\x00\x42"
-
 _BLOCK_SIZE = 32
 
 # Marker byte at block[4:6] of every histogram data block.  Used as
@@ -155,29 +127,19 @@ _FREQ_NUMERATOR = 512
 
 
 def _is_data_block(block: bytes) -> bool:
-    """Tight identification of a histogram data block.
+    """Tight identification of a histogram data block."""
-
-    Accepts both tail signatures.  ``block[4]`` alone is the marker -
-    ``block[5]`` is the high byte of the Tran peak and is non-zero on any
-    interval above 1.275 in/s, so it must not be part of the marker test.
-    The ``block[22] == 0`` constraint is what keeps trailer content out,
-    but it applies only to standard-tail blocks: terminal blocks carry
-    arbitrary bytes there.
-    """
     if len(block) < _BLOCK_SIZE:
         return False
+    if block[28:32] != _BLOCK_TAIL:
+        return False
+    if block[22:24] != b"\x00\x00":
+        return False
     if block[0] != 0x00:
         return False
-    if block[4] != _BLOCK_MARKER:
+    marker = block[4] | (block[5] << 8)
+    if marker != _BLOCK_MARKER:
         return False
-    # The 4-byte tail plus block[0]==0 and block[4]==0x0A is already six bytes
+    return True
-    # of constraint — enough to keep trailer content out.  There is NO extra
-    # test on block[22]: it was documented as a constant 0x00 but carries data
-    # on loud blocks, and rejecting those threw away the interval holding the
-    # event peak.  BE18350/T350L7HR.NL0H is the proof: its block 92 has
-    # block[22]=0x26 and a Tran peak of 0x0563 = 1379 counts = 6.895 in/s,
-    # exactly the device-reported PPV, while the file decoded to 0.015 in/s.
-    return block[28:32] in (_BLOCK_TAIL, _BLOCK_TAIL_TERMINAL)
 
 
 def _decode_block(block: bytes) -> Optional[dict]:
@@ -187,23 +149,33 @@ def _decode_block(block: bytes) -> Optional[dict]:
     Returns a record with per-channel peak counts (uint8) and
     half-periods (uint16 LE).
     """
-    # Every per-channel field is uint16 BIG-endian; only block_ctr is LE.
+    # Peak counts are uint8 at bytes [6] / [10] / [14] / [18].  The
-    # See the module docstring for the two superseded readings and why
+    # adjacent bytes [7] / [11] / [15] / [19] hold an annotation field
-    # each looked correct on quiet data.
+    # whose meaning isn't fully understood (empirically non-zero in
-    def _be16(i: int) -> int:
+    # intervals with sub-Hz or unmeasurable geo frequencies, mostly
-        return (block[i] << 8) | block[i + 1]
+    # zero otherwise — see test fixtures from BE9558/BE18003 corpora).
-
+    # Crucially, those annotation bytes are NOT the high byte of the
-    t_peak  = _be16(5)
+    # peak count: cross-correlating against BW's per-interval ASCII
-    t_halfp = _be16(7)
+    # export proves the peak is uint8 alone.
-    v_peak  = _be16(9)
+    #
-    v_halfp = _be16(11)
+    # Reading the peak as uint16 LE (the original interpretation) was
-    l_peak  = _be16(13)
+    # accidentally correct only because every block in the N844 fixture
-    l_halfp = _be16(15)
+    # corpus had a zero annotation byte; non-N844 events with non-zero
-    m_peak  = _be16(17)
+    # annotation bytes decoded to physically impossible peaks (e.g.
-    m_halfp = _be16(19)
+    # 268 in/s per channel) and produced 35× inflated PVS sums when
+    # first run against prod data.  See histogram_codec_re_status.md.
+    t_peak = block[6]
+    v_peak = block[10]
+    l_peak = block[14]
+    m_peak = block[18]
+    t_halfp = block[8]  | (block[9]  << 8)
+    v_halfp = block[12] | (block[13] << 8)
+    l_halfp = block[16] | (block[17] << 8)
+    m_halfp = block[20] | (block[21] << 8)
     segment_id = block[1]
     block_ctr  = block[2] | (block[3] << 8)
     var_meta   = bytes(block[24:28])
+    annotations = (block[7], block[11], block[15], block[19])
     return {
         "segment_id":  segment_id,
         "block_ctr":   block_ctr,
@@ -216,7 +188,7 @@ def _decode_block(block: bytes) -> Optional[dict]:
         "m_peak":      m_peak,
         "m_halfp":     m_halfp,
         "meta_var":    var_meta,
-        "is_terminal": block[28:32] == _BLOCK_TAIL_TERMINAL,
+        "annotations": annotations,
     }
 
 
@@ -252,23 +224,6 @@ def walk_body(body: bytes) -> List[dict]:
     return records
 
 
-def _walk_auto(body: bytes) -> List[dict]:
-    """Pick the block model by signature strength, not by which returns first.
-
-    The multi-interval variant announces itself with consecutive block headers
-    at an exact ``12 + 20*n`` stride — far stronger evidence than a handful of
-    scattered standard-tail blocks, which a multi-interval body will also yield
-    by coincidence.  Dispatching on "whichever decoder returns something"
-    handed 193 BE18193 files to the standard walker and produced peaks of
-    149 in/s against a 10 in/s full scale.
-    """
-    if detect_multi_interval_stride(body):
-        recs = walk_multi_interval_blocks(body)
-        if recs:
-            return recs
-    return walk_body(body)
-
-
 def decode_histogram_body(body: bytes) -> Optional[dict]:
     """Decode a histogram-mode body into per-channel peak-sample arrays.
 
@@ -284,7 +239,7 @@ def decode_histogram_body(body: bytes) -> Optional[dict]:
          to get 1-count ADC values, then ``count / 32767 * 10.0`` for in/s)
       - Mic channel:  use ``waveform_codec.mic_count_to_db(count)``
     """
-    records = _walk_auto(body)
+    records = walk_body(body)
     if not records:
         return None
     return {
@@ -306,7 +261,7 @@ def decode_histogram_body_full(body: bytes) -> Optional[List[dict]]:
 
     Returns ``None`` if the body has no valid blocks.
     """
-    records = _walk_auto(body)
+    records = walk_body(body)
     return records if records else None
 
 
@@ -326,162 +281,3 @@ def half_period_to_hz(halfp: int) -> Optional[float]:
 def geo_count_to_ins(count: int) -> float:
     """Convert a histogram geo peak count to in/s at Normal range."""
     return count * _GEO_LSB_INS
-
-
-# ── Multi-interval block variant (CONFIRMED 2026-08-26) ─────────────────────
-#
-# When the histogram interval is SHORTER than one minute, the device packs
-# several intervals into a single block so that every block still covers
-# exactly one minute of data:
-#
-#     interval size   intervals/block   stride
-#     1 minute        1                 32   <- the standard block above
-#     15 seconds      4                 92
-#     2 seconds       30                612
-#
-#     stride = 12 + n_intervals * 20
-#
-# Block layout:
-#     [0]      0x00
-#     [1]      segment_id (256 blocks per segment, same as the standard block)
-#     [2:4]    block_ctr uint16 LE (0x0100.., resets each segment)
-#     [4]      0x0a marker
-#     [5]      0x00
-#     [6 ...]  n x 20-byte interval records, each carrying 8 x uint16
-#              LITTLE-endian values:
-#                  T_peak, T_halfperiod, V_peak, V_halfperiod,
-#                  L_peak, L_halfperiod, M_peak, M_halfperiod
-#              then 2 more words; the first is 0x0000 on every real interval.
-#     [-6:]    6-byte block trailer
-#
-# ⚠ ENDIANNESS: the standard 32-byte block is BIG-endian.  This variant is
-# LITTLE-endian.  Do not share the accessor.
-#
-# These files previously decoded to nothing at all — 415 of them in the
-# production snapshot, 216 on BE18193 (2 s intervals) and 199 on BE9440
-# (15 s).  Before that they were being accepted by the WAVEFORM codec, which
-# returned garbage peaking up to 400x the device-reported PPV.
-#
-# Ground truth: BE9440/K440L3AQ.T70H (15 s intervals, 5,710 of them) decodes
-# against its Blastware ASCII export with 17,130/17,130 geo peak counts,
-# 22,840/22,840 frequencies and 5,710/5,710 mic dB(L) values matching exactly.
-
-_MULTI_HEADER_LEN   = 6
-_MULTI_RECORD_LEN   = 20
-_MULTI_TRAILER_LEN  = 6
-# At least 2 records: a 1-record block would have stride 12 + 20 = 32, which
-# collides with the standard big-endian block and mis-decodes it.
-_MULTI_MIN_RECORDS  = 2
-_MULTI_MAX_RECORDS  = 64
-
-# Geo full scale in 16-count units: 10.000 in/s / 0.005 = 2000.  A peak above
-# this is physically impossible and marks buffer garbage in a partial block.
-_GEO_MAX_COUNTS = 2000
-
-
-def _is_multi_header(body: bytes, off: int) -> bool:
-    return (off + _MULTI_HEADER_LEN <= len(body)
-            and body[off] == 0x00
-            and body[off + 4] == 0x0A
-            and body[off + 5] == 0x00)
-
-
-def detect_multi_interval_stride(body: bytes) -> Optional[int]:
-    """Block stride of a multi-interval histogram body, or None.
-
-    Found by locating the second block header; validated against
-    ``stride = 12 + n * 20`` and confirmed on a third block where present.
-    """
-    if not _is_multi_header(body, 0):
-        return None
-    lo = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MIN_RECORDS
-    hi = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MAX_RECORDS
-    for stride in range(lo, min(hi, len(body)) + 1, 2):
-        if (stride - 12) % _MULTI_RECORD_LEN:
-            continue
-        if not _is_multi_header(body, stride):
-            continue
-        # DECISIVE CHECK: consecutive blocks differ by exactly 1 in block_ctr.
-        # Without it this false-positives on ordinary standard-block bodies:
-        # those carry a header every 32 bytes, and 192 = 12 + 20*9 and
-        # 512 = 12 + 20*25 are both multiples of 32, so a stride "fits" while
-        # actually skipping 6 or 16 real blocks.  Sampling a standard body at
-        # stride 192 handed 9,082 files to the wrong decoder and produced peaks
-        # of 149 in/s against a 10 in/s full scale.
-        def _ctr(o: int) -> int:
-            return body[o + 2] | (body[o + 3] << 8)
-
-        if (_ctr(stride) - _ctr(0)) & 0xFFFF != 1:
-            continue
-        # Confirm on a third block WHEN ONE IS ACTUALLY PRESENT.  A body can
-        # be longer than two strides and still hold only two real blocks: a
-        # final *partial* block leaves trailing padding.  E.g. 51 intervals at
-        # 2 s = one full 30-interval block + a 21-interval remainder, in a
-        # 2787-byte body — long enough to demand a third header at 1224 that
-        # does not exist.  Requiring it unconditionally threw away the correct
-        # stride and the file decoded to nothing (BE18193 T193L0XM.CI0H).
-        # The block-counter check above is the decisive anti-false-positive
-        # test; this one is corroboration, so a missing third header means
-        # end-of-stream, not disqualification.
-        if (2 * stride + _MULTI_HEADER_LEN <= len(body)
-                and _is_multi_header(body, 2 * stride)):
-            if (_ctr(2 * stride) - _ctr(stride)) & 0xFFFF != 1:
-                continue
-        return stride
-    return None
-
-
-def walk_multi_interval_blocks(body: bytes,
-                               stride: Optional[int] = None) -> List[dict]:
-    """Decode a multi-interval histogram body into per-interval records."""
-    if stride is None:
-        stride = detect_multi_interval_stride(body)
-    if not stride:
-        return []
-    n_per_block = (stride - _MULTI_HEADER_LEN - _MULTI_TRAILER_LEN) // _MULTI_RECORD_LEN
-    if n_per_block < 1:
-        return []
-
-    def u16le(p: int) -> int:
-        return body[p] | (body[p + 1] << 8)
-
-    out: List[dict] = []
-    for off in range(0, len(body) - stride + 1, stride):
-        if not _is_multi_header(body, off):
-            break                      # end of the block run; trailer follows
-        for k in range(n_per_block):
-            q = off + _MULTI_HEADER_LEN + _MULTI_RECORD_LEN * k
-            out.append({
-                "_tail0": u16le(q + 16),
-                "segment_id": body[off + 1],
-                "block_ctr":  u16le(off + 2),
-                "t_peak": u16le(q),      "t_halfp": u16le(q + 2),
-                "v_peak": u16le(q + 4),  "v_halfp": u16le(q + 6),
-                "l_peak": u16le(q + 8),  "l_halfp": u16le(q + 10),
-                "m_peak": u16le(q + 12), "m_halfp": u16le(q + 14),
-                "meta_var": bytes(body[q + 16:q + 20]),
-                "is_terminal": False,
-            })
-    # A session ending mid-block leaves the remaining slots of the FINAL block
-    # filled with whatever was in the buffer.  Those decoded as peaks thousands
-    # of times the device-reported PPV, so they have to go — but only from the
-    # final block: a non-zero tail word occurs mid-file on real intervals, and
-    # trimming on that alone truncated four BE9440 files by up to 2,800
-    # intervals, while trimming purely from the end left garbage stranded
-    # behind one slot that happened to have a zero tail word.
-    #
-    # Within the final block, stop at the first slot that is not plausibly
-    # real: a non-zero tail word, or a geo peak above full scale.  16-count
-    # units put Normal-range full scale (10.000 in/s) at 2000 counts, so
-    # anything beyond that is physically impossible.
-    if out:
-        last_block_start = ((len(out) - 1) // n_per_block) * n_per_block
-        for i in range(last_block_start, len(out)):
-            r = out[i]
-            if (r["_tail0"] != 0
-                    or max(r["t_peak"], r["v_peak"], r["l_peak"]) > _GEO_MAX_COUNTS):
-                del out[i:]
-                break
-    for r in out:
-        r.pop("_tail0", None)
-    return out

minimateplus/models.py

@@ -352,14 +352,6 @@ class PeakValues:
     long: Optional[float] = None             # Longitudinal PPV (in/s) ✅
     micl: Optional[float] = None             # Air overpressure (psi) 🔶 (units uncertain)
     peak_vector_sum: Optional[float] = None  # Scalar geo PVS (in/s) ✅
-    tran_zc_freq: Optional[float] = None
-    vert_zc_freq: Optional[float] = None
-    long_zc_freq: Optional[float] = None
-    mic_zc_freq:  Optional[float] = None
-    tran_zc_above_range: bool = False
-    vert_zc_above_range: bool = False
-    long_zc_above_range: bool = False
-    mic_zc_above_range:  bool = False
 
 
 # ── Project / operator metadata ───────────────────────────────────────────────
@@ -544,15 +536,6 @@ class Event:
     pretrig_samples: Optional[int] = None   # from STRT record: pre-trigger sample count
     rectime_seconds: Optional[int] = None   # from STRT record: record duration (seconds)
 
-    # Sensor self-check traces keyed by channel label — the short diagnostic
-    # waveforms the unit records when it pulses each sensor before monitoring
-    # (geophone ring-downs + a mic pulse train).  Decoded from the binary by
-    # the per-series decoder (minimateplus.sensor_check / micromate.sensor_check)
-    # and carried here so the .h5 writer can persist them device-agnostically.
-    # Raw ADC counts; the source series' scale differs but the trace is a
-    # shape diagnostic (rendered fit-to-box).  None when absent.
-    sensor_check: Optional[dict] = None     # {"Tran": [...], ..., "MicL": [...]}
-
     # ── Debug / introspection ─────────────────────────────────────────────────
     # Raw 210-byte waveform record bytes, set when debug mode is active.
     # Exposed by the SFM server via ?debug=true so field layouts can be verified.

minimateplus/sensor_check.py

@@ -1,146 +0,0 @@
-r"""Decode the Blastware sensor self-check waveforms from a series-3 event binary.
-
-Reverse-engineered 2026-09-15 against 7 BE12844 (MiniMate Plus) oracle events.
-After the main waveform record-chain and the trailing metadata / per-channel
-calibration records, the binary carries four length-prefixed records tagged
-0x3c-0x3f: the sensor self-check traces the unit records when it pulses each
-sensor before monitoring.  Blastware draws these as the little waveforms in the
-"Sensor Check" strip on the right of the Event Report.
-
-  * 0x3c / 0x3d / 0x3e = Tran / Vert / Long geophone ring-downs (a damped
-    oscillation at the geophone's resonance, ~7-8 Hz at 1024 sps).
-  * 0x3f            = MicL, a pulse train at the mic self-test frequency
-    (~20 Hz), whose zero-crossing frequency is BW's mic "Channel Test" freq.
-
-Record framing (per record, all four chained by their length prefix)::
-
-    [len:2 BE][id:1][00 00][Nchan:1][12-byte header][delta stream][40 02][6B]
-    \_________________ payload (len bytes) _______________________________/
-
-The delta stream is ``payload[20 : len-8]`` (the ``40 02`` terminator sits at
-``len-8``, followed by 6 trailing bytes).  It uses the exact same 10/20/30/00
-delta-block tags as the main waveform codec
-(:mod:`minimateplus.waveform_codec`), decoded here from an implicit anchor of 0
-— so the traces come out in the same 16-count raw units as the main waveform
-(LSB = 0.005 in/s at Normal range for the geophones).
-"""
-from __future__ import annotations
-
-from typing import Dict, List
-
-from minimateplus.waveform_codec import walk_body
-
-# Record id → channel.  Order mirrors the trailing per-channel calibration
-# records (Tran / Vert / Long / MicL), confirmed against BW's sensor-check
-# frequencies on all 7 oracle events.
-_ID_TO_CHANNEL = {0x3C: "Tran", 0x3D: "Vert", 0x3E: "Long", 0x3F: "MicL"}
-_CHAIN_IDS = (0x3C, 0x3D, 0x3E, 0x3F)
-
-_HEADER_LEN = 20    # payload bytes before the delta stream
-_TRAILER_LEN = 8    # 40 02 terminator + 6 trailing bytes after the stream
-
-
-def _s4(nib: int) -> int:
-    """Sign-extend a 4-bit nibble delta."""
-    return nib - 16 if nib >= 8 else nib
-
-
-def _i8(byte: int) -> int:
-    """Sign-extend an 8-bit int delta."""
-    return byte - 256 if byte >= 128 else byte
-
-
-def _decode_delta_stream(buf: bytes) -> List[int]:
-    """Accumulate a 10/20/30/00 delta-block stream from an anchor of 0,
-    stopping at the 0x40 terminator.
-
-    Mirrors the block semantics in
-    :func:`minimateplus.waveform_codec.decode_waveform_v2` (fully decoded &
-    byte-exact as of 2026-05-11); see that module for the format details.
-    """
-    out: List[int] = []
-    cur = 0
-    for blk in walk_body(buf, 0):
-        fam = blk.tag_hi & 0xF0
-        if fam == 0x10:
-            # nibble deltas, high nibble first
-            for byte in blk.data:
-                for nib in ((byte >> 4) & 0xF, byte & 0xF):
-                    cur += _s4(nib)
-                    out.append(cur)
-        elif fam == 0x20:
-            # int8 deltas
-            for byte in blk.data:
-                cur += _i8(byte)
-                out.append(cur)
-        elif fam == 0x30:
-            # 12-bit signed deltas, packed as tag_lo/4 groups of 6 bytes
-            for g in range(blk.tag_lo // 4):
-                grp = blk.data[g * 6:(g + 1) * 6]
-                if len(grp) < 6:
-                    break
-                high_word = (grp[0] << 8) | grp[1]
-                for k in range(4):
-                    nib = (high_word >> (12 - 4 * k)) & 0xF
-                    v = (nib << 8) | grp[2 + k]
-                    if v >= 0x800:
-                        v -= 0x1000
-                    cur += v
-                    out.append(cur)
-        elif fam == 0x00:
-            # RLE zero-delta run (wide form carries the high nibble in the tag)
-            run = ((blk.tag_hi & 0x0F) << 8) | blk.tag_lo
-            out.extend([cur] * run)
-        elif fam == 0x40:
-            # segment / record terminator
-            break
-    return out
-
-
-def _find_chain(body: bytes):
-    """Locate the four length-prefixed sensor-check records.
-
-    Returns a list of ``(offset, id, length)`` or ``None``.  The chain is
-    validated by walking the ids 0x3c → 0x3d → 0x3e → 0x3f via their own length
-    prefixes, so a stray 0x3c byte in the waveform data cannot match.
-    """
-    for p in range(len(body) - 6):
-        if body[p + 2] == 0x3C and body[p + 3] == 0 and body[p + 4] == 0:
-            q = p
-            recs = []
-            ok = True
-            for expect in _CHAIN_IDS:
-                if q + 3 > len(body) or body[q + 2] != expect:
-                    ok = False
-                    break
-                length = int.from_bytes(body[q:q + 2], "big")
-                recs.append((q, expect, length))
-                q = q + 2 + length
-            if ok and len(recs) == 4:
-                return recs
-    return None
-
-
-def decode_sensor_check(raw: bytes) -> Dict[str, List[int]]:
-    """Decode the four sensor self-check traces from a series-3 event binary.
-
-    Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}`` in
-    raw decode units (same 16-count LSB as the main waveform), or ``{}`` if the
-    binary carries no sensor-check block (a histogram event, a non-series-3
-    file, or a unit/firmware that doesn't store it).
-    """
-    strt = raw.find(b"STRT")
-    if strt < 0 or len(raw) < strt + 21 + 26:
-        return {}
-    body = raw[strt + 21: len(raw) - 26]
-    chain = _find_chain(body)
-    if not chain:
-        return {}
-    out: Dict[str, List[int]] = {}
-    for off, rid, length in chain:
-        payload = body[off + 2: off + 2 + length]
-        if len(payload) < _HEADER_LEN + _TRAILER_LEN:
-            continue
-        stream = payload[_HEADER_LEN: length - _TRAILER_LEN]
-        out[_ID_TO_CHANNEL[rid]] = _decode_delta_stream(stream)
-    return out

minimateplus/waveform_codec.py

@@ -166,14 +166,8 @@ def find_data_start(body: bytes) -> int:
     # Try fixed offset 7 first (canonical preamble length).
     if len(body) >= 9:
         b, nn = body[7], body[8]
-        # Accept the same tag vocabulary ``walk_body`` accepts, including the
+        if (b in (0x00, 0x10, 0x20, 0x30) and nn % 4 == 0 and 0 < nn <= 0xFC) \
-        # wide-NN forms (``0X``/``1X``/``2X``) and the variable-width ``40 NN``
+                or (b == 0x40 and nn == 0x02):
-        # segment header.
-        if ((b & 0xF0) in (0x00, 0x10, 0x20) and nn % 4 == 0
-                and ((b & 0x0F) != 0 or 0 < nn <= 0xFC)) \
-                or (b == 0x30 and nn % 4 == 0 and 0 < nn <= 0xFC) \
-                or (b == 0x40 and 0 < nn <= 0x08) \
-                or is_tagless_segment_header(body, 7):
             return 7
     # Fall back to scanning the first 20 bytes.
     for i in range(min(20, len(body) - 1)):
@@ -184,31 +178,6 @@ def find_data_start(body: bytes) -> int:
     return -1
 
 
-# Channel-id byte carried in every segment header.  Previously mis-read as a
-# "monotonic uint32 LE counter"; it is really ``[channel][00][00][segment]``.
-# Verified 2026-08-25 on 1697/1697 segment headers across the ground-truth
-# corpus with zero disagreements against the decoded channel rotation.
-SEGMENT_CHANNEL_IDS = {0x46: "Tran", 0x47: "Vert", 0x48: "Long", 0x49: "MicL"}
-
-# A tagless segment header: the 14-byte tail of a ``40 NN`` header with no tag
-# and no previous-channel continuation deltas (the NN=0 case).
-_TAGLESS_HEADER_LEN = 14
-
-
-def is_tagless_segment_header(body: bytes, i: int) -> bool:
-    """True if a bare 14-byte segment header starts at *i*.
-
-    Layout ``[field2:2][len_to_next:2][channel_id:4][marker:2][anchors:4]``.
-    The discriminator is the 6 bytes at ``[4:10]``: a known channel id, two
-    zero bytes, a small segment index, and the ``01 00`` / ``02 00`` marker.
-    """
-    if i + _TAGLESS_HEADER_LEN > len(body):
-        return False
-    return (body[i + 4] in SEGMENT_CHANNEL_IDS
-            and body[i + 5] == 0x00 and body[i + 6] == 0x00
-            and body[i + 8] in (0x01, 0x02) and body[i + 9] == 0x00)
-
-
 def walk_body(body: bytes, start: Optional[int] = None) -> List[WaveformBlock]:
     """Walk the tagged-block sequence starting at *start* (auto-detected by default).
 
@@ -241,15 +210,9 @@ def walk_body(body: bytes, start: Optional[int] = None) -> List[WaveformBlock]:
             # Wide-NN int8 block: ``2X NN`` extends NN to 12 bits the same way.
             wide_nn = ((t0 & 0x0F) << 8) | t1
             length = wide_nn + 2
-        elif (t0 & 0xF0) == 0x00 and t1 % 4 == 0:
+        elif t0 == 0x00 and t1 % 4 == 0:
-            # ``00 NN`` RLE zero-delta run, plus its wide form ``0X NN``
-            # (X != 0) which extends NN to 12 bits exactly like ``1X``/``2X``:
-            # NN = ((t0 & 0x0F) << 8) | t1.  A narrow run maxes out at
-            # NN=0xFC, so quiet stretches longer than 252 samples must use
-            # the wide form.  Confirmed 2026-08-25 against six production
-            # events (e.g. ``01 0c`` = 268 repeats in K558LKOF.460W).
             length = 2
-        elif t0 == 0x30 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
+        elif t0 == 0x30 and t1 % 4 == 0 and 0 < t1 <= 0x10:
             # Data-section ``30 NN`` blocks carry NN 12-bit signed deltas packed
             # as NN/4 groups of (2-byte high-nibble field + 4 × int8 low byte).
             # Length = NN/4 × 6 + 2 = NN × 1.5 + 2 (= 8 for NN=4, 14 for NN=8,
@@ -266,28 +229,8 @@ def walk_body(body: bytes, start: Optional[int] = None) -> List[WaveformBlock]:
                 length = cand_data
             else:
                 length = cand_trailer
-        elif t0 == 0x40 and 0 < t1 <= 0x08:
+        elif t0 == 0x40 and t1 == 0x02:
-            # ``40 NN`` segment header.  NN is the number of int16 BE
+            length = 20
-            # continuation deltas the header carries for the PREVIOUS
-            # channel, so the header grows with NN:
-            #     length = 2 (tag) + 2*NN (deltas) + 14 (fixed tail)
-            # ``40 02`` (20 bytes) dominates, but ``40 01`` (18) and
-            # ``40 03`` (22) both occur in production files.  Confirmed
-            # 2026-08-25; the constant ``02 00`` marker moves with NN too
-            # (see :func:`parse_segment_header`).
-            length = 2 * t1 + 16
-        elif is_tagless_segment_header(body, i):
-            # Segment header with no ``40 NN`` tag (NN=0 — the previous channel
-            # needed no continuation deltas).  Emit it as a synthetic ``40 00``
-            # block whose ``data`` is the whole 14-byte record, so the nd=0
-            # offsets in :func:`decode_waveform_v2` line up unchanged.
-            blocks.append(WaveformBlock(
-                offset=i, tag_hi=0x40, tag_lo=0x00,
-                data=bytes(body[i : i + _TAGLESS_HEADER_LEN]),
-                length=_TAGLESS_HEADER_LEN,
-            ))
-            i += _TAGLESS_HEADER_LEN
-            continue
         else:
             # Unknown tag; stop.  Caller can inspect ``i`` to see where.
             break
@@ -313,7 +256,7 @@ def split_segments(blocks: List[WaveformBlock]) -> List[List[WaveformBlock]]:
     segments: List[List[WaveformBlock]] = []
     current: List[WaveformBlock] = []
     for b in blocks:
-        if b.tag_hi == 0x40:
+        if b.tag_hi == 0x40 and b.tag_lo == 0x02:
             if current:
                 segments.append(current)
             current = [b]
@@ -325,40 +268,23 @@ def split_segments(blocks: List[WaveformBlock]) -> List[List[WaveformBlock]]:
 
 
 def parse_segment_header(block: WaveformBlock) -> Optional[dict]:
-    """Decode the payload of a ``40 NN`` segment header.
+    """Decode the 18-byte payload of a ``40 02`` segment header.
 
-    NN (the tag's low byte) is the number of int16 BE continuation deltas
+    Returns a dict with the labelled fields, or None if *block* is not
-    the header carries for the PREVIOUS channel, so every field after
+    a ``40 02`` header.
-    those deltas shifts by ``2 * NN``.  The payload is ``2 * NN + 14``
-    bytes.  ``40 02`` is the common case; ``40 01`` and ``40 03`` also
-    occur in production files (confirmed 2026-08-25).
-
-    Returns a dict with the labelled fields, or None if *block* is not a
-    segment header or is too short.
     """
-    if block.tag_hi != 0x40 or block.tag_lo > 0x08:
+    if not (block.tag_hi == 0x40 and block.tag_lo == 0x02):
         return None
-    nd = block.tag_lo
+    if len(block.data) < 18:
-    if len(block.data) < 2 * nd + 14:
         return None
     p = block.data
-    counter = int.from_bytes(p[2 * nd + 4 : 2 * nd + 8], "little", signed=False)
+    counter = int.from_bytes(p[8:12], "little", signed=False)
     return {
-        "n_prev_deltas": nd,
+        "anchor_bytes": p[0:4],          # 4-byte field, role unconfirmed
-        # ``nd`` int16 BE deltas extending the previous channel.
+        "field2": p[4:8],                # 4-byte field, role unconfirmed
-        "prev_deltas": [
+        "counter": counter,              # uint32 LE — increments by 1 per segment
-            int.from_bytes(p[2 * k : 2 * k + 2], "big", signed=True)
+        "fixed_pattern": p[12:16],       # always b"\x02\x00\x00\x01"
-            for k in range(nd)
+        "tail": p[16:18],                # last 2 bytes
-        ],
-        "field2": p[2 * nd : 2 * nd + 4],   # 4-byte field, role unconfirmed
-        "counter": counter,                 # legacy: raw uint32 LE of the id field
-        "channel": SEGMENT_CHANNEL_IDS.get(p[2 * nd + 4]),
-        "segment_index": p[2 * nd + 7],
-        "marker": p[2 * nd + 8 : 2 * nd + 10],      # always b"\x02\x00"
-        "anchors": [
-            int.from_bytes(p[2 * nd + 10 : 2 * nd + 12], "big", signed=True),
-            int.from_bytes(p[2 * nd + 12 : 2 * nd + 14], "big", signed=True),
-        ],
     }
 
 
@@ -441,16 +367,8 @@ def decode_tran_initial(body: bytes) -> Optional[List[int]]:
     return out
 
 
-def decode_waveform_legacy(body: bytes) -> Optional[dict]:
+def decode_waveform_v2(body: bytes) -> Optional[dict]:
     """
-    SUPERSEDED 2026-08-25 — the tag-dispatch / segment-header model.
-
-    Retained because ``micromate/idf_file.py`` trial-decodes Thor IDFW bodies
-    at many candidate offsets and keeps whichever yields the most samples;
-    the record-chain decoder returns None where this one returned garbage,
-    which shifts that heuristic's winner.  Thor is pinned here until its own
-    body-offset search is reworked.  Do not use for series-3.
-
     Decode the body into per-channel sample arrays.
 
     Status (2026-05-11 evening — channel-rotation hypothesis CONFIRMED):
@@ -502,11 +420,8 @@ def decode_waveform_legacy(body: bytes) -> Optional[dict]:
                 for byte in blk.data:
                     cur += _i8(byte)
                     out[channel].append(cur)
-            elif (blk.tag_hi & 0xF0) == 0x00:
+            elif blk.tag_hi == 0x00:
-                # RLE zero-delta run.  Wide form ``0X NN`` carries the high
+                for _ in range(blk.tag_lo):
-                # nibble of a 12-bit NN in the tag byte, same as ``1X``/``2X``.
-                run = ((blk.tag_hi & 0x0F) << 8) | blk.tag_lo
-                for _ in range(run):
                     out[channel].append(cur)
             elif blk.tag_hi == 0x30:
                 # 12-bit signed deltas, packed as NN/4 groups of 6 bytes each:
@@ -546,54 +461,34 @@ def decode_waveform_legacy(body: bytes) -> Optional[dict]:
     # previous-channel extension deltas at every segment boundary.
     last_value = {"Tran": last_tran_value, "Vert": None, "Long": None, "MicL": None}
 
-    prev_channel = "Tran"
     for k, hi in enumerate(seg_idx):
-        header = blocks[hi]
-        # Channel comes from the header's own id byte, which is authoritative.
-        # The old rotation-by-position fallback is kept for headers whose id
-        # byte isn't one of the four known values — but a single missed or
-        # extra header would desync rotation and corrupt every later channel,
-        # which is exactly what tagless headers used to cause.
-        _nd = header.tag_lo
-        channel = None
-        if len(header.data) >= 2 * _nd + 8:
-            channel = SEGMENT_CHANNEL_IDS.get(header.data[2 * _nd + 4])
-        if channel is None:
         channel = rotation[k % 4]
-        # ``40 NN``: NN int16 BE continuation deltas for the previous channel
+        prev_channel = "Tran" if k == 0 else rotation[(k - 1) % 4]
-        # come first, so every later field shifts by 2*NN.  NN is usually 2
+        header = blocks[hi]
-        # but 1 and 3 both occur (confirmed 2026-08-25).
+        if len(header.data) < 18:
-        nd = header.tag_lo
-        if len(header.data) < 2 * nd + 14:
             continue
-        # Validate: real segment headers have the constant `02 00` marker
+        # Validate: real segment headers have bytes [12:14] = `02 00`.
-        # right after the counter.  Trailer/footer "40 NN" markers contain
+        # Trailer/footer "40 02" markers contain ASCII serial bytes or other
-        # ASCII serial bytes or other non-header data there and would
+        # non-header data there and would otherwise be mis-interpreted as
-        # otherwise be mis-read as segment headers, adding spurious tail
+        # segment headers, adding spurious samples at the tail.
-        # samples.
+        if header.data[12:14] != b"\x02\x00":
-        if header.data[2 * nd + 8 : 2 * nd + 10] != b"\x02\x00":
             break
-        # Extend the PREVIOUS channel by NN more samples.
+        # Extend the PREVIOUS channel by 2 more samples (deltas in bytes [0:4]).
+        prev_d0 = int.from_bytes(header.data[0:2], "big", signed=True)
+        prev_d1 = int.from_bytes(header.data[2:4], "big", signed=True)
         if last_value[prev_channel] is not None:
-            v = last_value[prev_channel]
+            v = last_value[prev_channel] + prev_d0
-            for d in range(nd):   # NB: not `k` — that's the segment index
+            out[prev_channel].append(v)
-                v += int.from_bytes(
+            v += prev_d1
-                    header.data[2 * d : 2 * d + 2], "big", signed=True
-                )
             out[prev_channel].append(v)
             last_value[prev_channel] = v
         # Anchor pair for THIS segment's channel.
-        c0 = int.from_bytes(
+        c0 = int.from_bytes(header.data[14:16], "big", signed=True)
-            header.data[2 * nd + 10 : 2 * nd + 12], "big", signed=True
+        c1 = int.from_bytes(header.data[16:18], "big", signed=True)
-        )
-        c1 = int.from_bytes(
-            header.data[2 * nd + 12 : 2 * nd + 14], "big", signed=True
-        )
         out[channel].extend([c0, c1])
         # Apply delta blocks for this segment.
         next_hi = seg_idx[k + 1] if k + 1 < len(seg_idx) else len(blocks)
         last_value[channel] = apply_blocks(channel, c1, hi + 1, next_hi)
-        prev_channel = channel
 
     return out
 
@@ -681,268 +576,3 @@ def decode_a5_frames(a5_frames) -> Optional[dict]:
     if decoded is None:
         return None
     return decoded_to_adc_counts(decoded)
-
-
-# ── Record-chain body model (CONFIRMED 2026-08-25) ──────────────────────────
-#
-# The body is NOT a flat tag-dispatch stream with ``40 NN`` segment headers.
-# It is a chain of self-delimiting per-channel RECORDS:
-#
-#     off+0   field2    uint16     purpose unknown (not a length, not a checksum)
-#     off+2   len       uint16 BE  next_record = off + 2 + len   <- authoritative
-#     off+4   chan_id   0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL
-#                       0x06 = end of waveform stream
-#     off+5   0x00
-#     off+6   0x00
-#     off+7   segment index
-#     off+8   mode      2 bytes, a 3-valued enum (see below)
-#     off+10  anchors   2 x int16 BE, ABSOLUTE — present only when mode is 02 00
-#
-# Mode semantics, all ground-truth verified:
-#     02 00   14-byte header; emit the 2 anchors, then blocks are CUMULATIVE deltas
-#     01 00   10-byte header; no anchors; blocks carry ABSOLUTE sample values
-#     00 03   10-byte header; NO TAGS AT ALL — the data section is raw 12-bit
-#             packed ABSOLUTE samples (6 bytes -> 4 samples)
-#
-# ``40 NN`` is an ordinary int16 BE DATA block (length 2*NN + 2), never a header.
-# The previous model read it as a variable-width segment header of length
-# 2*NN + 16, which is why walks drifted and channels came out unequal.
-#
-# Verified over the 1,388 series-3 waveform binaries in the production
-# snapshot: the length chain terminates on a 0x06 record in 1,387 of them (the
-# exception has an ambiguous footer, handled by the caller), and all four
-# channels come out at identical length in 1,388/1,388 — against 156/1,388
-# under the superseded model.  Against the 75 events with a preserved
-# Blastware ASCII export: sample-count exact 72/75 -> 75/75, fully exact
-# 70/75 -> 73/75.
-
-CHANNEL_IDS = {0x46: "Tran", 0x47: "Vert", 0x48: "Long", 0x49: "MicL"}
-STREAM_END_ID = 0x06
-
-MODE_DELTA    = (0x02, 0x00)
-MODE_ABSOLUTE = (0x01, 0x00)
-MODE_RAW12    = (0x00, 0x03)
-# Raw int16 BE absolute samples, 10-byte header, no tags — the same shape as
-# MODE_RAW12 but two bytes per sample instead of 1.5.  Found on Thor/Micromate
-# segment-0 records (2026-09-10): a `len=1032` record carries exactly
-# (1032 - 8) / 2 = 512 samples and reproduces Thor's own export 512/512
-# exactly.  Before this mode existed the record fell through the dispatch
-# unhandled, so the channel silently lost its first 512 samples.
-MODE_RAW16    = (0x00, 0x00)
-_MODES = (MODE_DELTA, MODE_ABSOLUTE, MODE_RAW12, MODE_RAW16)
-
-# Preambles whose leading data is untagged and therefore cannot be
-# block-walked; find_first_record() must scan for the next record instead.
-_UNTAGGED_MODES = (MODE_RAW12, MODE_RAW16)
-
-
-def _u16(b: bytes, p: int) -> int:
-    return (b[p] << 8) | b[p + 1]
-
-
-def _i16(b: bytes, p: int) -> int:
-    v = _u16(b, p)
-    return v - 0x10000 if v >= 0x8000 else v
-
-
-def data_block_len(body: bytes, p: int) -> Tuple[Optional[int], Optional[int]]:
-    """``(byte_length, n_samples)`` of the data block at *p*, or ``(None, None)``.
-
-    Data-section blocks only — there is no segment-header tag in this model.
-    ``30 NN`` has no trailer-length fallback here; that fallback corrupted
-    records whose ``30 NN`` sat near a record boundary.
-    """
-    if p + 2 > len(body):
-        return None, None
-    t0, t1 = body[p], body[p + 1]
-    hi = t0 & 0xF0
-    nn = ((t0 & 0x0F) << 8) | t1
-    if hi == 0x40:                          # int16 BE data block
-        # NN was capped at 0x08 until 2026-09-11.  That cap had no basis: the
-        # two corpora available at the time only ever used NN in {1,2,3,4,8},
-        # so it was never exercised.  Loud UM12947 events use NN of 12, 16,
-        # 20 ... up to 196, and every value above 8 halted the walk, which
-        # surfaced as silently short channels (walk_body/run stop at the first
-        # unrecognised tag rather than raising).  Verified against Thor's own
-        # exports: 22 length-mismatched files -> 0, and the affected corpus
-        # went to 1,476,242/1,476,249 samples exact.  The real bound is the
-        # buffer; the caller additionally clamps to the record end.
-        if nn == 0 or p + 2 * nn + 2 > len(body):
-            return None, None
-        return 2 * nn + 2, nn
-    if nn == 0 or nn % 4:
-        return None, None
-    if hi == 0x00:
-        return 2, nn                        # RLE hold
-    if hi == 0x10:
-        return nn // 2 + 2, nn              # 4-bit nibble
-    if hi == 0x20:
-        return nn + 2, nn                   # int8
-    if hi == 0x30:
-        return nn * 3 // 2 + 2, nn          # 12-bit packed
-    return None, None
-
-
-def unpack16(data: bytes) -> List[int]:
-    """Raw int16 BE absolute samples (MODE_RAW16)."""
-    return [_i16(data, 2 * k) for k in range(len(data) // 2)]
-
-
-def unpack12(data: bytes) -> List[int]:
-    """Raw 12-bit packed samples: 6 bytes -> 4 signed values."""
-    out: List[int] = []
-    for g in range(len(data) // 6):
-        hi = (data[6 * g] << 8) | data[6 * g + 1]
-        for k in range(4):
-            x = (((hi >> (12 - 4 * k)) & 0xF) << 8) | data[6 * g + 2 + k]
-            out.append(x - 0x1000 if x >= 0x800 else x)
-    return out
-
-
-def is_record(body: bytes, p: int) -> bool:
-    """True if a per-channel record header starts at *p*."""
-    return (p + 10 <= len(body)
-            and body[p + 4] in CHANNEL_IDS
-            and body[p + 5] == 0x00 and body[p + 6] == 0x00
-            and 8 <= _u16(body, p + 2) <= len(body) - p
-            and (body[p + 8], body[p + 9]) in _MODES)
-
-
-def find_first_record(body: bytes) -> Optional[int]:
-    """Offset of the first record, or None.
-
-    Under the normal ``00 02 00`` preamble the leading bytes are segment-0's
-    Tran blocks, so walk them.  Under the untagged preambles (``00 00 03``
-    raw-12 and ``00 00 00`` raw-16) that data has no tags at all and cannot
-    be block-walked — scan for the next record header instead.
-    """
-    if len(body) >= 3 and (body[1], body[2]) in _UNTAGGED_MODES:
-        scan_from = 3
-    else:
-        # Tagged preamble.  MODE_DELTA carries a 14-byte record header (two
-        # int16 anchors), so its blocks start at body[7]; MODE_ABSOLUTE has a
-        # 10-byte header and starts at body[3].
-        i = 3 if (len(body) >= 3 and (body[1], body[2]) == MODE_ABSOLUTE) else 7
-        while i < len(body):
-            if is_record(body, i):
-                nxt = i + 2 + _u16(body, i + 2)
-                if nxt + 5 <= len(body) and (is_record(body, nxt)
-                                             or body[nxt + 4] == STREAM_END_ID):
-                    return i
-            length, _ = data_block_len(body, i)
-            if length is None:
-                return None
-            i += length
-        return None
-    for i in range(scan_from, max(scan_from, len(body) - 10)):
-        if is_record(body, i):
-            nxt = i + 2 + _u16(body, i + 2)
-            if nxt + 5 <= len(body) and (is_record(body, nxt)
-                                         or body[nxt + 4] == STREAM_END_ID):
-                return i
-    return None
-
-
-def walk_records(body: bytes, first: Optional[int] = None) -> List[dict]:
-    """Follow the length chain from *first* to the ``0x06`` terminator."""
-    if first is None:
-        first = find_first_record(body)
-    out: List[dict] = []
-    if first is None:
-        return out
-    p, seen = first, set()
-    while p is not None and p + 10 <= len(body):
-        if p in seen:
-            break
-        seen.add(p)
-        cid = body[p + 4]
-        if cid == STREAM_END_ID or cid not in CHANNEL_IDS:
-            break
-        length = _u16(body, p + 2)
-        if length < 8 or p + 2 + length > len(body):
-            break
-        out.append({"offset": p, "channel": CHANNEL_IDS[cid],
-                    "segment_index": body[p + 7],
-                    "mode": (body[p + 8], body[p + 9]),
-                    "end": p + 2 + length})
-        p += 2 + length
-    return out
-
-
-def decode_waveform_v2(body: bytes) -> Optional[dict]:
-    """Decode a Blastware waveform body into per-channel sample arrays.
-
-    Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}``
-    in 16-count units (LSB = 0.005 in/s at Normal range), or None if *body*
-    is not a decodable waveform body.
-
-    Implements the record-chain model documented above.
-    """
-    if len(body) < 8 or body[0] != 0x00:
-        return None
-    preamble = (body[1], body[2])
-    if preamble not in (MODE_DELTA, MODE_ABSOLUTE, MODE_RAW12, MODE_RAW16):
-        return None
-    first = find_first_record(body)
-    if first is None:
-        return None
-
-    out: dict = {c: [] for c in ("Tran", "Vert", "Long", "MicL")}
-
-    def run(channel: str, start: int, end: int, absolute: bool) -> None:
-        cur = out[channel][-1] if out[channel] else 0
-        i = start
-        while i < end:
-            length, nn = data_block_len(body, i)
-            if length is None or i + length > end:
-                return          # stop this record; the chain resyncs at end
-            hi = body[i] & 0xF0
-            if hi == 0x00:
-                vals = [None] * nn
-            elif hi == 0x10:
-                vals = []
-                for k in range(nn):
-                    byte = body[i + 2 + k // 2]
-                    v = (byte >> 4) if k % 2 == 0 else (byte & 0xF)
-                    vals.append(v - 16 if v >= 8 else v)
-            elif hi == 0x20:
-                vals = [v - 256 if v >= 128 else v
-                        for v in body[i + 2:i + 2 + nn]]
-            elif hi == 0x30:
-                vals = unpack12(body[i + 2:i + length])
-            else:
-                vals = [_i16(body, i + 2 + 2 * k) for k in range(nn)]
-            for v in vals:
-                if v is None:
-                    pass                    # RLE hold, in delta AND absolute modes
-                elif absolute:
-                    cur = v
-                else:
-                    cur += v
-                out[channel].append(cur)
-            i += length
-
-    # Segment 0 is an implicit Tran record carried in the preamble.
-    if preamble == MODE_DELTA:
-        out["Tran"].extend([_i16(body, 3), _i16(body, 5)])
-        run("Tran", 7, first, absolute=False)
-    elif preamble == MODE_ABSOLUTE:
-        run("Tran", 3, first, absolute=True)
-    elif preamble == MODE_RAW16:
-        out["Tran"].extend(unpack16(body[3:first]))
-    else:
-        out["Tran"].extend(unpack12(body[3:first]))
-
-    for rec in walk_records(body, first):
-        ch, off, mode, end = (rec["channel"], rec["offset"],
-                              rec["mode"], rec["end"])
-        if mode == MODE_DELTA:
-            out[ch].extend([_i16(body, off + 10), _i16(body, off + 12)])
-            run(ch, off + 14, end, absolute=False)
-        elif mode == MODE_ABSOLUTE:
-            run(ch, off + 10, end, absolute=True)
-        elif mode == MODE_RAW12:
-            out[ch].extend(unpack12(body[off + 10:end]))
-        elif mode == MODE_RAW16:
-            out[ch].extend(unpack16(body[off + 10:end]))
-    return out

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "seismo-relay"
-version = "0.31.0"
+version = "0.20.0"
 description = "Python client and REST server for MiniMate Plus seismographs"
 requires-python = ">=3.10"
 dependencies = [

scratch/fake_unit.py

@@ -1,33 +0,0 @@
-"""Pretend to be a Micromate on a serial port: log what arrives, reply to POLL.
-
-Proves the modem's return path (serial -> TCP) independently of the real unit.
-"""
-import os, select, sys, termios, time
-
-path, baud = sys.argv[1], int(sys.argv[2]) if len(sys.argv) > 2 else 115200
-B = {9600: termios.B9600, 38400: termios.B38400, 115200: termios.B115200}[baud]
-fd = os.open(path, os.O_RDWR | os.O_NOCTTY | os.O_NONBLOCK)
-a = termios.tcgetattr(fd)
-a[0] = a[1] = a[3] = 0
-a[2] = termios.CS8 | termios.CREAD | termios.CLOCAL
-a[4] = a[5] = B
-a[6] = list(a[6]); a[6][termios.VMIN] = 0; a[6][termios.VTIME] = 0
-termios.tcsetattr(fd, termios.TCSANOW, a)
-termios.tcflush(fd, termios.TCIOFLUSH)
-
-# A real POLL probe reply, captured from UM12947 on 2026-09-24.
-REPLY = bytes.fromhex("0200c5a4000000000000300000000000000099") + b"\x03"
-
-print(f"fake unit on {path} @ {baud}; will answer any inbound frame", flush=True)
-while True:
-    r, _, _ = select.select([fd], [], [], 1.0)
-    if not r:
-        continue
-    data = os.read(fd, 4096)
-    if not data:
-        continue
-    ts = time.strftime("%H:%M:%S")
-    print(f"{ts}  IN  {len(data):3} B  {data.hex(' ')}", flush=True)
-    time.sleep(0.02)
-    os.write(fd, REPLY)
-    print(f"{ts}  OUT {len(REPLY):3} B  {REPLY.hex(' ')}  <- canned POLL reply", flush=True)

scratch/mm_frame_parse.py

@@ -1,202 +0,0 @@
-#!/usr/bin/env python3
-"""
-mm_frame_parse.py — parse Micromate (Series IV) frames out of a seismo_lab
-raw capture pair.
-
-Why this exists
----------------
-`minimateplus.framing.S3FrameParser` cannot see Micromate traffic.  It locates
-frames by scanning for `DLE STX`, and a Micromate response has **no leading
-DLE** — it starts at a bare `STX`.  It also expects `payload[1] == 0x10`, where
-the Micromate sends `0xC5` (Blastware firmware) or `0x03` (Thor firmware).
-
-The practical consequence, seen on the 9-24-26 setup-push capture: the
-Blastware-side requests parse fine (Thor emits Series III request frames), but
-**every device response is silently dropped or mis-framed** — so a capture that
-actually contains 12 acked writes looks like 12 unanswered requests.
-
-Destuffing
-----------
-One rule covers both directions: after the leading doubled `BW_CMD`, every
-`10 XX` pair on the wire destuffs to `XX`.  That includes `10 03` — Thor
-escapes literal `0x03` bytes in write data so they are not mistaken for ETX,
-exactly as Blastware does.
-
-That rule was chosen by evidence, not assumption: of the four candidates tried
-against the 9-24-26 capture's four data-carrying write frames, it is the only
-one under which all four checksums validate.  See
-`docs/micromate_protocol_reference.md` → *The write path*.
-
-Usage
------
-    python scratch/mm_frame_parse.py <capture-dir>
-    python scratch/mm_frame_parse.py <raw_bw.bin> <raw_s3.bin>
-    python scratch/mm_frame_parse.py <capture-dir> --dump 0x71
-"""
-
-from __future__ import annotations
-
-import argparse
-import sys
-from pathlib import Path
-
-DLE, STX, ETX, ACK = 0x10, 0x02, 0x03, 0x41
-
-# Request SUB -> short name.  Series III names where they carry over; the
-# Series IV additions are marked.
-SUBNAME = {
-    0x01: "DEVICE_INFO",
-    0x06: "STORAGE_RANGE",
-    0x08: "EVENT_INDEX",
-    0x0A: "WAVEFORM_HDR",
-    0x0C: "WAVEFORM_REC",
-    0x15: "SERIAL",
-    0x1A: "COMPLIANCE_CFG",
-    0x1C: "MONITOR_STATUS",
-    0x1E: "EVENT_HDR",
-    0x2C: "CALLHOME_CFG",
-    0x2E: "TRIGGER_CFG_READ",     # Series IV
-    0x3E: "OPERATOR",
-    0x41: "SETUP_NAME_READ",      # Series IV
-    0x5A: "BULK_DOWNLOAD",
-    0x5B: "POLL",
-    0x68: "EVENT_INDEX_WRITE",
-    0x69: "WAVEFORM_WRITE",
-    0x71: "COMPLIANCE_WRITE",
-    0x72: "CONFIRM_A",
-    0x73: "CONFIRM_B",
-    0x74: "CONFIRM_C",
-    0x82: "TRIGGER_WRITE",
-    0x83: "TRIGGER_CONFIRM",
-    0xDA: "SETUP_FILE_DECL",      # Series IV — names the target .MMB
-    0xFE: "FULL_CFG",
-}
-
-
-def destuff(blob: bytes, start: int, *, is_request: bool) -> tuple[bytes, int, int]:
-    """Destuff one frame starting at `start`.
-
-    Returns (payload, checksum, index_of_terminating_ETX).  `payload` excludes
-    the trailing checksum byte.  A request frame opens `ACK STX 10 10`; a
-    response opens with a bare `STX`.
-    """
-    i = start + (2 if is_request else 1)
-    out = bytearray()
-    if is_request:
-        # The doubled BW_CMD is the one guaranteed stuffed byte.
-        if blob[i : i + 2] != bytes([DLE, DLE]):
-            raise ValueError(f"@0x{start:04x}: request does not open with 10 10")
-        out.append(DLE)
-        i += 2
-    while i < len(blob):
-        b = blob[i]
-        if b == DLE and i + 1 < len(blob):
-            out.append(blob[i + 1])
-            i += 2
-            continue
-        if b == ETX:
-            break
-        out.append(b)
-        i += 1
-    if len(out) < 2:
-        raise ValueError(f"@0x{start:04x}: frame too short")
-    return bytes(out[:-1]), out[-1], i
-
-
-def frames(blob: bytes, *, is_request: bool):
-    """Yield (offset, payload, chk, checksum_kind)."""
-    i, n = 0, len(blob)
-    while i < n:
-        if is_request:
-            if not (blob[i] == ACK and i + 1 < n and blob[i + 1] == STX):
-                i += 1
-                continue
-        elif blob[i] != STX:
-            i += 1
-            continue
-        try:
-            payload, chk, end = destuff(blob, i, is_request=is_request)
-        except ValueError:
-            i += 1
-            continue
-        sum8 = sum(payload) & 0xFF
-        dle_aware = (sum(b for b in payload if b != DLE) & 0xFF)
-        if sum8 == chk:
-            kind = "SUM8"
-        elif dle_aware == chk:
-            kind = "DLE-aware"
-        else:
-            kind = "BAD"
-        yield i, payload, chk, kind
-        i = end + 1
-
-
-def describe(payload: bytes, is_request: bool) -> str:
-    if len(payload) < 3:
-        return "??"
-    sub = payload[2]
-    if is_request:
-        return SUBNAME.get(sub, f"SUB_{sub:02X}")
-    req = 0xFF - sub
-    return "rsp<-" + SUBNAME.get(req, f"SUB_{req:02X}")
-
-
-def report(path: Path, *, is_request: bool, dump_sub: int | None) -> None:
-    blob = path.read_bytes()
-    side = "Thor" if is_request else "unit"
-    print(f"== {side:4}  {path.name}  ({len(blob)} bytes)")
-    n_bad = 0
-    for idx, (off, p, chk, kind) in enumerate(frames(blob, is_request=is_request)):
-        if kind == "BAD":
-            n_bad += 1
-        sub = p[2] if len(p) > 2 else -1
-        flags = p[1] if len(p) > 1 else -1
-        # Requests carry offset at payload[4:6]; responses page at [3:5].
-        word = int.from_bytes(p[4:6] if is_request else p[3:5], "big")
-        data = len(p) - 16 if is_request else max(len(p) - 5, 0)
-        print(
-            f"  [{idx:2}] @0x{off:04x} payload={len(p):5} data={data:5} "
-            f"flags=0x{flags:02x} SUB=0x{sub:02x} {describe(p, is_request):18} "
-            f"{'offset' if is_request else 'page'}=0x{word:04x} chk={kind}"
-        )
-        if dump_sub is not None and sub == dump_sub:
-            body = p[16:] if is_request else p[5:]
-            print(f"       ---- data ({len(body)} bytes) ----")
-            for o in range(0, len(body), 16):
-                chunk = body[o : o + 16]
-                txt = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk)
-                print(f"       {o:06x}  {chunk.hex(' '):<47}  |{txt}|")
-    print(f"  -- {idx + 1} frames, {n_bad} bad checksum\n")
-
-
-def main() -> int:
-    ap = argparse.ArgumentParser(description=__doc__,
-                                 formatter_class=argparse.RawDescriptionHelpFormatter)
-    ap.add_argument("paths", nargs="+",
-                    help="a capture directory, or raw_bw.bin and raw_s3.bin")
-    ap.add_argument("--dump", default=None,
-                    help="hex-dump the data section of this SUB (e.g. 0x71)")
-    args = ap.parse_args()
-
-    dump_sub = int(args.dump, 0) if args.dump else None
-
-    if len(args.paths) == 1 and Path(args.paths[0]).is_dir():
-        d = Path(args.paths[0])
-        bw = sorted(d.glob("raw_bw_*.bin"))
-        s3 = sorted(d.glob("raw_s3_*.bin"))
-        if not bw or not s3:
-            print(f"{d}: need one raw_bw_*.bin and one raw_s3_*.bin", file=sys.stderr)
-            return 2
-        pairs = [(bw[0], True), (s3[0], False)]
-    elif len(args.paths) == 2:
-        pairs = [(Path(args.paths[0]), True), (Path(args.paths[1]), False)]
-    else:
-        ap.error("pass a capture directory, or exactly two .bin files")
-
-    for path, is_request in pairs:
-        report(path, is_request=is_request, dump_sub=dump_sub)
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())

scratch/nonmotion_scan.py

@@ -1,91 +0,0 @@
-#!/usr/bin/env python3
-"""Detect NON-MOTION on a geophone channel: |mean| / peak.
-
-A geophone is a velocity sensor with no DC response, so its output over a
-record must integrate to ~zero — the ground does not relocate. Real motion
-therefore sits roughly half above and half below zero. Anything electrical —
-a charge-injection spike, a step, a parked pedestal — is one-sided.
-
-    mp   = |mean| / peak            ~0 for motion, ~1 for a pedestal
-    frac_neg = share of samples < 0  ~0.3-0.5 for motion, ~0 for a fault
-
-Why this beats the pre-trigger floor (`offset_scan3.py`): that detector's
-`spread <= 0.02` gate rejects any record whose floor is MOVING, which is
-exactly what an onset is — it discarded the one BE18438 record in which the
-ramp was visible. This test is indifferent to whether the fault is a spike,
-a ramp or a flat pedestal; none of them cross zero.
-
-⚠ Not a rediscovery of the retracted v1 detector. v1 scored only the
-largest-peak axis and used the mean as a BASELINE estimator, where the median
-was required. Here the mean is the signal itself, per channel, and that is
-what the physics licenses.
-"""
-from __future__ import annotations
-import argparse, csv, re, statistics, sys
-from concurrent.futures import ProcessPoolExecutor, as_completed
-from pathlib import Path
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-from minimateplus.event_file_io import read_blastware_file
-
-GEO=("Tran","Vert","Long"); K=10.0/32000.0
-_WAVE=re.compile(r"\.[A-Za-z0-9]{2}0[Ww]$"); _STEM=re.compile(r"^([B-Z])(\d{3})")
-_SER=re.compile(rb"[A-Z]{2}\d{3,6}")
-
-def serial_of(name, path=None):
-    m=_STEM.match(name)
-    if not m: return "?"
-    num=(ord(m.group(1))-ord("B"))*1000+int(m.group(2))
-    if path is not None:
-        try:
-            for s in _SER.findall(Path(path).read_bytes()):
-                s=s.decode()
-                if s[2:].lstrip("0")==str(num): return s
-        except Exception: pass
-    return f"BE{num}"
-
-def scan(ps):
-    import logging; logging.disable(logging.WARNING)
-    p=Path(ps)
-    try: ev=read_blastware_file(p)
-    except Exception: return None
-    s=ev.raw_samples or {}
-    if not all(s.get(c) for c in GEO): return None
-    ts=ev.timestamp
-    stamp=(f"{ts.year:04d}-{ts.month:02d}-{ts.day:02d}T"
-           f"{ts.hour:02d}:{ts.minute:02d}:{ts.second:02d}") if ts else ""
-    ser=serial_of(p.name,p); out=[]
-    for ch in GEO:
-        a=[x*K for x in s[ch]]
-        pk=max(abs(x) for x in a)
-        if pk<=0: continue
-        out.append({"serial":ser,"timestamp":stamp,"filename":p.name,"channel":ch,
-                    "peak":round(pk,4),
-                    "mean":round(statistics.fmean(a),4),
-                    "mp":round(abs(statistics.fmean(a))/pk,4),
-                    "frac_neg":round(sum(1 for x in a if x<0)/len(a),4),
-                    "n":len(a)})
-    return out
-
-COLS=["serial","timestamp","filename","channel","peak","mean","mp","frac_neg","n"]
-
-def main():
-    ap=argparse.ArgumentParser()
-    ap.add_argument("--dir",required=True); ap.add_argument("--out",required=True)
-    ap.add_argument("--jobs",type=int,default=4)
-    a=ap.parse_args()
-    seen=set(); files=[]
-    for q in sorted(Path(a.dir).rglob("*")):
-        if q.is_file() and _WAVE.search(q.name) and q.name not in seen:
-            seen.add(q.name); files.append(str(q))
-    print(f"unique waveform binaries: {len(files)}",flush=True)
-    rows=[]
-    with ProcessPoolExecutor(max_workers=a.jobs) as ex:
-        for i,f in enumerate(as_completed([ex.submit(scan,p) for p in files]),1):
-            r=f.result()
-            if r: rows.extend(r)
-            if i%1000==0: print(f"  {i}/{len(files)}",flush=True)
-    with open(a.out,"w",newline="") as fh:
-        w=csv.DictWriter(fh,fieldnames=COLS); w.writeheader(); w.writerows(rows)
-    print(f"\nwrote {a.out}  ({len(rows)} channel-rows)")
-
-if __name__=="__main__": main()

scratch/offset_candidates.csv

@@ -1,275 +0,0 @@
-event_id,serial,timestamp,filename,channel,offset_ips,peak_ips,mean_over_peak,trigger_level_ips,already_flagged_ft
-1e7a7808-034c-423b-80e9-d6788da80993,BE18438,2025-11-15T08:57:40,T438LBX4.W40W,Vert,0.2722,0.2930,0.929,0.2,0
-346e383f-c5a1-41d1-b972-3d652e599d39,BE18438,2025-11-15T09:13:15,T438LBX5.M30W,Vert,0.2747,0.2930,0.938,0.2,0
-45ef3023-22c5-44a8-a606-a0900bc17861,BE18438,2025-11-15T09:16:13,T438LBX5.R10W,Vert,0.2929,0.3027,0.967,0.2,0
-1cbcd2aa-ec55-4e2f-af42-7178f26cc6a1,BE18438,2025-11-15T09:19:29,T438LBX5.WH0W,Vert,0.2816,0.2979,0.945,0.2,0
-caf43634-4528-4fd5-9674-5f4f563661c0,BE18438,2025-11-15T09:22:32,T438LBX6.1K0W,Vert,0.2883,0.2930,0.984,0.2,0
-456ae646-c834-4e5c-82f1-7df18b3440a1,BE18438,2025-11-15T09:25:31,T438LBX6.6J0W,Vert,0.3189,0.3223,0.989,0.2,0
-afb91ba2-351b-42a0-b33f-acebd5f1829d,BE18438,2025-11-15T09:28:29,T438LBX6.BH0W,Vert,0.3057,0.3076,0.994,0.2,0
-39847146-3537-4300-943d-01c0c771e9cd,BE18438,2025-11-15T09:31:26,T438LBX6.GE0W,Vert,0.3225,0.3271,0.986,0.2,0
-e47335c1-fc3f-4c90-88c6-8edd0f296ce4,BE18438,2025-11-15T09:34:24,T438LBX6.LC0W,Vert,0.3275,0.3320,0.986,0.2,0
-25b6eda6-ba98-422f-8834-94b6cbe34971,BE18438,2025-11-15T09:37:22,T438LBX6.QA0W,Vert,0.3398,0.3418,0.994,0.2,0
-7af99377-e47f-4914-8e91-ac7f7f50b8c6,BE18438,2025-11-15T09:40:20,T438LBX6.V80W,Vert,0.3515,0.3564,0.986,0.2,0
-c4330605-c257-49ef-aede-20deb641e7d7,BE18438,2025-11-15T10:09:48,T438LBX8.8C0W,Vert,0.3605,0.3955,0.912,0.2,0
-b665caf9-64ee-4352-a391-60c1ddeebc1e,BE18438,2026-02-25T10:58:04,T438LH66.GS0W,Vert,0.1803,0.1953,0.923,0.2,0
-3b0c32c3-fd16-4b83-9917-942a56dea168,BE18438,2026-02-25T18:12:04,T438LH6Q.K40W,Vert,0.1869,0.1953,0.957,0.2,0
-ca66e601-d10f-4424-9c26-08a688e7b08c,BE18438,2026-02-25T18:17:16,T438LH6Q.SS0W,Vert,0.1870,0.1953,0.958,0.2,0
-31aa30ba-eb0a-49a9-98ca-c77d2cc31275,BE18438,2026-02-25T18:21:11,T438LH6Q.ZB0W,Vert,0.1868,0.1953,0.956,0.2,0
-fdb55a93-fc91-4fdf-a7f4-fdd1a6b48f59,BE18438,2026-02-25T18:26:27,T438LH6R.830W,Vert,0.1893,0.1953,0.969,0.2,0
-cbd718d7-2378-4d74-8656-842f229d19a8,BE18438,2026-02-25T18:33:25,T438LH6R.JP0W,Vert,0.1887,0.1953,0.966,0.2,0
-bfaf58ec-6163-47eb-9d2a-70da880bd948,BE18438,2026-02-25T18:37:48,T438LH6R.R00W,Vert,0.1872,0.1953,0.959,0.2,0
-ddaa530b-886e-4415-b93a-f4ebf4c4a882,BE18438,2026-02-25T18:59:54,T438LH6S.RU0W,Vert,0.1874,0.1953,0.959,0.2,0
-57f15284-ab69-48fc-9660-8c347c80c681,BE18438,2026-02-25T19:15:58,T438LH6T.IM0W,Vert,0.1873,0.1953,0.959,0.2,0
-03715718-7aa8-4e1d-9b0b-c2ab75fc1975,BE18438,2026-02-25T19:19:58,T438LH6T.PA0W,Vert,0.1891,0.1953,0.968,0.2,0
-9f921cd5-1bf9-4713-aef6-41e7f780eff1,BE18438,2026-02-25T19:24:02,T438LH6T.W20W,Vert,0.1871,0.1953,0.958,0.2,0
-44994949-58d4-44fa-bd78-ed75b8acc667,BE18438,2026-02-25T19:44:57,T438LH6U.UX0W,Vert,0.1877,0.1953,0.961,0.2,0
-c4be01c3-1001-4e17-aecd-d6d26dbb09f3,BE18438,2026-02-25T19:49:19,T438LH6V.270W,Vert,0.1884,0.1953,0.964,0.2,0
-3db004e6-ee34-43c5-ae8e-f22894ead5d0,BE18438,2026-02-25T20:08:22,T438LH6V.XY0W,Vert,0.1878,0.1953,0.962,0.2,0
-d276ae48-0434-41e5-af43-e0fb366acf11,BE18438,2026-02-25T20:11:33,T438LH6W.390W,Vert,0.1880,0.1953,0.963,0.2,0
-c027dc12-88fd-4a45-872b-a6a945f2008e,BE18438,2026-02-25T20:14:35,T438LH6W.8B0W,Vert,0.1874,0.1953,0.959,0.2,0
-9665bc3a-3d08-4852-b008-37ef8bfd0023,BE18438,2026-02-25T20:18:51,T438LH6W.FF0W,Vert,0.1884,0.1953,0.965,0.2,0
-5fb5a05e-d7e2-4268-9b85-cfe8d82aa6c9,BE18438,2026-02-25T20:29:50,T438LH6W.XQ0W,Vert,0.1890,0.1953,0.968,0.2,0
-57ec0b10-9324-4459-a07e-0c8532b8928f,BE18438,2026-02-25T20:35:39,T438LH6X.7F0W,Vert,0.1889,0.1953,0.967,0.2,0
-ac86f73f-1174-4fae-a206-ecdcb73e1ffc,BE18438,2026-02-25T20:39:12,T438LH6X.DC0W,Vert,0.1887,0.1953,0.966,0.2,0
-dcf25ddd-9cf0-41c1-b109-f3ef2e490ce5,BE18438,2026-02-25T20:43:41,T438LH6X.KT0W,Vert,0.1871,0.1953,0.958,0.2,0
-aa79bbdd-2392-4ef6-8a5c-873a9f72746d,BE18438,2026-02-25T20:46:51,T438LH6X.Q30W,Vert,0.1877,0.1953,0.961,0.2,0
-ad46d1ce-b417-4c8c-b029-bb3b008ba809,BE18438,2026-02-25T20:53:33,T438LH6Y.190W,Vert,0.1882,0.1953,0.964,0.2,0
-3d3a8f34-86a7-4d70-a7e8-fa768a723325,BE18438,2026-02-26T07:03:39,T438LH7Q.A30W,Vert,0.1809,0.1953,0.926,0.2,0
-7817525f-dc3d-4f5e-b2cb-5af9ca12ede8,BE18438,2026-02-26T07:09:42,T438LH7Q.K60W,Vert,0.1814,0.1953,0.929,0.2,0
-a2091302-0b34-46c1-ab83-174be2b59bcf,BE18438,2026-02-26T13:12:35,T438LH87.CZ0W,Vert,0.1896,0.1953,0.971,0.2,0
-e1c0c9de-a900-479f-9572-9675b13296a9,BE18438,2026-02-26T13:24:46,T438LH87.XA0W,Vert,0.2237,0.2588,0.864,0.2,0
-76a6dc6b-88e1-434b-a137-f5e9b63f248e,BE18438,2026-02-26T13:27:43,T438LH88.270W,Vert,0.3185,0.3223,0.988,0.2,0
-281b508b-7982-4668-86af-9b0b28be608a,BE18438,2026-02-26T13:30:36,T438LH88.700W,Vert,0.3263,0.3320,0.983,0.2,0
-68bbaf77-bb23-40fd-94f9-f928728d90dc,BE18438,2026-02-26T13:33:28,T438LH88.BS0W,Vert,0.3207,0.3271,0.980,0.2,0
-5c975640-6ff1-441a-aa11-41642f6a6b31,BE18438,2026-02-26T13:36:21,T438LH88.GL0W,Vert,0.3261,0.3320,0.982,0.2,0
-dea248ac-1124-4bba-b054-953f86c852b2,BE18438,2026-02-26T13:40:34,T438LH88.NM0W,Vert,0.3271,0.3320,0.985,0.2,0
-4e463201-f008-41ad-9c52-11fcc3d93095,BE18438,2026-02-26T13:43:29,T438LH88.SH0W,Vert,0.3357,0.3418,0.982,0.2,0
-507262c5-d58f-4a05-a6a3-a6cbb5b4de20,BE18438,2026-02-26T13:46:24,T438LH88.XC0W,Vert,0.3372,0.3418,0.987,0.2,0
-4c2d8ee8-842f-4e44-9e03-ff34d26e4fdb,BE18438,2026-02-26T13:49:17,T438LH89.250W,Vert,0.3253,0.3320,0.980,0.2,0
-b184b112-603f-44f4-ae97-087493fbd8ce,BE18438,2026-02-26T13:52:10,T438LH89.6Y0W,Vert,0.3321,0.3369,0.986,0.2,0
-2ff1a552-5a2e-4396-9083-b9c8d334eed7,BE18438,2026-02-26T13:55:04,T438LH89.BS0W,Vert,0.3378,0.3418,0.988,0.2,0
-928e21a7-1990-4792-803e-58014d8140ce,BE18438,2026-02-26T13:57:57,T438LH89.GL0W,Vert,0.3341,0.3369,0.992,0.2,0
-f521b5db-6100-41e6-954e-0abdbf9660c1,BE18438,2026-02-26T14:00:51,T438LH89.LF0W,Vert,0.3421,0.3467,0.987,0.2,0
-efba962f-e511-41ef-9fc5-313f202a9d96,BE18438,2026-02-26T14:03:46,T438LH89.QA0W,Vert,0.3506,0.3564,0.984,0.2,0
-0b51e273-8d10-4494-bda6-8c75eaeec66f,BE18438,2026-02-26T14:06:41,T438LH89.V50W,Vert,0.3448,0.3516,0.981,0.2,0
-6d46e4ee-1647-4169-b6b2-33da32240eb9,BE18438,2026-02-26T14:09:35,T438LH89.ZZ0W,Vert,0.3463,0.3516,0.985,0.2,0
-1ca294cd-4dc2-41fb-9374-e277d4550761,BE18438,2026-02-26T14:12:28,T438LH8A.4S0W,Vert,0.3442,0.3467,0.993,0.2,0
-7b683dcf-1aed-4d9b-be1e-b10f92374398,BE18438,2026-02-26T14:15:23,T438LH8A.9N0W,Vert,0.3486,0.3516,0.992,0.2,0
-2b5cb8e7-c947-4914-bcb5-504f74603547,BE18438,2026-02-26T14:18:21,T438LH8A.EL0W,Vert,0.3519,0.3613,0.974,0.2,0
-5ea3a6bf-10de-4911-aedc-66daecf86b76,BE18438,2026-02-26T14:21:16,T438LH8A.JG0W,Vert,0.3460,0.3564,0.971,0.2,0
-d93be7f5-39ec-45fd-9949-fa531f2f9d34,BE18438,2026-02-26T14:24:13,T438LH8A.OD0W,Vert,0.3457,0.3516,0.983,0.2,0
-cfa113f2-8cee-4a1d-b1e7-0e936fa771e2,BE18438,2026-02-26T14:28:21,T438LH8A.V90W,Vert,0.3485,0.3564,0.978,0.2,0
-46d322ac-38cb-4c99-a4c2-9e9d7ad06171,BE18438,2026-02-26T14:31:17,T438LH8B.050W,Vert,0.3367,0.3467,0.971,0.2,0
-42500a83-bd67-4deb-8bb2-963ee2b8a7ce,BE18438,2026-02-26T14:34:11,T438LH8B.4Z0W,Vert,0.3365,0.3516,0.957,0.2,0
-3ccf6bda-2e88-473c-951d-9ea86d0d9363,BE18438,2026-02-26T14:37:05,T438LH8B.9T0W,Vert,0.3472,0.3564,0.974,0.2,0
-9050487d-d238-4631-b8e1-5a693a026542,BE18438,2026-02-26T14:39:58,T438LH8B.EM0W,Vert,0.3420,0.3564,0.959,0.2,0
-e62989e4-a326-4bae-8d47-a8becee96462,BE18438,2026-02-26T14:42:52,T438LH8B.JG0W,Vert,0.3430,0.3564,0.962,0.2,0
-f0e30fab-dc57-4189-8439-48df17007a2c,BE18438,2026-02-26T14:45:46,T438LH8B.OA0W,Vert,0.3389,0.3516,0.964,0.2,0
-03e1ca09-78df-4c6f-9915-0b6e4f2141d2,BE18438,2026-02-26T14:48:43,T438LH8B.T70W,Vert,0.3449,0.3564,0.968,0.2,0
-945d39f7-1b98-4a1f-80a8-6a18fd9df25f,BE18438,2026-02-26T14:51:37,T438LH8B.Y10W,Vert,0.3522,0.3613,0.975,0.2,0
-d27eb2e3-e313-4125-98f4-f9d6d3737d28,BE18438,2026-02-26T14:54:32,T438LH8C.2W0W,Vert,0.3502,0.3613,0.969,0.2,0
-916901cb-ebea-441e-a2a1-40f4e49f32f4,BE18438,2026-02-26T14:57:25,T438LH8C.7P0W,Vert,0.3524,0.3662,0.962,0.2,0
-5d8d7e29-fdbb-4fa3-bd25-506c9352a354,BE18438,2026-02-26T15:01:35,T438LH8C.EN0W,Vert,0.3530,0.3613,0.977,0.2,0
-c6c7a3af-b03b-43dc-97de-024f8e09b62f,BE18438,2026-02-26T15:04:27,T438LH8C.JF0W,Vert,0.3514,0.3613,0.973,0.2,0
-b6856898-3d10-4967-9577-7f17902f77ea,BE18438,2026-02-26T15:11:34,T438LH8C.VA0W,Vert,0.3566,0.3613,0.987,0.2,0
-7a4802f6-621e-4230-8d8c-d7f0c9a65bf2,BE18438,2026-02-26T15:14:24,T438LH8D.000W,Vert,0.3594,0.3711,0.968,0.2,0
-79f38b7e-ec56-417e-943a-8c944252c2b8,BE18438,2026-02-26T15:18:32,T438LH8D.6W0W,Vert,0.3550,0.3613,0.983,0.2,0
-1b4190c4-0d54-48ba-8ff5-8ddb0eb91e50,BE18438,2026-02-26T15:22:22,T438LH8D.DA0W,Vert,0.3571,0.3613,0.988,0.2,0
-de57b5ae-a3a1-4ede-9d2b-3e87bfb3fd19,BE9558,2026-04-14T11:16:32,K558LJN3.BK0W,Tran,0.3448,0.3662,0.942,0.2,0
-43afeaf5-02ec-41f6-9e23-0c9772821ed4,BE9558,2026-04-14T11:27:15,K558LJN3.TF0W,Tran,0.3094,0.3223,0.960,0.2,0
-8123c0ef-84c9-4f6c-8d82-9dc32e2e470d,BE9558,2026-04-14T14:45:30,K558LJNC.ZU0W,Tran,0.2721,0.3564,0.763,0.2,0
-8c787af3-596e-411b-9e10-29485fa5114f,BE9558,2026-04-29T16:18:47,K558LKF9.BB0W,Tran,0.2943,0.3027,0.972,0.2,0
-431928ff-b4c4-4caa-b933-acc917f3717c,BE9558,2026-05-04T15:02:30,K558LKOF.460W,Tran,0.4364,0.5225,0.835,0.2,0
-6c3c07f8-c36a-4493-acce-7441c9d22cec,BE9558,2026-05-15T08:50:06,K558LL8B.7I0W,Long,0.2892,0.2930,0.987,0.2,0
-13c268df-b652-422e-a642-6453f6a314aa,BE9558,2026-05-15T10:18:34,K558LL8F.AY0W,Long,0.2907,0.2979,0.976,0.2,0
-9b0d0871-8810-467d-806a-5bbfa4e667fe,BE9558,2026-05-15T15:52:00,K558LL8U.QO0W,Long,0.2659,0.2979,0.893,0.2,0
-092640f9-a944-48b7-b872-364d75c2c5e7,BE9558,2026-05-15T16:13:12,K558LL8V.Q00W,Long,0.2428,0.2979,0.815,0.2,0
-0573741a-96ab-4b36-af7a-b5bc11a79009,BE9558,2026-05-16T03:23:26,K558LL9Q.R20W,Long,0.2861,0.2930,0.976,0.2,0
-321f03ea-6696-47de-ad46-2be43d4d6cae,BE9558,2026-05-16T03:30:49,K558LL9R.3D0W,Long,0.2886,0.2930,0.985,0.2,0
-cc423a6a-3e3c-466a-b39c-6146ca7f34c8,BE9558,2026-05-16T03:33:55,K558LL9R.8J0W,Long,0.2880,0.2881,1.000,0.2,0
-efa52a14-cc68-4bea-aab4-4ee3a3cbff3a,BE9558,2026-05-16T03:36:55,K558LL9R.DJ0W,Long,0.2890,0.2930,0.986,0.2,0
-92895bfe-122f-4cdc-bd3d-40609633d278,BE9558,2026-05-16T03:43:41,K558LL9R.OT0W,Long,0.2880,0.2881,1.000,0.2,0
-2a4c81db-2307-44e5-9b27-d5154d98e38d,BE9558,2026-05-16T03:46:36,K558LL9R.TO0W,Long,0.2921,0.2979,0.981,0.2,0
-656b0fdc-e275-4c9c-8936-9eee267ea04e,BE9558,2026-05-16T03:49:28,K558LL9R.YG0W,Long,0.2966,0.2979,0.996,0.2,0
-3b9989ae-e0f2-4652-9724-c145603543d2,BE9558,2026-05-16T03:52:29,K558LL9S.3H0W,Long,0.2880,0.2930,0.983,0.2,0
-837d0d84-d609-4f4a-b542-b7e2b064ea22,BE9558,2026-05-16T03:56:32,K558LL9S.A80W,Long,0.2880,0.2930,0.983,0.2,0
-affcb008-ab99-4bc5-8f35-83dbe620499d,BE9558,2026-05-16T04:02:50,K558LL9S.KQ0W,Long,0.2896,0.2930,0.989,0.2,0
-e1f1c6d2-545c-451f-9018-714d52e20a05,BE9558,2026-05-16T04:05:47,K558LL9S.PN0W,Long,0.3024,0.3076,0.983,0.2,0
-93ef80f1-7e83-4703-a0f9-89bbfeeae6c3,BE9558,2026-05-16T04:08:41,K558LL9S.UH0W,Long,0.2976,0.2979,0.999,0.2,0
-5c118ca9-fc37-4080-876d-b3f57ce0b121,BE9558,2026-05-16T04:11:42,K558LL9S.ZI0W,Long,0.2881,0.2930,0.983,0.2,0
-d98b2eaa-7763-4c78-88fb-c21195cb9978,BE9558,2026-05-16T04:14:38,K558LL9T.4E0W,Long,0.2898,0.2930,0.989,0.2,0
-6a146a01-0d02-4413-b231-ecb4a6fde776,BE9558,2026-05-16T04:17:35,K558LL9T.9B0W,Long,0.2880,0.2930,0.983,0.2,0
-f6587a55-27e3-454f-997e-57e7516a296c,BE9558,2026-05-16T04:20:32,K558LL9T.E80W,Long,0.3046,0.3076,0.990,0.2,0
-1fb48250-2214-41e6-92cb-362aff1ba62b,BE9558,2026-05-16T04:26:05,K558LL9T.NH0W,Long,0.2876,0.2930,0.982,0.2,0
-2f4f3d06-1189-4e28-a1ac-5454ea586a0c,BE9558,2026-05-16T04:59:47,K558LL9V.7N0W,Long,0.2961,0.3027,0.978,0.2,0
-62b7bdab-b744-41e4-8d0e-c9141b9d4821,BE9558,2026-05-16T05:02:46,K558LL9V.CM0W,Long,0.2903,0.2930,0.991,0.2,0
-cadcb6dc-74c0-48b5-974c-87bb7a57816c,BE9558,2026-05-16T05:05:43,K558LL9V.HJ0W,Long,0.3090,0.3125,0.989,0.2,0
-a0235177-be5a-4f11-b741-c65744698e5b,BE9558,2026-05-16T05:08:35,K558LL9V.MB0W,Long,0.3167,0.3223,0.983,0.2,0
-1968c2f6-f52c-4544-a0ed-f79b91ce0da0,BE9558,2026-05-16T05:11:28,K558LL9V.R40W,Long,0.3446,0.3516,0.980,0.2,0
-caa06b94-8813-4949-b47b-61ffb16e61c4,BE9558,2026-05-16T05:14:22,K558LL9V.VY0W,Long,0.3318,0.3320,0.999,0.2,0
-fdaa8101-f033-47e6-9c65-abd76dea1870,BE9558,2026-05-16T05:17:16,K558LL9W.0S0W,Long,0.3407,0.3418,0.997,0.2,0
-29bd16bd-81b9-4ccd-a981-f1a80858536c,BE9558,2026-05-16T05:23:05,K558LL9W.AH0W,Long,0.3450,0.3516,0.981,0.2,0
-1f8bc862-d5fd-4cce-abc0-e2bd8c2f11fa,BE9558,2026-05-16T05:26:00,K558LL9W.FC0W,Long,0.3417,0.3467,0.986,0.2,0
-20220068-d498-4fca-9a2a-a7fd2ee52ab1,BE9558,2026-05-16T05:28:53,K558LL9W.K50W,Long,0.3516,0.3564,0.986,0.2,0
-9816a261-3118-417a-8334-ace36687ad8f,BE9558,2026-05-16T05:31:47,K558LL9W.OZ0W,Long,0.3416,0.3467,0.985,0.2,0
-5d2c76d7-543d-4557-9caf-996a69ab00bd,BE9558,2026-05-16T05:34:42,K558LL9W.TU0W,Long,0.3537,0.3564,0.992,0.2,0
-c7b5ae00-52e7-4920-aa27-49d3b3daff92,BE9558,2026-05-16T05:37:36,K558LL9W.YO0W,Long,0.3517,0.3564,0.987,0.2,0
-07db8fd8-ccef-4a5a-9d97-9f318f73a478,BE9558,2026-05-16T05:40:31,K558LL9X.3J0W,Long,0.3574,0.3613,0.989,0.2,0
-790fcff9-e266-4a12-87f0-a072990d2533,BE9558,2026-05-16T05:43:26,K558LL9X.8E0W,Long,0.3650,0.3662,0.997,0.2,0
-f4245f61-1dde-43dc-a121-98e102f6a193,BE9558,2026-05-16T05:50:28,K558LL9X.K40W,Long,0.3856,0.3906,0.987,0.2,0
-5a3bce7b-358e-4260-876f-3f853163d7cf,BE9558,2026-05-16T05:53:22,K558LL9X.OY0W,Long,0.3847,0.3857,0.997,0.2,0
-37c2b4b1-7a1b-43e7-b513-2c16f582109c,BE9558,2026-05-16T05:56:15,K558LL9X.TR0W,Long,0.3906,0.3955,0.988,0.2,0
-57fde109-1d8a-45a1-b398-5bd28c3e47f1,BE9558,2026-05-16T05:59:10,K558LL9X.YM0W,Long,0.4000,0.4004,0.999,0.2,0
-1e41ab40-93be-405c-a464-f7a57524d3b3,BE9558,2026-05-16T06:02:02,K558LL9Y.3E0W,Long,0.3825,0.3857,0.992,0.2,0
-e9d61eb5-fae8-4902-8fab-4828ffa8e6da,BE9558,2026-05-16T06:04:57,K558LL9Y.890W,Long,0.3802,0.3809,0.998,0.2,0
-c0e964e0-7788-4e41-bc41-6e479c3dc81b,BE9558,2026-05-16T06:07:51,K558LL9Y.D30W,Long,0.3954,0.4004,0.987,0.2,0
-0b2df8e5-1689-4fe6-abad-30970ba10aa6,BE9558,2026-05-16T06:16:06,K558LL9Y.QU0W,Long,0.4051,0.4053,1.000,0.2,0
-2bc6ce73-464b-469c-9e8f-3bf6f6cfae1d,BE9558,2026-05-16T06:21:51,K558LL9Z.0F0W,Long,0.3910,0.3955,0.989,0.2,0
-b8603ee1-616a-451e-a2cd-61f8f2634cd0,BE9558,2026-05-16T06:24:44,K558LL9Z.580W,Long,0.3969,0.4004,0.991,0.2,0
-1aecf582-66a2-4ebb-80f9-04b3aeba331a,BE9558,2026-05-16T06:27:39,K558LL9Z.A30W,Long,0.3915,0.3955,0.990,0.2,0
-885aedfd-7a1a-4120-8176-7c344dd3d8fe,BE9558,2026-05-16T06:30:30,K558LL9Z.EU0W,Long,0.4025,0.4053,0.993,0.2,0
-ec15930f-47f5-4752-afc2-427ee156ac95,BE9558,2026-05-16T06:33:25,K558LL9Z.JP0W,Long,0.4060,0.4102,0.990,0.2,0
-5ca23f47-2a38-4a54-99dd-041287aa9510,BE9558,2026-05-16T06:36:17,K558LL9Z.OH0W,Long,0.4054,0.4102,0.988,0.2,0
-03f371cf-81bc-4681-80f0-ef6fa437da85,BE9558,2026-05-16T06:39:09,K558LL9Z.T90W,Long,0.4052,0.4102,0.988,0.2,0
-ed386f54-140d-45e8-ad51-6585a58b4375,BE9558,2026-05-16T06:44:54,K558LLA0.2U0W,Long,0.3993,0.4004,0.997,0.2,0
-579f1c3f-4c1e-42a3-aa89-69aa93514033,BE9558,2026-05-16T06:47:47,K558LLA0.7N0W,Long,0.4001,0.4053,0.987,0.2,0
-57024de6-07ea-41f4-b916-c6adc3f51636,BE9558,2026-05-16T06:50:40,K558LLA0.CG0W,Long,0.4103,0.4150,0.989,0.2,0
-36bf5adf-afb5-4a5e-a106-1afbd1f56c7b,BE9558,2026-05-16T06:54:49,K558LLA0.JD0W,Long,0.4069,0.4150,0.980,0.2,0
-3e2f831b-5228-46ff-9acb-75d8c86f8bc1,BE9558,2026-05-16T06:57:43,K558LLA0.O70W,Long,0.4108,0.4150,0.990,0.2,0
-1775fa20-568b-4294-b9ca-561201a66c5b,BE9558,2026-05-16T07:00:38,K558LLA0.T20W,Long,0.3967,0.4004,0.991,0.2,0
-fd391469-03f9-4d8d-bccd-3af50c91e5b7,BE9558,2026-05-16T07:03:33,K558LLA0.XX0W,Long,0.3927,0.3955,0.993,0.2,0
-02b52ef6-b707-4dcb-b58f-43da02698832,BE9558,2026-05-16T07:06:27,K558LLA1.2R0W,Long,0.3824,0.3857,0.991,0.2,0
-8d45333f-e101-4249-9e27-913c583a0a8d,BE9558,2026-05-16T07:09:21,K558LLA1.7L0W,Long,0.4085,0.4150,0.984,0.2,0
-361d11ce-f42a-4d04-8270-ee3771fe4f09,BE9558,2026-05-16T07:12:14,K558LLA1.CE0W,Long,0.4017,0.4053,0.991,0.2,0
-59218125-ed5d-4457-aa8b-35ba03441363,BE9558,2026-05-16T07:15:07,K558LLA1.H70W,Long,0.3934,0.3955,0.995,0.2,0
-a603e55a-2ad6-4fc9-8c8c-0dc7be95c3e7,BE9558,2026-05-16T07:23:43,K558LLA1.VJ0W,Long,0.3806,0.3857,0.987,0.2,0
-c8d07902-93ea-4b03-9e69-9ad6ed5bffb1,BE9558,2026-05-16T07:26:36,K558LLA2.0C0W,Long,0.3807,0.3857,0.987,0.2,0
-4c46c0c7-42f0-4648-827e-1995f733b18f,BE9558,2026-05-16T07:30:44,K558LLA2.780W,Long,0.3667,0.3711,0.988,0.2,0
-04759a00-ba3e-4959-a4cb-be8fc1f3004e,BE9558,2026-05-16T07:36:28,K558LLA2.GS0W,Long,0.3514,0.3516,1.000,0.2,0
-0b6b0856-bd87-4a07-a5b4-f13d99e04a12,BE9558,2026-05-16T07:39:20,K558LLA2.LK0W,Long,0.3512,0.3516,0.999,0.2,0
-3d81a8bd-464d-4050-8a76-25e8373d1ddb,BE9558,2026-05-16T07:42:12,K558LLA2.QC0W,Long,0.3350,0.3369,0.994,0.2,0
-b8b9c0c4-c388-4acb-955a-be4ac356afce,BE9558,2026-05-16T07:45:04,K558LLA2.V40W,Long,0.3563,0.3564,1.000,0.2,0
-cb3d142a-fb84-4a21-a298-f61bb869b036,BE9558,2026-05-16T07:49:13,K558LLA3.210W,Long,0.3663,0.3711,0.987,0.2,0
-7401b19b-bc9d-43f0-b395-f4323aca5c72,BE9558,2026-05-16T07:52:06,K558LLA3.6U0W,Long,0.3612,0.3662,0.986,0.2,0
-2236df76-a9e6-401f-9176-ad565242f2cf,BE9558,2026-05-16T07:57:52,K558LLA3.GG0W,Long,0.3366,0.3369,0.999,0.2,0
-a645846c-cae4-4aa5-a2e6-85166bd4dd62,BE9558,2026-05-16T08:00:44,K558LLA3.L80W,Long,0.3265,0.3271,0.998,0.2,0
-7212bd33-99a9-4659-9f0b-c98cc8d1e8bc,BE9558,2026-05-16T08:04:51,K558LLA3.S30W,Long,0.3418,0.3467,0.986,0.2,0
-2cc4bf33-c63a-4c65-b0e2-5e61a9146a5d,BE9558,2026-05-16T08:07:43,K558LLA3.WV0W,Long,0.3405,0.3418,0.996,0.2,0
-4fedb486-5f1f-4623-a01f-083e829f0565,BE9558,2026-05-16T08:10:35,K558LLA4.1N0W,Long,0.3515,0.3564,0.986,0.2,0
-58aff0ba-0f9b-45e1-9bbe-a990d16192ae,BE9558,2026-05-16T08:13:27,K558LLA4.6F0W,Long,0.3513,0.3516,0.999,0.2,0
-e56d87e5-e1d4-4947-b6a5-3bbd2773c54a,BE9558,2026-05-16T08:20:03,K558LLA4.HF0W,Long,0.3564,0.3613,0.986,0.2,0
-0e96e61e-6a21-4e07-8e09-72d09dbfa1e6,BE9558,2026-05-16T08:25:21,K558LLA4.Q90W,Long,0.3565,0.3613,0.987,0.2,0
-19611f28-2da8-4362-a58e-dbed2e061379,BE9558,2026-05-16T08:34:19,K558LLA5.570W,Long,0.3647,0.3662,0.996,0.2,0
-bd580c1d-44d4-41fd-9f83-f462cda4099a,BE9558,2026-05-16T08:37:15,K558LLA5.A30W,Long,0.3536,0.3564,0.992,0.2,0
-2782df3b-3cb4-471a-a6f5-bbc4903a8ab9,BE9558,2026-05-16T08:44:00,K558LLA5.LC0W,Long,0.3612,0.3662,0.986,0.2,0
-d15a08f3-1c7c-4f80-8d9b-f5008c58c3d9,BE9558,2026-05-16T08:49:18,K558LLA5.U60W,Long,0.3610,0.3613,0.999,0.2,0
-03a10a6a-193a-476c-9543-f63818f69417,BE9558,2026-05-16T08:54:36,K558LLA6.300W,Long,0.3515,0.3516,1.000,0.2,0
-1c271856-8100-4c0d-bc8c-9357b47d7626,BE9558,2026-05-16T08:59:54,K558LLA6.BU0W,Long,0.3617,0.3662,0.988,0.2,0
-a85f125e-bfa2-42be-b877-2604b8d068c8,BE9558,2026-05-16T09:05:17,K558LLA6.KT0W,Long,0.3510,0.3516,0.998,0.2,0
-dde9aa71-d2e3-41ac-8388-f2bbddb6dae6,BE9558,2026-05-16T09:08:11,K558LLA6.PN0W,Long,0.3447,0.3467,0.994,0.2,0
-2f3f2be2-0ee3-422e-b3ad-83fb87512a2b,BE9558,2026-05-16T09:14:39,K558LLA7.0F0W,Long,0.3580,0.3613,0.991,0.2,0
-357a3034-8a9c-485e-8cb6-f526dd2baf6f,BE9558,2026-05-16T09:29:11,K558LLA7.ON0W,Long,0.3708,0.3760,0.986,0.2,0
-f8ad0274-8e6f-4714-82ff-d229e60f1442,BE9558,2026-05-16T09:32:06,K558LLA7.TI0W,Long,0.3567,0.3613,0.987,0.2,0
-42bb007a-2726-401a-b085-c25151b0dc24,BE9558,2026-05-16T09:35:02,K558LLA7.YE0W,Long,0.3575,0.3613,0.989,0.2,0
-c99bfc4a-db02-4bea-bea4-943c67485dba,BE9558,2026-05-16T09:37:58,K558LLA8.3A0W,Long,0.3527,0.3564,0.989,0.2,0
-8fc17de0-73f7-48bd-8d93-5451dce47e8e,BE9558,2026-05-16T09:40:52,K558LLA8.840W,Long,0.3613,0.3662,0.987,0.2,0
-bbee5276-03dd-4ca4-a7b9-aec02baac7e8,BE9558,2026-05-16T09:48:13,K558LLA8.KD0W,Long,0.3590,0.3613,0.993,0.2,0
-8dcdee10-db01-4224-9506-4181c3105d6f,BE9558,2026-05-16T09:53:31,K558LLA8.T70W,Long,0.3588,0.3613,0.993,0.2,0
-48c8be72-64d9-4da3-95b5-bbec2c5e1dbd,BE9558,2026-05-16T09:58:49,K558LLA9.210W,Long,0.3582,0.3662,0.978,0.2,0
-0da20919-7ce9-4feb-8ae9-12d9bb40c2b5,BE9558,2026-05-16T10:04:07,K558LLA9.AV0W,Long,0.3506,0.3516,0.997,0.2,0
-c2f7e9bc-8b93-42c3-96d3-e3af217dde35,BE9558,2026-05-16T10:09:25,K558LLA9.JP0W,Long,0.3506,0.3516,0.997,0.2,0
-522a0e42-d443-4447-bc71-af1832c6604e,BE9558,2026-05-16T10:14:43,K558LLA9.SJ0W,Long,0.3624,0.3662,0.990,0.2,0
-67f7e732-3c0b-439f-87c5-3c478a381418,BE9558,2026-05-16T10:20:01,K558LLAA.1D0W,Long,0.3494,0.3516,0.994,0.2,0
-251767e8-56f7-4091-a001-364d8e75c15c,BE9558,2026-05-16T10:25:19,K558LLAA.A70W,Long,0.3463,0.3516,0.985,0.2,0
-9457239d-a4ca-4476-b385-5e01fc8f6420,BE9558,2026-05-16T10:30:37,K558LLAA.J10W,Long,0.3436,0.3467,0.991,0.2,0
-43718d7c-9353-49e1-89b6-0841aeb1b276,BE9558,2026-05-16T10:35:55,K558LLAA.RV0W,Long,0.3452,0.3516,0.982,0.2,0
-f54ba557-ddfe-41d3-8f4c-8bbec9a3783c,BE9558,2026-05-16T10:41:13,K558LLAB.0P0W,Long,0.3419,0.3467,0.986,0.2,0
-dc26c758-ad99-49c1-b518-931368cdf8cf,BE9558,2026-05-16T10:46:31,K558LLAB.9J0W,Long,0.3463,0.3516,0.985,0.2,0
-10917eb2-d151-461c-a8cb-e9660da5a5b0,BE9558,2026-05-16T10:51:49,K558LLAB.ID0W,Long,0.3520,0.3564,0.987,0.2,0
-4938a5e6-d53b-40ee-be2b-06ec0e60398c,BE9558,2026-05-16T10:57:07,K558LLAB.R70W,Long,0.3541,0.3564,0.993,0.2,0
-0711b51d-55bd-48be-b51e-2aad67107a60,BE9558,2026-05-16T11:13:01,K558LLAC.HP0W,Long,0.3661,0.3711,0.986,0.2,0
-065564b6-aa49-40e7-9ea6-c75defdc60fe,BE9558,2026-05-16T11:18:19,K558LLAC.QJ0W,Long,0.3622,0.3662,0.989,0.2,0
-ffb6b230-1d70-4e0e-ba30-e945339a122a,BE9558,2026-05-16T11:23:37,K558LLAC.ZD0W,Long,0.3711,0.3760,0.987,0.2,0
-52b4da82-c1e0-4a8a-8365-51b090787005,BE9558,2026-05-16T11:28:55,K558LLAD.870W,Long,0.3644,0.3662,0.995,0.2,0
-cdbff3d8-fbfe-4bea-8bd6-c898800e81ba,BE9558,2026-05-16T11:39:31,K558LLAD.PV0W,Long,0.3609,0.3662,0.986,0.2,0
-3fb0bc30-1463-4cfc-9e7f-386c0eb2a34a,BE9558,2026-05-16T11:44:49,K558LLAD.YP0W,Long,0.3553,0.3564,0.997,0.2,0
-ba1f5ec8-aef7-4a9e-ac1f-276cf016e285,BE9558,2026-05-16T11:50:07,K558LLAE.7J0W,Long,0.3479,0.3516,0.990,0.2,0
-b1dd4df7-34a9-4512-a903-f8242b236fd2,BE9558,2026-05-16T11:55:25,K558LLAE.GD0W,Long,0.3427,0.3467,0.988,0.2,0
-1f23aa28-c026-4bc7-b3d1-7c78fd76a673,BE9558,2026-05-16T12:00:43,K558LLAE.P70W,Long,0.3454,0.3467,0.996,0.2,0
-0d91f54d-b20d-4a50-b7dd-d071c39a0690,BE9558,2026-05-16T12:06:01,K558LLAE.Y10W,Long,0.3431,0.3467,0.990,0.2,0
-12ccdecd-5fd9-4465-9813-3d6006ec32f3,BE9558,2026-05-16T12:11:19,K558LLAF.6V0W,Long,0.3433,0.3467,0.990,0.2,0
-a7024f1a-f071-48ab-820f-99bdf87366b5,BE9558,2026-05-16T12:16:37,K558LLAF.FP0W,Long,0.3361,0.3369,0.998,0.2,0
-de0fa439-30b8-4607-a3c3-7e17191e4624,BE9558,2026-05-16T12:21:55,K558LLAF.OJ0W,Long,0.3274,0.3320,0.986,0.2,0
-f9374ec3-05e5-477f-8e1a-0c862fed2a50,BE9558,2026-05-16T12:27:13,K558LLAF.XD0W,Long,0.3449,0.3467,0.995,0.2,0
-424cb275-3743-4174-95f0-0ca1205b374c,BE9558,2026-05-16T12:32:31,K558LLAG.670W,Long,0.3370,0.3418,0.986,0.2,0
-9accde13-cefa-41b8-bd0a-cef2ffed59b0,BE9558,2026-05-16T12:37:49,K558LLAG.F10W,Long,0.3465,0.3467,0.999,0.2,0
-4665be38-1956-4b14-b931-068ba9af5c5e,BE9558,2026-05-16T12:48:25,K558LLAG.WP0W,Long,0.3453,0.3516,0.982,0.2,0
-1aedf624-1a99-43e1-a8f2-f9d358b97f61,BE9558,2026-05-16T12:53:43,K558LLAH.5J0W,Long,0.3366,0.3418,0.985,0.2,0
-c15e1799-91d7-425e-ad17-2ba3d170d81e,BE9558,2026-05-16T13:04:19,K558LLAH.N70W,Long,0.3348,0.3369,0.994,0.2,0
-72d4c4ea-19ca-4cc6-9b04-7a2b70122c67,BE9558,2026-05-16T13:09:37,K558LLAH.W10W,Long,0.3272,0.3320,0.986,0.2,0
-b4ebf2c9-1f6d-4594-9810-40479e87c0a5,BE9558,2026-05-16T13:14:55,K558LLAI.4V0W,Long,0.3297,0.3320,0.993,0.2,0
-bb297e9e-548f-4dae-8b1a-0f161af2e766,BE9558,2026-05-16T13:20:13,K558LLAI.DP0W,Long,0.3272,0.3320,0.986,0.2,0
-87562c24-63db-48bd-9962-72a20a40f2ff,BE9558,2026-05-16T13:36:07,K558LLAJ.470W,Long,0.3174,0.3223,0.985,0.2,0
-01e9cc58-3bea-4f38-8499-a2c20e5f8550,BE9558,2026-05-16T13:46:52,K558LLAJ.M40W,Long,0.2880,0.2930,0.983,0.2,0
-5ae7503d-ae17-4063-9208-6a420699e1ab,BE9558,2026-05-16T14:00:35,K558LLAK.8Z0W,Long,0.2879,0.2930,0.983,0.2,0
-e009ea3c-1534-4154-8623-e180ed4db9fb,BE9558,2026-05-16T14:06:19,K558LLAK.IJ0W,Long,0.2882,0.2930,0.984,0.2,0
-cd4e21f5-512b-4b2b-8054-bec01aa06400,BE9558,2026-05-16T14:12:03,K558LLAK.S30W,Long,0.2857,0.2930,0.975,0.2,0
-ae5b780f-2038-4ccf-8bfc-b15b87be139e,BE9558,2026-05-16T14:17:22,K558LLAL.0Y0W,Long,0.2899,0.2930,0.989,0.2,0
-d8c3fb81-8c66-4ebc-b33a-0559bc03334b,BE9558,2026-05-16T14:22:40,K558LLAL.9S0W,Long,0.3009,0.3027,0.994,0.2,0
-990a4c7a-3c51-4d4a-8ab7-416ccb97d29c,BE9558,2026-05-16T14:27:58,K558LLAL.IM0W,Long,0.3057,0.3076,0.994,0.2,0
-68d7a2c3-d899-4e75-96fb-3bf16ae6ea6f,BE9558,2026-05-16T14:33:16,K558LLAL.RG0W,Long,0.3110,0.3125,0.995,0.2,0
-73505b8c-f912-4116-b903-a8a4c75524d3,BE9558,2026-05-16T14:38:34,K558LLAM.0A0W,Long,0.3172,0.3223,0.984,0.2,0
-8d3c9602-7efa-488a-801b-3228bd85ab14,BE9558,2026-05-16T14:43:51,K558LLAM.930W,Long,0.3136,0.3174,0.988,0.2,0
-243e16ee-860a-4960-bc15-b797ad7e9735,BE9558,2026-05-16T14:49:08,K558LLAM.HW0W,Long,0.3135,0.3174,0.988,0.2,0
-e90c8d90-6d41-4788-9ee6-a29e4bfcc16b,BE9558,2026-05-16T15:19:35,K558LLAN.WN0W,Long,0.3132,0.3174,0.987,0.2,0
-4182b0e7-7811-46b8-87d1-a4dd0c152f3b,BE9558,2026-05-16T15:28:45,K558LLAO.BX0W,Long,0.3023,0.3076,0.983,0.2,0
-28a790da-0e48-4fe4-b8ba-2c82cc6bfa79,BE9558,2026-05-16T15:34:02,K558LLAO.KQ0W,Long,0.3176,0.3223,0.985,0.2,0
-093d2fa3-7a4b-4f0f-abda-71155ebe2dfb,BE9558,2026-05-16T15:39:19,K558LLAO.TJ0W,Long,0.3223,0.3271,0.985,0.2,0
-68dc9560-4082-49b6-af04-9244773ffbc1,BE9558,2026-05-16T15:44:36,K558LLAP.2C0W,Long,0.3227,0.3271,0.986,0.2,0
-ad6b268a-165d-4b1d-b0b3-f58db7c9b0e4,BE9558,2026-05-16T15:49:53,K558LLAP.B50W,Long,0.3177,0.3223,0.986,0.2,0
-046c5cb4-56b3-48c0-83c2-bf1f30837940,BE9558,2026-05-16T15:55:10,K558LLAP.JY0W,Long,0.3174,0.3223,0.985,0.2,0
-1b76edf4-3b1c-4bcb-a73c-27334981c350,BE9558,2026-05-16T16:00:27,K558LLAP.SR0W,Long,0.3127,0.3174,0.985,0.2,0
-a0264171-a682-4396-ad8a-cda3b2964533,BE9558,2026-05-16T16:06:55,K558LLAQ.3J0W,Long,0.2885,0.2930,0.985,0.2,0
-707da1f6-ee81-4e57-abd5-98295c14651b,BE9558,2026-05-16T16:12:30,K558LLAQ.CU0W,Long,0.2880,0.2930,0.983,0.2,0
-b8115b64-8739-42c8-b541-713ab6b68bcd,BE9558,2026-05-16T16:17:53,K558LLAQ.LT0W,Long,0.2881,0.2930,0.983,0.2,0
-fb179c36-fef1-4df6-b0a0-2bd7792ea910,BE9558,2026-05-16T16:23:11,K558LLAQ.UN0W,Long,0.2933,0.2979,0.985,0.2,0
-0cec8793-c3c1-486b-a33a-4dbf9cb069b0,BE9558,2026-05-16T16:33:46,K558LLAR.CA0W,Long,0.3017,0.3027,0.997,0.2,0
-c531b052-ee04-4c66-888b-21e97445a615,BE9558,2026-05-16T16:49:37,K558LLAS.2P0W,Long,0.2922,0.2979,0.981,0.2,0
-0dc402f9-cab6-4c4a-8317-b8fc3c4b1ced,BE9558,2026-05-16T16:55:08,K558LLAS.BW0W,Long,0.2898,0.2930,0.989,0.2,0
-fd73733e-b42d-4a51-81d0-4acad6edfbd4,BE9558,2026-05-16T17:16:17,K558LLAT.B50W,Long,0.3115,0.3125,0.997,0.2,0
-6a254daa-ae27-4ac1-b334-58f776a6b276,BE9558,2026-05-16T17:26:51,K558LLAT.SR0W,Long,0.3174,0.3223,0.985,0.2,0
-0cc8e82f-5270-4508-beae-0d3af56095ca,BE9558,2026-05-16T17:37:27,K558LLAU.AF0W,Long,0.2899,0.2930,0.989,0.2,0
-00557a73-8dc3-4cd2-b880-c7820dc8b171,BE9558,2026-05-16T17:43:17,K558LLAU.K50W,Long,0.2881,0.2930,0.983,0.2,0
-8e5518d9-ff0a-4c09-9cf9-a681c84c0b6e,BE9558,2026-05-16T17:53:53,K558LLAV.1T0W,Long,0.2898,0.2930,0.989,0.2,0
-91dbd7e9-7344-400b-82b1-379f2ac4b49a,BE9558,2026-05-16T17:59:11,K558LLAV.AN0W,Long,0.2925,0.2930,0.998,0.2,0
-fa256408-d480-4376-95b5-d1a549c483ea,BE9558,2026-05-16T18:04:29,K558LLAV.JH0W,Long,0.2977,0.3027,0.983,0.2,0
-4ab6e66a-8364-49b6-82f1-86e86199e676,BE9558,2026-05-16T18:09:46,K558LLAV.SA0W,Long,0.2944,0.2979,0.988,0.2,0
-0a65fb7c-96e1-4ba0-85de-4620d88658c1,BE9558,2026-05-16T18:15:03,K558LLAW.130W,Long,0.2965,0.2979,0.995,0.2,0
-92b1ef3e-1696-4123-8662-163947737c83,BE9558,2026-05-16T18:20:20,K558LLAW.9W0W,Long,0.2976,0.3027,0.983,0.2,0
-9e3cb24c-4426-4220-9a73-389d0a96c94f,BE9558,2026-05-16T18:25:37,K558LLAW.IP0W,Long,0.2983,0.3027,0.985,0.2,0
-a3473215-4c46-45ab-9aa7-1720486ec4a8,BE9558,2026-05-16T18:30:54,K558LLAW.RI0W,Long,0.2956,0.2979,0.993,0.2,0
-449665b1-47c8-4bd2-984c-8f45b627e6d8,BE9558,2026-05-16T18:36:11,K558LLAX.0B0W,Long,0.2989,0.3027,0.987,0.2,0
-600cfabc-cffd-42e1-800c-70e75f312316,BE9558,2026-05-16T20:00:12,K558LLB0.WC0W,Long,0.3077,0.3125,0.985,0.2,0
-12bf5944-0194-4f5b-b486-a7acd7141c7f,BE9558,2026-05-16T20:04:28,K558LLB1.3G0W,Long,0.2888,0.2930,0.986,0.2,0
-f1fc913a-91bc-4240-a595-7dfe3091f9a2,BE9558,2026-05-16T20:07:29,K558LLB1.8H0W,Long,0.2977,0.3027,0.983,0.2,0
-e74f7ec3-20bb-4177-beb2-e802e5f801fd,BE9558,2026-05-16T20:10:26,K558LLB1.DE0W,Long,0.2978,0.3027,0.984,0.2,0
-27491c3a-b631-4643-818a-714523911178,BE9558,2026-05-16T20:16:21,K558LLB1.N90W,Long,0.2921,0.2930,0.997,0.2,0
-9d9196ce-11e2-4f28-b01c-3dcc60e60d42,BE9558,2026-05-16T20:22:28,K558LLB1.XG0W,Long,0.2881,0.2930,0.983,0.2,0
-a592d7c8-b643-4b25-9186-3d611ae22709,BE9558,2026-05-16T20:25:37,K558LLB2.2P0W,Long,0.2893,0.2930,0.987,0.2,0
-f709a3d5-27cd-4bb1-b91c-669e8ea11d77,BE9558,2026-05-16T20:28:57,K558LLB2.890W,Long,0.2880,0.2930,0.983,0.2,0
-cc9cad87-b87f-4b67-854a-ffb21e876a92,BE9558,2026-05-16T20:32:45,K558LLB2.EL0W,Long,0.2873,0.2930,0.980,0.2,0
-10732d26-6b49-4b14-a474-6c75e259d278,BE9558,2026-05-16T20:49:32,K558LLB3.6K0W,Long,0.2881,0.2930,0.983,0.2,0
-aa63957a-fdb2-4f17-af52-a63bb015119d,BE9558,2026-05-16T21:21:45,K558LLB4.O90W,Long,0.2880,0.2930,0.983,0.2,0
-8350440f-019d-45a4-b9f9-23058628e6f0,BE9558,2026-05-16T21:32:43,K558LLB5.6J0W,Long,0.2925,0.2930,0.999,0.2,0
-edfc10b2-dbe1-4527-a061-b0d670b5e5d2,BE9558,2026-05-16T21:35:37,K558LLB5.BD0W,Long,0.2891,0.2930,0.987,0.2,0
-f98001a3-88b2-4830-8112-b7c5164e99c3,BE9558,2026-05-16T21:38:40,K558LLB5.GG0W,Long,0.2877,0.2930,0.982,0.2,0
-f1e53e7b-b124-4fc5-82c2-504fb0c8b80d,BE9558,2026-05-16T21:41:46,K558LLB5.LM0W,Long,0.2881,0.2930,0.983,0.2,0
-7890637f-142e-4775-b48c-fd4f3b8ada77,BE9558,2026-05-16T21:45:14,K558LLB5.RE0W,Long,0.2880,0.2930,0.983,0.2,0
-7c97c10f-b413-467c-9b56-c18e423cdcfa,BE9558,2026-05-16T21:48:26,K558LLB5.WQ0W,Long,0.2886,0.2930,0.985,0.2,0
-8178cf22-8428-4f85-8199-0a43f7334a8f,BE9558,2026-05-16T21:51:47,K558LLB6.2B0W,Long,0.2881,0.2930,0.983,0.2,0
-eceb2997-deb4-403a-802f-50e93d23d423,BE9558,2026-05-16T21:57:18,K558LLB6.BI0W,Long,0.2881,0.2930,0.984,0.2,0
-9c26690c-1983-4b21-b8af-7c7967c014c3,BE9558,2026-05-16T22:12:21,K558LLB7.0L0W,Long,0.2876,0.2930,0.982,0.2,0
-78f67fc3-08b7-4317-96ba-ee5544e0b95b,BE9558,2026-05-16T22:19:34,K558LLB7.CM0W,Long,0.2880,0.2930,0.983,0.2,0
-7f9dad14-4f1b-492d-aa26-845a908ea894,BE9558,2026-05-16T22:27:40,K558LLB7.Q40W,Long,0.2881,0.2930,0.983,0.2,0
-f75c52b4-21b9-4065-baeb-5cfbd27d130a,BE9558,2026-05-16T22:33:50,K558LLB8.0E0W,Long,0.2881,0.2930,0.984,0.2,0
-10f7496d-fa76-40a7-8bef-dc14e8a41869,BE9558,2026-05-16T22:40:48,K558LLB8.C00W,Long,0.2878,0.2930,0.982,0.2,0

scratch/offset_hist_scan.py

@@ -1,234 +0,0 @@
-#!/usr/bin/env python3
-"""Offset detector — HISTOGRAM corpus (the other 90% of the archive).
-
-`offset_scan3.py` measures the pre-trigger floor in *waveform* samples.  That
-covers 6,577 of the archive's 70,112 unique series-3 files; the remaining
-63,535 are **histograms**, which carry no samples — only a per-interval,
-per-channel peak + half-period.  So the pre-trigger method cannot run on them.
-
-The histogram analogue of "the resting floor" is the **low percentile of the
-per-interval peaks**.  A histogram file is typically hours of continuous
-monitoring, so the great majority of its intervals are definitionally quiet;
-the bottom of that distribution is what the channel reads when nothing is
-happening.  A healthy channel bottoms out at 0.000-0.005 in/s.  A channel
-parked off zero cannot report a peak below its own displacement, so its floor
-is pinned up.
-
-⚠ The DC leakage into the histogram peak is PARTIAL.  Measured within-unit
-against episodes already established from the waveform scan:
-
-    BE18438  Vert  in-episode 0.0350 vs 0.0050 outside  (waveform pre = +0.18..+0.37)
-    BE12599  Tran  in-episode 0.0250 vs 0.0050 outside  (waveform pre = +0.03..+0.49)
-
-so the device's per-interval peak is evidently measured against a running /
-AC-coupled baseline that removes most, but not all, of the DC.  The residual
-is real and channel-specific, but the margin is ~5 quantisation counts rather
-than the ~70 the waveform detector enjoys.  Do not carry the waveform
-detector's 0.025 in/s floor across unexamined — calibrate on the CSV.
-
-Because the absolute floor also moves with site noise (traffic, wind, a
-generator), the statistic that matters most is the **cross-channel
-differential**: a channel's floor minus the quietest of the other two geo
-channels in the same file.  Site noise lifts all three together and cancels;
-a DC offset lifts one.
-
-This script does not decide anything.  It emits every candidate statistic per
-(file, channel) so thresholds can be calibrated against the waveform-derived
-ground truth in `offset_v3.csv` rather than guessed.
-
-Usage:
-    python scratch/offset_hist_scan.py --dir /home/serversdown/dl2-archive/files \
-        --out /home/serversdown/dl2-archive/offset_hist.csv --jobs 4
-"""
-from __future__ import annotations
-
-import argparse
-import csv
-import datetime
-import logging
-import re
-import statistics
-import sys
-from concurrent.futures import ProcessPoolExecutor, as_completed
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-from minimateplus.event_file_io import read_blastware_file  # noqa: E402
-
-GEO = ("Tran", "Vert", "Long")
-K = 10.0 / 32000.0          # ADC count -> in/s (see CLAUDE.md: full scale 32000)
-_HIST = re.compile(r"\.[A-Za-z0-9]{2}0[Hh]$")
-_STEM = re.compile(r"^([B-Z])(\d{3})")
-_B36 = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
-
-
-_SERIAL_RE = re.compile(rb"\b([A-Z]{2}\d{3,6})\b")
-
-
-def serial_of(name: str, path=None) -> str:
-    """Real serial for a BW file.
-
-    The filename encodes only the NUMBER: `<letter><3 digits>` where
-    letter = chr(ord('B') + serial // 1000).  The two-letter family prefix
-    ("BE", "BA", ...) is **not** in the filename, so it must be read out of
-    the file body.  Four units in the DL2 archive are BA, not BE — assuming
-    "BE" mislabels BA9229, BA10060, BA10895 and BA15957.
-    """
-    m = _STEM.match(name)
-    if not m:
-        return "?"
-    num = (ord(m.group(1)) - ord("B")) * 1000 + int(m.group(2))
-    if path is not None:
-        try:
-            for s in _SERIAL_RE.findall(Path(path).read_bytes()):
-                s = s.decode()
-                if s[2:].lstrip("0") == str(num):
-                    return s
-        except Exception:
-            pass
-    return f"BE{num}"          # last-resort fallback; prefix unverified
-
-
-def stem_time(name: str):
-    """Decode the filename's base-36 timestamp.  Epoch 1985-01-01, 1296 s/tick.
-
-    Preferred over the file's own footer timestamp only because it costs
-    nothing; the caller falls back to the decoded event when this fails.
-    """
-    try:
-        base, ext = name.rsplit(".", 1)
-        n = 0
-        for c in base[4:8].upper():
-            n = n * 36 + _B36.index(c)
-        ab = _B36.index(ext[0].upper()) * 36 + _B36.index(ext[1].upper())
-        return datetime.datetime(1985, 1, 1) + datetime.timedelta(seconds=n * 1296 + ab)
-    except Exception:
-        return None
-
-
-def _pct(sorted_vals, q):
-    """Nearest-rank percentile on an already-sorted list."""
-    if not sorted_vals:
-        return None
-    i = min(len(sorted_vals) - 1, max(0, int(len(sorted_vals) * q / 100.0)))
-    return sorted_vals[i]
-
-
-def scan(path_str: str):
-    logging.disable(logging.WARNING)   # per-worker: the codec warns on undecodables
-    p = Path(path_str)
-    try:
-        ev = read_blastware_file(p)
-    except Exception:
-        return None
-    s = ev.raw_samples or {}
-    if not any(s.get(c) for c in GEO):
-        return None
-
-    ts = stem_time(p.name) or ev.timestamp
-    stamp = ""
-    if ts is not None:
-        stamp = (f"{ts.year:04d}-{ts.month:02d}-{ts.day:02d}T"
-                 f"{ts.hour:02d}:{ts.minute:02d}:{ts.second:02d}")
-
-    # Per-channel floor candidates, in in/s.
-    stats = {}
-    for ch in GEO:
-        v = sorted(s.get(ch) or [])
-        if not v:
-            continue
-        stats[ch] = {
-            "n": len(v),
-            "min": v[0] * K,
-            "p1": _pct(v, 1) * K,
-            "p5": _pct(v, 5) * K,
-            "p10": _pct(v, 10) * K,
-            "p25": _pct(v, 25) * K,
-            "med": statistics.median(v) * K,
-            "peak": v[-1] * K,
-            "zeros": sum(1 for x in v if x == 0) / len(v),
-        }
-    if len(stats) < 2:      # need at least one sibling channel for the differential
-        return None
-
-    # Mic floor as a site-noise proxy (raw counts; the dB conversion is not
-    # needed — only its relative movement matters here).
-    mic = sorted(s.get("MicL") or [])
-    mic_p5 = _pct(mic, 5) if mic else ""
-
-    rows = []
-    for ch, st in stats.items():
-        others = [stats[o]["p5"] for o in stats if o != ch]
-        rows.append({
-            "serial": serial_of(p.name, p),
-            "timestamp": stamp,
-            "filename": p.name,
-            "channel": ch,
-            "n_intervals": st["n"],
-            "min": round(st["min"], 4),
-            "p1": round(st["p1"], 4),
-            "p5": round(st["p5"], 4),
-            "p10": round(st["p10"], 4),
-            "p25": round(st["p25"], 4),
-            "median": round(st["med"], 4),
-            "peak": round(st["peak"], 4),
-            "frac_zero": round(st["zeros"], 4),
-            # the site-noise-cancelling statistic: this channel's floor above
-            # the quietest sibling geo channel in the same file
-            "diff_p5": round(st["p5"] - min(others), 4),
-            "mic_p5": mic_p5,
-        })
-    return rows
-
-
-COLS = ["serial", "timestamp", "filename", "channel", "n_intervals",
-        "min", "p1", "p5", "p10", "p25", "median", "peak", "frac_zero",
-        "diff_p5", "mic_p5"]
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--dir", required=True)
-    ap.add_argument("--out", required=True)
-    ap.add_argument("--jobs", type=int, default=4)
-    ap.add_argument("--limit", type=int, default=0, help="stop after N files (smoke test)")
-    a = ap.parse_args()
-
-    # Dedupe by basename — the DL2 export keeps a byte-identical `Sent/`
-    # mirror of its root, which doubled two figures before it was caught.
-    seen, files = set(), []
-    for q in sorted(Path(a.dir).rglob("*")):
-        if q.is_file() and _HIST.search(q.name) and q.name not in seen:
-            seen.add(q.name)
-            files.append(str(q))
-    if a.limit:
-        files = files[:a.limit]
-    print(f"unique histogram binaries: {len(files)}", flush=True)
-
-    rows, undecodable = [], 0
-    with ProcessPoolExecutor(max_workers=a.jobs) as ex:
-        futs = [ex.submit(scan, f) for f in files]
-        for i, fut in enumerate(as_completed(futs), 1):
-            r = fut.result()
-            if r:
-                rows.extend(r)
-            else:
-                undecodable += 1
-            if i % 5000 == 0:
-                print(f"  {i}/{len(files)}", flush=True)
-
-    with open(a.out, "w", newline="") as fh:
-        w = csv.DictWriter(fh, fieldnames=COLS)
-        w.writeheader()
-        w.writerows(rows)
-
-    files_ok = len({r["filename"] for r in rows})
-    units = len({r["serial"] for r in rows})
-    ivals = sum(r["n_intervals"] for r in rows) // 3
-    print(f"\ndecoded {files_ok}/{len(files)} files "
-          f"({undecodable} undecodable), {units} units, ~{ivals/1e6:.1f}M intervals")
-    print(f"wrote {a.out}  ({len(rows)} channel-rows)")
-
-
-if __name__ == "__main__":
-    main()

scratch/offset_scan.py

@@ -1,171 +0,0 @@
-#!/usr/bin/env python3
-"""Scan series-3 waveform binaries for the 'offset' hardware fault.
-
-A healthy geophone trace is centred on zero.  An offset unit sits displaced,
-so the channel mean approaches its own peak.  Detector (unchanged from the
-2026-08-25 run, see memory note `offset-archive-analysis-backlog`):
-
-    dominant-axis  |mean| / peak > 0.7
-    AND            |mean| >= 0.9 * the unit's geo trigger level
-
-Trigger level is read from a paired _ASCII.TXT where one exists, otherwise
-from a per-serial median learned across that unit's ASCII files, otherwise
---default-trigger.
-
-Serial is decoded from the BW filename: prefix letter encodes thousands
-(chr(ord('B') + n)), next 3 digits the remainder -- T193 -> BE18193.
-
-Usage:
-  python scratch/offset_scan.py --dir <path> [--jobs N] --out offsets.csv
-"""
-from __future__ import annotations
-
-import argparse, csv, json, re, sys
-from collections import defaultdict
-from concurrent.futures import ProcessPoolExecutor, as_completed
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from minimateplus.event_file_io import read_blastware_file
-from minimateplus.bw_ascii_report import parse_report
-
-GEO = ("Tran", "Vert", "Long")
-_GEO_FS_COUNTS = 32000.0
-_WAVE_RE = re.compile(r"\.[A-Za-z0-9]{2}0[Ww]$")
-_STEM_RE = re.compile(r"^([B-Z])(\d{3})")
-
-MEAN_OVER_PEAK_MIN = 0.7
-TRIGGER_FRACTION   = 0.9
-
-
-def serial_from_name(name: str):
-    m = _STEM_RE.match(name)
-    if not m:
-        return None
-    letter, digits = m.group(1), m.group(2)
-    return f"BE{(ord(letter) - ord('B')) * 1000 + int(digits)}"
-
-
-def counts_to_ips(c, gr):
-    return c * (gr or 10.0) / _GEO_FS_COUNTS
-
-
-def scan_one(path_str: str, default_trigger: float) -> dict | None:
-    p = Path(path_str)
-    try:
-        gr, trig = 10.0, None
-        ap = p.with_name(p.name.replace(".", "_", 1) + "_ASCII.TXT") \
-            if False else p.parent / (p.stem + "_" + p.suffix.lstrip(".") + "_ASCII.TXT")
-        if ap.exists():
-            rep = parse_report(ap.read_text(errors="replace"))
-            gr = rep.geo_range_ips or 10.0
-            trig = rep.geo_trigger_level_ips
-        ev = read_blastware_file(p)
-        s = ev.raw_samples or {}
-        if not all(s.get(c) for c in GEO):
-            return None
-        best = None
-        for ch in GEO:
-            arr = s[ch]
-            n = len(arr)
-            if n == 0:
-                continue
-            mean = sum(arr) / n
-            peak = max(abs(v) for v in arr)
-            if peak == 0:
-                continue
-            ratio = abs(mean) / peak
-            if best is None or peak > best["peak_counts"]:
-                best = {"channel": ch, "mean_counts": mean,
-                        "peak_counts": peak, "ratio": ratio}
-        if best is None:
-            return None
-        ts = ev.timestamp
-        return {
-            "serial": serial_from_name(p.name) or "?",
-            "timestamp": (f"{ts.year:04d}-{ts.month:02d}-{ts.day:02d}T"
-                          f"{ts.hour:02d}:{ts.minute:02d}:{ts.second:02d}") if ts else "",
-            "filename": p.name,
-            "channel": best["channel"],
-            "offset_ips": round(counts_to_ips(best["mean_counts"], gr), 4),
-            "peak_ips": round(counts_to_ips(best["peak_counts"], gr), 4),
-            "mean_over_peak": round(best["ratio"], 3),
-            "trigger_level_ips": trig if trig is not None else "",
-            "geo_range_ips": gr,
-        }
-    except Exception:
-        return None
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--dir", required=True)
-    ap.add_argument("--jobs", type=int, default=4)
-    ap.add_argument("--limit", type=int, default=0)
-    ap.add_argument("--default-trigger", type=float, default=0.2)
-    ap.add_argument("--out", required=True)
-    a = ap.parse_args()
-
-    # The DL2 export keeps a byte-identical `Sent/` mirror of the root, so
-    # enumerate paths but keep only the first occurrence of each basename —
-    # otherwise every event is counted twice.
-    seen = set()
-    files = []
-    for q in sorted(Path(a.dir).rglob("*")):
-        if q.is_file() and _WAVE_RE.search(q.name) and q.name not in seen:
-            seen.add(q.name)
-            files.append(q)
-    if a.limit:
-        files = files[: a.limit]
-    print(f"waveform binaries to scan: {len(files)}", flush=True)
-
-    rows = []
-    with ProcessPoolExecutor(max_workers=a.jobs) as ex:
-        futs = [ex.submit(scan_one, str(p), a.default_trigger) for p in files]
-        for n, f in enumerate(as_completed(futs), 1):
-            r = f.result()
-            if r:
-                rows.append(r)
-            if n % 2000 == 0:
-                print(f"  {n}/{len(files)}", flush=True)
-
-    # learn per-serial trigger levels from the rows that had an ASCII
-    by_serial = defaultdict(list)
-    for r in rows:
-        if r["trigger_level_ips"] != "":
-            by_serial[r["serial"]].append(float(r["trigger_level_ips"]))
-    med = {}
-    for k, v in by_serial.items():
-        v.sort()
-        med[k] = v[len(v) // 2]
-
-    for r in rows:
-        if r["trigger_level_ips"] == "":
-            r["trigger_level_ips"] = med.get(r["serial"], a.default_trigger)
-        r["suspect"] = int(
-            r["mean_over_peak"] > MEAN_OVER_PEAK_MIN
-            and abs(r["offset_ips"]) >= TRIGGER_FRACTION * float(r["trigger_level_ips"])
-        )
-
-    cols = ["serial", "timestamp", "filename", "channel", "offset_ips", "peak_ips",
-            "mean_over_peak", "trigger_level_ips", "geo_range_ips", "suspect"]
-    with open(a.out, "w", newline="") as fh:
-        w = csv.DictWriter(fh, fieldnames=cols)
-        w.writeheader()
-        w.writerows(rows)
-
-    sus = [r for r in rows if r["suspect"]]
-    print(f"\nscanned {len(rows)} decodable waveforms")
-    print(f"suspect events: {len(sus)}")
-    per = defaultdict(int)
-    for r in sus:
-        per[r["serial"]] += 1
-    print(f"units with >=1 suspect event: {len(per)} of {len({r['serial'] for r in rows})}")
-    for s, n in sorted(per.items(), key=lambda x: -x[1])[:20]:
-        print(f"  {s:10} {n}")
-    print(f"\nwrote {a.out}")
-
-
-if __name__ == "__main__":
-    main()

scratch/offset_scan2.py

@@ -1,108 +0,0 @@
-#!/usr/bin/env python3
-"""Offset detector v2 — per-channel MEDIAN pedestal.
-
-Supersedes the dominant-axis / mean detector in offset_scan.py, which had two
-flaws that manufactured false "recoveries":
-
-  1. It scored only the axis with the largest peak, so a real event on one axis
-     hid a persistent pedestal on another.  BE12599 2026-08-21 read "clean"
-     because Long had a 1.065 in/s event, while Tran sat at +0.47 in/s.
-  2. It used the MEAN, which a real transient perturbs.  The median is the
-     resting baseline: most samples sit at it, so a blast does not move it.
-     Same event, Long: mean +0.0783 vs median -0.0050.
-
-Flags a CHANNEL when |median| >= --floor in/s (default 0.025 = 5 A/D counts,
-Instantel's own criterion; 1 A/D count = 0.005 in/s).
-
-Emits one row per (event, channel) so persistence can be tracked per channel.
-"""
-from __future__ import annotations
-import argparse, csv, re, statistics, sys
-from concurrent.futures import ProcessPoolExecutor, as_completed
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-from minimateplus.event_file_io import read_blastware_file
-
-GEO = ("Tran", "Vert", "Long")
-K = 10.0 / 32000.0                      # ADC counts -> in/s at the 10 in/s range
-_WAVE_RE = re.compile(r"\.[A-Za-z0-9]{2}0[Ww]$")
-_STEM_RE = re.compile(r"^([B-Z])(\d{3})")
-
-
-def serial_from_name(n):
-    m = _STEM_RE.match(n)
-    return f"BE{(ord(m.group(1))-ord('B'))*1000+int(m.group(2))}" if m else "?"
-
-
-def scan_one(ps):
-    p = Path(ps)
-    try:
-        ev = read_blastware_file(p)
-        s = ev.raw_samples or {}
-        if not all(s.get(c) for c in GEO):
-            return None
-        ts = ev.timestamp
-        stamp = (f"{ts.year:04d}-{ts.month:02d}-{ts.day:02d}T"
-                 f"{ts.hour:02d}:{ts.minute:02d}:{ts.second:02d}") if ts else ""
-        out = []
-        for ch in GEO:
-            a = s[ch]
-            out.append({
-                "serial": serial_from_name(p.name), "timestamp": stamp,
-                "filename": p.name, "channel": ch,
-                "median_ips": round(statistics.median(a) * K, 4),
-                "mean_ips":   round(statistics.fmean(a) * K, 4),
-                "peak_ips":   round(max(abs(v) for v in a) * K, 4),
-            })
-        return out
-    except Exception:
-        return None
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--dir", required=True)
-    ap.add_argument("--jobs", type=int, default=4)
-    ap.add_argument("--floor", type=float, default=0.025)
-    ap.add_argument("--out", required=True)
-    a = ap.parse_args()
-
-    seen, files = set(), []
-    for q in sorted(Path(a.dir).rglob("*")):
-        if q.is_file() and _WAVE_RE.search(q.name) and q.name not in seen:
-            seen.add(q.name); files.append(str(q))
-    print(f"unique waveform binaries: {len(files)}", flush=True)
-
-    rows = []
-    with ProcessPoolExecutor(max_workers=a.jobs) as ex:
-        for n, f in enumerate(as_completed([ex.submit(scan_one, p) for p in files]), 1):
-            r = f.result()
-            if r: rows.extend(r)
-            if n % 2000 == 0: print(f"  {n}/{len(files)}", flush=True)
-
-    for r in rows:
-        r["offset"] = int(abs(r["median_ips"]) >= a.floor)
-
-    cols = ["serial","timestamp","filename","channel","median_ips","mean_ips","peak_ips","offset"]
-    with open(a.out, "w", newline="") as fh:
-        w = csv.DictWriter(fh, fieldnames=cols); w.writeheader(); w.writerows(rows)
-
-    from collections import defaultdict
-    ev_flagged = {(r["serial"], r["filename"]) for r in rows if r["offset"]}
-    ev_all     = {(r["serial"], r["filename"]) for r in rows}
-    per = defaultdict(set)
-    for r in rows:
-        if r["offset"]: per[r["serial"]].add(r["filename"])
-    tot = defaultdict(set)
-    for r in rows: tot[r["serial"]].add(r["filename"])
-    print(f"\nfloor = {a.floor} in/s ({a.floor/0.005:.0f} A/D counts)")
-    print(f"events with >=1 offset channel: {len(ev_flagged)} of {len(ev_all)}")
-    print(f"units affected: {len(per)} of {len(tot)}")
-    for s in sorted(per, key=lambda s: -len(per[s])):
-        print(f"  {s:9} {len(per[s]):4} / {len(tot[s]):4} events")
-    print(f"\nwrote {a.out}")
-
-
-if __name__ == "__main__":
-    main()

scratch/offset_scan3.py

@@ -1,117 +0,0 @@
-#!/usr/bin/env python3
-"""Offset detector v3 — pre-trigger floor, with pre/mid/end consistency.
-
-Brian's method, and better than v2's whole-record median for one reason: the
-pre-trigger window is *definitionally* quiet (it is the buffer captured before
-the trigger fired), whereas a whole-record median is merely robust to the event.
-
-Per channel:
-    pre  = median of the first `pretrig_samples` samples (STRT record)
-    mid  = median of the middle third
-    end  = median of the final third
-    spread = max(pre,mid,end) - min(pre,mid,end)
-
-A DC offset is a *constant floor*: |pre| at or above the floor AND a small
-spread. A transient (settling, handling, a long-tailed event) moves one segment
-relative to the others and is rejected by the spread test.
-
-Floor default 0.025 in/s = 5 A/D counts (Instantel's own criterion; 1 count =
-0.005 in/s).  Quantisation is 0.005 in/s, so `spread` is measured in units of it.
-"""
-from __future__ import annotations
-import argparse, csv, re, statistics, sys
-from concurrent.futures import ProcessPoolExecutor, as_completed
-from pathlib import Path
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-from minimateplus.event_file_io import read_blastware_file
-
-GEO=("Tran","Vert","Long"); K=10.0/32000.0
-_WAVE=re.compile(r"\.[A-Za-z0-9]{2}0[Ww]$"); _STEM=re.compile(r"^([B-Z])(\d{3})")
-
-_SERIAL_RE = re.compile(rb"\b([A-Z]{2}\d{3,6})\b")
-
-
-def serial_of(name: str, path=None) -> str:
-    """Real serial for a BW file.
-
-    The filename encodes only the NUMBER: `<letter><3 digits>` where
-    letter = chr(ord('B') + serial // 1000).  The two-letter family prefix
-    ("BE", "BA", ...) is **not** in the filename, so it must be read out of
-    the file body.  Four units in the DL2 archive are BA, not BE — assuming
-    "BE" mislabels BA9229, BA10060, BA10895 and BA15957.
-    """
-    m = _STEM.match(name)
-    if not m:
-        return "?"
-    num = (ord(m.group(1)) - ord("B")) * 1000 + int(m.group(2))
-    if path is not None:
-        try:
-            for s in _SERIAL_RE.findall(Path(path).read_bytes()):
-                s = s.decode()
-                if s[2:].lstrip("0") == str(num):
-                    return s
-        except Exception:
-            pass
-    return f"BE{num}"          # last-resort fallback; prefix unverified
-
-def scan(ps):
-    p=Path(ps)
-    try:
-        ev=read_blastware_file(p); s=ev.raw_samples or {}
-        if not all(s.get(c) for c in GEO): return None
-        pre_n=ev.pretrig_samples
-        ts=ev.timestamp
-        stamp=(f"{ts.year:04d}-{ts.month:02d}-{ts.day:02d}T"
-               f"{ts.hour:02d}:{ts.minute:02d}:{ts.second:02d}") if ts else ""
-        out=[]
-        for ch in GEO:
-            a=s[ch]; n=len(a); t=n//3
-            pre = a[:pre_n] if (pre_n and 0 < pre_n < n) else a[:t]
-            mid, end = a[t:2*t], a[2*t:]
-            if not pre or not mid or not end: continue
-            v=[statistics.median(x)*K for x in (pre,mid,end)]
-            out.append({"serial":serial_of(p.name, p),"timestamp":stamp,
-                        "filename":p.name,"channel":ch,
-                        "pretrig_n": pre_n or 0,
-                        "pre":round(v[0],4),"mid":round(v[1],4),"end":round(v[2],4),
-                        "spread":round(max(v)-min(v),4),
-                        "peak":round(max(abs(x) for x in a)*K,4)})
-        return out
-    except Exception:
-        return None
-
-def main():
-    ap=argparse.ArgumentParser()
-    ap.add_argument("--dir",required=True); ap.add_argument("--jobs",type=int,default=4)
-    ap.add_argument("--floor",type=float,default=0.025)
-    ap.add_argument("--max-spread",type=float,default=0.02)
-    ap.add_argument("--out",required=True)
-    a=ap.parse_args()
-    seen=set(); files=[]
-    for q in sorted(Path(a.dir).rglob("*")):
-        if q.is_file() and _WAVE.search(q.name) and q.name not in seen:
-            seen.add(q.name); files.append(str(q))
-    print(f"unique waveform binaries: {len(files)}",flush=True)
-    rows=[]
-    with ProcessPoolExecutor(max_workers=a.jobs) as ex:
-        for i,f in enumerate(as_completed([ex.submit(scan,p) for p in files]),1):
-            r=f.result()
-            if r: rows.extend(r)
-            if i%2000==0: print(f"  {i}/{len(files)}",flush=True)
-    for r in rows:
-        r["offset"]=int(abs(r["pre"])>=a.floor and r["spread"]<=a.max_spread)
-    cols=["serial","timestamp","filename","channel","pretrig_n","pre","mid","end","spread","peak","offset"]
-    with open(a.out,"w",newline="") as fh:
-        w=csv.DictWriter(fh,fieldnames=cols); w.writeheader(); w.writerows(rows)
-    from collections import defaultdict
-    per=defaultdict(set); tot=defaultdict(set)
-    for r in rows:
-        tot[r["serial"]].add(r["filename"])
-        if r["offset"]: per[r["serial"]].add(r["filename"])
-    print(f"\nfloor={a.floor} in/s ({a.floor/0.005:.0f} counts)  max spread={a.max_spread}")
-    print(f"units affected: {len(per)} of {len(tot)}")
-    for s in sorted(per,key=lambda s:-len(per[s])):
-        print(f"  {s:9} {len(per[s]):4} / {len(tot[s]):4} events")
-    print(f"\nwrote {a.out}")
-
-if __name__=="__main__": main()

scratch/socat_log_split.py

@@ -1,119 +0,0 @@
-#!/usr/bin/env python3
-"""
-socat_log_split.py — recover a capture pair from a `socat -x` relay log.
-
-Why this exists
----------------
-The bench relay that puts Thor in front of a USB-attached Micromate is:
-
-    socat -d -d -x TCP-LISTEN:12345,reuseaddr,fork /dev/ttyACM0,raw,echo=0,b115200 \
-        > ~/mm-captures/socat_<ts>.log 2>&1
-
-`-x` makes socat hex-dump every byte it forwards, in both directions, with
-timestamps.  That log is therefore a **complete second copy of every capture**
-taken through the relay — independent of whether seismo_lab was recording.
-
-On 2026-09-25 that mattered: a capture's `.bin` files never made it off the
-Windows machine, and the session was rebuilt from this log instead.  When the
-real bins turned up later, the reconstruction was **byte-for-byte identical in
-both directions** (3,595 and 4,004 bytes).  So this is a validated fallback, not
-a lossy approximation.
-
-Log format
-----------
-```
-> 2026/09/25 00:30:35.000276659  length=21 from=0 to=20
- 41 02 10 10 00 5b 00 00 30 00 ...
-2026/09/25 00:30:35 socat[32190] N write(5, 0x..., 21) completed
-< 2026/09/25 00:30:35.000384100  length=64 from=0 to=63
- 02 00 c5 a4 00 00 30 00 ...
-```
-
-`>` is data heading toward the serial device (Thor → unit).  `<` is data coming
-back (unit → Thor).  Hex lines are space-separated and indented; socat's own
-status lines start with a date and carry no payload.
-
-Usage
------
-    # whole log
-    python scratch/socat_log_split.py socat_20260924_181248.log --out-dir ./recovered
-
-    # one session — line numbers from the "accepting connection" markers
-    grep -n "accepting connection" socat_*.log
-    python scratch/socat_log_split.py socat_*.log --from-line 919 --out-dir ./recovered
-
-Then parse the result as usual:
-
-    python scratch/mm_frame_parse.py recovered/raw_bw.bin recovered/raw_s3.bin
-
-⚠ A log spanning several sessions concatenates them.  Split by line number using
-the `accepting connection` markers, or the frame walk will run sessions together.
-"""
-
-from __future__ import annotations
-
-import argparse
-import re
-from pathlib import Path
-
-_HEX = re.compile(r"\A[0-9a-f]{2}\Z")
-_SOCAT_STATUS = re.compile(r"\A\d{4}/\d{2}/\d{2}")
-
-
-def split(lines) -> tuple[bytes, bytes]:
-    """Return (to_device, from_device) byte streams."""
-    to_dev, from_dev = bytearray(), bytearray()
-    cur = None
-    for line in lines:
-        if line.startswith(">"):
-            cur = to_dev
-            continue
-        if line.startswith("<"):
-            cur = from_dev
-            continue
-        if _SOCAT_STATUS.match(line):
-            # socat's own status line ends the current dump block.
-            cur = None
-            continue
-        if cur is None or not line.startswith(" "):
-            continue
-        toks = line.split()
-        if toks and all(_HEX.match(t) for t in toks):
-            cur.extend(int(t, 16) for t in toks)
-    return bytes(to_dev), bytes(from_dev)
-
-
-def main() -> int:
-    ap = argparse.ArgumentParser(
-        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
-    )
-    ap.add_argument("log", help="a socat -x log file")
-    ap.add_argument("--out-dir", default=".", help="where to write the .bin pair")
-    ap.add_argument("--from-line", type=int, default=1,
-                    help="first log line to read (1-based) — use the "
-                         "'accepting connection' marker of the session you want")
-    ap.add_argument("--to-line", type=int, default=None,
-                    help="last log line to read (1-based, inclusive)")
-    ap.add_argument("--prefix", default="raw", help="output basename prefix")
-    args = ap.parse_args()
-
-    lines = Path(args.log).read_text(errors="replace").splitlines()
-    lo = max(args.from_line - 1, 0)
-    hi = args.to_line if args.to_line is not None else len(lines)
-    to_dev, from_dev = split(lines[lo:hi])
-
-    out = Path(args.out_dir)
-    out.mkdir(parents=True, exist_ok=True)
-    bw = out / f"{args.prefix}_bw.bin"
-    s3 = out / f"{args.prefix}_s3.bin"
-    bw.write_bytes(to_dev)
-    s3.write_bytes(from_dev)
-    print(f"Thor -> unit  {len(to_dev):>7} bytes  {bw}")
-    print(f"unit -> Thor  {len(from_dev):>7} bytes  {s3}")
-    if not to_dev or not from_dev:
-        print("⚠ one direction is empty — check --from-line / --to-line")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

scratch/verify_against_ascii.py

@@ -1,221 +0,0 @@
-#!/usr/bin/env python3
-"""Verify the series-3 decoder against preserved Blastware ASCII exports.
-
-Pairs each `<stem>_<ext>_ASCII.TXT` with its binary `<stem>.<ext>`, decodes the
-binary with the production codec, and compares against BW's own export:
-
-  waveform  — per-channel sample counts, then every sample value
-  histogram — interval count, then every per-interval channel peak
-
-ADC counts convert as  ips = counts * geo_range_ips / 32000  (1 decoder unit =
-16 counts = 0.005 in/s at the 10 in/s range; see CLAUDE.md).
-
-Usage:
-  python scratch/verify_against_ascii.py --dir <path> [--limit N] [--jobs N]
-                                         [--out results.json] [--kind w|h|all]
-"""
-from __future__ import annotations
-
-import argparse, json, re, sys, traceback
-from concurrent.futures import ProcessPoolExecutor, as_completed
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from minimateplus.event_file_io import read_blastware_file
-from minimateplus.bw_ascii_report import parse_report_file, parse_report
-
-GEO = ("Tran", "Vert", "Long")
-_GEO_FS_COUNTS = 32000.0
-_ASCII_SUFFIX_RE = re.compile(r"_ASCII\.TXT$", re.IGNORECASE)
-
-
-def binary_for(ascii_path: Path) -> Path:
-    """H907KXOW_WC0H_ASCII.TXT -> H907KXOW.WC0H"""
-    stem = _ASCII_SUFFIX_RE.sub("", ascii_path.name)
-    if "_" not in stem:
-        return ascii_path.with_name(stem)
-    head, _, ext = stem.rpartition("_")
-    return ascii_path.with_name(f"{head}.{ext}")
-
-
-def counts_to_ips(counts, geo_range_ips):
-    r = geo_range_ips if geo_range_ips else 10.0
-    return counts * r / _GEO_FS_COUNTS
-
-
-def agrees(got, exp, geo_range_ips, tol=0.0006):
-    """True when decoded `got` matches BW's exported `exp`.
-
-    Saturation carve-out: when an event clips, BW clamps its export to the
-    channel's range maximum (and writes OORANGE for the summary PPV), while
-    the decoder faithfully reproduces raw counts that can sit a decoder unit
-    or two past nominal full scale (32016 counts observed = 10.005 in/s on
-    the 10 in/s range).  Same sign and both at/above the ceiling is agreement,
-    not a decode error.
-    """
-    if abs(got - exp) <= tol:
-        return True
-    r = geo_range_ips if geo_range_ips else 10.0
-    if abs(exp) >= r - tol and abs(got) >= r - tol and (got >= 0) == (exp >= 0):
-        return True
-    return False
-
-
-def parse_interval_table(text: str):
-    """Histogram interval rows: time, Tpk, Tfq, Vpk, Vfq, Lpk, Lfq, PVS, ..., micdB, micfq"""
-    rows = []
-    seen_header = False
-    for line in text.splitlines():
-        if "\t" not in line:
-            continue
-        cols = [c.strip().strip('"') for c in line.split("\t")]
-        cols = [c for c in cols if c != ""]
-        if not seen_header:
-            if any(c in ("Tran", "Vert", "Long") for c in cols):
-                seen_header = True
-            continue
-        if len(cols) < 7:
-            continue
-        if not re.match(r"^\d{1,2}:\d{2}:\d{2}$", cols[0]):
-            continue
-        def num(s):
-            try:
-                return float(s)
-            except ValueError:
-                return None
-        rows.append({"time": cols[0], "Tran": num(cols[1]),
-                     "Vert": num(cols[3]), "Long": num(cols[5])})
-    return rows
-
-
-def check_one(ascii_path_str: str) -> dict:
-    ap = Path(ascii_path_str)
-    bp = binary_for(ap)
-    res = {"ascii": ap.name, "binary": bp.name, "status": "?",
-           "kind": None, "detail": ""}
-    try:
-        if not bp.exists():
-            res["status"] = "no_binary"
-            return res
-        text = ap.read_text(errors="replace")
-        rep = parse_report(text, parse_samples=True)
-        ev = read_blastware_file(bp)
-        gr = rep.geo_range_ips
-        res["kind"] = kind = ("histogram"
-                              if (rep.event_type or "").lower().startswith(("full histogram", "histogram"))
-                              else "waveform")
-        samples = ev.raw_samples or {}
-        dec_n = {c: len(samples.get(c) or []) for c in GEO}
-
-        if kind == "histogram":
-            rows = parse_interval_table(text)
-            res["n_ascii"] = len(rows)
-            res["n_decoded"] = dec_n["Tran"]
-            if not rows:
-                res["status"] = "no_ascii_table"
-                return res
-            if dec_n["Tran"] == 0:
-                res["status"] = "decode_empty"
-                return res
-            if dec_n["Tran"] != len(rows):
-                res["status"] = "count_mismatch"
-                res["detail"] = f"decoded {dec_n['Tran']} vs ascii {len(rows)}"
-                return res
-            bad = 0
-            worst = 0.0
-            for i, row in enumerate(rows):
-                for ch in GEO:
-                    exp = row[ch]
-                    if exp is None:
-                        continue
-                    got = counts_to_ips(samples[ch][i], gr)
-                    if not agrees(got, exp, gr):
-                        bad += 1
-                        worst = max(worst, abs(got - exp))
-            res["worst_abs"] = round(worst, 6)
-            res["status"] = "exact" if bad == 0 else "value_mismatch"
-            if bad:
-                res["detail"] = f"{bad} interval-channel values off"
-            return res
-
-        # waveform
-        asc = rep.samples or []
-        res["n_ascii"] = len(asc)
-        res["n_decoded"] = dec_n["Tran"]
-        if not asc:
-            res["status"] = "no_ascii_table"
-            return res
-        if dec_n["Tran"] == 0:
-            res["status"] = "decode_empty"
-            return res
-        if len({dec_n[c] for c in GEO}) != 1:
-            res["status"] = "channel_len_mismatch"
-            res["detail"] = str(dec_n)
-            return res
-        if dec_n["Tran"] != len(asc):
-            res["status"] = "count_mismatch"
-            res["detail"] = f"decoded {dec_n['Tran']} vs ascii {len(asc)}"
-            return res
-        bad = 0
-        worst = 0.0
-        for i, quad in enumerate(asc):
-            for j, ch in enumerate(GEO):
-                exp = quad[j]
-                got = counts_to_ips(samples[ch][i], gr)
-                if not agrees(got, exp, gr):
-                    bad += 1
-                    worst = max(worst, abs(got - exp))
-        res["worst_abs"] = round(worst, 6)
-        res["status"] = "exact" if bad == 0 else "value_mismatch"
-        if bad:
-            res["detail"] = f"{bad} sample values off"
-        return res
-    except Exception as e:
-        res["status"] = "error"
-        res["detail"] = f"{type(e).__name__}: {e}"
-        return res
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--dir", required=True)
-    ap.add_argument("--limit", type=int, default=0)
-    ap.add_argument("--jobs", type=int, default=8)
-    ap.add_argument("--kind", choices=["w", "h", "all"], default="all")
-    ap.add_argument("--out", default=None)
-    a = ap.parse_args()
-
-    root = Path(a.dir)
-    files = sorted(p for p in root.rglob("*")
-                   if p.is_file() and p.name.upper().endswith("_ASCII.TXT"))
-    if a.kind != "all":
-        want = "0W" if a.kind == "w" else "0H"
-        files = [p for p in files
-                 if _ASCII_SUFFIX_RE.sub("", p.name).upper().endswith(want)]
-    if a.limit:
-        files = files[: a.limit]
-    print(f"pairs to check: {len(files)}", flush=True)
-
-    out = []
-    from collections import Counter
-    tally = Counter()
-    with ProcessPoolExecutor(max_workers=a.jobs) as ex:
-        futs = {ex.submit(check_one, str(p)): p for p in files}
-        for n, f in enumerate(as_completed(futs), 1):
-            r = f.result()
-            out.append(r)
-            tally[(r["kind"], r["status"])] += 1
-            if n % 500 == 0:
-                print(f"  {n}/{len(files)}", flush=True)
-
-    print("\n=== results ===")
-    for (kind, status), n in sorted(tally.items(), key=lambda x: -x[1]):
-        print(f"  {str(kind):10} {status:22} {n}")
-    if a.out:
-        Path(a.out).write_text(json.dumps(out, indent=1))
-        print(f"\nwrote {a.out}")
-
-
-if __name__ == "__main__":
-    main()

scratch/verify_thor_against_csv.py

@@ -1,228 +0,0 @@
-#!/usr/bin/env python3
-"""Verify the Thor / Micromate (series-4) IDF decoder against Thor's own exports.
-
-Sister harness to ``scratch/verify_against_ascii.py`` (series-3 / Blastware).
-
-Ground truth is the ``.IDFW.csv`` / ``.IDFH.csv`` file Thor writes next to each
-binary, under a sibling ``CSV/`` directory:
-
-    <dir>/UM13981_20220207084555.IDFW
-    <dir>/CSV/UM13981_20220207084555.IDFW.csv
-
-For waveforms the CSV carries a per-sample block of four columns
-(Tran, Vert, Long, Mic) in in/s and psi -- i.e. true per-sample ground truth,
-exactly what the BW ASCII exports give us for series-3.  The leading 2-column
-rows are the report header (PPV, sample rate, geo range, ...).
-
-Usage:
-    python scratch/verify_thor_against_csv.py [--root DIR] [--lsb FLOAT]
-                                              [--limit N] [--kind idfw|idfh|both]
-"""
-from __future__ import annotations
-
-import argparse
-import csv
-import os
-import statistics
-import sys
-from collections import Counter, defaultdict
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-from micromate import idf_file as M
-
-DEFAULT_ROOT = "/home/serversdown/thor-watcher/example-data"
-GEO = ("Tran", "Vert", "Long")
-
-
-def parse_export(path):
-    """Return (header_dict, sample_rows) from a Thor CSV export."""
-    hdr, rows = {}, []
-    with open(path, newline="", encoding="utf-8", errors="replace") as fh:
-        for rec in csv.reader(fh):
-            if len(rec) == 2:
-                hdr[rec[0].strip()] = rec[1].strip()
-            elif len(rec) >= 3:
-                try:
-                    rows.append([float(x) for x in rec])
-                except ValueError:
-                    pass
-    return hdr, rows
-
-
-def index_corpus(root):
-    """Map BASENAME.IDFW -> (binary_path, csv_path) for every paired file."""
-    exports, binaries = {}, {}
-    for dirpath, _dirs, files in os.walk(root):
-        for name in files:
-            up = name.upper()
-            full = os.path.join(dirpath, name)
-            if up.endswith(".IDFW.CSV") or up.endswith(".IDFH.CSV"):
-                exports.setdefault(name[:-4].upper(), full)
-            elif up.endswith(".IDFW") or up.endswith(".IDFH"):
-                binaries.setdefault(up, full)
-    return {k: (binaries[k], exports[k]) for k in binaries.keys() & exports.keys()}
-
-
-def hdr_float(hdr, key):
-    raw = hdr.get(key)
-    if not raw:
-        return None
-    try:
-        return float(raw.split()[0])
-    except (ValueError, IndexError):
-        return None
-
-
-def verify_waveform(binpath, csvpath, lsb):
-    """Compare one IDFW against its export.  Returns a result dict."""
-    out = {"file": os.path.basename(binpath), "status": "ok"}
-    try:
-        res = M.read_idf_file(binpath)
-    except NotImplementedError:
-        out["status"] = "not-thor"
-        return out
-    except Exception as exc:  # noqa: BLE001 - harness reports, never raises
-        out["status"] = "decode-error"
-        out["detail"] = f"{type(exc).__name__}: {exc}"
-        return out
-
-    hdr, rows = parse_export(csvpath)
-    if not rows:
-        out["status"] = "no-gt-samples"
-        return out
-
-    gt = {ch: [r[i] for r in rows] for i, ch in enumerate(GEO)}
-    out["gt_len"] = len(rows)
-    out["geo_range"] = hdr.get("GeoRange")
-
-    exact = total = 0
-    lens, chan_status = {}, {}
-    ppv_err = {}
-    for ch in GEO:
-        arr = res.samples.get(ch, [])
-        ref = gt[ch]
-        lens[ch] = len(arr)
-        if len(arr) != len(ref):
-            chan_status[ch] = "length"
-            continue
-        if not arr:
-            chan_status[ch] = "empty"
-            continue
-        hits = sum(1 for c, v in zip(arr, ref) if abs(c * lsb - v) < 5e-5)
-        exact += hits
-        total += len(arr)
-        chan_status[ch] = "exact" if hits == len(arr) else "value"
-        gp = hdr_float(hdr, f"{ch}PPV")
-        if gp:
-            ppv_err[ch] = (max(abs(c) for c in arr) * lsb - gp) / gp
-
-    out["lens"] = lens
-    out["chan_status"] = chan_status
-    out["exact"] = exact
-    out["total"] = total
-    out["ppv_err"] = ppv_err
-    if all(v == "exact" for v in chan_status.values()):
-        out["status"] = "exact"
-    elif any(v == "length" for v in chan_status.values()):
-        out["status"] = "length-mismatch"
-    else:
-        out["status"] = "value-mismatch"
-    return out
-
-
-def verify_histogram(binpath, csvpath, lsb):
-    out = {"file": os.path.basename(binpath), "status": "ok"}
-    try:
-        res = M.read_idf_file(binpath)
-    except NotImplementedError:
-        out["status"] = "not-thor"
-        return out
-    except Exception as exc:  # noqa: BLE001
-        out["status"] = "decode-error"
-        out["detail"] = f"{type(exc).__name__}: {exc}"
-        return out
-    hdr, _rows = parse_export(csvpath)
-    out["n_intervals"] = len(res.intervals or [])
-    errs = {}
-    for ch, attr in (("Tran", "transverse_ips"), ("Vert", "vertical_ips"),
-                     ("Long", "longitudinal_ips")):
-        gp = hdr_float(hdr, f"{ch}PPV")
-        dv = getattr(res.event.peaks, attr, None)
-        if gp and dv:
-            errs[ch] = (dv - gp) / gp
-    out["ppv_err"] = errs
-    out["status"] = "peaks" if errs else "no-gt-peaks"
-    return out
-
-
-def main():
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--root", default=DEFAULT_ROOT)
-    ap.add_argument("--lsb", type=float, default=M._GEO_LSB_IPS)
-    ap.add_argument("--limit", type=int, default=0)
-    ap.add_argument("--kind", choices=("idfw", "idfh", "both"), default="both")
-    ap.add_argument("--show", type=int, default=15, help="worst-N detail rows")
-    args = ap.parse_args()
-
-    pairs = index_corpus(args.root)
-    keys = sorted(pairs)
-    if args.kind != "both":
-        keys = [k for k in keys if k.endswith(args.kind.upper())]
-    if args.limit:
-        keys = keys[: args.limit]
-
-    print(f"root: {args.root}")
-    print(f"geo LSB under test: {args.lsb!r} in/s per count")
-    print(f"paired files: {len(keys)}\n")
-
-    wf, hg = [], []
-    for k in keys:
-        binpath, csvpath = pairs[k]
-        if k.endswith(".IDFW"):
-            wf.append(verify_waveform(binpath, csvpath, args.lsb))
-        else:
-            hg.append(verify_histogram(binpath, csvpath, args.lsb))
-
-    if wf:
-        st = Counter(r["status"] for r in wf)
-        ex = sum(r.get("exact", 0) for r in wf)
-        tot = sum(r.get("total", 0) for r in wf)
-        print("=" * 68)
-        print(f"WAVEFORM (IDFW): {len(wf)} files")
-        for s, n in st.most_common():
-            print(f"    {s:16} {n:5d}  ({100*n/len(wf):5.1f}%)")
-        if tot:
-            print(f"  per-sample exact: {ex}/{tot} = {100*ex/tot:.3f}%")
-        errs = [e for r in wf for e in r.get("ppv_err", {}).values()]
-        if errs:
-            print(f"  PPV rel-error: median {statistics.median(errs):+.4%}  "
-                  f"mean {statistics.mean(errs):+.4%}  "
-                  f"max|.| {max(abs(e) for e in errs):.4%}")
-        bad = [r for r in wf if r["status"] not in ("exact",)]
-        if bad:
-            print(f"\n  worst {min(args.show, len(bad))} of {len(bad)} non-exact:")
-            for r in bad[: args.show]:
-                print(f"    {r['file']:42} {r['status']:16} "
-                      f"lens={r.get('lens')} gt={r.get('gt_len')} "
-                      f"{r.get('detail','')}")
-
-    if hg:
-        st = Counter(r["status"] for r in hg)
-        print("=" * 68)
-        print(f"HISTOGRAM (IDFH): {len(hg)} files")
-        for s, n in st.most_common():
-            print(f"    {s:16} {n:5d}  ({100*n/len(hg):5.1f}%)")
-        errs = [e for r in hg for e in r.get("ppv_err", {}).values()]
-        if errs:
-            print(f"  PPV rel-error: median {statistics.median(errs):+.4%}  "
-                  f"mean {statistics.mean(errs):+.4%}  "
-                  f"max|.| {max(abs(e) for e in errs):.4%}")
-            within = lambda t: 100*sum(1 for e in errs if abs(e) <= t)/len(errs)
-            print(f"  within 0.5%: {within(0.005):.1f}%   "
-                  f"within 2%: {within(0.02):.1f}%   within 5%: {within(0.05):.1f}%")
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

scripts/backfill_event_shape.py

@@ -1,74 +0,0 @@
-#!/usr/bin/env python3
-"""Backfill events.shape_* and shape_offset_* from each event's .h5 samples. Idempotent."""
-from __future__ import annotations
-import argparse, logging, sys
-from pathlib import Path
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-from sfm.database import SeismoDb
-from sfm.waveform_store import WaveformStore
-from sfm.shape_metrics import shape_from_h5, offset_from_h5
-
-log = logging.getLogger("backfill_event_shape")
-
-def backfill_shape(db: SeismoDb, store: WaveformStore, *, dry_run: bool = False) -> dict:
-    counts = {"updated": 0, "skipped_no_h5": 0, "skipped_no_samples": 0,
-              "cleared_stale": 0}
-    for row in db.query_events(limit=1_000_000):
-        serial, filename = row.get("serial"), row.get("blastware_filename")
-        if not serial or not filename:
-            counts["skipped_no_h5"] += 1; continue
-        h5_path = store.hdf5_path_for(serial, filename)
-        if not h5_path.exists():
-            counts["skipped_no_h5"] += 1; continue
-        shape = shape_from_h5(h5_path)
-        offset = offset_from_h5(h5_path)
-        if shape is None:
-            # The .h5 can no longer yield a shape (fewer than 2 samples, or a
-            # flat trace).  Clear any previously stored value rather than
-            # leaving it behind — a stale shape outlives the decode it came
-            # from and silently feeds the false-trigger detector.  Seen after
-            # a decoder fix shrinks an event: 493 rows in the prod snapshot
-            # were carrying metrics from a superseded decode (2026-08-25).
-            if (row.get("shape_crest_factor") is not None
-                    or row.get("shape_offset") is not None):
-                if not dry_run:
-                    with db._connect() as conn:
-                        conn.execute(
-                            "UPDATE events SET shape_crest_factor=NULL, "
-                            "shape_near_peak_count=NULL, shape_sample_count=NULL, "
-                            "shape_axis=NULL, shape_offset=NULL, shape_offset_axis=NULL, "
-                            "shape_offset_pre=NULL, shape_offset_spread=NULL WHERE id=?",
-                            (row["id"],))
-                counts["cleared_stale"] += 1
-            counts["skipped_no_samples"] += 1; continue
-        if not dry_run:
-            with db._connect() as conn:
-                conn.execute(
-                    "UPDATE events SET shape_crest_factor=?, shape_near_peak_count=?, "
-                    "shape_sample_count=?, shape_axis=?, shape_offset=?, "
-                    "shape_offset_axis=?, shape_offset_pre=?, shape_offset_spread=? "
-                    "WHERE id=?",
-                    (shape["crest_factor"], shape["near_peak_count"],
-                     shape["sample_count"], shape["axis"],
-                     (1 if offset["offset"] else 0) if offset else None,
-                     offset["axis"] if offset else None,
-                     offset["pre"] if offset else None,
-                     offset["spread"] if offset else None,
-                     row["id"]))
-        counts["updated"] += 1
-    log.info("backfill_shape: %s", counts)
-    return counts
-
-def main(argv=None) -> int:
-    ap = argparse.ArgumentParser()
-    ap.add_argument("--db-path", required=True)
-    ap.add_argument("--store-root", required=True)
-    ap.add_argument("--dry-run", action="store_true")
-    a = ap.parse_args(argv)
-    logging.basicConfig(level=logging.INFO)
-    counts = backfill_shape(SeismoDb(a.db_path), WaveformStore(a.store_root), dry_run=a.dry_run)
-    print(counts)
-    return 0
-
-if __name__ == "__main__":
-    raise SystemExit(main())

scripts/backfill_event_zc_freq.py

@@ -1,98 +0,0 @@
-#!/usr/bin/env python3
-"""Backfill events.*_zc_freq / *_zc_above_range from .sfm.json sidecars.
-
-Idempotent. Reads each event's sidecar bw_report and writes the per-channel
-ZC frequency into the DB columns via a direct UPDATE. Events with no sidecar
-(or no bw_report freq) are left untouched. Mirrors scripts/backfill_record_type.py.
-"""
-from __future__ import annotations
-import argparse
-import logging
-import sys
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from sfm.database import SeismoDb
-from sfm.waveform_store import WaveformStore
-
-log = logging.getLogger("backfill_event_zc_freq")
-
-
-def _freqs_from_sidecar(sidecar: dict) -> dict | None:
-    bw = (sidecar or {}).get("bw_report") or {}
-    peaks = bw.get("peaks") or {}
-    mic = bw.get("mic") or {}
-    out = {}
-    for axis in ("tran", "vert", "long"):
-        chd = peaks.get(axis) or {}
-        out[f"{axis}_zc_freq"] = chd.get("zc_freq_hz")
-        out[f"{axis}_zc_above_range"] = 1 if chd.get("zc_freq_above_range") else 0
-    out["mic_zc_freq"] = mic.get("zc_freq_hz")
-    out["mic_zc_above_range"] = 1 if mic.get("zc_freq_above_range") else 0
-    # None if there's literally no frequency anywhere.
-    if all(out[f"{a}_zc_freq"] is None for a in ("tran", "vert", "long", "mic")):
-        return None
-    return out
-
-
-def backfill_zc_freq(db: SeismoDb, store, *, dry_run: bool = False) -> dict:
-    counts = {"updated": 0, "skipped_no_sidecar": 0, "skipped_no_freq": 0}
-    # Pull every event (paginate generously; adjust if a store has >100k events).
-    rows = db.query_events(limit=1_000_000)
-    for row in rows:
-        serial = row.get("serial")
-        filename = row.get("blastware_filename")
-        if not serial or not filename:
-            counts["skipped_no_sidecar"] += 1
-            continue
-        sidecar = store.load_sidecar(serial, filename)
-        if sidecar is None:
-            counts["skipped_no_sidecar"] += 1
-            continue
-        freqs = _freqs_from_sidecar(sidecar)
-        if freqs is None:
-            counts["skipped_no_freq"] += 1
-            continue
-        if dry_run:
-            counts["updated"] += 1
-            continue
-        with db._connect() as conn:
-            conn.execute(
-                """UPDATE events SET
-                     tran_zc_freq=?, vert_zc_freq=?, long_zc_freq=?, mic_zc_freq=?,
-                     tran_zc_above_range=?, vert_zc_above_range=?,
-                     long_zc_above_range=?, mic_zc_above_range=?
-                   WHERE id=?""",
-                (freqs["tran_zc_freq"], freqs["vert_zc_freq"], freqs["long_zc_freq"],
-                 freqs["mic_zc_freq"], freqs["tran_zc_above_range"],
-                 freqs["vert_zc_above_range"], freqs["long_zc_above_range"],
-                 freqs["mic_zc_above_range"], row["id"]),
-            )
-        counts["updated"] += 1
-    return counts
-
-
-def main(argv=None) -> int:
-    p = argparse.ArgumentParser(description=__doc__)
-    p.add_argument("--db-path", default=str(
-        Path(__file__).resolve().parent.parent / "bridges" / "captures" / "seismo_relay.db"))
-    p.add_argument("--store-root", default=None)
-    p.add_argument("--dry-run", action="store_true")
-    p.add_argument("-v", "--verbose", action="store_true")
-    args = p.parse_args(argv)
-    logging.basicConfig(level=logging.DEBUG if args.verbose else logging.INFO)
-    db_path = Path(args.db_path).expanduser().resolve()
-    store_root = (
-        Path(args.store_root).expanduser().resolve()
-        if args.store_root else db_path.parent / "waveforms"
-    )
-    db = SeismoDb(db_path)
-    store = WaveformStore(store_root)
-    counts = backfill_zc_freq(db, store, dry_run=args.dry_run)
-    log.info("backfill_zc_freq: %s", counts)
-    return 0
-
-
-if __name__ == "__main__":
-    raise SystemExit(main())

scripts/backfill_sidecars.py

@@ -136,7 +136,6 @@ def main(argv=None) -> int:
     db    = SeismoDb(db_path)
 
     written = skipped = errors = 0
-    stale_h5_removed = 0
     for serial_dir in sorted(p for p in store_root.iterdir() if p.is_dir()):
         serial = serial_dir.name
         for path in sorted(serial_dir.iterdir()):
@@ -389,16 +388,9 @@ def main(argv=None) -> int:
                 # waveform_codec.decode_waveform_v2 or histogram_codec.
                 # decode_histogram_body.  If samples are still empty after
                 # both codecs run, it's a genuine "we can't decode this
-                # file" case (truncated, malformed, or unknown mode).
+                # file" case (truncated, malformed, or unknown mode);
-                #
+                # skip the .h5 write so we don't replace whatever's
-                # In that case we REMOVE any existing .h5 rather than leave
+                # there with an empty placeholder.
-                # it.  Leaving it was the old behaviour and it silently
-                # preserved output from a superseded decoder: after the
-                # 2026-08-25 record-chain fix, 415 histogram files stopped
-                # decoding (an unmapped block variant on BE18193/BE9440) but
-                # kept .h5 files whose peaks were up to 400x the device's own
-                # reported PPV — garbage that fed the charts and the
-                # false-trigger detector with nothing marking it.
                 has_samples = bool(
                     ev.raw_samples and any(
                         ev.raw_samples.get(ch) for ch in ("Tran", "Vert", "Long", "MicL")
@@ -406,13 +398,6 @@ def main(argv=None) -> int:
                 )
                 hdf5_path = store.hdf5_path_for(serial, path.name)
                 hdf5_filename = hdf5_path.name if hdf5_path.exists() else None
-                if not has_samples and hdf5_path.exists():
-                    log.warning("%s: no samples decode — removing stale %s",
-                                path.name, hdf5_path.name)
-                    if not args.dry_run:
-                        hdf5_path.unlink()
-                    hdf5_filename = None
-                    stale_h5_removed += 1
                 hdf5_action = "kept"
                 need_h5 = (
                     not args.skip_hdf5
@@ -473,7 +458,7 @@ def main(argv=None) -> int:
                 log.error("backfill failed for %s: %s", path, exc, exc_info=args.verbose)
                 errors += 1
 
-    print(f"\nDone.  written={written}  skipped(uptodate)={skipped}  errors={errors}  stale_h5_removed={stale_h5_removed}")
+    print(f"\nDone.  written={written}  skipped(uptodate)={skipped}  errors={errors}")
     return 0 if errors == 0 else 1
 
 

scripts/backfill_thor_events.py

@@ -1,336 +0,0 @@
-"""
-scripts/backfill_thor_events.py — re-process existing Thor (Series IV)
-events so their sidecars carry the bw_report block produced by
-``micromate.idf_to_bw_report.build_bw_report_from_idf`` + their .h5
-clean-waveform files for IDFW events.
-
-Why this exists
-───────────────
-
-Thor events ingested before v0.21.0 (or during the v0.21.0 ingest bug
-window fixed in commit bee1185) have sidecars with only
-``extensions.idf_report`` — no ``bw_report`` block.  Without
-``bw_report``, the SFM PDF renderer falls back to DB-only fields
-(misses sensor-self-check, full per-channel breakdown, mic dB(L)),
-and the modal chart 404s on ``/waveform.json`` for IDFW events
-because no .h5 was written when the codec failed at ingest.
-
-Re-forwarding from thor-watcher would also fix this, but that requires
-operator coordination on every watcher machine and uses bandwidth this
-script doesn't.
-
-What this does
-──────────────
-
-Walks ``<store>/<serial>/<filename>`` for ``.IDFW`` / ``.IDFH`` files
-and, for each one:
-
-  1. Reads the existing sidecar (preserving review state + captured_at).
-  2. Re-runs ``micromate.idf_file.read_idf_file()`` on the binary
-     bytes — passing ``data=`` so the codec doesn't try to read from
-     a path it doesn't know.
-  3. Pulls ``extensions.idf_report`` (the raw parsed Thor dict the
-     v0.18.0+ ingest path already stashed) and runs the v0.21.0
-     ``build_bw_report_from_idf`` adapter against it.
-  4. Writes the refreshed sidecar with the new ``bw_report``,
-     bumped ``source.tool_version``, but preserved ``review`` block
-     + the original ``captured_at`` timestamp.
-  5. Regenerates the .h5 waveform file via the existing
-     ``event_hdf5`` writer.  For IDFW that's the decoded per-sample
-     stream; for IDFH it's a 1-sample-per-interval synthesised array
-     (peak ADC count per channel) so the renderer's bar-chart code
-     has data to group on.  Mic peak psi from the binary is merged
-     onto the IdfEvent before the bridge so the h5 writer's per-count
-     mic scale factor lands on a sensible value (without this the
-     mic chart on Thor events plots dB(L)-as-pseudo-psi and shows
-     bomb-level numbers).
-
-Idempotent.  Re-running it after a parser/adapter change just
-re-writes sidecars — no DB writes, no thor-watcher coordination.
-
-Usage
-─────
-
-    python scripts/backfill_thor_events.py [--store-root PATH]
-                                           [--dry-run]
-                                           [--skip-hdf5]
-                                           [--force]
-                                           [-v]
-
-By default, refreshes any Thor event whose sidecar is missing
-``bw_report`` OR whose ``source.tool_version`` is older than the
-current ``TOOL_VERSION``.  ``--force`` refreshes every Thor event
-regardless.
-"""
-
-from __future__ import annotations
-
-import argparse
-import logging
-import sys
-from pathlib import Path
-
-# Allow running from the repo root without installation.
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
-
-from minimateplus import event_file_io
-from sfm.waveform_store import WaveformStore
-
-log = logging.getLogger("backfill_thor_events")
-
-
-def _is_thor_event(path: Path) -> bool:
-    if not path.is_file():
-        return False
-    if path.name.endswith((".sfm.json", ".h5", "_ASCII.TXT")):
-        return False
-    return path.suffix.upper() in (".IDFW", ".IDFH")
-
-
-def _vtuple(s: str) -> tuple:
-    try:
-        return tuple(int(p) for p in str(s).split(".")[:3])
-    except Exception:
-        return (0, 0, 0)
-
-
-def main(argv=None) -> int:
-    p = argparse.ArgumentParser(description=__doc__)
-    p.add_argument(
-        "--db-path",
-        default=str(Path(__file__).resolve().parent.parent / "bridges" / "captures" / "seismo_relay.db"),
-        help="Used only to derive the default --store-root.",
-    )
-    p.add_argument("--store-root", default=None)
-    p.add_argument("--dry-run", action="store_true")
-    p.add_argument("--skip-hdf5", action="store_true",
-                   help="Don't regenerate .h5 files for IDFW events.")
-    p.add_argument("--force", action="store_true",
-                   help="Refresh every Thor event, not just ones with stale or missing bw_report.")
-    p.add_argument("-v", "--verbose", action="store_true")
-    args = p.parse_args(argv)
-
-    logging.basicConfig(
-        level=logging.DEBUG if args.verbose else logging.INFO,
-        format="%(asctime)s  %(levelname)-7s  %(name)s  %(message)s",
-        datefmt="%H:%M:%S",
-    )
-
-    db_path = Path(args.db_path).expanduser().resolve()
-    store_root = (
-        Path(args.store_root).expanduser().resolve()
-        if args.store_root else db_path.parent / "waveforms"
-    )
-    if not store_root.exists():
-        log.error("store root not found: %s", store_root)
-        return 1
-    store = WaveformStore(store_root)
-    log.info("store root: %s", store_root)
-    log.info("current TOOL_VERSION: %s", event_file_io.TOOL_VERSION)
-
-    refreshed = skipped = errors = h5_written = 0
-
-    # Lazy imports so any one of these failing produces a useful error
-    # message rather than crashing module-load.
-    from micromate.idf_file import read_idf_file
-    from micromate.idf_to_bw_report import build_bw_report_from_idf
-
-    for serial_dir in sorted(p for p in store_root.iterdir() if p.is_dir()):
-        serial = serial_dir.name
-        for path in sorted(serial_dir.iterdir()):
-            if not _is_thor_event(path):
-                continue
-
-            sidecar_path = store.sidecar_path_for(serial, path.name)
-            if not sidecar_path.exists():
-                log.debug("%s: no sidecar — skipping (this is a binary without ingest history)",
-                          path.name)
-                skipped += 1
-                continue
-
-            try:
-                existing = event_file_io.read_sidecar(sidecar_path)
-            except Exception as exc:
-                log.warning("%s: failed to read sidecar — %s", path.name, exc)
-                errors += 1
-                continue
-
-            has_bw_report = bool(existing.get("bw_report"))
-            existing_version = (existing.get("source") or {}).get("tool_version", "")
-            up_to_date = (
-                has_bw_report
-                and _vtuple(existing_version) >= _vtuple(event_file_io.TOOL_VERSION)
-            )
-            if up_to_date and not args.force:
-                skipped += 1
-                continue
-
-            # Re-decode the binary.  Catch + log; continue with .txt-only
-            # data if it fails (matches the live ingest path's behavior).
-            idf_samples = None
-            idf_intervals = None
-            binary_md = None
-            is_histogram = path.suffix.upper() == ".IDFH"
-            try:
-                binary_bytes = path.read_bytes()
-                res = read_idf_file(path, data=binary_bytes)
-                idf_samples = res.samples or None
-                idf_intervals = res.intervals
-                binary_md = res.binary_metadata
-                is_histogram = res.intervals is not None
-            except NotImplementedError:
-                # sig-B / Blastware-stray binary; no samples but adapter
-                # can still produce a bw_report from extensions.idf_report.
-                log.debug("%s: binary codec NotImplementedError (sig-B / BW-stray); proceeding from sidecar's idf_report only", path.name)
-            except Exception as exc:
-                log.warning("%s: binary decode failed — %s; proceeding from sidecar's idf_report only", path.name, exc)
-
-            # Run the adapter.  Pull report_dict from
-            # extensions.idf_report (the v0.18.0+ ingest preserved it).
-            report_dict = (existing.get("extensions") or {}).get("idf_report") or {}
-            if not report_dict and binary_md is None:
-                log.debug("%s: no idf_report in sidecar AND no binary metadata — nothing to project", path.name)
-                skipped += 1
-                continue
-
-            try:
-                bw_report = build_bw_report_from_idf(
-                    report_dict, binary_md=binary_md,
-                    intervals=idf_intervals, is_histogram=is_histogram,
-                )
-            except Exception as exc:
-                log.warning("%s: adapter failed — %s", path.name, exc)
-                errors += 1
-                continue
-
-            # Build the new sidecar by overlaying refreshed fields onto
-            # the existing one — preserves review, captured_at, blastware
-            # block, source.kind, etc.
-            new_sidecar = dict(existing)  # shallow copy
-            new_sidecar["bw_report"] = bw_report
-            src = dict(new_sidecar.get("source") or {})
-            src["tool_version"] = event_file_io.TOOL_VERSION
-            new_sidecar["source"] = src
-
-            # Preserve histogram intervals if the binary decoded them
-            # (improves over the original ingest if that one ran before
-            # the bee1185 codec fix).
-            if idf_intervals is not None:
-                ext = dict(new_sidecar.get("extensions") or {})
-                ext["idf_intervals"] = [
-                    {
-                        "offset":     iv.offset,
-                        "tran_peak":  iv.peak_count("Tran"),
-                        "tran_halfp": iv.tran_halfp,
-                        "tran_freq":  iv.freq_hz("Tran"),
-                        "vert_peak":  iv.peak_count("Vert"),
-                        "vert_halfp": iv.vert_halfp,
-                        "vert_freq":  iv.freq_hz("Vert"),
-                        "long_peak":  iv.peak_count("Long"),
-                        "long_halfp": iv.long_halfp,
-                        "long_freq":  iv.freq_hz("Long"),
-                        "mic_peak":   iv.peak_count("MicL"),
-                        "mic_halfp":  iv.micl_halfp,
-                        "mic_freq":   iv.freq_hz("MicL"),
-                    }
-                    for iv in idf_intervals
-                ]
-                new_sidecar["extensions"] = ext
-
-            if args.dry_run:
-                will_write_h5 = (idf_samples or idf_intervals) and not args.skip_hdf5
-                log.info("[DRY] %s/%s — would refresh sidecar (bw_report=%s, h5=%s)",
-                         serial, path.name,
-                         "wrote" if not has_bw_report else "refreshed",
-                         "would write" if will_write_h5 else "skipped")
-            else:
-                event_file_io.write_sidecar(sidecar_path, new_sidecar)
-                log.info("%s/%s — sidecar refreshed (bw_report=%s, intervals=%d)",
-                         serial, path.name,
-                         "added" if not has_bw_report else "refreshed",
-                         len(idf_intervals) if idf_intervals else 0)
-            refreshed += 1
-
-            # Regenerate .h5 by replaying the same IdfEvent → Event bridge
-            # save_imported_idf uses.  For IDFW we write the decoded per-
-            # sample arrays.  For IDFH we synthesise a 1-sample-per-interval
-            # array (peak ADC count per channel per interval) so the
-            # renderer's bar-chart code has something to group on.
-            # Pre-condition: either real samples (IDFW) or decoded intervals
-            # (IDFH).  Skip otherwise.
-            have_data = bool(idf_samples) or bool(idf_intervals)
-            if have_data and not args.skip_hdf5:
-                from sfm import event_hdf5
-                hdf5_path = store.hdf5_path_for(serial, path.name)
-                if args.dry_run:
-                    log.debug("[DRY] would write %s", hdf5_path.name)
-                else:
-                    try:
-                        from micromate import IdfEvent
-                        from minimateplus.event_file_io import file_sha256
-                        idf_event = IdfEvent.from_report(report_dict, path.name)
-
-                        # Merge the binary-derived mic peak psi (only the
-                        # binary path knows the proper psi value; the .txt
-                        # carries dB(L)).  Without this, the h5 writer's
-                        # per-count mic factor is computed against the
-                        # dB(L) value-as-pseudo-psi and the mic chart
-                        # scales wildly.
-                        if (binary_md is not None and res is not None
-                                and res.event.peaks.mic_pspl_psi is not None):
-                            idf_event.peaks.mic_pspl_psi = res.event.peaks.mic_pspl_psi
-
-                        sha256 = file_sha256(path)
-                        waveform_key = bytes.fromhex(sha256)[:16]
-                        ev = idf_event.to_minimateplus_event(waveform_key)
-
-                        if is_histogram and idf_intervals:
-                            # 1 sample per interval per channel — same
-                            # synthesis save_imported_idf uses.  The h5
-                            # writer's count×geo_fs/32768 conversion turns
-                            # each peak-ADC-count into the bar's physical
-                            # value.
-                            ev.raw_samples = {
-                                "Tran": [iv.peak_count("Tran") for iv in idf_intervals],
-                                "Vert": [iv.peak_count("Vert") for iv in idf_intervals],
-                                "Long": [iv.peak_count("Long") for iv in idf_intervals],
-                                "MicL": [iv.peak_count("MicL") for iv in idf_intervals],
-                            }
-                            ev.total_samples = ev.total_samples or len(idf_intervals)
-                        elif idf_samples:
-                            ev.raw_samples = idf_samples
-                            n_samp = max(
-                                (len(idf_samples.get(ch, []))
-                                 for ch in ("Tran", "Vert", "Long", "MicL")),
-                                default=0,
-                            )
-                            ev.total_samples = ev.total_samples or n_samp
-                            # Sensor self-check traces from the IDFW fixed
-                            # header, so regenerated .h5 files gain the v2
-                            # /sensor_check group (mirrors save_imported_idf).
-                            from micromate.sensor_check import decode_idf_sensor_check
-                            ev.sensor_check = decode_idf_sensor_check(binary_bytes) or None
-
-                        event_hdf5.write_event_hdf5(
-                            hdf5_path, ev,
-                            serial=serial,
-                            geo_range="normal",
-                            source_kind="idf-import",
-                            tool_version=event_file_io.TOOL_VERSION,
-                        )
-                        h5_written += 1
-                        log.debug("%s/%s — .h5 written (%s)",
-                                  serial, path.name,
-                                  f"{len(idf_intervals)} intervals" if is_histogram
-                                  else f"{sum(len(v) for v in (idf_samples or {}).values())} samples")
-                    except Exception as exc:
-                        log.warning("%s/%s — .h5 write failed: %s",
-                                    serial, path.name, exc)
-
-    log.info("Done.  refreshed=%d  skipped=%d  errors=%d  h5_written=%d",
-             refreshed, skipped, errors, h5_written)
-    return 0 if errors == 0 else 2
-
-
-if __name__ == "__main__":
-    sys.exit(main())

scripts/test_thor_render.py

@@ -1,91 +0,0 @@
-"""Re-ingest a prod IDFW + IDFH via the patched save_imported_idf and
-render both PDFs to confirm charts have data."""
-from __future__ import annotations
-import sys
-import json
-import datetime
-import tempfile
-from pathlib import Path
-
-sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
-
-from sfm.waveform_store import WaveformStore
-from sfm import report_pdf
-import h5py
-
-
-class FakeDb:
-    def __init__(self, event):
-        self.event = event
-    def get_event(self, _id):
-        return self.event
-
-
-def to_ts_iso(ts):
-    if ts is None:
-        return None
-    try:
-        return datetime.datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second).isoformat()
-    except Exception:
-        return None
-
-
-def render_case(idf_path: Path, serial: str, out_pdf: Path, h5_summary: bool = True):
-    with tempfile.TemporaryDirectory() as td:
-        store = WaveformStore(Path(td))
-        ev, rec = store.save_imported_idf(
-            idf_path.read_bytes(),
-            idf_path,
-            idf_report_text=None,    # production worst case: no .txt
-        )
-        print(f"=== {idf_path.name} ===")
-        print(f"  h5: {rec['hdf5_filename']}, sidecar: {rec['sidecar_filename']}")
-
-        h5p = Path(td) / serial / f"{idf_path.name}.h5"
-        if h5p.exists() and h5_summary:
-            with h5py.File(h5p) as h:
-                for ch in ("Tran", "Vert", "Long", "MicL"):
-                    ds = h.get(f"samples/{ch}")
-                    if ds is not None:
-                        n = ds.shape[0]
-                        mx = float(abs(ds[...]).max()) if n else 0
-                        print(f"  samples/{ch}: n={n}  max_abs={mx:.5f}")
-
-        record_type = "Histogram" if idf_path.suffix.upper() == ".IDFH" else "Waveform"
-        fake_row = {
-            "serial":              serial,
-            "blastware_filename":  rec["filename"],
-            "record_type":         record_type,
-            "timestamp":           to_ts_iso(ev.timestamp),
-            "sample_rate":         ev.sample_rate,
-            "project":             ev.project_info.project if ev.project_info else None,
-            "client":              ev.project_info.client if ev.project_info else None,
-            "operator":            ev.project_info.operator if ev.project_info else None,
-            "sensor_location":     ev.project_info.sensor_location if ev.project_info else None,
-            "created_at":          None,
-        }
-        rd = report_pdf.gather_report_data(FakeDb(fake_row), store, event_id="test-1")
-        print(f"  ReportData: channels={ {k: len(v) for k,v in rd.channels.items()} }")
-        if rd.is_histogram:
-            print(f"  histogram n_intervals={rd.histogram_n_intervals} interval_size={rd.histogram_interval_size}")
-        pdf = report_pdf.render_event_report_pdf(rd)
-        out_pdf.write_bytes(pdf)
-        print(f"  PDF: {out_pdf}  ({len(pdf)} bytes)")
-
-
-def main():
-    out_dir = Path("/tmp/thor_render_test"); out_dir.mkdir(exist_ok=True)
-    cases = [
-        # IDFW that decoded to preamble-only under the old codec
-        ("/home/serversdown/seismo-relay-prod-snap/waveforms/UM6047/UM6047_20250804154137.IDFW", "UM6047"),
-        # IDFW that worked under the old codec (validates no regression)
-        ("/home/serversdown/seismo-relay-prod-snap/waveforms/UM6047/UM6047_20250804104450.IDFW", "UM6047"),
-        # IDFH histogram
-        ("/home/serversdown/seismo-relay-prod-snap/waveforms/UM6047/UM6047_20250804190047.IDFH", "UM6047"),
-    ]
-    for path, serial in cases:
-        render_case(Path(path), serial, out_dir / f"{Path(path).name}.pdf")
-
-
-if __name__ == "__main__":
-    main()

seismo_lab.py

@@ -54,7 +54,6 @@ from s3_analyzer import (  # noqa: E402
     write_claude_export,
 )
 from frame_db import FrameDB  # noqa: E402
-from minimateplus.binary_annotate import annotate_blastware_binary  # noqa: E402
 
 # ── colour palette ────────────────────────────────────────────────────────────
 BG       = "#1e1e1e"
@@ -2676,95 +2675,6 @@ class DownloadPanel(tk.Frame):
         self._on_capture_ready(bw_path, s3_path, label)
 
 
-# ─────────────────────────────────────────────────────────────────────────────
-# Inspector panel — annotated hex view of a Series-3 binary
-# ─────────────────────────────────────────────────────────────────────────────
-
-class InspectorPanel(tk.Frame):
-    """Load any Series-3 waveform binary and read it as an annotated hex dump.
-
-    Regions the decoder understands (header, STRT, per-channel sample records,
-    footer) are labelled and colour-coded; everything the decoder cannot account
-    for is flagged UNKNOWN, so undecoded bytes stand out for hand-inspection.
-    """
-
-    _KIND_COLOR = {
-        "header":  ACCENT,
-        "strt":    YELLOW,
-        "sample":  COL_S3,
-        "footer":  FG_DIM,
-        "unknown": RED,
-    }
-
-    def __init__(self, parent: tk.Widget, initialdir=None, **kw) -> None:
-        super().__init__(parent, bg=BG, **kw)
-        self._path = None
-        self._initialdir = initialdir
-        self._build()
-
-    def _build(self) -> None:
-        bar = tk.Frame(self, bg=BG2)
-        bar.pack(side=tk.TOP, fill=tk.X)
-        tk.Button(bar, text="Open binary…", command=self._open, bg=BG3, fg=FG,
-                  relief=tk.FLAT, font=MONO, activebackground=ACCENT).pack(side=tk.LEFT, padx=6, pady=6)
-        self._path_var = tk.StringVar(value="(no file loaded)")
-        tk.Label(bar, textvariable=self._path_var, bg=BG2, fg=FG_DIM, font=MONO).pack(side=tk.LEFT, padx=6)
-        self._summary_var = tk.StringVar(value="")
-        tk.Label(bar, textvariable=self._summary_var, bg=BG2, fg=FG, font=MONO).pack(side=tk.RIGHT, padx=10)
-
-        legend = tk.Frame(self, bg=BG2)
-        legend.pack(side=tk.TOP, fill=tk.X)
-        tk.Label(legend, text="legend:", bg=BG2, fg=FG_DIM, font=MONO).pack(side=tk.LEFT, padx=(8, 2))
-        for kind, color in self._KIND_COLOR.items():
-            tk.Label(legend, text=f"■ {kind}", bg=BG2, fg=color, font=MONO).pack(side=tk.LEFT, padx=5, pady=2)
-
-        self._text = scrolledtext.ScrolledText(
-            self, bg=BG, fg=FG, insertbackground=FG, font=MONO, wrap=tk.NONE, borderwidth=0)
-        self._text.pack(side=tk.TOP, fill=tk.BOTH, expand=True)
-        for kind, color in self._KIND_COLOR.items():
-            self._text.tag_configure(kind, foreground=color)
-        self._text.tag_configure("label", foreground="#ffffff", font=("Consolas", 9, "bold"))
-        self._text.tag_configure("dim", foreground=FG_DIM)
-        self._text.configure(state=tk.DISABLED)
-
-    def _open(self) -> None:
-        p = filedialog.askopenfilename(title="Open a Series-3 binary", initialdir=self._initialdir)
-        if p:
-            self.load(Path(p))
-
-    def load(self, path: Path) -> None:
-        try:
-            raw = path.read_bytes()
-            spans = annotate_blastware_binary(raw)
-        except Exception as e:  # noqa: BLE001 — surface any read/annotate failure to the user
-            messagebox.showerror("Inspector", f"Failed to read/annotate:\n{path}\n\n{e}")
-            return
-        self._path = path
-        self._path_var.set(str(path))
-        self._render(raw, spans)
-
-    def _render(self, raw: bytes, spans) -> None:
-        t = self._text
-        t.configure(state=tk.NORMAL)
-        t.delete("1.0", tk.END)
-        unknown = sum(s.end - s.start for s in spans if s.kind == "unknown")
-        pct = 100 * unknown / max(1, len(raw))
-        self._summary_var.set(f"{len(raw)} B · {len(spans)} regions · {pct:.1f}% unknown")
-        for s in spans:
-            t.insert(tk.END, f"\n── {s.label}  [0x{s.start:04x}:0x{s.end:04x}]  {s.end - s.start} B ──\n", ("label",))
-            self._insert_hex(t, raw, s.start, s.end, s.kind)
-        t.configure(state=tk.DISABLED)
-
-    def _insert_hex(self, t: tk.Text, raw: bytes, start: int, end: int, kind: str) -> None:
-        for off in range(start, end, 16):
-            row = raw[off:min(off + 16, end)]
-            hx = " ".join(f"{b:02x}" for b in row).ljust(16 * 3 - 1)
-            txt = "".join(chr(b) if 32 <= b < 127 else "." for b in row)
-            t.insert(tk.END, f"  0x{off:04x}  ", ("dim",))
-            t.insert(tk.END, hx, (kind,))
-            t.insert(tk.END, f"  {txt}\n", ("dim",))
-
-
 # ─────────────────────────────────────────────────────────────────────────────
 # Main application window
 # ─────────────────────────────────────────────────────────────────────────────
@@ -2820,9 +2730,6 @@ class SeismoLab(tk.Tk):
         )
         nb.add(self._download_panel, text="  Download  ")
 
-        self._inspector_panel = InspectorPanel(nb)
-        nb.add(self._inspector_panel, text="  Inspector  ")
-
         self._nb = nb
         self.protocol("WM_DELETE_WINDOW", self._on_close)
 

sfm/compliance.py

@@ -1,133 +0,0 @@
-"""USBM RI8507 / OSMRE blasting compliance chart.
-
-Renders the velocity-vs-frequency compliance scatter Blastware draws on its Event
-Report: each channel's significant waveform cycles as ``(frequency, peak
-velocity)`` points on log-log axes against the regulatory limit curve(s). A point
-below the curve passes; above fails.
-
-Two pieces, kept separate so both can be reused/extended:
-  * ``limit_at`` / ``limit_curve`` — the regulatory limit curve(s), as data.
-  * ``channel_compliance_points`` — the per-cycle (freq, velocity) scatter, by
-    the zero-crossing method (matches Blastware: each channel's cloud tops out
-    at that channel's PPV).
-
-Limit curves (USBM RI8507 Figure B-1 / OSM 30 CFR 816.67), drawn CONTINUOUS — a
-constant-displacement bound (sloped, ``v = 2πf·d``) meets a constant-velocity
-plateau at the frequency where they're equal, so there are no vertical steps
-(matching how Blastware draws it). Two lines:
-  * **Drywall** (modern gypsum board) — 0.75 in/s plateau (solid).
-  * **Plaster** on wood lath (older homes) — 0.50 in/s plateau (dashed).
-Both use a 0.030 in low-frequency displacement bound and rise through a 0.010 in
-displacement bound to a 2.0 in/s high-frequency plateau. Values from USBM RI8507
-(Appendix B) / 30 CFR 816.67; ⚠ confirm the exact shape against a Blastware
-report before trusting for compliance.
-"""
-from __future__ import annotations
-
-import math
-from typing import Dict, Sequence, Tuple
-
-import numpy as np
-from matplotlib.ticker import FixedLocator, NullLocator
-
-# curve name → (low-freq "ultimate" displacement in, mid velocity plateau in/s,
-#               high-freq displacement in, high-freq velocity plateau in/s).
-# RI8507 Fig B-1 (p.74): ultimate max displacement 0.030 in (< ~4 Hz), plateau
-# 0.75 (Drywall) / 0.50 (plaster), rising diagonal at 0.008 in displacement up to
-# a 2.0 in/s plateau reached at ~40 Hz.
-_CURVES: Dict[str, Tuple[float, float, float, float]] = {
-    "Drywall": (0.030, 0.75, 0.008, 2.00),
-    "Plaster": (0.030, 0.50, 0.008, 2.00),
-}
-# how each curve is stroked on the chart
-_CURVE_STYLE = {"Drywall": {"ls": "-", "lw": 1.0}, "Plaster": {"ls": "--", "lw": 0.9}}
-
-STANDARDS = tuple(_CURVES)
-
-# Blastware's channel markers/colours on the compliance chart.
-_CHANNEL_STYLE = {
-    "Tran": ("+", "#d62728"),   # red +
-    "Vert": ("x", "#2ca02c"),   # green x
-    "Long": ("o", "#1f77b4"),   # blue o
-}
-
-
-def limit_at(freq_hz: float, curve: str = "Drywall") -> float:
-    """Max allowed PPV (in/s) at ``freq_hz`` for ``curve`` (continuous)."""
-    d_low, v_mid, d_high, v_high = _CURVES[curve]
-    f = max(freq_hz, 1.0)
-    f_a = v_mid / (2.0 * math.pi * d_low)    # disp_low → vel_mid
-    f_b = v_mid / (2.0 * math.pi * d_high)   # vel_mid → disp_high
-    f_c = v_high / (2.0 * math.pi * d_high)  # disp_high → vel_high
-    if f <= f_a:
-        return 2.0 * math.pi * f * d_low
-    if f <= f_b:
-        return v_mid
-    if f <= f_c:
-        return 2.0 * math.pi * f * d_high
-    return v_high
-
-
-def limit_curve(curve: str = "Drywall", fmin: float = 1.0, fmax: float = 100.0, n: int = 400):
-    """(freqs, limits) sampled across the band for plotting one curve."""
-    freqs = np.logspace(np.log10(fmin), np.log10(fmax), n)
-    return freqs, np.array([limit_at(f, curve) for f in freqs])
-
-
-def channel_compliance_points(
-    samples: Sequence[float], sps: float, fmin: float = 1.0, fmax: float = 100.0,
-    vmin: float = 0.0,
-) -> Tuple[np.ndarray, np.ndarray]:
-    """Per-cycle (frequency, peak velocity) scatter for one channel.
-
-    Zero-crossing method: split the trace at sign changes; each half-cycle
-    contributes one point at ``(1/(2·half_period), max|amplitude|)``. Matches
-    Blastware — the cloud's ceiling is the channel PPV. ``samples`` must be in the
-    velocity unit you want plotted (in/s). Points outside ``[fmin, fmax]`` or at
-    or below ``vmin`` are dropped.
-    """
-    x = np.asarray(samples, dtype=float)
-    if x.size < 3:
-        return np.empty(0), np.empty(0)
-    zc = np.where(np.diff(np.signbit(x)))[0]
-    freqs, vels = [], []
-    for a, b in zip(zc[:-1], zc[1:]):
-        half_period = (b - a) / sps
-        if half_period <= 0:
-            continue
-        freqs.append(1.0 / (2.0 * half_period))
-        vels.append(float(np.abs(x[a:b + 1]).max()))
-    f = np.array(freqs)
-    v = np.array(vels)
-    keep = (f >= fmin) & (f <= fmax) & (v > vmin)
-    return f[keep], v[keep]
-
-
-def draw_compliance_chart(ax, channels: Dict[str, Sequence[float]], sps: float) -> None:
-    """Draw the compliance chart (both limit curves + per-channel scatter)."""
-    for name, style in _CURVE_STYLE.items():
-        cf, cv = limit_curve(name)
-        ax.plot(cf, cv, color="#333", zorder=3, **style)
-
-    for ch, (marker, color) in _CHANNEL_STYLE.items():
-        samples = channels.get(ch)
-        if samples is None or len(samples) == 0:
-            continue
-        f, v = channel_compliance_points(samples, sps)
-        ax.scatter(f, v, marker=marker, s=12, c=color, linewidths=0.7, zorder=4, label=ch)
-
-    ax.set_xscale("log")
-    ax.set_yscale("log")
-    ax.set_xlim(1, 100)
-    ax.set_ylim(0.0394, 10)
-    ax.set_box_aspect(1)   # square plot box (log-log compliance charts are square)
-    xt = [1, 2, 5, 10, 20, 50, 100]
-    yt = [0.0394, 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]
-    ax.xaxis.set_major_locator(FixedLocator(xt)); ax.xaxis.set_minor_locator(NullLocator())
-    ax.yaxis.set_major_locator(FixedLocator(yt)); ax.yaxis.set_minor_locator(NullLocator())
-    ax.set_xticklabels([str(v) for v in xt])
-    ax.set_yticklabels([("%g" % v) for v in yt])
-    ax.set_xlabel("Frequency (Hz)", fontsize=7)
-    ax.set_ylabel("Velocity (in/s)", fontsize=7)
-    ax.tick_params(labelsize=6)
-    ax.grid(True, which="both", ls=":", lw=0.4, color="#ccc")

sfm/database.py

@@ -81,29 +81,11 @@ CREATE TABLE IF NOT EXISTS events (
     sample_rate             INTEGER,
     record_type             TEXT,                   -- "single_shot" | "continuous"
     false_trigger           INTEGER NOT NULL DEFAULT 0,  -- 0=no, 1=yes (manual flag)
-    reviewed_real           INTEGER NOT NULL DEFAULT 0,  -- 0=no, 1=operator-confirmed real (mutually exclusive with false_trigger)
-    false_trigger_reason    TEXT,                   -- optional FT cause ("offset", ...); NULL = none. Only meaningful when false_trigger=1.
     blastware_filename      TEXT,                   -- event file within waveform store; extension is per-event (AB0T encodes timestamp)
     blastware_filesize      INTEGER,                -- bytes; NULL if no event file saved
     a5_pickle_filename      TEXT,                   -- "<filename>.a5.pkl" sidecar
     sidecar_filename        TEXT,                   -- "<filename>.sfm.json" review/metadata sidecar
     device_family           TEXT,                   -- "series3" (MiniMate Plus / BW) | "series4" (Micromate / Thor) — drives per-family UI rendering (units, labels)
-    tran_zc_freq            REAL,                   -- Transverse ZC frequency (Hz)
-    vert_zc_freq            REAL,                   -- Vertical ZC frequency (Hz)
-    long_zc_freq            REAL,                   -- Longitudinal ZC frequency (Hz)
-    mic_zc_freq             REAL,                   -- Microphone ZC frequency (Hz)
-    tran_zc_above_range     INTEGER,                -- 1 = value is device ceiling, render ">N Hz"
-    vert_zc_above_range     INTEGER,
-    long_zc_above_range     INTEGER,
-    mic_zc_above_range      INTEGER,
-    shape_crest_factor      REAL,                   -- peak / rms of the triggering channel
-    shape_near_peak_count   INTEGER,                -- samples >= 0.5 * peak (FT: few; real: many)
-    shape_sample_count      INTEGER,                -- total samples (to normalize near_peak_count)
-    shape_axis              TEXT,                   -- geophone channel measured ("Tran"/"Vert"/"Long")
-    shape_offset            INTEGER,                -- 1 = DC-offset false trigger (pre-trigger baseline off zero + flat). Meaningful for waveforms only.
-    shape_offset_axis       TEXT,                   -- geo channel the offset was measured on
-    shape_offset_pre        REAL,                   -- pre-trigger baseline median (in/s)
-    shape_offset_spread     REAL,                   -- max(pre,mid,end) - min(...) in in/s; small = constant/DC
     created_at              TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
     UNIQUE(serial, timestamp)
 );
@@ -218,24 +200,6 @@ class SeismoDb:
             ("a5_pickle_filename", "TEXT"),
             ("sidecar_filename",   "TEXT"),
             ("device_family",      "TEXT"),
-            ("tran_zc_freq",        "REAL"),
-            ("vert_zc_freq",        "REAL"),
-            ("long_zc_freq",        "REAL"),
-            ("mic_zc_freq",         "REAL"),
-            ("tran_zc_above_range", "INTEGER"),
-            ("vert_zc_above_range", "INTEGER"),
-            ("long_zc_above_range", "INTEGER"),
-            ("mic_zc_above_range",  "INTEGER"),
-            ("shape_crest_factor",    "REAL"),
-            ("shape_near_peak_count", "INTEGER"),
-            ("shape_sample_count",    "INTEGER"),
-            ("shape_axis",            "TEXT"),
-            ("shape_offset",          "INTEGER"),
-            ("shape_offset_axis",     "TEXT"),
-            ("shape_offset_pre",      "REAL"),
-            ("shape_offset_spread",   "REAL"),
-            ("reviewed_real",         "INTEGER NOT NULL DEFAULT 0"),
-            ("false_trigger_reason",  "TEXT"),
         ):
             if col not in existing_cols:
                 log.info("_migrate: events ADD COLUMN %s %s", col, ddl)
@@ -435,16 +399,8 @@ class SeismoDb:
                              sample_rate, record_type,
                              blastware_filename, blastware_filesize,
                              a5_pickle_filename, sidecar_filename,
-                             device_family,
+                             device_family)
-                             tran_zc_freq, vert_zc_freq, long_zc_freq, mic_zc_freq,
+                        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
-                             tran_zc_above_range, vert_zc_above_range,
-                             long_zc_above_range, mic_zc_above_range,
-                             shape_crest_factor, shape_near_peak_count,
-                             shape_sample_count, shape_axis,
-                             shape_offset, shape_offset_axis,
-                             shape_offset_pre, shape_offset_spread)
-                        VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
-                                ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
                         """,
                         (
                             self._new_id(), serial, key, session_id, ts,
@@ -464,22 +420,6 @@ class SeismoDb:
                             rec.get("a5_pickle_filename"),
                             rec.get("sidecar_filename"),
                             device_family,
-                            pv.tran_zc_freq if pv else None,
-                            pv.vert_zc_freq if pv else None,
-                            pv.long_zc_freq if pv else None,
-                            pv.mic_zc_freq  if pv else None,
-                            (1 if (pv and pv.tran_zc_above_range) else 0),
-                            (1 if (pv and pv.vert_zc_above_range) else 0),
-                            (1 if (pv and pv.long_zc_above_range) else 0),
-                            (1 if (pv and pv.mic_zc_above_range)  else 0),
-                            rec.get("shape_crest_factor"),
-                            rec.get("shape_near_peak_count"),
-                            rec.get("shape_sample_count"),
-                            rec.get("shape_axis"),
-                            rec.get("shape_offset"),
-                            rec.get("shape_offset_axis"),
-                            rec.get("shape_offset_pre"),
-                            rec.get("shape_offset_spread"),
                         ),
                     )
                     inserted += 1
@@ -521,23 +461,7 @@ class SeismoDb:
                                blastware_filesize = ?,
                                a5_pickle_filename = ?,
                                sidecar_filename   = ?,
-                               device_family      = COALESCE(?, device_family),
+                               device_family      = COALESCE(?, device_family)
-                               tran_zc_freq        = ?,
-                               vert_zc_freq        = ?,
-                               long_zc_freq        = ?,
-                               mic_zc_freq         = ?,
-                               tran_zc_above_range = ?,
-                               vert_zc_above_range = ?,
-                               long_zc_above_range = ?,
-                               mic_zc_above_range  = ?,
-                               shape_crest_factor    = COALESCE(?, shape_crest_factor),
-                               shape_near_peak_count = COALESCE(?, shape_near_peak_count),
-                               shape_sample_count    = COALESCE(?, shape_sample_count),
-                               shape_axis            = COALESCE(?, shape_axis),
-                               shape_offset          = COALESCE(?, shape_offset),
-                               shape_offset_axis     = COALESCE(?, shape_offset_axis),
-                               shape_offset_pre      = COALESCE(?, shape_offset_pre),
-                               shape_offset_spread   = COALESCE(?, shape_offset_spread)
                          WHERE serial = ? AND timestamp = ?
                         """,
                         (
@@ -557,22 +481,6 @@ class SeismoDb:
                             rec.get("a5_pickle_filename") if rec else None,
                             rec.get("sidecar_filename")   if rec else None,
                             device_family,
-                            pv.tran_zc_freq if pv else None,
-                            pv.vert_zc_freq if pv else None,
-                            pv.long_zc_freq if pv else None,
-                            pv.mic_zc_freq  if pv else None,
-                            (1 if (pv and pv.tran_zc_above_range) else 0),
-                            (1 if (pv and pv.vert_zc_above_range) else 0),
-                            (1 if (pv and pv.long_zc_above_range) else 0),
-                            (1 if (pv and pv.mic_zc_above_range)  else 0),
-                            rec.get("shape_crest_factor")    if rec else None,
-                            rec.get("shape_near_peak_count") if rec else None,
-                            rec.get("shape_sample_count")    if rec else None,
-                            rec.get("shape_axis")            if rec else None,
-                            rec.get("shape_offset")          if rec else None,
-                            rec.get("shape_offset_axis")     if rec else None,
-                            rec.get("shape_offset_pre")      if rec else None,
-                            rec.get("shape_offset_spread")   if rec else None,
                             serial,
                             ts,
                         ),
@@ -627,138 +535,12 @@ class SeismoDb:
             ).fetchall()
         return [dict(r) for r in rows]
 
-    def find_twins(self, event_id: str, *, window_seconds: int | None = None) -> list[dict]:
-        """
-        Find this event's histogram/waveform twin(s): the SAME physical event
-        recorded both as a scheduled histogram and as a triggered waveform.
-
-        A real trigger is captured twice — once as a triggered waveform (stamped
-        at the trigger instant) and once inside the scheduled histogram whose
-        interval contains it (stamped at the histogram's interval start, e.g. the
-        7am/7pm call-in). The two can be HOURS apart in time yet report the same
-        serial and identical peak_vector_sum. Twins are therefore matched by:
-
-          * same serial,
-          * identical peak_vector_sum,
-          * OPPOSITE record type (one histogram, one waveform), and
-          * the waveform's timestamp falls within the histogram's interval —
-            from a histogram's timestamp up to the next histogram (same serial).
-
-        This replaces the old ±``window_seconds`` heuristic, which silently
-        missed twins more than a few minutes apart (a histogram's interval-start
-        stamp and the trigger instant routinely differ by hours). ``window_seconds``
-        is still accepted for backward compatibility but is ignored.
-
-        Returns [] if the event or a required field (serial / peak_vector_sum /
-        timestamp) is missing.
-
-        Caveat: identical-PVS matching remains a proxy for "same physical event"
-        — if the device clamps/saturates PVS (to sqrt(3) * geo_range), two
-        distinct saturated events could share a PVS. The added opposite-type and
-        interval constraints make a false pairing far less likely than the old
-        time-window match, and false_trigger/reviewed_real stay re-derivable from
-        the sidecar source of truth.
-        """
-        def _parse(ts):
-            if not ts:
-                return None
-            try:
-                return datetime.datetime.fromisoformat(str(ts).replace(" ", "T"))
-            except ValueError:
-                return None
-
-        def _is_hist(rt):
-            return str(rt or "").lower().startswith("hist")
-
-        row = self.get_event(event_id)
-        if not row:
-            return []
-        serial = row.get("serial"); pvs = row.get("peak_vector_sum")
-        t_target = _parse(row.get("timestamp"))
-        if serial is None or pvs is None or t_target is None:
-            return []
-        target_hist = _is_hist(row.get("record_type"))
-
-        with self._connect() as conn:
-            cand_rows = [dict(r) for r in conn.execute(
-                "SELECT * FROM events WHERE serial=? AND id!=? AND peak_vector_sum=?",
-                (serial, event_id, pvs)).fetchall()]
-            hist_ts = [r["timestamp"] for r in conn.execute(
-                "SELECT timestamp FROM events WHERE serial=? AND lower(record_type) LIKE 'hist%'",
-                (serial,)).fetchall()]
-
-        # Histogram interval-start times for this serial, sorted, to bound intervals.
-        starts = sorted(x for x in (_parse(t) for t in hist_ts) if x is not None)
-
-        def _interval_end(h_start):
-            # The next histogram strictly after h_start bounds the interval; else open-ended.
-            for x in starts:
-                if x > h_start:
-                    return x
-            return None
-
-        def _covers(h_start, w_time):
-            end = _interval_end(h_start)
-            return h_start <= w_time and (end is None or w_time < end)
-
-        twins = []
-        for c in cand_rows:
-            if _is_hist(c.get("record_type")) == target_hist:
-                continue  # twins are strictly cross-type (one histogram, one waveform)
-            c_time = _parse(c.get("timestamp"))
-            if c_time is None:
-                continue
-            h_start, w_time = (t_target, c_time) if target_hist else (c_time, t_target)
-            if _covers(h_start, w_time):
-                twins.append(c)
-        return twins
-
-    def propagate_review_to_twins(self, event_id: str, *, window_seconds: int | None = None) -> list[str]:
-        """
-        Copy this event's `false_trigger`/`reviewed_real`/`false_trigger_reason`
-        columns onto each of its histogram/waveform twins (see `find_twins`), so
-        flagging one twin flags both.  Returns the list of twin ids updated.
-
-        ``window_seconds`` is accepted for backward compatibility but ignored;
-        twin matching is now interval-based (see `find_twins`).
-        """
-        row = self.get_event(event_id)
-        if not row:
-            return []
-        ft = 1 if row.get("false_trigger") else 0
-        real = 1 if row.get("reviewed_real") else 0
-        # The reason is a subtype of the FT flag — carry it only when the source
-        # is actually a false trigger, so a confirmed-real twin never keeps one.
-        reason = row.get("false_trigger_reason") if ft else None
-        twins = self.find_twins(event_id)
-        moved = []
-        with self._connect() as conn:
-            for tw in twins:
-                conn.execute(
-                    "UPDATE events SET false_trigger=?, reviewed_real=?, false_trigger_reason=? WHERE id=?",
-                    (ft, real, reason, tw["id"]))
-                moved.append(tw["id"])
-        return moved
-
     def set_false_trigger(self, event_id: str, value: bool) -> bool:
-        """
+        """Set or clear the false_trigger flag on an event. Returns True if found."""
-        Set or clear the false_trigger flag on an event. Returns True if found.
-
-        Mirrors the 3-state exclusivity enforced by `update_event_review`:
-        setting false_trigger true also clears `reviewed_real` (a false
-        trigger can't also be a confirmed-real event). Clearing false_trigger
-        leaves `reviewed_real` untouched.
-        """
         with self._connect() as conn:
-            if value:
             cur = conn.execute(
-                    "UPDATE events SET false_trigger=1, reviewed_real=0 WHERE id=?",
+                "UPDATE events SET false_trigger=? WHERE id=?",
-                    (event_id,),
+                (1 if value else 0, event_id),
-                )
-            else:
-                cur = conn.execute(
-                    "UPDATE events SET false_trigger=0, false_trigger_reason=NULL WHERE id=?",
-                    (event_id,),
             )
         return cur.rowcount > 0
 
@@ -835,24 +617,17 @@ class SeismoDb:
         """
         Sync derived index columns from a sidecar's `review` block.
 
-        The derived indexes are `events.false_trigger` and
+        Currently the only derived index is `events.false_trigger` — kept
-        `events.reviewed_real` — kept in sync so `/db/events` queries don't
+        in sync so `/db/events?false_trigger=true` queries don't have to
-        have to scan every sidecar JSON on disk.  The sidecar JSON itself
+        scan every sidecar JSON on disk.  The sidecar JSON itself remains
-        remains the source of truth for the full review state.
+        the source of truth for the full review state.
-
-        The two columns are mutually exclusive: setting either one true
-        forces the other's column to 0.
 
         Returns True when the row exists, False otherwise.  No-op fields
-        (review without `false_trigger` or `reviewed_real`) leave both
+        (review without `false_trigger`) leave the column untouched.
-        columns untouched.
         """
         if not isinstance(review, dict):
             return False
-        has_ft = "false_trigger" in review
+        if "false_trigger" not in review:
-        has_real = "reviewed_real" in review
-        has_reason = "false_trigger_reason" in review
-        if not has_ft and not has_real and not has_reason:
             # Nothing derived to update; just confirm the row exists.
             with self._connect() as conn:
                 row = conn.execute(
@@ -860,28 +635,12 @@ class SeismoDb:
                 ).fetchone()
             return row is not None
 
-        sets = {}
+        flag = 1 if review.get("false_trigger") else 0
-        if has_ft:
-            sets["false_trigger"] = 1 if review.get("false_trigger") else 0
-        if has_real:
-            sets["reviewed_real"] = 1 if review.get("reviewed_real") else 0
-        if has_reason:
-            reason = review.get("false_trigger_reason") or None
-            sets["false_trigger_reason"] = reason
-            if reason:                      # a reason is a subtype of FT → implies FT
-                sets["false_trigger"] = 1
-        # mutual exclusivity: a true in one forces the other column to 0
-        if sets.get("false_trigger") == 1:
-            sets["reviewed_real"] = 0
-        if sets.get("reviewed_real") == 1:
-            sets["false_trigger"] = 0
-        # the reason is only meaningful while flagged FT — clear it if FT ends up 0
-        if sets.get("false_trigger") == 0:
-            sets["false_trigger_reason"] = None
-        assign = ", ".join(f"{k}=?" for k in sets)
-        params = list(sets.values()) + [event_id]
         with self._connect() as conn:
-            cur = conn.execute(f"UPDATE events SET {assign} WHERE id=?", params)
+            cur = conn.execute(
+                "UPDATE events SET false_trigger=? WHERE id=?",
+                (flag, event_id),
+            )
         return cur.rowcount > 0
 
     # ── Monitor log ───────────────────────────────────────────────────────────

sfm/db_snapshot.py

@@ -1,106 +0,0 @@
-"""
-db_snapshot.py — WAL-safe snapshot / restore helpers for seismo_relay.db and
-the waveform store, backing the /db/snapshot, /db/waveforms/recent.zip and
-/db/restore endpoints in server.py. Kept out of server.py so the logic is
-unit-testable without HTTP.
-"""
-from __future__ import annotations
-import os
-import shutil
-import sqlite3
-import zipfile
-import datetime
-from pathlib import Path
-
-
-def snapshot_db(src_path: Path, dest_path: Path) -> Path:
-    """WAL-safe copy of a live SQLite DB via the SQLite online backup API.
-    Consistent even while the source is being written under WAL, and never
-    blocks the source's writers."""
-    src = sqlite3.connect(str(src_path))
-    dest = sqlite3.connect(str(dest_path))
-    try:
-        with dest:
-            src.backup(dest)
-    finally:
-        src.close()
-        dest.close()
-    return Path(dest_path)
-
-
-def build_recent_waveforms_zip(db, store, n: int, dest_path: Path) -> dict:
-    """Zip every on-disk file (event file + .h5/.a5.pkl/.sfm.json siblings) for
-    the n most-recent events, arcname '<serial>/<filename>'. Events with no
-    resolvable files on disk are skipped. Returns {'events': int, 'files': int}."""
-    rows = db.query_events(limit=n, offset=0)
-    events = 0
-    files = 0
-    with zipfile.ZipFile(dest_path, "w", compression=zipfile.ZIP_DEFLATED) as zf:
-        for row in rows:
-            serial = row.get("serial")
-            fn = row.get("blastware_filename")
-            if not serial or not fn:
-                continue
-            serial_dir = store.root / serial
-            matched = sorted(serial_dir.glob(f"{fn}*")) if serial_dir.is_dir() else []
-            matched = [p for p in matched if p.is_file()]
-            if not matched:
-                continue
-            events += 1
-            for p in matched:
-                zf.write(p, arcname=f"{serial}/{p.name}")
-                files += 1
-    return {"events": events, "files": files}
-
-
-def sfm_db_restore_enabled() -> bool:
-    return os.getenv("SFM_DB_RESTORE_ENABLED", "").strip().lower() in ("1", "true", "yes", "on")
-
-
-def _assert_sqlite(path: Path) -> None:
-    with open(path, "rb") as f:
-        head = f.read(16)
-    if head != b"SQLite format 3\x00":
-        raise ValueError(f"{path} is not a SQLite database")
-
-
-def restore_db_and_waveforms(db_path: Path, waveform_root: Path,
-                             new_db_path: Path, waveforms_zip_path) -> dict:
-    """Safety-backup the current seismo_relay.db, atomically swap in
-    new_db_path, clear stale -wal/-shm sidecars, then unpack the waveforms zip
-    under waveform_root. SFM opens a fresh connection per call, so an in-place
-    atomic replace is safe."""
-    db_path = Path(db_path)
-    new_db_path = Path(new_db_path)
-    _assert_sqlite(new_db_path)
-
-    safety_name = None
-    if db_path.exists():
-        ts = datetime.datetime.utcnow().strftime("%Y%m%d_%H%M%S")
-        safety = db_path.with_name(f"{db_path.name}.pre-restore-{ts}")
-        snapshot_db(db_path, safety)
-        safety_name = safety.name
-
-    # Clear the OUTGOING db's WAL sidecars so they cannot attach to the incoming file.
-    for sfx in ("-wal", "-shm"):
-        side = Path(str(db_path) + sfx)
-        if side.exists():
-            side.unlink()
-
-    incoming = db_path.with_name(db_path.name + ".incoming")
-    shutil.copy2(new_db_path, incoming)
-    os.replace(incoming, db_path)   # atomic within the same directory
-
-    unpacked = 0
-    if waveforms_zip_path and Path(waveforms_zip_path).exists():
-        waveform_root = Path(waveform_root)
-        waveform_root.mkdir(parents=True, exist_ok=True)
-        with zipfile.ZipFile(waveforms_zip_path) as zf:
-            for m in zf.namelist():
-                if m.endswith("/") or m.startswith("/") or ".." in Path(m).parts:
-                    continue
-                zf.extract(m, waveform_root)
-                unpacked += 1
-
-    return {"ok": True, "db_restored": True,
-            "waveforms_unpacked": unpacked, "safety_backup": safety_name}

sfm/event_hdf5.py

@@ -12,11 +12,8 @@ Layout written to `<filename>.h5`:
   ├─ samples_int16/ (optional)
   │   ├─ Tran  (int16, raw ADC counts) shape: (N,)
   │   └─ ... per channel (only when present in the source)
-  ├─ sensor_check/ (optional, schema v2+)
-  │   ├─ Tran  (int32, raw counts)     shape: (M,)   M ≪ N
-  │   └─ ... per channel present in the source (MicL absent on 3-channel units)
   └─ root attrs (event metadata):
-       schema_version       int    = 2
+       schema_version       int    = 1
        kind                 str    = "sfm.event.hdf5"
        serial               str
        waveform_key         str    (8-hex)
@@ -67,7 +64,7 @@ from minimateplus.models import Event
 
 log = logging.getLogger(__name__)
 
-SCHEMA_VERSION = 2   # v2 adds the optional /sensor_check group
+SCHEMA_VERSION = 1
 HDF5_KIND = "sfm.event.hdf5"
 
 # Geophone full-scale velocity per range (in/s).  Confirmed in CLAUDE.md
@@ -80,20 +77,6 @@ _GEO_FS_BY_RANGE = {
 }
 _INT16_FS = 32768.0
 
-# Geophone full-scale count.  NOT 32768: the verified body codec emits geo
-# samples in 16-count units whose documented LSB is exactly 0.005 in/s, and
-# ``waveform_codec.decoded_to_adc_counts`` multiplies by 16 — so one ADC count
-# is 0.005/16 in/s and Normal range (10.000 in/s) is 10.0 / (0.005/16) = 32000
-# counts.  Using 32768 here made every geophone reading 2.3% low
-# (1 - 32000/32768 = 0.0234).
-#
-# Confirmed 2026-08-25 against 216 per-channel comparisons with the preserved
-# Blastware ASCII exports: 32000 gives 216/216 exact within 1 LSB (worst error
-# 0.005 in/s); 32768 gave 151/216 with a worst error of 0.238 in/s on a
-# 10 in/s event.  The mic path is unaffected — it back-solves its own scale
-# from the device-reported peak (see _mic_scale_factor).
-_GEO_INT16_FS = 32000.0
-
 # Default mic conversion: ADC count → psi.  Approximate; exact factor
 # depends on firmware reference voltage and mic sensitivity, neither of
 # which is independently confirmed.  We try to refine it from the device-
@@ -142,14 +125,15 @@ def _samples_to_float(
 ) -> np.ndarray:
     """Convert int16 ADC counts → float32 physical units.
 
-    Uses _GEO_INT16_FS=32000 (see the constant's rationale): one decoder
+    Uses _INT16_FS=32768 (not 32767) so that a count of -32768 maps to
-    unit (16 ADC counts) is exactly 0.005 in/s, so full scale is 32000
+    exactly -full_scale and +32767 maps to ~+full_scale * 32767/32768.
-    counts, not 32768.
+    Matches the device firmware's documented mapping (see CLAUDE.md
+    geo_hardware_constant rationale).
     """
     if not samples_int16:
         return np.array([], dtype=np.float32)
     arr = np.asarray(samples_int16, dtype=np.int32)  # int32 to avoid overflow during scale
-    return (arr.astype(np.float32) * (full_scale / _GEO_INT16_FS)).astype(np.float32)
+    return (arr.astype(np.float32) * (full_scale / _INT16_FS)).astype(np.float32)
 
 
 def _mic_scale_factor(
@@ -273,22 +257,6 @@ def write_event_hdf5(
                     )
             igrp.attrs["mic_psi_per_count"] = float(mic_factor)
 
-        # /sensor_check — optional short diagnostic self-check traces (schema
-        # v2+).  Raw ADC counts (a shape diagnostic; the per-series count scale
-        # differs, and the renderer fits each trace to its box).  Only channels
-        # the decoder found are written — 3-channel units carry no MicL.
-        sc = event.sensor_check or {}
-        if sc:
-            scgrp = f.create_group("sensor_check")
-            for ch in ("Tran", "Vert", "Long", "MicL"):
-                vals = sc.get(ch)
-                if vals:
-                    scgrp.create_dataset(
-                        ch, data=np.asarray(vals, dtype=np.int32),
-                        compression="gzip", compression_opts=4, shuffle=True,
-                    )
-            scgrp.attrs["units"] = "raw_counts"
-
     import os
     os.replace(tmp, path)
 
@@ -353,16 +321,6 @@ def read_event_hdf5(path: Union[str, Path]) -> dict:
             if mic_attr is not None:
                 mic_psi = float(mic_attr)
 
-        # /sensor_check — optional (schema v2+); absent on older files.
-        sensor_check = None
-        scgrp = f.get("sensor_check")
-        if scgrp is not None:
-            sensor_check = {}
-            for ch in ("Tran", "Vert", "Long", "MicL"):
-                ds = scgrp.get(ch)
-                if ds is not None:
-                    sensor_check[ch] = np.asarray(ds[()])
-
     return {
         "schema_version":     sv,
         "kind":               attrs.get("kind"),
@@ -370,7 +328,6 @@ def read_event_hdf5(path: Union[str, Path]) -> dict:
         "samples":            samples,
         "samples_int16":      samples_int16,
         "mic_psi_per_count":  mic_psi,
-        "sensor_check":       sensor_check,
     }
 
 
@@ -461,16 +418,11 @@ def plot_json_from_hdf5(
     event_id: Optional[str] = None,
     index: Optional[int] = None,
 ) -> dict:
-    """Build a `sfm.plot.v1` JSON dict from a stored .h5 file.
+    """Build a `sfm.plot.v1` JSON dict from a stored .h5 file."""
-
-    The dict also carries a top-level ``sensor_check`` key (the raw self-check
-    traces as ``{ch: [int]}``, or None) beyond the plot schema, so report
-    generation can read the traces from the same single .h5 load.
-    """
     data = read_event_hdf5(path)
     a = data["attrs"]
     s = data["samples"]
-    out = _build_plot_dict(
+    return _build_plot_dict(
         n_samples=len(s["Tran"]) if "Tran" in s else 0,
         sample_rate=int(a.get("sample_rate", 1024) or 1024),
         pretrig_samples=int(a.get("pretrig_samples", 0) or 0),
@@ -498,11 +450,6 @@ def plot_json_from_hdf5(
         event_id=event_id,
         index=index,
     )
-    scd = data.get("sensor_check")
-    out["sensor_check"] = (
-        {ch: v.tolist() for ch, v in scd.items()} if scd else None
-    )
-    return out
 
 
 def _build_plot_dict(

sfm/report_pdf.py

@@ -121,13 +121,6 @@ class ReportData:
     t0_ms:                  Optional[float] = None
     dt_ms:                  Optional[float] = None
 
-    # Sensor self-check traces — {ch: [samples]} in raw counts, read from the
-    # standardized .h5 (/sensor_check group, schema v2+) where the per-series
-    # decoder stored them at ingest.  The little diagnostic waveforms BW draws
-    # in its "Sensor Check" strip.  Empty when absent (pre-v2 .h5, histogram,
-    # or 3-channel unit's MicL).
-    sensor_check_waveforms: dict = field(default_factory=dict)
-
     # Record-type discriminator
     record_type:            Optional[str] = None
     is_histogram:           bool = False
@@ -253,8 +246,6 @@ def gather_report_data(
                 "peak_accel_g":       ch.get("peak_accel_g"),
                 "peak_disp_in":       ch.get("peak_disp_in"),
                 "sensor_check":       sc_ch.get("result"),
-                "sc_freq_hz":         sc_ch.get("freq_hz"),
-                "sc_ratio":           sc_ch.get("ratio"),
                 "peak_date":          peak_date,
                 "peak_time":          peak_time,
             })
@@ -296,12 +287,6 @@ def gather_report_data(
             rd.pretrig_samples  = ta.get("pretrig_samples")
             rd.t0_ms            = ta.get("t0_ms")
             rd.dt_ms            = ta.get("dt_ms")
-            # Sensor self-check traces — read from the standardized .h5 (schema
-            # v2+).  Device-agnostic: whichever decoder produced the event
-            # stored them at ingest, so SFM reads them here without knowing or
-            # caring about the source instrument series.  Empty on pre-v2 files
-            # (until backfilled) and on 3-channel / histogram events.
-            rd.sensor_check_waveforms = wf.get("sensor_check") or {}
         except Exception as exc:
             log.warning("gather_report_data: hdf5 read failed: %s", exc)
 
@@ -411,34 +396,9 @@ def _render_waveform_layout(fig, rd: ReportData) -> None:
     ax_stats = fig.add_subplot(gs[2]); ax_stats.axis("off")
     _draw_channel_stats_waveform(ax_stats, rd)
 
-    _draw_compliance_panel(fig, rd)
     _draw_waveform_subplot(fig, gs[3], rd)
 
 
-# Compliance-chart placement, in figure fractions.  Measured directly off a
-# Blastware Event Report PDF (ref-stuff/n844lqhbzt0w_bw_pdf.pdf) so the chart
-# matches BW's size and position: it spans from just under the header down
-# through the stats band, hard against the right page margin.  The left edge
-# leaves room for the y-axis tick labels + "Velocity (in/s)" title, which the
-# compacted stats table (see _draw_channel_stats_waveform) is sized to clear.
-_COMPLIANCE_BOX = (0.489, 0.502, 0.951, 0.867)   # x0, y0, x1, y1
-
-
-def _draw_compliance_panel(fig, rd: ReportData) -> None:
-    """Large USBM RI8507 compliance chart in the upper-right, sized and
-    positioned to match Blastware's Event Report (see _COMPLIANCE_BOX)."""
-    x0, y0, x1, y1 = _COMPLIANCE_BOX
-    fig.text((x0 + x1) / 2, y1 + 0.006, "USBM RI8507 And OSMRE", fontsize=9,
-             weight="bold", color="#333", ha="center", va="bottom")
-    if rd.channels and rd.sample_rate_sps:
-        from sfm.compliance import draw_compliance_chart
-        ax = fig.add_axes([x0, y0, x1 - x0, y1 - y0])
-        draw_compliance_chart(ax, rd.channels, rd.sample_rate_sps)
-    else:
-        fig.text((x0 + x1) / 2, (y0 + y1) / 2, "(no waveform data)", fontsize=8,
-                 color="#bbb", ha="center", va="center", style="italic")
-
-
 def _render_histogram_layout(fig, rd: ReportData) -> None:
     """Histogram layout: header / mic-only / per-channel stats / bar plot.
 
@@ -517,11 +477,11 @@ def _split_iso_to_date_time(iso: Optional[str]) -> tuple[Optional[str], Optional
         return (None, None)
 
 
-def _kv(ax, x, y, label, value, *, label_w=0.18, fontsize=8):
+def _kv(ax, x, y, label, value, *, label_w=0.18):
     """Render a 'Label  Value' row at axes-coordinates (x, y)."""
-    ax.text(x, y, label, fontsize=fontsize, color="#555", ha="left", va="top",
+    ax.text(x, y, label, fontsize=8, color="#555", ha="left", va="top",
             transform=ax.transAxes)
-    ax.text(x + label_w, y, _fmt(value), fontsize=fontsize, ha="left", va="top",
+    ax.text(x + label_w, y, _fmt(value), fontsize=8, ha="left", va="top",
             transform=ax.transAxes, family="monospace")
 
 
@@ -584,17 +544,14 @@ def _draw_header_columns(ax, rows_left, rd: ReportData) -> None:
         ("File Name", rd.file_name),
         ("Post Event Notes", rd.post_event_notes),
     ]
-    # fontsize 7.5 (BW's header is a touch smaller than our body text) + a
-    # tighter right-column value indent so the long serial+firmware line
-    # ("BE##### V ##.##-#.## MiniMate Plus") fits without running off the page.
     y = 0.95
     dy = 0.095
     for label, value in rows_left:
-        _kv(ax, 0.0, y, label, value, label_w=0.18, fontsize=7.5)
+        _kv(ax, 0.0, y, label, value, label_w=0.18)
         y -= dy
     y = 0.95
     for label, value in rows_right:
-        _kv(ax, 0.55, y, label, value, label_w=0.14, fontsize=7.5)
+        _kv(ax, 0.55, y, label, value, label_w=0.20)
         y -= dy
 
 
@@ -617,14 +574,19 @@ def _draw_mic_and_usbm(ax, rd: ReportData) -> None:
             transform=ax.transAxes, va="top")
     rows = _mic_rows(rd)
     y = 0.80
-    # Tighter label indent + slightly smaller font so the long "Channel Test
-    # Passed (Freq = … Amp = … mv)" line clears the enlarged compliance chart's
-    # left edge (_COMPLIANCE_BOX) instead of running behind it.
     for label, value in rows:
-        _kv(ax, 0.0, y, label, value, label_w=0.13, fontsize=7)
+        _kv(ax, 0.0, y, label, value, label_w=0.18)
         y -= 0.15
-    # The USBM compliance chart is drawn as its own large square panel spanning
+
-    # the mic + stats rows on the right — see _draw_compliance_panel().
+    # USBM chart placeholder — upper-right.  Real piecewise compliance
+    # curves are a separate work item; for now this just shows the title
+    # + a "see report" message so the layout is correct.
+    ax.text(0.72, 0.97, "USBM RI8507 And OSMRE",
+            fontsize=9, weight="bold", color="#333", ha="center", va="top",
+            transform=ax.transAxes)
+    ax.text(0.72, 0.50, "[compliance chart\ncoming soon]",
+            fontsize=8, color="#bbb", ha="center", va="center",
+            transform=ax.transAxes, style="italic")
 
 
 def _mic_rows(rd: ReportData) -> list[tuple[str, Optional[str]]]:
@@ -674,19 +636,16 @@ def _draw_channel_stats_waveform(ax, rd: ReportData) -> None:
         ("Peak Acceleration",    "peak_accel_g",   "g"),
         ("Peak Displacement",    "peak_disp_in",   "in"),
         ("Sensor Check",         "sensor_check",   ""),
-        # Sensor-check sub-rows (indented under "Sensor Check", like BW): the
-        # geophone ring-down frequency + overswing ratio from the self-check.
-        ("  Frequency",          "sc_freq_hz",     "Hz"),
-        ("  Overswing Ratio",    "sc_ratio",       ""),
     ]
-    # Compacted to the left half so the enlarged compliance chart (BW-sized,
+    _draw_stats_table(ax, rd, rows_spec)
-    # right against the page margin) has room — see _COMPLIANCE_BOX.
+    if rd.peak_vector_sum_ips is not None:
-    _draw_stats_table(
+        line = f"Peak Vector Sum   {rd.peak_vector_sum_ips:.3f} in/s"
-        ax, rd, rows_spec,
+        if rd.peak_vector_sum_time_s is not None:
-        bbox_width=0.42, fontsize=7.5,
+            line += f" At {rd.peak_vector_sum_time_s:.3f} sec."
-        col_widths=[0.185, 0.065, 0.065, 0.065, 0.040],
+        ax.text(0.0, -0.08, line, fontsize=9, weight="bold",
-    )
+                ha="left", va="top", transform=ax.transAxes)
-    _draw_pvs_summary(ax, rd, n_data_rows=len(rows_spec))
+        ax.text(0.0, -0.18, "NA: Not Applicable", fontsize=7, color="#888",
+                ha="left", va="top", transform=ax.transAxes)
 
 
 def _draw_channel_stats_histogram(ax, rd: ReportData) -> None:
@@ -704,81 +663,27 @@ def _draw_channel_stats_histogram(ax, rd: ReportData) -> None:
         ("Sensor Check", "sensor_check",    ""),
     ]
     _draw_stats_table(ax, rd, rows_spec)
-    _draw_pvs_summary(ax, rd, n_data_rows=len(rows_spec), histogram_when=True)
+    if rd.peak_vector_sum_ips is not None:
-
-
-def _draw_pvs_summary(
-    ax,
-    rd: ReportData,
-    *,
-    n_data_rows: int,
-    histogram_when: bool = False,
-) -> None:
-    """Render the Peak Vector Sum + 'NA: Not Applicable' caption below the
-    stats table.
-
-    Reads ``ax._stats_table_bottom`` (set by ``_draw_stats_table`` when
-    it pins the table via an explicit ``bbox``) so the PVS line lands
-    just below the table's known bottom edge instead of guessing at the
-    geometry.
-
-    Centered horizontally for visual balance (the previous left-aligned
-    x=0 landed under the label column, not the data, which looked off).
-    """
-    if rd.peak_vector_sum_ips is None:
-        return
-
         line = f"Peak Vector Sum   {rd.peak_vector_sum_ips:.3f} in/s"
-    if histogram_when and rd.peak_vector_sum_when_str:
+        # Histograms: "0.091 in/s on May 27, 2026 At 06:06:14"
-        # Histogram absolute date+time.  when_str is "HH:MM:SS Month DD, YYYY";
+        # The when_str is "HH:MM:SS Month DD, YYYY" — reformat for BW match.
-        # reformat to "<value> on <date> At <time>" to match BW.
+        if rd.peak_vector_sum_when_str:
             parts = rd.peak_vector_sum_when_str.split(" ", 1)
             if len(parts) == 2:
                 line += f" on {parts[1]} At {parts[0]}"
             else:
                 line += f" on {rd.peak_vector_sum_when_str}"
-    elif not histogram_when and rd.peak_vector_sum_time_s is not None:
+        ax.text(0.0, -0.08, line, fontsize=9, weight="bold",
-        line += f" At {rd.peak_vector_sum_time_s:.3f} sec."
+                ha="left", va="top", transform=ax.transAxes)
-
+        ax.text(0.0, -0.18, "NA: Not Applicable", fontsize=7, color="#888",
-    # _draw_stats_table stashes the bbox bottom on the axes so we don't
-    # have to guess geometry.  Falls back to a conservative default if
-    # the bbox approach hasn't run.
-    table_bottom_y = getattr(ax, "_stats_table_bottom", -0.10)
-    pvs_y = table_bottom_y - 0.04   # small gap below the table border
-
-    # Centered under the stats table for visual balance — looks intentional
-    # rather than offset.  When the table is compacted (waveform layout), it
-    # occupies only the left portion of the axes, so center on the table's
-    # width rather than the full axes (which would push the line under the
-    # compliance chart).  The original BW-replica had a "NA: Not Applicable"
-    # caption below this line; dropped because we use "—" for missing values.
-    table_w = getattr(ax, "_stats_table_width", 0.80)
-    if table_w < 0.79:
-        # Compacted (waveform) layout: left-align under the table, one point
-        # smaller, so the line clears the enlarged compliance chart's
-        # bottom-left tick labels on the right.
-        ax.text(0.0, pvs_y, line, fontsize=8, weight="bold",
                 ha="left", va="top", transform=ax.transAxes)
-    else:
-        ax.text(0.5, pvs_y, line, fontsize=9, weight="bold",
-                ha="center", va="top", transform=ax.transAxes)
 
 
-def _draw_stats_table(
+def _draw_stats_table(ax, rd: ReportData, rows_spec: list[tuple[str, str, str]]) -> None:
-    ax, rd: ReportData, rows_spec: list[tuple[str, str, str]],
-    *, bbox_width: float = 0.80, fontsize: float = 8,
-    col_widths: Optional[list[float]] = None,
-) -> None:
     """Render a per-channel stats table (Tran/Vert/Long).
 
     rows_spec: list of (label, field_name_in_channel_stats, unit_string)
-
-    ``bbox_width`` / ``col_widths`` / ``fontsize`` let a caller compact the
-    table (the waveform layout packs it into the left half to clear the
-    compliance chart; the histogram layout keeps the wider defaults).
     """
-    if col_widths is None:
-        col_widths = [0.28, 0.14, 0.14, 0.14, 0.10]
     headers = ["", "Tran", "Vert", "Long", ""]
     ch_lookup = {c["name"]: c for c in rd.channel_stats}
 
@@ -794,8 +699,6 @@ def _draw_stats_table(
             if field == "zc_freq_hz":
                 prefix = ">" if ch_rec.get("zc_freq_above_range") else ""
                 return f"{prefix}{val:.0f}"
-            if field in ("sc_freq_hz", "sc_ratio"):
-                return f"{val:.1f}"       # BW shows 1 decimal (7.5 Hz, 3.6)
             return f"{val:.3f}"
         return str(val)
 
@@ -808,29 +711,16 @@ def _draw_stats_table(
             _cell(field_name, "Long"),
             unit,
         ])
-    # Pin the table's position+size via bbox so we know exactly where
-    # the bottom edge lands.  Lets _draw_pvs_summary place the PVS line
-    # just below the table without guessing at row heights.
-    #
-    # bbox = [x, y, width, height] in axes coords.  Header + data rows
-    # at row_h each; horizontal extent matches sum(colWidths).
-    n_rows = len(table_data)        # header + data rows
-    row_h  = 0.12                   # axes-fraction per row (fits fontsize=8)
-    table_height = n_rows * row_h
-    table_bottom = 1.0 - table_height
     tbl = ax.table(
-        cellText=table_data,
+        cellText=table_data, loc="upper left",
-        colWidths=col_widths,
+        colWidths=[0.28, 0.14, 0.14, 0.14, 0.10],
         cellLoc="left", edges="open",
-        bbox=[0.0, table_bottom, bbox_width, table_height],
     )
     tbl.auto_set_font_size(False)
-    tbl.set_fontsize(fontsize)
+    tbl.set_fontsize(8)
+    tbl.scale(1, 1.4)
     for j in range(5):
         tbl[(0, j)].set_text_props(weight="bold", color="#555")
-    # Stash the bottom Y + width so _draw_pvs_summary can position itself.
-    ax._stats_table_bottom = table_bottom
-    ax._stats_table_width = bbox_width
 
 
 def _channel_axis_color(ch: str) -> str:
@@ -840,59 +730,27 @@ def _channel_axis_color(ch: str) -> str:
 def _draw_waveform_subplot(fig, gridspec_cell, rd: ReportData) -> None:
     """4-channel stacked waveform plot — Instantel printout order
     (MicL on top, Tran on bottom), shared x-axis in SECONDS, trigger
-    triangle markers at t=0, '0.0' baseline label on right of each.
+    triangle markers at t=0, '0.0' baseline label on right of each."""
-
-    When sensor self-check traces are present (rd.sensor_check_waveforms), a
-    narrow "Sensor Check" strip of per-channel mini-plots is drawn to the right,
-    aligned to the lanes — matching Blastware's Event Report.
-    """
-    from matplotlib.ticker import MaxNLocator
-
-    order = ["MicL", "Long", "Vert", "Tran"]
-    has_sc = bool(rd.sensor_check_waveforms)
-    if has_sc:
-        # main lanes + a narrow sensor-check strip column, flush against the
-        # main panel (BW shares the border — no gap), with the "0.0" baseline
-        # labels moved to the right of the strip.  Proportions match BW's
-        # Event Report (main ~0.75 / strip ~0.10 of the panel width).
-        inner = gridspec_cell.subgridspec(4, 2, width_ratios=[1.0, 0.13],
-                                          wspace=0.0, hspace=0.0)
-    else:
     inner = gridspec_cell.subgridspec(4, 1, hspace=0.0)
+    order = ["MicL", "Long", "Vert", "Tran"]
     sr = rd.sample_rate_sps or 1024
     # Convert ms-based time axis to seconds for the x-axis
     dt_s = (rd.dt_ms or (1000.0 / sr)) / 1000.0
     t0_s = (rd.t0_ms if rd.t0_ms is not None else 0.0) / 1000.0
 
-    # Shared geo scale across Long/Vert/Tran (matches the event modal + BW's
-    # single amp/div): all three geo lanes use ONE Y scale = the max |sample|
-    # across them (padded, floored), so relative amplitudes stay honest instead
-    # of each lane auto-zooming to its own peak.  Mic keeps its own (psi) scale.
-    GEO_FLOOR_INS = 0.05
-    _geo_amax = 0.0
-    for _gch in ("Long", "Vert", "Tran"):
-        for _x in (rd.channels.get(_gch) or []):
-            _a = abs(_x)
-            if _a > _geo_amax:
-                _geo_amax = _a
-    geo_shared = max(_geo_amax * 1.10, GEO_FLOOR_INS)
-
-    main_axes = []
-    sc_axes = []
     last_idx = len(order) - 1
     for i, ch in enumerate(order):
-        ax = fig.add_subplot(inner[i, 0] if has_sc else inner[i])
+        ax = fig.add_subplot(inner[i])
-        main_axes.append(ax)
         values = rd.channels.get(ch) or []
         times = [t0_s + j * dt_s for j in range(len(values))]
 
         if values:
             color = _channel_axis_color(ch)
             ax.plot(times, values, color=color, linewidth=0.5)
-            # Geo: one shared symmetric scale (honest relative amplitudes).
+            # Symmetric y-axis for geo; zero-anchored for mic.
-            # Mic: symmetric on its own psi scale (different unit).
             if ch != "MicL":
-                ax.set_ylim(-geo_shared, geo_shared)
+                amax = max((abs(v) for v in values), default=0.001)
+                ax.set_ylim(-amax * 1.10, amax * 1.10)
             else:
                 amax = max((abs(v) for v in values), default=0.001)
                 ax.set_ylim(-amax * 1.10, amax * 1.10)
@@ -900,10 +758,7 @@ def _draw_waveform_subplot(fig, gridspec_cell, rd: ReportData) -> None:
         # Channel label on the LEFT (matches BW)
         ax.set_ylabel(ch, fontsize=8, rotation=0, ha="right", va="center",
                       color=_channel_axis_color(ch), weight="bold", labelpad=14)
-        # "0.0" baseline label on the RIGHT (BW convention).  With the sensor-
+        # "0.0" on the RIGHT (BW convention)
-        # check strip attached, it goes to the right of the STRIP (drawn below);
-        # otherwise just outside the main lane.
-        if not has_sc:
         ax.text(1.005, 0.5, "0.0", transform=ax.transAxes,
                 fontsize=7, color="#555", va="center", ha="left")
 
@@ -920,53 +775,23 @@ def _draw_waveform_subplot(fig, gridspec_cell, rd: ReportData) -> None:
         else:
             ax.tick_params(axis="x", labelsize=7)
         ax.tick_params(axis="y", labelsize=6)
-        # Stacked lanes touch, so the top/bottom y-tick labels of adjacent lanes
-        # would overprint at the shared boundary.  Prune the extreme ticks so
-        # each boundary shows clean interior ticks (0.5 / 0.0 / -0.5) only.
-        ax.yaxis.set_major_locator(MaxNLocator(nbins=4, prune="both"))
-
-        # Sensor self-check mini-plot in the right strip (aligned to this lane).
-        if has_sc:
-            scx = fig.add_subplot(inner[i, 1])
-            sc_axes.append(scx)
-            sc_vals = rd.sensor_check_waveforms.get(ch) or []
-            if sc_vals:
-                _col = _channel_axis_color(ch)
-                # Faint zero baseline (BW draws the channel baseline through the
-                # strip) — reference for the one-sided geophone ring-downs.
-                scx.axhline(0.0, color=_col, linewidth=0.3, alpha=0.4)
-                scx.plot(range(len(sc_vals)), sc_vals, color=_col, linewidth=0.5)
-                # Fit the trace to the box (BW-style) rather than a symmetric
-                # scale: the geo self-checks are one-sided dips, so a symmetric
-                # scale would strand them in the bottom half with an empty top.
-                _lo, _hi = min(sc_vals), max(sc_vals)
-                _pad = 0.10 * ((_hi - _lo) or 1.0)
-                scx.set_ylim(_lo - _pad, _hi + _pad)
-            scx.set_xticks([]); scx.set_yticks([])
-            for _s in scx.spines.values():
-                _s.set_linewidth(0.4); _s.set_color("#999")
-            # "0.0" baseline label to the RIGHT of the strip (BW convention)
-            scx.text(1.10, 0.5, "0.0", transform=scx.transAxes,
-                     fontsize=7, color="#555", va="center", ha="left")
 
     # Trigger triangle marker ▼ above the top channel at t=0
-    top_ax = main_axes[0]  # MicL
+    top_ax = fig.axes[-4]  # MicL is the first added in this gridspec
     top_ax.plot([0], [top_ax.get_ylim()[1]], marker="v", color="black",
                 markersize=8, clip_on=False, zorder=10)
 
-    # "Sensor Check" caption under the strip (BW convention)
-    if has_sc and sc_axes:
-        pos = sc_axes[-1].get_position()
-        fig.text((pos.x0 + pos.x1) / 2, pos.y0 - 0.012, "Sensor Check",
-                 fontsize=7, color="#555", ha="center", va="top")
-
     # Compute scale-per-division for the footer (10 divs across the chart)
     # and find peak geo amplitude for the geo amp/div setting.
     total_s = times[-1] - times[0] if values else 0
     div_s = total_s / 10 if total_s > 0 else 0
-    # Footer div value reflects the SHARED geo scale (so it's correct for all
+    geo_amp_div = "—"
-    # three lanes, not just whichever one happened to be checked first).
+    for ch in ("Tran", "Vert", "Long"):
-    geo_amp_div = f"{(geo_shared * 2) / 10:.3f}" if _geo_amax > 0 else "—"
+        v = rd.channels.get(ch) or []
+        if v:
+            amax = max(abs(x) for x in v)
+            geo_amp_div = f"{(amax * 1.1 * 2) / 10:.3f}"
+            break
     fig.text(
         0.11, 0.030,
         f"Time(Seconds) {div_s:.2f} sec/div   Amplitude Geo: {geo_amp_div} in/s/div   Mic: 0.001 psi(L)/div",

sfm/server.py

@@ -67,13 +67,11 @@ from minimateplus.blastware_file import write_blastware_file, blastware_filename
 from minimateplus.client import _decode_a5_metadata_into, _decode_a5_waveform, _decode_event_count
 from minimateplus.framing import build_bw_write_frame, SESSION_RESET, POLL_PROBE, POLL_DATA
 from minimateplus.protocol import SUB_STOP_MONITORING
-from minimateplus.event_file_io import TOOL_VERSION as SFM_VERSION  # single source for the service version (release-bumped)
 from sfm import event_hdf5
 from sfm.cache import SFMCache, get_cache
 from sfm.database import SeismoDb
 from sfm.live_cache import LiveCache as _LiveCache
 from sfm.waveform_store import WaveformStore
-from sfm import db_snapshot
 
 logging.basicConfig(
     level=logging.INFO,
@@ -91,7 +89,7 @@ app = FastAPI(
         "Implements the minimateplus RS-232 protocol library.\n"
         "Proxied by terra-view at /api/sfm/*."
     ),
-    version=SFM_VERSION,
+    version="0.17.0",
 )
 
 # Allow requests from the waveform viewer opened as a local file (file://)
@@ -372,7 +370,7 @@ def _backfill_events(events: list, info: "DeviceInfo") -> None:
 @app.get("/health")
 def health() -> dict:
     """Service heartbeat.  No device I/O."""
-    return {"status": "ok", "service": "sfm", "version": SFM_VERSION}
+    return {"status": "ok", "service": "sfm", "version": "0.1.0"}
 
 
 @app.get("/", response_class=FileResponse)
@@ -1958,10 +1956,7 @@ def db_set_false_trigger(
     value:    bool = Query(..., description="True to flag as false trigger, False to clear"),
 ) -> dict:
     """
-    Set or clear the false_trigger flag on a single event.  Enforces the
+    Set or clear the false_trigger flag on a single event.
-    same 3-state exclusivity as the sidecar review PATCH (flagging false_trigger
-    clears reviewed_real) and propagates the resulting flags to the event's
-    histogram/waveform twin(s), same as the sidecar path.
 
     Used by the terra-view event review UI.
     Returns 404 if the event_id is not found.
@@ -1969,12 +1964,6 @@ def db_set_false_trigger(
     found = _get_db().set_false_trigger(event_id, value)
     if not found:
         raise HTTPException(status_code=404, detail=f"Event {event_id} not found")
-
-    try:
-        _get_db().propagate_review_to_twins(event_id)
-    except Exception as exc:
-        log.warning("twin review-propagation failed for %s: %s", event_id, exc)
-
     return {"status": "ok", "event_id": event_id, "false_trigger": value}
 
 
@@ -2491,13 +2480,6 @@ def db_event_sidecar_patch(event_id: str, body: SidecarPatchBody) -> dict:
     if body.review is not None:
         _get_db().update_event_review(event_id, new_sidecar.get("review", {}))
 
-        # Propagate the review to the event's histogram/waveform twin(s) so
-        # flagging one flags both (column-level; twins share serial+PVS+near time).
-        try:
-            _get_db().propagate_review_to_twins(event_id)
-        except Exception as exc:
-            log.warning("twin review-propagation failed for %s: %s", event_id, exc)
-
     return new_sidecar
 
 
@@ -2826,70 +2808,6 @@ def db_unit_waveforms_zip(
     return StreamingResponse(buf, media_type="application/zip", headers=headers)
 
 
-# ── Full-snapshot bundle endpoints (see sfm/db_snapshot.py) ───────────────────
-# SFM_DB_RESTORE_ENABLED gates /db/restore (dormant → 404). Set true on dev only.
-@app.get("/db/snapshot")
-def db_snapshot_download() -> FileResponse:
-    """WAL-safe copy of seismo_relay.db, for the terra-view full-snapshot bundle."""
-    import os
-    import tempfile
-    from starlette.background import BackgroundTask
-    fd, tmp = tempfile.mkstemp(suffix=".db")
-    os.close(fd)
-    db_snapshot.snapshot_db(_get_db().db_path, Path(tmp))
-    return FileResponse(
-        tmp, media_type="application/x-sqlite3", filename="seismo_relay.db",
-        background=BackgroundTask(os.unlink, tmp),
-    )
-
-
-@app.get("/db/waveforms/recent.zip")
-def db_waveforms_recent_zip(n: int = Query(200, ge=1, le=2000)) -> FileResponse:
-    """Zip of the n most-recent events' waveform + sidecar files."""
-    import os
-    import tempfile
-    from starlette.background import BackgroundTask
-    fd, tmp = tempfile.mkstemp(suffix=".zip")
-    os.close(fd)
-    stats = db_snapshot.build_recent_waveforms_zip(_get_db(), _get_store(), n, Path(tmp))
-    return FileResponse(
-        tmp, media_type="application/zip", filename="waveforms.zip",
-        headers={"X-Event-Count": str(stats["events"]), "X-File-Count": str(stats["files"])},
-        background=BackgroundTask(os.unlink, tmp),
-    )
-
-
-@app.post("/db/restore")
-def db_restore(
-    db: UploadFile = File(...),
-    waveforms: Optional[UploadFile] = File(None),
-) -> dict:
-    """Destructive: swap seismo_relay.db + unpack waveforms. Dormant (404) unless
-    SFM_DB_RESTORE_ENABLED is set."""
-    import os
-    import shutil as _shutil
-    import tempfile
-    if not db_snapshot.sfm_db_restore_enabled():
-        raise HTTPException(status_code=404, detail="Not found")
-
-    def _save(upload) -> Path:
-        fd, tmp = tempfile.mkstemp()
-        with os.fdopen(fd, "wb") as out:
-            _shutil.copyfileobj(upload.file, out)
-        return Path(tmp)
-
-    db_tmp = _save(db)
-    wf_tmp = _save(waveforms) if waveforms is not None else None
-    try:
-        return db_snapshot.restore_db_and_waveforms(
-            _get_db().db_path, _get_store().root, db_tmp, wf_tmp,
-        )
-    finally:
-        db_tmp.unlink(missing_ok=True)
-        if wf_tmp is not None:
-            wf_tmp.unlink(missing_ok=True)
-
-
 @app.get("/db/monitor_log")
 def db_monitor_log(
     serial:  Optional[str] = Query(None, description="Filter by unit serial"),

sfm/sfm_webapp.html

@@ -108,12 +108,6 @@
       color: var(--text);
     }
     .btn-ghost:hover { border-color: var(--blue-lt); color: var(--blue-lt); }
-    .btn-danger   { background: var(--red);    color: #fff; }
-    .btn-danger:hover:not(:disabled) { filter: brightness(1.15); }
-    .diag-result  { display:block; margin-top:6px; font-size:12px; opacity:.85;
-                    white-space:pre-wrap; word-break:break-word; }
-    .diag-result.ok    { color: var(--green); }
-    .diag-result.error { color: var(--red); }
     .btn:disabled { background: var(--surface2) !important; color: var(--text-mute) !important; cursor: not-allowed; border-color: var(--border2) !important; }
 
     /* #connect-btn styles moved to #live-connect-bar block */
@@ -916,7 +910,6 @@
   <button class="tab-btn"        data-tab="events"    onclick="switchTab('events')">Events</button>
   <button class="tab-btn"        data-tab="config"    onclick="switchTab('config')">Config</button>
   <button class="tab-btn"        data-tab="call-home" onclick="switchTab('call-home')">Call Home</button>
-  <button class="tab-btn"        data-tab="diagnostics" onclick="switchTab('diagnostics')">Diagnostics</button>
 </div>
 
 <!-- ════════════════════════════════════════════════════════════════
@@ -945,10 +938,6 @@
 <div id="tab-events" class="tab-pane" style="display:flex; flex-direction:column; overflow:hidden;">
 
   <div class="event-toolbar">
-    <button class="btn btn-ghost" id="load-events-btn" onclick="loadEventList()" disabled
-            title="Walk the device's event chain and list its stored events. This is the slow one — it reads every event header over the cellular link.">
-      ⟳ Load events
-    </button>
     <button class="btn btn-ghost" id="load-btn" onclick="loadWaveform()" disabled>Load Waveform</button>
     <button class="btn btn-ghost" id="save-btn" onclick="saveEventToDb()" disabled
             title="Download the full waveform from the device and save it to the SFM database + waveform store. Honors the Force refresh toggle.">
@@ -1216,77 +1205,6 @@
 
 </div><!-- end #tab-call-home -->
 
-<!-- ════════════════════════════════════════════════════════════════
-     TAB: Diagnostics
-═══════════════════════════════════════════════════════════════════ -->
-<div id="tab-diagnostics" class="tab-pane">
-
-  <div class="cfg-grid">
-
-    <div class="cfg-section">
-      <div class="cfg-section-title">Device State</div>
-      <div class="hint" style="margin-bottom:10px">
-        Fast probes — POLL plus one read each, about 2 s. None of these walk the event chain.
-      </div>
-      <div class="dev-table" id="diag-table"></div>
-      <div class="cfg-actions" style="margin-top:12px">
-        <button class="btn btn-ghost" id="diag-refresh-btn" onclick="refreshDiagnostics()" disabled>Refresh</button>
-        <span id="diag-status"></span>
-      </div>
-    </div>
-
-    <div class="cfg-section">
-      <div class="cfg-section-title">Actions</div>
-
-      <div class="cfg-field">
-        <label>Stop Monitoring</label>
-        <button class="btn btn-ghost" id="diag-stop-btn" onclick="diagStopMonitoring()" disabled>Send Stop (SUB 0x97)</button>
-        <div class="hint">Halts recording. On a unit triggering continuously, this is what breaks the call-home loop.</div>
-        <span class="diag-result" id="diag-stop-result"></span>
-      </div>
-
-      <div class="cfg-field">
-        <label>Disable Auto Call Home</label>
-        <button class="btn btn-ghost" id="diag-ach-btn" onclick="diagDisableAch()" disabled>Disable ACH</button>
-        <div class="hint">Stored events are left untouched (<code>rescue?erase=false</code>). The unit stops dialing out until ACH is re-enabled.</div>
-        <span class="diag-result" id="diag-ach-result"></span>
-      </div>
-
-      <div class="cfg-field">
-        <label>Erase All Events</label>
-        <input type="text" id="diag-erase-confirm" placeholder="Type the serial to enable"
-               oninput="diagCheckEraseConfirm()" autocomplete="off" />
-        <button class="btn btn-danger" id="diag-erase-btn" onclick="diagEraseEvents()" disabled>Erase Events</button>
-        <div class="hint">⚠ Permanent, and resets the event chain to key <code>0x01110000</code>. Download anything worth keeping first.</div>
-        <span class="diag-result" id="diag-erase-result"></span>
-      </div>
-    </div>
-
-    <div class="cfg-section">
-      <div class="cfg-section-title">Unresponsive Unit</div>
-      <div class="hint" style="margin-bottom:10px">
-        The escalation ladder from <code>docs/runbooks/wedged_unit_recovery.md</code>, for a unit too busy
-        to answer normal request/response. Prefer <b>Method A</b> — point the modem at an
-        <code>ach_server</code> and answer its call — before racing it with these.
-      </div>
-
-      <div class="cfg-field">
-        <label>Slow drip <span class="hint" style="display:inline">(one held session, a stop every 3 s)</span></label>
-        <button class="btn btn-ghost" id="diag-drip-btn" onclick="diagSlowDrip()" disabled>Run 120 s drip</button>
-        <div class="hint">Success is <code>bytes_received &gt; 0</code>. A full duration with <code>send_error: null</code> is <b>not</b> success on its own.</div>
-        <span class="diag-result" id="diag-drip-result"></span>
-      </div>
-
-      <div class="cfg-field">
-        <label>Blind stop <span class="hint" style="display:inline">(fire-and-forget, one attempt)</span></label>
-        <button class="btn btn-ghost" id="diag-blind-btn" onclick="diagBlindStop()" disabled>Send blind stop</button>
-        <span class="diag-result" id="diag-blind-result"></span>
-      </div>
-    </div>
-
-  </div>
-</div><!-- end #tab-diagnostics -->
-
 </div><!-- end #section-live -->
 
 <!-- ════════════════════════════════════════════════════════════════
@@ -1443,8 +1361,6 @@
 // ── State ──────────────────────────────────────────────────────────────────────
 let unitInfo      = null;
 let eventList     = [];
-let storageInfo   = null;   // /device/events/storage_range — cheap, read on connect
-let eventsLoaded  = false;  // the event chain walk is opt-in; see loadEventList()
 let currentEvent  = 0;
 let charts        = {};
 let geoAdcScale      = 6.206;
@@ -1542,7 +1458,6 @@ function switchTab(name) {
   if (name === 'units')    { if (!unitsLoaded)  loadUnits(); }
   if (name === 'monlog')   { if (!monlogLoaded) loadMonitorLog(); }
   if (name === 'sessions') { if (!sessLoaded)   loadSessions(); }
-  if (name === 'diagnostics' && devHost() && unitInfo) refreshDiagnostics();
 }
 
 // ── Connect ────────────────────────────────────────────────────────────────────
@@ -1563,13 +1478,18 @@ async function connectUnit() {
     btn.disabled = false; btn.textContent = 'Connect'; return;
   }
 
-  // Connecting deliberately does NOT walk the event chain.  That walk reads
+  setStatus('Fetching event list…', 'loading');
-  // every event header over the cellular link and can take minutes — or fail
+  try {
-  // outright on a unit whose buffer has wrapped past 0xFFFF.  Use the ~2 s
+    const r = await fetch(`${api()}/device/events?${deviceParams()}`);
-  // probes instead; the event list is opt-in via loadEventList().
+    if (!r.ok) { const e = await r.json().catch(() => ({})); throw new Error(e.detail || r.statusText); }
-  eventList = []; eventsLoaded = false;
+    const evData = await r.json();
-  setStatus('Reading device state…', 'loading');
+    eventList = evData.events || [];
-  storageInfo = await fetchJson(`/device/events/storage_range`).catch(() => null);
+    // Merge compliance from /device/events response (it re-reads it)
+    if (evData.device) unitInfo = { ...unitInfo, ...evData.device };
+  } catch (e) {
+    setStatus(`Event fetch failed: ${e.message}`, 'error');
+    btn.disabled = false; btn.textContent = 'Reconnect'; return;
+  }
 
   populateDeviceBar();
   populateDeviceTab();
@@ -1578,9 +1498,11 @@ async function connectUnit() {
 
   document.getElementById('device-bar').style.display    = 'flex';
   document.getElementById('monitor-panel').style.display = 'flex';
-  setEventButtonsEnabled();
+  document.getElementById('load-btn').disabled     = eventList.length === 0;
-  document.getElementById('load-events-btn').disabled = false;
+  document.getElementById('save-btn').disabled     = eventList.length === 0;
-  setDiagButtonsEnabled(true);
+  document.getElementById('download-btn').disabled = eventList.length === 0;
+  document.getElementById('prev-btn').disabled     = true;
+  document.getElementById('next-btn').disabled     = eventList.length <= 1;
   document.getElementById('cfg-read-btn').disabled  = false;
   document.getElementById('cfg-write-btn').disabled = false;
   document.getElementById('ch-read-btn').disabled   = false;
@@ -1588,9 +1510,7 @@ async function connectUnit() {
 
   btn.disabled = false; btn.textContent = 'Reconnect';
 
-  setStatus(storageInfo && storageInfo.is_empty
+  setStatus(`Connected — ${eventList.length} event${eventList.length !== 1 ? 's' : ''} stored.`, 'ok');
-    ? 'Connected — no events stored.'
-    : 'Connected. Event list not loaded (Events → Load events).', 'ok');
 
   // Fetch monitor status in background (non-blocking)
   refreshMonitorStatus().catch(() => {});
@@ -1602,48 +1522,6 @@ async function connectUnit() {
   }
 }
 
-// ── Shared fetch helper ────────────────────────────────────────────────────────
-async function fetchJson(path, opts) {
-  const sep = path.includes('?') ? '&' : '?';
-  const r = await fetch(`${api()}${path}${sep}${deviceParams()}`, opts);
-  const body = await r.json().catch(() => ({}));
-  if (!r.ok) throw new Error(body.detail || r.statusText);
-  return body;
-}
-
-function setEventButtonsEnabled() {
-  const n = eventList.length;
-  document.getElementById('load-btn').disabled     = n === 0;
-  document.getElementById('save-btn').disabled     = n === 0;
-  document.getElementById('download-btn').disabled = n === 0;
-  document.getElementById('prev-btn').disabled     = true;
-  document.getElementById('next-btn').disabled     = n <= 1;
-}
-
-// ── Event list (opt-in — this is the slow chain walk) ──────────────────────────
-async function loadEventList() {
-  if (!devHost()) { setStatus('Connect to a device first.', 'error'); return; }
-  const btn = document.getElementById('load-events-btn');
-  btn.disabled = true;
-  setStatus('Walking the event chain — this can take a while…', 'loading');
-  try {
-    const evData = await fetchJson('/device/events');
-    eventList = evData.events || [];
-    eventsLoaded = true;
-    // /device/events re-reads compliance; fold it in.
-    if (evData.device) unitInfo = { ...unitInfo, ...evData.device };
-  } catch (e) {
-    setStatus(`Event fetch failed: ${e.message}`, 'error');
-    btn.disabled = false; return;
-  }
-  populateDeviceBar();
-  populateDeviceTab();
-  populateEventChips();
-  setEventButtonsEnabled();
-  btn.disabled = false;
-  setStatus(`${eventList.length} event${eventList.length !== 1 ? 's' : ''} stored.`, 'ok');
-}
-
 // ── Device bar ─────────────────────────────────────────────────────────────────
 function populateDeviceBar() {
   qs('di-serial').textContent   = unitInfo.serial || '—';
@@ -1652,7 +1530,7 @@ function populateDeviceBar() {
   qs('di-sr').textContent       = cc.sample_rate ? `${cc.sample_rate} sps` : '—';
   qs('di-rt').textContent       = cc.record_time != null ? `${cc.record_time.toFixed(1)} s` : '—';
   qs('di-trig').textContent     = cc.trigger_level_geo != null ? `${cc.trigger_level_geo.toFixed(3)} in/s` : '—';
-  qs('di-count').textContent    = eventsLoaded ? eventList.length : '—';
+  qs('di-count').textContent    = eventList.length;
   qs('di-project').textContent  = cc.project || '—';
   qs('di-client').textContent   = cc.client  || '—';
   qs('di-operator').textContent = cc.operator || '—';
@@ -1782,8 +1660,7 @@ function populateDeviceTab() {
     { label:'DSP',            value: unitInfo.dsp_version || '—' },
     { label:'Model',          value: unitInfo.model || '—' },
     { label:'Manufacturer',   value: unitInfo.manufacturer || '—' },
-    { label:'Stored Events',  value: eventsLoaded ? eventList.length : 'not loaded' },
+    { label:'Stored Events',  value: eventList.length },
-    { label:'Storage Used',   value: storageUsedLabel() },
   ];
   for (const {label, value} of cardData) {
     const c = document.createElement('div');
@@ -1830,158 +1707,6 @@ function renderTable(id, rows) {
   }
 }
 
-// ── Diagnostics ────────────────────────────────────────────────────────────────
-// Everything here is a cheap probe (POLL + one read) or a single write.  None of
-// it walks the event chain.  See docs/runbooks/wedged_unit_recovery.md.
-
-function storageUsedLabel() {
-  if (!storageInfo) return '—';
-  if (storageInfo.is_empty) return 'empty';
-  const f = storageInfo.first_key, l = storageInfo.last_key;
-  return (f && l) ? `${f} → ${l}` : '—';
-}
-
-function setDiagButtonsEnabled(on) {
-  for (const id of ['diag-refresh-btn','diag-stop-btn','diag-ach-btn',
-                    'diag-drip-btn','diag-blind-btn']) {
-    const el = document.getElementById(id);
-    if (el) el.disabled = !on;
-  }
-  diagCheckEraseConfirm();
-}
-
-// Erase is guarded by typing the serial — auth answers "who", not "did you mean it".
-function diagCheckEraseConfirm() {
-  const box = document.getElementById('diag-erase-confirm');
-  const btn = document.getElementById('diag-erase-btn');
-  if (!box || !btn) return;
-  const serial = (unitInfo && unitInfo.serial) || '';
-  btn.disabled = !serial || box.value.trim().toUpperCase() !== serial.toUpperCase();
-}
-
-function diagResult(id, text, cls) {
-  const el = document.getElementById(id);
-  if (!el) return;
-  el.textContent = text;
-  el.className = 'diag-result' + (cls ? ' ' + cls : '');
-}
-
-async function refreshDiagnostics() {
-  if (!devHost()) return;
-  const st = document.getElementById('diag-status');
-  if (st) { st.textContent = 'Reading…'; st.className = 'loading'; }
-
-  const [mon, store, idx] = await Promise.all([
-    fetchJson('/device/monitor/status?force=true').catch(e => ({ _err: e.message })),
-    fetchJson('/device/events/storage_range').catch(e => ({ _err: e.message })),
-    fetchJson('/device/events/index').catch(e => ({ _err: e.message })),
-  ]);
-
-  if (!store._err) storageInfo = store;
-
-  const err = v => `<span style="color:var(--red)">${v}</span>`;
-  const rows = [];
-
-  rows.push(['Monitoring', mon._err ? err(mon._err)
-    : (mon.is_monitoring ? '<b>MONITORING</b>' : 'idle')]);
-  if (!mon._err) {
-    rows.push(['Battery', mon.battery_v != null ? `${mon.battery_v.toFixed(2)} V` : '—']);
-    if (mon.memory_total_bytes) {
-      const used = mon.memory_total_bytes - (mon.memory_free_bytes ?? 0);
-      const pct  = (used / mon.memory_total_bytes * 100).toFixed(1);
-      rows.push(['Memory used', `${used.toLocaleString()} / ${mon.memory_total_bytes.toLocaleString()} bytes (${pct}%)`]);
-    }
-  }
-
-  rows.push(['Event chain', store._err ? err(store._err) : storageUsedLabel()]);
-  if (!store._err) rows.push(['Chain empty', store.is_empty ? 'yes' : 'no']);
-
-  // SUB 0x08.  Known to report 0 on units with years of history — suspected
-  // field-offset bug in the decode, so show it but do not trust it.
-  rows.push(['Lifetime events', idx._err ? err(idx._err)
-    : `${idx.lifetime_count} <span class="hint" style="display:inline">(unreliable — see CHANGELOG)</span>`]);
-
-  renderTable('diag-table', rows);
-  populateDeviceTab();
-  if (st) { st.textContent = ''; st.className = ''; }
-}
-
-async function diagStopMonitoring() {
-  const btn = document.getElementById('diag-stop-btn');
-  btn.disabled = true; diagResult('diag-stop-result', 'Sending…');
-  try {
-    await fetchJson('/device/monitor/stop', { method: 'POST' });
-    diagResult('diag-stop-result', 'Stop acknowledged — recording halted.', 'ok');
-    refreshDiagnostics();
-  } catch (e) {
-    diagResult('diag-stop-result', `Failed: ${e.message}`, 'error');
-  }
-  btn.disabled = false;
-}
-
-async function diagDisableAch() {
-  const btn = document.getElementById('diag-ach-btn');
-  btn.disabled = true; diagResult('diag-ach-result', 'Writing call-home config…');
-  try {
-    const r = await fetchJson('/device/rescue?erase=false', { method: 'POST' });
-    const steps = (r.steps || []).map(s => s.step).join(' → ') || 'done';
-    diagResult('diag-ach-result', `ACH disabled (${steps}). Events untouched.`, 'ok');
-  } catch (e) {
-    diagResult('diag-ach-result', `Failed: ${e.message}`, 'error');
-  }
-  btn.disabled = false;
-}
-
-async function diagEraseEvents() {
-  const serial = (unitInfo && unitInfo.serial) || 'this unit';
-  if (!confirm(`Permanently erase ALL events on ${serial}?\n\nThis cannot be undone.`)) return;
-  const btn = document.getElementById('diag-erase-btn');
-  btn.disabled = true; diagResult('diag-erase-result', 'Erasing…');
-  try {
-    await fetchJson('/device/events/erase', { method: 'POST' });
-    diagResult('diag-erase-result', 'Events erased — chain reset to 0x01110000.', 'ok');
-    document.getElementById('diag-erase-confirm').value = '';
-    eventList = []; eventsLoaded = false;
-    setEventButtonsEnabled(); populateEventChips();
-    refreshDiagnostics();
-  } catch (e) {
-    diagResult('diag-erase-result', `Failed: ${e.message}`, 'error');
-  }
-  diagCheckEraseConfirm();
-}
-
-async function diagSlowDrip() {
-  const btn = document.getElementById('diag-drip-btn');
-  btn.disabled = true;
-  diagResult('diag-drip-result', 'Holding a session for 120 s…');
-  try {
-    const r = await fetchJson('/device/stop_monitoring_slow_drip?duration_s=120&interval_s=3',
-                              { method: 'POST' });
-    const good = (r.bytes_received || 0) > 0;
-    diagResult('diag-drip-result',
-      `drips ${r.drips_sent}  ·  held ${r.duration_s}s  ·  bytes back ${r.bytes_received}` +
-      (r.send_error ? `  ·  ${r.send_error}` : '') +
-      (good ? '  → device responded' : '  → no response; the modem may not be bridging'),
-      good ? 'ok' : 'error');
-  } catch (e) {
-    diagResult('diag-drip-result', `Failed: ${e.message}`, 'error');
-  }
-  btn.disabled = false;
-}
-
-async function diagBlindStop() {
-  const btn = document.getElementById('diag-blind-btn');
-  btn.disabled = true; diagResult('diag-blind-result', 'Sending…');
-  try {
-    const r = await fetchJson('/device/stop_monitoring_blind', { method: 'POST' });
-    diagResult('diag-blind-result',
-      `Sent ${r.bytes_sent ?? '?'} bytes, no response read (fire-and-forget).`, 'ok');
-  } catch (e) {
-    diagResult('diag-blind-result', `Failed: ${e.message}`, 'error');
-  }
-  btn.disabled = false;
-}
-
 // ── Config form ────────────────────────────────────────────────────────────────
 function populateConfigFromDeviceInfo() {
   if (!unitInfo) return;
@@ -3562,7 +3287,7 @@ if (currentSection === 'db') {
           <dt id="sc-l-bwsize">File size</dt>   <dd id="sc-f-bwsize">—</dd>
           <dt id="sc-l-sha">File sha256</dt>    <dd id="sc-f-sha">—</dd>
           <dt>Source kind</dt>      <dd id="sc-f-src">—</dd>
-          <dt title="When SFM received and stored this event — NOT the unit-local trigger time (see Timestamp at the top of the modal for that).">Time received</dt>
+          <dt title="When our server received and stored this event (sfm-db insert time, not the recording time)">Received by server at</dt>
                                     <dd id="sc-f-cap">—</dd>
         </dl>
       </div>

sfm/shape_metrics.py

@@ -1,127 +0,0 @@
-"""Waveform-shape metrics for false-trigger detection.
-
-A false trigger is an isolated impulse (quiet → spike → quiet); a real event
-rings for many cycles. Two numbers separate them: crest factor (how far the
-peak stands above the typical sample) and how many samples sit near the peak.
-"""
-from __future__ import annotations
-import numpy as np
-
-_GEO_CHANNELS = ("Tran", "Vert", "Long")
-NEAR_PEAK_FRACTION = 0.5   # a sample "near the peak" is >= this * peak amplitude
-
-
-def channel_shape(x) -> dict | None:
-    x = np.asarray(x, dtype=float)
-    if x.size < 2:
-        return None
-    peak = float(np.max(np.abs(x)))
-    if peak <= 0:
-        return None
-    rms = float(np.sqrt(np.mean(x ** 2)))
-    if rms <= 0:
-        return None
-    near = int(np.sum(np.abs(x) >= NEAR_PEAK_FRACTION * peak))
-    return {"crest_factor": peak / rms, "near_peak_count": near,
-            "sample_count": int(x.size)}
-
-
-def shape_from_samples(chans: dict) -> dict | None:
-    best_axis, best_peak, best_x = None, -1.0, None
-    for ax in _GEO_CHANNELS:
-        x = chans.get(ax)
-        if x is None:
-            continue
-        x = np.asarray(x, dtype=float)
-        if x.size < 2:
-            continue
-        p = float(np.max(np.abs(x)))
-        if p > best_peak:
-            best_axis, best_peak, best_x = ax, p, x
-    if best_axis is None:
-        return None
-    s = channel_shape(best_x)
-    if s is None:
-        return None
-    s["axis"] = best_axis
-    return s
-
-
-# ── Offset (DC-baseline) detection ────────────────────────────────────────────
-# A DC offset is a false trigger where the geophone baseline sits at a constant
-# non-zero floor (sensor bumped / settled / drifted) instead of oscillating
-# around zero. Brian's method (validated in scratch/offset_scan3.py): the
-# pre-trigger window is definitionally quiet, so a true offset shows |pre| off
-# zero AND stays flat across the record (pre ≈ mid ≈ end). A transient moves one
-# third relative to the others and is rejected by the spread test.
-# Thresholds are in in/s (the .h5 samples are already range-scaled); validated at
-# Normal range (10 in/s) — the only range in the fleet.
-OFFSET_FLOOR = 0.025        # |pre| at/above this reads as an off-zero baseline (5 A/D counts)
-OFFSET_MAX_SPREAD = 0.02    # max(pre,mid,end) - min(...) at/below this reads as flat/constant
-
-
-def _channel_offset(x, pretrig_n):
-    """Return (pre, spread, is_offset) for one channel, or None if unusable."""
-    x = np.asarray(x, dtype=float)
-    n = x.size
-    if n < 3:
-        return None
-    t = n // 3
-    pre = x[:pretrig_n] if (pretrig_n and 0 < pretrig_n < n) else x[:t]
-    mid, end = x[t:2 * t], x[2 * t:]
-    if pre.size == 0 or mid.size == 0 or end.size == 0:
-        return None
-    vals = [float(np.median(seg)) for seg in (pre, mid, end)]
-    spread = max(vals) - min(vals)
-    is_offset = abs(vals[0]) >= OFFSET_FLOOR and spread <= OFFSET_MAX_SPREAD
-    return vals[0], spread, is_offset
-
-
-def offset_from_samples(chans: dict, pretrig_n) -> dict | None:
-    """Detect a DC-offset false trigger across the geophone channels.
-
-    An event is offset if ANY geo channel's pre-trigger baseline is off zero and
-    flat across the record. Reports the tripping axis (or, if none trips, the
-    most-offset-like axis) with its ``pre``/``spread`` for transparency + tuning.
-    Returns None when no geo channel is usable.
-    """
-    results = []
-    for ax in _GEO_CHANNELS:
-        x = chans.get(ax)
-        if x is None:
-            continue
-        r = _channel_offset(x, pretrig_n)
-        if r is not None:
-            results.append((ax, r[0], r[1], r[2]))
-    if not results:
-        return None
-    offenders = [r for r in results if r[3]]
-    ax, pre, spread, _ = max(offenders or results, key=lambda r: abs(r[1]))
-    return {"offset": bool(offenders), "axis": ax,
-            "pre": round(pre, 6), "spread": round(spread, 6)}
-
-
-def shape_from_h5(path) -> dict | None:
-    import h5py
-    try:
-        with h5py.File(path, "r") as f:
-            chans = {ax: f[f"samples/{ax}"][:] for ax in _GEO_CHANNELS
-                     if f"samples/{ax}" in f}
-    except Exception:
-        return None
-    return shape_from_samples(chans)
-
-
-def offset_from_h5(path) -> dict | None:
-    """offset_from_samples fed from an event's .h5 (float32 in/s geo samples +
-    the pretrig_samples attribute)."""
-    import h5py
-    try:
-        with h5py.File(path, "r") as f:
-            chans = {ax: f[f"samples/{ax}"][:] for ax in _GEO_CHANNELS
-                     if f"samples/{ax}" in f}
-            pretrig_n = f.attrs.get("pretrig_samples")
-    except Exception:
-        return None
-    pretrig_n = int(pretrig_n) if pretrig_n is not None else 0
-    return offset_from_samples(chans, pretrig_n)

sfm/waveform_store.py

@@ -32,7 +32,6 @@ from __future__ import annotations
 import datetime
 import logging
 import pickle
-import re
 import shutil
 from pathlib import Path
 from typing import Optional, Union
@@ -42,7 +41,6 @@ from minimateplus.blastware_file import blastware_filename, write_blastware_file
 from minimateplus.framing import S3Frame
 from minimateplus.models import Event
 from sfm import event_hdf5
-from sfm.shape_metrics import shape_from_h5, offset_from_h5
 
 log = logging.getLogger("sfm.waveform_store")
 
@@ -264,20 +262,6 @@ class WaveformStore:
             serial, filename, filesize, len(a5_frames),
             hdf5_filename or "(skipped)", sidecar_path.name,
         )
-        _shape = shape_from_h5(hdf5_path) if hdf5_filename else None
-        _shape_rec = {
-            "shape_crest_factor":    _shape["crest_factor"],
-            "shape_near_peak_count": _shape["near_peak_count"],
-            "shape_sample_count":    _shape["sample_count"],
-            "shape_axis":            _shape["axis"],
-        } if _shape else {}
-        _offset = offset_from_h5(hdf5_path) if hdf5_filename else None
-        _offset_rec = {
-            "shape_offset":        1 if _offset["offset"] else 0,
-            "shape_offset_axis":   _offset["axis"],
-            "shape_offset_pre":    _offset["pre"],
-            "shape_offset_spread": _offset["spread"],
-        } if _offset else {}
         return {
             "filename":            filename,
             "filesize":            filesize,
@@ -285,8 +269,6 @@ class WaveformStore:
             "a5_pickle_filename":  a5_path.name,
             "hdf5_filename":       hdf5_filename,
             "sidecar_filename":    sidecar_path.name,
-            **_shape_rec,
-            **_offset_rec,
         }
 
     def save_imported_bw(
@@ -380,16 +362,8 @@ class WaveformStore:
 
         # Resolve serial.  blastware_filename derives a 4-char prefix from
         # the numeric serial (e.g. BE11529 → M529); we go the other way
-        # if a hint wasn't given.  The filename carries only the NUMBER,
+        # via the source filename if a hint wasn't given.
-        # so read the family prefix out of the body first — a BlastMate
+        serial = serial_hint or _serial_from_bw_filename(source_path.name) or "UNKNOWN"
-        # ("BA") filed as "BE" is a unit that does not exist.  The
-        # filename-only decoder stays as the last resort.
-        serial = (
-            serial_hint
-            or _serial_from_bw_bytes(bw_bytes, source_path.name)
-            or _serial_from_bw_filename(source_path.name)
-            or "UNKNOWN"
-        )
 
         # Use the source filename verbatim — it already encodes timestamp
         # + record type per BW's AB0T scheme, and we want to preserve it
@@ -471,20 +445,6 @@ class WaveformStore:
             "h5=%s (no .a5.pkl — A5 source unavailable for BW-imported files)",
             serial, filename, filesize, hdf5_filename or "(skipped)",
         )
-        _shape = shape_from_h5(hdf5_path) if hdf5_filename else None
-        _shape_rec = {
-            "shape_crest_factor":    _shape["crest_factor"],
-            "shape_near_peak_count": _shape["near_peak_count"],
-            "shape_sample_count":    _shape["sample_count"],
-            "shape_axis":            _shape["axis"],
-        } if _shape else {}
-        _offset = offset_from_h5(hdf5_path) if hdf5_filename else None
-        _offset_rec = {
-            "shape_offset":        1 if _offset["offset"] else 0,
-            "shape_offset_axis":   _offset["axis"],
-            "shape_offset_pre":    _offset["pre"],
-            "shape_offset_spread": _offset["spread"],
-        } if _offset else {}
         return ev, {
             "filename":           filename,
             "filesize":           filesize,
@@ -493,8 +453,6 @@ class WaveformStore:
             "hdf5_filename":      hdf5_filename,
             "sidecar_filename":   sidecar_path.name,
             "serial":             serial,
-            **_shape_rec,
-            **_offset_rec,
         }
 
     def save_imported_idf(
@@ -509,21 +467,21 @@ class WaveformStore:
         Ingest a Thor (Micromate Series IV) IDF event file (`.IDFW` or
         `.IDFH`) produced by Thor's TXT exporter.
 
+        Thor binaries are stored as opaque bytes — seismo-relay doesn't
+        yet decode the proprietary IDF binary format (codec slot lives
+        at ``micromate/idf_file.py``).  Device-authoritative metadata
+        comes from the paired ``.IDFW.txt`` / ``.IDFH.txt`` sidecar
+        when supplied.
+
         Workflow:
-          1. For sig-A `.IDFW` binaries, decode samples + binary metadata
+          1. Parse the paired TXT report (when supplied) via
-             via ``micromate.idf_file.read_idf_file()``.  Failure or
+             ``micromate.parse_idf_report`` → dict.
-             non-IDFW path falls through to the .txt-only flow.
+          2. Wrap parsed dict + filename into a typed ``micromate.IdfEvent``.
-          2. Parse the paired TXT report (when supplied) via
+          3. Copy bytes verbatim into ``<root>/<serial>/<filename>``.
-             ``micromate.parse_idf_report`` → dict.  TXT remains the
+          4. Bridge IdfEvent → ``minimateplus.Event`` (for the existing
-             source of truth for fields the binary doesn't yet supply
+             sidecar / DB insert machinery) via
-             (full peak set with ZC freq / Time of Peak, sensor self-check,
+             ``IdfEvent.to_minimateplus_event(waveform_key)``.
-             firmware string, project strings).
+          5. Write the ``.sfm.json`` sidecar with
-          3. Wrap parsed dict + filename into a typed ``micromate.IdfEvent``.
-          4. Copy bytes verbatim into ``<root>/<serial>/<filename>``.
-          5. Bridge IdfEvent → ``minimateplus.Event`` and attach
-             ``raw_samples`` from the binary decoder (when available).
-          6. Write the `.h5` clean-waveform file when samples decoded.
-          7. Write the ``.sfm.json`` sidecar with
              ``source.kind = "idf-import"`` and the full raw IDF report
              under ``extensions.idf_report``.
 
@@ -532,38 +490,7 @@ class WaveformStore:
         """
         from micromate import IdfEvent, parse_idf_report
 
-        # 1. Binary decode (sig-A IDFW and IDFH).  Non-fatal: any failure
+        # Parse the .txt sidecar (best-effort; non-fatal on failure).
-        # leaves samples / binary metadata unfilled and we proceed with
-        # the .txt path as before.
-        idf_samples: Optional[dict] = None
-        idf_intervals: Optional[list] = None
-        binary_md = None
-        binary_peaks = None
-        is_histogram = False
-        try:
-            from micromate.idf_file import read_idf_file
-            # Pass idf_bytes through `data=` — at this point in the flow
-            # the binary hasn't been written to disk yet, so the codec
-            # can't read from source_path.  We still pass source_path so
-            # the codec has the filename for error messages + .IDFH
-            # suffix detection.
-            res = read_idf_file(source_path, data=idf_bytes)
-            idf_samples = res.samples or None
-            idf_intervals = res.intervals
-            is_histogram = res.intervals is not None
-            binary_md = res.binary_metadata
-            binary_peaks = res.event.peaks
-        except NotImplementedError:
-            # sig-B — codec doesn't handle this yet.
-            pass
-        except Exception as exc:
-            log.warning(
-                "save_imported_idf: binary codec failed for %s: %s — "
-                "falling back to .txt-only ingest",
-                source_path.name, exc,
-            )
-
-        # 2. Parse the .txt sidecar (best-effort; non-fatal on failure).
         report_dict: dict = {}
         if idf_report_text is not None:
             try:
@@ -574,69 +501,17 @@ class WaveformStore:
                     exc,
                 )
 
-        # 3. Backfill report_dict with binary metadata for fields the
+        # Build the typed IdfEvent.  Filename is authoritative for
-        # .txt didn't supply.  Binary takes precedence on tied fields
-        # where the binary is more reliable (timestamp, sample_rate),
-        # and fills in fields entirely missing from the .txt.
-        if binary_md is not None:
-            if binary_md.serial and not report_dict.get("serial_number"):
-                report_dict["serial_number"] = binary_md.serial
-            if binary_md.event_datetime and not report_dict.get("event_datetime"):
-                report_dict["event_datetime"] = binary_md.event_datetime
-            if binary_md.sample_rate and not report_dict.get("sample_rate"):
-                report_dict["sample_rate"] = binary_md.sample_rate
-            if binary_md.record_time_sec and not report_dict.get("record_time_sec"):
-                report_dict["record_time_sec"] = binary_md.record_time_sec
-            # Calibration date (binary) vs calibration text (.txt) cohabit
-            # under different keys; no overwrite needed.
-            if binary_md.event_datetime and not report_dict.get("event_type"):
-                report_dict["event_type"] = (
-                    "Full Histogram" if is_histogram else "Full Waveform"
-                )
-
-        # Binary-derived peaks fill in when the .txt didn't supply them.
-        #
-        # The old justification for this precedence -- "binary peaks are ~3%
-        # low vs the .txt" -- was a decoder bug (geo LSB 0.0003 instead of
-        # 0.000310308) and was fixed 2026-09-10; the binary now agrees with
-        # Thor's own export per-sample.  The .txt still wins when present
-        # because it is what the operator sees in Thor's report.
-        #
-        # ⚠ One case where the .txt is the *less* accurate of the two:
-        # Thor floors displayed histogram PPV at 0.0050 in/s, so on quiet
-        # IDFH events the .txt reports 0.0050 while the binary decodes the
-        # true ~0.0025.  41.4% of prod IDFH sidecars carry a component PPV
-        # larger than their own vector sum because of it.  Left as-is
-        # deliberately, so stored peaks keep matching Thor's report.
-        if binary_peaks is not None:
-            if binary_peaks.transverse_ips and not report_dict.get("tran_ppv"):
-                report_dict["tran_ppv"] = binary_peaks.transverse_ips
-            if binary_peaks.vertical_ips and not report_dict.get("vert_ppv"):
-                report_dict["vert_ppv"] = binary_peaks.vertical_ips
-            if binary_peaks.longitudinal_ips and not report_dict.get("long_ppv"):
-                report_dict["long_ppv"] = binary_peaks.longitudinal_ips
-
-        # 4. Build the typed IdfEvent.  Filename is authoritative for
         # (serial, timestamp, kind); the report's event_datetime takes
         # precedence over the filename timestamp inside from_report().
         idf_event = IdfEvent.from_report(report_dict, source_path.name)
 
-        # The binary mic peak (psi) isn't carried through from_report() —
-        # IdfReport.from_dict only sees the .txt's dB(L) value.  Pull the
-        # binary-derived ``mic_pspl_psi`` onto the typed IdfEvent so the
-        # downstream bridge can populate ``PeakValues.micl`` (psi-shaped)
-        # and the h5 writer's per-count mic factor lands at a sensible
-        # value.  Without this, the h5 mic chart auto-scales against the
-        # dB(L) value-as-pseudo-psi and renders ~flat.
-        if binary_peaks is not None and binary_peaks.mic_pspl_psi is not None:
-            idf_event.peaks.mic_pspl_psi = binary_peaks.mic_pspl_psi
-
         # Operator-supplied serial_hint wins over the binary's filename
         # prefix when both are present (e.g. callers passing a known-good
         # serial that overrides a misnamed export).
         serial = serial_hint or idf_event.serial or "UNKNOWN"
 
-        # 5. Filesystem write of binary bytes.
+        # Filesystem write.
         filename = source_path.name
         bw_path = self._serial_dir(serial) / filename
         bw_path.write_bytes(idf_bytes)
@@ -648,64 +523,13 @@ class WaveformStore:
         # surrogate — every distinct binary maps to a distinct row.
         waveform_key = bytes.fromhex(sha256)[:16]
 
-        # 6. Bridge to minimateplus.Event for the existing sidecar / DB
+        # Bridge to minimateplus.Event for the existing sidecar / DB
         # insert paths.  See IdfEvent.to_minimateplus_event() for the
         # caveats of this bridge (mic units, missing fields → sidecar).
         ev = idf_event.to_minimateplus_event(waveform_key)
 
-        # Attach the decoded sample arrays.  Thor's decoder counts use
+        # Write the sidecar.  Source kind "idf-import" was added to the
-        # LSB = 0.0003 in/s for geo (vs BW's 16-count units at 0.005 in/s)
+        # allow-list in event_file_io.event_to_sidecar_dict for this.
-        # — the .h5 writer's geo_range="normal" yields LSB = 10/32768
-        # ≈ 0.000305 in/s, so plotted samples come out ~1.7% high.
-        # Acceptable known offset; refine with a Thor-aware h5 path later.
-        if idf_samples is not None:
-            ev.raw_samples = idf_samples
-            n_samples = max((len(idf_samples.get(ch, [])) for ch in ("Tran", "Vert", "Long", "MicL")), default=0)
-            ev.total_samples = ev.total_samples or n_samples
-            # Sensor self-check traces from the IDFW fixed header (waveform
-            # events only; {} on histograms / when absent).  Carried on the
-            # bridged Event so the .h5 writer persists them like series-3.
-            from micromate.sensor_check import decode_idf_sensor_check
-            ev.sensor_check = decode_idf_sensor_check(idf_bytes) or None
-
-        # For IDFH histograms there are no per-sample waveform arrays — the
-        # device stores one peak ADC count per interval per channel.  Synthesise
-        # a 1-sample-per-interval array so the existing h5+renderer pipeline
-        # (which groups samples down to ``n_intervals`` bars via max-per-group)
-        # produces a non-blank histogram chart.  Each "sample" is the peak ADC
-        # count for that interval, so the h5 writer's ``count × geo_fs/32768``
-        # conversion yields the right physical value for the bar height.
-        if is_histogram and idf_intervals:
-            hist_samples = {
-                "Tran": [iv.peak_count("Tran") for iv in idf_intervals],
-                "Vert": [iv.peak_count("Vert") for iv in idf_intervals],
-                "Long": [iv.peak_count("Long") for iv in idf_intervals],
-                "MicL": [iv.peak_count("MicL") for iv in idf_intervals],
-            }
-            ev.raw_samples = hist_samples
-            ev.total_samples = ev.total_samples or len(idf_intervals)
-
-        # 7. Write the .h5 clean-waveform file when we have samples to write
-        # (either the IDFW per-sample stream, or the IDFH synthesised per-
-        # interval peak array).  The renderer treats both shapes the same way.
-        hdf5_filename: Optional[str] = None
-        if ev.raw_samples:
-            hdf5_path = self.hdf5_path_for(serial, filename)
-            try:
-                event_hdf5.write_event_hdf5(
-                    hdf5_path, ev,
-                    serial=serial,
-                    geo_range="normal",   # Thor's geo full scale is also 10 in/s (Normal)
-                    source_kind="idf-import",
-                )
-                hdf5_filename = hdf5_path.name
-            except Exception as exc:
-                log.warning(
-                    "save_imported_idf: HDF5 write failed for %s: %s — continuing without .h5",
-                    hdf5_path, exc,
-                )
-
-        # 8. Write the sidecar.  Source kind "idf-import" is on the allow-list.
         sidecar_path = self.sidecar_path_for(serial, filename)
         existing_review = None
         if sidecar_path.exists():
@@ -730,85 +554,21 @@ class WaveformStore:
         # Time of Peak, sensor self-check, calibration, firmware).
         if report_dict:
             sidecar["extensions"]["idf_report"] = report_dict
-
-        # Project the IDF report into the BW report sidecar shape so the
-        # existing Event Report PDF pipeline (sfm/report_pdf.py) can
-        # render Thor events without needing a separate code path.  Thor
-        # data is 95% the same metric set as BW — the adapter handles
-        # the field-name mapping.
-        if report_dict or binary_md is not None:
-            try:
-                from micromate.idf_to_bw_report import build_bw_report_from_idf
-                sidecar["bw_report"] = build_bw_report_from_idf(
-                    report_dict or {},
-                    binary_md=binary_md,
-                    intervals=idf_intervals,
-                    is_histogram=is_histogram,
-                )
-            except Exception as exc:
-                log.warning(
-                    "save_imported_idf: idf→bw_report adapter failed for %s: %s — "
-                    "report PDF will fall back to DB-only fields",
-                    filename, exc,
-                )
-        # For histograms, also stash the binary-decoded per-interval
-        # records so the UI / report layer doesn't need to re-walk the
-        # IDFH file at render time.
-        if idf_intervals is not None:
-            sidecar["extensions"]["idf_intervals"] = [
-                {
-                    "offset":     iv.offset,
-                    "tran_peak":  iv.peak_count("Tran"),
-                    "tran_halfp": iv.tran_halfp,
-                    "tran_freq":  iv.freq_hz("Tran"),
-                    "vert_peak":  iv.peak_count("Vert"),
-                    "vert_halfp": iv.vert_halfp,
-                    "vert_freq":  iv.freq_hz("Vert"),
-                    "long_peak":  iv.peak_count("Long"),
-                    "long_halfp": iv.long_halfp,
-                    "long_freq":  iv.freq_hz("Long"),
-                    "mic_peak":   iv.peak_count("MicL"),
-                    "mic_halfp":  iv.micl_halfp,
-                    "mic_freq":   iv.freq_hz("MicL"),
-                }
-                for iv in idf_intervals
-            ]
         event_file_io.write_sidecar(sidecar_path, sidecar)
 
         log.info(
             "WaveformStore.save_imported_idf serial=%s filename=%s filesize=%d "
-            "kind=%s report_attached=%s binary_decoded=%s h5=%s intervals=%d",
+            "report_attached=%s",
-            serial, filename, filesize,
+            serial, filename, filesize, bool(report_dict),
-            "histogram" if is_histogram else "waveform",
-            bool(report_dict),
-            (idf_samples is not None) or (idf_intervals is not None),
-            hdf5_filename or "(skipped)",
-            len(idf_intervals) if idf_intervals else 0,
         )
-        _shape = shape_from_h5(hdf5_path) if hdf5_filename else None
-        _shape_rec = {
-            "shape_crest_factor":    _shape["crest_factor"],
-            "shape_near_peak_count": _shape["near_peak_count"],
-            "shape_sample_count":    _shape["sample_count"],
-            "shape_axis":            _shape["axis"],
-        } if _shape else {}
-        _offset = offset_from_h5(hdf5_path) if hdf5_filename else None
-        _offset_rec = {
-            "shape_offset":        1 if _offset["offset"] else 0,
-            "shape_offset_axis":   _offset["axis"],
-            "shape_offset_pre":    _offset["pre"],
-            "shape_offset_spread": _offset["spread"],
-        } if _offset else {}
         return ev, {
             "filename":           filename,
             "filesize":           filesize,
             "sha256":             sha256,
             "a5_pickle_filename": None,
-            "hdf5_filename":      hdf5_filename,
+            "hdf5_filename":      None,
             "sidecar_filename":   sidecar_path.name,
             "serial":             serial,
-            **_shape_rec,
-            **_offset_rec,
         }
 
     def load_a5(self, serial: str, filename: str) -> Optional[list[S3Frame]]:
@@ -865,24 +625,20 @@ class WaveformStore:
 
 # ── helpers ─────────────────────────────────────────────────────────────────────
 
-def _serial_number_from_bw_filename(name: str) -> Optional[int]:
+def _serial_from_bw_filename(name: str) -> Optional[str]:
     """
-    Reverse of `blastware_filename`'s serial-prefix encoding — the NUMBER only.
+    Reverse of `blastware_filename`'s serial-prefix encoding.
 
     BW filename format (V10.72): `<P><serial3><stem4>.<ext>`
     where P = chr(ord('B') + floor(serial // 1000))
     and serial3 = f"{serial % 1000:03d}".
 
     Examples (from CLAUDE.md verification archive):
-      P036... → 14036       H907... → 6907
+      P036... → BE14036       H907... → BE6907
-      M529... → 11529       T003... → 18003
+      M529... → BE11529       T003... → BE18003
-      L895... → 10895
 
-    ⚠ The filename encodes **only the number**.  The two-letter family
+    Returns the inferred BE-prefix serial (e.g. "BE11529") or None when
-    prefix is NOT in it — "BE" is a MiniMate Plus, "BA" a BlastMate — so
+    the filename doesn't match the expected pattern.
-    the prefix has to come from the file body (`_serial_from_bw_bytes`)
-    or from an explicit hint.  Returns None when the filename doesn't
-    match the expected pattern.
     """
     if not name:
         return None
@@ -895,43 +651,5 @@ def _serial_number_from_bw_filename(name: str) -> Optional[int]:
     if prefix_letter < "B":
         return None
     thousands = ord(prefix_letter) - ord("B")
-    return thousands * 1000 + int(base[1:4])
+    serial_num = thousands * 1000 + int(base[1:4])
-
+    return f"BE{serial_num}"
-
-_BW_SERIAL_RE = re.compile(rb"[A-Z]{2}\d{3,6}")
-
-
-def _serial_from_bw_bytes(data: bytes, name: str) -> Optional[str]:
-    """
-    Read the real serial — prefix included — out of a BW file body.
-
-    The body carries the serial as a plain ASCII string ("BE9558",
-    "BA10895").  We accept a candidate only when its numeric part matches
-    the number the filename encodes, which keeps a stray byte sequence in
-    the sample stream from being mistaken for a serial.
-
-    Returns None when the filename number can't be derived or no
-    candidate in the body agrees with it — the caller then falls back.
-    """
-    num = _serial_number_from_bw_filename(name)
-    if num is None or not data:
-        return None
-    for match in _BW_SERIAL_RE.findall(data):
-        candidate = match.decode("ascii", errors="replace")
-        if candidate[2:].lstrip("0") == str(num):
-            return candidate
-    return None
-
-
-def _serial_from_bw_filename(name: str) -> Optional[str]:
-    """
-    Best-effort serial from the filename alone.
-
-    ⚠ The family prefix is a **guess** — the filename does not carry it.
-    "BE" is right for every MiniMate Plus but wrong for a BlastMate, whose
-    serials start "BA".  Prefer `_serial_from_bw_bytes` whenever the file
-    body is at hand; this exists for callers that only have a name
-    (log lines, dry-run output).
-    """
-    num = _serial_number_from_bw_filename(name)
-    return None if num is None else f"BE{num}"

tests/test_backfill_event_shape.py

@@ -1,35 +0,0 @@
-from __future__ import annotations
-import numpy as np, h5py
-
-from sfm.database import SeismoDb
-from sfm.waveform_store import WaveformStore
-from scripts.backfill_event_shape import backfill_shape
-from minimateplus.models import Event
-
-
-def _h5(path, long):
-    with h5py.File(path, "w") as f:
-        g = f.create_group("samples")
-        for k in ("Tran", "Vert"):
-            g.create_dataset(k, data=np.zeros(1024, "float32"))
-        g.create_dataset("Long", data=np.asarray(long, "float32"))
-
-
-def test_backfill_updates_shape_and_is_idempotent(tmp_path):
-    db = SeismoDb(tmp_path / "s.db")
-    store = WaveformStore(tmp_path / "waveforms")
-    ev = Event(index=0)
-    ev._waveform_key = bytes.fromhex("0111abcd")
-    db.insert_events([ev], serial="BE1",
-                     waveform_records={ev._waveform_key.hex():
-                                       {"filename": "F.CE0W", "filesize": 10}})
-    # place the .h5 where store.paths_for expects it
-    long = np.zeros(1024); long[100] = 0.48
-    _h5(store.hdf5_path_for("BE1", "F.CE0W"), long)
-
-    c1 = backfill_shape(db, store)
-    assert c1["updated"] == 1
-    row = db.query_events(serial="BE1")[0]
-    assert row["shape_axis"] == "Long" and row["shape_near_peak_count"] <= 3
-    c2 = backfill_shape(db, store)                  # idempotent: re-run overwrites same values
-    assert db.query_events(serial="BE1")[0]["shape_crest_factor"] == row["shape_crest_factor"]

tests/test_binary_annotate.py

@@ -1,50 +0,0 @@
-"""Structural annotation of a Series-3 Blastware binary (for the seismo_lab
-Binary Inspector). The annotator maps byte ranges to labelled spans; anything
-the decoder can't account for is a first-class ``unknown`` span, so the whole
-file is tiled and the gaps (candidate FFT/spectral data) are visible.
-"""
-from pathlib import Path
-
-from minimateplus.binary_annotate import annotate_blastware_binary, Span
-
-# A known-good full-3-channel Series-3 waveform binary (the V70 cracking fixture).
-FIXTURE = Path(__file__).parent / "fixtures" / "5-11-26" / "M529LL1L.V70"
-
-
-def _raw() -> bytes:
-    return FIXTURE.read_bytes()
-
-
-def test_spans_tile_the_whole_file():
-    raw = _raw()
-    spans = annotate_blastware_binary(raw)
-    assert spans, "expected at least one span"
-    assert spans[0].start == 0
-    assert spans[-1].end == len(raw)
-    for a, b in zip(spans, spans[1:]):
-        assert a.end == b.start, f"gap/overlap between {a!r} and {b!r}"
-    for s in spans:
-        assert s.start < s.end, f"empty/negative span {s!r}"
-
-
-def test_strt_record_is_located():
-    raw = _raw()
-    spans = annotate_blastware_binary(raw)
-    strt = [s for s in spans if s.kind == "strt"]
-    assert strt, "expected a STRT region"
-    assert raw[strt[0].start : strt[0].start + 4] == b"STRT"
-
-
-def test_geo_sample_records_annotated():
-    raw = _raw()
-    spans = annotate_blastware_binary(raw)
-    chans = {s.label.split()[0] for s in spans if s.kind == "sample"}
-    # V70 is a full three-geo-channel event.
-    assert {"Tran", "Vert", "Long"} <= chans, f"expected geo records, got {chans}"
-
-
-def test_footer_is_last():
-    raw = _raw()
-    spans = annotate_blastware_binary(raw)
-    assert spans[-1].kind == "footer"
-    assert spans[-1].end - spans[-1].start == 26

tests/test_compliance.py

@@ -1,35 +0,0 @@
-"""USBM/OSMRE compliance curve + scatter logic (sfm.compliance).
-Rendering is verified visually against Blastware reports."""
-import math
-
-import numpy as np
-
-from sfm.compliance import limit_at, channel_compliance_points
-
-
-def test_osmre_velocity_segments():
-    assert abs(limit_at(6.0) - 0.75) < 1e-9      # 3.5–12 Hz flat
-    assert abs(limit_at(50.0) - 2.00) < 1e-9     # 30–100 Hz flat
-
-
-def test_displacement_segments():
-    assert abs(limit_at(2.0) - 2 * math.pi * 2.0 * 0.030) < 1e-9    # low-freq 0.030 in
-    assert abs(limit_at(20.0) - 2 * math.pi * 20.0 * 0.008) < 1e-9  # rising diagonal 0.008 in
-
-
-def test_limit_clamps_below_1hz():
-    assert limit_at(0.1) == limit_at(1.0)
-
-
-def test_scatter_ceiling_is_ppv_at_dominant_freq():
-    # ~27 Hz blast-like trace whose energy peaks mid-record (inside full cycles,
-    # as a real event does): the scatter cloud's ceiling is the trace PPV and the
-    # top point sits near the dominant frequency.
-    sps, n = 1024.0, 3328
-    t = np.arange(n) / sps
-    env = np.exp(-((t - 1.5) ** 2) / (2 * 0.3 ** 2))
-    x = 0.9 * env * np.sin(2 * np.pi * 27.0 * t)
-    f, v = channel_compliance_points(x, sps)
-    assert len(f) > 20
-    assert v.max() >= 0.99 * np.abs(x).max()
-    assert 20.0 < f[int(np.argmax(v))] < 35.0

tests/test_db_events_exposes_shape.py

@@ -1,29 +0,0 @@
-from __future__ import annotations
-import os, sys
-from pathlib import Path
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-from sfm.database import SeismoDb
-from minimateplus.models import Event
-
-SHAPE_COLS = (
-    "shape_crest_factor",
-    "shape_near_peak_count",
-    "shape_sample_count",
-    "shape_axis",
-)
-
-
-def test_query_events_row_includes_shape_keys(tmp_path: Path):
-    db = SeismoDb(tmp_path / "s.db")
-
-    ev = Event(index=0)
-    ev._waveform_key = bytes.fromhex("0111abcd")
-
-    db.insert_events([ev], serial="BE1")
-    row = db.query_events(serial="BE1")[0]
-
-    for k in SHAPE_COLS:
-        assert k in row
-        assert row[k] is None

tests/test_db_snapshot.py

@@ -1,163 +0,0 @@
-"""
-test_db_snapshot.py — unit tests for sfm/db_snapshot.py (snapshot/restore
-helpers behind the /db/snapshot, /db/waveforms/recent.zip, /db/restore routes).
-
-Run:
-    python -m pytest tests/test_db_snapshot.py -v
-"""
-from __future__ import annotations
-import os
-import sys
-import sqlite3
-from pathlib import Path
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-
-def _make_db(path: Path, rows: int = 3) -> None:
-    conn = sqlite3.connect(str(path))
-    conn.execute("CREATE TABLE t (id INTEGER PRIMARY KEY, v TEXT)")
-    conn.executemany("INSERT INTO t (v) VALUES (?)", [(f"row{i}",) for i in range(rows)])
-    conn.commit()
-    conn.close()
-
-
-def _seed_event(db_path: Path, serial: str, fn: str, ts: str) -> None:
-    conn = sqlite3.connect(str(db_path))
-    conn.execute(
-        "INSERT INTO events (id, serial, waveform_key, timestamp, "
-        "blastware_filename, a5_pickle_filename, sidecar_filename, false_trigger) "
-        "VALUES (?, ?, ?, ?, ?, ?, ?, 0)",
-        (f"id-{fn}", serial, fn[:8], ts, fn, f"{fn}.a5.pkl", f"{fn}.sfm.json"),
-    )
-    conn.commit()
-    conn.close()
-
-
-def test_snapshot_db_produces_queryable_copy(tmp_path):
-    from sfm.db_snapshot import snapshot_db
-    src = tmp_path / "src.db"
-    _make_db(src, rows=5)
-    dest = tmp_path / "snap.db"
-    out = snapshot_db(src, dest)
-    assert out == dest and dest.exists()
-    conn = sqlite3.connect(str(dest))
-    n = conn.execute("SELECT COUNT(*) FROM t").fetchone()[0]
-    conn.close()
-    assert n == 5
-
-
-def test_recent_waveforms_zip_bundles_all_per_event_files(tmp_path):
-    import zipfile
-    from sfm.database import SeismoDb
-    from sfm.waveform_store import WaveformStore
-    from sfm.db_snapshot import build_recent_waveforms_zip
-
-    db = SeismoDb(tmp_path / "seismo_relay.db")   # creates schema on init
-    store = WaveformStore(tmp_path / "waveforms")
-
-    def add(serial, fn, ts):
-        d = store.root / serial
-        d.mkdir(parents=True, exist_ok=True)
-        for suffix in ("", ".h5", ".a5.pkl", ".sfm.json"):
-            (d / f"{fn}{suffix}").write_bytes(b"data")
-        _seed_event(db.db_path, serial, fn, ts)
-
-    add("BE111", "AAAA1111.OE0H", "2026-06-01T00:00:00")
-    add("BE111", "BBBB2222.OE0H", "2026-06-02T00:00:00")   # newest
-
-    dest = tmp_path / "wf.zip"
-    stats = build_recent_waveforms_zip(db, store, n=1, dest_path=dest)
-
-    assert stats["events"] == 1
-    assert stats["files"] == 4
-    with zipfile.ZipFile(dest) as zf:
-        names = set(zf.namelist())
-    assert names == {
-        "BE111/BBBB2222.OE0H",
-        "BE111/BBBB2222.OE0H.h5",
-        "BE111/BBBB2222.OE0H.a5.pkl",
-        "BE111/BBBB2222.OE0H.sfm.json",
-    }
-
-
-def test_sfm_db_restore_enabled_reads_env(monkeypatch):
-    from sfm.db_snapshot import sfm_db_restore_enabled
-    monkeypatch.delenv("SFM_DB_RESTORE_ENABLED", raising=False)
-    assert sfm_db_restore_enabled() is False
-    monkeypatch.setenv("SFM_DB_RESTORE_ENABLED", "true")
-    assert sfm_db_restore_enabled() is True
-    monkeypatch.setenv("SFM_DB_RESTORE_ENABLED", "0")
-    assert sfm_db_restore_enabled() is False
-
-
-def test_restore_swaps_db_and_unpacks_waveforms(tmp_path):
-    import zipfile
-    from sfm.db_snapshot import restore_db_and_waveforms
-
-    db_path = tmp_path / "seismo_relay.db"
-    _make_db(db_path, rows=1)
-    (tmp_path / "seismo_relay.db-wal").write_bytes(b"stale")   # must be cleared
-
-    new_db = tmp_path / "incoming.db"
-    _make_db(new_db, rows=9)
-
-    wf_root = tmp_path / "waveforms"
-    wf_root.mkdir()
-    wf_zip = tmp_path / "wf.zip"
-    with zipfile.ZipFile(wf_zip, "w") as zf:
-        zf.writestr("BE111/AAAA.OE0H", b"x")
-        zf.writestr("BE111/AAAA.OE0H.h5", b"x")
-
-    result = restore_db_and_waveforms(db_path, wf_root, new_db, wf_zip)
-
-    conn = sqlite3.connect(str(db_path))
-    assert conn.execute("SELECT COUNT(*) FROM t").fetchone()[0] == 9
-    conn.close()
-    assert not (tmp_path / "seismo_relay.db-wal").exists()
-    assert (wf_root / "BE111" / "AAAA.OE0H.h5").exists()
-    assert result["ok"] and result["waveforms_unpacked"] == 2
-    assert result["safety_backup"] is not None
-
-
-def test_restore_rejects_non_sqlite(tmp_path):
-    import pytest
-    from sfm.db_snapshot import restore_db_and_waveforms
-    db_path = tmp_path / "seismo_relay.db"
-    _make_db(db_path, rows=1)
-    bogus = tmp_path / "bogus.db"
-    bogus.write_bytes(b"not a sqlite file at all")
-    wf_root = tmp_path / "waveforms"
-    wf_root.mkdir()
-    original_bytes = db_path.read_bytes()
-    with pytest.raises(ValueError):
-        restore_db_and_waveforms(db_path, wf_root, bogus, None)
-    assert db_path.read_bytes() == original_bytes
-    assert list(tmp_path.glob("*.pre-restore-*")) == []
-
-
-def test_restore_skips_unsafe_zip_members(tmp_path):
-    import zipfile
-    from sfm.db_snapshot import restore_db_and_waveforms
-
-    db_path = tmp_path / "seismo_relay.db"
-    _make_db(db_path, rows=1)
-
-    new_db = tmp_path / "incoming.db"
-    _make_db(new_db, rows=2)
-
-    wf_root = tmp_path / "waveforms"
-    wf_root.mkdir()
-    wf_zip = tmp_path / "wf.zip"
-    with zipfile.ZipFile(wf_zip, "w") as zf:
-        zf.writestr("BE111/ok.h5", b"good")
-        zf.writestr("../evil.txt", b"escape via parent dir")
-        zf.writestr("/tmp/abs_evil.txt", b"escape via absolute path")
-        zf.writestr("baddir/", b"")
-
-    result = restore_db_and_waveforms(db_path, wf_root, new_db, wf_zip)
-
-    assert (wf_root / "BE111" / "ok.h5").exists()
-    assert result["waveforms_unpacked"] == 1
-    assert not (wf_root.parent / "evil.txt").exists()
-    assert not Path("/tmp/abs_evil.txt").exists()

tests/test_db_snapshot_endpoints.py

@@ -1,53 +0,0 @@
-"""
-test_db_snapshot_endpoints.py — HTTP smoke tests for the /db/snapshot,
-/db/waveforms/recent.zip and /db/restore routes. Skipped if the FastAPI
-TestClient stack (httpx) is not installed.
-
-Run:
-    python -m pytest tests/test_db_snapshot_endpoints.py -v
-"""
-from __future__ import annotations
-import os
-import sys
-from pathlib import Path
-
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-import pytest
-
-pytest.importorskip("httpx")   # starlette TestClient needs httpx
-
-
-def _client(tmp_path, monkeypatch):
-    import sfm.server as srv
-    from sfm.database import SeismoDb
-    from sfm.waveform_store import WaveformStore
-    from starlette.testclient import TestClient
-    db = SeismoDb(tmp_path / "seismo_relay.db")
-    store = WaveformStore(tmp_path / "waveforms")
-    monkeypatch.setattr(srv, "_db", db, raising=False)
-    monkeypatch.setattr(srv, "_store", store, raising=False)
-    return TestClient(srv.app)
-
-
-def test_snapshot_endpoint_returns_sqlite(tmp_path, monkeypatch):
-    client = _client(tmp_path, monkeypatch)
-    r = client.get("/db/snapshot")
-    assert r.status_code == 200
-    assert r.content[:16] == b"SQLite format 3\x00"
-
-
-def test_recent_waveforms_endpoint_ok(tmp_path, monkeypatch):
-    client = _client(tmp_path, monkeypatch)
-    r = client.get("/db/waveforms/recent.zip", params={"n": 5})
-    assert r.status_code == 200
-    assert r.headers["content-type"] == "application/zip"
-
-
-def test_restore_dormant_without_env(tmp_path, monkeypatch):
-    monkeypatch.delenv("SFM_DB_RESTORE_ENABLED", raising=False)
-    client = _client(tmp_path, monkeypatch)
-    r = client.post(
-        "/db/restore",
-        files={"db": ("x.db", b"SQLite format 3\x00", "application/octet-stream")},
-    )
-    assert r.status_code == 404

tests/test_event_file_io.py

@@ -626,12 +626,3 @@ if __name__ == "__main__":
                 failed += 1
         print(f"\n{passed} passed, {failed} failed")
         sys.exit(0 if failed == 0 else 1)
-
-
-def test_peaks_from_samples_uses_32000_full_scale():
-    """`_peaks_from_samples` must use the same 32000-count geo full scale as
-    the .h5 writer, or sidecar peaks disagree with the plotted waveform by
-    2.3%.  See test_event_hdf5.test_geo_full_scale_count_is_32000."""
-    from minimateplus.event_file_io import _peaks_from_samples
-    pv = _peaks_from_samples({"Tran": [32000], "Vert": [0], "Long": [0], "MicL": []})
-    assert abs(pv.tran - 10.0) < 1e-4

tests/test_event_hdf5.py

@@ -99,14 +99,8 @@ def test_hdf5_round_trip_preserves_metadata(tmp_path: Path):
 
 
 def test_hdf5_samples_in_physical_units_normal_range(tmp_path: Path):
-    """Vert hits 32767 ADC counts → with Normal range FS=10 in/s that is
+    """Vert hits ADC full-scale (32767) → with Normal range FS=10 in/s,
-    ``10 * 32767/32000`` in/s.
+    the HDF5 sample value should be ≈ 10 * 32767/32768 in/s."""
-
-    Geo full scale is 32000 counts, not 32768 (see
-    test_geo_full_scale_count_is_32000), so 32767 counts sits slightly
-    ABOVE nominal full scale -- the ADC has headroom past 10.000 in/s,
-    which is why Blastware reports peaks like 10.14 in/s.  This test
-    previously asserted the 32768 scale and was wrong by 2.3%."""
     ev = _make_event_with_samples()
     h5 = tmp_path / "n.h5"
     event_hdf5.write_event_hdf5(h5, ev, serial="BE11529", geo_range="normal")
@@ -116,7 +110,7 @@ def test_hdf5_samples_in_physical_units_normal_range(tmp_path: Path):
     assert vert.dtype.name == "float32"
     assert max(abs(v) for v in vert) > 9.99   # full-scale ≈ 10.0
     # The dirac was at n//2 → 32767 ADC counts.
-    expected_peak = 10.0 * 32767 / 32000
+    expected_peak = 10.0 * 32767 / 32768
     assert abs(max(vert) - expected_peak) < 1e-3
 
 
@@ -128,7 +122,7 @@ def test_hdf5_samples_in_physical_units_sensitive_range(tmp_path: Path):
     data = event_hdf5.read_event_hdf5(h5)
 
     vert = data["samples"]["Vert"]
-    expected_peak = 1.250 * 32767 / 32000
+    expected_peak = 1.250 * 32767 / 32768
     assert abs(max(vert) - expected_peak) < 1e-4
 
 
@@ -300,35 +294,3 @@ if __name__ == "__main__":
                 failed += 1
         print(f"\n{passed} passed, {failed} failed")
         sys.exit(0 if failed == 0 else 1)
-
-
-# ── Geophone full-scale count ───────────────────────────────────────────────
-
-def test_geo_full_scale_count_is_32000():
-    """Geo full scale is 32000 ADC counts, not 32768.
-
-    The verified body codec emits geo samples in 16-count units with a
-    documented LSB of exactly 0.005 in/s, and ``decoded_to_adc_counts``
-    multiplies by 16 — so one ADC count is 0.005/16 in/s and Normal range
-    (10.000 in/s) is 10.0 / (0.005/16) = 32000 counts.
-
-    Using 32768 made every geophone reading 2.3% low (1 - 32000/32768).
-    Confirmed 2026-08-25 against 216 channel comparisons with preserved
-    Blastware ASCII exports: 32000 → 216/216 exact within 1 LSB;
-    32768 → 151/216, worst error 0.238 in/s on a 10 in/s event.
-    """
-    from sfm.event_hdf5 import _GEO_INT16_FS
-    assert _GEO_INT16_FS == 32000.0
-
-
-def test_samples_to_float_lsb_is_exactly_5_milli_ips():
-    """One decoder unit (= 16 ADC counts) must be exactly 0.005 in/s."""
-    from sfm.event_hdf5 import _samples_to_float
-    out = _samples_to_float([16], 10.0)
-    assert abs(float(out[0]) - 0.005) < 1e-9
-
-
-def test_samples_to_float_full_scale_count_maps_to_full_scale():
-    from sfm.event_hdf5 import _samples_to_float
-    assert abs(float(_samples_to_float([32000], 10.0)[0]) - 10.0) < 1e-4
-    assert abs(float(_samples_to_float([32000], 1.25)[0]) - 1.25) < 1e-5

tests/test_event_hdf5_sensor_check.py

@@ -1,71 +0,0 @@
-"""The event .h5 carries the sensor self-check traces (schema v2).
-
-The sensor check is decoded by the per-series decoder and attached to the
-standardized Event, so the .h5 writer persists it device-agnostically and SFM
-reads it back without knowing which instrument produced it.  Old v1 files (no
-sensor_check group) must still read cleanly.
-"""
-import tempfile
-from pathlib import Path
-
-import numpy as np
-
-from minimateplus.models import Event
-from minimateplus.event_file_io import read_blastware_file
-from sfm import event_hdf5
-
-S3_FIX = Path(__file__).parent / "fixtures" / "fft-oracle-2026-09-14" / "N844LQHB.ZT0W"
-
-
-def _write(ev, **kw):
-    d = Path(tempfile.mkdtemp())
-    p = d / "e.h5"
-    event_hdf5.write_event_hdf5(p, ev, serial="BE12844", **kw)
-    return p
-
-
-def test_sensor_check_roundtrips_through_hdf5():
-    ev = Event(index=0)
-    ev.raw_samples = {"Tran": [1, 2, -3], "Vert": [0, 1], "Long": [2], "MicL": [5, -5]}
-    ev.sample_rate = 1024
-    sc = {"Tran": [0, -990, -500, -100], "Vert": [0, -980, -480],
-          "Long": [0, -986, -470], "MicL": [0, -1800, 1800, -1800]}
-    ev.sensor_check = sc
-
-    r = event_hdf5.read_event_hdf5(_write(ev))
-    assert r["schema_version"] == 2
-    assert set(r["sensor_check"]) == {"Tran", "Vert", "Long", "MicL"}
-    for ch, vals in sc.items():
-        assert r["sensor_check"][ch].tolist() == vals
-
-
-def test_plot_json_carries_sensor_check():
-    ev = Event(index=0)
-    ev.raw_samples = {"Tran": [1, 2, 3]}
-    ev.sample_rate = 1024
-    ev.sensor_check = {"Tran": [0, -990, -500], "Vert": [0, -980],
-                       "Long": [0, -986]}   # 3-channel: no MicL
-    pj = event_hdf5.plot_json_from_hdf5(_write(ev))
-    assert pj["sensor_check"] is not None
-    assert "MicL" not in pj["sensor_check"]
-    assert pj["sensor_check"]["Tran"] == [0, -990, -500]
-
-
-def test_event_without_sensor_check_still_reads_as_v2():
-    ev = Event(index=0)
-    ev.raw_samples = {"Tran": [1, 2, 3]}
-    ev.sample_rate = 1024
-    r = event_hdf5.read_event_hdf5(_write(ev))
-    assert r["schema_version"] == 2
-    assert r["sensor_check"] is None
-    assert event_hdf5.plot_json_from_hdf5(_write(ev))["sensor_check"] is None
-
-
-def test_series3_decode_populates_event_sensor_check():
-    # The real series-3 decoder attaches the traces to the Event, so the
-    # ingest/backfill .h5 write picks them up with no extra plumbing.
-    ev = read_blastware_file(S3_FIX)
-    assert ev.sensor_check is not None
-    assert set(ev.sensor_check) == {"Tran", "Vert", "Long", "MicL"}
-    tran = np.asarray(ev.sensor_check["Tran"], dtype=float)
-    assert tran.min() < -800   # the geophone ring-down deflection

tests/test_event_timestamp.py

@@ -1,54 +0,0 @@
-"""Event timestamp decode — waveform trigger/stop vs histogram window start.
-
-The Blastware footer holds two timestamps: ts1 = footer[2:10], ts2 = footer[10:18].
-Their meaning depends on record type:
-
-  * Waveform: ts1 is the monitoring-SESSION start (e.g. 06:00 for a unit that
-    arms at 06:00 daily — shared across every event that day), and ts2 is THIS
-    event's recording STOP.  read_blastware_file used to stamp events with ts1 →
-    every waveform showed the session start (~4.5 h off).  Binary-only, the best
-    estimate is ts2 (the stop); the exact trigger BW displays (= ts2 - record
-    duration) comes from the paired report's event_datetime, since the binary
-    STRT record-time byte is a misparsed record-type marker.
-  * Histogram: ts1/ts2 are the ~24 h window [start, stop]; the event time is the
-    window start = ts1 (unchanged).
-"""
-import datetime
-from pathlib import Path
-
-from minimateplus.event_file_io import read_blastware_file, apply_report_to_event
-from minimateplus.bw_ascii_report import BwAsciiReport
-from minimateplus.models import Event
-
-FIX = Path(__file__).parent / "fixtures"
-WAVEFORM = FIX / "fft-oracle-2026-09-14" / "N844LQHB.ZT0W"   # footer ts2 = 2026-08-25 10:33:32
-HISTOGRAM = FIX / "ts-fix" / "K441LKZU.C30H"                 # window start 2026-05-10 19:04:50
-
-
-def _tuple(ts):
-    return (ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second)
-
-
-def test_waveform_timestamp_is_exact_trigger_from_binary():
-    ev = read_blastware_file(WAVEFORM)
-    # The EXACT Blastware trigger, from the binary alone: ts2 (stop 10:33:32)
-    # minus the config record time (3.0 s) = 10:33:29 — NOT the 06:00:13
-    # monitoring-session start the old decode used.
-    assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 33, 29), _tuple(ev.timestamp)
-
-
-def test_histogram_timestamp_is_window_start_unchanged():
-    ev = read_blastware_file(HISTOGRAM)
-    # Histogram event time = the window start (ts1); must NOT get the waveform
-    # ts2 treatment (that would land ~24 h off).
-    assert _tuple(ev.timestamp) == (2026, 5, 10, 19, 4, 50), _tuple(ev.timestamp)
-
-
-def test_report_event_datetime_is_authoritative_over_binary():
-    # The binary already yields the exact trigger, but a paired report stays
-    # authoritative (e.g. if the unit clock had drifted) — applying it wins.
-    ev = read_blastware_file(WAVEFORM)
-    assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 33, 29)   # exact, from binary
-    apply_report_to_event(ev, BwAsciiReport(
-        event_datetime=datetime.datetime(2026, 8, 25, 10, 35, 0)))
-    assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 35, 0)    # report wins