Compare commits
85
Commits
+798
-1
@@ -4,7 +4,804 @@ All notable changes to seismo-relay are documented here.
|
||||
|
||||
---
|
||||
|
||||
## [Unreleased]
|
||||
## Unreleased
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Waveform event times were the monitoring-session start, not the trigger
|
||||
(~hours off).** `read_blastware_file` stamped events with footer `ts1`, which
|
||||
for a waveform is the session start a unit shares across every event that day
|
||||
(a unit arming at 06:00 stamped 06:00 on all of them — the modal and PDF both
|
||||
showed it, since it's the stored value). The event time is footer `ts2` (the
|
||||
recording stop), and Blastware's trigger = `ts2 - record time`. The record
|
||||
time is a big-endian float32 in the recording-setup config block (30 bytes
|
||||
before the `Standard Recording Setup` marker), so the **exact trigger is now
|
||||
recovered from the binary alone** — all 7 BE12844 oracle events decode to
|
||||
their exact Blastware time (e.g. N844LQHB 10:33:29), no paired `.TXT` needed.
|
||||
Histograms keep `ts1` (the ~24 h window start). A paired report's
|
||||
`event_datetime` stays authoritative (unit-clock drift).
|
||||
⚠ **Needs a re-decode backfill** to correct existing stored events' timestamps.
|
||||
|
||||
### Added
|
||||
|
||||
- **Diagnostics tab in the SFM standalone webapp.** Surfaces the device
|
||||
endpoints that previously existed only as `curl`: `events/storage_range` and
|
||||
`events/index` alongside `monitor/status`, then stop monitoring, disable ACH
|
||||
(`rescue?erase=false`, so stored events survive), and erase. The wedged-unit
|
||||
ladder — slow drip and blind stop — sits under its own heading pointing at
|
||||
`docs/runbooks/wedged_unit_recovery.md`, with the reminder that `slow_drip`'s
|
||||
success signal is `bytes_received > 0` and not a clean duration. Erase is
|
||||
guarded by typing the unit's serial: auth answers *who*, not *did you mean
|
||||
it*, and Swagger's try-it-out button on `/device/events/erase` is live on
|
||||
`:8200/docs`.
|
||||
|
||||
- **`docs/sfm_tool_status.md`** — an honest per-capability maturity assessment:
|
||||
what is production-grade (the codec library, the data side), what is
|
||||
emergency-grade (the device side), what is a research artifact, the
|
||||
known-issues table, and the gap to a real tool. Also records the **5A
|
||||
page-boundary bug** as known: `parse_strt_end_offset()` discards the key's
|
||||
page byte, so once a unit has recorded more than 64 KB since its last erase,
|
||||
an event spanning the boundary reads an `end_offset` *behind* its own start —
|
||||
the chunk loop fetches nothing and TERM packs a negative `offset_word`, which
|
||||
500s. Reproduced on BE12599. Production is unaffected: it ingests complete
|
||||
files via the watcher path and never runs this walk.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Connecting to a unit no longer walks its event chain.** `/device/events`
|
||||
reads every event header over the cellular link; on a unit with a large or
|
||||
wrapped chain that takes minutes or fails outright, and it fired
|
||||
automatically on every connect. Connect now uses only ~2 s probes —
|
||||
`/device/info` (which already carried the compliance config the walk was
|
||||
re-reading) plus `events/storage_range` — and the Device tab gains an Event
|
||||
Chain card. The walk moved behind a **Load events** button in the Events
|
||||
toolbar. Knowing whether a unit's ACH is on no longer requires reading every
|
||||
event it has stored.
|
||||
|
||||
### Migration
|
||||
|
||||
**None.** Frontend and documentation only — no codec, waveform-store or DB
|
||||
change, no schema change, and no `TOOL_VERSION` bump. The webapp is served
|
||||
from the image, so the change appears after the next `sfm` rebuild.
|
||||
|
||||
---
|
||||
|
||||
## v0.31.0 — 2026-09-18
|
||||
|
||||
**Report parity, and a second way to rescue a runaway unit.** Two threads.
|
||||
|
||||
The first closes out Blastware Event/FFT-Report parity: the FFT, the USBM
|
||||
RI8507 compliance chart and the sensor self-check now render on the event
|
||||
report, reverse-engineered against BE12844 (MiniMate Plus) and UM (Thor)
|
||||
events. The sensor check is decoded for **both** series and standardized into
|
||||
the `.h5` (schema **v2**, a new `/sensor_check` group), so SFM serves it
|
||||
device-agnostically rather than decoding at report time. The Inspector — an
|
||||
annotated hex reader for series-3 binaries — is what made the trailing-block
|
||||
structure findable, and it earned its keep by *ruling out* a stored FFT block
|
||||
and proving Blastware computes it from the samples.
|
||||
|
||||
The second came out of a field emergency. BE12599's connector fault drove its
|
||||
Tran channel to its trigger level, so the unit recorded back-to-back and dialed
|
||||
the office ACH server every ~75 s, unreachable the whole time.
|
||||
`bridges/ach_server.py` gained `--stop-monitoring` / `--disable-ach` /
|
||||
`--rescue`, which **invert** the recovery: instead of racing a Stop into the
|
||||
gaps between dial-outs, point the modem's Destination at our own ACH server and
|
||||
answer the call. Proven in production the same night — the stop landed on the
|
||||
first call-in and held. See `docs/runbooks/wedged_unit_recovery.md`.
|
||||
|
||||
⚠ **This release owes prod a backfill** — see Migration below.
|
||||
|
||||
### Added
|
||||
- **Rescue-on-connect for `bridges/ach_server.py`** — `--stop-monitoring`
|
||||
(SUB 0x97), `--disable-ach` (SUB 0x2C read → 0x7E write → 0x7F confirm) and
|
||||
`--rescue` (both). They fire immediately after the startup handshake and
|
||||
**before** the event walk, so a unit that is recording back-to-back on a
|
||||
stuck-triggered geophone is quieted as early in the session as possible.
|
||||
Each action is independently guarded — a failure does not abort the download
|
||||
— and the outcome is written to `rescue.json` in the session directory.
|
||||
|
||||
This inverts the `docs/runbooks/wedged_unit_recovery.md` approach. That
|
||||
runbook reaches the unit *inbound* and clears the modem's Destination Address
|
||||
to stop it dialing. When the device is instead wedged mid-modem-init — ALEOS
|
||||
logs `tcpmode trying to send to invalid socket` and re-runs `Initialize Auto
|
||||
answer` every ~75 s, orphaning any held inbound session — inbound cannot win.
|
||||
Pointing the modem's Destination at an `ach_server` and letting the unit call
|
||||
*us* gives a device-initiated session the modem bridges properly.
|
||||
|
||||
⚠ Prefer `--stop-monitoring` alone on first contact. `--disable-ach` stops
|
||||
the unit calling, which is the only channel to a unit in this state; stopping
|
||||
the recording ends the call-home loop on its own when ACH is
|
||||
"after event recorded".
|
||||
|
||||
- **Blastware-compatible channel FFT (`waveform_fft`).** Reproduces Blastware's
|
||||
FFT Report: DC-removed, no window, zero-padded to 4096 (0.25 Hz bins at
|
||||
1024 sps), single-sided `2/N` amplitude. Matches Blastware's dominant
|
||||
frequency to the exact bin and the amplitude to report precision across all
|
||||
28 channels of the 7-event BE12844 oracle set. `channel_spectrum()` /
|
||||
`dominant_frequency()`; tests in `tests/test_waveform_fft.py`.
|
||||
|
||||
- **USBM RI8507 / OSMRE compliance chart on the event-report PDF
|
||||
(`sfm/compliance.py`).** The velocity-vs-frequency blasting-compliance
|
||||
scatter Blastware draws in the upper-right of its Event Report: each channel's
|
||||
significant cycles as `(frequency, peak velocity)` points (zero-crossing
|
||||
method, so each channel's cloud tops out at its PPV) plotted against the
|
||||
RI8507 Drywall (0.75 in/s) and plaster (0.50 in/s) limit curves, drawn
|
||||
continuous (constant-displacement bounds meeting the plateaus — no vertical
|
||||
steps). Sized and positioned to match a Blastware report, measured off the
|
||||
reference PDF. A technical breakdown of the curve is in
|
||||
`docs/ri8507_compliance_curve.md`.
|
||||
|
||||
- **Sensor self-check waveforms decoded and drawn — both series.** The little
|
||||
"Sensor Check" traces (geophone ring-downs — the transducer's damped impulse
|
||||
response — plus a MicL pulse train, the mic's known-signal gain check) are the
|
||||
unit's proof its sensors were healthy when it recorded the event.
|
||||
- **Series-3** (`minimateplus.sensor_check`): four records (`0x3c`–`0x3f`) in
|
||||
the binary's trailing block, same delta-block codec as the main waveform.
|
||||
Verified against all 7 BE12844 reports (mic zero-crossing = 20.1 Hz exact;
|
||||
geophone ring-downs ~7.5 Hz, overswing ~3.5).
|
||||
- **Series-4** (`micromate.sensor_check`): the same self-test in the Thor IDFW
|
||||
fixed header — four `01 0e 3c/3d/3e/3f` records (same channel ids) storing
|
||||
raw int16 traces; three-channel (mic-disabled) units carry only the three
|
||||
geophones. Validated by shape + cross-event consistency.
|
||||
- **Standardized into the `.h5`** (`/sensor_check`, schema v2): each series'
|
||||
decoder attaches the traces to the event at decode, the writer persists
|
||||
them, and `gather_report_data` reads them back — so SFM renders the strip
|
||||
(flush against the waveform panel) plus the **Sensor Check → Frequency /
|
||||
Overswing Ratio** sub-rows without knowing the source instrument.
|
||||
- Tests: `tests/test_sensor_check.py`, `tests/test_sensor_check_idf.py`,
|
||||
`tests/test_event_hdf5_sensor_check.py`.
|
||||
|
||||
- **Inspector tab in `seismo_lab.py` — annotated hex reader for series-3
|
||||
binaries (`minimateplus/binary_annotate.py`).** Tiles a raw Blastware file
|
||||
into labeled spans (header / STRT / body record-chain / trailing metadata +
|
||||
calibration + sensor-check records / footer) so a binary can be combed by eye.
|
||||
|
||||
### Fixed
|
||||
- **Event-report waveform panel — stacked-lane y-tick collision.** The lanes
|
||||
touch, so each lane's bottom `-1.0` overprinted the next lane's top `1.0` at
|
||||
the shared boundary. Prune the extreme ticks so each lane shows clean interior
|
||||
ticks only.
|
||||
- **Event-report header — serial+firmware line ran off the page.** The long
|
||||
`BE##### V ##.##-#.## MiniMate Plus` string overflowed the right margin;
|
||||
tighter right-column indent + BW's slightly smaller header size so it fits.
|
||||
|
||||
---
|
||||
|
||||
### Migration
|
||||
|
||||
⚠ **The sensor-check needs a backfill.** Existing `.h5` files are schema v1
|
||||
and carry no `/sensor_check` group, so their reports show no sensor-check strip
|
||||
until regenerated. `TOOL_VERSION` is bumped to **0.31.0**, so the standard
|
||||
backfill regenerates every event and picks up the traces with **no `--force`**:
|
||||
`scripts/backfill_thor_events.py` for series-4 (it already owed a v0.30.0 Thor
|
||||
backfill — this rides along) and the series-3 sidecar/shape backfill for
|
||||
MiniMate events. Purely additive — no decoded value changes, and v1 `.h5`
|
||||
files read fine until then (empty strip). DB backup first, as always.
|
||||
|
||||
⚠ Budget **~2 h on the NAS** — ~1.5 files/sec there versus ~85/sec on the dev
|
||||
box (gzip-4 in `sfm/event_hdf5.py` against a Synology CPU).
|
||||
|
||||
Everything else in this release owes nothing: the FFT, the USBM compliance
|
||||
chart and the `ach_server` rescue flags are additive and read data already on
|
||||
disk — no schema change, no DB migration.
|
||||
|
||||
---
|
||||
|
||||
## v0.30.0 — 2026-09-12
|
||||
|
||||
**The series-4 correctness release** — the Thor / Micromate counterpart to
|
||||
v0.26.0's series-3 work. The decoder is now verified per-sample against
|
||||
Thor's own CSV exports: **459 waveform files, 3,807,158 / 3,807,165 samples
|
||||
exact** across three independent ground-truth corpora, and production IDFW is
|
||||
**575/575** with zero truncations and zero decode failures. Series-3
|
||||
re-verified **unchanged at 14,338/14,338** after every shared-codec change.
|
||||
|
||||
⚠ **This release owes the prod store a Thor backfill.** Every stored
|
||||
series-4 geophone value is **3.3% low**, and histogram peaks from monitoring
|
||||
runs longer than ~4 hours can be far worse (the interval cap discarded the
|
||||
tail, frequently the part holding the peak). Run
|
||||
`scripts/backfill_thor_events.py` — `TOOL_VERSION` is bumped to `0.30.0`, so
|
||||
regeneration is gated correctly and **no `--force` is needed**. DB backup
|
||||
first. Series-3 events are untouched by this release and do not need
|
||||
re-running.
|
||||
|
||||
⚠ **Terra-View displays these values.** Series-4 geophone readings will rise
|
||||
~3.3% after the backfill, and some histogram PPVs will rise a great deal more.
|
||||
That is a correction, not a regression.
|
||||
|
||||
|
||||
### Fixed — event-report PDF used a per-trace geo Y scale
|
||||
|
||||
The waveform plot scaled each geo lane to its own peak, so a small channel
|
||||
filled its lane and looked as large as a big one, and the `Geo: X in/s/div`
|
||||
footer reflected only whichever channel was measured first — wrong for the
|
||||
other two. All three geo lanes now share one symmetric scale (max |sample|
|
||||
across them, padded, 0.05 in/s floor), matching the event modal and BW's
|
||||
single amp/div; the footer reflects that shared scale. Mic keeps its own psi
|
||||
scale. Large events are unchanged.
|
||||
|
||||
### Fixed — series-4 (Thor / Micromate) decoder is now per-sample exact
|
||||
|
||||
Verified against **Thor's own CSV exports**, which carry a per-sample
|
||||
four-column block beside every binary (`CSV/<name>.IDFW.csv`) — 1,012 paired
|
||||
files that had been sitting in the corpus unused. Previous notes asserted
|
||||
"Thor has no ASCII ground truth", which is why the decoder stayed pinned to a
|
||||
superseded walker with an unverifiable scale factor.
|
||||
|
||||
| metric | before | after |
|
||||
|---|---|---|
|
||||
| IDFW per-sample exact | 39.1% | **100.000%** (1,057,536/1,057,536) |
|
||||
| IDFW files fully exact | 0/153 | **153/153** |
|
||||
| IDFW PPV median error | −3.32% | **−0.002%** |
|
||||
| IDFH within 2% of Thor PPV | 51.1% | **100.0%** (858/858) |
|
||||
| prod IDFW PPV median error (8 units) | −3.3% | **−0.001%** |
|
||||
| decode cost | — | 6 ms/file |
|
||||
|
||||
Four independent root causes:
|
||||
|
||||
- **Geo LSB was `0.0003`, should be `0.000310308`** — the old value was Thor's
|
||||
4-decimal *display rounding* of the LSB mistaken for the LSB, so every
|
||||
series-4 geophone sample read **3.3% low**. Pinned to ±6e-11 by
|
||||
intersecting 991,415 rounding constraints; corroborated by the ±full-scale
|
||||
seed (`±32226`) in unwritten IDFH slots. Applies to IDFH too, which had a
|
||||
separate (also wrong) `10.0/32768`.
|
||||
- **IDFH histograms were capped at 250 intervals** — the segment validator
|
||||
required the interval counter's high byte to be zero, but the counter is a
|
||||
uint16 cumulative index, so every segment past interval 255 was rejected.
|
||||
Any run over ~4 hours lost its tail, often the part holding the peak.
|
||||
540/858 corpus files affected.
|
||||
- **Record mode `00 00` (raw int16, 10-byte header) was unhandled** — the
|
||||
record fell through the dispatch, silently dropping each channel's first
|
||||
512 samples. This produced the long-standing "loud events truncate"
|
||||
symptom. `MODE_ABSOLUTE` is now also accepted as a segment-0 preamble.
|
||||
- **Body-offset search matched `00 02 00` inside record headers** — picking a
|
||||
candidate part-way down the chain, which decodes a rotation-shifted body
|
||||
that drops each channel's segment 0. The search now anchors on record
|
||||
headers and takes the chain head.
|
||||
|
||||
Also fixes the separately-tracked "UM-series decodes ~1000× low" bug
|
||||
(`UM11402_20260406130113.IDFW` now matches its device report exactly).
|
||||
|
||||
Series-3 re-verified **unchanged at 14,338/14,338 exact** after the shared
|
||||
`waveform_codec` change.
|
||||
|
||||
⚠ **This is a codec change: the Thor store owes a regeneration.** Run
|
||||
`scripts/backfill_thor_events.py` (bump `TOOL_VERSION` first, or pass
|
||||
`--force`), DB backup first. All stored series-4 `.h5`/sidecar peaks are
|
||||
currently ~3.3% low, and histogram peaks for runs over ~4 hours may be
|
||||
badly low.
|
||||
|
||||
⚠ **Thor's histogram PPV has a 0.0050 in/s display floor** — 41.4% of prod
|
||||
IDFH sidecars report a component PPV larger than their own vector sum. On
|
||||
quiet files the decoder is now *more* accurate than that reference.
|
||||
|
||||
New: `scratch/verify_thor_against_csv.py`, `tests/test_idf_binary_codec.py`
|
||||
(10 tests, fixtures under `tests/fixtures/thor-idf/`).
|
||||
|
||||
### Fixed — mic-disabled (3-channel) units
|
||||
|
||||
Verified on a second corpus (`9-10-26-csv-req`: UM11402, UM12947, UM20147) —
|
||||
**139/139 waveforms per-sample exact (1,273,380 samples), 877/877 histograms
|
||||
within 2%** (was 66.9% and 56.6%).
|
||||
|
||||
- **Waveform body head sat below the scan floor.** A 3-channel unit's shorter
|
||||
header puts the record chain head at `0x0dba`, under the old
|
||||
`_BODY_SCAN_FLOOR` of `0x0E00`. The scan couldn't see it and fell through
|
||||
to the Vert segment-0 record, decoding a body shifted one position around
|
||||
the channel rotation — Vert came up exactly 512 samples short. Floor
|
||||
lowered to `0x0C00`; body-offset scoring now accepts 3 channels as "equal"
|
||||
instead of demanding 4.
|
||||
- **Histogram interval record is 56 bytes, not 72.** It is
|
||||
`16 × n_channels + 8`, so mic-disabled units pack 56. Assuming 72 read 7
|
||||
intervals out of every 10-interval segment then walked off alignment into
|
||||
garbage decoding as ~10 in/s peaks (errors up to +191,000%). The interval
|
||||
count now comes from the segment's cumulative counter and the stride is
|
||||
derived from it; also recovers 4 files that decoded no intervals at all.
|
||||
|
||||
Combined across both corpora: **292/292 waveform files, 2,330,916/2,330,916
|
||||
samples exact.** Production IDFW truncations 41 → 22.
|
||||
|
||||
### Fixed — `40 NN` int16 blocks with NN > 8
|
||||
|
||||
`data_block_len()` rejected any `40 NN` block with `NN > 0x08`. The cap had
|
||||
no evidence behind it: every corpus available when it was written used only
|
||||
NN ∈ {1,2,3,4,8}, so it was never exercised. Loud UM12947 events use NN of
|
||||
12, 16, 20 … up to 196, and because the block walker stops at the first
|
||||
unrecognised tag rather than raising, rejecting them surfaced as **silently
|
||||
short channels** (e.g. Tran 1812 / Vert 2132 / Long 2324 on a file whose
|
||||
export has 2324 for all three). The bound is the buffer, not a constant.
|
||||
|
||||
Verified against Thor exports for UM12947 (2025-07-14 … 09-25, 167
|
||||
waveforms): length mismatches **22 → 0**, **1,476,242/1,476,249** samples
|
||||
exact. These are not truncated recordings — the exports carry full sample
|
||||
counts.
|
||||
|
||||
`tests/test_waveform_codec.py` asserted the cap as intended behaviour; that
|
||||
assertion was wrong and has been replaced with one pinning the opposite,
|
||||
carrying the evidence.
|
||||
|
||||
### Result across all three ground-truth corpora
|
||||
|
||||
**459 waveform files, 3,807,158 / 3,807,165 samples exact.** Production
|
||||
IDFW: **575/575**, zero truncations, zero decode failures, median PPV error
|
||||
−0.0007% across 8 units. Series-3 re-verified **unchanged at 14,338/14,338**
|
||||
after every shared-codec change.
|
||||
|
||||
The 7 residual samples each differ by one 4th-decimal tick and are **Thor's
|
||||
own rounding**: intersecting the per-sample rounding constraints over that
|
||||
corpus is infeasible (the binding pair contradict by 2.3e-11, 7e-5 relative),
|
||||
so no single linear LSB reproduces every printed value. `_GEO_LSB_IPS` is
|
||||
already pinned to ~1e-11 — do not retune it to chase these.
|
||||
|
||||
---
|
||||
|
||||
## v0.29.0 — 2026-09-04
|
||||
|
||||
First release to reach prod since **v0.27.0**, so it ships **both** the
|
||||
`false_trigger_reason` column below *and* the v0.28.0 offset (DC-baseline)
|
||||
detector: v0.28.0 was version-bumped in-tree (`TOOL_VERSION`, CHANGELOG) but
|
||||
never tagged or deployed, so 0.29.0 is the first build to carry either to prod.
|
||||
Pairs with Terra-View ≥ 0.24.0. The `false_trigger_reason` column auto-migrates
|
||||
on startup; the offset detector still needs the shape backfill on the prod store
|
||||
(`scripts/backfill_event_shape.py`) to populate `shape_offset*` on existing rows.
|
||||
|
||||
### Added
|
||||
- **`events.false_trigger_reason` — optional FT cause.** A nullable `TEXT`
|
||||
column recording *why* an event is a false trigger (e.g. `"offset"`), as a
|
||||
subtype of the FT flag: setting a reason via the sidecar review PATCH implies
|
||||
`false_trigger=1`, and the reason is cleared whenever FT ends up 0
|
||||
(confirm-real, clear-FT, `set_false_trigger(false)`). `propagate_review_to_twins`
|
||||
carries the reason to the histogram/waveform twin alongside the flag.
|
||||
Auto-migrated (`_SCHEMA` + `_migrate` ADD COLUMN — not the Migration-1
|
||||
rebuild); exposed via `/db/events`. Terra-View surfaces it as a manual
|
||||
"Flag as offset" action + an `FT · offset` badge.
|
||||
|
||||
### Fixed
|
||||
- **BlastMate serials — the family prefix is read from the file, not guessed.**
|
||||
The Blastware filename encodes only the serial *number* (`L895…` → 10895);
|
||||
the two-letter prefix is not in it. `waveform_store` synthesised `"BE"`, so
|
||||
an imported **BlastMate** (serials `BA…`) was filed under a MiniMate Plus
|
||||
serial that does not exist — silently, and Terra-View read it straight
|
||||
through. `save_imported_bw` now resolves serial as hint → file body →
|
||||
filename guess, via a new `_serial_from_bw_bytes` that accepts a candidate
|
||||
only when its numeric part matches the filename. `client._decode_0a_partial_header`
|
||||
likewise matched a literal `b"BE"` in monitor-log partial records; on a
|
||||
BlastMate that returned −1 and skipped the whole block, losing the **geo
|
||||
threshold** along with the serial. It now matches any two-letter prefix and
|
||||
requires the NUL terminator — stricter than the search it replaces.
|
||||
|
||||
BlastMate is the MiniMate Plus's larger Series III sibling and its files are
|
||||
byte-compatible: all 1,493 in the DL2 archive decode through the existing
|
||||
codec at 100%, same four channels. **The serial string was the only thing
|
||||
blocking BlastMate support in SFM.** Four archive units were affected —
|
||||
BA9229, BA10060, BA10895, BA15957.
|
||||
|
||||
**No backfill and no `TOOL_VERSION` bump**: this changes which serial an
|
||||
*import* is filed under, not any decoded value, so existing sidecars and
|
||||
`.h5` files are untouched. **No migration either** — prod holds no BlastMate
|
||||
events (the archive's BA units last recorded 2018-10 through 2023-11; the
|
||||
prod backfill reaches back only to ~May 2025).
|
||||
|
||||
---
|
||||
|
||||
## v0.28.0 — 2026-09-02
|
||||
|
||||
**Offset (DC-baseline) false-trigger detector.** Productionizes the validated
|
||||
pre-trigger detector: a geophone event whose baseline sits off zero and stays
|
||||
flat across the record (sensor bumped / settled / drifted) is now flagged and
|
||||
surfaced in Terra-View as an `offset` false-trigger reason — catching offsets the
|
||||
crest/near-peak spike rule misses (an offset is low-crest and flat).
|
||||
|
||||
### Added
|
||||
- `shape_metrics.offset_from_samples` / `offset_from_h5`: per geophone channel,
|
||||
`|median(pre-trigger)| ≥ 0.025 in/s` AND `pre/mid/end spread ≤ 0.02` → offset;
|
||||
the consistency test rejects transients (a real event moves one third). Reads
|
||||
the `.h5` samples + the `pretrig_samples` attr, range-aware via the in/s float
|
||||
samples. Constants `OFFSET_FLOOR` / `OFFSET_MAX_SPREAD` are tunable.
|
||||
- `events.shape_offset` / `shape_offset_axis` / `shape_offset_pre` /
|
||||
`shape_offset_spread` columns (auto-migrated: `_SCHEMA` + the `_migrate`
|
||||
ADD COLUMN loop), computed at all three ingest paths and by
|
||||
`backfill_event_shape.py`, exposed via `/db/events`.
|
||||
|
||||
Requires the shape/offset backfill on the prod store to populate existing events:
|
||||
`python scripts/backfill_event_shape.py --db-path … --store-root …`.
|
||||
|
||||
---
|
||||
|
||||
## v0.27.0 — 2026-08-28
|
||||
|
||||
**Per-sample decoder verification at scale, plus the offset investigation.**
|
||||
The series-3 codec is now verified sample-by-sample against **14,338** preserved
|
||||
Blastware ASCII exports — 1,249 waveform and 13,089 histogram, spanning 45 units
|
||||
and files back to 2018. That is 11x the ground truth the production store
|
||||
carried, and it found one real codec bug (below).
|
||||
|
||||
### Fixed
|
||||
- **Sub-minute histograms with a partial final block decoded to nothing**
|
||||
(`histogram_codec.detect_multi_interval_stride`). The stride search confirmed
|
||||
itself on a third block header whenever the body was long enough to hold one —
|
||||
but a body can exceed two strides and still contain only two real blocks, because
|
||||
a *partial* final block leaves trailing padding. BE18193 `T193L0XM.CI0H` (51
|
||||
intervals at 2 s = one full 30-interval block plus a 21-interval remainder, in a
|
||||
2787-byte body) therefore had its correct stride of 612 discarded and produced an
|
||||
empty decode. A missing third header now means end-of-stream rather than
|
||||
disqualification; the block-counter check, which is what actually prevents the
|
||||
false positives that once mis-dispatched 9,082 files, is unchanged.
|
||||
|
||||
Found by decoding the full DL2 archive against its preserved Blastware ASCII
|
||||
exports. Across **63,535 unique** histogram binaries the fix recovers **4 files** —
|
||||
`K440HJCN.3C0H` and `K557IF1U.8K0H` (stride 252), `T191HVNP.0S0H` (92) and
|
||||
`T193L0XM.CI0H` (612) — with **zero** files regressed. Verification over all
|
||||
14,340 archive pairs goes 14,337 → 14,338 exact, the only remainder being two
|
||||
series-4 IDF files that belong to a different codec.
|
||||
|
||||
(The DL2 export keeps a byte-identical `Sent/` mirror of its root, so a naive
|
||||
walk double-counts every binary — 127,035 paths are 63,535 distinct files. The
|
||||
ASCII exports are *not* mirrored, so the 14,340 pair count is already distinct.)
|
||||
|
||||
**No prod backfill is required for this.** Verified after the fact: all four
|
||||
recovered files are archive-only — none exists in the production store or the
|
||||
events DB — and re-running stride detection over the production store's
|
||||
**10,215** histogram binaries shows **0 files whose decode changes**. The fix
|
||||
matters for future ingests of sub-minute histograms with a partial final block,
|
||||
not for anything already stored.
|
||||
|
||||
(`TOOL_VERSION` moves with the release, so whenever a backfill *is* next run for
|
||||
some other reason it will regenerate the whole store rather than skipping. That
|
||||
is harmless — the output is byte-identical for every currently-stored file — but
|
||||
it means the run takes its full ~2 hours on the NAS.)
|
||||
|
||||
- **Histogram/waveform twin matching is now interval-based** (`find_twins`). A real
|
||||
trigger is recorded twice — as a triggered waveform (stamped at the trigger instant)
|
||||
and inside the scheduled histogram whose interval contains it (stamped at the 7am/7pm
|
||||
interval start) — so the two twins can be **hours apart**. The old ±5-minute window
|
||||
silently missed them, which broke review propagation (flagging one twin didn't flag its
|
||||
twin). Twins are now matched by same serial + identical `peak_vector_sum` + opposite
|
||||
record type + the waveform falling within the histogram's interval (bounded by the next
|
||||
same-serial histogram). `window_seconds` is retained but ignored. Fixes terra-view #102
|
||||
sub-task 2.
|
||||
|
||||
- **`/health` reported a hard-coded `0.1.0`** instead of the real service version.
|
||||
`sfm/server.py` now derives its version from `minimateplus.event_file_io.TOOL_VERSION`,
|
||||
making that constant the single source of truth for the service version and the
|
||||
sidecar stamp alike — one place to bump at release.
|
||||
|
||||
- **`CLAUDE.md` had 793 NUL bytes appended** after its last line, which made `grep`
|
||||
treat the file as binary and silently skip it. Present since at least v0.21.0.
|
||||
Stripped.
|
||||
|
||||
### Added
|
||||
- **`docs/offset_investigation.md`** — a dated journal of the "offset" hardware
|
||||
fault: base rate, detector design, per-unit case files, ruled-out hypotheses
|
||||
(each kept with the evidence that killed it), and Instantel's own autozero
|
||||
procedure with its 2027–2069 acceptance window.
|
||||
- **`scratch/verify_against_ascii.py`** — decodes a corpus of BW binaries and
|
||||
diffs every sample against the paired `_ASCII.TXT`. Includes a saturation
|
||||
carve-out: BW clamps clipped events to the range maximum and writes `OORANGE`,
|
||||
while the decoder faithfully reports counts past nominal full scale.
|
||||
- **`scratch/offset_scan3.py`** — offset detector. Measures the resting floor in
|
||||
the *pre-trigger* window (definitionally quiet) and requires it to hold across
|
||||
pre / middle / end. Result: **5 of 45 units (11%)**, stable across a 2x
|
||||
threshold range. Supersedes `offset_scan.py` and `offset_scan2.py`, both kept
|
||||
as the reasoning trail.
|
||||
|
||||
### Verified
|
||||
- **19,244 healthy channel-events sit at a pre-trigger floor of exactly 0.000
|
||||
(62.7%), 94.5% within ±1 quantisation unit, median +0.0000.** No systematic
|
||||
zero-point bias in the decoder — an independent confirmation of the
|
||||
32000-count geo full scale, arrived at from a different direction than the
|
||||
ASCII sample comparisons.
|
||||
|
||||
---
|
||||
|
||||
## v0.26.0 — 2026-08-27
|
||||
|
||||
**Series-3 decode correctness.** Two body-model rewrites, a systematic
|
||||
scale error affecting every geophone reading ever produced, a recovered
|
||||
file format, and two artifact-hygiene bugs where stale files outlived the
|
||||
decodes that made them. All 11,603 series-3 binaries in the production
|
||||
snapshot now pass every check.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Series-3 health sweep: 11,603 / 11,603 binaries now clean on every check.**
|
||||
Swept every series-3 file with the live decoder against five independent
|
||||
checks — decode exceptions, zero samples, unequal geo channel lengths, peaks
|
||||
above range full scale, decoded peak vs the device-reported PPV, and waveform
|
||||
length vs the declared record time. Three real defects surfaced and were
|
||||
fixed:
|
||||
|
||||
- **`block[22]` is not a constant and must not be tested.** It was documented
|
||||
as always `0x00` but carries data on loud blocks, and rejecting those threw
|
||||
away the interval holding the event peak.
|
||||
`BE18350/T350L7HR.NL0H` block 92 has `block[22]=0x26` and a Tran peak of
|
||||
`0x0563` = 1379 counts = **6.895 in/s** — exactly the device-reported PPV —
|
||||
while the file as a whole decoded to 0.015 in/s. `block[0]==0x00`,
|
||||
`block[4]==0x0A` and the 4-byte tail are six bytes of constraint, which is
|
||||
what keeps trailer content out.
|
||||
|
||||
- **Block-model dispatch now goes on signature strength, not on whichever
|
||||
decoder returns first.** A multi-interval body also yields scattered
|
||||
standard-tail blocks by coincidence; dispatching on "first non-empty"
|
||||
handed 193 BE18193 files to the standard walker and produced peaks of
|
||||
149 in/s against a 10 in/s full scale.
|
||||
|
||||
- **Multi-interval stride detection requires the block counter to increment
|
||||
by exactly 1.** Without it the detector false-positives on ordinary
|
||||
standard-block bodies: those carry a header every 32 bytes, and
|
||||
`192 = 12 + 20×9` and `512 = 12 + 20×25` are both multiples of 32, so a
|
||||
stride "fits" while actually skipping 6 or 16 real blocks. That misrouted
|
||||
9,082 files.
|
||||
|
||||
Partial-block garbage is now trimmed within the final block only, stopping at
|
||||
the first slot with a non-zero tail word or a geo peak above full scale.
|
||||
Trimming purely from the end left garbage stranded behind one slot that
|
||||
happened to have a zero tail word; trimming on the tail word alone truncated
|
||||
four BE9440 files by up to 2,800 intervals.
|
||||
|
||||
- **Sub-minute histogram intervals are packed several to a block — 415 files
|
||||
recovered.** The device always writes one minute of data per block, so a
|
||||
shorter interval just means more intervals in a longer block:
|
||||
|
||||
| interval | intervals/block | stride |
|
||||
|---|---|---|
|
||||
| 1 min | 1 | 32 (the standard block) |
|
||||
| 15 s | 4 | 92 |
|
||||
| 2 s | 30 | 612 |
|
||||
|
||||
`stride = 12 + n * 20`. Each 20-byte record carries 8 × uint16
|
||||
**little**-endian values — peak and half-period per channel — plus a 2-word
|
||||
tail whose first word is `0000` on every real interval (a session ending
|
||||
mid-block leaves buffer garbage in the remaining slots, which decoded as
|
||||
peaks thousands of times the real value until that check was added).
|
||||
**The standard 32-byte block is big-endian; this variant is not.**
|
||||
|
||||
These 415 files (216 on BE18193 at 2 s intervals, 199 on BE9440 at 15 s)
|
||||
previously decoded to nothing at all — and before that were being accepted
|
||||
by the *waveform* codec, which returned garbage peaking up to 400× the
|
||||
device-reported PPV.
|
||||
|
||||
Ground truth `BE9440/K440L3AQ.T70H` — 5,710 intervals — matches its
|
||||
Blastware ASCII export on **17,130/17,130** geo peaks, **22,840/22,840**
|
||||
frequencies and **5,710/5,710** mic dB(L) values. Across all 455 affected
|
||||
files, **1,354/1,365 (99.2%)** channel peaks match the device-reported PPV;
|
||||
the 11 that don't are under-reads on BE9440 where the walk stops early.
|
||||
|
||||
- **`backfill_sidecars.py` now removes a stale `.h5` when nothing decodes.**
|
||||
It previously skipped the write "so we don't replace whatever's there with an
|
||||
empty placeholder", which silently preserved output from a superseded
|
||||
decoder. After the record-chain fix, 415 histogram files stopped decoding (an
|
||||
unmapped block variant on BE18193 and BE9440) but kept `.h5` files whose peaks
|
||||
ran up to **400× the device's own reported PPV** — garbage feeding the charts
|
||||
and the false-trigger detector with nothing marking it. Reports a
|
||||
`stale_h5_removed` count.
|
||||
|
||||
- **The series-3 waveform body is a RECORD CHAIN, not a tag stream — this
|
||||
supersedes the segment-header model, including the fixes made earlier the
|
||||
same day.**
|
||||
|
||||
Records are self-delimiting. `off+2` is a `uint16 BE` length and
|
||||
`next_record = off + 2 + len`; the chain ends on a record whose `chan_id` is
|
||||
`0x06`. `off+8` carries a 3-valued mode enum:
|
||||
|
||||
| mode | header | data section |
|
||||
|---|---|---|
|
||||
| `02 00` | 14 B | anchors, then **cumulative deltas** |
|
||||
| `01 00` | 10 B | no anchors, **absolute** values |
|
||||
| `00 03` | 10 B | **no tags at all** — raw 12-bit packed absolute |
|
||||
|
||||
**`40 NN` is an ordinary int16 BE data block** (`2*NN + 2`), never a segment
|
||||
header. Reading it as a `2*NN + 16` header is what made walks drift — and the
|
||||
"variable-prefix segment descriptors" reported earlier today were not a format
|
||||
feature at all, just walker drift of exactly
|
||||
`4 - (old_stop - true_record_start)` on all 25 affected files.
|
||||
|
||||
Measured against the production snapshot:
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| all four channels equal length | 156 / 1388 | **1388 / 1388** |
|
||||
| ASCII sample-count exact | 72 / 75 | **75 / 75** |
|
||||
| ASCII fully exact | 70 / 75 | **73 / 75** |
|
||||
| device PPV, waveform (live decode) | 1288 / 1306 | **1306 / 1306** |
|
||||
| device PPV, histogram (live decode) | 4434 / 4459 | **4458 / 4459** |
|
||||
|
||||
Mean absolute PPV ratio error on waveforms is now 0.00000. The 2 remaining
|
||||
ASCII imperfections differ by exactly 1 LSB on samples sitting at the
|
||||
±10.000 in/s rail.
|
||||
|
||||
**This also eliminated the walker-over-read class.** 24 of those 35 files
|
||||
were histograms that `read_blastware_file` fed to the *waveform* codec first;
|
||||
the old walker accepted them and returned garbage (one yielded 98,923
|
||||
"intervals"), while the record-chain decoder correctly returns `None` so they
|
||||
fall through to `histogram_codec`.
|
||||
|
||||
`00 03` records are decoded rather than skipped. Skipping them does not merely
|
||||
lose samples — it silently shifts the time base of everything after them on
|
||||
that channel (observed on `BE9558/K558LOF2.820W`, MicL displaced by exactly
|
||||
512 samples with nothing marking the gap).
|
||||
|
||||
Footer detection now prefers whichever `0e 08` candidate yields a chain
|
||||
terminating on `0x06`, since the signature can occur inside a sample stream.
|
||||
Blast radius: 1 file of 1,388.
|
||||
|
||||
The superseded model is retained as `decode_waveform_legacy` and pinned by
|
||||
`micromate/idf_file.py`, whose Thor IDFW body-offset search trial-decodes
|
||||
candidate offsets and keeps whichever yields the most samples — the new
|
||||
decoder correctly returns `None` where the old one returned garbage, which
|
||||
changes that heuristic's winner. Switching Thor over is deferred until that
|
||||
search is reworked to use the record chain directly.
|
||||
|
||||
- **Series-3 histogram block is uniformly big-endian, and the stream's final
|
||||
block has its own tail — the codec was clipping large peaks and dropping the
|
||||
last interval of nearly every histogram.**
|
||||
|
||||
- **Peaks and half-periods are `uint16` big-endian**, not `uint8` plus an
|
||||
"annotation" byte: `T_peak` `[5:7]`, `T_halfperiod` `[7:9]`, `V_peak`
|
||||
`[9:11]`, and so on. Only `block_ctr` `[2:4]` is little-endian. The old
|
||||
model silently **clipped any peak above 1.275 in/s** — the final interval
|
||||
of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in Blastware's own export and
|
||||
decoded as 0.590. The "annotation" byte was the half-period's high byte,
|
||||
which is why it was non-zero exactly on the sub-Hz intervals BW renders
|
||||
as `<1.0`.
|
||||
- **The marker is `block[4]` alone.** Testing `[4:6]` as a `uint16 LE`
|
||||
marker forced `block[5] == 0` — which is what capped the peak at one byte
|
||||
in the first place.
|
||||
- **The last block of each stream carries tail `9c 06 00 42`** instead of
|
||||
`1e 0a 00 00`, with arbitrary bytes at `[21:23]`. Rejecting it dropped
|
||||
the final interval of nearly every histogram, and that interval is
|
||||
frequently the one holding the event peak — so the file's reported PPV
|
||||
came out low.
|
||||
|
||||
Verified against **1211 production histograms** paired with their preserved
|
||||
Blastware ASCII exports, which carry a full per-interval data table:
|
||||
**1211/1211 now decode exactly** (interval count plus every per-interval
|
||||
peak), and 842,442 per-interval frequency comparisons match with zero
|
||||
mismatches. Before this fix: **1 of 1196**.
|
||||
|
||||
`decode_histogram_body_full` records now expose `is_terminal` in place of
|
||||
the removed `annotations` tuple.
|
||||
|
||||
- **Geophone full scale is 32000 ADC counts, not 32768 — every geo reading was
|
||||
2.3% low.** The verified body codec emits geo samples in 16-count units whose
|
||||
documented LSB is exactly 0.005 in/s, and `decoded_to_adc_counts` multiplies
|
||||
by 16, so one ADC count is `0.005/16` in/s and Normal range (10.000 in/s) is
|
||||
`10.0 / (0.005/16)` = **32000** counts. Both `sfm/event_hdf5.py` and
|
||||
`minimateplus/event_file_io.py` divided by 32768, scaling every geophone
|
||||
sample and every derived peak down by `1 - 32000/32768` = **2.34%**.
|
||||
|
||||
Measured against 216 per-channel comparisons with preserved Blastware ASCII
|
||||
exports: **32768 → 151/216 exact** (worst error 0.238 in/s on a 10 in/s
|
||||
event); **32000 → 216/216 exact**, worst error 0.005 in/s (exactly 1 LSB —
|
||||
pure quantization). The error scales with amplitude, so it was invisible on
|
||||
quiet events and worst on the loud ones that matter for compliance.
|
||||
|
||||
The mic path is unaffected — it back-solves its own per-count factor from the
|
||||
device-reported peak.
|
||||
|
||||
**Scope:** the scale lives in `_samples_to_float`, which every event passes
|
||||
through regardless of which codec produced the samples — so this affected
|
||||
**waveforms, histograms and series-4 (Thor IDF) alike**, not just waveforms.
|
||||
Verified after regeneration: series-3 histogram peaks vs their ASCII reports
|
||||
now sit at a median ratio of 1.0000 across 1,137 comparisons (0.9766 under
|
||||
32768); series-4 peaks vs device peaks moved from a median 0.960 to 0.983
|
||||
across 1,468 comparisons. The four block-framing fixes below are
|
||||
waveform-only — histograms decode via `histogram_codec.decode_histogram_body`,
|
||||
which is untouched.
|
||||
|
||||
- **Series-3 waveform codec: four block-framing cases caused silent channel
|
||||
truncation.** `walk_body` hit its unknown-tag `break` mid-stream and every
|
||||
channel decoded after that point came out short — typically Vert/Long/MicL,
|
||||
sometimes at a third of their true length, with no error raised.
|
||||
|
||||
- **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already handled for
|
||||
`1X NN` / `2X NN` also applies to the `00 NN` RLE tag. Runs longer than
|
||||
252 samples must use the wide form (e.g. `01 0c` = 268 repeats).
|
||||
- **`30 NN` with NN > 0x10** — the `0 < NN <= 0x10` guard was arbitrary;
|
||||
data-section `30 NN` blocks reach at least NN = 0x18. The length formula
|
||||
(`NN × 1.5 + 2`) was already correct.
|
||||
- **Variable-width `40 NN` segment headers** — NN is the *count of
|
||||
previous-channel continuation deltas*, so the header is `2 × NN + 16`
|
||||
bytes and every field after the deltas shifts by `2 × NN`. Only `40 02`
|
||||
(20 bytes) was handled; `40 01` (18) and `40 03` (22) both occur.
|
||||
- **Tagless segment headers** — a segment header can appear with no
|
||||
`40 NN` tag at all: just the 14-byte tail
|
||||
`[field2:2][len:2][channel_id:4][marker:2][anchors:4]`. This is the NN=0
|
||||
case (no continuation deltas needed, so no tag and no delta bytes). It is
|
||||
where the walk stopped in 7 of the 8 events still truncating after the
|
||||
first three fixes.
|
||||
|
||||
### Changed
|
||||
|
||||
- **Segment channel now comes from the header's own channel-id byte** rather
|
||||
than from rotation position. The field previously documented as a
|
||||
"monotonic uint32 LE counter" is really `[channel][00][00][segment_index]`
|
||||
with `0x46`=Tran `0x47`=Vert `0x48`=Long `0x49`=MicL — verified on
|
||||
**1697 of 1697** segment headers across the ground-truth corpus with zero
|
||||
disagreements. Rotation-by-position is kept only as a fallback for unknown
|
||||
ids; it was fragile because a single missed or extra header (exactly what
|
||||
tagless headers caused) desynced every channel after it.
|
||||
|
||||
- **`parse_segment_header` return shape** — now `n_prev_deltas`,
|
||||
`prev_deltas`, `marker`, `anchors`, `channel`, `segment_index` in place of
|
||||
the fixed-offset `anchor_bytes` / `fixed_pattern` / `tail` keys. The old
|
||||
`fixed_pattern` (`02 00 00 01`) conflated the 2-byte constant marker with
|
||||
the first anchor. `counter` is retained as the raw uint32 of the id field.
|
||||
|
||||
### Verification
|
||||
|
||||
Against the 75 ground-truth events (BW binary paired with its preserved
|
||||
`_ASCII.TXT` export), decoding end-to-end through the production path:
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| exact (full length, within 1 LSB) | 37 | **72** |
|
||||
| truncated | 23 | **3** |
|
||||
| full length, value error > 2 LSB | 15 | **0** |
|
||||
|
||||
Worst remaining error among the 72: 0.0050 in/s = exactly 1 LSB.
|
||||
|
||||
No regressions — the byte-exact fixture suite still passes, and the full-suite
|
||||
failure list is unchanged from baseline (16 pre-existing failures from
|
||||
gitignored fixtures).
|
||||
|
||||
### Notes
|
||||
|
||||
- **The "DC offset" symptom is _not_ a decode bug.** Events whose geo trace
|
||||
sits at a constant level instead of oscillating around zero
|
||||
(dominant-axis `|mean| / peak` >> 0) reproduce *exactly* in Blastware's own
|
||||
ASCII export — e.g. `BE12599/N599LQD7.8E0W` Tran reads mean +0.345,
|
||||
min +0.335, max +0.355 in both. It is a known recurring hardware fault (the
|
||||
operators call it an "offset"): the affected channel's baseline exceeds the
|
||||
unit's own geo trigger level, so the unit retriggers continuously and floods
|
||||
the ACH queue with garbage events. Store-wide it affects 2 units of 21 across
|
||||
6 episodes; see `scratch/offset_candidates.csv` and the project memory notes.
|
||||
|
||||
- ~~**Still open:** 3 of 75 ground-truth events truncate at a segment-header
|
||||
variant with a variable-width prefix.~~ **Resolved later the same day** — the
|
||||
record-chain rewrite (above) showed there is no variable prefix; it was
|
||||
walker drift. All 75 are now sample-count exact.
|
||||
|
||||
- **Still open after this release:**
|
||||
- **Series-4 (Thor / Micromate) is not verified** — UM-series sits at ~48%
|
||||
against device peaks with a ~1.7% systematic bias and a near-zero tail.
|
||||
Thor IDFW is pinned to `decode_waveform_legacy` deliberately.
|
||||
- **14 sensitive-range files** show a decoded/truth ratio of exactly 8.0
|
||||
(= 10.0/1.25) — a units bug, not a codec one. Never chased.
|
||||
- **`backfill_sidecars.py --force` also inserts DB rows** for store files
|
||||
that have none (1,286 on the snapshot; one-time per store), and the
|
||||
dry-run does not report that count before you commit to it.
|
||||
- **Verification is uneven:** per-sample proof on the 11% of files with a
|
||||
preserved `_ASCII.TXT`, peak-and-structure consistency on the other 89%.
|
||||
|
||||
---
|
||||
|
||||
## v0.25.0 — 2026-08-25
|
||||
|
||||
**reviewed_real 3-state review flag + twin review-propagation.** The
|
||||
`events` table and the `/db/events` feed now carry `reviewed_real`, a
|
||||
3-state review flag mutually exclusive with `false_trigger`, mirrored from
|
||||
the sidecar review block — plus histogram/waveform twin review-propagation
|
||||
(flagging one flags both, matched by serial + identical PVS + timestamp
|
||||
window).
|
||||
|
||||
### Added
|
||||
|
||||
- **`events` column** `reviewed_real` — `INTEGER NOT NULL DEFAULT 0`, added
|
||||
via the existing incremental `_migrate` ADD COLUMN pass (auto-migrates on
|
||||
`SeismoDb()` construction, no manual migration). `query_events` /
|
||||
`get_event` (and thus `/db/events`) return it automatically (`SELECT *`).
|
||||
- **Mutual exclusivity with `false_trigger`** — setting `reviewed_real=1`
|
||||
clears `false_trigger`, and vice versa, enforced on both review paths:
|
||||
the sidecar review PATCH (`update_event_review`) and the quick
|
||||
`PATCH /db/events/{id}/false_trigger` endpoint (`set_false_trigger`).
|
||||
- **`find_twins`** — matches an event's histogram/waveform twins by serial +
|
||||
identical peak-vector-sum + a timestamp window.
|
||||
- **`propagate_review_to_twins`** — copies an event's `false_trigger`/
|
||||
`reviewed_real` state onto its twins, wired into both the
|
||||
`PATCH /db/events/{id}/sidecar` review path and the quick
|
||||
`PATCH /db/events/{id}/false_trigger` path, so flagging one flags both
|
||||
regardless of which endpoint made the change.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -2,12 +2,152 @@
|
||||
|
||||
Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
|
||||
managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
|
||||
(Sierra Wireless RV50 / RV55). Current version: **v0.21.0**.
|
||||
(Sierra Wireless RV50 / RV55). Current version: **v0.31.0**.
|
||||
|
||||
Stack-level context — which repo owns what, and how the three project versions
|
||||
pair — lives in `../terra-view/docs/tmi-stack.md`, which is also loaded as
|
||||
`~/CLAUDE.md`.
|
||||
|
||||
---
|
||||
|
||||
## Where things stand (updated 2026-08-28)
|
||||
|
||||
Read this first when picking the project back up.
|
||||
|
||||
- **Series-3 decode is verified per-sample at scale (v0.27.0).** The full DL2
|
||||
archive decodes **14,338 / 14,338** paired files exactly against their
|
||||
preserved Blastware ASCII exports — 1,249 waveform + 13,089 histogram, 45
|
||||
units, files back to 2018. That is 11x the ground truth the prod store
|
||||
carried, and it supersedes the old "per-sample on 11%, peak-only on 89%"
|
||||
caveat. Harness: `scratch/verify_against_ascii.py` (note its saturation
|
||||
carve-out — BW clamps clipped events, the decoder reports true counts).
|
||||
Independent corroboration of the 32000-count scale: 19,244 healthy
|
||||
channel-events sit at a pre-trigger floor of exactly 0.000 (62.7%), 94.5%
|
||||
within ±1 quantisation unit, median +0.0000 — no zero-point bias.
|
||||
- **Series-4 (Thor / Micromate) is now verified per-sample (2026-09-10).**
|
||||
**1,057,536 / 1,057,536** geo samples across all 153 genuine Thor waveform
|
||||
files reproduce Thor's own CSV export exactly; IDFH peaks are within 2% on
|
||||
858/858 (median -0.004%). The ground truth was in the corpus all along —
|
||||
Thor writes `CSV/<name>.IDFW.csv` beside each binary with a **per-sample**
|
||||
four-column block. Harness: `scratch/verify_thor_against_csv.py`.
|
||||
Four bugs, all fixed: geo LSB was `0.0003` (display rounding of the real
|
||||
`0.000310308`, so every sample read **3.3% low**); the IDFH segment
|
||||
validator required a zero counter high byte, **capping every histogram at
|
||||
250 intervals**; record mode `00 00` (raw int16) was unhandled, silently
|
||||
dropping each channel's first 512 samples; and the body-offset search
|
||||
matched `00 02 00` *inside* record headers, decoding a rotation-shifted
|
||||
body. IDFW is no longer pinned to `decode_waveform_legacy`.
|
||||
Series-3 re-verified unchanged at 14,338/14,338 after the shared-codec
|
||||
change.
|
||||
- **Mic-disabled (3-channel) units are a distinct shape (2026-09-10).**
|
||||
Verified on a second corpus (`~/thor-csv-req`, UM11402/UM12947/UM20147):
|
||||
**139/139** waveforms per-sample exact, **877/877** histograms within 2%.
|
||||
Two structural differences: the shorter header puts the waveform record
|
||||
chain head at `0x0dba` (below the old `_BODY_SCAN_FLOOR` of `0x0E00`, so it
|
||||
was invisible and Vert came up exactly 512 short), and the histogram
|
||||
interval record is **56 bytes, not 72** — `16 × n_channels + 8`, derived per
|
||||
segment from the cumulative interval counter, never assumed.
|
||||
- **`40 NN` blocks are not capped at NN=8 (2026-09-11).** `data_block_len()`
|
||||
rejected `NN > 0x08`, a guard with no evidence behind it — the corpora
|
||||
available when it was written only used NN ∈ {1,2,3,4,8}. Loud UM12947
|
||||
events use NN up to 196, and since the walker stops at the first
|
||||
unrecognised tag rather than raising, this surfaced as silently short
|
||||
channels. Verified on 167 UM12947 waveforms: length mismatches 22 → 0,
|
||||
1,476,242/1,476,249 samples exact.
|
||||
- **Production IDFW is now 575/575** — zero truncations, zero decode
|
||||
failures, median PPV error −0.0007% across 8 units (was 41 truncated + 1
|
||||
failing, −3.3%). Across all three ground-truth corpora: **459 files,
|
||||
3,807,158/3,807,165 samples exact**; the 7 stragglers differ by one
|
||||
4th-decimal tick and are Thor's own rounding — no single linear LSB can
|
||||
reproduce every printed value (the constraints are infeasible by 7e-5
|
||||
relative), so do NOT retune `_GEO_LSB_IPS`.
|
||||
- **⚠ KNOWN BUG — the 5A walk breaks once a unit's buffer crosses 64 KB.**
|
||||
`parse_strt_end_offset()` returns only `(end_key[2] << 8) | end_key[3]`,
|
||||
discarding the key's page byte. An event starting at `0x0111F2A2` and ending
|
||||
at `0x0112_1010` therefore reads `end_offset = 0x1010` — *behind* its own
|
||||
start. The chunk loop then exits before fetching anything and TERM computes
|
||||
a negative `offset_word`, which `struct.pack(">H", ...)` rejects: the
|
||||
`/device/events` walk 500s. Reproduced on BE12599 (2026-09-19), which had
|
||||
78 KB stored and had rolled into page `0x12`.
|
||||
**Why it hid so long:** every 5A capture the walk was verified against came
|
||||
from a freshly-erased BE11529 — all three confirmed TERM examples in
|
||||
`framing.py` (`0x1ABE`, `0x21F2`, `0x417E`) sit inside page `0x11`. Prod is
|
||||
unaffected: it ingests complete files via BW ACH, never this walk.
|
||||
**Fixing it has two layers** — the arithmetic (`if end < start: end +=
|
||||
0x10000`) stops the crash and bounds the loop correctly; carrying the page
|
||||
byte through the chunk requests (`params[1]` 0x11 -> 0x12, counter rolling
|
||||
over) needs a BW capture of a spanning event first. Do not ship layer one
|
||||
alone without a loud truncation warning — a silently short event is the
|
||||
failure mode this codec has been bitten by repeatedly.
|
||||
- **Open, not blocking:** 14 sensitive-range files show an exact 8x
|
||||
(= 10.0/1.25) units discrepancy; `scripts/backfill_sidecars.py --force` also
|
||||
inserts DB rows for store files that have none (one-time per store) and the
|
||||
dry-run does not report that count.
|
||||
- **After any codec change, regenerate the store** — `backfill_sidecars.py`
|
||||
then `backfill_event_shape.py`, DB backup first. Stored `.h5` files do not
|
||||
update themselves. No `--force` needed as long as `TOOL_VERSION` was bumped
|
||||
(it gates regeneration). ⚠ On the office NAS this takes **~2 hours**
|
||||
(~1.5 files/sec vs 85/sec on the dev box — gzip-4 in `sfm/event_hdf5.py`
|
||||
against a Synology CPU). Budget it up front.
|
||||
**v0.27.0 does NOT owe prod a backfill** — verified: the partial-final-block
|
||||
fix changes 0 of the 10,215 histograms in the prod store (the 4 recovered
|
||||
files are archive-only and were never ingested).
|
||||
- **The "offset" hardware fault has its own journal** --
|
||||
`docs/offset_investigation.md`. **5 of 45 units (11%)**, and the fault is
|
||||
**persistent** — it stays until the geophone is serviced. Detect it with
|
||||
`scratch/offset_scan3.py`: the resting floor in the **pre-trigger** window,
|
||||
required to hold across pre/middle/end. Never score only the dominant-peak
|
||||
axis and never use the mean — both produce false recoveries (see the
|
||||
retraction banner in the journal). Instantel's autozero procedure and its
|
||||
2027-2069 acceptance window are recorded there too. Best open lead is
|
||||
`SUB 0x0E` (unimplemented), which may carry those very numbers.
|
||||
|
||||
|
||||
When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
|
||||
|
||||
---
|
||||
|
||||
## Changelog & release convention
|
||||
|
||||
**Feature branches do NOT touch `CHANGELOG.md`. Write the entry on `dev`, as
|
||||
part of finishing the merge, under `## Unreleased`. Cut the version on `dev` in a
|
||||
dedicated release commit when you are ready to ship to `main`.**
|
||||
|
||||
- **The changelog is written on `dev`, never on a feature branch.** With
|
||||
several branches in flight they all edit the same few lines at the top of
|
||||
the file and conflict every time. Writing it once, after the merge, also
|
||||
lets it describe what actually *landed* — including anything that changed
|
||||
during conflict resolution.
|
||||
- ⚠ **The merge is not finished until `## Unreleased` is updated.** Same sitting,
|
||||
not "later" — that is the one failure mode of writing it after the fact.
|
||||
Reconstruct from the branch's own commit messages:
|
||||
`git log --oneline dev..<branch>` before you merge, or
|
||||
`git log --oneline <merge-base>..<branch>` after.
|
||||
- **No preamble under `## Unreleased`** — just the `### Added` / `### Changed` /
|
||||
`### Fixed` lists. The themed opening paragraph gets written at release
|
||||
time, when the whole release is visible and can be named honestly. A theme
|
||||
written when the first item landed is stale by the third.
|
||||
- ⚠ **State the operational consequence** on any entry touching the codec, the
|
||||
waveform store, or the DB — **including when it is "none."** "requires
|
||||
`backfill_sidecars.py` + `backfill_event_shape.py`, ~2 h on the NAS",
|
||||
"`TOOL_VERSION` bumped", "no schema change, no migration". Silence is
|
||||
ambiguous; "none" is information. This repo's changelog is how future-you
|
||||
learns whether a deploy costs two hours.
|
||||
- **Releases are cut on judgement, not on a schedule or a merge.** `Unreleased`
|
||||
is the staging area for whatever is going into the next release; when enough
|
||||
has accumulated to be worth shipping, it gets a number and a date. Nothing
|
||||
about a merge to `dev` triggers a release.
|
||||
- **Cutting a release** is its own `chore(release): vX.Y.Z — <theme>` commit on
|
||||
`dev`, renaming `## Unreleased` → `## vX.Y.Z — YYYY-MM-DD` and touching:
|
||||
`CHANGELOG.md`, `pyproject.toml`, the version line in `CLAUDE.md` and
|
||||
`README.md`, and `minimateplus/event_file_io.py` (`TOOL_VERSION`) **when the
|
||||
codec changed** — that constant gates `.h5` regeneration.
|
||||
- **`main` carries only released versions.** No `## Unreleased` section there;
|
||||
it lands via the `dev` → `main` PR. `main` lagging `dev` by a version is
|
||||
normal.
|
||||
|
||||
---
|
||||
|
||||
## Architecture: three-tier conceptual model
|
||||
|
||||
seismo-relay is a **suite of cooperating components**, not a single app.
|
||||
@@ -73,20 +213,34 @@ should not import from `sfm/`, must not touch a DB, and have no I/O
|
||||
beyond reading files passed as arguments. Keep them pure — both
|
||||
tiers can then depend on them without circularity.
|
||||
|
||||
#### Thor IDF binary codec (2026-05-28)
|
||||
#### Thor IDF binary codec (updated 2026-09-10)
|
||||
|
||||
`micromate/idf_file.read_idf_file()` decodes both Thor IDFW
|
||||
(waveform) and IDFH (histogram) binaries.
|
||||
(waveform) and IDFH (histogram) binaries. **Verified per-sample
|
||||
against Thor's own CSV exports** — see
|
||||
`scratch/verify_thor_against_csv.py`.
|
||||
|
||||
- **IDFW** reuses `decode_waveform_v2()` on the body at fixed file
|
||||
offset `0x0f1f`. Sample fidelity is 87–99% byte-exact on quiet
|
||||
events; loud events hit the BW codec's known walker-stops-early
|
||||
limitation.
|
||||
- **IDFH** has its own segment-based decoder: `[len_be][0a 00 00 00]
|
||||
[00 NN][05 3f]` + N × 72-byte interval records (4 × 16-byte
|
||||
per-channel min/max/halfp). All 859 Thor IDFH corpus files
|
||||
decode (181,071 intervals); peak matches sidecar within ~1.8%
|
||||
(ADC quantization).
|
||||
- **IDFW** uses the series-3 record-chain `decode_waveform_v2()`. The
|
||||
body offset is **not** fixed: it is `<chain-head record> + 7`, found
|
||||
by `_find_waveform_body_offset()` anchoring on record headers. All
|
||||
**153/153** genuine Thor waveform files decode per-sample exact
|
||||
(1,057,536/1,057,536 samples).
|
||||
- **IDFH** segment header is `[len_be][0a 00 00 00][counter_be][05 3f]`,
|
||||
where `counter` is a **uint16 cumulative interval index** — it must
|
||||
not be constrained to a zero high byte (that capped histograms at 250
|
||||
intervals). Intervals whose `min > max` on all channels are unwritten
|
||||
slots carrying a ±full-scale seed and are skipped. 858/858 files land
|
||||
within 2% of Thor's PPV (median -0.004%).
|
||||
- **Geo LSB is `0.000310308` in/s per count** (full scale 10.0 in/s =
|
||||
32226.05 counts). Series-3's 32000-count scale does NOT apply.
|
||||
- **Record modes** are `02 00` deltas (14 B header), `01 00` absolute,
|
||||
`00 03` raw 12-bit, and `00 00` **raw int16** (all 10 B headers).
|
||||
`01 00` and `00 00` are also valid as the implicit segment-0 preamble.
|
||||
|
||||
⚠ **Thor's histogram PPV has a 0.0050 in/s display floor.** 41.4% of
|
||||
prod IDFH sidecars report a component PPV exceeding their own vector
|
||||
sum — impossible. On quiet files our decode is *more* accurate than
|
||||
the reference; do not "fix" the decoder to match it.
|
||||
|
||||
The two outlier `BE9439_*` files in the Thor example corpus are
|
||||
actually Series III Blastware binaries that share the `.IDFW`/`.IDFH`
|
||||
@@ -223,6 +377,118 @@ custom delta + RLE + variable-width codec.
|
||||
`NN + 2` for int8 blocks). Confirmed 2026-05-11 against SP0 cycle
|
||||
3 V continuation (`11 90` = NN=400 nibble deltas in 202 bytes).
|
||||
|
||||
### ⚠ SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN
|
||||
|
||||
Everything in this section below about `40 NN` segment headers, tagless
|
||||
headers, variable header widths and channel rotation describes a model that
|
||||
is **wrong**. The body is a chain of self-delimiting per-channel records:
|
||||
|
||||
off+2 len uint16 BE -> next_record = off + 2 + len
|
||||
off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL / 0x06 = END
|
||||
off+8 mode 02 00 = deltas+anchors (14B hdr)
|
||||
01 00 = ABSOLUTE values (10B hdr)
|
||||
00 03 = raw 12-bit absolute, NO TAGS (10B hdr)
|
||||
|
||||
`40 NN` is an ordinary int16 BE data block (`2*NN + 2`), never a header. The
|
||||
"variable prefix" of 0/2/4/6/8 bytes was walker drift, exactly
|
||||
`4 - (old_stop - true_record_start)`.
|
||||
|
||||
All four channels now come out equal length in **1388/1388** files (was
|
||||
156/1388); ASCII sample-count exact **75/75**, fully exact **73/75**; device
|
||||
PPV on a live decode **1306/1306** waveform, **4458/4459** histogram.
|
||||
|
||||
The old model survives as `decode_waveform_legacy` because
|
||||
`micromate/idf_file.py` pins it for Thor IDFW body-offset search.
|
||||
|
||||
### Framing cases added 2026-05-11 → 2026-08-25
|
||||
|
||||
Four more block-framing cases, each of which had been causing **silent
|
||||
channel truncation** — `walk_body` ends its loop on an unrecognised tag
|
||||
and `decode_waveform_v2` returns whatever channels it got, so an
|
||||
unhandled tag surfaces as short channels with no error raised. Found by
|
||||
diffing 75 production events against their preserved Blastware ASCII
|
||||
exports (`<store>/<serial>/<file>_ASCII.TXT`).
|
||||
|
||||
- **Wide-NN RLE `0X NN`** — the 12-bit NN encoding documented above for
|
||||
`1X`/`2X` **also applies to the `00 NN` RLE tag**. A narrow run maxes
|
||||
out at NN=0xFC, so a quiet stretch longer than 252 samples must use
|
||||
the wide form (e.g. `01 0c` = 268 repeats).
|
||||
- **`30 NN` is not capped at NN=0x10** — data-section blocks reach at
|
||||
least NN=0x18. The `NN × 1.5 + 2` length formula was already right;
|
||||
only the guard was wrong.
|
||||
- **`40 NN` segment headers are variable width** — NN is the *count of
|
||||
int16 BE continuation deltas for the PREVIOUS channel*, so the header
|
||||
is `2*NN + 16` bytes and every field after the deltas shifts by
|
||||
`2*NN`. `40 01` (18 B) and `40 03` (22 B) both occur alongside the
|
||||
common `40 02` (20 B).
|
||||
- **Tagless segment headers** — a header can appear with **no `40 NN`
|
||||
tag at all**: just the 14-byte tail
|
||||
`[field2:2][len:2][channel_id:4][marker:2][anchors:4]`. This is the
|
||||
NN=0 case (previous channel needed no continuation deltas).
|
||||
|
||||
**The header "counter" is really a channel id.** The 4-byte field long
|
||||
documented as a "monotonic uint32 LE counter" is
|
||||
`[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert
|
||||
`0x48`=Long `0x49`=MicL — verified on **1697/1697** segment headers
|
||||
across the corpus, zero disagreements. `decode_waveform_v2` now takes
|
||||
the channel from this field rather than from rotation position; a single
|
||||
missed or extra header (exactly what tagless headers caused) desyncs
|
||||
rotation and corrupts every channel after it.
|
||||
|
||||
Corpus result, end to end through the production path:
|
||||
**exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.**
|
||||
|
||||
### Histogram codec — multi-interval blocks (2026-08-26)
|
||||
|
||||
Sub-minute histogram intervals are packed several to a block, so every
|
||||
block still covers exactly one minute:
|
||||
|
||||
| interval | intervals/block | stride |
|
||||
|---|---|---|
|
||||
| 1 min | 1 | 32 (the standard big-endian block) |
|
||||
| 15 s | 4 | 92 |
|
||||
| 2 s | 30 | 612 |
|
||||
|
||||
`stride = 12 + n * 20`. Block = `[00][segment][ctr uint16 LE][0a][00]`,
|
||||
then n x 20-byte records of 8 x uint16 **LITTLE**-endian values
|
||||
(`T_peak, T_halfp, V_peak, V_halfp, L_peak, L_halfp, M_peak, M_halfp`)
|
||||
plus a 2-word tail whose first word is `0000` on every real interval,
|
||||
then a 6-byte block trailer.
|
||||
|
||||
⚠ The standard 32-byte block is BIG-endian; this variant is LITTLE-endian.
|
||||
|
||||
Recovers **415 files** (216 on BE18193, 199 on BE9440) that decoded to
|
||||
nothing. Ground truth `BE9440/K440L3AQ.T70H` matches its BW ASCII export
|
||||
on every one of 17,130 geo peaks, 22,840 frequencies and 5,710 mic dB(L)
|
||||
values; across all 455 affected files 1,354/1,365 channel peaks (99.2%)
|
||||
match the device-reported PPV.
|
||||
|
||||
### Histogram codec — corrected 2026-08-25
|
||||
|
||||
The histogram block is **uniformly big-endian**, and the stream's final
|
||||
block has its own tail signature. Two long-standing errors:
|
||||
|
||||
- **Peaks and half-periods are `uint16` big-endian**, not `uint8` +
|
||||
an "annotation" byte. `T_peak` is `[5:7]`, `T_halfperiod` `[7:9]`,
|
||||
`V_peak` `[9:11]`, and so on; only `block_ctr` at `[2:4]` is LE.
|
||||
The old model silently **clipped any peak above 1.275 in/s** — the
|
||||
final interval of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in BW's
|
||||
export and decoded as 0.590. The "annotation" byte was the
|
||||
half-period's high byte, which is why it was non-zero exactly on the
|
||||
sub-Hz intervals BW renders as `<1.0`.
|
||||
- **The marker is `block[4]` alone.** Testing `[4:6]` as a uint16 LE
|
||||
marker forced `block[5] == 0`, which is what capped the peak at one
|
||||
byte in the first place.
|
||||
- **The last block of the stream carries tail `9c 06 00 42`** instead of
|
||||
`1e 0a 00 00`, with arbitrary bytes at `[21:23]`. Rejecting it
|
||||
dropped the final interval of nearly every histogram — frequently the
|
||||
one holding the event peak, so the file's PPV read low.
|
||||
|
||||
Verified against 1211 production histograms paired with their BW ASCII
|
||||
exports: **1211/1211 decode exactly** (interval count plus every
|
||||
per-interval peak), and 842,442 per-interval frequency comparisons match
|
||||
with zero mismatches. Before: 1 of 1196.
|
||||
|
||||
### What's NOT solved
|
||||
|
||||
- **MicL channel conversion to dB(L)** — the codec emits MicL as
|
||||
@@ -230,9 +496,29 @@ custom delta + RLE + variable-width codec.
|
||||
shows mic in dB(L) with ~6 dB quantization steps. Need to map
|
||||
ADC counts → dB(L) for direct comparison; likely
|
||||
`dB = 20*log10(|counts|) + offset` or similar.
|
||||
- **Walker edge cases** — SP0/SS0/SV0 don't walk the full event due
|
||||
to block-length quirks past the first few segments. Every sample
|
||||
reached is correct; the walker just needs robustness improvements.
|
||||
- **Variable-prefix segment descriptors** — 3 of the 75 ground-truth
|
||||
production events still truncate. The walk reaches a segment header
|
||||
whose channel-id field is preceded by a *variable-width* prefix (2, 4
|
||||
or 6 bytes observed; the standard tagless form always has 4), carrying
|
||||
an `01 00` marker instead of `02 00`. The marker is **not** simply an
|
||||
anchor count — `01 00` records appear with both 2- and 4-byte anchor
|
||||
fields in the same file. Examples: `BE12599/N599LPNB.JF0W` @1155,
|
||||
`BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
|
||||
(The series-3 histogram codec was fixed 2026-08-25 — see below.)
|
||||
|
||||
- ~~**Micromate (UM-series) IDF decode is ~1000× low**~~ — FIXED 2026-09-10.
|
||||
`UM11402_20260406130113.IDFW` now decodes Tran 1.1168 / Vert 4.3220 /
|
||||
Long 0.9135, matching the device report exactly. Root cause was the
|
||||
body-offset search landing inside a record header plus the unhandled
|
||||
`00 00` record mode, not anything UM-specific.
|
||||
- ~~**Thor IDF per-count LSB**~~ — RESOLVED 2026-09-10. The 0.983 ratio was
|
||||
exactly `0.0003 / 0.000310308`. Thor's geo LSB is **0.000310308 in/s per
|
||||
count** (full scale 10.0 in/s = 32226.05 counts), pinned to ±6e-11 by
|
||||
intersecting 991,415 rounding constraints from Thor's own exports and
|
||||
corroborated by the ±full-scale seed (`±32226`) left in unwritten IDFH
|
||||
interval slots. Series-3's 32000-count scale does **not** carry over.
|
||||
Note `10.0/32226` is very slightly wrong — see
|
||||
`docs/idf_protocol_reference.md`.
|
||||
|
||||
### Decoded sample counts (across the fixture bundle)
|
||||
|
||||
@@ -264,6 +550,14 @@ then `decoded_to_adc_counts()` to scale to int16 ADC counts (geos × 16;
|
||||
mic pass-through). The `.h5` sidecars SFM produces now contain
|
||||
correct samples for any event without walker edge cases.
|
||||
|
||||
**Geo full scale is 32000 ADC counts, NOT 32768** (fixed 2026-08-25).
|
||||
One decoder unit = 16 ADC counts = exactly 0.005 in/s, so
|
||||
`10.000 in/s / (0.005/16)` = 32000. Consumers must use
|
||||
`sfm.event_hdf5._GEO_INT16_FS` / `event_file_io._GEO_INT16_FS` (both
|
||||
32000). Dividing by 32768 reads every geophone sample 2.34% low —
|
||||
that was a live bug in both modules until 2026-08-25. Mic is
|
||||
unaffected (it back-solves its scale from the device-reported peak).
|
||||
|
||||
The original int16 LE decoder is preserved as
|
||||
`_decode_a5_waveform_LEGACY` for reference but is not called.
|
||||
|
||||
@@ -1640,4 +1934,4 @@ body) because writing a dial string may require DLE escaping for embedded contro
|
||||
|
||||
To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern
|
||||
in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes
|
||||
inside write frame data (the naive parser terminates early at the escaped `0x03`). | ||||