From 8e808b09d41fa9dbe1fb1752b9eddf7f74947fa3 Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 28 Aug 2026 22:21:47 +0000 Subject: [PATCH] =?UTF-8?q?chore(release):=20v0.27.0=20=E2=80=94=20decoder?= =?UTF-8?q?=20verified=20at=20scale;=20offset=20investigation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bumps pyproject, TOOL_VERSION, README and CLAUDE.md to 0.27.0. sfm/server.py now derives its version from TOOL_VERSION (c8c4ec2), so that constant is the single source of truth for the service version and the sidecar stamp alike. What ships: - histogram partial-final-block fix (4 files recovered, 0 regressed) - interval-based find_twins matching (terra-view #102 sub-task 2) - /health no longer reports a hard-coded 0.1.0 - 793 NUL bytes stripped from CLAUDE.md (made grep skip it as binary) - docs/offset_investigation.md, and the offset detectors - scratch/verify_against_ascii.py Verification: the series-3 codec now decodes 14,338 / 14,338 archive pairs exactly against their Blastware ASCII exports (1,249 waveform + 13,089 histogram, 45 units, back to 2018) — 11x the ground truth the prod store carried, and it supersedes the old "per-sample on 11% of files" caveat. Independent check on the scale: 19,244 healthy channel-events sit at a pre-trigger floor of exactly 0.000 (62.7%), 94.5% within one quantisation unit, median +0.0000. No zero-point bias in the decoder. ⚠ TOOL_VERSION moved, so the next prod backfill regenerates the whole store (~2 hours on the NAS). That is intended — it is what publishes the 4 recovered histograms — but it is not a no-op; schedule it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog --- CHANGELOG.md | 41 ++++++++++++++++++++++++++++++++ CLAUDE.md | 44 ++++++++++++++++++++++------------- README.md | 2 +- minimateplus/event_file_io.py | 2 +- pyproject.toml | 2 +- 5 files changed, 72 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dd3626e..591a5de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,16 @@ All notable changes to seismo-relay are documented here. ## [Unreleased] +--- + +## v0.27.0 — 2026-08-28 + +**Per-sample decoder verification at scale, plus the offset investigation.** +The series-3 codec is now verified sample-by-sample against **14,338** preserved +Blastware ASCII exports — 1,249 waveform and 13,089 histogram, spanning 45 units +and files back to 2018. That is 11x the ground truth the production store +carried, and it found one real codec bug (below). + ### Fixed - **Sub-minute histograms with a partial final block decoded to nothing** (`histogram_codec.detect_multi_interval_stride`). The stride search confirmed @@ -43,6 +53,37 @@ All notable changes to seismo-relay are documented here. same-serial histogram). `window_seconds` is retained but ignored. Fixes terra-view #102 sub-task 2. +- **`/health` reported a hard-coded `0.1.0`** instead of the real service version. + `sfm/server.py` now derives its version from `minimateplus.event_file_io.TOOL_VERSION`, + making that constant the single source of truth for the service version and the + sidecar stamp alike — one place to bump at release. + +- **`CLAUDE.md` had 793 NUL bytes appended** after its last line, which made `grep` + treat the file as binary and silently skip it. Present since at least v0.21.0. + Stripped. + +### Added +- **`docs/offset_investigation.md`** — a dated journal of the "offset" hardware + fault: base rate, detector design, per-unit case files, ruled-out hypotheses + (each kept with the evidence that killed it), and Instantel's own autozero + procedure with its 2027–2069 acceptance window. +- **`scratch/verify_against_ascii.py`** — decodes a corpus of BW binaries and + diffs every sample against the paired `_ASCII.TXT`. Includes a saturation + carve-out: BW clamps clipped events to the range maximum and writes `OORANGE`, + while the decoder faithfully reports counts past nominal full scale. +- **`scratch/offset_scan3.py`** — offset detector. Measures the resting floor in + the *pre-trigger* window (definitionally quiet) and requires it to hold across + pre / middle / end. Result: **5 of 45 units (11%)**, stable across a 2x + threshold range. Supersedes `offset_scan.py` and `offset_scan2.py`, both kept + as the reasoning trail. + +### Verified +- **19,244 healthy channel-events sit at a pre-trigger floor of exactly 0.000 + (62.7%), 94.5% within ±1 quantisation unit, median +0.0000.** No systematic + zero-point bias in the decoder — an independent confirmation of the + 32000-count geo full scale, arrived at from a different direction than the + ASCII sample comparisons. + --- ## v0.26.0 — 2026-08-27 diff --git a/CLAUDE.md b/CLAUDE.md index c9a05e0..2e3ae32 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,21 +2,24 @@ Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem -(Sierra Wireless RV50 / RV55). Current version: **v0.26.0**. +(Sierra Wireless RV50 / RV55). Current version: **v0.27.0**. --- -## Where things stand (updated 2026-08-27) +## Where things stand (updated 2026-08-28) Read this first when picking the project back up. -- **Series-3 decode is correct and verified.** All 11,603 series-3 binaries in - the prod snapshot pass every check (channel lengths, peaks vs the device's - own reported PPV, nothing above full scale, length vs declared record time). - Ground truth: 1,211/1,211 histograms exact per-interval and 75/75 waveform - sample counts exact against preserved Blastware ASCII exports. - ⚠ That is per-sample proof on 11% of files and peak-only consistency on the - other 89% — see `docs/instantel_protocol_reference.md` §7.6.1. +- **Series-3 decode is verified per-sample at scale (v0.27.0).** The full DL2 + archive decodes **14,338 / 14,338** paired files exactly against their + preserved Blastware ASCII exports — 1,249 waveform + 13,089 histogram, 45 + units, files back to 2018. That is 11x the ground truth the prod store + carried, and it supersedes the old "per-sample on 11%, peak-only on 89%" + caveat. Harness: `scratch/verify_against_ascii.py` (note its saturation + carve-out — BW clamps clipped events, the decoder reports true counts). + Independent corroboration of the 32000-count scale: 19,244 healthy + channel-events sit at a pre-trigger floor of exactly 0.000 (62.7%), 94.5% + within ±1 quantisation unit, median +0.0000 — no zero-point bias. - **Series-4 (Thor / Micromate) is NOT verified.** UM-series sits at ~48% against device peaks with a ~1.7% systematic bias and a near-zero tail. Thor IDFW is pinned to `decode_waveform_legacy` deliberately. @@ -24,14 +27,23 @@ Read this first when picking the project back up. (= 10.0/1.25) units discrepancy; `scripts/backfill_sidecars.py --force` also inserts DB rows for store files that have none (one-time per store) and the dry-run does not report that count. -- **After any codec change, regenerate the store** — `backfill_sidecars.py - --force` then `backfill_event_shape.py`, DB backup first. Stored `.h5` files - do not update themselves. +- **After any codec change, regenerate the store** — `backfill_sidecars.py` + then `backfill_event_shape.py`, DB backup first. Stored `.h5` files do not + update themselves. No `--force` needed as long as `TOOL_VERSION` was bumped + (it gates regeneration). ⚠ On the office NAS this takes **~2 hours** + (~1.5 files/sec vs 85/sec on the dev box — gzip-4 in `sfm/event_hdf5.py` + against a Synology CPU). Budget it up front. + **v0.27.0 owes prod a backfill:** the partial-final-block fix recovers 4 + histograms that are still empty in the store. - **The "offset" hardware fault has its own journal** -- - `docs/offset_investigation.md`. Base rate settled at 5-6 of 45 units - (11-13%) across the DL2 archive; Instantel's own autozero procedure and - its 2027-2069 go/no-go window are recorded there. Best open lead is - `SUB 0x0E` (unimplemented), which may carry those numbers. + `docs/offset_investigation.md`. **5 of 45 units (11%)**, and the fault is + **persistent** — it stays until the geophone is serviced. Detect it with + `scratch/offset_scan3.py`: the resting floor in the **pre-trigger** window, + required to hold across pre/middle/end. Never score only the dominant-peak + axis and never use the mean — both produce false recoveries (see the + retraction banner in the journal). Instantel's autozero procedure and its + 2027-2069 acceptance window are recorded there too. Best open lead is + `SUB 0x0E` (unimplemented), which may carry those very numbers. When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document diff --git a/README.md b/README.md index 02f7644..4cb9676 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# seismo-relay `v0.26.0` +# seismo-relay `v0.27.0` A ground-up replacement for **Blastware** — Instantel's aging Windows-only software for managing seismographs. Supports both the **MiniMate Plus diff --git a/minimateplus/event_file_io.py b/minimateplus/event_file_io.py index b0393d8..09a5bf1 100644 --- a/minimateplus/event_file_io.py +++ b/minimateplus/event_file_io.py @@ -50,7 +50,7 @@ SIDECAR_KIND = "sfm.event" # bumped without a `pip install` re-run — leading to confusing stale # version stamps in sidecars. Bump this constant and CHANGELOG.md # together at release time. -TOOL_VERSION = "0.26.0" +TOOL_VERSION = "0.27.0" try: # Best-effort: prefer the installed metadata when it's NEWER than the diff --git a/pyproject.toml b/pyproject.toml index 24a4e1b..6153c0f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [project] name = "seismo-relay" -version = "0.26.0" +version = "0.27.0" description = "Python client and REST server for MiniMate Plus seismographs" requires-python = ">=3.10" dependencies = [