From 1daf693b3209fb599a0cf325093a8e95b44fa8be Mon Sep 17 00:00:00 2001 From: serversdown Date: Sat, 5 Sep 2026 02:08:36 +0000 Subject: [PATCH 1/4] =?UTF-8?q?feat(offset):=20scan=20the=20histogram=20co?= =?UTF-8?q?rpus=20=E2=80=94=20the=20other=2090%=20of=20the=20archive?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit offset_scan3.py covers only waveforms (6,577 unique binaries). The archive also holds 63,535 unique histograms, which the pre-trigger method cannot touch: a histogram carries no samples, only a per-interval per-channel peak. scratch/offset_hist_scan.py scans them — 63,505/63,535 decoded (99.95%), 43 units, 77.9M intervals. It emits every candidate floor statistic per (file, channel) rather than deciding anything, so thresholds get calibrated against the waveform ground truth instead of guessed. Journal §8b records the outcome. What survives is a site-quiet-gated cross-channel differential that independently confirms BE18438|Vert and BE9558|Tran+Long with a clean 2.5x separation gap and 0.037% day-level false alarm, threshold-insensitive across a 2.3x span — the first operating point in this investigation to pass that test cleanly. What it does not do, recorded just as plainly: it finds 2 of the 5 confirmed units, not 5. DC leakage into the interval peak is bimodal (0.9 on BE18438, 0.02 on BE12599), so a negative histogram result is not evidence of health. Per-channel attribution is not established (channel-scramble p = 0.769) and timing resolves to ~a month, not a day. Two dead ends buried for good: the absolute floor is retired (66% of its discrimination is a day/site confound), and zero-fraction is structurally impossible — the device clamps every interval peak at >= 1 A/D count. Two findings independent of the histograms: - offset_scan3's spread<=0.02 gate discards 18.8% of rows with |pre|>=0.025, concentrated on 41 unit-channels currently labelled clean; 4 would be sustained positives without it. The fleet label is three-state, not two. - The waveform corpus observes ~7% of the days a unit was deployed. BE10895 is reclassified from transient to a genuine Vert fault of a different subtype: 49.4% single-axis-dominant events, the highest in the fleet, all on Vert. The other six marginal units are clean. Not done: the 11 thin-coverage units were not screened, and no completeness audit was run. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog --- docs/offset_investigation.md | 172 ++++++++++++++++++++++++++++ scratch/offset_hist_scan.py | 215 +++++++++++++++++++++++++++++++++++ 2 files changed, 387 insertions(+) create mode 100644 scratch/offset_hist_scan.py diff --git a/docs/offset_investigation.md b/docs/offset_investigation.md index 20da227..537691c 100644 --- a/docs/offset_investigation.md +++ b/docs/offset_investigation.md @@ -58,6 +58,14 @@ Companion material: sensor-check failures. Do not try to use it as a screen. - **Cause is still unsettled.** Instantel's autozero fixes the minority of cases; the rest are hardware. We cannot yet tell which is which remotely. +- **The histogram corpus (63,535 files, 9.7x the waveforms) is now scanned too** — + see §8b. It independently confirms BE18438 and BE9558 with a clean 2.5x + separation, but detects only **2 of the 5** confirmed units, cannot attribute a + channel, and resolves time to ~a month. **A negative histogram result is not + evidence of health** — DC leakage into the interval peak varies 45x between units. +- **`offset_scan3.py` has a label defect** (§8b): its spread gate discards 18.8% of + high-|pre| rows onto units currently counted as clean. Re-cut before quoting any + precision number again. - **Best open lead:** `SUB 0x0E` (channel sensor data, 8 channels × 10 bytes, unimplemented) may carry the very numbers Instantel says to check against **2027–2069**. Untested. @@ -495,6 +503,165 @@ doubled two reported figures before it was caught. --- +## 8b. The histogram corpus — the other 90% of the archive (2026-09-04) + +Every result above §8 comes from **waveform** files. `offset_scan3.py` filters on +`\.[A-Za-z0-9]{2}0[Ww]$`, so the corpus it scanned is 6,577 unique binaries. The +archive also holds **63,535 unique histograms** — 9.7x more files — which the +pre-trigger method cannot touch, because a histogram carries no samples: only a +per-interval, per-channel peak and half-period. + +`scratch/offset_hist_scan.py` scans them. **63,505 of 63,535 decoded (99.95%), +43 units, 77.9M intervals.** Two of the 45 units have no histograms at all. +Output: `/home/serversdown/dl2-archive/offset_hist.csv` (190,515 channel-rows). + +### The premise, and how far it actually holds + +A histogram file is hours of continuous monitoring, so most of its intervals are +definitionally quiet, and a channel parked off zero cannot report a peak below +its own displacement. The signal is real — two within-unit contrasts, siblings +unmoved in both: + +| unit | channel | in-episode floor | outside | waveform \|pre\| same window | +|---|---|---|---|---| +| BE18438 | Vert | 0.0350 | 0.0050 | +0.18 .. +0.37 | +| BE12599 | Tran | 0.0250 | 0.0050 | +0.03 .. +0.49 | + +But the **leakage from a waveform pedestal into the histogram floor is bimodal, +not merely partial**: measured ratio ~0.9 on BE18438 Vert, ~0.7 on BE9558, +**~0.02 on BE12599** — two orders of magnitude on one instrument. The device +evidently measures each interval peak against a running baseline, and how much +DC survives that varies per unit. **Consequence: a negative histogram result +carries almost no information.** Do not read "clean in the histograms" as clean. + +### The detector that survived + + dmin(file, ch) = min[ch] - min over the other two geo channels, SAME file + gates (both hard): n_intervals >= 60 AND mic_p5 <= 5 raw counts + day statistic: median of dmin over that day's qualifying files + flag day at dmin >= 0.020 in/s (4 A/D counts) + episode at >= 3 CONSECUTIVE observed days + +**Result: BE18438|Vert, BE9558|Tran, BE9558|Long.** Threshold-insensitive — +the journal's own test for a real signal against a tuned one — and this is the +first operating point in the investigation that passes it cleanly. The identical +answer holds across: statistic `min` or `p5`; length gate 10/30/60/120/300; mic +gate 3/5/8/10; threshold 0.015–0.035 (a 2.3x span); persistence K = 2,3,4,5,7. + +Separation, ranked by highest floor sustained over 3 consecutive gated days +across all 135 unit-channels: + +| unit-channel | best3 | +|---|---| +| BE18438 Vert | 0.1650 | +| BE9558 Long | 0.0350 | +| BE9558 Tran | 0.0250 | +| *(2.5x gap)* | | +| BE7145 Tran | 0.0100 | +| entire rest of fleet | <= 0.0050 (one quantisation count) | + +Day-level false alarm: **37 of 99,432 gated unit-channel-days = 0.037%.** + +### What it does NOT do — read this before trusting it + +- **It finds 2 of the 5 confirmed units, not 5.** The site-quiet gate is what + makes it work and it is also what costs BE11529 and BE12599. BE11529's + four-day single-axis ramp (Tran 0.025 -> 0.055, both siblings pinned at 0.005) + is the most offset-shaped thing in the corpus outside the two detections, and + the gate discards it. +- **The positive class is two units.** Every threshold here is fitted to + BE18438 and BE9558, which contribute 22 of the 37 flagged days in the entire + corpus. No cross-validation is possible at n=2. +- **Per-channel attribution is NOT established.** Rotating the three geo channel + labels within each file — preserving every value, file and day, destroying + only channel identity — reproduces the episode *count* with p = 0.769 and the + label agreement at p = 0.038–0.077. Report a **unit and a window**; do not + name a geophone axis on the strength of this detector alone. +- **Timing resolution is ~1 month, not ~1 day.** A 30-day label shift still + scores 2 of 9 episode hits; the signal dies only past ~60 days. The day-level + series look far crisper than they are. +- **Ground truth here is a sibling detector, not a service record.** Agreement + between the two corpora is corroboration of a shared method. Nothing in this + section has been checked against an actual repair, calibration or RMA. + +### Dead ends — keep these dead + +- **Absolute floor (min / p1 / p5 / p10 / p25, thresholded alone) — RETIRED.** + Not fleet-comparable and mostly not about the channel. Scoring each cell using + *only the other two channels* — a statistic containing zero information about + the suspect channel — reaches AUC 0.746 against the same labels, versus 0.872 + for the absolute floor itself. **66% of its apparent discrimination is "that + day was noisy at that site."** Interval size alone moves its p99 7x (0.0350 at + 1 min vs 0.0050 at 2 s). And of all files with any channel above 0.025, 56.5% + have **all three** channels above it — common-mode, i.e. the wrong physics. +- **Zero-fraction — STRUCTURALLY IMPOSSIBLE, not merely weak.** The device never + reports a zero histogram interval peak. The value is a max over hundreds of + samples of a channel that always carries at least 1 count of noise, so it is + clamped at 1 A/D count (0.005 in/s). There is no zero to count. +- **Interval size, sample rate, geo range, firmware — refuted as confounds for + the differential.** All four are *file-level scalars*: they move all three geo + channels together, so they cannot produce a single-channel lift and the + within-file differential is immune to them by construction. Geo range is + identical across the three geo channels in **63,535 of 63,535** binaries. + (Interval size remains fatal to the *absolute*-floor version, above.) + +### Two findings that are independent of the histogram detector + +**1. `offset_scan3.py`'s `spread <= 0.02` gate is discarding real signal.** +It rejects **113 of the 600 channel-rows with |pre| >= 0.025 (18.8%)**, and the +rejections are not random — 92 of them fall across 41 unit-channels currently +labelled NEGATIVE. Four would become sustained positives under an +amplitude-only >=3-consecutive rule: **BE12599|Long (run of 8), BE18003|Vert +(4), BE10895|Vert (3), BE12844|Tran (3).** Until this is re-cut, the fleet label +is **three-state — POSITIVE / NEGATIVE / SPREAD-REJECTED(unknown)** — and the +third state should be excluded from both TP and FP counts rather than silently +scored as healthy. Every precision figure computed against the two-state label, +in this section and in §3, is affected. + +**2. The waveform corpus sees ~7% of the days a unit was deployed.** 2,627 +(unit, day) observations against the histogram corpus's 35,105 — 13.4x — with a +per-unit median ratio of 0.070. BE12599, a confirmed unit, is waveform-observed +on 39 of its 1,666 histogram-observed days (**2.3%**). Any statement of the form +"the fault was absent before date X" that rests on waveform coverage alone is +much weaker than its event count suggests. + +### BE10895 — reclassified (see also §4) + +Previously dismissed as a transient. The histogram record shows its **Vert** +quiet-minute floor at 0.005 on 62/62 qualifying files from 2023-07-07, then +0.010–0.015 on 48/58 files from 2023-08-03 to 08-27, while Tran moves on 2/58 +and Long on 9/58 and the site mic floor never leaves 1–3 counts. Independently, +**42 of its 85 waveform events (49.4%) are single-axis-dominant** — one geo peak +>= 10x both siblings and >= 0.05 in/s — the **highest rate in the 45-unit +fleet** (BE13117 36.1%, BE18438 29.4%), and **100% of it on Vert**. Vert +excursions of 0.1–1.5 in/s with Tran/Long at 0.005–0.035 are not ground motion. + +This is a genuine Vert-channel hardware fault, but **not the classic pedestal** — +the differential is only one A/D count. Caveat: its entire histogram record is a +single 52-day deployment ending 2023-08-27, so nothing says whether it +persisted, was serviced, or resolved. + +The other six marginal units — BE11007, BE17354, BE18004, BE18104, BE9557, +BE18003 — are **clean**. All seven cap at +0.005 to +0.007 (one A/D count) +lifetime under the quiet-site gate, against +0.175 for BE18438 Vert and +0.062 +for BE9558 Long. Three individual waveform flags fall in windows with **zero** +histogram coverage and are NO-DATA, not clean: BE18004|Tran 2024-10-16, +BE9557|Tran 2021-06-28, BE9557|Vert 2025-06-12. + +### Still open in this section + +- **The 11 thin-coverage units were not screened** (BE10202, BE11462, BE13779, + BE15760, BE15957, BE16754, BE16758, BE8081, BE8626, BE9229, BE9887 — each + under 20 waveform events, several with hundreds of histograms). This is the + population most likely to hold a previously unknown offset, and it is the one + slice of the plan that did not run. BE11462 was incidentally scored clean by + the full-archive pass; BE10202 has no histogram files at all. +- **No completeness audit was run** over the above. +- Re-cutting the ground truth three-state (finding 1) and re-scoring everything + against it. + +--- + ## 9. Chronology | date | event | @@ -512,3 +679,8 @@ doubled two reported figures before it was caught. | 2026-08-28 | Bimodality established; sensor check proven **blind** to offsets; `SUB 0x0E` identified as the best open lead. | | 2026-08-28 | **v1 detector retracted.** Brian challenged the "come and go" finding against field experience. Two flaws found: dominant-axis-only scoring and mean-instead-of-median. Corrected detector shows persistent pedestals on **8 of 45 units**, and the gaps are service windows. | | 2026-08-28 | **Detector v3 (Brian's method):** pre-trigger floor + pre/mid/end consistency. Healthy channels proven to sit at 0.000 +/-1 unit (94.5%), confirming no decoder zero-point bias. Final: **5 of 45 units (11%)**, threshold-insensitive. | +| 2026-09-04 | **Histogram corpus scanned** — 63,505 of 63,535 files, 43 units, 77.9M intervals (9.7x the waveform corpus). `scratch/offset_hist_scan.py`. | +| 2026-09-04 | Absolute-floor statistic **retired**: 66% of its discrimination is a day/site confound (other-channels-only AUC 0.746 vs 0.872). Zero-fraction shown **structurally impossible** — the device clamps every interval peak at >= 1 count. | +| 2026-09-04 | Site-quiet-gated cross-channel differential established: **BE18438 Vert, BE9558 Tran+Long**, threshold-insensitive over a 2.3x span. Finds only **2 of the 5** confirmed units — leakage into the histogram floor is bimodal (0.9 to 0.02), so a negative result carries almost no information. Per-channel attribution **not** established (channel-scramble p = 0.769). | +| 2026-09-04 | **BE10895 reclassified** from transient to a genuine Vert fault of a different subtype — 49.4% single-axis-dominant events, the highest in the fleet, 100% on Vert. The other six marginal units are clean. | +| 2026-09-04 | **Defect found in `offset_scan3.py`**: its `spread <= 0.02` gate discards 18.8% of rows with \|pre\| >= 0.025, concentrated on 41 negative unit-channels; 4 would be sustained positives without it. The fleet label is three-state, not two. | diff --git a/scratch/offset_hist_scan.py b/scratch/offset_hist_scan.py new file mode 100644 index 0000000..1a2b9a1 --- /dev/null +++ b/scratch/offset_hist_scan.py @@ -0,0 +1,215 @@ +#!/usr/bin/env python3 +"""Offset detector — HISTOGRAM corpus (the other 90% of the archive). + +`offset_scan3.py` measures the pre-trigger floor in *waveform* samples. That +covers 6,577 of the archive's 70,112 unique series-3 files; the remaining +63,535 are **histograms**, which carry no samples — only a per-interval, +per-channel peak + half-period. So the pre-trigger method cannot run on them. + +The histogram analogue of "the resting floor" is the **low percentile of the +per-interval peaks**. A histogram file is typically hours of continuous +monitoring, so the great majority of its intervals are definitionally quiet; +the bottom of that distribution is what the channel reads when nothing is +happening. A healthy channel bottoms out at 0.000-0.005 in/s. A channel +parked off zero cannot report a peak below its own displacement, so its floor +is pinned up. + +⚠ The DC leakage into the histogram peak is PARTIAL. Measured within-unit +against episodes already established from the waveform scan: + + BE18438 Vert in-episode 0.0350 vs 0.0050 outside (waveform pre = +0.18..+0.37) + BE12599 Tran in-episode 0.0250 vs 0.0050 outside (waveform pre = +0.03..+0.49) + +so the device's per-interval peak is evidently measured against a running / +AC-coupled baseline that removes most, but not all, of the DC. The residual +is real and channel-specific, but the margin is ~5 quantisation counts rather +than the ~70 the waveform detector enjoys. Do not carry the waveform +detector's 0.025 in/s floor across unexamined — calibrate on the CSV. + +Because the absolute floor also moves with site noise (traffic, wind, a +generator), the statistic that matters most is the **cross-channel +differential**: a channel's floor minus the quietest of the other two geo +channels in the same file. Site noise lifts all three together and cancels; +a DC offset lifts one. + +This script does not decide anything. It emits every candidate statistic per +(file, channel) so thresholds can be calibrated against the waveform-derived +ground truth in `offset_v3.csv` rather than guessed. + +Usage: + python scratch/offset_hist_scan.py --dir /home/serversdown/dl2-archive/files \ + --out /home/serversdown/dl2-archive/offset_hist.csv --jobs 4 +""" +from __future__ import annotations + +import argparse +import csv +import datetime +import logging +import re +import statistics +import sys +from concurrent.futures import ProcessPoolExecutor, as_completed +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) +from minimateplus.event_file_io import read_blastware_file # noqa: E402 + +GEO = ("Tran", "Vert", "Long") +K = 10.0 / 32000.0 # ADC count -> in/s (see CLAUDE.md: full scale 32000) +_HIST = re.compile(r"\.[A-Za-z0-9]{2}0[Hh]$") +_STEM = re.compile(r"^([B-Z])(\d{3})") +_B36 = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" + + +def serial_of(name: str) -> str: + """`P036L318.C80H` -> `BE14036`. See CLAUDE.md, serial encoding.""" + m = _STEM.match(name) + if not m: + return "?" + return f"BE{(ord(m.group(1)) - ord('B')) * 1000 + int(m.group(2))}" + + +def stem_time(name: str): + """Decode the filename's base-36 timestamp. Epoch 1985-01-01, 1296 s/tick. + + Preferred over the file's own footer timestamp only because it costs + nothing; the caller falls back to the decoded event when this fails. + """ + try: + base, ext = name.rsplit(".", 1) + n = 0 + for c in base[4:8].upper(): + n = n * 36 + _B36.index(c) + ab = _B36.index(ext[0].upper()) * 36 + _B36.index(ext[1].upper()) + return datetime.datetime(1985, 1, 1) + datetime.timedelta(seconds=n * 1296 + ab) + except Exception: + return None + + +def _pct(sorted_vals, q): + """Nearest-rank percentile on an already-sorted list.""" + if not sorted_vals: + return None + i = min(len(sorted_vals) - 1, max(0, int(len(sorted_vals) * q / 100.0))) + return sorted_vals[i] + + +def scan(path_str: str): + logging.disable(logging.WARNING) # per-worker: the codec warns on undecodables + p = Path(path_str) + try: + ev = read_blastware_file(p) + except Exception: + return None + s = ev.raw_samples or {} + if not any(s.get(c) for c in GEO): + return None + + ts = stem_time(p.name) or ev.timestamp + stamp = "" + if ts is not None: + stamp = (f"{ts.year:04d}-{ts.month:02d}-{ts.day:02d}T" + f"{ts.hour:02d}:{ts.minute:02d}:{ts.second:02d}") + + # Per-channel floor candidates, in in/s. + stats = {} + for ch in GEO: + v = sorted(s.get(ch) or []) + if not v: + continue + stats[ch] = { + "n": len(v), + "min": v[0] * K, + "p1": _pct(v, 1) * K, + "p5": _pct(v, 5) * K, + "p10": _pct(v, 10) * K, + "p25": _pct(v, 25) * K, + "med": statistics.median(v) * K, + "peak": v[-1] * K, + "zeros": sum(1 for x in v if x == 0) / len(v), + } + if len(stats) < 2: # need at least one sibling channel for the differential + return None + + # Mic floor as a site-noise proxy (raw counts; the dB conversion is not + # needed — only its relative movement matters here). + mic = sorted(s.get("MicL") or []) + mic_p5 = _pct(mic, 5) if mic else "" + + rows = [] + for ch, st in stats.items(): + others = [stats[o]["p5"] for o in stats if o != ch] + rows.append({ + "serial": serial_of(p.name), + "timestamp": stamp, + "filename": p.name, + "channel": ch, + "n_intervals": st["n"], + "min": round(st["min"], 4), + "p1": round(st["p1"], 4), + "p5": round(st["p5"], 4), + "p10": round(st["p10"], 4), + "p25": round(st["p25"], 4), + "median": round(st["med"], 4), + "peak": round(st["peak"], 4), + "frac_zero": round(st["zeros"], 4), + # the site-noise-cancelling statistic: this channel's floor above + # the quietest sibling geo channel in the same file + "diff_p5": round(st["p5"] - min(others), 4), + "mic_p5": mic_p5, + }) + return rows + + +COLS = ["serial", "timestamp", "filename", "channel", "n_intervals", + "min", "p1", "p5", "p10", "p25", "median", "peak", "frac_zero", + "diff_p5", "mic_p5"] + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--dir", required=True) + ap.add_argument("--out", required=True) + ap.add_argument("--jobs", type=int, default=4) + ap.add_argument("--limit", type=int, default=0, help="stop after N files (smoke test)") + a = ap.parse_args() + + # Dedupe by basename — the DL2 export keeps a byte-identical `Sent/` + # mirror of its root, which doubled two figures before it was caught. + seen, files = set(), [] + for q in sorted(Path(a.dir).rglob("*")): + if q.is_file() and _HIST.search(q.name) and q.name not in seen: + seen.add(q.name) + files.append(str(q)) + if a.limit: + files = files[:a.limit] + print(f"unique histogram binaries: {len(files)}", flush=True) + + rows, undecodable = [], 0 + with ProcessPoolExecutor(max_workers=a.jobs) as ex: + futs = [ex.submit(scan, f) for f in files] + for i, fut in enumerate(as_completed(futs), 1): + r = fut.result() + if r: + rows.extend(r) + else: + undecodable += 1 + if i % 5000 == 0: + print(f" {i}/{len(files)}", flush=True) + + with open(a.out, "w", newline="") as fh: + w = csv.DictWriter(fh, fieldnames=COLS) + w.writeheader() + w.writerows(rows) + + files_ok = len({r["filename"] for r in rows}) + units = len({r["serial"] for r in rows}) + ivals = sum(r["n_intervals"] for r in rows) // 3 + print(f"\ndecoded {files_ok}/{len(files)} files " + f"({undecodable} undecodable), {units} units, ~{ivals/1e6:.1f}M intervals") + print(f"wrote {a.out} ({len(rows)} channel-rows)") + + +if __name__ == "__main__": + main() From 9982938b0b7d3f5fd792b31a0b92d400f21056e7 Mon Sep 17 00:00:00 2001 From: serversdown Date: Sun, 6 Sep 2026 07:48:54 +0000 Subject: [PATCH 2/4] fix(offset): read the real serial from the file body, not "BE" + the number MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The BW filename encodes only the serial NUMBER — `<3 digits>` where letter = chr(ord('B') + serial // 1000), so `L895…` decodes to 10895. The two-letter family prefix is not in the filename at all, and every offset scanner synthesized it as f"BE{num}". Four of the 43 archive units are BA, not BE. Their binaries say so plainly: BA9229, BA10060, BA10895, BA15957. Brian caught BA10895 by recognising that no such unit as BE10895 exists. serial_of() now reads the serial string out of the file body and falls back to the old synthesis only when no matching string is found. No analysis changes: grouping was by the numeric part, which was always correct, and no unit number maps to more than one serial (checked across all 43). The same assumption is live in two production sites and is NOT touched here, because fixing ingest renames rows a running store and Terra-View already reads them: - sfm/waveform_store.py:870 `return f"BE{serial_num}"` on import - minimateplus/client.py:2538 `raw_data.find(b"BE")` in the monitor-log partial-record decode, which yields serial=None on a BA unit Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog --- scratch/offset_hist_scan.py | 27 +++++++++++++++++++++++---- scratch/offset_scan3.py | 30 ++++++++++++++++++++++++++---- 2 files changed, 49 insertions(+), 8 deletions(-) diff --git a/scratch/offset_hist_scan.py b/scratch/offset_hist_scan.py index 1a2b9a1..a43a097 100644 --- a/scratch/offset_hist_scan.py +++ b/scratch/offset_hist_scan.py @@ -62,12 +62,31 @@ _STEM = re.compile(r"^([B-Z])(\d{3})") _B36 = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ" -def serial_of(name: str) -> str: - """`P036L318.C80H` -> `BE14036`. See CLAUDE.md, serial encoding.""" +_SERIAL_RE = re.compile(rb"\b([A-Z]{2}\d{3,6})\b") + + +def serial_of(name: str, path=None) -> str: + """Real serial for a BW file. + + The filename encodes only the NUMBER: `<3 digits>` where + letter = chr(ord('B') + serial // 1000). The two-letter family prefix + ("BE", "BA", ...) is **not** in the filename, so it must be read out of + the file body. Four units in the DL2 archive are BA, not BE — assuming + "BE" mislabels BA9229, BA10060, BA10895 and BA15957. + """ m = _STEM.match(name) if not m: return "?" - return f"BE{(ord(m.group(1)) - ord('B')) * 1000 + int(m.group(2))}" + num = (ord(m.group(1)) - ord("B")) * 1000 + int(m.group(2)) + if path is not None: + try: + for s in _SERIAL_RE.findall(Path(path).read_bytes()): + s = s.decode() + if s[2:].lstrip("0") == str(num): + return s + except Exception: + pass + return f"BE{num}" # last-resort fallback; prefix unverified def stem_time(name: str): @@ -141,7 +160,7 @@ def scan(path_str: str): for ch, st in stats.items(): others = [stats[o]["p5"] for o in stats if o != ch] rows.append({ - "serial": serial_of(p.name), + "serial": serial_of(p.name, p), "timestamp": stamp, "filename": p.name, "channel": ch, diff --git a/scratch/offset_scan3.py b/scratch/offset_scan3.py index 2c2b953..3a3cde6 100644 --- a/scratch/offset_scan3.py +++ b/scratch/offset_scan3.py @@ -28,9 +28,31 @@ from minimateplus.event_file_io import read_blastware_file GEO=("Tran","Vert","Long"); K=10.0/32000.0 _WAVE=re.compile(r"\.[A-Za-z0-9]{2}0[Ww]$"); _STEM=re.compile(r"^([B-Z])(\d{3})") -def serial_of(n): - m=_STEM.match(n) - return f"BE{(ord(m.group(1))-ord('B'))*1000+int(m.group(2))}" if m else "?" +_SERIAL_RE = re.compile(rb"\b([A-Z]{2}\d{3,6})\b") + + +def serial_of(name: str, path=None) -> str: + """Real serial for a BW file. + + The filename encodes only the NUMBER: `<3 digits>` where + letter = chr(ord('B') + serial // 1000). The two-letter family prefix + ("BE", "BA", ...) is **not** in the filename, so it must be read out of + the file body. Four units in the DL2 archive are BA, not BE — assuming + "BE" mislabels BA9229, BA10060, BA10895 and BA15957. + """ + m = _STEM.match(name) + if not m: + return "?" + num = (ord(m.group(1)) - ord("B")) * 1000 + int(m.group(2)) + if path is not None: + try: + for s in _SERIAL_RE.findall(Path(path).read_bytes()): + s = s.decode() + if s[2:].lstrip("0") == str(num): + return s + except Exception: + pass + return f"BE{num}" # last-resort fallback; prefix unverified def scan(ps): p=Path(ps) @@ -48,7 +70,7 @@ def scan(ps): mid, end = a[t:2*t], a[2*t:] if not pre or not mid or not end: continue v=[statistics.median(x)*K for x in (pre,mid,end)] - out.append({"serial":serial_of(p.name),"timestamp":stamp, + out.append({"serial":serial_of(p.name, p),"timestamp":stamp, "filename":p.name,"channel":ch, "pretrig_n": pre_n or 0, "pre":round(v[0],4),"mid":round(v[1],4),"end":round(v[2],4), From 9ceff65bfb7ceae195d317cc8600f29044067a42 Mon Sep 17 00:00:00 2001 From: serversdown Date: Sun, 6 Sep 2026 07:58:00 +0000 Subject: [PATCH 3/4] fix(sfm): read the serial family prefix from the file, enabling BlastMates MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The BW filename encodes only the serial NUMBER — `<3 digits>`, so `L895…` is 10895 and nothing more. The two-letter family prefix is not in it: "BE" is a MiniMate Plus, "BA" a BlastMate. Both are Series III and their files are byte-identical in every way that matters — all 1,493 BlastMate binaries in the DL2 archive decode through the existing codec at 100%, same four channels — so the serial string was the only thing standing between SFM and BlastMate support. Two sites synthesised the prefix and got it wrong: - waveform_store `_serial_from_bw_filename` returned f"BE{num}" on import, so a BlastMate event was filed under a unit that does not exist, silently, and Terra-View read it straight through. Split into `_serial_number_from_bw_filename` (the number, which the filename really does carry) and a new `_serial_from_bw_bytes` that reads the serial out of the body and accepts it only when its numeric part agrees with the filename. save_imported_bw now prefers hint -> body -> filename guess. Verified against real archive bytes for BA9229, BA10060, BA10895, BA15957 and BE9558/BE11529/BE18003. - client `_decode_0a_partial_header` searched for a literal b"BE" in the monitor-log partial record. On a BlastMate that returns -1 and skips the whole block, so the geo threshold went missing along with the serial. Now matches any two-letter prefix, and requires the NUL terminator — stricter than the bare two-byte search it replaces. Nothing to migrate: no BlastMate events are in prod. The archive's BA units last recorded 2018-10 (BA9229, BA15957), 2023-08 (BA10895) and 2023-11 (BA10060), and the prod backfill only reaches back to ~May 2025. 21 tests. Suite: 309 passed, same 16 pre-existing failures as at HEAD (15 missing ASCII fixtures + one peak_values assertion, all untouched here). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog --- minimateplus/client.py | 10 +++- sfm/waveform_store.py | 71 +++++++++++++++++++++---- tests/test_serial_prefix.py | 101 ++++++++++++++++++++++++++++++++++++ 3 files changed, 171 insertions(+), 11 deletions(-) create mode 100644 tests/test_serial_prefix.py diff --git a/minimateplus/client.py b/minimateplus/client.py index d82604d..49f3e83 100644 --- a/minimateplus/client.py +++ b/minimateplus/client.py @@ -30,6 +30,7 @@ from __future__ import annotations import datetime import logging +import re import struct from typing import Optional @@ -2532,10 +2533,17 @@ def _decode_0a_partial_header(raw_data: bytes, index: int, key4: bytes) -> Optio ts2 = try_ts(raw_data[ts1_end + 1:ts1_end + 1 + ts_size]) # Extract serial and geo threshold from "BE11529\0" and "Geo: X.XXX in/s\0". + # + # Match any two-letter family prefix, not a literal "BE" — a BlastMate + # reports "BA10895", and the old `find(b"BE")` returned -1 on one. That + # skipped this whole block, so the geo threshold went missing along with + # the serial. Requiring the NUL terminator in the pattern also makes the + # match stricter than the bare two-byte search it replaces. serial: Optional[str] = None geo_ips: Optional[float] = None - serial_pos = raw_data.find(b"BE") + serial_match = re.search(rb"[A-Z]{2}\d{3,6}(?=\x00)", raw_data) + serial_pos = serial_match.start() if serial_match else -1 if serial_pos >= 0: # Read null-terminated serial starting at serial_pos. null_pos = raw_data.find(b"\x00", serial_pos) diff --git a/sfm/waveform_store.py b/sfm/waveform_store.py index 44d7020..8373e8d 100644 --- a/sfm/waveform_store.py +++ b/sfm/waveform_store.py @@ -32,6 +32,7 @@ from __future__ import annotations import datetime import logging import pickle +import re import shutil from pathlib import Path from typing import Optional, Union @@ -379,8 +380,16 @@ class WaveformStore: # Resolve serial. blastware_filename derives a 4-char prefix from # the numeric serial (e.g. BE11529 → M529); we go the other way - # via the source filename if a hint wasn't given. - serial = serial_hint or _serial_from_bw_filename(source_path.name) or "UNKNOWN" + # if a hint wasn't given. The filename carries only the NUMBER, + # so read the family prefix out of the body first — a BlastMate + # ("BA") filed as "BE" is a unit that does not exist. The + # filename-only decoder stays as the last resort. + serial = ( + serial_hint + or _serial_from_bw_bytes(bw_bytes, source_path.name) + or _serial_from_bw_filename(source_path.name) + or "UNKNOWN" + ) # Use the source filename verbatim — it already encodes timestamp # + record type per BW's AB0T scheme, and we want to preserve it @@ -840,20 +849,24 @@ class WaveformStore: # ── helpers ───────────────────────────────────────────────────────────────────── -def _serial_from_bw_filename(name: str) -> Optional[str]: +def _serial_number_from_bw_filename(name: str) -> Optional[int]: """ - Reverse of `blastware_filename`'s serial-prefix encoding. + Reverse of `blastware_filename`'s serial-prefix encoding — the NUMBER only. BW filename format (V10.72): `

.` where P = chr(ord('B') + floor(serial // 1000)) and serial3 = f"{serial % 1000:03d}". Examples (from CLAUDE.md verification archive): - P036... → BE14036 H907... → BE6907 - M529... → BE11529 T003... → BE18003 + P036... → 14036 H907... → 6907 + M529... → 11529 T003... → 18003 + L895... → 10895 - Returns the inferred BE-prefix serial (e.g. "BE11529") or None when - the filename doesn't match the expected pattern. + ⚠ The filename encodes **only the number**. The two-letter family + prefix is NOT in it — "BE" is a MiniMate Plus, "BA" a BlastMate — so + the prefix has to come from the file body (`_serial_from_bw_bytes`) + or from an explicit hint. Returns None when the filename doesn't + match the expected pattern. """ if not name: return None @@ -866,5 +879,43 @@ def _serial_from_bw_filename(name: str) -> Optional[str]: if prefix_letter < "B": return None thousands = ord(prefix_letter) - ord("B") - serial_num = thousands * 1000 + int(base[1:4]) - return f"BE{serial_num}" + return thousands * 1000 + int(base[1:4]) + + +_BW_SERIAL_RE = re.compile(rb"[A-Z]{2}\d{3,6}") + + +def _serial_from_bw_bytes(data: bytes, name: str) -> Optional[str]: + """ + Read the real serial — prefix included — out of a BW file body. + + The body carries the serial as a plain ASCII string ("BE9558", + "BA10895"). We accept a candidate only when its numeric part matches + the number the filename encodes, which keeps a stray byte sequence in + the sample stream from being mistaken for a serial. + + Returns None when the filename number can't be derived or no + candidate in the body agrees with it — the caller then falls back. + """ + num = _serial_number_from_bw_filename(name) + if num is None or not data: + return None + for match in _BW_SERIAL_RE.findall(data): + candidate = match.decode("ascii", errors="replace") + if candidate[2:].lstrip("0") == str(num): + return candidate + return None + + +def _serial_from_bw_filename(name: str) -> Optional[str]: + """ + Best-effort serial from the filename alone. + + ⚠ The family prefix is a **guess** — the filename does not carry it. + "BE" is right for every MiniMate Plus but wrong for a BlastMate, whose + serials start "BA". Prefer `_serial_from_bw_bytes` whenever the file + body is at hand; this exists for callers that only have a name + (log lines, dry-run output). + """ + num = _serial_number_from_bw_filename(name) + return None if num is None else f"BE{num}" diff --git a/tests/test_serial_prefix.py b/tests/test_serial_prefix.py new file mode 100644 index 0000000..9d2d111 --- /dev/null +++ b/tests/test_serial_prefix.py @@ -0,0 +1,101 @@ +"""The BW filename encodes the serial NUMBER, never the family prefix. + +"BE" is a MiniMate Plus; "BA" is a BlastMate. Both are Series III and their +files are byte-compatible — the whole archive's 1,493 BlastMate binaries +decode through the same codec at 100% — so the only thing that distinguishes +them downstream is the serial string, and that lives in the file body. + +Synthesising the prefix as "BE" files a BlastMate under a unit that does not +exist. Four units in the DL2 archive are affected: BA9229, BA10060, BA10895 +and BA15957. +""" +from __future__ import annotations + +import pytest + +from minimateplus.client import _decode_0a_partial_header +from sfm.waveform_store import ( + _serial_from_bw_bytes, + _serial_from_bw_filename, + _serial_number_from_bw_filename, +) + + +# ── the filename gives a number, and only a number ────────────────────────── + +@pytest.mark.parametrize("name,num", [ + ("P036L318.C80H", 14036), # BE14036 + ("H907KWRK.WB0H", 6907), # BE6907 + ("M529LKIQ.G10", 11529), # BE11529 + ("T003LQ9K.OE0H", 18003), # BE18003 + ("L895K63F.GE0W", 10895), # BA10895 — a BlastMate + ("K229HGQI.XO0W", 9229), # BA9229 — a BlastMate +]) +def test_number_from_filename(name, num): + assert _serial_number_from_bw_filename(name) == num + + +@pytest.mark.parametrize("name", ["", "not_a_bw_file.bin", "AB12", "1234ABCD.XX0W"]) +def test_number_from_filename_rejects_junk(name): + assert _serial_number_from_bw_filename(name) is None + + +def test_filename_only_decoder_is_a_guess(): + """It still answers "BE" — that is why it must not be the first choice.""" + assert _serial_from_bw_filename("L895K63F.GE0W") == "BE10895" + assert _serial_from_bw_filename("M529LKIQ.G10") == "BE11529" + assert _serial_from_bw_filename("nonsense") is None + + +# ── the body carries the truth ────────────────────────────────────────────── + +def _body(serial: bytes) -> bytes: + return b"\x00" * 32 + b"STRT" + b"\xff\xfe" + serial + b"\x00Geo: 0.254 in/s\x00" + + +def test_body_wins_for_a_blastmate(): + assert _serial_from_bw_bytes(_body(b"BA10895"), "L895K63F.GE0W") == "BA10895" + + +def test_body_wins_for_a_minimate(): + assert _serial_from_bw_bytes(_body(b"BE11529"), "M529LKIQ.G10") == "BE11529" + + +def test_body_candidate_must_match_the_filename_number(): + """A serial-shaped byte run that disagrees with the filename is ignored.""" + assert _serial_from_bw_bytes(_body(b"XX99999"), "L895K63F.GE0W") is None + + +def test_body_tolerates_a_leading_zero(): + assert _serial_from_bw_bytes(_body(b"BA09229"), "K229HGQI.XO0W") == "BA09229" + + +@pytest.mark.parametrize("data,name", [ + (b"", "L895K63F.GE0W"), # no bytes + (_body(b"BA10895"), "junk.bin"), # no derivable number +]) +def test_body_returns_none_when_it_cannot_decide(data, name): + assert _serial_from_bw_bytes(data, name) is None + + +# ── the live monitor-log path ─────────────────────────────────────────────── + +def _partial_record(serial: bytes) -> bytes: + """0x2C partial record: type, prefix, two 9-byte timestamps, then ASCII.""" + ts = bytes([11, 0x10, 4, 0x07, 0xE9, 0, 16, 2, 0]) # 2025-04-11 16:02:00 + return (bytes([0x2C]) + b"\x00" * 10 + ts + ts + + b"\x00\x00\x00\x00" + serial + b"\x00Geo: 0.254 in/s\x00") + + +@pytest.mark.parametrize("serial", [b"BE11529", b"BA10895", b"UM11719"]) +def test_monitor_log_reads_any_family_prefix(serial): + entry = _decode_0a_partial_header(_partial_record(serial), 0, b"\x01\x11\x00\x00") + assert entry is not None + assert entry.serial == serial.decode() + + +def test_monitor_log_geo_threshold_survives_a_blastmate(): + """The old find(b"BE") skipped the whole block, losing geo too.""" + entry = _decode_0a_partial_header(_partial_record(b"BA10895"), 0, b"\x01\x11\x00\x00") + assert entry is not None + assert entry.geo_threshold_ips == pytest.approx(0.254) From 84bb53e1852799738a9b03e445f2f459a8e29d84 Mon Sep 17 00:00:00 2001 From: serversdown Date: Sun, 6 Sep 2026 08:00:48 +0000 Subject: [PATCH 4/4] docs(offset): relabel the four BlastMate units BA, not BE MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BA9229, BA10060, BA10895 and BA15957 were reported throughout as BE — the scanners synthesised the family prefix, which the BW filename does not carry. Corrected across the journal with a note recording why, so the mistake is legible rather than silently patched. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog --- docs/offset_investigation.md | 37 +++++++++++++++++++++++++++++------- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/docs/offset_investigation.md b/docs/offset_investigation.md index 537691c..9a6b795 100644 --- a/docs/offset_investigation.md +++ b/docs/offset_investigation.md @@ -154,8 +154,8 @@ is the discriminator — and it is what the field experience predicts. | runs of >=3 consecutive | — | 29 | | runs of 1-2 events (noise) | — | 69 | -Units with a sustained pedestal: **BE9558, BE10895, BE11007, BE11529, BE12599, -BE13117, BE18003, BE18438**. BE10895 and BE18003 were invisible to v1. +Units with a sustained pedestal: **BE9558, BA10895, BE11007, BE11529, BE12599, +BE13117, BE18003, BE18438**. BA10895 and BE18003 were invisible to v1. **The affected channel is most often Vert**, which v1 got wrong — it named whichever axis had the largest peak. BE13117 and BE18438 are both Vert faults. @@ -226,7 +226,7 @@ signal from a tuned one: **BE9558, BE11529, BE12599, BE13117, BE18438.** -Unchanged across a 2x threshold range. BE11007 and BE10895 drop out — the +Unchanged across a 2x threshold range. BE11007 and BA10895 drop out — the spread test identifies them as transients, not pedestals. The 11% headline happens to match v1's, but the reasoning and the unit list @@ -612,7 +612,7 @@ It rejects **113 of the 600 channel-rows with |pre| >= 0.025 (18.8%)**, and the rejections are not random — 92 of them fall across 41 unit-channels currently labelled NEGATIVE. Four would become sustained positives under an amplitude-only >=3-consecutive rule: **BE12599|Long (run of 8), BE18003|Vert -(4), BE10895|Vert (3), BE12844|Tran (3).** Until this is re-cut, the fleet label +(4), BA10895|Vert (3), BE12844|Tran (3).** Until this is re-cut, the fleet label is **three-state — POSITIVE / NEGATIVE / SPREAD-REJECTED(unknown)** — and the third state should be excluded from both TP and FP counts rather than silently scored as healthy. Every precision figure computed against the two-state label, @@ -625,7 +625,7 @@ on 39 of its 1,666 histogram-observed days (**2.3%**). Any statement of the form "the fault was absent before date X" that rests on waveform coverage alone is much weaker than its event count suggests. -### BE10895 — reclassified (see also §4) +### BA10895 — reclassified (see also §4) Previously dismissed as a transient. The histogram record shows its **Vert** quiet-minute floor at 0.005 on 62/62 qualifying files from 2023-07-07, then @@ -651,7 +651,7 @@ BE9557|Tran 2021-06-28, BE9557|Vert 2025-06-12. ### Still open in this section - **The 11 thin-coverage units were not screened** (BE10202, BE11462, BE13779, - BE15760, BE15957, BE16754, BE16758, BE8081, BE8626, BE9229, BE9887 — each + BE15760, BA15957, BE16754, BE16758, BE8081, BE8626, BA9229, BE9887 — each under 20 waveform events, several with hundreds of histograms). This is the population most likely to hold a previously unknown offset, and it is the one slice of the plan that did not run. BE11462 was incidentally scored clean by @@ -662,6 +662,28 @@ BE9557|Tran 2021-06-28, BE9557|Vert 2025-06-12. --- +### ⚠ Serial prefixes — four of these units are BlastMates, not MiniMates + +Corrected 2026-09-06, after Brian queried "BA10895?" against a report that +said BE10895. He was right. The BW filename encodes the serial **number +only** — `L895` -> 10895 — and every offset scanner synthesised the family +prefix as `"BE"`. Four of the 43 archive units are **BA** (BlastMate, the +MiniMate Plus's bigger sibling; same Series III, byte-identical data): + +**BA9229, BA10060, BA10895, BA15957.** + +Read off the file bodies, which carry the serial verbatim. No analysis +changed — grouping was always on the numeric part, and no unit number maps +to two serials — but every earlier reference to "BE10895" and the other +three is a label error and has been corrected throughout this document. + +The same assumption was live in two production sites and is fixed +(`sfm/waveform_store.py`, `minimateplus/client.py`): the store would have +filed a BlastMate under a unit that does not exist, and the monitor-log +decoder lost the geo threshold along with the serial. See commit `9ceff65`. + +--- + ## 9. Chronology | date | event | @@ -682,5 +704,6 @@ BE9557|Tran 2021-06-28, BE9557|Vert 2025-06-12. | 2026-09-04 | **Histogram corpus scanned** — 63,505 of 63,535 files, 43 units, 77.9M intervals (9.7x the waveform corpus). `scratch/offset_hist_scan.py`. | | 2026-09-04 | Absolute-floor statistic **retired**: 66% of its discrimination is a day/site confound (other-channels-only AUC 0.746 vs 0.872). Zero-fraction shown **structurally impossible** — the device clamps every interval peak at >= 1 count. | | 2026-09-04 | Site-quiet-gated cross-channel differential established: **BE18438 Vert, BE9558 Tran+Long**, threshold-insensitive over a 2.3x span. Finds only **2 of the 5** confirmed units — leakage into the histogram floor is bimodal (0.9 to 0.02), so a negative result carries almost no information. Per-channel attribution **not** established (channel-scramble p = 0.769). | -| 2026-09-04 | **BE10895 reclassified** from transient to a genuine Vert fault of a different subtype — 49.4% single-axis-dominant events, the highest in the fleet, 100% on Vert. The other six marginal units are clean. | +| 2026-09-04 | **BA10895 reclassified** from transient to a genuine Vert fault of a different subtype — 49.4% single-axis-dominant events, the highest in the fleet, 100% on Vert. The other six marginal units are clean. | | 2026-09-04 | **Defect found in `offset_scan3.py`**: its `spread <= 0.02` gate discards 18.8% of rows with \|pre\| >= 0.025, concentrated on 41 negative unit-channels; 4 would be sustained positives without it. The fleet label is three-state, not two. | +| 2026-09-06 | **Four units relabelled BA, not BE** — BA9229, BA10060, BA10895, BA15957 are BlastMates. The BW filename carries only the serial number; the family prefix must be read from the file body. Fixed in the scanners and in two production sites. |