diff --git a/CLAUDE.md b/CLAUDE.md index f2c7e75..c9a05e0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,8 +27,11 @@ Read this first when picking the project back up. - **After any codec change, regenerate the store** — `backfill_sidecars.py --force` then `backfill_event_shape.py`, DB backup first. Stored `.h5` files do not update themselves. -- **Parked:** the "offset" hardware-fault investigation (Appendix E of the - protocol reference) pending the multi-year BW archive. +- **The "offset" hardware fault has its own journal** -- + `docs/offset_investigation.md`. Base rate settled at 5-6 of 45 units + (11-13%) across the DL2 archive; Instantel's own autozero procedure and + its 2027-2069 go/no-go window are recorded there. Best open lead is + `SUB 0x0E` (unimplemented), which may carry those numbers. When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document @@ -1803,4 +1806,4 @@ body) because writing a dial string may require DLE escaping for embedded contro To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes -inside write frame data (the naive parser terminates early at the escaped `0x03`). \ No newline at end of file +inside write frame data (the naive parser terminates early at the escaped `0x03`). diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index 4fecbad..a75393a 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -3524,6 +3524,11 @@ of them was, for a while. ### E.1 The "offset" fault +> **The full investigation now lives in `docs/offset_investigation.md`** -- +> base rate, detector definition, per-unit case files, ruled-out hypotheses, +> and Instantel's own autozero procedure with its 2027-2069 acceptance +> window. This appendix is kept as the protocol-side summary. + **Symptom.** One geophone channel's baseline steps away from zero and stays there. The trace still carries the real AC signal, but it rides on a DC pedestal of a few tenths of an in/s. Operators call this an diff --git a/docs/offset_investigation.md b/docs/offset_investigation.md new file mode 100644 index 0000000..daccf3c --- /dev/null +++ b/docs/offset_investigation.md @@ -0,0 +1,363 @@ +# The "offset" fault — investigation journal + +A running record of the **offset** hardware fault on Instantel Series III +seismographs: a geophone channel whose trace sits displaced from zero rather +than centred on it. + +This is a *journal*, not a spec. Findings are dated, dead ends are kept with +the reason they died, and every number says where it came from. When something +here is superseded, strike it and say why rather than deleting it — the point +is that a future session can tell what was actually established from what was +merely believed at the time. + +Companion material: +- `scratch/offset_scan.py` — the detector +- `scratch/verify_against_ascii.py` — decoder verification harness +- `docs/instantel_protocol_reference.md` — wire protocol, incl. the + unimplemented `SUB 0x0E` this investigation now wants + +--- + +## TL;DR (current state, 2026-08-28) + +- **It is real device data, not a decode bug.** Settled early and confirmed + against Blastware's own ASCII exports. +- **Base rate: 5–6 of 45 units (11–13%)** across the full DL2 archive, + 2018–2026. This *confirms* the earlier 2-of-21 (9.5%) estimate from the much + smaller Terra-View DB — survivorship bias from deleted events had **not** + concealed a wave of cases. +- **The fault is bimodal, not a drift continuum.** A unit is either clean or + grossly off. Loosening the amplitude threshold 11× adds no new units. +- **The unit's own sensor check cannot see it.** 102 offset events, zero + sensor-check failures. Do not try to use it as a screen. +- **Cause is still unsettled.** Instantel's autozero fixes the minority of + cases; the rest are hardware. We cannot yet tell which is which remotely. +- **Best open lead:** `SUB 0x0E` (channel sensor data, 8 channels × 10 bytes, + unimplemented) may carry the very numbers Instantel says to check against + **2027–2069**. Untested. + +--- + +## 1. What the fault looks like + +A healthy geophone trace is centred on zero. An offset channel is parked away +from zero, so the channel **mean approaches its own peak**. In Blastware the +signature is "parallel lines above or below the zero line" (Instantel's own +wording). + +Consequences observed in the field: +- The unit can **self-trigger on its own offset** when the displacement exceeds + the geo trigger level, producing streams of junk events with no ground + motion. Instantel has a separate FAQ for this symptom (13-0-22, *"Unit + triggers continuously without activity"*). +- Recorded PPV for that channel is meaningless while the fault persists. + +--- + +## 2. The detector + +Implemented in `scratch/offset_scan.py`. Operates on raw BW binaries only — no +DB, no sidecars. + +``` +for each series-3 waveform binary: + decode -> per-channel ADC counts + dominant axis = channel with the largest |peak| + flag when |mean| / peak > 0.70 + and |mean| >= 0.90 x the unit's geo trigger level +episodes = per-serial runs of flagged events, split on a >12 h gap +``` + +Why each term: + +| term | purpose | +|---|---| +| `\|mean\|/peak > 0.7` | the discriminator. A DC-parked trace has mean ≈ peak. | +| `\|mean\| >= 0.9 × trigger` | amplitude floor — suppresses quiet traces where mean and peak are both tiny and the ratio is meaningless. | +| dominant axis only | the fault is per-channel; scoring all three dilutes it. | +| 12 h episode gap | separates deployments/visits rather than counting events. | + +Trigger level comes from a paired `_ASCII.TXT` when one exists, else the +per-serial median learned from that unit's ASCII files, else 0.2 in/s. + +**Known limitation.** Event traces contain real ground motion, so this can only +see offsets large enough to *dominate* the trace. A mild offset on a real blast +is invisible. Instantel's A/D-mode check (§5) is the only thing that sees the +mild end. Our base rate is therefore a **gross-offset** rate. + +--- + +## 3. Archive results (2026-08-28) + +Source: DL2 event export, 6,577 **unique** series-3 waveforms, 45 units. +See [`dl2-archive`](#8-data-and-tooling) for the `Sent/` mirror trap. + +**283 suspect events, 15 episodes, 6 of 45 units (13.3%).** +Excluding BE11007 (§4, likely not an offset at all): **5 of 45 = 11.1%**. + +### Threshold sensitivity — the bimodality result + +Re-scoring the same corpus at a range of amplitude floors, with two +ratio cut-offs (1 A/D count = 0.005 in/s, see §5): + +| \|offset\| floor | m/p > 0.7 | m/p > 0.9 | +|---|---|---| +| 5 cts (0.025 in/s) — *Instantel's own* | 333 ev / 6 units | 279 ev / **5 units** | +| 10 cts (0.050) | 294 / 6 | 274 / 5 | +| 20 cts (0.100) | 250 / 5 | 244 / 4 | +| 40 cts (0.200) | 209 / 5 | 203 / 4 | +| 80 cts (0.400) | 152 / 4 | 148 / 2 | +| 160 cts (0.800) | 144 / 2 | 141 / 1 | + +Relaxing the floor by 11× (0.27 → 0.025 in/s) adds ~14% more events and **no +new units**. There is no population of mild offsets hiding below our threshold +*in event data*. Either a unit is clean or it is grossly off. + +--- + +## 4. Per-unit case files + +Ordered by severity. `m/p` medians are on the offending channel. + +### BE13117 — one violent day, never again +`145 / 454 events (32%)`, **1 episode**, 2023-05-04, 6.8 h. +Offset climbed **0.393 → 1.875 in/s within the episode**. `m/p` median +**0.996** — the trace is almost pure DC. No recurrence in the rest of its 454 +events. No ASCII files in the archive, so no calibration history. + +### BE18438 — recurring, months apart +`87 / 293 (30%)`, **2 episodes**: 2025-11-15 (1.2 h, n=12, 0.279 → 0.369) and +2026-02-25 (**28.8 h**, n=75, 0.183 → 0.366). `m/p` median 0.967. +Clean across all 196 events preceding its 2025-08-12 calibration. + +### BE9558 — six years apart +`38 / 196 (19%)`, **4 episodes**: 2020-02-11 (6.3 h, n=33, but only +0.068 → 0.086 — very mild), then 2026-04-14, 2026-04-29, 2026-05-04 +(0.28–0.45). `m/p` median 0.919. Calibrated 2026-06-26; 0/7 events flagged +after, but n=7 is far too small to call it fixed. + +### BE12599 — the live case ⚠ +`6 / 77 (8%)`, **6 single-event episodes, one per day at exactly 05:00**, +2026-08-17 → 2026-08-23. Offset rose 0.383 → 0.565 then fell back to 0.345. +`m/p` ≈ 0.965, geo trigger 0.3 in/s — **the offset exceeds the trigger level, +so the unit is triggering on its own fault**. Last calibrated 2025-08-12. + +This is the most recent and the most useful: a currently-faulting unit is the +natural experiment for the re-zero-vs-repair question (§7). + +### BE11529 — marginal +`4 / 99 (4%)`, 1 episode 2025-07-08, 0.4 h, offsets only 0.051 → 0.058 in/s. +`m/p` median 0.959, so DC-dominated, but the magnitude is near the noise of +this method. Treat as unconfirmed. + +### BE11007 — probably NOT an offset +`3 / 70 (4%)`, 1 episode 2022-01-17, offsets 5.500 → 6.904 in/s — by far the +largest. But `m/p` is only **0.719–0.738** against ≥0.9 for every other unit, +and the peaks are 7.6–9.4 in/s on a 10 in/s range. That reads as a **large +real blast with asymmetric ground motion**, not a parked trace. Excluded from +the headline base rate. + +--- + +## 5. Instantel's own procedure and thresholds + +From two Instantel technical-support FAQs supplied 2026-08-28 +(answers **13-0-21** *"How to determine offsets"* and **12-0-10** *"Removing +offsets on an Instantel Series III monitor"*; created 2008/2007, last updated +2009-03-06). + +### Identifying (13-0-21) + +1. Create or use an event with the **manual minimum trigger** set for the + connected geophone and microphone — i.e. an event that recorded no real data. +2. Save it and open in Blastware. +3. An offset shows as **parallel lines above or below the zero line**. +4. Put the unit in **A/D mode** — on Series III, press and hold `OPTION`, then + press `START MONITOR`. +5. **Display counts higher than 5**, with no vibration or overpressure present, + indicate an offset. + +### Removing — the autozero (12-0-10) + +1. Be in a **quiet area with low vibration**. +2. Power on the Blastmate III / Minimate Plus. +3. Connect the geophone and microphone — **LINEAR mic only**. + ⚠ *Do not connect an "A" weight microphone, regardless of what the monitor + displays.* +4. Press `Test`. +5. Wait for the **Sensor Check** results to appear. +6. Press `OPTION` and `START MONITOR` **simultaneously**. +7. `Performing Autozero` appears; press `Enter`. +8. Confirm the sensors are properly connected; press `Enter`. +9. Wait for the autozero to complete. +10. Press `Enter` twice → Main Menu, *Ready To Monitor*, offset corrected. + +### The go/no-go number — 2027 to 2069 + +> When you perform an Autozero on any Series III unit, the lists of numbers in +> the **X1 and X8 gains should all be between 2027 and 2069**. If not, repeat +> the Autozero. **If the numbers are extremely out of the specified range, then +> the unit should be sent in for repair.** +> +> If this process does not remove the offset problem, return the unit **and +> sensors** to Instantel for repair. + +This is the documented explanation for the field experience (Brian's dad, +2026-08-28) that **a re-zero works maybe 10% of the time** — the autozero only +recovers units whose zero reference is still near-correct. + +### Scale derivation (inference, well-supported — not proven) + +2048 is 12-bit midscale. Our codec's geo full scale is 32000 internal counts = +10 in/s, with 1 decoder unit = 16 counts = exactly 0.005 in/s +(`geo-full-scale-is-32000-counts`). ±2000 A/D counts about 2048 therefore maps +to ±10 in/s at **0.005 in/s per A/D count**. That makes: + +- Instantel's ">5 counts" threshold ≈ **0.025 in/s** +- the 2027–2069 window = **±21 counts = ±0.105 in/s** of tolerated zero error + +Consistent and mutually corroborating, but we have not confirmed the A/D-count +scale directly from a device reading. + +--- + +## 6. Ruled out — keep these dead + +### Condensation / humidity — DEAD (2026-08-25) +Proposed, then killed by its own controls: BE18438 stayed flat across a 10-hour +overnight gap, and only 2 of 21 units showed the fault while 19 sat in the same +weather. The apparent "diurnal cycle" was an artifact of binning by hour-of-day +across two days. See `waveform-dc-offset-is-real-device-data`. + +### Clipping as a false-positive source — RULED OUT (2026-08-28) +A rail-hitting trace would fake an offset (mean → peak). It isn't happening: +median suspect peak is only **10% of full scale**, p90 is 18.6%. Only BE11007's +3 events exceed 50% FS, and none reach 98%. + +### The sensor check as a predictor — DOES NOT WORK (2026-08-28) +Tested on 102 offset events across 4 units: + +| unit | state | n | failed | median ratio | median freq | +|---|---|---|---|---|---| +| BE11529 | offset | 4 | **0** | 3.90 | 7.6 | +| BE11529 | clean | 14 | 0 | 3.80 | 7.5 | +| BE12599 | offset | 6 | **0** | 4.00 | 7.4 | +| BE12599 | clean | 13 | 0 | 4.00 | 7.6 | +| BE18438 | offset | 87 | **0** | 3.70 | 7.6 | +| BE18438 | clean | 25 | 0 | 3.80 | 7.5 | +| BE9558 | offset | 5 | **0** | 3.90 | 7.8 | +| BE9558 | clean | 44 | 0 | 3.80 | 7.5 | + +Zero failures on either side and indistinguishable ratios/frequencies. The +swing test measures geophone frequency response and damping — it never examines +DC zero. **A grossly offset unit passes its own self-check.** This is why the +fault goes unnoticed until somebody looks at waveforms. + +### "Offsets develop N months after calibration" — CONFOUNDED, NOT A FINDING +Tempting, and it looked strong: + +| unit | suspect before latest cal | after | +|---|---|---| +| BE18438 | 0 / 196 | 87 / 97 | +| BE12599 | 0 / 62 | 6 / 15 | +| BE11529 | 0 / 82 | 4 / 17 | +| BE9558 | 38 / 189 | 0 / 7 | + +But bucketing suspects by months-since-calibration gives **one unit per bucket**: +`0–3mo={BE11529}`, `3–6 & 6–9mo={BE18438}`, `9–12mo={BE9558}`, +`12–15mo={BE12599}`. The apparent "51% failure rate at 6–9 months" is entirely +BE18438's single February 2026 episode. Five units with roughly one episode +each cannot support a population trend. **Do not re-derive this.** + +Also note: all affected units are calibrated on a **~12–13 month cadence**, so +"sent to Instantel" is the routine annual schedule, not evidence of a +fault-driven return. + +--- + +## 7. Open questions + +### Q1 — Is it a latched bad zero or analog degradation? +The question that decides everything. A latched zero is correctable (possibly +over the wire); degradation means a repair. Instantel's 2027–2069 rule implies +*both* populations exist, with the split roughly 10/90 in the field. + +**BE12599 is the natural experiment** — faulting as of 2026-08-23. Read its +values, run the autozero, read them again. + +### Q2 — Can we read the autozero numbers over the wire? (best lead) +Instantel says to check *"the lists of numbers in the **X1 and X8 gains**"* — +4 sensors × 2 gains = **8 channels**. The protocol reference already documents +an unimplemented command with exactly that shape: + +``` +SUB 0x0E -> RSP 0xF1 "channel sensor data" + 2-step read; channel selector in params[6:8] = 0x0000..0x0007 + data length 0x0A (10 bytes) per channel +``` + +Blastware's *Unit Channel Test* sequence: +`POLL×N → 0x15 → 0x01 → 0x08 → 0x01 → 0x0E×8 → 0x98×2 → 0x0E×8` +— note the **second `0x0E` pass carries live ADC readings**. + +**Hypothesis (untested):** `0x0E` returns the numbers Instantel wants compared +against 2027–2069. If true, SFM could diagnose an offset remotely *and* predict +whether a re-zero will succeed — converting a 10%/90% shipping gamble into a +decision made before packing a box. + +**How to test.** `bridges/ach_mitm.py` is a generic TCP proxy: + +```bash +python bridges/ach_mitm.py --bw-host --bw-port 9034 --listen-port 9999 +``` + +Point Blastware at the proxy and run **Unit Channel Test**. +⚠ In this topology the output filenames are reversed — the tool labels the +*connecting* side "unit", so `raw_s3_*.bin` holds Blastware's bytes and +`raw_bw_*.bin` the unit's. + +Capture priority: (1) BE12599 while faulting, (2) a known-good unit as control, +(3) before/after an autozero on the same unit. Eight 10-byte payloads with an +expected value near 2048 is a very constrained puzzle. + +### Q3 — What is the mild-offset rate? +Unmeasurable from event files (§2). Only the A/D-mode check sees it. Would +need a fleet sweep in A/D mode, or Q2 to succeed. + +### Q4 — Does an offset recur on the same unit after service? +BE9558 shows episodes in 2020 and 2026; BE18438 twice in four months. Suggestive +of recurrence, but service records aren't in the data — only calibration dates. + +--- + +## 8. Data and tooling + +| what | where | +|---|---| +| detector | `scratch/offset_scan.py` | +| current results | `/home/serversdown/dl2-archive/offset_archive.csv` | +| earlier candidate list (Terra-View DB, 274 events) | `scratch/offset_candidates.csv` | +| archive working copy | `/home/serversdown/dl2-archive/files/` | +| archive source | NAS `DeathStar` 10.0.0.2, `/volume1/Uploads/TMI/DL2-Event-backup-8-25-26/Event/autocall home/` | + +⚠ **The DL2 export keeps a byte-identical `Sent/` mirror of its root.** 13,077 +waveform paths are 6,577 distinct files. Always dedupe by basename — this +doubled two reported figures before it was caught. + +--- + +## 9. Chronology + +| date | event | +|---|---| +| 2026-08-25 | Reported as a *waveform decode bug* — traces with a DC offset. Investigation shows the offset is **real device data**; the decoder is correct. | +| 2026-08-25 | Brian relays his dad's description: a known hardware fault called an "offset"; usually sent to Instantel. | +| 2026-08-25 | First detection pass over the Terra-View DB: **2 of 21 units**, 274 events, 11 months. Flagged as vulnerable to survivorship bias — flooded events were routinely deleted. | +| 2026-08-25 | Condensation hypothesis proposed, then **killed by its own controls**. | +| 2026-08-25 | Parked pending the multi-year archive. | +| 2026-08-28 | DL2 archive pulled (33 GB, 546k files; 6.6 GB working set). | +| 2026-08-28 | Archive scan: **6 of 45 units**, 283 events, 15 episodes. Prior base rate **confirmed**, not overturned. | +| 2026-08-28 | Clipping ruled out; `m/p` established as the discriminator; BE11007 reclassified as probably a real blast. | +| 2026-08-28 | Calibration-timing correlation attempted and **rejected as confounded**. | +| 2026-08-28 | Instantel FAQs supplied: autozero procedure, the **2027–2069** window, the **>5 counts** threshold. Explains the ~10% re-zero success rate. | +| 2026-08-28 | Bimodality established; sensor check proven **blind** to offsets; `SUB 0x0E` identified as the best open lead. |