serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs: offset investigation journal; strip NUL corruption from CLAUDE.md

Browse Source 
Adds docs/offset_investigation.md — a dated journal of the "offset" hardware
fault, in the style of the codec status docs: findings with provenance, dead
ends kept with the reason they died, and per-unit case files.

Contents:
  - base rate 5-6 of 45 units (11-13%) over the DL2 archive, 2018-2026,
    confirming rather than overturning the earlier 2-of-21 estimate
  - the detector, with the rationale for each term and its known blind spot
    (event traces carry real motion, so only trace-dominating offsets show)
  - the bimodality result: relaxing the amplitude floor 11x adds no new units
  - Instantel's own procedure and thresholds from their FAQs 13-0-21 / 12-0-10:
    A/D-mode ">5 counts", the autozero key sequence, and the 2027-2069 X1/X8
    acceptance window that explains the ~10% field success rate of a re-zero
  - four ruled-out hypotheses, each with the evidence that killed it:
    condensation, clipping, the sensor check as a predictor (102 offset events,
    zero failures — a grossly offset unit passes its own self-check), and the
    calibration-timing correlation (confounded, one unit per time bucket)
  - open questions, chiefly whether SUB 0x0E carries the autozero numbers

Cross-referenced from CLAUDE.md and Appendix E of the protocol reference
(whose CRLF line endings are preserved).

Separately: CLAUDE.md had 793 NUL bytes appended after its last line. They
predate this work (present at least as far back as e42956a / v0.21.0) and made
grep treat the file as binary, silently skipping it. Stripped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
serversdown
2026-08-28 20:19:30 +00:00
co-authored by Claude Opus 5
parent 4839ddfa0e
commit 5f1ee5ba91
3 changed files with 374 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+6 -3
View File
@@ -27,8 +27,11 @@ Read this first when picking the project back up.
- **After any codec change, regenerate the store** — `backfill_sidecars.py
--force` then `backfill_event_shape.py`, DB backup first. Stored `.h5` files
do not update themselves.
- **Parked:** the "offset" hardware-fault investigation (Appendix E of the
protocol reference) pending the multi-year BW archive.
- **The "offset" hardware fault has its own journal** --
`docs/offset_investigation.md`. Base rate settled at 5-6 of 45 units
(11-13%) across the DL2 archive; Instantel's own autozero procedure and
its 2027-2069 go/no-go window are recorded there. Best open lead is
`SUB 0x0E` (unimplemented), which may carry those numbers.
When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
@@ -1803,4 +1806,4 @@ body) because writing a dial string may require DLE escaping for embedded contro
To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern
in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes
inside write frame data (the naive parser terminates early at the escaped `0x03`).
inside write frame data (the naive parser terminates early at the escaped `0x03`).
docs/instantel_protocol_reference.md
+5
View File
@@ -3524,6 +3524,11 @@ of them was, for a while.
### E.1 The "offset" fault
> **The full investigation now lives in `docs/offset_investigation.md`** --
> base rate, detector definition, per-unit case files, ruled-out hypotheses,
> and Instantel's own autozero procedure with its 2027-2069 acceptance
> window. This appendix is kept as the protocol-side summary.
**Symptom.** One geophone channel's baseline steps away from zero and
stays there. The trace still carries the real AC signal, but it rides
on a DC pedestal of a few tenths of an in/s. Operators call this an
docs/offset_investigation.md
+363
View File
@@ -0,0 +1,363 @@
# The "offset" fault — investigation journal
A running record of the **offset** hardware fault on Instantel Series III
seismographs: a geophone channel whose trace sits displaced from zero rather
than centred on it.
This is a *journal*, not a spec. Findings are dated, dead ends are kept with
the reason they died, and every number says where it came from. When something
here is superseded, strike it and say why rather than deleting it — the point
is that a future session can tell what was actually established from what was
merely believed at the time.
Companion material:
- `scratch/offset_scan.py` — the detector
- `scratch/verify_against_ascii.py` — decoder verification harness
- `docs/instantel_protocol_reference.md` — wire protocol, incl. the
unimplemented `SUB 0x0E` this investigation now wants
---
## TL;DR (current state, 2026-08-28)
- **It is real device data, not a decode bug.** Settled early and confirmed
against Blastware's own ASCII exports.
- **Base rate: 5–6 of 45 units (11–13%)** across the full DL2 archive,
2018–2026. This *confirms* the earlier 2-of-21 (9.5%) estimate from the much
smaller Terra-View DB — survivorship bias from deleted events had **not**
concealed a wave of cases.
- **The fault is bimodal, not a drift continuum.** A unit is either clean or
grossly off. Loosening the amplitude threshold 11× adds no new units.
- **The unit's own sensor check cannot see it.** 102 offset events, zero
sensor-check failures. Do not try to use it as a screen.
- **Cause is still unsettled.** Instantel's autozero fixes the minority of
cases; the rest are hardware. We cannot yet tell which is which remotely.
- **Best open lead:** `SUB 0x0E` (channel sensor data, 8 channels × 10 bytes,
unimplemented) may carry the very numbers Instantel says to check against
**2027–2069**. Untested.
---
## 1. What the fault looks like
A healthy geophone trace is centred on zero. An offset channel is parked away
from zero, so the channel **mean approaches its own peak**. In Blastware the
signature is "parallel lines above or below the zero line" (Instantel's own
wording).
Consequences observed in the field:
- The unit can **self-trigger on its own offset** when the displacement exceeds
the geo trigger level, producing streams of junk events with no ground
motion. Instantel has a separate FAQ for this symptom (13-0-22, *"Unit
triggers continuously without activity"*).
- Recorded PPV for that channel is meaningless while the fault persists.
---
## 2. The detector
Implemented in `scratch/offset_scan.py`. Operates on raw BW binaries only — no
DB, no sidecars.
```
for each series-3 waveform binary:
decode -> per-channel ADC counts
dominant axis = channel with the largest |peak|
flag when |mean| / peak > 0.70
and |mean| >= 0.90 x the unit's geo trigger level
episodes = per-serial runs of flagged events, split on a >12 h gap
```
Why each term:
| term | purpose |
|---|---|
| `\|mean\|/peak > 0.7` | the discriminator. A DC-parked trace has mean ≈ peak. |
| `\|mean\| >= 0.9 × trigger` | amplitude floor — suppresses quiet traces where mean and peak are both tiny and the ratio is meaningless. |
| dominant axis only | the fault is per-channel; scoring all three dilutes it. |
| 12 h episode gap | separates deployments/visits rather than counting events. |
Trigger level comes from a paired `_ASCII.TXT` when one exists, else the
per-serial median learned from that unit's ASCII files, else 0.2 in/s.
**Known limitation.** Event traces contain real ground motion, so this can only
see offsets large enough to *dominate* the trace. A mild offset on a real blast
is invisible. Instantel's A/D-mode check (§5) is the only thing that sees the
mild end. Our base rate is therefore a **gross-offset** rate.
---
## 3. Archive results (2026-08-28)
Source: DL2 event export, 6,577 **unique** series-3 waveforms, 45 units.
See [`dl2-archive`](#8-data-and-tooling) for the `Sent/` mirror trap.
**283 suspect events, 15 episodes, 6 of 45 units (13.3%).**
Excluding BE11007 (§4, likely not an offset at all): **5 of 45 = 11.1%**.
### Threshold sensitivity — the bimodality result
Re-scoring the same corpus at a range of amplitude floors, with two
ratio cut-offs (1 A/D count = 0.005 in/s, see §5):
| \|offset\| floor | m/p > 0.7 | m/p > 0.9 |
|---|---|---|
| 5 cts (0.025 in/s) — *Instantel's own* | 333 ev / 6 units | 279 ev / **5 units** |
| 10 cts (0.050) | 294 / 6 | 274 / 5 |
| 20 cts (0.100) | 250 / 5 | 244 / 4 |
| 40 cts (0.200) | 209 / 5 | 203 / 4 |
| 80 cts (0.400) | 152 / 4 | 148 / 2 |
| 160 cts (0.800) | 144 / 2 | 141 / 1 |
Relaxing the floor by 11× (0.27 → 0.025 in/s) adds ~14% more events and **no
new units**. There is no population of mild offsets hiding below our threshold
*in event data*. Either a unit is clean or it is grossly off.
---
## 4. Per-unit case files
Ordered by severity. `m/p` medians are on the offending channel.
### BE13117 — one violent day, never again
`145 / 454 events (32%)`, **1 episode**, 2023-05-04, 6.8 h.
Offset climbed **0.393 → 1.875 in/s within the episode**. `m/p` median
**0.996** — the trace is almost pure DC. No recurrence in the rest of its 454
events. No ASCII files in the archive, so no calibration history.
### BE18438 — recurring, months apart
`87 / 293 (30%)`, **2 episodes**: 2025-11-15 (1.2 h, n=12, 0.279 → 0.369) and
2026-02-25 (**28.8 h**, n=75, 0.183 → 0.366). `m/p` median 0.967.
Clean across all 196 events preceding its 2025-08-12 calibration.
### BE9558 — six years apart
`38 / 196 (19%)`, **4 episodes**: 2020-02-11 (6.3 h, n=33, but only
0.068 → 0.086 — very mild), then 2026-04-14, 2026-04-29, 2026-05-04
(0.28–0.45). `m/p` median 0.919. Calibrated 2026-06-26; 0/7 events flagged
after, but n=7 is far too small to call it fixed.
### BE12599 — the live case ⚠
`6 / 77 (8%)`, **6 single-event episodes, one per day at exactly 05:00**,
2026-08-17 → 2026-08-23. Offset rose 0.383 → 0.565 then fell back to 0.345.
`m/p` ≈ 0.965, geo trigger 0.3 in/s — **the offset exceeds the trigger level,
so the unit is triggering on its own fault**. Last calibrated 2025-08-12.
This is the most recent and the most useful: a currently-faulting unit is the
natural experiment for the re-zero-vs-repair question (§7).
### BE11529 — marginal
`4 / 99 (4%)`, 1 episode 2025-07-08, 0.4 h, offsets only 0.051 → 0.058 in/s.
`m/p` median 0.959, so DC-dominated, but the magnitude is near the noise of
this method. Treat as unconfirmed.
### BE11007 — probably NOT an offset
`3 / 70 (4%)`, 1 episode 2022-01-17, offsets 5.500 → 6.904 in/s — by far the
largest. But `m/p` is only **0.719–0.738** against ≥0.9 for every other unit,
and the peaks are 7.6–9.4 in/s on a 10 in/s range. That reads as a **large
real blast with asymmetric ground motion**, not a parked trace. Excluded from
the headline base rate.
---
## 5. Instantel's own procedure and thresholds
From two Instantel technical-support FAQs supplied 2026-08-28
(answers **13-0-21** *"How to determine offsets"* and **12-0-10** *"Removing
offsets on an Instantel Series III monitor"*; created 2008/2007, last updated
2009-03-06).
### Identifying (13-0-21)
1. Create or use an event with the **manual minimum trigger** set for the
connected geophone and microphone — i.e. an event that recorded no real data.
2. Save it and open in Blastware.
3. An offset shows as **parallel lines above or below the zero line**.
4. Put the unit in **A/D mode** — on Series III, press and hold `OPTION`, then
press `START MONITOR`.
5. **Display counts higher than 5**, with no vibration or overpressure present,
indicate an offset.
### Removing — the autozero (12-0-10)
1. Be in a **quiet area with low vibration**.
2. Power on the Blastmate III / Minimate Plus.
3. Connect the geophone and microphone — **LINEAR mic only**.
⚠ *Do not connect an "A" weight microphone, regardless of what the monitor
displays.*
4. Press `Test`.
5. Wait for the **Sensor Check** results to appear.
6. Press `OPTION` and `START MONITOR` **simultaneously**.
7. `Performing Autozero` appears; press `Enter`.
8. Confirm the sensors are properly connected; press `Enter`.
9. Wait for the autozero to complete.
10. Press `Enter` twice → Main Menu, *Ready To Monitor*, offset corrected.
### The go/no-go number — 2027 to 2069
> When you perform an Autozero on any Series III unit, the lists of numbers in
> the **X1 and X8 gains should all be between 2027 and 2069**. If not, repeat
> the Autozero. **If the numbers are extremely out of the specified range, then
> the unit should be sent in for repair.**
>
> If this process does not remove the offset problem, return the unit **and
> sensors** to Instantel for repair.
This is the documented explanation for the field experience (Brian's dad,
2026-08-28) that **a re-zero works maybe 10% of the time** — the autozero only
recovers units whose zero reference is still near-correct.
### Scale derivation (inference, well-supported — not proven)
2048 is 12-bit midscale. Our codec's geo full scale is 32000 internal counts =
10 in/s, with 1 decoder unit = 16 counts = exactly 0.005 in/s
(`geo-full-scale-is-32000-counts`). ±2000 A/D counts about 2048 therefore maps
to ±10 in/s at **0.005 in/s per A/D count**. That makes:
- Instantel's ">5 counts" threshold ≈ **0.025 in/s**
- the 2027–2069 window = **±21 counts = ±0.105 in/s** of tolerated zero error
Consistent and mutually corroborating, but we have not confirmed the A/D-count
scale directly from a device reading.
---
## 6. Ruled out — keep these dead
### Condensation / humidity — DEAD (2026-08-25)
Proposed, then killed by its own controls: BE18438 stayed flat across a 10-hour
overnight gap, and only 2 of 21 units showed the fault while 19 sat in the same
weather. The apparent "diurnal cycle" was an artifact of binning by hour-of-day
across two days. See `waveform-dc-offset-is-real-device-data`.
### Clipping as a false-positive source — RULED OUT (2026-08-28)
A rail-hitting trace would fake an offset (mean → peak). It isn't happening:
median suspect peak is only **10% of full scale**, p90 is 18.6%. Only BE11007's
3 events exceed 50% FS, and none reach 98%.
### The sensor check as a predictor — DOES NOT WORK (2026-08-28)
Tested on 102 offset events across 4 units:
| unit | state | n | failed | median ratio | median freq |
|---|---|---|---|---|---|
| BE11529 | offset | 4 | **0** | 3.90 | 7.6 |
| BE11529 | clean | 14 | 0 | 3.80 | 7.5 |
| BE12599 | offset | 6 | **0** | 4.00 | 7.4 |
| BE12599 | clean | 13 | 0 | 4.00 | 7.6 |
| BE18438 | offset | 87 | **0** | 3.70 | 7.6 |
| BE18438 | clean | 25 | 0 | 3.80 | 7.5 |
| BE9558 | offset | 5 | **0** | 3.90 | 7.8 |
| BE9558 | clean | 44 | 0 | 3.80 | 7.5 |
Zero failures on either side and indistinguishable ratios/frequencies. The
swing test measures geophone frequency response and damping — it never examines
DC zero. **A grossly offset unit passes its own self-check.** This is why the
fault goes unnoticed until somebody looks at waveforms.
### "Offsets develop N months after calibration" — CONFOUNDED, NOT A FINDING
Tempting, and it looked strong:
| unit | suspect before latest cal | after |
|---|---|---|
| BE18438 | 0 / 196 | 87 / 97 |
| BE12599 | 0 / 62 | 6 / 15 |
| BE11529 | 0 / 82 | 4 / 17 |
| BE9558 | 38 / 189 | 0 / 7 |
But bucketing suspects by months-since-calibration gives **one unit per bucket**:
`0–3mo={BE11529}`, `3–6 & 6–9mo={BE18438}`, `9–12mo={BE9558}`,
`12–15mo={BE12599}`. The apparent "51% failure rate at 6–9 months" is entirely
BE18438's single February 2026 episode. Five units with roughly one episode
each cannot support a population trend. **Do not re-derive this.**
Also note: all affected units are calibrated on a **~12–13 month cadence**, so
"sent to Instantel" is the routine annual schedule, not evidence of a
fault-driven return.
---
## 7. Open questions
### Q1 — Is it a latched bad zero or analog degradation?
The question that decides everything. A latched zero is correctable (possibly
over the wire); degradation means a repair. Instantel's 2027–2069 rule implies
*both* populations exist, with the split roughly 10/90 in the field.
**BE12599 is the natural experiment** — faulting as of 2026-08-23. Read its
values, run the autozero, read them again.
### Q2 — Can we read the autozero numbers over the wire? (best lead)
Instantel says to check *"the lists of numbers in the **X1 and X8 gains**"* —
4 sensors × 2 gains = **8 channels**. The protocol reference already documents
an unimplemented command with exactly that shape:
```
SUB 0x0E -> RSP 0xF1 "channel sensor data"
2-step read; channel selector in params[6:8] = 0x0000..0x0007
data length 0x0A (10 bytes) per channel
```
Blastware's *Unit Channel Test* sequence:
`POLL×N → 0x15 → 0x01 → 0x08 → 0x01 → 0x0E×8 → 0x98×2 → 0x0E×8`
— note the **second `0x0E` pass carries live ADC readings**.
**Hypothesis (untested):** `0x0E` returns the numbers Instantel wants compared
against 2027–2069. If true, SFM could diagnose an offset remotely *and* predict
whether a re-zero will succeed — converting a 10%/90% shipping gamble into a
decision made before packing a box.
**How to test.** `bridges/ach_mitm.py` is a generic TCP proxy:
```bash
python bridges/ach_mitm.py --bw-host <MODEM_IP> --bw-port 9034 --listen-port 9999
```
Point Blastware at the proxy and run **Unit Channel Test**.
⚠ In this topology the output filenames are reversed — the tool labels the
*connecting* side "unit", so `raw_s3_*.bin` holds Blastware's bytes and
`raw_bw_*.bin` the unit's.
Capture priority: (1) BE12599 while faulting, (2) a known-good unit as control,
(3) before/after an autozero on the same unit. Eight 10-byte payloads with an
expected value near 2048 is a very constrained puzzle.
### Q3 — What is the mild-offset rate?
Unmeasurable from event files (§2). Only the A/D-mode check sees it. Would
need a fleet sweep in A/D mode, or Q2 to succeed.
### Q4 — Does an offset recur on the same unit after service?
BE9558 shows episodes in 2020 and 2026; BE18438 twice in four months. Suggestive
of recurrence, but service records aren't in the data — only calibration dates.
---
## 8. Data and tooling
| what | where |
|---|---|
| detector | `scratch/offset_scan.py` |
| current results | `/home/serversdown/dl2-archive/offset_archive.csv` |
| earlier candidate list (Terra-View DB, 274 events) | `scratch/offset_candidates.csv` |
| archive working copy | `/home/serversdown/dl2-archive/files/` |
| archive source | NAS `DeathStar` 10.0.0.2, `/volume1/Uploads/TMI/DL2-Event-backup-8-25-26/Event/autocall home/` |
⚠ **The DL2 export keeps a byte-identical `Sent/` mirror of its root.** 13,077
waveform paths are 6,577 distinct files. Always dedupe by basename — this
doubled two reported figures before it was caught.
---
## 9. Chronology
| date | event |
|---|---|
| 2026-08-25 | Reported as a *waveform decode bug* — traces with a DC offset. Investigation shows the offset is **real device data**; the decoder is correct. |
| 2026-08-25 | Brian relays his dad's description: a known hardware fault called an "offset"; usually sent to Instantel. |
| 2026-08-25 | First detection pass over the Terra-View DB: **2 of 21 units**, 274 events, 11 months. Flagged as vulnerable to survivorship bias — flooded events were routinely deleted. |
| 2026-08-25 | Condensation hypothesis proposed, then **killed by its own controls**. |
| 2026-08-25 | Parked pending the multi-year archive. |
| 2026-08-28 | DL2 archive pulled (33 GB, 546k files; 6.6 GB working set). |
| 2026-08-28 | Archive scan: **6 of 45 units**, 283 events, 15 episodes. Prior base rate **confirmed**, not overturned. |
| 2026-08-28 | Clipping ruled out; `m/p` established as the discriminator; BE11007 reclassified as probably a real blast. |
| 2026-08-28 | Calibration-timing correlation attempted and **rejected as confounded**. |
| 2026-08-28 | Instantel FAQs supplied: autozero procedure, the **2027–2069** window, the **>5 counts** threshold. Explains the ~10% re-zero success rate. |
| 2026-08-28 | Bimodality established; sensor check proven **blind** to offsets; `SUB 0x0E` identified as the best open lead. |