diff --git a/CLAUDE.md b/CLAUDE.md index 1e5b2cb..9fc0941 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -223,6 +223,44 @@ custom delta + RLE + variable-width codec. `NN + 2` for int8 blocks). Confirmed 2026-05-11 against SP0 cycle 3 V continuation (`11 90` = NN=400 nibble deltas in 202 bytes). +### Framing cases added 2026-05-11 → 2026-08-25 + +Four more block-framing cases, each of which had been causing **silent +channel truncation** — `walk_body` ends its loop on an unrecognised tag +and `decode_waveform_v2` returns whatever channels it got, so an +unhandled tag surfaces as short channels with no error raised. Found by +diffing 75 production events against their preserved Blastware ASCII +exports (`//_ASCII.TXT`). + +- **Wide-NN RLE `0X NN`** — the 12-bit NN encoding documented above for + `1X`/`2X` **also applies to the `00 NN` RLE tag**. A narrow run maxes + out at NN=0xFC, so a quiet stretch longer than 252 samples must use + the wide form (e.g. `01 0c` = 268 repeats). +- **`30 NN` is not capped at NN=0x10** — data-section blocks reach at + least NN=0x18. The `NN × 1.5 + 2` length formula was already right; + only the guard was wrong. +- **`40 NN` segment headers are variable width** — NN is the *count of + int16 BE continuation deltas for the PREVIOUS channel*, so the header + is `2*NN + 16` bytes and every field after the deltas shifts by + `2*NN`. `40 01` (18 B) and `40 03` (22 B) both occur alongside the + common `40 02` (20 B). +- **Tagless segment headers** — a header can appear with **no `40 NN` + tag at all**: just the 14-byte tail + `[field2:2][len:2][channel_id:4][marker:2][anchors:4]`. This is the + NN=0 case (previous channel needed no continuation deltas). + +**The header "counter" is really a channel id.** The 4-byte field long +documented as a "monotonic uint32 LE counter" is +`[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert +`0x48`=Long `0x49`=MicL — verified on **1697/1697** segment headers +across the corpus, zero disagreements. `decode_waveform_v2` now takes +the channel from this field rather than from rotation position; a single +missed or extra header (exactly what tagless headers caused) desyncs +rotation and corrupts every channel after it. + +Corpus result, end to end through the production path: +**exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.** + ### What's NOT solved - **MicL channel conversion to dB(L)** — the codec emits MicL as @@ -230,9 +268,29 @@ custom delta + RLE + variable-width codec. shows mic in dB(L) with ~6 dB quantization steps. Need to map ADC counts → dB(L) for direct comparison; likely `dB = 20*log10(|counts|) + offset` or similar. -- **Walker edge cases** — SP0/SS0/SV0 don't walk the full event due - to block-length quirks past the first few segments. Every sample - reached is correct; the walker just needs robustness improvements. +- **Variable-prefix segment descriptors** — 3 of the 75 ground-truth + production events still truncate. The walk reaches a segment header + whose channel-id field is preceded by a *variable-width* prefix (2, 4 + or 6 bytes observed; the standard tagless form always has 4), carrying + an `01 00` marker instead of `02 00`. The marker is **not** simply an + anchor count — `01 00` records appear with both 2- and 4-byte anchor + fields in the same file. Examples: `BE12599/N599LPNB.JF0W` @1155, + `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485. +- **Histogram codec misses the peak interval** — 301 of 1141 series-3 + histogram channels (26%) decode a peak below 95% of the + device-reported PPV (one reads 0.0300 against a device PPV of 0.1200, + exactly 1/4). Not a scale error — that would be a uniform 2.34%. + Lives in `histogram_codec.decode_histogram_body`, untouched by the + 2026-08-25 waveform pass. +- **Micromate (UM-series) IDF decode is ~1000× low** — e.g. + `UM11402_20260406130113.IDFW` gives a Tran peak of 0.0009 in/s against + a device-reported 1.1168. The Thor IDF path decodes sanely, so this + is UM-specific. +- **Thor IDF per-count LSB** — after the 32000 geo full-scale + correction, series-4 Thor peaks sit at a median 0.983 of the + device-reported peak (was 0.960 under 32768). Closer but not exact; + Thor likely uses its own per-count LSB rather than the BW + 16-count/0.005 in/s convention. ### Decoded sample counts (across the fixture bundle) diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index 6c25d27..f4b118f 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -11,6 +11,7 @@ | Date | Section | Change | |---|---|---| +| 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. | | 2026-05-20 | §2, §3, §4.2, §5.1, §5.3, §6, §7.5b, §7.6.1, §7.6.3, §7.6.4, §7.7.2, §7.7.3, §7.7.5, §7.8.4, §7.8.7, §7.9, §8, §11, §12, §13, §14, §15, Appendix D | **DOC AUDIT PASS — accuracy sweep against `CLAUDE.md` + `minimateplus/` code.** Fixed: (1) S3 frames terminate on bare ETX, not DLE+ETX — §2/§3 rewritten. (2) §3 payload layout corrected — byte[1]=flags, byte[2]=SUB (was wrongly labelled DLE/ADDR). (3) §4.2 — probe responses do NOT carry data length; lengths are hardcoded `DATA_LENGTHS` constants. (4) §5.1 — removed stale duplicate "SUB 1C = TRIGGER CONFIG READ" row; SUB 0A lengths corrected from `0x30/0x26` to `0x46/0x2C` (real event / boundary marker). (5) §5.3 — added missing write-frame format (BW_CMD-only doubling, DLE-aware checksum, offset formula, ack format, SUB 71 chunk parameters). (6) §6 — fixed "SUB 06 → channel config read" → event storage range. (7) §7.5b / §8 — added the 10-byte `sub_code=0x03` continuous-mode timestamp variant alongside the 9-byte single-shot layout; peak vector sum location corrected from "fixed offset 87" to `tran_pos − 12` (label-relative). (8) §7.6 / §7.6.1 / §7.6.3 / §7.6.4 — switched compliance-anchor convention from the 10-byte form to the canonical 6-byte `\xbe\x80\x00\x00\x00\x00`; recording_mode confirmed at anchor−8 in BOTH read and write (was wrongly listed as anchor−3 write / anchor−4 read); sample_rate at anchor−6, histogram_interval at anchor−4, record_time at anchor+6; geo_range row added at channel_label+33. (9) §7.7.2 — token byte position corrected from `params[6]` to `params[7]`. (10) §7.8.4 — fi==9 skip marked FIXED (already removed from code); chunk-count totals updated. (11) §7.8.7 — TODO replaced with current state of `_decode_a5_metadata_into`. (12) §7.9 — Histogram Interval upgraded ❓ → ✅. (13) §11 — POLL example wire bytes corrected; SUB 5A row added to checksum table. (14) §13 — device-under-test updated for current primary unit (BE11529 / S338.17). (15) §14 — TCP Idle Timeout fixed (0→2 min); Data Forwarding Timeout units clarified. (16) §15 (renumbered from second §14) — open-question items already resolved in CLAUDE.md closed out. (17) Appendix D — extension taxonomy rewritten to reflect the AB0T timestamp encoding (D.5.2/D.5.3); EXTENSION REFUTED warning replaced with the resolved encoding. | | 2026-05-08 | §7.6.1 (RETRACTION) | **❌ RETRACTED — "raw int16 LE 8 bytes/sample-set" body codec was never validated.** The original 4-2-26 confirmation was based on misreading broken-decoder output (full-scale ±32K noise) as evidence the signal had saturated. BW's own 0C peaks for that capture (Tran=0.420 / Vert=3.870 / Long=0.495 in/s) prove the signal was NOT saturated — none of those exceed 13K ADC counts. No event in the project's archive has ever come close to saturation, yet the decoder consistently produces ±32K noise on every event. Conclusion: the body codec is not raw int16 LE; the actual encoding is open. Body byte distribution is heavily skewed (24% `0x00`, 10.5% `0x10`, lots of `10 XX` pairs) — likely a delta encoding with `0x10` as escape, but unverified. Retraction box added at top of §7.6.1; "fully-saturating event" claim removed from channel-identification note. The histogram codec in §7.6.2 IS verified and decoded correctly (different recording mode, 32-byte blocks); use it as a structural hint when reverse-engineering the waveform codec. | | 2026-02-26 | Initial | Document created from first hex dump analysis | @@ -1252,6 +1253,20 @@ pure quantization. This also explains why Blastware reports geo peaks slightly above nominal full scale (e.g. 10.14 in/s): the ADC has headroom past 32000. +**Scope — this is not waveform-specific.** The scale is applied where +ADC counts become physical units, which every event passes through +regardless of which codec produced the samples. Verified after +re-deriving the whole production store: + +| source | median ratio ours/device, 32768 | with 32000 | +|---|---|---| +| series-3 waveform (vs ASCII sample table) | 0.9766 | **1.0000** | +| series-3 histogram (vs ASCII PPV, n=1137) | 0.9766 | **1.0000** | +| series-4 Thor IDF (vs device peak, n=1468) | 0.960 | **0.983** | + +The series-4 figure is closer to correct but not exact — the Thor +per-count LSB is its own open question (see §15). + ###### Unmapped: variable-prefix segment descriptors ❓ OPEN Three of 75 ground-truth production events still truncate. In each, @@ -2994,7 +3009,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger |---|---|---|---| | Timestamp 6-byte format byte[3] — purpose of the separator `0x00` byte | LOW | 2026-02-26 | Not blocking; 9-byte waveform record format (§8.2) fully confirmed without this byte. | | `trail[0]` in serial number response — unit-specific byte, derivation unknown. `trail[1]` resolved as firmware minor version. | MEDIUM | 2026-02-26 | | -| Full channel ID mapping in SUB `5A` stream (01/02/03/04 → which sensor?) | MEDIUM | 2026-02-26 | | +| ~~Full channel ID mapping in SUB `5A` stream~~ — **RESOLVED 2026-08-25 for the waveform body:** every `40 NN` segment header carries `[channel_id][00][00][segment_index]` at `data[2*NN+4]`, with `0x46`=Tran `0x47`=Vert `0x48`=Long `0x49`=MicL. Verified on 1697/1697 segment headers with zero disagreements against the decoded channel rotation. See §7.6.1. | RESOLVED | 2026-02-26 | Resolved 2026-08-25 | | ~~Exact byte boundaries of project string fields in SUB `71` write frame~~ — **RESOLVED 2026-05-05:** project/client/operator/seis-loc/extended-notes come from SUB 5A metadata pages at counter `0x1002` / `0x1004` (§7.8.7), NOT from the SUB 71 write payload. `_decode_a5_metadata_into` locates them via ASCII label scans. | RESOLVED | 2026-02-26 | Resolved 2026-05-05 | | Purpose of SUB `09` / response `F6` — 202-byte read block | MEDIUM | 2026-02-26 | | | Purpose of SUB `2E` / response `D1` — 26-byte read block | MEDIUM | 2026-02-26 | | @@ -3021,6 +3036,10 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger | **ACH inbound server — RESOLVED.** `bridges/ach_server.py` implements full inbound ACH pipeline. `--clear-after-download` flag for delete-after-upload workflow. Post-erase key-reuse detection via `max_downloaded_key` high-water mark. | RESOLVED | 2026-04-11 | | | **Sensor Check dropdown byte location** — byte offset in 1A compliance config payload for the "Sensor Check: Before monitoring / After each event / Disabled" setting is NOT YET LOCATED. Confirmed: unit always runs with "Before monitoring" set. Need a capture with "Disabled" to diff. | MEDIUM | 2026-04-08 | Still open | | **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open | +| **Variable-prefix segment descriptors** — 3 of 75 ground-truth events still truncate. The walk reaches a segment header whose channel-id field is preceded by a *variable-width* prefix (2, 4 or 6 bytes observed; the standard tagless form always has 4), carrying an `01 00` marker instead of `02 00`. The marker is **not** simply an anchor count — `01 00` records appear with both 2- and 4-byte anchor fields in the same file, so prefix width and marker are not yet reconciled. Examples: `BE12599/N599LPNB.JF0W` @1155, `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485. | MEDIUM | 2026-08-25 | Still open | +| **Histogram codec misses the peak interval** — 301 of 1141 series-3 histogram channels (26%) decode a peak below 95% of the device-reported PPV; one example reads 0.0300 against a device PPV of 0.1200 (exactly 1/4). This is not a scale error (that would be a uniform 2.34%) — the interval carrying the peak is being missed or mis-attributed. Affects `histogram_codec.decode_histogram_body`, untouched by the 2026-08-25 waveform pass. | MEDIUM | 2026-08-25 | Still open | +| **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open | +| **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open | --- @@ -3446,4 +3465,98 @@ file[anc−3]: histogram_interval_LO *All findings reverse-engineered from live RS-232 bridge captures.* *Cross-referenced from 2026-03-02 with Instantel MiniMate Plus Operator Manual (716U0101 Rev 15).* -*This is a living document — append changelog entries and timestamps as new findings are confirmed or corrected.* \ No newline at end of file +*This is a living document — append changelog entries and timestamps as new findings are confirmed or corrected.* + +--- + +## Appendix E — Known Device Faults (field-observed, 2026-08-25) + +This appendix records *device* behaviour, not protocol. It exists +because these signatures are easy to mistake for decoder bugs — and one +of them was, for a while. + +### E.1 The "offset" fault + +**Symptom.** One geophone channel's baseline steps away from zero and +stays there. The trace still carries the real AC signal, but it rides +on a DC pedestal of a few tenths of an in/s. Operators call this an +"offset"; it is a known recurring hardware fault, historically resolved +by returning the unit to Instantel, shelving it until calibration, or a +per-channel re-zero in an advanced/professional Blastware build. + +**Why it floods the queue.** The pedestal exceeds the unit's own geo +trigger level, so the channel sits permanently above threshold and the +unit retriggers as fast as it can rearm — one event every 3–7 minutes +for as long as the fault persists. In the surveyed snapshot the worst +episode produced 193 events in 38 hours. + +| unit | window | events | channel | pedestal | that unit's trigger level | +|---|---|---|---|---|---| +| BE9558 | 2026-05-15 → 05-16 | 193 | Long | +0.335 | 0.200 | +| BE18438 | 2026-02-25 → 02-26 | 64 | Vert | +0.327 | 0.200 | +| BE18438 | 2025-11-15 | 12 | Vert | +0.312 | 0.200 | +| BE9558 | 2026-04-14 / 04-29 / 05-04 | 5 total | Tran | +0.29…+0.44 | 0.200 | + +**It is NOT a decode artifact.** These events reproduce *exactly* in +Blastware's own ASCII export — e.g. `BE12599/N599LQD7.8E0W` Tran reads +mean +0.345, min +0.335, max +0.355 in both our decode and BW's. Any +shape-based false-trigger detector must treat this as a real device +condition, not corrupt data. + +**Detection rule that works:** + +``` +dominant geo axis: |mean| / peak > 0.7 + AND: |mean| >= 0.9 x that unit's geo trigger level +group hits into episodes by serial with a 12 h gap +``` + +A bare `|mean|/peak` threshold is useless — a quiet trace sitting at the +0.010 in/s noise floor clears any ratio test. On the surveyed store the +bare rule flagged 7,184 events; the rule above flags 274, all real. +Candidate list: `scratch/offset_candidates.csv`. + +**What the data rules out.** + +- *Not the geophone.* The on-device sensor check passes on every unit, + offset or not — test frequency 7.2–8.1 Hz, damping ratio 3.3–4.7, + zero failures fleet-wide, including mid-episode. The coil and its + mechanical response are healthy. +- *Not the battery.* 6.6–6.8 V on the affected units, same as the rest. +- *Not environmental.* Only 2 units of 21 ever show it. BE17353 logged + 395 waveform events over nine months with zero occurrences; BE7145, + 295 events, zero. Weather acts on all of them equally. +- *Not condensation.* BE18438's Vert held 0.179 → 0.189 for 34 hours + **including a 10-hour overnight gap**, then stepped to 0.33 at 13:24 + in the afternoon. Dew would peak overnight; it was flat overnight. + +**What the data shows.** The pedestal is *piecewise constant* — it +holds rock-steady, survives power-off gaps, and changes only in discrete +steps. A small common-mode diurnal wobble rides on top of it, but that +wobble is present on the healthy channels too (it is the unit's normal +thermal breathing) and is not the fault. + +**Open: two candidate mechanisms.** + +1. *A latched bad zero.* These units run Sensor Check "Before + monitoring", so a baseline is captured at session start. Disturb the + geophone at that instant and the unit bakes a non-zero reading in as + "zero". Fits the discrete steps, the rock-steadiness between them, + surviving power cycles, the sensor check still passing, and clearing + on a clean re-zero. Also explains why it is episodic rather than + constant — it needs a disturbance at the moment of zeroing. +2. *A degrading analog front-end.* Fits BE9558 better: it was clean for + nine months, then escalated 3 → 1 → 1 → 193 events over five weeks + **and moved from Tran to Long**, which a single bad channel would + not do. + +BE18438 (same channel twice, months apart) looks like (1); BE9558 +(escalating, channel-migrating) looks like (2). The decisive test is to +force a clean re-zero on a faulted unit: if the offset clears and stays +clear it is (1) and fixable in the field — potentially over the wire, +since we already speak SUB `0x0E` (channel sensor data) and `0x98` +(trigger test), see §7.10. + +**Note:** the surveyed population is subject to survivorship bias — +flooded events were routinely deleted before this snapshot, so units +that look clean here may have had episodes that were scrubbed. diff --git a/docs/waveform_codec_re_status.md b/docs/waveform_codec_re_status.md index 47fbc41..1a0b864 100644 --- a/docs/waveform_codec_re_status.md +++ b/docs/waveform_codec_re_status.md @@ -250,6 +250,19 @@ that divided by 32768 read every geophone sample 2.34% low. Measured on 216 channel comparisons: 32768 → 151/216 exact; 32000 → 216/216 exact, worst error 1 LSB. +**Scope — not waveform-specific.** The scale is applied where ADC +counts become physical units, which every event passes through +regardless of source codec: + +| source | median ratio ours/device, 32768 | with 32000 | +|---|---|---| +| series-3 waveform (vs ASCII sample table) | 0.9766 | **1.0000** | +| series-3 histogram (vs ASCII PPV, n=1137) | 0.9766 | **1.0000** | +| series-4 Thor IDF (vs device peak, n=1468) | 0.960 | **0.983** | + +The four block-framing fixes are waveform-only — `histogram_codec` is +untouched by them. + ## Ground-truth corpus (2026-08-25) Beyond the bundled fixtures, the production waveform store keeps each