2026-08-27 13:43:01 -04:00
3 changed files with 189 additions and 5 deletions
@@ -223,6 +223,44 @@ custom delta + RLE + variable-width codec.
  `NN + 2` for int8 blocks).  Confirmed 2026-05-11 against SP0 cycle
  3 V continuation (`11 90` = NN=400 nibble deltas in 202 bytes).
 ### Framing cases added 2026-05-11 → 2026-08-25
 Four more block-framing cases, each of which had been causing **silent
 channel truncation** — `walk_body` ends its loop on an unrecognised tag
 and `decode_waveform_v2` returns whatever channels it got, so an
 unhandled tag surfaces as short channels with no error raised.  Found by
 diffing 75 production events against their preserved Blastware ASCII
 exports (`<store>/<serial>/<file>_ASCII.TXT`).
 - **Wide-NN RLE `0X NN`** — the 12-bit NN encoding documented above for
  `1X`/`2X` **also applies to the `00 NN` RLE tag**.  A narrow run maxes
  out at NN=0xFC, so a quiet stretch longer than 252 samples must use
  the wide form (e.g. `01 0c` = 268 repeats).
 - **`30 NN` is not capped at NN=0x10** — data-section blocks reach at
  least NN=0x18.  The `NN × 1.5 + 2` length formula was already right;
  only the guard was wrong.
 - **`40 NN` segment headers are variable width** — NN is the *count of
  int16 BE continuation deltas for the PREVIOUS channel*, so the header
  is `2*NN + 16` bytes and every field after the deltas shifts by
  `2*NN`.  `40 01` (18 B) and `40 03` (22 B) both occur alongside the
  common `40 02` (20 B).
 - **Tagless segment headers** — a header can appear with **no `40 NN`
  tag at all**: just the 14-byte tail
  `[field2:2][len:2][channel_id:4][marker:2][anchors:4]`.  This is the
  NN=0 case (previous channel needed no continuation deltas).
 **The header "counter" is really a channel id.**  The 4-byte field long
 documented as a "monotonic uint32 LE counter" is
 `[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert
 `0x48`=Long `0x49`=MicL — verified on **1697/1697** segment headers
 across the corpus, zero disagreements.  `decode_waveform_v2` now takes
 the channel from this field rather than from rotation position; a single
 missed or extra header (exactly what tagless headers caused) desyncs
 rotation and corrupts every channel after it.
 Corpus result, end to end through the production path:
 **exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.**
 ### What's NOT solved
 - **MicL channel conversion to dB(L)** — the codec emits MicL as
@@ -230,9 +268,29 @@ custom delta + RLE + variable-width codec.
  shows mic in dB(L) with ~6 dB quantization steps.  Need to map
  ADC counts → dB(L) for direct comparison; likely
  `dB = 20*log10(|counts|) + offset` or similar.
- **Walker edge cases** — SP0/SS0/SV0 don't walk the full event due
+- **Variable-prefix segment descriptors** — 3 of the 75 ground-truth
-  to block-length quirks past the first few segments.  Every sample
+  production events still truncate.  The walk reaches a segment header
-  reached is correct; the walker just needs robustness improvements.
+  whose channel-id field is preceded by a *variable-width* prefix (2, 4
  or 6 bytes observed; the standard tagless form always has 4), carrying
  an `01 00` marker instead of `02 00`.  The marker is **not** simply an
  anchor count — `01 00` records appear with both 2- and 4-byte anchor
  fields in the same file.  Examples: `BE12599/N599LPNB.JF0W` @1155,
  `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
 - **Histogram codec misses the peak interval** — 301 of 1141 series-3
  histogram channels (26%) decode a peak below 95% of the
  device-reported PPV (one reads 0.0300 against a device PPV of 0.1200,
  exactly 1/4).  Not a scale error — that would be a uniform 2.34%.
  Lives in `histogram_codec.decode_histogram_body`, untouched by the
  2026-08-25 waveform pass.
 - **Micromate (UM-series) IDF decode is ~1000× low** — e.g.
  `UM11402_20260406130113.IDFW` gives a Tran peak of 0.0009 in/s against
  a device-reported 1.1168.  The Thor IDF path decodes sanely, so this
  is UM-specific.
 - **Thor IDF per-count LSB** — after the 32000 geo full-scale
  correction, series-4 Thor peaks sit at a median 0.983 of the
  device-reported peak (was 0.960 under 32768).  Closer but not exact;
  Thor likely uses its own per-count LSB rather than the BW
  16-count/0.005 in/s convention.
 ### Decoded sample counts (across the fixture bundle)
@@ -11,6 +11,7 @@
 | Date | Section | Change |
 |---|---|---|
 | 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. |
 | 2026-05-20 | §2, §3, §4.2, §5.1, §5.3, §6, §7.5b, §7.6.1, §7.6.3, §7.6.4, §7.7.2, §7.7.3, §7.7.5, §7.8.4, §7.8.7, §7.9, §8, §11, §12, §13, §14, §15, Appendix D | **DOC AUDIT PASS — accuracy sweep against `CLAUDE.md` + `minimateplus/` code.** Fixed: (1) S3 frames terminate on bare ETX, not DLE+ETX — §2/§3 rewritten. (2) §3 payload layout corrected — byte[1]=flags, byte[2]=SUB (was wrongly labelled DLE/ADDR). (3) §4.2 — probe responses do NOT carry data length; lengths are hardcoded `DATA_LENGTHS` constants. (4) §5.1 — removed stale duplicate "SUB 1C = TRIGGER CONFIG READ" row; SUB 0A lengths corrected from `0x30/0x26` to `0x46/0x2C` (real event / boundary marker). (5) §5.3 — added missing write-frame format (BW_CMD-only doubling, DLE-aware checksum, offset formula, ack format, SUB 71 chunk parameters). (6) §6 — fixed "SUB 06 → channel config read" → event storage range. (7) §7.5b / §8 — added the 10-byte `sub_code=0x03` continuous-mode timestamp variant alongside the 9-byte single-shot layout; peak vector sum location corrected from "fixed offset 87" to `tran_pos − 12` (label-relative). (8) §7.6 / §7.6.1 / §7.6.3 / §7.6.4 — switched compliance-anchor convention from the 10-byte form to the canonical 6-byte `\xbe\x80\x00\x00\x00\x00`; recording_mode confirmed at anchor−8 in BOTH read and write (was wrongly listed as anchor−3 write / anchor−4 read); sample_rate at anchor−6, histogram_interval at anchor−4, record_time at anchor+6; geo_range row added at channel_label+33. (9) §7.7.2 — token byte position corrected from `params[6]` to `params[7]`. (10) §7.8.4 — fi==9 skip marked FIXED (already removed from code); chunk-count totals updated. (11) §7.8.7 — TODO replaced with current state of `_decode_a5_metadata_into`. (12) §7.9 — Histogram Interval upgraded ❓ → ✅. (13) §11 — POLL example wire bytes corrected; SUB 5A row added to checksum table. (14) §13 — device-under-test updated for current primary unit (BE11529 / S338.17). (15) §14 — TCP Idle Timeout fixed (0→2 min); Data Forwarding Timeout units clarified. (16) §15 (renumbered from second §14) — open-question items already resolved in CLAUDE.md closed out. (17) Appendix D — extension taxonomy rewritten to reflect the AB0T timestamp encoding (D.5.2/D.5.3); EXTENSION REFUTED warning replaced with the resolved encoding. |
 | 2026-05-08 | §7.6.1 (RETRACTION) | **❌ RETRACTED — "raw int16 LE 8 bytes/sample-set" body codec was never validated.** The original 4-2-26 confirmation was based on misreading broken-decoder output (full-scale ±32K noise) as evidence the signal had saturated. BW's own 0C peaks for that capture (Tran=0.420 / Vert=3.870 / Long=0.495 in/s) prove the signal was NOT saturated — none of those exceed 13K ADC counts. No event in the project's archive has ever come close to saturation, yet the decoder consistently produces ±32K noise on every event. Conclusion: the body codec is not raw int16 LE; the actual encoding is open. Body byte distribution is heavily skewed (24% `0x00`, 10.5% `0x10`, lots of `10 XX` pairs) — likely a delta encoding with `0x10` as escape, but unverified. Retraction box added at top of §7.6.1; "fully-saturating event" claim removed from channel-identification note. The histogram codec in §7.6.2 IS verified and decoded correctly (different recording mode, 32-byte blocks); use it as a structural hint when reverse-engineering the waveform codec. |
 | 2026-02-26 | Initial | Document created from first hex dump analysis |
@@ -1252,6 +1253,20 @@ pure quantization.
 This also explains why Blastware reports geo peaks slightly above
 nominal full scale (e.g. 10.14 in/s): the ADC has headroom past 32000.
 **Scope — this is not waveform-specific.**  The scale is applied where
 ADC counts become physical units, which every event passes through
 regardless of which codec produced the samples.  Verified after
 re-deriving the whole production store:
 | source | median ratio ours/device, 32768 | with 32000 |
 |---|---|---|
 | series-3 waveform (vs ASCII sample table) | 0.9766 | **1.0000** |
 | series-3 histogram (vs ASCII PPV, n=1137) | 0.9766 | **1.0000** |
 | series-4 Thor IDF (vs device peak, n=1468) | 0.960 | **0.983** |
 The series-4 figure is closer to correct but not exact — the Thor
 per-count LSB is its own open question (see §15).
 ###### Unmapped: variable-prefix segment descriptors ❓ OPEN
 Three of 75 ground-truth production events still truncate.  In each,
@@ -2994,7 +3009,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
 |---|---|---|---|
 | Timestamp 6-byte format byte[3] — purpose of the separator `0x00` byte | LOW | 2026-02-26 | Not blocking; 9-byte waveform record format (§8.2) fully confirmed without this byte. |
 | `trail[0]` in serial number response — unit-specific byte, derivation unknown. `trail[1]` resolved as firmware minor version. | MEDIUM | 2026-02-26 | |
-| Full channel ID mapping in SUB `5A` stream (01/02/03/04 → which sensor?) | MEDIUM | 2026-02-26 | |
+| ~~Full channel ID mapping in SUB `5A` stream~~ — **RESOLVED 2026-08-25 for the waveform body:** every `40 NN` segment header carries `[channel_id][00][00][segment_index]` at `data[2*NN+4]`, with `0x46`=Tran `0x47`=Vert `0x48`=Long `0x49`=MicL. Verified on 1697/1697 segment headers with zero disagreements against the decoded channel rotation. See §7.6.1. | RESOLVED | 2026-02-26 | Resolved 2026-08-25 |
 | ~~Exact byte boundaries of project string fields in SUB `71` write frame~~ — **RESOLVED 2026-05-05:** project/client/operator/seis-loc/extended-notes come from SUB 5A metadata pages at counter `0x1002` / `0x1004` (§7.8.7), NOT from the SUB 71 write payload. `_decode_a5_metadata_into` locates them via ASCII label scans. | RESOLVED | 2026-02-26 | Resolved 2026-05-05 |
 | Purpose of SUB `09` / response `F6` — 202-byte read block | MEDIUM | 2026-02-26 | |
 | Purpose of SUB `2E` / response `D1` — 26-byte read block | MEDIUM | 2026-02-26 | |
@@ -3021,6 +3036,10 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
 | **ACH inbound server — RESOLVED.** `bridges/ach_server.py` implements full inbound ACH pipeline. `--clear-after-download` flag for delete-after-upload workflow. Post-erase key-reuse detection via `max_downloaded_key` high-water mark. | RESOLVED | 2026-04-11 | |
 | **Sensor Check dropdown byte location** — byte offset in 1A compliance config payload for the "Sensor Check: Before monitoring / After each event / Disabled" setting is NOT YET LOCATED. Confirmed: unit always runs with "Before monitoring" set. Need a capture with "Disabled" to diff. | MEDIUM | 2026-04-08 | Still open |
 | **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open |
 | **Variable-prefix segment descriptors** — 3 of 75 ground-truth events still truncate. The walk reaches a segment header whose channel-id field is preceded by a *variable-width* prefix (2, 4 or 6 bytes observed; the standard tagless form always has 4), carrying an `01 00` marker instead of `02 00`. The marker is **not** simply an anchor count — `01 00` records appear with both 2- and 4-byte anchor fields in the same file, so prefix width and marker are not yet reconciled. Examples: `BE12599/N599LPNB.JF0W` @1155, `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485. | MEDIUM | 2026-08-25 | Still open |
 | **Histogram codec misses the peak interval** — 301 of 1141 series-3 histogram channels (26%) decode a peak below 95% of the device-reported PPV; one example reads 0.0300 against a device PPV of 0.1200 (exactly 1/4). This is not a scale error (that would be a uniform 2.34%) — the interval carrying the peak is being missed or mis-attributed. Affects `histogram_codec.decode_histogram_body`, untouched by the 2026-08-25 waveform pass. | MEDIUM | 2026-08-25 | Still open |
 | **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open |
 | **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open |
 ---
@@ -3446,4 +3465,98 @@ file[anc−3]: histogram_interval_LO
 *All findings reverse-engineered from live RS-232 bridge captures.*  
 *Cross-referenced from 2026-03-02 with Instantel MiniMate Plus Operator Manual (716U0101 Rev 15).*  
-*This is a living document — append changelog entries and timestamps as new findings are confirmed or corrected.*
+*This is a living document — append changelog entries and timestamps as new findings are confirmed or corrected.*
 ---
 ## Appendix E — Known Device Faults (field-observed, 2026-08-25)
 This appendix records *device* behaviour, not protocol.  It exists
 because these signatures are easy to mistake for decoder bugs — and one
 of them was, for a while.
 ### E.1 The "offset" fault
 **Symptom.**  One geophone channel's baseline steps away from zero and
 stays there.  The trace still carries the real AC signal, but it rides
 on a DC pedestal of a few tenths of an in/s.  Operators call this an
 "offset"; it is a known recurring hardware fault, historically resolved
 by returning the unit to Instantel, shelving it until calibration, or a
 per-channel re-zero in an advanced/professional Blastware build.
 **Why it floods the queue.**  The pedestal exceeds the unit's own geo
 trigger level, so the channel sits permanently above threshold and the
 unit retriggers as fast as it can rearm — one event every 3–7 minutes
 for as long as the fault persists.  In the surveyed snapshot the worst
 episode produced 193 events in 38 hours.
 | unit | window | events | channel | pedestal | that unit's trigger level |
 |---|---|---|---|---|---|
 | BE9558 | 2026-05-15 → 05-16 | 193 | Long | +0.335 | 0.200 |
 | BE18438 | 2026-02-25 → 02-26 | 64 | Vert | +0.327 | 0.200 |
 | BE18438 | 2025-11-15 | 12 | Vert | +0.312 | 0.200 |
 | BE9558 | 2026-04-14 / 04-29 / 05-04 | 5 total | Tran | +0.29…+0.44 | 0.200 |
 **It is NOT a decode artifact.**  These events reproduce *exactly* in
 Blastware's own ASCII export — e.g. `BE12599/N599LQD7.8E0W` Tran reads
 mean +0.345, min +0.335, max +0.355 in both our decode and BW's.  Any
 shape-based false-trigger detector must treat this as a real device
 condition, not corrupt data.
 **Detection rule that works:**
 ```
 dominant geo axis:  |mean| / peak  > 0.7
              AND:  |mean| >= 0.9 x that unit's geo trigger level
 group hits into episodes by serial with a 12 h gap
 ```
 A bare `|mean|/peak` threshold is useless — a quiet trace sitting at the
 0.010 in/s noise floor clears any ratio test.  On the surveyed store the
 bare rule flagged 7,184 events; the rule above flags 274, all real.
 Candidate list: `scratch/offset_candidates.csv`.
 **What the data rules out.**
 - *Not the geophone.*  The on-device sensor check passes on every unit,
  offset or not — test frequency 7.2–8.1 Hz, damping ratio 3.3–4.7,
  zero failures fleet-wide, including mid-episode.  The coil and its
  mechanical response are healthy.
 - *Not the battery.*  6.6–6.8 V on the affected units, same as the rest.
 - *Not environmental.*  Only 2 units of 21 ever show it.  BE17353 logged
  395 waveform events over nine months with zero occurrences; BE7145,
  295 events, zero.  Weather acts on all of them equally.
 - *Not condensation.*  BE18438's Vert held 0.179 → 0.189 for 34 hours
  **including a 10-hour overnight gap**, then stepped to 0.33 at 13:24
  in the afternoon.  Dew would peak overnight; it was flat overnight.
 **What the data shows.**  The pedestal is *piecewise constant* — it
 holds rock-steady, survives power-off gaps, and changes only in discrete
 steps.  A small common-mode diurnal wobble rides on top of it, but that
 wobble is present on the healthy channels too (it is the unit's normal
 thermal breathing) and is not the fault.
 **Open: two candidate mechanisms.**
 1. *A latched bad zero.*  These units run Sensor Check "Before
   monitoring", so a baseline is captured at session start.  Disturb the
   geophone at that instant and the unit bakes a non-zero reading in as
   "zero".  Fits the discrete steps, the rock-steadiness between them,
   surviving power cycles, the sensor check still passing, and clearing
   on a clean re-zero.  Also explains why it is episodic rather than
   constant — it needs a disturbance at the moment of zeroing.
 2. *A degrading analog front-end.*  Fits BE9558 better: it was clean for
   nine months, then escalated 3 → 1 → 1 → 193 events over five weeks
   **and moved from Tran to Long**, which a single bad channel would
   not do.
 BE18438 (same channel twice, months apart) looks like (1); BE9558
 (escalating, channel-migrating) looks like (2).  The decisive test is to
 force a clean re-zero on a faulted unit: if the offset clears and stays
 clear it is (1) and fixable in the field — potentially over the wire,
 since we already speak SUB `0x0E` (channel sensor data) and `0x98`
 (trigger test), see §7.10.
 **Note:** the surveyed population is subject to survivorship bias —
 flooded events were routinely deleted before this snapshot, so units
 that look clean here may have had episodes that were scrubbed.
@@ -250,6 +250,19 @@ that divided by 32768 read every geophone sample 2.34% low.  Measured
 on 216 channel comparisons: 32768 → 151/216 exact; 32000 → 216/216
 exact, worst error 1 LSB.
 **Scope — not waveform-specific.**  The scale is applied where ADC
 counts become physical units, which every event passes through
 regardless of source codec:
 | source | median ratio ours/device, 32768 | with 32000 |
 |---|---|---|
 | series-3 waveform (vs ASCII sample table) | 0.9766 | **1.0000** |
 | series-3 histogram (vs ASCII PPV, n=1137) | 0.9766 | **1.0000** |
 | series-4 Thor IDF (vs device peak, n=1468) | 0.960 | **0.983** |
 The four block-framing fixes are waveform-only — `histogram_codec` is
 untouched by them.
 ## Ground-truth corpus (2026-08-25)
 Beyond the bundled fixtures, the production waveform store keeps each