serversdownandClaude Opus 5
9bb95003e9
Supersedes the segment-header model entirely, including the fixes made earlier today. Found via multi-agent structural analysis of the 25 files that stalled the walker, then verified independently. Records are self-delimiting: off+2 is a uint16 BE length, next_record = off + 2 + len, and the chain ends on a record whose chan_id is 0x06. off+8 carries a 3-valued mode enum: 02 00 14-byte header, 2 anchors, then CUMULATIVE delta blocks 01 00 10-byte header, no anchors, blocks are ABSOLUTE values 00 03 10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute `40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header. Reading it as a 2*NN + 16 header is what made walks drift — the "variable-prefix segment descriptors" reported earlier today were not a format feature, just walker drift of exactly 4 - (old_stop - true_record_start), on all 25 affected files. Measured on the production snapshot: all four channels equal length 156/1388 -> 1388/1388 ASCII sample-count exact 72/75 -> 75/75 ASCII fully exact 70/75 -> 73/75 device PPV waveform (live) 1288/1306 -> 1306/1306 (mean err 0.00000) device PPV histogram (live) 4434/4459 -> 4458/4459 Also eliminates the walker-over-read class: 24 of those 35 files were histograms that read_blastware_file fed to the waveform codec first; the old walker accepted them and returned garbage (one yielded 98,923 "intervals"), while the record-chain decoder returns None so they fall through to histogram_codec. 00 03 records are DECODED, not skipped. Skipping them silently shifts the time base of everything after them on that channel — BE9558/ K558LOF2.820W had MicL displaced by exactly 512 samples with nothing marking the gap. Footer detection now prefers the 0e 08 candidate whose body yields a chain terminating on 0x06; the signature can occur inside a sample stream. Blast radius 1 file of 1388. The superseded model survives as decode_waveform_legacy, pinned by micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes candidates and keeps whichever yields the most samples, so the new decoder returning None where the old returned garbage changes that heuristic's winner. Deferred until that search uses the record chain. Tests: 253 passed (+11), failure list unchanged from baseline. The 9 tests pinning the superseded model are retargeted at decode_waveform_legacy, which still implements it. NOTE: stored .h5 files need regenerating — nearly all get longer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
351 lines
15 KiB
Markdown
351 lines
15 KiB
Markdown
> ## SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN
|
||
>
|
||
> The tag-dispatch model described in this document — `40 NN` segment headers,
|
||
> tagless headers, channel rotation — is **wrong**. It produced nearly-correct
|
||
> output only because the block table happens to tile the data sections.
|
||
>
|
||
> The body is a chain of self-delimiting per-channel records: `off+2` is a
|
||
> uint16 BE length, `next = off + 2 + len`, and the chain ends on a record whose
|
||
> chan_id is `0x06`. A 3-valued mode enum at `off+8` selects delta / absolute /
|
||
> raw-12-bit semantics. `40 NN` is an ordinary int16 BE data block.
|
||
>
|
||
> See the record-chain section of `docs/instantel_protocol_reference.md` §7.6.1
|
||
> and the implementation in `minimateplus/waveform_codec.py`.
|
||
>
|
||
> Result: all four channels equal length in 1388/1388 files (was 156/1388);
|
||
> ASCII sample-count exact 75/75, fully exact 73/75; device PPV 1306/1306.
|
||
>
|
||
> This document is retained as the reasoning trail.
|
||
|
||
# Waveform body codec — FULLY DECODED (2026-05-11)
|
||
|
||
This is the **clean working note** for the body-codec reverse-engineering
|
||
effort. It supersedes scattered claims elsewhere when they conflict.
|
||
The deep historical record (with retractions, dead ends, and dated
|
||
analyses) lives in `docs/instantel_protocol_reference.md §7.6.1`; the
|
||
authoritative implementation lives in `minimateplus/waveform_codec.py`.
|
||
|
||
## TL;DR
|
||
|
||
**The codec is fully decoded.** Every block type, every channel, every
|
||
event in the fixture bundle decodes byte-exact against BW's ASCII
|
||
export.
|
||
|
||
| Block type | Meaning | Verified |
|
||
|---|---|---|
|
||
| `10 NN` | 4-bit signed nibble deltas | ✅ |
|
||
| `20 NN` | int8 signed deltas | ✅ |
|
||
| `00 NN` | run-length-encoded zero deltas | ✅ |
|
||
| `30 NN` | 12-bit signed packed deltas | ✅ NEW (2026-05-11 late) |
|
||
| `40 02` | segment header (anchor pair + prev-channel extension) | ✅ |
|
||
|
||
Channels rotate **Tran → Vert → Long → MicL** per segment. Each
|
||
channel-segment carries ~512 samples (2-sample anchor pair + 508
|
||
deltas + 2-sample continuation in next segment's header).
|
||
|
||
## What decodes byte-exact today
|
||
|
||
**Every decoded sample across every fixture event matches truth. Zero
|
||
divergences.**
|
||
|
||
| Event | Description | Tran | Vert | Long | Total |
|
||
|---|---|---|---|---|---|
|
||
| event-a (5-8) | quiet, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
|
||
| event-c (5-8) | quiet, 1 sec | 1280 ✓ | 1280 ✓ | 1280 ✓ | 3840 |
|
||
| event-d (5-8) | quiet, 1 sec | 1280 ✓ | 1280 ✓ | 1280 ✓ | 3840 |
|
||
| JQ0 (5-11) | Vert-heavy, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
|
||
| V70 (5-11) | Mic-heavy, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
|
||
| SP0 (5-11) | loud all, 3 sec | 2048 ✓ | 1538 ✓ | 1536 ✓ | 5122 |
|
||
| SS0 (5-11) | loud-from-start | 734 ✓ | 512 ✓ | 512 ✓ | 1758 |
|
||
| SV0 (5-11) | loud-from-start | 1024 ✓ | 578 ✓ | 512 ✓ | 2114 |
|
||
| event-b (5-8) | quiet, 2 sec | 512 ✓ | 226 ✓ | 0 | 738 |
|
||
|
||
That's **47,364 ADC samples decoded byte-exact, zero errors.**
|
||
|
||
Three full 3-sec events (event-a, JQ0, V70) decode end-to-end across
|
||
all three geo channels.
|
||
|
||
The events where fewer samples are decoded (SP0, SS0, SV0, event-b)
|
||
are limited by the walker stopping at certain block-length edge cases,
|
||
not by decoder correctness — every sample the walker reaches is
|
||
correct.
|
||
|
||
## What's still open
|
||
|
||
- **Tail samples on SS0/SV0** — these two events decode all but the
|
||
last 1–7 samples per channel (out of 3079). Likely the same
|
||
"last segment is truncated" pattern. Minor; doesn't affect the
|
||
bulk of the data.
|
||
|
||
## Sample counts (72,972 byte-exact total)
|
||
|
||
| Event | Tran | Vert | Long | Status |
|
||
|---|---|---|---|---|
|
||
| event-a | 3328 | 3328 | 3328 | full |
|
||
| event-b | 2304 | 2304 | 2304 | full |
|
||
| event-c | 1280 | 1280 | 1280 | full |
|
||
| event-d | 1280 | 1280 | 1280 | full |
|
||
| JQ0 | 3328 | 3328 | 3328 | full |
|
||
| V70 | 3328 | 3328 | 3328 | full |
|
||
| SP0 | 3328 | 3328 | 3328 | full |
|
||
| SS0 | 3078 | 3072 | 3072 | minus 1–7 tail samples |
|
||
| SV0 | 3078 | 3072 | 3072 | minus 1–7 tail samples |
|
||
|
||
## What's now wired into production (2026-05-11 late)
|
||
|
||
- **`client.py:_decode_a5_waveform`** — now uses
|
||
`decode_a5_frames(a5_frames)` instead of the broken int16 LE decoder.
|
||
`event.raw_samples` is populated with int16 ADC counts that flow
|
||
through the existing `sfm/event_hdf5.py` scaling pipeline unchanged.
|
||
Legacy decoder is preserved as `_decode_a5_waveform_LEGACY` for
|
||
reference but is not called.
|
||
|
||
- **MicL → dB(L) conversion** — exposed as
|
||
`waveform_codec.mic_count_to_db(count)`. Verified against BW
|
||
display values (count=1 → 81.94 dB; count=813 → 140.14 dB; matches
|
||
the V70 mic-heavy fixture exactly).
|
||
|
||
- **`decode_a5_frames(a5_frames)`** — production entry point that
|
||
reconstructs the BW-binary body from A5 frames (via the new
|
||
`blastware_file.extract_body_bytes` helper) and runs the verified
|
||
codec. Returns the same `raw_samples` dict shape the consumers
|
||
already expect.
|
||
|
||
## What's solved
|
||
|
||
### Block framing
|
||
|
||
| Tag | Length | Meaning |
|
||
|----------|-----------------------|------------------------------------------|
|
||
| `10 NN` | NN/2 + 2 bytes | 4-bit nibble deltas (2 per byte; high |
|
||
| | | nibble first; signed 0..7 / 8..F = -8..-1)|
|
||
| `20 NN` | NN + 2 bytes | int8 signed deltas (1 per byte) |
|
||
| `00 NN` | 2 bytes | RLE: append NN copies of current value |
|
||
| `30 NN` | NN*1.5 + 2 in data | 12-bit signed deltas (see below). |
|
||
| | section, NN*4 trailer | |
|
||
| `40 NN` | 2*NN + 16 bytes | Segment header (NN = prev-channel deltas)|
|
||
|
||
NN is always a multiple of 4.
|
||
|
||
**Wide-NN forms.** `10`, `20` *and* `00` all support a 12-bit NN:
|
||
when NN would exceed 0xFC the low nibble of the tag byte carries NN's
|
||
high nibble, so `NN = ((tag & 0x0F) << 8) | nn_byte`. Confirmed for
|
||
`1X`/`2X` in 2026-05-11 and for `0X` (RLE) in 2026-08-25 — e.g.
|
||
`01 0c` = a 268-sample zero-delta run.
|
||
|
||
**`40 NN` is variable width.** NN counts the int16 BE continuation
|
||
deltas the header carries for the *previous* channel, so the header is
|
||
`2*NN + 16` bytes and every field after the deltas shifts by `2*NN`.
|
||
`40 01` (18 B) and `40 03` (22 B) both occur alongside the common
|
||
`40 02` (20 B). Confirmed 2026-08-25.
|
||
|
||
Implementation: `walk_body()` in `minimateplus/waveform_codec.py`.
|
||
|
||
### 7-byte preamble
|
||
|
||
```
|
||
body[0:3] = 00 02 00 magic
|
||
body[3:5] = Tran[0] int16 BE in 16-count units (LSB = 0.005 in/s)
|
||
body[5:7] = Tran[1] int16 BE in 16-count units
|
||
```
|
||
|
||
### Tran channel, segment 0
|
||
|
||
Segment 0 (everything before the first `40 02`) encodes Tran samples
|
||
only. Starting from preamble anchors Tran[0] and Tran[1], each block
|
||
contributes to a running cumulative:
|
||
|
||
- `10 NN` → append NN nibble-deltas
|
||
- `20 NN` → append NN int8-deltas
|
||
- `00 NN` → append NN copies of current value (RLE)
|
||
- `40 02` → end segment 0
|
||
|
||
Verified byte-exact:
|
||
|
||
| Event | Description | Segment 0 size | Match |
|
||
|---|---|---|---|
|
||
| `M529LL1A.SP0` | Loud, 0.25 s pretrig | 510 | 510/510 ✓ |
|
||
| `M529LL1A.SV0` | Loud from sample 0 | 58 | 58/58 ✓ (stops at first `30 NN`) |
|
||
| `M529LL1A.SS0` | Loud from sample 0 | 42 | 42/42 ✓ (stops at first `30 04`) |
|
||
| `M529LL1L.JQ0` | Vert-heavy | 510 | 510/510 ✓ |
|
||
| `M529LL1L.V70` | Mic-heavy (140 dB) | 510 | 510/510 ✓ |
|
||
|
||
Implementation: `decode_tran_initial()`.
|
||
|
||
### Segment header (`40 02`, 20 bytes total) — REWRITTEN 2026-05-11
|
||
|
||
| Payload offset | Field | Status |
|
||
|---|---|---|
|
||
| [0:2] | Previous-channel delta — 1st extension sample (int16 BE) | ✅ confirmed |
|
||
| [2:4] | Previous-channel delta — 2nd extension sample (int16 BE) | ✅ confirmed |
|
||
| [4:6] | Unknown (likely checksum) | ❓ open |
|
||
| [6:8] | Byte length to next segment header − 2 (uint16 BE) | ✅ confirmed |
|
||
| [8:12] | Monotonic uint32 LE counter (starts ~0x47) | ✅ confirmed |
|
||
| [12:14] | Constant `02 00` | ✅ confirmed |
|
||
| [14:16] | THIS segment's channel — sample 0 anchor (int16 BE, 16-count units) | ✅ confirmed |
|
||
| [16:18] | THIS segment's channel — sample 1 anchor (int16 BE, 16-count units) | ✅ confirmed |
|
||
|
||
**Key insight (2026-05-11 late):** every segment carries 510 main
|
||
samples (2 anchor + 508 deltas) PLUS 2 continuation samples that live
|
||
in the NEXT segment header. So each channel-segment effectively spans
|
||
512 sample-sets. The continuation lives in the next segment because
|
||
the segment header is also a channel-switch point, so it's a natural
|
||
place to "extend the channel we're leaving" before "starting the
|
||
channel we're entering."
|
||
|
||
This is the same structure as the body preamble (which carries
|
||
Tran[0] and Tran[1] as int16 BE) — every channel uses the same
|
||
"2 anchors + delta stream" layout.
|
||
|
||
## Channel rotation — VERIFIED 2026-05-11
|
||
|
||
```
|
||
(initial body) → Tran samples 0..509 (preamble + delta blocks)
|
||
segment 0 hdr ext+anchor → Vert samples 0..511 ← anchor in hdr [14:18]
|
||
segment 1 hdr ext+anchor → Long samples 0..511
|
||
segment 2 hdr ext+anchor → Mic samples 0..511
|
||
segment 3 hdr ext+anchor → Tran samples 510..1021 (continuation)
|
||
segment 4 hdr ext+anchor → Vert samples 512..1023
|
||
segment 5 hdr ext+anchor → Long samples 512..1023
|
||
segment 6 hdr ext+anchor → Mic samples 512..1023
|
||
segment 7 hdr ext+anchor → Tran samples 1022..1533
|
||
...
|
||
```
|
||
|
||
Implementation: `decode_waveform_v2()` returns
|
||
`{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}` with
|
||
each channel's samples in 16-count units. All verified ranges in the
|
||
TL;DR table above are now locked in by pytest regression tests.
|
||
|
||
## What's still open
|
||
|
||
1. **`30 NN` block content.** These blocks appear in high-amplitude
|
||
regions (sample-set deltas exceeding what int8 in `20 NN` can
|
||
express). The decoder currently steps over them, which loses
|
||
precision for the affected samples. Likely a packed multi-byte
|
||
delta format (12-bit or 16-bit per delta) — initial guesses didn't
|
||
match cleanly, needs more careful analysis.
|
||
|
||
2. **MicL decoding.** The mic channel's anchor pair appears in the
|
||
third segment of each rotation cycle in the same format as the
|
||
geo channels, but the BW ASCII export shows mic in dB(L) (~6 dB
|
||
quantization steps), so direct integer comparison against ADC
|
||
units doesn't work. Need to figure out the ADC-counts → dB(L)
|
||
conversion or pull the mic ADC counts from somewhere else in the
|
||
file format.
|
||
|
||
3. **Walker fix for event-b.** The original quiet bundle's event-b
|
||
still bails out partway through. Lower priority since the other
|
||
7 events walk cleanly.
|
||
|
||
4. **Variable-prefix segment descriptors** (found 2026-08-25).
|
||
3 of 75 ground-truth production events still truncate. The walk
|
||
reaches a segment header whose channel-id field is preceded by a
|
||
variable-width prefix (2, 4 or 6 bytes observed; the standard
|
||
tagless form always has 4). These also carry an `01 00` marker
|
||
instead of `02 00`. The marker is not simply an anchor count —
|
||
records with `01 00` appear with both 2- and 4-byte anchor fields in
|
||
the same file. Examples: `BE12599/N599LPNB.JF0W` @1155,
|
||
`BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
|
||
|
||
## Segment header: channel id and tagless form — 2026-08-25
|
||
|
||
The 4-byte field previously read as a "monotonic uint32 LE counter" is
|
||
`[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert
|
||
`0x48`=Long `0x49`=MicL. Verified on **1697/1697** segment headers in
|
||
the ground-truth corpus, zero disagreements. `decode_waveform_v2` now
|
||
takes the channel from this field instead of rotation position.
|
||
|
||
A segment header may also appear **without its `40 NN` tag** — just the
|
||
14-byte tail `[field2:2][len:2][channel_id:4][marker:2][anchors:4]`
|
||
(the NN=0 case). `is_tagless_segment_header()` detects it from the six
|
||
bytes at `[4:10]`.
|
||
|
||
## Geo scale: full scale is 32000 counts — 2026-08-25
|
||
|
||
One decoder unit (16 ADC counts) is exactly 0.005 in/s, so Normal range
|
||
(10.000 in/s) is `10.0 / (0.005/16)` = **32000** ADC counts. Consumers
|
||
that divided by 32768 read every geophone sample 2.34% low. Measured
|
||
on 216 channel comparisons: 32768 → 151/216 exact; 32000 → 216/216
|
||
exact, worst error 1 LSB.
|
||
|
||
**Scope — not waveform-specific.** The scale is applied where ADC
|
||
counts become physical units, which every event passes through
|
||
regardless of source codec:
|
||
|
||
| source | median ratio ours/device, 32768 | with 32000 |
|
||
|---|---|---|
|
||
| series-3 waveform (vs ASCII sample table) | 0.9766 | **1.0000** |
|
||
| series-3 histogram (vs ASCII PPV, n=1137) | 0.9766 | **1.0000** |
|
||
| series-4 Thor IDF (vs device peak, n=1468) | 0.960 | **0.983** |
|
||
|
||
The four block-framing fixes are waveform-only — `histogram_codec` is
|
||
untouched by them.
|
||
|
||
## Ground-truth corpus (2026-08-25)
|
||
|
||
Beyond the bundled fixtures, the production waveform store keeps each
|
||
event's original Blastware ASCII export at
|
||
`<store>/<serial>/<filename>_ASCII.TXT`. 75 series-3 waveform events
|
||
have both the BW binary and the ASCII, giving a per-sample regression
|
||
corpus far wider than the 9 bundled fixtures. Current standing:
|
||
**72 decode exactly** (full length, within 1 LSB — the worst error is
|
||
0.0050 in/s, which is exactly 1 LSB of quantization) and 3 truncate
|
||
(item 4 above). Zero events have full-length value errors.
|
||
|
||
## `30 NN` block format — CRACKED 2026-05-11 late
|
||
|
||
The `30 NN` block carries `NN` 12-bit signed deltas, packed as `NN/4`
|
||
groups of 6 bytes each. Within each 6-byte group:
|
||
|
||
```
|
||
bytes [0:2] = 16 bits = 4 × 4-bit "high nibbles" (MSB-first)
|
||
bytes [2:6] = 4 × int8 "low bytes"
|
||
|
||
For k in 0..3:
|
||
high_nibble = (header_word >> (12 - 4*k)) & 0xF
|
||
raw_12 = (high_nibble << 8) | low_byte[k]
|
||
delta[k] = raw_12 - 0x1000 if raw_12 >= 0x800 else raw_12
|
||
```
|
||
|
||
The block's total length is `NN × 1.5 + 2` bytes (tag included). This
|
||
is what was tripping up the earlier walker, which used `NN × 4` (the
|
||
trailer-section formula) instead.
|
||
|
||
Why 12-bit and not 16-bit: 12-bit signed range is ±2047, which in
|
||
16-count units = ±10.2 in/s — almost exactly the ±10 in/s full-scale
|
||
range of the geophone at Normal range. The codec sizes its widest
|
||
delta to cover the worst-case sample-to-sample change.
|
||
|
||
Verified against all 14 `30 NN` blocks across the bundled fixture
|
||
events. Every delta decodes byte-exact against BW's ASCII export.
|
||
|
||
## Test fixtures
|
||
|
||
Committed under `tests/fixtures/`:
|
||
|
||
- `decode-re-5-8-26/event-a..event-d/`: original quiet bundle (4 events,
|
||
PPV < 1 in/s). These have Tran ≈ 0 throughout, so segment-0 decode
|
||
works but the loud-amplitude tests (preamble anchors, `30 NN`) are
|
||
uninformative.
|
||
- `5-11-26/M529LL1A.{SP0,SS0,SV0}`: loud bundle (PPV 6-7 in/s on all
|
||
channels). These cracked the Tran codec.
|
||
- `5-11-26/M529LL1L.{JQ0,V70}`: targeted captures. JQ0 is Vert-heavy,
|
||
V70 is Mic-heavy (140 dB). These cracked the `00 NN` RLE rule.
|
||
|
||
Each fixture has a `.TXT` Blastware ASCII export as ground truth.
|
||
|
||
## Tests
|
||
|
||
`tests/test_waveform_codec.py` (40 tests, all passing) locks in:
|
||
|
||
- Block framing (5 tag types with correct lengths).
|
||
- Walker contiguity (no gaps or overlaps).
|
||
- Segment header parsing (counter monotonicity, fixed-pattern check).
|
||
- `decode_tran_initial` against ground-truth Tran samples for all
|
||
fixture events.
|
||
|
||
When you crack the next piece, **add fixture tests against ground-truth
|
||
samples** for that piece before moving on. Don't let unverified code
|
||
ship without a regression lock-in.
|