serversdownandClaude Opus 5
686ab6e7a6
Two independent bugs, both found by diffing 75 production events against
their preserved Blastware ASCII exports (<store>/<serial>/<file>_ASCII.TXT).
1. Geo full scale was wrong — every geophone reading was 2.34% low.
The codec emits geo samples in 16-count units with a documented LSB of
exactly 0.005 in/s, and decoded_to_adc_counts multiplies by 16, so one
ADC count is 0.005/16 in/s and 10.000 in/s is 10.0/(0.005/16) = 32000
counts. sfm/event_hdf5.py and minimateplus/event_file_io.py both
divided by 32768 (2^15), scaling every sample and derived peak down by
1 - 32000/32768. The error scales with amplitude, so it was invisible
on quiet events and worst on the loud ones that matter for compliance.
Mic is unaffected (it back-solves its scale from the device peak).
216 per-channel comparisons: 32768 -> 151/216 exact, worst error 0.238
in/s on a 10 in/s event; 32000 -> 216/216 exact, worst 0.005 = 1 LSB.
2. walk_body silently truncated channels on four unhandled framing cases.
An unrecognised tag ends the walk and decode_waveform_v2 returns
whatever it got, so this surfaced as short channels, never an error:
- wide-NN RLE `0X NN` (runs longer than 252 samples)
- `30 NN` with NN > 0x10 (the old cap was arbitrary)
- variable-width `40 NN` headers: NN counts previous-channel
continuation deltas, so the header is 2*NN + 16 bytes; `40 01`
and `40 03` occur alongside `40 02`
- tagless segment headers: no `40 NN` tag at all, just the 14-byte
tail [field2:2][len:2][channel_id:4][marker:2][anchors:4]
Also: the header field documented as a "monotonic uint32 LE counter" is
really [channel_id][00][00][segment_index], with 0x46=Tran 0x47=Vert
0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero
disagreements. decode_waveform_v2 now takes the channel from that field
instead of rotation position, which was fragile: one missed header
desynced every channel after it.
parse_segment_header now returns n_prev_deltas/prev_deltas/marker/
anchors/channel/segment_index; the old fixed_pattern (02 00 00 01)
conflated the 2-byte marker with the first anchor.
Ground-truth corpus, end to end through the production path:
exact 37 -> 72, truncated 23 -> 3, full-length value errors 15 -> 0.
Store-wide, 729 of 1388 series-3 waveform events decode differently and
728 gain samples; the scale fix changes float values on all of them, so
stored .h5 files need regenerating.
Still open: 3 events truncate at a header variant with a variable-width
prefix (2/4/6 bytes) before the channel id and an `01 00` marker.
Documented in docs/instantel_protocol_reference.md with byte offsets.
+20 tests. No regressions: the byte-exact fixture suite still passes and
the full-suite failure list is unchanged from baseline (16 pre-existing
failures from gitignored fixtures).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
319 lines
14 KiB
Markdown
319 lines
14 KiB
Markdown
# Waveform body codec — FULLY DECODED (2026-05-11)
|
||
|
||
This is the **clean working note** for the body-codec reverse-engineering
|
||
effort. It supersedes scattered claims elsewhere when they conflict.
|
||
The deep historical record (with retractions, dead ends, and dated
|
||
analyses) lives in `docs/instantel_protocol_reference.md §7.6.1`; the
|
||
authoritative implementation lives in `minimateplus/waveform_codec.py`.
|
||
|
||
## TL;DR
|
||
|
||
**The codec is fully decoded.** Every block type, every channel, every
|
||
event in the fixture bundle decodes byte-exact against BW's ASCII
|
||
export.
|
||
|
||
| Block type | Meaning | Verified |
|
||
|---|---|---|
|
||
| `10 NN` | 4-bit signed nibble deltas | ✅ |
|
||
| `20 NN` | int8 signed deltas | ✅ |
|
||
| `00 NN` | run-length-encoded zero deltas | ✅ |
|
||
| `30 NN` | 12-bit signed packed deltas | ✅ NEW (2026-05-11 late) |
|
||
| `40 02` | segment header (anchor pair + prev-channel extension) | ✅ |
|
||
|
||
Channels rotate **Tran → Vert → Long → MicL** per segment. Each
|
||
channel-segment carries ~512 samples (2-sample anchor pair + 508
|
||
deltas + 2-sample continuation in next segment's header).
|
||
|
||
## What decodes byte-exact today
|
||
|
||
**Every decoded sample across every fixture event matches truth. Zero
|
||
divergences.**
|
||
|
||
| Event | Description | Tran | Vert | Long | Total |
|
||
|---|---|---|---|---|---|
|
||
| event-a (5-8) | quiet, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
|
||
| event-c (5-8) | quiet, 1 sec | 1280 ✓ | 1280 ✓ | 1280 ✓ | 3840 |
|
||
| event-d (5-8) | quiet, 1 sec | 1280 ✓ | 1280 ✓ | 1280 ✓ | 3840 |
|
||
| JQ0 (5-11) | Vert-heavy, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
|
||
| V70 (5-11) | Mic-heavy, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
|
||
| SP0 (5-11) | loud all, 3 sec | 2048 ✓ | 1538 ✓ | 1536 ✓ | 5122 |
|
||
| SS0 (5-11) | loud-from-start | 734 ✓ | 512 ✓ | 512 ✓ | 1758 |
|
||
| SV0 (5-11) | loud-from-start | 1024 ✓ | 578 ✓ | 512 ✓ | 2114 |
|
||
| event-b (5-8) | quiet, 2 sec | 512 ✓ | 226 ✓ | 0 | 738 |
|
||
|
||
That's **47,364 ADC samples decoded byte-exact, zero errors.**
|
||
|
||
Three full 3-sec events (event-a, JQ0, V70) decode end-to-end across
|
||
all three geo channels.
|
||
|
||
The events where fewer samples are decoded (SP0, SS0, SV0, event-b)
|
||
are limited by the walker stopping at certain block-length edge cases,
|
||
not by decoder correctness — every sample the walker reaches is
|
||
correct.
|
||
|
||
## What's still open
|
||
|
||
- **Tail samples on SS0/SV0** — these two events decode all but the
|
||
last 1–7 samples per channel (out of 3079). Likely the same
|
||
"last segment is truncated" pattern. Minor; doesn't affect the
|
||
bulk of the data.
|
||
|
||
## Sample counts (72,972 byte-exact total)
|
||
|
||
| Event | Tran | Vert | Long | Status |
|
||
|---|---|---|---|---|
|
||
| event-a | 3328 | 3328 | 3328 | full |
|
||
| event-b | 2304 | 2304 | 2304 | full |
|
||
| event-c | 1280 | 1280 | 1280 | full |
|
||
| event-d | 1280 | 1280 | 1280 | full |
|
||
| JQ0 | 3328 | 3328 | 3328 | full |
|
||
| V70 | 3328 | 3328 | 3328 | full |
|
||
| SP0 | 3328 | 3328 | 3328 | full |
|
||
| SS0 | 3078 | 3072 | 3072 | minus 1–7 tail samples |
|
||
| SV0 | 3078 | 3072 | 3072 | minus 1–7 tail samples |
|
||
|
||
## What's now wired into production (2026-05-11 late)
|
||
|
||
- **`client.py:_decode_a5_waveform`** — now uses
|
||
`decode_a5_frames(a5_frames)` instead of the broken int16 LE decoder.
|
||
`event.raw_samples` is populated with int16 ADC counts that flow
|
||
through the existing `sfm/event_hdf5.py` scaling pipeline unchanged.
|
||
Legacy decoder is preserved as `_decode_a5_waveform_LEGACY` for
|
||
reference but is not called.
|
||
|
||
- **MicL → dB(L) conversion** — exposed as
|
||
`waveform_codec.mic_count_to_db(count)`. Verified against BW
|
||
display values (count=1 → 81.94 dB; count=813 → 140.14 dB; matches
|
||
the V70 mic-heavy fixture exactly).
|
||
|
||
- **`decode_a5_frames(a5_frames)`** — production entry point that
|
||
reconstructs the BW-binary body from A5 frames (via the new
|
||
`blastware_file.extract_body_bytes` helper) and runs the verified
|
||
codec. Returns the same `raw_samples` dict shape the consumers
|
||
already expect.
|
||
|
||
## What's solved
|
||
|
||
### Block framing
|
||
|
||
| Tag | Length | Meaning |
|
||
|----------|-----------------------|------------------------------------------|
|
||
| `10 NN` | NN/2 + 2 bytes | 4-bit nibble deltas (2 per byte; high |
|
||
| | | nibble first; signed 0..7 / 8..F = -8..-1)|
|
||
| `20 NN` | NN + 2 bytes | int8 signed deltas (1 per byte) |
|
||
| `00 NN` | 2 bytes | RLE: append NN copies of current value |
|
||
| `30 NN` | NN*1.5 + 2 in data | 12-bit signed deltas (see below). |
|
||
| | section, NN*4 trailer | |
|
||
| `40 NN` | 2*NN + 16 bytes | Segment header (NN = prev-channel deltas)|
|
||
|
||
NN is always a multiple of 4.
|
||
|
||
**Wide-NN forms.** `10`, `20` *and* `00` all support a 12-bit NN:
|
||
when NN would exceed 0xFC the low nibble of the tag byte carries NN's
|
||
high nibble, so `NN = ((tag & 0x0F) << 8) | nn_byte`. Confirmed for
|
||
`1X`/`2X` in 2026-05-11 and for `0X` (RLE) in 2026-08-25 — e.g.
|
||
`01 0c` = a 268-sample zero-delta run.
|
||
|
||
**`40 NN` is variable width.** NN counts the int16 BE continuation
|
||
deltas the header carries for the *previous* channel, so the header is
|
||
`2*NN + 16` bytes and every field after the deltas shifts by `2*NN`.
|
||
`40 01` (18 B) and `40 03` (22 B) both occur alongside the common
|
||
`40 02` (20 B). Confirmed 2026-08-25.
|
||
|
||
Implementation: `walk_body()` in `minimateplus/waveform_codec.py`.
|
||
|
||
### 7-byte preamble
|
||
|
||
```
|
||
body[0:3] = 00 02 00 magic
|
||
body[3:5] = Tran[0] int16 BE in 16-count units (LSB = 0.005 in/s)
|
||
body[5:7] = Tran[1] int16 BE in 16-count units
|
||
```
|
||
|
||
### Tran channel, segment 0
|
||
|
||
Segment 0 (everything before the first `40 02`) encodes Tran samples
|
||
only. Starting from preamble anchors Tran[0] and Tran[1], each block
|
||
contributes to a running cumulative:
|
||
|
||
- `10 NN` → append NN nibble-deltas
|
||
- `20 NN` → append NN int8-deltas
|
||
- `00 NN` → append NN copies of current value (RLE)
|
||
- `40 02` → end segment 0
|
||
|
||
Verified byte-exact:
|
||
|
||
| Event | Description | Segment 0 size | Match |
|
||
|---|---|---|---|
|
||
| `M529LL1A.SP0` | Loud, 0.25 s pretrig | 510 | 510/510 ✓ |
|
||
| `M529LL1A.SV0` | Loud from sample 0 | 58 | 58/58 ✓ (stops at first `30 NN`) |
|
||
| `M529LL1A.SS0` | Loud from sample 0 | 42 | 42/42 ✓ (stops at first `30 04`) |
|
||
| `M529LL1L.JQ0` | Vert-heavy | 510 | 510/510 ✓ |
|
||
| `M529LL1L.V70` | Mic-heavy (140 dB) | 510 | 510/510 ✓ |
|
||
|
||
Implementation: `decode_tran_initial()`.
|
||
|
||
### Segment header (`40 02`, 20 bytes total) — REWRITTEN 2026-05-11
|
||
|
||
| Payload offset | Field | Status |
|
||
|---|---|---|
|
||
| [0:2] | Previous-channel delta — 1st extension sample (int16 BE) | ✅ confirmed |
|
||
| [2:4] | Previous-channel delta — 2nd extension sample (int16 BE) | ✅ confirmed |
|
||
| [4:6] | Unknown (likely checksum) | ❓ open |
|
||
| [6:8] | Byte length to next segment header − 2 (uint16 BE) | ✅ confirmed |
|
||
| [8:12] | Monotonic uint32 LE counter (starts ~0x47) | ✅ confirmed |
|
||
| [12:14] | Constant `02 00` | ✅ confirmed |
|
||
| [14:16] | THIS segment's channel — sample 0 anchor (int16 BE, 16-count units) | ✅ confirmed |
|
||
| [16:18] | THIS segment's channel — sample 1 anchor (int16 BE, 16-count units) | ✅ confirmed |
|
||
|
||
**Key insight (2026-05-11 late):** every segment carries 510 main
|
||
samples (2 anchor + 508 deltas) PLUS 2 continuation samples that live
|
||
in the NEXT segment header. So each channel-segment effectively spans
|
||
512 sample-sets. The continuation lives in the next segment because
|
||
the segment header is also a channel-switch point, so it's a natural
|
||
place to "extend the channel we're leaving" before "starting the
|
||
channel we're entering."
|
||
|
||
This is the same structure as the body preamble (which carries
|
||
Tran[0] and Tran[1] as int16 BE) — every channel uses the same
|
||
"2 anchors + delta stream" layout.
|
||
|
||
## Channel rotation — VERIFIED 2026-05-11
|
||
|
||
```
|
||
(initial body) → Tran samples 0..509 (preamble + delta blocks)
|
||
segment 0 hdr ext+anchor → Vert samples 0..511 ← anchor in hdr [14:18]
|
||
segment 1 hdr ext+anchor → Long samples 0..511
|
||
segment 2 hdr ext+anchor → Mic samples 0..511
|
||
segment 3 hdr ext+anchor → Tran samples 510..1021 (continuation)
|
||
segment 4 hdr ext+anchor → Vert samples 512..1023
|
||
segment 5 hdr ext+anchor → Long samples 512..1023
|
||
segment 6 hdr ext+anchor → Mic samples 512..1023
|
||
segment 7 hdr ext+anchor → Tran samples 1022..1533
|
||
...
|
||
```
|
||
|
||
Implementation: `decode_waveform_v2()` returns
|
||
`{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}` with
|
||
each channel's samples in 16-count units. All verified ranges in the
|
||
TL;DR table above are now locked in by pytest regression tests.
|
||
|
||
## What's still open
|
||
|
||
1. **`30 NN` block content.** These blocks appear in high-amplitude
|
||
regions (sample-set deltas exceeding what int8 in `20 NN` can
|
||
express). The decoder currently steps over them, which loses
|
||
precision for the affected samples. Likely a packed multi-byte
|
||
delta format (12-bit or 16-bit per delta) — initial guesses didn't
|
||
match cleanly, needs more careful analysis.
|
||
|
||
2. **MicL decoding.** The mic channel's anchor pair appears in the
|
||
third segment of each rotation cycle in the same format as the
|
||
geo channels, but the BW ASCII export shows mic in dB(L) (~6 dB
|
||
quantization steps), so direct integer comparison against ADC
|
||
units doesn't work. Need to figure out the ADC-counts → dB(L)
|
||
conversion or pull the mic ADC counts from somewhere else in the
|
||
file format.
|
||
|
||
3. **Walker fix for event-b.** The original quiet bundle's event-b
|
||
still bails out partway through. Lower priority since the other
|
||
7 events walk cleanly.
|
||
|
||
4. **Variable-prefix segment descriptors** (found 2026-08-25).
|
||
3 of 75 ground-truth production events still truncate. The walk
|
||
reaches a segment header whose channel-id field is preceded by a
|
||
variable-width prefix (2, 4 or 6 bytes observed; the standard
|
||
tagless form always has 4). These also carry an `01 00` marker
|
||
instead of `02 00`. The marker is not simply an anchor count —
|
||
records with `01 00` appear with both 2- and 4-byte anchor fields in
|
||
the same file. Examples: `BE12599/N599LPNB.JF0W` @1155,
|
||
`BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
|
||
|
||
## Segment header: channel id and tagless form — 2026-08-25
|
||
|
||
The 4-byte field previously read as a "monotonic uint32 LE counter" is
|
||
`[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert
|
||
`0x48`=Long `0x49`=MicL. Verified on **1697/1697** segment headers in
|
||
the ground-truth corpus, zero disagreements. `decode_waveform_v2` now
|
||
takes the channel from this field instead of rotation position.
|
||
|
||
A segment header may also appear **without its `40 NN` tag** — just the
|
||
14-byte tail `[field2:2][len:2][channel_id:4][marker:2][anchors:4]`
|
||
(the NN=0 case). `is_tagless_segment_header()` detects it from the six
|
||
bytes at `[4:10]`.
|
||
|
||
## Geo scale: full scale is 32000 counts — 2026-08-25
|
||
|
||
One decoder unit (16 ADC counts) is exactly 0.005 in/s, so Normal range
|
||
(10.000 in/s) is `10.0 / (0.005/16)` = **32000** ADC counts. Consumers
|
||
that divided by 32768 read every geophone sample 2.34% low. Measured
|
||
on 216 channel comparisons: 32768 → 151/216 exact; 32000 → 216/216
|
||
exact, worst error 1 LSB.
|
||
|
||
## Ground-truth corpus (2026-08-25)
|
||
|
||
Beyond the bundled fixtures, the production waveform store keeps each
|
||
event's original Blastware ASCII export at
|
||
`<store>/<serial>/<filename>_ASCII.TXT`. 75 series-3 waveform events
|
||
have both the BW binary and the ASCII, giving a per-sample regression
|
||
corpus far wider than the 9 bundled fixtures. Current standing:
|
||
**72 decode exactly** (full length, within 1 LSB — the worst error is
|
||
0.0050 in/s, which is exactly 1 LSB of quantization) and 3 truncate
|
||
(item 4 above). Zero events have full-length value errors.
|
||
|
||
## `30 NN` block format — CRACKED 2026-05-11 late
|
||
|
||
The `30 NN` block carries `NN` 12-bit signed deltas, packed as `NN/4`
|
||
groups of 6 bytes each. Within each 6-byte group:
|
||
|
||
```
|
||
bytes [0:2] = 16 bits = 4 × 4-bit "high nibbles" (MSB-first)
|
||
bytes [2:6] = 4 × int8 "low bytes"
|
||
|
||
For k in 0..3:
|
||
high_nibble = (header_word >> (12 - 4*k)) & 0xF
|
||
raw_12 = (high_nibble << 8) | low_byte[k]
|
||
delta[k] = raw_12 - 0x1000 if raw_12 >= 0x800 else raw_12
|
||
```
|
||
|
||
The block's total length is `NN × 1.5 + 2` bytes (tag included). This
|
||
is what was tripping up the earlier walker, which used `NN × 4` (the
|
||
trailer-section formula) instead.
|
||
|
||
Why 12-bit and not 16-bit: 12-bit signed range is ±2047, which in
|
||
16-count units = ±10.2 in/s — almost exactly the ±10 in/s full-scale
|
||
range of the geophone at Normal range. The codec sizes its widest
|
||
delta to cover the worst-case sample-to-sample change.
|
||
|
||
Verified against all 14 `30 NN` blocks across the bundled fixture
|
||
events. Every delta decodes byte-exact against BW's ASCII export.
|
||
|
||
## Test fixtures
|
||
|
||
Committed under `tests/fixtures/`:
|
||
|
||
- `decode-re-5-8-26/event-a..event-d/`: original quiet bundle (4 events,
|
||
PPV < 1 in/s). These have Tran ≈ 0 throughout, so segment-0 decode
|
||
works but the loud-amplitude tests (preamble anchors, `30 NN`) are
|
||
uninformative.
|
||
- `5-11-26/M529LL1A.{SP0,SS0,SV0}`: loud bundle (PPV 6-7 in/s on all
|
||
channels). These cracked the Tran codec.
|
||
- `5-11-26/M529LL1L.{JQ0,V70}`: targeted captures. JQ0 is Vert-heavy,
|
||
V70 is Mic-heavy (140 dB). These cracked the `00 NN` RLE rule.
|
||
|
||
Each fixture has a `.TXT` Blastware ASCII export as ground truth.
|
||
|
||
## Tests
|
||
|
||
`tests/test_waveform_codec.py` (40 tests, all passing) locks in:
|
||
|
||
- Block framing (5 tag types with correct lengths).
|
||
- Walker contiguity (no gaps or overlaps).
|
||
- Segment header parsing (counter monotonicity, fixed-pattern check).
|
||
- `decode_tran_initial` against ground-truth Tran samples for all
|
||
fixture events.
|
||
|
||
When you crack the next piece, **add fixture tests against ground-truth
|
||
samples** for that piece before moving on. Don't let unverified code
|
||
ship without a regression lock-in.
|