fix(codec): geo full scale is 32000 counts; 4 walker framing cases; channel-id from header
Two independent bugs, both found by diffing 75 production events against
their preserved Blastware ASCII exports (<store>/<serial>/<file>_ASCII.TXT).
1. Geo full scale was wrong — every geophone reading was 2.34% low.
The codec emits geo samples in 16-count units with a documented LSB of
exactly 0.005 in/s, and decoded_to_adc_counts multiplies by 16, so one
ADC count is 0.005/16 in/s and 10.000 in/s is 10.0/(0.005/16) = 32000
counts. sfm/event_hdf5.py and minimateplus/event_file_io.py both
divided by 32768 (2^15), scaling every sample and derived peak down by
1 - 32000/32768. The error scales with amplitude, so it was invisible
on quiet events and worst on the loud ones that matter for compliance.
Mic is unaffected (it back-solves its scale from the device peak).
216 per-channel comparisons: 32768 -> 151/216 exact, worst error 0.238
in/s on a 10 in/s event; 32000 -> 216/216 exact, worst 0.005 = 1 LSB.
2. walk_body silently truncated channels on four unhandled framing cases.
An unrecognised tag ends the walk and decode_waveform_v2 returns
whatever it got, so this surfaced as short channels, never an error:
- wide-NN RLE `0X NN` (runs longer than 252 samples)
- `30 NN` with NN > 0x10 (the old cap was arbitrary)
- variable-width `40 NN` headers: NN counts previous-channel
continuation deltas, so the header is 2*NN + 16 bytes; `40 01`
and `40 03` occur alongside `40 02`
- tagless segment headers: no `40 NN` tag at all, just the 14-byte
tail [field2:2][len:2][channel_id:4][marker:2][anchors:4]
Also: the header field documented as a "monotonic uint32 LE counter" is
really [channel_id][00][00][segment_index], with 0x46=Tran 0x47=Vert
0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero
disagreements. decode_waveform_v2 now takes the channel from that field
instead of rotation position, which was fragile: one missed header
desynced every channel after it.
parse_segment_header now returns n_prev_deltas/prev_deltas/marker/
anchors/channel/segment_index; the old fixed_pattern (02 00 00 01)
conflated the 2-byte marker with the first anchor.
Ground-truth corpus, end to end through the production path:
exact 37 -> 72, truncated 23 -> 3, full-length value errors 15 -> 0.
Store-wide, 729 of 1388 series-3 waveform events decode differently and
728 gain samples; the scale fix changes float values on all of them, so
stored .h5 files need regenerating.
Still open: 3 events truncate at a header variant with a variable-width
prefix (2/4/6 bytes) before the channel id and an `01 00` marker.
Documented in docs/instantel_protocol_reference.md with byte offsets.
+20 tests. No regressions: the byte-exact fixture suite still passes and
the full-suite failure list is unchanged from baseline (16 pre-existing
failures from gitignored fixtures).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
Claude Opus 5
@@ -1101,14 +1101,51 @@ Every block starts with a 2-byte tag. Five tag types are confirmed:
|
||||
|-----------|-------------------------------------|-----------------------|
|
||||
| ``10 NN`` | Small-delta data block | NN/2 + 2 bytes |
|
||||
| ``20 NN`` | Literal data block (int8-shaped) | NN + 2 bytes |
|
||||
| ``00 NN`` | 2-byte marker between data blocks | 2 bytes |
|
||||
| ``00 NN`` | RLE zero-delta run | 2 bytes |
|
||||
| ``30 NN`` | Trailer summary block | NN × 4 bytes |
|
||||
| ``40 02`` | Segment header | 20 bytes (fixed) |
|
||||
| ``40 NN`` | Segment header | 2 × NN + 16 bytes |
|
||||
|
||||
NN is always a multiple of 4. ``10 NN`` and ``20 NN`` data blocks
|
||||
alternate with ``00 NN`` markers — every ``10/20 NN`` block is
|
||||
followed by a ``00 NN`` marker before the next data block.
|
||||
|
||||
###### Wide-NN forms — ``0X NN`` (CONFIRMED 2026-08-25)
|
||||
|
||||
The 12-bit wide-NN encoding already documented for ``1X NN`` /
|
||||
``2X NN`` (low nibble of the tag byte carries the high nibble of NN,
|
||||
so effective ``NN = ((tag & 0x0F) << 8) | NN``) **also applies to the
|
||||
``00 NN`` RLE tag.** A narrow RLE run maxes out at NN = 0xFC, so a
|
||||
quiet stretch longer than 252 samples must use the wide form.
|
||||
|
||||
Confirmed against six production events, e.g. ``01 0c`` (NN = 268) in
|
||||
``BE9558/K558LKOF.460W``. Before this was handled, the walker hit its
|
||||
unknown-tag break at the first long quiet run and silently truncated
|
||||
every channel decoded after that point.
|
||||
|
||||
###### ``30 NN`` is not capped at NN = 0x10 (CONFIRMED 2026-08-25)
|
||||
|
||||
Data-section ``30 NN`` blocks occur with NN up to at least 0x18 (24),
|
||||
e.g. ``30 18`` in ``BE18193/T193LQ45.NN0W`` and ``30 14`` in
|
||||
``BE18193/T193LQ9W.AF0W``. The data-section length formula
|
||||
(``NN × 1.5 + 2``) holds for these; only the earlier ``NN ≤ 0x10``
|
||||
guard was wrong.
|
||||
|
||||
###### ``40 NN`` segment headers are variable width (CONFIRMED 2026-08-25)
|
||||
|
||||
``40 02`` is the common case, but **NN is the count of int16 BE
|
||||
continuation deltas the header carries for the *previous* channel**, so
|
||||
the header grows with NN and every field after the deltas shifts by
|
||||
``2 × NN``:
|
||||
|
||||
```
|
||||
length = 2 (tag) + 2 × NN (prev-channel deltas) + 14 (fixed tail)
|
||||
```
|
||||
|
||||
``40 01`` (18 bytes) and ``40 03`` (22 bytes) both occur in production
|
||||
files — see ``BE12599/N599LP1S.UO0W`` and ``BE18438/T438LO30.GA0W``.
|
||||
In each case the constant ``02 00`` marker sits at ``data[2×NN+8]`` and
|
||||
the following tag lands exactly on a valid block boundary.
|
||||
|
||||
##### Segments
|
||||
|
||||
The body is divided into segments separated by ``40 02`` segment headers.
|
||||
@@ -1129,20 +1166,110 @@ fit fewer. Observed first-segment sizes in the bundled fixtures:
|
||||
based on incomplete walks; that figure is wrong. Segments are
|
||||
flash-page-sized in bytes, not sample-count-sized.
|
||||
|
||||
The 18-byte ``40 02`` payload structure:
|
||||
The ``40 NN`` payload structure (offsets shown for the common NN=2 /
|
||||
18-byte-payload case; add ``2 × (NN − 2)`` to every offset from ``[4:6]``
|
||||
onward for other widths):
|
||||
|
||||
| Offset | Field | Status |
|
||||
|-----------|---------------------------------------------|-------------|
|
||||
| [0:2] | T_delta at first sample of new segment | ✅ confirmed|
|
||||
| | (int16 BE, in 16-count units) | |
|
||||
| [2:4] | Likely T_delta at sample seg_start+1 | 🟡 likely |
|
||||
| [4:6] | Unknown (varies; possibly a checksum) | ❓ open |
|
||||
| [6:8] | Byte length to next segment header − 2 | ✅ confirmed|
|
||||
| | (uint16 BE; useful for walker pre-scan) | |
|
||||
| [8:12] | Monotonic uint32 LE counter | ✅ confirmed|
|
||||
| | (starts ~0x47, increments by 1 per segment) | |
|
||||
| [12:14] | Constant ``02 00`` | ✅ confirmed|
|
||||
| [14:18] | Unknown 4-byte field | ❓ open |
|
||||
| Offset (NN=2) | Generic | Field | Status |
|
||||
|---------------|------------------|----------------------------------------|-------------|
|
||||
| [0:4] | [0 : 2NN] | NN × int16 BE continuation deltas for | ✅ confirmed|
|
||||
| | | the PREVIOUS channel (16-count units) | |
|
||||
| [4:6] | [2NN : 2NN+2] | Unknown (varies; possibly a checksum) | ❓ open |
|
||||
| [6:8] | [2NN+2 : 2NN+4] | Byte length to next segment header − 2 | 🟡 likely |
|
||||
| | | (uint16 BE; off by ±4 on some files) | |
|
||||
| [8:12] | [2NN+4 : 2NN+8] | Monotonic uint32 LE counter | ✅ confirmed|
|
||||
| | | (starts ~0x47, +1 per segment) | |
|
||||
| [12:14] | [2NN+8 : 2NN+10] | Constant ``02 00`` | ✅ confirmed|
|
||||
| [14:18] | [2NN+10 : 2NN+14]| THIS channel's 2-sample anchor pair | ✅ confirmed|
|
||||
| | | (2 × int16 BE) | |
|
||||
|
||||
⚠️ An earlier draft listed ``[14:18]`` as an "unknown 4-byte field" and
|
||||
``[12:16]`` as a constant ``02 00 00 01``. Both were wrong: the constant
|
||||
is only the 2-byte ``02 00``, and the four bytes after it are the anchor
|
||||
pair the decoder needs. Corrected 2026-08-25.
|
||||
|
||||
###### Tagless segment headers (CONFIRMED 2026-08-25)
|
||||
|
||||
A segment header can appear with **no ``40 NN`` tag at all** — just the
|
||||
14-byte tail:
|
||||
|
||||
```
|
||||
[field2:2][len_to_next:2][channel_id:4][marker:2][anchors:4]
|
||||
```
|
||||
|
||||
This is the NN=0 case: the previous channel needed no continuation
|
||||
deltas, so there is no tag and no delta bytes. Detect it by the six
|
||||
bytes at ``[4:10]`` — a known channel id, two zero bytes, a small
|
||||
segment index, then the ``01 00`` / ``02 00`` marker.
|
||||
|
||||
It is where the walk stopped in 7 of the 8 events that still truncated
|
||||
after the wide-RLE / ``30 NN`` / variable-width-``40 NN`` fixes.
|
||||
|
||||
###### The header "counter" is really a channel id (CONFIRMED 2026-08-25)
|
||||
|
||||
The 4-byte field previously documented as a *"monotonic uint32 LE
|
||||
counter (starts ~0x47, increments by 1 per segment)"* is actually:
|
||||
|
||||
```
|
||||
[channel_id:1][00][00][segment_index:1]
|
||||
```
|
||||
|
||||
| channel_id | channel |
|
||||
|---|---|
|
||||
| ``0x46`` | Tran |
|
||||
| ``0x47`` | Vert |
|
||||
| ``0x48`` | Long |
|
||||
| ``0x49`` | MicL |
|
||||
|
||||
Verified on **1697 of 1697** segment headers across the ground-truth
|
||||
corpus — every one agrees with the channel the rotation would assign,
|
||||
zero disagreements, no other id values observed. The old reading was
|
||||
plausible because the id byte cycles 0x46→0x47→0x48→0x49 and the
|
||||
segment index increments, which *looks* monotonic in LE.
|
||||
|
||||
Decoders should take the channel from this field rather than from
|
||||
rotation position: one missed or extra header (exactly what tagless
|
||||
headers used to cause) desyncs rotation and corrupts every channel
|
||||
after it.
|
||||
|
||||
###### Geophone full scale is 32000 counts, not 32768 (CONFIRMED 2026-08-25)
|
||||
|
||||
The body codec emits geo samples in 16-count units whose LSB is exactly
|
||||
**0.005 in/s**. With the consumer-side ``×16`` to ADC counts, one ADC
|
||||
count is ``0.005 / 16`` in/s, so Normal range (10.000 in/s) is
|
||||
|
||||
```
|
||||
10.0 / (0.005 / 16) = 32000 counts
|
||||
```
|
||||
|
||||
Dividing by 32768 scales every geophone sample and every derived peak
|
||||
down by ``1 - 32000/32768`` = **2.34%**. Measured on 216 per-channel
|
||||
comparisons against preserved Blastware ASCII exports: 32768 gave
|
||||
151/216 exact (worst error 0.238 in/s on a 10 in/s event); 32000 gives
|
||||
216/216 exact with a worst error of 0.005 in/s — exactly 1 LSB, i.e.
|
||||
pure quantization.
|
||||
|
||||
This also explains why Blastware reports geo peaks slightly above
|
||||
nominal full scale (e.g. 10.14 in/s): the ADC has headroom past 32000.
|
||||
|
||||
###### Unmapped: variable-prefix segment descriptors ❓ OPEN
|
||||
|
||||
Three of 75 ground-truth production events still truncate. In each,
|
||||
the walk reaches a segment header whose channel-id field is preceded by
|
||||
a **variable-width prefix** — 2, 4 or 6 bytes have all been observed,
|
||||
where the standard tagless form always has 4 (``field2`` + ``len``).
|
||||
These records also carry the ``01 00`` marker rather than ``02 00``,
|
||||
and appear packed back-to-back with little or no sample data between
|
||||
them.
|
||||
|
||||
The ``01 00`` marker is *not* simply an anchor count: records carrying
|
||||
it have been seen with both 2-byte and 4-byte anchor fields in the same
|
||||
file, so the prefix width and the marker are not yet reconciled.
|
||||
|
||||
The decoder stops cleanly at these rather than emitting garbage.
|
||||
Examples: ``BE12599/N599LPNB.JF0W`` at body offset 1155 (2-byte
|
||||
prefix), ``BE12599/N599LPWJ.980W`` at 849 (6-byte prefix),
|
||||
``BE9558/K558LOF2.820W`` at 1485.
|
||||
|
||||
Examples from event-c (1 sec single-shot):
|
||||
|
||||
|
||||
@@ -102,12 +102,24 @@ correct.
|
||||
| | | nibble first; signed 0..7 / 8..F = -8..-1)|
|
||||
| `20 NN` | NN + 2 bytes | int8 signed deltas (1 per byte) |
|
||||
| `00 NN` | 2 bytes | RLE: append NN copies of current value |
|
||||
| `30 NN` | NN*2 in data section, | Unknown content. Only in loud-from- |
|
||||
| | NN*4 in trailer | start events. |
|
||||
| `40 02` | 20 bytes (fixed) | Segment header |
|
||||
| `30 NN` | NN*1.5 + 2 in data | 12-bit signed deltas (see below). |
|
||||
| | section, NN*4 trailer | |
|
||||
| `40 NN` | 2*NN + 16 bytes | Segment header (NN = prev-channel deltas)|
|
||||
|
||||
NN is always a multiple of 4.
|
||||
|
||||
**Wide-NN forms.** `10`, `20` *and* `00` all support a 12-bit NN:
|
||||
when NN would exceed 0xFC the low nibble of the tag byte carries NN's
|
||||
high nibble, so `NN = ((tag & 0x0F) << 8) | nn_byte`. Confirmed for
|
||||
`1X`/`2X` in 2026-05-11 and for `0X` (RLE) in 2026-08-25 — e.g.
|
||||
`01 0c` = a 268-sample zero-delta run.
|
||||
|
||||
**`40 NN` is variable width.** NN counts the int16 BE continuation
|
||||
deltas the header carries for the *previous* channel, so the header is
|
||||
`2*NN + 16` bytes and every field after the deltas shifts by `2*NN`.
|
||||
`40 01` (18 B) and `40 03` (22 B) both occur alongside the common
|
||||
`40 02` (20 B). Confirmed 2026-08-25.
|
||||
|
||||
Implementation: `walk_body()` in `minimateplus/waveform_codec.py`.
|
||||
|
||||
### 7-byte preamble
|
||||
@@ -207,6 +219,48 @@ TL;DR table above are now locked in by pytest regression tests.
|
||||
still bails out partway through. Lower priority since the other
|
||||
7 events walk cleanly.
|
||||
|
||||
4. **Variable-prefix segment descriptors** (found 2026-08-25).
|
||||
3 of 75 ground-truth production events still truncate. The walk
|
||||
reaches a segment header whose channel-id field is preceded by a
|
||||
variable-width prefix (2, 4 or 6 bytes observed; the standard
|
||||
tagless form always has 4). These also carry an `01 00` marker
|
||||
instead of `02 00`. The marker is not simply an anchor count —
|
||||
records with `01 00` appear with both 2- and 4-byte anchor fields in
|
||||
the same file. Examples: `BE12599/N599LPNB.JF0W` @1155,
|
||||
`BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
|
||||
|
||||
## Segment header: channel id and tagless form — 2026-08-25
|
||||
|
||||
The 4-byte field previously read as a "monotonic uint32 LE counter" is
|
||||
`[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert
|
||||
`0x48`=Long `0x49`=MicL. Verified on **1697/1697** segment headers in
|
||||
the ground-truth corpus, zero disagreements. `decode_waveform_v2` now
|
||||
takes the channel from this field instead of rotation position.
|
||||
|
||||
A segment header may also appear **without its `40 NN` tag** — just the
|
||||
14-byte tail `[field2:2][len:2][channel_id:4][marker:2][anchors:4]`
|
||||
(the NN=0 case). `is_tagless_segment_header()` detects it from the six
|
||||
bytes at `[4:10]`.
|
||||
|
||||
## Geo scale: full scale is 32000 counts — 2026-08-25
|
||||
|
||||
One decoder unit (16 ADC counts) is exactly 0.005 in/s, so Normal range
|
||||
(10.000 in/s) is `10.0 / (0.005/16)` = **32000** ADC counts. Consumers
|
||||
that divided by 32768 read every geophone sample 2.34% low. Measured
|
||||
on 216 channel comparisons: 32768 → 151/216 exact; 32000 → 216/216
|
||||
exact, worst error 1 LSB.
|
||||
|
||||
## Ground-truth corpus (2026-08-25)
|
||||
|
||||
Beyond the bundled fixtures, the production waveform store keeps each
|
||||
event's original Blastware ASCII export at
|
||||
`<store>/<serial>/<filename>_ASCII.TXT`. 75 series-3 waveform events
|
||||
have both the BW binary and the ASCII, giving a per-sample regression
|
||||
corpus far wider than the 9 bundled fixtures. Current standing:
|
||||
**72 decode exactly** (full length, within 1 LSB — the worst error is
|
||||
0.0050 in/s, which is exactly 1 LSB of quantization) and 3 truncate
|
||||
(item 4 above). Zero events have full-length value errors.
|
||||
|
||||
## `30 NN` block format — CRACKED 2026-05-11 late
|
||||
|
||||
The `30 NN` block carries `NN` 12-bit signed deltas, packed as `NN/4`
|
||||
|
||||
Reference in New Issue
Block a user