serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(codec): geo full scale is 32000 counts; 4 walker framing cases; channel-id from header

Browse Source 
Two independent bugs, both found by diffing 75 production events against
their preserved Blastware ASCII exports (<store>/<serial>/<file>_ASCII.TXT).

1. Geo full scale was wrong — every geophone reading was 2.34% low.
   The codec emits geo samples in 16-count units with a documented LSB of
   exactly 0.005 in/s, and decoded_to_adc_counts multiplies by 16, so one
   ADC count is 0.005/16 in/s and 10.000 in/s is 10.0/(0.005/16) = 32000
   counts.  sfm/event_hdf5.py and minimateplus/event_file_io.py both
   divided by 32768 (2^15), scaling every sample and derived peak down by
   1 - 32000/32768.  The error scales with amplitude, so it was invisible
   on quiet events and worst on the loud ones that matter for compliance.
   Mic is unaffected (it back-solves its scale from the device peak).

   216 per-channel comparisons: 32768 -> 151/216 exact, worst error 0.238
   in/s on a 10 in/s event; 32000 -> 216/216 exact, worst 0.005 = 1 LSB.

2. walk_body silently truncated channels on four unhandled framing cases.
   An unrecognised tag ends the walk and decode_waveform_v2 returns
   whatever it got, so this surfaced as short channels, never an error:
     - wide-NN RLE `0X NN` (runs longer than 252 samples)
     - `30 NN` with NN > 0x10 (the old cap was arbitrary)
     - variable-width `40 NN` headers: NN counts previous-channel
       continuation deltas, so the header is 2*NN + 16 bytes; `40 01`
       and `40 03` occur alongside `40 02`
     - tagless segment headers: no `40 NN` tag at all, just the 14-byte
       tail [field2:2][len:2][channel_id:4][marker:2][anchors:4]

Also: the header field documented as a "monotonic uint32 LE counter" is
really [channel_id][00][00][segment_index], with 0x46=Tran 0x47=Vert
0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero
disagreements.  decode_waveform_v2 now takes the channel from that field
instead of rotation position, which was fragile: one missed header
desynced every channel after it.

parse_segment_header now returns n_prev_deltas/prev_deltas/marker/
anchors/channel/segment_index; the old fixed_pattern (02 00 00 01)
conflated the 2-byte marker with the first anchor.

Ground-truth corpus, end to end through the production path:
  exact 37 -> 72, truncated 23 -> 3, full-length value errors 15 -> 0.
Store-wide, 729 of 1388 series-3 waveform events decode differently and
728 gain samples; the scale fix changes float values on all of them, so
stored .h5 files need regenerating.

Still open: 3 events truncate at a header variant with a variable-width
prefix (2/4/6 bytes) before the channel id and an `01 00` marker.
Documented in docs/instantel_protocol_reference.md with byte offsets.

+20 tests.  No regressions: the byte-exact fixture suite still passes and
the full-suite failure list is unchanged from baseline (16 pre-existing
failures from gitignored fixtures).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
serversdown
2026-08-25 08:11:11 +00:00
co-authored by Claude Opus 5
parent 37043a47e9
commit 686ab6e7a6
10 changed files with 645 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/waveform_codec_re_status.md
+57 -3
View File
@@ -102,12 +102,24 @@ correct.
| | | nibble first; signed 0..7 / 8..F = -8..-1)|
| `20 NN` | NN + 2 bytes | int8 signed deltas (1 per byte) |
| `00 NN` | 2 bytes | RLE: append NN copies of current value |
| `30 NN` | NN*2 in data section, | Unknown content. Only in loud-from- |
| | NN*4 in trailer | start events. |
| `40 02` | 20 bytes (fixed) | Segment header |
| `30 NN` | NN*1.5 + 2 in data | 12-bit signed deltas (see below). |
| | section, NN*4 trailer | |
| `40 NN` | 2*NN + 16 bytes | Segment header (NN = prev-channel deltas)|
NN is always a multiple of 4.
**Wide-NN forms.** `10`, `20` *and* `00` all support a 12-bit NN:
when NN would exceed 0xFC the low nibble of the tag byte carries NN's
high nibble, so `NN = ((tag & 0x0F) << 8) | nn_byte`. Confirmed for
`1X`/`2X` in 2026-05-11 and for `0X` (RLE) in 2026-08-25 — e.g.
`01 0c` = a 268-sample zero-delta run.
**`40 NN` is variable width.** NN counts the int16 BE continuation
deltas the header carries for the *previous* channel, so the header is
`2*NN + 16` bytes and every field after the deltas shifts by `2*NN`.
`40 01` (18 B) and `40 03` (22 B) both occur alongside the common
`40 02` (20 B). Confirmed 2026-08-25.
Implementation: `walk_body()` in `minimateplus/waveform_codec.py`.
### 7-byte preamble
@@ -207,6 +219,48 @@ TL;DR table above are now locked in by pytest regression tests.
still bails out partway through. Lower priority since the other
7 events walk cleanly.
4. **Variable-prefix segment descriptors** (found 2026-08-25).
3 of 75 ground-truth production events still truncate. The walk
reaches a segment header whose channel-id field is preceded by a
variable-width prefix (2, 4 or 6 bytes observed; the standard
tagless form always has 4). These also carry an `01 00` marker
instead of `02 00`. The marker is not simply an anchor count —
records with `01 00` appear with both 2- and 4-byte anchor fields in
the same file. Examples: `BE12599/N599LPNB.JF0W` @1155,
`BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
## Segment header: channel id and tagless form — 2026-08-25
The 4-byte field previously read as a "monotonic uint32 LE counter" is
`[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert
`0x48`=Long `0x49`=MicL. Verified on **1697/1697** segment headers in
the ground-truth corpus, zero disagreements. `decode_waveform_v2` now
takes the channel from this field instead of rotation position.
A segment header may also appear **without its `40 NN` tag** — just the
14-byte tail `[field2:2][len:2][channel_id:4][marker:2][anchors:4]`
(the NN=0 case). `is_tagless_segment_header()` detects it from the six
bytes at `[4:10]`.
## Geo scale: full scale is 32000 counts — 2026-08-25
One decoder unit (16 ADC counts) is exactly 0.005 in/s, so Normal range
(10.000 in/s) is `10.0 / (0.005/16)` = **32000** ADC counts. Consumers
that divided by 32768 read every geophone sample 2.34% low. Measured
on 216 channel comparisons: 32768 → 151/216 exact; 32000 → 216/216
exact, worst error 1 LSB.
## Ground-truth corpus (2026-08-25)
Beyond the bundled fixtures, the production waveform store keeps each
event's original Blastware ASCII export at
`<store>/<serial>/<filename>_ASCII.TXT`. 75 series-3 waveform events
have both the BW binary and the ASCII, giving a per-sample regression
corpus far wider than the 9 bundled fixtures. Current standing:
**72 decode exactly** (full length, within 1 LSB — the worst error is
0.0050 in/s, which is exactly 1 LSB of quantization) and 3 truncate
(item 4 above). Zero events have full-length value errors.
## `30 NN` block format — CRACKED 2026-05-11 late
The `30 NN` block carries `NN` 12-bit signed deltas, packed as `NN/4`