serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(codec): geo full scale is 32000 counts; 4 walker framing cases; channel-id from header

Browse Source 
Two independent bugs, both found by diffing 75 production events against
their preserved Blastware ASCII exports (<store>/<serial>/<file>_ASCII.TXT).

1. Geo full scale was wrong — every geophone reading was 2.34% low.
   The codec emits geo samples in 16-count units with a documented LSB of
   exactly 0.005 in/s, and decoded_to_adc_counts multiplies by 16, so one
   ADC count is 0.005/16 in/s and 10.000 in/s is 10.0/(0.005/16) = 32000
   counts.  sfm/event_hdf5.py and minimateplus/event_file_io.py both
   divided by 32768 (2^15), scaling every sample and derived peak down by
   1 - 32000/32768.  The error scales with amplitude, so it was invisible
   on quiet events and worst on the loud ones that matter for compliance.
   Mic is unaffected (it back-solves its scale from the device peak).

   216 per-channel comparisons: 32768 -> 151/216 exact, worst error 0.238
   in/s on a 10 in/s event; 32000 -> 216/216 exact, worst 0.005 = 1 LSB.

2. walk_body silently truncated channels on four unhandled framing cases.
   An unrecognised tag ends the walk and decode_waveform_v2 returns
   whatever it got, so this surfaced as short channels, never an error:
     - wide-NN RLE `0X NN` (runs longer than 252 samples)
     - `30 NN` with NN > 0x10 (the old cap was arbitrary)
     - variable-width `40 NN` headers: NN counts previous-channel
       continuation deltas, so the header is 2*NN + 16 bytes; `40 01`
       and `40 03` occur alongside `40 02`
     - tagless segment headers: no `40 NN` tag at all, just the 14-byte
       tail [field2:2][len:2][channel_id:4][marker:2][anchors:4]

Also: the header field documented as a "monotonic uint32 LE counter" is
really [channel_id][00][00][segment_index], with 0x46=Tran 0x47=Vert
0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero
disagreements.  decode_waveform_v2 now takes the channel from that field
instead of rotation position, which was fragile: one missed header
desynced every channel after it.

parse_segment_header now returns n_prev_deltas/prev_deltas/marker/
anchors/channel/segment_index; the old fixed_pattern (02 00 00 01)
conflated the 2-byte marker with the first anchor.

Ground-truth corpus, end to end through the production path:
  exact 37 -> 72, truncated 23 -> 3, full-length value errors 15 -> 0.
Store-wide, 729 of 1388 series-3 waveform events decode differently and
728 gain samples; the scale fix changes float values on all of them, so
stored .h5 files need regenerating.

Still open: 3 events truncate at a header variant with a variable-width
prefix (2/4/6 bytes) before the channel id and an `01 00` marker.
Documented in docs/instantel_protocol_reference.md with byte offsets.

+20 tests.  No regressions: the byte-exact fixture suite still passes and
the full-suite failure list is unchanged from baseline (16 pre-existing
failures from gitignored fixtures).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
serversdown
2026-08-25 08:11:11 +00:00
co-authored by Claude Opus 5
parent 37043a47e9
commit 686ab6e7a6
10 changed files with 645 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
minimateplus/event_file_io.py
+7 -2
View File
@@ -659,6 +659,11 @@ def file_sha256(path: Union[str, Path], chunk_size: int = 65536) -> str:
_GEO_NORMAL_FS_INS = 10.0
_GEO_SENSITIVE_FS_INS = 1.250
_INT16_FS = 32768.0
# Geophone full-scale count — 32000, not 32768. One decoder unit (16 ADC
# counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Must match
# sfm.event_hdf5._GEO_INT16_FS or sidecar peaks disagree with the plotted
# waveform by 2.3%. Confirmed 2026-08-25 against the BW ASCII corpus.
_GEO_INT16_FS = 32000.0
# Microphone scale factor, psi per ADC count. Approximate — exact factor
# depends on the geophone-vs-mic ADC scaling and the firmware reference.
@@ -728,7 +733,7 @@ def _peaks_from_samples(samples: dict[str, list[int]]) -> PeakValues:
if not ch:
return 0.0
m = max(abs(int(v)) for v in ch)
return m / _INT16_FS * _GEO_NORMAL_FS_INS
return m / _GEO_INT16_FS * _GEO_NORMAL_FS_INS
tran = _peak_ins(samples.get("Tran", []))
vert = _peak_ins(samples.get("Vert", []))
@@ -742,7 +747,7 @@ def _peaks_from_samples(samples: dict[str, list[int]]) -> PeakValues:
pvs = 0.0
n = min(len(samples.get("Tran", [])), len(samples.get("Vert", [])), len(samples.get("Long", [])))
if n:
scale = _GEO_NORMAL_FS_INS / _INT16_FS
scale = _GEO_NORMAL_FS_INS / _GEO_INT16_FS
T = samples["Tran"]; V = samples["Vert"]; L = samples["Long"]
for i in range(n):
t = T[i] * scale
minimateplus/waveform_codec.py
+134 -37
View File
@@ -166,8 +166,14 @@ def find_data_start(body: bytes) -> int:
# Try fixed offset 7 first (canonical preamble length).
if len(body) >= 9:
b, nn = body[7], body[8]
if (b in (0x00, 0x10, 0x20, 0x30) and nn % 4 == 0 and 0 < nn <= 0xFC) \
or (b == 0x40 and nn == 0x02):
# Accept the same tag vocabulary ``walk_body`` accepts, including the
# wide-NN forms (``0X``/``1X``/``2X``) and the variable-width ``40 NN``
# segment header.
if ((b & 0xF0) in (0x00, 0x10, 0x20) and nn % 4 == 0
and ((b & 0x0F) != 0 or 0 < nn <= 0xFC)) \
or (b == 0x30 and nn % 4 == 0 and 0 < nn <= 0xFC) \
or (b == 0x40 and 0 < nn <= 0x08) \
or is_tagless_segment_header(body, 7):
return 7
# Fall back to scanning the first 20 bytes.
for i in range(min(20, len(body) - 1)):
@@ -178,6 +184,31 @@ def find_data_start(body: bytes) -> int:
return -1
# Channel-id byte carried in every segment header. Previously mis-read as a
# "monotonic uint32 LE counter"; it is really ``[channel][00][00][segment]``.
# Verified 2026-08-25 on 1697/1697 segment headers across the ground-truth
# corpus with zero disagreements against the decoded channel rotation.
SEGMENT_CHANNEL_IDS = {0x46: "Tran", 0x47: "Vert", 0x48: "Long", 0x49: "MicL"}
# A tagless segment header: the 14-byte tail of a ``40 NN`` header with no tag
# and no previous-channel continuation deltas (the NN=0 case).
_TAGLESS_HEADER_LEN = 14
def is_tagless_segment_header(body: bytes, i: int) -> bool:
"""True if a bare 14-byte segment header starts at *i*.
Layout ``[field2:2][len_to_next:2][channel_id:4][marker:2][anchors:4]``.
The discriminator is the 6 bytes at ``[4:10]``: a known channel id, two
zero bytes, a small segment index, and the ``01 00`` / ``02 00`` marker.
"""
if i + _TAGLESS_HEADER_LEN > len(body):
return False
return (body[i + 4] in SEGMENT_CHANNEL_IDS
and body[i + 5] == 0x00 and body[i + 6] == 0x00
and body[i + 8] in (0x01, 0x02) and body[i + 9] == 0x00)
def walk_body(body: bytes, start: Optional[int] = None) -> List[WaveformBlock]:
"""Walk the tagged-block sequence starting at *start* (auto-detected by default).
@@ -210,9 +241,15 @@ def walk_body(body: bytes, start: Optional[int] = None) -> List[WaveformBlock]:
# Wide-NN int8 block: ``2X NN`` extends NN to 12 bits the same way.
wide_nn = ((t0 & 0x0F) << 8) | t1
length = wide_nn + 2
elif t0 == 0x00 and t1 % 4 == 0:
elif (t0 & 0xF0) == 0x00 and t1 % 4 == 0:
# ``00 NN`` RLE zero-delta run, plus its wide form ``0X NN``
# (X != 0) which extends NN to 12 bits exactly like ``1X``/``2X``:
# NN = ((t0 & 0x0F) << 8) | t1. A narrow run maxes out at
# NN=0xFC, so quiet stretches longer than 252 samples must use
# the wide form. Confirmed 2026-08-25 against six production
# events (e.g. ``01 0c`` = 268 repeats in K558LKOF.460W).
length = 2
elif t0 == 0x30 and t1 % 4 == 0 and 0 < t1 <= 0x10:
elif t0 == 0x30 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
# Data-section ``30 NN`` blocks carry NN 12-bit signed deltas packed
# as NN/4 groups of (2-byte high-nibble field + 4 × int8 low byte).
# Length = NN/4 × 6 + 2 = NN × 1.5 + 2 (= 8 for NN=4, 14 for NN=8,
@@ -229,8 +266,28 @@ def walk_body(body: bytes, start: Optional[int] = None) -> List[WaveformBlock]:
length = cand_data
else:
length = cand_trailer
elif t0 == 0x40 and t1 == 0x02:
length = 20
elif t0 == 0x40 and 0 < t1 <= 0x08:
# ``40 NN`` segment header. NN is the number of int16 BE
# continuation deltas the header carries for the PREVIOUS
# channel, so the header grows with NN:
# length = 2 (tag) + 2*NN (deltas) + 14 (fixed tail)
# ``40 02`` (20 bytes) dominates, but ``40 01`` (18) and
# ``40 03`` (22) both occur in production files. Confirmed
# 2026-08-25; the constant ``02 00`` marker moves with NN too
# (see :func:`parse_segment_header`).
length = 2 * t1 + 16
elif is_tagless_segment_header(body, i):
# Segment header with no ``40 NN`` tag (NN=0 — the previous channel
# needed no continuation deltas). Emit it as a synthetic ``40 00``
# block whose ``data`` is the whole 14-byte record, so the nd=0
# offsets in :func:`decode_waveform_v2` line up unchanged.
blocks.append(WaveformBlock(
offset=i, tag_hi=0x40, tag_lo=0x00,
data=bytes(body[i : i + _TAGLESS_HEADER_LEN]),
length=_TAGLESS_HEADER_LEN,
))
i += _TAGLESS_HEADER_LEN
continue
else:
# Unknown tag; stop. Caller can inspect ``i`` to see where.
break
@@ -256,7 +313,7 @@ def split_segments(blocks: List[WaveformBlock]) -> List[List[WaveformBlock]]:
segments: List[List[WaveformBlock]] = []
current: List[WaveformBlock] = []
for b in blocks:
if b.tag_hi == 0x40 and b.tag_lo == 0x02:
if b.tag_hi == 0x40:
if current:
segments.append(current)
current = [b]
@@ -268,23 +325,40 @@ def split_segments(blocks: List[WaveformBlock]) -> List[List[WaveformBlock]]:
def parse_segment_header(block: WaveformBlock) -> Optional[dict]:
"""Decode the 18-byte payload of a ``40 02`` segment header.
"""Decode the payload of a ``40 NN`` segment header.
Returns a dict with the labelled fields, or None if *block* is not
a ``40 02`` header.
NN (the tag's low byte) is the number of int16 BE continuation deltas
the header carries for the PREVIOUS channel, so every field after
those deltas shifts by ``2 * NN``. The payload is ``2 * NN + 14``
bytes. ``40 02`` is the common case; ``40 01`` and ``40 03`` also
occur in production files (confirmed 2026-08-25).
Returns a dict with the labelled fields, or None if *block* is not a
segment header or is too short.
"""
if not (block.tag_hi == 0x40 and block.tag_lo == 0x02):
if block.tag_hi != 0x40 or block.tag_lo > 0x08:
return None
if len(block.data) < 18:
nd = block.tag_lo
if len(block.data) < 2 * nd + 14:
return None
p = block.data
counter = int.from_bytes(p[8:12], "little", signed=False)
counter = int.from_bytes(p[2 * nd + 4 : 2 * nd + 8], "little", signed=False)
return {
"anchor_bytes": p[0:4], # 4-byte field, role unconfirmed
"field2": p[4:8], # 4-byte field, role unconfirmed
"counter": counter, # uint32 LE — increments by 1 per segment
"fixed_pattern": p[12:16], # always b"\x02\x00\x00\x01"
"tail": p[16:18], # last 2 bytes
"n_prev_deltas": nd,
# ``nd`` int16 BE deltas extending the previous channel.
"prev_deltas": [
int.from_bytes(p[2 * k : 2 * k + 2], "big", signed=True)
for k in range(nd)
],
"field2": p[2 * nd : 2 * nd + 4], # 4-byte field, role unconfirmed
"counter": counter, # legacy: raw uint32 LE of the id field
"channel": SEGMENT_CHANNEL_IDS.get(p[2 * nd + 4]),
"segment_index": p[2 * nd + 7],
"marker": p[2 * nd + 8 : 2 * nd + 10], # always b"\x02\x00"
"anchors": [
int.from_bytes(p[2 * nd + 10 : 2 * nd + 12], "big", signed=True),
int.from_bytes(p[2 * nd + 12 : 2 * nd + 14], "big", signed=True),
],
}
@@ -420,8 +494,11 @@ def decode_waveform_v2(body: bytes) -> Optional[dict]:
for byte in blk.data:
cur += _i8(byte)
out[channel].append(cur)
elif blk.tag_hi == 0x00:
for _ in range(blk.tag_lo):
elif (blk.tag_hi & 0xF0) == 0x00:
# RLE zero-delta run. Wide form ``0X NN`` carries the high
# nibble of a 12-bit NN in the tag byte, same as ``1X``/``2X``.
run = ((blk.tag_hi & 0x0F) << 8) | blk.tag_lo
for _ in range(run):
out[channel].append(cur)
elif blk.tag_hi == 0x30:
# 12-bit signed deltas, packed as NN/4 groups of 6 bytes each:
@@ -461,34 +538,54 @@ def decode_waveform_v2(body: bytes) -> Optional[dict]:
# previous-channel extension deltas at every segment boundary.
last_value = {"Tran": last_tran_value, "Vert": None, "Long": None, "MicL": None}
prev_channel = "Tran"
for k, hi in enumerate(seg_idx):
channel = rotation[k % 4]
prev_channel = "Tran" if k == 0 else rotation[(k - 1) % 4]
header = blocks[hi]
if len(header.data) < 18:
# Channel comes from the header's own id byte, which is authoritative.
# The old rotation-by-position fallback is kept for headers whose id
# byte isn't one of the four known values — but a single missed or
# extra header would desync rotation and corrupt every later channel,
# which is exactly what tagless headers used to cause.
_nd = header.tag_lo
channel = None
if len(header.data) >= 2 * _nd + 8:
channel = SEGMENT_CHANNEL_IDS.get(header.data[2 * _nd + 4])
if channel is None:
channel = rotation[k % 4]
# ``40 NN``: NN int16 BE continuation deltas for the previous channel
# come first, so every later field shifts by 2*NN. NN is usually 2
# but 1 and 3 both occur (confirmed 2026-08-25).
nd = header.tag_lo
if len(header.data) < 2 * nd + 14:
continue
# Validate: real segment headers have bytes [12:14] = `02 00`.
# Trailer/footer "40 02" markers contain ASCII serial bytes or other
# non-header data there and would otherwise be mis-interpreted as
# segment headers, adding spurious samples at the tail.
if header.data[12:14] != b"\x02\x00":
# Validate: real segment headers have the constant `02 00` marker
# right after the counter. Trailer/footer "40 NN" markers contain
# ASCII serial bytes or other non-header data there and would
# otherwise be mis-read as segment headers, adding spurious tail
# samples.
if header.data[2 * nd + 8 : 2 * nd + 10] != b"\x02\x00":
break
# Extend the PREVIOUS channel by 2 more samples (deltas in bytes [0:4]).
prev_d0 = int.from_bytes(header.data[0:2], "big", signed=True)
prev_d1 = int.from_bytes(header.data[2:4], "big", signed=True)
# Extend the PREVIOUS channel by NN more samples.
if last_value[prev_channel] is not None:
v = last_value[prev_channel] + prev_d0
out[prev_channel].append(v)
v += prev_d1
out[prev_channel].append(v)
v = last_value[prev_channel]
for d in range(nd): # NB: not `k` — that's the segment index
v += int.from_bytes(
header.data[2 * d : 2 * d + 2], "big", signed=True
)
out[prev_channel].append(v)
last_value[prev_channel] = v
# Anchor pair for THIS segment's channel.
c0 = int.from_bytes(header.data[14:16], "big", signed=True)
c1 = int.from_bytes(header.data[16:18], "big", signed=True)
c0 = int.from_bytes(
header.data[2 * nd + 10 : 2 * nd + 12], "big", signed=True
)
c1 = int.from_bytes(
header.data[2 * nd + 12 : 2 * nd + 14], "big", signed=True
)
out[channel].extend([c0, c1])
# Apply delta blocks for this segment.
next_hi = seg_idx[k + 1] if k + 1 < len(seg_idx) else len(blocks)
last_value[channel] = apply_blocks(channel, c1, hi + 1, next_hi)
prev_channel = channel
return out