serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(series4): support mic-disabled (3-channel) Thor units

Browse Source 
Verified against a second Thor corpus (9-10-26-csv-req: UM11402, UM12947,
UM20147) with per-sample CSV exports: 139/139 waveforms exact
(1,273,380/1,273,380 samples) and 877/877 histograms within 2% of Thor's
reported PPV -- up from 66.9% and 56.6%.

Some units run with the microphone disabled, which changes two structural
things that were both hardcoded to the 4-channel shape:

- Waveform body head sat below the scan floor. A 3-channel unit has a
  shorter fixed header and puts its record chain head at 0x0dba, under the
  old _BODY_SCAN_FLOOR of 0x0E00. The scan could not see it and fell
  through to the Vert segment-0 record, decoding a body shifted one
  position around the channel rotation -- Vert came up exactly 512 samples
  short. Floor lowered to 0x0C00. The body-offset scoring also had to stop
  requiring four channels, or `equal` is permanently False for these events
  and the pick falls back to raw sample count.

- Histogram interval record is 56 bytes, not 72. It is
  16 * n_channels + 8, and is not inferable from the segment length alone.
  The interval count now comes from the segment's cumulative counter
  (n = counter - prev_counter) and the stride is derived from it. Assuming
  72 read 7 intervals out of every 10-interval segment, then walked off
  alignment into garbage that decoded as ~10 in/s peaks -- inflating some
  files' PPV by up to 191,000%. Also recovers 4 files that previously
  decoded no intervals at all.

Combined across both corpora: 292/292 waveform files,
2,330,916/2,330,916 samples exact. Production IDFW truncations 41 -> 22.
Series-3 unaffected (no shared-codec change in this commit; last full run
14,338/14,338).

Known open, diagnosed but NOT verified: the remaining 22 unequal + 1 failing
production IDFW files (all UM12947, 2025-07-14..09-23) stop the block walker
on tag 40 0c. data_block_len() caps the 40 NN int16 block at NN > 0x08 while
those files use NN up to 196. Both verified corpora only ever use
NN in {1,2,3,4,8}, so the cap is untested there and lifting it leaves both at
100.000% -- which is not evidence it decodes these correctly. Deliberately
not shipped; needs Thor CSV exports for UM12947 in that date range.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
This commit is contained in:
serversdown
2026-09-10 20:00:56 +00:00
co-authored by Claude Opus 5
parent 726c2ce1b5
commit c07aaa552c
5 changed files with 240 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
micromate/idf_file.py
+76 -23
View File
@@ -94,7 +94,15 @@ _BODY_MAGIC = b"\x00\x02\x00"
# fixed-header region where the same magic legitimately appears inside
# channel-test records and the compliance block (offsets 0x015d, 0x091c,
# 0x0ae2, 0x0d30 in observed events).
_BODY_SCAN_FLOOR = 0x0E00
# Lowered from 0x0E00 to 0x0C00 (2026-09-10). Three-channel events -- mic
# disabled -- have a shorter fixed header and put their record chain head at
# 0x0dba, below the old floor. The head was therefore invisible to the scan,
# which fell through to the *Vert* segment-0 record and decoded a body shifted
# one position around the channel rotation. 46 of 139 files in the
# 9-10-26-csv-req corpus were affected; all 46 became per-sample exact once
# the head was reachable. The floor still skips the fixed-header region,
# where `is_record()` can match channel-test records (0x015d, 0x091c, 0x0ae2).
_BODY_SCAN_FLOOR = 0x0C00
# Cap on trial decodes per file. Chain-head detection normally yields one
# or two candidates; the cap only bounds the worst case on a corrupt file.
@@ -140,7 +148,13 @@ _GEO_LSB_IPS = 0.000310308
_MIC_LSB_PSI = 2.14e-6
# IDFH histogram constants.
_IDFH_INTERVAL_SIZE = 72 # bytes per per-interval record
# Bytes per interval record = 16 per channel + an 8-byte tail, so a
# 4-channel unit uses 72 and a mic-disabled 3-channel unit uses 56. It is
# NOT a constant: derive it per segment from the interval counter (see
# decode_idfh_body). This value survives only as the 4-channel default.
_IDFH_INTERVAL_SIZE = 72 # bytes per per-interval record (4 channels)
_IDFH_CHANNEL_BLOCK = 16 # bytes per channel inside an interval record
_IDFH_INTERVAL_TAIL = 8 # bytes after the per-channel blocks
_IDFH_SEGMENT_HEADER = 10 # bytes: [len_be 2B][0a 00 00 00 4B][00 NN 2B][05 3f 2B]
_IDFH_SEGMENT_TAIL = 2 # bytes after the interval data block, before next marker
_IDFH_HALFP_FREQ_NUM = 512.0 # freq_hz = NUM / halfp; halfp ≤ 5 means ">100 Hz" sentinel
@@ -314,7 +328,10 @@ def _find_waveform_body_offset(buf: bytes) -> Optional[int]:
# A "real" body has more than just the 2-sample preamble.
if total <= 2:
continue
equal = len(lengths) == 4 and len(set(lengths)) == 1
# >= 3 rather than == 4: a mic-disabled event has only the three geo
# channels, and demanding four made `equal` permanently False for
# them, leaving the pick to raw sample count alone.
equal = len(lengths) >= 3 and len(set(lengths)) == 1
score = (equal, total)
if best is None or score > best:
best, best_off = score, j
@@ -375,6 +392,12 @@ class IdfhInterval:
micl_min: int
micl_max: int
micl_halfp: int
# 4 on a normal unit; 3 when the microphone is disabled, in which case the
# micl_* fields are absent from the record and read as zero.
n_channels: int = 4
def has_channel(self, channel: str) -> bool:
return channel != "MicL" or self.n_channels >= 4
def peak_count(self, channel: str) -> int:
mn = getattr(self, f"{channel.lower()}_min")
@@ -412,22 +435,30 @@ def _is_unwritten_interval(interval: "IdfhInterval") -> bool:
requiring every channel to be inverted keeps this from ever firing on
genuine data.
"""
return all(
mn > mx
for mn, mx in (
(interval.tran_min, interval.tran_max),
(interval.vert_min, interval.vert_max),
(interval.long_min, interval.long_max),
(interval.micl_min, interval.micl_max),
)
)
pairs = [
(interval.tran_min, interval.tran_max),
(interval.vert_min, interval.vert_max),
(interval.long_min, interval.long_max),
]
if interval.has_channel("MicL"):
pairs.append((interval.micl_min, interval.micl_max))
return all(mn > mx for mn, mx in pairs)
def _decode_idfh_interval(buf72: bytes, offset: int) -> IdfhInterval:
"""Decode one 72-byte interval record into per-channel min/max/halfp."""
def _decode_idfh_interval(buf72: bytes, offset: int,
n_channels: int = 4) -> IdfhInterval:
"""Decode one interval record into per-channel min/max/halfp.
The record is ``n_channels`` × 16-byte blocks plus an 8-byte tail, so it
is 72 bytes on a normal unit and 56 when the microphone is disabled.
Missing channels read as zero.
"""
import struct
fields = []
for i in range(4):
if i >= n_channels:
fields.extend([0, 0, 0])
continue
block = buf72[i * 16 : (i + 1) * 16]
mn = struct.unpack_from(">h", block, 0)[0]
mx = struct.unpack_from(">h", block, 2)[0]
@@ -443,6 +474,7 @@ def _decode_idfh_interval(buf72: bytes, offset: int) -> IdfhInterval:
vert_min=fields[3], vert_max=fields[4], vert_halfp=fields[5],
long_min=fields[6], long_max=fields[7], long_halfp=fields[8],
micl_min=fields[9], micl_max=fields[10], micl_halfp=fields[11],
n_channels=n_channels,
)
@@ -469,6 +501,7 @@ def decode_idfh_body(buf: bytes) -> list:
"""
intervals: list = []
i = 0
prev_counter = -1 # so the first segment's n = counter + 1
while True:
j = buf.find(b"\x0a\x00\x00\x00", i)
if j < 0 or j < 2:
@@ -479,28 +512,43 @@ def decode_idfh_body(buf: bytes) -> list:
i = j + 1
continue
length = int.from_bytes(buf[j - 2 : j], "big")
n = (length - _IDFH_SEGMENT_HEADER) // _IDFH_INTERVAL_SIZE
counter = int.from_bytes(buf[j + 4 : j + 6], "big")
header_start = j - 2
if length < _IDFH_SEGMENT_HEADER or header_start + length > len(buf):
# Truncated / bogus length — not a real segment header.
i = j + 1
continue
# The counter is the cumulative index of this segment's LAST interval,
# so the interval count is its delta from the previous segment. That
# gives the record stride, which is NOT fixed: 16 bytes per channel
# plus an 8-byte tail, so 72 for a 4-channel unit and 56 for a
# mic-disabled 3-channel one. Assuming 72 unconditionally made every
# 3-channel histogram read 7 intervals per 10-interval segment,
# walking off alignment into garbage that decoded as ~10 in/s peaks.
n = counter - prev_counter
if n <= 0:
i = j + 1
continue
header_start = j - 2
if header_start + length > len(buf):
# Truncated / bogus length — not a real segment header.
stride = (length - _IDFH_SEGMENT_HEADER) // n
n_channels, remainder = divmod(stride - _IDFH_INTERVAL_TAIL,
_IDFH_CHANNEL_BLOCK)
if remainder or not (1 <= n_channels <= 4):
i = j + 1
continue
interval_start = header_start + _IDFH_SEGMENT_HEADER
for k in range(n):
off = interval_start + k * _IDFH_INTERVAL_SIZE
if off + _IDFH_INTERVAL_SIZE > len(buf):
off = interval_start + k * stride
if off + stride > len(buf):
break
chunk = buf[off : off + _IDFH_INTERVAL_SIZE]
interval = _decode_idfh_interval(chunk, off)
chunk = buf[off : off + stride]
interval = _decode_idfh_interval(chunk, off, n_channels)
if _is_unwritten_interval(interval):
# Reserved-but-never-recorded slot: the min/max accumulators
# still hold their ±full-scale seed. Counting it would
# fabricate a 10.0 in/s peak on every channel.
continue
intervals.append(interval)
prev_counter = counter
# Advance past this segment + the 2-byte tail.
i = header_start + length + _IDFH_SEGMENT_TAIL
return intervals
@@ -580,7 +628,12 @@ def read_idf_file(
peak_long = max((iv.peak_ips("Long") for iv in intervals), default=0.0)
# Mic peak in psi — Thor stores per-interval mic ADC counts in the
# binary; convert the max count to psi via the per-count factor.
mic_peak_count = max((iv.peak_count("MicL") for iv in intervals), default=0)
# Skip on a mic-disabled (3-channel) unit: those records carry no mic
# block at all, so peak_count("MicL") would report a synthetic zero.
mic_peak_count = max(
(iv.peak_count("MicL") for iv in intervals if iv.has_channel("MicL")),
default=0,
)
mic_peak_psi = mic_count_to_psi(mic_peak_count) if mic_peak_count else None
rep = IdfReport(
serial_number=md.serial,