""" histogram_codec.py — decoder for MiniMate Plus histogram-mode event bodies. FULLY DECODED 2026-05-20. Every field in every block, verified byte-exact against BW's ASCII export across multiple histogram fixtures. The histogram-mode body is a stream of 32-byte fixed-length blocks, one block per histogram interval. Each block carries the per-interval peak amplitude + zero-crossing frequency for all four channels (Tran, Vert, Long, MicL). ──────────────────────────────────────────────────────────────────────────── Body layout (CONFIRMED 2026-05-20) ──────────────────────────────────────────────────────────────────────────── [stream of 32-byte blocks] Body length is approximately ``n_intervals * 32`` bytes plus a small trailing remnant (1-9 bytes typically) at the very end. Walker should iterate 32-stride and stop before the tail. ──────────────────────────────────────────────────────────────────────────── 32-byte block layout ──────────────────────────────────────────────────────────────────────────── [0] 0x00 always-zero tag [1] segment_id (uint8) 0x00..0x03 - 256 blocks per segment [2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, ...) [4] 0x0a (uint8) constant marker (= 10) [5:7] T_peak_count uint16 BE Tran peak (count x 0.005 -> in/s) [7:9] T_halfperiod uint16 BE Tran half-period in samples (freq = 512 / halfp) [9:11] V_peak_count uint16 BE [11:13] V_halfperiod uint16 BE [13:15] L_peak_count uint16 BE [15:17] L_halfperiod uint16 BE [17:19] M_peak_count uint16 BE MicL peak (count -> dB via mic_count_to_db) [19:21] M_halfperiod uint16 BE MicL half-period in samples [21:23] 0x00 0x00 constant on standard blocks [24:28] 4-byte variable purpose unknown (possibly CRC or timestamp delta) [28:32] block-end signature see "Two block tails" below **Every per-channel field is uint16 BIG-endian** (confirmed 2026-08-25). Only ``block_ctr`` at [2:4] is little-endian. HISTORY - two earlier readings of this block were wrong in ways that cancelled out on quiet data: 1. *peak as uint16 LE at [6:8]* - produced 268 in/s peaks on any interval whose next byte was non-zero. 2. *peak as uint8 at [6] with an "annotation" byte at [7]* - correct for every peak below 256 counts (1.275 in/s), but it silently **clipped larger peaks**: the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590 in/s. The "annotation" byte was never an annotation - it is the high byte of the big-endian half-period, which is why it was non-zero exactly on the sub-Hz intervals BW renders as "<1.0". Both readings also forced ``block[5] == 0`` via a bogus ``uint16 LE`` marker check at [4:6], which is what capped the peak at one byte. The marker is ``block[4]`` alone. Verified 2026-08-25 against 1211 production histograms paired with their Blastware ASCII exports: **1211/1211 decode exactly** (interval count plus every per-interval peak), and 842,442 per-interval frequency comparisons match with **zero** mismatches. Two block tails --------------- Standard blocks end with ``1e 0a 00 00``. The **final block of the stream** ends with ``9c 06 00 42`` instead, and carries arbitrary bytes at [21:23]. Rejecting it dropped the last interval of nearly every histogram - and the last interval is frequently the one holding the event peak, so the file's reported PPV came out low. Observed in 1206 of 1211 production histograms, always positioned after every standard-tail block. Block-identification anchor: ``block[0] == 0x00`` AND ``block[4] == 0x0A`` AND the tail is one of the two signatures above; standard-tail blocks additionally require ``block[22] == 0x00``. ──────────────────────────────────────────────────────────────────────────── Per-channel encoding ──────────────────────────────────────────────────────────────────────────── Geophone channels (Tran, Vert, Long): - peak_count × 0.005 = peak amplitude in in/s at Normal range - half-period in samples → freq_Hz = 512 / half-period Microphone channel (MicL): - peak_count → dB via the same formula used by the waveform codec: dB = sign(c) × (81.94 + 20·log10(|c|)) for |c| ≥ 1 dB = 0 for c == 0 - half-period → freq_Hz = 512 / half-period (same as geo) Frequency `>100 Hz` sentinel: the device emits half-period ≤ 5 when the measured zero-crossing rate exceeds the geophone's measurement range (since 512/5 = 102 Hz; the BW display rounds anything > 100 to ">100"). ──────────────────────────────────────────────────────────────────────────── Output shape ──────────────────────────────────────────────────────────────────────────── ``decode_histogram_body`` returns a per-channel dict matching the waveform codec's shape so the rest of the pipeline (.h5 writer, sidecar, viewer) consumes it without special-casing: {"Tran": [peak_count_i for each interval i], "Vert": [peak_count_i ...], "Long": [peak_count_i ...], "MicL": [peak_count_i ...]} Values are in **16-count units for geo** (LSB = 0.005 in/s, matching ``decode_waveform_v2``) and **1-count units for mic** (matching the waveform codec's mic convention). Run through ``waveform_codec.decoded_to_adc_counts`` to scale geo to 1-count ADC. Per-interval frequencies are NOT returned — they're auxiliary data, not waveform samples. Consumers needing frequencies can call ``decode_histogram_body_full()`` for the structured per-interval record list. """ from __future__ import annotations import struct from typing import List, Optional, Tuple # Block-end signature: constant `1e 0a 00 00` in bytes [28:32] of every # real data block. More distinctive than the byte-22 `00 00` (which # matches many false positives), so we anchor on this. _BLOCK_TAIL = b"\x1e\x0a\x00\x00" # The final block of a histogram stream ends with this instead. It is a # real data block - same layout - and holds the last interval. See the # module docstring, "Two block tails". _BLOCK_TAIL_TERMINAL = b"\x9c\x06\x00\x42" _BLOCK_SIZE = 32 # Marker byte at block[4:6] of every histogram data block. Used as # additional validation that we're looking at a real block. _BLOCK_MARKER = 10 # Geo peak scaling: stored as "count × 0.005 in/s" where 1 count = one # 0.005 in/s display quantum. Equivalent to the waveform codec's # 16-count-unit output (1 unit = 0.005 in/s = 16 ADC counts). _GEO_LSB_INS = 0.005 # Frequency formula: freq_Hz = _FREQ_NUMERATOR / half_period_samples. # Empirically determined to be 512 (= sample_rate / 2, where sample rate # is 1024 sps for the standard MiniMate Plus configuration). _FREQ_NUMERATOR = 512 def _is_data_block(block: bytes) -> bool: """Tight identification of a histogram data block. Accepts both tail signatures. ``block[4]`` alone is the marker - ``block[5]`` is the high byte of the Tran peak and is non-zero on any interval above 1.275 in/s, so it must not be part of the marker test. The ``block[22] == 0`` constraint is what keeps trailer content out, but it applies only to standard-tail blocks: terminal blocks carry arbitrary bytes there. """ if len(block) < _BLOCK_SIZE: return False if block[0] != 0x00: return False if block[4] != _BLOCK_MARKER: return False # The 4-byte tail plus block[0]==0 and block[4]==0x0A is already six bytes # of constraint — enough to keep trailer content out. There is NO extra # test on block[22]: it was documented as a constant 0x00 but carries data # on loud blocks, and rejecting those threw away the interval holding the # event peak. BE18350/T350L7HR.NL0H is the proof: its block 92 has # block[22]=0x26 and a Tran peak of 0x0563 = 1379 counts = 6.895 in/s, # exactly the device-reported PPV, while the file decoded to 0.015 in/s. return block[28:32] in (_BLOCK_TAIL, _BLOCK_TAIL_TERMINAL) def _decode_block(block: bytes) -> Optional[dict]: """Decode one 32-byte histogram block. Caller must have validated with ``_is_data_block`` first. Returns a record with per-channel peak counts (uint8) and half-periods (uint16 LE). """ # Every per-channel field is uint16 BIG-endian; only block_ctr is LE. # See the module docstring for the two superseded readings and why # each looked correct on quiet data. def _be16(i: int) -> int: return (block[i] << 8) | block[i + 1] t_peak = _be16(5) t_halfp = _be16(7) v_peak = _be16(9) v_halfp = _be16(11) l_peak = _be16(13) l_halfp = _be16(15) m_peak = _be16(17) m_halfp = _be16(19) segment_id = block[1] block_ctr = block[2] | (block[3] << 8) var_meta = bytes(block[24:28]) return { "segment_id": segment_id, "block_ctr": block_ctr, "t_peak": t_peak, "t_halfp": t_halfp, "v_peak": v_peak, "v_halfp": v_halfp, "l_peak": l_peak, "l_halfp": l_halfp, "m_peak": m_peak, "m_halfp": m_halfp, "meta_var": var_meta, "is_terminal": block[28:32] == _BLOCK_TAIL_TERMINAL, } def walk_body(body: bytes) -> List[dict]: """Walk the body and return one dict per histogram interval. Iterates 32-byte strides from offset 0. Yields a decoded record for every block that passes ``_is_data_block`` validation. Stops when the remaining bytes are too short to form a complete block. In Histogram+Continuous mode the body interleaves data blocks with other 32-byte content (likely continuous-mode waveform blocks) that fail the data-block validation; the walker naturally skips them without losing 32-byte alignment. Use ``block_ctr`` from each returned record to map back to the original interval index — the record list is sparse when other block types are interleaved. """ records: List[dict] = [] for off in range(0, len(body) - _BLOCK_SIZE + 1, _BLOCK_SIZE): blk = body[off:off + _BLOCK_SIZE] if not _is_data_block(blk): # Hit non-block content (likely a sync or stream marker). # Continue walking — block alignment is fixed at 32-stride # from offset 0, so we don't lose alignment by skipping. continue decoded = _decode_block(blk) if decoded is None: # Block validated as a histogram block but had peak fields # outside the plausible range — undocumented extension. # Skip rather than propagating bogus PVS contributions. continue records.append(decoded) return records def _walk_auto(body: bytes) -> List[dict]: """Pick the block model by signature strength, not by which returns first. The multi-interval variant announces itself with consecutive block headers at an exact ``12 + 20*n`` stride — far stronger evidence than a handful of scattered standard-tail blocks, which a multi-interval body will also yield by coincidence. Dispatching on "whichever decoder returns something" handed 193 BE18193 files to the standard walker and produced peaks of 149 in/s against a 10 in/s full scale. """ if detect_multi_interval_stride(body): recs = walk_multi_interval_blocks(body) if recs: return recs return walk_body(body) def decode_histogram_body(body: bytes) -> Optional[dict]: """Decode a histogram-mode body into per-channel peak-sample arrays. Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}`` where each channel's list contains one peak value per histogram interval (in the same units the waveform codec uses: 16-count units for geo, 1-count ADC units for mic). Returns ``None`` if the body doesn't contain any valid histogram blocks. To convert to physical units: - Geo channels: ``count * 0.005`` = peak in in/s at Normal range (or run through ``waveform_codec.decoded_to_adc_counts`` first to get 1-count ADC values, then ``count / 32767 * 10.0`` for in/s) - Mic channel: use ``waveform_codec.mic_count_to_db(count)`` """ records = _walk_auto(body) if not records: return None return { "Tran": [r["t_peak"] for r in records], "Vert": [r["v_peak"] for r in records], "Long": [r["l_peak"] for r in records], "MicL": [r["m_peak"] for r in records], } def decode_histogram_body_full(body: bytes) -> Optional[List[dict]]: """Decode a histogram-mode body into the full per-interval record list. Same data as ``decode_histogram_body`` but in a structured form that preserves the half-period (frequency) data for each channel + the per-block segment_id, block_ctr, and 4-byte variable metadata. Useful for diagnostic tools, sidecar enrichment, and future-codec work. Returns ``None`` if the body has no valid blocks. """ records = _walk_auto(body) return records if records else None def half_period_to_hz(halfp: int) -> Optional[float]: """Convert a half-period in samples to frequency in Hz. Returns ``None`` for half-period ≤ 5 — the device emits values in that range when the measured zero-crossing rate exceeds 100 Hz (the BW display reports `>100 Hz` for such cases). Callers can treat ``None`` as the `>100 Hz` sentinel. """ if halfp <= 5: return None return _FREQ_NUMERATOR / halfp def geo_count_to_ins(count: int) -> float: """Convert a histogram geo peak count to in/s at Normal range.""" return count * _GEO_LSB_INS # ── Multi-interval block variant (CONFIRMED 2026-08-26) ───────────────────── # # When the histogram interval is SHORTER than one minute, the device packs # several intervals into a single block so that every block still covers # exactly one minute of data: # # interval size intervals/block stride # 1 minute 1 32 <- the standard block above # 15 seconds 4 92 # 2 seconds 30 612 # # stride = 12 + n_intervals * 20 # # Block layout: # [0] 0x00 # [1] segment_id (256 blocks per segment, same as the standard block) # [2:4] block_ctr uint16 LE (0x0100.., resets each segment) # [4] 0x0a marker # [5] 0x00 # [6 ...] n x 20-byte interval records, each carrying 8 x uint16 # LITTLE-endian values: # T_peak, T_halfperiod, V_peak, V_halfperiod, # L_peak, L_halfperiod, M_peak, M_halfperiod # then 2 more words; the first is 0x0000 on every real interval. # [-6:] 6-byte block trailer # # ⚠ ENDIANNESS: the standard 32-byte block is BIG-endian. This variant is # LITTLE-endian. Do not share the accessor. # # These files previously decoded to nothing at all — 415 of them in the # production snapshot, 216 on BE18193 (2 s intervals) and 199 on BE9440 # (15 s). Before that they were being accepted by the WAVEFORM codec, which # returned garbage peaking up to 400x the device-reported PPV. # # Ground truth: BE9440/K440L3AQ.T70H (15 s intervals, 5,710 of them) decodes # against its Blastware ASCII export with 17,130/17,130 geo peak counts, # 22,840/22,840 frequencies and 5,710/5,710 mic dB(L) values matching exactly. _MULTI_HEADER_LEN = 6 _MULTI_RECORD_LEN = 20 _MULTI_TRAILER_LEN = 6 # At least 2 records: a 1-record block would have stride 12 + 20 = 32, which # collides with the standard big-endian block and mis-decodes it. _MULTI_MIN_RECORDS = 2 _MULTI_MAX_RECORDS = 64 # Geo full scale in 16-count units: 10.000 in/s / 0.005 = 2000. A peak above # this is physically impossible and marks buffer garbage in a partial block. _GEO_MAX_COUNTS = 2000 def _is_multi_header(body: bytes, off: int) -> bool: return (off + _MULTI_HEADER_LEN <= len(body) and body[off] == 0x00 and body[off + 4] == 0x0A and body[off + 5] == 0x00) def detect_multi_interval_stride(body: bytes) -> Optional[int]: """Block stride of a multi-interval histogram body, or None. Found by locating the second block header; validated against ``stride = 12 + n * 20`` and confirmed on a third block where present. """ if not _is_multi_header(body, 0): return None lo = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MIN_RECORDS hi = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MAX_RECORDS for stride in range(lo, min(hi, len(body)) + 1, 2): if (stride - 12) % _MULTI_RECORD_LEN: continue if not _is_multi_header(body, stride): continue # DECISIVE CHECK: consecutive blocks differ by exactly 1 in block_ctr. # Without it this false-positives on ordinary standard-block bodies: # those carry a header every 32 bytes, and 192 = 12 + 20*9 and # 512 = 12 + 20*25 are both multiples of 32, so a stride "fits" while # actually skipping 6 or 16 real blocks. Sampling a standard body at # stride 192 handed 9,082 files to the wrong decoder and produced peaks # of 149 in/s against a 10 in/s full scale. def _ctr(o: int) -> int: return body[o + 2] | (body[o + 3] << 8) if (_ctr(stride) - _ctr(0)) & 0xFFFF != 1: continue # Confirm on a third block WHEN ONE IS ACTUALLY PRESENT. A body can # be longer than two strides and still hold only two real blocks: a # final *partial* block leaves trailing padding. E.g. 51 intervals at # 2 s = one full 30-interval block + a 21-interval remainder, in a # 2787-byte body — long enough to demand a third header at 1224 that # does not exist. Requiring it unconditionally threw away the correct # stride and the file decoded to nothing (BE18193 T193L0XM.CI0H). # The block-counter check above is the decisive anti-false-positive # test; this one is corroboration, so a missing third header means # end-of-stream, not disqualification. if (2 * stride + _MULTI_HEADER_LEN <= len(body) and _is_multi_header(body, 2 * stride)): if (_ctr(2 * stride) - _ctr(stride)) & 0xFFFF != 1: continue return stride return None def walk_multi_interval_blocks(body: bytes, stride: Optional[int] = None) -> List[dict]: """Decode a multi-interval histogram body into per-interval records.""" if stride is None: stride = detect_multi_interval_stride(body) if not stride: return [] n_per_block = (stride - _MULTI_HEADER_LEN - _MULTI_TRAILER_LEN) // _MULTI_RECORD_LEN if n_per_block < 1: return [] def u16le(p: int) -> int: return body[p] | (body[p + 1] << 8) out: List[dict] = [] for off in range(0, len(body) - stride + 1, stride): if not _is_multi_header(body, off): break # end of the block run; trailer follows for k in range(n_per_block): q = off + _MULTI_HEADER_LEN + _MULTI_RECORD_LEN * k out.append({ "_tail0": u16le(q + 16), "segment_id": body[off + 1], "block_ctr": u16le(off + 2), "t_peak": u16le(q), "t_halfp": u16le(q + 2), "v_peak": u16le(q + 4), "v_halfp": u16le(q + 6), "l_peak": u16le(q + 8), "l_halfp": u16le(q + 10), "m_peak": u16le(q + 12), "m_halfp": u16le(q + 14), "meta_var": bytes(body[q + 16:q + 20]), "is_terminal": False, }) # A session ending mid-block leaves the remaining slots of the FINAL block # filled with whatever was in the buffer. Those decoded as peaks thousands # of times the device-reported PPV, so they have to go — but only from the # final block: a non-zero tail word occurs mid-file on real intervals, and # trimming on that alone truncated four BE9440 files by up to 2,800 # intervals, while trimming purely from the end left garbage stranded # behind one slot that happened to have a zero tail word. # # Within the final block, stop at the first slot that is not plausibly # real: a non-zero tail word, or a geo peak above full scale. 16-count # units put Normal-range full scale (10.000 in/s) at 2000 counts, so # anything beyond that is physically impossible. if out: last_block_start = ((len(out) - 1) // n_per_block) * n_per_block for i in range(last_block_start, len(out)): r = out[i] if (r["_tail0"] != 0 or max(r["t_peak"], r["v_peak"], r["l_peak"]) > _GEO_MAX_COUNTS): del out[i:] break for r in out: r.pop("_tail0", None) return out