seismo-relay/docs/histogram_codec_re_status.md

SUPERSEDED 2026-08-25 — the block is uniformly BIG-ENDIAN

The uint8 peak / annotation byte model described below is wrong, though it decoded quiet data correctly. The real layout:

• Every per-channel field is uint16 big-endian. T_peak is [5:7], T_halfperiod [7:9], V_peak [9:11], and so on. Only block_ctr at [2:4] is little-endian.
• The marker is block[4] alone, not a uint16 LE at [4:6]. Testing [4:6] == 10 forced block[5] == 0, which is exactly what capped every geo peak at one byte (255 counts = 1.275 in/s).
• The "annotation" byte was never an annotation — it is the high byte of the big-endian half-period. That is why it was non-zero precisely on the sub-Hz intervals Blastware renders as <1.0.
• The final block of the stream carries tail 9c 06 00 42 instead of 1e 0a 00 00, and arbitrary bytes at [21:23]. Rejecting it dropped the last interval of nearly every histogram — often the one holding the event peak, so the file's PPV read low.

Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly (interval count plus every per-interval peak), and 842,442 per-interval frequency comparisons match with zero mismatches. The uint8 model scored 1204/1211 — the seven failures are exactly the files containing a peak above 1.275 in/s.

The section below is retained as the reasoning trail.

Histogram body codec — FULLY DECODED (2026-05-20)

Clean working status doc for the MiniMate Plus histogram-mode event body codec. Companion to waveform_codec_re_status.md. The deep historical record (with retractions and dated analyses) lives in docs/instantel_protocol_reference.md §7.6.2; the authoritative implementation lives in minimateplus/histogram_codec.py.

TL;DR

The codec is fully decoded. Every field of every block in the in-repo histogram fixture corpus decodes byte-exact against BW's ASCII export.

26 regression tests pass against ~3,500 blocks across 5 in-repo fixtures, plus a synthetic regression block taken from a real BE9558 prod event to lock in the uint8-peak interpretation.

Important correction (2026-05-21): the per-channel peak count is uint8 at byte[6]/[10]/[14]/[18], NOT uint16 LE at byte[6:8] etc. The N844 fixture corpus the original RE was done against has zero values in bytes [7]/[11]/[15]/[19] for every block, so the two interpretations happened to be equivalent. Cross-correlating non-N844 events (BE9558 Tran-drift, BE18003 Histogram+Continuous) against BW's per-interval ASCII export — 4 channels × ~1400 blocks per event × multiple events = 100% byte-exact only when the peak is read as uint8. Reading as uint16 LE produced peaks up to 268 in/s per channel and 35× inflated PVS sums when first deployed to prod (rolled back, root-caused, and fixed in commit 7183b95+1).

Body format

body = [stream of 32-byte data blocks] + [small trailing remnant]

Each block represents one histogram interval. Block layout:

[0]    0x00                      always-zero tag
[1]    segment_id (uint8)        0x00..0x03 — 256 blocks per segment
[2:4]  block_ctr (uint16 LE)     resets each segment (0x0100, 0x0101, …)
[4:6]  0x000a (uint16 LE)        constant marker (= 10)
[6]    T_peak_count   uint8      Tran peak (count × 0.005 → in/s at Normal,
                                  max 1.275 in/s — fits in uint8)
[7]    T_annotation   uint8      empirically non-zero on intervals with sub-Hz
                                  or unmeasurable freq; meaning not fully RE'd
[8:10] T_halfperiod   uint16 LE  Tran half-period in samples
                                  (freq_Hz = 512 / halfp; ≤ 5 means ">100 Hz")
[10]   V_peak_count   uint8      Vert peak
[11]   V_annotation   uint8
[12:14] V_halfperiod  uint16 LE  Vert freq half-period
[14]   L_peak_count   uint8      Long peak
[15]   L_annotation   uint8
[16:18] L_halfperiod  uint16 LE  Long freq half-period
[18]   M_peak_count   uint8      MicL peak count
                                  (dB via waveform_codec.mic_count_to_db)
[19]   M_annotation   uint8
[20:22] M_halfperiod  uint16 LE  MicL freq half-period
[22:24] 0x00 0x00                constant
[24:28] 4-byte variable          purpose unknown — possibly CRC,
                                  timestamp delta, or psi(L) numeric;
                                  not needed for waveform reconstruction
[28:32] 0x1e 0x0a 0x00 0x00      constant block-end signature

Reliable block-identification anchor:

block[22:24] == b"\x00\x00" and block[28:32] == b"\x1e\x0a\x00\x00"

(The 1e 0a 00 00 constant tail is the most distinctive signature.)

Per-channel encoding

Channel	Peak encoding	Frequency encoding
Tran	count × 0.005 = in/s at Normal range	freq_Hz = 512 / halfperiod
Vert	same	same
Long	same	same
MicL	count → dB via mic_count_to_db(count) (same formula as waveform codec)	same

>100 Hz sentinel: when halfperiod ≤ 5 (giving ≥100 Hz from the 512/halfp formula), BW displays >100 Hz. Codec's half_period_to_hz returns None in this range.

Verified facts (cross-checked against fixture corpus)

Example: N844L6Z8.ZR0H block 130 → all 8 decoded fields byte-exact:

binary samples [10, 6, 24, 4, 18, 5, 21, 5, 9]
TXT row        [0.030, 21, 0.020, 28, 0.025, 24, 0.040, 0.000, 95.92, 57]

slot[0] = 10                                  marker
slot[1] = 6  × 0.005 = 0.030 in/s         ✓ T_peak
slot[2] = 24 → 512/24 = 21.3 → 21 Hz      ✓ T_freq
slot[3] = 4  × 0.005 = 0.020 in/s         ✓ V_peak
slot[4] = 18 → 512/18 = 28.4 → 28 Hz      ✓ V_freq
slot[5] = 5  × 0.005 = 0.025 in/s         ✓ L_peak
slot[6] = 21 → 512/21 = 24.4 → 24 Hz      ✓ L_freq
slot[7] = 5  → 81.94 + 20·log10(5) = 95.92 dB  ✓ M_peak
slot[8] = 9  → 512/9 = 56.9 → 57 Hz       ✓ M_freq

Verified test coverage

tests/test_histogram_codec.py (24 tests):

• Block walking: yields one record per .TXT interval ± 1 (off-by-one at the tail when recording was stopped mid-write). Segment-ID groups of 256 blocks confirmed.
• Geo peaks: every block of N844L20G, N844L6Z8, N844L6XE, N844L23B matches .TXT within the 0.0005 in/s quantization step.
• Geo freqs: every block of N844L6Z8 and N844L6XE matches .TXT within 1 Hz (BW display rounds). >100 Hz sentinel handled correctly.
• Mic dB: every block of N844L6XE, N844L23B, N844L6Z8 matches .TXT within 0.1 dB (BW display precision).
• Mic freq: matches .TXT within 1 Hz across active blocks.

What's NOT yet decoded

• Annotation bytes (block[7]/[11]/[15]/[19]). Empirically non-zero on intervals where the per-channel ZC frequency comes out as N/A or sub-Hz (<1.0, 1.X). Hypothesis tested in the RE session: byte != 0 ↔ sub-Hz freq. Only ~50% correlation across the K558 corpus, so the relationship is more complex. Possibilities: time-of-peak-within-interval, halfp extension for very-long-period signals, or a debug/diagnostic field the firmware writes opportunistically. Doesn't affect peak amplitudes or waveform reconstruction. Captured as record["annotations"] for future RE.
• 4-byte variable metadata field (bytes 24:28). Not needed for waveform reconstruction. Speculation: per-block CRC, sub-second timestamp offset, or a Mic psi(L) count not in the 9 samples. Punt until something needs it.
• Geo PVS (TXT col 7, e.g. "0.040 in/s"). Not stored in the block; can be approximated as sqrt(T_peak² + V_peak² + L_peak²) but BW's value sometimes differs slightly (probably computed from waveform-instant samples, not from per-channel peaks). Punt — the .h5 consumers don't need PVS as a sample channel.
• Mic psi(L) value (TXT col 8). TXT shows it as a small psi value derived from the dB measurement. Not in the 9 samples. Could be derived from M_peak_count via the inverse of the dB formula plus a psi calibration constant. Defer.

Output shape

decode_histogram_body returns the standard 4-channel dict that mirrors waveform_codec.decode_waveform_v2's output:

{
    "Tran": [peak_count_per_interval, ...],   # 16-count units (LSB = 0.005 in/s)
    "Vert": [..., ...],
    "Long": [..., ...],
    "MicL": [..., ...],                       # raw ADC counts
}

Run through waveform_codec.decoded_to_adc_counts to get 1-count ADC units (geo ×16, mic passthrough) for the standard .h5 writer.

For the full per-interval record with frequencies + metadata, use decode_histogram_body_full().

Where it's wired

• minimateplus/event_file_io.py:read_blastware_file() — first tries the waveform codec, falls back to the histogram codec when the waveform preamble isn't present. Same output shape, same downstream pipeline.
• scripts/backfill_sidecars.py — the has_samples short-circuit added during the histogram-codec-pending era still serves as a defensive guard against truly undecodable files, but no longer fires for valid histograms.

Companion reference

• docs/waveform_codec_re_status.md — sibling status doc for the much-more-complex waveform-mode codec.
• docs/instantel_protocol_reference.md §7.6.2 — historical protocol-reference entry. Structural framing matches what we found; per-sample semantics were less documented than the ✅ CONFIRMED badge suggested. This doc supersedes §7.6.2 where they conflict on confidence level.