diff --git a/CHANGELOG.md b/CHANGELOG.md index cb7aa9c..179ad17 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,34 @@ All notable changes to seismo-relay are documented here. ### Fixed +- **Sub-minute histogram intervals are packed several to a block — 415 files + recovered.** The device always writes one minute of data per block, so a + shorter interval just means more intervals in a longer block: + + | interval | intervals/block | stride | + |---|---|---| + | 1 min | 1 | 32 (the standard block) | + | 15 s | 4 | 92 | + | 2 s | 30 | 612 | + + `stride = 12 + n * 20`. Each 20-byte record carries 8 × uint16 + **little**-endian values — peak and half-period per channel — plus a 2-word + tail whose first word is `0000` on every real interval (a session ending + mid-block leaves buffer garbage in the remaining slots, which decoded as + peaks thousands of times the real value until that check was added). + **The standard 32-byte block is big-endian; this variant is not.** + + These 415 files (216 on BE18193 at 2 s intervals, 199 on BE9440 at 15 s) + previously decoded to nothing at all — and before that were being accepted + by the *waveform* codec, which returned garbage peaking up to 400× the + device-reported PPV. + + Ground truth `BE9440/K440L3AQ.T70H` — 5,710 intervals — matches its + Blastware ASCII export on **17,130/17,130** geo peaks, **22,840/22,840** + frequencies and **5,710/5,710** mic dB(L) values. Across all 455 affected + files, **1,354/1,365 (99.2%)** channel peaks match the device-reported PPV; + the 11 that don't are under-reads on BE9440 where the walk stops early. + - **`backfill_sidecars.py` now removes a stale `.h5` when nothing decodes.** It previously skipped the write "so we don't replace whatever's there with an empty placeholder", which silently preserved output from a superseded diff --git a/CLAUDE.md b/CLAUDE.md index 0af777e..49c4a1d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -284,6 +284,31 @@ rotation and corrupts every channel after it. Corpus result, end to end through the production path: **exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.** +### Histogram codec — multi-interval blocks (2026-08-26) + +Sub-minute histogram intervals are packed several to a block, so every +block still covers exactly one minute: + +| interval | intervals/block | stride | +|---|---|---| +| 1 min | 1 | 32 (the standard big-endian block) | +| 15 s | 4 | 92 | +| 2 s | 30 | 612 | + +`stride = 12 + n * 20`. Block = `[00][segment][ctr uint16 LE][0a][00]`, +then n x 20-byte records of 8 x uint16 **LITTLE**-endian values +(`T_peak, T_halfp, V_peak, V_halfp, L_peak, L_halfp, M_peak, M_halfp`) +plus a 2-word tail whose first word is `0000` on every real interval, +then a 6-byte block trailer. + +⚠ The standard 32-byte block is BIG-endian; this variant is LITTLE-endian. + +Recovers **415 files** (216 on BE18193, 199 on BE9440) that decoded to +nothing. Ground truth `BE9440/K440L3AQ.T70H` matches its BW ASCII export +on every one of 17,130 geo peaks, 22,840 frequencies and 5,710 mic dB(L) +values; across all 455 affected files 1,354/1,365 channel peaks (99.2%) +match the device-reported PPV. + ### Histogram codec — corrected 2026-08-25 The histogram block is **uniformly big-endian**, and the stream's final diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index 371e800..4fecbad 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -11,6 +11,7 @@ | Date | Section | Change | |---|---|---| +| 2026-08-26 | S7.6.2, S15 | **MULTI-INTERVAL HISTOGRAM BLOCKS - sub-minute intervals pack several per block.** The device always writes one minute of data per block, so a shorter histogram interval means more intervals packed into a longer block: 1 min -> 1 (the standard 32-byte block), 15 s -> 4 (stride 92), 2 s -> 30 (stride 612), with `stride = 12 + n * 20`. Each 20-byte record holds 8 x uint16 **little**-endian values (peak/half-period per channel) plus a 2-word tail whose first word is `0000` on every real interval. The standard block is big-endian - the variant is not. This recovers **415 files** (216 on BE18193 at 2 s, 199 on BE9440 at 15 s) that previously decoded to nothing, and before that were being accepted by the WAVEFORM codec and returning garbage up to 400x the device-reported PPV. | | 2026-08-25 (3) | S7.6.1, S15 | **THE WAVEFORM BODY IS A RECORD CHAIN, NOT A TAG STREAM — supersedes the segment-header model entirely.** Records are self-delimiting: `off+2` is a uint16 BE length and `next = off + 2 + len`; the chain ends on a record whose chan_id is `0x06`. `off+8` holds a 3-valued mode enum - `02 00` (14-byte header, anchors, cumulative deltas), `01 00` (10-byte, no anchors, ABSOLUTE values), `00 03` (10-byte, no tags at all, raw 12-bit absolute). **`40 NN` is an ordinary int16 BE data block of length 2*NN+2**, never a header; reading it as a 2*NN+16 header is what made walks drift, and the 'variable prefix' of 0/2/4/6/8 bytes reported earlier the same day was walker drift, exactly `4 - (old_stop - true_record_start)`. Verified: chain terminates on `06` in 1387/1388 files; all four channels equal length in **1388/1388** (was 156/1388); ASCII sample-count exact 72/75 -> **75/75**, fully exact 70/75 -> **73/75**; device PPV on a live decode **1306/1306** waveform (mean abs ratio error 0.00000) and **4458/4459** histogram. Also eliminated the walker-over-read class: 24 of those files were histograms the waveform codec was wrongly accepting. The superseded model is retained as `decode_waveform_legacy` because `micromate/idf_file.py` pins it for Thor IDFW body-offset search. | | 2026-08-25 (2) | S7.6.2, S15 | **HISTOGRAM BLOCK IS BIG-ENDIAN + terminal block tail.** The 32-byte histogram block's per-channel fields are uint16 **big-endian** (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], `V_halfperiod` [11:13], `L_peak` [13:15], `L_halfperiod` [15:17], `M_peak` [17:19], `M_halfperiod` [19:21]); only `block_ctr` [2:4] is little-endian. The marker is `block[4]` alone - the previous uint16 LE marker test at [4:6] forced `block[5] == 0` and thereby capped every geo peak at 255 counts (1.275 in/s), silently clipping larger peaks. The byte previously documented as a per-channel "annotation" is the high byte of the big-endian half-period, which is why it was non-zero exactly on sub-Hz intervals. Separately, the **final block of each stream carries tail `9c 06 00 42`** rather than `1e 0a 00 00` and holds arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram. Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly, plus 842,442 per-interval frequency comparisons with zero mismatches (previously 1 of 1196 files fully correct). | | 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. | @@ -3083,7 +3084,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger | **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open | | ~~**Variable-prefix segment descriptors**~~ - **RESOLVED 2026-08-25:** there is no variable prefix. The body is a chain of self-delimiting records (see S7.6.1); the 0/2/4/6/8-byte prefix was walker drift from reading `40 NN` as a segment header. All 25 affected files now chain cleanly to the `06` terminator. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 | | ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 | -| **Unmapped histogram block variant (415 files, 2 units)** - after the 2026-08-25 record-chain fix these stop decoding entirely: 216 on BE18193 and 199 on BE9440, all histograms. Their bodies open `00 00 00 01 0a 00`, which is a valid block header (marker `0a` at [4], block_ctr 256 at [2:4] LE), but `block[28:32]` is neither the standard `1e 0a 00 00` nor the terminal `9c 06 00 42`, and no fixed stride between 8 and 64 bytes puts a marker at [4] consistently. Bodies are very large (one is 360,573 bytes). Previously these were being decoded by the WAVEFORM codec, which accepted them and returned garbage peaking up to 400x the device-reported PPV - so this is a pre-existing gap the fix merely exposed, not a regression. Their stale `.h5` files are now removed by `scripts/backfill_sidecars.py` rather than left behind. Example: `BE18193/T193L2X2.1J0H`. | MEDIUM | 2026-08-25 | Still open | +| ~~**Unmapped histogram block variant (415 files, 2 units)**~~ - **RESOLVED 2026-08-26.** When the histogram interval is SHORTER than one minute the device packs several intervals into a single block so each block still covers exactly one minute: 1 min -> 1 interval (the standard 32-byte block), 15 s -> 4 (stride 92), 2 s -> 30 (stride 612). `stride = 12 + n * 20`. Block is `[00][segment][ctr uint16 LE][0a][00]` then n x 20-byte records of 8 x uint16 **LITTLE**-endian `T_peak,T_halfp,V_peak,V_halfp,L_peak,L_halfp,M_peak,M_halfp` plus a 2-word tail whose first word is `0000` on every real interval (a session ending mid-block leaves buffer garbage in the remaining slots), then a 6-byte block trailer. **Note the endianness flip** - the standard 32-byte block is big-endian. Ground truth `BE9440/K440L3AQ.T70H` (5,710 intervals at 15 s) decodes with 17,130/17,130 geo peaks, 22,840/22,840 frequencies and 5,710/5,710 mic dB(L) matching its Blastware ASCII export exactly; across all 455 affected files 1,354/1,365 channel peaks (99.2%) match the device-reported PPV. | RESOLVED | 2026-08-25 | Resolved 2026-08-26 | | **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open | | **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open | diff --git a/minimateplus/histogram_codec.py b/minimateplus/histogram_codec.py index d60bf07..b6746ec 100644 --- a/minimateplus/histogram_codec.py +++ b/minimateplus/histogram_codec.py @@ -263,7 +263,7 @@ def decode_histogram_body(body: bytes) -> Optional[dict]: to get 1-count ADC values, then ``count / 32767 * 10.0`` for in/s) - Mic channel: use ``waveform_codec.mic_count_to_db(count)`` """ - records = walk_body(body) + records = walk_body(body) or walk_multi_interval_blocks(body) if not records: return None return { @@ -285,7 +285,7 @@ def decode_histogram_body_full(body: bytes) -> Optional[List[dict]]: Returns ``None`` if the body has no valid blocks. """ - records = walk_body(body) + records = walk_body(body) or walk_multi_interval_blocks(body) return records if records else None @@ -305,3 +305,118 @@ def half_period_to_hz(halfp: int) -> Optional[float]: def geo_count_to_ins(count: int) -> float: """Convert a histogram geo peak count to in/s at Normal range.""" return count * _GEO_LSB_INS + + +# ── Multi-interval block variant (CONFIRMED 2026-08-26) ───────────────────── +# +# When the histogram interval is SHORTER than one minute, the device packs +# several intervals into a single block so that every block still covers +# exactly one minute of data: +# +# interval size intervals/block stride +# 1 minute 1 32 <- the standard block above +# 15 seconds 4 92 +# 2 seconds 30 612 +# +# stride = 12 + n_intervals * 20 +# +# Block layout: +# [0] 0x00 +# [1] segment_id (256 blocks per segment, same as the standard block) +# [2:4] block_ctr uint16 LE (0x0100.., resets each segment) +# [4] 0x0a marker +# [5] 0x00 +# [6 ...] n x 20-byte interval records, each carrying 8 x uint16 +# LITTLE-endian values: +# T_peak, T_halfperiod, V_peak, V_halfperiod, +# L_peak, L_halfperiod, M_peak, M_halfperiod +# then 2 more words; the first is 0x0000 on every real interval. +# [-6:] 6-byte block trailer +# +# ⚠ ENDIANNESS: the standard 32-byte block is BIG-endian. This variant is +# LITTLE-endian. Do not share the accessor. +# +# These files previously decoded to nothing at all — 415 of them in the +# production snapshot, 216 on BE18193 (2 s intervals) and 199 on BE9440 +# (15 s). Before that they were being accepted by the WAVEFORM codec, which +# returned garbage peaking up to 400x the device-reported PPV. +# +# Ground truth: BE9440/K440L3AQ.T70H (15 s intervals, 5,710 of them) decodes +# against its Blastware ASCII export with 17,130/17,130 geo peak counts, +# 22,840/22,840 frequencies and 5,710/5,710 mic dB(L) values matching exactly. + +_MULTI_HEADER_LEN = 6 +_MULTI_RECORD_LEN = 20 +_MULTI_TRAILER_LEN = 6 +# At least 2 records: a 1-record block would have stride 12 + 20 = 32, which +# collides with the standard big-endian block and mis-decodes it. +_MULTI_MIN_RECORDS = 2 +_MULTI_MAX_RECORDS = 64 + + +def _is_multi_header(body: bytes, off: int) -> bool: + return (off + _MULTI_HEADER_LEN <= len(body) + and body[off] == 0x00 + and body[off + 4] == 0x0A + and body[off + 5] == 0x00) + + +def detect_multi_interval_stride(body: bytes) -> Optional[int]: + """Block stride of a multi-interval histogram body, or None. + + Found by locating the second block header; validated against + ``stride = 12 + n * 20`` and confirmed on a third block where present. + """ + if not _is_multi_header(body, 0): + return None + lo = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MIN_RECORDS + hi = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MAX_RECORDS + for stride in range(lo, min(hi, len(body)) + 1, 2): + if (stride - 12) % _MULTI_RECORD_LEN: + continue + if not _is_multi_header(body, stride): + continue + # confirm on a third block when the body is long enough + if 2 * stride + _MULTI_HEADER_LEN <= len(body) and not _is_multi_header(body, 2 * stride): + continue + return stride + return None + + +def walk_multi_interval_blocks(body: bytes, + stride: Optional[int] = None) -> List[dict]: + """Decode a multi-interval histogram body into per-interval records.""" + if stride is None: + stride = detect_multi_interval_stride(body) + if not stride: + return [] + n_per_block = (stride - _MULTI_HEADER_LEN - _MULTI_TRAILER_LEN) // _MULTI_RECORD_LEN + if n_per_block < 1: + return [] + + def u16le(p: int) -> int: + return body[p] | (body[p + 1] << 8) + + out: List[dict] = [] + for off in range(0, len(body) - stride + 1, stride): + if not _is_multi_header(body, off): + break # end of the block run; trailer follows + for k in range(n_per_block): + q = off + _MULTI_HEADER_LEN + _MULTI_RECORD_LEN * k + # The first word of each record's 2-word tail is 0x0000 on every + # real interval. A session ending mid-block leaves the remaining + # slots filled with whatever was in the buffer; emitting those + # produced peaks thousands of times the device-reported PPV. + if u16le(q + 16) != 0: + return out + out.append({ + "segment_id": body[off + 1], + "block_ctr": u16le(off + 2), + "t_peak": u16le(q), "t_halfp": u16le(q + 2), + "v_peak": u16le(q + 4), "v_halfp": u16le(q + 6), + "l_peak": u16le(q + 8), "l_halfp": u16le(q + 10), + "m_peak": u16le(q + 12), "m_halfp": u16le(q + 14), + "meta_var": bytes(body[q + 16:q + 20]), + "is_terminal": False, + }) + return out diff --git a/tests/test_histogram_codec.py b/tests/test_histogram_codec.py index e37f60e..9d24817 100644 --- a/tests/test_histogram_codec.py +++ b/tests/test_histogram_codec.py @@ -471,3 +471,164 @@ def test_marker_is_single_byte_not_uint16(): r = decode_histogram_body_full(_mk_block(t_peak=0x0676)) assert r is not None, "block[5] != 0 must not disqualify the block" assert r[0]["t_peak"] == 0x0676 + + +# ── Multi-interval block variant (2026-08-26) ─────────────────────────────── +# +# When the histogram interval is SHORTER than one minute the device packs +# several intervals into one block, so that every block still covers exactly +# one minute of data: +# +# interval intervals/block stride +# 1 min 1 32 (the standard big-endian block) +# 15 s 4 92 +# 2 s 30 612 +# +# stride = 12 + n_intervals * 20 +# +# Block layout: +# [0] 0x00 +# [1] segment_id (256 blocks per segment) +# [2:4] block_ctr uint16 LE +# [4] 0x0a marker +# [5] 0x00 +# [6 ...] n x 20-byte interval records, each 8 x uint16 LITTLE-endian: +# T_peak, T_halfp, V_peak, V_halfp, +# L_peak, L_halfp, M_peak, M_halfp +# followed by 2 words (first is 0x0000) +# [-6:] 6-byte block trailer +# +# NOTE the endianness flip: the standard 32-byte block is big-endian, this +# variant is little-endian. +# +# Ground truth: BE9440/K440L3AQ.T70H (15 s intervals, 5,710 of them) decodes +# with 17,130/17,130 geo peak counts, 22,840/22,840 frequencies and +# 5,710/5,710 mic dB(L) values matching its Blastware ASCII export exactly. + +from minimateplus.histogram_codec import ( # noqa: E402 + detect_multi_interval_stride, + walk_multi_interval_blocks, +) + + +def _mk_multi_block(intervals, seg=0, ctr=256): + """Build one multi-interval block from a list of 8-tuples.""" + b = bytearray() + b += bytes([0x00, seg]) + b += int(ctr).to_bytes(2, "little") + b += bytes([0x0A, 0x00]) + for iv in intervals: + for v in iv: + b += int(v).to_bytes(2, "little") + b += (0).to_bytes(2, "little") + b += (5).to_bytes(2, "little") + b += bytes(6) + assert len(b) == 12 + 20 * len(intervals) + return bytes(b) + + +def test_stride_is_twelve_plus_twenty_per_interval(): + for n in (4, 30): + ivs = [(1, 1, 2, 2, 3, 3, 4, 4)] * n + body = _mk_multi_block(ivs, ctr=256) + _mk_multi_block(ivs, ctr=257) + assert detect_multi_interval_stride(body) == 12 + 20 * n + + +def test_multi_interval_block_decodes_all_four_channels(): + ivs = [(1, 1, 2, 2, 3, 3, 4, 4), (5, 6, 7, 8, 9, 10, 11, 12)] + body = _mk_multi_block(ivs) + _mk_multi_block(ivs, ctr=257) + recs = walk_multi_interval_blocks(body) + assert len(recs) == 4 + r = recs[0] + assert (r["t_peak"], r["v_peak"], r["l_peak"], r["m_peak"]) == (1, 2, 3, 4) + assert (r["t_halfp"], r["v_halfp"], r["l_halfp"], r["m_halfp"]) == (1, 2, 3, 4) + assert recs[1]["t_peak"] == 5 and recs[1]["m_halfp"] == 12 + + +def test_multi_interval_values_are_little_endian(): + """The standard 32-byte block is big-endian; this variant is not. + + A peak of 0x0100 must decode as 256, not 1. + """ + ivs = [(0x0100, 1, 1, 1, 1, 1, 1, 1), (1, 1, 1, 1, 1, 1, 1, 1)] + body = _mk_multi_block(ivs) + _mk_multi_block(ivs, ctr=257) + assert walk_multi_interval_blocks(body)[0]["t_peak"] == 0x0100 + + +def test_decode_histogram_body_falls_back_to_the_variant(): + ivs = [(1, 1, 2, 2, 3, 3, 4, 4)] * 4 + body = _mk_multi_block(ivs) + _mk_multi_block(ivs, ctr=257) + ch = decode_histogram_body(body) + assert ch is not None + assert len(ch["Tran"]) == 8 + assert ch["Long"][0] == 3 + + +def test_standard_blocks_still_take_precedence(): + """A body of standard 32-byte blocks must not be re-read as the variant.""" + std = _mk_block(t_peak=7) + _mk_block(t_peak=9, ctr=257, + tail=b"\x9c\x06\x00\x42") + ch = decode_histogram_body(std) + assert ch is not None and ch["Tran"] == [7, 9] + + +# ── Ground truth for the multi-interval variant ───────────────────────────── +# Fixture is gitignored (like the rest of tests/fixtures); skips when absent. + +_MULTI_FIXTURE = os.path.join( + os.path.dirname(__file__), "fixtures", "histogram-multi-interval", + "K440L3AQ.T70H", +) + + +@pytest.mark.skipif(not os.path.exists(_MULTI_FIXTURE), + reason="multi-interval fixture not present") +def test_multi_interval_matches_blastware_ascii_exactly(): + """BE9440/K440L3AQ.T70H — 5,710 intervals at 15 s, 4 per 92-byte block. + + Every geo peak, every frequency and every mic dB(L) in the file matches + the Blastware ASCII export: 17,130 / 22,840 / 5,710 values, zero + mismatches. Before this decoder the file produced nothing at all. + """ + import math + import re as _re + from minimateplus import blastware_file as _bwf + + raw = open(_MULTI_FIXTURE, "rb").read() + bs = _bwf._WAVEFORM_HEADER_SIZE + 21 + pos, fp = bs, -1 + while True: + pos = raw.find(b"\x0e\x08", pos) + if pos < 0 or pos + 26 > len(raw): + break + if 2015 <= ((raw[pos + 4] << 8) | raw[pos + 5]) <= 2050: + fp = pos + break + pos += 1 + recs = walk_multi_interval_blocks(raw[bs:fp]) + + rows = [] + for line in open(_MULTI_FIXTURE + "_ASCII.TXT", errors="replace"): + p = [x.strip() for x in line.split("\t")] + if len(p) >= 11 and _re.match(r"^\d{2}:\d{2}:\d{2}$", p[0]): + rows.append(p) + + assert len(recs) == len(rows) == 5710 + + def want_count(x): + return round(float(x) / 0.005) + + for rec, row in zip(recs, rows): + assert rec["t_peak"] == want_count(row[1]) + assert rec["v_peak"] == want_count(row[3]) + assert rec["l_peak"] == want_count(row[5]) + # mic dB(L) + assert abs((81.94 + 20 * math.log10(rec["m_peak"])) - float(row[9])) <= 0.06 + # frequency: half-period <= 5 is BW's ">100 Hz" sentinel + for hp, cell in ((rec["t_halfp"], row[2]), (rec["v_halfp"], row[4]), + (rec["l_halfp"], row[6]), (rec["m_halfp"], row[10])): + hz = None if hp <= 5 else 512.0 / hp + if cell.startswith(">"): + assert hz is None + elif not cell.startswith("<"): + assert hz is not None and abs(hz - float(cell)) <= max(0.55, float(cell) * 0.02)