update to 0.26.0. Big chonking update including 0.23, 0.24, and 0.25 as well. #33
@@ -8,6 +8,34 @@ All notable changes to seismo-relay are documented here.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Sub-minute histogram intervals are packed several to a block — 415 files
|
||||
recovered.** The device always writes one minute of data per block, so a
|
||||
shorter interval just means more intervals in a longer block:
|
||||
|
||||
| interval | intervals/block | stride |
|
||||
|---|---|---|
|
||||
| 1 min | 1 | 32 (the standard block) |
|
||||
| 15 s | 4 | 92 |
|
||||
| 2 s | 30 | 612 |
|
||||
|
||||
`stride = 12 + n * 20`. Each 20-byte record carries 8 × uint16
|
||||
**little**-endian values — peak and half-period per channel — plus a 2-word
|
||||
tail whose first word is `0000` on every real interval (a session ending
|
||||
mid-block leaves buffer garbage in the remaining slots, which decoded as
|
||||
peaks thousands of times the real value until that check was added).
|
||||
**The standard 32-byte block is big-endian; this variant is not.**
|
||||
|
||||
These 415 files (216 on BE18193 at 2 s intervals, 199 on BE9440 at 15 s)
|
||||
previously decoded to nothing at all — and before that were being accepted
|
||||
by the *waveform* codec, which returned garbage peaking up to 400× the
|
||||
device-reported PPV.
|
||||
|
||||
Ground truth `BE9440/K440L3AQ.T70H` — 5,710 intervals — matches its
|
||||
Blastware ASCII export on **17,130/17,130** geo peaks, **22,840/22,840**
|
||||
frequencies and **5,710/5,710** mic dB(L) values. Across all 455 affected
|
||||
files, **1,354/1,365 (99.2%)** channel peaks match the device-reported PPV;
|
||||
the 11 that don't are under-reads on BE9440 where the walk stops early.
|
||||
|
||||
- **`backfill_sidecars.py` now removes a stale `.h5` when nothing decodes.**
|
||||
It previously skipped the write "so we don't replace whatever's there with an
|
||||
empty placeholder", which silently preserved output from a superseded
|
||||
|
||||
@@ -284,6 +284,31 @@ rotation and corrupts every channel after it.
|
||||
Corpus result, end to end through the production path:
|
||||
**exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.**
|
||||
|
||||
### Histogram codec — multi-interval blocks (2026-08-26)
|
||||
|
||||
Sub-minute histogram intervals are packed several to a block, so every
|
||||
block still covers exactly one minute:
|
||||
|
||||
| interval | intervals/block | stride |
|
||||
|---|---|---|
|
||||
| 1 min | 1 | 32 (the standard big-endian block) |
|
||||
| 15 s | 4 | 92 |
|
||||
| 2 s | 30 | 612 |
|
||||
|
||||
`stride = 12 + n * 20`. Block = `[00][segment][ctr uint16 LE][0a][00]`,
|
||||
then n x 20-byte records of 8 x uint16 **LITTLE**-endian values
|
||||
(`T_peak, T_halfp, V_peak, V_halfp, L_peak, L_halfp, M_peak, M_halfp`)
|
||||
plus a 2-word tail whose first word is `0000` on every real interval,
|
||||
then a 6-byte block trailer.
|
||||
|
||||
⚠ The standard 32-byte block is BIG-endian; this variant is LITTLE-endian.
|
||||
|
||||
Recovers **415 files** (216 on BE18193, 199 on BE9440) that decoded to
|
||||
nothing. Ground truth `BE9440/K440L3AQ.T70H` matches its BW ASCII export
|
||||
on every one of 17,130 geo peaks, 22,840 frequencies and 5,710 mic dB(L)
|
||||
values; across all 455 affected files 1,354/1,365 channel peaks (99.2%)
|
||||
match the device-reported PPV.
|
||||
|
||||
### Histogram codec — corrected 2026-08-25
|
||||
|
||||
The histogram block is **uniformly big-endian**, and the stream's final
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
|
||||
| Date | Section | Change |
|
||||
|---|---|---|
|
||||
| 2026-08-26 | S7.6.2, S15 | **MULTI-INTERVAL HISTOGRAM BLOCKS - sub-minute intervals pack several per block.** The device always writes one minute of data per block, so a shorter histogram interval means more intervals packed into a longer block: 1 min -> 1 (the standard 32-byte block), 15 s -> 4 (stride 92), 2 s -> 30 (stride 612), with `stride = 12 + n * 20`. Each 20-byte record holds 8 x uint16 **little**-endian values (peak/half-period per channel) plus a 2-word tail whose first word is `0000` on every real interval. The standard block is big-endian - the variant is not. This recovers **415 files** (216 on BE18193 at 2 s, 199 on BE9440 at 15 s) that previously decoded to nothing, and before that were being accepted by the WAVEFORM codec and returning garbage up to 400x the device-reported PPV. |
|
||||
| 2026-08-25 (3) | S7.6.1, S15 | **THE WAVEFORM BODY IS A RECORD CHAIN, NOT A TAG STREAM — supersedes the segment-header model entirely.** Records are self-delimiting: `off+2` is a uint16 BE length and `next = off + 2 + len`; the chain ends on a record whose chan_id is `0x06`. `off+8` holds a 3-valued mode enum - `02 00` (14-byte header, anchors, cumulative deltas), `01 00` (10-byte, no anchors, ABSOLUTE values), `00 03` (10-byte, no tags at all, raw 12-bit absolute). **`40 NN` is an ordinary int16 BE data block of length 2*NN+2**, never a header; reading it as a 2*NN+16 header is what made walks drift, and the 'variable prefix' of 0/2/4/6/8 bytes reported earlier the same day was walker drift, exactly `4 - (old_stop - true_record_start)`. Verified: chain terminates on `06` in 1387/1388 files; all four channels equal length in **1388/1388** (was 156/1388); ASCII sample-count exact 72/75 -> **75/75**, fully exact 70/75 -> **73/75**; device PPV on a live decode **1306/1306** waveform (mean abs ratio error 0.00000) and **4458/4459** histogram. Also eliminated the walker-over-read class: 24 of those files were histograms the waveform codec was wrongly accepting. The superseded model is retained as `decode_waveform_legacy` because `micromate/idf_file.py` pins it for Thor IDFW body-offset search. |
|
||||
| 2026-08-25 (2) | S7.6.2, S15 | **HISTOGRAM BLOCK IS BIG-ENDIAN + terminal block tail.** The 32-byte histogram block's per-channel fields are uint16 **big-endian** (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], `V_halfperiod` [11:13], `L_peak` [13:15], `L_halfperiod` [15:17], `M_peak` [17:19], `M_halfperiod` [19:21]); only `block_ctr` [2:4] is little-endian. The marker is `block[4]` alone - the previous uint16 LE marker test at [4:6] forced `block[5] == 0` and thereby capped every geo peak at 255 counts (1.275 in/s), silently clipping larger peaks. The byte previously documented as a per-channel "annotation" is the high byte of the big-endian half-period, which is why it was non-zero exactly on sub-Hz intervals. Separately, the **final block of each stream carries tail `9c 06 00 42`** rather than `1e 0a 00 00` and holds arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram. Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly, plus 842,442 per-interval frequency comparisons with zero mismatches (previously 1 of 1196 files fully correct). |
|
||||
| 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. |
|
||||
@@ -3083,7 +3084,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
|
||||
| **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open |
|
||||
| ~~**Variable-prefix segment descriptors**~~ - **RESOLVED 2026-08-25:** there is no variable prefix. The body is a chain of self-delimiting records (see S7.6.1); the 0/2/4/6/8-byte prefix was walker drift from reading `40 NN` as a segment header. All 25 affected files now chain cleanly to the `06` terminator. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
|
||||
| ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
|
||||
| **Unmapped histogram block variant (415 files, 2 units)** - after the 2026-08-25 record-chain fix these stop decoding entirely: 216 on BE18193 and 199 on BE9440, all histograms. Their bodies open `00 00 00 01 0a 00`, which is a valid block header (marker `0a` at [4], block_ctr 256 at [2:4] LE), but `block[28:32]` is neither the standard `1e 0a 00 00` nor the terminal `9c 06 00 42`, and no fixed stride between 8 and 64 bytes puts a marker at [4] consistently. Bodies are very large (one is 360,573 bytes). Previously these were being decoded by the WAVEFORM codec, which accepted them and returned garbage peaking up to 400x the device-reported PPV - so this is a pre-existing gap the fix merely exposed, not a regression. Their stale `.h5` files are now removed by `scripts/backfill_sidecars.py` rather than left behind. Example: `BE18193/T193L2X2.1J0H`. | MEDIUM | 2026-08-25 | Still open |
|
||||
| ~~**Unmapped histogram block variant (415 files, 2 units)**~~ - **RESOLVED 2026-08-26.** When the histogram interval is SHORTER than one minute the device packs several intervals into a single block so each block still covers exactly one minute: 1 min -> 1 interval (the standard 32-byte block), 15 s -> 4 (stride 92), 2 s -> 30 (stride 612). `stride = 12 + n * 20`. Block is `[00][segment][ctr uint16 LE][0a][00]` then n x 20-byte records of 8 x uint16 **LITTLE**-endian `T_peak,T_halfp,V_peak,V_halfp,L_peak,L_halfp,M_peak,M_halfp` plus a 2-word tail whose first word is `0000` on every real interval (a session ending mid-block leaves buffer garbage in the remaining slots), then a 6-byte block trailer. **Note the endianness flip** - the standard 32-byte block is big-endian. Ground truth `BE9440/K440L3AQ.T70H` (5,710 intervals at 15 s) decodes with 17,130/17,130 geo peaks, 22,840/22,840 frequencies and 5,710/5,710 mic dB(L) matching its Blastware ASCII export exactly; across all 455 affected files 1,354/1,365 channel peaks (99.2%) match the device-reported PPV. | RESOLVED | 2026-08-25 | Resolved 2026-08-26 |
|
||||
| **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open |
|
||||
| **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open |
|
||||
|
||||
|
||||
@@ -263,7 +263,7 @@ def decode_histogram_body(body: bytes) -> Optional[dict]:
|
||||
to get 1-count ADC values, then ``count / 32767 * 10.0`` for in/s)
|
||||
- Mic channel: use ``waveform_codec.mic_count_to_db(count)``
|
||||
"""
|
||||
records = walk_body(body)
|
||||
records = walk_body(body) or walk_multi_interval_blocks(body)
|
||||
if not records:
|
||||
return None
|
||||
return {
|
||||
@@ -285,7 +285,7 @@ def decode_histogram_body_full(body: bytes) -> Optional[List[dict]]:
|
||||
|
||||
Returns ``None`` if the body has no valid blocks.
|
||||
"""
|
||||
records = walk_body(body)
|
||||
records = walk_body(body) or walk_multi_interval_blocks(body)
|
||||
return records if records else None
|
||||
|
||||
|
||||
@@ -305,3 +305,118 @@ def half_period_to_hz(halfp: int) -> Optional[float]:
|
||||
def geo_count_to_ins(count: int) -> float:
|
||||
"""Convert a histogram geo peak count to in/s at Normal range."""
|
||||
return count * _GEO_LSB_INS
|
||||
|
||||
|
||||
# ── Multi-interval block variant (CONFIRMED 2026-08-26) ─────────────────────
|
||||
#
|
||||
# When the histogram interval is SHORTER than one minute, the device packs
|
||||
# several intervals into a single block so that every block still covers
|
||||
# exactly one minute of data:
|
||||
#
|
||||
# interval size intervals/block stride
|
||||
# 1 minute 1 32 <- the standard block above
|
||||
# 15 seconds 4 92
|
||||
# 2 seconds 30 612
|
||||
#
|
||||
# stride = 12 + n_intervals * 20
|
||||
#
|
||||
# Block layout:
|
||||
# [0] 0x00
|
||||
# [1] segment_id (256 blocks per segment, same as the standard block)
|
||||
# [2:4] block_ctr uint16 LE (0x0100.., resets each segment)
|
||||
# [4] 0x0a marker
|
||||
# [5] 0x00
|
||||
# [6 ...] n x 20-byte interval records, each carrying 8 x uint16
|
||||
# LITTLE-endian values:
|
||||
# T_peak, T_halfperiod, V_peak, V_halfperiod,
|
||||
# L_peak, L_halfperiod, M_peak, M_halfperiod
|
||||
# then 2 more words; the first is 0x0000 on every real interval.
|
||||
# [-6:] 6-byte block trailer
|
||||
#
|
||||
# ⚠ ENDIANNESS: the standard 32-byte block is BIG-endian. This variant is
|
||||
# LITTLE-endian. Do not share the accessor.
|
||||
#
|
||||
# These files previously decoded to nothing at all — 415 of them in the
|
||||
# production snapshot, 216 on BE18193 (2 s intervals) and 199 on BE9440
|
||||
# (15 s). Before that they were being accepted by the WAVEFORM codec, which
|
||||
# returned garbage peaking up to 400x the device-reported PPV.
|
||||
#
|
||||
# Ground truth: BE9440/K440L3AQ.T70H (15 s intervals, 5,710 of them) decodes
|
||||
# against its Blastware ASCII export with 17,130/17,130 geo peak counts,
|
||||
# 22,840/22,840 frequencies and 5,710/5,710 mic dB(L) values matching exactly.
|
||||
|
||||
_MULTI_HEADER_LEN = 6
|
||||
_MULTI_RECORD_LEN = 20
|
||||
_MULTI_TRAILER_LEN = 6
|
||||
# At least 2 records: a 1-record block would have stride 12 + 20 = 32, which
|
||||
# collides with the standard big-endian block and mis-decodes it.
|
||||
_MULTI_MIN_RECORDS = 2
|
||||
_MULTI_MAX_RECORDS = 64
|
||||
|
||||
|
||||
def _is_multi_header(body: bytes, off: int) -> bool:
|
||||
return (off + _MULTI_HEADER_LEN <= len(body)
|
||||
and body[off] == 0x00
|
||||
and body[off + 4] == 0x0A
|
||||
and body[off + 5] == 0x00)
|
||||
|
||||
|
||||
def detect_multi_interval_stride(body: bytes) -> Optional[int]:
|
||||
"""Block stride of a multi-interval histogram body, or None.
|
||||
|
||||
Found by locating the second block header; validated against
|
||||
``stride = 12 + n * 20`` and confirmed on a third block where present.
|
||||
"""
|
||||
if not _is_multi_header(body, 0):
|
||||
return None
|
||||
lo = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MIN_RECORDS
|
||||
hi = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MAX_RECORDS
|
||||
for stride in range(lo, min(hi, len(body)) + 1, 2):
|
||||
if (stride - 12) % _MULTI_RECORD_LEN:
|
||||
continue
|
||||
if not _is_multi_header(body, stride):
|
||||
continue
|
||||
# confirm on a third block when the body is long enough
|
||||
if 2 * stride + _MULTI_HEADER_LEN <= len(body) and not _is_multi_header(body, 2 * stride):
|
||||
continue
|
||||
return stride
|
||||
return None
|
||||
|
||||
|
||||
def walk_multi_interval_blocks(body: bytes,
|
||||
stride: Optional[int] = None) -> List[dict]:
|
||||
"""Decode a multi-interval histogram body into per-interval records."""
|
||||
if stride is None:
|
||||
stride = detect_multi_interval_stride(body)
|
||||
if not stride:
|
||||
return []
|
||||
n_per_block = (stride - _MULTI_HEADER_LEN - _MULTI_TRAILER_LEN) // _MULTI_RECORD_LEN
|
||||
if n_per_block < 1:
|
||||
return []
|
||||
|
||||
def u16le(p: int) -> int:
|
||||
return body[p] | (body[p + 1] << 8)
|
||||
|
||||
out: List[dict] = []
|
||||
for off in range(0, len(body) - stride + 1, stride):
|
||||
if not _is_multi_header(body, off):
|
||||
break # end of the block run; trailer follows
|
||||
for k in range(n_per_block):
|
||||
q = off + _MULTI_HEADER_LEN + _MULTI_RECORD_LEN * k
|
||||
# The first word of each record's 2-word tail is 0x0000 on every
|
||||
# real interval. A session ending mid-block leaves the remaining
|
||||
# slots filled with whatever was in the buffer; emitting those
|
||||
# produced peaks thousands of times the device-reported PPV.
|
||||
if u16le(q + 16) != 0:
|
||||
return out
|
||||
out.append({
|
||||
"segment_id": body[off + 1],
|
||||
"block_ctr": u16le(off + 2),
|
||||
"t_peak": u16le(q), "t_halfp": u16le(q + 2),
|
||||
"v_peak": u16le(q + 4), "v_halfp": u16le(q + 6),
|
||||
"l_peak": u16le(q + 8), "l_halfp": u16le(q + 10),
|
||||
"m_peak": u16le(q + 12), "m_halfp": u16le(q + 14),
|
||||
"meta_var": bytes(body[q + 16:q + 20]),
|
||||
"is_terminal": False,
|
||||
})
|
||||
return out
|
||||
|
||||
@@ -471,3 +471,164 @@ def test_marker_is_single_byte_not_uint16():
|
||||
r = decode_histogram_body_full(_mk_block(t_peak=0x0676))
|
||||
assert r is not None, "block[5] != 0 must not disqualify the block"
|
||||
assert r[0]["t_peak"] == 0x0676
|
||||
|
||||
|
||||
# ── Multi-interval block variant (2026-08-26) ───────────────────────────────
|
||||
#
|
||||
# When the histogram interval is SHORTER than one minute the device packs
|
||||
# several intervals into one block, so that every block still covers exactly
|
||||
# one minute of data:
|
||||
#
|
||||
# interval intervals/block stride
|
||||
# 1 min 1 32 (the standard big-endian block)
|
||||
# 15 s 4 92
|
||||
# 2 s 30 612
|
||||
#
|
||||
# stride = 12 + n_intervals * 20
|
||||
#
|
||||
# Block layout:
|
||||
# [0] 0x00
|
||||
# [1] segment_id (256 blocks per segment)
|
||||
# [2:4] block_ctr uint16 LE
|
||||
# [4] 0x0a marker
|
||||
# [5] 0x00
|
||||
# [6 ...] n x 20-byte interval records, each 8 x uint16 LITTLE-endian:
|
||||
# T_peak, T_halfp, V_peak, V_halfp,
|
||||
# L_peak, L_halfp, M_peak, M_halfp
|
||||
# followed by 2 words (first is 0x0000)
|
||||
# [-6:] 6-byte block trailer
|
||||
#
|
||||
# NOTE the endianness flip: the standard 32-byte block is big-endian, this
|
||||
# variant is little-endian.
|
||||
#
|
||||
# Ground truth: BE9440/K440L3AQ.T70H (15 s intervals, 5,710 of them) decodes
|
||||
# with 17,130/17,130 geo peak counts, 22,840/22,840 frequencies and
|
||||
# 5,710/5,710 mic dB(L) values matching its Blastware ASCII export exactly.
|
||||
|
||||
from minimateplus.histogram_codec import ( # noqa: E402
|
||||
detect_multi_interval_stride,
|
||||
walk_multi_interval_blocks,
|
||||
)
|
||||
|
||||
|
||||
def _mk_multi_block(intervals, seg=0, ctr=256):
|
||||
"""Build one multi-interval block from a list of 8-tuples."""
|
||||
b = bytearray()
|
||||
b += bytes([0x00, seg])
|
||||
b += int(ctr).to_bytes(2, "little")
|
||||
b += bytes([0x0A, 0x00])
|
||||
for iv in intervals:
|
||||
for v in iv:
|
||||
b += int(v).to_bytes(2, "little")
|
||||
b += (0).to_bytes(2, "little")
|
||||
b += (5).to_bytes(2, "little")
|
||||
b += bytes(6)
|
||||
assert len(b) == 12 + 20 * len(intervals)
|
||||
return bytes(b)
|
||||
|
||||
|
||||
def test_stride_is_twelve_plus_twenty_per_interval():
|
||||
for n in (4, 30):
|
||||
ivs = [(1, 1, 2, 2, 3, 3, 4, 4)] * n
|
||||
body = _mk_multi_block(ivs, ctr=256) + _mk_multi_block(ivs, ctr=257)
|
||||
assert detect_multi_interval_stride(body) == 12 + 20 * n
|
||||
|
||||
|
||||
def test_multi_interval_block_decodes_all_four_channels():
|
||||
ivs = [(1, 1, 2, 2, 3, 3, 4, 4), (5, 6, 7, 8, 9, 10, 11, 12)]
|
||||
body = _mk_multi_block(ivs) + _mk_multi_block(ivs, ctr=257)
|
||||
recs = walk_multi_interval_blocks(body)
|
||||
assert len(recs) == 4
|
||||
r = recs[0]
|
||||
assert (r["t_peak"], r["v_peak"], r["l_peak"], r["m_peak"]) == (1, 2, 3, 4)
|
||||
assert (r["t_halfp"], r["v_halfp"], r["l_halfp"], r["m_halfp"]) == (1, 2, 3, 4)
|
||||
assert recs[1]["t_peak"] == 5 and recs[1]["m_halfp"] == 12
|
||||
|
||||
|
||||
def test_multi_interval_values_are_little_endian():
|
||||
"""The standard 32-byte block is big-endian; this variant is not.
|
||||
|
||||
A peak of 0x0100 must decode as 256, not 1.
|
||||
"""
|
||||
ivs = [(0x0100, 1, 1, 1, 1, 1, 1, 1), (1, 1, 1, 1, 1, 1, 1, 1)]
|
||||
body = _mk_multi_block(ivs) + _mk_multi_block(ivs, ctr=257)
|
||||
assert walk_multi_interval_blocks(body)[0]["t_peak"] == 0x0100
|
||||
|
||||
|
||||
def test_decode_histogram_body_falls_back_to_the_variant():
|
||||
ivs = [(1, 1, 2, 2, 3, 3, 4, 4)] * 4
|
||||
body = _mk_multi_block(ivs) + _mk_multi_block(ivs, ctr=257)
|
||||
ch = decode_histogram_body(body)
|
||||
assert ch is not None
|
||||
assert len(ch["Tran"]) == 8
|
||||
assert ch["Long"][0] == 3
|
||||
|
||||
|
||||
def test_standard_blocks_still_take_precedence():
|
||||
"""A body of standard 32-byte blocks must not be re-read as the variant."""
|
||||
std = _mk_block(t_peak=7) + _mk_block(t_peak=9, ctr=257,
|
||||
tail=b"\x9c\x06\x00\x42")
|
||||
ch = decode_histogram_body(std)
|
||||
assert ch is not None and ch["Tran"] == [7, 9]
|
||||
|
||||
|
||||
# ── Ground truth for the multi-interval variant ─────────────────────────────
|
||||
# Fixture is gitignored (like the rest of tests/fixtures); skips when absent.
|
||||
|
||||
_MULTI_FIXTURE = os.path.join(
|
||||
os.path.dirname(__file__), "fixtures", "histogram-multi-interval",
|
||||
"K440L3AQ.T70H",
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not os.path.exists(_MULTI_FIXTURE),
|
||||
reason="multi-interval fixture not present")
|
||||
def test_multi_interval_matches_blastware_ascii_exactly():
|
||||
"""BE9440/K440L3AQ.T70H — 5,710 intervals at 15 s, 4 per 92-byte block.
|
||||
|
||||
Every geo peak, every frequency and every mic dB(L) in the file matches
|
||||
the Blastware ASCII export: 17,130 / 22,840 / 5,710 values, zero
|
||||
mismatches. Before this decoder the file produced nothing at all.
|
||||
"""
|
||||
import math
|
||||
import re as _re
|
||||
from minimateplus import blastware_file as _bwf
|
||||
|
||||
raw = open(_MULTI_FIXTURE, "rb").read()
|
||||
bs = _bwf._WAVEFORM_HEADER_SIZE + 21
|
||||
pos, fp = bs, -1
|
||||
while True:
|
||||
pos = raw.find(b"\x0e\x08", pos)
|
||||
if pos < 0 or pos + 26 > len(raw):
|
||||
break
|
||||
if 2015 <= ((raw[pos + 4] << 8) | raw[pos + 5]) <= 2050:
|
||||
fp = pos
|
||||
break
|
||||
pos += 1
|
||||
recs = walk_multi_interval_blocks(raw[bs:fp])
|
||||
|
||||
rows = []
|
||||
for line in open(_MULTI_FIXTURE + "_ASCII.TXT", errors="replace"):
|
||||
p = [x.strip() for x in line.split("\t")]
|
||||
if len(p) >= 11 and _re.match(r"^\d{2}:\d{2}:\d{2}$", p[0]):
|
||||
rows.append(p)
|
||||
|
||||
assert len(recs) == len(rows) == 5710
|
||||
|
||||
def want_count(x):
|
||||
return round(float(x) / 0.005)
|
||||
|
||||
for rec, row in zip(recs, rows):
|
||||
assert rec["t_peak"] == want_count(row[1])
|
||||
assert rec["v_peak"] == want_count(row[3])
|
||||
assert rec["l_peak"] == want_count(row[5])
|
||||
# mic dB(L)
|
||||
assert abs((81.94 + 20 * math.log10(rec["m_peak"])) - float(row[9])) <= 0.06
|
||||
# frequency: half-period <= 5 is BW's ">100 Hz" sentinel
|
||||
for hp, cell in ((rec["t_halfp"], row[2]), (rec["v_halfp"], row[4]),
|
||||
(rec["l_halfp"], row[6]), (rec["m_halfp"], row[10])):
|
||||
hz = None if hp <= 5 else 512.0 / hp
|
||||
if cell.startswith(">"):
|
||||
assert hz is None
|
||||
elif not cell.startswith("<"):
|
||||
assert hz is not None and abs(hz - float(cell)) <= max(0.55, float(cell) * 0.02)
|
||||
|
||||
Reference in New Issue
Block a user