feat(histogram): decode multi-interval blocks — recovers 415 files

Sub-minute histogram intervals are packed several to a block so that
every block still covers exactly one minute of data:

    interval   intervals/block   stride
    1 minute   1                 32     <- the standard big-endian block
    15 s       4                 92
    2 s        30                612

    stride = 12 + n * 20

Block = [00][segment][ctr uint16 LE][0a][00], then n x 20-byte records of
8 x uint16 LITTLE-endian values (T_peak, T_halfp, V_peak, V_halfp,
L_peak, L_halfp, M_peak, M_halfp) plus a 2-word tail whose first word is
0000 on every real interval, then a 6-byte block trailer.

The standard 32-byte block is BIG-endian; this variant is LITTLE-endian.

The tail-word check matters: a session ending mid-block leaves buffer
garbage in the remaining interval slots, which decoded as peaks
thousands of times the real value.  Stride detection also requires at
least 2 records, since a 1-record block would have stride 32 and
collide with the standard block.

Recovers 415 files that decoded to nothing: 216 on BE18193 (2 s
intervals) and 199 on BE9440 (15 s).  Before decoding to nothing they
were being accepted by the WAVEFORM codec, which returned garbage
peaking up to 400x the device-reported PPV.

Ground truth BE9440/K440L3AQ.T70H (5,710 intervals) matches its
Blastware ASCII export exactly: 17,130/17,130 geo peaks, 22,840/22,840
frequencies, 5,710/5,710 mic dB(L).  Across all 455 affected files,
1,354/1,365 channel peaks (99.2%) match the device-reported PPV; the 11
that don't are under-reads on BE9440 where the walk stops early.

Fixture (binary + ASCII) saved under tests/fixtures/, which is
gitignored per repo practice — the ground-truth test skips when absent.

Tests: 258 passed, failure list unchanged from baseline.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
2026-08-26 04:59:00 +00:00
co-authored by Claude Opus 5
parent 4c58a532de
commit 306104354b
5 changed files with 333 additions and 3 deletions
+28
View File
@@ -8,6 +8,34 @@ All notable changes to seismo-relay are documented here.
### Fixed ### Fixed
- **Sub-minute histogram intervals are packed several to a block — 415 files
recovered.** The device always writes one minute of data per block, so a
shorter interval just means more intervals in a longer block:
| interval | intervals/block | stride |
|---|---|---|
| 1 min | 1 | 32 (the standard block) |
| 15 s | 4 | 92 |
| 2 s | 30 | 612 |
`stride = 12 + n * 20`. Each 20-byte record carries 8 × uint16
**little**-endian values — peak and half-period per channel — plus a 2-word
tail whose first word is `0000` on every real interval (a session ending
mid-block leaves buffer garbage in the remaining slots, which decoded as
peaks thousands of times the real value until that check was added).
**The standard 32-byte block is big-endian; this variant is not.**
These 415 files (216 on BE18193 at 2 s intervals, 199 on BE9440 at 15 s)
previously decoded to nothing at all — and before that were being accepted
by the *waveform* codec, which returned garbage peaking up to 400× the
device-reported PPV.
Ground truth `BE9440/K440L3AQ.T70H` — 5,710 intervals — matches its
Blastware ASCII export on **17,130/17,130** geo peaks, **22,840/22,840**
frequencies and **5,710/5,710** mic dB(L) values. Across all 455 affected
files, **1,354/1,365 (99.2%)** channel peaks match the device-reported PPV;
the 11 that don't are under-reads on BE9440 where the walk stops early.
- **`backfill_sidecars.py` now removes a stale `.h5` when nothing decodes.** - **`backfill_sidecars.py` now removes a stale `.h5` when nothing decodes.**
It previously skipped the write "so we don't replace whatever's there with an It previously skipped the write "so we don't replace whatever's there with an
empty placeholder", which silently preserved output from a superseded empty placeholder", which silently preserved output from a superseded
+25
View File
@@ -284,6 +284,31 @@ rotation and corrupts every channel after it.
Corpus result, end to end through the production path: Corpus result, end to end through the production path:
**exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.** **exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.**
### Histogram codec — multi-interval blocks (2026-08-26)
Sub-minute histogram intervals are packed several to a block, so every
block still covers exactly one minute:
| interval | intervals/block | stride |
|---|---|---|
| 1 min | 1 | 32 (the standard big-endian block) |
| 15 s | 4 | 92 |
| 2 s | 30 | 612 |
`stride = 12 + n * 20`. Block = `[00][segment][ctr uint16 LE][0a][00]`,
then n x 20-byte records of 8 x uint16 **LITTLE**-endian values
(`T_peak, T_halfp, V_peak, V_halfp, L_peak, L_halfp, M_peak, M_halfp`)
plus a 2-word tail whose first word is `0000` on every real interval,
then a 6-byte block trailer.
⚠ The standard 32-byte block is BIG-endian; this variant is LITTLE-endian.
Recovers **415 files** (216 on BE18193, 199 on BE9440) that decoded to
nothing. Ground truth `BE9440/K440L3AQ.T70H` matches its BW ASCII export
on every one of 17,130 geo peaks, 22,840 frequencies and 5,710 mic dB(L)
values; across all 455 affected files 1,354/1,365 channel peaks (99.2%)
match the device-reported PPV.
### Histogram codec — corrected 2026-08-25 ### Histogram codec — corrected 2026-08-25
The histogram block is **uniformly big-endian**, and the stream's final The histogram block is **uniformly big-endian**, and the stream's final
+2 -1
View File
@@ -11,6 +11,7 @@
| Date | Section | Change | | Date | Section | Change |
|---|---|---| |---|---|---|
| 2026-08-26 | S7.6.2, S15 | **MULTI-INTERVAL HISTOGRAM BLOCKS - sub-minute intervals pack several per block.** The device always writes one minute of data per block, so a shorter histogram interval means more intervals packed into a longer block: 1 min -> 1 (the standard 32-byte block), 15 s -> 4 (stride 92), 2 s -> 30 (stride 612), with `stride = 12 + n * 20`. Each 20-byte record holds 8 x uint16 **little**-endian values (peak/half-period per channel) plus a 2-word tail whose first word is `0000` on every real interval. The standard block is big-endian - the variant is not. This recovers **415 files** (216 on BE18193 at 2 s, 199 on BE9440 at 15 s) that previously decoded to nothing, and before that were being accepted by the WAVEFORM codec and returning garbage up to 400x the device-reported PPV. |
| 2026-08-25 (3) | S7.6.1, S15 | **THE WAVEFORM BODY IS A RECORD CHAIN, NOT A TAG STREAM — supersedes the segment-header model entirely.** Records are self-delimiting: `off+2` is a uint16 BE length and `next = off + 2 + len`; the chain ends on a record whose chan_id is `0x06`. `off+8` holds a 3-valued mode enum - `02 00` (14-byte header, anchors, cumulative deltas), `01 00` (10-byte, no anchors, ABSOLUTE values), `00 03` (10-byte, no tags at all, raw 12-bit absolute). **`40 NN` is an ordinary int16 BE data block of length 2*NN+2**, never a header; reading it as a 2*NN+16 header is what made walks drift, and the 'variable prefix' of 0/2/4/6/8 bytes reported earlier the same day was walker drift, exactly `4 - (old_stop - true_record_start)`. Verified: chain terminates on `06` in 1387/1388 files; all four channels equal length in **1388/1388** (was 156/1388); ASCII sample-count exact 72/75 -> **75/75**, fully exact 70/75 -> **73/75**; device PPV on a live decode **1306/1306** waveform (mean abs ratio error 0.00000) and **4458/4459** histogram. Also eliminated the walker-over-read class: 24 of those files were histograms the waveform codec was wrongly accepting. The superseded model is retained as `decode_waveform_legacy` because `micromate/idf_file.py` pins it for Thor IDFW body-offset search. | | 2026-08-25 (3) | S7.6.1, S15 | **THE WAVEFORM BODY IS A RECORD CHAIN, NOT A TAG STREAM — supersedes the segment-header model entirely.** Records are self-delimiting: `off+2` is a uint16 BE length and `next = off + 2 + len`; the chain ends on a record whose chan_id is `0x06`. `off+8` holds a 3-valued mode enum - `02 00` (14-byte header, anchors, cumulative deltas), `01 00` (10-byte, no anchors, ABSOLUTE values), `00 03` (10-byte, no tags at all, raw 12-bit absolute). **`40 NN` is an ordinary int16 BE data block of length 2*NN+2**, never a header; reading it as a 2*NN+16 header is what made walks drift, and the 'variable prefix' of 0/2/4/6/8 bytes reported earlier the same day was walker drift, exactly `4 - (old_stop - true_record_start)`. Verified: chain terminates on `06` in 1387/1388 files; all four channels equal length in **1388/1388** (was 156/1388); ASCII sample-count exact 72/75 -> **75/75**, fully exact 70/75 -> **73/75**; device PPV on a live decode **1306/1306** waveform (mean abs ratio error 0.00000) and **4458/4459** histogram. Also eliminated the walker-over-read class: 24 of those files were histograms the waveform codec was wrongly accepting. The superseded model is retained as `decode_waveform_legacy` because `micromate/idf_file.py` pins it for Thor IDFW body-offset search. |
| 2026-08-25 (2) | S7.6.2, S15 | **HISTOGRAM BLOCK IS BIG-ENDIAN + terminal block tail.** The 32-byte histogram block's per-channel fields are uint16 **big-endian** (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], `V_halfperiod` [11:13], `L_peak` [13:15], `L_halfperiod` [15:17], `M_peak` [17:19], `M_halfperiod` [19:21]); only `block_ctr` [2:4] is little-endian. The marker is `block[4]` alone - the previous uint16 LE marker test at [4:6] forced `block[5] == 0` and thereby capped every geo peak at 255 counts (1.275 in/s), silently clipping larger peaks. The byte previously documented as a per-channel "annotation" is the high byte of the big-endian half-period, which is why it was non-zero exactly on sub-Hz intervals. Separately, the **final block of each stream carries tail `9c 06 00 42`** rather than `1e 0a 00 00` and holds arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram. Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly, plus 842,442 per-interval frequency comparisons with zero mismatches (previously 1 of 1196 files fully correct). | | 2026-08-25 (2) | S7.6.2, S15 | **HISTOGRAM BLOCK IS BIG-ENDIAN + terminal block tail.** The 32-byte histogram block's per-channel fields are uint16 **big-endian** (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], `V_halfperiod` [11:13], `L_peak` [13:15], `L_halfperiod` [15:17], `M_peak` [17:19], `M_halfperiod` [19:21]); only `block_ctr` [2:4] is little-endian. The marker is `block[4]` alone - the previous uint16 LE marker test at [4:6] forced `block[5] == 0` and thereby capped every geo peak at 255 counts (1.275 in/s), silently clipping larger peaks. The byte previously documented as a per-channel "annotation" is the high byte of the big-endian half-period, which is why it was non-zero exactly on sub-Hz intervals. Separately, the **final block of each stream carries tail `9c 06 00 42`** rather than `1e 0a 00 00` and holds arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram. Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly, plus 842,442 per-interval frequency comparisons with zero mismatches (previously 1 of 1196 files fully correct). |
| 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. | | 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. |
@@ -3083,7 +3084,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
| **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open | | **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open |
| ~~**Variable-prefix segment descriptors**~~ - **RESOLVED 2026-08-25:** there is no variable prefix. The body is a chain of self-delimiting records (see S7.6.1); the 0/2/4/6/8-byte prefix was walker drift from reading `40 NN` as a segment header. All 25 affected files now chain cleanly to the `06` terminator. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 | | ~~**Variable-prefix segment descriptors**~~ - **RESOLVED 2026-08-25:** there is no variable prefix. The body is a chain of self-delimiting records (see S7.6.1); the 0/2/4/6/8-byte prefix was walker drift from reading `40 NN` as a segment header. All 25 affected files now chain cleanly to the `06` terminator. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
| ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 | | ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
| **Unmapped histogram block variant (415 files, 2 units)** - after the 2026-08-25 record-chain fix these stop decoding entirely: 216 on BE18193 and 199 on BE9440, all histograms. Their bodies open `00 00 00 01 0a 00`, which is a valid block header (marker `0a` at [4], block_ctr 256 at [2:4] LE), but `block[28:32]` is neither the standard `1e 0a 00 00` nor the terminal `9c 06 00 42`, and no fixed stride between 8 and 64 bytes puts a marker at [4] consistently. Bodies are very large (one is 360,573 bytes). Previously these were being decoded by the WAVEFORM codec, which accepted them and returned garbage peaking up to 400x the device-reported PPV - so this is a pre-existing gap the fix merely exposed, not a regression. Their stale `.h5` files are now removed by `scripts/backfill_sidecars.py` rather than left behind. Example: `BE18193/T193L2X2.1J0H`. | MEDIUM | 2026-08-25 | Still open | | ~~**Unmapped histogram block variant (415 files, 2 units)**~~ - **RESOLVED 2026-08-26.** When the histogram interval is SHORTER than one minute the device packs several intervals into a single block so each block still covers exactly one minute: 1 min -> 1 interval (the standard 32-byte block), 15 s -> 4 (stride 92), 2 s -> 30 (stride 612). `stride = 12 + n * 20`. Block is `[00][segment][ctr uint16 LE][0a][00]` then n x 20-byte records of 8 x uint16 **LITTLE**-endian `T_peak,T_halfp,V_peak,V_halfp,L_peak,L_halfp,M_peak,M_halfp` plus a 2-word tail whose first word is `0000` on every real interval (a session ending mid-block leaves buffer garbage in the remaining slots), then a 6-byte block trailer. **Note the endianness flip** - the standard 32-byte block is big-endian. Ground truth `BE9440/K440L3AQ.T70H` (5,710 intervals at 15 s) decodes with 17,130/17,130 geo peaks, 22,840/22,840 frequencies and 5,710/5,710 mic dB(L) matching its Blastware ASCII export exactly; across all 455 affected files 1,354/1,365 channel peaks (99.2%) match the device-reported PPV. | RESOLVED | 2026-08-25 | Resolved 2026-08-26 |
| **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open | | **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open |
| **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open | | **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open |
+117 -2
View File
@@ -263,7 +263,7 @@ def decode_histogram_body(body: bytes) -> Optional[dict]:
to get 1-count ADC values, then ``count / 32767 * 10.0`` for in/s) to get 1-count ADC values, then ``count / 32767 * 10.0`` for in/s)
- Mic channel: use ``waveform_codec.mic_count_to_db(count)`` - Mic channel: use ``waveform_codec.mic_count_to_db(count)``
""" """
records = walk_body(body) records = walk_body(body) or walk_multi_interval_blocks(body)
if not records: if not records:
return None return None
return { return {
@@ -285,7 +285,7 @@ def decode_histogram_body_full(body: bytes) -> Optional[List[dict]]:
Returns ``None`` if the body has no valid blocks. Returns ``None`` if the body has no valid blocks.
""" """
records = walk_body(body) records = walk_body(body) or walk_multi_interval_blocks(body)
return records if records else None return records if records else None
@@ -305,3 +305,118 @@ def half_period_to_hz(halfp: int) -> Optional[float]:
def geo_count_to_ins(count: int) -> float: def geo_count_to_ins(count: int) -> float:
"""Convert a histogram geo peak count to in/s at Normal range.""" """Convert a histogram geo peak count to in/s at Normal range."""
return count * _GEO_LSB_INS return count * _GEO_LSB_INS
# ── Multi-interval block variant (CONFIRMED 2026-08-26) ─────────────────────
#
# When the histogram interval is SHORTER than one minute, the device packs
# several intervals into a single block so that every block still covers
# exactly one minute of data:
#
# interval size intervals/block stride
# 1 minute 1 32 <- the standard block above
# 15 seconds 4 92
# 2 seconds 30 612
#
# stride = 12 + n_intervals * 20
#
# Block layout:
# [0] 0x00
# [1] segment_id (256 blocks per segment, same as the standard block)
# [2:4] block_ctr uint16 LE (0x0100.., resets each segment)
# [4] 0x0a marker
# [5] 0x00
# [6 ...] n x 20-byte interval records, each carrying 8 x uint16
# LITTLE-endian values:
# T_peak, T_halfperiod, V_peak, V_halfperiod,
# L_peak, L_halfperiod, M_peak, M_halfperiod
# then 2 more words; the first is 0x0000 on every real interval.
# [-6:] 6-byte block trailer
#
# ⚠ ENDIANNESS: the standard 32-byte block is BIG-endian. This variant is
# LITTLE-endian. Do not share the accessor.
#
# These files previously decoded to nothing at all — 415 of them in the
# production snapshot, 216 on BE18193 (2 s intervals) and 199 on BE9440
# (15 s). Before that they were being accepted by the WAVEFORM codec, which
# returned garbage peaking up to 400x the device-reported PPV.
#
# Ground truth: BE9440/K440L3AQ.T70H (15 s intervals, 5,710 of them) decodes
# against its Blastware ASCII export with 17,130/17,130 geo peak counts,
# 22,840/22,840 frequencies and 5,710/5,710 mic dB(L) values matching exactly.
_MULTI_HEADER_LEN = 6
_MULTI_RECORD_LEN = 20
_MULTI_TRAILER_LEN = 6
# At least 2 records: a 1-record block would have stride 12 + 20 = 32, which
# collides with the standard big-endian block and mis-decodes it.
_MULTI_MIN_RECORDS = 2
_MULTI_MAX_RECORDS = 64
def _is_multi_header(body: bytes, off: int) -> bool:
return (off + _MULTI_HEADER_LEN <= len(body)
and body[off] == 0x00
and body[off + 4] == 0x0A
and body[off + 5] == 0x00)
def detect_multi_interval_stride(body: bytes) -> Optional[int]:
"""Block stride of a multi-interval histogram body, or None.
Found by locating the second block header; validated against
``stride = 12 + n * 20`` and confirmed on a third block where present.
"""
if not _is_multi_header(body, 0):
return None
lo = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MIN_RECORDS
hi = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MAX_RECORDS
for stride in range(lo, min(hi, len(body)) + 1, 2):
if (stride - 12) % _MULTI_RECORD_LEN:
continue
if not _is_multi_header(body, stride):
continue
# confirm on a third block when the body is long enough
if 2 * stride + _MULTI_HEADER_LEN <= len(body) and not _is_multi_header(body, 2 * stride):
continue
return stride
return None
def walk_multi_interval_blocks(body: bytes,
stride: Optional[int] = None) -> List[dict]:
"""Decode a multi-interval histogram body into per-interval records."""
if stride is None:
stride = detect_multi_interval_stride(body)
if not stride:
return []
n_per_block = (stride - _MULTI_HEADER_LEN - _MULTI_TRAILER_LEN) // _MULTI_RECORD_LEN
if n_per_block < 1:
return []
def u16le(p: int) -> int:
return body[p] | (body[p + 1] << 8)
out: List[dict] = []
for off in range(0, len(body) - stride + 1, stride):
if not _is_multi_header(body, off):
break # end of the block run; trailer follows
for k in range(n_per_block):
q = off + _MULTI_HEADER_LEN + _MULTI_RECORD_LEN * k
# The first word of each record's 2-word tail is 0x0000 on every
# real interval. A session ending mid-block leaves the remaining
# slots filled with whatever was in the buffer; emitting those
# produced peaks thousands of times the device-reported PPV.
if u16le(q + 16) != 0:
return out
out.append({
"segment_id": body[off + 1],
"block_ctr": u16le(off + 2),
"t_peak": u16le(q), "t_halfp": u16le(q + 2),
"v_peak": u16le(q + 4), "v_halfp": u16le(q + 6),
"l_peak": u16le(q + 8), "l_halfp": u16le(q + 10),
"m_peak": u16le(q + 12), "m_halfp": u16le(q + 14),
"meta_var": bytes(body[q + 16:q + 20]),
"is_terminal": False,
})
return out
+161
View File
@@ -471,3 +471,164 @@ def test_marker_is_single_byte_not_uint16():
r = decode_histogram_body_full(_mk_block(t_peak=0x0676)) r = decode_histogram_body_full(_mk_block(t_peak=0x0676))
assert r is not None, "block[5] != 0 must not disqualify the block" assert r is not None, "block[5] != 0 must not disqualify the block"
assert r[0]["t_peak"] == 0x0676 assert r[0]["t_peak"] == 0x0676
# ── Multi-interval block variant (2026-08-26) ───────────────────────────────
#
# When the histogram interval is SHORTER than one minute the device packs
# several intervals into one block, so that every block still covers exactly
# one minute of data:
#
# interval intervals/block stride
# 1 min 1 32 (the standard big-endian block)
# 15 s 4 92
# 2 s 30 612
#
# stride = 12 + n_intervals * 20
#
# Block layout:
# [0] 0x00
# [1] segment_id (256 blocks per segment)
# [2:4] block_ctr uint16 LE
# [4] 0x0a marker
# [5] 0x00
# [6 ...] n x 20-byte interval records, each 8 x uint16 LITTLE-endian:
# T_peak, T_halfp, V_peak, V_halfp,
# L_peak, L_halfp, M_peak, M_halfp
# followed by 2 words (first is 0x0000)
# [-6:] 6-byte block trailer
#
# NOTE the endianness flip: the standard 32-byte block is big-endian, this
# variant is little-endian.
#
# Ground truth: BE9440/K440L3AQ.T70H (15 s intervals, 5,710 of them) decodes
# with 17,130/17,130 geo peak counts, 22,840/22,840 frequencies and
# 5,710/5,710 mic dB(L) values matching its Blastware ASCII export exactly.
from minimateplus.histogram_codec import ( # noqa: E402
detect_multi_interval_stride,
walk_multi_interval_blocks,
)
def _mk_multi_block(intervals, seg=0, ctr=256):
"""Build one multi-interval block from a list of 8-tuples."""
b = bytearray()
b += bytes([0x00, seg])
b += int(ctr).to_bytes(2, "little")
b += bytes([0x0A, 0x00])
for iv in intervals:
for v in iv:
b += int(v).to_bytes(2, "little")
b += (0).to_bytes(2, "little")
b += (5).to_bytes(2, "little")
b += bytes(6)
assert len(b) == 12 + 20 * len(intervals)
return bytes(b)
def test_stride_is_twelve_plus_twenty_per_interval():
for n in (4, 30):
ivs = [(1, 1, 2, 2, 3, 3, 4, 4)] * n
body = _mk_multi_block(ivs, ctr=256) + _mk_multi_block(ivs, ctr=257)
assert detect_multi_interval_stride(body) == 12 + 20 * n
def test_multi_interval_block_decodes_all_four_channels():
ivs = [(1, 1, 2, 2, 3, 3, 4, 4), (5, 6, 7, 8, 9, 10, 11, 12)]
body = _mk_multi_block(ivs) + _mk_multi_block(ivs, ctr=257)
recs = walk_multi_interval_blocks(body)
assert len(recs) == 4
r = recs[0]
assert (r["t_peak"], r["v_peak"], r["l_peak"], r["m_peak"]) == (1, 2, 3, 4)
assert (r["t_halfp"], r["v_halfp"], r["l_halfp"], r["m_halfp"]) == (1, 2, 3, 4)
assert recs[1]["t_peak"] == 5 and recs[1]["m_halfp"] == 12
def test_multi_interval_values_are_little_endian():
"""The standard 32-byte block is big-endian; this variant is not.
A peak of 0x0100 must decode as 256, not 1.
"""
ivs = [(0x0100, 1, 1, 1, 1, 1, 1, 1), (1, 1, 1, 1, 1, 1, 1, 1)]
body = _mk_multi_block(ivs) + _mk_multi_block(ivs, ctr=257)
assert walk_multi_interval_blocks(body)[0]["t_peak"] == 0x0100
def test_decode_histogram_body_falls_back_to_the_variant():
ivs = [(1, 1, 2, 2, 3, 3, 4, 4)] * 4
body = _mk_multi_block(ivs) + _mk_multi_block(ivs, ctr=257)
ch = decode_histogram_body(body)
assert ch is not None
assert len(ch["Tran"]) == 8
assert ch["Long"][0] == 3
def test_standard_blocks_still_take_precedence():
"""A body of standard 32-byte blocks must not be re-read as the variant."""
std = _mk_block(t_peak=7) + _mk_block(t_peak=9, ctr=257,
tail=b"\x9c\x06\x00\x42")
ch = decode_histogram_body(std)
assert ch is not None and ch["Tran"] == [7, 9]
# ── Ground truth for the multi-interval variant ─────────────────────────────
# Fixture is gitignored (like the rest of tests/fixtures); skips when absent.
_MULTI_FIXTURE = os.path.join(
os.path.dirname(__file__), "fixtures", "histogram-multi-interval",
"K440L3AQ.T70H",
)
@pytest.mark.skipif(not os.path.exists(_MULTI_FIXTURE),
reason="multi-interval fixture not present")
def test_multi_interval_matches_blastware_ascii_exactly():
"""BE9440/K440L3AQ.T70H — 5,710 intervals at 15 s, 4 per 92-byte block.
Every geo peak, every frequency and every mic dB(L) in the file matches
the Blastware ASCII export: 17,130 / 22,840 / 5,710 values, zero
mismatches. Before this decoder the file produced nothing at all.
"""
import math
import re as _re
from minimateplus import blastware_file as _bwf
raw = open(_MULTI_FIXTURE, "rb").read()
bs = _bwf._WAVEFORM_HEADER_SIZE + 21
pos, fp = bs, -1
while True:
pos = raw.find(b"\x0e\x08", pos)
if pos < 0 or pos + 26 > len(raw):
break
if 2015 <= ((raw[pos + 4] << 8) | raw[pos + 5]) <= 2050:
fp = pos
break
pos += 1
recs = walk_multi_interval_blocks(raw[bs:fp])
rows = []
for line in open(_MULTI_FIXTURE + "_ASCII.TXT", errors="replace"):
p = [x.strip() for x in line.split("\t")]
if len(p) >= 11 and _re.match(r"^\d{2}:\d{2}:\d{2}$", p[0]):
rows.append(p)
assert len(recs) == len(rows) == 5710
def want_count(x):
return round(float(x) / 0.005)
for rec, row in zip(recs, rows):
assert rec["t_peak"] == want_count(row[1])
assert rec["v_peak"] == want_count(row[3])
assert rec["l_peak"] == want_count(row[5])
# mic dB(L)
assert abs((81.94 + 20 * math.log10(rec["m_peak"])) - float(row[9])) <= 0.06
# frequency: half-period <= 5 is BW's ">100 Hz" sentinel
for hp, cell in ((rec["t_halfp"], row[2]), (rec["v_halfp"], row[4]),
(rec["l_halfp"], row[6]), (rec["m_halfp"], row[10])):
hz = None if hp <= 5 else 512.0 / hp
if cell.startswith(">"):
assert hz is None
elif not cell.startswith("<"):
assert hz is not None and abs(hz - float(cell)) <= max(0.55, float(cell) * 0.02)