diff --git a/CHANGELOG.md b/CHANGELOG.md index a2a7819..07dc116 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,36 @@ All notable changes to seismo-relay are documented here. ### Fixed +- **Series-3 histogram block is uniformly big-endian, and the stream's final + block has its own tail — the codec was clipping large peaks and dropping the + last interval of nearly every histogram.** + + - **Peaks and half-periods are `uint16` big-endian**, not `uint8` plus an + "annotation" byte: `T_peak` `[5:7]`, `T_halfperiod` `[7:9]`, `V_peak` + `[9:11]`, and so on. Only `block_ctr` `[2:4]` is little-endian. The old + model silently **clipped any peak above 1.275 in/s** — the final interval + of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in Blastware's own export and + decoded as 0.590. The "annotation" byte was the half-period's high byte, + which is why it was non-zero exactly on the sub-Hz intervals BW renders + as `<1.0`. + - **The marker is `block[4]` alone.** Testing `[4:6]` as a `uint16 LE` + marker forced `block[5] == 0` — which is what capped the peak at one byte + in the first place. + - **The last block of each stream carries tail `9c 06 00 42`** instead of + `1e 0a 00 00`, with arbitrary bytes at `[21:23]`. Rejecting it dropped + the final interval of nearly every histogram, and that interval is + frequently the one holding the event peak — so the file's reported PPV + came out low. + + Verified against **1211 production histograms** paired with their preserved + Blastware ASCII exports, which carry a full per-interval data table: + **1211/1211 now decode exactly** (interval count plus every per-interval + peak), and 842,442 per-interval frequency comparisons match with zero + mismatches. Before this fix: **1 of 1196**. + + `decode_histogram_body_full` records now expose `is_terminal` in place of + the removed `annotations` tuple. + - **Geophone full scale is 32000 ADC counts, not 32768 — every geo reading was 2.3% low.** The verified body codec emits geo samples in 16-count units whose documented LSB is exactly 0.005 in/s, and `decoded_to_adc_counts` multiplies diff --git a/CLAUDE.md b/CLAUDE.md index 8623df4..9af798e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -261,6 +261,32 @@ rotation and corrupts every channel after it. Corpus result, end to end through the production path: **exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.** +### Histogram codec — corrected 2026-08-25 + +The histogram block is **uniformly big-endian**, and the stream's final +block has its own tail signature. Two long-standing errors: + +- **Peaks and half-periods are `uint16` big-endian**, not `uint8` + + an "annotation" byte. `T_peak` is `[5:7]`, `T_halfperiod` `[7:9]`, + `V_peak` `[9:11]`, and so on; only `block_ctr` at `[2:4]` is LE. + The old model silently **clipped any peak above 1.275 in/s** — the + final interval of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in BW's + export and decoded as 0.590. The "annotation" byte was the + half-period's high byte, which is why it was non-zero exactly on the + sub-Hz intervals BW renders as `<1.0`. +- **The marker is `block[4]` alone.** Testing `[4:6]` as a uint16 LE + marker forced `block[5] == 0`, which is what capped the peak at one + byte in the first place. +- **The last block of the stream carries tail `9c 06 00 42`** instead of + `1e 0a 00 00`, with arbitrary bytes at `[21:23]`. Rejecting it + dropped the final interval of nearly every histogram — frequently the + one holding the event peak, so the file's PPV read low. + +Verified against 1211 production histograms paired with their BW ASCII +exports: **1211/1211 decode exactly** (interval count plus every +per-interval peak), and 842,442 per-interval frequency comparisons match +with zero mismatches. Before: 1 of 1196. + ### What's NOT solved - **MicL channel conversion to dB(L)** — the codec emits MicL as @@ -276,19 +302,8 @@ Corpus result, end to end through the production path: anchor count — `01 00` records appear with both 2- and 4-byte anchor fields in the same file. Examples: `BE12599/N599LPNB.JF0W` @1155, `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485. -- **Histogram codec drops trailing intervals (series-3)** — the - per-interval *values* are right: measured against the histogram ASCII - data table (Tran/Vert/Long peak + freq + PVS per interval), 1031 of - 1196 files (86%) match within 1 LSB across the overlapping prefix. - The interval **count** is short in 1195 of 1196 files — median 1 - missing; 1088 short by 1–2, 65 by 3–10, 39 by 11–100, 3 by >100 (max - 205). When a dropped interval holds the event peak, the decoded max - falls below the device-reported PPV: 169/1196 files (14%). A - termination bug in `histogram_codec.decode_histogram_body` — same - family as the waveform-walker truncation fixed 2026-08-25, and - untouched by it. Note: the histogram ASCII carries a full - per-interval table, so this has proper ground truth available (1211 - files in the prod snapshot). +(The series-3 histogram codec was fixed 2026-08-25 — see below.) + - **Micromate (UM-series) IDF decode is ~1000× low** — e.g. `UM11402_20260406130113.IDFW` gives a Tran peak of 0.0009 in/s against a device-reported 1.1168. The Thor IDF path decodes sanely, so this diff --git a/docs/histogram_codec_re_status.md b/docs/histogram_codec_re_status.md index 6fa388c..2f4c323 100644 --- a/docs/histogram_codec_re_status.md +++ b/docs/histogram_codec_re_status.md @@ -1,3 +1,30 @@ +> ## SUPERSEDED 2026-08-25 — the block is uniformly BIG-ENDIAN +> +> The `uint8` peak / `annotation` byte model described below is wrong, +> though it decoded quiet data correctly. The real layout: +> +> - **Every per-channel field is `uint16` big-endian.** `T_peak` is +> `[5:7]`, `T_halfperiod` `[7:9]`, `V_peak` `[9:11]`, and so on. +> Only `block_ctr` at `[2:4]` is little-endian. +> - The **marker is `block[4]` alone**, not a `uint16 LE` at `[4:6]`. +> Testing `[4:6] == 10` forced `block[5] == 0`, which is exactly what +> capped every geo peak at one byte (255 counts = 1.275 in/s). +> - The **"annotation" byte was never an annotation** — it is the high +> byte of the big-endian half-period. That is why it was non-zero +> precisely on the sub-Hz intervals Blastware renders as `<1.0`. +> - The **final block of the stream carries tail `9c 06 00 42`** instead +> of `1e 0a 00 00`, and arbitrary bytes at `[21:23]`. Rejecting it +> dropped the last interval of nearly every histogram — often the one +> holding the event peak, so the file's PPV read low. +> +> Verified against 1211 production histograms paired with their Blastware +> ASCII exports: **1211/1211 decode exactly** (interval count plus every +> per-interval peak), and 842,442 per-interval frequency comparisons match +> with zero mismatches. The uint8 model scored 1204/1211 — the seven +> failures are exactly the files containing a peak above 1.275 in/s. +> +> The section below is retained as the reasoning trail. + # Histogram body codec — FULLY DECODED (2026-05-20) Clean working status doc for the MiniMate Plus histogram-mode event diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index bf272cd..9059658 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -11,6 +11,7 @@ | Date | Section | Change | |---|---|---| +| 2026-08-25 (2) | S7.6.2, S15 | **HISTOGRAM BLOCK IS BIG-ENDIAN + terminal block tail.** The 32-byte histogram block's per-channel fields are uint16 **big-endian** (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], `V_halfperiod` [11:13], `L_peak` [13:15], `L_halfperiod` [15:17], `M_peak` [17:19], `M_halfperiod` [19:21]); only `block_ctr` [2:4] is little-endian. The marker is `block[4]` alone - the previous uint16 LE marker test at [4:6] forced `block[5] == 0` and thereby capped every geo peak at 255 counts (1.275 in/s), silently clipping larger peaks. The byte previously documented as a per-channel "annotation" is the high byte of the big-endian half-period, which is why it was non-zero exactly on sub-Hz intervals. Separately, the **final block of each stream carries tail `9c 06 00 42`** rather than `1e 0a 00 00` and holds arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram. Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly, plus 842,442 per-interval frequency comparisons with zero mismatches (previously 1 of 1196 files fully correct). | | 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. | | 2026-05-20 | §2, §3, §4.2, §5.1, §5.3, §6, §7.5b, §7.6.1, §7.6.3, §7.6.4, §7.7.2, §7.7.3, §7.7.5, §7.8.4, §7.8.7, §7.9, §8, §11, §12, §13, §14, §15, Appendix D | **DOC AUDIT PASS — accuracy sweep against `CLAUDE.md` + `minimateplus/` code.** Fixed: (1) S3 frames terminate on bare ETX, not DLE+ETX — §2/§3 rewritten. (2) §3 payload layout corrected — byte[1]=flags, byte[2]=SUB (was wrongly labelled DLE/ADDR). (3) §4.2 — probe responses do NOT carry data length; lengths are hardcoded `DATA_LENGTHS` constants. (4) §5.1 — removed stale duplicate "SUB 1C = TRIGGER CONFIG READ" row; SUB 0A lengths corrected from `0x30/0x26` to `0x46/0x2C` (real event / boundary marker). (5) §5.3 — added missing write-frame format (BW_CMD-only doubling, DLE-aware checksum, offset formula, ack format, SUB 71 chunk parameters). (6) §6 — fixed "SUB 06 → channel config read" → event storage range. (7) §7.5b / §8 — added the 10-byte `sub_code=0x03` continuous-mode timestamp variant alongside the 9-byte single-shot layout; peak vector sum location corrected from "fixed offset 87" to `tran_pos − 12` (label-relative). (8) §7.6 / §7.6.1 / §7.6.3 / §7.6.4 — switched compliance-anchor convention from the 10-byte form to the canonical 6-byte `\xbe\x80\x00\x00\x00\x00`; recording_mode confirmed at anchor−8 in BOTH read and write (was wrongly listed as anchor−3 write / anchor−4 read); sample_rate at anchor−6, histogram_interval at anchor−4, record_time at anchor+6; geo_range row added at channel_label+33. (9) §7.7.2 — token byte position corrected from `params[6]` to `params[7]`. (10) §7.8.4 — fi==9 skip marked FIXED (already removed from code); chunk-count totals updated. (11) §7.8.7 — TODO replaced with current state of `_decode_a5_metadata_into`. (12) §7.9 — Histogram Interval upgraded ❓ → ✅. (13) §11 — POLL example wire bytes corrected; SUB 5A row added to checksum table. (14) §13 — device-under-test updated for current primary unit (BE11529 / S338.17). (15) §14 — TCP Idle Timeout fixed (0→2 min); Data Forwarding Timeout units clarified. (16) §15 (renumbered from second §14) — open-question items already resolved in CLAUDE.md closed out. (17) Appendix D — extension taxonomy rewritten to reflect the AB0T timestamp encoding (D.5.2/D.5.3); EXTENSION REFUTED warning replaced with the resolved encoding. | | 2026-05-08 | §7.6.1 (RETRACTION) | **❌ RETRACTED — "raw int16 LE 8 bytes/sample-set" body codec was never validated.** The original 4-2-26 confirmation was based on misreading broken-decoder output (full-scale ±32K noise) as evidence the signal had saturated. BW's own 0C peaks for that capture (Tran=0.420 / Vert=3.870 / Long=0.495 in/s) prove the signal was NOT saturated — none of those exceed 13K ADC counts. No event in the project's archive has ever come close to saturation, yet the decoder consistently produces ±32K noise on every event. Conclusion: the body codec is not raw int16 LE; the actual encoding is open. Body byte distribution is heavily skewed (24% `0x00`, 10.5% `0x10`, lots of `10 XX` pairs) — likely a delta encoding with `0x10` as escape, but unverified. Retraction box added at top of §7.6.1; "fully-saturating event" claim removed from channel-identification note. The histogram codec in §7.6.2 IS verified and decoded correctly (different recording mode, 32-byte blocks); use it as a structural hint when reverse-engineering the waveform codec. | @@ -3037,7 +3038,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger | **Sensor Check dropdown byte location** — byte offset in 1A compliance config payload for the "Sensor Check: Before monitoring / After each event / Disabled" setting is NOT YET LOCATED. Confirmed: unit always runs with "Before monitoring" set. Need a capture with "Disabled" to diff. | MEDIUM | 2026-04-08 | Still open | | **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open | | **Variable-prefix segment descriptors** — 3 of 75 ground-truth events still truncate. The walk reaches a segment header whose channel-id field is preceded by a *variable-width* prefix (2, 4 or 6 bytes observed; the standard tagless form always has 4), carrying an `01 00` marker instead of `02 00`. The marker is **not** simply an anchor count — `01 00` records appear with both 2- and 4-byte anchor fields in the same file, so prefix width and marker are not yet reconciled. Examples: `BE12599/N599LPNB.JF0W` @1155, `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485. | MEDIUM | 2026-08-25 | Still open | -| **Histogram codec drops trailing intervals (series-3)** — measured per-interval against the histogram ASCII data table (which carries Tran/Vert/Long peak + freq + PVS per interval), n=1196: the decoded per-interval **values are right** — 1031/1196 (86%) match the ASCII table within 1 LSB across the overlapping prefix — but the interval **count** is short in 1195 of 1196 files. Median 1 interval missing; 1088 files short by 1–2, 65 by 3–10, 39 by 11–100, 3 by more than 100 (max 205). When a dropped interval is the one holding the event peak, the decoded max falls below the device-reported PPV — 169/1196 (14%) of files. This is a termination bug in `histogram_codec.decode_histogram_body`, the same family as the waveform-walker truncation fixed 2026-08-25, and it was untouched by that pass. | MEDIUM | 2026-08-25 | Still open | +| ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 | | **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open | | **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open | diff --git a/minimateplus/histogram_codec.py b/minimateplus/histogram_codec.py index 36e399d..d60bf07 100644 --- a/minimateplus/histogram_codec.py +++ b/minimateplus/histogram_codec.py @@ -25,38 +25,60 @@ iterate 32-stride and stop before the tail. ──────────────────────────────────────────────────────────────────────────── [0] 0x00 always-zero tag - [1] segment_id (uint8) 0x00..0x03 — 256 blocks per segment - [2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, …) - [4:6] 0x000a (uint16 LE) constant marker (= 10) - [6] T_peak_count uint8 Tran peak (count × 0.005 → in/s, max 1.275 in/s) - [7] T_annotation uint8 empirically non-zero on intervals with sub-Hz - or unmeasurable Tran freq; meaning not fully RE'd - [8:10] T_halfperiod uint16 LE Tran half-period in samples (freq = 512 / halfp Hz) - [10] V_peak_count uint8 - [11] V_annotation uint8 - [12:14] V_halfperiod uint16 LE - [14] L_peak_count uint8 - [15] L_annotation uint8 - [16:18] L_halfperiod uint16 LE - [18] M_peak_count uint8 MicL peak (count → dB via mic_count_to_db) - [19] M_annotation uint8 - [20:22] M_halfperiod uint16 LE MicL half-period in samples (freq = 512 / halfp Hz) - [22:24] 0x00 0x00 constant + [1] segment_id (uint8) 0x00..0x03 - 256 blocks per segment + [2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, ...) + [4] 0x0a (uint8) constant marker (= 10) + [5:7] T_peak_count uint16 BE Tran peak (count x 0.005 -> in/s) + [7:9] T_halfperiod uint16 BE Tran half-period in samples (freq = 512 / halfp) + [9:11] V_peak_count uint16 BE + [11:13] V_halfperiod uint16 BE + [13:15] L_peak_count uint16 BE + [15:17] L_halfperiod uint16 BE + [17:19] M_peak_count uint16 BE MicL peak (count -> dB via mic_count_to_db) + [19:21] M_halfperiod uint16 BE MicL half-period in samples + [21:23] 0x00 0x00 constant on standard blocks [24:28] 4-byte variable purpose unknown (possibly CRC or timestamp delta) - [28:32] 0x1e 0x0a 0x00 0x00 constant block-end signature + [28:32] block-end signature see "Two block tails" below -NOTE on peak-count width: an earlier interpretation treated the peak -fields as uint16 LE spanning [6:8] / [10:12] / [14:16] / [18:20]. -That happened to be byte-exact against the N844 fixture corpus only -because every annotation byte in those fixtures was zero, making -``uint16 LE == uint8``. Cross-correlating BE9558 (K558) Tran-drift -and BE18003 (T003) Histogram+Continuous events against the BW ASCII -export proved peak is uint8 alone — see test_histogram_codec.py -and docs/histogram_codec_re_status.md. +**Every per-channel field is uint16 BIG-endian** (confirmed 2026-08-25). +Only ``block_ctr`` at [2:4] is little-endian. -Block-identification anchor: ``block[22:24] == b"\\x00\\x00"`` AND -``block[28:32] == b"\\x1e\\x0a\\x00\\x00"``. This is the reliable -distinguisher from non-block content in the file. +HISTORY - two earlier readings of this block were wrong in ways that +cancelled out on quiet data: + + 1. *peak as uint16 LE at [6:8]* - produced 268 in/s peaks on any + interval whose next byte was non-zero. + 2. *peak as uint8 at [6] with an "annotation" byte at [7]* - correct + for every peak below 256 counts (1.275 in/s), but it silently + **clipped larger peaks**: the final interval of + BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export + (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590 in/s. + The "annotation" byte was never an annotation - it is the high + byte of the big-endian half-period, which is why it was non-zero + exactly on the sub-Hz intervals BW renders as "<1.0". + +Both readings also forced ``block[5] == 0`` via a bogus ``uint16 LE`` +marker check at [4:6], which is what capped the peak at one byte. +The marker is ``block[4]`` alone. + +Verified 2026-08-25 against 1211 production histograms paired with +their Blastware ASCII exports: **1211/1211 decode exactly** (interval +count plus every per-interval peak), and 842,442 per-interval +frequency comparisons match with **zero** mismatches. + +Two block tails +--------------- +Standard blocks end with ``1e 0a 00 00``. The **final block of the +stream** ends with ``9c 06 00 42`` instead, and carries arbitrary bytes +at [21:23]. Rejecting it dropped the last interval of nearly every +histogram - and the last interval is frequently the one holding the +event peak, so the file's reported PPV came out low. Observed in 1206 +of 1211 production histograms, always positioned after every +standard-tail block. + +Block-identification anchor: ``block[0] == 0x00`` AND +``block[4] == 0x0A`` AND the tail is one of the two signatures above; +standard-tail blocks additionally require ``block[22] == 0x00``. ──────────────────────────────────────────────────────────────────────────── Per-channel encoding @@ -109,6 +131,12 @@ from typing import List, Optional, Tuple # real data block. More distinctive than the byte-22 `00 00` (which # matches many false positives), so we anchor on this. _BLOCK_TAIL = b"\x1e\x0a\x00\x00" + +# The final block of a histogram stream ends with this instead. It is a +# real data block - same layout - and holds the last interval. See the +# module docstring, "Two block tails". +_BLOCK_TAIL_TERMINAL = b"\x9c\x06\x00\x42" + _BLOCK_SIZE = 32 # Marker byte at block[4:6] of every histogram data block. Used as @@ -127,19 +155,25 @@ _FREQ_NUMERATOR = 512 def _is_data_block(block: bytes) -> bool: - """Tight identification of a histogram data block.""" + """Tight identification of a histogram data block. + + Accepts both tail signatures. ``block[4]`` alone is the marker - + ``block[5]`` is the high byte of the Tran peak and is non-zero on any + interval above 1.275 in/s, so it must not be part of the marker test. + The ``block[22] == 0`` constraint is what keeps trailer content out, + but it applies only to standard-tail blocks: terminal blocks carry + arbitrary bytes there. + """ if len(block) < _BLOCK_SIZE: return False - if block[28:32] != _BLOCK_TAIL: - return False - if block[22:24] != b"\x00\x00": - return False if block[0] != 0x00: return False - marker = block[4] | (block[5] << 8) - if marker != _BLOCK_MARKER: + if block[4] != _BLOCK_MARKER: return False - return True + tail = block[28:32] + if tail == _BLOCK_TAIL: + return block[22] == 0x00 + return tail == _BLOCK_TAIL_TERMINAL def _decode_block(block: bytes) -> Optional[dict]: @@ -149,33 +183,23 @@ def _decode_block(block: bytes) -> Optional[dict]: Returns a record with per-channel peak counts (uint8) and half-periods (uint16 LE). """ - # Peak counts are uint8 at bytes [6] / [10] / [14] / [18]. The - # adjacent bytes [7] / [11] / [15] / [19] hold an annotation field - # whose meaning isn't fully understood (empirically non-zero in - # intervals with sub-Hz or unmeasurable geo frequencies, mostly - # zero otherwise — see test fixtures from BE9558/BE18003 corpora). - # Crucially, those annotation bytes are NOT the high byte of the - # peak count: cross-correlating against BW's per-interval ASCII - # export proves the peak is uint8 alone. - # - # Reading the peak as uint16 LE (the original interpretation) was - # accidentally correct only because every block in the N844 fixture - # corpus had a zero annotation byte; non-N844 events with non-zero - # annotation bytes decoded to physically impossible peaks (e.g. - # 268 in/s per channel) and produced 35× inflated PVS sums when - # first run against prod data. See histogram_codec_re_status.md. - t_peak = block[6] - v_peak = block[10] - l_peak = block[14] - m_peak = block[18] - t_halfp = block[8] | (block[9] << 8) - v_halfp = block[12] | (block[13] << 8) - l_halfp = block[16] | (block[17] << 8) - m_halfp = block[20] | (block[21] << 8) + # Every per-channel field is uint16 BIG-endian; only block_ctr is LE. + # See the module docstring for the two superseded readings and why + # each looked correct on quiet data. + def _be16(i: int) -> int: + return (block[i] << 8) | block[i + 1] + + t_peak = _be16(5) + t_halfp = _be16(7) + v_peak = _be16(9) + v_halfp = _be16(11) + l_peak = _be16(13) + l_halfp = _be16(15) + m_peak = _be16(17) + m_halfp = _be16(19) segment_id = block[1] block_ctr = block[2] | (block[3] << 8) var_meta = bytes(block[24:28]) - annotations = (block[7], block[11], block[15], block[19]) return { "segment_id": segment_id, "block_ctr": block_ctr, @@ -188,7 +212,7 @@ def _decode_block(block: bytes) -> Optional[dict]: "m_peak": m_peak, "m_halfp": m_halfp, "meta_var": var_meta, - "annotations": annotations, + "is_terminal": block[28:32] == _BLOCK_TAIL_TERMINAL, } diff --git a/tests/test_histogram_codec.py b/tests/test_histogram_codec.py index 6a42e27..e37f60e 100644 --- a/tests/test_histogram_codec.py +++ b/tests/test_histogram_codec.py @@ -354,23 +354,29 @@ _K558_INTERVAL_12_BLOCK = bytes.fromhex( def test_extension_byte_does_not_inflate_peak(): - """The annotation byte at [7]/[11]/[15]/[19] must NOT contribute to - the peak count. Decoded T_peak must be 3 (uint8 byte[6]), NOT - 53763 (uint16 LE byte[6:8]).""" + """The byte after each peak must NOT contribute to the peak count. + + Still true, but for a different reason than originally recorded: the + block is uniformly **big-endian**, so T_peak is uint16 BE at [5:7] + (= 3 here) and the 0xd2 at [7] is the HIGH BYTE of the big-endian + half-period at [7:9], not an "annotation" field. Reading the peak as + uint16 LE at [6:8] gave 53763 → 268 in/s, which is what this test was + written to prevent. + """ body = _K558_INTERVAL_12_BLOCK records = decode_histogram_body_full(body) assert records is not None assert len(records) == 1 r = records[0] - assert r["t_peak"] == 3, f"T_peak should be 3 (uint8), got {r['t_peak']}" + assert r["t_peak"] == 3, f"T_peak should be 3, got {r['t_peak']}" assert r["v_peak"] == 2 assert r["l_peak"] == 2 assert r["m_peak"] == 16 - # Half-periods unchanged — still uint16 LE. - assert r["t_halfp"] == 0x0045 # 69 → 7.4 Hz + # Half-period is uint16 BE — 0xd245 = 53829 samples → 0.0095 Hz, which + # is exactly the sub-Hz drift BW rendered as "<1.0" for this interval. + assert r["t_halfp"] == 0xd245 + assert half_period_to_hz(r["t_halfp"]) < 1.0 assert r["m_halfp"] == 6 # → 85.3 Hz - # Annotation byte is preserved (for future RE) but does not affect peak. - assert r["annotations"] == (0xd2, 0x00, 0x00, 0x00) def test_extension_byte_decoded_to_correct_in_s(): @@ -383,3 +389,85 @@ def test_extension_byte_decoded_to_correct_in_s(): assert channels["Vert"] == [2] assert channels["Long"] == [2] assert channels["MicL"] == [16] + + +# ── Big-endian block layout + terminal block (2026-08-25) ─────────────────── +# +# Verified against 1211 production histograms paired with their Blastware +# ASCII exports: 1211/1211 decode exactly (interval count + every +# per-interval peak), and 842,442 per-interval frequency comparisons match +# with zero mismatches. + +def _mk_block(t_peak=3, t_halfp=69, v_peak=2, v_halfp=69, l_peak=2, l_halfp=69, + m_peak=16, m_halfp=6, seg=0, ctr=256, tail=b"\x1e\x0a\x00\x00", + b22=0x00): + """Build one synthetic 32-byte histogram block, big-endian throughout.""" + b = bytearray(32) + b[0] = 0x00 + b[1] = seg + b[2], b[3] = ctr & 0xFF, (ctr >> 8) & 0xFF # block_ctr is uint16 LE + b[4] = 0x0A # marker, uint8 + for off, val in ((5, t_peak), (7, t_halfp), (9, v_peak), (11, v_halfp), + (13, l_peak), (15, l_halfp), (17, m_peak), (19, m_halfp)): + b[off], b[off + 1] = (val >> 8) & 0xFF, val & 0xFF # uint16 BE + b[22] = b22 + b[28:32] = tail + return bytes(b) + + +def test_geo_peak_is_uint16_be_not_uint8(): + """A peak above the uint8 ceiling (255 counts = 1.275 in/s) must decode. + + Real example: BE18193/T193LQ9K.OE0H's final interval reads 8.270 in/s + in the BW export = 1654 counts = 0x0676, which needs both bytes. + Reading only byte[6] gave 0x76 = 118 = 0.590 in/s. + """ + r = decode_histogram_body_full(_mk_block(t_peak=1654)) + assert r is not None and len(r) == 1 + assert r[0]["t_peak"] == 1654 + assert geo_count_to_ins(r[0]["t_peak"]) == pytest.approx(8.27) + + +def test_half_period_is_uint16_be(): + """Half-period spans two bytes big-endian; a large value is sub-Hz.""" + r = decode_histogram_body_full(_mk_block(t_halfp=53829)) + assert r[0]["t_halfp"] == 53829 + assert half_period_to_hz(53829) == pytest.approx(512 / 53829) + + +def test_terminal_block_tail_is_accepted(): + """The LAST block of a histogram stream carries tail `9c 06 00 42` + instead of `1e 0a 00 00`. Rejecting it dropped the final interval — + which is where the event peak often lives. Observed in 1206 of 1211 + production histograms, always after every standard-tail block.""" + body = _mk_block(ctr=256) + _mk_block(ctr=257, t_peak=1654, + tail=b"\x9c\x06\x00\x42") + ch = decode_histogram_body(body) + assert ch is not None + assert ch["Tran"] == [3, 1654], "terminal block must not be dropped" + + +def test_terminal_block_exempt_from_byte22_constraint(): + """Terminal blocks carry arbitrary bytes at [22:24]; only standard + blocks hold 0x00 there. Requiring it dropped the final interval on + files such as BE18438/T438LO30.DC0H.""" + body = _mk_block(ctr=256) + _mk_block(ctr=257, tail=b"\x9c\x06\x00\x42", + b22=0x01) + ch = decode_histogram_body(body) + assert ch is not None and len(ch["Tran"]) == 2 + + +def test_standard_block_still_requires_byte22_zero(): + """The [22] == 0x00 constraint is what keeps trailer content out, so it + must still apply to standard-tail blocks.""" + ch = decode_histogram_body(_mk_block(b22=0x01)) + assert ch is None + + +def test_marker_is_single_byte_not_uint16(): + """block[4] alone is the 0x0A marker. Treating [4:6] as a uint16 LE + marker forced block[5] to zero, which capped every geo peak at 255 + counts — block[5] is the peak's high byte.""" + r = decode_histogram_body_full(_mk_block(t_peak=0x0676)) + assert r is not None, "block[5] != 0 must not disqualify the block" + assert r[0]["t_peak"] == 0x0676