fix(histogram): block is big-endian + terminal block tail — 1/1196 to 1211/1211
Two errors in the series-3 histogram block model, both found by diffing against the per-interval data table in the preserved Blastware ASCII exports (1211 files in the prod snapshot — far stronger ground truth than the header PPV used previously). 1. The block is uniformly BIG-ENDIAN. Peaks and half-periods are uint16 BE (T_peak [5:7], T_halfperiod [7:9], V_peak [9:11], V_halfperiod [11:13], L_peak [13:15], L_halfperiod [15:17], M_peak [17:19], M_halfperiod [19:21]); only block_ctr [2:4] is little-endian. The old uint8-peak model silently CLIPPED any peak above 1.275 in/s: the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590. The byte documented as a per-channel "annotation" was never an annotation — it is the half-period's high byte, which is exactly why it was non-zero on the sub-Hz intervals BW renders as "<1.0". The marker is block[4] alone. Testing [4:6] as a uint16 LE marker forced block[5] == 0, which is what capped the peak at one byte. 2. The final block of each stream carries tail 9c 06 00 42 instead of 1e 0a 00 00, and holds arbitrary bytes at [21:23]. Rejecting it dropped the last interval of nearly every histogram — frequently the interval holding the event peak, so the file's PPV read low. Verified end to end through the production path: 1211/1211 histograms decode exactly (interval count + every per-interval peak), plus 842,442 per-interval frequency comparisons with zero mismatches. Previously 1 of 1196 files was fully correct. decode_histogram_body_full records expose `is_terminal` in place of the removed `annotations` tuple. +6 tests. No regressions: full-suite failure list unchanged from baseline. NOTE: stored histogram .h5 files need regenerating to pick this up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
Claude Opus 5
@@ -8,6 +8,36 @@ All notable changes to seismo-relay are documented here.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Series-3 histogram block is uniformly big-endian, and the stream's final
|
||||
block has its own tail — the codec was clipping large peaks and dropping the
|
||||
last interval of nearly every histogram.**
|
||||
|
||||
- **Peaks and half-periods are `uint16` big-endian**, not `uint8` plus an
|
||||
"annotation" byte: `T_peak` `[5:7]`, `T_halfperiod` `[7:9]`, `V_peak`
|
||||
`[9:11]`, and so on. Only `block_ctr` `[2:4]` is little-endian. The old
|
||||
model silently **clipped any peak above 1.275 in/s** — the final interval
|
||||
of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in Blastware's own export and
|
||||
decoded as 0.590. The "annotation" byte was the half-period's high byte,
|
||||
which is why it was non-zero exactly on the sub-Hz intervals BW renders
|
||||
as `<1.0`.
|
||||
- **The marker is `block[4]` alone.** Testing `[4:6]` as a `uint16 LE`
|
||||
marker forced `block[5] == 0` — which is what capped the peak at one byte
|
||||
in the first place.
|
||||
- **The last block of each stream carries tail `9c 06 00 42`** instead of
|
||||
`1e 0a 00 00`, with arbitrary bytes at `[21:23]`. Rejecting it dropped
|
||||
the final interval of nearly every histogram, and that interval is
|
||||
frequently the one holding the event peak — so the file's reported PPV
|
||||
came out low.
|
||||
|
||||
Verified against **1211 production histograms** paired with their preserved
|
||||
Blastware ASCII exports, which carry a full per-interval data table:
|
||||
**1211/1211 now decode exactly** (interval count plus every per-interval
|
||||
peak), and 842,442 per-interval frequency comparisons match with zero
|
||||
mismatches. Before this fix: **1 of 1196**.
|
||||
|
||||
`decode_histogram_body_full` records now expose `is_terminal` in place of
|
||||
the removed `annotations` tuple.
|
||||
|
||||
- **Geophone full scale is 32000 ADC counts, not 32768 — every geo reading was
|
||||
2.3% low.** The verified body codec emits geo samples in 16-count units whose
|
||||
documented LSB is exactly 0.005 in/s, and `decoded_to_adc_counts` multiplies
|
||||
|
||||
@@ -261,6 +261,32 @@ rotation and corrupts every channel after it.
|
||||
Corpus result, end to end through the production path:
|
||||
**exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.**
|
||||
|
||||
### Histogram codec — corrected 2026-08-25
|
||||
|
||||
The histogram block is **uniformly big-endian**, and the stream's final
|
||||
block has its own tail signature. Two long-standing errors:
|
||||
|
||||
- **Peaks and half-periods are `uint16` big-endian**, not `uint8` +
|
||||
an "annotation" byte. `T_peak` is `[5:7]`, `T_halfperiod` `[7:9]`,
|
||||
`V_peak` `[9:11]`, and so on; only `block_ctr` at `[2:4]` is LE.
|
||||
The old model silently **clipped any peak above 1.275 in/s** — the
|
||||
final interval of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in BW's
|
||||
export and decoded as 0.590. The "annotation" byte was the
|
||||
half-period's high byte, which is why it was non-zero exactly on the
|
||||
sub-Hz intervals BW renders as `<1.0`.
|
||||
- **The marker is `block[4]` alone.** Testing `[4:6]` as a uint16 LE
|
||||
marker forced `block[5] == 0`, which is what capped the peak at one
|
||||
byte in the first place.
|
||||
- **The last block of the stream carries tail `9c 06 00 42`** instead of
|
||||
`1e 0a 00 00`, with arbitrary bytes at `[21:23]`. Rejecting it
|
||||
dropped the final interval of nearly every histogram — frequently the
|
||||
one holding the event peak, so the file's PPV read low.
|
||||
|
||||
Verified against 1211 production histograms paired with their BW ASCII
|
||||
exports: **1211/1211 decode exactly** (interval count plus every
|
||||
per-interval peak), and 842,442 per-interval frequency comparisons match
|
||||
with zero mismatches. Before: 1 of 1196.
|
||||
|
||||
### What's NOT solved
|
||||
|
||||
- **MicL channel conversion to dB(L)** — the codec emits MicL as
|
||||
@@ -276,19 +302,8 @@ Corpus result, end to end through the production path:
|
||||
anchor count — `01 00` records appear with both 2- and 4-byte anchor
|
||||
fields in the same file. Examples: `BE12599/N599LPNB.JF0W` @1155,
|
||||
`BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
|
||||
- **Histogram codec drops trailing intervals (series-3)** — the
|
||||
per-interval *values* are right: measured against the histogram ASCII
|
||||
data table (Tran/Vert/Long peak + freq + PVS per interval), 1031 of
|
||||
1196 files (86%) match within 1 LSB across the overlapping prefix.
|
||||
The interval **count** is short in 1195 of 1196 files — median 1
|
||||
missing; 1088 short by 1–2, 65 by 3–10, 39 by 11–100, 3 by >100 (max
|
||||
205). When a dropped interval holds the event peak, the decoded max
|
||||
falls below the device-reported PPV: 169/1196 files (14%). A
|
||||
termination bug in `histogram_codec.decode_histogram_body` — same
|
||||
family as the waveform-walker truncation fixed 2026-08-25, and
|
||||
untouched by it. Note: the histogram ASCII carries a full
|
||||
per-interval table, so this has proper ground truth available (1211
|
||||
files in the prod snapshot).
|
||||
(The series-3 histogram codec was fixed 2026-08-25 — see below.)
|
||||
|
||||
- **Micromate (UM-series) IDF decode is ~1000× low** — e.g.
|
||||
`UM11402_20260406130113.IDFW` gives a Tran peak of 0.0009 in/s against
|
||||
a device-reported 1.1168. The Thor IDF path decodes sanely, so this
|
||||
|
||||
@@ -1,3 +1,30 @@
|
||||
> ## SUPERSEDED 2026-08-25 — the block is uniformly BIG-ENDIAN
|
||||
>
|
||||
> The `uint8` peak / `annotation` byte model described below is wrong,
|
||||
> though it decoded quiet data correctly. The real layout:
|
||||
>
|
||||
> - **Every per-channel field is `uint16` big-endian.** `T_peak` is
|
||||
> `[5:7]`, `T_halfperiod` `[7:9]`, `V_peak` `[9:11]`, and so on.
|
||||
> Only `block_ctr` at `[2:4]` is little-endian.
|
||||
> - The **marker is `block[4]` alone**, not a `uint16 LE` at `[4:6]`.
|
||||
> Testing `[4:6] == 10` forced `block[5] == 0`, which is exactly what
|
||||
> capped every geo peak at one byte (255 counts = 1.275 in/s).
|
||||
> - The **"annotation" byte was never an annotation** — it is the high
|
||||
> byte of the big-endian half-period. That is why it was non-zero
|
||||
> precisely on the sub-Hz intervals Blastware renders as `<1.0`.
|
||||
> - The **final block of the stream carries tail `9c 06 00 42`** instead
|
||||
> of `1e 0a 00 00`, and arbitrary bytes at `[21:23]`. Rejecting it
|
||||
> dropped the last interval of nearly every histogram — often the one
|
||||
> holding the event peak, so the file's PPV read low.
|
||||
>
|
||||
> Verified against 1211 production histograms paired with their Blastware
|
||||
> ASCII exports: **1211/1211 decode exactly** (interval count plus every
|
||||
> per-interval peak), and 842,442 per-interval frequency comparisons match
|
||||
> with zero mismatches. The uint8 model scored 1204/1211 — the seven
|
||||
> failures are exactly the files containing a peak above 1.275 in/s.
|
||||
>
|
||||
> The section below is retained as the reasoning trail.
|
||||
|
||||
# Histogram body codec — FULLY DECODED (2026-05-20)
|
||||
|
||||
Clean working status doc for the MiniMate Plus histogram-mode event
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
|
||||
| Date | Section | Change |
|
||||
|---|---|---|
|
||||
| 2026-08-25 (2) | S7.6.2, S15 | **HISTOGRAM BLOCK IS BIG-ENDIAN + terminal block tail.** The 32-byte histogram block's per-channel fields are uint16 **big-endian** (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], `V_halfperiod` [11:13], `L_peak` [13:15], `L_halfperiod` [15:17], `M_peak` [17:19], `M_halfperiod` [19:21]); only `block_ctr` [2:4] is little-endian. The marker is `block[4]` alone - the previous uint16 LE marker test at [4:6] forced `block[5] == 0` and thereby capped every geo peak at 255 counts (1.275 in/s), silently clipping larger peaks. The byte previously documented as a per-channel "annotation" is the high byte of the big-endian half-period, which is why it was non-zero exactly on sub-Hz intervals. Separately, the **final block of each stream carries tail `9c 06 00 42`** rather than `1e 0a 00 00` and holds arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram. Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly, plus 842,442 per-interval frequency comparisons with zero mismatches (previously 1 of 1196 files fully correct). |
|
||||
| 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. |
|
||||
| 2026-05-20 | §2, §3, §4.2, §5.1, §5.3, §6, §7.5b, §7.6.1, §7.6.3, §7.6.4, §7.7.2, §7.7.3, §7.7.5, §7.8.4, §7.8.7, §7.9, §8, §11, §12, §13, §14, §15, Appendix D | **DOC AUDIT PASS — accuracy sweep against `CLAUDE.md` + `minimateplus/` code.** Fixed: (1) S3 frames terminate on bare ETX, not DLE+ETX — §2/§3 rewritten. (2) §3 payload layout corrected — byte[1]=flags, byte[2]=SUB (was wrongly labelled DLE/ADDR). (3) §4.2 — probe responses do NOT carry data length; lengths are hardcoded `DATA_LENGTHS` constants. (4) §5.1 — removed stale duplicate "SUB 1C = TRIGGER CONFIG READ" row; SUB 0A lengths corrected from `0x30/0x26` to `0x46/0x2C` (real event / boundary marker). (5) §5.3 — added missing write-frame format (BW_CMD-only doubling, DLE-aware checksum, offset formula, ack format, SUB 71 chunk parameters). (6) §6 — fixed "SUB 06 → channel config read" → event storage range. (7) §7.5b / §8 — added the 10-byte `sub_code=0x03` continuous-mode timestamp variant alongside the 9-byte single-shot layout; peak vector sum location corrected from "fixed offset 87" to `tran_pos − 12` (label-relative). (8) §7.6 / §7.6.1 / §7.6.3 / §7.6.4 — switched compliance-anchor convention from the 10-byte form to the canonical 6-byte `\xbe\x80\x00\x00\x00\x00`; recording_mode confirmed at anchor−8 in BOTH read and write (was wrongly listed as anchor−3 write / anchor−4 read); sample_rate at anchor−6, histogram_interval at anchor−4, record_time at anchor+6; geo_range row added at channel_label+33. (9) §7.7.2 — token byte position corrected from `params[6]` to `params[7]`. (10) §7.8.4 — fi==9 skip marked FIXED (already removed from code); chunk-count totals updated. (11) §7.8.7 — TODO replaced with current state of `_decode_a5_metadata_into`. (12) §7.9 — Histogram Interval upgraded ❓ → ✅. (13) §11 — POLL example wire bytes corrected; SUB 5A row added to checksum table. (14) §13 — device-under-test updated for current primary unit (BE11529 / S338.17). (15) §14 — TCP Idle Timeout fixed (0→2 min); Data Forwarding Timeout units clarified. (16) §15 (renumbered from second §14) — open-question items already resolved in CLAUDE.md closed out. (17) Appendix D — extension taxonomy rewritten to reflect the AB0T timestamp encoding (D.5.2/D.5.3); EXTENSION REFUTED warning replaced with the resolved encoding. |
|
||||
| 2026-05-08 | §7.6.1 (RETRACTION) | **❌ RETRACTED — "raw int16 LE 8 bytes/sample-set" body codec was never validated.** The original 4-2-26 confirmation was based on misreading broken-decoder output (full-scale ±32K noise) as evidence the signal had saturated. BW's own 0C peaks for that capture (Tran=0.420 / Vert=3.870 / Long=0.495 in/s) prove the signal was NOT saturated — none of those exceed 13K ADC counts. No event in the project's archive has ever come close to saturation, yet the decoder consistently produces ±32K noise on every event. Conclusion: the body codec is not raw int16 LE; the actual encoding is open. Body byte distribution is heavily skewed (24% `0x00`, 10.5% `0x10`, lots of `10 XX` pairs) — likely a delta encoding with `0x10` as escape, but unverified. Retraction box added at top of §7.6.1; "fully-saturating event" claim removed from channel-identification note. The histogram codec in §7.6.2 IS verified and decoded correctly (different recording mode, 32-byte blocks); use it as a structural hint when reverse-engineering the waveform codec. |
|
||||
@@ -3037,7 +3038,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
|
||||
| **Sensor Check dropdown byte location** — byte offset in 1A compliance config payload for the "Sensor Check: Before monitoring / After each event / Disabled" setting is NOT YET LOCATED. Confirmed: unit always runs with "Before monitoring" set. Need a capture with "Disabled" to diff. | MEDIUM | 2026-04-08 | Still open |
|
||||
| **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open |
|
||||
| **Variable-prefix segment descriptors** — 3 of 75 ground-truth events still truncate. The walk reaches a segment header whose channel-id field is preceded by a *variable-width* prefix (2, 4 or 6 bytes observed; the standard tagless form always has 4), carrying an `01 00` marker instead of `02 00`. The marker is **not** simply an anchor count — `01 00` records appear with both 2- and 4-byte anchor fields in the same file, so prefix width and marker are not yet reconciled. Examples: `BE12599/N599LPNB.JF0W` @1155, `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485. | MEDIUM | 2026-08-25 | Still open |
|
||||
| **Histogram codec drops trailing intervals (series-3)** — measured per-interval against the histogram ASCII data table (which carries Tran/Vert/Long peak + freq + PVS per interval), n=1196: the decoded per-interval **values are right** — 1031/1196 (86%) match the ASCII table within 1 LSB across the overlapping prefix — but the interval **count** is short in 1195 of 1196 files. Median 1 interval missing; 1088 files short by 1–2, 65 by 3–10, 39 by 11–100, 3 by more than 100 (max 205). When a dropped interval is the one holding the event peak, the decoded max falls below the device-reported PPV — 169/1196 (14%) of files. This is a termination bug in `histogram_codec.decode_histogram_body`, the same family as the waveform-walker truncation fixed 2026-08-25, and it was untouched by that pass. | MEDIUM | 2026-08-25 | Still open |
|
||||
| ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
|
||||
| **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open |
|
||||
| **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open |
|
||||
|
||||
|
||||
@@ -25,38 +25,60 @@ iterate 32-stride and stop before the tail.
|
||||
────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
[0] 0x00 always-zero tag
|
||||
[1] segment_id (uint8) 0x00..0x03 — 256 blocks per segment
|
||||
[2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, …)
|
||||
[4:6] 0x000a (uint16 LE) constant marker (= 10)
|
||||
[6] T_peak_count uint8 Tran peak (count × 0.005 → in/s, max 1.275 in/s)
|
||||
[7] T_annotation uint8 empirically non-zero on intervals with sub-Hz
|
||||
or unmeasurable Tran freq; meaning not fully RE'd
|
||||
[8:10] T_halfperiod uint16 LE Tran half-period in samples (freq = 512 / halfp Hz)
|
||||
[10] V_peak_count uint8
|
||||
[11] V_annotation uint8
|
||||
[12:14] V_halfperiod uint16 LE
|
||||
[14] L_peak_count uint8
|
||||
[15] L_annotation uint8
|
||||
[16:18] L_halfperiod uint16 LE
|
||||
[18] M_peak_count uint8 MicL peak (count → dB via mic_count_to_db)
|
||||
[19] M_annotation uint8
|
||||
[20:22] M_halfperiod uint16 LE MicL half-period in samples (freq = 512 / halfp Hz)
|
||||
[22:24] 0x00 0x00 constant
|
||||
[1] segment_id (uint8) 0x00..0x03 - 256 blocks per segment
|
||||
[2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, ...)
|
||||
[4] 0x0a (uint8) constant marker (= 10)
|
||||
[5:7] T_peak_count uint16 BE Tran peak (count x 0.005 -> in/s)
|
||||
[7:9] T_halfperiod uint16 BE Tran half-period in samples (freq = 512 / halfp)
|
||||
[9:11] V_peak_count uint16 BE
|
||||
[11:13] V_halfperiod uint16 BE
|
||||
[13:15] L_peak_count uint16 BE
|
||||
[15:17] L_halfperiod uint16 BE
|
||||
[17:19] M_peak_count uint16 BE MicL peak (count -> dB via mic_count_to_db)
|
||||
[19:21] M_halfperiod uint16 BE MicL half-period in samples
|
||||
[21:23] 0x00 0x00 constant on standard blocks
|
||||
[24:28] 4-byte variable purpose unknown (possibly CRC or timestamp delta)
|
||||
[28:32] 0x1e 0x0a 0x00 0x00 constant block-end signature
|
||||
[28:32] block-end signature see "Two block tails" below
|
||||
|
||||
NOTE on peak-count width: an earlier interpretation treated the peak
|
||||
fields as uint16 LE spanning [6:8] / [10:12] / [14:16] / [18:20].
|
||||
That happened to be byte-exact against the N844 fixture corpus only
|
||||
because every annotation byte in those fixtures was zero, making
|
||||
``uint16 LE == uint8``. Cross-correlating BE9558 (K558) Tran-drift
|
||||
and BE18003 (T003) Histogram+Continuous events against the BW ASCII
|
||||
export proved peak is uint8 alone — see test_histogram_codec.py
|
||||
and docs/histogram_codec_re_status.md.
|
||||
**Every per-channel field is uint16 BIG-endian** (confirmed 2026-08-25).
|
||||
Only ``block_ctr`` at [2:4] is little-endian.
|
||||
|
||||
Block-identification anchor: ``block[22:24] == b"\\x00\\x00"`` AND
|
||||
``block[28:32] == b"\\x1e\\x0a\\x00\\x00"``. This is the reliable
|
||||
distinguisher from non-block content in the file.
|
||||
HISTORY - two earlier readings of this block were wrong in ways that
|
||||
cancelled out on quiet data:
|
||||
|
||||
1. *peak as uint16 LE at [6:8]* - produced 268 in/s peaks on any
|
||||
interval whose next byte was non-zero.
|
||||
2. *peak as uint8 at [6] with an "annotation" byte at [7]* - correct
|
||||
for every peak below 256 counts (1.275 in/s), but it silently
|
||||
**clipped larger peaks**: the final interval of
|
||||
BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export
|
||||
(1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590 in/s.
|
||||
The "annotation" byte was never an annotation - it is the high
|
||||
byte of the big-endian half-period, which is why it was non-zero
|
||||
exactly on the sub-Hz intervals BW renders as "<1.0".
|
||||
|
||||
Both readings also forced ``block[5] == 0`` via a bogus ``uint16 LE``
|
||||
marker check at [4:6], which is what capped the peak at one byte.
|
||||
The marker is ``block[4]`` alone.
|
||||
|
||||
Verified 2026-08-25 against 1211 production histograms paired with
|
||||
their Blastware ASCII exports: **1211/1211 decode exactly** (interval
|
||||
count plus every per-interval peak), and 842,442 per-interval
|
||||
frequency comparisons match with **zero** mismatches.
|
||||
|
||||
Two block tails
|
||||
---------------
|
||||
Standard blocks end with ``1e 0a 00 00``. The **final block of the
|
||||
stream** ends with ``9c 06 00 42`` instead, and carries arbitrary bytes
|
||||
at [21:23]. Rejecting it dropped the last interval of nearly every
|
||||
histogram - and the last interval is frequently the one holding the
|
||||
event peak, so the file's reported PPV came out low. Observed in 1206
|
||||
of 1211 production histograms, always positioned after every
|
||||
standard-tail block.
|
||||
|
||||
Block-identification anchor: ``block[0] == 0x00`` AND
|
||||
``block[4] == 0x0A`` AND the tail is one of the two signatures above;
|
||||
standard-tail blocks additionally require ``block[22] == 0x00``.
|
||||
|
||||
────────────────────────────────────────────────────────────────────────────
|
||||
Per-channel encoding
|
||||
@@ -109,6 +131,12 @@ from typing import List, Optional, Tuple
|
||||
# real data block. More distinctive than the byte-22 `00 00` (which
|
||||
# matches many false positives), so we anchor on this.
|
||||
_BLOCK_TAIL = b"\x1e\x0a\x00\x00"
|
||||
|
||||
# The final block of a histogram stream ends with this instead. It is a
|
||||
# real data block - same layout - and holds the last interval. See the
|
||||
# module docstring, "Two block tails".
|
||||
_BLOCK_TAIL_TERMINAL = b"\x9c\x06\x00\x42"
|
||||
|
||||
_BLOCK_SIZE = 32
|
||||
|
||||
# Marker byte at block[4:6] of every histogram data block. Used as
|
||||
@@ -127,19 +155,25 @@ _FREQ_NUMERATOR = 512
|
||||
|
||||
|
||||
def _is_data_block(block: bytes) -> bool:
|
||||
"""Tight identification of a histogram data block."""
|
||||
"""Tight identification of a histogram data block.
|
||||
|
||||
Accepts both tail signatures. ``block[4]`` alone is the marker -
|
||||
``block[5]`` is the high byte of the Tran peak and is non-zero on any
|
||||
interval above 1.275 in/s, so it must not be part of the marker test.
|
||||
The ``block[22] == 0`` constraint is what keeps trailer content out,
|
||||
but it applies only to standard-tail blocks: terminal blocks carry
|
||||
arbitrary bytes there.
|
||||
"""
|
||||
if len(block) < _BLOCK_SIZE:
|
||||
return False
|
||||
if block[28:32] != _BLOCK_TAIL:
|
||||
return False
|
||||
if block[22:24] != b"\x00\x00":
|
||||
return False
|
||||
if block[0] != 0x00:
|
||||
return False
|
||||
marker = block[4] | (block[5] << 8)
|
||||
if marker != _BLOCK_MARKER:
|
||||
if block[4] != _BLOCK_MARKER:
|
||||
return False
|
||||
return True
|
||||
tail = block[28:32]
|
||||
if tail == _BLOCK_TAIL:
|
||||
return block[22] == 0x00
|
||||
return tail == _BLOCK_TAIL_TERMINAL
|
||||
|
||||
|
||||
def _decode_block(block: bytes) -> Optional[dict]:
|
||||
@@ -149,33 +183,23 @@ def _decode_block(block: bytes) -> Optional[dict]:
|
||||
Returns a record with per-channel peak counts (uint8) and
|
||||
half-periods (uint16 LE).
|
||||
"""
|
||||
# Peak counts are uint8 at bytes [6] / [10] / [14] / [18]. The
|
||||
# adjacent bytes [7] / [11] / [15] / [19] hold an annotation field
|
||||
# whose meaning isn't fully understood (empirically non-zero in
|
||||
# intervals with sub-Hz or unmeasurable geo frequencies, mostly
|
||||
# zero otherwise — see test fixtures from BE9558/BE18003 corpora).
|
||||
# Crucially, those annotation bytes are NOT the high byte of the
|
||||
# peak count: cross-correlating against BW's per-interval ASCII
|
||||
# export proves the peak is uint8 alone.
|
||||
#
|
||||
# Reading the peak as uint16 LE (the original interpretation) was
|
||||
# accidentally correct only because every block in the N844 fixture
|
||||
# corpus had a zero annotation byte; non-N844 events with non-zero
|
||||
# annotation bytes decoded to physically impossible peaks (e.g.
|
||||
# 268 in/s per channel) and produced 35× inflated PVS sums when
|
||||
# first run against prod data. See histogram_codec_re_status.md.
|
||||
t_peak = block[6]
|
||||
v_peak = block[10]
|
||||
l_peak = block[14]
|
||||
m_peak = block[18]
|
||||
t_halfp = block[8] | (block[9] << 8)
|
||||
v_halfp = block[12] | (block[13] << 8)
|
||||
l_halfp = block[16] | (block[17] << 8)
|
||||
m_halfp = block[20] | (block[21] << 8)
|
||||
# Every per-channel field is uint16 BIG-endian; only block_ctr is LE.
|
||||
# See the module docstring for the two superseded readings and why
|
||||
# each looked correct on quiet data.
|
||||
def _be16(i: int) -> int:
|
||||
return (block[i] << 8) | block[i + 1]
|
||||
|
||||
t_peak = _be16(5)
|
||||
t_halfp = _be16(7)
|
||||
v_peak = _be16(9)
|
||||
v_halfp = _be16(11)
|
||||
l_peak = _be16(13)
|
||||
l_halfp = _be16(15)
|
||||
m_peak = _be16(17)
|
||||
m_halfp = _be16(19)
|
||||
segment_id = block[1]
|
||||
block_ctr = block[2] | (block[3] << 8)
|
||||
var_meta = bytes(block[24:28])
|
||||
annotations = (block[7], block[11], block[15], block[19])
|
||||
return {
|
||||
"segment_id": segment_id,
|
||||
"block_ctr": block_ctr,
|
||||
@@ -188,7 +212,7 @@ def _decode_block(block: bytes) -> Optional[dict]:
|
||||
"m_peak": m_peak,
|
||||
"m_halfp": m_halfp,
|
||||
"meta_var": var_meta,
|
||||
"annotations": annotations,
|
||||
"is_terminal": block[28:32] == _BLOCK_TAIL_TERMINAL,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -354,23 +354,29 @@ _K558_INTERVAL_12_BLOCK = bytes.fromhex(
|
||||
|
||||
|
||||
def test_extension_byte_does_not_inflate_peak():
|
||||
"""The annotation byte at [7]/[11]/[15]/[19] must NOT contribute to
|
||||
the peak count. Decoded T_peak must be 3 (uint8 byte[6]), NOT
|
||||
53763 (uint16 LE byte[6:8])."""
|
||||
"""The byte after each peak must NOT contribute to the peak count.
|
||||
|
||||
Still true, but for a different reason than originally recorded: the
|
||||
block is uniformly **big-endian**, so T_peak is uint16 BE at [5:7]
|
||||
(= 3 here) and the 0xd2 at [7] is the HIGH BYTE of the big-endian
|
||||
half-period at [7:9], not an "annotation" field. Reading the peak as
|
||||
uint16 LE at [6:8] gave 53763 → 268 in/s, which is what this test was
|
||||
written to prevent.
|
||||
"""
|
||||
body = _K558_INTERVAL_12_BLOCK
|
||||
records = decode_histogram_body_full(body)
|
||||
assert records is not None
|
||||
assert len(records) == 1
|
||||
r = records[0]
|
||||
assert r["t_peak"] == 3, f"T_peak should be 3 (uint8), got {r['t_peak']}"
|
||||
assert r["t_peak"] == 3, f"T_peak should be 3, got {r['t_peak']}"
|
||||
assert r["v_peak"] == 2
|
||||
assert r["l_peak"] == 2
|
||||
assert r["m_peak"] == 16
|
||||
# Half-periods unchanged — still uint16 LE.
|
||||
assert r["t_halfp"] == 0x0045 # 69 → 7.4 Hz
|
||||
# Half-period is uint16 BE — 0xd245 = 53829 samples → 0.0095 Hz, which
|
||||
# is exactly the sub-Hz drift BW rendered as "<1.0" for this interval.
|
||||
assert r["t_halfp"] == 0xd245
|
||||
assert half_period_to_hz(r["t_halfp"]) < 1.0
|
||||
assert r["m_halfp"] == 6 # → 85.3 Hz
|
||||
# Annotation byte is preserved (for future RE) but does not affect peak.
|
||||
assert r["annotations"] == (0xd2, 0x00, 0x00, 0x00)
|
||||
|
||||
|
||||
def test_extension_byte_decoded_to_correct_in_s():
|
||||
@@ -383,3 +389,85 @@ def test_extension_byte_decoded_to_correct_in_s():
|
||||
assert channels["Vert"] == [2]
|
||||
assert channels["Long"] == [2]
|
||||
assert channels["MicL"] == [16]
|
||||
|
||||
|
||||
# ── Big-endian block layout + terminal block (2026-08-25) ───────────────────
|
||||
#
|
||||
# Verified against 1211 production histograms paired with their Blastware
|
||||
# ASCII exports: 1211/1211 decode exactly (interval count + every
|
||||
# per-interval peak), and 842,442 per-interval frequency comparisons match
|
||||
# with zero mismatches.
|
||||
|
||||
def _mk_block(t_peak=3, t_halfp=69, v_peak=2, v_halfp=69, l_peak=2, l_halfp=69,
|
||||
m_peak=16, m_halfp=6, seg=0, ctr=256, tail=b"\x1e\x0a\x00\x00",
|
||||
b22=0x00):
|
||||
"""Build one synthetic 32-byte histogram block, big-endian throughout."""
|
||||
b = bytearray(32)
|
||||
b[0] = 0x00
|
||||
b[1] = seg
|
||||
b[2], b[3] = ctr & 0xFF, (ctr >> 8) & 0xFF # block_ctr is uint16 LE
|
||||
b[4] = 0x0A # marker, uint8
|
||||
for off, val in ((5, t_peak), (7, t_halfp), (9, v_peak), (11, v_halfp),
|
||||
(13, l_peak), (15, l_halfp), (17, m_peak), (19, m_halfp)):
|
||||
b[off], b[off + 1] = (val >> 8) & 0xFF, val & 0xFF # uint16 BE
|
||||
b[22] = b22
|
||||
b[28:32] = tail
|
||||
return bytes(b)
|
||||
|
||||
|
||||
def test_geo_peak_is_uint16_be_not_uint8():
|
||||
"""A peak above the uint8 ceiling (255 counts = 1.275 in/s) must decode.
|
||||
|
||||
Real example: BE18193/T193LQ9K.OE0H's final interval reads 8.270 in/s
|
||||
in the BW export = 1654 counts = 0x0676, which needs both bytes.
|
||||
Reading only byte[6] gave 0x76 = 118 = 0.590 in/s.
|
||||
"""
|
||||
r = decode_histogram_body_full(_mk_block(t_peak=1654))
|
||||
assert r is not None and len(r) == 1
|
||||
assert r[0]["t_peak"] == 1654
|
||||
assert geo_count_to_ins(r[0]["t_peak"]) == pytest.approx(8.27)
|
||||
|
||||
|
||||
def test_half_period_is_uint16_be():
|
||||
"""Half-period spans two bytes big-endian; a large value is sub-Hz."""
|
||||
r = decode_histogram_body_full(_mk_block(t_halfp=53829))
|
||||
assert r[0]["t_halfp"] == 53829
|
||||
assert half_period_to_hz(53829) == pytest.approx(512 / 53829)
|
||||
|
||||
|
||||
def test_terminal_block_tail_is_accepted():
|
||||
"""The LAST block of a histogram stream carries tail `9c 06 00 42`
|
||||
instead of `1e 0a 00 00`. Rejecting it dropped the final interval —
|
||||
which is where the event peak often lives. Observed in 1206 of 1211
|
||||
production histograms, always after every standard-tail block."""
|
||||
body = _mk_block(ctr=256) + _mk_block(ctr=257, t_peak=1654,
|
||||
tail=b"\x9c\x06\x00\x42")
|
||||
ch = decode_histogram_body(body)
|
||||
assert ch is not None
|
||||
assert ch["Tran"] == [3, 1654], "terminal block must not be dropped"
|
||||
|
||||
|
||||
def test_terminal_block_exempt_from_byte22_constraint():
|
||||
"""Terminal blocks carry arbitrary bytes at [22:24]; only standard
|
||||
blocks hold 0x00 there. Requiring it dropped the final interval on
|
||||
files such as BE18438/T438LO30.DC0H."""
|
||||
body = _mk_block(ctr=256) + _mk_block(ctr=257, tail=b"\x9c\x06\x00\x42",
|
||||
b22=0x01)
|
||||
ch = decode_histogram_body(body)
|
||||
assert ch is not None and len(ch["Tran"]) == 2
|
||||
|
||||
|
||||
def test_standard_block_still_requires_byte22_zero():
|
||||
"""The [22] == 0x00 constraint is what keeps trailer content out, so it
|
||||
must still apply to standard-tail blocks."""
|
||||
ch = decode_histogram_body(_mk_block(b22=0x01))
|
||||
assert ch is None
|
||||
|
||||
|
||||
def test_marker_is_single_byte_not_uint16():
|
||||
"""block[4] alone is the 0x0A marker. Treating [4:6] as a uint16 LE
|
||||
marker forced block[5] to zero, which capped every geo peak at 255
|
||||
counts — block[5] is the peak's high byte."""
|
||||
r = decode_histogram_body_full(_mk_block(t_peak=0x0676))
|
||||
assert r is not None, "block[5] != 0 must not disqualify the block"
|
||||
assert r[0]["t_peak"] == 0x0676
|
||||
|
||||
Reference in New Issue
Block a user