diff --git a/CHANGELOG.md b/CHANGELOG.md index 07dc116..94bc028 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,62 @@ All notable changes to seismo-relay are documented here. ### Fixed +- **The series-3 waveform body is a RECORD CHAIN, not a tag stream — this + supersedes the segment-header model, including the fixes made earlier the + same day.** + + Records are self-delimiting. `off+2` is a `uint16 BE` length and + `next_record = off + 2 + len`; the chain ends on a record whose `chan_id` is + `0x06`. `off+8` carries a 3-valued mode enum: + + | mode | header | data section | + |---|---|---| + | `02 00` | 14 B | anchors, then **cumulative deltas** | + | `01 00` | 10 B | no anchors, **absolute** values | + | `00 03` | 10 B | **no tags at all** — raw 12-bit packed absolute | + + **`40 NN` is an ordinary int16 BE data block** (`2*NN + 2`), never a segment + header. Reading it as a `2*NN + 16` header is what made walks drift — and the + "variable-prefix segment descriptors" reported earlier today were not a format + feature at all, just walker drift of exactly + `4 - (old_stop - true_record_start)` on all 25 affected files. + + Measured against the production snapshot: + + | | before | after | + |---|---|---| + | all four channels equal length | 156 / 1388 | **1388 / 1388** | + | ASCII sample-count exact | 72 / 75 | **75 / 75** | + | ASCII fully exact | 70 / 75 | **73 / 75** | + | device PPV, waveform (live decode) | 1288 / 1306 | **1306 / 1306** | + | device PPV, histogram (live decode) | 4434 / 4459 | **4458 / 4459** | + + Mean absolute PPV ratio error on waveforms is now 0.00000. The 2 remaining + ASCII imperfections differ by exactly 1 LSB on samples sitting at the + ±10.000 in/s rail. + + **This also eliminated the walker-over-read class.** 24 of those 35 files + were histograms that `read_blastware_file` fed to the *waveform* codec first; + the old walker accepted them and returned garbage (one yielded 98,923 + "intervals"), while the record-chain decoder correctly returns `None` so they + fall through to `histogram_codec`. + + `00 03` records are decoded rather than skipped. Skipping them does not merely + lose samples — it silently shifts the time base of everything after them on + that channel (observed on `BE9558/K558LOF2.820W`, MicL displaced by exactly + 512 samples with nothing marking the gap). + + Footer detection now prefers whichever `0e 08` candidate yields a chain + terminating on `0x06`, since the signature can occur inside a sample stream. + Blast radius: 1 file of 1,388. + + The superseded model is retained as `decode_waveform_legacy` and pinned by + `micromate/idf_file.py`, whose Thor IDFW body-offset search trial-decodes + candidate offsets and keeps whichever yields the most samples — the new + decoder correctly returns `None` where the old one returned garbage, which + changes that heuristic's winner. Switching Thor over is deferred until that + search is reworked to use the record chain directly. + - **Series-3 histogram block is uniformly big-endian, and the stream's final block has its own tail — the codec was clipping large peaks and dropping the last interval of nearly every histogram.** diff --git a/CLAUDE.md b/CLAUDE.md index 9af798e..0af777e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -223,6 +223,29 @@ custom delta + RLE + variable-width codec. `NN + 2` for int8 blocks). Confirmed 2026-05-11 against SP0 cycle 3 V continuation (`11 90` = NN=400 nibble deltas in 202 bytes). +### ⚠ SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN + +Everything in this section below about `40 NN` segment headers, tagless +headers, variable header widths and channel rotation describes a model that +is **wrong**. The body is a chain of self-delimiting per-channel records: + + off+2 len uint16 BE -> next_record = off + 2 + len + off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL / 0x06 = END + off+8 mode 02 00 = deltas+anchors (14B hdr) + 01 00 = ABSOLUTE values (10B hdr) + 00 03 = raw 12-bit absolute, NO TAGS (10B hdr) + +`40 NN` is an ordinary int16 BE data block (`2*NN + 2`), never a header. The +"variable prefix" of 0/2/4/6/8 bytes was walker drift, exactly +`4 - (old_stop - true_record_start)`. + +All four channels now come out equal length in **1388/1388** files (was +156/1388); ASCII sample-count exact **75/75**, fully exact **73/75**; device +PPV on a live decode **1306/1306** waveform, **4458/4459** histogram. + +The old model survives as `decode_waveform_legacy` because +`micromate/idf_file.py` pins it for Thor IDFW body-offset search. + ### Framing cases added 2026-05-11 → 2026-08-25 Four more block-framing cases, each of which had been causing **silent diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index 9059658..1242d6a 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -11,6 +11,7 @@ | Date | Section | Change | |---|---|---| +| 2026-08-25 (3) | S7.6.1, S15 | **THE WAVEFORM BODY IS A RECORD CHAIN, NOT A TAG STREAM — supersedes the segment-header model entirely.** Records are self-delimiting: `off+2` is a uint16 BE length and `next = off + 2 + len`; the chain ends on a record whose chan_id is `0x06`. `off+8` holds a 3-valued mode enum - `02 00` (14-byte header, anchors, cumulative deltas), `01 00` (10-byte, no anchors, ABSOLUTE values), `00 03` (10-byte, no tags at all, raw 12-bit absolute). **`40 NN` is an ordinary int16 BE data block of length 2*NN+2**, never a header; reading it as a 2*NN+16 header is what made walks drift, and the 'variable prefix' of 0/2/4/6/8 bytes reported earlier the same day was walker drift, exactly `4 - (old_stop - true_record_start)`. Verified: chain terminates on `06` in 1387/1388 files; all four channels equal length in **1388/1388** (was 156/1388); ASCII sample-count exact 72/75 -> **75/75**, fully exact 70/75 -> **73/75**; device PPV on a live decode **1306/1306** waveform (mean abs ratio error 0.00000) and **4458/4459** histogram. Also eliminated the walker-over-read class: 24 of those files were histograms the waveform codec was wrongly accepting. The superseded model is retained as `decode_waveform_legacy` because `micromate/idf_file.py` pins it for Thor IDFW body-offset search. | | 2026-08-25 (2) | S7.6.2, S15 | **HISTOGRAM BLOCK IS BIG-ENDIAN + terminal block tail.** The 32-byte histogram block's per-channel fields are uint16 **big-endian** (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], `V_halfperiod` [11:13], `L_peak` [13:15], `L_halfperiod` [15:17], `M_peak` [17:19], `M_halfperiod` [19:21]); only `block_ctr` [2:4] is little-endian. The marker is `block[4]` alone - the previous uint16 LE marker test at [4:6] forced `block[5] == 0` and thereby capped every geo peak at 255 counts (1.275 in/s), silently clipping larger peaks. The byte previously documented as a per-channel "annotation" is the high byte of the big-endian half-period, which is why it was non-zero exactly on sub-Hz intervals. Separately, the **final block of each stream carries tail `9c 06 00 42`** rather than `1e 0a 00 00` and holds arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram. Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly, plus 842,442 per-interval frequency comparisons with zero mismatches (previously 1 of 1196 files fully correct). | | 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. | | 2026-05-20 | §2, §3, §4.2, §5.1, §5.3, §6, §7.5b, §7.6.1, §7.6.3, §7.6.4, §7.7.2, §7.7.3, §7.7.5, §7.8.4, §7.8.7, §7.9, §8, §11, §12, §13, §14, §15, Appendix D | **DOC AUDIT PASS — accuracy sweep against `CLAUDE.md` + `minimateplus/` code.** Fixed: (1) S3 frames terminate on bare ETX, not DLE+ETX — §2/§3 rewritten. (2) §3 payload layout corrected — byte[1]=flags, byte[2]=SUB (was wrongly labelled DLE/ADDR). (3) §4.2 — probe responses do NOT carry data length; lengths are hardcoded `DATA_LENGTHS` constants. (4) §5.1 — removed stale duplicate "SUB 1C = TRIGGER CONFIG READ" row; SUB 0A lengths corrected from `0x30/0x26` to `0x46/0x2C` (real event / boundary marker). (5) §5.3 — added missing write-frame format (BW_CMD-only doubling, DLE-aware checksum, offset formula, ack format, SUB 71 chunk parameters). (6) §6 — fixed "SUB 06 → channel config read" → event storage range. (7) §7.5b / §8 — added the 10-byte `sub_code=0x03` continuous-mode timestamp variant alongside the 9-byte single-shot layout; peak vector sum location corrected from "fixed offset 87" to `tran_pos − 12` (label-relative). (8) §7.6 / §7.6.1 / §7.6.3 / §7.6.4 — switched compliance-anchor convention from the 10-byte form to the canonical 6-byte `\xbe\x80\x00\x00\x00\x00`; recording_mode confirmed at anchor−8 in BOTH read and write (was wrongly listed as anchor−3 write / anchor−4 read); sample_rate at anchor−6, histogram_interval at anchor−4, record_time at anchor+6; geo_range row added at channel_label+33. (9) §7.7.2 — token byte position corrected from `params[6]` to `params[7]`. (10) §7.8.4 — fi==9 skip marked FIXED (already removed from code); chunk-count totals updated. (11) §7.8.7 — TODO replaced with current state of `_decode_a5_metadata_into`. (12) §7.9 — Histogram Interval upgraded ❓ → ✅. (13) §11 — POLL example wire bytes corrected; SUB 5A row added to checksum table. (14) §13 — device-under-test updated for current primary unit (BE11529 / S338.17). (15) §14 — TCP Idle Timeout fixed (0→2 min); Data Forwarding Timeout units clarified. (16) §15 (renumbered from second §14) — open-question items already resolved in CLAUDE.md closed out. (17) Appendix D — extension taxonomy rewritten to reflect the AB0T timestamp encoding (D.5.2/D.5.3); EXTENSION REFUTED warning replaced with the resolved encoding. | @@ -1268,36 +1269,79 @@ re-deriving the whole production store: The series-4 figure is closer to correct but not exact — the Thor per-count LSB is its own open question (see §15). -###### Unmapped: variable-prefix segment descriptors ❓ OPEN +###### SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN, not a tag stream -Three of 75 ground-truth production events still truncate. In each, -the walk reaches a segment header whose channel-id field is preceded by -a **variable-width prefix** — 2, 4 or 6 bytes have all been observed, -where the standard tagless form always has 4 (``field2`` + ``len``). -These records also carry the ``01 00`` marker rather than ``02 00``, -and appear packed back-to-back with little or no sample data between -them. +Everything above about ``40 NN`` segment headers, tagless headers, variable +header widths and channel rotation describes a model that is **wrong**. It +produced nearly-correct output because the block table happens to tile the +data sections correctly, but the framing is not what the device writes. -The ``01 00`` marker is *not* simply an anchor count: records carrying -it have been seen with both 2-byte and 4-byte anchor fields in the same -file, so the prefix width and the marker are not yet reconciled. - -The decoder stops cleanly at these rather than emitting garbage. -Examples: ``BE12599/N599LPNB.JF0W`` at body offset 1155 (2-byte -prefix), ``BE12599/N599LPWJ.980W`` at 849 (6-byte prefix), -``BE9558/K558LOF2.820W`` at 1485. - -Examples from event-c (1 sec single-shot): +The body is a chain of **self-delimiting per-channel records**: ``` -Segment header 1 (offset 235): - 40 02 | 00 00 00 00 | 0a 4b 01 1e | 47 00 00 00 | 02 00 00 01 | 00 01 - ^counter=0x47 -Segment header 2 (offset 523): - 40 02 | ff fe ff fe | 13 f5 01 06 | 48 00 00 00 | 02 00 00 01 | 00 02 - ^counter=0x48 (+1) +off+0 field2 uint16 purpose unknown (not a length, not a checksum) +off+2 len uint16 BE next_record = off + 2 + len <- authoritative +off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL + 0x06 = END OF WAVEFORM STREAM +off+5 0x00 +off+6 0x00 +off+7 segment index +off+8 mode 2 bytes, a 3-valued enum +off+10 anchors 2 x int16 BE, ABSOLUTE -- present ONLY when mode == 02 00 ``` +**Mode enum**, all three ground-truth verified: + +| mode | header | data section | +|---|---|---| +| ``02 00`` | 14 bytes | anchors emitted, then blocks are **cumulative deltas** | +| ``01 00`` | 10 bytes | no anchors, blocks carry **absolute** sample values | +| ``00 03`` | 10 bytes | **no tags at all** — raw 12-bit packed absolute samples | + +Census over the 1,388 production series-3 waveform binaries: +``02 00`` x 32,617, ``01 00`` x 74, ``00 03`` x 71. + +**``40 NN`` is an ordinary int16 BE data block** of length ``2*NN + 2`` +(1 <= NN <= 8), never a header. The superseded model read it as a header of +length ``2*NN + 16``, which is exactly why walks drifted: the "variable prefix" +of 0/2/4/6/8 bytes reported earlier was walker drift, precisely +``4 - (old_stop - true_record_start)``, on all 25 affected files. + +Block table for data sections (``NN = ((tag_hi & 0x0F) << 8) | tag_lo``): + +| tag | length | samples | +|---|---|---| +| ``0X NN`` | 2 | NN (RLE hold — holds the previous value in BOTH delta and absolute modes) | +| ``1X NN`` | NN/2 + 2 | NN (4-bit nibble) | +| ``2X NN`` | NN + 2 | NN (int8) | +| ``30 NN`` | NN*1.5 + 2 | NN (12-bit packed) | +| ``40 NN`` | 2*NN + 2 | NN (int16 BE) | + +The ``30 NN`` "trailer length = NN*4" fallback must NOT be applied inside a +record — it corrupts records whose ``30 NN`` sits near a boundary. + +**The preamble is segment 0's implicit Tran record.** ``body[1:3]`` carries +the same mode pair: ``00 02 00`` (1,387 of 1,388 files) means two int16 BE +anchors at ``body[3:7]`` then delta blocks; ``00 00 03`` (1 file, +``BE13121/O121L4L1.KF0W``) means raw 12-bit absolute from ``body[3]``, which +cannot be block-walked — the first record must be located by scanning. + +**Verification.** The length chain terminates on a ``0x06`` record in 1,387 of +1,388 files (the exception has an ambiguous footer signature inside its sample +stream). All four channels come out at identical length in **1,388/1,388**, +against 156/1,388 under the superseded model. Against the 75 events with a +preserved Blastware ASCII export: sample-count exact **72/75 -> 75/75**, fully +exact **70/75 -> 73/75** (the 2 remaining differ by exactly 1 LSB on samples +sitting at the +-10.000 in/s rail). Against device-reported PPVs on a live +decode: waveform **1306/1306** exact with mean absolute ratio error 0.00000; +histogram **4458/4459**. + +This also eliminated the walker-over-read class entirely. 24 of those 35 +"histogram" over-reads were histogram files that ``read_blastware_file`` fed to +the *waveform* codec first; the old walker accepted them and returned garbage +(one produced 98,923 "intervals"), while the record-chain decoder correctly +returns None so they fall through to ``histogram_codec``. + ##### Trailer The trailer (after the last segment's data) is a sequence of 32-byte @@ -3037,7 +3081,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger | **ACH inbound server — RESOLVED.** `bridges/ach_server.py` implements full inbound ACH pipeline. `--clear-after-download` flag for delete-after-upload workflow. Post-erase key-reuse detection via `max_downloaded_key` high-water mark. | RESOLVED | 2026-04-11 | | | **Sensor Check dropdown byte location** — byte offset in 1A compliance config payload for the "Sensor Check: Before monitoring / After each event / Disabled" setting is NOT YET LOCATED. Confirmed: unit always runs with "Before monitoring" set. Need a capture with "Disabled" to diff. | MEDIUM | 2026-04-08 | Still open | | **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open | -| **Variable-prefix segment descriptors** — 3 of 75 ground-truth events still truncate. The walk reaches a segment header whose channel-id field is preceded by a *variable-width* prefix (2, 4 or 6 bytes observed; the standard tagless form always has 4), carrying an `01 00` marker instead of `02 00`. The marker is **not** simply an anchor count — `01 00` records appear with both 2- and 4-byte anchor fields in the same file, so prefix width and marker are not yet reconciled. Examples: `BE12599/N599LPNB.JF0W` @1155, `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485. | MEDIUM | 2026-08-25 | Still open | +| ~~**Variable-prefix segment descriptors**~~ - **RESOLVED 2026-08-25:** there is no variable prefix. The body is a chain of self-delimiting records (see S7.6.1); the 0/2/4/6/8-byte prefix was walker drift from reading `40 NN` as a segment header. All 25 affected files now chain cleanly to the `06` terminator. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 | | ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 | | **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open | | **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open | diff --git a/docs/waveform_codec_re_status.md b/docs/waveform_codec_re_status.md index 1a0b864..6d9ff71 100644 --- a/docs/waveform_codec_re_status.md +++ b/docs/waveform_codec_re_status.md @@ -1,3 +1,22 @@ +> ## SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN +> +> The tag-dispatch model described in this document — `40 NN` segment headers, +> tagless headers, channel rotation — is **wrong**. It produced nearly-correct +> output only because the block table happens to tile the data sections. +> +> The body is a chain of self-delimiting per-channel records: `off+2` is a +> uint16 BE length, `next = off + 2 + len`, and the chain ends on a record whose +> chan_id is `0x06`. A 3-valued mode enum at `off+8` selects delta / absolute / +> raw-12-bit semantics. `40 NN` is an ordinary int16 BE data block. +> +> See the record-chain section of `docs/instantel_protocol_reference.md` §7.6.1 +> and the implementation in `minimateplus/waveform_codec.py`. +> +> Result: all four channels equal length in 1388/1388 files (was 156/1388); +> ASCII sample-count exact 75/75, fully exact 73/75; device PPV 1306/1306. +> +> This document is retained as the reasoning trail. + # Waveform body codec — FULLY DECODED (2026-05-11) This is the **clean working note** for the body-codec reverse-engineering diff --git a/micromate/idf_file.py b/micromate/idf_file.py index f3db878..60937b8 100644 --- a/micromate/idf_file.py +++ b/micromate/idf_file.py @@ -47,7 +47,19 @@ from dataclasses import dataclass from pathlib import Path from typing import Optional, Union -from minimateplus.waveform_codec import decode_waveform_v2 +# Thor IDFW bodies are pinned to the SUPERSEDED tag-dispatch decoder. +# +# _find_waveform_body_offset() trial-decodes every candidate offset and keeps +# whichever yields the most samples. The series-3 record-chain decoder +# correctly returns None where the legacy walker returned garbage, which +# changes that heuristic's winner on 33 of 577 files. The net effect measured +# 2026-08-25 was positive (all-channels-equal 8/577 -> 506/577, mean abs PPV +# error 0.228 -> 0.173 in/s) but Thor has no ASCII ground truth in the corpus +# and its geo scaling is separately suspect, so the switch is deferred until +# the body-offset search is reworked to use the record chain directly. +from minimateplus.waveform_codec import ( + decode_waveform_legacy as decode_waveform_v2, +) from .models import IdfEvent, IdfPeaks, IdfReport diff --git a/minimateplus/event_file_io.py b/minimateplus/event_file_io.py index 5d3a01f..15b6641 100644 --- a/minimateplus/event_file_io.py +++ b/minimateplus/event_file_io.py @@ -27,6 +27,7 @@ from typing import Optional, Union from .models import Event, PeakValues, ProjectInfo, Timestamp from . import blastware_file as _bw # avoid circular reference at module load from .bw_ascii_report import BwAsciiReport +from . import waveform_codec as _wc from .waveform_codec import decode_waveform_v2, decoded_to_adc_counts from .histogram_codec import decode_histogram_body @@ -843,7 +844,13 @@ def read_blastware_file(path: Union[str, Path]) -> Event: # Footer: locate the 0e 08 marker, validating the year is in a sane range. body_start = _bw._WAVEFORM_HEADER_SIZE + 21 - footer_pos = -1 + # The 0e 08 + plausible-year footer signature can occur inside the sample + # stream. Collect every candidate and prefer the first whose body yields a + # waveform record chain terminating on the 0x06 marker; fall back to the + # first candidate otherwise. Blast radius measured 2026-08-25: changes the + # chosen footer on exactly 1 of 1,388 series-3 waveform files + # (BE17353/S353L4O5.OX0W, false positive at 3800, real footer at 8576). + footer_candidates = [] pos = body_start while True: pos = raw.find(b"\x0e\x08", pos) @@ -851,10 +858,24 @@ def read_blastware_file(path: Union[str, Path]) -> Event: break yr = (raw[pos + 4] << 8) | raw[pos + 5] if 2015 <= yr <= 2050: - footer_pos = pos - break + footer_candidates.append(pos) pos += 1 + footer_pos = -1 + for cand in footer_candidates: + cand_body = raw[body_start:cand] + try: + recs = _wc.walk_records(cand_body) + except Exception: + recs = [] + if recs: + tail = recs[-1]["end"] + if tail + 5 <= len(cand_body) and cand_body[tail + 4] == _wc.STREAM_END_ID: + footer_pos = cand + break + if footer_pos < 0 and footer_candidates: + footer_pos = footer_candidates[0] + if footer_pos < 0 and len(raw) >= 26: footer_pos = len(raw) - 26 if footer_pos < body_start: diff --git a/minimateplus/waveform_codec.py b/minimateplus/waveform_codec.py index 0c96422..9a74780 100644 --- a/minimateplus/waveform_codec.py +++ b/minimateplus/waveform_codec.py @@ -441,8 +441,16 @@ def decode_tran_initial(body: bytes) -> Optional[List[int]]: return out -def decode_waveform_v2(body: bytes) -> Optional[dict]: +def decode_waveform_legacy(body: bytes) -> Optional[dict]: """ + SUPERSEDED 2026-08-25 — the tag-dispatch / segment-header model. + + Retained because ``micromate/idf_file.py`` trial-decodes Thor IDFW bodies + at many candidate offsets and keeps whichever yields the most samples; + the record-chain decoder returns None where this one returned garbage, + which shifts that heuristic's winner. Thor is pinned here until its own + body-offset search is reworked. Do not use for series-3. + Decode the body into per-channel sample arrays. Status (2026-05-11 evening — channel-rotation hypothesis CONFIRMED): @@ -673,3 +681,231 @@ def decode_a5_frames(a5_frames) -> Optional[dict]: if decoded is None: return None return decoded_to_adc_counts(decoded) + + +# ── Record-chain body model (CONFIRMED 2026-08-25) ────────────────────────── +# +# The body is NOT a flat tag-dispatch stream with ``40 NN`` segment headers. +# It is a chain of self-delimiting per-channel RECORDS: +# +# off+0 field2 uint16 purpose unknown (not a length, not a checksum) +# off+2 len uint16 BE next_record = off + 2 + len <- authoritative +# off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL +# 0x06 = end of waveform stream +# off+5 0x00 +# off+6 0x00 +# off+7 segment index +# off+8 mode 2 bytes, a 3-valued enum (see below) +# off+10 anchors 2 x int16 BE, ABSOLUTE — present only when mode is 02 00 +# +# Mode semantics, all ground-truth verified: +# 02 00 14-byte header; emit the 2 anchors, then blocks are CUMULATIVE deltas +# 01 00 10-byte header; no anchors; blocks carry ABSOLUTE sample values +# 00 03 10-byte header; NO TAGS AT ALL — the data section is raw 12-bit +# packed ABSOLUTE samples (6 bytes -> 4 samples) +# +# ``40 NN`` is an ordinary int16 BE DATA block (length 2*NN + 2), never a header. +# The previous model read it as a variable-width segment header of length +# 2*NN + 16, which is why walks drifted and channels came out unequal. +# +# Verified over the 1,388 series-3 waveform binaries in the production +# snapshot: the length chain terminates on a 0x06 record in 1,387 of them (the +# exception has an ambiguous footer, handled by the caller), and all four +# channels come out at identical length in 1,388/1,388 — against 156/1,388 +# under the superseded model. Against the 75 events with a preserved +# Blastware ASCII export: sample-count exact 72/75 -> 75/75, fully exact +# 70/75 -> 73/75. + +CHANNEL_IDS = {0x46: "Tran", 0x47: "Vert", 0x48: "Long", 0x49: "MicL"} +STREAM_END_ID = 0x06 + +MODE_DELTA = (0x02, 0x00) +MODE_ABSOLUTE = (0x01, 0x00) +MODE_RAW12 = (0x00, 0x03) +_MODES = (MODE_DELTA, MODE_ABSOLUTE, MODE_RAW12) + + +def _u16(b: bytes, p: int) -> int: + return (b[p] << 8) | b[p + 1] + + +def _i16(b: bytes, p: int) -> int: + v = _u16(b, p) + return v - 0x10000 if v >= 0x8000 else v + + +def data_block_len(body: bytes, p: int) -> Tuple[Optional[int], Optional[int]]: + """``(byte_length, n_samples)`` of the data block at *p*, or ``(None, None)``. + + Data-section blocks only — there is no segment-header tag in this model. + ``30 NN`` has no trailer-length fallback here; that fallback corrupted + records whose ``30 NN`` sat near a record boundary. + """ + if p + 2 > len(body): + return None, None + t0, t1 = body[p], body[p + 1] + hi = t0 & 0xF0 + nn = ((t0 & 0x0F) << 8) | t1 + if hi == 0x40: # int16 BE data block + return (None, None) if (nn == 0 or nn > 0x08) else (2 * nn + 2, nn) + if nn == 0 or nn % 4: + return None, None + if hi == 0x00: + return 2, nn # RLE hold + if hi == 0x10: + return nn // 2 + 2, nn # 4-bit nibble + if hi == 0x20: + return nn + 2, nn # int8 + if hi == 0x30: + return nn * 3 // 2 + 2, nn # 12-bit packed + return None, None + + +def unpack12(data: bytes) -> List[int]: + """Raw 12-bit packed samples: 6 bytes -> 4 signed values.""" + out: List[int] = [] + for g in range(len(data) // 6): + hi = (data[6 * g] << 8) | data[6 * g + 1] + for k in range(4): + x = (((hi >> (12 - 4 * k)) & 0xF) << 8) | data[6 * g + 2 + k] + out.append(x - 0x1000 if x >= 0x800 else x) + return out + + +def is_record(body: bytes, p: int) -> bool: + """True if a per-channel record header starts at *p*.""" + return (p + 10 <= len(body) + and body[p + 4] in CHANNEL_IDS + and body[p + 5] == 0x00 and body[p + 6] == 0x00 + and 8 <= _u16(body, p + 2) <= len(body) - p + and (body[p + 8], body[p + 9]) in _MODES) + + +def find_first_record(body: bytes) -> Optional[int]: + """Offset of the first record, or None. + + Under the normal ``00 02 00`` preamble the leading bytes are segment-0's + Tran blocks, so walk them. Under the ``00 00 03`` preamble that data is + raw 12-bit with no tags at all and cannot be block-walked — scan instead. + """ + if len(body) >= 3 and (body[1], body[2]) == MODE_RAW12: + scan_from = 3 + else: + i = 7 + while i < len(body): + if is_record(body, i): + nxt = i + 2 + _u16(body, i + 2) + if nxt + 5 <= len(body) and (is_record(body, nxt) + or body[nxt + 4] == STREAM_END_ID): + return i + length, _ = data_block_len(body, i) + if length is None: + return None + i += length + return None + for i in range(scan_from, max(scan_from, len(body) - 10)): + if is_record(body, i): + nxt = i + 2 + _u16(body, i + 2) + if nxt + 5 <= len(body) and (is_record(body, nxt) + or body[nxt + 4] == STREAM_END_ID): + return i + return None + + +def walk_records(body: bytes, first: Optional[int] = None) -> List[dict]: + """Follow the length chain from *first* to the ``0x06`` terminator.""" + if first is None: + first = find_first_record(body) + out: List[dict] = [] + if first is None: + return out + p, seen = first, set() + while p is not None and p + 10 <= len(body): + if p in seen: + break + seen.add(p) + cid = body[p + 4] + if cid == STREAM_END_ID or cid not in CHANNEL_IDS: + break + length = _u16(body, p + 2) + if length < 8 or p + 2 + length > len(body): + break + out.append({"offset": p, "channel": CHANNEL_IDS[cid], + "segment_index": body[p + 7], + "mode": (body[p + 8], body[p + 9]), + "end": p + 2 + length}) + p += 2 + length + return out + + +def decode_waveform_v2(body: bytes) -> Optional[dict]: + """Decode a Blastware waveform body into per-channel sample arrays. + + Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}`` + in 16-count units (LSB = 0.005 in/s at Normal range), or None if *body* + is not a decodable waveform body. + + Implements the record-chain model documented above. + """ + if len(body) < 8 or body[0] != 0x00: + return None + preamble = (body[1], body[2]) + if preamble not in (MODE_DELTA, MODE_RAW12): + return None + first = find_first_record(body) + if first is None: + return None + + out: dict = {c: [] for c in ("Tran", "Vert", "Long", "MicL")} + + def run(channel: str, start: int, end: int, absolute: bool) -> None: + cur = out[channel][-1] if out[channel] else 0 + i = start + while i < end: + length, nn = data_block_len(body, i) + if length is None or i + length > end: + return # stop this record; the chain resyncs at end + hi = body[i] & 0xF0 + if hi == 0x00: + vals = [None] * nn + elif hi == 0x10: + vals = [] + for k in range(nn): + byte = body[i + 2 + k // 2] + v = (byte >> 4) if k % 2 == 0 else (byte & 0xF) + vals.append(v - 16 if v >= 8 else v) + elif hi == 0x20: + vals = [v - 256 if v >= 128 else v + for v in body[i + 2:i + 2 + nn]] + elif hi == 0x30: + vals = unpack12(body[i + 2:i + length]) + else: + vals = [_i16(body, i + 2 + 2 * k) for k in range(nn)] + for v in vals: + if v is None: + pass # RLE hold, in delta AND absolute modes + elif absolute: + cur = v + else: + cur += v + out[channel].append(cur) + i += length + + # Segment 0 is an implicit Tran record carried in the preamble. + if preamble == MODE_DELTA: + out["Tran"].extend([_i16(body, 3), _i16(body, 5)]) + run("Tran", 7, first, absolute=False) + else: + out["Tran"].extend(unpack12(body[3:first])) + + for rec in walk_records(body, first): + ch, off, mode, end = (rec["channel"], rec["offset"], + rec["mode"], rec["end"]) + if mode == MODE_DELTA: + out[ch].extend([_i16(body, off + 10), _i16(body, off + 12)]) + run(ch, off + 14, end, absolute=False) + elif mode == MODE_ABSOLUTE: + run(ch, off + 10, end, absolute=True) + elif mode == MODE_RAW12: + out[ch].extend(unpack12(body[off + 10:end])) + return out diff --git a/tests/test_waveform_codec.py b/tests/test_waveform_codec.py index 2d3290f..eebcf9d 100644 --- a/tests/test_waveform_codec.py +++ b/tests/test_waveform_codec.py @@ -14,6 +14,7 @@ import pytest from minimateplus.waveform_codec import ( WaveformBlock, + decode_waveform_legacy, decode_tran_initial, decode_waveform_v2, decoded_to_adc_counts, @@ -548,9 +549,18 @@ def test_walk_body_wide_rle_block(): assert blocks[0].length == 2 +# NOTE (2026-08-25): the four tests below assert the SUPERSEDED tag-dispatch +# model — `40 NN` as a variable-width segment header, tagless headers, channel +# from rotation. The body format is really a chain of self-delimiting records +# (see the record-chain section of waveform_codec.py), so `decode_waveform_v2` +# no longer behaves this way. They are retargeted at `decode_waveform_legacy`, +# which still implements the old model and is pinned by micromate/idf_file.py +# for Thor IDFW bodies. + + def test_decode_wide_rle_repeats_full_run(): """A wide RLE run repeats the running value NN times, not NN & 0xFF.""" - decoded = decode_waveform_v2(_synth(b"\x01\x0c")) + decoded = decode_waveform_legacy(_synth(b"\x01\x0c")) # 2 preamble anchors + 268 repeats assert len(decoded["Tran"]) == 2 + 268 assert set(decoded["Tran"]) == {0} @@ -589,7 +599,7 @@ def test_segment_header_anchors_track_header_width(nn, hdr_len): data[2 * nn + 8 : 2 * nn + 10] = b"\x02\x00" data[2 * nn + 10 : 2 * nn + 12] = (7).to_bytes(2, "big") # anchor 0 data[2 * nn + 12 : 2 * nn + 14] = (9).to_bytes(2, "big") # anchor 1 - decoded = decode_waveform_v2(_synth(bytes([0x40, nn]) + bytes(data))) + decoded = decode_waveform_legacy(_synth(bytes([0x40, nn]) + bytes(data))) assert decoded["Vert"][:2] == [7, 9] @@ -629,7 +639,7 @@ def test_tagless_header_carries_full_14_bytes_as_data(): def test_tagless_header_anchors_and_channel_id(): """Anchors decode from data[10:14]; the channel comes from the id byte.""" - decoded = decode_waveform_v2(_synth(_tagless(chan_id=0x48, a0=11, a1=13))) + decoded = decode_waveform_legacy(_synth(_tagless(chan_id=0x48, a0=11, a1=13))) assert decoded["Long"][:2] == [11, 13] # 0x48 → Long, not rotation assert decoded["Vert"] == [] @@ -642,10 +652,146 @@ def test_segment_channel_comes_from_id_not_rotation(chan_id, name): would put the second one on the next channel and corrupt both.""" body = _synth(_tagless(chan_id=chan_id, seg=1, a0=5, a1=6), _tagless(chan_id=chan_id, seg=2, a0=7, a1=8)) - decoded = decode_waveform_v2(body) + decoded = decode_waveform_legacy(body) # Tran additionally carries the body preamble's 2 anchors (both 0 here). expected = [0, 0, 5, 6, 7, 8] if name == "Tran" else [5, 6, 7, 8] assert decoded[name] == expected for other in ("Tran", "Vert", "Long", "MicL"): if other != name: assert decoded[other] == ([0, 0] if other == "Tran" else []) + + +# ── Record-chain body model (2026-08-25) ──────────────────────────────────── +# +# The body is a chain of self-delimiting per-channel records, not a flat +# tag-dispatch stream. Verified over 1,388 production series-3 waveform +# binaries: the chain terminates on a 0x06 record in 1,387 of them and all +# four channels come out at identical length in 1,388/1,388 (was 156/1,388). +# Against the 75 events with a preserved Blastware ASCII export: sample-count +# exact 72/75 -> 75/75, fully exact 70/75 -> 73/75. + +from minimateplus.waveform_codec import ( # noqa: E402 + CHANNEL_IDS, + MODE_ABSOLUTE, + MODE_DELTA, + MODE_RAW12, + STREAM_END_ID, + data_block_len, + find_first_record, + is_record, + unpack12, + walk_records, +) + + +def _rec(chan_id, mode, payload, seg=0, field2=b"\x00\x00", anchors=None): + """Build one self-delimiting record.""" + head = bytearray() + head += bytes([chan_id, 0x00, 0x00, seg]) + head += bytes(mode) + if anchors is not None: + for a in anchors: + head += int(a).to_bytes(2, "big", signed=True) + body = bytes(head) + payload + return field2 + (len(body) + 2).to_bytes(2, "big") + body + + +def _terminator(): + return b"\x00\x00" + (8).to_bytes(2, "big") + bytes([STREAM_END_ID, 0, 0, 0, 0, 0]) + + +def _body(*records, preamble=b"\x00\x02\x00", seg0=b"\x00\x00\x00\x00"): + return preamble + seg0 + b"".join(records) + _terminator() + + +def test_forty_nn_is_a_data_block_not_a_segment_header(): + """`40 NN` is an int16 BE data block of length 2*NN + 2. + + The superseded model read it as a segment header of length 2*NN + 16, + which is what made walks drift and channels come out unequal. + """ + assert data_block_len(b"\x40\x02\x00\x01\x00\x02", 0) == (6, 2) + assert data_block_len(b"\x40\x08" + bytes(16), 0) == (18, 8) + # NN > 8 is not a data block + assert data_block_len(b"\x40\x0c" + bytes(24), 0) == (None, None) + + +def test_record_chain_is_followed_by_length_not_by_tag_sniffing(): + payload = b"\x00\x04" # RLE hold x4 + body = _body(_rec(0x47, MODE_DELTA, payload, anchors=(3, 5))) + recs = walk_records(body) + assert len(recs) == 1 + assert recs[0]["channel"] == "Vert" + assert recs[0]["mode"] == MODE_DELTA + + +def test_chain_terminates_on_channel_id_06(): + body = _body(_rec(0x47, MODE_DELTA, b"\x00\x04", anchors=(1, 1)), + _rec(0x48, MODE_DELTA, b"\x00\x04", anchors=(2, 2))) + assert [r["channel"] for r in walk_records(body)] == ["Vert", "Long"] + assert not is_record(body, len(body) - 10) # the terminator is not a record + + +def test_mode_delta_emits_anchors_then_accumulates(): + # two anchors, then an int8 block of +1,+1,+1,+1 + body = _body(_rec(0x47, MODE_DELTA, b"\x20\x04\x01\x01\x01\x01", + anchors=(10, 11))) + d = decode_waveform_v2(body) + assert d["Vert"] == [10, 11, 12, 13, 14, 15] + + +def test_mode_absolute_replaces_rather_than_accumulates(): + """mode `01 00`: no anchors, and block values are ABSOLUTE samples.""" + body = _body(_rec(0x48, MODE_ABSOLUTE, b"\x20\x04\x0a\x0b\x0c\x0d")) + d = decode_waveform_v2(body) + assert d["Long"] == [10, 11, 12, 13], "01 00 blocks are absolute, not deltas" + + +def test_mode_absolute_rle_holds_the_previous_value(): + body = _body(_rec(0x48, MODE_ABSOLUTE, b"\x20\x04\x07\x07\x07\x07\x00\x04")) + d = decode_waveform_v2(body) + assert d["Long"] == [7, 7, 7, 7, 7, 7, 7, 7] + + +def test_mode_raw12_has_no_tags_at_all(): + """mode `00 03`: the whole data section is raw 12-bit absolute samples. + + Decoding these matters for the time base — skipping the record would + displace every later sample on that channel (observed on + BE9558/K558LOF2.820W, MicL shifted by exactly 512). + """ + packed = bytes([0x01, 0x23, 0x04, 0x05, 0x06, 0x07]) # 4 samples + body = _body(_rec(0x49, MODE_RAW12, packed)) + d = decode_waveform_v2(body) + assert d["MicL"] == unpack12(packed) + assert len(d["MicL"]) == 4 + + +def test_unpack12_sign_extends(): + assert unpack12(bytes([0x00, 0x00, 0x01, 0x02, 0x03, 0x04])) == [1, 2, 3, 4] + # high nibble 0x8 -> negative + assert unpack12(bytes([0x80, 0x00, 0x00, 0x00, 0x00, 0x00]))[0] == -2048 + + +def test_channel_comes_from_the_record_id(): + for cid, name in CHANNEL_IDS.items(): + body = _body(_rec(cid, MODE_ABSOLUTE, b"\x20\x04\x01\x02\x03\x04")) + d = decode_waveform_v2(body) + assert d[name][-4:] == [1, 2, 3, 4], f"{name} misrouted" + + +def test_raw12_preamble_is_scanned_not_block_walked(): + """A `00 00 03` preamble carries raw 12-bit data from body[3] with no tags, + so find_first_record must scan rather than block-walk. One production file + has this (BE13121/O121L4L1.KF0W); block-walking returns None on it.""" + packed = bytes([0x00, 0x00, 0x01, 0x02, 0x03, 0x04]) + body = _body(_rec(0x47, MODE_DELTA, b"\x00\x04", anchors=(1, 1)), + preamble=b"\x00\x00\x03", seg0=packed) + assert find_first_record(body) == 3 + len(packed) + d = decode_waveform_v2(body) + assert d["Tran"] == unpack12(packed) + + +def test_returns_none_when_no_record_chain(): + assert decode_waveform_v2(b"\x00\x02\x00" + bytes(40)) is None + assert decode_waveform_v2(b"") is None