fix(codec): the waveform body is a record chain, not a tag stream
Supersedes the segment-header model entirely, including the fixes made earlier today. Found via multi-agent structural analysis of the 25 files that stalled the walker, then verified independently. Records are self-delimiting: off+2 is a uint16 BE length, next_record = off + 2 + len, and the chain ends on a record whose chan_id is 0x06. off+8 carries a 3-valued mode enum: 02 00 14-byte header, 2 anchors, then CUMULATIVE delta blocks 01 00 10-byte header, no anchors, blocks are ABSOLUTE values 00 03 10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute `40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header. Reading it as a 2*NN + 16 header is what made walks drift — the "variable-prefix segment descriptors" reported earlier today were not a format feature, just walker drift of exactly 4 - (old_stop - true_record_start), on all 25 affected files. Measured on the production snapshot: all four channels equal length 156/1388 -> 1388/1388 ASCII sample-count exact 72/75 -> 75/75 ASCII fully exact 70/75 -> 73/75 device PPV waveform (live) 1288/1306 -> 1306/1306 (mean err 0.00000) device PPV histogram (live) 4434/4459 -> 4458/4459 Also eliminates the walker-over-read class: 24 of those 35 files were histograms that read_blastware_file fed to the waveform codec first; the old walker accepted them and returned garbage (one yielded 98,923 "intervals"), while the record-chain decoder returns None so they fall through to histogram_codec. 00 03 records are DECODED, not skipped. Skipping them silently shifts the time base of everything after them on that channel — BE9558/ K558LOF2.820W had MicL displaced by exactly 512 samples with nothing marking the gap. Footer detection now prefers the 0e 08 candidate whose body yields a chain terminating on 0x06; the signature can occur inside a sample stream. Blast radius 1 file of 1388. The superseded model survives as decode_waveform_legacy, pinned by micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes candidates and keeps whichever yields the most samples, so the new decoder returning None where the old returned garbage changes that heuristic's winner. Deferred until that search uses the record chain. Tests: 253 passed (+11), failure list unchanged from baseline. The 9 tests pinning the superseded model are retargeted at decode_waveform_legacy, which still implements it. NOTE: stored .h5 files need regenerating — nearly all get longer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
This commit is contained in:
Claude Opus 5
@@ -8,6 +8,62 @@ All notable changes to seismo-relay are documented here.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The series-3 waveform body is a RECORD CHAIN, not a tag stream — this
|
||||
supersedes the segment-header model, including the fixes made earlier the
|
||||
same day.**
|
||||
|
||||
Records are self-delimiting. `off+2` is a `uint16 BE` length and
|
||||
`next_record = off + 2 + len`; the chain ends on a record whose `chan_id` is
|
||||
`0x06`. `off+8` carries a 3-valued mode enum:
|
||||
|
||||
| mode | header | data section |
|
||||
|---|---|---|
|
||||
| `02 00` | 14 B | anchors, then **cumulative deltas** |
|
||||
| `01 00` | 10 B | no anchors, **absolute** values |
|
||||
| `00 03` | 10 B | **no tags at all** — raw 12-bit packed absolute |
|
||||
|
||||
**`40 NN` is an ordinary int16 BE data block** (`2*NN + 2`), never a segment
|
||||
header. Reading it as a `2*NN + 16` header is what made walks drift — and the
|
||||
"variable-prefix segment descriptors" reported earlier today were not a format
|
||||
feature at all, just walker drift of exactly
|
||||
`4 - (old_stop - true_record_start)` on all 25 affected files.
|
||||
|
||||
Measured against the production snapshot:
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| all four channels equal length | 156 / 1388 | **1388 / 1388** |
|
||||
| ASCII sample-count exact | 72 / 75 | **75 / 75** |
|
||||
| ASCII fully exact | 70 / 75 | **73 / 75** |
|
||||
| device PPV, waveform (live decode) | 1288 / 1306 | **1306 / 1306** |
|
||||
| device PPV, histogram (live decode) | 4434 / 4459 | **4458 / 4459** |
|
||||
|
||||
Mean absolute PPV ratio error on waveforms is now 0.00000. The 2 remaining
|
||||
ASCII imperfections differ by exactly 1 LSB on samples sitting at the
|
||||
±10.000 in/s rail.
|
||||
|
||||
**This also eliminated the walker-over-read class.** 24 of those 35 files
|
||||
were histograms that `read_blastware_file` fed to the *waveform* codec first;
|
||||
the old walker accepted them and returned garbage (one yielded 98,923
|
||||
"intervals"), while the record-chain decoder correctly returns `None` so they
|
||||
fall through to `histogram_codec`.
|
||||
|
||||
`00 03` records are decoded rather than skipped. Skipping them does not merely
|
||||
lose samples — it silently shifts the time base of everything after them on
|
||||
that channel (observed on `BE9558/K558LOF2.820W`, MicL displaced by exactly
|
||||
512 samples with nothing marking the gap).
|
||||
|
||||
Footer detection now prefers whichever `0e 08` candidate yields a chain
|
||||
terminating on `0x06`, since the signature can occur inside a sample stream.
|
||||
Blast radius: 1 file of 1,388.
|
||||
|
||||
The superseded model is retained as `decode_waveform_legacy` and pinned by
|
||||
`micromate/idf_file.py`, whose Thor IDFW body-offset search trial-decodes
|
||||
candidate offsets and keeps whichever yields the most samples — the new
|
||||
decoder correctly returns `None` where the old one returned garbage, which
|
||||
changes that heuristic's winner. Switching Thor over is deferred until that
|
||||
search is reworked to use the record chain directly.
|
||||
|
||||
- **Series-3 histogram block is uniformly big-endian, and the stream's final
|
||||
block has its own tail — the codec was clipping large peaks and dropping the
|
||||
last interval of nearly every histogram.**
|
||||
|
||||
@@ -223,6 +223,29 @@ custom delta + RLE + variable-width codec.
|
||||
`NN + 2` for int8 blocks). Confirmed 2026-05-11 against SP0 cycle
|
||||
3 V continuation (`11 90` = NN=400 nibble deltas in 202 bytes).
|
||||
|
||||
### ⚠ SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN
|
||||
|
||||
Everything in this section below about `40 NN` segment headers, tagless
|
||||
headers, variable header widths and channel rotation describes a model that
|
||||
is **wrong**. The body is a chain of self-delimiting per-channel records:
|
||||
|
||||
off+2 len uint16 BE -> next_record = off + 2 + len
|
||||
off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL / 0x06 = END
|
||||
off+8 mode 02 00 = deltas+anchors (14B hdr)
|
||||
01 00 = ABSOLUTE values (10B hdr)
|
||||
00 03 = raw 12-bit absolute, NO TAGS (10B hdr)
|
||||
|
||||
`40 NN` is an ordinary int16 BE data block (`2*NN + 2`), never a header. The
|
||||
"variable prefix" of 0/2/4/6/8 bytes was walker drift, exactly
|
||||
`4 - (old_stop - true_record_start)`.
|
||||
|
||||
All four channels now come out equal length in **1388/1388** files (was
|
||||
156/1388); ASCII sample-count exact **75/75**, fully exact **73/75**; device
|
||||
PPV on a live decode **1306/1306** waveform, **4458/4459** histogram.
|
||||
|
||||
The old model survives as `decode_waveform_legacy` because
|
||||
`micromate/idf_file.py` pins it for Thor IDFW body-offset search.
|
||||
|
||||
### Framing cases added 2026-05-11 → 2026-08-25
|
||||
|
||||
Four more block-framing cases, each of which had been causing **silent
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
|
||||
| Date | Section | Change |
|
||||
|---|---|---|
|
||||
| 2026-08-25 (3) | S7.6.1, S15 | **THE WAVEFORM BODY IS A RECORD CHAIN, NOT A TAG STREAM — supersedes the segment-header model entirely.** Records are self-delimiting: `off+2` is a uint16 BE length and `next = off + 2 + len`; the chain ends on a record whose chan_id is `0x06`. `off+8` holds a 3-valued mode enum - `02 00` (14-byte header, anchors, cumulative deltas), `01 00` (10-byte, no anchors, ABSOLUTE values), `00 03` (10-byte, no tags at all, raw 12-bit absolute). **`40 NN` is an ordinary int16 BE data block of length 2*NN+2**, never a header; reading it as a 2*NN+16 header is what made walks drift, and the 'variable prefix' of 0/2/4/6/8 bytes reported earlier the same day was walker drift, exactly `4 - (old_stop - true_record_start)`. Verified: chain terminates on `06` in 1387/1388 files; all four channels equal length in **1388/1388** (was 156/1388); ASCII sample-count exact 72/75 -> **75/75**, fully exact 70/75 -> **73/75**; device PPV on a live decode **1306/1306** waveform (mean abs ratio error 0.00000) and **4458/4459** histogram. Also eliminated the walker-over-read class: 24 of those files were histograms the waveform codec was wrongly accepting. The superseded model is retained as `decode_waveform_legacy` because `micromate/idf_file.py` pins it for Thor IDFW body-offset search. |
|
||||
| 2026-08-25 (2) | S7.6.2, S15 | **HISTOGRAM BLOCK IS BIG-ENDIAN + terminal block tail.** The 32-byte histogram block's per-channel fields are uint16 **big-endian** (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], `V_halfperiod` [11:13], `L_peak` [13:15], `L_halfperiod` [15:17], `M_peak` [17:19], `M_halfperiod` [19:21]); only `block_ctr` [2:4] is little-endian. The marker is `block[4]` alone - the previous uint16 LE marker test at [4:6] forced `block[5] == 0` and thereby capped every geo peak at 255 counts (1.275 in/s), silently clipping larger peaks. The byte previously documented as a per-channel "annotation" is the high byte of the big-endian half-period, which is why it was non-zero exactly on sub-Hz intervals. Separately, the **final block of each stream carries tail `9c 06 00 42`** rather than `1e 0a 00 00` and holds arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram. Verified against 1211 production histograms paired with their Blastware ASCII exports: 1211/1211 decode exactly, plus 842,442 per-interval frequency comparisons with zero mismatches (previously 1 of 1196 files fully correct). |
|
||||
| 2026-08-25 | §7.6.1, §15, Appendix E (NEW) | **BODY CODEC + SCALE PASS — five findings, all verified against 75 production events paired with their preserved Blastware ASCII exports.** (1) **Geo full scale is 32000 ADC counts, not 32768** — one decoder unit (16 counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Consumers dividing by 32768 read every geophone sample and derived peak **2.34% low**; the error scales with amplitude so it was invisible on quiet events and worst on loud ones. 216 per-channel comparisons: 32768 → 151/216 exact (worst 0.238 in/s on a 10 in/s event); 32000 → 216/216 exact, worst 1 LSB. Affects waveforms, histograms and series-4 alike. (2) **Wide-NN RLE `0X NN`** — the 12-bit NN encoding already known for `1X`/`2X` also applies to the `00 NN` RLE tag (runs > 252 samples). (3) **`30 NN` is not capped at NN=0x10** — data-section blocks reach at least 0x18; the length formula was already right. (4) **`40 NN` segment headers are variable width** — NN counts the previous-channel continuation deltas, so the header is `2*NN + 16` bytes; `40 01` and `40 03` occur alongside `40 02`. A header can also appear **tagless** (the NN=0 case): just the 14-byte tail. (5) **The header field documented as a "monotonic uint32 LE counter" is really `[channel_id][00][00][segment_index]`** with 0x46=Tran 0x47=Vert 0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero disagreements. Decoders should take the channel from this field, not from rotation position. Items (2)–(5) each caused **silent channel truncation**: an unhandled tag ends the walk and the decoder returns short channels with no error. Corpus result end-to-end: exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0. New Appendix E documents the field-observed "offset" device fault. |
|
||||
| 2026-05-20 | §2, §3, §4.2, §5.1, §5.3, §6, §7.5b, §7.6.1, §7.6.3, §7.6.4, §7.7.2, §7.7.3, §7.7.5, §7.8.4, §7.8.7, §7.9, §8, §11, §12, §13, §14, §15, Appendix D | **DOC AUDIT PASS — accuracy sweep against `CLAUDE.md` + `minimateplus/` code.** Fixed: (1) S3 frames terminate on bare ETX, not DLE+ETX — §2/§3 rewritten. (2) §3 payload layout corrected — byte[1]=flags, byte[2]=SUB (was wrongly labelled DLE/ADDR). (3) §4.2 — probe responses do NOT carry data length; lengths are hardcoded `DATA_LENGTHS` constants. (4) §5.1 — removed stale duplicate "SUB 1C = TRIGGER CONFIG READ" row; SUB 0A lengths corrected from `0x30/0x26` to `0x46/0x2C` (real event / boundary marker). (5) §5.3 — added missing write-frame format (BW_CMD-only doubling, DLE-aware checksum, offset formula, ack format, SUB 71 chunk parameters). (6) §6 — fixed "SUB 06 → channel config read" → event storage range. (7) §7.5b / §8 — added the 10-byte `sub_code=0x03` continuous-mode timestamp variant alongside the 9-byte single-shot layout; peak vector sum location corrected from "fixed offset 87" to `tran_pos − 12` (label-relative). (8) §7.6 / §7.6.1 / §7.6.3 / §7.6.4 — switched compliance-anchor convention from the 10-byte form to the canonical 6-byte `\xbe\x80\x00\x00\x00\x00`; recording_mode confirmed at anchor−8 in BOTH read and write (was wrongly listed as anchor−3 write / anchor−4 read); sample_rate at anchor−6, histogram_interval at anchor−4, record_time at anchor+6; geo_range row added at channel_label+33. (9) §7.7.2 — token byte position corrected from `params[6]` to `params[7]`. (10) §7.8.4 — fi==9 skip marked FIXED (already removed from code); chunk-count totals updated. (11) §7.8.7 — TODO replaced with current state of `_decode_a5_metadata_into`. (12) §7.9 — Histogram Interval upgraded ❓ → ✅. (13) §11 — POLL example wire bytes corrected; SUB 5A row added to checksum table. (14) §13 — device-under-test updated for current primary unit (BE11529 / S338.17). (15) §14 — TCP Idle Timeout fixed (0→2 min); Data Forwarding Timeout units clarified. (16) §15 (renumbered from second §14) — open-question items already resolved in CLAUDE.md closed out. (17) Appendix D — extension taxonomy rewritten to reflect the AB0T timestamp encoding (D.5.2/D.5.3); EXTENSION REFUTED warning replaced with the resolved encoding. |
|
||||
@@ -1268,36 +1269,79 @@ re-deriving the whole production store:
|
||||
The series-4 figure is closer to correct but not exact — the Thor
|
||||
per-count LSB is its own open question (see §15).
|
||||
|
||||
###### Unmapped: variable-prefix segment descriptors ❓ OPEN
|
||||
###### SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN, not a tag stream
|
||||
|
||||
Three of 75 ground-truth production events still truncate. In each,
|
||||
the walk reaches a segment header whose channel-id field is preceded by
|
||||
a **variable-width prefix** — 2, 4 or 6 bytes have all been observed,
|
||||
where the standard tagless form always has 4 (``field2`` + ``len``).
|
||||
These records also carry the ``01 00`` marker rather than ``02 00``,
|
||||
and appear packed back-to-back with little or no sample data between
|
||||
them.
|
||||
Everything above about ``40 NN`` segment headers, tagless headers, variable
|
||||
header widths and channel rotation describes a model that is **wrong**. It
|
||||
produced nearly-correct output because the block table happens to tile the
|
||||
data sections correctly, but the framing is not what the device writes.
|
||||
|
||||
The ``01 00`` marker is *not* simply an anchor count: records carrying
|
||||
it have been seen with both 2-byte and 4-byte anchor fields in the same
|
||||
file, so the prefix width and the marker are not yet reconciled.
|
||||
|
||||
The decoder stops cleanly at these rather than emitting garbage.
|
||||
Examples: ``BE12599/N599LPNB.JF0W`` at body offset 1155 (2-byte
|
||||
prefix), ``BE12599/N599LPWJ.980W`` at 849 (6-byte prefix),
|
||||
``BE9558/K558LOF2.820W`` at 1485.
|
||||
|
||||
Examples from event-c (1 sec single-shot):
|
||||
The body is a chain of **self-delimiting per-channel records**:
|
||||
|
||||
```
|
||||
Segment header 1 (offset 235):
|
||||
40 02 | 00 00 00 00 | 0a 4b 01 1e | 47 00 00 00 | 02 00 00 01 | 00 01
|
||||
^counter=0x47
|
||||
Segment header 2 (offset 523):
|
||||
40 02 | ff fe ff fe | 13 f5 01 06 | 48 00 00 00 | 02 00 00 01 | 00 02
|
||||
^counter=0x48 (+1)
|
||||
off+0 field2 uint16 purpose unknown (not a length, not a checksum)
|
||||
off+2 len uint16 BE next_record = off + 2 + len <- authoritative
|
||||
off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL
|
||||
0x06 = END OF WAVEFORM STREAM
|
||||
off+5 0x00
|
||||
off+6 0x00
|
||||
off+7 segment index
|
||||
off+8 mode 2 bytes, a 3-valued enum
|
||||
off+10 anchors 2 x int16 BE, ABSOLUTE -- present ONLY when mode == 02 00
|
||||
```
|
||||
|
||||
**Mode enum**, all three ground-truth verified:
|
||||
|
||||
| mode | header | data section |
|
||||
|---|---|---|
|
||||
| ``02 00`` | 14 bytes | anchors emitted, then blocks are **cumulative deltas** |
|
||||
| ``01 00`` | 10 bytes | no anchors, blocks carry **absolute** sample values |
|
||||
| ``00 03`` | 10 bytes | **no tags at all** — raw 12-bit packed absolute samples |
|
||||
|
||||
Census over the 1,388 production series-3 waveform binaries:
|
||||
``02 00`` x 32,617, ``01 00`` x 74, ``00 03`` x 71.
|
||||
|
||||
**``40 NN`` is an ordinary int16 BE data block** of length ``2*NN + 2``
|
||||
(1 <= NN <= 8), never a header. The superseded model read it as a header of
|
||||
length ``2*NN + 16``, which is exactly why walks drifted: the "variable prefix"
|
||||
of 0/2/4/6/8 bytes reported earlier was walker drift, precisely
|
||||
``4 - (old_stop - true_record_start)``, on all 25 affected files.
|
||||
|
||||
Block table for data sections (``NN = ((tag_hi & 0x0F) << 8) | tag_lo``):
|
||||
|
||||
| tag | length | samples |
|
||||
|---|---|---|
|
||||
| ``0X NN`` | 2 | NN (RLE hold — holds the previous value in BOTH delta and absolute modes) |
|
||||
| ``1X NN`` | NN/2 + 2 | NN (4-bit nibble) |
|
||||
| ``2X NN`` | NN + 2 | NN (int8) |
|
||||
| ``30 NN`` | NN*1.5 + 2 | NN (12-bit packed) |
|
||||
| ``40 NN`` | 2*NN + 2 | NN (int16 BE) |
|
||||
|
||||
The ``30 NN`` "trailer length = NN*4" fallback must NOT be applied inside a
|
||||
record — it corrupts records whose ``30 NN`` sits near a boundary.
|
||||
|
||||
**The preamble is segment 0's implicit Tran record.** ``body[1:3]`` carries
|
||||
the same mode pair: ``00 02 00`` (1,387 of 1,388 files) means two int16 BE
|
||||
anchors at ``body[3:7]`` then delta blocks; ``00 00 03`` (1 file,
|
||||
``BE13121/O121L4L1.KF0W``) means raw 12-bit absolute from ``body[3]``, which
|
||||
cannot be block-walked — the first record must be located by scanning.
|
||||
|
||||
**Verification.** The length chain terminates on a ``0x06`` record in 1,387 of
|
||||
1,388 files (the exception has an ambiguous footer signature inside its sample
|
||||
stream). All four channels come out at identical length in **1,388/1,388**,
|
||||
against 156/1,388 under the superseded model. Against the 75 events with a
|
||||
preserved Blastware ASCII export: sample-count exact **72/75 -> 75/75**, fully
|
||||
exact **70/75 -> 73/75** (the 2 remaining differ by exactly 1 LSB on samples
|
||||
sitting at the +-10.000 in/s rail). Against device-reported PPVs on a live
|
||||
decode: waveform **1306/1306** exact with mean absolute ratio error 0.00000;
|
||||
histogram **4458/4459**.
|
||||
|
||||
This also eliminated the walker-over-read class entirely. 24 of those 35
|
||||
"histogram" over-reads were histogram files that ``read_blastware_file`` fed to
|
||||
the *waveform* codec first; the old walker accepted them and returned garbage
|
||||
(one produced 98,923 "intervals"), while the record-chain decoder correctly
|
||||
returns None so they fall through to ``histogram_codec``.
|
||||
|
||||
##### Trailer
|
||||
|
||||
The trailer (after the last segment's data) is a sequence of 32-byte
|
||||
@@ -3037,7 +3081,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
|
||||
| **ACH inbound server — RESOLVED.** `bridges/ach_server.py` implements full inbound ACH pipeline. `--clear-after-download` flag for delete-after-upload workflow. Post-erase key-reuse detection via `max_downloaded_key` high-water mark. | RESOLVED | 2026-04-11 | |
|
||||
| **Sensor Check dropdown byte location** — byte offset in 1A compliance config payload for the "Sensor Check: Before monitoring / After each event / Disabled" setting is NOT YET LOCATED. Confirmed: unit always runs with "Before monitoring" set. Need a capture with "Disabled" to diff. | MEDIUM | 2026-04-08 | Still open |
|
||||
| **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open |
|
||||
| **Variable-prefix segment descriptors** — 3 of 75 ground-truth events still truncate. The walk reaches a segment header whose channel-id field is preceded by a *variable-width* prefix (2, 4 or 6 bytes observed; the standard tagless form always has 4), carrying an `01 00` marker instead of `02 00`. The marker is **not** simply an anchor count — `01 00` records appear with both 2- and 4-byte anchor fields in the same file, so prefix width and marker are not yet reconciled. Examples: `BE12599/N599LPNB.JF0W` @1155, `BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485. | MEDIUM | 2026-08-25 | Still open |
|
||||
| ~~**Variable-prefix segment descriptors**~~ - **RESOLVED 2026-08-25:** there is no variable prefix. The body is a chain of self-delimiting records (see S7.6.1); the 0/2/4/6/8-byte prefix was walker drift from reading `40 NN` as a segment header. All 25 affected files now chain cleanly to the `06` terminator. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
|
||||
| ~~**Histogram codec drops trailing intervals (series-3)**~~ - **RESOLVED 2026-08-25.** Two errors, both in the block model. (1) The block is **uniformly big-endian**: peaks and half-periods are uint16 BE (`T_peak` [5:7], `T_halfperiod` [7:9], `V_peak` [9:11], ...); only `block_ctr` [2:4] is LE. The old uint8-peak + "annotation"-byte model silently clipped any peak above 1.275 in/s, and the "annotation" byte was really the half-period high byte - non-zero exactly on the sub-Hz intervals BW renders `<1.0`. The marker is `block[4]` alone; testing [4:6] as a uint16 LE marker forced `block[5] == 0`, which is what capped the peak at one byte. (2) The **final block of the stream carries tail `9c 06 00 42`** instead of `1e 0a 00 00`, with arbitrary bytes at [21:23]; rejecting it dropped the last interval of nearly every histogram - often the one holding the event peak. Verified on 1211 production histograms vs their BW ASCII exports: **1211/1211 exact** (interval count + every per-interval peak) and 842,442 frequency comparisons with zero mismatches; was 1/1196. | RESOLVED | 2026-08-25 | Resolved 2026-08-25 |
|
||||
| **Micromate (UM-series) IDF decode is ~1000x low** — e.g. `UM11402_20260406130113.IDFW` decodes a Tran peak of 0.0009 in/s against a device-reported 1.1168. Distinct from the Thor IDF path, which decodes sanely. Suspect a different per-count LSB or a body offset that does not hold for UM-series files. | MEDIUM | 2026-08-25 | Still open |
|
||||
| **Thor IDF per-count LSB** — after the 32000 geo full-scale correction, series-4 Thor peaks sit at a median 0.983 of the device-reported peak (was 0.960 under 32768). Closer, but the residual ~1.7% suggests Thor uses its own per-count LSB rather than the BW 16-count/0.005 in/s convention. A code comment in `sfm/waveform_store.py` claims Thor's LSB is 0.0003 in/s, which would predict Thor reading *high* — the measurement shows the opposite, so that comment is unverified. | LOW | 2026-08-25 | Still open |
|
||||
|
||||
@@ -1,3 +1,22 @@
|
||||
> ## SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN
|
||||
>
|
||||
> The tag-dispatch model described in this document — `40 NN` segment headers,
|
||||
> tagless headers, channel rotation — is **wrong**. It produced nearly-correct
|
||||
> output only because the block table happens to tile the data sections.
|
||||
>
|
||||
> The body is a chain of self-delimiting per-channel records: `off+2` is a
|
||||
> uint16 BE length, `next = off + 2 + len`, and the chain ends on a record whose
|
||||
> chan_id is `0x06`. A 3-valued mode enum at `off+8` selects delta / absolute /
|
||||
> raw-12-bit semantics. `40 NN` is an ordinary int16 BE data block.
|
||||
>
|
||||
> See the record-chain section of `docs/instantel_protocol_reference.md` §7.6.1
|
||||
> and the implementation in `minimateplus/waveform_codec.py`.
|
||||
>
|
||||
> Result: all four channels equal length in 1388/1388 files (was 156/1388);
|
||||
> ASCII sample-count exact 75/75, fully exact 73/75; device PPV 1306/1306.
|
||||
>
|
||||
> This document is retained as the reasoning trail.
|
||||
|
||||
# Waveform body codec — FULLY DECODED (2026-05-11)
|
||||
|
||||
This is the **clean working note** for the body-codec reverse-engineering
|
||||
|
||||
+13
-1
@@ -47,7 +47,19 @@ from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Optional, Union
|
||||
|
||||
from minimateplus.waveform_codec import decode_waveform_v2
|
||||
# Thor IDFW bodies are pinned to the SUPERSEDED tag-dispatch decoder.
|
||||
#
|
||||
# _find_waveform_body_offset() trial-decodes every candidate offset and keeps
|
||||
# whichever yields the most samples. The series-3 record-chain decoder
|
||||
# correctly returns None where the legacy walker returned garbage, which
|
||||
# changes that heuristic's winner on 33 of 577 files. The net effect measured
|
||||
# 2026-08-25 was positive (all-channels-equal 8/577 -> 506/577, mean abs PPV
|
||||
# error 0.228 -> 0.173 in/s) but Thor has no ASCII ground truth in the corpus
|
||||
# and its geo scaling is separately suspect, so the switch is deferred until
|
||||
# the body-offset search is reworked to use the record chain directly.
|
||||
from minimateplus.waveform_codec import (
|
||||
decode_waveform_legacy as decode_waveform_v2,
|
||||
)
|
||||
|
||||
from .models import IdfEvent, IdfPeaks, IdfReport
|
||||
|
||||
|
||||
@@ -27,6 +27,7 @@ from typing import Optional, Union
|
||||
from .models import Event, PeakValues, ProjectInfo, Timestamp
|
||||
from . import blastware_file as _bw # avoid circular reference at module load
|
||||
from .bw_ascii_report import BwAsciiReport
|
||||
from . import waveform_codec as _wc
|
||||
from .waveform_codec import decode_waveform_v2, decoded_to_adc_counts
|
||||
from .histogram_codec import decode_histogram_body
|
||||
|
||||
@@ -843,7 +844,13 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
|
||||
|
||||
# Footer: locate the 0e 08 marker, validating the year is in a sane range.
|
||||
body_start = _bw._WAVEFORM_HEADER_SIZE + 21
|
||||
footer_pos = -1
|
||||
# The 0e 08 + plausible-year footer signature can occur inside the sample
|
||||
# stream. Collect every candidate and prefer the first whose body yields a
|
||||
# waveform record chain terminating on the 0x06 marker; fall back to the
|
||||
# first candidate otherwise. Blast radius measured 2026-08-25: changes the
|
||||
# chosen footer on exactly 1 of 1,388 series-3 waveform files
|
||||
# (BE17353/S353L4O5.OX0W, false positive at 3800, real footer at 8576).
|
||||
footer_candidates = []
|
||||
pos = body_start
|
||||
while True:
|
||||
pos = raw.find(b"\x0e\x08", pos)
|
||||
@@ -851,10 +858,24 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
|
||||
break
|
||||
yr = (raw[pos + 4] << 8) | raw[pos + 5]
|
||||
if 2015 <= yr <= 2050:
|
||||
footer_pos = pos
|
||||
break
|
||||
footer_candidates.append(pos)
|
||||
pos += 1
|
||||
|
||||
footer_pos = -1
|
||||
for cand in footer_candidates:
|
||||
cand_body = raw[body_start:cand]
|
||||
try:
|
||||
recs = _wc.walk_records(cand_body)
|
||||
except Exception:
|
||||
recs = []
|
||||
if recs:
|
||||
tail = recs[-1]["end"]
|
||||
if tail + 5 <= len(cand_body) and cand_body[tail + 4] == _wc.STREAM_END_ID:
|
||||
footer_pos = cand
|
||||
break
|
||||
if footer_pos < 0 and footer_candidates:
|
||||
footer_pos = footer_candidates[0]
|
||||
|
||||
if footer_pos < 0 and len(raw) >= 26:
|
||||
footer_pos = len(raw) - 26
|
||||
if footer_pos < body_start:
|
||||
|
||||
@@ -441,8 +441,16 @@ def decode_tran_initial(body: bytes) -> Optional[List[int]]:
|
||||
return out
|
||||
|
||||
|
||||
def decode_waveform_v2(body: bytes) -> Optional[dict]:
|
||||
def decode_waveform_legacy(body: bytes) -> Optional[dict]:
|
||||
"""
|
||||
SUPERSEDED 2026-08-25 — the tag-dispatch / segment-header model.
|
||||
|
||||
Retained because ``micromate/idf_file.py`` trial-decodes Thor IDFW bodies
|
||||
at many candidate offsets and keeps whichever yields the most samples;
|
||||
the record-chain decoder returns None where this one returned garbage,
|
||||
which shifts that heuristic's winner. Thor is pinned here until its own
|
||||
body-offset search is reworked. Do not use for series-3.
|
||||
|
||||
Decode the body into per-channel sample arrays.
|
||||
|
||||
Status (2026-05-11 evening — channel-rotation hypothesis CONFIRMED):
|
||||
@@ -673,3 +681,231 @@ def decode_a5_frames(a5_frames) -> Optional[dict]:
|
||||
if decoded is None:
|
||||
return None
|
||||
return decoded_to_adc_counts(decoded)
|
||||
|
||||
|
||||
# ── Record-chain body model (CONFIRMED 2026-08-25) ──────────────────────────
|
||||
#
|
||||
# The body is NOT a flat tag-dispatch stream with ``40 NN`` segment headers.
|
||||
# It is a chain of self-delimiting per-channel RECORDS:
|
||||
#
|
||||
# off+0 field2 uint16 purpose unknown (not a length, not a checksum)
|
||||
# off+2 len uint16 BE next_record = off + 2 + len <- authoritative
|
||||
# off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL
|
||||
# 0x06 = end of waveform stream
|
||||
# off+5 0x00
|
||||
# off+6 0x00
|
||||
# off+7 segment index
|
||||
# off+8 mode 2 bytes, a 3-valued enum (see below)
|
||||
# off+10 anchors 2 x int16 BE, ABSOLUTE — present only when mode is 02 00
|
||||
#
|
||||
# Mode semantics, all ground-truth verified:
|
||||
# 02 00 14-byte header; emit the 2 anchors, then blocks are CUMULATIVE deltas
|
||||
# 01 00 10-byte header; no anchors; blocks carry ABSOLUTE sample values
|
||||
# 00 03 10-byte header; NO TAGS AT ALL — the data section is raw 12-bit
|
||||
# packed ABSOLUTE samples (6 bytes -> 4 samples)
|
||||
#
|
||||
# ``40 NN`` is an ordinary int16 BE DATA block (length 2*NN + 2), never a header.
|
||||
# The previous model read it as a variable-width segment header of length
|
||||
# 2*NN + 16, which is why walks drifted and channels came out unequal.
|
||||
#
|
||||
# Verified over the 1,388 series-3 waveform binaries in the production
|
||||
# snapshot: the length chain terminates on a 0x06 record in 1,387 of them (the
|
||||
# exception has an ambiguous footer, handled by the caller), and all four
|
||||
# channels come out at identical length in 1,388/1,388 — against 156/1,388
|
||||
# under the superseded model. Against the 75 events with a preserved
|
||||
# Blastware ASCII export: sample-count exact 72/75 -> 75/75, fully exact
|
||||
# 70/75 -> 73/75.
|
||||
|
||||
CHANNEL_IDS = {0x46: "Tran", 0x47: "Vert", 0x48: "Long", 0x49: "MicL"}
|
||||
STREAM_END_ID = 0x06
|
||||
|
||||
MODE_DELTA = (0x02, 0x00)
|
||||
MODE_ABSOLUTE = (0x01, 0x00)
|
||||
MODE_RAW12 = (0x00, 0x03)
|
||||
_MODES = (MODE_DELTA, MODE_ABSOLUTE, MODE_RAW12)
|
||||
|
||||
|
||||
def _u16(b: bytes, p: int) -> int:
|
||||
return (b[p] << 8) | b[p + 1]
|
||||
|
||||
|
||||
def _i16(b: bytes, p: int) -> int:
|
||||
v = _u16(b, p)
|
||||
return v - 0x10000 if v >= 0x8000 else v
|
||||
|
||||
|
||||
def data_block_len(body: bytes, p: int) -> Tuple[Optional[int], Optional[int]]:
|
||||
"""``(byte_length, n_samples)`` of the data block at *p*, or ``(None, None)``.
|
||||
|
||||
Data-section blocks only — there is no segment-header tag in this model.
|
||||
``30 NN`` has no trailer-length fallback here; that fallback corrupted
|
||||
records whose ``30 NN`` sat near a record boundary.
|
||||
"""
|
||||
if p + 2 > len(body):
|
||||
return None, None
|
||||
t0, t1 = body[p], body[p + 1]
|
||||
hi = t0 & 0xF0
|
||||
nn = ((t0 & 0x0F) << 8) | t1
|
||||
if hi == 0x40: # int16 BE data block
|
||||
return (None, None) if (nn == 0 or nn > 0x08) else (2 * nn + 2, nn)
|
||||
if nn == 0 or nn % 4:
|
||||
return None, None
|
||||
if hi == 0x00:
|
||||
return 2, nn # RLE hold
|
||||
if hi == 0x10:
|
||||
return nn // 2 + 2, nn # 4-bit nibble
|
||||
if hi == 0x20:
|
||||
return nn + 2, nn # int8
|
||||
if hi == 0x30:
|
||||
return nn * 3 // 2 + 2, nn # 12-bit packed
|
||||
return None, None
|
||||
|
||||
|
||||
def unpack12(data: bytes) -> List[int]:
|
||||
"""Raw 12-bit packed samples: 6 bytes -> 4 signed values."""
|
||||
out: List[int] = []
|
||||
for g in range(len(data) // 6):
|
||||
hi = (data[6 * g] << 8) | data[6 * g + 1]
|
||||
for k in range(4):
|
||||
x = (((hi >> (12 - 4 * k)) & 0xF) << 8) | data[6 * g + 2 + k]
|
||||
out.append(x - 0x1000 if x >= 0x800 else x)
|
||||
return out
|
||||
|
||||
|
||||
def is_record(body: bytes, p: int) -> bool:
|
||||
"""True if a per-channel record header starts at *p*."""
|
||||
return (p + 10 <= len(body)
|
||||
and body[p + 4] in CHANNEL_IDS
|
||||
and body[p + 5] == 0x00 and body[p + 6] == 0x00
|
||||
and 8 <= _u16(body, p + 2) <= len(body) - p
|
||||
and (body[p + 8], body[p + 9]) in _MODES)
|
||||
|
||||
|
||||
def find_first_record(body: bytes) -> Optional[int]:
|
||||
"""Offset of the first record, or None.
|
||||
|
||||
Under the normal ``00 02 00`` preamble the leading bytes are segment-0's
|
||||
Tran blocks, so walk them. Under the ``00 00 03`` preamble that data is
|
||||
raw 12-bit with no tags at all and cannot be block-walked — scan instead.
|
||||
"""
|
||||
if len(body) >= 3 and (body[1], body[2]) == MODE_RAW12:
|
||||
scan_from = 3
|
||||
else:
|
||||
i = 7
|
||||
while i < len(body):
|
||||
if is_record(body, i):
|
||||
nxt = i + 2 + _u16(body, i + 2)
|
||||
if nxt + 5 <= len(body) and (is_record(body, nxt)
|
||||
or body[nxt + 4] == STREAM_END_ID):
|
||||
return i
|
||||
length, _ = data_block_len(body, i)
|
||||
if length is None:
|
||||
return None
|
||||
i += length
|
||||
return None
|
||||
for i in range(scan_from, max(scan_from, len(body) - 10)):
|
||||
if is_record(body, i):
|
||||
nxt = i + 2 + _u16(body, i + 2)
|
||||
if nxt + 5 <= len(body) and (is_record(body, nxt)
|
||||
or body[nxt + 4] == STREAM_END_ID):
|
||||
return i
|
||||
return None
|
||||
|
||||
|
||||
def walk_records(body: bytes, first: Optional[int] = None) -> List[dict]:
|
||||
"""Follow the length chain from *first* to the ``0x06`` terminator."""
|
||||
if first is None:
|
||||
first = find_first_record(body)
|
||||
out: List[dict] = []
|
||||
if first is None:
|
||||
return out
|
||||
p, seen = first, set()
|
||||
while p is not None and p + 10 <= len(body):
|
||||
if p in seen:
|
||||
break
|
||||
seen.add(p)
|
||||
cid = body[p + 4]
|
||||
if cid == STREAM_END_ID or cid not in CHANNEL_IDS:
|
||||
break
|
||||
length = _u16(body, p + 2)
|
||||
if length < 8 or p + 2 + length > len(body):
|
||||
break
|
||||
out.append({"offset": p, "channel": CHANNEL_IDS[cid],
|
||||
"segment_index": body[p + 7],
|
||||
"mode": (body[p + 8], body[p + 9]),
|
||||
"end": p + 2 + length})
|
||||
p += 2 + length
|
||||
return out
|
||||
|
||||
|
||||
def decode_waveform_v2(body: bytes) -> Optional[dict]:
|
||||
"""Decode a Blastware waveform body into per-channel sample arrays.
|
||||
|
||||
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}``
|
||||
in 16-count units (LSB = 0.005 in/s at Normal range), or None if *body*
|
||||
is not a decodable waveform body.
|
||||
|
||||
Implements the record-chain model documented above.
|
||||
"""
|
||||
if len(body) < 8 or body[0] != 0x00:
|
||||
return None
|
||||
preamble = (body[1], body[2])
|
||||
if preamble not in (MODE_DELTA, MODE_RAW12):
|
||||
return None
|
||||
first = find_first_record(body)
|
||||
if first is None:
|
||||
return None
|
||||
|
||||
out: dict = {c: [] for c in ("Tran", "Vert", "Long", "MicL")}
|
||||
|
||||
def run(channel: str, start: int, end: int, absolute: bool) -> None:
|
||||
cur = out[channel][-1] if out[channel] else 0
|
||||
i = start
|
||||
while i < end:
|
||||
length, nn = data_block_len(body, i)
|
||||
if length is None or i + length > end:
|
||||
return # stop this record; the chain resyncs at end
|
||||
hi = body[i] & 0xF0
|
||||
if hi == 0x00:
|
||||
vals = [None] * nn
|
||||
elif hi == 0x10:
|
||||
vals = []
|
||||
for k in range(nn):
|
||||
byte = body[i + 2 + k // 2]
|
||||
v = (byte >> 4) if k % 2 == 0 else (byte & 0xF)
|
||||
vals.append(v - 16 if v >= 8 else v)
|
||||
elif hi == 0x20:
|
||||
vals = [v - 256 if v >= 128 else v
|
||||
for v in body[i + 2:i + 2 + nn]]
|
||||
elif hi == 0x30:
|
||||
vals = unpack12(body[i + 2:i + length])
|
||||
else:
|
||||
vals = [_i16(body, i + 2 + 2 * k) for k in range(nn)]
|
||||
for v in vals:
|
||||
if v is None:
|
||||
pass # RLE hold, in delta AND absolute modes
|
||||
elif absolute:
|
||||
cur = v
|
||||
else:
|
||||
cur += v
|
||||
out[channel].append(cur)
|
||||
i += length
|
||||
|
||||
# Segment 0 is an implicit Tran record carried in the preamble.
|
||||
if preamble == MODE_DELTA:
|
||||
out["Tran"].extend([_i16(body, 3), _i16(body, 5)])
|
||||
run("Tran", 7, first, absolute=False)
|
||||
else:
|
||||
out["Tran"].extend(unpack12(body[3:first]))
|
||||
|
||||
for rec in walk_records(body, first):
|
||||
ch, off, mode, end = (rec["channel"], rec["offset"],
|
||||
rec["mode"], rec["end"])
|
||||
if mode == MODE_DELTA:
|
||||
out[ch].extend([_i16(body, off + 10), _i16(body, off + 12)])
|
||||
run(ch, off + 14, end, absolute=False)
|
||||
elif mode == MODE_ABSOLUTE:
|
||||
run(ch, off + 10, end, absolute=True)
|
||||
elif mode == MODE_RAW12:
|
||||
out[ch].extend(unpack12(body[off + 10:end]))
|
||||
return out
|
||||
|
||||
@@ -14,6 +14,7 @@ import pytest
|
||||
|
||||
from minimateplus.waveform_codec import (
|
||||
WaveformBlock,
|
||||
decode_waveform_legacy,
|
||||
decode_tran_initial,
|
||||
decode_waveform_v2,
|
||||
decoded_to_adc_counts,
|
||||
@@ -548,9 +549,18 @@ def test_walk_body_wide_rle_block():
|
||||
assert blocks[0].length == 2
|
||||
|
||||
|
||||
# NOTE (2026-08-25): the four tests below assert the SUPERSEDED tag-dispatch
|
||||
# model — `40 NN` as a variable-width segment header, tagless headers, channel
|
||||
# from rotation. The body format is really a chain of self-delimiting records
|
||||
# (see the record-chain section of waveform_codec.py), so `decode_waveform_v2`
|
||||
# no longer behaves this way. They are retargeted at `decode_waveform_legacy`,
|
||||
# which still implements the old model and is pinned by micromate/idf_file.py
|
||||
# for Thor IDFW bodies.
|
||||
|
||||
|
||||
def test_decode_wide_rle_repeats_full_run():
|
||||
"""A wide RLE run repeats the running value NN times, not NN & 0xFF."""
|
||||
decoded = decode_waveform_v2(_synth(b"\x01\x0c"))
|
||||
decoded = decode_waveform_legacy(_synth(b"\x01\x0c"))
|
||||
# 2 preamble anchors + 268 repeats
|
||||
assert len(decoded["Tran"]) == 2 + 268
|
||||
assert set(decoded["Tran"]) == {0}
|
||||
@@ -589,7 +599,7 @@ def test_segment_header_anchors_track_header_width(nn, hdr_len):
|
||||
data[2 * nn + 8 : 2 * nn + 10] = b"\x02\x00"
|
||||
data[2 * nn + 10 : 2 * nn + 12] = (7).to_bytes(2, "big") # anchor 0
|
||||
data[2 * nn + 12 : 2 * nn + 14] = (9).to_bytes(2, "big") # anchor 1
|
||||
decoded = decode_waveform_v2(_synth(bytes([0x40, nn]) + bytes(data)))
|
||||
decoded = decode_waveform_legacy(_synth(bytes([0x40, nn]) + bytes(data)))
|
||||
assert decoded["Vert"][:2] == [7, 9]
|
||||
|
||||
|
||||
@@ -629,7 +639,7 @@ def test_tagless_header_carries_full_14_bytes_as_data():
|
||||
|
||||
def test_tagless_header_anchors_and_channel_id():
|
||||
"""Anchors decode from data[10:14]; the channel comes from the id byte."""
|
||||
decoded = decode_waveform_v2(_synth(_tagless(chan_id=0x48, a0=11, a1=13)))
|
||||
decoded = decode_waveform_legacy(_synth(_tagless(chan_id=0x48, a0=11, a1=13)))
|
||||
assert decoded["Long"][:2] == [11, 13] # 0x48 → Long, not rotation
|
||||
assert decoded["Vert"] == []
|
||||
|
||||
@@ -642,10 +652,146 @@ def test_segment_channel_comes_from_id_not_rotation(chan_id, name):
|
||||
would put the second one on the next channel and corrupt both."""
|
||||
body = _synth(_tagless(chan_id=chan_id, seg=1, a0=5, a1=6),
|
||||
_tagless(chan_id=chan_id, seg=2, a0=7, a1=8))
|
||||
decoded = decode_waveform_v2(body)
|
||||
decoded = decode_waveform_legacy(body)
|
||||
# Tran additionally carries the body preamble's 2 anchors (both 0 here).
|
||||
expected = [0, 0, 5, 6, 7, 8] if name == "Tran" else [5, 6, 7, 8]
|
||||
assert decoded[name] == expected
|
||||
for other in ("Tran", "Vert", "Long", "MicL"):
|
||||
if other != name:
|
||||
assert decoded[other] == ([0, 0] if other == "Tran" else [])
|
||||
|
||||
|
||||
# ── Record-chain body model (2026-08-25) ────────────────────────────────────
|
||||
#
|
||||
# The body is a chain of self-delimiting per-channel records, not a flat
|
||||
# tag-dispatch stream. Verified over 1,388 production series-3 waveform
|
||||
# binaries: the chain terminates on a 0x06 record in 1,387 of them and all
|
||||
# four channels come out at identical length in 1,388/1,388 (was 156/1,388).
|
||||
# Against the 75 events with a preserved Blastware ASCII export: sample-count
|
||||
# exact 72/75 -> 75/75, fully exact 70/75 -> 73/75.
|
||||
|
||||
from minimateplus.waveform_codec import ( # noqa: E402
|
||||
CHANNEL_IDS,
|
||||
MODE_ABSOLUTE,
|
||||
MODE_DELTA,
|
||||
MODE_RAW12,
|
||||
STREAM_END_ID,
|
||||
data_block_len,
|
||||
find_first_record,
|
||||
is_record,
|
||||
unpack12,
|
||||
walk_records,
|
||||
)
|
||||
|
||||
|
||||
def _rec(chan_id, mode, payload, seg=0, field2=b"\x00\x00", anchors=None):
|
||||
"""Build one self-delimiting record."""
|
||||
head = bytearray()
|
||||
head += bytes([chan_id, 0x00, 0x00, seg])
|
||||
head += bytes(mode)
|
||||
if anchors is not None:
|
||||
for a in anchors:
|
||||
head += int(a).to_bytes(2, "big", signed=True)
|
||||
body = bytes(head) + payload
|
||||
return field2 + (len(body) + 2).to_bytes(2, "big") + body
|
||||
|
||||
|
||||
def _terminator():
|
||||
return b"\x00\x00" + (8).to_bytes(2, "big") + bytes([STREAM_END_ID, 0, 0, 0, 0, 0])
|
||||
|
||||
|
||||
def _body(*records, preamble=b"\x00\x02\x00", seg0=b"\x00\x00\x00\x00"):
|
||||
return preamble + seg0 + b"".join(records) + _terminator()
|
||||
|
||||
|
||||
def test_forty_nn_is_a_data_block_not_a_segment_header():
|
||||
"""`40 NN` is an int16 BE data block of length 2*NN + 2.
|
||||
|
||||
The superseded model read it as a segment header of length 2*NN + 16,
|
||||
which is what made walks drift and channels come out unequal.
|
||||
"""
|
||||
assert data_block_len(b"\x40\x02\x00\x01\x00\x02", 0) == (6, 2)
|
||||
assert data_block_len(b"\x40\x08" + bytes(16), 0) == (18, 8)
|
||||
# NN > 8 is not a data block
|
||||
assert data_block_len(b"\x40\x0c" + bytes(24), 0) == (None, None)
|
||||
|
||||
|
||||
def test_record_chain_is_followed_by_length_not_by_tag_sniffing():
|
||||
payload = b"\x00\x04" # RLE hold x4
|
||||
body = _body(_rec(0x47, MODE_DELTA, payload, anchors=(3, 5)))
|
||||
recs = walk_records(body)
|
||||
assert len(recs) == 1
|
||||
assert recs[0]["channel"] == "Vert"
|
||||
assert recs[0]["mode"] == MODE_DELTA
|
||||
|
||||
|
||||
def test_chain_terminates_on_channel_id_06():
|
||||
body = _body(_rec(0x47, MODE_DELTA, b"\x00\x04", anchors=(1, 1)),
|
||||
_rec(0x48, MODE_DELTA, b"\x00\x04", anchors=(2, 2)))
|
||||
assert [r["channel"] for r in walk_records(body)] == ["Vert", "Long"]
|
||||
assert not is_record(body, len(body) - 10) # the terminator is not a record
|
||||
|
||||
|
||||
def test_mode_delta_emits_anchors_then_accumulates():
|
||||
# two anchors, then an int8 block of +1,+1,+1,+1
|
||||
body = _body(_rec(0x47, MODE_DELTA, b"\x20\x04\x01\x01\x01\x01",
|
||||
anchors=(10, 11)))
|
||||
d = decode_waveform_v2(body)
|
||||
assert d["Vert"] == [10, 11, 12, 13, 14, 15]
|
||||
|
||||
|
||||
def test_mode_absolute_replaces_rather_than_accumulates():
|
||||
"""mode `01 00`: no anchors, and block values are ABSOLUTE samples."""
|
||||
body = _body(_rec(0x48, MODE_ABSOLUTE, b"\x20\x04\x0a\x0b\x0c\x0d"))
|
||||
d = decode_waveform_v2(body)
|
||||
assert d["Long"] == [10, 11, 12, 13], "01 00 blocks are absolute, not deltas"
|
||||
|
||||
|
||||
def test_mode_absolute_rle_holds_the_previous_value():
|
||||
body = _body(_rec(0x48, MODE_ABSOLUTE, b"\x20\x04\x07\x07\x07\x07\x00\x04"))
|
||||
d = decode_waveform_v2(body)
|
||||
assert d["Long"] == [7, 7, 7, 7, 7, 7, 7, 7]
|
||||
|
||||
|
||||
def test_mode_raw12_has_no_tags_at_all():
|
||||
"""mode `00 03`: the whole data section is raw 12-bit absolute samples.
|
||||
|
||||
Decoding these matters for the time base — skipping the record would
|
||||
displace every later sample on that channel (observed on
|
||||
BE9558/K558LOF2.820W, MicL shifted by exactly 512).
|
||||
"""
|
||||
packed = bytes([0x01, 0x23, 0x04, 0x05, 0x06, 0x07]) # 4 samples
|
||||
body = _body(_rec(0x49, MODE_RAW12, packed))
|
||||
d = decode_waveform_v2(body)
|
||||
assert d["MicL"] == unpack12(packed)
|
||||
assert len(d["MicL"]) == 4
|
||||
|
||||
|
||||
def test_unpack12_sign_extends():
|
||||
assert unpack12(bytes([0x00, 0x00, 0x01, 0x02, 0x03, 0x04])) == [1, 2, 3, 4]
|
||||
# high nibble 0x8 -> negative
|
||||
assert unpack12(bytes([0x80, 0x00, 0x00, 0x00, 0x00, 0x00]))[0] == -2048
|
||||
|
||||
|
||||
def test_channel_comes_from_the_record_id():
|
||||
for cid, name in CHANNEL_IDS.items():
|
||||
body = _body(_rec(cid, MODE_ABSOLUTE, b"\x20\x04\x01\x02\x03\x04"))
|
||||
d = decode_waveform_v2(body)
|
||||
assert d[name][-4:] == [1, 2, 3, 4], f"{name} misrouted"
|
||||
|
||||
|
||||
def test_raw12_preamble_is_scanned_not_block_walked():
|
||||
"""A `00 00 03` preamble carries raw 12-bit data from body[3] with no tags,
|
||||
so find_first_record must scan rather than block-walk. One production file
|
||||
has this (BE13121/O121L4L1.KF0W); block-walking returns None on it."""
|
||||
packed = bytes([0x00, 0x00, 0x01, 0x02, 0x03, 0x04])
|
||||
body = _body(_rec(0x47, MODE_DELTA, b"\x00\x04", anchors=(1, 1)),
|
||||
preamble=b"\x00\x00\x03", seg0=packed)
|
||||
assert find_first_record(body) == 3 + len(packed)
|
||||
d = decode_waveform_v2(body)
|
||||
assert d["Tran"] == unpack12(packed)
|
||||
|
||||
|
||||
def test_returns_none_when_no_record_chain():
|
||||
assert decode_waveform_v2(b"\x00\x02\x00" + bytes(40)) is None
|
||||
assert decode_waveform_v2(b"") is None
|
||||
|
||||
Reference in New Issue
Block a user