# IDF Protocol Reference — Thor / Micromate Series IV Starting-point reference for reverse-engineering Instantel's Micromate Series IV event-file format. Sibling to [instantel_protocol_reference.md](instantel_protocol_reference.md) (the Series III "Rosetta Stone") — this doc holds what we know so far and the open questions still to crack. > ⚠ **The "Status (2026-05-28)" block below is SUPERSEDED.** Its geo LSB > (0.0003), its IDFH scale (`/32768 × 10`), its fixed body offset (`0x0f1f`) > and its "87–99% byte-exact / loud events truncate" caveat were all wrong or > incomplete. See **[Verified against Thor's own exports > (2026-09-10)](#verified-against-thors-own-exports-2026-09-10)** — the > decoder is now per-sample exact on 1,057,536/1,057,536 samples. The block > is kept only for the reverse-engineering trail. **Status (2026-05-28, SUPERSEDED):** ASCII text sidecar fully decoded (1,014 sample files round-trip). **Thor IDFW** binary now decodes via `micromate.idf_file.read_idf_file()` — reuses the BW segment-rotated block codec verbatim at fixed body offset `0x0f1f`; metadata (serial, timestamp, sample_rate, record_time, calibration_date) extracted from the binary header. Sample fidelity is 87–99% byte-exact on quiet events; loud events hit the BW codec's known walker-stops-early limitation. Residual ~3% drift on per-sample deltas (likely a Thor-specific 12-bit delta refinement not yet modelled). **Thor IDFH histograms also decoded.** Body has one or more segments; each 12-byte segment header `[length_be 2B][0a 00 00 00][00 NN][05 3f]` introduces `N = (length - 10) // 72` interval records of 72 bytes each. Each interval = 4 × 16-byte per-channel records: `[int16 min][int16 max][int16 ??][uint16 halfp][2B 00][uint16 ??][2B 00][uint16 ??]`. Geo peak `= max(|min|, |max|) / 32768 × 10` in/s (matches sidecar ~1.8%); freq `= 512 / halfp` Hz (None for halfp ≤ 5 → ">100" sentinel). Corpus: **all 859 Thor IDFH files decode, 181,071 intervals**. Wired through `read_idf_file()` → `save_imported_idf()` → sidecar's `extensions.idf_intervals`. **Note on the BE9439 outliers in the example corpus:** Two files (`BE9439_20200713131747.IDFW` and `BE9439_20200713124251.IDFH`) are **Series III Blastware** binaries, not Thor. Provenance: TMI tried to use Thor to manage auto-call-homes for Series III units; the experiment didn't work out, but it did leave a few BW event files in Thor's per-serial directory structure with `.IDFW`/`.IDFH` extensions — Thor's forwarder applied its own naming convention to the BW bodies it was relaying. Their header `10 00 01 80 00 00 Instantel STRT ff fe ` is the BW SUB 5A STRT record, not a Thor body preamble. The reader detects them by signature and raises `NotImplementedError` pointing callers at `read_blastware_file()`, which extracts BW-format peaks from them. **Still NYI for Thor IDFH:** per-channel `int16 field4` (possibly time-of-peak); the two uint16 fields (probably PVS contributions); 8-byte interval tail (PVS data); mic dB(L) exact conversion constant. ## Verified against Thor's own exports (2026-09-10) **The series-4 decoder is now per-sample exact.** 1,057,536 / 1,057,536 geophone samples across all 153 genuine Thor waveform files reproduce Thor's own CSV export exactly; histogram peaks land within 2% on 858/858 files (median error −0.004%). ### Ground truth — it was there all along Thor writes `TXT/`, `CSV/`, `XML/` and `PDF/` exports beside every binary: ``` /UM13981_20220207084555.IDFW /CSV/UM13981_20220207084555.IDFW.csv ``` The **CSV carries a per-sample block** — four columns (Tran, Vert, Long, Mic) in in/s and psi, after the 2-column report header. That is the series-4 equivalent of Blastware's `_ASCII.TXT` exports, and it gives 1,012 paired files (152 IDFW + 860 IDFH). Earlier notes in this file and in `micromate/idf_file.py` asserted "Thor has no ASCII ground truth in the corpus"; that was wrong, and it is why the decoder sat pinned to a superseded walker with a scaling constant nobody could check. Harness: `scratch/verify_thor_against_csv.py`. ### Geo LSB = 0.000310308 in/s per count (NOT 0.0003) The old 0.0003 was read off the smallest non-zero sample in the exports — but that is Thor's **4-decimal display rounding of the LSB, not the LSB**. It read every series-4 geophone sample **3.3% low**. The quantisation ladder gives it away: counts 1..6 export as 0.0003, 0.0006, 0.0009, 0.0012, 0.0016, 0.0019 — an LSB of exactly 0.0003 would end 0.0015, 0.0018. Each exported sample constrains the LSB to the window that rounds to its printed value. Intersecting 991,415 such constraints gives ``` LSB ∈ [0.000310307933, 0.000310308057] width 1.2e-10 ``` so `_GEO_LSB_IPS = 0.000310308`, i.e. full scale 10.0 in/s = **32226.05 counts**. Corroboration: an IDFH interval that never recorded keeps its min/max accumulator at its ±full-scale seed, and that seed is `(min=+32226, max=-32226)`. ⚠ The tempting closed form `10.0/32226` is very slightly wrong — it lands 4.5e-10 above the feasible window and loses 78 boundary samples while never winning one. **Series III uses 32000 counts for the same 10.0 in/s, so the two generations do not share a scale.** Independently confirmed on 8 production units (UM6047, UM11402, UM11719, UM12947, UM13981, UM14133, UM20146, UM20147): every unit's median PPV error against its device-reported peak moved from −3.3% to within ±0.03%. It is a global constant, not a per-unit calibration. ### IDFH segment header: the counter is a uint16, and it is cumulative ``` [length_be 2B][0a 00 00 00][counter_be 2B][05 3f] ``` `counter` is the **0-based cumulative index of the last interval in the segment** — 9, 19, 29, ... for the usual 10-intervals-per-segment layout (`length` = 730). The validator used to require `counter`'s high byte to be `0x00`. That silently **capped every histogram at 250 intervals**: once the cumulative counter passed 255 the high byte went non-zero and every later segment was rejected. Any run longer than ~4 hours lost its tail — frequently the part holding the event peak, so the file's PPV read low. **540 of 858 corpus files were affected**; fixing it moved histogram peaks from 48.3% to 93.8% within 0.5% of Thor's reported PPV. ### Unwritten interval slots carry a ±full-scale seed An interval the device reserved but never wrote keeps `min = +32226`, `max = -32226` on all four channels — `min > max`, impossible for real data. Decoded naively it yields a 10.0 in/s peak on every channel and, being a max-over-intervals, poisons the whole file's PPV. Rare but real: exactly 1 of 497,611 corpus intervals, and it inflated that file's Long PPV from 0.0081 to 10.0 in/s. The inversion is all-or-nothing across channels (0 partial cases), so requiring every channel to be inverted is a safe test. ### Record mode `00 00` — raw int16 absolute (MODE_RAW16) The record chain's mode field at `off+8` takes a fourth value: | mode | meaning | header | |---|---|---| | `02 00` | deltas + two int16 anchors | 14 B | | `01 00` | absolute, tagged blocks | 10 B | | `00 03` | raw 12-bit absolute, untagged | 10 B | | **`00 00`** | **raw int16 BE absolute, untagged** | **10 B** | A `MODE_RAW16` record with `length = 1032` carries exactly `(1032 - 8) / 2 = 512` samples and reproduced Thor's export **512/512 exactly** on first test. Thor uses it for segment 0 (the pre-trigger window) on some events. Before this mode existed the record fell through the dispatch unhandled, so the channel silently lost its first 512 samples — which is what produced the "loud events truncate" symptom. `MODE_ABSOLUTE` is also valid as a **preamble** (the implicit segment-0 Tran record); its tagged blocks start at `body[3]`, not `body[7]`, because its header is 10 bytes rather than 14. ### Body offset is not fixed at 0x0f1f — and 0x0f1f is really a record + 7 A "body offset" is ` + 7`, so that `body[0]` is the segment index and `body[1:3]` is the mode. The canonical `0x0f1f` is simply the record at `0x0f18`. Searching for the literal preamble `00 02 00` finds only MODE_DELTA bodies, and worse, it **matches the `[seg][mode]` bytes inside any record header**, so the scan could pick a candidate part-way down the chain. That decodes a plausible-looking but rotation-shifted body which drops each channel's segment 0 — the real cause of the remaining truncations. `_find_waveform_body_offset()` now anchors on record headers (the ` 00 00` signature at `+4`, validated with `is_record()`), takes the **chain head** — a record no other record's length field points at — and trial-decodes `head + 7`, preferring the candidate where all four channels come out the same length. ⚠ Do **not** scan for candidate preambles instead: `MODE_RAW16` is `00 00`, so every run of three zero bytes looks like a body start and each costs a full trial decode (~0.5 s/file measured, vs 6 ms/file now). ### Mic-disabled units are a distinct shape (2026-09-10, second corpus) Some units run with the microphone disabled — **3 channels, not 4** — and that changes two structural things. Confirmed on the `9-10-26-csv-req` corpus (UM11402, UM12947, UM20147): 139/139 waveforms and 877/877 histograms. **Waveform: the body starts earlier.** A 3-channel unit has a shorter fixed header and puts its record chain head at **`0x0dba`**, below the old `_BODY_SCAN_FLOOR` of `0x0E00`. The head was therefore invisible to the scan, which fell through to the *Vert* segment-0 record and decoded a body shifted one position around the channel rotation. The signature is unmistakable: ``` Tran 3072 / Vert 2560 / Long 3072 / MicL 0 <- Vert exactly 512 short ``` 46 of 139 files in that corpus were affected; all 46 became per-sample exact once the floor dropped to `0x0C00`. Note the body-offset scoring also had to stop requiring four channels — `len(lengths) >= 3`, not `== 4`, or `equal` is permanently False for these events and the pick falls back to raw sample count. **Histogram: the interval record is 56 bytes, not 72.** ``` interval_size = 16 × n_channels + 8 (72 for 4 channels, 56 for 3) ``` It is **not a constant**, and it cannot be inferred from `length` alone. Derive the interval count from the segment counter — it is cumulative, so `n = counter - previous_counter` — and then `stride = (length - 10) / n`. `n_channels` follows from `(stride - 8) / 16`. Assuming 72 read 7 intervals out of each 10-interval segment and then walked off alignment into garbage that decoded as ~10 in/s peaks — inflating those files' PPV by up to 191,000%. Fixing it moved the second corpus from 56.6% to **100.0%** of histograms within 2% of Thor's reported PPV, and recovered 4 files that previously decoded no intervals at all. ### What is still open - **23 of 575 production IDFW files (4%)** still decode with unequal channel lengths (22) or fail outright (1). All are **UM12947, 2025-07-14 to 2025-09-23**. The `Tran/Vert/Long 3072 / MicL 0` group was the 3-channel shape above and is fixed; what remains is ragged, e.g. `T1812 / V2132 / L2324 / M2324`. **Diagnosed but NOT verified.** These files stop the block walker on tag `40 0c`, because `data_block_len()` caps the `40 NN` int16 block at `NN > 0x08`: ```python if hi == 0x40: # int16 BE data block return (None, None) if (nn == 0 or nn > 0x08) else (2 * nn + 2, nn) ``` The affected files use NN of 12, 16, 20 … up to 196 — every value above 8 halts the walk, which is why the channel comes up short. Both verified corpora only ever use NN ∈ {1, 2, 3, 4, 8}, so the cap has never been exercised, and lifting it leaves both at 100.000%. ⚠ **Do not ship the cap change on that evidence.** "Doesn't regress the known-good corpus" is not "produces correct values here", and a too-loose guard can accept a false `40 NN` inside data and emit plausible-but-wrong samples — the exact failure mode this whole effort was about. It needs Thor CSV exports for UM12947 events between **2025-07-14 and 2025-09-23**; the `9-10-26-csv-req` upload starts at 2025-09-25 and misses them. - Mic → psi scale is still the rough `2.14e-6` regression, not derived. - Per-channel `int16 field4` in the IDFH interval record (possibly time-of-peak) and the 8-byte tail (PVS data) remain undecoded. ⚠ **Thor's histogram PPV has a display floor of 0.0050 in/s.** In the production store 6,080 sidecar PPV values are exactly 0.0050 (next most common value: 275 occurrences), and **41.4% of IDFH sidecars report a component PPV larger than their own vector sum** — geometrically impossible. On those quiet files the decoder's ~0.0025 in/s is *more* accurate than the reference; do not "fix" the decoder to match it. ### Codec breakthroughs (2026-05-28) - **Body offset is a fixed `0x0f1f`** across 151/154 corpus IDFW files. Preceded by a 4-byte record-type marker (`46 00 00 00`) + magic preamble `00 02 00 [Tran[0] BE] [Tran[1] BE]`. - **Sample stream is BW's segment-rotated block codec verbatim.** Thor reuses `10 NN` (nibble), `20 NN` (int8), `00 NN` (RLE), `30 NN` (packed12), `40 02` (segment header) tags with the same semantics. Channel rotation Tran→Vert→Long→MicL. - **Geo LSB = 0.0003 in/s** (not BW's 0.005), because Thor's 16-bit ADC range maps to 10 in/s without the 16-count BW quantization step. - **Mic ≈ 2.14×10⁻⁶ psi/count** (rough scale; refine after channel block calibration constants are decoded). - **BW compliance anchor `\xbe\x80\x00\x00\x00\x00` reappears at IDFW offset 0x952** — sample_rate at anchor−6 (uint16 BE), record_time at anchor+6 (float32 BE), same layout as BW. - **Event timestamp at offset 0x97A** — 8 bytes `[day][month] [year_be][unk][hour][min][sec]`. Stop-time mirrors at 0x982. - **Serial as null-terminated ASCII at 0x14E**. - **Calibration date** at 0x194–0x197 (day, month, year_be). - Per-sample residual drift of ~3% suggests Thor encodes int8/nibble deltas with an extra refinement bit that BW doesn't carry — unsolved; errors resync within a few samples so cumulative impact is small. --- ## File model ### Filename convention ``` _. ``` - **SERIAL** — literal device serial, two-letter prefix + numeric suffix. Examples seen: `UM11719`, `UM13981`, `UM20147`, `BE9439`. Unlike Series III BW filenames (`M529LK44.AB0`, base-36 stem), Series IV filenames carry the serial in plain text. - **YYYYMMDDHHMMSS** — 14-char ASCII timestamp in **device local time** (no timezone marker). - **KIND** — `IDFH` for histograms, `IDFW` for waveforms. The `.IDFH.txt` / `.IDFW.txt` ASCII sidecar lives in a `TXT/` **subfolder** of the unit's directory, not alongside the binary. This pairing convention is encoded in `event_forwarder.idf_report_path()`. ### Directory layout ``` C:\THORDATA\ └── \ └── \ ← unit serial dir ├── UM12345_20260520100000.MLG ← monitor log (not events) ├── UM12345_20260520100000.IDFH ← histogram event (binary) ├── UM12345_20260520100000.IDFW ← waveform event (binary) ├── UM12345_20260520100000.IDFW.CDB ← cache-DB variant (skip) ├── TXT\ │ ├── UM12345_20260520100000.IDFH.txt ← histogram ASCII sidecar │ └── UM12345_20260520100000.IDFW.txt ← waveform ASCII sidecar ├── CSV\, HTML\, PDF\, XML\ ← operator-facing derived exports └── ... ``` The `.IDFW.CDB` files share the binary's basename but appear to be a separate cache/database variant. Their first 8 bytes match the **old**-firmware Thor signature (see below) regardless of which signature the paired `.IDFW` uses. Purpose unknown; sizes vary wildly (observed 123 B → 40,491 B). Thor-watcher's forwarder deliberately skips them. ### Sample corpus The `thor-watcher/example-data/THORDATA_example/` tree carries **1,014 paired .IDFW / .IDFH + .txt files** spanning 2020–2023 across nine units (UM11719, UM13981, UM20147, …, plus BE9439 from 2020). This is the reverse-engineering ground truth. --- ## ASCII sidecar (`.IDFW.txt` / `.IDFH.txt`) — fully decoded Shape: plain text, one `"Key : Value"` line per metadata field, followed for waveforms by a tab-separated sample table headed by the literal line `Waveform Data Channels`. Parsed by [`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py). See [`micromate/models.py`](../micromate/models.py) for the typed `IdfReport` shape. ### Notable conventions - **Units are native to Thor** — geophone in **in/s**, microphone in **dB(L)** (not psi like Series III BW reports), frequency in Hz, acceleration in g, displacement in in. - **Below-threshold readings** appear as the literal string `<0.005 in/s` (155 occurrences in the sample corpus) — the parser strips the `<` and treats the numeric remainder as the value. - **Out-of-range / not-measured** values appear as `N/A` — parser drops the field rather than letting the string leak into a numeric column. - **Firmware string** observed: `Micromate ISEE 11.0AK`. - **TitleString1..4** are operator-defined free-text slots; Thor's default labels map them to Location / Client / Company / Notes, which the parser surfaces as `project` / `client` / `operator` / `notes`. - **Histogram sidecars** use `HistogramStartDate` / `HistogramStartTime` in place of waveform's `EventDate` / `EventTime`. Parser falls through to either. - **Histogram tabular block** lacks the `Waveform Data Channels` marker; instead it's a multi-line column header followed by per-interval rows (`