seismo-relay/docs/idf_protocol_reference.md

IDF Protocol Reference — Thor / Micromate Series IV

Starting-point reference for reverse-engineering Instantel's Micromate Series IV event-file format. Sibling to instantel_protocol_reference.md (the Series III "Rosetta Stone") — this doc holds what we know so far and the open questions still to crack.

⚠ The "Status (2026-05-28)" block below is SUPERSEDED. Its geo LSB (0.0003), its IDFH scale (/32768 × 10), its fixed body offset (0x0f1f) and its "87–99% byte-exact / loud events truncate" caveat were all wrong or incomplete. See Verified against Thor's own exports (2026-09-10) — the decoder is now per-sample exact on 1,057,536/1,057,536 samples. The block is kept only for the reverse-engineering trail.

Status (2026-05-28, SUPERSEDED): ASCII text sidecar fully decoded (1,014 sample files round-trip). Thor IDFW binary now decodes via micromate.idf_file.read_idf_file() — reuses the BW segment-rotated block codec verbatim at fixed body offset 0x0f1f; metadata (serial, timestamp, sample_rate, record_time, calibration_date) extracted from the binary header. Sample fidelity is 87–99% byte-exact on quiet events; loud events hit the BW codec's known walker-stops-early limitation. Residual ~3% drift on per-sample deltas (likely a Thor-specific 12-bit delta refinement not yet modelled).

Thor IDFH histograms also decoded. Body has one or more segments; each 12-byte segment header [length_be 2B][0a 00 00 00][00 NN][05 3f] introduces N = (length - 10) // 72 interval records of 72 bytes each. Each interval = 4 × 16-byte per-channel records: [int16 min][int16 max][int16 ??][uint16 halfp][2B 00][uint16 ??][2B 00][uint16 ??]. Geo peak = max(|min|, |max|) / 32768 × 10 in/s (matches sidecar ~1.8%); freq = 512 / halfp Hz (None for halfp ≤ 5 → ">100" sentinel). Corpus: all 859 Thor IDFH files decode, 181,071 intervals. Wired through read_idf_file() → save_imported_idf() → sidecar's extensions.idf_intervals.

Note on the BE9439 outliers in the example corpus: Two files (BE9439_20200713131747.IDFW and BE9439_20200713124251.IDFH) are Series III Blastware binaries, not Thor. Provenance: TMI tried to use Thor to manage auto-call-homes for Series III units; the experiment didn't work out, but it did leave a few BW event files in Thor's per-serial directory structure with .IDFW/.IDFH extensions — Thor's forwarder applied its own naming convention to the BW bodies it was relaying. Their header 10 00 01 80 00 00 Instantel STRT ff fe <end_key> <start_key> is the BW SUB 5A STRT record, not a Thor body preamble. The reader detects them by signature and raises NotImplementedError pointing callers at read_blastware_file(), which extracts BW-format peaks from them.

Still NYI for Thor IDFH: per-channel int16 field4 (possibly time-of-peak); the two uint16 fields (probably PVS contributions); 8-byte interval tail (PVS data); mic dB(L) exact conversion constant.

Verified against Thor's own exports (2026-09-10)

The series-4 decoder is now per-sample exact. 1,057,536 / 1,057,536 geophone samples across all 153 genuine Thor waveform files reproduce Thor's own CSV export exactly; histogram peaks land within 2% on 858/858 files (median error −0.004%).

Ground truth — it was there all along

Thor writes TXT/, CSV/, XML/ and PDF/ exports beside every binary:

<serial dir>/UM13981_20220207084555.IDFW
<serial dir>/CSV/UM13981_20220207084555.IDFW.csv

The CSV carries a per-sample block — four columns (Tran, Vert, Long, Mic) in in/s and psi, after the 2-column report header. That is the series-4 equivalent of Blastware's _ASCII.TXT exports, and it gives 1,012 paired files (152 IDFW + 860 IDFH). Earlier notes in this file and in micromate/idf_file.py asserted "Thor has no ASCII ground truth in the corpus"; that was wrong, and it is why the decoder sat pinned to a superseded walker with a scaling constant nobody could check.

Harness: scratch/verify_thor_against_csv.py.

Geo LSB = 0.000310308 in/s per count (NOT 0.0003)

The old 0.0003 was read off the smallest non-zero sample in the exports — but that is Thor's 4-decimal display rounding of the LSB, not the LSB. It read every series-4 geophone sample 3.3% low. The quantisation ladder gives it away: counts 1..6 export as 0.0003, 0.0006, 0.0009, 0.0012, 0.0016, 0.0019 — an LSB of exactly 0.0003 would end 0.0015, 0.0018.

Each exported sample constrains the LSB to the window that rounds to its printed value. Intersecting 991,415 such constraints gives

LSB ∈ [0.000310307933, 0.000310308057]      width 1.2e-10

so _GEO_LSB_IPS = 0.000310308, i.e. full scale 10.0 in/s = 32226.05 counts. Corroboration: an IDFH interval that never recorded keeps its min/max accumulator at its ±full-scale seed, and that seed is (min=+32226, max=-32226). ⚠ The tempting closed form 10.0/32226 is very slightly wrong — it lands 4.5e-10 above the feasible window and loses 78 boundary samples while never winning one. Series III uses 32000 counts for the same 10.0 in/s, so the two generations do not share a scale.

Independently confirmed on 8 production units (UM6047, UM11402, UM11719, UM12947, UM13981, UM14133, UM20146, UM20147): every unit's median PPV error against its device-reported peak moved from −3.3% to within ±0.03%. It is a global constant, not a per-unit calibration.

IDFH segment header: the counter is a uint16, and it is cumulative

[length_be 2B][0a 00 00 00][counter_be 2B][05 3f]

counter is the 0-based cumulative index of the last interval in the segment — 9, 19, 29, ... for the usual 10-intervals-per-segment layout (length = 730).

The validator used to require counter's high byte to be 0x00. That silently capped every histogram at 250 intervals: once the cumulative counter passed 255 the high byte went non-zero and every later segment was rejected. Any run longer than ~4 hours lost its tail — frequently the part holding the event peak, so the file's PPV read low. 540 of 858 corpus files were affected; fixing it moved histogram peaks from 48.3% to 93.8% within 0.5% of Thor's reported PPV.

Unwritten interval slots carry a ±full-scale seed

An interval the device reserved but never wrote keeps min = +32226, max = -32226 on all four channels — min > max, impossible for real data. Decoded naively it yields a 10.0 in/s peak on every channel and, being a max-over-intervals, poisons the whole file's PPV. Rare but real: exactly 1 of 497,611 corpus intervals, and it inflated that file's Long PPV from 0.0081 to 10.0 in/s. The inversion is all-or-nothing across channels (0 partial cases), so requiring every channel to be inverted is a safe test.

Record mode 00 00 — raw int16 absolute (MODE_RAW16)

The record chain's mode field at off+8 takes a fourth value:

mode	meaning	header
02 00	deltas + two int16 anchors	14 B
01 00	absolute, tagged blocks	10 B
00 03	raw 12-bit absolute, untagged	10 B
00 00	raw int16 BE absolute, untagged	10 B

A MODE_RAW16 record with length = 1032 carries exactly (1032 - 8) / 2 = 512 samples and reproduced Thor's export 512/512 exactly on first test. Thor uses it for segment 0 (the pre-trigger window) on some events. Before this mode existed the record fell through the dispatch unhandled, so the channel silently lost its first 512 samples — which is what produced the "loud events truncate" symptom.

MODE_ABSOLUTE is also valid as a preamble (the implicit segment-0 Tran record); its tagged blocks start at body[3], not body[7], because its header is 10 bytes rather than 14.

Body offset is not fixed at 0x0f1f — and 0x0f1f is really a record + 7

A "body offset" is <record start> + 7, so that body[0] is the segment index and body[1:3] is the mode. The canonical 0x0f1f is simply the record at 0x0f18.

Searching for the literal preamble 00 02 00 finds only MODE_DELTA bodies, and worse, it matches the [seg][mode] bytes inside any record header, so the scan could pick a candidate part-way down the chain. That decodes a plausible-looking but rotation-shifted body which drops each channel's segment 0 — the real cause of the remaining truncations.

_find_waveform_body_offset() now anchors on record headers (the <channel_id> 00 00 signature at +4, validated with is_record()), takes the chain head — a record no other record's length field points at — and trial-decodes head + 7, preferring the candidate where all four channels come out the same length.

⚠ Do not scan for candidate preambles instead: MODE_RAW16 is 00 00, so every run of three zero bytes looks like a body start and each costs a full trial decode (~0.5 s/file measured, vs 6 ms/file now).

Mic-disabled units are a distinct shape (2026-09-10, second corpus)

Some units run with the microphone disabled — 3 channels, not 4 — and that changes two structural things. Confirmed on the 9-10-26-csv-req corpus (UM11402, UM12947, UM20147): 139/139 waveforms and 877/877 histograms.

Waveform: the body starts earlier. A 3-channel unit has a shorter fixed header and puts its record chain head at 0x0dba, below the old _BODY_SCAN_FLOOR of 0x0E00. The head was therefore invisible to the scan, which fell through to the Vert segment-0 record and decoded a body shifted one position around the channel rotation. The signature is unmistakable:

Tran 3072 / Vert 2560 / Long 3072 / MicL 0      <- Vert exactly 512 short

46 of 139 files in that corpus were affected; all 46 became per-sample exact once the floor dropped to 0x0C00. Note the body-offset scoring also had to stop requiring four channels — len(lengths) >= 3, not == 4, or equal is permanently False for these events and the pick falls back to raw sample count.

Histogram: the interval record is 56 bytes, not 72.

interval_size = 16 × n_channels + 8        (72 for 4 channels, 56 for 3)

It is not a constant, and it cannot be inferred from length alone. Derive the interval count from the segment counter — it is cumulative, so n = counter - previous_counter — and then stride = (length - 10) / n. n_channels follows from (stride - 8) / 16.

Assuming 72 read 7 intervals out of each 10-interval segment and then walked off alignment into garbage that decoded as ~10 in/s peaks — inflating those files' PPV by up to 191,000%. Fixing it moved the second corpus from 56.6% to 100.0% of histograms within 2% of Thor's reported PPV, and recovered 4 files that previously decoded no intervals at all.

What is still open

• 23 of 575 production IDFW files (4%) still decode with unequal channel lengths (22) or fail outright (1). All are UM12947, 2025-07-14 to 2025-09-23. The Tran/Vert/Long 3072 / MicL 0 group was the 3-channel shape above and is fixed; what remains is ragged, e.g. T1812 / V2132 / L2324 / M2324.

  Diagnosed but NOT verified. These files stop the block walker on tag 40 0c, because data_block_len() caps the 40 NN int16 block at NN > 0x08:

  if hi == 0x40:                     # int16 BE data block
      return (None, None) if (nn == 0 or nn > 0x08) else (2 * nn + 2, nn)
  

  The affected files use NN of 12, 16, 20 … up to 196 — every value above 8 halts the walk, which is why the channel comes up short. Both verified corpora only ever use NN ∈ {1, 2, 3, 4, 8}, so the cap has never been exercised, and lifting it leaves both at 100.000%.

  ⚠ Do not ship the cap change on that evidence. "Doesn't regress the known-good corpus" is not "produces correct values here", and a too-loose guard can accept a false 40 NN inside data and emit plausible-but-wrong samples — the exact failure mode this whole effort was about. It needs Thor CSV exports for UM12947 events between 2025-07-14 and 2025-09-23; the 9-10-26-csv-req upload starts at 2025-09-25 and misses them.

• Mic → psi scale is still the rough 2.14e-6 regression, not derived.

• Per-channel int16 field4 in the IDFH interval record (possibly time-of-peak) and the 8-byte tail (PVS data) remain undecoded.

⚠ Thor's histogram PPV has a display floor of 0.0050 in/s. In the production store 6,080 sidecar PPV values are exactly 0.0050 (next most common value: 275 occurrences), and 41.4% of IDFH sidecars report a component PPV larger than their own vector sum — geometrically impossible. On those quiet files the decoder's ~0.0025 in/s is more accurate than the reference; do not "fix" the decoder to match it.

Codec breakthroughs (2026-05-28)

• Body offset is a fixed 0x0f1f across 151/154 corpus IDFW files. Preceded by a 4-byte record-type marker (46 00 00 00)
  • magic preamble 00 02 00 [Tran[0] BE] [Tran[1] BE].
• Sample stream is BW's segment-rotated block codec verbatim. Thor reuses 10 NN (nibble), 20 NN (int8), 00 NN (RLE), 30 NN (packed12), 40 02 (segment header) tags with the same semantics. Channel rotation Tran→Vert→Long→MicL.
• Geo LSB = 0.0003 in/s (not BW's 0.005), because Thor's 16-bit ADC range maps to 10 in/s without the 16-count BW quantization step.
• Mic ≈ 2.14×10⁻⁶ psi/count (rough scale; refine after channel block calibration constants are decoded).
• BW compliance anchor \xbe\x80\x00\x00\x00\x00 reappears at IDFW offset 0x952 — sample_rate at anchor−6 (uint16 BE), record_time at anchor+6 (float32 BE), same layout as BW.
• Event timestamp at offset 0x97A — 8 bytes [day][month] [year_be][unk][hour][min][sec]. Stop-time mirrors at 0x982.
• Serial as null-terminated ASCII at 0x14E.
• Calibration date at 0x194–0x197 (day, month, year_be).
• Per-sample residual drift of ~3% suggests Thor encodes int8/nibble deltas with an extra refinement bit that BW doesn't carry — unsolved; errors resync within a few samples so cumulative impact is small.

---

File model

Filename convention

<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>

• SERIAL — literal device serial, two-letter prefix + numeric suffix. Examples seen: UM11719, UM13981, UM20147, BE9439. Unlike Series III BW filenames (M529LK44.AB0, base-36 stem), Series IV filenames carry the serial in plain text.
• YYYYMMDDHHMMSS — 14-char ASCII timestamp in device local time (no timezone marker).
• KIND — IDFH for histograms, IDFW for waveforms.

The .IDFH.txt / .IDFW.txt ASCII sidecar lives in a TXT/ subfolder of the unit's directory, not alongside the binary. This pairing convention is encoded in event_forwarder.idf_report_path().

Directory layout

C:\THORDATA\
└── <Project>\
    └── <UM####>\                  ← unit serial dir
        ├── UM12345_20260520100000.MLG     ← monitor log (not events)
        ├── UM12345_20260520100000.IDFH    ← histogram event (binary)
        ├── UM12345_20260520100000.IDFW    ← waveform event (binary)
        ├── UM12345_20260520100000.IDFW.CDB ← cache-DB variant (skip)
        ├── TXT\
        │   ├── UM12345_20260520100000.IDFH.txt    ← histogram ASCII sidecar
        │   └── UM12345_20260520100000.IDFW.txt    ← waveform  ASCII sidecar
        ├── CSV\, HTML\, PDF\, XML\        ← operator-facing derived exports
        └── ...

The .IDFW.CDB files share the binary's basename but appear to be a separate cache/database variant. Their first 8 bytes match the old-firmware Thor signature (see below) regardless of which signature the paired .IDFW uses. Purpose unknown; sizes vary wildly (observed 123 B → 40,491 B). Thor-watcher's forwarder deliberately skips them.

Sample corpus

The thor-watcher/example-data/THORDATA_example/ tree carries 1,014 paired .IDFW / .IDFH + .txt files spanning 2020–2023 across nine units (UM11719, UM13981, UM20147, …, plus BE9439 from 2020). This is the reverse-engineering ground truth.

---

ASCII sidecar (.IDFW.txt / .IDFH.txt) — fully decoded

Shape: plain text, one "Key : Value" line per metadata field, followed for waveforms by a tab-separated sample table headed by the literal line Waveform Data Channels. Parsed by micromate/idf_ascii_report.py. See micromate/models.py for the typed IdfReport shape.

Notable conventions

• Units are native to Thor — geophone in in/s, microphone in dB(L) (not psi like Series III BW reports), frequency in Hz, acceleration in g, displacement in in.
• Below-threshold readings appear as the literal string <0.005 in/s (155 occurrences in the sample corpus) — the parser strips the < and treats the numeric remainder as the value.
• Out-of-range / not-measured values appear as N/A — parser drops the field rather than letting the string leak into a numeric column.
• Firmware string observed: Micromate ISEE 11.0AK.
• TitleString1..4 are operator-defined free-text slots; Thor's default labels map them to Location / Client / Company / Notes, which the parser surfaces as project / client / operator / notes.
• Histogram sidecars use HistogramStartDate / HistogramStartTime in place of waveform's EventDate / EventTime. Parser falls through to either.
• Histogram tabular block lacks the Waveform Data Channels marker; instead it's a multi-line column header followed by per-interval rows (<date> <time> <tran-ppv> <freq> ...). Parser silently ignores lines after the metadata block since they lack a colon-separated key : value shape (the timestamps DO contain colons but produce garbage keys that don't collide with any recognised field).

---

Binary header signatures (observed)

Hex dump of the first 32 bytes across 1,014 sample files reveals two distinct file signatures, both anchored by the literal ASCII string "\x00Instantel\x00" at offset 6–16:

Signature A — newer firmware (1,012 files, 99.8% of corpus)

00000000: 0012 0100 0000 496e 7374 616e 7465 6c00   ......Instantel.
00000010: 0000 a695 002e b500 4f70 6572 6174 6f72   ........Operator
                                ^^^^^^^^^^^^^^^^
                                operator/title string starts at 0x18

Header bytes 0–5: 00 12 01 00 00 00. Followed immediately by the 8-byte ASCII tag, then 6 unknown bytes, then ASCII operator-supplied strings (Operator name, etc.) and on through the project / client / title strings. No STRT record observed in this layout.

Signature B — older firmware (2 files: BE9439 from 2020)

00000000: 1000 0180 0000 496e 7374 616e 7465 6c00   ......Instantel.
00000010: 072c 0012 0300 5354 5254 fffe 0111 2340   .,....STRT....#@
                          ^^^^^^^^^                ^^^^^^^^^
                          STRT magic               4-byte end_key
00000020: 0111 0000 2e5f 00ac 4600 0000 0200 0000   ....._..F.......
          ^^^^^^^^^             ^^^
          4-byte start_key      0x46 (BW WAVEHDR record-type marker)

Header bytes 0–5: 10 00 01 80 00 00. The structure after the Instantel magic is byte-for-byte identical to a BW SUB 5A probe-response STRT record as documented in instantel_protocol_reference.md → "SUB 5A — STRT record encodes end_offset". Specifically:

Offset	Bytes	Meaning (per BW reference)
0x14	53 54 52 54	STRT magic
0x18	ff fe	STRT sentinel
0x1A	01 11 23 40	end_key (4 bytes)
0x1E	01 11 00 00	start_key (4 bytes)
0x26	46	0x46 waveform-record type marker

Hypothesis: Older Micromate firmware writes a wrapped BW-format event into the .IDFW file — essentially the same on-disk shape as a Series III device, with the new filename convention applied at export time. Newer firmware (signature A) abandoned the BW-compatible layout for an Instantel-specific format.

If that hypothesis holds, the 2 signature-B files can already be parsed via minimateplus/event_file_io.read_blastware_file() — worth testing. The 1,012 signature-A files are the real reverse-engineering target.

.IDFW.CDB cache files

Always carry signature B (10 00 01 80 ...), even when the paired .IDFW carries signature A. Plausible explanation: the CDB is an internal Thor cache-database export that retains the legacy BW-style record layout regardless of the user-facing .IDFW format version. Not currently consumed by the forwarder.

---

File-size patterns (Signature A, the main target)

Survey of 1,012 signature-A files:

Event type	Typical size	Source of variance
.IDFW 2-sec	9,200 – 10,500 B	Operator-supplied strings (TitleString1..4) of varying length
.IDFH	2,944 – 4,076 B	Histogram interval count (record duration / interval)

Naive arithmetic for 2-sec waveform:

• 4 channels × 2 sec × 1024 sps = 8,192 samples
• At 2 bytes/sample (int16) = 16,384 sample bytes → file would be > 16 KB
• Observed: ~9–10 KB
• → samples are likely 1 byte each (int8 quantised), or stored with bit-packing / delta encoding, or only one channel's full-rate samples are stored with the others reconstructed arithmetically. Verifying this is the first RE milestone.

Project-string–length variance (~1 KB across the corpus) is consistent with the file carrying a single copy of each TitleString1..4 plus operator + setup-name as null-padded ASCII regions.

---

Open questions

The reverse-engineering targets, roughly in dependency order:

1. Sample encoding (signature A) — int8? int16 LE/BE? Bit-packed? Delta-coded? Per-channel interleaved or sequential blocks?
2. Header field layout (signature A) — where do sample_rate, record_time, channel count, and per-channel peaks live in the binary? The ASCII sidecar gives the device-authoritative values, so binary fields can be confirmed by diff.
3. Operator-string offsets — Operator at 0x18 is the first visible string in signature-A files; the rest (project, client, notes, setup) follow. Need to map exact offsets and null-padding conventions.
4. Signature-B → BW codec compatibility — does minimateplus/event_file_io.read_blastware_file() actually parse the 2 BE9439 signature-B files as-is? If yes, the OLD-format ingest is free.
5. .IDFW.CDB purpose — is it an internal Thor cache, a ring-buffer dump, or something else? Worth a single small effort to characterise so we know what we're skipping.
6. Footer / checksum — every BW event file has a footer; does IDF? Where does the per-channel sample block end?

---

Reverse-engineering playbook (when we start)

The Series III BW codec took ~2 months of MITM wire captures because we didn't have ground-truth metadata. Thor's situation is substantially better:

• Ground truth is on disk. Every binary in example-data/ has a paired .IDFW.txt carrying the full decoded sample table (Waveform Data Channels block — see any sample file in thor-watcher/example-data/.../TXT/). Aligning binary bytes to the table's float-per-row values gives an immediate per-byte hypothesis test.
• Cross-event diffing. 1,012 signature-A samples from 9 units spanning 4 years means any field that varies between events is immediately localisable. Fields that are constant across all files (firmware ID, channel labels, format-version word) are also immediately localisable by complementary search.
• No protocol surface. Files at rest, not a wire dialect. No DLE stuffing, no inner-frame parsing, no probe/data two-step.

Suggested first session (2-4 hours): hand-decode UM11719_20231219162723.IDFW (10,290 bytes) against its TXT/UM11719_20231219162723.IDFW.txt sample table (the 2-sec waveform at 1024 sps × 4 channels = 8,192 sample rows). Find the first per-channel sample value (0.0003 in the Tran column at t=0) in the binary. Confirms sample encoding. Everything else flows from there.

---

Code seams ready to receive the codec

When the codec lands, it goes into micromate/idf_file.py (currently a stub raising NotImplementedError). Public API:

from micromate import IdfEvent
from micromate.idf_file import read_idf_file

event: IdfEvent = read_idf_file(Path("UM11719_20231219163444.IDFW"))
# event.peaks.transverse_ips, event.timestamp, event.raw_samples, ...

The ingest pipeline (WaveformStore.save_imported_idf) currently builds the IdfEvent from the .txt parser only. Once read_idf_file() works, the binary becomes authoritative; the .txt parser drops to fast-path metadata cross-check. Operators who don't enable Thor's TXT exporter still get fully populated events.

---

See also

• instantel_protocol_reference.md — Series III BW protocol reference (the Rosetta Stone). STRT record format, DLE framing, BW filename encoding.
• micromate/idf_ascii_report.py — .txt sidecar parser.
• micromate/models.py — IdfEvent, IdfReport typed dataclasses.
• micromate/idf_file.py — placeholder for the binary codec.
• thor-watcher/example-data/THORDATA_example/ — 1,014 paired binary + .txt files for codec validation.