342 lines
16 KiB
Markdown
342 lines
16 KiB
Markdown
# IDF Protocol Reference — Thor / Micromate Series IV
|
||
|
||
Starting-point reference for reverse-engineering Instantel's Micromate
|
||
Series IV event-file format. Sibling to
|
||
[instantel_protocol_reference.md](instantel_protocol_reference.md) (the
|
||
Series III "Rosetta Stone") — this doc holds what we know so far and
|
||
the open questions still to crack.
|
||
|
||
**Status (2026-05-28):** ASCII text sidecar fully decoded (1,014
|
||
sample files round-trip). **Thor IDFW** binary now decodes via
|
||
`micromate.idf_file.read_idf_file()` — reuses the BW segment-rotated
|
||
block codec verbatim at fixed body offset `0x0f1f`; metadata (serial,
|
||
timestamp, sample_rate, record_time, calibration_date) extracted from
|
||
the binary header. Sample fidelity is 87–99% byte-exact on quiet
|
||
events; loud events hit the BW codec's known walker-stops-early
|
||
limitation. Residual ~3% drift on per-sample deltas (likely a
|
||
Thor-specific 12-bit delta refinement not yet modelled).
|
||
|
||
**Thor IDFH histograms also decoded.** Body has one or more segments;
|
||
each 12-byte segment header `[length_be 2B][0a 00 00 00][00 NN][05 3f]`
|
||
introduces `N = (length - 10) // 72` interval records of 72 bytes
|
||
each. Each interval = 4 × 16-byte per-channel records:
|
||
`[int16 min][int16 max][int16 ??][uint16 halfp][2B 00][uint16 ??][2B 00][uint16 ??]`.
|
||
Geo peak `= max(|min|, |max|) / 32768 × 10` in/s (matches sidecar
|
||
~1.8%); freq `= 512 / halfp` Hz (None for halfp ≤ 5 → ">100"
|
||
sentinel). Corpus: **all 859 Thor IDFH files decode, 181,071
|
||
intervals**. Wired through `read_idf_file()` →
|
||
`save_imported_idf()` → sidecar's `extensions.idf_intervals`.
|
||
|
||
**Note on the BE9439 outliers in the example corpus:** Two files
|
||
(`BE9439_20200713131747.IDFW` and `BE9439_20200713124251.IDFH`) are
|
||
**Series III Blastware** binaries, not Thor. Provenance: TMI tried
|
||
to use Thor to manage auto-call-homes for Series III units; the
|
||
experiment didn't work out, but it did leave a few BW event files
|
||
in Thor's per-serial directory structure with `.IDFW`/`.IDFH`
|
||
extensions — Thor's forwarder applied its own naming convention to
|
||
the BW bodies it was relaying. Their header `10 00 01 80 00 00
|
||
Instantel STRT ff fe <end_key> <start_key>` is the BW SUB 5A STRT
|
||
record, not a Thor body preamble. The reader detects them by
|
||
signature and raises `NotImplementedError` pointing callers at
|
||
`read_blastware_file()`, which extracts BW-format peaks from them.
|
||
|
||
**Still NYI for Thor IDFH:** per-channel `int16 field4` (possibly
|
||
time-of-peak); the two uint16 fields (probably PVS contributions);
|
||
8-byte interval tail (PVS data); mic dB(L) exact conversion constant.
|
||
|
||
### Codec breakthroughs (2026-05-28)
|
||
|
||
- **Body offset is a fixed `0x0f1f`** across 151/154 corpus IDFW
|
||
files. Preceded by a 4-byte record-type marker (`46 00 00 00`)
|
||
+ magic preamble `00 02 00 [Tran[0] BE] [Tran[1] BE]`.
|
||
- **Sample stream is BW's segment-rotated block codec verbatim.**
|
||
Thor reuses `10 NN` (nibble), `20 NN` (int8), `00 NN` (RLE),
|
||
`30 NN` (packed12), `40 02` (segment header) tags with the same
|
||
semantics. Channel rotation Tran→Vert→Long→MicL.
|
||
- **Geo LSB = 0.0003 in/s** (not BW's 0.005), because Thor's 16-bit
|
||
ADC range maps to 10 in/s without the 16-count BW quantization step.
|
||
- **Mic ≈ 2.14×10⁻⁶ psi/count** (rough scale; refine after channel
|
||
block calibration constants are decoded).
|
||
- **BW compliance anchor `\xbe\x80\x00\x00\x00\x00` reappears at
|
||
IDFW offset 0x952** — sample_rate at anchor−6 (uint16 BE),
|
||
record_time at anchor+6 (float32 BE), same layout as BW.
|
||
- **Event timestamp at offset 0x97A** — 8 bytes `[day][month]
|
||
[year_be][unk][hour][min][sec]`. Stop-time mirrors at 0x982.
|
||
- **Serial as null-terminated ASCII at 0x14E**.
|
||
- **Calibration date** at 0x194–0x197 (day, month, year_be).
|
||
- Per-sample residual drift of ~3% suggests Thor encodes int8/nibble
|
||
deltas with an extra refinement bit that BW doesn't carry —
|
||
unsolved; errors resync within a few samples so cumulative impact
|
||
is small.
|
||
|
||
---
|
||
|
||
## File model
|
||
|
||
### Filename convention
|
||
|
||
```
|
||
<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>
|
||
```
|
||
|
||
- **SERIAL** — literal device serial, two-letter prefix + numeric
|
||
suffix. Examples seen: `UM11719`, `UM13981`, `UM20147`, `BE9439`.
|
||
Unlike Series III BW filenames (`M529LK44.AB0`, base-36 stem),
|
||
Series IV filenames carry the serial in plain text.
|
||
- **YYYYMMDDHHMMSS** — 14-char ASCII timestamp in **device local
|
||
time** (no timezone marker).
|
||
- **KIND** — `IDFH` for histograms, `IDFW` for waveforms.
|
||
|
||
The `.IDFH.txt` / `.IDFW.txt` ASCII sidecar lives in a `TXT/`
|
||
**subfolder** of the unit's directory, not alongside the binary.
|
||
This pairing convention is encoded in
|
||
`event_forwarder.idf_report_path()`.
|
||
|
||
### Directory layout
|
||
|
||
```
|
||
C:\THORDATA\
|
||
└── <Project>\
|
||
└── <UM####>\ ← unit serial dir
|
||
├── UM12345_20260520100000.MLG ← monitor log (not events)
|
||
├── UM12345_20260520100000.IDFH ← histogram event (binary)
|
||
├── UM12345_20260520100000.IDFW ← waveform event (binary)
|
||
├── UM12345_20260520100000.IDFW.CDB ← cache-DB variant (skip)
|
||
├── TXT\
|
||
│ ├── UM12345_20260520100000.IDFH.txt ← histogram ASCII sidecar
|
||
│ └── UM12345_20260520100000.IDFW.txt ← waveform ASCII sidecar
|
||
├── CSV\, HTML\, PDF\, XML\ ← operator-facing derived exports
|
||
└── ...
|
||
```
|
||
|
||
The `.IDFW.CDB` files share the binary's basename but appear to be a
|
||
separate cache/database variant. Their first 8 bytes match the
|
||
**old**-firmware Thor signature (see below) regardless of which
|
||
signature the paired `.IDFW` uses. Purpose unknown; sizes vary
|
||
wildly (observed 123 B → 40,491 B). Thor-watcher's forwarder
|
||
deliberately skips them.
|
||
|
||
### Sample corpus
|
||
|
||
The `thor-watcher/example-data/THORDATA_example/` tree carries
|
||
**1,014 paired .IDFW / .IDFH + .txt files** spanning 2020–2023
|
||
across nine units (UM11719, UM13981, UM20147, …, plus BE9439 from
|
||
2020). This is the reverse-engineering ground truth.
|
||
|
||
---
|
||
|
||
## ASCII sidecar (`.IDFW.txt` / `.IDFH.txt`) — fully decoded
|
||
|
||
Shape: plain text, one `"Key : Value"` line per metadata field,
|
||
followed for waveforms by a tab-separated sample table headed by
|
||
the literal line `Waveform Data Channels`. Parsed by
|
||
[`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py).
|
||
See [`micromate/models.py`](../micromate/models.py) for the typed
|
||
`IdfReport` shape.
|
||
|
||
### Notable conventions
|
||
|
||
- **Units are native to Thor** — geophone in **in/s**, microphone in
|
||
**dB(L)** (not psi like Series III BW reports), frequency in Hz,
|
||
acceleration in g, displacement in in.
|
||
- **Below-threshold readings** appear as the literal string
|
||
`<0.005 in/s` (155 occurrences in the sample corpus) — the parser
|
||
strips the `<` and treats the numeric remainder as the value.
|
||
- **Out-of-range / not-measured** values appear as `N/A` — parser
|
||
drops the field rather than letting the string leak into a numeric
|
||
column.
|
||
- **Firmware string** observed: `Micromate ISEE 11.0AK`.
|
||
- **TitleString1..4** are operator-defined free-text slots; Thor's
|
||
default labels map them to Location / Client / Company / Notes,
|
||
which the parser surfaces as `project` / `client` / `operator` /
|
||
`notes`.
|
||
- **Histogram sidecars** use `HistogramStartDate` / `HistogramStartTime`
|
||
in place of waveform's `EventDate` / `EventTime`. Parser falls
|
||
through to either.
|
||
- **Histogram tabular block** lacks the `Waveform Data Channels`
|
||
marker; instead it's a multi-line column header followed by
|
||
per-interval rows (`<date> <time> <tran-ppv> <freq> ...`). Parser
|
||
silently ignores lines after the metadata block since they lack a
|
||
colon-separated `key : value` shape (the timestamps DO contain
|
||
colons but produce garbage keys that don't collide with any
|
||
recognised field).
|
||
|
||
---
|
||
|
||
## Binary header signatures (observed)
|
||
|
||
Hex dump of the first 32 bytes across 1,014 sample files reveals
|
||
**two distinct file signatures**, both anchored by the literal
|
||
ASCII string `"\x00Instantel\x00"` at offset 6–16:
|
||
|
||
### Signature A — newer firmware (1,012 files, 99.8% of corpus)
|
||
|
||
```
|
||
00000000: 0012 0100 0000 496e 7374 616e 7465 6c00 ......Instantel.
|
||
00000010: 0000 a695 002e b500 4f70 6572 6174 6f72 ........Operator
|
||
^^^^^^^^^^^^^^^^
|
||
operator/title string starts at 0x18
|
||
```
|
||
|
||
Header bytes 0–5: `00 12 01 00 00 00`. Followed immediately by the
|
||
8-byte ASCII tag, then 6 unknown bytes, then ASCII operator-supplied
|
||
strings (Operator name, etc.) and on through the project / client /
|
||
title strings. No `STRT` record observed in this layout.
|
||
|
||
### Signature B — older firmware (2 files: BE9439 from 2020)
|
||
|
||
```
|
||
00000000: 1000 0180 0000 496e 7374 616e 7465 6c00 ......Instantel.
|
||
00000010: 072c 0012 0300 5354 5254 fffe 0111 2340 .,....STRT....#@
|
||
^^^^^^^^^ ^^^^^^^^^
|
||
STRT magic 4-byte end_key
|
||
00000020: 0111 0000 2e5f 00ac 4600 0000 0200 0000 ....._..F.......
|
||
^^^^^^^^^ ^^^
|
||
4-byte start_key 0x46 (BW WAVEHDR record-type marker)
|
||
```
|
||
|
||
Header bytes 0–5: `10 00 01 80 00 00`. The structure after the
|
||
`Instantel` magic is **byte-for-byte identical to a BW SUB 5A
|
||
probe-response STRT record** as documented in
|
||
[instantel_protocol_reference.md → "SUB 5A — STRT record encodes
|
||
end_offset"](instantel_protocol_reference.md). Specifically:
|
||
|
||
| Offset | Bytes | Meaning (per BW reference) |
|
||
|--------|---------------------|--------------------------------------|
|
||
| 0x14 | `53 54 52 54` | `STRT` magic |
|
||
| 0x18 | `ff fe` | STRT sentinel |
|
||
| 0x1A | `01 11 23 40` | `end_key` (4 bytes) |
|
||
| 0x1E | `01 11 00 00` | `start_key` (4 bytes) |
|
||
| 0x26 | `46` | `0x46` waveform-record type marker |
|
||
|
||
**Hypothesis:** Older Micromate firmware writes a wrapped BW-format
|
||
event into the `.IDFW` file — essentially the same on-disk shape as
|
||
a Series III device, with the new filename convention applied at
|
||
export time. Newer firmware (signature A) abandoned the
|
||
BW-compatible layout for an Instantel-specific format.
|
||
|
||
If that hypothesis holds, the 2 signature-B files can already be
|
||
parsed via `minimateplus/event_file_io.read_blastware_file()` — worth
|
||
testing. The 1,012 signature-A files are the real reverse-engineering
|
||
target.
|
||
|
||
### `.IDFW.CDB` cache files
|
||
|
||
Always carry signature B (`10 00 01 80 ...`), even when the paired
|
||
`.IDFW` carries signature A. Plausible explanation: the CDB is an
|
||
internal Thor cache-database export that retains the legacy BW-style
|
||
record layout regardless of the user-facing `.IDFW` format version.
|
||
Not currently consumed by the forwarder.
|
||
|
||
---
|
||
|
||
## File-size patterns (Signature A, the main target)
|
||
|
||
Survey of 1,012 signature-A files:
|
||
|
||
| Event type | Typical size | Source of variance |
|
||
|--------------|-------------------|----------------------------------------------|
|
||
| `.IDFW` 2-sec | 9,200 – 10,500 B | Operator-supplied strings (TitleString1..4) of varying length |
|
||
| `.IDFH` | 2,944 – 4,076 B | Histogram interval count (record duration / interval) |
|
||
|
||
**Naive arithmetic for 2-sec waveform:**
|
||
- 4 channels × 2 sec × 1024 sps = 8,192 samples
|
||
- At 2 bytes/sample (int16) = 16,384 sample bytes → file would be > 16 KB
|
||
- Observed: ~9–10 KB
|
||
- → samples are likely **1 byte each** (int8 quantised), **or** stored
|
||
with bit-packing / delta encoding, **or** only one channel's
|
||
full-rate samples are stored with the others reconstructed
|
||
arithmetically. Verifying this is the **first RE milestone**.
|
||
|
||
Project-string–length variance (~1 KB across the corpus) is consistent
|
||
with the file carrying a single copy of each TitleString1..4 plus
|
||
operator + setup-name as null-padded ASCII regions.
|
||
|
||
---
|
||
|
||
## Open questions
|
||
|
||
The reverse-engineering targets, roughly in dependency order:
|
||
|
||
1. **Sample encoding (signature A)** — int8? int16 LE/BE? Bit-packed?
|
||
Delta-coded? Per-channel interleaved or sequential blocks?
|
||
2. **Header field layout (signature A)** — where do sample_rate,
|
||
record_time, channel count, and per-channel peaks live in the
|
||
binary? The ASCII sidecar gives the device-authoritative values,
|
||
so binary fields can be confirmed by diff.
|
||
3. **Operator-string offsets** — `Operator` at 0x18 is the first
|
||
visible string in signature-A files; the rest (project, client,
|
||
notes, setup) follow. Need to map exact offsets and null-padding
|
||
conventions.
|
||
4. **Signature-B → BW codec compatibility** — does
|
||
`minimateplus/event_file_io.read_blastware_file()` actually parse
|
||
the 2 BE9439 signature-B files as-is? If yes, the OLD-format
|
||
ingest is free.
|
||
5. **`.IDFW.CDB` purpose** — is it an internal Thor cache, a
|
||
ring-buffer dump, or something else? Worth a single small effort
|
||
to characterise so we know what we're skipping.
|
||
6. **Footer / checksum** — every BW event file has a footer; does
|
||
IDF? Where does the per-channel sample block end?
|
||
|
||
---
|
||
|
||
## Reverse-engineering playbook (when we start)
|
||
|
||
The Series III BW codec took ~2 months of MITM wire captures
|
||
because we didn't have ground-truth metadata. Thor's situation is
|
||
**substantially better**:
|
||
|
||
- **Ground truth is on disk.** Every binary in `example-data/`
|
||
has a paired `.IDFW.txt` carrying the full decoded sample table
|
||
(`Waveform Data Channels` block — see any sample file in
|
||
`thor-watcher/example-data/.../TXT/`). Aligning binary bytes
|
||
to the table's float-per-row values gives an immediate per-byte
|
||
hypothesis test.
|
||
- **Cross-event diffing.** 1,012 signature-A samples from 9 units
|
||
spanning 4 years means any field that varies between events is
|
||
immediately localisable. Fields that are constant across all
|
||
files (firmware ID, channel labels, format-version word) are also
|
||
immediately localisable by complementary search.
|
||
- **No protocol surface.** Files at rest, not a wire dialect. No
|
||
DLE stuffing, no inner-frame parsing, no probe/data two-step.
|
||
|
||
Suggested first session (2-4 hours): hand-decode `UM11719_20231219162723.IDFW`
|
||
(10,290 bytes) against its `TXT/UM11719_20231219162723.IDFW.txt`
|
||
sample table (the 2-sec waveform at 1024 sps × 4 channels = 8,192
|
||
sample rows). Find the first per-channel sample value (`0.0003` in
|
||
the Tran column at t=0) in the binary. Confirms sample encoding.
|
||
Everything else flows from there.
|
||
|
||
---
|
||
|
||
## Code seams ready to receive the codec
|
||
|
||
When the codec lands, it goes into
|
||
[`micromate/idf_file.py`](../micromate/idf_file.py) (currently a
|
||
stub raising `NotImplementedError`). Public API:
|
||
|
||
```python
|
||
from micromate import IdfEvent
|
||
from micromate.idf_file import read_idf_file
|
||
|
||
event: IdfEvent = read_idf_file(Path("UM11719_20231219163444.IDFW"))
|
||
# event.peaks.transverse_ips, event.timestamp, event.raw_samples, ...
|
||
```
|
||
|
||
The ingest pipeline (`WaveformStore.save_imported_idf`) currently
|
||
builds the `IdfEvent` from the `.txt` parser only. Once
|
||
`read_idf_file()` works, the binary becomes authoritative; the
|
||
`.txt` parser drops to fast-path metadata cross-check. Operators
|
||
who don't enable Thor's TXT exporter still get fully populated
|
||
events.
|
||
|
||
---
|
||
|
||
## See also
|
||
|
||
- [instantel_protocol_reference.md](instantel_protocol_reference.md) — Series III BW protocol reference (the Rosetta Stone). STRT record format, DLE framing, BW filename encoding.
|
||
- [`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py) — `.txt` sidecar parser.
|
||
- [`micromate/models.py`](../micromate/models.py) — `IdfEvent`, `IdfReport` typed dataclasses.
|
||
- [`micromate/idf_file.py`](../micromate/idf_file.py) — placeholder for the binary codec.
|
||
- [`thor-watcher/example-data/THORDATA_example/`](../../thor-watcher/example-data/) — 1,014 paired binary + .txt files for codec validation.
|