serversdown
ecc935482b
Tighten the Series III / Series IV boundary so UI and storage dispatch
on a clean signal instead of sniffing filenames or applying magnitude
heuristics.
Phase 1 — events.device_family column ("series3" | "series4"):
self-applying migration with filename-based backfill of existing rows
(1,132 backfilled on prod 2026-05-20); plumbed through every import
path (BW endpoint, IDF endpoint, ACH server, BW CLI, sidecar
backfill); UPSERT preserves via COALESCE; UI dispatches on it.
Phase 2 — extract micromate/ package alongside minimateplus/:
native IdfEvent / IdfReport / IdfPeaks / IdfProjectInfo /
IdfSensorCheck (mic in dB(L), not pseudo-psi); moved
idf_ascii_report.py from sfm/ to micromate/; refactored
save_imported_idf to use IdfEvent and bridge to minimateplus.Event at
the SQL-insert boundary; idf_file.py stub for the future binary codec.
Phase 3 prep — docs/idf_protocol_reference.md captures the two
observed Thor binary header signatures (1,012 newer-firmware files vs
2 old files whose layout is byte-for-byte BW-STRT-compatible), file-size
hints suggesting int8 sample encoding, open questions in dependency
order, and a concrete first-session plan for cracking the codec.
Also rolled in the v0.18.1 hotfixes that motivated this work:
- idf_ascii_report parser now handles "<0.005 in/s" (below-threshold)
and "N/A" markers without leaving raw strings in numeric DB columns.
- sfm_webapp.html: defensive _ppvFmt / mic formatter so future
data-shape drift can't kill the whole events table render.
All 1,014 example-data sidecars round-trip through the new package.
See CHANGELOG.md for full notes.
65 lines
2.8 KiB
Python
65 lines
2.8 KiB
Python
"""
|
|
micromate/idf_file.py — placeholder for the Thor IDF binary codec.
|
|
|
|
Thor's ``.IDFH`` (histogram) and ``.IDFW`` (waveform) event files are an
|
|
Instantel proprietary binary format that has not yet been reverse-
|
|
engineered. Today seismo-relay treats them as opaque blobs:
|
|
``WaveformStore.save_imported_idf`` stores the bytes verbatim and reads
|
|
all device-authoritative metadata from the paired ``.IDFW.txt`` /
|
|
``.IDFH.txt`` ASCII sidecar (parsed by ``idf_ascii_report.py``).
|
|
|
|
When we crack the binary codec — same reverse-engineering playbook we
|
|
used to byte-perfect-parse Series III BW files (see
|
|
``docs/instantel_protocol_reference.md`` and ``minimateplus/event_file_io.py``)
|
|
— this module will grow:
|
|
|
|
- ``read_idf_file(path) -> IdfEvent``
|
|
Parse a ``.IDFW``/``.IDFH`` binary and return a fully populated
|
|
``IdfEvent`` whose waveform-sample arrays come from the binary
|
|
(the .txt sidecar's tabular sample block being a best-effort
|
|
check). Lets us ingest Thor events even when the operator
|
|
hasn't enabled the .txt exporter — closing the
|
|
``had_report=False`` gap that the thor-watcher forwarder
|
|
currently tolerates as a known limitation.
|
|
|
|
- ``write_idf_file(path, event)`` (eventually)
|
|
Round-trip event reconstruction, used for verifying the codec
|
|
against captured device files the way ``write_blastware_file``
|
|
verifies the Series III codec.
|
|
|
|
- Helpers for decoding the binary's per-channel sample arrays into
|
|
physical units, the per-event flash buffer's monitor-log records,
|
|
etc.
|
|
|
|
The reverse-engineering path: pair every ``.IDFW`` binary in
|
|
``thor-watcher/example-data/`` with its sibling ``.IDFW.txt``, treating
|
|
the txt's "Waveform Data Channels" block as ground-truth, and align
|
|
the binary's per-channel int16-or-similar arrays against it. Header
|
|
fields (sample rate, channel count, record time, timestamps) sit before
|
|
the sample block — same approach as the BW codec where ASCII strings
|
|
inside the binary (``Project:``, ``Client:``, etc.) anchored field
|
|
discovery.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
from typing import Union
|
|
|
|
from .models import IdfEvent
|
|
|
|
|
|
def read_idf_file(path: Union[str, Path]) -> "IdfEvent":
|
|
"""Parse a Thor ``.IDFW``/``.IDFH`` binary into an ``IdfEvent``.
|
|
|
|
Not yet implemented. When implemented, this will be the canonical
|
|
entry point for reading Thor binaries — the ASCII sidecar parser
|
|
becomes an optional fast-path metadata supplement rather than the
|
|
sole source of device-authoritative data.
|
|
"""
|
|
raise NotImplementedError(
|
|
"IDF binary codec not yet implemented; the .IDFW/.IDFH binary format "
|
|
"is undecoded. Use parse_idf_report() on the paired .txt sidecar "
|
|
"for device-authoritative metadata."
|
|
)
|