seismo-relay v0.19.0 — device-family separation + micromate/ package
Tighten the Series III / Series IV boundary so UI and storage dispatch
on a clean signal instead of sniffing filenames or applying magnitude
heuristics.
Phase 1 — events.device_family column ("series3" | "series4"):
self-applying migration with filename-based backfill of existing rows
(1,132 backfilled on prod 2026-05-20); plumbed through every import
path (BW endpoint, IDF endpoint, ACH server, BW CLI, sidecar
backfill); UPSERT preserves via COALESCE; UI dispatches on it.
Phase 2 — extract micromate/ package alongside minimateplus/:
native IdfEvent / IdfReport / IdfPeaks / IdfProjectInfo /
IdfSensorCheck (mic in dB(L), not pseudo-psi); moved
idf_ascii_report.py from sfm/ to micromate/; refactored
save_imported_idf to use IdfEvent and bridge to minimateplus.Event at
the SQL-insert boundary; idf_file.py stub for the future binary codec.
Phase 3 prep — docs/idf_protocol_reference.md captures the two
observed Thor binary header signatures (1,012 newer-firmware files vs
2 old files whose layout is byte-for-byte BW-STRT-compatible), file-size
hints suggesting int8 sample encoding, open questions in dependency
order, and a concrete first-session plan for cracking the codec.
Also rolled in the v0.18.1 hotfixes that motivated this work:
- idf_ascii_report parser now handles "<0.005 in/s" (below-threshold)
and "N/A" markers without leaving raw strings in numeric DB columns.
- sfm_webapp.html: defensive _ppvFmt / mic formatter so future
data-shape drift can't kill the whole events table render.
All 1,014 example-data sidecars round-trip through the new package.
See CHANGELOG.md for full notes.
This commit is contained in:
@@ -0,0 +1,64 @@
|
||||
"""
|
||||
micromate/idf_file.py — placeholder for the Thor IDF binary codec.
|
||||
|
||||
Thor's ``.IDFH`` (histogram) and ``.IDFW`` (waveform) event files are an
|
||||
Instantel proprietary binary format that has not yet been reverse-
|
||||
engineered. Today seismo-relay treats them as opaque blobs:
|
||||
``WaveformStore.save_imported_idf`` stores the bytes verbatim and reads
|
||||
all device-authoritative metadata from the paired ``.IDFW.txt`` /
|
||||
``.IDFH.txt`` ASCII sidecar (parsed by ``idf_ascii_report.py``).
|
||||
|
||||
When we crack the binary codec — same reverse-engineering playbook we
|
||||
used to byte-perfect-parse Series III BW files (see
|
||||
``docs/instantel_protocol_reference.md`` and ``minimateplus/event_file_io.py``)
|
||||
— this module will grow:
|
||||
|
||||
- ``read_idf_file(path) -> IdfEvent``
|
||||
Parse a ``.IDFW``/``.IDFH`` binary and return a fully populated
|
||||
``IdfEvent`` whose waveform-sample arrays come from the binary
|
||||
(the .txt sidecar's tabular sample block being a best-effort
|
||||
check). Lets us ingest Thor events even when the operator
|
||||
hasn't enabled the .txt exporter — closing the
|
||||
``had_report=False`` gap that the thor-watcher forwarder
|
||||
currently tolerates as a known limitation.
|
||||
|
||||
- ``write_idf_file(path, event)`` (eventually)
|
||||
Round-trip event reconstruction, used for verifying the codec
|
||||
against captured device files the way ``write_blastware_file``
|
||||
verifies the Series III codec.
|
||||
|
||||
- Helpers for decoding the binary's per-channel sample arrays into
|
||||
physical units, the per-event flash buffer's monitor-log records,
|
||||
etc.
|
||||
|
||||
The reverse-engineering path: pair every ``.IDFW`` binary in
|
||||
``thor-watcher/example-data/`` with its sibling ``.IDFW.txt``, treating
|
||||
the txt's "Waveform Data Channels" block as ground-truth, and align
|
||||
the binary's per-channel int16-or-similar arrays against it. Header
|
||||
fields (sample rate, channel count, record time, timestamps) sit before
|
||||
the sample block — same approach as the BW codec where ASCII strings
|
||||
inside the binary (``Project:``, ``Client:``, etc.) anchored field
|
||||
discovery.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Union
|
||||
|
||||
from .models import IdfEvent
|
||||
|
||||
|
||||
def read_idf_file(path: Union[str, Path]) -> "IdfEvent":
|
||||
"""Parse a Thor ``.IDFW``/``.IDFH`` binary into an ``IdfEvent``.
|
||||
|
||||
Not yet implemented. When implemented, this will be the canonical
|
||||
entry point for reading Thor binaries — the ASCII sidecar parser
|
||||
becomes an optional fast-path metadata supplement rather than the
|
||||
sole source of device-authoritative data.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
"IDF binary codec not yet implemented; the .IDFW/.IDFH binary format "
|
||||
"is undecoded. Use parse_idf_report() on the paired .txt sidecar "
|
||||
"for device-authoritative metadata."
|
||||
)
|
||||
Reference in New Issue
Block a user