serversdown/seismo-relay
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

seismo-relay v0.19.0 — device-family separation + micromate/ package

Browse Source 
Tighten the Series III / Series IV boundary so UI and storage dispatch
on a clean signal instead of sniffing filenames or applying magnitude
heuristics.

Phase 1 — events.device_family column ("series3" | "series4"):
  self-applying migration with filename-based backfill of existing rows
  (1,132 backfilled on prod 2026-05-20); plumbed through every import
  path (BW endpoint, IDF endpoint, ACH server, BW CLI, sidecar
  backfill); UPSERT preserves via COALESCE; UI dispatches on it.

Phase 2 — extract micromate/ package alongside minimateplus/:
  native IdfEvent / IdfReport / IdfPeaks / IdfProjectInfo /
  IdfSensorCheck (mic in dB(L), not pseudo-psi); moved
  idf_ascii_report.py from sfm/ to micromate/; refactored
  save_imported_idf to use IdfEvent and bridge to minimateplus.Event at
  the SQL-insert boundary; idf_file.py stub for the future binary codec.

Phase 3 prep — docs/idf_protocol_reference.md captures the two
observed Thor binary header signatures (1,012 newer-firmware files vs
2 old files whose layout is byte-for-byte BW-STRT-compatible), file-size
hints suggesting int8 sample encoding, open questions in dependency
order, and a concrete first-session plan for cracking the codec.

Also rolled in the v0.18.1 hotfixes that motivated this work:
  - idf_ascii_report parser now handles "<0.005 in/s" (below-threshold)
    and "N/A" markers without leaving raw strings in numeric DB columns.
  - sfm_webapp.html: defensive _ppvFmt / mic formatter so future
    data-shape drift can't kill the whole events table render.

All 1,014 example-data sidecars round-trip through the new package.
See CHANGELOG.md for full notes.
This commit is contained in:
serversdown
2026-05-20 15:19:49 +00:00
parent e95ac692ee
commit ecc935482b
11 changed files with 966 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sfm/idf_ascii_report.py
-315
View File
@@ -1,315 +0,0 @@
"""
sfm/idf_ascii_report.py — parse Thor (Micromate Series IV) IDF ASCII reports.
Thor exports a `.IDFW.txt` or `.IDFH.txt` sidecar next to each `.IDFW`
(waveform) or `.IDFH` (histogram) event binary. Each sidecar is a
plain-text file with `"Key : Value"` lines covering the full device-
authoritative event metadata — PPV per channel, ZC Freq, Time of Peak,
Peak Acceleration / Displacement, sensor self-check results, project
strings, calibration date, battery level, etc. — followed by a raw
waveform-samples block headed by the literal line "Waveform Data Channels".
This is the Thor analogue of `minimateplus/bw_ascii_report.py` for the
Blastware (Series III) report format. The parser is intentionally
permissive: we extract everything we recognise into a flat dict and
silently ignore anything we don't. Downstream callers parse units
(`"0.2119 in/s"` → 0.2119) only on the fields they need.
Example input (truncated):
"EventType : Full Waveform"
"SampleRate : 1024 sps"
"EventTime : 16:27:23"
"EventDate : 2023-12-19"
"TranPPV : 0.0251 in/s"
"VertPPV : 0.2119 in/s"
"LongPPV : 0.0282 in/s"
"PeakVectorSum : 0.2131 in/s"
"MicPSPL : 99.4 dB(L)"
"TranZCFreq : 6.5 Hz"
"SerialNumber : UM11719"
"Version : Micromate ISEE 11.0AK"
"FileName : UM11719_20231219162723.IDFW"
"BatteryLevel : 3.8 volts"
"Calibration : November 22, 2023 by Instantel"
"TranTestResults : Passed"
"TitleString1 : UPMC Presby-Loc 3-Level1-1R Elevator Rm"
Waveform Data Channels
Tran Vert Long MicL
0.0003 -0.0003 0.0003 0.00013
...
"""
from __future__ import annotations
import datetime
import re
from typing import Any, Dict, Optional, Tuple, Union
# Lines look like: "Key : Value" (quotes literal, single ":" separator)
_LINE_RE = re.compile(r'^\s*"?([^":]+?)"?\s*:\s*"?(.*?)"?\s*$')
# Marker that ends the metadata block — everything after is raw sample data.
_WAVEFORM_BLOCK_MARKER = "waveform data channels"
def _normalize_key(raw: str) -> str:
"""Convert "TranPPV" / "PreTriggerLength" → snake_case."""
s = raw.strip()
# Insert underscore between lower→upper / digit→letter transitions
s = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", "_", s)
s = re.sub(r"(?<=[A-Z])(?=[A-Z][a-z])", "_", s)
s = s.replace("-", "_").replace(" ", "_")
return s.lower()
def _strip_unit_suffix(value: str) -> str:
"""Return the numeric part of values like "0.2119 in/s" → "0.2119".
Also strips Thor's below/above-threshold prefixes:
"<0.005 in/s" → "0.005" (below-noise-floor reading)
">100 Hz" → "100" (above-measurement-range reading)
"""
parts = value.strip().split()
token = parts[0] if parts else value.strip()
if token.startswith("<") or token.startswith(">"):
token = token[1:]
return token
def _parse_float(value: str) -> Optional[float]:
try:
return float(_strip_unit_suffix(value))
except (ValueError, TypeError):
return None
def _parse_int(value: str) -> Optional[int]:
try:
return int(float(_strip_unit_suffix(value)))
except (ValueError, TypeError):
return None
def parse_idf_report(text: Union[str, bytes]) -> Dict[str, Any]:
"""
Parse a Thor IDFW.txt / IDFH.txt sidecar.
Returns a flat dict with two kinds of entries:
- **Raw fields** — every `Key : Value` line, keyed by snake_case
of the original key, value as a string (unit suffix preserved).
Lets callers grab any field we haven't explicitly normalised.
- **Derived fields** — a curated set with parsed types:
* `serial_number` str
* `event_type` str ("Full Waveform" / "Full Histogram")
* `event_datetime` ISO-8601 string ("YYYY-MM-DDTHH:MM:SS") when
both EventDate and EventTime are present
* `sample_rate` int (samples/sec)
* `tran_ppv`,`vert_ppv`,`long_ppv` float (in/s)
* `mic_ppv` float (dB or psi — same units as MicPSPL)
* `peak_vector_sum` float (in/s)
* `tran_zc_freq`,`vert_zc_freq`,`long_zc_freq` float (Hz)
* `record_time_sec` float (seconds)
* `pre_trigger_sec` float (seconds)
* `project` str (from TitleString1 — Thor's location)
* `client` str (TitleString2)
* `operator` str (TitleString3 — company/operator)
* `notes` str (TitleString4)
* `setup` str
* `version` str (firmware)
* `battery_volts` float
* `calibration_text` str (e.g. "November 22, 2023 by Instantel")
* `tran_test_passed`, `vert_test_passed`, `long_test_passed`,
`mic_test_passed` bool ("Passed" → True; anything else → False)
* `filename` str (FileName line — useful sanity check)
Stops parsing at the literal "Waveform Data Channels" line; the
raw-samples block is left to whoever wants to decode the binary.
Input may be `str` or `bytes` (`utf-8`/`latin-1` tolerant).
"""
if isinstance(text, bytes):
try:
text = text.decode("utf-8")
except UnicodeDecodeError:
text = text.decode("latin-1", errors="replace")
raw: Dict[str, str] = {}
for line in text.splitlines():
stripped = line.strip()
if not stripped:
continue
if stripped.lower().startswith(_WAVEFORM_BLOCK_MARKER):
break
m = _LINE_RE.match(stripped)
if not m:
continue
key = _normalize_key(m.group(1))
value = m.group(2).strip()
# Multi-value lines (Channel, Units, etc.) — coalesce by appending.
if key in raw:
raw[key] = raw[key] + "; " + value
else:
raw[key] = value
out: Dict[str, Any] = dict(raw) # keep all raw fields
# ── Derived fields ───────────────────────────────────────────────────────
def _take(*candidates: str) -> Optional[str]:
for c in candidates:
if c in raw:
return raw[c]
return None
# Event identity
if "serial_number" in raw:
out["serial_number"] = raw["serial_number"]
if "event_type" in raw:
out["event_type"] = raw["event_type"]
if "file_name" in raw:
out["filename"] = raw["file_name"]
# Combined date+time. Waveform sidecars use "EventDate" / "EventTime";
# histogram sidecars use "HistogramStartDate" / "HistogramStartTime".
# Prefer the event_* names when both are present.
ed = raw.get("event_date") or raw.get("histogram_start_date")
et = raw.get("event_time") or raw.get("histogram_start_time")
if ed and et:
try:
dt = datetime.datetime.strptime(f"{ed} {et}", "%Y-%m-%d %H:%M:%S")
out["event_datetime"] = dt.isoformat()
except ValueError:
pass
# Numeric scalars. For every field we typify here, we MUST drop the
# raw string copy from `out` when parsing fails — Thor writes things
# like "<0.005 in/s" (below threshold) and "N/A" (not measured) that
# would otherwise linger in `out` as strings, sneak into SQLite REAL
# columns via permissive type affinity, and then crash the JS
# frontend on `.toFixed(...)`.
int_fields = ("sample_rate",)
for key in int_fields:
v = raw.get(key)
if v is None:
continue
iv = _parse_int(v)
if iv is not None:
out[key] = iv
else:
out.pop(key, None)
float_fields = (
"tran_ppv", "vert_ppv", "long_ppv", "peak_vector_sum",
"tran_zc_freq", "vert_zc_freq", "long_zc_freq",
"tran_peak_acceleration", "vert_peak_acceleration",
"long_peak_acceleration",
"tran_peak_displacement", "vert_peak_displacement",
"long_peak_displacement",
"tran_time_of_peak", "vert_time_of_peak", "long_time_of_peak",
"mic_time_of_peak", "mic_zc_freq",
)
for key in float_fields:
v = raw.get(key)
if v is None:
continue
fv = _parse_float(v)
if fv is not None:
out[key] = fv
else:
out.pop(key, None)
# Microphone — Thor reports MicPSPL (dB(L)) which is the closest
# analogue to BW's mic_ppv. The raw "99.4 dB(L)" string stays in
# `out` under the original `mic_pspl` key for display; the parsed
# float goes in `mic_ppv`.
mic = raw.get("mic_pspl")
if mic is not None:
fv = _parse_float(mic)
if fv is not None:
out["mic_ppv"] = fv
# Record / pre-trigger duration — same drop-on-failure discipline.
rt = raw.get("record_time")
if rt is not None:
fv = _parse_float(rt)
if fv is not None:
out["record_time_sec"] = fv
pt = raw.get("pre_trigger_length")
if pt is not None:
fv = _parse_float(pt)
if fv is not None:
out["pre_trigger_sec"] = fv
# Project / client / operator / location strings. Thor's title
# strings are operator-defined; conventional mapping (per Thor's
# default TitleNote labels in the example data):
# TitleString1 = Location → project (sensor location identifier)
# TitleString2 = Client → client
# TitleString3 = Company → operator (the monitoring company)
# TitleString4 = Notes → notes
out["project"] = _take("title_string1")
out["client"] = _take("title_string2")
out["operator"] = _take("title_string3", "operator")
out["notes"] = _take("title_string4", "post_event_note")
if "setup" in raw:
out["setup"] = raw["setup"]
if "version" in raw:
out["version"] = raw["version"]
# Battery (e.g. "3.8 volts" → 3.8)
bl = raw.get("battery_level")
if bl is not None:
fv = _parse_float(bl)
if fv is not None:
out["battery_volts"] = fv
# Calibration line is free-form (e.g. "November 22, 2023 by Instantel").
if "calibration" in raw:
out["calibration_text"] = raw["calibration"]
# Sensor self-check results — bool flags
for key, out_key in (
("tran_test_results", "tran_test_passed"),
("vert_test_results", "vert_test_passed"),
("long_test_results", "long_test_passed"),
("mic_test_results", "mic_test_passed"),
):
v = raw.get(key)
if v is not None:
out[out_key] = v.strip().lower() == "passed"
return out
def serial_from_filename(name: str) -> Optional[str]:
"""Convenience: pull the serial prefix from a Thor event filename.
Thor uses the literal serial as the filename prefix:
UM11719_20231219163444.IDFW → "UM11719"
BE9439_20200713124251.IDFH → "BE9439"
"""
m = re.match(r"^([A-Z]{2}\d+)_\d{14}\.(IDFH|IDFW)(?:\.txt)?$",
name, re.IGNORECASE)
return m.group(1).upper() if m else None
def parse_event_filename(name: str) -> Optional[Tuple[str, datetime.datetime, str]]:
"""Parse `<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>` → (serial, datetime, kind).
`kind` is "IDFH" or "IDFW" (upper-case). Returns None on no match.
"""
m = re.match(r"^([A-Z]{2}\d+)_(\d{14})\.(IDFH|IDFW)$",
name, re.IGNORECASE)
if not m:
return None
try:
ts = datetime.datetime.strptime(m.group(2), "%Y%m%d%H%M%S")
except ValueError:
return None
return m.group(1).upper(), ts, m.group(3).upper()
sfm/server.py
+1 -1
View File
@@ -2491,7 +2491,7 @@ async def db_import_idf_file(
1. Pair the binary with its `<binary>.txt` ASCII report when one
is present in the same upload.
2. Parse the report via `sfm.idf_ascii_report.parse_idf_report`
2. Parse the report via `micromate.parse_idf_report`
and copy the binary into the persistent store via
`WaveformStore.save_imported_idf`, writing a `.sfm.json`
sidecar with `source.kind = "idf-import"`.
sfm/waveform_store.py
+32 -89
View File
@@ -426,99 +426,48 @@ class WaveformStore:
`.IDFH`) produced by Thor's TXT exporter.
Thor binaries are stored as opaque bytes — seismo-relay doesn't
decode the proprietary IDF binary format. Device-authoritative
metadata comes from the paired `.IDFW.txt` / `.IDFH.txt` sidecar
when supplied; we parse that text and surface its fields onto
the returned Event so the SFM database row has real PPV/project
values instead of NULLs.
yet decode the proprietary IDF binary format (codec slot lives
at ``micromate/idf_file.py``). Device-authoritative metadata
comes from the paired ``.IDFW.txt`` / ``.IDFH.txt`` sidecar
when supplied.
Workflow:
1. Parse the paired TXT report (when supplied) via
`sfm.idf_ascii_report.parse_idf_report`.
2. Build a minimal `Event` populated from the report fields
(timestamp, peaks, project info, sample_rate, record_type).
3. Resolve serial from filename prefix or `serial_hint`.
4. Copy bytes verbatim into <root>/<serial>/<filename>.
5. Write the `.sfm.json` sidecar with source.kind = "idf-import".
``micromate.parse_idf_report`` → dict.
2. Wrap parsed dict + filename into a typed ``micromate.IdfEvent``.
3. Copy bytes verbatim into ``<root>/<serial>/<filename>``.
4. Bridge IdfEvent → ``minimateplus.Event`` (for the existing
sidecar / DB insert machinery) via
``IdfEvent.to_minimateplus_event(waveform_key)``.
5. Write the ``.sfm.json`` sidecar with
``source.kind = "idf-import"`` and the full raw IDF report
under ``extensions.idf_report``.
Returns (event, record_dict) so the endpoint can both insert
Returns ``(event, record_dict)`` so the endpoint can both insert
into SeismoDb and surface the parsed event.
"""
from sfm.idf_ascii_report import (
parse_idf_report,
parse_event_filename,
serial_from_filename as _idf_serial_from_filename,
)
from minimateplus.models import (
Event, PeakValues, ProjectInfo, Timestamp,
)
from micromate import IdfEvent, parse_idf_report
# Parse the .txt sidecar (best-effort; non-fatal on failure).
report: dict = {}
report_dict: dict = {}
if idf_report_text is not None:
try:
report = parse_idf_report(idf_report_text)
report_dict = parse_idf_report(idf_report_text)
except Exception as exc:
log.warning(
"save_imported_idf: report parse failed: %s — continuing without it",
exc,
)
# Resolve serial: prefer the explicit hint, fall back to filename prefix.
serial = (
serial_hint
or report.get("serial_number")
or _idf_serial_from_filename(source_path.name)
or "UNKNOWN"
)
# Build the typed IdfEvent. Filename is authoritative for
# (serial, timestamp, kind); the report's event_datetime takes
# precedence over the filename timestamp inside from_report().
idf_event = IdfEvent.from_report(report_dict, source_path.name)
# Resolve event timestamp + kind from the filename (always present).
parsed_name = parse_event_filename(source_path.name)
kind = "Waveform"
ts_dt: Optional[datetime.datetime] = None
if parsed_name is not None:
_, ts_dt, kind_token = parsed_name
kind = "Histogram" if kind_token == "IDFH" else "Waveform"
# Report's event_datetime is the device-authoritative value; prefer it.
if "event_datetime" in report:
try:
ts_dt = datetime.datetime.fromisoformat(report["event_datetime"])
except (TypeError, ValueError):
pass
ts_obj: Optional[Timestamp] = None
if ts_dt is not None:
ts_obj = Timestamp(
raw=bytes(9),
flag=0,
year=ts_dt.year,
unknown_byte=0,
month=ts_dt.month,
day=ts_dt.day,
hour=ts_dt.hour,
minute=ts_dt.minute,
second=ts_dt.second,
)
# Build PeakValues from the report (fields are None when absent).
pv = PeakValues(
tran=report.get("tran_ppv"),
vert=report.get("vert_ppv"),
long=report.get("long_ppv"),
micl=report.get("mic_ppv"),
peak_vector_sum=report.get("peak_vector_sum"),
)
# Build ProjectInfo. See idf_ascii_report — Thor's title strings
# carry project / client / company / notes in TitleString1..4.
pi = ProjectInfo(
setup_name=report.get("setup"),
project=report.get("project"),
client=report.get("client"),
operator=report.get("operator"),
sensor_location=None, # Thor folds location into TitleString1 = project
notes=report.get("notes"),
)
# Operator-supplied serial_hint wins over the binary's filename
# prefix when both are present (e.g. callers passing a known-good
# serial that overrides a misnamed export).
serial = serial_hint or idf_event.serial or "UNKNOWN"
# Filesystem write.
filename = source_path.name
@@ -532,16 +481,10 @@ class WaveformStore:
# surrogate — every distinct binary maps to a distinct row.
waveform_key = bytes.fromhex(sha256)[:16]
ev = Event(
index=0,
timestamp=ts_obj,
sample_rate=report.get("sample_rate"),
peak_values=pv,
project_info=pi,
record_type=kind,
rectime_seconds=report.get("record_time_sec"),
)
ev._waveform_key = waveform_key
# Bridge to minimateplus.Event for the existing sidecar / DB
# insert paths. See IdfEvent.to_minimateplus_event() for the
# caveats of this bridge (mic units, missing fields → sidecar).
ev = idf_event.to_minimateplus_event(waveform_key)
# Write the sidecar. Source kind "idf-import" was added to the
# allow-list in event_file_io.event_to_sidecar_dict for this.
@@ -567,14 +510,14 @@ class WaveformStore:
# consumers can recover the rich derived fields that don't fit
# the BW-shaped event model (Peak Acceleration / Displacement,
# Time of Peak, sensor self-check, calibration, firmware).
if report:
sidecar["extensions"]["idf_report"] = report
if report_dict:
sidecar["extensions"]["idf_report"] = report_dict
event_file_io.write_sidecar(sidecar_path, sidecar)
log.info(
"WaveformStore.save_imported_idf serial=%s filename=%s filesize=%d "
"report_attached=%s",
serial, filename, filesize, bool(report),
serial, filename, filesize, bool(report_dict),
)
return ev, {
"filename": filename,