v0.19.0 - minimate compatability + family separation #22
@@ -4,6 +4,33 @@ All notable changes to seismo-relay are documented here.
|
||||
|
||||
---
|
||||
|
||||
## v0.19.0 — 2026-05-20
|
||||
|
||||
The "device-family separation" release. Tightens the boundary between Series III (MiniMate Plus / Blastware) and Series IV (Micromate / Thor) so the UI and storage layer dispatch deterministically by family instead of sniffing filename extensions or magnitude heuristics.
|
||||
|
||||
### Added — Phase 1: `device_family` column on `events`
|
||||
|
||||
- **`events.device_family TEXT`** — new column carrying `"series3"` or `"series4"`. Populated by every import path (`/db/import/blastware_file`, `/db/import/idf_file`, ACH server, BW CLI, sidecar backfill script). Returned through `/db/events` since `query_events` uses `SELECT *`.
|
||||
- **Self-applying migration** — on startup, `ALTER TABLE ... ADD COLUMN` lands the new column; a follow-on `UPDATE` backfills existing rows from the binary filename extension (`.IDFH`/`.IDFW` → `series4`, everything else → `series3`). No manual SQL needed.
|
||||
- **UPSERT preserves family** — re-imports without an explicit family don't blank existing rows (`COALESCE(?, device_family)`).
|
||||
- **UI dispatches on the column** — `sfm_webapp.html` events-table mic formatter now branches on `ev.device_family === 'series4'` (Thor stores native dB(L); BW stores psi). Modal uses `source.kind === 'idf-import'` from the sidecar (sidecars don't carry the DB column). Source-files section labels changed from "BW filename / BW filesize / BW sha256" to format-neutral "Event file / File size / File sha256".
|
||||
|
||||
### Added — Phase 2: `micromate/` package alongside `minimateplus/`
|
||||
|
||||
- **`micromate/`** — new sibling package for the Thor / Micromate Series IV device. Currently scoped to offline-file ingest; live-device support (TCP transport, framing, protocol, client) will land here when reverse-engineering happens.
|
||||
- `micromate/idf_ascii_report.py` — moved from `sfm/idf_ascii_report.py`. No behaviour change.
|
||||
- `micromate/models.py` — typed `IdfReport`, `IdfEvent`, `IdfPeaks`, `IdfProjectInfo`, `IdfSensorCheck`. Stores mic in native `mic_pspl_dbl` (dB(L)) instead of the pseudo-psi shoehorn that the BW-shaped model uses. `IdfEvent.from_report()` constructs from a parsed dict + filename; `IdfEvent.to_minimateplus_event(waveform_key)` bridges to the existing sidecar / DB-insert machinery.
|
||||
- `micromate/idf_file.py` — placeholder for the binary codec (`.IDFH` / `.IDFW`). Stubbed `read_idf_file()` raises `NotImplementedError`; documents the planned reverse-engineering path.
|
||||
- **`WaveformStore.save_imported_idf`** refactored to use the native `IdfEvent` and bridge at the SQL-insert boundary. Cleaner separation of "parse a Thor event" (in `micromate/`) from "store it on disk + write a sidecar" (in `sfm/waveform_store.py`).
|
||||
- **Tests** — `tests/test_idf_ascii_report.py` imports updated to `micromate.idf_ascii_report`. All 1,014 example-data sidecars round-trip through `IdfEvent.from_report()` without errors.
|
||||
|
||||
### Companion releases
|
||||
|
||||
- **thor-watcher** unaffected — it talks to the relay over HTTP only. No version bump needed.
|
||||
- **terra-view** unaffected today; can use `device_family` in its event-detail rendering when convenient.
|
||||
|
||||
---
|
||||
|
||||
## v0.18.0 — 2026-05-19
|
||||
|
||||
The "Thor / Series IV ingest adapter" release. Seismo-relay can now accept event files from Instantel Micromate Series IV (Thor) units alongside the existing MiniMate Plus (Series III) Blastware pipeline.
|
||||
|
||||
@@ -1,7 +1,11 @@
|
||||
# seismo-relay `v0.17.0`
|
||||
# seismo-relay `v0.19.0`
|
||||
|
||||
A ground-up replacement for **Blastware** — Instantel's aging Windows-only
|
||||
software for managing MiniMate Plus seismographs.
|
||||
software for managing seismographs. Supports both the **MiniMate Plus
|
||||
(Series III)** and the **Micromate (Series IV / "Thor")** families:
|
||||
Series III via the live RS-232 / TCP wire protocol *and* Blastware ACH file
|
||||
ingest; Series IV currently via Thor TXT-paired IDF file ingest, with the
|
||||
binary codec on the roadmap.
|
||||
|
||||
Built in Python. Runs on Windows, Linux, or macOS. Connects to instruments
|
||||
over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
||||
@@ -19,6 +23,18 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
||||
> every Blastware ACH event lands in SeismoDb with device-authoritative
|
||||
> peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data,
|
||||
> without depending on the still-undecoded waveform body codec.
|
||||
> **v0.18.0 (2026-05-19)** adds Thor / Micromate Series IV ingest at
|
||||
> `/db/import/idf_file` — paired with **thor-watcher v0.3.0**, every
|
||||
> `.IDFH` / `.IDFW` event file (plus its `.txt` sidecar) lands in
|
||||
> SeismoDb the same way BW events do. See
|
||||
> [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md) for
|
||||
> the IDF format reference and reverse-engineering plan.
|
||||
> **v0.19.0 (2026-05-20)** separates Series III and Series IV at the
|
||||
> code level: new `micromate/` package alongside `minimateplus/`, new
|
||||
> `events.device_family` DB column ("series3" / "series4") so the UI
|
||||
> and storage layer dispatch deterministically instead of sniffing
|
||||
> filenames. Self-applying migration backfills existing rows from the
|
||||
> binary filename extension.
|
||||
> See [CHANGELOG.md](CHANGELOG.md) for full version history.
|
||||
|
||||
---
|
||||
@@ -29,17 +45,25 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
||||
seismo-relay/
|
||||
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Download + Console tabs)
|
||||
│
|
||||
├── minimateplus/ ← MiniMate Plus client library
|
||||
├── minimateplus/ ← Series III (MiniMate Plus) client library
|
||||
│ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport
|
||||
│ ├── protocol.py ← DLE frame layer, SUB command dispatch
|
||||
│ ├── client.py ← High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …)
|
||||
│ ├── framing.py ← Frame builders, DLE codec, S3FrameParser
|
||||
│ ├── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig, …
|
||||
│ ├── bw_ascii_report.py ← Parse BW per-event ASCII reports (.TXT sidecars)
|
||||
│ ├── event_file_io.py ← Read BW binaries, write .sfm.json sidecars
|
||||
│ └── blastware_file.py ← Write events to Blastware-compatible .AB0 files
|
||||
│
|
||||
├── micromate/ ← Series IV (Micromate / Thor) client library (NEW v0.19)
|
||||
│ ├── models.py ← IdfEvent, IdfReport, IdfPeaks, IdfProjectInfo, IdfSensorCheck (mic in native dB(L))
|
||||
│ ├── idf_ascii_report.py ← Parse Thor .IDFW.txt / .IDFH.txt event sidecars
|
||||
│ └── idf_file.py ← Stub for the .IDFW / .IDFH binary codec (reverse-engineering pending)
|
||||
│
|
||||
├── sfm/ ← SFM REST API server (FastAPI, port 8200)
|
||||
│ ├── server.py ← Live device endpoints + DB query endpoints + caching
|
||||
│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions, sessions table)
|
||||
│ ├── server.py ← Live device endpoints + DB query + ingest endpoints + caching
|
||||
│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions)
|
||||
│ ├── waveform_store.py ← On-disk store for BW + IDF event binaries + .sfm.json sidecars
|
||||
│ └── sfm_webapp.html ← Embedded web UI with Call Home config tab
|
||||
│
|
||||
├── bridges/
|
||||
@@ -56,7 +80,8 @@ seismo-relay/
|
||||
│ └── frame_db.py ← SQLite frame database
|
||||
│
|
||||
└── docs/
|
||||
└── instantel_protocol_reference.md ← Reverse-engineered protocol spec
|
||||
├── instantel_protocol_reference.md ← Series III protocol spec (the Rosetta Stone)
|
||||
└── idf_protocol_reference.md ← Series IV (Thor IDF) format reference + codec RE plan
|
||||
```
|
||||
|
||||
---
|
||||
@@ -148,11 +173,23 @@ Query the SQLite database written by `ach_server.py`. All read-only except
|
||||
| Method | URL | Description |
|
||||
|--------|-----|-------------|
|
||||
| `GET` | `/db/units` | All known serials with summary stats |
|
||||
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger) |
|
||||
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger). Response rows include `device_family` ("series3" / "series4") so clients dispatch on unit type without sniffing filenames. |
|
||||
| `GET` | `/db/monitor_log` | Monitoring intervals |
|
||||
| `GET` | `/db/sessions` | ACH call-home session history |
|
||||
| `PATCH` | `/db/events/{id}/false_trigger?value=true` | Flag / unflag false triggers |
|
||||
|
||||
### File ingest endpoints
|
||||
|
||||
Used by watcher daemons to push field-collected event files into the SFM DB
|
||||
+ waveform store. Both accept multipart uploads of binary event files
|
||||
optionally paired with their ASCII sidecar reports; both dedup by
|
||||
`(serial, timestamp)` and UPSERT device-authoritative fields on re-import.
|
||||
|
||||
| Method | URL | Description |
|
||||
|--------|-----|-------------|
|
||||
| `POST` | `/db/import/blastware_file` | Series III: `.AB0*` / `.N00` binaries + paired `_ASCII.TXT`. Source: `series3-watcher`. |
|
||||
| `POST` | `/db/import/idf_file` | Series IV: `.IDFH` / `.IDFW` binaries + paired `.IDFW.txt` / `.IDFH.txt`. Source: `thor-watcher`. |
|
||||
|
||||
---
|
||||
|
||||
## minimateplus library
|
||||
@@ -214,22 +251,77 @@ not per individual event).
|
||||
|
||||
---
|
||||
|
||||
## micromate library
|
||||
|
||||
Series IV / Thor support, sibling to `minimateplus`. Currently scoped to
|
||||
offline-file ingest from Thor's TXT exporter; live-device protocol is
|
||||
deferred until the binary codec is cracked.
|
||||
|
||||
```python
|
||||
from micromate import IdfEvent, parse_idf_report
|
||||
|
||||
# Parse a .IDFW.txt / .IDFH.txt sidecar (1014 example files round-trip cleanly)
|
||||
text = open("UM11719_20231219162723.IDFW.txt").read()
|
||||
report_dict = parse_idf_report(text) # permissive dict
|
||||
|
||||
# Wrap into a typed event using the device-native binary filename
|
||||
event = IdfEvent.from_report(report_dict, "UM11719_20231219162723.IDFW")
|
||||
|
||||
event.serial # "UM11719"
|
||||
event.kind # "Waveform" or "Histogram"
|
||||
event.peaks.transverse_ips # 0.0251 (in/s, native unit)
|
||||
event.peaks.mic_pspl_dbl # 99.4 (dB(L), Thor's native mic unit — NOT psi)
|
||||
event.project_info.project # "UPMC Presby-Loc 3-Level1-1R Elevator Rm"
|
||||
event.sensor_check.tran # True (passed self-check)
|
||||
event.firmware_version # "Micromate ISEE 11.0AK"
|
||||
event.calibration_text # "November 22, 2023 by Instantel"
|
||||
|
||||
# Bridge to the existing minimateplus.Event shape for the DB / sidecar paths
|
||||
# (waveform_key is a 16-byte sha256 prefix when ingesting from a binary file)
|
||||
bridged_event = event.to_minimateplus_event(waveform_key=b"\x00" * 16)
|
||||
```
|
||||
|
||||
The binary codec (`.IDFW` / `.IDFH` event files themselves) is on the
|
||||
roadmap — see [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md)
|
||||
for everything known so far, the two observed file signatures, and the
|
||||
reverse-engineering plan. The `micromate/idf_file.py` stub is where
|
||||
`read_idf_file()` will land.
|
||||
|
||||
---
|
||||
|
||||
## Database
|
||||
|
||||
`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode) using the
|
||||
`SeismoDb` persistence layer. Four tables, all unit-keyed by serial number:
|
||||
`ach_server.py` and the file-ingest endpoints write to
|
||||
`bridges/captures/seismo_relay.db` (SQLite, WAL mode) via the `SeismoDb`
|
||||
persistence layer. Three tables, all unit-keyed by serial number:
|
||||
|
||||
| Table | Key | Contents |
|
||||
|-------|-----|----------|
|
||||
| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds |
|
||||
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag |
|
||||
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
|
||||
| `events.false_trigger` | Boolean flag | PATCH endpoint to mark/unmark false triggers for review |
|
||||
| `events` | UUID, UNIQUE(serial, timestamp) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag, **`device_family`** ("series3" / "series4"), `blastware_filename` (binary at-rest in `waveforms/`), sidecar references |
|
||||
| `monitor_log` | UUID, UNIQUE(serial, start_time) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
|
||||
|
||||
Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs never
|
||||
produce duplicate rows. Post-erase key reuse is handled automatically via the
|
||||
high-water mark in `ach_state.json`. Key-based state tracking allows correct
|
||||
handling of device erasures (external or post-download).
|
||||
**Deduplication is by `(serial, timestamp)`** — the device clock is the
|
||||
stable natural key. Repeat call-homes or re-runs UPSERT the row in place,
|
||||
refreshing every device-authoritative field (peaks, project strings,
|
||||
sample_rate, file references) so the latest writer wins. `false_trigger`
|
||||
and `device_family` are preserved across UPSERTs. Earlier versions used
|
||||
`(serial, waveform_key)` for dedup, but the device's event-key counter
|
||||
resets to `0x01110000` after every erase, so timestamps are the correct
|
||||
dedup field. Migration handles the transition transparently on first
|
||||
startup.
|
||||
|
||||
**`device_family` (added v0.19.0)** discriminates Series III from Series
|
||||
IV at the SQL level. Set by every import path; the UI dispatches on it
|
||||
to render mic units correctly (Series III: psi → dBL conversion; Series
|
||||
IV: native dBL passthrough). Existing rows are backfilled at first
|
||||
startup of v0.19.0+ by sniffing the binary filename extension.
|
||||
|
||||
The on-disk waveform store lives at `bridges/captures/waveforms/<serial>/`
|
||||
and holds the original event binaries (BW `.AB0*` / `.N00` for Series III,
|
||||
`.IDFH` / `.IDFW` for Series IV) plus their `.sfm.json` review/metadata
|
||||
sidecars. Series III events also produce `.a5.pkl` source-frame pickles
|
||||
and `.h5` clean-waveform exports; Series IV doesn't yet (pending codec).
|
||||
|
||||
---
|
||||
|
||||
@@ -311,18 +403,27 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
|
||||
|
||||
## Key Features
|
||||
|
||||
**Device support:**
|
||||
- [x] Full read/write/erase pipelines
|
||||
**Series III (MiniMate Plus) device support:**
|
||||
- [x] Full read/write/erase pipelines over RS-232 or TCP/cellular
|
||||
- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings)
|
||||
- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries)
|
||||
- [x] Monitor control (start/stop, status polling, battery/memory)
|
||||
- [x] Monitor log entries (continuous monitoring intervals without full waveform download)
|
||||
- [x] Blastware file ingest at `/db/import/blastware_file` (paired with `series3-watcher`)
|
||||
|
||||
**Series IV (Micromate / Thor) device support:**
|
||||
- [x] Thor IDF file ingest at `/db/import/idf_file` (paired with `thor-watcher`, v0.18.0+)
|
||||
- [x] Native `IdfEvent` / `IdfReport` typed models — mic in dB(L), full title strings, sensor self-check, calibration, firmware version
|
||||
- [x] Parser verified against 1,014 paired `.txt` sidecars in `thor-watcher/example-data/`
|
||||
- [ ] Binary `.IDFW` / `.IDFH` codec — pending (see Roadmap + [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md))
|
||||
- [ ] Live-device protocol — pending codec
|
||||
|
||||
**Data persistence:**
|
||||
- [x] SQLite database (`seismo_relay.db`) with 4 tables: ach_sessions, events, monitor_log, plus false_trigger flag
|
||||
- [x] Deduplication by waveform key (handles re-runs and repeat call-homes)
|
||||
- [x] Post-erase key-reuse detection (tracks high-water mark)
|
||||
- [x] Session state (`ach_state.json`) with downloaded keys and max key
|
||||
- [x] SQLite database (`seismo_relay.db`) with `events`, `monitor_log`, `ach_sessions` tables
|
||||
- [x] Per-row `device_family` column ("series3" / "series4") for clean UI / unit-of-measurement dispatch (v0.19.0+)
|
||||
- [x] Deduplication by `(serial, timestamp)` — natural key handles post-erase counter resets
|
||||
- [x] UPSERT on re-import refreshes every device-authoritative field (peaks, project, sample_rate); preserves operator review state (`false_trigger`)
|
||||
- [x] Post-erase key-reuse detection (tracks high-water mark in `ach_state.json`)
|
||||
|
||||
**REST API:**
|
||||
- [x] Live device endpoints with in-memory caching (`_LiveCache`)
|
||||
@@ -330,6 +431,7 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
|
||||
- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH)
|
||||
- [x] Call Home config read/write endpoints
|
||||
- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`)
|
||||
- [x] Import endpoints for both device families (`/db/import/blastware_file`, `/db/import/idf_file`)
|
||||
|
||||
**File output (v0.7+, byte-perfect as of v0.14.3):**
|
||||
- [x] Blastware-compatible `.AB0` / `.G10` file generation (waveform + metadata)
|
||||
@@ -359,8 +461,10 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
|
||||
|
||||
### High-impact (unblocks product features)
|
||||
|
||||
- [ ] **Waveform body codec reverse-engineering.** The 5A bulk-stream body is some kind of compressed/encoded format (not raw int16 LE as previously assumed — see §7.6.1 retraction in `docs/instantel_protocol_reference.md`). Structural framing is ~50% decoded on branch `claude/codec-re-cBGNe` (tagged-block walker, segment counters); per-byte sample mapping is still open. Until this lands, the in-app waveform viewer renders garbage and BW-import peak values fall back to `_peaks_from_samples()` saturation noise. Workaround: pair every BW-imported event with its `_ASCII.TXT` so the device-authoritative peaks land in the DB regardless of codec.
|
||||
- [ ] **In-app waveform viewer accuracy.** Depends on codec decode. Plot.v1 JSON pipeline + viewer skeleton already exist; will start showing real waveforms automatically once `_decode_a5_waveform` produces correct samples.
|
||||
- [ ] **Series III waveform body codec reverse-engineering.** The 5A bulk-stream body is some kind of compressed/encoded format (not raw int16 LE as previously assumed — see §7.6.1 retraction in `docs/instantel_protocol_reference.md`). Structural framing is ~50% decoded on branch `claude/codec-re-cBGNe` (tagged-block walker, segment counters); per-byte sample mapping is still open. Until this lands, the in-app waveform viewer renders garbage and BW-import peak values fall back to `_peaks_from_samples()` saturation noise. Workaround: pair every BW-imported event with its `_ASCII.TXT` so the device-authoritative peaks land in the DB regardless of codec.
|
||||
- [ ] **Series IV (Thor IDF) binary codec reverse-engineering.** `.IDFH` / `.IDFW` files are currently stored opaquely by `WaveformStore.save_imported_idf`, with all metadata sourced from the paired `.txt` sidecar. This works because thor-watcher forwards both files together, but operators who haven't enabled Thor's TXT exporter get rows with NULL peaks. Cracking the binary closes that gap and unlocks waveform display. Starting-point reference at [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md) — two observed file signatures (1,012 newer-firmware files + 2 old files whose layout matches the Series III STRT-record format), suggested first-session plan (~2-4 hrs), 1,014 paired binary+txt files available as ground truth in `thor-watcher/example-data/`. Code seam ready at `micromate/idf_file.py`.
|
||||
- [ ] **In-app waveform viewer accuracy.** Depends on Series III codec decode. Plot.v1 JSON pipeline + viewer skeleton already exist; will start showing real waveforms automatically once `_decode_a5_waveform` produces correct samples. Series IV waveforms come online when the IDF codec lands.
|
||||
- [ ] **Series IV live-device support.** Once the IDF binary is decoded, extend `micromate/` with `transport.py` / `framing.py` / `protocol.py` / `client.py` mirroring the `minimateplus/` package layout — depends on capturing Thor's wire protocol (TCP / RS-232 captures TBD).
|
||||
- [ ] **Terra-view integration** — seismo-relay router, unit detail page, VISON-style event listing.
|
||||
- [ ] **Vibration summary reports** — highest legit PPV per project → Word doc (false-trigger filtering first).
|
||||
|
||||
|
||||
@@ -516,6 +516,7 @@ class AchSession:
|
||||
serial=serial or self.peer,
|
||||
session_id=None,
|
||||
waveform_records=waveform_records,
|
||||
device_family="series3",
|
||||
)
|
||||
_ml_ins, _ml_skip = self.db.insert_monitor_log(
|
||||
new_monitor_entries, session_id=None
|
||||
|
||||
@@ -0,0 +1,284 @@
|
||||
# IDF Protocol Reference — Thor / Micromate Series IV
|
||||
|
||||
Starting-point reference for reverse-engineering Instantel's Micromate
|
||||
Series IV event-file format. Sibling to
|
||||
[instantel_protocol_reference.md](instantel_protocol_reference.md) (the
|
||||
Series III "Rosetta Stone") — this doc holds what we know so far and
|
||||
the open questions still to crack.
|
||||
|
||||
**Status (2026-05-20):** ASCII text sidecar fully decoded (1,014
|
||||
sample files round-trip). Binary `.IDFH` / `.IDFW` codec
|
||||
**not yet implemented** — binaries are stored opaquely by
|
||||
`WaveformStore.save_imported_idf`, with metadata sourced from the
|
||||
paired `.txt` sidecar.
|
||||
|
||||
---
|
||||
|
||||
## File model
|
||||
|
||||
### Filename convention
|
||||
|
||||
```
|
||||
<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>
|
||||
```
|
||||
|
||||
- **SERIAL** — literal device serial, two-letter prefix + numeric
|
||||
suffix. Examples seen: `UM11719`, `UM13981`, `UM20147`, `BE9439`.
|
||||
Unlike Series III BW filenames (`M529LK44.AB0`, base-36 stem),
|
||||
Series IV filenames carry the serial in plain text.
|
||||
- **YYYYMMDDHHMMSS** — 14-char ASCII timestamp in **device local
|
||||
time** (no timezone marker).
|
||||
- **KIND** — `IDFH` for histograms, `IDFW` for waveforms.
|
||||
|
||||
The `.IDFH.txt` / `.IDFW.txt` ASCII sidecar lives in a `TXT/`
|
||||
**subfolder** of the unit's directory, not alongside the binary.
|
||||
This pairing convention is encoded in
|
||||
`event_forwarder.idf_report_path()`.
|
||||
|
||||
### Directory layout
|
||||
|
||||
```
|
||||
C:\THORDATA\
|
||||
└── <Project>\
|
||||
└── <UM####>\ ← unit serial dir
|
||||
├── UM12345_20260520100000.MLG ← monitor log (not events)
|
||||
├── UM12345_20260520100000.IDFH ← histogram event (binary)
|
||||
├── UM12345_20260520100000.IDFW ← waveform event (binary)
|
||||
├── UM12345_20260520100000.IDFW.CDB ← cache-DB variant (skip)
|
||||
├── TXT\
|
||||
│ ├── UM12345_20260520100000.IDFH.txt ← histogram ASCII sidecar
|
||||
│ └── UM12345_20260520100000.IDFW.txt ← waveform ASCII sidecar
|
||||
├── CSV\, HTML\, PDF\, XML\ ← operator-facing derived exports
|
||||
└── ...
|
||||
```
|
||||
|
||||
The `.IDFW.CDB` files share the binary's basename but appear to be a
|
||||
separate cache/database variant. Their first 8 bytes match the
|
||||
**old**-firmware Thor signature (see below) regardless of which
|
||||
signature the paired `.IDFW` uses. Purpose unknown; sizes vary
|
||||
wildly (observed 123 B → 40,491 B). Thor-watcher's forwarder
|
||||
deliberately skips them.
|
||||
|
||||
### Sample corpus
|
||||
|
||||
The `thor-watcher/example-data/THORDATA_example/` tree carries
|
||||
**1,014 paired .IDFW / .IDFH + .txt files** spanning 2020–2023
|
||||
across nine units (UM11719, UM13981, UM20147, …, plus BE9439 from
|
||||
2020). This is the reverse-engineering ground truth.
|
||||
|
||||
---
|
||||
|
||||
## ASCII sidecar (`.IDFW.txt` / `.IDFH.txt`) — fully decoded
|
||||
|
||||
Shape: plain text, one `"Key : Value"` line per metadata field,
|
||||
followed for waveforms by a tab-separated sample table headed by
|
||||
the literal line `Waveform Data Channels`. Parsed by
|
||||
[`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py).
|
||||
See [`micromate/models.py`](../micromate/models.py) for the typed
|
||||
`IdfReport` shape.
|
||||
|
||||
### Notable conventions
|
||||
|
||||
- **Units are native to Thor** — geophone in **in/s**, microphone in
|
||||
**dB(L)** (not psi like Series III BW reports), frequency in Hz,
|
||||
acceleration in g, displacement in in.
|
||||
- **Below-threshold readings** appear as the literal string
|
||||
`<0.005 in/s` (155 occurrences in the sample corpus) — the parser
|
||||
strips the `<` and treats the numeric remainder as the value.
|
||||
- **Out-of-range / not-measured** values appear as `N/A` — parser
|
||||
drops the field rather than letting the string leak into a numeric
|
||||
column.
|
||||
- **Firmware string** observed: `Micromate ISEE 11.0AK`.
|
||||
- **TitleString1..4** are operator-defined free-text slots; Thor's
|
||||
default labels map them to Location / Client / Company / Notes,
|
||||
which the parser surfaces as `project` / `client` / `operator` /
|
||||
`notes`.
|
||||
- **Histogram sidecars** use `HistogramStartDate` / `HistogramStartTime`
|
||||
in place of waveform's `EventDate` / `EventTime`. Parser falls
|
||||
through to either.
|
||||
- **Histogram tabular block** lacks the `Waveform Data Channels`
|
||||
marker; instead it's a multi-line column header followed by
|
||||
per-interval rows (`<date> <time> <tran-ppv> <freq> ...`). Parser
|
||||
silently ignores lines after the metadata block since they lack a
|
||||
colon-separated `key : value` shape (the timestamps DO contain
|
||||
colons but produce garbage keys that don't collide with any
|
||||
recognised field).
|
||||
|
||||
---
|
||||
|
||||
## Binary header signatures (observed)
|
||||
|
||||
Hex dump of the first 32 bytes across 1,014 sample files reveals
|
||||
**two distinct file signatures**, both anchored by the literal
|
||||
ASCII string `"\x00Instantel\x00"` at offset 6–16:
|
||||
|
||||
### Signature A — newer firmware (1,012 files, 99.8% of corpus)
|
||||
|
||||
```
|
||||
00000000: 0012 0100 0000 496e 7374 616e 7465 6c00 ......Instantel.
|
||||
00000010: 0000 a695 002e b500 4f70 6572 6174 6f72 ........Operator
|
||||
^^^^^^^^^^^^^^^^
|
||||
operator/title string starts at 0x18
|
||||
```
|
||||
|
||||
Header bytes 0–5: `00 12 01 00 00 00`. Followed immediately by the
|
||||
8-byte ASCII tag, then 6 unknown bytes, then ASCII operator-supplied
|
||||
strings (Operator name, etc.) and on through the project / client /
|
||||
title strings. No `STRT` record observed in this layout.
|
||||
|
||||
### Signature B — older firmware (2 files: BE9439 from 2020)
|
||||
|
||||
```
|
||||
00000000: 1000 0180 0000 496e 7374 616e 7465 6c00 ......Instantel.
|
||||
00000010: 072c 0012 0300 5354 5254 fffe 0111 2340 .,....STRT....#@
|
||||
^^^^^^^^^ ^^^^^^^^^
|
||||
STRT magic 4-byte end_key
|
||||
00000020: 0111 0000 2e5f 00ac 4600 0000 0200 0000 ....._..F.......
|
||||
^^^^^^^^^ ^^^
|
||||
4-byte start_key 0x46 (BW WAVEHDR record-type marker)
|
||||
```
|
||||
|
||||
Header bytes 0–5: `10 00 01 80 00 00`. The structure after the
|
||||
`Instantel` magic is **byte-for-byte identical to a BW SUB 5A
|
||||
probe-response STRT record** as documented in
|
||||
[instantel_protocol_reference.md → "SUB 5A — STRT record encodes
|
||||
end_offset"](instantel_protocol_reference.md). Specifically:
|
||||
|
||||
| Offset | Bytes | Meaning (per BW reference) |
|
||||
|--------|---------------------|--------------------------------------|
|
||||
| 0x14 | `53 54 52 54` | `STRT` magic |
|
||||
| 0x18 | `ff fe` | STRT sentinel |
|
||||
| 0x1A | `01 11 23 40` | `end_key` (4 bytes) |
|
||||
| 0x1E | `01 11 00 00` | `start_key` (4 bytes) |
|
||||
| 0x26 | `46` | `0x46` waveform-record type marker |
|
||||
|
||||
**Hypothesis:** Older Micromate firmware writes a wrapped BW-format
|
||||
event into the `.IDFW` file — essentially the same on-disk shape as
|
||||
a Series III device, with the new filename convention applied at
|
||||
export time. Newer firmware (signature A) abandoned the
|
||||
BW-compatible layout for an Instantel-specific format.
|
||||
|
||||
If that hypothesis holds, the 2 signature-B files can already be
|
||||
parsed via `minimateplus/event_file_io.read_blastware_file()` — worth
|
||||
testing. The 1,012 signature-A files are the real reverse-engineering
|
||||
target.
|
||||
|
||||
### `.IDFW.CDB` cache files
|
||||
|
||||
Always carry signature B (`10 00 01 80 ...`), even when the paired
|
||||
`.IDFW` carries signature A. Plausible explanation: the CDB is an
|
||||
internal Thor cache-database export that retains the legacy BW-style
|
||||
record layout regardless of the user-facing `.IDFW` format version.
|
||||
Not currently consumed by the forwarder.
|
||||
|
||||
---
|
||||
|
||||
## File-size patterns (Signature A, the main target)
|
||||
|
||||
Survey of 1,012 signature-A files:
|
||||
|
||||
| Event type | Typical size | Source of variance |
|
||||
|--------------|-------------------|----------------------------------------------|
|
||||
| `.IDFW` 2-sec | 9,200 – 10,500 B | Operator-supplied strings (TitleString1..4) of varying length |
|
||||
| `.IDFH` | 2,944 – 4,076 B | Histogram interval count (record duration / interval) |
|
||||
|
||||
**Naive arithmetic for 2-sec waveform:**
|
||||
- 4 channels × 2 sec × 1024 sps = 8,192 samples
|
||||
- At 2 bytes/sample (int16) = 16,384 sample bytes → file would be > 16 KB
|
||||
- Observed: ~9–10 KB
|
||||
- → samples are likely **1 byte each** (int8 quantised), **or** stored
|
||||
with bit-packing / delta encoding, **or** only one channel's
|
||||
full-rate samples are stored with the others reconstructed
|
||||
arithmetically. Verifying this is the **first RE milestone**.
|
||||
|
||||
Project-string–length variance (~1 KB across the corpus) is consistent
|
||||
with the file carrying a single copy of each TitleString1..4 plus
|
||||
operator + setup-name as null-padded ASCII regions.
|
||||
|
||||
---
|
||||
|
||||
## Open questions
|
||||
|
||||
The reverse-engineering targets, roughly in dependency order:
|
||||
|
||||
1. **Sample encoding (signature A)** — int8? int16 LE/BE? Bit-packed?
|
||||
Delta-coded? Per-channel interleaved or sequential blocks?
|
||||
2. **Header field layout (signature A)** — where do sample_rate,
|
||||
record_time, channel count, and per-channel peaks live in the
|
||||
binary? The ASCII sidecar gives the device-authoritative values,
|
||||
so binary fields can be confirmed by diff.
|
||||
3. **Operator-string offsets** — `Operator` at 0x18 is the first
|
||||
visible string in signature-A files; the rest (project, client,
|
||||
notes, setup) follow. Need to map exact offsets and null-padding
|
||||
conventions.
|
||||
4. **Signature-B → BW codec compatibility** — does
|
||||
`minimateplus/event_file_io.read_blastware_file()` actually parse
|
||||
the 2 BE9439 signature-B files as-is? If yes, the OLD-format
|
||||
ingest is free.
|
||||
5. **`.IDFW.CDB` purpose** — is it an internal Thor cache, a
|
||||
ring-buffer dump, or something else? Worth a single small effort
|
||||
to characterise so we know what we're skipping.
|
||||
6. **Footer / checksum** — every BW event file has a footer; does
|
||||
IDF? Where does the per-channel sample block end?
|
||||
|
||||
---
|
||||
|
||||
## Reverse-engineering playbook (when we start)
|
||||
|
||||
The Series III BW codec took ~2 months of MITM wire captures
|
||||
because we didn't have ground-truth metadata. Thor's situation is
|
||||
**substantially better**:
|
||||
|
||||
- **Ground truth is on disk.** Every binary in `example-data/`
|
||||
has a paired `.IDFW.txt` carrying the full decoded sample table
|
||||
(`Waveform Data Channels` block — see any sample file in
|
||||
`thor-watcher/example-data/.../TXT/`). Aligning binary bytes
|
||||
to the table's float-per-row values gives an immediate per-byte
|
||||
hypothesis test.
|
||||
- **Cross-event diffing.** 1,012 signature-A samples from 9 units
|
||||
spanning 4 years means any field that varies between events is
|
||||
immediately localisable. Fields that are constant across all
|
||||
files (firmware ID, channel labels, format-version word) are also
|
||||
immediately localisable by complementary search.
|
||||
- **No protocol surface.** Files at rest, not a wire dialect. No
|
||||
DLE stuffing, no inner-frame parsing, no probe/data two-step.
|
||||
|
||||
Suggested first session (2-4 hours): hand-decode `UM11719_20231219162723.IDFW`
|
||||
(10,290 bytes) against its `TXT/UM11719_20231219162723.IDFW.txt`
|
||||
sample table (the 2-sec waveform at 1024 sps × 4 channels = 8,192
|
||||
sample rows). Find the first per-channel sample value (`0.0003` in
|
||||
the Tran column at t=0) in the binary. Confirms sample encoding.
|
||||
Everything else flows from there.
|
||||
|
||||
---
|
||||
|
||||
## Code seams ready to receive the codec
|
||||
|
||||
When the codec lands, it goes into
|
||||
[`micromate/idf_file.py`](../micromate/idf_file.py) (currently a
|
||||
stub raising `NotImplementedError`). Public API:
|
||||
|
||||
```python
|
||||
from micromate import IdfEvent
|
||||
from micromate.idf_file import read_idf_file
|
||||
|
||||
event: IdfEvent = read_idf_file(Path("UM11719_20231219163444.IDFW"))
|
||||
# event.peaks.transverse_ips, event.timestamp, event.raw_samples, ...
|
||||
```
|
||||
|
||||
The ingest pipeline (`WaveformStore.save_imported_idf`) currently
|
||||
builds the `IdfEvent` from the `.txt` parser only. Once
|
||||
`read_idf_file()` works, the binary becomes authoritative; the
|
||||
`.txt` parser drops to fast-path metadata cross-check. Operators
|
||||
who don't enable Thor's TXT exporter still get fully populated
|
||||
events.
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- [instantel_protocol_reference.md](instantel_protocol_reference.md) — Series III BW protocol reference (the Rosetta Stone). STRT record format, DLE framing, BW filename encoding.
|
||||
- [`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py) — `.txt` sidecar parser.
|
||||
- [`micromate/models.py`](../micromate/models.py) — `IdfEvent`, `IdfReport` typed dataclasses.
|
||||
- [`micromate/idf_file.py`](../micromate/idf_file.py) — placeholder for the binary codec.
|
||||
- [`thor-watcher/example-data/THORDATA_example/`](../../thor-watcher/example-data/) — 1,014 paired binary + .txt files for codec validation.
|
||||
@@ -0,0 +1,48 @@
|
||||
"""
|
||||
micromate — Instantel Micromate (Series IV) device library.
|
||||
|
||||
Sibling of ``minimateplus`` (the Series III library). Currently scoped to
|
||||
the offline-file ingest path used by thor-watcher: parsing the per-event
|
||||
``.IDFH``/``.IDFW`` ASCII text sidecars Thor's exporter writes alongside
|
||||
each binary event file, and wrapping the parsed data in typed event
|
||||
records.
|
||||
|
||||
Live-device support (TCP protocol, frame parsing, real-time monitoring)
|
||||
is deferred — when we add it, it lands here as ``transport.py`` /
|
||||
``framing.py`` / ``protocol.py`` / ``client.py``, mirroring the
|
||||
``minimateplus`` package layout.
|
||||
|
||||
Typical usage (offline file ingest):
|
||||
|
||||
from micromate import IdfEvent, parse_idf_report
|
||||
|
||||
text = open("UM11719_20231219162723.IDFW.txt").read()
|
||||
rep = parse_idf_report(text) # dict
|
||||
event = IdfEvent.from_report(rep, "UM11719_20231219162723.IDFW")
|
||||
print(event.serial, event.peaks.transverse_ips, event.mic_pspl_dbl)
|
||||
"""
|
||||
|
||||
from .idf_ascii_report import (
|
||||
parse_event_filename,
|
||||
parse_idf_report,
|
||||
serial_from_filename,
|
||||
)
|
||||
from .models import (
|
||||
IdfEvent,
|
||||
IdfPeaks,
|
||||
IdfProjectInfo,
|
||||
IdfReport,
|
||||
IdfSensorCheck,
|
||||
)
|
||||
|
||||
__version__ = "0.1.0"
|
||||
__all__ = [
|
||||
"IdfEvent",
|
||||
"IdfPeaks",
|
||||
"IdfProjectInfo",
|
||||
"IdfReport",
|
||||
"IdfSensorCheck",
|
||||
"parse_event_filename",
|
||||
"parse_idf_report",
|
||||
"serial_from_filename",
|
||||
]
|
||||
@@ -1,5 +1,5 @@
|
||||
"""
|
||||
sfm/idf_ascii_report.py — parse Thor (Micromate Series IV) IDF ASCII reports.
|
||||
micromate/idf_ascii_report.py — parse Thor (Micromate Series IV) IDF ASCII reports.
|
||||
|
||||
Thor exports a `.IDFW.txt` or `.IDFH.txt` sidecar next to each `.IDFW`
|
||||
(waveform) or `.IDFH` (histogram) event binary. Each sidecar is a
|
||||
@@ -65,9 +65,17 @@ def _normalize_key(raw: str) -> str:
|
||||
|
||||
|
||||
def _strip_unit_suffix(value: str) -> str:
|
||||
"""Return the numeric part of values like "0.2119 in/s" → "0.2119"."""
|
||||
"""Return the numeric part of values like "0.2119 in/s" → "0.2119".
|
||||
|
||||
Also strips Thor's below/above-threshold prefixes:
|
||||
"<0.005 in/s" → "0.005" (below-noise-floor reading)
|
||||
">100 Hz" → "100" (above-measurement-range reading)
|
||||
"""
|
||||
parts = value.strip().split()
|
||||
return parts[0] if parts else value.strip()
|
||||
token = parts[0] if parts else value.strip()
|
||||
if token.startswith("<") or token.startswith(">"):
|
||||
token = token[1:]
|
||||
return token
|
||||
|
||||
|
||||
def _parse_float(value: str) -> Optional[float]:
|
||||
@@ -178,38 +186,54 @@ def parse_idf_report(text: Union[str, bytes]) -> Dict[str, Any]:
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
# Numeric scalars
|
||||
for key in ("sample_rate",):
|
||||
# Numeric scalars. For every field we typify here, we MUST drop the
|
||||
# raw string copy from `out` when parsing fails — Thor writes things
|
||||
# like "<0.005 in/s" (below threshold) and "N/A" (not measured) that
|
||||
# would otherwise linger in `out` as strings, sneak into SQLite REAL
|
||||
# columns via permissive type affinity, and then crash the JS
|
||||
# frontend on `.toFixed(...)`.
|
||||
int_fields = ("sample_rate",)
|
||||
for key in int_fields:
|
||||
v = raw.get(key)
|
||||
if v is not None:
|
||||
iv = _parse_int(v)
|
||||
if iv is not None:
|
||||
out[key] = iv
|
||||
if v is None:
|
||||
continue
|
||||
iv = _parse_int(v)
|
||||
if iv is not None:
|
||||
out[key] = iv
|
||||
else:
|
||||
out.pop(key, None)
|
||||
|
||||
for key in ("tran_ppv", "vert_ppv", "long_ppv", "peak_vector_sum",
|
||||
"tran_zc_freq", "vert_zc_freq", "long_zc_freq",
|
||||
"tran_peak_acceleration", "vert_peak_acceleration",
|
||||
"long_peak_acceleration",
|
||||
"tran_peak_displacement", "vert_peak_displacement",
|
||||
"long_peak_displacement",
|
||||
"tran_time_of_peak", "vert_time_of_peak", "long_time_of_peak",
|
||||
"mic_time_of_peak", "mic_zc_freq"):
|
||||
float_fields = (
|
||||
"tran_ppv", "vert_ppv", "long_ppv", "peak_vector_sum",
|
||||
"tran_zc_freq", "vert_zc_freq", "long_zc_freq",
|
||||
"tran_peak_acceleration", "vert_peak_acceleration",
|
||||
"long_peak_acceleration",
|
||||
"tran_peak_displacement", "vert_peak_displacement",
|
||||
"long_peak_displacement",
|
||||
"tran_time_of_peak", "vert_time_of_peak", "long_time_of_peak",
|
||||
"mic_time_of_peak", "mic_zc_freq",
|
||||
)
|
||||
for key in float_fields:
|
||||
v = raw.get(key)
|
||||
if v is not None:
|
||||
fv = _parse_float(v)
|
||||
if fv is not None:
|
||||
out[key] = fv
|
||||
if v is None:
|
||||
continue
|
||||
fv = _parse_float(v)
|
||||
if fv is not None:
|
||||
out[key] = fv
|
||||
else:
|
||||
out.pop(key, None)
|
||||
|
||||
# Microphone — Thor reports MicPSPL (dB(L)) which is the closest
|
||||
# analogue to BW's mic_ppv. Stored as a float; units are in the
|
||||
# original raw field (`mic_pspl` raw entry preserves "99.4 dB(L)").
|
||||
# analogue to BW's mic_ppv. The raw "99.4 dB(L)" string stays in
|
||||
# `out` under the original `mic_pspl` key for display; the parsed
|
||||
# float goes in `mic_ppv`.
|
||||
mic = raw.get("mic_pspl")
|
||||
if mic is not None:
|
||||
fv = _parse_float(mic)
|
||||
if fv is not None:
|
||||
out["mic_ppv"] = fv
|
||||
|
||||
# Record / pre-trigger duration
|
||||
# Record / pre-trigger duration — same drop-on-failure discipline.
|
||||
rt = raw.get("record_time")
|
||||
if rt is not None:
|
||||
fv = _parse_float(rt)
|
||||
@@ -0,0 +1,64 @@
|
||||
"""
|
||||
micromate/idf_file.py — placeholder for the Thor IDF binary codec.
|
||||
|
||||
Thor's ``.IDFH`` (histogram) and ``.IDFW`` (waveform) event files are an
|
||||
Instantel proprietary binary format that has not yet been reverse-
|
||||
engineered. Today seismo-relay treats them as opaque blobs:
|
||||
``WaveformStore.save_imported_idf`` stores the bytes verbatim and reads
|
||||
all device-authoritative metadata from the paired ``.IDFW.txt`` /
|
||||
``.IDFH.txt`` ASCII sidecar (parsed by ``idf_ascii_report.py``).
|
||||
|
||||
When we crack the binary codec — same reverse-engineering playbook we
|
||||
used to byte-perfect-parse Series III BW files (see
|
||||
``docs/instantel_protocol_reference.md`` and ``minimateplus/event_file_io.py``)
|
||||
— this module will grow:
|
||||
|
||||
- ``read_idf_file(path) -> IdfEvent``
|
||||
Parse a ``.IDFW``/``.IDFH`` binary and return a fully populated
|
||||
``IdfEvent`` whose waveform-sample arrays come from the binary
|
||||
(the .txt sidecar's tabular sample block being a best-effort
|
||||
check). Lets us ingest Thor events even when the operator
|
||||
hasn't enabled the .txt exporter — closing the
|
||||
``had_report=False`` gap that the thor-watcher forwarder
|
||||
currently tolerates as a known limitation.
|
||||
|
||||
- ``write_idf_file(path, event)`` (eventually)
|
||||
Round-trip event reconstruction, used for verifying the codec
|
||||
against captured device files the way ``write_blastware_file``
|
||||
verifies the Series III codec.
|
||||
|
||||
- Helpers for decoding the binary's per-channel sample arrays into
|
||||
physical units, the per-event flash buffer's monitor-log records,
|
||||
etc.
|
||||
|
||||
The reverse-engineering path: pair every ``.IDFW`` binary in
|
||||
``thor-watcher/example-data/`` with its sibling ``.IDFW.txt``, treating
|
||||
the txt's "Waveform Data Channels" block as ground-truth, and align
|
||||
the binary's per-channel int16-or-similar arrays against it. Header
|
||||
fields (sample rate, channel count, record time, timestamps) sit before
|
||||
the sample block — same approach as the BW codec where ASCII strings
|
||||
inside the binary (``Project:``, ``Client:``, etc.) anchored field
|
||||
discovery.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Union
|
||||
|
||||
from .models import IdfEvent
|
||||
|
||||
|
||||
def read_idf_file(path: Union[str, Path]) -> "IdfEvent":
|
||||
"""Parse a Thor ``.IDFW``/``.IDFH`` binary into an ``IdfEvent``.
|
||||
|
||||
Not yet implemented. When implemented, this will be the canonical
|
||||
entry point for reading Thor binaries — the ASCII sidecar parser
|
||||
becomes an optional fast-path metadata supplement rather than the
|
||||
sole source of device-authoritative data.
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
"IDF binary codec not yet implemented; the .IDFW/.IDFH binary format "
|
||||
"is undecoded. Use parse_idf_report() on the paired .txt sidecar "
|
||||
"for device-authoritative metadata."
|
||||
)
|
||||
@@ -0,0 +1,377 @@
|
||||
"""
|
||||
Micromate (Series IV / Thor) native data models.
|
||||
|
||||
These are the right-shaped dataclasses for Thor data — Thor measures
|
||||
the microphone in dB(L) directly, so this model carries
|
||||
``mic_pspl_dbl`` rather than the pseudo-``psi`` shoehorn that
|
||||
``minimateplus.PeakValues`` uses for Series III BW data.
|
||||
|
||||
The ingest pipeline today goes:
|
||||
|
||||
.IDFW.txt → parse_idf_report() → dict
|
||||
dict → IdfEvent.from_report() → IdfEvent (typed)
|
||||
IdfEvent → IdfEvent.to_minimateplus_event() → shape DB / sidecar
|
||||
machinery expects
|
||||
|
||||
The ``to_minimateplus_event()`` bridge is a temporary boundary — when we
|
||||
crack the binary IDF codec and have richer per-event data to store, the
|
||||
DB schema will grow Series-IV-specific columns and the bridge will
|
||||
shrink or disappear.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import datetime
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Dict, Optional, Tuple
|
||||
|
||||
|
||||
# ── IdfReport ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class IdfReport:
|
||||
"""Typed wrapper around the dict returned by ``parse_idf_report``.
|
||||
|
||||
All fields optional — Thor's exporter is permissive and some IDF .txt
|
||||
files (especially histograms) omit fields that waveform sidecars
|
||||
include. Use ``.raw`` for any field this dataclass hasn't surfaced
|
||||
yet (the parser keeps every recognised key in the raw dict).
|
||||
"""
|
||||
|
||||
# Identity / kind
|
||||
serial_number: Optional[str] = None
|
||||
event_type: Optional[str] = None # "Full Waveform" | "Full Histogram"
|
||||
event_datetime: Optional[datetime.datetime] = None
|
||||
filename: Optional[str] = None # echoed by Thor's exporter
|
||||
|
||||
# Sampling / timing
|
||||
sample_rate: Optional[int] = None # samples/sec
|
||||
record_time_sec: Optional[float] = None
|
||||
pre_trigger_sec: Optional[float] = None
|
||||
|
||||
# Geophone peaks (in/s)
|
||||
tran_ppv: Optional[float] = None
|
||||
vert_ppv: Optional[float] = None
|
||||
long_ppv: Optional[float] = None
|
||||
peak_vector_sum: Optional[float] = None
|
||||
|
||||
# Microphone — Thor's native unit is dB(L), NOT psi.
|
||||
mic_pspl_dbl: Optional[float] = None
|
||||
|
||||
# Zero-crossing frequencies (Hz)
|
||||
tran_zc_freq: Optional[float] = None
|
||||
vert_zc_freq: Optional[float] = None
|
||||
long_zc_freq: Optional[float] = None
|
||||
mic_zc_freq: Optional[float] = None
|
||||
|
||||
# Per-channel time of peak (sec, since event start)
|
||||
tran_time_of_peak: Optional[float] = None
|
||||
vert_time_of_peak: Optional[float] = None
|
||||
long_time_of_peak: Optional[float] = None
|
||||
mic_time_of_peak: Optional[float] = None
|
||||
|
||||
# Derived per-channel motion
|
||||
tran_peak_acceleration: Optional[float] = None # g
|
||||
vert_peak_acceleration: Optional[float] = None
|
||||
long_peak_acceleration: Optional[float] = None
|
||||
tran_peak_displacement: Optional[float] = None # in
|
||||
vert_peak_displacement: Optional[float] = None
|
||||
long_peak_displacement: Optional[float] = None
|
||||
|
||||
# Operator-supplied strings (Thor's TitleString1..4 → semantic slots)
|
||||
project: Optional[str] = None # TitleString1
|
||||
client: Optional[str] = None # TitleString2
|
||||
operator: Optional[str] = None # TitleString3
|
||||
notes: Optional[str] = None # TitleString4 / PostEventNote
|
||||
setup: Optional[str] = None # setup file name
|
||||
|
||||
# Sensor self-check results
|
||||
tran_test_passed: Optional[bool] = None
|
||||
vert_test_passed: Optional[bool] = None
|
||||
long_test_passed: Optional[bool] = None
|
||||
mic_test_passed: Optional[bool] = None
|
||||
|
||||
# Device-fixed metadata
|
||||
firmware_version: Optional[str] = None
|
||||
calibration_text: Optional[str] = None
|
||||
battery_volts: Optional[float] = None
|
||||
|
||||
# Original parser dict — preserves every recognised key (including
|
||||
# raw unit-suffixed strings) for forward-compatible field access.
|
||||
raw: Dict[str, Any] = field(default_factory=dict, repr=False)
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, d: Dict[str, Any]) -> "IdfReport":
|
||||
"""Build an IdfReport from the dict returned by ``parse_idf_report``."""
|
||||
ed = d.get("event_datetime")
|
||||
if isinstance(ed, str):
|
||||
try:
|
||||
ed = datetime.datetime.fromisoformat(ed)
|
||||
except ValueError:
|
||||
ed = None
|
||||
|
||||
return cls(
|
||||
serial_number = d.get("serial_number"),
|
||||
event_type = d.get("event_type"),
|
||||
event_datetime = ed if isinstance(ed, datetime.datetime) else None,
|
||||
filename = d.get("filename"),
|
||||
sample_rate = d.get("sample_rate"),
|
||||
record_time_sec = d.get("record_time_sec"),
|
||||
pre_trigger_sec = d.get("pre_trigger_sec"),
|
||||
tran_ppv = d.get("tran_ppv"),
|
||||
vert_ppv = d.get("vert_ppv"),
|
||||
long_ppv = d.get("long_ppv"),
|
||||
peak_vector_sum = d.get("peak_vector_sum"),
|
||||
mic_pspl_dbl = d.get("mic_ppv"), # parser names it mic_ppv (legacy)
|
||||
tran_zc_freq = d.get("tran_zc_freq"),
|
||||
vert_zc_freq = d.get("vert_zc_freq"),
|
||||
long_zc_freq = d.get("long_zc_freq"),
|
||||
mic_zc_freq = d.get("mic_zc_freq"),
|
||||
tran_time_of_peak = d.get("tran_time_of_peak"),
|
||||
vert_time_of_peak = d.get("vert_time_of_peak"),
|
||||
long_time_of_peak = d.get("long_time_of_peak"),
|
||||
mic_time_of_peak = d.get("mic_time_of_peak"),
|
||||
tran_peak_acceleration = d.get("tran_peak_acceleration"),
|
||||
vert_peak_acceleration = d.get("vert_peak_acceleration"),
|
||||
long_peak_acceleration = d.get("long_peak_acceleration"),
|
||||
tran_peak_displacement = d.get("tran_peak_displacement"),
|
||||
vert_peak_displacement = d.get("vert_peak_displacement"),
|
||||
long_peak_displacement = d.get("long_peak_displacement"),
|
||||
project = d.get("project"),
|
||||
client = d.get("client"),
|
||||
operator = d.get("operator"),
|
||||
notes = d.get("notes"),
|
||||
setup = d.get("setup"),
|
||||
tran_test_passed = d.get("tran_test_passed"),
|
||||
vert_test_passed = d.get("vert_test_passed"),
|
||||
long_test_passed = d.get("long_test_passed"),
|
||||
mic_test_passed = d.get("mic_test_passed"),
|
||||
firmware_version = d.get("version"),
|
||||
calibration_text = d.get("calibration_text"),
|
||||
battery_volts = d.get("battery_volts"),
|
||||
raw = d,
|
||||
)
|
||||
|
||||
|
||||
# ── IdfPeaks / IdfProjectInfo / IdfSensorCheck (narrow grouping types) ───────
|
||||
|
||||
|
||||
@dataclass
|
||||
class IdfPeaks:
|
||||
"""Geophone + mic peak values for one Thor event. Native Thor units."""
|
||||
transverse_ips: Optional[float] = None # in/s
|
||||
vertical_ips: Optional[float] = None # in/s
|
||||
longitudinal_ips: Optional[float] = None # in/s
|
||||
peak_vector_sum_ips: Optional[float] = None # in/s
|
||||
mic_pspl_dbl: Optional[float] = None # dB(L)
|
||||
|
||||
|
||||
@dataclass
|
||||
class IdfProjectInfo:
|
||||
"""Operator-supplied strings from Thor's TitleString1..4."""
|
||||
project: Optional[str] = None
|
||||
client: Optional[str] = None
|
||||
operator: Optional[str] = None
|
||||
notes: Optional[str] = None
|
||||
setup: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class IdfSensorCheck:
|
||||
"""Per-channel pass/fail from Thor's self-test."""
|
||||
tran: Optional[bool] = None
|
||||
vert: Optional[bool] = None
|
||||
long: Optional[bool] = None
|
||||
mic: Optional[bool] = None
|
||||
|
||||
|
||||
# ── IdfEvent ─────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class IdfEvent:
|
||||
"""A single Thor / Micromate Series IV event.
|
||||
|
||||
Built from a parsed .IDFW.txt or .IDFH.txt sidecar via
|
||||
``IdfEvent.from_report()``. The filename is the authoritative
|
||||
source for serial + timestamp + kind; the .txt provides
|
||||
device-authoritative peak values, frequencies, project strings,
|
||||
sensor self-check, firmware, calibration.
|
||||
"""
|
||||
|
||||
# Identity
|
||||
serial: str
|
||||
timestamp: datetime.datetime
|
||||
kind: str # "Waveform" | "Histogram"
|
||||
filename: str # device-native binary filename, e.g. "UM11719_20231219163444.IDFW"
|
||||
|
||||
# Sampling / timing
|
||||
sample_rate: Optional[int] = None
|
||||
record_time_sec: Optional[float] = None
|
||||
pre_trigger_sec: Optional[float] = None
|
||||
|
||||
# Peaks
|
||||
peaks: IdfPeaks = field(default_factory=IdfPeaks)
|
||||
|
||||
# Per-channel frequencies (Hz)
|
||||
tran_zc_freq: Optional[float] = None
|
||||
vert_zc_freq: Optional[float] = None
|
||||
long_zc_freq: Optional[float] = None
|
||||
mic_zc_freq: Optional[float] = None
|
||||
|
||||
# Project strings
|
||||
project_info: IdfProjectInfo = field(default_factory=IdfProjectInfo)
|
||||
|
||||
# Sensor self-check
|
||||
sensor_check: IdfSensorCheck = field(default_factory=IdfSensorCheck)
|
||||
|
||||
# Device-fixed
|
||||
firmware_version: Optional[str] = None
|
||||
calibration_text: Optional[str] = None
|
||||
battery_volts: Optional[float] = None
|
||||
|
||||
# The full parsed report — preserves anything not surfaced as a typed field
|
||||
report: IdfReport = field(default_factory=IdfReport)
|
||||
|
||||
@classmethod
|
||||
def from_report(
|
||||
cls,
|
||||
report: Any,
|
||||
filename: str,
|
||||
) -> "IdfEvent":
|
||||
"""Build an IdfEvent from a parsed report (dict or IdfReport) and
|
||||
the device-native binary filename.
|
||||
|
||||
The filename is authoritative for serial + timestamp + kind:
|
||||
Thor's filenames are literal ``<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>``
|
||||
and the device's own clock is the canonical event timestamp.
|
||||
If the report carries an ``event_datetime`` that differs from
|
||||
what's in the filename, the report wins (it has finer-grained
|
||||
device-reported time-of-trigger semantics).
|
||||
"""
|
||||
from .idf_ascii_report import parse_event_filename
|
||||
|
||||
# Normalise input to IdfReport
|
||||
if isinstance(report, IdfReport):
|
||||
rep = report
|
||||
elif isinstance(report, dict):
|
||||
rep = IdfReport.from_dict(report)
|
||||
else:
|
||||
raise TypeError(
|
||||
f"report must be IdfReport or dict; got {type(report).__name__}"
|
||||
)
|
||||
|
||||
# Filename → (serial, timestamp, kind). Required — fall back to
|
||||
# report-supplied values only if filename parsing fails.
|
||||
parsed = parse_event_filename(filename)
|
||||
if parsed is not None:
|
||||
fn_serial, fn_ts, fn_kind = parsed
|
||||
kind = "Histogram" if fn_kind == "IDFH" else "Waveform"
|
||||
else:
|
||||
fn_serial = rep.serial_number or "UNKNOWN"
|
||||
fn_ts = rep.event_datetime or datetime.datetime(1970, 1, 1)
|
||||
kind = "Waveform" if (rep.event_type or "").lower().startswith("full waveform") else "Histogram"
|
||||
|
||||
# Prefer report's event_datetime (device-authoritative) over the filename.
|
||||
ts = rep.event_datetime or fn_ts
|
||||
serial = rep.serial_number or fn_serial
|
||||
|
||||
return cls(
|
||||
serial=serial,
|
||||
timestamp=ts,
|
||||
kind=kind,
|
||||
filename=filename,
|
||||
sample_rate=rep.sample_rate,
|
||||
record_time_sec=rep.record_time_sec,
|
||||
pre_trigger_sec=rep.pre_trigger_sec,
|
||||
peaks=IdfPeaks(
|
||||
transverse_ips = rep.tran_ppv,
|
||||
vertical_ips = rep.vert_ppv,
|
||||
longitudinal_ips = rep.long_ppv,
|
||||
peak_vector_sum_ips = rep.peak_vector_sum,
|
||||
mic_pspl_dbl = rep.mic_pspl_dbl,
|
||||
),
|
||||
tran_zc_freq=rep.tran_zc_freq,
|
||||
vert_zc_freq=rep.vert_zc_freq,
|
||||
long_zc_freq=rep.long_zc_freq,
|
||||
mic_zc_freq=rep.mic_zc_freq,
|
||||
project_info=IdfProjectInfo(
|
||||
project=rep.project,
|
||||
client=rep.client,
|
||||
operator=rep.operator,
|
||||
notes=rep.notes,
|
||||
setup=rep.setup,
|
||||
),
|
||||
sensor_check=IdfSensorCheck(
|
||||
tran=rep.tran_test_passed,
|
||||
vert=rep.vert_test_passed,
|
||||
long=rep.long_test_passed,
|
||||
mic=rep.mic_test_passed,
|
||||
),
|
||||
firmware_version=rep.firmware_version,
|
||||
calibration_text=rep.calibration_text,
|
||||
battery_volts=rep.battery_volts,
|
||||
report=rep,
|
||||
)
|
||||
|
||||
# ── Bridge to minimateplus shape (for the existing DB / sidecar paths) ──
|
||||
|
||||
def to_minimateplus_event(self, waveform_key: bytes) -> Any:
|
||||
"""Project this Thor event into the shape ``minimateplus.Event``
|
||||
carries, so it can flow through the existing
|
||||
``SeismoDb.insert_events()`` and ``event_to_sidecar_dict()``
|
||||
machinery without those code paths needing to know about Thor.
|
||||
|
||||
Caveats of the bridge:
|
||||
- ``mic_ppv`` on the produced Event carries Thor's dB(L) value
|
||||
verbatim — the UI distinguishes via the ``device_family``
|
||||
column (Phase 1). Don't run the BW psi→dBL converter on
|
||||
Series IV rows.
|
||||
- Many Thor-specific fields (Peak Acceleration / Displacement,
|
||||
sensor self-check, calibration) don't have a slot in
|
||||
``Event``. The full IdfReport is preserved on the
|
||||
``.sfm.json`` sidecar under ``extensions.idf_report`` via
|
||||
``save_imported_idf`` — that's the source of truth for them.
|
||||
"""
|
||||
from minimateplus.models import (
|
||||
Event, PeakValues, ProjectInfo, Timestamp,
|
||||
)
|
||||
|
||||
ts_obj = Timestamp(
|
||||
raw=bytes(9),
|
||||
flag=0,
|
||||
year=self.timestamp.year,
|
||||
unknown_byte=0,
|
||||
month=self.timestamp.month,
|
||||
day=self.timestamp.day,
|
||||
hour=self.timestamp.hour,
|
||||
minute=self.timestamp.minute,
|
||||
second=self.timestamp.second,
|
||||
)
|
||||
pv = PeakValues(
|
||||
tran=self.peaks.transverse_ips,
|
||||
vert=self.peaks.vertical_ips,
|
||||
long=self.peaks.longitudinal_ips,
|
||||
micl=self.peaks.mic_pspl_dbl, # dB(L) — see caveat above
|
||||
peak_vector_sum=self.peaks.peak_vector_sum_ips,
|
||||
)
|
||||
pi = ProjectInfo(
|
||||
setup_name=self.project_info.setup,
|
||||
project=self.project_info.project,
|
||||
client=self.project_info.client,
|
||||
operator=self.project_info.operator,
|
||||
sensor_location=None, # Thor folds location into project string
|
||||
notes=self.project_info.notes,
|
||||
)
|
||||
ev = Event(
|
||||
index=0,
|
||||
timestamp=ts_obj,
|
||||
sample_rate=self.sample_rate,
|
||||
peak_values=pv,
|
||||
project_info=pi,
|
||||
record_type=self.kind,
|
||||
rectime_seconds=self.record_time_sec,
|
||||
)
|
||||
ev._waveform_key = waveform_key
|
||||
return ev
|
||||
+3
-3
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "seismo-relay"
|
||||
version = "0.18.0"
|
||||
version = "0.19.0"
|
||||
description = "Python client and REST server for MiniMate Plus seismographs"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = [
|
||||
@@ -18,6 +18,6 @@ dependencies = [
|
||||
]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
# Auto-discovers minimateplus/, sfm/, bridges/ as packages
|
||||
# Auto-discovers minimateplus/, micromate/, sfm/, bridges/ as packages
|
||||
where = ["."]
|
||||
include = ["minimateplus*", "sfm*", "bridges*"]
|
||||
include = ["minimateplus*", "micromate*", "sfm*", "bridges*"]
|
||||
|
||||
@@ -326,6 +326,7 @@ def main(argv=None) -> int:
|
||||
}}
|
||||
if ev._waveform_key else None
|
||||
),
|
||||
device_family="series3",
|
||||
)
|
||||
except Exception as exc:
|
||||
log.warning("DB upsert failed for %s: %s", path.name, exc)
|
||||
|
||||
+56
-3
@@ -85,6 +85,7 @@ CREATE TABLE IF NOT EXISTS events (
|
||||
blastware_filesize INTEGER, -- bytes; NULL if no event file saved
|
||||
a5_pickle_filename TEXT, -- "<filename>.a5.pkl" sidecar
|
||||
sidecar_filename TEXT, -- "<filename>.sfm.json" review/metadata sidecar
|
||||
device_family TEXT, -- "series3" (MiniMate Plus / BW) | "series4" (Micromate / Thor) — drives per-family UI rendering (units, labels)
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
|
||||
UNIQUE(serial, timestamp)
|
||||
);
|
||||
@@ -198,11 +199,53 @@ class SeismoDb:
|
||||
("blastware_filesize", "INTEGER"),
|
||||
("a5_pickle_filename", "TEXT"),
|
||||
("sidecar_filename", "TEXT"),
|
||||
("device_family", "TEXT"),
|
||||
):
|
||||
if col not in existing_cols:
|
||||
log.info("_migrate: events ADD COLUMN %s %s", col, ddl)
|
||||
conn.execute(f"ALTER TABLE events ADD COLUMN {col} {ddl}")
|
||||
|
||||
# Migration 1c: backfill device_family for existing rows by sniffing
|
||||
# the device-native binary filename's extension. Thor (Micromate
|
||||
# Series IV) writes `.IDFH` / `.IDFW`; MiniMate Plus (Series III)
|
||||
# writes `.AB0*` / `.N00` / `.<base36>` Blastware extensions. We do
|
||||
# this here rather than from sidecars so the migration is fully
|
||||
# self-contained (doesn't need the waveform-store root) and runs at
|
||||
# DB-init time. Only fills NULL device_family so re-runs are no-ops.
|
||||
rebackfill = conn.execute(
|
||||
"SELECT COUNT(*) FROM events WHERE device_family IS NULL"
|
||||
).fetchone()
|
||||
if rebackfill and rebackfill[0] > 0:
|
||||
log.info("_migrate: backfilling device_family for %d events", rebackfill[0])
|
||||
# Series IV (Thor IDF) — extension is exactly .IDFH or .IDFW
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE events
|
||||
SET device_family = 'series4'
|
||||
WHERE device_family IS NULL
|
||||
AND (
|
||||
UPPER(blastware_filename) LIKE '%.IDFH'
|
||||
OR UPPER(blastware_filename) LIKE '%.IDFW'
|
||||
)
|
||||
"""
|
||||
)
|
||||
# Everything else with a filename → Series III (Blastware family)
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE events
|
||||
SET device_family = 'series3'
|
||||
WHERE device_family IS NULL
|
||||
AND blastware_filename IS NOT NULL
|
||||
"""
|
||||
)
|
||||
# Rows with no filename (e.g. older monitor_log-derived events)
|
||||
# stay NULL — UI handles NULL as "unknown family".
|
||||
remaining = conn.execute(
|
||||
"SELECT COUNT(*) FROM events WHERE device_family IS NULL"
|
||||
).fetchone()[0]
|
||||
log.info("_migrate: device_family backfill complete (remaining NULL=%d)",
|
||||
remaining)
|
||||
|
||||
# Migration 2: change monitor_log UNIQUE from (serial, waveform_key) to
|
||||
# (serial, start_time) — same reasoning as events.
|
||||
row = conn.execute(
|
||||
@@ -302,6 +345,7 @@ class SeismoDb:
|
||||
serial: str,
|
||||
session_id: Optional[str] = None,
|
||||
waveform_records: Optional[dict[str, dict]] = None,
|
||||
device_family: Optional[str] = None,
|
||||
) -> tuple[int, int]:
|
||||
"""
|
||||
Insert triggered events. Silently skips duplicates (serial+timestamp).
|
||||
@@ -316,6 +360,11 @@ class SeismoDb:
|
||||
(dedup hit), the matching waveform record is upserted onto the
|
||||
existing row so a re-download via the live endpoint refreshes the
|
||||
file metadata.
|
||||
|
||||
``device_family`` (optional): "series3" (MiniMate Plus / Blastware) or
|
||||
"series4" (Micromate / Thor). Drives per-family UI rendering — most
|
||||
importantly the mic-unit convention (psi vs dB(L)). Set on every
|
||||
insert and overwritten on every UPSERT so the latest writer wins.
|
||||
"""
|
||||
inserted = skipped = 0
|
||||
wave_recs = waveform_records or {}
|
||||
@@ -349,8 +398,9 @@ class SeismoDb:
|
||||
project, client, operator, sensor_location,
|
||||
sample_rate, record_type,
|
||||
blastware_filename, blastware_filesize,
|
||||
a5_pickle_filename, sidecar_filename)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
a5_pickle_filename, sidecar_filename,
|
||||
device_family)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
self._new_id(), serial, key, session_id, ts,
|
||||
@@ -369,6 +419,7 @@ class SeismoDb:
|
||||
rec.get("filesize"),
|
||||
rec.get("a5_pickle_filename"),
|
||||
rec.get("sidecar_filename"),
|
||||
device_family,
|
||||
),
|
||||
)
|
||||
inserted += 1
|
||||
@@ -409,7 +460,8 @@ class SeismoDb:
|
||||
blastware_filename = ?,
|
||||
blastware_filesize = ?,
|
||||
a5_pickle_filename = ?,
|
||||
sidecar_filename = ?
|
||||
sidecar_filename = ?,
|
||||
device_family = COALESCE(?, device_family)
|
||||
WHERE serial = ? AND timestamp = ?
|
||||
""",
|
||||
(
|
||||
@@ -428,6 +480,7 @@ class SeismoDb:
|
||||
rec.get("filesize") if rec else None,
|
||||
rec.get("a5_pickle_filename") if rec else None,
|
||||
rec.get("sidecar_filename") if rec else None,
|
||||
device_family,
|
||||
serial,
|
||||
ts,
|
||||
),
|
||||
|
||||
@@ -166,6 +166,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||
{ev._waveform_key.hex(): rec}
|
||||
if ev._waveform_key else None
|
||||
),
|
||||
device_family="series3",
|
||||
)
|
||||
tag = "OK " if ins else ("SKIP" if sk else "OK ")
|
||||
print(f" [{tag}] {path.name} → {rec['filename']} "
|
||||
|
||||
+4
-1
@@ -918,6 +918,7 @@ def device_event_blastware_file(
|
||||
[ev],
|
||||
serial=serial,
|
||||
waveform_records={ev._waveform_key.hex(): rec},
|
||||
device_family="series3",
|
||||
)
|
||||
log.info(
|
||||
"blastware_file: persisted to store (%s, %d bytes)",
|
||||
@@ -2434,6 +2435,7 @@ async def db_import_blastware_file(
|
||||
ev._waveform_key.hex(): rec
|
||||
if ev._waveform_key else None
|
||||
} if ev._waveform_key else None,
|
||||
device_family="series3",
|
||||
)
|
||||
results.append({
|
||||
"filename": filename,
|
||||
@@ -2489,7 +2491,7 @@ async def db_import_idf_file(
|
||||
|
||||
1. Pair the binary with its `<binary>.txt` ASCII report when one
|
||||
is present in the same upload.
|
||||
2. Parse the report via `sfm.idf_ascii_report.parse_idf_report`
|
||||
2. Parse the report via `micromate.parse_idf_report`
|
||||
and copy the binary into the persistent store via
|
||||
`WaveformStore.save_imported_idf`, writing a `.sfm.json`
|
||||
sidecar with `source.kind = "idf-import"`.
|
||||
@@ -2558,6 +2560,7 @@ async def db_import_idf_file(
|
||||
waveform_records={
|
||||
ev._waveform_key.hex(): rec
|
||||
} if ev._waveform_key else None,
|
||||
device_family="series4",
|
||||
)
|
||||
results.append({
|
||||
"filename": filename,
|
||||
|
||||
+36
-12
@@ -2285,13 +2285,16 @@ let sessLoaded = false;
|
||||
const _unitSerials = new Set();
|
||||
|
||||
function _ppvClass(v) {
|
||||
if (v == null) return '';
|
||||
if (v >= 2.0) return 'ppv-high';
|
||||
if (v >= 0.5) return 'ppv-warn';
|
||||
const n = (v == null) ? null : Number(v);
|
||||
if (n == null || !isFinite(n)) return '';
|
||||
if (n >= 2.0) return 'ppv-high';
|
||||
if (n >= 0.5) return 'ppv-warn';
|
||||
return 'ppv-ok';
|
||||
}
|
||||
function _ppvFmt(v) {
|
||||
return v != null ? v.toFixed(5) : '—';
|
||||
if (v == null) return '—';
|
||||
const n = typeof v === 'number' ? v : Number(v);
|
||||
return isFinite(n) ? n.toFixed(5) : String(v);
|
||||
}
|
||||
function _fmtTs(ts) {
|
||||
if (!ts) return '—';
|
||||
@@ -2386,7 +2389,14 @@ async function loadHistory() {
|
||||
<td class="${_ppvClass(ev.vert_ppv)}">${_ppvFmt(ev.vert_ppv)}</td>
|
||||
<td class="${_ppvClass(ev.long_ppv)}">${_ppvFmt(ev.long_ppv)}</td>
|
||||
<td class="${_ppvClass(pvs)}">${_ppvFmt(pvs)}</td>
|
||||
<td class="td-dim">${ev.mic_ppv != null && ev.mic_ppv > 0 ? (20 * Math.log10(ev.mic_ppv / DBL_REF)).toFixed(1) + ' dBL' : '—'}</td>
|
||||
<td class="td-dim">${(() => {
|
||||
const m = ev.mic_ppv == null ? null : Number(ev.mic_ppv);
|
||||
if (m == null || !isFinite(m) || m <= 0) return '—';
|
||||
// Series III (MiniMate Plus / BW) stores mic_ppv as psi → convert.
|
||||
// Series IV (Micromate / Thor) already stores dB(L) → display direct.
|
||||
if (ev.device_family === 'series4') return m.toFixed(1) + ' dBL';
|
||||
return (20 * Math.log10(m / DBL_REF)).toFixed(1) + ' dBL';
|
||||
})()}</td>
|
||||
<td class="td-text">${ev.project ?? '—'}</td>
|
||||
<td class="td-text">${ev.client ?? '—'}</td>
|
||||
<td class="td-dim">${ev.record_type ?? '—'}</td>
|
||||
@@ -2447,11 +2457,25 @@ function _renderSidecar(data) {
|
||||
|
||||
document.getElementById('sc-title').textContent = `Event — ${bw.filename || ev.waveform_key || 'unknown'}`;
|
||||
|
||||
const fmtPpv = v => (v == null ? '—' : Number(v).toFixed(5) + ' in/s');
|
||||
const fmtPpv = v => {
|
||||
if (v == null) return '—';
|
||||
const n = Number(v);
|
||||
return isFinite(n) ? n.toFixed(5) + ' in/s' : String(v);
|
||||
};
|
||||
// Map sidecar source.kind → device family (Series IV ingest path is
|
||||
// "idf-import"; everything else is Series III today). The events-list
|
||||
// table uses ev.device_family from the DB row, but sidecars don't carry
|
||||
// that column — source.kind is the equivalent signal here.
|
||||
const family = ((src.kind || '') === 'idf-import') ? 'series4' : 'series3';
|
||||
const fmtMic = v => {
|
||||
if (v == null || v <= 0) return '—';
|
||||
const dbl = 20 * Math.log10(v / DBL_REF);
|
||||
return `${dbl.toFixed(1)} dBL (${v.toExponential(2)} psi)`;
|
||||
if (v == null) return '—';
|
||||
const n = Number(v);
|
||||
if (!isFinite(n) || n <= 0) return '—';
|
||||
// Series IV (Micromate / Thor) stores mic as dB(L); Series III (BW)
|
||||
// stores it as psi and we render both for cross-reference.
|
||||
if (family === 'series4') return `${n.toFixed(1)} dBL`;
|
||||
const dbl = 20 * Math.log10(n / DBL_REF);
|
||||
return `${dbl.toFixed(1)} dBL (${n.toExponential(2)} psi)`;
|
||||
};
|
||||
|
||||
document.getElementById('sc-f-serial').textContent = ev.serial || '—';
|
||||
@@ -2746,9 +2770,9 @@ document.getElementById('api-base').value = window.location.origin;
|
||||
<div class="sc-section">
|
||||
<h4>Source / files</h4>
|
||||
<dl class="sc-grid">
|
||||
<dt>BW filename</dt> <dd id="sc-f-bw">—</dd>
|
||||
<dt>BW filesize</dt> <dd id="sc-f-bwsize">—</dd>
|
||||
<dt>BW sha256</dt> <dd id="sc-f-sha">—</dd>
|
||||
<dt id="sc-l-bw">Event file</dt> <dd id="sc-f-bw">—</dd>
|
||||
<dt id="sc-l-bwsize">File size</dt> <dd id="sc-f-bwsize">—</dd>
|
||||
<dt id="sc-l-sha">File sha256</dt> <dd id="sc-f-sha">—</dd>
|
||||
<dt>Source kind</dt> <dd id="sc-f-src">—</dd>
|
||||
<dt>Captured at</dt> <dd id="sc-f-cap">—</dd>
|
||||
</dl>
|
||||
|
||||
+32
-89
@@ -426,99 +426,48 @@ class WaveformStore:
|
||||
`.IDFH`) produced by Thor's TXT exporter.
|
||||
|
||||
Thor binaries are stored as opaque bytes — seismo-relay doesn't
|
||||
decode the proprietary IDF binary format. Device-authoritative
|
||||
metadata comes from the paired `.IDFW.txt` / `.IDFH.txt` sidecar
|
||||
when supplied; we parse that text and surface its fields onto
|
||||
the returned Event so the SFM database row has real PPV/project
|
||||
values instead of NULLs.
|
||||
yet decode the proprietary IDF binary format (codec slot lives
|
||||
at ``micromate/idf_file.py``). Device-authoritative metadata
|
||||
comes from the paired ``.IDFW.txt`` / ``.IDFH.txt`` sidecar
|
||||
when supplied.
|
||||
|
||||
Workflow:
|
||||
1. Parse the paired TXT report (when supplied) via
|
||||
`sfm.idf_ascii_report.parse_idf_report`.
|
||||
2. Build a minimal `Event` populated from the report fields
|
||||
(timestamp, peaks, project info, sample_rate, record_type).
|
||||
3. Resolve serial from filename prefix or `serial_hint`.
|
||||
4. Copy bytes verbatim into <root>/<serial>/<filename>.
|
||||
5. Write the `.sfm.json` sidecar with source.kind = "idf-import".
|
||||
``micromate.parse_idf_report`` → dict.
|
||||
2. Wrap parsed dict + filename into a typed ``micromate.IdfEvent``.
|
||||
3. Copy bytes verbatim into ``<root>/<serial>/<filename>``.
|
||||
4. Bridge IdfEvent → ``minimateplus.Event`` (for the existing
|
||||
sidecar / DB insert machinery) via
|
||||
``IdfEvent.to_minimateplus_event(waveform_key)``.
|
||||
5. Write the ``.sfm.json`` sidecar with
|
||||
``source.kind = "idf-import"`` and the full raw IDF report
|
||||
under ``extensions.idf_report``.
|
||||
|
||||
Returns (event, record_dict) so the endpoint can both insert
|
||||
Returns ``(event, record_dict)`` so the endpoint can both insert
|
||||
into SeismoDb and surface the parsed event.
|
||||
"""
|
||||
from sfm.idf_ascii_report import (
|
||||
parse_idf_report,
|
||||
parse_event_filename,
|
||||
serial_from_filename as _idf_serial_from_filename,
|
||||
)
|
||||
from minimateplus.models import (
|
||||
Event, PeakValues, ProjectInfo, Timestamp,
|
||||
)
|
||||
from micromate import IdfEvent, parse_idf_report
|
||||
|
||||
# Parse the .txt sidecar (best-effort; non-fatal on failure).
|
||||
report: dict = {}
|
||||
report_dict: dict = {}
|
||||
if idf_report_text is not None:
|
||||
try:
|
||||
report = parse_idf_report(idf_report_text)
|
||||
report_dict = parse_idf_report(idf_report_text)
|
||||
except Exception as exc:
|
||||
log.warning(
|
||||
"save_imported_idf: report parse failed: %s — continuing without it",
|
||||
exc,
|
||||
)
|
||||
|
||||
# Resolve serial: prefer the explicit hint, fall back to filename prefix.
|
||||
serial = (
|
||||
serial_hint
|
||||
or report.get("serial_number")
|
||||
or _idf_serial_from_filename(source_path.name)
|
||||
or "UNKNOWN"
|
||||
)
|
||||
# Build the typed IdfEvent. Filename is authoritative for
|
||||
# (serial, timestamp, kind); the report's event_datetime takes
|
||||
# precedence over the filename timestamp inside from_report().
|
||||
idf_event = IdfEvent.from_report(report_dict, source_path.name)
|
||||
|
||||
# Resolve event timestamp + kind from the filename (always present).
|
||||
parsed_name = parse_event_filename(source_path.name)
|
||||
kind = "Waveform"
|
||||
ts_dt: Optional[datetime.datetime] = None
|
||||
if parsed_name is not None:
|
||||
_, ts_dt, kind_token = parsed_name
|
||||
kind = "Histogram" if kind_token == "IDFH" else "Waveform"
|
||||
# Report's event_datetime is the device-authoritative value; prefer it.
|
||||
if "event_datetime" in report:
|
||||
try:
|
||||
ts_dt = datetime.datetime.fromisoformat(report["event_datetime"])
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
ts_obj: Optional[Timestamp] = None
|
||||
if ts_dt is not None:
|
||||
ts_obj = Timestamp(
|
||||
raw=bytes(9),
|
||||
flag=0,
|
||||
year=ts_dt.year,
|
||||
unknown_byte=0,
|
||||
month=ts_dt.month,
|
||||
day=ts_dt.day,
|
||||
hour=ts_dt.hour,
|
||||
minute=ts_dt.minute,
|
||||
second=ts_dt.second,
|
||||
)
|
||||
|
||||
# Build PeakValues from the report (fields are None when absent).
|
||||
pv = PeakValues(
|
||||
tran=report.get("tran_ppv"),
|
||||
vert=report.get("vert_ppv"),
|
||||
long=report.get("long_ppv"),
|
||||
micl=report.get("mic_ppv"),
|
||||
peak_vector_sum=report.get("peak_vector_sum"),
|
||||
)
|
||||
|
||||
# Build ProjectInfo. See idf_ascii_report — Thor's title strings
|
||||
# carry project / client / company / notes in TitleString1..4.
|
||||
pi = ProjectInfo(
|
||||
setup_name=report.get("setup"),
|
||||
project=report.get("project"),
|
||||
client=report.get("client"),
|
||||
operator=report.get("operator"),
|
||||
sensor_location=None, # Thor folds location into TitleString1 = project
|
||||
notes=report.get("notes"),
|
||||
)
|
||||
# Operator-supplied serial_hint wins over the binary's filename
|
||||
# prefix when both are present (e.g. callers passing a known-good
|
||||
# serial that overrides a misnamed export).
|
||||
serial = serial_hint or idf_event.serial or "UNKNOWN"
|
||||
|
||||
# Filesystem write.
|
||||
filename = source_path.name
|
||||
@@ -532,16 +481,10 @@ class WaveformStore:
|
||||
# surrogate — every distinct binary maps to a distinct row.
|
||||
waveform_key = bytes.fromhex(sha256)[:16]
|
||||
|
||||
ev = Event(
|
||||
index=0,
|
||||
timestamp=ts_obj,
|
||||
sample_rate=report.get("sample_rate"),
|
||||
peak_values=pv,
|
||||
project_info=pi,
|
||||
record_type=kind,
|
||||
rectime_seconds=report.get("record_time_sec"),
|
||||
)
|
||||
ev._waveform_key = waveform_key
|
||||
# Bridge to minimateplus.Event for the existing sidecar / DB
|
||||
# insert paths. See IdfEvent.to_minimateplus_event() for the
|
||||
# caveats of this bridge (mic units, missing fields → sidecar).
|
||||
ev = idf_event.to_minimateplus_event(waveform_key)
|
||||
|
||||
# Write the sidecar. Source kind "idf-import" was added to the
|
||||
# allow-list in event_file_io.event_to_sidecar_dict for this.
|
||||
@@ -567,14 +510,14 @@ class WaveformStore:
|
||||
# consumers can recover the rich derived fields that don't fit
|
||||
# the BW-shaped event model (Peak Acceleration / Displacement,
|
||||
# Time of Peak, sensor self-check, calibration, firmware).
|
||||
if report:
|
||||
sidecar["extensions"]["idf_report"] = report
|
||||
if report_dict:
|
||||
sidecar["extensions"]["idf_report"] = report_dict
|
||||
event_file_io.write_sidecar(sidecar_path, sidecar)
|
||||
|
||||
log.info(
|
||||
"WaveformStore.save_imported_idf serial=%s filename=%s filesize=%d "
|
||||
"report_attached=%s",
|
||||
serial, filename, filesize, bool(report),
|
||||
serial, filename, filesize, bool(report_dict),
|
||||
)
|
||||
return ev, {
|
||||
"filename": filename,
|
||||
|
||||
@@ -21,7 +21,7 @@ import pytest
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from sfm.idf_ascii_report import (
|
||||
from micromate.idf_ascii_report import (
|
||||
parse_event_filename,
|
||||
parse_idf_report,
|
||||
serial_from_filename,
|
||||
|
||||
Reference in New Issue
Block a user