seismo-relay v0.19.0 — device-family separation + micromate/ package
Tighten the Series III / Series IV boundary so UI and storage dispatch
on a clean signal instead of sniffing filenames or applying magnitude
heuristics.
Phase 1 — events.device_family column ("series3" | "series4"):
self-applying migration with filename-based backfill of existing rows
(1,132 backfilled on prod 2026-05-20); plumbed through every import
path (BW endpoint, IDF endpoint, ACH server, BW CLI, sidecar
backfill); UPSERT preserves via COALESCE; UI dispatches on it.
Phase 2 — extract micromate/ package alongside minimateplus/:
native IdfEvent / IdfReport / IdfPeaks / IdfProjectInfo /
IdfSensorCheck (mic in dB(L), not pseudo-psi); moved
idf_ascii_report.py from sfm/ to micromate/; refactored
save_imported_idf to use IdfEvent and bridge to minimateplus.Event at
the SQL-insert boundary; idf_file.py stub for the future binary codec.
Phase 3 prep — docs/idf_protocol_reference.md captures the two
observed Thor binary header signatures (1,012 newer-firmware files vs
2 old files whose layout is byte-for-byte BW-STRT-compatible), file-size
hints suggesting int8 sample encoding, open questions in dependency
order, and a concrete first-session plan for cracking the codec.
Also rolled in the v0.18.1 hotfixes that motivated this work:
- idf_ascii_report parser now handles "<0.005 in/s" (below-threshold)
and "N/A" markers without leaving raw strings in numeric DB columns.
- sfm_webapp.html: defensive _ppvFmt / mic formatter so future
data-shape drift can't kill the whole events table render.
All 1,014 example-data sidecars round-trip through the new package.
See CHANGELOG.md for full notes.
This commit is contained in:
@@ -4,6 +4,33 @@ All notable changes to seismo-relay are documented here.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## v0.19.0 — 2026-05-20
|
||||||
|
|
||||||
|
The "device-family separation" release. Tightens the boundary between Series III (MiniMate Plus / Blastware) and Series IV (Micromate / Thor) so the UI and storage layer dispatch deterministically by family instead of sniffing filename extensions or magnitude heuristics.
|
||||||
|
|
||||||
|
### Added — Phase 1: `device_family` column on `events`
|
||||||
|
|
||||||
|
- **`events.device_family TEXT`** — new column carrying `"series3"` or `"series4"`. Populated by every import path (`/db/import/blastware_file`, `/db/import/idf_file`, ACH server, BW CLI, sidecar backfill script). Returned through `/db/events` since `query_events` uses `SELECT *`.
|
||||||
|
- **Self-applying migration** — on startup, `ALTER TABLE ... ADD COLUMN` lands the new column; a follow-on `UPDATE` backfills existing rows from the binary filename extension (`.IDFH`/`.IDFW` → `series4`, everything else → `series3`). No manual SQL needed.
|
||||||
|
- **UPSERT preserves family** — re-imports without an explicit family don't blank existing rows (`COALESCE(?, device_family)`).
|
||||||
|
- **UI dispatches on the column** — `sfm_webapp.html` events-table mic formatter now branches on `ev.device_family === 'series4'` (Thor stores native dB(L); BW stores psi). Modal uses `source.kind === 'idf-import'` from the sidecar (sidecars don't carry the DB column). Source-files section labels changed from "BW filename / BW filesize / BW sha256" to format-neutral "Event file / File size / File sha256".
|
||||||
|
|
||||||
|
### Added — Phase 2: `micromate/` package alongside `minimateplus/`
|
||||||
|
|
||||||
|
- **`micromate/`** — new sibling package for the Thor / Micromate Series IV device. Currently scoped to offline-file ingest; live-device support (TCP transport, framing, protocol, client) will land here when reverse-engineering happens.
|
||||||
|
- `micromate/idf_ascii_report.py` — moved from `sfm/idf_ascii_report.py`. No behaviour change.
|
||||||
|
- `micromate/models.py` — typed `IdfReport`, `IdfEvent`, `IdfPeaks`, `IdfProjectInfo`, `IdfSensorCheck`. Stores mic in native `mic_pspl_dbl` (dB(L)) instead of the pseudo-psi shoehorn that the BW-shaped model uses. `IdfEvent.from_report()` constructs from a parsed dict + filename; `IdfEvent.to_minimateplus_event(waveform_key)` bridges to the existing sidecar / DB-insert machinery.
|
||||||
|
- `micromate/idf_file.py` — placeholder for the binary codec (`.IDFH` / `.IDFW`). Stubbed `read_idf_file()` raises `NotImplementedError`; documents the planned reverse-engineering path.
|
||||||
|
- **`WaveformStore.save_imported_idf`** refactored to use the native `IdfEvent` and bridge at the SQL-insert boundary. Cleaner separation of "parse a Thor event" (in `micromate/`) from "store it on disk + write a sidecar" (in `sfm/waveform_store.py`).
|
||||||
|
- **Tests** — `tests/test_idf_ascii_report.py` imports updated to `micromate.idf_ascii_report`. All 1,014 example-data sidecars round-trip through `IdfEvent.from_report()` without errors.
|
||||||
|
|
||||||
|
### Companion releases
|
||||||
|
|
||||||
|
- **thor-watcher** unaffected — it talks to the relay over HTTP only. No version bump needed.
|
||||||
|
- **terra-view** unaffected today; can use `device_family` in its event-detail rendering when convenient.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## v0.18.0 — 2026-05-19
|
## v0.18.0 — 2026-05-19
|
||||||
|
|
||||||
The "Thor / Series IV ingest adapter" release. Seismo-relay can now accept event files from Instantel Micromate Series IV (Thor) units alongside the existing MiniMate Plus (Series III) Blastware pipeline.
|
The "Thor / Series IV ingest adapter" release. Seismo-relay can now accept event files from Instantel Micromate Series IV (Thor) units alongside the existing MiniMate Plus (Series III) Blastware pipeline.
|
||||||
|
|||||||
@@ -1,7 +1,11 @@
|
|||||||
# seismo-relay `v0.17.0`
|
# seismo-relay `v0.19.0`
|
||||||
|
|
||||||
A ground-up replacement for **Blastware** — Instantel's aging Windows-only
|
A ground-up replacement for **Blastware** — Instantel's aging Windows-only
|
||||||
software for managing MiniMate Plus seismographs.
|
software for managing seismographs. Supports both the **MiniMate Plus
|
||||||
|
(Series III)** and the **Micromate (Series IV / "Thor")** families:
|
||||||
|
Series III via the live RS-232 / TCP wire protocol *and* Blastware ACH file
|
||||||
|
ingest; Series IV currently via Thor TXT-paired IDF file ingest, with the
|
||||||
|
binary codec on the roadmap.
|
||||||
|
|
||||||
Built in Python. Runs on Windows, Linux, or macOS. Connects to instruments
|
Built in Python. Runs on Windows, Linux, or macOS. Connects to instruments
|
||||||
over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
||||||
@@ -19,6 +23,18 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
|||||||
> every Blastware ACH event lands in SeismoDb with device-authoritative
|
> every Blastware ACH event lands in SeismoDb with device-authoritative
|
||||||
> peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data,
|
> peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data,
|
||||||
> without depending on the still-undecoded waveform body codec.
|
> without depending on the still-undecoded waveform body codec.
|
||||||
|
> **v0.18.0 (2026-05-19)** adds Thor / Micromate Series IV ingest at
|
||||||
|
> `/db/import/idf_file` — paired with **thor-watcher v0.3.0**, every
|
||||||
|
> `.IDFH` / `.IDFW` event file (plus its `.txt` sidecar) lands in
|
||||||
|
> SeismoDb the same way BW events do. See
|
||||||
|
> [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md) for
|
||||||
|
> the IDF format reference and reverse-engineering plan.
|
||||||
|
> **v0.19.0 (2026-05-20)** separates Series III and Series IV at the
|
||||||
|
> code level: new `micromate/` package alongside `minimateplus/`, new
|
||||||
|
> `events.device_family` DB column ("series3" / "series4") so the UI
|
||||||
|
> and storage layer dispatch deterministically instead of sniffing
|
||||||
|
> filenames. Self-applying migration backfills existing rows from the
|
||||||
|
> binary filename extension.
|
||||||
> See [CHANGELOG.md](CHANGELOG.md) for full version history.
|
> See [CHANGELOG.md](CHANGELOG.md) for full version history.
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -29,17 +45,25 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
|||||||
seismo-relay/
|
seismo-relay/
|
||||||
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Download + Console tabs)
|
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Download + Console tabs)
|
||||||
│
|
│
|
||||||
├── minimateplus/ ← MiniMate Plus client library
|
├── minimateplus/ ← Series III (MiniMate Plus) client library
|
||||||
│ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport
|
│ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport
|
||||||
│ ├── protocol.py ← DLE frame layer, SUB command dispatch
|
│ ├── protocol.py ← DLE frame layer, SUB command dispatch
|
||||||
│ ├── client.py ← High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …)
|
│ ├── client.py ← High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …)
|
||||||
│ ├── framing.py ← Frame builders, DLE codec, S3FrameParser
|
│ ├── framing.py ← Frame builders, DLE codec, S3FrameParser
|
||||||
│ ├── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig, …
|
│ ├── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig, …
|
||||||
|
│ ├── bw_ascii_report.py ← Parse BW per-event ASCII reports (.TXT sidecars)
|
||||||
|
│ ├── event_file_io.py ← Read BW binaries, write .sfm.json sidecars
|
||||||
│ └── blastware_file.py ← Write events to Blastware-compatible .AB0 files
|
│ └── blastware_file.py ← Write events to Blastware-compatible .AB0 files
|
||||||
│
|
│
|
||||||
|
├── micromate/ ← Series IV (Micromate / Thor) client library (NEW v0.19)
|
||||||
|
│ ├── models.py ← IdfEvent, IdfReport, IdfPeaks, IdfProjectInfo, IdfSensorCheck (mic in native dB(L))
|
||||||
|
│ ├── idf_ascii_report.py ← Parse Thor .IDFW.txt / .IDFH.txt event sidecars
|
||||||
|
│ └── idf_file.py ← Stub for the .IDFW / .IDFH binary codec (reverse-engineering pending)
|
||||||
|
│
|
||||||
├── sfm/ ← SFM REST API server (FastAPI, port 8200)
|
├── sfm/ ← SFM REST API server (FastAPI, port 8200)
|
||||||
│ ├── server.py ← Live device endpoints + DB query endpoints + caching
|
│ ├── server.py ← Live device endpoints + DB query + ingest endpoints + caching
|
||||||
│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions, sessions table)
|
│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions)
|
||||||
|
│ ├── waveform_store.py ← On-disk store for BW + IDF event binaries + .sfm.json sidecars
|
||||||
│ └── sfm_webapp.html ← Embedded web UI with Call Home config tab
|
│ └── sfm_webapp.html ← Embedded web UI with Call Home config tab
|
||||||
│
|
│
|
||||||
├── bridges/
|
├── bridges/
|
||||||
@@ -56,7 +80,8 @@ seismo-relay/
|
|||||||
│ └── frame_db.py ← SQLite frame database
|
│ └── frame_db.py ← SQLite frame database
|
||||||
│
|
│
|
||||||
└── docs/
|
└── docs/
|
||||||
└── instantel_protocol_reference.md ← Reverse-engineered protocol spec
|
├── instantel_protocol_reference.md ← Series III protocol spec (the Rosetta Stone)
|
||||||
|
└── idf_protocol_reference.md ← Series IV (Thor IDF) format reference + codec RE plan
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -148,11 +173,23 @@ Query the SQLite database written by `ach_server.py`. All read-only except
|
|||||||
| Method | URL | Description |
|
| Method | URL | Description |
|
||||||
|--------|-----|-------------|
|
|--------|-----|-------------|
|
||||||
| `GET` | `/db/units` | All known serials with summary stats |
|
| `GET` | `/db/units` | All known serials with summary stats |
|
||||||
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger) |
|
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger). Response rows include `device_family` ("series3" / "series4") so clients dispatch on unit type without sniffing filenames. |
|
||||||
| `GET` | `/db/monitor_log` | Monitoring intervals |
|
| `GET` | `/db/monitor_log` | Monitoring intervals |
|
||||||
| `GET` | `/db/sessions` | ACH call-home session history |
|
| `GET` | `/db/sessions` | ACH call-home session history |
|
||||||
| `PATCH` | `/db/events/{id}/false_trigger?value=true` | Flag / unflag false triggers |
|
| `PATCH` | `/db/events/{id}/false_trigger?value=true` | Flag / unflag false triggers |
|
||||||
|
|
||||||
|
### File ingest endpoints
|
||||||
|
|
||||||
|
Used by watcher daemons to push field-collected event files into the SFM DB
|
||||||
|
+ waveform store. Both accept multipart uploads of binary event files
|
||||||
|
optionally paired with their ASCII sidecar reports; both dedup by
|
||||||
|
`(serial, timestamp)` and UPSERT device-authoritative fields on re-import.
|
||||||
|
|
||||||
|
| Method | URL | Description |
|
||||||
|
|--------|-----|-------------|
|
||||||
|
| `POST` | `/db/import/blastware_file` | Series III: `.AB0*` / `.N00` binaries + paired `_ASCII.TXT`. Source: `series3-watcher`. |
|
||||||
|
| `POST` | `/db/import/idf_file` | Series IV: `.IDFH` / `.IDFW` binaries + paired `.IDFW.txt` / `.IDFH.txt`. Source: `thor-watcher`. |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## minimateplus library
|
## minimateplus library
|
||||||
@@ -214,22 +251,77 @@ not per individual event).
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## micromate library
|
||||||
|
|
||||||
|
Series IV / Thor support, sibling to `minimateplus`. Currently scoped to
|
||||||
|
offline-file ingest from Thor's TXT exporter; live-device protocol is
|
||||||
|
deferred until the binary codec is cracked.
|
||||||
|
|
||||||
|
```python
|
||||||
|
from micromate import IdfEvent, parse_idf_report
|
||||||
|
|
||||||
|
# Parse a .IDFW.txt / .IDFH.txt sidecar (1014 example files round-trip cleanly)
|
||||||
|
text = open("UM11719_20231219162723.IDFW.txt").read()
|
||||||
|
report_dict = parse_idf_report(text) # permissive dict
|
||||||
|
|
||||||
|
# Wrap into a typed event using the device-native binary filename
|
||||||
|
event = IdfEvent.from_report(report_dict, "UM11719_20231219162723.IDFW")
|
||||||
|
|
||||||
|
event.serial # "UM11719"
|
||||||
|
event.kind # "Waveform" or "Histogram"
|
||||||
|
event.peaks.transverse_ips # 0.0251 (in/s, native unit)
|
||||||
|
event.peaks.mic_pspl_dbl # 99.4 (dB(L), Thor's native mic unit — NOT psi)
|
||||||
|
event.project_info.project # "UPMC Presby-Loc 3-Level1-1R Elevator Rm"
|
||||||
|
event.sensor_check.tran # True (passed self-check)
|
||||||
|
event.firmware_version # "Micromate ISEE 11.0AK"
|
||||||
|
event.calibration_text # "November 22, 2023 by Instantel"
|
||||||
|
|
||||||
|
# Bridge to the existing minimateplus.Event shape for the DB / sidecar paths
|
||||||
|
# (waveform_key is a 16-byte sha256 prefix when ingesting from a binary file)
|
||||||
|
bridged_event = event.to_minimateplus_event(waveform_key=b"\x00" * 16)
|
||||||
|
```
|
||||||
|
|
||||||
|
The binary codec (`.IDFW` / `.IDFH` event files themselves) is on the
|
||||||
|
roadmap — see [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md)
|
||||||
|
for everything known so far, the two observed file signatures, and the
|
||||||
|
reverse-engineering plan. The `micromate/idf_file.py` stub is where
|
||||||
|
`read_idf_file()` will land.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Database
|
## Database
|
||||||
|
|
||||||
`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode) using the
|
`ach_server.py` and the file-ingest endpoints write to
|
||||||
`SeismoDb` persistence layer. Four tables, all unit-keyed by serial number:
|
`bridges/captures/seismo_relay.db` (SQLite, WAL mode) via the `SeismoDb`
|
||||||
|
persistence layer. Three tables, all unit-keyed by serial number:
|
||||||
|
|
||||||
| Table | Key | Contents |
|
| Table | Key | Contents |
|
||||||
|-------|-----|----------|
|
|-------|-----|----------|
|
||||||
| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds |
|
| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds |
|
||||||
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag |
|
| `events` | UUID, UNIQUE(serial, timestamp) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag, **`device_family`** ("series3" / "series4"), `blastware_filename` (binary at-rest in `waveforms/`), sidecar references |
|
||||||
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
|
| `monitor_log` | UUID, UNIQUE(serial, start_time) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
|
||||||
| `events.false_trigger` | Boolean flag | PATCH endpoint to mark/unmark false triggers for review |
|
|
||||||
|
|
||||||
Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs never
|
**Deduplication is by `(serial, timestamp)`** — the device clock is the
|
||||||
produce duplicate rows. Post-erase key reuse is handled automatically via the
|
stable natural key. Repeat call-homes or re-runs UPSERT the row in place,
|
||||||
high-water mark in `ach_state.json`. Key-based state tracking allows correct
|
refreshing every device-authoritative field (peaks, project strings,
|
||||||
handling of device erasures (external or post-download).
|
sample_rate, file references) so the latest writer wins. `false_trigger`
|
||||||
|
and `device_family` are preserved across UPSERTs. Earlier versions used
|
||||||
|
`(serial, waveform_key)` for dedup, but the device's event-key counter
|
||||||
|
resets to `0x01110000` after every erase, so timestamps are the correct
|
||||||
|
dedup field. Migration handles the transition transparently on first
|
||||||
|
startup.
|
||||||
|
|
||||||
|
**`device_family` (added v0.19.0)** discriminates Series III from Series
|
||||||
|
IV at the SQL level. Set by every import path; the UI dispatches on it
|
||||||
|
to render mic units correctly (Series III: psi → dBL conversion; Series
|
||||||
|
IV: native dBL passthrough). Existing rows are backfilled at first
|
||||||
|
startup of v0.19.0+ by sniffing the binary filename extension.
|
||||||
|
|
||||||
|
The on-disk waveform store lives at `bridges/captures/waveforms/<serial>/`
|
||||||
|
and holds the original event binaries (BW `.AB0*` / `.N00` for Series III,
|
||||||
|
`.IDFH` / `.IDFW` for Series IV) plus their `.sfm.json` review/metadata
|
||||||
|
sidecars. Series III events also produce `.a5.pkl` source-frame pickles
|
||||||
|
and `.h5` clean-waveform exports; Series IV doesn't yet (pending codec).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -311,18 +403,27 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
|
|||||||
|
|
||||||
## Key Features
|
## Key Features
|
||||||
|
|
||||||
**Device support:**
|
**Series III (MiniMate Plus) device support:**
|
||||||
- [x] Full read/write/erase pipelines
|
- [x] Full read/write/erase pipelines over RS-232 or TCP/cellular
|
||||||
- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings)
|
- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings)
|
||||||
- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries)
|
- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries)
|
||||||
- [x] Monitor control (start/stop, status polling, battery/memory)
|
- [x] Monitor control (start/stop, status polling, battery/memory)
|
||||||
- [x] Monitor log entries (continuous monitoring intervals without full waveform download)
|
- [x] Monitor log entries (continuous monitoring intervals without full waveform download)
|
||||||
|
- [x] Blastware file ingest at `/db/import/blastware_file` (paired with `series3-watcher`)
|
||||||
|
|
||||||
|
**Series IV (Micromate / Thor) device support:**
|
||||||
|
- [x] Thor IDF file ingest at `/db/import/idf_file` (paired with `thor-watcher`, v0.18.0+)
|
||||||
|
- [x] Native `IdfEvent` / `IdfReport` typed models — mic in dB(L), full title strings, sensor self-check, calibration, firmware version
|
||||||
|
- [x] Parser verified against 1,014 paired `.txt` sidecars in `thor-watcher/example-data/`
|
||||||
|
- [ ] Binary `.IDFW` / `.IDFH` codec — pending (see Roadmap + [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md))
|
||||||
|
- [ ] Live-device protocol — pending codec
|
||||||
|
|
||||||
**Data persistence:**
|
**Data persistence:**
|
||||||
- [x] SQLite database (`seismo_relay.db`) with 4 tables: ach_sessions, events, monitor_log, plus false_trigger flag
|
- [x] SQLite database (`seismo_relay.db`) with `events`, `monitor_log`, `ach_sessions` tables
|
||||||
- [x] Deduplication by waveform key (handles re-runs and repeat call-homes)
|
- [x] Per-row `device_family` column ("series3" / "series4") for clean UI / unit-of-measurement dispatch (v0.19.0+)
|
||||||
- [x] Post-erase key-reuse detection (tracks high-water mark)
|
- [x] Deduplication by `(serial, timestamp)` — natural key handles post-erase counter resets
|
||||||
- [x] Session state (`ach_state.json`) with downloaded keys and max key
|
- [x] UPSERT on re-import refreshes every device-authoritative field (peaks, project, sample_rate); preserves operator review state (`false_trigger`)
|
||||||
|
- [x] Post-erase key-reuse detection (tracks high-water mark in `ach_state.json`)
|
||||||
|
|
||||||
**REST API:**
|
**REST API:**
|
||||||
- [x] Live device endpoints with in-memory caching (`_LiveCache`)
|
- [x] Live device endpoints with in-memory caching (`_LiveCache`)
|
||||||
@@ -330,6 +431,7 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
|
|||||||
- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH)
|
- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH)
|
||||||
- [x] Call Home config read/write endpoints
|
- [x] Call Home config read/write endpoints
|
||||||
- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`)
|
- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`)
|
||||||
|
- [x] Import endpoints for both device families (`/db/import/blastware_file`, `/db/import/idf_file`)
|
||||||
|
|
||||||
**File output (v0.7+, byte-perfect as of v0.14.3):**
|
**File output (v0.7+, byte-perfect as of v0.14.3):**
|
||||||
- [x] Blastware-compatible `.AB0` / `.G10` file generation (waveform + metadata)
|
- [x] Blastware-compatible `.AB0` / `.G10` file generation (waveform + metadata)
|
||||||
@@ -359,8 +461,10 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
|
|||||||
|
|
||||||
### High-impact (unblocks product features)
|
### High-impact (unblocks product features)
|
||||||
|
|
||||||
- [ ] **Waveform body codec reverse-engineering.** The 5A bulk-stream body is some kind of compressed/encoded format (not raw int16 LE as previously assumed — see §7.6.1 retraction in `docs/instantel_protocol_reference.md`). Structural framing is ~50% decoded on branch `claude/codec-re-cBGNe` (tagged-block walker, segment counters); per-byte sample mapping is still open. Until this lands, the in-app waveform viewer renders garbage and BW-import peak values fall back to `_peaks_from_samples()` saturation noise. Workaround: pair every BW-imported event with its `_ASCII.TXT` so the device-authoritative peaks land in the DB regardless of codec.
|
- [ ] **Series III waveform body codec reverse-engineering.** The 5A bulk-stream body is some kind of compressed/encoded format (not raw int16 LE as previously assumed — see §7.6.1 retraction in `docs/instantel_protocol_reference.md`). Structural framing is ~50% decoded on branch `claude/codec-re-cBGNe` (tagged-block walker, segment counters); per-byte sample mapping is still open. Until this lands, the in-app waveform viewer renders garbage and BW-import peak values fall back to `_peaks_from_samples()` saturation noise. Workaround: pair every BW-imported event with its `_ASCII.TXT` so the device-authoritative peaks land in the DB regardless of codec.
|
||||||
- [ ] **In-app waveform viewer accuracy.** Depends on codec decode. Plot.v1 JSON pipeline + viewer skeleton already exist; will start showing real waveforms automatically once `_decode_a5_waveform` produces correct samples.
|
- [ ] **Series IV (Thor IDF) binary codec reverse-engineering.** `.IDFH` / `.IDFW` files are currently stored opaquely by `WaveformStore.save_imported_idf`, with all metadata sourced from the paired `.txt` sidecar. This works because thor-watcher forwards both files together, but operators who haven't enabled Thor's TXT exporter get rows with NULL peaks. Cracking the binary closes that gap and unlocks waveform display. Starting-point reference at [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md) — two observed file signatures (1,012 newer-firmware files + 2 old files whose layout matches the Series III STRT-record format), suggested first-session plan (~2-4 hrs), 1,014 paired binary+txt files available as ground truth in `thor-watcher/example-data/`. Code seam ready at `micromate/idf_file.py`.
|
||||||
|
- [ ] **In-app waveform viewer accuracy.** Depends on Series III codec decode. Plot.v1 JSON pipeline + viewer skeleton already exist; will start showing real waveforms automatically once `_decode_a5_waveform` produces correct samples. Series IV waveforms come online when the IDF codec lands.
|
||||||
|
- [ ] **Series IV live-device support.** Once the IDF binary is decoded, extend `micromate/` with `transport.py` / `framing.py` / `protocol.py` / `client.py` mirroring the `minimateplus/` package layout — depends on capturing Thor's wire protocol (TCP / RS-232 captures TBD).
|
||||||
- [ ] **Terra-view integration** — seismo-relay router, unit detail page, VISON-style event listing.
|
- [ ] **Terra-view integration** — seismo-relay router, unit detail page, VISON-style event listing.
|
||||||
- [ ] **Vibration summary reports** — highest legit PPV per project → Word doc (false-trigger filtering first).
|
- [ ] **Vibration summary reports** — highest legit PPV per project → Word doc (false-trigger filtering first).
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,284 @@
|
|||||||
|
# IDF Protocol Reference — Thor / Micromate Series IV
|
||||||
|
|
||||||
|
Starting-point reference for reverse-engineering Instantel's Micromate
|
||||||
|
Series IV event-file format. Sibling to
|
||||||
|
[instantel_protocol_reference.md](instantel_protocol_reference.md) (the
|
||||||
|
Series III "Rosetta Stone") — this doc holds what we know so far and
|
||||||
|
the open questions still to crack.
|
||||||
|
|
||||||
|
**Status (2026-05-20):** ASCII text sidecar fully decoded (1,014
|
||||||
|
sample files round-trip). Binary `.IDFH` / `.IDFW` codec
|
||||||
|
**not yet implemented** — binaries are stored opaquely by
|
||||||
|
`WaveformStore.save_imported_idf`, with metadata sourced from the
|
||||||
|
paired `.txt` sidecar.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File model
|
||||||
|
|
||||||
|
### Filename convention
|
||||||
|
|
||||||
|
```
|
||||||
|
<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>
|
||||||
|
```
|
||||||
|
|
||||||
|
- **SERIAL** — literal device serial, two-letter prefix + numeric
|
||||||
|
suffix. Examples seen: `UM11719`, `UM13981`, `UM20147`, `BE9439`.
|
||||||
|
Unlike Series III BW filenames (`M529LK44.AB0`, base-36 stem),
|
||||||
|
Series IV filenames carry the serial in plain text.
|
||||||
|
- **YYYYMMDDHHMMSS** — 14-char ASCII timestamp in **device local
|
||||||
|
time** (no timezone marker).
|
||||||
|
- **KIND** — `IDFH` for histograms, `IDFW` for waveforms.
|
||||||
|
|
||||||
|
The `.IDFH.txt` / `.IDFW.txt` ASCII sidecar lives in a `TXT/`
|
||||||
|
**subfolder** of the unit's directory, not alongside the binary.
|
||||||
|
This pairing convention is encoded in
|
||||||
|
`event_forwarder.idf_report_path()`.
|
||||||
|
|
||||||
|
### Directory layout
|
||||||
|
|
||||||
|
```
|
||||||
|
C:\THORDATA\
|
||||||
|
└── <Project>\
|
||||||
|
└── <UM####>\ ← unit serial dir
|
||||||
|
├── UM12345_20260520100000.MLG ← monitor log (not events)
|
||||||
|
├── UM12345_20260520100000.IDFH ← histogram event (binary)
|
||||||
|
├── UM12345_20260520100000.IDFW ← waveform event (binary)
|
||||||
|
├── UM12345_20260520100000.IDFW.CDB ← cache-DB variant (skip)
|
||||||
|
├── TXT\
|
||||||
|
│ ├── UM12345_20260520100000.IDFH.txt ← histogram ASCII sidecar
|
||||||
|
│ └── UM12345_20260520100000.IDFW.txt ← waveform ASCII sidecar
|
||||||
|
├── CSV\, HTML\, PDF\, XML\ ← operator-facing derived exports
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
The `.IDFW.CDB` files share the binary's basename but appear to be a
|
||||||
|
separate cache/database variant. Their first 8 bytes match the
|
||||||
|
**old**-firmware Thor signature (see below) regardless of which
|
||||||
|
signature the paired `.IDFW` uses. Purpose unknown; sizes vary
|
||||||
|
wildly (observed 123 B → 40,491 B). Thor-watcher's forwarder
|
||||||
|
deliberately skips them.
|
||||||
|
|
||||||
|
### Sample corpus
|
||||||
|
|
||||||
|
The `thor-watcher/example-data/THORDATA_example/` tree carries
|
||||||
|
**1,014 paired .IDFW / .IDFH + .txt files** spanning 2020–2023
|
||||||
|
across nine units (UM11719, UM13981, UM20147, …, plus BE9439 from
|
||||||
|
2020). This is the reverse-engineering ground truth.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ASCII sidecar (`.IDFW.txt` / `.IDFH.txt`) — fully decoded
|
||||||
|
|
||||||
|
Shape: plain text, one `"Key : Value"` line per metadata field,
|
||||||
|
followed for waveforms by a tab-separated sample table headed by
|
||||||
|
the literal line `Waveform Data Channels`. Parsed by
|
||||||
|
[`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py).
|
||||||
|
See [`micromate/models.py`](../micromate/models.py) for the typed
|
||||||
|
`IdfReport` shape.
|
||||||
|
|
||||||
|
### Notable conventions
|
||||||
|
|
||||||
|
- **Units are native to Thor** — geophone in **in/s**, microphone in
|
||||||
|
**dB(L)** (not psi like Series III BW reports), frequency in Hz,
|
||||||
|
acceleration in g, displacement in in.
|
||||||
|
- **Below-threshold readings** appear as the literal string
|
||||||
|
`<0.005 in/s` (155 occurrences in the sample corpus) — the parser
|
||||||
|
strips the `<` and treats the numeric remainder as the value.
|
||||||
|
- **Out-of-range / not-measured** values appear as `N/A` — parser
|
||||||
|
drops the field rather than letting the string leak into a numeric
|
||||||
|
column.
|
||||||
|
- **Firmware string** observed: `Micromate ISEE 11.0AK`.
|
||||||
|
- **TitleString1..4** are operator-defined free-text slots; Thor's
|
||||||
|
default labels map them to Location / Client / Company / Notes,
|
||||||
|
which the parser surfaces as `project` / `client` / `operator` /
|
||||||
|
`notes`.
|
||||||
|
- **Histogram sidecars** use `HistogramStartDate` / `HistogramStartTime`
|
||||||
|
in place of waveform's `EventDate` / `EventTime`. Parser falls
|
||||||
|
through to either.
|
||||||
|
- **Histogram tabular block** lacks the `Waveform Data Channels`
|
||||||
|
marker; instead it's a multi-line column header followed by
|
||||||
|
per-interval rows (`<date> <time> <tran-ppv> <freq> ...`). Parser
|
||||||
|
silently ignores lines after the metadata block since they lack a
|
||||||
|
colon-separated `key : value` shape (the timestamps DO contain
|
||||||
|
colons but produce garbage keys that don't collide with any
|
||||||
|
recognised field).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Binary header signatures (observed)
|
||||||
|
|
||||||
|
Hex dump of the first 32 bytes across 1,014 sample files reveals
|
||||||
|
**two distinct file signatures**, both anchored by the literal
|
||||||
|
ASCII string `"\x00Instantel\x00"` at offset 6–16:
|
||||||
|
|
||||||
|
### Signature A — newer firmware (1,012 files, 99.8% of corpus)
|
||||||
|
|
||||||
|
```
|
||||||
|
00000000: 0012 0100 0000 496e 7374 616e 7465 6c00 ......Instantel.
|
||||||
|
00000010: 0000 a695 002e b500 4f70 6572 6174 6f72 ........Operator
|
||||||
|
^^^^^^^^^^^^^^^^
|
||||||
|
operator/title string starts at 0x18
|
||||||
|
```
|
||||||
|
|
||||||
|
Header bytes 0–5: `00 12 01 00 00 00`. Followed immediately by the
|
||||||
|
8-byte ASCII tag, then 6 unknown bytes, then ASCII operator-supplied
|
||||||
|
strings (Operator name, etc.) and on through the project / client /
|
||||||
|
title strings. No `STRT` record observed in this layout.
|
||||||
|
|
||||||
|
### Signature B — older firmware (2 files: BE9439 from 2020)
|
||||||
|
|
||||||
|
```
|
||||||
|
00000000: 1000 0180 0000 496e 7374 616e 7465 6c00 ......Instantel.
|
||||||
|
00000010: 072c 0012 0300 5354 5254 fffe 0111 2340 .,....STRT....#@
|
||||||
|
^^^^^^^^^ ^^^^^^^^^
|
||||||
|
STRT magic 4-byte end_key
|
||||||
|
00000020: 0111 0000 2e5f 00ac 4600 0000 0200 0000 ....._..F.......
|
||||||
|
^^^^^^^^^ ^^^
|
||||||
|
4-byte start_key 0x46 (BW WAVEHDR record-type marker)
|
||||||
|
```
|
||||||
|
|
||||||
|
Header bytes 0–5: `10 00 01 80 00 00`. The structure after the
|
||||||
|
`Instantel` magic is **byte-for-byte identical to a BW SUB 5A
|
||||||
|
probe-response STRT record** as documented in
|
||||||
|
[instantel_protocol_reference.md → "SUB 5A — STRT record encodes
|
||||||
|
end_offset"](instantel_protocol_reference.md). Specifically:
|
||||||
|
|
||||||
|
| Offset | Bytes | Meaning (per BW reference) |
|
||||||
|
|--------|---------------------|--------------------------------------|
|
||||||
|
| 0x14 | `53 54 52 54` | `STRT` magic |
|
||||||
|
| 0x18 | `ff fe` | STRT sentinel |
|
||||||
|
| 0x1A | `01 11 23 40` | `end_key` (4 bytes) |
|
||||||
|
| 0x1E | `01 11 00 00` | `start_key` (4 bytes) |
|
||||||
|
| 0x26 | `46` | `0x46` waveform-record type marker |
|
||||||
|
|
||||||
|
**Hypothesis:** Older Micromate firmware writes a wrapped BW-format
|
||||||
|
event into the `.IDFW` file — essentially the same on-disk shape as
|
||||||
|
a Series III device, with the new filename convention applied at
|
||||||
|
export time. Newer firmware (signature A) abandoned the
|
||||||
|
BW-compatible layout for an Instantel-specific format.
|
||||||
|
|
||||||
|
If that hypothesis holds, the 2 signature-B files can already be
|
||||||
|
parsed via `minimateplus/event_file_io.read_blastware_file()` — worth
|
||||||
|
testing. The 1,012 signature-A files are the real reverse-engineering
|
||||||
|
target.
|
||||||
|
|
||||||
|
### `.IDFW.CDB` cache files
|
||||||
|
|
||||||
|
Always carry signature B (`10 00 01 80 ...`), even when the paired
|
||||||
|
`.IDFW` carries signature A. Plausible explanation: the CDB is an
|
||||||
|
internal Thor cache-database export that retains the legacy BW-style
|
||||||
|
record layout regardless of the user-facing `.IDFW` format version.
|
||||||
|
Not currently consumed by the forwarder.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File-size patterns (Signature A, the main target)
|
||||||
|
|
||||||
|
Survey of 1,012 signature-A files:
|
||||||
|
|
||||||
|
| Event type | Typical size | Source of variance |
|
||||||
|
|--------------|-------------------|----------------------------------------------|
|
||||||
|
| `.IDFW` 2-sec | 9,200 – 10,500 B | Operator-supplied strings (TitleString1..4) of varying length |
|
||||||
|
| `.IDFH` | 2,944 – 4,076 B | Histogram interval count (record duration / interval) |
|
||||||
|
|
||||||
|
**Naive arithmetic for 2-sec waveform:**
|
||||||
|
- 4 channels × 2 sec × 1024 sps = 8,192 samples
|
||||||
|
- At 2 bytes/sample (int16) = 16,384 sample bytes → file would be > 16 KB
|
||||||
|
- Observed: ~9–10 KB
|
||||||
|
- → samples are likely **1 byte each** (int8 quantised), **or** stored
|
||||||
|
with bit-packing / delta encoding, **or** only one channel's
|
||||||
|
full-rate samples are stored with the others reconstructed
|
||||||
|
arithmetically. Verifying this is the **first RE milestone**.
|
||||||
|
|
||||||
|
Project-string–length variance (~1 KB across the corpus) is consistent
|
||||||
|
with the file carrying a single copy of each TitleString1..4 plus
|
||||||
|
operator + setup-name as null-padded ASCII regions.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
The reverse-engineering targets, roughly in dependency order:
|
||||||
|
|
||||||
|
1. **Sample encoding (signature A)** — int8? int16 LE/BE? Bit-packed?
|
||||||
|
Delta-coded? Per-channel interleaved or sequential blocks?
|
||||||
|
2. **Header field layout (signature A)** — where do sample_rate,
|
||||||
|
record_time, channel count, and per-channel peaks live in the
|
||||||
|
binary? The ASCII sidecar gives the device-authoritative values,
|
||||||
|
so binary fields can be confirmed by diff.
|
||||||
|
3. **Operator-string offsets** — `Operator` at 0x18 is the first
|
||||||
|
visible string in signature-A files; the rest (project, client,
|
||||||
|
notes, setup) follow. Need to map exact offsets and null-padding
|
||||||
|
conventions.
|
||||||
|
4. **Signature-B → BW codec compatibility** — does
|
||||||
|
`minimateplus/event_file_io.read_blastware_file()` actually parse
|
||||||
|
the 2 BE9439 signature-B files as-is? If yes, the OLD-format
|
||||||
|
ingest is free.
|
||||||
|
5. **`.IDFW.CDB` purpose** — is it an internal Thor cache, a
|
||||||
|
ring-buffer dump, or something else? Worth a single small effort
|
||||||
|
to characterise so we know what we're skipping.
|
||||||
|
6. **Footer / checksum** — every BW event file has a footer; does
|
||||||
|
IDF? Where does the per-channel sample block end?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Reverse-engineering playbook (when we start)
|
||||||
|
|
||||||
|
The Series III BW codec took ~2 months of MITM wire captures
|
||||||
|
because we didn't have ground-truth metadata. Thor's situation is
|
||||||
|
**substantially better**:
|
||||||
|
|
||||||
|
- **Ground truth is on disk.** Every binary in `example-data/`
|
||||||
|
has a paired `.IDFW.txt` carrying the full decoded sample table
|
||||||
|
(`Waveform Data Channels` block — see any sample file in
|
||||||
|
`thor-watcher/example-data/.../TXT/`). Aligning binary bytes
|
||||||
|
to the table's float-per-row values gives an immediate per-byte
|
||||||
|
hypothesis test.
|
||||||
|
- **Cross-event diffing.** 1,012 signature-A samples from 9 units
|
||||||
|
spanning 4 years means any field that varies between events is
|
||||||
|
immediately localisable. Fields that are constant across all
|
||||||
|
files (firmware ID, channel labels, format-version word) are also
|
||||||
|
immediately localisable by complementary search.
|
||||||
|
- **No protocol surface.** Files at rest, not a wire dialect. No
|
||||||
|
DLE stuffing, no inner-frame parsing, no probe/data two-step.
|
||||||
|
|
||||||
|
Suggested first session (2-4 hours): hand-decode `UM11719_20231219162723.IDFW`
|
||||||
|
(10,290 bytes) against its `TXT/UM11719_20231219162723.IDFW.txt`
|
||||||
|
sample table (the 2-sec waveform at 1024 sps × 4 channels = 8,192
|
||||||
|
sample rows). Find the first per-channel sample value (`0.0003` in
|
||||||
|
the Tran column at t=0) in the binary. Confirms sample encoding.
|
||||||
|
Everything else flows from there.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Code seams ready to receive the codec
|
||||||
|
|
||||||
|
When the codec lands, it goes into
|
||||||
|
[`micromate/idf_file.py`](../micromate/idf_file.py) (currently a
|
||||||
|
stub raising `NotImplementedError`). Public API:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from micromate import IdfEvent
|
||||||
|
from micromate.idf_file import read_idf_file
|
||||||
|
|
||||||
|
event: IdfEvent = read_idf_file(Path("UM11719_20231219163444.IDFW"))
|
||||||
|
# event.peaks.transverse_ips, event.timestamp, event.raw_samples, ...
|
||||||
|
```
|
||||||
|
|
||||||
|
The ingest pipeline (`WaveformStore.save_imported_idf`) currently
|
||||||
|
builds the `IdfEvent` from the `.txt` parser only. Once
|
||||||
|
`read_idf_file()` works, the binary becomes authoritative; the
|
||||||
|
`.txt` parser drops to fast-path metadata cross-check. Operators
|
||||||
|
who don't enable Thor's TXT exporter still get fully populated
|
||||||
|
events.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## See also
|
||||||
|
|
||||||
|
- [instantel_protocol_reference.md](instantel_protocol_reference.md) — Series III BW protocol reference (the Rosetta Stone). STRT record format, DLE framing, BW filename encoding.
|
||||||
|
- [`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py) — `.txt` sidecar parser.
|
||||||
|
- [`micromate/models.py`](../micromate/models.py) — `IdfEvent`, `IdfReport` typed dataclasses.
|
||||||
|
- [`micromate/idf_file.py`](../micromate/idf_file.py) — placeholder for the binary codec.
|
||||||
|
- [`thor-watcher/example-data/THORDATA_example/`](../../thor-watcher/example-data/) — 1,014 paired binary + .txt files for codec validation.
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
"""
|
||||||
|
micromate — Instantel Micromate (Series IV) device library.
|
||||||
|
|
||||||
|
Sibling of ``minimateplus`` (the Series III library). Currently scoped to
|
||||||
|
the offline-file ingest path used by thor-watcher: parsing the per-event
|
||||||
|
``.IDFH``/``.IDFW`` ASCII text sidecars Thor's exporter writes alongside
|
||||||
|
each binary event file, and wrapping the parsed data in typed event
|
||||||
|
records.
|
||||||
|
|
||||||
|
Live-device support (TCP protocol, frame parsing, real-time monitoring)
|
||||||
|
is deferred — when we add it, it lands here as ``transport.py`` /
|
||||||
|
``framing.py`` / ``protocol.py`` / ``client.py``, mirroring the
|
||||||
|
``minimateplus`` package layout.
|
||||||
|
|
||||||
|
Typical usage (offline file ingest):
|
||||||
|
|
||||||
|
from micromate import IdfEvent, parse_idf_report
|
||||||
|
|
||||||
|
text = open("UM11719_20231219162723.IDFW.txt").read()
|
||||||
|
rep = parse_idf_report(text) # dict
|
||||||
|
event = IdfEvent.from_report(rep, "UM11719_20231219162723.IDFW")
|
||||||
|
print(event.serial, event.peaks.transverse_ips, event.mic_pspl_dbl)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .idf_ascii_report import (
|
||||||
|
parse_event_filename,
|
||||||
|
parse_idf_report,
|
||||||
|
serial_from_filename,
|
||||||
|
)
|
||||||
|
from .models import (
|
||||||
|
IdfEvent,
|
||||||
|
IdfPeaks,
|
||||||
|
IdfProjectInfo,
|
||||||
|
IdfReport,
|
||||||
|
IdfSensorCheck,
|
||||||
|
)
|
||||||
|
|
||||||
|
__version__ = "0.1.0"
|
||||||
|
__all__ = [
|
||||||
|
"IdfEvent",
|
||||||
|
"IdfPeaks",
|
||||||
|
"IdfProjectInfo",
|
||||||
|
"IdfReport",
|
||||||
|
"IdfSensorCheck",
|
||||||
|
"parse_event_filename",
|
||||||
|
"parse_idf_report",
|
||||||
|
"serial_from_filename",
|
||||||
|
]
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
"""
|
"""
|
||||||
sfm/idf_ascii_report.py — parse Thor (Micromate Series IV) IDF ASCII reports.
|
micromate/idf_ascii_report.py — parse Thor (Micromate Series IV) IDF ASCII reports.
|
||||||
|
|
||||||
Thor exports a `.IDFW.txt` or `.IDFH.txt` sidecar next to each `.IDFW`
|
Thor exports a `.IDFW.txt` or `.IDFH.txt` sidecar next to each `.IDFW`
|
||||||
(waveform) or `.IDFH` (histogram) event binary. Each sidecar is a
|
(waveform) or `.IDFH` (histogram) event binary. Each sidecar is a
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
"""
|
||||||
|
micromate/idf_file.py — placeholder for the Thor IDF binary codec.
|
||||||
|
|
||||||
|
Thor's ``.IDFH`` (histogram) and ``.IDFW`` (waveform) event files are an
|
||||||
|
Instantel proprietary binary format that has not yet been reverse-
|
||||||
|
engineered. Today seismo-relay treats them as opaque blobs:
|
||||||
|
``WaveformStore.save_imported_idf`` stores the bytes verbatim and reads
|
||||||
|
all device-authoritative metadata from the paired ``.IDFW.txt`` /
|
||||||
|
``.IDFH.txt`` ASCII sidecar (parsed by ``idf_ascii_report.py``).
|
||||||
|
|
||||||
|
When we crack the binary codec — same reverse-engineering playbook we
|
||||||
|
used to byte-perfect-parse Series III BW files (see
|
||||||
|
``docs/instantel_protocol_reference.md`` and ``minimateplus/event_file_io.py``)
|
||||||
|
— this module will grow:
|
||||||
|
|
||||||
|
- ``read_idf_file(path) -> IdfEvent``
|
||||||
|
Parse a ``.IDFW``/``.IDFH`` binary and return a fully populated
|
||||||
|
``IdfEvent`` whose waveform-sample arrays come from the binary
|
||||||
|
(the .txt sidecar's tabular sample block being a best-effort
|
||||||
|
check). Lets us ingest Thor events even when the operator
|
||||||
|
hasn't enabled the .txt exporter — closing the
|
||||||
|
``had_report=False`` gap that the thor-watcher forwarder
|
||||||
|
currently tolerates as a known limitation.
|
||||||
|
|
||||||
|
- ``write_idf_file(path, event)`` (eventually)
|
||||||
|
Round-trip event reconstruction, used for verifying the codec
|
||||||
|
against captured device files the way ``write_blastware_file``
|
||||||
|
verifies the Series III codec.
|
||||||
|
|
||||||
|
- Helpers for decoding the binary's per-channel sample arrays into
|
||||||
|
physical units, the per-event flash buffer's monitor-log records,
|
||||||
|
etc.
|
||||||
|
|
||||||
|
The reverse-engineering path: pair every ``.IDFW`` binary in
|
||||||
|
``thor-watcher/example-data/`` with its sibling ``.IDFW.txt``, treating
|
||||||
|
the txt's "Waveform Data Channels" block as ground-truth, and align
|
||||||
|
the binary's per-channel int16-or-similar arrays against it. Header
|
||||||
|
fields (sample rate, channel count, record time, timestamps) sit before
|
||||||
|
the sample block — same approach as the BW codec where ASCII strings
|
||||||
|
inside the binary (``Project:``, ``Client:``, etc.) anchored field
|
||||||
|
discovery.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Union
|
||||||
|
|
||||||
|
from .models import IdfEvent
|
||||||
|
|
||||||
|
|
||||||
|
def read_idf_file(path: Union[str, Path]) -> "IdfEvent":
|
||||||
|
"""Parse a Thor ``.IDFW``/``.IDFH`` binary into an ``IdfEvent``.
|
||||||
|
|
||||||
|
Not yet implemented. When implemented, this will be the canonical
|
||||||
|
entry point for reading Thor binaries — the ASCII sidecar parser
|
||||||
|
becomes an optional fast-path metadata supplement rather than the
|
||||||
|
sole source of device-authoritative data.
|
||||||
|
"""
|
||||||
|
raise NotImplementedError(
|
||||||
|
"IDF binary codec not yet implemented; the .IDFW/.IDFH binary format "
|
||||||
|
"is undecoded. Use parse_idf_report() on the paired .txt sidecar "
|
||||||
|
"for device-authoritative metadata."
|
||||||
|
)
|
||||||
@@ -0,0 +1,377 @@
|
|||||||
|
"""
|
||||||
|
Micromate (Series IV / Thor) native data models.
|
||||||
|
|
||||||
|
These are the right-shaped dataclasses for Thor data — Thor measures
|
||||||
|
the microphone in dB(L) directly, so this model carries
|
||||||
|
``mic_pspl_dbl`` rather than the pseudo-``psi`` shoehorn that
|
||||||
|
``minimateplus.PeakValues`` uses for Series III BW data.
|
||||||
|
|
||||||
|
The ingest pipeline today goes:
|
||||||
|
|
||||||
|
.IDFW.txt → parse_idf_report() → dict
|
||||||
|
dict → IdfEvent.from_report() → IdfEvent (typed)
|
||||||
|
IdfEvent → IdfEvent.to_minimateplus_event() → shape DB / sidecar
|
||||||
|
machinery expects
|
||||||
|
|
||||||
|
The ``to_minimateplus_event()`` bridge is a temporary boundary — when we
|
||||||
|
crack the binary IDF codec and have richer per-event data to store, the
|
||||||
|
DB schema will grow Series-IV-specific columns and the bridge will
|
||||||
|
shrink or disappear.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import datetime
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from typing import Any, Dict, Optional, Tuple
|
||||||
|
|
||||||
|
|
||||||
|
# ── IdfReport ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class IdfReport:
|
||||||
|
"""Typed wrapper around the dict returned by ``parse_idf_report``.
|
||||||
|
|
||||||
|
All fields optional — Thor's exporter is permissive and some IDF .txt
|
||||||
|
files (especially histograms) omit fields that waveform sidecars
|
||||||
|
include. Use ``.raw`` for any field this dataclass hasn't surfaced
|
||||||
|
yet (the parser keeps every recognised key in the raw dict).
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Identity / kind
|
||||||
|
serial_number: Optional[str] = None
|
||||||
|
event_type: Optional[str] = None # "Full Waveform" | "Full Histogram"
|
||||||
|
event_datetime: Optional[datetime.datetime] = None
|
||||||
|
filename: Optional[str] = None # echoed by Thor's exporter
|
||||||
|
|
||||||
|
# Sampling / timing
|
||||||
|
sample_rate: Optional[int] = None # samples/sec
|
||||||
|
record_time_sec: Optional[float] = None
|
||||||
|
pre_trigger_sec: Optional[float] = None
|
||||||
|
|
||||||
|
# Geophone peaks (in/s)
|
||||||
|
tran_ppv: Optional[float] = None
|
||||||
|
vert_ppv: Optional[float] = None
|
||||||
|
long_ppv: Optional[float] = None
|
||||||
|
peak_vector_sum: Optional[float] = None
|
||||||
|
|
||||||
|
# Microphone — Thor's native unit is dB(L), NOT psi.
|
||||||
|
mic_pspl_dbl: Optional[float] = None
|
||||||
|
|
||||||
|
# Zero-crossing frequencies (Hz)
|
||||||
|
tran_zc_freq: Optional[float] = None
|
||||||
|
vert_zc_freq: Optional[float] = None
|
||||||
|
long_zc_freq: Optional[float] = None
|
||||||
|
mic_zc_freq: Optional[float] = None
|
||||||
|
|
||||||
|
# Per-channel time of peak (sec, since event start)
|
||||||
|
tran_time_of_peak: Optional[float] = None
|
||||||
|
vert_time_of_peak: Optional[float] = None
|
||||||
|
long_time_of_peak: Optional[float] = None
|
||||||
|
mic_time_of_peak: Optional[float] = None
|
||||||
|
|
||||||
|
# Derived per-channel motion
|
||||||
|
tran_peak_acceleration: Optional[float] = None # g
|
||||||
|
vert_peak_acceleration: Optional[float] = None
|
||||||
|
long_peak_acceleration: Optional[float] = None
|
||||||
|
tran_peak_displacement: Optional[float] = None # in
|
||||||
|
vert_peak_displacement: Optional[float] = None
|
||||||
|
long_peak_displacement: Optional[float] = None
|
||||||
|
|
||||||
|
# Operator-supplied strings (Thor's TitleString1..4 → semantic slots)
|
||||||
|
project: Optional[str] = None # TitleString1
|
||||||
|
client: Optional[str] = None # TitleString2
|
||||||
|
operator: Optional[str] = None # TitleString3
|
||||||
|
notes: Optional[str] = None # TitleString4 / PostEventNote
|
||||||
|
setup: Optional[str] = None # setup file name
|
||||||
|
|
||||||
|
# Sensor self-check results
|
||||||
|
tran_test_passed: Optional[bool] = None
|
||||||
|
vert_test_passed: Optional[bool] = None
|
||||||
|
long_test_passed: Optional[bool] = None
|
||||||
|
mic_test_passed: Optional[bool] = None
|
||||||
|
|
||||||
|
# Device-fixed metadata
|
||||||
|
firmware_version: Optional[str] = None
|
||||||
|
calibration_text: Optional[str] = None
|
||||||
|
battery_volts: Optional[float] = None
|
||||||
|
|
||||||
|
# Original parser dict — preserves every recognised key (including
|
||||||
|
# raw unit-suffixed strings) for forward-compatible field access.
|
||||||
|
raw: Dict[str, Any] = field(default_factory=dict, repr=False)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_dict(cls, d: Dict[str, Any]) -> "IdfReport":
|
||||||
|
"""Build an IdfReport from the dict returned by ``parse_idf_report``."""
|
||||||
|
ed = d.get("event_datetime")
|
||||||
|
if isinstance(ed, str):
|
||||||
|
try:
|
||||||
|
ed = datetime.datetime.fromisoformat(ed)
|
||||||
|
except ValueError:
|
||||||
|
ed = None
|
||||||
|
|
||||||
|
return cls(
|
||||||
|
serial_number = d.get("serial_number"),
|
||||||
|
event_type = d.get("event_type"),
|
||||||
|
event_datetime = ed if isinstance(ed, datetime.datetime) else None,
|
||||||
|
filename = d.get("filename"),
|
||||||
|
sample_rate = d.get("sample_rate"),
|
||||||
|
record_time_sec = d.get("record_time_sec"),
|
||||||
|
pre_trigger_sec = d.get("pre_trigger_sec"),
|
||||||
|
tran_ppv = d.get("tran_ppv"),
|
||||||
|
vert_ppv = d.get("vert_ppv"),
|
||||||
|
long_ppv = d.get("long_ppv"),
|
||||||
|
peak_vector_sum = d.get("peak_vector_sum"),
|
||||||
|
mic_pspl_dbl = d.get("mic_ppv"), # parser names it mic_ppv (legacy)
|
||||||
|
tran_zc_freq = d.get("tran_zc_freq"),
|
||||||
|
vert_zc_freq = d.get("vert_zc_freq"),
|
||||||
|
long_zc_freq = d.get("long_zc_freq"),
|
||||||
|
mic_zc_freq = d.get("mic_zc_freq"),
|
||||||
|
tran_time_of_peak = d.get("tran_time_of_peak"),
|
||||||
|
vert_time_of_peak = d.get("vert_time_of_peak"),
|
||||||
|
long_time_of_peak = d.get("long_time_of_peak"),
|
||||||
|
mic_time_of_peak = d.get("mic_time_of_peak"),
|
||||||
|
tran_peak_acceleration = d.get("tran_peak_acceleration"),
|
||||||
|
vert_peak_acceleration = d.get("vert_peak_acceleration"),
|
||||||
|
long_peak_acceleration = d.get("long_peak_acceleration"),
|
||||||
|
tran_peak_displacement = d.get("tran_peak_displacement"),
|
||||||
|
vert_peak_displacement = d.get("vert_peak_displacement"),
|
||||||
|
long_peak_displacement = d.get("long_peak_displacement"),
|
||||||
|
project = d.get("project"),
|
||||||
|
client = d.get("client"),
|
||||||
|
operator = d.get("operator"),
|
||||||
|
notes = d.get("notes"),
|
||||||
|
setup = d.get("setup"),
|
||||||
|
tran_test_passed = d.get("tran_test_passed"),
|
||||||
|
vert_test_passed = d.get("vert_test_passed"),
|
||||||
|
long_test_passed = d.get("long_test_passed"),
|
||||||
|
mic_test_passed = d.get("mic_test_passed"),
|
||||||
|
firmware_version = d.get("version"),
|
||||||
|
calibration_text = d.get("calibration_text"),
|
||||||
|
battery_volts = d.get("battery_volts"),
|
||||||
|
raw = d,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ── IdfPeaks / IdfProjectInfo / IdfSensorCheck (narrow grouping types) ───────
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class IdfPeaks:
|
||||||
|
"""Geophone + mic peak values for one Thor event. Native Thor units."""
|
||||||
|
transverse_ips: Optional[float] = None # in/s
|
||||||
|
vertical_ips: Optional[float] = None # in/s
|
||||||
|
longitudinal_ips: Optional[float] = None # in/s
|
||||||
|
peak_vector_sum_ips: Optional[float] = None # in/s
|
||||||
|
mic_pspl_dbl: Optional[float] = None # dB(L)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class IdfProjectInfo:
|
||||||
|
"""Operator-supplied strings from Thor's TitleString1..4."""
|
||||||
|
project: Optional[str] = None
|
||||||
|
client: Optional[str] = None
|
||||||
|
operator: Optional[str] = None
|
||||||
|
notes: Optional[str] = None
|
||||||
|
setup: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class IdfSensorCheck:
|
||||||
|
"""Per-channel pass/fail from Thor's self-test."""
|
||||||
|
tran: Optional[bool] = None
|
||||||
|
vert: Optional[bool] = None
|
||||||
|
long: Optional[bool] = None
|
||||||
|
mic: Optional[bool] = None
|
||||||
|
|
||||||
|
|
||||||
|
# ── IdfEvent ─────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class IdfEvent:
|
||||||
|
"""A single Thor / Micromate Series IV event.
|
||||||
|
|
||||||
|
Built from a parsed .IDFW.txt or .IDFH.txt sidecar via
|
||||||
|
``IdfEvent.from_report()``. The filename is the authoritative
|
||||||
|
source for serial + timestamp + kind; the .txt provides
|
||||||
|
device-authoritative peak values, frequencies, project strings,
|
||||||
|
sensor self-check, firmware, calibration.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Identity
|
||||||
|
serial: str
|
||||||
|
timestamp: datetime.datetime
|
||||||
|
kind: str # "Waveform" | "Histogram"
|
||||||
|
filename: str # device-native binary filename, e.g. "UM11719_20231219163444.IDFW"
|
||||||
|
|
||||||
|
# Sampling / timing
|
||||||
|
sample_rate: Optional[int] = None
|
||||||
|
record_time_sec: Optional[float] = None
|
||||||
|
pre_trigger_sec: Optional[float] = None
|
||||||
|
|
||||||
|
# Peaks
|
||||||
|
peaks: IdfPeaks = field(default_factory=IdfPeaks)
|
||||||
|
|
||||||
|
# Per-channel frequencies (Hz)
|
||||||
|
tran_zc_freq: Optional[float] = None
|
||||||
|
vert_zc_freq: Optional[float] = None
|
||||||
|
long_zc_freq: Optional[float] = None
|
||||||
|
mic_zc_freq: Optional[float] = None
|
||||||
|
|
||||||
|
# Project strings
|
||||||
|
project_info: IdfProjectInfo = field(default_factory=IdfProjectInfo)
|
||||||
|
|
||||||
|
# Sensor self-check
|
||||||
|
sensor_check: IdfSensorCheck = field(default_factory=IdfSensorCheck)
|
||||||
|
|
||||||
|
# Device-fixed
|
||||||
|
firmware_version: Optional[str] = None
|
||||||
|
calibration_text: Optional[str] = None
|
||||||
|
battery_volts: Optional[float] = None
|
||||||
|
|
||||||
|
# The full parsed report — preserves anything not surfaced as a typed field
|
||||||
|
report: IdfReport = field(default_factory=IdfReport)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_report(
|
||||||
|
cls,
|
||||||
|
report: Any,
|
||||||
|
filename: str,
|
||||||
|
) -> "IdfEvent":
|
||||||
|
"""Build an IdfEvent from a parsed report (dict or IdfReport) and
|
||||||
|
the device-native binary filename.
|
||||||
|
|
||||||
|
The filename is authoritative for serial + timestamp + kind:
|
||||||
|
Thor's filenames are literal ``<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>``
|
||||||
|
and the device's own clock is the canonical event timestamp.
|
||||||
|
If the report carries an ``event_datetime`` that differs from
|
||||||
|
what's in the filename, the report wins (it has finer-grained
|
||||||
|
device-reported time-of-trigger semantics).
|
||||||
|
"""
|
||||||
|
from .idf_ascii_report import parse_event_filename
|
||||||
|
|
||||||
|
# Normalise input to IdfReport
|
||||||
|
if isinstance(report, IdfReport):
|
||||||
|
rep = report
|
||||||
|
elif isinstance(report, dict):
|
||||||
|
rep = IdfReport.from_dict(report)
|
||||||
|
else:
|
||||||
|
raise TypeError(
|
||||||
|
f"report must be IdfReport or dict; got {type(report).__name__}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Filename → (serial, timestamp, kind). Required — fall back to
|
||||||
|
# report-supplied values only if filename parsing fails.
|
||||||
|
parsed = parse_event_filename(filename)
|
||||||
|
if parsed is not None:
|
||||||
|
fn_serial, fn_ts, fn_kind = parsed
|
||||||
|
kind = "Histogram" if fn_kind == "IDFH" else "Waveform"
|
||||||
|
else:
|
||||||
|
fn_serial = rep.serial_number or "UNKNOWN"
|
||||||
|
fn_ts = rep.event_datetime or datetime.datetime(1970, 1, 1)
|
||||||
|
kind = "Waveform" if (rep.event_type or "").lower().startswith("full waveform") else "Histogram"
|
||||||
|
|
||||||
|
# Prefer report's event_datetime (device-authoritative) over the filename.
|
||||||
|
ts = rep.event_datetime or fn_ts
|
||||||
|
serial = rep.serial_number or fn_serial
|
||||||
|
|
||||||
|
return cls(
|
||||||
|
serial=serial,
|
||||||
|
timestamp=ts,
|
||||||
|
kind=kind,
|
||||||
|
filename=filename,
|
||||||
|
sample_rate=rep.sample_rate,
|
||||||
|
record_time_sec=rep.record_time_sec,
|
||||||
|
pre_trigger_sec=rep.pre_trigger_sec,
|
||||||
|
peaks=IdfPeaks(
|
||||||
|
transverse_ips = rep.tran_ppv,
|
||||||
|
vertical_ips = rep.vert_ppv,
|
||||||
|
longitudinal_ips = rep.long_ppv,
|
||||||
|
peak_vector_sum_ips = rep.peak_vector_sum,
|
||||||
|
mic_pspl_dbl = rep.mic_pspl_dbl,
|
||||||
|
),
|
||||||
|
tran_zc_freq=rep.tran_zc_freq,
|
||||||
|
vert_zc_freq=rep.vert_zc_freq,
|
||||||
|
long_zc_freq=rep.long_zc_freq,
|
||||||
|
mic_zc_freq=rep.mic_zc_freq,
|
||||||
|
project_info=IdfProjectInfo(
|
||||||
|
project=rep.project,
|
||||||
|
client=rep.client,
|
||||||
|
operator=rep.operator,
|
||||||
|
notes=rep.notes,
|
||||||
|
setup=rep.setup,
|
||||||
|
),
|
||||||
|
sensor_check=IdfSensorCheck(
|
||||||
|
tran=rep.tran_test_passed,
|
||||||
|
vert=rep.vert_test_passed,
|
||||||
|
long=rep.long_test_passed,
|
||||||
|
mic=rep.mic_test_passed,
|
||||||
|
),
|
||||||
|
firmware_version=rep.firmware_version,
|
||||||
|
calibration_text=rep.calibration_text,
|
||||||
|
battery_volts=rep.battery_volts,
|
||||||
|
report=rep,
|
||||||
|
)
|
||||||
|
|
||||||
|
# ── Bridge to minimateplus shape (for the existing DB / sidecar paths) ──
|
||||||
|
|
||||||
|
def to_minimateplus_event(self, waveform_key: bytes) -> Any:
|
||||||
|
"""Project this Thor event into the shape ``minimateplus.Event``
|
||||||
|
carries, so it can flow through the existing
|
||||||
|
``SeismoDb.insert_events()`` and ``event_to_sidecar_dict()``
|
||||||
|
machinery without those code paths needing to know about Thor.
|
||||||
|
|
||||||
|
Caveats of the bridge:
|
||||||
|
- ``mic_ppv`` on the produced Event carries Thor's dB(L) value
|
||||||
|
verbatim — the UI distinguishes via the ``device_family``
|
||||||
|
column (Phase 1). Don't run the BW psi→dBL converter on
|
||||||
|
Series IV rows.
|
||||||
|
- Many Thor-specific fields (Peak Acceleration / Displacement,
|
||||||
|
sensor self-check, calibration) don't have a slot in
|
||||||
|
``Event``. The full IdfReport is preserved on the
|
||||||
|
``.sfm.json`` sidecar under ``extensions.idf_report`` via
|
||||||
|
``save_imported_idf`` — that's the source of truth for them.
|
||||||
|
"""
|
||||||
|
from minimateplus.models import (
|
||||||
|
Event, PeakValues, ProjectInfo, Timestamp,
|
||||||
|
)
|
||||||
|
|
||||||
|
ts_obj = Timestamp(
|
||||||
|
raw=bytes(9),
|
||||||
|
flag=0,
|
||||||
|
year=self.timestamp.year,
|
||||||
|
unknown_byte=0,
|
||||||
|
month=self.timestamp.month,
|
||||||
|
day=self.timestamp.day,
|
||||||
|
hour=self.timestamp.hour,
|
||||||
|
minute=self.timestamp.minute,
|
||||||
|
second=self.timestamp.second,
|
||||||
|
)
|
||||||
|
pv = PeakValues(
|
||||||
|
tran=self.peaks.transverse_ips,
|
||||||
|
vert=self.peaks.vertical_ips,
|
||||||
|
long=self.peaks.longitudinal_ips,
|
||||||
|
micl=self.peaks.mic_pspl_dbl, # dB(L) — see caveat above
|
||||||
|
peak_vector_sum=self.peaks.peak_vector_sum_ips,
|
||||||
|
)
|
||||||
|
pi = ProjectInfo(
|
||||||
|
setup_name=self.project_info.setup,
|
||||||
|
project=self.project_info.project,
|
||||||
|
client=self.project_info.client,
|
||||||
|
operator=self.project_info.operator,
|
||||||
|
sensor_location=None, # Thor folds location into project string
|
||||||
|
notes=self.project_info.notes,
|
||||||
|
)
|
||||||
|
ev = Event(
|
||||||
|
index=0,
|
||||||
|
timestamp=ts_obj,
|
||||||
|
sample_rate=self.sample_rate,
|
||||||
|
peak_values=pv,
|
||||||
|
project_info=pi,
|
||||||
|
record_type=self.kind,
|
||||||
|
rectime_seconds=self.record_time_sec,
|
||||||
|
)
|
||||||
|
ev._waveform_key = waveform_key
|
||||||
|
return ev
|
||||||
+3
-3
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "seismo-relay"
|
name = "seismo-relay"
|
||||||
version = "0.18.0"
|
version = "0.19.0"
|
||||||
description = "Python client and REST server for MiniMate Plus seismographs"
|
description = "Python client and REST server for MiniMate Plus seismographs"
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
@@ -18,6 +18,6 @@ dependencies = [
|
|||||||
]
|
]
|
||||||
|
|
||||||
[tool.setuptools.packages.find]
|
[tool.setuptools.packages.find]
|
||||||
# Auto-discovers minimateplus/, sfm/, bridges/ as packages
|
# Auto-discovers minimateplus/, micromate/, sfm/, bridges/ as packages
|
||||||
where = ["."]
|
where = ["."]
|
||||||
include = ["minimateplus*", "sfm*", "bridges*"]
|
include = ["minimateplus*", "micromate*", "sfm*", "bridges*"]
|
||||||
|
|||||||
+1
-1
@@ -2491,7 +2491,7 @@ async def db_import_idf_file(
|
|||||||
|
|
||||||
1. Pair the binary with its `<binary>.txt` ASCII report when one
|
1. Pair the binary with its `<binary>.txt` ASCII report when one
|
||||||
is present in the same upload.
|
is present in the same upload.
|
||||||
2. Parse the report via `sfm.idf_ascii_report.parse_idf_report`
|
2. Parse the report via `micromate.parse_idf_report`
|
||||||
and copy the binary into the persistent store via
|
and copy the binary into the persistent store via
|
||||||
`WaveformStore.save_imported_idf`, writing a `.sfm.json`
|
`WaveformStore.save_imported_idf`, writing a `.sfm.json`
|
||||||
sidecar with `source.kind = "idf-import"`.
|
sidecar with `source.kind = "idf-import"`.
|
||||||
|
|||||||
+32
-89
@@ -426,99 +426,48 @@ class WaveformStore:
|
|||||||
`.IDFH`) produced by Thor's TXT exporter.
|
`.IDFH`) produced by Thor's TXT exporter.
|
||||||
|
|
||||||
Thor binaries are stored as opaque bytes — seismo-relay doesn't
|
Thor binaries are stored as opaque bytes — seismo-relay doesn't
|
||||||
decode the proprietary IDF binary format. Device-authoritative
|
yet decode the proprietary IDF binary format (codec slot lives
|
||||||
metadata comes from the paired `.IDFW.txt` / `.IDFH.txt` sidecar
|
at ``micromate/idf_file.py``). Device-authoritative metadata
|
||||||
when supplied; we parse that text and surface its fields onto
|
comes from the paired ``.IDFW.txt`` / ``.IDFH.txt`` sidecar
|
||||||
the returned Event so the SFM database row has real PPV/project
|
when supplied.
|
||||||
values instead of NULLs.
|
|
||||||
|
|
||||||
Workflow:
|
Workflow:
|
||||||
1. Parse the paired TXT report (when supplied) via
|
1. Parse the paired TXT report (when supplied) via
|
||||||
`sfm.idf_ascii_report.parse_idf_report`.
|
``micromate.parse_idf_report`` → dict.
|
||||||
2. Build a minimal `Event` populated from the report fields
|
2. Wrap parsed dict + filename into a typed ``micromate.IdfEvent``.
|
||||||
(timestamp, peaks, project info, sample_rate, record_type).
|
3. Copy bytes verbatim into ``<root>/<serial>/<filename>``.
|
||||||
3. Resolve serial from filename prefix or `serial_hint`.
|
4. Bridge IdfEvent → ``minimateplus.Event`` (for the existing
|
||||||
4. Copy bytes verbatim into <root>/<serial>/<filename>.
|
sidecar / DB insert machinery) via
|
||||||
5. Write the `.sfm.json` sidecar with source.kind = "idf-import".
|
``IdfEvent.to_minimateplus_event(waveform_key)``.
|
||||||
|
5. Write the ``.sfm.json`` sidecar with
|
||||||
|
``source.kind = "idf-import"`` and the full raw IDF report
|
||||||
|
under ``extensions.idf_report``.
|
||||||
|
|
||||||
Returns (event, record_dict) so the endpoint can both insert
|
Returns ``(event, record_dict)`` so the endpoint can both insert
|
||||||
into SeismoDb and surface the parsed event.
|
into SeismoDb and surface the parsed event.
|
||||||
"""
|
"""
|
||||||
from sfm.idf_ascii_report import (
|
from micromate import IdfEvent, parse_idf_report
|
||||||
parse_idf_report,
|
|
||||||
parse_event_filename,
|
|
||||||
serial_from_filename as _idf_serial_from_filename,
|
|
||||||
)
|
|
||||||
from minimateplus.models import (
|
|
||||||
Event, PeakValues, ProjectInfo, Timestamp,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Parse the .txt sidecar (best-effort; non-fatal on failure).
|
# Parse the .txt sidecar (best-effort; non-fatal on failure).
|
||||||
report: dict = {}
|
report_dict: dict = {}
|
||||||
if idf_report_text is not None:
|
if idf_report_text is not None:
|
||||||
try:
|
try:
|
||||||
report = parse_idf_report(idf_report_text)
|
report_dict = parse_idf_report(idf_report_text)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
log.warning(
|
log.warning(
|
||||||
"save_imported_idf: report parse failed: %s — continuing without it",
|
"save_imported_idf: report parse failed: %s — continuing without it",
|
||||||
exc,
|
exc,
|
||||||
)
|
)
|
||||||
|
|
||||||
# Resolve serial: prefer the explicit hint, fall back to filename prefix.
|
# Build the typed IdfEvent. Filename is authoritative for
|
||||||
serial = (
|
# (serial, timestamp, kind); the report's event_datetime takes
|
||||||
serial_hint
|
# precedence over the filename timestamp inside from_report().
|
||||||
or report.get("serial_number")
|
idf_event = IdfEvent.from_report(report_dict, source_path.name)
|
||||||
or _idf_serial_from_filename(source_path.name)
|
|
||||||
or "UNKNOWN"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Resolve event timestamp + kind from the filename (always present).
|
# Operator-supplied serial_hint wins over the binary's filename
|
||||||
parsed_name = parse_event_filename(source_path.name)
|
# prefix when both are present (e.g. callers passing a known-good
|
||||||
kind = "Waveform"
|
# serial that overrides a misnamed export).
|
||||||
ts_dt: Optional[datetime.datetime] = None
|
serial = serial_hint or idf_event.serial or "UNKNOWN"
|
||||||
if parsed_name is not None:
|
|
||||||
_, ts_dt, kind_token = parsed_name
|
|
||||||
kind = "Histogram" if kind_token == "IDFH" else "Waveform"
|
|
||||||
# Report's event_datetime is the device-authoritative value; prefer it.
|
|
||||||
if "event_datetime" in report:
|
|
||||||
try:
|
|
||||||
ts_dt = datetime.datetime.fromisoformat(report["event_datetime"])
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
pass
|
|
||||||
|
|
||||||
ts_obj: Optional[Timestamp] = None
|
|
||||||
if ts_dt is not None:
|
|
||||||
ts_obj = Timestamp(
|
|
||||||
raw=bytes(9),
|
|
||||||
flag=0,
|
|
||||||
year=ts_dt.year,
|
|
||||||
unknown_byte=0,
|
|
||||||
month=ts_dt.month,
|
|
||||||
day=ts_dt.day,
|
|
||||||
hour=ts_dt.hour,
|
|
||||||
minute=ts_dt.minute,
|
|
||||||
second=ts_dt.second,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Build PeakValues from the report (fields are None when absent).
|
|
||||||
pv = PeakValues(
|
|
||||||
tran=report.get("tran_ppv"),
|
|
||||||
vert=report.get("vert_ppv"),
|
|
||||||
long=report.get("long_ppv"),
|
|
||||||
micl=report.get("mic_ppv"),
|
|
||||||
peak_vector_sum=report.get("peak_vector_sum"),
|
|
||||||
)
|
|
||||||
|
|
||||||
# Build ProjectInfo. See idf_ascii_report — Thor's title strings
|
|
||||||
# carry project / client / company / notes in TitleString1..4.
|
|
||||||
pi = ProjectInfo(
|
|
||||||
setup_name=report.get("setup"),
|
|
||||||
project=report.get("project"),
|
|
||||||
client=report.get("client"),
|
|
||||||
operator=report.get("operator"),
|
|
||||||
sensor_location=None, # Thor folds location into TitleString1 = project
|
|
||||||
notes=report.get("notes"),
|
|
||||||
)
|
|
||||||
|
|
||||||
# Filesystem write.
|
# Filesystem write.
|
||||||
filename = source_path.name
|
filename = source_path.name
|
||||||
@@ -532,16 +481,10 @@ class WaveformStore:
|
|||||||
# surrogate — every distinct binary maps to a distinct row.
|
# surrogate — every distinct binary maps to a distinct row.
|
||||||
waveform_key = bytes.fromhex(sha256)[:16]
|
waveform_key = bytes.fromhex(sha256)[:16]
|
||||||
|
|
||||||
ev = Event(
|
# Bridge to minimateplus.Event for the existing sidecar / DB
|
||||||
index=0,
|
# insert paths. See IdfEvent.to_minimateplus_event() for the
|
||||||
timestamp=ts_obj,
|
# caveats of this bridge (mic units, missing fields → sidecar).
|
||||||
sample_rate=report.get("sample_rate"),
|
ev = idf_event.to_minimateplus_event(waveform_key)
|
||||||
peak_values=pv,
|
|
||||||
project_info=pi,
|
|
||||||
record_type=kind,
|
|
||||||
rectime_seconds=report.get("record_time_sec"),
|
|
||||||
)
|
|
||||||
ev._waveform_key = waveform_key
|
|
||||||
|
|
||||||
# Write the sidecar. Source kind "idf-import" was added to the
|
# Write the sidecar. Source kind "idf-import" was added to the
|
||||||
# allow-list in event_file_io.event_to_sidecar_dict for this.
|
# allow-list in event_file_io.event_to_sidecar_dict for this.
|
||||||
@@ -567,14 +510,14 @@ class WaveformStore:
|
|||||||
# consumers can recover the rich derived fields that don't fit
|
# consumers can recover the rich derived fields that don't fit
|
||||||
# the BW-shaped event model (Peak Acceleration / Displacement,
|
# the BW-shaped event model (Peak Acceleration / Displacement,
|
||||||
# Time of Peak, sensor self-check, calibration, firmware).
|
# Time of Peak, sensor self-check, calibration, firmware).
|
||||||
if report:
|
if report_dict:
|
||||||
sidecar["extensions"]["idf_report"] = report
|
sidecar["extensions"]["idf_report"] = report_dict
|
||||||
event_file_io.write_sidecar(sidecar_path, sidecar)
|
event_file_io.write_sidecar(sidecar_path, sidecar)
|
||||||
|
|
||||||
log.info(
|
log.info(
|
||||||
"WaveformStore.save_imported_idf serial=%s filename=%s filesize=%d "
|
"WaveformStore.save_imported_idf serial=%s filename=%s filesize=%d "
|
||||||
"report_attached=%s",
|
"report_attached=%s",
|
||||||
serial, filename, filesize, bool(report),
|
serial, filename, filesize, bool(report_dict),
|
||||||
)
|
)
|
||||||
return ev, {
|
return ev, {
|
||||||
"filename": filename,
|
"filename": filename,
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ import pytest
|
|||||||
|
|
||||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
|
||||||
from sfm.idf_ascii_report import (
|
from micromate.idf_ascii_report import (
|
||||||
parse_event_filename,
|
parse_event_filename,
|
||||||
parse_idf_report,
|
parse_idf_report,
|
||||||
serial_from_filename,
|
serial_from_filename,
|
||||||
|
|||||||
Reference in New Issue
Block a user