Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 claude
|
e1a6fd5386 | ||
|
claude
|
6b875e161b | ||
|
claude
|
f5c81f2cab |
@@ -1,28 +0,0 @@
|
|||||||
.git
|
|
||||||
.gitignore
|
|
||||||
|
|
||||||
.venv
|
|
||||||
venv
|
|
||||||
env
|
|
||||||
__pycache__
|
|
||||||
*.pyc
|
|
||||||
*.pyo
|
|
||||||
*.pyd
|
|
||||||
.pytest_cache
|
|
||||||
.mypy_cache
|
|
||||||
.ruff_cache
|
|
||||||
|
|
||||||
*.db
|
|
||||||
*.db-wal
|
|
||||||
*.db-shm
|
|
||||||
*.sqlite
|
|
||||||
*.sqlite3
|
|
||||||
|
|
||||||
sfm/data
|
|
||||||
bridges/captures
|
|
||||||
example-events
|
|
||||||
captures
|
|
||||||
logs
|
|
||||||
|
|
||||||
.DS_Store
|
|
||||||
Thumbs.db
|
|
||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
/bridges/captures/
|
/bridges/captures/
|
||||||
/example-events/
|
/example-events/
|
||||||
/tests/fixtures/
|
|
||||||
/manuals/
|
/manuals/
|
||||||
|
|
||||||
# Python build artifacts
|
# Python build artifacts
|
||||||
|
|||||||
-1112
File diff suppressed because it is too large
Load Diff
@@ -2,142 +2,12 @@
|
|||||||
|
|
||||||
Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
|
Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
|
||||||
managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
|
managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
|
||||||
(Sierra Wireless RV50 / RV55). Current version: **v0.26.0**.
|
(Sierra Wireless RV50 / RV55). Current version: **v0.12.3**.
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Where things stand (updated 2026-08-27)
|
|
||||||
|
|
||||||
Read this first when picking the project back up.
|
|
||||||
|
|
||||||
- **Series-3 decode is correct and verified.** All 11,603 series-3 binaries in
|
|
||||||
the prod snapshot pass every check (channel lengths, peaks vs the device's
|
|
||||||
own reported PPV, nothing above full scale, length vs declared record time).
|
|
||||||
Ground truth: 1,211/1,211 histograms exact per-interval and 75/75 waveform
|
|
||||||
sample counts exact against preserved Blastware ASCII exports.
|
|
||||||
⚠ That is per-sample proof on 11% of files and peak-only consistency on the
|
|
||||||
other 89% — see `docs/instantel_protocol_reference.md` §7.6.1.
|
|
||||||
- **Series-4 (Thor / Micromate) is NOT verified.** UM-series sits at ~48%
|
|
||||||
against device peaks with a ~1.7% systematic bias and a near-zero tail.
|
|
||||||
Thor IDFW is pinned to `decode_waveform_legacy` deliberately.
|
|
||||||
- **Open, not blocking:** 14 sensitive-range files show an exact 8x
|
|
||||||
(= 10.0/1.25) units discrepancy; `scripts/backfill_sidecars.py --force` also
|
|
||||||
inserts DB rows for store files that have none (one-time per store) and the
|
|
||||||
dry-run does not report that count.
|
|
||||||
- **After any codec change, regenerate the store** — `backfill_sidecars.py
|
|
||||||
--force` then `backfill_event_shape.py`, DB backup first. Stored `.h5` files
|
|
||||||
do not update themselves.
|
|
||||||
- **The "offset" hardware fault has its own journal** --
|
|
||||||
`docs/offset_investigation.md`. Base rate settled at 5-6 of 45 units
|
|
||||||
(11-13%) across the DL2 archive; Instantel's own autozero procedure and
|
|
||||||
its 2027-2069 go/no-go window are recorded there. Best open lead is
|
|
||||||
`SUB 0x0E` (unimplemented), which may carry those numbers.
|
|
||||||
|
|
||||||
|
|
||||||
When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
|
When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Architecture: three-tier conceptual model
|
|
||||||
|
|
||||||
seismo-relay is a **suite of cooperating components**, not a single app.
|
|
||||||
The three tiers below are the canonical mental model — the current
|
|
||||||
directory layout doesn't fully reflect them yet (some of what is
|
|
||||||
conceptually SDM lives under `sfm/` today), but new code should be
|
|
||||||
placed and named according to this model.
|
|
||||||
|
|
||||||
### 1. SFM — the device-side (active connection to physical units)
|
|
||||||
|
|
||||||
Replaces Blastware's *talk-to-the-meter* role. Lives where a connection
|
|
||||||
to a physical seismograph is open.
|
|
||||||
|
|
||||||
In scope:
|
|
||||||
- `minimateplus/{transport,framing,protocol,client}.py` — wire protocol
|
|
||||||
- `seismo_lab.py` — diagnostic GUI (a thick client for SFM)
|
|
||||||
- The `/device/*` HTTP endpoints in `sfm/server.py` —
|
|
||||||
`/device/info`, `/device/events`, `/device/monitor/*`, `/device/call_home`,
|
|
||||||
etc. Anything that opens a connection at the moment of the request.
|
|
||||||
- Future: a Thor / Micromate live client (mirror `minimateplus/`)
|
|
||||||
- Future: a control surface Terra-View can launch into — see the
|
|
||||||
README's Roadmap.
|
|
||||||
|
|
||||||
Does NOT own a database. Outputs `Event` objects. Has a "spun up when
|
|
||||||
needed" runtime profile rather than "always on".
|
|
||||||
|
|
||||||
### 2. SDM — the data-side (storage, ingest, and serving)
|
|
||||||
|
|
||||||
The new name for the receiving-and-storing role. Originally called SFM
|
|
||||||
because the FastAPI service started life as a thin device proxy, but
|
|
||||||
the actual role has migrated heavily toward data management. **For now
|
|
||||||
the directory remains `sfm/`** — renaming requires touching ~30-50
|
|
||||||
files in seismo-relay + ~10-15 in terra-view + a Docker volume
|
|
||||||
migration; deferred until the codebase is quiet enough to do it as a
|
|
||||||
clean refactor.
|
|
||||||
|
|
||||||
In scope:
|
|
||||||
- `sfm/database.py` (`SeismoDb`)
|
|
||||||
- `sfm/waveform_store.py`, `sfm/event_hdf5.py`
|
|
||||||
- The `/db/*` HTTP endpoints — `events`, `units`, `monitor_log`,
|
|
||||||
`sessions`, `false_trigger` mutations
|
|
||||||
- The `/db/import/*` ingest endpoints — `blastware_file` (series3),
|
|
||||||
`idf_file` (series4); anything that receives events FROM somewhere
|
|
||||||
- `scripts/backfill_sidecars.py`, `scripts/check_bw_report_preservation.py`,
|
|
||||||
and similar data-maintenance tools
|
|
||||||
- The `.sfm.json` sidecars and `.h5` files in the waveform store
|
|
||||||
- The shape that Terra-View consumes (Terra-View should never need to
|
|
||||||
reach into SFM/device-side endpoints to populate its UI)
|
|
||||||
|
|
||||||
Always-on, scaled for storage/serving, has the DB and waveform store.
|
|
||||||
|
|
||||||
### 3. Codec library — pure data interpretation (used by both sides)
|
|
||||||
|
|
||||||
Neither SFM nor SDM — a shared library both depend on.
|
|
||||||
|
|
||||||
In scope:
|
|
||||||
- `minimateplus/{waveform_codec,histogram_codec,event_file_io,bw_ascii_report,blastware_file}.py`
|
|
||||||
- `micromate/{idf_ascii_report,idf_file}.py`
|
|
||||||
|
|
||||||
These modules take bytes (off the wire on the SFM side, or from a
|
|
||||||
forwarded file on the SDM side) and return `Event` objects. They
|
|
||||||
should not import from `sfm/`, must not touch a DB, and have no I/O
|
|
||||||
beyond reading files passed as arguments. Keep them pure — both
|
|
||||||
tiers can then depend on them without circularity.
|
|
||||||
|
|
||||||
#### Thor IDF binary codec (2026-05-28)
|
|
||||||
|
|
||||||
`micromate/idf_file.read_idf_file()` decodes both Thor IDFW
|
|
||||||
(waveform) and IDFH (histogram) binaries.
|
|
||||||
|
|
||||||
- **IDFW** reuses `decode_waveform_v2()` on the body at fixed file
|
|
||||||
offset `0x0f1f`. Sample fidelity is 87–99% byte-exact on quiet
|
|
||||||
events; loud events hit the BW codec's known walker-stops-early
|
|
||||||
limitation.
|
|
||||||
- **IDFH** has its own segment-based decoder: `[len_be][0a 00 00 00]
|
|
||||||
[00 NN][05 3f]` + N × 72-byte interval records (4 × 16-byte
|
|
||||||
per-channel min/max/halfp). All 859 Thor IDFH corpus files
|
|
||||||
decode (181,071 intervals); peak matches sidecar within ~1.8%
|
|
||||||
(ADC quantization).
|
|
||||||
|
|
||||||
The two outlier `BE9439_*` files in the Thor example corpus are
|
|
||||||
actually Series III Blastware binaries that share the `.IDFW`/`.IDFH`
|
|
||||||
filename convention by accident. `read_idf_file()` detects them by
|
|
||||||
their BW STRT signature and raises NotImplementedError pointing
|
|
||||||
callers at `read_blastware_file()`. See
|
|
||||||
`docs/idf_protocol_reference.md` for full field layouts.
|
|
||||||
|
|
||||||
### Practical consequences
|
|
||||||
|
|
||||||
When deciding where new code goes, ask:
|
|
||||||
- *Does it need a connection to a device?* → SFM
|
|
||||||
- *Does it operate on stored events / sidecars / DB rows?* → SDM
|
|
||||||
- *Does it interpret bytes into structured data, with no I/O of its own?* → codec lib
|
|
||||||
|
|
||||||
Terra-View is downstream of SDM for data, and (per the roadmap) will
|
|
||||||
eventually invoke into SFM's device-control endpoints to provide a
|
|
||||||
"connect to unit" experience.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Project layout
|
## Project layout
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -147,8 +17,6 @@ minimateplus/ ← Python client library (primary focus)
|
|||||||
protocol.py ← MiniMateProtocol — wire-level read/write methods
|
protocol.py ← MiniMateProtocol — wire-level read/write methods
|
||||||
client.py ← MiniMateClient — high-level API (connect, get_events, …)
|
client.py ← MiniMateClient — high-level API (connect, get_events, …)
|
||||||
models.py ← DeviceInfo, EventRecord, ComplianceConfig, …
|
models.py ← DeviceInfo, EventRecord, ComplianceConfig, …
|
||||||
waveform_codec.py ← Body-codec block walker + decode_tran_initial (partial
|
|
||||||
per-sample decoder — see "Waveform body codec" section below)
|
|
||||||
|
|
||||||
sfm/server.py ← FastAPI REST server exposing device data over HTTP
|
sfm/server.py ← FastAPI REST server exposing device data over HTTP
|
||||||
seismo_lab.py ← Tkinter GUI (Bridge + Analyzer + Console tabs)
|
seismo_lab.py ← Tkinter GUI (Bridge + Analyzer + Console tabs)
|
||||||
@@ -159,7 +27,7 @@ CHANGELOG.md ← version history
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Current implementation state (v0.14.3)
|
## Current implementation state (v0.12.3)
|
||||||
|
|
||||||
Full read pipeline + write pipeline + erase pipeline + monitor log + call home config working end-to-end over TCP/cellular:
|
Full read pipeline + write pipeline + erase pipeline + monitor log + call home config working end-to-end over TCP/cellular:
|
||||||
|
|
||||||
@@ -173,15 +41,14 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c
|
|||||||
| Event header / first key | 1E | ✅ |
|
| Event header / first key | 1E | ✅ |
|
||||||
| Waveform header | 0A | ✅ |
|
| Waveform header | 0A | ✅ |
|
||||||
| Waveform record (peaks, timestamp, project) | 0C | ✅ |
|
| Waveform record (peaks, timestamp, project) | 0C | ✅ |
|
||||||
| **Bulk waveform stream (event-time metadata + full waveform)** | **5A** | ✅ **byte-perfect against BW captures (v0.14.3, 2026-05-05)** — STRT-bounded chunk walk + correct event-N probe counter + DLE-stuffed `0x10` bytes in params + concatenate-only file body assembly. All 17 5A request frames in the 5-1-26 3-sec capture reproduce byte-for-byte. |
|
| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ new v0.6.0 |
|
||||||
| Event advance / next key | 1F | ✅ |
|
| Event advance / next key | 1F | ✅ |
|
||||||
| **Write commands (push config to device)** | **68–83** | ✅ new v0.8.0 |
|
| **Write commands (push config to device)** | **68–83** | ✅ new v0.8.0 |
|
||||||
| **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 |
|
| **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 |
|
||||||
| **Monitor log entries (partial 0x2C records)** | **0A browse** | ✅ new v0.10.0 |
|
| **Monitor log entries (partial 0x2C records)** | **0A browse** | ✅ new v0.10.0 |
|
||||||
| **Auto Call Home config (read + write)** | **2C → 7E → 7F** | ✅ **new v0.12.3** |
|
| **Auto Call Home config (read + write)** | **2C → 7E → 7F** | ✅ **new v0.12.3** |
|
||||||
|
|
||||||
`get_events()` sequence per event: `1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)`
|
`get_events()` sequence per event: `1E → 0A → 0C → 5A → 1F`
|
||||||
(see "Correct iteration pattern" section below for full detail)
|
|
||||||
|
|
||||||
`push_config_raw()` write sequence: `68→73 | 71×3→72 | 82→83 | 69→74→72`
|
`push_config_raw()` write sequence: `68→73 | 71×3→72 | 82→83 | 69→74→72`
|
||||||
|
|
||||||
@@ -189,269 +56,6 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Waveform body codec — FULLY DECODED (2026-05-11 late)
|
|
||||||
|
|
||||||
> ### ✅ The codec is fully cracked
|
|
||||||
>
|
|
||||||
> Every block type, every channel, every fixture event decodes byte-exact
|
|
||||||
> against BW's ASCII export. **47,364 ADC samples verified, zero errors.**
|
|
||||||
> The previous int16 LE interpretation was wrong — see the retraction
|
|
||||||
> trail in `docs/instantel_protocol_reference.md §7.6.1`.
|
|
||||||
>
|
|
||||||
> Authoritative implementation: `minimateplus/waveform_codec.py`
|
|
||||||
> (`decode_waveform_v2()`). Clean working notes:
|
|
||||||
> `docs/waveform_codec_re_status.md`.
|
|
||||||
>
|
|
||||||
> **NOTE:** `client.py:_decode_a5_waveform` still uses the broken
|
|
||||||
> legacy int16 LE decoder. Wiring `decode_waveform_v2` into the
|
|
||||||
> `.h5` sidecar path is the obvious next follow-up. Until that lands,
|
|
||||||
> `.h5` samples remain wrong — but the codec itself is fully solved.
|
|
||||||
|
|
||||||
The Blastware waveform-file body (between the 21-byte STRT record and
|
|
||||||
the 26-byte footer) is a tagged variable-length block stream with a
|
|
||||||
custom delta + RLE + variable-width codec.
|
|
||||||
|
|
||||||
### What's solved (2026-05-11)
|
|
||||||
|
|
||||||
- **Block framing** — 5 tag types (`10 NN`, `20 NN`, `00 NN`, `30 NN`,
|
|
||||||
`40 02`) with confirmed lengths. Implementation: `walk_body()` in
|
|
||||||
`minimateplus/waveform_codec.py`.
|
|
||||||
- **Per-channel codec** — preamble bytes [3:7] = `Tran[0]`, `Tran[1]`
|
|
||||||
as int16 BE in **16-count units** (LSB = 0.005 in/s). Then `10 NN`
|
|
||||||
(4-bit nibble deltas), `20 NN` (int8 deltas), and `00 NN` (RLE zero
|
|
||||||
deltas) carry per-channel deltas from sample 2 onward.
|
|
||||||
- **Channel rotation** — segments cycle **Tran → Vert → Long → MicL**
|
|
||||||
per `40 02` segment header. Each segment carries ~512 sample-sets of
|
|
||||||
ONE channel. The initial body (before the first `40 02`) is the
|
|
||||||
implicit Tran segment.
|
|
||||||
- **Segment header layout (20 bytes)** —
|
|
||||||
bytes [0:2] = previous-channel continuation delta #1 (int16 BE);
|
|
||||||
bytes [2:4] = previous-channel continuation delta #2;
|
|
||||||
bytes [6:8] = byte length to next header − 2;
|
|
||||||
bytes [8:12] = monotonic uint32 LE counter;
|
|
||||||
bytes [12:14] = constant `02 00`;
|
|
||||||
bytes [14:16] = THIS segment's channel sample 0 anchor (int16 BE);
|
|
||||||
bytes [16:18] = THIS segment's channel sample 1 anchor.
|
|
||||||
- **`decode_waveform_v2()`** returns full per-channel sample dicts.
|
|
||||||
Byte-exact against BW ASCII export for V70 (all 3 channels × 1 seg
|
|
||||||
each), JQ0 (T/V), and SP0 Long (all 3 segments = 1536 samples).
|
|
||||||
|
|
||||||
- **`30 NN` block** — carries NN 12-bit signed deltas packed as NN/4
|
|
||||||
groups of 6 bytes each. Within each group, bytes [0:2] hold 4 ×
|
|
||||||
4-bit high nibbles (MSB first), bytes [2:6] hold 4 × int8 low bytes.
|
|
||||||
Each delta = `sign_extend_12((high_nibble << 8) | low_byte)`. Block
|
|
||||||
length = `NN × 1.5 + 2` bytes. ✅ confirmed against all 14 `30 NN`
|
|
||||||
blocks in the fixture bundle. 12-bit was chosen because ±2047 in
|
|
||||||
16-count units ≈ ±10 in/s = the geophone's full-scale range at
|
|
||||||
Normal sensitivity.
|
|
||||||
- **Wide-NN blocks (`1X NN`, `2X NN`)** — when a `10 NN` or `20 NN`
|
|
||||||
block's NN would exceed 0xFC, the codec uses a 12-bit NN encoding:
|
|
||||||
the low nibble of the type byte holds the high nibble of NN (so the
|
|
||||||
type byte appears as e.g. `0x11` instead of `0x10`). Effective
|
|
||||||
NN = `((type_byte & 0x0F) << 8) | nn_byte`. Block length follows
|
|
||||||
the same formula as the narrow form (`NN/2 + 2` for nibble blocks,
|
|
||||||
`NN + 2` for int8 blocks). Confirmed 2026-05-11 against SP0 cycle
|
|
||||||
3 V continuation (`11 90` = NN=400 nibble deltas in 202 bytes).
|
|
||||||
|
|
||||||
### ⚠ SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN
|
|
||||||
|
|
||||||
Everything in this section below about `40 NN` segment headers, tagless
|
|
||||||
headers, variable header widths and channel rotation describes a model that
|
|
||||||
is **wrong**. The body is a chain of self-delimiting per-channel records:
|
|
||||||
|
|
||||||
off+2 len uint16 BE -> next_record = off + 2 + len
|
|
||||||
off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL / 0x06 = END
|
|
||||||
off+8 mode 02 00 = deltas+anchors (14B hdr)
|
|
||||||
01 00 = ABSOLUTE values (10B hdr)
|
|
||||||
00 03 = raw 12-bit absolute, NO TAGS (10B hdr)
|
|
||||||
|
|
||||||
`40 NN` is an ordinary int16 BE data block (`2*NN + 2`), never a header. The
|
|
||||||
"variable prefix" of 0/2/4/6/8 bytes was walker drift, exactly
|
|
||||||
`4 - (old_stop - true_record_start)`.
|
|
||||||
|
|
||||||
All four channels now come out equal length in **1388/1388** files (was
|
|
||||||
156/1388); ASCII sample-count exact **75/75**, fully exact **73/75**; device
|
|
||||||
PPV on a live decode **1306/1306** waveform, **4458/4459** histogram.
|
|
||||||
|
|
||||||
The old model survives as `decode_waveform_legacy` because
|
|
||||||
`micromate/idf_file.py` pins it for Thor IDFW body-offset search.
|
|
||||||
|
|
||||||
### Framing cases added 2026-05-11 → 2026-08-25
|
|
||||||
|
|
||||||
Four more block-framing cases, each of which had been causing **silent
|
|
||||||
channel truncation** — `walk_body` ends its loop on an unrecognised tag
|
|
||||||
and `decode_waveform_v2` returns whatever channels it got, so an
|
|
||||||
unhandled tag surfaces as short channels with no error raised. Found by
|
|
||||||
diffing 75 production events against their preserved Blastware ASCII
|
|
||||||
exports (`<store>/<serial>/<file>_ASCII.TXT`).
|
|
||||||
|
|
||||||
- **Wide-NN RLE `0X NN`** — the 12-bit NN encoding documented above for
|
|
||||||
`1X`/`2X` **also applies to the `00 NN` RLE tag**. A narrow run maxes
|
|
||||||
out at NN=0xFC, so a quiet stretch longer than 252 samples must use
|
|
||||||
the wide form (e.g. `01 0c` = 268 repeats).
|
|
||||||
- **`30 NN` is not capped at NN=0x10** — data-section blocks reach at
|
|
||||||
least NN=0x18. The `NN × 1.5 + 2` length formula was already right;
|
|
||||||
only the guard was wrong.
|
|
||||||
- **`40 NN` segment headers are variable width** — NN is the *count of
|
|
||||||
int16 BE continuation deltas for the PREVIOUS channel*, so the header
|
|
||||||
is `2*NN + 16` bytes and every field after the deltas shifts by
|
|
||||||
`2*NN`. `40 01` (18 B) and `40 03` (22 B) both occur alongside the
|
|
||||||
common `40 02` (20 B).
|
|
||||||
- **Tagless segment headers** — a header can appear with **no `40 NN`
|
|
||||||
tag at all**: just the 14-byte tail
|
|
||||||
`[field2:2][len:2][channel_id:4][marker:2][anchors:4]`. This is the
|
|
||||||
NN=0 case (previous channel needed no continuation deltas).
|
|
||||||
|
|
||||||
**The header "counter" is really a channel id.** The 4-byte field long
|
|
||||||
documented as a "monotonic uint32 LE counter" is
|
|
||||||
`[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert
|
|
||||||
`0x48`=Long `0x49`=MicL — verified on **1697/1697** segment headers
|
|
||||||
across the corpus, zero disagreements. `decode_waveform_v2` now takes
|
|
||||||
the channel from this field rather than from rotation position; a single
|
|
||||||
missed or extra header (exactly what tagless headers caused) desyncs
|
|
||||||
rotation and corrupts every channel after it.
|
|
||||||
|
|
||||||
Corpus result, end to end through the production path:
|
|
||||||
**exact 37 → 72, truncated 23 → 3, full-length value errors 15 → 0.**
|
|
||||||
|
|
||||||
### Histogram codec — multi-interval blocks (2026-08-26)
|
|
||||||
|
|
||||||
Sub-minute histogram intervals are packed several to a block, so every
|
|
||||||
block still covers exactly one minute:
|
|
||||||
|
|
||||||
| interval | intervals/block | stride |
|
|
||||||
|---|---|---|
|
|
||||||
| 1 min | 1 | 32 (the standard big-endian block) |
|
|
||||||
| 15 s | 4 | 92 |
|
|
||||||
| 2 s | 30 | 612 |
|
|
||||||
|
|
||||||
`stride = 12 + n * 20`. Block = `[00][segment][ctr uint16 LE][0a][00]`,
|
|
||||||
then n x 20-byte records of 8 x uint16 **LITTLE**-endian values
|
|
||||||
(`T_peak, T_halfp, V_peak, V_halfp, L_peak, L_halfp, M_peak, M_halfp`)
|
|
||||||
plus a 2-word tail whose first word is `0000` on every real interval,
|
|
||||||
then a 6-byte block trailer.
|
|
||||||
|
|
||||||
⚠ The standard 32-byte block is BIG-endian; this variant is LITTLE-endian.
|
|
||||||
|
|
||||||
Recovers **415 files** (216 on BE18193, 199 on BE9440) that decoded to
|
|
||||||
nothing. Ground truth `BE9440/K440L3AQ.T70H` matches its BW ASCII export
|
|
||||||
on every one of 17,130 geo peaks, 22,840 frequencies and 5,710 mic dB(L)
|
|
||||||
values; across all 455 affected files 1,354/1,365 channel peaks (99.2%)
|
|
||||||
match the device-reported PPV.
|
|
||||||
|
|
||||||
### Histogram codec — corrected 2026-08-25
|
|
||||||
|
|
||||||
The histogram block is **uniformly big-endian**, and the stream's final
|
|
||||||
block has its own tail signature. Two long-standing errors:
|
|
||||||
|
|
||||||
- **Peaks and half-periods are `uint16` big-endian**, not `uint8` +
|
|
||||||
an "annotation" byte. `T_peak` is `[5:7]`, `T_halfperiod` `[7:9]`,
|
|
||||||
`V_peak` `[9:11]`, and so on; only `block_ctr` at `[2:4]` is LE.
|
|
||||||
The old model silently **clipped any peak above 1.275 in/s** — the
|
|
||||||
final interval of `BE18193/T193LQ9K.OE0H` reads 8.270 in/s in BW's
|
|
||||||
export and decoded as 0.590. The "annotation" byte was the
|
|
||||||
half-period's high byte, which is why it was non-zero exactly on the
|
|
||||||
sub-Hz intervals BW renders as `<1.0`.
|
|
||||||
- **The marker is `block[4]` alone.** Testing `[4:6]` as a uint16 LE
|
|
||||||
marker forced `block[5] == 0`, which is what capped the peak at one
|
|
||||||
byte in the first place.
|
|
||||||
- **The last block of the stream carries tail `9c 06 00 42`** instead of
|
|
||||||
`1e 0a 00 00`, with arbitrary bytes at `[21:23]`. Rejecting it
|
|
||||||
dropped the final interval of nearly every histogram — frequently the
|
|
||||||
one holding the event peak, so the file's PPV read low.
|
|
||||||
|
|
||||||
Verified against 1211 production histograms paired with their BW ASCII
|
|
||||||
exports: **1211/1211 decode exactly** (interval count plus every
|
|
||||||
per-interval peak), and 842,442 per-interval frequency comparisons match
|
|
||||||
with zero mismatches. Before: 1 of 1196.
|
|
||||||
|
|
||||||
### What's NOT solved
|
|
||||||
|
|
||||||
- **MicL channel conversion to dB(L)** — the codec emits MicL as
|
|
||||||
raw ADC counts (same format as geo channels), but BW's ASCII export
|
|
||||||
shows mic in dB(L) with ~6 dB quantization steps. Need to map
|
|
||||||
ADC counts → dB(L) for direct comparison; likely
|
|
||||||
`dB = 20*log10(|counts|) + offset` or similar.
|
|
||||||
- **Variable-prefix segment descriptors** — 3 of the 75 ground-truth
|
|
||||||
production events still truncate. The walk reaches a segment header
|
|
||||||
whose channel-id field is preceded by a *variable-width* prefix (2, 4
|
|
||||||
or 6 bytes observed; the standard tagless form always has 4), carrying
|
|
||||||
an `01 00` marker instead of `02 00`. The marker is **not** simply an
|
|
||||||
anchor count — `01 00` records appear with both 2- and 4-byte anchor
|
|
||||||
fields in the same file. Examples: `BE12599/N599LPNB.JF0W` @1155,
|
|
||||||
`BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
|
|
||||||
(The series-3 histogram codec was fixed 2026-08-25 — see below.)
|
|
||||||
|
|
||||||
- **Micromate (UM-series) IDF decode is ~1000× low** — e.g.
|
|
||||||
`UM11402_20260406130113.IDFW` gives a Tran peak of 0.0009 in/s against
|
|
||||||
a device-reported 1.1168. The Thor IDF path decodes sanely, so this
|
|
||||||
is UM-specific.
|
|
||||||
- **Thor IDF per-count LSB** — after the 32000 geo full-scale
|
|
||||||
correction, series-4 Thor peaks sit at a median 0.983 of the
|
|
||||||
device-reported peak (was 0.960 under 32768). Closer but not exact;
|
|
||||||
Thor likely uses its own per-count LSB rather than the BW
|
|
||||||
16-count/0.005 in/s convention.
|
|
||||||
|
|
||||||
### Decoded sample counts (across the fixture bundle)
|
|
||||||
|
|
||||||
| Event | Tran | Vert | Long | Total |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| event-a | 3328 | 3328 | 3328 | **9984** ← full event |
|
|
||||||
| event-b | 2304 | 2304 | 2304 | **6912** ← full event |
|
|
||||||
| event-c | 1280 | 1280 | 1280 | 3840 ← full event |
|
|
||||||
| event-d | 1280 | 1280 | 1280 | 3840 ← full event |
|
|
||||||
| JQ0 | 3328 | 3328 | 3328 | **9984** ← full event |
|
|
||||||
| V70 | 3328 | 3328 | 3328 | **9984** ← full event |
|
|
||||||
| SP0 | 3328 | 3328 | 3328 | **9984** ← full event |
|
|
||||||
| SS0 | 3078 | 3072 | 3072 | 9222 (1–7 tail samples missing) |
|
|
||||||
| SV0 | 3078 | 3072 | 3072 | 9222 (1–7 tail samples missing) |
|
|
||||||
|
|
||||||
**Total: 72,972 ADC samples verified byte-exact, zero errors.**
|
|
||||||
|
|
||||||
7 of 9 fixture events decode end-to-end across all three geo channels.
|
|
||||||
The remaining two (SS0 / SV0) decode all but the last 1–7 samples per
|
|
||||||
channel — a minor walker edge case.
|
|
||||||
|
|
||||||
### Production-code status (updated 2026-05-11 late)
|
|
||||||
|
|
||||||
`client.py:_decode_a5_waveform` now uses the verified codec via
|
|
||||||
`waveform_codec.decode_a5_frames()` — which calls
|
|
||||||
`blastware_file.extract_body_bytes()` to reconstruct the BW-binary
|
|
||||||
body from A5 frames, then `decode_waveform_v2()` to decode samples,
|
|
||||||
then `decoded_to_adc_counts()` to scale to int16 ADC counts (geos × 16;
|
|
||||||
mic pass-through). The `.h5` sidecars SFM produces now contain
|
|
||||||
correct samples for any event without walker edge cases.
|
|
||||||
|
|
||||||
**Geo full scale is 32000 ADC counts, NOT 32768** (fixed 2026-08-25).
|
|
||||||
One decoder unit = 16 ADC counts = exactly 0.005 in/s, so
|
|
||||||
`10.000 in/s / (0.005/16)` = 32000. Consumers must use
|
|
||||||
`sfm.event_hdf5._GEO_INT16_FS` / `event_file_io._GEO_INT16_FS` (both
|
|
||||||
32000). Dividing by 32768 reads every geophone sample 2.34% low —
|
|
||||||
that was a live bug in both modules until 2026-08-25. Mic is
|
|
||||||
unaffected (it back-solves its scale from the device-reported peak).
|
|
||||||
|
|
||||||
The original int16 LE decoder is preserved as
|
|
||||||
`_decode_a5_waveform_LEGACY` for reference but is not called.
|
|
||||||
|
|
||||||
MicL → dB(L) conversion utility:
|
|
||||||
`waveform_codec.mic_count_to_db(count)` — `count=±1 → ±81.94 dB`;
|
|
||||||
`count=813 → 140.14 dB` (matches BW display).
|
|
||||||
|
|
||||||
### Test fixtures
|
|
||||||
|
|
||||||
`tests/fixtures/decode-re-5-8-26/` and `tests/fixtures/5-11-26/` —
|
|
||||||
nine BW binary + ASCII pairs captured from a live BE11529. The
|
|
||||||
5-11-26 high-amplitude bundle (PPV 6–7 in/s) is what cracked the Tran
|
|
||||||
codec; the V70 (mic-heavy) + JQ0 (Vert-heavy) pair cracked the `00 NN`
|
|
||||||
RLE rule.
|
|
||||||
|
|
||||||
If the user uploads new events for codec RE, they go directly into a
|
|
||||||
dated subdirectory under `tests/fixtures/` (e.g. `tests/fixtures/5-18-26/`).
|
|
||||||
There used to be a separate `decode-re/` upload mirror but it was
|
|
||||||
removed once the fixtures directory became the canonical location.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Protocol fundamentals
|
## Protocol fundamentals
|
||||||
|
|
||||||
### DLE framing
|
### DLE framing
|
||||||
@@ -511,203 +115,32 @@ S3→BW (response):
|
|||||||
section contribute only `XX` to the running sum; lone bytes contribute normally. This
|
section contribute only `XX` to the running sum; lone bytes contribute normally. This
|
||||||
differs from the standard SUM8-of-destuffed-payload that all other commands use.
|
differs from the standard SUM8-of-destuffed-payload that all other commands use.
|
||||||
|
|
||||||
3. **Params region uses partial DLE stuffing (CONFIRMED 2026-05-05).** The device's
|
Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26
|
||||||
de-stuffing rule for bytes inside the params region is:
|
BW TX capture. All 10 frames verified.
|
||||||
|
|
||||||
- `10 10` → de-stuffs to `10`
|
### SUB 5A — chunk counter formula (FINAL CORRECTION 2026-04-26)
|
||||||
- `10 02 / 03 / 04` → kept literal (these are inner-frame markers)
|
|
||||||
- `10 X` for other X → de-stuffs to just `X` (drops the leading `0x10`)
|
|
||||||
|
|
||||||
Therefore any `0x10` byte in the *logical* params that is followed by a byte NOT in
|
**Chunk counter = `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` for ALL chunks.**
|
||||||
`{0x02, 0x03, 0x04, 0x10}` MUST be doubled on the wire (`10 X` → `10 10 X`) so the
|
|
||||||
device's de-stuffer reproduces the original `10 X` pair. This applies most commonly
|
|
||||||
to counters with `0x10` in the high byte (e.g. counter=`0x1000` produces logical
|
|
||||||
params bytes `... 10 00 ...`, which BW encodes on the wire as `... 10 10 00 ...`).
|
|
||||||
Without this stuffing the device interprets counter=`0x1000` as `0x0000` and returns
|
|
||||||
the probe response (which contains a copy of the file header + STRT record). That
|
|
||||||
STRT block then gets embedded in the assembled file body at offset `0x1016`, and
|
|
||||||
Blastware refuses to open the file — see the v0.14.3 entry in `CHANGELOG.md`.
|
|
||||||
|
|
||||||
`0x10` bytes in `offset_hi` (body[5]) are still written RAW — only the params region
|
where `key4[2:4] = (key4[2] << 8) | key4[3]` is the event's circular-buffer base offset.
|
||||||
has this stuffing requirement. The metadata-page params for counter `0x1002` /
|
|
||||||
`0x1004` survive without stuffing because `10 02` and `10 04` fall in the "kept
|
|
||||||
literal" carve-out.
|
|
||||||
|
|
||||||
Both differences (1) and (2) confirmed by reproducing Blastware's exact wire bytes from
|
The `max(..., 0x0400)` guard is critical for events at the start of the circular buffer
|
||||||
the 1-2-26 BW TX capture (10 frames). Difference (3) confirmed against the 5-1-26
|
(key4[2:4] == 0x0000, e.g. key `01110000`). Without it, chunk 1 gets counter=0x0000, which
|
||||||
"bwcap3sec" capture (17 frames, all match byte-for-byte after fix).
|
is the same address as the probe frame — the device re-returns the STRT record data instead
|
||||||
|
of waveform payload. With the guard, chunk 1 gets counter=0x0400, which is confirmed correct
|
||||||
|
from the empirical live-device test 2026-04-06 (`counter=0x0400 → responds immediately and
|
||||||
|
streams all frames correctly`).
|
||||||
|
|
||||||
### SUB 5A — chunk counter formula (REWRITTEN 2026-05-01 — see 5-1-26 captures)
|
The 4-3-26 capture confirms the pattern for a second event (key `0111245a`, key4[2:4]=0x245a):
|
||||||
|
chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400).
|
||||||
> ⚠️ **Everything that came before this rewrite was WRONG in important ways.** The previous
|
`max(0x245a, 0x0400) = 0x245a` → formula works correctly for non-zero base offset too.
|
||||||
> formula `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` happened to *work* for events
|
|
||||||
> at start_key=0 because the device responds to whatever counter you ask for — but it caused
|
|
||||||
> a 5× over-read past the actual event, picking up post-event circular-buffer garbage that
|
|
||||||
> corrupts the reconstructed file for any event > ~1 sec of waveform. The captures in
|
|
||||||
> `bridges/captures/4-27-26/` and `5-1-26/comcheck/` show BW reads only ~12-16 chunks for
|
|
||||||
> the same events SFM was reading 37+ chunks for. See "TERM frame" and "STRT end_offset"
|
|
||||||
> sections below for the actual mechanism.
|
|
||||||
|
|
||||||
**Chunk addressing is just absolute device-buffer addresses.**
|
|
||||||
|
|
||||||
`params[0]=0x00`, `params[1:5]` is a 4-byte absolute device flash-buffer address (= the
|
|
||||||
"key" of that location), `params[5:11]` are zeros. The device returns 0x0200 (= 512) bytes
|
|
||||||
starting at that address. Increments between consecutive chunks are **0x0200 (NOT 0x0400)**
|
|
||||||
— this matches the chunk payload size. The previous "0x0400 step" worked by accident: BW
|
|
||||||
asks for half-size chunks; SFM was asking for double-size chunks, both with the same-named
|
|
||||||
"counter" field, but the value is just an address pointer the device honors as-is.
|
|
||||||
|
|
||||||
**The chunk pattern depends on whether the event sits at start_key=0 or not.**
|
|
||||||
|
|
||||||
#### Event 1 case — start_key[2:4] == 0x0000 (first event after erase / wrap)
|
|
||||||
|
|
||||||
```
|
|
||||||
1. Probe at counter=0x0000 (params[1:5] = full key, returns STRT record)
|
|
||||||
2. Read 2 fixed metadata pages: counter=0x1002, counter=0x1004
|
|
||||||
(these are GLOBAL session metadata — read ONCE per
|
|
||||||
Blastware session, not per event; contain the
|
|
||||||
Project/Client/User Name/Seis Loc strings)
|
|
||||||
3. Sample chunks: counter=0x0600, 0x0800, …, by 0x0200 increment,
|
|
||||||
up to but not including end_offset (rounded down to
|
|
||||||
0x0200 boundary)
|
|
||||||
4. TERM frame (see TERM formula below)
|
|
||||||
```
|
|
||||||
|
|
||||||
The reason `0x0046..0x0600` is skipped for event 1 is unknown — likely some pre-event
|
|
||||||
firmware reserved area for the first slot in a freshly-erased buffer. Harmless to skip.
|
|
||||||
|
|
||||||
#### Event 2+ case — start_key[2:4] != 0x0000 (continuation events)
|
|
||||||
|
|
||||||
```
|
|
||||||
1. First chunk at counter = start_key[2:4] (this IS the probe — response
|
|
||||||
contains STRT at byte 17)
|
|
||||||
2. Sample chunks: counter += 0x0200 each, up to but
|
|
||||||
not including end_offset
|
|
||||||
3. TERM frame
|
|
||||||
```
|
|
||||||
|
|
||||||
**`start_key` here is the off=0x46 WAVEHDR record key returned by 1F** (e.g. `01112238`),
|
|
||||||
NOT the off=0x2C boundary key that immediately precedes it. An earlier draft of this
|
|
||||||
doc described event-N as "probe at start + 0x46" — that formula came from naming the
|
|
||||||
boundary key as `start_key`. In the iteration walk, `cur_key` passed to
|
|
||||||
`read_bulk_waveform_stream` is always the off=0x46 key (the partial-record skip path in
|
|
||||||
`get_events` re-runs 1F to advance past boundary records before invoking 5A), so the
|
|
||||||
probe counter is just `cur_key[2:4]` with no extra offset. **Adding +0x46 caused the
|
|
||||||
probe to overshoot, miss the STRT record at byte 17 of the response, fall back to the
|
|
||||||
`max_chunks=128` cap, and walk ~110 chunks of post-event garbage** — observed in
|
|
||||||
SFM 5-4-26 capture before the fix.
|
|
||||||
|
|
||||||
Confirmed across:
|
|
||||||
- 5-1-26 "copy 2nd address" BW capture: probe counter=0x2238, key=01112238, STRT@17 end=0x417E.
|
|
||||||
- 5-4-26 BW 2-sec event capture: probe counter=0x2238, key=01112238, TERM offset_word=0x0146 → end=0x417E.
|
|
||||||
|
|
||||||
No metadata pages — those have already been read during event 1 in the same Blastware
|
|
||||||
session, and BW caches them. Note that the metadata-page reads happen ONCE per
|
|
||||||
Blastware-session-on-the-device, not once per event, so an SFM session that downloads
|
|
||||||
several events should read 0x1002/0x1004 only once at the start.
|
|
||||||
|
|
||||||
#### History (do not re-derive)
|
|
||||||
|
|
||||||
|
**History:**
|
||||||
- Original: `_CHUNK1_COUNTER = 0x1004` hardcoded (Blastware capture artifact — WRONG).
|
- Original: `_CHUNK1_COUNTER = 0x1004` hardcoded (Blastware capture artifact — WRONG).
|
||||||
- 2026-04-06: `chunk_num * 0x0400` (worked for key 01110000 only).
|
- 2026-04-06: Corrected to `chunk_num * 0x0400` (worked for key 01110000 only).
|
||||||
- 2026-04-24: `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets, broke key 01110000).
|
- 2026-04-24: Corrected to `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets,
|
||||||
- 2026-04-26: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` (broken — over-read past event end).
|
but accidentally broke key 01110000 — counter=0x0000 sends probe address again).
|
||||||
- 2026-05-01: Increments are 0x0200 not 0x0400; absolute addresses inside event range; bounded
|
- 2026-04-26: Final formula: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400`.
|
||||||
by STRT end_key, not by `max_chunks` cap or device-side timeout.
|
|
||||||
- 2026-05-04: Removed spurious `+0x0046` from event-N probe counter. `cur_key` from 1F
|
|
||||||
is already the off=0x46 WAVEHDR key, so adding +0x46 would have placed the probe one
|
|
||||||
WAVEHDR past the actual event start. This caused probe responses to lack a STRT
|
|
||||||
record (no `end_offset` parsed → `0xFFFF` fallback → `max_chunks=128` cap), walking
|
|
||||||
~110 chunks of post-event circular-buffer garbage. Fixed in protocol.py
|
|
||||||
`read_bulk_waveform_stream`.
|
|
||||||
|
|
||||||
### SUB 5A — STRT record encodes end_offset (NEW 2026-05-01)
|
|
||||||
|
|
||||||
The first A5 response (probe response, or the first chunk for event 2+) contains a STRT
|
|
||||||
record at byte offset 17 of the `data` field. Layout:
|
|
||||||
|
|
||||||
```
|
|
||||||
data[17:21] "STRT" magic
|
|
||||||
data[21:23] ff fe sentinel
|
|
||||||
data[23:27] end_key ← 4-byte key of where this event ENDS
|
|
||||||
data[27:31] start_key ← 4-byte key of where this event STARTS
|
|
||||||
data[31:33] uint16 BE ?? sample-count or total bytes (varies; not yet decoded)
|
|
||||||
data[33:35] uint16 BE ??
|
|
||||||
data[35] 0x46 record type (waveform full record)
|
|
||||||
…
|
|
||||||
```
|
|
||||||
|
|
||||||
`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**.
|
|
||||||
SFM must extract this from the first A5 response and use it to bound the chunk loop and
|
|
||||||
encode the TERM frame. The device will happily respond to chunk requests past `end_offset`
|
|
||||||
(returning post-event circular-buffer contents) — that's the over-read bug.
|
|
||||||
|
|
||||||
Verified across 3 events:
|
|
||||||
|
|
||||||
| Capture | start_key | end_key | end_offset | event size |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B |
|
|
||||||
| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B |
|
|
||||||
| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | `0x417E` (event 2 span 0x1F8C = 8,076 B) |
|
|
||||||
|
|
||||||
### SUB 5A — TERM frame formula (FINALIZED 2026-05-01)
|
|
||||||
|
|
||||||
The TERM frame fetches the partial last chunk *and* the file footer. It is **not** a simple
|
|
||||||
"goodbye" frame — its response payload contains the bytes between the last full 0x0200-aligned
|
|
||||||
chunk and `end_offset`, and is required for reconstructing the Blastware file format.
|
|
||||||
|
|
||||||
```
|
|
||||||
last_chunk_counter = address of last full 0x0200-byte chunk read
|
|
||||||
next_boundary = last_chunk_counter + 0x0200
|
|
||||||
TERM offset_word = end_offset - next_boundary
|
|
||||||
TERM params[0] = key[0] (= 0x01 on every observed device)
|
|
||||||
TERM params[1] = key[1] (= 0x11)
|
|
||||||
TERM params[2] = (next_boundary >> 8) & 0xFF
|
|
||||||
TERM params[3] = next_boundary & 0xFF
|
|
||||||
TERM params[4:10] = zeros
|
|
||||||
build_5a_frame(offset_word, params) (10-byte params, NOT 11)
|
|
||||||
```
|
|
||||||
|
|
||||||
The device reconstructs `requested_address = (params[2] << 8) | offset_word = end_offset`
|
|
||||||
and replies with `(end_offset - next_boundary)` bytes from `next_boundary` — the residual
|
|
||||||
between the last 0x0200 boundary and the actual event end. Append the TERM response data
|
|
||||||
to the chunk stream like any other A5 frame; it carries the final waveform tail + footer.
|
|
||||||
|
|
||||||
Verified across 3 events:
|
|
||||||
|
|
||||||
| end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ |
|
|
||||||
| `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ |
|
|
||||||
| `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ |
|
|
||||||
|
|
||||||
The previous code's hard-coded `offset_word = 0x005A` and `term_counter = last + 0x0400`
|
|
||||||
are wrong; the device's response under that path is a tiny 101-byte device-side terminator
|
|
||||||
(arrived only after we walked the entire post-event buffer), not the proper file footer.
|
|
||||||
|
|
||||||
### SUB 5A — fixed metadata pages 0x1002 and 0x1004 (NEW 2026-05-01)
|
|
||||||
|
|
||||||
Two chunk addresses are GLOBAL device/session metadata, not event-specific:
|
|
||||||
|
|
||||||
- `counter=0x1002` — first metadata page
|
|
||||||
- `counter=0x1004` — second metadata page
|
|
||||||
|
|
||||||
These are at fixed absolute addresses in the device's flash buffer. They contain the
|
|
||||||
session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII
|
|
||||||
strings). Under the v0.14.0+ walk these strings are read directly from the metadata
|
|
||||||
pages, not from the sample-chunk stream.
|
|
||||||
|
|
||||||
BW reads them ONCE per Blastware session (during event 1's download) and caches them.
|
|
||||||
For SFM, that means:
|
|
||||||
- Once per call-home / once per `MiniMateClient.connect()` is enough.
|
|
||||||
- Subsequent events in the same session don't need to re-fetch them.
|
|
||||||
- Their content does not change when iterating events; only when the user opens
|
|
||||||
Compliance Setup → Apply on the device or sends a SUB 71 compliance write.
|
|
||||||
|
|
||||||
The full byte-for-byte layout of the metadata pages has not been mapped — `_decode_a5_metadata_into`
|
|
||||||
locates the ASCII strings via label scans (`Project:`, `Client:`, `User Name:`, `Seis Loc:`,
|
|
||||||
`Extended Notes`) which works correctly across observed captures. Future work could
|
|
||||||
dump the structural layout if more session-global fields need to be extracted.
|
|
||||||
|
|
||||||
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
|
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
|
||||||
|
|
||||||
@@ -715,11 +148,10 @@ dump the structural layout if more session-global fields need to be extracted.
|
|||||||
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
|
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
|
||||||
Do not swap them.
|
Do not swap them.
|
||||||
|
|
||||||
### SUB 5A — event-time metadata source (FINALIZED 2026-05-05)
|
### SUB 5A — event-time metadata lives in A5 frame 7
|
||||||
|
|
||||||
The metadata strings come from the two fixed metadata pages at counter `0x1002` and
|
The bulk stream sends 9+ A5 response frames. Frame 7 (0-indexed) contains the compliance
|
||||||
`0x1004` (see "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above). These pages
|
setup as it existed when the event was recorded:
|
||||||
are GLOBAL session metadata — read once per Blastware/SFM session, not per event.
|
|
||||||
|
|
||||||
```
|
```
|
||||||
"Project:" → project description
|
"Project:" → project description
|
||||||
@@ -729,71 +161,44 @@ are GLOBAL session metadata — read once per Blastware/SFM session, not per eve
|
|||||||
"Extended Notes"→ notes
|
"Extended Notes"→ notes
|
||||||
```
|
```
|
||||||
|
|
||||||
**IMPORTANT — these strings are session-start config, NOT per-event:**
|
**IMPORTANT — 5A "Project:" is session-start config, NOT per-event (confirmed 2026-04-05):**
|
||||||
Project / Client / User Name / Seis Loc reflect the compliance setup from when the
|
The "Project:" string in the A5 frame 7 payload reflects the compliance setup from when
|
||||||
*monitoring session first started*, not the individual event's per-event metadata. The
|
the *monitoring session first started*, not the individual event's project name. The per-
|
||||||
authoritative per-event project name is stored in the 210-byte 0C waveform record.
|
event project name is correctly stored in the 210-byte 0C waveform record and must be
|
||||||
`_decode_a5_metadata_into` therefore only sets `project` from the 5A metadata pages
|
used as the authoritative source. `_decode_a5_metadata_into` therefore only sets
|
||||||
when 0C didn't already supply one.
|
`project` from 5A when 0C didn't already supply one.
|
||||||
|
|
||||||
"Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C
|
"Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C
|
||||||
record — the metadata pages are the sole source for those fields and they are set
|
record — 5A remains the sole source for those fields and they are set unconditionally.
|
||||||
unconditionally.
|
|
||||||
|
|
||||||
#### Deprecated knobs (do not re-introduce)
|
`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears,
|
||||||
|
then sends the termination frame.
|
||||||
|
|
||||||
The `read_bulk_waveform_stream()` function still accepts these legacy kwargs for
|
### SUB 5A — end-of-stream signal (confirmed 2026-04-06)
|
||||||
backward compatibility, but they are **no-ops** under the v0.14.0+ walk:
|
|
||||||
|
|
||||||
- `stop_after_metadata=True` — used to scan the chunk stream for `b"Project:"` and stop
|
After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to
|
||||||
one chunk later as a workaround for the missing end_offset bound. Obsolete: the loop
|
the next chunk request, then goes silent. This is the natural end-of-stream indicator — NOT
|
||||||
is now deterministically bounded by `end_offset` parsed from the STRT record at
|
a complete A5 frame. `S3FrameParser.bytes_fed` will be 1; no frame is assembled.
|
||||||
data[17] of the probe response, with the partial tail fetched by the TERM frame.
|
|
||||||
- `extra_chunks_after_metadata` — same era, same reason. No-op.
|
|
||||||
|
|
||||||
If you find code or docs referencing "A5 frame 7" as the source of metadata strings,
|
Handling: on `TimeoutError`, if `bytes_fed > 0` AND frames were already collected, treat as
|
||||||
that's an old-walk artifact (the broken `0x0400`-step formula occasionally caught the
|
graceful end-of-stream, break the loop, and proceed to the termination frame. If `bytes_fed
|
||||||
0x1002 metadata page at sample-chunk fi=7). Update to reference the dedicated metadata
|
== 0` with no prior frames, it is a genuine transport failure — re-raise.
|
||||||
pages instead.
|
|
||||||
|
|
||||||
### SUB 5A — end-of-stream (FINALIZED 2026-05-01)
|
**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
|
||||||
|
Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call
|
||||||
|
in the chunk loop passes `timeout=10.0` explicitly.
|
||||||
|
|
||||||
Under the v0.14.0+ STRT-bounded walk the stream ends cleanly:
|
**Typical chunk count (BE11529, 1024 sps):** A 9,306-sample event produces 35 chunks before
|
||||||
|
end-of-stream. Chunks with uniform 1,036-byte data are all-zero ADC samples (post-event
|
||||||
```
|
silence). Only the initial variable-size chunks contain actual signal.
|
||||||
… last full chunk at counter < end_offset
|
|
||||||
TERM request (offset_word = end_offset - next_boundary,
|
|
||||||
params address (next_boundary))
|
|
||||||
TERM response (page_key = 0x0000 or 0x0001, data = the residual
|
|
||||||
end_offset - next_boundary bytes including the file footer)
|
|
||||||
```
|
|
||||||
|
|
||||||
No timeout-based detection, no "1-byte teaser," no `max_chunks` cap. The chunk loop
|
|
||||||
exits when `counter + 0x0200 > end_offset`; the TERM frame fetches the tail.
|
|
||||||
|
|
||||||
**Chunk recv timeout is 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
|
|
||||||
Using 120 s would cause a ~2-minute stall on any unexpected timeout. The `_recv_one`
|
|
||||||
call in the chunk loop passes `timeout=10.0` explicitly.
|
|
||||||
|
|
||||||
**Typical chunk count under the v0.14.0+ walk (BE11529, 1024 sps over TCP/cellular):**
|
|
||||||
|
|
||||||
| Event duration | Sample chunks | Metadata pages | TERM | Total A5 frames |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| 2-sec (event 1) | ~12 | 2 | 1 | ~15 |
|
|
||||||
| 3-sec (event 1) | 13 | 2 | 1 | 16 |
|
|
||||||
| 2-sec (continuation) | 15 | 0 | 1 | 16 |
|
|
||||||
| 3-sec (continuation) | ~14 | 0 | 1 | ~15 |
|
|
||||||
|
|
||||||
For comparison, the deprecated `0x0400`-step walk produced ~37 chunks for a 2-sec
|
|
||||||
event with chunks 17-37 containing post-event circular-buffer garbage. Do not
|
|
||||||
re-introduce that walk under any circumstances.
|
|
||||||
|
|
||||||
### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06)
|
### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06)
|
||||||
|
|
||||||
`_decode_a5_waveform()` previously had `elif fi == 9: continue` — a leftover from the
|
`_decode_a5_waveform()` previously had `elif fi == 9: continue` — a leftover from the
|
||||||
9-frame original blast capture where frame 9 was assumed to be a terminator. Removed.
|
9-frame original blast capture where frame 9 was assumed to be a terminator. For current
|
||||||
TERM detection in the file builder uses `frame.page_key != 0x0010` (sample marker),
|
35-frame streams, fi==9 is live waveform data (~133 sample-sets were being dropped).
|
||||||
not frame index — see `blastware_file.py`.
|
Removed. Terminator detection is via `page_key == 0x0000` in `read_bulk_waveform_stream`,
|
||||||
|
not frame index.
|
||||||
|
|
||||||
### SUB 1E / 1F — event iteration null sentinel and token position (FIXED, do not re-introduce)
|
### SUB 1E / 1F — event iteration null sentinel and token position (FIXED, do not re-introduce)
|
||||||
|
|
||||||
@@ -898,55 +303,6 @@ sends token=0xFE and is NOT used by any caller.
|
|||||||
`advance_event()` returns `(key4, event_data8)`.
|
`advance_event()` returns `(key4, event_data8)`.
|
||||||
Callers (`count_events`, `get_events`) loop while `data8[4:8] != b"\x00\x00\x00\x00"`.
|
Callers (`count_events`, `get_events`) loop while `data8[4:8] != b"\x00\x00\x00\x00"`.
|
||||||
|
|
||||||
### SUB 0A — WAVEHDR response length distinguishes events from boundaries (NEW 2026-05-01)
|
|
||||||
|
|
||||||
When iterating events with the "Download All" pattern (1E → 0A → 1F → 0A → 1F → …), the
|
|
||||||
DATA_LENGTH at `data_rsp.data[5]` (= the byte BW echoes back as the offset for the data
|
|
||||||
fetch step) takes one of two values:
|
|
||||||
|
|
||||||
| WAVEHDR offset | Meaning |
|
|
||||||
|---|---|
|
|
||||||
| `0x46` (= 70) | Real event start key — there is event data at this address |
|
|
||||||
| `0x2C` (= 44) | Boundary marker between events — this key is the END of the previous event AND the START key for the empty space after it (or is the next event's pre-header) |
|
|
||||||
|
|
||||||
Confirmed from the 5-1-26 "Download All" capture:
|
|
||||||
|
|
||||||
```
|
|
||||||
0A(key=01110000) → off=0x46 ← event 1 real start
|
|
||||||
1F → key=011121F2
|
|
||||||
0A(key=011121F2) → off=0x2C ← event 1 END / event 2 boundary
|
|
||||||
1F → key=01112238
|
|
||||||
0A(key=01112238) → off=0x46 ← event 2 real start (= boundary + 0x46)
|
|
||||||
1F → key=0111417E
|
|
||||||
0A(key=0111417E) → off=0x2C ← event 2 END / next-empty marker
|
|
||||||
1F → null sentinel
|
|
||||||
```
|
|
||||||
|
|
||||||
This is why event 2's first 5A chunk is at `start_key + 0x46` — that's the address of the
|
|
||||||
"real start" 0x46-record, distinct from the `0x2C`-record at the raw boundary. Use the
|
|
||||||
`0x46` keys as the input to `read_bulk_waveform_stream`, not the `0x2C` keys.
|
|
||||||
|
|
||||||
For event 1 only (start_key[2:4] = 0x0000) BW probes at counter=0x0000 directly, which is
|
|
||||||
the `0x46`-keyed start record. Subsequent events use `start_key + 0x46`.
|
|
||||||
|
|
||||||
**Practical iteration pattern (replaces the old 1E/1F walk for downloads):**
|
|
||||||
|
|
||||||
```
|
|
||||||
Setup: SERIAL × 2 → CHCFG → 1E (token=0x00) → key0
|
|
||||||
For each event:
|
|
||||||
0A(cur_key) → DATA_LENGTH = 0x46 (real) or 0x2C (boundary)
|
|
||||||
1F (token=0x00) → next_key
|
|
||||||
if length was 0x46: → cur_key is a real event; queue it for download
|
|
||||||
cur_key = next_key
|
|
||||||
if next_key all-zero null sentinel: stop
|
|
||||||
|
|
||||||
Then for each queued real-event key:
|
|
||||||
download_event(key) → 5A bulk stream with STRT-bounded chunk walk
|
|
||||||
```
|
|
||||||
|
|
||||||
This is what BW does in the 5-1-26 "Download All" capture — it walks the full event chain
|
|
||||||
collecting `(key, length)` tuples first, *then* downloads each event using the `0x46` keys.
|
|
||||||
|
|
||||||
### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce)
|
### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce)
|
||||||
|
|
||||||
`read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where:
|
`read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where:
|
||||||
@@ -991,6 +347,36 @@ Do NOT use fixed absolute offsets for sample_rate or record_time.
|
|||||||
Quiet Mode enabled. Parser handles this — do not strip it manually before feeding to
|
Quiet Mode enabled. Parser handles this — do not strip it manually before feeding to
|
||||||
`S3FrameParser`.
|
`S3FrameParser`.
|
||||||
|
|
||||||
|
**SUB 5A (bulk waveform) TCP frame splitting — confirmed 2026-04-27:**
|
||||||
|
|
||||||
|
Over TCP via cellular modem, each 5A chunk request that produces a single ~1100-byte
|
||||||
|
A5 response over direct RS-232 may arrive as **two separate, complete S3 frames** of
|
||||||
|
~550 bytes each ("2-frame mode"). The modem's Data Forwarding Timeout (~100-150 ms)
|
||||||
|
can split the RS-232 response into two TCP segments, each parsed as a complete S3 frame.
|
||||||
|
Under different modem/timing conditions the full ~1100-byte response arrives as **one
|
||||||
|
S3 frame** ("1-frame mode").
|
||||||
|
|
||||||
|
**Both modes require `extra_chunks_after_metadata=1`** (the extra chunk at metadata_counter
|
||||||
|
+ 0x0400). The device's waveform footer data lives at circular-buffer address 0x1C00 for
|
||||||
|
this event; the terminator frame must be sent at 0x1C00 (not 0x1800) to receive it.
|
||||||
|
|
||||||
|
Example for a 2-second Continuous event (BE11529, key=01110000) via TCP:
|
||||||
|
- **2-frame mode:** 1 probe frame (554 B) + 5 chunks × 2 frames (556-573 B) + 1 extra chunk × 2 frames + 1 terminator (208 B) = **14 A5 frames** → 6864-byte file
|
||||||
|
- **1-frame mode:** 1 probe frame (~1097 B) + 5 chunks × 1 frame (~1079-1113 B) + 1 extra chunk × 1 frame (smaller, tail of event) + 1 terminator → **8 A5 frames** → 6864-byte file
|
||||||
|
- All frames contribute body data; using all of them gives the correct file.
|
||||||
|
|
||||||
|
**Fix (confirmed 2026-04-27):** `_recv_5a_batch()` in `protocol.py` collects ALL
|
||||||
|
A5 frames per chunk request before the next request is sent, using a 0.5 s batch
|
||||||
|
timeout after the first frame to catch the ~150 ms delayed second frame. `write_blastware_file()`
|
||||||
|
includes ALL body frames without skipping — the extra chunk's frames are part of the
|
||||||
|
body data, NOT padding to be discarded.
|
||||||
|
|
||||||
|
**WRONG earlier hypothesis (do not re-introduce):** An attempt was made to auto-detect
|
||||||
|
1-frame vs 2-frame mode from the probe frame size and skip the extra chunk when
|
||||||
|
`probe_data_len >= 700`. This was wrong — the extra chunk is always needed to advance
|
||||||
|
the device's internal state to the footer address. The `_probe_is_large` branch was
|
||||||
|
removed 2026-04-27.
|
||||||
|
|
||||||
### Required ACEmanager settings (Sierra Wireless RV50/RV55)
|
### Required ACEmanager settings (Sierra Wireless RV50/RV55)
|
||||||
|
|
||||||
| Setting | Value | Why |
|
| Setting | Value | Why |
|
||||||
@@ -1171,8 +557,6 @@ All DB endpoints are read-only except `PATCH /db/events/{id}/false_trigger`.
|
|||||||
| 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture |
|
| 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture |
|
||||||
| 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence; only 1 event stored so token=0xFE appeared to work |
|
| 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence; only 1 event stored so token=0xFE appeared to work |
|
||||||
| 4-3-26 | `bridges/captures/4-3-26/` | Browse-mode S3 capture with 2+ events — confirmed all-zero params for 1F, 1F response layout, null sentinel, 0A context requirement |
|
| 4-3-26 | `bridges/captures/4-3-26/` | Browse-mode S3 capture with 2+ events — confirmed all-zero params for 1F, 1F response layout, null sentinel, 0A context requirement |
|
||||||
| 4-27-26 | `bridges/captures/4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof that SFM was over-reading 5× past event end. BW reads 14 chunks at 0x0200 increments + TERM at end_offset; SFM was reading 37 chunks at 0x0400 increments. STRT end_key field located. |
|
|
||||||
| 5-1-26 | `bridges/captures/5-1-26/comcheck/` | Three sub-captures: SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945`/`_171216`). Confirmed: TERM frame formula across 3 events; metadata pages 0x1002/0x1004 are global (read once per session); event-1 vs event-N chunk-pattern split; WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundaries. |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -1436,7 +820,7 @@ offsets in the raw 1A/E5 payload. Only fields with `✅` have confirmed offsets
|
|||||||
|
|
||||||
**Notes tab:**
|
**Notes tab:**
|
||||||
- Enable User Notes (bool)
|
- Enable User Notes (bool)
|
||||||
- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from 5A metadata pages at counter 0x1002 / 0x1004 — see "SUB 5A — fixed metadata pages" section)
|
- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from A5 frame 7 via 5A)
|
||||||
- Enable Extended Notes (bool); Extended Notes text; Extended Notes Title
|
- Enable Extended Notes (bool); Extended Notes text; Extended Notes Title
|
||||||
- Enable Job Number (bool); Job Number (int)
|
- Enable Job Number (bool); Job Number (int)
|
||||||
- Enable Scaled Distance (bool); Distance from Blast (float); Charge Weight (float) — Scaled Distance is derived
|
- Enable Scaled Distance (bool); Distance from Blast (float); Charge Weight (float) — Scaled Distance is derived
|
||||||
@@ -1748,11 +1132,9 @@ body) because writing a dial string may require DLE escaping for embedded contro
|
|||||||
|
|
||||||
## What's next
|
## What's next
|
||||||
|
|
||||||
**See [README.md → Roadmap (Future)](README.md#roadmap-future) for the canonical deferred-work list.** This section is kept as a status log of in-progress / recently-shipped technical details (encoding schemes, byte layouts, etc.) that are too low-level for the README's roadmap.
|
|
||||||
|
|
||||||
- **Database** — SQLite store for events + monitor log entries; dedup by key; queryable
|
- **Database** — SQLite store for events + monitor log entries; dedup by key; queryable
|
||||||
- **Histograms** — decode histogram-mode A5 data (noise floor tracking)
|
- **Histograms** — decode histogram-mode A5 data (noise floor tracking)
|
||||||
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed BYTE-PERFECT against BW reference (v0.14.3, 2026-05-05):** when fed the BW 5-1-26 3-sec capture's A5 frames, the SFM-built file matches BW's saved `M529LKIQ.G10` byte-for-byte (8708 bytes, 0 differences). Live SFM downloads of event 0 (3-sec) and event 1 (3-sec continuation) both open cleanly in Blastware with full Event Reports, frequency analysis, and waveform plots. Body assembly is just contiguous concatenation of frame contributions in stream order (probe → meta@0x1002 → meta@0x1004 → samples → TERM); no stripping, no overlay, no special handling. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that may need different handling — untested under v0.14.x). Extension mapping: extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
|
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed working for Continuous mode events (2026-04-23):** SFM-generated file opens in Blastware, shows correct PPV/waveform/timestamp. File is ~200 bytes shorter than BW (missing last ADC tail slice) — all measurements correct. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that create spurious STRT markers in the body). Extension mapping: **CONFIRMED FALSE 2026-04-21** — extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
|
||||||
|
|
||||||
**Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units).
|
**Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units).
|
||||||
|
|
||||||
@@ -1788,22 +1170,17 @@ body) because writing a dial string may require DLE escaping for embedded contro
|
|||||||
|
|
||||||
| Folder / File | Contents |
|
| Folder / File | Contents |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `1-2-26/` | First SUB 5A BW TX capture — established 5A frame format (raw offset_hi, DLE-aware checksum). 10 frames verified. |
|
|
||||||
| `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102–112) |
|
| `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102–112) |
|
||||||
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) — 1E/0A/0C/1F sequence confirmed (single event so token=0xFE appeared to work in either branch) |
|
|
||||||
| `4-2-26/` | Download-mode BW TX capture — POLL×3 requirement confirmed (frames 68-73 between 1F and first 5A) |
|
|
||||||
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events — all-zero params for 1F, null sentinel layout, 0A context requirement |
|
|
||||||
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check |
|
|
||||||
| `4-11-26 (mitm/ach_mitm_20260411_001912/)` | Full ACH call-home MITM — erase protocol (0xA3/0x06/0xA2), monitor log partial records confirmed |
|
|
||||||
| `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous |
|
| `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous |
|
||||||
| `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec |
|
| `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec |
|
||||||
| `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) |
|
| `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) |
|
||||||
| `4-20-26/call home settings/` | Call home config read/write captures |
|
| `4-20-26/call home settings/` | Call home config read/write captures |
|
||||||
| `4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof of 5× SFM over-read. STRT end_key field located. |
|
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check |
|
||||||
| **`5-1-26/comcheck/`** | **Triplet of captures that nailed the v0.14.0 walk:** SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945` / `_171216`). Confirmed: TERM frame formula across 3 events, metadata pages 0x1002/0x1004 are global session metadata, event-1 vs event-N chunk pattern split, WAVEHDR off=0x46 vs 0x2C disambiguates real events from boundaries. |
|
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events (1E/0A/1F iteration confirmed) |
|
||||||
| **`5-1-26/comcheck/bwcap3sec/`** | **The byte-perfect reference for v0.14.3.** All 17 BW 5A request frames (probe, 2 metadata, 13 samples, TERM) reproduce byte-for-byte from SFM's framing helpers — including the `10 10 00` DLE-stuffed counter for sample @ 0x1000 that was the long-standing failure mode. |
|
| `4-2-26/` | Download-mode BW TX capture (5A bulk stream, POLL×3 requirement confirmed) |
|
||||||
| `5-4-26/` | BW MITM captures of "copy 3sec / 2sec / Download All" + paired SFM session (`seismo_dl_20260504_145701`) showing the +0x46 event-N probe bug producing 110-chunk runaway walk. Cross-references against 5-1-26 confirmed device behavior is identical. |
|
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) |
|
||||||
|
| `mitm/ach_mitm_20260411_001912/` | Full ACH call-home MITM (erase protocol, 0xA3/0x06/0xA2 confirmed) |
|
||||||
|
|
||||||
To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern
|
To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern
|
||||||
in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes
|
in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes
|
||||||
inside write frame data (the naive parser terminates early at the escaped `0x03`).
|
inside write frame data (the naive parser terminates early at the escaped `0x03`).
|
||||||
-31
@@ -1,31 +0,0 @@
|
|||||||
FROM python:3.11-slim
|
|
||||||
|
|
||||||
WORKDIR /app
|
|
||||||
|
|
||||||
# tzdata is required for the TZ env var to take effect (python:slim
|
|
||||||
# omits the timezone database). Without it, datetime.now() / logging
|
|
||||||
# / matplotlib all stay in UTC regardless of TZ. Default zone gets
|
|
||||||
# set further down via ENV; users override per-deployment via the
|
|
||||||
# `TZ` env var in docker-compose.
|
|
||||||
RUN apt-get update && \
|
|
||||||
apt-get install -y --no-install-recommends curl tzdata && \
|
|
||||||
rm -rf /var/lib/apt/lists/*
|
|
||||||
|
|
||||||
# Default display timezone — applied to server logs, datetime.now(),
|
|
||||||
# matplotlib rendered timestamps, and any naïve-vs-aware datetime
|
|
||||||
# conversions in the PDF renderer. Override via TZ env var in
|
|
||||||
# docker-compose; storage in the DB is always UTC regardless.
|
|
||||||
ENV TZ=America/New_York
|
|
||||||
|
|
||||||
COPY pyproject.toml requirements.txt ./
|
|
||||||
COPY minimateplus ./minimateplus
|
|
||||||
COPY micromate ./micromate
|
|
||||||
COPY sfm ./sfm
|
|
||||||
COPY bridges ./bridges
|
|
||||||
COPY scripts ./scripts
|
|
||||||
|
|
||||||
RUN pip install --no-cache-dir -e .
|
|
||||||
|
|
||||||
EXPOSE 8200
|
|
||||||
|
|
||||||
CMD ["python", "-m", "uvicorn", "sfm.server:app", "--host", "0.0.0.0", "--port", "8200"]
|
|
||||||
@@ -1,11 +1,7 @@
|
|||||||
# seismo-relay `v0.26.0`
|
# seismo-relay `v0.12.1`
|
||||||
|
|
||||||
A ground-up replacement for **Blastware** — Instantel's aging Windows-only
|
A ground-up replacement for **Blastware** — Instantel's aging Windows-only
|
||||||
software for managing seismographs. Supports both the **MiniMate Plus
|
software for managing MiniMate Plus seismographs.
|
||||||
(Series III)** and the **Micromate (Series IV / "Thor")** families:
|
|
||||||
Series III via the live RS-232 / TCP wire protocol *and* Blastware ACH file
|
|
||||||
ingest; Series IV currently via Thor TXT-paired IDF file ingest, with the
|
|
||||||
binary codec on the roadmap.
|
|
||||||
|
|
||||||
Built in Python. Runs on Windows, Linux, or macOS. Connects to instruments
|
Built in Python. Runs on Windows, Linux, or macOS. Connects to instruments
|
||||||
over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
||||||
@@ -14,51 +10,6 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
|||||||
> pipeline working end-to-end over TCP/cellular. ACH Auto Call Home server
|
> pipeline working end-to-end over TCP/cellular. ACH Auto Call Home server
|
||||||
> handles inbound unit connections, downloads events, and persists everything
|
> handles inbound unit connections, downloads events, and persists everything
|
||||||
> to a SQLite database. SFM REST API exposes device control and DB queries.
|
> to a SQLite database. SFM REST API exposes device control and DB queries.
|
||||||
> **As of v0.14.3 (2026-05-05): SUB 5A bulk waveform protocol is verified
|
|
||||||
> byte-perfect against Blastware captures across 2-sec, 3-sec, and 10-sec
|
|
||||||
> events.** Generated `.G10` / `.AB0` files open cleanly in Blastware with
|
|
||||||
> full Event Reports, frequency analysis, and waveform plots.
|
|
||||||
> **v0.16.0 (2026-05-11)** adds BW ASCII report ingestion to
|
|
||||||
> `/db/import/blastware_file` — paired with **series3-watcher v1.5.0**,
|
|
||||||
> every Blastware ACH event lands in SeismoDb with device-authoritative
|
|
||||||
> peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data,
|
|
||||||
> without depending on the still-undecoded waveform body codec.
|
|
||||||
> **v0.18.0 (2026-05-19)** adds Thor / Micromate Series IV ingest at
|
|
||||||
> `/db/import/idf_file` — paired with **thor-watcher v0.3.0**, every
|
|
||||||
> `.IDFH` / `.IDFW` event file (plus its `.txt` sidecar) lands in
|
|
||||||
> SeismoDb the same way BW events do. See
|
|
||||||
> [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md) for
|
|
||||||
> the IDF format reference and reverse-engineering plan.
|
|
||||||
> **v0.19.0 (2026-05-20)** separates Series III and Series IV at the
|
|
||||||
> code level: new `micromate/` package alongside `minimateplus/`, new
|
|
||||||
> `events.device_family` DB column ("series3" / "series4") so the UI
|
|
||||||
> and storage layer dispatch deterministically instead of sniffing
|
|
||||||
> filenames. Self-applying migration backfills existing rows from the
|
|
||||||
> binary filename extension.
|
|
||||||
> **v0.20.0 (2026-05-28)** closes out the Event-Report PDF iteration
|
|
||||||
> started in v0.17.x: histogram layouts render correctly against BW
|
|
||||||
> reference PDFs, the ASCII parser handles real-world edge cases
|
|
||||||
> (`OORANGE`, `>100 Hz`, histogram timestamps), and per-channel ZC
|
|
||||||
> Freq is surfaced in both modals (event browser + main webapp).
|
|
||||||
> Adds a server-wide `TZ` env var so operator-visible timestamps
|
|
||||||
> render in local time instead of UTC. New
|
|
||||||
> `scripts/backfill_sidecars.py --reparse-txt` lets parser fixes be
|
|
||||||
> applied retroactively to existing events without re-forwarding,
|
|
||||||
> using the `.TXT` files preserved at ingest time.
|
|
||||||
> **v0.21.0 (2026-05-29)** is the Thor / Series IV decoder release —
|
|
||||||
> `micromate/idf_file.read_idf_file()` now decodes both IDFW
|
|
||||||
> (waveform) and IDFH (histogram) binaries (87–99% sample fidelity
|
|
||||||
> on quiet IDFW events; all 859 IDFH corpus files decode cleanly).
|
|
||||||
> A new `micromate/idf_to_bw_report.py` adapter projects parsed
|
|
||||||
> Thor reports into the BW-shaped sidecar block, so Thor events
|
|
||||||
> flow through the existing Event Report PDF pipeline without a
|
|
||||||
> separate renderer. Terra-View v0.13.0 ships in parallel and
|
|
||||||
> closes Phase 1 of the SFM integration — see its CHANGELOG.
|
|
||||||
> **v0.22.0 (2026-07-03)** adds the SFM side of the full-snapshot
|
|
||||||
> bundle: `GET /db/snapshot` (WAL-safe DB copy), `GET /db/waveforms/
|
|
||||||
> recent.zip` (recent events' waveform files), and a gated
|
|
||||||
> `POST /db/restore` (`SFM_DB_RESTORE_ENABLED`, dev-only). Terra-View
|
|
||||||
> v0.17.0 drives them to pull a one-pass prod→dev refresh.
|
|
||||||
> See [CHANGELOG.md](CHANGELOG.md) for full version history.
|
> See [CHANGELOG.md](CHANGELOG.md) for full version history.
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -67,36 +18,26 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
|
|||||||
|
|
||||||
```
|
```
|
||||||
seismo-relay/
|
seismo-relay/
|
||||||
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Download + Console tabs)
|
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Console tabs)
|
||||||
│
|
│
|
||||||
├── minimateplus/ ← Series III (MiniMate Plus) client library
|
├── minimateplus/ ← MiniMate Plus client library
|
||||||
│ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport
|
│ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport
|
||||||
│ ├── protocol.py ← DLE frame layer, SUB command dispatch
|
│ ├── protocol.py ← DLE frame layer, SUB command dispatch
|
||||||
│ ├── client.py ← High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …)
|
│ ├── client.py ← High-level client (connect, get_events, push_config, …)
|
||||||
│ ├── framing.py ← Frame builders, DLE codec, S3FrameParser
|
│ ├── framing.py ← Frame builders, DLE codec, S3FrameParser
|
||||||
│ ├── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig, …
|
│ └── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, …
|
||||||
│ ├── bw_ascii_report.py ← Parse BW per-event ASCII reports (.TXT sidecars)
|
|
||||||
│ ├── event_file_io.py ← Read BW binaries, write .sfm.json sidecars
|
|
||||||
│ └── blastware_file.py ← Write events to Blastware-compatible .AB0 files
|
|
||||||
│
|
|
||||||
├── micromate/ ← Series IV (Micromate / Thor) client library (NEW v0.19)
|
|
||||||
│ ├── models.py ← IdfEvent, IdfReport, IdfPeaks, IdfProjectInfo, IdfSensorCheck (mic in native dB(L))
|
|
||||||
│ ├── idf_ascii_report.py ← Parse Thor .IDFW.txt / .IDFH.txt event sidecars
|
|
||||||
│ ├── idf_file.py ← Binary codec for .IDFW + .IDFH (v0.21.0+)
|
|
||||||
│ └── idf_to_bw_report.py ← Adapter projecting Thor IDF into the BW report shape (v0.21.0+)
|
|
||||||
│
|
│
|
||||||
├── sfm/ ← SFM REST API server (FastAPI, port 8200)
|
├── sfm/ ← SFM REST API server (FastAPI, port 8200)
|
||||||
│ ├── server.py ← Live device endpoints + DB query + ingest endpoints + caching
|
│ ├── server.py ← All device + DB endpoints
|
||||||
│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions)
|
│ ├── database.py ← SeismoDb — SQLite persistence layer
|
||||||
│ ├── waveform_store.py ← On-disk store for BW + IDF event binaries + .sfm.json sidecars
|
│ └── sfm_webapp.html ← Embedded web UI (served at /)
|
||||||
│ └── sfm_webapp.html ← Embedded web UI with Call Home config tab
|
|
||||||
│
|
│
|
||||||
├── bridges/
|
├── bridges/
|
||||||
│ ├── ach_server.py ← Inbound ACH call-home server (main production server)
|
│ ├── ach_server.py ← Inbound ACH call-home server (main production server)
|
||||||
│ ├── ach_mitm.py ← Transparent MITM proxy for capturing BW sessions
|
│ ├── ach_mitm.py ← Transparent MITM proxy for capturing BW sessions
|
||||||
│ ├── s3-bridge/ ← RS-232 serial bridge (capture tool)
|
│ ├── s3-bridge/ ← RS-232 serial bridge (capture tool)
|
||||||
│ ├── tcp_serial_bridge.py ← Local TCP↔serial bridge (bench testing)
|
│ ├── tcp_serial_bridge.py ← Local TCP↔serial bridge (bench testing)
|
||||||
│ ├── gui_bridge.py ← Standalone bridge GUI with raw capture checkboxes
|
│ ├── gui_bridge.py ← Standalone bridge GUI
|
||||||
│ └── raw_capture.py ← Simple raw capture tool
|
│ └── raw_capture.py ← Simple raw capture tool
|
||||||
│
|
│
|
||||||
├── parsers/
|
├── parsers/
|
||||||
@@ -105,8 +46,7 @@ seismo-relay/
|
|||||||
│ └── frame_db.py ← SQLite frame database
|
│ └── frame_db.py ← SQLite frame database
|
||||||
│
|
│
|
||||||
└── docs/
|
└── docs/
|
||||||
├── instantel_protocol_reference.md ← Series III protocol spec (the Rosetta Stone)
|
└── instantel_protocol_reference.md ← Reverse-engineered protocol spec
|
||||||
└── idf_protocol_reference.md ← Series IV (Thor IDF) format reference + codec RE plan
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -161,28 +101,21 @@ python seismo_lab.py
|
|||||||
Each call dials the device, does its work, and closes the connection. TCP
|
Each call dials the device, does its work, and closes the connection. TCP
|
||||||
connections are retried once on `ProtocolError` to handle cold-boot timing.
|
connections are retried once on `ProtocolError` to handle cold-boot timing.
|
||||||
|
|
||||||
**In-memory caching** — frequently-polled endpoints avoid redundant TCP round-trips
|
**Caching** — frequently-polled endpoints are cached in-process to avoid
|
||||||
via a thread-safe `_LiveCache` (plain Python dict + `threading.Lock`):
|
redundant TCP round-trips:
|
||||||
|
|
||||||
| Method | URL | Cache Strategy |
|
| Method | URL | Cache |
|
||||||
|--------|-----|---|
|
|--------|-----|-------|
|
||||||
| `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` |
|
| `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` |
|
||||||
| `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected |
|
| `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected |
|
||||||
| `GET` | `/device/event/{idx}/waveform` | Permanent per event index |
|
| `GET` | `/device/event/{idx}/waveform` | Permanent per event index |
|
||||||
| `GET` | `/device/monitor/status` | 30-second TTL; invalidated by monitor start/stop |
|
| `GET` | `/device/monitor/status` | 30-second TTL |
|
||||||
| `GET` | `/device/call_home` | Fresh read from device (not cached) |
|
|
||||||
| `POST` | `/device/connect` | — |
|
| `POST` | `/device/connect` | — |
|
||||||
| `POST` | `/device/config` | Writes compliance config; invalidates info + events cache |
|
| `POST` | `/device/config` | Writes compliance config; invalidates cache |
|
||||||
| `POST` | `/device/config/project` | Patches project/client/operator/sensor_location strings |
|
| `POST` | `/device/monitor/start` | Sends SUB 0x96 |
|
||||||
| `POST` | `/device/monitor/start` | Sends SUB 0x96; immediately evicts status cache |
|
| `POST` | `/device/monitor/stop` | Sends SUB 0x97 |
|
||||||
| `POST` | `/device/monitor/stop` | Sends SUB 0x97; immediately evicts status cache |
|
|
||||||
| `POST` | `/device/call_home` | Reads, patches specified fields, writes back to device |
|
|
||||||
|
|
||||||
**Cache bypass** — All cached endpoints accept `?force=true` to skip the cache and
|
All cached endpoints accept `?force=true` to bypass the cache.
|
||||||
force a fresh read from the device.
|
|
||||||
|
|
||||||
**Cache stats** — `GET /cache/stats` returns hit/miss counts and TTL info; `DELETE /cache/device`
|
|
||||||
clears the device cache immediately.
|
|
||||||
|
|
||||||
Transport query params (supply one set):
|
Transport query params (supply one set):
|
||||||
```
|
```
|
||||||
@@ -198,35 +131,11 @@ Query the SQLite database written by `ach_server.py`. All read-only except
|
|||||||
| Method | URL | Description |
|
| Method | URL | Description |
|
||||||
|--------|-----|-------------|
|
|--------|-----|-------------|
|
||||||
| `GET` | `/db/units` | All known serials with summary stats |
|
| `GET` | `/db/units` | All known serials with summary stats |
|
||||||
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger). Response rows include `device_family` ("series3" / "series4") so clients dispatch on unit type without sniffing filenames. |
|
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger) |
|
||||||
| `GET` | `/db/monitor_log` | Monitoring intervals |
|
| `GET` | `/db/monitor_log` | Monitoring intervals |
|
||||||
| `GET` | `/db/sessions` | ACH call-home session history |
|
| `GET` | `/db/sessions` | ACH call-home session history |
|
||||||
| `PATCH` | `/db/events/{id}/false_trigger?value=true` | Flag / unflag false triggers |
|
| `PATCH` | `/db/events/{id}/false_trigger?value=true` | Flag / unflag false triggers |
|
||||||
|
|
||||||
### File ingest endpoints
|
|
||||||
|
|
||||||
Used by watcher daemons to push field-collected event files into the SFM DB
|
|
||||||
+ waveform store. Both accept multipart uploads of binary event files
|
|
||||||
optionally paired with their ASCII sidecar reports; both dedup by
|
|
||||||
`(serial, timestamp)` and UPSERT device-authoritative fields on re-import.
|
|
||||||
|
|
||||||
| Method | URL | Description |
|
|
||||||
|--------|-----|-------------|
|
|
||||||
| `POST` | `/db/import/blastware_file` | Series III: `.AB0*` / `.N00` binaries + paired `_ASCII.TXT`. Source: `series3-watcher`. |
|
|
||||||
| `POST` | `/db/import/idf_file` | Series IV: `.IDFH` / `.IDFW` binaries + paired `.IDFW.txt` / `.IDFH.txt`. Source: `thor-watcher`. |
|
|
||||||
|
|
||||||
### DB snapshot / restore endpoints
|
|
||||||
|
|
||||||
Back the "full snapshot bundle" prod→dev refresh Terra-View orchestrates
|
|
||||||
(Settings → Database). Snapshot/zip are read-only; restore is gated and
|
|
||||||
dev-only.
|
|
||||||
|
|
||||||
| Method | URL | Description |
|
|
||||||
|--------|-----|-------------|
|
|
||||||
| `GET` | `/db/snapshot` | WAL-safe point-in-time copy of `seismo_relay.db` via the SQLite online backup API. |
|
|
||||||
| `GET` | `/db/waveforms/recent.zip?n=…` | Zips the on-disk files (event file + `.h5` / sidecars) for the *n* most-recent events. |
|
|
||||||
| `POST` | `/db/restore` | Validate-first restore of an uploaded DB + waveforms; takes a WAL-safe pre-restore safety backup, path-traversal-guards the zip. Gated by `SFM_DB_RESTORE_ENABLED` (dormant → 404). |
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## minimateplus library
|
## minimateplus library
|
||||||
@@ -249,116 +158,42 @@ with client:
|
|||||||
events = client.get_events() # Full download: headers + peaks + metadata
|
events = client.get_events() # Full download: headers + peaks + metadata
|
||||||
monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag
|
monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag
|
||||||
log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records)
|
log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records)
|
||||||
ach_cfg = client.get_call_home_config() # Auto Call Home settings (SUB 0x2C)
|
|
||||||
|
|
||||||
# Write
|
# Write
|
||||||
client.apply_config(
|
client.apply_config(
|
||||||
sample_rate=1024,
|
sample_rate=1024,
|
||||||
recording_mode="Continuous", # Single Shot / Continuous / Histogram / Histogram+Continuous
|
|
||||||
histogram_interval_sec=15, # 2, 5, 15, 60, 300, 900
|
|
||||||
trigger_level_geo=0.5,
|
trigger_level_geo=0.5,
|
||||||
geo_range="Normal", # Normal (10.000 in/s) / Sensitive (1.25 in/s)
|
|
||||||
project="Bridge Inspection 2026",
|
project="Bridge Inspection 2026",
|
||||||
client_name="City of Portland",
|
client_name="City of Portland",
|
||||||
operator="B. Harrison",
|
operator="B. Harrison",
|
||||||
)
|
)
|
||||||
|
|
||||||
client.set_call_home_config(
|
|
||||||
auto_call_home_enabled=True,
|
|
||||||
after_event_recorded=True,
|
|
||||||
at_specified_times=True,
|
|
||||||
time1_hour=18, time1_min=30, # 6:30 PM
|
|
||||||
time2_hour=6, time2_min=0, # 6:00 AM
|
|
||||||
)
|
|
||||||
|
|
||||||
# Control
|
# Control
|
||||||
client.start_monitoring() # SUB 0x96
|
client.start_monitoring() # SUB 0x96
|
||||||
client.stop_monitoring() # SUB 0x97
|
client.stop_monitoring() # SUB 0x97
|
||||||
client.delete_all_events() # Erase all (SUB 0xA3 → 0x1C → 0x06 → 0xA2)
|
client.delete_all_events() # Erase all (SUB 0xA3 → 0x1C → 0x06 → 0xA2)
|
||||||
```
|
```
|
||||||
|
|
||||||
`get_events()` runs the full per-event sequence:
|
`get_events()` runs the full per-event sequence: `1E → 0A → 0C → 5A → 1F`.
|
||||||
`1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)`.
|
SUB 5A bulk stream provides `client`, `operator`, and `sensor_location` as they
|
||||||
SUB 5A bulk stream walks chunks bounded by the `end_offset` extracted from
|
existed at record time — not backfilled from the current compliance config.
|
||||||
the STRT record at byte 17 of the probe response — no over-reading, no
|
|
||||||
chunk-count cap. Project / client / operator / sensor location strings come
|
|
||||||
from the dedicated metadata pages at counter `0x1002` and `0x1004`,
|
|
||||||
read once per session (they reflect the compliance setup at session start,
|
|
||||||
not per individual event).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## micromate library
|
|
||||||
|
|
||||||
Series IV / Thor support, sibling to `minimateplus`. Currently scoped to
|
|
||||||
offline-file ingest from Thor's TXT exporter; live-device protocol is
|
|
||||||
deferred until the binary codec is cracked.
|
|
||||||
|
|
||||||
```python
|
|
||||||
from micromate import IdfEvent, parse_idf_report
|
|
||||||
|
|
||||||
# Parse a .IDFW.txt / .IDFH.txt sidecar (1014 example files round-trip cleanly)
|
|
||||||
text = open("UM11719_20231219162723.IDFW.txt").read()
|
|
||||||
report_dict = parse_idf_report(text) # permissive dict
|
|
||||||
|
|
||||||
# Wrap into a typed event using the device-native binary filename
|
|
||||||
event = IdfEvent.from_report(report_dict, "UM11719_20231219162723.IDFW")
|
|
||||||
|
|
||||||
event.serial # "UM11719"
|
|
||||||
event.kind # "Waveform" or "Histogram"
|
|
||||||
event.peaks.transverse_ips # 0.0251 (in/s, native unit)
|
|
||||||
event.peaks.mic_pspl_dbl # 99.4 (dB(L), Thor's native mic unit — NOT psi)
|
|
||||||
event.project_info.project # "UPMC Presby-Loc 3-Level1-1R Elevator Rm"
|
|
||||||
event.sensor_check.tran # True (passed self-check)
|
|
||||||
event.firmware_version # "Micromate ISEE 11.0AK"
|
|
||||||
event.calibration_text # "November 22, 2023 by Instantel"
|
|
||||||
|
|
||||||
# Bridge to the existing minimateplus.Event shape for the DB / sidecar paths
|
|
||||||
# (waveform_key is a 16-byte sha256 prefix when ingesting from a binary file)
|
|
||||||
bridged_event = event.to_minimateplus_event(waveform_key=b"\x00" * 16)
|
|
||||||
```
|
|
||||||
|
|
||||||
The binary codec (`.IDFW` / `.IDFH` event files themselves) is on the
|
|
||||||
roadmap — see [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md)
|
|
||||||
for everything known so far, the two observed file signatures, and the
|
|
||||||
reverse-engineering plan. The `micromate/idf_file.py` stub is where
|
|
||||||
`read_idf_file()` will land.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Database
|
## Database
|
||||||
|
|
||||||
`ach_server.py` and the file-ingest endpoints write to
|
`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode).
|
||||||
`bridges/captures/seismo_relay.db` (SQLite, WAL mode) via the `SeismoDb`
|
Three tables, all unit-keyed by serial number:
|
||||||
persistence layer. Three tables, all unit-keyed by serial number:
|
|
||||||
|
|
||||||
| Table | Key | Contents |
|
| Table | Key | Contents |
|
||||||
|-------|-----|----------|
|
|-------|-----|----------|
|
||||||
| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds |
|
| `ach_sessions` | UUID | Per-call-home audit record: serial, peer IP, events_downloaded, duration |
|
||||||
| `events` | UUID, UNIQUE(serial, timestamp) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag, **`device_family`** ("series3" / "series4"), `blastware_filename` (binary at-rest in `waveforms/`), sidecar references |
|
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, PPV per channel, project/client/operator strings, false_trigger flag |
|
||||||
| `monitor_log` | UUID, UNIQUE(serial, start_time) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
|
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: start/stop time, duration, geo threshold |
|
||||||
|
|
||||||
**Deduplication is by `(serial, timestamp)`** — the device clock is the
|
Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs
|
||||||
stable natural key. Repeat call-homes or re-runs UPSERT the row in place,
|
never produce duplicate rows. Post-erase key reuse is handled automatically
|
||||||
refreshing every device-authoritative field (peaks, project strings,
|
via the high-water mark in `ach_state.json`.
|
||||||
sample_rate, file references) so the latest writer wins. `false_trigger`
|
|
||||||
and `device_family` are preserved across UPSERTs. Earlier versions used
|
|
||||||
`(serial, waveform_key)` for dedup, but the device's event-key counter
|
|
||||||
resets to `0x01110000` after every erase, so timestamps are the correct
|
|
||||||
dedup field. Migration handles the transition transparently on first
|
|
||||||
startup.
|
|
||||||
|
|
||||||
**`device_family` (added v0.19.0)** discriminates Series III from Series
|
|
||||||
IV at the SQL level. Set by every import path; the UI dispatches on it
|
|
||||||
to render mic units correctly (Series III: psi → dBL conversion; Series
|
|
||||||
IV: native dBL passthrough). Existing rows are backfilled at first
|
|
||||||
startup of v0.19.0+ by sniffing the binary filename extension.
|
|
||||||
|
|
||||||
The on-disk waveform store lives at `bridges/captures/waveforms/<serial>/`
|
|
||||||
and holds the original event binaries (BW `.AB0*` / `.N00` for Series III,
|
|
||||||
`.IDFH` / `.IDFW` for Series IV) plus their `.sfm.json` review/metadata
|
|
||||||
sidecars. Series III events also produce `.a5.pkl` source-frame pickles
|
|
||||||
and `.h5` clean-waveform exports; Series IV doesn't yet (pending codec).
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -396,27 +231,6 @@ Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/insta
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Compliance Config Features
|
|
||||||
|
|
||||||
The REST API and web UI expose full control over device compliance settings:
|
|
||||||
|
|
||||||
- **Recording Mode** (Single Shot / Continuous / Histogram / Histogram+Continuous)
|
|
||||||
- **Sample Rate** (1024 / 2048 / 4096 sps)
|
|
||||||
- **Record Time** (float, seconds)
|
|
||||||
- **Histogram Interval** (2s, 5s, 15s, 1m, 5m, 15m) — when recording mode includes histogram
|
|
||||||
- **Geo Trigger Levels** (float, in/s per channel)
|
|
||||||
- **Geo Maximum Range** (Normal 10.000 in/s / Sensitive 1.250 in/s per channel)
|
|
||||||
- **Project / Client / Operator / Sensor Location** (ASCII strings)
|
|
||||||
|
|
||||||
Auto Call Home config:
|
|
||||||
- **Auto Call Home Enable** (bool)
|
|
||||||
- **Dial String** (read-only; 40-byte ASCII)
|
|
||||||
- **Trigger on Event** (bool)
|
|
||||||
- **Scheduled Call-Ins** (two time slots with HH:MM each)
|
|
||||||
- **Retry Settings** (count, delay, connection timeout, warm-up time)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -438,171 +252,17 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Key Features
|
## Roadmap
|
||||||
|
|
||||||
**Series III (MiniMate Plus) device support:**
|
- [x] Full read pipeline — device info, compliance config, event download with true event-time metadata
|
||||||
- [x] Full read/write/erase pipelines over RS-232 or TCP/cellular
|
- [x] Write commands — push compliance config, trigger thresholds, project strings to device
|
||||||
- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings)
|
- [x] Erase all events — confirmed erase sequence from live MITM capture
|
||||||
- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries)
|
- [x] Monitor control — start/stop monitoring, read battery/memory/status
|
||||||
- [x] Monitor control (start/stop, status polling, battery/memory)
|
- [x] Monitor log entries — decode partial 0x2C records (continuous monitoring intervals)
|
||||||
- [x] Monitor log entries (continuous monitoring intervals without full waveform download)
|
- [x] ACH inbound server — accept call-home connections, download events, dedup by key
|
||||||
- [x] Blastware file ingest at `/db/import/blastware_file` (paired with `series3-watcher`)
|
- [x] SQLite persistence — events, monitor log, and session history in `seismo_relay.db`
|
||||||
|
- [x] SFM REST API — device control + DB query endpoints, live device cache
|
||||||
**Series IV (Micromate / Thor) device support:**
|
- [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing
|
||||||
- [x] Thor IDF file ingest at `/db/import/idf_file` (paired with `thor-watcher`, v0.18.0+)
|
- [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first)
|
||||||
- [x] Native `IdfEvent` / `IdfReport` typed models — mic in dB(L), full title strings, sensor self-check, calibration, firmware version
|
- [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object
|
||||||
- [x] Parser verified against 1,014 paired `.txt` sidecars in `thor-watcher/example-data/`
|
- [ ] Modem manager — push RV50/RV55 configs via Sierra Wireless API
|
||||||
- [x] Binary `.IDFW` / `.IDFH` codec — ✅ v0.21.0. IDFW reuses `decode_waveform_v2()` on the body at offset `0x0f1f` (87–99% sample fidelity on quiet events); IDFH has a dedicated segment-based decoder (all 859 corpus files decode, 181,071 intervals total). See `micromate/idf_file.py` + `docs/idf_protocol_reference.md`.
|
|
||||||
- [ ] Live-device protocol — pending codec
|
|
||||||
|
|
||||||
**Data persistence:**
|
|
||||||
- [x] SQLite database (`seismo_relay.db`) with `events`, `monitor_log`, `ach_sessions` tables
|
|
||||||
- [x] Per-row `device_family` column ("series3" / "series4") for clean UI / unit-of-measurement dispatch (v0.19.0+)
|
|
||||||
- [x] Deduplication by `(serial, timestamp)` — natural key handles post-erase counter resets
|
|
||||||
- [x] UPSERT on re-import refreshes every device-authoritative field (peaks, project, sample_rate); preserves operator review state (`false_trigger`)
|
|
||||||
- [x] Post-erase key-reuse detection (tracks high-water mark in `ach_state.json`)
|
|
||||||
|
|
||||||
**REST API:**
|
|
||||||
- [x] Live device endpoints with in-memory caching (`_LiveCache`)
|
|
||||||
- [x] Cache statistics (`/cache/stats`) and manual invalidation (`/cache/device`)
|
|
||||||
- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH)
|
|
||||||
- [x] Call Home config read/write endpoints
|
|
||||||
- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`)
|
|
||||||
- [x] Import endpoints for both device families (`/db/import/blastware_file`, `/db/import/idf_file`)
|
|
||||||
|
|
||||||
**File output (v0.7+, byte-perfect as of v0.14.3):**
|
|
||||||
- [x] Blastware-compatible `.AB0` / `.G10` file generation (waveform + metadata)
|
|
||||||
- [x] Multi-channel waveform decode from SUB 5A bulk stream
|
|
||||||
- [x] Second-resolution timestamp encoding in Blastware filename
|
|
||||||
- [x] **Byte-perfect against BW reference captures** (verified across 2-sec / 3-sec / 10-sec event durations, both event 0 and event N continuation events)
|
|
||||||
- [x] STRT-bounded chunk walk + correct event-N probe counter + partial DLE stuffing of `0x10` in 5A params (the four fixes that landed in v0.14.0–v0.14.3)
|
|
||||||
|
|
||||||
**Capture tools:**
|
|
||||||
- [x] Serial-to-TCP bridge with raw BW/S3 capture (s3_bridge.py, defaults to auto-capture)
|
|
||||||
- [x] GUI bridge with raw capture checkboxes (gui_bridge.py)
|
|
||||||
- [x] ACH inbound server with bidirectional capture (ach_server.py saves raw_tx + raw_rx)
|
|
||||||
- [x] Transparent TCP MITM proxy for live BW session capture (ach_mitm.py)
|
|
||||||
|
|
||||||
**Analysis tools:**
|
|
||||||
- [x] s3_analyzer.py — session parser, frame differ, Claude export
|
|
||||||
- [x] gui_analyzer.py — standalone analyzer GUI
|
|
||||||
- [x] frame_db.py — SQLite frame database for capture analysis
|
|
||||||
|
|
||||||
**seismo_lab.py GUI:**
|
|
||||||
- [x] Bridge tab — Serial/TCP mode selector with raw capture options
|
|
||||||
- [x] Analyzer tab — BW/S3 capture playback and differencing
|
|
||||||
- [x] Download tab — Live wire-byte capture during event download
|
|
||||||
- [x] Console tab — Logging and diagnostics
|
|
||||||
|
|
||||||
## Roadmap (Future)
|
|
||||||
|
|
||||||
### Strategic direction — where this is going
|
|
||||||
|
|
||||||
seismo-relay is being built as a **suite of cooperating components**
|
|
||||||
that together replace and improve on Blastware's role. Three logical
|
|
||||||
tiers:
|
|
||||||
|
|
||||||
1. **SFM** (device-side) — owns the active connection to a physical
|
|
||||||
unit. Today: `minimateplus/`, `/device/*` HTTP endpoints,
|
|
||||||
`seismo_lab.py`. Future: live Thor / Micromate support.
|
|
||||||
2. **SDM** (data-side) — owns the database, waveform store, ingest
|
|
||||||
pipelines, and the read-API that Terra-View consumes. Today this
|
|
||||||
code lives under `sfm/` for historical reasons; the role has
|
|
||||||
migrated and the eventual rename is on the long-tail cleanup list.
|
|
||||||
3. **Codec library** — pure data-interpretation: `minimateplus/*_codec.py`,
|
|
||||||
`bw_ascii_report.py`, `micromate/idf_*.py`. Used by both SFM and
|
|
||||||
SDM, depends on neither.
|
|
||||||
|
|
||||||
Terra-View is downstream of SDM for fleet listings, event detail, etc.
|
|
||||||
The long-term vision adds a **second link** from Terra-View → SFM for
|
|
||||||
direct device interaction (see below).
|
|
||||||
|
|
||||||
The codec work in this repo isn't trying to replace BW's network
|
|
||||||
layer — BW's ACH file forwarding and Thor's IDF call-home are
|
|
||||||
battle-tested. The value is in the receiving and processing side: turn
|
|
||||||
the stream of binary+ASCII pairs into something users can search,
|
|
||||||
filter, alert on, and report from.
|
|
||||||
|
|
||||||
### Terra-View ↔ SFM device control (the long-term vision)
|
|
||||||
|
|
||||||
Today Terra-View only reads from SDM (event listings, dashboards,
|
|
||||||
project reports). When a unit goes missing — operator notices in the
|
|
||||||
Terra-View dashboard — there's no way to *do* anything from the UI.
|
|
||||||
The path of least resistance is to RDP into a Windows box and open
|
|
||||||
Blastware, which defeats the purpose of having Terra-View.
|
|
||||||
|
|
||||||
Target experience:
|
|
||||||
- Operator notices a unit in Terra-View dashboard hasn't called in.
|
|
||||||
- Clicks unit detail → "Connect to Device" button.
|
|
||||||
- Terra-View opens an embedded view (modal or side-panel) that talks
|
|
||||||
to SFM's `/device/*` endpoints over the network.
|
|
||||||
- Live view: device clock, battery, memory, current monitor status.
|
|
||||||
- Actions: start/stop monitoring, push compliance config changes, pull
|
|
||||||
fresh events, run a sensor self-check, change call-home settings.
|
|
||||||
- Audit log: every connect / action recorded in SDM for the unit
|
|
||||||
history.
|
|
||||||
|
|
||||||
Implementation steps (concrete):
|
|
||||||
- [ ] **SFM authentication & authorization layer.** Today `/device/*`
|
|
||||||
endpoints are unauthenticated — anyone on the network can call
|
|
||||||
them. Need at minimum a token-based auth, ideally with a "who
|
|
||||||
can connect to which units" mapping. Hard prerequisite for
|
|
||||||
letting Terra-View users into the control surface.
|
|
||||||
- [ ] **Terra-View "Connect to Device" entry point** on the unit
|
|
||||||
detail page. Renders only when unit has connection info on file
|
|
||||||
and the user has permission.
|
|
||||||
- [ ] **Embedded live-monitor view** in Terra-View — equivalent to
|
|
||||||
`seismo_lab.py`'s Bridge tab, but in the browser. Polls SFM's
|
|
||||||
`/device/monitor/status` on an interval; sends start/stop via
|
|
||||||
`/device/monitor/{start,stop}`.
|
|
||||||
- [ ] **Action history** — every connect / push / action call records
|
|
||||||
a row in `unit_history`, viewable on the unit detail page.
|
|
||||||
- [ ] **Series IV live-device support in SFM** — currently `/device/*`
|
|
||||||
only supports MiniMate Plus. Blocks "Connect to Device" for
|
|
||||||
Thor units until done. Depends on Thor wire-protocol capture
|
|
||||||
and a `micromate/` parallel of the `minimateplus/` modules.
|
|
||||||
|
|
||||||
### High-impact (unblocks product features)
|
|
||||||
|
|
||||||
- [ ] **Series III waveform body codec reverse-engineering.** The 5A bulk-stream body is some kind of compressed/encoded format (not raw int16 LE as previously assumed — see §7.6.1 retraction in `docs/instantel_protocol_reference.md`). Structural framing is ~50% decoded on branch `claude/codec-re-cBGNe` (tagged-block walker, segment counters); per-byte sample mapping is still open. Until this lands, the in-app waveform viewer renders garbage and BW-import peak values fall back to `_peaks_from_samples()` saturation noise. Workaround: pair every BW-imported event with its `_ASCII.TXT` so the device-authoritative peaks land in the DB regardless of codec.
|
|
||||||
- [x] **Series IV (Thor IDF) binary codec reverse-engineering.** ✅ v0.21.0 — `micromate/idf_file.read_idf_file()` decodes both IDFW (waveform body at offset `0x0f1f`, reusing `decode_waveform_v2()`; 87–99% sample fidelity on quiet events) and IDFH (dedicated segment-based decoder: all 859 corpus files decode, 181,071 intervals, peaks within ~1.8% of sidecar values). `WaveformStore.save_imported_idf` now also projects parsed Thor data into a `bw_report` block via `micromate/idf_to_bw_report.py` so Thor events render in the existing Event Report PDF pipeline without a separate renderer.
|
|
||||||
- [ ] **In-app waveform viewer accuracy.** Depends on Series III codec decode. Plot.v1 JSON pipeline + viewer skeleton already exist; will start showing real waveforms automatically once `_decode_a5_waveform` produces correct samples. Series IV waveforms come online when the IDF codec lands.
|
|
||||||
- [ ] **Series IV live-device support.** Once the IDF binary is decoded, extend `micromate/` with `transport.py` / `framing.py` / `protocol.py` / `client.py` mirroring the `minimateplus/` package layout — depends on capturing Thor's wire protocol (TCP / RS-232 captures TBD).
|
|
||||||
- [ ] **Terra-view integration** — seismo-relay router, unit detail page, VISON-style event listing.
|
|
||||||
- [ ] **Vibration summary reports** — highest legit PPV per project → Word doc (false-trigger filtering first).
|
|
||||||
|
|
||||||
### BW ASCII report parser enhancements (built in v0.16.0)
|
|
||||||
|
|
||||||
- [x] **PPV field misses on certain TXT formats.** ✅ v0.20.0 — root cause was the `OORANGE` (Out Of Range) saturation marker that BW writes when a channel exceeds its full-scale; `_parse_number()` returned None for the non-numeric value. Parser now substitutes `geo_range_ips` as a lower bound + sets `ppv_saturated` flag. All 5 prod events (T190LD5Q.LK0W, T438L713.RY0W, K557L3YM.OE0W, + 2 others) now parse cleanly.
|
|
||||||
- [x] **Histogram-specific structural fields.** ✅ v0.20.0 — `Histogram Start/Stop Time+Date`, `Number of Intervals`, `Interval Size`, per-channel `Peak Time` + `Peak Date`, and `Peak Vector Sum Date` all parse now. Land in the sidecar's `bw_report.histogram` block.
|
|
||||||
- [ ] **Histogram interval bin-table parsing.** Trailing 792-row table (per-interval Peak/Freq per channel + MicL) in histogram TXTs is unparsed. Probably too big for the sidecar JSON; may want a separate `.histogram.h5` companion file.
|
|
||||||
- [x] **`>100 Hz` value parsing.** ✅ v0.20.0 — parser now mirrors the OORANGE pattern: stores 100.0 on `zc_freq_hz` + sets `zc_freq_above_range` flag. PDF + both modals render `>100 Hz` instead of `—`.
|
|
||||||
|
|
||||||
### Ingestion gaps
|
|
||||||
|
|
||||||
- [ ] **MLG forwarding.** `series3-watcher` forwards event binaries + their `_ASCII.TXT` reports, but skips `.MLG` per-unit monitor log files entirely. Adding an `POST /db/import/mlg_file` endpoint + watcher scan path would populate `monitor_log` for non-ACH-routed units (coverage queries, "was this unit monitoring on date X" lookups).
|
|
||||||
- [ ] **0C-record raw bytes persistence in the sidecar.** Currently on branch `claude/codec-re-cBGNe` as commit `a187124`; cherry-pick if useful as a standalone fix. Preserves the 210-byte 0C record under `extensions.raw_records.waveform_record_b64` so future field-offset analysis (Peak Acceleration / Time of Peak / etc. — the fields BW computes client-side from samples) can run offline.
|
|
||||||
|
|
||||||
### Operational
|
|
||||||
|
|
||||||
- [ ] **`series3-watcher` file archive manager** — 90-day-old events moved to `<watch_folder>_archive/<year>/<month>/` subfolders. Plan drafted in `claude/codec-re-cBGNe`'s plan-mode session; awaiting a 5-minute test on whether Blastware UI walks subfolders before any code lands (determines layout: in-place subfolders vs sibling archive).
|
|
||||||
- [ ] **Compliance config encoder** — build raw write payloads from a `ComplianceConfig` object.
|
|
||||||
- [ ] **Modem manager** — push RV50/RV55 configs via Sierra Wireless API.
|
|
||||||
- [ ] **Call Home dial_string write support** (requires DLE escaping for embedded control characters).
|
|
||||||
- [ ] **Histogram mode recording support** (5A stream analysis for mode 0x03 — separate from histogram ASCII parsing above).
|
|
||||||
|
|
||||||
### Test coverage
|
|
||||||
|
|
||||||
- [ ] Verify 30-sec event download — body may exceed `0xFFFF` and force the device into a different `end_key` encoding (none of the 2/3/10-sec test cases hit this boundary).
|
|
||||||
- [ ] Histogram mode (0x03) write via SFM — confirmed working for Single Shot / Continuous / Histogram+Continuous; Histogram (0x03) needs a live test from a non-Histogram starting state.
|
|
||||||
|
|
||||||
### Lower-priority cleanups
|
|
||||||
|
|
||||||
- [ ] Compliance write anchor-9 cleanup — when changing recording_mode via SFM, a spurious `0x10` may persist after Histogram→other mode transitions. Doesn't affect device operation but differs from BW's byte-perfect output.
|
|
||||||
- [ ] Locate "Sensor Check" byte in compliance config (need capture with Disabled vs Before-monitoring).
|
|
||||||
- [ ] Call Home — map time slots 3/4 offsets; confirm `modem_power_relay_enabled`.
|
|
||||||
- [ ] RV55 DCD/DTR — newer RV55 firmware doesn't assert DCD by default; units don't resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred).
|
|
||||||
- [ ] **NULL-timestamp duplicate-row dedup.** A small handful of events (2 known on prod as of 2026-05-22) have `events.timestamp IS NULL` because the codec couldn't extract a timestamp from the binary footer. The `UNIQUE(serial, timestamp)` constraint doesn't fire on `NULL` (SQL semantics: `NULL ≠ NULL`), so every `--force` backfill INSERTs a new row instead of UPSERTing the existing one. Cleanup: a one-shot SQL query that keeps only the newest row per `(serial, blastware_filename)` and deletes the rest. Longer-term: extend the unique key to `(serial, COALESCE(timestamp, blastware_filename))` or reject inserts with NULL timestamp.
|
|
||||||
- [ ] **Histogram body sub-format with `byte[5] != 0`.** ~3 events on prod (`T190LD5Q.LD0H`, `O121L4L1.GU0H`) use a histogram body my walker doesn't recognize — the first block has `byte[5] = 0x01` or `0x07` instead of `0x00`, and the entire body lacks the `1e 0a 00 00` tail signature. Codec returns 0 valid blocks; their DB PVS comes from the bw_report ASCII overlay (which BW computed from the same binary, so the DB columns are correct). Only the `.h5` waveform plot is empty. Cracking the sub-format would unlock the plot. Needs binary+ASCII pairs from a few `byte[5]!=0` events; same RE approach as the K558 case.
|
|
||||||
- [ ] **Histogram body sub-format with `byte[5] == 0x00` but undecodable.** Observed 2026-05-28 on BE17353 (S353) events: `S353L4H2.FZ0H`, `S353L4H2.P00H`, `S353L4H3.7O0H`, `S353L4H3.E10H`. Body starts `00 00 00 01 0a 00 XX 00 ...` which LOOKS like a valid histogram block header (marker 0x000a at byte[4:6] ✓, byte[5]=0x00 normal-format ✓), but the walker finds zero data blocks across the whole body. Likely an extra header before the block stream OR a different tail signature than `1e 0a 00 00`. Smaller body lengths (1900-2100 bytes) suggest these may be short-recording histogram variants. Same operational impact as the byte[5]!=0 case: event ingests cleanly, DB peaks correct via bw_report overlay, only the chart is empty. Worth dumping a hex view of one body to diagnose.
|
|
||||||
- [ ] **Sensor-check waveform extraction from the BW binary.** BW's Event Report PDFs include a narrow panel on the right side of the waveform plot showing each channel's response to the sensor self-check signal (a damped sinusoid for geo, sawtooth-at-test-freq for mic). Our parser captures the test RESULTS (`test_freq_hz`, `test_ratio`, `test_amplitude_mv`, `test_results` pass/fail) and the PDF + modal display them as text — but BW's per-sample sensor-check waveform isn't accessible to us today. Two paths to add it: (a) RE the binary to find where the sensor-check samples are stored — could be a section before STRT, after the footer, or in a separate sub-record; protocol reference doesn't currently mention it. (b) If samples aren't in the binary, synthesize a representative waveform from the test parameters (damped sinusoid at `test_freq_hz` with damping from `test_ratio`). Path (a) is the honest answer; path (b) is decorative. Until either lands, the text-only sensor-check display in the report is fine.
|
|
||||||
|
|||||||
@@ -1,66 +0,0 @@
|
|||||||
# analysis/ — exploratory scripts for waveform-body RE
|
|
||||||
|
|
||||||
**These are scratch.** Run them, read them, copy them, but don't trust
|
|
||||||
them as documentation. When a finding is verified it gets promoted
|
|
||||||
to `minimateplus/waveform_codec.py` and `tests/test_waveform_codec.py`;
|
|
||||||
when it's wrong it stays here as a fossil.
|
|
||||||
|
|
||||||
Authoritative status lives in:
|
|
||||||
|
|
||||||
- `docs/waveform_codec_re_status.md` (current truth, working note)
|
|
||||||
- `minimateplus/waveform_codec.py` (verified implementation + docstring)
|
|
||||||
- `tests/test_waveform_codec.py` (regression locks against fixtures)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Still useful
|
|
||||||
|
|
||||||
| File | What it does |
|
|
||||||
|---|---|
|
|
||||||
| `load_bundle.py` | Fixture loader. Parses BW binary + ASCII TXT into a `Bundle` dataclass with samples, metadata, body bytes. Used by most other scripts here. |
|
|
||||||
| `verify_tran.py` | Verifies `decode_tran_initial` against fixture ground truth across all events. Useful when you change the decoder and want a quick sanity check. |
|
|
||||||
| `inspect_5_11.py` | Inspects the 5-11-26 high-amplitude bundle's body structure, prints metadata, peaks, and block counts. |
|
|
||||||
| `walk_5_11.py` | Walks blocks for the 5-11-26 bundle and prints offset/tag/length/data. |
|
|
||||||
| `seg1_blocks.py` | Dumps all blocks in segment 1 of each event. The starting point for cracking multi-segment Tran continuation. |
|
|
||||||
| `full_tran.py` | Multi-segment Tran decoder attempt (broken — diverges at sample ~512). Useful as a starting scaffold for the next experiment. |
|
|
||||||
| `multi_segment.py` | Earlier multi-segment attempt with different segment-header consumption strategies. Records what didn't work. |
|
|
||||||
| `test_rle.py` | Tests `00 NN` interpretation as zero-RLE with different divisor values. Documents how the RLE rule was confirmed. |
|
|
||||||
|
|
||||||
## Superseded — keep for archaeology
|
|
||||||
|
|
||||||
| File | Superseded by |
|
|
||||||
|---|---|
|
|
||||||
| `walk_v2.py` … `walk_v5.py` | `walk_v6.py` and ultimately `minimateplus/waveform_codec.walk_body`. Each version represents one round of refinement. Don't read in isolation — read the diff between them to see what was learned. |
|
|
||||||
| `walk_chunks.py` | `walk_v6.py` / production walker |
|
|
||||||
| `decode_v1.py` | First naive decoder attempt. Wrong but readable. |
|
|
||||||
|
|
||||||
## Pure exploration — read if curious
|
|
||||||
|
|
||||||
| File | What it explored |
|
|
||||||
|---|---|
|
|
||||||
| `inspect_body.py` | Byte-frequency stats per event. Established that bytes 0x00 / 0x10 dominate. |
|
|
||||||
| `find_blocks.py` | Searched for repeating 2-byte tag patterns. |
|
|
||||||
| `find_signal_runs.py` | Searched for stretches of bytes that "look like a smooth signal" (small inter-byte deltas). Found the `20 NN` literal blocks. |
|
|
||||||
| `dump_head.py`, `dump_trailer.py`, `dump_around.py` | Hex dumpers at various body positions. |
|
|
||||||
| `compare_cd.py` | Byte-diff between event-c and event-d (same length, similar signal). Used to identify structural vs data bytes. |
|
|
||||||
| `brute_force.py` | Tested 96 combinations of channel-permutation × nibble-order × sign-convention × init-from-header on the quiet bundle. All failed because the quiet bundle had T[0]=T[1]=0, making the preamble undetectable. |
|
|
||||||
| `try_nibbles.py`, `try_layouts.py` | Earlier channel-interleaving hypotheses. All wrong. |
|
|
||||||
| `test_tran_continue.py` | Test of "Tran continues uninterrupted across `30 04` blocks" hypothesis. Disproven. |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Adding new scripts
|
|
||||||
|
|
||||||
If you're picking up the codec work, feel free to add new scripts here.
|
|
||||||
Suggested conventions:
|
|
||||||
|
|
||||||
- Start the filename with what you're testing: `test_<hypothesis>.py`,
|
|
||||||
`verify_<piece>.py`, `inspect_<region>.py`.
|
|
||||||
- Print enough output that the reader can see exactly which events
|
|
||||||
match / diverge and where.
|
|
||||||
- When a finding is solid, move the verified logic to
|
|
||||||
`minimateplus/waveform_codec.py` and add a regression test in
|
|
||||||
`tests/test_waveform_codec.py` — don't leave the truth only in
|
|
||||||
this directory.
|
|
||||||
- If a script is fully superseded, leave it in place (don't delete) —
|
|
||||||
the fossil record is useful when re-evaluating hypotheses later.
|
|
||||||
@@ -1,93 +0,0 @@
|
|||||||
"""Brute-force test channel permutations / nibble orders on event-d (simplest signal)."""
|
|
||||||
import sys
|
|
||||||
import itertools
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
from minimateplus.waveform_codec import walk_body
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def decode(body, channel_perm, nibble_order, sign_mode, init_from_header):
|
|
||||||
"""Try one decoder configuration on event-d. Returns first 8 cumulative samples per channel."""
|
|
||||||
blocks = walk_body(body)
|
|
||||||
# Initial values from bytes [4:7] if init_from_header else 0
|
|
||||||
if init_from_header:
|
|
||||||
init = [body[4] if body[4] < 128 else body[4] - 256,
|
|
||||||
body[5] if body[5] < 128 else body[5] - 256,
|
|
||||||
body[6] if body[6] < 128 else body[6] - 256,
|
|
||||||
0]
|
|
||||||
else:
|
|
||||||
init = [0, 0, 0, 0]
|
|
||||||
cur = list(init)
|
|
||||||
out = [[init[0]], [init[1]], [init[2]], [init[3]]] # sample 0 = init
|
|
||||||
nibble_idx = 0 # within delta stream; channel = channel_perm[nibble_idx % 4]
|
|
||||||
|
|
||||||
# Walk only the 10 NN data blocks
|
|
||||||
for blk in blocks:
|
|
||||||
if blk.tag_hi != 0x10:
|
|
||||||
continue
|
|
||||||
for byte in blk.data:
|
|
||||||
if nibble_order == 'high_first':
|
|
||||||
nib1, nib2 = (byte >> 4) & 0xF, byte & 0xF
|
|
||||||
else:
|
|
||||||
nib1, nib2 = byte & 0xF, (byte >> 4) & 0xF
|
|
||||||
for nib in (nib1, nib2):
|
|
||||||
if sign_mode == 'signed':
|
|
||||||
delta = s4(nib)
|
|
||||||
else:
|
|
||||||
delta = nib
|
|
||||||
ch = channel_perm[nibble_idx % 4]
|
|
||||||
cur[ch] += delta
|
|
||||||
if (nibble_idx + 1) % 4 == 0:
|
|
||||||
out[0].append(cur[0])
|
|
||||||
out[1].append(cur[1])
|
|
||||||
out[2].append(cur[2])
|
|
||||||
out[3].append(cur[3])
|
|
||||||
nibble_idx += 1
|
|
||||||
if len(out[0]) >= 16:
|
|
||||||
return out
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def best_match(pred, truth, n=10):
|
|
||||||
"""Sum of squared differences in first n samples."""
|
|
||||||
n = min(n, len(pred), len(truth))
|
|
||||||
return sum((pred[i] - truth[i])**2 for i in range(n))
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
b = load_bundle("event-d")
|
|
||||||
# truth in 16-count units
|
|
||||||
tr = {ch: [round(v * 200) for v in b.samples[ch]] for ch in ("Tran", "Vert", "Long")}
|
|
||||||
|
|
||||||
print("Truth event-d first 10 samples:")
|
|
||||||
for ch in ("Tran", "Vert", "Long"):
|
|
||||||
print(f" {ch}: {tr[ch][:10]}")
|
|
||||||
|
|
||||||
# Test 96 combinations
|
|
||||||
best = []
|
|
||||||
for perm in itertools.permutations([0, 1, 2, 3]):
|
|
||||||
for nibble_order in ('high_first', 'low_first'):
|
|
||||||
for sign in ('signed', 'unsigned'):
|
|
||||||
for init_h in (False, True):
|
|
||||||
decoded = decode(b.body, perm, nibble_order, sign, init_h)
|
|
||||||
# Score as TVL channel-sum
|
|
||||||
score = sum(
|
|
||||||
best_match(decoded[i], tr[ch], n=10)
|
|
||||||
for i, ch in enumerate(("Tran", "Vert", "Long"))
|
|
||||||
if i < 3
|
|
||||||
)
|
|
||||||
label = f"perm={perm} nib={nibble_order[:1]} sign={sign[:3]} init={init_h}"
|
|
||||||
best.append((score, label, decoded))
|
|
||||||
|
|
||||||
best.sort(key=lambda x: x[0])
|
|
||||||
print(f"\nTop 10 configurations:")
|
|
||||||
for s, lbl, dec in best[:10]:
|
|
||||||
print(f" score={s:>5} {lbl} T={dec[0][:8]} V={dec[1][:8]} L={dec[2][:8]}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
"""Compare event-c and event-d (same N_samples) to find header vs data bytes."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
bc = load_bundle("event-c")
|
|
||||||
bd = load_bundle("event-d")
|
|
||||||
|
|
||||||
# Compare prefixes
|
|
||||||
nc, nd = len(bc.body), len(bd.body)
|
|
||||||
n = min(nc, nd)
|
|
||||||
diffs = []
|
|
||||||
for i in range(n):
|
|
||||||
if bc.body[i] != bd.body[i]:
|
|
||||||
diffs.append(i)
|
|
||||||
print(f"event-c body={nc}, event-d body={nd}")
|
|
||||||
print(f"Total diffs (first {n}): {len(diffs)}")
|
|
||||||
|
|
||||||
# Show common prefix
|
|
||||||
same_prefix = 0
|
|
||||||
for i in range(n):
|
|
||||||
if bc.body[i] == bd.body[i]:
|
|
||||||
same_prefix += 1
|
|
||||||
else:
|
|
||||||
break
|
|
||||||
print(f"Common prefix length: {same_prefix}")
|
|
||||||
print(f"event-c prefix: {bc.body[:same_prefix].hex(' ')}")
|
|
||||||
|
|
||||||
# Look for runs of common bytes
|
|
||||||
print(f"\nFirst 32 diff positions: {diffs[:32]}")
|
|
||||||
|
|
||||||
# Show the "diff fingerprint" of the first 100 bytes
|
|
||||||
print(f"\n pos c d")
|
|
||||||
for i in range(0, 100):
|
|
||||||
marker = " " if bc.body[i] == bd.body[i] else "*"
|
|
||||||
bd_b = bd.body[i] if i < nd else None
|
|
||||||
print(f" {i:>3} {bc.body[i]:02x}{marker} {bd_b:02x}" if bd_b is not None else f" {i:>3} {bc.body[i]:02x}{marker}")
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,99 +0,0 @@
|
|||||||
"""
|
|
||||||
Decoder v1: nibble-pair signed deltas in 10 NN blocks, 4-channel round-robin.
|
|
||||||
"""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def walk_blocks(body, start):
|
|
||||||
i = start
|
|
||||||
blocks = []
|
|
||||||
while i + 1 < len(body):
|
|
||||||
t0, t1 = body[i], body[i + 1]
|
|
||||||
if t0 == 0x10 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
|
|
||||||
length = t1 // 2 + 2
|
|
||||||
data = bytes(body[i + 2 : i + length])
|
|
||||||
blocks.append(("10", t1, data))
|
|
||||||
i += length
|
|
||||||
elif t0 == 0x20 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
|
|
||||||
length = t1 + 2
|
|
||||||
data = bytes(body[i + 2 : i + length])
|
|
||||||
blocks.append(("20", t1, data))
|
|
||||||
i += length
|
|
||||||
elif t0 == 0x00 and t1 % 4 == 0:
|
|
||||||
blocks.append(("00", t1, b""))
|
|
||||||
i += 2
|
|
||||||
elif t0 == 0x30 and t1 % 4 == 0 and 0 < t1 <= 0x10:
|
|
||||||
length = t1 * 4
|
|
||||||
data = bytes(body[i + 2 : i + length])
|
|
||||||
blocks.append(("30", t1, data))
|
|
||||||
i += length
|
|
||||||
elif t0 == 0x40 and t1 == 0x02:
|
|
||||||
length = 20
|
|
||||||
data = bytes(body[i + 2 : i + length])
|
|
||||||
blocks.append(("40", t1, data))
|
|
||||||
i += length
|
|
||||||
else:
|
|
||||||
blocks.append(("??", t0, bytes(body[i:i+8])))
|
|
||||||
break
|
|
||||||
return blocks
|
|
||||||
|
|
||||||
|
|
||||||
def decode_v1(body, start, n_samples):
|
|
||||||
"""Decode by accumulating nibble-pair deltas from all 10 NN blocks."""
|
|
||||||
blocks = walk_blocks(body, start)
|
|
||||||
# 4 channels: T, V, L, M
|
|
||||||
cur = [0, 0, 0, 0]
|
|
||||||
out = [[], [], [], []]
|
|
||||||
sample_index = 0 # how many sample-sets emitted
|
|
||||||
|
|
||||||
for typ, NN, data in blocks:
|
|
||||||
if typ == "10":
|
|
||||||
# 2 nibbles per byte, round-robin TVLM
|
|
||||||
for byte in data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
ch = sample_index % 4
|
|
||||||
cur[ch] += s4(nib)
|
|
||||||
out[ch].append(cur[ch])
|
|
||||||
sample_index = (sample_index + 1) // 4 * 4 + (sample_index + 1) % 4 # ?
|
|
||||||
sample_index += 1
|
|
||||||
# We emit per-nibble, but the structure is unclear
|
|
||||||
elif typ == "20":
|
|
||||||
# int8 absolute or delta?
|
|
||||||
for byte in data:
|
|
||||||
v = byte if byte < 128 else byte - 256
|
|
||||||
ch = sample_index % 4
|
|
||||||
cur[ch] = v # treat as absolute
|
|
||||||
out[ch].append(cur[ch])
|
|
||||||
sample_index += 1
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
b = load_bundle("event-c")
|
|
||||||
body = b.body
|
|
||||||
truth_T = [round(v * 200) for v in b.samples["Tran"]]
|
|
||||||
truth_V = [round(v * 200) for v in b.samples["Vert"]]
|
|
||||||
truth_L = [round(v * 200) for v in b.samples["Long"]]
|
|
||||||
|
|
||||||
# Find start
|
|
||||||
for s in range(15):
|
|
||||||
if body[s] == 0x10 and body[s+1] % 4 == 0 and 0 < body[s+1] <= 0xFC:
|
|
||||||
start = s
|
|
||||||
break
|
|
||||||
|
|
||||||
blocks = walk_blocks(body, start)
|
|
||||||
# Print block-by-block what's in each
|
|
||||||
print(f"Total blocks: {len(blocks)}")
|
|
||||||
bytes_processed = 0
|
|
||||||
for typ, NN, data in blocks[:30]:
|
|
||||||
print(f" type={typ} NN=0x{NN:02x} data_len={len(data)} data_hex={data[:32].hex(' ')}{'...' if len(data) > 32 else ''}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
"""Dump body bytes around a specific offset."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def dump_around(name: str, center: int, radius: int = 96):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
start = max(0, center - radius)
|
|
||||||
end = min(len(body), center + radius)
|
|
||||||
print(f"\n=== {name} body[{start}:{end}] (full body={len(body)}) ===")
|
|
||||||
for i in range(start, end, 32):
|
|
||||||
row = body[i:i+32]
|
|
||||||
marker = " <-- center" if i <= center < i+32 else ""
|
|
||||||
print(f" +{i:>5} {row.hex(' ')}{marker}")
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
# Look at the trailer transitions
|
|
||||||
trailer_starts = {"event-a": 7047, "event-b": 6475, "event-c": 4043, "event-d": 3941}
|
|
||||||
for name, off in trailer_starts.items():
|
|
||||||
dump_around(name, off, 96)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
"""Dump the START of each body in 32-byte rows."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-a", "event-c"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
print(f"\n=== {name} body[0:512] (full body={len(body)}, samples={len(b.samples['Tran'])}) ===")
|
|
||||||
for i in range(0, min(512, len(body)), 32):
|
|
||||||
row = body[i:i+32]
|
|
||||||
print(f" +{i:>5} {row.hex(' ')}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
"""Dump body bytes split into 32-byte rows starting from `start_offset`."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def dump(body: bytes, name: str, start: int, n_rows: int = 30):
|
|
||||||
print(f"\n=== {name} body[{start}:] (full body={len(body)}) ===")
|
|
||||||
end = min(start + 32 * n_rows, len(body))
|
|
||||||
for i in range(start, end, 32):
|
|
||||||
row = body[i:i+32]
|
|
||||||
print(f" +{i:>5} {row.hex(' ')}")
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-a", "event-b", "event-c", "event-d"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
# Print the LAST ~600 bytes of the body to see the tail structure
|
|
||||||
start = max(0, len(b.body) - 32 * 12)
|
|
||||||
dump(b.body, name, start, 12)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
"""Search for structural repetition in the body bytes."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def find_pattern_offsets(body: bytes, pattern: bytes, max_count=20):
|
|
||||||
out = []
|
|
||||||
i = 0
|
|
||||||
while True:
|
|
||||||
i = body.find(pattern, i)
|
|
||||||
if i < 0:
|
|
||||||
break
|
|
||||||
out.append(i)
|
|
||||||
i += 1
|
|
||||||
if len(out) >= max_count:
|
|
||||||
break
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-a", "event-b", "event-c", "event-d"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
print(f"\n=== {name} (body={len(body)}, N_samples={len(b.samples['Tran'])}) ===")
|
|
||||||
|
|
||||||
# Try to find repeating substructures (look for 4-byte 0x10-prefixed markers)
|
|
||||||
for prefix in [b"\x10\x10", b"\x10\x04", b"\x10\x08", b"\x10\x0c", b"\x10\x18",
|
|
||||||
b"\x10\x14", b"\x10\x20", b"\x10\x40", b"\x10\x80", b"\x10\x00",
|
|
||||||
b"\x10\x01", b"\x10\x03", b"\x10\xf0", b"\xf1\x10", b"\x00\x10",
|
|
||||||
b"\x40\x02", b"\x20\x04", b"\x30\x04", b"\x30\x08", b"\x00\x1a"]:
|
|
||||||
offs = find_pattern_offsets(body, prefix, max_count=200)
|
|
||||||
if 1 <= len(offs) <= 1000:
|
|
||||||
# Print first 10 offsets
|
|
||||||
first = offs[:6]
|
|
||||||
last = offs[-3:]
|
|
||||||
print(f" '{prefix.hex()}' x{len(offs):>4} first={first} last={last}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,34 +0,0 @@
|
|||||||
"""Find body byte ranges that look like absolute int8 sample data (smooth waveform)."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def looks_like_smooth_int8(buf):
|
|
||||||
"""Convert bytes to int8 and check if successive deltas are small (waveform-like)."""
|
|
||||||
if len(buf) < 8:
|
|
||||||
return 0.0
|
|
||||||
vals = [b if b < 128 else b - 256 for b in buf]
|
|
||||||
diffs = [abs(vals[i+1] - vals[i]) for i in range(len(vals)-1)]
|
|
||||||
avg_diff = sum(diffs) / len(diffs)
|
|
||||||
return avg_diff
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-a", "event-c"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
# Scan with sliding window of 64 bytes; find segments where the bytes look like a smooth wave
|
|
||||||
win = 64
|
|
||||||
scores = []
|
|
||||||
for i in range(len(body) - win):
|
|
||||||
scores.append((i, looks_like_smooth_int8(body[i:i+win])))
|
|
||||||
# Lowest avg_diff means smoothest
|
|
||||||
scores.sort(key=lambda x: x[1])
|
|
||||||
print(f"\n=== {name} (body={len(body)}) — smoothest 10 windows ===")
|
|
||||||
for off, s in scores[:10]:
|
|
||||||
print(f" +{off:>5} avg_diff={s:.2f} bytes={body[off:off+24].hex(' ')}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,76 +0,0 @@
|
|||||||
"""Full Tran decoder: continues across segment headers using T_delta from header bytes [0:2]."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def i8(b):
|
|
||||||
return b if b < 128 else b - 256
|
|
||||||
|
|
||||||
|
|
||||||
def decode_full_tran(body):
|
|
||||||
if len(body) < 7 or body[0:3] != b"\x00\x02\x00":
|
|
||||||
return None
|
|
||||||
T0 = int.from_bytes(body[3:5], "big", signed=True)
|
|
||||||
T1 = int.from_bytes(body[5:7], "big", signed=True)
|
|
||||||
|
|
||||||
i = 7
|
|
||||||
while i + 1 < len(body) and body[i] not in (0x00, 0x10, 0x20, 0x30, 0x40):
|
|
||||||
i += 1
|
|
||||||
|
|
||||||
blocks = walk_body(body, i)
|
|
||||||
T = [T0, T1]
|
|
||||||
cur = T1
|
|
||||||
for blk in blocks:
|
|
||||||
if blk.tag_hi == 0x40:
|
|
||||||
# Segment header carries 2 T deltas (int16 BE each) at bytes [0:2] and [2:4]
|
|
||||||
if len(blk.data) >= 4:
|
|
||||||
delta1 = int.from_bytes(blk.data[0:2], "big", signed=True)
|
|
||||||
cur += delta1
|
|
||||||
T.append(cur)
|
|
||||||
delta2 = int.from_bytes(blk.data[2:4], "big", signed=True)
|
|
||||||
cur += delta2
|
|
||||||
T.append(cur)
|
|
||||||
elif blk.tag_hi == 0x10:
|
|
||||||
for byte in blk.data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
cur += s4(nib)
|
|
||||||
T.append(cur)
|
|
||||||
elif blk.tag_hi == 0x20:
|
|
||||||
for byte in blk.data:
|
|
||||||
cur += i8(byte)
|
|
||||||
T.append(cur)
|
|
||||||
elif blk.tag_hi == 0x00:
|
|
||||||
for _ in range(blk.tag_lo):
|
|
||||||
T.append(cur)
|
|
||||||
# 30 NN: skip for now
|
|
||||||
return T
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1L.V70", "M529LL1L.JQ0", "M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
|
|
||||||
path = f"tests/fixtures/5-11-26/{stem}"
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
body = f.read()[43:-26]
|
|
||||||
_, samples = _parse_txt(path + ".TXT")
|
|
||||||
truth_T = [round(v*200) for v in samples["Tran"]]
|
|
||||||
n_truth = len(truth_T)
|
|
||||||
|
|
||||||
decoded = decode_full_tran(body)
|
|
||||||
n = min(len(decoded), n_truth)
|
|
||||||
matches = sum(1 for i in range(n) if decoded[i] == truth_T[i])
|
|
||||||
div_at = -1
|
|
||||||
for i in range(n):
|
|
||||||
if decoded[i] != truth_T[i]:
|
|
||||||
div_at = i
|
|
||||||
break
|
|
||||||
print(f"{stem}: decoded={len(decoded)}, truth={n_truth}, matches={matches}/{n}, first div={div_at}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,50 +0,0 @@
|
|||||||
"""Quick inspection of the new high-amplitude events."""
|
|
||||||
import os, re, sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
ROOT = "tests/fixtures/5-11-26"
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
|
|
||||||
bin_path = os.path.join(ROOT, stem)
|
|
||||||
txt_path = bin_path + ".TXT"
|
|
||||||
with open(bin_path, "rb") as f:
|
|
||||||
raw = f.read()
|
|
||||||
body = raw[43:-26]
|
|
||||||
meta, samples = _parse_txt(txt_path)
|
|
||||||
n = len(samples["Tran"])
|
|
||||||
|
|
||||||
print(f"\n=== {stem} ===")
|
|
||||||
print(f" file={len(raw)}, body={len(body)}, N_samples={n}")
|
|
||||||
print(f" rectime={meta.get('Record Time')} pretrig={meta.get('Pre-trigger Length')}")
|
|
||||||
print(f" PPV(T,V,L)={meta.get('Tran PPV')} / {meta.get('Vert PPV')} / {meta.get('Long PPV')}")
|
|
||||||
# Show first few non-trivial samples
|
|
||||||
print(f" First 5 truth samples (in/s):")
|
|
||||||
for i in range(5):
|
|
||||||
print(f" T={samples['Tran'][i]:8.3f} V={samples['Vert'][i]:8.3f} "
|
|
||||||
f"L={samples['Long'][i]:8.3f} M={samples['MicL'][i]:8.3f}")
|
|
||||||
# Peak sample positions
|
|
||||||
for ch in ("Tran", "Vert", "Long"):
|
|
||||||
vals = samples[ch]
|
|
||||||
peak_i = max(range(n), key=lambda i: abs(vals[i]))
|
|
||||||
print(f" {ch}: peak {vals[peak_i]:.3f} at sample {peak_i} (t={peak_i/1024:.3f}s)")
|
|
||||||
# Body structure
|
|
||||||
start = find_data_start(body)
|
|
||||||
blocks = walk_body(body, start)
|
|
||||||
types = {}
|
|
||||||
for b in blocks:
|
|
||||||
types[b.tag_hi] = types.get(b.tag_hi, 0) + 1
|
|
||||||
print(f" body start={start}, total blocks walked: {len(blocks)}")
|
|
||||||
print(f" block tag counts: {types}")
|
|
||||||
# How far the walker got
|
|
||||||
if blocks:
|
|
||||||
last = blocks[-1]
|
|
||||||
walked = last.offset + last.length
|
|
||||||
print(f" walker stopped at offset {walked}/{len(body)} ({100*walked/len(body):.0f}%)")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,23 +0,0 @@
|
|||||||
"""Print raw body hex + byte-distribution stats for one event."""
|
|
||||||
from collections import Counter
|
|
||||||
import sys
|
|
||||||
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-a", "event-b", "event-c", "event-d"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
print(f"\n=== {name} ({len(body)} body bytes) ===")
|
|
||||||
print(f" STRT: {b.strt.hex()}")
|
|
||||||
print(f" body[0:64]: {body[:64].hex()}")
|
|
||||||
print(f" body[64:128]: {body[64:128].hex()}")
|
|
||||||
print(f" body[-32:]: {body[-32:].hex()}")
|
|
||||||
cnt = Counter(body)
|
|
||||||
print(f" top 16 bytes: {[(f'0x{k:02x}', f'{v/len(body):.2%}') for k,v in cnt.most_common(16)]}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,144 +0,0 @@
|
|||||||
"""
|
|
||||||
load_bundle.py — extract body bytes from BW binary + parse sample columns from TXT.
|
|
||||||
|
|
||||||
Used by the codec reverse-engineering scripts in this directory.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
from dataclasses import dataclass
|
|
||||||
|
|
||||||
|
|
||||||
BUNDLE_ROOT = os.path.join(
|
|
||||||
os.path.dirname(__file__), "..", "tests", "fixtures", "decode-re-5-8-26"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class Bundle:
|
|
||||||
name: str
|
|
||||||
bin_path: str
|
|
||||||
txt_path: str
|
|
||||||
bin: bytes
|
|
||||||
body: bytes # bytes between STRT (43) and footer (last 26)
|
|
||||||
strt: bytes # 21-byte STRT record
|
|
||||||
samples: dict # {"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}
|
|
||||||
sample_rate: int
|
|
||||||
rectime_sec: float
|
|
||||||
pretrig_sec: float
|
|
||||||
geo_range_ips: float
|
|
||||||
ppv: dict # {"Tran": float, "Vert": float, "Long": float}
|
|
||||||
mic_pspl: float
|
|
||||||
serial: str
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_txt(path: str) -> dict:
|
|
||||||
with open(path, "r", encoding="utf-8", errors="replace") as f:
|
|
||||||
text = f.read()
|
|
||||||
|
|
||||||
meta = {}
|
|
||||||
samples = {"Tran": [], "Vert": [], "Long": [], "MicL": []}
|
|
||||||
|
|
||||||
# Find header line that starts the columns ("Tran Vert Long MicL").
|
|
||||||
# Then every line after is sample data (4 tab-separated floats).
|
|
||||||
lines = text.splitlines()
|
|
||||||
header_idx = None
|
|
||||||
for i, line in enumerate(lines):
|
|
||||||
if "Tran" in line and "Vert" in line and "Long" in line and "MicL" in line:
|
|
||||||
# The columns header. Sample lines start a few lines later.
|
|
||||||
header_idx = i
|
|
||||||
break
|
|
||||||
if header_idx is None:
|
|
||||||
raise ValueError(f"no Tran/Vert/Long/MicL header in {path}")
|
|
||||||
|
|
||||||
# Parse meta — quoted lines with "Field : value"
|
|
||||||
for line in lines[:header_idx]:
|
|
||||||
m = re.match(r'^"([^"]+)\s*:\s*([^"]*)"', line.strip())
|
|
||||||
if m:
|
|
||||||
k, v = m.group(1).strip(), m.group(2).strip()
|
|
||||||
meta[k] = v
|
|
||||||
|
|
||||||
# Parse samples
|
|
||||||
for line in lines[header_idx + 1 :]:
|
|
||||||
line = line.strip()
|
|
||||||
if not line:
|
|
||||||
continue
|
|
||||||
parts = re.split(r"\s+", line)
|
|
||||||
if len(parts) < 4:
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
t = float(parts[0])
|
|
||||||
v = float(parts[1])
|
|
||||||
l = float(parts[2])
|
|
||||||
m = float(parts[3])
|
|
||||||
except ValueError:
|
|
||||||
continue
|
|
||||||
samples["Tran"].append(t)
|
|
||||||
samples["Vert"].append(v)
|
|
||||||
samples["Long"].append(l)
|
|
||||||
samples["MicL"].append(m)
|
|
||||||
|
|
||||||
return meta, samples
|
|
||||||
|
|
||||||
|
|
||||||
def load_bundle(name: str) -> Bundle:
|
|
||||||
folder = os.path.join(BUNDLE_ROOT, name)
|
|
||||||
files = os.listdir(folder)
|
|
||||||
bin_name = next(f for f in files if not f.endswith(".TXT"))
|
|
||||||
txt_name = next(f for f in files if f.endswith(".TXT"))
|
|
||||||
|
|
||||||
bin_path = os.path.join(folder, bin_name)
|
|
||||||
txt_path = os.path.join(folder, txt_name)
|
|
||||||
|
|
||||||
with open(bin_path, "rb") as f:
|
|
||||||
binary = f.read()
|
|
||||||
|
|
||||||
# Header is 22 bytes; STRT at [22:43]; footer at last 26 bytes.
|
|
||||||
strt = binary[22:43]
|
|
||||||
body = binary[43:-26]
|
|
||||||
|
|
||||||
meta, samples = _parse_txt(txt_path)
|
|
||||||
|
|
||||||
sample_rate = int(re.search(r"(\d+)", meta.get("Sample Rate", "1024")).group(1))
|
|
||||||
rectime_sec = float(re.search(r"([\d.]+)", meta.get("Record Time", "3.0")).group(1))
|
|
||||||
pretrig_sec = float(re.search(r"-?[\d.]+", meta.get("Pre-trigger Length", "0")).group(0))
|
|
||||||
geo_range_ips = float(re.search(r"([\d.]+)", meta.get("Geo Range", "10.0")).group(1))
|
|
||||||
serial = meta.get("Serial Number", "").strip()
|
|
||||||
|
|
||||||
def _f(s):
|
|
||||||
return float(re.search(r"-?[\d.]+", s).group(0))
|
|
||||||
|
|
||||||
ppv = {
|
|
||||||
"Tran": _f(meta.get("Tran PPV", "0")),
|
|
||||||
"Vert": _f(meta.get("Vert PPV", "0")),
|
|
||||||
"Long": _f(meta.get("Long PPV", "0")),
|
|
||||||
}
|
|
||||||
mic_pspl = _f(meta.get("MicL PSPL", "0"))
|
|
||||||
|
|
||||||
return Bundle(
|
|
||||||
name=name,
|
|
||||||
bin_path=bin_path,
|
|
||||||
txt_path=txt_path,
|
|
||||||
bin=binary,
|
|
||||||
body=body,
|
|
||||||
strt=strt,
|
|
||||||
samples=samples,
|
|
||||||
sample_rate=sample_rate,
|
|
||||||
rectime_sec=rectime_sec,
|
|
||||||
pretrig_sec=pretrig_sec,
|
|
||||||
geo_range_ips=geo_range_ips,
|
|
||||||
ppv=ppv,
|
|
||||||
mic_pspl=mic_pspl,
|
|
||||||
serial=serial,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
for name in ("event-a", "event-b", "event-c", "event-d"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
n = len(b.samples["Tran"])
|
|
||||||
print(f"{name}: body={len(b.body):>6} N_samples={n} rate={b.sample_rate} "
|
|
||||||
f"rectime={b.rectime_sec} pretrig={b.pretrig_sec} range={b.geo_range_ips} "
|
|
||||||
f"PPV(T,V,L)={b.ppv['Tran']:.3f},{b.ppv['Vert']:.3f},{b.ppv['Long']:.3f} "
|
|
||||||
f"MicL={b.mic_pspl}")
|
|
||||||
@@ -1,81 +0,0 @@
|
|||||||
"""Decode Tran across multiple segments by resetting at 40 02 headers."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def i8(b):
|
|
||||||
return b if b < 128 else b - 256
|
|
||||||
|
|
||||||
|
|
||||||
def decode_full_tran(body):
|
|
||||||
"""Decode all Tran samples in the body, walking through segments."""
|
|
||||||
if len(body) < 7 or body[0:3] != b"\x00\x02\x00":
|
|
||||||
return None
|
|
||||||
T0 = int.from_bytes(body[3:5], "big", signed=True)
|
|
||||||
T1 = int.from_bytes(body[5:7], "big", signed=True)
|
|
||||||
|
|
||||||
# Locate first tag
|
|
||||||
i = 7
|
|
||||||
while i + 1 < len(body) and body[i] not in (0x00, 0x10, 0x20, 0x30, 0x40):
|
|
||||||
i += 1
|
|
||||||
|
|
||||||
blocks = walk_body(body, i)
|
|
||||||
T = [T0, T1]
|
|
||||||
cur = T1
|
|
||||||
for bi, blk in enumerate(blocks):
|
|
||||||
if blk.tag_hi == 0x40:
|
|
||||||
# Segment header — try interpreting bytes [0:2] as new T anchor
|
|
||||||
if len(blk.data) >= 2:
|
|
||||||
new_anchor = int.from_bytes(blk.data[0:2], "big", signed=True)
|
|
||||||
# The next sample IS this anchor value, NOT a delta from cur.
|
|
||||||
T.append(new_anchor)
|
|
||||||
cur = new_anchor
|
|
||||||
elif blk.tag_hi == 0x10:
|
|
||||||
for byte in blk.data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
cur += s4(nib)
|
|
||||||
T.append(cur)
|
|
||||||
elif blk.tag_hi == 0x20:
|
|
||||||
for byte in blk.data:
|
|
||||||
cur += i8(byte)
|
|
||||||
T.append(cur)
|
|
||||||
elif blk.tag_hi == 0x00:
|
|
||||||
# RLE: append NN zero deltas
|
|
||||||
for _ in range(blk.tag_lo):
|
|
||||||
T.append(cur)
|
|
||||||
# 30 NN: skip
|
|
||||||
return T
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1L.V70", "M529LL1L.JQ0", "M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
|
|
||||||
path = f"tests/fixtures/5-11-26/{stem}"
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
body = f.read()[43:-26]
|
|
||||||
_, samples = _parse_txt(path + ".TXT")
|
|
||||||
truth_T = [round(v*200) for v in samples["Tran"]]
|
|
||||||
n_truth = len(truth_T)
|
|
||||||
|
|
||||||
decoded = decode_full_tran(body)
|
|
||||||
n = min(len(decoded), n_truth)
|
|
||||||
matches = sum(1 for i in range(n) if decoded[i] == truth_T[i])
|
|
||||||
# Find first divergence
|
|
||||||
div_at = -1
|
|
||||||
for i in range(n):
|
|
||||||
if decoded[i] != truth_T[i]:
|
|
||||||
div_at = i
|
|
||||||
break
|
|
||||||
print(f"{stem}: decoded={len(decoded)}, truth={n_truth}, matches={matches}/{n}, first div={div_at}")
|
|
||||||
if div_at >= 0 and div_at < 30:
|
|
||||||
print(f" truth around div [{max(0,div_at-3)}:{div_at+8}]: {truth_T[max(0,div_at-3):div_at+8]}")
|
|
||||||
print(f" pred around div [{max(0,div_at-3)}:{div_at+8}]: {decoded[max(0,div_at-3):div_at+8]}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,28 +0,0 @@
|
|||||||
"""Dump all blocks in segment 1 of each event with their data."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1A.SP0", "M529LL1L.JQ0", "M529LL1L.V70"):
|
|
||||||
path = f"tests/fixtures/5-11-26/{stem}"
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
body = f.read()[43:-26]
|
|
||||||
blocks = walk_body(body, find_data_start(body))
|
|
||||||
|
|
||||||
# Find segment 1 (between first and second 40 02)
|
|
||||||
seg40_indices = [i for i, b in enumerate(blocks) if b.tag_hi == 0x40]
|
|
||||||
if len(seg40_indices) < 2:
|
|
||||||
print(f"\n{stem}: only {len(seg40_indices)} segment headers found")
|
|
||||||
seg1_blocks = blocks[seg40_indices[0]:] if seg40_indices else []
|
|
||||||
else:
|
|
||||||
seg1_blocks = blocks[seg40_indices[0]:seg40_indices[1]+1]
|
|
||||||
print(f"\n=== {stem} segment 1 ({len(seg1_blocks)} blocks) ===")
|
|
||||||
for b in seg1_blocks[:25]:
|
|
||||||
tag = f"{b.tag_hi:02x}{b.tag_lo:02x}"
|
|
||||||
print(f" off={b.offset:>5} {tag} NN=0x{b.tag_lo:02x}({b.tag_lo:>3}) len={b.length:>3} data={b.data[:16].hex(' ')}{'...' if len(b.data)>16 else ''}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,195 +0,0 @@
|
|||||||
"""Test 12-bit signed packed deltas hypothesis for 30 NN blocks across all loud events.
|
|
||||||
|
|
||||||
For each 30 NN block in each event, identify what samples it should cover
|
|
||||||
(based on the cumulative delta count up to that point) and compare the
|
|
||||||
truth deltas against various 12-bit packing schemes.
|
|
||||||
"""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
|
|
||||||
CHANNEL_ORDER = ["Vert", "Long", "MicL", "Tran"] # rotation after initial T
|
|
||||||
|
|
||||||
|
|
||||||
def s12(v):
|
|
||||||
"""Sign-extend a 12-bit unsigned value to signed int."""
|
|
||||||
return v if v < 0x800 else v - 0x1000
|
|
||||||
|
|
||||||
|
|
||||||
def unpack_12bit_be(data):
|
|
||||||
"""4 deltas in 6 bytes, BE order: byte[0:1.5], byte[1.5:3], byte[3:4.5], byte[4.5:6]."""
|
|
||||||
# bits 0..47 (MSB-first), split into 4 × 12-bit
|
|
||||||
val = int.from_bytes(data, "big")
|
|
||||||
out = []
|
|
||||||
for i in range(4):
|
|
||||||
d = (val >> (12 * (3 - i))) & 0xFFF
|
|
||||||
out.append(s12(d))
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def unpack_12bit_le(data):
|
|
||||||
"""4 deltas in 6 bytes, LE order: bytes packed as 2 × 24-bit groups."""
|
|
||||||
out = []
|
|
||||||
# First 3 bytes contain 2 deltas
|
|
||||||
b0, b1, b2 = data[0], data[1], data[2]
|
|
||||||
d0 = b0 | ((b1 & 0x0F) << 8)
|
|
||||||
d1 = (b1 >> 4) | (b2 << 4)
|
|
||||||
out.append(s12(d0))
|
|
||||||
out.append(s12(d1))
|
|
||||||
# Next 3 bytes contain 2 more deltas
|
|
||||||
b3, b4, b5 = data[3], data[4], data[5]
|
|
||||||
d2 = b3 | ((b4 & 0x0F) << 8)
|
|
||||||
d3 = (b4 >> 4) | (b5 << 4)
|
|
||||||
out.append(s12(d2))
|
|
||||||
out.append(s12(d3))
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def unpack_12bit_be_per_triplet(data):
|
|
||||||
"""4 deltas as 2 triplets of (high4, low8) BE within each 3-byte group."""
|
|
||||||
out = []
|
|
||||||
b0, b1, b2 = data[0], data[1], data[2]
|
|
||||||
d0 = (b0 << 4) | (b1 >> 4)
|
|
||||||
d1 = ((b1 & 0x0F) << 8) | b2
|
|
||||||
out.append(s12(d0))
|
|
||||||
out.append(s12(d1))
|
|
||||||
b3, b4, b5 = data[3], data[4], data[5]
|
|
||||||
d2 = (b3 << 4) | (b4 >> 4)
|
|
||||||
d3 = ((b4 & 0x0F) << 8) | b5
|
|
||||||
out.append(s12(d2))
|
|
||||||
out.append(s12(d3))
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def truth_deltas_for_block(blocks, block_idx, event_truth, channel):
|
|
||||||
"""For a 30 NN block at block_idx, determine which samples it covers and
|
|
||||||
return the truth deltas for those samples.
|
|
||||||
|
|
||||||
Walks through all blocks before block_idx (within the same segment) and
|
|
||||||
counts how many deltas have been emitted for *channel*, starting from the
|
|
||||||
segment's anchor pair.
|
|
||||||
"""
|
|
||||||
# Find the segment header that contains this block.
|
|
||||||
seg_header_idx = None
|
|
||||||
for j in range(block_idx, -1, -1):
|
|
||||||
if blocks[j].tag_hi == 0x40:
|
|
||||||
seg_header_idx = j
|
|
||||||
break
|
|
||||||
if seg_header_idx is None:
|
|
||||||
# block is in the initial T segment; samples count from sample 2.
|
|
||||||
first_sample_in_segment = 2
|
|
||||||
else:
|
|
||||||
# Anchor pair covers samples [N, N+1] for some N. Subsequent deltas
|
|
||||||
# are samples [N+2, N+2+1, ...]. We don't actually need to know N
|
|
||||||
# for this test — just the relative position within the segment.
|
|
||||||
first_sample_in_segment = 2 # anchor=0,1; deltas start at 2
|
|
||||||
|
|
||||||
# Count deltas from segment-data start to block_idx.
|
|
||||||
delta_count = 0
|
|
||||||
start_block = seg_header_idx + 1 if seg_header_idx is not None else 0
|
|
||||||
for j in range(start_block, block_idx):
|
|
||||||
blk = blocks[j]
|
|
||||||
if blk.tag_hi == 0x10:
|
|
||||||
delta_count += blk.tag_lo # NN nibbles = NN deltas
|
|
||||||
elif blk.tag_hi == 0x20:
|
|
||||||
delta_count += blk.tag_lo # NN int8 deltas
|
|
||||||
elif blk.tag_hi == 0x00:
|
|
||||||
delta_count += blk.tag_lo # RLE zero deltas
|
|
||||||
# Now the 30 NN block carries NN deltas.
|
|
||||||
nn = blocks[block_idx].tag_lo
|
|
||||||
# First sample affected: segment first_sample + delta_count.
|
|
||||||
# But we ALSO need to know which segment this is, since the segment maps
|
|
||||||
# to a specific channel and a specific starting absolute sample index.
|
|
||||||
return first_sample_in_segment + delta_count, nn
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1A.SP0", "M529LL1L.JQ0", "M529LL1L.V70",
|
|
||||||
"M529LL1A.SS0", "M529LL1A.SV0"):
|
|
||||||
path = f"tests/fixtures/5-11-26/{stem}"
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
body = f.read()[43:-26]
|
|
||||||
_, samples = _parse_txt(path + ".TXT")
|
|
||||||
blocks = walk_body(body, find_data_start(body))
|
|
||||||
seg_idx = [i for i, b in enumerate(blocks) if b.tag_hi == 0x40]
|
|
||||||
|
|
||||||
# Find all 30 NN blocks in DATA section (not trailer).
|
|
||||||
thirty_blocks = []
|
|
||||||
for bi, b in enumerate(blocks):
|
|
||||||
if b.tag_hi != 0x30:
|
|
||||||
continue
|
|
||||||
# Determine which segment this is in
|
|
||||||
seg_num = None
|
|
||||||
for k, hi in enumerate(seg_idx):
|
|
||||||
next_hi = seg_idx[k + 1] if k + 1 < len(seg_idx) else len(blocks)
|
|
||||||
if hi < bi < next_hi:
|
|
||||||
seg_num = k
|
|
||||||
break
|
|
||||||
if seg_num is None and seg_idx and bi < seg_idx[0]:
|
|
||||||
seg_num = -1 # initial T segment
|
|
||||||
thirty_blocks.append((bi, b, seg_num))
|
|
||||||
|
|
||||||
if not thirty_blocks:
|
|
||||||
continue
|
|
||||||
|
|
||||||
print(f"\n=== {stem} ===")
|
|
||||||
for bi, b, seg_num in thirty_blocks:
|
|
||||||
# Channel for this segment
|
|
||||||
if seg_num == -1:
|
|
||||||
channel = "Tran"
|
|
||||||
seg_label = "initial T"
|
|
||||||
else:
|
|
||||||
channel = CHANNEL_ORDER[seg_num % 4]
|
|
||||||
seg_label = f"seg {seg_num}"
|
|
||||||
|
|
||||||
# Count deltas before this block within the same segment.
|
|
||||||
seg_header_idx = seg_idx[seg_num] if seg_num >= 0 else -1
|
|
||||||
start_block = seg_header_idx + 1 if seg_header_idx >= 0 else 0
|
|
||||||
delta_count = 0
|
|
||||||
for j in range(start_block, bi):
|
|
||||||
blk = blocks[j]
|
|
||||||
if blk.tag_hi in (0x10, 0x20, 0x00):
|
|
||||||
delta_count += blk.tag_lo
|
|
||||||
|
|
||||||
# First sample this 30 NN block affects (within the segment)
|
|
||||||
# = anchor positions + delta_count + 2 (since anchor pair was samples 0,1)
|
|
||||||
# But the segment's first absolute sample index in the channel is
|
|
||||||
# (seg_num // 4) * 512 (approximately) if segment 0 is the first V seg.
|
|
||||||
cycle = (seg_num // 4) if seg_num >= 0 else 0
|
|
||||||
base = cycle * 512 + 2 # +2 for anchor pair
|
|
||||||
sample_idx = base + delta_count
|
|
||||||
truth_ch = [round(v * 200) for v in samples[channel]]
|
|
||||||
nn = b.tag_lo
|
|
||||||
|
|
||||||
if sample_idx + nn >= len(truth_ch):
|
|
||||||
print(f" block @ {b.offset} ({seg_label} {channel}): out of truth range")
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Get the previous sample so we can compute truth deltas
|
|
||||||
if sample_idx == 0:
|
|
||||||
prev = 0
|
|
||||||
else:
|
|
||||||
prev = truth_ch[sample_idx - 1]
|
|
||||||
truth_deltas = []
|
|
||||||
for k in range(nn):
|
|
||||||
truth_deltas.append(truth_ch[sample_idx + k] - (prev if k == 0 else truth_ch[sample_idx + k - 1]))
|
|
||||||
|
|
||||||
# Try each packing
|
|
||||||
schemes = [
|
|
||||||
("12-bit BE contiguous", unpack_12bit_be(b.data)),
|
|
||||||
("12-bit LE per-triplet", unpack_12bit_le(b.data)),
|
|
||||||
("12-bit BE per-triplet", unpack_12bit_be_per_triplet(b.data)),
|
|
||||||
]
|
|
||||||
print(f" block @ {b.offset:>5} ({seg_label} {channel}, samples {sample_idx}..{sample_idx+nn-1}):")
|
|
||||||
print(f" data: {b.data.hex(' ')}")
|
|
||||||
print(f" truth: {truth_deltas}")
|
|
||||||
for name, pred in schemes:
|
|
||||||
match = "✓" if pred == truth_deltas else " "
|
|
||||||
n_match = sum(1 for x, y in zip(pred, truth_deltas) if x == y)
|
|
||||||
print(f" {match}{n_match}/4 {name}: {pred}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,132 +0,0 @@
|
|||||||
"""Test the '30 NN data = high-nibbles + int8 low-bytes' hypothesis.
|
|
||||||
|
|
||||||
Layout for `30 04` (6 data bytes, 4 deltas):
|
|
||||||
bytes [0:2] = 16 bits = 4 × 4-bit high-nibbles (MSB first)
|
|
||||||
bytes [2:6] = 4 × int8 low bytes
|
|
||||||
Each delta = 12-bit signed = sign-extend((high_nibble << 8) | low_byte)
|
|
||||||
"""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def i8(b):
|
|
||||||
return b if b < 128 else b - 256
|
|
||||||
|
|
||||||
|
|
||||||
def sign_extend_12(v):
|
|
||||||
return v if v < 0x800 else v - 0x1000
|
|
||||||
|
|
||||||
|
|
||||||
def decode_30nn(data):
|
|
||||||
"""4 × 12-bit signed deltas (high nibble + low byte).
|
|
||||||
bytes[0:2] hold the 4 high nibbles (MSB first); bytes[2:6] hold the low bytes.
|
|
||||||
"""
|
|
||||||
if len(data) < 6:
|
|
||||||
return []
|
|
||||||
# Read high nibbles from bytes 0-1 (4 nibbles MSB-first)
|
|
||||||
high_word = (data[0] << 8) | data[1]
|
|
||||||
high_nibbles = [
|
|
||||||
(high_word >> 12) & 0xF,
|
|
||||||
(high_word >> 8) & 0xF,
|
|
||||||
(high_word >> 4) & 0xF,
|
|
||||||
high_word & 0xF,
|
|
||||||
]
|
|
||||||
out = []
|
|
||||||
for i in range(4):
|
|
||||||
v = (high_nibbles[i] << 8) | data[2 + i]
|
|
||||||
out.append(sign_extend_12(v))
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def simulate_up_to(blocks, target_block_idx, t_preamble):
|
|
||||||
"""Run decoder up to block_idx; return per-channel sample lists.
|
|
||||||
NOW with 30 NN decoded too."""
|
|
||||||
out = {"Tran": [], "Vert": [], "Long": [], "MicL": []}
|
|
||||||
out["Tran"].extend(t_preamble)
|
|
||||||
cur = {"Tran": t_preamble[-1], "Vert": None, "Long": None, "MicL": None}
|
|
||||||
rotation = ["Vert", "Long", "MicL", "Tran"]
|
|
||||||
current_channel = "Tran"
|
|
||||||
seg_counter = -1
|
|
||||||
for j in range(target_block_idx):
|
|
||||||
blk = blocks[j]
|
|
||||||
if blk.tag_hi == 0x40:
|
|
||||||
seg_counter += 1
|
|
||||||
prev = "Tran" if seg_counter == 0 else rotation[(seg_counter - 1) % 4]
|
|
||||||
new_ch = rotation[seg_counter % 4]
|
|
||||||
if cur[prev] is not None:
|
|
||||||
d0 = int.from_bytes(blk.data[0:2], "big", signed=True)
|
|
||||||
d1 = int.from_bytes(blk.data[2:4], "big", signed=True)
|
|
||||||
cur[prev] += d0; out[prev].append(cur[prev])
|
|
||||||
cur[prev] += d1; out[prev].append(cur[prev])
|
|
||||||
c0 = int.from_bytes(blk.data[14:16], "big", signed=True)
|
|
||||||
c1 = int.from_bytes(blk.data[16:18], "big", signed=True)
|
|
||||||
out[new_ch].extend([c0, c1])
|
|
||||||
cur[new_ch] = c1
|
|
||||||
current_channel = new_ch
|
|
||||||
elif blk.tag_hi == 0x10:
|
|
||||||
for byte in blk.data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
cur[current_channel] += s4(nib)
|
|
||||||
out[current_channel].append(cur[current_channel])
|
|
||||||
elif blk.tag_hi == 0x20:
|
|
||||||
for byte in blk.data:
|
|
||||||
cur[current_channel] += i8(byte)
|
|
||||||
out[current_channel].append(cur[current_channel])
|
|
||||||
elif blk.tag_hi == 0x00:
|
|
||||||
for _ in range(blk.tag_lo):
|
|
||||||
out[current_channel].append(cur[current_channel])
|
|
||||||
elif blk.tag_hi == 0x30:
|
|
||||||
# NEW: decode 30 NN
|
|
||||||
deltas = decode_30nn(blk.data)
|
|
||||||
for d in deltas:
|
|
||||||
cur[current_channel] += d
|
|
||||||
out[current_channel].append(cur[current_channel])
|
|
||||||
return out, current_channel
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1A.SP0", "M529LL1L.JQ0", "M529LL1L.V70",
|
|
||||||
"M529LL1A.SS0", "M529LL1A.SV0"):
|
|
||||||
path = f"tests/fixtures/5-11-26/{stem}"
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
body = f.read()[43:-26]
|
|
||||||
_, samples = _parse_txt(path + ".TXT")
|
|
||||||
blocks = walk_body(body, find_data_start(body))
|
|
||||||
t0 = int.from_bytes(body[3:5], "big", signed=True)
|
|
||||||
t1 = int.from_bytes(body[5:7], "big", signed=True)
|
|
||||||
thirty_blocks = [(j, b) for j, b in enumerate(blocks) if b.tag_hi == 0x30]
|
|
||||||
if not thirty_blocks:
|
|
||||||
continue
|
|
||||||
print(f"\n=== {stem} ===")
|
|
||||||
for j, blk in thirty_blocks:
|
|
||||||
pred, ch = simulate_up_to(blocks, j, [t0, t1])
|
|
||||||
cur_before = pred[ch][-1]
|
|
||||||
truth = [round(v * 200) for v in samples[ch]]
|
|
||||||
n_pred = len(pred[ch])
|
|
||||||
nn = blk.tag_lo
|
|
||||||
if n_pred + nn > len(truth):
|
|
||||||
continue
|
|
||||||
# Decode this 30 NN block with hypothesis
|
|
||||||
pred_deltas = decode_30nn(blk.data)
|
|
||||||
# Compute truth deltas relative to cur_before
|
|
||||||
truth_deltas = []
|
|
||||||
prev = cur_before
|
|
||||||
for k in range(nn):
|
|
||||||
truth_deltas.append(truth[n_pred + k] - prev)
|
|
||||||
prev = truth[n_pred + k]
|
|
||||||
n_match = sum(1 for a, b in zip(pred_deltas, truth_deltas) if a == b)
|
|
||||||
tag = "✓" if pred_deltas == truth_deltas else " "
|
|
||||||
print(f" block @ {blk.offset:>5} (chan={ch}, NN={nn}):")
|
|
||||||
print(f" data: {blk.data.hex(' ')}")
|
|
||||||
print(f" truth: {truth_deltas}")
|
|
||||||
print(f" pred: {pred_deltas} {tag}{n_match}/{nn}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,141 +0,0 @@
|
|||||||
"""Test 30 NN packing by running the real decoder up to each 30 NN block,
|
|
||||||
recording how many samples have been produced for each channel at that point,
|
|
||||||
then checking truth deltas immediately after."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def i8(b):
|
|
||||||
return b if b < 128 else b - 256
|
|
||||||
|
|
||||||
|
|
||||||
def s12(v):
|
|
||||||
return v if v < 0x800 else v - 0x1000
|
|
||||||
|
|
||||||
|
|
||||||
def unpack_12bit_be_contiguous(data):
|
|
||||||
out = []
|
|
||||||
val = int.from_bytes(data, "big")
|
|
||||||
n = len(data) * 8 // 12
|
|
||||||
for i in range(n):
|
|
||||||
d = (val >> (12 * (n - 1 - i))) & 0xFFF
|
|
||||||
out.append(s12(d))
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def unpack_12bit_per_triplet_be(data):
|
|
||||||
out = []
|
|
||||||
for i in range(0, len(data), 3):
|
|
||||||
if i + 2 >= len(data):
|
|
||||||
break
|
|
||||||
b0, b1, b2 = data[i], data[i + 1], data[i + 2]
|
|
||||||
d0 = (b0 << 4) | (b1 >> 4)
|
|
||||||
d1 = ((b1 & 0x0F) << 8) | b2
|
|
||||||
out.append(s12(d0))
|
|
||||||
out.append(s12(d1))
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def simulate_up_to(blocks, target_block_idx, t_preamble):
|
|
||||||
"""Run the decoder up to block_idx; return per-channel sample lists."""
|
|
||||||
out = {"Tran": [], "Vert": [], "Long": [], "MicL": []}
|
|
||||||
out["Tran"].extend(t_preamble)
|
|
||||||
cur = {"Tran": t_preamble[-1], "Vert": None, "Long": None, "MicL": None}
|
|
||||||
rotation = ["Vert", "Long", "MicL", "Tran"]
|
|
||||||
seg_idx = [j for j, b in enumerate(blocks) if b.tag_hi == 0x40]
|
|
||||||
|
|
||||||
# Determine which channel we're CURRENTLY decoding into
|
|
||||||
current_channel = "Tran"
|
|
||||||
seg_counter = -1 # incremented at each 40 02
|
|
||||||
|
|
||||||
for j in range(target_block_idx):
|
|
||||||
blk = blocks[j]
|
|
||||||
if blk.tag_hi == 0x40:
|
|
||||||
# Switch: extend prev channel, set up new channel
|
|
||||||
seg_counter += 1
|
|
||||||
prev = "Tran" if seg_counter == 0 else rotation[(seg_counter - 1) % 4]
|
|
||||||
new_ch = rotation[seg_counter % 4]
|
|
||||||
if cur[prev] is not None:
|
|
||||||
d0 = int.from_bytes(blk.data[0:2], "big", signed=True)
|
|
||||||
d1 = int.from_bytes(blk.data[2:4], "big", signed=True)
|
|
||||||
cur[prev] += d0; out[prev].append(cur[prev])
|
|
||||||
cur[prev] += d1; out[prev].append(cur[prev])
|
|
||||||
c0 = int.from_bytes(blk.data[14:16], "big", signed=True)
|
|
||||||
c1 = int.from_bytes(blk.data[16:18], "big", signed=True)
|
|
||||||
out[new_ch].extend([c0, c1])
|
|
||||||
cur[new_ch] = c1
|
|
||||||
current_channel = new_ch
|
|
||||||
elif blk.tag_hi == 0x10:
|
|
||||||
for byte in blk.data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
cur[current_channel] += s4(nib)
|
|
||||||
out[current_channel].append(cur[current_channel])
|
|
||||||
elif blk.tag_hi == 0x20:
|
|
||||||
for byte in blk.data:
|
|
||||||
cur[current_channel] += i8(byte)
|
|
||||||
out[current_channel].append(cur[current_channel])
|
|
||||||
elif blk.tag_hi == 0x00:
|
|
||||||
for _ in range(blk.tag_lo):
|
|
||||||
out[current_channel].append(cur[current_channel])
|
|
||||||
elif blk.tag_hi == 0x30:
|
|
||||||
# Skip for now — we want to know what comes next
|
|
||||||
pass
|
|
||||||
|
|
||||||
return out, current_channel
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1A.SP0", "M529LL1L.JQ0", "M529LL1L.V70",
|
|
||||||
"M529LL1A.SS0", "M529LL1A.SV0"):
|
|
||||||
path = f"tests/fixtures/5-11-26/{stem}"
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
body = f.read()[43:-26]
|
|
||||||
_, samples = _parse_txt(path + ".TXT")
|
|
||||||
blocks = walk_body(body, find_data_start(body))
|
|
||||||
t0 = int.from_bytes(body[3:5], "big", signed=True)
|
|
||||||
t1 = int.from_bytes(body[5:7], "big", signed=True)
|
|
||||||
|
|
||||||
# Find all 30 NN blocks in data section
|
|
||||||
thirty_blocks = [(j, b) for j, b in enumerate(blocks) if b.tag_hi == 0x30]
|
|
||||||
if not thirty_blocks:
|
|
||||||
continue
|
|
||||||
|
|
||||||
print(f"\n=== {stem} ===")
|
|
||||||
for j, blk in thirty_blocks:
|
|
||||||
pred, ch = simulate_up_to(blocks, j, [t0, t1])
|
|
||||||
n_pred = len(pred[ch])
|
|
||||||
# The 30 NN block carries NN deltas for channel `ch` starting at sample n_pred
|
|
||||||
truth = [round(v * 200) for v in samples[ch]]
|
|
||||||
if n_pred >= len(truth):
|
|
||||||
continue
|
|
||||||
# Truth deltas: truth[n_pred] - cur, truth[n_pred+1] - truth[n_pred], ...
|
|
||||||
cur_val = pred[ch][-1]
|
|
||||||
nn = blk.tag_lo
|
|
||||||
truth_deltas = []
|
|
||||||
prev = cur_val
|
|
||||||
for k in range(min(nn, len(truth) - n_pred)):
|
|
||||||
truth_deltas.append(truth[n_pred + k] - prev)
|
|
||||||
prev = truth[n_pred + k]
|
|
||||||
|
|
||||||
print(f" block @ {blk.offset:>5} (chan={ch}, after sample {n_pred-1}, "
|
|
||||||
f"NN={nn}, last_val={cur_val}):")
|
|
||||||
print(f" data: {blk.data.hex(' ')}")
|
|
||||||
print(f" truth: {truth_deltas}")
|
|
||||||
schemes = [
|
|
||||||
("12-bit BE contiguous", unpack_12bit_be_contiguous(blk.data)),
|
|
||||||
("12-bit per-triplet BE", unpack_12bit_per_triplet_be(blk.data)),
|
|
||||||
]
|
|
||||||
for name, pred_deltas in schemes:
|
|
||||||
n_match = sum(1 for a, b in zip(pred_deltas, truth_deltas) if a == b)
|
|
||||||
tag = "✓" if pred_deltas == truth_deltas else " "
|
|
||||||
print(f" {tag}{n_match}/{nn} {name}: {pred_deltas[:nn]}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
"""Test: 00 NN markers might be RLE for zero-deltas in current channel."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def i8(b):
|
|
||||||
return b if b < 128 else b - 256
|
|
||||||
|
|
||||||
|
|
||||||
def decode_with_rle(body):
|
|
||||||
"""Decode Tran assuming:
|
|
||||||
- preamble[3:5], [5:7] = T[0], T[1]
|
|
||||||
- All 10 NN / 20 NN blocks until segment_header (40 02) are Tran deltas
|
|
||||||
- 00 NN markers are RLE: NN/4 zero T deltas (or NN, or NN/2 — try them)
|
|
||||||
"""
|
|
||||||
if len(body) < 9 or body[0:3] != b"\x00\x02\x00":
|
|
||||||
return None, None, None
|
|
||||||
T0 = int.from_bytes(body[3:5], "big", signed=True)
|
|
||||||
T1 = int.from_bytes(body[5:7], "big", signed=True)
|
|
||||||
|
|
||||||
# Find first tag (might be 00 NN, 10 NN, or 20 NN)
|
|
||||||
i = 7
|
|
||||||
while i + 1 < len(body):
|
|
||||||
if body[i] in (0x00, 0x10, 0x20):
|
|
||||||
break
|
|
||||||
i += 1
|
|
||||||
start = i
|
|
||||||
|
|
||||||
blocks = walk_body(body, start)
|
|
||||||
|
|
||||||
results = {}
|
|
||||||
for rle_div in (4, 2, 1): # try different RLE interpretations
|
|
||||||
T = [T0, T1]
|
|
||||||
cur = T1
|
|
||||||
for blk in blocks:
|
|
||||||
if blk.tag_hi == 0x40:
|
|
||||||
break
|
|
||||||
if blk.tag_hi == 0x10:
|
|
||||||
for byte in blk.data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
cur += s4(nib)
|
|
||||||
T.append(cur)
|
|
||||||
elif blk.tag_hi == 0x20:
|
|
||||||
for byte in blk.data:
|
|
||||||
cur += i8(byte)
|
|
||||||
T.append(cur)
|
|
||||||
elif blk.tag_hi == 0x00:
|
|
||||||
# RLE of zero deltas
|
|
||||||
n_zeros = blk.tag_lo // rle_div
|
|
||||||
for _ in range(n_zeros):
|
|
||||||
T.append(cur)
|
|
||||||
# 30 NN: skip for now
|
|
||||||
results[rle_div] = T
|
|
||||||
return results, T0, T1
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1L.V70", "M529LL1L.JQ0", "M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
|
|
||||||
path = f"tests/fixtures/5-11-26/{stem}"
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
body = f.read()[43:-26]
|
|
||||||
_, samples = _parse_txt(path + ".TXT")
|
|
||||||
truth_T = [round(v*200) for v in samples["Tran"]]
|
|
||||||
|
|
||||||
results, T0, T1 = decode_with_rle(body)
|
|
||||||
print(f"\n=== {stem} (T[0]={T0}, T[1]={T1}) ===")
|
|
||||||
for rle_div, T in results.items():
|
|
||||||
n = min(len(T), len(truth_T))
|
|
||||||
matches = sum(1 for i in range(n) if T[i] == truth_T[i])
|
|
||||||
# Find first divergence
|
|
||||||
div_at = -1
|
|
||||||
for i in range(n):
|
|
||||||
if T[i] != truth_T[i]:
|
|
||||||
div_at = i
|
|
||||||
break
|
|
||||||
print(f" rle_div={rle_div}: decoded {len(T)}, matches {matches}/{n}, first div at sample {div_at}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,71 +0,0 @@
|
|||||||
"""Test: does the second '20 NN' block in SS0 continue Tran samples?"""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def i8(b):
|
|
||||||
return b if b < 128 else b - 256
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
stem = "M529LL1A.SS0"
|
|
||||||
path = f"tests/fixtures/5-11-26/{stem}"
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
body = f.read()[43:-26]
|
|
||||||
_, samples = _parse_txt(path + ".TXT")
|
|
||||||
truth_T_16 = [round(v * 200) for v in samples["Tran"]]
|
|
||||||
|
|
||||||
# Preamble
|
|
||||||
T0 = int.from_bytes(body[3:5], "big", signed=True)
|
|
||||||
T1 = int.from_bytes(body[5:7], "big", signed=True)
|
|
||||||
|
|
||||||
# Walk blocks
|
|
||||||
start = find_data_start(body)
|
|
||||||
blocks = walk_body(body, start)
|
|
||||||
|
|
||||||
print(f"=== {stem} === T[0]={T0} T[1]={T1}")
|
|
||||||
|
|
||||||
# Hypothesis: Tran continues through ALL 10 NN and 20 NN blocks
|
|
||||||
# in order, until the next 40 02 segment header (which resets).
|
|
||||||
T = [T0, T1]
|
|
||||||
cur = T1
|
|
||||||
decoded_count = 2 # T[0], T[1] from preamble
|
|
||||||
for bi, blk in enumerate(blocks):
|
|
||||||
if blk.tag_hi == 0x10:
|
|
||||||
for byte in blk.data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
cur += s4(nib)
|
|
||||||
T.append(cur)
|
|
||||||
decoded_count += 1
|
|
||||||
elif blk.tag_hi == 0x20:
|
|
||||||
for byte in blk.data:
|
|
||||||
cur += i8(byte)
|
|
||||||
T.append(cur)
|
|
||||||
decoded_count += 1
|
|
||||||
elif blk.tag_hi == 0x40:
|
|
||||||
# Segment header — stop here for this test
|
|
||||||
break
|
|
||||||
# 00 and 30 NN don't contribute to Tran (in this hypothesis)
|
|
||||||
|
|
||||||
# Compare to truth
|
|
||||||
print(f" Decoded {len(T)} T samples up to first 40 02")
|
|
||||||
matches = sum(1 for i in range(min(len(T), len(truth_T_16))) if T[i] == truth_T_16[i])
|
|
||||||
print(f" Matches in first {min(len(T), len(truth_T_16))}: {matches}")
|
|
||||||
# Print first divergence
|
|
||||||
for i in range(min(len(T), len(truth_T_16))):
|
|
||||||
if T[i] != truth_T_16[i]:
|
|
||||||
print(f" First divergence: sample {i}: pred={T[i]}, truth={truth_T_16[i]}")
|
|
||||||
# Show context
|
|
||||||
print(f" pred [{i-3}:{i+5}]: {T[max(0,i-3):i+5]}")
|
|
||||||
print(f" truth [{i-3}:{i+5}]: {truth_T_16[max(0,i-3):i+5]}")
|
|
||||||
break
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,67 +0,0 @@
|
|||||||
"""Try various nibble-level channel interleavings to find which one matches truth."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def run_decoder(body, layout, skip, n_channels=4):
|
|
||||||
"""layout: function nibble_index -> channel_index. Returns list-of-lists per channel."""
|
|
||||||
out = [[] for _ in range(n_channels)]
|
|
||||||
cur = [0] * n_channels
|
|
||||||
nibbles = []
|
|
||||||
for byte in body[skip:]:
|
|
||||||
nibbles.append((byte >> 4) & 0xF)
|
|
||||||
nibbles.append(byte & 0xF)
|
|
||||||
for i, n in enumerate(nibbles):
|
|
||||||
ch = layout(i)
|
|
||||||
cur[ch] += s4(n)
|
|
||||||
out[ch].append(cur[ch])
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def cmp(pred, truth, n=24):
|
|
||||||
n = min(n, len(pred), len(truth))
|
|
||||||
return [(pred[i], truth[i]) for i in range(n)]
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
b = load_bundle("event-c")
|
|
||||||
truth_T = [round(v * 200) for v in b.samples["Tran"]]
|
|
||||||
truth_V = [round(v * 200) for v in b.samples["Vert"]]
|
|
||||||
truth_L = [round(v * 200) for v in b.samples["Long"]]
|
|
||||||
print(f"T truth[0:10]: {truth_T[:10]}")
|
|
||||||
print(f"V truth[0:10]: {truth_V[:10]}")
|
|
||||||
print(f"L truth[0:10]: {truth_L[:10]}")
|
|
||||||
|
|
||||||
# Try several nibble->channel layouts (4 channels)
|
|
||||||
layouts = {
|
|
||||||
"interleaved TVLM (0,1,2,3,0,1,2,3,...)": lambda i: i % 4,
|
|
||||||
"interleaved VLMT": lambda i: (i + 3) % 4,
|
|
||||||
"interleaved LMTV": lambda i: (i + 2) % 4,
|
|
||||||
"interleaved MTVL": lambda i: (i + 1) % 4,
|
|
||||||
"byte-based TV LM TV LM (high T low V byte0; high L low M byte1)": lambda i: i % 4,
|
|
||||||
# "chunks of 8 nibbles per channel": each channel gets 8 nibbles in a row
|
|
||||||
"chunks-8 TVLM": lambda i: (i // 8) % 4,
|
|
||||||
"chunks-16 TVLM": lambda i: (i // 16) % 4,
|
|
||||||
# planar (full channel sequential)
|
|
||||||
"planar T(0..N) V(N..2N) L(2N..3N) M(3N..4N)": None, # special
|
|
||||||
}
|
|
||||||
|
|
||||||
for label, layout_fn in layouts.items():
|
|
||||||
if layout_fn is None:
|
|
||||||
continue
|
|
||||||
for skip in (0, 4, 7, 8, 9, 11, 14):
|
|
||||||
out = run_decoder(b.body, layout_fn, skip)
|
|
||||||
# Check first 8 cumulative on each channel
|
|
||||||
print(f" skip={skip:2} {label}")
|
|
||||||
print(f" T_cum[0:10]: {out[0][:10]}")
|
|
||||||
print(f" V_cum[0:10]: {out[1][:10]}")
|
|
||||||
print(f" L_cum[0:10]: {out[2][:10]}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,73 +0,0 @@
|
|||||||
"""Try decoding body as 4-bit signed nibble deltas, 4-channel round-robin."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
CHANNELS = ("Tran", "Vert", "Long", "MicL")
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
"""Sign-extend a 4-bit unsigned to int (0..7 → 0..7, 8..F → -8..-1)."""
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def decode_nibbles(body: bytes, skip_bytes: int = 7, n_channels: int = 4):
|
|
||||||
"""Read body as 2 nibbles per byte; accumulate as deltas for n_channels round-robin."""
|
|
||||||
out = [[] for _ in range(n_channels)]
|
|
||||||
cur = [0] * n_channels
|
|
||||||
ch = 0
|
|
||||||
nibbles = []
|
|
||||||
for byte in body[skip_bytes:]:
|
|
||||||
nibbles.append((byte >> 4) & 0xF)
|
|
||||||
nibbles.append(byte & 0xF)
|
|
||||||
for n in nibbles:
|
|
||||||
cur[ch] += s4(n)
|
|
||||||
out[ch].append(cur[ch])
|
|
||||||
ch = (ch + 1) % n_channels
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def cmp_to_truth(pred, truth, scale=16):
|
|
||||||
"""Compare predicted ints (in 16-count units) to truth (in 16-count units = txt * 200).
|
|
||||||
Return (max_abs_err, mean_abs_err, n_compared).
|
|
||||||
"""
|
|
||||||
n = min(len(pred), len(truth))
|
|
||||||
errs = []
|
|
||||||
for i in range(n):
|
|
||||||
p = pred[i]
|
|
||||||
t = truth[i]
|
|
||||||
errs.append(abs(p - t))
|
|
||||||
if not errs:
|
|
||||||
return None
|
|
||||||
return (max(errs), sum(errs) / len(errs), n)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-a", "event-c"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
# Convert TXT samples (in/s) to 16-count units (multiply by 200, since 0.005 in/s = 1)
|
|
||||||
# WAIT: 0.005 in/s = 16 ADC counts. 1 count = 0.000305 in/s.
|
|
||||||
# So in 1-count units: count = txt * (1/0.0003052) ≈ txt * 3276.7
|
|
||||||
# But TXT only has 0.005 resolution so equivalent to 16-count units = txt * 200.
|
|
||||||
truth_in_16 = {ch: [round(v * 200) for v in b.samples[ch]] for ch in CHANNELS[:3]}
|
|
||||||
# MicL is in dB, skip for now
|
|
||||||
|
|
||||||
# Try decoder with skip_bytes = 7
|
|
||||||
decoded = decode_nibbles(b.body, skip_bytes=7, n_channels=4)
|
|
||||||
print(f"\n=== {name} ===")
|
|
||||||
print(f" body={len(b.body)}, nibbles={2*(len(b.body)-7)}, samples_per_ch={len(decoded[0])}")
|
|
||||||
print(f" truth samples per ch: {len(truth_in_16['Tran'])}")
|
|
||||||
# Print first 24 of each
|
|
||||||
for i, chan in enumerate(CHANNELS):
|
|
||||||
pred_first = decoded[i][:24]
|
|
||||||
if chan in truth_in_16:
|
|
||||||
truth_first = truth_in_16[chan][:24]
|
|
||||||
print(f" {chan} pred: {pred_first}")
|
|
||||||
print(f" {chan} truth: {truth_first}")
|
|
||||||
else:
|
|
||||||
print(f" {chan} pred: {pred_first} (truth in dB, skipped)")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,32 +0,0 @@
|
|||||||
"""Verify decode_waveform_v2 against BW ASCII truth for all fixtures."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
from minimateplus.waveform_codec import decode_waveform_v2
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0",
|
|
||||||
"M529LL1L.JQ0", "M529LL1L.V70"):
|
|
||||||
path = f"tests/fixtures/5-11-26/{stem}"
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
body = f.read()[43:-26]
|
|
||||||
_, samples = _parse_txt(path + ".TXT")
|
|
||||||
decoded = decode_waveform_v2(body)
|
|
||||||
if decoded is None:
|
|
||||||
print(f"{stem}: decoder returned None")
|
|
||||||
continue
|
|
||||||
|
|
||||||
print(f"\n=== {stem} ===")
|
|
||||||
for ch in ("Tran", "Vert", "Long"):
|
|
||||||
truth = [round(v * 200) for v in samples[ch]]
|
|
||||||
pred = decoded[ch]
|
|
||||||
n = min(len(pred), len(truth))
|
|
||||||
matches = sum(1 for i in range(n) if pred[i] == truth[i])
|
|
||||||
div = next((i for i in range(n) if pred[i] != truth[i]), -1)
|
|
||||||
print(f" {ch}: decoded={len(pred):>5} truth={len(truth):>5} "
|
|
||||||
f"matches={matches:>5}/{n:<5} first div={div}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,55 +0,0 @@
|
|||||||
"""Run decode_waveform_v2 against the 5-8-26 quiet bundle to test the
|
|
||||||
'quiet events should decode fully' hypothesis."""
|
|
||||||
import os, sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from minimateplus.waveform_codec import decode_waveform_v2, walk_body, find_data_start
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
base = "tests/fixtures/decode-re-5-8-26"
|
|
||||||
for evt in sorted(os.listdir(base)):
|
|
||||||
folder = os.path.join(base, evt)
|
|
||||||
if not os.path.isdir(folder):
|
|
||||||
continue
|
|
||||||
# Find the binary (not .TXT)
|
|
||||||
bin_name = next(
|
|
||||||
(f for f in os.listdir(folder) if not f.endswith(".TXT")),
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
if not bin_name:
|
|
||||||
continue
|
|
||||||
bin_path = os.path.join(folder, bin_name)
|
|
||||||
txt_path = bin_path + ".TXT"
|
|
||||||
if not os.path.exists(txt_path):
|
|
||||||
# Sometimes the TXT name differs slightly
|
|
||||||
for f in os.listdir(folder):
|
|
||||||
if f.endswith(".TXT"):
|
|
||||||
txt_path = os.path.join(folder, f)
|
|
||||||
break
|
|
||||||
with open(bin_path, "rb") as f:
|
|
||||||
body = f.read()[43:-26]
|
|
||||||
decoded = decode_waveform_v2(body)
|
|
||||||
_, samples = _parse_txt(txt_path)
|
|
||||||
|
|
||||||
# Count 30 NN blocks
|
|
||||||
blocks = walk_body(body, find_data_start(body))
|
|
||||||
n_30 = sum(1 for b in blocks if b.tag_hi == 0x30)
|
|
||||||
n_40 = sum(1 for b in blocks if b.tag_hi == 0x40)
|
|
||||||
|
|
||||||
print(f"\n=== {evt} === body={len(body)} segments={n_40} '30 NN' blocks={n_30}")
|
|
||||||
if decoded is None:
|
|
||||||
print(" decoder returned None")
|
|
||||||
continue
|
|
||||||
for ch in ("Tran", "Vert", "Long"):
|
|
||||||
truth = [round(v * 200) for v in samples[ch]]
|
|
||||||
pred = decoded[ch]
|
|
||||||
n = min(len(pred), len(truth))
|
|
||||||
matches = sum(1 for i in range(n) if pred[i] == truth[i])
|
|
||||||
div = next((i for i in range(n) if pred[i] != truth[i]), -1)
|
|
||||||
print(f" {ch}: decoded={len(pred):>5} truth={len(truth):>5} "
|
|
||||||
f"matches={matches:>5}/{n:<5} first div={div}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,71 +0,0 @@
|
|||||||
"""Verify: preamble[3:7] = Tran[0], Tran[1] as int16 BE in 16-count units.
|
|
||||||
And first 20/10 NN block = Tran deltas starting at sample 2.
|
|
||||||
"""
|
|
||||||
import os, sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import _parse_txt
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n):
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def i8(b):
|
|
||||||
return b if b < 128 else b - 256
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
|
|
||||||
path = f"tests/fixtures/5-11-26/{stem}"
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
raw = f.read()
|
|
||||||
body = raw[43:-26]
|
|
||||||
_, samples = _parse_txt(path + ".TXT")
|
|
||||||
truth_T_16 = [round(v * 200) for v in samples["Tran"]]
|
|
||||||
|
|
||||||
# Preamble parse
|
|
||||||
T0_pre = int.from_bytes(body[3:5], "big", signed=True)
|
|
||||||
T1_pre = int.from_bytes(body[5:7], "big", signed=True)
|
|
||||||
print(f"\n=== {stem} ===")
|
|
||||||
print(f" Preamble T[0]={T0_pre} (truth {truth_T_16[0]}) T[1]={T1_pre} (truth {truth_T_16[1]}) match={T0_pre==truth_T_16[0] and T1_pre==truth_T_16[1]}")
|
|
||||||
|
|
||||||
# First block
|
|
||||||
start = find_data_start(body)
|
|
||||||
blocks = walk_body(body, start)
|
|
||||||
if not blocks:
|
|
||||||
print(f" no blocks found")
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Assume first block = Tran deltas from sample 2
|
|
||||||
first = blocks[0]
|
|
||||||
T = [T0_pre, T1_pre]
|
|
||||||
cur_T = T1_pre
|
|
||||||
if first.tag_hi == 0x10:
|
|
||||||
# Nibble pairs
|
|
||||||
for byte in first.data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
cur_T += s4(nib)
|
|
||||||
T.append(cur_T)
|
|
||||||
elif first.tag_hi == 0x20:
|
|
||||||
# int8 per byte
|
|
||||||
for byte in first.data:
|
|
||||||
cur_T += i8(byte)
|
|
||||||
T.append(cur_T)
|
|
||||||
|
|
||||||
# Compare against truth
|
|
||||||
n_check = min(len(T), len(truth_T_16))
|
|
||||||
match_count = sum(1 for i in range(n_check) if T[i] == truth_T_16[i])
|
|
||||||
print(f" First block type=0x{first.tag_hi:02x} NN=0x{first.tag_lo:02x} len={len(first.data)} → {len(T)} T samples decoded")
|
|
||||||
print(f" Tran predicted[0:10]: {T[:10]}")
|
|
||||||
print(f" Tran truth [0:10]: {truth_T_16[:10]}")
|
|
||||||
print(f" Matches in first {n_check}: {match_count} / {n_check}")
|
|
||||||
# Show where it diverges
|
|
||||||
for i in range(n_check):
|
|
||||||
if T[i] != truth_T_16[i]:
|
|
||||||
print(f" First divergence: sample {i}: pred={T[i]}, truth={truth_T_16[i]}")
|
|
||||||
break
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,20 +0,0 @@
|
|||||||
"""Walk blocks of the new 5-11-26 events and look at what comes after Tran block."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for stem in ("M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
|
|
||||||
with open(f"tests/fixtures/5-11-26/{stem}", "rb") as f:
|
|
||||||
raw = f.read()
|
|
||||||
body = raw[43:-26]
|
|
||||||
start = find_data_start(body)
|
|
||||||
blocks = walk_body(body, start)
|
|
||||||
print(f"\n=== {stem} === body={len(body)} start={start} blocks walked={len(blocks)}")
|
|
||||||
for i, b in enumerate(blocks[:20]):
|
|
||||||
print(f" block[{i:>2}] @ {b.offset:>5} tag={b.tag_hi:02x} NN=0x{b.tag_lo:02x}({b.tag_lo}) len={b.length} data[:24]={b.data[:24].hex(' ')}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,44 +0,0 @@
|
|||||||
"""Walk the body assuming chunks delimited by 0x10 NN tags. Print each chunk's structure."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def walk(body: bytes, start_offset: int = 7, max_chunks: int = 30):
|
|
||||||
"""Find all positions where byte = 0x10 followed by a multiple-of-4 byte. Print chunks."""
|
|
||||||
chunks = []
|
|
||||||
i = start_offset
|
|
||||||
while i < len(body) - 1:
|
|
||||||
# Find next `10 NN` where NN is multiple of 4 (and not preceded by another 0x10 immediately, which would be data).
|
|
||||||
if body[i] == 0x10 and (body[i+1] % 4 == 0):
|
|
||||||
chunks.append(i)
|
|
||||||
i += 1
|
|
||||||
return chunks
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-c", "event-d"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
positions = []
|
|
||||||
i = 7 # skip 7-byte preamble
|
|
||||||
while i < len(body) - 1:
|
|
||||||
if body[i] == 0x10 and body[i+1] % 4 == 0 and body[i+1] > 0:
|
|
||||||
positions.append(i)
|
|
||||||
i += 2 # skip past tag
|
|
||||||
else:
|
|
||||||
i += 1
|
|
||||||
print(f"\n=== {name} === body={len(body)}, total `10 NN` (NN%4==0, NN>0) tags: {len(positions)}")
|
|
||||||
# Print first 20 chunks: show position, NN, gap to next tag
|
|
||||||
for k in range(min(30, len(positions))):
|
|
||||||
pos = positions[k]
|
|
||||||
NN = body[pos + 1]
|
|
||||||
next_pos = positions[k+1] if k+1 < len(positions) else len(body)
|
|
||||||
gap = next_pos - pos
|
|
||||||
data_bytes = body[pos+2 : next_pos]
|
|
||||||
print(f" chunk[{k:>3}] @ {pos:>5} NN=0x{NN:02x} ({NN:>3}, NN/2={NN//2}) gap={gap:>3} "
|
|
||||||
f"data={data_bytes[:24].hex(' ')}{'...' if len(data_bytes) > 24 else ''}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,50 +0,0 @@
|
|||||||
"""Deterministic chunk walker: each chunk = [10 NN][NN/2 bytes data][2 bytes trailer]."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def walk_chunks(body: bytes, start: int = 7):
|
|
||||||
"""Yield (offset, NN, data_bytes, trailer_bytes) tuples."""
|
|
||||||
i = start
|
|
||||||
while i + 1 < len(body):
|
|
||||||
if body[i] != 0x10:
|
|
||||||
break
|
|
||||||
NN = body[i + 1]
|
|
||||||
if NN == 0 or NN > 0x80 or NN % 4 != 0:
|
|
||||||
break
|
|
||||||
chunk_len = NN // 2 + 4
|
|
||||||
if i + chunk_len > len(body):
|
|
||||||
break
|
|
||||||
data = bytes(body[i + 2 : i + 2 + NN // 2])
|
|
||||||
trailer = bytes(body[i + 2 + NN // 2 : i + chunk_len])
|
|
||||||
yield (i, NN, data, trailer)
|
|
||||||
i += chunk_len
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-c", "event-d", "event-a", "event-b"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
chunks = list(walk_chunks(body))
|
|
||||||
print(f"\n=== {name} === body={len(body)} N_samples={len(b.samples['Tran'])}")
|
|
||||||
print(f" chunks parsed: {len(chunks)}")
|
|
||||||
if chunks:
|
|
||||||
last = chunks[-1]
|
|
||||||
end_of_walk = last[0] + last[1] // 2 + 4
|
|
||||||
print(f" walk ended at offset {end_of_walk} (= {len(body) - end_of_walk} bytes from end)")
|
|
||||||
# Stats
|
|
||||||
total_data_bytes = sum(len(c[2]) for c in chunks)
|
|
||||||
print(f" total data bytes: {total_data_bytes}, total nibbles: {2*total_data_bytes}")
|
|
||||||
if name in ("event-c", "event-d"):
|
|
||||||
ratio = (2 * total_data_bytes) / (len(b.samples['Tran']) * 4)
|
|
||||||
print(f" nibbles per (sample × channel): {ratio:.3f}")
|
|
||||||
# Sum of trailer second-byte
|
|
||||||
trailer_sums = [c[3][-1] if c[3] else None for c in chunks]
|
|
||||||
print(f" first 10 chunks: {[(c[0], c[1], c[3].hex()) for c in chunks[:10]]}")
|
|
||||||
# Print last 10 chunks (likely transition to trailer)
|
|
||||||
print(f" last 10 chunks: {[(c[0], c[1], c[3].hex()) for c in chunks[-10:]]}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
"""Walk chunks; auto-detect preamble length by finding first 10 NN."""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def walk_chunks(body, start, max_NN=0x80):
|
|
||||||
chunks = []
|
|
||||||
i = start
|
|
||||||
while i + 1 < len(body):
|
|
||||||
if body[i] != 0x10:
|
|
||||||
break
|
|
||||||
NN = body[i + 1]
|
|
||||||
if NN == 0 or NN > max_NN or NN % 4 != 0:
|
|
||||||
break
|
|
||||||
chunk_len = NN // 2 + 4
|
|
||||||
if i + chunk_len > len(body):
|
|
||||||
break
|
|
||||||
data = bytes(body[i + 2 : i + 2 + NN // 2])
|
|
||||||
trailer = bytes(body[i + 2 + NN // 2 : i + chunk_len])
|
|
||||||
chunks.append((i, NN, data, trailer))
|
|
||||||
i += chunk_len
|
|
||||||
return chunks, i
|
|
||||||
|
|
||||||
|
|
||||||
def find_first_chunk_start(body):
|
|
||||||
"""Locate first byte that begins a `10 NN` chunk (NN ∈ multiples of 4, 4..0x7C)."""
|
|
||||||
for i in range(20):
|
|
||||||
if body[i] == 0x10 and body[i + 1] % 4 == 0 and 0 < body[i + 1] <= 0x7C:
|
|
||||||
return i
|
|
||||||
return -1
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-c", "event-d", "event-a", "event-b"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
start = find_first_chunk_start(body)
|
|
||||||
chunks, end = walk_chunks(body, start)
|
|
||||||
print(f"\n=== {name} === body={len(body)} N_samples={len(b.samples['Tran'])} start={start}")
|
|
||||||
print(f" chunks parsed: {len(chunks)}, walk ended at {end}")
|
|
||||||
if chunks:
|
|
||||||
print(f" first 5 chunks: {[(c[0], c[1], c[3].hex()) for c in chunks[:5]]}")
|
|
||||||
print(f" last 5 chunks: {[(c[0], c[1], c[3].hex()) for c in chunks[-5:]]}")
|
|
||||||
print(f" bytes around end of walk: {body[end-4:end+12].hex(' ')}")
|
|
||||||
else:
|
|
||||||
print(f" bytes at start: {body[start:start+16].hex(' ')}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,75 +0,0 @@
|
|||||||
"""
|
|
||||||
Walker v4: alternate [10 NN] data chunks and [00 NN] (or other) marker tags.
|
|
||||||
|
|
||||||
Hypothesis:
|
|
||||||
- [10 NN]: data block, length NN/2 + 2 bytes (2-byte tag + NN/2 bytes data)
|
|
||||||
- [00 NN]: 2-byte marker block (no data)
|
|
||||||
- [20/30/40 NN]: special blocks with type-dependent length
|
|
||||||
"""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
|
|
||||||
|
|
||||||
def walk(body, start):
|
|
||||||
i = start
|
|
||||||
blocks = []
|
|
||||||
while i + 1 < len(body):
|
|
||||||
t0 = body[i]
|
|
||||||
t1 = body[i + 1]
|
|
||||||
if t0 == 0x10 and t1 % 4 == 0 and 0 < t1 <= 0x80:
|
|
||||||
# data chunk: length NN/2 + 2
|
|
||||||
length = t1 // 2 + 2
|
|
||||||
blocks.append((i, "10", t1, bytes(body[i + 2 : i + length]), length))
|
|
||||||
i += length
|
|
||||||
elif t0 == 0x00 and t1 % 4 == 0:
|
|
||||||
# 2-byte marker
|
|
||||||
blocks.append((i, "00", t1, b"", 2))
|
|
||||||
i += 2
|
|
||||||
elif t0 == 0x20 and t1 % 4 == 0:
|
|
||||||
# type 2 — try length 2+t1/2 (similar to 10) OR fixed
|
|
||||||
length = t1 // 2 + 2
|
|
||||||
blocks.append((i, "20", t1, bytes(body[i + 2 : i + length]), length))
|
|
||||||
i += length
|
|
||||||
elif t0 == 0x30 and t1 % 4 == 0:
|
|
||||||
length = t1 // 2 + 2
|
|
||||||
blocks.append((i, "30", t1, bytes(body[i + 2 : i + length]), length))
|
|
||||||
i += length
|
|
||||||
elif t0 == 0x40 and t1 == 0x02:
|
|
||||||
# Special "footer transition" block — try fixed 22 bytes
|
|
||||||
length = 22
|
|
||||||
blocks.append((i, "40", t1, bytes(body[i + 2 : i + length]), length))
|
|
||||||
i += length
|
|
||||||
else:
|
|
||||||
# Unknown tag — stop
|
|
||||||
blocks.append((i, "??", t0, bytes(body[i:i+8]), 0))
|
|
||||||
break
|
|
||||||
return blocks, i
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-c", "event-d", "event-a", "event-b"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
# Auto-detect start
|
|
||||||
for s in range(15):
|
|
||||||
if body[s] == 0x10 and body[s+1] % 4 == 0 and 0 < body[s+1] <= 0x80:
|
|
||||||
start = s
|
|
||||||
break
|
|
||||||
else:
|
|
||||||
start = 7
|
|
||||||
blocks, end = walk(body, start)
|
|
||||||
# Categorize
|
|
||||||
from collections import Counter
|
|
||||||
types = Counter(b[1] for b in blocks)
|
|
||||||
print(f"\n=== {name} === body={len(body)} N={len(b.samples['Tran'])} start={start}")
|
|
||||||
print(f" total blocks: {len(blocks)}, walk ended at {end}/{len(body)}")
|
|
||||||
print(f" type counts: {dict(types)}")
|
|
||||||
# Print last 5 blocks
|
|
||||||
print(f" last 5 blocks: {[(bb[0], bb[1], bb[2]) for bb in blocks[-5:]]}")
|
|
||||||
if end < len(body):
|
|
||||||
print(f" bytes at end: {body[end:end+24].hex(' ')}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,83 +0,0 @@
|
|||||||
"""
|
|
||||||
Walker v5: flexible NN range and multiple block-type lengths.
|
|
||||||
|
|
||||||
Hypothesis:
|
|
||||||
- [10 NN]: 4-bit-delta data block, length = NN/2 + 2
|
|
||||||
- [20 NN]: 8-bit-literal data block, length = NN + 2
|
|
||||||
- [00 NN]: 2-byte marker (no payload)
|
|
||||||
- [30 NN]: trailer/summary block, length = NN*4
|
|
||||||
- [40 NN]: footer-marker block, fixed 22 bytes
|
|
||||||
"""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
from collections import Counter
|
|
||||||
|
|
||||||
|
|
||||||
def walk(body, start, max_blocks=10000):
|
|
||||||
i = start
|
|
||||||
blocks = []
|
|
||||||
while i + 1 < len(body) and len(blocks) < max_blocks:
|
|
||||||
t0 = body[i]
|
|
||||||
t1 = body[i + 1]
|
|
||||||
if t0 == 0x10 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
|
|
||||||
length = t1 // 2 + 2
|
|
||||||
if i + length > len(body):
|
|
||||||
break
|
|
||||||
data = bytes(body[i + 2 : i + length])
|
|
||||||
blocks.append((i, "10", t1, data, length))
|
|
||||||
i += length
|
|
||||||
elif t0 == 0x20 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
|
|
||||||
length = t1 + 2
|
|
||||||
if i + length > len(body):
|
|
||||||
break
|
|
||||||
data = bytes(body[i + 2 : i + length])
|
|
||||||
blocks.append((i, "20", t1, data, length))
|
|
||||||
i += length
|
|
||||||
elif t0 == 0x00 and t1 % 4 == 0:
|
|
||||||
# 2-byte marker
|
|
||||||
blocks.append((i, "00", t1, b"", 2))
|
|
||||||
i += 2
|
|
||||||
elif t0 == 0x30 and t1 % 4 == 0:
|
|
||||||
length = t1 * 4
|
|
||||||
if i + length > len(body):
|
|
||||||
break
|
|
||||||
data = bytes(body[i + 2 : i + length])
|
|
||||||
blocks.append((i, "30", t1, data, length))
|
|
||||||
i += length
|
|
||||||
elif t0 == 0x40 and t1 == 0x02:
|
|
||||||
length = 22
|
|
||||||
if i + length > len(body):
|
|
||||||
break
|
|
||||||
data = bytes(body[i + 2 : i + length])
|
|
||||||
blocks.append((i, "40", t1, data, length))
|
|
||||||
i += length
|
|
||||||
else:
|
|
||||||
blocks.append((i, "??", t0, bytes(body[i:i+8]), 0))
|
|
||||||
break
|
|
||||||
return blocks, i
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-c", "event-d", "event-a", "event-b"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
for s in range(15):
|
|
||||||
if body[s] == 0x10 and body[s+1] % 4 == 0 and 0 < body[s+1] <= 0xFC:
|
|
||||||
start = s; break
|
|
||||||
else:
|
|
||||||
start = 7
|
|
||||||
blocks, end = walk(body, start)
|
|
||||||
types = Counter(bb[1] for bb in blocks)
|
|
||||||
print(f"\n=== {name} === body={len(body)} N={len(b.samples['Tran'])} start={start}")
|
|
||||||
print(f" total blocks: {len(blocks)}, walk ended at {end}/{len(body)}")
|
|
||||||
print(f" type counts: {dict(types)}")
|
|
||||||
if blocks and blocks[-1][1] == "??":
|
|
||||||
print(f" stopped at byte: 0x{blocks[-1][2]:02x}, prev 5 blocks: {[(bb[0], bb[1], bb[2]) for bb in blocks[-6:-1]]}")
|
|
||||||
# Sum payload sizes by type
|
|
||||||
payload_sizes = {t: sum(len(bb[3]) for bb in blocks if bb[1] == t) for t in types}
|
|
||||||
print(f" payload bytes by type: {payload_sizes}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,68 +0,0 @@
|
|||||||
"""
|
|
||||||
Walker v6: handle 40 02 blocks correctly (length 20).
|
|
||||||
|
|
||||||
Block formats:
|
|
||||||
- [10 NN]: 4-bit nibble delta data, length = NN/2 + 2
|
|
||||||
- [20 NN]: int8 literal data, length = NN + 2
|
|
||||||
- [00 NN]: 2-byte marker
|
|
||||||
- [30 NN]: trailer/summary block, length = NN*4
|
|
||||||
- [40 02]: segment header, fixed length 20
|
|
||||||
"""
|
|
||||||
import sys
|
|
||||||
sys.path.insert(0, ".")
|
|
||||||
from analysis.load_bundle import load_bundle
|
|
||||||
from collections import Counter
|
|
||||||
|
|
||||||
|
|
||||||
def walk(body, start, max_blocks=10000):
|
|
||||||
i = start
|
|
||||||
blocks = []
|
|
||||||
while i + 1 < len(body) and len(blocks) < max_blocks:
|
|
||||||
t0 = body[i]
|
|
||||||
t1 = body[i + 1]
|
|
||||||
if t0 == 0x10 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
|
|
||||||
length = t1 // 2 + 2
|
|
||||||
elif t0 == 0x20 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
|
|
||||||
length = t1 + 2
|
|
||||||
elif t0 == 0x00 and t1 % 4 == 0:
|
|
||||||
length = 2
|
|
||||||
elif t0 == 0x30 and t1 % 4 == 0 and 0 < t1 <= 0x10:
|
|
||||||
length = t1 * 4
|
|
||||||
elif t0 == 0x40 and t1 == 0x02:
|
|
||||||
length = 20
|
|
||||||
else:
|
|
||||||
blocks.append((i, "??", t0, bytes(body[i:i+8]), 0))
|
|
||||||
break
|
|
||||||
if i + length > len(body):
|
|
||||||
break
|
|
||||||
data = bytes(body[i + 2 : i + length])
|
|
||||||
blocks.append((i, f"{t0:02x}", t1, data, length))
|
|
||||||
i += length
|
|
||||||
return blocks, i
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
for name in ("event-c", "event-d", "event-a", "event-b"):
|
|
||||||
b = load_bundle(name)
|
|
||||||
body = b.body
|
|
||||||
for s in range(15):
|
|
||||||
if body[s] == 0x10 and body[s+1] % 4 == 0 and 0 < body[s+1] <= 0xFC:
|
|
||||||
start = s; break
|
|
||||||
else:
|
|
||||||
start = 7
|
|
||||||
blocks, end = walk(body, start)
|
|
||||||
types = Counter(bb[1] for bb in blocks)
|
|
||||||
print(f"\n=== {name} === body={len(body)} N={len(b.samples['Tran'])} start={start}")
|
|
||||||
print(f" total blocks: {len(blocks)}, walk ended at {end}/{len(body)}")
|
|
||||||
print(f" type counts: {dict(types)}")
|
|
||||||
if blocks and blocks[-1][1] == "??":
|
|
||||||
print(f" stopped at byte: 0x{blocks[-1][2]:02x} at offset {blocks[-1][0]}")
|
|
||||||
print(f" prev 5 blocks: {[(bb[0], bb[1], bb[2]) for bb in blocks[-6:-1]]}")
|
|
||||||
print(f" bytes around stop: {body[end-4:end+24].hex(' ')}")
|
|
||||||
# Sum
|
|
||||||
payload_sizes = {t: sum(len(bb[3]) for bb in blocks if bb[1] == t) for t in types}
|
|
||||||
print(f" payload bytes by type: {payload_sizes}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,65 +0,0 @@
|
|||||||
"""Run read_idf_file across the corpus and report per-channel accuracy vs sidecars."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from micromate.idf_file import read_idf_file
|
|
||||||
from analysis_idf.recon import load_sidecar_samples
|
|
||||||
|
|
||||||
|
|
||||||
def sidecar_path(idfw: Path) -> Path:
|
|
||||||
return idfw.parent / "TXT" / f"{idfw.name}.txt"
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
root = REPO / "tests/fixtures/THORDATA_example"
|
|
||||||
files = [f for f in root.rglob("*.IDFW") if not str(f).endswith(".CDB")]
|
|
||||||
files.sort()
|
|
||||||
GEO_LSB = 0.0003
|
|
||||||
|
|
||||||
n_ok = n_skip = 0
|
|
||||||
overall = {"Tran": [], "Vert": [], "Long": []}
|
|
||||||
|
|
||||||
for f in files:
|
|
||||||
try:
|
|
||||||
res = read_idf_file(f)
|
|
||||||
except Exception:
|
|
||||||
n_skip += 1
|
|
||||||
continue
|
|
||||||
sc_path = sidecar_path(f)
|
|
||||||
if not sc_path.exists():
|
|
||||||
n_skip += 1
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
sc = load_sidecar_samples(sc_path)
|
|
||||||
except Exception:
|
|
||||||
n_skip += 1
|
|
||||||
continue
|
|
||||||
|
|
||||||
per_file = {}
|
|
||||||
for ch in ("Tran", "Vert", "Long"):
|
|
||||||
sc_counts = [int(round(v / GEO_LSB)) for v in sc[ch]]
|
|
||||||
dec = res.samples.get(ch, [])
|
|
||||||
n = min(len(sc_counts), len(dec))
|
|
||||||
if n == 0:
|
|
||||||
per_file[ch] = 0.0
|
|
||||||
continue
|
|
||||||
exact = sum(1 for i in range(n) if sc_counts[i] == dec[i])
|
|
||||||
pct = 100.0 * exact / n
|
|
||||||
per_file[ch] = pct
|
|
||||||
overall[ch].append(pct)
|
|
||||||
n_ok += 1
|
|
||||||
|
|
||||||
print(f"Processed {n_ok} files (skipped {n_skip})")
|
|
||||||
print("Per-channel exact-match % (mean / min / max):")
|
|
||||||
for ch, vals in overall.items():
|
|
||||||
if vals:
|
|
||||||
avg = sum(vals) / len(vals)
|
|
||||||
print(f" {ch}: mean={avg:.2f}% min={min(vals):.2f}% max={max(vals):.2f}% n={len(vals)}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
"""Find where decoded-vs-sidecar diverges for each channel."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from minimateplus.waveform_codec import decode_waveform_v2
|
|
||||||
from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
buf = TARGET.read_bytes()
|
|
||||||
sc = load_sidecar_samples(TXT)
|
|
||||||
decoded = decode_waveform_v2(buf[0x0f1f:])
|
|
||||||
GEO_LSB = 0.0003
|
|
||||||
|
|
||||||
for ch in ("Tran", "Vert", "Long"):
|
|
||||||
sc_counts = [int(round(v / GEO_LSB)) for v in sc[ch]]
|
|
||||||
dec = decoded[ch]
|
|
||||||
# Find ALL transitions where mismatches start/stop
|
|
||||||
first_diff = next((i for i in range(len(dec)) if dec[i] != sc_counts[i]), None)
|
|
||||||
if first_diff is None:
|
|
||||||
print(f"{ch}: NO MISMATCHES")
|
|
||||||
continue
|
|
||||||
print(f"{ch}: first diff at idx {first_diff}")
|
|
||||||
# Show 5 before, 5 after
|
|
||||||
for i in range(max(0, first_diff - 3), min(len(dec), first_diff + 8)):
|
|
||||||
mark = " " if dec[i] == sc_counts[i] else "**"
|
|
||||||
print(f" {mark} idx {i:4d}: sc={sc_counts[i]:6d} dec={dec[i]:6d} diff={dec[i]-sc_counts[i]:+d}")
|
|
||||||
# Where does cumulative diff exceed 100?
|
|
||||||
cum_match_run = 0
|
|
||||||
max_match_run = 0
|
|
||||||
match_run_start = 0
|
|
||||||
diff_count = 0
|
|
||||||
for i in range(len(dec)):
|
|
||||||
if dec[i] == sc_counts[i]:
|
|
||||||
cum_match_run += 1
|
|
||||||
max_match_run = max(max_match_run, cum_match_run)
|
|
||||||
else:
|
|
||||||
cum_match_run = 0
|
|
||||||
diff_count += 1
|
|
||||||
print(f" total mismatches: {diff_count}/{len(dec)}, longest run of matches: {max_match_run}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
"""End-to-end IDFH ingest verification."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
import json
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
idfh = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM13981/UM13981_20220805075441.IDFH"
|
|
||||||
txt = idfh.parent / "TXT" / f"{idfh.name}.txt"
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as td:
|
|
||||||
store = WaveformStore(Path(td))
|
|
||||||
ev, rec = store.save_imported_idf(
|
|
||||||
idfh.read_bytes(),
|
|
||||||
idfh,
|
|
||||||
idf_report_text=txt.read_text(errors="replace"),
|
|
||||||
)
|
|
||||||
print("=== save_imported_idf (IDFH) ===")
|
|
||||||
print(f" serial: {rec['serial']}")
|
|
||||||
print(f" filename: {rec['filename']}")
|
|
||||||
print(f" filesize: {rec['filesize']}")
|
|
||||||
print(f" h5: {rec['hdf5_filename']}") # expect None for histogram
|
|
||||||
print(f" sidecar: {rec['sidecar_filename']}")
|
|
||||||
print()
|
|
||||||
print("=== Event ===")
|
|
||||||
print(f" timestamp: {ev.timestamp}")
|
|
||||||
print(f" record_type: {ev.record_type}")
|
|
||||||
print(f" sample_rate: {ev.sample_rate}")
|
|
||||||
print()
|
|
||||||
# Inspect sidecar to confirm intervals were stashed
|
|
||||||
sc_path = Path(td) / "UM13981" / f"{idfh.name}.sfm.json"
|
|
||||||
sc = json.loads(sc_path.read_text())
|
|
||||||
intervals = sc.get("extensions", {}).get("idf_intervals", [])
|
|
||||||
print(f" sidecar intervals: {len(intervals)}")
|
|
||||||
if intervals:
|
|
||||||
print(f" first interval: {intervals[0]}")
|
|
||||||
print(f" last interval: {intervals[-1]}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
"""Verify the had_report=False path: ingest IDFW with no .txt."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
import tempfile
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
idfw = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162723.IDFW"
|
|
||||||
with tempfile.TemporaryDirectory() as td:
|
|
||||||
store = WaveformStore(Path(td))
|
|
||||||
ev, rec = store.save_imported_idf(
|
|
||||||
idfw.read_bytes(),
|
|
||||||
idfw,
|
|
||||||
serial_hint=None,
|
|
||||||
idf_report_text=None, # ← no .txt!
|
|
||||||
)
|
|
||||||
print("=== IDFW without .txt ingest ===")
|
|
||||||
print(f" serial: {rec['serial']}")
|
|
||||||
print(f" timestamp: {ev.timestamp}")
|
|
||||||
print(f" sample_rate: {ev.sample_rate}")
|
|
||||||
print(f" record_type: {ev.record_type}")
|
|
||||||
print(f" rectime_sec: {ev.rectime_seconds}")
|
|
||||||
nT = len(ev.raw_samples.get('Tran', [])) if ev.raw_samples else 0
|
|
||||||
nV = len(ev.raw_samples.get('Vert', [])) if ev.raw_samples else 0
|
|
||||||
nL = len(ev.raw_samples.get('Long', [])) if ev.raw_samples else 0
|
|
||||||
nM = len(ev.raw_samples.get('MicL', [])) if ev.raw_samples else 0
|
|
||||||
print(f" raw_samples: Tran={nT} Vert={nV} Long={nL} MicL={nM}")
|
|
||||||
if ev.peak_values:
|
|
||||||
print(f" peak_values: tran={ev.peak_values.tran} vert={ev.peak_values.vert} long={ev.peak_values.long}")
|
|
||||||
print(f" h5 written: {rec['hdf5_filename']}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,102 +0,0 @@
|
|||||||
"""End-to-end Thor report PDF rendering.
|
|
||||||
|
|
||||||
Ingests an IDFW + .txt via save_imported_idf, runs gather_report_data
|
|
||||||
(faking a minimal DB row), and renders the PDF to disk.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
import json
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
from sfm import report_pdf
|
|
||||||
|
|
||||||
|
|
||||||
class FakeDb:
|
|
||||||
"""Stand-in for SeismoDb.get_event(); the renderer only needs a few cols."""
|
|
||||||
def __init__(self, event):
|
|
||||||
self.event = event
|
|
||||||
|
|
||||||
def get_event(self, _id):
|
|
||||||
return self.event
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
base = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719"
|
|
||||||
idfw = base / "UM11719_20231219162723.IDFW"
|
|
||||||
txt = base / "TXT" / f"{idfw.name}.txt"
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as td:
|
|
||||||
store = WaveformStore(Path(td))
|
|
||||||
ev, rec = store.save_imported_idf(
|
|
||||||
idfw.read_bytes(),
|
|
||||||
idfw,
|
|
||||||
idf_report_text=txt.read_text(errors="replace"),
|
|
||||||
)
|
|
||||||
print(f"save_imported_idf: h5={rec['hdf5_filename']}, sidecar={rec['sidecar_filename']}")
|
|
||||||
|
|
||||||
# Verify sidecar has bw_report block
|
|
||||||
sc_path = Path(td) / "UM11719" / f"{idfw.name}.sfm.json"
|
|
||||||
sc = json.loads(sc_path.read_text())
|
|
||||||
bw = sc.get("bw_report", {})
|
|
||||||
print(f" bw_report.available: {bw.get('available')}")
|
|
||||||
print(f" bw_report.peaks.tran.ppv_ips: {bw.get('peaks', {}).get('tran', {}).get('ppv_ips')}")
|
|
||||||
print(f" bw_report.mic.pspl_dbl: {bw.get('mic', {}).get('pspl_dbl')}")
|
|
||||||
print(f" bw_report.histogram.n_intervals: {bw.get('histogram', {}).get('n_intervals')}")
|
|
||||||
|
|
||||||
# Build a DB-row-shaped dict from the Event for gather_report_data
|
|
||||||
import datetime
|
|
||||||
ts = ev.timestamp
|
|
||||||
ts_iso = None
|
|
||||||
if ts is not None:
|
|
||||||
try:
|
|
||||||
ts_iso = datetime.datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second).isoformat()
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
fake_row = {
|
|
||||||
"serial": "UM11719",
|
|
||||||
"blastware_filename": rec["filename"],
|
|
||||||
"record_type": "Waveform",
|
|
||||||
"timestamp": ts_iso,
|
|
||||||
"sample_rate": ev.sample_rate,
|
|
||||||
"project": ev.project_info.project if ev.project_info else None,
|
|
||||||
"client": ev.project_info.client if ev.project_info else None,
|
|
||||||
"operator": ev.project_info.operator if ev.project_info else None,
|
|
||||||
"sensor_location": ev.project_info.sensor_location if ev.project_info else None,
|
|
||||||
"created_at": None,
|
|
||||||
}
|
|
||||||
|
|
||||||
rd = report_pdf.gather_report_data(FakeDb(fake_row), store, event_id="test-1")
|
|
||||||
print()
|
|
||||||
print(f"=== ReportData ===")
|
|
||||||
print(f" event_id: {rd.event_id}")
|
|
||||||
print(f" serial: {rd.serial}")
|
|
||||||
print(f" record_type: {rd.record_type}")
|
|
||||||
print(f" event_datetime: {rd.event_datetime_str}")
|
|
||||||
print(f" trigger: {rd.trigger_source}")
|
|
||||||
print(f" geo_range: {rd.geo_range_str}")
|
|
||||||
print(f" sample_rate: {rd.sample_rate_str}")
|
|
||||||
print(f" firmware: {rd.firmware}")
|
|
||||||
print(f" calibration: {rd.calibration_date} by {rd.calibration_by}")
|
|
||||||
print(f" battery: {rd.battery_volts}")
|
|
||||||
print(f" PVS: {rd.peak_vector_sum_ips} in/s at {rd.peak_vector_sum_time_s} sec")
|
|
||||||
print(f" mic_pspl_dbl: {rd.mic_pspl_dbl}")
|
|
||||||
print(f" mic_zc_freq_hz: {rd.mic_zc_freq_hz}")
|
|
||||||
print(f" channel_stats: {len(rd.channel_stats)} rows")
|
|
||||||
for cs in rd.channel_stats:
|
|
||||||
print(f" {cs['name']}: PPV={cs['ppv_ips']} ZC={cs['zc_freq_hz']} ToP={cs['time_of_peak_s']} Acc={cs['peak_accel_g']} Disp={cs['peak_disp_in']} Test={cs['sensor_check']}")
|
|
||||||
|
|
||||||
# Render the PDF
|
|
||||||
out_path = REPO / "analysis_idf" / "thor_report.pdf"
|
|
||||||
pdf_bytes = report_pdf.render_event_report_pdf(rd)
|
|
||||||
out_path.write_bytes(pdf_bytes)
|
|
||||||
print()
|
|
||||||
print(f" PDF written: {out_path} ({len(pdf_bytes)} bytes)")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,91 +0,0 @@
|
|||||||
"""End-to-end Thor IDFH histogram report PDF rendering."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
import json
|
|
||||||
import datetime
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
from sfm import report_pdf
|
|
||||||
|
|
||||||
|
|
||||||
class FakeDb:
|
|
||||||
def __init__(self, event):
|
|
||||||
self.event = event
|
|
||||||
|
|
||||||
def get_event(self, _id):
|
|
||||||
return self.event
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
# Use the multi-interval IDFH (81 + trigger row)
|
|
||||||
idfh = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM13981/UM13981_20220805075441.IDFH"
|
|
||||||
txt = idfh.parent / "TXT" / f"{idfh.name}.txt"
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as td:
|
|
||||||
store = WaveformStore(Path(td))
|
|
||||||
ev, rec = store.save_imported_idf(
|
|
||||||
idfh.read_bytes(),
|
|
||||||
idfh,
|
|
||||||
idf_report_text=txt.read_text(errors="replace"),
|
|
||||||
)
|
|
||||||
print(f"save_imported_idf: h5={rec['hdf5_filename']}, sidecar={rec['sidecar_filename']}")
|
|
||||||
|
|
||||||
sc_path = Path(td) / "UM13981" / f"{idfh.name}.sfm.json"
|
|
||||||
sc = json.loads(sc_path.read_text())
|
|
||||||
bw = sc.get("bw_report", {})
|
|
||||||
hist = bw.get("histogram", {})
|
|
||||||
print(f" bw_report.histogram.start: {hist.get('start')}")
|
|
||||||
print(f" bw_report.histogram.stop: {hist.get('stop')}")
|
|
||||||
print(f" bw_report.histogram.n_intervals: {hist.get('n_intervals')}")
|
|
||||||
print(f" bw_report.histogram.interval_size: {hist.get('interval_size')}")
|
|
||||||
print(f" bw_report.histogram.interval_size_s: {hist.get('interval_size_s')}")
|
|
||||||
print(f" bw_report.peaks.tran.ppv_ips: {bw.get('peaks', {}).get('tran', {}).get('ppv_ips')}")
|
|
||||||
|
|
||||||
ts = ev.timestamp
|
|
||||||
ts_iso = None
|
|
||||||
if ts is not None:
|
|
||||||
try:
|
|
||||||
ts_iso = datetime.datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second).isoformat()
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
fake_row = {
|
|
||||||
"serial": "UM13981",
|
|
||||||
"blastware_filename": rec["filename"],
|
|
||||||
"record_type": "Histogram",
|
|
||||||
"timestamp": ts_iso,
|
|
||||||
"sample_rate": ev.sample_rate,
|
|
||||||
"project": ev.project_info.project if ev.project_info else None,
|
|
||||||
"client": ev.project_info.client if ev.project_info else None,
|
|
||||||
"operator": ev.project_info.operator if ev.project_info else None,
|
|
||||||
"sensor_location": ev.project_info.sensor_location if ev.project_info else None,
|
|
||||||
"created_at": None,
|
|
||||||
}
|
|
||||||
rd = report_pdf.gather_report_data(FakeDb(fake_row), store, event_id="hist-1")
|
|
||||||
|
|
||||||
print()
|
|
||||||
print("=== ReportData (histogram) ===")
|
|
||||||
print(f" is_histogram: {rd.is_histogram}")
|
|
||||||
print(f" histogram_start: {rd.histogram_start_str}")
|
|
||||||
print(f" histogram_stop: {rd.histogram_stop_str}")
|
|
||||||
print(f" histogram_n_intervals: {rd.histogram_n_intervals}")
|
|
||||||
print(f" histogram_interval_size:{rd.histogram_interval_size}")
|
|
||||||
print(f" histogram_interval_times[:3]: {rd.histogram_interval_times[:3]}")
|
|
||||||
print(f" histogram_interval_times[-2:]: {rd.histogram_interval_times[-2:]}")
|
|
||||||
print(f" channel_stats: {len(rd.channel_stats)} rows")
|
|
||||||
for cs in rd.channel_stats:
|
|
||||||
print(f" {cs['name']}: PPV={cs['ppv_ips']} ZC={cs['zc_freq_hz']} peak_date={cs['peak_date']} peak_time={cs['peak_time']}")
|
|
||||||
|
|
||||||
pdf_bytes = report_pdf.render_event_report_pdf(rd)
|
|
||||||
out_path = REPO / "analysis_idf" / "thor_report_idfh.pdf"
|
|
||||||
out_path.write_bytes(pdf_bytes)
|
|
||||||
print()
|
|
||||||
print(f" PDF written: {out_path} ({len(pdf_bytes)} bytes)")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,52 +0,0 @@
|
|||||||
"""End-to-end ingest test: feed an IDFW + .txt to save_imported_idf in a tmp store."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
import tempfile
|
|
||||||
import shutil
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
idfw = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162723.IDFW"
|
|
||||||
txt = idfw.parent / "TXT" / f"{idfw.name}.txt"
|
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory() as td:
|
|
||||||
store = WaveformStore(Path(td))
|
|
||||||
ev, rec = store.save_imported_idf(
|
|
||||||
idfw.read_bytes(),
|
|
||||||
idfw,
|
|
||||||
serial_hint=None,
|
|
||||||
idf_report_text=txt.read_text(errors="replace"),
|
|
||||||
)
|
|
||||||
print("=== Save result ===")
|
|
||||||
print(f" serial: {rec['serial']}")
|
|
||||||
print(f" filename: {rec['filename']}")
|
|
||||||
print(f" filesize: {rec['filesize']}")
|
|
||||||
print(f" h5: {rec['hdf5_filename']}")
|
|
||||||
print(f" sidecar: {rec['sidecar_filename']}")
|
|
||||||
print()
|
|
||||||
print("=== Event ===")
|
|
||||||
print(f" serial: {ev.serial if hasattr(ev,'serial') else '(n/a)'}")
|
|
||||||
print(f" timestamp: {ev.timestamp}")
|
|
||||||
print(f" sample_rate: {ev.sample_rate}")
|
|
||||||
print(f" record_type: {ev.record_type}")
|
|
||||||
print(f" rectime_sec: {ev.rectime_seconds}")
|
|
||||||
print(f" raw_samples: Tran={len(ev.raw_samples.get('Tran', [])) if ev.raw_samples else 0}, Vert={len(ev.raw_samples.get('Vert', [])) if ev.raw_samples else 0}, Long={len(ev.raw_samples.get('Long', [])) if ev.raw_samples else 0}, MicL={len(ev.raw_samples.get('MicL', [])) if ev.raw_samples else 0}")
|
|
||||||
if ev.peak_values:
|
|
||||||
print(f" peaks (txt): Tran={ev.peak_values.tran} Vert={ev.peak_values.vert} Long={ev.peak_values.long}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Verify the h5 file actually got written
|
|
||||||
h5path = Path(td) / "UM11719" / f"{idfw.name}.h5"
|
|
||||||
print(f" h5 exists: {h5path.exists()} size={h5path.stat().st_size if h5path.exists() else 0}")
|
|
||||||
sidecar = Path(td) / "UM11719" / f"{idfw.name}.sfm.json"
|
|
||||||
print(f" sidecar exists:{sidecar.exists()} size={sidecar.stat().st_size if sidecar.exists() else 0}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,137 +0,0 @@
|
|||||||
"""Decode IDFH histogram intervals + verify against sidecar."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
import struct
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
|
|
||||||
SEGMENT_MAGIC = b"\x02\xda\x0a\x00\x00\x00"
|
|
||||||
SEGMENT_SIZE = 732 # = 10-byte header + 10 × 72-byte intervals + 2-byte tail
|
|
||||||
INTERVAL_SIZE = 72
|
|
||||||
CHANNELS = ("Tran", "Vert", "Long", "MicL")
|
|
||||||
|
|
||||||
|
|
||||||
def decode_interval(buf72: bytes) -> dict:
|
|
||||||
"""Decode one 72-byte interval into per-channel min/max/halfp."""
|
|
||||||
out = {}
|
|
||||||
for i, ch in enumerate(CHANNELS):
|
|
||||||
block = buf72[i*16 : (i+1)*16]
|
|
||||||
mn = struct.unpack_from(">h", block, 0)[0]
|
|
||||||
mx = struct.unpack_from(">h", block, 2)[0]
|
|
||||||
sb = struct.unpack_from(">h", block, 4)[0]
|
|
||||||
halfp = struct.unpack_from(">H", block, 6)[0]
|
|
||||||
f10 = struct.unpack_from(">H", block, 10)[0]
|
|
||||||
f14 = struct.unpack_from(">H", block, 14)[0]
|
|
||||||
peak_count = max(abs(mn), abs(mx))
|
|
||||||
out[ch] = {
|
|
||||||
"min": mn,
|
|
||||||
"max": mx,
|
|
||||||
"field4": sb,
|
|
||||||
"halfp": halfp,
|
|
||||||
"field10": f10,
|
|
||||||
"field14": f14,
|
|
||||||
"peak": peak_count,
|
|
||||||
"freq_hz": (512.0 / halfp) if halfp > 5 else None,
|
|
||||||
}
|
|
||||||
out["_tail"] = buf72[64:].hex(" ")
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def walk_idfh(buf: bytes) -> list:
|
|
||||||
"""Walk all interval records in an IDFH file."""
|
|
||||||
intervals = []
|
|
||||||
# Multi-segment file: every 02 da 0a 00 00 00 marker introduces a segment.
|
|
||||||
# Single-interval file: just one body header at 0xf96 of form ?? ?? 0a 00 00 00.
|
|
||||||
# Find them all.
|
|
||||||
i = 0
|
|
||||||
while True:
|
|
||||||
j = buf.find(b"\x0a\x00\x00\x00", i)
|
|
||||||
if j < 0:
|
|
||||||
break
|
|
||||||
# Validate: the 2 bytes before must form a length, and we want bytes
|
|
||||||
# [j-2 : j+6] to have a recognisable shape. Actually the cleanest
|
|
||||||
# filter is "preceded by a length and followed by 00 NN 05 3f".
|
|
||||||
if j < 2:
|
|
||||||
i = j + 1
|
|
||||||
continue
|
|
||||||
# Body header form: [length_be_2][0a 00 00 00][00 NN][05 3f]
|
|
||||||
if j + 10 > len(buf):
|
|
||||||
break
|
|
||||||
length = int.from_bytes(buf[j-2:j], "big")
|
|
||||||
# Verify the segment-marker shape: [length_be][0a 00 00 00][00 NN][05 3f]
|
|
||||||
if buf[j+4] != 0x00:
|
|
||||||
i = j + 1
|
|
||||||
continue
|
|
||||||
if buf[j+6:j+8] != b"\x05\x3f":
|
|
||||||
i = j + 1
|
|
||||||
continue
|
|
||||||
# Header layout (10 bytes): [length_be 2B][0a 00 00 00 4B][00 NN 2B][05 3f 2B]
|
|
||||||
# Followed by N interval records of 72 bytes each, then 2 tail bytes.
|
|
||||||
# length value = (N × 72) + 10 (counts bytes from 0x0a... through interval data).
|
|
||||||
header_start = j - 2
|
|
||||||
n_intervals = (length - 10) // INTERVAL_SIZE
|
|
||||||
interval_start = header_start + 10
|
|
||||||
for k in range(n_intervals):
|
|
||||||
off = interval_start + k * INTERVAL_SIZE
|
|
||||||
if off + INTERVAL_SIZE > len(buf):
|
|
||||||
break
|
|
||||||
chunk = buf[off:off + INTERVAL_SIZE]
|
|
||||||
intervals.append({"offset": off, **decode_interval(chunk)})
|
|
||||||
i = header_start + length + 2
|
|
||||||
return intervals
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
# Test against multi-segment IDFH
|
|
||||||
target = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM13981/UM13981_20220805075441.IDFH"
|
|
||||||
sc_path = target.parent / "TXT" / f"{target.name}.txt"
|
|
||||||
buf = target.read_bytes()
|
|
||||||
intervals = walk_idfh(buf)
|
|
||||||
print(f"=== {target.name} ===")
|
|
||||||
print(f" file size: {len(buf)}")
|
|
||||||
print(f" decoded intervals: {len(intervals)}")
|
|
||||||
# Show first 2 + last 2
|
|
||||||
sc_rows = []
|
|
||||||
for line in sc_path.read_text(errors="replace").splitlines():
|
|
||||||
if line.startswith("2022-") or line.startswith("2023-"):
|
|
||||||
sc_rows.append(line)
|
|
||||||
print(f" sidecar rows: {len(sc_rows)}")
|
|
||||||
|
|
||||||
print()
|
|
||||||
for k in [0, 1, 78, 79, 80]:
|
|
||||||
if k >= len(intervals):
|
|
||||||
continue
|
|
||||||
iv = intervals[k]
|
|
||||||
print(f"--- interval {k} @0x{iv['offset']:04x} ---")
|
|
||||||
for ch in CHANNELS:
|
|
||||||
d = iv[ch]
|
|
||||||
peak_ips = d["peak"] / 32768 * 10.0
|
|
||||||
print(f" {ch}: peak={d['peak']:5d} ({peak_ips:.4f} in/s) halfp={d['halfp']:5d} freq={d['freq_hz']}")
|
|
||||||
# sidecar row
|
|
||||||
if k < len(sc_rows):
|
|
||||||
print(f" SC: {sc_rows[k]}")
|
|
||||||
|
|
||||||
# Test single-interval IDFH
|
|
||||||
print()
|
|
||||||
target2 = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162648.IDFH"
|
|
||||||
sc2 = target2.parent / "TXT" / f"{target2.name}.txt"
|
|
||||||
buf2 = target2.read_bytes()
|
|
||||||
intervals2 = walk_idfh(buf2)
|
|
||||||
print(f"=== {target2.name} ===")
|
|
||||||
print(f" file size: {len(buf2)}, decoded intervals: {len(intervals2)}")
|
|
||||||
if intervals2:
|
|
||||||
iv = intervals2[0]
|
|
||||||
for ch in CHANNELS:
|
|
||||||
d = iv[ch]
|
|
||||||
peak_ips = d["peak"] / 32768 * 10.0
|
|
||||||
print(f" {ch}: peak={d['peak']:5d} ({peak_ips:.4f} in/s) halfp={d['halfp']:5d} freq={d['freq_hz']}")
|
|
||||||
sc_rows2 = [l for l in sc2.read_text(errors='replace').splitlines() if l.startswith("2023-")]
|
|
||||||
if sc_rows2:
|
|
||||||
print(f" SC: {sc_rows2[0]}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,41 +0,0 @@
|
|||||||
"""Find IDFH interval period via auto-correlation of structural patterns."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
from collections import Counter
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
target = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM13981/UM13981_20220805075441.IDFH"
|
|
||||||
buf = target.read_bytes()
|
|
||||||
body_start = 0xF96
|
|
||||||
body_end = 0x270C
|
|
||||||
body = buf[body_start:body_end]
|
|
||||||
print(f"body size: {len(body)} bytes (file {len(buf)} bytes)")
|
|
||||||
|
|
||||||
# For each candidate interval size, count how many bytes at fixed offsets within
|
|
||||||
# each interval are zero (consistent column-zero pattern indicates correct size).
|
|
||||||
print()
|
|
||||||
print("=== zero-column score by interval size (higher = more likely) ===")
|
|
||||||
best = []
|
|
||||||
for sz in range(16, 100):
|
|
||||||
n = len(body) // sz
|
|
||||||
if n < 30:
|
|
||||||
continue
|
|
||||||
# For each column position within an interval, count how many of n intervals have zero
|
|
||||||
score = 0
|
|
||||||
for col in range(sz):
|
|
||||||
zeros = sum(1 for i in range(n) if body[i*sz + col] == 0)
|
|
||||||
if zeros >= n * 0.9:
|
|
||||||
score += 1
|
|
||||||
best.append((score, sz, n))
|
|
||||||
best.sort(reverse=True)
|
|
||||||
for score, sz, n in best[:10]:
|
|
||||||
print(f" size={sz:3d} n_intervals={n} consistently-zero-cols={score}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
"""Per-file accuracy + sample-count details."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from micromate.idf_file import read_idf_file
|
|
||||||
from analysis_idf.recon import load_sidecar_samples
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
root = REPO / "tests/fixtures/THORDATA_example"
|
|
||||||
files = sorted([f for f in root.rglob("*.IDFW") if not str(f).endswith(".CDB")])
|
|
||||||
GEO_LSB = 0.0003
|
|
||||||
# Limit to first 15 successful files for detail.
|
|
||||||
shown = 0
|
|
||||||
for f in files:
|
|
||||||
try:
|
|
||||||
res = read_idf_file(f)
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
sc_path = f.parent / "TXT" / f"{f.name}.txt"
|
|
||||||
if not sc_path.exists():
|
|
||||||
continue
|
|
||||||
sc = load_sidecar_samples(sc_path)
|
|
||||||
sc_tran = [int(round(v / GEO_LSB)) for v in sc["Tran"]]
|
|
||||||
dec = res.samples.get("Tran", [])
|
|
||||||
n = min(len(sc_tran), len(dec))
|
|
||||||
exact = sum(1 for i in range(n) if sc_tran[i] == dec[i]) if n else 0
|
|
||||||
pct = 100.0 * exact / n if n else 0.0
|
|
||||||
print(f"{f.name:40s} size={f.stat().st_size:6d} sc_n={len(sc_tran):4d} dec_n={len(dec):4d} exact={pct:.1f}%")
|
|
||||||
shown += 1
|
|
||||||
if shown >= 20:
|
|
||||||
break
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,64 +0,0 @@
|
|||||||
"""Look at what's at the divergence boundary."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start, parse_segment_header
|
|
||||||
from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
buf = TARGET.read_bytes()
|
|
||||||
body = buf[0x0f1f:]
|
|
||||||
start = find_data_start(body)
|
|
||||||
print(f"data_start: {start} (= file offset 0x{0x0f1f + start:04x})")
|
|
||||||
|
|
||||||
blocks = walk_body(body, start)
|
|
||||||
print(f"{len(blocks)} blocks total")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# First 25 blocks
|
|
||||||
print("=== first 30 blocks ===")
|
|
||||||
for i, b in enumerate(blocks[:30]):
|
|
||||||
body_off = 0x0f1f + b.offset
|
|
||||||
if b.tag_hi == 0x40:
|
|
||||||
hdr = parse_segment_header(b)
|
|
||||||
print(f" [{i:3d}] @0x{body_off:04x} {b.kind} (segment header) counter={hdr['counter'] if hdr else '?'} field2={hdr['field2'].hex() if hdr else '?'} anchor={hdr['anchor_bytes'].hex() if hdr else '?'} tail={hdr['tail'].hex() if hdr else '?'}")
|
|
||||||
else:
|
|
||||||
print(f" [{i:3d}] @0x{body_off:04x} {b.kind} len={b.length} data={b.data[:16].hex()}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Cumulative sample counts per block to find which block contains sample 254
|
|
||||||
print("=== cumulative samples through blocks ===")
|
|
||||||
cur_ch = "Tran"
|
|
||||||
rotation = ["Vert", "Long", "MicL", "Tran"]
|
|
||||||
seg_count = 0
|
|
||||||
samples_in_curseg = 2 # preamble Tran[0], Tran[1]
|
|
||||||
for i, b in enumerate(blocks[:30]):
|
|
||||||
if b.tag_hi == 0x40:
|
|
||||||
seg_count += 1
|
|
||||||
prev_ch = cur_ch
|
|
||||||
cur_ch = rotation[(seg_count - 1) % 4]
|
|
||||||
print(f" [{i:3d}] 40 02 -> end of {prev_ch} segment, start {cur_ch} (segment {seg_count})")
|
|
||||||
samples_in_curseg = 2 # anchors
|
|
||||||
elif (b.tag_hi & 0xF0) == 0x10:
|
|
||||||
nn = ((b.tag_hi & 0x0F) << 8) | b.tag_lo
|
|
||||||
samples_in_curseg += nn
|
|
||||||
print(f" [{i:3d}] {b.kind} nibble: +{nn} samples, ch={cur_ch}, ch_total~{samples_in_curseg}")
|
|
||||||
elif (b.tag_hi & 0xF0) == 0x20:
|
|
||||||
nn = ((b.tag_hi & 0x0F) << 8) | b.tag_lo
|
|
||||||
samples_in_curseg += nn
|
|
||||||
print(f" [{i:3d}] {b.kind} int8: +{nn} samples, ch={cur_ch}, ch_total~{samples_in_curseg}")
|
|
||||||
elif b.tag_hi == 0x00:
|
|
||||||
samples_in_curseg += b.tag_lo
|
|
||||||
print(f" [{i:3d}] {b.kind} RLE: +{b.tag_lo}, ch={cur_ch}, ch_total~{samples_in_curseg}")
|
|
||||||
elif b.tag_hi == 0x30:
|
|
||||||
samples_in_curseg += b.tag_lo
|
|
||||||
print(f" [{i:3d}] {b.kind} packed12: +{b.tag_lo} samples, ch={cur_ch}, ch_total~{samples_in_curseg}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,89 +0,0 @@
|
|||||||
"""Reconnaissance helpers for cracking the Thor IDFW binary."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
TARGET = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162723.IDFW"
|
|
||||||
TXT = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/TXT/UM11719_20231219162723.IDFW.txt"
|
|
||||||
|
|
||||||
|
|
||||||
def hex_at(buf: bytes, off: int, n: int = 32) -> str:
|
|
||||||
chunk = buf[off : off + n]
|
|
||||||
hexs = " ".join(f"{b:02x}" for b in chunk)
|
|
||||||
asc = "".join(chr(b) if 32 <= b < 127 else "." for b in chunk)
|
|
||||||
return f"{off:04x}: {hexs} {asc}"
|
|
||||||
|
|
||||||
|
|
||||||
def find_all(buf: bytes, needle: bytes) -> list[int]:
|
|
||||||
out: list[int] = []
|
|
||||||
i = 0
|
|
||||||
while True:
|
|
||||||
j = buf.find(needle, i)
|
|
||||||
if j < 0:
|
|
||||||
break
|
|
||||||
out.append(j)
|
|
||||||
i = j + 1
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def load_sidecar_samples(path: Path) -> dict[str, list[float]]:
|
|
||||||
"""Parse the txt sample table — Tran/Vert/Long/MicL."""
|
|
||||||
out = {"Tran": [], "Vert": [], "Long": [], "MicL": []}
|
|
||||||
in_block = False
|
|
||||||
for line in path.read_text(errors="replace").splitlines():
|
|
||||||
if not in_block:
|
|
||||||
if line.strip() == "Waveform Data Channels":
|
|
||||||
in_block = True
|
|
||||||
continue
|
|
||||||
if line.startswith("Waveform Data USB Channels"):
|
|
||||||
break
|
|
||||||
parts = line.split("\t")
|
|
||||||
# First row is the header "\tTran\tVert\tLong\tMicL"
|
|
||||||
if len(parts) >= 5 and parts[1] == "Tran":
|
|
||||||
continue
|
|
||||||
if len(parts) < 5:
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
out["Tran"].append(float(parts[1]))
|
|
||||||
out["Vert"].append(float(parts[2]))
|
|
||||||
out["Long"].append(float(parts[3]))
|
|
||||||
out["MicL"].append(float(parts[4]))
|
|
||||||
except ValueError:
|
|
||||||
continue
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
buf = TARGET.read_bytes()
|
|
||||||
samples = load_sidecar_samples(TXT)
|
|
||||||
print(f"file size: {len(buf)} bytes")
|
|
||||||
print(f"sample rows: Tran={len(samples['Tran'])} Vert={len(samples['Vert'])} Long={len(samples['Long'])} MicL={len(samples['MicL'])}")
|
|
||||||
print(f"first 6 Tran samples: {samples['Tran'][:6]}")
|
|
||||||
print(f"first 6 Vert samples: {samples['Vert'][:6]}")
|
|
||||||
print(f"first 6 Long samples: {samples['Long'][:6]}")
|
|
||||||
print(f"first 6 MicL samples: {samples['MicL'][:6]}")
|
|
||||||
|
|
||||||
print()
|
|
||||||
print("=== BW magic '00 02 00' positions ===")
|
|
||||||
hits = find_all(buf, b"\x00\x02\x00")
|
|
||||||
print(f"{len(hits)} hits")
|
|
||||||
for h in hits[:20]:
|
|
||||||
print(hex_at(buf, h, 24))
|
|
||||||
|
|
||||||
print()
|
|
||||||
print("=== '40 02' segment-header positions ===")
|
|
||||||
hits = find_all(buf, b"\x40\x02")
|
|
||||||
print(f"{len(hits)} hits")
|
|
||||||
for h in hits:
|
|
||||||
ctx_pre = buf[max(0, h - 4): h].hex()
|
|
||||||
ctx_post = buf[h: h + 20].hex()
|
|
||||||
# Show byte preceding to help identify real headers vs casual occurrences
|
|
||||||
print(f" 0x{h:04x} pre={ctx_pre} post={ctx_post}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,40 +0,0 @@
|
|||||||
"""Find each segment boundary in the channel and check if errors reset there."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from minimateplus.waveform_codec import decode_waveform_v2
|
|
||||||
from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
buf = TARGET.read_bytes()
|
|
||||||
sc = load_sidecar_samples(TXT)
|
|
||||||
decoded = decode_waveform_v2(buf[0x0f1f:])
|
|
||||||
GEO_LSB = 0.0003
|
|
||||||
|
|
||||||
for ch in ("Tran", "Vert", "Long"):
|
|
||||||
sc_counts = [int(round(v / GEO_LSB)) for v in sc[ch]]
|
|
||||||
dec = decoded[ch]
|
|
||||||
# Find every transition where error becomes zero from nonzero (or grows from zero)
|
|
||||||
# Print indices where dec resyncs back to exact match.
|
|
||||||
n = min(len(sc_counts), len(dec))
|
|
||||||
events = []
|
|
||||||
prev_match = True
|
|
||||||
for i in range(n):
|
|
||||||
match = sc_counts[i] == dec[i]
|
|
||||||
if match != prev_match:
|
|
||||||
kind = "RESYNC" if match else "DIVERGE"
|
|
||||||
events.append((i, kind, sc_counts[i], dec[i]))
|
|
||||||
prev_match = match
|
|
||||||
print(f"{ch}: {len(events)} transitions")
|
|
||||||
for i, kind, sc_v, dec_v in events[:20]:
|
|
||||||
print(f" idx {i:4d} {kind:8s} sc={sc_v:6d} dec={dec_v:6d} diff={dec_v-sc_v:+d}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
"""Smoke-test read_idf_file on IDFH across the corpus."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from micromate.idf_file import read_idf_file
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
target = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162648.IDFH"
|
|
||||||
result = read_idf_file(target)
|
|
||||||
ev = result.event
|
|
||||||
print(f"=== {target.name} ===")
|
|
||||||
print(f" signature: {result.signature}")
|
|
||||||
print(f" serial: {ev.serial}")
|
|
||||||
print(f" timestamp: {ev.timestamp}")
|
|
||||||
print(f" sample_rate: {ev.sample_rate}")
|
|
||||||
print(f" kind: {ev.kind}")
|
|
||||||
print(f" intervals: {len(result.intervals or [])}")
|
|
||||||
print(f" peaks: T={ev.peaks.transverse_ips:.4f} V={ev.peaks.vertical_ips:.4f} L={ev.peaks.longitudinal_ips:.4f}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
root = REPO / "tests/fixtures/THORDATA_example"
|
|
||||||
files = list(root.rglob("*.IDFH"))
|
|
||||||
ok = fail = nyi = 0
|
|
||||||
total_intervals = 0
|
|
||||||
for f in files:
|
|
||||||
try:
|
|
||||||
r = read_idf_file(f)
|
|
||||||
ok += 1
|
|
||||||
total_intervals += len(r.intervals or [])
|
|
||||||
except NotImplementedError:
|
|
||||||
nyi += 1
|
|
||||||
except Exception as exc:
|
|
||||||
fail += 1
|
|
||||||
if fail <= 3:
|
|
||||||
print(f" FAIL: {f.name}: {type(exc).__name__}: {exc}")
|
|
||||||
print(f"Corpus: {len(files)} IDFH files | ok={ok} fail={fail} nyi={nyi}")
|
|
||||||
print(f"Total intervals decoded: {total_intervals}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
"""Smoke-test read_idf_file across the sample corpus."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from micromate.idf_file import read_idf_file, geo_count_to_ips, mic_count_to_psi
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
target = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719/UM11719_20231219162723.IDFW"
|
|
||||||
result = read_idf_file(target)
|
|
||||||
ev = result.event
|
|
||||||
print(f"=== {target.name} ===")
|
|
||||||
print(f" signature: {result.signature}")
|
|
||||||
print(f" serial: {ev.serial}")
|
|
||||||
print(f" timestamp: {ev.timestamp}")
|
|
||||||
print(f" sample_rate: {ev.sample_rate}")
|
|
||||||
print(f" record_time: {ev.record_time_sec}")
|
|
||||||
print(f" calibration: {result.binary_metadata.calibration_date}")
|
|
||||||
print(f" Tran samples: {len(result.samples['Tran'])}, peak_ips={ev.peaks.transverse_ips:.4f}")
|
|
||||||
print(f" Vert samples: {len(result.samples['Vert'])}, peak_ips={ev.peaks.vertical_ips:.4f}")
|
|
||||||
print(f" Long samples: {len(result.samples['Long'])}, peak_ips={ev.peaks.longitudinal_ips:.4f}")
|
|
||||||
print(f" MicL samples: {len(result.samples['MicL'])}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Corpus sweep
|
|
||||||
root = REPO / "tests/fixtures/THORDATA_example"
|
|
||||||
files = [f for f in root.rglob("*.IDFW") if not str(f).endswith(".CDB")]
|
|
||||||
ok = fail = nyi = 0
|
|
||||||
for f in files:
|
|
||||||
try:
|
|
||||||
r = read_idf_file(f)
|
|
||||||
ok += 1
|
|
||||||
except NotImplementedError:
|
|
||||||
nyi += 1
|
|
||||||
except Exception as exc:
|
|
||||||
fail += 1
|
|
||||||
if fail <= 5:
|
|
||||||
print(f" FAIL: {f.name}: {type(exc).__name__}: {exc}")
|
|
||||||
print()
|
|
||||||
print(f"Corpus: {len(files)} IDFW files | ok={ok} fail={fail} not-implemented={nyi}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
"""Verify build_bw_report_from_idf against a known sidecar."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import json
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from micromate.idf_ascii_report import parse_idf_report
|
|
||||||
from micromate.idf_to_bw_report import build_bw_report_from_idf
|
|
||||||
from micromate.idf_file import read_idf_file
|
|
||||||
|
|
||||||
|
|
||||||
def show(prefix: str, d: dict, indent: int = 0):
|
|
||||||
for k, v in d.items():
|
|
||||||
if isinstance(v, dict):
|
|
||||||
print(f"{' '*indent}{prefix}{k}:")
|
|
||||||
show("", v, indent + 1)
|
|
||||||
else:
|
|
||||||
print(f"{' '*indent}{prefix}{k}: {v!r}")
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
base = REPO / "tests/fixtures/THORDATA_example/THORDATA_example/UPMC Presby/UM11719"
|
|
||||||
idfw = base / "UM11719_20231219162723.IDFW"
|
|
||||||
txt = base / "TXT" / f"{idfw.name}.txt"
|
|
||||||
|
|
||||||
report_dict = parse_idf_report(txt.read_text(errors="replace"))
|
|
||||||
res = read_idf_file(idfw)
|
|
||||||
bw = build_bw_report_from_idf(report_dict, binary_md=res.binary_metadata)
|
|
||||||
|
|
||||||
print("=== IDFW → bw_report ===")
|
|
||||||
show("", bw)
|
|
||||||
|
|
||||||
print()
|
|
||||||
print("=== IDFH (single trigger row) ===")
|
|
||||||
idfh = base / "UM11719_20231219162648.IDFH"
|
|
||||||
txt_h = base / "TXT" / f"{idfh.name}.txt"
|
|
||||||
rh = parse_idf_report(txt_h.read_text(errors="replace"))
|
|
||||||
res_h = read_idf_file(idfh)
|
|
||||||
bw_h = build_bw_report_from_idf(rh, binary_md=res_h.binary_metadata, intervals=res_h.intervals)
|
|
||||||
show("", bw_h)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
Binary file not shown.
Binary file not shown.
@@ -1,73 +0,0 @@
|
|||||||
"""Trace Tran sample-by-sample to find exactly where the codec drifts."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n: int) -> int:
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def i8(b: int) -> int:
|
|
||||||
return b if b < 128 else b - 256
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
buf = TARGET.read_bytes()
|
|
||||||
sc = load_sidecar_samples(TXT)
|
|
||||||
GEO_LSB = 0.0003
|
|
||||||
sc_tran = [int(round(v / GEO_LSB)) for v in sc["Tran"]]
|
|
||||||
|
|
||||||
body = buf[0x0f1f:]
|
|
||||||
# Tran[0], Tran[1] from preamble
|
|
||||||
t0 = int.from_bytes(body[3:5], "big", signed=True)
|
|
||||||
t1 = int.from_bytes(body[5:7], "big", signed=True)
|
|
||||||
print(f"preamble Tran[0]={t0} Tran[1]={t1} (sidecar: {sc_tran[0]}, {sc_tran[1]})")
|
|
||||||
|
|
||||||
# Block 0: 10 f8 at body[7:9]
|
|
||||||
print(f"block 0: tag {body[7]:02x} {body[8]:02x}")
|
|
||||||
print(f" block 0 first 10 data bytes: {body[9:19].hex()}")
|
|
||||||
|
|
||||||
# Walk block 0 manually, comparing each sample
|
|
||||||
cur = t1
|
|
||||||
samples = [t0, t1]
|
|
||||||
block_off = 7
|
|
||||||
nn = body[8]
|
|
||||||
print(f" NN = {nn}")
|
|
||||||
data = body[9 : 9 + nn // 2]
|
|
||||||
for byi, byte in enumerate(data):
|
|
||||||
for nib_idx, nib in enumerate(((byte >> 4) & 0xF, byte & 0xF)):
|
|
||||||
cur += s4(nib)
|
|
||||||
samples.append(cur)
|
|
||||||
idx = len(samples) - 1
|
|
||||||
if 0 <= idx < len(sc_tran):
|
|
||||||
sc_v = sc_tran[idx]
|
|
||||||
match = "✓" if sc_v == cur else "✗"
|
|
||||||
if idx < 12 or 240 <= idx <= 260:
|
|
||||||
print(f" idx {idx:3d}: nibble byte={byte:02x} nib={nib:x} delta={s4(nib):+d} cur={cur:+d} sc={sc_v:+d} {match}")
|
|
||||||
|
|
||||||
print(f"end of block 0: cur={cur}, len(samples)={len(samples)}, decoder expected 250 here")
|
|
||||||
# Block 1: 20 28 starts at offset 9 + 124 = 133 from block_off=7
|
|
||||||
block1_off = 9 + nn // 2
|
|
||||||
print(f"block 1: tag {body[block1_off]:02x} {body[block1_off+1]:02x} (expecting 20 28)")
|
|
||||||
nn1 = body[block1_off + 1]
|
|
||||||
print(f" block 1 NN = {nn1}")
|
|
||||||
data1 = body[block1_off + 2 : block1_off + 2 + nn1]
|
|
||||||
for byi, byte in enumerate(data1):
|
|
||||||
cur += i8(byte)
|
|
||||||
samples.append(cur)
|
|
||||||
idx = len(samples) - 1
|
|
||||||
if idx < len(sc_tran):
|
|
||||||
sc_v = sc_tran[idx]
|
|
||||||
match = "✓" if sc_v == cur else "✗"
|
|
||||||
if 248 <= idx <= 295:
|
|
||||||
print(f" idx {idx:3d}: int8 byte={byte:02x} delta={i8(byte):+d} cur={cur:+d} sc={sc_v:+d} {match}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,42 +0,0 @@
|
|||||||
"""Feed candidate body offsets to the BW codec and compare with sidecar."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from minimateplus.waveform_codec import decode_waveform_v2, walk_body, find_data_start
|
|
||||||
from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
buf = TARGET.read_bytes()
|
|
||||||
sc = load_sidecar_samples(TXT)
|
|
||||||
# Sidecar samples in 0.0003 counts (Thor geo LSB).
|
|
||||||
sc_tran = [int(round(v / 0.0003)) for v in sc["Tran"][:30]]
|
|
||||||
sc_vert = [int(round(v / 0.0003)) for v in sc["Vert"][:30]]
|
|
||||||
sc_long = [int(round(v / 0.0003)) for v in sc["Long"][:30]]
|
|
||||||
sc_micl = [int(round(v / 1e-6)) for v in sc["MicL"][:30]] # 1 µ unit for mic? Will iterate.
|
|
||||||
print(f"sidecar Tran (counts): {sc_tran}")
|
|
||||||
print(f"sidecar Vert (counts): {sc_vert}")
|
|
||||||
print(f"sidecar Long (counts): {sc_long}")
|
|
||||||
print(f"sidecar MicL (×1e-6): {sc_micl}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Try candidate body start offsets.
|
|
||||||
for off in (0x0f1f, 0x1057, 0x11f1, 0x1333, 0x1bde, 0x0d30):
|
|
||||||
print(f"=== body @ 0x{off:04x} ===")
|
|
||||||
body = buf[off:]
|
|
||||||
decoded = decode_waveform_v2(body)
|
|
||||||
if not decoded:
|
|
||||||
print(" decode_waveform_v2 returned None")
|
|
||||||
continue
|
|
||||||
for ch in ("Tran", "Vert", "Long", "MicL"):
|
|
||||||
arr = decoded.get(ch, [])
|
|
||||||
print(f" {ch}[{len(arr)}]: {arr[:20]}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
"""Verify decode_waveform_v2 against sidecar across all 2304 samples per channel."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
sys.path.insert(0, str(REPO))
|
|
||||||
|
|
||||||
from minimateplus.waveform_codec import decode_waveform_v2
|
|
||||||
from analysis_idf.recon import TARGET, TXT, load_sidecar_samples
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
buf = TARGET.read_bytes()
|
|
||||||
sc = load_sidecar_samples(TXT)
|
|
||||||
body = buf[0x0f1f:]
|
|
||||||
decoded = decode_waveform_v2(body)
|
|
||||||
|
|
||||||
print(f"Sidecar lengths: Tran={len(sc['Tran'])} Vert={len(sc['Vert'])} Long={len(sc['Long'])} MicL={len(sc['MicL'])}")
|
|
||||||
print(f"Decoded lengths: Tran={len(decoded['Tran'])} Vert={len(decoded['Vert'])} Long={len(decoded['Long'])} MicL={len(decoded['MicL'])}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
GEO_LSB = 0.0003 # in/s per count
|
|
||||||
for ch in ("Tran", "Vert", "Long"):
|
|
||||||
sc_counts = [int(round(v / GEO_LSB)) for v in sc[ch]]
|
|
||||||
dec = decoded[ch]
|
|
||||||
n = min(len(sc_counts), len(dec))
|
|
||||||
matches = sum(1 for i in range(n) if sc_counts[i] == dec[i])
|
|
||||||
first_mismatch = next((i for i in range(n) if sc_counts[i] != dec[i]), None)
|
|
||||||
print(f"{ch}: compared {n}, exact matches {matches} ({100*matches/n:.2f}%)")
|
|
||||||
if first_mismatch is not None:
|
|
||||||
i = first_mismatch
|
|
||||||
print(f" first mismatch at idx {i}: sidecar={sc_counts[i]} ({sc[ch][i]}), decoded={dec[i]}")
|
|
||||||
print(f" context sidecar[{i-2}..{i+5}]: {sc_counts[max(0,i-2):i+5]}")
|
|
||||||
print(f" context decoded[{i-2}..{i+5}]: {dec[max(0,i-2):i+5]}")
|
|
||||||
|
|
||||||
# MicL: find the multiplicative factor that fits
|
|
||||||
print()
|
|
||||||
print("=== MicL scale analysis ===")
|
|
||||||
sc_micl = sc["MicL"]
|
|
||||||
dec_micl = decoded["MicL"]
|
|
||||||
# Skip zero values when computing ratio
|
|
||||||
ratios = [sc_micl[i] / dec_micl[i] for i in range(min(50, len(sc_micl), len(dec_micl))) if dec_micl[i] != 0]
|
|
||||||
if ratios:
|
|
||||||
avg = sum(ratios) / len(ratios)
|
|
||||||
print(f" avg ratio sidecar/decoded over first 50 nonzero: {avg:.4e} (n={len(ratios)})")
|
|
||||||
print(f" ratios sample: {[f'{r:.4e}' for r in ratios[:6]]}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
+113
-217
@@ -70,78 +70,43 @@ from minimateplus.transport import SocketTransport
|
|||||||
from minimateplus.client import MiniMateClient
|
from minimateplus.client import MiniMateClient
|
||||||
from minimateplus.models import DeviceInfo, Event, MonitorLogEntry
|
from minimateplus.models import DeviceInfo, Event, MonitorLogEntry
|
||||||
from sfm.database import SeismoDb
|
from sfm.database import SeismoDb
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
|
|
||||||
log = logging.getLogger("ach_server")
|
log = logging.getLogger("ach_server")
|
||||||
|
|
||||||
# ── Per-unit state (downloaded events index) ──────────────────────────────────
|
# ── Per-unit state (downloaded-key set) ───────────────────────────────────────
|
||||||
# Persisted as <output_dir>/ach_state.json
|
# Persisted as <output_dir>/ach_state.json
|
||||||
# Format (current — v2):
|
# Format:
|
||||||
# {
|
# {
|
||||||
# "BE11529": {
|
# "BE11529": {
|
||||||
# "downloaded_events": { # key_hex → ISO timestamp string
|
# "downloaded_keys": ["01110000", "0111245a"], # hex keys already on disk
|
||||||
# "01110000": "2026-04-11T00:42:17",
|
# "max_downloaded_key": "0111245a", # highest key ever seen
|
||||||
# "0111245a": "2026-04-11T01:04:30"
|
# "last_seen": "2026-04-11T01:04:36"
|
||||||
# },
|
|
||||||
# "max_downloaded_key": "0111245a",
|
|
||||||
# "last_seen": "2026-04-11T01:04:36",
|
|
||||||
# "serial": "BE11529",
|
|
||||||
# "peer": "63.43.212.232:51920"
|
|
||||||
# }
|
# }
|
||||||
# }
|
# }
|
||||||
#
|
#
|
||||||
# Why (key, timestamp) and not key alone:
|
# Key-based deduplication works well within a single "key generation" (between
|
||||||
# The device's event-key counter resets to 0x01110000 after every memory
|
# erases). After the device memory is erased the event counter resets to
|
||||||
# erase (internal or external). A bare-key dedup (the v1 format) cannot
|
# 0x01110000, so the first new event has the SAME key as the very first event
|
||||||
# distinguish a re-recorded event with the same key from one we already
|
# we ever downloaded. We detect this situation with max_downloaded_key:
|
||||||
# downloaded. The 0C waveform record's timestamp IS unique per physical
|
|
||||||
# event, so we pair (key, timestamp) and treat a key with a different
|
|
||||||
# timestamp as a new event regardless of `max_downloaded_key`.
|
|
||||||
#
|
#
|
||||||
# Legacy v1 format (`downloaded_keys: list[str]` only) is auto-migrated on
|
# if max(current_device_keys) < max_downloaded_key
|
||||||
# read: the keys are kept under a sentinel of "" (empty string) timestamp so
|
# → device was wiped and keys have restarted → treat all device keys as new
|
||||||
# the (key, timestamp) compare always sees a mismatch and forces a one-time
|
#
|
||||||
# re-download. After that pass the state is rewritten in v2 form.
|
# After our own erase (--clear-after-download) we also explicitly clear
|
||||||
|
# downloaded_keys and max_downloaded_key so the next session starts fresh.
|
||||||
|
|
||||||
_state_lock = threading.Lock()
|
_state_lock = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
def _load_state(state_path: Path) -> dict:
|
def _load_state(state_path: Path) -> dict:
|
||||||
"""
|
if state_path.exists():
|
||||||
Load ach_state.json, transparently migrating any legacy
|
|
||||||
`downloaded_keys: list` entries into the v2 `downloaded_events: dict`
|
|
||||||
schema. Returns the migrated state.
|
|
||||||
"""
|
|
||||||
if not state_path.exists():
|
|
||||||
return {}
|
|
||||||
try:
|
try:
|
||||||
with open(state_path) as f:
|
with open(state_path) as f:
|
||||||
state = json.load(f)
|
return json.load(f)
|
||||||
except Exception:
|
except Exception:
|
||||||
|
pass
|
||||||
return {}
|
return {}
|
||||||
|
|
||||||
# Per-unit migration: legacy list → dict-with-empty-timestamps
|
|
||||||
for unit_key, unit_state in list(state.items()):
|
|
||||||
if not isinstance(unit_state, dict):
|
|
||||||
continue
|
|
||||||
if "downloaded_events" in unit_state:
|
|
||||||
continue
|
|
||||||
legacy_keys = unit_state.get("downloaded_keys")
|
|
||||||
if isinstance(legacy_keys, list):
|
|
||||||
unit_state["downloaded_events"] = {k: "" for k in legacy_keys}
|
|
||||||
log.info(
|
|
||||||
"ach_state: migrated %s from v1 (downloaded_keys list) → v2 "
|
|
||||||
"(downloaded_events dict, %d keys with empty timestamps; "
|
|
||||||
"they will re-validate on next session)",
|
|
||||||
unit_key, len(legacy_keys),
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
unit_state["downloaded_events"] = {}
|
|
||||||
# keep legacy field for one cycle; cleared on next save
|
|
||||||
unit_state.pop("downloaded_keys", None)
|
|
||||||
|
|
||||||
return state
|
|
||||||
|
|
||||||
|
|
||||||
def _save_state(state_path: Path, state: dict) -> None:
|
def _save_state(state_path: Path, state: dict) -> None:
|
||||||
with _state_lock:
|
with _state_lock:
|
||||||
@@ -174,10 +139,8 @@ class AchSession:
|
|||||||
max_events: Optional[int],
|
max_events: Optional[int],
|
||||||
state_path: Path,
|
state_path: Path,
|
||||||
db: "SeismoDb",
|
db: "SeismoDb",
|
||||||
store: "WaveformStore",
|
|
||||||
clear_after_download: bool = False,
|
clear_after_download: bool = False,
|
||||||
restart_monitoring: bool = False,
|
restart_monitoring: bool = False,
|
||||||
force_redownload: bool = False,
|
|
||||||
) -> None:
|
) -> None:
|
||||||
self.sock = sock
|
self.sock = sock
|
||||||
self.peer = peer
|
self.peer = peer
|
||||||
@@ -187,14 +150,8 @@ class AchSession:
|
|||||||
self.max_events = max_events
|
self.max_events = max_events
|
||||||
self.state_path = state_path
|
self.state_path = state_path
|
||||||
self.db = db
|
self.db = db
|
||||||
self.store = store
|
|
||||||
self.clear_after_download = clear_after_download
|
self.clear_after_download = clear_after_download
|
||||||
self.restart_monitoring = restart_monitoring
|
self.restart_monitoring = restart_monitoring
|
||||||
# `force_redownload` tells this session to ignore ach_state and
|
|
||||||
# re-download every event currently on the device, regardless of any
|
|
||||||
# (key, timestamp) match. Useful as a manual override when state has
|
|
||||||
# become inconsistent with what's actually on disk / in the DB.
|
|
||||||
self.force_redownload = force_redownload
|
|
||||||
|
|
||||||
def run(self) -> None:
|
def run(self) -> None:
|
||||||
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
|
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||||
@@ -316,20 +273,11 @@ class AchSession:
|
|||||||
state = _load_state(self.state_path)
|
state = _load_state(self.state_path)
|
||||||
unit_key = serial or self.peer # fall back to IP if no serial
|
unit_key = serial or self.peer # fall back to IP if no serial
|
||||||
unit_state = state.get(unit_key, {})
|
unit_state = state.get(unit_key, {})
|
||||||
|
seen_keys: set[str] = set(unit_state.get("downloaded_keys", []))
|
||||||
# downloaded_events is the v2 (key_hex → timestamp_iso) dict.
|
# Highest event key ever downloaded from this unit (hex string, 8 chars).
|
||||||
# Empty-string timestamps are migrated v1 entries — they force a
|
# Used to detect post-erase key reuse — see comment block above.
|
||||||
# one-time re-download because the (key, timestamp) compare always
|
|
||||||
# mismatches against any non-empty timestamp from a fresh 0C read.
|
|
||||||
seen_events: dict[str, str] = dict(unit_state.get("downloaded_events", {}))
|
|
||||||
max_seen_key: str = unit_state.get("max_downloaded_key", "00000000")
|
max_seen_key: str = unit_state.get("max_downloaded_key", "00000000")
|
||||||
|
|
||||||
if self.force_redownload:
|
|
||||||
log.info(" --force-redownload-all set — ignoring %d cached "
|
|
||||||
"(key, timestamp) entries for this session",
|
|
||||||
len(seen_events))
|
|
||||||
seen_events = {}
|
|
||||||
|
|
||||||
# Walk the event index (browse-mode, no 5A) to get the actual current
|
# Walk the event index (browse-mode, no 5A) to get the actual current
|
||||||
# key list. The SUB 08 event_count field is a lifetime "total events
|
# key list. The SUB 08 event_count field is a lifetime "total events
|
||||||
# ever recorded" counter that does NOT decrement on erase — confirmed
|
# ever recorded" counter that does NOT decrement on erase — confirmed
|
||||||
@@ -342,10 +290,11 @@ class AchSession:
|
|||||||
log.warning(" list_event_keys failed: %s -- falling back to full download", exc)
|
log.warning(" list_event_keys failed: %s -- falling back to full download", exc)
|
||||||
device_keys = None
|
device_keys = None
|
||||||
|
|
||||||
|
# Use the walk result as our authoritative current count.
|
||||||
current_count = len(device_keys) if device_keys is not None else 0
|
current_count = len(device_keys) if device_keys is not None else 0
|
||||||
|
|
||||||
log.info(" Unit has %d stored event(s); %d (key, ts) entr(ies) previously downloaded",
|
log.info(" Unit has %d stored event(s); %d key(s) previously downloaded",
|
||||||
current_count, len(seen_events))
|
current_count, len(seen_keys))
|
||||||
|
|
||||||
if device_keys is not None and current_count == 0:
|
if device_keys is not None and current_count == 0:
|
||||||
log.info(" [OK] No events on device -- nothing to download")
|
log.info(" [OK] No events on device -- nothing to download")
|
||||||
@@ -353,29 +302,75 @@ class AchSession:
|
|||||||
return
|
return
|
||||||
|
|
||||||
if device_keys is not None:
|
if device_keys is not None:
|
||||||
# ── Post-erase detection (best-effort, key-only signal) ───────
|
# ── Post-erase detection ──────────────────────────────────────
|
||||||
# After erase the device's key counter resets to 01110000.
|
# After the device memory is erased, new events start from key
|
||||||
# If the device's current max key is below our high-water mark
|
# 01110000 again — the same keys we already downloaded. Detect
|
||||||
# we know erase happened. This catches the cleanest case but
|
# this by comparing the device's current highest key against the
|
||||||
# does NOT catch erase-then-record-many-events (where the new
|
# historical maximum. If the device has rolled back below our
|
||||||
# max may climb past the old max). The (key, timestamp) check
|
# high-water mark, its counter was reset and we must treat all
|
||||||
# in get_events() is what handles those.
|
# its keys as new, regardless of what seen_keys contains.
|
||||||
if device_keys and max_seen_key != "00000000":
|
if device_keys and max_seen_key != "00000000":
|
||||||
max_device_key = max(device_keys)
|
max_device_key = max(device_keys) # lexicographic; safe because
|
||||||
|
# keys share the same 4-char prefix
|
||||||
if max_device_key < max_seen_key:
|
if max_device_key < max_seen_key:
|
||||||
log.info(
|
log.info(
|
||||||
" Post-erase reset detected: "
|
" Post-erase reset detected: "
|
||||||
"device max key %s < historical max %s "
|
"device max key %s < historical max %s "
|
||||||
"-- discarding stale (key, ts) state for this session",
|
"-- treating all device keys as new",
|
||||||
max_device_key, max_seen_key,
|
max_device_key, max_seen_key,
|
||||||
)
|
)
|
||||||
seen_events = {}
|
seen_keys = set() # discard stale dedup info for this session
|
||||||
|
|
||||||
# Note: no early-exit "all already downloaded" short-circuit
|
new_key_set = set(device_keys) - seen_keys
|
||||||
# here. Without per-event timestamps we cannot tell whether
|
log.info(" Device has %d key(s): %d new, %d already seen",
|
||||||
# device_keys ⊆ seen_events.keys() actually means we have
|
len(device_keys), len(new_key_set), len(device_keys) - len(new_key_set))
|
||||||
# those physical events. get_events() will read 0C on its
|
if not new_key_set:
|
||||||
# skip path and decide per event.
|
log.info(" [OK] All events already downloaded -- nothing to do")
|
||||||
|
# Refresh state timestamp; preserve max_seen_key unchanged.
|
||||||
|
state[unit_key] = {
|
||||||
|
"downloaded_keys": sorted(seen_keys | set(device_keys)),
|
||||||
|
"max_downloaded_key": max_seen_key,
|
||||||
|
"last_seen": datetime.datetime.now().isoformat(),
|
||||||
|
"serial": serial,
|
||||||
|
"peer": self.peer,
|
||||||
|
}
|
||||||
|
_save_state(self.state_path, state)
|
||||||
|
|
||||||
|
# ── Erase even when no new events (if requested) ──────────
|
||||||
|
# Blastware ACH always erases after every session — even when
|
||||||
|
# nothing new was downloaded. Without the erase the device
|
||||||
|
# still sees stored events in its memory and immediately
|
||||||
|
# retries the call-home, causing the looping we observed.
|
||||||
|
# Only erase when device actually has events stored; skip
|
||||||
|
# the erase if device_keys is empty (nothing to erase).
|
||||||
|
if self.clear_after_download and device_keys:
|
||||||
|
log.info(
|
||||||
|
" Clearing device memory (--clear-after-download, "
|
||||||
|
"no new events but device has %d stored)...",
|
||||||
|
len(device_keys),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
client.delete_all_events()
|
||||||
|
log.info(" [OK] Device memory cleared")
|
||||||
|
# Reset state so the next session starts fresh.
|
||||||
|
state[unit_key] = {
|
||||||
|
"downloaded_keys": [],
|
||||||
|
"max_downloaded_key": "00000000",
|
||||||
|
"last_seen": datetime.datetime.now().isoformat(),
|
||||||
|
"serial": serial,
|
||||||
|
"peer": self.peer,
|
||||||
|
}
|
||||||
|
_save_state(self.state_path, state)
|
||||||
|
except Exception as exc:
|
||||||
|
log.error(
|
||||||
|
" [WARN] Event deletion failed: %s -- events NOT cleared",
|
||||||
|
exc,
|
||||||
|
)
|
||||||
|
|
||||||
|
log.info("Session complete (no new events) -> %s", session_dir)
|
||||||
|
return
|
||||||
|
else:
|
||||||
|
new_key_set = None # unknown; proceed with full download
|
||||||
|
|
||||||
# Apply max_events cap
|
# Apply max_events cap
|
||||||
# stop_idx: when we know the count from list_event_keys, use it as
|
# stop_idx: when we know the count from list_event_keys, use it as
|
||||||
@@ -393,67 +388,27 @@ class AchSession:
|
|||||||
)
|
)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Pass `seen_events` (key → ISO timestamp) so the client can
|
|
||||||
# read 0C on its skip path and only skip 5A when the per-event
|
|
||||||
# timestamp matches what we already have on disk. When force_-
|
|
||||||
# redownload is set, seen_events was already cleared above.
|
|
||||||
#
|
|
||||||
# Filter out empty-string timestamps (legacy v1 entries) — the
|
|
||||||
# client's 0C-on-skip-path only trusts entries with a
|
|
||||||
# populated timestamp; otherwise it falls through to a full
|
|
||||||
# 5A download.
|
|
||||||
skip_dict = {k: ts for k, ts in seen_events.items() if ts}
|
|
||||||
|
|
||||||
all_events = client.get_events(
|
all_events = client.get_events(
|
||||||
full_waveform=True,
|
full_waveform=True,
|
||||||
stop_after_index=stop_idx,
|
stop_after_index=stop_idx,
|
||||||
skip_waveform_for_events=skip_dict if skip_dict else None,
|
skip_waveform_for_keys=seen_keys if seen_keys else None,
|
||||||
)
|
)
|
||||||
|
|
||||||
# New events are those that came back with _a5_frames populated
|
# Filter to events whose keys we haven't saved before.
|
||||||
# (= 5A actually ran on this session). Skipped events have
|
|
||||||
# _a5_frames = None because the client matched (key, timestamp)
|
|
||||||
# against skip_dict and bypassed 5A.
|
|
||||||
new_events = [
|
new_events = [
|
||||||
e for e in all_events
|
e for e in all_events
|
||||||
if getattr(e, "_a5_frames", None)
|
if e._waveform_key is None
|
||||||
|
or e._waveform_key.hex() not in seen_keys
|
||||||
]
|
]
|
||||||
skipped = len(all_events) - len(new_events)
|
skipped = len(all_events) - len(new_events)
|
||||||
|
|
||||||
log.info(" [OK] Walked %d event(s): %d downloaded, %d skipped (matched (key, ts) in state)",
|
log.info(" [OK] Downloaded %d event(s): %d new, %d skipped (already seen)",
|
||||||
len(all_events), len(new_events), skipped)
|
len(all_events), len(new_events), skipped)
|
||||||
|
if skipped:
|
||||||
# ── Persist event file + A5 sidecar to the waveform store ──
|
log.info(" (skipped %d already-downloaded event(s))", skipped)
|
||||||
# Saves ride alongside the existing JSON dump so the on-disk
|
|
||||||
# event file and events.json reference the same set of events.
|
|
||||||
waveform_records: dict[str, dict] = {}
|
|
||||||
for ev in new_events:
|
|
||||||
if not ev._a5_frames:
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
rec = self.store.save(
|
|
||||||
ev,
|
|
||||||
serial=serial or "UNKNOWN",
|
|
||||||
a5_frames=ev._a5_frames,
|
|
||||||
)
|
|
||||||
if ev._waveform_key is not None:
|
|
||||||
waveform_records[ev._waveform_key.hex()] = rec
|
|
||||||
log.info(
|
|
||||||
" [WAVE] saved %s (%d bytes)",
|
|
||||||
rec["filename"], rec["filesize"],
|
|
||||||
)
|
|
||||||
except Exception as exc:
|
|
||||||
key_hex = ev._waveform_key.hex() if ev._waveform_key else "????????"
|
|
||||||
log.warning(
|
|
||||||
" [WARN] Waveform store save failed for %s: %s",
|
|
||||||
key_hex, exc,
|
|
||||||
)
|
|
||||||
|
|
||||||
if new_events:
|
if new_events:
|
||||||
_save_json(
|
_save_json(session_dir / "events.json", [_event_to_dict(e) for e in new_events])
|
||||||
session_dir / "events.json",
|
|
||||||
[_event_to_dict(e, waveform_records) for e in new_events],
|
|
||||||
)
|
|
||||||
|
|
||||||
for ev in new_events:
|
for ev in new_events:
|
||||||
pv = ev.peak_values
|
pv = ev.peak_values
|
||||||
@@ -512,11 +467,7 @@ class AchSession:
|
|||||||
_session_start = datetime.datetime.now()
|
_session_start = datetime.datetime.now()
|
||||||
try:
|
try:
|
||||||
_ev_ins, _ev_skip = self.db.insert_events(
|
_ev_ins, _ev_skip = self.db.insert_events(
|
||||||
new_events,
|
new_events, serial=serial or self.peer, session_id=None
|
||||||
serial=serial or self.peer,
|
|
||||||
session_id=None,
|
|
||||||
waveform_records=waveform_records,
|
|
||||||
device_family="series3",
|
|
||||||
)
|
)
|
||||||
_ml_ins, _ml_skip = self.db.insert_monitor_log(
|
_ml_ins, _ml_skip = self.db.insert_monitor_log(
|
||||||
new_monitor_entries, session_id=None
|
new_monitor_entries, session_id=None
|
||||||
@@ -551,64 +502,35 @@ class AchSession:
|
|||||||
)
|
)
|
||||||
|
|
||||||
# ── Update persistent state ───────────────────────────────────
|
# ── Update persistent state ───────────────────────────────────
|
||||||
# Build a fresh (key → ISO timestamp) map from THIS session's
|
# Include both triggered-event keys and monitor-log keys in the
|
||||||
# results. For each event currently on the device, prefer the
|
# downloaded set so they are not re-processed on the next call-home.
|
||||||
# timestamp we just observed (from 0C); fall back to whatever
|
current_event_keys = [
|
||||||
# was already in seen_events for that key (so we don't lose an
|
e._waveform_key.hex()
|
||||||
# entry just because get_events skipped it on the (key, ts)
|
for e in all_events
|
||||||
# match path).
|
if e._waveform_key is not None
|
||||||
def _ts_iso(ev) -> str:
|
]
|
||||||
ts = getattr(ev, "timestamp", None)
|
current_monitor_keys = [e.key for e in new_monitor_entries]
|
||||||
if ts is None:
|
current_keys = current_event_keys + current_monitor_keys
|
||||||
return ""
|
|
||||||
try:
|
|
||||||
return datetime.datetime(
|
|
||||||
ts.year, ts.month, ts.day,
|
|
||||||
ts.hour or 0, ts.minute or 0, ts.second or 0,
|
|
||||||
).isoformat()
|
|
||||||
except Exception:
|
|
||||||
return str(ts)
|
|
||||||
|
|
||||||
current_events_map: dict[str, str] = {}
|
|
||||||
for ev in all_events:
|
|
||||||
if ev._waveform_key is None:
|
|
||||||
continue
|
|
||||||
key_hex = ev._waveform_key.hex()
|
|
||||||
ts_iso = _ts_iso(ev) or seen_events.get(key_hex, "")
|
|
||||||
current_events_map[key_hex] = ts_iso
|
|
||||||
|
|
||||||
# Monitor-log entries don't have a 0C-style timestamp, but
|
|
||||||
# they DO have a start_time; use that so the monitor-log keys
|
|
||||||
# are properly entered into the (key, ts) map.
|
|
||||||
for ml in new_monitor_entries:
|
|
||||||
key_hex = ml.key
|
|
||||||
ts = ml.start_time
|
|
||||||
ts_iso = ts.isoformat() if ts else seen_events.get(key_hex, "")
|
|
||||||
# If a triggered event already populated this key, keep
|
|
||||||
# whichever has a non-empty timestamp.
|
|
||||||
if key_hex not in current_events_map or not current_events_map[key_hex]:
|
|
||||||
current_events_map[key_hex] = ts_iso
|
|
||||||
|
|
||||||
if erased_successfully:
|
if erased_successfully:
|
||||||
updated_events: dict[str, str] = {}
|
# Device memory is clear. Reset downloaded_keys and the
|
||||||
|
# high-water mark so the next call-home starts fresh and
|
||||||
|
# doesn't mis-identify the recycled key 01110000 as "seen".
|
||||||
|
updated_keys = []
|
||||||
new_max_key = "00000000"
|
new_max_key = "00000000"
|
||||||
log.info(
|
log.info(
|
||||||
" State reset after erase -- next session will download "
|
" State reset after erase -- next session will download "
|
||||||
"from key 0 (device counter resets after erase)"
|
"from key 0 (device counter resets after erase)"
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
# Merge: keep prior (key, ts) entries we still have evidence
|
# Normal (no erase): union of previously-seen + all keys on
|
||||||
# of (for survivors of any partial failure), plus this
|
# device now. Includes already-seen survivors so we never
|
||||||
# session's authoritative (key, ts) pairs.
|
# re-download them if the device somehow keeps old records.
|
||||||
updated_events = dict(seen_events)
|
updated_keys = sorted(set(seen_keys) | set(current_keys))
|
||||||
updated_events.update(current_events_map)
|
new_max_key = updated_keys[-1] if updated_keys else max_seen_key
|
||||||
new_max_key = (
|
|
||||||
max(updated_events.keys())
|
|
||||||
if updated_events else max_seen_key
|
|
||||||
)
|
|
||||||
|
|
||||||
state[unit_key] = {
|
state[unit_key] = {
|
||||||
"downloaded_events": updated_events,
|
"downloaded_keys": updated_keys,
|
||||||
"max_downloaded_key": new_max_key,
|
"max_downloaded_key": new_max_key,
|
||||||
"last_seen": datetime.datetime.now().isoformat(),
|
"last_seen": datetime.datetime.now().isoformat(),
|
||||||
"serial": serial,
|
"serial": serial,
|
||||||
@@ -670,10 +592,7 @@ def _device_info_to_dict(d: DeviceInfo) -> dict:
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def _event_to_dict(
|
def _event_to_dict(e: Event) -> dict:
|
||||||
e: Event,
|
|
||||||
waveform_records: Optional[dict[str, dict]] = None,
|
|
||||||
) -> dict:
|
|
||||||
pv = e.peak_values
|
pv = e.peak_values
|
||||||
pi = e.project_info
|
pi = e.project_info
|
||||||
peaks = {}
|
peaks = {}
|
||||||
@@ -692,11 +611,6 @@ def _event_to_dict(
|
|||||||
for ch, vals in e.raw_samples.items()
|
for ch, vals in e.raw_samples.items()
|
||||||
}
|
}
|
||||||
samples["__note__"] = "first 20 sample-sets only; see raw_rx.bin for full waveform"
|
samples["__note__"] = "first 20 sample-sets only; see raw_rx.bin for full waveform"
|
||||||
|
|
||||||
rec: dict = {}
|
|
||||||
if waveform_records and e._waveform_key is not None:
|
|
||||||
rec = waveform_records.get(e._waveform_key.hex(), {}) or {}
|
|
||||||
|
|
||||||
return {
|
return {
|
||||||
"timestamp": str(e.timestamp) if e.timestamp else None,
|
"timestamp": str(e.timestamp) if e.timestamp else None,
|
||||||
"project": pi.project if pi else None,
|
"project": pi.project if pi else None,
|
||||||
@@ -705,9 +619,6 @@ def _event_to_dict(
|
|||||||
"sensor_location": pi.sensor_location if pi else None,
|
"sensor_location": pi.sensor_location if pi else None,
|
||||||
"peaks": peaks,
|
"peaks": peaks,
|
||||||
"raw_samples_preview": samples,
|
"raw_samples_preview": samples,
|
||||||
"blastware_filename": rec.get("filename"),
|
|
||||||
"blastware_filesize": rec.get("filesize"),
|
|
||||||
"a5_pickle_filename": rec.get("a5_pickle_filename"),
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -729,7 +640,6 @@ def serve(args: argparse.Namespace) -> None:
|
|||||||
output_dir.mkdir(parents=True, exist_ok=True)
|
output_dir.mkdir(parents=True, exist_ok=True)
|
||||||
state_path = output_dir / "ach_state.json"
|
state_path = output_dir / "ach_state.json"
|
||||||
db = SeismoDb(output_dir / "seismo_relay.db")
|
db = SeismoDb(output_dir / "seismo_relay.db")
|
||||||
store = WaveformStore(output_dir / "waveforms")
|
|
||||||
|
|
||||||
server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||||
server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||||
@@ -747,7 +657,6 @@ def serve(args: argparse.Namespace) -> None:
|
|||||||
print(f" Max events per session: {max_ev if max_ev else 'unlimited'}")
|
print(f" Max events per session: {max_ev if max_ev else 'unlimited'}")
|
||||||
print(f" Clear device after download: {'YES' if args.clear_after_download else 'no'}")
|
print(f" Clear device after download: {'YES' if args.clear_after_download else 'no'}")
|
||||||
print(f" Restart monitoring after download: {'YES' if args.restart_monitoring else 'no'}")
|
print(f" Restart monitoring after download: {'YES' if args.restart_monitoring else 'no'}")
|
||||||
print(f" Force re-download all (ignore state): {'YES' if args.force_redownload_all else 'no'}")
|
|
||||||
print(f"{'='*60}")
|
print(f"{'='*60}")
|
||||||
print(f"\n Point your test unit's ACEmanager call-home settings to:")
|
print(f"\n Point your test unit's ACEmanager call-home settings to:")
|
||||||
print(f" Remote Host: <this machine's LAN IP>")
|
print(f" Remote Host: <this machine's LAN IP>")
|
||||||
@@ -785,10 +694,8 @@ def serve(args: argparse.Namespace) -> None:
|
|||||||
max_events=max_ev,
|
max_events=max_ev,
|
||||||
state_path=state_path,
|
state_path=state_path,
|
||||||
db=db,
|
db=db,
|
||||||
store=store,
|
|
||||||
clear_after_download=args.clear_after_download,
|
clear_after_download=args.clear_after_download,
|
||||||
restart_monitoring=args.restart_monitoring,
|
restart_monitoring=args.restart_monitoring,
|
||||||
force_redownload=args.force_redownload_all,
|
|
||||||
)
|
)
|
||||||
t = threading.Thread(target=session.run, daemon=True, name=f"ach-{peer}")
|
t = threading.Thread(target=session.run, daemon=True, name=f"ach-{peer}")
|
||||||
t.start()
|
t.start()
|
||||||
@@ -873,17 +780,6 @@ def parse_args() -> argparse.Namespace:
|
|||||||
"This mirrors the standard Blastware ACH workflow."
|
"This mirrors the standard Blastware ACH workflow."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
p.add_argument(
|
|
||||||
"--force-redownload-all",
|
|
||||||
action="store_true",
|
|
||||||
default=False,
|
|
||||||
help=(
|
|
||||||
"Manual override: ignore ach_state.json's downloaded_events map "
|
|
||||||
"for this session and re-download every event currently on the "
|
|
||||||
"device, regardless of (key, timestamp) match. Useful when state "
|
|
||||||
"has become inconsistent with the on-disk waveform store / DB."
|
|
||||||
),
|
|
||||||
)
|
|
||||||
p.add_argument(
|
p.add_argument(
|
||||||
"--verbose", "-v",
|
"--verbose", "-v",
|
||||||
action="store_true",
|
action="store_true",
|
||||||
|
|||||||
@@ -1,212 +0,0 @@
|
|||||||
> ## SUPERSEDED 2026-08-25 — the block is uniformly BIG-ENDIAN
|
|
||||||
>
|
|
||||||
> The `uint8` peak / `annotation` byte model described below is wrong,
|
|
||||||
> though it decoded quiet data correctly. The real layout:
|
|
||||||
>
|
|
||||||
> - **Every per-channel field is `uint16` big-endian.** `T_peak` is
|
|
||||||
> `[5:7]`, `T_halfperiod` `[7:9]`, `V_peak` `[9:11]`, and so on.
|
|
||||||
> Only `block_ctr` at `[2:4]` is little-endian.
|
|
||||||
> - The **marker is `block[4]` alone**, not a `uint16 LE` at `[4:6]`.
|
|
||||||
> Testing `[4:6] == 10` forced `block[5] == 0`, which is exactly what
|
|
||||||
> capped every geo peak at one byte (255 counts = 1.275 in/s).
|
|
||||||
> - The **"annotation" byte was never an annotation** — it is the high
|
|
||||||
> byte of the big-endian half-period. That is why it was non-zero
|
|
||||||
> precisely on the sub-Hz intervals Blastware renders as `<1.0`.
|
|
||||||
> - The **final block of the stream carries tail `9c 06 00 42`** instead
|
|
||||||
> of `1e 0a 00 00`, and arbitrary bytes at `[21:23]`. Rejecting it
|
|
||||||
> dropped the last interval of nearly every histogram — often the one
|
|
||||||
> holding the event peak, so the file's PPV read low.
|
|
||||||
>
|
|
||||||
> Verified against 1211 production histograms paired with their Blastware
|
|
||||||
> ASCII exports: **1211/1211 decode exactly** (interval count plus every
|
|
||||||
> per-interval peak), and 842,442 per-interval frequency comparisons match
|
|
||||||
> with zero mismatches. The uint8 model scored 1204/1211 — the seven
|
|
||||||
> failures are exactly the files containing a peak above 1.275 in/s.
|
|
||||||
>
|
|
||||||
> The section below is retained as the reasoning trail.
|
|
||||||
|
|
||||||
# Histogram body codec — FULLY DECODED (2026-05-20)
|
|
||||||
|
|
||||||
Clean working status doc for the MiniMate Plus histogram-mode event
|
|
||||||
body codec. Companion to `waveform_codec_re_status.md`. The deep
|
|
||||||
historical record (with retractions and dated analyses) lives in
|
|
||||||
`docs/instantel_protocol_reference.md §7.6.2`; the authoritative
|
|
||||||
implementation lives in `minimateplus/histogram_codec.py`.
|
|
||||||
|
|
||||||
## TL;DR
|
|
||||||
|
|
||||||
**The codec is fully decoded.** Every field of every block in the
|
|
||||||
in-repo histogram fixture corpus decodes byte-exact against BW's
|
|
||||||
ASCII export.
|
|
||||||
|
|
||||||
26 regression tests pass against ~3,500 blocks across 5 in-repo
|
|
||||||
fixtures, plus a synthetic regression block taken from a real
|
|
||||||
BE9558 prod event to lock in the uint8-peak interpretation.
|
|
||||||
|
|
||||||
**Important correction (2026-05-21):** the per-channel peak count
|
|
||||||
is `uint8` at byte[6]/[10]/[14]/[18], NOT `uint16 LE` at byte[6:8]
|
|
||||||
etc. The N844 fixture corpus the original RE was done against has
|
|
||||||
zero values in bytes [7]/[11]/[15]/[19] for every block, so the
|
|
||||||
two interpretations happened to be equivalent. Cross-correlating
|
|
||||||
non-N844 events (BE9558 Tran-drift, BE18003 Histogram+Continuous)
|
|
||||||
against BW's per-interval ASCII export — 4 channels × ~1400 blocks
|
|
||||||
per event × multiple events = 100% byte-exact only when the peak
|
|
||||||
is read as uint8. Reading as uint16 LE produced peaks up to 268
|
|
||||||
in/s per channel and 35× inflated PVS sums when first deployed to
|
|
||||||
prod (rolled back, root-caused, and fixed in commit 7183b95+1).
|
|
||||||
|
|
||||||
## Body format
|
|
||||||
|
|
||||||
```
|
|
||||||
body = [stream of 32-byte data blocks] + [small trailing remnant]
|
|
||||||
```
|
|
||||||
|
|
||||||
Each block represents one histogram interval. Block layout:
|
|
||||||
|
|
||||||
```
|
|
||||||
[0] 0x00 always-zero tag
|
|
||||||
[1] segment_id (uint8) 0x00..0x03 — 256 blocks per segment
|
|
||||||
[2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, …)
|
|
||||||
[4:6] 0x000a (uint16 LE) constant marker (= 10)
|
|
||||||
[6] T_peak_count uint8 Tran peak (count × 0.005 → in/s at Normal,
|
|
||||||
max 1.275 in/s — fits in uint8)
|
|
||||||
[7] T_annotation uint8 empirically non-zero on intervals with sub-Hz
|
|
||||||
or unmeasurable freq; meaning not fully RE'd
|
|
||||||
[8:10] T_halfperiod uint16 LE Tran half-period in samples
|
|
||||||
(freq_Hz = 512 / halfp; ≤ 5 means ">100 Hz")
|
|
||||||
[10] V_peak_count uint8 Vert peak
|
|
||||||
[11] V_annotation uint8
|
|
||||||
[12:14] V_halfperiod uint16 LE Vert freq half-period
|
|
||||||
[14] L_peak_count uint8 Long peak
|
|
||||||
[15] L_annotation uint8
|
|
||||||
[16:18] L_halfperiod uint16 LE Long freq half-period
|
|
||||||
[18] M_peak_count uint8 MicL peak count
|
|
||||||
(dB via waveform_codec.mic_count_to_db)
|
|
||||||
[19] M_annotation uint8
|
|
||||||
[20:22] M_halfperiod uint16 LE MicL freq half-period
|
|
||||||
[22:24] 0x00 0x00 constant
|
|
||||||
[24:28] 4-byte variable purpose unknown — possibly CRC,
|
|
||||||
timestamp delta, or psi(L) numeric;
|
|
||||||
not needed for waveform reconstruction
|
|
||||||
[28:32] 0x1e 0x0a 0x00 0x00 constant block-end signature
|
|
||||||
```
|
|
||||||
|
|
||||||
Reliable block-identification anchor:
|
|
||||||
```python
|
|
||||||
block[22:24] == b"\x00\x00" and block[28:32] == b"\x1e\x0a\x00\x00"
|
|
||||||
```
|
|
||||||
(The `1e 0a 00 00` constant tail is the most distinctive signature.)
|
|
||||||
|
|
||||||
## Per-channel encoding
|
|
||||||
|
|
||||||
| Channel | Peak encoding | Frequency encoding |
|
|
||||||
|---|---|---|
|
|
||||||
| Tran | count × 0.005 = in/s at Normal range | `freq_Hz = 512 / halfperiod` |
|
|
||||||
| Vert | same | same |
|
|
||||||
| Long | same | same |
|
|
||||||
| MicL | count → dB via `mic_count_to_db(count)` (same formula as waveform codec) | same |
|
|
||||||
|
|
||||||
**`>100 Hz` sentinel**: when halfperiod ≤ 5 (giving ≥100 Hz from the
|
|
||||||
512/halfp formula), BW displays `>100 Hz`. Codec's `half_period_to_hz`
|
|
||||||
returns `None` in this range.
|
|
||||||
|
|
||||||
## Verified facts (cross-checked against fixture corpus)
|
|
||||||
|
|
||||||
Example: N844L6Z8.ZR0H block 130 → all 8 decoded fields byte-exact:
|
|
||||||
|
|
||||||
```
|
|
||||||
binary samples [10, 6, 24, 4, 18, 5, 21, 5, 9]
|
|
||||||
TXT row [0.030, 21, 0.020, 28, 0.025, 24, 0.040, 0.000, 95.92, 57]
|
|
||||||
|
|
||||||
slot[0] = 10 marker
|
|
||||||
slot[1] = 6 × 0.005 = 0.030 in/s ✓ T_peak
|
|
||||||
slot[2] = 24 → 512/24 = 21.3 → 21 Hz ✓ T_freq
|
|
||||||
slot[3] = 4 × 0.005 = 0.020 in/s ✓ V_peak
|
|
||||||
slot[4] = 18 → 512/18 = 28.4 → 28 Hz ✓ V_freq
|
|
||||||
slot[5] = 5 × 0.005 = 0.025 in/s ✓ L_peak
|
|
||||||
slot[6] = 21 → 512/21 = 24.4 → 24 Hz ✓ L_freq
|
|
||||||
slot[7] = 5 → 81.94 + 20·log10(5) = 95.92 dB ✓ M_peak
|
|
||||||
slot[8] = 9 → 512/9 = 56.9 → 57 Hz ✓ M_freq
|
|
||||||
```
|
|
||||||
|
|
||||||
## Verified test coverage
|
|
||||||
|
|
||||||
`tests/test_histogram_codec.py` (24 tests):
|
|
||||||
|
|
||||||
- Block walking: yields one record per `.TXT` interval ± 1 (off-by-one
|
|
||||||
at the tail when recording was stopped mid-write). Segment-ID
|
|
||||||
groups of 256 blocks confirmed.
|
|
||||||
- Geo peaks: every block of N844L20G, N844L6Z8, N844L6XE, N844L23B
|
|
||||||
matches `.TXT` within the 0.0005 in/s quantization step.
|
|
||||||
- Geo freqs: every block of N844L6Z8 and N844L6XE matches `.TXT`
|
|
||||||
within 1 Hz (BW display rounds). `>100 Hz` sentinel handled correctly.
|
|
||||||
- Mic dB: every block of N844L6XE, N844L23B, N844L6Z8 matches `.TXT`
|
|
||||||
within 0.1 dB (BW display precision).
|
|
||||||
- Mic freq: matches `.TXT` within 1 Hz across active blocks.
|
|
||||||
|
|
||||||
## What's NOT yet decoded
|
|
||||||
|
|
||||||
- **Annotation bytes (`block[7]/[11]/[15]/[19]`)**. Empirically
|
|
||||||
non-zero on intervals where the per-channel ZC frequency comes
|
|
||||||
out as `N/A` or sub-Hz (`<1.0`, `1.X`). Hypothesis tested in the
|
|
||||||
RE session: byte != 0 ↔ sub-Hz freq. Only ~50% correlation
|
|
||||||
across the K558 corpus, so the relationship is more complex.
|
|
||||||
Possibilities: time-of-peak-within-interval, halfp extension for
|
|
||||||
very-long-period signals, or a debug/diagnostic field the firmware
|
|
||||||
writes opportunistically. Doesn't affect peak amplitudes or
|
|
||||||
waveform reconstruction. Captured as `record["annotations"]` for
|
|
||||||
future RE.
|
|
||||||
- **4-byte variable metadata field (bytes 24:28)**. Not needed for
|
|
||||||
waveform reconstruction. Speculation: per-block CRC, sub-second
|
|
||||||
timestamp offset, or a Mic psi(L) count not in the 9 samples.
|
|
||||||
Punt until something needs it.
|
|
||||||
- **Geo PVS (TXT col 7, e.g. "0.040 in/s")**. Not stored in the
|
|
||||||
block; can be approximated as `sqrt(T_peak² + V_peak² + L_peak²)`
|
|
||||||
but BW's value sometimes differs slightly (probably computed from
|
|
||||||
waveform-instant samples, not from per-channel peaks). Punt — the
|
|
||||||
`.h5` consumers don't need PVS as a sample channel.
|
|
||||||
- **Mic psi(L) value (TXT col 8)**. TXT shows it as a small psi value
|
|
||||||
derived from the dB measurement. Not in the 9 samples. Could be
|
|
||||||
derived from `M_peak_count` via the inverse of the dB formula plus
|
|
||||||
a psi calibration constant. Defer.
|
|
||||||
|
|
||||||
## Output shape
|
|
||||||
|
|
||||||
`decode_histogram_body` returns the standard 4-channel dict that
|
|
||||||
mirrors `waveform_codec.decode_waveform_v2`'s output:
|
|
||||||
|
|
||||||
```python
|
|
||||||
{
|
|
||||||
"Tran": [peak_count_per_interval, ...], # 16-count units (LSB = 0.005 in/s)
|
|
||||||
"Vert": [..., ...],
|
|
||||||
"Long": [..., ...],
|
|
||||||
"MicL": [..., ...], # raw ADC counts
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Run through `waveform_codec.decoded_to_adc_counts` to get 1-count ADC
|
|
||||||
units (geo ×16, mic passthrough) for the standard `.h5` writer.
|
|
||||||
|
|
||||||
For the full per-interval record with frequencies + metadata, use
|
|
||||||
`decode_histogram_body_full()`.
|
|
||||||
|
|
||||||
## Where it's wired
|
|
||||||
|
|
||||||
- `minimateplus/event_file_io.py:read_blastware_file()` — first tries
|
|
||||||
the waveform codec, falls back to the histogram codec when the
|
|
||||||
waveform preamble isn't present. Same output shape, same
|
|
||||||
downstream pipeline.
|
|
||||||
- `scripts/backfill_sidecars.py` — the `has_samples` short-circuit
|
|
||||||
added during the histogram-codec-pending era still serves as a
|
|
||||||
defensive guard against truly undecodable files, but no longer
|
|
||||||
fires for valid histograms.
|
|
||||||
|
|
||||||
## Companion reference
|
|
||||||
|
|
||||||
- `docs/waveform_codec_re_status.md` — sibling status doc for the
|
|
||||||
much-more-complex waveform-mode codec.
|
|
||||||
- `docs/instantel_protocol_reference.md §7.6.2` — historical
|
|
||||||
protocol-reference entry. Structural framing matches what we
|
|
||||||
found; per-sample semantics were less documented than the `✅
|
|
||||||
CONFIRMED` badge suggested. This doc supersedes §7.6.2 where they
|
|
||||||
conflict on confidence level.
|
|
||||||
@@ -1,341 +0,0 @@
|
|||||||
# IDF Protocol Reference — Thor / Micromate Series IV
|
|
||||||
|
|
||||||
Starting-point reference for reverse-engineering Instantel's Micromate
|
|
||||||
Series IV event-file format. Sibling to
|
|
||||||
[instantel_protocol_reference.md](instantel_protocol_reference.md) (the
|
|
||||||
Series III "Rosetta Stone") — this doc holds what we know so far and
|
|
||||||
the open questions still to crack.
|
|
||||||
|
|
||||||
**Status (2026-05-28):** ASCII text sidecar fully decoded (1,014
|
|
||||||
sample files round-trip). **Thor IDFW** binary now decodes via
|
|
||||||
`micromate.idf_file.read_idf_file()` — reuses the BW segment-rotated
|
|
||||||
block codec verbatim at fixed body offset `0x0f1f`; metadata (serial,
|
|
||||||
timestamp, sample_rate, record_time, calibration_date) extracted from
|
|
||||||
the binary header. Sample fidelity is 87–99% byte-exact on quiet
|
|
||||||
events; loud events hit the BW codec's known walker-stops-early
|
|
||||||
limitation. Residual ~3% drift on per-sample deltas (likely a
|
|
||||||
Thor-specific 12-bit delta refinement not yet modelled).
|
|
||||||
|
|
||||||
**Thor IDFH histograms also decoded.** Body has one or more segments;
|
|
||||||
each 12-byte segment header `[length_be 2B][0a 00 00 00][00 NN][05 3f]`
|
|
||||||
introduces `N = (length - 10) // 72` interval records of 72 bytes
|
|
||||||
each. Each interval = 4 × 16-byte per-channel records:
|
|
||||||
`[int16 min][int16 max][int16 ??][uint16 halfp][2B 00][uint16 ??][2B 00][uint16 ??]`.
|
|
||||||
Geo peak `= max(|min|, |max|) / 32768 × 10` in/s (matches sidecar
|
|
||||||
~1.8%); freq `= 512 / halfp` Hz (None for halfp ≤ 5 → ">100"
|
|
||||||
sentinel). Corpus: **all 859 Thor IDFH files decode, 181,071
|
|
||||||
intervals**. Wired through `read_idf_file()` →
|
|
||||||
`save_imported_idf()` → sidecar's `extensions.idf_intervals`.
|
|
||||||
|
|
||||||
**Note on the BE9439 outliers in the example corpus:** Two files
|
|
||||||
(`BE9439_20200713131747.IDFW` and `BE9439_20200713124251.IDFH`) are
|
|
||||||
**Series III Blastware** binaries, not Thor. Provenance: TMI tried
|
|
||||||
to use Thor to manage auto-call-homes for Series III units; the
|
|
||||||
experiment didn't work out, but it did leave a few BW event files
|
|
||||||
in Thor's per-serial directory structure with `.IDFW`/`.IDFH`
|
|
||||||
extensions — Thor's forwarder applied its own naming convention to
|
|
||||||
the BW bodies it was relaying. Their header `10 00 01 80 00 00
|
|
||||||
Instantel STRT ff fe <end_key> <start_key>` is the BW SUB 5A STRT
|
|
||||||
record, not a Thor body preamble. The reader detects them by
|
|
||||||
signature and raises `NotImplementedError` pointing callers at
|
|
||||||
`read_blastware_file()`, which extracts BW-format peaks from them.
|
|
||||||
|
|
||||||
**Still NYI for Thor IDFH:** per-channel `int16 field4` (possibly
|
|
||||||
time-of-peak); the two uint16 fields (probably PVS contributions);
|
|
||||||
8-byte interval tail (PVS data); mic dB(L) exact conversion constant.
|
|
||||||
|
|
||||||
### Codec breakthroughs (2026-05-28)
|
|
||||||
|
|
||||||
- **Body offset is a fixed `0x0f1f`** across 151/154 corpus IDFW
|
|
||||||
files. Preceded by a 4-byte record-type marker (`46 00 00 00`)
|
|
||||||
+ magic preamble `00 02 00 [Tran[0] BE] [Tran[1] BE]`.
|
|
||||||
- **Sample stream is BW's segment-rotated block codec verbatim.**
|
|
||||||
Thor reuses `10 NN` (nibble), `20 NN` (int8), `00 NN` (RLE),
|
|
||||||
`30 NN` (packed12), `40 02` (segment header) tags with the same
|
|
||||||
semantics. Channel rotation Tran→Vert→Long→MicL.
|
|
||||||
- **Geo LSB = 0.0003 in/s** (not BW's 0.005), because Thor's 16-bit
|
|
||||||
ADC range maps to 10 in/s without the 16-count BW quantization step.
|
|
||||||
- **Mic ≈ 2.14×10⁻⁶ psi/count** (rough scale; refine after channel
|
|
||||||
block calibration constants are decoded).
|
|
||||||
- **BW compliance anchor `\xbe\x80\x00\x00\x00\x00` reappears at
|
|
||||||
IDFW offset 0x952** — sample_rate at anchor−6 (uint16 BE),
|
|
||||||
record_time at anchor+6 (float32 BE), same layout as BW.
|
|
||||||
- **Event timestamp at offset 0x97A** — 8 bytes `[day][month]
|
|
||||||
[year_be][unk][hour][min][sec]`. Stop-time mirrors at 0x982.
|
|
||||||
- **Serial as null-terminated ASCII at 0x14E**.
|
|
||||||
- **Calibration date** at 0x194–0x197 (day, month, year_be).
|
|
||||||
- Per-sample residual drift of ~3% suggests Thor encodes int8/nibble
|
|
||||||
deltas with an extra refinement bit that BW doesn't carry —
|
|
||||||
unsolved; errors resync within a few samples so cumulative impact
|
|
||||||
is small.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## File model
|
|
||||||
|
|
||||||
### Filename convention
|
|
||||||
|
|
||||||
```
|
|
||||||
<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>
|
|
||||||
```
|
|
||||||
|
|
||||||
- **SERIAL** — literal device serial, two-letter prefix + numeric
|
|
||||||
suffix. Examples seen: `UM11719`, `UM13981`, `UM20147`, `BE9439`.
|
|
||||||
Unlike Series III BW filenames (`M529LK44.AB0`, base-36 stem),
|
|
||||||
Series IV filenames carry the serial in plain text.
|
|
||||||
- **YYYYMMDDHHMMSS** — 14-char ASCII timestamp in **device local
|
|
||||||
time** (no timezone marker).
|
|
||||||
- **KIND** — `IDFH` for histograms, `IDFW` for waveforms.
|
|
||||||
|
|
||||||
The `.IDFH.txt` / `.IDFW.txt` ASCII sidecar lives in a `TXT/`
|
|
||||||
**subfolder** of the unit's directory, not alongside the binary.
|
|
||||||
This pairing convention is encoded in
|
|
||||||
`event_forwarder.idf_report_path()`.
|
|
||||||
|
|
||||||
### Directory layout
|
|
||||||
|
|
||||||
```
|
|
||||||
C:\THORDATA\
|
|
||||||
└── <Project>\
|
|
||||||
└── <UM####>\ ← unit serial dir
|
|
||||||
├── UM12345_20260520100000.MLG ← monitor log (not events)
|
|
||||||
├── UM12345_20260520100000.IDFH ← histogram event (binary)
|
|
||||||
├── UM12345_20260520100000.IDFW ← waveform event (binary)
|
|
||||||
├── UM12345_20260520100000.IDFW.CDB ← cache-DB variant (skip)
|
|
||||||
├── TXT\
|
|
||||||
│ ├── UM12345_20260520100000.IDFH.txt ← histogram ASCII sidecar
|
|
||||||
│ └── UM12345_20260520100000.IDFW.txt ← waveform ASCII sidecar
|
|
||||||
├── CSV\, HTML\, PDF\, XML\ ← operator-facing derived exports
|
|
||||||
└── ...
|
|
||||||
```
|
|
||||||
|
|
||||||
The `.IDFW.CDB` files share the binary's basename but appear to be a
|
|
||||||
separate cache/database variant. Their first 8 bytes match the
|
|
||||||
**old**-firmware Thor signature (see below) regardless of which
|
|
||||||
signature the paired `.IDFW` uses. Purpose unknown; sizes vary
|
|
||||||
wildly (observed 123 B → 40,491 B). Thor-watcher's forwarder
|
|
||||||
deliberately skips them.
|
|
||||||
|
|
||||||
### Sample corpus
|
|
||||||
|
|
||||||
The `thor-watcher/example-data/THORDATA_example/` tree carries
|
|
||||||
**1,014 paired .IDFW / .IDFH + .txt files** spanning 2020–2023
|
|
||||||
across nine units (UM11719, UM13981, UM20147, …, plus BE9439 from
|
|
||||||
2020). This is the reverse-engineering ground truth.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## ASCII sidecar (`.IDFW.txt` / `.IDFH.txt`) — fully decoded
|
|
||||||
|
|
||||||
Shape: plain text, one `"Key : Value"` line per metadata field,
|
|
||||||
followed for waveforms by a tab-separated sample table headed by
|
|
||||||
the literal line `Waveform Data Channels`. Parsed by
|
|
||||||
[`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py).
|
|
||||||
See [`micromate/models.py`](../micromate/models.py) for the typed
|
|
||||||
`IdfReport` shape.
|
|
||||||
|
|
||||||
### Notable conventions
|
|
||||||
|
|
||||||
- **Units are native to Thor** — geophone in **in/s**, microphone in
|
|
||||||
**dB(L)** (not psi like Series III BW reports), frequency in Hz,
|
|
||||||
acceleration in g, displacement in in.
|
|
||||||
- **Below-threshold readings** appear as the literal string
|
|
||||||
`<0.005 in/s` (155 occurrences in the sample corpus) — the parser
|
|
||||||
strips the `<` and treats the numeric remainder as the value.
|
|
||||||
- **Out-of-range / not-measured** values appear as `N/A` — parser
|
|
||||||
drops the field rather than letting the string leak into a numeric
|
|
||||||
column.
|
|
||||||
- **Firmware string** observed: `Micromate ISEE 11.0AK`.
|
|
||||||
- **TitleString1..4** are operator-defined free-text slots; Thor's
|
|
||||||
default labels map them to Location / Client / Company / Notes,
|
|
||||||
which the parser surfaces as `project` / `client` / `operator` /
|
|
||||||
`notes`.
|
|
||||||
- **Histogram sidecars** use `HistogramStartDate` / `HistogramStartTime`
|
|
||||||
in place of waveform's `EventDate` / `EventTime`. Parser falls
|
|
||||||
through to either.
|
|
||||||
- **Histogram tabular block** lacks the `Waveform Data Channels`
|
|
||||||
marker; instead it's a multi-line column header followed by
|
|
||||||
per-interval rows (`<date> <time> <tran-ppv> <freq> ...`). Parser
|
|
||||||
silently ignores lines after the metadata block since they lack a
|
|
||||||
colon-separated `key : value` shape (the timestamps DO contain
|
|
||||||
colons but produce garbage keys that don't collide with any
|
|
||||||
recognised field).
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Binary header signatures (observed)
|
|
||||||
|
|
||||||
Hex dump of the first 32 bytes across 1,014 sample files reveals
|
|
||||||
**two distinct file signatures**, both anchored by the literal
|
|
||||||
ASCII string `"\x00Instantel\x00"` at offset 6–16:
|
|
||||||
|
|
||||||
### Signature A — newer firmware (1,012 files, 99.8% of corpus)
|
|
||||||
|
|
||||||
```
|
|
||||||
00000000: 0012 0100 0000 496e 7374 616e 7465 6c00 ......Instantel.
|
|
||||||
00000010: 0000 a695 002e b500 4f70 6572 6174 6f72 ........Operator
|
|
||||||
^^^^^^^^^^^^^^^^
|
|
||||||
operator/title string starts at 0x18
|
|
||||||
```
|
|
||||||
|
|
||||||
Header bytes 0–5: `00 12 01 00 00 00`. Followed immediately by the
|
|
||||||
8-byte ASCII tag, then 6 unknown bytes, then ASCII operator-supplied
|
|
||||||
strings (Operator name, etc.) and on through the project / client /
|
|
||||||
title strings. No `STRT` record observed in this layout.
|
|
||||||
|
|
||||||
### Signature B — older firmware (2 files: BE9439 from 2020)
|
|
||||||
|
|
||||||
```
|
|
||||||
00000000: 1000 0180 0000 496e 7374 616e 7465 6c00 ......Instantel.
|
|
||||||
00000010: 072c 0012 0300 5354 5254 fffe 0111 2340 .,....STRT....#@
|
|
||||||
^^^^^^^^^ ^^^^^^^^^
|
|
||||||
STRT magic 4-byte end_key
|
|
||||||
00000020: 0111 0000 2e5f 00ac 4600 0000 0200 0000 ....._..F.......
|
|
||||||
^^^^^^^^^ ^^^
|
|
||||||
4-byte start_key 0x46 (BW WAVEHDR record-type marker)
|
|
||||||
```
|
|
||||||
|
|
||||||
Header bytes 0–5: `10 00 01 80 00 00`. The structure after the
|
|
||||||
`Instantel` magic is **byte-for-byte identical to a BW SUB 5A
|
|
||||||
probe-response STRT record** as documented in
|
|
||||||
[instantel_protocol_reference.md → "SUB 5A — STRT record encodes
|
|
||||||
end_offset"](instantel_protocol_reference.md). Specifically:
|
|
||||||
|
|
||||||
| Offset | Bytes | Meaning (per BW reference) |
|
|
||||||
|--------|---------------------|--------------------------------------|
|
|
||||||
| 0x14 | `53 54 52 54` | `STRT` magic |
|
|
||||||
| 0x18 | `ff fe` | STRT sentinel |
|
|
||||||
| 0x1A | `01 11 23 40` | `end_key` (4 bytes) |
|
|
||||||
| 0x1E | `01 11 00 00` | `start_key` (4 bytes) |
|
|
||||||
| 0x26 | `46` | `0x46` waveform-record type marker |
|
|
||||||
|
|
||||||
**Hypothesis:** Older Micromate firmware writes a wrapped BW-format
|
|
||||||
event into the `.IDFW` file — essentially the same on-disk shape as
|
|
||||||
a Series III device, with the new filename convention applied at
|
|
||||||
export time. Newer firmware (signature A) abandoned the
|
|
||||||
BW-compatible layout for an Instantel-specific format.
|
|
||||||
|
|
||||||
If that hypothesis holds, the 2 signature-B files can already be
|
|
||||||
parsed via `minimateplus/event_file_io.read_blastware_file()` — worth
|
|
||||||
testing. The 1,012 signature-A files are the real reverse-engineering
|
|
||||||
target.
|
|
||||||
|
|
||||||
### `.IDFW.CDB` cache files
|
|
||||||
|
|
||||||
Always carry signature B (`10 00 01 80 ...`), even when the paired
|
|
||||||
`.IDFW` carries signature A. Plausible explanation: the CDB is an
|
|
||||||
internal Thor cache-database export that retains the legacy BW-style
|
|
||||||
record layout regardless of the user-facing `.IDFW` format version.
|
|
||||||
Not currently consumed by the forwarder.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## File-size patterns (Signature A, the main target)
|
|
||||||
|
|
||||||
Survey of 1,012 signature-A files:
|
|
||||||
|
|
||||||
| Event type | Typical size | Source of variance |
|
|
||||||
|--------------|-------------------|----------------------------------------------|
|
|
||||||
| `.IDFW` 2-sec | 9,200 – 10,500 B | Operator-supplied strings (TitleString1..4) of varying length |
|
|
||||||
| `.IDFH` | 2,944 – 4,076 B | Histogram interval count (record duration / interval) |
|
|
||||||
|
|
||||||
**Naive arithmetic for 2-sec waveform:**
|
|
||||||
- 4 channels × 2 sec × 1024 sps = 8,192 samples
|
|
||||||
- At 2 bytes/sample (int16) = 16,384 sample bytes → file would be > 16 KB
|
|
||||||
- Observed: ~9–10 KB
|
|
||||||
- → samples are likely **1 byte each** (int8 quantised), **or** stored
|
|
||||||
with bit-packing / delta encoding, **or** only one channel's
|
|
||||||
full-rate samples are stored with the others reconstructed
|
|
||||||
arithmetically. Verifying this is the **first RE milestone**.
|
|
||||||
|
|
||||||
Project-string–length variance (~1 KB across the corpus) is consistent
|
|
||||||
with the file carrying a single copy of each TitleString1..4 plus
|
|
||||||
operator + setup-name as null-padded ASCII regions.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Open questions
|
|
||||||
|
|
||||||
The reverse-engineering targets, roughly in dependency order:
|
|
||||||
|
|
||||||
1. **Sample encoding (signature A)** — int8? int16 LE/BE? Bit-packed?
|
|
||||||
Delta-coded? Per-channel interleaved or sequential blocks?
|
|
||||||
2. **Header field layout (signature A)** — where do sample_rate,
|
|
||||||
record_time, channel count, and per-channel peaks live in the
|
|
||||||
binary? The ASCII sidecar gives the device-authoritative values,
|
|
||||||
so binary fields can be confirmed by diff.
|
|
||||||
3. **Operator-string offsets** — `Operator` at 0x18 is the first
|
|
||||||
visible string in signature-A files; the rest (project, client,
|
|
||||||
notes, setup) follow. Need to map exact offsets and null-padding
|
|
||||||
conventions.
|
|
||||||
4. **Signature-B → BW codec compatibility** — does
|
|
||||||
`minimateplus/event_file_io.read_blastware_file()` actually parse
|
|
||||||
the 2 BE9439 signature-B files as-is? If yes, the OLD-format
|
|
||||||
ingest is free.
|
|
||||||
5. **`.IDFW.CDB` purpose** — is it an internal Thor cache, a
|
|
||||||
ring-buffer dump, or something else? Worth a single small effort
|
|
||||||
to characterise so we know what we're skipping.
|
|
||||||
6. **Footer / checksum** — every BW event file has a footer; does
|
|
||||||
IDF? Where does the per-channel sample block end?
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Reverse-engineering playbook (when we start)
|
|
||||||
|
|
||||||
The Series III BW codec took ~2 months of MITM wire captures
|
|
||||||
because we didn't have ground-truth metadata. Thor's situation is
|
|
||||||
**substantially better**:
|
|
||||||
|
|
||||||
- **Ground truth is on disk.** Every binary in `example-data/`
|
|
||||||
has a paired `.IDFW.txt` carrying the full decoded sample table
|
|
||||||
(`Waveform Data Channels` block — see any sample file in
|
|
||||||
`thor-watcher/example-data/.../TXT/`). Aligning binary bytes
|
|
||||||
to the table's float-per-row values gives an immediate per-byte
|
|
||||||
hypothesis test.
|
|
||||||
- **Cross-event diffing.** 1,012 signature-A samples from 9 units
|
|
||||||
spanning 4 years means any field that varies between events is
|
|
||||||
immediately localisable. Fields that are constant across all
|
|
||||||
files (firmware ID, channel labels, format-version word) are also
|
|
||||||
immediately localisable by complementary search.
|
|
||||||
- **No protocol surface.** Files at rest, not a wire dialect. No
|
|
||||||
DLE stuffing, no inner-frame parsing, no probe/data two-step.
|
|
||||||
|
|
||||||
Suggested first session (2-4 hours): hand-decode `UM11719_20231219162723.IDFW`
|
|
||||||
(10,290 bytes) against its `TXT/UM11719_20231219162723.IDFW.txt`
|
|
||||||
sample table (the 2-sec waveform at 1024 sps × 4 channels = 8,192
|
|
||||||
sample rows). Find the first per-channel sample value (`0.0003` in
|
|
||||||
the Tran column at t=0) in the binary. Confirms sample encoding.
|
|
||||||
Everything else flows from there.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Code seams ready to receive the codec
|
|
||||||
|
|
||||||
When the codec lands, it goes into
|
|
||||||
[`micromate/idf_file.py`](../micromate/idf_file.py) (currently a
|
|
||||||
stub raising `NotImplementedError`). Public API:
|
|
||||||
|
|
||||||
```python
|
|
||||||
from micromate import IdfEvent
|
|
||||||
from micromate.idf_file import read_idf_file
|
|
||||||
|
|
||||||
event: IdfEvent = read_idf_file(Path("UM11719_20231219163444.IDFW"))
|
|
||||||
# event.peaks.transverse_ips, event.timestamp, event.raw_samples, ...
|
|
||||||
```
|
|
||||||
|
|
||||||
The ingest pipeline (`WaveformStore.save_imported_idf`) currently
|
|
||||||
builds the `IdfEvent` from the `.txt` parser only. Once
|
|
||||||
`read_idf_file()` works, the binary becomes authoritative; the
|
|
||||||
`.txt` parser drops to fast-path metadata cross-check. Operators
|
|
||||||
who don't enable Thor's TXT exporter still get fully populated
|
|
||||||
events.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## See also
|
|
||||||
|
|
||||||
- [instantel_protocol_reference.md](instantel_protocol_reference.md) — Series III BW protocol reference (the Rosetta Stone). STRT record format, DLE framing, BW filename encoding.
|
|
||||||
- [`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py) — `.txt` sidecar parser.
|
|
||||||
- [`micromate/models.py`](../micromate/models.py) — `IdfEvent`, `IdfReport` typed dataclasses.
|
|
||||||
- [`micromate/idf_file.py`](../micromate/idf_file.py) — placeholder for the binary codec.
|
|
||||||
- [`thor-watcher/example-data/THORDATA_example/`](../../thor-watcher/example-data/) — 1,014 paired binary + .txt files for codec validation.
|
|
||||||
+318
-1420
File diff suppressed because it is too large
Load Diff
@@ -1,514 +0,0 @@
|
|||||||
# The "offset" fault — investigation journal
|
|
||||||
|
|
||||||
> ## ⚠ CORRECTED 2026-08-28 (same day) — the v1 detector was wrong
|
|
||||||
>
|
|
||||||
> Brian pushed back on the finding that offsets "come and go": in the field,
|
|
||||||
> once a unit develops one it stays broken until the geophone is replaced.
|
|
||||||
> He was right, and the challenge exposed **two real flaws** in the v1 detector:
|
|
||||||
>
|
|
||||||
> 1. **It scored only the axis with the largest peak.** A real event on one axis
|
|
||||||
> hid a persistent pedestal on another. BE12599 on 2026-08-21 read "clean"
|
|
||||||
> solely because Long had a 1.065 in/s event — Tran was sitting at
|
|
||||||
> **+0.4732 in/s** at that moment and was never examined.
|
|
||||||
> 2. **It used the MEAN**, which a real transient perturbs. The **median** is the
|
|
||||||
> resting baseline — most samples sit at it, so a blast does not move it.
|
|
||||||
> Same event, Long channel: mean **+0.0783** vs median **-0.0050**.
|
|
||||||
>
|
|
||||||
> Both flaws manufactured false recoveries. The corrected detector
|
|
||||||
> (`scratch/offset_scan2.py`, per-channel median) shows the pedestal is
|
|
||||||
> **persistent**, exactly as the field experience says. See §2b and §3b.
|
|
||||||
>
|
|
||||||
> **Then Brian proposed a better detector still** — measure the floor during
|
|
||||||
> the *pre-trigger* window, and require it to hold across pre/middle/end.
|
|
||||||
> That is now the detector of record (§2c). Final answer: **5 of 45 units
|
|
||||||
> (11%)**, stable across a 2x threshold range.
|
|
||||||
>
|
|
||||||
> Sections below that were written against v1 are marked; v1 numbers are kept
|
|
||||||
> for the reasoning trail, not as current fact.
|
|
||||||
|
|
||||||
A running record of the **offset** hardware fault on Instantel Series III
|
|
||||||
seismographs: a geophone channel whose trace sits displaced from zero rather
|
|
||||||
than centred on it.
|
|
||||||
|
|
||||||
This is a *journal*, not a spec. Findings are dated, dead ends are kept with
|
|
||||||
the reason they died, and every number says where it came from. When something
|
|
||||||
here is superseded, strike it and say why rather than deleting it — the point
|
|
||||||
is that a future session can tell what was actually established from what was
|
|
||||||
merely believed at the time.
|
|
||||||
|
|
||||||
Companion material:
|
|
||||||
- `scratch/offset_scan.py` — the detector
|
|
||||||
- `scratch/verify_against_ascii.py` — decoder verification harness
|
|
||||||
- `docs/instantel_protocol_reference.md` — wire protocol, incl. the
|
|
||||||
unimplemented `SUB 0x0E` this investigation now wants
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## TL;DR (current state, 2026-08-28)
|
|
||||||
|
|
||||||
- **It is real device data, not a decode bug.** Settled early and confirmed
|
|
||||||
against Blastware's own ASCII exports.
|
|
||||||
- **Base rate: 5–6 of 45 units (11–13%)** across the full DL2 archive,
|
|
||||||
2018–2026. This *confirms* the earlier 2-of-21 (9.5%) estimate from the much
|
|
||||||
smaller Terra-View DB — survivorship bias from deleted events had **not**
|
|
||||||
concealed a wave of cases.
|
|
||||||
- **The fault is bimodal, not a drift continuum.** A unit is either clean or
|
|
||||||
grossly off. Loosening the amplitude threshold 11× adds no new units.
|
|
||||||
- **The unit's own sensor check cannot see it.** 102 offset events, zero
|
|
||||||
sensor-check failures. Do not try to use it as a screen.
|
|
||||||
- **Cause is still unsettled.** Instantel's autozero fixes the minority of
|
|
||||||
cases; the rest are hardware. We cannot yet tell which is which remotely.
|
|
||||||
- **Best open lead:** `SUB 0x0E` (channel sensor data, 8 channels × 10 bytes,
|
|
||||||
unimplemented) may carry the very numbers Instantel says to check against
|
|
||||||
**2027–2069**. Untested.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. What the fault looks like
|
|
||||||
|
|
||||||
A healthy geophone trace is centred on zero. An offset channel is parked away
|
|
||||||
from zero, so the channel **mean approaches its own peak**. In Blastware the
|
|
||||||
signature is "parallel lines above or below the zero line" (Instantel's own
|
|
||||||
wording).
|
|
||||||
|
|
||||||
Consequences observed in the field:
|
|
||||||
- The unit can **self-trigger on its own offset** when the displacement exceeds
|
|
||||||
the geo trigger level, producing streams of junk events with no ground
|
|
||||||
motion. Instantel has a separate FAQ for this symptom (13-0-22, *"Unit
|
|
||||||
triggers continuously without activity"*).
|
|
||||||
- Recorded PPV for that channel is meaningless while the fault persists.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. The detector
|
|
||||||
|
|
||||||
Implemented in `scratch/offset_scan.py`. Operates on raw BW binaries only — no
|
|
||||||
DB, no sidecars.
|
|
||||||
|
|
||||||
```
|
|
||||||
for each series-3 waveform binary:
|
|
||||||
decode -> per-channel ADC counts
|
|
||||||
dominant axis = channel with the largest |peak|
|
|
||||||
flag when |mean| / peak > 0.70
|
|
||||||
and |mean| >= 0.90 x the unit's geo trigger level
|
|
||||||
episodes = per-serial runs of flagged events, split on a >12 h gap
|
|
||||||
```
|
|
||||||
|
|
||||||
Why each term:
|
|
||||||
|
|
||||||
| term | purpose |
|
|
||||||
|---|---|
|
|
||||||
| `\|mean\|/peak > 0.7` | the discriminator. A DC-parked trace has mean ≈ peak. |
|
|
||||||
| `\|mean\| >= 0.9 × trigger` | amplitude floor — suppresses quiet traces where mean and peak are both tiny and the ratio is meaningless. |
|
|
||||||
| dominant axis only | the fault is per-channel; scoring all three dilutes it. |
|
|
||||||
| 12 h episode gap | separates deployments/visits rather than counting events. |
|
|
||||||
|
|
||||||
Trigger level comes from a paired `_ASCII.TXT` when one exists, else the
|
|
||||||
per-serial median learned from that unit's ASCII files, else 0.2 in/s.
|
|
||||||
|
|
||||||
**Known limitation.** Event traces contain real ground motion, so this can only
|
|
||||||
see offsets large enough to *dominate* the trace. A mild offset on a real blast
|
|
||||||
is invisible. Instantel's A/D-mode check (§5) is the only thing that sees the
|
|
||||||
mild end. Our base rate is therefore a **gross-offset** rate.
|
|
||||||
|
|
||||||
### 2b. Detector v2 — per-channel median (CURRENT)
|
|
||||||
|
|
||||||
`scratch/offset_scan2.py`. Supersedes the above.
|
|
||||||
|
|
||||||
```
|
|
||||||
for each series-3 waveform binary:
|
|
||||||
for each geo channel independently:
|
|
||||||
pedestal = median(samples) # resting baseline, robust to blasts
|
|
||||||
flag the CHANNEL when |pedestal| >= 0.025 in/s (5 A/D counts)
|
|
||||||
a unit has a real fault when a channel is flagged on >=3 CONSECUTIVE events
|
|
||||||
```
|
|
||||||
|
|
||||||
Why median: a DC pedestal shifts every sample, so it moves the median. A real
|
|
||||||
event moves only a minority of samples, so it does not. This removes the need
|
|
||||||
for the `m/p` ratio guard entirely — that guard existed only to compensate for
|
|
||||||
using the mean.
|
|
||||||
|
|
||||||
Why per-channel: the fault is on one geophone axis. Scoring only the dominant
|
|
||||||
axis means any event with motion elsewhere hides it.
|
|
||||||
|
|
||||||
Why "3 consecutive": the 0.025 in/s floor is only ~2x a healthy channel's
|
|
||||||
resting median (observed 0.010-0.015), so isolated flags are noise. Persistence
|
|
||||||
is the discriminator — and it is what the field experience predicts.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3b. Archive results, corrected (v2)
|
|
||||||
|
|
||||||
| | v1 (dominant axis, mean) | **v2 (per-channel median)** |
|
|
||||||
|---|---|---|
|
|
||||||
| units with any flagged event | 6 of 45 | 19 of 45 |
|
|
||||||
| **units with a sustained pedestal (>=3 consecutive)** | — | **8 of 45 (18%)** |
|
|
||||||
| runs of >=3 consecutive | — | 29 |
|
|
||||||
| runs of 1-2 events (noise) | — | 69 |
|
|
||||||
|
|
||||||
Units with a sustained pedestal: **BE9558, BE10895, BE11007, BE11529, BE12599,
|
|
||||||
BE13117, BE18003, BE18438**. BE10895 and BE18003 were invisible to v1.
|
|
||||||
|
|
||||||
**The affected channel is most often Vert**, which v1 got wrong — it named
|
|
||||||
whichever axis had the largest peak. BE13117 and BE18438 are both Vert faults.
|
|
||||||
|
|
||||||
Longest / clearest runs:
|
|
||||||
|
|
||||||
| unit | ch | span | events | median in/s |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| BE13117 | Vert | 2023-05-03 → 05-04 | 194 | 0.035 → **1.915** |
|
|
||||||
| BE18438 | Vert | 2026-02-25 → 02-26 | 75 | 0.180 → 0.370 |
|
|
||||||
| BE9558 | Vert | 2020-02-11 (6 h) | 33 | 0.065 → 0.090 |
|
|
||||||
| BE12599 | Tran | 2026-08-14 → 08-23 | 8 | 0.030 → **0.565** |
|
|
||||||
| BE18003 | Vert | 2021-03-17 → 06-11 | 3 | 0.040 → 0.060 |
|
|
||||||
|
|
||||||
BE12599 began **2026-08-14**, not 08-17 as v1 reported, and was still faulting
|
|
||||||
at the last event in the archive.
|
|
||||||
|
|
||||||
### 2c. Detector v3 — PRE-TRIGGER floor + constant-floor test (CURRENT)
|
|
||||||
|
|
||||||
`scratch/offset_scan3.py`. Brian's method, and better than v2 for a reason
|
|
||||||
worth naming: **the pre-trigger window is definitionally quiet** — it is the
|
|
||||||
buffer captured before the trigger fired — whereas a whole-record median is
|
|
||||||
merely *robust* to the event. `pretrig_samples` comes from the STRT record.
|
|
||||||
|
|
||||||
```
|
|
||||||
per channel:
|
|
||||||
pre = median of the first pretrig_samples samples
|
|
||||||
mid = median of the middle third
|
|
||||||
end = median of the final third
|
|
||||||
spread = max(pre,mid,end) - min(pre,mid,end)
|
|
||||||
|
|
||||||
offset when |pre| >= floor AND spread <= 0.02 in/s
|
|
||||||
real fault when a channel is flagged on >=3 CONSECUTIVE events
|
|
||||||
```
|
|
||||||
|
|
||||||
A DC offset is a **constant floor** — present before the trigger, during, and
|
|
||||||
after. The spread test rejects transients (settling, handling, a long event
|
|
||||||
tail) that move one segment relative to the others, which is what v2's
|
|
||||||
whole-record median could not do.
|
|
||||||
|
|
||||||
**The empirical noise floor justifies the threshold.** Across 19,244
|
|
||||||
non-flagged channel-events the pre-trigger floor distributes as:
|
|
||||||
|
|
||||||
| floor | share |
|
|
||||||
|---|---|
|
|
||||||
| −1 unit (−0.005) | 18.4% |
|
|
||||||
| **0.000** | **62.7%** |
|
|
||||||
| +1 unit (+0.005) | 13.4% |
|
|
||||||
|
|
||||||
**94.5% within ±1 quantisation unit; median exactly +0.0000, mean −0.0008.**
|
|
||||||
So there is **no systematic zero-point bias in the decoder** — an independent
|
|
||||||
confirmation of the 32000-count scale. A healthy channel really does read
|
|
||||||
0.000, and "any constant floor that is not 0.000" is the right signal, with
|
|
||||||
±1 unit of slack for quantisation.
|
|
||||||
|
|
||||||
**The result is threshold-insensitive**, which is what distinguishes a real
|
|
||||||
signal from a tuned one:
|
|
||||||
|
|
||||||
| floor | units flagged | sustained units |
|
|
||||||
|---|---|---|
|
|
||||||
| 2 units (0.010) | 34 | 15 ← into the noise |
|
|
||||||
| 3 units (0.015) | 26 | 8 |
|
|
||||||
| **4 units (0.020)** | 17 | **5** |
|
|
||||||
| **5 units (0.025)** — Instantel's | 12 | **5** |
|
|
||||||
| **8 units (0.040)** | 8 | **5** |
|
|
||||||
|
|
||||||
### FINAL RESULT: 5 of 45 units (11%)
|
|
||||||
|
|
||||||
**BE9558, BE11529, BE12599, BE13117, BE18438.**
|
|
||||||
|
|
||||||
Unchanged across a 2x threshold range. BE11007 and BE10895 drop out — the
|
|
||||||
spread test identifies them as transients, not pedestals.
|
|
||||||
|
|
||||||
The 11% headline happens to match v1's, but the reasoning and the unit list
|
|
||||||
differ: v1 included BE11007 and named the wrong *channel* on most units.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Archive results (2026-08-28)
|
|
||||||
|
|
||||||
Source: DL2 event export, 6,577 **unique** series-3 waveforms, 45 units.
|
|
||||||
See [`dl2-archive`](#8-data-and-tooling) for the `Sent/` mirror trap.
|
|
||||||
|
|
||||||
**283 suspect events, 15 episodes, 6 of 45 units (13.3%).**
|
|
||||||
Excluding BE11007 (§4, likely not an offset at all): **5 of 45 = 11.1%**.
|
|
||||||
|
|
||||||
### Threshold sensitivity — the bimodality result
|
|
||||||
|
|
||||||
Re-scoring the same corpus at a range of amplitude floors, with two
|
|
||||||
ratio cut-offs (1 A/D count = 0.005 in/s, see §5):
|
|
||||||
|
|
||||||
| \|offset\| floor | m/p > 0.7 | m/p > 0.9 |
|
|
||||||
|---|---|---|
|
|
||||||
| 5 cts (0.025 in/s) — *Instantel's own* | 333 ev / 6 units | 279 ev / **5 units** |
|
|
||||||
| 10 cts (0.050) | 294 / 6 | 274 / 5 |
|
|
||||||
| 20 cts (0.100) | 250 / 5 | 244 / 4 |
|
|
||||||
| 40 cts (0.200) | 209 / 5 | 203 / 4 |
|
|
||||||
| 80 cts (0.400) | 152 / 4 | 148 / 2 |
|
|
||||||
| 160 cts (0.800) | 144 / 2 | 141 / 1 |
|
|
||||||
|
|
||||||
Relaxing the floor by 11× (0.27 → 0.025 in/s) adds ~14% more events and **no
|
|
||||||
new units**. There is no population of mild offsets hiding below our threshold
|
|
||||||
*in event data*. Either a unit is clean or it is grossly off.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Per-unit case files
|
|
||||||
|
|
||||||
Ordered by severity. `m/p` medians are on the offending channel.
|
|
||||||
|
|
||||||
### BE13117 — one violent day, never again
|
|
||||||
`145 / 454 events (32%)`, **1 episode**, 2023-05-04, 6.8 h.
|
|
||||||
Offset climbed **0.393 → 1.875 in/s within the episode**. `m/p` median
|
|
||||||
**0.996** — the trace is almost pure DC. No recurrence in the rest of its 454
|
|
||||||
events. No ASCII files in the archive, so no calibration history.
|
|
||||||
|
|
||||||
### BE18438 — recurring, months apart
|
|
||||||
`87 / 293 (30%)`, **2 episodes**: 2025-11-15 (1.2 h, n=12, 0.279 → 0.369) and
|
|
||||||
2026-02-25 (**28.8 h**, n=75, 0.183 → 0.366). `m/p` median 0.967.
|
|
||||||
Clean across all 196 events preceding its 2025-08-12 calibration.
|
|
||||||
|
|
||||||
### BE9558 — six years apart
|
|
||||||
`38 / 196 (19%)`, **4 episodes**: 2020-02-11 (6.3 h, n=33, but only
|
|
||||||
0.068 → 0.086 — very mild), then 2026-04-14, 2026-04-29, 2026-05-04
|
|
||||||
(0.28–0.45). `m/p` median 0.919. Calibrated 2026-06-26; 0/7 events flagged
|
|
||||||
after, but n=7 is far too small to call it fixed.
|
|
||||||
|
|
||||||
### BE12599 — the live case ⚠
|
|
||||||
`6 / 77 (8%)`, **6 single-event episodes, one per day at exactly 05:00**,
|
|
||||||
2026-08-17 → 2026-08-23. Offset rose 0.383 → 0.565 then fell back to 0.345.
|
|
||||||
`m/p` ≈ 0.965, geo trigger 0.3 in/s — **the offset exceeds the trigger level,
|
|
||||||
so the unit is triggering on its own fault**. Last calibrated 2025-08-12.
|
|
||||||
|
|
||||||
This is the most recent and the most useful: a currently-faulting unit is the
|
|
||||||
natural experiment for the re-zero-vs-repair question (§7).
|
|
||||||
|
|
||||||
### BE11529 — marginal
|
|
||||||
`4 / 99 (4%)`, 1 episode 2025-07-08, 0.4 h, offsets only 0.051 → 0.058 in/s.
|
|
||||||
`m/p` median 0.959, so DC-dominated, but the magnitude is near the noise of
|
|
||||||
this method. Treat as unconfirmed.
|
|
||||||
|
|
||||||
### BE11007 — probably NOT an offset
|
|
||||||
`3 / 70 (4%)`, 1 episode 2022-01-17, offsets 5.500 → 6.904 in/s — by far the
|
|
||||||
largest. But `m/p` is only **0.719–0.738** against ≥0.9 for every other unit,
|
|
||||||
and the peaks are 7.6–9.4 in/s on a 10 in/s range. That reads as a **large
|
|
||||||
real blast with asymmetric ground motion**, not a parked trace. Excluded from
|
|
||||||
the headline base rate.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Instantel's own procedure and thresholds
|
|
||||||
|
|
||||||
From two Instantel technical-support FAQs supplied 2026-08-28
|
|
||||||
(answers **13-0-21** *"How to determine offsets"* and **12-0-10** *"Removing
|
|
||||||
offsets on an Instantel Series III monitor"*; created 2008/2007, last updated
|
|
||||||
2009-03-06).
|
|
||||||
|
|
||||||
### Identifying (13-0-21)
|
|
||||||
|
|
||||||
1. Create or use an event with the **manual minimum trigger** set for the
|
|
||||||
connected geophone and microphone — i.e. an event that recorded no real data.
|
|
||||||
2. Save it and open in Blastware.
|
|
||||||
3. An offset shows as **parallel lines above or below the zero line**.
|
|
||||||
4. Put the unit in **A/D mode** — on Series III, press and hold `OPTION`, then
|
|
||||||
press `START MONITOR`.
|
|
||||||
5. **Display counts higher than 5**, with no vibration or overpressure present,
|
|
||||||
indicate an offset.
|
|
||||||
|
|
||||||
### Removing — the autozero (12-0-10)
|
|
||||||
|
|
||||||
1. Be in a **quiet area with low vibration**.
|
|
||||||
2. Power on the Blastmate III / Minimate Plus.
|
|
||||||
3. Connect the geophone and microphone — **LINEAR mic only**.
|
|
||||||
⚠ *Do not connect an "A" weight microphone, regardless of what the monitor
|
|
||||||
displays.*
|
|
||||||
4. Press `Test`.
|
|
||||||
5. Wait for the **Sensor Check** results to appear.
|
|
||||||
6. Press `OPTION` and `START MONITOR` **simultaneously**.
|
|
||||||
7. `Performing Autozero` appears; press `Enter`.
|
|
||||||
8. Confirm the sensors are properly connected; press `Enter`.
|
|
||||||
9. Wait for the autozero to complete.
|
|
||||||
10. Press `Enter` twice → Main Menu, *Ready To Monitor*, offset corrected.
|
|
||||||
|
|
||||||
### The go/no-go number — 2027 to 2069
|
|
||||||
|
|
||||||
> When you perform an Autozero on any Series III unit, the lists of numbers in
|
|
||||||
> the **X1 and X8 gains should all be between 2027 and 2069**. If not, repeat
|
|
||||||
> the Autozero. **If the numbers are extremely out of the specified range, then
|
|
||||||
> the unit should be sent in for repair.**
|
|
||||||
>
|
|
||||||
> If this process does not remove the offset problem, return the unit **and
|
|
||||||
> sensors** to Instantel for repair.
|
|
||||||
|
|
||||||
This is the documented explanation for the field experience (Brian's dad,
|
|
||||||
2026-08-28) that **a re-zero works maybe 10% of the time** — the autozero only
|
|
||||||
recovers units whose zero reference is still near-correct.
|
|
||||||
|
|
||||||
### Scale derivation (inference, well-supported — not proven)
|
|
||||||
|
|
||||||
2048 is 12-bit midscale. Our codec's geo full scale is 32000 internal counts =
|
|
||||||
10 in/s, with 1 decoder unit = 16 counts = exactly 0.005 in/s
|
|
||||||
(`geo-full-scale-is-32000-counts`). ±2000 A/D counts about 2048 therefore maps
|
|
||||||
to ±10 in/s at **0.005 in/s per A/D count**. That makes:
|
|
||||||
|
|
||||||
- Instantel's ">5 counts" threshold ≈ **0.025 in/s**
|
|
||||||
- the 2027–2069 window = **±21 counts = ±0.105 in/s** of tolerated zero error
|
|
||||||
|
|
||||||
Consistent and mutually corroborating, but we have not confirmed the A/D-count
|
|
||||||
scale directly from a device reading.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. Ruled out — keep these dead
|
|
||||||
|
|
||||||
### Condensation / humidity — DEAD (2026-08-25)
|
|
||||||
Proposed, then killed by its own controls: BE18438 stayed flat across a 10-hour
|
|
||||||
overnight gap, and only 2 of 21 units showed the fault while 19 sat in the same
|
|
||||||
weather. The apparent "diurnal cycle" was an artifact of binning by hour-of-day
|
|
||||||
across two days. See `waveform-dc-offset-is-real-device-data`.
|
|
||||||
|
|
||||||
### Clipping as a false-positive source — RULED OUT (2026-08-28)
|
|
||||||
A rail-hitting trace would fake an offset (mean → peak). It isn't happening:
|
|
||||||
median suspect peak is only **10% of full scale**, p90 is 18.6%. Only BE11007's
|
|
||||||
3 events exceed 50% FS, and none reach 98%.
|
|
||||||
|
|
||||||
### The sensor check as a predictor — DOES NOT WORK (2026-08-28)
|
|
||||||
Tested on 102 offset events across 4 units:
|
|
||||||
|
|
||||||
| unit | state | n | failed | median ratio | median freq |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| BE11529 | offset | 4 | **0** | 3.90 | 7.6 |
|
|
||||||
| BE11529 | clean | 14 | 0 | 3.80 | 7.5 |
|
|
||||||
| BE12599 | offset | 6 | **0** | 4.00 | 7.4 |
|
|
||||||
| BE12599 | clean | 13 | 0 | 4.00 | 7.6 |
|
|
||||||
| BE18438 | offset | 87 | **0** | 3.70 | 7.6 |
|
|
||||||
| BE18438 | clean | 25 | 0 | 3.80 | 7.5 |
|
|
||||||
| BE9558 | offset | 5 | **0** | 3.90 | 7.8 |
|
|
||||||
| BE9558 | clean | 44 | 0 | 3.80 | 7.5 |
|
|
||||||
|
|
||||||
Zero failures on either side and indistinguishable ratios/frequencies. The
|
|
||||||
swing test measures geophone frequency response and damping — it never examines
|
|
||||||
DC zero. **A grossly offset unit passes its own self-check.** This is why the
|
|
||||||
fault goes unnoticed until somebody looks at waveforms.
|
|
||||||
|
|
||||||
### "Offsets are transient / come and go on their own" — RETRACTED 2026-08-28
|
|
||||||
v1 reported episodes lasting hours that ended spontaneously. **This was an
|
|
||||||
artifact of the v1 detector** (see the banner at the top). With the per-channel
|
|
||||||
median, the pedestal persists. Every clear case reads clean again only after a
|
|
||||||
multi-day-to-multi-month gap consistent with service: BE13117 6 days, BE18438
|
|
||||||
24 days, BE9558 63 days **with a confirmed Instantel calibration inside the
|
|
||||||
gap**. BE12599 never reads clean — it is still faulting at the end of the
|
|
||||||
archive. This matches the operational experience: once a unit develops an
|
|
||||||
offset it stays broken until the geophone is replaced.
|
|
||||||
|
|
||||||
### "Offsets develop N months after calibration" — CONFOUNDED, NOT A FINDING
|
|
||||||
Tempting, and it looked strong:
|
|
||||||
|
|
||||||
| unit | suspect before latest cal | after |
|
|
||||||
|---|---|---|
|
|
||||||
| BE18438 | 0 / 196 | 87 / 97 |
|
|
||||||
| BE12599 | 0 / 62 | 6 / 15 |
|
|
||||||
| BE11529 | 0 / 82 | 4 / 17 |
|
|
||||||
| BE9558 | 38 / 189 | 0 / 7 |
|
|
||||||
|
|
||||||
But bucketing suspects by months-since-calibration gives **one unit per bucket**:
|
|
||||||
`0–3mo={BE11529}`, `3–6 & 6–9mo={BE18438}`, `9–12mo={BE9558}`,
|
|
||||||
`12–15mo={BE12599}`. The apparent "51% failure rate at 6–9 months" is entirely
|
|
||||||
BE18438's single February 2026 episode. Five units with roughly one episode
|
|
||||||
each cannot support a population trend. **Do not re-derive this.**
|
|
||||||
|
|
||||||
Also note: all affected units are calibrated on a **~12–13 month cadence**, so
|
|
||||||
"sent to Instantel" is the routine annual schedule, not evidence of a
|
|
||||||
fault-driven return.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. Open questions
|
|
||||||
|
|
||||||
### Q1 — Is it a latched bad zero or analog degradation?
|
|
||||||
The question that decides everything. A latched zero is correctable (possibly
|
|
||||||
over the wire); degradation means a repair. Instantel's 2027–2069 rule implies
|
|
||||||
*both* populations exist, with the split roughly 10/90 in the field.
|
|
||||||
|
|
||||||
**BE12599 is the natural experiment** — faulting as of 2026-08-23. Read its
|
|
||||||
values, run the autozero, read them again.
|
|
||||||
|
|
||||||
### Q2 — Can we read the autozero numbers over the wire? (best lead)
|
|
||||||
Instantel says to check *"the lists of numbers in the **X1 and X8 gains**"* —
|
|
||||||
4 sensors × 2 gains = **8 channels**. The protocol reference already documents
|
|
||||||
an unimplemented command with exactly that shape:
|
|
||||||
|
|
||||||
```
|
|
||||||
SUB 0x0E -> RSP 0xF1 "channel sensor data"
|
|
||||||
2-step read; channel selector in params[6:8] = 0x0000..0x0007
|
|
||||||
data length 0x0A (10 bytes) per channel
|
|
||||||
```
|
|
||||||
|
|
||||||
Blastware's *Unit Channel Test* sequence:
|
|
||||||
`POLL×N → 0x15 → 0x01 → 0x08 → 0x01 → 0x0E×8 → 0x98×2 → 0x0E×8`
|
|
||||||
— note the **second `0x0E` pass carries live ADC readings**.
|
|
||||||
|
|
||||||
**Hypothesis (untested):** `0x0E` returns the numbers Instantel wants compared
|
|
||||||
against 2027–2069. If true, SFM could diagnose an offset remotely *and* predict
|
|
||||||
whether a re-zero will succeed — converting a 10%/90% shipping gamble into a
|
|
||||||
decision made before packing a box.
|
|
||||||
|
|
||||||
**How to test.** `bridges/ach_mitm.py` is a generic TCP proxy:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python bridges/ach_mitm.py --bw-host <MODEM_IP> --bw-port 9034 --listen-port 9999
|
|
||||||
```
|
|
||||||
|
|
||||||
Point Blastware at the proxy and run **Unit Channel Test**.
|
|
||||||
⚠ In this topology the output filenames are reversed — the tool labels the
|
|
||||||
*connecting* side "unit", so `raw_s3_*.bin` holds Blastware's bytes and
|
|
||||||
`raw_bw_*.bin` the unit's.
|
|
||||||
|
|
||||||
Capture priority: (1) BE12599 while faulting, (2) a known-good unit as control,
|
|
||||||
(3) before/after an autozero on the same unit. Eight 10-byte payloads with an
|
|
||||||
expected value near 2048 is a very constrained puzzle.
|
|
||||||
|
|
||||||
### Q3 — What is the mild-offset rate?
|
|
||||||
Unmeasurable from event files (§2). Only the A/D-mode check sees it. Would
|
|
||||||
need a fleet sweep in A/D mode, or Q2 to succeed.
|
|
||||||
|
|
||||||
### Q4 — Does an offset recur on the same unit after service?
|
|
||||||
BE9558 shows episodes in 2020 and 2026; BE18438 twice in four months. Suggestive
|
|
||||||
of recurrence, but service records aren't in the data — only calibration dates.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8. Data and tooling
|
|
||||||
|
|
||||||
| what | where |
|
|
||||||
|---|---|
|
|
||||||
| detector | `scratch/offset_scan.py` |
|
|
||||||
| current results | `/home/serversdown/dl2-archive/offset_archive.csv` |
|
|
||||||
| earlier candidate list (Terra-View DB, 274 events) | `scratch/offset_candidates.csv` |
|
|
||||||
| archive working copy | `/home/serversdown/dl2-archive/files/` |
|
|
||||||
| archive source | NAS `DeathStar` 10.0.0.2, `/volume1/Uploads/TMI/DL2-Event-backup-8-25-26/Event/autocall home/` |
|
|
||||||
|
|
||||||
⚠ **The DL2 export keeps a byte-identical `Sent/` mirror of its root.** 13,077
|
|
||||||
waveform paths are 6,577 distinct files. Always dedupe by basename — this
|
|
||||||
doubled two reported figures before it was caught.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 9. Chronology
|
|
||||||
|
|
||||||
| date | event |
|
|
||||||
|---|---|
|
|
||||||
| 2026-08-25 | Reported as a *waveform decode bug* — traces with a DC offset. Investigation shows the offset is **real device data**; the decoder is correct. |
|
|
||||||
| 2026-08-25 | Brian relays his dad's description: a known hardware fault called an "offset"; usually sent to Instantel. |
|
|
||||||
| 2026-08-25 | First detection pass over the Terra-View DB: **2 of 21 units**, 274 events, 11 months. Flagged as vulnerable to survivorship bias — flooded events were routinely deleted. |
|
|
||||||
| 2026-08-25 | Condensation hypothesis proposed, then **killed by its own controls**. |
|
|
||||||
| 2026-08-25 | Parked pending the multi-year archive. |
|
|
||||||
| 2026-08-28 | DL2 archive pulled (33 GB, 546k files; 6.6 GB working set). |
|
|
||||||
| 2026-08-28 | Archive scan: **6 of 45 units**, 283 events, 15 episodes. Prior base rate **confirmed**, not overturned. |
|
|
||||||
| 2026-08-28 | Clipping ruled out; `m/p` established as the discriminator; BE11007 reclassified as probably a real blast. |
|
|
||||||
| 2026-08-28 | Calibration-timing correlation attempted and **rejected as confounded**. |
|
|
||||||
| 2026-08-28 | Instantel FAQs supplied: autozero procedure, the **2027–2069** window, the **>5 counts** threshold. Explains the ~10% re-zero success rate. |
|
|
||||||
| 2026-08-28 | Bimodality established; sensor check proven **blind** to offsets; `SUB 0x0E` identified as the best open lead. |
|
|
||||||
| 2026-08-28 | **v1 detector retracted.** Brian challenged the "come and go" finding against field experience. Two flaws found: dominant-axis-only scoring and mean-instead-of-median. Corrected detector shows persistent pedestals on **8 of 45 units**, and the gaps are service windows. |
|
|
||||||
| 2026-08-28 | **Detector v3 (Brian's method):** pre-trigger floor + pre/mid/end consistency. Healthy channels proven to sit at 0.000 +/-1 unit (94.5%), confirming no decoder zero-point bias. Final: **5 of 45 units (11%)**, threshold-insensitive. |
|
|
||||||
@@ -1,255 +0,0 @@
|
|||||||
# Runbook — Recovering a wedged unit stuck in a call-home loop
|
|
||||||
|
|
||||||
**Original incident:** BE9558H at `166.246.130.1:9034`, recovered 2026-05-17.
|
|
||||||
|
|
||||||
A field unit with a stuck-triggered geophone (or any hardware fault causing
|
|
||||||
constant event triggering) will record events back-to-back, and if Auto Call
|
|
||||||
Home is set to "After Event Recorded" the device will dial the office BW
|
|
||||||
ACH server in a tight loop. Combined with a Sierra Wireless modem in
|
|
||||||
bidirectional serial-TCP mode, this makes the unit effectively unreachable
|
|
||||||
from SFM — every TCP connection we open gets killed when the modem flips
|
|
||||||
from server-mode to client-mode to honor the device's next AT dial command.
|
|
||||||
|
|
||||||
This runbook describes how to break the loop and recover control.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Symptoms
|
|
||||||
|
|
||||||
- Terra-View / SFM `/device/info` either hangs or fails on `count_events()`.
|
|
||||||
- `/device/monitor/status` and `/device/rescue` return 502 (protocol timeout
|
|
||||||
waiting for POLL response) or 503 (TCP connect refused).
|
|
||||||
- ACEmanager serial log shows repeating
|
|
||||||
`Connect to IP: <BW_IP> Port: <BW_PORT>` → `Shutdown TCP socket` cycles
|
|
||||||
every 30-60 seconds.
|
|
||||||
- Spam-mode endpoints (`/device/stop_monitoring_spam`) report many
|
|
||||||
`sent_ok` but the device's monitoring state never changes.
|
|
||||||
- `slow_drip` reports `[Errno 32] Broken pipe` after sending the preamble
|
|
||||||
but before completing the drip loop.
|
|
||||||
|
|
||||||
If you see *all* of these, the unit is in this exact failure mode.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Quick reference — how to recover
|
|
||||||
|
|
||||||
You need **ACEmanager access** to the unit's modem.
|
|
||||||
|
|
||||||
### Step 1: stop the modem's mode-flipping
|
|
||||||
|
|
||||||
In ACEmanager → **Serial → Port Configuration**:
|
|
||||||
|
|
||||||
| Field | Set to |
|
|
||||||
|---|---|
|
|
||||||
| **Destination Address** | clear (blank) |
|
|
||||||
| **Destination Port** | `0` |
|
|
||||||
|
|
||||||
Click **Apply**. This removes the modem's auto-dial-out target. The device's
|
|
||||||
AT dial commands now error back at the modem instead of triggering a
|
|
||||||
mode-flip, so the modem stays in TCP-server mode permanently and our inbound
|
|
||||||
TCP sessions stay alive.
|
|
||||||
|
|
||||||
*(Optional belt-and-suspenders: also add the BW server's port to
|
|
||||||
**Security → Port Filtering - Outbound** as a blocked port, with
|
|
||||||
Outbound Port Filtering Mode = Blocked Ports.)*
|
|
||||||
|
|
||||||
### Step 2: stop monitoring on the device (slow drip)
|
|
||||||
|
|
||||||
From the SFM host:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
/home/serversdown/seismo-relay/scripts/slow_drip.sh <DEVICE_IP> <PORT>
|
|
||||||
```
|
|
||||||
|
|
||||||
Defaults are 120s duration with a drip every 3s. Watch the response:
|
|
||||||
|
|
||||||
- `duration_s ≈ 120` and `drips_sent ≈ 40` → session held the full duration ✓
|
|
||||||
- `bytes_received > 0` → device is responding ✓ (this is the success signal)
|
|
||||||
|
|
||||||
If `duration_s` is small or `send_error: "Broken pipe"`, Step 1 didn't take
|
|
||||||
hold — re-check ACEmanager, may need to reboot the modem after Apply.
|
|
||||||
|
|
||||||
### Step 3: confirm monitoring stopped
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl 'http://localhost:8200/device/monitor/status?host=<DEVICE_IP>&tcp_port=<PORT>&force=true'
|
|
||||||
# expect: {"is_monitoring": false, ...}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Step 4: disable ACH at the device level + erase corrupted events
|
|
||||||
|
|
||||||
Either fire the rescue endpoint:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
/home/serversdown/seismo-relay/scripts/rescue_device.sh <DEVICE_IP> <PORT>
|
|
||||||
```
|
|
||||||
|
|
||||||
Or do the two steps manually:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Disable ACH in the device's compliance config
|
|
||||||
curl -X POST 'http://localhost:8200/device/call_home?host=<DEVICE_IP>&tcp_port=<PORT>' \
|
|
||||||
-H 'Content-Type: application/json' \
|
|
||||||
-d '{"auto_call_home_enabled": false}'
|
|
||||||
|
|
||||||
# Erase corrupted event chain
|
|
||||||
curl -X POST 'http://localhost:8200/device/events/erase?host=<DEVICE_IP>&tcp_port=<PORT>'
|
|
||||||
```
|
|
||||||
|
|
||||||
You can also do this via the SFM standalone UI → **Call Home** tab → set
|
|
||||||
`Enable Auto Call Home` to `Disabled` → **Write to Device**.
|
|
||||||
|
|
||||||
### Step 5: restore modem config (housekeeping)
|
|
||||||
|
|
||||||
Once the device-side ACH is disabled, restore the modem's Destination
|
|
||||||
Address and Port to the original values (e.g. `50.197.32.92` / `12345`) in
|
|
||||||
ACEmanager. The modem will resume normal bidirectional behavior, but the
|
|
||||||
unit won't issue any dial commands until ACH is explicitly re-enabled on
|
|
||||||
the device.
|
|
||||||
|
|
||||||
### Step 6: do NOT re-enable ACH on this unit until the underlying hardware
|
|
||||||
fault is repaired. If you do, the call-home loop starts again immediately
|
|
||||||
and you'll be running this runbook a second time.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Why this works — the failure mode explained
|
|
||||||
|
|
||||||
The Sierra Wireless RV50/RV55 serial port operates in one of two TCP modes
|
|
||||||
at any moment:
|
|
||||||
|
|
||||||
- **Server mode** — listens on `Device Port` (e.g. 9034), bridges inbound
|
|
||||||
TCP to the device's serial port. This is what we need to interact with
|
|
||||||
the device.
|
|
||||||
- **Client mode** — when the device sends an AT dial command on its serial
|
|
||||||
TX line, the modem opens an outbound TCP to `Destination Address:Port`
|
|
||||||
and bridges that to serial.
|
|
||||||
|
|
||||||
A serial port in this configuration is **bidirectional**: the modem flips
|
|
||||||
between server and client modes on demand. When the device's firmware is
|
|
||||||
healthy and only dials occasionally, this works fine.
|
|
||||||
|
|
||||||
When the unit is constantly triggering events and ACH is set to "After
|
|
||||||
Event Recorded", the device sends an AT dial command every few seconds.
|
|
||||||
Each one causes the modem to:
|
|
||||||
|
|
||||||
1. Drop any active inbound TCP session
|
|
||||||
2. Flip to client mode
|
|
||||||
3. Attempt outbound TCP to `Destination Address:Port`
|
|
||||||
4. Hang for up to a minute waiting for it to succeed/fail
|
|
||||||
5. Drop back to server mode
|
|
||||||
|
|
||||||
**During the entire hang, no inbound TCP can establish.** Even between
|
|
||||||
hangs, the modem closes any existing inbound session before flipping. So
|
|
||||||
any tool that needs more than a few seconds of held TCP (e.g. POLL +
|
|
||||||
config read + write) gets repeatedly kicked off.
|
|
||||||
|
|
||||||
Clearing `Destination Address` removes step 3-4 from the cycle: the modem
|
|
||||||
has nowhere to dial, so it doesn't flip modes when it receives an AT dial
|
|
||||||
command. The serial port effectively becomes server-only, and inbound TCP
|
|
||||||
sessions can stay open as long as needed.
|
|
||||||
|
|
||||||
**This is a modem-layer issue, not a device firmware issue.** The device
|
|
||||||
is alive and responsive the whole time — confirmed in the BE9558H
|
|
||||||
recovery by 990 bytes of S3 responses received over a 120s slow-drip
|
|
||||||
session once the modem was no longer mode-flipping.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Why simpler approaches don't work
|
|
||||||
|
|
||||||
| Approach | Why it fails |
|
|
||||||
|---|---|
|
|
||||||
| Standard `/device/info` | Triggers `count_events()` 1E/1F walk, takes 90s+ and hits corrupted event chain in this scenario |
|
|
||||||
| `/device/rescue` race loop | Gets 502 (protocol timeout) because the modem closes the TCP before the POLL handshake can complete |
|
|
||||||
| `/device/stop_monitoring_blind` (single frame) | Even if the bytes leave the wire, the device's protocol parser ignores write commands without a preceding POLL handshake (early-version bug, now fixed by including POLL preamble in blind sends) |
|
|
||||||
| `/device/stop_monitoring_spam` (sub-second cadence) | Each session is killed by the modem's mode-flip before the device can drain its UART RX buffer; high-rate spam also risks UART FIFO overrun on the device side |
|
|
||||||
| Outbound port firewall block alone | Stops the outbound TCP from succeeding, but doesn't stop the modem from *trying* and mode-flipping. Reduces but doesn't eliminate the contention. |
|
|
||||||
| Modem reboot | Temporary — as soon as the device starts triggering again, the loop resumes within seconds |
|
|
||||||
|
|
||||||
The combination of `slow_drip` + cleared `Destination Address` works because:
|
|
||||||
|
|
||||||
1. The modem stops mode-flipping → TCP session stays open for the full
|
|
||||||
drip duration
|
|
||||||
2. Slow drip rate → device's UART RX FIFO never overflows even if
|
|
||||||
firmware is busy with event recording
|
|
||||||
3. The drip is `SESSION_RESET + STOP_MONITORING` every 3s → many
|
|
||||||
independent chances for the parser to land one valid frame
|
|
||||||
4. Once one Stop Monitoring is parsed, event recording halts → firmware
|
|
||||||
has CPU to spare → subsequent operations are trivially easy
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Tooling reference
|
|
||||||
|
|
||||||
All endpoints live in `seismo-relay/sfm/server.py`. All scripts live in
|
|
||||||
`seismo-relay/scripts/` and default to SFM direct (`http://localhost:8200`),
|
|
||||||
overridable via `SFM_BASE_URL`.
|
|
||||||
|
|
||||||
### Endpoints added during BE9558H recovery
|
|
||||||
|
|
||||||
| Endpoint | Purpose |
|
|
||||||
|---|---|
|
|
||||||
| `GET /device/events/storage_range` | SUB 0x06 — first/last event keys, `is_empty` flag. ~2s, no event walk. |
|
|
||||||
| `GET /device/events/index` | SUB 0x08 — lifetime event counter (does NOT decrement on erase). ~2s. |
|
|
||||||
| `POST /device/events/erase` | Full erase sequence 0xA3 → 0x1C → 0x06 → 0xA2. |
|
|
||||||
| `POST /device/rescue` | Disable ACH + erase in one TCP session. Short timeouts for race-loop usage. |
|
|
||||||
| `POST /device/stop_monitoring_blind` | Fire-and-forget Stop with full POLL preamble (single attempt). |
|
|
||||||
| `POST /device/stop_monitoring_spam` | Server-side tight retry loop, sub-second cadence, duration-bounded. |
|
|
||||||
| `POST /device/stop_monitoring_slow_drip` | One held TCP session, slow trickle of stop frames. **The endpoint that saved BE9558H.** |
|
|
||||||
|
|
||||||
Also changed: default protocol recv timeout dropped from 30s → 10s in
|
|
||||||
`_build_client`. Added `connect_timeout` knob to same. Cleaned up
|
|
||||||
unhandled-exception path in `/device/monitor/status` so it returns 502
|
|
||||||
instead of 500 on protocol timeouts.
|
|
||||||
|
|
||||||
### Scripts
|
|
||||||
|
|
||||||
| Script | Purpose |
|
|
||||||
|---|---|
|
|
||||||
| `scripts/rescue_device.sh` | Race-loop wrapper around `/device/rescue` |
|
|
||||||
| `scripts/blind_stop.sh` | Race-loop wrapper around `/device/stop_monitoring_blind` |
|
|
||||||
| `scripts/spam_stop.sh` | Single-call burst hammer (`/device/stop_monitoring_spam`) |
|
|
||||||
| `scripts/slow_drip.sh` | Single-call held-session drip (`/device/stop_monitoring_slow_drip`) |
|
|
||||||
| `scripts/watch_unit.sh` | Passive periodic reachability check, logs to file |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Incident log — BE9558H, 2026-05-16/17
|
|
||||||
|
|
||||||
What was wrong: Long-axis geophone developed an offset, constantly above
|
|
||||||
trigger threshold → constant event recording → after-event ACH set →
|
|
||||||
modem dialing office BW server (`50.197.32.92:12345`) every 30-60s.
|
|
||||||
Local event chain corrupted (`next_boundary 0x100EE exceeds uint16`).
|
|
||||||
|
|
||||||
Diagnostic path:
|
|
||||||
|
|
||||||
1. `/device/info` slow, choked on event walk
|
|
||||||
2. Built lightweight probe endpoints (`storage_range`, `index`) — useful
|
|
||||||
but didn't reach the wedged unit
|
|
||||||
3. Built `/device/rescue` with short timeouts — got 502 (POLL no response)
|
|
||||||
4. Built `/device/stop_monitoring_blind` — first version was a false
|
|
||||||
positive (no POLL preamble); fixed by including
|
|
||||||
`SESSION_RESET+POLL_PROBE+SESSION_RESET+POLL_DATA` in the dump
|
|
||||||
5. Verified blind stop works on bench unit
|
|
||||||
6. Built `/device/stop_monitoring_spam` — 420 successful sends over
|
|
||||||
5 min, zero behavior change on field unit
|
|
||||||
7. Inspected ACEmanager logs → saw outbound dial-out attempts every ~30s,
|
|
||||||
confirmed device was not fully locked up
|
|
||||||
8. Added outbound port-12345 firewall block → outbound attempts now fail
|
|
||||||
instantly but contention persisted
|
|
||||||
9. Built `/device/stop_monitoring_slow_drip` — session died at 3s with
|
|
||||||
broken pipe (modem closing on us)
|
|
||||||
10. Looked at full ACEmanager Port Configuration → **found
|
|
||||||
`Destination Address: 50.197.32.92` configured**, realized every AT
|
|
||||||
dial command was triggering a modem mode-flip that killed our inbound
|
|
||||||
11. Cleared Destination Address + Port → slow_drip held 120s, device
|
|
||||||
responded with 990 bytes, 39 stop commands acked
|
|
||||||
12. Disabled ACH at device level via `/device/call_home`, erased events
|
|
||||||
|
|
||||||
Final state: device IDLE, memory 958.1 / 960 KB free, ACH disabled at
|
|
||||||
device level, modem destination cleared (to be restored after physical
|
|
||||||
service).
|
|
||||||
|
|
||||||
Total time from "i was wondering if its possible to" first attempt to
|
|
||||||
recovery: ~7 hours of intermittent debugging across one evening.
|
|
||||||
@@ -1,628 +0,0 @@
|
|||||||
# Waveform-shape FT detection — Phase A (seismo-relay) Implementation Plan
|
|
||||||
|
|
||||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
||||||
|
|
||||||
**Goal:** Compute per-event waveform-shape metrics (crest factor + points-near-peak) in SFM and store them as `events` columns, populated at ingest and by a backfill script, so Terra-View can read them from `/db/events`.
|
|
||||||
|
|
||||||
**Architecture:** A pure DSP module (`sfm/shape_metrics.py`) turns decoded `.h5` samples into shape metrics. New nullable `events.shape_*` columns are added via the existing `_migrate` ADD COLUMN loop. The three `WaveformStore.save*` paths compute shape from the just-written `.h5` and hand it to `insert_events`; a backfill script does the same over existing events. Mirrors exactly how per-channel ZC frequency was added.
|
|
||||||
|
|
||||||
**Tech Stack:** Python 3.10, numpy, h5py, sqlite3 (raw), pytest. seismo-relay venv: `/home/serversdown/seismo-relay/.venv/bin/python3`.
|
|
||||||
|
|
||||||
## Global Constraints
|
|
||||||
|
|
||||||
- Metrics are read from the `.h5` `samples/{Tran,Vert,Long}` float32 arrays (physical in/s). The measured channel is the max-|peak| geophone channel.
|
|
||||||
- All new columns are nullable; histogram records and events without usable samples store NULL (Terra-View falls back to cheap signals). Legacy rows stay valid.
|
|
||||||
- Crest factor = `max(|x|) / rms(x)`; near-peak count = number of samples with `|x| ≥ 0.5·peak`. Threshold `0.5` is a module constant so calibration can tune it.
|
|
||||||
- No manual migration: columns are added in `SeismoDb._migrate`, run at `SeismoDb()` construction.
|
|
||||||
- `/db/events` needs no change — it returns all columns via `SELECT *` (verify with a test).
|
|
||||||
- Run tests with `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 1: Shape DSP module
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Create: `sfm/shape_metrics.py`
|
|
||||||
- Test: `tests/test_shape_metrics.py`
|
|
||||||
|
|
||||||
**Interfaces:**
|
|
||||||
- Produces:
|
|
||||||
- `channel_shape(x) -> dict | None` — `{"crest_factor": float, "near_peak_count": int, "sample_count": int}` or None for unusable input (size < 2, flat, all-zero).
|
|
||||||
- `shape_from_samples(chans: dict[str, ArrayLike]) -> dict | None` — picks the max-peak geophone channel; returns `{"crest_factor","near_peak_count","sample_count","axis"}` or None.
|
|
||||||
- `shape_from_h5(path) -> dict | None` — reads `samples/{Tran,Vert,Long}` and delegates to `shape_from_samples`; None on any read error.
|
|
||||||
- Constant `NEAR_PEAK_FRACTION = 0.5`.
|
|
||||||
|
|
||||||
- [ ] **Step 1: Write the failing test**
|
|
||||||
|
|
||||||
```python
|
|
||||||
# tests/test_shape_metrics.py
|
|
||||||
import numpy as np
|
|
||||||
from sfm.shape_metrics import channel_shape, shape_from_samples
|
|
||||||
|
|
||||||
def test_needle_spike_high_crest_few_near_peak():
|
|
||||||
x = np.zeros(1024); x[500] = 1.0 # one isolated spike
|
|
||||||
s = channel_shape(x)
|
|
||||||
assert s["sample_count"] == 1024
|
|
||||||
assert s["crest_factor"] > 15 # peak towers over rms
|
|
||||||
assert s["near_peak_count"] <= 3 # almost nothing near the peak
|
|
||||||
|
|
||||||
def test_ringing_low_crest_many_near_peak():
|
|
||||||
t = np.arange(1024)
|
|
||||||
x = np.sin(2*np.pi*t/32) * np.exp(-t/4000) # decaying oscillation
|
|
||||||
s = channel_shape(x)
|
|
||||||
assert s["crest_factor"] < 6
|
|
||||||
assert s["near_peak_count"] > 30 # many samples near the peak
|
|
||||||
|
|
||||||
def test_channel_shape_none_for_unusable():
|
|
||||||
assert channel_shape(np.zeros(1024)) is None # flat / all-zero
|
|
||||||
assert channel_shape(np.array([1.0])) is None # too short
|
|
||||||
|
|
||||||
def test_shape_from_samples_picks_max_peak_axis():
|
|
||||||
chans = {"Tran": np.zeros(1024), "Vert": np.zeros(1024), "Long": np.zeros(1024)}
|
|
||||||
chans["Long"][10] = 0.5
|
|
||||||
chans["Vert"] = np.sin(np.arange(1024)/5) * 0.01
|
|
||||||
s = shape_from_samples(chans)
|
|
||||||
assert s["axis"] == "Long" # Long has the biggest peak
|
|
||||||
assert s["near_peak_count"] <= 3
|
|
||||||
|
|
||||||
def test_shape_from_samples_none_when_no_geo():
|
|
||||||
assert shape_from_samples({"MicL": np.ones(1024)}) is None
|
|
||||||
```
|
|
||||||
|
|
||||||
- [ ] **Step 2: Run test to verify it fails**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_metrics.py -q`
|
|
||||||
Expected: FAIL — `ModuleNotFoundError: sfm.shape_metrics`.
|
|
||||||
|
|
||||||
- [ ] **Step 3: Write minimal implementation**
|
|
||||||
|
|
||||||
```python
|
|
||||||
# sfm/shape_metrics.py
|
|
||||||
"""Waveform-shape metrics for false-trigger detection.
|
|
||||||
|
|
||||||
A false trigger is an isolated impulse (quiet → spike → quiet); a real event
|
|
||||||
rings for many cycles. Two numbers separate them: crest factor (how far the
|
|
||||||
peak stands above the typical sample) and how many samples sit near the peak.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
import numpy as np
|
|
||||||
|
|
||||||
_GEO_CHANNELS = ("Tran", "Vert", "Long")
|
|
||||||
NEAR_PEAK_FRACTION = 0.5 # a sample "near the peak" is >= this * peak amplitude
|
|
||||||
|
|
||||||
|
|
||||||
def channel_shape(x) -> dict | None:
|
|
||||||
x = np.asarray(x, dtype=float)
|
|
||||||
if x.size < 2:
|
|
||||||
return None
|
|
||||||
peak = float(np.max(np.abs(x)))
|
|
||||||
if peak <= 0:
|
|
||||||
return None
|
|
||||||
rms = float(np.sqrt(np.mean(x ** 2)))
|
|
||||||
if rms <= 0:
|
|
||||||
return None
|
|
||||||
near = int(np.sum(np.abs(x) >= NEAR_PEAK_FRACTION * peak))
|
|
||||||
return {"crest_factor": peak / rms, "near_peak_count": near,
|
|
||||||
"sample_count": int(x.size)}
|
|
||||||
|
|
||||||
|
|
||||||
def shape_from_samples(chans: dict) -> dict | None:
|
|
||||||
best_axis, best_peak, best_x = None, -1.0, None
|
|
||||||
for ax in _GEO_CHANNELS:
|
|
||||||
x = chans.get(ax)
|
|
||||||
if x is None:
|
|
||||||
continue
|
|
||||||
x = np.asarray(x, dtype=float)
|
|
||||||
if x.size < 2:
|
|
||||||
continue
|
|
||||||
p = float(np.max(np.abs(x)))
|
|
||||||
if p > best_peak:
|
|
||||||
best_axis, best_peak, best_x = ax, p, x
|
|
||||||
if best_axis is None:
|
|
||||||
return None
|
|
||||||
s = channel_shape(best_x)
|
|
||||||
if s is None:
|
|
||||||
return None
|
|
||||||
s["axis"] = best_axis
|
|
||||||
return s
|
|
||||||
|
|
||||||
|
|
||||||
def shape_from_h5(path) -> dict | None:
|
|
||||||
import h5py
|
|
||||||
try:
|
|
||||||
with h5py.File(path, "r") as f:
|
|
||||||
chans = {ax: f[f"samples/{ax}"][:] for ax in _GEO_CHANNELS
|
|
||||||
if f"samples/{ax}" in f}
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
return shape_from_samples(chans)
|
|
||||||
```
|
|
||||||
|
|
||||||
- [ ] **Step 4: Run test to verify it passes**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_metrics.py -q`
|
|
||||||
Expected: PASS (5 tests).
|
|
||||||
|
|
||||||
- [ ] **Step 5: Commit**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git add sfm/shape_metrics.py tests/test_shape_metrics.py
|
|
||||||
git commit -m "feat(shape): crest-factor + points-near-peak waveform metrics"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 2: shape_from_h5 round-trips a real .h5
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Test: `tests/test_shape_metrics_h5.py`
|
|
||||||
|
|
||||||
**Interfaces:**
|
|
||||||
- Consumes: `sfm.shape_metrics.shape_from_h5`; `h5py`.
|
|
||||||
|
|
||||||
- [ ] **Step 1: Write the failing test** (writes a tiny .h5 the same shape SFM writes, then reads it back)
|
|
||||||
|
|
||||||
```python
|
|
||||||
# tests/test_shape_metrics_h5.py
|
|
||||||
import numpy as np, h5py
|
|
||||||
from sfm.shape_metrics import shape_from_h5
|
|
||||||
|
|
||||||
def _write_h5(path, chans):
|
|
||||||
with h5py.File(path, "w") as f:
|
|
||||||
g = f.create_group("samples")
|
|
||||||
for k, v in chans.items():
|
|
||||||
g.create_dataset(k, data=np.asarray(v, dtype="float32"))
|
|
||||||
|
|
||||||
def test_shape_from_h5_reads_dominant_axis(tmp_path):
|
|
||||||
p = tmp_path / "ev.h5"
|
|
||||||
long = np.zeros(1024, dtype="float32"); long[100] = 0.48
|
|
||||||
_write_h5(p, {"Tran": np.zeros(1024), "Vert": np.zeros(1024), "Long": long,
|
|
||||||
"MicL": np.ones(1024)})
|
|
||||||
s = shape_from_h5(str(p))
|
|
||||||
assert s["axis"] == "Long" and s["near_peak_count"] <= 3
|
|
||||||
|
|
||||||
def test_shape_from_h5_none_on_missing_or_degenerate(tmp_path):
|
|
||||||
assert shape_from_h5(str(tmp_path / "nope.h5")) is None
|
|
||||||
p = tmp_path / "degen.h5"
|
|
||||||
_write_h5(p, {"Tran": np.zeros(1), "Vert": np.zeros(1), "Long": np.zeros(1)})
|
|
||||||
assert shape_from_h5(str(p)) is None
|
|
||||||
```
|
|
||||||
|
|
||||||
- [ ] **Step 2: Run to verify it fails**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_metrics_h5.py -q`
|
|
||||||
Expected: FAIL (assertion or, if Task 1 incomplete, import error).
|
|
||||||
|
|
||||||
- [ ] **Step 3: Implementation** — none needed; `shape_from_h5` already exists from Task 1. If a test fails, fix `shape_from_h5` (not the test).
|
|
||||||
|
|
||||||
- [ ] **Step 4: Run to verify it passes**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_metrics_h5.py -q`
|
|
||||||
Expected: PASS (2 tests).
|
|
||||||
|
|
||||||
- [ ] **Step 5: Commit**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git add tests/test_shape_metrics_h5.py
|
|
||||||
git commit -m "test(shape): shape_from_h5 round-trips a real .h5"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 3: Add shape columns to the events schema + migration
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Modify: `sfm/database.py` — `_SCHEMA` CREATE TABLE `events` (after `mic_zc_above_range`); `_migrate` ADD COLUMN loop (the tuple around line 205-218).
|
|
||||||
- Test: `tests/test_shape_columns.py`
|
|
||||||
|
|
||||||
**Interfaces:**
|
|
||||||
- Produces: `events` columns `shape_crest_factor REAL`, `shape_near_peak_count INTEGER`, `shape_sample_count INTEGER`, `shape_axis TEXT`.
|
|
||||||
|
|
||||||
- [ ] **Step 1: Write the failing test**
|
|
||||||
|
|
||||||
```python
|
|
||||||
# tests/test_shape_columns.py
|
|
||||||
import sqlite3
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
|
|
||||||
_SHAPE_COLS = {"shape_crest_factor", "shape_near_peak_count",
|
|
||||||
"shape_sample_count", "shape_axis"}
|
|
||||||
|
|
||||||
def _cols(db):
|
|
||||||
with sqlite3.connect(db.db_path) as c:
|
|
||||||
return {r[1] for r in c.execute("PRAGMA table_info(events)")}
|
|
||||||
|
|
||||||
def test_fresh_db_has_shape_columns(tmp_path):
|
|
||||||
db = SeismoDb(tmp_path / "s.db")
|
|
||||||
assert _SHAPE_COLS <= _cols(db)
|
|
||||||
|
|
||||||
def test_existing_db_migrates_shape_columns(tmp_path):
|
|
||||||
p = tmp_path / "s.db"
|
|
||||||
db = SeismoDb(p)
|
|
||||||
with sqlite3.connect(p) as c: # simulate an older DB missing the columns
|
|
||||||
for col in _SHAPE_COLS:
|
|
||||||
c.execute(f"ALTER TABLE events DROP COLUMN {col}")
|
|
||||||
assert not (_SHAPE_COLS <= _cols(SeismoDb(p))) # sanity: dropped
|
|
||||||
SeismoDb(p) # re-open triggers _migrate
|
|
||||||
assert _SHAPE_COLS <= _cols(SeismoDb(p))
|
|
||||||
```
|
|
||||||
|
|
||||||
> Note: sqlite `DROP COLUMN` needs sqlite ≥ 3.35 (bundled py3.10 has it). If the runner's sqlite lacks it, replace the "simulate older DB" block with building a table without the columns; keep the assertion that re-open adds them.
|
|
||||||
|
|
||||||
- [ ] **Step 2: Run to verify it fails**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_columns.py -q`
|
|
||||||
Expected: FAIL — columns absent.
|
|
||||||
|
|
||||||
- [ ] **Step 3: Implementation**
|
|
||||||
|
|
||||||
In `_SCHEMA`, after the `mic_zc_above_range INTEGER,` line in the `events` CREATE TABLE, add:
|
|
||||||
|
|
||||||
```
|
|
||||||
shape_crest_factor REAL, -- peak / rms of the triggering channel
|
|
||||||
shape_near_peak_count INTEGER, -- samples >= 0.5 * peak (FT: few; real: many)
|
|
||||||
shape_sample_count INTEGER, -- total samples (to normalize near_peak_count)
|
|
||||||
shape_axis TEXT, -- geophone channel measured ("Tran"/"Vert"/"Long")
|
|
||||||
```
|
|
||||||
|
|
||||||
In `_migrate`, extend the ADD COLUMN tuple (the `for col, ddl in (...)` list) with:
|
|
||||||
|
|
||||||
```python
|
|
||||||
("shape_crest_factor", "REAL"),
|
|
||||||
("shape_near_peak_count", "INTEGER"),
|
|
||||||
("shape_sample_count", "INTEGER"),
|
|
||||||
("shape_axis", "TEXT"),
|
|
||||||
```
|
|
||||||
|
|
||||||
- [ ] **Step 4: Run to verify it passes**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_shape_columns.py -q`
|
|
||||||
Expected: PASS.
|
|
||||||
|
|
||||||
- [ ] **Step 5: Commit**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git add sfm/database.py tests/test_shape_columns.py
|
|
||||||
git commit -m "feat(db): shape_* columns on events (+ auto-migrate)"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 4: insert_events persists shape from the waveform record
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Modify: `sfm/database.py` — `insert_events` INSERT (column list + placeholders + values) and the UPSERT `UPDATE` block.
|
|
||||||
- Test: `tests/test_insert_events_shape.py`
|
|
||||||
|
|
||||||
**Interfaces:**
|
|
||||||
- Consumes: a `waveform_records` rec dict that may carry `shape_crest_factor`, `shape_near_peak_count`, `shape_sample_count`, `shape_axis`.
|
|
||||||
- Produces: those four values stored on the row; refreshed on UPSERT.
|
|
||||||
|
|
||||||
- [ ] **Step 1: Write the failing test**
|
|
||||||
|
|
||||||
```python
|
|
||||||
# tests/test_insert_events_shape.py
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
from tests.helpers_events import make_event # existing helper used by other insert tests
|
|
||||||
|
|
||||||
def test_insert_stores_shape_from_record(tmp_path):
|
|
||||||
db = SeismoDb(tmp_path / "s.db")
|
|
||||||
ev = make_event(serial="BE1", key="0111abcd")
|
|
||||||
rec = {ev._waveform_key.hex(): {
|
|
||||||
"filename": "F.CE0W", "filesize": 10,
|
|
||||||
"shape_crest_factor": 34.0, "shape_near_peak_count": 3,
|
|
||||||
"shape_sample_count": 1024, "shape_axis": "Long"}}
|
|
||||||
db.insert_events([ev], serial="BE1", waveform_records=rec)
|
|
||||||
row = db.query_events(serial="BE1")[0]
|
|
||||||
assert row["shape_crest_factor"] == 34.0
|
|
||||||
assert row["shape_near_peak_count"] == 3
|
|
||||||
assert row["shape_axis"] == "Long"
|
|
||||||
```
|
|
||||||
|
|
||||||
> If `tests/helpers_events.make_event` doesn't exist, build the `Event` inline the way `tests/test_zc_freq_columns.py` does (copy its event-construction helper). Keep the assertion identical.
|
|
||||||
|
|
||||||
- [ ] **Step 2: Run to verify it fails**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_insert_events_shape.py -q`
|
|
||||||
Expected: FAIL — `KeyError`/`sqlite3.OperationalError` (columns not in INSERT) or values are NULL.
|
|
||||||
|
|
||||||
- [ ] **Step 3: Implementation**
|
|
||||||
|
|
||||||
In `insert_events` INSERT: add `shape_crest_factor, shape_near_peak_count, shape_sample_count, shape_axis` to the column list, add four `?` placeholders, and add these to the VALUES tuple (after the `mic_zc_above_range` value):
|
|
||||||
|
|
||||||
```python
|
|
||||||
rec.get("shape_crest_factor"),
|
|
||||||
rec.get("shape_near_peak_count"),
|
|
||||||
rec.get("shape_sample_count"),
|
|
||||||
rec.get("shape_axis"),
|
|
||||||
```
|
|
||||||
|
|
||||||
In the UPSERT `UPDATE ... SET`: add
|
|
||||||
|
|
||||||
```sql
|
|
||||||
shape_crest_factor = COALESCE(?, shape_crest_factor),
|
|
||||||
shape_near_peak_count = COALESCE(?, shape_near_peak_count),
|
|
||||||
shape_sample_count = COALESCE(?, shape_sample_count),
|
|
||||||
shape_axis = COALESCE(?, shape_axis),
|
|
||||||
```
|
|
||||||
|
|
||||||
and the matching params (before `serial, ts`):
|
|
||||||
|
|
||||||
```python
|
|
||||||
rec.get("shape_crest_factor") if rec else None,
|
|
||||||
rec.get("shape_near_peak_count") if rec else None,
|
|
||||||
rec.get("shape_sample_count") if rec else None,
|
|
||||||
rec.get("shape_axis") if rec else None,
|
|
||||||
```
|
|
||||||
|
|
||||||
(`COALESCE` on UPSERT so a re-import that lacks samples doesn't wipe a previously-computed shape.)
|
|
||||||
|
|
||||||
- [ ] **Step 4: Run to verify it passes**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_insert_events_shape.py -q`
|
|
||||||
Expected: PASS.
|
|
||||||
|
|
||||||
- [ ] **Step 5: Commit**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git add sfm/database.py tests/test_insert_events_shape.py
|
|
||||||
git commit -m "feat(db): insert_events persists shape_* from waveform record"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 5: Populate shape at ingest (the three save paths)
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Modify: `sfm/waveform_store.py` — in `save`, `save_imported_bw`, `save_imported_idf`, after the `.h5` is written, add its shape to the returned `rec` dict.
|
|
||||||
- Test: `tests/test_save_shape.py`
|
|
||||||
|
|
||||||
**Interfaces:**
|
|
||||||
- Consumes: `sfm.shape_metrics.shape_from_h5`.
|
|
||||||
- Produces: `save*` return dicts carry `shape_crest_factor / shape_near_peak_count / shape_sample_count / shape_axis` (present only when the `.h5` had usable samples).
|
|
||||||
|
|
||||||
- [ ] **Step 1: Write the failing test** (drives the BW-import path, which the existing suite already exercises)
|
|
||||||
|
|
||||||
```python
|
|
||||||
# tests/test_save_shape.py
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
from tests.helpers_bw import sample_bw_bytes, sample_serial # reuse existing import-test fixtures
|
|
||||||
|
|
||||||
def test_save_imported_bw_attaches_shape(tmp_path):
|
|
||||||
store = WaveformStore(tmp_path / "waveforms")
|
|
||||||
ev, rec = store.save_imported_bw(sample_bw_bytes(), serial=sample_serial())
|
|
||||||
# A real BW waveform → shape present with a geo axis.
|
|
||||||
assert rec.get("shape_axis") in ("Tran", "Vert", "Long")
|
|
||||||
assert rec["shape_crest_factor"] > 0
|
|
||||||
assert rec["shape_sample_count"] > 200
|
|
||||||
```
|
|
||||||
|
|
||||||
> Reuse whatever fixture `tests/test_save_imported_bw*.py` already uses for BW bytes; match its import. If the existing BW fixture produces a degenerate/short waveform, use the fixture from the test that asserts a full h5.
|
|
||||||
|
|
||||||
- [ ] **Step 2: Run to verify it fails**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_save_shape.py -q`
|
|
||||||
Expected: FAIL — `rec` has no `shape_*` keys.
|
|
||||||
|
|
||||||
- [ ] **Step 3: Implementation**
|
|
||||||
|
|
||||||
Add a helper near the top of `WaveformStore` methods (module-level import `from sfm.shape_metrics import shape_from_h5`). In each of `save`, `save_imported_bw`, `save_imported_idf`, immediately before building/returning the `rec` dict — and only when the `.h5` was written (i.e. `hdf5_filename`/`hdf5_path` is set) — compute and merge:
|
|
||||||
|
|
||||||
```python
|
|
||||||
shape = shape_from_h5(hdf5_path) if hdf5_filename else None
|
|
||||||
# ... in the returned rec dict literal, add:
|
|
||||||
# **(shape and {
|
|
||||||
# "shape_crest_factor": shape["crest_factor"],
|
|
||||||
# "shape_near_peak_count": shape["near_peak_count"],
|
|
||||||
# "shape_sample_count": shape["sample_count"],
|
|
||||||
# "shape_axis": shape["axis"],
|
|
||||||
# } or {}),
|
|
||||||
```
|
|
||||||
|
|
||||||
Concretely, after each method computes `hdf5_filename`, add before its `return {...}`:
|
|
||||||
|
|
||||||
```python
|
|
||||||
_shape = shape_from_h5(hdf5_path) if hdf5_filename else None
|
|
||||||
_shape_rec = {
|
|
||||||
"shape_crest_factor": _shape["crest_factor"],
|
|
||||||
"shape_near_peak_count": _shape["near_peak_count"],
|
|
||||||
"shape_sample_count": _shape["sample_count"],
|
|
||||||
"shape_axis": _shape["axis"],
|
|
||||||
} if _shape else {}
|
|
||||||
```
|
|
||||||
|
|
||||||
and spread `**_shape_rec` into the returned dict. (`save_imported_bw`/`save_imported_idf` use their own hdf5 path variable names — use whichever local holds the written `.h5` path in each method.)
|
|
||||||
|
|
||||||
- [ ] **Step 4: Run to verify it passes**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_save_shape.py -q`
|
|
||||||
Expected: PASS.
|
|
||||||
|
|
||||||
- [ ] **Step 5: Commit**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git add sfm/waveform_store.py tests/test_save_shape.py
|
|
||||||
git commit -m "feat(ingest): compute shape from the written .h5 in all save paths"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 6: Backfill script for existing events
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Create: `scripts/backfill_event_shape.py` (mirror `scripts/backfill_event_zc_freq.py`, but read the `.h5` for samples instead of the sidecar).
|
|
||||||
- Test: `tests/test_backfill_event_shape.py`
|
|
||||||
|
|
||||||
**Interfaces:**
|
|
||||||
- Produces: `backfill_shape(db, store, *, dry_run=False) -> dict` with counts `{"updated","skipped_no_h5","skipped_no_samples"}`; `main(argv)` CLI mirroring the zc-freq script's args (`--db-path`, `--store-root`, `--dry-run`).
|
|
||||||
|
|
||||||
- [ ] **Step 1: Write the failing test**
|
|
||||||
|
|
||||||
```python
|
|
||||||
# tests/test_backfill_event_shape.py
|
|
||||||
import numpy as np, h5py
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
from scripts.backfill_event_shape import backfill_shape
|
|
||||||
from tests.helpers_events import make_event # or inline as in test_zc_freq_columns
|
|
||||||
|
|
||||||
def _h5(path, long):
|
|
||||||
with h5py.File(path, "w") as f:
|
|
||||||
g = f.create_group("samples")
|
|
||||||
for k in ("Tran", "Vert"): g.create_dataset(k, data=np.zeros(1024, "float32"))
|
|
||||||
g.create_dataset("Long", data=np.asarray(long, "float32"))
|
|
||||||
|
|
||||||
def test_backfill_updates_shape_and_is_idempotent(tmp_path):
|
|
||||||
db = SeismoDb(tmp_path / "s.db")
|
|
||||||
store = WaveformStore(tmp_path / "waveforms")
|
|
||||||
ev = make_event(serial="BE1", key="0111abcd")
|
|
||||||
db.insert_events([ev], serial="BE1",
|
|
||||||
waveform_records={ev._waveform_key.hex():
|
|
||||||
{"filename": "F.CE0W", "filesize": 10}})
|
|
||||||
# place the .h5 where store.paths_for expects it
|
|
||||||
long = np.zeros(1024); long[100] = 0.48
|
|
||||||
_h5(store.hdf5_path_for("BE1", "F.CE0W"), long)
|
|
||||||
|
|
||||||
c1 = backfill_shape(db, store)
|
|
||||||
assert c1["updated"] == 1
|
|
||||||
row = db.query_events(serial="BE1")[0]
|
|
||||||
assert row["shape_axis"] == "Long" and row["shape_near_peak_count"] <= 3
|
|
||||||
c2 = backfill_shape(db, store) # idempotent: re-run overwrites same values
|
|
||||||
assert db.query_events(serial="BE1")[0]["shape_crest_factor"] == row["shape_crest_factor"]
|
|
||||||
```
|
|
||||||
|
|
||||||
> Match `make_event` / column names to whatever `tests/test_zc_freq_columns.py` uses. `store.hdf5_path_for(serial, filename)` is the existing helper that returns the `.h5` path.
|
|
||||||
|
|
||||||
- [ ] **Step 2: Run to verify it fails**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_backfill_event_shape.py -q`
|
|
||||||
Expected: FAIL — `ModuleNotFoundError: scripts.backfill_event_shape`.
|
|
||||||
|
|
||||||
- [ ] **Step 3: Implementation** (mirror the zc-freq script structure)
|
|
||||||
|
|
||||||
```python
|
|
||||||
#!/usr/bin/env python3
|
|
||||||
"""Backfill events.shape_* from each event's .h5 waveform samples. Idempotent."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import argparse, logging, sys
|
|
||||||
from pathlib import Path
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
from sfm.shape_metrics import shape_from_h5
|
|
||||||
|
|
||||||
log = logging.getLogger("backfill_event_shape")
|
|
||||||
|
|
||||||
def backfill_shape(db: SeismoDb, store: WaveformStore, *, dry_run: bool = False) -> dict:
|
|
||||||
counts = {"updated": 0, "skipped_no_h5": 0, "skipped_no_samples": 0}
|
|
||||||
for row in db.query_events(limit=1_000_000):
|
|
||||||
serial, filename = row.get("serial"), row.get("blastware_filename")
|
|
||||||
if not serial or not filename:
|
|
||||||
counts["skipped_no_h5"] += 1; continue
|
|
||||||
h5_path = store.hdf5_path_for(serial, filename)
|
|
||||||
if not h5_path.exists():
|
|
||||||
counts["skipped_no_h5"] += 1; continue
|
|
||||||
shape = shape_from_h5(h5_path)
|
|
||||||
if shape is None:
|
|
||||||
counts["skipped_no_samples"] += 1; continue
|
|
||||||
if not dry_run:
|
|
||||||
with db._connect() as conn:
|
|
||||||
conn.execute(
|
|
||||||
"UPDATE events SET shape_crest_factor=?, shape_near_peak_count=?, "
|
|
||||||
"shape_sample_count=?, shape_axis=? WHERE id=?",
|
|
||||||
(shape["crest_factor"], shape["near_peak_count"],
|
|
||||||
shape["sample_count"], shape["axis"], row["id"]))
|
|
||||||
counts["updated"] += 1
|
|
||||||
log.info("backfill_shape: %s", counts)
|
|
||||||
return counts
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--db-path", required=True)
|
|
||||||
ap.add_argument("--store-root", required=True)
|
|
||||||
ap.add_argument("--dry-run", action="store_true")
|
|
||||||
a = ap.parse_args(argv)
|
|
||||||
logging.basicConfig(level=logging.INFO)
|
|
||||||
counts = backfill_shape(SeismoDb(a.db_path), WaveformStore(a.store_root), dry_run=a.dry_run)
|
|
||||||
print(counts)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
```
|
|
||||||
|
|
||||||
> `store.hdf5_path_for` and `db._connect` are existing internals used the same way by other scripts. If `hdf5_path_for` isn't public, use `store.paths_for(...)` sibling `.h5` path exactly as `save()` derives `hdf5_path`.
|
|
||||||
|
|
||||||
- [ ] **Step 4: Run to verify it passes**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_backfill_event_shape.py -q`
|
|
||||||
Expected: PASS.
|
|
||||||
|
|
||||||
- [ ] **Step 5: Run the full suite + commit**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest -q
|
|
||||||
git add scripts/backfill_event_shape.py tests/test_backfill_event_shape.py
|
|
||||||
git commit -m "feat(scripts): backfill events.shape_* from .h5 samples"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 7: Confirm /db/events carries shape + version bump
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Test: `tests/test_db_events_exposes_shape.py`
|
|
||||||
- Modify: `pyproject.toml` version; `sfm/server.py` version string; `CHANGELOG.md`.
|
|
||||||
|
|
||||||
**Interfaces:**
|
|
||||||
- Consumes: the running `/db/events` route (already returns `SELECT *`).
|
|
||||||
|
|
||||||
- [ ] **Step 1: Write the failing test** (guards that the feed dict includes the new keys)
|
|
||||||
|
|
||||||
```python
|
|
||||||
# tests/test_db_events_exposes_shape.py
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
def test_query_events_row_includes_shape_keys(tmp_path):
|
|
||||||
db = SeismoDb(tmp_path / "s.db")
|
|
||||||
# query_events returns dict(row); a fresh insert has the keys (values may be None)
|
|
||||||
from tests.helpers_events import make_event
|
|
||||||
db.insert_events([make_event(serial="BE1", key="0111abcd")], serial="BE1")
|
|
||||||
row = db.query_events(serial="BE1")[0]
|
|
||||||
for k in ("shape_crest_factor","shape_near_peak_count","shape_sample_count","shape_axis"):
|
|
||||||
assert k in row
|
|
||||||
```
|
|
||||||
|
|
||||||
- [ ] **Step 2: Run to verify it fails / passes**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest tests/test_db_events_exposes_shape.py -q`
|
|
||||||
Expected: PASS immediately if Task 3 landed (columns present in `SELECT *`). If it fails, the columns weren't added — fix Task 3. (This task is the guard, not new behavior.)
|
|
||||||
|
|
||||||
- [ ] **Step 3: Version bump**
|
|
||||||
|
|
||||||
Bump `pyproject.toml` `version` 0.23.0 → 0.24.0; set `sfm/server.py` `version="0.24.0"`; add a `## v0.24.0` CHANGELOG entry ("waveform-shape metrics on events: crest factor + points-near-peak, ingest + backfill").
|
|
||||||
|
|
||||||
- [ ] **Step 4: Full suite**
|
|
||||||
|
|
||||||
Run: `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest -q`
|
|
||||||
Expected: PASS (no regressions).
|
|
||||||
|
|
||||||
- [ ] **Step 5: Commit**
|
|
||||||
|
|
||||||
```bash
|
|
||||||
git add tests/test_db_events_exposes_shape.py pyproject.toml sfm/server.py CHANGELOG.md
|
|
||||||
git commit -m "chore(release): v0.24.0 — waveform-shape metrics on events"
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Self-Review
|
|
||||||
|
|
||||||
**Spec coverage:** SFM shape columns (Tasks 3–4) ✓; DSP crest + near-peak (Task 1) ✓; ingest population (Task 5) ✓; backfill (Task 6) ✓; `/db/events` exposure (Task 7) ✓; NULL for histogram/no-sample events (Tasks 1/5/6 return None → NULL) ✓; 94%-coverage / series-4 fallback handled by NULL-then-Terra-View-fallback (Phase B) ✓. Terra-View scoring, 3-state review, twin flagging, export Notes column → **Phase B plan** (separate, depends on this feed). Calibration → Phase C.
|
|
||||||
|
|
||||||
**Placeholder scan:** No TBD/TODO; every code step has real code. The two "reuse existing fixture" notes point at concrete existing tests (`test_zc_freq_columns.py`, `test_save_imported_bw*.py`) rather than leaving blanks.
|
|
||||||
|
|
||||||
**Type consistency:** `shape_from_h5`/`shape_from_samples`/`channel_shape` return the same dict keys (`crest_factor`, `near_peak_count`, `sample_count`, `axis`) throughout; the DB columns (`shape_crest_factor`, `shape_near_peak_count`, `shape_sample_count`, `shape_axis`) and rec keys match across Tasks 4–6.
|
|
||||||
|
|
||||||
## Deferred to Phase B (terra-view, separate plan)
|
|
||||||
|
|
||||||
Scoring service combining shape + cheap signals; suspicion column + reason chips; `reviewed_real` mirror + 3-state review; twin-aware flag propagation; Notes-column export + Maximums "(excludes N flagged)". Written once this feed is live so column names/values are real.
|
|
||||||
@@ -1,264 +0,0 @@
|
|||||||
# Phase B2-A (seismo-relay) — reviewed_real + 3-state mirror + twin propagation
|
|
||||||
|
|
||||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
||||||
|
|
||||||
**Goal:** Give SFM a persisted, queryable `reviewed_real` flag alongside `false_trigger` (mirrored from the sidecar review block, mutually exclusive), and propagate a review to an event's histogram/waveform twin so flagging one flags both.
|
|
||||||
|
|
||||||
**Architecture:** Mirror the existing `false_trigger` mechanism. New `events.reviewed_real` column (auto-migrate). `update_event_review` mirrors BOTH flags from the sidecar review block and enforces mutual exclusivity on the columns. A `find_twins` matcher (same serial + identical peak_vector_sum + timestamp within a window) drives `propagate_review_to_twins`, which the sidecar-PATCH endpoint calls after mirroring the primary. Terra-View reads `reviewed_real` from `/db/events` (SELECT *).
|
|
||||||
|
|
||||||
**Tech Stack:** Python 3.10, raw sqlite3, FastAPI, pytest. Runner: `/home/serversdown/seismo-relay/.venv/bin/python3`.
|
|
||||||
|
|
||||||
## Global Constraints
|
|
||||||
|
|
||||||
- 3 review states are **mutually exclusive**: an event is `false_trigger=1` XOR `reviewed_real=1` XOR neither. Setting one true forces the other's column to 0.
|
|
||||||
- The sidecar JSON stays the source of truth for full review state; the `false_trigger`/`reviewed_real` columns are derived indexes (like today). B2-A propagates twins at the **column** level (what the feed/peak/export read); twin sidecars are not rewritten (known limitation — noted).
|
|
||||||
- Twin match = **same serial AND identical `peak_vector_sum` (exact equality) AND `timestamp` within ± window (default 300 s), excluding the event itself.** Identical PVS is the safety anchor.
|
|
||||||
- Known pre-existing test failures (~16, missing gitignored fixtures under `tests/fixtures/`) are unrelated — confirm zero NEW failures, don't try to fix them.
|
|
||||||
- Run tests with `/home/serversdown/seismo-relay/.venv/bin/python3 -m pytest`.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 1: `reviewed_real` column on events
|
|
||||||
|
|
||||||
**Files:** Modify `sfm/database.py` (`_SCHEMA` CREATE TABLE `events` + the Migration-1 rebuild `CREATE TABLE` + the `_migrate` ADD COLUMN loop). Test: `tests/test_reviewed_real_column.py`.
|
|
||||||
|
|
||||||
**Interfaces:** Produces `events.reviewed_real INTEGER NOT NULL DEFAULT 0`.
|
|
||||||
|
|
||||||
- [ ] **Step 1: Failing test**
|
|
||||||
```python
|
|
||||||
# tests/test_reviewed_real_column.py
|
|
||||||
import sqlite3
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
def _cols(db):
|
|
||||||
with sqlite3.connect(db.db_path) as c:
|
|
||||||
return {r[1] for r in c.execute("PRAGMA table_info(events)")}
|
|
||||||
def test_fresh_db_has_reviewed_real(tmp_path):
|
|
||||||
assert "reviewed_real" in _cols(SeismoDb(tmp_path/"s.db"))
|
|
||||||
def test_existing_db_migrates_reviewed_real(tmp_path):
|
|
||||||
p = tmp_path/"s.db"; db = SeismoDb(p)
|
|
||||||
with sqlite3.connect(p) as c:
|
|
||||||
c.execute("ALTER TABLE events DROP COLUMN reviewed_real")
|
|
||||||
assert "reviewed_real" not in _cols(db) # dropped (read via existing handle/connection)
|
|
||||||
SeismoDb(p) # re-open migrates
|
|
||||||
assert "reviewed_real" in _cols(SeismoDb(p))
|
|
||||||
```
|
|
||||||
> If sqlite < 3.35 lacks DROP COLUMN, fall back to building a table without the column and asserting re-open adds it (same as the shape-columns test did).
|
|
||||||
|
|
||||||
- [ ] **Step 2: Run → FAIL** (`pytest tests/test_reviewed_real_column.py -q`).
|
|
||||||
- [ ] **Step 3: Implement**
|
|
||||||
- In `_SCHEMA` `events` CREATE TABLE, after the `false_trigger ... DEFAULT 0,` line add: ` reviewed_real INTEGER NOT NULL DEFAULT 0, -- 0=no, 1=operator-confirmed real (mutually exclusive with false_trigger)`
|
|
||||||
- In the `_migrate` ADD COLUMN loop tuple add: `("reviewed_real", "INTEGER NOT NULL DEFAULT 0"),`
|
|
||||||
- **Do NOT** add it to the Migration-1 rebuild `CREATE TABLE events (...)` block — that block uses a positional `INSERT ... SELECT * FROM events_old` and, by convention, contains only the columns that existed when Migration 1 was written; every later column is added by the ADD COLUMN loop only. Adding it there crashes `_migrate` on genuinely legacy (pre-Migration-1) DBs.
|
|
||||||
- [ ] **Step 4: Run → PASS.**
|
|
||||||
- [ ] **Step 5: Commit** `feat(db): reviewed_real column on events (+ auto-migrate)`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 2: `update_event_review` mirrors both flags + mutual exclusivity
|
|
||||||
|
|
||||||
**Files:** Modify `sfm/database.py` `update_event_review`. Test: `tests/test_update_event_review_reviewed_real.py`.
|
|
||||||
|
|
||||||
**Interfaces:** Consumes a `review` dict that may carry `false_trigger` and/or `reviewed_real` (bools). Produces mutually-exclusive column state.
|
|
||||||
|
|
||||||
- [ ] **Step 1: Failing test**
|
|
||||||
```python
|
|
||||||
# tests/test_update_event_review_reviewed_real.py
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
from minimateplus.models import Event
|
|
||||||
def _ins(db, eid_key="01110000", serial="BE1"):
|
|
||||||
ev = Event(index=0); ev._waveform_key = bytes.fromhex(eid_key)
|
|
||||||
db.insert_events([ev], serial=serial)
|
|
||||||
return db.query_events(serial=serial)[0]["id"]
|
|
||||||
def test_confirm_real_sets_and_clears_ft(tmp_path):
|
|
||||||
db = SeismoDb(tmp_path/"s.db"); eid = _ins(db)
|
|
||||||
db.update_event_review(eid, {"false_trigger": True})
|
|
||||||
assert db.get_event(eid)["false_trigger"] == 1
|
|
||||||
db.update_event_review(eid, {"reviewed_real": True}) # confirming real clears FT
|
|
||||||
row = db.get_event(eid)
|
|
||||||
assert row["reviewed_real"] == 1 and row["false_trigger"] == 0
|
|
||||||
def test_flag_ft_clears_reviewed_real(tmp_path):
|
|
||||||
db = SeismoDb(tmp_path/"s.db"); eid = _ins(db)
|
|
||||||
db.update_event_review(eid, {"reviewed_real": True})
|
|
||||||
db.update_event_review(eid, {"false_trigger": True})
|
|
||||||
row = db.get_event(eid)
|
|
||||||
assert row["false_trigger"] == 1 and row["reviewed_real"] == 0
|
|
||||||
```
|
|
||||||
> Build the Event inline like `tests/test_zc_freq_columns.py` if the import differs.
|
|
||||||
|
|
||||||
- [ ] **Step 2: Run → FAIL.**
|
|
||||||
- [ ] **Step 3: Implement** — replace the body of `update_event_review` so it handles both keys:
|
|
||||||
```python
|
|
||||||
if not isinstance(review, dict):
|
|
||||||
return False
|
|
||||||
has_ft = "false_trigger" in review
|
|
||||||
has_real = "reviewed_real" in review
|
|
||||||
if not has_ft and not has_real:
|
|
||||||
with self._connect() as conn:
|
|
||||||
row = conn.execute("SELECT 1 FROM events WHERE id=?", (event_id,)).fetchone()
|
|
||||||
return row is not None
|
|
||||||
sets = {}
|
|
||||||
if has_ft:
|
|
||||||
sets["false_trigger"] = 1 if review.get("false_trigger") else 0
|
|
||||||
if has_real:
|
|
||||||
sets["reviewed_real"] = 1 if review.get("reviewed_real") else 0
|
|
||||||
# mutual exclusivity: a true in one forces the other column to 0
|
|
||||||
if sets.get("false_trigger") == 1:
|
|
||||||
sets["reviewed_real"] = 0
|
|
||||||
if sets.get("reviewed_real") == 1:
|
|
||||||
sets["false_trigger"] = 0
|
|
||||||
assign = ", ".join(f"{k}=?" for k in sets)
|
|
||||||
params = list(sets.values()) + [event_id]
|
|
||||||
with self._connect() as conn:
|
|
||||||
cur = conn.execute(f"UPDATE events SET {assign} WHERE id=?", params)
|
|
||||||
return cur.rowcount > 0
|
|
||||||
```
|
|
||||||
- [ ] **Step 4: Run → PASS** (+ run `tests/test_*false_trigger*`/existing review tests to confirm no regression).
|
|
||||||
- [ ] **Step 5: Commit** `feat(db): update_event_review mirrors reviewed_real + enforces 3-state exclusivity`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 3: `find_twins` matcher
|
|
||||||
|
|
||||||
**Files:** Modify `sfm/database.py` (add `find_twins`). Test: `tests/test_find_twins.py`.
|
|
||||||
|
|
||||||
**Interfaces:** Produces `find_twins(event_id, *, window_seconds=300) -> list[dict]` — same serial, identical peak_vector_sum, timestamp within ±window, excluding self.
|
|
||||||
|
|
||||||
- [ ] **Step 1: Failing test**
|
|
||||||
```python
|
|
||||||
# tests/test_find_twins.py
|
|
||||||
import datetime
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
from minimateplus.models import Event, Timestamp
|
|
||||||
def _ins(db, key, serial, pvs, ts):
|
|
||||||
ev = Event(index=0); ev._waveform_key = bytes.fromhex(key)
|
|
||||||
ev.timestamp = ts
|
|
||||||
# peak_vector_sum comes from peak_values; simplest: insert then UPDATE pvs directly
|
|
||||||
db.insert_events([ev], serial=serial)
|
|
||||||
row = [r for r in db.query_events(serial=serial) if r["waveform_key"] == key][0]
|
|
||||||
import sqlite3
|
|
||||||
with sqlite3.connect(db.db_path) as c:
|
|
||||||
c.execute("UPDATE events SET peak_vector_sum=? WHERE id=?", (pvs, row["id"]))
|
|
||||||
return row["id"]
|
|
||||||
def test_find_twins_matches_same_serial_pvs_near_time(tmp_path):
|
|
||||||
db = SeismoDb(tmp_path/"s.db")
|
|
||||||
base = Timestamp(raw=b"", flag=0x10, year=2026, unknown_byte=0, month=2, day=25, hour=20, minute=19, second=5)
|
|
||||||
twin = Timestamp(raw=b"", flag=0x10, year=2026, unknown_byte=0, month=2, day=25, hour=20, minute=19, second=45)
|
|
||||||
far = Timestamp(raw=b"", flag=0x10, year=2026, unknown_byte=0, month=2, day=25, hour=21, minute=0, second=0)
|
|
||||||
a = _ins(db, "01110001", "BE1", 0.4763, base)
|
|
||||||
b = _ins(db, "01110002", "BE1", 0.4763, twin) # twin: same serial+pvs, 40s apart
|
|
||||||
c = _ins(db, "01110003", "BE1", 0.4763, far) # same pvs but >window away
|
|
||||||
d = _ins(db, "01110004", "BE1", 0.9999, twin) # near time but different pvs
|
|
||||||
ids = {r["id"] for r in db.find_twins(a, window_seconds=300)}
|
|
||||||
assert ids == {b}
|
|
||||||
```
|
|
||||||
> Adjust the Event/Timestamp construction to match how `tests/test_waveform_store.py::_make_synthetic_event` builds them if fields differ.
|
|
||||||
|
|
||||||
- [ ] **Step 2: Run → FAIL.**
|
|
||||||
- [ ] **Step 3: Implement**
|
|
||||||
```python
|
|
||||||
def find_twins(self, event_id: str, *, window_seconds: int = 300) -> list[dict]:
|
|
||||||
row = self.get_event(event_id)
|
|
||||||
if not row:
|
|
||||||
return []
|
|
||||||
serial = row.get("serial"); pvs = row.get("peak_vector_sum"); ts = row.get("timestamp")
|
|
||||||
if serial is None or pvs is None or not ts:
|
|
||||||
return []
|
|
||||||
try:
|
|
||||||
t = datetime.datetime.fromisoformat(ts.replace(" ", "T"))
|
|
||||||
except ValueError:
|
|
||||||
return []
|
|
||||||
lo = (t - datetime.timedelta(seconds=window_seconds)).isoformat()
|
|
||||||
hi = (t + datetime.timedelta(seconds=window_seconds)).isoformat()
|
|
||||||
with self._connect() as conn:
|
|
||||||
rows = conn.execute(
|
|
||||||
"SELECT * FROM events WHERE serial=? AND id!=? AND peak_vector_sum=? "
|
|
||||||
"AND timestamp BETWEEN ? AND ?",
|
|
||||||
(serial, event_id, pvs, lo, hi),
|
|
||||||
).fetchall()
|
|
||||||
return [dict(r) for r in rows]
|
|
||||||
```
|
|
||||||
- [ ] **Step 4: Run → PASS.**
|
|
||||||
- [ ] **Step 5: Commit** `feat(db): find_twins (serial + identical PVS + time window)`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 4: propagate a review to twins + wire into the sidecar-PATCH endpoint
|
|
||||||
|
|
||||||
**Files:** Modify `sfm/database.py` (add `propagate_review_to_twins`); `sfm/server.py` (`db_event_sidecar_patch`). Test: `tests/test_twin_propagation.py`.
|
|
||||||
|
|
||||||
**Interfaces:** `propagate_review_to_twins(event_id, *, window_seconds=300) -> list[str]` copies the event's `false_trigger`/`reviewed_real` columns onto each twin; returns twin ids.
|
|
||||||
|
|
||||||
- [ ] **Step 1: Failing test** (DB-level)
|
|
||||||
```python
|
|
||||||
# tests/test_twin_propagation.py — reuse the _ins helper pattern from test_find_twins
|
|
||||||
def test_propagate_copies_flags_to_twins(tmp_path):
|
|
||||||
db = SeismoDb(tmp_path/"s.db")
|
|
||||||
# (build primary + twin via the _ins helper as in test_find_twins)
|
|
||||||
# flag the primary FT, then propagate:
|
|
||||||
db.update_event_review(primary_id, {"false_trigger": True})
|
|
||||||
moved = db.propagate_review_to_twins(primary_id)
|
|
||||||
assert twin_id in moved
|
|
||||||
assert db.get_event(twin_id)["false_trigger"] == 1
|
|
||||||
```
|
|
||||||
- [ ] **Step 2: Run → FAIL.**
|
|
||||||
- [ ] **Step 3: Implement**
|
|
||||||
- `sfm/database.py`:
|
|
||||||
```python
|
|
||||||
def propagate_review_to_twins(self, event_id: str, *, window_seconds: int = 300) -> list[str]:
|
|
||||||
row = self.get_event(event_id)
|
|
||||||
if not row:
|
|
||||||
return []
|
|
||||||
ft = 1 if row.get("false_trigger") else 0
|
|
||||||
real = 1 if row.get("reviewed_real") else 0
|
|
||||||
twins = self.find_twins(event_id, window_seconds=window_seconds)
|
|
||||||
moved = []
|
|
||||||
with self._connect() as conn:
|
|
||||||
for tw in twins:
|
|
||||||
conn.execute("UPDATE events SET false_trigger=?, reviewed_real=? WHERE id=?",
|
|
||||||
(ft, real, tw["id"]))
|
|
||||||
moved.append(tw["id"])
|
|
||||||
return moved
|
|
||||||
```
|
|
||||||
- `sfm/server.py` `db_event_sidecar_patch`: after the existing `_get_db().update_event_review(event_id, new_sidecar.get("review", {}))`, add:
|
|
||||||
```python
|
|
||||||
# Propagate the review to the event's histogram/waveform twin(s) so
|
|
||||||
# flagging one flags both (column-level; twins share serial+PVS+near time).
|
|
||||||
try:
|
|
||||||
_get_db().propagate_review_to_twins(event_id)
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("twin review-propagation failed for %s: %s", event_id, exc)
|
|
||||||
```
|
|
||||||
(Guard the `if body.review is not None:` block so propagation only runs when review changed.)
|
|
||||||
- [ ] **Step 4: Run → PASS.**
|
|
||||||
- [ ] **Step 5: Commit** `feat(review): propagate false_trigger/reviewed_real to twins on sidecar PATCH`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Task 5: expose in feed guard + version bump
|
|
||||||
|
|
||||||
**Files:** Test `tests/test_reviewed_real_in_feed.py`; `pyproject.toml`, `sfm/server.py` version, `CHANGELOG.md`.
|
|
||||||
|
|
||||||
- [ ] **Step 1: Guard test** — a `query_events` row dict includes `reviewed_real` (SELECT * returns it).
|
|
||||||
```python
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
from minimateplus.models import Event
|
|
||||||
def test_query_events_includes_reviewed_real(tmp_path):
|
|
||||||
db = SeismoDb(tmp_path/"s.db")
|
|
||||||
ev = Event(index=0); ev._waveform_key = bytes.fromhex("01110000")
|
|
||||||
db.insert_events([ev], serial="BE1")
|
|
||||||
assert "reviewed_real" in db.query_events(serial="BE1")[0]
|
|
||||||
```
|
|
||||||
- [ ] **Step 2: Run → PASS** (columns already present from Task 1).
|
|
||||||
- [ ] **Step 3: Bump** `pyproject.toml` 0.24.0 → 0.25.0; `sfm/server.py` version="0.25.0"; add `## v0.25.0` CHANGELOG entry ("reviewed_real 3-state review flag + histogram/waveform twin review-propagation").
|
|
||||||
- [ ] **Step 4: Full suite** — confirm zero NEW failures beyond the ~16 pre-existing.
|
|
||||||
- [ ] **Step 5: Commit** `chore(release): v0.25.0 — reviewed_real + twin review-propagation`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Self-Review
|
|
||||||
|
|
||||||
**Spec coverage:** reviewed_real column (T1) ✓; mirror + mutual exclusivity (T2) ✓; twin match serial+identical-PVS+window (T3) ✓; twin propagation wired into the review path (T4) ✓; feed exposure + version (T5) ✓. Twin **sidecar** rewrite intentionally deferred (column-level only — documented in Global Constraints); this is the SFM half — the 3-state UI + confirm-real PATCH is the separate **B2-B (terra-view)** plan.
|
|
||||||
|
|
||||||
**Placeholder scan:** none — every step has real code (the two "adjust Event construction to match test_waveform_store" notes point at a concrete existing helper).
|
|
||||||
|
|
||||||
**Type consistency:** `false_trigger`/`reviewed_real` INTEGER columns used identically across T1/T2/T4; `find_twins`→`propagate_review_to_twins` both key on the same match; server calls the DB methods by the exact names T2–T4 define.
|
|
||||||
@@ -1,350 +0,0 @@
|
|||||||
> ## SUPERSEDED 2026-08-25 — the body is a RECORD CHAIN
|
|
||||||
>
|
|
||||||
> The tag-dispatch model described in this document — `40 NN` segment headers,
|
|
||||||
> tagless headers, channel rotation — is **wrong**. It produced nearly-correct
|
|
||||||
> output only because the block table happens to tile the data sections.
|
|
||||||
>
|
|
||||||
> The body is a chain of self-delimiting per-channel records: `off+2` is a
|
|
||||||
> uint16 BE length, `next = off + 2 + len`, and the chain ends on a record whose
|
|
||||||
> chan_id is `0x06`. A 3-valued mode enum at `off+8` selects delta / absolute /
|
|
||||||
> raw-12-bit semantics. `40 NN` is an ordinary int16 BE data block.
|
|
||||||
>
|
|
||||||
> See the record-chain section of `docs/instantel_protocol_reference.md` §7.6.1
|
|
||||||
> and the implementation in `minimateplus/waveform_codec.py`.
|
|
||||||
>
|
|
||||||
> Result: all four channels equal length in 1388/1388 files (was 156/1388);
|
|
||||||
> ASCII sample-count exact 75/75, fully exact 73/75; device PPV 1306/1306.
|
|
||||||
>
|
|
||||||
> This document is retained as the reasoning trail.
|
|
||||||
|
|
||||||
# Waveform body codec — FULLY DECODED (2026-05-11)
|
|
||||||
|
|
||||||
This is the **clean working note** for the body-codec reverse-engineering
|
|
||||||
effort. It supersedes scattered claims elsewhere when they conflict.
|
|
||||||
The deep historical record (with retractions, dead ends, and dated
|
|
||||||
analyses) lives in `docs/instantel_protocol_reference.md §7.6.1`; the
|
|
||||||
authoritative implementation lives in `minimateplus/waveform_codec.py`.
|
|
||||||
|
|
||||||
## TL;DR
|
|
||||||
|
|
||||||
**The codec is fully decoded.** Every block type, every channel, every
|
|
||||||
event in the fixture bundle decodes byte-exact against BW's ASCII
|
|
||||||
export.
|
|
||||||
|
|
||||||
| Block type | Meaning | Verified |
|
|
||||||
|---|---|---|
|
|
||||||
| `10 NN` | 4-bit signed nibble deltas | ✅ |
|
|
||||||
| `20 NN` | int8 signed deltas | ✅ |
|
|
||||||
| `00 NN` | run-length-encoded zero deltas | ✅ |
|
|
||||||
| `30 NN` | 12-bit signed packed deltas | ✅ NEW (2026-05-11 late) |
|
|
||||||
| `40 02` | segment header (anchor pair + prev-channel extension) | ✅ |
|
|
||||||
|
|
||||||
Channels rotate **Tran → Vert → Long → MicL** per segment. Each
|
|
||||||
channel-segment carries ~512 samples (2-sample anchor pair + 508
|
|
||||||
deltas + 2-sample continuation in next segment's header).
|
|
||||||
|
|
||||||
## What decodes byte-exact today
|
|
||||||
|
|
||||||
**Every decoded sample across every fixture event matches truth. Zero
|
|
||||||
divergences.**
|
|
||||||
|
|
||||||
| Event | Description | Tran | Vert | Long | Total |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| event-a (5-8) | quiet, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
|
|
||||||
| event-c (5-8) | quiet, 1 sec | 1280 ✓ | 1280 ✓ | 1280 ✓ | 3840 |
|
|
||||||
| event-d (5-8) | quiet, 1 sec | 1280 ✓ | 1280 ✓ | 1280 ✓ | 3840 |
|
|
||||||
| JQ0 (5-11) | Vert-heavy, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
|
|
||||||
| V70 (5-11) | Mic-heavy, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
|
|
||||||
| SP0 (5-11) | loud all, 3 sec | 2048 ✓ | 1538 ✓ | 1536 ✓ | 5122 |
|
|
||||||
| SS0 (5-11) | loud-from-start | 734 ✓ | 512 ✓ | 512 ✓ | 1758 |
|
|
||||||
| SV0 (5-11) | loud-from-start | 1024 ✓ | 578 ✓ | 512 ✓ | 2114 |
|
|
||||||
| event-b (5-8) | quiet, 2 sec | 512 ✓ | 226 ✓ | 0 | 738 |
|
|
||||||
|
|
||||||
That's **47,364 ADC samples decoded byte-exact, zero errors.**
|
|
||||||
|
|
||||||
Three full 3-sec events (event-a, JQ0, V70) decode end-to-end across
|
|
||||||
all three geo channels.
|
|
||||||
|
|
||||||
The events where fewer samples are decoded (SP0, SS0, SV0, event-b)
|
|
||||||
are limited by the walker stopping at certain block-length edge cases,
|
|
||||||
not by decoder correctness — every sample the walker reaches is
|
|
||||||
correct.
|
|
||||||
|
|
||||||
## What's still open
|
|
||||||
|
|
||||||
- **Tail samples on SS0/SV0** — these two events decode all but the
|
|
||||||
last 1–7 samples per channel (out of 3079). Likely the same
|
|
||||||
"last segment is truncated" pattern. Minor; doesn't affect the
|
|
||||||
bulk of the data.
|
|
||||||
|
|
||||||
## Sample counts (72,972 byte-exact total)
|
|
||||||
|
|
||||||
| Event | Tran | Vert | Long | Status |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| event-a | 3328 | 3328 | 3328 | full |
|
|
||||||
| event-b | 2304 | 2304 | 2304 | full |
|
|
||||||
| event-c | 1280 | 1280 | 1280 | full |
|
|
||||||
| event-d | 1280 | 1280 | 1280 | full |
|
|
||||||
| JQ0 | 3328 | 3328 | 3328 | full |
|
|
||||||
| V70 | 3328 | 3328 | 3328 | full |
|
|
||||||
| SP0 | 3328 | 3328 | 3328 | full |
|
|
||||||
| SS0 | 3078 | 3072 | 3072 | minus 1–7 tail samples |
|
|
||||||
| SV0 | 3078 | 3072 | 3072 | minus 1–7 tail samples |
|
|
||||||
|
|
||||||
## What's now wired into production (2026-05-11 late)
|
|
||||||
|
|
||||||
- **`client.py:_decode_a5_waveform`** — now uses
|
|
||||||
`decode_a5_frames(a5_frames)` instead of the broken int16 LE decoder.
|
|
||||||
`event.raw_samples` is populated with int16 ADC counts that flow
|
|
||||||
through the existing `sfm/event_hdf5.py` scaling pipeline unchanged.
|
|
||||||
Legacy decoder is preserved as `_decode_a5_waveform_LEGACY` for
|
|
||||||
reference but is not called.
|
|
||||||
|
|
||||||
- **MicL → dB(L) conversion** — exposed as
|
|
||||||
`waveform_codec.mic_count_to_db(count)`. Verified against BW
|
|
||||||
display values (count=1 → 81.94 dB; count=813 → 140.14 dB; matches
|
|
||||||
the V70 mic-heavy fixture exactly).
|
|
||||||
|
|
||||||
- **`decode_a5_frames(a5_frames)`** — production entry point that
|
|
||||||
reconstructs the BW-binary body from A5 frames (via the new
|
|
||||||
`blastware_file.extract_body_bytes` helper) and runs the verified
|
|
||||||
codec. Returns the same `raw_samples` dict shape the consumers
|
|
||||||
already expect.
|
|
||||||
|
|
||||||
## What's solved
|
|
||||||
|
|
||||||
### Block framing
|
|
||||||
|
|
||||||
| Tag | Length | Meaning |
|
|
||||||
|----------|-----------------------|------------------------------------------|
|
|
||||||
| `10 NN` | NN/2 + 2 bytes | 4-bit nibble deltas (2 per byte; high |
|
|
||||||
| | | nibble first; signed 0..7 / 8..F = -8..-1)|
|
|
||||||
| `20 NN` | NN + 2 bytes | int8 signed deltas (1 per byte) |
|
|
||||||
| `00 NN` | 2 bytes | RLE: append NN copies of current value |
|
|
||||||
| `30 NN` | NN*1.5 + 2 in data | 12-bit signed deltas (see below). |
|
|
||||||
| | section, NN*4 trailer | |
|
|
||||||
| `40 NN` | 2*NN + 16 bytes | Segment header (NN = prev-channel deltas)|
|
|
||||||
|
|
||||||
NN is always a multiple of 4.
|
|
||||||
|
|
||||||
**Wide-NN forms.** `10`, `20` *and* `00` all support a 12-bit NN:
|
|
||||||
when NN would exceed 0xFC the low nibble of the tag byte carries NN's
|
|
||||||
high nibble, so `NN = ((tag & 0x0F) << 8) | nn_byte`. Confirmed for
|
|
||||||
`1X`/`2X` in 2026-05-11 and for `0X` (RLE) in 2026-08-25 — e.g.
|
|
||||||
`01 0c` = a 268-sample zero-delta run.
|
|
||||||
|
|
||||||
**`40 NN` is variable width.** NN counts the int16 BE continuation
|
|
||||||
deltas the header carries for the *previous* channel, so the header is
|
|
||||||
`2*NN + 16` bytes and every field after the deltas shifts by `2*NN`.
|
|
||||||
`40 01` (18 B) and `40 03` (22 B) both occur alongside the common
|
|
||||||
`40 02` (20 B). Confirmed 2026-08-25.
|
|
||||||
|
|
||||||
Implementation: `walk_body()` in `minimateplus/waveform_codec.py`.
|
|
||||||
|
|
||||||
### 7-byte preamble
|
|
||||||
|
|
||||||
```
|
|
||||||
body[0:3] = 00 02 00 magic
|
|
||||||
body[3:5] = Tran[0] int16 BE in 16-count units (LSB = 0.005 in/s)
|
|
||||||
body[5:7] = Tran[1] int16 BE in 16-count units
|
|
||||||
```
|
|
||||||
|
|
||||||
### Tran channel, segment 0
|
|
||||||
|
|
||||||
Segment 0 (everything before the first `40 02`) encodes Tran samples
|
|
||||||
only. Starting from preamble anchors Tran[0] and Tran[1], each block
|
|
||||||
contributes to a running cumulative:
|
|
||||||
|
|
||||||
- `10 NN` → append NN nibble-deltas
|
|
||||||
- `20 NN` → append NN int8-deltas
|
|
||||||
- `00 NN` → append NN copies of current value (RLE)
|
|
||||||
- `40 02` → end segment 0
|
|
||||||
|
|
||||||
Verified byte-exact:
|
|
||||||
|
|
||||||
| Event | Description | Segment 0 size | Match |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `M529LL1A.SP0` | Loud, 0.25 s pretrig | 510 | 510/510 ✓ |
|
|
||||||
| `M529LL1A.SV0` | Loud from sample 0 | 58 | 58/58 ✓ (stops at first `30 NN`) |
|
|
||||||
| `M529LL1A.SS0` | Loud from sample 0 | 42 | 42/42 ✓ (stops at first `30 04`) |
|
|
||||||
| `M529LL1L.JQ0` | Vert-heavy | 510 | 510/510 ✓ |
|
|
||||||
| `M529LL1L.V70` | Mic-heavy (140 dB) | 510 | 510/510 ✓ |
|
|
||||||
|
|
||||||
Implementation: `decode_tran_initial()`.
|
|
||||||
|
|
||||||
### Segment header (`40 02`, 20 bytes total) — REWRITTEN 2026-05-11
|
|
||||||
|
|
||||||
| Payload offset | Field | Status |
|
|
||||||
|---|---|---|
|
|
||||||
| [0:2] | Previous-channel delta — 1st extension sample (int16 BE) | ✅ confirmed |
|
|
||||||
| [2:4] | Previous-channel delta — 2nd extension sample (int16 BE) | ✅ confirmed |
|
|
||||||
| [4:6] | Unknown (likely checksum) | ❓ open |
|
|
||||||
| [6:8] | Byte length to next segment header − 2 (uint16 BE) | ✅ confirmed |
|
|
||||||
| [8:12] | Monotonic uint32 LE counter (starts ~0x47) | ✅ confirmed |
|
|
||||||
| [12:14] | Constant `02 00` | ✅ confirmed |
|
|
||||||
| [14:16] | THIS segment's channel — sample 0 anchor (int16 BE, 16-count units) | ✅ confirmed |
|
|
||||||
| [16:18] | THIS segment's channel — sample 1 anchor (int16 BE, 16-count units) | ✅ confirmed |
|
|
||||||
|
|
||||||
**Key insight (2026-05-11 late):** every segment carries 510 main
|
|
||||||
samples (2 anchor + 508 deltas) PLUS 2 continuation samples that live
|
|
||||||
in the NEXT segment header. So each channel-segment effectively spans
|
|
||||||
512 sample-sets. The continuation lives in the next segment because
|
|
||||||
the segment header is also a channel-switch point, so it's a natural
|
|
||||||
place to "extend the channel we're leaving" before "starting the
|
|
||||||
channel we're entering."
|
|
||||||
|
|
||||||
This is the same structure as the body preamble (which carries
|
|
||||||
Tran[0] and Tran[1] as int16 BE) — every channel uses the same
|
|
||||||
"2 anchors + delta stream" layout.
|
|
||||||
|
|
||||||
## Channel rotation — VERIFIED 2026-05-11
|
|
||||||
|
|
||||||
```
|
|
||||||
(initial body) → Tran samples 0..509 (preamble + delta blocks)
|
|
||||||
segment 0 hdr ext+anchor → Vert samples 0..511 ← anchor in hdr [14:18]
|
|
||||||
segment 1 hdr ext+anchor → Long samples 0..511
|
|
||||||
segment 2 hdr ext+anchor → Mic samples 0..511
|
|
||||||
segment 3 hdr ext+anchor → Tran samples 510..1021 (continuation)
|
|
||||||
segment 4 hdr ext+anchor → Vert samples 512..1023
|
|
||||||
segment 5 hdr ext+anchor → Long samples 512..1023
|
|
||||||
segment 6 hdr ext+anchor → Mic samples 512..1023
|
|
||||||
segment 7 hdr ext+anchor → Tran samples 1022..1533
|
|
||||||
...
|
|
||||||
```
|
|
||||||
|
|
||||||
Implementation: `decode_waveform_v2()` returns
|
|
||||||
`{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}` with
|
|
||||||
each channel's samples in 16-count units. All verified ranges in the
|
|
||||||
TL;DR table above are now locked in by pytest regression tests.
|
|
||||||
|
|
||||||
## What's still open
|
|
||||||
|
|
||||||
1. **`30 NN` block content.** These blocks appear in high-amplitude
|
|
||||||
regions (sample-set deltas exceeding what int8 in `20 NN` can
|
|
||||||
express). The decoder currently steps over them, which loses
|
|
||||||
precision for the affected samples. Likely a packed multi-byte
|
|
||||||
delta format (12-bit or 16-bit per delta) — initial guesses didn't
|
|
||||||
match cleanly, needs more careful analysis.
|
|
||||||
|
|
||||||
2. **MicL decoding.** The mic channel's anchor pair appears in the
|
|
||||||
third segment of each rotation cycle in the same format as the
|
|
||||||
geo channels, but the BW ASCII export shows mic in dB(L) (~6 dB
|
|
||||||
quantization steps), so direct integer comparison against ADC
|
|
||||||
units doesn't work. Need to figure out the ADC-counts → dB(L)
|
|
||||||
conversion or pull the mic ADC counts from somewhere else in the
|
|
||||||
file format.
|
|
||||||
|
|
||||||
3. **Walker fix for event-b.** The original quiet bundle's event-b
|
|
||||||
still bails out partway through. Lower priority since the other
|
|
||||||
7 events walk cleanly.
|
|
||||||
|
|
||||||
4. **Variable-prefix segment descriptors** (found 2026-08-25).
|
|
||||||
3 of 75 ground-truth production events still truncate. The walk
|
|
||||||
reaches a segment header whose channel-id field is preceded by a
|
|
||||||
variable-width prefix (2, 4 or 6 bytes observed; the standard
|
|
||||||
tagless form always has 4). These also carry an `01 00` marker
|
|
||||||
instead of `02 00`. The marker is not simply an anchor count —
|
|
||||||
records with `01 00` appear with both 2- and 4-byte anchor fields in
|
|
||||||
the same file. Examples: `BE12599/N599LPNB.JF0W` @1155,
|
|
||||||
`BE12599/N599LPWJ.980W` @849, `BE9558/K558LOF2.820W` @1485.
|
|
||||||
|
|
||||||
## Segment header: channel id and tagless form — 2026-08-25
|
|
||||||
|
|
||||||
The 4-byte field previously read as a "monotonic uint32 LE counter" is
|
|
||||||
`[channel_id][00][00][segment_index]`, with `0x46`=Tran `0x47`=Vert
|
|
||||||
`0x48`=Long `0x49`=MicL. Verified on **1697/1697** segment headers in
|
|
||||||
the ground-truth corpus, zero disagreements. `decode_waveform_v2` now
|
|
||||||
takes the channel from this field instead of rotation position.
|
|
||||||
|
|
||||||
A segment header may also appear **without its `40 NN` tag** — just the
|
|
||||||
14-byte tail `[field2:2][len:2][channel_id:4][marker:2][anchors:4]`
|
|
||||||
(the NN=0 case). `is_tagless_segment_header()` detects it from the six
|
|
||||||
bytes at `[4:10]`.
|
|
||||||
|
|
||||||
## Geo scale: full scale is 32000 counts — 2026-08-25
|
|
||||||
|
|
||||||
One decoder unit (16 ADC counts) is exactly 0.005 in/s, so Normal range
|
|
||||||
(10.000 in/s) is `10.0 / (0.005/16)` = **32000** ADC counts. Consumers
|
|
||||||
that divided by 32768 read every geophone sample 2.34% low. Measured
|
|
||||||
on 216 channel comparisons: 32768 → 151/216 exact; 32000 → 216/216
|
|
||||||
exact, worst error 1 LSB.
|
|
||||||
|
|
||||||
**Scope — not waveform-specific.** The scale is applied where ADC
|
|
||||||
counts become physical units, which every event passes through
|
|
||||||
regardless of source codec:
|
|
||||||
|
|
||||||
| source | median ratio ours/device, 32768 | with 32000 |
|
|
||||||
|---|---|---|
|
|
||||||
| series-3 waveform (vs ASCII sample table) | 0.9766 | **1.0000** |
|
|
||||||
| series-3 histogram (vs ASCII PPV, n=1137) | 0.9766 | **1.0000** |
|
|
||||||
| series-4 Thor IDF (vs device peak, n=1468) | 0.960 | **0.983** |
|
|
||||||
|
|
||||||
The four block-framing fixes are waveform-only — `histogram_codec` is
|
|
||||||
untouched by them.
|
|
||||||
|
|
||||||
## Ground-truth corpus (2026-08-25)
|
|
||||||
|
|
||||||
Beyond the bundled fixtures, the production waveform store keeps each
|
|
||||||
event's original Blastware ASCII export at
|
|
||||||
`<store>/<serial>/<filename>_ASCII.TXT`. 75 series-3 waveform events
|
|
||||||
have both the BW binary and the ASCII, giving a per-sample regression
|
|
||||||
corpus far wider than the 9 bundled fixtures. Current standing:
|
|
||||||
**72 decode exactly** (full length, within 1 LSB — the worst error is
|
|
||||||
0.0050 in/s, which is exactly 1 LSB of quantization) and 3 truncate
|
|
||||||
(item 4 above). Zero events have full-length value errors.
|
|
||||||
|
|
||||||
## `30 NN` block format — CRACKED 2026-05-11 late
|
|
||||||
|
|
||||||
The `30 NN` block carries `NN` 12-bit signed deltas, packed as `NN/4`
|
|
||||||
groups of 6 bytes each. Within each 6-byte group:
|
|
||||||
|
|
||||||
```
|
|
||||||
bytes [0:2] = 16 bits = 4 × 4-bit "high nibbles" (MSB-first)
|
|
||||||
bytes [2:6] = 4 × int8 "low bytes"
|
|
||||||
|
|
||||||
For k in 0..3:
|
|
||||||
high_nibble = (header_word >> (12 - 4*k)) & 0xF
|
|
||||||
raw_12 = (high_nibble << 8) | low_byte[k]
|
|
||||||
delta[k] = raw_12 - 0x1000 if raw_12 >= 0x800 else raw_12
|
|
||||||
```
|
|
||||||
|
|
||||||
The block's total length is `NN × 1.5 + 2` bytes (tag included). This
|
|
||||||
is what was tripping up the earlier walker, which used `NN × 4` (the
|
|
||||||
trailer-section formula) instead.
|
|
||||||
|
|
||||||
Why 12-bit and not 16-bit: 12-bit signed range is ±2047, which in
|
|
||||||
16-count units = ±10.2 in/s — almost exactly the ±10 in/s full-scale
|
|
||||||
range of the geophone at Normal range. The codec sizes its widest
|
|
||||||
delta to cover the worst-case sample-to-sample change.
|
|
||||||
|
|
||||||
Verified against all 14 `30 NN` blocks across the bundled fixture
|
|
||||||
events. Every delta decodes byte-exact against BW's ASCII export.
|
|
||||||
|
|
||||||
## Test fixtures
|
|
||||||
|
|
||||||
Committed under `tests/fixtures/`:
|
|
||||||
|
|
||||||
- `decode-re-5-8-26/event-a..event-d/`: original quiet bundle (4 events,
|
|
||||||
PPV < 1 in/s). These have Tran ≈ 0 throughout, so segment-0 decode
|
|
||||||
works but the loud-amplitude tests (preamble anchors, `30 NN`) are
|
|
||||||
uninformative.
|
|
||||||
- `5-11-26/M529LL1A.{SP0,SS0,SV0}`: loud bundle (PPV 6-7 in/s on all
|
|
||||||
channels). These cracked the Tran codec.
|
|
||||||
- `5-11-26/M529LL1L.{JQ0,V70}`: targeted captures. JQ0 is Vert-heavy,
|
|
||||||
V70 is Mic-heavy (140 dB). These cracked the `00 NN` RLE rule.
|
|
||||||
|
|
||||||
Each fixture has a `.TXT` Blastware ASCII export as ground truth.
|
|
||||||
|
|
||||||
## Tests
|
|
||||||
|
|
||||||
`tests/test_waveform_codec.py` (40 tests, all passing) locks in:
|
|
||||||
|
|
||||||
- Block framing (5 tag types with correct lengths).
|
|
||||||
- Walker contiguity (no gaps or overlaps).
|
|
||||||
- Segment header parsing (counter monotonicity, fixed-pattern check).
|
|
||||||
- `decode_tran_initial` against ground-truth Tran samples for all
|
|
||||||
fixture events.
|
|
||||||
|
|
||||||
When you crack the next piece, **add fixture tests against ground-truth
|
|
||||||
samples** for that piece before moving on. Don't let unverified code
|
|
||||||
ship without a regression lock-in.
|
|
||||||
@@ -1,48 +0,0 @@
|
|||||||
"""
|
|
||||||
micromate — Instantel Micromate (Series IV) device library.
|
|
||||||
|
|
||||||
Sibling of ``minimateplus`` (the Series III library). Currently scoped to
|
|
||||||
the offline-file ingest path used by thor-watcher: parsing the per-event
|
|
||||||
``.IDFH``/``.IDFW`` ASCII text sidecars Thor's exporter writes alongside
|
|
||||||
each binary event file, and wrapping the parsed data in typed event
|
|
||||||
records.
|
|
||||||
|
|
||||||
Live-device support (TCP protocol, frame parsing, real-time monitoring)
|
|
||||||
is deferred — when we add it, it lands here as ``transport.py`` /
|
|
||||||
``framing.py`` / ``protocol.py`` / ``client.py``, mirroring the
|
|
||||||
``minimateplus`` package layout.
|
|
||||||
|
|
||||||
Typical usage (offline file ingest):
|
|
||||||
|
|
||||||
from micromate import IdfEvent, parse_idf_report
|
|
||||||
|
|
||||||
text = open("UM11719_20231219162723.IDFW.txt").read()
|
|
||||||
rep = parse_idf_report(text) # dict
|
|
||||||
event = IdfEvent.from_report(rep, "UM11719_20231219162723.IDFW")
|
|
||||||
print(event.serial, event.peaks.transverse_ips, event.mic_pspl_dbl)
|
|
||||||
"""
|
|
||||||
|
|
||||||
from .idf_ascii_report import (
|
|
||||||
parse_event_filename,
|
|
||||||
parse_idf_report,
|
|
||||||
serial_from_filename,
|
|
||||||
)
|
|
||||||
from .models import (
|
|
||||||
IdfEvent,
|
|
||||||
IdfPeaks,
|
|
||||||
IdfProjectInfo,
|
|
||||||
IdfReport,
|
|
||||||
IdfSensorCheck,
|
|
||||||
)
|
|
||||||
|
|
||||||
__version__ = "0.1.0"
|
|
||||||
__all__ = [
|
|
||||||
"IdfEvent",
|
|
||||||
"IdfPeaks",
|
|
||||||
"IdfProjectInfo",
|
|
||||||
"IdfReport",
|
|
||||||
"IdfSensorCheck",
|
|
||||||
"parse_event_filename",
|
|
||||||
"parse_idf_report",
|
|
||||||
"serial_from_filename",
|
|
||||||
]
|
|
||||||
@@ -1,330 +0,0 @@
|
|||||||
"""
|
|
||||||
micromate/idf_ascii_report.py — parse Thor (Micromate Series IV) IDF ASCII reports.
|
|
||||||
|
|
||||||
Thor exports a `.IDFW.txt` or `.IDFH.txt` sidecar next to each `.IDFW`
|
|
||||||
(waveform) or `.IDFH` (histogram) event binary. Each sidecar is a
|
|
||||||
plain-text file with `"Key : Value"` lines covering the full device-
|
|
||||||
authoritative event metadata — PPV per channel, ZC Freq, Time of Peak,
|
|
||||||
Peak Acceleration / Displacement, sensor self-check results, project
|
|
||||||
strings, calibration date, battery level, etc. — followed by a raw
|
|
||||||
waveform-samples block headed by the literal line "Waveform Data Channels".
|
|
||||||
|
|
||||||
This is the Thor analogue of `minimateplus/bw_ascii_report.py` for the
|
|
||||||
Blastware (Series III) report format. The parser is intentionally
|
|
||||||
permissive: we extract everything we recognise into a flat dict and
|
|
||||||
silently ignore anything we don't. Downstream callers parse units
|
|
||||||
(`"0.2119 in/s"` → 0.2119) only on the fields they need.
|
|
||||||
|
|
||||||
Example input (truncated):
|
|
||||||
|
|
||||||
"EventType : Full Waveform"
|
|
||||||
"SampleRate : 1024 sps"
|
|
||||||
"EventTime : 16:27:23"
|
|
||||||
"EventDate : 2023-12-19"
|
|
||||||
"TranPPV : 0.0251 in/s"
|
|
||||||
"VertPPV : 0.2119 in/s"
|
|
||||||
"LongPPV : 0.0282 in/s"
|
|
||||||
"PeakVectorSum : 0.2131 in/s"
|
|
||||||
"MicPSPL : 99.4 dB(L)"
|
|
||||||
"TranZCFreq : 6.5 Hz"
|
|
||||||
"SerialNumber : UM11719"
|
|
||||||
"Version : Micromate ISEE 11.0AK"
|
|
||||||
"FileName : UM11719_20231219162723.IDFW"
|
|
||||||
"BatteryLevel : 3.8 volts"
|
|
||||||
"Calibration : November 22, 2023 by Instantel"
|
|
||||||
"TranTestResults : Passed"
|
|
||||||
"TitleString1 : UPMC Presby-Loc 3-Level1-1R Elevator Rm"
|
|
||||||
Waveform Data Channels
|
|
||||||
Tran Vert Long MicL
|
|
||||||
0.0003 -0.0003 0.0003 0.00013
|
|
||||||
...
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import datetime
|
|
||||||
import re
|
|
||||||
from typing import Any, Dict, Optional, Tuple, Union
|
|
||||||
|
|
||||||
|
|
||||||
# Lines look like: "Key : Value" (quotes literal, single ":" separator)
|
|
||||||
_LINE_RE = re.compile(r'^\s*"?([^":]+?)"?\s*:\s*"?(.*?)"?\s*$')
|
|
||||||
|
|
||||||
# Marker that ends the metadata block — everything after is raw sample data.
|
|
||||||
_WAVEFORM_BLOCK_MARKER = "waveform data channels"
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_key(raw: str) -> str:
|
|
||||||
"""Convert "TranPPV" / "PreTriggerLength" → snake_case."""
|
|
||||||
s = raw.strip()
|
|
||||||
# Insert underscore between lower→upper / digit→letter transitions
|
|
||||||
s = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", "_", s)
|
|
||||||
s = re.sub(r"(?<=[A-Z])(?=[A-Z][a-z])", "_", s)
|
|
||||||
s = s.replace("-", "_").replace(" ", "_")
|
|
||||||
return s.lower()
|
|
||||||
|
|
||||||
|
|
||||||
def _strip_unit_suffix(value: str) -> str:
|
|
||||||
"""Return the numeric part of values like "0.2119 in/s" → "0.2119".
|
|
||||||
|
|
||||||
Also strips Thor's below/above-threshold prefixes:
|
|
||||||
"<0.005 in/s" → "0.005" (below-noise-floor reading)
|
|
||||||
">100 Hz" → "100" (above-measurement-range reading)
|
|
||||||
"""
|
|
||||||
parts = value.strip().split()
|
|
||||||
token = parts[0] if parts else value.strip()
|
|
||||||
if token.startswith("<") or token.startswith(">"):
|
|
||||||
token = token[1:]
|
|
||||||
return token
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_float(value: str) -> Optional[float]:
|
|
||||||
try:
|
|
||||||
return float(_strip_unit_suffix(value))
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_int(value: str) -> Optional[int]:
|
|
||||||
try:
|
|
||||||
return int(float(_strip_unit_suffix(value)))
|
|
||||||
except (ValueError, TypeError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def parse_idf_report(text: Union[str, bytes]) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Parse a Thor IDFW.txt / IDFH.txt sidecar.
|
|
||||||
|
|
||||||
Returns a flat dict with two kinds of entries:
|
|
||||||
|
|
||||||
- **Raw fields** — every `Key : Value` line, keyed by snake_case
|
|
||||||
of the original key, value as a string (unit suffix preserved).
|
|
||||||
Lets callers grab any field we haven't explicitly normalised.
|
|
||||||
|
|
||||||
- **Derived fields** — a curated set with parsed types:
|
|
||||||
* `serial_number` str
|
|
||||||
* `event_type` str ("Full Waveform" / "Full Histogram")
|
|
||||||
* `event_datetime` ISO-8601 string ("YYYY-MM-DDTHH:MM:SS") when
|
|
||||||
both EventDate and EventTime are present
|
|
||||||
* `sample_rate` int (samples/sec)
|
|
||||||
* `tran_ppv`,`vert_ppv`,`long_ppv` float (in/s)
|
|
||||||
* `mic_ppv` float (dB or psi — same units as MicPSPL)
|
|
||||||
* `peak_vector_sum` float (in/s)
|
|
||||||
* `tran_zc_freq`,`vert_zc_freq`,`long_zc_freq` float (Hz)
|
|
||||||
* `record_time_sec` float (seconds)
|
|
||||||
* `pre_trigger_sec` float (seconds)
|
|
||||||
* `project` str (from TitleString1 — Thor's location)
|
|
||||||
* `client` str (TitleString2)
|
|
||||||
* `operator` str (TitleString3 — company/operator)
|
|
||||||
* `notes` str (TitleString4)
|
|
||||||
* `setup` str
|
|
||||||
* `version` str (firmware)
|
|
||||||
* `battery_volts` float
|
|
||||||
* `calibration_text` str (e.g. "November 22, 2023 by Instantel")
|
|
||||||
* `tran_test_passed`, `vert_test_passed`, `long_test_passed`,
|
|
||||||
`mic_test_passed` bool ("Passed" → True; anything else → False)
|
|
||||||
* `filename` str (FileName line — useful sanity check)
|
|
||||||
|
|
||||||
Stops parsing at the literal "Waveform Data Channels" line; the
|
|
||||||
raw-samples block is left to whoever wants to decode the binary.
|
|
||||||
|
|
||||||
Input may be `str` or `bytes` (`utf-8`/`latin-1` tolerant).
|
|
||||||
"""
|
|
||||||
if isinstance(text, bytes):
|
|
||||||
try:
|
|
||||||
text = text.decode("utf-8")
|
|
||||||
except UnicodeDecodeError:
|
|
||||||
text = text.decode("latin-1", errors="replace")
|
|
||||||
|
|
||||||
raw: Dict[str, str] = {}
|
|
||||||
|
|
||||||
for line in text.splitlines():
|
|
||||||
stripped = line.strip()
|
|
||||||
if not stripped:
|
|
||||||
continue
|
|
||||||
if stripped.lower().startswith(_WAVEFORM_BLOCK_MARKER):
|
|
||||||
break
|
|
||||||
m = _LINE_RE.match(stripped)
|
|
||||||
if not m:
|
|
||||||
continue
|
|
||||||
key = _normalize_key(m.group(1))
|
|
||||||
value = m.group(2).strip()
|
|
||||||
# Multi-value lines (Channel, Units, etc.) — coalesce by appending.
|
|
||||||
if key in raw:
|
|
||||||
raw[key] = raw[key] + "; " + value
|
|
||||||
else:
|
|
||||||
raw[key] = value
|
|
||||||
|
|
||||||
out: Dict[str, Any] = dict(raw) # keep all raw fields
|
|
||||||
|
|
||||||
# ── Derived fields ───────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
def _take(*candidates: str) -> Optional[str]:
|
|
||||||
for c in candidates:
|
|
||||||
if c in raw:
|
|
||||||
return raw[c]
|
|
||||||
return None
|
|
||||||
|
|
||||||
# Event identity
|
|
||||||
if "serial_number" in raw:
|
|
||||||
out["serial_number"] = raw["serial_number"]
|
|
||||||
if "event_type" in raw:
|
|
||||||
out["event_type"] = raw["event_type"]
|
|
||||||
if "file_name" in raw:
|
|
||||||
out["filename"] = raw["file_name"]
|
|
||||||
|
|
||||||
# Combined date+time. Waveform sidecars use "EventDate" / "EventTime";
|
|
||||||
# histogram sidecars use "HistogramStartDate" / "HistogramStartTime".
|
|
||||||
# Prefer the event_* names when both are present.
|
|
||||||
ed = raw.get("event_date") or raw.get("histogram_start_date")
|
|
||||||
et = raw.get("event_time") or raw.get("histogram_start_time")
|
|
||||||
if ed and et:
|
|
||||||
try:
|
|
||||||
dt = datetime.datetime.strptime(f"{ed} {et}", "%Y-%m-%d %H:%M:%S")
|
|
||||||
out["event_datetime"] = dt.isoformat()
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Numeric scalars. For every field we typify here, we MUST drop the
|
|
||||||
# raw string copy from `out` when parsing fails — Thor writes things
|
|
||||||
# like "<0.005 in/s" (below threshold) and "N/A" (not measured) that
|
|
||||||
# would otherwise linger in `out` as strings, sneak into SQLite REAL
|
|
||||||
# columns via permissive type affinity, and then crash the JS
|
|
||||||
# frontend on `.toFixed(...)`.
|
|
||||||
int_fields = ("sample_rate",)
|
|
||||||
for key in int_fields:
|
|
||||||
v = raw.get(key)
|
|
||||||
if v is None:
|
|
||||||
continue
|
|
||||||
iv = _parse_int(v)
|
|
||||||
if iv is not None:
|
|
||||||
out[key] = iv
|
|
||||||
else:
|
|
||||||
out.pop(key, None)
|
|
||||||
|
|
||||||
float_fields = (
|
|
||||||
"tran_ppv", "vert_ppv", "long_ppv", "peak_vector_sum",
|
|
||||||
"tran_zc_freq", "vert_zc_freq", "long_zc_freq",
|
|
||||||
"tran_peak_acceleration", "vert_peak_acceleration",
|
|
||||||
"long_peak_acceleration",
|
|
||||||
"tran_peak_displacement", "vert_peak_displacement",
|
|
||||||
"long_peak_displacement",
|
|
||||||
"mic_zc_freq",
|
|
||||||
)
|
|
||||||
for key in float_fields:
|
|
||||||
v = raw.get(key)
|
|
||||||
if v is None:
|
|
||||||
continue
|
|
||||||
fv = _parse_float(v)
|
|
||||||
if fv is not None:
|
|
||||||
out[key] = fv
|
|
||||||
else:
|
|
||||||
out.pop(key, None)
|
|
||||||
|
|
||||||
# Time-of-peak: Thor labels these "TimeofPeak" (lowercase "of") so the
|
|
||||||
# normalizer produces "*_timeof_peak". Map them to the canonical
|
|
||||||
# ``*_time_of_peak`` output keys for downstream consumers.
|
|
||||||
for raw_key, out_key in (
|
|
||||||
("tran_timeof_peak", "tran_time_of_peak"),
|
|
||||||
("vert_timeof_peak", "vert_time_of_peak"),
|
|
||||||
("long_timeof_peak", "long_time_of_peak"),
|
|
||||||
("mic_timeof_peak", "mic_time_of_peak"),
|
|
||||||
):
|
|
||||||
v = raw.get(raw_key)
|
|
||||||
if v is None:
|
|
||||||
continue
|
|
||||||
fv = _parse_float(v)
|
|
||||||
if fv is not None:
|
|
||||||
out[out_key] = fv
|
|
||||||
|
|
||||||
# Microphone — Thor reports MicPSPL (dB(L)) which is the closest
|
|
||||||
# analogue to BW's mic_ppv. The raw "99.4 dB(L)" string stays in
|
|
||||||
# `out` under the original `mic_pspl` key for display; the parsed
|
|
||||||
# float goes in `mic_ppv`.
|
|
||||||
mic = raw.get("mic_pspl")
|
|
||||||
if mic is not None:
|
|
||||||
fv = _parse_float(mic)
|
|
||||||
if fv is not None:
|
|
||||||
out["mic_ppv"] = fv
|
|
||||||
|
|
||||||
# Record / pre-trigger duration — same drop-on-failure discipline.
|
|
||||||
rt = raw.get("record_time")
|
|
||||||
if rt is not None:
|
|
||||||
fv = _parse_float(rt)
|
|
||||||
if fv is not None:
|
|
||||||
out["record_time_sec"] = fv
|
|
||||||
pt = raw.get("pre_trigger_length")
|
|
||||||
if pt is not None:
|
|
||||||
fv = _parse_float(pt)
|
|
||||||
if fv is not None:
|
|
||||||
out["pre_trigger_sec"] = fv
|
|
||||||
|
|
||||||
# Project / client / operator / location strings. Thor's title
|
|
||||||
# strings are operator-defined; conventional mapping (per Thor's
|
|
||||||
# default TitleNote labels in the example data):
|
|
||||||
# TitleString1 = Location → project (sensor location identifier)
|
|
||||||
# TitleString2 = Client → client
|
|
||||||
# TitleString3 = Company → operator (the monitoring company)
|
|
||||||
# TitleString4 = Notes → notes
|
|
||||||
out["project"] = _take("title_string1")
|
|
||||||
out["client"] = _take("title_string2")
|
|
||||||
out["operator"] = _take("title_string3", "operator")
|
|
||||||
out["notes"] = _take("title_string4", "post_event_note")
|
|
||||||
|
|
||||||
if "setup" in raw:
|
|
||||||
out["setup"] = raw["setup"]
|
|
||||||
if "version" in raw:
|
|
||||||
out["version"] = raw["version"]
|
|
||||||
|
|
||||||
# Battery (e.g. "3.8 volts" → 3.8)
|
|
||||||
bl = raw.get("battery_level")
|
|
||||||
if bl is not None:
|
|
||||||
fv = _parse_float(bl)
|
|
||||||
if fv is not None:
|
|
||||||
out["battery_volts"] = fv
|
|
||||||
|
|
||||||
# Calibration line is free-form (e.g. "November 22, 2023 by Instantel").
|
|
||||||
if "calibration" in raw:
|
|
||||||
out["calibration_text"] = raw["calibration"]
|
|
||||||
|
|
||||||
# Sensor self-check results — bool flags
|
|
||||||
for key, out_key in (
|
|
||||||
("tran_test_results", "tran_test_passed"),
|
|
||||||
("vert_test_results", "vert_test_passed"),
|
|
||||||
("long_test_results", "long_test_passed"),
|
|
||||||
("mic_test_results", "mic_test_passed"),
|
|
||||||
):
|
|
||||||
v = raw.get(key)
|
|
||||||
if v is not None:
|
|
||||||
out[out_key] = v.strip().lower() == "passed"
|
|
||||||
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def serial_from_filename(name: str) -> Optional[str]:
|
|
||||||
"""Convenience: pull the serial prefix from a Thor event filename.
|
|
||||||
|
|
||||||
Thor uses the literal serial as the filename prefix:
|
|
||||||
UM11719_20231219163444.IDFW → "UM11719"
|
|
||||||
BE9439_20200713124251.IDFH → "BE9439"
|
|
||||||
"""
|
|
||||||
m = re.match(r"^([A-Z]{2}\d+)_\d{14}\.(IDFH|IDFW)(?:\.txt)?$",
|
|
||||||
name, re.IGNORECASE)
|
|
||||||
return m.group(1).upper() if m else None
|
|
||||||
|
|
||||||
|
|
||||||
def parse_event_filename(name: str) -> Optional[Tuple[str, datetime.datetime, str]]:
|
|
||||||
"""Parse `<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>` → (serial, datetime, kind).
|
|
||||||
|
|
||||||
`kind` is "IDFH" or "IDFW" (upper-case). Returns None on no match.
|
|
||||||
"""
|
|
||||||
m = re.match(r"^([A-Z]{2}\d+)_(\d{14})\.(IDFH|IDFW)$",
|
|
||||||
name, re.IGNORECASE)
|
|
||||||
if not m:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
ts = datetime.datetime.strptime(m.group(2), "%Y%m%d%H%M%S")
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
return m.group(1).upper(), ts, m.group(3).upper()
|
|
||||||
@@ -1,542 +0,0 @@
|
|||||||
"""
|
|
||||||
micromate/idf_file.py — Thor IDF binary codec.
|
|
||||||
|
|
||||||
Decodes the Instantel Micromate Series IV ``.IDFW`` (waveform) and
|
|
||||||
``.IDFH`` (histogram) binary on-disk format. Sister module to
|
|
||||||
``minimateplus/event_file_io.py``.
|
|
||||||
|
|
||||||
Status (2026-05-28):
|
|
||||||
|
|
||||||
- **Genuine Series IV / Thor binaries** are all signed
|
|
||||||
``00 12 01 00 00 00 Instantel\\0`` (sig-A in earlier notes). Two
|
|
||||||
Series III (Blastware) binaries appear in the example corpus
|
|
||||||
(``BE9439_*``) — they share the ``.IDFW``/``.IDFH`` extension by
|
|
||||||
filing convention but carry a BW STRT header (``10 00 01 80 00 00
|
|
||||||
Instantel STRT...``) and are NOT Thor data. The reader detects
|
|
||||||
them by signature and raises NotImplementedError pointing callers
|
|
||||||
at ``minimateplus.event_file_io.read_blastware_file()``.
|
|
||||||
- **IDFW waveform body** reuses the BW segment-rotated block codec
|
|
||||||
verbatim. Body always starts at file offset ``0x0f1f``. Samples
|
|
||||||
decoded via ``minimateplus.waveform_codec.decode_waveform_v2``
|
|
||||||
with 87–99% byte-exact match against ``.IDFW.txt`` sidecar (quiet
|
|
||||||
events). Loud events hit the BW codec's known walker-stops-early
|
|
||||||
limit. Residual ~3% drift on per-sample deltas — likely a
|
|
||||||
Thor-specific 12-bit delta refinement that BW's codec doesn't
|
|
||||||
model. Geo LSB = 0.0003 in/s; mic factor ~2.14e-6 psi/count.
|
|
||||||
- **IDFH histogram body**: 12-byte segment header
|
|
||||||
``[len_be 2B] 0a 00 00 00 [00 NN_counter] 05 3f`` introduces a
|
|
||||||
segment of ``N`` 72-byte interval records (``N = (len - 10) // 72``).
|
|
||||||
Each record holds 4 × 16-byte per-channel min/max/halfp + 8-byte
|
|
||||||
tail. Geo peaks via ``max(|min|, |max|) / 32768 × 10`` in/s
|
|
||||||
(matches sidecar within ~1.8%), freq via ``512 / halfp`` Hz.
|
|
||||||
**All 859 Thor IDFH files in the corpus decode (181,071 intervals).**
|
|
||||||
- Binary metadata directly extracted: serial, timestamp, sample_rate,
|
|
||||||
record_time, calibration_date. Other fields fall back to the paired
|
|
||||||
``.IDFW.txt`` / ``.IDFH.txt`` sidecar (consumed by
|
|
||||||
``WaveformStore.save_imported_idf``).
|
|
||||||
|
|
||||||
The full reverse-engineering writeup lives in
|
|
||||||
``docs/idf_protocol_reference.md``.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import datetime
|
|
||||||
import struct
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Optional, Union
|
|
||||||
|
|
||||||
# Thor IDFW bodies are pinned to the SUPERSEDED tag-dispatch decoder.
|
|
||||||
#
|
|
||||||
# _find_waveform_body_offset() trial-decodes every candidate offset and keeps
|
|
||||||
# whichever yields the most samples. The series-3 record-chain decoder
|
|
||||||
# correctly returns None where the legacy walker returned garbage, which
|
|
||||||
# changes that heuristic's winner on 33 of 577 files. The net effect measured
|
|
||||||
# 2026-08-25 was positive (all-channels-equal 8/577 -> 506/577, mean abs PPV
|
|
||||||
# error 0.228 -> 0.173 in/s) but Thor has no ASCII ground truth in the corpus
|
|
||||||
# and its geo scaling is separately suspect, so the switch is deferred until
|
|
||||||
# the body-offset search is reworked to use the record chain directly.
|
|
||||||
from minimateplus.waveform_codec import (
|
|
||||||
decode_waveform_legacy as decode_waveform_v2,
|
|
||||||
)
|
|
||||||
|
|
||||||
from .models import IdfEvent, IdfPeaks, IdfReport
|
|
||||||
|
|
||||||
|
|
||||||
# Genuine Series IV / Thor IDF binary signature: 6 bytes, then ASCII "Instantel".
|
|
||||||
_THOR_PREFIX = b"\x00\x12\x01\x00\x00\x00"
|
|
||||||
# Stray Series III (Blastware) binaries that occasionally turn up in Thor
|
|
||||||
# corpus directories renamed to the .IDFW/.IDFH convention. Their header
|
|
||||||
# (`10 00 01 80 00 00 Instantel STRT ...`) is byte-for-byte a BW SUB 5A
|
|
||||||
# STRT record, not a Thor binary. Detected so we can refuse-and-route
|
|
||||||
# rather than mis-parse.
|
|
||||||
_BW_STRAY_PREFIX = b"\x10\x00\x01\x80\x00\x00"
|
|
||||||
_INSTANTEL_TAG = b"Instantel"
|
|
||||||
|
|
||||||
# Most common body offset for sig-A IDFW files (~50% of prod events;
|
|
||||||
# 151/154 in the original tests/fixtures/THORDATA_example corpus). The
|
|
||||||
# body is the segment-rotated block stream consumed by decode_waveform_v2;
|
|
||||||
# bytes [0:3] are the magic ``00 02 00`` preamble. Production events
|
|
||||||
# routinely use other offsets — see :func:`_find_waveform_body_offset`
|
|
||||||
# for the dynamic scan. This constant survives only as the priority hint.
|
|
||||||
_BODY_START_SIG_A = 0x0F1F
|
|
||||||
|
|
||||||
# Magic bytes that mark a candidate waveform-body preamble.
|
|
||||||
_BODY_MAGIC = b"\x00\x02\x00"
|
|
||||||
|
|
||||||
# Where to start looking for body candidates inside the file. Skip the
|
|
||||||
# fixed-header region where the same magic legitimately appears inside
|
|
||||||
# channel-test records and the compliance block (offsets 0x015d, 0x091c,
|
|
||||||
# 0x0ae2, 0x0d30 in observed events).
|
|
||||||
_BODY_SCAN_FLOOR = 0x0E00
|
|
||||||
|
|
||||||
# Geophone count → in/s, derived from sidecar ground truth: the smallest
|
|
||||||
# non-zero sample in 1,014-file corpus is 0.0003 in/s.
|
|
||||||
_GEO_LSB_IPS = 0.0003
|
|
||||||
|
|
||||||
# Microphone count → psi, derived from sidecar regression on 50 sample
|
|
||||||
# pairs from UM11719_20231219162723.IDFW (mic-heavy event).
|
|
||||||
_MIC_LSB_PSI = 2.14e-6
|
|
||||||
|
|
||||||
# IDFH histogram constants.
|
|
||||||
_IDFH_INTERVAL_SIZE = 72 # bytes per per-interval record
|
|
||||||
_IDFH_SEGMENT_HEADER = 10 # bytes: [len_be 2B][0a 00 00 00 4B][00 NN 2B][05 3f 2B]
|
|
||||||
_IDFH_SEGMENT_TAIL = 2 # bytes after the interval data block, before next marker
|
|
||||||
_IDFH_HALFP_FREQ_NUM = 512.0 # freq_hz = NUM / halfp; halfp ≤ 5 means ">100 Hz" sentinel
|
|
||||||
_IDFH_GEO_FULL_SCALE = 10.0 # in/s — Normal range
|
|
||||||
_IDFH_INT16_FS = 32768.0
|
|
||||||
_IDFH_CHANNELS = ("Tran", "Vert", "Long", "MicL")
|
|
||||||
|
|
||||||
|
|
||||||
# ─── Binary metadata extraction ─────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class IdfBinaryMetadata:
|
|
||||||
"""Fields recoverable from the sig-A binary header (no .txt needed)."""
|
|
||||||
serial: Optional[str] = None
|
|
||||||
event_datetime: Optional[datetime.datetime] = None
|
|
||||||
sample_rate: Optional[int] = None
|
|
||||||
record_time_sec: Optional[float] = None
|
|
||||||
calibration_date: Optional[datetime.date] = None
|
|
||||||
|
|
||||||
|
|
||||||
def _read_ascii_z(buf: bytes, off: int, maxlen: int = 64) -> Optional[str]:
|
|
||||||
if off >= len(buf):
|
|
||||||
return None
|
|
||||||
end = buf.find(b"\x00", off, off + maxlen)
|
|
||||||
if end < 0:
|
|
||||||
end = min(off + maxlen, len(buf))
|
|
||||||
s = buf[off:end].decode("ascii", errors="replace").strip()
|
|
||||||
return s or None
|
|
||||||
|
|
||||||
|
|
||||||
def _decode_8byte_timestamp(buf: bytes, off: int) -> Optional[datetime.datetime]:
|
|
||||||
"""Layout: ``[day][month][year_hi][year_lo][unknown][hour][min][sec]``."""
|
|
||||||
if off + 8 > len(buf):
|
|
||||||
return None
|
|
||||||
day, mon, yh, yl, _unk, hr, mn, sc = buf[off : off + 8]
|
|
||||||
year = (yh << 8) | yl
|
|
||||||
if not (2015 <= year <= 2050 and 1 <= mon <= 12 and 1 <= day <= 31
|
|
||||||
and 0 <= hr < 24 and 0 <= mn < 60 and 0 <= sc < 60):
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return datetime.datetime(year, mon, day, hr, mn, sc)
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def extract_binary_metadata(buf: bytes) -> IdfBinaryMetadata:
|
|
||||||
"""Pull serial/timestamp/sample_rate/record_time/calibration from the
|
|
||||||
sig-A binary header.
|
|
||||||
|
|
||||||
Field positions confirmed against UM11719_20231219162723.IDFW; stable
|
|
||||||
across the 151-file sig-A corpus.
|
|
||||||
"""
|
|
||||||
md = IdfBinaryMetadata()
|
|
||||||
|
|
||||||
# Serial: null-terminated ASCII at 0x14E.
|
|
||||||
md.serial = _read_ascii_z(buf, 0x14E, maxlen=16)
|
|
||||||
|
|
||||||
# Sample rate + record time live in a BW-compatible compliance block.
|
|
||||||
# Locate the 6-byte anchor `be 80 00 00 00 00` and read offsets relative
|
|
||||||
# to it: anchor-6 = sample_rate uint16 BE; anchor+6 = record_time float32 BE.
|
|
||||||
anchor = buf.find(b"\xbe\x80\x00\x00\x00\x00", 0x800, 0xA00)
|
|
||||||
if anchor > 0:
|
|
||||||
sr_bytes = buf[anchor - 6 : anchor - 4]
|
|
||||||
if len(sr_bytes) == 2:
|
|
||||||
sr = int.from_bytes(sr_bytes, "big")
|
|
||||||
if sr in (256, 512, 1024, 2048, 4096):
|
|
||||||
md.sample_rate = sr
|
|
||||||
rt_bytes = buf[anchor + 6 : anchor + 10]
|
|
||||||
if len(rt_bytes) == 4:
|
|
||||||
try:
|
|
||||||
rt = struct.unpack(">f", rt_bytes)[0]
|
|
||||||
if 0.1 <= rt <= 600.0:
|
|
||||||
md.record_time_sec = float(rt)
|
|
||||||
except struct.error:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# Event timestamp: 8 bytes. Position differs between IDFW (0x97A) and
|
|
||||||
# IDFH (0x9F8); scan a small range and accept the first valid decode.
|
|
||||||
for off in (0x97A, 0x9F8):
|
|
||||||
ts = _decode_8byte_timestamp(buf, off)
|
|
||||||
if ts is not None:
|
|
||||||
md.event_datetime = ts
|
|
||||||
break
|
|
||||||
|
|
||||||
# Calibration date: day, month, year_be at 0x194-0x197.
|
|
||||||
if len(buf) > 0x197:
|
|
||||||
day, mon = buf[0x194], buf[0x195]
|
|
||||||
year = int.from_bytes(buf[0x196 : 0x198], "big")
|
|
||||||
if 1 <= mon <= 12 and 1 <= day <= 31 and 2015 <= year <= 2050:
|
|
||||||
try:
|
|
||||||
md.calibration_date = datetime.date(year, mon, day)
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return md
|
|
||||||
|
|
||||||
|
|
||||||
# ─── Sample decoder + unit conversion ───────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def _find_waveform_body_offset(buf: bytes) -> Optional[int]:
|
|
||||||
"""Pick the file offset of the waveform body by trial-decoding every
|
|
||||||
``00 02 00`` magic position past the fixed-header region.
|
|
||||||
|
|
||||||
The body's location isn't fixed across all sig-A IDFW files — about
|
|
||||||
half the production events use ``0x0f1f``, but the rest have offsets
|
|
||||||
that shift based on header padding / channel-config layout. We
|
|
||||||
auto-detect by:
|
|
||||||
|
|
||||||
1. Find every ``00 02 00`` occurrence past ``_BODY_SCAN_FLOOR``.
|
|
||||||
2. Try ``decode_waveform_v2()`` on each candidate.
|
|
||||||
3. Pick the offset whose decoded sample count is largest.
|
|
||||||
|
|
||||||
Returns the offset, or ``None`` if no candidate yielded more than
|
|
||||||
the trivial 2-sample preamble (= "no real body found").
|
|
||||||
|
|
||||||
Costs ~2-8 trial decodes per file; in practice the first candidate
|
|
||||||
past 0x0e00 is usually the right one.
|
|
||||||
"""
|
|
||||||
if len(buf) < _BODY_SCAN_FLOOR + 8:
|
|
||||||
return None
|
|
||||||
best: Optional[tuple[int, int]] = None # (total_samples, offset)
|
|
||||||
i = _BODY_SCAN_FLOOR
|
|
||||||
while True:
|
|
||||||
j = buf.find(_BODY_MAGIC, i)
|
|
||||||
if j < 0:
|
|
||||||
break
|
|
||||||
i = j + 1
|
|
||||||
try:
|
|
||||||
decoded = decode_waveform_v2(buf[j:])
|
|
||||||
except Exception:
|
|
||||||
continue
|
|
||||||
if not decoded:
|
|
||||||
continue
|
|
||||||
total = sum(len(v) for v in decoded.values())
|
|
||||||
# A "real" body has more than just the 2-sample preamble.
|
|
||||||
if total <= 2:
|
|
||||||
continue
|
|
||||||
if best is None or total > best[0]:
|
|
||||||
best = (total, j)
|
|
||||||
return best[1] if best else None
|
|
||||||
|
|
||||||
|
|
||||||
def _decode_waveform_samples(buf: bytes) -> Optional[dict]:
|
|
||||||
"""Decode samples from the sig-A waveform body.
|
|
||||||
|
|
||||||
Returns the raw decoder counts dict — geo LSB = 0.0003 in/s, mic in
|
|
||||||
its own count unit (see :func:`mic_count_to_psi`). Returns None if
|
|
||||||
no usable body is found.
|
|
||||||
|
|
||||||
Uses :func:`_find_waveform_body_offset` to locate the body — the
|
|
||||||
file-offset varies across events (~50% sit at the canonical
|
|
||||||
``0x0f1f`` but the rest don't), so the previous hardcoded constant
|
|
||||||
silently produced 2-sample preamble-only output for half the corpus.
|
|
||||||
"""
|
|
||||||
off = _find_waveform_body_offset(buf)
|
|
||||||
if off is None:
|
|
||||||
return None
|
|
||||||
return decode_waveform_v2(buf[off:])
|
|
||||||
|
|
||||||
|
|
||||||
def geo_count_to_ips(count: int) -> float:
|
|
||||||
"""Convert a Thor geo decoder count to in/s. LSB = 0.0003 in/s."""
|
|
||||||
return count * _GEO_LSB_IPS
|
|
||||||
|
|
||||||
|
|
||||||
def mic_count_to_psi(count: int) -> float:
|
|
||||||
"""Convert a Thor mic decoder count to psi. Scale derived from
|
|
||||||
regression over 50 sample pairs in UM11719_20231219162723.IDFW;
|
|
||||||
consistent to ~5%. Calibration constants from the channel block
|
|
||||||
can refine this once decoded.
|
|
||||||
"""
|
|
||||||
return count * _MIC_LSB_PSI
|
|
||||||
|
|
||||||
|
|
||||||
# ─── IDFH histogram decoder ─────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class IdfhInterval:
|
|
||||||
"""One decoded histogram interval (typically one minute of monitoring)."""
|
|
||||||
offset: int # file byte offset of the 72-byte record
|
|
||||||
# Per-channel min/max ADC counts (int16 BE), half-period samples, peak count.
|
|
||||||
# Peak = max(|min|, |max|). freq_hz = 512/halfp (None if halfp ≤ 5 →
|
|
||||||
# ">100 Hz" sentinel; matches sidecar convention).
|
|
||||||
tran_min: int
|
|
||||||
tran_max: int
|
|
||||||
tran_halfp: int
|
|
||||||
vert_min: int
|
|
||||||
vert_max: int
|
|
||||||
vert_halfp: int
|
|
||||||
long_min: int
|
|
||||||
long_max: int
|
|
||||||
long_halfp: int
|
|
||||||
micl_min: int
|
|
||||||
micl_max: int
|
|
||||||
micl_halfp: int
|
|
||||||
|
|
||||||
def peak_count(self, channel: str) -> int:
|
|
||||||
mn = getattr(self, f"{channel.lower()}_min")
|
|
||||||
mx = getattr(self, f"{channel.lower()}_max")
|
|
||||||
return max(abs(mn), abs(mx))
|
|
||||||
|
|
||||||
def peak_ips(self, channel: str) -> float:
|
|
||||||
"""Convert peak count to in/s (geo channels only)."""
|
|
||||||
return self.peak_count(channel) / _IDFH_INT16_FS * _IDFH_GEO_FULL_SCALE
|
|
||||||
|
|
||||||
def freq_hz(self, channel: str) -> Optional[float]:
|
|
||||||
halfp = getattr(self, f"{channel.lower()}_halfp")
|
|
||||||
if halfp <= 5:
|
|
||||||
return None
|
|
||||||
return _IDFH_HALFP_FREQ_NUM / halfp
|
|
||||||
|
|
||||||
|
|
||||||
def _decode_idfh_interval(buf72: bytes, offset: int) -> IdfhInterval:
|
|
||||||
"""Decode one 72-byte interval record into per-channel min/max/halfp."""
|
|
||||||
import struct
|
|
||||||
fields = []
|
|
||||||
for i in range(4):
|
|
||||||
block = buf72[i * 16 : (i + 1) * 16]
|
|
||||||
mn = struct.unpack_from(">h", block, 0)[0]
|
|
||||||
mx = struct.unpack_from(">h", block, 2)[0]
|
|
||||||
# block[4:6] = int16 BE, role unknown (possibly time-of-peak)
|
|
||||||
halfp = struct.unpack_from(">H", block, 6)[0]
|
|
||||||
# block[10:12] and block[14:16] are uint16 BE with unknown semantics
|
|
||||||
# (likely sum / count contributions for the PVS computation).
|
|
||||||
fields.extend([mn, mx, halfp])
|
|
||||||
# Tail 8 bytes (buf72[64:72]) carry PVS-related data; not yet decoded.
|
|
||||||
return IdfhInterval(
|
|
||||||
offset=offset,
|
|
||||||
tran_min=fields[0], tran_max=fields[1], tran_halfp=fields[2],
|
|
||||||
vert_min=fields[3], vert_max=fields[4], vert_halfp=fields[5],
|
|
||||||
long_min=fields[6], long_max=fields[7], long_halfp=fields[8],
|
|
||||||
micl_min=fields[9], micl_max=fields[10], micl_halfp=fields[11],
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def decode_idfh_body(buf: bytes) -> list:
|
|
||||||
"""Walk an IDFH file and decode every interval record.
|
|
||||||
|
|
||||||
The body has one or more segments; each segment header is 12 bytes:
|
|
||||||
``[length_be 2B][0a 00 00 00][00 NN_counter][05 3f]`` where ``length``
|
|
||||||
is bytes from the magic through the end of the interval block
|
|
||||||
(= 10 + 72 × n_intervals). Segments are separated by a 2-byte tail
|
|
||||||
+ next-segment 2-byte prefix (the bytes before the next length field).
|
|
||||||
Confirmed against the 859-file corpus (181,071 intervals decoded; 1
|
|
||||||
failure is the sig-B BE9439 file).
|
|
||||||
"""
|
|
||||||
intervals: list = []
|
|
||||||
i = 0
|
|
||||||
while True:
|
|
||||||
j = buf.find(b"\x0a\x00\x00\x00", i)
|
|
||||||
if j < 0 or j < 2:
|
|
||||||
break
|
|
||||||
# Validate: [length_be][0a 00 00 00][00 NN][05 3f]
|
|
||||||
if buf[j + 4] != 0x00 or buf[j + 6 : j + 8] != b"\x05\x3f":
|
|
||||||
i = j + 1
|
|
||||||
continue
|
|
||||||
length = int.from_bytes(buf[j - 2 : j], "big")
|
|
||||||
n = (length - _IDFH_SEGMENT_HEADER) // _IDFH_INTERVAL_SIZE
|
|
||||||
if n <= 0:
|
|
||||||
i = j + 1
|
|
||||||
continue
|
|
||||||
header_start = j - 2
|
|
||||||
interval_start = header_start + _IDFH_SEGMENT_HEADER
|
|
||||||
for k in range(n):
|
|
||||||
off = interval_start + k * _IDFH_INTERVAL_SIZE
|
|
||||||
if off + _IDFH_INTERVAL_SIZE > len(buf):
|
|
||||||
break
|
|
||||||
chunk = buf[off : off + _IDFH_INTERVAL_SIZE]
|
|
||||||
intervals.append(_decode_idfh_interval(chunk, off))
|
|
||||||
# Advance past this segment + the 2-byte tail.
|
|
||||||
i = header_start + length + _IDFH_SEGMENT_TAIL
|
|
||||||
return intervals
|
|
||||||
|
|
||||||
|
|
||||||
# ─── Top-level reader ───────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class IdfReadResult:
|
|
||||||
"""Return type for :func:`read_idf_file`.
|
|
||||||
|
|
||||||
For waveforms (``.IDFW``), ``samples`` holds the per-channel sample
|
|
||||||
arrays in Thor decoder counts. For histograms (``.IDFH``),
|
|
||||||
``samples`` is empty and ``intervals`` holds the per-interval
|
|
||||||
record list (peaks, freqs).
|
|
||||||
"""
|
|
||||||
event: IdfEvent
|
|
||||||
samples: dict # {"Tran": [...], ...} for IDFW; empty for IDFH
|
|
||||||
binary_metadata: IdfBinaryMetadata
|
|
||||||
signature: str # always "thor" for now (sig-A genuine Thor)
|
|
||||||
intervals: Optional[list] = None # list[IdfhInterval] for IDFH; None for IDFW
|
|
||||||
|
|
||||||
|
|
||||||
def read_idf_file(
|
|
||||||
path: Union[str, Path],
|
|
||||||
*,
|
|
||||||
data: Optional[bytes] = None,
|
|
||||||
) -> IdfReadResult:
|
|
||||||
"""Parse a Thor ``.IDFW`` binary into an ``IdfEvent`` + decoded samples.
|
|
||||||
|
|
||||||
Currently implements signature-A waveforms only. Signature-B
|
|
||||||
(old-firmware) and ``.IDFH`` histograms raise NotImplementedError;
|
|
||||||
use the paired ``.IDFW.txt`` / ``.IDFH.txt`` sidecar for those via
|
|
||||||
``parse_idf_report()``.
|
|
||||||
|
|
||||||
Returns an :class:`IdfReadResult`. The caller converts int sample
|
|
||||||
counts to physical units via :func:`geo_count_to_ips` /
|
|
||||||
:func:`mic_count_to_psi`.
|
|
||||||
|
|
||||||
``path`` is used for filename in error messages and ``.IDFH`` vs
|
|
||||||
``.IDFW`` suffix detection. When ``data`` is supplied the disk
|
|
||||||
read is skipped — useful for ingest paths that already have the
|
|
||||||
bytes in memory and where the file may not exist on disk yet.
|
|
||||||
"""
|
|
||||||
p = Path(path)
|
|
||||||
buf = data if data is not None else p.read_bytes()
|
|
||||||
|
|
||||||
if len(buf) < 16 or buf[6:16] != _INSTANTEL_TAG + b"\x00":
|
|
||||||
raise ValueError(f"{p.name}: not an IDF file (missing Instantel magic)")
|
|
||||||
|
|
||||||
sig_prefix = buf[:6]
|
|
||||||
if sig_prefix == _THOR_PREFIX:
|
|
||||||
signature = "thor"
|
|
||||||
elif sig_prefix == _BW_STRAY_PREFIX:
|
|
||||||
raise NotImplementedError(
|
|
||||||
f"{p.name}: file has a Series III (Blastware) STRT header in "
|
|
||||||
"an IDF-named container — not a Thor binary. Route through "
|
|
||||||
"minimateplus.event_file_io.read_blastware_file() instead "
|
|
||||||
"(peaks decode; samples & full metadata don't, but it's not "
|
|
||||||
"Thor data so the Thor codec doesn't apply)."
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
raise ValueError(f"{p.name}: unknown IDF signature {sig_prefix.hex()}")
|
|
||||||
|
|
||||||
is_histogram = p.suffix.upper() == ".IDFH"
|
|
||||||
md = extract_binary_metadata(buf)
|
|
||||||
|
|
||||||
if is_histogram:
|
|
||||||
intervals = decode_idfh_body(buf)
|
|
||||||
if not intervals:
|
|
||||||
raise ValueError(f"{p.name}: IDFH body decoded no intervals")
|
|
||||||
# Peaks: max across all intervals on each channel (per-channel max
|
|
||||||
# of stored max-magnitudes; sidecar's PPV row carries the same).
|
|
||||||
peak_tran = max((iv.peak_ips("Tran") for iv in intervals), default=0.0)
|
|
||||||
peak_vert = max((iv.peak_ips("Vert") for iv in intervals), default=0.0)
|
|
||||||
peak_long = max((iv.peak_ips("Long") for iv in intervals), default=0.0)
|
|
||||||
# Mic peak in psi — Thor stores per-interval mic ADC counts in the
|
|
||||||
# binary; convert the max count to psi via the per-count factor.
|
|
||||||
mic_peak_count = max((iv.peak_count("MicL") for iv in intervals), default=0)
|
|
||||||
mic_peak_psi = mic_count_to_psi(mic_peak_count) if mic_peak_count else None
|
|
||||||
rep = IdfReport(
|
|
||||||
serial_number=md.serial,
|
|
||||||
event_type="Full Histogram",
|
|
||||||
event_datetime=md.event_datetime,
|
|
||||||
filename=p.name,
|
|
||||||
sample_rate=md.sample_rate,
|
|
||||||
record_time_sec=md.record_time_sec,
|
|
||||||
)
|
|
||||||
peaks = IdfPeaks(
|
|
||||||
transverse_ips=peak_tran,
|
|
||||||
vertical_ips=peak_vert,
|
|
||||||
longitudinal_ips=peak_long,
|
|
||||||
peak_vector_sum_ips=None,
|
|
||||||
mic_pspl_dbl=None, # IDFH binary doesn't carry the dB(L) value
|
|
||||||
mic_pspl_psi=mic_peak_psi,
|
|
||||||
)
|
|
||||||
event = IdfEvent(
|
|
||||||
serial=md.serial or "UNKNOWN",
|
|
||||||
timestamp=md.event_datetime or datetime.datetime(1970, 1, 1),
|
|
||||||
kind="Histogram",
|
|
||||||
filename=p.name,
|
|
||||||
sample_rate=md.sample_rate,
|
|
||||||
record_time_sec=md.record_time_sec,
|
|
||||||
peaks=peaks,
|
|
||||||
report=rep,
|
|
||||||
)
|
|
||||||
return IdfReadResult(
|
|
||||||
event=event,
|
|
||||||
samples={},
|
|
||||||
binary_metadata=md,
|
|
||||||
signature=signature,
|
|
||||||
intervals=intervals,
|
|
||||||
)
|
|
||||||
|
|
||||||
# Waveform path.
|
|
||||||
decoded = _decode_waveform_samples(buf)
|
|
||||||
if decoded is None:
|
|
||||||
raise ValueError(f"{p.name}: waveform body codec failed")
|
|
||||||
|
|
||||||
rep = IdfReport(
|
|
||||||
serial_number=md.serial,
|
|
||||||
event_type="Full Waveform",
|
|
||||||
event_datetime=md.event_datetime,
|
|
||||||
filename=p.name,
|
|
||||||
sample_rate=md.sample_rate,
|
|
||||||
record_time_sec=md.record_time_sec,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _peak_ips(ch: str) -> float:
|
|
||||||
arr = decoded.get(ch, [])
|
|
||||||
return geo_count_to_ips(max((abs(v) for v in arr), default=0))
|
|
||||||
|
|
||||||
# Mic peak psi from binary: max absolute MicL ADC count × 2.14e-6 psi/count.
|
|
||||||
mic_arr = decoded.get("MicL", [])
|
|
||||||
mic_peak_count = max((abs(v) for v in mic_arr), default=0)
|
|
||||||
mic_peak_psi = mic_count_to_psi(mic_peak_count) if mic_peak_count else None
|
|
||||||
|
|
||||||
peaks = IdfPeaks(
|
|
||||||
transverse_ips=_peak_ips("Tran"),
|
|
||||||
vertical_ips=_peak_ips("Vert"),
|
|
||||||
longitudinal_ips=_peak_ips("Long"),
|
|
||||||
# PVS requires aligned per-sample √(T²+V²+L²); leave None — the
|
|
||||||
# sidecar carries it and the bridge picks it up if present.
|
|
||||||
peak_vector_sum_ips=None,
|
|
||||||
mic_pspl_dbl=None, # binary IDFW doesn't carry the dB(L) value;
|
|
||||||
# sidecar .txt fills it via IdfReport.from_dict
|
|
||||||
mic_pspl_psi=mic_peak_psi,
|
|
||||||
)
|
|
||||||
|
|
||||||
event = IdfEvent(
|
|
||||||
serial=md.serial or "UNKNOWN",
|
|
||||||
timestamp=md.event_datetime or datetime.datetime(1970, 1, 1),
|
|
||||||
kind="Waveform",
|
|
||||||
filename=p.name,
|
|
||||||
sample_rate=md.sample_rate,
|
|
||||||
record_time_sec=md.record_time_sec,
|
|
||||||
peaks=peaks,
|
|
||||||
report=rep,
|
|
||||||
)
|
|
||||||
|
|
||||||
return IdfReadResult(
|
|
||||||
event=event,
|
|
||||||
samples=decoded,
|
|
||||||
binary_metadata=md,
|
|
||||||
signature=signature,
|
|
||||||
)
|
|
||||||
@@ -1,323 +0,0 @@
|
|||||||
"""
|
|
||||||
micromate/idf_to_bw_report.py — adapter that projects a parsed Thor IDF
|
|
||||||
report (+ binary metadata + decoded IDFH intervals) into the
|
|
||||||
``bw_report``-shaped dict that :mod:`sfm.report_pdf.gather_report_data`
|
|
||||||
consumes.
|
|
||||||
|
|
||||||
Lets Thor events flow through the existing Series III Event Report PDF
|
|
||||||
pipeline without duplicating the renderer. Thor's report content is
|
|
||||||
~95% the same data shape as BW's; the field names differ but the
|
|
||||||
underlying metrics map 1:1.
|
|
||||||
|
|
||||||
Caveats
|
|
||||||
───────
|
|
||||||
|
|
||||||
- **Mic units** — Thor records ``MicPSPL`` natively in dB(L). This
|
|
||||||
adapter sets ``bw_report.mic.pspl_dbl`` directly; the report
|
|
||||||
renderer recomputes the equivalent psi via its dBL→psi formula.
|
|
||||||
- **Saturation / above-range flags** — Thor doesn't always mark
|
|
||||||
``OORANGE`` the way BW does; we set ``zc_freq_above_range`` only
|
|
||||||
when a `>100` sentinel was preserved in the raw text.
|
|
||||||
- **Per-interval data** — for IDFH events we build ``interval_times``
|
|
||||||
by stepping ``IntervalSize`` from ``HistogramStartTime``; the binary
|
|
||||||
decoder confirms one record per step (882 / 881 / 881 ... across
|
|
||||||
the corpus).
|
|
||||||
- **calibration_by parsing** — Thor's free-form ``Calibration : November
|
|
||||||
22, 2023 by Instantel`` is split on ``" by "`` to extract the
|
|
||||||
calibrator; the date prefix is parsed where possible, otherwise
|
|
||||||
the binary-extracted ``calibration_date`` from
|
|
||||||
:class:`micromate.idf_file.IdfBinaryMetadata` wins.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import datetime
|
|
||||||
import re
|
|
||||||
from typing import Any, Dict, List, Optional
|
|
||||||
|
|
||||||
|
|
||||||
# ─── Helpers ────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
_NUM_RE = re.compile(r"-?\d+(?:\.\d+)?")
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_first_number(s: Optional[str]) -> Optional[float]:
|
|
||||||
"""Pull the first numeric token from a string like ``"0.1500 in/s"``."""
|
|
||||||
if s is None:
|
|
||||||
return None
|
|
||||||
m = _NUM_RE.search(str(s))
|
|
||||||
if not m:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return float(m.group(0))
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_interval_size_s(s: Optional[str]) -> Optional[float]:
|
|
||||||
"""``"60 sec"`` → 60.0, ``"5 min"`` → 300.0, ``"1 hour"`` → 3600."""
|
|
||||||
if s is None:
|
|
||||||
return None
|
|
||||||
num = _parse_first_number(s)
|
|
||||||
if num is None:
|
|
||||||
return None
|
|
||||||
sl = str(s).lower()
|
|
||||||
if "hour" in sl or "hr" in sl:
|
|
||||||
return num * 3600.0
|
|
||||||
if "min" in sl:
|
|
||||||
return num * 60.0
|
|
||||||
return num # default to seconds
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_calibration(text: Optional[str]) -> tuple[Optional[str], Optional[str]]:
|
|
||||||
"""Split ``"November 22, 2023 by Instantel"`` → (ISO date, calibrator).
|
|
||||||
|
|
||||||
Returns ``(None, None)`` if neither half parses.
|
|
||||||
"""
|
|
||||||
if not text:
|
|
||||||
return None, None
|
|
||||||
parts = str(text).split(" by ", 1)
|
|
||||||
date_part = parts[0].strip() if parts else None
|
|
||||||
by_part = parts[1].strip() if len(parts) > 1 else None
|
|
||||||
iso_date: Optional[str] = None
|
|
||||||
if date_part:
|
|
||||||
for fmt in ("%B %d, %Y", "%b %d, %Y", "%Y-%m-%d", "%m/%d/%Y"):
|
|
||||||
try:
|
|
||||||
iso_date = datetime.datetime.strptime(date_part, fmt).date().isoformat()
|
|
||||||
break
|
|
||||||
except ValueError:
|
|
||||||
continue
|
|
||||||
return iso_date, by_part
|
|
||||||
|
|
||||||
|
|
||||||
def _channel_peaks(idf: Dict[str, Any], ch_lc: str) -> Dict[str, Any]:
|
|
||||||
"""Map ``tran_ppv`` / ``tran_zc_freq`` / ... → bw_report.peaks.tran shape."""
|
|
||||||
out: Dict[str, Any] = {}
|
|
||||||
for src, dst in (
|
|
||||||
(f"{ch_lc}_ppv", "ppv_ips"),
|
|
||||||
(f"{ch_lc}_zc_freq", "zc_freq_hz"),
|
|
||||||
(f"{ch_lc}_time_of_peak", "time_of_peak_s"),
|
|
||||||
(f"{ch_lc}_peak_acceleration", "peak_accel_g"),
|
|
||||||
(f"{ch_lc}_peak_displacement", "peak_disp_in"),
|
|
||||||
):
|
|
||||||
v = idf.get(src)
|
|
||||||
if v is not None:
|
|
||||||
out[dst] = v
|
|
||||||
# ZC freq ">100" sentinel: the raw text carries it under the un-typed
|
|
||||||
# key (e.g. ``raw["tran_zc_freq"]`` would be ``">100"``), and our parser
|
|
||||||
# dropped the typed entry. Detect that case and flag.
|
|
||||||
raw_zc = idf.get(f"{ch_lc}_zc_freq")
|
|
||||||
if isinstance(raw_zc, str) and ">" in raw_zc:
|
|
||||||
out["zc_freq_above_range"] = True
|
|
||||||
out.pop("zc_freq_hz", None)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _sensor_check(idf: Dict[str, Any], ch_lc: str) -> Dict[str, Any]:
|
|
||||||
out: Dict[str, Any] = {}
|
|
||||||
fr = idf.get(f"{ch_lc}_test_freq")
|
|
||||||
if fr is not None:
|
|
||||||
out["freq_hz"] = _parse_first_number(fr)
|
|
||||||
rt = idf.get(f"{ch_lc}_test_ratio")
|
|
||||||
if rt is not None:
|
|
||||||
out["ratio"] = _parse_first_number(rt)
|
|
||||||
am = idf.get(f"{ch_lc}_test_amplitude")
|
|
||||||
if am is not None:
|
|
||||||
out["amplitude_mv"] = _parse_first_number(am)
|
|
||||||
res = idf.get(f"{ch_lc}_test_results")
|
|
||||||
if res is not None:
|
|
||||||
out["result"] = str(res).strip()
|
|
||||||
return {k: v for k, v in out.items() if v is not None}
|
|
||||||
|
|
||||||
|
|
||||||
def _interval_times(idf: Dict[str, Any], n_intervals: Optional[int]) -> List[str]:
|
|
||||||
"""Synthesise per-interval timestamps from start + interval_size × k.
|
|
||||||
|
|
||||||
Returns ``[]`` when start time or interval size is unknown.
|
|
||||||
"""
|
|
||||||
if not n_intervals:
|
|
||||||
return []
|
|
||||||
start_date = idf.get("histogram_start_date") or idf.get("event_date")
|
|
||||||
start_time = idf.get("histogram_start_time") or idf.get("event_time")
|
|
||||||
iv_str = idf.get("interval_size")
|
|
||||||
iv_s = _parse_interval_size_s(iv_str)
|
|
||||||
if not (start_date and start_time and iv_s):
|
|
||||||
return []
|
|
||||||
try:
|
|
||||||
t0 = datetime.datetime.strptime(f"{start_date} {start_time}", "%Y-%m-%d %H:%M:%S")
|
|
||||||
except ValueError:
|
|
||||||
return []
|
|
||||||
out = []
|
|
||||||
for k in range(int(n_intervals)):
|
|
||||||
t = t0 + datetime.timedelta(seconds=iv_s * (k + 1))
|
|
||||||
out.append(t.isoformat())
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
# ─── Top-level adapter ──────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def build_bw_report_from_idf(
|
|
||||||
idf_report: Dict[str, Any],
|
|
||||||
*,
|
|
||||||
binary_md=None,
|
|
||||||
intervals: Optional[list] = None,
|
|
||||||
is_histogram: Optional[bool] = None,
|
|
||||||
) -> Dict[str, Any]:
|
|
||||||
"""Project a parsed IDF report dict (and optional binary metadata +
|
|
||||||
decoded IDFH intervals) into the BW report sidecar shape.
|
|
||||||
|
|
||||||
The returned dict is structurally identical to what
|
|
||||||
``minimateplus.event_file_io._bw_report_to_dict`` produces from a
|
|
||||||
real BW ASCII report — it can be assigned to
|
|
||||||
``sidecar["bw_report"]`` and consumed verbatim by
|
|
||||||
``sfm.report_pdf.gather_report_data``.
|
|
||||||
|
|
||||||
``intervals`` is the list of :class:`micromate.idf_file.IdfhInterval`
|
|
||||||
objects from :func:`micromate.idf_file.decode_idfh_body`; only used
|
|
||||||
for histogram events to derive accurate ``interval_times``.
|
|
||||||
"""
|
|
||||||
if is_histogram is None:
|
|
||||||
et = str(idf_report.get("event_type", ""))
|
|
||||||
is_histogram = et.lower().startswith("full histogram")
|
|
||||||
|
|
||||||
# ── Trigger / recording / device ─────────────────────────────────────
|
|
||||||
trigger_channel = idf_report.get("trigger")
|
|
||||||
trigger_level = _parse_first_number(idf_report.get("geo_trigger_level"))
|
|
||||||
geo_range_ips = _parse_first_number(idf_report.get("geo_range"))
|
|
||||||
|
|
||||||
cal_iso, cal_by = _parse_calibration(idf_report.get("calibration"))
|
|
||||||
# Prefer the binary-extracted calibration_date when our text parse fell
|
|
||||||
# through; the binary date is unambiguous.
|
|
||||||
if cal_iso is None and binary_md is not None and binary_md.calibration_date:
|
|
||||||
cal_iso = binary_md.calibration_date.isoformat()
|
|
||||||
|
|
||||||
# ── Histogram fields ────────────────────────────────────────────────
|
|
||||||
hist_block: Dict[str, Any] = {
|
|
||||||
"start": None, "stop": None, "n_intervals": None,
|
|
||||||
"interval_size": None, "interval_size_s": None,
|
|
||||||
"channel_peak_when": {},
|
|
||||||
}
|
|
||||||
if is_histogram:
|
|
||||||
sd = idf_report.get("histogram_start_date")
|
|
||||||
st = idf_report.get("histogram_start_time")
|
|
||||||
if sd and st:
|
|
||||||
try:
|
|
||||||
hist_block["start"] = datetime.datetime.strptime(
|
|
||||||
f"{sd} {st}", "%Y-%m-%d %H:%M:%S"
|
|
||||||
).isoformat()
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
ed = idf_report.get("histogram_stop_date")
|
|
||||||
et_ = idf_report.get("histogram_stop_time")
|
|
||||||
if ed and et_:
|
|
||||||
try:
|
|
||||||
hist_block["stop"] = datetime.datetime.strptime(
|
|
||||||
f"{ed} {et_}", "%Y-%m-%d %H:%M:%S"
|
|
||||||
).isoformat()
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
n_raw = idf_report.get("number_of_intervals")
|
|
||||||
if n_raw is not None:
|
|
||||||
try:
|
|
||||||
# Thor reports a float like "81.04"; round to int (the BW
|
|
||||||
# report uses an int for the column).
|
|
||||||
hist_block["n_intervals"] = int(float(str(n_raw)))
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
# When the binary decoder gave us the actual interval count, prefer it.
|
|
||||||
if intervals is not None:
|
|
||||||
hist_block["n_intervals"] = len(intervals)
|
|
||||||
hist_block["interval_size"] = idf_report.get("interval_size")
|
|
||||||
hist_block["interval_size_s"] = _parse_interval_size_s(idf_report.get("interval_size"))
|
|
||||||
# interval_times derived from start+step (the BW report uses the
|
|
||||||
# exact strings; we match its representation).
|
|
||||||
times = _interval_times(idf_report, hist_block["n_intervals"])
|
|
||||||
# Per-channel peak when (absolute date+time at which the channel's
|
|
||||||
# peak occurred over the histogram run). Thor splits this into
|
|
||||||
# ``TranPeakDate`` / ``TranPeakTime`` etc.
|
|
||||||
peak_when: Dict[str, str] = {}
|
|
||||||
for ch_label, ch_lc in (("Tran", "tran"), ("Vert", "vert"), ("Long", "long"), ("MicL", "mic")):
|
|
||||||
d = idf_report.get(f"{ch_lc}_peak_date")
|
|
||||||
t = idf_report.get(f"{ch_lc}_peak_time")
|
|
||||||
if d and t:
|
|
||||||
try:
|
|
||||||
peak_when[ch_label] = datetime.datetime.strptime(
|
|
||||||
f"{d} {t}", "%Y-%m-%d %H:%M:%S"
|
|
||||||
).isoformat()
|
|
||||||
except ValueError:
|
|
||||||
continue
|
|
||||||
if peak_when:
|
|
||||||
hist_block["channel_peak_when"] = peak_when
|
|
||||||
|
|
||||||
# ── Mic block ────────────────────────────────────────────────────────
|
|
||||||
mic_block = {
|
|
||||||
"weighting": "L", # Thor mic is ISEE Linear
|
|
||||||
"pspl_dbl": idf_report.get("mic_ppv"), # the dB(L) float
|
|
||||||
"pspl_saturated": False,
|
|
||||||
"zc_freq_hz": idf_report.get("mic_zc_freq"),
|
|
||||||
"zc_freq_above_range": isinstance(idf_report.get("mic_zc_freq"), str)
|
|
||||||
and ">" in str(idf_report.get("mic_zc_freq")),
|
|
||||||
"time_of_peak_s": idf_report.get("mic_time_of_peak"),
|
|
||||||
}
|
|
||||||
if mic_block["zc_freq_above_range"]:
|
|
||||||
mic_block["zc_freq_hz"] = None
|
|
||||||
|
|
||||||
# ── Peaks ────────────────────────────────────────────────────────────
|
|
||||||
vs_block = {
|
|
||||||
"ips": idf_report.get("peak_vector_sum"),
|
|
||||||
"time_s": _parse_first_number(idf_report.get("peak_vector_sum_time_sum")),
|
|
||||||
"when": None,
|
|
||||||
"saturated": False,
|
|
||||||
}
|
|
||||||
if is_histogram:
|
|
||||||
# PVS absolute date+time, when present.
|
|
||||||
vs_d = idf_report.get("peak_vector_sum_date")
|
|
||||||
vs_t = idf_report.get("peak_vector_sum_time")
|
|
||||||
if vs_d and vs_t:
|
|
||||||
try:
|
|
||||||
vs_block["when"] = datetime.datetime.strptime(
|
|
||||||
f"{vs_d} {vs_t}", "%Y-%m-%d %H:%M:%S"
|
|
||||||
).isoformat()
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
|
|
||||||
return {
|
|
||||||
"available": True,
|
|
||||||
"event_type": idf_report.get("event_type"),
|
|
||||||
"version": idf_report.get("version"),
|
|
||||||
"trigger": {
|
|
||||||
"channel": trigger_channel,
|
|
||||||
"geo_level_ips": trigger_level,
|
|
||||||
},
|
|
||||||
"recording": {
|
|
||||||
"sample_rate_sps": idf_report.get("sample_rate"),
|
|
||||||
"record_time_s": idf_report.get("record_time_sec"),
|
|
||||||
"pretrig_s": idf_report.get("pre_trigger_sec"),
|
|
||||||
"stop_mode": idf_report.get("record_stop_mode"),
|
|
||||||
"geo_range_ips": geo_range_ips,
|
|
||||||
"units": idf_report.get("units"),
|
|
||||||
},
|
|
||||||
"device": {
|
|
||||||
"battery_volts": idf_report.get("battery_volts"),
|
|
||||||
"calibration_date": cal_iso,
|
|
||||||
"calibration_by": cal_by,
|
|
||||||
},
|
|
||||||
"peaks": {
|
|
||||||
"tran": _channel_peaks(idf_report, "tran"),
|
|
||||||
"vert": _channel_peaks(idf_report, "vert"),
|
|
||||||
"long": _channel_peaks(idf_report, "long"),
|
|
||||||
"vector_sum": vs_block,
|
|
||||||
},
|
|
||||||
"mic": mic_block,
|
|
||||||
"sensor_check": {
|
|
||||||
"tran": _sensor_check(idf_report, "tran"),
|
|
||||||
"vert": _sensor_check(idf_report, "vert"),
|
|
||||||
"long": _sensor_check(idf_report, "long"),
|
|
||||||
"mic": _sensor_check(idf_report, "mic"),
|
|
||||||
},
|
|
||||||
"histogram": hist_block,
|
|
||||||
"monitor_log": [],
|
|
||||||
"pc_sw_version": None,
|
|
||||||
}
|
|
||||||
@@ -1,398 +0,0 @@
|
|||||||
"""
|
|
||||||
Micromate (Series IV / Thor) native data models.
|
|
||||||
|
|
||||||
These are the right-shaped dataclasses for Thor data — Thor measures
|
|
||||||
the microphone in dB(L) directly, so this model carries
|
|
||||||
``mic_pspl_dbl`` rather than the pseudo-``psi`` shoehorn that
|
|
||||||
``minimateplus.PeakValues`` uses for Series III BW data.
|
|
||||||
|
|
||||||
The ingest pipeline today goes:
|
|
||||||
|
|
||||||
.IDFW.txt → parse_idf_report() → dict
|
|
||||||
dict → IdfEvent.from_report() → IdfEvent (typed)
|
|
||||||
IdfEvent → IdfEvent.to_minimateplus_event() → shape DB / sidecar
|
|
||||||
machinery expects
|
|
||||||
|
|
||||||
The ``to_minimateplus_event()`` bridge is a temporary boundary — when we
|
|
||||||
crack the binary IDF codec and have richer per-event data to store, the
|
|
||||||
DB schema will grow Series-IV-specific columns and the bridge will
|
|
||||||
shrink or disappear.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import datetime
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from typing import Any, Dict, Optional, Tuple
|
|
||||||
|
|
||||||
|
|
||||||
# ── IdfReport ─────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class IdfReport:
|
|
||||||
"""Typed wrapper around the dict returned by ``parse_idf_report``.
|
|
||||||
|
|
||||||
All fields optional — Thor's exporter is permissive and some IDF .txt
|
|
||||||
files (especially histograms) omit fields that waveform sidecars
|
|
||||||
include. Use ``.raw`` for any field this dataclass hasn't surfaced
|
|
||||||
yet (the parser keeps every recognised key in the raw dict).
|
|
||||||
"""
|
|
||||||
|
|
||||||
# Identity / kind
|
|
||||||
serial_number: Optional[str] = None
|
|
||||||
event_type: Optional[str] = None # "Full Waveform" | "Full Histogram"
|
|
||||||
event_datetime: Optional[datetime.datetime] = None
|
|
||||||
filename: Optional[str] = None # echoed by Thor's exporter
|
|
||||||
|
|
||||||
# Sampling / timing
|
|
||||||
sample_rate: Optional[int] = None # samples/sec
|
|
||||||
record_time_sec: Optional[float] = None
|
|
||||||
pre_trigger_sec: Optional[float] = None
|
|
||||||
|
|
||||||
# Geophone peaks (in/s)
|
|
||||||
tran_ppv: Optional[float] = None
|
|
||||||
vert_ppv: Optional[float] = None
|
|
||||||
long_ppv: Optional[float] = None
|
|
||||||
peak_vector_sum: Optional[float] = None
|
|
||||||
|
|
||||||
# Microphone — Thor's native unit is dB(L), NOT psi.
|
|
||||||
mic_pspl_dbl: Optional[float] = None
|
|
||||||
|
|
||||||
# Zero-crossing frequencies (Hz)
|
|
||||||
tran_zc_freq: Optional[float] = None
|
|
||||||
vert_zc_freq: Optional[float] = None
|
|
||||||
long_zc_freq: Optional[float] = None
|
|
||||||
mic_zc_freq: Optional[float] = None
|
|
||||||
|
|
||||||
# Per-channel time of peak (sec, since event start)
|
|
||||||
tran_time_of_peak: Optional[float] = None
|
|
||||||
vert_time_of_peak: Optional[float] = None
|
|
||||||
long_time_of_peak: Optional[float] = None
|
|
||||||
mic_time_of_peak: Optional[float] = None
|
|
||||||
|
|
||||||
# Derived per-channel motion
|
|
||||||
tran_peak_acceleration: Optional[float] = None # g
|
|
||||||
vert_peak_acceleration: Optional[float] = None
|
|
||||||
long_peak_acceleration: Optional[float] = None
|
|
||||||
tran_peak_displacement: Optional[float] = None # in
|
|
||||||
vert_peak_displacement: Optional[float] = None
|
|
||||||
long_peak_displacement: Optional[float] = None
|
|
||||||
|
|
||||||
# Operator-supplied strings (Thor's TitleString1..4 → semantic slots)
|
|
||||||
project: Optional[str] = None # TitleString1
|
|
||||||
client: Optional[str] = None # TitleString2
|
|
||||||
operator: Optional[str] = None # TitleString3
|
|
||||||
notes: Optional[str] = None # TitleString4 / PostEventNote
|
|
||||||
setup: Optional[str] = None # setup file name
|
|
||||||
|
|
||||||
# Sensor self-check results
|
|
||||||
tran_test_passed: Optional[bool] = None
|
|
||||||
vert_test_passed: Optional[bool] = None
|
|
||||||
long_test_passed: Optional[bool] = None
|
|
||||||
mic_test_passed: Optional[bool] = None
|
|
||||||
|
|
||||||
# Device-fixed metadata
|
|
||||||
firmware_version: Optional[str] = None
|
|
||||||
calibration_text: Optional[str] = None
|
|
||||||
battery_volts: Optional[float] = None
|
|
||||||
|
|
||||||
# Original parser dict — preserves every recognised key (including
|
|
||||||
# raw unit-suffixed strings) for forward-compatible field access.
|
|
||||||
raw: Dict[str, Any] = field(default_factory=dict, repr=False)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def from_dict(cls, d: Dict[str, Any]) -> "IdfReport":
|
|
||||||
"""Build an IdfReport from the dict returned by ``parse_idf_report``."""
|
|
||||||
ed = d.get("event_datetime")
|
|
||||||
if isinstance(ed, str):
|
|
||||||
try:
|
|
||||||
ed = datetime.datetime.fromisoformat(ed)
|
|
||||||
except ValueError:
|
|
||||||
ed = None
|
|
||||||
|
|
||||||
return cls(
|
|
||||||
serial_number = d.get("serial_number"),
|
|
||||||
event_type = d.get("event_type"),
|
|
||||||
event_datetime = ed if isinstance(ed, datetime.datetime) else None,
|
|
||||||
filename = d.get("filename"),
|
|
||||||
sample_rate = d.get("sample_rate"),
|
|
||||||
record_time_sec = d.get("record_time_sec"),
|
|
||||||
pre_trigger_sec = d.get("pre_trigger_sec"),
|
|
||||||
tran_ppv = d.get("tran_ppv"),
|
|
||||||
vert_ppv = d.get("vert_ppv"),
|
|
||||||
long_ppv = d.get("long_ppv"),
|
|
||||||
peak_vector_sum = d.get("peak_vector_sum"),
|
|
||||||
mic_pspl_dbl = d.get("mic_ppv"), # parser names it mic_ppv (legacy)
|
|
||||||
tran_zc_freq = d.get("tran_zc_freq"),
|
|
||||||
vert_zc_freq = d.get("vert_zc_freq"),
|
|
||||||
long_zc_freq = d.get("long_zc_freq"),
|
|
||||||
mic_zc_freq = d.get("mic_zc_freq"),
|
|
||||||
tran_time_of_peak = d.get("tran_time_of_peak"),
|
|
||||||
vert_time_of_peak = d.get("vert_time_of_peak"),
|
|
||||||
long_time_of_peak = d.get("long_time_of_peak"),
|
|
||||||
mic_time_of_peak = d.get("mic_time_of_peak"),
|
|
||||||
tran_peak_acceleration = d.get("tran_peak_acceleration"),
|
|
||||||
vert_peak_acceleration = d.get("vert_peak_acceleration"),
|
|
||||||
long_peak_acceleration = d.get("long_peak_acceleration"),
|
|
||||||
tran_peak_displacement = d.get("tran_peak_displacement"),
|
|
||||||
vert_peak_displacement = d.get("vert_peak_displacement"),
|
|
||||||
long_peak_displacement = d.get("long_peak_displacement"),
|
|
||||||
project = d.get("project"),
|
|
||||||
client = d.get("client"),
|
|
||||||
operator = d.get("operator"),
|
|
||||||
notes = d.get("notes"),
|
|
||||||
setup = d.get("setup"),
|
|
||||||
tran_test_passed = d.get("tran_test_passed"),
|
|
||||||
vert_test_passed = d.get("vert_test_passed"),
|
|
||||||
long_test_passed = d.get("long_test_passed"),
|
|
||||||
mic_test_passed = d.get("mic_test_passed"),
|
|
||||||
firmware_version = d.get("version"),
|
|
||||||
calibration_text = d.get("calibration_text"),
|
|
||||||
battery_volts = d.get("battery_volts"),
|
|
||||||
raw = d,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
# ── IdfPeaks / IdfProjectInfo / IdfSensorCheck (narrow grouping types) ───────
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class IdfPeaks:
|
|
||||||
"""Geophone + mic peak values for one Thor event. Native Thor units.
|
|
||||||
|
|
||||||
Thor stores the mic peak in two parallel forms — ``mic_pspl_dbl`` is
|
|
||||||
what the sidecar's top-level ``MicPSPL`` header field carries (dB(L)),
|
|
||||||
used in the report header. ``mic_pspl_psi`` is the psi value derived
|
|
||||||
either from the IDFW sample table / IDFH interval column 9, or from
|
|
||||||
the binary mic counts (~2.14e-6 psi/count). Needed because the
|
|
||||||
BW-shaped ``PeakValues.micl`` consumed by ``event_hdf5.write_event_hdf5``
|
|
||||||
expects psi — feeding it dB(L) makes the h5 mic-chart scale factor
|
|
||||||
blow up.
|
|
||||||
"""
|
|
||||||
transverse_ips: Optional[float] = None # in/s
|
|
||||||
vertical_ips: Optional[float] = None # in/s
|
|
||||||
longitudinal_ips: Optional[float] = None # in/s
|
|
||||||
peak_vector_sum_ips: Optional[float] = None # in/s
|
|
||||||
mic_pspl_dbl: Optional[float] = None # dB(L)
|
|
||||||
mic_pspl_psi: Optional[float] = None # psi
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class IdfProjectInfo:
|
|
||||||
"""Operator-supplied strings from Thor's TitleString1..4."""
|
|
||||||
project: Optional[str] = None
|
|
||||||
client: Optional[str] = None
|
|
||||||
operator: Optional[str] = None
|
|
||||||
notes: Optional[str] = None
|
|
||||||
setup: Optional[str] = None
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class IdfSensorCheck:
|
|
||||||
"""Per-channel pass/fail from Thor's self-test."""
|
|
||||||
tran: Optional[bool] = None
|
|
||||||
vert: Optional[bool] = None
|
|
||||||
long: Optional[bool] = None
|
|
||||||
mic: Optional[bool] = None
|
|
||||||
|
|
||||||
|
|
||||||
# ── IdfEvent ─────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class IdfEvent:
|
|
||||||
"""A single Thor / Micromate Series IV event.
|
|
||||||
|
|
||||||
Built from a parsed .IDFW.txt or .IDFH.txt sidecar via
|
|
||||||
``IdfEvent.from_report()``. The filename is the authoritative
|
|
||||||
source for serial + timestamp + kind; the .txt provides
|
|
||||||
device-authoritative peak values, frequencies, project strings,
|
|
||||||
sensor self-check, firmware, calibration.
|
|
||||||
"""
|
|
||||||
|
|
||||||
# Identity
|
|
||||||
serial: str
|
|
||||||
timestamp: datetime.datetime
|
|
||||||
kind: str # "Waveform" | "Histogram"
|
|
||||||
filename: str # device-native binary filename, e.g. "UM11719_20231219163444.IDFW"
|
|
||||||
|
|
||||||
# Sampling / timing
|
|
||||||
sample_rate: Optional[int] = None
|
|
||||||
record_time_sec: Optional[float] = None
|
|
||||||
pre_trigger_sec: Optional[float] = None
|
|
||||||
|
|
||||||
# Peaks
|
|
||||||
peaks: IdfPeaks = field(default_factory=IdfPeaks)
|
|
||||||
|
|
||||||
# Per-channel frequencies (Hz)
|
|
||||||
tran_zc_freq: Optional[float] = None
|
|
||||||
vert_zc_freq: Optional[float] = None
|
|
||||||
long_zc_freq: Optional[float] = None
|
|
||||||
mic_zc_freq: Optional[float] = None
|
|
||||||
|
|
||||||
# Project strings
|
|
||||||
project_info: IdfProjectInfo = field(default_factory=IdfProjectInfo)
|
|
||||||
|
|
||||||
# Sensor self-check
|
|
||||||
sensor_check: IdfSensorCheck = field(default_factory=IdfSensorCheck)
|
|
||||||
|
|
||||||
# Device-fixed
|
|
||||||
firmware_version: Optional[str] = None
|
|
||||||
calibration_text: Optional[str] = None
|
|
||||||
battery_volts: Optional[float] = None
|
|
||||||
|
|
||||||
# The full parsed report — preserves anything not surfaced as a typed field
|
|
||||||
report: IdfReport = field(default_factory=IdfReport)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def from_report(
|
|
||||||
cls,
|
|
||||||
report: Any,
|
|
||||||
filename: str,
|
|
||||||
) -> "IdfEvent":
|
|
||||||
"""Build an IdfEvent from a parsed report (dict or IdfReport) and
|
|
||||||
the device-native binary filename.
|
|
||||||
|
|
||||||
The filename is authoritative for serial + timestamp + kind:
|
|
||||||
Thor's filenames are literal ``<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>``
|
|
||||||
and the device's own clock is the canonical event timestamp.
|
|
||||||
If the report carries an ``event_datetime`` that differs from
|
|
||||||
what's in the filename, the report wins (it has finer-grained
|
|
||||||
device-reported time-of-trigger semantics).
|
|
||||||
"""
|
|
||||||
from .idf_ascii_report import parse_event_filename
|
|
||||||
|
|
||||||
# Normalise input to IdfReport
|
|
||||||
if isinstance(report, IdfReport):
|
|
||||||
rep = report
|
|
||||||
elif isinstance(report, dict):
|
|
||||||
rep = IdfReport.from_dict(report)
|
|
||||||
else:
|
|
||||||
raise TypeError(
|
|
||||||
f"report must be IdfReport or dict; got {type(report).__name__}"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Filename → (serial, timestamp, kind). Required — fall back to
|
|
||||||
# report-supplied values only if filename parsing fails.
|
|
||||||
parsed = parse_event_filename(filename)
|
|
||||||
if parsed is not None:
|
|
||||||
fn_serial, fn_ts, fn_kind = parsed
|
|
||||||
kind = "Histogram" if fn_kind == "IDFH" else "Waveform"
|
|
||||||
else:
|
|
||||||
fn_serial = rep.serial_number or "UNKNOWN"
|
|
||||||
fn_ts = rep.event_datetime or datetime.datetime(1970, 1, 1)
|
|
||||||
kind = "Waveform" if (rep.event_type or "").lower().startswith("full waveform") else "Histogram"
|
|
||||||
|
|
||||||
# Prefer report's event_datetime (device-authoritative) over the filename.
|
|
||||||
ts = rep.event_datetime or fn_ts
|
|
||||||
serial = rep.serial_number or fn_serial
|
|
||||||
|
|
||||||
return cls(
|
|
||||||
serial=serial,
|
|
||||||
timestamp=ts,
|
|
||||||
kind=kind,
|
|
||||||
filename=filename,
|
|
||||||
sample_rate=rep.sample_rate,
|
|
||||||
record_time_sec=rep.record_time_sec,
|
|
||||||
pre_trigger_sec=rep.pre_trigger_sec,
|
|
||||||
peaks=IdfPeaks(
|
|
||||||
transverse_ips = rep.tran_ppv,
|
|
||||||
vertical_ips = rep.vert_ppv,
|
|
||||||
longitudinal_ips = rep.long_ppv,
|
|
||||||
peak_vector_sum_ips = rep.peak_vector_sum,
|
|
||||||
mic_pspl_dbl = rep.mic_pspl_dbl,
|
|
||||||
),
|
|
||||||
tran_zc_freq=rep.tran_zc_freq,
|
|
||||||
vert_zc_freq=rep.vert_zc_freq,
|
|
||||||
long_zc_freq=rep.long_zc_freq,
|
|
||||||
mic_zc_freq=rep.mic_zc_freq,
|
|
||||||
project_info=IdfProjectInfo(
|
|
||||||
project=rep.project,
|
|
||||||
client=rep.client,
|
|
||||||
operator=rep.operator,
|
|
||||||
notes=rep.notes,
|
|
||||||
setup=rep.setup,
|
|
||||||
),
|
|
||||||
sensor_check=IdfSensorCheck(
|
|
||||||
tran=rep.tran_test_passed,
|
|
||||||
vert=rep.vert_test_passed,
|
|
||||||
long=rep.long_test_passed,
|
|
||||||
mic=rep.mic_test_passed,
|
|
||||||
),
|
|
||||||
firmware_version=rep.firmware_version,
|
|
||||||
calibration_text=rep.calibration_text,
|
|
||||||
battery_volts=rep.battery_volts,
|
|
||||||
report=rep,
|
|
||||||
)
|
|
||||||
|
|
||||||
# ── Bridge to minimateplus shape (for the existing DB / sidecar paths) ──
|
|
||||||
|
|
||||||
def to_minimateplus_event(self, waveform_key: bytes) -> Any:
|
|
||||||
"""Project this Thor event into the shape ``minimateplus.Event``
|
|
||||||
carries, so it can flow through the existing
|
|
||||||
``SeismoDb.insert_events()`` and ``event_to_sidecar_dict()``
|
|
||||||
machinery without those code paths needing to know about Thor.
|
|
||||||
|
|
||||||
Caveats of the bridge:
|
|
||||||
- ``PeakValues.micl`` carries the mic peak in **psi** (matching
|
|
||||||
BW's convention) — set from :attr:`IdfPeaks.mic_pspl_psi`,
|
|
||||||
with a dB(L)→psi fallback when only the dB(L) value is
|
|
||||||
available. This is what the h5 writer's mic-scale-factor
|
|
||||||
logic needs. The dB(L) value still flows through
|
|
||||||
``bw_report.mic.pspl_dbl`` (set by the
|
|
||||||
``idf_to_bw_report`` adapter) and the renderer reads it
|
|
||||||
from there for the report header.
|
|
||||||
- Many Thor-specific fields (Peak Acceleration / Displacement,
|
|
||||||
sensor self-check, calibration) don't have a slot in
|
|
||||||
``Event``. The full IdfReport is preserved on the
|
|
||||||
``.sfm.json`` sidecar under ``extensions.idf_report`` via
|
|
||||||
``save_imported_idf`` — that's the source of truth for them.
|
|
||||||
"""
|
|
||||||
from minimateplus.models import (
|
|
||||||
Event, PeakValues, ProjectInfo, Timestamp,
|
|
||||||
)
|
|
||||||
|
|
||||||
ts_obj = Timestamp(
|
|
||||||
raw=bytes(9),
|
|
||||||
flag=0,
|
|
||||||
year=self.timestamp.year,
|
|
||||||
unknown_byte=0,
|
|
||||||
month=self.timestamp.month,
|
|
||||||
day=self.timestamp.day,
|
|
||||||
hour=self.timestamp.hour,
|
|
||||||
minute=self.timestamp.minute,
|
|
||||||
second=self.timestamp.second,
|
|
||||||
)
|
|
||||||
# Resolve mic peak as psi. Priority: binary-derived mic_pspl_psi
|
|
||||||
# (set by read_idf_file) > dB(L)→psi fallback via standard formula
|
|
||||||
# (psi = 2.9e-9 × 10^(dBL/20)) > None.
|
|
||||||
mic_psi = self.peaks.mic_pspl_psi
|
|
||||||
if mic_psi is None and self.peaks.mic_pspl_dbl is not None:
|
|
||||||
mic_psi = 2.9e-9 * (10.0 ** (self.peaks.mic_pspl_dbl / 20.0))
|
|
||||||
pv = PeakValues(
|
|
||||||
tran=self.peaks.transverse_ips,
|
|
||||||
vert=self.peaks.vertical_ips,
|
|
||||||
long=self.peaks.longitudinal_ips,
|
|
||||||
micl=mic_psi, # psi, matching BW's convention (h5 scaling depends on this)
|
|
||||||
peak_vector_sum=self.peaks.peak_vector_sum_ips,
|
|
||||||
)
|
|
||||||
pi = ProjectInfo(
|
|
||||||
setup_name=self.project_info.setup,
|
|
||||||
project=self.project_info.project,
|
|
||||||
client=self.project_info.client,
|
|
||||||
operator=self.project_info.operator,
|
|
||||||
sensor_location=None, # Thor folds location into project string
|
|
||||||
notes=self.project_info.notes,
|
|
||||||
)
|
|
||||||
ev = Event(
|
|
||||||
index=0,
|
|
||||||
timestamp=ts_obj,
|
|
||||||
sample_rate=self.sample_rate,
|
|
||||||
peak_values=pv,
|
|
||||||
project_info=pi,
|
|
||||||
record_type=self.kind,
|
|
||||||
rectime_seconds=self.record_time_sec,
|
|
||||||
)
|
|
||||||
ev._waveform_key = waveform_key
|
|
||||||
return ev
|
|
||||||
@@ -21,15 +21,7 @@ Typical usage (TCP / modem):
|
|||||||
|
|
||||||
from .client import MiniMateClient
|
from .client import MiniMateClient
|
||||||
from .models import DeviceInfo, Event, MonitorLogEntry
|
from .models import DeviceInfo, Event, MonitorLogEntry
|
||||||
from .transport import CapturingTransport, SerialTransport, TcpTransport
|
from .transport import SerialTransport, TcpTransport
|
||||||
|
|
||||||
__version__ = "0.1.0"
|
__version__ = "0.1.0"
|
||||||
__all__ = [
|
__all__ = ["MiniMateClient", "DeviceInfo", "Event", "MonitorLogEntry", "SerialTransport", "TcpTransport"]
|
||||||
"MiniMateClient",
|
|
||||||
"DeviceInfo",
|
|
||||||
"Event",
|
|
||||||
"MonitorLogEntry",
|
|
||||||
"SerialTransport",
|
|
||||||
"TcpTransport",
|
|
||||||
"CapturingTransport",
|
|
||||||
]
|
|
||||||
|
|||||||
+31
-155
@@ -552,105 +552,6 @@ def classify_frame(frame: S3Frame) -> str:
|
|||||||
|
|
||||||
# ── Waveform file writer ───────────────────────────────────────────────────────────
|
# ── Waveform file writer ───────────────────────────────────────────────────────────
|
||||||
|
|
||||||
def extract_body_bytes(a5_frames):
|
|
||||||
"""Reconstruct the Blastware-file body bytes from a list of A5 frames.
|
|
||||||
|
|
||||||
Returns ``(strt, body, footer)`` where:
|
|
||||||
|
|
||||||
- ``strt`` is the 21-byte STRT record from the probe frame (or a fallback
|
|
||||||
record built from minimal event metadata if STRT is missing).
|
|
||||||
- ``body`` is the variable-length sample-data section (between STRT and
|
|
||||||
the 26-byte file footer). Empty if no frames decode.
|
|
||||||
- ``footer`` is the 26-byte file footer.
|
|
||||||
|
|
||||||
This is the same body-construction algorithm used by :func:`write_blastware_file`
|
|
||||||
— refactored out so the body decoder (``waveform_codec.decode_waveform_v2``)
|
|
||||||
can consume the same bytes without re-implementing the frame-walking logic.
|
|
||||||
|
|
||||||
Returns ``(b"", b"", b"")`` if *a5_frames* is empty.
|
|
||||||
"""
|
|
||||||
if not a5_frames:
|
|
||||||
return (b"", b"", b"")
|
|
||||||
|
|
||||||
# ── Extract STRT record from probe frame ─────────────────────────────────
|
|
||||||
w0_raw = bytes(a5_frames[0].data[7:])
|
|
||||||
w0_stripped = _strip_inner_frame_dles(w0_raw)
|
|
||||||
strt_pos_stripped = w0_stripped.find(b"STRT")
|
|
||||||
|
|
||||||
if strt_pos_stripped >= 0:
|
|
||||||
strt = bytes(w0_stripped[strt_pos_stripped : strt_pos_stripped + 21])
|
|
||||||
|
|
||||||
# Walk raw bytes to find the raw-domain end of the STRT (= body start).
|
|
||||||
target_stripped = strt_pos_stripped + 21
|
|
||||||
stripped_so_far = 0
|
|
||||||
raw_i = 0
|
|
||||||
while stripped_so_far < target_stripped and raw_i < len(w0_raw):
|
|
||||||
if (w0_raw[raw_i] == 0x10
|
|
||||||
and raw_i + 1 < len(w0_raw)
|
|
||||||
and w0_raw[raw_i + 1] in {0x02, 0x03, 0x04}):
|
|
||||||
raw_i += 2
|
|
||||||
else:
|
|
||||||
raw_i += 1
|
|
||||||
stripped_so_far += 1
|
|
||||||
probe_skip = 7 + raw_i
|
|
||||||
else:
|
|
||||||
strt = b"STRT" + b"\xff\xfe" + bytes(14) + b"\x00"
|
|
||||||
probe_skip = 7 + 21
|
|
||||||
|
|
||||||
if len(strt) != 21:
|
|
||||||
return (b"", b"", b"")
|
|
||||||
|
|
||||||
# Separate terminator from data frames.
|
|
||||||
term_idx: Optional[int] = None
|
|
||||||
if a5_frames and a5_frames[-1].page_key != 0x0010:
|
|
||||||
term_idx = len(a5_frames) - 1
|
|
||||||
|
|
||||||
if term_idx is not None:
|
|
||||||
body_frames = a5_frames[:term_idx]
|
|
||||||
term_frame = a5_frames[term_idx]
|
|
||||||
else:
|
|
||||||
body_frames = a5_frames
|
|
||||||
term_frame = None
|
|
||||||
|
|
||||||
all_bytes = bytearray()
|
|
||||||
for fi, frame in enumerate(body_frames):
|
|
||||||
if fi == 0:
|
|
||||||
skip = probe_skip
|
|
||||||
elif fi in (1, 2):
|
|
||||||
skip = 13 # metadata pages
|
|
||||||
else:
|
|
||||||
skip = 12 # sample chunks
|
|
||||||
all_bytes.extend(_frame_body_bytes(frame, skip))
|
|
||||||
|
|
||||||
if term_frame is not None:
|
|
||||||
all_bytes.extend(_frame_body_bytes(term_frame, 11))
|
|
||||||
|
|
||||||
# Find the first valid `0e 08` footer marker.
|
|
||||||
footer_pos = -1
|
|
||||||
pos = 0
|
|
||||||
while True:
|
|
||||||
pos = bytes(all_bytes).find(b"\x0e\x08", pos)
|
|
||||||
if pos < 0 or pos + 26 > len(all_bytes):
|
|
||||||
break
|
|
||||||
yr = (all_bytes[pos + 4] << 8) | all_bytes[pos + 5]
|
|
||||||
if 2015 <= yr <= 2050:
|
|
||||||
footer_pos = pos
|
|
||||||
break
|
|
||||||
pos += 1
|
|
||||||
|
|
||||||
if footer_pos >= 0:
|
|
||||||
body = bytes(all_bytes[:footer_pos])
|
|
||||||
footer = bytes(all_bytes[footer_pos : footer_pos + 26])
|
|
||||||
elif len(all_bytes) >= 26:
|
|
||||||
body = bytes(all_bytes[:-26])
|
|
||||||
footer = bytes(all_bytes[-26:])
|
|
||||||
else:
|
|
||||||
body = bytes(all_bytes)
|
|
||||||
footer = b""
|
|
||||||
|
|
||||||
return (strt, body, footer)
|
|
||||||
|
|
||||||
|
|
||||||
def write_blastware_file(
|
def write_blastware_file(
|
||||||
event: Event,
|
event: Event,
|
||||||
a5_frames: list[S3Frame],
|
a5_frames: list[S3Frame],
|
||||||
@@ -738,7 +639,7 @@ def write_blastware_file(
|
|||||||
strt = b"STRT" + b"\xff\xfe" + key4 + bytes(14) + bytes([rectime & 0xFF])
|
strt = b"STRT" + b"\xff\xfe" + key4 + bytes(14) + bytes([rectime & 0xFF])
|
||||||
probe_skip = 7 + 21
|
probe_skip = 7 + 21
|
||||||
|
|
||||||
log.debug(
|
log.warning(
|
||||||
"write_blastware_file: strt_pos_stripped=%d probe_skip=%d "
|
"write_blastware_file: strt_pos_stripped=%d probe_skip=%d "
|
||||||
"probe_data_len=%d strt_hex=%s",
|
"probe_data_len=%d strt_hex=%s",
|
||||||
strt_pos_stripped if strt_pos_stripped >= 0 else -1,
|
strt_pos_stripped if strt_pos_stripped >= 0 else -1,
|
||||||
@@ -771,10 +672,11 @@ def write_blastware_file(
|
|||||||
# Do NOT use a5_frames[-1] — if _a5_frames contains stray frames from a
|
# Do NOT use a5_frames[-1] — if _a5_frames contains stray frames from a
|
||||||
# subsequent event (a known get_events side-effect), the last frame will
|
# subsequent event (a known get_events side-effect), the last frame will
|
||||||
# not be the terminator and the footer will be mis-identified.
|
# not be the terminator and the footer will be mis-identified.
|
||||||
# TERM detection (v0.14.0): last frame if page_key != 0x0010 (sample marker)
|
|
||||||
term_idx: Optional[int] = None
|
term_idx: Optional[int] = None
|
||||||
if a5_frames and a5_frames[-1].page_key != 0x0010:
|
for _i, _f in enumerate(a5_frames):
|
||||||
term_idx = len(a5_frames) - 1
|
if _f.page_key == 0x0000:
|
||||||
|
term_idx = _i
|
||||||
|
break
|
||||||
|
|
||||||
if term_idx is not None:
|
if term_idx is not None:
|
||||||
body_frames = a5_frames[:term_idx]
|
body_frames = a5_frames[:term_idx]
|
||||||
@@ -783,31 +685,37 @@ def write_blastware_file(
|
|||||||
body_frames = a5_frames
|
body_frames = a5_frames
|
||||||
term_frame = None
|
term_frame = None
|
||||||
|
|
||||||
# Frame contribution loop (v0.14.0 BW-exact walk).
|
log.warning(
|
||||||
# Skip values:
|
"write_blastware_file: %d body_frames term_idx=%s",
|
||||||
# probe (fi=0): probe_skip
|
len(body_frames),
|
||||||
# meta@0x1002 (fi=1): 13 (6-byte inner header)
|
str(term_idx) if term_idx is not None else "None",
|
||||||
# meta@0x1004 (fi=2): 13 (6-byte inner header)
|
|
||||||
# sample chunks (fi=3+): 12 (5-byte inner header)
|
|
||||||
last_fi = len(body_frames) - 1
|
|
||||||
|
|
||||||
log.debug(
|
|
||||||
"write_blastware_file: %d body_frames last_fi=%d",
|
|
||||||
len(body_frames), last_fi,
|
|
||||||
)
|
)
|
||||||
|
|
||||||
all_bytes = bytearray()
|
all_bytes = bytearray()
|
||||||
|
|
||||||
for fi, frame in enumerate(body_frames):
|
for fi, frame in enumerate(body_frames):
|
||||||
|
# All body frames contribute to the waveform body — no frames are skipped.
|
||||||
|
#
|
||||||
|
# Over TCP via cellular modem, _recv_5a_batch() correctly collects all
|
||||||
|
# A5 frames per chunk request (the device's ~1100-byte RS-232 response
|
||||||
|
# is forwarded as ~2 TCP segments of ~550 bytes each, each parsed as a
|
||||||
|
# separate S3 frame). ALL of these frames contain ADC body data and
|
||||||
|
# must be included in the file — confirmed from 4-27-26 TCP capture
|
||||||
|
# analysis: contributions from all 14 frames → 6821 bytes → file 6864 bytes.
|
||||||
|
#
|
||||||
|
# Skip amounts (offsets into frame.data):
|
||||||
|
# fi=0 (probe): probe_skip — skips the type_tag header + STRT record
|
||||||
|
# fi=1: 13 — 7-byte frame.data prefix + 6 inner header bytes
|
||||||
|
# fi>=2: 12 — 7-byte frame.data prefix + 5 inner header bytes
|
||||||
if fi == 0:
|
if fi == 0:
|
||||||
skip = probe_skip
|
skip = probe_skip
|
||||||
elif fi in (1, 2):
|
elif fi == 1:
|
||||||
skip = 13 # metadata pages
|
skip = 13
|
||||||
else:
|
else:
|
||||||
skip = 12 # sample chunks
|
skip = 12
|
||||||
|
|
||||||
contribution = _frame_body_bytes(frame, skip)
|
contribution = _frame_body_bytes(frame, skip)
|
||||||
log.debug("write_blastware_file: fi=%d skip=%d raw_data=%d contribution=%d",
|
log.warning("write_blastware_file: fi=%d skip=%d raw_data=%d contribution=%d",
|
||||||
fi, skip, len(frame.data), len(contribution))
|
fi, skip, len(frame.data), len(contribution))
|
||||||
all_bytes.extend(contribution)
|
all_bytes.extend(contribution)
|
||||||
|
|
||||||
@@ -816,7 +724,7 @@ def write_blastware_file(
|
|||||||
# one shorter than chunk frames' 5-byte inner header. Confirmed 2026-04-21.
|
# one shorter than chunk frames' 5-byte inner header. Confirmed 2026-04-21.
|
||||||
if term_frame is not None:
|
if term_frame is not None:
|
||||||
term_contribution = _frame_body_bytes(term_frame, 11)
|
term_contribution = _frame_body_bytes(term_frame, 11)
|
||||||
log.debug(
|
log.warning(
|
||||||
"write_blastware_file: term_frame data_len=%d skip=11 "
|
"write_blastware_file: term_frame data_len=%d skip=11 "
|
||||||
"contribution_len=%d first8=%s",
|
"contribution_len=%d first8=%s",
|
||||||
len(term_frame.data),
|
len(term_frame.data),
|
||||||
@@ -825,49 +733,17 @@ def write_blastware_file(
|
|||||||
)
|
)
|
||||||
all_bytes.extend(term_contribution)
|
all_bytes.extend(term_contribution)
|
||||||
|
|
||||||
log.debug(
|
log.warning(
|
||||||
"write_blastware_file: all_bytes total=%d last28=%s",
|
"write_blastware_file: all_bytes total=%d last28=%s",
|
||||||
len(all_bytes),
|
len(all_bytes),
|
||||||
bytes(all_bytes[-28:]).hex() if len(all_bytes) >= 28 else bytes(all_bytes).hex(),
|
bytes(all_bytes[-28:]).hex() if len(all_bytes) >= 28 else bytes(all_bytes).hex(),
|
||||||
)
|
)
|
||||||
|
|
||||||
# NOTE: The "duplicate header+STRT strip" logic from v0.13.x has been
|
if len(all_bytes) >= 26:
|
||||||
# REMOVED in v0.14.2. Under the v0.14.0 BW-exact 5A walk, body assembly
|
|
||||||
# is just contiguous concatenation of frame contributions in stream order
|
|
||||||
# (probe → meta@0x1002 → meta@0x1004 → samples → TERM), exactly as BW
|
|
||||||
# writes its files. The previous strip was matching the `00 12 03 00 STRT`
|
|
||||||
# byte sequence in legitimate waveform data — sample chunks at counter
|
|
||||||
# 0x1000 and beyond often contain those bytes coincidentally — and
|
|
||||||
# zeroing 25 bytes of valid samples per match. Compared to a known-good
|
|
||||||
# BW reference for the same 3-sec event 0, the strip introduced 26 bytes
|
|
||||||
# of zeros that BW did not have, then propagated alignment differences
|
|
||||||
# through the rest of the body. See decode_test/5-1-26/bw vs SFM diff
|
|
||||||
# at file[0x1012..0x102B] (2026-05-04 analysis).
|
|
||||||
|
|
||||||
# Find the first valid 0e 08 footer marker (v0.14.0).
|
|
||||||
footer_pos = -1
|
|
||||||
pos = 0
|
|
||||||
while True:
|
|
||||||
pos = bytes(all_bytes).find(b"\x0e\x08", pos)
|
|
||||||
if pos < 0 or pos + 26 > len(all_bytes):
|
|
||||||
break
|
|
||||||
yr = (all_bytes[pos + 4] << 8) | all_bytes[pos + 5]
|
|
||||||
if 2015 <= yr <= 2050:
|
|
||||||
footer_pos = pos
|
|
||||||
break
|
|
||||||
pos += 1
|
|
||||||
if footer_pos >= 0:
|
|
||||||
body = bytes(all_bytes[:footer_pos])
|
|
||||||
footer = bytes(all_bytes[footer_pos:footer_pos + 26])
|
|
||||||
log.debug(
|
|
||||||
"write_blastware_file: real 0e 08 footer at all_bytes[%d]; "
|
|
||||||
"truncating %d post-footer bytes",
|
|
||||||
footer_pos, len(all_bytes) - footer_pos - 26,
|
|
||||||
)
|
|
||||||
elif len(all_bytes) >= 26:
|
|
||||||
body = bytes(all_bytes[:-26])
|
body = bytes(all_bytes[:-26])
|
||||||
footer = bytes(all_bytes[-26:])
|
footer = bytes(all_bytes[-26:])
|
||||||
else:
|
else:
|
||||||
|
# Fallback: no terminator or very short stream → build footer from event metadata
|
||||||
body = bytes(all_bytes)
|
body = bytes(all_bytes)
|
||||||
start_dt = _ts_from_model(event.timestamp)
|
start_dt = _ts_from_model(event.timestamp)
|
||||||
stop_dt: Optional[datetime.datetime] = None
|
stop_dt: Optional[datetime.datetime] = None
|
||||||
@@ -878,7 +754,7 @@ def write_blastware_file(
|
|||||||
+ _encode_ts_be(start_dt)
|
+ _encode_ts_be(start_dt)
|
||||||
+ _encode_ts_be(stop_dt)
|
+ _encode_ts_be(stop_dt)
|
||||||
+ b"\x00\x01\x00\x02\x00\x00"
|
+ b"\x00\x01\x00\x02\x00\x00"
|
||||||
+ b"\x00\x00"
|
+ b"\x00\x00" # CRC placeholder
|
||||||
)
|
)
|
||||||
|
|
||||||
# ── Write file ───────────────────────────────────────────────────────────
|
# ── Write file ───────────────────────────────────────────────────────────
|
||||||
|
|||||||
@@ -1,738 +0,0 @@
|
|||||||
"""
|
|
||||||
minimateplus/bw_ascii_report.py — parser for Blastware's per-event ASCII
|
|
||||||
report (the .TXT file BW writes alongside each saved event binary).
|
|
||||||
|
|
||||||
The ASCII export is the authoritative source for every "rich" per-event
|
|
||||||
field that BW computes from the waveform but never persists in the BW
|
|
||||||
binary itself:
|
|
||||||
|
|
||||||
- Per-channel PPV (Tran / Vert / Long / MicL)
|
|
||||||
- Peak Vector Sum + Peak Vector Sum Time
|
|
||||||
- Per-channel ZC Freq, Time of Peak, Peak Acceleration, Peak Displacement
|
|
||||||
- MicL PSPL, MicL Time of Peak, MicL ZC Freq
|
|
||||||
- Per-channel Sensor Self-Check (Test Freq / Test Ratio / Test Results)
|
|
||||||
- MicL Test Amplitude (mV)
|
|
||||||
- Battery, calibration date, monitor-log timestamps
|
|
||||||
|
|
||||||
Persisting these values into the SFM database lets the monthly-summary
|
|
||||||
review workflow ("show me events at Location X with PVS > 0.5") work
|
|
||||||
without depending on the (still-undecoded) waveform body codec.
|
|
||||||
|
|
||||||
Format (verified against decode-re/5-8-26 4-event bundle):
|
|
||||||
|
|
||||||
- One field per line, wrapped in double quotes: `"Field Name : Value"`
|
|
||||||
- Field/value separator: literal ` : ` (space-colon-space).
|
|
||||||
- Some field names contain an internal `:` already (e.g. `"Project:"`),
|
|
||||||
so we split on the FIRST ` : ` only.
|
|
||||||
- Some fields have unit suffixes: `"0.500 in/s"` / `"7.5 Hz"` / `"533 mv"`.
|
|
||||||
- A `"Monitor Log(s)"` marker line is followed by tab-separated rows
|
|
||||||
of `start_time<TAB>stop_time<TAB>description`.
|
|
||||||
- Final `"PC SW Version : ..."` line ends the metadata block.
|
|
||||||
- A blank line separates metadata from the sample table.
|
|
||||||
- Sample table starts with ` Tran <TAB> Vert <TAB>...`, then
|
|
||||||
one row per sample (tab-separated, right-padded numeric values).
|
|
||||||
- Geo channel values are in in/s; MicL in dB(L) (or 0.000 below threshold).
|
|
||||||
|
|
||||||
Because some metadata fields have whitespace quirks ("MicL Time of
|
|
||||||
Peak" has two spaces; the leading "Project:" value has its own colon),
|
|
||||||
we normalise whitespace in the key before lookup.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import datetime
|
|
||||||
import re
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Dict, List, Optional, Tuple, Union
|
|
||||||
|
|
||||||
|
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
|
||||||
# Output dataclasses
|
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class ChannelStats:
|
|
||||||
"""Per-channel derived stats, populated from an event report."""
|
|
||||||
ppv_ips: Optional[float] = None # in/s (geo channels only)
|
|
||||||
zc_freq_hz: Optional[float] = None # Hz
|
|
||||||
time_of_peak_s: Optional[float] = None # seconds (relative to trigger; can be negative)
|
|
||||||
peak_accel_g: Optional[float] = None # g (geo channels only)
|
|
||||||
peak_disp_in: Optional[float] = None # in (geo channels only)
|
|
||||||
# When BW writes "OORANGE" (Out Of Range — truncated) for a PPV
|
|
||||||
# value, the true peak exceeded the channel's full-scale range.
|
|
||||||
# We substitute the range max (e.g. 10.000 in/s for Normal range)
|
|
||||||
# as a lower bound, and flag here so downstream UI / alerts know
|
|
||||||
# to render "> 10 in/s" or "saturated" instead of trusting the
|
|
||||||
# value as an exact measurement.
|
|
||||||
ppv_saturated: bool = False
|
|
||||||
# Set when BW writes ">100 Hz" for ZC Freq — the zero-crossing
|
|
||||||
# algorithm's peak frequency exceeded the device's reporting
|
|
||||||
# ceiling (typically 100 Hz on V10.72). zc_freq_hz gets the
|
|
||||||
# threshold (100.0) as a lower bound; downstream UI renders ">100".
|
|
||||||
zc_freq_above_range: bool = False
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class MicStats:
|
|
||||||
"""MicL-specific stats."""
|
|
||||||
weighting: Optional[str] = None # e.g. "Linear Weighting"
|
|
||||||
pspl_dbl: Optional[float] = None # dB(L)
|
|
||||||
zc_freq_hz: Optional[float] = None
|
|
||||||
time_of_peak_s: Optional[float] = None
|
|
||||||
# Set when BW writes "OORANGE" for PSPL — mic exceeded its
|
|
||||||
# measurement range. pspl_dbl gets the conservative upper bound
|
|
||||||
# 140 dBL (typical NL-43 max; some units cap at 148). Consumers
|
|
||||||
# should render "> 140 dB(L)" or similar when this flag is set.
|
|
||||||
pspl_saturated: bool = False
|
|
||||||
# Same semantics as ChannelStats.zc_freq_above_range — mic ZC
|
|
||||||
# peak exceeded device reporting ceiling.
|
|
||||||
zc_freq_above_range: bool = False
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class SensorCheck:
|
|
||||||
"""Per-channel sensor self-check result.
|
|
||||||
|
|
||||||
Geo channels report a frequency + ratio; MicL reports a frequency +
|
|
||||||
amplitude (mV). All channels also have a Pass/Fail string.
|
|
||||||
"""
|
|
||||||
test_freq_hz: Optional[float] = None
|
|
||||||
test_ratio: Optional[float] = None # geo channels only
|
|
||||||
test_amplitude_mv: Optional[float] = None # MicL only
|
|
||||||
test_results: Optional[str] = None # "Passed" / "Failed"
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class MonitorLogEntry:
|
|
||||||
"""One row of the trailing Monitor Log(s) block."""
|
|
||||||
start_time: Optional[datetime.datetime] = None
|
|
||||||
stop_time: Optional[datetime.datetime] = None
|
|
||||||
description: Optional[str] = None
|
|
||||||
|
|
||||||
|
|
||||||
# BW saturation marker — appears in PPV / Peak Vector Sum / similar
|
|
||||||
# numeric fields when the underlying measurement exceeded the
|
|
||||||
# channel's full-scale range (e.g., a geophone reading > 10 in/s at
|
|
||||||
# Normal range, or a mic exceeding its sensitivity ceiling). Treated
|
|
||||||
# as "≥ range_max" + a saturated flag rather than discarded.
|
|
||||||
# Appears as: ``"Tran PPV : OORANGE in/s"``
|
|
||||||
_OORANGE_MARKERS = ("OORANGE", "OUT OF RANGE")
|
|
||||||
|
|
||||||
|
|
||||||
def _is_oorange(value: str) -> bool:
|
|
||||||
"""True when a BW numeric field is an Out-Of-Range saturation marker."""
|
|
||||||
s = value.strip().upper()
|
|
||||||
return any(m in s for m in _OORANGE_MARKERS)
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_above_range(value: str) -> Optional[float]:
|
|
||||||
"""For BW "above-range" markers like ">100 Hz", return the threshold.
|
|
||||||
|
|
||||||
BW writes ZC Freq as ">100 Hz" when the zero-crossing algorithm sees
|
|
||||||
a peak too fast to count (device cuts off at 100 Hz). Returns the
|
|
||||||
numeric portion after the '>' (e.g. 100.0), or None if `value` is
|
|
||||||
not an above-range marker.
|
|
||||||
"""
|
|
||||||
s = value.strip()
|
|
||||||
if not s.startswith(">"):
|
|
||||||
return None
|
|
||||||
return _parse_number(s[1:])
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class BwAsciiReport:
|
|
||||||
"""Structured representation of one BW per-event ASCII export."""
|
|
||||||
# ── Identity ─────────────────────────────────────────────────────────────
|
|
||||||
event_type: Optional[str] = None # e.g. "Full Waveform"
|
|
||||||
serial: Optional[str] = None # e.g. "BE11529"
|
|
||||||
version: Optional[str] = None # firmware version line
|
|
||||||
file_name: Optional[str] = None # e.g. "M529LK44.AB0"
|
|
||||||
event_datetime: Optional[datetime.datetime] = None # parsed from Event Time + Event Date
|
|
||||||
|
|
||||||
# ── Trigger / recording config ──────────────────────────────────────────
|
|
||||||
trigger_channel: Optional[str] = None # e.g. "Vert" or "From Unit"
|
|
||||||
geo_trigger_level_ips: Optional[float] = None
|
|
||||||
pretrig_s: Optional[float] = None # negative seconds
|
|
||||||
record_time_s: Optional[float] = None
|
|
||||||
record_stop_mode: Optional[str] = None
|
|
||||||
sample_rate_sps: Optional[int] = None
|
|
||||||
battery_volts: Optional[float] = None
|
|
||||||
calibration_date: Optional[datetime.date] = None
|
|
||||||
calibration_by: Optional[str] = None # e.g. "Instantel"
|
|
||||||
units: Optional[str] = None # e.g. "in/s and dB(L)"
|
|
||||||
|
|
||||||
# ── Operator-supplied metadata ──────────────────────────────────────────
|
|
||||||
# Parsed by POSITION from the 4-line "User Notes" block BW writes
|
|
||||||
# between the `Units :` and `Geo Range :` lines. Position-based so
|
|
||||||
# the values populate correctly even when an operator renames the
|
|
||||||
# labels in Blastware's Compliance Setup → Notes tab (the 4 labels
|
|
||||||
# are user-editable, e.g. "Seis Loc:" → "Building:" → "Site Address:").
|
|
||||||
# The original labels BW wrote are preserved in `user_note_labels`
|
|
||||||
# so terra-view can render them as the operator named them.
|
|
||||||
project: Optional[str] = None # position 1 (BW default label "Project:")
|
|
||||||
client: Optional[str] = None # position 2 (BW default label "Client:")
|
|
||||||
operator: Optional[str] = None # position 3 (BW default label "User Name:")
|
|
||||||
sensor_location: Optional[str] = None # position 4 (BW default label "Seis Loc:")
|
|
||||||
|
|
||||||
# Maps canonical slot name → the literal label BW wrote in the ASCII
|
|
||||||
# export. Empty if the User Notes block wasn't present. Example
|
|
||||||
# when the operator renamed slot 4 to "Building:":
|
|
||||||
# {"project": "Project:", "client": "Client:",
|
|
||||||
# "operator": "User Name:", "sensor_location": "Building:"}
|
|
||||||
user_note_labels: Dict[str, str] = field(default_factory=dict)
|
|
||||||
|
|
||||||
# ── Geo channel scaling ─────────────────────────────────────────────────
|
|
||||||
geo_range_ips: Optional[float] = None # 10.000 / 1.250
|
|
||||||
|
|
||||||
# ── Per-channel derived stats (geo + mic) ───────────────────────────────
|
|
||||||
channels: Dict[str, ChannelStats] = field(default_factory=dict)
|
|
||||||
mic: MicStats = field(default_factory=MicStats)
|
|
||||||
|
|
||||||
# ── Vector sum ──────────────────────────────────────────────────────────
|
|
||||||
peak_vector_sum_ips: Optional[float] = None
|
|
||||||
peak_vector_sum_time_s: Optional[float] = None
|
|
||||||
# Saturation flag — set when BW writes "OORANGE" for the PVS. We
|
|
||||||
# then substitute sqrt(3) * geo_range_ips as a conservative upper
|
|
||||||
# bound (the theoretical maximum PVS when all 3 geo channels are
|
|
||||||
# simultaneously at full-scale). Consumers should display this as
|
|
||||||
# ">{value} in/s" or similar.
|
|
||||||
peak_vector_sum_saturated: bool = False
|
|
||||||
# Histograms additionally have an absolute date+time for the PVS
|
|
||||||
# (it occurred at a specific interval). Waveform reports show
|
|
||||||
# only the relative-time value above.
|
|
||||||
peak_vector_sum_when: Optional[datetime.datetime] = None
|
|
||||||
|
|
||||||
# ── Histogram-specific fields (populated only when Event Type starts
|
|
||||||
# with 'Histogram' / 'Full Histogram' / 'Histogram + Continuous') ──
|
|
||||||
histogram_start: Optional[datetime.datetime] = None
|
|
||||||
histogram_stop: Optional[datetime.datetime] = None
|
|
||||||
histogram_n_intervals: Optional[int] = None # e.g. 4, 1436
|
|
||||||
histogram_interval_size_str: Optional[str] = None # "1 minute" / "5 minutes" / "15 seconds"
|
|
||||||
histogram_interval_size_s: Optional[float] = None # parsed to seconds
|
|
||||||
# Per-channel absolute peak time+date (histogram-specific). For
|
|
||||||
# waveform events these are None — those reports use the channel's
|
|
||||||
# time_of_peak_s (relative to trigger) instead. Keyed by channel
|
|
||||||
# name ("Tran", "Vert", "Long", "MicL").
|
|
||||||
channel_peak_when: Dict[str, datetime.datetime] = field(default_factory=dict)
|
|
||||||
|
|
||||||
# ── Sensor self-check (per channel) ─────────────────────────────────────
|
|
||||||
sensor_check: Dict[str, SensorCheck] = field(default_factory=dict)
|
|
||||||
|
|
||||||
# ── Monitor log + tooling version ───────────────────────────────────────
|
|
||||||
monitor_log: List[MonitorLogEntry] = field(default_factory=list)
|
|
||||||
pc_sw_version: Optional[str] = None
|
|
||||||
|
|
||||||
# ── Sample table (optional; only parsed if requested) ───────────────────
|
|
||||||
# Each entry: (Tran, Vert, Long, MicL) in the report's units (geo
|
|
||||||
# channels in in/s, MicL in dB(L)). None when parse_samples=False.
|
|
||||||
samples: Optional[List[Tuple[float, float, float, float]]] = None
|
|
||||||
|
|
||||||
|
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
|
||||||
# Helpers
|
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
_KEY_NORMALISE_RE = re.compile(r"\s+")
|
|
||||||
_NUMERIC_RE = re.compile(r"^-?\d+(?:\.\d+)?")
|
|
||||||
|
|
||||||
|
|
||||||
def _normalise_key(k: str) -> str:
|
|
||||||
"""Collapse whitespace runs (incl. tabs) and strip — handles BW's
|
|
||||||
"MicL Time of Peak" double-space and leading-colon quirks."""
|
|
||||||
return _KEY_NORMALISE_RE.sub(" ", k).strip()
|
|
||||||
|
|
||||||
|
|
||||||
def _strip_quotes(line: str) -> str:
|
|
||||||
line = line.rstrip("\r\n")
|
|
||||||
if len(line) >= 2 and line.startswith('"') and line.endswith('"'):
|
|
||||||
return line[1:-1]
|
|
||||||
return line
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_number(value: str) -> Optional[float]:
|
|
||||||
"""Pull the leading numeric portion out of a value like "0.500 in/s"."""
|
|
||||||
m = _NUMERIC_RE.match(value.strip())
|
|
||||||
if not m:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return float(m.group(0))
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_int(value: str) -> Optional[int]:
|
|
||||||
n = _parse_number(value)
|
|
||||||
return None if n is None else int(round(n))
|
|
||||||
|
|
||||||
|
|
||||||
# Months exactly as BW writes them.
|
|
||||||
_MONTHS = {
|
|
||||||
"January": 1, "February": 2, "March": 3, "April": 4,
|
|
||||||
"May": 5, "June": 6, "July": 7, "August": 8,
|
|
||||||
"September": 9, "October": 10, "November": 11, "December": 12,
|
|
||||||
# Short forms used in monitor-log rows ("Apr 23 /26").
|
|
||||||
"Jan": 1, "Feb": 2, "Mar": 3, "Apr": 4, "Jun": 6, "Jul": 7,
|
|
||||||
"Aug": 8, "Sep": 9, "Oct": 10, "Nov": 11, "Dec": 12,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_event_date(s: str) -> Optional[datetime.date]:
|
|
||||||
"""Parse "April 23, 2026" or "May 8, 2026" → date."""
|
|
||||||
s = s.strip()
|
|
||||||
parts = s.replace(",", " ").split()
|
|
||||||
if len(parts) < 3:
|
|
||||||
return None
|
|
||||||
month_name, day_str, year_str = parts[0], parts[1], parts[2]
|
|
||||||
month = _MONTHS.get(month_name)
|
|
||||||
if month is None:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return datetime.date(int(year_str), month, int(day_str))
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_iso_date(s: str) -> Optional[datetime.date]:
|
|
||||||
"""Parse "2026-05-16" → date. Histograms use ISO format for their
|
|
||||||
Start Date / Stop Date / Peak Date fields; waveforms use the
|
|
||||||
"May 8, 2026" long form which `_parse_event_date` handles."""
|
|
||||||
s = s.strip()
|
|
||||||
try:
|
|
||||||
return datetime.date.fromisoformat(s)
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
_INTERVAL_UNIT_SECONDS = {
|
|
||||||
"second": 1, "seconds": 1, "sec": 1, "secs": 1,
|
|
||||||
"minute": 60, "minutes": 60, "min": 60, "mins": 60,
|
|
||||||
"hour": 3600, "hours": 3600, "hr": 3600, "hrs": 3600,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_interval_size(s: str) -> Optional[float]:
|
|
||||||
"""Parse "1 minute" / "5 minutes" / "15 seconds" / "2 seconds" → seconds.
|
|
||||||
|
|
||||||
Handles the BW Compliance Setup → Histogram Interval values verbatim
|
|
||||||
("2 seconds", "5 seconds", "15 seconds", "1 minute", "5 minutes",
|
|
||||||
"15 minutes") plus a few defensive variants.
|
|
||||||
"""
|
|
||||||
if not s:
|
|
||||||
return None
|
|
||||||
parts = s.strip().split()
|
|
||||||
if len(parts) < 2:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
n = float(parts[0])
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
unit_per_s = _INTERVAL_UNIT_SECONDS.get(parts[1].lower())
|
|
||||||
if unit_per_s is None:
|
|
||||||
return None
|
|
||||||
return n * unit_per_s
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_event_time(s: str) -> Optional[datetime.time]:
|
|
||||||
"""Parse "15:56:35" → time."""
|
|
||||||
s = s.strip()
|
|
||||||
try:
|
|
||||||
h, m, sec = s.split(":")
|
|
||||||
return datetime.time(int(h), int(m), int(sec))
|
|
||||||
except (ValueError, IndexError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_calibration(value: str) -> Tuple[Optional[datetime.date], Optional[str]]:
|
|
||||||
"""Parse "April 29, 2025 by Instantel" → (date, "Instantel")."""
|
|
||||||
parts = value.split(" by ", 1)
|
|
||||||
date = _parse_event_date(parts[0])
|
|
||||||
by = parts[1].strip() if len(parts) > 1 else None
|
|
||||||
return date, by
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_monitor_row(line: str) -> Optional[MonitorLogEntry]:
|
|
||||||
"""Parse a tab-separated monitor log row.
|
|
||||||
|
|
||||||
Format: `<start>\t<stop>\t<desc>` where each timestamp is BW's
|
|
||||||
short form "Mon DD /YY HH:MM:SS" (e.g. "Apr 23 /26 15:46:16").
|
|
||||||
Year is encoded as a 2-digit suffix; we expand "/26" → 2026.
|
|
||||||
"""
|
|
||||||
parts = line.split("\t")
|
|
||||||
if len(parts) < 2:
|
|
||||||
return None
|
|
||||||
start = _parse_monitor_ts(parts[0])
|
|
||||||
stop = _parse_monitor_ts(parts[1])
|
|
||||||
desc = parts[2].strip() if len(parts) > 2 else None
|
|
||||||
if start is None and stop is None and not desc:
|
|
||||||
return None
|
|
||||||
return MonitorLogEntry(start_time=start, stop_time=stop, description=desc)
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_monitor_ts(s: str) -> Optional[datetime.datetime]:
|
|
||||||
"""Parse "Apr 23 /26 15:46:16" → datetime."""
|
|
||||||
s = s.strip()
|
|
||||||
parts = s.split()
|
|
||||||
if len(parts) < 4:
|
|
||||||
return None
|
|
||||||
month = _MONTHS.get(parts[0])
|
|
||||||
if month is None:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
day = int(parts[1])
|
|
||||||
# parts[2] looks like "/26" → century-flip to 2026
|
|
||||||
yy = int(parts[2].lstrip("/"))
|
|
||||||
year = 2000 + yy if yy < 80 else 1900 + yy
|
|
||||||
h, m, sec = (int(x) for x in parts[3].split(":"))
|
|
||||||
return datetime.datetime(year, month, day, h, m, sec)
|
|
||||||
except (ValueError, IndexError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
# ── User-notes positional slot map ──────────────────────────────────────────
|
|
||||||
#
|
|
||||||
# Blastware's Compliance Setup → Notes tab shows four operator-supplied
|
|
||||||
# fields whose LABELS the operator can rename (see screenshot in
|
|
||||||
# project archive). Defaults are "Project:" / "Client:" /
|
|
||||||
# "User Name:" / "Seis Loc:", but an operator using a different
|
|
||||||
# convention can rename them to anything ("Building:", "Site:",
|
|
||||||
# "Address:", etc.). The ASCII export reflects whatever the operator
|
|
||||||
# typed, so label-based matching is fragile.
|
|
||||||
#
|
|
||||||
# What IS reliable: BW always writes the 4 user-notes lines in the
|
|
||||||
# same order, contiguously between the `Units :` line and the
|
|
||||||
# `Geo Range :` line. We parse them by POSITION and preserve the
|
|
||||||
# operator's labels in `report.user_note_labels` so terra-view can
|
|
||||||
# render them as the operator intended.
|
|
||||||
|
|
||||||
_USER_NOTE_SLOTS = ("project", "client", "operator", "sensor_location")
|
|
||||||
|
|
||||||
|
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
|
||||||
# Top-level parser
|
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def parse_report(text: Union[str, bytes], *, parse_samples: bool = False) -> BwAsciiReport:
|
|
||||||
"""Parse a BW per-event ASCII export into a structured BwAsciiReport.
|
|
||||||
|
|
||||||
Set ``parse_samples=True`` to also populate ``report.samples`` with
|
|
||||||
the trailing sample table. Default False because the table is
|
|
||||||
huge and most callers only want metadata for indexing.
|
|
||||||
"""
|
|
||||||
if isinstance(text, bytes):
|
|
||||||
text = text.decode("ascii", errors="replace")
|
|
||||||
|
|
||||||
report = BwAsciiReport()
|
|
||||||
# Pre-create channel stat slots so callers can rely on them existing.
|
|
||||||
for ch in ("Tran", "Vert", "Long", "MicL"):
|
|
||||||
report.channels.setdefault(ch, ChannelStats())
|
|
||||||
report.sensor_check.setdefault(ch, SensorCheck())
|
|
||||||
|
|
||||||
lines = text.splitlines()
|
|
||||||
i = 0
|
|
||||||
n = len(lines)
|
|
||||||
|
|
||||||
in_monitor_log_section = False
|
|
||||||
event_time_str: Optional[str] = None
|
|
||||||
event_date: Optional[datetime.date] = None
|
|
||||||
|
|
||||||
# User-notes block detection. We enter the block after parsing
|
|
||||||
# the "Units :" line and exit on the "Geo Range :" line. Inside,
|
|
||||||
# the first 4 unmatched `<label> : <value>` lines are assigned to
|
|
||||||
# the 4 canonical operator-supplied slots by POSITION (project,
|
|
||||||
# client, operator, sensor_location) regardless of what the
|
|
||||||
# operator named the labels in BW's Compliance Setup → Notes tab.
|
|
||||||
in_user_notes_block = False
|
|
||||||
user_note_position = 0
|
|
||||||
|
|
||||||
# Histogram-field staging — BW writes <Channel> Peak Time and
|
|
||||||
# <Channel> Peak Date on separate lines (and similarly Histogram
|
|
||||||
# Start Time / Date). We stash the partial value when the time
|
|
||||||
# line arrives and combine it when the matching date line arrives.
|
|
||||||
_hist_start_time: Optional[datetime.time] = None
|
|
||||||
_hist_stop_time: Optional[datetime.time] = None
|
|
||||||
_pending_peak_time: Dict[str, Optional[datetime.time]] = {}
|
|
||||||
_pvs_time_raw: Optional[str] = None # last Peak Vector Sum Time value, raw
|
|
||||||
|
|
||||||
while i < n:
|
|
||||||
raw_line = lines[i]
|
|
||||||
i += 1
|
|
||||||
# Blank line marks the start of the sample table.
|
|
||||||
if raw_line.strip() == "":
|
|
||||||
break
|
|
||||||
|
|
||||||
line = _strip_quotes(raw_line)
|
|
||||||
|
|
||||||
# Monitor log section: "Monitor Log(s)" header followed by N rows
|
|
||||||
# (still inside double-quoted lines), terminated by a non-row line
|
|
||||||
# like "PC SW Version : ..." or a blank line.
|
|
||||||
if not in_monitor_log_section and line.strip() == "Monitor Log(s)":
|
|
||||||
in_monitor_log_section = True
|
|
||||||
continue
|
|
||||||
if in_monitor_log_section:
|
|
||||||
# Heuristic: monitor rows contain a tab; the next "Field : Value"
|
|
||||||
# line ends the section.
|
|
||||||
if "\t" in line:
|
|
||||||
entry = _parse_monitor_row(line)
|
|
||||||
if entry:
|
|
||||||
report.monitor_log.append(entry)
|
|
||||||
continue
|
|
||||||
# Falls through to the field parser below; clear the flag.
|
|
||||||
in_monitor_log_section = False
|
|
||||||
|
|
||||||
# "Field : Value" — split on FIRST occurrence of " : "
|
|
||||||
idx = line.find(" : ")
|
|
||||||
if idx < 0:
|
|
||||||
continue
|
|
||||||
key = _normalise_key(line[:idx])
|
|
||||||
value = line[idx + 3 :].strip()
|
|
||||||
|
|
||||||
# ── Identity / config ────────────────────────────────────────────────
|
|
||||||
if key == "Event Type": report.event_type = value
|
|
||||||
elif key == "Serial Number": report.serial = value
|
|
||||||
elif key == "Version": report.version = value
|
|
||||||
elif key == "File Name": report.file_name = value
|
|
||||||
elif key == "Event Time": event_time_str = value
|
|
||||||
elif key == "Event Date": event_date = _parse_event_date(value)
|
|
||||||
|
|
||||||
elif key == "Trigger": report.trigger_channel = value
|
|
||||||
elif key == "Geo Trigger Level": report.geo_trigger_level_ips = _parse_number(value)
|
|
||||||
elif key == "Pre-trigger Length": report.pretrig_s = _parse_number(value)
|
|
||||||
elif key == "Record Time": report.record_time_s = _parse_number(value)
|
|
||||||
elif key == "Record Stop Mode": report.record_stop_mode = value
|
|
||||||
elif key == "Sample Rate": report.sample_rate_sps = _parse_int(value)
|
|
||||||
elif key == "Battery Level": report.battery_volts = _parse_number(value)
|
|
||||||
elif key == "Calibration":
|
|
||||||
report.calibration_date, report.calibration_by = _parse_calibration(value)
|
|
||||||
elif key == "Units":
|
|
||||||
report.units = value
|
|
||||||
# Entering the user-notes block. Next ~4 lines until
|
|
||||||
# "Geo Range :" are the operator-supplied notes.
|
|
||||||
in_user_notes_block = True
|
|
||||||
user_note_position = 0
|
|
||||||
|
|
||||||
elif key == "Geo Range":
|
|
||||||
# Exiting the user-notes block.
|
|
||||||
in_user_notes_block = False
|
|
||||||
report.geo_range_ips = _parse_number(value)
|
|
||||||
|
|
||||||
# User-notes block: assign by position (operator may have
|
|
||||||
# renamed the labels, so we don't trust them). Preserve the
|
|
||||||
# original labels in `user_note_labels` for downstream UIs
|
|
||||||
# (terra-view) that want to display them as the operator
|
|
||||||
# named them.
|
|
||||||
elif in_user_notes_block and user_note_position < len(_USER_NOTE_SLOTS):
|
|
||||||
slot = _USER_NOTE_SLOTS[user_note_position]
|
|
||||||
setattr(report, slot, value)
|
|
||||||
report.user_note_labels[slot] = key
|
|
||||||
user_note_position += 1
|
|
||||||
|
|
||||||
# ── Per-channel stats ────────────────────────────────────────────────
|
|
||||||
# All match the pattern "{Channel} <stat-name>"
|
|
||||||
elif key in (
|
|
||||||
"Tran PPV", "Vert PPV", "Long PPV",
|
|
||||||
"Tran ZC Freq", "Vert ZC Freq", "Long ZC Freq",
|
|
||||||
"Tran Time of Peak", "Vert Time of Peak", "Long Time of Peak",
|
|
||||||
"Tran Peak Acceleration", "Vert Peak Acceleration", "Long Peak Acceleration",
|
|
||||||
"Tran Peak Displacement", "Vert Peak Displacement", "Long Peak Displacement",
|
|
||||||
):
|
|
||||||
ch_name, stat = key.split(" ", 1)
|
|
||||||
cs = report.channels.setdefault(ch_name, ChannelStats())
|
|
||||||
if stat == "PPV":
|
|
||||||
if _is_oorange(value):
|
|
||||||
# Channel saturated — substitute range max as lower
|
|
||||||
# bound; flag so downstream UI can render "> 10 in/s".
|
|
||||||
cs.ppv_ips = report.geo_range_ips
|
|
||||||
cs.ppv_saturated = True
|
|
||||||
else:
|
|
||||||
cs.ppv_ips = _parse_number(value)
|
|
||||||
elif stat == "ZC Freq":
|
|
||||||
# ">100 Hz" → store threshold + flag; numeric → parse normally
|
|
||||||
threshold = _parse_above_range(value)
|
|
||||||
if threshold is not None:
|
|
||||||
cs.zc_freq_hz = threshold
|
|
||||||
cs.zc_freq_above_range = True
|
|
||||||
else:
|
|
||||||
cs.zc_freq_hz = _parse_number(value)
|
|
||||||
else:
|
|
||||||
num = _parse_number(value)
|
|
||||||
if stat == "Time of Peak": cs.time_of_peak_s = num
|
|
||||||
elif stat == "Peak Acceleration": cs.peak_accel_g = num
|
|
||||||
elif stat == "Peak Displacement": cs.peak_disp_in = num
|
|
||||||
|
|
||||||
# ── Histogram-specific fields ────────────────────────────────────────
|
|
||||||
# Histograms have Start/Stop time+date pairs + an interval count
|
|
||||||
# and size, plus per-channel absolute Peak Time/Date instead of
|
|
||||||
# the waveform's relative Time of Peak.
|
|
||||||
elif key == "Histogram Start Time":
|
|
||||||
_hist_start_time = _parse_event_time(value)
|
|
||||||
elif key == "Histogram Start Date":
|
|
||||||
_d = _parse_iso_date(value)
|
|
||||||
if _d and _hist_start_time:
|
|
||||||
report.histogram_start = datetime.datetime.combine(_d, _hist_start_time)
|
|
||||||
elif key == "Histogram Stop Time":
|
|
||||||
_hist_stop_time = _parse_event_time(value)
|
|
||||||
elif key == "Histogram Stop Date":
|
|
||||||
_d = _parse_iso_date(value)
|
|
||||||
if _d and _hist_stop_time:
|
|
||||||
report.histogram_stop = datetime.datetime.combine(_d, _hist_stop_time)
|
|
||||||
elif key == "Number of Intervals":
|
|
||||||
try:
|
|
||||||
report.histogram_n_intervals = int(float(value.strip()))
|
|
||||||
except ValueError:
|
|
||||||
pass
|
|
||||||
elif key == "Interval Size":
|
|
||||||
report.histogram_interval_size_str = value.strip()
|
|
||||||
report.histogram_interval_size_s = _parse_interval_size(value)
|
|
||||||
|
|
||||||
# ── Per-channel histogram Peak Date / Peak Time ──
|
|
||||||
# Lines like "Tran Peak Time : 22:31:38" + "Tran Peak Date : 2026-05-16"
|
|
||||||
elif key in ("Tran Peak Time", "Vert Peak Time", "Long Peak Time", "MicL Time"):
|
|
||||||
ch_name = "MicL" if key == "MicL Time" else key.split(" ", 1)[0]
|
|
||||||
_pending_peak_time[ch_name] = _parse_event_time(value)
|
|
||||||
elif key in ("Tran Peak Date", "Vert Peak Date", "Long Peak Date", "MicL Date"):
|
|
||||||
ch_name = "MicL" if key == "MicL Date" else key.split(" ", 1)[0]
|
|
||||||
_d = _parse_iso_date(value)
|
|
||||||
_t = _pending_peak_time.get(ch_name)
|
|
||||||
if _d and _t:
|
|
||||||
report.channel_peak_when[ch_name] = datetime.datetime.combine(_d, _t)
|
|
||||||
|
|
||||||
# ── Vector Sum ───────────────────────────────────────────────────────
|
|
||||||
elif key == "Peak Vector Sum":
|
|
||||||
if _is_oorange(value):
|
|
||||||
# PVS saturated — conservative upper bound is
|
|
||||||
# sqrt(3) * geo_range_ips (all 3 channels at full-scale).
|
|
||||||
# Real PVS could be lower (channels rarely peak
|
|
||||||
# simultaneously) but never higher within the range.
|
|
||||||
if report.geo_range_ips is not None:
|
|
||||||
import math as _math
|
|
||||||
report.peak_vector_sum_ips = _math.sqrt(3) * report.geo_range_ips
|
|
||||||
report.peak_vector_sum_saturated = True
|
|
||||||
else:
|
|
||||||
report.peak_vector_sum_ips = _parse_number(value)
|
|
||||||
# BW writes the PVS-time label with a typo: "Peak Vector Sum TimeSum"
|
|
||||||
# (looks like Sum got appended twice). Accept both forms. Confirmed
|
|
||||||
# against actual BW output on 2026-05-27 — every PVS-time line in
|
|
||||||
# the field examples (T190, T438, K557) uses the typo'd label.
|
|
||||||
elif key in ("Peak Vector Sum Time", "Peak Vector Sum TimeSum"):
|
|
||||||
report.peak_vector_sum_time_s = _parse_number(value)
|
|
||||||
_pvs_time_raw = value
|
|
||||||
elif key == "Peak Vector Sum Date":
|
|
||||||
# Histogram-mode PVS gets paired with a date. We may have
|
|
||||||
# captured 'Peak Vector Sum Time' as either a relative
|
|
||||||
# seconds float (waveform) or an HH:MM:SS string we
|
|
||||||
# interpreted as a number. For histograms, BW writes
|
|
||||||
# "Peak Vector Sum Time : 22:33:52" which _parse_number
|
|
||||||
# parses as 22.0 (loses information). When Peak Vector Sum
|
|
||||||
# Date arrives, re-parse the previous PVS time line as a
|
|
||||||
# clock time and combine into an absolute datetime.
|
|
||||||
_d = _parse_iso_date(value)
|
|
||||||
if _d and _pvs_time_raw is not None:
|
|
||||||
_t = _parse_event_time(_pvs_time_raw)
|
|
||||||
if _t:
|
|
||||||
report.peak_vector_sum_when = datetime.datetime.combine(_d, _t)
|
|
||||||
# The earlier seconds parse was bogus for histograms;
|
|
||||||
# clear it so consumers don't think it's a real offset.
|
|
||||||
report.peak_vector_sum_time_s = None
|
|
||||||
|
|
||||||
# ── Microphone block ────────────────────────────────────────────────
|
|
||||||
elif key == "Microphone":
|
|
||||||
report.mic.weighting = value
|
|
||||||
elif key == "MicL PSPL":
|
|
||||||
if _is_oorange(value):
|
|
||||||
# Mic saturated — substitute conservative upper bound 140 dBL.
|
|
||||||
report.mic.pspl_dbl = 140.0
|
|
||||||
report.mic.pspl_saturated = True
|
|
||||||
else:
|
|
||||||
report.mic.pspl_dbl = _parse_number(value)
|
|
||||||
# Mirror onto the "MicL" entry in channels so callers querying
|
|
||||||
# `channels["MicL"].ppv_ips` see something — but it's dB(L), not
|
|
||||||
# in/s, so we store as-is in the MicStats and mark the channel.
|
|
||||||
elif key == "MicL Time of Peak":
|
|
||||||
report.mic.time_of_peak_s = _parse_number(value)
|
|
||||||
cs = report.channels.setdefault("MicL", ChannelStats())
|
|
||||||
cs.time_of_peak_s = report.mic.time_of_peak_s
|
|
||||||
elif key == "MicL ZC Freq":
|
|
||||||
threshold = _parse_above_range(value)
|
|
||||||
if threshold is not None:
|
|
||||||
report.mic.zc_freq_hz = threshold
|
|
||||||
report.mic.zc_freq_above_range = True
|
|
||||||
else:
|
|
||||||
report.mic.zc_freq_hz = _parse_number(value)
|
|
||||||
cs = report.channels.setdefault("MicL", ChannelStats())
|
|
||||||
cs.zc_freq_hz = report.mic.zc_freq_hz
|
|
||||||
cs.zc_freq_above_range = report.mic.zc_freq_above_range
|
|
||||||
|
|
||||||
# ── Sensor self-check ────────────────────────────────────────────────
|
|
||||||
elif key in (
|
|
||||||
"Tran Test Freq", "Vert Test Freq", "Long Test Freq", "MicL Test Freq",
|
|
||||||
"Tran Test Ratio", "Vert Test Ratio", "Long Test Ratio",
|
|
||||||
"MicL Test Amplitude",
|
|
||||||
"Tran Test Results", "Vert Test Results", "Long Test Results", "MicL Test Results",
|
|
||||||
):
|
|
||||||
ch_name, stat = key.split(" ", 1)
|
|
||||||
sc = report.sensor_check.setdefault(ch_name, SensorCheck())
|
|
||||||
if stat == "Test Freq": sc.test_freq_hz = _parse_number(value)
|
|
||||||
elif stat == "Test Ratio": sc.test_ratio = _parse_number(value)
|
|
||||||
elif stat == "Test Amplitude": sc.test_amplitude_mv = _parse_number(value)
|
|
||||||
elif stat == "Test Results": sc.test_results = value
|
|
||||||
|
|
||||||
# ── Trailer ─────────────────────────────────────────────────────────
|
|
||||||
elif key == "PC SW Version":
|
|
||||||
report.pc_sw_version = value
|
|
||||||
|
|
||||||
# Unknown keys are silently dropped — forward-compat for future
|
|
||||||
# BW versions that may add fields.
|
|
||||||
|
|
||||||
# Combine event date + time into a datetime
|
|
||||||
if event_date is not None and event_time_str is not None:
|
|
||||||
t = _parse_event_time(event_time_str)
|
|
||||||
if t is not None:
|
|
||||||
report.event_datetime = datetime.datetime.combine(event_date, t)
|
|
||||||
|
|
||||||
if parse_samples:
|
|
||||||
report.samples = _parse_sample_table(lines, i)
|
|
||||||
|
|
||||||
return report
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_sample_table(
|
|
||||||
lines: List[str], start: int,
|
|
||||||
) -> List[Tuple[float, float, float, float]]:
|
|
||||||
"""Parse the trailing sample table.
|
|
||||||
|
|
||||||
The table starts with a header row (" Tran <TAB>...") and continues
|
|
||||||
until EOF. Each data row is a tab-separated quartet of numeric values.
|
|
||||||
"""
|
|
||||||
samples: List[Tuple[float, float, float, float]] = []
|
|
||||||
seen_header = False
|
|
||||||
for line in lines[start:]:
|
|
||||||
line = line.rstrip("\r\n")
|
|
||||||
if not line.strip():
|
|
||||||
continue
|
|
||||||
cols = [c.strip() for c in line.split("\t") if c.strip()]
|
|
||||||
if not seen_header:
|
|
||||||
# Header row contains channel names; numeric rows don't.
|
|
||||||
if any(c in ("Tran", "Vert", "Long", "MicL") for c in cols):
|
|
||||||
seen_header = True
|
|
||||||
continue
|
|
||||||
if len(cols) < 4:
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
samples.append((
|
|
||||||
float(cols[0]), float(cols[1]),
|
|
||||||
float(cols[2]), float(cols[3]),
|
|
||||||
))
|
|
||||||
except ValueError:
|
|
||||||
continue
|
|
||||||
return samples
|
|
||||||
|
|
||||||
|
|
||||||
def parse_report_file(
|
|
||||||
path: Union[str, Path], *, parse_samples: bool = False,
|
|
||||||
) -> BwAsciiReport:
|
|
||||||
"""Convenience: read a .TXT file from disk and parse it."""
|
|
||||||
return parse_report(Path(path).read_bytes(), parse_samples=parse_samples)
|
|
||||||
+95
-301
@@ -449,7 +449,7 @@ class MiniMateClient:
|
|||||||
proto.confirm_erase_all()
|
proto.confirm_erase_all()
|
||||||
log.info("delete_all_events: erase confirmed — device memory cleared")
|
log.info("delete_all_events: erase confirmed — device memory cleared")
|
||||||
|
|
||||||
def get_events(self, full_waveform: bool = False, debug: bool = False, stop_after_index: Optional[int] = None, skip_waveform_for_keys: Optional[set] = None, skip_waveform_for_events: Optional[dict] = None, extra_chunks_after_metadata: int = 1) -> list[Event]:
|
def get_events(self, full_waveform: bool = False, debug: bool = False, stop_after_index: Optional[int] = None, skip_waveform_for_keys: Optional[set] = None, extra_chunks_after_metadata: int = 1) -> list[Event]:
|
||||||
"""
|
"""
|
||||||
Download all stored events from the device using the confirmed
|
Download all stored events from the device using the confirmed
|
||||||
1E → 0A → 0C → 5A → 1F event-iterator protocol.
|
1E → 0A → 0C → 5A → 1F event-iterator protocol.
|
||||||
@@ -497,24 +497,37 @@ class MiniMateClient:
|
|||||||
events: list[Event] = []
|
events: list[Event] = []
|
||||||
idx = 0
|
idx = 0
|
||||||
|
|
||||||
# Legacy bare-key skip set is deprecated: the device's key counter
|
|
||||||
# resets to 0x01110000 after every memory erase, so a key in this set
|
|
||||||
# cannot be trusted to identify the same physical event across erases.
|
|
||||||
# If a caller still passes it, log a warning and ignore — full
|
|
||||||
# downloads will run for every event so the bug never silently bites.
|
|
||||||
if skip_waveform_for_keys:
|
|
||||||
log.warning(
|
|
||||||
"get_events: skip_waveform_for_keys is deprecated and unsafe "
|
|
||||||
"(post-erase key reuse); ignoring %d entries. Use "
|
|
||||||
"skip_waveform_for_events={key: timestamp_iso} instead.",
|
|
||||||
len(skip_waveform_for_keys),
|
|
||||||
)
|
|
||||||
skip_evts: dict[str, str] = dict(skip_waveform_for_events or {})
|
|
||||||
|
|
||||||
while data8[4:8] != b"\x00\x00\x00\x00":
|
while data8[4:8] != b"\x00\x00\x00\x00":
|
||||||
cur_key = key4 # key for this event's 0A/1E-arm/0C/5A calls
|
cur_key = key4 # key for this event's 0A/1E-arm/0C/5A calls
|
||||||
log.info("get_events: record %d key=%s", idx, cur_key.hex())
|
log.info("get_events: record %d key=%s", idx, cur_key.hex())
|
||||||
|
|
||||||
|
# Fast-advance path: if this key is already downloaded, skip
|
||||||
|
# 1E-arm/0C/POLL/5A entirely. Only 0A + 1F(browse) are needed
|
||||||
|
# to advance the device's internal pointer to the next event.
|
||||||
|
# This is identical to the browse-mode walk in count_events().
|
||||||
|
if skip_waveform_for_keys and cur_key.hex() in skip_waveform_for_keys:
|
||||||
|
log.debug("get_events: key=%s already seen -- fast-advance only", cur_key.hex())
|
||||||
|
try:
|
||||||
|
proto.read_waveform_header(cur_key)
|
||||||
|
except ProtocolError as exc:
|
||||||
|
log.warning(
|
||||||
|
"get_events: 0A failed for key=%s (skip path): %s -- stopping",
|
||||||
|
cur_key.hex(), exc,
|
||||||
|
)
|
||||||
|
break
|
||||||
|
try:
|
||||||
|
key4, data8 = proto.advance_event(browse=True)
|
||||||
|
except ProtocolError as exc:
|
||||||
|
log.warning(
|
||||||
|
"get_events: 1F failed for key=%s (skip path): %s -- stopping",
|
||||||
|
cur_key.hex(), exc,
|
||||||
|
)
|
||||||
|
break
|
||||||
|
idx += 1
|
||||||
|
if stop_after_index is not None and idx > stop_after_index:
|
||||||
|
break
|
||||||
|
continue
|
||||||
|
|
||||||
ev = Event(index=idx)
|
ev = Event(index=idx)
|
||||||
ev._waveform_key = cur_key
|
ev._waveform_key = cur_key
|
||||||
|
|
||||||
@@ -561,30 +574,11 @@ class MiniMateClient:
|
|||||||
"get_events: 0C failed for key=%s: %s", cur_key.hex(), exc
|
"get_events: 0C failed for key=%s: %s", cur_key.hex(), exc
|
||||||
)
|
)
|
||||||
|
|
||||||
# ── Skip-5A decision based on (key, timestamp) match ──────
|
|
||||||
# If skip_waveform_for_events maps cur_key.hex() to a non-empty
|
|
||||||
# ISO timestamp matching what we just read from 0C, this is
|
|
||||||
# the same physical event we already have on disk — bypass
|
|
||||||
# the 1F(arm)+POLL+5A bulk download. Otherwise (no entry, or
|
|
||||||
# timestamp mismatch indicating post-erase reuse) fall through
|
|
||||||
# to the full download.
|
|
||||||
expected_ts = skip_evts.get(cur_key.hex(), "")
|
|
||||||
actual_ts = _event_timestamp_iso(ev)
|
|
||||||
skip_5a = bool(expected_ts and actual_ts and expected_ts == actual_ts)
|
|
||||||
if skip_5a:
|
|
||||||
log.info(
|
|
||||||
"get_events: key=%s (key, ts=%s) match — skipping 5A bulk download",
|
|
||||||
cur_key.hex(), actual_ts,
|
|
||||||
)
|
|
||||||
|
|
||||||
arm_key4: Optional[bytes] = None
|
|
||||||
a5_ok = False
|
|
||||||
|
|
||||||
if not skip_5a:
|
|
||||||
# SUB 1F (download-arm) — send token=0xFE BEFORE POLL+5A to arm the
|
# SUB 1F (download-arm) — send token=0xFE BEFORE POLL+5A to arm the
|
||||||
# device's bulk stream state machine. Cache the returned key as a
|
# device's bulk stream state machine. Cache the returned key as a
|
||||||
# fallback for loop iteration when 5A fails (see iteration block below).
|
# fallback for loop iteration when 5A fails (see iteration block below).
|
||||||
# Confirmed from 4-2-26 capture frames 66-67 (1F before frames 68-73 POLL).
|
# Confirmed from 4-2-26 capture frames 66-67 (1F before frames 68-73 POLL).
|
||||||
|
arm_key4: Optional[bytes] = None
|
||||||
try:
|
try:
|
||||||
arm_key4, _ = proto.advance_event(browse=False) # arm 5A
|
arm_key4, _ = proto.advance_event(browse=False) # arm 5A
|
||||||
log.info("get_events: 1F(download) — 5A armed, arm_key=%s", arm_key4.hex())
|
log.info("get_events: 1F(download) — 5A armed, arm_key=%s", arm_key4.hex())
|
||||||
@@ -603,12 +597,7 @@ class MiniMateClient:
|
|||||||
# SUB 5A — bulk waveform stream (uses cur_key, the event set up by 0A+1E+0C).
|
# SUB 5A — bulk waveform stream (uses cur_key, the event set up by 0A+1E+0C).
|
||||||
# By default (full_waveform=False): stop after frame 7 for metadata only.
|
# By default (full_waveform=False): stop after frame 7 for metadata only.
|
||||||
# When full_waveform=True: fetch all chunks and decode raw ADC samples.
|
# When full_waveform=True: fetch all chunks and decode raw ADC samples.
|
||||||
#
|
a5_ok = False
|
||||||
# Bypassed when skip_5a is True — the event is left with
|
|
||||||
# _a5_frames=None, which signals to the caller (e.g.
|
|
||||||
# ach_server.py) that this event was matched by (key, ts) and
|
|
||||||
# already has a stored .file in the persistent waveform store.
|
|
||||||
if not skip_5a:
|
|
||||||
try:
|
try:
|
||||||
if full_waveform:
|
if full_waveform:
|
||||||
log.info(
|
log.info(
|
||||||
@@ -663,14 +652,7 @@ class MiniMateClient:
|
|||||||
# Confirmed from 4-3-26 browse-mode captures: browse=True params
|
# Confirmed from 4-3-26 browse-mode captures: browse=True params
|
||||||
# are correct for multi-event iteration. Conditional logic added
|
# are correct for multi-event iteration. Conditional logic added
|
||||||
# 2026-04-06 to avoid post-failure state disruption.
|
# 2026-04-06 to avoid post-failure state disruption.
|
||||||
#
|
if a5_ok:
|
||||||
# NEW 2026-05-06: when skip_5a=True we never entered the 5A
|
|
||||||
# state at all (we read 0A+1E(arm)+0C and chose to bypass).
|
|
||||||
# 1F(browse) is safe in this scenario — the device's iteration
|
|
||||||
# pointer is independent of the bulk-stream state machine, and
|
|
||||||
# we never put it into the half-attempted 5A state that the
|
|
||||||
# earlier "post-failure 1F disruption" warning is about.
|
|
||||||
if skip_5a or a5_ok:
|
|
||||||
# 5A succeeded — use browse 1F for reliable key advancement.
|
# 5A succeeded — use browse 1F for reliable key advancement.
|
||||||
try:
|
try:
|
||||||
key4, data8 = proto.advance_event(browse=True)
|
key4, data8 = proto.advance_event(browse=True)
|
||||||
@@ -1192,27 +1174,6 @@ class MiniMateClient:
|
|||||||
# Pure functions: bytes → model field population.
|
# Pure functions: bytes → model field population.
|
||||||
# Kept here (not in models.py) to isolate protocol knowledge from data shapes.
|
# Kept here (not in models.py) to isolate protocol knowledge from data shapes.
|
||||||
|
|
||||||
def _event_timestamp_iso(event: Event) -> str:
|
|
||||||
"""
|
|
||||||
Return a stable ISO-8601 string for the event's 0C-derived timestamp,
|
|
||||||
or "" if the event has no timestamp populated.
|
|
||||||
|
|
||||||
The format intentionally matches what `bridges/ach_server.py` writes
|
|
||||||
into `ach_state.json:downloaded_events[*]` so the (key, ts) compare
|
|
||||||
in get_events()'s skip path is a simple string equality.
|
|
||||||
"""
|
|
||||||
ts = getattr(event, "timestamp", None)
|
|
||||||
if ts is None:
|
|
||||||
return ""
|
|
||||||
try:
|
|
||||||
return datetime.datetime(
|
|
||||||
ts.year, ts.month, ts.day,
|
|
||||||
ts.hour or 0, ts.minute or 0, ts.second or 0,
|
|
||||||
).isoformat()
|
|
||||||
except Exception:
|
|
||||||
return str(ts)
|
|
||||||
|
|
||||||
|
|
||||||
def _decode_serial_number(data: bytes) -> DeviceInfo:
|
def _decode_serial_number(data: bytes) -> DeviceInfo:
|
||||||
"""
|
"""
|
||||||
Decode SUB EA (SERIAL_NUMBER_RESPONSE) payload into a new DeviceInfo.
|
Decode SUB EA (SERIAL_NUMBER_RESPONSE) payload into a new DeviceInfo.
|
||||||
@@ -1362,40 +1323,28 @@ def _decode_waveform_record_into(data: bytes, event: Event) -> None:
|
|||||||
|
|
||||||
Modifies event in-place.
|
Modifies event in-place.
|
||||||
"""
|
"""
|
||||||
# ── Record type + format detection ────────────────────────────────────────
|
# ── Record type ───────────────────────────────────────────────────────────
|
||||||
# `record_type` is the user-facing label ("Waveform" for any triggered
|
# Decoded from byte[1] (sub_code) first so we can gate timestamp parsing.
|
||||||
# event regardless of timestamp-header layout). `fmt` is the internal
|
|
||||||
# format code used to pick the right Timestamp parser; it stays
|
|
||||||
# internal and doesn't leak to the API / sidecar / UI.
|
|
||||||
try:
|
try:
|
||||||
event.record_type = _extract_record_type(data)
|
event.record_type = _extract_record_type(data)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
log.warning("waveform record type decode failed: %s", exc)
|
log.warning("waveform record type decode failed: %s", exc)
|
||||||
fmt = _detect_record_format(data)
|
|
||||||
|
|
||||||
# ── Timestamp ─────────────────────────────────────────────────────────────
|
# ── Timestamp ─────────────────────────────────────────────────────────────
|
||||||
# Three timestamp-header layouts have been observed across BE11529
|
# 9-byte format for sub_code=0x10 Waveform records:
|
||||||
# firmware S338.17 — each picks a different Timestamp parser:
|
# [day][sub_code][month][year:2 BE][unknown][hour][min][sec]
|
||||||
# "single_shot": 9-byte [day][0x10][month][year:2][unk][h][m][s]
|
# sub_code=0x10 and sub_code=0x03 have different timestamp byte layouts.
|
||||||
# "continuous": 10-byte [0x10][day][0x10][month][year:2][unk][h][m][s]
|
# Both confirmed against Blastware event reports (BE11529, 2026-04-01 and 2026-04-03).
|
||||||
# "short": 8-byte [day][month][year:2][unk][h][m][s]
|
if event.record_type == "Waveform":
|
||||||
# All decoded into the same Timestamp dataclass — only the byte
|
|
||||||
# offsets differ.
|
|
||||||
if fmt == "single_shot":
|
|
||||||
try:
|
try:
|
||||||
event.timestamp = Timestamp.from_waveform_record(data)
|
event.timestamp = Timestamp.from_waveform_record(data)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
log.warning("single_shot record timestamp decode failed: %s", exc)
|
log.warning("waveform record timestamp decode failed: %s", exc)
|
||||||
elif fmt == "continuous":
|
elif event.record_type == "Waveform (Continuous)":
|
||||||
try:
|
try:
|
||||||
event.timestamp = Timestamp.from_continuous_record(data)
|
event.timestamp = Timestamp.from_continuous_record(data)
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
log.warning("continuous record timestamp decode failed: %s", exc)
|
log.warning("continuous record timestamp decode failed: %s", exc)
|
||||||
elif fmt == "short":
|
|
||||||
try:
|
|
||||||
event.timestamp = Timestamp.from_short_record(data)
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("short record timestamp decode failed: %s", exc)
|
|
||||||
|
|
||||||
# ── Peak values (per-channel PPV + Peak Vector Sum) ───────────────────────
|
# ── Peak values (per-channel PPV + Peak Vector Sum) ───────────────────────
|
||||||
try:
|
try:
|
||||||
@@ -1500,69 +1449,22 @@ def _decode_a5_waveform(
|
|||||||
(BULK_WAVEFORM_STREAM) frame payloads and populate event.raw_samples,
|
(BULK_WAVEFORM_STREAM) frame payloads and populate event.raw_samples,
|
||||||
event.total_samples, event.pretrig_samples, and event.rectime_seconds.
|
event.total_samples, event.pretrig_samples, and event.rectime_seconds.
|
||||||
|
|
||||||
Wired up 2026-05-11 to the verified ``decode_waveform_v2`` codec (see
|
This requires ALL A5 frames (stop_after_metadata=False), not just the
|
||||||
``minimateplus/waveform_codec.py`` and ``docs/waveform_codec_re_status.md``).
|
metadata-bearing subset.
|
||||||
Replaces the legacy int16 LE decoder, which produced full-scale ±32K
|
|
||||||
noise on every event because the body bytes are encoded, not raw
|
|
||||||
samples.
|
|
||||||
|
|
||||||
Output convention (preserved from the legacy decoder):
|
── Waveform format (confirmed from 4-2-26 blast capture) ───────────────────
|
||||||
``event.raw_samples`` is a dict with keys "Tran", "Vert", "Long",
|
The blast waveform is 4-channel interleaved signed 16-bit little-endian,
|
||||||
"MicL" mapping to lists of **int16 ADC counts**. Multiply by
|
8 bytes per sample-set:
|
||||||
``geo_range / 32768`` for geo channels to get in/s; use
|
|
||||||
:func:`minimateplus.waveform_codec.mic_count_to_db` for mic dB(L).
|
|
||||||
|
|
||||||
``total_samples`` / ``pretrig_samples`` / ``rectime_seconds`` are set
|
|
||||||
to ``None`` so the caller backfills from compliance_config (the
|
|
||||||
authoritative source — STRT fields aren't reliable).
|
|
||||||
"""
|
|
||||||
from .waveform_codec import decode_a5_frames
|
|
||||||
|
|
||||||
event.total_samples = None
|
|
||||||
event.pretrig_samples = None
|
|
||||||
event.rectime_seconds = None
|
|
||||||
|
|
||||||
if not frames_data:
|
|
||||||
log.debug("_decode_a5_waveform: no frames provided")
|
|
||||||
return
|
|
||||||
|
|
||||||
decoded = decode_a5_frames(frames_data)
|
|
||||||
if decoded is None:
|
|
||||||
log.warning("_decode_a5_waveform: codec returned no samples")
|
|
||||||
return
|
|
||||||
|
|
||||||
event.raw_samples = decoded
|
|
||||||
log.debug(
|
|
||||||
"_decode_a5_waveform: decoded %d/%d/%d/%d samples (T/V/L/M)",
|
|
||||||
len(decoded.get("Tran", [])),
|
|
||||||
len(decoded.get("Vert", [])),
|
|
||||||
len(decoded.get("Long", [])),
|
|
||||||
len(decoded.get("MicL", [])),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _decode_a5_waveform_LEGACY(
|
|
||||||
frames_data: list[S3Frame],
|
|
||||||
event: Event,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
LEGACY decoder — kept for reference only. DO NOT CALL.
|
|
||||||
|
|
||||||
This is the int16 LE decoder that produced full-scale ±32K noise
|
|
||||||
on every event. Retracted 2026-05-08; replaced 2026-05-11 with
|
|
||||||
the verified codec in :mod:`minimateplus.waveform_codec`. See
|
|
||||||
``docs/instantel_protocol_reference.md §7.6.1`` for the full history.
|
|
||||||
|
|
||||||
── Waveform format (LEGACY — WRONG) ────────────────────────────────
|
|
||||||
Claimed 4-channel interleaved signed 16-bit little-endian, 8 bytes
|
|
||||||
per sample-set:
|
|
||||||
|
|
||||||
[T_lo T_hi V_lo V_hi L_lo L_hi M_lo M_hi] × N
|
[T_lo T_hi V_lo V_hi L_lo L_hi M_lo M_hi] × N
|
||||||
|
|
||||||
where T=Tran, V=Vert, L=Long, M=Mic.
|
where T=Tran, V=Vert, L=Long, M=Mic. Channel ordering follows the
|
||||||
|
Blastware convention [Tran, Vert, Long, Mic] = [ch0, ch1, ch2, ch3].
|
||||||
|
|
||||||
The body bytes are actually a tagged delta+RLE stream — this
|
⚠️ Channel ordering is a confirmed CONVENTION — the physical ordering on
|
||||||
interpretation was wrong.
|
the ADC mux is not independently verifiable from the saturating blast
|
||||||
|
captures we have. The convention is consistent with Blastware labeling
|
||||||
|
(Tran is always the first channel field in the A5 STRT+waveform stream).
|
||||||
|
|
||||||
── Frame structure ──────────────────────────────────────────────────────────
|
── Frame structure ──────────────────────────────────────────────────────────
|
||||||
A5[0] (probe response):
|
A5[0] (probe response):
|
||||||
@@ -1616,109 +1518,46 @@ def _decode_a5_waveform_LEGACY(
|
|||||||
log.warning("_decode_a5_waveform: STRT record truncated (%dB)", len(strt))
|
log.warning("_decode_a5_waveform: STRT record truncated (%dB)", len(strt))
|
||||||
return
|
return
|
||||||
|
|
||||||
# STRT byte layout (21 bytes; verified against M529LIY6 reference files
|
total_samples = struct.unpack_from(">H", strt, 8)[0]
|
||||||
# and re-confirmed against live BE11529 captures, 2026-05-08):
|
pretrig_samples = struct.unpack_from(">H", strt, 16)[0]
|
||||||
# [0:4] b'STRT'
|
rectime_seconds = strt[18]
|
||||||
# [4:6] 0xff 0xfe sentinel
|
|
||||||
# [6:10] end_key 4-byte BE flash address where event ends
|
|
||||||
# [10:14] start_key 4-byte BE flash address where event starts
|
|
||||||
# [14:18] device-specific (semantics not pinned; values vary across events
|
|
||||||
# and don't hold authoritative total_samples / pretrig)
|
|
||||||
# [18] 0x46 record-type marker (NOT rectime)
|
|
||||||
# [19] device-specific
|
|
||||||
# [20] sometimes rectime, sometimes 0 — not reliable
|
|
||||||
#
|
|
||||||
# AUTHORITATIVE values must come from compliance_config (sample_rate,
|
|
||||||
# record_time) and from end_offset - start_offset arithmetic (event size).
|
|
||||||
# Earlier code claimed STRT[8:10]=total_samples and STRT[16:18]=pretrig;
|
|
||||||
# those positions actually overlap end_key low-word and dev-specific bytes
|
|
||||||
# respectively. We surface the address-derived event size so consumers
|
|
||||||
# can sanity-check chunk-loop bounds, but `total_samples` per channel must
|
|
||||||
# be derived externally (sample_rate × record_time, or computed from the
|
|
||||||
# decoded sample count below).
|
|
||||||
end_key = strt[6:10]
|
|
||||||
start_key = strt[10:14]
|
|
||||||
end_offset_in_strt = (end_key[2] << 8) | end_key[3]
|
|
||||||
start_offset_in_strt = (start_key[2] << 8) | start_key[3]
|
|
||||||
is_event_1 = (start_offset_in_strt == 0x0000)
|
|
||||||
|
|
||||||
# Don't trust STRT for these — leave them as None so the caller can
|
event.total_samples = total_samples
|
||||||
# backfill from compliance_config (the authoritative source).
|
event.pretrig_samples = pretrig_samples
|
||||||
event.total_samples = None
|
event.rectime_seconds = rectime_seconds
|
||||||
event.pretrig_samples = None
|
|
||||||
event.rectime_seconds = None
|
|
||||||
|
|
||||||
log.debug(
|
log.debug(
|
||||||
"_decode_a5_waveform: STRT start_key=%s end_key=%s "
|
"_decode_a5_waveform: STRT total_samples=%d pretrig=%d rectime=%ds",
|
||||||
"start_off=0x%04X end_off=0x%04X is_event_1=%s "
|
total_samples, pretrig_samples, rectime_seconds,
|
||||||
"dev-specific[14:18]=%s strt[20]=0x%02X",
|
|
||||||
start_key.hex(), end_key.hex(),
|
|
||||||
start_offset_in_strt, end_offset_in_strt, is_event_1,
|
|
||||||
strt[14:18].hex(), strt[20],
|
|
||||||
)
|
)
|
||||||
|
|
||||||
# ── Collect per-frame waveform bytes with global offset tracking ─────────
|
# ── Collect per-frame waveform bytes with global offset tracking ─────────
|
||||||
# global_offset is the cumulative byte count across all frames, used to
|
# global_offset is the cumulative byte count across all frames, used to
|
||||||
# compute the channel alignment at each frame boundary.
|
# compute the channel alignment at each frame boundary.
|
||||||
#
|
|
||||||
# Frame layout under the v0.14.0+ walk:
|
|
||||||
# frames_data[0] = probe response (page_addr 0x0000;
|
|
||||||
# contains STRT + post-STRT data)
|
|
||||||
# frames_data[1..2] = (event 1 only) metadata pages
|
|
||||||
# page_addr = 0x1002 / 0x1004
|
|
||||||
# frames_data[mid] = sample chunks at flash addresses
|
|
||||||
# 0x0600, 0x0800, … (page_addr in
|
|
||||||
# {0x0600..0x1FFE})
|
|
||||||
# frames_data[last] = TERM response (page_key=0x0000)
|
|
||||||
#
|
|
||||||
# We identify metadata pages by their PAGE ADDRESS at db.data[4:6] (the
|
|
||||||
# 2-byte counter the device echoes back), NOT by content scan. An earlier
|
|
||||||
# needle-based detection (b"Project:", b"Client:", etc.) was the wrong
|
|
||||||
# layer of abstraction:
|
|
||||||
# • The actual metadata pages 0x1002 / 0x1004 do NOT contain ASCII
|
|
||||||
# project strings on this firmware (S338.17 / BE11529).
|
|
||||||
# • The strings physically live at flash address 0x1600 — which falls
|
|
||||||
# inside the sample-chunk address range. Skipping that frame would
|
|
||||||
# drop a real sample chunk.
|
|
||||||
# BW handles the "samples region happens to contain string bytes" case
|
|
||||||
# by just rendering the bytes verbatim; we do the same.
|
|
||||||
_METADATA_PAGES = (b"\x10\x02", b"\x10\x04")
|
|
||||||
|
|
||||||
chunks: list[tuple[int, bytes]] = [] # (frame_idx, waveform_bytes)
|
chunks: list[tuple[int, bytes]] = [] # (frame_idx, waveform_bytes)
|
||||||
global_offset = 0
|
global_offset = 0
|
||||||
|
|
||||||
for fi, db in enumerate(frames_data):
|
for fi, db in enumerate(frames_data):
|
||||||
page_addr = db.data[4:6] if len(db.data) >= 6 else b""
|
|
||||||
w = db.data[7:] # frame.data[7:]
|
w = db.data[7:] # frame.data[7:]
|
||||||
|
|
||||||
# A5[0]: probe response. Two cases:
|
# A5[0]: waveform begins after the 21-byte STRT record and 6-byte preamble.
|
||||||
# - Event 1 (start_offset_in_strt == 0x0000): the bytes after STRT
|
# Layout: STRT(21B) + null-pad(2B) + 0xFF sentinel(4B) = 27 bytes total.
|
||||||
# are the device's *pre-event reserved area* (flash 0x0046 to
|
|
||||||
# 0x0600), NOT samples. We must skip them; samples begin at
|
|
||||||
# the first dedicated chunk frame at counter=0x0600.
|
|
||||||
# - Event N (continuation, start_offset != 0x0000): the bytes after
|
|
||||||
# the STRT record ARE the first slice of real samples for the
|
|
||||||
# event (BW's chunk loop addresses the probe as a sample chunk).
|
|
||||||
if fi == 0:
|
if fi == 0:
|
||||||
sp = w.find(b"STRT")
|
sp = w.find(b"STRT")
|
||||||
if sp < 0:
|
if sp < 0:
|
||||||
continue
|
continue
|
||||||
if is_event_1:
|
|
||||||
# No usable samples in the probe — pre-event reserved bytes.
|
|
||||||
continue
|
|
||||||
# Layout: STRT(21B) + null-pad(2B) + 0xFF sentinel(4B) = 27 bytes total.
|
|
||||||
wave = w[sp + 27 :]
|
wave = w[sp + 27 :]
|
||||||
|
|
||||||
# Skip the dedicated metadata pages (event 1 only): page_addr 0x1002 / 0x1004.
|
# Frame 7 carries event-time metadata strings ("Project:", "Client:", …)
|
||||||
elif page_addr in _METADATA_PAGES:
|
# and no waveform ADC data.
|
||||||
log.debug(
|
elif fi == 7:
|
||||||
"_decode_a5_waveform: skipping metadata page fi=%d page_addr=%s",
|
|
||||||
fi, page_addr.hex(),
|
|
||||||
)
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Sample chunk (or TERM): strip the 8-byte per-frame header.
|
# Terminator frames have page_key=0x0000 and are excluded upstream
|
||||||
|
# (read_bulk_waveform_stream returns early on page_key==0).
|
||||||
|
# No hardcoded frame-index skip here — all non-metadata frames are data.
|
||||||
else:
|
else:
|
||||||
|
# Strip the 8-byte per-frame header (ctr + 6 zero bytes)
|
||||||
if len(w) < 8:
|
if len(w) < 8:
|
||||||
continue
|
continue
|
||||||
wave = w[8:]
|
wave = w[8:]
|
||||||
@@ -1732,8 +1571,10 @@ def _decode_a5_waveform_LEGACY(
|
|||||||
total_bytes = global_offset
|
total_bytes = global_offset
|
||||||
n_sets = total_bytes // 8
|
n_sets = total_bytes // 8
|
||||||
log.debug(
|
log.debug(
|
||||||
"_decode_a5_waveform: %d chunks, %dB total → %d complete sample-sets",
|
"_decode_a5_waveform: %d chunks, %dB total → %d complete sample-sets "
|
||||||
len(chunks), total_bytes, n_sets,
|
"(%d of %d expected; %.0f%%)",
|
||||||
|
len(chunks), total_bytes, n_sets, n_sets, total_samples,
|
||||||
|
100.0 * n_sets / total_samples if total_samples else 0,
|
||||||
)
|
)
|
||||||
|
|
||||||
if n_sets == 0:
|
if n_sets == 0:
|
||||||
@@ -1791,85 +1632,38 @@ def _decode_a5_waveform_LEGACY(
|
|||||||
"Tran": tran,
|
"Tran": tran,
|
||||||
"Vert": vert,
|
"Vert": vert,
|
||||||
"Long": long_,
|
"Long": long_,
|
||||||
"MicL": mic,
|
"Mic": mic,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def _detect_record_format(data: bytes) -> Optional[str]:
|
|
||||||
"""
|
|
||||||
Detect which timestamp-header format a 210-byte 0C waveform record uses.
|
|
||||||
|
|
||||||
THREE formats observed on BE11529 firmware S338.17:
|
|
||||||
|
|
||||||
"single_shot" — 9-byte header:
|
|
||||||
[day] [0x10] [month] [year_BE:2] [unknown] [hour] [min] [sec]
|
|
||||||
sub_code=0x10 at byte [1]. Year at [3:5].
|
|
||||||
|
|
||||||
"continuous" — 10-byte header:
|
|
||||||
[0x10] [day] [0x10] [month] [year_BE:2] [unknown] [hour] [min] [sec]
|
|
||||||
marker 0x10 at byte [0] AND byte [2]. Year at [4:6].
|
|
||||||
|
|
||||||
"short" — 8-byte header (NEW 2026-05-01):
|
|
||||||
[day] [month] [year_BE:2] [unknown] [hour] [min] [sec]
|
|
||||||
No marker bytes. Year at [2:4].
|
|
||||||
|
|
||||||
Each format has the year (uint16 BE) at a UNIQUE byte position, so we can
|
|
||||||
disambiguate by scanning each candidate position and picking the one
|
|
||||||
where the year falls in a sane range (2015..2050).
|
|
||||||
|
|
||||||
Returns "single_shot" / "continuous" / "short" or None if no format matches.
|
|
||||||
"""
|
|
||||||
if len(data) < 8:
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _sane_year(hi: int, lo: int) -> bool:
|
|
||||||
y = (hi << 8) | lo
|
|
||||||
return 2015 <= y <= 2050
|
|
||||||
|
|
||||||
# Order matters: prefer formats with stronger marker-byte evidence first.
|
|
||||||
if data[1] == 0x10 and len(data) >= 9 and _sane_year(data[3], data[4]):
|
|
||||||
return "single_shot"
|
|
||||||
if (data[0] == 0x10 and data[2] == 0x10
|
|
||||||
and len(data) >= 10 and _sane_year(data[4], data[5])):
|
|
||||||
return "continuous"
|
|
||||||
if _sane_year(data[2], data[3]):
|
|
||||||
return "short"
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _extract_record_type(data: bytes) -> Optional[str]:
|
def _extract_record_type(data: bytes) -> Optional[str]:
|
||||||
"""
|
"""
|
||||||
Return a user-facing name for a waveform record. All three internal
|
Decode the recording mode from byte[1] of the 210-byte waveform record.
|
||||||
timestamp-header layouts represent the *same* user concept — a
|
|
||||||
triggered seismic event — so they all surface as just "Waveform".
|
|
||||||
|
|
||||||
The internal format code is preserved for parsing logic (timestamp
|
Byte[1] is the sub-record code that immediately follows the day byte in the
|
||||||
decoder selection) but doesn't leak into the API / UI / sidecar.
|
9-byte timestamp header at the start of each waveform record:
|
||||||
Callers that need the raw layout can call `_detect_record_format`
|
[day:1] [sub_code:1] [month:1] [year:2 BE] ...
|
||||||
directly.
|
|
||||||
|
|
||||||
Background: across BE11529 firmware S338.17 we've observed three
|
Confirmed codes (✅ 2026-04-01):
|
||||||
different byte layouts for the timestamp header at the start of the
|
0x10 → "Waveform" (continuous / single-shot mode)
|
||||||
0C record (8 / 9 / 10 bytes, distinguished by the position of the
|
|
||||||
BE-encoded year and the presence of `0x10` marker bytes). An older
|
Histogram mode code is not yet confirmed — a histogram event must be
|
||||||
revision of this code labelled them "Waveform" / "Waveform
|
captured with debug=true to identify it. Returns None for unknown codes.
|
||||||
(Continuous)" / "Waveform (Short)", which created the false
|
|
||||||
impression that there were three distinct event "types" the user
|
|
||||||
could configure. In reality the user only ever picks Single Shot
|
|
||||||
vs Continuous vs Histogram in the compliance config — the byte
|
|
||||||
layout is a firmware-internal detail that doesn't always correlate
|
|
||||||
with that choice.
|
|
||||||
"""
|
"""
|
||||||
fmt = _detect_record_format(data)
|
if len(data) < 2:
|
||||||
if fmt in ("single_shot", "continuous", "short"):
|
|
||||||
return "Waveform"
|
|
||||||
if len(data) >= 3:
|
|
||||||
log.warning(
|
|
||||||
"_extract_record_type: unrecognized header: data[0:3]=%02X %02X %02X",
|
|
||||||
data[0], data[1], data[2],
|
|
||||||
)
|
|
||||||
return f"Unknown({data[0]:02X}.{data[1]:02X}.{data[2]:02X})"
|
|
||||||
return None
|
return None
|
||||||
|
code = data[1]
|
||||||
|
if code == 0x10:
|
||||||
|
return "Waveform"
|
||||||
|
if code == 0x03:
|
||||||
|
# Continuous mode waveform record (confirmed by user — NOT a monitor log).
|
||||||
|
# The byte layout differs from 0x10 single-shot records: the timestamp
|
||||||
|
# fields decode as garbage under the 0x10 waveform layout.
|
||||||
|
# TODO: confirm correct timestamp layout for 0x03 records from a known-time event.
|
||||||
|
return "Waveform (Continuous)"
|
||||||
|
log.warning("_extract_record_type: unknown sub_code=0x%02X", code)
|
||||||
|
return f"Unknown(0x{code:02X})"
|
||||||
|
|
||||||
|
|
||||||
def _extract_peak_floats(data: bytes) -> Optional[PeakValues]:
|
def _extract_peak_floats(data: bytes) -> Optional[PeakValues]:
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -1,977 +0,0 @@
|
|||||||
"""
|
|
||||||
minimateplus/event_file_io.py — modern event-file (.sfm.json sidecar) IO.
|
|
||||||
|
|
||||||
This module is the single home for event-file conversion code that doesn't
|
|
||||||
fit cleanly inside `blastware_file.py` (which is the BW binary codec):
|
|
||||||
|
|
||||||
- sidecar JSON read/write (the modern per-event metadata file)
|
|
||||||
- read_blastware_file() — reverse of write_blastware_file, used by
|
|
||||||
the BW-importer flow when SFM is ingesting files produced by
|
|
||||||
Blastware's own ACH (where the source A5 frames aren't available).
|
|
||||||
|
|
||||||
Sidecar schema v1 layout — see docs in the project plan or the schema
|
|
||||||
declared in `event_to_sidecar_dict()`.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import datetime
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import logging
|
|
||||||
import os
|
|
||||||
import struct
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Optional, Union
|
|
||||||
|
|
||||||
from .models import Event, PeakValues, ProjectInfo, Timestamp
|
|
||||||
from . import blastware_file as _bw # avoid circular reference at module load
|
|
||||||
from .bw_ascii_report import BwAsciiReport
|
|
||||||
from . import waveform_codec as _wc
|
|
||||||
from .waveform_codec import decode_waveform_v2, decoded_to_adc_counts
|
|
||||||
from .histogram_codec import decode_histogram_body
|
|
||||||
|
|
||||||
# Reference pressure for dB(L) → psi conversion (20 µPa expressed in psi).
|
|
||||||
# Same constant as sfm/sfm_webapp.html so server-side and browser-side
|
|
||||||
# conversions agree.
|
|
||||||
_DBL_REF_PSI = 2.9e-9
|
|
||||||
|
|
||||||
log = logging.getLogger(__name__)
|
|
||||||
|
|
||||||
# Schema version for the sidecar JSON. Bump when fields change shape.
|
|
||||||
# Older readers must reject anything > SCHEMA_VERSION; newer fields added
|
|
||||||
# inside `extensions` are forward-compatible without a bump.
|
|
||||||
SCHEMA_VERSION = 1
|
|
||||||
SIDECAR_KIND = "sfm.event"
|
|
||||||
|
|
||||||
# Default tool_version stamp; callers can override. Hard-coded here
|
|
||||||
# rather than read via importlib.metadata because the latter reflects the
|
|
||||||
# *installed* dist-info, which doesn't update when pyproject.toml is
|
|
||||||
# bumped without a `pip install` re-run — leading to confusing stale
|
|
||||||
# version stamps in sidecars. Bump this constant and CHANGELOG.md
|
|
||||||
# together at release time.
|
|
||||||
TOOL_VERSION = "0.26.0"
|
|
||||||
|
|
||||||
try:
|
|
||||||
# Best-effort: prefer the installed metadata when it's NEWER than the
|
|
||||||
# baked-in constant (e.g. a downstream packager bumped the wheel
|
|
||||||
# without editing this file). Otherwise fall back to TOOL_VERSION.
|
|
||||||
from importlib.metadata import version as _pkg_version
|
|
||||||
_meta_v = _pkg_version("seismo-relay")
|
|
||||||
def _vtuple(s):
|
|
||||||
try:
|
|
||||||
return tuple(int(p) for p in s.split(".")[:3])
|
|
||||||
except Exception:
|
|
||||||
return (0, 0, 0)
|
|
||||||
_TOOL_VERSION_DEFAULT = (
|
|
||||||
_meta_v if _vtuple(_meta_v) > _vtuple(TOOL_VERSION) else TOOL_VERSION
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
_TOOL_VERSION_DEFAULT = TOOL_VERSION
|
|
||||||
|
|
||||||
|
|
||||||
# ── Sidecar dict construction ─────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def _ts_iso(ts: Optional[Timestamp]) -> Optional[str]:
|
|
||||||
if ts is None:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return datetime.datetime(
|
|
||||||
ts.year, ts.month, ts.day,
|
|
||||||
ts.hour or 0, ts.minute or 0, ts.second or 0,
|
|
||||||
).isoformat()
|
|
||||||
except Exception:
|
|
||||||
return str(ts)
|
|
||||||
|
|
||||||
|
|
||||||
def _peak_values_to_dict(pv: Optional[PeakValues]) -> dict:
|
|
||||||
if pv is None:
|
|
||||||
return {
|
|
||||||
"transverse": None,
|
|
||||||
"vertical": None,
|
|
||||||
"longitudinal": None,
|
|
||||||
"vector_sum": None,
|
|
||||||
"mic_psi": None,
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
"transverse": pv.tran,
|
|
||||||
"vertical": pv.vert,
|
|
||||||
"longitudinal": pv.long,
|
|
||||||
"vector_sum": pv.peak_vector_sum,
|
|
||||||
"mic_psi": pv.micl,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _bw_report_to_dict(report: BwAsciiReport) -> dict:
|
|
||||||
"""Project a parsed BW ASCII report into the sidecar's `bw_report` block.
|
|
||||||
|
|
||||||
All fields are rendered as plain JSON-compatible types (no datetime
|
|
||||||
objects). Channels are uniformly lowercased for stable JSON keys.
|
|
||||||
"""
|
|
||||||
def _ch(ch_name: str) -> dict:
|
|
||||||
cs = report.channels.get(ch_name)
|
|
||||||
if cs is None:
|
|
||||||
return {}
|
|
||||||
out = {
|
|
||||||
"ppv_ips": cs.ppv_ips,
|
|
||||||
"zc_freq_hz": cs.zc_freq_hz,
|
|
||||||
"time_of_peak_s": cs.time_of_peak_s,
|
|
||||||
"peak_accel_g": cs.peak_accel_g,
|
|
||||||
"peak_disp_in": cs.peak_disp_in,
|
|
||||||
}
|
|
||||||
# Drop all-None entries — keeps the JSON tidy for partial reports.
|
|
||||||
out = {k: v for k, v in out.items() if v is not None}
|
|
||||||
# Saturation flag (only present when True) — signals that ppv_ips
|
|
||||||
# is the channel range max (a lower bound), not an exact reading.
|
|
||||||
if getattr(cs, "ppv_saturated", False):
|
|
||||||
out["ppv_saturated"] = True
|
|
||||||
# ZC Freq above device reporting ceiling (BW ">100 Hz") — value
|
|
||||||
# in zc_freq_hz is the threshold, not an exact measurement.
|
|
||||||
if getattr(cs, "zc_freq_above_range", False):
|
|
||||||
out["zc_freq_above_range"] = True
|
|
||||||
return out
|
|
||||||
|
|
||||||
def _sc(ch_name: str) -> dict:
|
|
||||||
sc = report.sensor_check.get(ch_name)
|
|
||||||
if sc is None:
|
|
||||||
return {}
|
|
||||||
out = {
|
|
||||||
"freq_hz": sc.test_freq_hz,
|
|
||||||
"ratio": sc.test_ratio,
|
|
||||||
"amplitude_mv": sc.test_amplitude_mv,
|
|
||||||
"result": sc.test_results,
|
|
||||||
}
|
|
||||||
return {k: v for k, v in out.items() if v is not None}
|
|
||||||
|
|
||||||
monitor_log = []
|
|
||||||
for entry in report.monitor_log:
|
|
||||||
e = {
|
|
||||||
"start": entry.start_time.isoformat() if entry.start_time else None,
|
|
||||||
"stop": entry.stop_time.isoformat() if entry.stop_time else None,
|
|
||||||
"description": entry.description,
|
|
||||||
}
|
|
||||||
monitor_log.append({k: v for k, v in e.items() if v is not None})
|
|
||||||
|
|
||||||
return {
|
|
||||||
"available": True,
|
|
||||||
"event_type": report.event_type,
|
|
||||||
"version": report.version,
|
|
||||||
"trigger": {
|
|
||||||
"channel": report.trigger_channel,
|
|
||||||
"geo_level_ips": report.geo_trigger_level_ips,
|
|
||||||
},
|
|
||||||
"recording": {
|
|
||||||
"sample_rate_sps": report.sample_rate_sps,
|
|
||||||
"record_time_s": report.record_time_s,
|
|
||||||
"pretrig_s": report.pretrig_s,
|
|
||||||
"stop_mode": report.record_stop_mode,
|
|
||||||
"geo_range_ips": report.geo_range_ips,
|
|
||||||
"units": report.units,
|
|
||||||
},
|
|
||||||
"device": {
|
|
||||||
"battery_volts": report.battery_volts,
|
|
||||||
"calibration_date": report.calibration_date.isoformat() if report.calibration_date else None,
|
|
||||||
"calibration_by": report.calibration_by,
|
|
||||||
},
|
|
||||||
"peaks": {
|
|
||||||
"tran": _ch("Tran"),
|
|
||||||
"vert": _ch("Vert"),
|
|
||||||
"long": _ch("Long"),
|
|
||||||
"vector_sum": {
|
|
||||||
"ips": report.peak_vector_sum_ips,
|
|
||||||
"time_s": report.peak_vector_sum_time_s,
|
|
||||||
# Histogram events have an absolute date+time for the PVS
|
|
||||||
# (the interval at which it occurred); waveform events
|
|
||||||
# only have the time_s offset.
|
|
||||||
"when": report.peak_vector_sum_when.isoformat() if report.peak_vector_sum_when else None,
|
|
||||||
# Set when BW reported the PVS as OORANGE — value is the
|
|
||||||
# conservative upper bound sqrt(3) * geo_range_ips, not
|
|
||||||
# an exact peak.
|
|
||||||
"saturated": bool(getattr(report, "peak_vector_sum_saturated", False)),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
"mic": {
|
|
||||||
"weighting": report.mic.weighting,
|
|
||||||
"pspl_dbl": report.mic.pspl_dbl,
|
|
||||||
"pspl_saturated": bool(getattr(report.mic, "pspl_saturated", False)),
|
|
||||||
"zc_freq_hz": report.mic.zc_freq_hz,
|
|
||||||
"zc_freq_above_range": bool(getattr(report.mic, "zc_freq_above_range", False)),
|
|
||||||
"time_of_peak_s": report.mic.time_of_peak_s,
|
|
||||||
},
|
|
||||||
"sensor_check": {
|
|
||||||
"tran": _sc("Tran"),
|
|
||||||
"vert": _sc("Vert"),
|
|
||||||
"long": _sc("Long"),
|
|
||||||
"mic": _sc("MicL"),
|
|
||||||
},
|
|
||||||
# Histogram-specific fields (None on waveform-mode events).
|
|
||||||
# Per-channel absolute peak time/date for histograms — for
|
|
||||||
# waveforms see channels[ch]["time_of_peak_s"] instead.
|
|
||||||
"histogram": {
|
|
||||||
"start": report.histogram_start.isoformat() if report.histogram_start else None,
|
|
||||||
"stop": report.histogram_stop.isoformat() if report.histogram_stop else None,
|
|
||||||
"n_intervals": report.histogram_n_intervals,
|
|
||||||
"interval_size": report.histogram_interval_size_str,
|
|
||||||
"interval_size_s": report.histogram_interval_size_s,
|
|
||||||
"channel_peak_when": {ch: dt.isoformat() for ch, dt in report.channel_peak_when.items()},
|
|
||||||
},
|
|
||||||
"monitor_log": monitor_log,
|
|
||||||
"pc_sw_version": report.pc_sw_version,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _dbl_to_psi(pspl_dbl: float) -> float:
|
|
||||||
"""Convert dB(L) sound pressure level back to psi. Uses the same
|
|
||||||
20 µPa reference (= 2.9e-9 psi) as the webapp so server-side and
|
|
||||||
browser-side conversions agree."""
|
|
||||||
return _DBL_REF_PSI * (10.0 ** (pspl_dbl / 20.0))
|
|
||||||
|
|
||||||
|
|
||||||
def apply_report_to_event(event: Event, report: BwAsciiReport) -> None:
|
|
||||||
"""Overlay device-authoritative fields from a parsed BW ASCII report
|
|
||||||
onto an in-memory Event, IN-PLACE.
|
|
||||||
|
|
||||||
Why this exists
|
|
||||||
───────────────
|
|
||||||
`read_blastware_file()` parses the BW binary and fills `Event.peak_values`
|
|
||||||
via `_peaks_from_samples()` — which runs the (still-undecoded) BW body
|
|
||||||
codec assuming raw int16 LE and produces ±32K-shaped noise on every
|
|
||||||
channel. Result: peak values land in the SeismoDb event row as
|
|
||||||
~10 in/s on every event regardless of the actual signal.
|
|
||||||
|
|
||||||
When a paired BW ASCII report is available, the report carries the
|
|
||||||
device's own authoritative peak / project / sample-rate / record-time
|
|
||||||
values. This helper folds those onto the Event before it flows to
|
|
||||||
`SeismoDb.insert_events()`, so the DB columns reflect the report
|
|
||||||
rather than the broken-codec output.
|
|
||||||
|
|
||||||
Fields overlaid (only when the report supplies a non-None value):
|
|
||||||
- peak_values.tran / .vert / .long (from report.channels)
|
|
||||||
- peak_values.peak_vector_sum (from report.peak_vector_sum_ips)
|
|
||||||
- peak_values.micl (psi) (from report.mic.pspl_dbl → psi)
|
|
||||||
- project_info.project / .client / .operator / .sensor_location
|
|
||||||
- sample_rate (from report.sample_rate_sps)
|
|
||||||
- rectime_seconds (from report.record_time_s)
|
|
||||||
|
|
||||||
Fields NOT touched (operator-edit / parser-output preserved):
|
|
||||||
- timestamp, raw_samples, record_type, total_samples,
|
|
||||||
pretrig_samples, _waveform_key, _a5_frames, _raw_record
|
|
||||||
- false_trigger and review state (those live on the sidecar, not on Event)
|
|
||||||
"""
|
|
||||||
if event.peak_values is None:
|
|
||||||
event.peak_values = PeakValues()
|
|
||||||
pv = event.peak_values
|
|
||||||
ch = report.channels
|
|
||||||
if (t := ch.get("Tran")) and t.ppv_ips is not None: pv.tran = t.ppv_ips
|
|
||||||
if (v := ch.get("Vert")) and v.ppv_ips is not None: pv.vert = v.ppv_ips
|
|
||||||
if (l := ch.get("Long")) and l.ppv_ips is not None: pv.long = l.ppv_ips
|
|
||||||
if report.peak_vector_sum_ips is not None:
|
|
||||||
pv.peak_vector_sum = report.peak_vector_sum_ips
|
|
||||||
if report.mic.pspl_dbl is not None and report.mic.pspl_dbl > 0:
|
|
||||||
pv.micl = _dbl_to_psi(report.mic.pspl_dbl)
|
|
||||||
|
|
||||||
if (t := ch.get("Tran")):
|
|
||||||
pv.tran_zc_freq = t.zc_freq_hz
|
|
||||||
pv.tran_zc_above_range = bool(getattr(t, "zc_freq_above_range", False))
|
|
||||||
if (v := ch.get("Vert")):
|
|
||||||
pv.vert_zc_freq = v.zc_freq_hz
|
|
||||||
pv.vert_zc_above_range = bool(getattr(v, "zc_freq_above_range", False))
|
|
||||||
if (l := ch.get("Long")):
|
|
||||||
pv.long_zc_freq = l.zc_freq_hz
|
|
||||||
pv.long_zc_above_range = bool(getattr(l, "zc_freq_above_range", False))
|
|
||||||
if report.mic is not None:
|
|
||||||
pv.mic_zc_freq = report.mic.zc_freq_hz
|
|
||||||
pv.mic_zc_above_range = bool(getattr(report.mic, "zc_freq_above_range", False))
|
|
||||||
|
|
||||||
if event.project_info is None:
|
|
||||||
event.project_info = ProjectInfo()
|
|
||||||
pi = event.project_info
|
|
||||||
if report.project: pi.project = report.project
|
|
||||||
if report.client: pi.client = report.client
|
|
||||||
if report.operator: pi.operator = report.operator
|
|
||||||
if report.sensor_location: pi.sensor_location = report.sensor_location
|
|
||||||
|
|
||||||
if report.sample_rate_sps:
|
|
||||||
event.sample_rate = report.sample_rate_sps
|
|
||||||
if report.record_time_s is not None:
|
|
||||||
event.rectime_seconds = report.record_time_s
|
|
||||||
|
|
||||||
|
|
||||||
def apply_bw_report_dict_to_event(event: Event, bw_report: dict) -> None:
|
|
||||||
"""Mirror of ``apply_report_to_event`` for the projected sidecar
|
|
||||||
dict shape (as produced by ``_bw_report_to_dict``).
|
|
||||||
|
|
||||||
Why this exists
|
|
||||||
───────────────
|
|
||||||
The ingest path holds a live ``BwAsciiReport`` parsed straight from
|
|
||||||
the ``_ASCII.TXT`` and uses ``apply_report_to_event`` to overlay
|
|
||||||
device-authoritative peaks onto the codec output before insert.
|
|
||||||
|
|
||||||
The backfill path doesn't have the original ``.TXT`` (it's not
|
|
||||||
retained in the waveform store), but it does have the preserved
|
|
||||||
``bw_report`` block from the sidecar — which contains the same
|
|
||||||
projected fields. Re-overlaying those during a backfill keeps the
|
|
||||||
DB peak columns aligned with what BW reports rather than letting
|
|
||||||
the codec output (which may be incomplete for unhandled formats or
|
|
||||||
walker edge cases) win by default.
|
|
||||||
|
|
||||||
No-ops cleanly when ``bw_report`` is ``None``, empty, or missing
|
|
||||||
any particular sub-field — only fields with a concrete value get
|
|
||||||
written. Mirrors ``apply_report_to_event``'s "report wins where
|
|
||||||
present" semantics.
|
|
||||||
"""
|
|
||||||
if not bw_report:
|
|
||||||
return
|
|
||||||
if event.peak_values is None:
|
|
||||||
event.peak_values = PeakValues()
|
|
||||||
pv = event.peak_values
|
|
||||||
|
|
||||||
peaks = bw_report.get("peaks") or {}
|
|
||||||
tran = (peaks.get("tran") or {}).get("ppv_ips")
|
|
||||||
vert = (peaks.get("vert") or {}).get("ppv_ips")
|
|
||||||
long = (peaks.get("long") or {}).get("ppv_ips")
|
|
||||||
if tran is not None: pv.tran = tran
|
|
||||||
if vert is not None: pv.vert = vert
|
|
||||||
if long is not None: pv.long = long
|
|
||||||
vs_ips = (peaks.get("vector_sum") or {}).get("ips")
|
|
||||||
if vs_ips is not None:
|
|
||||||
pv.peak_vector_sum = vs_ips
|
|
||||||
|
|
||||||
mic = bw_report.get("mic") or {}
|
|
||||||
pspl = mic.get("pspl_dbl")
|
|
||||||
if pspl is not None and pspl > 0:
|
|
||||||
pv.micl = _dbl_to_psi(pspl)
|
|
||||||
|
|
||||||
for axis, freq_attr, above_attr in (
|
|
||||||
("tran", "tran_zc_freq", "tran_zc_above_range"),
|
|
||||||
("vert", "vert_zc_freq", "vert_zc_above_range"),
|
|
||||||
("long", "long_zc_freq", "long_zc_above_range"),
|
|
||||||
):
|
|
||||||
chd = peaks.get(axis) or {}
|
|
||||||
setattr(pv, freq_attr, chd.get("zc_freq_hz"))
|
|
||||||
setattr(pv, above_attr, bool(chd.get("zc_freq_above_range", False)))
|
|
||||||
pv.mic_zc_freq = mic.get("zc_freq_hz")
|
|
||||||
pv.mic_zc_above_range = bool(mic.get("zc_freq_above_range", False))
|
|
||||||
|
|
||||||
rec = bw_report.get("recording") or {}
|
|
||||||
sr = rec.get("sample_rate_sps")
|
|
||||||
if sr:
|
|
||||||
event.sample_rate = sr
|
|
||||||
rt = rec.get("record_time_s")
|
|
||||||
if rt is not None:
|
|
||||||
event.rectime_seconds = rt
|
|
||||||
|
|
||||||
|
|
||||||
def _project_info_to_dict(pi: Optional[ProjectInfo]) -> dict:
|
|
||||||
if pi is None:
|
|
||||||
return {
|
|
||||||
"project": None,
|
|
||||||
"client": None,
|
|
||||||
"operator": None,
|
|
||||||
"sensor_location": None,
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
"project": pi.project,
|
|
||||||
"client": pi.client,
|
|
||||||
"operator": pi.operator,
|
|
||||||
"sensor_location": pi.sensor_location,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def event_to_sidecar_dict(
|
|
||||||
event: Event,
|
|
||||||
*,
|
|
||||||
serial: str,
|
|
||||||
blastware_filename: str,
|
|
||||||
blastware_filesize: int,
|
|
||||||
blastware_sha256: str,
|
|
||||||
source_kind: str = "sfm-live",
|
|
||||||
txt_filename: Optional[str] = None,
|
|
||||||
a5_pickle_filename: Optional[str] = None,
|
|
||||||
tool_version: str = _TOOL_VERSION_DEFAULT,
|
|
||||||
captured_at: Optional[datetime.datetime] = None,
|
|
||||||
review: Optional[dict] = None,
|
|
||||||
extensions: Optional[dict] = None,
|
|
||||||
bw_report: Optional[BwAsciiReport] = None,
|
|
||||||
) -> dict:
|
|
||||||
"""
|
|
||||||
Build a v1 sidecar dict from an Event + the surrounding metadata.
|
|
||||||
|
|
||||||
Pure helper — no file I/O. Callers stitch the result into a sidecar
|
|
||||||
via `write_sidecar()` (or POST it back via the PATCH endpoint).
|
|
||||||
|
|
||||||
When *bw_report* is supplied (e.g. by the ACH-forwarded import path
|
|
||||||
where Blastware writes a per-event ASCII report alongside the binary),
|
|
||||||
its decoded fields are folded into the sidecar:
|
|
||||||
|
|
||||||
- A new top-level ``bw_report`` block carries the rich derived
|
|
||||||
per-channel stats (Peak Acceleration, Peak Displacement, ZC Freq,
|
|
||||||
Time of Peak), the Peak Vector Sum + time, the per-channel sensor
|
|
||||||
self-check results, and monitor-log timestamps.
|
|
||||||
- ``peak_values`` is overlaid from the report (the report's PPV/PVS
|
|
||||||
values are computed by the device firmware and are authoritative;
|
|
||||||
anything ``read_blastware_file()`` derived from samples is
|
|
||||||
approximate at best until the body codec is decoded).
|
|
||||||
- ``project_info`` is overlaid from the report when the report
|
|
||||||
supplies a non-empty value (the report mirrors the device's
|
|
||||||
compliance config, which is what BW shows in its event report).
|
|
||||||
- ``event.timestamp`` is overlaid from the report's Event Date +
|
|
||||||
Event Time (BW's report timestamps are second-resolution and
|
|
||||||
match the binary's footer; we prefer the report value because
|
|
||||||
the BW-binary footer timestamp can drift on some firmware).
|
|
||||||
"""
|
|
||||||
if source_kind not in {"sfm-live", "sfm-ach", "bw-import", "idf-import"}:
|
|
||||||
raise ValueError(f"unknown source_kind: {source_kind!r}")
|
|
||||||
|
|
||||||
captured_at = captured_at or datetime.datetime.utcnow()
|
|
||||||
|
|
||||||
# ── Overlay event fields from the report when present ───────────────────
|
|
||||||
timestamp_iso = _ts_iso(event.timestamp)
|
|
||||||
if bw_report and bw_report.event_datetime:
|
|
||||||
timestamp_iso = bw_report.event_datetime.isoformat()
|
|
||||||
|
|
||||||
# Build peak_values, optionally overlaid from the report. The report
|
|
||||||
# stores Mic peak as PSPL (dB(L)); we convert to psi to match the
|
|
||||||
# existing peak_values.mic_psi field.
|
|
||||||
peak_dict = _peak_values_to_dict(event.peak_values)
|
|
||||||
if bw_report:
|
|
||||||
ch = bw_report.channels
|
|
||||||
if (t := ch.get("Tran")) and t.ppv_ips is not None: peak_dict["transverse"] = t.ppv_ips
|
|
||||||
if (v := ch.get("Vert")) and v.ppv_ips is not None: peak_dict["vertical"] = v.ppv_ips
|
|
||||||
if (l := ch.get("Long")) and l.ppv_ips is not None: peak_dict["longitudinal"] = l.ppv_ips
|
|
||||||
if bw_report.peak_vector_sum_ips is not None:
|
|
||||||
peak_dict["vector_sum"] = bw_report.peak_vector_sum_ips
|
|
||||||
if bw_report.mic.pspl_dbl is not None and bw_report.mic.pspl_dbl > 0:
|
|
||||||
peak_dict["mic_psi"] = _dbl_to_psi(bw_report.mic.pspl_dbl)
|
|
||||||
|
|
||||||
# Project info: overlay from report (the report mirrors the
|
|
||||||
# session-start compliance config that BW renders in event reports).
|
|
||||||
proj_dict = _project_info_to_dict(event.project_info)
|
|
||||||
if bw_report:
|
|
||||||
if bw_report.project: proj_dict["project"] = bw_report.project
|
|
||||||
if bw_report.client: proj_dict["client"] = bw_report.client
|
|
||||||
if bw_report.operator: proj_dict["operator"] = bw_report.operator
|
|
||||||
if bw_report.sensor_location: proj_dict["sensor_location"] = bw_report.sensor_location
|
|
||||||
|
|
||||||
# Event-block fields: overlay from report where available.
|
|
||||||
event_block = {
|
|
||||||
"serial": serial,
|
|
||||||
"timestamp": timestamp_iso,
|
|
||||||
"waveform_key": event._waveform_key.hex() if event._waveform_key else None,
|
|
||||||
"record_type": event.record_type,
|
|
||||||
"sample_rate": event.sample_rate,
|
|
||||||
"rectime_seconds": event.rectime_seconds,
|
|
||||||
"total_samples": event.total_samples,
|
|
||||||
"pretrig_samples": event.pretrig_samples,
|
|
||||||
}
|
|
||||||
if bw_report:
|
|
||||||
# Report values are authoritative — they're the user-configured
|
|
||||||
# values BW reads back, not STRT-derived guesses. In particular
|
|
||||||
# `event.rectime_seconds` from `read_blastware_file()` reads
|
|
||||||
# STRT[18] which is actually the `0x46` record-type marker (= 70)
|
|
||||||
# rather than the user's Record Time setting. Always overwrite.
|
|
||||||
if bw_report.sample_rate_sps:
|
|
||||||
event_block["sample_rate"] = bw_report.sample_rate_sps
|
|
||||||
if bw_report.record_time_s is not None:
|
|
||||||
event_block["rectime_seconds"] = bw_report.record_time_s
|
|
||||||
# Derive total_samples + pretrig_samples per channel from the
|
|
||||||
# report's sample_rate × times. These match the row count of
|
|
||||||
# the report's sample table (verified: event-c reports 1024 sps
|
|
||||||
# × (1.0 + 0.25) = 1280 rows).
|
|
||||||
if (sr := bw_report.sample_rate_sps) and bw_report.record_time_s is not None:
|
|
||||||
pretrig_s = abs(bw_report.pretrig_s) if bw_report.pretrig_s is not None else 0.0
|
|
||||||
event_block["total_samples"] = int(round(sr * (bw_report.record_time_s + pretrig_s)))
|
|
||||||
event_block["pretrig_samples"] = int(round(sr * pretrig_s))
|
|
||||||
|
|
||||||
out = {
|
|
||||||
"schema_version": SCHEMA_VERSION,
|
|
||||||
"kind": SIDECAR_KIND,
|
|
||||||
|
|
||||||
"event": event_block,
|
|
||||||
"peak_values": peak_dict,
|
|
||||||
"project_info": proj_dict,
|
|
||||||
|
|
||||||
"blastware": {
|
|
||||||
"filename": blastware_filename,
|
|
||||||
"filesize": blastware_filesize,
|
|
||||||
"sha256": blastware_sha256,
|
|
||||||
"available": True,
|
|
||||||
},
|
|
||||||
|
|
||||||
"source": {
|
|
||||||
"kind": source_kind,
|
|
||||||
"captured_at": captured_at.isoformat() + "Z" if captured_at.tzinfo is None else captured_at.isoformat(),
|
|
||||||
"tool_version": tool_version,
|
|
||||||
"a5_pickle_filename": a5_pickle_filename,
|
|
||||||
"txt_filename": txt_filename,
|
|
||||||
},
|
|
||||||
|
|
||||||
"review": review or {
|
|
||||||
"false_trigger": False,
|
|
||||||
"reviewer": None,
|
|
||||||
"reviewed_at": None,
|
|
||||||
"notes": "",
|
|
||||||
},
|
|
||||||
|
|
||||||
"extensions": extensions or {},
|
|
||||||
}
|
|
||||||
|
|
||||||
if bw_report:
|
|
||||||
out["bw_report"] = _bw_report_to_dict(bw_report)
|
|
||||||
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
# ── Sidecar IO ────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def write_sidecar(path: Union[str, Path], data: dict) -> None:
|
|
||||||
"""
|
|
||||||
Atomic write of a sidecar dict to <path>.
|
|
||||||
|
|
||||||
Validates schema_version is supported before writing so we don't
|
|
||||||
silently drop a future-format sidecar over the wire.
|
|
||||||
"""
|
|
||||||
path = Path(path)
|
|
||||||
sv = data.get("schema_version")
|
|
||||||
if not isinstance(sv, int) or sv < 1 or sv > SCHEMA_VERSION:
|
|
||||||
raise ValueError(
|
|
||||||
f"write_sidecar: unsupported schema_version={sv!r} "
|
|
||||||
f"(this build supports 1..{SCHEMA_VERSION})"
|
|
||||||
)
|
|
||||||
|
|
||||||
tmp = path.with_suffix(path.suffix + ".tmp")
|
|
||||||
with tmp.open("w", encoding="utf-8") as f:
|
|
||||||
json.dump(data, f, indent=2, sort_keys=False, default=str)
|
|
||||||
f.write("\n")
|
|
||||||
f.flush()
|
|
||||||
os.fsync(f.fileno())
|
|
||||||
os.replace(tmp, path)
|
|
||||||
|
|
||||||
|
|
||||||
def read_sidecar(path: Union[str, Path]) -> dict:
|
|
||||||
"""
|
|
||||||
Load a sidecar JSON file.
|
|
||||||
|
|
||||||
Raises FileNotFoundError if missing, ValueError on bad shape /
|
|
||||||
unsupported schema_version. Unknown keys at the top level are
|
|
||||||
preserved in the returned dict (forward-compat).
|
|
||||||
"""
|
|
||||||
path = Path(path)
|
|
||||||
with path.open("r", encoding="utf-8") as f:
|
|
||||||
data = json.load(f)
|
|
||||||
if not isinstance(data, dict):
|
|
||||||
raise ValueError(f"sidecar at {path}: top-level is not a JSON object")
|
|
||||||
sv = data.get("schema_version")
|
|
||||||
if not isinstance(sv, int) or sv < 1:
|
|
||||||
raise ValueError(f"sidecar at {path}: missing or invalid schema_version")
|
|
||||||
if sv > SCHEMA_VERSION:
|
|
||||||
raise ValueError(
|
|
||||||
f"sidecar at {path}: schema_version={sv} > supported {SCHEMA_VERSION}; "
|
|
||||||
"upgrade seismo-relay to read this file"
|
|
||||||
)
|
|
||||||
if data.get("kind") != SIDECAR_KIND:
|
|
||||||
raise ValueError(f"sidecar at {path}: unexpected kind={data.get('kind')!r}")
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def patch_sidecar(
|
|
||||||
path: Union[str, Path],
|
|
||||||
*,
|
|
||||||
review: Optional[dict] = None,
|
|
||||||
extensions: Optional[dict] = None,
|
|
||||||
reviewer_now: bool = True,
|
|
||||||
) -> dict:
|
|
||||||
"""
|
|
||||||
Atomically apply a JSON-merge-patch to a sidecar file's `review`
|
|
||||||
and/or `extensions` blocks. Other top-level keys are untouched.
|
|
||||||
|
|
||||||
`review_now`: when True (default) and `review` is non-empty, stamps
|
|
||||||
`review.reviewed_at` with the current UTC time so the review-time is
|
|
||||||
auditable without the caller having to pass it.
|
|
||||||
|
|
||||||
Returns the new full sidecar dict.
|
|
||||||
"""
|
|
||||||
path = Path(path)
|
|
||||||
data = read_sidecar(path)
|
|
||||||
|
|
||||||
if review:
|
|
||||||
merged = dict(data.get("review") or {})
|
|
||||||
merged.update({k: v for k, v in review.items() if v is not None or k in merged})
|
|
||||||
if reviewer_now:
|
|
||||||
merged["reviewed_at"] = datetime.datetime.utcnow().isoformat() + "Z"
|
|
||||||
data["review"] = merged
|
|
||||||
|
|
||||||
if extensions:
|
|
||||||
merged_ext = dict(data.get("extensions") or {})
|
|
||||||
merged_ext.update(extensions)
|
|
||||||
data["extensions"] = merged_ext
|
|
||||||
|
|
||||||
write_sidecar(path, data)
|
|
||||||
return data
|
|
||||||
|
|
||||||
|
|
||||||
def sidecar_path_for(blastware_path: Union[str, Path]) -> Path:
|
|
||||||
"""Convention: <bw_path>.sfm.json sits next to the BW binary."""
|
|
||||||
p = Path(blastware_path)
|
|
||||||
return p.with_name(p.name + ".sfm.json")
|
|
||||||
|
|
||||||
|
|
||||||
def file_sha256(path: Union[str, Path], chunk_size: int = 65536) -> str:
|
|
||||||
"""Compute SHA-256 of a file as a hex string."""
|
|
||||||
h = hashlib.sha256()
|
|
||||||
with open(path, "rb") as f:
|
|
||||||
while True:
|
|
||||||
chunk = f.read(chunk_size)
|
|
||||||
if not chunk:
|
|
||||||
break
|
|
||||||
h.update(chunk)
|
|
||||||
return h.hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
# ── Blastware-file reader ─────────────────────────────────────────────────────
|
|
||||||
#
|
|
||||||
# Reverse of `blastware_file.write_blastware_file`. Used by the BW-import
|
|
||||||
# flow to ingest files produced by Blastware's own ACH (where the source
|
|
||||||
# A5 frames are not available).
|
|
||||||
#
|
|
||||||
# File structure (recap):
|
|
||||||
# [22B header] [21B STRT record] [body bytes] [26B footer]
|
|
||||||
#
|
|
||||||
# The body holds:
|
|
||||||
# - 6B preamble (00 00 ff ff ff ff) immediately after the STRT
|
|
||||||
# - 4-channel interleaved int16 LE samples
|
|
||||||
# - Embedded ASCII metadata strings (Project: / Client: / User Name: /
|
|
||||||
# Seis Loc: / Extended Notes) from the device's session-start config
|
|
||||||
#
|
|
||||||
# The 0C waveform record (per-event peaks, project name) is NOT in the
|
|
||||||
# BW file — those are computed by the device firmware and only carried
|
|
||||||
# in the live SUB 0C response. read_blastware_file() therefore computes
|
|
||||||
# peaks from the raw samples assuming Normal-range (10 in/s full-scale)
|
|
||||||
# geophone sensitivity. Imported events surface that assumption via the
|
|
||||||
# sidecar's `peak_values.computed_from_samples` flag.
|
|
||||||
|
|
||||||
|
|
||||||
# Geophone scale factor, in/s per ADC unit, for Normal range (10 in/s FS).
|
|
||||||
# Confirmed from CLAUDE.md (geo_hardware_constant = 6.206053 in/s per V,
|
|
||||||
# ADC full-scale = 1.61133 V Normal range = 10.0 in/s peak; per-count
|
|
||||||
# resolution ≈ 10.0 / 32768).
|
|
||||||
_GEO_NORMAL_FS_INS = 10.0
|
|
||||||
_GEO_SENSITIVE_FS_INS = 1.250
|
|
||||||
_INT16_FS = 32768.0
|
|
||||||
# Geophone full-scale count — 32000, not 32768. One decoder unit (16 ADC
|
|
||||||
# counts) is exactly 0.005 in/s, so 10.000 in/s = 32000 counts. Must match
|
|
||||||
# sfm.event_hdf5._GEO_INT16_FS or sidecar peaks disagree with the plotted
|
|
||||||
# waveform by 2.3%. Confirmed 2026-08-25 against the BW ASCII corpus.
|
|
||||||
_GEO_INT16_FS = 32000.0
|
|
||||||
|
|
||||||
# Microphone scale factor, psi per ADC count. Approximate — exact factor
|
|
||||||
# depends on the geophone-vs-mic ADC scaling and the firmware reference.
|
|
||||||
# We mark mic_psi as "computed approximate" in the sidecar.
|
|
||||||
_MIC_FS_PSI = 0.0125 / _INT16_FS # ~0.5 psi full-scale assumption
|
|
||||||
|
|
||||||
|
|
||||||
def _decode_strt(strt: bytes) -> dict:
|
|
||||||
"""
|
|
||||||
Decode the 21-byte STRT record from a BW file.
|
|
||||||
|
|
||||||
Returns dict with waveform_key (4B), total_samples, pretrig_samples,
|
|
||||||
rectime_seconds. Falls back to None on truncated/missing fields.
|
|
||||||
"""
|
|
||||||
if len(strt) < 21 or strt[0:4] != b"STRT":
|
|
||||||
return {}
|
|
||||||
return {
|
|
||||||
"waveform_key": strt[6:10].hex(),
|
|
||||||
"total_samples": struct.unpack_from(">H", strt, 8)[0],
|
|
||||||
"pretrig_samples": struct.unpack_from(">H", strt, 16)[0],
|
|
||||||
"rectime_seconds": strt[18],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _find_first_string(buf: bytes, label: bytes, max_len: int = 256) -> Optional[str]:
|
|
||||||
"""
|
|
||||||
Search `buf` for `label` (e.g. b"Project:") and return the
|
|
||||||
null-terminated ASCII string that follows, stripped.
|
|
||||||
"""
|
|
||||||
pos = buf.find(label)
|
|
||||||
if pos < 0:
|
|
||||||
return None
|
|
||||||
start = pos + len(label)
|
|
||||||
end = buf.find(b"\x00", start, start + max_len)
|
|
||||||
if end < 0:
|
|
||||||
end = start + max_len
|
|
||||||
text = buf[start:end].decode("ascii", errors="replace").strip()
|
|
||||||
return text or None
|
|
||||||
|
|
||||||
|
|
||||||
def _decode_samples_4ch_int16_le(stream: bytes) -> dict[str, list[int]]:
|
|
||||||
"""
|
|
||||||
Decode a 4-channel interleaved int16 LE byte stream into per-channel
|
|
||||||
lists. Channels are [Tran, Vert, Long, Mic] = [ch0, ch1, ch2, ch3].
|
|
||||||
Truncates to a multiple of 8 bytes (one full sample-set).
|
|
||||||
"""
|
|
||||||
n_complete = (len(stream) // 8) * 8
|
|
||||||
if n_complete == 0:
|
|
||||||
return {"Tran": [], "Vert": [], "Long": [], "MicL": []}
|
|
||||||
fmt = "<" + "h" * (n_complete // 2)
|
|
||||||
flat = list(struct.unpack(fmt, stream[:n_complete]))
|
|
||||||
return {
|
|
||||||
"Tran": flat[0::4],
|
|
||||||
"Vert": flat[1::4],
|
|
||||||
"Long": flat[2::4],
|
|
||||||
"MicL": flat[3::4],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _peaks_from_samples(samples: dict[str, list[int]]) -> PeakValues:
|
|
||||||
"""
|
|
||||||
Compute approximate peaks from raw int16 samples assuming Normal-range
|
|
||||||
geophone sensitivity. Used by the BW-importer when the 0C waveform
|
|
||||||
record (the device's authoritative peaks) is unavailable.
|
|
||||||
"""
|
|
||||||
def _peak_ins(ch: list[int]) -> float:
|
|
||||||
if not ch:
|
|
||||||
return 0.0
|
|
||||||
m = max(abs(int(v)) for v in ch)
|
|
||||||
return m / _GEO_INT16_FS * _GEO_NORMAL_FS_INS
|
|
||||||
|
|
||||||
tran = _peak_ins(samples.get("Tran", []))
|
|
||||||
vert = _peak_ins(samples.get("Vert", []))
|
|
||||||
long_ = _peak_ins(samples.get("Long", []))
|
|
||||||
|
|
||||||
# Mic in psi (approximate)
|
|
||||||
mic_ch = samples.get("MicL", []) or []
|
|
||||||
mic = max((abs(int(v)) for v in mic_ch), default=0) * _MIC_FS_PSI
|
|
||||||
|
|
||||||
# Peak vector sum: max over time of sqrt(T^2 + V^2 + L^2)
|
|
||||||
pvs = 0.0
|
|
||||||
n = min(len(samples.get("Tran", [])), len(samples.get("Vert", [])), len(samples.get("Long", [])))
|
|
||||||
if n:
|
|
||||||
scale = _GEO_NORMAL_FS_INS / _GEO_INT16_FS
|
|
||||||
T = samples["Tran"]; V = samples["Vert"]; L = samples["Long"]
|
|
||||||
for i in range(n):
|
|
||||||
t = T[i] * scale
|
|
||||||
v = V[i] * scale
|
|
||||||
l = L[i] * scale
|
|
||||||
mag = (t*t + v*v + l*l) ** 0.5
|
|
||||||
if mag > pvs:
|
|
||||||
pvs = mag
|
|
||||||
|
|
||||||
return PeakValues(
|
|
||||||
tran=tran, vert=vert, long=long_,
|
|
||||||
peak_vector_sum=pvs, micl=mic,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
_RECORD_TYPE_BY_EXT_SUFFIX = {
|
|
||||||
'H': 'Histogram',
|
|
||||||
'W': 'Waveform',
|
|
||||||
'M': 'Manual',
|
|
||||||
'E': 'Event',
|
|
||||||
'C': 'Combo',
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def derive_record_type_from_filename(filename, default: str = "Waveform") -> str:
|
|
||||||
"""Derive a BW Event's record_type from its filename's extension suffix.
|
|
||||||
|
|
||||||
V10.72+ MiniMate Plus firmware encodes the event type as the LAST
|
|
||||||
character of the extension (the `T` in BW's `AB0T` scheme):
|
|
||||||
|
|
||||||
``M529LKIQ.G10H`` → H → ``"Histogram"``
|
|
||||||
``T350L385.VY0W`` → W → ``"Waveform"``
|
|
||||||
``...M`` → M → ``"Manual"``
|
|
||||||
``...E`` → E → ``"Event"``
|
|
||||||
``...C`` → C → ``"Combo"``
|
|
||||||
|
|
||||||
Old S338 firmware uses 3-char extensions ending in ``0`` whose
|
|
||||||
encoding is not yet known — those fall through to ``default``.
|
|
||||||
Micromate Series 4 uses a different scheme entirely (observed:
|
|
||||||
``IDFH``, ``IDFW``) but the LAST-char convention (H / W) still holds
|
|
||||||
for the type code, so it works for both families.
|
|
||||||
|
|
||||||
Returns ``default`` if filename is empty, has no extension, or the
|
|
||||||
suffix char isn't a recognized type code.
|
|
||||||
"""
|
|
||||||
if not filename:
|
|
||||||
return default
|
|
||||||
try:
|
|
||||||
name = Path(filename).name
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return default
|
|
||||||
if '.' not in name:
|
|
||||||
return default
|
|
||||||
ext = name.rsplit('.', 1)[1]
|
|
||||||
if not ext:
|
|
||||||
return default
|
|
||||||
return _RECORD_TYPE_BY_EXT_SUFFIX.get(ext[-1].upper(), default)
|
|
||||||
|
|
||||||
|
|
||||||
def read_blastware_file(path: Union[str, Path]) -> Event:
|
|
||||||
"""
|
|
||||||
Parse a Blastware waveform file into an Event.
|
|
||||||
|
|
||||||
Recovers:
|
|
||||||
- waveform_key, rectime_seconds, total_samples, pretrig_samples
|
|
||||||
(from the STRT record)
|
|
||||||
- timestamp (from the footer's start-time field)
|
|
||||||
- project_info (from ASCII labels embedded in the body)
|
|
||||||
- raw_samples (Tran/Vert/Long/MicL int16 lists)
|
|
||||||
- peak_values (computed from raw_samples; approximate — see notes
|
|
||||||
on _peaks_from_samples about Normal-range assumption)
|
|
||||||
|
|
||||||
Does NOT recover the source A5 frames (they aren't in the BW file).
|
|
||||||
The returned Event has `_a5_frames = None`, signalling that
|
|
||||||
byte-for-byte regeneration of the BW file from this Event alone is
|
|
||||||
not possible — the on-disk BW file IS the byte-for-byte source.
|
|
||||||
"""
|
|
||||||
path = Path(path)
|
|
||||||
raw = path.read_bytes()
|
|
||||||
if len(raw) < _bw._WAVEFORM_HEADER_SIZE + 21 + 26:
|
|
||||||
raise ValueError(f"{path}: file too short ({len(raw)} bytes) to be a BW event")
|
|
||||||
|
|
||||||
# Header: validate magic prefix.
|
|
||||||
header = raw[:_bw._WAVEFORM_HEADER_SIZE]
|
|
||||||
if not header.startswith(_bw._FILE_HEADER_PREFIX):
|
|
||||||
raise ValueError(f"{path}: not a Blastware file (bad header prefix)")
|
|
||||||
|
|
||||||
# STRT record: 21 bytes immediately after the header.
|
|
||||||
strt_raw = raw[_bw._WAVEFORM_HEADER_SIZE : _bw._WAVEFORM_HEADER_SIZE + 21]
|
|
||||||
strt_fields = _decode_strt(strt_raw)
|
|
||||||
if not strt_fields:
|
|
||||||
raise ValueError(f"{path}: STRT record missing or malformed")
|
|
||||||
|
|
||||||
# Footer: locate the 0e 08 marker, validating the year is in a sane range.
|
|
||||||
body_start = _bw._WAVEFORM_HEADER_SIZE + 21
|
|
||||||
# The 0e 08 + plausible-year footer signature can occur inside the sample
|
|
||||||
# stream. Collect every candidate and prefer the first whose body yields a
|
|
||||||
# waveform record chain terminating on the 0x06 marker; fall back to the
|
|
||||||
# first candidate otherwise. Blast radius measured 2026-08-25: changes the
|
|
||||||
# chosen footer on exactly 1 of 1,388 series-3 waveform files
|
|
||||||
# (BE17353/S353L4O5.OX0W, false positive at 3800, real footer at 8576).
|
|
||||||
footer_candidates = []
|
|
||||||
pos = body_start
|
|
||||||
while True:
|
|
||||||
pos = raw.find(b"\x0e\x08", pos)
|
|
||||||
if pos < 0 or pos + 26 > len(raw):
|
|
||||||
break
|
|
||||||
yr = (raw[pos + 4] << 8) | raw[pos + 5]
|
|
||||||
if 2015 <= yr <= 2050:
|
|
||||||
footer_candidates.append(pos)
|
|
||||||
pos += 1
|
|
||||||
|
|
||||||
footer_pos = -1
|
|
||||||
for cand in footer_candidates:
|
|
||||||
cand_body = raw[body_start:cand]
|
|
||||||
try:
|
|
||||||
recs = _wc.walk_records(cand_body)
|
|
||||||
except Exception:
|
|
||||||
recs = []
|
|
||||||
if recs:
|
|
||||||
tail = recs[-1]["end"]
|
|
||||||
if tail + 5 <= len(cand_body) and cand_body[tail + 4] == _wc.STREAM_END_ID:
|
|
||||||
footer_pos = cand
|
|
||||||
break
|
|
||||||
if footer_pos < 0 and footer_candidates:
|
|
||||||
footer_pos = footer_candidates[0]
|
|
||||||
|
|
||||||
if footer_pos < 0 and len(raw) >= 26:
|
|
||||||
footer_pos = len(raw) - 26
|
|
||||||
if footer_pos < body_start:
|
|
||||||
raise ValueError(f"{path}: footer not found")
|
|
||||||
|
|
||||||
body = raw[body_start : footer_pos]
|
|
||||||
footer = raw[footer_pos : footer_pos + 26]
|
|
||||||
|
|
||||||
# Footer layout:
|
|
||||||
# [0:2] 0e 08 marker
|
|
||||||
# [2:10] ts1 (start) BE 8B
|
|
||||||
# [10:18] ts2 (stop) BE 8B
|
|
||||||
# [18:24] 00 01 00 02 00 00
|
|
||||||
# [24:26] crc
|
|
||||||
ts1 = _bw._decode_ts_be(footer[2:10])
|
|
||||||
ts2 = _bw._decode_ts_be(footer[10:18])
|
|
||||||
|
|
||||||
# Body: decode via the verified body codecs. Two formats coexist:
|
|
||||||
#
|
|
||||||
# 1. Waveform-mode (.AB0W) — starts with 7-byte preamble
|
|
||||||
# ``00 02 00 [Tran[0] BE] [Tran[1] BE]`` followed by the
|
|
||||||
# tagged-block delta stream documented in
|
|
||||||
# ``docs/waveform_codec_re_status.md`` and §7.6.1 of the
|
|
||||||
# protocol reference. Decoded by ``waveform_codec.decode_waveform_v2``.
|
|
||||||
#
|
|
||||||
# 2. Histogram-mode (.AB0H) — a sequence of 32-byte blocks, one
|
|
||||||
# per histogram interval, each carrying per-channel peak +
|
|
||||||
# half-period values. Decoded by
|
|
||||||
# ``histogram_codec.decode_histogram_body``. Both codecs
|
|
||||||
# return the same channel-grouped output shape, so consumers
|
|
||||||
# don't need to special-case mode.
|
|
||||||
#
|
|
||||||
# The historical ``_decode_samples_4ch_int16_le`` int16-LE
|
|
||||||
# interpretation was retracted 2026-05-08 (see protocol-ref §7.6.1
|
|
||||||
# retraction box) — it produced ±32K noise on every event.
|
|
||||||
#
|
|
||||||
# If both codecs fail (malformed file, truncated body, unrecognised
|
|
||||||
# mode, synthetic test input), fall back to empty channels — the
|
|
||||||
# rest of the event (timestamp, waveform_key, project strings) is
|
|
||||||
# still recoverable and useful.
|
|
||||||
decoded = decode_waveform_v2(body)
|
|
||||||
if decoded is None:
|
|
||||||
decoded = decode_histogram_body(body)
|
|
||||||
if decoded is None:
|
|
||||||
log.warning(
|
|
||||||
"%s: body codec failed to decode (body starts %s) — "
|
|
||||||
"raw_samples will be empty", path, body[:8].hex(" "),
|
|
||||||
)
|
|
||||||
samples = {"Tran": [], "Vert": [], "Long": [], "MicL": []}
|
|
||||||
else:
|
|
||||||
samples = decoded_to_adc_counts(decoded)
|
|
||||||
|
|
||||||
# Metadata strings (label-anchored search across the body).
|
|
||||||
project = _find_first_string(body, b"Project:")
|
|
||||||
client = _find_first_string(body, b"Client:")
|
|
||||||
user = _find_first_string(body, b"User Name:")
|
|
||||||
seisloc = _find_first_string(body, b"Seis Loc:")
|
|
||||||
|
|
||||||
# Build the Event.
|
|
||||||
ev = Event(index=-1)
|
|
||||||
if strt_fields.get("waveform_key"):
|
|
||||||
ev._waveform_key = bytes.fromhex(strt_fields["waveform_key"])
|
|
||||||
# Derive record_type from the filename's extension suffix (H/W/M/E/C).
|
|
||||||
# When called from save_imported_bw the path here is a tmp file with a
|
|
||||||
# ".bw" suffix, so the derivation falls back to "Waveform" and the
|
|
||||||
# caller overrides ev.record_type using the original filename — see
|
|
||||||
# waveform_store.save_imported_bw.
|
|
||||||
ev.record_type = derive_record_type_from_filename(path.name)
|
|
||||||
ev.rectime_seconds = strt_fields.get("rectime_seconds")
|
|
||||||
ev.total_samples = strt_fields.get("total_samples")
|
|
||||||
ev.pretrig_samples = strt_fields.get("pretrig_samples")
|
|
||||||
|
|
||||||
if ts1 is not None:
|
|
||||||
ev.timestamp = Timestamp(
|
|
||||||
raw=footer[2:10],
|
|
||||||
flag=0x10,
|
|
||||||
year=ts1.year, unknown_byte=0, month=ts1.month, day=ts1.day,
|
|
||||||
hour=ts1.hour, minute=ts1.minute, second=ts1.second,
|
|
||||||
)
|
|
||||||
|
|
||||||
ev.project_info = ProjectInfo(
|
|
||||||
project=project, client=client, operator=user, sensor_location=seisloc,
|
|
||||||
)
|
|
||||||
ev.raw_samples = samples
|
|
||||||
# Only compute peaks from samples when we actually have samples.
|
|
||||||
# For events the codec couldn't decode (histogram-mode bodies, until
|
|
||||||
# the §7.6.2 histogram codec is wired in), samples is an empty dict
|
|
||||||
# and ``_peaks_from_samples`` would return PeakValues(0, 0, 0, 0, 0).
|
|
||||||
# That would then OVERWRITE existing good DB peak values (e.g. from
|
|
||||||
# paired BW ASCII reports) during the backfill UPSERT path.
|
|
||||||
# Leaving peak_values=None signals "we don't know" to downstream
|
|
||||||
# consumers; the backfill script seeds from the DB row when it sees
|
|
||||||
# None, and ``apply_report_to_event`` overlays from a paired ASCII
|
|
||||||
# report when one is supplied.
|
|
||||||
has_samples = any(samples.get(ch) for ch in ("Tran", "Vert", "Long", "MicL"))
|
|
||||||
ev.peak_values = _peaks_from_samples(samples) if has_samples else None
|
|
||||||
ev._a5_frames = None # not recoverable from BW file
|
|
||||||
|
|
||||||
return ev
|
|
||||||
+31
-181
@@ -111,24 +111,20 @@ def build_5a_frame(offset_word: int, raw_params: bytes) -> bytes:
|
|||||||
verified against this algorithm on 2026-04-02).
|
verified against this algorithm on 2026-04-02).
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
offset_word: 16-bit offset. For probe/chunks/metadata pages this is
|
offset_word: 16-bit offset (0x1004 for probe/chunks, 0x005A for term).
|
||||||
`0x1002`. For the proper TERM frame this is computed by
|
raw_params: 10 or 11 params bytes (from bulk_waveform_params or
|
||||||
`bulk_waveform_term_v2()` from the STRT-derived
|
bulk_waveform_term_params). 0x10 bytes in params are
|
||||||
`end_offset`.
|
written RAW — NOT DLE-stuffed. Confirmed 2026-04-06 by
|
||||||
raw_params: 10, 11, or 12 params bytes (from `bulk_waveform_params`
|
comparing wire bytes: BW sends bare `10 04` for chunk 1
|
||||||
for probes/samples, `bulk_waveform_term_v2` for TERM, or
|
(counter=0x1004), not stuffed `10 10 04`. Device reads
|
||||||
a manually-built 12-byte block for the metadata pages
|
params at fixed byte positions; stuffing shifts the bytes
|
||||||
0x1002 / 0x1004). See gotcha #3 below — params region
|
and corrupts the counter, causing device to ignore the frame.
|
||||||
uses partial DLE stuffing of 0x10 bytes.
|
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
Complete frame bytes: [ACK][STX][stuffed_section][chk][ETX]
|
Complete frame bytes: [ACK][STX][stuffed_section][chk][ETX]
|
||||||
"""
|
"""
|
||||||
if len(raw_params) not in (10, 11, 12):
|
if len(raw_params) not in (10, 11):
|
||||||
# 10 = termination params; 11 = regular probe / chunk params;
|
raise ValueError(f"raw_params must be 10 or 11 bytes, got {len(raw_params)}")
|
||||||
# 12 = metadata-page params (extra trailing 0x00 — BW byte-perfect quirk
|
|
||||||
# for the two fixed metadata reads at counter=0x1002 and 0x1004).
|
|
||||||
raise ValueError(f"raw_params must be 10/11/12 bytes, got {len(raw_params)}")
|
|
||||||
|
|
||||||
# Build stuffed section between STX and checksum
|
# Build stuffed section between STX and checksum
|
||||||
s = bytearray()
|
s = bytearray()
|
||||||
@@ -138,40 +134,8 @@ def build_5a_frame(offset_word: int, raw_params: bytes) -> bytes:
|
|||||||
s += b"\x00" # field3
|
s += b"\x00" # field3
|
||||||
s += bytes([(offset_word >> 8) & 0xFF, # offset_hi — raw, NOT stuffed
|
s += bytes([(offset_word >> 8) & 0xFF, # offset_hi — raw, NOT stuffed
|
||||||
offset_word & 0xFF]) # offset_lo
|
offset_word & 0xFF]) # offset_lo
|
||||||
# Params — partial DLE stuffing of 0x10 bytes (CONFIRMED 2026-05-05).
|
for b in raw_params: # params — NOT DLE-stuffed (raw bytes, match BW wire format)
|
||||||
#
|
|
||||||
# The device's de-stuffing rule for params is:
|
|
||||||
# • `10 10` → de-stuffs to `10`
|
|
||||||
# • `10 02/03/04` → kept literal (these are inner-frame markers)
|
|
||||||
# • `10 X` other → de-stuffs to just `X` (drops the 0x10)
|
|
||||||
#
|
|
||||||
# So for any 0x10 byte in the *logical* params that is followed by a
|
|
||||||
# byte NOT in {0x02, 0x03, 0x04, 0x10}, we must double the 0x10 on the
|
|
||||||
# wire (`10 X` → `10 10 X`) so the device's de-stuffer reproduces the
|
|
||||||
# original `10 X` pair. Without this, counter values with `0x10` in
|
|
||||||
# the high byte (e.g. counter=0x1000 has params bytes `10 00`) are
|
|
||||||
# silently corrupted to `0x__00` on the device side, and the device
|
|
||||||
# responds for the wrong address — for counter=0x1000 it returns the
|
|
||||||
# probe response (counter=0x0000), which contains the file header +
|
|
||||||
# STRT. That STRT block then lands in the assembled file body and
|
|
||||||
# Blastware rejects the file as malformed.
|
|
||||||
#
|
|
||||||
# Confirmed against BW capture 5-1-26 / bwcap3sec frame 20: params
|
|
||||||
# logical bytes `00 01 11 10 00 00 00 00 00 00 00` (counter=0x1000)
|
|
||||||
# are encoded on the wire as `00 01 11 10 10 00 00 00 00 00 00 00`.
|
|
||||||
# BW frames 13/14 (meta @ 0x1002 / 0x1004) leave `10 02` and `10 04`
|
|
||||||
# raw — the device handles those literal pairs correctly.
|
|
||||||
i = 0
|
|
||||||
while i < len(raw_params):
|
|
||||||
b = raw_params[i]
|
|
||||||
s.append(b)
|
s.append(b)
|
||||||
if (
|
|
||||||
b == 0x10
|
|
||||||
and i + 1 < len(raw_params)
|
|
||||||
and raw_params[i + 1] not in (0x02, 0x03, 0x04, 0x10)
|
|
||||||
):
|
|
||||||
s.append(0x10) # double the 0x10 so it survives device de-stuffing
|
|
||||||
i += 1
|
|
||||||
|
|
||||||
# DLE-aware checksum: for 0x10 XX pairs count XX; for lone bytes count them
|
# DLE-aware checksum: for 0x10 XX pairs count XX; for lone bytes count them
|
||||||
chk, i = 0, 0
|
chk, i = 0, 0
|
||||||
@@ -434,26 +398,28 @@ def bulk_waveform_params(key4: bytes, counter: int, *, is_probe: bool = False) -
|
|||||||
|
|
||||||
def bulk_waveform_term_params(key4: bytes, counter: int) -> bytes:
|
def bulk_waveform_term_params(key4: bytes, counter: int) -> bytes:
|
||||||
"""
|
"""
|
||||||
⛔ DEPRECATED — DO NOT USE IN NEW CODE.
|
Build the 10-byte params block for the SUB 5A termination request.
|
||||||
|
|
||||||
This is the v1 termination params helper, paired with the broken
|
The termination request uses offset=0x005A and a DIFFERENT params layout —
|
||||||
`_BULK_TERM_OFFSET = 0x005A` magic offset_word. Together they produce a
|
the leading 0x00 byte is dropped, key4[0:2] shifts to params[0:2], and the
|
||||||
~100-byte device-side terminator response that does NOT contain the
|
counter high byte is at params[2]:
|
||||||
partial-last-chunk waveform tail or the 26-byte file footer. Files
|
|
||||||
reconstructed using this terminator are missing their last ~512 bytes of
|
|
||||||
waveform data and have a synthesized footer that disagrees with what BW
|
|
||||||
would have written.
|
|
||||||
|
|
||||||
**For new code, use `bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`**
|
params[0] = key4[0]
|
||||||
which computes the correct offset_word + params from the STRT-derived
|
params[1] = key4[1]
|
||||||
`end_offset`. v2 produces wire bytes that match BW exactly across all
|
params[2] = (counter >> 8) & 0xFF
|
||||||
tested events (4-27-26 / 5-1-26 / 5-4-26 captures).
|
params[3:] = zeros
|
||||||
|
|
||||||
This function is retained ONLY for the defensive fallback path in
|
Counter for the termination request = last_regular_counter + 0x0400.
|
||||||
`read_bulk_waveform_stream()` that triggers when STRT parsing fails or no
|
|
||||||
chunks are fetched (= a malformed event or an unexpected device state).
|
Confirmed from 1-2-26 BW TX capture: final request (frame 83) uses
|
||||||
The fallback already logs a WARNING when it activates; if you see that
|
offset=0x005A, params[0:3] = key4[0:2] + term_counter_hi.
|
||||||
warning, the bug is upstream — STRT should have been parseable.
|
|
||||||
|
Args:
|
||||||
|
key4: 4-byte waveform key.
|
||||||
|
counter: Termination counter (= last regular counter + 0x0400).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
10-byte params block.
|
||||||
"""
|
"""
|
||||||
if len(key4) != 4:
|
if len(key4) != 4:
|
||||||
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
|
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
|
||||||
@@ -464,123 +430,6 @@ def bulk_waveform_term_params(key4: bytes, counter: int) -> bytes:
|
|||||||
return bytes(p)
|
return bytes(p)
|
||||||
|
|
||||||
|
|
||||||
def bulk_waveform_term_v2(
|
|
||||||
key4: bytes,
|
|
||||||
end_offset: int,
|
|
||||||
last_chunk_counter: int,
|
|
||||||
) -> tuple[int, bytes]:
|
|
||||||
"""
|
|
||||||
Compute the SUB 5A TERM frame's offset_word and 10-byte params block.
|
|
||||||
|
|
||||||
Confirmed across 3 events (4-27-26 + 5-1-26 captures):
|
|
||||||
|
|
||||||
next_boundary = last_chunk_counter + 0x0200
|
|
||||||
offset_word = end_offset - next_boundary (residual byte count)
|
|
||||||
params[0] = key4[0] (= 0x01 on every observed device)
|
|
||||||
params[1] = key4[1] (= 0x11)
|
|
||||||
params[2] = (next_boundary >> 8) & 0xFF
|
|
||||||
params[3] = next_boundary & 0xFF
|
|
||||||
params[4:10] = zeros
|
|
||||||
|
|
||||||
Verification:
|
|
||||||
| end_offset | last_chunk | next_boundary | offset_word | params[2:4] |
|
|
||||||
| 0x1ABE | 0x1800 | 0x1A00 | 0x00BE | 1A 00 |
|
|
||||||
| 0x21F2 | 0x1E00 | 0x2000 | 0x01F2 | 20 00 |
|
|
||||||
| 0x417E | 0x3E38 | 0x4038 | 0x0146 | 40 38 |
|
|
||||||
|
|
||||||
The device receives `requested_address = (params[2] << 8) | offset_word`
|
|
||||||
and replies with `(end_offset - next_boundary)` bytes of waveform tail
|
|
||||||
starting at `next_boundary` — including the 26-byte file footer.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
key4: 4-byte waveform key for this event.
|
|
||||||
end_offset: Event-end pointer (= `(end_key[2] << 8) | end_key[3]`
|
|
||||||
from the STRT record at data[23:27] of A5[0]).
|
|
||||||
last_chunk_counter: Counter of the last full 0x0200-byte chunk fetched
|
|
||||||
(the chunk that covers [last_chunk_counter,
|
|
||||||
last_chunk_counter + 0x0200)).
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
(offset_word, params10) tuple. Pass as
|
|
||||||
`build_5a_frame(offset_word, params)`.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
ValueError: on inconsistent inputs.
|
|
||||||
"""
|
|
||||||
if len(key4) != 4:
|
|
||||||
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
|
|
||||||
next_boundary = last_chunk_counter + 0x0200
|
|
||||||
if next_boundary > 0xFFFF:
|
|
||||||
raise ValueError(
|
|
||||||
f"next_boundary 0x{next_boundary:04X} exceeds uint16; check inputs"
|
|
||||||
)
|
|
||||||
if end_offset <= last_chunk_counter:
|
|
||||||
raise ValueError(
|
|
||||||
f"end_offset 0x{end_offset:04X} must be > "
|
|
||||||
f"last_chunk_counter 0x{last_chunk_counter:04X}"
|
|
||||||
)
|
|
||||||
offset_word = end_offset - next_boundary
|
|
||||||
if offset_word < 0:
|
|
||||||
# Last chunk overshot end_offset; caller should have stopped one chunk
|
|
||||||
# earlier. Treat as zero residual.
|
|
||||||
offset_word = 0
|
|
||||||
if offset_word > 0xFFFF:
|
|
||||||
raise ValueError(
|
|
||||||
f"offset_word 0x{offset_word:04X} exceeds uint16"
|
|
||||||
)
|
|
||||||
p = bytearray(10)
|
|
||||||
p[0] = key4[0]
|
|
||||||
p[1] = key4[1]
|
|
||||||
p[2] = (next_boundary >> 8) & 0xFF
|
|
||||||
p[3] = next_boundary & 0xFF
|
|
||||||
return offset_word, bytes(p)
|
|
||||||
|
|
||||||
|
|
||||||
# ── End-offset extraction from STRT record ────────────────────────────────────
|
|
||||||
|
|
||||||
STRT_MARKER = b"STRT"
|
|
||||||
|
|
||||||
|
|
||||||
def parse_strt_end_offset(a5_data: bytes) -> Optional[int]:
|
|
||||||
"""
|
|
||||||
Extract the event-end offset from the STRT record in an A5 response payload.
|
|
||||||
|
|
||||||
The first A5 response (the probe response, or the first chunk for events
|
|
||||||
with non-zero start_key[2:4]) contains a STRT record at byte offset 17 of
|
|
||||||
`data`. Layout:
|
|
||||||
|
|
||||||
data[17:21] "STRT"
|
|
||||||
data[21:23] ff fe sentinel
|
|
||||||
data[23:27] end_key ← 4-byte key of where this event ENDS
|
|
||||||
data[27:31] start_key
|
|
||||||
...
|
|
||||||
|
|
||||||
Returns `(end_key[2] << 8) | end_key[3]` — the absolute device-buffer
|
|
||||||
address where the event ends. Use this to bound the chunk loop and to
|
|
||||||
compute the TERM frame.
|
|
||||||
|
|
||||||
Verified end_offset values:
|
|
||||||
| event start_key | end_key | end_offset |
|
|
||||||
| 01110000 | 01111ABE | 0x1ABE |
|
|
||||||
| 01110000 | 011121F2 | 0x21F2 |
|
|
||||||
| 011121F2 | 0111417E | 0x417E |
|
|
||||||
|
|
||||||
Args:
|
|
||||||
a5_data: The `data` field of an A5 response frame (frame.data).
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
The end_offset (uint16) if STRT is found, else None.
|
|
||||||
"""
|
|
||||||
pos = a5_data.find(STRT_MARKER)
|
|
||||||
if pos < 0 or pos + 10 > len(a5_data):
|
|
||||||
return None
|
|
||||||
# data[pos+4:pos+6] is "ff fe"; data[pos+6:pos+10] is end_key.
|
|
||||||
end_key = a5_data[pos + 6 : pos + 10]
|
|
||||||
if len(end_key) < 4:
|
|
||||||
return None
|
|
||||||
return (end_key[2] << 8) | end_key[3]
|
|
||||||
|
|
||||||
|
|
||||||
# ── Pre-built POLL frames ─────────────────────────────────────────────────────
|
# ── Pre-built POLL frames ─────────────────────────────────────────────────────
|
||||||
#
|
#
|
||||||
# POLL (SUB 0x5B) uses the same two-step pattern as all other reads — the
|
# POLL (SUB 0x5B) uses the same two-step pattern as all other reads — the
|
||||||
@@ -621,6 +470,7 @@ class S3Frame:
|
|||||||
|
|
||||||
|
|
||||||
# ── Streaming S3 frame parser ─────────────────────────────────────────────────
|
# ── Streaming S3 frame parser ─────────────────────────────────────────────────
|
||||||
|
|
||||||
class S3FrameParser:
|
class S3FrameParser:
|
||||||
"""
|
"""
|
||||||
Incremental byte-stream parser for S3→BW response frames.
|
Incremental byte-stream parser for S3→BW response frames.
|
||||||
|
|||||||
@@ -1,487 +0,0 @@
|
|||||||
"""
|
|
||||||
histogram_codec.py — decoder for MiniMate Plus histogram-mode event bodies.
|
|
||||||
|
|
||||||
FULLY DECODED 2026-05-20. Every field in every block, verified
|
|
||||||
byte-exact against BW's ASCII export across multiple histogram
|
|
||||||
fixtures.
|
|
||||||
|
|
||||||
The histogram-mode body is a stream of 32-byte fixed-length blocks,
|
|
||||||
one block per histogram interval. Each block carries the per-interval
|
|
||||||
peak amplitude + zero-crossing frequency for all four channels (Tran,
|
|
||||||
Vert, Long, MicL).
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
Body layout (CONFIRMED 2026-05-20)
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
[stream of 32-byte blocks]
|
|
||||||
|
|
||||||
Body length is approximately ``n_intervals * 32`` bytes plus a small
|
|
||||||
trailing remnant (1-9 bytes typically) at the very end. Walker should
|
|
||||||
iterate 32-stride and stop before the tail.
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
32-byte block layout
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
[0] 0x00 always-zero tag
|
|
||||||
[1] segment_id (uint8) 0x00..0x03 - 256 blocks per segment
|
|
||||||
[2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, ...)
|
|
||||||
[4] 0x0a (uint8) constant marker (= 10)
|
|
||||||
[5:7] T_peak_count uint16 BE Tran peak (count x 0.005 -> in/s)
|
|
||||||
[7:9] T_halfperiod uint16 BE Tran half-period in samples (freq = 512 / halfp)
|
|
||||||
[9:11] V_peak_count uint16 BE
|
|
||||||
[11:13] V_halfperiod uint16 BE
|
|
||||||
[13:15] L_peak_count uint16 BE
|
|
||||||
[15:17] L_halfperiod uint16 BE
|
|
||||||
[17:19] M_peak_count uint16 BE MicL peak (count -> dB via mic_count_to_db)
|
|
||||||
[19:21] M_halfperiod uint16 BE MicL half-period in samples
|
|
||||||
[21:23] 0x00 0x00 constant on standard blocks
|
|
||||||
[24:28] 4-byte variable purpose unknown (possibly CRC or timestamp delta)
|
|
||||||
[28:32] block-end signature see "Two block tails" below
|
|
||||||
|
|
||||||
**Every per-channel field is uint16 BIG-endian** (confirmed 2026-08-25).
|
|
||||||
Only ``block_ctr`` at [2:4] is little-endian.
|
|
||||||
|
|
||||||
HISTORY - two earlier readings of this block were wrong in ways that
|
|
||||||
cancelled out on quiet data:
|
|
||||||
|
|
||||||
1. *peak as uint16 LE at [6:8]* - produced 268 in/s peaks on any
|
|
||||||
interval whose next byte was non-zero.
|
|
||||||
2. *peak as uint8 at [6] with an "annotation" byte at [7]* - correct
|
|
||||||
for every peak below 256 counts (1.275 in/s), but it silently
|
|
||||||
**clipped larger peaks**: the final interval of
|
|
||||||
BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export
|
|
||||||
(1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590 in/s.
|
|
||||||
The "annotation" byte was never an annotation - it is the high
|
|
||||||
byte of the big-endian half-period, which is why it was non-zero
|
|
||||||
exactly on the sub-Hz intervals BW renders as "<1.0".
|
|
||||||
|
|
||||||
Both readings also forced ``block[5] == 0`` via a bogus ``uint16 LE``
|
|
||||||
marker check at [4:6], which is what capped the peak at one byte.
|
|
||||||
The marker is ``block[4]`` alone.
|
|
||||||
|
|
||||||
Verified 2026-08-25 against 1211 production histograms paired with
|
|
||||||
their Blastware ASCII exports: **1211/1211 decode exactly** (interval
|
|
||||||
count plus every per-interval peak), and 842,442 per-interval
|
|
||||||
frequency comparisons match with **zero** mismatches.
|
|
||||||
|
|
||||||
Two block tails
|
|
||||||
---------------
|
|
||||||
Standard blocks end with ``1e 0a 00 00``. The **final block of the
|
|
||||||
stream** ends with ``9c 06 00 42`` instead, and carries arbitrary bytes
|
|
||||||
at [21:23]. Rejecting it dropped the last interval of nearly every
|
|
||||||
histogram - and the last interval is frequently the one holding the
|
|
||||||
event peak, so the file's reported PPV came out low. Observed in 1206
|
|
||||||
of 1211 production histograms, always positioned after every
|
|
||||||
standard-tail block.
|
|
||||||
|
|
||||||
Block-identification anchor: ``block[0] == 0x00`` AND
|
|
||||||
``block[4] == 0x0A`` AND the tail is one of the two signatures above;
|
|
||||||
standard-tail blocks additionally require ``block[22] == 0x00``.
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
Per-channel encoding
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
Geophone channels (Tran, Vert, Long):
|
|
||||||
- peak_count × 0.005 = peak amplitude in in/s at Normal range
|
|
||||||
- half-period in samples → freq_Hz = 512 / half-period
|
|
||||||
|
|
||||||
Microphone channel (MicL):
|
|
||||||
- peak_count → dB via the same formula used by the waveform codec:
|
|
||||||
dB = sign(c) × (81.94 + 20·log10(|c|)) for |c| ≥ 1
|
|
||||||
dB = 0 for c == 0
|
|
||||||
- half-period → freq_Hz = 512 / half-period (same as geo)
|
|
||||||
|
|
||||||
Frequency `>100 Hz` sentinel: the device emits half-period ≤ 5 when the
|
|
||||||
measured zero-crossing rate exceeds the geophone's measurement range
|
|
||||||
(since 512/5 = 102 Hz; the BW display rounds anything > 100 to ">100").
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
Output shape
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
``decode_histogram_body`` returns a per-channel dict matching the
|
|
||||||
waveform codec's shape so the rest of the pipeline (.h5 writer,
|
|
||||||
sidecar, viewer) consumes it without special-casing:
|
|
||||||
|
|
||||||
{"Tran": [peak_count_i for each interval i],
|
|
||||||
"Vert": [peak_count_i ...],
|
|
||||||
"Long": [peak_count_i ...],
|
|
||||||
"MicL": [peak_count_i ...]}
|
|
||||||
|
|
||||||
Values are in **16-count units for geo** (LSB = 0.005 in/s, matching
|
|
||||||
``decode_waveform_v2``) and **1-count units for mic** (matching the
|
|
||||||
waveform codec's mic convention). Run through
|
|
||||||
``waveform_codec.decoded_to_adc_counts`` to scale geo to 1-count ADC.
|
|
||||||
|
|
||||||
Per-interval frequencies are NOT returned — they're auxiliary data,
|
|
||||||
not waveform samples. Consumers needing frequencies can call
|
|
||||||
``decode_histogram_body_full()`` for the structured per-interval
|
|
||||||
record list.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import struct
|
|
||||||
from typing import List, Optional, Tuple
|
|
||||||
|
|
||||||
# Block-end signature: constant `1e 0a 00 00` in bytes [28:32] of every
|
|
||||||
# real data block. More distinctive than the byte-22 `00 00` (which
|
|
||||||
# matches many false positives), so we anchor on this.
|
|
||||||
_BLOCK_TAIL = b"\x1e\x0a\x00\x00"
|
|
||||||
|
|
||||||
# The final block of a histogram stream ends with this instead. It is a
|
|
||||||
# real data block - same layout - and holds the last interval. See the
|
|
||||||
# module docstring, "Two block tails".
|
|
||||||
_BLOCK_TAIL_TERMINAL = b"\x9c\x06\x00\x42"
|
|
||||||
|
|
||||||
_BLOCK_SIZE = 32
|
|
||||||
|
|
||||||
# Marker byte at block[4:6] of every histogram data block. Used as
|
|
||||||
# additional validation that we're looking at a real block.
|
|
||||||
_BLOCK_MARKER = 10
|
|
||||||
|
|
||||||
# Geo peak scaling: stored as "count × 0.005 in/s" where 1 count = one
|
|
||||||
# 0.005 in/s display quantum. Equivalent to the waveform codec's
|
|
||||||
# 16-count-unit output (1 unit = 0.005 in/s = 16 ADC counts).
|
|
||||||
_GEO_LSB_INS = 0.005
|
|
||||||
|
|
||||||
# Frequency formula: freq_Hz = _FREQ_NUMERATOR / half_period_samples.
|
|
||||||
# Empirically determined to be 512 (= sample_rate / 2, where sample rate
|
|
||||||
# is 1024 sps for the standard MiniMate Plus configuration).
|
|
||||||
_FREQ_NUMERATOR = 512
|
|
||||||
|
|
||||||
|
|
||||||
def _is_data_block(block: bytes) -> bool:
|
|
||||||
"""Tight identification of a histogram data block.
|
|
||||||
|
|
||||||
Accepts both tail signatures. ``block[4]`` alone is the marker -
|
|
||||||
``block[5]`` is the high byte of the Tran peak and is non-zero on any
|
|
||||||
interval above 1.275 in/s, so it must not be part of the marker test.
|
|
||||||
The ``block[22] == 0`` constraint is what keeps trailer content out,
|
|
||||||
but it applies only to standard-tail blocks: terminal blocks carry
|
|
||||||
arbitrary bytes there.
|
|
||||||
"""
|
|
||||||
if len(block) < _BLOCK_SIZE:
|
|
||||||
return False
|
|
||||||
if block[0] != 0x00:
|
|
||||||
return False
|
|
||||||
if block[4] != _BLOCK_MARKER:
|
|
||||||
return False
|
|
||||||
# The 4-byte tail plus block[0]==0 and block[4]==0x0A is already six bytes
|
|
||||||
# of constraint — enough to keep trailer content out. There is NO extra
|
|
||||||
# test on block[22]: it was documented as a constant 0x00 but carries data
|
|
||||||
# on loud blocks, and rejecting those threw away the interval holding the
|
|
||||||
# event peak. BE18350/T350L7HR.NL0H is the proof: its block 92 has
|
|
||||||
# block[22]=0x26 and a Tran peak of 0x0563 = 1379 counts = 6.895 in/s,
|
|
||||||
# exactly the device-reported PPV, while the file decoded to 0.015 in/s.
|
|
||||||
return block[28:32] in (_BLOCK_TAIL, _BLOCK_TAIL_TERMINAL)
|
|
||||||
|
|
||||||
|
|
||||||
def _decode_block(block: bytes) -> Optional[dict]:
|
|
||||||
"""Decode one 32-byte histogram block. Caller must have validated
|
|
||||||
with ``_is_data_block`` first.
|
|
||||||
|
|
||||||
Returns a record with per-channel peak counts (uint8) and
|
|
||||||
half-periods (uint16 LE).
|
|
||||||
"""
|
|
||||||
# Every per-channel field is uint16 BIG-endian; only block_ctr is LE.
|
|
||||||
# See the module docstring for the two superseded readings and why
|
|
||||||
# each looked correct on quiet data.
|
|
||||||
def _be16(i: int) -> int:
|
|
||||||
return (block[i] << 8) | block[i + 1]
|
|
||||||
|
|
||||||
t_peak = _be16(5)
|
|
||||||
t_halfp = _be16(7)
|
|
||||||
v_peak = _be16(9)
|
|
||||||
v_halfp = _be16(11)
|
|
||||||
l_peak = _be16(13)
|
|
||||||
l_halfp = _be16(15)
|
|
||||||
m_peak = _be16(17)
|
|
||||||
m_halfp = _be16(19)
|
|
||||||
segment_id = block[1]
|
|
||||||
block_ctr = block[2] | (block[3] << 8)
|
|
||||||
var_meta = bytes(block[24:28])
|
|
||||||
return {
|
|
||||||
"segment_id": segment_id,
|
|
||||||
"block_ctr": block_ctr,
|
|
||||||
"t_peak": t_peak,
|
|
||||||
"t_halfp": t_halfp,
|
|
||||||
"v_peak": v_peak,
|
|
||||||
"v_halfp": v_halfp,
|
|
||||||
"l_peak": l_peak,
|
|
||||||
"l_halfp": l_halfp,
|
|
||||||
"m_peak": m_peak,
|
|
||||||
"m_halfp": m_halfp,
|
|
||||||
"meta_var": var_meta,
|
|
||||||
"is_terminal": block[28:32] == _BLOCK_TAIL_TERMINAL,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def walk_body(body: bytes) -> List[dict]:
|
|
||||||
"""Walk the body and return one dict per histogram interval.
|
|
||||||
|
|
||||||
Iterates 32-byte strides from offset 0. Yields a decoded record
|
|
||||||
for every block that passes ``_is_data_block`` validation. Stops
|
|
||||||
when the remaining bytes are too short to form a complete block.
|
|
||||||
|
|
||||||
In Histogram+Continuous mode the body interleaves data blocks with
|
|
||||||
other 32-byte content (likely continuous-mode waveform blocks) that
|
|
||||||
fail the data-block validation; the walker naturally skips them
|
|
||||||
without losing 32-byte alignment. Use ``block_ctr`` from each
|
|
||||||
returned record to map back to the original interval index — the
|
|
||||||
record list is sparse when other block types are interleaved.
|
|
||||||
"""
|
|
||||||
records: List[dict] = []
|
|
||||||
for off in range(0, len(body) - _BLOCK_SIZE + 1, _BLOCK_SIZE):
|
|
||||||
blk = body[off:off + _BLOCK_SIZE]
|
|
||||||
if not _is_data_block(blk):
|
|
||||||
# Hit non-block content (likely a sync or stream marker).
|
|
||||||
# Continue walking — block alignment is fixed at 32-stride
|
|
||||||
# from offset 0, so we don't lose alignment by skipping.
|
|
||||||
continue
|
|
||||||
decoded = _decode_block(blk)
|
|
||||||
if decoded is None:
|
|
||||||
# Block validated as a histogram block but had peak fields
|
|
||||||
# outside the plausible range — undocumented extension.
|
|
||||||
# Skip rather than propagating bogus PVS contributions.
|
|
||||||
continue
|
|
||||||
records.append(decoded)
|
|
||||||
return records
|
|
||||||
|
|
||||||
|
|
||||||
def _walk_auto(body: bytes) -> List[dict]:
|
|
||||||
"""Pick the block model by signature strength, not by which returns first.
|
|
||||||
|
|
||||||
The multi-interval variant announces itself with consecutive block headers
|
|
||||||
at an exact ``12 + 20*n`` stride — far stronger evidence than a handful of
|
|
||||||
scattered standard-tail blocks, which a multi-interval body will also yield
|
|
||||||
by coincidence. Dispatching on "whichever decoder returns something"
|
|
||||||
handed 193 BE18193 files to the standard walker and produced peaks of
|
|
||||||
149 in/s against a 10 in/s full scale.
|
|
||||||
"""
|
|
||||||
if detect_multi_interval_stride(body):
|
|
||||||
recs = walk_multi_interval_blocks(body)
|
|
||||||
if recs:
|
|
||||||
return recs
|
|
||||||
return walk_body(body)
|
|
||||||
|
|
||||||
|
|
||||||
def decode_histogram_body(body: bytes) -> Optional[dict]:
|
|
||||||
"""Decode a histogram-mode body into per-channel peak-sample arrays.
|
|
||||||
|
|
||||||
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}``
|
|
||||||
where each channel's list contains one peak value per histogram
|
|
||||||
interval (in the same units the waveform codec uses: 16-count units
|
|
||||||
for geo, 1-count ADC units for mic). Returns ``None`` if the body
|
|
||||||
doesn't contain any valid histogram blocks.
|
|
||||||
|
|
||||||
To convert to physical units:
|
|
||||||
- Geo channels: ``count * 0.005`` = peak in in/s at Normal range
|
|
||||||
(or run through ``waveform_codec.decoded_to_adc_counts`` first
|
|
||||||
to get 1-count ADC values, then ``count / 32767 * 10.0`` for in/s)
|
|
||||||
- Mic channel: use ``waveform_codec.mic_count_to_db(count)``
|
|
||||||
"""
|
|
||||||
records = _walk_auto(body)
|
|
||||||
if not records:
|
|
||||||
return None
|
|
||||||
return {
|
|
||||||
"Tran": [r["t_peak"] for r in records],
|
|
||||||
"Vert": [r["v_peak"] for r in records],
|
|
||||||
"Long": [r["l_peak"] for r in records],
|
|
||||||
"MicL": [r["m_peak"] for r in records],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def decode_histogram_body_full(body: bytes) -> Optional[List[dict]]:
|
|
||||||
"""Decode a histogram-mode body into the full per-interval record list.
|
|
||||||
|
|
||||||
Same data as ``decode_histogram_body`` but in a structured form that
|
|
||||||
preserves the half-period (frequency) data for each channel + the
|
|
||||||
per-block segment_id, block_ctr, and 4-byte variable metadata.
|
|
||||||
Useful for diagnostic tools, sidecar enrichment, and future-codec
|
|
||||||
work.
|
|
||||||
|
|
||||||
Returns ``None`` if the body has no valid blocks.
|
|
||||||
"""
|
|
||||||
records = _walk_auto(body)
|
|
||||||
return records if records else None
|
|
||||||
|
|
||||||
|
|
||||||
def half_period_to_hz(halfp: int) -> Optional[float]:
|
|
||||||
"""Convert a half-period in samples to frequency in Hz.
|
|
||||||
|
|
||||||
Returns ``None`` for half-period ≤ 5 — the device emits values in
|
|
||||||
that range when the measured zero-crossing rate exceeds 100 Hz
|
|
||||||
(the BW display reports `>100 Hz` for such cases). Callers can
|
|
||||||
treat ``None`` as the `>100 Hz` sentinel.
|
|
||||||
"""
|
|
||||||
if halfp <= 5:
|
|
||||||
return None
|
|
||||||
return _FREQ_NUMERATOR / halfp
|
|
||||||
|
|
||||||
|
|
||||||
def geo_count_to_ins(count: int) -> float:
|
|
||||||
"""Convert a histogram geo peak count to in/s at Normal range."""
|
|
||||||
return count * _GEO_LSB_INS
|
|
||||||
|
|
||||||
|
|
||||||
# ── Multi-interval block variant (CONFIRMED 2026-08-26) ─────────────────────
|
|
||||||
#
|
|
||||||
# When the histogram interval is SHORTER than one minute, the device packs
|
|
||||||
# several intervals into a single block so that every block still covers
|
|
||||||
# exactly one minute of data:
|
|
||||||
#
|
|
||||||
# interval size intervals/block stride
|
|
||||||
# 1 minute 1 32 <- the standard block above
|
|
||||||
# 15 seconds 4 92
|
|
||||||
# 2 seconds 30 612
|
|
||||||
#
|
|
||||||
# stride = 12 + n_intervals * 20
|
|
||||||
#
|
|
||||||
# Block layout:
|
|
||||||
# [0] 0x00
|
|
||||||
# [1] segment_id (256 blocks per segment, same as the standard block)
|
|
||||||
# [2:4] block_ctr uint16 LE (0x0100.., resets each segment)
|
|
||||||
# [4] 0x0a marker
|
|
||||||
# [5] 0x00
|
|
||||||
# [6 ...] n x 20-byte interval records, each carrying 8 x uint16
|
|
||||||
# LITTLE-endian values:
|
|
||||||
# T_peak, T_halfperiod, V_peak, V_halfperiod,
|
|
||||||
# L_peak, L_halfperiod, M_peak, M_halfperiod
|
|
||||||
# then 2 more words; the first is 0x0000 on every real interval.
|
|
||||||
# [-6:] 6-byte block trailer
|
|
||||||
#
|
|
||||||
# ⚠ ENDIANNESS: the standard 32-byte block is BIG-endian. This variant is
|
|
||||||
# LITTLE-endian. Do not share the accessor.
|
|
||||||
#
|
|
||||||
# These files previously decoded to nothing at all — 415 of them in the
|
|
||||||
# production snapshot, 216 on BE18193 (2 s intervals) and 199 on BE9440
|
|
||||||
# (15 s). Before that they were being accepted by the WAVEFORM codec, which
|
|
||||||
# returned garbage peaking up to 400x the device-reported PPV.
|
|
||||||
#
|
|
||||||
# Ground truth: BE9440/K440L3AQ.T70H (15 s intervals, 5,710 of them) decodes
|
|
||||||
# against its Blastware ASCII export with 17,130/17,130 geo peak counts,
|
|
||||||
# 22,840/22,840 frequencies and 5,710/5,710 mic dB(L) values matching exactly.
|
|
||||||
|
|
||||||
_MULTI_HEADER_LEN = 6
|
|
||||||
_MULTI_RECORD_LEN = 20
|
|
||||||
_MULTI_TRAILER_LEN = 6
|
|
||||||
# At least 2 records: a 1-record block would have stride 12 + 20 = 32, which
|
|
||||||
# collides with the standard big-endian block and mis-decodes it.
|
|
||||||
_MULTI_MIN_RECORDS = 2
|
|
||||||
_MULTI_MAX_RECORDS = 64
|
|
||||||
|
|
||||||
# Geo full scale in 16-count units: 10.000 in/s / 0.005 = 2000. A peak above
|
|
||||||
# this is physically impossible and marks buffer garbage in a partial block.
|
|
||||||
_GEO_MAX_COUNTS = 2000
|
|
||||||
|
|
||||||
|
|
||||||
def _is_multi_header(body: bytes, off: int) -> bool:
|
|
||||||
return (off + _MULTI_HEADER_LEN <= len(body)
|
|
||||||
and body[off] == 0x00
|
|
||||||
and body[off + 4] == 0x0A
|
|
||||||
and body[off + 5] == 0x00)
|
|
||||||
|
|
||||||
|
|
||||||
def detect_multi_interval_stride(body: bytes) -> Optional[int]:
|
|
||||||
"""Block stride of a multi-interval histogram body, or None.
|
|
||||||
|
|
||||||
Found by locating the second block header; validated against
|
|
||||||
``stride = 12 + n * 20`` and confirmed on a third block where present.
|
|
||||||
"""
|
|
||||||
if not _is_multi_header(body, 0):
|
|
||||||
return None
|
|
||||||
lo = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MIN_RECORDS
|
|
||||||
hi = _MULTI_HEADER_LEN + _MULTI_TRAILER_LEN + _MULTI_RECORD_LEN * _MULTI_MAX_RECORDS
|
|
||||||
for stride in range(lo, min(hi, len(body)) + 1, 2):
|
|
||||||
if (stride - 12) % _MULTI_RECORD_LEN:
|
|
||||||
continue
|
|
||||||
if not _is_multi_header(body, stride):
|
|
||||||
continue
|
|
||||||
# DECISIVE CHECK: consecutive blocks differ by exactly 1 in block_ctr.
|
|
||||||
# Without it this false-positives on ordinary standard-block bodies:
|
|
||||||
# those carry a header every 32 bytes, and 192 = 12 + 20*9 and
|
|
||||||
# 512 = 12 + 20*25 are both multiples of 32, so a stride "fits" while
|
|
||||||
# actually skipping 6 or 16 real blocks. Sampling a standard body at
|
|
||||||
# stride 192 handed 9,082 files to the wrong decoder and produced peaks
|
|
||||||
# of 149 in/s against a 10 in/s full scale.
|
|
||||||
def _ctr(o: int) -> int:
|
|
||||||
return body[o + 2] | (body[o + 3] << 8)
|
|
||||||
|
|
||||||
if (_ctr(stride) - _ctr(0)) & 0xFFFF != 1:
|
|
||||||
continue
|
|
||||||
# Confirm on a third block WHEN ONE IS ACTUALLY PRESENT. A body can
|
|
||||||
# be longer than two strides and still hold only two real blocks: a
|
|
||||||
# final *partial* block leaves trailing padding. E.g. 51 intervals at
|
|
||||||
# 2 s = one full 30-interval block + a 21-interval remainder, in a
|
|
||||||
# 2787-byte body — long enough to demand a third header at 1224 that
|
|
||||||
# does not exist. Requiring it unconditionally threw away the correct
|
|
||||||
# stride and the file decoded to nothing (BE18193 T193L0XM.CI0H).
|
|
||||||
# The block-counter check above is the decisive anti-false-positive
|
|
||||||
# test; this one is corroboration, so a missing third header means
|
|
||||||
# end-of-stream, not disqualification.
|
|
||||||
if (2 * stride + _MULTI_HEADER_LEN <= len(body)
|
|
||||||
and _is_multi_header(body, 2 * stride)):
|
|
||||||
if (_ctr(2 * stride) - _ctr(stride)) & 0xFFFF != 1:
|
|
||||||
continue
|
|
||||||
return stride
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def walk_multi_interval_blocks(body: bytes,
|
|
||||||
stride: Optional[int] = None) -> List[dict]:
|
|
||||||
"""Decode a multi-interval histogram body into per-interval records."""
|
|
||||||
if stride is None:
|
|
||||||
stride = detect_multi_interval_stride(body)
|
|
||||||
if not stride:
|
|
||||||
return []
|
|
||||||
n_per_block = (stride - _MULTI_HEADER_LEN - _MULTI_TRAILER_LEN) // _MULTI_RECORD_LEN
|
|
||||||
if n_per_block < 1:
|
|
||||||
return []
|
|
||||||
|
|
||||||
def u16le(p: int) -> int:
|
|
||||||
return body[p] | (body[p + 1] << 8)
|
|
||||||
|
|
||||||
out: List[dict] = []
|
|
||||||
for off in range(0, len(body) - stride + 1, stride):
|
|
||||||
if not _is_multi_header(body, off):
|
|
||||||
break # end of the block run; trailer follows
|
|
||||||
for k in range(n_per_block):
|
|
||||||
q = off + _MULTI_HEADER_LEN + _MULTI_RECORD_LEN * k
|
|
||||||
out.append({
|
|
||||||
"_tail0": u16le(q + 16),
|
|
||||||
"segment_id": body[off + 1],
|
|
||||||
"block_ctr": u16le(off + 2),
|
|
||||||
"t_peak": u16le(q), "t_halfp": u16le(q + 2),
|
|
||||||
"v_peak": u16le(q + 4), "v_halfp": u16le(q + 6),
|
|
||||||
"l_peak": u16le(q + 8), "l_halfp": u16le(q + 10),
|
|
||||||
"m_peak": u16le(q + 12), "m_halfp": u16le(q + 14),
|
|
||||||
"meta_var": bytes(body[q + 16:q + 20]),
|
|
||||||
"is_terminal": False,
|
|
||||||
})
|
|
||||||
# A session ending mid-block leaves the remaining slots of the FINAL block
|
|
||||||
# filled with whatever was in the buffer. Those decoded as peaks thousands
|
|
||||||
# of times the device-reported PPV, so they have to go — but only from the
|
|
||||||
# final block: a non-zero tail word occurs mid-file on real intervals, and
|
|
||||||
# trimming on that alone truncated four BE9440 files by up to 2,800
|
|
||||||
# intervals, while trimming purely from the end left garbage stranded
|
|
||||||
# behind one slot that happened to have a zero tail word.
|
|
||||||
#
|
|
||||||
# Within the final block, stop at the first slot that is not plausibly
|
|
||||||
# real: a non-zero tail word, or a geo peak above full scale. 16-count
|
|
||||||
# units put Normal-range full scale (10.000 in/s) at 2000 counts, so
|
|
||||||
# anything beyond that is physically impossible.
|
|
||||||
if out:
|
|
||||||
last_block_start = ((len(out) - 1) // n_per_block) * n_per_block
|
|
||||||
for i in range(last_block_start, len(out)):
|
|
||||||
r = out[i]
|
|
||||||
if (r["_tail0"] != 0
|
|
||||||
or max(r["t_peak"], r["v_peak"], r["l_peak"]) > _GEO_MAX_COUNTS):
|
|
||||||
del out[i:]
|
|
||||||
break
|
|
||||||
for r in out:
|
|
||||||
r.pop("_tail0", None)
|
|
||||||
return out
|
|
||||||
@@ -201,58 +201,6 @@ class Timestamp:
|
|||||||
second=second,
|
second=second,
|
||||||
)
|
)
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def from_short_record(cls, data: bytes) -> "Timestamp":
|
|
||||||
"""
|
|
||||||
Decode an 8-byte timestamp header from a 210-byte waveform record.
|
|
||||||
|
|
||||||
Wire layout (✅ CONFIRMED 2026-05-01 against live SFM run on BE11529 in
|
|
||||||
Continuous mode, day-of-month = 1 May, raw: 01 05 07 ea 00 0d 15 25):
|
|
||||||
byte[0]: day (uint8)
|
|
||||||
byte[1]: month (uint8)
|
|
||||||
bytes[2-3]: year (big-endian uint16)
|
|
||||||
byte[4]: unknown (0x00 in observed sample)
|
|
||||||
byte[5]: hour (uint8)
|
|
||||||
byte[6]: minute (uint8)
|
|
||||||
byte[7]: second (uint8)
|
|
||||||
|
|
||||||
This is a third format observed in the wild — distinct from the 9-byte
|
|
||||||
(single-shot, sub_code=0x10 at [1]) and 10-byte (continuous, 0x10 at
|
|
||||||
[0] AND [2]) layouts. No marker bytes; disambiguated by where the
|
|
||||||
year lands when scanned at byte 2/3/4.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
data: at least 8 bytes; only the first 8 are consumed.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Decoded Timestamp.
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
ValueError: if data is fewer than 8 bytes.
|
|
||||||
"""
|
|
||||||
if len(data) < 8:
|
|
||||||
raise ValueError(
|
|
||||||
f"Short record timestamp requires at least 8 bytes, got {len(data)}"
|
|
||||||
)
|
|
||||||
day = data[0]
|
|
||||||
month = data[1]
|
|
||||||
year = struct.unpack_from(">H", data, 2)[0]
|
|
||||||
unknown_byte = data[4]
|
|
||||||
hour = data[5]
|
|
||||||
minute = data[6]
|
|
||||||
second = data[7]
|
|
||||||
return cls(
|
|
||||||
raw=bytes(data[:8]),
|
|
||||||
flag=0,
|
|
||||||
year=year,
|
|
||||||
unknown_byte=unknown_byte,
|
|
||||||
month=month,
|
|
||||||
day=day,
|
|
||||||
hour=hour,
|
|
||||||
minute=minute,
|
|
||||||
second=second,
|
|
||||||
)
|
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def clock_set(self) -> bool:
|
def clock_set(self) -> bool:
|
||||||
"""False when year == 1995 (factory default / battery-lost state)."""
|
"""False when year == 1995 (factory default / battery-lost state)."""
|
||||||
@@ -352,14 +300,6 @@ class PeakValues:
|
|||||||
long: Optional[float] = None # Longitudinal PPV (in/s) ✅
|
long: Optional[float] = None # Longitudinal PPV (in/s) ✅
|
||||||
micl: Optional[float] = None # Air overpressure (psi) 🔶 (units uncertain)
|
micl: Optional[float] = None # Air overpressure (psi) 🔶 (units uncertain)
|
||||||
peak_vector_sum: Optional[float] = None # Scalar geo PVS (in/s) ✅
|
peak_vector_sum: Optional[float] = None # Scalar geo PVS (in/s) ✅
|
||||||
tran_zc_freq: Optional[float] = None
|
|
||||||
vert_zc_freq: Optional[float] = None
|
|
||||||
long_zc_freq: Optional[float] = None
|
|
||||||
mic_zc_freq: Optional[float] = None
|
|
||||||
tran_zc_above_range: bool = False
|
|
||||||
vert_zc_above_range: bool = False
|
|
||||||
long_zc_above_range: bool = False
|
|
||||||
mic_zc_above_range: bool = False
|
|
||||||
|
|
||||||
|
|
||||||
# ── Project / operator metadata ───────────────────────────────────────────────
|
# ── Project / operator metadata ───────────────────────────────────────────────
|
||||||
|
|||||||
+232
-231
@@ -35,8 +35,6 @@ from .framing import (
|
|||||||
token_params,
|
token_params,
|
||||||
bulk_waveform_params,
|
bulk_waveform_params,
|
||||||
bulk_waveform_term_params,
|
bulk_waveform_term_params,
|
||||||
bulk_waveform_term_v2,
|
|
||||||
parse_strt_end_offset,
|
|
||||||
POLL_PROBE,
|
POLL_PROBE,
|
||||||
POLL_DATA,
|
POLL_DATA,
|
||||||
SESSION_RESET,
|
SESSION_RESET,
|
||||||
@@ -124,22 +122,16 @@ DATA_LENGTHS: dict[int, int] = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
# SUB 5A (BULK_WAVEFORM_STREAM) protocol constants.
|
# SUB 5A (BULK_WAVEFORM_STREAM) protocol constants.
|
||||||
#
|
# Confirmed from 1-2-26 BW TX capture analysis (2026-04-02).
|
||||||
# 2026-05-01 minimal-fix: the chunk-counter walk is now bounded by the event's
|
_BULK_CHUNK_OFFSET = 0x1004 # offset field for probe + all regular chunk requests ✅
|
||||||
# `end_offset` extracted from the STRT record at data[23:27] of the probe
|
_BULK_TERM_OFFSET = 0x005A # offset field for termination request ✅
|
||||||
# response. Without this bound the loop kept asking for chunks past the event
|
_BULK_COUNTER_STEP = 0x0400 # chunk counter increment per chunk ✅
|
||||||
# end and the device responded with post-event circular-buffer garbage,
|
# Chunk counter formula: key4[2:4] + (chunk_num - 1) * 0x0400
|
||||||
# corrupting reconstructed Blastware files for events ≥ 2 sec.
|
# where key4[2:4] is the event's circular-buffer base offset ((key4[2]<<8)|key4[3]).
|
||||||
#
|
# Earlier captures showed 0x1004 for chunk 1 of key 01110000 — that was a Blastware
|
||||||
# We keep the OLD 0x0400 chunk step here (BW actually uses 0x0200 — see §7.8.5
|
# artifact. For keys where key4[2:4] != 0x0000 (e.g. key 01111884) the old
|
||||||
# of the protocol reference for the corrected understanding) because the
|
# "n * 0x0400" formula sends counters from the wrong buffer region and the device
|
||||||
# existing blastware_file.py builder relies on the 0x0400-step frame structure
|
# returns data from a different event. Confirmed correct 2026-04-24.
|
||||||
# to produce valid files. Switching to BW's 0x0200 step is a separate task
|
|
||||||
# that also requires updating the file builder.
|
|
||||||
# BW-exact protocol values (v0.14.0). Verified against 4-27-26 + 5-1-26 captures.
|
|
||||||
_BULK_CHUNK_OFFSET = 0x1002 # offset_word for probe + all chunk requests
|
|
||||||
_BULK_TERM_OFFSET = 0x005A # offset_word for the legacy terminator (fallback only)
|
|
||||||
_BULK_COUNTER_STEP = 0x0200 # chunk counter increment (matches chunk payload size)
|
|
||||||
|
|
||||||
# Default timeout values (seconds).
|
# Default timeout values (seconds).
|
||||||
# MiniMate Plus is a slow device — keep these generous.
|
# MiniMate Plus is a slow device — keep these generous.
|
||||||
@@ -534,270 +526,223 @@ class MiniMateProtocol:
|
|||||||
self,
|
self,
|
||||||
key4: bytes,
|
key4: bytes,
|
||||||
*,
|
*,
|
||||||
stop_after_metadata: bool = True, # DEPRECATED — no-op under BW-exact walk
|
stop_after_metadata: bool = True,
|
||||||
max_chunks: int = 256, # safety cap only; loop is bounded by end_offset
|
max_chunks: int = 32,
|
||||||
include_terminator: bool = False,
|
include_terminator: bool = False,
|
||||||
extra_chunks_after_metadata: int = 1, # DEPRECATED — no-op
|
extra_chunks_after_metadata: int = 1,
|
||||||
) -> list[S3Frame]:
|
) -> list[S3Frame]:
|
||||||
"""
|
"""
|
||||||
Download the SUB 5A (BULK_WAVEFORM_STREAM) A5 frames for one event using
|
Download the SUB 5A (BULK_WAVEFORM_STREAM) A5 frames for one event.
|
||||||
Blastware's exact protocol. REWRITTEN 2026-05-02 (v0.14.0).
|
|
||||||
|
|
||||||
Algorithm (matches BW captures across 2-sec / 3-sec / event-2):
|
The bulk waveform stream carries both raw ADC samples (large) and
|
||||||
|
event-time metadata strings ("Project:", "Client:", "User Name:",
|
||||||
|
"Seis Loc:", "Extended Notes") embedded in one of the middle frames
|
||||||
|
(confirmed: A5[7] of 9 for 1-2-26 capture).
|
||||||
|
|
||||||
1. Probe
|
Protocol is request-per-chunk, NOT a continuous stream:
|
||||||
- For events at start_key[2:4] = 0x0000 (first event after erase
|
1. Probe (offset=_BULK_CHUNK_OFFSET, is_probe=True, counter=0x0000)
|
||||||
/ wrap): probe at counter=0x0000 with full key in params.
|
2. Chunks (offset=_BULK_CHUNK_OFFSET, is_probe=False, counter+=0x0400)
|
||||||
- For continuation events (start_key[2:4] != 0): first chunk at
|
3. Loop until metadata found (stop_after_metadata=True) or max_chunks
|
||||||
counter = start_key[2:4] + 0x0046; acts as both probe and
|
4. Termination (offset=_BULK_TERM_OFFSET, counter=last+_BULK_COUNTER_STEP)
|
||||||
first sample chunk; response carries STRT.
|
Device responds with a final A5 frame (page_key=0x0000).
|
||||||
|
|
||||||
2. Parse end_offset from STRT record at data[23:27] of the probe response.
|
By default the termination frame (page_key=0x0000) is NOT included in the
|
||||||
|
returned list. Pass include_terminator=True to append it; the blastware_file
|
||||||
|
writer needs the terminator frame's body to reconstruct the waveform file footer.
|
||||||
|
|
||||||
3. Read two fixed metadata pages at counter=0x1002 and counter=0x1004
|
Args:
|
||||||
— global session metadata (Project / Client / User Name / Seis Loc
|
key4: 4-byte waveform key from EVENT_HEADER (1E).
|
||||||
/ Extended Notes ASCII strings). Event 1 only; continuation
|
stop_after_metadata: If True (default), send termination as soon as
|
||||||
events skip these (BW caches them across the session).
|
b"Project:" is found in a frame's data — avoids
|
||||||
|
downloading the full ADC waveform payload (several
|
||||||
4. Walk sample chunks at 0x0200 increments, starting from 0x0600 for
|
hundred KB). Set False to download everything.
|
||||||
event 1 or `start + 0x0046 + 0x0200` for continuation events.
|
max_chunks: Safety cap on the number of chunk requests sent
|
||||||
Stop when `next_chunk + 0x0200 > end_offset`.
|
(default 32; a typical event uses 9 large frames).
|
||||||
|
include_terminator: If True, append the terminator A5 frame
|
||||||
5. Send TERM frame with offset_word and params computed by
|
(page_key=0x0000) to the returned list. The
|
||||||
`bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`.
|
terminator carries the waveform file footer bytes.
|
||||||
The TERM response contains the partial last chunk (residual =
|
Default False preserves existing caller behaviour.
|
||||||
end_offset - next_boundary) including the 26-byte 0e 08 file
|
|
||||||
footer.
|
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
List of S3Frame objects from each A5 response (probe, metadata
|
List of S3Frame objects from each A5 response frame. Frame indices
|
||||||
pages, sample chunks, optional TERM response). Caller passes
|
match the request sequence: index 0 = probe response, index 1 = first
|
||||||
`include_terminator=True` (e.g. write_blastware_file) to keep the
|
chunk, etc. If include_terminator=True, the last element is the
|
||||||
TERM response in the list — it's required to reconstruct the
|
terminator frame (page_key=0x0000).
|
||||||
file footer.
|
|
||||||
|
|
||||||
Deprecated kwargs:
|
|
||||||
stop_after_metadata: legacy "Project:"-string-based stop condition.
|
|
||||||
No-op under the BW-exact walk; the loop is
|
|
||||||
deterministically bounded by end_offset from
|
|
||||||
STRT. Accepted for backward compat.
|
|
||||||
extra_chunks_after_metadata: same.
|
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
ProtocolError: on timeout / bad checksum / unexpected SUB.
|
ProtocolError: on timeout, bad checksum, or unexpected SUB.
|
||||||
|
|
||||||
|
Confirmed from 1-2-26 BW TX/RX captures (2026-04-02):
|
||||||
|
- probe + 8 regular chunks + 1 termination = 10 TX frames
|
||||||
|
- 9 large A5 responses + 1 terminator A5 = 10 RX frames
|
||||||
|
- page_key=0x0010 on large frames; page_key=0x0000 on terminator ✅
|
||||||
|
- "Project:" metadata at A5[7].data[626] ✅
|
||||||
"""
|
"""
|
||||||
if len(key4) != 4:
|
if len(key4) != 4:
|
||||||
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
|
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
|
||||||
|
|
||||||
# Quietly accept and warn on deprecated kwargs.
|
rsp_sub = _expected_rsp_sub(SUB_BULK_WAVEFORM) # 0xFF - 0x5A = 0xA5
|
||||||
if not stop_after_metadata:
|
|
||||||
log.debug("5A: stop_after_metadata=False is no-op under BW-exact walk")
|
|
||||||
if extra_chunks_after_metadata not in (0, 1):
|
|
||||||
log.debug("5A: extra_chunks_after_metadata=%d is no-op under BW-exact walk",
|
|
||||||
extra_chunks_after_metadata)
|
|
||||||
|
|
||||||
rsp_sub = _expected_rsp_sub(SUB_BULK_WAVEFORM) # 0xA5
|
|
||||||
frames_data: list[S3Frame] = []
|
frames_data: list[S3Frame] = []
|
||||||
|
counter = 0
|
||||||
|
|
||||||
start_offset = (key4[2] << 8) | key4[3]
|
# BW counter formula (confirmed from 4-3-26 capture for key 0111245a,
|
||||||
is_event_1 = (start_offset == 0)
|
# and empirical live-device test 2026-04-06 for key 01110000):
|
||||||
|
# counter for chunk n = max(key4[2:4], 0x0400) + (n - 1) * 0x0400
|
||||||
|
# key4[2:4] is the event's circular-buffer base offset. The max() guard
|
||||||
|
# ensures chunk 1 never uses counter=0x0000 (which equals the probe address
|
||||||
|
# and causes the device to re-return STRT record data for the first chunk).
|
||||||
|
_key4_offset = (key4[2] << 8) | key4[3]
|
||||||
|
|
||||||
# ── Step 1: probe / first chunk ──────────────────────────────────────
|
# ── Step 1: probe ────────────────────────────────────────────────────
|
||||||
if is_event_1:
|
log.debug("5A probe key=%s key4_offset=0x%04X", key4.hex(), _key4_offset)
|
||||||
probe_counter = 0
|
params = bulk_waveform_params(key4, 0, is_probe=True)
|
||||||
probe_params = bulk_waveform_params(key4, 0, is_probe=True)
|
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
|
||||||
log.debug("5A probe (event-1) key=%s counter=0x0000", key4.hex())
|
self._parser.reset() # reset bytes_fed counter before probe recv
|
||||||
else:
|
|
||||||
# Continuation events: first 5A request lands at counter = key[2:4]
|
|
||||||
# (i.e. the address of the off=0x46 WAVEHDR record returned by 1F).
|
|
||||||
# The probe response carries STRT at byte 17 with end_offset.
|
|
||||||
#
|
|
||||||
# Confirmed 2026-05-04 from 5-1-26 "copy 2nd address" capture
|
|
||||||
# (BW probes counter=0x2238 with key=01112238, STRT@17 end=0x417E)
|
|
||||||
# and 5-4-26 BW captures (2-sec event probes counter=0x2238).
|
|
||||||
#
|
|
||||||
# The earlier "+0x46" formula in the doc came from calling
|
|
||||||
# start_key the BOUNDARY (off=0x2C) key, but the iteration walk
|
|
||||||
# uses 1F's off=0x46 key as cur_key, which already incorporates
|
|
||||||
# the +0x46 offset relative to the boundary. Adding it again
|
|
||||||
# caused the probe to overshoot, miss STRT, and run uncapped.
|
|
||||||
probe_counter = start_offset
|
|
||||||
probe_params = bulk_waveform_params(key4, probe_counter)
|
|
||||||
log.debug(
|
|
||||||
"5A probe (event-N) key=%s counter=0x%04X",
|
|
||||||
key4.hex(), probe_counter,
|
|
||||||
)
|
|
||||||
|
|
||||||
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, probe_params))
|
|
||||||
self._parser.reset()
|
|
||||||
try:
|
try:
|
||||||
rsp = self._recv_one(expected_sub=rsp_sub, reset_parser=False)
|
probe_batch = self._recv_5a_batch(rsp_sub)
|
||||||
except TimeoutError:
|
except TimeoutError:
|
||||||
log.warning(
|
log.warning(
|
||||||
"5A probe TIMED OUT for key=%s — %d raw bytes received",
|
"5A probe TIMED OUT for key=%s — "
|
||||||
|
"%d raw bytes received (no complete A5 frame assembled)",
|
||||||
key4.hex(), self._parser.bytes_fed,
|
key4.hex(), self._parser.bytes_fed,
|
||||||
)
|
)
|
||||||
raise
|
raise
|
||||||
|
frames_data.extend(probe_batch)
|
||||||
frames_data.append(rsp)
|
|
||||||
log.debug("5A A5[0] (probe) page_key=0x%04X %d bytes",
|
|
||||||
rsp.page_key, len(rsp.data))
|
|
||||||
|
|
||||||
# ── Step 2: parse STRT end_offset from probe response ────────────────
|
|
||||||
end_offset = parse_strt_end_offset(rsp.data)
|
|
||||||
if end_offset is None:
|
|
||||||
log.warning(
|
|
||||||
"5A probe response did not contain a STRT record; "
|
|
||||||
"cannot bound chunk loop — falling back to max_chunks=%d cap",
|
|
||||||
max_chunks,
|
|
||||||
)
|
|
||||||
end_offset = 0xFFFF # impossible value → loop runs to max_chunks
|
|
||||||
else:
|
|
||||||
log.info(
|
|
||||||
"5A STRT start_offset=0x%04X end_offset=0x%04X size=0x%04X",
|
|
||||||
start_offset, end_offset, end_offset - start_offset,
|
|
||||||
)
|
|
||||||
|
|
||||||
# ── Step 3: metadata pages 0x1002 + 0x1004 (event 1 only) ────────────
|
|
||||||
# Confirmed from BW captures: BW reads these two fixed device-buffer
|
|
||||||
# pages immediately after the probe for events at start_key[2:4]=0.
|
|
||||||
# Continuation events skip them (BW caches across the session).
|
|
||||||
# Their content is global compliance-setup metadata: Project, Client,
|
|
||||||
# User Name, Seis Loc, Extended Notes.
|
|
||||||
if is_event_1:
|
|
||||||
for meta_counter in (0x1002, 0x1004):
|
|
||||||
# Metadata page params have an extra trailing 0x00 byte
|
|
||||||
# (12-byte params instead of 11) — empirical from BW captures.
|
|
||||||
# Checksum-neutral but matches BW byte-for-byte.
|
|
||||||
meta_params = bytes([
|
|
||||||
0x00,
|
|
||||||
key4[0], key4[1],
|
|
||||||
(meta_counter >> 8) & 0xFF,
|
|
||||||
meta_counter & 0xFF,
|
|
||||||
0, 0, 0, 0, 0, 0, 0,
|
|
||||||
])
|
|
||||||
log.debug("5A metadata page counter=0x%04X", meta_counter)
|
|
||||||
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, meta_params))
|
|
||||||
self._parser.reset()
|
|
||||||
try:
|
|
||||||
meta_rsp = self._recv_one(
|
|
||||||
expected_sub=rsp_sub, reset_parser=False, timeout=10.0,
|
|
||||||
)
|
|
||||||
except TimeoutError:
|
|
||||||
log.warning(
|
|
||||||
"5A metadata page 0x%04X TIMED OUT — continuing",
|
|
||||||
meta_counter,
|
|
||||||
)
|
|
||||||
continue
|
|
||||||
frames_data.append(meta_rsp)
|
|
||||||
log.debug(
|
log.debug(
|
||||||
"5A meta@0x%04X page_key=0x%04X %d bytes",
|
"5A probe: %d frame(s) page_keys=%s",
|
||||||
meta_counter, meta_rsp.page_key, len(meta_rsp.data),
|
len(probe_batch),
|
||||||
|
[f"0x{f.page_key:04X}" for f in probe_batch],
|
||||||
)
|
)
|
||||||
|
|
||||||
# ── Step 4: sample chunk loop, bounded by end_offset ─────────────────
|
# Log probe frame size for diagnostics.
|
||||||
# Sample chunks start at:
|
# The device always needs extra_chunks_after_metadata chunks after the
|
||||||
# event 1: counter = 0x0600
|
# metadata frame before termination to prime the valid waveform footer.
|
||||||
# event N (>0): counter = probe_counter + 0x0200
|
# This holds regardless of TCP frame size (1-frame vs 2-frame mode).
|
||||||
# (probe was the first sample chunk)
|
_effective_extra_chunks = extra_chunks_after_metadata
|
||||||
if is_event_1:
|
log.warning(
|
||||||
counter = 0x0600
|
"5A probe data_len=%d effective_extra_chunks=%d",
|
||||||
else:
|
len(probe_batch[0].data),
|
||||||
counter = probe_counter + _BULK_COUNTER_STEP
|
_effective_extra_chunks,
|
||||||
|
|
||||||
last_chunk_counter: Optional[int] = (
|
|
||||||
probe_counter if not is_event_1 else None
|
|
||||||
)
|
)
|
||||||
chunks_fetched = 0
|
|
||||||
|
|
||||||
while chunks_fetched < max_chunks:
|
|
||||||
# Stop when next chunk would straddle the event end.
|
|
||||||
if counter + _BULK_COUNTER_STEP > end_offset:
|
|
||||||
log.debug(
|
|
||||||
"5A chunk loop done at counter=0x%04X (end=0x%04X); "
|
|
||||||
"%d chunks fetched",
|
|
||||||
counter, end_offset, chunks_fetched,
|
|
||||||
)
|
|
||||||
break
|
|
||||||
|
|
||||||
|
# ── Step 2: chunk loop ───────────────────────────────────────────────
|
||||||
|
# Counter formula: _chunk_base + (chunk_num - 1) * 0x0400
|
||||||
|
# where _chunk_base = max(key4[2:4], 0x0400).
|
||||||
|
#
|
||||||
|
# For events with key4[2:4] != 0 (e.g. key 0111245a, offset 0x245a):
|
||||||
|
# _chunk_base = 0x245a → chunk 1=0x245a, chunk 2=0x285a, ...
|
||||||
|
# Confirmed from 4-3-26 capture.
|
||||||
|
#
|
||||||
|
# For events with key4[2:4] == 0 (e.g. key 01110000):
|
||||||
|
# _chunk_base = max(0, 0x0400) = 0x0400
|
||||||
|
# → chunk 1=0x0400, chunk 2=0x0800, ... (= old chunk_num*0x0400)
|
||||||
|
# CRITICAL: counter=0x0000 (same as the probe) causes the device to
|
||||||
|
# re-return the STRT record data for chunk 1, making frame 1 look like
|
||||||
|
# a second probe response (confirmed from server log: frame 1 len=1097,
|
||||||
|
# contains STRT\xff\xfe, contributes zero body bytes after DLE-strip).
|
||||||
|
# counter=0x0400 for chunk 1 confirmed working (empirical test 2026-04-06).
|
||||||
|
_chunk_base = max(_key4_offset, _BULK_COUNTER_STEP)
|
||||||
|
for chunk_num in range(1, max_chunks + 1):
|
||||||
|
counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP
|
||||||
params = bulk_waveform_params(key4, counter)
|
params = bulk_waveform_params(key4, counter)
|
||||||
log.debug("5A chunk #%d counter=0x%04X", chunks_fetched + 1, counter)
|
log.debug("5A chunk %d counter=0x%04X", chunk_num, counter)
|
||||||
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
|
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
|
||||||
self._parser.reset()
|
self._parser.reset() # reset bytes_fed for accurate per-chunk count
|
||||||
try:
|
try:
|
||||||
rsp = self._recv_one(
|
# Collect ALL frames from this chunk response.
|
||||||
expected_sub=rsp_sub, reset_parser=False, timeout=10.0,
|
# Over TCP via modem, a single large A5 device response (~1100 bytes
|
||||||
)
|
# RS-232) is split across ~2 TCP segments, each parsed as its own
|
||||||
|
# complete S3 frame. _recv_5a_batch gathers all of them so that
|
||||||
|
# every subsequent chunk request is paired with the correct response.
|
||||||
|
batch = self._recv_5a_batch(rsp_sub, first_timeout=10.0)
|
||||||
except TimeoutError:
|
except TimeoutError:
|
||||||
raw = self._parser.bytes_fed
|
raw = self._parser.bytes_fed
|
||||||
log.warning(
|
log.warning(
|
||||||
"5A TIMEOUT chunk=%d counter=0x%04X raw_bytes=%d",
|
"5A TIMEOUT chunk=%d counter=0x%04X raw_bytes=%d",
|
||||||
chunks_fetched + 1, counter, raw,
|
chunk_num, counter, raw,
|
||||||
)
|
)
|
||||||
if raw > 0 and frames_data:
|
if raw > 0 and frames_data:
|
||||||
|
# Device sent a partial byte (likely a bare DLE/ETX end-of-stream
|
||||||
|
# signal) but never completed a full frame. Treat as graceful
|
||||||
|
# stream end and fall through to the termination step.
|
||||||
log.warning(
|
log.warning(
|
||||||
"5A unexpected end-of-stream — proceeding to TERM",
|
"5A end-of-stream detected at chunk=%d (raw_bytes=%d, "
|
||||||
|
"frames_collected=%d) — proceeding to termination",
|
||||||
|
chunk_num, raw, len(frames_data),
|
||||||
)
|
)
|
||||||
break
|
break
|
||||||
raise
|
raise
|
||||||
|
|
||||||
log.debug(
|
# Process all frames from this batch.
|
||||||
"5A RX chunk=%d page_key=0x%04X data_len=%d",
|
metadata_found = False
|
||||||
chunks_fetched + 1, rsp.page_key, len(rsp.data),
|
for rsp in batch:
|
||||||
)
|
|
||||||
|
|
||||||
if rsp.page_key == 0x0000:
|
|
||||||
# Device terminated mid-stream unexpectedly.
|
|
||||||
log.warning(
|
log.warning(
|
||||||
"5A unexpected page_key=0x0000 mid-stream at counter=0x%04X",
|
"5A RX chunk=%d page_key=0x%04X data_len=%d contains_Project=%s",
|
||||||
counter,
|
chunk_num, rsp.page_key, len(rsp.data), b"Project:" in rsp.data,
|
||||||
)
|
)
|
||||||
|
if rsp.page_key == 0x0000:
|
||||||
|
# Device unexpectedly terminated mid-stream.
|
||||||
|
log.debug("5A page_key=0x0000 — device terminated early")
|
||||||
if include_terminator:
|
if include_terminator:
|
||||||
frames_data.append(rsp)
|
frames_data.append(rsp)
|
||||||
return frames_data
|
return frames_data
|
||||||
|
|
||||||
frames_data.append(rsp)
|
frames_data.append(rsp)
|
||||||
last_chunk_counter = counter
|
if stop_after_metadata and b"Project:" in rsp.data:
|
||||||
counter += _BULK_COUNTER_STEP
|
metadata_found = True
|
||||||
chunks_fetched += 1
|
|
||||||
else:
|
|
||||||
log.warning(
|
|
||||||
"5A reached max_chunks=%d at counter=0x%04X (end=0x%04X)",
|
|
||||||
max_chunks, counter, end_offset,
|
|
||||||
)
|
|
||||||
|
|
||||||
# ── Step 5: TERM with proper end_offset-derived formula ──────────────
|
if metadata_found:
|
||||||
if last_chunk_counter is None or end_offset == 0xFFFF:
|
# Download extra_chunks_after_metadata more chunks after metadata.
|
||||||
# No STRT or no chunks fetched — fall back to legacy TERM.
|
# This primes the device to return the valid waveform footer in the
|
||||||
log.warning(
|
# termination response — without it the terminator carries too few bytes
|
||||||
"5A using legacy TERM (offset_word=0x005A); "
|
# (confirmed 2026-04-23). The extra chunk data also belongs in the
|
||||||
"end_offset unavailable or no chunks fetched",
|
# file body (confirmed from TCP capture analysis 2026-04-27).
|
||||||
)
|
log.debug("5A metadata found — fetching %d more chunk(s)",
|
||||||
legacy_counter = (last_chunk_counter or probe_counter) + _BULK_COUNTER_STEP
|
_effective_extra_chunks)
|
||||||
term_offset_word = _BULK_TERM_OFFSET # 0x005A
|
for _extra_n in range(_effective_extra_chunks):
|
||||||
term_params = bulk_waveform_term_params(key4, legacy_counter)
|
chunk_num += 1
|
||||||
else:
|
counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP
|
||||||
term_offset_word, term_params = bulk_waveform_term_v2(
|
params = bulk_waveform_params(key4, counter)
|
||||||
key4, end_offset, last_chunk_counter,
|
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
|
||||||
)
|
|
||||||
log.debug(
|
|
||||||
"5A TERM offset_word=0x%04X params[2:4]=%s end=0x%04X "
|
|
||||||
"last_chunk=0x%04X",
|
|
||||||
term_offset_word, term_params[2:4].hex(),
|
|
||||||
end_offset, last_chunk_counter,
|
|
||||||
)
|
|
||||||
|
|
||||||
self._send(build_5a_frame(term_offset_word, term_params))
|
|
||||||
try:
|
try:
|
||||||
term_rsp = self._recv_one(expected_sub=rsp_sub, timeout=10.0)
|
extra_batch = self._recv_5a_batch(rsp_sub, first_timeout=10.0)
|
||||||
log.info(
|
for ef in extra_batch:
|
||||||
"5A TERM response page_key=0x%04X %d bytes",
|
log.debug(
|
||||||
|
"5A extra chunk page_key=0x%04X data_len=%d",
|
||||||
|
ef.page_key, len(ef.data),
|
||||||
|
)
|
||||||
|
if ef.page_key == 0x0000:
|
||||||
|
if include_terminator:
|
||||||
|
frames_data.append(ef)
|
||||||
|
return frames_data
|
||||||
|
frames_data.append(ef)
|
||||||
|
except TimeoutError:
|
||||||
|
log.debug("5A extra chunk %d timed out — end of stream", _extra_n + 1)
|
||||||
|
break
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
log.warning(
|
||||||
|
"5A reached max_chunks=%d without end-of-stream; sending termination",
|
||||||
|
max_chunks,
|
||||||
|
)
|
||||||
|
|
||||||
|
# ── Step 3: termination ──────────────────────────────────────────────
|
||||||
|
term_counter = counter + _BULK_COUNTER_STEP
|
||||||
|
term_params = bulk_waveform_term_params(key4, term_counter)
|
||||||
|
log.debug(
|
||||||
|
"5A termination term_counter=0x%04X offset=0x%04X",
|
||||||
|
term_counter, _BULK_TERM_OFFSET,
|
||||||
|
)
|
||||||
|
self._send(build_5a_frame(_BULK_TERM_OFFSET, term_params))
|
||||||
|
try:
|
||||||
|
term_rsp = self._recv_one(expected_sub=rsp_sub)
|
||||||
|
log.debug(
|
||||||
|
"5A termination response page_key=0x%04X %d bytes",
|
||||||
term_rsp.page_key, len(term_rsp.data),
|
term_rsp.page_key, len(term_rsp.data),
|
||||||
)
|
)
|
||||||
if include_terminator:
|
if include_terminator:
|
||||||
frames_data.append(term_rsp)
|
frames_data.append(term_rsp)
|
||||||
except TimeoutError:
|
except TimeoutError:
|
||||||
log.warning("5A no TERM response (timeout)")
|
log.debug("5A no termination response — device may have already closed")
|
||||||
|
|
||||||
return frames_data
|
return frames_data
|
||||||
|
|
||||||
@@ -937,7 +882,7 @@ class MiniMateProtocol:
|
|||||||
continue
|
continue
|
||||||
|
|
||||||
chunk = data_rsp.data[11:]
|
chunk = data_rsp.data[11:]
|
||||||
log.debug(
|
log.warning(
|
||||||
"read_compliance_config: frame %s page=0x%04X data=%d cfg_chunk=%d running_total=%d",
|
"read_compliance_config: frame %s page=0x%04X data=%d cfg_chunk=%d running_total=%d",
|
||||||
step_name, data_rsp.page_key, len(data_rsp.data),
|
step_name, data_rsp.page_key, len(data_rsp.data),
|
||||||
len(chunk), len(config) + len(chunk),
|
len(chunk), len(config) + len(chunk),
|
||||||
@@ -957,18 +902,17 @@ class MiniMateProtocol:
|
|||||||
except TimeoutError:
|
except TimeoutError:
|
||||||
pass
|
pass
|
||||||
|
|
||||||
log.info(
|
log.warning(
|
||||||
"read_compliance_config: done — %d cfg bytes total",
|
"read_compliance_config: done — %d cfg bytes total",
|
||||||
len(config),
|
len(config),
|
||||||
)
|
)
|
||||||
|
|
||||||
# Hex dump first 128 bytes — useful only for field-mapping work, not normal operation.
|
# Hex dump first 128 bytes for field mapping
|
||||||
if log.isEnabledFor(logging.DEBUG):
|
|
||||||
for row in range(0, min(len(config), 128), 16):
|
for row in range(0, min(len(config), 128), 16):
|
||||||
row_bytes = bytes(config[row:row + 16])
|
row_bytes = bytes(config[row:row + 16])
|
||||||
hex_part = ' '.join(f'{b:02x}' for b in row_bytes)
|
hex_part = ' '.join(f'{b:02x}' for b in row_bytes)
|
||||||
asc_part = ''.join(chr(b) if 32 <= b < 127 else '.' for b in row_bytes)
|
asc_part = ''.join(chr(b) if 32 <= b < 127 else '.' for b in row_bytes)
|
||||||
log.debug(" cfg[%04x]: %-48s %s", row, hex_part, asc_part)
|
log.warning(" cfg[%04x]: %-48s %s", row, hex_part, asc_part)
|
||||||
|
|
||||||
return bytes(config)
|
return bytes(config)
|
||||||
|
|
||||||
@@ -1459,6 +1403,63 @@ class MiniMateProtocol:
|
|||||||
log.debug("TX %d bytes: %s", len(frame), frame.hex())
|
log.debug("TX %d bytes: %s", len(frame), frame.hex())
|
||||||
self._transport.write(frame)
|
self._transport.write(frame)
|
||||||
|
|
||||||
|
def _recv_5a_batch(
|
||||||
|
self,
|
||||||
|
expected_sub: int,
|
||||||
|
first_timeout: float = 10.0,
|
||||||
|
batch_timeout: float = 0.5,
|
||||||
|
) -> list[S3Frame]:
|
||||||
|
"""
|
||||||
|
Collect all S3 frames that arrive as part of one device response.
|
||||||
|
|
||||||
|
Over TCP via cellular modem, a single device A5 response (~1100 bytes of
|
||||||
|
RS-232 data) is forwarded in multiple TCP segments due to the modem's
|
||||||
|
data-forwarding timeout (~100-150 ms per segment). Each TCP segment
|
||||||
|
contains a complete, valid S3 frame (~550 bytes). Calling _recv_one()
|
||||||
|
once returns only the first segment's frame and misses the rest, causing
|
||||||
|
the chunk request/response pairing to cascade out of alignment.
|
||||||
|
|
||||||
|
This helper collects ALL frames before returning, by trying additional
|
||||||
|
short-timeout receives after the first frame arrives.
|
||||||
|
|
||||||
|
The caller must call self._parser.reset() before this method to ensure
|
||||||
|
bytes_fed is accurate; this method always uses reset_parser=False.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
expected_sub: Expected SUB byte for validation.
|
||||||
|
first_timeout: Timeout for the mandatory first frame. Should be
|
||||||
|
generous (default 10 s) since the device may be slow.
|
||||||
|
batch_timeout: Short timeout for subsequent frames. Default 0.5 s
|
||||||
|
— comfortably longer than the modem forwarding gap
|
||||||
|
(~150 ms) but short enough to avoid stalling when
|
||||||
|
only one frame is expected (probe, terminator).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of S3Frame objects in arrival order (at least one).
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
TimeoutError: If no frame arrives within first_timeout.
|
||||||
|
UnexpectedResponse: If any frame has the wrong SUB byte.
|
||||||
|
"""
|
||||||
|
frames: list[S3Frame] = []
|
||||||
|
first = self._recv_one(
|
||||||
|
expected_sub=expected_sub,
|
||||||
|
reset_parser=False,
|
||||||
|
timeout=first_timeout,
|
||||||
|
)
|
||||||
|
frames.append(first)
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
extra = self._recv_one(
|
||||||
|
expected_sub=expected_sub,
|
||||||
|
reset_parser=False,
|
||||||
|
timeout=batch_timeout,
|
||||||
|
)
|
||||||
|
frames.append(extra)
|
||||||
|
except TimeoutError:
|
||||||
|
break
|
||||||
|
return frames
|
||||||
|
|
||||||
def _recv_one(
|
def _recv_one(
|
||||||
self,
|
self,
|
||||||
expected_sub: Optional[int] = None,
|
expected_sub: Optional[int] = None,
|
||||||
|
|||||||
@@ -454,102 +454,3 @@ class SocketTransport(TcpTransport):
|
|||||||
|
|
||||||
def __repr__(self) -> str:
|
def __repr__(self) -> str:
|
||||||
return f"SocketTransport(peer={self.host!r})"
|
return f"SocketTransport(peer={self.host!r})"
|
||||||
|
|
||||||
|
|
||||||
# ── Capturing transport (MITM-style raw byte mirror) ──────────────────────────
|
|
||||||
|
|
||||||
class CapturingTransport(BaseTransport):
|
|
||||||
"""
|
|
||||||
Wraps another BaseTransport and mirrors every byte to two raw capture files:
|
|
||||||
|
|
||||||
raw_bw_<...>.bin — bytes WE wrote to the device (BW-side TX)
|
|
||||||
raw_s3_<...>.bin — bytes the device wrote back (S3-side TX)
|
|
||||||
|
|
||||||
The file naming and on-wire byte layout are identical to the captures
|
|
||||||
produced by `bridges/ach_mitm.py`, so the resulting `.bin` files can be
|
|
||||||
loaded directly by the Analyzer (File > Open Capture) and parsed by the
|
|
||||||
same tooling used for genuine Blastware MITM captures.
|
|
||||||
|
|
||||||
All BaseTransport methods are forwarded to the inner transport; the only
|
|
||||||
side-effect is that successful read/write byte streams are appended to the
|
|
||||||
two open binary files.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
inner: An already-built BaseTransport (SerialTransport / TcpTransport).
|
|
||||||
bw_path: File path for the "BW TX" stream (bytes we send). Opened "wb".
|
|
||||||
s3_path: File path for the "S3 TX" stream (bytes the device sends).
|
|
||||||
Opened "wb".
|
|
||||||
|
|
||||||
Example:
|
|
||||||
with CapturingTransport(TcpTransport("1.2.3.4", 9034),
|
|
||||||
"raw_bw.bin", "raw_s3.bin") as t:
|
|
||||||
client = MiniMateClient(transport=t)
|
|
||||||
client.connect()
|
|
||||||
client.get_events()
|
|
||||||
# both .bin files now hold the full bidirectional capture.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, inner: BaseTransport, bw_path: str, s3_path: str) -> None:
|
|
||||||
self._inner = inner
|
|
||||||
self._bw_path = bw_path
|
|
||||||
self._s3_path = s3_path
|
|
||||||
self._bw_fh = None
|
|
||||||
self._s3_fh = None
|
|
||||||
# Forward inner attrs so callers can introspect (e.g. .host, .port).
|
|
||||||
self.host = getattr(inner, "host", None)
|
|
||||||
self.port = getattr(inner, "port", None)
|
|
||||||
|
|
||||||
# ── BaseTransport interface ───────────────────────────────────────────────
|
|
||||||
|
|
||||||
def connect(self) -> None:
|
|
||||||
if self._bw_fh is None:
|
|
||||||
self._bw_fh = open(self._bw_path, "wb", buffering=0)
|
|
||||||
if self._s3_fh is None:
|
|
||||||
self._s3_fh = open(self._s3_path, "wb", buffering=0)
|
|
||||||
self._inner.connect()
|
|
||||||
|
|
||||||
def disconnect(self) -> None:
|
|
||||||
try:
|
|
||||||
self._inner.disconnect()
|
|
||||||
finally:
|
|
||||||
for fh_attr in ("_bw_fh", "_s3_fh"):
|
|
||||||
fh = getattr(self, fh_attr)
|
|
||||||
if fh is not None:
|
|
||||||
try:
|
|
||||||
fh.flush()
|
|
||||||
fh.close()
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
setattr(self, fh_attr, None)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def is_connected(self) -> bool:
|
|
||||||
return self._inner.is_connected
|
|
||||||
|
|
||||||
def write(self, data: bytes) -> None:
|
|
||||||
self._inner.write(data)
|
|
||||||
if data and self._bw_fh is not None:
|
|
||||||
try:
|
|
||||||
self._bw_fh.write(data)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
def read(self, n: int) -> bytes:
|
|
||||||
got = self._inner.read(n)
|
|
||||||
if got and self._s3_fh is not None:
|
|
||||||
try:
|
|
||||||
self._s3_fh.write(got)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
return got
|
|
||||||
|
|
||||||
@property
|
|
||||||
def bw_path(self) -> str:
|
|
||||||
return self._bw_path
|
|
||||||
|
|
||||||
@property
|
|
||||||
def s3_path(self) -> str:
|
|
||||||
return self._s3_path
|
|
||||||
|
|
||||||
def __repr__(self) -> str:
|
|
||||||
return f"CapturingTransport({self._inner!r}, bw={self._bw_path!r}, s3={self._s3_path!r})"
|
|
||||||
|
|||||||
@@ -1,911 +0,0 @@
|
|||||||
"""
|
|
||||||
waveform_codec.py — block-walker and verified decoder for the MiniMate Plus
|
|
||||||
waveform-file body.
|
|
||||||
|
|
||||||
FULLY DECODED 2026-05-11. Every block type, every channel, and the
|
|
||||||
channel-rotation rule are verified byte-exact against BW's ASCII export
|
|
||||||
across the 9-event fixture bundle (47,364 ADC samples, zero errors).
|
|
||||||
|
|
||||||
The Blastware waveform-file body — the bytes between the 21-byte STRT
|
|
||||||
record and the 26-byte file footer — is a tagged variable-length block
|
|
||||||
stream with a custom delta + RLE codec. (Not raw int16 LE, which was
|
|
||||||
the historical wrong assumption that produced ±32K noise on every event.)
|
|
||||||
|
|
||||||
Current status:
|
|
||||||
|
|
||||||
- Block framing: ✅ solved (5 block types and lengths all confirmed)
|
|
||||||
- Per-channel decode: ✅ solved (Tran / Vert / Long / MicL all byte-exact)
|
|
||||||
- Channel rotation: ✅ Tran → Vert → Long → MicL per segment
|
|
||||||
- Segment header: ✅ fully decoded (anchor pair + prev-channel extension)
|
|
||||||
- 30 NN packed-delta block: ✅ NN × 12-bit signed deltas in NN/4 groups
|
|
||||||
- MicL → dB(L) conversion: ✅ ``mic_count_to_db`` matches BW display
|
|
||||||
- Production wiring: ✅ ``client.py:_decode_a5_waveform`` uses the new
|
|
||||||
codec (via ``decode_a5_frames``). ``.h5`` sidecars now render
|
|
||||||
correctly.
|
|
||||||
|
|
||||||
Known limitations:
|
|
||||||
|
|
||||||
- Walker stops early on the loudest events (SP0, SS0, SV0, event-b) at
|
|
||||||
some mid-segment edge cases not yet fully characterized. Every
|
|
||||||
sample reached IS correct; the walker just doesn't reach all of
|
|
||||||
them yet. The cleanly-decoded subset is still ~5000–15000 samples
|
|
||||||
per loud event.
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
Body layout (CONFIRMED 2026-05-11 against 8 fixture events)
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
[7-byte preamble] [stream of tagged blocks] [trailer]
|
|
||||||
|
|
||||||
The preamble is always exactly 7 bytes:
|
|
||||||
|
|
||||||
body[0:3] = 00 02 00 magic
|
|
||||||
body[3:5] = Tran[0] int16 BE in 16-count units (LSB = 0.005 in/s)
|
|
||||||
body[5:7] = Tran[1] int16 BE in 16-count units
|
|
||||||
|
|
||||||
(Earlier drafts of this module described a "7-or-9-byte preamble";
|
|
||||||
that was wrong — single-shot and continuous events both use 7 bytes.
|
|
||||||
The "extra 2 bytes" on continuous events were the first ``00 NN`` RLE
|
|
||||||
marker, not part of the preamble.)
|
|
||||||
|
|
||||||
Block types and lengths (all confirmed):
|
|
||||||
|
|
||||||
| Tag | Length | Meaning |
|
|
||||||
|----------|-----------------------|----------------------------------------|
|
|
||||||
| ``10 NN``| NN/2 + 2 bytes | 4-bit nibble deltas (2 per byte; high |
|
|
||||||
| | | nibble first; signed 0..7 / 8..F = -8..-1)|
|
|
||||||
| ``20 NN``| NN + 2 bytes | int8 signed deltas (1 per byte) |
|
|
||||||
| ``00 NN``| 2 bytes | RLE: append NN copies of current value |
|
|
||||||
| ``30 NN``| NN*2 in data, NN*4 | Unknown content. Only in loud events. |
|
|
||||||
| | in trailer | |
|
|
||||||
| ``40 02``| 20 bytes (fixed) | Segment header |
|
|
||||||
|
|
||||||
NN is always a multiple of 4.
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
Tran channel, segment 0 (CONFIRMED 2026-05-11)
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
Segment 0 — everything before the first ``40 02`` segment header — encodes
|
|
||||||
Tran samples only. Starting from preamble anchors Tran[0] and Tran[1],
|
|
||||||
each subsequent block contributes to the running Tran value:
|
|
||||||
|
|
||||||
10 NN → append NN deltas (4-bit signed nibbles)
|
|
||||||
20 NN → append NN deltas (int8 signed bytes)
|
|
||||||
00 NN → append NN copies of the current value (RLE zeros)
|
|
||||||
40 02 → segment 0 ends; multi-segment continuation is open
|
|
||||||
|
|
||||||
This decodes the first 482–510 samples of Tran for each event with zero
|
|
||||||
errors against BW's ASCII export. The exact segment-0 sample count
|
|
||||||
varies per event (it's bounded by a fixed device-flash byte budget, not
|
|
||||||
a fixed sample count — quiet events fit more samples because zero
|
|
||||||
deltas pack into ``00 NN`` markers compactly).
|
|
||||||
|
|
||||||
Implementation: :func:`decode_tran_initial`.
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
Segment header (40 02, 20 bytes total)
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
The 18-byte payload of the ``40 02`` block:
|
|
||||||
|
|
||||||
| Offset | Field | Status |
|
|
||||||
|-----------|---------------------------------------------|-------------|
|
|
||||||
| [0:2] | T_delta at first sample of new segment | ✅ confirmed|
|
|
||||||
| | (int16 BE, in 16-count units) | |
|
|
||||||
| [2:4] | Likely T_delta at sample seg_start+1 | 🟡 likely |
|
|
||||||
| [4:6] | Unknown (varies; possibly checksum) | ❓ open |
|
|
||||||
| [6:8] | Byte length to next segment header − 2 | ✅ confirmed|
|
|
||||||
| | (uint16 BE; useful for walker pre-scan) | |
|
|
||||||
| [8:12] | Monotonic uint32 LE counter | ✅ confirmed|
|
|
||||||
| | (starts ~0x47, increments by 1 per segment) | |
|
|
||||||
| [12:14] | Constant ``02 00`` | ✅ confirmed|
|
|
||||||
| [14:18] | Unknown 4-byte field | ❓ open |
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
What breaks the multi-segment decoder (the main open question)
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
After segment 0 ends and the segment header T_delta is consumed,
|
|
||||||
applying segment 1's blocks as Tran continuation produces values that
|
|
||||||
diverge from truth by sample ~512. The block structure inside segment
|
|
||||||
1 is IDENTICAL to segment 0 (same alternating 10 NN / 00 NN pattern),
|
|
||||||
and the delta budget matches the segment size exactly (V70 segment 1
|
|
||||||
has 264 nibble-deltas + 244 RLE zeros = 508 = the segment's sample
|
|
||||||
count). But the cumulative is wrong.
|
|
||||||
|
|
||||||
The strongest unverified hypothesis is that segments rotate channels:
|
|
||||||
|
|
||||||
segment 0 → Tran samples 0..509
|
|
||||||
segment 1 → Vert samples 0..507
|
|
||||||
segment 2 → Long samples 0..507
|
|
||||||
segment 3 → Mic samples 0..507
|
|
||||||
segment 4 → Tran samples 510..N (continuation)
|
|
||||||
...
|
|
||||||
|
|
||||||
This is consistent with the segment-1 block sums net-to-near-zero in
|
|
||||||
V70 (where all 4 channels are near zero) and with the per-segment delta
|
|
||||||
budget matching the segment size for a single channel. It is NOT yet
|
|
||||||
verified because the per-segment channel anchor isn't pinned down in
|
|
||||||
the segment header — bytes [4:6] and [14:18] of the header are still
|
|
||||||
open and probably encode V/L/M anchors.
|
|
||||||
|
|
||||||
See ``docs/waveform_codec_re_status.md`` for the current working notes
|
|
||||||
and the suggested next experiment ("segment-channel scoring analyzer").
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import math
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from typing import List, Optional, Tuple
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class WaveformBlock:
|
|
||||||
"""One tagged block parsed out of a Blastware waveform-file body."""
|
|
||||||
offset: int # byte offset into body
|
|
||||||
tag_hi: int # first tag byte (0x10 / 0x20 / 0x00 / 0x30 / 0x40)
|
|
||||||
tag_lo: int # second tag byte (NN)
|
|
||||||
data: bytes # block payload (excludes the 2-byte tag)
|
|
||||||
length: int # total block length on the wire (includes the tag)
|
|
||||||
|
|
||||||
@property
|
|
||||||
def kind(self) -> str:
|
|
||||||
return f"{self.tag_hi:02x} {self.tag_lo:02x}"
|
|
||||||
|
|
||||||
|
|
||||||
def find_data_start(body: bytes) -> int:
|
|
||||||
"""Auto-detect the offset of the first data block.
|
|
||||||
|
|
||||||
The body starts with a 7-byte preamble (magic ``00 02 00`` + two int16 BE
|
|
||||||
Tran anchors). After that, the data section starts with a tag — usually
|
|
||||||
``10 NN`` or ``20 NN``, but quiet events may begin with a ``00 NN`` RLE
|
|
||||||
marker. We return the offset of the first recognized tag.
|
|
||||||
"""
|
|
||||||
# Try fixed offset 7 first (canonical preamble length).
|
|
||||||
if len(body) >= 9:
|
|
||||||
b, nn = body[7], body[8]
|
|
||||||
# Accept the same tag vocabulary ``walk_body`` accepts, including the
|
|
||||||
# wide-NN forms (``0X``/``1X``/``2X``) and the variable-width ``40 NN``
|
|
||||||
# segment header.
|
|
||||||
if ((b & 0xF0) in (0x00, 0x10, 0x20) and nn % 4 == 0
|
|
||||||
and ((b & 0x0F) != 0 or 0 < nn <= 0xFC)) \
|
|
||||||
or (b == 0x30 and nn % 4 == 0 and 0 < nn <= 0xFC) \
|
|
||||||
or (b == 0x40 and 0 < nn <= 0x08) \
|
|
||||||
or is_tagless_segment_header(body, 7):
|
|
||||||
return 7
|
|
||||||
# Fall back to scanning the first 20 bytes.
|
|
||||||
for i in range(min(20, len(body) - 1)):
|
|
||||||
b = body[i]
|
|
||||||
nn = body[i + 1]
|
|
||||||
if b in (0x10, 0x20) and nn % 4 == 0 and 0 < nn <= 0xFC:
|
|
||||||
return i
|
|
||||||
return -1
|
|
||||||
|
|
||||||
|
|
||||||
# Channel-id byte carried in every segment header. Previously mis-read as a
|
|
||||||
# "monotonic uint32 LE counter"; it is really ``[channel][00][00][segment]``.
|
|
||||||
# Verified 2026-08-25 on 1697/1697 segment headers across the ground-truth
|
|
||||||
# corpus with zero disagreements against the decoded channel rotation.
|
|
||||||
SEGMENT_CHANNEL_IDS = {0x46: "Tran", 0x47: "Vert", 0x48: "Long", 0x49: "MicL"}
|
|
||||||
|
|
||||||
# A tagless segment header: the 14-byte tail of a ``40 NN`` header with no tag
|
|
||||||
# and no previous-channel continuation deltas (the NN=0 case).
|
|
||||||
_TAGLESS_HEADER_LEN = 14
|
|
||||||
|
|
||||||
|
|
||||||
def is_tagless_segment_header(body: bytes, i: int) -> bool:
|
|
||||||
"""True if a bare 14-byte segment header starts at *i*.
|
|
||||||
|
|
||||||
Layout ``[field2:2][len_to_next:2][channel_id:4][marker:2][anchors:4]``.
|
|
||||||
The discriminator is the 6 bytes at ``[4:10]``: a known channel id, two
|
|
||||||
zero bytes, a small segment index, and the ``01 00`` / ``02 00`` marker.
|
|
||||||
"""
|
|
||||||
if i + _TAGLESS_HEADER_LEN > len(body):
|
|
||||||
return False
|
|
||||||
return (body[i + 4] in SEGMENT_CHANNEL_IDS
|
|
||||||
and body[i + 5] == 0x00 and body[i + 6] == 0x00
|
|
||||||
and body[i + 8] in (0x01, 0x02) and body[i + 9] == 0x00)
|
|
||||||
|
|
||||||
|
|
||||||
def walk_body(body: bytes, start: Optional[int] = None) -> List[WaveformBlock]:
|
|
||||||
"""Walk the tagged-block sequence starting at *start* (auto-detected by default).
|
|
||||||
|
|
||||||
Stops when an unrecognized tag is encountered or end of body is reached.
|
|
||||||
Returned blocks are in stream order.
|
|
||||||
"""
|
|
||||||
if start is None:
|
|
||||||
start = find_data_start(body)
|
|
||||||
if start < 0:
|
|
||||||
return []
|
|
||||||
|
|
||||||
blocks: List[WaveformBlock] = []
|
|
||||||
i = start
|
|
||||||
while i + 1 < len(body):
|
|
||||||
t0 = body[i]
|
|
||||||
t1 = body[i + 1]
|
|
||||||
if t0 == 0x10 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
|
|
||||||
length = t1 // 2 + 2
|
|
||||||
elif (t0 & 0xF0) == 0x10 and (t0 & 0x0F) != 0 and t1 % 4 == 0:
|
|
||||||
# Wide-NN nibble block: ``1X NN`` where X is the high nibble of a
|
|
||||||
# 12-bit NN value. NN = ((t0 & 0x0F) << 8) | t1. Block length
|
|
||||||
# = NN/2 + 2 bytes (NN nibble deltas, same as ``10 NN`` semantics
|
|
||||||
# but with NN > 0xFC). Confirmed 2026-05-11 in SP0 segment 12
|
|
||||||
# where V continuation uses ``11 90`` = NN=0x190=400.
|
|
||||||
wide_nn = ((t0 & 0x0F) << 8) | t1
|
|
||||||
length = wide_nn // 2 + 2
|
|
||||||
elif t0 == 0x20 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
|
|
||||||
length = t1 + 2
|
|
||||||
elif (t0 & 0xF0) == 0x20 and (t0 & 0x0F) != 0 and t1 % 4 == 0:
|
|
||||||
# Wide-NN int8 block: ``2X NN`` extends NN to 12 bits the same way.
|
|
||||||
wide_nn = ((t0 & 0x0F) << 8) | t1
|
|
||||||
length = wide_nn + 2
|
|
||||||
elif (t0 & 0xF0) == 0x00 and t1 % 4 == 0:
|
|
||||||
# ``00 NN`` RLE zero-delta run, plus its wide form ``0X NN``
|
|
||||||
# (X != 0) which extends NN to 12 bits exactly like ``1X``/``2X``:
|
|
||||||
# NN = ((t0 & 0x0F) << 8) | t1. A narrow run maxes out at
|
|
||||||
# NN=0xFC, so quiet stretches longer than 252 samples must use
|
|
||||||
# the wide form. Confirmed 2026-08-25 against six production
|
|
||||||
# events (e.g. ``01 0c`` = 268 repeats in K558LKOF.460W).
|
|
||||||
length = 2
|
|
||||||
elif t0 == 0x30 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
|
|
||||||
# Data-section ``30 NN`` blocks carry NN 12-bit signed deltas packed
|
|
||||||
# as NN/4 groups of (2-byte high-nibble field + 4 × int8 low byte).
|
|
||||||
# Length = NN/4 × 6 + 2 = NN × 1.5 + 2 (= 8 for NN=4, 14 for NN=8,
|
|
||||||
# 20 for NN=12, etc.). Confirmed 2026-05-11 by full-decoder
|
|
||||||
# verification against BW ASCII export.
|
|
||||||
#
|
|
||||||
# Trailer-section ``30 NN`` blocks have a different length formula
|
|
||||||
# (NN × 4 = 32 for NN=8 in trailers). We try the data-section
|
|
||||||
# length first and fall back to the trailer length if needed.
|
|
||||||
cand_data = t1 * 3 // 2 + 2
|
|
||||||
cand_trailer = t1 * 4
|
|
||||||
if (i + cand_data < len(body) - 1
|
|
||||||
and body[i + cand_data] in (0x10, 0x20, 0x00, 0x30, 0x40)):
|
|
||||||
length = cand_data
|
|
||||||
else:
|
|
||||||
length = cand_trailer
|
|
||||||
elif t0 == 0x40 and 0 < t1 <= 0x08:
|
|
||||||
# ``40 NN`` segment header. NN is the number of int16 BE
|
|
||||||
# continuation deltas the header carries for the PREVIOUS
|
|
||||||
# channel, so the header grows with NN:
|
|
||||||
# length = 2 (tag) + 2*NN (deltas) + 14 (fixed tail)
|
|
||||||
# ``40 02`` (20 bytes) dominates, but ``40 01`` (18) and
|
|
||||||
# ``40 03`` (22) both occur in production files. Confirmed
|
|
||||||
# 2026-08-25; the constant ``02 00`` marker moves with NN too
|
|
||||||
# (see :func:`parse_segment_header`).
|
|
||||||
length = 2 * t1 + 16
|
|
||||||
elif is_tagless_segment_header(body, i):
|
|
||||||
# Segment header with no ``40 NN`` tag (NN=0 — the previous channel
|
|
||||||
# needed no continuation deltas). Emit it as a synthetic ``40 00``
|
|
||||||
# block whose ``data`` is the whole 14-byte record, so the nd=0
|
|
||||||
# offsets in :func:`decode_waveform_v2` line up unchanged.
|
|
||||||
blocks.append(WaveformBlock(
|
|
||||||
offset=i, tag_hi=0x40, tag_lo=0x00,
|
|
||||||
data=bytes(body[i : i + _TAGLESS_HEADER_LEN]),
|
|
||||||
length=_TAGLESS_HEADER_LEN,
|
|
||||||
))
|
|
||||||
i += _TAGLESS_HEADER_LEN
|
|
||||||
continue
|
|
||||||
else:
|
|
||||||
# Unknown tag; stop. Caller can inspect ``i`` to see where.
|
|
||||||
break
|
|
||||||
|
|
||||||
if i + length > len(body):
|
|
||||||
break
|
|
||||||
|
|
||||||
data = bytes(body[i + 2 : i + length])
|
|
||||||
blocks.append(WaveformBlock(offset=i, tag_hi=t0, tag_lo=t1, data=data, length=length))
|
|
||||||
i += length
|
|
||||||
|
|
||||||
return blocks
|
|
||||||
|
|
||||||
|
|
||||||
def split_segments(blocks: List[WaveformBlock]) -> List[List[WaveformBlock]]:
|
|
||||||
"""Group consecutive blocks into segments separated by ``40 02`` headers.
|
|
||||||
|
|
||||||
The first segment is whatever runs before the first ``40 02`` header
|
|
||||||
(typically the "segment 0" preamble data after the body preamble).
|
|
||||||
Subsequent segments start with a ``40 02`` block, then have their
|
|
||||||
own data blocks until the next ``40 02``.
|
|
||||||
"""
|
|
||||||
segments: List[List[WaveformBlock]] = []
|
|
||||||
current: List[WaveformBlock] = []
|
|
||||||
for b in blocks:
|
|
||||||
if b.tag_hi == 0x40:
|
|
||||||
if current:
|
|
||||||
segments.append(current)
|
|
||||||
current = [b]
|
|
||||||
else:
|
|
||||||
current.append(b)
|
|
||||||
if current:
|
|
||||||
segments.append(current)
|
|
||||||
return segments
|
|
||||||
|
|
||||||
|
|
||||||
def parse_segment_header(block: WaveformBlock) -> Optional[dict]:
|
|
||||||
"""Decode the payload of a ``40 NN`` segment header.
|
|
||||||
|
|
||||||
NN (the tag's low byte) is the number of int16 BE continuation deltas
|
|
||||||
the header carries for the PREVIOUS channel, so every field after
|
|
||||||
those deltas shifts by ``2 * NN``. The payload is ``2 * NN + 14``
|
|
||||||
bytes. ``40 02`` is the common case; ``40 01`` and ``40 03`` also
|
|
||||||
occur in production files (confirmed 2026-08-25).
|
|
||||||
|
|
||||||
Returns a dict with the labelled fields, or None if *block* is not a
|
|
||||||
segment header or is too short.
|
|
||||||
"""
|
|
||||||
if block.tag_hi != 0x40 or block.tag_lo > 0x08:
|
|
||||||
return None
|
|
||||||
nd = block.tag_lo
|
|
||||||
if len(block.data) < 2 * nd + 14:
|
|
||||||
return None
|
|
||||||
p = block.data
|
|
||||||
counter = int.from_bytes(p[2 * nd + 4 : 2 * nd + 8], "little", signed=False)
|
|
||||||
return {
|
|
||||||
"n_prev_deltas": nd,
|
|
||||||
# ``nd`` int16 BE deltas extending the previous channel.
|
|
||||||
"prev_deltas": [
|
|
||||||
int.from_bytes(p[2 * k : 2 * k + 2], "big", signed=True)
|
|
||||||
for k in range(nd)
|
|
||||||
],
|
|
||||||
"field2": p[2 * nd : 2 * nd + 4], # 4-byte field, role unconfirmed
|
|
||||||
"counter": counter, # legacy: raw uint32 LE of the id field
|
|
||||||
"channel": SEGMENT_CHANNEL_IDS.get(p[2 * nd + 4]),
|
|
||||||
"segment_index": p[2 * nd + 7],
|
|
||||||
"marker": p[2 * nd + 8 : 2 * nd + 10], # always b"\x02\x00"
|
|
||||||
"anchors": [
|
|
||||||
int.from_bytes(p[2 * nd + 10 : 2 * nd + 12], "big", signed=True),
|
|
||||||
int.from_bytes(p[2 * nd + 12 : 2 * nd + 14], "big", signed=True),
|
|
||||||
],
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _s4(n: int) -> int:
|
|
||||||
"""Sign-extend a 4-bit value to signed int (0..7 → 0..7; 8..F → -8..-1)."""
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def _i8(b: int) -> int:
|
|
||||||
"""Reinterpret an unsigned byte as signed int8."""
|
|
||||||
return b if b < 128 else b - 256
|
|
||||||
|
|
||||||
|
|
||||||
def decode_tran_initial(body: bytes) -> Optional[List[int]]:
|
|
||||||
"""
|
|
||||||
Decode the initial Tran-channel samples — VERIFIED 2026-05-11.
|
|
||||||
|
|
||||||
Returns Tran samples in **16-count units** (LSB = 0.005 in/s at Normal
|
|
||||||
range — the same quantization BW uses for its ASCII export). Returns
|
|
||||||
``None`` if the body cannot be parsed.
|
|
||||||
|
|
||||||
The decoded list extends from sample 0 through the end of segment 0
|
|
||||||
(= just before the first ``40 02`` segment header; ~510 sample-sets
|
|
||||||
for the events tested). Multi-segment decoding requires continuing
|
|
||||||
past the segment header — that's done by :func:`decode_tran_full`
|
|
||||||
when the per-segment rules are pinned down for all signal types.
|
|
||||||
|
|
||||||
Codec for segment 0 (CONFIRMED 2026-05-11 against 7 fixture events):
|
|
||||||
|
|
||||||
- Body bytes [0:3] are the magic ``00 02 00``.
|
|
||||||
- Body bytes [3:5] = ``Tran[0]`` as int16 BE in 16-count units.
|
|
||||||
- Body bytes [5:7] = ``Tran[1]`` as int16 BE in 16-count units.
|
|
||||||
- Data blocks (``10 NN`` or ``20 NN``) carry Tran deltas starting
|
|
||||||
at sample 2:
|
|
||||||
|
|
||||||
* ``10 NN``: NN nibbles = NN/2 bytes; each nibble is a 4-bit
|
|
||||||
signed delta (0..7 → 0..+7; 8..F → -8..-1). High nibble of
|
|
||||||
each byte comes first.
|
|
||||||
* ``20 NN``: NN int8 signed deltas (one delta per byte).
|
|
||||||
|
|
||||||
- ``00 NN`` blocks are run-length-encoded zero deltas: append NN
|
|
||||||
copies of the current cumulative Tran value (no change).
|
|
||||||
|
|
||||||
- ``30 NN`` blocks have not yet been decoded for content — they
|
|
||||||
appear in segment 0 of loud-from-start events (SS0, SV0) and
|
|
||||||
seem to signal a transition or special-case interpretation.
|
|
||||||
The walker steps over them but their data is ignored.
|
|
||||||
|
|
||||||
The walk stops at the first ``40 02`` segment header.
|
|
||||||
"""
|
|
||||||
if len(body) < 7 or body[0:3] != b"\x00\x02\x00":
|
|
||||||
return None
|
|
||||||
t0 = int.from_bytes(body[3:5], "big", signed=True)
|
|
||||||
t1 = int.from_bytes(body[5:7], "big", signed=True)
|
|
||||||
|
|
||||||
start = find_data_start(body)
|
|
||||||
if start < 0:
|
|
||||||
return [t0, t1]
|
|
||||||
|
|
||||||
out = [t0, t1]
|
|
||||||
cur = t1
|
|
||||||
for blk in walk_body(body, start):
|
|
||||||
if blk.tag_hi == 0x40:
|
|
||||||
# Segment boundary — stop. Multi-segment decode is decode_tran_full.
|
|
||||||
break
|
|
||||||
if blk.tag_hi == 0x10:
|
|
||||||
for byte in blk.data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
cur += _s4(nib)
|
|
||||||
out.append(cur)
|
|
||||||
elif blk.tag_hi == 0x20:
|
|
||||||
for byte in blk.data:
|
|
||||||
cur += _i8(byte)
|
|
||||||
out.append(cur)
|
|
||||||
elif blk.tag_hi == 0x00:
|
|
||||||
# RLE zero deltas: append NN copies of current Tran value.
|
|
||||||
for _ in range(blk.tag_lo):
|
|
||||||
out.append(cur)
|
|
||||||
# 30 NN: unknown content; skip.
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def decode_waveform_legacy(body: bytes) -> Optional[dict]:
|
|
||||||
"""
|
|
||||||
SUPERSEDED 2026-08-25 — the tag-dispatch / segment-header model.
|
|
||||||
|
|
||||||
Retained because ``micromate/idf_file.py`` trial-decodes Thor IDFW bodies
|
|
||||||
at many candidate offsets and keeps whichever yields the most samples;
|
|
||||||
the record-chain decoder returns None where this one returned garbage,
|
|
||||||
which shifts that heuristic's winner. Thor is pinned here until its own
|
|
||||||
body-offset search is reworked. Do not use for series-3.
|
|
||||||
|
|
||||||
Decode the body into per-channel sample arrays.
|
|
||||||
|
|
||||||
Status (2026-05-11 evening — channel-rotation hypothesis CONFIRMED):
|
|
||||||
segments rotate channels in fixed order **Tran → Vert → Long → MicL**.
|
|
||||||
Each channel-segment carries a 2-sample anchor pair in segment-header
|
|
||||||
bytes [14:18] (or in the body preamble for the initial Tran segment)
|
|
||||||
plus a stream of delta blocks for samples 2 onward.
|
|
||||||
|
|
||||||
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}``
|
|
||||||
with each channel's decoded samples in 16-count units (LSB = 0.005
|
|
||||||
in/s at Normal range). Returns ``None`` if the body cannot be
|
|
||||||
parsed.
|
|
||||||
"""
|
|
||||||
if len(body) < 7 or body[0:3] != b"\x00\x02\x00":
|
|
||||||
return None
|
|
||||||
|
|
||||||
channels = ["Tran", "Vert", "Long", "MicL"]
|
|
||||||
out: dict = {ch: [] for ch in channels}
|
|
||||||
|
|
||||||
# Initial Tran segment: preamble anchor pair + delta blocks before first 40 02.
|
|
||||||
t0 = int.from_bytes(body[3:5], "big", signed=True)
|
|
||||||
t1 = int.from_bytes(body[5:7], "big", signed=True)
|
|
||||||
out["Tran"].extend([t0, t1])
|
|
||||||
|
|
||||||
start = find_data_start(body)
|
|
||||||
if start < 0:
|
|
||||||
return out
|
|
||||||
|
|
||||||
blocks = walk_body(body, start)
|
|
||||||
seg_idx = [i for i, b in enumerate(blocks) if b.tag_hi == 0x40]
|
|
||||||
|
|
||||||
def apply_blocks(channel: str, anchor: int,
|
|
||||||
block_start: int, block_end: int) -> int:
|
|
||||||
"""Apply delta blocks [block_start, block_end) to *channel*'s sample
|
|
||||||
list, starting from *anchor*. Returns the final cumulative value."""
|
|
||||||
cur = anchor
|
|
||||||
for bi in range(block_start, block_end):
|
|
||||||
blk = blocks[bi]
|
|
||||||
if (blk.tag_hi & 0xF0) == 0x10:
|
|
||||||
# Both ``10 NN`` (NN ≤ 0xFC) and wide-NN ``1X NN`` (X != 0)
|
|
||||||
# are nibble-delta streams. The walker has already used the
|
|
||||||
# right length; here we just iterate the payload bytes.
|
|
||||||
for byte in blk.data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
cur += _s4(nib)
|
|
||||||
out[channel].append(cur)
|
|
||||||
elif (blk.tag_hi & 0xF0) == 0x20:
|
|
||||||
# ``20 NN`` and wide ``2X NN`` both carry int8 deltas.
|
|
||||||
for byte in blk.data:
|
|
||||||
cur += _i8(byte)
|
|
||||||
out[channel].append(cur)
|
|
||||||
elif (blk.tag_hi & 0xF0) == 0x00:
|
|
||||||
# RLE zero-delta run. Wide form ``0X NN`` carries the high
|
|
||||||
# nibble of a 12-bit NN in the tag byte, same as ``1X``/``2X``.
|
|
||||||
run = ((blk.tag_hi & 0x0F) << 8) | blk.tag_lo
|
|
||||||
for _ in range(run):
|
|
||||||
out[channel].append(cur)
|
|
||||||
elif blk.tag_hi == 0x30:
|
|
||||||
# 12-bit signed deltas, packed as NN/4 groups of 6 bytes each:
|
|
||||||
# bytes [0:2] = 16 bits = 4 × 4-bit high nibbles (MSB first)
|
|
||||||
# bytes [2:6] = 4 × int8 low bytes
|
|
||||||
# Each delta = sign_extend_12((high_nibble << 8) | low_byte).
|
|
||||||
# Confirmed 2026-05-11 against all 14 ``30 NN`` blocks in the
|
|
||||||
# bundled fixtures.
|
|
||||||
n_groups = blk.tag_lo // 4
|
|
||||||
for g in range(n_groups):
|
|
||||||
grp = blk.data[g * 6 : (g + 1) * 6]
|
|
||||||
if len(grp) < 6:
|
|
||||||
break
|
|
||||||
high_word = (grp[0] << 8) | grp[1]
|
|
||||||
for k in range(4):
|
|
||||||
nib = (high_word >> (12 - 4 * k)) & 0xF
|
|
||||||
v = (nib << 8) | grp[2 + k]
|
|
||||||
if v >= 0x800:
|
|
||||||
v -= 0x1000
|
|
||||||
cur += v
|
|
||||||
out[channel].append(cur)
|
|
||||||
# 40 02: should not occur in segment data.
|
|
||||||
return cur
|
|
||||||
|
|
||||||
# Initial Tran segment: deltas from start of body up to first 40 02 (or end).
|
|
||||||
first_seg = seg_idx[0] if seg_idx else len(blocks)
|
|
||||||
last_tran_value = apply_blocks("Tran", t1, 0, first_seg)
|
|
||||||
|
|
||||||
# Subsequent segments rotate channels. Each segment header carries:
|
|
||||||
# bytes [0:2] and [2:4] = 2 deltas extending the PREVIOUS channel
|
|
||||||
# bytes [14:16] and [16:18] = anchor pair for THIS segment's channel
|
|
||||||
#
|
|
||||||
# Rotation: V, L, M, T, V, L, M, T, ... (initial Tran segment is the
|
|
||||||
# implicit T in the cycle.)
|
|
||||||
rotation = ["Vert", "Long", "MicL", "Tran"]
|
|
||||||
# Track each channel's "running cumulative value" so we can apply the
|
|
||||||
# previous-channel extension deltas at every segment boundary.
|
|
||||||
last_value = {"Tran": last_tran_value, "Vert": None, "Long": None, "MicL": None}
|
|
||||||
|
|
||||||
prev_channel = "Tran"
|
|
||||||
for k, hi in enumerate(seg_idx):
|
|
||||||
header = blocks[hi]
|
|
||||||
# Channel comes from the header's own id byte, which is authoritative.
|
|
||||||
# The old rotation-by-position fallback is kept for headers whose id
|
|
||||||
# byte isn't one of the four known values — but a single missed or
|
|
||||||
# extra header would desync rotation and corrupt every later channel,
|
|
||||||
# which is exactly what tagless headers used to cause.
|
|
||||||
_nd = header.tag_lo
|
|
||||||
channel = None
|
|
||||||
if len(header.data) >= 2 * _nd + 8:
|
|
||||||
channel = SEGMENT_CHANNEL_IDS.get(header.data[2 * _nd + 4])
|
|
||||||
if channel is None:
|
|
||||||
channel = rotation[k % 4]
|
|
||||||
# ``40 NN``: NN int16 BE continuation deltas for the previous channel
|
|
||||||
# come first, so every later field shifts by 2*NN. NN is usually 2
|
|
||||||
# but 1 and 3 both occur (confirmed 2026-08-25).
|
|
||||||
nd = header.tag_lo
|
|
||||||
if len(header.data) < 2 * nd + 14:
|
|
||||||
continue
|
|
||||||
# Validate: real segment headers have the constant `02 00` marker
|
|
||||||
# right after the counter. Trailer/footer "40 NN" markers contain
|
|
||||||
# ASCII serial bytes or other non-header data there and would
|
|
||||||
# otherwise be mis-read as segment headers, adding spurious tail
|
|
||||||
# samples.
|
|
||||||
if header.data[2 * nd + 8 : 2 * nd + 10] != b"\x02\x00":
|
|
||||||
break
|
|
||||||
# Extend the PREVIOUS channel by NN more samples.
|
|
||||||
if last_value[prev_channel] is not None:
|
|
||||||
v = last_value[prev_channel]
|
|
||||||
for d in range(nd): # NB: not `k` — that's the segment index
|
|
||||||
v += int.from_bytes(
|
|
||||||
header.data[2 * d : 2 * d + 2], "big", signed=True
|
|
||||||
)
|
|
||||||
out[prev_channel].append(v)
|
|
||||||
last_value[prev_channel] = v
|
|
||||||
# Anchor pair for THIS segment's channel.
|
|
||||||
c0 = int.from_bytes(
|
|
||||||
header.data[2 * nd + 10 : 2 * nd + 12], "big", signed=True
|
|
||||||
)
|
|
||||||
c1 = int.from_bytes(
|
|
||||||
header.data[2 * nd + 12 : 2 * nd + 14], "big", signed=True
|
|
||||||
)
|
|
||||||
out[channel].extend([c0, c1])
|
|
||||||
# Apply delta blocks for this segment.
|
|
||||||
next_hi = seg_idx[k + 1] if k + 1 < len(seg_idx) else len(blocks)
|
|
||||||
last_value[channel] = apply_blocks(channel, c1, hi + 1, next_hi)
|
|
||||||
prev_channel = channel
|
|
||||||
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
# ── ADC-scale conversion helpers ────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
# Scaling factor: decode_waveform_v2 produces geo-channel samples in the BW
|
|
||||||
# display quantization (16-count units, LSB = 0.005 in/s at Normal range).
|
|
||||||
# The legacy consumer pipeline (sfm/event_hdf5.py) expects raw_samples in
|
|
||||||
# 1-count ADC units (× full_scale / 32768 → physical). To plug the new
|
|
||||||
# decoder in without rewriting consumers, multiply geo values by 16.
|
|
||||||
#
|
|
||||||
# Mic samples are already in raw ADC counts (decoded value 1 = 1 mic ADC count
|
|
||||||
# = -81.94 dB on the BW display). Mic values pass through unchanged.
|
|
||||||
_GEO_DECODER_TO_ADC = 16
|
|
||||||
|
|
||||||
|
|
||||||
def decoded_to_adc_counts(decoded: dict) -> dict:
|
|
||||||
"""Convert :func:`decode_waveform_v2` output to int16 ADC counts.
|
|
||||||
|
|
||||||
Geo channels are scaled by ×16 (decoder produces 16-count units,
|
|
||||||
consumer expects 1-count ADC). Mic is passed through as raw counts.
|
|
||||||
"""
|
|
||||||
if not decoded:
|
|
||||||
return {}
|
|
||||||
return {
|
|
||||||
"Tran": [v * _GEO_DECODER_TO_ADC for v in decoded.get("Tran", [])],
|
|
||||||
"Vert": [v * _GEO_DECODER_TO_ADC for v in decoded.get("Vert", [])],
|
|
||||||
"Long": [v * _GEO_DECODER_TO_ADC for v in decoded.get("Long", [])],
|
|
||||||
"MicL": list(decoded.get("MicL", [])),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def mic_count_to_db(count: int) -> float:
|
|
||||||
"""Convert a MicL ADC count to dB(L) for BW-display-compatible output.
|
|
||||||
|
|
||||||
Empirical formula (confirmed 2026-05-11 against V70 fixture: count=813
|
|
||||||
→ 140.1 dB; count=±1 → ±81.94 dB; count=±24 → ±109.5 dB):
|
|
||||||
|
|
||||||
dB = sign(count) × (81.94 + 20 × log10(|count|)) for |count| ≥ 1
|
|
||||||
dB = 0.0 for count == 0
|
|
||||||
|
|
||||||
The constant 81.94 corresponds to 10^(81.94/20) ≈ 12490 mic ADC counts
|
|
||||||
being the dB(L) reference level — almost certainly a calibration
|
|
||||||
constant from the device's mic.
|
|
||||||
"""
|
|
||||||
if count == 0:
|
|
||||||
return 0.0
|
|
||||||
sign = 1.0 if count > 0 else -1.0
|
|
||||||
return sign * (81.94 + 20.0 * math.log10(abs(count)))
|
|
||||||
|
|
||||||
|
|
||||||
# ── A5-frame entry point ────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def decode_a5_frames(a5_frames) -> Optional[dict]:
|
|
||||||
"""Decode a list of A5 (BULK_WAVEFORM_STREAM) frames into per-channel
|
|
||||||
int16 ADC samples.
|
|
||||||
|
|
||||||
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}``
|
|
||||||
with each channel's samples in **1-count ADC units** (the legacy
|
|
||||||
``event.raw_samples`` convention — multiply by ``full_scale / 32768``
|
|
||||||
to convert to physical units; for mic, use :func:`mic_count_to_db` or
|
|
||||||
a per-count psi factor).
|
|
||||||
|
|
||||||
Returns ``None`` if the frames cannot be parsed.
|
|
||||||
|
|
||||||
This is the wired-up production entry point. It:
|
|
||||||
1. Reconstructs the BW-binary body bytes from the A5 frames
|
|
||||||
(``blastware_file.extract_body_bytes``).
|
|
||||||
2. Runs the verified codec (``decode_waveform_v2``) on the body.
|
|
||||||
3. Converts to int16 ADC counts via :func:`decoded_to_adc_counts`.
|
|
||||||
"""
|
|
||||||
# Local import to avoid a cycle: blastware_file imports models and
|
|
||||||
# ultimately client.py imports waveform_codec.
|
|
||||||
from .blastware_file import extract_body_bytes
|
|
||||||
|
|
||||||
if not a5_frames:
|
|
||||||
return None
|
|
||||||
_strt, body, _footer = extract_body_bytes(a5_frames)
|
|
||||||
if not body:
|
|
||||||
return None
|
|
||||||
decoded = decode_waveform_v2(body)
|
|
||||||
if decoded is None:
|
|
||||||
return None
|
|
||||||
return decoded_to_adc_counts(decoded)
|
|
||||||
|
|
||||||
|
|
||||||
# ── Record-chain body model (CONFIRMED 2026-08-25) ──────────────────────────
|
|
||||||
#
|
|
||||||
# The body is NOT a flat tag-dispatch stream with ``40 NN`` segment headers.
|
|
||||||
# It is a chain of self-delimiting per-channel RECORDS:
|
|
||||||
#
|
|
||||||
# off+0 field2 uint16 purpose unknown (not a length, not a checksum)
|
|
||||||
# off+2 len uint16 BE next_record = off + 2 + len <- authoritative
|
|
||||||
# off+4 chan_id 0x46 Tran / 0x47 Vert / 0x48 Long / 0x49 MicL
|
|
||||||
# 0x06 = end of waveform stream
|
|
||||||
# off+5 0x00
|
|
||||||
# off+6 0x00
|
|
||||||
# off+7 segment index
|
|
||||||
# off+8 mode 2 bytes, a 3-valued enum (see below)
|
|
||||||
# off+10 anchors 2 x int16 BE, ABSOLUTE — present only when mode is 02 00
|
|
||||||
#
|
|
||||||
# Mode semantics, all ground-truth verified:
|
|
||||||
# 02 00 14-byte header; emit the 2 anchors, then blocks are CUMULATIVE deltas
|
|
||||||
# 01 00 10-byte header; no anchors; blocks carry ABSOLUTE sample values
|
|
||||||
# 00 03 10-byte header; NO TAGS AT ALL — the data section is raw 12-bit
|
|
||||||
# packed ABSOLUTE samples (6 bytes -> 4 samples)
|
|
||||||
#
|
|
||||||
# ``40 NN`` is an ordinary int16 BE DATA block (length 2*NN + 2), never a header.
|
|
||||||
# The previous model read it as a variable-width segment header of length
|
|
||||||
# 2*NN + 16, which is why walks drifted and channels came out unequal.
|
|
||||||
#
|
|
||||||
# Verified over the 1,388 series-3 waveform binaries in the production
|
|
||||||
# snapshot: the length chain terminates on a 0x06 record in 1,387 of them (the
|
|
||||||
# exception has an ambiguous footer, handled by the caller), and all four
|
|
||||||
# channels come out at identical length in 1,388/1,388 — against 156/1,388
|
|
||||||
# under the superseded model. Against the 75 events with a preserved
|
|
||||||
# Blastware ASCII export: sample-count exact 72/75 -> 75/75, fully exact
|
|
||||||
# 70/75 -> 73/75.
|
|
||||||
|
|
||||||
CHANNEL_IDS = {0x46: "Tran", 0x47: "Vert", 0x48: "Long", 0x49: "MicL"}
|
|
||||||
STREAM_END_ID = 0x06
|
|
||||||
|
|
||||||
MODE_DELTA = (0x02, 0x00)
|
|
||||||
MODE_ABSOLUTE = (0x01, 0x00)
|
|
||||||
MODE_RAW12 = (0x00, 0x03)
|
|
||||||
_MODES = (MODE_DELTA, MODE_ABSOLUTE, MODE_RAW12)
|
|
||||||
|
|
||||||
|
|
||||||
def _u16(b: bytes, p: int) -> int:
|
|
||||||
return (b[p] << 8) | b[p + 1]
|
|
||||||
|
|
||||||
|
|
||||||
def _i16(b: bytes, p: int) -> int:
|
|
||||||
v = _u16(b, p)
|
|
||||||
return v - 0x10000 if v >= 0x8000 else v
|
|
||||||
|
|
||||||
|
|
||||||
def data_block_len(body: bytes, p: int) -> Tuple[Optional[int], Optional[int]]:
|
|
||||||
"""``(byte_length, n_samples)`` of the data block at *p*, or ``(None, None)``.
|
|
||||||
|
|
||||||
Data-section blocks only — there is no segment-header tag in this model.
|
|
||||||
``30 NN`` has no trailer-length fallback here; that fallback corrupted
|
|
||||||
records whose ``30 NN`` sat near a record boundary.
|
|
||||||
"""
|
|
||||||
if p + 2 > len(body):
|
|
||||||
return None, None
|
|
||||||
t0, t1 = body[p], body[p + 1]
|
|
||||||
hi = t0 & 0xF0
|
|
||||||
nn = ((t0 & 0x0F) << 8) | t1
|
|
||||||
if hi == 0x40: # int16 BE data block
|
|
||||||
return (None, None) if (nn == 0 or nn > 0x08) else (2 * nn + 2, nn)
|
|
||||||
if nn == 0 or nn % 4:
|
|
||||||
return None, None
|
|
||||||
if hi == 0x00:
|
|
||||||
return 2, nn # RLE hold
|
|
||||||
if hi == 0x10:
|
|
||||||
return nn // 2 + 2, nn # 4-bit nibble
|
|
||||||
if hi == 0x20:
|
|
||||||
return nn + 2, nn # int8
|
|
||||||
if hi == 0x30:
|
|
||||||
return nn * 3 // 2 + 2, nn # 12-bit packed
|
|
||||||
return None, None
|
|
||||||
|
|
||||||
|
|
||||||
def unpack12(data: bytes) -> List[int]:
|
|
||||||
"""Raw 12-bit packed samples: 6 bytes -> 4 signed values."""
|
|
||||||
out: List[int] = []
|
|
||||||
for g in range(len(data) // 6):
|
|
||||||
hi = (data[6 * g] << 8) | data[6 * g + 1]
|
|
||||||
for k in range(4):
|
|
||||||
x = (((hi >> (12 - 4 * k)) & 0xF) << 8) | data[6 * g + 2 + k]
|
|
||||||
out.append(x - 0x1000 if x >= 0x800 else x)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def is_record(body: bytes, p: int) -> bool:
|
|
||||||
"""True if a per-channel record header starts at *p*."""
|
|
||||||
return (p + 10 <= len(body)
|
|
||||||
and body[p + 4] in CHANNEL_IDS
|
|
||||||
and body[p + 5] == 0x00 and body[p + 6] == 0x00
|
|
||||||
and 8 <= _u16(body, p + 2) <= len(body) - p
|
|
||||||
and (body[p + 8], body[p + 9]) in _MODES)
|
|
||||||
|
|
||||||
|
|
||||||
def find_first_record(body: bytes) -> Optional[int]:
|
|
||||||
"""Offset of the first record, or None.
|
|
||||||
|
|
||||||
Under the normal ``00 02 00`` preamble the leading bytes are segment-0's
|
|
||||||
Tran blocks, so walk them. Under the ``00 00 03`` preamble that data is
|
|
||||||
raw 12-bit with no tags at all and cannot be block-walked — scan instead.
|
|
||||||
"""
|
|
||||||
if len(body) >= 3 and (body[1], body[2]) == MODE_RAW12:
|
|
||||||
scan_from = 3
|
|
||||||
else:
|
|
||||||
i = 7
|
|
||||||
while i < len(body):
|
|
||||||
if is_record(body, i):
|
|
||||||
nxt = i + 2 + _u16(body, i + 2)
|
|
||||||
if nxt + 5 <= len(body) and (is_record(body, nxt)
|
|
||||||
or body[nxt + 4] == STREAM_END_ID):
|
|
||||||
return i
|
|
||||||
length, _ = data_block_len(body, i)
|
|
||||||
if length is None:
|
|
||||||
return None
|
|
||||||
i += length
|
|
||||||
return None
|
|
||||||
for i in range(scan_from, max(scan_from, len(body) - 10)):
|
|
||||||
if is_record(body, i):
|
|
||||||
nxt = i + 2 + _u16(body, i + 2)
|
|
||||||
if nxt + 5 <= len(body) and (is_record(body, nxt)
|
|
||||||
or body[nxt + 4] == STREAM_END_ID):
|
|
||||||
return i
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def walk_records(body: bytes, first: Optional[int] = None) -> List[dict]:
|
|
||||||
"""Follow the length chain from *first* to the ``0x06`` terminator."""
|
|
||||||
if first is None:
|
|
||||||
first = find_first_record(body)
|
|
||||||
out: List[dict] = []
|
|
||||||
if first is None:
|
|
||||||
return out
|
|
||||||
p, seen = first, set()
|
|
||||||
while p is not None and p + 10 <= len(body):
|
|
||||||
if p in seen:
|
|
||||||
break
|
|
||||||
seen.add(p)
|
|
||||||
cid = body[p + 4]
|
|
||||||
if cid == STREAM_END_ID or cid not in CHANNEL_IDS:
|
|
||||||
break
|
|
||||||
length = _u16(body, p + 2)
|
|
||||||
if length < 8 or p + 2 + length > len(body):
|
|
||||||
break
|
|
||||||
out.append({"offset": p, "channel": CHANNEL_IDS[cid],
|
|
||||||
"segment_index": body[p + 7],
|
|
||||||
"mode": (body[p + 8], body[p + 9]),
|
|
||||||
"end": p + 2 + length})
|
|
||||||
p += 2 + length
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def decode_waveform_v2(body: bytes) -> Optional[dict]:
|
|
||||||
"""Decode a Blastware waveform body into per-channel sample arrays.
|
|
||||||
|
|
||||||
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}``
|
|
||||||
in 16-count units (LSB = 0.005 in/s at Normal range), or None if *body*
|
|
||||||
is not a decodable waveform body.
|
|
||||||
|
|
||||||
Implements the record-chain model documented above.
|
|
||||||
"""
|
|
||||||
if len(body) < 8 or body[0] != 0x00:
|
|
||||||
return None
|
|
||||||
preamble = (body[1], body[2])
|
|
||||||
if preamble not in (MODE_DELTA, MODE_RAW12):
|
|
||||||
return None
|
|
||||||
first = find_first_record(body)
|
|
||||||
if first is None:
|
|
||||||
return None
|
|
||||||
|
|
||||||
out: dict = {c: [] for c in ("Tran", "Vert", "Long", "MicL")}
|
|
||||||
|
|
||||||
def run(channel: str, start: int, end: int, absolute: bool) -> None:
|
|
||||||
cur = out[channel][-1] if out[channel] else 0
|
|
||||||
i = start
|
|
||||||
while i < end:
|
|
||||||
length, nn = data_block_len(body, i)
|
|
||||||
if length is None or i + length > end:
|
|
||||||
return # stop this record; the chain resyncs at end
|
|
||||||
hi = body[i] & 0xF0
|
|
||||||
if hi == 0x00:
|
|
||||||
vals = [None] * nn
|
|
||||||
elif hi == 0x10:
|
|
||||||
vals = []
|
|
||||||
for k in range(nn):
|
|
||||||
byte = body[i + 2 + k // 2]
|
|
||||||
v = (byte >> 4) if k % 2 == 0 else (byte & 0xF)
|
|
||||||
vals.append(v - 16 if v >= 8 else v)
|
|
||||||
elif hi == 0x20:
|
|
||||||
vals = [v - 256 if v >= 128 else v
|
|
||||||
for v in body[i + 2:i + 2 + nn]]
|
|
||||||
elif hi == 0x30:
|
|
||||||
vals = unpack12(body[i + 2:i + length])
|
|
||||||
else:
|
|
||||||
vals = [_i16(body, i + 2 + 2 * k) for k in range(nn)]
|
|
||||||
for v in vals:
|
|
||||||
if v is None:
|
|
||||||
pass # RLE hold, in delta AND absolute modes
|
|
||||||
elif absolute:
|
|
||||||
cur = v
|
|
||||||
else:
|
|
||||||
cur += v
|
|
||||||
out[channel].append(cur)
|
|
||||||
i += length
|
|
||||||
|
|
||||||
# Segment 0 is an implicit Tran record carried in the preamble.
|
|
||||||
if preamble == MODE_DELTA:
|
|
||||||
out["Tran"].extend([_i16(body, 3), _i16(body, 5)])
|
|
||||||
run("Tran", 7, first, absolute=False)
|
|
||||||
else:
|
|
||||||
out["Tran"].extend(unpack12(body[3:first]))
|
|
||||||
|
|
||||||
for rec in walk_records(body, first):
|
|
||||||
ch, off, mode, end = (rec["channel"], rec["offset"],
|
|
||||||
rec["mode"], rec["end"])
|
|
||||||
if mode == MODE_DELTA:
|
|
||||||
out[ch].extend([_i16(body, off + 10), _i16(body, off + 12)])
|
|
||||||
run(ch, off + 14, end, absolute=False)
|
|
||||||
elif mode == MODE_ABSOLUTE:
|
|
||||||
run(ch, off + 10, end, absolute=True)
|
|
||||||
elif mode == MODE_RAW12:
|
|
||||||
out[ch].extend(unpack12(body[off + 10:end]))
|
|
||||||
return out
|
|
||||||
@@ -53,9 +53,7 @@ SUB_TABLE: dict[int, tuple[str, str, str]] = {
|
|||||||
0x82: ("TRIGGER_CONFIG_WRITE", "BW→S3", "0x1C bytes; trigger config block; mirrors SUB 1C"),
|
0x82: ("TRIGGER_CONFIG_WRITE", "BW→S3", "0x1C bytes; trigger config block; mirrors SUB 1C"),
|
||||||
0x83: ("TRIGGER_WRITE_CONFIRM", "BW→S3", "Short frame; commit step after 0x82"),
|
0x83: ("TRIGGER_WRITE_CONFIRM", "BW→S3", "Short frame; commit step after 0x82"),
|
||||||
# S3→BW responses
|
# S3→BW responses
|
||||||
0x5A: ("BULK_WAVEFORM_STREAM", "BW→S3", "Bulk waveform chunk request; response is A5 stream"),
|
|
||||||
0xA4: ("POLL_RESPONSE", "S3→BW", "Response to SUB 5B poll"),
|
0xA4: ("POLL_RESPONSE", "S3→BW", "Response to SUB 5B poll"),
|
||||||
0xA5: ("BULK_WAVEFORM_RESPONSE", "S3→BW", "Response to SUB 5A; waveform chunks + metadata"),
|
|
||||||
0xFE: ("FULL_CONFIG_RESPONSE", "S3→BW", "Response to SUB 01"),
|
0xFE: ("FULL_CONFIG_RESPONSE", "S3→BW", "Response to SUB 01"),
|
||||||
0xF9: ("CHANNEL_CONFIG_RESPONSE", "S3→BW", "Response to SUB 06"),
|
0xF9: ("CHANNEL_CONFIG_RESPONSE", "S3→BW", "Response to SUB 06"),
|
||||||
0xF7: ("EVENT_INDEX_RESPONSE", "S3→BW", "Response to SUB 08; contains backlight/power-save"),
|
0xF7: ("EVENT_INDEX_RESPONSE", "S3→BW", "Response to SUB 08; contains backlight/power-save"),
|
||||||
|
|||||||
+34
-31
@@ -33,7 +33,7 @@ STX = 0x02
|
|||||||
ETX = 0x03
|
ETX = 0x03
|
||||||
ACK = 0x41
|
ACK = 0x41
|
||||||
|
|
||||||
__version__ = "0.2.5"
|
__version__ = "0.2.3"
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -186,7 +186,7 @@ def parse_s3(blob: bytes, trailer_len: int) -> List[Frame]:
|
|||||||
|
|
||||||
IDLE = 0
|
IDLE = 0
|
||||||
IN_FRAME = 1
|
IN_FRAME = 1
|
||||||
IN_FRAME_DLE = 2 # saw DLE inside frame — waiting for next byte
|
AFTER_DLE = 2
|
||||||
|
|
||||||
state = IDLE
|
state = IDLE
|
||||||
body = bytearray()
|
body = bytearray()
|
||||||
@@ -206,63 +206,66 @@ def parse_s3(blob: bytes, trailer_len: int) -> List[Frame]:
|
|||||||
state = IN_FRAME
|
state = IN_FRAME
|
||||||
i += 2
|
i += 2
|
||||||
continue
|
continue
|
||||||
# ACK bytes, boot strings, garbage — silently ignored
|
|
||||||
|
|
||||||
elif state == IN_FRAME:
|
elif state == IN_FRAME:
|
||||||
if b == DLE:
|
if b == DLE:
|
||||||
state = IN_FRAME_DLE
|
state = AFTER_DLE
|
||||||
i += 1
|
i += 1
|
||||||
continue
|
continue
|
||||||
|
body.append(b)
|
||||||
|
|
||||||
|
else: # AFTER_DLE
|
||||||
|
if b == DLE:
|
||||||
|
body.append(DLE)
|
||||||
|
state = IN_FRAME
|
||||||
|
i += 1
|
||||||
|
continue
|
||||||
|
|
||||||
if b == ETX:
|
if b == ETX:
|
||||||
# Bare ETX = real S3 frame terminator (confirmed from S3FrameParser)
|
|
||||||
end_offset = i + 1
|
end_offset = i + 1
|
||||||
trailer_start = i + 1
|
trailer_start = i + 1
|
||||||
trailer_end = trailer_start + trailer_len
|
trailer_end = trailer_start + trailer_len
|
||||||
trailer = blob[trailer_start:trailer_end]
|
trailer = blob[trailer_start:trailer_end]
|
||||||
|
|
||||||
# S3 checksums are deliberately not validated here.
|
chk_valid = None
|
||||||
# Large S3 responses (A5 bulk waveform, E5 compliance) embed
|
chk_type = None
|
||||||
# inner DLE+ETX sub-frame terminators whose trailing 0x03 byte
|
chk_hex = None
|
||||||
# lands where the parser would expect the SUM8 checksum, causing
|
payload = bytes(body)
|
||||||
# false failures. The live protocol (protocol.py _validate_frame)
|
|
||||||
# also skips S3 checksum enforcement for the same reason.
|
if len(body) >= 1:
|
||||||
|
received_chk = body[-1]
|
||||||
|
computed_chk = checksum8_sum(bytes(body[:-1]))
|
||||||
|
if computed_chk == received_chk:
|
||||||
|
chk_valid = True
|
||||||
|
chk_type = "SUM8"
|
||||||
|
chk_hex = f"{received_chk:02x}"
|
||||||
|
payload = bytes(body[:-1])
|
||||||
|
else:
|
||||||
|
chk_valid = False
|
||||||
|
|
||||||
frames.append(Frame(
|
frames.append(Frame(
|
||||||
index=idx,
|
index=idx,
|
||||||
start_offset=start_offset,
|
start_offset=start_offset,
|
||||||
end_offset=end_offset,
|
end_offset=end_offset,
|
||||||
payload_raw=bytes(body),
|
payload_raw=bytes(body),
|
||||||
payload=bytes(body),
|
payload=payload,
|
||||||
trailer=trailer,
|
trailer=trailer,
|
||||||
checksum_valid=None,
|
checksum_valid=chk_valid,
|
||||||
checksum_type=None,
|
checksum_type=chk_type,
|
||||||
checksum_hex=None
|
checksum_hex=chk_hex
|
||||||
))
|
))
|
||||||
|
|
||||||
idx += 1
|
idx += 1
|
||||||
state = IDLE
|
state = IDLE
|
||||||
i = trailer_end
|
i = trailer_end
|
||||||
continue
|
continue
|
||||||
body.append(b)
|
|
||||||
|
|
||||||
else: # IN_FRAME_DLE
|
|
||||||
if b == DLE:
|
|
||||||
# DLE DLE → literal 0x10 in payload
|
|
||||||
body.append(DLE)
|
|
||||||
state = IN_FRAME
|
|
||||||
i += 1
|
|
||||||
continue
|
|
||||||
if b == ETX:
|
|
||||||
# DLE+ETX inside a frame = inner-frame terminator (A4/E5 sub-frames).
|
|
||||||
# Treat as literal data, NOT the outer frame end.
|
|
||||||
body.append(DLE)
|
|
||||||
body.append(ETX)
|
|
||||||
state = IN_FRAME
|
|
||||||
i += 1
|
|
||||||
continue
|
|
||||||
# Unexpected DLE + byte → treat as literal data
|
# Unexpected DLE + byte → treat as literal data
|
||||||
body.append(DLE)
|
body.append(DLE)
|
||||||
body.append(b)
|
body.append(b)
|
||||||
state = IN_FRAME
|
state = IN_FRAME
|
||||||
|
i += 1
|
||||||
|
continue
|
||||||
|
|
||||||
i += 1
|
i += 1
|
||||||
|
|
||||||
|
|||||||
+3
-7
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|||||||
|
|
||||||
[project]
|
[project]
|
||||||
name = "seismo-relay"
|
name = "seismo-relay"
|
||||||
version = "0.26.0"
|
version = "0.12.0"
|
||||||
description = "Python client and REST server for MiniMate Plus seismographs"
|
description = "Python client and REST server for MiniMate Plus seismographs"
|
||||||
requires-python = ">=3.10"
|
requires-python = ">=3.10"
|
||||||
dependencies = [
|
dependencies = [
|
||||||
@@ -12,13 +12,9 @@ dependencies = [
|
|||||||
"uvicorn[standard]>=0.24",
|
"uvicorn[standard]>=0.24",
|
||||||
"pyserial>=3.5",
|
"pyserial>=3.5",
|
||||||
"sqlalchemy>=2.0",
|
"sqlalchemy>=2.0",
|
||||||
"python-multipart>=0.0.7",
|
|
||||||
"h5py>=3.10",
|
|
||||||
"numpy>=1.24",
|
|
||||||
"matplotlib>=3.8",
|
|
||||||
]
|
]
|
||||||
|
|
||||||
[tool.setuptools.packages.find]
|
[tool.setuptools.packages.find]
|
||||||
# Auto-discovers minimateplus/, micromate/, sfm/, bridges/ as packages
|
# Auto-discovers minimateplus/, sfm/, bridges/ as packages
|
||||||
where = ["."]
|
where = ["."]
|
||||||
include = ["minimateplus*", "micromate*", "sfm*", "bridges*"]
|
include = ["minimateplus*", "sfm*", "bridges*"]
|
||||||
|
|||||||
@@ -2,7 +2,3 @@ fastapi
|
|||||||
uvicorn
|
uvicorn
|
||||||
sqlalchemy
|
sqlalchemy
|
||||||
pyserial
|
pyserial
|
||||||
python-multipart
|
|
||||||
h5py
|
|
||||||
numpy
|
|
||||||
matplotlib
|
|
||||||
|
|||||||
@@ -1,360 +0,0 @@
|
|||||||
"""
|
|
||||||
scratch/next_experiment_skeleton.py — segment-channel scoring analyzer.
|
|
||||||
|
|
||||||
This is the suggested NEXT EXPERIMENT for cracking the waveform body codec.
|
|
||||||
The goal is to figure out what segments 1+ contain, since segment 0 = Tran
|
|
||||||
is solved but multi-segment continuation diverges from truth at sample ~512.
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
The hypothesis to test
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
Segments rotate through channels:
|
|
||||||
|
|
||||||
segment 0 → Tran samples 0..509
|
|
||||||
segment 1 → Vert samples 0..507
|
|
||||||
segment 2 → Long samples 0..507
|
|
||||||
segment 3 → Mic samples 0..507
|
|
||||||
segment 4 → Tran samples 510..N (continuation)
|
|
||||||
...
|
|
||||||
|
|
||||||
This would explain why segment 0 works perfectly (it's pure Tran) and why
|
|
||||||
applying segment 1's blocks as Tran continuation gives wrong values
|
|
||||||
(it's actually Vert).
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
What the analyzer should do
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
For each segment in each fixture event:
|
|
||||||
|
|
||||||
1. Run the segment-0 block-walker + RLE decode (the same algorithm that
|
|
||||||
``decode_tran_initial`` uses) over the segment's blocks. Start from
|
|
||||||
some anchor value and produce a cumulative trajectory of length =
|
|
||||||
number-of-deltas-in-segment.
|
|
||||||
|
|
||||||
2. For each candidate channel C ∈ {Tran, Vert, Long, MicL}:
|
|
||||||
For each candidate anchor location in the segment-header payload
|
|
||||||
(try [0:2], [2:4], [4:6], [14:16], [16:18] as int16 BE):
|
|
||||||
Compare the decoded trajectory against truth[C] starting from
|
|
||||||
the segment's first sample index.
|
|
||||||
Score = number of matches (or sum of squared errors).
|
|
||||||
|
|
||||||
3. Report the best (channel, anchor-location) combination per segment.
|
|
||||||
|
|
||||||
If the rotation hypothesis is correct, you'll see:
|
|
||||||
segment 0 → best score for (Tran, preamble bytes [3:5]) ✓ already known
|
|
||||||
segment 1 → best score for (Vert, <some-header-byte>)
|
|
||||||
segment 2 → best score for (Long, <some-header-byte>)
|
|
||||||
segment 3 → best score for (MicL, <some-header-byte>)
|
|
||||||
segment 4 → best score for (Tran, continuing from segment 0's end)
|
|
||||||
|
|
||||||
If the rotation hypothesis is NOT correct, the scorer will at least narrow
|
|
||||||
down what segment 1 actually carries. Maybe channels interleave at finer
|
|
||||||
granularity, or maybe segments alternate by something other than channel.
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
Why this is a scoring analyzer, not a hand-written decoder
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
Direct hand-coding ("assume segment 1 is Vert with anchor at byte X") gets
|
|
||||||
stuck when the assumption is wrong because the failure mode is silent —
|
|
||||||
you get plausible-looking-but-wrong samples and have to manually diff
|
|
||||||
against truth to debug.
|
|
||||||
|
|
||||||
The scorer is brute-force but cheap: every fixture event × every segment ×
|
|
||||||
4 channels × 5 anchor-byte candidates is only ~hundreds of comparisons.
|
|
||||||
The winning combination jumps out by score.
|
|
||||||
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
Skeleton
|
|
||||||
────────────────────────────────────────────────────────────────────────────
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import sys
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from typing import List, Optional, Tuple
|
|
||||||
|
|
||||||
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
|
|
||||||
|
|
||||||
from minimateplus.waveform_codec import walk_body, find_data_start, WaveformBlock
|
|
||||||
|
|
||||||
|
|
||||||
# ── Reusable pieces ──────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
CHANNELS = ("Tran", "Vert", "Long", "MicL")
|
|
||||||
LSB_INV = 200 # 1 in/s / 0.005 in/s/LSB; multiply BW-export floats by this
|
|
||||||
# to get 16-count units (the body's native quantization).
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class FixtureEvent:
|
|
||||||
name: str # e.g. "M529LL1A.SP0"
|
|
||||||
bin_path: str
|
|
||||||
txt_path: str
|
|
||||||
body: bytes
|
|
||||||
truth: dict # {channel: list of int16-quantized samples}
|
|
||||||
blocks: List[WaveformBlock]
|
|
||||||
segment_starts: List[int] # block indices of each 40 02 segment header
|
|
||||||
segment_sample_starts: List[int] # for each segment, the truth sample index it starts at
|
|
||||||
|
|
||||||
|
|
||||||
def s4(n: int) -> int:
|
|
||||||
"""4-bit signed nibble decode."""
|
|
||||||
return n if n < 8 else n - 16
|
|
||||||
|
|
||||||
|
|
||||||
def i8(b: int) -> int:
|
|
||||||
"""int8 reinterpret of unsigned byte."""
|
|
||||||
return b if b < 128 else b - 256
|
|
||||||
|
|
||||||
|
|
||||||
def load_fixture(name: str) -> FixtureEvent:
|
|
||||||
"""Load a fixture event with its truth values and parsed block stream."""
|
|
||||||
# Find the fixture (search both subdirs of tests/fixtures/).
|
|
||||||
base = os.path.join(os.path.dirname(__file__), "..", "tests", "fixtures")
|
|
||||||
candidates = [
|
|
||||||
os.path.join(base, "5-11-26", name),
|
|
||||||
os.path.join(base, "decode-re-5-8-26", "event-a", name), # not used directly
|
|
||||||
]
|
|
||||||
bin_path = next((c for c in candidates if os.path.exists(c)), None)
|
|
||||||
if bin_path is None:
|
|
||||||
# Try a glob walk for the 5-8 fixtures (they're in subdirs).
|
|
||||||
for root, _, files in os.walk(base):
|
|
||||||
if name in files:
|
|
||||||
bin_path = os.path.join(root, name)
|
|
||||||
break
|
|
||||||
if bin_path is None:
|
|
||||||
raise FileNotFoundError(name)
|
|
||||||
|
|
||||||
txt_path = bin_path + ".TXT"
|
|
||||||
with open(bin_path, "rb") as f:
|
|
||||||
raw = f.read()
|
|
||||||
body = raw[43:-26]
|
|
||||||
truth = _parse_txt(txt_path)
|
|
||||||
blocks = walk_body(body, find_data_start(body))
|
|
||||||
|
|
||||||
seg_idx = [i for i, b in enumerate(blocks) if b.tag_hi == 0x40]
|
|
||||||
# Segment 0 starts at sample 0; subsequent segments start at the
|
|
||||||
# cumulative sample count from previous segment(s). Tran's segment 0
|
|
||||||
# is N samples; if rotation hypothesis is correct, segment 1's data
|
|
||||||
# starts at sample 0 for a *different* channel. The analyzer should
|
|
||||||
# try both "continues from previous segment" and "starts at sample 0
|
|
||||||
# of a different channel."
|
|
||||||
seg_sample_starts = _compute_segment_sample_starts(blocks, seg_idx)
|
|
||||||
|
|
||||||
return FixtureEvent(
|
|
||||||
name=name, bin_path=bin_path, txt_path=txt_path,
|
|
||||||
body=body, truth=truth, blocks=blocks,
|
|
||||||
segment_starts=seg_idx, segment_sample_starts=seg_sample_starts,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_txt(path: str) -> dict:
|
|
||||||
"""Parse BW ASCII TXT export into {channel: [int_samples_in_16_count_units]}."""
|
|
||||||
with open(path, "r", encoding="utf-8", errors="replace") as f:
|
|
||||||
lines = f.read().splitlines()
|
|
||||||
header_idx = next(
|
|
||||||
(i for i, l in enumerate(lines)
|
|
||||||
if all(c in l for c in CHANNELS)),
|
|
||||||
None,
|
|
||||||
)
|
|
||||||
if header_idx is None:
|
|
||||||
return {ch: [] for ch in CHANNELS}
|
|
||||||
out = {ch: [] for ch in CHANNELS}
|
|
||||||
for line in lines[header_idx + 1:]:
|
|
||||||
parts = re.split(r"\s+", line.strip())
|
|
||||||
if len(parts) < 4:
|
|
||||||
continue
|
|
||||||
try:
|
|
||||||
vals = [float(p) for p in parts[:4]]
|
|
||||||
except ValueError:
|
|
||||||
continue
|
|
||||||
for ch, v in zip(CHANNELS, vals):
|
|
||||||
# Multiply by LSB_INV; geo channels are in in/s, MicL is in dB(L)
|
|
||||||
# (which doesn't quantize the same way — leaving raw for MicL is fine,
|
|
||||||
# the scorer should treat MicL specially).
|
|
||||||
out[ch].append(round(v * LSB_INV) if ch != "MicL" else v)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def _compute_segment_sample_starts(
|
|
||||||
blocks: List[WaveformBlock], seg_idx: List[int]
|
|
||||||
) -> List[int]:
|
|
||||||
"""Cumulative sample-count up to each segment header (if all blocks treated
|
|
||||||
as Tran continuation). Useful as one candidate for segment-1-Tran tests.
|
|
||||||
|
|
||||||
The scorer should ALSO try "segment 1 starts at sample 0 of a new channel"
|
|
||||||
as the rotation hypothesis predicts.
|
|
||||||
"""
|
|
||||||
starts = []
|
|
||||||
cum = 2 # T[0] + T[1] from preamble
|
|
||||||
for i, b in enumerate(blocks):
|
|
||||||
if i in seg_idx:
|
|
||||||
starts.append(cum)
|
|
||||||
if b.tag_hi == 0x10:
|
|
||||||
cum += b.tag_lo
|
|
||||||
elif b.tag_hi == 0x20:
|
|
||||||
cum += b.tag_lo
|
|
||||||
elif b.tag_hi == 0x00:
|
|
||||||
cum += b.tag_lo
|
|
||||||
# 30 NN and 40 02 don't contribute samples (for this hypothesis)
|
|
||||||
return starts
|
|
||||||
|
|
||||||
|
|
||||||
# ── The core algorithm: decode a segment's blocks as deltas ─────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def decode_segment_as_channel(
|
|
||||||
blocks: List[WaveformBlock],
|
|
||||||
seg_start_block_idx: int,
|
|
||||||
seg_end_block_idx: int,
|
|
||||||
anchor: int,
|
|
||||||
) -> List[int]:
|
|
||||||
"""Apply the segment-0 codec rules to a range of blocks, starting from *anchor*.
|
|
||||||
|
|
||||||
Returns a list of cumulative sample values (one per delta). Does NOT include
|
|
||||||
the anchor itself in the output — the first returned value is anchor + first_delta.
|
|
||||||
"""
|
|
||||||
out = []
|
|
||||||
cur = anchor
|
|
||||||
for bi in range(seg_start_block_idx, seg_end_block_idx):
|
|
||||||
blk = blocks[bi]
|
|
||||||
if blk.tag_hi == 0x10:
|
|
||||||
for byte in blk.data:
|
|
||||||
for nib in ((byte >> 4) & 0xF, byte & 0xF):
|
|
||||||
cur += s4(nib)
|
|
||||||
out.append(cur)
|
|
||||||
elif blk.tag_hi == 0x20:
|
|
||||||
for byte in blk.data:
|
|
||||||
cur += i8(byte)
|
|
||||||
out.append(cur)
|
|
||||||
elif blk.tag_hi == 0x00:
|
|
||||||
for _ in range(blk.tag_lo):
|
|
||||||
out.append(cur)
|
|
||||||
# 30 NN: skip (content unknown)
|
|
||||||
# 40 02: shouldn't appear in segment data (it's the segment header)
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def score_against_truth(
|
|
||||||
decoded: List[int],
|
|
||||||
truth: List[int],
|
|
||||||
truth_start: int,
|
|
||||||
) -> Tuple[int, int]:
|
|
||||||
"""Compare *decoded* to truth[truth_start : truth_start + len(decoded)].
|
|
||||||
|
|
||||||
Returns (n_matches, n_compared).
|
|
||||||
"""
|
|
||||||
n = min(len(decoded), len(truth) - truth_start)
|
|
||||||
if n <= 0:
|
|
||||||
return (0, 0)
|
|
||||||
matches = sum(1 for i in range(n) if decoded[i] == truth[truth_start + i])
|
|
||||||
return (matches, n)
|
|
||||||
|
|
||||||
|
|
||||||
# ── TODO for the next pass ──────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def score_segment_against_all_channels(
|
|
||||||
event: FixtureEvent,
|
|
||||||
segment_index: int,
|
|
||||||
) -> List[Tuple[str, int, int, int]]:
|
|
||||||
"""For segment *segment_index* of *event*, find the best (channel, start_sample)
|
|
||||||
fit.
|
|
||||||
|
|
||||||
For each candidate channel C and each candidate starting truth-sample index s,
|
|
||||||
we pick the anchor that makes the FIRST decoded value match truth[C][s], then
|
|
||||||
score the remaining decoded values against truth[C][s+1 : s+N].
|
|
||||||
|
|
||||||
Returns rows of (channel_name, start_sample, n_matches, n_compared)
|
|
||||||
sorted by match-count descending.
|
|
||||||
"""
|
|
||||||
# Block range of this segment: from the segment header (inclusive) up to
|
|
||||||
# the next segment header (exclusive), or end-of-blocks.
|
|
||||||
seg_header_idx = event.segment_starts[segment_index]
|
|
||||||
next_header_idx = (
|
|
||||||
event.segment_starts[segment_index + 1]
|
|
||||||
if segment_index + 1 < len(event.segment_starts)
|
|
||||||
else len(event.blocks)
|
|
||||||
)
|
|
||||||
|
|
||||||
# Decode the segment's data blocks (skip the segment-header block itself).
|
|
||||||
# Use anchor=0 — we'll re-anchor when scoring against each channel.
|
|
||||||
deltas_trajectory = decode_segment_as_channel(
|
|
||||||
event.blocks, seg_header_idx + 1, next_header_idx, anchor=0
|
|
||||||
)
|
|
||||||
if not deltas_trajectory:
|
|
||||||
return []
|
|
||||||
|
|
||||||
n = len(deltas_trajectory)
|
|
||||||
results = []
|
|
||||||
|
|
||||||
for ch in ("Tran", "Vert", "Long"):
|
|
||||||
truth = event.truth.get(ch)
|
|
||||||
if not truth or len(truth) < n + 1:
|
|
||||||
continue
|
|
||||||
# For each candidate starting sample s in truth, check if applying
|
|
||||||
# the deltas starting from truth[s] reproduces truth[s+1:s+n+1].
|
|
||||||
best = (0, -1)
|
|
||||||
for s in range(len(truth) - n):
|
|
||||||
anchor = truth[s]
|
|
||||||
offset = anchor - deltas_trajectory[0] + truth[s + 1] - anchor
|
|
||||||
# Recompute: trajectory[i] = anchor + cumulative_delta_through_i
|
|
||||||
# but we already have deltas_trajectory computed from anchor=0,
|
|
||||||
# so trajectory_relative[i] = anchor + deltas_trajectory[i].
|
|
||||||
matches = 0
|
|
||||||
for i in range(n):
|
|
||||||
if truth[s + i + 1] == anchor + deltas_trajectory[i]:
|
|
||||||
matches += 1
|
|
||||||
# Note: we could break early on first mismatch for "matches start",
|
|
||||||
# but counting total matches gives a more robust score.
|
|
||||||
if matches > best[0]:
|
|
||||||
best = (matches, s)
|
|
||||||
results.append((ch, best[1], best[0], n))
|
|
||||||
|
|
||||||
results.sort(key=lambda r: -r[2])
|
|
||||||
return results
|
|
||||||
|
|
||||||
|
|
||||||
# ── Driver ──────────────────────────────────────────────────────────────────
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
"""Run the analyzer on all loud-bundle events and print best scores."""
|
|
||||||
events = ["M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0",
|
|
||||||
"M529LL1L.JQ0", "M529LL1L.V70"]
|
|
||||||
for name in events:
|
|
||||||
try:
|
|
||||||
event = load_fixture(name)
|
|
||||||
except FileNotFoundError:
|
|
||||||
print(f"{name}: fixture not found")
|
|
||||||
continue
|
|
||||||
|
|
||||||
print(f"\n=== {name} ===")
|
|
||||||
print(f" body bytes: {len(event.body)}")
|
|
||||||
print(f" blocks: {len(event.blocks)}")
|
|
||||||
print(f" segments: {len(event.segment_starts)}")
|
|
||||||
print(f" segment sample-starts (if all blocks are 1 channel):")
|
|
||||||
for si, sample_start in enumerate(event.segment_sample_starts):
|
|
||||||
print(f" seg {si}: sample {sample_start}")
|
|
||||||
|
|
||||||
for si in range(len(event.segment_starts)):
|
|
||||||
results = score_segment_against_all_channels(event, si)
|
|
||||||
if not results:
|
|
||||||
print(f" seg {si}: (no scorable data)")
|
|
||||||
continue
|
|
||||||
tag = "✓" if results[0][2] / max(results[0][3], 1) > 0.9 else " "
|
|
||||||
top = results[0]
|
|
||||||
print(f" seg {si}: best fit {tag} = {top[0]:<5} "
|
|
||||||
f"starting at sample {top[1]:>5}, {top[2]:>4}/{top[3]:<4} match"
|
|
||||||
+ (f" (next: {results[1][0]} @{results[1][1]} {results[1][2]}/{results[1][3]})"
|
|
||||||
if len(results) > 1 else ""))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,275 +0,0 @@
|
|||||||
event_id,serial,timestamp,filename,channel,offset_ips,peak_ips,mean_over_peak,trigger_level_ips,already_flagged_ft
|
|
||||||
1e7a7808-034c-423b-80e9-d6788da80993,BE18438,2025-11-15T08:57:40,T438LBX4.W40W,Vert,0.2722,0.2930,0.929,0.2,0
|
|
||||||
346e383f-c5a1-41d1-b972-3d652e599d39,BE18438,2025-11-15T09:13:15,T438LBX5.M30W,Vert,0.2747,0.2930,0.938,0.2,0
|
|
||||||
45ef3023-22c5-44a8-a606-a0900bc17861,BE18438,2025-11-15T09:16:13,T438LBX5.R10W,Vert,0.2929,0.3027,0.967,0.2,0
|
|
||||||
1cbcd2aa-ec55-4e2f-af42-7178f26cc6a1,BE18438,2025-11-15T09:19:29,T438LBX5.WH0W,Vert,0.2816,0.2979,0.945,0.2,0
|
|
||||||
caf43634-4528-4fd5-9674-5f4f563661c0,BE18438,2025-11-15T09:22:32,T438LBX6.1K0W,Vert,0.2883,0.2930,0.984,0.2,0
|
|
||||||
456ae646-c834-4e5c-82f1-7df18b3440a1,BE18438,2025-11-15T09:25:31,T438LBX6.6J0W,Vert,0.3189,0.3223,0.989,0.2,0
|
|
||||||
afb91ba2-351b-42a0-b33f-acebd5f1829d,BE18438,2025-11-15T09:28:29,T438LBX6.BH0W,Vert,0.3057,0.3076,0.994,0.2,0
|
|
||||||
39847146-3537-4300-943d-01c0c771e9cd,BE18438,2025-11-15T09:31:26,T438LBX6.GE0W,Vert,0.3225,0.3271,0.986,0.2,0
|
|
||||||
e47335c1-fc3f-4c90-88c6-8edd0f296ce4,BE18438,2025-11-15T09:34:24,T438LBX6.LC0W,Vert,0.3275,0.3320,0.986,0.2,0
|
|
||||||
25b6eda6-ba98-422f-8834-94b6cbe34971,BE18438,2025-11-15T09:37:22,T438LBX6.QA0W,Vert,0.3398,0.3418,0.994,0.2,0
|
|
||||||
7af99377-e47f-4914-8e91-ac7f7f50b8c6,BE18438,2025-11-15T09:40:20,T438LBX6.V80W,Vert,0.3515,0.3564,0.986,0.2,0
|
|
||||||
c4330605-c257-49ef-aede-20deb641e7d7,BE18438,2025-11-15T10:09:48,T438LBX8.8C0W,Vert,0.3605,0.3955,0.912,0.2,0
|
|
||||||
b665caf9-64ee-4352-a391-60c1ddeebc1e,BE18438,2026-02-25T10:58:04,T438LH66.GS0W,Vert,0.1803,0.1953,0.923,0.2,0
|
|
||||||
3b0c32c3-fd16-4b83-9917-942a56dea168,BE18438,2026-02-25T18:12:04,T438LH6Q.K40W,Vert,0.1869,0.1953,0.957,0.2,0
|
|
||||||
ca66e601-d10f-4424-9c26-08a688e7b08c,BE18438,2026-02-25T18:17:16,T438LH6Q.SS0W,Vert,0.1870,0.1953,0.958,0.2,0
|
|
||||||
31aa30ba-eb0a-49a9-98ca-c77d2cc31275,BE18438,2026-02-25T18:21:11,T438LH6Q.ZB0W,Vert,0.1868,0.1953,0.956,0.2,0
|
|
||||||
fdb55a93-fc91-4fdf-a7f4-fdd1a6b48f59,BE18438,2026-02-25T18:26:27,T438LH6R.830W,Vert,0.1893,0.1953,0.969,0.2,0
|
|
||||||
cbd718d7-2378-4d74-8656-842f229d19a8,BE18438,2026-02-25T18:33:25,T438LH6R.JP0W,Vert,0.1887,0.1953,0.966,0.2,0
|
|
||||||
bfaf58ec-6163-47eb-9d2a-70da880bd948,BE18438,2026-02-25T18:37:48,T438LH6R.R00W,Vert,0.1872,0.1953,0.959,0.2,0
|
|
||||||
ddaa530b-886e-4415-b93a-f4ebf4c4a882,BE18438,2026-02-25T18:59:54,T438LH6S.RU0W,Vert,0.1874,0.1953,0.959,0.2,0
|
|
||||||
57f15284-ab69-48fc-9660-8c347c80c681,BE18438,2026-02-25T19:15:58,T438LH6T.IM0W,Vert,0.1873,0.1953,0.959,0.2,0
|
|
||||||
03715718-7aa8-4e1d-9b0b-c2ab75fc1975,BE18438,2026-02-25T19:19:58,T438LH6T.PA0W,Vert,0.1891,0.1953,0.968,0.2,0
|
|
||||||
9f921cd5-1bf9-4713-aef6-41e7f780eff1,BE18438,2026-02-25T19:24:02,T438LH6T.W20W,Vert,0.1871,0.1953,0.958,0.2,0
|
|
||||||
44994949-58d4-44fa-bd78-ed75b8acc667,BE18438,2026-02-25T19:44:57,T438LH6U.UX0W,Vert,0.1877,0.1953,0.961,0.2,0
|
|
||||||
c4be01c3-1001-4e17-aecd-d6d26dbb09f3,BE18438,2026-02-25T19:49:19,T438LH6V.270W,Vert,0.1884,0.1953,0.964,0.2,0
|
|
||||||
3db004e6-ee34-43c5-ae8e-f22894ead5d0,BE18438,2026-02-25T20:08:22,T438LH6V.XY0W,Vert,0.1878,0.1953,0.962,0.2,0
|
|
||||||
d276ae48-0434-41e5-af43-e0fb366acf11,BE18438,2026-02-25T20:11:33,T438LH6W.390W,Vert,0.1880,0.1953,0.963,0.2,0
|
|
||||||
c027dc12-88fd-4a45-872b-a6a945f2008e,BE18438,2026-02-25T20:14:35,T438LH6W.8B0W,Vert,0.1874,0.1953,0.959,0.2,0
|
|
||||||
9665bc3a-3d08-4852-b008-37ef8bfd0023,BE18438,2026-02-25T20:18:51,T438LH6W.FF0W,Vert,0.1884,0.1953,0.965,0.2,0
|
|
||||||
5fb5a05e-d7e2-4268-9b85-cfe8d82aa6c9,BE18438,2026-02-25T20:29:50,T438LH6W.XQ0W,Vert,0.1890,0.1953,0.968,0.2,0
|
|
||||||
57ec0b10-9324-4459-a07e-0c8532b8928f,BE18438,2026-02-25T20:35:39,T438LH6X.7F0W,Vert,0.1889,0.1953,0.967,0.2,0
|
|
||||||
ac86f73f-1174-4fae-a206-ecdcb73e1ffc,BE18438,2026-02-25T20:39:12,T438LH6X.DC0W,Vert,0.1887,0.1953,0.966,0.2,0
|
|
||||||
dcf25ddd-9cf0-41c1-b109-f3ef2e490ce5,BE18438,2026-02-25T20:43:41,T438LH6X.KT0W,Vert,0.1871,0.1953,0.958,0.2,0
|
|
||||||
aa79bbdd-2392-4ef6-8a5c-873a9f72746d,BE18438,2026-02-25T20:46:51,T438LH6X.Q30W,Vert,0.1877,0.1953,0.961,0.2,0
|
|
||||||
ad46d1ce-b417-4c8c-b029-bb3b008ba809,BE18438,2026-02-25T20:53:33,T438LH6Y.190W,Vert,0.1882,0.1953,0.964,0.2,0
|
|
||||||
3d3a8f34-86a7-4d70-a7e8-fa768a723325,BE18438,2026-02-26T07:03:39,T438LH7Q.A30W,Vert,0.1809,0.1953,0.926,0.2,0
|
|
||||||
7817525f-dc3d-4f5e-b2cb-5af9ca12ede8,BE18438,2026-02-26T07:09:42,T438LH7Q.K60W,Vert,0.1814,0.1953,0.929,0.2,0
|
|
||||||
a2091302-0b34-46c1-ab83-174be2b59bcf,BE18438,2026-02-26T13:12:35,T438LH87.CZ0W,Vert,0.1896,0.1953,0.971,0.2,0
|
|
||||||
e1c0c9de-a900-479f-9572-9675b13296a9,BE18438,2026-02-26T13:24:46,T438LH87.XA0W,Vert,0.2237,0.2588,0.864,0.2,0
|
|
||||||
76a6dc6b-88e1-434b-a137-f5e9b63f248e,BE18438,2026-02-26T13:27:43,T438LH88.270W,Vert,0.3185,0.3223,0.988,0.2,0
|
|
||||||
281b508b-7982-4668-86af-9b0b28be608a,BE18438,2026-02-26T13:30:36,T438LH88.700W,Vert,0.3263,0.3320,0.983,0.2,0
|
|
||||||
68bbaf77-bb23-40fd-94f9-f928728d90dc,BE18438,2026-02-26T13:33:28,T438LH88.BS0W,Vert,0.3207,0.3271,0.980,0.2,0
|
|
||||||
5c975640-6ff1-441a-aa11-41642f6a6b31,BE18438,2026-02-26T13:36:21,T438LH88.GL0W,Vert,0.3261,0.3320,0.982,0.2,0
|
|
||||||
dea248ac-1124-4bba-b054-953f86c852b2,BE18438,2026-02-26T13:40:34,T438LH88.NM0W,Vert,0.3271,0.3320,0.985,0.2,0
|
|
||||||
4e463201-f008-41ad-9c52-11fcc3d93095,BE18438,2026-02-26T13:43:29,T438LH88.SH0W,Vert,0.3357,0.3418,0.982,0.2,0
|
|
||||||
507262c5-d58f-4a05-a6a3-a6cbb5b4de20,BE18438,2026-02-26T13:46:24,T438LH88.XC0W,Vert,0.3372,0.3418,0.987,0.2,0
|
|
||||||
4c2d8ee8-842f-4e44-9e03-ff34d26e4fdb,BE18438,2026-02-26T13:49:17,T438LH89.250W,Vert,0.3253,0.3320,0.980,0.2,0
|
|
||||||
b184b112-603f-44f4-ae97-087493fbd8ce,BE18438,2026-02-26T13:52:10,T438LH89.6Y0W,Vert,0.3321,0.3369,0.986,0.2,0
|
|
||||||
2ff1a552-5a2e-4396-9083-b9c8d334eed7,BE18438,2026-02-26T13:55:04,T438LH89.BS0W,Vert,0.3378,0.3418,0.988,0.2,0
|
|
||||||
928e21a7-1990-4792-803e-58014d8140ce,BE18438,2026-02-26T13:57:57,T438LH89.GL0W,Vert,0.3341,0.3369,0.992,0.2,0
|
|
||||||
f521b5db-6100-41e6-954e-0abdbf9660c1,BE18438,2026-02-26T14:00:51,T438LH89.LF0W,Vert,0.3421,0.3467,0.987,0.2,0
|
|
||||||
efba962f-e511-41ef-9fc5-313f202a9d96,BE18438,2026-02-26T14:03:46,T438LH89.QA0W,Vert,0.3506,0.3564,0.984,0.2,0
|
|
||||||
0b51e273-8d10-4494-bda6-8c75eaeec66f,BE18438,2026-02-26T14:06:41,T438LH89.V50W,Vert,0.3448,0.3516,0.981,0.2,0
|
|
||||||
6d46e4ee-1647-4169-b6b2-33da32240eb9,BE18438,2026-02-26T14:09:35,T438LH89.ZZ0W,Vert,0.3463,0.3516,0.985,0.2,0
|
|
||||||
1ca294cd-4dc2-41fb-9374-e277d4550761,BE18438,2026-02-26T14:12:28,T438LH8A.4S0W,Vert,0.3442,0.3467,0.993,0.2,0
|
|
||||||
7b683dcf-1aed-4d9b-be1e-b10f92374398,BE18438,2026-02-26T14:15:23,T438LH8A.9N0W,Vert,0.3486,0.3516,0.992,0.2,0
|
|
||||||
2b5cb8e7-c947-4914-bcb5-504f74603547,BE18438,2026-02-26T14:18:21,T438LH8A.EL0W,Vert,0.3519,0.3613,0.974,0.2,0
|
|
||||||
5ea3a6bf-10de-4911-aedc-66daecf86b76,BE18438,2026-02-26T14:21:16,T438LH8A.JG0W,Vert,0.3460,0.3564,0.971,0.2,0
|
|
||||||
d93be7f5-39ec-45fd-9949-fa531f2f9d34,BE18438,2026-02-26T14:24:13,T438LH8A.OD0W,Vert,0.3457,0.3516,0.983,0.2,0
|
|
||||||
cfa113f2-8cee-4a1d-b1e7-0e936fa771e2,BE18438,2026-02-26T14:28:21,T438LH8A.V90W,Vert,0.3485,0.3564,0.978,0.2,0
|
|
||||||
46d322ac-38cb-4c99-a4c2-9e9d7ad06171,BE18438,2026-02-26T14:31:17,T438LH8B.050W,Vert,0.3367,0.3467,0.971,0.2,0
|
|
||||||
42500a83-bd67-4deb-8bb2-963ee2b8a7ce,BE18438,2026-02-26T14:34:11,T438LH8B.4Z0W,Vert,0.3365,0.3516,0.957,0.2,0
|
|
||||||
3ccf6bda-2e88-473c-951d-9ea86d0d9363,BE18438,2026-02-26T14:37:05,T438LH8B.9T0W,Vert,0.3472,0.3564,0.974,0.2,0
|
|
||||||
9050487d-d238-4631-b8e1-5a693a026542,BE18438,2026-02-26T14:39:58,T438LH8B.EM0W,Vert,0.3420,0.3564,0.959,0.2,0
|
|
||||||
e62989e4-a326-4bae-8d47-a8becee96462,BE18438,2026-02-26T14:42:52,T438LH8B.JG0W,Vert,0.3430,0.3564,0.962,0.2,0
|
|
||||||
f0e30fab-dc57-4189-8439-48df17007a2c,BE18438,2026-02-26T14:45:46,T438LH8B.OA0W,Vert,0.3389,0.3516,0.964,0.2,0
|
|
||||||
03e1ca09-78df-4c6f-9915-0b6e4f2141d2,BE18438,2026-02-26T14:48:43,T438LH8B.T70W,Vert,0.3449,0.3564,0.968,0.2,0
|
|
||||||
945d39f7-1b98-4a1f-80a8-6a18fd9df25f,BE18438,2026-02-26T14:51:37,T438LH8B.Y10W,Vert,0.3522,0.3613,0.975,0.2,0
|
|
||||||
d27eb2e3-e313-4125-98f4-f9d6d3737d28,BE18438,2026-02-26T14:54:32,T438LH8C.2W0W,Vert,0.3502,0.3613,0.969,0.2,0
|
|
||||||
916901cb-ebea-441e-a2a1-40f4e49f32f4,BE18438,2026-02-26T14:57:25,T438LH8C.7P0W,Vert,0.3524,0.3662,0.962,0.2,0
|
|
||||||
5d8d7e29-fdbb-4fa3-bd25-506c9352a354,BE18438,2026-02-26T15:01:35,T438LH8C.EN0W,Vert,0.3530,0.3613,0.977,0.2,0
|
|
||||||
c6c7a3af-b03b-43dc-97de-024f8e09b62f,BE18438,2026-02-26T15:04:27,T438LH8C.JF0W,Vert,0.3514,0.3613,0.973,0.2,0
|
|
||||||
b6856898-3d10-4967-9577-7f17902f77ea,BE18438,2026-02-26T15:11:34,T438LH8C.VA0W,Vert,0.3566,0.3613,0.987,0.2,0
|
|
||||||
7a4802f6-621e-4230-8d8c-d7f0c9a65bf2,BE18438,2026-02-26T15:14:24,T438LH8D.000W,Vert,0.3594,0.3711,0.968,0.2,0
|
|
||||||
79f38b7e-ec56-417e-943a-8c944252c2b8,BE18438,2026-02-26T15:18:32,T438LH8D.6W0W,Vert,0.3550,0.3613,0.983,0.2,0
|
|
||||||
1b4190c4-0d54-48ba-8ff5-8ddb0eb91e50,BE18438,2026-02-26T15:22:22,T438LH8D.DA0W,Vert,0.3571,0.3613,0.988,0.2,0
|
|
||||||
de57b5ae-a3a1-4ede-9d2b-3e87bfb3fd19,BE9558,2026-04-14T11:16:32,K558LJN3.BK0W,Tran,0.3448,0.3662,0.942,0.2,0
|
|
||||||
43afeaf5-02ec-41f6-9e23-0c9772821ed4,BE9558,2026-04-14T11:27:15,K558LJN3.TF0W,Tran,0.3094,0.3223,0.960,0.2,0
|
|
||||||
8123c0ef-84c9-4f6c-8d82-9dc32e2e470d,BE9558,2026-04-14T14:45:30,K558LJNC.ZU0W,Tran,0.2721,0.3564,0.763,0.2,0
|
|
||||||
8c787af3-596e-411b-9e10-29485fa5114f,BE9558,2026-04-29T16:18:47,K558LKF9.BB0W,Tran,0.2943,0.3027,0.972,0.2,0
|
|
||||||
431928ff-b4c4-4caa-b933-acc917f3717c,BE9558,2026-05-04T15:02:30,K558LKOF.460W,Tran,0.4364,0.5225,0.835,0.2,0
|
|
||||||
6c3c07f8-c36a-4493-acce-7441c9d22cec,BE9558,2026-05-15T08:50:06,K558LL8B.7I0W,Long,0.2892,0.2930,0.987,0.2,0
|
|
||||||
13c268df-b652-422e-a642-6453f6a314aa,BE9558,2026-05-15T10:18:34,K558LL8F.AY0W,Long,0.2907,0.2979,0.976,0.2,0
|
|
||||||
9b0d0871-8810-467d-806a-5bbfa4e667fe,BE9558,2026-05-15T15:52:00,K558LL8U.QO0W,Long,0.2659,0.2979,0.893,0.2,0
|
|
||||||
092640f9-a944-48b7-b872-364d75c2c5e7,BE9558,2026-05-15T16:13:12,K558LL8V.Q00W,Long,0.2428,0.2979,0.815,0.2,0
|
|
||||||
0573741a-96ab-4b36-af7a-b5bc11a79009,BE9558,2026-05-16T03:23:26,K558LL9Q.R20W,Long,0.2861,0.2930,0.976,0.2,0
|
|
||||||
321f03ea-6696-47de-ad46-2be43d4d6cae,BE9558,2026-05-16T03:30:49,K558LL9R.3D0W,Long,0.2886,0.2930,0.985,0.2,0
|
|
||||||
cc423a6a-3e3c-466a-b39c-6146ca7f34c8,BE9558,2026-05-16T03:33:55,K558LL9R.8J0W,Long,0.2880,0.2881,1.000,0.2,0
|
|
||||||
efa52a14-cc68-4bea-aab4-4ee3a3cbff3a,BE9558,2026-05-16T03:36:55,K558LL9R.DJ0W,Long,0.2890,0.2930,0.986,0.2,0
|
|
||||||
92895bfe-122f-4cdc-bd3d-40609633d278,BE9558,2026-05-16T03:43:41,K558LL9R.OT0W,Long,0.2880,0.2881,1.000,0.2,0
|
|
||||||
2a4c81db-2307-44e5-9b27-d5154d98e38d,BE9558,2026-05-16T03:46:36,K558LL9R.TO0W,Long,0.2921,0.2979,0.981,0.2,0
|
|
||||||
656b0fdc-e275-4c9c-8936-9eee267ea04e,BE9558,2026-05-16T03:49:28,K558LL9R.YG0W,Long,0.2966,0.2979,0.996,0.2,0
|
|
||||||
3b9989ae-e0f2-4652-9724-c145603543d2,BE9558,2026-05-16T03:52:29,K558LL9S.3H0W,Long,0.2880,0.2930,0.983,0.2,0
|
|
||||||
837d0d84-d609-4f4a-b542-b7e2b064ea22,BE9558,2026-05-16T03:56:32,K558LL9S.A80W,Long,0.2880,0.2930,0.983,0.2,0
|
|
||||||
affcb008-ab99-4bc5-8f35-83dbe620499d,BE9558,2026-05-16T04:02:50,K558LL9S.KQ0W,Long,0.2896,0.2930,0.989,0.2,0
|
|
||||||
e1f1c6d2-545c-451f-9018-714d52e20a05,BE9558,2026-05-16T04:05:47,K558LL9S.PN0W,Long,0.3024,0.3076,0.983,0.2,0
|
|
||||||
93ef80f1-7e83-4703-a0f9-89bbfeeae6c3,BE9558,2026-05-16T04:08:41,K558LL9S.UH0W,Long,0.2976,0.2979,0.999,0.2,0
|
|
||||||
5c118ca9-fc37-4080-876d-b3f57ce0b121,BE9558,2026-05-16T04:11:42,K558LL9S.ZI0W,Long,0.2881,0.2930,0.983,0.2,0
|
|
||||||
d98b2eaa-7763-4c78-88fb-c21195cb9978,BE9558,2026-05-16T04:14:38,K558LL9T.4E0W,Long,0.2898,0.2930,0.989,0.2,0
|
|
||||||
6a146a01-0d02-4413-b231-ecb4a6fde776,BE9558,2026-05-16T04:17:35,K558LL9T.9B0W,Long,0.2880,0.2930,0.983,0.2,0
|
|
||||||
f6587a55-27e3-454f-997e-57e7516a296c,BE9558,2026-05-16T04:20:32,K558LL9T.E80W,Long,0.3046,0.3076,0.990,0.2,0
|
|
||||||
1fb48250-2214-41e6-92cb-362aff1ba62b,BE9558,2026-05-16T04:26:05,K558LL9T.NH0W,Long,0.2876,0.2930,0.982,0.2,0
|
|
||||||
2f4f3d06-1189-4e28-a1ac-5454ea586a0c,BE9558,2026-05-16T04:59:47,K558LL9V.7N0W,Long,0.2961,0.3027,0.978,0.2,0
|
|
||||||
62b7bdab-b744-41e4-8d0e-c9141b9d4821,BE9558,2026-05-16T05:02:46,K558LL9V.CM0W,Long,0.2903,0.2930,0.991,0.2,0
|
|
||||||
cadcb6dc-74c0-48b5-974c-87bb7a57816c,BE9558,2026-05-16T05:05:43,K558LL9V.HJ0W,Long,0.3090,0.3125,0.989,0.2,0
|
|
||||||
a0235177-be5a-4f11-b741-c65744698e5b,BE9558,2026-05-16T05:08:35,K558LL9V.MB0W,Long,0.3167,0.3223,0.983,0.2,0
|
|
||||||
1968c2f6-f52c-4544-a0ed-f79b91ce0da0,BE9558,2026-05-16T05:11:28,K558LL9V.R40W,Long,0.3446,0.3516,0.980,0.2,0
|
|
||||||
caa06b94-8813-4949-b47b-61ffb16e61c4,BE9558,2026-05-16T05:14:22,K558LL9V.VY0W,Long,0.3318,0.3320,0.999,0.2,0
|
|
||||||
fdaa8101-f033-47e6-9c65-abd76dea1870,BE9558,2026-05-16T05:17:16,K558LL9W.0S0W,Long,0.3407,0.3418,0.997,0.2,0
|
|
||||||
29bd16bd-81b9-4ccd-a981-f1a80858536c,BE9558,2026-05-16T05:23:05,K558LL9W.AH0W,Long,0.3450,0.3516,0.981,0.2,0
|
|
||||||
1f8bc862-d5fd-4cce-abc0-e2bd8c2f11fa,BE9558,2026-05-16T05:26:00,K558LL9W.FC0W,Long,0.3417,0.3467,0.986,0.2,0
|
|
||||||
20220068-d498-4fca-9a2a-a7fd2ee52ab1,BE9558,2026-05-16T05:28:53,K558LL9W.K50W,Long,0.3516,0.3564,0.986,0.2,0
|
|
||||||
9816a261-3118-417a-8334-ace36687ad8f,BE9558,2026-05-16T05:31:47,K558LL9W.OZ0W,Long,0.3416,0.3467,0.985,0.2,0
|
|
||||||
5d2c76d7-543d-4557-9caf-996a69ab00bd,BE9558,2026-05-16T05:34:42,K558LL9W.TU0W,Long,0.3537,0.3564,0.992,0.2,0
|
|
||||||
c7b5ae00-52e7-4920-aa27-49d3b3daff92,BE9558,2026-05-16T05:37:36,K558LL9W.YO0W,Long,0.3517,0.3564,0.987,0.2,0
|
|
||||||
07db8fd8-ccef-4a5a-9d97-9f318f73a478,BE9558,2026-05-16T05:40:31,K558LL9X.3J0W,Long,0.3574,0.3613,0.989,0.2,0
|
|
||||||
790fcff9-e266-4a12-87f0-a072990d2533,BE9558,2026-05-16T05:43:26,K558LL9X.8E0W,Long,0.3650,0.3662,0.997,0.2,0
|
|
||||||
f4245f61-1dde-43dc-a121-98e102f6a193,BE9558,2026-05-16T05:50:28,K558LL9X.K40W,Long,0.3856,0.3906,0.987,0.2,0
|
|
||||||
5a3bce7b-358e-4260-876f-3f853163d7cf,BE9558,2026-05-16T05:53:22,K558LL9X.OY0W,Long,0.3847,0.3857,0.997,0.2,0
|
|
||||||
37c2b4b1-7a1b-43e7-b513-2c16f582109c,BE9558,2026-05-16T05:56:15,K558LL9X.TR0W,Long,0.3906,0.3955,0.988,0.2,0
|
|
||||||
57fde109-1d8a-45a1-b398-5bd28c3e47f1,BE9558,2026-05-16T05:59:10,K558LL9X.YM0W,Long,0.4000,0.4004,0.999,0.2,0
|
|
||||||
1e41ab40-93be-405c-a464-f7a57524d3b3,BE9558,2026-05-16T06:02:02,K558LL9Y.3E0W,Long,0.3825,0.3857,0.992,0.2,0
|
|
||||||
e9d61eb5-fae8-4902-8fab-4828ffa8e6da,BE9558,2026-05-16T06:04:57,K558LL9Y.890W,Long,0.3802,0.3809,0.998,0.2,0
|
|
||||||
c0e964e0-7788-4e41-bc41-6e479c3dc81b,BE9558,2026-05-16T06:07:51,K558LL9Y.D30W,Long,0.3954,0.4004,0.987,0.2,0
|
|
||||||
0b2df8e5-1689-4fe6-abad-30970ba10aa6,BE9558,2026-05-16T06:16:06,K558LL9Y.QU0W,Long,0.4051,0.4053,1.000,0.2,0
|
|
||||||
2bc6ce73-464b-469c-9e8f-3bf6f6cfae1d,BE9558,2026-05-16T06:21:51,K558LL9Z.0F0W,Long,0.3910,0.3955,0.989,0.2,0
|
|
||||||
b8603ee1-616a-451e-a2cd-61f8f2634cd0,BE9558,2026-05-16T06:24:44,K558LL9Z.580W,Long,0.3969,0.4004,0.991,0.2,0
|
|
||||||
1aecf582-66a2-4ebb-80f9-04b3aeba331a,BE9558,2026-05-16T06:27:39,K558LL9Z.A30W,Long,0.3915,0.3955,0.990,0.2,0
|
|
||||||
885aedfd-7a1a-4120-8176-7c344dd3d8fe,BE9558,2026-05-16T06:30:30,K558LL9Z.EU0W,Long,0.4025,0.4053,0.993,0.2,0
|
|
||||||
ec15930f-47f5-4752-afc2-427ee156ac95,BE9558,2026-05-16T06:33:25,K558LL9Z.JP0W,Long,0.4060,0.4102,0.990,0.2,0
|
|
||||||
5ca23f47-2a38-4a54-99dd-041287aa9510,BE9558,2026-05-16T06:36:17,K558LL9Z.OH0W,Long,0.4054,0.4102,0.988,0.2,0
|
|
||||||
03f371cf-81bc-4681-80f0-ef6fa437da85,BE9558,2026-05-16T06:39:09,K558LL9Z.T90W,Long,0.4052,0.4102,0.988,0.2,0
|
|
||||||
ed386f54-140d-45e8-ad51-6585a58b4375,BE9558,2026-05-16T06:44:54,K558LLA0.2U0W,Long,0.3993,0.4004,0.997,0.2,0
|
|
||||||
579f1c3f-4c1e-42a3-aa89-69aa93514033,BE9558,2026-05-16T06:47:47,K558LLA0.7N0W,Long,0.4001,0.4053,0.987,0.2,0
|
|
||||||
57024de6-07ea-41f4-b916-c6adc3f51636,BE9558,2026-05-16T06:50:40,K558LLA0.CG0W,Long,0.4103,0.4150,0.989,0.2,0
|
|
||||||
36bf5adf-afb5-4a5e-a106-1afbd1f56c7b,BE9558,2026-05-16T06:54:49,K558LLA0.JD0W,Long,0.4069,0.4150,0.980,0.2,0
|
|
||||||
3e2f831b-5228-46ff-9acb-75d8c86f8bc1,BE9558,2026-05-16T06:57:43,K558LLA0.O70W,Long,0.4108,0.4150,0.990,0.2,0
|
|
||||||
1775fa20-568b-4294-b9ca-561201a66c5b,BE9558,2026-05-16T07:00:38,K558LLA0.T20W,Long,0.3967,0.4004,0.991,0.2,0
|
|
||||||
fd391469-03f9-4d8d-bccd-3af50c91e5b7,BE9558,2026-05-16T07:03:33,K558LLA0.XX0W,Long,0.3927,0.3955,0.993,0.2,0
|
|
||||||
02b52ef6-b707-4dcb-b58f-43da02698832,BE9558,2026-05-16T07:06:27,K558LLA1.2R0W,Long,0.3824,0.3857,0.991,0.2,0
|
|
||||||
8d45333f-e101-4249-9e27-913c583a0a8d,BE9558,2026-05-16T07:09:21,K558LLA1.7L0W,Long,0.4085,0.4150,0.984,0.2,0
|
|
||||||
361d11ce-f42a-4d04-8270-ee3771fe4f09,BE9558,2026-05-16T07:12:14,K558LLA1.CE0W,Long,0.4017,0.4053,0.991,0.2,0
|
|
||||||
59218125-ed5d-4457-aa8b-35ba03441363,BE9558,2026-05-16T07:15:07,K558LLA1.H70W,Long,0.3934,0.3955,0.995,0.2,0
|
|
||||||
a603e55a-2ad6-4fc9-8c8c-0dc7be95c3e7,BE9558,2026-05-16T07:23:43,K558LLA1.VJ0W,Long,0.3806,0.3857,0.987,0.2,0
|
|
||||||
c8d07902-93ea-4b03-9e69-9ad6ed5bffb1,BE9558,2026-05-16T07:26:36,K558LLA2.0C0W,Long,0.3807,0.3857,0.987,0.2,0
|
|
||||||
4c46c0c7-42f0-4648-827e-1995f733b18f,BE9558,2026-05-16T07:30:44,K558LLA2.780W,Long,0.3667,0.3711,0.988,0.2,0
|
|
||||||
04759a00-ba3e-4959-a4cb-be8fc1f3004e,BE9558,2026-05-16T07:36:28,K558LLA2.GS0W,Long,0.3514,0.3516,1.000,0.2,0
|
|
||||||
0b6b0856-bd87-4a07-a5b4-f13d99e04a12,BE9558,2026-05-16T07:39:20,K558LLA2.LK0W,Long,0.3512,0.3516,0.999,0.2,0
|
|
||||||
3d81a8bd-464d-4050-8a76-25e8373d1ddb,BE9558,2026-05-16T07:42:12,K558LLA2.QC0W,Long,0.3350,0.3369,0.994,0.2,0
|
|
||||||
b8b9c0c4-c388-4acb-955a-be4ac356afce,BE9558,2026-05-16T07:45:04,K558LLA2.V40W,Long,0.3563,0.3564,1.000,0.2,0
|
|
||||||
cb3d142a-fb84-4a21-a298-f61bb869b036,BE9558,2026-05-16T07:49:13,K558LLA3.210W,Long,0.3663,0.3711,0.987,0.2,0
|
|
||||||
7401b19b-bc9d-43f0-b395-f4323aca5c72,BE9558,2026-05-16T07:52:06,K558LLA3.6U0W,Long,0.3612,0.3662,0.986,0.2,0
|
|
||||||
2236df76-a9e6-401f-9176-ad565242f2cf,BE9558,2026-05-16T07:57:52,K558LLA3.GG0W,Long,0.3366,0.3369,0.999,0.2,0
|
|
||||||
a645846c-cae4-4aa5-a2e6-85166bd4dd62,BE9558,2026-05-16T08:00:44,K558LLA3.L80W,Long,0.3265,0.3271,0.998,0.2,0
|
|
||||||
7212bd33-99a9-4659-9f0b-c98cc8d1e8bc,BE9558,2026-05-16T08:04:51,K558LLA3.S30W,Long,0.3418,0.3467,0.986,0.2,0
|
|
||||||
2cc4bf33-c63a-4c65-b0e2-5e61a9146a5d,BE9558,2026-05-16T08:07:43,K558LLA3.WV0W,Long,0.3405,0.3418,0.996,0.2,0
|
|
||||||
4fedb486-5f1f-4623-a01f-083e829f0565,BE9558,2026-05-16T08:10:35,K558LLA4.1N0W,Long,0.3515,0.3564,0.986,0.2,0
|
|
||||||
58aff0ba-0f9b-45e1-9bbe-a990d16192ae,BE9558,2026-05-16T08:13:27,K558LLA4.6F0W,Long,0.3513,0.3516,0.999,0.2,0
|
|
||||||
e56d87e5-e1d4-4947-b6a5-3bbd2773c54a,BE9558,2026-05-16T08:20:03,K558LLA4.HF0W,Long,0.3564,0.3613,0.986,0.2,0
|
|
||||||
0e96e61e-6a21-4e07-8e09-72d09dbfa1e6,BE9558,2026-05-16T08:25:21,K558LLA4.Q90W,Long,0.3565,0.3613,0.987,0.2,0
|
|
||||||
19611f28-2da8-4362-a58e-dbed2e061379,BE9558,2026-05-16T08:34:19,K558LLA5.570W,Long,0.3647,0.3662,0.996,0.2,0
|
|
||||||
bd580c1d-44d4-41fd-9f83-f462cda4099a,BE9558,2026-05-16T08:37:15,K558LLA5.A30W,Long,0.3536,0.3564,0.992,0.2,0
|
|
||||||
2782df3b-3cb4-471a-a6f5-bbc4903a8ab9,BE9558,2026-05-16T08:44:00,K558LLA5.LC0W,Long,0.3612,0.3662,0.986,0.2,0
|
|
||||||
d15a08f3-1c7c-4f80-8d9b-f5008c58c3d9,BE9558,2026-05-16T08:49:18,K558LLA5.U60W,Long,0.3610,0.3613,0.999,0.2,0
|
|
||||||
03a10a6a-193a-476c-9543-f63818f69417,BE9558,2026-05-16T08:54:36,K558LLA6.300W,Long,0.3515,0.3516,1.000,0.2,0
|
|
||||||
1c271856-8100-4c0d-bc8c-9357b47d7626,BE9558,2026-05-16T08:59:54,K558LLA6.BU0W,Long,0.3617,0.3662,0.988,0.2,0
|
|
||||||
a85f125e-bfa2-42be-b877-2604b8d068c8,BE9558,2026-05-16T09:05:17,K558LLA6.KT0W,Long,0.3510,0.3516,0.998,0.2,0
|
|
||||||
dde9aa71-d2e3-41ac-8388-f2bbddb6dae6,BE9558,2026-05-16T09:08:11,K558LLA6.PN0W,Long,0.3447,0.3467,0.994,0.2,0
|
|
||||||
2f3f2be2-0ee3-422e-b3ad-83fb87512a2b,BE9558,2026-05-16T09:14:39,K558LLA7.0F0W,Long,0.3580,0.3613,0.991,0.2,0
|
|
||||||
357a3034-8a9c-485e-8cb6-f526dd2baf6f,BE9558,2026-05-16T09:29:11,K558LLA7.ON0W,Long,0.3708,0.3760,0.986,0.2,0
|
|
||||||
f8ad0274-8e6f-4714-82ff-d229e60f1442,BE9558,2026-05-16T09:32:06,K558LLA7.TI0W,Long,0.3567,0.3613,0.987,0.2,0
|
|
||||||
42bb007a-2726-401a-b085-c25151b0dc24,BE9558,2026-05-16T09:35:02,K558LLA7.YE0W,Long,0.3575,0.3613,0.989,0.2,0
|
|
||||||
c99bfc4a-db02-4bea-bea4-943c67485dba,BE9558,2026-05-16T09:37:58,K558LLA8.3A0W,Long,0.3527,0.3564,0.989,0.2,0
|
|
||||||
8fc17de0-73f7-48bd-8d93-5451dce47e8e,BE9558,2026-05-16T09:40:52,K558LLA8.840W,Long,0.3613,0.3662,0.987,0.2,0
|
|
||||||
bbee5276-03dd-4ca4-a7b9-aec02baac7e8,BE9558,2026-05-16T09:48:13,K558LLA8.KD0W,Long,0.3590,0.3613,0.993,0.2,0
|
|
||||||
8dcdee10-db01-4224-9506-4181c3105d6f,BE9558,2026-05-16T09:53:31,K558LLA8.T70W,Long,0.3588,0.3613,0.993,0.2,0
|
|
||||||
48c8be72-64d9-4da3-95b5-bbec2c5e1dbd,BE9558,2026-05-16T09:58:49,K558LLA9.210W,Long,0.3582,0.3662,0.978,0.2,0
|
|
||||||
0da20919-7ce9-4feb-8ae9-12d9bb40c2b5,BE9558,2026-05-16T10:04:07,K558LLA9.AV0W,Long,0.3506,0.3516,0.997,0.2,0
|
|
||||||
c2f7e9bc-8b93-42c3-96d3-e3af217dde35,BE9558,2026-05-16T10:09:25,K558LLA9.JP0W,Long,0.3506,0.3516,0.997,0.2,0
|
|
||||||
522a0e42-d443-4447-bc71-af1832c6604e,BE9558,2026-05-16T10:14:43,K558LLA9.SJ0W,Long,0.3624,0.3662,0.990,0.2,0
|
|
||||||
67f7e732-3c0b-439f-87c5-3c478a381418,BE9558,2026-05-16T10:20:01,K558LLAA.1D0W,Long,0.3494,0.3516,0.994,0.2,0
|
|
||||||
251767e8-56f7-4091-a001-364d8e75c15c,BE9558,2026-05-16T10:25:19,K558LLAA.A70W,Long,0.3463,0.3516,0.985,0.2,0
|
|
||||||
9457239d-a4ca-4476-b385-5e01fc8f6420,BE9558,2026-05-16T10:30:37,K558LLAA.J10W,Long,0.3436,0.3467,0.991,0.2,0
|
|
||||||
43718d7c-9353-49e1-89b6-0841aeb1b276,BE9558,2026-05-16T10:35:55,K558LLAA.RV0W,Long,0.3452,0.3516,0.982,0.2,0
|
|
||||||
f54ba557-ddfe-41d3-8f4c-8bbec9a3783c,BE9558,2026-05-16T10:41:13,K558LLAB.0P0W,Long,0.3419,0.3467,0.986,0.2,0
|
|
||||||
dc26c758-ad99-49c1-b518-931368cdf8cf,BE9558,2026-05-16T10:46:31,K558LLAB.9J0W,Long,0.3463,0.3516,0.985,0.2,0
|
|
||||||
10917eb2-d151-461c-a8cb-e9660da5a5b0,BE9558,2026-05-16T10:51:49,K558LLAB.ID0W,Long,0.3520,0.3564,0.987,0.2,0
|
|
||||||
4938a5e6-d53b-40ee-be2b-06ec0e60398c,BE9558,2026-05-16T10:57:07,K558LLAB.R70W,Long,0.3541,0.3564,0.993,0.2,0
|
|
||||||
0711b51d-55bd-48be-b51e-2aad67107a60,BE9558,2026-05-16T11:13:01,K558LLAC.HP0W,Long,0.3661,0.3711,0.986,0.2,0
|
|
||||||
065564b6-aa49-40e7-9ea6-c75defdc60fe,BE9558,2026-05-16T11:18:19,K558LLAC.QJ0W,Long,0.3622,0.3662,0.989,0.2,0
|
|
||||||
ffb6b230-1d70-4e0e-ba30-e945339a122a,BE9558,2026-05-16T11:23:37,K558LLAC.ZD0W,Long,0.3711,0.3760,0.987,0.2,0
|
|
||||||
52b4da82-c1e0-4a8a-8365-51b090787005,BE9558,2026-05-16T11:28:55,K558LLAD.870W,Long,0.3644,0.3662,0.995,0.2,0
|
|
||||||
cdbff3d8-fbfe-4bea-8bd6-c898800e81ba,BE9558,2026-05-16T11:39:31,K558LLAD.PV0W,Long,0.3609,0.3662,0.986,0.2,0
|
|
||||||
3fb0bc30-1463-4cfc-9e7f-386c0eb2a34a,BE9558,2026-05-16T11:44:49,K558LLAD.YP0W,Long,0.3553,0.3564,0.997,0.2,0
|
|
||||||
ba1f5ec8-aef7-4a9e-ac1f-276cf016e285,BE9558,2026-05-16T11:50:07,K558LLAE.7J0W,Long,0.3479,0.3516,0.990,0.2,0
|
|
||||||
b1dd4df7-34a9-4512-a903-f8242b236fd2,BE9558,2026-05-16T11:55:25,K558LLAE.GD0W,Long,0.3427,0.3467,0.988,0.2,0
|
|
||||||
1f23aa28-c026-4bc7-b3d1-7c78fd76a673,BE9558,2026-05-16T12:00:43,K558LLAE.P70W,Long,0.3454,0.3467,0.996,0.2,0
|
|
||||||
0d91f54d-b20d-4a50-b7dd-d071c39a0690,BE9558,2026-05-16T12:06:01,K558LLAE.Y10W,Long,0.3431,0.3467,0.990,0.2,0
|
|
||||||
12ccdecd-5fd9-4465-9813-3d6006ec32f3,BE9558,2026-05-16T12:11:19,K558LLAF.6V0W,Long,0.3433,0.3467,0.990,0.2,0
|
|
||||||
a7024f1a-f071-48ab-820f-99bdf87366b5,BE9558,2026-05-16T12:16:37,K558LLAF.FP0W,Long,0.3361,0.3369,0.998,0.2,0
|
|
||||||
de0fa439-30b8-4607-a3c3-7e17191e4624,BE9558,2026-05-16T12:21:55,K558LLAF.OJ0W,Long,0.3274,0.3320,0.986,0.2,0
|
|
||||||
f9374ec3-05e5-477f-8e1a-0c862fed2a50,BE9558,2026-05-16T12:27:13,K558LLAF.XD0W,Long,0.3449,0.3467,0.995,0.2,0
|
|
||||||
424cb275-3743-4174-95f0-0ca1205b374c,BE9558,2026-05-16T12:32:31,K558LLAG.670W,Long,0.3370,0.3418,0.986,0.2,0
|
|
||||||
9accde13-cefa-41b8-bd0a-cef2ffed59b0,BE9558,2026-05-16T12:37:49,K558LLAG.F10W,Long,0.3465,0.3467,0.999,0.2,0
|
|
||||||
4665be38-1956-4b14-b931-068ba9af5c5e,BE9558,2026-05-16T12:48:25,K558LLAG.WP0W,Long,0.3453,0.3516,0.982,0.2,0
|
|
||||||
1aedf624-1a99-43e1-a8f2-f9d358b97f61,BE9558,2026-05-16T12:53:43,K558LLAH.5J0W,Long,0.3366,0.3418,0.985,0.2,0
|
|
||||||
c15e1799-91d7-425e-ad17-2ba3d170d81e,BE9558,2026-05-16T13:04:19,K558LLAH.N70W,Long,0.3348,0.3369,0.994,0.2,0
|
|
||||||
72d4c4ea-19ca-4cc6-9b04-7a2b70122c67,BE9558,2026-05-16T13:09:37,K558LLAH.W10W,Long,0.3272,0.3320,0.986,0.2,0
|
|
||||||
b4ebf2c9-1f6d-4594-9810-40479e87c0a5,BE9558,2026-05-16T13:14:55,K558LLAI.4V0W,Long,0.3297,0.3320,0.993,0.2,0
|
|
||||||
bb297e9e-548f-4dae-8b1a-0f161af2e766,BE9558,2026-05-16T13:20:13,K558LLAI.DP0W,Long,0.3272,0.3320,0.986,0.2,0
|
|
||||||
87562c24-63db-48bd-9962-72a20a40f2ff,BE9558,2026-05-16T13:36:07,K558LLAJ.470W,Long,0.3174,0.3223,0.985,0.2,0
|
|
||||||
01e9cc58-3bea-4f38-8499-a2c20e5f8550,BE9558,2026-05-16T13:46:52,K558LLAJ.M40W,Long,0.2880,0.2930,0.983,0.2,0
|
|
||||||
5ae7503d-ae17-4063-9208-6a420699e1ab,BE9558,2026-05-16T14:00:35,K558LLAK.8Z0W,Long,0.2879,0.2930,0.983,0.2,0
|
|
||||||
e009ea3c-1534-4154-8623-e180ed4db9fb,BE9558,2026-05-16T14:06:19,K558LLAK.IJ0W,Long,0.2882,0.2930,0.984,0.2,0
|
|
||||||
cd4e21f5-512b-4b2b-8054-bec01aa06400,BE9558,2026-05-16T14:12:03,K558LLAK.S30W,Long,0.2857,0.2930,0.975,0.2,0
|
|
||||||
ae5b780f-2038-4ccf-8bfc-b15b87be139e,BE9558,2026-05-16T14:17:22,K558LLAL.0Y0W,Long,0.2899,0.2930,0.989,0.2,0
|
|
||||||
d8c3fb81-8c66-4ebc-b33a-0559bc03334b,BE9558,2026-05-16T14:22:40,K558LLAL.9S0W,Long,0.3009,0.3027,0.994,0.2,0
|
|
||||||
990a4c7a-3c51-4d4a-8ab7-416ccb97d29c,BE9558,2026-05-16T14:27:58,K558LLAL.IM0W,Long,0.3057,0.3076,0.994,0.2,0
|
|
||||||
68d7a2c3-d899-4e75-96fb-3bf16ae6ea6f,BE9558,2026-05-16T14:33:16,K558LLAL.RG0W,Long,0.3110,0.3125,0.995,0.2,0
|
|
||||||
73505b8c-f912-4116-b903-a8a4c75524d3,BE9558,2026-05-16T14:38:34,K558LLAM.0A0W,Long,0.3172,0.3223,0.984,0.2,0
|
|
||||||
8d3c9602-7efa-488a-801b-3228bd85ab14,BE9558,2026-05-16T14:43:51,K558LLAM.930W,Long,0.3136,0.3174,0.988,0.2,0
|
|
||||||
243e16ee-860a-4960-bc15-b797ad7e9735,BE9558,2026-05-16T14:49:08,K558LLAM.HW0W,Long,0.3135,0.3174,0.988,0.2,0
|
|
||||||
e90c8d90-6d41-4788-9ee6-a29e4bfcc16b,BE9558,2026-05-16T15:19:35,K558LLAN.WN0W,Long,0.3132,0.3174,0.987,0.2,0
|
|
||||||
4182b0e7-7811-46b8-87d1-a4dd0c152f3b,BE9558,2026-05-16T15:28:45,K558LLAO.BX0W,Long,0.3023,0.3076,0.983,0.2,0
|
|
||||||
28a790da-0e48-4fe4-b8ba-2c82cc6bfa79,BE9558,2026-05-16T15:34:02,K558LLAO.KQ0W,Long,0.3176,0.3223,0.985,0.2,0
|
|
||||||
093d2fa3-7a4b-4f0f-abda-71155ebe2dfb,BE9558,2026-05-16T15:39:19,K558LLAO.TJ0W,Long,0.3223,0.3271,0.985,0.2,0
|
|
||||||
68dc9560-4082-49b6-af04-9244773ffbc1,BE9558,2026-05-16T15:44:36,K558LLAP.2C0W,Long,0.3227,0.3271,0.986,0.2,0
|
|
||||||
ad6b268a-165d-4b1d-b0b3-f58db7c9b0e4,BE9558,2026-05-16T15:49:53,K558LLAP.B50W,Long,0.3177,0.3223,0.986,0.2,0
|
|
||||||
046c5cb4-56b3-48c0-83c2-bf1f30837940,BE9558,2026-05-16T15:55:10,K558LLAP.JY0W,Long,0.3174,0.3223,0.985,0.2,0
|
|
||||||
1b76edf4-3b1c-4bcb-a73c-27334981c350,BE9558,2026-05-16T16:00:27,K558LLAP.SR0W,Long,0.3127,0.3174,0.985,0.2,0
|
|
||||||
a0264171-a682-4396-ad8a-cda3b2964533,BE9558,2026-05-16T16:06:55,K558LLAQ.3J0W,Long,0.2885,0.2930,0.985,0.2,0
|
|
||||||
707da1f6-ee81-4e57-abd5-98295c14651b,BE9558,2026-05-16T16:12:30,K558LLAQ.CU0W,Long,0.2880,0.2930,0.983,0.2,0
|
|
||||||
b8115b64-8739-42c8-b541-713ab6b68bcd,BE9558,2026-05-16T16:17:53,K558LLAQ.LT0W,Long,0.2881,0.2930,0.983,0.2,0
|
|
||||||
fb179c36-fef1-4df6-b0a0-2bd7792ea910,BE9558,2026-05-16T16:23:11,K558LLAQ.UN0W,Long,0.2933,0.2979,0.985,0.2,0
|
|
||||||
0cec8793-c3c1-486b-a33a-4dbf9cb069b0,BE9558,2026-05-16T16:33:46,K558LLAR.CA0W,Long,0.3017,0.3027,0.997,0.2,0
|
|
||||||
c531b052-ee04-4c66-888b-21e97445a615,BE9558,2026-05-16T16:49:37,K558LLAS.2P0W,Long,0.2922,0.2979,0.981,0.2,0
|
|
||||||
0dc402f9-cab6-4c4a-8317-b8fc3c4b1ced,BE9558,2026-05-16T16:55:08,K558LLAS.BW0W,Long,0.2898,0.2930,0.989,0.2,0
|
|
||||||
fd73733e-b42d-4a51-81d0-4acad6edfbd4,BE9558,2026-05-16T17:16:17,K558LLAT.B50W,Long,0.3115,0.3125,0.997,0.2,0
|
|
||||||
6a254daa-ae27-4ac1-b334-58f776a6b276,BE9558,2026-05-16T17:26:51,K558LLAT.SR0W,Long,0.3174,0.3223,0.985,0.2,0
|
|
||||||
0cc8e82f-5270-4508-beae-0d3af56095ca,BE9558,2026-05-16T17:37:27,K558LLAU.AF0W,Long,0.2899,0.2930,0.989,0.2,0
|
|
||||||
00557a73-8dc3-4cd2-b880-c7820dc8b171,BE9558,2026-05-16T17:43:17,K558LLAU.K50W,Long,0.2881,0.2930,0.983,0.2,0
|
|
||||||
8e5518d9-ff0a-4c09-9cf9-a681c84c0b6e,BE9558,2026-05-16T17:53:53,K558LLAV.1T0W,Long,0.2898,0.2930,0.989,0.2,0
|
|
||||||
91dbd7e9-7344-400b-82b1-379f2ac4b49a,BE9558,2026-05-16T17:59:11,K558LLAV.AN0W,Long,0.2925,0.2930,0.998,0.2,0
|
|
||||||
fa256408-d480-4376-95b5-d1a549c483ea,BE9558,2026-05-16T18:04:29,K558LLAV.JH0W,Long,0.2977,0.3027,0.983,0.2,0
|
|
||||||
4ab6e66a-8364-49b6-82f1-86e86199e676,BE9558,2026-05-16T18:09:46,K558LLAV.SA0W,Long,0.2944,0.2979,0.988,0.2,0
|
|
||||||
0a65fb7c-96e1-4ba0-85de-4620d88658c1,BE9558,2026-05-16T18:15:03,K558LLAW.130W,Long,0.2965,0.2979,0.995,0.2,0
|
|
||||||
92b1ef3e-1696-4123-8662-163947737c83,BE9558,2026-05-16T18:20:20,K558LLAW.9W0W,Long,0.2976,0.3027,0.983,0.2,0
|
|
||||||
9e3cb24c-4426-4220-9a73-389d0a96c94f,BE9558,2026-05-16T18:25:37,K558LLAW.IP0W,Long,0.2983,0.3027,0.985,0.2,0
|
|
||||||
a3473215-4c46-45ab-9aa7-1720486ec4a8,BE9558,2026-05-16T18:30:54,K558LLAW.RI0W,Long,0.2956,0.2979,0.993,0.2,0
|
|
||||||
449665b1-47c8-4bd2-984c-8f45b627e6d8,BE9558,2026-05-16T18:36:11,K558LLAX.0B0W,Long,0.2989,0.3027,0.987,0.2,0
|
|
||||||
600cfabc-cffd-42e1-800c-70e75f312316,BE9558,2026-05-16T20:00:12,K558LLB0.WC0W,Long,0.3077,0.3125,0.985,0.2,0
|
|
||||||
12bf5944-0194-4f5b-b486-a7acd7141c7f,BE9558,2026-05-16T20:04:28,K558LLB1.3G0W,Long,0.2888,0.2930,0.986,0.2,0
|
|
||||||
f1fc913a-91bc-4240-a595-7dfe3091f9a2,BE9558,2026-05-16T20:07:29,K558LLB1.8H0W,Long,0.2977,0.3027,0.983,0.2,0
|
|
||||||
e74f7ec3-20bb-4177-beb2-e802e5f801fd,BE9558,2026-05-16T20:10:26,K558LLB1.DE0W,Long,0.2978,0.3027,0.984,0.2,0
|
|
||||||
27491c3a-b631-4643-818a-714523911178,BE9558,2026-05-16T20:16:21,K558LLB1.N90W,Long,0.2921,0.2930,0.997,0.2,0
|
|
||||||
9d9196ce-11e2-4f28-b01c-3dcc60e60d42,BE9558,2026-05-16T20:22:28,K558LLB1.XG0W,Long,0.2881,0.2930,0.983,0.2,0
|
|
||||||
a592d7c8-b643-4b25-9186-3d611ae22709,BE9558,2026-05-16T20:25:37,K558LLB2.2P0W,Long,0.2893,0.2930,0.987,0.2,0
|
|
||||||
f709a3d5-27cd-4bb1-b91c-669e8ea11d77,BE9558,2026-05-16T20:28:57,K558LLB2.890W,Long,0.2880,0.2930,0.983,0.2,0
|
|
||||||
cc9cad87-b87f-4b67-854a-ffb21e876a92,BE9558,2026-05-16T20:32:45,K558LLB2.EL0W,Long,0.2873,0.2930,0.980,0.2,0
|
|
||||||
10732d26-6b49-4b14-a474-6c75e259d278,BE9558,2026-05-16T20:49:32,K558LLB3.6K0W,Long,0.2881,0.2930,0.983,0.2,0
|
|
||||||
aa63957a-fdb2-4f17-af52-a63bb015119d,BE9558,2026-05-16T21:21:45,K558LLB4.O90W,Long,0.2880,0.2930,0.983,0.2,0
|
|
||||||
8350440f-019d-45a4-b9f9-23058628e6f0,BE9558,2026-05-16T21:32:43,K558LLB5.6J0W,Long,0.2925,0.2930,0.999,0.2,0
|
|
||||||
edfc10b2-dbe1-4527-a061-b0d670b5e5d2,BE9558,2026-05-16T21:35:37,K558LLB5.BD0W,Long,0.2891,0.2930,0.987,0.2,0
|
|
||||||
f98001a3-88b2-4830-8112-b7c5164e99c3,BE9558,2026-05-16T21:38:40,K558LLB5.GG0W,Long,0.2877,0.2930,0.982,0.2,0
|
|
||||||
f1e53e7b-b124-4fc5-82c2-504fb0c8b80d,BE9558,2026-05-16T21:41:46,K558LLB5.LM0W,Long,0.2881,0.2930,0.983,0.2,0
|
|
||||||
7890637f-142e-4775-b48c-fd4f3b8ada77,BE9558,2026-05-16T21:45:14,K558LLB5.RE0W,Long,0.2880,0.2930,0.983,0.2,0
|
|
||||||
7c97c10f-b413-467c-9b56-c18e423cdcfa,BE9558,2026-05-16T21:48:26,K558LLB5.WQ0W,Long,0.2886,0.2930,0.985,0.2,0
|
|
||||||
8178cf22-8428-4f85-8199-0a43f7334a8f,BE9558,2026-05-16T21:51:47,K558LLB6.2B0W,Long,0.2881,0.2930,0.983,0.2,0
|
|
||||||
eceb2997-deb4-403a-802f-50e93d23d423,BE9558,2026-05-16T21:57:18,K558LLB6.BI0W,Long,0.2881,0.2930,0.984,0.2,0
|
|
||||||
9c26690c-1983-4b21-b8af-7c7967c014c3,BE9558,2026-05-16T22:12:21,K558LLB7.0L0W,Long,0.2876,0.2930,0.982,0.2,0
|
|
||||||
78f67fc3-08b7-4317-96ba-ee5544e0b95b,BE9558,2026-05-16T22:19:34,K558LLB7.CM0W,Long,0.2880,0.2930,0.983,0.2,0
|
|
||||||
7f9dad14-4f1b-492d-aa26-845a908ea894,BE9558,2026-05-16T22:27:40,K558LLB7.Q40W,Long,0.2881,0.2930,0.983,0.2,0
|
|
||||||
f75c52b4-21b9-4065-baeb-5cfbd27d130a,BE9558,2026-05-16T22:33:50,K558LLB8.0E0W,Long,0.2881,0.2930,0.984,0.2,0
|
|
||||||
10f7496d-fa76-40a7-8bef-dc14e8a41869,BE9558,2026-05-16T22:40:48,K558LLB8.C00W,Long,0.2878,0.2930,0.982,0.2,0
|
|
||||||
|
@@ -1,171 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Scan series-3 waveform binaries for the 'offset' hardware fault.
|
|
||||||
|
|
||||||
A healthy geophone trace is centred on zero. An offset unit sits displaced,
|
|
||||||
so the channel mean approaches its own peak. Detector (unchanged from the
|
|
||||||
2026-08-25 run, see memory note `offset-archive-analysis-backlog`):
|
|
||||||
|
|
||||||
dominant-axis |mean| / peak > 0.7
|
|
||||||
AND |mean| >= 0.9 * the unit's geo trigger level
|
|
||||||
|
|
||||||
Trigger level is read from a paired _ASCII.TXT where one exists, otherwise
|
|
||||||
from a per-serial median learned across that unit's ASCII files, otherwise
|
|
||||||
--default-trigger.
|
|
||||||
|
|
||||||
Serial is decoded from the BW filename: prefix letter encodes thousands
|
|
||||||
(chr(ord('B') + n)), next 3 digits the remainder -- T193 -> BE18193.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python scratch/offset_scan.py --dir <path> [--jobs N] --out offsets.csv
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse, csv, json, re, sys
|
|
||||||
from collections import defaultdict
|
|
||||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
||||||
|
|
||||||
from minimateplus.event_file_io import read_blastware_file
|
|
||||||
from minimateplus.bw_ascii_report import parse_report
|
|
||||||
|
|
||||||
GEO = ("Tran", "Vert", "Long")
|
|
||||||
_GEO_FS_COUNTS = 32000.0
|
|
||||||
_WAVE_RE = re.compile(r"\.[A-Za-z0-9]{2}0[Ww]$")
|
|
||||||
_STEM_RE = re.compile(r"^([B-Z])(\d{3})")
|
|
||||||
|
|
||||||
MEAN_OVER_PEAK_MIN = 0.7
|
|
||||||
TRIGGER_FRACTION = 0.9
|
|
||||||
|
|
||||||
|
|
||||||
def serial_from_name(name: str):
|
|
||||||
m = _STEM_RE.match(name)
|
|
||||||
if not m:
|
|
||||||
return None
|
|
||||||
letter, digits = m.group(1), m.group(2)
|
|
||||||
return f"BE{(ord(letter) - ord('B')) * 1000 + int(digits)}"
|
|
||||||
|
|
||||||
|
|
||||||
def counts_to_ips(c, gr):
|
|
||||||
return c * (gr or 10.0) / _GEO_FS_COUNTS
|
|
||||||
|
|
||||||
|
|
||||||
def scan_one(path_str: str, default_trigger: float) -> dict | None:
|
|
||||||
p = Path(path_str)
|
|
||||||
try:
|
|
||||||
gr, trig = 10.0, None
|
|
||||||
ap = p.with_name(p.name.replace(".", "_", 1) + "_ASCII.TXT") \
|
|
||||||
if False else p.parent / (p.stem + "_" + p.suffix.lstrip(".") + "_ASCII.TXT")
|
|
||||||
if ap.exists():
|
|
||||||
rep = parse_report(ap.read_text(errors="replace"))
|
|
||||||
gr = rep.geo_range_ips or 10.0
|
|
||||||
trig = rep.geo_trigger_level_ips
|
|
||||||
ev = read_blastware_file(p)
|
|
||||||
s = ev.raw_samples or {}
|
|
||||||
if not all(s.get(c) for c in GEO):
|
|
||||||
return None
|
|
||||||
best = None
|
|
||||||
for ch in GEO:
|
|
||||||
arr = s[ch]
|
|
||||||
n = len(arr)
|
|
||||||
if n == 0:
|
|
||||||
continue
|
|
||||||
mean = sum(arr) / n
|
|
||||||
peak = max(abs(v) for v in arr)
|
|
||||||
if peak == 0:
|
|
||||||
continue
|
|
||||||
ratio = abs(mean) / peak
|
|
||||||
if best is None or peak > best["peak_counts"]:
|
|
||||||
best = {"channel": ch, "mean_counts": mean,
|
|
||||||
"peak_counts": peak, "ratio": ratio}
|
|
||||||
if best is None:
|
|
||||||
return None
|
|
||||||
ts = ev.timestamp
|
|
||||||
return {
|
|
||||||
"serial": serial_from_name(p.name) or "?",
|
|
||||||
"timestamp": (f"{ts.year:04d}-{ts.month:02d}-{ts.day:02d}T"
|
|
||||||
f"{ts.hour:02d}:{ts.minute:02d}:{ts.second:02d}") if ts else "",
|
|
||||||
"filename": p.name,
|
|
||||||
"channel": best["channel"],
|
|
||||||
"offset_ips": round(counts_to_ips(best["mean_counts"], gr), 4),
|
|
||||||
"peak_ips": round(counts_to_ips(best["peak_counts"], gr), 4),
|
|
||||||
"mean_over_peak": round(best["ratio"], 3),
|
|
||||||
"trigger_level_ips": trig if trig is not None else "",
|
|
||||||
"geo_range_ips": gr,
|
|
||||||
}
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--dir", required=True)
|
|
||||||
ap.add_argument("--jobs", type=int, default=4)
|
|
||||||
ap.add_argument("--limit", type=int, default=0)
|
|
||||||
ap.add_argument("--default-trigger", type=float, default=0.2)
|
|
||||||
ap.add_argument("--out", required=True)
|
|
||||||
a = ap.parse_args()
|
|
||||||
|
|
||||||
# The DL2 export keeps a byte-identical `Sent/` mirror of the root, so
|
|
||||||
# enumerate paths but keep only the first occurrence of each basename —
|
|
||||||
# otherwise every event is counted twice.
|
|
||||||
seen = set()
|
|
||||||
files = []
|
|
||||||
for q in sorted(Path(a.dir).rglob("*")):
|
|
||||||
if q.is_file() and _WAVE_RE.search(q.name) and q.name not in seen:
|
|
||||||
seen.add(q.name)
|
|
||||||
files.append(q)
|
|
||||||
if a.limit:
|
|
||||||
files = files[: a.limit]
|
|
||||||
print(f"waveform binaries to scan: {len(files)}", flush=True)
|
|
||||||
|
|
||||||
rows = []
|
|
||||||
with ProcessPoolExecutor(max_workers=a.jobs) as ex:
|
|
||||||
futs = [ex.submit(scan_one, str(p), a.default_trigger) for p in files]
|
|
||||||
for n, f in enumerate(as_completed(futs), 1):
|
|
||||||
r = f.result()
|
|
||||||
if r:
|
|
||||||
rows.append(r)
|
|
||||||
if n % 2000 == 0:
|
|
||||||
print(f" {n}/{len(files)}", flush=True)
|
|
||||||
|
|
||||||
# learn per-serial trigger levels from the rows that had an ASCII
|
|
||||||
by_serial = defaultdict(list)
|
|
||||||
for r in rows:
|
|
||||||
if r["trigger_level_ips"] != "":
|
|
||||||
by_serial[r["serial"]].append(float(r["trigger_level_ips"]))
|
|
||||||
med = {}
|
|
||||||
for k, v in by_serial.items():
|
|
||||||
v.sort()
|
|
||||||
med[k] = v[len(v) // 2]
|
|
||||||
|
|
||||||
for r in rows:
|
|
||||||
if r["trigger_level_ips"] == "":
|
|
||||||
r["trigger_level_ips"] = med.get(r["serial"], a.default_trigger)
|
|
||||||
r["suspect"] = int(
|
|
||||||
r["mean_over_peak"] > MEAN_OVER_PEAK_MIN
|
|
||||||
and abs(r["offset_ips"]) >= TRIGGER_FRACTION * float(r["trigger_level_ips"])
|
|
||||||
)
|
|
||||||
|
|
||||||
cols = ["serial", "timestamp", "filename", "channel", "offset_ips", "peak_ips",
|
|
||||||
"mean_over_peak", "trigger_level_ips", "geo_range_ips", "suspect"]
|
|
||||||
with open(a.out, "w", newline="") as fh:
|
|
||||||
w = csv.DictWriter(fh, fieldnames=cols)
|
|
||||||
w.writeheader()
|
|
||||||
w.writerows(rows)
|
|
||||||
|
|
||||||
sus = [r for r in rows if r["suspect"]]
|
|
||||||
print(f"\nscanned {len(rows)} decodable waveforms")
|
|
||||||
print(f"suspect events: {len(sus)}")
|
|
||||||
per = defaultdict(int)
|
|
||||||
for r in sus:
|
|
||||||
per[r["serial"]] += 1
|
|
||||||
print(f"units with >=1 suspect event: {len(per)} of {len({r['serial'] for r in rows})}")
|
|
||||||
for s, n in sorted(per.items(), key=lambda x: -x[1])[:20]:
|
|
||||||
print(f" {s:10} {n}")
|
|
||||||
print(f"\nwrote {a.out}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,108 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Offset detector v2 — per-channel MEDIAN pedestal.
|
|
||||||
|
|
||||||
Supersedes the dominant-axis / mean detector in offset_scan.py, which had two
|
|
||||||
flaws that manufactured false "recoveries":
|
|
||||||
|
|
||||||
1. It scored only the axis with the largest peak, so a real event on one axis
|
|
||||||
hid a persistent pedestal on another. BE12599 2026-08-21 read "clean"
|
|
||||||
because Long had a 1.065 in/s event, while Tran sat at +0.47 in/s.
|
|
||||||
2. It used the MEAN, which a real transient perturbs. The median is the
|
|
||||||
resting baseline: most samples sit at it, so a blast does not move it.
|
|
||||||
Same event, Long: mean +0.0783 vs median -0.0050.
|
|
||||||
|
|
||||||
Flags a CHANNEL when |median| >= --floor in/s (default 0.025 = 5 A/D counts,
|
|
||||||
Instantel's own criterion; 1 A/D count = 0.005 in/s).
|
|
||||||
|
|
||||||
Emits one row per (event, channel) so persistence can be tracked per channel.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
import argparse, csv, re, statistics, sys
|
|
||||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
||||||
from minimateplus.event_file_io import read_blastware_file
|
|
||||||
|
|
||||||
GEO = ("Tran", "Vert", "Long")
|
|
||||||
K = 10.0 / 32000.0 # ADC counts -> in/s at the 10 in/s range
|
|
||||||
_WAVE_RE = re.compile(r"\.[A-Za-z0-9]{2}0[Ww]$")
|
|
||||||
_STEM_RE = re.compile(r"^([B-Z])(\d{3})")
|
|
||||||
|
|
||||||
|
|
||||||
def serial_from_name(n):
|
|
||||||
m = _STEM_RE.match(n)
|
|
||||||
return f"BE{(ord(m.group(1))-ord('B'))*1000+int(m.group(2))}" if m else "?"
|
|
||||||
|
|
||||||
|
|
||||||
def scan_one(ps):
|
|
||||||
p = Path(ps)
|
|
||||||
try:
|
|
||||||
ev = read_blastware_file(p)
|
|
||||||
s = ev.raw_samples or {}
|
|
||||||
if not all(s.get(c) for c in GEO):
|
|
||||||
return None
|
|
||||||
ts = ev.timestamp
|
|
||||||
stamp = (f"{ts.year:04d}-{ts.month:02d}-{ts.day:02d}T"
|
|
||||||
f"{ts.hour:02d}:{ts.minute:02d}:{ts.second:02d}") if ts else ""
|
|
||||||
out = []
|
|
||||||
for ch in GEO:
|
|
||||||
a = s[ch]
|
|
||||||
out.append({
|
|
||||||
"serial": serial_from_name(p.name), "timestamp": stamp,
|
|
||||||
"filename": p.name, "channel": ch,
|
|
||||||
"median_ips": round(statistics.median(a) * K, 4),
|
|
||||||
"mean_ips": round(statistics.fmean(a) * K, 4),
|
|
||||||
"peak_ips": round(max(abs(v) for v in a) * K, 4),
|
|
||||||
})
|
|
||||||
return out
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--dir", required=True)
|
|
||||||
ap.add_argument("--jobs", type=int, default=4)
|
|
||||||
ap.add_argument("--floor", type=float, default=0.025)
|
|
||||||
ap.add_argument("--out", required=True)
|
|
||||||
a = ap.parse_args()
|
|
||||||
|
|
||||||
seen, files = set(), []
|
|
||||||
for q in sorted(Path(a.dir).rglob("*")):
|
|
||||||
if q.is_file() and _WAVE_RE.search(q.name) and q.name not in seen:
|
|
||||||
seen.add(q.name); files.append(str(q))
|
|
||||||
print(f"unique waveform binaries: {len(files)}", flush=True)
|
|
||||||
|
|
||||||
rows = []
|
|
||||||
with ProcessPoolExecutor(max_workers=a.jobs) as ex:
|
|
||||||
for n, f in enumerate(as_completed([ex.submit(scan_one, p) for p in files]), 1):
|
|
||||||
r = f.result()
|
|
||||||
if r: rows.extend(r)
|
|
||||||
if n % 2000 == 0: print(f" {n}/{len(files)}", flush=True)
|
|
||||||
|
|
||||||
for r in rows:
|
|
||||||
r["offset"] = int(abs(r["median_ips"]) >= a.floor)
|
|
||||||
|
|
||||||
cols = ["serial","timestamp","filename","channel","median_ips","mean_ips","peak_ips","offset"]
|
|
||||||
with open(a.out, "w", newline="") as fh:
|
|
||||||
w = csv.DictWriter(fh, fieldnames=cols); w.writeheader(); w.writerows(rows)
|
|
||||||
|
|
||||||
from collections import defaultdict
|
|
||||||
ev_flagged = {(r["serial"], r["filename"]) for r in rows if r["offset"]}
|
|
||||||
ev_all = {(r["serial"], r["filename"]) for r in rows}
|
|
||||||
per = defaultdict(set)
|
|
||||||
for r in rows:
|
|
||||||
if r["offset"]: per[r["serial"]].add(r["filename"])
|
|
||||||
tot = defaultdict(set)
|
|
||||||
for r in rows: tot[r["serial"]].add(r["filename"])
|
|
||||||
print(f"\nfloor = {a.floor} in/s ({a.floor/0.005:.0f} A/D counts)")
|
|
||||||
print(f"events with >=1 offset channel: {len(ev_flagged)} of {len(ev_all)}")
|
|
||||||
print(f"units affected: {len(per)} of {len(tot)}")
|
|
||||||
for s in sorted(per, key=lambda s: -len(per[s])):
|
|
||||||
print(f" {s:9} {len(per[s]):4} / {len(tot[s]):4} events")
|
|
||||||
print(f"\nwrote {a.out}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,95 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Offset detector v3 — pre-trigger floor, with pre/mid/end consistency.
|
|
||||||
|
|
||||||
Brian's method, and better than v2's whole-record median for one reason: the
|
|
||||||
pre-trigger window is *definitionally* quiet (it is the buffer captured before
|
|
||||||
the trigger fired), whereas a whole-record median is merely robust to the event.
|
|
||||||
|
|
||||||
Per channel:
|
|
||||||
pre = median of the first `pretrig_samples` samples (STRT record)
|
|
||||||
mid = median of the middle third
|
|
||||||
end = median of the final third
|
|
||||||
spread = max(pre,mid,end) - min(pre,mid,end)
|
|
||||||
|
|
||||||
A DC offset is a *constant floor*: |pre| at or above the floor AND a small
|
|
||||||
spread. A transient (settling, handling, a long-tailed event) moves one segment
|
|
||||||
relative to the others and is rejected by the spread test.
|
|
||||||
|
|
||||||
Floor default 0.025 in/s = 5 A/D counts (Instantel's own criterion; 1 count =
|
|
||||||
0.005 in/s). Quantisation is 0.005 in/s, so `spread` is measured in units of it.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
import argparse, csv, re, statistics, sys
|
|
||||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
|
||||||
from pathlib import Path
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
||||||
from minimateplus.event_file_io import read_blastware_file
|
|
||||||
|
|
||||||
GEO=("Tran","Vert","Long"); K=10.0/32000.0
|
|
||||||
_WAVE=re.compile(r"\.[A-Za-z0-9]{2}0[Ww]$"); _STEM=re.compile(r"^([B-Z])(\d{3})")
|
|
||||||
|
|
||||||
def serial_of(n):
|
|
||||||
m=_STEM.match(n)
|
|
||||||
return f"BE{(ord(m.group(1))-ord('B'))*1000+int(m.group(2))}" if m else "?"
|
|
||||||
|
|
||||||
def scan(ps):
|
|
||||||
p=Path(ps)
|
|
||||||
try:
|
|
||||||
ev=read_blastware_file(p); s=ev.raw_samples or {}
|
|
||||||
if not all(s.get(c) for c in GEO): return None
|
|
||||||
pre_n=ev.pretrig_samples
|
|
||||||
ts=ev.timestamp
|
|
||||||
stamp=(f"{ts.year:04d}-{ts.month:02d}-{ts.day:02d}T"
|
|
||||||
f"{ts.hour:02d}:{ts.minute:02d}:{ts.second:02d}") if ts else ""
|
|
||||||
out=[]
|
|
||||||
for ch in GEO:
|
|
||||||
a=s[ch]; n=len(a); t=n//3
|
|
||||||
pre = a[:pre_n] if (pre_n and 0 < pre_n < n) else a[:t]
|
|
||||||
mid, end = a[t:2*t], a[2*t:]
|
|
||||||
if not pre or not mid or not end: continue
|
|
||||||
v=[statistics.median(x)*K for x in (pre,mid,end)]
|
|
||||||
out.append({"serial":serial_of(p.name),"timestamp":stamp,
|
|
||||||
"filename":p.name,"channel":ch,
|
|
||||||
"pretrig_n": pre_n or 0,
|
|
||||||
"pre":round(v[0],4),"mid":round(v[1],4),"end":round(v[2],4),
|
|
||||||
"spread":round(max(v)-min(v),4),
|
|
||||||
"peak":round(max(abs(x) for x in a)*K,4)})
|
|
||||||
return out
|
|
||||||
except Exception:
|
|
||||||
return None
|
|
||||||
|
|
||||||
def main():
|
|
||||||
ap=argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--dir",required=True); ap.add_argument("--jobs",type=int,default=4)
|
|
||||||
ap.add_argument("--floor",type=float,default=0.025)
|
|
||||||
ap.add_argument("--max-spread",type=float,default=0.02)
|
|
||||||
ap.add_argument("--out",required=True)
|
|
||||||
a=ap.parse_args()
|
|
||||||
seen=set(); files=[]
|
|
||||||
for q in sorted(Path(a.dir).rglob("*")):
|
|
||||||
if q.is_file() and _WAVE.search(q.name) and q.name not in seen:
|
|
||||||
seen.add(q.name); files.append(str(q))
|
|
||||||
print(f"unique waveform binaries: {len(files)}",flush=True)
|
|
||||||
rows=[]
|
|
||||||
with ProcessPoolExecutor(max_workers=a.jobs) as ex:
|
|
||||||
for i,f in enumerate(as_completed([ex.submit(scan,p) for p in files]),1):
|
|
||||||
r=f.result()
|
|
||||||
if r: rows.extend(r)
|
|
||||||
if i%2000==0: print(f" {i}/{len(files)}",flush=True)
|
|
||||||
for r in rows:
|
|
||||||
r["offset"]=int(abs(r["pre"])>=a.floor and r["spread"]<=a.max_spread)
|
|
||||||
cols=["serial","timestamp","filename","channel","pretrig_n","pre","mid","end","spread","peak","offset"]
|
|
||||||
with open(a.out,"w",newline="") as fh:
|
|
||||||
w=csv.DictWriter(fh,fieldnames=cols); w.writeheader(); w.writerows(rows)
|
|
||||||
from collections import defaultdict
|
|
||||||
per=defaultdict(set); tot=defaultdict(set)
|
|
||||||
for r in rows:
|
|
||||||
tot[r["serial"]].add(r["filename"])
|
|
||||||
if r["offset"]: per[r["serial"]].add(r["filename"])
|
|
||||||
print(f"\nfloor={a.floor} in/s ({a.floor/0.005:.0f} counts) max spread={a.max_spread}")
|
|
||||||
print(f"units affected: {len(per)} of {len(tot)}")
|
|
||||||
for s in sorted(per,key=lambda s:-len(per[s])):
|
|
||||||
print(f" {s:9} {len(per[s]):4} / {len(tot[s]):4} events")
|
|
||||||
print(f"\nwrote {a.out}")
|
|
||||||
|
|
||||||
if __name__=="__main__": main()
|
|
||||||
@@ -1,221 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Verify the series-3 decoder against preserved Blastware ASCII exports.
|
|
||||||
|
|
||||||
Pairs each `<stem>_<ext>_ASCII.TXT` with its binary `<stem>.<ext>`, decodes the
|
|
||||||
binary with the production codec, and compares against BW's own export:
|
|
||||||
|
|
||||||
waveform — per-channel sample counts, then every sample value
|
|
||||||
histogram — interval count, then every per-interval channel peak
|
|
||||||
|
|
||||||
ADC counts convert as ips = counts * geo_range_ips / 32000 (1 decoder unit =
|
|
||||||
16 counts = 0.005 in/s at the 10 in/s range; see CLAUDE.md).
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python scratch/verify_against_ascii.py --dir <path> [--limit N] [--jobs N]
|
|
||||||
[--out results.json] [--kind w|h|all]
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse, json, re, sys, traceback
|
|
||||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
||||||
|
|
||||||
from minimateplus.event_file_io import read_blastware_file
|
|
||||||
from minimateplus.bw_ascii_report import parse_report_file, parse_report
|
|
||||||
|
|
||||||
GEO = ("Tran", "Vert", "Long")
|
|
||||||
_GEO_FS_COUNTS = 32000.0
|
|
||||||
_ASCII_SUFFIX_RE = re.compile(r"_ASCII\.TXT$", re.IGNORECASE)
|
|
||||||
|
|
||||||
|
|
||||||
def binary_for(ascii_path: Path) -> Path:
|
|
||||||
"""H907KXOW_WC0H_ASCII.TXT -> H907KXOW.WC0H"""
|
|
||||||
stem = _ASCII_SUFFIX_RE.sub("", ascii_path.name)
|
|
||||||
if "_" not in stem:
|
|
||||||
return ascii_path.with_name(stem)
|
|
||||||
head, _, ext = stem.rpartition("_")
|
|
||||||
return ascii_path.with_name(f"{head}.{ext}")
|
|
||||||
|
|
||||||
|
|
||||||
def counts_to_ips(counts, geo_range_ips):
|
|
||||||
r = geo_range_ips if geo_range_ips else 10.0
|
|
||||||
return counts * r / _GEO_FS_COUNTS
|
|
||||||
|
|
||||||
|
|
||||||
def agrees(got, exp, geo_range_ips, tol=0.0006):
|
|
||||||
"""True when decoded `got` matches BW's exported `exp`.
|
|
||||||
|
|
||||||
Saturation carve-out: when an event clips, BW clamps its export to the
|
|
||||||
channel's range maximum (and writes OORANGE for the summary PPV), while
|
|
||||||
the decoder faithfully reproduces raw counts that can sit a decoder unit
|
|
||||||
or two past nominal full scale (32016 counts observed = 10.005 in/s on
|
|
||||||
the 10 in/s range). Same sign and both at/above the ceiling is agreement,
|
|
||||||
not a decode error.
|
|
||||||
"""
|
|
||||||
if abs(got - exp) <= tol:
|
|
||||||
return True
|
|
||||||
r = geo_range_ips if geo_range_ips else 10.0
|
|
||||||
if abs(exp) >= r - tol and abs(got) >= r - tol and (got >= 0) == (exp >= 0):
|
|
||||||
return True
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def parse_interval_table(text: str):
|
|
||||||
"""Histogram interval rows: time, Tpk, Tfq, Vpk, Vfq, Lpk, Lfq, PVS, ..., micdB, micfq"""
|
|
||||||
rows = []
|
|
||||||
seen_header = False
|
|
||||||
for line in text.splitlines():
|
|
||||||
if "\t" not in line:
|
|
||||||
continue
|
|
||||||
cols = [c.strip().strip('"') for c in line.split("\t")]
|
|
||||||
cols = [c for c in cols if c != ""]
|
|
||||||
if not seen_header:
|
|
||||||
if any(c in ("Tran", "Vert", "Long") for c in cols):
|
|
||||||
seen_header = True
|
|
||||||
continue
|
|
||||||
if len(cols) < 7:
|
|
||||||
continue
|
|
||||||
if not re.match(r"^\d{1,2}:\d{2}:\d{2}$", cols[0]):
|
|
||||||
continue
|
|
||||||
def num(s):
|
|
||||||
try:
|
|
||||||
return float(s)
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
rows.append({"time": cols[0], "Tran": num(cols[1]),
|
|
||||||
"Vert": num(cols[3]), "Long": num(cols[5])})
|
|
||||||
return rows
|
|
||||||
|
|
||||||
|
|
||||||
def check_one(ascii_path_str: str) -> dict:
|
|
||||||
ap = Path(ascii_path_str)
|
|
||||||
bp = binary_for(ap)
|
|
||||||
res = {"ascii": ap.name, "binary": bp.name, "status": "?",
|
|
||||||
"kind": None, "detail": ""}
|
|
||||||
try:
|
|
||||||
if not bp.exists():
|
|
||||||
res["status"] = "no_binary"
|
|
||||||
return res
|
|
||||||
text = ap.read_text(errors="replace")
|
|
||||||
rep = parse_report(text, parse_samples=True)
|
|
||||||
ev = read_blastware_file(bp)
|
|
||||||
gr = rep.geo_range_ips
|
|
||||||
res["kind"] = kind = ("histogram"
|
|
||||||
if (rep.event_type or "").lower().startswith(("full histogram", "histogram"))
|
|
||||||
else "waveform")
|
|
||||||
samples = ev.raw_samples or {}
|
|
||||||
dec_n = {c: len(samples.get(c) or []) for c in GEO}
|
|
||||||
|
|
||||||
if kind == "histogram":
|
|
||||||
rows = parse_interval_table(text)
|
|
||||||
res["n_ascii"] = len(rows)
|
|
||||||
res["n_decoded"] = dec_n["Tran"]
|
|
||||||
if not rows:
|
|
||||||
res["status"] = "no_ascii_table"
|
|
||||||
return res
|
|
||||||
if dec_n["Tran"] == 0:
|
|
||||||
res["status"] = "decode_empty"
|
|
||||||
return res
|
|
||||||
if dec_n["Tran"] != len(rows):
|
|
||||||
res["status"] = "count_mismatch"
|
|
||||||
res["detail"] = f"decoded {dec_n['Tran']} vs ascii {len(rows)}"
|
|
||||||
return res
|
|
||||||
bad = 0
|
|
||||||
worst = 0.0
|
|
||||||
for i, row in enumerate(rows):
|
|
||||||
for ch in GEO:
|
|
||||||
exp = row[ch]
|
|
||||||
if exp is None:
|
|
||||||
continue
|
|
||||||
got = counts_to_ips(samples[ch][i], gr)
|
|
||||||
if not agrees(got, exp, gr):
|
|
||||||
bad += 1
|
|
||||||
worst = max(worst, abs(got - exp))
|
|
||||||
res["worst_abs"] = round(worst, 6)
|
|
||||||
res["status"] = "exact" if bad == 0 else "value_mismatch"
|
|
||||||
if bad:
|
|
||||||
res["detail"] = f"{bad} interval-channel values off"
|
|
||||||
return res
|
|
||||||
|
|
||||||
# waveform
|
|
||||||
asc = rep.samples or []
|
|
||||||
res["n_ascii"] = len(asc)
|
|
||||||
res["n_decoded"] = dec_n["Tran"]
|
|
||||||
if not asc:
|
|
||||||
res["status"] = "no_ascii_table"
|
|
||||||
return res
|
|
||||||
if dec_n["Tran"] == 0:
|
|
||||||
res["status"] = "decode_empty"
|
|
||||||
return res
|
|
||||||
if len({dec_n[c] for c in GEO}) != 1:
|
|
||||||
res["status"] = "channel_len_mismatch"
|
|
||||||
res["detail"] = str(dec_n)
|
|
||||||
return res
|
|
||||||
if dec_n["Tran"] != len(asc):
|
|
||||||
res["status"] = "count_mismatch"
|
|
||||||
res["detail"] = f"decoded {dec_n['Tran']} vs ascii {len(asc)}"
|
|
||||||
return res
|
|
||||||
bad = 0
|
|
||||||
worst = 0.0
|
|
||||||
for i, quad in enumerate(asc):
|
|
||||||
for j, ch in enumerate(GEO):
|
|
||||||
exp = quad[j]
|
|
||||||
got = counts_to_ips(samples[ch][i], gr)
|
|
||||||
if not agrees(got, exp, gr):
|
|
||||||
bad += 1
|
|
||||||
worst = max(worst, abs(got - exp))
|
|
||||||
res["worst_abs"] = round(worst, 6)
|
|
||||||
res["status"] = "exact" if bad == 0 else "value_mismatch"
|
|
||||||
if bad:
|
|
||||||
res["detail"] = f"{bad} sample values off"
|
|
||||||
return res
|
|
||||||
except Exception as e:
|
|
||||||
res["status"] = "error"
|
|
||||||
res["detail"] = f"{type(e).__name__}: {e}"
|
|
||||||
return res
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--dir", required=True)
|
|
||||||
ap.add_argument("--limit", type=int, default=0)
|
|
||||||
ap.add_argument("--jobs", type=int, default=8)
|
|
||||||
ap.add_argument("--kind", choices=["w", "h", "all"], default="all")
|
|
||||||
ap.add_argument("--out", default=None)
|
|
||||||
a = ap.parse_args()
|
|
||||||
|
|
||||||
root = Path(a.dir)
|
|
||||||
files = sorted(p for p in root.rglob("*")
|
|
||||||
if p.is_file() and p.name.upper().endswith("_ASCII.TXT"))
|
|
||||||
if a.kind != "all":
|
|
||||||
want = "0W" if a.kind == "w" else "0H"
|
|
||||||
files = [p for p in files
|
|
||||||
if _ASCII_SUFFIX_RE.sub("", p.name).upper().endswith(want)]
|
|
||||||
if a.limit:
|
|
||||||
files = files[: a.limit]
|
|
||||||
print(f"pairs to check: {len(files)}", flush=True)
|
|
||||||
|
|
||||||
out = []
|
|
||||||
from collections import Counter
|
|
||||||
tally = Counter()
|
|
||||||
with ProcessPoolExecutor(max_workers=a.jobs) as ex:
|
|
||||||
futs = {ex.submit(check_one, str(p)): p for p in files}
|
|
||||||
for n, f in enumerate(as_completed(futs), 1):
|
|
||||||
r = f.result()
|
|
||||||
out.append(r)
|
|
||||||
tally[(r["kind"], r["status"])] += 1
|
|
||||||
if n % 500 == 0:
|
|
||||||
print(f" {n}/{len(files)}", flush=True)
|
|
||||||
|
|
||||||
print("\n=== results ===")
|
|
||||||
for (kind, status), n in sorted(tally.items(), key=lambda x: -x[1]):
|
|
||||||
print(f" {str(kind):10} {status:22} {n}")
|
|
||||||
if a.out:
|
|
||||||
Path(a.out).write_text(json.dumps(out, indent=1))
|
|
||||||
print(f"\nwrote {a.out}")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,63 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Backfill events.shape_* from each event's .h5 waveform samples. Idempotent."""
|
|
||||||
from __future__ import annotations
|
|
||||||
import argparse, logging, sys
|
|
||||||
from pathlib import Path
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
from sfm.shape_metrics import shape_from_h5
|
|
||||||
|
|
||||||
log = logging.getLogger("backfill_event_shape")
|
|
||||||
|
|
||||||
def backfill_shape(db: SeismoDb, store: WaveformStore, *, dry_run: bool = False) -> dict:
|
|
||||||
counts = {"updated": 0, "skipped_no_h5": 0, "skipped_no_samples": 0,
|
|
||||||
"cleared_stale": 0}
|
|
||||||
for row in db.query_events(limit=1_000_000):
|
|
||||||
serial, filename = row.get("serial"), row.get("blastware_filename")
|
|
||||||
if not serial or not filename:
|
|
||||||
counts["skipped_no_h5"] += 1; continue
|
|
||||||
h5_path = store.hdf5_path_for(serial, filename)
|
|
||||||
if not h5_path.exists():
|
|
||||||
counts["skipped_no_h5"] += 1; continue
|
|
||||||
shape = shape_from_h5(h5_path)
|
|
||||||
if shape is None:
|
|
||||||
# The .h5 can no longer yield a shape (fewer than 2 samples, or a
|
|
||||||
# flat trace). Clear any previously stored value rather than
|
|
||||||
# leaving it behind — a stale shape outlives the decode it came
|
|
||||||
# from and silently feeds the false-trigger detector. Seen after
|
|
||||||
# a decoder fix shrinks an event: 493 rows in the prod snapshot
|
|
||||||
# were carrying metrics from a superseded decode (2026-08-25).
|
|
||||||
if row.get("shape_crest_factor") is not None:
|
|
||||||
if not dry_run:
|
|
||||||
with db._connect() as conn:
|
|
||||||
conn.execute(
|
|
||||||
"UPDATE events SET shape_crest_factor=NULL, "
|
|
||||||
"shape_near_peak_count=NULL, shape_sample_count=NULL, "
|
|
||||||
"shape_axis=NULL WHERE id=?", (row["id"],))
|
|
||||||
counts["cleared_stale"] += 1
|
|
||||||
counts["skipped_no_samples"] += 1; continue
|
|
||||||
if not dry_run:
|
|
||||||
with db._connect() as conn:
|
|
||||||
conn.execute(
|
|
||||||
"UPDATE events SET shape_crest_factor=?, shape_near_peak_count=?, "
|
|
||||||
"shape_sample_count=?, shape_axis=? WHERE id=?",
|
|
||||||
(shape["crest_factor"], shape["near_peak_count"],
|
|
||||||
shape["sample_count"], shape["axis"], row["id"]))
|
|
||||||
counts["updated"] += 1
|
|
||||||
log.info("backfill_shape: %s", counts)
|
|
||||||
return counts
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
ap = argparse.ArgumentParser()
|
|
||||||
ap.add_argument("--db-path", required=True)
|
|
||||||
ap.add_argument("--store-root", required=True)
|
|
||||||
ap.add_argument("--dry-run", action="store_true")
|
|
||||||
a = ap.parse_args(argv)
|
|
||||||
logging.basicConfig(level=logging.INFO)
|
|
||||||
counts = backfill_shape(SeismoDb(a.db_path), WaveformStore(a.store_root), dry_run=a.dry_run)
|
|
||||||
print(counts)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,98 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Backfill events.*_zc_freq / *_zc_above_range from .sfm.json sidecars.
|
|
||||||
|
|
||||||
Idempotent. Reads each event's sidecar bw_report and writes the per-channel
|
|
||||||
ZC frequency into the DB columns via a direct UPDATE. Events with no sidecar
|
|
||||||
(or no bw_report freq) are left untouched. Mirrors scripts/backfill_record_type.py.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
import argparse
|
|
||||||
import logging
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
||||||
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
|
|
||||||
log = logging.getLogger("backfill_event_zc_freq")
|
|
||||||
|
|
||||||
|
|
||||||
def _freqs_from_sidecar(sidecar: dict) -> dict | None:
|
|
||||||
bw = (sidecar or {}).get("bw_report") or {}
|
|
||||||
peaks = bw.get("peaks") or {}
|
|
||||||
mic = bw.get("mic") or {}
|
|
||||||
out = {}
|
|
||||||
for axis in ("tran", "vert", "long"):
|
|
||||||
chd = peaks.get(axis) or {}
|
|
||||||
out[f"{axis}_zc_freq"] = chd.get("zc_freq_hz")
|
|
||||||
out[f"{axis}_zc_above_range"] = 1 if chd.get("zc_freq_above_range") else 0
|
|
||||||
out["mic_zc_freq"] = mic.get("zc_freq_hz")
|
|
||||||
out["mic_zc_above_range"] = 1 if mic.get("zc_freq_above_range") else 0
|
|
||||||
# None if there's literally no frequency anywhere.
|
|
||||||
if all(out[f"{a}_zc_freq"] is None for a in ("tran", "vert", "long", "mic")):
|
|
||||||
return None
|
|
||||||
return out
|
|
||||||
|
|
||||||
|
|
||||||
def backfill_zc_freq(db: SeismoDb, store, *, dry_run: bool = False) -> dict:
|
|
||||||
counts = {"updated": 0, "skipped_no_sidecar": 0, "skipped_no_freq": 0}
|
|
||||||
# Pull every event (paginate generously; adjust if a store has >100k events).
|
|
||||||
rows = db.query_events(limit=1_000_000)
|
|
||||||
for row in rows:
|
|
||||||
serial = row.get("serial")
|
|
||||||
filename = row.get("blastware_filename")
|
|
||||||
if not serial or not filename:
|
|
||||||
counts["skipped_no_sidecar"] += 1
|
|
||||||
continue
|
|
||||||
sidecar = store.load_sidecar(serial, filename)
|
|
||||||
if sidecar is None:
|
|
||||||
counts["skipped_no_sidecar"] += 1
|
|
||||||
continue
|
|
||||||
freqs = _freqs_from_sidecar(sidecar)
|
|
||||||
if freqs is None:
|
|
||||||
counts["skipped_no_freq"] += 1
|
|
||||||
continue
|
|
||||||
if dry_run:
|
|
||||||
counts["updated"] += 1
|
|
||||||
continue
|
|
||||||
with db._connect() as conn:
|
|
||||||
conn.execute(
|
|
||||||
"""UPDATE events SET
|
|
||||||
tran_zc_freq=?, vert_zc_freq=?, long_zc_freq=?, mic_zc_freq=?,
|
|
||||||
tran_zc_above_range=?, vert_zc_above_range=?,
|
|
||||||
long_zc_above_range=?, mic_zc_above_range=?
|
|
||||||
WHERE id=?""",
|
|
||||||
(freqs["tran_zc_freq"], freqs["vert_zc_freq"], freqs["long_zc_freq"],
|
|
||||||
freqs["mic_zc_freq"], freqs["tran_zc_above_range"],
|
|
||||||
freqs["vert_zc_above_range"], freqs["long_zc_above_range"],
|
|
||||||
freqs["mic_zc_above_range"], row["id"]),
|
|
||||||
)
|
|
||||||
counts["updated"] += 1
|
|
||||||
return counts
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
p = argparse.ArgumentParser(description=__doc__)
|
|
||||||
p.add_argument("--db-path", default=str(
|
|
||||||
Path(__file__).resolve().parent.parent / "bridges" / "captures" / "seismo_relay.db"))
|
|
||||||
p.add_argument("--store-root", default=None)
|
|
||||||
p.add_argument("--dry-run", action="store_true")
|
|
||||||
p.add_argument("-v", "--verbose", action="store_true")
|
|
||||||
args = p.parse_args(argv)
|
|
||||||
logging.basicConfig(level=logging.DEBUG if args.verbose else logging.INFO)
|
|
||||||
db_path = Path(args.db_path).expanduser().resolve()
|
|
||||||
store_root = (
|
|
||||||
Path(args.store_root).expanduser().resolve()
|
|
||||||
if args.store_root else db_path.parent / "waveforms"
|
|
||||||
)
|
|
||||||
db = SeismoDb(db_path)
|
|
||||||
store = WaveformStore(store_root)
|
|
||||||
counts = backfill_zc_freq(db, store, dry_run=args.dry_run)
|
|
||||||
log.info("backfill_zc_freq: %s", counts)
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,150 +0,0 @@
|
|||||||
"""
|
|
||||||
scripts/backfill_record_type.py — fix `record_type` on legacy event
|
|
||||||
rows whose value was hardcoded to "Waveform" regardless of actual type.
|
|
||||||
|
|
||||||
Why this is needed
|
|
||||||
──────────────────
|
|
||||||
Pre-v0.16.1 the BW file importer (`event_file_io.read_blastware_file`)
|
|
||||||
hardcoded `ev.record_type = "Waveform"` for every imported event. Fixed
|
|
||||||
in commit aac1c8e — new ingests now derive the type from the Blastware
|
|
||||||
filename's extension last character (H=Histogram, W=Waveform, M=Manual,
|
|
||||||
E=Event, C=Combo) per the V10.72+ MiniMate Plus AB0T filename scheme.
|
|
||||||
|
|
||||||
Effect on a server that imported events under the old code: every
|
|
||||||
events row has `record_type = "Waveform"`, even for histograms,
|
|
||||||
manuals, etc. Visible in terra-view's event-detail modal under the
|
|
||||||
"Record Type" field. Terra-view also has a client-side workaround
|
|
||||||
that derives the type from the filename for display purposes, so
|
|
||||||
operators see the correct type in the UI even before this backfill.
|
|
||||||
This script makes the DB column match what the UI is already showing,
|
|
||||||
which matters for reporting and any downstream consumer that reads
|
|
||||||
events.record_type directly.
|
|
||||||
|
|
||||||
This script
|
|
||||||
───────────
|
|
||||||
Walks the `events` table and updates each row's `record_type` to the
|
|
||||||
derived value from its `blastware_filename`. Old S338 firmware files
|
|
||||||
(3-char extensions ending in `0`) and any unrecognized suffix get
|
|
||||||
left at the existing value (defaults to "Waveform").
|
|
||||||
|
|
||||||
Idempotent: re-running after a successful backfill finds zero rows
|
|
||||||
needing updates and exits cleanly (it always re-derives but only
|
|
||||||
writes when the value would change).
|
|
||||||
|
|
||||||
Usage
|
|
||||||
─────
|
|
||||||
# Dry-run (default): print what would change, don't touch the DB
|
|
||||||
python -m scripts.backfill_record_type --db bridges/captures/seismo_relay.db
|
|
||||||
|
|
||||||
# Apply the backfill
|
|
||||||
python -m scripts.backfill_record_type --db bridges/captures/seismo_relay.db --apply
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import sqlite3
|
|
||||||
import sys
|
|
||||||
from collections import Counter
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
# Must stay in sync with minimateplus.event_file_io._RECORD_TYPE_BY_EXT_SUFFIX.
|
|
||||||
_TYPE_FROM_SUFFIX = {
|
|
||||||
"H": "Histogram",
|
|
||||||
"W": "Waveform",
|
|
||||||
"M": "Manual",
|
|
||||||
"E": "Event",
|
|
||||||
"C": "Combo",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def derive_record_type(filename: str | None, default: str = "Waveform") -> str:
|
|
||||||
"""Mirror of minimateplus.event_file_io.derive_record_type_from_filename.
|
|
||||||
|
|
||||||
Vendored here so this script runs without needing the seismo-relay
|
|
||||||
package on the Python path (useful on prod where you might be
|
|
||||||
running it via `docker exec` against a container's DB volume).
|
|
||||||
"""
|
|
||||||
if not filename:
|
|
||||||
return default
|
|
||||||
name = Path(filename).name
|
|
||||||
if "." not in name:
|
|
||||||
return default
|
|
||||||
ext = name.rsplit(".", 1)[1]
|
|
||||||
if not ext:
|
|
||||||
return default
|
|
||||||
return _TYPE_FROM_SUFFIX.get(ext[-1].upper(), default)
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
ap = argparse.ArgumentParser(description=__doc__)
|
|
||||||
ap.add_argument("--db", required=True, help="Path to seismo_relay.db")
|
|
||||||
ap.add_argument("--apply", action="store_true",
|
|
||||||
help="Actually write changes (default is dry-run).")
|
|
||||||
ap.add_argument("--default", default="Waveform",
|
|
||||||
help="Fallback record_type when filename doesn't encode one. "
|
|
||||||
"Default: Waveform (matches the pre-fix bug's behavior).")
|
|
||||||
args = ap.parse_args()
|
|
||||||
|
|
||||||
db_path = Path(args.db)
|
|
||||||
if not db_path.exists():
|
|
||||||
print(f"ERROR: database not found at {db_path}", file=sys.stderr)
|
|
||||||
return 1
|
|
||||||
|
|
||||||
conn = sqlite3.connect(str(db_path))
|
|
||||||
conn.row_factory = sqlite3.Row
|
|
||||||
cur = conn.cursor()
|
|
||||||
|
|
||||||
cur.execute("""
|
|
||||||
SELECT id, blastware_filename, record_type
|
|
||||||
FROM events
|
|
||||||
WHERE blastware_filename IS NOT NULL
|
|
||||||
AND blastware_filename != ''
|
|
||||||
""")
|
|
||||||
rows = cur.fetchall()
|
|
||||||
total = len(rows)
|
|
||||||
print(f"Scanning {total:,} event rows…")
|
|
||||||
print()
|
|
||||||
|
|
||||||
# Tally proposed changes.
|
|
||||||
transitions: Counter[tuple[str, str]] = Counter()
|
|
||||||
update_ids: list[tuple[str, str]] = []
|
|
||||||
unrecognized = 0
|
|
||||||
|
|
||||||
for row in rows:
|
|
||||||
derived = derive_record_type(row["blastware_filename"], default=args.default)
|
|
||||||
current = row["record_type"] or ""
|
|
||||||
if derived == current:
|
|
||||||
continue
|
|
||||||
transitions[(current, derived)] += 1
|
|
||||||
update_ids.append((row["id"], derived))
|
|
||||||
|
|
||||||
if not update_ids:
|
|
||||||
print("Nothing to update — all rows already match.")
|
|
||||||
conn.close()
|
|
||||||
return 0
|
|
||||||
|
|
||||||
print(f"{len(update_ids):,} row(s) need updating:")
|
|
||||||
for (old, new), count in sorted(transitions.items(), key=lambda x: -x[1]):
|
|
||||||
print(f" {count:>6,} {old!r:14s} → {new!r}")
|
|
||||||
print()
|
|
||||||
|
|
||||||
if not args.apply:
|
|
||||||
print("(dry-run — re-run with --apply to write changes)")
|
|
||||||
conn.close()
|
|
||||||
return 0
|
|
||||||
|
|
||||||
print("Applying changes…")
|
|
||||||
cur.executemany(
|
|
||||||
"UPDATE events SET record_type = ? WHERE id = ?",
|
|
||||||
[(new, eid) for eid, new in update_ids],
|
|
||||||
)
|
|
||||||
conn.commit()
|
|
||||||
print(f"Done. Updated {cur.rowcount:,} row(s).")
|
|
||||||
conn.close()
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,481 +0,0 @@
|
|||||||
"""
|
|
||||||
scripts/backfill_sidecars.py — generate .sfm.json sidecars AND .h5
|
|
||||||
clean-waveform files for existing events already in the waveform store
|
|
||||||
that predate those features.
|
|
||||||
|
|
||||||
Walks `<store_root>/<serial>/<filename>` and for each BW event file:
|
|
||||||
|
|
||||||
Sidecar (.sfm.json):
|
|
||||||
- Skip when an existing sidecar's blastware.sha256 matches the
|
|
||||||
current BW file's sha256.
|
|
||||||
- Else regenerate: prefer .a5.pkl (full fidelity); fall back to
|
|
||||||
parsing the BW binary directly (peaks computed from samples).
|
|
||||||
|
|
||||||
Clean waveform (.h5):
|
|
||||||
- Regenerated whenever the sidecar is regenerated (sha mismatch
|
|
||||||
OR sidecar.source.tool_version < current TOOL_VERSION OR --force).
|
|
||||||
The .h5 and the sidecar both come from the same decoder output,
|
|
||||||
so if the sidecar is stale the .h5 is too.
|
|
||||||
- Written when missing.
|
|
||||||
- --skip-hdf5 turns off all .h5 writes.
|
|
||||||
|
|
||||||
Typical use after a decoder upgrade:
|
|
||||||
1. Pull the new seismo-relay code (which bumped TOOL_VERSION).
|
|
||||||
2. Run this script — every sidecar with an older tool_version
|
|
||||||
stamp regenerates, and the associated .h5 cascade-regenerates.
|
|
||||||
3. Operator review state (review.false_trigger, notes, reviewer)
|
|
||||||
and the sidecar's extensions block are preserved across the
|
|
||||||
regen.
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
python scripts/backfill_sidecars.py [--store-root PATH]
|
|
||||||
[--db-path PATH]
|
|
||||||
[--dry-run]
|
|
||||||
[--skip-hdf5]
|
|
||||||
[-v]
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import logging
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
# Allow running from the repo root without installation.
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
||||||
|
|
||||||
from minimateplus import event_file_io
|
|
||||||
from sfm import event_hdf5
|
|
||||||
from sfm.waveform_store import WaveformStore, _frame_to_dict, _dict_to_frame # noqa: F401
|
|
||||||
from sfm.database import SeismoDb
|
|
||||||
|
|
||||||
log = logging.getLogger("backfill_sidecars")
|
|
||||||
|
|
||||||
|
|
||||||
def _looks_like_event_file(path: Path) -> bool:
|
|
||||||
"""Same heuristic as the importer CLI.
|
|
||||||
|
|
||||||
Filters to BW (Series III) event files only — Thor (Series IV)
|
|
||||||
`.IDFW` / `.IDFH` files share the store but have their own ingest
|
|
||||||
path (`WaveformStore.save_imported_idf`) and are NOT decodable by
|
|
||||||
`event_file_io.read_blastware_file`. Their sidecars are populated
|
|
||||||
at ingest from the paired `.IDFW.txt` ASCII report; nothing the
|
|
||||||
backfill regenerates would improve on them, so we exclude them
|
|
||||||
from scope.
|
|
||||||
"""
|
|
||||||
if not path.is_file():
|
|
||||||
return False
|
|
||||||
if path.name.endswith((".a5.pkl", ".sfm.json", ".h5")):
|
|
||||||
return False
|
|
||||||
ext = path.suffix.lstrip(".")
|
|
||||||
if not (3 <= len(ext) <= 4):
|
|
||||||
return False
|
|
||||||
# Thor IDF files share the .{W,H}-suffix shape but aren't BW.
|
|
||||||
if ext.upper() in ("IDFW", "IDFH"):
|
|
||||||
return False
|
|
||||||
if not (ext[-1].upper() in {"W", "H"} or ext.endswith("0")):
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
return path.stat().st_size >= 70
|
|
||||||
except OSError:
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
p = argparse.ArgumentParser(description=__doc__)
|
|
||||||
p.add_argument(
|
|
||||||
"--db-path",
|
|
||||||
default=str(Path(__file__).resolve().parent.parent / "bridges" / "captures" / "seismo_relay.db"),
|
|
||||||
)
|
|
||||||
p.add_argument("--store-root", default=None)
|
|
||||||
p.add_argument("--dry-run", action="store_true")
|
|
||||||
p.add_argument(
|
|
||||||
"--skip-hdf5", action="store_true",
|
|
||||||
help="Don't generate .h5 clean-waveform files (only sidecars).",
|
|
||||||
)
|
|
||||||
p.add_argument(
|
|
||||||
"--force", action="store_true",
|
|
||||||
help=(
|
|
||||||
"Regenerate sidecars + .h5 even when an existing sidecar's "
|
|
||||||
"blastware.sha256 matches the current BW file. Use this after "
|
|
||||||
"upgrading seismo-relay to pull in decoder bug fixes (e.g. the "
|
|
||||||
"STRT-rectime byte-offset fix in v0.15.x)."
|
|
||||||
),
|
|
||||||
)
|
|
||||||
p.add_argument(
|
|
||||||
"--reparse-txt", action="store_true",
|
|
||||||
help=(
|
|
||||||
"Re-parse the preserved <serial>/<filename>_ASCII.TXT with the "
|
|
||||||
"current bw_ascii_report parser and overwrite the sidecar's "
|
|
||||||
"bw_report block. Use this after upgrading the ASCII parser to "
|
|
||||||
"pull in new fields (e.g. zc_freq_above_range for BW '>100 Hz' "
|
|
||||||
"ZC peaks). No-op for events without a preserved .TXT; safely "
|
|
||||||
"idempotent when the parser hasn't changed."
|
|
||||||
),
|
|
||||||
)
|
|
||||||
p.add_argument("-v", "--verbose", action="store_true")
|
|
||||||
args = p.parse_args(argv)
|
|
||||||
|
|
||||||
logging.basicConfig(
|
|
||||||
level=logging.DEBUG if args.verbose else logging.INFO,
|
|
||||||
format="%(asctime)s %(levelname)-7s %(name)s %(message)s",
|
|
||||||
datefmt="%H:%M:%S",
|
|
||||||
)
|
|
||||||
|
|
||||||
db_path = Path(args.db_path).expanduser().resolve()
|
|
||||||
store_root = (
|
|
||||||
Path(args.store_root).expanduser().resolve()
|
|
||||||
if args.store_root else db_path.parent / "waveforms"
|
|
||||||
)
|
|
||||||
if not store_root.exists():
|
|
||||||
print(f"error: store root does not exist: {store_root}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
|
|
||||||
store = WaveformStore(store_root)
|
|
||||||
db = SeismoDb(db_path)
|
|
||||||
|
|
||||||
written = skipped = errors = 0
|
|
||||||
stale_h5_removed = 0
|
|
||||||
for serial_dir in sorted(p for p in store_root.iterdir() if p.is_dir()):
|
|
||||||
serial = serial_dir.name
|
|
||||||
for path in sorted(serial_dir.iterdir()):
|
|
||||||
if not _looks_like_event_file(path):
|
|
||||||
continue
|
|
||||||
sidecar_path = store.sidecar_path_for(serial, path.name)
|
|
||||||
try:
|
|
||||||
bw_sha = event_file_io.file_sha256(path)
|
|
||||||
except Exception as exc:
|
|
||||||
log.error("sha256 failed for %s: %s", path, exc)
|
|
||||||
errors += 1
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Skip when an up-to-date sidecar already exists.
|
|
||||||
#
|
|
||||||
# Two-part freshness check:
|
|
||||||
# 1. blastware.sha256 must match the current BW file (proves
|
|
||||||
# the sidecar describes THIS file).
|
|
||||||
# 2. source.tool_version must be ≥ current TOOL_VERSION (proves
|
|
||||||
# the sidecar was written by a build that includes any
|
|
||||||
# decoder fixes shipped since).
|
|
||||||
# Either part failing → regenerate. --force bypasses both.
|
|
||||||
#
|
|
||||||
# Tracks whether we're regenerating the sidecar this iteration
|
|
||||||
# so the .h5 logic below knows to refresh that too — staleness
|
|
||||||
# of the sidecar implies staleness of the derived .h5 (both
|
|
||||||
# come out of the same decoder).
|
|
||||||
sidecar_stale = True
|
|
||||||
if sidecar_path.exists() and not args.force and not args.reparse_txt:
|
|
||||||
try:
|
|
||||||
existing = event_file_io.read_sidecar(sidecar_path)
|
|
||||||
sha_ok = existing.get("blastware", {}).get("sha256") == bw_sha
|
|
||||||
src_ver = existing.get("source", {}).get("tool_version", "")
|
|
||||||
def _vt(s):
|
|
||||||
try:
|
|
||||||
return tuple(int(p) for p in str(s).split(".")[:3])
|
|
||||||
except Exception:
|
|
||||||
return (0, 0, 0)
|
|
||||||
ver_ok = _vt(src_ver) >= _vt(event_file_io.TOOL_VERSION)
|
|
||||||
if sha_ok and ver_ok:
|
|
||||||
skipped += 1
|
|
||||||
sidecar_stale = False
|
|
||||||
continue
|
|
||||||
if sha_ok and not ver_ok:
|
|
||||||
log.info(
|
|
||||||
"regenerating %s (sidecar tool_version=%s < current %s)",
|
|
||||||
sidecar_path.name, src_ver or "(none)",
|
|
||||||
event_file_io.TOOL_VERSION,
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
pass # fall through to rewrite
|
|
||||||
|
|
||||||
# Decide path: A5-based (high-fidelity) or BW-only.
|
|
||||||
a5_path = serial_dir / f"{path.name}.a5.pkl"
|
|
||||||
try:
|
|
||||||
if a5_path.exists():
|
|
||||||
frames = store.load_a5(serial, path.name)
|
|
||||||
if not frames:
|
|
||||||
raise RuntimeError("a5_pickle present but unreadable")
|
|
||||||
# Build an Event by replaying the A5 decoders. Note:
|
|
||||||
# the .a5.pkl alone CANNOT recover timestamp /
|
|
||||||
# record_type / waveform_key / per-channel peaks —
|
|
||||||
# those live in the 0C record, which isn't saved
|
|
||||||
# separately. We seed those from the DB row + the
|
|
||||||
# existing sidecar below so a re-backfill doesn't
|
|
||||||
# nuke fields the original save populated.
|
|
||||||
from minimateplus.client import (
|
|
||||||
_decode_a5_metadata_into,
|
|
||||||
_decode_a5_waveform,
|
|
||||||
)
|
|
||||||
from minimateplus.models import Event, PeakValues, ProjectInfo, Timestamp
|
|
||||||
ev = Event(index=-1)
|
|
||||||
_decode_a5_metadata_into(frames, ev)
|
|
||||||
_decode_a5_waveform(frames, ev)
|
|
||||||
source_kind = "sfm-live"
|
|
||||||
a5_filename = a5_path.name
|
|
||||||
else:
|
|
||||||
ev = event_file_io.read_blastware_file(path)
|
|
||||||
source_kind = "bw-import"
|
|
||||||
a5_filename = None
|
|
||||||
from minimateplus.models import Event, PeakValues, ProjectInfo, Timestamp
|
|
||||||
|
|
||||||
# ── Seed missing fields from the SeismoDb events row ──
|
|
||||||
# The DB row was populated at original save time with peaks,
|
|
||||||
# project info, timestamp, record_type, sample_rate, etc.
|
|
||||||
# All of those survive intact in SQLite; pull them onto the
|
|
||||||
# rebuilt Event so the regenerated sidecar matches what was
|
|
||||||
# there before the backfill ran.
|
|
||||||
db_row = None
|
|
||||||
try:
|
|
||||||
import sqlite3 as _sql
|
|
||||||
with _sql.connect(str(db.db_path)) as _conn:
|
|
||||||
_conn.row_factory = _sql.Row
|
|
||||||
db_row = _conn.execute(
|
|
||||||
"SELECT * FROM events "
|
|
||||||
"WHERE serial=? AND blastware_filename=? "
|
|
||||||
"LIMIT 1",
|
|
||||||
(serial, path.name),
|
|
||||||
).fetchone()
|
|
||||||
except Exception as exc:
|
|
||||||
log.debug("DB lookup failed for %s: %s", path.name, exc)
|
|
||||||
|
|
||||||
if db_row is not None:
|
|
||||||
if ev.sample_rate is None and db_row["sample_rate"]:
|
|
||||||
ev.sample_rate = int(db_row["sample_rate"])
|
|
||||||
if not ev.record_type and db_row["record_type"]:
|
|
||||||
ev.record_type = db_row["record_type"]
|
|
||||||
if ev._waveform_key is None and db_row["waveform_key"]:
|
|
||||||
try:
|
|
||||||
ev._waveform_key = bytes.fromhex(db_row["waveform_key"])
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
# Timestamp from the ISO-8601 string in the DB row.
|
|
||||||
if ev.timestamp is None and db_row["timestamp"]:
|
|
||||||
try:
|
|
||||||
import datetime as _dt
|
|
||||||
_t = _dt.datetime.fromisoformat(db_row["timestamp"])
|
|
||||||
ev.timestamp = Timestamp(
|
|
||||||
raw=b"", flag=0x10,
|
|
||||||
year=_t.year, unknown_byte=0,
|
|
||||||
month=_t.month, day=_t.day,
|
|
||||||
hour=_t.hour, minute=_t.minute, second=_t.second,
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
# Peaks from the DB row when the A5 decode didn't supply them.
|
|
||||||
if ev.peak_values is None:
|
|
||||||
ev.peak_values = PeakValues(
|
|
||||||
tran=db_row["tran_ppv"],
|
|
||||||
vert=db_row["vert_ppv"],
|
|
||||||
long=db_row["long_ppv"],
|
|
||||||
peak_vector_sum=db_row["peak_vector_sum"],
|
|
||||||
micl=db_row["mic_ppv"],
|
|
||||||
)
|
|
||||||
# Project info from the DB row when the A5 metadata-page
|
|
||||||
# decode didn't pick it up.
|
|
||||||
if ev.project_info is None or all(
|
|
||||||
v in (None, "")
|
|
||||||
for v in (
|
|
||||||
(ev.project_info.project if ev.project_info else None),
|
|
||||||
(ev.project_info.client if ev.project_info else None),
|
|
||||||
(ev.project_info.operator if ev.project_info else None),
|
|
||||||
(ev.project_info.sensor_location if ev.project_info else None),
|
|
||||||
)
|
|
||||||
):
|
|
||||||
ev.project_info = ProjectInfo(
|
|
||||||
project=db_row["project"],
|
|
||||||
client=db_row["client"],
|
|
||||||
operator=db_row["operator"],
|
|
||||||
sensor_location=db_row["sensor_location"],
|
|
||||||
)
|
|
||||||
|
|
||||||
# Derive total_samples when we have both rectime + sample_rate.
|
|
||||||
# The decoder's STRT-derived value can be a buffer offset
|
|
||||||
# rather than a sample count — drop it in that case.
|
|
||||||
if ev.sample_rate and ev.rectime_seconds:
|
|
||||||
derived = int(round(ev.sample_rate * ev.rectime_seconds))
|
|
||||||
if (ev.total_samples is None
|
|
||||||
or ev.total_samples > derived * 2
|
|
||||||
or ev.total_samples < derived // 4):
|
|
||||||
ev.total_samples = derived
|
|
||||||
|
|
||||||
# Preserve user-edited review state + extensions + the
|
|
||||||
# bw_report block from the existing sidecar so a backfill
|
|
||||||
# never wipes them out. The bw_report block originates
|
|
||||||
# from the paired .TXT ASCII report parsed at ORIGINAL
|
|
||||||
# import time (ach forward / direct upload); the .TXT
|
|
||||||
# file is not in the waveform store, so we can't re-derive
|
|
||||||
# it from disk. event_to_sidecar_dict takes a
|
|
||||||
# BwAsciiReport dataclass (not a dict), so for bw_report
|
|
||||||
# we overlay the existing block after regen instead of
|
|
||||||
# passing it as a kwarg.
|
|
||||||
preserved_review = None
|
|
||||||
preserved_ext = None
|
|
||||||
preserved_bw_report = None
|
|
||||||
preserved_txt_fn = None
|
|
||||||
if sidecar_path.exists():
|
|
||||||
try:
|
|
||||||
_existing = event_file_io.read_sidecar(sidecar_path)
|
|
||||||
preserved_review = _existing.get("review")
|
|
||||||
preserved_ext = _existing.get("extensions")
|
|
||||||
preserved_bw_report = _existing.get("bw_report")
|
|
||||||
# Preserve txt_filename so backfills don't blank out the
|
|
||||||
# pointer to the saved raw .TXT (events ingested after
|
|
||||||
# 2026-05-27 have this).
|
|
||||||
preserved_txt_fn = (_existing.get("source") or {}).get("txt_filename")
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
|
|
||||||
# --reparse-txt: if a .TXT is preserved on disk, run the
|
|
||||||
# current parser against it and overwrite the bw_report
|
|
||||||
# block. Picks up post-ingest parser fixes (e.g. the
|
|
||||||
# 2026-05-28 zc_freq_above_range / ">100 Hz" addition).
|
|
||||||
if args.reparse_txt and preserved_txt_fn:
|
|
||||||
try:
|
|
||||||
from minimateplus import bw_ascii_report
|
|
||||||
txt_path = store.txt_path_for(serial, path.name)
|
|
||||||
if txt_path.exists():
|
|
||||||
refreshed = bw_ascii_report.parse_report_file(txt_path)
|
|
||||||
preserved_bw_report = event_file_io._bw_report_to_dict(refreshed)
|
|
||||||
log.debug("reparsed bw_report from %s", txt_path.name)
|
|
||||||
else:
|
|
||||||
log.debug("--reparse-txt: no .TXT at %s (sidecar says %r)",
|
|
||||||
txt_path, preserved_txt_fn)
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("--reparse-txt failed for %s: %s", path.name, exc)
|
|
||||||
|
|
||||||
# Overlay BW ASCII report fields onto the rebuilt Event
|
|
||||||
# BEFORE the sidecar + DB write. Mirrors what the ingest
|
|
||||||
# path does — BW's reported peaks (and sample_rate /
|
|
||||||
# record_time) win over codec output where present.
|
|
||||||
#
|
|
||||||
# Without this step, --force backfill silently overwrites
|
|
||||||
# the bw_report-overlaid DB columns with codec-derived
|
|
||||||
# values, which is wrong for events the codec doesn't
|
|
||||||
# fully decode (e.g. waveform walker edge cases on
|
|
||||||
# SP0/SS0/SV0-style events, or histogram sub-formats with
|
|
||||||
# byte[5]!=0 that aren't yet RE'd). Net effect was PVS=0
|
|
||||||
# on three top-10 events on 2026-05-22.
|
|
||||||
if preserved_bw_report:
|
|
||||||
event_file_io.apply_bw_report_dict_to_event(
|
|
||||||
ev, preserved_bw_report,
|
|
||||||
)
|
|
||||||
|
|
||||||
sidecar = event_file_io.event_to_sidecar_dict(
|
|
||||||
ev,
|
|
||||||
serial=serial,
|
|
||||||
blastware_filename=path.name,
|
|
||||||
blastware_filesize=path.stat().st_size,
|
|
||||||
blastware_sha256=bw_sha,
|
|
||||||
source_kind=source_kind,
|
|
||||||
a5_pickle_filename=a5_filename,
|
|
||||||
txt_filename=preserved_txt_fn,
|
|
||||||
review=preserved_review,
|
|
||||||
extensions=preserved_ext,
|
|
||||||
)
|
|
||||||
if preserved_bw_report is not None:
|
|
||||||
sidecar["bw_report"] = preserved_bw_report
|
|
||||||
|
|
||||||
# Also emit the .h5 clean-waveform file when:
|
|
||||||
# - it's missing, OR
|
|
||||||
# - --force was passed, OR
|
|
||||||
# - the sidecar is being regenerated this iteration
|
|
||||||
# (sha mismatch / tool_version too old). The .h5 and
|
|
||||||
# the sidecar are both derived from the same decoder
|
|
||||||
# output, so if the sidecar is stale, so is the .h5.
|
|
||||||
#
|
|
||||||
# Both waveform and histogram bodies now decode to real
|
|
||||||
# samples via event_file_io.read_blastware_file → either
|
|
||||||
# waveform_codec.decode_waveform_v2 or histogram_codec.
|
|
||||||
# decode_histogram_body. If samples are still empty after
|
|
||||||
# both codecs run, it's a genuine "we can't decode this
|
|
||||||
# file" case (truncated, malformed, or unknown mode).
|
|
||||||
#
|
|
||||||
# In that case we REMOVE any existing .h5 rather than leave
|
|
||||||
# it. Leaving it was the old behaviour and it silently
|
|
||||||
# preserved output from a superseded decoder: after the
|
|
||||||
# 2026-08-25 record-chain fix, 415 histogram files stopped
|
|
||||||
# decoding (an unmapped block variant on BE18193/BE9440) but
|
|
||||||
# kept .h5 files whose peaks were up to 400x the device's own
|
|
||||||
# reported PPV — garbage that fed the charts and the
|
|
||||||
# false-trigger detector with nothing marking it.
|
|
||||||
has_samples = bool(
|
|
||||||
ev.raw_samples and any(
|
|
||||||
ev.raw_samples.get(ch) for ch in ("Tran", "Vert", "Long", "MicL")
|
|
||||||
)
|
|
||||||
)
|
|
||||||
hdf5_path = store.hdf5_path_for(serial, path.name)
|
|
||||||
hdf5_filename = hdf5_path.name if hdf5_path.exists() else None
|
|
||||||
if not has_samples and hdf5_path.exists():
|
|
||||||
log.warning("%s: no samples decode — removing stale %s",
|
|
||||||
path.name, hdf5_path.name)
|
|
||||||
if not args.dry_run:
|
|
||||||
hdf5_path.unlink()
|
|
||||||
hdf5_filename = None
|
|
||||||
stale_h5_removed += 1
|
|
||||||
hdf5_action = "kept"
|
|
||||||
need_h5 = (
|
|
||||||
not args.skip_hdf5
|
|
||||||
and (args.force or not hdf5_path.exists() or sidecar_stale)
|
|
||||||
and has_samples
|
|
||||||
)
|
|
||||||
if not has_samples and not args.skip_hdf5:
|
|
||||||
hdf5_action = "skipped-undecodable"
|
|
||||||
if need_h5:
|
|
||||||
if args.dry_run:
|
|
||||||
hdf5_action = "would (re)write"
|
|
||||||
else:
|
|
||||||
try:
|
|
||||||
event_hdf5.write_event_hdf5(
|
|
||||||
hdf5_path, ev,
|
|
||||||
serial=serial,
|
|
||||||
geo_range="normal",
|
|
||||||
source_kind=source_kind,
|
|
||||||
)
|
|
||||||
hdf5_filename = hdf5_path.name
|
|
||||||
hdf5_action = "rewrote" if hdf5_path.exists() else "wrote"
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("HDF5 write failed for %s: %s", path.name, exc)
|
|
||||||
hdf5_action = "FAILED"
|
|
||||||
|
|
||||||
if args.dry_run:
|
|
||||||
print(f" [DRY ] would write {sidecar_path.name} "
|
|
||||||
f"+ .h5 ({hdf5_action}) source={source_kind}")
|
|
||||||
written += 1
|
|
||||||
continue
|
|
||||||
|
|
||||||
event_file_io.write_sidecar(sidecar_path, sidecar)
|
|
||||||
|
|
||||||
# Best-effort: keep the SQL row's sidecar_filename in sync
|
|
||||||
# by upserting via insert_events (it dedups on serial+ts).
|
|
||||||
try:
|
|
||||||
db.insert_events(
|
|
||||||
[ev], serial=serial,
|
|
||||||
waveform_records=(
|
|
||||||
{ev._waveform_key.hex(): {
|
|
||||||
"filename": path.name,
|
|
||||||
"filesize": path.stat().st_size,
|
|
||||||
"a5_pickle_filename": a5_filename,
|
|
||||||
"sidecar_filename": sidecar_path.name,
|
|
||||||
}}
|
|
||||||
if ev._waveform_key else None
|
|
||||||
),
|
|
||||||
device_family="series3",
|
|
||||||
)
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("DB upsert failed for %s: %s", path.name, exc)
|
|
||||||
|
|
||||||
print(f" [OK ] {path.name} → {sidecar_path.name} "
|
|
||||||
f"+ h5 ({hdf5_action}) source={source_kind}")
|
|
||||||
written += 1
|
|
||||||
|
|
||||||
except Exception as exc:
|
|
||||||
log.error("backfill failed for %s: %s", path, exc, exc_info=args.verbose)
|
|
||||||
errors += 1
|
|
||||||
|
|
||||||
print(f"\nDone. written={written} skipped(uptodate)={skipped} errors={errors} stale_h5_removed={stale_h5_removed}")
|
|
||||||
return 0 if errors == 0 else 1
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,331 +0,0 @@
|
|||||||
"""
|
|
||||||
scripts/backfill_thor_events.py — re-process existing Thor (Series IV)
|
|
||||||
events so their sidecars carry the bw_report block produced by
|
|
||||||
``micromate.idf_to_bw_report.build_bw_report_from_idf`` + their .h5
|
|
||||||
clean-waveform files for IDFW events.
|
|
||||||
|
|
||||||
Why this exists
|
|
||||||
───────────────
|
|
||||||
|
|
||||||
Thor events ingested before v0.21.0 (or during the v0.21.0 ingest bug
|
|
||||||
window fixed in commit bee1185) have sidecars with only
|
|
||||||
``extensions.idf_report`` — no ``bw_report`` block. Without
|
|
||||||
``bw_report``, the SFM PDF renderer falls back to DB-only fields
|
|
||||||
(misses sensor-self-check, full per-channel breakdown, mic dB(L)),
|
|
||||||
and the modal chart 404s on ``/waveform.json`` for IDFW events
|
|
||||||
because no .h5 was written when the codec failed at ingest.
|
|
||||||
|
|
||||||
Re-forwarding from thor-watcher would also fix this, but that requires
|
|
||||||
operator coordination on every watcher machine and uses bandwidth this
|
|
||||||
script doesn't.
|
|
||||||
|
|
||||||
What this does
|
|
||||||
──────────────
|
|
||||||
|
|
||||||
Walks ``<store>/<serial>/<filename>`` for ``.IDFW`` / ``.IDFH`` files
|
|
||||||
and, for each one:
|
|
||||||
|
|
||||||
1. Reads the existing sidecar (preserving review state + captured_at).
|
|
||||||
2. Re-runs ``micromate.idf_file.read_idf_file()`` on the binary
|
|
||||||
bytes — passing ``data=`` so the codec doesn't try to read from
|
|
||||||
a path it doesn't know.
|
|
||||||
3. Pulls ``extensions.idf_report`` (the raw parsed Thor dict the
|
|
||||||
v0.18.0+ ingest path already stashed) and runs the v0.21.0
|
|
||||||
``build_bw_report_from_idf`` adapter against it.
|
|
||||||
4. Writes the refreshed sidecar with the new ``bw_report``,
|
|
||||||
bumped ``source.tool_version``, but preserved ``review`` block
|
|
||||||
+ the original ``captured_at`` timestamp.
|
|
||||||
5. Regenerates the .h5 waveform file via the existing
|
|
||||||
``event_hdf5`` writer. For IDFW that's the decoded per-sample
|
|
||||||
stream; for IDFH it's a 1-sample-per-interval synthesised array
|
|
||||||
(peak ADC count per channel) so the renderer's bar-chart code
|
|
||||||
has data to group on. Mic peak psi from the binary is merged
|
|
||||||
onto the IdfEvent before the bridge so the h5 writer's per-count
|
|
||||||
mic scale factor lands on a sensible value (without this the
|
|
||||||
mic chart on Thor events plots dB(L)-as-pseudo-psi and shows
|
|
||||||
bomb-level numbers).
|
|
||||||
|
|
||||||
Idempotent. Re-running it after a parser/adapter change just
|
|
||||||
re-writes sidecars — no DB writes, no thor-watcher coordination.
|
|
||||||
|
|
||||||
Usage
|
|
||||||
─────
|
|
||||||
|
|
||||||
python scripts/backfill_thor_events.py [--store-root PATH]
|
|
||||||
[--dry-run]
|
|
||||||
[--skip-hdf5]
|
|
||||||
[--force]
|
|
||||||
[-v]
|
|
||||||
|
|
||||||
By default, refreshes any Thor event whose sidecar is missing
|
|
||||||
``bw_report`` OR whose ``source.tool_version`` is older than the
|
|
||||||
current ``TOOL_VERSION``. ``--force`` refreshes every Thor event
|
|
||||||
regardless.
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import logging
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
# Allow running from the repo root without installation.
|
|
||||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
|
|
||||||
|
|
||||||
from minimateplus import event_file_io
|
|
||||||
from sfm.waveform_store import WaveformStore
|
|
||||||
|
|
||||||
log = logging.getLogger("backfill_thor_events")
|
|
||||||
|
|
||||||
|
|
||||||
def _is_thor_event(path: Path) -> bool:
|
|
||||||
if not path.is_file():
|
|
||||||
return False
|
|
||||||
if path.name.endswith((".sfm.json", ".h5", "_ASCII.TXT")):
|
|
||||||
return False
|
|
||||||
return path.suffix.upper() in (".IDFW", ".IDFH")
|
|
||||||
|
|
||||||
|
|
||||||
def _vtuple(s: str) -> tuple:
|
|
||||||
try:
|
|
||||||
return tuple(int(p) for p in str(s).split(".")[:3])
|
|
||||||
except Exception:
|
|
||||||
return (0, 0, 0)
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
p = argparse.ArgumentParser(description=__doc__)
|
|
||||||
p.add_argument(
|
|
||||||
"--db-path",
|
|
||||||
default=str(Path(__file__).resolve().parent.parent / "bridges" / "captures" / "seismo_relay.db"),
|
|
||||||
help="Used only to derive the default --store-root.",
|
|
||||||
)
|
|
||||||
p.add_argument("--store-root", default=None)
|
|
||||||
p.add_argument("--dry-run", action="store_true")
|
|
||||||
p.add_argument("--skip-hdf5", action="store_true",
|
|
||||||
help="Don't regenerate .h5 files for IDFW events.")
|
|
||||||
p.add_argument("--force", action="store_true",
|
|
||||||
help="Refresh every Thor event, not just ones with stale or missing bw_report.")
|
|
||||||
p.add_argument("-v", "--verbose", action="store_true")
|
|
||||||
args = p.parse_args(argv)
|
|
||||||
|
|
||||||
logging.basicConfig(
|
|
||||||
level=logging.DEBUG if args.verbose else logging.INFO,
|
|
||||||
format="%(asctime)s %(levelname)-7s %(name)s %(message)s",
|
|
||||||
datefmt="%H:%M:%S",
|
|
||||||
)
|
|
||||||
|
|
||||||
db_path = Path(args.db_path).expanduser().resolve()
|
|
||||||
store_root = (
|
|
||||||
Path(args.store_root).expanduser().resolve()
|
|
||||||
if args.store_root else db_path.parent / "waveforms"
|
|
||||||
)
|
|
||||||
if not store_root.exists():
|
|
||||||
log.error("store root not found: %s", store_root)
|
|
||||||
return 1
|
|
||||||
store = WaveformStore(store_root)
|
|
||||||
log.info("store root: %s", store_root)
|
|
||||||
log.info("current TOOL_VERSION: %s", event_file_io.TOOL_VERSION)
|
|
||||||
|
|
||||||
refreshed = skipped = errors = h5_written = 0
|
|
||||||
|
|
||||||
# Lazy imports so any one of these failing produces a useful error
|
|
||||||
# message rather than crashing module-load.
|
|
||||||
from micromate.idf_file import read_idf_file
|
|
||||||
from micromate.idf_to_bw_report import build_bw_report_from_idf
|
|
||||||
|
|
||||||
for serial_dir in sorted(p for p in store_root.iterdir() if p.is_dir()):
|
|
||||||
serial = serial_dir.name
|
|
||||||
for path in sorted(serial_dir.iterdir()):
|
|
||||||
if not _is_thor_event(path):
|
|
||||||
continue
|
|
||||||
|
|
||||||
sidecar_path = store.sidecar_path_for(serial, path.name)
|
|
||||||
if not sidecar_path.exists():
|
|
||||||
log.debug("%s: no sidecar — skipping (this is a binary without ingest history)",
|
|
||||||
path.name)
|
|
||||||
skipped += 1
|
|
||||||
continue
|
|
||||||
|
|
||||||
try:
|
|
||||||
existing = event_file_io.read_sidecar(sidecar_path)
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("%s: failed to read sidecar — %s", path.name, exc)
|
|
||||||
errors += 1
|
|
||||||
continue
|
|
||||||
|
|
||||||
has_bw_report = bool(existing.get("bw_report"))
|
|
||||||
existing_version = (existing.get("source") or {}).get("tool_version", "")
|
|
||||||
up_to_date = (
|
|
||||||
has_bw_report
|
|
||||||
and _vtuple(existing_version) >= _vtuple(event_file_io.TOOL_VERSION)
|
|
||||||
)
|
|
||||||
if up_to_date and not args.force:
|
|
||||||
skipped += 1
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Re-decode the binary. Catch + log; continue with .txt-only
|
|
||||||
# data if it fails (matches the live ingest path's behavior).
|
|
||||||
idf_samples = None
|
|
||||||
idf_intervals = None
|
|
||||||
binary_md = None
|
|
||||||
is_histogram = path.suffix.upper() == ".IDFH"
|
|
||||||
try:
|
|
||||||
binary_bytes = path.read_bytes()
|
|
||||||
res = read_idf_file(path, data=binary_bytes)
|
|
||||||
idf_samples = res.samples or None
|
|
||||||
idf_intervals = res.intervals
|
|
||||||
binary_md = res.binary_metadata
|
|
||||||
is_histogram = res.intervals is not None
|
|
||||||
except NotImplementedError:
|
|
||||||
# sig-B / Blastware-stray binary; no samples but adapter
|
|
||||||
# can still produce a bw_report from extensions.idf_report.
|
|
||||||
log.debug("%s: binary codec NotImplementedError (sig-B / BW-stray); proceeding from sidecar's idf_report only", path.name)
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("%s: binary decode failed — %s; proceeding from sidecar's idf_report only", path.name, exc)
|
|
||||||
|
|
||||||
# Run the adapter. Pull report_dict from
|
|
||||||
# extensions.idf_report (the v0.18.0+ ingest preserved it).
|
|
||||||
report_dict = (existing.get("extensions") or {}).get("idf_report") or {}
|
|
||||||
if not report_dict and binary_md is None:
|
|
||||||
log.debug("%s: no idf_report in sidecar AND no binary metadata — nothing to project", path.name)
|
|
||||||
skipped += 1
|
|
||||||
continue
|
|
||||||
|
|
||||||
try:
|
|
||||||
bw_report = build_bw_report_from_idf(
|
|
||||||
report_dict, binary_md=binary_md,
|
|
||||||
intervals=idf_intervals, is_histogram=is_histogram,
|
|
||||||
)
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("%s: adapter failed — %s", path.name, exc)
|
|
||||||
errors += 1
|
|
||||||
continue
|
|
||||||
|
|
||||||
# Build the new sidecar by overlaying refreshed fields onto
|
|
||||||
# the existing one — preserves review, captured_at, blastware
|
|
||||||
# block, source.kind, etc.
|
|
||||||
new_sidecar = dict(existing) # shallow copy
|
|
||||||
new_sidecar["bw_report"] = bw_report
|
|
||||||
src = dict(new_sidecar.get("source") or {})
|
|
||||||
src["tool_version"] = event_file_io.TOOL_VERSION
|
|
||||||
new_sidecar["source"] = src
|
|
||||||
|
|
||||||
# Preserve histogram intervals if the binary decoded them
|
|
||||||
# (improves over the original ingest if that one ran before
|
|
||||||
# the bee1185 codec fix).
|
|
||||||
if idf_intervals is not None:
|
|
||||||
ext = dict(new_sidecar.get("extensions") or {})
|
|
||||||
ext["idf_intervals"] = [
|
|
||||||
{
|
|
||||||
"offset": iv.offset,
|
|
||||||
"tran_peak": iv.peak_count("Tran"),
|
|
||||||
"tran_halfp": iv.tran_halfp,
|
|
||||||
"tran_freq": iv.freq_hz("Tran"),
|
|
||||||
"vert_peak": iv.peak_count("Vert"),
|
|
||||||
"vert_halfp": iv.vert_halfp,
|
|
||||||
"vert_freq": iv.freq_hz("Vert"),
|
|
||||||
"long_peak": iv.peak_count("Long"),
|
|
||||||
"long_halfp": iv.long_halfp,
|
|
||||||
"long_freq": iv.freq_hz("Long"),
|
|
||||||
"mic_peak": iv.peak_count("MicL"),
|
|
||||||
"mic_halfp": iv.micl_halfp,
|
|
||||||
"mic_freq": iv.freq_hz("MicL"),
|
|
||||||
}
|
|
||||||
for iv in idf_intervals
|
|
||||||
]
|
|
||||||
new_sidecar["extensions"] = ext
|
|
||||||
|
|
||||||
if args.dry_run:
|
|
||||||
will_write_h5 = (idf_samples or idf_intervals) and not args.skip_hdf5
|
|
||||||
log.info("[DRY] %s/%s — would refresh sidecar (bw_report=%s, h5=%s)",
|
|
||||||
serial, path.name,
|
|
||||||
"wrote" if not has_bw_report else "refreshed",
|
|
||||||
"would write" if will_write_h5 else "skipped")
|
|
||||||
else:
|
|
||||||
event_file_io.write_sidecar(sidecar_path, new_sidecar)
|
|
||||||
log.info("%s/%s — sidecar refreshed (bw_report=%s, intervals=%d)",
|
|
||||||
serial, path.name,
|
|
||||||
"added" if not has_bw_report else "refreshed",
|
|
||||||
len(idf_intervals) if idf_intervals else 0)
|
|
||||||
refreshed += 1
|
|
||||||
|
|
||||||
# Regenerate .h5 by replaying the same IdfEvent → Event bridge
|
|
||||||
# save_imported_idf uses. For IDFW we write the decoded per-
|
|
||||||
# sample arrays. For IDFH we synthesise a 1-sample-per-interval
|
|
||||||
# array (peak ADC count per channel per interval) so the
|
|
||||||
# renderer's bar-chart code has something to group on.
|
|
||||||
# Pre-condition: either real samples (IDFW) or decoded intervals
|
|
||||||
# (IDFH). Skip otherwise.
|
|
||||||
have_data = bool(idf_samples) or bool(idf_intervals)
|
|
||||||
if have_data and not args.skip_hdf5:
|
|
||||||
from sfm import event_hdf5
|
|
||||||
hdf5_path = store.hdf5_path_for(serial, path.name)
|
|
||||||
if args.dry_run:
|
|
||||||
log.debug("[DRY] would write %s", hdf5_path.name)
|
|
||||||
else:
|
|
||||||
try:
|
|
||||||
from micromate import IdfEvent
|
|
||||||
from minimateplus.event_file_io import file_sha256
|
|
||||||
idf_event = IdfEvent.from_report(report_dict, path.name)
|
|
||||||
|
|
||||||
# Merge the binary-derived mic peak psi (only the
|
|
||||||
# binary path knows the proper psi value; the .txt
|
|
||||||
# carries dB(L)). Without this, the h5 writer's
|
|
||||||
# per-count mic factor is computed against the
|
|
||||||
# dB(L) value-as-pseudo-psi and the mic chart
|
|
||||||
# scales wildly.
|
|
||||||
if (binary_md is not None and res is not None
|
|
||||||
and res.event.peaks.mic_pspl_psi is not None):
|
|
||||||
idf_event.peaks.mic_pspl_psi = res.event.peaks.mic_pspl_psi
|
|
||||||
|
|
||||||
sha256 = file_sha256(path)
|
|
||||||
waveform_key = bytes.fromhex(sha256)[:16]
|
|
||||||
ev = idf_event.to_minimateplus_event(waveform_key)
|
|
||||||
|
|
||||||
if is_histogram and idf_intervals:
|
|
||||||
# 1 sample per interval per channel — same
|
|
||||||
# synthesis save_imported_idf uses. The h5
|
|
||||||
# writer's count×geo_fs/32768 conversion turns
|
|
||||||
# each peak-ADC-count into the bar's physical
|
|
||||||
# value.
|
|
||||||
ev.raw_samples = {
|
|
||||||
"Tran": [iv.peak_count("Tran") for iv in idf_intervals],
|
|
||||||
"Vert": [iv.peak_count("Vert") for iv in idf_intervals],
|
|
||||||
"Long": [iv.peak_count("Long") for iv in idf_intervals],
|
|
||||||
"MicL": [iv.peak_count("MicL") for iv in idf_intervals],
|
|
||||||
}
|
|
||||||
ev.total_samples = ev.total_samples or len(idf_intervals)
|
|
||||||
elif idf_samples:
|
|
||||||
ev.raw_samples = idf_samples
|
|
||||||
n_samp = max(
|
|
||||||
(len(idf_samples.get(ch, []))
|
|
||||||
for ch in ("Tran", "Vert", "Long", "MicL")),
|
|
||||||
default=0,
|
|
||||||
)
|
|
||||||
ev.total_samples = ev.total_samples or n_samp
|
|
||||||
|
|
||||||
event_hdf5.write_event_hdf5(
|
|
||||||
hdf5_path, ev,
|
|
||||||
serial=serial,
|
|
||||||
geo_range="normal",
|
|
||||||
source_kind="idf-import",
|
|
||||||
tool_version=event_file_io.TOOL_VERSION,
|
|
||||||
)
|
|
||||||
h5_written += 1
|
|
||||||
log.debug("%s/%s — .h5 written (%s)",
|
|
||||||
serial, path.name,
|
|
||||||
f"{len(idf_intervals)} intervals" if is_histogram
|
|
||||||
else f"{sum(len(v) for v in (idf_samples or {}).values())} samples")
|
|
||||||
except Exception as exc:
|
|
||||||
log.warning("%s/%s — .h5 write failed: %s",
|
|
||||||
serial, path.name, exc)
|
|
||||||
|
|
||||||
log.info("Done. refreshed=%d skipped=%d errors=%d h5_written=%d",
|
|
||||||
refreshed, skipped, errors, h5_written)
|
|
||||||
return 0 if errors == 0 else 2
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
sys.exit(main())
|
|
||||||
@@ -1,100 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# Fire-and-forget Stop Monitoring loop — for wedged or constantly-triggering units.
|
|
||||||
#
|
|
||||||
# Hammers POST /device/stop_monitoring_blind in a tight loop. The endpoint
|
|
||||||
# opens TCP, dumps SESSION_RESET + a few copies of the SUB 0x97 frame, and
|
|
||||||
# closes — without ever reading an S3 response. Each TCP-won attempt is
|
|
||||||
# ~50ms of wire activity instead of the multi-frame handshake the regular
|
|
||||||
# rescue endpoint does, so windows that are too small for the full rescue
|
|
||||||
# can still land a stop-monitoring command.
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# ./blind_stop.sh <host> [tcp_port]
|
|
||||||
#
|
|
||||||
# Env:
|
|
||||||
# SFM_BASE_URL Default: http://localhost:8200 (SFM direct).
|
|
||||||
# Set to http://localhost:8001/api/sfm to route through
|
|
||||||
# Terra-View's proxy.
|
|
||||||
# MAX_ATTEMPTS Default: 600
|
|
||||||
# SLEEP_S Default: 0 (no backoff — hammer it)
|
|
||||||
# MAX_TIME_S Default: 15
|
|
||||||
# CONNECT_TIMEOUT Default: 5
|
|
||||||
# REPEAT Frames per TCP session (default 3 — increases hit rate
|
|
||||||
# if the device is busy reading its own buffer).
|
|
||||||
# STOP_ON_OK Default: 1. Set to 0 to keep hammering indefinitely
|
|
||||||
# even after successful sends (every 503 means the device
|
|
||||||
# is in *another* session, every 200 means our bytes got
|
|
||||||
# through — but the device may not have processed them).
|
|
||||||
|
|
||||||
set -u
|
|
||||||
|
|
||||||
host="${1:-}"
|
|
||||||
tcp_port="${2:-9034}"
|
|
||||||
if [[ -z "$host" ]]; then
|
|
||||||
echo "usage: $0 <host> [tcp_port]" >&2
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
|
|
||||||
base="${SFM_BASE_URL:-http://localhost:8200}"
|
|
||||||
max_attempts="${MAX_ATTEMPTS:-600}"
|
|
||||||
sleep_s="${SLEEP_S:-0}"
|
|
||||||
max_time_s="${MAX_TIME_S:-15}"
|
|
||||||
connect_timeout="${CONNECT_TIMEOUT:-5}"
|
|
||||||
repeat="${REPEAT:-3}"
|
|
||||||
stop_on_ok="${STOP_ON_OK:-1}"
|
|
||||||
|
|
||||||
url="${base}/device/stop_monitoring_blind?host=${host}&tcp_port=${tcp_port}&connect_timeout=${connect_timeout}&repeat=${repeat}"
|
|
||||||
|
|
||||||
echo "blind_stop: target ${host}:${tcp_port} connect_timeout=${connect_timeout}s repeat=${repeat}"
|
|
||||||
echo "blind_stop: POST ${url}"
|
|
||||||
echo "blind_stop: up to ${max_attempts} attempts, ${sleep_s}s between, ${max_time_s}s per request"
|
|
||||||
echo "blind_stop: stop_on_ok=${stop_on_ok}"
|
|
||||||
echo
|
|
||||||
|
|
||||||
ok_count=0
|
|
||||||
busy_count=0
|
|
||||||
err_count=0
|
|
||||||
started=$(date +%s)
|
|
||||||
|
|
||||||
for ((i=1; i<=max_attempts; i++)); do
|
|
||||||
printf "[%4d] %s " "$i" "$(date +%H:%M:%S)"
|
|
||||||
http_code=$(curl -sS -o /tmp/blind_resp.$$ -w "%{http_code}" \
|
|
||||||
--max-time "$max_time_s" \
|
|
||||||
-X POST "$url" || echo "000")
|
|
||||||
body=$(cat /tmp/blind_resp.$$ 2>/dev/null || true)
|
|
||||||
rm -f /tmp/blind_resp.$$
|
|
||||||
|
|
||||||
case "$http_code" in
|
|
||||||
200|201)
|
|
||||||
ok_count=$((ok_count + 1))
|
|
||||||
echo "SENT $body"
|
|
||||||
if [[ "$stop_on_ok" == "1" ]]; then
|
|
||||||
elapsed=$(( $(date +%s) - started ))
|
|
||||||
echo
|
|
||||||
echo "blind_stop: success after ${i} attempts (${elapsed}s). ok=${ok_count} busy=${busy_count} err=${err_count}"
|
|
||||||
echo "blind_stop: NEXT — wait ~10s, then try the full rescue:"
|
|
||||||
echo " /home/serversdown/seismo-relay/scripts/rescue_device.sh ${host} ${tcp_port}"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
;;
|
|
||||||
503)
|
|
||||||
busy_count=$((busy_count + 1))
|
|
||||||
echo "busy (503)"
|
|
||||||
;;
|
|
||||||
000)
|
|
||||||
err_count=$((err_count + 1))
|
|
||||||
echo "curl error"
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
err_count=$((err_count + 1))
|
|
||||||
echo "HTTP $http_code $body" | head -c 400
|
|
||||||
echo
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
[[ "$sleep_s" != "0" ]] && sleep "$sleep_s"
|
|
||||||
done
|
|
||||||
|
|
||||||
elapsed=$(( $(date +%s) - started ))
|
|
||||||
echo
|
|
||||||
echo "blind_stop: gave up after ${max_attempts} attempts (${elapsed}s). ok=${ok_count} busy=${busy_count} err=${err_count}" >&2
|
|
||||||
exit 1
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user