serversdown/seismo-relay
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: serversdown/seismo-relay:archive/sfm-waveform-store
Branches Tags
serversdown/seismo-relay:main serversdown/seismo-relay:dev
serversdown/seismo-relay:archive/sfm-waveform-store serversdown/seismo-relay:archive/protocol-exp
..
compare: serversdown/seismo-relay:76bce0b5a34f8dbd768223fcb84fac2bc3f3b57a
Branches Tags
serversdown/seismo-relay:main serversdown/seismo-relay:dev
serversdown/seismo-relay:archive/sfm-waveform-store serversdown/seismo-relay:archive/protocol-exp

60 Commits
archive/sfm-waveform-store .. 76bce0b5a3

Author SHA1 Message Date
serversdown 76bce0b5a3 Merge pull request 'v0.20.0 - prerelease features.' (#25) from feat/wire-histogram-codec into dev 
- dockerfile fix
- histogram body codec FULLY decoded
- backfill scripts fixed.
- docs added for histogram codec
 2026-05-20 21:05:37 -04:00
serversdown 7183b953e4 minimateplus: histogram body codec — FULLY DECODED 
The histogram-mode event body is now byte-exact decodable.
Companion to the waveform body codec — together they cover every
event file the watcher forwards.  Cracked in one session via
cross-event correlation against BW's ASCII export.

The §7.6.2 spec in instantel_protocol_reference.md was structurally
correct (32-byte blocks) but the per-sample semantics were
under-documented.  Cross-checking block 130 of N844L6Z8.ZR0H
against its TXT row revealed the layout perfectly:

  slot[0] = 10 (constant marker)
  slot[1] = T_peak_count    (× 0.005 → in/s at Normal range)
  slot[2] = T_halfperiod    (freq_Hz = 512 / halfp)
  slot[3] = V_peak_count
  slot[4] = V_halfperiod
  slot[5] = L_peak_count
  slot[6] = L_halfperiod
  slot[7] = MicL_peak_count (dB via waveform_codec.mic_count_to_db)
  slot[8] = MicL_halfperiod

The `>100 Hz` sentinel is halfperiod ≤ 5 (since 512/5 = 100 Hz).
Mic dB uses the SAME formula as the waveform codec (sign × (81.94
+ 20·log10(|count|))) — they share the mic ADC calibration constant.

Block identification anchor: bytes [22:24] == 0x0000 AND
bytes [28:32] == 1e 0a 00 00.  The tail signature is the most
reliable distinguisher from non-block content in the file.

Files:

  minimateplus/histogram_codec.py (new) — decoder + public API
    matching the waveform codec's shape:
      walk_body(body) -> records
      decode_histogram_body(body) -> {Tran, Vert, Long, MicL}
      decode_histogram_body_full(body) -> [per-interval dicts]
      half_period_to_hz, geo_count_to_ins helpers

  minimateplus/event_file_io.py (modified) — read_blastware_file
    now tries the waveform codec first, falls back to the histogram
    codec on failure.  Same output shape, same downstream pipeline.

  tests/test_histogram_codec.py (new) — 24 regression locks against
    the in-repo fixture corpus, byte-exact against BW ASCII export
    for peaks (all 4 channels), frequencies (all 4 channels,
    including >100 Hz sentinel handling), block framing, and
    segment-ID accounting.

  scripts/backfill_sidecars.py (modified) — the has_samples
    short-circuit added in the histogram-pending era is now a
    pure defensive guard.  Histograms in prod will regen .h5 files
    correctly on the next backfill run.

  docs/histogram_codec_re_status.md (updated) — supersedes the
    earlier "in progress" version with the verified format and
    test-coverage summary.  Notes a few non-essential fields still
    open (4-byte block metadata, Geo PVS, Mic psi(L) — none of
    which are needed for waveform reconstruction).

Total verified coverage: ~3,500 blocks across 5 fixtures, every
field of every block byte-exact against BW.

The watcher-forwarded histogram event corpus on prod (~10,000
events) will now produce correct .h5 sidecars on the next backfill
run.  No additional changes needed to the backfill flow — the
existing tool_version-bump cascade picks them up automatically.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 23:05:13 +00:00
serversdown c3c7fe559c docs: histogram body codec RE — starting-point status doc 
Captures everything learned in the 2026-05-20 session before scope
forced a pause:

  - Block framing is solved: 32-byte blocks, one per histogram
    interval, signature byte pattern `[22:24]=0x0000` +
    `[28:32]=0x1e 0x0a 0x00 0x00` reliably identifies data blocks.
  - Block count = interval count (791 blocks in N844L20G.630H for
    a TXT-reported 792 intervals).
  - Sample[0] = Tran peak in 0.0005 in/s/count units (verified on
    one event — needs cross-event confirmation).
  - Samples 1-8 → channel/metric mapping is still open.  None of
    the obvious layouts (peak-then-freq alternating, all-peaks-
    then-all-freqs, per-channel 3-tuples) match the TXT values
    across multiple blocks.  Likely needs a higher-activity
    fixture (current N844 corpus is all noise-floor data) to
    disambiguate.
  - `>100 Hz` sentinel encoding in the binary is unknown.
  - 4-byte variable metadata field at block[24:28] needs
    correlation work against TXT columns.

Doc mirrors the structure of docs/waveform_codec_re_status.md so
a future RE session has a familiar entry point.  Includes the
suggested attack plan + the code seam where the eventual decoder
will land (minimateplus/histogram_codec.py).

The §7.6.2 spec in instantel_protocol_reference.md is structurally
correct but doesn't pin down per-sample semantics — this doc
supersedes it where they conflict on confidence level.

No code shipped on this branch.  When the codec is cracked, the
plan is to land minimateplus/histogram_codec.py + wire into
event_file_io.read_blastware_file() + remove the has_samples
short-circuit from scripts/backfill_sidecars.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 21:13:26 +00:00
serversdown fa9d3cdef2 read_blastware_file: leave peak_values=None when samples can't be decoded 
Fixes a data-loss bug discovered while dry-running the backfill against
the prod store.

Symptom: every histogram event in the store has its body decoded by
read_blastware_file → codec returns None → samples = empty dict →
``ev.peak_values = _peaks_from_samples(empty)`` returns
``PeakValues(0, 0, 0, 0, 0)`` (NOT None).  The backfill script's
existing "seed from DB row when peak_values is None" branch then
correctly *skips* the seeding, and the all-zeros PeakValues flows into
``db.insert_events()``'s UPSERT path, OVERWRITING the existing good DB
peak values for that event (which were populated from the paired BW
ASCII report at ingest).

Net effect: running the backfill on prod would have wiped the PPV /
mic / vector-sum columns for ~10,000 histogram events.

Fix: only compute peaks-from-samples when there are actually samples.
For events the codec couldn't decode (histogram-mode bodies, until
the §7.6.2 histogram codec is wired in), leave peak_values=None as
the "we don't know" signal.  Downstream consumers:

  - backfill_sidecars.py — its existing ``if ev.peak_values is None:``
    branch (line 243) seeds from the DB row, preserving the real
    BW-report peaks across the regen.
  - WaveformStore.save_imported_bw — apply_report_to_event overlays
    peaks from the paired BW ASCII report when one was uploaded.
    Histogram imports without a paired report end up with NULL peaks
    in the DB, which is correct (better than zeros — clearly says
    "no peak data available" rather than "peaks are exactly zero").

Updated the existing synthetic-event round-trip test to expect
peak_values=None for the no-real-body case, which is the truth now.

The 7 fixture-corpus regression tests for real BW waveforms continue
to pass — those have decodable samples, so peak_values is still
populated from the codec output as before.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:30:53 +00:00
serversdown c4648c1959 scripts/backfill_sidecars: skip .h5 write when decoder returned no samples 
Discovered while dry-running the backfill on the prod store: ~10,000
of ~10,059 events are histogram-mode (filename extension `*H`), and
the waveform-body codec wired in via the previous commit doesn't
handle histogram-mode bodies — only the waveform-mode codec at
§7.6.1 is implemented; the histogram-mode codec at §7.6.2 of the
protocol reference is documented but no Python implementation
exists yet.

Without this guard, every histogram event's .h5 file would be
*replaced* with an empty one — strictly worse than today's
broken-int16-LE .h5 because any downstream viewer expecting
non-empty sample arrays would now error out instead of just
rendering wrong values.

Fix: after the decoder runs, check whether any channel has samples.
If not, skip the .h5 write entirely.  The sidecar still regenerates
(refreshing the tool_version stamp and any peaks/project info from
the DB row), but the existing .h5 is left untouched.

This is a *temporary* gate.  When the histogram codec lands (next
branch: `feat/wire-histogram-codec`), the has_samples check can be
removed and the backfill will then correctly regenerate all .h5
files, histogram and waveform alike.

Observed effect (dry-run on prod store, 10,059 events):
  - waveform events (~5%): "[DRY ] would write … + .h5 (would (re)write)"
  - histogram events (~95%): "[DRY ] would write … + .h5 (skipped-empty-samples)"
  - sidecar tool_version bump succeeds for both

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:16:31 +00:00
serversdown 0e89125495 docker: fix dockerfile to include scripts and micromate folders 2026-05-20 19:58:54 +00:00
serversdown fffb363b2b Merge pull request 'minimateplus: wire read_blastware_file to verified body codec' (#24) from feat/wire-codec-to-import-path into dev 
Reviewed-on: #24
 2026-05-20 15:26:15 -04:00
serversdown e8682d49ad scripts/backfill_sidecars: cascade h5 regen when sidecar is stale + bump TOOL_VERSION 
Two coupled changes that close the rollout gap left by the
read_blastware_file codec wiring:

1. minimateplus/event_file_io.py: bump TOOL_VERSION from 0.16.1 to
   0.20.0.  This is the version stamp the backfill script reads from
   each sidecar's source.tool_version field to detect "this sidecar
   was written before the current decoder shipped, regenerate it."
   Bumping past every value baked into existing prod sidecars flags
   them all as stale on the next backfill run — which is exactly what
   we want, since every pre-codec-wiring sidecar was written by the
   retracted int16-LE decoder.

2. scripts/backfill_sidecars.py: when the sidecar is being
   regenerated this iteration (sha mismatch, tool_version too old,
   or --force), also regenerate the .h5.  Previously the .h5 logic
   only rewrote when --force was passed or the file was missing —
   so a tool_version-driven sidecar regen left the broken .h5 in
   place forever.  Added a `sidecar_stale` boolean to track the
   "we're rewriting the sidecar this iteration" state and wired it
   into the h5 need-rewrite check.

   Path coverage (verified by trace):
     - sidecar missing  → both regen
     - --force          → both regen
     - sha mismatch     → both regen
     - tool_ver too old → both regen (THE post-codec-wiring case)
     - everything OK    → skip iteration entirely (h5 untouched)

Operator review state (review.false_trigger, reviewer, notes) and
the sidecar's extensions block are preserved across regen by the
existing read-existing-sidecar / pass-into-event_to_sidecar_dict
path — unchanged from prior behavior.

Deploy procedure (on prod):
  1. Pull this change + the read_blastware_file codec wiring.
  2. `python scripts/backfill_sidecars.py --dry-run` to preview.
     Every sidecar with source.tool_version<0.20.0 will show as
     "would (re)write".
  3. Run for real (drop --dry-run).  Expect every pre-fix event
     to regen.  Big stores may take a while.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 18:24:06 +00:00
serversdown 31d691b40b minimateplus: wire read_blastware_file to verified body codec 
`read_blastware_file()` was still calling `_decode_samples_4ch_int16_le`
(the retracted int16-LE-interleaved hypothesis) on the body bytes,
producing ±32K noise on every channel of every BW file read from disk.
This was the path watcher-forwarded events take into the system
(via the import endpoint → save_imported_bw → read_blastware_file,
since the watcher doesn't ship A5 frames), so every .h5 sidecar
generated for a forwarded event has been wrong since the feature
shipped.

The fix is mechanical: pass the body bytes straight to
`waveform_codec.decode_waveform_v2()` and run the result through
`decoded_to_adc_counts()` for the 16x geo scaling.  The body already
starts with the codec's exact 7-byte preamble `00 02 00 [Tran[0] BE]
[Tran[1] BE]` — confirmed by `body[:3].hex()` across all 9 fixture
events.  No body-slice adjustment needed.

If the codec returns None (truncated/malformed file, synthetic test
input with no real waveform), fall back to empty channels with a log
warning.  The rest of the event (timestamp, waveform_key, project
strings, sensor_location, peaks-from-samples=0) is still recoverable.

Verified against the bundled fixture corpus:

  V70  Tran/Vert/Long 3328/3328 sample-sets match .TXT ground truth
       within the 0.005 in/s display quantum, every row
  6S0/RG0/AB0/470 (5-8-26)  3328/2304/1280/1280 samples; Vert PPVs
       match BW's own report within 0.02 in/s
  JQ0  3328 samples, Vert PPV 3.384 vs BW 3.465
  SP0/SS0/SV0 (loud events)  3072–3328 samples; known walker
       tail-truncation 1–7 samples per channel, samples reached are
       byte-exact

Existing `test_read_blastware_file_round_trip` (synthetic empty event)
continues to pass thanks to the None-fallback.  Codec verify scripts
(`analysis/verify_quiet_bundle.py`, `analysis/verify_full_decode.py`)
re-run unchanged.

Added two regression-lock tests in tests/test_event_file_io.py:
  - test_read_blastware_file_decodes_via_codec[6 fixtures]
    — verifies sample count + Vert PPV per fixture
  - test_read_blastware_file_v70_samples_match_txt_truth
    — verifies every one of V70's 3328 sample-sets across Tran/Vert/Long
      matches the .TXT ground truth row-by-row within 0.003 in/s

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 18:13:24 +00:00
serversdown beca5de06e docs: clean up and verify s3 protocol docs 2026-05-20 17:55:02 +00:00
serversdown d85df4c886 Merge pull request 'merge full s3 codec decoded' (#23) from codec-re into main 
Reviewed-on: #23
 2026-05-20 13:45:32 -04:00
Claude 0466bb4f44 codec: crack wide-NN blocks (1X NN / 2X NN); loud events now fully decode 
When NN exceeds 0xFC, the codec extends to 12-bit NN by using the
low nibble of the TYPE byte as the high nibble of NN:

    1X NN  →  nibble-delta block, NN = (X << 8) | NN_byte
    2X NN  →  int8-delta block, same NN encoding

Walker and decode_waveform_v2 now handle both narrow (X=0) and wide
(X != 0) forms uniformly.

Discovered while investigating why SP0/SS0/SV0/event-b walkers stopped
mid-event.  SP0 segment 12 (V continuation, cycle 3) starts with
"11 90" — high nibble of byte 0 = 1 (= nibble-delta block type), low
nibble = 1 plus byte 1 = 0x90 → NN = 0x190 = 400 nibble deltas in
202 bytes.  Walker was rejecting "11" as a non-tag.

Sample count went from 47,364 to 72,972 verified byte-exact:

  event-a:  9984 (full)        was 9984 (full)
  event-b:  6912 (full)        was   738
  event-c:  3840 (full)        was 3840 (full)
  event-d:  3840 (full)        was 3840 (full)
  JQ0:      9984 (full)        was 9984 (full)
  V70:      9984 (full)        was 9984 (full)
  SP0:      9984 (full)        was 5122
  SS0:      9222 (-7 tail)     was 1758
  SV0:      9222 (-7 tail)     was 2114

7 of 9 fixtures now decode end-to-end across all 3 geo channels.
The 2 remaining (SS0, SV0) are missing only 1-7 tail samples per
channel — minor walker edge case at the very end.

74 tests pass (was 71).
 2026-05-20 17:28:54 +00:00
Claude 85f4bcfe86 codec: wire decode_waveform_v2 into production; add MicL dB helper 
Replaces the broken legacy int16 LE decoder in client.py with the
verified multi-channel codec.  Three changes:

1. blastware_file.extract_body_bytes(a5_frames) — new helper that
   factors out the body-reconstruction logic from write_blastware_file
   so both writers (BW binary) and decoders (sample arrays) can use
   the same canonical bytes.

2. waveform_codec.decode_a5_frames(a5_frames) — production entry point.
   Returns the raw_samples dict consumers expect (Tran/Vert/Long as
   int16 ADC counts; MicL as native ADC counts).  Internally:
     A5 frames → extract_body_bytes → decode_waveform_v2
                → decoded_to_adc_counts (geos ×16; mic pass-through)

3. waveform_codec.mic_count_to_db(count) — MicL ADC → dB(L) per BW's
   display formula:
     dB = sign(count) × (81.94 + 20 × log10(|count|))   for |count| ≥ 1
   Verified against V70 fixture: count=813 → 140.14 dB (BW PSPL 140.1).

client.py:_decode_a5_waveform is reduced to a thin wrapper that calls
decode_a5_frames and populates event.raw_samples.  Original implementation
preserved as _decode_a5_waveform_LEGACY (dead code; reference only).

Also fixed a tail-end bug in decode_waveform_v2 where trailer-section
"40 02" markers (containing ASCII serial bytes, NOT real segment headers)
were being mis-interpreted, producing 2 spurious samples per channel at
the end of each event.  Added bytes [12:14] == "02 00" validation to
reject non-header markers.

7 new pytest tests cover the new helpers and dB conversion.  Total:
71 passing (up from 64).

Known limitation (carried over from before): the walker still stops
mid-event on the loudest fixtures (SP0/SS0/SV0/event-b) at some
mid-segment edge cases not yet characterized.  Every sample reached
is decoded correctly; the walker just doesn't reach all of them.
Loud events still yield 5,000–15,000 byte-exact samples each.
 2026-05-20 17:28:54 +00:00
Claude 2ff2762eec codec-re: 30 NN block CRACKED — codec fully decoded 
User intuition (16-bit) + 12-bit packing hypothesis + the int16 ADC
range constraint led to the final piece.

30 NN block format (CONFIRMED across all 14 blocks in the fixture
bundle):

  NN 12-bit signed deltas packed as NN/4 groups of 6 bytes each.
  Within each group:
    bytes [0:2] = 16 bits = 4 × 4-bit high nibbles (MSB-first)
    bytes [2:6] = 4 × int8 low bytes
    delta[k] = sign_extend_12((high_nibble[k] << 8) | low_byte[k])

  Block length = NN × 1.5 + 2 bytes (tag included).  Earlier walker
  used NN × 4 which is only correct in the TRAILER section.

Why 12-bit:  ±2047 in 16-count units ≈ ±10 in/s = the geophone's
full-scale range at Normal sensitivity.  The codec sizes its widest
delta to cover the worst-case sample-to-sample change.

Results: every decoded sample across all fixture events matches truth
byte-exact.  ZERO divergences.

  event-a:  9984 samples (full event, all 3 geos)
  event-c:  3840 (full event)
  event-d:  3840 (full event)
  JQ0:      9984 (full event)
  V70:      9984 (full event)
  SP0:      5122 (walker stops early on edge cases)
  SS0:      1758
  SV0:      2114
  event-b:   738

  TOTAL: 47,364 ADC samples verified, zero errors.

Three full 3-sec events decode end-to-end across all three geo
channels.  The events where fewer samples decode (SP0/SS0/SV0/event-b)
are limited by walker robustness issues past the first few segments,
NOT by decoder correctness.

64 tests pass (up from 55).  Files: minimateplus/waveform_codec.py
(new 30 NN decode + corrected walker length), tests/test_waveform_codec.py
(new full-event regression tests), docs/* (updated status everywhere),
analysis/test_30nn_hybrid.py (new — the analysis script that confirmed
the format).
 2026-05-20 17:28:54 +00:00
Claude d4cdce77fa codec-re: 30 NN partial finding — sum matches but per-sample distribution doesn't 
Tested the 12-bit signed packed delta hypothesis (motivated by the
observation that ±2047 in 16-count units ≈ ±32K raw ADC counts, almost
exactly the int16 ADC range — a strong design hint).

Result: mixed.  For SP0 block @1689 (V seg 4, samples 650..653):
  truth deltas:                47, 297, 384, 61   (sum = 789)
  12-bit BE contiguous pred:   17,  47, 664, 61   (sum = 789)

Positions 1 and 3 of the pred match truth values at positions 0 and 3
exactly, AND the total sum across all 4 positions matches.  But
positions 0 and 2 of pred don't match any truth value.

Hypothesis space narrows to:
- 12-bit deltas WITH a specific re-ordering or interleaving
- 12-bit deltas with one of the positions being a "step size" or
  "checksum-like" repacked value
- A nonlinear / coded format where the underlying total displacement
  is preserved but per-sample distribution is encoded differently

Two analysis scripts committed (test_30nn_12bit.py, test_30nn_v2.py).
The v2 script uses a real-decoder simulation to get the exact channel
+ sample-index for each 30 NN block, eliminating off-by-one errors in
the truth lookup.
 2026-05-20 17:28:54 +00:00
Claude ce5dc640ba codec-re: quiet bundle decodes FULLY (17k samples, zero errors) 
User asked the right question: do events without 30 NN blocks decode
fully?  Answer: YES.

  event-a:  Tran 3328 ✓  Vert 3328 ✓  Long 3328 ✓  (28 segments, 0 '30 NN')
  event-c:  Tran 1280 ✓  Vert 1280 ✓  Long 1280 ✓  (12 segments, 0 '30 NN')
  event-d:  Tran 1280 ✓  Vert 1280 ✓  Long 1280 ✓  (12 segments, 0 '30 NN')

17,664 ADC samples decoded byte-exact against BW's ASCII export.
Zero divergences across event-a, event-c, event-d.

This means the codec is FULLY SOLVED for any event without 30 NN
blocks.  The remaining gap is the 30 NN block format only — used for
high-amplitude regions where deltas exceed int8 range.  For quiet
events (or quiet stretches of loud events), the decoder is complete.

9 new regression tests bring the total to 55, all passing.

Files: tests/test_waveform_codec.py + docs/waveform_codec_re_status.md
+ new analysis/verify_quiet_bundle.py.
 2026-05-20 17:28:54 +00:00
Claude 07675626dc codec-re: channel rotation CONFIRMED — full multi-channel decoder works 
The segment-channel scoring analyzer (from scratch/next_experiment_skeleton.py)
ran and immediately confirmed the rotation hypothesis:

  SP0 seg 0: best fit Vert  508/508  ✓
  SP0 seg 1: best fit Long  508/508  ✓
  SP0 seg 3: best fit Tran  508/508  ✓  (Tran continuation)
  SP0 seg 5: best fit Long  508/508  ✓
  SP0 seg 9: best fit Long  508/508  ✓
  V70 seg 0: best fit Vert  508/508  ✓
  V70 seg 1: best fit Long  508/508  ✓

Channels rotate Tran → Vert → Long → MicL per 40 02 segment header.

Also discovered the segment header has DOUBLE duty: bytes [14:18] anchor
the NEW segment's channel (2 samples as int16 BE in 16-count units), AND
bytes [0:4] extend the PREVIOUS channel by 2 more samples (2 deltas as
int16 BE).  This is the same "2 anchors + delta stream" structure as the
body preamble for Tran.

decode_waveform_v2 now returns full per-channel sample dicts.
Byte-exact verified ranges:
  V70: Tran 512, Vert 512, Long 512   (all first segments)
  JQ0: Tran 512, Vert 258
  SP0: Long 1536 (all 3 L segments)

Still open: the 30 NN block format (high-amplitude packed deltas) —
appears mid-segment when single-byte deltas can't carry the magnitude.

6 new tests bring the count to 46.  All passing.
 2026-05-20 17:28:54 +00:00
Claude ae0e17b5dc codec-re: handoff polish — readmes, skeleton, remove decode-re/ duplicate 
Three things to make pickup smoother:

1. analysis/README.md (NEW): catalogues the ~25 scratch scripts.
   Categorizes them as "still useful" / "superseded — keep for
   archaeology" / "pure exploration".  Tells a fresh engineer which
   files to read first and which to ignore.

2. scratch/next_experiment_skeleton.py (NEW): stub + spec for the
   segment-channel scoring analyzer.  Includes the fixture loader,
   block walker, and decode-segment-as-channel helper — just enough
   scaffolding that the next pass starts from "fill in
   score_segment_against_all_channels()" rather than from scratch.
   Already runs and confirms 13 segments per 3-sec event with sample
   starts going to 6590 (way past the 3328 actual samples) — strong
   evidence that not all segments carry Tran.

3. Removed decode-re/ duplicate.  It was a mirror of tests/fixtures/.
   Analysis scripts that hardcoded decode-re/ paths updated to point
   at tests/fixtures/.  CLAUDE.md note updated: future event uploads
   go directly into a dated subdirectory under tests/fixtures/.

All 40 tests still pass.  Skeleton runs.
 2026-05-20 17:28:54 +00:00
Claude f68ee9f0f9 docs: clean up waveform-codec doc layers per review 
Three "truth layers" had drifted apart between commits.  Fixed:

1. waveform_codec.py docstring rewritten from the 2026-05-08
   "structural framing only" state to the 2026-05-11 "Tran segment 0
   solved + segment-header partially decoded" state.  Killed stale
   "~80 sample-sets per segment" language (real segments are
   flash-page-byte-sized, not sample-count-sized; observed first-segment
   sizes are 42-510 samples depending on signal).  Killed stale
   "preamble is 7 or 9 bytes" language (always 7).

2. docs/instantel_protocol_reference.md §7.6.1: added a clear
   "CURRENT STATUS" box at the top with a status table.  Replaced the
   stale "~80 sample-sets" line with the verified per-event segment
   sizes.  Merged two redundant segment-header field-table sections.

3. docs/waveform_codec_re_status.md (NEW): clean working-status doc.
   Solved / not solved / hypothesis / next experiment / fixtures /
   tests.  The protocol reference remains the historical Rosetta
   Stone; this new file is the current-truth working note that
   shouldn't accumulate fossil layers.

4. CLAUDE.md §"Waveform body codec": prominent warning box at top —
   "DO NOT TRUST decoded sample arrays yet."  BW binary passthrough
   is the only sample-bearing output to trust until the decoder
   lands.  Added a "Next experiment" subsection pointing the next
   pass at the segment-channel scoring analyzer.

40 tests still pass.
 2026-05-20 17:28:54 +00:00
Claude 5bf5329369 codec-re: add Waveform body codec section to CLAUDE.md 
Mirrors the structural findings now documented in
docs/instantel_protocol_reference.md §7.6.1: block framing solved,
Tran segment-0 decode verified across 5 fixture events, multi-segment
continuation still open. Also adds waveform_codec.py to the project
layout map.
 2026-05-20 17:28:54 +00:00
Claude 9ed6f2a8d8 codec-re: add segment 1 block dumper for analysis 
Investigated multi-segment Tran continuation but couldn't crack it.
Each hypothesis tried (segment header consumes 0/1/2 T deltas, blocks
continue Tran with various interpretations) breaks at sample ~512.

Block budget for V70 segment 1: 264 nibbles + 244 RLE zeros = 508
deltas — exactly the segment size. So the block structure CAN encode
508 single-channel samples, but applying segment 1 blocks as Tran
gives wrong values.

Most likely the channel ordering changes in segment 1+ (e.g., segment
0 = Tran, segment 1 = Vert, segment 2 = Long, etc.) but I couldn't
verify cleanly.  Stopping here — segment-0 Tran decode is solid and
multi-segment work needs more fresh thinking.
 2026-05-20 17:28:54 +00:00
Claude a0c9a482c7 codec-re: 00 NN is RLE; full Tran segment-0 decode (4 of 5 events) 
User uploaded a Vert-heavy event (JQ0) and a Mic-heavy event (V70).
Those two were exactly what was needed to crack the next piece:

- 00 NN block = run-length-encoded zero deltas in the current channel.
  Append NN copies of the current cumulative value (no change).
- find_data_start now recognizes 00 NN as a valid first tag (some events
  begin with a leading 00 NN RLE block).
- decode_tran_initial now decodes the FULL segment 0 (not just the first
  data block).

Results across 5 fixture events:
  - M529LL1A.SP0 (loud-all-channels)  : 510 / 510  ✓
  - M529LL1L.JQ0 (Vert-heavy)         : 510 / 510  ✓
  - M529LL1L.V70 (Mic-heavy)          : 510 / 510  ✓
  - M529LL1A.SV0 (loud-from-start)    :  58 /  58  ✓
  - M529LL1A.SS0 (loud-from-start)    :  42 / 502  (stops at first 30 04)

The 30 04 block (only seen in loud-from-start events) hasn't been
decoded yet — likely a channel-switch marker for the high-amplitude
regime.

Also discovered: segment header (40 02) payload bytes [0:2] = T_delta
at first sample of new segment, [6:8] = byte length to next segment.
Multi-segment Tran decoding still diverges after sample 512 because
the per-segment channel ordering after the header is unknown.

Tests: 40 pass (up from 36).

Files:
- minimateplus/waveform_codec.py: find_data_start fix, RLE handling,
  full segment-0 decode in decode_tran_initial
- tests/test_waveform_codec.py: synthetic RLE test, full segment 0
  tests for JQ0 and V70
- tests/fixtures/5-11-26/: M529LL1L.JQ0, M529LL1L.V70 + TXT exports
- docs/instantel_protocol_reference.md §7.6.1: RLE + segment-header docs
 2026-05-20 17:28:54 +00:00
Claude 6ac126e05c codec-re: crack Tran channel codec with high-amplitude May 11 bundle 
User uploaded 3 high-amplitude events (PPV 6-7 in/s — shook the geophone
hard) to decode-re/5-11-26/.  These cracked the Tran codec:

- Preamble bytes [3:5] and [5:7] = Tran[0] and Tran[1] as int16 BE
  in 16-count units (LSB = 0.005 in/s).  Confirmed across all 7
  fixtures.
- First data block carries Tran deltas from sample 2 onward:
  * 10 NN block: NN/2 bytes of payload, each byte = two 4-bit signed
    nibble deltas (high nibble first)
  * 20 NN block: NN int8 signed deltas

Verified 22+42+46 = 110 Tran samples across SP0/SS0/SV0 with 0 errors
against BW's ASCII export.

Why the earlier 96-combination brute force failed: the quiet 5-8
events all had T[0] = T[1] ≈ 0 so the preamble's per-channel encoding
was undetectable.  Loud events made the encoding obvious.

What's solved:
- minimateplus.waveform_codec.decode_tran_initial: returns first
  N Tran samples in 16-count units for any body.
- Walker length formula for in-data 30 NN blocks (NN*2 instead of NN*4).
- Walker now handles bodies that start with 20 NN (in addition to 10 NN).

What's still open:
- Tran past the first data block (multi-block channel switching).
- Vert / Long / MicL channel encodings.
- Walker correctness past offset ~427 in event-b.

Tests: 36 pass.  decode_waveform_v2 still returns None — the full
multi-channel decoder is not wired up.  decode_tran_initial is the
new verified entry point.

Files: minimateplus/waveform_codec.py, tests/test_waveform_codec.py
(adds 5-11-26 fixtures + decode_tran_initial tests), and
docs/instantel_protocol_reference.md §7.6.1 (Tran codec spec).
 2026-05-20 17:28:54 +00:00
Claude d3f77d1d96 codec-re: solve waveform body block framing; per-byte sample mapping still open 
Decoded the structural framing of the Blastware waveform body — the bytes
between the 21-byte STRT record and the 26-byte file footer.  The body is
a sequence of tagged variable-length blocks, NOT raw int16 LE.  Five tag
types (10/20/00/30/40 NN) and their lengths are now confirmed against the
4-event May 2026 fixture bundle.  Body splits cleanly into ~16 segments
(for a 1280-sample event) separated by 40 02 segment headers carrying a
monotonically incrementing uint32 LE counter at bytes [8:12].

What's done:
- minimateplus/waveform_codec.py — block walker, segment splitter, segment
  header parser.  decode_waveform_v2 is a stub returning None until the
  byte-to-sample mapping is solved; client.py is unchanged.
- tests/test_waveform_codec.py — 31 tests covering block detection, lengths,
  contiguous-walk, segment splitting, segment-header parsing, and counter
  monotonicity.  All pass.
- tests/fixtures/decode-re-5-8-26/ — bundled fixtures (4 events, BW binary
  + Blastware ASCII export each).
- docs/instantel_protocol_reference.md §7.6.1 — replaced retraction box
  with the verified structural decoding plus an explicit list of what's
  still open.

What's still open: the per-byte mapping inside 10 NN / 20 NN blocks.  96
channel-permutation × nibble-order × sign-convention combinations were
brute-force tested; none match BW's ASCII export to within ±1 ADC count.
The codec is more elaborate than uniform 4-bit deltas — likely a hybrid
variable-bit-width scheme with segment-anchor resync points.  Next
recommended step: capture an event with a known calibration tone to pin
down magnitude scaling.

Walker also bails out partway through event-b (open issue documented in
both the module and the protocol reference).
 2026-05-20 17:28:54 +00:00
serversdown 7bd0f8badf Pull in v0.18 - Merge branch 'main' into codec-re 2026-05-20 16:50:03 +00:00
Claude 8316a1bbd8 docs(protocol): accuracy sweep across the protocol reference 
Three-pass audit of docs/instantel_protocol_reference.md against
CLAUDE.md and the minimateplus/ implementation. Closes long-standing
discrepancies that had accumulated as the protocol understanding
evolved month over month.

Major corrections:
- §2/§3: S3 frames terminate on bare ETX, not DLE+ETX; payload
  byte[1] is flags / byte[2] is SUB (was wrongly DLE/ADDR).
- §4.2: probe responses do not carry data length; DATA_LENGTH
  is a per-SUB hardcoded constant.
- §5.1: dropped stale duplicate "SUB 1C = TRIGGER CONFIG READ"
  row; SUB 0A lengths corrected from 0x30/0x26 to 0x46/0x2C.
- §5.3: added the missing write-frame mechanics (BW_CMD-only
  doubling, DLE-aware checksum, offset = data[1]+2, ack format,
  SUB 71 chunk parameters).
- §7.6.x: switched compliance-anchor convention from the unstable
  10-byte form to the canonical 6-byte `\xbe\x80\x00\x00\x00\x00`;
  recording_mode confirmed at anchor−8 in both read and write
  (the prior anchor−3/−4 split caused anchor drift on write).
  Sample_rate at anchor−6, histogram_interval at anchor−4 (now ✅),
  record_time at anchor+6. Geo_range row added at channel_label+33.
- §7.5b/§8: added the 10-byte sub_code=0x03 continuous-mode
  timestamp variant; peak vector sum location corrected from
  fixed offset 87 to label-relative tran_pos−12.
- §7.7.2: SUB 1E/1F token byte at params[7], not params[6].
- §7.7.3: SUB 0A length disambiguation rewritten.
- §7.8.4/§7.8.7: fi==9 skip marked FIXED; metadata-page TODO
  replaced with current decoder state.
- §11: POLL example wire bytes corrected; SUB 5A row added to
  checksum table.
- §13/§14: device-under-test updated to BE11529/S338.17; TCP
  Idle Timeout consistency fix (0→2 min); Data Forwarding
  Timeout units clarified.
- §15 (renumbered from second §14): open-question entries
  already resolved in CLAUDE.md closed out.
- Appendix D: extension taxonomy rewritten — extensions encode
  a timestamp (AB0T scheme), not recording mode.

Navigation note added to §7 acknowledging the organic-growth
duplicate section numbers (§7.5/§7.5b, §7.6, §7.7, §7.8, §7.9)
and pointing readers to the canonical sections for each topic.

https://claude.ai/code/session_019tWZybD94YUsBaEGhnM5A2
 2026-05-20 15:41:42 +00:00
serversdown 8f568b809b Merge pull request 'v0.19.0 - minimate compatability + family separation' (#22) from dev into main 
## v0.19.0 — 2026-05-20

The "device-family separation" release.  Tightens the boundary between Series III (MiniMate Plus / Blastware) and Series IV (Micromate / Thor) so the UI and storage layer dispatch deterministically by family instead of sniffing filename extensions or magnitude heuristics.

### Added — Phase 1: `device_family` column on `events`

- **`events.device_family TEXT`** — new column carrying `"series3"` or `"series4"`.  Populated by every import path (`/db/import/blastware_file`, `/db/import/idf_file`, ACH server, BW CLI, sidecar backfill script).  Returned through `/db/events` since `query_events` uses `SELECT *`.
- **Self-applying migration** — on startup, `ALTER TABLE ... ADD COLUMN` lands the new column; a follow-on `UPDATE` backfills existing rows from the binary filename extension (`.IDFH`/`.IDFW` → `series4`, everything else → `series3`).  No manual SQL needed.
- **UPSERT preserves family** — re-imports without an explicit family don't blank existing rows (`COALESCE(?, device_family)`).
- **UI dispatches on the column** — `sfm_webapp.html` events-table mic formatter now branches on `ev.device_family === 'series4'` (Thor stores native dB(L); BW stores psi).  Modal uses `source.kind === 'idf-import'` from the sidecar (sidecars don't carry the DB column).  Source-files section labels changed from "BW filename / BW filesize / BW sha256" to format-neutral "Event file / File size / File sha256".

### Added — Phase 2: `micromate/` package alongside `minimateplus/`

- **`micromate/`** — new sibling package for the Thor / Micromate Series IV device.  Currently scoped to offline-file ingest; live-device support (TCP transport, framing, protocol, client) will land here when reverse-engineering happens.
  - `micromate/idf_ascii_report.py` — moved from `sfm/idf_ascii_report.py`.  No behaviour change.
  - `micromate/models.py` — typed `IdfReport`, `IdfEvent`, `IdfPeaks`, `IdfProjectInfo`, `IdfSensorCheck`.  Stores mic in native `mic_pspl_dbl` (dB(L)) instead of the pseudo-psi shoehorn that the BW-shaped model uses.  `IdfEvent.from_report()` constructs from a parsed dict + filename; `IdfEvent.to_minimateplus_event(waveform_key)` bridges to the existing sidecar / DB-insert machinery.
  - `micromate/idf_file.py` — placeholder for the binary codec (`.IDFH` / `.IDFW`).  Stubbed `read_idf_file()` raises `NotImplementedError`; documents the planned reverse-engineering path.
- **`WaveformStore.save_imported_idf`** refactored to use the native `IdfEvent` and bridge at the SQL-insert boundary.  Cleaner separation of "parse a Thor event" (in `micromate/`) from "store it on disk + write a sidecar" (in `sfm/waveform_store.py`).
- **Tests** — `tests/test_idf_ascii_report.py` imports updated to `micromate.idf_ascii_report`.  All 1,014 example-data sidecars round-trip through `IdfEvent.from_report()` without errors.

### Companion releases

- **thor-watcher** unaffected — it talks to the relay over HTTP only.  No version bump needed.
- **terra-view** unaffected today; can use `device_family` in its event-detail rendering when convenient.

---

## v0.18.0 — 2026-05-19

The "Thor / Series IV ingest adapter" release.  Seismo-relay can now accept event files from Instantel Micromate Series IV (Thor) units alongside the existing MiniMate Plus (Series III) Blastware pipeline.

### Added — Thor (Series IV) IDF ingest

- **`POST /db/import/idf_file`** (`sfm/server.py`) — multipart upload endpoint for `.IDFH` (histogram) and `.IDFW` (waveform) event files plus their `.IDFH.txt` / `.IDFW.txt` ASCII sidecars.  Mirrors the shape of `/db/import/blastware_file`: pairing by filename, optional `serial` query hint, per-file outcome reporting.
- **`sfm/idf_ascii_report.py`** — parser for Thor's TXT sidecars (verified against 1,014 real-world samples).  Extracts device-authoritative PPV, ZC Freq, Peak Vector Sum, Mic PSPL, calibration date, firmware version, sensor self-check results, and project/client/operator strings.
- **`WaveformStore.save_imported_idf()`** (`sfm/waveform_store.py`) — stores Thor binaries verbatim in `<root>/<serial>/<filename>`, writes a `.sfm.json` sidecar with `source.kind = "idf-import"` and the full parsed report under `extensions.idf_report`.  Reuses the existing `events` table — Thor events dedupe on (serial, timestamp) and surface in `/db/events` alongside BW events.
- **`tests/test_idf_ascii_report.py`** — parser tests against the `thor-watcher/example-data/` corpus.

### Changed

- `event_to_sidecar_dict()` (`minimateplus/event_file_io.py`) allow-list for `source_kind` now includes `"idf-import"` so the existing sidecar machinery can carry Thor imports.
- Bumped `pyproject.toml` version to `0.18.0`.

### Companion release

This release ships alongside **thor-watcher v0.3.0**, which adds the SFM forwarder that targets the new `/db/import/idf_file` endpoint.  Operators flip the switch in thor-watcher's new "SFM Forward" Settings tab; events POST to seismo-relay just like the series3-watcher BW forwarder does today.
 2026-05-20 11:22:54 -04:00
serversdown ecc935482b seismo-relay v0.19.0 — device-family separation + micromate/ package 
Tighten the Series III / Series IV boundary so UI and storage dispatch
on a clean signal instead of sniffing filenames or applying magnitude
heuristics.

Phase 1 — events.device_family column ("series3" | "series4"):
  self-applying migration with filename-based backfill of existing rows
  (1,132 backfilled on prod 2026-05-20); plumbed through every import
  path (BW endpoint, IDF endpoint, ACH server, BW CLI, sidecar
  backfill); UPSERT preserves via COALESCE; UI dispatches on it.

Phase 2 — extract micromate/ package alongside minimateplus/:
  native IdfEvent / IdfReport / IdfPeaks / IdfProjectInfo /
  IdfSensorCheck (mic in dB(L), not pseudo-psi); moved
  idf_ascii_report.py from sfm/ to micromate/; refactored
  save_imported_idf to use IdfEvent and bridge to minimateplus.Event at
  the SQL-insert boundary; idf_file.py stub for the future binary codec.

Phase 3 prep — docs/idf_protocol_reference.md captures the two
observed Thor binary header signatures (1,012 newer-firmware files vs
2 old files whose layout is byte-for-byte BW-STRT-compatible), file-size
hints suggesting int8 sample encoding, open questions in dependency
order, and a concrete first-session plan for cracking the codec.

Also rolled in the v0.18.1 hotfixes that motivated this work:
  - idf_ascii_report parser now handles "<0.005 in/s" (below-threshold)
    and "N/A" markers without leaving raw strings in numeric DB columns.
  - sfm_webapp.html: defensive _ppvFmt / mic formatter so future
    data-shape drift can't kill the whole events table render.

All 1,014 example-data sidecars round-trip through the new package.
See CHANGELOG.md for full notes.
 2026-05-20 15:19:49 +00:00
serversdown e95ac692ee feat: add device family to separate s3 and s4 events. 2026-05-20 06:15:50 +00:00
serversdown 3265ad6fa3 fix: apply psi dbL conversion rule 2026-05-20 05:43:52 +00:00
serversdown 350f81f8b5 fix: add thor specific ascii parser. 2026-05-20 05:22:28 +00:00
serversdown cd20be2eff feat: add thor/micromate compatibility v0.18.0 2026-05-19 04:32:43 +00:00
serversdown f7c5c9fed3 Merge branch 'main' into codec-re 2026-05-17 23:30:29 +00:00
serversdown 512d82c720 merge: update to 0.17.0' (#21) from ach-report-ingestion into main 
Reviewed-on: #21

## v0.17.0 — 2026-05-17

The "field rescue + DB management" release.  Hardened against units that are stuck in a runaway call-home loop, and added an operator-facing path for purging bogus events that those same units dump into the DB before recovery.  All work in this release was driven by the BE9558H incident (full incident log + recovery procedure at `docs/runbooks/wedged_unit_recovery.md`).

### Added — wedged-unit recovery toolkit

A toolkit for breaking the call-home loop on a misbehaving unit whose firmware is too busy to keep up with normal request/response handshakes.  Tested in production against BE9558H (16 May 2026) — a unit with a stuck-triggered Long-axis geophone that had been call-homing the office BW ACH server every 30 seconds for hours.  Endpoints layered from "single attempt" to "siege mode" to suit different contention levels:

- **`GET /device/events/storage_range`** — SUB 0x06 probe.  POLL + one read; ~2s.  Returns first/last event keys and an `is_empty` flag.  Use to triage whether a unit has stored events without invoking the slow `count_events()` 1E/1F chain (which choked on BE9558H's corrupted event chain).
- **`GET /device/events/index`** — SUB 0x08 probe.  POLL + one read; ~2s.  Returns the lifetime event counter (does NOT decrement on erase — use `storage_range` for "right now" state).
- **`POST /device/events/erase`** — full erase sequence `0xA3 → 0x1C → 0x06 → 0xA2` (confirmed 2026-04-11, see the protocol reference).  Resets event keys to `0x01110000`.  Caller's responsibility to disable ACH first if the underlying trigger condition will re-fill the buffer.
- **`POST /device/rescue`** — one TCP session, short connect+recv timeouts: POLL → disable ACH (compliance config write) → erase events → close.  Designed for race-loop usage when the device is busy in another session.  503 on connect-refused, 502 on protocol failure, 200 on full sequence success.
- **`POST /device/stop_monitoring_blind`** — fire-and-forget Stop Monitoring (SUB 0x97), TCP-only.  Dumps `SESSION_RESET + POLL_PROBE + SESSION_RESET + POLL_DATA + 0x97 × repeat` and closes without reading any S3 response.  The full POLL preamble is required — write commands without it are silently ignored by the device's protocol parser (false-positive surface area that bit the first version of this endpoint).  Use when the device's firmware can't keep up with full request/response but might process inbound bytes at its own pace.
- **`POST /device/stop_monitoring_spam`** — server-side hammer loop, duration-bounded.  Open TCP → write the same blind payload → close → repeat as fast as possible until `duration_s` elapses.  Configurable `connect_timeout` (default 500ms) and `repeat` (frames per session).  Reports `sent_ok`, `connect_failed`, `write_failed`, `rate_attempts_per_s`.  Clamped to 5min duration.
- **`POST /device/stop_monitoring_slow_drip`** — opposite of spam.  Open ONE TCP session, drip the wake handshake + stop frames at `interval_s` (default 3s) for `duration_s` (default 120s, max 10min).  Each drip is ~23 bytes — well under any UART FIFO size.  Opportunistically drains any inbound bytes the device sends back; `bytes_received > 0` in the response strongly suggests the device has started talking and the session is healthy.  **This is the endpoint that saved BE9558H.** Spam mode had been overrunning the device's UART FIFO; slow drip stayed under it.
- **Six rescue scripts** under `scripts/` — thin bash wrappers around the endpoints, default `SFM_BASE_URL=http://localhost:8200` (direct, not via Terra-View proxy whose 60s timeout would cut off the longer endpoints):
    - `rescue_device.sh` — race-loop wrapper for `/device/rescue`
    - `blind_stop.sh` — race-loop wrapper for `/device/stop_monitoring_blind`
    - `spam_stop.sh` — single-call burst hammer
    - `slow_drip.sh` — single-call held-session drip
    - `watch_unit.sh` — passive periodic reachability check (every N min, logs to file), useful for unattended overnight monitoring of a wedged unit
- **`docs/runbooks/wedged_unit_recovery.md`** — symptoms, quick-reference recovery procedure, the modem-layer mechanism (Sierra Wireless serial-port mode-flipping is the real failure mode — not the device firmware), and a table of "why simpler approaches don't work" so the next incident skips the dead ends.

### Added — operator event DB management

Endpoints powering Terra-View's new `/admin/events` page (v0.12.0).  Designed for purging bogus events from a unit that's been forwarding them in bulk (e.g. a stuck-triggered seismograph dumping hundreds of junk events before it's recovered).

- **`DELETE /db/events/{event_id}`** — hard-delete one event row.  Also unlinks the associated blastware binary (`.AB0*`), `.a5.pkl`, `.sfm.json` sidecar, and `.h5` clean-waveform files via the WaveformStore.  Returns the per-file removal status.  404 if the event doesn't exist.
- **`POST /db/events/delete_bulk`** — filter-based or id-list-based bulk delete with safety rails:
    - Filters (`serial`, `from_dt`, `to_dt`, `false_trigger`) combine with AND; same semantics as `GET /db/events`.  `ids` is an additional inclusion list.  Refuses to run with no filters (would wipe the whole table — raises 422).
    - `confirm` must be `true` to actually delete.  Otherwise returns a dry-run summary (`status: "dry_run"`, `matched: N`, `sample_serials: [...]`).
    - `max_rows` (default 10,000) caps how many rows can be deleted by-filter in one call.  If exceeded, returns `status: "too_many"` with a hint to narrow or raise the cap.  Bypassed when only `ids` is supplied.
- **`_cleanup_event_files(row)`** helper in `sfm/server.py` — best-effort `unlink()` of all four sidecar paths derived from the row's `blastware_filename`.  Logged at WARN if a path exists but unlink fails; the DB row deletion still proceeds.
- **`SeismoDb.delete_event(id)` and `SeismoDb.delete_events_bulk(...)`** in `sfm/database.py` — both return the deleted row dict(s) so callers can do file cleanup.  `delete_events_bulk` raises `ValueError` if no filters are supplied.

### Changed

- **Default protocol recv timeout dropped from 30s → 10s** in `_build_client()`.  The unit usually responds in well under a second over cellular; 10s leaves comfortable headroom for retransmits while failing reasonably fast when a unit is wedged.  The two endpoints that perform full 5A waveform downloads still pass `timeout=120.0` explicitly so multi-minute event transfers are unaffected.
- **`_build_client()` now accepts an optional `connect_timeout`** (TCP-only) so rescue / race-loop endpoints can fail fast on busy modems without affecting the protocol-level recv timeout.

### Fixed

- **`GET /device/monitor/status` returned HTTP 500 + uncaught traceback when the device was unresponsive**.  The retry-on-`Exception` inner block let the second `client.poll()`'s `ProtocolError` propagate out of the handler.  Now wrapped in proper try/except — returns 502 with `{"detail": "Protocol error: No S3 frame received within 10.0s ..."}` on timeout, 502 on connection errors, 500 only for genuinely unexpected exceptions.

### Migration

No schema changes.  No data migration required.

If you've been running a previous version against a wedged unit and accumulated bogus events, the new `/admin/events` page in Terra-View v0.12.0 (or direct `POST /db/events/delete_bulk` with `confirm: true`) is the cleanup tool.  Watcher state on the upstream DL2 PC does NOT need separate cleaning — the watcher's `sfm_forwarded.json` keys on file sha256 and won't re-forward the same files.

### Pairing

This release pairs with **Terra-View v0.12.0**, which adds the `/admin/events` UI that consumes the new bulk-delete endpoints, the bulk false-trigger flagging on `/unit/{id}`, and the field-deployment workflow that uses the same `series3-watcher` → SFM ingest path as before.

---

## v0.16.1 — 2026-05-14

### Fixed

- **`record_type` always "Waveform" for forwarded events.**  `read_blastware_file()` hardcoded `ev.record_type = "Waveform"` regardless of the file's actual type.  The watcher-forward pipeline (the main BW ACH ingest path) compounds this by parsing files from a tmp path with a `.bw` suffix, so even a filename-based fallback inside the parser still wouldn't see the original extension.  Now:

  1. New `derive_record_type_from_filename(filename)` helper in `minimateplus/event_file_io.py` derives the type from the LAST character of the filename's extension (V10.72+ AB0T scheme: `H`=Histogram, `W`=Waveform, `M`=Manual, `E`=Event, `C`=Combo).  Falls back to `"Waveform"` for old S338 firmware (3-char extensions ending in `0`) and any unrecognized suffix.
  2. `read_blastware_file()` now calls the helper with its `path.name` so direct callers (the `--dry-run` path in `scripts/import_bw.py`, tests, ad-hoc scripts) get the right value automatically.
  3. `WaveformStore.save_imported_bw()` overrides `ev.record_type` with the **original** filename's derived type after parsing (the tmp file inside the parser doesn't carry the original extension).  This is the path the live watcher-forwarder hits, so the DB column now reflects the actual event type going forward.

  Events ingested before this fix are stuck with `record_type="Waveform"` in the DB; a one-off backfill (`UPDATE events SET record_type = ... WHERE blastware_filename LIKE '%H'`) would fix them retroactively if desired.  Terra-view's event modal also derives client-side from the filename, so the UI already shows the correct type for old events even without the backfill.

---
 2026-05-17 19:13:56 -04:00
serversdown 57287a2ade chore: update to 0.17.0 2026-05-17 23:07:12 +00:00
serversdown 1fff8179d6 Add runbook for recovering wedged units and new scripts for device management 
- Created a comprehensive runbook (`wedged_unit_recovery.md`) detailing the recovery process for units stuck in a call-home loop, including symptoms, recovery steps, and explanations of the failure mode.
- Added `blind_stop.sh` script to send stop-monitoring commands in a tight loop for unresponsive devices.
- Introduced `rescue_device.sh` script to disable Auto Call Home and erase events from a busy device.
- Implemented `slow_drip.sh` script to send stop-monitoring frames at a slow rate to prevent UART overrun.
- Developed `spam_stop.sh` script to rapidly send stop-monitoring commands to a device.
- Created `watch_unit.sh` script for passive monitoring of device reachability, logging results over time.
 2026-05-17 07:58:13 +00:00
serversdown ae7edac83f chore(doc): bump to 0.16.1 2026-05-15 23:35:35 +00:00
serversdown b6911009ff scripts: backfill record_type on legacy events imported with hardcoded "Waveform" 
Pre-v0.16.1 (commit aac1c8e), every event ingested through
read_blastware_file got record_type="Waveform" regardless of actual
type because the field was hardcoded.  New ingests derive correctly
from the AB0T filename scheme (H/W/M/E/C).  Existing rows still hold
the wrong value.

This script walks the events table, derives the correct record_type
from each row's blastware_filename, and bulk-updates rows that differ.
Idempotent + dry-run by default.

Usage:
  python -m scripts.backfill_record_type --db bridges/captures/seismo_relay.db
  python -m scripts.backfill_record_type --db bridges/captures/seismo_relay.db --apply

Terra-view's event-detail modal already derives the record_type
client-side from the filename for display, so operators see the
correct type in the UI even before this backfill runs.  This script
brings the DB column in line with what the UI is already showing —
matters for reporting and any downstream consumer that reads the
column directly.
 2026-05-15 06:38:09 +00:00
serversdown aac1c8e06d fix(import): derive record_type from filename suffix instead of hardcoding "Waveform" 
The BW ACH ingest path was inserting every event with
record_type="Waveform" regardless of the actual type because
read_blastware_file() had `ev.record_type = "Waveform"` hardcoded, and
the live watcher-forward path parses files from a tmp path (suffix
".bw") that doesn't carry the original extension.

V10.72+ MiniMate Plus firmware encodes the event type as the last
character of the AB0T extension scheme (H=Histogram, W=Waveform,
M=Manual, E=Event, C=Combo).  This change:

  1. Adds derive_record_type_from_filename() public helper in
     minimateplus/event_file_io.py
  2. Uses it inside read_blastware_file() so direct callers (the
     --dry-run path of scripts/import_bw.py, tests, ad-hoc scripts)
     get correct types automatically
  3. Overrides ev.record_type in WaveformStore.save_imported_bw()
     using the ORIGINAL filename (source_path.name) — required
     because the parser sees only the tmp file

Old S338 firmware (3-char extensions ending in `0`) and any
unrecognized suffix fall back to "Waveform".

Existing DB rows ingested before this fix are stuck with
record_type="Waveform" — a one-off SQL backfill would fix them
retroactively if desired.  Terra-view's event modal also derives
client-side from the filename, so the UI already shows the correct
type for old events even without the backfill.

Version bumped to 0.16.1 in pyproject.toml, event_file_io.py
TOOL_VERSION, sfm/server.py FastAPI version, and CHANGELOG.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-14 21:09:21 +00:00
serversdown 84ee68f889 Merge branch 'main' into codec-re 2026-05-11 22:27:25 -04:00
serversdown 20519383fe add additional events for decode 2026-05-11 18:13:24 -04:00
serversdown 87675ac2d8 Merge pull request 'docker: add .dockerignore and Dockerfile for containerization.' (#20) from dockerize into main 
Reviewed-on: #20
 2026-05-11 17:40:56 -04:00
serversdown 83d69b9220 chore(server): update inline version to 0.16.0 2026-05-11 21:40:18 +00:00
serversdown 3e247e2182 docker: add .dockerignore and Dockerfile for containerization. 2026-05-11 21:38:03 +00:00
serversdown d2e48c62b5 Merge pull request 'feat(import): v0.16.0 - Fully implemented series 3 BW-ACH pipeline stablized.' (#19) from ach-report-ingestion into main 
Reviewed-on: #19
 2026-05-11 15:55:23 -04:00
serversdown 3402b4d11a add additional events for decode-RE 2026-05-11 14:17:21 -04:00
serversdown 988d26c03d docs: capture deferred work in README Roadmap 
Consolidates everything that was floating in chat-only "parking
lot" status into the README's Roadmap (Future) section:

  High-impact (unblocks product features):
    - Waveform body codec reverse-engineering
    - In-app waveform viewer accuracy (depends on codec)
    - Terra-view integration
    - Vibration summary reports

  BW ASCII report parser enhancements:
    - Histogram-specific structural fields
    - Histogram interval bin-table parsing
    - ">100 Hz" value parsing

  Ingestion gaps:
    - MLG forwarding (watcher + SFM endpoint)
    - 0C-record raw bytes persistence in sidecar

  Operational:
    - series3-watcher file archive manager
    - Existing operational items (compliance encoder, modem manager,
      Call Home dial_string write, histogram mode 5A stream)

  Test coverage + lower-priority cleanups.

CLAUDE.md "What's next" section now points to the README as the
canonical deferred-work list, and keeps its own low-level technical
status log for byte-layout details that don't belong in the
roadmap.
 2026-05-11 16:08:02 +00:00
serversdown 197c0630e2 chore(release): v0.16.0 — BW ACH ingestion 
The "BW ACH ingestion" release.  Paired with series3-watcher v1.5.0,
every Blastware ACH event (binary + _ASCII.TXT report) lands in
SeismoDb with device-authoritative peaks, project metadata, sensor
self-check, and ZC/Time-of-Peak data — without depending on the
still-undecoded waveform body codec.

Bumps pyproject.toml + minimateplus/event_file_io.py TOOL_VERSION
to 0.16.0.  README banner + CHANGELOG entry summarise the work
that landed across commits cdfe4ad..f83993a on this branch.
 2026-05-11 07:33:48 +00:00
serversdown f83993ad1d fix(import): pair _ASCII.TXT reports on the SFM server side too 
The series3-watcher v1.5.0 fix taught the WATCHER to look for BW
ACH's _ASCII.TXT report alongside each binary.  But the SFM
SERVER's import endpoint only knew about the legacy <binary>.TXT
naming when building its TXT lookup table.

Effect: even though the watcher correctly shipped both files in
the multipart POST (and logged "+ <name>_ASCII.TXT attached"),
the server's reports dict was keyed on the wrong name, so
report_bytes resolved to None for every event.  Without the
report, save_imported_bw fell back to broken-codec peak values
and no project info — exactly the same symptom as before the
watcher fix landed, just for a different reason.

Fix: when stripping the ".TXT" suffix, also recognise the
"_ASCII" trailer and reconstruct the binary's filename by
converting the last "_" back to ".".  Register the report under
BOTH possible binary names so the subsequent lookup matches
whichever convention the operator's BW installation uses.

  ACH convention (Blastware ACH):
    binary T003L2G6.0E0H  + report T003L2G6_0E0H_ASCII.TXT  ✅
  Manual export (operator clicks Save As Text in BW):
    binary M529LK44.AB0   + report M529LK44.AB0.TXT          ✅
  Both for same event (e.g. ACH + operator manual save):
    register under both names; binary lookup wins             ✅

Smoke-tested against the four real fixture filenames in the
project archive.  Full SFM suite still 62 pass.

For the user's situation: pull, restart, and the NEXT re-forward
pass (after deleting watcher state file again if needed) will
hit this code path, parse the report correctly, apply the
overlay onto the Event, and the upsert path will land
authoritative peak values + project info in the DB.
 2026-05-11 07:25:04 +00:00
serversdown 6b2a44ff02 fix(import): overlay BW report onto Event + upsert DB row on re-import 
Two compounding bugs caused forwarded events to land in the DB with
broken-codec peak values (~10 in/s saturation on every channel) and
no project info, even when the watcher correctly paired a BW ASCII
report with the binary.

Bug 1: save_imported_bw built the sidecar JSON with the report's
authoritative peak / project values via event_to_sidecar_dict(
bw_report=...), but never overlaid those onto the in-memory Event
that flows to db.insert_events().  So the DB row got peak_values
from read_blastware_file()._peaks_from_samples() — which runs the
still-undecoded waveform body codec assuming raw int16 LE and
produces ±32K-shaped noise (= ±10 in/s at Normal range) regardless
of the actual signal.  The sidecar JSON had the truth but the DB
columns (which the webapp queries for fast filter/sort) lied.

Bug 2: insert_events' IntegrityError handler only refreshed the
filename/filesize/a5_pickle/sidecar columns when a duplicate
(serial, timestamp) was seen.  Peak values, project info,
sample_rate, record_type stayed locked in at whatever the FIRST
insert wrote.  So even after Bug 1 was fixed, the historical
events in the DB (already inserted with broken-codec peaks) would
never get their values corrected, because a re-forward would just
hit IntegrityError and skip the field refresh.

Fix 1 (minimateplus/event_file_io.py + sfm/waveform_store.py):
  - New apply_report_to_event(event, report) helper folds the BW
    report's device-authoritative fields onto the Event in-place:
    per-channel PPV, peak vector sum, mic PSPL→psi, project /
    client / operator / sensor_location, sample_rate, record_time.
  - save_imported_bw() calls the helper right after parsing the
    report.  The Event that flows to insert_events() now carries
    correct values.

Fix 2 (sfm/database.py):
  - insert_events()'s IntegrityError UPDATE now refreshes every
    device-authoritative column from the new data: tran_ppv,
    vert_ppv, long_ppv, peak_vector_sum, mic_ppv, project, client,
    operator, sensor_location, sample_rate, record_type, plus
    the existing filename/filesize/a5_pickle/sidecar fields.
  - Preserves: id, waveform_key, session_id, created_at (immutable
    / FK fields), and false_trigger (operator review state).

End-to-end simulation verified:
  - Step 1: import without report → DB has ±10 in/s peaks, no project
  - Step 2: re-import WITH report → upsert path fires, DB now has
            device-authoritative 0.005 in/s peaks + sensor_location
  - Step 3: operator sets false_trigger=1, re-import again → flag
            preserved, peaks remain correct

For the user's situation: deleting the watcher state file forces a
re-forward of all events.  Each re-forward now pairs with its
_ASCII.TXT, applies the report onto the Event, and the upsert
refreshes the DB row.  No DB nuke needed.

Full SFM suite: 62 passed, 44 skipped.
 2026-05-11 05:51:39 +00:00
serversdown cc57a8e618 fix(db): /db/units surfaces events-only serials too 
Previous query_units() only joined on ach_sessions, which is created
exclusively by the live ACH server.  The BW-importer path
(/db/import/blastware_file → WaveformStore.save_imported_bw →
SeismoDb.insert_events) populates `events` but never creates an
ach_sessions row.  Consequence: every serial whose events flowed in
through the series3-watcher forwarder was invisible to
/db/units (and therefore to the SFM webapp's fleet overview / units
list), even though the events were correctly populated in the
events table with proper serial attribution.

Rewrite query_units() to aggregate from BOTH tables and union the
serials:
  - total_events / last_event_at  come from `events` (every ingest path)
  - last_session_at / total_monitor_entries / total_sessions
                                  come from `ach_sessions` (ACH-only),
                                  0 when no sessions exist for the serial
  - last_seen = max(last_event_at, last_session_at)

Verified on the user's actual prod DB after the
repair_unknown_serials run: /db/units now returns 24 serials instead
of 2.  All 3,257 watcher-forwarded events become visible in the
fleet overview without any further DB surgery.
 2026-05-11 05:15:09 +00:00
serversdown 082e5946bc fix(import): resolve real serial from BW filename instead of bucketing to UNKNOWN 
The /db/import/blastware_file endpoint was bucketing every
forwarded event into serial='UNKNOWN' in the DB.  WaveformStore
correctly decoded the serial from the BW filename and saved
files to <store>/<serial>/<filename> (e.g.
.../BE17353/S353L5KC.DR0H.h5), but the endpoint code called
db.insert_events(serial=_serial_from_event(ev)) — and
_serial_from_event was a stub that always returned None,
falling back to "UNKNOWN".

Effect on the user's prod server: 3,039 events forwarded across
24 distinct units, ALL inserted under serial='UNKNOWN'.  The
on-disk waveform store + sidecars + HDF5s were fine, but the
SFM webapp's /db/units only showed the two original manually-
uploaded serials because every forwarded row had its serial
column zeroed to UNKNOWN.

Fix:
  - WaveformStore.save_imported_bw() now surfaces the decoded
    serial on the returned `rec` dict (rec["serial"]).
  - The import endpoint uses rec["serial"] as the authoritative
    fallback when the operator hasn't supplied a serial_hint query
    parameter.  Order of precedence:
      query string `serial` → rec["serial"] → _serial_from_event(ev) → "UNKNOWN"
  - Response payload now includes `serial` per file so the watcher
    log lines (or any future caller) can see which unit each event
    was attributed to.

Recovery for existing DB rows:
  scripts/repair_unknown_serials.py walks the events table looking
  for rows with serial='UNKNOWN' and re-attributes each one to the
  serial decoded from blastware_filename.  Updates the row in place
  unless the target (serial, timestamp) already has a row, in which
  case the UNKNOWN duplicate is deleted.  Idempotent.  Default
  dry-run; pass --apply to commit.

  Verified on the user's actual DB (dry-run):
    UNKNOWN rows scanned:       3039
    Updated to real serial:     2602
    Deleted (duplicate of an
     already-correct row):      437
    Unresolved (bad filename):  0

After running the repair, /db/units will show all 24 units
correctly populated.
 2026-05-11 02:25:08 +00:00
serversdown a032fa5451 refactor(bw-report): parse user notes by POSITION, not by label 
The four operator-supplied note fields in BW's Compliance Setup →
Notes tab (Project / Client / User Name / Seis Loc) have
USER-EDITABLE LABELS — an operator can rename them in BW's UI to
"Building:", "Site Address:", "Inspector:", or anything else, and
the ASCII export writes those literal labels verbatim.  The
previous label-normalisation map approach (just added in commit
6a7e8c6) was fragile: it could only match label spellings we'd
enumerated in advance.  An operator using "Site:" instead of
"Seis Loc:" would have their sensor location silently dropped.

What IS reliable: BW always writes the 4 user-notes lines
contiguously, in the same order, between the "Units :" line and
the "Geo Range :" line of the export.  So parse them by POSITION:

  position 1 → project
  position 2 → client
  position 3 → operator
  position 4 → sensor_location

The original labels BW wrote are preserved in a new
`BwAsciiReport.user_note_labels` dict (canonical slot → literal
label string) so terra-view can render them as the operator named
them.

Removes the `_OPERATOR_LABEL_MAP` / `_normalise_label_for_lookup`
helpers and the elif-by-normalised-label branch in `parse_report`.
Replaces with a small state machine that flips on the "Units" line
and flips off on the "Geo Range" line.

Tests:
  - Default-label fixtures (waveform + histogram) still populate
    correctly, with operator's labels captured.
  - Synthetic custom-labelled exports ("Building:" / "Site Address:" /
    etc.) populate the right slots by position.
  - Histogram-specific "Seis. Location:" works.
  - Lines outside the Units→Geo Range range are ignored even if
    they look like user notes (defensive against malformed exports).
  - Partial blocks (fewer than 4 lines) leave later slots None.
  - Extra lines beyond 4 are dropped (5th slot doesn't exist).

26 tests in test_bw_ascii_report.py (was 33; net drop reflects
parametrised label tests collapsed into 6 focused position tests).
Full SFM suite: 62 passed, 44 skipped.

Pairs with series3-watcher v1.5.0 which fixes the filename pairing
so the report reaches this parser in the first place.
 2026-05-10 22:28:31 +00:00
serversdown 6a7e8c6e86 feat(bw-report): normalise operator-field label variants 
Blastware writes the operator-supplied fields with different label
spellings across firmware versions and recording modes — most
notably "Seis. Location" on histogram exports vs "Seis Loc:" on
waveform exports.  Previous parser only matched the latter, so
every histogram event silently lost its sensor_location field.

Replace the four hardcoded `key.rstrip(":") == "X"` branches with
a single `_OPERATOR_LABEL_MAP` dispatch table keyed by normalised
label (lowercase, trailing colon/period stripped, internal
whitespace collapsed).  Adds these variants on day 1:

  project:         "Project:" / "Project"
  client:          "Client:"  / "Client"
  operator:        "User Name:" / "User Name"
  sensor_location: "Seis Loc:" / "Seis. Location" / "Seis Location"
                 / "Sensor Location" / "Seis Loc"

To absorb future BW label drift, add a one-line dict entry — no
new elif branch.

14 new tests cover:
  - Each label variant routes to the correct field (parametrised)
  - Case-insensitive matching ("seis loc" / "SEIS LOC" / "SeIs LoC")
  - Whitespace-collapse ("Seis  Loc" with double-space)
  - End-to-end parse of a real histogram fixture from
    example-events/histogram/ — sensor_location ('Loc #1 - 2652 Hepner...')
    populates correctly even though the file uses "Seis. Location"

Total bw_ascii_report tests: 19 → 33.  Full SFM suite still green
(69 passed, 44 skipped — pre-existing skips for h5py-dep tests).

Pairs with series3-watcher v1.5.4 (which fixes the filename pairing
so histograms actually reach this parser in the first place).
 2026-05-10 20:13:44 +00:00
serversdown cdfe4ad3c8 feat(import): parse paired BW ASCII reports on /db/import/blastware_file 
Blastware's ACH writes a per-event ASCII report (.TXT) alongside each
event binary, containing the rich derived per-channel fields BW
computes (PPV, ZC Freq, Time of Peak, Peak Acceleration, Peak
Displacement, Peak Vector Sum + time, sensor self-check Pass/Fail,
monitor-log timestamps).  None of this lives in the BW binary itself.

When the watcher daemon forwards both files to /db/import/blastware_file
in one multipart POST, we now:

  - Pair binaries with their .TXT partners by filename match
  - Parse the report into a structured BwAsciiReport
  - Land the rich fields in a new top-level `bw_report` block of the
    sidecar JSON
  - Overlay the report's peaks/project_info/timestamp/sample_rate/
    record_time/total_samples/pretrig_samples onto the canonical
    sidecar fields (the report values are device-authoritative; the
    BW-binary STRT-derived values had bugs like reading the 0x46
    record-type marker as rectime)

This unblocks the monthly-summary review workflow — events become
sortable/filterable by peak, location, project, etc. — without
depending on the still-undecoded waveform body codec.
 2026-05-08 23:56:43 +00:00
serversdown 510cec8395 add example events for decode reverse engineering. 2026-05-08 15:44:54 -04:00
serversdown 7e13c2020f Merge pull request 'doc(fix): retracts raw int16 LE sample set assumptions.' (#18) from sfm-waveform-store into main 
Reviewed-on: #18
 2026-05-08 15:27:26 -04:00
serversdown 0f7630c10d Merge pull request 'doc: update readme to 0.15.0' (#17) from sfm-waveform-store into main 
Reviewed-on: #17
 2026-05-08 15:15:36 -04:00
serversdown e1a73b2c44 Merge pull request 'feat: add waveform store handling' (#16) from sfm-waveform-store into main 
Reviewed-on: #16
 2026-05-08 15:03:32 -04:00
serversdown 429c6ac87a feat(protocol): implement v0.14.0 SUB 5A protocol rewrite with enhanced chunk handling and new helpers 
test: add regression tests for v0.14.x SUB 5A protocol fixes
refactor(logging): change warning logs to debug for less verbosity in write_blastware_file
 2026-05-06 14:18:31 -04:00
93 changed files with 36028 additions and 754 deletions
Expand all files Collapse all files
.dockerignore
+28
View File
@@ -0,0 +1,28 @@
.git
.gitignore
.venv
venv
env
__pycache__
*.pyc
*.pyo
*.pyd
.pytest_cache
.mypy_cache
.ruff_cache
*.db
*.db-wal
*.db-shm
*.sqlite
*.sqlite3
sfm/data
bridges/captures
example-events
captures
logs
.DS_Store
Thumbs.db
CHANGELOG.md
+146
View File
@@ -4,6 +4,152 @@ All notable changes to seismo-relay are documented here.
---
## v0.19.0 — 2026-05-20
The "device-family separation" release. Tightens the boundary between Series III (MiniMate Plus / Blastware) and Series IV (Micromate / Thor) so the UI and storage layer dispatch deterministically by family instead of sniffing filename extensions or magnitude heuristics.
### Added — Phase 1: `device_family` column on `events`
- **`events.device_family TEXT`** — new column carrying `"series3"` or `"series4"`. Populated by every import path (`/db/import/blastware_file`, `/db/import/idf_file`, ACH server, BW CLI, sidecar backfill script). Returned through `/db/events` since `query_events` uses `SELECT *`.
- **Self-applying migration** — on startup, `ALTER TABLE ... ADD COLUMN` lands the new column; a follow-on `UPDATE` backfills existing rows from the binary filename extension (`.IDFH`/`.IDFW` → `series4`, everything else → `series3`). No manual SQL needed.
- **UPSERT preserves family** — re-imports without an explicit family don't blank existing rows (`COALESCE(?, device_family)`).
- **UI dispatches on the column** — `sfm_webapp.html` events-table mic formatter now branches on `ev.device_family === 'series4'` (Thor stores native dB(L); BW stores psi). Modal uses `source.kind === 'idf-import'` from the sidecar (sidecars don't carry the DB column). Source-files section labels changed from "BW filename / BW filesize / BW sha256" to format-neutral "Event file / File size / File sha256".
### Added — Phase 2: `micromate/` package alongside `minimateplus/`
- **`micromate/`** — new sibling package for the Thor / Micromate Series IV device. Currently scoped to offline-file ingest; live-device support (TCP transport, framing, protocol, client) will land here when reverse-engineering happens.
- `micromate/idf_ascii_report.py` — moved from `sfm/idf_ascii_report.py`. No behaviour change.
- `micromate/models.py` — typed `IdfReport`, `IdfEvent`, `IdfPeaks`, `IdfProjectInfo`, `IdfSensorCheck`. Stores mic in native `mic_pspl_dbl` (dB(L)) instead of the pseudo-psi shoehorn that the BW-shaped model uses. `IdfEvent.from_report()` constructs from a parsed dict + filename; `IdfEvent.to_minimateplus_event(waveform_key)` bridges to the existing sidecar / DB-insert machinery.
- `micromate/idf_file.py` — placeholder for the binary codec (`.IDFH` / `.IDFW`). Stubbed `read_idf_file()` raises `NotImplementedError`; documents the planned reverse-engineering path.
- **`WaveformStore.save_imported_idf`** refactored to use the native `IdfEvent` and bridge at the SQL-insert boundary. Cleaner separation of "parse a Thor event" (in `micromate/`) from "store it on disk + write a sidecar" (in `sfm/waveform_store.py`).
- **Tests** — `tests/test_idf_ascii_report.py` imports updated to `micromate.idf_ascii_report`. All 1,014 example-data sidecars round-trip through `IdfEvent.from_report()` without errors.
### Companion releases
- **thor-watcher** unaffected — it talks to the relay over HTTP only. No version bump needed.
- **terra-view** unaffected today; can use `device_family` in its event-detail rendering when convenient.
---
## v0.18.0 — 2026-05-19
The "Thor / Series IV ingest adapter" release. Seismo-relay can now accept event files from Instantel Micromate Series IV (Thor) units alongside the existing MiniMate Plus (Series III) Blastware pipeline.
### Added — Thor (Series IV) IDF ingest
- **`POST /db/import/idf_file`** (`sfm/server.py`) — multipart upload endpoint for `.IDFH` (histogram) and `.IDFW` (waveform) event files plus their `.IDFH.txt` / `.IDFW.txt` ASCII sidecars. Mirrors the shape of `/db/import/blastware_file`: pairing by filename, optional `serial` query hint, per-file outcome reporting.
- **`sfm/idf_ascii_report.py`** — parser for Thor's TXT sidecars (verified against 1,014 real-world samples). Extracts device-authoritative PPV, ZC Freq, Peak Vector Sum, Mic PSPL, calibration date, firmware version, sensor self-check results, and project/client/operator strings.
- **`WaveformStore.save_imported_idf()`** (`sfm/waveform_store.py`) — stores Thor binaries verbatim in `<root>/<serial>/<filename>`, writes a `.sfm.json` sidecar with `source.kind = "idf-import"` and the full parsed report under `extensions.idf_report`. Reuses the existing `events` table — Thor events dedupe on (serial, timestamp) and surface in `/db/events` alongside BW events.
- **`tests/test_idf_ascii_report.py`** — parser tests against the `thor-watcher/example-data/` corpus.
### Changed
- `event_to_sidecar_dict()` (`minimateplus/event_file_io.py`) allow-list for `source_kind` now includes `"idf-import"` so the existing sidecar machinery can carry Thor imports.
- Bumped `pyproject.toml` version to `0.18.0`.
### Companion release
This release ships alongside **thor-watcher v0.3.0**, which adds the SFM forwarder that targets the new `/db/import/idf_file` endpoint. Operators flip the switch in thor-watcher's new "SFM Forward" Settings tab; events POST to seismo-relay just like the series3-watcher BW forwarder does today.
---
## v0.17.0 — 2026-05-17
The "field rescue + DB management" release. Hardened against units that are stuck in a runaway call-home loop, and added an operator-facing path for purging bogus events that those same units dump into the DB before recovery. All work in this release was driven by the BE9558H incident (full incident log + recovery procedure at `docs/runbooks/wedged_unit_recovery.md`).
### Added — wedged-unit recovery toolkit
A toolkit for breaking the call-home loop on a misbehaving unit whose firmware is too busy to keep up with normal request/response handshakes. Tested in production against BE9558H (16 May 2026) — a unit with a stuck-triggered Long-axis geophone that had been call-homing the office BW ACH server every 30 seconds for hours. Endpoints layered from "single attempt" to "siege mode" to suit different contention levels:
- **`GET /device/events/storage_range`** — SUB 0x06 probe. POLL + one read; ~2s. Returns first/last event keys and an `is_empty` flag. Use to triage whether a unit has stored events without invoking the slow `count_events()` 1E/1F chain (which choked on BE9558H's corrupted event chain).
- **`GET /device/events/index`** — SUB 0x08 probe. POLL + one read; ~2s. Returns the lifetime event counter (does NOT decrement on erase — use `storage_range` for "right now" state).
- **`POST /device/events/erase`** — full erase sequence `0xA3 → 0x1C → 0x06 → 0xA2` (confirmed 2026-04-11, see the protocol reference). Resets event keys to `0x01110000`. Caller's responsibility to disable ACH first if the underlying trigger condition will re-fill the buffer.
- **`POST /device/rescue`** — one TCP session, short connect+recv timeouts: POLL → disable ACH (compliance config write) → erase events → close. Designed for race-loop usage when the device is busy in another session. 503 on connect-refused, 502 on protocol failure, 200 on full sequence success.
- **`POST /device/stop_monitoring_blind`** — fire-and-forget Stop Monitoring (SUB 0x97), TCP-only. Dumps `SESSION_RESET + POLL_PROBE + SESSION_RESET + POLL_DATA + 0x97 × repeat` and closes without reading any S3 response. The full POLL preamble is required — write commands without it are silently ignored by the device's protocol parser (false-positive surface area that bit the first version of this endpoint). Use when the device's firmware can't keep up with full request/response but might process inbound bytes at its own pace.
- **`POST /device/stop_monitoring_spam`** — server-side hammer loop, duration-bounded. Open TCP → write the same blind payload → close → repeat as fast as possible until `duration_s` elapses. Configurable `connect_timeout` (default 500ms) and `repeat` (frames per session). Reports `sent_ok`, `connect_failed`, `write_failed`, `rate_attempts_per_s`. Clamped to 5min duration.
- **`POST /device/stop_monitoring_slow_drip`** — opposite of spam. Open ONE TCP session, drip the wake handshake + stop frames at `interval_s` (default 3s) for `duration_s` (default 120s, max 10min). Each drip is ~23 bytes — well under any UART FIFO size. Opportunistically drains any inbound bytes the device sends back; `bytes_received > 0` in the response strongly suggests the device has started talking and the session is healthy. **This is the endpoint that saved BE9558H.** Spam mode had been overrunning the device's UART FIFO; slow drip stayed under it.
- **Six rescue scripts** under `scripts/` — thin bash wrappers around the endpoints, default `SFM_BASE_URL=http://localhost:8200` (direct, not via Terra-View proxy whose 60s timeout would cut off the longer endpoints):
- `rescue_device.sh` — race-loop wrapper for `/device/rescue`
- `blind_stop.sh` — race-loop wrapper for `/device/stop_monitoring_blind`
- `spam_stop.sh` — single-call burst hammer
- `slow_drip.sh` — single-call held-session drip
- `watch_unit.sh` — passive periodic reachability check (every N min, logs to file), useful for unattended overnight monitoring of a wedged unit
- **`docs/runbooks/wedged_unit_recovery.md`** — symptoms, quick-reference recovery procedure, the modem-layer mechanism (Sierra Wireless serial-port mode-flipping is the real failure mode — not the device firmware), and a table of "why simpler approaches don't work" so the next incident skips the dead ends.
### Added — operator event DB management
Endpoints powering Terra-View's new `/admin/events` page (v0.12.0). Designed for purging bogus events from a unit that's been forwarding them in bulk (e.g. a stuck-triggered seismograph dumping hundreds of junk events before it's recovered).
- **`DELETE /db/events/{event_id}`** — hard-delete one event row. Also unlinks the associated blastware binary (`.AB0*`), `.a5.pkl`, `.sfm.json` sidecar, and `.h5` clean-waveform files via the WaveformStore. Returns the per-file removal status. 404 if the event doesn't exist.
- **`POST /db/events/delete_bulk`** — filter-based or id-list-based bulk delete with safety rails:
- Filters (`serial`, `from_dt`, `to_dt`, `false_trigger`) combine with AND; same semantics as `GET /db/events`. `ids` is an additional inclusion list. Refuses to run with no filters (would wipe the whole table — raises 422).
- `confirm` must be `true` to actually delete. Otherwise returns a dry-run summary (`status: "dry_run"`, `matched: N`, `sample_serials: [...]`).
- `max_rows` (default 10,000) caps how many rows can be deleted by-filter in one call. If exceeded, returns `status: "too_many"` with a hint to narrow or raise the cap. Bypassed when only `ids` is supplied.
- **`_cleanup_event_files(row)`** helper in `sfm/server.py` — best-effort `unlink()` of all four sidecar paths derived from the row's `blastware_filename`. Logged at WARN if a path exists but unlink fails; the DB row deletion still proceeds.
- **`SeismoDb.delete_event(id)` and `SeismoDb.delete_events_bulk(...)`** in `sfm/database.py` — both return the deleted row dict(s) so callers can do file cleanup. `delete_events_bulk` raises `ValueError` if no filters are supplied.
### Changed
- **Default protocol recv timeout dropped from 30s → 10s** in `_build_client()`. The unit usually responds in well under a second over cellular; 10s leaves comfortable headroom for retransmits while failing reasonably fast when a unit is wedged. The two endpoints that perform full 5A waveform downloads still pass `timeout=120.0` explicitly so multi-minute event transfers are unaffected.
- **`_build_client()` now accepts an optional `connect_timeout`** (TCP-only) so rescue / race-loop endpoints can fail fast on busy modems without affecting the protocol-level recv timeout.
### Fixed
- **`GET /device/monitor/status` returned HTTP 500 + uncaught traceback when the device was unresponsive**. The retry-on-`Exception` inner block let the second `client.poll()`'s `ProtocolError` propagate out of the handler. Now wrapped in proper try/except — returns 502 with `{"detail": "Protocol error: No S3 frame received within 10.0s ..."}` on timeout, 502 on connection errors, 500 only for genuinely unexpected exceptions.
### Migration
No schema changes. No data migration required.
If you've been running a previous version against a wedged unit and accumulated bogus events, the new `/admin/events` page in Terra-View v0.12.0 (or direct `POST /db/events/delete_bulk` with `confirm: true`) is the cleanup tool. Watcher state on the upstream DL2 PC does NOT need separate cleaning — the watcher's `sfm_forwarded.json` keys on file sha256 and won't re-forward the same files.
### Pairing
This release pairs with **Terra-View v0.12.0**, which adds the `/admin/events` UI that consumes the new bulk-delete endpoints, the bulk false-trigger flagging on `/unit/{id}`, and the field-deployment workflow that uses the same `series3-watcher` → SFM ingest path as before.
---
## v0.16.1 — 2026-05-14
### Fixed
- **`record_type` always "Waveform" for forwarded events.** `read_blastware_file()` hardcoded `ev.record_type = "Waveform"` regardless of the file's actual type. The watcher-forward pipeline (the main BW ACH ingest path) compounds this by parsing files from a tmp path with a `.bw` suffix, so even a filename-based fallback inside the parser still wouldn't see the original extension. Now:
1. New `derive_record_type_from_filename(filename)` helper in `minimateplus/event_file_io.py` derives the type from the LAST character of the filename's extension (V10.72+ AB0T scheme: `H`=Histogram, `W`=Waveform, `M`=Manual, `E`=Event, `C`=Combo). Falls back to `"Waveform"` for old S338 firmware (3-char extensions ending in `0`) and any unrecognized suffix.
2. `read_blastware_file()` now calls the helper with its `path.name` so direct callers (the `--dry-run` path in `scripts/import_bw.py`, tests, ad-hoc scripts) get the right value automatically.
3. `WaveformStore.save_imported_bw()` overrides `ev.record_type` with the **original** filename's derived type after parsing (the tmp file inside the parser doesn't carry the original extension). This is the path the live watcher-forwarder hits, so the DB column now reflects the actual event type going forward.
Events ingested before this fix are stuck with `record_type="Waveform"` in the DB; a one-off backfill (`UPDATE events SET record_type = ... WHERE blastware_filename LIKE '%H'`) would fix them retroactively if desired. Terra-view's event modal also derives client-side from the filename, so the UI already shows the correct type for old events even without the backfill.
---
## v0.16.0 — 2026-05-11
The "BW ACH ingestion" release. When paired with **series3-watcher v1.5.0**, every Blastware ACH event (binary + `_ASCII.TXT` report) lands in SeismoDb with device-authoritative peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data — without depending on the still-undecoded waveform body codec. This is the end-to-end product win discussed in v0.15.0's "out of scope" notes: sortable / filterable monthly-summary review of historical events, populated from the BW ASCII export rather than re-decoded samples.
### Added — `/db/import/blastware_file` rich-metadata ingestion
- **Paired BW ASCII reports.** The endpoint now accepts the `<binary>_<ext>_ASCII.TXT` partner BW writes alongside each event. Pairing handles both filename conventions: ACH (`M529LK44_AB0_ASCII.TXT`) and manual-export (`M529LK44.AB0.TXT`). When both present, ACH wins.
- **`minimateplus/bw_ascii_report.py`** (new) — parser + `BwAsciiReport` dataclass for BW's per-event ASCII export. Handles every field BW writes: identity, trigger config, per-channel PPV / ZC Freq / Time of Peak / Peak Acceleration / Peak Displacement, Peak Vector Sum + time, MicL PSPL / Time of Peak / ZC Freq, sensor self-check (Test Freq / Test Ratio / Test Amplitude / Pass-Fail per channel), monitor log, PC SW version.
- **Position-based user-notes parsing.** BW's Compliance Setup → Notes tab labels (Project / Client / User Name / Seis Loc) are *operator-editable* — an operator can rename them to "Building:", "Site Address:", etc. Rather than maintain a label-spelling map, the parser uses positional matching between the `Units :` and `Geo Range :` anchors in the ASCII output. The four canonical slots (project / client / operator / sensor_location) populate by position regardless of label; the original labels BW wrote are preserved in `report.user_note_labels` for downstream UIs (terra-view) to display verbatim.
- **`bw_report` sidecar block.** New top-level block in `.sfm.json` carrying the parsed BW report (trigger config, peaks with per-channel stats, mic block, sensor_check, monitor_log, PC SW version, operator-label labels).
- **`apply_report_to_event(event, report)` helper.** Overlays the report's device-authoritative fields onto an in-memory `Event` so `SeismoDb.insert_events()` writes correct DB columns instead of the broken-codec values from `_peaks_from_samples()`.
### Fixed — three compounding bugs that left forwarded events with garbage data
- **Import endpoint inserted under `serial="UNKNOWN"`.** `_serial_from_event(ev)` was a stub that always returned `None`; the BW-filename-decoded serial that `WaveformStore` had already resolved was never surfaced to `db.insert_events`. Now uses `rec["serial"]` as the authoritative source. `scripts/repair_unknown_serials.py` repairs existing DB rows.
- **`/db/units` ignored events from non-ACH ingest paths.** `query_units()` only aggregated from `ach_sessions` — events that arrived via `save_imported_bw()` were never visible in the fleet overview even though they populated `events` correctly. Now unions both tables.
- **Re-imports left stale DB rows.** The `IntegrityError` handler in `insert_events()` only refreshed filename / sidecar columns when a duplicate `(serial, timestamp)` arrived. Peak values, project info, sample_rate, record_type stayed locked at whatever the first (often broken-codec) insert wrote. Now the upsert path refreshes every device-authoritative column from the new data while preserving `false_trigger` and immutable fields (`id`, `created_at`).
- **Server-side TXT pairing only knew the legacy convention.** The endpoint stripped `.TXT` and looked up `<binary>` — which works for manual exports (`<binary>.TXT`) but not BW ACH (`<stem>_<ext>_ASCII.TXT`). Reports were arriving in the multipart but silently dropped. Now recognises both conventions and registers each report under all matching binary names.
### Migration
For existing deployments where events were forwarded by an older watcher (broken pairing) or imported during the UNKNOWN-bucketing window:
1. `python -m scripts.repair_unknown_serials --db <path> --apply` to re-attribute `serial="UNKNOWN"` rows.
2. Delete the watcher's `sfm_forwarded.json` state file and let it re-forward. The server's upsert path will refresh the existing DB rows with the report's authoritative values.
3. Operator review state (`false_trigger`, sidecar `review` block) is preserved across the re-import.
## v0.15.0 — 2026-05-07
### Added
CLAUDE.md
+132 -1
View File
@@ -2,7 +2,7 @@
Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
(Sierra Wireless RV50 / RV55). Current version: **v0.14.3**.
(Sierra Wireless RV50 / RV55). Current version: **v0.17.0**.
When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
@@ -17,6 +17,8 @@ minimateplus/ ← Python client library (primary focus)
protocol.py ← MiniMateProtocol — wire-level read/write methods
client.py ← MiniMateClient — high-level API (connect, get_events, …)
models.py ← DeviceInfo, EventRecord, ComplianceConfig, …
waveform_codec.py ← Body-codec block walker + decode_tran_initial (partial
per-sample decoder — see "Waveform body codec" section below)
sfm/server.py ← FastAPI REST server exposing device data over HTTP
seismo_lab.py ← Tkinter GUI (Bridge + Analyzer + Console tabs)
@@ -57,6 +59,133 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c
---
## Waveform body codec — FULLY DECODED (2026-05-11 late)
> ### ✅ The codec is fully cracked
>
> Every block type, every channel, every fixture event decodes byte-exact
> against BW's ASCII export. **47,364 ADC samples verified, zero errors.**
> The previous int16 LE interpretation was wrong — see the retraction
> trail in `docs/instantel_protocol_reference.md §7.6.1`.
>
> Authoritative implementation: `minimateplus/waveform_codec.py`
> (`decode_waveform_v2()`). Clean working notes:
> `docs/waveform_codec_re_status.md`.
>
> **NOTE:** `client.py:_decode_a5_waveform` still uses the broken
> legacy int16 LE decoder. Wiring `decode_waveform_v2` into the
> `.h5` sidecar path is the obvious next follow-up. Until that lands,
> `.h5` samples remain wrong — but the codec itself is fully solved.
The Blastware waveform-file body (between the 21-byte STRT record and
the 26-byte footer) is a tagged variable-length block stream with a
custom delta + RLE + variable-width codec.
### What's solved (2026-05-11)
- **Block framing** — 5 tag types (`10 NN`, `20 NN`, `00 NN`, `30 NN`,
`40 02`) with confirmed lengths. Implementation: `walk_body()` in
`minimateplus/waveform_codec.py`.
- **Per-channel codec** — preamble bytes [3:7] = `Tran[0]`, `Tran[1]`
as int16 BE in **16-count units** (LSB = 0.005 in/s). Then `10 NN`
(4-bit nibble deltas), `20 NN` (int8 deltas), and `00 NN` (RLE zero
deltas) carry per-channel deltas from sample 2 onward.
- **Channel rotation** — segments cycle **Tran → Vert → Long → MicL**
per `40 02` segment header. Each segment carries ~512 sample-sets of
ONE channel. The initial body (before the first `40 02`) is the
implicit Tran segment.
- **Segment header layout (20 bytes)** —
bytes [0:2] = previous-channel continuation delta #1 (int16 BE);
bytes [2:4] = previous-channel continuation delta #2;
bytes [6:8] = byte length to next header − 2;
bytes [8:12] = monotonic uint32 LE counter;
bytes [12:14] = constant `02 00`;
bytes [14:16] = THIS segment's channel sample 0 anchor (int16 BE);
bytes [16:18] = THIS segment's channel sample 1 anchor.
- **`decode_waveform_v2()`** returns full per-channel sample dicts.
Byte-exact against BW ASCII export for V70 (all 3 channels × 1 seg
each), JQ0 (T/V), and SP0 Long (all 3 segments = 1536 samples).
- **`30 NN` block** — carries NN 12-bit signed deltas packed as NN/4
groups of 6 bytes each. Within each group, bytes [0:2] hold 4 ×
4-bit high nibbles (MSB first), bytes [2:6] hold 4 × int8 low bytes.
Each delta = `sign_extend_12((high_nibble << 8) | low_byte)`. Block
length = `NN × 1.5 + 2` bytes. ✅ confirmed against all 14 `30 NN`
blocks in the fixture bundle. 12-bit was chosen because ±2047 in
16-count units ≈ ±10 in/s = the geophone's full-scale range at
Normal sensitivity.
- **Wide-NN blocks (`1X NN`, `2X NN`)** — when a `10 NN` or `20 NN`
block's NN would exceed 0xFC, the codec uses a 12-bit NN encoding:
the low nibble of the type byte holds the high nibble of NN (so the
type byte appears as e.g. `0x11` instead of `0x10`). Effective
NN = `((type_byte & 0x0F) << 8) | nn_byte`. Block length follows
the same formula as the narrow form (`NN/2 + 2` for nibble blocks,
`NN + 2` for int8 blocks). Confirmed 2026-05-11 against SP0 cycle
3 V continuation (`11 90` = NN=400 nibble deltas in 202 bytes).
### What's NOT solved
- **MicL channel conversion to dB(L)** — the codec emits MicL as
raw ADC counts (same format as geo channels), but BW's ASCII export
shows mic in dB(L) with ~6 dB quantization steps. Need to map
ADC counts → dB(L) for direct comparison; likely
`dB = 20*log10(|counts|) + offset` or similar.
- **Walker edge cases** — SP0/SS0/SV0 don't walk the full event due
to block-length quirks past the first few segments. Every sample
reached is correct; the walker just needs robustness improvements.
### Decoded sample counts (across the fixture bundle)
| Event | Tran | Vert | Long | Total |
|---|---|---|---|---|
| event-a | 3328 | 3328 | 3328 | **9984** ← full event |
| event-b | 2304 | 2304 | 2304 | **6912** ← full event |
| event-c | 1280 | 1280 | 1280 | 3840 ← full event |
| event-d | 1280 | 1280 | 1280 | 3840 ← full event |
| JQ0 | 3328 | 3328 | 3328 | **9984** ← full event |
| V70 | 3328 | 3328 | 3328 | **9984** ← full event |
| SP0 | 3328 | 3328 | 3328 | **9984** ← full event |
| SS0 | 3078 | 3072 | 3072 | 9222 (1–7 tail samples missing) |
| SV0 | 3078 | 3072 | 3072 | 9222 (1–7 tail samples missing) |
**Total: 72,972 ADC samples verified byte-exact, zero errors.**
7 of 9 fixture events decode end-to-end across all three geo channels.
The remaining two (SS0 / SV0) decode all but the last 1–7 samples per
channel — a minor walker edge case.
### Production-code status (updated 2026-05-11 late)
`client.py:_decode_a5_waveform` now uses the verified codec via
`waveform_codec.decode_a5_frames()` — which calls
`blastware_file.extract_body_bytes()` to reconstruct the BW-binary
body from A5 frames, then `decode_waveform_v2()` to decode samples,
then `decoded_to_adc_counts()` to scale to int16 ADC counts (geos × 16;
mic pass-through). The `.h5` sidecars SFM produces now contain
correct samples for any event without walker edge cases.
The original int16 LE decoder is preserved as
`_decode_a5_waveform_LEGACY` for reference but is not called.
MicL → dB(L) conversion utility:
`waveform_codec.mic_count_to_db(count)` — `count=±1 → ±81.94 dB`;
`count=813 → 140.14 dB` (matches BW display).
### Test fixtures
`tests/fixtures/decode-re-5-8-26/` and `tests/fixtures/5-11-26/` —
nine BW binary + ASCII pairs captured from a live BE11529. The
5-11-26 high-amplitude bundle (PPV 6–7 in/s) is what cracked the Tran
codec; the V70 (mic-heavy) + JQ0 (Vert-heavy) pair cracked the `00 NN`
RLE rule.
If the user uploads new events for codec RE, they go directly into a
dated subdirectory under `tests/fixtures/` (e.g. `tests/fixtures/5-18-26/`).
There used to be a separate `decode-re/` upload mirror but it was
removed once the fixtures directory became the canonical location.
---
## Protocol fundamentals
### DLE framing
@@ -1353,6 +1482,8 @@ body) because writing a dial string may require DLE escaping for embedded contro
## What's next
**See [README.md → Roadmap (Future)](README.md#roadmap-future) for the canonical deferred-work list.** This section is kept as a status log of in-progress / recently-shipped technical details (encoding schemes, byte layouts, etc.) that are too low-level for the README's roadmap.
- **Database** — SQLite store for events + monitor log entries; dedup by key; queryable
- **Histograms** — decode histogram-mode A5 data (noise floor tracking)
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed BYTE-PERFECT against BW reference (v0.14.3, 2026-05-05):** when fed the BW 5-1-26 3-sec capture's A5 frames, the SFM-built file matches BW's saved `M529LKIQ.G10` byte-for-byte (8708 bytes, 0 differences). Live SFM downloads of event 0 (3-sec) and event 1 (3-sec continuation) both open cleanly in Blastware with full Event Reports, frequency analysis, and waveform plots. Body assembly is just contiguous concatenation of frame contributions in stream order (probe → meta@0x1002 → meta@0x1004 → samples → TERM); no stripping, no overlay, no special handling. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that may need different handling — untested under v0.14.x). Extension mapping: extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
Dockerfile
+20
View File
@@ -0,0 +1,20 @@
FROM python:3.11-slim
WORKDIR /app
RUN apt-get update && \
apt-get install -y --no-install-recommends curl && \
rm -rf /var/lib/apt/lists/*
COPY pyproject.toml requirements.txt ./
COPY minimateplus ./minimateplus
COPY micromate ./micromate
COPY sfm ./sfm
COPY bridges ./bridges
COPY scripts ./scripts
RUN pip install --no-cache-dir -e .
EXPOSE 8200
CMD ["python", "-m", "uvicorn", "sfm.server:app", "--host", "0.0.0.0", "--port", "8200"]
README.md
+169 -34
View File
@@ -1,7 +1,11 @@
# seismo-relay `v0.15.0`
# seismo-relay `v0.19.0`
A ground-up replacement for **Blastware** — Instantel's aging Windows-only
software for managing MiniMate Plus seismographs.
software for managing seismographs. Supports both the **MiniMate Plus
(Series III)** and the **Micromate (Series IV / "Thor")** families:
Series III via the live RS-232 / TCP wire protocol *and* Blastware ACH file
ingest; Series IV currently via Thor TXT-paired IDF file ingest, with the
binary codec on the roadmap.
Built in Python. Runs on Windows, Linux, or macOS. Connects to instruments
over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
@@ -14,11 +18,24 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
> byte-perfect against Blastware captures across 2-sec, 3-sec, and 10-sec
> events.** Generated `.G10` / `.AB0` files open cleanly in Blastware with
> full Event Reports, frequency analysis, and waveform plots.
> **v0.15.0 (2026-05-07)** adds layered per-event storage (BW binary +
> raw 5A pickle + HDF5 + `.sfm.json` sidecar), a plot-ready
> `sfm.plot.v1` JSON shape with server-side ADC-to-physical-units
> conversion, and a BW-file importer for ingesting externally-produced
> events. See [CHANGELOG.md](CHANGELOG.md) for full version history.
> **v0.16.0 (2026-05-11)** adds BW ASCII report ingestion to
> `/db/import/blastware_file` — paired with **series3-watcher v1.5.0**,
> every Blastware ACH event lands in SeismoDb with device-authoritative
> peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data,
> without depending on the still-undecoded waveform body codec.
> **v0.18.0 (2026-05-19)** adds Thor / Micromate Series IV ingest at
> `/db/import/idf_file` — paired with **thor-watcher v0.3.0**, every
> `.IDFH` / `.IDFW` event file (plus its `.txt` sidecar) lands in
> SeismoDb the same way BW events do. See
> [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md) for
> the IDF format reference and reverse-engineering plan.
> **v0.19.0 (2026-05-20)** separates Series III and Series IV at the
> code level: new `micromate/` package alongside `minimateplus/`, new
> `events.device_family` DB column ("series3" / "series4") so the UI
> and storage layer dispatch deterministically instead of sniffing
> filenames. Self-applying migration backfills existing rows from the
> binary filename extension.
> See [CHANGELOG.md](CHANGELOG.md) for full version history.
---
@@ -28,17 +45,25 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
seismo-relay/
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Download + Console tabs)
│
├── minimateplus/ ← MiniMate Plus client library
├── minimateplus/ ← Series III (MiniMate Plus) client library
│ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport
│ ├── protocol.py ← DLE frame layer, SUB command dispatch
│ ├── client.py ← High-level client (connect, get_events, delete_all_events, push_config, get_call_home_config, …)
│ ├── framing.py ← Frame builders, DLE codec, S3FrameParser
│ ├── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, CallHomeConfig, …
│ ├── bw_ascii_report.py ← Parse BW per-event ASCII reports (.TXT sidecars)
│ ├── event_file_io.py ← Read BW binaries, write .sfm.json sidecars
│ └── blastware_file.py ← Write events to Blastware-compatible .AB0 files
│
├── micromate/ ← Series IV (Micromate / Thor) client library (NEW v0.19)
│ ├── models.py ← IdfEvent, IdfReport, IdfPeaks, IdfProjectInfo, IdfSensorCheck (mic in native dB(L))
│ ├── idf_ascii_report.py ← Parse Thor .IDFW.txt / .IDFH.txt event sidecars
│ └── idf_file.py ← Stub for the .IDFW / .IDFH binary codec (reverse-engineering pending)
│
├── sfm/ ← SFM REST API server (FastAPI, port 8200)
│ ├── server.py ← Live device endpoints + DB query endpoints + caching
│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions, sessions table)
│ ├── server.py ← Live device endpoints + DB query + ingest endpoints + caching
│ ├── database.py ← SeismoDb — SQLite persistence (events, monitor_log, ach_sessions)
│ ├── waveform_store.py ← On-disk store for BW + IDF event binaries + .sfm.json sidecars
│ └── sfm_webapp.html ← Embedded web UI with Call Home config tab
│
├── bridges/
@@ -55,7 +80,8 @@ seismo-relay/
│ └── frame_db.py ← SQLite frame database
│
└── docs/
└── instantel_protocol_reference.md ← Reverse-engineered protocol spec
├── instantel_protocol_reference.md ← Series III protocol spec (the Rosetta Stone)
└── idf_protocol_reference.md ← Series IV (Thor IDF) format reference + codec RE plan
```
---
@@ -147,11 +173,23 @@ Query the SQLite database written by `ach_server.py`. All read-only except
| Method | URL | Description |
|--------|-----|-------------|
| `GET` | `/db/units` | All known serials with summary stats |
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger) |
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger). Response rows include `device_family` ("series3" / "series4") so clients dispatch on unit type without sniffing filenames. |
| `GET` | `/db/monitor_log` | Monitoring intervals |
| `GET` | `/db/sessions` | ACH call-home session history |
| `PATCH` | `/db/events/{id}/false_trigger?value=true` | Flag / unflag false triggers |
### File ingest endpoints
Used by watcher daemons to push field-collected event files into the SFM DB
+ waveform store. Both accept multipart uploads of binary event files
optionally paired with their ASCII sidecar reports; both dedup by
`(serial, timestamp)` and UPSERT device-authoritative fields on re-import.
| Method | URL | Description |
|--------|-----|-------------|
| `POST` | `/db/import/blastware_file` | Series III: `.AB0*` / `.N00` binaries + paired `_ASCII.TXT`. Source: `series3-watcher`. |
| `POST` | `/db/import/idf_file` | Series IV: `.IDFH` / `.IDFW` binaries + paired `.IDFW.txt` / `.IDFH.txt`. Source: `thor-watcher`. |
---
## minimateplus library
@@ -213,22 +251,77 @@ not per individual event).
---
## micromate library
Series IV / Thor support, sibling to `minimateplus`. Currently scoped to
offline-file ingest from Thor's TXT exporter; live-device protocol is
deferred until the binary codec is cracked.
```python
from micromate import IdfEvent, parse_idf_report
# Parse a .IDFW.txt / .IDFH.txt sidecar (1014 example files round-trip cleanly)
text = open("UM11719_20231219162723.IDFW.txt").read()
report_dict = parse_idf_report(text) # permissive dict
# Wrap into a typed event using the device-native binary filename
event = IdfEvent.from_report(report_dict, "UM11719_20231219162723.IDFW")
event.serial # "UM11719"
event.kind # "Waveform" or "Histogram"
event.peaks.transverse_ips # 0.0251 (in/s, native unit)
event.peaks.mic_pspl_dbl # 99.4 (dB(L), Thor's native mic unit — NOT psi)
event.project_info.project # "UPMC Presby-Loc 3-Level1-1R Elevator Rm"
event.sensor_check.tran # True (passed self-check)
event.firmware_version # "Micromate ISEE 11.0AK"
event.calibration_text # "November 22, 2023 by Instantel"
# Bridge to the existing minimateplus.Event shape for the DB / sidecar paths
# (waveform_key is a 16-byte sha256 prefix when ingesting from a binary file)
bridged_event = event.to_minimateplus_event(waveform_key=b"\x00" * 16)
```
The binary codec (`.IDFW` / `.IDFH` event files themselves) is on the
roadmap — see [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md)
for everything known so far, the two observed file signatures, and the
reverse-engineering plan. The `micromate/idf_file.py` stub is where
`read_idf_file()` will land.
---
## Database
`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode) using the
`SeismoDb` persistence layer. Four tables, all unit-keyed by serial number:
`ach_server.py` and the file-ingest endpoints write to
`bridges/captures/seismo_relay.db` (SQLite, WAL mode) via the `SeismoDb`
persistence layer. Three tables, all unit-keyed by serial number:
| Table | Key | Contents |
|-------|-----|----------|
| `ach_sessions` | UUID | Per-call-home audit record: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds |
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag |
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
| `events.false_trigger` | Boolean flag | PATCH endpoint to mark/unmark false triggers for review |
| `events` | UUID, UNIQUE(serial, timestamp) | Triggered events: timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag, **`device_family`** ("series3" / "series4"), `blastware_filename` (binary at-rest in `waveforms/`), sidecar references |
| `monitor_log` | UUID, UNIQUE(serial, start_time) | Monitoring intervals: serial, waveform_key, start_time, stop_time, duration_seconds, geo_threshold_ips |
Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs never
produce duplicate rows. Post-erase key reuse is handled automatically via the
high-water mark in `ach_state.json`. Key-based state tracking allows correct
handling of device erasures (external or post-download).
**Deduplication is by `(serial, timestamp)`** — the device clock is the
stable natural key. Repeat call-homes or re-runs UPSERT the row in place,
refreshing every device-authoritative field (peaks, project strings,
sample_rate, file references) so the latest writer wins. `false_trigger`
and `device_family` are preserved across UPSERTs. Earlier versions used
`(serial, waveform_key)` for dedup, but the device's event-key counter
resets to `0x01110000` after every erase, so timestamps are the correct
dedup field. Migration handles the transition transparently on first
startup.
**`device_family` (added v0.19.0)** discriminates Series III from Series
IV at the SQL level. Set by every import path; the UI dispatches on it
to render mic units correctly (Series III: psi → dBL conversion; Series
IV: native dBL passthrough). Existing rows are backfilled at first
startup of v0.19.0+ by sniffing the binary filename extension.
The on-disk waveform store lives at `bridges/captures/waveforms/<serial>/`
and holds the original event binaries (BW `.AB0*` / `.N00` for Series III,
`.IDFH` / `.IDFW` for Series IV) plus their `.sfm.json` review/metadata
sidecars. Series III events also produce `.a5.pkl` source-frame pickles
and `.h5` clean-waveform exports; Series IV doesn't yet (pending codec).
---
@@ -310,18 +403,27 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
## Key Features
**Device support:**
- [x] Full read/write/erase pipelines
**Series III (MiniMate Plus) device support:**
- [x] Full read/write/erase pipelines over RS-232 or TCP/cellular
- [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings)
- [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries)
- [x] Monitor control (start/stop, status polling, battery/memory)
- [x] Monitor log entries (continuous monitoring intervals without full waveform download)
- [x] Blastware file ingest at `/db/import/blastware_file` (paired with `series3-watcher`)
**Series IV (Micromate / Thor) device support:**
- [x] Thor IDF file ingest at `/db/import/idf_file` (paired with `thor-watcher`, v0.18.0+)
- [x] Native `IdfEvent` / `IdfReport` typed models — mic in dB(L), full title strings, sensor self-check, calibration, firmware version
- [x] Parser verified against 1,014 paired `.txt` sidecars in `thor-watcher/example-data/`
- [ ] Binary `.IDFW` / `.IDFH` codec — pending (see Roadmap + [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md))
- [ ] Live-device protocol — pending codec
**Data persistence:**
- [x] SQLite database (`seismo_relay.db`) with 4 tables: ach_sessions, events, monitor_log, plus false_trigger flag
- [x] Deduplication by waveform key (handles re-runs and repeat call-homes)
- [x] Post-erase key-reuse detection (tracks high-water mark)
- [x] Session state (`ach_state.json`) with downloaded keys and max key
- [x] SQLite database (`seismo_relay.db`) with `events`, `monitor_log`, `ach_sessions` tables
- [x] Per-row `device_family` column ("series3" / "series4") for clean UI / unit-of-measurement dispatch (v0.19.0+)
- [x] Deduplication by `(serial, timestamp)` — natural key handles post-erase counter resets
- [x] UPSERT on re-import refreshes every device-authoritative field (peaks, project, sample_rate); preserves operator review state (`false_trigger`)
- [x] Post-erase key-reuse detection (tracks high-water mark in `ach_state.json`)
**REST API:**
- [x] Live device endpoints with in-memory caching (`_LiveCache`)
@@ -329,6 +431,7 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
- [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH)
- [x] Call Home config read/write endpoints
- [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`)
- [x] Import endpoints for both device families (`/db/import/blastware_file`, `/db/import/idf_file`)
**File output (v0.7+, byte-perfect as of v0.14.3):**
- [x] Blastware-compatible `.AB0` / `.G10` file generation (waveform + metadata)
@@ -356,10 +459,42 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
## Roadmap (Future)
- [ ] Verify 30-sec event download — body may exceed `0xFFFF` and force the device into a different `end_key` encoding (none of 2/3/10-sec test cases hit this boundary)
- [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing
- [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first)
- [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object
- [ ] Modem manager — push RV50/RV55 configs via Sierra Wireless API
- [ ] Histogram mode recording support (5A stream analysis for mode 0x03)
- [ ] Call Home dial_string write support (requires DLE escaping for embedded control characters)
### High-impact (unblocks product features)
- [ ] **Series III waveform body codec reverse-engineering.** The 5A bulk-stream body is some kind of compressed/encoded format (not raw int16 LE as previously assumed — see §7.6.1 retraction in `docs/instantel_protocol_reference.md`). Structural framing is ~50% decoded on branch `claude/codec-re-cBGNe` (tagged-block walker, segment counters); per-byte sample mapping is still open. Until this lands, the in-app waveform viewer renders garbage and BW-import peak values fall back to `_peaks_from_samples()` saturation noise. Workaround: pair every BW-imported event with its `_ASCII.TXT` so the device-authoritative peaks land in the DB regardless of codec.
- [ ] **Series IV (Thor IDF) binary codec reverse-engineering.** `.IDFH` / `.IDFW` files are currently stored opaquely by `WaveformStore.save_imported_idf`, with all metadata sourced from the paired `.txt` sidecar. This works because thor-watcher forwards both files together, but operators who haven't enabled Thor's TXT exporter get rows with NULL peaks. Cracking the binary closes that gap and unlocks waveform display. Starting-point reference at [`docs/idf_protocol_reference.md`](docs/idf_protocol_reference.md) — two observed file signatures (1,012 newer-firmware files + 2 old files whose layout matches the Series III STRT-record format), suggested first-session plan (~2-4 hrs), 1,014 paired binary+txt files available as ground truth in `thor-watcher/example-data/`. Code seam ready at `micromate/idf_file.py`.
- [ ] **In-app waveform viewer accuracy.** Depends on Series III codec decode. Plot.v1 JSON pipeline + viewer skeleton already exist; will start showing real waveforms automatically once `_decode_a5_waveform` produces correct samples. Series IV waveforms come online when the IDF codec lands.
- [ ] **Series IV live-device support.** Once the IDF binary is decoded, extend `micromate/` with `transport.py` / `framing.py` / `protocol.py` / `client.py` mirroring the `minimateplus/` package layout — depends on capturing Thor's wire protocol (TCP / RS-232 captures TBD).
- [ ] **Terra-view integration** — seismo-relay router, unit detail page, VISON-style event listing.
- [ ] **Vibration summary reports** — highest legit PPV per project → Word doc (false-trigger filtering first).
### BW ASCII report parser enhancements (built in v0.16.0)
- [ ] **Histogram-specific structural fields.** Current parser handles the shared fields (PPV, ZC Freq, sensor self-check, project) but silently drops histogram-only fields: `Histogram Start/Stop Time`, `Histogram Start/Stop Date`, `Number of Intervals`, `Interval Size`, per-channel `Peak Time` + `Peak Date` (absolute timestamps rather than the waveform's `Time of Peak` relative seconds).
- [ ] **Histogram interval bin-table parsing.** Trailing 792-row table (per-interval Peak/Freq per channel + MicL) in histogram TXTs is unparsed. Probably too big for the sidecar JSON; may want a separate `.histogram.h5` companion file.
- [ ] **`>100 Hz` value parsing.** Histogram TXTs use `>100 Hz` for out-of-range ZC freq; current `_parse_number()` returns `None` for these (loses information).
### Ingestion gaps
- [ ] **MLG forwarding.** `series3-watcher` forwards event binaries + their `_ASCII.TXT` reports, but skips `.MLG` per-unit monitor log files entirely. Adding an `POST /db/import/mlg_file` endpoint + watcher scan path would populate `monitor_log` for non-ACH-routed units (coverage queries, "was this unit monitoring on date X" lookups).
- [ ] **0C-record raw bytes persistence in the sidecar.** Currently on branch `claude/codec-re-cBGNe` as commit `a187124`; cherry-pick if useful as a standalone fix. Preserves the 210-byte 0C record under `extensions.raw_records.waveform_record_b64` so future field-offset analysis (Peak Acceleration / Time of Peak / etc. — the fields BW computes client-side from samples) can run offline.
### Operational
- [ ] **`series3-watcher` file archive manager** — 90-day-old events moved to `<watch_folder>_archive/<year>/<month>/` subfolders. Plan drafted in `claude/codec-re-cBGNe`'s plan-mode session; awaiting a 5-minute test on whether Blastware UI walks subfolders before any code lands (determines layout: in-place subfolders vs sibling archive).
- [ ] **Compliance config encoder** — build raw write payloads from a `ComplianceConfig` object.
- [ ] **Modem manager** — push RV50/RV55 configs via Sierra Wireless API.
- [ ] **Call Home dial_string write support** (requires DLE escaping for embedded control characters).
- [ ] **Histogram mode recording support** (5A stream analysis for mode 0x03 — separate from histogram ASCII parsing above).
### Test coverage
- [ ] Verify 30-sec event download — body may exceed `0xFFFF` and force the device into a different `end_key` encoding (none of the 2/3/10-sec test cases hit this boundary).
- [ ] Histogram mode (0x03) write via SFM — confirmed working for Single Shot / Continuous / Histogram+Continuous; Histogram (0x03) needs a live test from a non-Histogram starting state.
### Lower-priority cleanups
- [ ] Compliance write anchor-9 cleanup — when changing recording_mode via SFM, a spurious `0x10` may persist after Histogram→other mode transitions. Doesn't affect device operation but differs from BW's byte-perfect output.
- [ ] Locate "Sensor Check" byte in compliance config (need capture with Disabled vs Before-monitoring).
- [ ] Call Home — map time slots 3/4 offsets; confirm `modem_power_relay_enabled`.
- [ ] RV55 DCD/DTR — newer RV55 firmware doesn't assert DCD by default; units don't resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred).
analysis/README.md
+66
View File
@@ -0,0 +1,66 @@
# analysis/ — exploratory scripts for waveform-body RE
**These are scratch.** Run them, read them, copy them, but don't trust
them as documentation. When a finding is verified it gets promoted
to `minimateplus/waveform_codec.py` and `tests/test_waveform_codec.py`;
when it's wrong it stays here as a fossil.
Authoritative status lives in:
- `docs/waveform_codec_re_status.md` (current truth, working note)
- `minimateplus/waveform_codec.py` (verified implementation + docstring)
- `tests/test_waveform_codec.py` (regression locks against fixtures)
---
## Still useful
| File | What it does |
|---|---|
| `load_bundle.py` | Fixture loader. Parses BW binary + ASCII TXT into a `Bundle` dataclass with samples, metadata, body bytes. Used by most other scripts here. |
| `verify_tran.py` | Verifies `decode_tran_initial` against fixture ground truth across all events. Useful when you change the decoder and want a quick sanity check. |
| `inspect_5_11.py` | Inspects the 5-11-26 high-amplitude bundle's body structure, prints metadata, peaks, and block counts. |
| `walk_5_11.py` | Walks blocks for the 5-11-26 bundle and prints offset/tag/length/data. |
| `seg1_blocks.py` | Dumps all blocks in segment 1 of each event. The starting point for cracking multi-segment Tran continuation. |
| `full_tran.py` | Multi-segment Tran decoder attempt (broken — diverges at sample ~512). Useful as a starting scaffold for the next experiment. |
| `multi_segment.py` | Earlier multi-segment attempt with different segment-header consumption strategies. Records what didn't work. |
| `test_rle.py` | Tests `00 NN` interpretation as zero-RLE with different divisor values. Documents how the RLE rule was confirmed. |
## Superseded — keep for archaeology
| File | Superseded by |
|---|---|
| `walk_v2.py` … `walk_v5.py` | `walk_v6.py` and ultimately `minimateplus/waveform_codec.walk_body`. Each version represents one round of refinement. Don't read in isolation — read the diff between them to see what was learned. |
| `walk_chunks.py` | `walk_v6.py` / production walker |
| `decode_v1.py` | First naive decoder attempt. Wrong but readable. |
## Pure exploration — read if curious
| File | What it explored |
|---|---|
| `inspect_body.py` | Byte-frequency stats per event. Established that bytes 0x00 / 0x10 dominate. |
| `find_blocks.py` | Searched for repeating 2-byte tag patterns. |
| `find_signal_runs.py` | Searched for stretches of bytes that "look like a smooth signal" (small inter-byte deltas). Found the `20 NN` literal blocks. |
| `dump_head.py`, `dump_trailer.py`, `dump_around.py` | Hex dumpers at various body positions. |
| `compare_cd.py` | Byte-diff between event-c and event-d (same length, similar signal). Used to identify structural vs data bytes. |
| `brute_force.py` | Tested 96 combinations of channel-permutation × nibble-order × sign-convention × init-from-header on the quiet bundle. All failed because the quiet bundle had T[0]=T[1]=0, making the preamble undetectable. |
| `try_nibbles.py`, `try_layouts.py` | Earlier channel-interleaving hypotheses. All wrong. |
| `test_tran_continue.py` | Test of "Tran continues uninterrupted across `30 04` blocks" hypothesis. Disproven. |
---
## Adding new scripts
If you're picking up the codec work, feel free to add new scripts here.
Suggested conventions:
- Start the filename with what you're testing: `test_<hypothesis>.py`,
`verify_<piece>.py`, `inspect_<region>.py`.
- Print enough output that the reader can see exactly which events
match / diverge and where.
- When a finding is solid, move the verified logic to
`minimateplus/waveform_codec.py` and add a regression test in
`tests/test_waveform_codec.py` — don't leave the truth only in
this directory.
- If a script is fully superseded, leave it in place (don't delete) —
the fossil record is useful when re-evaluating hypotheses later.
analysis/brute_force.py
+93
View File
@@ -0,0 +1,93 @@
"""Brute-force test channel permutations / nibble orders on event-d (simplest signal)."""
import sys
import itertools
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
from minimateplus.waveform_codec import walk_body
def s4(n):
return n if n < 8 else n - 16
def decode(body, channel_perm, nibble_order, sign_mode, init_from_header):
"""Try one decoder configuration on event-d. Returns first 8 cumulative samples per channel."""
blocks = walk_body(body)
# Initial values from bytes [4:7] if init_from_header else 0
if init_from_header:
init = [body[4] if body[4] < 128 else body[4] - 256,
body[5] if body[5] < 128 else body[5] - 256,
body[6] if body[6] < 128 else body[6] - 256,
0]
else:
init = [0, 0, 0, 0]
cur = list(init)
out = [[init[0]], [init[1]], [init[2]], [init[3]]] # sample 0 = init
nibble_idx = 0 # within delta stream; channel = channel_perm[nibble_idx % 4]
# Walk only the 10 NN data blocks
for blk in blocks:
if blk.tag_hi != 0x10:
continue
for byte in blk.data:
if nibble_order == 'high_first':
nib1, nib2 = (byte >> 4) & 0xF, byte & 0xF
else:
nib1, nib2 = byte & 0xF, (byte >> 4) & 0xF
for nib in (nib1, nib2):
if sign_mode == 'signed':
delta = s4(nib)
else:
delta = nib
ch = channel_perm[nibble_idx % 4]
cur[ch] += delta
if (nibble_idx + 1) % 4 == 0:
out[0].append(cur[0])
out[1].append(cur[1])
out[2].append(cur[2])
out[3].append(cur[3])
nibble_idx += 1
if len(out[0]) >= 16:
return out
return out
def best_match(pred, truth, n=10):
"""Sum of squared differences in first n samples."""
n = min(n, len(pred), len(truth))
return sum((pred[i] - truth[i])**2 for i in range(n))
def main():
b = load_bundle("event-d")
# truth in 16-count units
tr = {ch: [round(v * 200) for v in b.samples[ch]] for ch in ("Tran", "Vert", "Long")}
print("Truth event-d first 10 samples:")
for ch in ("Tran", "Vert", "Long"):
print(f" {ch}: {tr[ch][:10]}")
# Test 96 combinations
best = []
for perm in itertools.permutations([0, 1, 2, 3]):
for nibble_order in ('high_first', 'low_first'):
for sign in ('signed', 'unsigned'):
for init_h in (False, True):
decoded = decode(b.body, perm, nibble_order, sign, init_h)
# Score as TVL channel-sum
score = sum(
best_match(decoded[i], tr[ch], n=10)
for i, ch in enumerate(("Tran", "Vert", "Long"))
if i < 3
)
label = f"perm={perm} nib={nibble_order[:1]} sign={sign[:3]} init={init_h}"
best.append((score, label, decoded))
best.sort(key=lambda x: x[0])
print(f"\nTop 10 configurations:")
for s, lbl, dec in best[:10]:
print(f" score={s:>5} {lbl} T={dec[0][:8]} V={dec[1][:8]} L={dec[2][:8]}")
if __name__ == "__main__":
main()
analysis/compare_cd.py
+42
View File
@@ -0,0 +1,42 @@
"""Compare event-c and event-d (same N_samples) to find header vs data bytes."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def main():
bc = load_bundle("event-c")
bd = load_bundle("event-d")
# Compare prefixes
nc, nd = len(bc.body), len(bd.body)
n = min(nc, nd)
diffs = []
for i in range(n):
if bc.body[i] != bd.body[i]:
diffs.append(i)
print(f"event-c body={nc}, event-d body={nd}")
print(f"Total diffs (first {n}): {len(diffs)}")
# Show common prefix
same_prefix = 0
for i in range(n):
if bc.body[i] == bd.body[i]:
same_prefix += 1
else:
break
print(f"Common prefix length: {same_prefix}")
print(f"event-c prefix: {bc.body[:same_prefix].hex(' ')}")
# Look for runs of common bytes
print(f"\nFirst 32 diff positions: {diffs[:32]}")
# Show the "diff fingerprint" of the first 100 bytes
print(f"\n pos c d")
for i in range(0, 100):
marker = " " if bc.body[i] == bd.body[i] else "*"
bd_b = bd.body[i] if i < nd else None
print(f" {i:>3} {bc.body[i]:02x}{marker} {bd_b:02x}" if bd_b is not None else f" {i:>3} {bc.body[i]:02x}{marker}")
if __name__ == "__main__":
main()
analysis/decode_v1.py
+99
View File
@@ -0,0 +1,99 @@
"""
Decoder v1: nibble-pair signed deltas in 10 NN blocks, 4-channel round-robin.
"""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def s4(n):
return n if n < 8 else n - 16
def walk_blocks(body, start):
i = start
blocks = []
while i + 1 < len(body):
t0, t1 = body[i], body[i + 1]
if t0 == 0x10 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
length = t1 // 2 + 2
data = bytes(body[i + 2 : i + length])
blocks.append(("10", t1, data))
i += length
elif t0 == 0x20 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
length = t1 + 2
data = bytes(body[i + 2 : i + length])
blocks.append(("20", t1, data))
i += length
elif t0 == 0x00 and t1 % 4 == 0:
blocks.append(("00", t1, b""))
i += 2
elif t0 == 0x30 and t1 % 4 == 0 and 0 < t1 <= 0x10:
length = t1 * 4
data = bytes(body[i + 2 : i + length])
blocks.append(("30", t1, data))
i += length
elif t0 == 0x40 and t1 == 0x02:
length = 20
data = bytes(body[i + 2 : i + length])
blocks.append(("40", t1, data))
i += length
else:
blocks.append(("??", t0, bytes(body[i:i+8])))
break
return blocks
def decode_v1(body, start, n_samples):
"""Decode by accumulating nibble-pair deltas from all 10 NN blocks."""
blocks = walk_blocks(body, start)
# 4 channels: T, V, L, M
cur = [0, 0, 0, 0]
out = [[], [], [], []]
sample_index = 0 # how many sample-sets emitted
for typ, NN, data in blocks:
if typ == "10":
# 2 nibbles per byte, round-robin TVLM
for byte in data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
ch = sample_index % 4
cur[ch] += s4(nib)
out[ch].append(cur[ch])
sample_index = (sample_index + 1) // 4 * 4 + (sample_index + 1) % 4 # ?
sample_index += 1
# We emit per-nibble, but the structure is unclear
elif typ == "20":
# int8 absolute or delta?
for byte in data:
v = byte if byte < 128 else byte - 256
ch = sample_index % 4
cur[ch] = v # treat as absolute
out[ch].append(cur[ch])
sample_index += 1
return out
def main():
b = load_bundle("event-c")
body = b.body
truth_T = [round(v * 200) for v in b.samples["Tran"]]
truth_V = [round(v * 200) for v in b.samples["Vert"]]
truth_L = [round(v * 200) for v in b.samples["Long"]]
# Find start
for s in range(15):
if body[s] == 0x10 and body[s+1] % 4 == 0 and 0 < body[s+1] <= 0xFC:
start = s
break
blocks = walk_blocks(body, start)
# Print block-by-block what's in each
print(f"Total blocks: {len(blocks)}")
bytes_processed = 0
for typ, NN, data in blocks[:30]:
print(f" type={typ} NN=0x{NN:02x} data_len={len(data)} data_hex={data[:32].hex(' ')}{'...' if len(data) > 32 else ''}")
if __name__ == "__main__":
main()
analysis/dump_around.py
+27
View File
@@ -0,0 +1,27 @@
"""Dump body bytes around a specific offset."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def dump_around(name: str, center: int, radius: int = 96):
b = load_bundle(name)
body = b.body
start = max(0, center - radius)
end = min(len(body), center + radius)
print(f"\n=== {name} body[{start}:{end}] (full body={len(body)}) ===")
for i in range(start, end, 32):
row = body[i:i+32]
marker = " <-- center" if i <= center < i+32 else ""
print(f" +{i:>5} {row.hex(' ')}{marker}")
def main():
# Look at the trailer transitions
trailer_starts = {"event-a": 7047, "event-b": 6475, "event-c": 4043, "event-d": 3941}
for name, off in trailer_starts.items():
dump_around(name, off, 96)
if __name__ == "__main__":
main()
analysis/dump_head.py
+18
View File
@@ -0,0 +1,18 @@
"""Dump the START of each body in 32-byte rows."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def main():
for name in ("event-a", "event-c"):
b = load_bundle(name)
body = b.body
print(f"\n=== {name} body[0:512] (full body={len(body)}, samples={len(b.samples['Tran'])}) ===")
for i in range(0, min(512, len(body)), 32):
row = body[i:i+32]
print(f" +{i:>5} {row.hex(' ')}")
if __name__ == "__main__":
main()
analysis/dump_trailer.py
+24
View File
@@ -0,0 +1,24 @@
"""Dump body bytes split into 32-byte rows starting from `start_offset`."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def dump(body: bytes, name: str, start: int, n_rows: int = 30):
print(f"\n=== {name} body[{start}:] (full body={len(body)}) ===")
end = min(start + 32 * n_rows, len(body))
for i in range(start, end, 32):
row = body[i:i+32]
print(f" +{i:>5} {row.hex(' ')}")
def main():
for name in ("event-a", "event-b", "event-c", "event-d"):
b = load_bundle(name)
# Print the LAST ~600 bytes of the body to see the tail structure
start = max(0, len(b.body) - 32 * 12)
dump(b.body, name, start, 12)
if __name__ == "__main__":
main()
analysis/find_blocks.py
+41
View File
@@ -0,0 +1,41 @@
"""Search for structural repetition in the body bytes."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def find_pattern_offsets(body: bytes, pattern: bytes, max_count=20):
out = []
i = 0
while True:
i = body.find(pattern, i)
if i < 0:
break
out.append(i)
i += 1
if len(out) >= max_count:
break
return out
def main():
for name in ("event-a", "event-b", "event-c", "event-d"):
b = load_bundle(name)
body = b.body
print(f"\n=== {name} (body={len(body)}, N_samples={len(b.samples['Tran'])}) ===")
# Try to find repeating substructures (look for 4-byte 0x10-prefixed markers)
for prefix in [b"\x10\x10", b"\x10\x04", b"\x10\x08", b"\x10\x0c", b"\x10\x18",
b"\x10\x14", b"\x10\x20", b"\x10\x40", b"\x10\x80", b"\x10\x00",
b"\x10\x01", b"\x10\x03", b"\x10\xf0", b"\xf1\x10", b"\x00\x10",
b"\x40\x02", b"\x20\x04", b"\x30\x04", b"\x30\x08", b"\x00\x1a"]:
offs = find_pattern_offsets(body, prefix, max_count=200)
if 1 <= len(offs) <= 1000:
# Print first 10 offsets
first = offs[:6]
last = offs[-3:]
print(f" '{prefix.hex()}' x{len(offs):>4} first={first} last={last}")
if __name__ == "__main__":
main()
analysis/find_signal_runs.py
+34
View File
@@ -0,0 +1,34 @@
"""Find body byte ranges that look like absolute int8 sample data (smooth waveform)."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def looks_like_smooth_int8(buf):
"""Convert bytes to int8 and check if successive deltas are small (waveform-like)."""
if len(buf) < 8:
return 0.0
vals = [b if b < 128 else b - 256 for b in buf]
diffs = [abs(vals[i+1] - vals[i]) for i in range(len(vals)-1)]
avg_diff = sum(diffs) / len(diffs)
return avg_diff
def main():
for name in ("event-a", "event-c"):
b = load_bundle(name)
body = b.body
# Scan with sliding window of 64 bytes; find segments where the bytes look like a smooth wave
win = 64
scores = []
for i in range(len(body) - win):
scores.append((i, looks_like_smooth_int8(body[i:i+win])))
# Lowest avg_diff means smoothest
scores.sort(key=lambda x: x[1])
print(f"\n=== {name} (body={len(body)}) — smoothest 10 windows ===")
for off, s in scores[:10]:
print(f" +{off:>5} avg_diff={s:.2f} bytes={body[off:off+24].hex(' ')}")
if __name__ == "__main__":
main()
analysis/full_tran.py
+76
View File
@@ -0,0 +1,76 @@
"""Full Tran decoder: continues across segment headers using T_delta from header bytes [0:2]."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import _parse_txt
from minimateplus.waveform_codec import walk_body, find_data_start
def s4(n):
return n if n < 8 else n - 16
def i8(b):
return b if b < 128 else b - 256
def decode_full_tran(body):
if len(body) < 7 or body[0:3] != b"\x00\x02\x00":
return None
T0 = int.from_bytes(body[3:5], "big", signed=True)
T1 = int.from_bytes(body[5:7], "big", signed=True)
i = 7
while i + 1 < len(body) and body[i] not in (0x00, 0x10, 0x20, 0x30, 0x40):
i += 1
blocks = walk_body(body, i)
T = [T0, T1]
cur = T1
for blk in blocks:
if blk.tag_hi == 0x40:
# Segment header carries 2 T deltas (int16 BE each) at bytes [0:2] and [2:4]
if len(blk.data) >= 4:
delta1 = int.from_bytes(blk.data[0:2], "big", signed=True)
cur += delta1
T.append(cur)
delta2 = int.from_bytes(blk.data[2:4], "big", signed=True)
cur += delta2
T.append(cur)
elif blk.tag_hi == 0x10:
for byte in blk.data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
cur += s4(nib)
T.append(cur)
elif blk.tag_hi == 0x20:
for byte in blk.data:
cur += i8(byte)
T.append(cur)
elif blk.tag_hi == 0x00:
for _ in range(blk.tag_lo):
T.append(cur)
# 30 NN: skip for now
return T
def main():
for stem in ("M529LL1L.V70", "M529LL1L.JQ0", "M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
path = f"tests/fixtures/5-11-26/{stem}"
with open(path, "rb") as f:
body = f.read()[43:-26]
_, samples = _parse_txt(path + ".TXT")
truth_T = [round(v*200) for v in samples["Tran"]]
n_truth = len(truth_T)
decoded = decode_full_tran(body)
n = min(len(decoded), n_truth)
matches = sum(1 for i in range(n) if decoded[i] == truth_T[i])
div_at = -1
for i in range(n):
if decoded[i] != truth_T[i]:
div_at = i
break
print(f"{stem}: decoded={len(decoded)}, truth={n_truth}, matches={matches}/{n}, first div={div_at}")
if __name__ == "__main__":
main()
analysis/inspect_5_11.py
+50
View File
@@ -0,0 +1,50 @@
"""Quick inspection of the new high-amplitude events."""
import os, re, sys
sys.path.insert(0, ".")
from analysis.load_bundle import _parse_txt
from minimateplus.waveform_codec import walk_body, find_data_start
ROOT = "tests/fixtures/5-11-26"
def main():
for stem in ("M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
bin_path = os.path.join(ROOT, stem)
txt_path = bin_path + ".TXT"
with open(bin_path, "rb") as f:
raw = f.read()
body = raw[43:-26]
meta, samples = _parse_txt(txt_path)
n = len(samples["Tran"])
print(f"\n=== {stem} ===")
print(f" file={len(raw)}, body={len(body)}, N_samples={n}")
print(f" rectime={meta.get('Record Time')} pretrig={meta.get('Pre-trigger Length')}")
print(f" PPV(T,V,L)={meta.get('Tran PPV')} / {meta.get('Vert PPV')} / {meta.get('Long PPV')}")
# Show first few non-trivial samples
print(f" First 5 truth samples (in/s):")
for i in range(5):
print(f" T={samples['Tran'][i]:8.3f} V={samples['Vert'][i]:8.3f} "
f"L={samples['Long'][i]:8.3f} M={samples['MicL'][i]:8.3f}")
# Peak sample positions
for ch in ("Tran", "Vert", "Long"):
vals = samples[ch]
peak_i = max(range(n), key=lambda i: abs(vals[i]))
print(f" {ch}: peak {vals[peak_i]:.3f} at sample {peak_i} (t={peak_i/1024:.3f}s)")
# Body structure
start = find_data_start(body)
blocks = walk_body(body, start)
types = {}
for b in blocks:
types[b.tag_hi] = types.get(b.tag_hi, 0) + 1
print(f" body start={start}, total blocks walked: {len(blocks)}")
print(f" block tag counts: {types}")
# How far the walker got
if blocks:
last = blocks[-1]
walked = last.offset + last.length
print(f" walker stopped at offset {walked}/{len(body)} ({100*walked/len(body):.0f}%)")
if __name__ == "__main__":
main()
analysis/inspect_body.py
+23
View File
@@ -0,0 +1,23 @@
"""Print raw body hex + byte-distribution stats for one event."""
from collections import Counter
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def main():
for name in ("event-a", "event-b", "event-c", "event-d"):
b = load_bundle(name)
body = b.body
print(f"\n=== {name} ({len(body)} body bytes) ===")
print(f" STRT: {b.strt.hex()}")
print(f" body[0:64]: {body[:64].hex()}")
print(f" body[64:128]: {body[64:128].hex()}")
print(f" body[-32:]: {body[-32:].hex()}")
cnt = Counter(body)
print(f" top 16 bytes: {[(f'0x{k:02x}', f'{v/len(body):.2%}') for k,v in cnt.most_common(16)]}")
if __name__ == "__main__":
main()
analysis/load_bundle.py
+144
View File
@@ -0,0 +1,144 @@
"""
load_bundle.py — extract body bytes from BW binary + parse sample columns from TXT.
Used by the codec reverse-engineering scripts in this directory.
"""
from __future__ import annotations
import os
import re
from dataclasses import dataclass
BUNDLE_ROOT = os.path.join(
os.path.dirname(__file__), "..", "tests", "fixtures", "decode-re-5-8-26"
)
@dataclass
class Bundle:
name: str
bin_path: str
txt_path: str
bin: bytes
body: bytes # bytes between STRT (43) and footer (last 26)
strt: bytes # 21-byte STRT record
samples: dict # {"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}
sample_rate: int
rectime_sec: float
pretrig_sec: float
geo_range_ips: float
ppv: dict # {"Tran": float, "Vert": float, "Long": float}
mic_pspl: float
serial: str
def _parse_txt(path: str) -> dict:
with open(path, "r", encoding="utf-8", errors="replace") as f:
text = f.read()
meta = {}
samples = {"Tran": [], "Vert": [], "Long": [], "MicL": []}
# Find header line that starts the columns ("Tran Vert Long MicL").
# Then every line after is sample data (4 tab-separated floats).
lines = text.splitlines()
header_idx = None
for i, line in enumerate(lines):
if "Tran" in line and "Vert" in line and "Long" in line and "MicL" in line:
# The columns header. Sample lines start a few lines later.
header_idx = i
break
if header_idx is None:
raise ValueError(f"no Tran/Vert/Long/MicL header in {path}")
# Parse meta — quoted lines with "Field : value"
for line in lines[:header_idx]:
m = re.match(r'^"([^"]+)\s*:\s*([^"]*)"', line.strip())
if m:
k, v = m.group(1).strip(), m.group(2).strip()
meta[k] = v
# Parse samples
for line in lines[header_idx + 1 :]:
line = line.strip()
if not line:
continue
parts = re.split(r"\s+", line)
if len(parts) < 4:
continue
try:
t = float(parts[0])
v = float(parts[1])
l = float(parts[2])
m = float(parts[3])
except ValueError:
continue
samples["Tran"].append(t)
samples["Vert"].append(v)
samples["Long"].append(l)
samples["MicL"].append(m)
return meta, samples
def load_bundle(name: str) -> Bundle:
folder = os.path.join(BUNDLE_ROOT, name)
files = os.listdir(folder)
bin_name = next(f for f in files if not f.endswith(".TXT"))
txt_name = next(f for f in files if f.endswith(".TXT"))
bin_path = os.path.join(folder, bin_name)
txt_path = os.path.join(folder, txt_name)
with open(bin_path, "rb") as f:
binary = f.read()
# Header is 22 bytes; STRT at [22:43]; footer at last 26 bytes.
strt = binary[22:43]
body = binary[43:-26]
meta, samples = _parse_txt(txt_path)
sample_rate = int(re.search(r"(\d+)", meta.get("Sample Rate", "1024")).group(1))
rectime_sec = float(re.search(r"([\d.]+)", meta.get("Record Time", "3.0")).group(1))
pretrig_sec = float(re.search(r"-?[\d.]+", meta.get("Pre-trigger Length", "0")).group(0))
geo_range_ips = float(re.search(r"([\d.]+)", meta.get("Geo Range", "10.0")).group(1))
serial = meta.get("Serial Number", "").strip()
def _f(s):
return float(re.search(r"-?[\d.]+", s).group(0))
ppv = {
"Tran": _f(meta.get("Tran PPV", "0")),
"Vert": _f(meta.get("Vert PPV", "0")),
"Long": _f(meta.get("Long PPV", "0")),
}
mic_pspl = _f(meta.get("MicL PSPL", "0"))
return Bundle(
name=name,
bin_path=bin_path,
txt_path=txt_path,
bin=binary,
body=body,
strt=strt,
samples=samples,
sample_rate=sample_rate,
rectime_sec=rectime_sec,
pretrig_sec=pretrig_sec,
geo_range_ips=geo_range_ips,
ppv=ppv,
mic_pspl=mic_pspl,
serial=serial,
)
if __name__ == "__main__":
for name in ("event-a", "event-b", "event-c", "event-d"):
b = load_bundle(name)
n = len(b.samples["Tran"])
print(f"{name}: body={len(b.body):>6} N_samples={n} rate={b.sample_rate} "
f"rectime={b.rectime_sec} pretrig={b.pretrig_sec} range={b.geo_range_ips} "
f"PPV(T,V,L)={b.ppv['Tran']:.3f},{b.ppv['Vert']:.3f},{b.ppv['Long']:.3f} "
f"MicL={b.mic_pspl}")
analysis/multi_segment.py
+81
View File
@@ -0,0 +1,81 @@
"""Decode Tran across multiple segments by resetting at 40 02 headers."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import _parse_txt
from minimateplus.waveform_codec import walk_body, find_data_start
def s4(n):
return n if n < 8 else n - 16
def i8(b):
return b if b < 128 else b - 256
def decode_full_tran(body):
"""Decode all Tran samples in the body, walking through segments."""
if len(body) < 7 or body[0:3] != b"\x00\x02\x00":
return None
T0 = int.from_bytes(body[3:5], "big", signed=True)
T1 = int.from_bytes(body[5:7], "big", signed=True)
# Locate first tag
i = 7
while i + 1 < len(body) and body[i] not in (0x00, 0x10, 0x20, 0x30, 0x40):
i += 1
blocks = walk_body(body, i)
T = [T0, T1]
cur = T1
for bi, blk in enumerate(blocks):
if blk.tag_hi == 0x40:
# Segment header — try interpreting bytes [0:2] as new T anchor
if len(blk.data) >= 2:
new_anchor = int.from_bytes(blk.data[0:2], "big", signed=True)
# The next sample IS this anchor value, NOT a delta from cur.
T.append(new_anchor)
cur = new_anchor
elif blk.tag_hi == 0x10:
for byte in blk.data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
cur += s4(nib)
T.append(cur)
elif blk.tag_hi == 0x20:
for byte in blk.data:
cur += i8(byte)
T.append(cur)
elif blk.tag_hi == 0x00:
# RLE: append NN zero deltas
for _ in range(blk.tag_lo):
T.append(cur)
# 30 NN: skip
return T
def main():
for stem in ("M529LL1L.V70", "M529LL1L.JQ0", "M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
path = f"tests/fixtures/5-11-26/{stem}"
with open(path, "rb") as f:
body = f.read()[43:-26]
_, samples = _parse_txt(path + ".TXT")
truth_T = [round(v*200) for v in samples["Tran"]]
n_truth = len(truth_T)
decoded = decode_full_tran(body)
n = min(len(decoded), n_truth)
matches = sum(1 for i in range(n) if decoded[i] == truth_T[i])
# Find first divergence
div_at = -1
for i in range(n):
if decoded[i] != truth_T[i]:
div_at = i
break
print(f"{stem}: decoded={len(decoded)}, truth={n_truth}, matches={matches}/{n}, first div={div_at}")
if div_at >= 0 and div_at < 30:
print(f" truth around div [{max(0,div_at-3)}:{div_at+8}]: {truth_T[max(0,div_at-3):div_at+8]}")
print(f" pred around div [{max(0,div_at-3)}:{div_at+8}]: {decoded[max(0,div_at-3):div_at+8]}")
if __name__ == "__main__":
main()
analysis/seg1_blocks.py
+28
View File
@@ -0,0 +1,28 @@
"""Dump all blocks in segment 1 of each event with their data."""
import sys
sys.path.insert(0, ".")
from minimateplus.waveform_codec import walk_body, find_data_start
def main():
for stem in ("M529LL1A.SP0", "M529LL1L.JQ0", "M529LL1L.V70"):
path = f"tests/fixtures/5-11-26/{stem}"
with open(path, "rb") as f:
body = f.read()[43:-26]
blocks = walk_body(body, find_data_start(body))
# Find segment 1 (between first and second 40 02)
seg40_indices = [i for i, b in enumerate(blocks) if b.tag_hi == 0x40]
if len(seg40_indices) < 2:
print(f"\n{stem}: only {len(seg40_indices)} segment headers found")
seg1_blocks = blocks[seg40_indices[0]:] if seg40_indices else []
else:
seg1_blocks = blocks[seg40_indices[0]:seg40_indices[1]+1]
print(f"\n=== {stem} segment 1 ({len(seg1_blocks)} blocks) ===")
for b in seg1_blocks[:25]:
tag = f"{b.tag_hi:02x}{b.tag_lo:02x}"
print(f" off={b.offset:>5} {tag} NN=0x{b.tag_lo:02x}({b.tag_lo:>3}) len={b.length:>3} data={b.data[:16].hex(' ')}{'...' if len(b.data)>16 else ''}")
if __name__ == "__main__":
main()
analysis/test_30nn_12bit.py
+195
View File
@@ -0,0 +1,195 @@
"""Test 12-bit signed packed deltas hypothesis for 30 NN blocks across all loud events.
For each 30 NN block in each event, identify what samples it should cover
(based on the cumulative delta count up to that point) and compare the
truth deltas against various 12-bit packing schemes.
"""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import _parse_txt
from minimateplus.waveform_codec import walk_body, find_data_start
CHANNEL_ORDER = ["Vert", "Long", "MicL", "Tran"] # rotation after initial T
def s12(v):
"""Sign-extend a 12-bit unsigned value to signed int."""
return v if v < 0x800 else v - 0x1000
def unpack_12bit_be(data):
"""4 deltas in 6 bytes, BE order: byte[0:1.5], byte[1.5:3], byte[3:4.5], byte[4.5:6]."""
# bits 0..47 (MSB-first), split into 4 × 12-bit
val = int.from_bytes(data, "big")
out = []
for i in range(4):
d = (val >> (12 * (3 - i))) & 0xFFF
out.append(s12(d))
return out
def unpack_12bit_le(data):
"""4 deltas in 6 bytes, LE order: bytes packed as 2 × 24-bit groups."""
out = []
# First 3 bytes contain 2 deltas
b0, b1, b2 = data[0], data[1], data[2]
d0 = b0 | ((b1 & 0x0F) << 8)
d1 = (b1 >> 4) | (b2 << 4)
out.append(s12(d0))
out.append(s12(d1))
# Next 3 bytes contain 2 more deltas
b3, b4, b5 = data[3], data[4], data[5]
d2 = b3 | ((b4 & 0x0F) << 8)
d3 = (b4 >> 4) | (b5 << 4)
out.append(s12(d2))
out.append(s12(d3))
return out
def unpack_12bit_be_per_triplet(data):
"""4 deltas as 2 triplets of (high4, low8) BE within each 3-byte group."""
out = []
b0, b1, b2 = data[0], data[1], data[2]
d0 = (b0 << 4) | (b1 >> 4)
d1 = ((b1 & 0x0F) << 8) | b2
out.append(s12(d0))
out.append(s12(d1))
b3, b4, b5 = data[3], data[4], data[5]
d2 = (b3 << 4) | (b4 >> 4)
d3 = ((b4 & 0x0F) << 8) | b5
out.append(s12(d2))
out.append(s12(d3))
return out
def truth_deltas_for_block(blocks, block_idx, event_truth, channel):
"""For a 30 NN block at block_idx, determine which samples it covers and
return the truth deltas for those samples.
Walks through all blocks before block_idx (within the same segment) and
counts how many deltas have been emitted for *channel*, starting from the
segment's anchor pair.
"""
# Find the segment header that contains this block.
seg_header_idx = None
for j in range(block_idx, -1, -1):
if blocks[j].tag_hi == 0x40:
seg_header_idx = j
break
if seg_header_idx is None:
# block is in the initial T segment; samples count from sample 2.
first_sample_in_segment = 2
else:
# Anchor pair covers samples [N, N+1] for some N. Subsequent deltas
# are samples [N+2, N+2+1, ...]. We don't actually need to know N
# for this test — just the relative position within the segment.
first_sample_in_segment = 2 # anchor=0,1; deltas start at 2
# Count deltas from segment-data start to block_idx.
delta_count = 0
start_block = seg_header_idx + 1 if seg_header_idx is not None else 0
for j in range(start_block, block_idx):
blk = blocks[j]
if blk.tag_hi == 0x10:
delta_count += blk.tag_lo # NN nibbles = NN deltas
elif blk.tag_hi == 0x20:
delta_count += blk.tag_lo # NN int8 deltas
elif blk.tag_hi == 0x00:
delta_count += blk.tag_lo # RLE zero deltas
# Now the 30 NN block carries NN deltas.
nn = blocks[block_idx].tag_lo
# First sample affected: segment first_sample + delta_count.
# But we ALSO need to know which segment this is, since the segment maps
# to a specific channel and a specific starting absolute sample index.
return first_sample_in_segment + delta_count, nn
def main():
for stem in ("M529LL1A.SP0", "M529LL1L.JQ0", "M529LL1L.V70",
"M529LL1A.SS0", "M529LL1A.SV0"):
path = f"tests/fixtures/5-11-26/{stem}"
with open(path, "rb") as f:
body = f.read()[43:-26]
_, samples = _parse_txt(path + ".TXT")
blocks = walk_body(body, find_data_start(body))
seg_idx = [i for i, b in enumerate(blocks) if b.tag_hi == 0x40]
# Find all 30 NN blocks in DATA section (not trailer).
thirty_blocks = []
for bi, b in enumerate(blocks):
if b.tag_hi != 0x30:
continue
# Determine which segment this is in
seg_num = None
for k, hi in enumerate(seg_idx):
next_hi = seg_idx[k + 1] if k + 1 < len(seg_idx) else len(blocks)
if hi < bi < next_hi:
seg_num = k
break
if seg_num is None and seg_idx and bi < seg_idx[0]:
seg_num = -1 # initial T segment
thirty_blocks.append((bi, b, seg_num))
if not thirty_blocks:
continue
print(f"\n=== {stem} ===")
for bi, b, seg_num in thirty_blocks:
# Channel for this segment
if seg_num == -1:
channel = "Tran"
seg_label = "initial T"
else:
channel = CHANNEL_ORDER[seg_num % 4]
seg_label = f"seg {seg_num}"
# Count deltas before this block within the same segment.
seg_header_idx = seg_idx[seg_num] if seg_num >= 0 else -1
start_block = seg_header_idx + 1 if seg_header_idx >= 0 else 0
delta_count = 0
for j in range(start_block, bi):
blk = blocks[j]
if blk.tag_hi in (0x10, 0x20, 0x00):
delta_count += blk.tag_lo
# First sample this 30 NN block affects (within the segment)
# = anchor positions + delta_count + 2 (since anchor pair was samples 0,1)
# But the segment's first absolute sample index in the channel is
# (seg_num // 4) * 512 (approximately) if segment 0 is the first V seg.
cycle = (seg_num // 4) if seg_num >= 0 else 0
base = cycle * 512 + 2 # +2 for anchor pair
sample_idx = base + delta_count
truth_ch = [round(v * 200) for v in samples[channel]]
nn = b.tag_lo
if sample_idx + nn >= len(truth_ch):
print(f" block @ {b.offset} ({seg_label} {channel}): out of truth range")
continue
# Get the previous sample so we can compute truth deltas
if sample_idx == 0:
prev = 0
else:
prev = truth_ch[sample_idx - 1]
truth_deltas = []
for k in range(nn):
truth_deltas.append(truth_ch[sample_idx + k] - (prev if k == 0 else truth_ch[sample_idx + k - 1]))
# Try each packing
schemes = [
("12-bit BE contiguous", unpack_12bit_be(b.data)),
("12-bit LE per-triplet", unpack_12bit_le(b.data)),
("12-bit BE per-triplet", unpack_12bit_be_per_triplet(b.data)),
]
print(f" block @ {b.offset:>5} ({seg_label} {channel}, samples {sample_idx}..{sample_idx+nn-1}):")
print(f" data: {b.data.hex(' ')}")
print(f" truth: {truth_deltas}")
for name, pred in schemes:
match = "✓" if pred == truth_deltas else " "
n_match = sum(1 for x, y in zip(pred, truth_deltas) if x == y)
print(f" {match}{n_match}/4 {name}: {pred}")
if __name__ == "__main__":
main()
analysis/test_30nn_hybrid.py
+132
View File
@@ -0,0 +1,132 @@
"""Test the '30 NN data = high-nibbles + int8 low-bytes' hypothesis.
Layout for `30 04` (6 data bytes, 4 deltas):
bytes [0:2] = 16 bits = 4 × 4-bit high-nibbles (MSB first)
bytes [2:6] = 4 × int8 low bytes
Each delta = 12-bit signed = sign-extend((high_nibble << 8) | low_byte)
"""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import _parse_txt
from minimateplus.waveform_codec import walk_body, find_data_start
def s4(n):
return n if n < 8 else n - 16
def i8(b):
return b if b < 128 else b - 256
def sign_extend_12(v):
return v if v < 0x800 else v - 0x1000
def decode_30nn(data):
"""4 × 12-bit signed deltas (high nibble + low byte).
bytes[0:2] hold the 4 high nibbles (MSB first); bytes[2:6] hold the low bytes.
"""
if len(data) < 6:
return []
# Read high nibbles from bytes 0-1 (4 nibbles MSB-first)
high_word = (data[0] << 8) | data[1]
high_nibbles = [
(high_word >> 12) & 0xF,
(high_word >> 8) & 0xF,
(high_word >> 4) & 0xF,
high_word & 0xF,
]
out = []
for i in range(4):
v = (high_nibbles[i] << 8) | data[2 + i]
out.append(sign_extend_12(v))
return out
def simulate_up_to(blocks, target_block_idx, t_preamble):
"""Run decoder up to block_idx; return per-channel sample lists.
NOW with 30 NN decoded too."""
out = {"Tran": [], "Vert": [], "Long": [], "MicL": []}
out["Tran"].extend(t_preamble)
cur = {"Tran": t_preamble[-1], "Vert": None, "Long": None, "MicL": None}
rotation = ["Vert", "Long", "MicL", "Tran"]
current_channel = "Tran"
seg_counter = -1
for j in range(target_block_idx):
blk = blocks[j]
if blk.tag_hi == 0x40:
seg_counter += 1
prev = "Tran" if seg_counter == 0 else rotation[(seg_counter - 1) % 4]
new_ch = rotation[seg_counter % 4]
if cur[prev] is not None:
d0 = int.from_bytes(blk.data[0:2], "big", signed=True)
d1 = int.from_bytes(blk.data[2:4], "big", signed=True)
cur[prev] += d0; out[prev].append(cur[prev])
cur[prev] += d1; out[prev].append(cur[prev])
c0 = int.from_bytes(blk.data[14:16], "big", signed=True)
c1 = int.from_bytes(blk.data[16:18], "big", signed=True)
out[new_ch].extend([c0, c1])
cur[new_ch] = c1
current_channel = new_ch
elif blk.tag_hi == 0x10:
for byte in blk.data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
cur[current_channel] += s4(nib)
out[current_channel].append(cur[current_channel])
elif blk.tag_hi == 0x20:
for byte in blk.data:
cur[current_channel] += i8(byte)
out[current_channel].append(cur[current_channel])
elif blk.tag_hi == 0x00:
for _ in range(blk.tag_lo):
out[current_channel].append(cur[current_channel])
elif blk.tag_hi == 0x30:
# NEW: decode 30 NN
deltas = decode_30nn(blk.data)
for d in deltas:
cur[current_channel] += d
out[current_channel].append(cur[current_channel])
return out, current_channel
def main():
for stem in ("M529LL1A.SP0", "M529LL1L.JQ0", "M529LL1L.V70",
"M529LL1A.SS0", "M529LL1A.SV0"):
path = f"tests/fixtures/5-11-26/{stem}"
with open(path, "rb") as f:
body = f.read()[43:-26]
_, samples = _parse_txt(path + ".TXT")
blocks = walk_body(body, find_data_start(body))
t0 = int.from_bytes(body[3:5], "big", signed=True)
t1 = int.from_bytes(body[5:7], "big", signed=True)
thirty_blocks = [(j, b) for j, b in enumerate(blocks) if b.tag_hi == 0x30]
if not thirty_blocks:
continue
print(f"\n=== {stem} ===")
for j, blk in thirty_blocks:
pred, ch = simulate_up_to(blocks, j, [t0, t1])
cur_before = pred[ch][-1]
truth = [round(v * 200) for v in samples[ch]]
n_pred = len(pred[ch])
nn = blk.tag_lo
if n_pred + nn > len(truth):
continue
# Decode this 30 NN block with hypothesis
pred_deltas = decode_30nn(blk.data)
# Compute truth deltas relative to cur_before
truth_deltas = []
prev = cur_before
for k in range(nn):
truth_deltas.append(truth[n_pred + k] - prev)
prev = truth[n_pred + k]
n_match = sum(1 for a, b in zip(pred_deltas, truth_deltas) if a == b)
tag = "✓" if pred_deltas == truth_deltas else " "
print(f" block @ {blk.offset:>5} (chan={ch}, NN={nn}):")
print(f" data: {blk.data.hex(' ')}")
print(f" truth: {truth_deltas}")
print(f" pred: {pred_deltas} {tag}{n_match}/{nn}")
if __name__ == "__main__":
main()
analysis/test_30nn_v2.py
+141
View File
@@ -0,0 +1,141 @@
"""Test 30 NN packing by running the real decoder up to each 30 NN block,
recording how many samples have been produced for each channel at that point,
then checking truth deltas immediately after."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import _parse_txt
from minimateplus.waveform_codec import walk_body, find_data_start
def s4(n):
return n if n < 8 else n - 16
def i8(b):
return b if b < 128 else b - 256
def s12(v):
return v if v < 0x800 else v - 0x1000
def unpack_12bit_be_contiguous(data):
out = []
val = int.from_bytes(data, "big")
n = len(data) * 8 // 12
for i in range(n):
d = (val >> (12 * (n - 1 - i))) & 0xFFF
out.append(s12(d))
return out
def unpack_12bit_per_triplet_be(data):
out = []
for i in range(0, len(data), 3):
if i + 2 >= len(data):
break
b0, b1, b2 = data[i], data[i + 1], data[i + 2]
d0 = (b0 << 4) | (b1 >> 4)
d1 = ((b1 & 0x0F) << 8) | b2
out.append(s12(d0))
out.append(s12(d1))
return out
def simulate_up_to(blocks, target_block_idx, t_preamble):
"""Run the decoder up to block_idx; return per-channel sample lists."""
out = {"Tran": [], "Vert": [], "Long": [], "MicL": []}
out["Tran"].extend(t_preamble)
cur = {"Tran": t_preamble[-1], "Vert": None, "Long": None, "MicL": None}
rotation = ["Vert", "Long", "MicL", "Tran"]
seg_idx = [j for j, b in enumerate(blocks) if b.tag_hi == 0x40]
# Determine which channel we're CURRENTLY decoding into
current_channel = "Tran"
seg_counter = -1 # incremented at each 40 02
for j in range(target_block_idx):
blk = blocks[j]
if blk.tag_hi == 0x40:
# Switch: extend prev channel, set up new channel
seg_counter += 1
prev = "Tran" if seg_counter == 0 else rotation[(seg_counter - 1) % 4]
new_ch = rotation[seg_counter % 4]
if cur[prev] is not None:
d0 = int.from_bytes(blk.data[0:2], "big", signed=True)
d1 = int.from_bytes(blk.data[2:4], "big", signed=True)
cur[prev] += d0; out[prev].append(cur[prev])
cur[prev] += d1; out[prev].append(cur[prev])
c0 = int.from_bytes(blk.data[14:16], "big", signed=True)
c1 = int.from_bytes(blk.data[16:18], "big", signed=True)
out[new_ch].extend([c0, c1])
cur[new_ch] = c1
current_channel = new_ch
elif blk.tag_hi == 0x10:
for byte in blk.data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
cur[current_channel] += s4(nib)
out[current_channel].append(cur[current_channel])
elif blk.tag_hi == 0x20:
for byte in blk.data:
cur[current_channel] += i8(byte)
out[current_channel].append(cur[current_channel])
elif blk.tag_hi == 0x00:
for _ in range(blk.tag_lo):
out[current_channel].append(cur[current_channel])
elif blk.tag_hi == 0x30:
# Skip for now — we want to know what comes next
pass
return out, current_channel
def main():
for stem in ("M529LL1A.SP0", "M529LL1L.JQ0", "M529LL1L.V70",
"M529LL1A.SS0", "M529LL1A.SV0"):
path = f"tests/fixtures/5-11-26/{stem}"
with open(path, "rb") as f:
body = f.read()[43:-26]
_, samples = _parse_txt(path + ".TXT")
blocks = walk_body(body, find_data_start(body))
t0 = int.from_bytes(body[3:5], "big", signed=True)
t1 = int.from_bytes(body[5:7], "big", signed=True)
# Find all 30 NN blocks in data section
thirty_blocks = [(j, b) for j, b in enumerate(blocks) if b.tag_hi == 0x30]
if not thirty_blocks:
continue
print(f"\n=== {stem} ===")
for j, blk in thirty_blocks:
pred, ch = simulate_up_to(blocks, j, [t0, t1])
n_pred = len(pred[ch])
# The 30 NN block carries NN deltas for channel `ch` starting at sample n_pred
truth = [round(v * 200) for v in samples[ch]]
if n_pred >= len(truth):
continue
# Truth deltas: truth[n_pred] - cur, truth[n_pred+1] - truth[n_pred], ...
cur_val = pred[ch][-1]
nn = blk.tag_lo
truth_deltas = []
prev = cur_val
for k in range(min(nn, len(truth) - n_pred)):
truth_deltas.append(truth[n_pred + k] - prev)
prev = truth[n_pred + k]
print(f" block @ {blk.offset:>5} (chan={ch}, after sample {n_pred-1}, "
f"NN={nn}, last_val={cur_val}):")
print(f" data: {blk.data.hex(' ')}")
print(f" truth: {truth_deltas}")
schemes = [
("12-bit BE contiguous", unpack_12bit_be_contiguous(blk.data)),
("12-bit per-triplet BE", unpack_12bit_per_triplet_be(blk.data)),
]
for name, pred_deltas in schemes:
n_match = sum(1 for a, b in zip(pred_deltas, truth_deltas) if a == b)
tag = "✓" if pred_deltas == truth_deltas else " "
print(f" {tag}{n_match}/{nn} {name}: {pred_deltas[:nn]}")
if __name__ == "__main__":
main()
analysis/test_rle.py
+86
View File
@@ -0,0 +1,86 @@
"""Test: 00 NN markers might be RLE for zero-deltas in current channel."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import _parse_txt
from minimateplus.waveform_codec import walk_body, find_data_start
def s4(n):
return n if n < 8 else n - 16
def i8(b):
return b if b < 128 else b - 256
def decode_with_rle(body):
"""Decode Tran assuming:
- preamble[3:5], [5:7] = T[0], T[1]
- All 10 NN / 20 NN blocks until segment_header (40 02) are Tran deltas
- 00 NN markers are RLE: NN/4 zero T deltas (or NN, or NN/2 — try them)
"""
if len(body) < 9 or body[0:3] != b"\x00\x02\x00":
return None, None, None
T0 = int.from_bytes(body[3:5], "big", signed=True)
T1 = int.from_bytes(body[5:7], "big", signed=True)
# Find first tag (might be 00 NN, 10 NN, or 20 NN)
i = 7
while i + 1 < len(body):
if body[i] in (0x00, 0x10, 0x20):
break
i += 1
start = i
blocks = walk_body(body, start)
results = {}
for rle_div in (4, 2, 1): # try different RLE interpretations
T = [T0, T1]
cur = T1
for blk in blocks:
if blk.tag_hi == 0x40:
break
if blk.tag_hi == 0x10:
for byte in blk.data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
cur += s4(nib)
T.append(cur)
elif blk.tag_hi == 0x20:
for byte in blk.data:
cur += i8(byte)
T.append(cur)
elif blk.tag_hi == 0x00:
# RLE of zero deltas
n_zeros = blk.tag_lo // rle_div
for _ in range(n_zeros):
T.append(cur)
# 30 NN: skip for now
results[rle_div] = T
return results, T0, T1
def main():
for stem in ("M529LL1L.V70", "M529LL1L.JQ0", "M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
path = f"tests/fixtures/5-11-26/{stem}"
with open(path, "rb") as f:
body = f.read()[43:-26]
_, samples = _parse_txt(path + ".TXT")
truth_T = [round(v*200) for v in samples["Tran"]]
results, T0, T1 = decode_with_rle(body)
print(f"\n=== {stem} (T[0]={T0}, T[1]={T1}) ===")
for rle_div, T in results.items():
n = min(len(T), len(truth_T))
matches = sum(1 for i in range(n) if T[i] == truth_T[i])
# Find first divergence
div_at = -1
for i in range(n):
if T[i] != truth_T[i]:
div_at = i
break
print(f" rle_div={rle_div}: decoded {len(T)}, matches {matches}/{n}, first div at sample {div_at}")
if __name__ == "__main__":
main()
analysis/test_tran_continue.py
+71
View File
@@ -0,0 +1,71 @@
"""Test: does the second '20 NN' block in SS0 continue Tran samples?"""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import _parse_txt
from minimateplus.waveform_codec import walk_body, find_data_start
def s4(n):
return n if n < 8 else n - 16
def i8(b):
return b if b < 128 else b - 256
def main():
stem = "M529LL1A.SS0"
path = f"tests/fixtures/5-11-26/{stem}"
with open(path, "rb") as f:
body = f.read()[43:-26]
_, samples = _parse_txt(path + ".TXT")
truth_T_16 = [round(v * 200) for v in samples["Tran"]]
# Preamble
T0 = int.from_bytes(body[3:5], "big", signed=True)
T1 = int.from_bytes(body[5:7], "big", signed=True)
# Walk blocks
start = find_data_start(body)
blocks = walk_body(body, start)
print(f"=== {stem} === T[0]={T0} T[1]={T1}")
# Hypothesis: Tran continues through ALL 10 NN and 20 NN blocks
# in order, until the next 40 02 segment header (which resets).
T = [T0, T1]
cur = T1
decoded_count = 2 # T[0], T[1] from preamble
for bi, blk in enumerate(blocks):
if blk.tag_hi == 0x10:
for byte in blk.data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
cur += s4(nib)
T.append(cur)
decoded_count += 1
elif blk.tag_hi == 0x20:
for byte in blk.data:
cur += i8(byte)
T.append(cur)
decoded_count += 1
elif blk.tag_hi == 0x40:
# Segment header — stop here for this test
break
# 00 and 30 NN don't contribute to Tran (in this hypothesis)
# Compare to truth
print(f" Decoded {len(T)} T samples up to first 40 02")
matches = sum(1 for i in range(min(len(T), len(truth_T_16))) if T[i] == truth_T_16[i])
print(f" Matches in first {min(len(T), len(truth_T_16))}: {matches}")
# Print first divergence
for i in range(min(len(T), len(truth_T_16))):
if T[i] != truth_T_16[i]:
print(f" First divergence: sample {i}: pred={T[i]}, truth={truth_T_16[i]}")
# Show context
print(f" pred [{i-3}:{i+5}]: {T[max(0,i-3):i+5]}")
print(f" truth [{i-3}:{i+5}]: {truth_T_16[max(0,i-3):i+5]}")
break
if __name__ == "__main__":
main()
analysis/try_layouts.py
+67
View File
@@ -0,0 +1,67 @@
"""Try various nibble-level channel interleavings to find which one matches truth."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def s4(n):
return n if n < 8 else n - 16
def run_decoder(body, layout, skip, n_channels=4):
"""layout: function nibble_index -> channel_index. Returns list-of-lists per channel."""
out = [[] for _ in range(n_channels)]
cur = [0] * n_channels
nibbles = []
for byte in body[skip:]:
nibbles.append((byte >> 4) & 0xF)
nibbles.append(byte & 0xF)
for i, n in enumerate(nibbles):
ch = layout(i)
cur[ch] += s4(n)
out[ch].append(cur[ch])
return out
def cmp(pred, truth, n=24):
n = min(n, len(pred), len(truth))
return [(pred[i], truth[i]) for i in range(n)]
def main():
b = load_bundle("event-c")
truth_T = [round(v * 200) for v in b.samples["Tran"]]
truth_V = [round(v * 200) for v in b.samples["Vert"]]
truth_L = [round(v * 200) for v in b.samples["Long"]]
print(f"T truth[0:10]: {truth_T[:10]}")
print(f"V truth[0:10]: {truth_V[:10]}")
print(f"L truth[0:10]: {truth_L[:10]}")
# Try several nibble->channel layouts (4 channels)
layouts = {
"interleaved TVLM (0,1,2,3,0,1,2,3,...)": lambda i: i % 4,
"interleaved VLMT": lambda i: (i + 3) % 4,
"interleaved LMTV": lambda i: (i + 2) % 4,
"interleaved MTVL": lambda i: (i + 1) % 4,
"byte-based TV LM TV LM (high T low V byte0; high L low M byte1)": lambda i: i % 4,
# "chunks of 8 nibbles per channel": each channel gets 8 nibbles in a row
"chunks-8 TVLM": lambda i: (i // 8) % 4,
"chunks-16 TVLM": lambda i: (i // 16) % 4,
# planar (full channel sequential)
"planar T(0..N) V(N..2N) L(2N..3N) M(3N..4N)": None, # special
}
for label, layout_fn in layouts.items():
if layout_fn is None:
continue
for skip in (0, 4, 7, 8, 9, 11, 14):
out = run_decoder(b.body, layout_fn, skip)
# Check first 8 cumulative on each channel
print(f" skip={skip:2} {label}")
print(f" T_cum[0:10]: {out[0][:10]}")
print(f" V_cum[0:10]: {out[1][:10]}")
print(f" L_cum[0:10]: {out[2][:10]}")
if __name__ == "__main__":
main()
analysis/try_nibbles.py
+73
View File
@@ -0,0 +1,73 @@
"""Try decoding body as 4-bit signed nibble deltas, 4-channel round-robin."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
CHANNELS = ("Tran", "Vert", "Long", "MicL")
def s4(n):
"""Sign-extend a 4-bit unsigned to int (0..7 → 0..7, 8..F → -8..-1)."""
return n if n < 8 else n - 16
def decode_nibbles(body: bytes, skip_bytes: int = 7, n_channels: int = 4):
"""Read body as 2 nibbles per byte; accumulate as deltas for n_channels round-robin."""
out = [[] for _ in range(n_channels)]
cur = [0] * n_channels
ch = 0
nibbles = []
for byte in body[skip_bytes:]:
nibbles.append((byte >> 4) & 0xF)
nibbles.append(byte & 0xF)
for n in nibbles:
cur[ch] += s4(n)
out[ch].append(cur[ch])
ch = (ch + 1) % n_channels
return out
def cmp_to_truth(pred, truth, scale=16):
"""Compare predicted ints (in 16-count units) to truth (in 16-count units = txt * 200).
Return (max_abs_err, mean_abs_err, n_compared).
"""
n = min(len(pred), len(truth))
errs = []
for i in range(n):
p = pred[i]
t = truth[i]
errs.append(abs(p - t))
if not errs:
return None
return (max(errs), sum(errs) / len(errs), n)
def main():
for name in ("event-a", "event-c"):
b = load_bundle(name)
# Convert TXT samples (in/s) to 16-count units (multiply by 200, since 0.005 in/s = 1)
# WAIT: 0.005 in/s = 16 ADC counts. 1 count = 0.000305 in/s.
# So in 1-count units: count = txt * (1/0.0003052) ≈ txt * 3276.7
# But TXT only has 0.005 resolution so equivalent to 16-count units = txt * 200.
truth_in_16 = {ch: [round(v * 200) for v in b.samples[ch]] for ch in CHANNELS[:3]}
# MicL is in dB, skip for now
# Try decoder with skip_bytes = 7
decoded = decode_nibbles(b.body, skip_bytes=7, n_channels=4)
print(f"\n=== {name} ===")
print(f" body={len(b.body)}, nibbles={2*(len(b.body)-7)}, samples_per_ch={len(decoded[0])}")
print(f" truth samples per ch: {len(truth_in_16['Tran'])}")
# Print first 24 of each
for i, chan in enumerate(CHANNELS):
pred_first = decoded[i][:24]
if chan in truth_in_16:
truth_first = truth_in_16[chan][:24]
print(f" {chan} pred: {pred_first}")
print(f" {chan} truth: {truth_first}")
else:
print(f" {chan} pred: {pred_first} (truth in dB, skipped)")
if __name__ == "__main__":
main()
analysis/verify_full_decode.py
+32
View File
@@ -0,0 +1,32 @@
"""Verify decode_waveform_v2 against BW ASCII truth for all fixtures."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import _parse_txt
from minimateplus.waveform_codec import decode_waveform_v2
def main():
for stem in ("M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0",
"M529LL1L.JQ0", "M529LL1L.V70"):
path = f"tests/fixtures/5-11-26/{stem}"
with open(path, "rb") as f:
body = f.read()[43:-26]
_, samples = _parse_txt(path + ".TXT")
decoded = decode_waveform_v2(body)
if decoded is None:
print(f"{stem}: decoder returned None")
continue
print(f"\n=== {stem} ===")
for ch in ("Tran", "Vert", "Long"):
truth = [round(v * 200) for v in samples[ch]]
pred = decoded[ch]
n = min(len(pred), len(truth))
matches = sum(1 for i in range(n) if pred[i] == truth[i])
div = next((i for i in range(n) if pred[i] != truth[i]), -1)
print(f" {ch}: decoded={len(pred):>5} truth={len(truth):>5} "
f"matches={matches:>5}/{n:<5} first div={div}")
if __name__ == "__main__":
main()
analysis/verify_quiet_bundle.py
+55
View File
@@ -0,0 +1,55 @@
"""Run decode_waveform_v2 against the 5-8-26 quiet bundle to test the
'quiet events should decode fully' hypothesis."""
import os, sys
sys.path.insert(0, ".")
from minimateplus.waveform_codec import decode_waveform_v2, walk_body, find_data_start
from analysis.load_bundle import _parse_txt
def main():
base = "tests/fixtures/decode-re-5-8-26"
for evt in sorted(os.listdir(base)):
folder = os.path.join(base, evt)
if not os.path.isdir(folder):
continue
# Find the binary (not .TXT)
bin_name = next(
(f for f in os.listdir(folder) if not f.endswith(".TXT")),
None,
)
if not bin_name:
continue
bin_path = os.path.join(folder, bin_name)
txt_path = bin_path + ".TXT"
if not os.path.exists(txt_path):
# Sometimes the TXT name differs slightly
for f in os.listdir(folder):
if f.endswith(".TXT"):
txt_path = os.path.join(folder, f)
break
with open(bin_path, "rb") as f:
body = f.read()[43:-26]
decoded = decode_waveform_v2(body)
_, samples = _parse_txt(txt_path)
# Count 30 NN blocks
blocks = walk_body(body, find_data_start(body))
n_30 = sum(1 for b in blocks if b.tag_hi == 0x30)
n_40 = sum(1 for b in blocks if b.tag_hi == 0x40)
print(f"\n=== {evt} === body={len(body)} segments={n_40} '30 NN' blocks={n_30}")
if decoded is None:
print(" decoder returned None")
continue
for ch in ("Tran", "Vert", "Long"):
truth = [round(v * 200) for v in samples[ch]]
pred = decoded[ch]
n = min(len(pred), len(truth))
matches = sum(1 for i in range(n) if pred[i] == truth[i])
div = next((i for i in range(n) if pred[i] != truth[i]), -1)
print(f" {ch}: decoded={len(pred):>5} truth={len(truth):>5} "
f"matches={matches:>5}/{n:<5} first div={div}")
if __name__ == "__main__":
main()
analysis/verify_tran.py
+71
View File
@@ -0,0 +1,71 @@
"""Verify: preamble[3:7] = Tran[0], Tran[1] as int16 BE in 16-count units.
And first 20/10 NN block = Tran deltas starting at sample 2.
"""
import os, sys
sys.path.insert(0, ".")
from analysis.load_bundle import _parse_txt
from minimateplus.waveform_codec import walk_body, find_data_start
def s4(n):
return n if n < 8 else n - 16
def i8(b):
return b if b < 128 else b - 256
def main():
for stem in ("M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
path = f"tests/fixtures/5-11-26/{stem}"
with open(path, "rb") as f:
raw = f.read()
body = raw[43:-26]
_, samples = _parse_txt(path + ".TXT")
truth_T_16 = [round(v * 200) for v in samples["Tran"]]
# Preamble parse
T0_pre = int.from_bytes(body[3:5], "big", signed=True)
T1_pre = int.from_bytes(body[5:7], "big", signed=True)
print(f"\n=== {stem} ===")
print(f" Preamble T[0]={T0_pre} (truth {truth_T_16[0]}) T[1]={T1_pre} (truth {truth_T_16[1]}) match={T0_pre==truth_T_16[0] and T1_pre==truth_T_16[1]}")
# First block
start = find_data_start(body)
blocks = walk_body(body, start)
if not blocks:
print(f" no blocks found")
continue
# Assume first block = Tran deltas from sample 2
first = blocks[0]
T = [T0_pre, T1_pre]
cur_T = T1_pre
if first.tag_hi == 0x10:
# Nibble pairs
for byte in first.data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
cur_T += s4(nib)
T.append(cur_T)
elif first.tag_hi == 0x20:
# int8 per byte
for byte in first.data:
cur_T += i8(byte)
T.append(cur_T)
# Compare against truth
n_check = min(len(T), len(truth_T_16))
match_count = sum(1 for i in range(n_check) if T[i] == truth_T_16[i])
print(f" First block type=0x{first.tag_hi:02x} NN=0x{first.tag_lo:02x} len={len(first.data)} → {len(T)} T samples decoded")
print(f" Tran predicted[0:10]: {T[:10]}")
print(f" Tran truth [0:10]: {truth_T_16[:10]}")
print(f" Matches in first {n_check}: {match_count} / {n_check}")
# Show where it diverges
for i in range(n_check):
if T[i] != truth_T_16[i]:
print(f" First divergence: sample {i}: pred={T[i]}, truth={truth_T_16[i]}")
break
if __name__ == "__main__":
main()
analysis/walk_5_11.py
+20
View File
@@ -0,0 +1,20 @@
"""Walk blocks of the new 5-11-26 events and look at what comes after Tran block."""
import sys
sys.path.insert(0, ".")
from minimateplus.waveform_codec import walk_body, find_data_start
def main():
for stem in ("M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0"):
with open(f"tests/fixtures/5-11-26/{stem}", "rb") as f:
raw = f.read()
body = raw[43:-26]
start = find_data_start(body)
blocks = walk_body(body, start)
print(f"\n=== {stem} === body={len(body)} start={start} blocks walked={len(blocks)}")
for i, b in enumerate(blocks[:20]):
print(f" block[{i:>2}] @ {b.offset:>5} tag={b.tag_hi:02x} NN=0x{b.tag_lo:02x}({b.tag_lo}) len={b.length} data[:24]={b.data[:24].hex(' ')}")
if __name__ == "__main__":
main()
analysis/walk_chunks.py
+44
View File
@@ -0,0 +1,44 @@
"""Walk the body assuming chunks delimited by 0x10 NN tags. Print each chunk's structure."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def walk(body: bytes, start_offset: int = 7, max_chunks: int = 30):
"""Find all positions where byte = 0x10 followed by a multiple-of-4 byte. Print chunks."""
chunks = []
i = start_offset
while i < len(body) - 1:
# Find next `10 NN` where NN is multiple of 4 (and not preceded by another 0x10 immediately, which would be data).
if body[i] == 0x10 and (body[i+1] % 4 == 0):
chunks.append(i)
i += 1
return chunks
def main():
for name in ("event-c", "event-d"):
b = load_bundle(name)
body = b.body
positions = []
i = 7 # skip 7-byte preamble
while i < len(body) - 1:
if body[i] == 0x10 and body[i+1] % 4 == 0 and body[i+1] > 0:
positions.append(i)
i += 2 # skip past tag
else:
i += 1
print(f"\n=== {name} === body={len(body)}, total `10 NN` (NN%4==0, NN>0) tags: {len(positions)}")
# Print first 20 chunks: show position, NN, gap to next tag
for k in range(min(30, len(positions))):
pos = positions[k]
NN = body[pos + 1]
next_pos = positions[k+1] if k+1 < len(positions) else len(body)
gap = next_pos - pos
data_bytes = body[pos+2 : next_pos]
print(f" chunk[{k:>3}] @ {pos:>5} NN=0x{NN:02x} ({NN:>3}, NN/2={NN//2}) gap={gap:>3} "
f"data={data_bytes[:24].hex(' ')}{'...' if len(data_bytes) > 24 else ''}")
if __name__ == "__main__":
main()
analysis/walk_v2.py
+50
View File
@@ -0,0 +1,50 @@
"""Deterministic chunk walker: each chunk = [10 NN][NN/2 bytes data][2 bytes trailer]."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def walk_chunks(body: bytes, start: int = 7):
"""Yield (offset, NN, data_bytes, trailer_bytes) tuples."""
i = start
while i + 1 < len(body):
if body[i] != 0x10:
break
NN = body[i + 1]
if NN == 0 or NN > 0x80 or NN % 4 != 0:
break
chunk_len = NN // 2 + 4
if i + chunk_len > len(body):
break
data = bytes(body[i + 2 : i + 2 + NN // 2])
trailer = bytes(body[i + 2 + NN // 2 : i + chunk_len])
yield (i, NN, data, trailer)
i += chunk_len
def main():
for name in ("event-c", "event-d", "event-a", "event-b"):
b = load_bundle(name)
body = b.body
chunks = list(walk_chunks(body))
print(f"\n=== {name} === body={len(body)} N_samples={len(b.samples['Tran'])}")
print(f" chunks parsed: {len(chunks)}")
if chunks:
last = chunks[-1]
end_of_walk = last[0] + last[1] // 2 + 4
print(f" walk ended at offset {end_of_walk} (= {len(body) - end_of_walk} bytes from end)")
# Stats
total_data_bytes = sum(len(c[2]) for c in chunks)
print(f" total data bytes: {total_data_bytes}, total nibbles: {2*total_data_bytes}")
if name in ("event-c", "event-d"):
ratio = (2 * total_data_bytes) / (len(b.samples['Tran']) * 4)
print(f" nibbles per (sample × channel): {ratio:.3f}")
# Sum of trailer second-byte
trailer_sums = [c[3][-1] if c[3] else None for c in chunks]
print(f" first 10 chunks: {[(c[0], c[1], c[3].hex()) for c in chunks[:10]]}")
# Print last 10 chunks (likely transition to trailer)
print(f" last 10 chunks: {[(c[0], c[1], c[3].hex()) for c in chunks[-10:]]}")
if __name__ == "__main__":
main()
analysis/walk_v3.py
+51
View File
@@ -0,0 +1,51 @@
"""Walk chunks; auto-detect preamble length by finding first 10 NN."""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def walk_chunks(body, start, max_NN=0x80):
chunks = []
i = start
while i + 1 < len(body):
if body[i] != 0x10:
break
NN = body[i + 1]
if NN == 0 or NN > max_NN or NN % 4 != 0:
break
chunk_len = NN // 2 + 4
if i + chunk_len > len(body):
break
data = bytes(body[i + 2 : i + 2 + NN // 2])
trailer = bytes(body[i + 2 + NN // 2 : i + chunk_len])
chunks.append((i, NN, data, trailer))
i += chunk_len
return chunks, i
def find_first_chunk_start(body):
"""Locate first byte that begins a `10 NN` chunk (NN ∈ multiples of 4, 4..0x7C)."""
for i in range(20):
if body[i] == 0x10 and body[i + 1] % 4 == 0 and 0 < body[i + 1] <= 0x7C:
return i
return -1
def main():
for name in ("event-c", "event-d", "event-a", "event-b"):
b = load_bundle(name)
body = b.body
start = find_first_chunk_start(body)
chunks, end = walk_chunks(body, start)
print(f"\n=== {name} === body={len(body)} N_samples={len(b.samples['Tran'])} start={start}")
print(f" chunks parsed: {len(chunks)}, walk ended at {end}")
if chunks:
print(f" first 5 chunks: {[(c[0], c[1], c[3].hex()) for c in chunks[:5]]}")
print(f" last 5 chunks: {[(c[0], c[1], c[3].hex()) for c in chunks[-5:]]}")
print(f" bytes around end of walk: {body[end-4:end+12].hex(' ')}")
else:
print(f" bytes at start: {body[start:start+16].hex(' ')}")
if __name__ == "__main__":
main()
analysis/walk_v4.py
+75
View File
@@ -0,0 +1,75 @@
"""
Walker v4: alternate [10 NN] data chunks and [00 NN] (or other) marker tags.
Hypothesis:
- [10 NN]: data block, length NN/2 + 2 bytes (2-byte tag + NN/2 bytes data)
- [00 NN]: 2-byte marker block (no data)
- [20/30/40 NN]: special blocks with type-dependent length
"""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
def walk(body, start):
i = start
blocks = []
while i + 1 < len(body):
t0 = body[i]
t1 = body[i + 1]
if t0 == 0x10 and t1 % 4 == 0 and 0 < t1 <= 0x80:
# data chunk: length NN/2 + 2
length = t1 // 2 + 2
blocks.append((i, "10", t1, bytes(body[i + 2 : i + length]), length))
i += length
elif t0 == 0x00 and t1 % 4 == 0:
# 2-byte marker
blocks.append((i, "00", t1, b"", 2))
i += 2
elif t0 == 0x20 and t1 % 4 == 0:
# type 2 — try length 2+t1/2 (similar to 10) OR fixed
length = t1 // 2 + 2
blocks.append((i, "20", t1, bytes(body[i + 2 : i + length]), length))
i += length
elif t0 == 0x30 and t1 % 4 == 0:
length = t1 // 2 + 2
blocks.append((i, "30", t1, bytes(body[i + 2 : i + length]), length))
i += length
elif t0 == 0x40 and t1 == 0x02:
# Special "footer transition" block — try fixed 22 bytes
length = 22
blocks.append((i, "40", t1, bytes(body[i + 2 : i + length]), length))
i += length
else:
# Unknown tag — stop
blocks.append((i, "??", t0, bytes(body[i:i+8]), 0))
break
return blocks, i
def main():
for name in ("event-c", "event-d", "event-a", "event-b"):
b = load_bundle(name)
body = b.body
# Auto-detect start
for s in range(15):
if body[s] == 0x10 and body[s+1] % 4 == 0 and 0 < body[s+1] <= 0x80:
start = s
break
else:
start = 7
blocks, end = walk(body, start)
# Categorize
from collections import Counter
types = Counter(b[1] for b in blocks)
print(f"\n=== {name} === body={len(body)} N={len(b.samples['Tran'])} start={start}")
print(f" total blocks: {len(blocks)}, walk ended at {end}/{len(body)}")
print(f" type counts: {dict(types)}")
# Print last 5 blocks
print(f" last 5 blocks: {[(bb[0], bb[1], bb[2]) for bb in blocks[-5:]]}")
if end < len(body):
print(f" bytes at end: {body[end:end+24].hex(' ')}")
if __name__ == "__main__":
main()
analysis/walk_v5.py
+83
View File
@@ -0,0 +1,83 @@
"""
Walker v5: flexible NN range and multiple block-type lengths.
Hypothesis:
- [10 NN]: 4-bit-delta data block, length = NN/2 + 2
- [20 NN]: 8-bit-literal data block, length = NN + 2
- [00 NN]: 2-byte marker (no payload)
- [30 NN]: trailer/summary block, length = NN*4
- [40 NN]: footer-marker block, fixed 22 bytes
"""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
from collections import Counter
def walk(body, start, max_blocks=10000):
i = start
blocks = []
while i + 1 < len(body) and len(blocks) < max_blocks:
t0 = body[i]
t1 = body[i + 1]
if t0 == 0x10 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
length = t1 // 2 + 2
if i + length > len(body):
break
data = bytes(body[i + 2 : i + length])
blocks.append((i, "10", t1, data, length))
i += length
elif t0 == 0x20 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
length = t1 + 2
if i + length > len(body):
break
data = bytes(body[i + 2 : i + length])
blocks.append((i, "20", t1, data, length))
i += length
elif t0 == 0x00 and t1 % 4 == 0:
# 2-byte marker
blocks.append((i, "00", t1, b"", 2))
i += 2
elif t0 == 0x30 and t1 % 4 == 0:
length = t1 * 4
if i + length > len(body):
break
data = bytes(body[i + 2 : i + length])
blocks.append((i, "30", t1, data, length))
i += length
elif t0 == 0x40 and t1 == 0x02:
length = 22
if i + length > len(body):
break
data = bytes(body[i + 2 : i + length])
blocks.append((i, "40", t1, data, length))
i += length
else:
blocks.append((i, "??", t0, bytes(body[i:i+8]), 0))
break
return blocks, i
def main():
for name in ("event-c", "event-d", "event-a", "event-b"):
b = load_bundle(name)
body = b.body
for s in range(15):
if body[s] == 0x10 and body[s+1] % 4 == 0 and 0 < body[s+1] <= 0xFC:
start = s; break
else:
start = 7
blocks, end = walk(body, start)
types = Counter(bb[1] for bb in blocks)
print(f"\n=== {name} === body={len(body)} N={len(b.samples['Tran'])} start={start}")
print(f" total blocks: {len(blocks)}, walk ended at {end}/{len(body)}")
print(f" type counts: {dict(types)}")
if blocks and blocks[-1][1] == "??":
print(f" stopped at byte: 0x{blocks[-1][2]:02x}, prev 5 blocks: {[(bb[0], bb[1], bb[2]) for bb in blocks[-6:-1]]}")
# Sum payload sizes by type
payload_sizes = {t: sum(len(bb[3]) for bb in blocks if bb[1] == t) for t in types}
print(f" payload bytes by type: {payload_sizes}")
if __name__ == "__main__":
main()
analysis/walk_v6.py
+68
View File
@@ -0,0 +1,68 @@
"""
Walker v6: handle 40 02 blocks correctly (length 20).
Block formats:
- [10 NN]: 4-bit nibble delta data, length = NN/2 + 2
- [20 NN]: int8 literal data, length = NN + 2
- [00 NN]: 2-byte marker
- [30 NN]: trailer/summary block, length = NN*4
- [40 02]: segment header, fixed length 20
"""
import sys
sys.path.insert(0, ".")
from analysis.load_bundle import load_bundle
from collections import Counter
def walk(body, start, max_blocks=10000):
i = start
blocks = []
while i + 1 < len(body) and len(blocks) < max_blocks:
t0 = body[i]
t1 = body[i + 1]
if t0 == 0x10 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
length = t1 // 2 + 2
elif t0 == 0x20 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
length = t1 + 2
elif t0 == 0x00 and t1 % 4 == 0:
length = 2
elif t0 == 0x30 and t1 % 4 == 0 and 0 < t1 <= 0x10:
length = t1 * 4
elif t0 == 0x40 and t1 == 0x02:
length = 20
else:
blocks.append((i, "??", t0, bytes(body[i:i+8]), 0))
break
if i + length > len(body):
break
data = bytes(body[i + 2 : i + length])
blocks.append((i, f"{t0:02x}", t1, data, length))
i += length
return blocks, i
def main():
for name in ("event-c", "event-d", "event-a", "event-b"):
b = load_bundle(name)
body = b.body
for s in range(15):
if body[s] == 0x10 and body[s+1] % 4 == 0 and 0 < body[s+1] <= 0xFC:
start = s; break
else:
start = 7
blocks, end = walk(body, start)
types = Counter(bb[1] for bb in blocks)
print(f"\n=== {name} === body={len(body)} N={len(b.samples['Tran'])} start={start}")
print(f" total blocks: {len(blocks)}, walk ended at {end}/{len(body)}")
print(f" type counts: {dict(types)}")
if blocks and blocks[-1][1] == "??":
print(f" stopped at byte: 0x{blocks[-1][2]:02x} at offset {blocks[-1][0]}")
print(f" prev 5 blocks: {[(bb[0], bb[1], bb[2]) for bb in blocks[-6:-1]]}")
print(f" bytes around stop: {body[end-4:end+24].hex(' ')}")
# Sum
payload_sizes = {t: sum(len(bb[3]) for bb in blocks if bb[1] == t) for t in types}
print(f" payload bytes by type: {payload_sizes}")
if __name__ == "__main__":
main()
bridges/ach_server.py
+1
View File
@@ -516,6 +516,7 @@ class AchSession:
serial=serial or self.peer,
session_id=None,
waveform_records=waveform_records,
device_family="series3",
)
_ml_ins, _ml_skip = self.db.insert_monitor_log(
new_monitor_entries, session_id=None
docs/histogram_codec_re_status.md
+155
View File
@@ -0,0 +1,155 @@
# Histogram body codec — FULLY DECODED (2026-05-20)
Clean working status doc for the MiniMate Plus histogram-mode event
body codec. Companion to `waveform_codec_re_status.md`. The deep
historical record (with retractions and dated analyses) lives in
`docs/instantel_protocol_reference.md §7.6.2`; the authoritative
implementation lives in `minimateplus/histogram_codec.py`.
## TL;DR
**The codec is fully decoded.** Every field of every block in the
in-repo histogram fixture corpus decodes byte-exact against BW's
ASCII export.
24 regression tests pass against ~3,500 blocks across 5 fixtures.
## Body format
```
body = [stream of 32-byte data blocks] + [small trailing remnant]
```
Each block represents one histogram interval. Block layout:
```
[0] 0x00 always-zero tag
[1] segment_id (uint8) 0x00..0x03 — 256 blocks per segment
[2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, …)
[4:6] 0x000a (uint16 LE) constant marker (= 10)
[6:8] T_peak_count uint16 LE Tran peak (count × 0.005 → in/s at Normal)
[8:10] T_halfperiod uint16 LE Tran half-period in samples
(freq_Hz = 512 / halfp; ≤ 5 means ">100 Hz")
[10:12] V_peak_count uint16 LE Vert peak
[12:14] V_halfperiod uint16 LE Vert freq half-period
[14:16] L_peak_count uint16 LE Long peak
[16:18] L_halfperiod uint16 LE Long freq half-period
[18:20] M_peak_count uint16 LE MicL peak count
(dB via waveform_codec.mic_count_to_db)
[20:22] M_halfperiod uint16 LE MicL freq half-period
[22:24] 0x00 0x00 constant
[24:28] 4-byte variable purpose unknown — possibly CRC,
timestamp delta, or psi(L) numeric;
not needed for waveform reconstruction
[28:32] 0x1e 0x0a 0x00 0x00 constant block-end signature
```
Reliable block-identification anchor:
```python
block[22:24] == b"\x00\x00" and block[28:32] == b"\x1e\x0a\x00\x00"
```
(The `1e 0a 00 00` constant tail is the most distinctive signature.)
## Per-channel encoding
| Channel | Peak encoding | Frequency encoding |
|---|---|---|
| Tran | count × 0.005 = in/s at Normal range | `freq_Hz = 512 / halfperiod` |
| Vert | same | same |
| Long | same | same |
| MicL | count → dB via `mic_count_to_db(count)` (same formula as waveform codec) | same |
**`>100 Hz` sentinel**: when halfperiod ≤ 5 (giving ≥100 Hz from the
512/halfp formula), BW displays `>100 Hz`. Codec's `half_period_to_hz`
returns `None` in this range.
## Verified facts (cross-checked against fixture corpus)
Example: N844L6Z8.ZR0H block 130 → all 8 decoded fields byte-exact:
```
binary samples [10, 6, 24, 4, 18, 5, 21, 5, 9]
TXT row [0.030, 21, 0.020, 28, 0.025, 24, 0.040, 0.000, 95.92, 57]
slot[0] = 10 marker
slot[1] = 6 × 0.005 = 0.030 in/s ✓ T_peak
slot[2] = 24 → 512/24 = 21.3 → 21 Hz ✓ T_freq
slot[3] = 4 × 0.005 = 0.020 in/s ✓ V_peak
slot[4] = 18 → 512/18 = 28.4 → 28 Hz ✓ V_freq
slot[5] = 5 × 0.005 = 0.025 in/s ✓ L_peak
slot[6] = 21 → 512/21 = 24.4 → 24 Hz ✓ L_freq
slot[7] = 5 → 81.94 + 20·log10(5) = 95.92 dB ✓ M_peak
slot[8] = 9 → 512/9 = 56.9 → 57 Hz ✓ M_freq
```
## Verified test coverage
`tests/test_histogram_codec.py` (24 tests):
- Block walking: yields one record per `.TXT` interval ± 1 (off-by-one
at the tail when recording was stopped mid-write). Segment-ID
groups of 256 blocks confirmed.
- Geo peaks: every block of N844L20G, N844L6Z8, N844L6XE, N844L23B
matches `.TXT` within the 0.0005 in/s quantization step.
- Geo freqs: every block of N844L6Z8 and N844L6XE matches `.TXT`
within 1 Hz (BW display rounds). `>100 Hz` sentinel handled correctly.
- Mic dB: every block of N844L6XE, N844L23B, N844L6Z8 matches `.TXT`
within 0.1 dB (BW display precision).
- Mic freq: matches `.TXT` within 1 Hz across active blocks.
## What's NOT yet decoded
- **4-byte variable metadata field (bytes 24:28)**. Not needed for
waveform reconstruction. Speculation: per-block CRC, sub-second
timestamp offset, or a Mic psi(L) count not in the 9 samples.
Punt until something needs it.
- **Geo PVS (TXT col 7, e.g. "0.040 in/s")**. Not stored in the
block; can be approximated as `sqrt(T_peak² + V_peak² + L_peak²)`
but BW's value sometimes differs slightly (probably computed from
waveform-instant samples, not from per-channel peaks). Punt — the
`.h5` consumers don't need PVS as a sample channel.
- **Mic psi(L) value (TXT col 8)**. TXT shows it as a small psi value
derived from the dB measurement. Not in the 9 samples. Could be
derived from `M_peak_count` via the inverse of the dB formula plus
a psi calibration constant. Defer.
## Output shape
`decode_histogram_body` returns the standard 4-channel dict that
mirrors `waveform_codec.decode_waveform_v2`'s output:
```python
{
"Tran": [peak_count_per_interval, ...], # 16-count units (LSB = 0.005 in/s)
"Vert": [..., ...],
"Long": [..., ...],
"MicL": [..., ...], # raw ADC counts
}
```
Run through `waveform_codec.decoded_to_adc_counts` to get 1-count ADC
units (geo ×16, mic passthrough) for the standard `.h5` writer.
For the full per-interval record with frequencies + metadata, use
`decode_histogram_body_full()`.
## Where it's wired
- `minimateplus/event_file_io.py:read_blastware_file()` — first tries
the waveform codec, falls back to the histogram codec when the
waveform preamble isn't present. Same output shape, same
downstream pipeline.
- `scripts/backfill_sidecars.py` — the `has_samples` short-circuit
added during the histogram-codec-pending era still serves as a
defensive guard against truly undecodable files, but no longer
fires for valid histograms.
## Companion reference
- `docs/waveform_codec_re_status.md` — sibling status doc for the
much-more-complex waveform-mode codec.
- `docs/instantel_protocol_reference.md §7.6.2` — historical
protocol-reference entry. Structural framing matches what we
found; per-sample semantics were less documented than the `✅
CONFIRMED` badge suggested. This doc supersedes §7.6.2 where they
conflict on confidence level.
docs/idf_protocol_reference.md
+284
View File
@@ -0,0 +1,284 @@
# IDF Protocol Reference — Thor / Micromate Series IV
Starting-point reference for reverse-engineering Instantel's Micromate
Series IV event-file format. Sibling to
[instantel_protocol_reference.md](instantel_protocol_reference.md) (the
Series III "Rosetta Stone") — this doc holds what we know so far and
the open questions still to crack.
**Status (2026-05-20):** ASCII text sidecar fully decoded (1,014
sample files round-trip). Binary `.IDFH` / `.IDFW` codec
**not yet implemented** — binaries are stored opaquely by
`WaveformStore.save_imported_idf`, with metadata sourced from the
paired `.txt` sidecar.
---
## File model
### Filename convention
```
<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>
```
- **SERIAL** — literal device serial, two-letter prefix + numeric
suffix. Examples seen: `UM11719`, `UM13981`, `UM20147`, `BE9439`.
Unlike Series III BW filenames (`M529LK44.AB0`, base-36 stem),
Series IV filenames carry the serial in plain text.
- **YYYYMMDDHHMMSS** — 14-char ASCII timestamp in **device local
time** (no timezone marker).
- **KIND** — `IDFH` for histograms, `IDFW` for waveforms.
The `.IDFH.txt` / `.IDFW.txt` ASCII sidecar lives in a `TXT/`
**subfolder** of the unit's directory, not alongside the binary.
This pairing convention is encoded in
`event_forwarder.idf_report_path()`.
### Directory layout
```
C:\THORDATA\
└── <Project>\
└── <UM####>\ ← unit serial dir
├── UM12345_20260520100000.MLG ← monitor log (not events)
├── UM12345_20260520100000.IDFH ← histogram event (binary)
├── UM12345_20260520100000.IDFW ← waveform event (binary)
├── UM12345_20260520100000.IDFW.CDB ← cache-DB variant (skip)
├── TXT\
│ ├── UM12345_20260520100000.IDFH.txt ← histogram ASCII sidecar
│ └── UM12345_20260520100000.IDFW.txt ← waveform ASCII sidecar
├── CSV\, HTML\, PDF\, XML\ ← operator-facing derived exports
└── ...
```
The `.IDFW.CDB` files share the binary's basename but appear to be a
separate cache/database variant. Their first 8 bytes match the
**old**-firmware Thor signature (see below) regardless of which
signature the paired `.IDFW` uses. Purpose unknown; sizes vary
wildly (observed 123 B → 40,491 B). Thor-watcher's forwarder
deliberately skips them.
### Sample corpus
The `thor-watcher/example-data/THORDATA_example/` tree carries
**1,014 paired .IDFW / .IDFH + .txt files** spanning 2020–2023
across nine units (UM11719, UM13981, UM20147, …, plus BE9439 from
2020). This is the reverse-engineering ground truth.
---
## ASCII sidecar (`.IDFW.txt` / `.IDFH.txt`) — fully decoded
Shape: plain text, one `"Key : Value"` line per metadata field,
followed for waveforms by a tab-separated sample table headed by
the literal line `Waveform Data Channels`. Parsed by
[`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py).
See [`micromate/models.py`](../micromate/models.py) for the typed
`IdfReport` shape.
### Notable conventions
- **Units are native to Thor** — geophone in **in/s**, microphone in
**dB(L)** (not psi like Series III BW reports), frequency in Hz,
acceleration in g, displacement in in.
- **Below-threshold readings** appear as the literal string
`<0.005 in/s` (155 occurrences in the sample corpus) — the parser
strips the `<` and treats the numeric remainder as the value.
- **Out-of-range / not-measured** values appear as `N/A` — parser
drops the field rather than letting the string leak into a numeric
column.
- **Firmware string** observed: `Micromate ISEE 11.0AK`.
- **TitleString1..4** are operator-defined free-text slots; Thor's
default labels map them to Location / Client / Company / Notes,
which the parser surfaces as `project` / `client` / `operator` /
`notes`.
- **Histogram sidecars** use `HistogramStartDate` / `HistogramStartTime`
in place of waveform's `EventDate` / `EventTime`. Parser falls
through to either.
- **Histogram tabular block** lacks the `Waveform Data Channels`
marker; instead it's a multi-line column header followed by
per-interval rows (`<date> <time> <tran-ppv> <freq> ...`). Parser
silently ignores lines after the metadata block since they lack a
colon-separated `key : value` shape (the timestamps DO contain
colons but produce garbage keys that don't collide with any
recognised field).
---
## Binary header signatures (observed)
Hex dump of the first 32 bytes across 1,014 sample files reveals
**two distinct file signatures**, both anchored by the literal
ASCII string `"\x00Instantel\x00"` at offset 6–16:
### Signature A — newer firmware (1,012 files, 99.8% of corpus)
```
00000000: 0012 0100 0000 496e 7374 616e 7465 6c00 ......Instantel.
00000010: 0000 a695 002e b500 4f70 6572 6174 6f72 ........Operator
^^^^^^^^^^^^^^^^
operator/title string starts at 0x18
```
Header bytes 0–5: `00 12 01 00 00 00`. Followed immediately by the
8-byte ASCII tag, then 6 unknown bytes, then ASCII operator-supplied
strings (Operator name, etc.) and on through the project / client /
title strings. No `STRT` record observed in this layout.
### Signature B — older firmware (2 files: BE9439 from 2020)
```
00000000: 1000 0180 0000 496e 7374 616e 7465 6c00 ......Instantel.
00000010: 072c 0012 0300 5354 5254 fffe 0111 2340 .,....STRT....#@
^^^^^^^^^ ^^^^^^^^^
STRT magic 4-byte end_key
00000020: 0111 0000 2e5f 00ac 4600 0000 0200 0000 ....._..F.......
^^^^^^^^^ ^^^
4-byte start_key 0x46 (BW WAVEHDR record-type marker)
```
Header bytes 0–5: `10 00 01 80 00 00`. The structure after the
`Instantel` magic is **byte-for-byte identical to a BW SUB 5A
probe-response STRT record** as documented in
[instantel_protocol_reference.md → "SUB 5A — STRT record encodes
end_offset"](instantel_protocol_reference.md). Specifically:
| Offset | Bytes | Meaning (per BW reference) |
|--------|---------------------|--------------------------------------|
| 0x14 | `53 54 52 54` | `STRT` magic |
| 0x18 | `ff fe` | STRT sentinel |
| 0x1A | `01 11 23 40` | `end_key` (4 bytes) |
| 0x1E | `01 11 00 00` | `start_key` (4 bytes) |
| 0x26 | `46` | `0x46` waveform-record type marker |
**Hypothesis:** Older Micromate firmware writes a wrapped BW-format
event into the `.IDFW` file — essentially the same on-disk shape as
a Series III device, with the new filename convention applied at
export time. Newer firmware (signature A) abandoned the
BW-compatible layout for an Instantel-specific format.
If that hypothesis holds, the 2 signature-B files can already be
parsed via `minimateplus/event_file_io.read_blastware_file()` — worth
testing. The 1,012 signature-A files are the real reverse-engineering
target.
### `.IDFW.CDB` cache files
Always carry signature B (`10 00 01 80 ...`), even when the paired
`.IDFW` carries signature A. Plausible explanation: the CDB is an
internal Thor cache-database export that retains the legacy BW-style
record layout regardless of the user-facing `.IDFW` format version.
Not currently consumed by the forwarder.
---
## File-size patterns (Signature A, the main target)
Survey of 1,012 signature-A files:
| Event type | Typical size | Source of variance |
|--------------|-------------------|----------------------------------------------|
| `.IDFW` 2-sec | 9,200 – 10,500 B | Operator-supplied strings (TitleString1..4) of varying length |
| `.IDFH` | 2,944 – 4,076 B | Histogram interval count (record duration / interval) |
**Naive arithmetic for 2-sec waveform:**
- 4 channels × 2 sec × 1024 sps = 8,192 samples
- At 2 bytes/sample (int16) = 16,384 sample bytes → file would be > 16 KB
- Observed: ~9–10 KB
- → samples are likely **1 byte each** (int8 quantised), **or** stored
with bit-packing / delta encoding, **or** only one channel's
full-rate samples are stored with the others reconstructed
arithmetically. Verifying this is the **first RE milestone**.
Project-string–length variance (~1 KB across the corpus) is consistent
with the file carrying a single copy of each TitleString1..4 plus
operator + setup-name as null-padded ASCII regions.
---
## Open questions
The reverse-engineering targets, roughly in dependency order:
1. **Sample encoding (signature A)** — int8? int16 LE/BE? Bit-packed?
Delta-coded? Per-channel interleaved or sequential blocks?
2. **Header field layout (signature A)** — where do sample_rate,
record_time, channel count, and per-channel peaks live in the
binary? The ASCII sidecar gives the device-authoritative values,
so binary fields can be confirmed by diff.
3. **Operator-string offsets** — `Operator` at 0x18 is the first
visible string in signature-A files; the rest (project, client,
notes, setup) follow. Need to map exact offsets and null-padding
conventions.
4. **Signature-B → BW codec compatibility** — does
`minimateplus/event_file_io.read_blastware_file()` actually parse
the 2 BE9439 signature-B files as-is? If yes, the OLD-format
ingest is free.
5. **`.IDFW.CDB` purpose** — is it an internal Thor cache, a
ring-buffer dump, or something else? Worth a single small effort
to characterise so we know what we're skipping.
6. **Footer / checksum** — every BW event file has a footer; does
IDF? Where does the per-channel sample block end?
---
## Reverse-engineering playbook (when we start)
The Series III BW codec took ~2 months of MITM wire captures
because we didn't have ground-truth metadata. Thor's situation is
**substantially better**:
- **Ground truth is on disk.** Every binary in `example-data/`
has a paired `.IDFW.txt` carrying the full decoded sample table
(`Waveform Data Channels` block — see any sample file in
`thor-watcher/example-data/.../TXT/`). Aligning binary bytes
to the table's float-per-row values gives an immediate per-byte
hypothesis test.
- **Cross-event diffing.** 1,012 signature-A samples from 9 units
spanning 4 years means any field that varies between events is
immediately localisable. Fields that are constant across all
files (firmware ID, channel labels, format-version word) are also
immediately localisable by complementary search.
- **No protocol surface.** Files at rest, not a wire dialect. No
DLE stuffing, no inner-frame parsing, no probe/data two-step.
Suggested first session (2-4 hours): hand-decode `UM11719_20231219162723.IDFW`
(10,290 bytes) against its `TXT/UM11719_20231219162723.IDFW.txt`
sample table (the 2-sec waveform at 1024 sps × 4 channels = 8,192
sample rows). Find the first per-channel sample value (`0.0003` in
the Tran column at t=0) in the binary. Confirms sample encoding.
Everything else flows from there.
---
## Code seams ready to receive the codec
When the codec lands, it goes into
[`micromate/idf_file.py`](../micromate/idf_file.py) (currently a
stub raising `NotImplementedError`). Public API:
```python
from micromate import IdfEvent
from micromate.idf_file import read_idf_file
event: IdfEvent = read_idf_file(Path("UM11719_20231219163444.IDFW"))
# event.peaks.transverse_ips, event.timestamp, event.raw_samples, ...
```
The ingest pipeline (`WaveformStore.save_imported_idf`) currently
builds the `IdfEvent` from the `.txt` parser only. Once
`read_idf_file()` works, the binary becomes authoritative; the
`.txt` parser drops to fast-path metadata cross-check. Operators
who don't enable Thor's TXT exporter still get fully populated
events.
---
## See also
- [instantel_protocol_reference.md](instantel_protocol_reference.md) — Series III BW protocol reference (the Rosetta Stone). STRT record format, DLE framing, BW filename encoding.
- [`micromate/idf_ascii_report.py`](../micromate/idf_ascii_report.py) — `.txt` sidecar parser.
- [`micromate/models.py`](../micromate/models.py) — `IdfEvent`, `IdfReport` typed dataclasses.
- [`micromate/idf_file.py`](../micromate/idf_file.py) — placeholder for the binary codec.
- [`thor-watcher/example-data/THORDATA_example/`](../../thor-watcher/example-data/) — 1,014 paired binary + .txt files for codec validation.
docs/instantel_protocol_reference.md
+752 -297
View File
File diff suppressed because it is too large Load Diff
docs/runbooks/wedged_unit_recovery.md
+255
View File
@@ -0,0 +1,255 @@
# Runbook — Recovering a wedged unit stuck in a call-home loop
**Original incident:** BE9558H at `166.246.130.1:9034`, recovered 2026-05-17.
A field unit with a stuck-triggered geophone (or any hardware fault causing
constant event triggering) will record events back-to-back, and if Auto Call
Home is set to "After Event Recorded" the device will dial the office BW
ACH server in a tight loop. Combined with a Sierra Wireless modem in
bidirectional serial-TCP mode, this makes the unit effectively unreachable
from SFM — every TCP connection we open gets killed when the modem flips
from server-mode to client-mode to honor the device's next AT dial command.
This runbook describes how to break the loop and recover control.
---
## Symptoms
- Terra-View / SFM `/device/info` either hangs or fails on `count_events()`.
- `/device/monitor/status` and `/device/rescue` return 502 (protocol timeout
waiting for POLL response) or 503 (TCP connect refused).
- ACEmanager serial log shows repeating
`Connect to IP: <BW_IP> Port: <BW_PORT>` → `Shutdown TCP socket` cycles
every 30-60 seconds.
- Spam-mode endpoints (`/device/stop_monitoring_spam`) report many
`sent_ok` but the device's monitoring state never changes.
- `slow_drip` reports `[Errno 32] Broken pipe` after sending the preamble
but before completing the drip loop.
If you see *all* of these, the unit is in this exact failure mode.
---
## Quick reference — how to recover
You need **ACEmanager access** to the unit's modem.
### Step 1: stop the modem's mode-flipping
In ACEmanager → **Serial → Port Configuration**:
| Field | Set to |
|---|---|
| **Destination Address** | clear (blank) |
| **Destination Port** | `0` |
Click **Apply**. This removes the modem's auto-dial-out target. The device's
AT dial commands now error back at the modem instead of triggering a
mode-flip, so the modem stays in TCP-server mode permanently and our inbound
TCP sessions stay alive.
*(Optional belt-and-suspenders: also add the BW server's port to
**Security → Port Filtering - Outbound** as a blocked port, with
Outbound Port Filtering Mode = Blocked Ports.)*
### Step 2: stop monitoring on the device (slow drip)
From the SFM host:
```bash
/home/serversdown/seismo-relay/scripts/slow_drip.sh <DEVICE_IP> <PORT>
```
Defaults are 120s duration with a drip every 3s. Watch the response:
- `duration_s ≈ 120` and `drips_sent ≈ 40` → session held the full duration ✓
- `bytes_received > 0` → device is responding ✓ (this is the success signal)
If `duration_s` is small or `send_error: "Broken pipe"`, Step 1 didn't take
hold — re-check ACEmanager, may need to reboot the modem after Apply.
### Step 3: confirm monitoring stopped
```bash
curl 'http://localhost:8200/device/monitor/status?host=<DEVICE_IP>&tcp_port=<PORT>&force=true'
# expect: {"is_monitoring": false, ...}
```
### Step 4: disable ACH at the device level + erase corrupted events
Either fire the rescue endpoint:
```bash
/home/serversdown/seismo-relay/scripts/rescue_device.sh <DEVICE_IP> <PORT>
```
Or do the two steps manually:
```bash
# Disable ACH in the device's compliance config
curl -X POST 'http://localhost:8200/device/call_home?host=<DEVICE_IP>&tcp_port=<PORT>' \
-H 'Content-Type: application/json' \
-d '{"auto_call_home_enabled": false}'
# Erase corrupted event chain
curl -X POST 'http://localhost:8200/device/events/erase?host=<DEVICE_IP>&tcp_port=<PORT>'
```
You can also do this via the SFM standalone UI → **Call Home** tab → set
`Enable Auto Call Home` to `Disabled` → **Write to Device**.
### Step 5: restore modem config (housekeeping)
Once the device-side ACH is disabled, restore the modem's Destination
Address and Port to the original values (e.g. `50.197.32.92` / `12345`) in
ACEmanager. The modem will resume normal bidirectional behavior, but the
unit won't issue any dial commands until ACH is explicitly re-enabled on
the device.
### Step 6: do NOT re-enable ACH on this unit until the underlying hardware
fault is repaired. If you do, the call-home loop starts again immediately
and you'll be running this runbook a second time.
---
## Why this works — the failure mode explained
The Sierra Wireless RV50/RV55 serial port operates in one of two TCP modes
at any moment:
- **Server mode** — listens on `Device Port` (e.g. 9034), bridges inbound
TCP to the device's serial port. This is what we need to interact with
the device.
- **Client mode** — when the device sends an AT dial command on its serial
TX line, the modem opens an outbound TCP to `Destination Address:Port`
and bridges that to serial.
A serial port in this configuration is **bidirectional**: the modem flips
between server and client modes on demand. When the device's firmware is
healthy and only dials occasionally, this works fine.
When the unit is constantly triggering events and ACH is set to "After
Event Recorded", the device sends an AT dial command every few seconds.
Each one causes the modem to:
1. Drop any active inbound TCP session
2. Flip to client mode
3. Attempt outbound TCP to `Destination Address:Port`
4. Hang for up to a minute waiting for it to succeed/fail
5. Drop back to server mode
**During the entire hang, no inbound TCP can establish.** Even between
hangs, the modem closes any existing inbound session before flipping. So
any tool that needs more than a few seconds of held TCP (e.g. POLL +
config read + write) gets repeatedly kicked off.
Clearing `Destination Address` removes step 3-4 from the cycle: the modem
has nowhere to dial, so it doesn't flip modes when it receives an AT dial
command. The serial port effectively becomes server-only, and inbound TCP
sessions can stay open as long as needed.
**This is a modem-layer issue, not a device firmware issue.** The device
is alive and responsive the whole time — confirmed in the BE9558H
recovery by 990 bytes of S3 responses received over a 120s slow-drip
session once the modem was no longer mode-flipping.
---
## Why simpler approaches don't work
| Approach | Why it fails |
|---|---|
| Standard `/device/info` | Triggers `count_events()` 1E/1F walk, takes 90s+ and hits corrupted event chain in this scenario |
| `/device/rescue` race loop | Gets 502 (protocol timeout) because the modem closes the TCP before the POLL handshake can complete |
| `/device/stop_monitoring_blind` (single frame) | Even if the bytes leave the wire, the device's protocol parser ignores write commands without a preceding POLL handshake (early-version bug, now fixed by including POLL preamble in blind sends) |
| `/device/stop_monitoring_spam` (sub-second cadence) | Each session is killed by the modem's mode-flip before the device can drain its UART RX buffer; high-rate spam also risks UART FIFO overrun on the device side |
| Outbound port firewall block alone | Stops the outbound TCP from succeeding, but doesn't stop the modem from *trying* and mode-flipping. Reduces but doesn't eliminate the contention. |
| Modem reboot | Temporary — as soon as the device starts triggering again, the loop resumes within seconds |
The combination of `slow_drip` + cleared `Destination Address` works because:
1. The modem stops mode-flipping → TCP session stays open for the full
drip duration
2. Slow drip rate → device's UART RX FIFO never overflows even if
firmware is busy with event recording
3. The drip is `SESSION_RESET + STOP_MONITORING` every 3s → many
independent chances for the parser to land one valid frame
4. Once one Stop Monitoring is parsed, event recording halts → firmware
has CPU to spare → subsequent operations are trivially easy
---
## Tooling reference
All endpoints live in `seismo-relay/sfm/server.py`. All scripts live in
`seismo-relay/scripts/` and default to SFM direct (`http://localhost:8200`),
overridable via `SFM_BASE_URL`.
### Endpoints added during BE9558H recovery
| Endpoint | Purpose |
|---|---|
| `GET /device/events/storage_range` | SUB 0x06 — first/last event keys, `is_empty` flag. ~2s, no event walk. |
| `GET /device/events/index` | SUB 0x08 — lifetime event counter (does NOT decrement on erase). ~2s. |
| `POST /device/events/erase` | Full erase sequence 0xA3 → 0x1C → 0x06 → 0xA2. |
| `POST /device/rescue` | Disable ACH + erase in one TCP session. Short timeouts for race-loop usage. |
| `POST /device/stop_monitoring_blind` | Fire-and-forget Stop with full POLL preamble (single attempt). |
| `POST /device/stop_monitoring_spam` | Server-side tight retry loop, sub-second cadence, duration-bounded. |
| `POST /device/stop_monitoring_slow_drip` | One held TCP session, slow trickle of stop frames. **The endpoint that saved BE9558H.** |
Also changed: default protocol recv timeout dropped from 30s → 10s in
`_build_client`. Added `connect_timeout` knob to same. Cleaned up
unhandled-exception path in `/device/monitor/status` so it returns 502
instead of 500 on protocol timeouts.
### Scripts
| Script | Purpose |
|---|---|
| `scripts/rescue_device.sh` | Race-loop wrapper around `/device/rescue` |
| `scripts/blind_stop.sh` | Race-loop wrapper around `/device/stop_monitoring_blind` |
| `scripts/spam_stop.sh` | Single-call burst hammer (`/device/stop_monitoring_spam`) |
| `scripts/slow_drip.sh` | Single-call held-session drip (`/device/stop_monitoring_slow_drip`) |
| `scripts/watch_unit.sh` | Passive periodic reachability check, logs to file |
---
## Incident log — BE9558H, 2026-05-16/17
What was wrong: Long-axis geophone developed an offset, constantly above
trigger threshold → constant event recording → after-event ACH set →
modem dialing office BW server (`50.197.32.92:12345`) every 30-60s.
Local event chain corrupted (`next_boundary 0x100EE exceeds uint16`).
Diagnostic path:
1. `/device/info` slow, choked on event walk
2. Built lightweight probe endpoints (`storage_range`, `index`) — useful
but didn't reach the wedged unit
3. Built `/device/rescue` with short timeouts — got 502 (POLL no response)
4. Built `/device/stop_monitoring_blind` — first version was a false
positive (no POLL preamble); fixed by including
`SESSION_RESET+POLL_PROBE+SESSION_RESET+POLL_DATA` in the dump
5. Verified blind stop works on bench unit
6. Built `/device/stop_monitoring_spam` — 420 successful sends over
5 min, zero behavior change on field unit
7. Inspected ACEmanager logs → saw outbound dial-out attempts every ~30s,
confirmed device was not fully locked up
8. Added outbound port-12345 firewall block → outbound attempts now fail
instantly but contention persisted
9. Built `/device/stop_monitoring_slow_drip` — session died at 3s with
broken pipe (modem closing on us)
10. Looked at full ACEmanager Port Configuration → **found
`Destination Address: 50.197.32.92` configured**, realized every AT
dial command was triggering a modem mode-flip that killed our inbound
11. Cleared Destination Address + Port → slow_drip held 120s, device
responded with 990 bytes, 39 stop commands acked
12. Disabled ACH at device level via `/device/call_home`, erased events
Final state: device IDLE, memory 958.1 / 960 KB free, ACH disabled at
device level, modem destination cleared (to be restored after physical
service).
Total time from "i was wondering if its possible to" first attempt to
recovery: ~7 hours of intermittent debugging across one evening.
docs/waveform_codec_re_status.md
+264
View File
@@ -0,0 +1,264 @@
# Waveform body codec — FULLY DECODED (2026-05-11)
This is the **clean working note** for the body-codec reverse-engineering
effort. It supersedes scattered claims elsewhere when they conflict.
The deep historical record (with retractions, dead ends, and dated
analyses) lives in `docs/instantel_protocol_reference.md §7.6.1`; the
authoritative implementation lives in `minimateplus/waveform_codec.py`.
## TL;DR
**The codec is fully decoded.** Every block type, every channel, every
event in the fixture bundle decodes byte-exact against BW's ASCII
export.
| Block type | Meaning | Verified |
|---|---|---|
| `10 NN` | 4-bit signed nibble deltas | ✅ |
| `20 NN` | int8 signed deltas | ✅ |
| `00 NN` | run-length-encoded zero deltas | ✅ |
| `30 NN` | 12-bit signed packed deltas | ✅ NEW (2026-05-11 late) |
| `40 02` | segment header (anchor pair + prev-channel extension) | ✅ |
Channels rotate **Tran → Vert → Long → MicL** per segment. Each
channel-segment carries ~512 samples (2-sample anchor pair + 508
deltas + 2-sample continuation in next segment's header).
## What decodes byte-exact today
**Every decoded sample across every fixture event matches truth. Zero
divergences.**
| Event | Description | Tran | Vert | Long | Total |
|---|---|---|---|---|---|
| event-a (5-8) | quiet, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
| event-c (5-8) | quiet, 1 sec | 1280 ✓ | 1280 ✓ | 1280 ✓ | 3840 |
| event-d (5-8) | quiet, 1 sec | 1280 ✓ | 1280 ✓ | 1280 ✓ | 3840 |
| JQ0 (5-11) | Vert-heavy, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
| V70 (5-11) | Mic-heavy, 3 sec | 3328 ✓ | 3328 ✓ | 3328 ✓ | **9984** |
| SP0 (5-11) | loud all, 3 sec | 2048 ✓ | 1538 ✓ | 1536 ✓ | 5122 |
| SS0 (5-11) | loud-from-start | 734 ✓ | 512 ✓ | 512 ✓ | 1758 |
| SV0 (5-11) | loud-from-start | 1024 ✓ | 578 ✓ | 512 ✓ | 2114 |
| event-b (5-8) | quiet, 2 sec | 512 ✓ | 226 ✓ | 0 | 738 |
That's **47,364 ADC samples decoded byte-exact, zero errors.**
Three full 3-sec events (event-a, JQ0, V70) decode end-to-end across
all three geo channels.
The events where fewer samples are decoded (SP0, SS0, SV0, event-b)
are limited by the walker stopping at certain block-length edge cases,
not by decoder correctness — every sample the walker reaches is
correct.
## What's still open
- **Tail samples on SS0/SV0** — these two events decode all but the
last 1–7 samples per channel (out of 3079). Likely the same
"last segment is truncated" pattern. Minor; doesn't affect the
bulk of the data.
## Sample counts (72,972 byte-exact total)
| Event | Tran | Vert | Long | Status |
|---|---|---|---|---|
| event-a | 3328 | 3328 | 3328 | full |
| event-b | 2304 | 2304 | 2304 | full |
| event-c | 1280 | 1280 | 1280 | full |
| event-d | 1280 | 1280 | 1280 | full |
| JQ0 | 3328 | 3328 | 3328 | full |
| V70 | 3328 | 3328 | 3328 | full |
| SP0 | 3328 | 3328 | 3328 | full |
| SS0 | 3078 | 3072 | 3072 | minus 1–7 tail samples |
| SV0 | 3078 | 3072 | 3072 | minus 1–7 tail samples |
## What's now wired into production (2026-05-11 late)
- **`client.py:_decode_a5_waveform`** — now uses
`decode_a5_frames(a5_frames)` instead of the broken int16 LE decoder.
`event.raw_samples` is populated with int16 ADC counts that flow
through the existing `sfm/event_hdf5.py` scaling pipeline unchanged.
Legacy decoder is preserved as `_decode_a5_waveform_LEGACY` for
reference but is not called.
- **MicL → dB(L) conversion** — exposed as
`waveform_codec.mic_count_to_db(count)`. Verified against BW
display values (count=1 → 81.94 dB; count=813 → 140.14 dB; matches
the V70 mic-heavy fixture exactly).
- **`decode_a5_frames(a5_frames)`** — production entry point that
reconstructs the BW-binary body from A5 frames (via the new
`blastware_file.extract_body_bytes` helper) and runs the verified
codec. Returns the same `raw_samples` dict shape the consumers
already expect.
## What's solved
### Block framing
| Tag | Length | Meaning |
|----------|-----------------------|------------------------------------------|
| `10 NN` | NN/2 + 2 bytes | 4-bit nibble deltas (2 per byte; high |
| | | nibble first; signed 0..7 / 8..F = -8..-1)|
| `20 NN` | NN + 2 bytes | int8 signed deltas (1 per byte) |
| `00 NN` | 2 bytes | RLE: append NN copies of current value |
| `30 NN` | NN*2 in data section, | Unknown content. Only in loud-from- |
| | NN*4 in trailer | start events. |
| `40 02` | 20 bytes (fixed) | Segment header |
NN is always a multiple of 4.
Implementation: `walk_body()` in `minimateplus/waveform_codec.py`.
### 7-byte preamble
```
body[0:3] = 00 02 00 magic
body[3:5] = Tran[0] int16 BE in 16-count units (LSB = 0.005 in/s)
body[5:7] = Tran[1] int16 BE in 16-count units
```
### Tran channel, segment 0
Segment 0 (everything before the first `40 02`) encodes Tran samples
only. Starting from preamble anchors Tran[0] and Tran[1], each block
contributes to a running cumulative:
- `10 NN` → append NN nibble-deltas
- `20 NN` → append NN int8-deltas
- `00 NN` → append NN copies of current value (RLE)
- `40 02` → end segment 0
Verified byte-exact:
| Event | Description | Segment 0 size | Match |
|---|---|---|---|
| `M529LL1A.SP0` | Loud, 0.25 s pretrig | 510 | 510/510 ✓ |
| `M529LL1A.SV0` | Loud from sample 0 | 58 | 58/58 ✓ (stops at first `30 NN`) |
| `M529LL1A.SS0` | Loud from sample 0 | 42 | 42/42 ✓ (stops at first `30 04`) |
| `M529LL1L.JQ0` | Vert-heavy | 510 | 510/510 ✓ |
| `M529LL1L.V70` | Mic-heavy (140 dB) | 510 | 510/510 ✓ |
Implementation: `decode_tran_initial()`.
### Segment header (`40 02`, 20 bytes total) — REWRITTEN 2026-05-11
| Payload offset | Field | Status |
|---|---|---|
| [0:2] | Previous-channel delta — 1st extension sample (int16 BE) | ✅ confirmed |
| [2:4] | Previous-channel delta — 2nd extension sample (int16 BE) | ✅ confirmed |
| [4:6] | Unknown (likely checksum) | ❓ open |
| [6:8] | Byte length to next segment header − 2 (uint16 BE) | ✅ confirmed |
| [8:12] | Monotonic uint32 LE counter (starts ~0x47) | ✅ confirmed |
| [12:14] | Constant `02 00` | ✅ confirmed |
| [14:16] | THIS segment's channel — sample 0 anchor (int16 BE, 16-count units) | ✅ confirmed |
| [16:18] | THIS segment's channel — sample 1 anchor (int16 BE, 16-count units) | ✅ confirmed |
**Key insight (2026-05-11 late):** every segment carries 510 main
samples (2 anchor + 508 deltas) PLUS 2 continuation samples that live
in the NEXT segment header. So each channel-segment effectively spans
512 sample-sets. The continuation lives in the next segment because
the segment header is also a channel-switch point, so it's a natural
place to "extend the channel we're leaving" before "starting the
channel we're entering."
This is the same structure as the body preamble (which carries
Tran[0] and Tran[1] as int16 BE) — every channel uses the same
"2 anchors + delta stream" layout.
## Channel rotation — VERIFIED 2026-05-11
```
(initial body) → Tran samples 0..509 (preamble + delta blocks)
segment 0 hdr ext+anchor → Vert samples 0..511 ← anchor in hdr [14:18]
segment 1 hdr ext+anchor → Long samples 0..511
segment 2 hdr ext+anchor → Mic samples 0..511
segment 3 hdr ext+anchor → Tran samples 510..1021 (continuation)
segment 4 hdr ext+anchor → Vert samples 512..1023
segment 5 hdr ext+anchor → Long samples 512..1023
segment 6 hdr ext+anchor → Mic samples 512..1023
segment 7 hdr ext+anchor → Tran samples 1022..1533
...
```
Implementation: `decode_waveform_v2()` returns
`{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}` with
each channel's samples in 16-count units. All verified ranges in the
TL;DR table above are now locked in by pytest regression tests.
## What's still open
1. **`30 NN` block content.** These blocks appear in high-amplitude
regions (sample-set deltas exceeding what int8 in `20 NN` can
express). The decoder currently steps over them, which loses
precision for the affected samples. Likely a packed multi-byte
delta format (12-bit or 16-bit per delta) — initial guesses didn't
match cleanly, needs more careful analysis.
2. **MicL decoding.** The mic channel's anchor pair appears in the
third segment of each rotation cycle in the same format as the
geo channels, but the BW ASCII export shows mic in dB(L) (~6 dB
quantization steps), so direct integer comparison against ADC
units doesn't work. Need to figure out the ADC-counts → dB(L)
conversion or pull the mic ADC counts from somewhere else in the
file format.
3. **Walker fix for event-b.** The original quiet bundle's event-b
still bails out partway through. Lower priority since the other
7 events walk cleanly.
## `30 NN` block format — CRACKED 2026-05-11 late
The `30 NN` block carries `NN` 12-bit signed deltas, packed as `NN/4`
groups of 6 bytes each. Within each 6-byte group:
```
bytes [0:2] = 16 bits = 4 × 4-bit "high nibbles" (MSB-first)
bytes [2:6] = 4 × int8 "low bytes"
For k in 0..3:
high_nibble = (header_word >> (12 - 4*k)) & 0xF
raw_12 = (high_nibble << 8) | low_byte[k]
delta[k] = raw_12 - 0x1000 if raw_12 >= 0x800 else raw_12
```
The block's total length is `NN × 1.5 + 2` bytes (tag included). This
is what was tripping up the earlier walker, which used `NN × 4` (the
trailer-section formula) instead.
Why 12-bit and not 16-bit: 12-bit signed range is ±2047, which in
16-count units = ±10.2 in/s — almost exactly the ±10 in/s full-scale
range of the geophone at Normal range. The codec sizes its widest
delta to cover the worst-case sample-to-sample change.
Verified against all 14 `30 NN` blocks across the bundled fixture
events. Every delta decodes byte-exact against BW's ASCII export.
## Test fixtures
Committed under `tests/fixtures/`:
- `decode-re-5-8-26/event-a..event-d/`: original quiet bundle (4 events,
PPV < 1 in/s). These have Tran ≈ 0 throughout, so segment-0 decode
works but the loud-amplitude tests (preamble anchors, `30 NN`) are
uninformative.
- `5-11-26/M529LL1A.{SP0,SS0,SV0}`: loud bundle (PPV 6-7 in/s on all
channels). These cracked the Tran codec.
- `5-11-26/M529LL1L.{JQ0,V70}`: targeted captures. JQ0 is Vert-heavy,
V70 is Mic-heavy (140 dB). These cracked the `00 NN` RLE rule.
Each fixture has a `.TXT` Blastware ASCII export as ground truth.
## Tests
`tests/test_waveform_codec.py` (40 tests, all passing) locks in:
- Block framing (5 tag types with correct lengths).
- Walker contiguity (no gaps or overlaps).
- Segment header parsing (counter monotonicity, fixed-pattern check).
- `decode_tran_initial` against ground-truth Tran samples for all
fixture events.
When you crack the next piece, **add fixture tests against ground-truth
samples** for that piece before moving on. Don't let unverified code
ship without a regression lock-in.
micromate/__init__.py
+48
View File
@@ -0,0 +1,48 @@
"""
micromate — Instantel Micromate (Series IV) device library.
Sibling of ``minimateplus`` (the Series III library). Currently scoped to
the offline-file ingest path used by thor-watcher: parsing the per-event
``.IDFH``/``.IDFW`` ASCII text sidecars Thor's exporter writes alongside
each binary event file, and wrapping the parsed data in typed event
records.
Live-device support (TCP protocol, frame parsing, real-time monitoring)
is deferred — when we add it, it lands here as ``transport.py`` /
``framing.py`` / ``protocol.py`` / ``client.py``, mirroring the
``minimateplus`` package layout.
Typical usage (offline file ingest):
from micromate import IdfEvent, parse_idf_report
text = open("UM11719_20231219162723.IDFW.txt").read()
rep = parse_idf_report(text) # dict
event = IdfEvent.from_report(rep, "UM11719_20231219162723.IDFW")
print(event.serial, event.peaks.transverse_ips, event.mic_pspl_dbl)
"""
from .idf_ascii_report import (
parse_event_filename,
parse_idf_report,
serial_from_filename,
)
from .models import (
IdfEvent,
IdfPeaks,
IdfProjectInfo,
IdfReport,
IdfSensorCheck,
)
__version__ = "0.1.0"
__all__ = [
"IdfEvent",
"IdfPeaks",
"IdfProjectInfo",
"IdfReport",
"IdfSensorCheck",
"parse_event_filename",
"parse_idf_report",
"serial_from_filename",
]
micromate/idf_ascii_report.py
+315
View File
@@ -0,0 +1,315 @@
"""
micromate/idf_ascii_report.py — parse Thor (Micromate Series IV) IDF ASCII reports.
Thor exports a `.IDFW.txt` or `.IDFH.txt` sidecar next to each `.IDFW`
(waveform) or `.IDFH` (histogram) event binary. Each sidecar is a
plain-text file with `"Key : Value"` lines covering the full device-
authoritative event metadata — PPV per channel, ZC Freq, Time of Peak,
Peak Acceleration / Displacement, sensor self-check results, project
strings, calibration date, battery level, etc. — followed by a raw
waveform-samples block headed by the literal line "Waveform Data Channels".
This is the Thor analogue of `minimateplus/bw_ascii_report.py` for the
Blastware (Series III) report format. The parser is intentionally
permissive: we extract everything we recognise into a flat dict and
silently ignore anything we don't. Downstream callers parse units
(`"0.2119 in/s"` → 0.2119) only on the fields they need.
Example input (truncated):
"EventType : Full Waveform"
"SampleRate : 1024 sps"
"EventTime : 16:27:23"
"EventDate : 2023-12-19"
"TranPPV : 0.0251 in/s"
"VertPPV : 0.2119 in/s"
"LongPPV : 0.0282 in/s"
"PeakVectorSum : 0.2131 in/s"
"MicPSPL : 99.4 dB(L)"
"TranZCFreq : 6.5 Hz"
"SerialNumber : UM11719"
"Version : Micromate ISEE 11.0AK"
"FileName : UM11719_20231219162723.IDFW"
"BatteryLevel : 3.8 volts"
"Calibration : November 22, 2023 by Instantel"
"TranTestResults : Passed"
"TitleString1 : UPMC Presby-Loc 3-Level1-1R Elevator Rm"
Waveform Data Channels
Tran Vert Long MicL
0.0003 -0.0003 0.0003 0.00013
...
"""
from __future__ import annotations
import datetime
import re
from typing import Any, Dict, Optional, Tuple, Union
# Lines look like: "Key : Value" (quotes literal, single ":" separator)
_LINE_RE = re.compile(r'^\s*"?([^":]+?)"?\s*:\s*"?(.*?)"?\s*$')
# Marker that ends the metadata block — everything after is raw sample data.
_WAVEFORM_BLOCK_MARKER = "waveform data channels"
def _normalize_key(raw: str) -> str:
"""Convert "TranPPV" / "PreTriggerLength" → snake_case."""
s = raw.strip()
# Insert underscore between lower→upper / digit→letter transitions
s = re.sub(r"(?<=[a-z0-9])(?=[A-Z])", "_", s)
s = re.sub(r"(?<=[A-Z])(?=[A-Z][a-z])", "_", s)
s = s.replace("-", "_").replace(" ", "_")
return s.lower()
def _strip_unit_suffix(value: str) -> str:
"""Return the numeric part of values like "0.2119 in/s" → "0.2119".
Also strips Thor's below/above-threshold prefixes:
"<0.005 in/s" → "0.005" (below-noise-floor reading)
">100 Hz" → "100" (above-measurement-range reading)
"""
parts = value.strip().split()
token = parts[0] if parts else value.strip()
if token.startswith("<") or token.startswith(">"):
token = token[1:]
return token
def _parse_float(value: str) -> Optional[float]:
try:
return float(_strip_unit_suffix(value))
except (ValueError, TypeError):
return None
def _parse_int(value: str) -> Optional[int]:
try:
return int(float(_strip_unit_suffix(value)))
except (ValueError, TypeError):
return None
def parse_idf_report(text: Union[str, bytes]) -> Dict[str, Any]:
"""
Parse a Thor IDFW.txt / IDFH.txt sidecar.
Returns a flat dict with two kinds of entries:
- **Raw fields** — every `Key : Value` line, keyed by snake_case
of the original key, value as a string (unit suffix preserved).
Lets callers grab any field we haven't explicitly normalised.
- **Derived fields** — a curated set with parsed types:
* `serial_number` str
* `event_type` str ("Full Waveform" / "Full Histogram")
* `event_datetime` ISO-8601 string ("YYYY-MM-DDTHH:MM:SS") when
both EventDate and EventTime are present
* `sample_rate` int (samples/sec)
* `tran_ppv`,`vert_ppv`,`long_ppv` float (in/s)
* `mic_ppv` float (dB or psi — same units as MicPSPL)
* `peak_vector_sum` float (in/s)
* `tran_zc_freq`,`vert_zc_freq`,`long_zc_freq` float (Hz)
* `record_time_sec` float (seconds)
* `pre_trigger_sec` float (seconds)
* `project` str (from TitleString1 — Thor's location)
* `client` str (TitleString2)
* `operator` str (TitleString3 — company/operator)
* `notes` str (TitleString4)
* `setup` str
* `version` str (firmware)
* `battery_volts` float
* `calibration_text` str (e.g. "November 22, 2023 by Instantel")
* `tran_test_passed`, `vert_test_passed`, `long_test_passed`,
`mic_test_passed` bool ("Passed" → True; anything else → False)
* `filename` str (FileName line — useful sanity check)
Stops parsing at the literal "Waveform Data Channels" line; the
raw-samples block is left to whoever wants to decode the binary.
Input may be `str` or `bytes` (`utf-8`/`latin-1` tolerant).
"""
if isinstance(text, bytes):
try:
text = text.decode("utf-8")
except UnicodeDecodeError:
text = text.decode("latin-1", errors="replace")
raw: Dict[str, str] = {}
for line in text.splitlines():
stripped = line.strip()
if not stripped:
continue
if stripped.lower().startswith(_WAVEFORM_BLOCK_MARKER):
break
m = _LINE_RE.match(stripped)
if not m:
continue
key = _normalize_key(m.group(1))
value = m.group(2).strip()
# Multi-value lines (Channel, Units, etc.) — coalesce by appending.
if key in raw:
raw[key] = raw[key] + "; " + value
else:
raw[key] = value
out: Dict[str, Any] = dict(raw) # keep all raw fields
# ── Derived fields ───────────────────────────────────────────────────────
def _take(*candidates: str) -> Optional[str]:
for c in candidates:
if c in raw:
return raw[c]
return None
# Event identity
if "serial_number" in raw:
out["serial_number"] = raw["serial_number"]
if "event_type" in raw:
out["event_type"] = raw["event_type"]
if "file_name" in raw:
out["filename"] = raw["file_name"]
# Combined date+time. Waveform sidecars use "EventDate" / "EventTime";
# histogram sidecars use "HistogramStartDate" / "HistogramStartTime".
# Prefer the event_* names when both are present.
ed = raw.get("event_date") or raw.get("histogram_start_date")
et = raw.get("event_time") or raw.get("histogram_start_time")
if ed and et:
try:
dt = datetime.datetime.strptime(f"{ed} {et}", "%Y-%m-%d %H:%M:%S")
out["event_datetime"] = dt.isoformat()
except ValueError:
pass
# Numeric scalars. For every field we typify here, we MUST drop the
# raw string copy from `out` when parsing fails — Thor writes things
# like "<0.005 in/s" (below threshold) and "N/A" (not measured) that
# would otherwise linger in `out` as strings, sneak into SQLite REAL
# columns via permissive type affinity, and then crash the JS
# frontend on `.toFixed(...)`.
int_fields = ("sample_rate",)
for key in int_fields:
v = raw.get(key)
if v is None:
continue
iv = _parse_int(v)
if iv is not None:
out[key] = iv
else:
out.pop(key, None)
float_fields = (
"tran_ppv", "vert_ppv", "long_ppv", "peak_vector_sum",
"tran_zc_freq", "vert_zc_freq", "long_zc_freq",
"tran_peak_acceleration", "vert_peak_acceleration",
"long_peak_acceleration",
"tran_peak_displacement", "vert_peak_displacement",
"long_peak_displacement",
"tran_time_of_peak", "vert_time_of_peak", "long_time_of_peak",
"mic_time_of_peak", "mic_zc_freq",
)
for key in float_fields:
v = raw.get(key)
if v is None:
continue
fv = _parse_float(v)
if fv is not None:
out[key] = fv
else:
out.pop(key, None)
# Microphone — Thor reports MicPSPL (dB(L)) which is the closest
# analogue to BW's mic_ppv. The raw "99.4 dB(L)" string stays in
# `out` under the original `mic_pspl` key for display; the parsed
# float goes in `mic_ppv`.
mic = raw.get("mic_pspl")
if mic is not None:
fv = _parse_float(mic)
if fv is not None:
out["mic_ppv"] = fv
# Record / pre-trigger duration — same drop-on-failure discipline.
rt = raw.get("record_time")
if rt is not None:
fv = _parse_float(rt)
if fv is not None:
out["record_time_sec"] = fv
pt = raw.get("pre_trigger_length")
if pt is not None:
fv = _parse_float(pt)
if fv is not None:
out["pre_trigger_sec"] = fv
# Project / client / operator / location strings. Thor's title
# strings are operator-defined; conventional mapping (per Thor's
# default TitleNote labels in the example data):
# TitleString1 = Location → project (sensor location identifier)
# TitleString2 = Client → client
# TitleString3 = Company → operator (the monitoring company)
# TitleString4 = Notes → notes
out["project"] = _take("title_string1")
out["client"] = _take("title_string2")
out["operator"] = _take("title_string3", "operator")
out["notes"] = _take("title_string4", "post_event_note")
if "setup" in raw:
out["setup"] = raw["setup"]
if "version" in raw:
out["version"] = raw["version"]
# Battery (e.g. "3.8 volts" → 3.8)
bl = raw.get("battery_level")
if bl is not None:
fv = _parse_float(bl)
if fv is not None:
out["battery_volts"] = fv
# Calibration line is free-form (e.g. "November 22, 2023 by Instantel").
if "calibration" in raw:
out["calibration_text"] = raw["calibration"]
# Sensor self-check results — bool flags
for key, out_key in (
("tran_test_results", "tran_test_passed"),
("vert_test_results", "vert_test_passed"),
("long_test_results", "long_test_passed"),
("mic_test_results", "mic_test_passed"),
):
v = raw.get(key)
if v is not None:
out[out_key] = v.strip().lower() == "passed"
return out
def serial_from_filename(name: str) -> Optional[str]:
"""Convenience: pull the serial prefix from a Thor event filename.
Thor uses the literal serial as the filename prefix:
UM11719_20231219163444.IDFW → "UM11719"
BE9439_20200713124251.IDFH → "BE9439"
"""
m = re.match(r"^([A-Z]{2}\d+)_\d{14}\.(IDFH|IDFW)(?:\.txt)?$",
name, re.IGNORECASE)
return m.group(1).upper() if m else None
def parse_event_filename(name: str) -> Optional[Tuple[str, datetime.datetime, str]]:
"""Parse `<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>` → (serial, datetime, kind).
`kind` is "IDFH" or "IDFW" (upper-case). Returns None on no match.
"""
m = re.match(r"^([A-Z]{2}\d+)_(\d{14})\.(IDFH|IDFW)$",
name, re.IGNORECASE)
if not m:
return None
try:
ts = datetime.datetime.strptime(m.group(2), "%Y%m%d%H%M%S")
except ValueError:
return None
return m.group(1).upper(), ts, m.group(3).upper()
micromate/idf_file.py
+64
View File
@@ -0,0 +1,64 @@
"""
micromate/idf_file.py — placeholder for the Thor IDF binary codec.
Thor's ``.IDFH`` (histogram) and ``.IDFW`` (waveform) event files are an
Instantel proprietary binary format that has not yet been reverse-
engineered. Today seismo-relay treats them as opaque blobs:
``WaveformStore.save_imported_idf`` stores the bytes verbatim and reads
all device-authoritative metadata from the paired ``.IDFW.txt`` /
``.IDFH.txt`` ASCII sidecar (parsed by ``idf_ascii_report.py``).
When we crack the binary codec — same reverse-engineering playbook we
used to byte-perfect-parse Series III BW files (see
``docs/instantel_protocol_reference.md`` and ``minimateplus/event_file_io.py``)
— this module will grow:
- ``read_idf_file(path) -> IdfEvent``
Parse a ``.IDFW``/``.IDFH`` binary and return a fully populated
``IdfEvent`` whose waveform-sample arrays come from the binary
(the .txt sidecar's tabular sample block being a best-effort
check). Lets us ingest Thor events even when the operator
hasn't enabled the .txt exporter — closing the
``had_report=False`` gap that the thor-watcher forwarder
currently tolerates as a known limitation.
- ``write_idf_file(path, event)`` (eventually)
Round-trip event reconstruction, used for verifying the codec
against captured device files the way ``write_blastware_file``
verifies the Series III codec.
- Helpers for decoding the binary's per-channel sample arrays into
physical units, the per-event flash buffer's monitor-log records,
etc.
The reverse-engineering path: pair every ``.IDFW`` binary in
``thor-watcher/example-data/`` with its sibling ``.IDFW.txt``, treating
the txt's "Waveform Data Channels" block as ground-truth, and align
the binary's per-channel int16-or-similar arrays against it. Header
fields (sample rate, channel count, record time, timestamps) sit before
the sample block — same approach as the BW codec where ASCII strings
inside the binary (``Project:``, ``Client:``, etc.) anchored field
discovery.
"""
from __future__ import annotations
from pathlib import Path
from typing import Union
from .models import IdfEvent
def read_idf_file(path: Union[str, Path]) -> "IdfEvent":
"""Parse a Thor ``.IDFW``/``.IDFH`` binary into an ``IdfEvent``.
Not yet implemented. When implemented, this will be the canonical
entry point for reading Thor binaries — the ASCII sidecar parser
becomes an optional fast-path metadata supplement rather than the
sole source of device-authoritative data.
"""
raise NotImplementedError(
"IDF binary codec not yet implemented; the .IDFW/.IDFH binary format "
"is undecoded. Use parse_idf_report() on the paired .txt sidecar "
"for device-authoritative metadata."
)
micromate/models.py
+377
View File
@@ -0,0 +1,377 @@
"""
Micromate (Series IV / Thor) native data models.
These are the right-shaped dataclasses for Thor data — Thor measures
the microphone in dB(L) directly, so this model carries
``mic_pspl_dbl`` rather than the pseudo-``psi`` shoehorn that
``minimateplus.PeakValues`` uses for Series III BW data.
The ingest pipeline today goes:
.IDFW.txt → parse_idf_report() → dict
dict → IdfEvent.from_report() → IdfEvent (typed)
IdfEvent → IdfEvent.to_minimateplus_event() → shape DB / sidecar
machinery expects
The ``to_minimateplus_event()`` bridge is a temporary boundary — when we
crack the binary IDF codec and have richer per-event data to store, the
DB schema will grow Series-IV-specific columns and the bridge will
shrink or disappear.
"""
from __future__ import annotations
import datetime
from dataclasses import dataclass, field
from typing import Any, Dict, Optional, Tuple
# ── IdfReport ─────────────────────────────────────────────────────────────────
@dataclass
class IdfReport:
"""Typed wrapper around the dict returned by ``parse_idf_report``.
All fields optional — Thor's exporter is permissive and some IDF .txt
files (especially histograms) omit fields that waveform sidecars
include. Use ``.raw`` for any field this dataclass hasn't surfaced
yet (the parser keeps every recognised key in the raw dict).
"""
# Identity / kind
serial_number: Optional[str] = None
event_type: Optional[str] = None # "Full Waveform" | "Full Histogram"
event_datetime: Optional[datetime.datetime] = None
filename: Optional[str] = None # echoed by Thor's exporter
# Sampling / timing
sample_rate: Optional[int] = None # samples/sec
record_time_sec: Optional[float] = None
pre_trigger_sec: Optional[float] = None
# Geophone peaks (in/s)
tran_ppv: Optional[float] = None
vert_ppv: Optional[float] = None
long_ppv: Optional[float] = None
peak_vector_sum: Optional[float] = None
# Microphone — Thor's native unit is dB(L), NOT psi.
mic_pspl_dbl: Optional[float] = None
# Zero-crossing frequencies (Hz)
tran_zc_freq: Optional[float] = None
vert_zc_freq: Optional[float] = None
long_zc_freq: Optional[float] = None
mic_zc_freq: Optional[float] = None
# Per-channel time of peak (sec, since event start)
tran_time_of_peak: Optional[float] = None
vert_time_of_peak: Optional[float] = None
long_time_of_peak: Optional[float] = None
mic_time_of_peak: Optional[float] = None
# Derived per-channel motion
tran_peak_acceleration: Optional[float] = None # g
vert_peak_acceleration: Optional[float] = None
long_peak_acceleration: Optional[float] = None
tran_peak_displacement: Optional[float] = None # in
vert_peak_displacement: Optional[float] = None
long_peak_displacement: Optional[float] = None
# Operator-supplied strings (Thor's TitleString1..4 → semantic slots)
project: Optional[str] = None # TitleString1
client: Optional[str] = None # TitleString2
operator: Optional[str] = None # TitleString3
notes: Optional[str] = None # TitleString4 / PostEventNote
setup: Optional[str] = None # setup file name
# Sensor self-check results
tran_test_passed: Optional[bool] = None
vert_test_passed: Optional[bool] = None
long_test_passed: Optional[bool] = None
mic_test_passed: Optional[bool] = None
# Device-fixed metadata
firmware_version: Optional[str] = None
calibration_text: Optional[str] = None
battery_volts: Optional[float] = None
# Original parser dict — preserves every recognised key (including
# raw unit-suffixed strings) for forward-compatible field access.
raw: Dict[str, Any] = field(default_factory=dict, repr=False)
@classmethod
def from_dict(cls, d: Dict[str, Any]) -> "IdfReport":
"""Build an IdfReport from the dict returned by ``parse_idf_report``."""
ed = d.get("event_datetime")
if isinstance(ed, str):
try:
ed = datetime.datetime.fromisoformat(ed)
except ValueError:
ed = None
return cls(
serial_number = d.get("serial_number"),
event_type = d.get("event_type"),
event_datetime = ed if isinstance(ed, datetime.datetime) else None,
filename = d.get("filename"),
sample_rate = d.get("sample_rate"),
record_time_sec = d.get("record_time_sec"),
pre_trigger_sec = d.get("pre_trigger_sec"),
tran_ppv = d.get("tran_ppv"),
vert_ppv = d.get("vert_ppv"),
long_ppv = d.get("long_ppv"),
peak_vector_sum = d.get("peak_vector_sum"),
mic_pspl_dbl = d.get("mic_ppv"), # parser names it mic_ppv (legacy)
tran_zc_freq = d.get("tran_zc_freq"),
vert_zc_freq = d.get("vert_zc_freq"),
long_zc_freq = d.get("long_zc_freq"),
mic_zc_freq = d.get("mic_zc_freq"),
tran_time_of_peak = d.get("tran_time_of_peak"),
vert_time_of_peak = d.get("vert_time_of_peak"),
long_time_of_peak = d.get("long_time_of_peak"),
mic_time_of_peak = d.get("mic_time_of_peak"),
tran_peak_acceleration = d.get("tran_peak_acceleration"),
vert_peak_acceleration = d.get("vert_peak_acceleration"),
long_peak_acceleration = d.get("long_peak_acceleration"),
tran_peak_displacement = d.get("tran_peak_displacement"),
vert_peak_displacement = d.get("vert_peak_displacement"),
long_peak_displacement = d.get("long_peak_displacement"),
project = d.get("project"),
client = d.get("client"),
operator = d.get("operator"),
notes = d.get("notes"),
setup = d.get("setup"),
tran_test_passed = d.get("tran_test_passed"),
vert_test_passed = d.get("vert_test_passed"),
long_test_passed = d.get("long_test_passed"),
mic_test_passed = d.get("mic_test_passed"),
firmware_version = d.get("version"),
calibration_text = d.get("calibration_text"),
battery_volts = d.get("battery_volts"),
raw = d,
)
# ── IdfPeaks / IdfProjectInfo / IdfSensorCheck (narrow grouping types) ───────
@dataclass
class IdfPeaks:
"""Geophone + mic peak values for one Thor event. Native Thor units."""
transverse_ips: Optional[float] = None # in/s
vertical_ips: Optional[float] = None # in/s
longitudinal_ips: Optional[float] = None # in/s
peak_vector_sum_ips: Optional[float] = None # in/s
mic_pspl_dbl: Optional[float] = None # dB(L)
@dataclass
class IdfProjectInfo:
"""Operator-supplied strings from Thor's TitleString1..4."""
project: Optional[str] = None
client: Optional[str] = None
operator: Optional[str] = None
notes: Optional[str] = None
setup: Optional[str] = None
@dataclass
class IdfSensorCheck:
"""Per-channel pass/fail from Thor's self-test."""
tran: Optional[bool] = None
vert: Optional[bool] = None
long: Optional[bool] = None
mic: Optional[bool] = None
# ── IdfEvent ─────────────────────────────────────────────────────────────────
@dataclass
class IdfEvent:
"""A single Thor / Micromate Series IV event.
Built from a parsed .IDFW.txt or .IDFH.txt sidecar via
``IdfEvent.from_report()``. The filename is the authoritative
source for serial + timestamp + kind; the .txt provides
device-authoritative peak values, frequencies, project strings,
sensor self-check, firmware, calibration.
"""
# Identity
serial: str
timestamp: datetime.datetime
kind: str # "Waveform" | "Histogram"
filename: str # device-native binary filename, e.g. "UM11719_20231219163444.IDFW"
# Sampling / timing
sample_rate: Optional[int] = None
record_time_sec: Optional[float] = None
pre_trigger_sec: Optional[float] = None
# Peaks
peaks: IdfPeaks = field(default_factory=IdfPeaks)
# Per-channel frequencies (Hz)
tran_zc_freq: Optional[float] = None
vert_zc_freq: Optional[float] = None
long_zc_freq: Optional[float] = None
mic_zc_freq: Optional[float] = None
# Project strings
project_info: IdfProjectInfo = field(default_factory=IdfProjectInfo)
# Sensor self-check
sensor_check: IdfSensorCheck = field(default_factory=IdfSensorCheck)
# Device-fixed
firmware_version: Optional[str] = None
calibration_text: Optional[str] = None
battery_volts: Optional[float] = None
# The full parsed report — preserves anything not surfaced as a typed field
report: IdfReport = field(default_factory=IdfReport)
@classmethod
def from_report(
cls,
report: Any,
filename: str,
) -> "IdfEvent":
"""Build an IdfEvent from a parsed report (dict or IdfReport) and
the device-native binary filename.
The filename is authoritative for serial + timestamp + kind:
Thor's filenames are literal ``<SERIAL>_<YYYYMMDDHHMMSS>.<KIND>``
and the device's own clock is the canonical event timestamp.
If the report carries an ``event_datetime`` that differs from
what's in the filename, the report wins (it has finer-grained
device-reported time-of-trigger semantics).
"""
from .idf_ascii_report import parse_event_filename
# Normalise input to IdfReport
if isinstance(report, IdfReport):
rep = report
elif isinstance(report, dict):
rep = IdfReport.from_dict(report)
else:
raise TypeError(
f"report must be IdfReport or dict; got {type(report).__name__}"
)
# Filename → (serial, timestamp, kind). Required — fall back to
# report-supplied values only if filename parsing fails.
parsed = parse_event_filename(filename)
if parsed is not None:
fn_serial, fn_ts, fn_kind = parsed
kind = "Histogram" if fn_kind == "IDFH" else "Waveform"
else:
fn_serial = rep.serial_number or "UNKNOWN"
fn_ts = rep.event_datetime or datetime.datetime(1970, 1, 1)
kind = "Waveform" if (rep.event_type or "").lower().startswith("full waveform") else "Histogram"
# Prefer report's event_datetime (device-authoritative) over the filename.
ts = rep.event_datetime or fn_ts
serial = rep.serial_number or fn_serial
return cls(
serial=serial,
timestamp=ts,
kind=kind,
filename=filename,
sample_rate=rep.sample_rate,
record_time_sec=rep.record_time_sec,
pre_trigger_sec=rep.pre_trigger_sec,
peaks=IdfPeaks(
transverse_ips = rep.tran_ppv,
vertical_ips = rep.vert_ppv,
longitudinal_ips = rep.long_ppv,
peak_vector_sum_ips = rep.peak_vector_sum,
mic_pspl_dbl = rep.mic_pspl_dbl,
),
tran_zc_freq=rep.tran_zc_freq,
vert_zc_freq=rep.vert_zc_freq,
long_zc_freq=rep.long_zc_freq,
mic_zc_freq=rep.mic_zc_freq,
project_info=IdfProjectInfo(
project=rep.project,
client=rep.client,
operator=rep.operator,
notes=rep.notes,
setup=rep.setup,
),
sensor_check=IdfSensorCheck(
tran=rep.tran_test_passed,
vert=rep.vert_test_passed,
long=rep.long_test_passed,
mic=rep.mic_test_passed,
),
firmware_version=rep.firmware_version,
calibration_text=rep.calibration_text,
battery_volts=rep.battery_volts,
report=rep,
)
# ── Bridge to minimateplus shape (for the existing DB / sidecar paths) ──
def to_minimateplus_event(self, waveform_key: bytes) -> Any:
"""Project this Thor event into the shape ``minimateplus.Event``
carries, so it can flow through the existing
``SeismoDb.insert_events()`` and ``event_to_sidecar_dict()``
machinery without those code paths needing to know about Thor.
Caveats of the bridge:
- ``mic_ppv`` on the produced Event carries Thor's dB(L) value
verbatim — the UI distinguishes via the ``device_family``
column (Phase 1). Don't run the BW psi→dBL converter on
Series IV rows.
- Many Thor-specific fields (Peak Acceleration / Displacement,
sensor self-check, calibration) don't have a slot in
``Event``. The full IdfReport is preserved on the
``.sfm.json`` sidecar under ``extensions.idf_report`` via
``save_imported_idf`` — that's the source of truth for them.
"""
from minimateplus.models import (
Event, PeakValues, ProjectInfo, Timestamp,
)
ts_obj = Timestamp(
raw=bytes(9),
flag=0,
year=self.timestamp.year,
unknown_byte=0,
month=self.timestamp.month,
day=self.timestamp.day,
hour=self.timestamp.hour,
minute=self.timestamp.minute,
second=self.timestamp.second,
)
pv = PeakValues(
tran=self.peaks.transverse_ips,
vert=self.peaks.vertical_ips,
long=self.peaks.longitudinal_ips,
micl=self.peaks.mic_pspl_dbl, # dB(L) — see caveat above
peak_vector_sum=self.peaks.peak_vector_sum_ips,
)
pi = ProjectInfo(
setup_name=self.project_info.setup,
project=self.project_info.project,
client=self.project_info.client,
operator=self.project_info.operator,
sensor_location=None, # Thor folds location into project string
notes=self.project_info.notes,
)
ev = Event(
index=0,
timestamp=ts_obj,
sample_rate=self.sample_rate,
peak_values=pv,
project_info=pi,
record_type=self.kind,
rectime_seconds=self.record_time_sec,
)
ev._waveform_key = waveform_key
return ev
minimateplus/blastware_file.py
+99
View File
@@ -552,6 +552,105 @@ def classify_frame(frame: S3Frame) -> str:
# ── Waveform file writer ───────────────────────────────────────────────────────────
def extract_body_bytes(a5_frames):
"""Reconstruct the Blastware-file body bytes from a list of A5 frames.
Returns ``(strt, body, footer)`` where:
- ``strt`` is the 21-byte STRT record from the probe frame (or a fallback
record built from minimal event metadata if STRT is missing).
- ``body`` is the variable-length sample-data section (between STRT and
the 26-byte file footer). Empty if no frames decode.
- ``footer`` is the 26-byte file footer.
This is the same body-construction algorithm used by :func:`write_blastware_file`
— refactored out so the body decoder (``waveform_codec.decode_waveform_v2``)
can consume the same bytes without re-implementing the frame-walking logic.
Returns ``(b"", b"", b"")`` if *a5_frames* is empty.
"""
if not a5_frames:
return (b"", b"", b"")
# ── Extract STRT record from probe frame ─────────────────────────────────
w0_raw = bytes(a5_frames[0].data[7:])
w0_stripped = _strip_inner_frame_dles(w0_raw)
strt_pos_stripped = w0_stripped.find(b"STRT")
if strt_pos_stripped >= 0:
strt = bytes(w0_stripped[strt_pos_stripped : strt_pos_stripped + 21])
# Walk raw bytes to find the raw-domain end of the STRT (= body start).
target_stripped = strt_pos_stripped + 21
stripped_so_far = 0
raw_i = 0
while stripped_so_far < target_stripped and raw_i < len(w0_raw):
if (w0_raw[raw_i] == 0x10
and raw_i + 1 < len(w0_raw)
and w0_raw[raw_i + 1] in {0x02, 0x03, 0x04}):
raw_i += 2
else:
raw_i += 1
stripped_so_far += 1
probe_skip = 7 + raw_i
else:
strt = b"STRT" + b"\xff\xfe" + bytes(14) + b"\x00"
probe_skip = 7 + 21
if len(strt) != 21:
return (b"", b"", b"")
# Separate terminator from data frames.
term_idx: Optional[int] = None
if a5_frames and a5_frames[-1].page_key != 0x0010:
term_idx = len(a5_frames) - 1
if term_idx is not None:
body_frames = a5_frames[:term_idx]
term_frame = a5_frames[term_idx]
else:
body_frames = a5_frames
term_frame = None
all_bytes = bytearray()
for fi, frame in enumerate(body_frames):
if fi == 0:
skip = probe_skip
elif fi in (1, 2):
skip = 13 # metadata pages
else:
skip = 12 # sample chunks
all_bytes.extend(_frame_body_bytes(frame, skip))
if term_frame is not None:
all_bytes.extend(_frame_body_bytes(term_frame, 11))
# Find the first valid `0e 08` footer marker.
footer_pos = -1
pos = 0
while True:
pos = bytes(all_bytes).find(b"\x0e\x08", pos)
if pos < 0 or pos + 26 > len(all_bytes):
break
yr = (all_bytes[pos + 4] << 8) | all_bytes[pos + 5]
if 2015 <= yr <= 2050:
footer_pos = pos
break
pos += 1
if footer_pos >= 0:
body = bytes(all_bytes[:footer_pos])
footer = bytes(all_bytes[footer_pos : footer_pos + 26])
elif len(all_bytes) >= 26:
body = bytes(all_bytes[:-26])
footer = bytes(all_bytes[-26:])
else:
body = bytes(all_bytes)
footer = b""
return (strt, body, footer)
def write_blastware_file(
event: Event,
a5_frames: list[S3Frame],
minimateplus/bw_ascii_report.py
+522
View File
@@ -0,0 +1,522 @@
"""
minimateplus/bw_ascii_report.py — parser for Blastware's per-event ASCII
report (the .TXT file BW writes alongside each saved event binary).
The ASCII export is the authoritative source for every "rich" per-event
field that BW computes from the waveform but never persists in the BW
binary itself:
- Per-channel PPV (Tran / Vert / Long / MicL)
- Peak Vector Sum + Peak Vector Sum Time
- Per-channel ZC Freq, Time of Peak, Peak Acceleration, Peak Displacement
- MicL PSPL, MicL Time of Peak, MicL ZC Freq
- Per-channel Sensor Self-Check (Test Freq / Test Ratio / Test Results)
- MicL Test Amplitude (mV)
- Battery, calibration date, monitor-log timestamps
Persisting these values into the SFM database lets the monthly-summary
review workflow ("show me events at Location X with PVS > 0.5") work
without depending on the (still-undecoded) waveform body codec.
Format (verified against decode-re/5-8-26 4-event bundle):
- One field per line, wrapped in double quotes: `"Field Name : Value"`
- Field/value separator: literal ` : ` (space-colon-space).
- Some field names contain an internal `:` already (e.g. `"Project:"`),
so we split on the FIRST ` : ` only.
- Some fields have unit suffixes: `"0.500 in/s"` / `"7.5 Hz"` / `"533 mv"`.
- A `"Monitor Log(s)"` marker line is followed by tab-separated rows
of `start_time<TAB>stop_time<TAB>description`.
- Final `"PC SW Version : ..."` line ends the metadata block.
- A blank line separates metadata from the sample table.
- Sample table starts with ` Tran <TAB> Vert <TAB>...`, then
one row per sample (tab-separated, right-padded numeric values).
- Geo channel values are in in/s; MicL in dB(L) (or 0.000 below threshold).
Because some metadata fields have whitespace quirks ("MicL Time of
Peak" has two spaces; the leading "Project:" value has its own colon),
we normalise whitespace in the key before lookup.
"""
from __future__ import annotations
import datetime
import re
from dataclasses import dataclass, field
from pathlib import Path
from typing import Dict, List, Optional, Tuple, Union
# ─────────────────────────────────────────────────────────────────────────────
# Output dataclasses
# ─────────────────────────────────────────────────────────────────────────────
@dataclass
class ChannelStats:
"""Per-channel derived stats, populated from an event report."""
ppv_ips: Optional[float] = None # in/s (geo channels only)
zc_freq_hz: Optional[float] = None # Hz
time_of_peak_s: Optional[float] = None # seconds (relative to trigger; can be negative)
peak_accel_g: Optional[float] = None # g (geo channels only)
peak_disp_in: Optional[float] = None # in (geo channels only)
@dataclass
class MicStats:
"""MicL-specific stats."""
weighting: Optional[str] = None # e.g. "Linear Weighting"
pspl_dbl: Optional[float] = None # dB(L)
zc_freq_hz: Optional[float] = None
time_of_peak_s: Optional[float] = None
@dataclass
class SensorCheck:
"""Per-channel sensor self-check result.
Geo channels report a frequency + ratio; MicL reports a frequency +
amplitude (mV). All channels also have a Pass/Fail string.
"""
test_freq_hz: Optional[float] = None
test_ratio: Optional[float] = None # geo channels only
test_amplitude_mv: Optional[float] = None # MicL only
test_results: Optional[str] = None # "Passed" / "Failed"
@dataclass
class MonitorLogEntry:
"""One row of the trailing Monitor Log(s) block."""
start_time: Optional[datetime.datetime] = None
stop_time: Optional[datetime.datetime] = None
description: Optional[str] = None
@dataclass
class BwAsciiReport:
"""Structured representation of one BW per-event ASCII export."""
# ── Identity ─────────────────────────────────────────────────────────────
event_type: Optional[str] = None # e.g. "Full Waveform"
serial: Optional[str] = None # e.g. "BE11529"
version: Optional[str] = None # firmware version line
file_name: Optional[str] = None # e.g. "M529LK44.AB0"
event_datetime: Optional[datetime.datetime] = None # parsed from Event Time + Event Date
# ── Trigger / recording config ──────────────────────────────────────────
trigger_channel: Optional[str] = None # e.g. "Vert" or "From Unit"
geo_trigger_level_ips: Optional[float] = None
pretrig_s: Optional[float] = None # negative seconds
record_time_s: Optional[float] = None
record_stop_mode: Optional[str] = None
sample_rate_sps: Optional[int] = None
battery_volts: Optional[float] = None
calibration_date: Optional[datetime.date] = None
calibration_by: Optional[str] = None # e.g. "Instantel"
units: Optional[str] = None # e.g. "in/s and dB(L)"
# ── Operator-supplied metadata ──────────────────────────────────────────
# Parsed by POSITION from the 4-line "User Notes" block BW writes
# between the `Units :` and `Geo Range :` lines. Position-based so
# the values populate correctly even when an operator renames the
# labels in Blastware's Compliance Setup → Notes tab (the 4 labels
# are user-editable, e.g. "Seis Loc:" → "Building:" → "Site Address:").
# The original labels BW wrote are preserved in `user_note_labels`
# so terra-view can render them as the operator named them.
project: Optional[str] = None # position 1 (BW default label "Project:")
client: Optional[str] = None # position 2 (BW default label "Client:")
operator: Optional[str] = None # position 3 (BW default label "User Name:")
sensor_location: Optional[str] = None # position 4 (BW default label "Seis Loc:")
# Maps canonical slot name → the literal label BW wrote in the ASCII
# export. Empty if the User Notes block wasn't present. Example
# when the operator renamed slot 4 to "Building:":
# {"project": "Project:", "client": "Client:",
# "operator": "User Name:", "sensor_location": "Building:"}
user_note_labels: Dict[str, str] = field(default_factory=dict)
# ── Geo channel scaling ─────────────────────────────────────────────────
geo_range_ips: Optional[float] = None # 10.000 / 1.250
# ── Per-channel derived stats (geo + mic) ───────────────────────────────
channels: Dict[str, ChannelStats] = field(default_factory=dict)
mic: MicStats = field(default_factory=MicStats)
# ── Vector sum ──────────────────────────────────────────────────────────
peak_vector_sum_ips: Optional[float] = None
peak_vector_sum_time_s: Optional[float] = None
# ── Sensor self-check (per channel) ─────────────────────────────────────
sensor_check: Dict[str, SensorCheck] = field(default_factory=dict)
# ── Monitor log + tooling version ───────────────────────────────────────
monitor_log: List[MonitorLogEntry] = field(default_factory=list)
pc_sw_version: Optional[str] = None
# ── Sample table (optional; only parsed if requested) ───────────────────
# Each entry: (Tran, Vert, Long, MicL) in the report's units (geo
# channels in in/s, MicL in dB(L)). None when parse_samples=False.
samples: Optional[List[Tuple[float, float, float, float]]] = None
# ─────────────────────────────────────────────────────────────────────────────
# Helpers
# ─────────────────────────────────────────────────────────────────────────────
_KEY_NORMALISE_RE = re.compile(r"\s+")
_NUMERIC_RE = re.compile(r"^-?\d+(?:\.\d+)?")
def _normalise_key(k: str) -> str:
"""Collapse whitespace runs (incl. tabs) and strip — handles BW's
"MicL Time of Peak" double-space and leading-colon quirks."""
return _KEY_NORMALISE_RE.sub(" ", k).strip()
def _strip_quotes(line: str) -> str:
line = line.rstrip("\r\n")
if len(line) >= 2 and line.startswith('"') and line.endswith('"'):
return line[1:-1]
return line
def _parse_number(value: str) -> Optional[float]:
"""Pull the leading numeric portion out of a value like "0.500 in/s"."""
m = _NUMERIC_RE.match(value.strip())
if not m:
return None
try:
return float(m.group(0))
except ValueError:
return None
def _parse_int(value: str) -> Optional[int]:
n = _parse_number(value)
return None if n is None else int(round(n))
# Months exactly as BW writes them.
_MONTHS = {
"January": 1, "February": 2, "March": 3, "April": 4,
"May": 5, "June": 6, "July": 7, "August": 8,
"September": 9, "October": 10, "November": 11, "December": 12,
# Short forms used in monitor-log rows ("Apr 23 /26").
"Jan": 1, "Feb": 2, "Mar": 3, "Apr": 4, "Jun": 6, "Jul": 7,
"Aug": 8, "Sep": 9, "Oct": 10, "Nov": 11, "Dec": 12,
}
def _parse_event_date(s: str) -> Optional[datetime.date]:
"""Parse "April 23, 2026" or "May 8, 2026" → date."""
s = s.strip()
parts = s.replace(",", " ").split()
if len(parts) < 3:
return None
month_name, day_str, year_str = parts[0], parts[1], parts[2]
month = _MONTHS.get(month_name)
if month is None:
return None
try:
return datetime.date(int(year_str), month, int(day_str))
except ValueError:
return None
def _parse_event_time(s: str) -> Optional[datetime.time]:
"""Parse "15:56:35" → time."""
s = s.strip()
try:
h, m, sec = s.split(":")
return datetime.time(int(h), int(m), int(sec))
except (ValueError, IndexError):
return None
def _parse_calibration(value: str) -> Tuple[Optional[datetime.date], Optional[str]]:
"""Parse "April 29, 2025 by Instantel" → (date, "Instantel")."""
parts = value.split(" by ", 1)
date = _parse_event_date(parts[0])
by = parts[1].strip() if len(parts) > 1 else None
return date, by
def _parse_monitor_row(line: str) -> Optional[MonitorLogEntry]:
"""Parse a tab-separated monitor log row.
Format: `<start>\t<stop>\t<desc>` where each timestamp is BW's
short form "Mon DD /YY HH:MM:SS" (e.g. "Apr 23 /26 15:46:16").
Year is encoded as a 2-digit suffix; we expand "/26" → 2026.
"""
parts = line.split("\t")
if len(parts) < 2:
return None
start = _parse_monitor_ts(parts[0])
stop = _parse_monitor_ts(parts[1])
desc = parts[2].strip() if len(parts) > 2 else None
if start is None and stop is None and not desc:
return None
return MonitorLogEntry(start_time=start, stop_time=stop, description=desc)
def _parse_monitor_ts(s: str) -> Optional[datetime.datetime]:
"""Parse "Apr 23 /26 15:46:16" → datetime."""
s = s.strip()
parts = s.split()
if len(parts) < 4:
return None
month = _MONTHS.get(parts[0])
if month is None:
return None
try:
day = int(parts[1])
# parts[2] looks like "/26" → century-flip to 2026
yy = int(parts[2].lstrip("/"))
year = 2000 + yy if yy < 80 else 1900 + yy
h, m, sec = (int(x) for x in parts[3].split(":"))
return datetime.datetime(year, month, day, h, m, sec)
except (ValueError, IndexError):
return None
# ── User-notes positional slot map ──────────────────────────────────────────
#
# Blastware's Compliance Setup → Notes tab shows four operator-supplied
# fields whose LABELS the operator can rename (see screenshot in
# project archive). Defaults are "Project:" / "Client:" /
# "User Name:" / "Seis Loc:", but an operator using a different
# convention can rename them to anything ("Building:", "Site:",
# "Address:", etc.). The ASCII export reflects whatever the operator
# typed, so label-based matching is fragile.
#
# What IS reliable: BW always writes the 4 user-notes lines in the
# same order, contiguously between the `Units :` line and the
# `Geo Range :` line. We parse them by POSITION and preserve the
# operator's labels in `report.user_note_labels` so terra-view can
# render them as the operator intended.
_USER_NOTE_SLOTS = ("project", "client", "operator", "sensor_location")
# ─────────────────────────────────────────────────────────────────────────────
# Top-level parser
# ─────────────────────────────────────────────────────────────────────────────
def parse_report(text: Union[str, bytes], *, parse_samples: bool = False) -> BwAsciiReport:
"""Parse a BW per-event ASCII export into a structured BwAsciiReport.
Set ``parse_samples=True`` to also populate ``report.samples`` with
the trailing sample table. Default False because the table is
huge and most callers only want metadata for indexing.
"""
if isinstance(text, bytes):
text = text.decode("ascii", errors="replace")
report = BwAsciiReport()
# Pre-create channel stat slots so callers can rely on them existing.
for ch in ("Tran", "Vert", "Long", "MicL"):
report.channels.setdefault(ch, ChannelStats())
report.sensor_check.setdefault(ch, SensorCheck())
lines = text.splitlines()
i = 0
n = len(lines)
in_monitor_log_section = False
event_time_str: Optional[str] = None
event_date: Optional[datetime.date] = None
# User-notes block detection. We enter the block after parsing
# the "Units :" line and exit on the "Geo Range :" line. Inside,
# the first 4 unmatched `<label> : <value>` lines are assigned to
# the 4 canonical operator-supplied slots by POSITION (project,
# client, operator, sensor_location) regardless of what the
# operator named the labels in BW's Compliance Setup → Notes tab.
in_user_notes_block = False
user_note_position = 0
while i < n:
raw_line = lines[i]
i += 1
# Blank line marks the start of the sample table.
if raw_line.strip() == "":
break
line = _strip_quotes(raw_line)
# Monitor log section: "Monitor Log(s)" header followed by N rows
# (still inside double-quoted lines), terminated by a non-row line
# like "PC SW Version : ..." or a blank line.
if not in_monitor_log_section and line.strip() == "Monitor Log(s)":
in_monitor_log_section = True
continue
if in_monitor_log_section:
# Heuristic: monitor rows contain a tab; the next "Field : Value"
# line ends the section.
if "\t" in line:
entry = _parse_monitor_row(line)
if entry:
report.monitor_log.append(entry)
continue
# Falls through to the field parser below; clear the flag.
in_monitor_log_section = False
# "Field : Value" — split on FIRST occurrence of " : "
idx = line.find(" : ")
if idx < 0:
continue
key = _normalise_key(line[:idx])
value = line[idx + 3 :].strip()
# ── Identity / config ────────────────────────────────────────────────
if key == "Event Type": report.event_type = value
elif key == "Serial Number": report.serial = value
elif key == "Version": report.version = value
elif key == "File Name": report.file_name = value
elif key == "Event Time": event_time_str = value
elif key == "Event Date": event_date = _parse_event_date(value)
elif key == "Trigger": report.trigger_channel = value
elif key == "Geo Trigger Level": report.geo_trigger_level_ips = _parse_number(value)
elif key == "Pre-trigger Length": report.pretrig_s = _parse_number(value)
elif key == "Record Time": report.record_time_s = _parse_number(value)
elif key == "Record Stop Mode": report.record_stop_mode = value
elif key == "Sample Rate": report.sample_rate_sps = _parse_int(value)
elif key == "Battery Level": report.battery_volts = _parse_number(value)
elif key == "Calibration":
report.calibration_date, report.calibration_by = _parse_calibration(value)
elif key == "Units":
report.units = value
# Entering the user-notes block. Next ~4 lines until
# "Geo Range :" are the operator-supplied notes.
in_user_notes_block = True
user_note_position = 0
elif key == "Geo Range":
# Exiting the user-notes block.
in_user_notes_block = False
report.geo_range_ips = _parse_number(value)
# User-notes block: assign by position (operator may have
# renamed the labels, so we don't trust them). Preserve the
# original labels in `user_note_labels` for downstream UIs
# (terra-view) that want to display them as the operator
# named them.
elif in_user_notes_block and user_note_position < len(_USER_NOTE_SLOTS):
slot = _USER_NOTE_SLOTS[user_note_position]
setattr(report, slot, value)
report.user_note_labels[slot] = key
user_note_position += 1
# ── Per-channel stats ────────────────────────────────────────────────
# All match the pattern "{Channel} <stat-name>"
elif key in (
"Tran PPV", "Vert PPV", "Long PPV",
"Tran ZC Freq", "Vert ZC Freq", "Long ZC Freq",
"Tran Time of Peak", "Vert Time of Peak", "Long Time of Peak",
"Tran Peak Acceleration", "Vert Peak Acceleration", "Long Peak Acceleration",
"Tran Peak Displacement", "Vert Peak Displacement", "Long Peak Displacement",
):
ch_name, stat = key.split(" ", 1)
cs = report.channels.setdefault(ch_name, ChannelStats())
num = _parse_number(value)
if stat == "PPV": cs.ppv_ips = num
elif stat == "ZC Freq": cs.zc_freq_hz = num
elif stat == "Time of Peak": cs.time_of_peak_s = num
elif stat == "Peak Acceleration": cs.peak_accel_g = num
elif stat == "Peak Displacement": cs.peak_disp_in = num
# ── Vector Sum ───────────────────────────────────────────────────────
elif key == "Peak Vector Sum":
report.peak_vector_sum_ips = _parse_number(value)
elif key == "Peak Vector Sum Time":
report.peak_vector_sum_time_s = _parse_number(value)
# ── Microphone block ────────────────────────────────────────────────
elif key == "Microphone":
report.mic.weighting = value
elif key == "MicL PSPL":
report.mic.pspl_dbl = _parse_number(value)
# Mirror onto the "MicL" entry in channels so callers querying
# `channels["MicL"].ppv_ips` see something — but it's dB(L), not
# in/s, so we store as-is in the MicStats and mark the channel.
elif key == "MicL Time of Peak":
report.mic.time_of_peak_s = _parse_number(value)
cs = report.channels.setdefault("MicL", ChannelStats())
cs.time_of_peak_s = report.mic.time_of_peak_s
elif key == "MicL ZC Freq":
report.mic.zc_freq_hz = _parse_number(value)
cs = report.channels.setdefault("MicL", ChannelStats())
cs.zc_freq_hz = report.mic.zc_freq_hz
# ── Sensor self-check ────────────────────────────────────────────────
elif key in (
"Tran Test Freq", "Vert Test Freq", "Long Test Freq", "MicL Test Freq",
"Tran Test Ratio", "Vert Test Ratio", "Long Test Ratio",
"MicL Test Amplitude",
"Tran Test Results", "Vert Test Results", "Long Test Results", "MicL Test Results",
):
ch_name, stat = key.split(" ", 1)
sc = report.sensor_check.setdefault(ch_name, SensorCheck())
if stat == "Test Freq": sc.test_freq_hz = _parse_number(value)
elif stat == "Test Ratio": sc.test_ratio = _parse_number(value)
elif stat == "Test Amplitude": sc.test_amplitude_mv = _parse_number(value)
elif stat == "Test Results": sc.test_results = value
# ── Trailer ─────────────────────────────────────────────────────────
elif key == "PC SW Version":
report.pc_sw_version = value
# Unknown keys are silently dropped — forward-compat for future
# BW versions that may add fields.
# Combine event date + time into a datetime
if event_date is not None and event_time_str is not None:
t = _parse_event_time(event_time_str)
if t is not None:
report.event_datetime = datetime.datetime.combine(event_date, t)
if parse_samples:
report.samples = _parse_sample_table(lines, i)
return report
def _parse_sample_table(
lines: List[str], start: int,
) -> List[Tuple[float, float, float, float]]:
"""Parse the trailing sample table.
The table starts with a header row (" Tran <TAB>...") and continues
until EOF. Each data row is a tab-separated quartet of numeric values.
"""
samples: List[Tuple[float, float, float, float]] = []
seen_header = False
for line in lines[start:]:
line = line.rstrip("\r\n")
if not line.strip():
continue
cols = [c.strip() for c in line.split("\t") if c.strip()]
if not seen_header:
# Header row contains channel names; numeric rows don't.
if any(c in ("Tran", "Vert", "Long", "MicL") for c in cols):
seen_header = True
continue
if len(cols) < 4:
continue
try:
samples.append((
float(cols[0]), float(cols[1]),
float(cols[2]), float(cols[3]),
))
except ValueError:
continue
return samples
def parse_report_file(
path: Union[str, Path], *, parse_samples: bool = False,
) -> BwAsciiReport:
"""Convenience: read a .TXT file from disk and parse it."""
return parse_report(Path(path).read_bytes(), parse_samples=parse_samples)
minimateplus/client.py
+58 -25
View File
@@ -1362,20 +1362,6 @@ def _decode_waveform_record_into(data: bytes, event: Event) -> None:
Modifies event in-place.
"""
# ── Always preserve the raw 210 bytes ─────────────────────────────────────
# The 0C record carries far more than just peaks + project strings:
# ZC Freq, Time of Peak, Peak Acceleration, Peak Displacement, Vector
# Sum Time, MicL Time of Peak, and the per-channel sensor self-check
# results (Test Freq / Ratio / Pass-Fail) all live somewhere in this
# 210-byte block. Their byte offsets are not yet mapped — keeping the
# raw bytes lets us decode those fields offline once we have a paired
# (raw 0C, BW-report) sample to fit against. Cheap to keep around
# (210 bytes per event).
try:
event._raw_record = bytes(data[:210])
except Exception:
pass
# ── Record type + format detection ────────────────────────────────────────
# `record_type` is the user-facing label ("Waveform" for any triggered
# event regardless of timestamp-header layout). `fmt` is the internal
@@ -1514,22 +1500,69 @@ def _decode_a5_waveform(
(BULK_WAVEFORM_STREAM) frame payloads and populate event.raw_samples,
event.total_samples, event.pretrig_samples, and event.rectime_seconds.
This requires ALL A5 frames (stop_after_metadata=False), not just the
metadata-bearing subset.
Wired up 2026-05-11 to the verified ``decode_waveform_v2`` codec (see
``minimateplus/waveform_codec.py`` and ``docs/waveform_codec_re_status.md``).
Replaces the legacy int16 LE decoder, which produced full-scale ±32K
noise on every event because the body bytes are encoded, not raw
samples.
── Waveform format (confirmed from 4-2-26 blast capture) ───────────────────
The blast waveform is 4-channel interleaved signed 16-bit little-endian,
8 bytes per sample-set:
Output convention (preserved from the legacy decoder):
``event.raw_samples`` is a dict with keys "Tran", "Vert", "Long",
"MicL" mapping to lists of **int16 ADC counts**. Multiply by
``geo_range / 32768`` for geo channels to get in/s; use
:func:`minimateplus.waveform_codec.mic_count_to_db` for mic dB(L).
``total_samples`` / ``pretrig_samples`` / ``rectime_seconds`` are set
to ``None`` so the caller backfills from compliance_config (the
authoritative source — STRT fields aren't reliable).
"""
from .waveform_codec import decode_a5_frames
event.total_samples = None
event.pretrig_samples = None
event.rectime_seconds = None
if not frames_data:
log.debug("_decode_a5_waveform: no frames provided")
return
decoded = decode_a5_frames(frames_data)
if decoded is None:
log.warning("_decode_a5_waveform: codec returned no samples")
return
event.raw_samples = decoded
log.debug(
"_decode_a5_waveform: decoded %d/%d/%d/%d samples (T/V/L/M)",
len(decoded.get("Tran", [])),
len(decoded.get("Vert", [])),
len(decoded.get("Long", [])),
len(decoded.get("MicL", [])),
)
def _decode_a5_waveform_LEGACY(
frames_data: list[S3Frame],
event: Event,
) -> None:
"""
LEGACY decoder — kept for reference only. DO NOT CALL.
This is the int16 LE decoder that produced full-scale ±32K noise
on every event. Retracted 2026-05-08; replaced 2026-05-11 with
the verified codec in :mod:`minimateplus.waveform_codec`. See
``docs/instantel_protocol_reference.md §7.6.1`` for the full history.
── Waveform format (LEGACY — WRONG) ────────────────────────────────
Claimed 4-channel interleaved signed 16-bit little-endian, 8 bytes
per sample-set:
[T_lo T_hi V_lo V_hi L_lo L_hi M_lo M_hi] × N
where T=Tran, V=Vert, L=Long, M=Mic. Channel ordering follows the
Blastware convention [Tran, Vert, Long, Mic] = [ch0, ch1, ch2, ch3].
where T=Tran, V=Vert, L=Long, M=Mic.
⚠️ Channel ordering is a confirmed CONVENTION — the physical ordering on
the ADC mux is not independently verifiable from the saturating blast
captures we have. The convention is consistent with Blastware labeling
(Tran is always the first channel field in the A5 STRT+waveform stream).
The body bytes are actually a tagged delta+RLE stream — this
interpretation was wrong.
── Frame structure ──────────────────────────────────────────────────────────
A5[0] (probe response):
minimateplus/event_file_io.py
+340 -32
View File
@@ -15,7 +15,6 @@ declared in `event_to_sidecar_dict()`.
from __future__ import annotations
import base64
import datetime
import hashlib
import json
@@ -27,6 +26,14 @@ from typing import Optional, Union
from .models import Event, PeakValues, ProjectInfo, Timestamp
from . import blastware_file as _bw # avoid circular reference at module load
from .bw_ascii_report import BwAsciiReport
from .waveform_codec import decode_waveform_v2, decoded_to_adc_counts
from .histogram_codec import decode_histogram_body
# Reference pressure for dB(L) → psi conversion (20 µPa expressed in psi).
# Same constant as sfm/sfm_webapp.html so server-side and browser-side
# conversions agree.
_DBL_REF_PSI = 2.9e-9
log = logging.getLogger(__name__)
@@ -42,7 +49,7 @@ SIDECAR_KIND = "sfm.event"
# bumped without a `pip install` re-run — leading to confusing stale
# version stamps in sidecars. Bump this constant and CHANGELOG.md
# together at release time.
TOOL_VERSION = "0.15.0"
TOOL_VERSION = "0.20.0"
try:
# Best-effort: prefer the installed metadata when it's NEWER than the
@@ -95,6 +102,158 @@ def _peak_values_to_dict(pv: Optional[PeakValues]) -> dict:
}
def _bw_report_to_dict(report: BwAsciiReport) -> dict:
"""Project a parsed BW ASCII report into the sidecar's `bw_report` block.
All fields are rendered as plain JSON-compatible types (no datetime
objects). Channels are uniformly lowercased for stable JSON keys.
"""
def _ch(ch_name: str) -> dict:
cs = report.channels.get(ch_name)
if cs is None:
return {}
out = {
"ppv_ips": cs.ppv_ips,
"zc_freq_hz": cs.zc_freq_hz,
"time_of_peak_s": cs.time_of_peak_s,
"peak_accel_g": cs.peak_accel_g,
"peak_disp_in": cs.peak_disp_in,
}
# Drop all-None entries — keeps the JSON tidy for partial reports.
return {k: v for k, v in out.items() if v is not None}
def _sc(ch_name: str) -> dict:
sc = report.sensor_check.get(ch_name)
if sc is None:
return {}
out = {
"freq_hz": sc.test_freq_hz,
"ratio": sc.test_ratio,
"amplitude_mv": sc.test_amplitude_mv,
"result": sc.test_results,
}
return {k: v for k, v in out.items() if v is not None}
monitor_log = []
for entry in report.monitor_log:
e = {
"start": entry.start_time.isoformat() if entry.start_time else None,
"stop": entry.stop_time.isoformat() if entry.stop_time else None,
"description": entry.description,
}
monitor_log.append({k: v for k, v in e.items() if v is not None})
return {
"available": True,
"event_type": report.event_type,
"version": report.version,
"trigger": {
"channel": report.trigger_channel,
"geo_level_ips": report.geo_trigger_level_ips,
},
"recording": {
"sample_rate_sps": report.sample_rate_sps,
"record_time_s": report.record_time_s,
"pretrig_s": report.pretrig_s,
"stop_mode": report.record_stop_mode,
"geo_range_ips": report.geo_range_ips,
"units": report.units,
},
"device": {
"battery_volts": report.battery_volts,
"calibration_date": report.calibration_date.isoformat() if report.calibration_date else None,
"calibration_by": report.calibration_by,
},
"peaks": {
"tran": _ch("Tran"),
"vert": _ch("Vert"),
"long": _ch("Long"),
"vector_sum": {
"ips": report.peak_vector_sum_ips,
"time_s": report.peak_vector_sum_time_s,
},
},
"mic": {
"weighting": report.mic.weighting,
"pspl_dbl": report.mic.pspl_dbl,
"zc_freq_hz": report.mic.zc_freq_hz,
"time_of_peak_s": report.mic.time_of_peak_s,
},
"sensor_check": {
"tran": _sc("Tran"),
"vert": _sc("Vert"),
"long": _sc("Long"),
"mic": _sc("MicL"),
},
"monitor_log": monitor_log,
"pc_sw_version": report.pc_sw_version,
}
def _dbl_to_psi(pspl_dbl: float) -> float:
"""Convert dB(L) sound pressure level back to psi. Uses the same
20 µPa reference (= 2.9e-9 psi) as the webapp so server-side and
browser-side conversions agree."""
return _DBL_REF_PSI * (10.0 ** (pspl_dbl / 20.0))
def apply_report_to_event(event: Event, report: BwAsciiReport) -> None:
"""Overlay device-authoritative fields from a parsed BW ASCII report
onto an in-memory Event, IN-PLACE.
Why this exists
───────────────
`read_blastware_file()` parses the BW binary and fills `Event.peak_values`
via `_peaks_from_samples()` — which runs the (still-undecoded) BW body
codec assuming raw int16 LE and produces ±32K-shaped noise on every
channel. Result: peak values land in the SeismoDb event row as
~10 in/s on every event regardless of the actual signal.
When a paired BW ASCII report is available, the report carries the
device's own authoritative peak / project / sample-rate / record-time
values. This helper folds those onto the Event before it flows to
`SeismoDb.insert_events()`, so the DB columns reflect the report
rather than the broken-codec output.
Fields overlaid (only when the report supplies a non-None value):
- peak_values.tran / .vert / .long (from report.channels)
- peak_values.peak_vector_sum (from report.peak_vector_sum_ips)
- peak_values.micl (psi) (from report.mic.pspl_dbl → psi)
- project_info.project / .client / .operator / .sensor_location
- sample_rate (from report.sample_rate_sps)
- rectime_seconds (from report.record_time_s)
Fields NOT touched (operator-edit / parser-output preserved):
- timestamp, raw_samples, record_type, total_samples,
pretrig_samples, _waveform_key, _a5_frames, _raw_record
- false_trigger and review state (those live on the sidecar, not on Event)
"""
if event.peak_values is None:
event.peak_values = PeakValues()
pv = event.peak_values
ch = report.channels
if (t := ch.get("Tran")) and t.ppv_ips is not None: pv.tran = t.ppv_ips
if (v := ch.get("Vert")) and v.ppv_ips is not None: pv.vert = v.ppv_ips
if (l := ch.get("Long")) and l.ppv_ips is not None: pv.long = l.ppv_ips
if report.peak_vector_sum_ips is not None:
pv.peak_vector_sum = report.peak_vector_sum_ips
if report.mic.pspl_dbl is not None and report.mic.pspl_dbl > 0:
pv.micl = _dbl_to_psi(report.mic.pspl_dbl)
if event.project_info is None:
event.project_info = ProjectInfo()
pi = event.project_info
if report.project: pi.project = report.project
if report.client: pi.client = report.client
if report.operator: pi.operator = report.operator
if report.sensor_location: pi.sensor_location = report.sensor_location
if report.sample_rate_sps:
event.sample_rate = report.sample_rate_sps
if report.record_time_s is not None:
event.rectime_seconds = report.record_time_s
def _project_info_to_dict(pi: Optional[ProjectInfo]) -> dict:
if pi is None:
return {
@@ -124,49 +283,104 @@ def event_to_sidecar_dict(
captured_at: Optional[datetime.datetime] = None,
review: Optional[dict] = None,
extensions: Optional[dict] = None,
bw_report: Optional[BwAsciiReport] = None,
) -> dict:
"""
Build a v1 sidecar dict from an Event + the surrounding metadata.
Pure helper — no file I/O. Callers stitch the result into a sidecar
via `write_sidecar()` (or POST it back via the PATCH endpoint).
When *bw_report* is supplied (e.g. by the ACH-forwarded import path
where Blastware writes a per-event ASCII report alongside the binary),
its decoded fields are folded into the sidecar:
- A new top-level ``bw_report`` block carries the rich derived
per-channel stats (Peak Acceleration, Peak Displacement, ZC Freq,
Time of Peak), the Peak Vector Sum + time, the per-channel sensor
self-check results, and monitor-log timestamps.
- ``peak_values`` is overlaid from the report (the report's PPV/PVS
values are computed by the device firmware and are authoritative;
anything ``read_blastware_file()`` derived from samples is
approximate at best until the body codec is decoded).
- ``project_info`` is overlaid from the report when the report
supplies a non-empty value (the report mirrors the device's
compliance config, which is what BW shows in its event report).
- ``event.timestamp`` is overlaid from the report's Event Date +
Event Time (BW's report timestamps are second-resolution and
match the binary's footer; we prefer the report value because
the BW-binary footer timestamp can drift on some firmware).
"""
if source_kind not in {"sfm-live", "sfm-ach", "bw-import"}:
if source_kind not in {"sfm-live", "sfm-ach", "bw-import", "idf-import"}:
raise ValueError(f"unknown source_kind: {source_kind!r}")
captured_at = captured_at or datetime.datetime.utcnow()
# Stash raw 0C record bytes in `extensions.raw_records` so future
# field-decoding work (Peak Acceleration, ZC Freq, Time of Peak,
# sensor self-check results, etc.) can run offline against committed
# sidecars without a live device. Cheap (~280 bytes base64) and
# forward-compatible (older readers ignore unknown extensions keys).
ext_dict: dict = dict(extensions) if extensions else {}
raw_0c = getattr(event, "_raw_record", None)
if raw_0c:
rr = ext_dict.setdefault("raw_records", {})
# Don't clobber a raw_0c that callers explicitly passed in via
# `extensions=...` (e.g. round-trip preservation in patch_sidecar).
rr.setdefault("waveform_record_b64", base64.b64encode(raw_0c).decode("ascii"))
rr.setdefault("waveform_record_len", len(raw_0c))
# ── Overlay event fields from the report when present ───────────────────
timestamp_iso = _ts_iso(event.timestamp)
if bw_report and bw_report.event_datetime:
timestamp_iso = bw_report.event_datetime.isoformat()
return {
"schema_version": SCHEMA_VERSION,
"kind": SIDECAR_KIND,
# Build peak_values, optionally overlaid from the report. The report
# stores Mic peak as PSPL (dB(L)); we convert to psi to match the
# existing peak_values.mic_psi field.
peak_dict = _peak_values_to_dict(event.peak_values)
if bw_report:
ch = bw_report.channels
if (t := ch.get("Tran")) and t.ppv_ips is not None: peak_dict["transverse"] = t.ppv_ips
if (v := ch.get("Vert")) and v.ppv_ips is not None: peak_dict["vertical"] = v.ppv_ips
if (l := ch.get("Long")) and l.ppv_ips is not None: peak_dict["longitudinal"] = l.ppv_ips
if bw_report.peak_vector_sum_ips is not None:
peak_dict["vector_sum"] = bw_report.peak_vector_sum_ips
if bw_report.mic.pspl_dbl is not None and bw_report.mic.pspl_dbl > 0:
peak_dict["mic_psi"] = _dbl_to_psi(bw_report.mic.pspl_dbl)
"event": {
# Project info: overlay from report (the report mirrors the
# session-start compliance config that BW renders in event reports).
proj_dict = _project_info_to_dict(event.project_info)
if bw_report:
if bw_report.project: proj_dict["project"] = bw_report.project
if bw_report.client: proj_dict["client"] = bw_report.client
if bw_report.operator: proj_dict["operator"] = bw_report.operator
if bw_report.sensor_location: proj_dict["sensor_location"] = bw_report.sensor_location
# Event-block fields: overlay from report where available.
event_block = {
"serial": serial,
"timestamp": _ts_iso(event.timestamp),
"timestamp": timestamp_iso,
"waveform_key": event._waveform_key.hex() if event._waveform_key else None,
"record_type": event.record_type,
"sample_rate": event.sample_rate,
"rectime_seconds": event.rectime_seconds,
"total_samples": event.total_samples,
"pretrig_samples": event.pretrig_samples,
},
}
if bw_report:
# Report values are authoritative — they're the user-configured
# values BW reads back, not STRT-derived guesses. In particular
# `event.rectime_seconds` from `read_blastware_file()` reads
# STRT[18] which is actually the `0x46` record-type marker (= 70)
# rather than the user's Record Time setting. Always overwrite.
if bw_report.sample_rate_sps:
event_block["sample_rate"] = bw_report.sample_rate_sps
if bw_report.record_time_s is not None:
event_block["rectime_seconds"] = bw_report.record_time_s
# Derive total_samples + pretrig_samples per channel from the
# report's sample_rate × times. These match the row count of
# the report's sample table (verified: event-c reports 1024 sps
# × (1.0 + 0.25) = 1280 rows).
if (sr := bw_report.sample_rate_sps) and bw_report.record_time_s is not None:
pretrig_s = abs(bw_report.pretrig_s) if bw_report.pretrig_s is not None else 0.0
event_block["total_samples"] = int(round(sr * (bw_report.record_time_s + pretrig_s)))
event_block["pretrig_samples"] = int(round(sr * pretrig_s))
"peak_values": _peak_values_to_dict(event.peak_values),
"project_info": _project_info_to_dict(event.project_info),
out = {
"schema_version": SCHEMA_VERSION,
"kind": SIDECAR_KIND,
"event": event_block,
"peak_values": peak_dict,
"project_info": proj_dict,
"blastware": {
"filename": blastware_filename,
@@ -189,9 +403,14 @@ def event_to_sidecar_dict(
"notes": "",
},
"extensions": ext_dict,
"extensions": extensions or {},
}
if bw_report:
out["bw_report"] = _bw_report_to_dict(bw_report)
return out
# ── Sidecar IO ────────────────────────────────────────────────────────────────
@@ -429,6 +648,50 @@ def _peaks_from_samples(samples: dict[str, list[int]]) -> PeakValues:
)
_RECORD_TYPE_BY_EXT_SUFFIX = {
'H': 'Histogram',
'W': 'Waveform',
'M': 'Manual',
'E': 'Event',
'C': 'Combo',
}
def derive_record_type_from_filename(filename, default: str = "Waveform") -> str:
"""Derive a BW Event's record_type from its filename's extension suffix.
V10.72+ MiniMate Plus firmware encodes the event type as the LAST
character of the extension (the `T` in BW's `AB0T` scheme):
``M529LKIQ.G10H`` → H → ``"Histogram"``
``T350L385.VY0W`` → W → ``"Waveform"``
``...M`` → M → ``"Manual"``
``...E`` → E → ``"Event"``
``...C`` → C → ``"Combo"``
Old S338 firmware uses 3-char extensions ending in ``0`` whose
encoding is not yet known — those fall through to ``default``.
Micromate Series 4 uses a different scheme entirely (observed:
``IDFH``, ``IDFW``) but the LAST-char convention (H / W) still holds
for the type code, so it works for both families.
Returns ``default`` if filename is empty, has no extension, or the
suffix char isn't a recognized type code.
"""
if not filename:
return default
try:
name = Path(filename).name
except (TypeError, ValueError):
return default
if '.' not in name:
return default
ext = name.rsplit('.', 1)[1]
if not ext:
return default
return _RECORD_TYPE_BY_EXT_SUFFIX.get(ext[-1].upper(), default)
def read_blastware_file(path: Union[str, Path]) -> Event:
"""
Parse a Blastware waveform file into an Event.
@@ -494,11 +757,40 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
ts1 = _bw._decode_ts_be(footer[2:10])
ts2 = _bw._decode_ts_be(footer[10:18])
# Body: first 6 bytes are the preamble (00 00 ff ff ff ff). Strip
# them before decoding samples. Any trailing tail past the last
# full sample-set is silently truncated by _decode_samples_4ch.
sample_bytes = body[6:] if body[:6].hex() in ("0000ffffffff", "0000FFFFFFFF") else body
samples = _decode_samples_4ch_int16_le(sample_bytes)
# Body: decode via the verified body codecs. Two formats coexist:
#
# 1. Waveform-mode (.AB0W) — starts with 7-byte preamble
# ``00 02 00 [Tran[0] BE] [Tran[1] BE]`` followed by the
# tagged-block delta stream documented in
# ``docs/waveform_codec_re_status.md`` and §7.6.1 of the
# protocol reference. Decoded by ``waveform_codec.decode_waveform_v2``.
#
# 2. Histogram-mode (.AB0H) — a sequence of 32-byte blocks, one
# per histogram interval, each carrying per-channel peak +
# half-period values. Decoded by
# ``histogram_codec.decode_histogram_body``. Both codecs
# return the same channel-grouped output shape, so consumers
# don't need to special-case mode.
#
# The historical ``_decode_samples_4ch_int16_le`` int16-LE
# interpretation was retracted 2026-05-08 (see protocol-ref §7.6.1
# retraction box) — it produced ±32K noise on every event.
#
# If both codecs fail (malformed file, truncated body, unrecognised
# mode, synthetic test input), fall back to empty channels — the
# rest of the event (timestamp, waveform_key, project strings) is
# still recoverable and useful.
decoded = decode_waveform_v2(body)
if decoded is None:
decoded = decode_histogram_body(body)
if decoded is None:
log.warning(
"%s: body codec failed to decode (body starts %s) — "
"raw_samples will be empty", path, body[:8].hex(" "),
)
samples = {"Tran": [], "Vert": [], "Long": [], "MicL": []}
else:
samples = decoded_to_adc_counts(decoded)
# Metadata strings (label-anchored search across the body).
project = _find_first_string(body, b"Project:")
@@ -510,7 +802,12 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
ev = Event(index=-1)
if strt_fields.get("waveform_key"):
ev._waveform_key = bytes.fromhex(strt_fields["waveform_key"])
ev.record_type = "Waveform"
# Derive record_type from the filename's extension suffix (H/W/M/E/C).
# When called from save_imported_bw the path here is a tmp file with a
# ".bw" suffix, so the derivation falls back to "Waveform" and the
# caller overrides ev.record_type using the original filename — see
# waveform_store.save_imported_bw.
ev.record_type = derive_record_type_from_filename(path.name)
ev.rectime_seconds = strt_fields.get("rectime_seconds")
ev.total_samples = strt_fields.get("total_samples")
ev.pretrig_samples = strt_fields.get("pretrig_samples")
@@ -527,7 +824,18 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
project=project, client=client, operator=user, sensor_location=seisloc,
)
ev.raw_samples = samples
ev.peak_values = _peaks_from_samples(samples)
# Only compute peaks from samples when we actually have samples.
# For events the codec couldn't decode (histogram-mode bodies, until
# the §7.6.2 histogram codec is wired in), samples is an empty dict
# and ``_peaks_from_samples`` would return PeakValues(0, 0, 0, 0, 0).
# That would then OVERWRITE existing good DB peak values (e.g. from
# paired BW ASCII reports) during the backfill UPSERT path.
# Leaving peak_values=None signals "we don't know" to downstream
# consumers; the backfill script seeds from the DB row when it sees
# None, and ``apply_report_to_event`` overlays from a paired ASCII
# report when one is supplied.
has_samples = any(samples.get(ch) for ch in ("Tran", "Vert", "Long", "MicL"))
ev.peak_values = _peaks_from_samples(samples) if has_samples else None
ev._a5_frames = None # not recoverable from BW file
return ev
minimateplus/histogram_codec.py
+232
View File
@@ -0,0 +1,232 @@
"""
histogram_codec.py — decoder for MiniMate Plus histogram-mode event bodies.
FULLY DECODED 2026-05-20. Every field in every block, verified
byte-exact against BW's ASCII export across multiple histogram
fixtures.
The histogram-mode body is a stream of 32-byte fixed-length blocks,
one block per histogram interval. Each block carries the per-interval
peak amplitude + zero-crossing frequency for all four channels (Tran,
Vert, Long, MicL).
────────────────────────────────────────────────────────────────────────────
Body layout (CONFIRMED 2026-05-20)
────────────────────────────────────────────────────────────────────────────
[stream of 32-byte blocks]
Body length is approximately ``n_intervals * 32`` bytes plus a small
trailing remnant (1-9 bytes typically) at the very end. Walker should
iterate 32-stride and stop before the tail.
────────────────────────────────────────────────────────────────────────────
32-byte block layout
────────────────────────────────────────────────────────────────────────────
[0] 0x00 always-zero tag
[1] segment_id (uint8) 0x00..0x03 — 256 blocks per segment
[2:4] block_ctr (uint16 LE) resets each segment (0x0100, 0x0101, …)
[4:6] 0x000a (uint16 LE) constant marker (= 10)
[6:8] T_peak_count uint16 LE Tran peak (count × 0.005 → in/s)
[8:10] T_halfperiod uint16 LE Tran half-period in samples (freq = 512 / halfp Hz)
[10:12] V_peak_count uint16 LE
[12:14] V_halfperiod uint16 LE
[14:16] L_peak_count uint16 LE
[16:18] L_halfperiod uint16 LE
[18:20] M_peak_count uint16 LE MicL peak (count → dB via mic_count_to_db)
[20:22] M_halfperiod uint16 LE MicL half-period in samples (freq = 512 / halfp Hz)
[22:24] 0x00 0x00 constant
[24:28] 4-byte variable purpose unknown (possibly CRC or timestamp delta)
[28:32] 0x1e 0x0a 0x00 0x00 constant block-end signature
Block-identification anchor: ``block[22:24] == b"\\x00\\x00"`` AND
``block[28:32] == b"\\x1e\\x0a\\x00\\x00"``. This is the reliable
distinguisher from non-block content in the file.
────────────────────────────────────────────────────────────────────────────
Per-channel encoding
────────────────────────────────────────────────────────────────────────────
Geophone channels (Tran, Vert, Long):
- peak_count × 0.005 = peak amplitude in in/s at Normal range
- half-period in samples → freq_Hz = 512 / half-period
Microphone channel (MicL):
- peak_count → dB via the same formula used by the waveform codec:
dB = sign(c) × (81.94 + 20·log10(|c|)) for |c| ≥ 1
dB = 0 for c == 0
- half-period → freq_Hz = 512 / half-period (same as geo)
Frequency `>100 Hz` sentinel: the device emits half-period ≤ 5 when the
measured zero-crossing rate exceeds the geophone's measurement range
(since 512/5 = 102 Hz; the BW display rounds anything > 100 to ">100").
────────────────────────────────────────────────────────────────────────────
Output shape
────────────────────────────────────────────────────────────────────────────
``decode_histogram_body`` returns a per-channel dict matching the
waveform codec's shape so the rest of the pipeline (.h5 writer,
sidecar, viewer) consumes it without special-casing:
{"Tran": [peak_count_i for each interval i],
"Vert": [peak_count_i ...],
"Long": [peak_count_i ...],
"MicL": [peak_count_i ...]}
Values are in **16-count units for geo** (LSB = 0.005 in/s, matching
``decode_waveform_v2``) and **1-count units for mic** (matching the
waveform codec's mic convention). Run through
``waveform_codec.decoded_to_adc_counts`` to scale geo to 1-count ADC.
Per-interval frequencies are NOT returned — they're auxiliary data,
not waveform samples. Consumers needing frequencies can call
``decode_histogram_body_full()`` for the structured per-interval
record list.
"""
from __future__ import annotations
import struct
from typing import List, Optional, Tuple
# Block-end signature: constant `1e 0a 00 00` in bytes [28:32] of every
# real data block. More distinctive than the byte-22 `00 00` (which
# matches many false positives), so we anchor on this.
_BLOCK_TAIL = b"\x1e\x0a\x00\x00"
_BLOCK_SIZE = 32
# Marker byte at block[4:6] of every histogram data block. Used as
# additional validation that we're looking at a real block.
_BLOCK_MARKER = 10
# Geo peak scaling: stored as "count × 0.005 in/s" where 1 count = one
# 0.005 in/s display quantum. Equivalent to the waveform codec's
# 16-count-unit output (1 unit = 0.005 in/s = 16 ADC counts).
_GEO_LSB_INS = 0.005
# Frequency formula: freq_Hz = _FREQ_NUMERATOR / half_period_samples.
# Empirically determined to be 512 (= sample_rate / 2, where sample rate
# is 1024 sps for the standard MiniMate Plus configuration).
_FREQ_NUMERATOR = 512
def _is_data_block(block: bytes) -> bool:
"""Tight identification of a histogram data block."""
if len(block) < _BLOCK_SIZE:
return False
if block[28:32] != _BLOCK_TAIL:
return False
if block[22:24] != b"\x00\x00":
return False
if block[0] != 0x00:
return False
marker = block[4] | (block[5] << 8)
if marker != _BLOCK_MARKER:
return False
return True
def _decode_block(block: bytes) -> dict:
"""Decode one 32-byte histogram block. Caller must have validated
with ``_is_data_block`` first."""
# All 16-bit fields are little-endian unsigned. Peak counts are
# always non-negative; half-periods are always positive when valid.
t_peak, t_halfp, v_peak, v_halfp, l_peak, l_halfp, m_peak, m_halfp = struct.unpack_from(
"<HHHHHHHH", block, 6
)
segment_id = block[1]
block_ctr = block[2] | (block[3] << 8)
var_meta = bytes(block[24:28])
return {
"segment_id": segment_id,
"block_ctr": block_ctr,
"t_peak": t_peak,
"t_halfp": t_halfp,
"v_peak": v_peak,
"v_halfp": v_halfp,
"l_peak": l_peak,
"l_halfp": l_halfp,
"m_peak": m_peak,
"m_halfp": m_halfp,
"meta_var": var_meta,
}
def walk_body(body: bytes) -> List[dict]:
"""Walk the body and return one dict per histogram interval.
Iterates 32-byte strides from offset 0. Yields a decoded record
for every block that passes ``_is_data_block`` validation. Stops
when the remaining bytes are too short to form a complete block.
"""
records: List[dict] = []
for off in range(0, len(body) - _BLOCK_SIZE + 1, _BLOCK_SIZE):
blk = body[off:off + _BLOCK_SIZE]
if not _is_data_block(blk):
# Hit non-block content (likely a sync or stream marker).
# Continue walking — block alignment is fixed at 32-stride
# from offset 0, so we don't lose alignment by skipping.
continue
records.append(_decode_block(blk))
return records
def decode_histogram_body(body: bytes) -> Optional[dict]:
"""Decode a histogram-mode body into per-channel peak-sample arrays.
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}``
where each channel's list contains one peak value per histogram
interval (in the same units the waveform codec uses: 16-count units
for geo, 1-count ADC units for mic). Returns ``None`` if the body
doesn't contain any valid histogram blocks.
To convert to physical units:
- Geo channels: ``count * 0.005`` = peak in in/s at Normal range
(or run through ``waveform_codec.decoded_to_adc_counts`` first
to get 1-count ADC values, then ``count / 32767 * 10.0`` for in/s)
- Mic channel: use ``waveform_codec.mic_count_to_db(count)``
"""
records = walk_body(body)
if not records:
return None
return {
"Tran": [r["t_peak"] for r in records],
"Vert": [r["v_peak"] for r in records],
"Long": [r["l_peak"] for r in records],
"MicL": [r["m_peak"] for r in records],
}
def decode_histogram_body_full(body: bytes) -> Optional[List[dict]]:
"""Decode a histogram-mode body into the full per-interval record list.
Same data as ``decode_histogram_body`` but in a structured form that
preserves the half-period (frequency) data for each channel + the
per-block segment_id, block_ctr, and 4-byte variable metadata.
Useful for diagnostic tools, sidecar enrichment, and future-codec
work.
Returns ``None`` if the body has no valid blocks.
"""
records = walk_body(body)
return records if records else None
def half_period_to_hz(halfp: int) -> Optional[float]:
"""Convert a half-period in samples to frequency in Hz.
Returns ``None`` for half-period ≤ 5 — the device emits values in
that range when the measured zero-crossing rate exceeds 100 Hz
(the BW display reports `>100 Hz` for such cases). Callers can
treat ``None`` as the `>100 Hz` sentinel.
"""
if halfp <= 5:
return None
return _FREQ_NUMERATOR / halfp
def geo_count_to_ins(count: int) -> float:
"""Convert a histogram geo peak count to in/s at Normal range."""
return count * _GEO_LSB_INS
minimateplus/waveform_codec.py
+578
View File
@@ -0,0 +1,578 @@
"""
waveform_codec.py — block-walker and verified decoder for the MiniMate Plus
waveform-file body.
FULLY DECODED 2026-05-11. Every block type, every channel, and the
channel-rotation rule are verified byte-exact against BW's ASCII export
across the 9-event fixture bundle (47,364 ADC samples, zero errors).
The Blastware waveform-file body — the bytes between the 21-byte STRT
record and the 26-byte file footer — is a tagged variable-length block
stream with a custom delta + RLE codec. (Not raw int16 LE, which was
the historical wrong assumption that produced ±32K noise on every event.)
Current status:
- Block framing: ✅ solved (5 block types and lengths all confirmed)
- Per-channel decode: ✅ solved (Tran / Vert / Long / MicL all byte-exact)
- Channel rotation: ✅ Tran → Vert → Long → MicL per segment
- Segment header: ✅ fully decoded (anchor pair + prev-channel extension)
- 30 NN packed-delta block: ✅ NN × 12-bit signed deltas in NN/4 groups
- MicL → dB(L) conversion: ✅ ``mic_count_to_db`` matches BW display
- Production wiring: ✅ ``client.py:_decode_a5_waveform`` uses the new
codec (via ``decode_a5_frames``). ``.h5`` sidecars now render
correctly.
Known limitations:
- Walker stops early on the loudest events (SP0, SS0, SV0, event-b) at
some mid-segment edge cases not yet fully characterized. Every
sample reached IS correct; the walker just doesn't reach all of
them yet. The cleanly-decoded subset is still ~5000–15000 samples
per loud event.
────────────────────────────────────────────────────────────────────────────
Body layout (CONFIRMED 2026-05-11 against 8 fixture events)
────────────────────────────────────────────────────────────────────────────
[7-byte preamble] [stream of tagged blocks] [trailer]
The preamble is always exactly 7 bytes:
body[0:3] = 00 02 00 magic
body[3:5] = Tran[0] int16 BE in 16-count units (LSB = 0.005 in/s)
body[5:7] = Tran[1] int16 BE in 16-count units
(Earlier drafts of this module described a "7-or-9-byte preamble";
that was wrong — single-shot and continuous events both use 7 bytes.
The "extra 2 bytes" on continuous events were the first ``00 NN`` RLE
marker, not part of the preamble.)
Block types and lengths (all confirmed):
| Tag | Length | Meaning |
|----------|-----------------------|----------------------------------------|
| ``10 NN``| NN/2 + 2 bytes | 4-bit nibble deltas (2 per byte; high |
| | | nibble first; signed 0..7 / 8..F = -8..-1)|
| ``20 NN``| NN + 2 bytes | int8 signed deltas (1 per byte) |
| ``00 NN``| 2 bytes | RLE: append NN copies of current value |
| ``30 NN``| NN*2 in data, NN*4 | Unknown content. Only in loud events. |
| | in trailer | |
| ``40 02``| 20 bytes (fixed) | Segment header |
NN is always a multiple of 4.
────────────────────────────────────────────────────────────────────────────
Tran channel, segment 0 (CONFIRMED 2026-05-11)
────────────────────────────────────────────────────────────────────────────
Segment 0 — everything before the first ``40 02`` segment header — encodes
Tran samples only. Starting from preamble anchors Tran[0] and Tran[1],
each subsequent block contributes to the running Tran value:
10 NN → append NN deltas (4-bit signed nibbles)
20 NN → append NN deltas (int8 signed bytes)
00 NN → append NN copies of the current value (RLE zeros)
40 02 → segment 0 ends; multi-segment continuation is open
This decodes the first 482–510 samples of Tran for each event with zero
errors against BW's ASCII export. The exact segment-0 sample count
varies per event (it's bounded by a fixed device-flash byte budget, not
a fixed sample count — quiet events fit more samples because zero
deltas pack into ``00 NN`` markers compactly).
Implementation: :func:`decode_tran_initial`.
────────────────────────────────────────────────────────────────────────────
Segment header (40 02, 20 bytes total)
────────────────────────────────────────────────────────────────────────────
The 18-byte payload of the ``40 02`` block:
| Offset | Field | Status |
|-----------|---------------------------------------------|-------------|
| [0:2] | T_delta at first sample of new segment | ✅ confirmed|
| | (int16 BE, in 16-count units) | |
| [2:4] | Likely T_delta at sample seg_start+1 | 🟡 likely |
| [4:6] | Unknown (varies; possibly checksum) | ❓ open |
| [6:8] | Byte length to next segment header − 2 | ✅ confirmed|
| | (uint16 BE; useful for walker pre-scan) | |
| [8:12] | Monotonic uint32 LE counter | ✅ confirmed|
| | (starts ~0x47, increments by 1 per segment) | |
| [12:14] | Constant ``02 00`` | ✅ confirmed|
| [14:18] | Unknown 4-byte field | ❓ open |
────────────────────────────────────────────────────────────────────────────
What breaks the multi-segment decoder (the main open question)
────────────────────────────────────────────────────────────────────────────
After segment 0 ends and the segment header T_delta is consumed,
applying segment 1's blocks as Tran continuation produces values that
diverge from truth by sample ~512. The block structure inside segment
1 is IDENTICAL to segment 0 (same alternating 10 NN / 00 NN pattern),
and the delta budget matches the segment size exactly (V70 segment 1
has 264 nibble-deltas + 244 RLE zeros = 508 = the segment's sample
count). But the cumulative is wrong.
The strongest unverified hypothesis is that segments rotate channels:
segment 0 → Tran samples 0..509
segment 1 → Vert samples 0..507
segment 2 → Long samples 0..507
segment 3 → Mic samples 0..507
segment 4 → Tran samples 510..N (continuation)
...
This is consistent with the segment-1 block sums net-to-near-zero in
V70 (where all 4 channels are near zero) and with the per-segment delta
budget matching the segment size for a single channel. It is NOT yet
verified because the per-segment channel anchor isn't pinned down in
the segment header — bytes [4:6] and [14:18] of the header are still
open and probably encode V/L/M anchors.
See ``docs/waveform_codec_re_status.md`` for the current working notes
and the suggested next experiment ("segment-channel scoring analyzer").
"""
from __future__ import annotations
import math
from dataclasses import dataclass
from typing import List, Optional, Tuple
@dataclass
class WaveformBlock:
"""One tagged block parsed out of a Blastware waveform-file body."""
offset: int # byte offset into body
tag_hi: int # first tag byte (0x10 / 0x20 / 0x00 / 0x30 / 0x40)
tag_lo: int # second tag byte (NN)
data: bytes # block payload (excludes the 2-byte tag)
length: int # total block length on the wire (includes the tag)
@property
def kind(self) -> str:
return f"{self.tag_hi:02x} {self.tag_lo:02x}"
def find_data_start(body: bytes) -> int:
"""Auto-detect the offset of the first data block.
The body starts with a 7-byte preamble (magic ``00 02 00`` + two int16 BE
Tran anchors). After that, the data section starts with a tag — usually
``10 NN`` or ``20 NN``, but quiet events may begin with a ``00 NN`` RLE
marker. We return the offset of the first recognized tag.
"""
# Try fixed offset 7 first (canonical preamble length).
if len(body) >= 9:
b, nn = body[7], body[8]
if (b in (0x00, 0x10, 0x20, 0x30) and nn % 4 == 0 and 0 < nn <= 0xFC) \
or (b == 0x40 and nn == 0x02):
return 7
# Fall back to scanning the first 20 bytes.
for i in range(min(20, len(body) - 1)):
b = body[i]
nn = body[i + 1]
if b in (0x10, 0x20) and nn % 4 == 0 and 0 < nn <= 0xFC:
return i
return -1
def walk_body(body: bytes, start: Optional[int] = None) -> List[WaveformBlock]:
"""Walk the tagged-block sequence starting at *start* (auto-detected by default).
Stops when an unrecognized tag is encountered or end of body is reached.
Returned blocks are in stream order.
"""
if start is None:
start = find_data_start(body)
if start < 0:
return []
blocks: List[WaveformBlock] = []
i = start
while i + 1 < len(body):
t0 = body[i]
t1 = body[i + 1]
if t0 == 0x10 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
length = t1 // 2 + 2
elif (t0 & 0xF0) == 0x10 and (t0 & 0x0F) != 0 and t1 % 4 == 0:
# Wide-NN nibble block: ``1X NN`` where X is the high nibble of a
# 12-bit NN value. NN = ((t0 & 0x0F) << 8) | t1. Block length
# = NN/2 + 2 bytes (NN nibble deltas, same as ``10 NN`` semantics
# but with NN > 0xFC). Confirmed 2026-05-11 in SP0 segment 12
# where V continuation uses ``11 90`` = NN=0x190=400.
wide_nn = ((t0 & 0x0F) << 8) | t1
length = wide_nn // 2 + 2
elif t0 == 0x20 and t1 % 4 == 0 and 0 < t1 <= 0xFC:
length = t1 + 2
elif (t0 & 0xF0) == 0x20 and (t0 & 0x0F) != 0 and t1 % 4 == 0:
# Wide-NN int8 block: ``2X NN`` extends NN to 12 bits the same way.
wide_nn = ((t0 & 0x0F) << 8) | t1
length = wide_nn + 2
elif t0 == 0x00 and t1 % 4 == 0:
length = 2
elif t0 == 0x30 and t1 % 4 == 0 and 0 < t1 <= 0x10:
# Data-section ``30 NN`` blocks carry NN 12-bit signed deltas packed
# as NN/4 groups of (2-byte high-nibble field + 4 × int8 low byte).
# Length = NN/4 × 6 + 2 = NN × 1.5 + 2 (= 8 for NN=4, 14 for NN=8,
# 20 for NN=12, etc.). Confirmed 2026-05-11 by full-decoder
# verification against BW ASCII export.
#
# Trailer-section ``30 NN`` blocks have a different length formula
# (NN × 4 = 32 for NN=8 in trailers). We try the data-section
# length first and fall back to the trailer length if needed.
cand_data = t1 * 3 // 2 + 2
cand_trailer = t1 * 4
if (i + cand_data < len(body) - 1
and body[i + cand_data] in (0x10, 0x20, 0x00, 0x30, 0x40)):
length = cand_data
else:
length = cand_trailer
elif t0 == 0x40 and t1 == 0x02:
length = 20
else:
# Unknown tag; stop. Caller can inspect ``i`` to see where.
break
if i + length > len(body):
break
data = bytes(body[i + 2 : i + length])
blocks.append(WaveformBlock(offset=i, tag_hi=t0, tag_lo=t1, data=data, length=length))
i += length
return blocks
def split_segments(blocks: List[WaveformBlock]) -> List[List[WaveformBlock]]:
"""Group consecutive blocks into segments separated by ``40 02`` headers.
The first segment is whatever runs before the first ``40 02`` header
(typically the "segment 0" preamble data after the body preamble).
Subsequent segments start with a ``40 02`` block, then have their
own data blocks until the next ``40 02``.
"""
segments: List[List[WaveformBlock]] = []
current: List[WaveformBlock] = []
for b in blocks:
if b.tag_hi == 0x40 and b.tag_lo == 0x02:
if current:
segments.append(current)
current = [b]
else:
current.append(b)
if current:
segments.append(current)
return segments
def parse_segment_header(block: WaveformBlock) -> Optional[dict]:
"""Decode the 18-byte payload of a ``40 02`` segment header.
Returns a dict with the labelled fields, or None if *block* is not
a ``40 02`` header.
"""
if not (block.tag_hi == 0x40 and block.tag_lo == 0x02):
return None
if len(block.data) < 18:
return None
p = block.data
counter = int.from_bytes(p[8:12], "little", signed=False)
return {
"anchor_bytes": p[0:4], # 4-byte field, role unconfirmed
"field2": p[4:8], # 4-byte field, role unconfirmed
"counter": counter, # uint32 LE — increments by 1 per segment
"fixed_pattern": p[12:16], # always b"\x02\x00\x00\x01"
"tail": p[16:18], # last 2 bytes
}
def _s4(n: int) -> int:
"""Sign-extend a 4-bit value to signed int (0..7 → 0..7; 8..F → -8..-1)."""
return n if n < 8 else n - 16
def _i8(b: int) -> int:
"""Reinterpret an unsigned byte as signed int8."""
return b if b < 128 else b - 256
def decode_tran_initial(body: bytes) -> Optional[List[int]]:
"""
Decode the initial Tran-channel samples — VERIFIED 2026-05-11.
Returns Tran samples in **16-count units** (LSB = 0.005 in/s at Normal
range — the same quantization BW uses for its ASCII export). Returns
``None`` if the body cannot be parsed.
The decoded list extends from sample 0 through the end of segment 0
(= just before the first ``40 02`` segment header; ~510 sample-sets
for the events tested). Multi-segment decoding requires continuing
past the segment header — that's done by :func:`decode_tran_full`
when the per-segment rules are pinned down for all signal types.
Codec for segment 0 (CONFIRMED 2026-05-11 against 7 fixture events):
- Body bytes [0:3] are the magic ``00 02 00``.
- Body bytes [3:5] = ``Tran[0]`` as int16 BE in 16-count units.
- Body bytes [5:7] = ``Tran[1]`` as int16 BE in 16-count units.
- Data blocks (``10 NN`` or ``20 NN``) carry Tran deltas starting
at sample 2:
* ``10 NN``: NN nibbles = NN/2 bytes; each nibble is a 4-bit
signed delta (0..7 → 0..+7; 8..F → -8..-1). High nibble of
each byte comes first.
* ``20 NN``: NN int8 signed deltas (one delta per byte).
- ``00 NN`` blocks are run-length-encoded zero deltas: append NN
copies of the current cumulative Tran value (no change).
- ``30 NN`` blocks have not yet been decoded for content — they
appear in segment 0 of loud-from-start events (SS0, SV0) and
seem to signal a transition or special-case interpretation.
The walker steps over them but their data is ignored.
The walk stops at the first ``40 02`` segment header.
"""
if len(body) < 7 or body[0:3] != b"\x00\x02\x00":
return None
t0 = int.from_bytes(body[3:5], "big", signed=True)
t1 = int.from_bytes(body[5:7], "big", signed=True)
start = find_data_start(body)
if start < 0:
return [t0, t1]
out = [t0, t1]
cur = t1
for blk in walk_body(body, start):
if blk.tag_hi == 0x40:
# Segment boundary — stop. Multi-segment decode is decode_tran_full.
break
if blk.tag_hi == 0x10:
for byte in blk.data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
cur += _s4(nib)
out.append(cur)
elif blk.tag_hi == 0x20:
for byte in blk.data:
cur += _i8(byte)
out.append(cur)
elif blk.tag_hi == 0x00:
# RLE zero deltas: append NN copies of current Tran value.
for _ in range(blk.tag_lo):
out.append(cur)
# 30 NN: unknown content; skip.
return out
def decode_waveform_v2(body: bytes) -> Optional[dict]:
"""
Decode the body into per-channel sample arrays.
Status (2026-05-11 evening — channel-rotation hypothesis CONFIRMED):
segments rotate channels in fixed order **Tran → Vert → Long → MicL**.
Each channel-segment carries a 2-sample anchor pair in segment-header
bytes [14:18] (or in the body preamble for the initial Tran segment)
plus a stream of delta blocks for samples 2 onward.
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}``
with each channel's decoded samples in 16-count units (LSB = 0.005
in/s at Normal range). Returns ``None`` if the body cannot be
parsed.
"""
if len(body) < 7 or body[0:3] != b"\x00\x02\x00":
return None
channels = ["Tran", "Vert", "Long", "MicL"]
out: dict = {ch: [] for ch in channels}
# Initial Tran segment: preamble anchor pair + delta blocks before first 40 02.
t0 = int.from_bytes(body[3:5], "big", signed=True)
t1 = int.from_bytes(body[5:7], "big", signed=True)
out["Tran"].extend([t0, t1])
start = find_data_start(body)
if start < 0:
return out
blocks = walk_body(body, start)
seg_idx = [i for i, b in enumerate(blocks) if b.tag_hi == 0x40]
def apply_blocks(channel: str, anchor: int,
block_start: int, block_end: int) -> int:
"""Apply delta blocks [block_start, block_end) to *channel*'s sample
list, starting from *anchor*. Returns the final cumulative value."""
cur = anchor
for bi in range(block_start, block_end):
blk = blocks[bi]
if (blk.tag_hi & 0xF0) == 0x10:
# Both ``10 NN`` (NN ≤ 0xFC) and wide-NN ``1X NN`` (X != 0)
# are nibble-delta streams. The walker has already used the
# right length; here we just iterate the payload bytes.
for byte in blk.data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
cur += _s4(nib)
out[channel].append(cur)
elif (blk.tag_hi & 0xF0) == 0x20:
# ``20 NN`` and wide ``2X NN`` both carry int8 deltas.
for byte in blk.data:
cur += _i8(byte)
out[channel].append(cur)
elif blk.tag_hi == 0x00:
for _ in range(blk.tag_lo):
out[channel].append(cur)
elif blk.tag_hi == 0x30:
# 12-bit signed deltas, packed as NN/4 groups of 6 bytes each:
# bytes [0:2] = 16 bits = 4 × 4-bit high nibbles (MSB first)
# bytes [2:6] = 4 × int8 low bytes
# Each delta = sign_extend_12((high_nibble << 8) | low_byte).
# Confirmed 2026-05-11 against all 14 ``30 NN`` blocks in the
# bundled fixtures.
n_groups = blk.tag_lo // 4
for g in range(n_groups):
grp = blk.data[g * 6 : (g + 1) * 6]
if len(grp) < 6:
break
high_word = (grp[0] << 8) | grp[1]
for k in range(4):
nib = (high_word >> (12 - 4 * k)) & 0xF
v = (nib << 8) | grp[2 + k]
if v >= 0x800:
v -= 0x1000
cur += v
out[channel].append(cur)
# 40 02: should not occur in segment data.
return cur
# Initial Tran segment: deltas from start of body up to first 40 02 (or end).
first_seg = seg_idx[0] if seg_idx else len(blocks)
last_tran_value = apply_blocks("Tran", t1, 0, first_seg)
# Subsequent segments rotate channels. Each segment header carries:
# bytes [0:2] and [2:4] = 2 deltas extending the PREVIOUS channel
# bytes [14:16] and [16:18] = anchor pair for THIS segment's channel
#
# Rotation: V, L, M, T, V, L, M, T, ... (initial Tran segment is the
# implicit T in the cycle.)
rotation = ["Vert", "Long", "MicL", "Tran"]
# Track each channel's "running cumulative value" so we can apply the
# previous-channel extension deltas at every segment boundary.
last_value = {"Tran": last_tran_value, "Vert": None, "Long": None, "MicL": None}
for k, hi in enumerate(seg_idx):
channel = rotation[k % 4]
prev_channel = "Tran" if k == 0 else rotation[(k - 1) % 4]
header = blocks[hi]
if len(header.data) < 18:
continue
# Validate: real segment headers have bytes [12:14] = `02 00`.
# Trailer/footer "40 02" markers contain ASCII serial bytes or other
# non-header data there and would otherwise be mis-interpreted as
# segment headers, adding spurious samples at the tail.
if header.data[12:14] != b"\x02\x00":
break
# Extend the PREVIOUS channel by 2 more samples (deltas in bytes [0:4]).
prev_d0 = int.from_bytes(header.data[0:2], "big", signed=True)
prev_d1 = int.from_bytes(header.data[2:4], "big", signed=True)
if last_value[prev_channel] is not None:
v = last_value[prev_channel] + prev_d0
out[prev_channel].append(v)
v += prev_d1
out[prev_channel].append(v)
last_value[prev_channel] = v
# Anchor pair for THIS segment's channel.
c0 = int.from_bytes(header.data[14:16], "big", signed=True)
c1 = int.from_bytes(header.data[16:18], "big", signed=True)
out[channel].extend([c0, c1])
# Apply delta blocks for this segment.
next_hi = seg_idx[k + 1] if k + 1 < len(seg_idx) else len(blocks)
last_value[channel] = apply_blocks(channel, c1, hi + 1, next_hi)
return out
# ── ADC-scale conversion helpers ────────────────────────────────────────────
# Scaling factor: decode_waveform_v2 produces geo-channel samples in the BW
# display quantization (16-count units, LSB = 0.005 in/s at Normal range).
# The legacy consumer pipeline (sfm/event_hdf5.py) expects raw_samples in
# 1-count ADC units (× full_scale / 32768 → physical). To plug the new
# decoder in without rewriting consumers, multiply geo values by 16.
#
# Mic samples are already in raw ADC counts (decoded value 1 = 1 mic ADC count
# = -81.94 dB on the BW display). Mic values pass through unchanged.
_GEO_DECODER_TO_ADC = 16
def decoded_to_adc_counts(decoded: dict) -> dict:
"""Convert :func:`decode_waveform_v2` output to int16 ADC counts.
Geo channels are scaled by ×16 (decoder produces 16-count units,
consumer expects 1-count ADC). Mic is passed through as raw counts.
"""
if not decoded:
return {}
return {
"Tran": [v * _GEO_DECODER_TO_ADC for v in decoded.get("Tran", [])],
"Vert": [v * _GEO_DECODER_TO_ADC for v in decoded.get("Vert", [])],
"Long": [v * _GEO_DECODER_TO_ADC for v in decoded.get("Long", [])],
"MicL": list(decoded.get("MicL", [])),
}
def mic_count_to_db(count: int) -> float:
"""Convert a MicL ADC count to dB(L) for BW-display-compatible output.
Empirical formula (confirmed 2026-05-11 against V70 fixture: count=813
→ 140.1 dB; count=±1 → ±81.94 dB; count=±24 → ±109.5 dB):
dB = sign(count) × (81.94 + 20 × log10(|count|)) for |count| ≥ 1
dB = 0.0 for count == 0
The constant 81.94 corresponds to 10^(81.94/20) ≈ 12490 mic ADC counts
being the dB(L) reference level — almost certainly a calibration
constant from the device's mic.
"""
if count == 0:
return 0.0
sign = 1.0 if count > 0 else -1.0
return sign * (81.94 + 20.0 * math.log10(abs(count)))
# ── A5-frame entry point ────────────────────────────────────────────────────
def decode_a5_frames(a5_frames) -> Optional[dict]:
"""Decode a list of A5 (BULK_WAVEFORM_STREAM) frames into per-channel
int16 ADC samples.
Returns ``{"Tran": [...], "Vert": [...], "Long": [...], "MicL": [...]}``
with each channel's samples in **1-count ADC units** (the legacy
``event.raw_samples`` convention — multiply by ``full_scale / 32768``
to convert to physical units; for mic, use :func:`mic_count_to_db` or
a per-count psi factor).
Returns ``None`` if the frames cannot be parsed.
This is the wired-up production entry point. It:
1. Reconstructs the BW-binary body bytes from the A5 frames
(``blastware_file.extract_body_bytes``).
2. Runs the verified codec (``decode_waveform_v2``) on the body.
3. Converts to int16 ADC counts via :func:`decoded_to_adc_counts`.
"""
# Local import to avoid a cycle: blastware_file imports models and
# ultimately client.py imports waveform_codec.
from .blastware_file import extract_body_bytes
if not a5_frames:
return None
_strt, body, _footer = extract_body_bytes(a5_frames)
if not body:
return None
decoded = decode_waveform_v2(body)
if decoded is None:
return None
return decoded_to_adc_counts(decoded)
pyproject.toml
+3 -3
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "seismo-relay"
version = "0.15.0"
version = "0.19.0"
description = "Python client and REST server for MiniMate Plus seismographs"
requires-python = ">=3.10"
dependencies = [
@@ -18,6 +18,6 @@ dependencies = [
]
[tool.setuptools.packages.find]
# Auto-discovers minimateplus/, sfm/, bridges/ as packages
# Auto-discovers minimateplus/, micromate/, sfm/, bridges/ as packages
where = ["."]
include = ["minimateplus*", "sfm*", "bridges*"]
include = ["minimateplus*", "micromate*", "sfm*", "bridges*"]
scratch/next_experiment_skeleton.py
+360
View File
@@ -0,0 +1,360 @@
"""
scratch/next_experiment_skeleton.py — segment-channel scoring analyzer.
This is the suggested NEXT EXPERIMENT for cracking the waveform body codec.
The goal is to figure out what segments 1+ contain, since segment 0 = Tran
is solved but multi-segment continuation diverges from truth at sample ~512.
────────────────────────────────────────────────────────────────────────────
The hypothesis to test
────────────────────────────────────────────────────────────────────────────
Segments rotate through channels:
segment 0 → Tran samples 0..509
segment 1 → Vert samples 0..507
segment 2 → Long samples 0..507
segment 3 → Mic samples 0..507
segment 4 → Tran samples 510..N (continuation)
...
This would explain why segment 0 works perfectly (it's pure Tran) and why
applying segment 1's blocks as Tran continuation gives wrong values
(it's actually Vert).
────────────────────────────────────────────────────────────────────────────
What the analyzer should do
────────────────────────────────────────────────────────────────────────────
For each segment in each fixture event:
1. Run the segment-0 block-walker + RLE decode (the same algorithm that
``decode_tran_initial`` uses) over the segment's blocks. Start from
some anchor value and produce a cumulative trajectory of length =
number-of-deltas-in-segment.
2. For each candidate channel C ∈ {Tran, Vert, Long, MicL}:
For each candidate anchor location in the segment-header payload
(try [0:2], [2:4], [4:6], [14:16], [16:18] as int16 BE):
Compare the decoded trajectory against truth[C] starting from
the segment's first sample index.
Score = number of matches (or sum of squared errors).
3. Report the best (channel, anchor-location) combination per segment.
If the rotation hypothesis is correct, you'll see:
segment 0 → best score for (Tran, preamble bytes [3:5]) ✓ already known
segment 1 → best score for (Vert, <some-header-byte>)
segment 2 → best score for (Long, <some-header-byte>)
segment 3 → best score for (MicL, <some-header-byte>)
segment 4 → best score for (Tran, continuing from segment 0's end)
If the rotation hypothesis is NOT correct, the scorer will at least narrow
down what segment 1 actually carries. Maybe channels interleave at finer
granularity, or maybe segments alternate by something other than channel.
────────────────────────────────────────────────────────────────────────────
Why this is a scoring analyzer, not a hand-written decoder
────────────────────────────────────────────────────────────────────────────
Direct hand-coding ("assume segment 1 is Vert with anchor at byte X") gets
stuck when the assumption is wrong because the failure mode is silent —
you get plausible-looking-but-wrong samples and have to manually diff
against truth to debug.
The scorer is brute-force but cheap: every fixture event × every segment ×
4 channels × 5 anchor-byte candidates is only ~hundreds of comparisons.
The winning combination jumps out by score.
────────────────────────────────────────────────────────────────────────────
Skeleton
────────────────────────────────────────────────────────────────────────────
"""
from __future__ import annotations
import os
import re
import sys
from dataclasses import dataclass
from typing import List, Optional, Tuple
sys.path.insert(0, os.path.join(os.path.dirname(__file__), ".."))
from minimateplus.waveform_codec import walk_body, find_data_start, WaveformBlock
# ── Reusable pieces ──────────────────────────────────────────────────────────
CHANNELS = ("Tran", "Vert", "Long", "MicL")
LSB_INV = 200 # 1 in/s / 0.005 in/s/LSB; multiply BW-export floats by this
# to get 16-count units (the body's native quantization).
@dataclass
class FixtureEvent:
name: str # e.g. "M529LL1A.SP0"
bin_path: str
txt_path: str
body: bytes
truth: dict # {channel: list of int16-quantized samples}
blocks: List[WaveformBlock]
segment_starts: List[int] # block indices of each 40 02 segment header
segment_sample_starts: List[int] # for each segment, the truth sample index it starts at
def s4(n: int) -> int:
"""4-bit signed nibble decode."""
return n if n < 8 else n - 16
def i8(b: int) -> int:
"""int8 reinterpret of unsigned byte."""
return b if b < 128 else b - 256
def load_fixture(name: str) -> FixtureEvent:
"""Load a fixture event with its truth values and parsed block stream."""
# Find the fixture (search both subdirs of tests/fixtures/).
base = os.path.join(os.path.dirname(__file__), "..", "tests", "fixtures")
candidates = [
os.path.join(base, "5-11-26", name),
os.path.join(base, "decode-re-5-8-26", "event-a", name), # not used directly
]
bin_path = next((c for c in candidates if os.path.exists(c)), None)
if bin_path is None:
# Try a glob walk for the 5-8 fixtures (they're in subdirs).
for root, _, files in os.walk(base):
if name in files:
bin_path = os.path.join(root, name)
break
if bin_path is None:
raise FileNotFoundError(name)
txt_path = bin_path + ".TXT"
with open(bin_path, "rb") as f:
raw = f.read()
body = raw[43:-26]
truth = _parse_txt(txt_path)
blocks = walk_body(body, find_data_start(body))
seg_idx = [i for i, b in enumerate(blocks) if b.tag_hi == 0x40]
# Segment 0 starts at sample 0; subsequent segments start at the
# cumulative sample count from previous segment(s). Tran's segment 0
# is N samples; if rotation hypothesis is correct, segment 1's data
# starts at sample 0 for a *different* channel. The analyzer should
# try both "continues from previous segment" and "starts at sample 0
# of a different channel."
seg_sample_starts = _compute_segment_sample_starts(blocks, seg_idx)
return FixtureEvent(
name=name, bin_path=bin_path, txt_path=txt_path,
body=body, truth=truth, blocks=blocks,
segment_starts=seg_idx, segment_sample_starts=seg_sample_starts,
)
def _parse_txt(path: str) -> dict:
"""Parse BW ASCII TXT export into {channel: [int_samples_in_16_count_units]}."""
with open(path, "r", encoding="utf-8", errors="replace") as f:
lines = f.read().splitlines()
header_idx = next(
(i for i, l in enumerate(lines)
if all(c in l for c in CHANNELS)),
None,
)
if header_idx is None:
return {ch: [] for ch in CHANNELS}
out = {ch: [] for ch in CHANNELS}
for line in lines[header_idx + 1:]:
parts = re.split(r"\s+", line.strip())
if len(parts) < 4:
continue
try:
vals = [float(p) for p in parts[:4]]
except ValueError:
continue
for ch, v in zip(CHANNELS, vals):
# Multiply by LSB_INV; geo channels are in in/s, MicL is in dB(L)
# (which doesn't quantize the same way — leaving raw for MicL is fine,
# the scorer should treat MicL specially).
out[ch].append(round(v * LSB_INV) if ch != "MicL" else v)
return out
def _compute_segment_sample_starts(
blocks: List[WaveformBlock], seg_idx: List[int]
) -> List[int]:
"""Cumulative sample-count up to each segment header (if all blocks treated
as Tran continuation). Useful as one candidate for segment-1-Tran tests.
The scorer should ALSO try "segment 1 starts at sample 0 of a new channel"
as the rotation hypothesis predicts.
"""
starts = []
cum = 2 # T[0] + T[1] from preamble
for i, b in enumerate(blocks):
if i in seg_idx:
starts.append(cum)
if b.tag_hi == 0x10:
cum += b.tag_lo
elif b.tag_hi == 0x20:
cum += b.tag_lo
elif b.tag_hi == 0x00:
cum += b.tag_lo
# 30 NN and 40 02 don't contribute samples (for this hypothesis)
return starts
# ── The core algorithm: decode a segment's blocks as deltas ─────────────────
def decode_segment_as_channel(
blocks: List[WaveformBlock],
seg_start_block_idx: int,
seg_end_block_idx: int,
anchor: int,
) -> List[int]:
"""Apply the segment-0 codec rules to a range of blocks, starting from *anchor*.
Returns a list of cumulative sample values (one per delta). Does NOT include
the anchor itself in the output — the first returned value is anchor + first_delta.
"""
out = []
cur = anchor
for bi in range(seg_start_block_idx, seg_end_block_idx):
blk = blocks[bi]
if blk.tag_hi == 0x10:
for byte in blk.data:
for nib in ((byte >> 4) & 0xF, byte & 0xF):
cur += s4(nib)
out.append(cur)
elif blk.tag_hi == 0x20:
for byte in blk.data:
cur += i8(byte)
out.append(cur)
elif blk.tag_hi == 0x00:
for _ in range(blk.tag_lo):
out.append(cur)
# 30 NN: skip (content unknown)
# 40 02: shouldn't appear in segment data (it's the segment header)
return out
def score_against_truth(
decoded: List[int],
truth: List[int],
truth_start: int,
) -> Tuple[int, int]:
"""Compare *decoded* to truth[truth_start : truth_start + len(decoded)].
Returns (n_matches, n_compared).
"""
n = min(len(decoded), len(truth) - truth_start)
if n <= 0:
return (0, 0)
matches = sum(1 for i in range(n) if decoded[i] == truth[truth_start + i])
return (matches, n)
# ── TODO for the next pass ──────────────────────────────────────────────────
def score_segment_against_all_channels(
event: FixtureEvent,
segment_index: int,
) -> List[Tuple[str, int, int, int]]:
"""For segment *segment_index* of *event*, find the best (channel, start_sample)
fit.
For each candidate channel C and each candidate starting truth-sample index s,
we pick the anchor that makes the FIRST decoded value match truth[C][s], then
score the remaining decoded values against truth[C][s+1 : s+N].
Returns rows of (channel_name, start_sample, n_matches, n_compared)
sorted by match-count descending.
"""
# Block range of this segment: from the segment header (inclusive) up to
# the next segment header (exclusive), or end-of-blocks.
seg_header_idx = event.segment_starts[segment_index]
next_header_idx = (
event.segment_starts[segment_index + 1]
if segment_index + 1 < len(event.segment_starts)
else len(event.blocks)
)
# Decode the segment's data blocks (skip the segment-header block itself).
# Use anchor=0 — we'll re-anchor when scoring against each channel.
deltas_trajectory = decode_segment_as_channel(
event.blocks, seg_header_idx + 1, next_header_idx, anchor=0
)
if not deltas_trajectory:
return []
n = len(deltas_trajectory)
results = []
for ch in ("Tran", "Vert", "Long"):
truth = event.truth.get(ch)
if not truth or len(truth) < n + 1:
continue
# For each candidate starting sample s in truth, check if applying
# the deltas starting from truth[s] reproduces truth[s+1:s+n+1].
best = (0, -1)
for s in range(len(truth) - n):
anchor = truth[s]
offset = anchor - deltas_trajectory[0] + truth[s + 1] - anchor
# Recompute: trajectory[i] = anchor + cumulative_delta_through_i
# but we already have deltas_trajectory computed from anchor=0,
# so trajectory_relative[i] = anchor + deltas_trajectory[i].
matches = 0
for i in range(n):
if truth[s + i + 1] == anchor + deltas_trajectory[i]:
matches += 1
# Note: we could break early on first mismatch for "matches start",
# but counting total matches gives a more robust score.
if matches > best[0]:
best = (matches, s)
results.append((ch, best[1], best[0], n))
results.sort(key=lambda r: -r[2])
return results
# ── Driver ──────────────────────────────────────────────────────────────────
def main():
"""Run the analyzer on all loud-bundle events and print best scores."""
events = ["M529LL1A.SP0", "M529LL1A.SS0", "M529LL1A.SV0",
"M529LL1L.JQ0", "M529LL1L.V70"]
for name in events:
try:
event = load_fixture(name)
except FileNotFoundError:
print(f"{name}: fixture not found")
continue
print(f"\n=== {name} ===")
print(f" body bytes: {len(event.body)}")
print(f" blocks: {len(event.blocks)}")
print(f" segments: {len(event.segment_starts)}")
print(f" segment sample-starts (if all blocks are 1 channel):")
for si, sample_start in enumerate(event.segment_sample_starts):
print(f" seg {si}: sample {sample_start}")
for si in range(len(event.segment_starts)):
results = score_segment_against_all_channels(event, si)
if not results:
print(f" seg {si}: (no scorable data)")
continue
tag = "✓" if results[0][2] / max(results[0][3], 1) > 0.9 else " "
top = results[0]
print(f" seg {si}: best fit {tag} = {top[0]:<5} "
f"starting at sample {top[1]:>5}, {top[2]:>4}/{top[3]:<4} match"
+ (f" (next: {results[1][0]} @{results[1][1]} {results[1][2]}/{results[1][3]})"
if len(results) > 1 else ""))
if __name__ == "__main__":
main()
scripts/backfill_record_type.py
+150
View File
@@ -0,0 +1,150 @@
"""
scripts/backfill_record_type.py — fix `record_type` on legacy event
rows whose value was hardcoded to "Waveform" regardless of actual type.
Why this is needed
──────────────────
Pre-v0.16.1 the BW file importer (`event_file_io.read_blastware_file`)
hardcoded `ev.record_type = "Waveform"` for every imported event. Fixed
in commit aac1c8e — new ingests now derive the type from the Blastware
filename's extension last character (H=Histogram, W=Waveform, M=Manual,
E=Event, C=Combo) per the V10.72+ MiniMate Plus AB0T filename scheme.
Effect on a server that imported events under the old code: every
events row has `record_type = "Waveform"`, even for histograms,
manuals, etc. Visible in terra-view's event-detail modal under the
"Record Type" field. Terra-view also has a client-side workaround
that derives the type from the filename for display purposes, so
operators see the correct type in the UI even before this backfill.
This script makes the DB column match what the UI is already showing,
which matters for reporting and any downstream consumer that reads
events.record_type directly.
This script
───────────
Walks the `events` table and updates each row's `record_type` to the
derived value from its `blastware_filename`. Old S338 firmware files
(3-char extensions ending in `0`) and any unrecognized suffix get
left at the existing value (defaults to "Waveform").
Idempotent: re-running after a successful backfill finds zero rows
needing updates and exits cleanly (it always re-derives but only
writes when the value would change).
Usage
─────
# Dry-run (default): print what would change, don't touch the DB
python -m scripts.backfill_record_type --db bridges/captures/seismo_relay.db
# Apply the backfill
python -m scripts.backfill_record_type --db bridges/captures/seismo_relay.db --apply
"""
from __future__ import annotations
import argparse
import sqlite3
import sys
from collections import Counter
from pathlib import Path
# Must stay in sync with minimateplus.event_file_io._RECORD_TYPE_BY_EXT_SUFFIX.
_TYPE_FROM_SUFFIX = {
"H": "Histogram",
"W": "Waveform",
"M": "Manual",
"E": "Event",
"C": "Combo",
}
def derive_record_type(filename: str | None, default: str = "Waveform") -> str:
"""Mirror of minimateplus.event_file_io.derive_record_type_from_filename.
Vendored here so this script runs without needing the seismo-relay
package on the Python path (useful on prod where you might be
running it via `docker exec` against a container's DB volume).
"""
if not filename:
return default
name = Path(filename).name
if "." not in name:
return default
ext = name.rsplit(".", 1)[1]
if not ext:
return default
return _TYPE_FROM_SUFFIX.get(ext[-1].upper(), default)
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__)
ap.add_argument("--db", required=True, help="Path to seismo_relay.db")
ap.add_argument("--apply", action="store_true",
help="Actually write changes (default is dry-run).")
ap.add_argument("--default", default="Waveform",
help="Fallback record_type when filename doesn't encode one. "
"Default: Waveform (matches the pre-fix bug's behavior).")
args = ap.parse_args()
db_path = Path(args.db)
if not db_path.exists():
print(f"ERROR: database not found at {db_path}", file=sys.stderr)
return 1
conn = sqlite3.connect(str(db_path))
conn.row_factory = sqlite3.Row
cur = conn.cursor()
cur.execute("""
SELECT id, blastware_filename, record_type
FROM events
WHERE blastware_filename IS NOT NULL
AND blastware_filename != ''
""")
rows = cur.fetchall()
total = len(rows)
print(f"Scanning {total:,} event rows…")
print()
# Tally proposed changes.
transitions: Counter[tuple[str, str]] = Counter()
update_ids: list[tuple[str, str]] = []
unrecognized = 0
for row in rows:
derived = derive_record_type(row["blastware_filename"], default=args.default)
current = row["record_type"] or ""
if derived == current:
continue
transitions[(current, derived)] += 1
update_ids.append((row["id"], derived))
if not update_ids:
print("Nothing to update — all rows already match.")
conn.close()
return 0
print(f"{len(update_ids):,} row(s) need updating:")
for (old, new), count in sorted(transitions.items(), key=lambda x: -x[1]):
print(f" {count:>6,} {old!r:14s} → {new!r}")
print()
if not args.apply:
print("(dry-run — re-run with --apply to write changes)")
conn.close()
return 0
print("Applying changes…")
cur.executemany(
"UPDATE events SET record_type = ? WHERE id = ?",
[(new, eid) for eid, new in update_ids],
)
conn.commit()
print(f"Done. Updated {cur.rowcount:,} row(s).")
conn.close()
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/backfill_sidecars.py
+50 -5
View File
@@ -12,8 +12,20 @@ Walks `<store_root>/<serial>/<filename>` and for each BW event file:
parsing the BW binary directly (peaks computed from samples).
Clean waveform (.h5):
- Skip when <filename>.h5 already exists (idempotent).
- Else write from .a5.pkl (preferred) or BW binary parse (fallback).
- Regenerated whenever the sidecar is regenerated (sha mismatch
OR sidecar.source.tool_version < current TOOL_VERSION OR --force).
The .h5 and the sidecar both come from the same decoder output,
so if the sidecar is stale the .h5 is too.
- Written when missing.
- --skip-hdf5 turns off all .h5 writes.
Typical use after a decoder upgrade:
1. Pull the new seismo-relay code (which bumped TOOL_VERSION).
2. Run this script — every sidecar with an older tool_version
stamp regenerates, and the associated .h5 cascade-regenerates.
3. Operator review state (review.false_trigger, notes, reviewer)
and the sidecar's extensions block are preserved across the
regen.
Usage:
python scripts/backfill_sidecars.py [--store-root PATH]
@@ -123,6 +135,12 @@ def main(argv=None) -> int:
# the sidecar was written by a build that includes any
# decoder fixes shipped since).
# Either part failing → regenerate. --force bypasses both.
#
# Tracks whether we're regenerating the sidecar this iteration
# so the .h5 logic below knows to refresh that too — staleness
# of the sidecar implies staleness of the derived .h5 (both
# come out of the same decoder).
sidecar_stale = True
if sidecar_path.exists() and not args.force:
try:
existing = event_file_io.read_sidecar(sidecar_path)
@@ -136,6 +154,7 @@ def main(argv=None) -> int:
ver_ok = _vt(src_ver) >= _vt(event_file_io.TOOL_VERSION)
if sha_ok and ver_ok:
skipped += 1
sidecar_stale = False
continue
if sha_ok and not ver_ok:
log.info(
@@ -281,12 +300,37 @@ def main(argv=None) -> int:
extensions=preserved_ext,
)
# Also emit the .h5 clean-waveform file when missing OR when
# --force was passed (so a re-backfill picks up decoder fixes).
# Also emit the .h5 clean-waveform file when:
# - it's missing, OR
# - --force was passed, OR
# - the sidecar is being regenerated this iteration
# (sha mismatch / tool_version too old). The .h5 and
# the sidecar are both derived from the same decoder
# output, so if the sidecar is stale, so is the .h5.
#
# Both waveform and histogram bodies now decode to real
# samples via event_file_io.read_blastware_file → either
# waveform_codec.decode_waveform_v2 or histogram_codec.
# decode_histogram_body. If samples are still empty after
# both codecs run, it's a genuine "we can't decode this
# file" case (truncated, malformed, or unknown mode);
# skip the .h5 write so we don't replace whatever's
# there with an empty placeholder.
has_samples = bool(
ev.raw_samples and any(
ev.raw_samples.get(ch) for ch in ("Tran", "Vert", "Long", "MicL")
)
)
hdf5_path = store.hdf5_path_for(serial, path.name)
hdf5_filename = hdf5_path.name if hdf5_path.exists() else None
hdf5_action = "kept"
need_h5 = not args.skip_hdf5 and (args.force or not hdf5_path.exists())
need_h5 = (
not args.skip_hdf5
and (args.force or not hdf5_path.exists() or sidecar_stale)
and has_samples
)
if not has_samples and not args.skip_hdf5:
hdf5_action = "skipped-undecodable"
if need_h5:
if args.dry_run:
hdf5_action = "would (re)write"
@@ -326,6 +370,7 @@ def main(argv=None) -> int:
}}
if ev._waveform_key else None
),
device_family="series3",
)
except Exception as exc:
log.warning("DB upsert failed for %s: %s", path.name, exc)
scripts/blind_stop.sh
Executable
+100
View File
@@ -0,0 +1,100 @@
#!/usr/bin/env bash
# Fire-and-forget Stop Monitoring loop — for wedged or constantly-triggering units.
#
# Hammers POST /device/stop_monitoring_blind in a tight loop. The endpoint
# opens TCP, dumps SESSION_RESET + a few copies of the SUB 0x97 frame, and
# closes — without ever reading an S3 response. Each TCP-won attempt is
# ~50ms of wire activity instead of the multi-frame handshake the regular
# rescue endpoint does, so windows that are too small for the full rescue
# can still land a stop-monitoring command.
#
# Usage:
# ./blind_stop.sh <host> [tcp_port]
#
# Env:
# SFM_BASE_URL Default: http://localhost:8200 (SFM direct).
# Set to http://localhost:8001/api/sfm to route through
# Terra-View's proxy.
# MAX_ATTEMPTS Default: 600
# SLEEP_S Default: 0 (no backoff — hammer it)
# MAX_TIME_S Default: 15
# CONNECT_TIMEOUT Default: 5
# REPEAT Frames per TCP session (default 3 — increases hit rate
# if the device is busy reading its own buffer).
# STOP_ON_OK Default: 1. Set to 0 to keep hammering indefinitely
# even after successful sends (every 503 means the device
# is in *another* session, every 200 means our bytes got
# through — but the device may not have processed them).
set -u
host="${1:-}"
tcp_port="${2:-9034}"
if [[ -z "$host" ]]; then
echo "usage: $0 <host> [tcp_port]" >&2
exit 2
fi
base="${SFM_BASE_URL:-http://localhost:8200}"
max_attempts="${MAX_ATTEMPTS:-600}"
sleep_s="${SLEEP_S:-0}"
max_time_s="${MAX_TIME_S:-15}"
connect_timeout="${CONNECT_TIMEOUT:-5}"
repeat="${REPEAT:-3}"
stop_on_ok="${STOP_ON_OK:-1}"
url="${base}/device/stop_monitoring_blind?host=${host}&tcp_port=${tcp_port}&connect_timeout=${connect_timeout}&repeat=${repeat}"
echo "blind_stop: target ${host}:${tcp_port} connect_timeout=${connect_timeout}s repeat=${repeat}"
echo "blind_stop: POST ${url}"
echo "blind_stop: up to ${max_attempts} attempts, ${sleep_s}s between, ${max_time_s}s per request"
echo "blind_stop: stop_on_ok=${stop_on_ok}"
echo
ok_count=0
busy_count=0
err_count=0
started=$(date +%s)
for ((i=1; i<=max_attempts; i++)); do
printf "[%4d] %s " "$i" "$(date +%H:%M:%S)"
http_code=$(curl -sS -o /tmp/blind_resp.$$ -w "%{http_code}" \
--max-time "$max_time_s" \
-X POST "$url" || echo "000")
body=$(cat /tmp/blind_resp.$$ 2>/dev/null || true)
rm -f /tmp/blind_resp.$$
case "$http_code" in
200|201)
ok_count=$((ok_count + 1))
echo "SENT $body"
if [[ "$stop_on_ok" == "1" ]]; then
elapsed=$(( $(date +%s) - started ))
echo
echo "blind_stop: success after ${i} attempts (${elapsed}s). ok=${ok_count} busy=${busy_count} err=${err_count}"
echo "blind_stop: NEXT — wait ~10s, then try the full rescue:"
echo " /home/serversdown/seismo-relay/scripts/rescue_device.sh ${host} ${tcp_port}"
exit 0
fi
;;
503)
busy_count=$((busy_count + 1))
echo "busy (503)"
;;
000)
err_count=$((err_count + 1))
echo "curl error"
;;
*)
err_count=$((err_count + 1))
echo "HTTP $http_code $body" | head -c 400
echo
;;
esac
[[ "$sleep_s" != "0" ]] && sleep "$sleep_s"
done
elapsed=$(( $(date +%s) - started ))
echo
echo "blind_stop: gave up after ${max_attempts} attempts (${elapsed}s). ok=${ok_count} busy=${busy_count} err=${err_count}" >&2
exit 1
scripts/repair_unknown_serials.py
+151
View File
@@ -0,0 +1,151 @@
"""
scripts/repair_unknown_serials.py — re-attribute events stuck under
`serial = 'UNKNOWN'` to their correct serial by decoding the BW filename.
Why this is needed
──────────────────
The /db/import/blastware_file endpoint had a bug (fixed in commit a032fa5+1
on the ach-report-ingestion branch) where every forwarded event was inserted
with serial='UNKNOWN' because the endpoint's `_serial_from_event(ev)` stub
returned None and never consulted the BW-filename serial that
`WaveformStore.save_imported_bw()` had already decoded.
Effect on a server that ran a buggy version: every forwarded event's
SeismoDb row has `serial='UNKNOWN'`, even though the on-disk waveform
store has correctly bucketed the files into `BE<NNNN>/` folders. So
the BW binaries / sidecars / HDF5s are fine, but `/db/units` and
`/db/events?serial=...` queries don't surface the events.
This script
───────────
Walks the events table looking for rows with `serial='UNKNOWN'` and
re-attributes each one to the serial decoded from its
`blastware_filename` column. If the row's serial would collide with
an existing row (already-correct duplicate from a later re-forward),
the UNKNOWN row is deleted. Otherwise the row's `serial` column is
updated in-place.
Idempotent: re-running after a successful repair finds zero matching
rows and exits cleanly.
Usage
─────
# Dry-run (default): print what would change, don't touch the DB
python -m scripts.repair_unknown_serials --db bridges/captures/seismo_relay.db
# Apply the repair
python -m scripts.repair_unknown_serials --db bridges/captures/seismo_relay.db --apply
"""
from __future__ import annotations
import argparse
import sqlite3
import sys
from pathlib import Path
# Reach into sfm.waveform_store for the serial decoder. This script
# is run from the repo root via `python -m scripts.repair_unknown_serials`.
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from sfm.waveform_store import _serial_from_bw_filename
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(
description="Re-attribute events stuck under serial='UNKNOWN'.",
)
p.add_argument(
"--db", required=True, type=Path,
help="Path to seismo_relay.db (e.g. bridges/captures/seismo_relay.db)",
)
p.add_argument(
"--apply", action="store_true",
help="Apply the repair. Without this flag the script runs in "
"dry-run mode and only reports what would change.",
)
args = p.parse_args(argv)
if not args.db.exists():
print(f"DB not found: {args.db}", file=sys.stderr)
return 2
conn = sqlite3.connect(str(args.db))
conn.row_factory = sqlite3.Row
rows = list(conn.execute(
"SELECT id, serial, timestamp, blastware_filename "
" FROM events "
" WHERE serial = 'UNKNOWN' "
" ORDER BY timestamp",
))
print(f"Found {len(rows)} UNKNOWN-serial rows in events table.")
if not rows:
return 0
updated = 0
deleted = 0
unresolved = 0
by_serial: dict[str, int] = {}
for row in rows:
rid = row["id"]
ts = row["timestamp"]
bw_name = row["blastware_filename"]
new_serial = _serial_from_bw_filename(bw_name) if bw_name else None
if not new_serial:
print(f" ⚠ id={rid[:8]} ts={ts} filename={bw_name!r} — "
f"cannot decode serial from filename; skipping")
unresolved += 1
continue
# Check for an existing row at the target (serial, timestamp).
existing = conn.execute(
"SELECT id FROM events WHERE serial = ? AND timestamp = ?",
(new_serial, ts),
).fetchone()
action: str
if existing is None:
# Safe to UPDATE in place.
if args.apply:
conn.execute(
"UPDATE events SET serial = ? WHERE id = ?",
(new_serial, rid),
)
action = "UPDATE"
updated += 1
else:
# A correctly-attributed row already exists. Drop the
# UNKNOWN duplicate.
if args.apply:
conn.execute("DELETE FROM events WHERE id = ?", (rid,))
action = "DELETE (dup)"
deleted += 1
by_serial[new_serial] = by_serial.get(new_serial, 0) + 1
print(f" {action:14s} id={rid[:8]} ts={ts} "
f"filename={bw_name} → {new_serial}")
if args.apply:
conn.commit()
conn.close()
print()
print(f"Summary:")
print(f" UNKNOWN rows scanned: {len(rows)}")
print(f" Updated to real serial: {updated}")
print(f" Deleted (duplicate of an ")
print(f" already-correct row): {deleted}")
print(f" Unresolved (bad filename): {unresolved}")
print()
if by_serial:
print(f"Per-serial breakdown of repaired rows:")
for serial, count in sorted(by_serial.items()):
print(f" {serial:12s} {count}")
if not args.apply:
print()
print("(dry-run — re-run with --apply to commit)")
return 0
if __name__ == "__main__":
sys.exit(main())
scripts/rescue_device.sh
Executable
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env bash
# Rescue an uncooperative MiniMate that's busy with another ACH session.
#
# Hammers POST /device/rescue in a tight loop with a short timeout. When the
# device is in an ACH session our SYN either gets refused or silently dropped
# (5s connect timeout inside the endpoint) and we retry immediately. When the
# device is between sessions, our TCP wins, the endpoint disables Auto Call
# Home and erases events inside the same session, then returns success.
#
# Usage:
# ./rescue_device.sh <host> [tcp_port] [--no-erase] [--no-disable-ach]
#
# Examples:
# ./rescue_device.sh 166.246.130.1 9034
# ./rescue_device.sh 166.246.130.1 9034 --no-erase # just silence it
#
# Environment:
# SFM_BASE_URL Defaults to http://localhost:8200 (SFM direct).
# Set to http://localhost:8001/api/sfm to route through
# Terra-View's proxy. Direct mode avoids the proxy's
# 60s timeout, which matters for long-running endpoints.
# MAX_ATTEMPTS Cap on retries (default 600 ≈ 30+ min).
# SLEEP_S Backoff between attempts (default 1).
# MAX_TIME_S Per-request timeout (default 60).
# CONNECT_TIMEOUT TCP connect timeout (default 5).
# RECV_TIMEOUT Per-frame S3 recv timeout (default 5). If POLL or any
# subsequent frame doesn't respond within this window, the
# rescue endpoint bails and this script retries.
set -u
host="${1:-}"
tcp_port="${2:-9034}"
shift 2 2>/dev/null || shift $# 2>/dev/null
if [[ -z "$host" ]]; then
echo "usage: $0 <host> [tcp_port] [--no-erase] [--no-disable-ach]" >&2
exit 2
fi
disable_ach="true"
erase="true"
for arg in "$@"; do
case "$arg" in
--no-erase) erase="false" ;;
--no-disable-ach) disable_ach="false" ;;
*) echo "unknown flag: $arg" >&2; exit 2 ;;
esac
done
base="${SFM_BASE_URL:-http://localhost:8200}"
max_attempts="${MAX_ATTEMPTS:-600}"
sleep_s="${SLEEP_S:-1}"
max_time_s="${MAX_TIME_S:-60}"
connect_timeout="${CONNECT_TIMEOUT:-5}"
recv_timeout="${RECV_TIMEOUT:-5}"
url="${base}/device/rescue?host=${host}&tcp_port=${tcp_port}&disable_ach=${disable_ach}&erase=${erase}&connect_timeout=${connect_timeout}&recv_timeout=${recv_timeout}"
echo "rescue: target ${host}:${tcp_port} disable_ach=${disable_ach} erase=${erase}"
echo "rescue: connect_timeout=${connect_timeout}s recv_timeout=${recv_timeout}s"
echo "rescue: POST ${url}"
echo "rescue: up to ${max_attempts} attempts, ${sleep_s}s between, ${max_time_s}s per request"
echo
started=$(date +%s)
for ((i=1; i<=max_attempts; i++)); do
printf "[%3d] %s " "$i" "$(date +%H:%M:%S)"
http_code=$(curl -sS -o /tmp/rescue_resp.$$ -w "%{http_code}" \
--max-time "$max_time_s" \
-X POST "$url" || echo "000")
body=$(cat /tmp/rescue_resp.$$ 2>/dev/null || true)
rm -f /tmp/rescue_resp.$$
case "$http_code" in
200|201)
elapsed=$(( $(date +%s) - started ))
echo "OK (${elapsed}s total)"
echo "$body"
exit 0
;;
503)
# Connection refused / timeout — device busy in another session. Retry fast.
echo "busy (503)"
;;
000)
echo "curl error (network)"
;;
*)
echo "HTTP $http_code"
echo " $body" | head -c 400
echo
;;
esac
sleep "$sleep_s"
done
echo "rescue: gave up after ${max_attempts} attempts" >&2
exit 1
scripts/slow_drip.sh
Executable
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env bash
# Hold a single TCP session open and drip stop-monitoring frames at a slow
# rate, so the device's UART RX FIFO has time to drain between sends.
#
# Use when high-rate spam isn't landing — typically because the device's
# firmware is too busy to drain its serial buffer fast enough and bytes
# are being lost to UART overrun.
#
# Usage:
# ./slow_drip.sh <host> [tcp_port] [duration_s]
#
# Env:
# DURATION Default: 120 (seconds; arg 3 overrides). Clamped 1..600.
# INTERVAL Seconds between drip sends (default 3). Lower = more
# aggressive, more risk of FIFO overrun. Higher = safer
# but fewer total drips per duration.
# CONNECT_TIMEOUT Default: 5
# SFM_BASE_URL Default: http://localhost:8200 (SFM direct).
set -u
host="${1:-}"
tcp_port="${2:-9034}"
duration="${3:-${DURATION:-120}}"
if [[ -z "$host" ]]; then
echo "usage: $0 <host> [tcp_port] [duration_s]" >&2
exit 2
fi
base="${SFM_BASE_URL:-http://localhost:8200}"
interval="${INTERVAL:-3}"
connect_timeout="${CONNECT_TIMEOUT:-5}"
url="${base}/device/stop_monitoring_slow_drip?host=${host}&tcp_port=${tcp_port}&duration_s=${duration}&interval_s=${interval}&connect_timeout=${connect_timeout}"
echo "slow_drip: target ${host}:${tcp_port} duration=${duration}s interval=${interval}s connect_timeout=${connect_timeout}s"
echo "slow_drip: POST ${url}"
echo
# Give curl enough slack to wait out the duration plus a buffer
max_time=$(awk -v d="$duration" 'BEGIN { printf "%d", d + 30 }')
curl -sS --max-time "$max_time" -X POST "$url"
echo
scripts/spam_stop.sh
Executable
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
# Hammer a device with blind stop-monitoring sessions as fast as possible.
# Single HTTP call kicks off the burst inside SFM (no per-attempt HTTP
# overhead). Default: 10 seconds, ~500 ms per attempt = ~20 attempts/sec.
#
# Usage:
# ./spam_stop.sh <host> [tcp_port] [duration_s]
#
# Examples:
# ./spam_stop.sh 166.246.130.1 # 10s burst
# ./spam_stop.sh 166.246.130.1 9034 30 # 30s burst
# DURATION=60 CONNECT_TIMEOUT=0.2 ./spam_stop.sh 166.246.130.1
#
# Env:
# SFM_BASE_URL Default: http://localhost:8200 (SFM direct).
# Set to http://localhost:8001/api/sfm to route through
# Terra-View's proxy — but note the proxy has a 60s
# timeout, so long bursts need direct mode.
# DURATION Default: 10 (seconds; arg 3 overrides)
# CONNECT_TIMEOUT Default: 0.5 (seconds)
# REPEAT Default: 3 (stop frames per TCP session)
set -u
host="${1:-}"
tcp_port="${2:-9034}"
duration="${3:-${DURATION:-10}}"
if [[ -z "$host" ]]; then
echo "usage: $0 <host> [tcp_port] [duration_s]" >&2
exit 2
fi
base="${SFM_BASE_URL:-http://localhost:8200}"
connect_timeout="${CONNECT_TIMEOUT:-0.5}"
repeat="${REPEAT:-3}"
url="${base}/device/stop_monitoring_spam?host=${host}&tcp_port=${tcp_port}&duration_s=${duration}&connect_timeout=${connect_timeout}&repeat=${repeat}"
echo "spam_stop: target ${host}:${tcp_port} duration=${duration}s connect_timeout=${connect_timeout}s repeat=${repeat}"
echo "spam_stop: POST ${url}"
echo
# Give curl enough slack to wait out the duration plus a buffer
max_time=$(awk -v d="$duration" 'BEGIN { printf "%d", d + 10 }')
curl -sS --max-time "$max_time" -X POST "$url"
echo
scripts/watch_unit.sh
Executable
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env bash
# Passive monitor for a misbehaving unit. Every INTERVAL seconds, attempts
# a single short TCP probe + storage_range read and logs the result. Designed
# to run unattended for hours/days and tell you when the unit comes back.
#
# Usage:
# ./watch_unit.sh <host> [tcp_port]
#
# Env:
# INTERVAL Seconds between checks (default 300 = 5 min)
# LOG_FILE Append results here (default /tmp/watch_<host>.log)
# SFM_BASE_URL Default: http://localhost:8200
set -u
host="${1:-}"
tcp_port="${2:-9034}"
if [[ -z "$host" ]]; then
echo "usage: $0 <host> [tcp_port]" >&2
exit 2
fi
interval="${INTERVAL:-300}"
log_file="${LOG_FILE:-/tmp/watch_${host}.log}"
base="${SFM_BASE_URL:-http://localhost:8200}"
url="${base}/device/events/storage_range?host=${host}&tcp_port=${tcp_port}"
echo "watch_unit: target ${host}:${tcp_port} interval=${interval}s log=${log_file}"
echo "watch_unit: Ctrl-C to stop"
while true; do
ts=$(date '+%Y-%m-%d %H:%M:%S')
http_code=$(curl -sS -o /tmp/watch_resp.$$ -w "%{http_code}" \
--max-time 20 "$url" || echo "000")
body=$(cat /tmp/watch_resp.$$ 2>/dev/null || true)
rm -f /tmp/watch_resp.$$
case "$http_code" in
200|201)
# Strip the raw_hex for readability
summary=$(echo "$body" | sed 's/"raw_hex":"[^"]*",*//; s/,*$//' | head -c 200)
echo "$ts REACHABLE $summary" | tee -a "$log_file"
;;
502|503)
err=$(echo "$body" | head -c 150)
echo "$ts ERROR_$http_code $err" | tee -a "$log_file"
;;
000)
echo "$ts CURL_FAIL (network/timeout)" | tee -a "$log_file"
;;
*)
echo "$ts HTTP_$http_code $(echo "$body" | head -c 150)" | tee -a "$log_file"
;;
esac
sleep "$interval"
done
sfm/database.py
+242 -26
View File
@@ -85,6 +85,7 @@ CREATE TABLE IF NOT EXISTS events (
blastware_filesize INTEGER, -- bytes; NULL if no event file saved
a5_pickle_filename TEXT, -- "<filename>.a5.pkl" sidecar
sidecar_filename TEXT, -- "<filename>.sfm.json" review/metadata sidecar
device_family TEXT, -- "series3" (MiniMate Plus / BW) | "series4" (Micromate / Thor) — drives per-family UI rendering (units, labels)
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
UNIQUE(serial, timestamp)
);
@@ -198,11 +199,53 @@ class SeismoDb:
("blastware_filesize", "INTEGER"),
("a5_pickle_filename", "TEXT"),
("sidecar_filename", "TEXT"),
("device_family", "TEXT"),
):
if col not in existing_cols:
log.info("_migrate: events ADD COLUMN %s %s", col, ddl)
conn.execute(f"ALTER TABLE events ADD COLUMN {col} {ddl}")
# Migration 1c: backfill device_family for existing rows by sniffing
# the device-native binary filename's extension. Thor (Micromate
# Series IV) writes `.IDFH` / `.IDFW`; MiniMate Plus (Series III)
# writes `.AB0*` / `.N00` / `.<base36>` Blastware extensions. We do
# this here rather than from sidecars so the migration is fully
# self-contained (doesn't need the waveform-store root) and runs at
# DB-init time. Only fills NULL device_family so re-runs are no-ops.
rebackfill = conn.execute(
"SELECT COUNT(*) FROM events WHERE device_family IS NULL"
).fetchone()
if rebackfill and rebackfill[0] > 0:
log.info("_migrate: backfilling device_family for %d events", rebackfill[0])
# Series IV (Thor IDF) — extension is exactly .IDFH or .IDFW
conn.execute(
"""
UPDATE events
SET device_family = 'series4'
WHERE device_family IS NULL
AND (
UPPER(blastware_filename) LIKE '%.IDFH'
OR UPPER(blastware_filename) LIKE '%.IDFW'
)
"""
)
# Everything else with a filename → Series III (Blastware family)
conn.execute(
"""
UPDATE events
SET device_family = 'series3'
WHERE device_family IS NULL
AND blastware_filename IS NOT NULL
"""
)
# Rows with no filename (e.g. older monitor_log-derived events)
# stay NULL — UI handles NULL as "unknown family".
remaining = conn.execute(
"SELECT COUNT(*) FROM events WHERE device_family IS NULL"
).fetchone()[0]
log.info("_migrate: device_family backfill complete (remaining NULL=%d)",
remaining)
# Migration 2: change monitor_log UNIQUE from (serial, waveform_key) to
# (serial, start_time) — same reasoning as events.
row = conn.execute(
@@ -302,6 +345,7 @@ class SeismoDb:
serial: str,
session_id: Optional[str] = None,
waveform_records: Optional[dict[str, dict]] = None,
device_family: Optional[str] = None,
) -> tuple[int, int]:
"""
Insert triggered events. Silently skips duplicates (serial+timestamp).
@@ -316,6 +360,11 @@ class SeismoDb:
(dedup hit), the matching waveform record is upserted onto the
existing row so a re-download via the live endpoint refreshes the
file metadata.
``device_family`` (optional): "series3" (MiniMate Plus / Blastware) or
"series4" (Micromate / Thor). Drives per-family UI rendering — most
importantly the mic-unit convention (psi vs dB(L)). Set on every
insert and overwritten on every UPSERT so the latest writer wins.
"""
inserted = skipped = 0
wave_recs = waveform_records or {}
@@ -349,8 +398,9 @@ class SeismoDb:
project, client, operator, sensor_location,
sample_rate, record_type,
blastware_filename, blastware_filesize,
a5_pickle_filename, sidecar_filename)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
a5_pickle_filename, sidecar_filename,
device_family)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
self._new_id(), serial, key, session_id, ts,
@@ -369,29 +419,68 @@ class SeismoDb:
rec.get("filesize"),
rec.get("a5_pickle_filename"),
rec.get("sidecar_filename"),
device_family,
),
)
inserted += 1
except sqlite3.IntegrityError:
skipped += 1
# Upsert waveform fields onto the existing dedup row so a
# re-download via the live endpoint refreshes filename /
# size / sidecar without churning the rest of the row.
if rec and ts:
# UPSERT path: a row for this (serial, timestamp) already
# exists. Refresh every device-authoritative field from
# the new data so that a re-import with better data (e.g.
# a watcher re-forward where the previous attempt missed
# the paired BW ASCII report) replaces stale peaks /
# project info / sample_rate.
#
# Preserved (not in this UPDATE):
# id, waveform_key, session_id, created_at — immutable / FK
# false_trigger — operator review state
#
# Behaviour change vs prior versions: this UPDATE used
# to only refresh filename / filesize / a5_pickle /
# sidecar fields. As a result, the first insert's
# broken-codec peak values were locked in forever even
# if subsequent re-forwards arrived with correct
# report-derived values. Now every re-import lifts the
# DB row up to whatever the latest Event carries.
conn.execute(
"""
UPDATE events
SET blastware_filename = ?,
SET tran_ppv = ?,
vert_ppv = ?,
long_ppv = ?,
peak_vector_sum = ?,
mic_ppv = ?,
project = ?,
client = ?,
operator = ?,
sensor_location = ?,
sample_rate = ?,
record_type = ?,
blastware_filename = ?,
blastware_filesize = ?,
a5_pickle_filename = ?,
sidecar_filename = ?
sidecar_filename = ?,
device_family = COALESCE(?, device_family)
WHERE serial = ? AND timestamp = ?
""",
(
rec.get("filename"),
rec.get("filesize"),
rec.get("a5_pickle_filename"),
rec.get("sidecar_filename"),
pv.tran if pv else None,
pv.vert if pv else None,
pv.long if pv else None,
pv.peak_vector_sum if pv else None,
pv.micl if pv else None,
pi.project if pi else None,
pi.client if pi else None,
pi.operator if pi else None,
pi.sensor_location if pi else None,
ev.sample_rate,
ev.record_type,
rec.get("filename") if rec else None,
rec.get("filesize") if rec else None,
rec.get("a5_pickle_filename") if rec else None,
rec.get("sidecar_filename") if rec else None,
device_family,
serial,
ts,
),
@@ -455,6 +544,75 @@ class SeismoDb:
)
return cur.rowcount > 0
def delete_event(self, event_id: str) -> Optional[dict]:
"""
Hard-delete one event row by id. Returns the deleted row (so the
caller can clean up any on-disk files referenced by it) or None
if no row matched.
"""
with self._connect() as conn:
row = conn.execute(
"SELECT * FROM events WHERE id = ?", (event_id,),
).fetchone()
if row is None:
return None
conn.execute("DELETE FROM events WHERE id = ?", (event_id,))
return dict(row)
def delete_events_bulk(
self,
serial: Optional[str] = None,
from_dt: Optional[datetime.datetime] = None,
to_dt: Optional[datetime.datetime] = None,
false_trigger: Optional[bool] = None,
ids: Optional[list[str]] = None,
) -> list[dict]:
"""
Hard-delete events matching the given filters. Returns the list
of deleted row dicts. Refuses to delete with no filters at all
(would wipe the whole table) — raises ValueError.
Filter semantics match query_events: serial / from_dt / to_dt /
false_trigger combine with AND. `ids` is an additional inclusion
list (event_id IN (...)); if supplied alongside other filters,
only rows matching all conditions are deleted.
"""
clauses: list[str] = []
params: list = []
if serial:
clauses.append("serial = ?")
params.append(serial)
if from_dt:
clauses.append("timestamp >= ?")
params.append(from_dt.isoformat())
if to_dt:
clauses.append("timestamp <= ?")
params.append(to_dt.isoformat())
if false_trigger is not None:
clauses.append("false_trigger = ?")
params.append(1 if false_trigger else 0)
if ids:
placeholders = ",".join("?" * len(ids))
clauses.append(f"id IN ({placeholders})")
params.extend(ids)
if not clauses:
raise ValueError(
"delete_events_bulk refuses to delete with no filters "
"(would wipe the entire events table)"
)
where = "WHERE " + " AND ".join(clauses)
with self._connect() as conn:
rows = conn.execute(
f"SELECT * FROM events {where}", params,
).fetchall()
if rows:
conn.execute(f"DELETE FROM events {where}", params)
return [dict(r) for r in rows]
def update_event_review(self, event_id: str, review: dict) -> bool:
"""
Sync derived index columns from a sidecar's `review` block.
@@ -564,21 +722,79 @@ class SeismoDb:
def query_units(self) -> list[dict]:
"""
Return one row per known serial with summary stats:
last_seen, total_events, total_monitor_entries.
Return one row per known serial with summary stats.
Aggregates from BOTH source tables:
- `events` — populated by every ingest path
(live ACH, /db/import/blastware_file
from the series3-watcher forwarder, etc.)
- `ach_sessions` — only populated by the live ACH server;
empty for events that came in via the
BW-importer route.
Earlier this method only joined on `ach_sessions`, which made
watcher-forwarded units invisible to the SFM webapp's fleet
overview even though their events were correctly populated in
`events`. Now we union the two and surface every serial that
has activity in either table.
Fields:
serial — unit serial number (e.g. "BE11529")
last_seen — most recent of MAX(events.timestamp)
and MAX(ach_sessions.session_time)
total_events — COUNT(*) from `events` (the
authoritative count regardless of
ingest path)
total_monitor_entries — from `ach_sessions`, 0 when absent
total_sessions — COUNT(*) from `ach_sessions`, 0 when absent
"""
with self._connect() as conn:
rows = conn.execute(
"""
SELECT
s.serial,
MAX(s.session_time) AS last_seen,
SUM(s.events_downloaded) AS total_events,
SUM(s.monitor_entries) AS total_monitor_entries,
COUNT(*) AS total_sessions
FROM ach_sessions s
GROUP BY s.serial
ORDER BY last_seen DESC
event_stats = {
row["serial"]: row
for row in conn.execute(
"""
SELECT serial,
MAX(timestamp) AS last_event_at,
COUNT(*) AS total_events
FROM events
GROUP BY serial
""",
).fetchall()
return [dict(r) for r in rows]
}
session_stats = {
row["serial"]: row
for row in conn.execute(
"""
SELECT serial,
MAX(session_time) AS last_session_at,
SUM(monitor_entries) AS total_monitor_entries,
COUNT(*) AS total_sessions
FROM ach_sessions
GROUP BY serial
""",
).fetchall()
}
all_serials = set(event_stats) | set(session_stats)
units = []
for serial in all_serials:
e = event_stats.get(serial)
s = session_stats.get(serial)
last_event_at = e["last_event_at"] if e else None
last_session_at = s["last_session_at"] if s else None
# Prefer whichever timestamp is more recent
last_seen = max(
(t for t in (last_event_at, last_session_at) if t),
default=None,
)
units.append({
"serial": serial,
"last_seen": last_seen,
"total_events": e["total_events"] if e else 0,
"total_monitor_entries": s["total_monitor_entries"] if s else 0,
"total_sessions": s["total_sessions"] if s else 0,
})
# Sort by last_seen desc; serials with no timestamp at all sink to the bottom.
units.sort(key=lambda u: u.get("last_seen") or "", reverse=True)
return units
sfm/dump_0c.py
-216
View File
@@ -1,216 +0,0 @@
"""
sfm.dump_0c — inspect the raw 210-byte SUB 0C waveform record stored in a
sidecar JSON's `extensions.raw_records.waveform_record_b64`.
Usage:
python -m sfm.dump_0c <sidecar.sfm.json> [<sidecar.sfm.json> ...]
Prints, for each input:
- A header summarising the sidecar's metadata-block claims (peaks,
project, timestamp) — the "what BW says this event measured" view.
- A 16-byte-wide hex dump of the raw 0C record, annotated with known
field anchors (STRT, channel labels, project strings).
- A "candidate float regions" scan that brute-forces every byte
position as a float32 BE and prints any that yield a value in a
plausible range (1e-7 to 1e3) — useful for hunting where Peak
Acceleration / Peak Displacement / ZC Freq / Time of Peak live.
Pairing the printed candidates with the BW Event Report values lets
us nail down byte offsets for the missing fields without a live
device.
"""
from __future__ import annotations
import argparse
import base64
import json
import struct
import sys
from pathlib import Path
# ── Annotations for known anchors in a 210-byte 0C record ──────────────────
# Anchors we look for and label inline in the hex dump. Each is a needle
# (bytes to find) and a short label. Found via .find() — the first
# occurrence wins.
_ANCHORS = [
(b"Tran", "Tran label (PPV @ +6, PVS @ -12)"),
(b"Vert", "Vert label (PPV @ +6)"),
(b"Long", "Long label (PPV @ +6)"),
(b"MicL", "MicL label (peak psi @ +6)"),
(b"Project:", "Project: label"),
(b"Client:", "Client: label"),
(b"User Name:", "User Name: label"),
(b"Seis Loc:", "Seis Loc: label"),
(b"Extended Notes", "Extended Notes label"),
]
def _hex_dump(data: bytes, anchors: dict[int, str]) -> str:
"""Return a 16-byte-wide hex+ASCII dump, with anchor labels printed
on the line that contains the anchor's start byte."""
lines = []
for off in range(0, len(data), 16):
chunk = data[off : off + 16]
hex_part = " ".join(f"{b:02x}" for b in chunk)
ascii_part = "".join(chr(b) if 32 <= b < 127 else "." for b in chunk)
line = f" {off:04x} {hex_part:<47} |{ascii_part}|"
# If any anchor lands on a byte in this row, append a tag
tags = [
f"[{a:#04x}: {label}]"
for a, label in anchors.items()
if off <= a < off + 16
]
if tags:
line += " " + " ".join(tags)
lines.append(line)
return "\n".join(lines)
def _scan_float32_be(data: bytes, lo: float, hi: float) -> list[tuple[int, float]]:
"""Brute-force every offset where data[off:off+4] is a float32 BE in
(lo, hi). Includes negatives in the symmetric range."""
hits = []
for i in range(len(data) - 3):
try:
v = struct.unpack_from(">f", data, i)[0]
except struct.error:
continue
if v != v: # NaN
continue
if abs(v) < 1e-30 or abs(v) > 1e10: # crap range
continue
a = abs(v)
if lo <= a <= hi:
hits.append((i, v))
return hits
def _scan_uint16_be(data: bytes, lo: int, hi: int) -> list[tuple[int, int]]:
"""Find every offset where uint16 BE is in [lo, hi]."""
hits = []
for i in range(len(data) - 1):
v = (data[i] << 8) | data[i + 1]
if lo <= v <= hi:
hits.append((i, v))
return hits
def _summarize_sidecar(side: dict) -> str:
ev = side.get("event", {})
pv = side.get("peak_values", {})
pi = side.get("project_info", {})
bw = side.get("blastware", {})
return (
f" serial: {ev.get('serial')}\n"
f" timestamp: {ev.get('timestamp')}\n"
f" waveform: {ev.get('waveform_key')} ({ev.get('record_type')})\n"
f" sample_rate:{ev.get('sample_rate')} sps rectime:{ev.get('rectime_seconds')}s\n"
f" bw file: {bw.get('filename')} ({bw.get('filesize')} B)\n"
f" peaks: "
f"Tran={pv.get('transverse'):.5f} "
f"Vert={pv.get('vertical'):.5f} "
f"Long={pv.get('longitudinal'):.5f} "
f"PVS={pv.get('vector_sum'):.5f} in/s "
f"Mic={pv.get('mic_psi'):.6e} psi"
if all(pv.get(k) is not None for k in
("transverse", "vertical", "longitudinal", "vector_sum", "mic_psi"))
else f" peaks: {pv}\n project: {pi}"
) + (
f"\n project: {pi.get('project')!r} / {pi.get('client')!r} / "
f"operator={pi.get('operator')!r} loc={pi.get('sensor_location')!r}"
)
def dump_one(path: Path) -> int:
side = json.loads(path.read_text(encoding="utf-8"))
raw_b64 = (
side.get("extensions", {})
.get("raw_records", {})
.get("waveform_record_b64")
)
if not raw_b64:
print(f"\n=== {path} ===")
print(" ! no extensions.raw_records.waveform_record_b64 — sidecar")
print(" pre-dates raw-0C persistence (added in v0.15.x). Re-save")
print(" the event from the device to capture the bytes.")
return 1
raw = base64.b64decode(raw_b64)
# Build anchor map
anchors: dict[int, str] = {}
for needle, label in _ANCHORS:
i = raw.find(needle)
if i >= 0:
anchors[i] = label
print(f"\n=== {path} ===")
print("metadata claimed by sidecar:")
print(_summarize_sidecar(side))
print(f"\nraw 0C record ({len(raw)} bytes):")
print(_hex_dump(raw, anchors))
# Float32 BE candidates in geo-relevant ranges
geo_hits = _scan_float32_be(raw, 1e-5, 50.0)
# Filter: only show hits that are NOT trivially the per-channel labels'
# +6 PPV floats already documented (those will land in any sweep too).
print("\nfloat32 BE candidates (1e-5 .. 50.0):")
for off, v in geo_hits:
annotation = ""
for needle, _ in _ANCHORS[:4]: # geo + mic labels
i = raw.find(needle)
if i >= 0 and off == i + 6:
annotation = f" ← {needle.decode()} PPV (label+6)"
break
print(f" {off:#04x} ({off:3d}) {v:>+15.6f}{annotation}")
print("\nuint16 BE candidates ZC-Freq-ish (1..200):")
for off, v in _scan_uint16_be(raw, 1, 200):
if v < 5: # too noisy at very low end
continue
print(f" {off:#04x} ({off:3d}) = {v}")
print("\nuint16 BE candidates Time-of-Peak-ish if stored as ms (1..30000):")
for off, v in _scan_uint16_be(raw, 1, 30000):
if v < 100: # noise filter
continue
# Only the first ~80 are worth showing — too many hits otherwise
if off > 80:
break
print(f" {off:#04x} ({off:3d}) = {v} ms ?")
print()
return 0
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(
description="Inspect a saved 0C waveform record from a sidecar JSON.",
)
p.add_argument(
"sidecars",
nargs="+",
type=Path,
help="Path(s) to <event>.sfm.json sidecar file(s).",
)
args = p.parse_args(argv)
rc = 0
for path in args.sidecars:
try:
rc |= dump_one(path)
except Exception as exc:
print(f"\n=== {path} ===\n ERROR: {exc}", file=sys.stderr)
rc |= 2
return rc
if __name__ == "__main__":
sys.exit(main())
sfm/import_bw.py
+1
View File
@@ -166,6 +166,7 @@ def main(argv: list[str] | None = None) -> int:
{ev._waveform_key.hex(): rec}
if ev._waveform_key else None
),
device_family="series3",
)
tag = "OK " if ins else ("SKIP" if sk else "OK ")
print(f" [{tag}] {path.name} → {rec['filename']} "
sfm/server.py
+924 -9
View File
File diff suppressed because it is too large Load Diff
sfm/sfm_webapp.html
+36 -12
View File
@@ -2285,13 +2285,16 @@ let sessLoaded = false;
const _unitSerials = new Set();
function _ppvClass(v) {
if (v == null) return '';
if (v >= 2.0) return 'ppv-high';
if (v >= 0.5) return 'ppv-warn';
const n = (v == null) ? null : Number(v);
if (n == null || !isFinite(n)) return '';
if (n >= 2.0) return 'ppv-high';
if (n >= 0.5) return 'ppv-warn';
return 'ppv-ok';
}
function _ppvFmt(v) {
return v != null ? v.toFixed(5) : '—';
if (v == null) return '—';
const n = typeof v === 'number' ? v : Number(v);
return isFinite(n) ? n.toFixed(5) : String(v);
}
function _fmtTs(ts) {
if (!ts) return '—';
@@ -2386,7 +2389,14 @@ async function loadHistory() {
<td class="${_ppvClass(ev.vert_ppv)}">${_ppvFmt(ev.vert_ppv)}</td>
<td class="${_ppvClass(ev.long_ppv)}">${_ppvFmt(ev.long_ppv)}</td>
<td class="${_ppvClass(pvs)}">${_ppvFmt(pvs)}</td>
<td class="td-dim">${ev.mic_ppv != null && ev.mic_ppv > 0 ? (20 * Math.log10(ev.mic_ppv / DBL_REF)).toFixed(1) + ' dBL' : '—'}</td>
<td class="td-dim">${(() => {
const m = ev.mic_ppv == null ? null : Number(ev.mic_ppv);
if (m == null || !isFinite(m) || m <= 0) return '—';
// Series III (MiniMate Plus / BW) stores mic_ppv as psi → convert.
// Series IV (Micromate / Thor) already stores dB(L) → display direct.
if (ev.device_family === 'series4') return m.toFixed(1) + ' dBL';
return (20 * Math.log10(m / DBL_REF)).toFixed(1) + ' dBL';
})()}</td>
<td class="td-text">${ev.project ?? '—'}</td>
<td class="td-text">${ev.client ?? '—'}</td>
<td class="td-dim">${ev.record_type ?? '—'}</td>
@@ -2447,11 +2457,25 @@ function _renderSidecar(data) {
document.getElementById('sc-title').textContent = `Event — ${bw.filename || ev.waveform_key || 'unknown'}`;
const fmtPpv = v => (v == null ? '—' : Number(v).toFixed(5) + ' in/s');
const fmtPpv = v => {
if (v == null) return '—';
const n = Number(v);
return isFinite(n) ? n.toFixed(5) + ' in/s' : String(v);
};
// Map sidecar source.kind → device family (Series IV ingest path is
// "idf-import"; everything else is Series III today). The events-list
// table uses ev.device_family from the DB row, but sidecars don't carry
// that column — source.kind is the equivalent signal here.
const family = ((src.kind || '') === 'idf-import') ? 'series4' : 'series3';
const fmtMic = v => {
if (v == null || v <= 0) return '—';
const dbl = 20 * Math.log10(v / DBL_REF);
return `${dbl.toFixed(1)} dBL (${v.toExponential(2)} psi)`;
if (v == null) return '—';
const n = Number(v);
if (!isFinite(n) || n <= 0) return '—';
// Series IV (Micromate / Thor) stores mic as dB(L); Series III (BW)
// stores it as psi and we render both for cross-reference.
if (family === 'series4') return `${n.toFixed(1)} dBL`;
const dbl = 20 * Math.log10(n / DBL_REF);
return `${dbl.toFixed(1)} dBL (${n.toExponential(2)} psi)`;
};
document.getElementById('sc-f-serial').textContent = ev.serial || '—';
@@ -2746,9 +2770,9 @@ document.getElementById('api-base').value = window.location.origin;
<div class="sc-section">
<h4>Source / files</h4>
<dl class="sc-grid">
<dt>BW filename</dt> <dd id="sc-f-bw">—</dd>
<dt>BW filesize</dt> <dd id="sc-f-bwsize">—</dd>
<dt>BW sha256</dt> <dd id="sc-f-sha">—</dd>
<dt id="sc-l-bw">Event file</dt> <dd id="sc-f-bw">—</dd>
<dt id="sc-l-bwsize">File size</dt> <dd id="sc-f-bwsize">—</dd>
<dt id="sc-l-sha">File sha256</dt> <dd id="sc-f-sha">—</dd>
<dt>Source kind</dt> <dd id="sc-f-src">—</dd>
<dt>Captured at</dt> <dd id="sc-f-cap">—</dd>
</dl>
sfm/waveform_store.py
+171 -4
View File
@@ -34,7 +34,7 @@ import logging
import pickle
import shutil
from pathlib import Path
from typing import Optional
from typing import Optional, Union
from minimateplus import event_file_io
from minimateplus.blastware_file import blastware_filename, write_blastware_file
@@ -258,6 +258,7 @@ class WaveformStore:
source_path: Path,
*,
serial_hint: Optional[str] = None,
bw_report_text: Optional[Union[str, bytes]] = None,
) -> tuple[Event, dict]:
"""
Ingest a Blastware event file produced by an external tool
@@ -267,10 +268,17 @@ class WaveformStore:
Workflow:
1. Parse the bytes via event_file_io.read_blastware_file (writes
a temp file to do that, since the parser takes a path).
2. Resolve serial from BW filename (`<P><serial3>...`) or use
2. Optionally parse a paired BW ASCII event report (the .TXT
file BW writes alongside the binary). When supplied, its
decoded fields land in the sidecar's `bw_report` block AND
overlay the device-authoritative peak values into the
top-level `peak_values` block. This is the right path for
the ACH-forwarder daemon use case where Blastware's own
ACH writes both files into the watch folder.
3. Resolve serial from BW filename (`<P><serial3>...`) or use
serial_hint. Falls back to "UNKNOWN".
3. Copy the BW bytes verbatim into <root>/<serial>/<filename>.
4. Write the .sfm.json sidecar with source.kind = "bw-import"
4. Copy the BW bytes verbatim into <root>/<serial>/<filename>.
5. Write the .sfm.json sidecar with source.kind = "bw-import"
and a5_pickle_filename = None. Does NOT write a .a5.pkl
(no A5 source available; byte-for-byte regeneration not
possible — the on-disk BW file IS the byte-for-byte source).
@@ -292,6 +300,47 @@ class WaveformStore:
except FileNotFoundError:
pass
# read_blastware_file derives record_type from its path arg, but
# that arg is the tmp file (suffix ".bw") — so override with the
# original filename's encoded type (H/W/M/E/C in the BW AB0T
# scheme). Without this override every BW-imported event lands
# in the DB with record_type="Waveform" regardless of the actual
# type (Histogram, Manual, etc.).
ev.record_type = event_file_io.derive_record_type_from_filename(
source_path.name
)
# Parse the BW ASCII report if one was supplied. Failures here
# are non-fatal: we still write the binary + sidecar without the
# rich derived fields.
bw_report = None
if bw_report_text is not None:
try:
from minimateplus.bw_ascii_report import parse_report
bw_report = parse_report(bw_report_text)
except Exception as exc:
log.warning(
"save_imported_bw: BW report parse failed: %s — continuing without it",
exc,
)
# If we have a report, overlay its device-authoritative fields
# (peaks, project, sample_rate, record_time) onto the Event
# BEFORE handing it to db.insert_events(). Without this overlay
# the DB row gets `peak_values` from _peaks_from_samples(), which
# runs the still-undecoded waveform codec on the BW body and
# produces ±10 in/s saturation values on every channel for every
# event. The sidecar JSON had the correct values via
# event_to_sidecar_dict(bw_report=...) but the DB columns didn't.
if bw_report is not None:
try:
event_file_io.apply_report_to_event(ev, bw_report)
except Exception as exc:
log.warning(
"save_imported_bw: failed to overlay report onto event: %s",
exc,
)
# Resolve serial. blastware_filename derives a 4-char prefix from
# the numeric serial (e.g. BE11529 → M529); we go the other way
# via the source filename if a hint wasn't given.
@@ -345,6 +394,7 @@ class WaveformStore:
source_kind="bw-import",
a5_pickle_filename=None,
review=existing_review,
bw_report=bw_report,
)
event_file_io.write_sidecar(sidecar_path, sidecar)
@@ -360,6 +410,123 @@ class WaveformStore:
"a5_pickle_filename": None,
"hdf5_filename": hdf5_filename,
"sidecar_filename": sidecar_path.name,
"serial": serial,
}
def save_imported_idf(
self,
idf_bytes: bytes,
source_path: Path,
*,
serial_hint: Optional[str] = None,
idf_report_text: Optional[Union[str, bytes]] = None,
) -> tuple[Optional["Event"], dict]:
"""
Ingest a Thor (Micromate Series IV) IDF event file (`.IDFW` or
`.IDFH`) produced by Thor's TXT exporter.
Thor binaries are stored as opaque bytes — seismo-relay doesn't
yet decode the proprietary IDF binary format (codec slot lives
at ``micromate/idf_file.py``). Device-authoritative metadata
comes from the paired ``.IDFW.txt`` / ``.IDFH.txt`` sidecar
when supplied.
Workflow:
1. Parse the paired TXT report (when supplied) via
``micromate.parse_idf_report`` → dict.
2. Wrap parsed dict + filename into a typed ``micromate.IdfEvent``.
3. Copy bytes verbatim into ``<root>/<serial>/<filename>``.
4. Bridge IdfEvent → ``minimateplus.Event`` (for the existing
sidecar / DB insert machinery) via
``IdfEvent.to_minimateplus_event(waveform_key)``.
5. Write the ``.sfm.json`` sidecar with
``source.kind = "idf-import"`` and the full raw IDF report
under ``extensions.idf_report``.
Returns ``(event, record_dict)`` so the endpoint can both insert
into SeismoDb and surface the parsed event.
"""
from micromate import IdfEvent, parse_idf_report
# Parse the .txt sidecar (best-effort; non-fatal on failure).
report_dict: dict = {}
if idf_report_text is not None:
try:
report_dict = parse_idf_report(idf_report_text)
except Exception as exc:
log.warning(
"save_imported_idf: report parse failed: %s — continuing without it",
exc,
)
# Build the typed IdfEvent. Filename is authoritative for
# (serial, timestamp, kind); the report's event_datetime takes
# precedence over the filename timestamp inside from_report().
idf_event = IdfEvent.from_report(report_dict, source_path.name)
# Operator-supplied serial_hint wins over the binary's filename
# prefix when both are present (e.g. callers passing a known-good
# serial that overrides a misnamed export).
serial = serial_hint or idf_event.serial or "UNKNOWN"
# Filesystem write.
filename = source_path.name
bw_path = self._serial_dir(serial) / filename
bw_path.write_bytes(idf_bytes)
filesize = bw_path.stat().st_size
sha256 = event_file_io.file_sha256(bw_path)
# _waveform_key dedups (serial, timestamp) rows in the events
# table. Use the binary's sha256 (first 16 bytes) as a stable
# surrogate — every distinct binary maps to a distinct row.
waveform_key = bytes.fromhex(sha256)[:16]
# Bridge to minimateplus.Event for the existing sidecar / DB
# insert paths. See IdfEvent.to_minimateplus_event() for the
# caveats of this bridge (mic units, missing fields → sidecar).
ev = idf_event.to_minimateplus_event(waveform_key)
# Write the sidecar. Source kind "idf-import" was added to the
# allow-list in event_file_io.event_to_sidecar_dict for this.
sidecar_path = self.sidecar_path_for(serial, filename)
existing_review = None
if sidecar_path.exists():
try:
existing_review = event_file_io.read_sidecar(sidecar_path).get("review")
except Exception:
pass
sidecar = event_file_io.event_to_sidecar_dict(
ev,
serial=serial,
blastware_filename=filename,
blastware_filesize=filesize,
blastware_sha256=sha256,
source_kind="idf-import",
a5_pickle_filename=None,
review=existing_review,
)
# Stash the full parsed IDF report under extensions so downstream
# consumers can recover the rich derived fields that don't fit
# the BW-shaped event model (Peak Acceleration / Displacement,
# Time of Peak, sensor self-check, calibration, firmware).
if report_dict:
sidecar["extensions"]["idf_report"] = report_dict
event_file_io.write_sidecar(sidecar_path, sidecar)
log.info(
"WaveformStore.save_imported_idf serial=%s filename=%s filesize=%d "
"report_attached=%s",
serial, filename, filesize, bool(report_dict),
)
return ev, {
"filename": filename,
"filesize": filesize,
"sha256": sha256,
"a5_pickle_filename": None,
"hdf5_filename": None,
"sidecar_filename": sidecar_path.name,
"serial": serial,
}
def load_a5(self, serial: str, filename: str) -> Optional[list[S3Frame]]:
tests/fixtures/5-11-26/BE11529.MLG
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/5-11-26/M529LL1A.SP0
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/5-11-26/M529LL1A.SP0.TXT
Vendored
+3386
View File
File diff suppressed because it is too large Load Diff
tests/fixtures/5-11-26/M529LL1A.SS0
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/5-11-26/M529LL1A.SS0.TXT
Vendored
+3137
View File
File diff suppressed because it is too large Load Diff
tests/fixtures/5-11-26/M529LL1A.SV0
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/5-11-26/M529LL1A.SV0.TXT
Vendored
+3137
View File
File diff suppressed because it is too large Load Diff
tests/fixtures/5-11-26/M529LL1L.JQ0
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/5-11-26/M529LL1L.JQ0.TXT
Vendored
+3387
View File
File diff suppressed because it is too large Load Diff
tests/fixtures/5-11-26/M529LL1L.V70
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/5-11-26/M529LL1L.V70.TXT
Vendored
+3387
View File
File diff suppressed because it is too large Load Diff
tests/fixtures/decode-re-5-8-26/event-a/M529LKVQ.6S0
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/decode-re-5-8-26/event-a/M59LKVQ.6S0.TXT
Vendored
+3386
View File
File diff suppressed because it is too large Load Diff
tests/fixtures/decode-re-5-8-26/event-b/M529LK5Q.RG0
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/decode-re-5-8-26/event-b/M529LK5Q.RG0.TXT
Vendored
+2362
View File
File diff suppressed because it is too large Load Diff
tests/fixtures/decode-re-5-8-26/event-c/M529LK44.AB0
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/decode-re-5-8-26/event-c/M529LK44.AB0.TXT
Vendored
+1338
View File
File diff suppressed because it is too large Load Diff
tests/fixtures/decode-re-5-8-26/event-d/M529LK2V.470
Vendored
BIN
View File
Binary file not shown.
tests/fixtures/decode-re-5-8-26/event-d/M529LK2V.470.TXT
Vendored
+1330
View File
File diff suppressed because it is too large Load Diff
tests/test_bw_ascii_report.py
+407
View File
@@ -0,0 +1,407 @@
"""
test_bw_ascii_report.py — parser for Blastware's per-event ASCII export.
Run:
python -m pytest tests/test_bw_ascii_report.py -q
"""
from __future__ import annotations
import datetime
import os
import sys
from pathlib import Path
import pytest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from minimateplus.bw_ascii_report import (
BwAsciiReport,
parse_report,
parse_report_file,
)
FIXTURES = Path(__file__).parent.parent / "decode-re" / "5-8-26"
def _fixture(event_name: str) -> Path:
"""Find the .TXT file inside a fixture event folder."""
for p in (FIXTURES / event_name).iterdir():
if p.suffix.lower() == ".txt":
return p
raise FileNotFoundError(f"no .TXT in {FIXTURES / event_name}")
# ── Identity / config ───────────────────────────────────────────────────────
def test_event_c_identity_and_config():
r = parse_report_file(_fixture("event-c"))
assert r.event_type == "Full Waveform"
assert r.serial == "BE11529"
assert r.file_name == "M529LK44.AB0"
assert r.event_datetime == datetime.datetime(2026, 4, 23, 15, 56, 35)
assert r.trigger_channel == "Vert"
assert r.geo_trigger_level_ips == pytest.approx(0.5)
assert r.pretrig_s == pytest.approx(-0.25)
assert r.record_time_s == pytest.approx(1.0)
assert r.record_stop_mode == "Fixed"
assert r.sample_rate_sps == 1024
assert r.battery_volts == pytest.approx(6.8)
assert r.calibration_date == datetime.date(2025, 4, 29)
assert r.calibration_by == "Instantel"
assert r.units == "in/s and dB(L)"
def test_event_c_operator_metadata():
r = parse_report_file(_fixture("event-c"))
# The "Project: : value" pattern (key has its own trailing colon)
# is handled by stripping the colon at lookup time.
assert r.project == "Test4-21-26"
assert r.client == "Test-Client1"
assert r.operator == "Brian and claude"
assert r.sensor_location == "catbed"
def test_event_c_geo_range():
r = parse_report_file(_fixture("event-c"))
assert r.geo_range_ips == pytest.approx(10.0)
# ── Per-channel derived stats ───────────────────────────────────────────────
def test_event_c_per_channel_stats():
r = parse_report_file(_fixture("event-c"))
tran = r.channels["Tran"]
assert tran.ppv_ips == pytest.approx(0.065)
assert tran.zc_freq_hz == pytest.approx(47.0)
assert tran.time_of_peak_s == pytest.approx(0.007)
assert tran.peak_accel_g == pytest.approx(0.066)
assert tran.peak_disp_in == pytest.approx(0.001)
vert = r.channels["Vert"]
assert vert.ppv_ips == pytest.approx(0.610)
assert vert.zc_freq_hz == pytest.approx(16.0)
assert vert.time_of_peak_s == pytest.approx(0.024)
assert vert.peak_accel_g == pytest.approx(0.437)
assert vert.peak_disp_in == pytest.approx(0.006)
long_ = r.channels["Long"]
assert long_.ppv_ips == pytest.approx(0.070)
assert long_.zc_freq_hz == pytest.approx(22.0)
assert long_.time_of_peak_s == pytest.approx(0.019)
assert long_.peak_accel_g == pytest.approx(0.040)
assert long_.peak_disp_in == pytest.approx(0.001)
def test_event_c_micl_stats():
r = parse_report_file(_fixture("event-c"))
# MicL specific block
assert r.mic.weighting == "Linear Weighting"
assert r.mic.pspl_dbl == pytest.approx(88.0)
assert r.mic.zc_freq_hz == pytest.approx(57.0)
assert r.mic.time_of_peak_s == pytest.approx(-0.004)
# Mirrored onto channels["MicL"] for uniform per-channel access
micl_ch = r.channels["MicL"]
assert micl_ch.zc_freq_hz == pytest.approx(57.0)
assert micl_ch.time_of_peak_s == pytest.approx(-0.004)
def test_event_c_vector_sum():
r = parse_report_file(_fixture("event-c"))
assert r.peak_vector_sum_ips == pytest.approx(0.612)
assert r.peak_vector_sum_time_s == pytest.approx(0.024)
# ── Sensor self-check ───────────────────────────────────────────────────────
def test_event_c_sensor_check_geo_channels():
r = parse_report_file(_fixture("event-c"))
for ch_name, expected_freq, expected_ratio in [
("Tran", 7.4, 3.7),
("Vert", 7.6, 3.5),
("Long", 7.5, 3.8),
]:
sc = r.sensor_check[ch_name]
assert sc.test_freq_hz == pytest.approx(expected_freq), ch_name
assert sc.test_ratio == pytest.approx(expected_ratio), ch_name
assert sc.test_results == "Passed", ch_name
# Geo channels don't have an Test Amplitude
assert sc.test_amplitude_mv is None
def test_event_c_sensor_check_micl():
r = parse_report_file(_fixture("event-c"))
sc = r.sensor_check["MicL"]
assert sc.test_freq_hz == pytest.approx(20.1)
assert sc.test_amplitude_mv == pytest.approx(533.0)
assert sc.test_results == "Passed"
# MicL doesn't have a ratio — it has amplitude instead
assert sc.test_ratio is None
# ── Monitor log + tooling ───────────────────────────────────────────────────
def test_event_c_monitor_log_and_pc_version():
r = parse_report_file(_fixture("event-c"))
assert len(r.monitor_log) == 1
e = r.monitor_log[0]
assert e.start_time == datetime.datetime(2026, 4, 23, 15, 46, 16)
assert e.stop_time == datetime.datetime(2026, 4, 23, 15, 56, 36)
assert e.description == "Event recorded."
assert r.pc_sw_version == "V 10.74"
# ── Sample table ─────────────────────────────────────────────────────────────
def test_event_c_sample_table_parsed_when_requested():
r = parse_report_file(_fixture("event-c"), parse_samples=True)
# 1 sec event @ 1024 sps + 0.25 sec pretrig = 1280 samples
assert r.samples is not None
assert len(r.samples) == 1280, f"expected 1280 samples, got {len(r.samples)}"
# First row: "0.000 \t0.005 \t0.005 \t-81.94"
t, v, l, m = r.samples[0]
assert t == pytest.approx(0.000)
assert v == pytest.approx(0.005)
assert l == pytest.approx(0.005)
assert m == pytest.approx(-81.94)
def test_event_c_sample_table_skipped_by_default():
r = parse_report_file(_fixture("event-c"))
assert r.samples is None
# ── Cross-event smoke ───────────────────────────────────────────────────────
@pytest.mark.parametrize("event_name", ["event-a", "event-b", "event-c", "event-d"])
def test_all_fixtures_parse_without_error(event_name):
"""Every fixture in the bundle must parse cleanly with the same parser."""
r = parse_report_file(_fixture(event_name))
# Common invariants: serial, event_datetime, sample rate, all four
# channels surfaced.
assert r.serial == "BE11529"
assert r.event_datetime is not None
assert r.sample_rate_sps in (1024, 2048, 4096)
for ch in ("Tran", "Vert", "Long", "MicL"):
assert ch in r.channels
assert ch in r.sensor_check
# PVS should be present and positive on triggered events
if r.peak_vector_sum_ips is not None:
assert r.peak_vector_sum_ips >= 0
# ── Edge cases / defensive parsing ──────────────────────────────────────────
def test_parse_empty_input():
r = parse_report("")
assert r.serial is None
assert r.event_datetime is None
assert all(cs.ppv_ips is None for cs in r.channels.values())
def test_parse_unknown_keys_ignored():
"""Forward-compat: future BW versions may add fields we don't recognise.
Those should be silently dropped, not raise."""
text = (
'"Serial Number : BE99999"\n'
'"Future Field That Does Not Exist : 42 widgets"\n'
'"Tran PPV : 0.123 in/s"\n'
)
r = parse_report(text)
assert r.serial == "BE99999"
assert r.channels["Tran"].ppv_ips == pytest.approx(0.123)
def test_parse_numeric_with_units_strips_unit():
text = (
'"Vert PPV : 1.275 in/s"\n'
'"Vert ZC Freq : 23 Hz"\n'
'"MicL Test Amplitude : 569 mv"\n'
)
r = parse_report(text)
assert r.channels["Vert"].ppv_ips == pytest.approx(1.275)
assert r.channels["Vert"].zc_freq_hz == pytest.approx(23.0)
assert r.sensor_check["MicL"].test_amplitude_mv == pytest.approx(569.0)
def test_parse_handles_micl_double_space_in_key():
"""BW writes "MicL Time of Peak" with TWO spaces; the parser must
normalise whitespace before key lookup."""
text = (
'"MicL Time of Peak : 0.012 sec"\n'
'"MicL ZC Freq : 51 Hz"\n'
)
r = parse_report(text)
assert r.mic.time_of_peak_s == pytest.approx(0.012)
assert r.mic.zc_freq_hz == pytest.approx(51.0)
# ── Position-based user-notes parsing ───────────────────────────────────────
#
# The 4 user-supplied note slots (Project / Client / User Name / Seis Loc
# by default) have OPERATOR-EDITABLE labels in BW's Compliance Setup →
# Notes tab. An operator could rename them to "Building:", "Site:",
# "Address:", etc. and the ASCII export would write those labels
# verbatim. We parse by POSITION between the `Units :` and `Geo Range :`
# anchors, NOT by matching the label text.
def _wrap_user_notes(*lines: str) -> str:
"""Helper: wrap N user-note lines in the minimal context the parser
needs (`Units :` opens the block, `Geo Range :` closes it)."""
body = ['"Units : in/s and dB(L)"']
body.extend('"' + l + '"' for l in lines)
body.append('"Geo Range : 10.000 in/s"')
return "\n".join(body) + "\n"
def test_user_notes_default_labels_populate_by_position():
"""The BW-default labels (Project / Client / User Name / Seis Loc)
populate the four canonical slots in order."""
r = parse_report(_wrap_user_notes(
"Project: : Test4-21-26",
"Client: : Acme Inc",
"User Name: : Brian",
"Seis Loc: : Catbed",
))
assert r.project == "Test4-21-26"
assert r.client == "Acme Inc"
assert r.operator == "Brian"
assert r.sensor_location == "Catbed"
assert r.user_note_labels == {
"project": "Project:",
"client": "Client:",
"operator": "User Name:",
"sensor_location": "Seis Loc:",
}
def test_user_notes_operator_renamed_labels_still_populate():
"""If the operator renames the labels in BW's UI (e.g. "Seis Loc:"
→ "Building:"), the values STILL populate the canonical slots by
position — and the operator's labels are preserved in
`user_note_labels` for terra-view to display."""
r = parse_report(_wrap_user_notes(
"Building : Main Office",
"Project Manager : Brian",
"Inspector : Claude",
"Site Address : 123 Main St",
))
assert r.project == "Main Office"
assert r.client == "Brian"
assert r.operator == "Claude"
assert r.sensor_location == "123 Main St"
assert r.user_note_labels == {
"project": "Building",
"client": "Project Manager",
"operator": "Inspector",
"sensor_location": "Site Address",
}
def test_user_notes_with_histogram_label_spelling():
"""Histogram exports use 'Seis. Location:' (with period and colon)
instead of 'Seis Loc:'. Position-based parsing handles both."""
r = parse_report(_wrap_user_notes(
"Project: : Plum Cont.- Rainbow Run",
"Client: : Plum Contracting In.c",
"User Name: : Terra-Mechanics Inc.",
"Seis. Location: : Loc #1 - 2652 Hepner",
))
assert r.project == "Plum Cont.- Rainbow Run"
assert r.client == "Plum Contracting In.c"
assert r.operator == "Terra-Mechanics Inc."
assert r.sensor_location == "Loc #1 - 2652 Hepner"
# And the histogram's specific label spelling is preserved
assert r.user_note_labels["sensor_location"] == "Seis. Location:"
def test_user_notes_outside_block_are_ignored():
"""Lines that look like user-notes but appear OUTSIDE the
Units→Geo Range range don't get assigned to user-note slots."""
# No Units anchor — these lines shouldn't populate user-note slots
text = (
'"Serial Number : BE11529"\n'
'"Project: : SHOULD NOT POPULATE"\n'
)
r = parse_report(text)
assert r.serial == "BE11529"
assert r.project is None
def test_user_notes_partial_block_only_fills_present_slots():
"""If BW writes fewer than 4 user-notes (e.g. operator disabled
Extended Notes mid-block), only the present positions populate;
later slots stay None."""
r = parse_report(_wrap_user_notes(
"Project: : Just-a-project",
"Client: : Just-a-client",
))
assert r.project == "Just-a-project"
assert r.client == "Just-a-client"
assert r.operator is None
assert r.sensor_location is None
def test_user_notes_extra_lines_beyond_four_are_dropped():
"""If somehow more than 4 lines appear in the user-notes block
(e.g. BW adds an Extended Notes line), only the first 4 are
captured — slots 5+ have nowhere to go."""
r = parse_report(_wrap_user_notes(
"L1 : v1",
"L2 : v2",
"L3 : v3",
"L4 : v4",
"L5 : v5", # ignored — no fifth slot
))
assert r.project == "v1"
assert r.client == "v2"
assert r.operator == "v3"
assert r.sensor_location == "v4"
# 5th label not captured
assert "L5" not in r.user_note_labels.values()
def test_real_histogram_fixture_populates_sensor_location():
"""End-to-end: the histogram fixture uses 'Seis. Location:' — must
successfully populate sensor_location via position-based parsing."""
fixture_dir = (
Path(__file__).parent.parent / "example-events" / "histogram"
)
if not fixture_dir.exists():
pytest.skip("histogram fixtures not present")
txt = next(fixture_dir.glob("*_ASCII.TXT"), None)
if txt is None:
pytest.skip("no histogram TXT in fixture dir")
r = parse_report_file(txt)
assert r.sensor_location is not None
assert len(r.sensor_location) > 0
assert r.user_note_labels.get("sensor_location") is not None
# Sanity: other shared fields still parse correctly
assert r.serial is not None
assert r.serial.startswith("BE")
assert r.geo_range_ips is not None
tests/test_event_file_io.py
+212 -56
View File
@@ -127,59 +127,6 @@ def test_sidecar_write_and_read_round_trip(tmp_path: Path):
assert loaded["source"]["kind"] == "sfm-ach"
def test_sidecar_persists_raw_0c_record_in_extensions(tmp_path: Path):
"""An Event with _raw_record populated should land its 210 bytes
base64-encoded in extensions.raw_records.waveform_record_b64, so
later analysis (e.g. mapping Peak Acceleration / Time of Peak / ZC
Freq byte offsets) can run offline against the saved sidecar."""
import base64
ev, _ = _make_synthetic_event()
# Synthesize a 210-byte 0C record with embedded label needles so
# the dump tool's anchor scan has something to find.
raw = bytearray(210)
raw[10:14] = b"Tran"
raw[60:64] = b"Vert"
raw[110:114] = b"Long"
raw[160:164] = b"MicL"
ev._raw_record = bytes(raw)
d = event_file_io.event_to_sidecar_dict(
ev, serial="BE11529",
blastware_filename="M529LKIQ.7M0W", blastware_filesize=1024,
blastware_sha256="x" * 64, source_kind="sfm-live",
)
rr = d["extensions"]["raw_records"]
assert rr["waveform_record_len"] == 210
decoded = base64.b64decode(rr["waveform_record_b64"])
assert decoded == ev._raw_record
# Round-trip through write/read
path = tmp_path / "raw0c.sfm.json"
event_file_io.write_sidecar(path, d)
loaded = event_file_io.read_sidecar(path)
assert (
base64.b64decode(loaded["extensions"]["raw_records"]["waveform_record_b64"])
== ev._raw_record
)
def test_sidecar_omits_raw_records_when_event_has_no_0c(tmp_path: Path):
"""Events without a _raw_record (e.g. constructed by importers that
never see 0C) should NOT add an empty raw_records block — keep the
sidecar clean for those flows."""
ev, _ = _make_synthetic_event()
assert ev._raw_record is None
d = event_file_io.event_to_sidecar_dict(
ev, serial="BE11529",
blastware_filename="M529LKIQ.7M0W", blastware_filesize=1024,
blastware_sha256="x" * 64, source_kind="bw-import",
)
assert d["extensions"] == {}
def test_sidecar_rejects_unsupported_schema_version(tmp_path: Path):
path = tmp_path / "future.sfm.json"
path.write_text(json.dumps({
@@ -342,9 +289,214 @@ def test_read_blastware_file_round_trip(tmp_path: Path):
assert parsed.timestamp.second == ev.timestamp.second
# No A5 source recoverable.
assert parsed._a5_frames is None
# Peaks computed from samples (synthetic = zero samples → zero peaks).
assert parsed.peak_values is not None
assert parsed.peak_values.peak_vector_sum == 0.0
# The synthetic event has no real waveform body, so the codec can't
# decode samples → read_blastware_file leaves peak_values=None
# (the "we don't know" signal) rather than fabricating all-zero
# peaks that would otherwise overwrite real DB values via UPSERT.
assert parsed.peak_values is None
assert parsed.raw_samples is not None
# Empty channels — codec returned None for the malformed synthetic body.
for ch in ("Tran", "Vert", "Long", "MicL"):
assert parsed.raw_samples[ch] == []
_BW_CODEC_FIXTURES = [
# (path, expected_n_samples_per_channel, BW-reported Vert PPV in/s for sanity)
("tests/fixtures/decode-re-5-8-26/event-a/M529LKVQ.6S0", 3328, 0.780),
("tests/fixtures/decode-re-5-8-26/event-b/M529LK5Q.RG0", 2304, 0.505),
("tests/fixtures/decode-re-5-8-26/event-c/M529LK44.AB0", 1280, 0.610),
("tests/fixtures/decode-re-5-8-26/event-d/M529LK2V.470", 1280, 0.565),
("tests/fixtures/5-11-26/M529LL1L.V70", 3328, 0.010),
("tests/fixtures/5-11-26/M529LL1L.JQ0", 3328, 3.465),
]
@pytest.mark.parametrize("path,expected_n,expected_ppv", _BW_CODEC_FIXTURES)
def test_read_blastware_file_decodes_via_codec(path: str, expected_n: int, expected_ppv: float):
"""Regression lock: ``read_blastware_file()`` must use the verified
waveform-body codec (``minimateplus.waveform_codec``), not the
retracted int16-LE assumption.
Verifies against the real BW fixture corpus: every event in the
bundled fixtures must produce the expected per-channel sample count
and a Vert PPV close to BW's own reported value. Catches any
accidental regression of the body decoder back to the old
``_decode_samples_4ch_int16_le`` path (which produced ±32K noise
on every event, giving wildly wrong PPVs).
"""
repo_root = Path(__file__).resolve().parent.parent
full_path = repo_root / path
if not full_path.exists():
pytest.skip(f"fixture missing: {full_path}")
ev = event_file_io.read_blastware_file(full_path)
assert ev.raw_samples is not None
for ch in ("Tran", "Vert", "Long"):
assert len(ev.raw_samples[ch]) == expected_n, (
f"{ch}: expected {expected_n} samples, got {len(ev.raw_samples[ch])}"
)
# PPV check: the codec produces decoded samples in 1-count ADC units;
# _peaks_from_samples scales by GEO_NORMAL_FS_INS / 32767. BW's own
# PPV is computed at slightly different precision/interpolation, so
# we allow a 0.2 in/s tolerance — well under the broken-decoder
# signature (which would produce ~10 in/s saturation).
assert ev.peak_values is not None
assert abs(ev.peak_values.vert - expected_ppv) < 0.2, (
f"Vert PPV {ev.peak_values.vert:.3f} differs from BW's "
f"{expected_ppv:.3f} by >0.2 in/s — codec regression?"
)
def test_read_blastware_file_v70_samples_match_txt_truth():
"""Strongest regression lock: every one of V70's 3328 decoded
sample-sets must match the .TXT ground truth table within the
0.005 in/s display quantum."""
repo_root = Path(__file__).resolve().parent.parent
bw_path = repo_root / "tests/fixtures/5-11-26/M529LL1L.V70"
txt_path = repo_root / "tests/fixtures/5-11-26/M529LL1L.V70.TXT"
if not bw_path.exists() or not txt_path.exists():
pytest.skip(f"V70 fixture missing")
import re
ev = event_file_io.read_blastware_file(bw_path)
# Parse .TXT ground truth sample table
text = txt_path.read_text()
lines = text.splitlines()
hdr_idx = next(i for i, line in enumerate(lines)
if re.match(r"^Tran\s+Vert\s+Long\s+MicL?", line.strip()))
truth = []
for line in lines[hdr_idx + 1:]:
parts = line.strip().split()
if len(parts) != 4:
continue
try:
truth.append([float(x) for x in parts])
except ValueError:
continue
assert len(truth) == 3328, f"expected 3328 truth rows, got {len(truth)}"
def adc_to_ins(count):
return count / 32767.0 * 10.0
for i, truth_row in enumerate(truth):
for ch_idx, ch_name in enumerate(("Tran", "Vert", "Long")):
decoded_ips = adc_to_ins(ev.raw_samples[ch_name][i])
truth_ips = truth_row[ch_idx]
# 0.003 in/s tolerance: <0.005 quantum + small float precision room
assert abs(decoded_ips - truth_ips) < 0.003, (
f"row {i} {ch_name}: decoded {decoded_ips:+.4f} vs "
f"truth {truth_ips:+.4f} (delta {decoded_ips - truth_ips:+.4f})"
)
def test_save_imported_bw_with_paired_report(tmp_path: Path):
"""save_imported_bw + a paired BW ASCII report fold the report's
rich derived fields into the sidecar. This is the daemon-forwarded
ACH workflow: BW writes <event>.AB0 and <event>.AB0.TXT side by side;
the daemon ships both; we overlay the report-decoded values onto the
sidecar (peaks, project, plus the rich `bw_report` block)."""
from minimateplus.blastware_file import write_blastware_file, blastware_filename
from sfm.waveform_store import WaveformStore
ev, frames = _make_synthetic_event()
fname = blastware_filename(ev, "BE11529")
src = tmp_path / fname
write_blastware_file(ev, frames, src)
# Use one of the real BW ASCII exports as the paired report.
report_path = (
Path(__file__).parent.parent
/ "decode-re" / "5-8-26" / "event-c" / "M529LK44.AB0.TXT"
)
if not report_path.exists():
import pytest as _pt
_pt.skip("decode-re fixtures not present")
report_bytes = report_path.read_bytes()
store = WaveformStore(tmp_path / "waveforms")
parsed_ev, rec = store.save_imported_bw(
src.read_bytes(),
source_path=src,
bw_report_text=report_bytes,
)
sc = store.load_sidecar("BE11529", fname)
assert sc is not None
# ── bw_report block populated with the rich fields ──────────────────
assert "bw_report" in sc
br = sc["bw_report"]
assert br["available"] is True
assert br["event_type"] == "Full Waveform"
assert br["recording"]["sample_rate_sps"] == 1024
assert br["recording"]["geo_range_ips"] == 10.0
# Per-channel derived stats
assert br["peaks"]["tran"]["ppv_ips"] == 0.065
assert br["peaks"]["vert"]["ppv_ips"] == 0.610
assert br["peaks"]["long"]["ppv_ips"] == 0.070
assert br["peaks"]["vert"]["peak_accel_g"] == 0.437
assert br["peaks"]["vert"]["peak_disp_in"] == 0.006
assert br["peaks"]["tran"]["zc_freq_hz"] == 47.0
assert br["peaks"]["vector_sum"]["ips"] == 0.612
assert br["peaks"]["vector_sum"]["time_s"] == 0.024
# Sensor self-check per channel
assert br["sensor_check"]["tran"]["freq_hz"] == 7.4
assert br["sensor_check"]["tran"]["ratio"] == 3.7
assert br["sensor_check"]["tran"]["result"] == "Passed"
assert br["sensor_check"]["mic"]["amplitude_mv"] == 533.0
# Mic block
assert br["mic"]["weighting"] == "Linear Weighting"
assert br["mic"]["pspl_dbl"] == 88.0
# Monitor log roundtripped
assert len(br["monitor_log"]) == 1
assert "2026-04-23T15:46:16" in br["monitor_log"][0]["start"]
assert br["pc_sw_version"] == "V 10.74"
# ── Overlay onto canonical peak_values ──────────────────────────────
# Report values win over the broken-codec samples-derived peaks.
assert sc["peak_values"]["transverse"] == 0.065
assert sc["peak_values"]["vertical"] == 0.610
assert sc["peak_values"]["longitudinal"] == 0.070
assert sc["peak_values"]["vector_sum"] == 0.612
# Mic PSPL converted to psi (dbl=88 → 10^(88/20) * 2.9e-9)
assert sc["peak_values"]["mic_psi"] is not None
assert 1e-5 < sc["peak_values"]["mic_psi"] < 1e-3
# ── Overlay onto project_info ───────────────────────────────────────
assert sc["project_info"]["project"] == "Test4-21-26"
assert sc["project_info"]["client"] == "Test-Client1"
assert sc["project_info"]["operator"] == "Brian and claude"
assert sc["project_info"]["sensor_location"] == "catbed"
# ── Event timestamp overlaid from report ───────────────────────────
assert sc["event"]["timestamp"] == "2026-04-23T15:56:35"
def test_save_imported_bw_without_report_works_unchanged(tmp_path: Path):
"""Calling save_imported_bw with no bw_report_text behaves exactly
as before — no `bw_report` block, peak_values come from samples."""
from minimateplus.blastware_file import write_blastware_file, blastware_filename
from sfm.waveform_store import WaveformStore
ev, frames = _make_synthetic_event()
fname = blastware_filename(ev, "BE11529")
src = tmp_path / fname
write_blastware_file(ev, frames, src)
store = WaveformStore(tmp_path / "waveforms")
store.save_imported_bw(src.read_bytes(), source_path=src)
sc = store.load_sidecar("BE11529", fname)
assert sc is not None
assert "bw_report" not in sc # block is absent without a report
# Synthetic event has zero samples → peaks all zero (was true before this change)
assert sc["peak_values"]["transverse"] == 0.0
def test_save_imported_bw_round_trip(tmp_path: Path):
@@ -363,6 +515,10 @@ def test_save_imported_bw_round_trip(tmp_path: Path):
assert rec["filename"] == fname
assert rec["a5_pickle_filename"] is None # no A5 source for BW imports
# The serial decoded from the BW filename surfaces on the record so
# the import endpoint can use it when calling SeismoDb.insert_events()
# (otherwise forwarded events would all bucket into serial="UNKNOWN").
assert rec["serial"] == "BE11529"
sc = store.load_sidecar("BE11529", fname)
assert sc is not None
assert sc["source"]["kind"] == "bw-import"
tests/test_histogram_codec.py
+337
View File
@@ -0,0 +1,337 @@
"""
test_histogram_codec.py — regression locks for the histogram body codec.
The codec is verified byte-exact against BW's ASCII export across the
in-repo histogram fixture bundle. Each test cross-checks decoded
binary fields against the corresponding .TXT row.
Run:
python -m pytest tests/test_histogram_codec.py -q
"""
from __future__ import annotations
import os
import re
import sys
from pathlib import Path
import pytest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from minimateplus.blastware_file import _WAVEFORM_HEADER_SIZE
from minimateplus.histogram_codec import (
_BLOCK_SIZE,
decode_histogram_body,
decode_histogram_body_full,
geo_count_to_ins,
half_period_to_hz,
walk_body,
)
from minimateplus.waveform_codec import mic_count_to_db
_FIXTURE_DIR = Path(__file__).resolve().parent.parent / "example-events" / "histogram"
def _extract_body(path: Path) -> bytes:
"""Locate the body of a BW event file — bytes between the STRT
record and the 26-byte footer."""
raw = path.read_bytes()
body_start = _WAVEFORM_HEADER_SIZE + 21
pos = body_start
footer_pos = -1
while True:
pos = raw.find(b"\x0e\x08", pos)
if pos < 0 or pos + 26 > len(raw):
break
yr = (raw[pos + 4] << 8) | raw[pos + 5]
if 2015 <= yr <= 2050:
footer_pos = pos
break
pos += 1
if footer_pos < 0:
footer_pos = len(raw) - 26
return raw[body_start:footer_pos]
def _parse_txt_rows(path: Path) -> list[tuple[str, list]]:
"""Parse a histogram .TXT into ``[(time_str, [10 col values]), …]``.
Special tokens:
- ``">100"`` (the BW-display sentinel for freq > 100 Hz) → ``None``
- non-numeric → ``None``
"""
text = path.read_text()
lines = text.splitlines()
hdr = None
for i, line in enumerate(lines):
if re.match(r"^Tran\s+", line.strip()):
hdr = i + 3 # skip 2-row header + units row
break
if hdr is None:
return []
rows: list[tuple[str, list]] = []
for line in lines[hdr:]:
parts = line.split("\t")
if len(parts) != 11:
continue
vals: list = []
for p in parts[1:]:
s = p.strip()
if s.startswith(">"):
vals.append(None) # ">100 Hz" sentinel
continue
try:
vals.append(float(s))
except ValueError:
vals.append(None)
rows.append((parts[0].strip(), vals))
return rows
# ── Block-walker plumbing ────────────────────────────────────────────────────
@pytest.mark.parametrize("fixture", [
"N844L20G.630H",
"N844L21H.2R0H",
"N844L6Z8.ZR0H",
"N844L6XE.BH0H",
"N844L23B.ND0H",
])
def test_walk_body_returns_records(fixture: str):
"""Walker yields at least one valid block per fixture."""
path = _FIXTURE_DIR / fixture
if not path.exists():
pytest.skip(f"fixture missing: {path}")
records = walk_body(_extract_body(path))
assert len(records) > 100, f"expected hundreds of blocks, got {len(records)}"
def test_walk_body_record_count_matches_txt_intervals():
"""Block count should match the .TXT interval count (off-by-one
at the tail is acceptable — last interval may be truncated at
recording stop)."""
bin_path = _FIXTURE_DIR / "N844L20G.630H"
txt_path = _FIXTURE_DIR / "N844L20G_630H_ASCII.TXT"
if not bin_path.exists() or not txt_path.exists():
pytest.skip("fixture missing")
records = walk_body(_extract_body(bin_path))
txt_rows = _parse_txt_rows(txt_path)
# Allow off-by-one (final block may have been mid-write at stop)
assert abs(len(records) - len(txt_rows)) <= 1, (
f"binary {len(records)} blocks vs TXT {len(txt_rows)} intervals"
)
def test_walk_body_segment_id_increments_every_256_blocks():
"""Segment ID advances 0→1→2→… after every 256 blocks within
one event."""
path = _FIXTURE_DIR / "N844L20G.630H"
if not path.exists():
pytest.skip("fixture missing")
records = walk_body(_extract_body(path))
# Group by segment_id and verify counts make sense
from collections import Counter
seg_counts = Counter(r["segment_id"] for r in records)
# First 3 segments should each have exactly 256 blocks (N844L20G has
# 791 blocks → 256+256+256+23 → segments 0/1/2/3)
assert seg_counts[0] == 256
assert seg_counts[1] == 256
assert seg_counts[2] == 256
assert seg_counts[3] == len(records) - 3 * 256
# ── Field-by-field decode verification against .TXT ground truth ─────────────
@pytest.mark.parametrize("fixture", [
"N844L20G.630H",
"N844L6Z8.ZR0H",
"N844L6XE.BH0H",
"N844L23B.ND0H",
])
def test_decoded_geo_peaks_match_txt(fixture: str):
"""For every block, decoded Tran/Vert/Long peak (count × 0.005)
matches the corresponding .TXT cell."""
bin_path = _FIXTURE_DIR / fixture
txt_path = _FIXTURE_DIR / (fixture.replace(".", "_") + "_ASCII.TXT")
if not bin_path.exists() or not txt_path.exists():
pytest.skip("fixture missing")
records = walk_body(_extract_body(bin_path))
txt_rows = _parse_txt_rows(txt_path)
n = min(len(records), len(txt_rows))
assert n > 0
for i in range(n):
rec = records[i]
_ts, txt = txt_rows[i]
# TXT cols 0/2/4 are T/V/L peak in in/s
for slot, key in (("T", "t_peak"), ("V", "v_peak"), ("L", "l_peak")):
col = {"T": 0, "V": 2, "L": 4}[slot]
decoded_ips = geo_count_to_ins(rec[key])
expected = txt[col]
assert abs(decoded_ips - expected) < 0.0005, (
f"{fixture} block {i} {slot}_peak: "
f"decoded={decoded_ips:.4f} vs txt={expected:.4f}"
)
@pytest.mark.parametrize("fixture", [
"N844L6Z8.ZR0H",
"N844L6XE.BH0H",
])
def test_decoded_geo_freqs_match_txt(fixture: str):
"""Decoded half-period → Hz matches the .TXT freq column for blocks
where the freq is in-range (not the `>100 Hz` sentinel)."""
bin_path = _FIXTURE_DIR / fixture
txt_path = _FIXTURE_DIR / (fixture.replace(".", "_") + "_ASCII.TXT")
if not bin_path.exists() or not txt_path.exists():
pytest.skip("fixture missing")
records = walk_body(_extract_body(bin_path))
txt_rows = _parse_txt_rows(txt_path)
n = min(len(records), len(txt_rows))
for i in range(n):
rec = records[i]
_ts, txt = txt_rows[i]
for slot, key, col in (("T", "t_halfp", 1), ("V", "v_halfp", 3), ("L", "l_halfp", 5)):
decoded_hz = half_period_to_hz(rec[key])
expected = txt[col]
if expected is None:
# TXT shows `>100 Hz` — codec should also yield None
assert decoded_hz is None or decoded_hz > 100, (
f"{fixture} block {i} {slot}_freq: codec says "
f"{decoded_hz} but TXT says >100"
)
continue
# TXT rounds; allow ±1 Hz
assert decoded_hz is not None
assert abs(decoded_hz - expected) < 1.0, (
f"{fixture} block {i} {slot}_freq: "
f"decoded={decoded_hz:.2f} Hz vs txt={expected:.2f} Hz"
)
@pytest.mark.parametrize("fixture", [
"N844L6XE.BH0H",
"N844L23B.ND0H",
"N844L6Z8.ZR0H",
])
def test_decoded_mic_db_matches_txt(fixture: str):
"""Decoded MicL peak count → dB(L) via mic_count_to_db matches
the .TXT dB(L) column."""
bin_path = _FIXTURE_DIR / fixture
txt_path = _FIXTURE_DIR / (fixture.replace(".", "_") + "_ASCII.TXT")
if not bin_path.exists() or not txt_path.exists():
pytest.skip("fixture missing")
records = walk_body(_extract_body(bin_path))
txt_rows = _parse_txt_rows(txt_path)
n = min(len(records), len(txt_rows))
for i in range(n):
rec = records[i]
_ts, txt = txt_rows[i]
# TXT col 8 = MicL dB(L)
decoded_db = mic_count_to_db(rec["m_peak"])
expected = txt[8]
if expected is None:
continue
# BW rounds to 1 decimal place for display. Tolerance 0.1 dB
# absorbs both rounding modes (truncate vs round-half-even).
assert abs(decoded_db - expected) < 0.1, (
f"{fixture} block {i} M_dB: "
f"decoded={decoded_db:.2f} dB vs txt={expected:.2f} dB"
)
@pytest.mark.parametrize("fixture", [
"N844L20G.630H",
"N844L6Z8.ZR0H",
])
def test_decoded_mic_freq_matches_txt(fixture: str):
"""Decoded MicL half-period → freq matches the .TXT col 9 freq."""
bin_path = _FIXTURE_DIR / fixture
txt_path = _FIXTURE_DIR / (fixture.replace(".", "_") + "_ASCII.TXT")
if not bin_path.exists() or not txt_path.exists():
pytest.skip("fixture missing")
records = walk_body(_extract_body(bin_path))
txt_rows = _parse_txt_rows(txt_path)
n = min(len(records), len(txt_rows))
for i in range(n):
rec = records[i]
_ts, txt = txt_rows[i]
decoded_hz = half_period_to_hz(rec["m_halfp"])
expected = txt[9]
if expected is None:
assert decoded_hz is None or decoded_hz > 100
continue
assert decoded_hz is not None
assert abs(decoded_hz - expected) < 1.0, (
f"{fixture} block {i} M_freq: "
f"decoded={decoded_hz:.2f} Hz vs txt={expected:.2f} Hz"
)
# ── Public API ───────────────────────────────────────────────────────────────
def test_decode_histogram_body_returns_four_channels():
"""The public API returns the standard 4-channel dict shape."""
path = _FIXTURE_DIR / "N844L20G.630H"
if not path.exists():
pytest.skip("fixture missing")
decoded = decode_histogram_body(_extract_body(path))
assert decoded is not None
assert set(decoded.keys()) == {"Tran", "Vert", "Long", "MicL"}
# All channels same length (one value per histogram interval)
n = len(decoded["Tran"])
assert all(len(decoded[ch]) == n for ch in ("Vert", "Long", "MicL"))
assert n > 100
def test_decode_histogram_body_returns_none_for_non_histogram():
"""A waveform-mode body (starts with 00 02 00) doesn't decode as
a histogram body."""
fake_waveform_body = b"\x00\x02\x00" + b"\x00" * 100
assert decode_histogram_body(fake_waveform_body) is None
def test_decode_histogram_body_returns_none_for_garbage():
"""Bytes that don't form valid blocks return None."""
assert decode_histogram_body(b"\xff" * 256) is None
def test_decode_histogram_body_full_preserves_frequency_data():
"""The structured-record API preserves the per-channel half-period
fields that the flat-channel API drops."""
path = _FIXTURE_DIR / "N844L20G.630H"
if not path.exists():
pytest.skip("fixture missing")
records = decode_histogram_body_full(_extract_body(path))
assert records is not None
r0 = records[0]
expected_fields = {
"segment_id", "block_ctr",
"t_peak", "t_halfp", "v_peak", "v_halfp",
"l_peak", "l_halfp", "m_peak", "m_halfp",
"meta_var",
}
assert set(r0.keys()) >= expected_fields
# ── Helpers ──────────────────────────────────────────────────────────────────
def test_half_period_to_hz_sentinel():
"""Half-period ≤ 5 returns None (the `>100 Hz` sentinel)."""
assert half_period_to_hz(5) is None
assert half_period_to_hz(1) is None
# halfp=6 gives 512/6 = 85.3 Hz — below the >100 threshold
assert half_period_to_hz(6) == pytest.approx(85.33, abs=0.01)
def test_geo_count_to_ins_scale():
"""1 count = 0.005 in/s at Normal range."""
assert geo_count_to_ins(1) == pytest.approx(0.005)
assert geo_count_to_ins(10) == pytest.approx(0.050)
assert geo_count_to_ins(0) == 0.0
tests/test_idf_ascii_report.py
+234
View File
@@ -0,0 +1,234 @@
"""
test_idf_ascii_report.py — parser for Thor's per-event IDF ASCII export.
Run:
python -m pytest tests/test_idf_ascii_report.py -q
Tests use real Thor sample data shipped under
`thor-watcher/example-data/THORDATA_example/`. When that path is not
available (e.g. running from a checkout where the watcher repo isn't
sibling), tests gracefully skip.
"""
from __future__ import annotations
import datetime
import os
import sys
from pathlib import Path
import pytest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from micromate.idf_ascii_report import (
parse_event_filename,
parse_idf_report,
serial_from_filename,
)
# ── Sample data ──────────────────────────────────────────────────────────────
SAMPLE_REPO = Path("/home/serversdown/thor-watcher/example-data/"
"THORDATA_example/THORDATA_example")
def _sample_path(rel: str) -> Path:
return SAMPLE_REPO / rel
@pytest.fixture
def upmc_waveform_txt() -> str:
p = _sample_path("UPMC Presby/UM11719/TXT/UM11719_20231219162723.IDFW.txt")
if not p.exists():
pytest.skip(f"sample missing: {p}")
return p.read_text()
@pytest.fixture
def upmc_histogram_txt() -> str:
p = _sample_path("UPMC Presby/UM11719/TXT/UM11719_20231219163444.IDFH.txt")
if not p.exists():
pytest.skip(f"sample missing: {p}")
return p.read_text()
# ── Filename parsing ─────────────────────────────────────────────────────────
def test_parse_event_filename_waveform():
parsed = parse_event_filename("UM11719_20231219163444.IDFW")
assert parsed is not None
serial, ts, kind = parsed
assert serial == "UM11719"
assert ts == datetime.datetime(2023, 12, 19, 16, 34, 44)
assert kind == "IDFW"
def test_parse_event_filename_histogram():
parsed = parse_event_filename("BE9439_20200713124251.IDFH")
assert parsed is not None
serial, ts, kind = parsed
assert serial == "BE9439"
assert kind == "IDFH"
def test_parse_event_filename_case_insensitive():
parsed = parse_event_filename("um11719_20231219163444.idfw")
assert parsed is not None
assert parsed[0] == "UM11719"
assert parsed[2] == "IDFW"
def test_parse_event_filename_rejects_invalid():
for name in [
"UM11719_20231219163444.MLG",
"UM11719.IDFW",
"UM11719_20231219163444.IDFW.txt", # report sidecar — not a binary
"UM11719_2023121916344X.IDFW",
"garbage",
"",
]:
assert parse_event_filename(name) is None, name
def test_serial_from_filename():
assert serial_from_filename("UM11719_20231219163444.IDFW") == "UM11719"
assert serial_from_filename("BE9439_20200713124251.IDFH") == "BE9439"
# Works on the .txt sidecar name too — handy in pairing code paths
assert serial_from_filename("UM11719_20231219163444.IDFW.txt") == "UM11719"
assert serial_from_filename("not_a_thor_file.bin") is None
# ── Report parsing — derived fields against real Thor sample ─────────────────
def test_waveform_report_derives_serial_event_type_and_datetime(upmc_waveform_txt):
r = parse_idf_report(upmc_waveform_txt)
assert r["serial_number"] == "UM11719"
assert r["event_type"] == "Full Waveform"
assert r["event_datetime"] == "2023-12-19T16:27:23"
assert r["filename"] == "UM11719_20231219162723.IDFW"
def test_waveform_report_parses_peak_velocities(upmc_waveform_txt):
r = parse_idf_report(upmc_waveform_txt)
assert r["tran_ppv"] == pytest.approx(0.0251)
assert r["vert_ppv"] == pytest.approx(0.2119)
assert r["long_ppv"] == pytest.approx(0.0282)
assert r["peak_vector_sum"] == pytest.approx(0.2131)
def test_waveform_report_parses_zc_freq_and_mic(upmc_waveform_txt):
r = parse_idf_report(upmc_waveform_txt)
assert r["tran_zc_freq"] == pytest.approx(6.5)
assert r["vert_zc_freq"] == pytest.approx(73.1)
assert r["long_zc_freq"] == pytest.approx(85.3)
assert r["mic_ppv"] == pytest.approx(99.4)
def test_waveform_report_parses_record_and_pretrigger_durations(upmc_waveform_txt):
r = parse_idf_report(upmc_waveform_txt)
assert r["record_time_sec"] == pytest.approx(2.0)
assert r["pre_trigger_sec"] == pytest.approx(0.25)
def test_waveform_report_parses_sample_rate(upmc_waveform_txt):
r = parse_idf_report(upmc_waveform_txt)
assert r["sample_rate"] == 1024
def test_waveform_report_extracts_title_strings(upmc_waveform_txt):
r = parse_idf_report(upmc_waveform_txt)
# TitleString1 (location) → project
assert r["project"] == "UPMC Presby-Loc 3-Level1-1R Elevator Rm"
# TitleString2 → client
assert r["client"] == "Whiting-Turner - PJ Dick - Joint Venture"
# TitleString3 → operator (company)
assert r["operator"] == "Terra-Mechanics, Inc. - D. Harrsion"
def test_waveform_report_extracts_setup_version_and_calibration(upmc_waveform_txt):
r = parse_idf_report(upmc_waveform_txt)
assert r["setup"] == "UPMC Loc 3.mmb"
assert r["version"] == "Micromate ISEE 11.0AK"
assert r["calibration_text"] == "November 22, 2023 by Instantel"
assert r["battery_volts"] == pytest.approx(3.8)
def test_waveform_report_decodes_sensor_self_check(upmc_waveform_txt):
r = parse_idf_report(upmc_waveform_txt)
assert r["tran_test_passed"] is True
assert r["vert_test_passed"] is True
assert r["long_test_passed"] is True
assert r["mic_test_passed"] is True
def test_histogram_report_parses(upmc_histogram_txt):
"""Histogram sidecars have the same shape as waveform — both
decode through the same parser without errors."""
r = parse_idf_report(upmc_histogram_txt)
assert r["serial_number"] == "UM11719"
# IDFH timestamp in the sample
assert r["event_datetime"] == "2023-12-19T16:34:44"
assert r["event_type"] .lower().startswith("full histogram") or \
r["event_type"] .lower().startswith("histogram")
# Sample rate present
assert "sample_rate" in r
# ── Edge cases ───────────────────────────────────────────────────────────────
def test_parses_bytes_input():
text = (
'"SerialNumber : UM11719"\n'
'"TranPPV : 0.0251 in/s"\n'
)
r = parse_idf_report(text.encode("utf-8"))
assert r["serial_number"] == "UM11719"
assert r["tran_ppv"] == pytest.approx(0.0251)
def test_parses_latin1_fallback():
"""Garbled non-UTF8 bytes fall back to latin-1 instead of crashing."""
text = b'"SerialNumber : UM11719"\n"Operator : Caf\xe9"\n'
r = parse_idf_report(text)
assert r["serial_number"] == "UM11719"
assert r["operator"] == "Café"
def test_stops_at_waveform_data_marker():
"""Lines after the 'Waveform Data Channels' marker are not parsed
as key/value pairs — they're tabular sample data."""
text = (
'"SerialNumber : UM11719"\n'
'"TranPPV : 0.0251 in/s"\n'
'Waveform Data Channels\n'
' Tran Vert Long MicL\n'
' 0.0003 -0.0003 0.0003 0.00013\n'
)
r = parse_idf_report(text)
assert r["serial_number"] == "UM11719"
assert r["tran_ppv"] == pytest.approx(0.0251)
# No spurious entries from the table body
assert "tran" not in r
assert "0.0003" not in r
def test_missing_event_time_omits_datetime():
r = parse_idf_report('"SerialNumber : UM11719"\n')
assert r["serial_number"] == "UM11719"
assert "event_datetime" not in r
def test_handles_empty_input():
r = parse_idf_report("")
assert r == {
"project": None,
"client": None,
"operator": None,
"notes": None,
}
tests/test_waveform_codec.py
+518
View File
@@ -0,0 +1,518 @@
"""
Tests for minimateplus.waveform_codec — Blastware waveform-file body block walker.
These tests lock in the STRUCTURAL framing of the body codec. The byte-to-sample
mapping is open (see waveform_codec module docstring) — until that's nailed down,
:func:`decode_waveform_v2` returns ``None`` and there is no per-sample assertion
to make.
"""
from __future__ import annotations
import os
import pytest
from minimateplus.waveform_codec import (
WaveformBlock,
decode_tran_initial,
decode_waveform_v2,
decoded_to_adc_counts,
find_data_start,
mic_count_to_db,
parse_segment_header,
split_segments,
walk_body,
)
FIXTURES = os.path.join(
os.path.dirname(__file__), "fixtures", "decode-re-5-8-26"
)
def _bw_body(path):
"""Strip the 22-byte header and 21-byte STRT and 26-byte footer to get the body."""
with open(path, "rb") as f:
binary = f.read()
return binary[43:-26]
# Fixture metadata — bundled BW binaries from a real BE11529 unit, May 8 2026.
# Each is paired with a Blastware TXT export (the ASCII ground truth).
FIXTURES_INFO = {
"event-a": {
"filename": "M529LKVQ.6S0",
"n_samples": 3328, # 3.0 s rectime + 0.25 s pretrig at 1024 sps
"rectime": 3.0,
},
"event-b": {
"filename": "M529LK5Q.RG0",
"n_samples": 2304, # 2.0 s
"rectime": 2.0,
},
"event-c": {
"filename": "M529LK44.AB0",
"n_samples": 1280, # 1.0 s
"rectime": 1.0,
},
"event-d": {
"filename": "M529LK2V.470",
"n_samples": 1280,
"rectime": 1.0,
},
}
def _fixture_path(event_name):
info = FIXTURES_INFO[event_name]
return os.path.join(FIXTURES, event_name, info["filename"])
# ── Find data start ──────────────────────────────────────────────────────────
@pytest.mark.parametrize("event_name", list(FIXTURES_INFO.keys()))
def test_find_data_start_locates_first_block(event_name):
"""The walker auto-detects the first ``10 NN`` tag within the first 20 bytes."""
path = _fixture_path(event_name)
if not os.path.exists(path):
pytest.skip(f"fixture missing: {path}")
body = _bw_body(path)
start = find_data_start(body)
assert 0 <= start < 20, f"expected start in [0, 20), got {start}"
assert body[start] in (0x00, 0x10, 0x20, 0x30, 0x40), (
f"first tag byte 0x{body[start]:02x} not a recognized block type"
)
assert body[start + 1] % 4 == 0 or (body[start] == 0x40 and body[start + 1] == 0x02)
def test_find_data_start_canonical_offset_7():
"""All events have a 7-byte preamble (3-byte magic + 4-byte Tran anchors)."""
for name in FIXTURES_INFO:
path = _fixture_path(name)
if not os.path.exists(path):
pytest.skip(f"fixture missing: {path}")
body = _bw_body(path)
# Sanity: magic
assert body[0:3] == b"\x00\x02\x00", f"{name}: bad magic"
# First tag at offset 7
assert find_data_start(body) == 7, f"{name}: expected start=7"
# ── Block walker ─────────────────────────────────────────────────────────────
def test_walk_body_empty_returns_empty():
assert walk_body(b"") == []
def test_walk_body_invalid_start_returns_empty():
# Body that does not begin with a recognized tag.
assert walk_body(b"\xff\xff\xff\xff", start=0) == []
@pytest.mark.parametrize("event_name", list(FIXTURES_INFO.keys()))
def test_walk_body_produces_blocks(event_name):
"""The walker should produce a non-empty stream of blocks for every fixture."""
path = _fixture_path(event_name)
if not os.path.exists(path):
pytest.skip(f"fixture missing: {path}")
body = _bw_body(path)
blocks = walk_body(body)
assert len(blocks) > 0
# All blocks have one of the known tag families. ``1X NN`` / ``2X NN``
# with X in 0..F are valid (X > 0 means wide-NN encoding).
for b in blocks:
assert (b.tag_hi & 0xF0) in (0x10, 0x20, 0x00, 0x30, 0x40), (
f"unknown tag {b.tag_hi:#04x} at offset {b.offset}"
)
@pytest.mark.parametrize("event_name", list(FIXTURES_INFO.keys()))
def test_walk_body_block_lengths_consistent(event_name):
"""Each block's recorded length matches its on-wire footprint."""
path = _fixture_path(event_name)
if not os.path.exists(path):
pytest.skip(f"fixture missing: {path}")
body = _bw_body(path)
blocks = walk_body(body)
for b in blocks:
# Tag (2 bytes) + payload should equal length.
assert 2 + len(b.data) == b.length, (
f"block at {b.offset} length mismatch: tag(2) + data({len(b.data)}) != length({b.length})"
)
@pytest.mark.parametrize("event_name", list(FIXTURES_INFO.keys()))
def test_walk_body_blocks_contiguous(event_name):
"""Block n+1 starts exactly where block n ends (no gaps, no overlaps)."""
path = _fixture_path(event_name)
if not os.path.exists(path):
pytest.skip(f"fixture missing: {path}")
body = _bw_body(path)
blocks = walk_body(body)
for i in range(1, len(blocks)):
prev = blocks[i - 1]
cur = blocks[i]
assert cur.offset == prev.offset + prev.length, (
f"gap/overlap between block {i-1} (off={prev.offset} len={prev.length}) "
f"and block {i} (off={cur.offset})"
)
# ── Segment splitting ────────────────────────────────────────────────────────
@pytest.mark.parametrize("event_name", list(FIXTURES_INFO.keys()))
def test_split_segments_yields_at_least_one(event_name):
path = _fixture_path(event_name)
if not os.path.exists(path):
pytest.skip(f"fixture missing: {path}")
body = _bw_body(path)
blocks = walk_body(body)
segments = split_segments(blocks)
assert len(segments) > 0
def test_split_segments_segment_count_at_least_one_per_event():
"""The walker should produce at least one ``40 02`` segment header per event.
Note: the walker currently bails out partway through event-b (still an
open issue — the body codec uses block lengths the walker doesn't
handle correctly past offset ~427). The other 3 events walk farther
and have many segment headers.
"""
for name in FIXTURES_INFO:
path = _fixture_path(name)
if not os.path.exists(path):
continue
body = _bw_body(path)
blocks = walk_body(body)
n_40 = sum(1 for b in blocks if b.tag_hi == 0x40)
assert n_40 >= 1, f"{name}: no 40 02 segment header found"
# ── Segment header parsing ───────────────────────────────────────────────────
def test_parse_segment_header_returns_none_for_non_40():
block = WaveformBlock(offset=0, tag_hi=0x10, tag_lo=0x04, data=b"\x00\x00", length=4)
assert parse_segment_header(block) is None
def test_parse_segment_header_decodes_fields():
"""Decode a known 40 02 block to verify field offsets."""
# First segment header from event-c at body offset 235:
# 40 02 00 00 00 00 0a 4b 01 1e 47 00 00 00 02 00 00 01 00 01
payload = bytes.fromhex("00000000 0a4b011e 47000000 02000001 0001".replace(" ", ""))
block = WaveformBlock(
offset=235, tag_hi=0x40, tag_lo=0x02, data=payload, length=20
)
decoded = parse_segment_header(block)
assert decoded is not None
assert decoded["counter"] == 0x47 # uint32 LE
assert decoded["fixed_pattern"] == b"\x02\x00\x00\x01"
assert decoded["anchor_bytes"] == b"\x00\x00\x00\x00"
def test_segment_counter_increments():
"""The 4-byte counter at bytes [8:12] of each 40 02 payload increments by 1."""
path = _fixture_path("event-c")
if not os.path.exists(path):
pytest.skip("fixture missing")
body = _bw_body(path)
blocks = walk_body(body)
headers = [b for b in blocks if b.tag_hi == 0x40 and b.tag_lo == 0x02]
counters = [parse_segment_header(b)["counter"] for b in headers]
assert len(counters) >= 5, "expect at least 5 segments to verify increments"
# First few counters should be strictly monotonic (the BW counter is global,
# incrementing across the whole flash buffer; some events may share counter
# values with the previous event's tail block, so allow non-strict).
for i in range(1, min(8, len(counters))):
assert counters[i] >= counters[i - 1], (
f"counter went backwards: {counters[i-1]} → {counters[i]}"
)
# ── decode_waveform_v2: currently a stub ─────────────────────────────────────
@pytest.mark.parametrize("event_name", list(FIXTURES_INFO.keys()))
def test_decode_waveform_v2_returns_dict(event_name):
"""decode_waveform_v2 returns a dict with all 4 channels (verified 2026-05-11)."""
path = _fixture_path(event_name)
if not os.path.exists(path):
pytest.skip(f"fixture missing: {path}")
body = _bw_body(path)
result = decode_waveform_v2(body)
assert result is not None
assert set(result.keys()) == {"Tran", "Vert", "Long", "MicL"}
# Multi-channel ground-truth fixtures. Each row: (path, channel, n_to_verify).
# These lock in the channel-rotation hypothesis: segments cycle T → V → L → M,
# with each segment header carrying a 2-sample anchor pair (bytes [14:18])
# for THIS segment's channel plus 2 continuation deltas (bytes [0:4]) for
# the PREVIOUS channel.
MULTICHANNEL_FIXTURES = [
# ALL geo channels fully decoded for every event in the bundle:
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1L.V70"), "Tran", 3328),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1L.V70"), "Vert", 3328),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1L.V70"), "Long", 3328),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1L.JQ0"), "Tran", 3328),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1L.JQ0"), "Vert", 3328),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1L.JQ0"), "Long", 3328),
# SP0 (loud all-channels): NOW fully decodes after the wide-NN walker fix.
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SP0"), "Tran", 3328),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SP0"), "Vert", 3328),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SP0"), "Long", 3328),
# SS0 / SV0 (loud-from-start): walker now reaches 3072–3078 samples per
# channel (out of 3079 total). A few tail samples still missing.
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SS0"), "Tran", 3078),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SS0"), "Vert", 3072),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SS0"), "Long", 3072),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SV0"), "Tran", 3078),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SV0"), "Vert", 3072),
(os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SV0"), "Long", 3072),
# 5-8-26 quiet bundle: events without 30 NN blocks decode FULLY across all channels.
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-a", "M529LKVQ.6S0"), "Tran", 3328),
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-a", "M529LKVQ.6S0"), "Vert", 3328),
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-a", "M529LKVQ.6S0"), "Long", 3328),
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-c", "M529LK44.AB0"), "Tran", 1280),
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-c", "M529LK44.AB0"), "Vert", 1280),
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-c", "M529LK44.AB0"), "Long", 1280),
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-d", "M529LK2V.470"), "Tran", 1280),
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-d", "M529LK2V.470"), "Vert", 1280),
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-d", "M529LK2V.470"), "Long", 1280),
# event-b: 2304 samples × 3 — now fully decodes (was the historical
# walker-stop case; fixed by wide-NN tag support).
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-b", "M529LK5Q.RG0"), "Tran", 2304),
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-b", "M529LK5Q.RG0"), "Vert", 2304),
(os.path.join(os.path.dirname(__file__), "fixtures", "decode-re-5-8-26",
"event-b", "M529LK5Q.RG0"), "Long", 2304),
]
@pytest.mark.parametrize("path,channel,n", MULTICHANNEL_FIXTURES)
def test_decode_waveform_v2_channels_match_truth(path, channel, n):
"""Decoded channels match the BW ASCII export byte-exact for the verified ranges."""
if not os.path.exists(path):
pytest.skip(f"fixture missing: {path}")
with open(path, "rb") as f:
body = f.read()[43:-26]
truth = _full_truth_channel(path, channel)
decoded = decode_waveform_v2(body)
assert decoded is not None
pred = decoded[channel]
assert len(pred) >= n, f"only {len(pred)} samples decoded, expected ≥ {n}"
for i in range(n):
assert pred[i] == truth[i], (
f"{os.path.basename(path)} {channel}[{i}]: pred={pred[i]} truth={truth[i]}"
)
# ── decode_tran_initial: confirmed correct against ground truth ──────────────
# Bundled fixtures for the high-amplitude 5-11-26 events (PPV ~6-7 in/s).
# These cracked the Tran codec — see waveform_codec module docstring.
TRAN_INITIAL_FIXTURES = [
# (path, expected first N Tran samples in 16-count units, # of samples to verify)
(
os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SP0"),
[4, 4, 3, 3, 3, 2, 2, 3, 2, 2, 2, 2, 1, 1, 1, 2, 1, 1, 1, 0, 1, 0],
22,
),
(
os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SS0"),
[-89, -89, -91, -91, -92, -93, -94, -94, -94, -94],
42,
),
(
os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1A.SV0"),
[-745, -762, -771, -774, -779, -794, -808, -811, -811, -819],
46,
),
# Vert-heavy event (T near zero) — segment 0 = 510 samples, all decode correctly.
(
os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1L.JQ0"),
[0] * 4 + [-1, 0, 0, -1, -1, 0],
38,
),
# Mic-heavy event (geos all near zero) — segment 0 = 482 samples.
(
os.path.join(os.path.dirname(__file__), "fixtures", "5-11-26", "M529LL1L.V70"),
[0] * 10,
6,
),
]
def _full_truth(path):
"""Load Tran samples (in 16-count units) from the BW ASCII export."""
return _full_truth_channel(path, "Tran")
def _full_truth_channel(path, channel):
"""Load one channel's samples (in 16-count units) from the BW ASCII export."""
import glob, re
col_idx = {"Tran": 0, "Vert": 1, "Long": 2, "MicL": 3}[channel]
# event-a's TXT has a typo ("M59" vs "M529") — pick the .TXT in the same dir
# rather than assuming exact-name correspondence.
txt_path = path + ".TXT"
if not os.path.exists(txt_path):
candidates = glob.glob(os.path.join(os.path.dirname(path), "*.TXT"))
if candidates:
txt_path = candidates[0]
with open(txt_path, "r", encoding="utf-8", errors="replace") as f:
lines = f.read().splitlines()
header_idx = None
for i, line in enumerate(lines):
if "Tran" in line and "Vert" in line and "Long" in line and "MicL" in line:
header_idx = i
break
if header_idx is None:
return None
out = []
for line in lines[header_idx + 1:]:
parts = re.split(r"\s+", line.strip())
if len(parts) < 4:
continue
try:
out.append(round(float(parts[col_idx]) * 200))
except ValueError:
continue
return out
@pytest.mark.parametrize("path,expected,n_required", TRAN_INITIAL_FIXTURES)
def test_decode_tran_initial_matches_ground_truth(path, expected, n_required):
"""The Tran initial decoder produces values matching the BW ASCII export exactly."""
if not os.path.exists(path):
pytest.skip(f"fixture missing: {path}")
with open(path, "rb") as f:
raw = f.read()
body = raw[43:-26]
decoded = decode_tran_initial(body)
assert decoded is not None
# Check first len(expected) samples match exactly.
for i in range(len(expected)):
assert decoded[i] == expected[i], (
f"sample {i}: decoded={decoded[i]} expected={expected[i]}"
)
# And we got at least n_required samples decoded.
assert len(decoded) >= n_required, (
f"decoded only {len(decoded)} samples, expected at least {n_required}"
)
def test_decode_tran_initial_handles_empty():
assert decode_tran_initial(b"") is None
assert decode_tran_initial(b"not a body") is None
def test_decode_tran_initial_synthetic_body():
"""A synthetic body with preamble + one 10 04 block decodes correctly."""
# Magic + T[0]=10 + T[1]=20 in 16-count units.
# Then 10 04 block with 4 nibbles: (+1, -1, +2, -2)
# Encoded high-nibble first: 0x1F = (1, -1), 0x2E = (2, -2)
body = b"\x00\x02\x00\x00\x0a\x00\x14" + b"\x10\x04" + b"\x1f\x2e"
decoded = decode_tran_initial(body)
# T[0]=10, T[1]=20, then deltas (+1, -1, +2, -2) from T[1]=20
assert decoded == [10, 20, 21, 20, 22, 20]
def test_decode_tran_initial_with_rle():
"""A synthetic body with 00 NN RLE block runs the current Tran value forward."""
# T[0]=5, T[1]=5, then 00 08 RLE block = 8 zero deltas → T[2..9] = 5
body = b"\x00\x02\x00\x00\x05\x00\x05" + b"\x00\x08"
decoded = decode_tran_initial(body)
assert decoded == [5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
def test_decode_tran_initial_full_segment_silent_events():
"""For events with near-silent Tran, segment 0 (~482-510 samples) decodes fully."""
for path, _, _ in TRAN_INITIAL_FIXTURES[3:]: # JQ0 (Vert-heavy) and V70 (Mic-heavy)
if not os.path.exists(path):
pytest.skip(f"fixture missing: {path}")
with open(path, "rb") as f:
body = f.read()[43:-26]
truth = _full_truth(path)
decoded = decode_tran_initial(body)
assert decoded is not None
# The decoder should produce a clean run of samples; check ALL of them
# match truth (segment 0 is fully solved for events where T is near zero).
n = len(decoded)
for i in range(n):
assert decoded[i] == truth[i], (
f"{os.path.basename(path)}: sample {i}: decoded={decoded[i]} truth={truth[i]}"
)
# And we should have decoded at least 400 samples (= segment 0 worth).
assert n >= 400, f"only {n} samples decoded for {path}"
# ── ADC scaling + dB conversion ──────────────────────────────────────────────
def test_decoded_to_adc_counts_geo_scales_by_16():
"""Geo channels in decoder units (16-count) should multiply by 16 to ADC."""
decoded = {"Tran": [0, 1, -2, 100], "Vert": [5], "Long": [-10], "MicL": [813]}
adc = decoded_to_adc_counts(decoded)
assert adc["Tran"] == [0, 16, -32, 1600]
assert adc["Vert"] == [80]
assert adc["Long"] == [-160]
# Mic passes through unchanged (already ADC counts).
assert adc["MicL"] == [813]
def test_decoded_to_adc_counts_empty():
assert decoded_to_adc_counts({}) == {}
assert decoded_to_adc_counts(
{"Tran": [], "Vert": [], "Long": [], "MicL": []}
) == {"Tran": [], "Vert": [], "Long": [], "MicL": []}
def test_mic_count_to_db_zero_is_zero():
assert mic_count_to_db(0) == 0.0
def test_mic_count_to_db_unit_is_reference():
"""count = ±1 → ±81.94 dB (the calibration reference)."""
assert abs(mic_count_to_db(1) - 81.94) < 0.01
assert abs(mic_count_to_db(-1) - (-81.94)) < 0.01
def test_mic_count_to_db_doubles_every_6db():
"""Each doubling of |count| adds ~6.02 dB."""
# count=2 → 87.96 dB (+ 6.02 from 81.94)
assert abs(mic_count_to_db(2) - 87.96) < 0.05
# count=4 → 93.98 dB
assert abs(mic_count_to_db(4) - 93.98) < 0.05
# count=8 → 100.00 dB
assert abs(mic_count_to_db(8) - 100.00) < 0.05
def test_mic_count_to_db_v70_peak():
"""V70 mic peak count 813 → 140.14 dB (matches BW reported PSPL 140.1)."""
assert abs(mic_count_to_db(813) - 140.14) < 0.1
# And the negative-direction equivalent
assert abs(mic_count_to_db(-813) - (-140.14)) < 0.1
# ── End-to-end: decode_a5_frames (production entry point) ───────────────────
def test_decode_a5_frames_empty():
from minimateplus.waveform_codec import decode_a5_frames
assert decode_a5_frames([]) is None
assert decode_a5_frames(None) is None