From 8bfebadd46d86341ea007675ac889cca1af28350 Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Wed, 15 Apr 2026 02:01:28 -0400 Subject: [PATCH] Revert "fix: improve metadata frame detection and update version to v0.12.1" This reverts commit ad7b064b67ac93b1b4bd5b0e65499f7a2ab337d9. --- CHANGELOG.md | 23 ---------------- CLAUDE.md | 37 +++++-------------------- README.md | 6 ++-- docs/instantel_protocol_reference.md | 31 ++++++++------------- minimateplus/client.py | 41 ++++------------------------ minimateplus/protocol.py | 24 ++-------------- sfm/sfm_webapp.html | 12 +++----- 7 files changed, 34 insertions(+), 140 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ef56e59..defc04f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,29 +4,6 @@ All notable changes to seismo-relay are documented here. --- -## v0.12.1 — 2026-04-15 - -### Fixed - -- **Metadata frame not detected when compliance fields are unconfigured** — the - 5A bulk waveform stream contains one metadata frame carrying ASCII compliance - strings. The decoder skipped it only if `b"Project:"` appeared in the frame - payload. On devices where Project/Client/User Name/Seis Loc are all blank, the - device omits those label strings entirely, so the check returned False and the - frame was decoded as ADC waveform samples. The serial number bytes (`"BE11529\0"`) - appeared at sample ~929 followed by `0xFF 0xFF` fill, truncating the second half - of every waveform and producing a flat-line at 0 in/s after ~660 ms. - - Fix: `_decode_a5_waveform` now checks a tuple of needles (`_METADATA_FRAME_NEEDLES`) - against the full frame bytes (`db`, not `db[7:]`): `b"Project:"`, `b"Client:"`, - `b"User Name:"`, `b"Seis Loc:"`, `b"Extended Notes"`, and `b"Geo: "`. The geo - threshold label is always present (monitoring cannot operate without a configured - geo trigger level) and is the reliable universal anchor. The same tuple is used - in `read_bulk_waveform_stream` for the `stop_after_metadata` early-exit path. - The log records which needle matched, aiding future diagnosis. - ---- - ## v0.12.0 — 2026-04-13 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 36e8365..94f0db0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem -(Sierra Wireless RV50 / RV55). Current version: **v0.12.1**. +(Sierra Wireless RV50 / RV55). Current version: **v0.12.0**. --- @@ -137,10 +137,10 @@ counter and streams data for any valid 5A request; using `chunk_num * 0x0400` is confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes. Do not swap them. -### SUB 5A — event-time metadata lives in a middle A5 frame (position variable) +### SUB 5A — event-time metadata lives in A5 frame 7 -The bulk stream sends 9+ A5 response frames. One of the middle frames contains the -compliance setup as it existed when the event was recorded: +The bulk stream sends 9+ A5 response frames. Frame 7 (0-indexed) contains the compliance +setup as it existed when the event was recorded: ``` "Project:" → project description @@ -148,14 +148,10 @@ compliance setup as it existed when the event was recorded: "User Name:" → operator name ← NOT in the 0C record "Seis Loc:" → sensor location ← NOT in the 0C record "Extended Notes"→ notes -"Geo: " → geo threshold (in/s) — always present ``` -**Metadata frame position is variable:** fi==7 for blast events (4-2-26 capture), -fi==6 for desk-thump events (2026-04-14 confirmed). - **IMPORTANT — 5A "Project:" is session-start config, NOT per-event (confirmed 2026-04-05):** -The "Project:" string in the metadata frame reflects the compliance setup from when +The "Project:" string in the A5 frame 7 payload reflects the compliance setup from when the *monitoring session first started*, not the individual event's project name. The per- event project name is correctly stored in the 210-byte 0C waveform record and must be used as the authoritative source. `_decode_a5_metadata_into` therefore only sets @@ -164,27 +160,8 @@ used as the authoritative source. `_decode_a5_metadata_into` therefore only set "Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C record — 5A remains the sole source for those fields and they are set unconditionally. -**Metadata frame detection uses `_METADATA_FRAME_NEEDLES` (CORRECTED 2026-04-15):** -`stop_after_metadata=True` stops the 5A loop as soon as any needle in -`_METADATA_FRAME_NEEDLES` is found in the frame, then sends the termination frame. -`_decode_a5_waveform` uses the same tuple to skip the frame instead of decoding it as ADC -samples. The tuple is: - -```python -_METADATA_FRAME_NEEDLES = ( - b"Project:", b"Client:", b"User Name:", b"Seis Loc:", - b"Extended Notes", b"Geo: ", -) -``` - -**Do NOT check only `b"Project:"`.** The old single-needle check caused a critical bug -(confirmed 2026-04-15): the metadata frame was decoded as waveform ADC samples, making -the serial number bytes `"BE11529\0"` appear at sample ~929 followed by `0xFF 0xFF` fill, -truncating the second half of every waveform. Root cause unknown (may be that the -Project/Client etc. label strings appear differently under certain conditions), but -`b"Geo: "` is the reliable universal anchor — monitoring cannot operate without a geo -threshold configured. The check is applied against the full `db` (= `rsp.data`) bytes, -not `db[7:]`, to eliminate any off-by-7 edge case. +`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears, +then sends the termination frame. ### SUB 5A — STRT record layout and rectime_seconds (CORRECTED 2026-04-14) diff --git a/README.md b/README.md index 47aac6f..1c86de0 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# seismo-relay `v0.12.1` +# seismo-relay `v0.12.0` A ground-up replacement for **Blastware** — Instantel's aging Windows-only software for managing MiniMate Plus seismographs. @@ -263,4 +263,6 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows. - [x] SQLite persistence — events, monitor log, and session history in `seismo_relay.db` - [x] SFM REST API — device control + DB query endpoints, live device cache - [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing -- [ ] Vibra \ No newline at end of file +- [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first) +- [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object +- [ ] Modem manager — push RV50/RV55 configs via Sierra Wireless API diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index aa93090..356cb07 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -103,7 +103,6 @@ | 2026-04-11 | §5.1 | **CONFIRMED — SUB 0x06 (CHANNEL CONFIG READ) now confirmed as event storage range.** Two-step read, data offset = 0x24 (36 bytes). Token=0xFE at params[7]. Last 8 bytes of response: first stored event key (bytes −8:−4) and last stored event key (bytes −4:). Both equal `01110000` when device memory is empty. Used by Blastware to verify erase completion. | | 2026-04-11 | §7.11 (NEW) | **NEW — §7.11 Erase-All Protocol added.** Full wire sequence, SUB 0x06 storage range payload layout, post-erase key counter reset (resets to `0x01110000`). Confirmed from 4-11-26 MITM capture of live Blastware ACH session. | | 2026-04-11 | §14.6 | **RESOLVED — ACH Session Lifecycle is no longer "Future".** `bridges/ach_server.py` fully implements inbound ACH: POLL handshake, device info, event download. State tracked via `ach_state.json` (key-based, with `max_downloaded_key` for post-erase detection). `--clear-after-download` flag added for the standard delete-after-upload workflow. | -| 2026-04-15 | §7.8.3 | **CORRECTED — Metadata frame detection must use multi-needle tuple, not `b"Project:"` alone.** Using only `b"Project:"` caused the metadata frame to be decoded as waveform ADC samples under certain conditions (serial number bytes `"BE11529\0"` appeared at sample ~929, followed by `0xFF 0xFF` fill). Root cause not fully characterised but `b"Project:"` alone is unreliable. Fix: check `_METADATA_FRAME_NEEDLES = (b"Project:", b"Client:", b"User Name:", b"Seis Loc:", b"Extended Notes", b"Geo: ")` against the full `rsp.data` bytes. `b"Geo: "` is the universal fallback — monitoring cannot operate without a configured geo threshold. Applied to both `_decode_a5_waveform` (client.py) and `read_bulk_waveform_stream` (protocol.py). | --- @@ -824,9 +823,9 @@ Waveform: starts at byte 8 of db[7:] | Frame index | Contents | |---|---| | A5[0] | Probe response: STRT record + first waveform chunk | -| A5[fi] | Event-time metadata strings — position variable (fi==7 blast capture, fi==6 desk-thump; detect via `_METADATA_FRAME_NEEDLES`) | -| A5[last] | Terminator frame (page_key=0x0000) — ignored | -| All others | Waveform chunks | +| A5[7] | Event-time metadata strings only (no waveform data) | +| A5[9] | Terminator frame (page_key=0x0000) — ignored | +| A5[1..6,8] | Waveform chunks | **Confirmed from 4-2-26 blast capture (total_samples=9306, pretrig=298, rate=1024 Hz):** @@ -844,7 +843,7 @@ Total: 7633B → 954 naive sample-sets, 948 alignment-corrected ``` Only 948 of 9306 sample-sets captured (10%) — `stop_after_metadata=True` terminated -download after the metadata frame (A5[7] in this capture) was received. +download after A5[7] was received. **Channel identification note:** The 4-2-26 blast saturated all four geophone channels to near-maximum ADC output (~32000–32617 counts). Channel ordering [Tran, Vert, Long, Mic] @@ -1188,22 +1187,15 @@ Two critical differences from `build_bw_frame`: > this equals `n * 0x0400` since `key4[2:4] = 0x0000`. The monotonic formula is correct > for all keys encountered on this device. -The `stop_after_metadata=True` flag causes the loop to stop as soon as any needle in -`_METADATA_FRAME_NEEDLES` is found in the A5 frame data, typically after 7–9 chunks. -A termination frame is always sent before returning. - -**⚠️ CORRECTED 2026-04-15 — do NOT check only `b"Project:"`.** The single-needle check -caused a critical waveform corruption bug: the metadata frame was decoded as ADC samples, -making serial number bytes `"BE11529\0"` appear at sample ~929 followed by `0xFF 0xFF` -fill, truncating the second half of every waveform. Use `_METADATA_FRAME_NEEDLES` (see -`protocol.py`) which includes `b"Geo: "` as the reliable universal anchor. +The `stop_after_metadata=True` flag causes the loop to stop as soon as `b"Project:"` is +found in the accumulated A5 frame data, typically after 7–9 chunks. A termination frame +is always sent before returning. #### 7.8.3 A5 Frame Layout -Each A5 response frame contains a chunk of raw bulk data. One middle frame (position -variable: fi==7 for blast events, fi==6 for desk-thump events) carries the compliance -text block with project-info label-value pairs. The `client` layer searches for ASCII -labels with a null-terminated value read: +Each A5 response frame contains a chunk of raw bulk data. Frame 7 of the stream carries the +compliance text block with all project-info label-value pairs. The `client` layer searches +for ASCII labels with a null-terminated value read: ``` "Project:" → null-terminated project name @@ -1211,10 +1203,9 @@ labels with a null-terminated value read: "User Name:" → null-terminated operator name "Seis Loc:" → null-terminated sensor location "Extended Notes" → null-terminated notes -"Geo: " → geo threshold (float string, in/s) — ALWAYS present ``` -All fields reflect the **setup at event-record time**, not the current device config. +All five fields reflect the **setup at event-record time**, not the current device config. #### 7.8.4 End-of-Stream Behaviour and Chunk Timing diff --git a/minimateplus/client.py b/minimateplus/client.py index 277f70e..6973214 100644 --- a/minimateplus/client.py +++ b/minimateplus/client.py @@ -76,22 +76,6 @@ _COMPLIANCE_SLOT_SIZE = 64 _COMPLIANCE_VALUE_OFFSET = 22 _COMPLIANCE_VALUE_MAX = _COMPLIANCE_SLOT_SIZE - _COMPLIANCE_VALUE_OFFSET # 42 -# Needles used to identify the 5A metadata frame in _decode_a5_waveform. -# The frame carries compliance-setup ASCII strings embedded in the bulk stream. -# Checking multiple needles is required because devices with unconfigured -# Project/Client/etc. fields omit those label strings entirely — the sole -# reliable anchor then becomes b"Geo: " (always present, geo threshold is -# required for monitoring). We also check db (full rsp.data) rather than -# db[7:] to avoid any off-by-7 miss if a label landed in the prefix bytes. -_METADATA_FRAME_NEEDLES: tuple[bytes, ...] = ( - b"Project:", - b"Client:", - b"User Name:", - b"Seis Loc:", - b"Extended Notes", - b"Geo: ", -) - # ── MiniMateClient ──────────────────────────────────────────────────────────── @@ -1472,29 +1456,14 @@ def _decode_a5_waveform( wave[:24].hex(' '), ) - # Metadata frame: contains compliance-setup ASCII strings. + # Metadata frame: contains "Project:", "Client:", etc. strings. # Originally assumed to be always fi==7 (A5[7] in 4-2-26 blast capture), # but confirmed variable position — it appears at whatever chunk index the # device places it (observed at fi=6 for desk-thump events 2026-04-14). - # - # Detection uses MULTIPLE needles against the full frame bytes (db, NOT w) - # because: - # 1. Devices with no Project/Client/etc. configured omit those label strings - # entirely — checking only b"Project:" misses the frame for unconfigured - # units (confirmed 2026-04-15: Brian's device had no fields set, so - # "Project:" was absent and the frame was decoded as ADC samples, causing - # the serial number bytes "BE11529\0" to appear as waveform data). - # 2. Checking db (full rsp.data) rather than w (db[7:]) eliminates any - # off-by-7 risk if a label were to land in the first 7 bytes. - # 3. b"Geo: " is always present (geo threshold is required for monitoring) - # and serves as the universal fallback anchor. - elif any(needle in db for needle in _METADATA_FRAME_NEEDLES): - log.info( - "_decode_a5_waveform: fi=%d skipped (metadata frame, " - "needle=%r)", - fi, - next(n for n in _METADATA_FRAME_NEEDLES if n in db), - ) + # Skip ANY frame whose raw bytes contain b"Project:" — this is the same + # anchor used by stop_after_metadata in read_bulk_waveform_stream. + elif b"Project:" in w: + log.info("_decode_a5_waveform: fi=%d skipped (metadata frame)", fi) continue # Terminator frames have page_key=0x0000 and are excluded upstream diff --git a/minimateplus/protocol.py b/minimateplus/protocol.py index f3d5dfe..35e99cf 100644 --- a/minimateplus/protocol.py +++ b/minimateplus/protocol.py @@ -125,20 +125,6 @@ _BULK_COUNTER_STEP = 0x0400 # chunk counter increment per chunk ✅ # protocol requirement. Confirmed 2026-04-06: 0x0400 for chunk 1 works; 0x1004 # causes a 120-second device timeout. Formula n * 0x0400 is used for all chunks. -# Needles for identifying the 5A metadata frame. -# Devices with no Project/Client/etc. configured omit those label strings, -# so b"Geo: " (geo threshold — always present) is the universal fallback. -# Mirror of _METADATA_FRAME_NEEDLES in client.py; kept here for the -# stop_after_metadata check in read_bulk_waveform_stream. -_METADATA_FRAME_NEEDLES: tuple[bytes, ...] = ( - b"Project:", - b"Client:", - b"User Name:", - b"Seis Loc:", - b"Extended Notes", - b"Geo: ", -) - # Default timeout values (seconds). # MiniMate Plus is a slow device — keep these generous. DEFAULT_RECV_TIMEOUT = 10.0 @@ -631,13 +617,9 @@ class MiniMateProtocol: break raise - # Detect metadata frame using the same multi-needle set used by - # _decode_a5_waveform. Devices with no Project/Client/etc. - # configured omit those label strings; b"Geo: " is the fallback. - _is_metadata = any(n in rsp.data for n in _METADATA_FRAME_NEEDLES) log.warning( - "5A RX chunk=%d page_key=0x%04X data_len=%d is_metadata=%s", - chunk_num, rsp.page_key, len(rsp.data), _is_metadata, + "5A RX chunk=%d page_key=0x%04X data_len=%d contains_Project=%s", + chunk_num, rsp.page_key, len(rsp.data), b"Project:" in rsp.data, ) if rsp.page_key == 0x0000: @@ -647,7 +629,7 @@ class MiniMateProtocol: frames_data.append(rsp.data) - if stop_after_metadata and _is_metadata: + if stop_after_metadata and b"Project:" in rsp.data: log.debug("5A A5[%d] metadata found — stopping early", chunk_num) break else: diff --git a/sfm/sfm_webapp.html b/sfm/sfm_webapp.html index 3433df5..3078fa4 100644 --- a/sfm/sfm_webapp.html +++ b/sfm/sfm_webapp.html @@ -1574,10 +1574,7 @@ function _buildWaveformCharts(data, chartsEl, emptyEl, chartsStore) { return; } - // Clip to configured record window — device streams extra zero-padded frames - // beyond total_samples; showing them just adds a flat tail to every chart. - const display = data.total_samples ? Math.min(data.total_samples, decoded) : decoded; - const times = Array.from({length: display}, (_, i) => ((i - pretrig) / sr * 1000).toFixed(2)); + const times = Array.from({length: decoded}, (_, i) => ((i - pretrig) / sr * 1000).toFixed(2)); if (emptyEl) emptyEl.style.display = 'none'; chartsEl.style.display = 'flex'; chartsEl.style.flexDirection = 'column'; @@ -1595,7 +1592,7 @@ function _buildWaveformCharts(data, chartsEl, emptyEl, chartsStore) { if (isGeo) { const scale = geoRange / 32767; - plotData = samples.slice(0, display).map(s => s * scale); + plotData = samples.map(s => s * scale); // Use the device-recorded peak from the 0C waveform record — authoritative // and matches Blastware. Computing from raw samples can catch rogue // near-full-scale values from decoding artifacts. @@ -1606,10 +1603,9 @@ function _buildWaveformCharts(data, chartsEl, emptyEl, chartsStore) { ttFmt = v => `${ch}: ${v.toFixed(5)} in/s`; tickFmt = v => v.toFixed(4); } else { - const clippedMic = samples.slice(0, display); - const peakCounts = Math.max(...clippedMic.map(Math.abs)); + const peakCounts = Math.max(...samples.map(Math.abs)); const micScale = (micPeakPsi !== null && peakCounts > 0) ? Math.abs(micPeakPsi) / peakCounts : 1.0; - plotData = clippedMic.map(s => s * micScale); + plotData = samples.map(s => s * micScale); const peakPsi = Math.max(...plotData.map(Math.abs)); const peakDbl = peakPsi > 0 ? 20 * Math.log10(peakPsi / DBL_REF) : -Infinity; peakLabel = `${peakDbl.toFixed(1)} dBL`;