From 0fbb39c21abf8f31f0e78d1045a89a0c02a18805 Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Fri, 1 May 2026 18:37:34 -0400 Subject: [PATCH 01/11] Big event bugfix. see details: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## v0.13.0 — 2026-05-01 ### Fixed - **SUB 5A bulk waveform stream — over-read bug for events ≥ 2 sec.** `read_bulk_waveform_stream` was walking the chunk counter past the actual end of the event, picking up post-event circular-buffer garbage that corrupted reconstructed Blastware files for any waveform > ~1 sec. The loop now extracts the event's `end_offset` from the STRT record at `data[23:27]` of the probe response and stops the chunk walk when the next counter would step past it. Verified against three BW MITM captures (4-27-26 + 5-1-26): 2-sec event drops from 37 over-read chunks to 7 bounded chunks; 3-sec drops to 9; non-zero-start "event 2" drops to 9. ### Added - `framing.bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)` — computes the corrected SUB 5A TERM frame's `(offset_word, params)` per the formula confirmed across all 3 BW captures. Not yet wired into `read_bulk_waveform_stream` (the legacy TERM is still used to preserve the existing `blastware_file.write_blastware_file` frame-structure expectations); available for the next iteration that switches to BW's 0x0200 chunk step. - `framing.parse_strt_end_offset(a5_data)` — extracts the event-end pointer from the STRT record in an A5 response payload. --- CHANGELOG.md | 54 ++++++++++++++ CLAUDE.md | 4 +- minimateplus/framing.py | 148 ++++++++++++++++++++++++++++++++++----- minimateplus/protocol.py | 55 ++++++++++++--- seismo_lab.py | 132 ++-------------------------------- 5 files changed, 235 insertions(+), 158 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 609d889..3737924 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,60 @@ All notable changes to seismo-relay are documented here. --- +## v0.13.0 — 2026-05-01 + +### Fixed + +- **SUB 5A bulk waveform stream — over-read bug for events ≥ 2 sec.** + `read_bulk_waveform_stream` was walking the chunk counter past the actual + end of the event, picking up post-event circular-buffer garbage that + corrupted reconstructed Blastware files for any waveform > ~1 sec. The + loop now extracts the event's `end_offset` from the STRT record at + `data[23:27]` of the probe response and stops the chunk walk when the next + counter would step past it. Verified against three BW MITM captures + (4-27-26 + 5-1-26): 2-sec event drops from 37 over-read chunks to 7 + bounded chunks; 3-sec drops to 9; non-zero-start "event 2" drops to 9. + +### Added + +- `framing.bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)` — + computes the corrected SUB 5A TERM frame's `(offset_word, params)` per the + formula confirmed across all 3 BW captures. Not yet wired into + `read_bulk_waveform_stream` (the legacy TERM is still used to preserve the + existing `blastware_file.write_blastware_file` frame-structure expectations); + available for the next iteration that switches to BW's 0x0200 chunk step. +- `framing.parse_strt_end_offset(a5_data)` — extracts the event-end pointer + from the STRT record in an A5 response payload. + +### Documentation + +- **CLAUDE.md and `docs/instantel_protocol_reference.md` extensively + rewritten** to reflect the corrected SUB 5A protocol. See: + - CLAUDE.md "SUB 5A — chunk counter formula (REWRITTEN 2026-05-01)" + - CLAUDE.md "SUB 5A — STRT record encodes end_offset" + - CLAUDE.md "SUB 5A — TERM frame formula" + - CLAUDE.md "SUB 5A — fixed metadata pages 0x1002 and 0x1004" + - CLAUDE.md "SUB 0A — WAVEHDR response length distinguishes events from + boundaries" (0x46 = real event, 0x2C = boundary marker) + - protocol reference §7.8.5 / §7.8.6 / §7.8.7 / §7.8.8 +- The previous chunk-counter formula (`max(key4[2:4], 0x0400) + (chunk-1) * + 0x0400`) is now marked DEPRECATED and explicitly tagged WRONG with + pointers to the new sections, so future work doesn't re-derive it. + +### Known minor diffs vs Blastware (deferred to a follow-up) + +- We still use the OLD 0x0400 chunk step rather than BW's 0x0200; switching + also requires updating `blastware_file.write_blastware_file`'s skip values + and "extra chunk after metadata" logic, which depends on a fresh capture + to verify. +- We still use the legacy fixed `offset_word=0x005A` TERM frame rather than + BW's `end_offset - next_boundary` formula, for the same reason. +- Two fixed metadata pages at counter `0x1002` and `0x1004` are not yet + read explicitly; under the current 0x0400 walk their content is reachable + via the sample chunk that covers buffer addresses `[0x1000, 0x1400)`. + +--- + ## v0.12.5 — 2026-04-21 ### Changed diff --git a/CLAUDE.md b/CLAUDE.md index 16089fe..240e96b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem -(Sierra Wireless RV50 / RV55). Current version: **v0.12.3**. +(Sierra Wireless RV50 / RV55). Current version: **v0.13.0**. When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document @@ -41,7 +41,7 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c | Event header / first key | 1E | ✅ | | Waveform header | 0A | ✅ | | Waveform record (peaks, timestamp, project) | 0C | ✅ | -| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ new v0.6.0 | +| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ over-read bug fixed v0.13.0 (chunk loop bounded by STRT end_offset); minor wire diffs vs BW deferred — see "SUB 5A — chunk counter formula" | | Event advance / next key | 1F | ✅ | | **Write commands (push config to device)** | **68–83** | ✅ new v0.8.0 | | **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 | diff --git a/minimateplus/framing.py b/minimateplus/framing.py index 3adf4ce..2011cc9 100644 --- a/minimateplus/framing.py +++ b/minimateplus/framing.py @@ -123,8 +123,11 @@ def build_5a_frame(offset_word: int, raw_params: bytes) -> bytes: Returns: Complete frame bytes: [ACK][STX][stuffed_section][chk][ETX] """ - if len(raw_params) not in (10, 11): - raise ValueError(f"raw_params must be 10 or 11 bytes, got {len(raw_params)}") + if len(raw_params) not in (10, 11, 12): + # 10 = termination params; 11 = regular probe / chunk params; + # 12 = metadata-page params (extra trailing 0x00 — BW byte-perfect quirk + # for the two fixed metadata reads at counter=0x1002 and 0x1004). + raise ValueError(f"raw_params must be 10/11/12 bytes, got {len(raw_params)}") # Build stuffed section between STX and checksum s = bytearray() @@ -398,28 +401,21 @@ def bulk_waveform_params(key4: bytes, counter: int, *, is_probe: bool = False) - def bulk_waveform_term_params(key4: bytes, counter: int) -> bytes: """ - Build the 10-byte params block for the SUB 5A termination request. + DEPRECATED 2026-05-01 — see bulk_waveform_term_v2(). - The termination request uses offset=0x005A and a DIFFERENT params layout — - the leading 0x00 byte is dropped, key4[0:2] shifts to params[0:2], and the - counter high byte is at params[2]: + Build the 10-byte params block for the SUB 5A termination request, OLD layout + (used in conjunction with the fixed offset_word=0x005A). Kept for backward + compatibility — produces a tiny ~100-byte device-side terminator response + rather than the proper partial-last-chunk + footer payload that BW gets. params[0] = key4[0] params[1] = key4[1] params[2] = (counter >> 8) & 0xFF params[3:] = zeros - Counter for the termination request = last_regular_counter + 0x0400. - - Confirmed from 1-2-26 BW TX capture: final request (frame 83) uses - offset=0x005A, params[0:3] = key4[0:2] + term_counter_hi. - - Args: - key4: 4-byte waveform key. - counter: Termination counter (= last regular counter + 0x0400). - - Returns: - 10-byte params block. + Use bulk_waveform_term_v2() for new code — it computes the verified + offset_word + params from end_offset (extracted from STRT) and the last + chunk counter. """ if len(key4) != 4: raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}") @@ -430,6 +426,123 @@ def bulk_waveform_term_params(key4: bytes, counter: int) -> bytes: return bytes(p) +def bulk_waveform_term_v2( + key4: bytes, + end_offset: int, + last_chunk_counter: int, +) -> tuple[int, bytes]: + """ + Compute the SUB 5A TERM frame's offset_word and 10-byte params block. + + Confirmed across 3 events (4-27-26 + 5-1-26 captures): + + next_boundary = last_chunk_counter + 0x0200 + offset_word = end_offset - next_boundary (residual byte count) + params[0] = key4[0] (= 0x01 on every observed device) + params[1] = key4[1] (= 0x11) + params[2] = (next_boundary >> 8) & 0xFF + params[3] = next_boundary & 0xFF + params[4:10] = zeros + + Verification: + | end_offset | last_chunk | next_boundary | offset_word | params[2:4] | + | 0x1ABE | 0x1800 | 0x1A00 | 0x00BE | 1A 00 | + | 0x21F2 | 0x1E00 | 0x2000 | 0x01F2 | 20 00 | + | 0x417E | 0x3E38 | 0x4038 | 0x0146 | 40 38 | + + The device receives `requested_address = (params[2] << 8) | offset_word` + and replies with `(end_offset - next_boundary)` bytes of waveform tail + starting at `next_boundary` — including the 26-byte file footer. + + Args: + key4: 4-byte waveform key for this event. + end_offset: Event-end pointer (= `(end_key[2] << 8) | end_key[3]` + from the STRT record at data[23:27] of A5[0]). + last_chunk_counter: Counter of the last full 0x0200-byte chunk fetched + (the chunk that covers [last_chunk_counter, + last_chunk_counter + 0x0200)). + + Returns: + (offset_word, params10) tuple. Pass as + `build_5a_frame(offset_word, params)`. + + Raises: + ValueError: on inconsistent inputs. + """ + if len(key4) != 4: + raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}") + next_boundary = last_chunk_counter + 0x0200 + if next_boundary > 0xFFFF: + raise ValueError( + f"next_boundary 0x{next_boundary:04X} exceeds uint16; check inputs" + ) + if end_offset <= last_chunk_counter: + raise ValueError( + f"end_offset 0x{end_offset:04X} must be > " + f"last_chunk_counter 0x{last_chunk_counter:04X}" + ) + offset_word = end_offset - next_boundary + if offset_word < 0: + # Last chunk overshot end_offset; caller should have stopped one chunk + # earlier. Treat as zero residual. + offset_word = 0 + if offset_word > 0xFFFF: + raise ValueError( + f"offset_word 0x{offset_word:04X} exceeds uint16" + ) + p = bytearray(10) + p[0] = key4[0] + p[1] = key4[1] + p[2] = (next_boundary >> 8) & 0xFF + p[3] = next_boundary & 0xFF + return offset_word, bytes(p) + + +# ── End-offset extraction from STRT record ──────────────────────────────────── + +STRT_MARKER = b"STRT" + + +def parse_strt_end_offset(a5_data: bytes) -> Optional[int]: + """ + Extract the event-end offset from the STRT record in an A5 response payload. + + The first A5 response (the probe response, or the first chunk for events + with non-zero start_key[2:4]) contains a STRT record at byte offset 17 of + `data`. Layout: + + data[17:21] "STRT" + data[21:23] ff fe sentinel + data[23:27] end_key ← 4-byte key of where this event ENDS + data[27:31] start_key + ... + + Returns `(end_key[2] << 8) | end_key[3]` — the absolute device-buffer + address where the event ends. Use this to bound the chunk loop and to + compute the TERM frame. + + Verified end_offset values: + | event start_key | end_key | end_offset | + | 01110000 | 01111ABE | 0x1ABE | + | 01110000 | 011121F2 | 0x21F2 | + | 011121F2 | 0111417E | 0x417E | + + Args: + a5_data: The `data` field of an A5 response frame (frame.data). + + Returns: + The end_offset (uint16) if STRT is found, else None. + """ + pos = a5_data.find(STRT_MARKER) + if pos < 0 or pos + 10 > len(a5_data): + return None + # data[pos+4:pos+6] is "ff fe"; data[pos+6:pos+10] is end_key. + end_key = a5_data[pos + 6 : pos + 10] + if len(end_key) < 4: + return None + return (end_key[2] << 8) | end_key[3] + + # ── Pre-built POLL frames ───────────────────────────────────────────────────── # # POLL (SUB 0x5B) uses the same two-step pattern as all other reads — the @@ -470,7 +583,6 @@ class S3Frame: # ── Streaming S3 frame parser ───────────────────────────────────────────────── - class S3FrameParser: """ Incremental byte-stream parser for S3→BW response frames. diff --git a/minimateplus/protocol.py b/minimateplus/protocol.py index 0a69f93..2abeb1a 100644 --- a/minimateplus/protocol.py +++ b/minimateplus/protocol.py @@ -35,6 +35,8 @@ from .framing import ( token_params, bulk_waveform_params, bulk_waveform_term_params, + bulk_waveform_term_v2, + parse_strt_end_offset, POLL_PROBE, POLL_DATA, SESSION_RESET, @@ -122,16 +124,21 @@ DATA_LENGTHS: dict[int, int] = { } # SUB 5A (BULK_WAVEFORM_STREAM) protocol constants. -# Confirmed from 1-2-26 BW TX capture analysis (2026-04-02). -_BULK_CHUNK_OFFSET = 0x1004 # offset field for probe + all regular chunk requests ✅ -_BULK_TERM_OFFSET = 0x005A # offset field for termination request ✅ -_BULK_COUNTER_STEP = 0x0400 # chunk counter increment per chunk ✅ -# Chunk counter formula: key4[2:4] + (chunk_num - 1) * 0x0400 -# where key4[2:4] is the event's circular-buffer base offset ((key4[2]<<8)|key4[3]). -# Earlier captures showed 0x1004 for chunk 1 of key 01110000 — that was a Blastware -# artifact. For keys where key4[2:4] != 0x0000 (e.g. key 01111884) the old -# "n * 0x0400" formula sends counters from the wrong buffer region and the device -# returns data from a different event. Confirmed correct 2026-04-24. +# +# 2026-05-01 minimal-fix: the chunk-counter walk is now bounded by the event's +# `end_offset` extracted from the STRT record at data[23:27] of the probe +# response. Without this bound the loop kept asking for chunks past the event +# end and the device responded with post-event circular-buffer garbage, +# corrupting reconstructed Blastware files for events ≥ 2 sec. +# +# We keep the OLD 0x0400 chunk step here (BW actually uses 0x0200 — see §7.8.5 +# of the protocol reference for the corrected understanding) because the +# existing blastware_file.py builder relies on the 0x0400-step frame structure +# to produce valid files. Switching to BW's 0x0200 step is a separate task +# that also requires updating the file builder. +_BULK_CHUNK_OFFSET = 0x1004 # offset_word for probe + all chunk requests +_BULK_TERM_OFFSET = 0x005A # offset_word for the legacy terminator +_BULK_COUNTER_STEP = 0x0400 # chunk counter increment # Default timeout values (seconds). # MiniMate Plus is a slow device — keep these generous. @@ -610,6 +617,24 @@ class MiniMateProtocol: frames_data.append(rsp) log.debug("5A A5[0] page_key=0x%04X %d bytes", rsp.page_key, len(rsp.data)) + # ── Parse STRT end_offset from probe response (NEW 2026-05-01) ──────── + # The first A5 response contains a STRT record at data[17:]. The + # bytes at data[23:27] are the event's end-key, whose low 16 bits + # are the absolute device-buffer address where the event ends. Use + # this to bound the chunk loop and stop the over-read past event end. + # See docs/instantel_protocol_reference.md §7.8.5 and CLAUDE.md + # "SUB 5A — STRT record encodes end_offset". + _end_offset = parse_strt_end_offset(rsp.data) + if _end_offset is None: + # Defensive fallback — let max_chunks cap the walk. + log.warning("5A: STRT not found in probe; cannot bound chunk loop") + _end_offset = 0xFFFF + else: + log.debug( + "5A STRT start_offset=0x%04X end_offset=0x%04X size=0x%04X", + _key4_offset, _end_offset, _end_offset - _key4_offset, + ) + # ── Step 2: chunk loop ─────────────────────────────────────────────── # Counter formula: _chunk_base + (chunk_num - 1) * 0x0400 # where _chunk_base = max(key4[2:4], 0x0400). @@ -629,6 +654,16 @@ class MiniMateProtocol: _chunk_base = max(_key4_offset, _BULK_COUNTER_STEP) for chunk_num in range(1, max_chunks + 1): counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP + # Stop when we'd step past the event end (NEW 2026-05-01). Without + # this, the device returns post-event circular-buffer data which + # corrupts the reconstructed file for events ≥ 2 sec. + if counter >= _end_offset: + log.debug( + "5A chunk loop done at counter=0x%04X (end=0x%04X); " + "%d chunks fetched", + counter, _end_offset, len(frames_data), + ) + break params = bulk_waveform_params(key4, counter) log.debug("5A chunk %d counter=0x%04X", chunk_num, counter) self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params)) diff --git a/seismo_lab.py b/seismo_lab.py index 723a921..f78e6a1 100644 --- a/seismo_lab.py +++ b/seismo_lab.py @@ -114,8 +114,6 @@ class BridgePanel(tk.Frame): on_capture_complete(bw_path, s3_path, label)— a capture segment finished """ - def __init__(self, parent: tk.Widget, on_bridge_started, on_bridge_stopped, - on_capture_started=None, on_capture_complete=None, **kw): def __init__(self, parent: tk.Widget, on_bridge_started, on_bridge_stopped, on_capture_started=None, on_capture_complete=None, **kw): super().__init__(parent, bg=BG2, **kw) @@ -141,10 +139,6 @@ class BridgePanel(tk.Frame): self._cap_history: list[dict] = [] # {label, status, bw, s3} # mode self._mode = tk.StringVar(value="serial") - # Capture state - self._capturing = False - self._cap_label: Optional[str] = None - self._cap_history: list[dict] = [] # {label, status, bw, s3} self._build() self._poll_stdout() self._poll_tcp_log() @@ -246,18 +240,6 @@ class BridgePanel(tk.Frame): command=self._stop_capture, state="disabled") self.stop_cap_btn.pack(side=tk.LEFT, padx=4) - tk.Frame(btn_row, bg=BG2, width=16).pack(side=tk.LEFT) # spacer - - self.cap_btn = tk.Button(btn_row, text="⬤ New Capture", bg=ORANGE, fg="#000000", - relief="flat", padx=10, cursor="hand2", font=MONO_B, - command=self._start_capture, state="disabled") - self.cap_btn.pack(side=tk.LEFT, padx=4) - - self.stop_cap_btn = tk.Button(btn_row, text="■ Stop Capture", bg=BG3, fg=RED, - relief="flat", padx=10, cursor="hand2", font=MONO_B, - command=self._stop_capture, state="disabled") - self.stop_cap_btn.pack(side=tk.LEFT, padx=4) - self.mark_btn = tk.Button(btn_row, text="Add Mark", bg=BG3, fg=FG, relief="flat", padx=10, cursor="hand2", font=MONO, command=self.add_mark, state="disabled") @@ -319,7 +301,6 @@ class BridgePanel(tk.Frame): # Log output self.log_view = scrolledtext.ScrolledText( - self, height=14, font=MONO_SM, self, height=14, font=MONO_SM, bg=BG, fg=FG, insertbackground=FG, relief="flat", state="disabled", @@ -462,15 +443,12 @@ class BridgePanel(tk.Frame): self.start_btn.configure(state="disabled") self.stop_btn.configure(state="normal", bg=RED) self.cap_btn.configure(state="normal") - self.cap_btn.configure(state="normal") self._append_log(f"== Bridge started [{ts}] ==\n") self._append_log(" Click 'New Capture' when ready to record.\n") - self._on_started(struct_bin_path) - # Notify parent — no raw files yet, just the structured bin path self._on_started(struct_bin_path) - def stop_bridge(self) -> None: + def _stop_serial(self) -> None: if self.process and self.process.poll() is None: self.process.terminate() try: @@ -480,17 +458,6 @@ class BridgePanel(tk.Frame): self._bridge_ended() self._on_stopped() - def _bridge_ended(self) -> None: - self.status_var.set("Stopped") - self.start_btn.configure(state="normal") - self.stop_btn.configure(state="disabled", bg=BG3) - self.cap_btn.configure(state="disabled") - self.stop_cap_btn.configure(state="disabled", bg=BG3) - self.mark_btn.configure(state="disabled") - self._capturing = False - self._cap_label = None - self._append_log("== Bridge stopped ==\n") - def _reader_thread(self) -> None: if not self.process or not self.process.stdout: return @@ -531,9 +498,7 @@ class BridgePanel(tk.Frame): # ── capture control ─────────────────────────────────────────────────── def _start_capture(self) -> None: - """Ask for a label and tell the bridge to start writing raw tap files.""" - if not self.process or self.process.poll() is not None: - return + """Ask for a label and start writing raw tap files (serial subprocess or TCP files).""" label = simpledialog.askstring( "New Capture", "Label for this capture\n(e.g. 'recording_mode_continuous').\nLeave blank for timestamp only:", @@ -542,88 +507,6 @@ class BridgePanel(tk.Frame): if label is None: return # user hit Cancel label = label.strip() - try: - self.process.stdin.write(f"CAP_START:{label}\n") - self.process.stdin.flush() - except Exception as e: - messagebox.showerror("Error", f"Failed to start capture:\n{e}") - return - self._capturing = True - self._cap_label = label or datetime.datetime.now().strftime("%H%M%S") - self.cap_btn.configure(state="disabled") - self.stop_cap_btn.configure(state="normal", bg=RED) - self.mark_btn.configure(state="normal") - self._append_log(f"[CAPTURE] Starting: {self._cap_label!r}...\n") - # Add to history as recording (paths filled in when [CAP_START] arrives) - self._cap_history.append({"label": self._cap_label, "status": "recording", - "bw": None, "s3": None}) - self._refresh_hist() - - def _stop_capture(self) -> None: - """Tell the bridge to flush and close the current raw tap files.""" - if not self.process or self.process.poll() is not None: - return - try: - self.process.stdin.write("CAP_STOP\n") - self.process.stdin.flush() - except Exception as e: - messagebox.showerror("Error", f"Failed to stop capture:\n{e}") - # UI is updated when [CAP_STOP] arrives in stdout - - def _on_cap_started_msg(self, bw_path: str, s3_path: str) -> None: - """Called when bridge confirms capture has started (files are open).""" - # Fill in paths for the last 'recording' history entry - for entry in reversed(self._cap_history): - if entry["status"] == "recording" and entry["bw"] is None: - entry["bw"] = bw_path - entry["s3"] = s3_path - break - if self._on_cap_started: - self._on_cap_started(bw_path, s3_path, self._cap_label or "") - - def _on_cap_stopped_msg(self, bw_path: str, s3_path: str) -> None: - """Called when bridge confirms capture has stopped (files are closed).""" - label = self._cap_label or "capture" - # Mark history entry as done - for entry in reversed(self._cap_history): - if entry["status"] == "recording": - entry["status"] = "done" - entry["bw"] = bw_path - entry["s3"] = s3_path - break - self._refresh_hist() - self._capturing = False - self._cap_label = None - self.cap_btn.configure(state="normal") - self.stop_cap_btn.configure(state="disabled", bg=BG3) - self._append_log(f"[CAPTURE] Done: {label!r} — ready in Analyzer\n") - if self._on_cap_complete: - self._on_cap_complete(bw_path, s3_path, label) - - def _refresh_hist(self) -> None: - self._hist_lb.delete(0, tk.END) - for entry in self._cap_history: - icon = "🔴" if entry["status"] == "recording" else "✅" - label = entry["label"] or "(unlabeled)" - self._hist_lb.insert(tk.END, f" {icon} {label}") - if self._cap_history: - self._hist_lb.see(tk.END) - - def _on_hist_dblclick(self, _e=None) -> None: - sel = self._hist_lb.curselection() - if not sel: - return - entry = self._cap_history[sel[0]] - if entry["status"] == "done" and entry["bw"] and entry["s3"]: - if self._on_cap_complete: - self._on_cap_complete(entry["bw"], entry["s3"], entry["label"]) - - # ── mark ────────────────────────────────────────────────────────────── - - def add_mark(self) -> None: - if not self.process or not self.process.stdin or self.process.poll() is not None: - return - label = label.strip() self._capturing = True self._cap_label = label or datetime.datetime.now().strftime("%H%M%S") @@ -664,6 +547,7 @@ class BridgePanel(tk.Frame): self._append_log(f"[CAPTURE] Starting: {self._cap_label!r}...\n") def _stop_capture(self) -> None: + """Flush and close the current raw tap files (TCP) or signal the bridge subprocess (serial).""" if self._mode.get() == "tcp": with self._tcp_cap_lock: bw_path = self._tcp_cap_bw_path @@ -686,6 +570,7 @@ class BridgePanel(tk.Frame): self.process.stdin.flush() except Exception as e: messagebox.showerror("Error", f"Failed to stop capture:\n{e}") + # UI is updated when [CAP_STOP] arrives in stdout # ── TCP mode ────────────────────────────────────────────────────────── @@ -1173,14 +1058,6 @@ class AnalyzerPanel(tk.Frame): self.state.bw_path = bwp self._do_analyze(s3p, bwp) - def _browse_bin(self) -> None: - path = filedialog.askopenfilename( - title="Select session .bin file", - filetypes=[("Binary", "*.bin"), ("All files", "*.*")], - ) - if path: - self.bin_var.set(path) - def _do_analyze(self, s3_path: Path, bw_path: Path) -> None: self.status_var.set("Parsing...") self.update_idletasks() @@ -1611,7 +1488,6 @@ class AnalyzerPanel(tk.Frame): w.configure(state="normal"); w.insert(tk.END, "\n"); w.configure(state="disabled") -# ───────────────────────────────────────────────────────────────────────────── # ───────────────────────────────────────────────────────────────────────────── # Serial Watch panel — tap the RS-232 line between device and modem # ───────────────────────────────────────────────────────────────────────────── From d758825c672703c9e51e3e3dfcfe906604b871e1 Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Fri, 1 May 2026 20:28:55 -0400 Subject: [PATCH 02/11] fix(protocol): correct continuous-mode record header classification for accurate timestamp extraction --- CHANGELOG.md | 21 ++++++++++++++++ CLAUDE.md | 2 +- minimateplus/client.py | 55 +++++++++++++++++++++++++++--------------- 3 files changed, 58 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3737924..a72537f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,27 @@ All notable changes to seismo-relay are documented here. --- +## v0.13.1 — 2026-05-01 + +### Fixed + +- **`_extract_record_type` — Continuous-mode record headers misclassified as Unknown.** + In single-shot mode the 0C waveform record's 9-byte header puts the sub_code + marker `0x10` at byte 1, with the day at byte 0. In Continuous mode the + header is 10 bytes with the marker at byte 0 *and* byte 2, and the day at + byte 1. Previous logic only inspected byte 1 and treated any value other + than `0x10` / `0x03` as `"Unknown"`, which prevented `event.timestamp` from + being populated for any continuous-mode event whose day-of-month wasn't + exactly 3 or 16. As a downstream effect, `blastware_filename()` saw + `event.timestamp == None`, fell back to `stem="0000"` / `ab="00"`, and + produced filenames like `M5290000.000`. Discovered from a live SFM run on + BE11529 in continuous mode (day-of-month = 5). + Now disambiguates by checking BOTH byte 0 and byte 2: if both are `0x10`, + it's the 10-byte continuous header; else if byte 1 is `0x10`, it's the + 9-byte single-shot header. Day-of-month no longer matters. + +--- + ## v0.13.0 — 2026-05-01 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 240e96b..f908461 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem -(Sierra Wireless RV50 / RV55). Current version: **v0.13.0**. +(Sierra Wireless RV50 / RV55). Current version: **v0.13.1**. When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document diff --git a/minimateplus/client.py b/minimateplus/client.py index 8f01f3d..12664c0 100644 --- a/minimateplus/client.py +++ b/minimateplus/client.py @@ -1638,31 +1638,48 @@ def _decode_a5_waveform( def _extract_record_type(data: bytes) -> Optional[str]: """ - Decode the recording mode from byte[1] of the 210-byte waveform record. + Detect the waveform record format by inspecting the first 3 bytes of the + 210-byte record returned by SUB 0C. - Byte[1] is the sub-record code that immediately follows the day byte in the - 9-byte timestamp header at the start of each waveform record: - [day:1] [sub_code:1] [month:1] [year:2 BE] ... + Two formats exist (confirmed from BE11529 captures and CLAUDE.md docs): - Confirmed codes (✅ 2026-04-01): - 0x10 → "Waveform" (continuous / single-shot mode) + Single-shot mode — 9-byte header: + data[0] = day + data[1] = 0x10 ← sub_code marker + data[2] = month + data[3:5] = year (BE) + ... - Histogram mode code is not yet confirmed — a histogram event must be - captured with debug=true to identify it. Returns None for unknown codes. + Continuous mode — 10-byte header: + data[0] = 0x10 ← marker A + data[1] = day ← variable (NOT 0x10) + data[2] = 0x10 ← marker B + data[3] = month + data[4:6] = year (BE) + ... + + Disambiguate by checking BOTH data[0] and data[2]: + - data[0]==0x10 AND data[2]==0x10 → Continuous (10-byte header) + - data[1]==0x10 → Single-shot (9-byte header) + - otherwise → Unknown + + Previous logic only checked data[1] and so mis-classified continuous-mode + records as "Unknown(0xXX)" wherever day != 0x10 — see filename + M5290000.000 regression report (2026-05-01 SFM log). """ - if len(data) < 2: + if len(data) < 3: return None - code = data[1] - if code == 0x10: - return "Waveform" - if code == 0x03: - # Continuous mode waveform record (confirmed by user — NOT a monitor log). - # The byte layout differs from 0x10 single-shot records: the timestamp - # fields decode as garbage under the 0x10 waveform layout. - # TODO: confirm correct timestamp layout for 0x03 records from a known-time event. + # 10-byte continuous format: 0x10 markers at byte 0 AND byte 2 + if data[0] == 0x10 and data[2] == 0x10: return "Waveform (Continuous)" - log.warning("_extract_record_type: unknown sub_code=0x%02X", code) - return f"Unknown(0x{code:02X})" + # 9-byte single-shot format: 0x10 sub_code marker at byte 1 + if data[1] == 0x10: + return "Waveform" + log.warning( + "_extract_record_type: unrecognized header: data[0:3]=%02X %02X %02X", + data[0], data[1], data[2], + ) + return f"Unknown({data[0]:02X}.{data[1]:02X}.{data[2]:02X})" def _extract_peak_floats(data: bytes) -> Optional[PeakValues]: From 45e61fbcaf14c3160d75ecddfdb225a953dd7518 Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Sun, 3 May 2026 01:20:21 -0400 Subject: [PATCH 03/11] big refactor of waveform protocol. --- CHANGELOG.md | 31 +++ CLAUDE.md | 2 +- minimateplus/blastware_file.py | 101 +++++----- minimateplus/client.py | 107 ++++++---- minimateplus/models.py | 52 +++++ minimateplus/protocol.py | 354 ++++++++++++++++++--------------- sfm/server.py | 40 ++-- 7 files changed, 409 insertions(+), 278 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a72537f..e98b2f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,34 @@ All notable changes to seismo-relay are documented here. --- +## v0.13.2 — 2026-05-01 + +### Fixed + +- **`_extract_record_type` — third 0C-record header format ("short", 8 bytes).** + A live SFM download against BE11529 produced files named `M5290000.000` + (zero-stamped) because the 0C waveform record's first bytes were + `01 05 07 ea ...` — neither the 9-byte single-shot layout (`0x10` at byte 1) + nor the 10-byte continuous layout (`0x10` at bytes 0 and 2). Investigation + showed this is a third format observed in the wild: an 8-byte header with no + marker bytes at all (`[day][month][year_BE:2][unknown][hour][min][sec]`). + The detection logic now scans the year (uint16 BE) at byte 2 / byte 3 / byte + 4 and picks whichever offset returns a sensible year (2015–2050) — each + format has the year at a unique position so this disambiguates cleanly. + - New format → `event.record_type = "Waveform (Short)"`, + `Timestamp.from_short_record()`. + - Existing single-shot and continuous parsers unchanged. + - The user's event from May 1, 2026 13:21:37 now correctly resolves to a + filename like `M529LKIQ.G10` instead of `M5290000.000`. + +### Added + +- `Timestamp.from_short_record(data)` — decodes the 8-byte header. +- `_detect_record_format(data)` — internal helper returning + `"single_shot" / "continuous" / "short" / None` via year-position scan. + +--- + ## v0.13.1 — 2026-05-01 ### Fixed @@ -23,6 +51,9 @@ All notable changes to seismo-relay are documented here. it's the 10-byte continuous header; else if byte 1 is `0x10`, it's the 9-byte single-shot header. Day-of-month no longer matters. + *Superseded by v0.13.2 — the user's actual record uses a third 8-byte format + with no `0x10` markers, which v0.13.1 still misclassified.* + --- ## v0.13.0 — 2026-05-01 diff --git a/CLAUDE.md b/CLAUDE.md index f908461..f06d849 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem -(Sierra Wireless RV50 / RV55). Current version: **v0.13.1**. +(Sierra Wireless RV50 / RV55). Current version: **v0.13.2**. When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document diff --git a/minimateplus/blastware_file.py b/minimateplus/blastware_file.py index fedf229..774f2b1 100644 --- a/minimateplus/blastware_file.py +++ b/minimateplus/blastware_file.py @@ -672,11 +672,13 @@ def write_blastware_file( # Do NOT use a5_frames[-1] — if _a5_frames contains stray frames from a # subsequent event (a known get_events side-effect), the last frame will # not be the terminator and the footer will be mis-identified. + # TERM detection (v0.14.0): + # Legacy walk: TERM has page_key == 0x0000. + # BW-exact walk: TERM has page_key != 0x0010 (e.g. 0x0001). + # The TERM is always the LAST frame in the list (include_terminator=True). term_idx: Optional[int] = None - for _i, _f in enumerate(a5_frames): - if _f.page_key == 0x0000: - term_idx = _i - break + if a5_frames and a5_frames[-1].page_key != 0x0010: + term_idx = len(a5_frames) - 1 if term_idx is not None: body_frames = a5_frames[:term_idx] @@ -685,64 +687,33 @@ def write_blastware_file( body_frames = a5_frames term_frame = None - # ── Identify first metadata frame and skip "extra chunks" ─────────────── - # When extra_chunks_after_metadata=1 in read_bulk_waveform_stream(), the - # frame list is: [probe, data..., metadata, extra_chunk, terminator]. - # The extra_chunk is downloaded to prime the TCP terminator response — its - # ADC data is NOT part of the Blastware file body. Skip it. + # Frame contribution loop (v0.14.0 BW-exact walk). + # Frame structure: + # Event 1: [probe] [meta@0x1002] [meta@0x1004] [samples ...] [TERM-not-in-body] + # Event N: [probe@start+0x46] [samples ...] [TERM-not-in-body] # - # Rule: any frame at index strictly between first_metadata_fi and last_fi - # (the final frame) is an extra chunk and must be excluded. - # - # If no metadata frame exists (e.g. full_waveform download), first_metadata_fi - # is None and no frames are skipped — all frames contribute normally. - first_metadata_fi: Optional[int] = None - for _fi_scan, _frame_scan in enumerate(body_frames): - if _fi_scan > 0 and any(m in bytes(_frame_scan.data) for m in _METADATA_FRAME_MARKERS): - first_metadata_fi = _fi_scan - break + # Skip values per frame (confirmed from byte-diff vs BW-saved file + # M529LKIQ.G10): + # probe (fi=0): probe_skip (depends on STRT position) + # meta@0x1002 (fi=1): 13 (6-byte inner header) + # meta@0x1004 (fi=2): 13 (6-byte inner header) + # sample chunks (fi=3+): 12 (5-byte inner header) last_fi = len(body_frames) - 1 - log.warning( - "write_blastware_file: %d body_frames first_metadata_fi=%s last_fi=%d", - len(body_frames), - str(first_metadata_fi) if first_metadata_fi is not None else "None", - last_fi, + log.debug( + "write_blastware_file: %d body_frames last_fi=%d", + len(body_frames), last_fi, ) all_bytes = bytearray() for fi, frame in enumerate(body_frames): - # Skip "extra chunk" frames: frames after the first metadata frame but - # before the last frame (terminator). These prime the TCP terminator but - # their ADC data must NOT appear in the Blastware file body. - if (first_metadata_fi is not None - and fi > first_metadata_fi - and fi < last_fi): - log.warning( - "write_blastware_file: fi=%d SKIP (extra chunk after metadata fi=%d last_fi=%d)", - fi, first_metadata_fi, last_fi, - ) - continue - if fi == 0: - # Probe frame: always process regardless of classification. - # It holds the STRT record; probe_skip positions us past it. skip = probe_skip + elif fi in (1, 2): + skip = 13 # metadata pages else: - # ALL subsequent frames are included unconditionally — no filtering on - # frame type. In the A5 stream, frame 0 is always the probe response; - # frames 1+ are always data (waveform chunks, compliance config, or - # compliance continuation). Classification is for logging only. - # - # DO NOT gate on classify_frame() here: - # - "probe_or_strt" at fi>0 is always a false positive — ADC binary - # data can coincidentally contain b"STRT\xff\xfe" (confirmed from - # live capture: frames 1 and 5 matched on event key=01110000). - # - "metadata" frames must be included (compliance config body). - # - The compliance block spans 2 frames; skipping either produces a - # truncated file that Blastware rejects. - skip = 13 if fi == 1 else 12 + skip = 12 # sample chunks contribution = _frame_body_bytes(frame, skip) log.warning("write_blastware_file: fi=%d skip=%d raw_data=%d contribution=%d", @@ -769,11 +740,33 @@ def write_blastware_file( bytes(all_bytes[-28:]).hex() if len(all_bytes) >= 28 else bytes(all_bytes).hex(), ) - if len(all_bytes) >= 26: + # Find the first valid 0e 08 footer marker (v0.14.0). The device's + # TERM response contains the real Blastware footer; older walks + # accidentally fetched data past the footer. Validate by checking the + # year field (uint16 BE at offset+4) is in 2015..2050. + footer_pos = -1 + pos = 0 + while True: + pos = bytes(all_bytes).find(b"\x0e\x08", pos) + if pos < 0 or pos + 26 > len(all_bytes): + break + yr = (all_bytes[pos + 4] << 8) | all_bytes[pos + 5] + if 2015 <= yr <= 2050: + footer_pos = pos + break + pos += 1 + if footer_pos >= 0: + body = bytes(all_bytes[:footer_pos]) + footer = bytes(all_bytes[footer_pos:footer_pos + 26]) + log.warning( + "write_blastware_file: real 0e 08 footer at all_bytes[%d]; " + "truncating %d post-footer bytes", + footer_pos, len(all_bytes) - footer_pos - 26, + ) + elif len(all_bytes) >= 26: body = bytes(all_bytes[:-26]) footer = bytes(all_bytes[-26:]) else: - # Fallback: no terminator or very short stream → build footer from event metadata body = bytes(all_bytes) start_dt = _ts_from_model(event.timestamp) stop_dt: Optional[datetime.datetime] = None @@ -784,7 +777,7 @@ def write_blastware_file( + _encode_ts_be(start_dt) + _encode_ts_be(stop_dt) + b"\x00\x01\x00\x02\x00\x00" - + b"\x00\x00" # CRC placeholder + + b"\x00\x00" ) # ── Write file ─────────────────────────────────────────────────────────── diff --git a/minimateplus/client.py b/minimateplus/client.py index 12664c0..7b1f9eb 100644 --- a/minimateplus/client.py +++ b/minimateplus/client.py @@ -1345,6 +1345,11 @@ def _decode_waveform_record_into(data: bytes, event: Event) -> None: event.timestamp = Timestamp.from_continuous_record(data) except Exception as exc: log.warning("continuous record timestamp decode failed: %s", exc) + elif event.record_type == "Waveform (Short)": + try: + event.timestamp = Timestamp.from_short_record(data) + except Exception as exc: + log.warning("short record timestamp decode failed: %s", exc) # ── Peak values (per-channel PPV + Peak Vector Sum) ─────────────────────── try: @@ -1636,51 +1641,73 @@ def _decode_a5_waveform( } +def _detect_record_format(data: bytes) -> Optional[str]: + """ + Detect which timestamp-header format a 210-byte 0C waveform record uses. + + THREE formats observed on BE11529 firmware S338.17: + + "single_shot" — 9-byte header: + [day] [0x10] [month] [year_BE:2] [unknown] [hour] [min] [sec] + sub_code=0x10 at byte [1]. Year at [3:5]. + + "continuous" — 10-byte header: + [0x10] [day] [0x10] [month] [year_BE:2] [unknown] [hour] [min] [sec] + marker 0x10 at byte [0] AND byte [2]. Year at [4:6]. + + "short" — 8-byte header (NEW 2026-05-01): + [day] [month] [year_BE:2] [unknown] [hour] [min] [sec] + No marker bytes. Year at [2:4]. + + Each format has the year (uint16 BE) at a UNIQUE byte position, so we can + disambiguate by scanning each candidate position and picking the one + where the year falls in a sane range (2015..2050). + + Returns "single_shot" / "continuous" / "short" or None if no format matches. + """ + if len(data) < 8: + return None + + def _sane_year(hi: int, lo: int) -> bool: + y = (hi << 8) | lo + return 2015 <= y <= 2050 + + # Order matters: prefer formats with stronger marker-byte evidence first. + if data[1] == 0x10 and len(data) >= 9 and _sane_year(data[3], data[4]): + return "single_shot" + if (data[0] == 0x10 and data[2] == 0x10 + and len(data) >= 10 and _sane_year(data[4], data[5])): + return "continuous" + if _sane_year(data[2], data[3]): + return "short" + return None + + def _extract_record_type(data: bytes) -> Optional[str]: """ - Detect the waveform record format by inspecting the first 3 bytes of the - 210-byte record returned by SUB 0C. + Return a human-readable name for the waveform record format detected + in the first bytes of a 210-byte 0C record. - Two formats exist (confirmed from BE11529 captures and CLAUDE.md docs): - - Single-shot mode — 9-byte header: - data[0] = day - data[1] = 0x10 ← sub_code marker - data[2] = month - data[3:5] = year (BE) - ... - - Continuous mode — 10-byte header: - data[0] = 0x10 ← marker A - data[1] = day ← variable (NOT 0x10) - data[2] = 0x10 ← marker B - data[3] = month - data[4:6] = year (BE) - ... - - Disambiguate by checking BOTH data[0] and data[2]: - - data[0]==0x10 AND data[2]==0x10 → Continuous (10-byte header) - - data[1]==0x10 → Single-shot (9-byte header) - - otherwise → Unknown - - Previous logic only checked data[1] and so mis-classified continuous-mode - records as "Unknown(0xXX)" wherever day != 0x10 — see filename - M5290000.000 regression report (2026-05-01 SFM log). + Maps to the format codes returned by _detect_record_format(): + "single_shot" → "Waveform" + "continuous" → "Waveform (Continuous)" + "short" → "Waveform (Short)" + None → "Unknown(XX.YY.ZZ)" """ - if len(data) < 3: - return None - # 10-byte continuous format: 0x10 markers at byte 0 AND byte 2 - if data[0] == 0x10 and data[2] == 0x10: - return "Waveform (Continuous)" - # 9-byte single-shot format: 0x10 sub_code marker at byte 1 - if data[1] == 0x10: + fmt = _detect_record_format(data) + if fmt == "single_shot": return "Waveform" - log.warning( - "_extract_record_type: unrecognized header: data[0:3]=%02X %02X %02X", - data[0], data[1], data[2], - ) - return f"Unknown({data[0]:02X}.{data[1]:02X}.{data[2]:02X})" - + if fmt == "continuous": + return "Waveform (Continuous)" + if fmt == "short": + return "Waveform (Short)" + if len(data) >= 3: + log.warning( + "_extract_record_type: unrecognized header: data[0:3]=%02X %02X %02X", + data[0], data[1], data[2], + ) + return f"Unknown({data[0]:02X}.{data[1]:02X}.{data[2]:02X})" + return None def _extract_peak_floats(data: bytes) -> Optional[PeakValues]: """ diff --git a/minimateplus/models.py b/minimateplus/models.py index 47d4028..91d6344 100644 --- a/minimateplus/models.py +++ b/minimateplus/models.py @@ -201,6 +201,58 @@ class Timestamp: second=second, ) + @classmethod + def from_short_record(cls, data: bytes) -> "Timestamp": + """ + Decode an 8-byte timestamp header from a 210-byte waveform record. + + Wire layout (✅ CONFIRMED 2026-05-01 against live SFM run on BE11529 in + Continuous mode, day-of-month = 1 May, raw: 01 05 07 ea 00 0d 15 25): + byte[0]: day (uint8) + byte[1]: month (uint8) + bytes[2-3]: year (big-endian uint16) + byte[4]: unknown (0x00 in observed sample) + byte[5]: hour (uint8) + byte[6]: minute (uint8) + byte[7]: second (uint8) + + This is a third format observed in the wild — distinct from the 9-byte + (single-shot, sub_code=0x10 at [1]) and 10-byte (continuous, 0x10 at + [0] AND [2]) layouts. No marker bytes; disambiguated by where the + year lands when scanned at byte 2/3/4. + + Args: + data: at least 8 bytes; only the first 8 are consumed. + + Returns: + Decoded Timestamp. + + Raises: + ValueError: if data is fewer than 8 bytes. + """ + if len(data) < 8: + raise ValueError( + f"Short record timestamp requires at least 8 bytes, got {len(data)}" + ) + day = data[0] + month = data[1] + year = struct.unpack_from(">H", data, 2)[0] + unknown_byte = data[4] + hour = data[5] + minute = data[6] + second = data[7] + return cls( + raw=bytes(data[:8]), + flag=0, + year=year, + unknown_byte=unknown_byte, + month=month, + day=day, + hour=hour, + minute=minute, + second=second, + ) + @property def clock_set(self) -> bool: """False when year == 1995 (factory default / battery-lost state).""" diff --git a/minimateplus/protocol.py b/minimateplus/protocol.py index 2abeb1a..48fbfbe 100644 --- a/minimateplus/protocol.py +++ b/minimateplus/protocol.py @@ -136,9 +136,10 @@ DATA_LENGTHS: dict[int, int] = { # existing blastware_file.py builder relies on the 0x0400-step frame structure # to produce valid files. Switching to BW's 0x0200 step is a separate task # that also requires updating the file builder. -_BULK_CHUNK_OFFSET = 0x1004 # offset_word for probe + all chunk requests -_BULK_TERM_OFFSET = 0x005A # offset_word for the legacy terminator -_BULK_COUNTER_STEP = 0x0400 # chunk counter increment +# BW-exact protocol values (v0.14.0). Verified against 4-27-26 + 5-1-26 captures. +_BULK_CHUNK_OFFSET = 0x1002 # offset_word for probe + all chunk requests +_BULK_TERM_OFFSET = 0x005A # offset_word for the legacy terminator (fallback only) +_BULK_COUNTER_STEP = 0x0200 # chunk counter increment (matches chunk payload size) # Default timeout values (seconds). # MiniMate Plus is a slow device — keep these generous. @@ -533,231 +534,260 @@ class MiniMateProtocol: self, key4: bytes, *, - stop_after_metadata: bool = True, - max_chunks: int = 32, + stop_after_metadata: bool = True, # DEPRECATED — no-op under BW-exact walk + max_chunks: int = 256, # safety cap only; loop is bounded by end_offset include_terminator: bool = False, - extra_chunks_after_metadata: int = 1, + extra_chunks_after_metadata: int = 1, # DEPRECATED — no-op ) -> list[S3Frame]: """ - Download the SUB 5A (BULK_WAVEFORM_STREAM) A5 frames for one event. + Download the SUB 5A (BULK_WAVEFORM_STREAM) A5 frames for one event using + Blastware's exact protocol. REWRITTEN 2026-05-02 (v0.14.0). - The bulk waveform stream carries both raw ADC samples (large) and - event-time metadata strings ("Project:", "Client:", "User Name:", - "Seis Loc:", "Extended Notes") embedded in one of the middle frames - (confirmed: A5[7] of 9 for 1-2-26 capture). + Algorithm (matches BW captures across 2-sec / 3-sec / event-2): - Protocol is request-per-chunk, NOT a continuous stream: - 1. Probe (offset=_BULK_CHUNK_OFFSET, is_probe=True, counter=0x0000) - 2. Chunks (offset=_BULK_CHUNK_OFFSET, is_probe=False, counter+=0x0400) - 3. Loop until metadata found (stop_after_metadata=True) or max_chunks - 4. Termination (offset=_BULK_TERM_OFFSET, counter=last+_BULK_COUNTER_STEP) - Device responds with a final A5 frame (page_key=0x0000). + 1. Probe + - For events at start_key[2:4] = 0x0000 (first event after erase + / wrap): probe at counter=0x0000 with full key in params. + - For continuation events (start_key[2:4] != 0): first chunk at + counter = start_key[2:4] + 0x0046; acts as both probe and + first sample chunk; response carries STRT. - By default the termination frame (page_key=0x0000) is NOT included in the - returned list. Pass include_terminator=True to append it; the blastware_file - writer needs the terminator frame's body to reconstruct the waveform file footer. + 2. Parse end_offset from STRT record at data[23:27] of the probe response. - Args: - key4: 4-byte waveform key from EVENT_HEADER (1E). - stop_after_metadata: If True (default), send termination as soon as - b"Project:" is found in a frame's data — avoids - downloading the full ADC waveform payload (several - hundred KB). Set False to download everything. - max_chunks: Safety cap on the number of chunk requests sent - (default 32; a typical event uses 9 large frames). - include_terminator: If True, append the terminator A5 frame - (page_key=0x0000) to the returned list. The - terminator carries the waveform file footer bytes. - Default False preserves existing caller behaviour. + 3. Read two fixed metadata pages at counter=0x1002 and counter=0x1004 + — global session metadata (Project / Client / User Name / Seis Loc + / Extended Notes ASCII strings). Event 1 only; continuation + events skip these (BW caches them across the session). + + 4. Walk sample chunks at 0x0200 increments, starting from 0x0600 for + event 1 or `start + 0x0046 + 0x0200` for continuation events. + Stop when `next_chunk + 0x0200 > end_offset`. + + 5. Send TERM frame with offset_word and params computed by + `bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)`. + The TERM response contains the partial last chunk (residual = + end_offset - next_boundary) including the 26-byte 0e 08 file + footer. Returns: - List of S3Frame objects from each A5 response frame. Frame indices - match the request sequence: index 0 = probe response, index 1 = first - chunk, etc. If include_terminator=True, the last element is the - terminator frame (page_key=0x0000). + List of S3Frame objects from each A5 response (probe, metadata + pages, sample chunks, optional TERM response). Caller passes + `include_terminator=True` (e.g. write_blastware_file) to keep the + TERM response in the list — it's required to reconstruct the + file footer. + + Deprecated kwargs: + stop_after_metadata: legacy "Project:"-string-based stop condition. + No-op under the BW-exact walk; the loop is + deterministically bounded by end_offset from + STRT. Accepted for backward compat. + extra_chunks_after_metadata: same. Raises: - ProtocolError: on timeout, bad checksum, or unexpected SUB. - - Confirmed from 1-2-26 BW TX/RX captures (2026-04-02): - - probe + 8 regular chunks + 1 termination = 10 TX frames - - 9 large A5 responses + 1 terminator A5 = 10 RX frames - - page_key=0x0010 on large frames; page_key=0x0000 on terminator ✅ - - "Project:" metadata at A5[7].data[626] ✅ + ProtocolError: on timeout / bad checksum / unexpected SUB. """ if len(key4) != 4: raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}") - rsp_sub = _expected_rsp_sub(SUB_BULK_WAVEFORM) # 0xFF - 0x5A = 0xA5 + # Quietly accept and warn on deprecated kwargs. + if not stop_after_metadata: + log.debug("5A: stop_after_metadata=False is no-op under BW-exact walk") + if extra_chunks_after_metadata not in (0, 1): + log.debug("5A: extra_chunks_after_metadata=%d is no-op under BW-exact walk", + extra_chunks_after_metadata) + + rsp_sub = _expected_rsp_sub(SUB_BULK_WAVEFORM) # 0xA5 frames_data: list[S3Frame] = [] - counter = 0 - # BW counter formula (confirmed from 4-3-26 capture for key 0111245a, - # and empirical live-device test 2026-04-06 for key 01110000): - # counter for chunk n = max(key4[2:4], 0x0400) + (n - 1) * 0x0400 - # key4[2:4] is the event's circular-buffer base offset. The max() guard - # ensures chunk 1 never uses counter=0x0000 (which equals the probe address - # and causes the device to re-return STRT record data for the first chunk). - _key4_offset = (key4[2] << 8) | key4[3] + start_offset = (key4[2] << 8) | key4[3] + is_event_1 = (start_offset == 0) - # ── Step 1: probe ──────────────────────────────────────────────────── - log.debug("5A probe key=%s key4_offset=0x%04X", key4.hex(), _key4_offset) - params = bulk_waveform_params(key4, 0, is_probe=True) - self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params)) - self._parser.reset() # reset bytes_fed counter before probe recv + # ── Step 1: probe / first chunk ────────────────────────────────────── + if is_event_1: + probe_counter = 0 + probe_params = bulk_waveform_params(key4, 0, is_probe=True) + log.debug("5A probe (event-1) key=%s counter=0x0000", key4.hex()) + else: + # Continuation events: first 5A request lands at start+0x0046, + # acting as both probe and first sample chunk. Confirmed from + # 5-1-26 "copy 2nd address event" capture. + probe_counter = start_offset + 0x0046 + probe_params = bulk_waveform_params(key4, probe_counter) + log.debug( + "5A probe (event-N) key=%s counter=0x%04X (start+0x46)", + key4.hex(), probe_counter, + ) + + self._send(build_5a_frame(_BULK_CHUNK_OFFSET, probe_params)) + self._parser.reset() try: rsp = self._recv_one(expected_sub=rsp_sub, reset_parser=False) except TimeoutError: log.warning( - "5A probe TIMED OUT for key=%s — " - "%d raw bytes received (no complete A5 frame assembled)", + "5A probe TIMED OUT for key=%s — %d raw bytes received", key4.hex(), self._parser.bytes_fed, ) raise - frames_data.append(rsp) - log.debug("5A A5[0] page_key=0x%04X %d bytes", rsp.page_key, len(rsp.data)) - # ── Parse STRT end_offset from probe response (NEW 2026-05-01) ──────── - # The first A5 response contains a STRT record at data[17:]. The - # bytes at data[23:27] are the event's end-key, whose low 16 bits - # are the absolute device-buffer address where the event ends. Use - # this to bound the chunk loop and stop the over-read past event end. - # See docs/instantel_protocol_reference.md §7.8.5 and CLAUDE.md - # "SUB 5A — STRT record encodes end_offset". - _end_offset = parse_strt_end_offset(rsp.data) - if _end_offset is None: - # Defensive fallback — let max_chunks cap the walk. - log.warning("5A: STRT not found in probe; cannot bound chunk loop") - _end_offset = 0xFFFF + frames_data.append(rsp) + log.debug("5A A5[0] (probe) page_key=0x%04X %d bytes", + rsp.page_key, len(rsp.data)) + + # ── Step 2: parse STRT end_offset from probe response ──────────────── + end_offset = parse_strt_end_offset(rsp.data) + if end_offset is None: + log.warning( + "5A probe response did not contain a STRT record; " + "cannot bound chunk loop — falling back to max_chunks=%d cap", + max_chunks, + ) + end_offset = 0xFFFF # impossible value → loop runs to max_chunks else: - log.debug( + log.info( "5A STRT start_offset=0x%04X end_offset=0x%04X size=0x%04X", - _key4_offset, _end_offset, _end_offset - _key4_offset, + start_offset, end_offset, end_offset - start_offset, ) - # ── Step 2: chunk loop ─────────────────────────────────────────────── - # Counter formula: _chunk_base + (chunk_num - 1) * 0x0400 - # where _chunk_base = max(key4[2:4], 0x0400). - # - # For events with key4[2:4] != 0 (e.g. key 0111245a, offset 0x245a): - # _chunk_base = 0x245a → chunk 1=0x245a, chunk 2=0x285a, ... - # Confirmed from 4-3-26 capture. - # - # For events with key4[2:4] == 0 (e.g. key 01110000): - # _chunk_base = max(0, 0x0400) = 0x0400 - # → chunk 1=0x0400, chunk 2=0x0800, ... (= old chunk_num*0x0400) - # CRITICAL: counter=0x0000 (same as the probe) causes the device to - # re-return the STRT record data for chunk 1, making frame 1 look like - # a second probe response (confirmed from server log: frame 1 len=1097, - # contains STRT\xff\xfe, contributes zero body bytes after DLE-strip). - # counter=0x0400 for chunk 1 confirmed working (empirical test 2026-04-06). - _chunk_base = max(_key4_offset, _BULK_COUNTER_STEP) - for chunk_num in range(1, max_chunks + 1): - counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP - # Stop when we'd step past the event end (NEW 2026-05-01). Without - # this, the device returns post-event circular-buffer data which - # corrupts the reconstructed file for events ≥ 2 sec. - if counter >= _end_offset: + # ── Step 3: metadata pages 0x1002 + 0x1004 (event 1 only) ──────────── + # Confirmed from BW captures: BW reads these two fixed device-buffer + # pages immediately after the probe for events at start_key[2:4]=0. + # Continuation events skip them (BW caches across the session). + # Their content is global compliance-setup metadata: Project, Client, + # User Name, Seis Loc, Extended Notes. + if is_event_1: + for meta_counter in (0x1002, 0x1004): + # Metadata page params have an extra trailing 0x00 byte + # (12-byte params instead of 11) — empirical from BW captures. + # Checksum-neutral but matches BW byte-for-byte. + meta_params = bytes([ + 0x00, + key4[0], key4[1], + (meta_counter >> 8) & 0xFF, + meta_counter & 0xFF, + 0, 0, 0, 0, 0, 0, 0, + ]) + log.debug("5A metadata page counter=0x%04X", meta_counter) + self._send(build_5a_frame(_BULK_CHUNK_OFFSET, meta_params)) + self._parser.reset() + try: + meta_rsp = self._recv_one( + expected_sub=rsp_sub, reset_parser=False, timeout=10.0, + ) + except TimeoutError: + log.warning( + "5A metadata page 0x%04X TIMED OUT — continuing", + meta_counter, + ) + continue + frames_data.append(meta_rsp) + log.debug( + "5A meta@0x%04X page_key=0x%04X %d bytes", + meta_counter, meta_rsp.page_key, len(meta_rsp.data), + ) + + # ── Step 4: sample chunk loop, bounded by end_offset ───────────────── + # Sample chunks start at: + # event 1: counter = 0x0600 + # event N (>0): counter = probe_counter + 0x0200 + # (probe was the first sample chunk) + if is_event_1: + counter = 0x0600 + else: + counter = probe_counter + _BULK_COUNTER_STEP + + last_chunk_counter: Optional[int] = ( + probe_counter if not is_event_1 else None + ) + chunks_fetched = 0 + + while chunks_fetched < max_chunks: + # Stop when next chunk would straddle the event end. + if counter + _BULK_COUNTER_STEP > end_offset: log.debug( "5A chunk loop done at counter=0x%04X (end=0x%04X); " "%d chunks fetched", - counter, _end_offset, len(frames_data), + counter, end_offset, chunks_fetched, ) break - params = bulk_waveform_params(key4, counter) - log.debug("5A chunk %d counter=0x%04X", chunk_num, counter) + + params = bulk_waveform_params(key4, counter) + log.debug("5A chunk #%d counter=0x%04X", chunks_fetched + 1, counter) self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params)) - self._parser.reset() # reset bytes_fed for accurate per-chunk count + self._parser.reset() try: - rsp = self._recv_one(expected_sub=rsp_sub, reset_parser=False, timeout=10.0) + rsp = self._recv_one( + expected_sub=rsp_sub, reset_parser=False, timeout=10.0, + ) except TimeoutError: raw = self._parser.bytes_fed log.warning( "5A TIMEOUT chunk=%d counter=0x%04X raw_bytes=%d", - chunk_num, counter, raw, + chunks_fetched + 1, counter, raw, ) if raw > 0 and frames_data: - # Device sent a partial byte (likely a bare DLE/ETX end-of-stream - # signal) but never completed a full frame. Treat as graceful - # stream end and fall through to the termination step. log.warning( - "5A end-of-stream detected at chunk=%d (raw_bytes=%d, " - "frames_collected=%d) — proceeding to termination", - chunk_num, raw, len(frames_data), + "5A unexpected end-of-stream — proceeding to TERM", ) break raise - log.warning( - "5A RX chunk=%d page_key=0x%04X data_len=%d contains_Project=%s", - chunk_num, rsp.page_key, len(rsp.data), b"Project:" in rsp.data, + log.debug( + "5A RX chunk=%d page_key=0x%04X data_len=%d", + chunks_fetched + 1, rsp.page_key, len(rsp.data), ) if rsp.page_key == 0x0000: - # Device unexpectedly terminated mid-stream (no termination needed). - log.debug("5A A5[%d] page_key=0x0000 — device terminated early", chunk_num) + # Device terminated mid-stream unexpectedly. + log.warning( + "5A unexpected page_key=0x0000 mid-stream at counter=0x%04X", + counter, + ) if include_terminator: frames_data.append(rsp) return frames_data frames_data.append(rsp) - - if stop_after_metadata and b"Project:" in rsp.data: - # Download exactly one more chunk after finding metadata — this is - # what Blastware does. The extra chunk contains the tail ADC data - # and primes the device to return a valid footer in the termination - # response. Without it, termination returns an empty ack with no - # footer bytes (confirmed 2026-04-23 from HxD comparison). - # Download extra_chunks_after_metadata more chunks past the - # metadata. The caller calculates this from record_time and - # sample_rate so we download exactly the right amount of ADC - # data — no more, no less — before terminating. - # The device returns the footer in the termination response only - # after the right amount of data has been consumed. - log.debug("5A A5[%d] metadata found — fetching %d more chunk(s)", - chunk_num, extra_chunks_after_metadata) - for _extra_n in range(extra_chunks_after_metadata): - chunk_num += 1 - counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP - params = bulk_waveform_params(key4, counter) - self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params)) - try: - extra = self._recv_one(expected_sub=rsp_sub, timeout=10.0) - log.debug("5A A5[%d] extra chunk page_key=0x%04X data_len=%d", - chunk_num, extra.page_key, len(extra.data)) - if extra.page_key == 0x0000: - if include_terminator: - frames_data.append(extra) - return frames_data - frames_data.append(extra) - except TimeoutError: - log.debug("5A extra chunk %d timed out — end of stream", _extra_n + 1) - break - break + last_chunk_counter = counter + counter += _BULK_COUNTER_STEP + chunks_fetched += 1 else: log.warning( - "5A reached max_chunks=%d without end-of-stream; sending termination", - max_chunks, + "5A reached max_chunks=%d at counter=0x%04X (end=0x%04X)", + max_chunks, counter, end_offset, ) - # ── Step 3: termination ────────────────────────────────────────────── - term_counter = counter + _BULK_COUNTER_STEP - term_params = bulk_waveform_term_params(key4, term_counter) - log.debug( - "5A termination term_counter=0x%04X offset=0x%04X", - term_counter, _BULK_TERM_OFFSET, - ) - self._send(build_5a_frame(_BULK_TERM_OFFSET, term_params)) - try: - term_rsp = self._recv_one(expected_sub=rsp_sub) + # ── Step 5: TERM with proper end_offset-derived formula ────────────── + if last_chunk_counter is None or end_offset == 0xFFFF: + # No STRT or no chunks fetched — fall back to legacy TERM. + log.warning( + "5A using legacy TERM (offset_word=0x005A); " + "end_offset unavailable or no chunks fetched", + ) + legacy_counter = (last_chunk_counter or probe_counter) + _BULK_COUNTER_STEP + term_offset_word = _BULK_TERM_OFFSET # 0x005A + term_params = bulk_waveform_term_params(key4, legacy_counter) + else: + term_offset_word, term_params = bulk_waveform_term_v2( + key4, end_offset, last_chunk_counter, + ) log.debug( - "5A termination response page_key=0x%04X %d bytes", + "5A TERM offset_word=0x%04X params[2:4]=%s end=0x%04X " + "last_chunk=0x%04X", + term_offset_word, term_params[2:4].hex(), + end_offset, last_chunk_counter, + ) + + self._send(build_5a_frame(term_offset_word, term_params)) + try: + term_rsp = self._recv_one(expected_sub=rsp_sub, timeout=10.0) + log.info( + "5A TERM response page_key=0x%04X %d bytes", term_rsp.page_key, len(term_rsp.data), ) if include_terminator: frames_data.append(term_rsp) except TimeoutError: - log.debug("5A no termination response — device may have already closed") + log.warning("5A no TERM response (timeout)") return frames_data diff --git a/sfm/server.py b/sfm/server.py index 3fc4bb2..9683254 100644 --- a/sfm/server.py +++ b/sfm/server.py @@ -37,6 +37,7 @@ from __future__ import annotations import datetime import logging import sys +import tempfile import threading import time from pathlib import Path @@ -863,8 +864,8 @@ def device_event_blastware_file( Supply either *port* (serial) or *host* (TCP/modem). - The file is written to /tmp and streamed back as a binary download. - Blastware can open it directly — filename encodes serial + timestamp. + The file is written to the OS temp directory and streamed back as a binary + download. Blastware can open it directly — filename encodes serial + timestamp. Filename format: 0 - prefix letter = chr(ord('B') + floor(serial_numeric / 1000)) @@ -885,26 +886,13 @@ def device_event_blastware_file( def _do(): with _build_client(port, baud, host, tcp_port, timeout=120.0) as client: info = client.connect() - # Use stop_after_metadata=True (full_waveform=False) with 1 extra - # chunk after "Project:". The extra chunk is required to prime the - # device over TCP: termination at term_counter=metadata_counter+0x0400 - # returns only ~90 bytes (no useful footer) over TCP/cellular, but - # termination at metadata_counter+0x0800 (one chunk later) returns - # the full 737-byte frame containing the footer. - # - # Confirmed from 4-26-26 BW RS-232 capture: BW terminates at 0x1800 - # without an extra chunk (works on RS-232 but not TCP). - # write_blastware_file() automatically skips the extra chunk's - # contribution — only the probe+ADC+metadata+terminator bytes appear - # in the output file. - # - # full_waveform=True (natural end-of-stream) downloads ALL chunks - # including post-event silence (35+ chunks for a 9-sec event at - # 1024 sps) — this produces 24KB+ files that Blastware rejects. + # Under v0.14.0 BW-exact 5A walk, the chunk loop is bounded by + # the event end_offset extracted from STRT. No more + # stop_after_metadata / extra_chunks gymnastics — these + # kwargs are now no-ops. events = client.get_events( full_waveform=False, stop_after_index=index, - extra_chunks_after_metadata=1, ) matching = [ev for ev in events if ev.index == index] return matching[0] if matching else None, info @@ -940,8 +928,18 @@ def device_event_blastware_file( # Build filename using the same algorithm Blastware uses filename = blastware_filename(ev, serial) - # Write to /tmp so FastAPI can stream it back - out_path = Path("/tmp") / filename + # Write to OS temp dir (cross-platform: /tmp on Linux/macOS, + # %TEMP% on Windows) so FastAPI can stream it back via FileResponse. + out_path = Path(tempfile.gettempdir()) / filename + # Delete any stale file at this path before writing. On Windows we have + # observed the new (smaller) file getting trailing zero-bytes from the + # previous (larger) file when filesystem semantics around open(...,"wb") + # don't truncate cleanly (e.g. through a synced folder). Explicit unlink + # eliminates that ambiguity. + try: + out_path.unlink() + except FileNotFoundError: + pass write_blastware_file(ev, a5_frames, out_path) log.info( "blastware_file: wrote %s (%d A5 frames, serial=%s)", From b66cc9d0753a37dd124de5e2b782780e44ccfa5c Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Mon, 4 May 2026 14:28:11 -0400 Subject: [PATCH 04/11] fix(blastware_file): update TERM detection logic and strip duplicate header blocks for accurate file writing --- minimateplus/blastware_file.py | 38 ++++++++++++++++++++-------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/minimateplus/blastware_file.py b/minimateplus/blastware_file.py index 774f2b1..f708435 100644 --- a/minimateplus/blastware_file.py +++ b/minimateplus/blastware_file.py @@ -672,10 +672,7 @@ def write_blastware_file( # Do NOT use a5_frames[-1] — if _a5_frames contains stray frames from a # subsequent event (a known get_events side-effect), the last frame will # not be the terminator and the footer will be mis-identified. - # TERM detection (v0.14.0): - # Legacy walk: TERM has page_key == 0x0000. - # BW-exact walk: TERM has page_key != 0x0010 (e.g. 0x0001). - # The TERM is always the LAST frame in the list (include_terminator=True). + # TERM detection (v0.14.0): last frame if page_key != 0x0010 (sample marker) term_idx: Optional[int] = None if a5_frames and a5_frames[-1].page_key != 0x0010: term_idx = len(a5_frames) - 1 @@ -688,13 +685,8 @@ def write_blastware_file( term_frame = None # Frame contribution loop (v0.14.0 BW-exact walk). - # Frame structure: - # Event 1: [probe] [meta@0x1002] [meta@0x1004] [samples ...] [TERM-not-in-body] - # Event N: [probe@start+0x46] [samples ...] [TERM-not-in-body] - # - # Skip values per frame (confirmed from byte-diff vs BW-saved file - # M529LKIQ.G10): - # probe (fi=0): probe_skip (depends on STRT position) + # Skip values: + # probe (fi=0): probe_skip # meta@0x1002 (fi=1): 13 (6-byte inner header) # meta@0x1004 (fi=2): 13 (6-byte inner header) # sample chunks (fi=3+): 12 (5-byte inner header) @@ -740,10 +732,26 @@ def write_blastware_file( bytes(all_bytes[-28:]).hex() if len(all_bytes) >= 28 else bytes(all_bytes).hex(), ) - # Find the first valid 0e 08 footer marker (v0.14.0). The device's - # TERM response contains the real Blastware footer; older walks - # accidentally fetched data past the footer. Validate by checking the - # year field (uint16 BE at offset+4) is in 2015..2050. + # Strip embedded "duplicate header+STRT" blocks from body (v0.14.1). + # Chunk@0x1000 sometimes lands on the device's metadata-mirror page, + # whose response includes a 25-byte "00 12 03 00 STRT ..." block that + # mirrors the file's own header + STRT record. BW treats embedded STRT + # markers as second-event starts and rejects the file. Replace these + # blocks with zeros to preserve file size + alignment. + needle = b"\x00\x12\x03\x00STRT" + pos = bytes(all_bytes).find(needle) + while pos >= 0: + end = pos + 25 + if end <= len(all_bytes): + all_bytes[pos:end] = b"\x00" * 25 + log.warning( + "write_blastware_file: stripped duplicate header+STRT at " + "all_bytes[%d:%d] (replaced with 25 zero-bytes)", + pos, end, + ) + pos = bytes(all_bytes).find(needle, end) + + # Find the first valid 0e 08 footer marker (v0.14.0). footer_pos = -1 pos = 0 while True: From 7b62c790a9890a94660f9f61575ceaaed8d16cdb Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Mon, 4 May 2026 14:30:46 -0400 Subject: [PATCH 05/11] fix(seismo-lab): remove duplicate capture history list --- seismo_lab.py | 25 ------------------------- 1 file changed, 25 deletions(-) diff --git a/seismo_lab.py b/seismo_lab.py index f78e6a1..1e84d0e 100644 --- a/seismo_lab.py +++ b/seismo_lab.py @@ -271,31 +271,6 @@ class BridgePanel(tk.Frame): self._hist_lb.pack(side=tk.LEFT, fill=tk.X, expand=True) self._hist_lb.bind("", self._on_hist_dblclick) - tk.Label(hist_outer, text="dbl-click to reload", bg=BG2, fg=FG_DIM, - font=MONO_SM, anchor="e").pack(side=tk.RIGHT, padx=6) - - # Capture history list - hist_outer = tk.Frame(self, bg=BG2) - hist_outer.pack(side=tk.TOP, fill=tk.X, padx=4, pady=(2, 0)) - - tk.Label(hist_outer, text="Captures:", bg=BG2, fg=FG_DIM, - font=MONO_SM, anchor="w").pack(side=tk.LEFT, padx=(4, 6)) - - hist_inner = tk.Frame(hist_outer, bg=BG2) - hist_inner.pack(side=tk.LEFT, fill=tk.X, expand=True) - - self._hist_lb = tk.Listbox( - hist_inner, bg=BG3, fg=FG, font=MONO_SM, - height=3, relief="flat", selectbackground=BG, - selectforeground=ACCENT, activestyle="none", - highlightthickness=0, - ) - hist_vsb = ttk.Scrollbar(hist_inner, orient="vertical", command=self._hist_lb.yview) - self._hist_lb.configure(yscrollcommand=hist_vsb.set) - hist_vsb.pack(side=tk.RIGHT, fill=tk.Y) - self._hist_lb.pack(side=tk.LEFT, fill=tk.X, expand=True) - self._hist_lb.bind("", self._on_hist_dblclick) - tk.Label(hist_outer, text="dbl-click to reload", bg=BG2, fg=FG_DIM, font=MONO_SM, anchor="e").pack(side=tk.RIGHT, padx=6) From 744473888344ae221e6185801e6eb35c0fa65015 Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Tue, 5 May 2026 16:46:35 -0400 Subject: [PATCH 06/11] debug(protocol): event-N probe is now at counter = start_offset instead of start_offset + 0x46 --- CHANGELOG.md | 23 +++++++++++++++++++++++ CLAUDE.md | 27 ++++++++++++++++++++++++--- docs/instantel_protocol_reference.md | 21 ++++++++++++++++++--- minimateplus/protocol.py | 20 +++++++++++++++----- 4 files changed, 80 insertions(+), 11 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 42236a6..e8e9177 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,29 @@ All notable changes to seismo-relay are documented here. --- +## v0.14.1 — 2026-05-04 + +### Fixed + +- **`read_bulk_waveform_stream` — event-N probe counter off-by-`0x46`.** + Continuation events (start_key[2:4] != 0) were being probed at counter + `start_offset + 0x0046` instead of just `start_offset`. In the iteration + walk, `cur_key` from 1F is already the off=0x46 WAVEHDR record key, so the + earlier formula effectively double-counted the WAVEHDR offset. The probe + landed one WAVEHDR past the actual event start, the response no longer + contained the STRT record at byte 17, `parse_strt_end_offset` returned + `None`, and the chunk loop fell back to the `max_chunks=128` cap — walking + ~110 chunks of post-event circular-buffer garbage. Verified against the + 5-1-26 "copy 2nd address" and 5-4-26 BW 2-sec event captures: BW probes + counter=`0x2238` with key=`01112238` and STRT is present at byte 17 of + the response (end_offset=`0x417E`). +- **CLAUDE.md / docs/instantel_protocol_reference.md** — corrected the + event-N section to clarify that `start_key` in those formulas is the + off=0x46 key, not the off=0x2C boundary key, and removed the spurious + `+0x46` from the chunk-walk pseudocode. + +--- + ## v0.12.6 — 2026-05-01 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index dbfa3c2..bd2a92a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem -(Sierra Wireless RV50 / RV55). Current version: **v0.13.2**. +(Sierra Wireless RV50 / RV55). Current version: **v0.14.1**. When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document @@ -160,13 +160,28 @@ firmware reserved area for the first slot in a freshly-erased buffer. Harmless #### Event 2+ case — start_key[2:4] != 0x0000 (continuation events) ``` -1. First chunk at counter = start_key[2:4] + 0x0046 (this IS the probe — response - contains STRT) +1. First chunk at counter = start_key[2:4] (this IS the probe — response + contains STRT at byte 17) 2. Sample chunks: counter += 0x0200 each, up to but not including end_offset 3. TERM frame ``` +**`start_key` here is the off=0x46 WAVEHDR record key returned by 1F** (e.g. `01112238`), +NOT the off=0x2C boundary key that immediately precedes it. An earlier draft of this +doc described event-N as "probe at start + 0x46" — that formula came from naming the +boundary key as `start_key`. In the iteration walk, `cur_key` passed to +`read_bulk_waveform_stream` is always the off=0x46 key (the partial-record skip path in +`get_events` re-runs 1F to advance past boundary records before invoking 5A), so the +probe counter is just `cur_key[2:4]` with no extra offset. **Adding +0x46 caused the +probe to overshoot, miss the STRT record at byte 17 of the response, fall back to the +`max_chunks=128` cap, and walk ~110 chunks of post-event garbage** — observed in +SFM 5-4-26 capture before the fix. + +Confirmed across: +- 5-1-26 "copy 2nd address" BW capture: probe counter=0x2238, key=01112238, STRT@17 end=0x417E. +- 5-4-26 BW 2-sec event capture: probe counter=0x2238, key=01112238, TERM offset_word=0x0146 → end=0x417E. + No metadata pages — those have already been read during event 1 in the same Blastware session, and BW caches them. Note that the metadata-page reads happen ONCE per Blastware-session-on-the-device, not once per event, so an SFM session that downloads @@ -180,6 +195,12 @@ several events should read 0x1002/0x1004 only once at the start. - 2026-04-26: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` (broken — over-read past event end). - 2026-05-01: Increments are 0x0200 not 0x0400; absolute addresses inside event range; bounded by STRT end_key, not by `max_chunks` cap or device-side timeout. +- 2026-05-04: Removed spurious `+0x0046` from event-N probe counter. `cur_key` from 1F + is already the off=0x46 WAVEHDR key, so adding +0x46 would have placed the probe one + WAVEHDR past the actual event start. This caused probe responses to lack a STRT + record (no `end_offset` parsed → `0xFFFF` fallback → `max_chunks=128` cap), walking + ~110 chunks of post-event circular-buffer garbage. Fixed in protocol.py + `read_bulk_waveform_stream`. ### SUB 5A — STRT record encodes end_offset (NEW 2026-05-01) diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index 0d90732..af8d1fc 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -1383,12 +1383,26 @@ the first slot in a freshly-erased buffer. Harmless to skip; BW does the same. **Event 2+ / start_key[2:4] != 0x0000** (continuation events in a populated buffer): ``` -1. First chunk at counter = start_key[2:4] + 0x0046 ← acts as both probe and first - sample chunk; response carries STRT +1. First chunk at counter = start_key[2:4] ← acts as both probe and first + sample chunk; response carries STRT at byte 17 2. Walk sample chunks counter += 0x0200 each 3. TERM ``` +**`start_key` here is the off=0x46 WAVEHDR record key returned by 1F** (e.g. `01112238`), +NOT the off=0x2C boundary key that immediately precedes it. An earlier draft of this +spec described event-N as "probe at start + 0x46" — that formula was correct only if +"start" meant the boundary key (0x21F2 in the 5-1-26 event 2 case). In the iteration +walk used by SFM and BW, `cur_key` passed into the 5A flow is always the off=0x46 key, +so the probe counter equals `cur_key[2:4]` with no extra offset. Adding +0x46 places +the probe one WAVEHDR past the actual event start, the response no longer contains +STRT at byte 17, and the chunk loop falls back to the `max_chunks` cap. + +Confirmed: +- 5-1-26 "copy 2nd address" BW capture: probe counter=0x2238 with key=01112238; A5[0] + has STRT@17 with end_offset=0x417E. +- 5-4-26 BW 2-sec event capture: same probe counter=0x2238, same end_offset=0x417E. + **No metadata-page reads.** Pages 0x1002/0x1004 are session-global and were already read during event 1 in the same Blastware session. In SFM, treat metadata pages as a once- per-`MiniMateClient.connect()` (or once-per-call-home) read, not per-event. @@ -1399,7 +1413,8 @@ per-`MiniMateClient.connect()` (or once-per-call-home) read, not per-event. |---|---|---|---|---|---| | 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B | 0x0600 (event-1 case) | | 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B | 0x0600 (event-1 case) | -| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | event 2 size = 0x1F8C = 8,076 B | 0x2238 (= 0x21F2 + 0x46) | +| 5-1-26 "copy 2nd address" / DA event 2 | `01112238` (= 1F result) | `0111417E` | `0x417E`, span 0x1F8C = 8,076 B | 0x2238 (= cur_key[2:4]) | +| 5-4-26 BW 2-sec event | `01112238` | `0111417E` | `0x417E` | 0x2238 (= cur_key[2:4]) | #### 7.8.6 TERM Frame Formula (NEW 2026-05-01) ✅ diff --git a/minimateplus/protocol.py b/minimateplus/protocol.py index 48fbfbe..3fb3b05 100644 --- a/minimateplus/protocol.py +++ b/minimateplus/protocol.py @@ -608,13 +608,23 @@ class MiniMateProtocol: probe_params = bulk_waveform_params(key4, 0, is_probe=True) log.debug("5A probe (event-1) key=%s counter=0x0000", key4.hex()) else: - # Continuation events: first 5A request lands at start+0x0046, - # acting as both probe and first sample chunk. Confirmed from - # 5-1-26 "copy 2nd address event" capture. - probe_counter = start_offset + 0x0046 + # Continuation events: first 5A request lands at counter = key[2:4] + # (i.e. the address of the off=0x46 WAVEHDR record returned by 1F). + # The probe response carries STRT at byte 17 with end_offset. + # + # Confirmed 2026-05-04 from 5-1-26 "copy 2nd address" capture + # (BW probes counter=0x2238 with key=01112238, STRT@17 end=0x417E) + # and 5-4-26 BW captures (2-sec event probes counter=0x2238). + # + # The earlier "+0x46" formula in the doc came from calling + # start_key the BOUNDARY (off=0x2C) key, but the iteration walk + # uses 1F's off=0x46 key as cur_key, which already incorporates + # the +0x46 offset relative to the boundary. Adding it again + # caused the probe to overshoot, miss STRT, and run uncapped. + probe_counter = start_offset probe_params = bulk_waveform_params(key4, probe_counter) log.debug( - "5A probe (event-N) key=%s counter=0x%04X (start+0x46)", + "5A probe (event-N) key=%s counter=0x%04X", key4.hex(), probe_counter, ) From eefec0bd6463d3304b58adc65b4fd838dea652a2 Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Tue, 5 May 2026 17:48:40 -0400 Subject: [PATCH 07/11] fix(blastware_file): remove harmful "duplicate header+STRT" strip logic to preserve valid waveform data --- CHANGELOG.md | 22 ++++++++++++++++++++++ CLAUDE.md | 2 +- minimateplus/blastware_file.py | 30 ++++++++++++------------------ 3 files changed, 35 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e8e9177..01c0dde 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,28 @@ All notable changes to seismo-relay are documented here. --- +## v0.14.2 — 2026-05-04 + +### Fixed + +- **`blastware_file.py` — removed harmful "duplicate header+STRT" strip.** + The v0.13.x strip logic was matching the byte sequence `00 12 03 00 STRT` + in legitimate waveform data — sample chunks at counter `0x1000` and + beyond often contain those bytes coincidentally — and zeroing 25 bytes + of valid samples per match. This is why event 0 (event-1 case in the + protocol) downloads of >1-sec recordings always failed in BW: the strip + destroyed real data at body offset `0x1012..0x102B` and propagated + alignment differences through the rest of the body. Sub-1-sec events + worked because their `end_offset` was below `0x1002`, so no sample + chunks landed in the metadata-page region and the strip's needle never + matched. Verified fix by re-feeding the BW 5-1-26 "copy 3sec" capture's + A5 frames into the file builder: output is now byte-identical to BW's + saved `M529LKIQ.G10` reference (8708 bytes, 0 differences). +- BW already concatenates frame contributions in stream order without + any de-duplication; SFM now does the same. + +--- + ## v0.14.1 — 2026-05-04 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index bd2a92a..6894e6a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem -(Sierra Wireless RV50 / RV55). Current version: **v0.14.1**. +(Sierra Wireless RV50 / RV55). Current version: **v0.14.2**. When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document diff --git a/minimateplus/blastware_file.py b/minimateplus/blastware_file.py index f708435..ee73390 100644 --- a/minimateplus/blastware_file.py +++ b/minimateplus/blastware_file.py @@ -732,24 +732,18 @@ def write_blastware_file( bytes(all_bytes[-28:]).hex() if len(all_bytes) >= 28 else bytes(all_bytes).hex(), ) - # Strip embedded "duplicate header+STRT" blocks from body (v0.14.1). - # Chunk@0x1000 sometimes lands on the device's metadata-mirror page, - # whose response includes a 25-byte "00 12 03 00 STRT ..." block that - # mirrors the file's own header + STRT record. BW treats embedded STRT - # markers as second-event starts and rejects the file. Replace these - # blocks with zeros to preserve file size + alignment. - needle = b"\x00\x12\x03\x00STRT" - pos = bytes(all_bytes).find(needle) - while pos >= 0: - end = pos + 25 - if end <= len(all_bytes): - all_bytes[pos:end] = b"\x00" * 25 - log.warning( - "write_blastware_file: stripped duplicate header+STRT at " - "all_bytes[%d:%d] (replaced with 25 zero-bytes)", - pos, end, - ) - pos = bytes(all_bytes).find(needle, end) + # NOTE: The "duplicate header+STRT strip" logic from v0.13.x has been + # REMOVED in v0.14.2. Under the v0.14.0 BW-exact 5A walk, body assembly + # is just contiguous concatenation of frame contributions in stream order + # (probe → meta@0x1002 → meta@0x1004 → samples → TERM), exactly as BW + # writes its files. The previous strip was matching the `00 12 03 00 STRT` + # byte sequence in legitimate waveform data — sample chunks at counter + # 0x1000 and beyond often contain those bytes coincidentally — and + # zeroing 25 bytes of valid samples per match. Compared to a known-good + # BW reference for the same 3-sec event 0, the strip introduced 26 bytes + # of zeros that BW did not have, then propagated alignment differences + # through the rest of the body. See decode_test/5-1-26/bw vs SFM diff + # at file[0x1012..0x102B] (2026-05-04 analysis). # Find the first valid 0e 08 footer marker (v0.14.0). footer_pos = -1 From a27693242db41d11ba7dd5303455b068453d4a75 Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Tue, 5 May 2026 18:28:28 -0400 Subject: [PATCH 08/11] fix(protocol): implement partial DLE stuffing for 0x10 bytes in params to prevent request corruption --- CHANGELOG.md | 49 +++++++++++++++++++++++++++++++++++++++++ CLAUDE.md | 29 +++++++++++++++++++++--- minimateplus/framing.py | 34 +++++++++++++++++++++++++++- 3 files changed, 108 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 01c0dde..29e50d1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,55 @@ All notable changes to seismo-relay are documented here. --- +## v0.14.3 — 2026-05-05 + +### Fixed + +- **`build_5a_frame` — DLE-stuffing rule for 0x10 bytes in params (the + long-standing >1-sec event 0 "won't open in BW" bug).** + + Previously `build_5a_frame` wrote params bytes RAW with no DLE stuffing, + based on the incorrect assumption that the device handled all `0x10` + bytes in params literally. It does not. The device's actual de-stuffing + rule for the params region is: + + - `10 10` → de-stuffs to `10` + - `10 02/03/04` → kept literal (inner-frame markers) + - `10 X` for other X → de-stuffs to just `X` (drops the `0x10`) + + When the counter passed in params has `0x10` in the high byte (e.g. + counter=`0x1000` produces params bytes `... 10 00 ...`), the device + silently corrupts the request to counter=`0x__00` and responds with + whatever lives at that wrong address. For counter=0x1000 the wrong + address was 0x0000, so the response was a copy of the file header + + STRT record. That STRT block then got embedded in the assembled body + at file offset `0x1016`, and Blastware refused to open the file + (interprets the second STRT as a malformed multi-event file). + + This explains the entire >1-sec event-0 failure pattern: + + - 1-sec events have `end_offset < 0x1000`, so the chunk walk never + requests counter `0x10__` and the bug never triggers. + - 2-sec / 3-sec / longer events all need a chunk at counter `0x1000` + (and longer events also need `0x1200`, `0x1400`, etc., none of which + have `0x10` in the high byte except `0x1000`). Just one corrupted + response is enough to embed STRT in the body and break the file. + + Verified against BW 5-1-26 "copy 3sec" capture: all 17 5A request + frames (probe + 2 metadata pages + 13 sample chunks + TERM) now match + BW's wire output **byte-for-byte**, including the doubled `10 10 00` + for counter=0x1000. + +### Notes + +- `0x10` bytes in `offset_hi` (the standalone offset field at body[5]) + are still written RAW — confirmed correct per the 1-2-26 capture. +- BW's actual encoding of `10 02` / `10 04` for meta pages 0x1002 / + 0x1004 is *not* doubled — it relies on the device keeping `10 02` + and `10 04` as literal pairs. This is preserved by the fix. + +--- + ## v0.14.2 — 2026-05-04 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md index 6894e6a..0675bd7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -2,7 +2,7 @@ Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem -(Sierra Wireless RV50 / RV55). Current version: **v0.14.2**. +(Sierra Wireless RV50 / RV55). Current version: **v0.14.3**. When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document @@ -115,8 +115,31 @@ S3→BW (response): section contribute only `XX` to the running sum; lone bytes contribute normally. This differs from the standard SUM8-of-destuffed-payload that all other commands use. -Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26 -BW TX capture. All 10 frames verified. +3. **Params region uses partial DLE stuffing (CONFIRMED 2026-05-05).** The device's + de-stuffing rule for bytes inside the params region is: + + - `10 10` → de-stuffs to `10` + - `10 02 / 03 / 04` → kept literal (these are inner-frame markers) + - `10 X` for other X → de-stuffs to just `X` (drops the leading `0x10`) + + Therefore any `0x10` byte in the *logical* params that is followed by a byte NOT in + `{0x02, 0x03, 0x04, 0x10}` MUST be doubled on the wire (`10 X` → `10 10 X`) so the + device's de-stuffer reproduces the original `10 X` pair. This applies most commonly + to counters with `0x10` in the high byte (e.g. counter=`0x1000` produces logical + params bytes `... 10 00 ...`, which BW encodes on the wire as `... 10 10 00 ...`). + Without this stuffing the device interprets counter=`0x1000` as `0x0000` and returns + the probe response (which contains a copy of the file header + STRT record). That + STRT block then gets embedded in the assembled file body at offset `0x1016`, and + Blastware refuses to open the file — see the v0.14.3 entry in `CHANGELOG.md`. + + `0x10` bytes in `offset_hi` (body[5]) are still written RAW — only the params region + has this stuffing requirement. The metadata-page params for counter `0x1002` / + `0x1004` survive without stuffing because `10 02` and `10 04` fall in the "kept + literal" carve-out. + +Both differences (1) and (2) confirmed by reproducing Blastware's exact wire bytes from +the 1-2-26 BW TX capture (10 frames). Difference (3) confirmed against the 5-1-26 +"bwcap3sec" capture (17 frames, all match byte-for-byte after fix). ### SUB 5A — chunk counter formula (REWRITTEN 2026-05-01 — see 5-1-26 captures) diff --git a/minimateplus/framing.py b/minimateplus/framing.py index 2011cc9..e26e0f0 100644 --- a/minimateplus/framing.py +++ b/minimateplus/framing.py @@ -137,8 +137,40 @@ def build_5a_frame(offset_word: int, raw_params: bytes) -> bytes: s += b"\x00" # field3 s += bytes([(offset_word >> 8) & 0xFF, # offset_hi — raw, NOT stuffed offset_word & 0xFF]) # offset_lo - for b in raw_params: # params — NOT DLE-stuffed (raw bytes, match BW wire format) + # Params — partial DLE stuffing of 0x10 bytes (CONFIRMED 2026-05-05). + # + # The device's de-stuffing rule for params is: + # • `10 10` → de-stuffs to `10` + # • `10 02/03/04` → kept literal (these are inner-frame markers) + # • `10 X` other → de-stuffs to just `X` (drops the 0x10) + # + # So for any 0x10 byte in the *logical* params that is followed by a + # byte NOT in {0x02, 0x03, 0x04, 0x10}, we must double the 0x10 on the + # wire (`10 X` → `10 10 X`) so the device's de-stuffer reproduces the + # original `10 X` pair. Without this, counter values with `0x10` in + # the high byte (e.g. counter=0x1000 has params bytes `10 00`) are + # silently corrupted to `0x__00` on the device side, and the device + # responds for the wrong address — for counter=0x1000 it returns the + # probe response (counter=0x0000), which contains the file header + + # STRT. That STRT block then lands in the assembled file body and + # Blastware rejects the file as malformed. + # + # Confirmed against BW capture 5-1-26 / bwcap3sec frame 20: params + # logical bytes `00 01 11 10 00 00 00 00 00 00 00` (counter=0x1000) + # are encoded on the wire as `00 01 11 10 10 00 00 00 00 00 00 00`. + # BW frames 13/14 (meta @ 0x1002 / 0x1004) leave `10 02` and `10 04` + # raw — the device handles those literal pairs correctly. + i = 0 + while i < len(raw_params): + b = raw_params[i] s.append(b) + if ( + b == 0x10 + and i + 1 < len(raw_params) + and raw_params[i + 1] not in (0x02, 0x03, 0x04, 0x10) + ): + s.append(0x10) # double the 0x10 so it survives device de-stuffing + i += 1 # DLE-aware checksum: for 0x10 XX pairs count XX; for lone bytes count them chk, i = 0, 0 From c914a15e12a942b34d6c2ff0326dc110dc06e9af Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Tue, 5 May 2026 20:37:52 -0400 Subject: [PATCH 09/11] docs: update for v0.14.3 - Full continuous waveform download successful! --- CLAUDE.md | 139 +++++++++++++++------------ docs/instantel_protocol_reference.md | 133 +++++++++++++++++++------ 2 files changed, 181 insertions(+), 91 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 0675bd7..627c0cb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,7 +27,7 @@ CHANGELOG.md ← version history --- -## Current implementation state (v0.12.3) +## Current implementation state (v0.14.3) Full read pipeline + write pipeline + erase pipeline + monitor log + call home config working end-to-end over TCP/cellular: @@ -41,14 +41,15 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c | Event header / first key | 1E | ✅ | | Waveform header | 0A | ✅ | | Waveform record (peaks, timestamp, project) | 0C | ✅ | -| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ over-read bug fixed v0.13.0 (chunk loop bounded by STRT end_offset); minor wire diffs vs BW deferred — see "SUB 5A — chunk counter formula" | +| **Bulk waveform stream (event-time metadata + full waveform)** | **5A** | ✅ **byte-perfect against BW captures (v0.14.3, 2026-05-05)** — STRT-bounded chunk walk + correct event-N probe counter + DLE-stuffed `0x10` bytes in params + concatenate-only file body assembly. All 17 5A request frames in the 5-1-26 3-sec capture reproduce byte-for-byte. | | Event advance / next key | 1F | ✅ | | **Write commands (push config to device)** | **68–83** | ✅ new v0.8.0 | | **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 | | **Monitor log entries (partial 0x2C records)** | **0A browse** | ✅ new v0.10.0 | | **Auto Call Home config (read + write)** | **2C → 7E → 7F** | ✅ **new v0.12.3** | -`get_events()` sequence per event: `1E → 0A → 0C → 5A → 1F` +`get_events()` sequence per event: `1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)` +(see "Correct iteration pattern" section below for full detail) `push_config_raw()` write sequence: `68→73 | 71×3→72 | 82→83 | 69→74→72` @@ -298,9 +299,8 @@ Two chunk addresses are GLOBAL device/session metadata, not event-specific: These are at fixed absolute addresses in the device's flash buffer. They contain the session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII -strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the -new walk these strings come from the dedicated metadata pages, not from the sample-chunk -stream. +strings). Under the v0.14.0+ walk these strings are read directly from the metadata +pages, not from the sample-chunk stream. BW reads them ONCE per Blastware session (during event 1's download) and caches them. For SFM, that means: @@ -309,9 +309,10 @@ For SFM, that means: - Their content does not change when iterating events; only when the user opens Compliance Setup → Apply on the device or sends a SUB 71 compliance write. -The contents have not been byte-for-byte decoded yet — first task on the implementation -side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the -strings we currently extract from A5[7]. +The full byte-for-byte layout of the metadata pages has not been mapped — `_decode_a5_metadata_into` +locates the ASCII strings via label scans (`Project:`, `Client:`, `User Name:`, `Seis Loc:`, +`Extended Notes`) which works correctly across observed captures. Future work could +dump the structural layout if more session-global fields need to be extracted. ### SUB 5A — params are 11 bytes for chunk frames, 10 for termination @@ -319,16 +320,11 @@ strings we currently extract from A5[7]. confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes. Do not swap them. -### SUB 5A — event-time metadata source (UPDATED 2026-05-01) +### SUB 5A — event-time metadata source (FINALIZED 2026-05-05) -> **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A -> bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at -> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because -> the broken counter formula was scanning the wrong region. -> -> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and -> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A -> sample-chunk stream itself does NOT contain these strings any more under the new walk. +The metadata strings come from the two fixed metadata pages at counter `0x1002` and +`0x1004` (see "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above). These pages +are GLOBAL session metadata — read once per Blastware/SFM session, not per event. ``` "Project:" → project description @@ -338,55 +334,71 @@ Do not swap them. "Extended Notes"→ notes ``` -**IMPORTANT — 5A "Project:" is session-start config, NOT per-event (confirmed 2026-04-05):** -The "Project:" string in the A5 frame 7 payload reflects the compliance setup from when -the *monitoring session first started*, not the individual event's project name. The per- -event project name is correctly stored in the 210-byte 0C waveform record and must be -used as the authoritative source. `_decode_a5_metadata_into` therefore only sets -`project` from 5A when 0C didn't already supply one. +**IMPORTANT — these strings are session-start config, NOT per-event:** +Project / Client / User Name / Seis Loc reflect the compliance setup from when the +*monitoring session first started*, not the individual event's per-event metadata. The +authoritative per-event project name is stored in the 210-byte 0C waveform record. +`_decode_a5_metadata_into` therefore only sets `project` from the 5A metadata pages +when 0C didn't already supply one. "Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C -record — 5A remains the sole source for those fields and they are set unconditionally. +record — the metadata pages are the sole source for those fields and they are set +unconditionally. -> ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and -> stops one chunk later) is a workaround for the missing end_offset bound — when the new -> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is -> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM -> frame. +#### Deprecated knobs (do not re-introduce) -### SUB 5A — end-of-stream — UPDATED 2026-05-01 +The `read_bulk_waveform_stream()` function still accepts these legacy kwargs for +backward compatibility, but they are **no-ops** under the v0.14.0+ walk: -> **Previous understanding (now known to be a symptom, not a feature):** "After streaming -> all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was -> not the device's natural end-of-event signal — it was the device's response when SFM had -> walked clean off the end of the addressable buffer region after over-reading by ~5×. -> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the -> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`, -> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser. +- `stop_after_metadata=True` — used to scan the chunk stream for `b"Project:"` and stop + one chunk later as a workaround for the missing end_offset bound. Obsolete: the loop + is now deterministically bounded by `end_offset` parsed from the STRT record at + data[17] of the probe response, with the partial tail fetched by the TERM frame. +- `extra_chunks_after_metadata` — same era, same reason. No-op. -The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful -defence-in-depth fallback for malformed events or unexpected device states, but should not -be the primary loop-exit condition. +If you find code or docs referencing "A5 frame 7" as the source of metadata strings, +that's an old-walk artifact (the broken `0x0400`-step formula occasionally caught the +0x1002 metadata page at sample-chunk fi=7). Update to reference the dedicated metadata +pages instead. -**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each. -Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call -in the chunk loop passes `timeout=10.0` explicitly. +### SUB 5A — end-of-stream (FINALIZED 2026-05-01) -**Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):** -A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames. -A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames. -An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames. +Under the v0.14.0+ STRT-bounded walk the stream ends cleanly: -Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks -17-37 containing post-event circular-buffer garbage that corrupted the file body. +``` +… last full chunk at counter < end_offset +TERM request (offset_word = end_offset - next_boundary, + params address (next_boundary)) +TERM response (page_key = 0x0000 or 0x0001, data = the residual + end_offset - next_boundary bytes including the file footer) +``` + +No timeout-based detection, no "1-byte teaser," no `max_chunks` cap. The chunk loop +exits when `counter + 0x0200 > end_offset`; the TERM frame fetches the tail. + +**Chunk recv timeout is 10 s, not the default 120 s.** Chunks arrive within ~1 s each. +Using 120 s would cause a ~2-minute stall on any unexpected timeout. The `_recv_one` +call in the chunk loop passes `timeout=10.0` explicitly. + +**Typical chunk count under the v0.14.0+ walk (BE11529, 1024 sps over TCP/cellular):** + +| Event duration | Sample chunks | Metadata pages | TERM | Total A5 frames | +|---|---|---|---|---| +| 2-sec (event 1) | ~12 | 2 | 1 | ~15 | +| 3-sec (event 1) | 13 | 2 | 1 | 16 | +| 2-sec (continuation) | 15 | 0 | 1 | 16 | +| 3-sec (continuation) | ~14 | 0 | 1 | ~15 | + +For comparison, the deprecated `0x0400`-step walk produced ~37 chunks for a 2-sec +event with chunks 17-37 containing post-event circular-buffer garbage. Do not +re-introduce that walk under any circumstances. ### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06) `_decode_a5_waveform()` previously had `elif fi == 9: continue` — a leftover from the -9-frame original blast capture where frame 9 was assumed to be a terminator. For current -35-frame streams, fi==9 is live waveform data (~133 sample-sets were being dropped). -Removed. Terminator detection is via `page_key == 0x0000` in `read_bulk_waveform_stream`, -not frame index. +9-frame original blast capture where frame 9 was assumed to be a terminator. Removed. +TERM detection in the file builder uses `frame.page_key != 0x0010` (sample marker), +not frame index — see `blastware_file.py`. ### SUB 1E / 1F — event iteration null sentinel and token position (FIXED, do not re-introduce) @@ -1029,7 +1041,7 @@ offsets in the raw 1A/E5 payload. Only fields with `✅` have confirmed offsets **Notes tab:** - Enable User Notes (bool) -- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from A5 frame 7 via 5A) +- Project, Client, User Name, Seis Loc (ASCII strings) ✅ (sourced from 5A metadata pages at counter 0x1002 / 0x1004 — see "SUB 5A — fixed metadata pages" section) - Enable Extended Notes (bool); Extended Notes text; Extended Notes Title - Enable Job Number (bool); Job Number (int) - Enable Scaled Distance (bool); Distance from Blast (float); Charge Weight (float) — Scaled Distance is derived @@ -1343,7 +1355,7 @@ body) because writing a dial string may require DLE escaping for embedded contro - **Database** — SQLite store for events + monitor log entries; dedup by key; queryable - **Histograms** — decode histogram-mode A5 data (noise floor tracking) -- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed working for Continuous mode events (2026-04-23):** SFM-generated file opens in Blastware, shows correct PPV/waveform/timestamp. File is ~200 bytes shorter than BW (missing last ADC tail slice) — all measurements correct. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that create spurious STRT markers in the body). Extension mapping: **CONFIRMED FALSE 2026-04-21** — extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<4-char-base36-stem>` +- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed BYTE-PERFECT against BW reference (v0.14.3, 2026-05-05):** when fed the BW 5-1-26 3-sec capture's A5 frames, the SFM-built file matches BW's saved `M529LKIQ.G10` byte-for-byte (8708 bytes, 0 differences). Live SFM downloads of event 0 (3-sec) and event 1 (3-sec continuation) both open cleanly in Blastware with full Event Reports, frequency analysis, and waveform plots. Body assembly is just contiguous concatenation of frame contributions in stream order (probe → meta@0x1002 → meta@0x1004 → samples → TERM); no stripping, no overlay, no special handling. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that may need different handling — untested under v0.14.x). Extension mapping: extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<4-char-base36-stem>` **Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units). @@ -1379,16 +1391,21 @@ body) because writing a dial string may require DLE escaping for embedded contro | Folder / File | Contents | |---|---| +| `1-2-26/` | First SUB 5A BW TX capture — established 5A frame format (raw offset_hi, DLE-aware checksum). 10 frames verified. | | `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102–112) | +| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) — 1E/0A/0C/1F sequence confirmed (single event so token=0xFE appeared to work in either branch) | +| `4-2-26/` | Download-mode BW TX capture — POLL×3 requirement confirmed (frames 68-73 between 1F and first 5A) | +| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events — all-zero params for 1F, null sentinel layout, 0A context requirement | +| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check | +| `4-11-26 (mitm/ach_mitm_20260411_001912/)` | Full ACH call-home MITM — erase protocol (0xA3/0x06/0xA2), monitor log partial records confirmed | | `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous | | `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec | | `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) | | `4-20-26/call home settings/` | Call home config read/write captures | -| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check | -| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events (1E/0A/1F iteration confirmed) | -| `4-2-26/` | Download-mode BW TX capture (5A bulk stream, POLL×3 requirement confirmed) | -| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) | -| `mitm/ach_mitm_20260411_001912/` | Full ACH call-home MITM (erase protocol, 0xA3/0x06/0xA2 confirmed) | +| `4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof of 5× SFM over-read. STRT end_key field located. | +| **`5-1-26/comcheck/`** | **Triplet of captures that nailed the v0.14.0 walk:** SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945` / `_171216`). Confirmed: TERM frame formula across 3 events, metadata pages 0x1002/0x1004 are global session metadata, event-1 vs event-N chunk pattern split, WAVEHDR off=0x46 vs 0x2C disambiguates real events from boundaries. | +| **`5-1-26/comcheck/bwcap3sec/`** | **The byte-perfect reference for v0.14.3.** All 17 BW 5A request frames (probe, 2 metadata, 13 samples, TERM) reproduce byte-for-byte from SFM's framing helpers — including the `10 10 00` DLE-stuffed counter for sample @ 0x1000 that was the long-standing failure mode. | +| `5-4-26/` | BW MITM captures of "copy 3sec / 2sec / Download All" + paired SFM session (`seismo_dl_20260504_145701`) showing the +0x46 event-N probe bug producing 110-chunk runaway walk. Cross-references against 5-1-26 confirmed device behavior is identical. | To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes diff --git a/docs/instantel_protocol_reference.md b/docs/instantel_protocol_reference.md index af8d1fc..ad2d841 100644 --- a/docs/instantel_protocol_reference.md +++ b/docs/instantel_protocol_reference.md @@ -111,6 +111,9 @@ | 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. | | 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. | | 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. | +| 2026-05-04 | §7.8.5, §7.8.8 | **CORRECTED — Event-N probe counter is just `start_offset`, NOT `start_offset + 0x0046`.** The `+0x46` formula in the original §7.8.5 was based on calling the off=0x2C boundary key the "start_key", but in the iteration walk `cur_key` passed into `read_bulk_waveform_stream` is always the off=0x46 WAVEHDR record key from 1F (the partial-record skip path in `get_events` re-runs 1F to advance past 0x2C boundary records). Adding +0x46 placed the probe one WAVEHDR past the actual event start; the response no longer contained STRT at byte 17, `parse_strt_end_offset` returned None, and the chunk loop fell back to the `max_chunks=128` cap, walking ~110 chunks of post-event circular-buffer garbage. Confirmed against both the 5-1-26 "copy 2nd address" capture (probe at counter=0x2238 with key=01112238) and the 5-4-26 BW 2-sec event capture. Fixed in protocol.py `read_bulk_waveform_stream` v0.14.1. | +| 2026-05-05 | §7.8.1 (rule #3 added) | **CONFIRMED — Partial DLE stuffing of `0x10` bytes in 5A params region.** The device's de-stuffing rule for the SUB 5A params region is: `10 10` → `10`, `10 02/03/04` → kept literal (inner-frame markers), `10 X` for any other X → de-stuffs to just `X` (drops the `0x10`). Therefore any `0x10` byte in the logical params followed by a byte NOT in {0x02, 0x03, 0x04, 0x10} MUST be doubled on the wire. This affects counters with `0x10` in the high byte — most importantly counter=`0x1000`, where logical params bytes `... 10 00 ...` were being sent raw and the device de-stuffed `10 00` to just `00`, returning the response for counter=0x0000 (= the file header + STRT). That STRT block then ended up embedded in the assembled file body at file offset `0x1016` and Blastware refused to open the file. This was the root cause of the long-standing ">1-sec event 0 won't open in BW" pattern (1-sec events worked because their `end_offset < 0x1000`, so no chunk request ever needed counter `0x10__`). All 17 5A request frames in the 5-1-26 bwcap3sec capture (probe + 2 meta + 13 samples + TERM) now match BW byte-for-byte after the fix. Fixed in framing.py `build_5a_frame` v0.14.3. | +| 2026-05-05 | §7.8 / Blastware file format | **CONFIRMED — File body assembly is contiguous concatenation, no de-duplication.** The "duplicate header+STRT strip" hack from v0.13.x was actively destroying valid waveform data — sample chunks at counter `0x1000` and beyond often coincidentally contain the byte sequence `00 12 03 00 STRT` in their delta-encoded ADC stream, and the strip was zeroing 25 bytes per match. Removed in v0.14.2. The correct file body is: probe contribution + meta@0x1002 + meta@0x1004 + sample contributions in stream order + TERM contribution. Verified byte-perfect against BW reference `M529LKIQ.G10` (8708 bytes, 0 differences) when fed the same A5 frames as the BW capture. | --- @@ -261,7 +264,7 @@ Step 4 — Device sends actual data payload: | `0A` | **WAVEFORM HEADER READ** | Checks record type for a given waveform key. Variable DATA_LENGTH: 0x30=full bin, 0x26=partial bin. Key at params[4..7]. Required before every 1F call to establish device waveform context. | ✅ CONFIRMED 2026-03-31 | | `0C` | **FULL WAVEFORM RECORD** | Downloads 210-byte waveform/histogram record. Sub_code at byte[1]: 0x10=Waveform (9-byte timestamp hdr), 0x03=Waveform-continuous (10-byte hdr, 1-byte shift). PPV floats at label+6 (search "Tran"/"Vert"/"Long"/"MicL"). Peak Vector Sum at tran_label−12 (NOT fixed offset). Key at params[4..7], DATA_LENGTH=0xD2. | ✅ CONFIRMED 2026-04-03 | | `1F` | **EVENT ADVANCE** | Advances to next waveform key. Token byte at params[7] (⚠️ NOT params[6]): 0x00=browse (all-zero params), 0xFE=download (arm 5A state machine). Returns next key at data[11:15]; null sentinel when data[15:19]=0x00000000. Requires preceding 0A to establish context. Browse 1F must ONLY be called after successful 5A — calling it after a failed 5A disrupts device state for the next event's 5A probe. | ✅ CONFIRMED 2026-04-06 | -| `5A` | **BULK WAVEFORM STREAM** | Bulk download of raw ADC sample data. Non-standard frame format: offset_hi=0x10 sent raw (not DLE-stuffed), DLE-aware checksum. Requires 1E-arm + 0C + 1F(0xFE) + POLL×3 before first probe. A5[7] contains event-time metadata (Project:/Client:/User Name:/Seis Loc:). 9+ A5 frames for full waveform; stop_after_metadata=True exits after A5[7]. | ✅ CONFIRMED 2026-04-06 | +| `5A` | **BULK WAVEFORM STREAM** | Bulk download of raw ADC sample data. Non-standard frame format: offset_hi=0x10 sent raw (not DLE-stuffed), DLE-aware checksum, **partial DLE stuffing of 0x10 in params** (`10 X` where X∉{02,03,04,10} must be doubled to `10 10 X` — see §7.8). Requires 1E-arm + 0C + 1F(0xFE) + POLL×3 before first probe. Walk: probe at counter=`start_offset` (event 1: 0x0000) → metadata pages 0x1002 + 0x1004 (event 1 only) → sample chunks at 0x0600, 0x0800, …, step 0x0200, bounded by `end_offset` parsed from STRT@data[17] of probe response → TERM frame at residual offset_word. Project:/Client:/User Name:/Seis Loc: live in the metadata pages, NOT in the sample-chunk stream. | ✅ CONFIRMED 2026-05-05 (BYTE-PERFECT vs BW capture) | | `24` | **WAVEFORM PAGE A?** | Paged waveform read, possibly channel group A. | 🔶 INFERRED | | `25` | **WAVEFORM PAGE B?** | Paged waveform read, possibly channel group B. | 🔶 INFERRED | | `09` | **UNKNOWN READ A** | Read command, response (`F6`) returns 0xCA (202) bytes. Purpose unknown. | 🔶 INFERRED | @@ -837,6 +840,20 @@ MicL: 39 64 1D AA = 0.0000875 psi ### 7.6 Bulk Waveform Stream (SUB A5) — Raw ADC Sample Records +> ⛔ **§7.6 below describes the deprecated `0x0400`-step walk and is RETAINED FOR HISTORICAL CONTEXT ONLY.** +> The "A5[7] is metadata", "A5[9] is terminator", and chunk-counter frame-index claims in this section +> are all artifacts of the broken walk that was overrunning past event end by ~5×. +> +> **For the corrected protocol (v0.14.0+), use:** +> - **§7.8.5** — chunk addressing (probe at `start_offset`, samples step 0x0200, bounded by STRT `end_offset`) +> - **§7.8.6** — TERM frame formula +> - **§7.8.7** — fixed metadata pages 0x1002 / 0x1004 (this is where Project / Client / User Name / Seis Loc +> strings actually live — NOT in any sample-chunk frame) +> - **§7.8.8** — multi-event "Download All" sequence +> +> The waveform sample encoding (4-channel interleaved s16 LE, 8 bytes per sample-set) described in §7.6.1 +> below is still correct. Only the frame-indexing claims and metadata-source claims are wrong. + **Two distinct formats exist depending on recording mode. Both confirmed from captures.** --- @@ -1119,20 +1136,26 @@ Near-ambient: 0x3C75C28F = 0.015 in/s (histogram event, near-zero ambient) **Project strings** — ASCII label-value pairs (search for label, read null-terminated value): ``` -"Project:" → project description (in 0C record ✅) -"Client:" → client name (in SUB 5A / A5 frame 7 ✅ — NOT in 0C) -"User Name:" → operator / user (in SUB 5A / A5 frame 7 ✅ — NOT in 0C) -"Seis Loc:" → sensor location (in SUB 5A / A5 frame 7 ✅ — NOT in 0C) -"Extended Notes"→ notes field (in SUB 5A / A5 frame 7 ✅) +"Project:" → project description (in 0C record ✅, also mirrored in metadata pages) +"Client:" → client name (in SUB 5A metadata pages ✅ — NOT in 0C) +"User Name:" → operator / user (in SUB 5A metadata pages ✅ — NOT in 0C) +"Seis Loc:" → sensor location (in SUB 5A metadata pages ✅ — NOT in 0C) +"Extended Notes"→ notes field (in SUB 5A metadata pages ✅) ``` -> ✅ **2026-04-02 — CONFIRMED:** `Client:`, `User Name:`, and `Seis Loc:` are sourced from -> **SUB 5A (bulk waveform stream)**, specifically A5 frame 7 of the multi-frame response. -> They are NOT present in the 210-byte SUB 0C waveform record. The strings reflect the -> compliance setup that was active when the event was recorded on the device — making SUB 5A -> the authoritative source for true event-time metadata. The `get_events()` client method -> now issues a SUB 5A request after each 0C download (`stop_after_metadata=True`) and -> overwrites `event.project_info` with the decoded fields. +> ✅ **UPDATED 2026-05-05:** `Client:`, `User Name:`, and `Seis Loc:` come from the +> dedicated **SUB 5A metadata pages at counter `0x1002` and `0x1004`** — see §7.8.7. +> They are NOT present in the 210-byte SUB 0C waveform record. +> +> An earlier draft of this doc claimed they came from "A5 frame 7" of the bulk waveform +> stream — that was an artifact of the deprecated `0x0400`-step walk where the broken +> chunk counter formula happened to land sample-chunk fi=7 on top of the 0x1002 metadata +> page. Under the corrected v0.14.0+ walk (§7.8.5), sample chunks at `0x1000` / `0x1200` +> contain ordinary waveform data, and the metadata pages are read separately. +> +> The strings reflect the compliance setup that was active when the *monitoring session* +> first started (not per-event). `get_events()` reads the metadata pages once at the start +> of the SFM session and the decoded values are stamped onto every event in that session. --- @@ -1166,7 +1189,9 @@ return events ### 7.7.7 Updated Download Loop with SUB 5A Metadata -> ✅ **Added 2026-04-02.** Confirmed working on BE11529 over TCP/cellular. +> ⛔ **The loop in this subsection is DEPRECATED — it uses the broken `stop_after_metadata=True` +> hack and the wrong sequence ordering.** See §7.8.5–§7.8.8 for the corrected protocol. +> The pseudocode below is preserved as historical record only. ```python key4, _ = proto.read_event_first() # SUB 1E @@ -1201,13 +1226,25 @@ return events ### 7.8 SUB 5A — Bulk Waveform Stream (event-time metadata) -> ✅ **Added 2026-04-02.** Frame format confirmed by reproducing Blastware wire bytes -> byte-for-byte from the 1-2-26 BW capture. +> ✅ **§7.8.1 (frame format) — added 2026-04-02; v0.14.3 partial DLE stuffing finalized 2026-05-05.** +> Frame format confirmed by reproducing Blastware wire bytes byte-for-byte across the 1-2-26 +> capture (10 frames) and the 5-1-26 bwcap3sec capture (17 frames, all match including the +> DLE-stuffed `10 10 00` for counter=0x1000). -SUB 5A initiates a bulk transfer of the raw sample data for a stored event. The response is a -sequence of A5 frames. Frame 7 (0-indexed) contains the full compliance setup as it existed -when the event was recorded — including `Client:`, `User Name:`, `Seis Loc:`, and -`Extended Notes` ASCII label-value pairs. +SUB 5A initiates a bulk transfer of the raw sample data for a stored event. The response is +a sequence of A5 frames. Project-info ASCII strings (`Project:`, `Client:`, `User Name:`, +`Seis Loc:`, `Extended Notes`) live in the dedicated metadata pages at counter `0x1002` +and `0x1004` (see §7.8.7), not in the sample-chunk stream. + +**For the corrected protocol read in order:** +- §7.8.1 — frame format (raw `offset_hi`, DLE-aware checksum, partial DLE stuffing of params) +- §7.8.5 — chunk addressing (probe → metadata pages → samples → TERM, all bounded by `end_offset`) +- §7.8.6 — TERM frame formula +- §7.8.7 — fixed metadata pages 0x1002 / 0x1004 +- §7.8.8 — multi-event "Download All" sequence + +§7.8.2–§7.8.4 are retained as historical record of earlier (incorrect) understandings — +do not implement against them. #### 7.8.1 Frame Format @@ -1218,7 +1255,7 @@ SUB 5A uses a **non-standard frame layout** that differs from all other BW→S3 41 02 10 10 00 5A 00 ^^raw^^ ^^raw^^ ^^stuffed^^ ``` -Two critical differences from `build_bw_frame`: +Three critical differences from `build_bw_frame`: 1. **`offset_hi` is sent raw, not DLE-stuffed.** When `offset_hi = 0x10`, the wire carries a bare `0x10` — NOT the stuffed `10 10` that `build_bw_frame` would produce. The device @@ -1227,6 +1264,31 @@ Two critical differences from `build_bw_frame`: 2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen, only `XX` is added to the running sum; lone bytes are added normally. +3. **Partial DLE stuffing of `0x10` bytes in the params region** (CONFIRMED 2026-05-05). + The device's de-stuffing rule for the params region is: + + - `10 10` → de-stuffs to `10` + - `10 02 / 03 / 04` → kept literal (these are inner-frame markers) + - `10 X` for other X → de-stuffs to just `X` (drops the leading `0x10`) + + Therefore any `0x10` byte in the *logical* params that is followed by a byte NOT in + `{0x02, 0x03, 0x04, 0x10}` MUST be doubled on the wire (`10 X` → `10 10 X`) so the + device's de-stuffer reproduces the original `10 X` pair. This applies most commonly + to counters with `0x10` in the high byte (e.g. counter=`0x1000` produces logical + params bytes `... 10 00 ...`, which BW encodes on the wire as `... 10 10 00 ...`). + Without this stuffing the device interprets counter=`0x1000` as `0x0000` and returns + the probe response (= a copy of the file header + STRT record); that STRT block then + ends up embedded in the assembled file body and Blastware refuses to open the file. + + `0x10` bytes in `offset_hi` are still written RAW per (1) above — only the params + region has this stuffing requirement. Metadata-page params for counter `0x1002` / + `0x1004` survive without stuffing because `10 02` / `10 04` fall in the "kept literal" + carve-out. + + Verified against BW 5-1-26 bwcap3sec frame 20: params logical bytes + `00 01 11 10 00 00 00 00 00 00 00` (counter=0x1000) are encoded on the wire as + `00 01 11 10 10 00 00 00 00 00 00 00` (12 wire bytes for 11 logical bytes). + #### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol) > ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new @@ -1267,11 +1329,20 @@ when the broken 0x0400-step walk passed the global metadata pages at 0x1002/0x10 the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004, not from the sample-chunk stream — see §7.8.7. -#### 7.8.3 A5 Frame Layout +#### 7.8.3 A5 Frame Layout — DEPRECATED 2026-05-01 -Each A5 response frame contains a chunk of raw bulk data. Frame 7 of the stream carries the -compliance text block with all project-info label-value pairs. The `client` layer searches -for ASCII labels with a null-terminated value read: +> ⛔ **The "Frame 7 carries the compliance text block" claim below is WRONG.** It was +> an artifact of the deprecated `0x0400`-step walk where the broken counter formula +> happened to land sample-chunk fi=7 on top of the 0x1002 metadata page in flash. +> Under the corrected v0.14.0+ walk (§7.8.5), Frame 7 of the sample-chunk sequence is +> just sample-chunk #5 (counter=0x1000), and contains either ordinary waveform data or — +> critically when DLE-stuffing of params is wrong (§7.8.1.3) — a duplicate file header + +> STRT block when the device misinterprets counter=0x1000 as 0x0000. See §7.8.7 for the +> actual source of these strings. + +Historical claim (NOT TO BE IMPLEMENTED): each A5 response frame contains a chunk of raw +bulk data; Frame 7 of the stream carries the compliance text block with all project-info +label-value pairs: ``` "Project:" → null-terminated project name @@ -1281,7 +1352,9 @@ for ASCII labels with a null-terminated value read: "Extended Notes" → null-terminated notes ``` -All five fields reflect the **setup at event-record time**, not the current device config. +All five fields do reflect the **setup at event-record time**, not the current device +config. But the source is the metadata pages (§7.8.7), not "Frame 7" of the sample +stream. #### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01 @@ -1560,10 +1633,10 @@ Fields visible in the Blastware "Compliance Setup" dialog. ✅ = byte offset co | Field | Values / Type | Status | |---|---|---| | Enable User Notes | bool | ❓ | -| Project | ASCII string | ✅ (sourced from A5 frame 7 via SUB 5A) | -| Client | ASCII string | ✅ (sourced from A5 frame 7) | -| User Name | ASCII string | ✅ (sourced from A5 frame 7) | -| Seis Loc | ASCII string | ✅ (sourced from A5 frame 7) | +| Project | ASCII string | ✅ (sourced from SUB 5A metadata pages at counter `0x1002` / `0x1004` — see §7.8.7) | +| Client | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) | +| User Name | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) | +| Seis Loc | ASCII string | ✅ (sourced from SUB 5A metadata pages — see §7.8.7) | | Enable Extended Notes | bool | ❓ | | Extended Notes | ASCII text | ❓ | | Extended Notes Title | ASCII string | ❓ | From ebfe9877faa369114bd0ce55a8aee72e6ef8c737 Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Tue, 5 May 2026 20:39:47 -0400 Subject: [PATCH 10/11] doc: update changelog to 0.14.3 --- CHANGELOG.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 29e50d1..9c2dda0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -98,25 +98,6 @@ All notable changes to seismo-relay are documented here. --- -## v0.12.6 — 2026-05-01 - -### Fixed - -- **`blastware_file.py` — waveform frame classification** — A5 frame classification for - waveform-only vs header-only frames now uses `frame.record_type` instead of frame index. - Only waveform frames (0x46) are written to the file body; metadata frames are skipped. - Fixes spurious data corruption from incorrectly classified frames. - -- **`s3_analyzer.py` — A5/5A frame naming** — Bulk waveform stream frames (SUB 5A response) - are now correctly labeled "A5" in analyzer output instead of being conflated with other - multi-frame responses (SUB A4, E5, etc.). - -- **`S3FrameParser` — frame terminator detection** — Corrected the bare ETX terminator - detection. Frame termination is now correctly identified by a standalone `ETX=0x03` byte, - not by the `DLE+ETX` sequence (which is part of the payload when it appears within a frame). - ---- - ## v0.13.2 — 2026-05-01 ### Fixed @@ -223,6 +204,25 @@ All notable changes to seismo-relay are documented here. --- +## v0.12.6 — 2026-05-01 + +### Fixed + +- **`blastware_file.py` — waveform frame classification** — A5 frame classification for + waveform-only vs header-only frames now uses `frame.record_type` instead of frame index. + Only waveform frames (0x46) are written to the file body; metadata frames are skipped. + Fixes spurious data corruption from incorrectly classified frames. + +- **`s3_analyzer.py` — A5/5A frame naming** — Bulk waveform stream frames (SUB 5A response) + are now correctly labeled "A5" in analyzer output instead of being conflated with other + multi-frame responses (SUB A4, E5, etc.). + +- **`S3FrameParser` — frame terminator detection** — Corrected the bare ETX terminator + detection. Frame termination is now correctly identified by a standalone `ETX=0x03` byte, + not by the `DLE+ETX` sequence (which is part of the payload when it appears within a frame). + +--- + ## v0.12.5 — 2026-04-21 ### Added From 29ebc7565609100133e6613d30326745e7a14572 Mon Sep 17 00:00:00 2001 From: Brian Harrison Date: Tue, 5 May 2026 20:48:58 -0400 Subject: [PATCH 11/11] doc: update readme v0.14.3 --- README.md | 40 ++++++++++++++++++++++++++-------------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index e0aafb7..27dda6c 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# seismo-relay `v0.12.6` +# seismo-relay `v0.14.3` A ground-up replacement for **Blastware** — Instantel's aging Windows-only software for managing MiniMate Plus seismographs. @@ -10,6 +10,10 @@ over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55). > pipeline working end-to-end over TCP/cellular. ACH Auto Call Home server > handles inbound unit connections, downloads events, and persists everything > to a SQLite database. SFM REST API exposes device control and DB queries. +> **As of v0.14.3 (2026-05-05): SUB 5A bulk waveform protocol is verified +> byte-perfect against Blastware captures across 2-sec, 3-sec, and 10-sec +> events.** Generated `.G10` / `.AB0` files open cleanly in Blastware with +> full Event Reports, frequency analysis, and waveform plots. > See [CHANGELOG.md](CHANGELOG.md) for full version history. --- @@ -194,9 +198,14 @@ with client: client.delete_all_events() # Erase all (SUB 0xA3 → 0x1C → 0x06 → 0xA2) ``` -`get_events()` runs the full per-event sequence: `1E → 0A → 0C → 5A → 1F`. -SUB 5A bulk stream provides `client`, `operator`, and `sensor_location` as they -existed at record time — not backfilled from the current compliance config. +`get_events()` runs the full per-event sequence: +`1E → 0A → 1E(arm token=0xFE) → 0C → 1F(arm) → POLL×3 → 5A → 1F(browse)`. +SUB 5A bulk stream walks chunks bounded by the `end_offset` extracted from +the STRT record at byte 17 of the probe response — no over-reading, no +chunk-count cap. Project / client / operator / sensor location strings come +from the dedicated metadata pages at counter `0x1002` and `0x1004`, +read once per session (they reflect the compliance setup at session start, +not per individual event). --- @@ -253,7 +262,7 @@ Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/insta --- -## Compliance Config Features (v0.12.2–v0.12.3) +## Compliance Config Features The REST API and web UI expose full control over device compliance settings: @@ -295,34 +304,36 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows. --- -## Key Features (v0.10–v0.12) +## Key Features -**Device support (v0.12.5):** +**Device support:** - [x] Full read/write/erase pipelines - [x] Compliance config (recording mode, sample rate, histogram interval, geo sensitivity, project strings) - [x] Auto Call Home config (read/write ACH settings, dial string, time slots, retries) - [x] Monitor control (start/stop, status polling, battery/memory) - [x] Monitor log entries (continuous monitoring intervals without full waveform download) -**Data persistence (v0.11):** +**Data persistence:** - [x] SQLite database (`seismo_relay.db`) with 4 tables: ach_sessions, events, monitor_log, plus false_trigger flag - [x] Deduplication by waveform key (handles re-runs and repeat call-homes) - [x] Post-erase key-reuse detection (tracks high-water mark) - [x] Session state (`ach_state.json`) with downloaded keys and max key -**REST API (v0.12.1):** +**REST API:** - [x] Live device endpoints with in-memory caching (`_LiveCache`) - [x] Cache statistics (`/cache/stats`) and manual invalidation (`/cache/device`) - [x] DB query endpoints (units, events, monitor_log, sessions, false_trigger PATCH) - [x] Call Home config read/write endpoints - [x] Blastware file download endpoint (`/device/event/{index}/blastware_file`) -**File output (v0.7+):** -- [x] Blastware-compatible `.AB0` file generation (waveform + metadata) +**File output (v0.7+, byte-perfect as of v0.14.3):** +- [x] Blastware-compatible `.AB0` / `.G10` file generation (waveform + metadata) - [x] Multi-channel waveform decode from SUB 5A bulk stream - [x] Second-resolution timestamp encoding in Blastware filename +- [x] **Byte-perfect against BW reference captures** (verified across 2-sec / 3-sec / 10-sec event durations, both event 0 and event N continuation events) +- [x] STRT-bounded chunk walk + correct event-N probe counter + partial DLE stuffing of `0x10` in 5A params (the four fixes that landed in v0.14.0–v0.14.3) -**Capture tools (v0.12.5):** +**Capture tools:** - [x] Serial-to-TCP bridge with raw BW/S3 capture (s3_bridge.py, defaults to auto-capture) - [x] GUI bridge with raw capture checkboxes (gui_bridge.py) - [x] ACH inbound server with bidirectional capture (ach_server.py saves raw_tx + raw_rx) @@ -333,14 +344,15 @@ Use **com0com** or **VSPD** to create the virtual COM pair on Windows. - [x] gui_analyzer.py — standalone analyzer GUI - [x] frame_db.py — SQLite frame database for capture analysis -**seismo_lab.py GUI (v0.12.5):** +**seismo_lab.py GUI:** - [x] Bridge tab — Serial/TCP mode selector with raw capture options - [x] Analyzer tab — BW/S3 capture playback and differencing -- [x] Download tab — Live wire-byte capture during event download (new v0.12.5) +- [x] Download tab — Live wire-byte capture during event download - [x] Console tab — Logging and diagnostics ## Roadmap (Future) +- [ ] Verify 30-sec event download — body may exceed `0xFFFF` and force the device into a different `end_key` encoding (none of 2/3/10-sec test cases hit this boundary) - [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing - [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first) - [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object