serversdown/seismo-relay
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Fix: Removed duplicates from merge botch. Stable version of seismo_lab.py

Browse Source
This commit is contained in:
Brian Harrison
2026-05-01 17:34:41 -04:00
parent 738b39f3cb
commit 1ef55521b1
3 changed files with 458 additions and 237 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/instantel_protocol_reference.md
+225 -50
View File
@@ -110,6 +110,7 @@
| 2026-04-21 | §7.6.2, §5.3 | **CORRECTED — compliance_raw contains wire-encoded bytes, NOT logical bytes.** S3FrameParser appends DLE+ETX inner-frame pairs as two literal bytes to the frame body. Any `0x03` values in the compliance config appear in `compliance_raw` as `\x10\x03` (two bytes), not as a single `0x03`. The previous claim "S3FrameParser handles this transparently so compliance_raw contains logical (destuffed) bytes" was wrong. Consequence: `compliance_raw` is the wire-encoded E5 payload; anchor-relative reads work correctly because the anchor position automatically accounts for any DLE-encoded bytes before it. For write-back, round-tripping `compliance_raw` verbatim sends the correct wire bytes to the device. **DLE ETX escaping in write frames:** Blastware escapes `0x03` bytes in write frame data as `\x10\x03` on wire; our `build_bw_write_frame` does not (writes data raw). Device is confirmed to accept raw writes for all tested modes — likely uses the offset/length field for write frame framing, not ETX scanning. |
| 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. |
| 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. |
| 2026-05-01 | §7.8.2, §7.8.5 (NEW), §7.8.6 (NEW), §7.8.7 (NEW) | **REWRITTEN — SUB 5A bulk waveform stream protocol.** Five BW MITM captures (4-27-26 "open 2sec waveform" + "copy event to disk", 5-1-26 BW 3-sec + 2nd-event + Download All) prove that the previous chunk-counter formula `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` over-reads 5× past the actual event end. BW reads ~12-16 chunks per event at **0x0200 increments (NOT 0x0400)**, bounded by `end_offset` extracted from the STRT record at `data[23:27]` of the first A5 response. **TERM frame formula corrected:** `offset_word = end_offset - next_boundary`, `params[2:4] = next_boundary BE` where `next_boundary = last_chunk_counter + 0x0200`. Verified across 3 events (offsets 0x1ABE, 0x21F2, 0x417E). **Metadata pages 0x1002 / 0x1004** are global, fixed-address device pages containing Project/Client/User Name/Seis Loc/Extended Notes — read ONCE per Blastware session (not per event). **Event-1 vs event-N split:** events at start_key[2:4]=0 use probe@0x0000 + metadata pages + sample chunks at 0x0600 onward; continuation events skip metadata and start at start_key+0x0046. **WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundary markers** — the "Download All" pattern walks 1E/0A/1F to map all event keys+lengths upfront, then downloads each `0x46`-keyed event in turn. Old `stop_after_metadata=True` knob is a workaround for the missing end_offset bound and becomes obsolete under the new walk. See new §7.8.5 / §7.8.6 / §7.8.7 for full details. |
---
@@ -1226,7 +1227,24 @@ Two critical differences from `build_bw_frame`:
2. **DLE-aware checksum.** Walking the full frame byte sequence: when a `10 XX` pair is seen,
only `XX` is added to the running sum; lone bytes are added normally.
#### 7.8.2 Request Sequence
#### 7.8.2 Request Sequence — DEPRECATED 2026-05-01 (see §7.8.5–§7.8.7 for the corrected protocol)
> ⛔ **The 0x0400-step / max(key4[2:4], 0x0400) formula in this section is WRONG.** Five new
> BW MITM captures (4-27-26 + 5-1-26) prove the actual chunk increment is **0x0200**, the
> chunk loop is bounded by `end_offset` from the STRT record (not by chunk count or by a
> device-side timeout), and the TERM frame's `offset_word=0x005A` magic is incorrect — the
> real TERM offset_word is computed from `end_offset` and the last chunk address. Under the
> deprecated formula SFM over-reads roughly 5× past the actual event end into post-event
> circular-buffer garbage, corrupting reconstructed Blastware files for any waveform ≥ 2 sec.
>
> The whole "stop_after_metadata + one extra chunk + 0e 08 footer" workaround in this
> section was compensating for the missing end_offset bound. It is obsoleted by the
> STRT-bounded walk in §7.8.5.
>
> **Read this section for historical context only.** For the correct protocol, jump to:
> - §7.8.5 — chunk addressing and the STRT end_offset
> - §7.8.6 — TERM frame formula
> - §7.8.7 — fixed metadata pages 0x1002 and 0x1004
| Frame | offset_word | counter | params | Purpose |
|---|---|---|---|---|
@@ -1237,45 +1255,17 @@ Two critical differences from `build_bw_frame`:
| … | … | … | … | … |
| Termination | `0x005A` | `max(key4[2:4], 0x0400) + N * 0x0400` | 10 bytes | End transfer |
> ⚠️ **2026-04-06 CORRECTED — chunk counter is `key4[2:4] + (N-1) * 0x0400`.**
> The 4-2-26 BW TX capture showed counter=0x1004 for chunk 1 of key `01110000`, leading to
> an interim "monotonic n * 0x0400" formula. This was accidentally correct because
> `key4[2:4] == 0x0000` for that event.
>
> **2026-04-24 CORRECTION:** The counter is an absolute circular-buffer address.
> BW's true formula is `key4[2:4] + (chunk_num - 1) * 0x0400` where `key4[2:4]` is the
> event's storage base offset (`(key4[2]<<8) | key4[3]`). For keys where
> `key4[2:4] != 0x0000` (e.g. key `01111884`), using `n * 0x0400` sends requests into the
> wrong buffer region — the device returns data from a completely different event.
>
> **2026-04-26 FINAL CORRECTION:** The formula `key4[2:4] + (N-1) * 0x0400` is wrong when
> `key4[2:4] == 0x0000` (e.g. event key `01110000`, the very first event after a device erase).
> Counter=0x0000 for chunk 1 is the same address as the probe frame — the device re-returns
> the STRT record data instead of waveform payload (frame 1 has len=1097, same as probe, and
> contains `b"STRT\xff\xfe"`, contributing zero waveform bytes).
> Final formula: `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400`.
> For key `01110000`: chunk 1 = 0x0400 (confirmed working, empirical test 2026-04-06).
> For key `0111245a`: chunk 1 = 0x245a (unchanged, confirmed from 4-3-26 capture).
> Historical correction notes (left in place to deter re-derivation of the same wrong formula):
> the table above was the result of three iterative "corrections" between 2026-04-06 and
> 2026-04-26 that progressively narrowed in on the wrong answer because every test was on
> events with `key4[2:4]=0` and the device responds to whatever counter you ask for. The
> 5-1-26 captures with a non-zero start_key event (`01112238`) finally exposed the bug.
The `stop_after_metadata=True` flag causes the loop to stop as soon as `b"Project:"` is
found in the accumulated A5 frame data, typically after 4–9 chunks. A termination frame
is always sent before returning.
**IMPORTANT — one extra chunk required after "Project:" for valid file footer (confirmed 2026-04-23):**
When writing a Blastware-compatible waveform file, stopping immediately at "Project:" and
sending termination produces an empty termination response with no footer bytes (`0e 08`
marker missing). Blastware downloads exactly **one more chunk** after finding "Project:"
before sending termination — that extra chunk primes the device to return valid footer
bytes (monitoring start/stop timestamps) in the termination response.
`read_bulk_waveform_stream(stop_after_metadata=True)` implements this: after the "Project:"
chunk is received, one additional chunk is requested before breaking. The termination
response (`include_terminator=True`) then contains the correct `0e 08` footer.
**do NOT use `full_waveform=True` for Blastware file writing** — for events with long
post-event silence (35 chunks), the silence chunks contain embedded device-internal
pointer structures that produce spurious STRT markers in the file body. Blastware only
downloads 4–5 chunks (metadata + one signal chunk) regardless of event length.
The `stop_after_metadata=True` flag (deprecated as a primary loop-exit) scanned for
`b"Project:"` in the chunk stream because the metadata strings happened to be reachable
when the broken 0x0400-step walk passed the global metadata pages at 0x1002/0x1004. Under
the corrected walk, those strings come from explicit reads at counter=0x1002 and 0x1004,
not from the sample-chunk stream — see §7.8.7.
#### 7.8.3 A5 Frame Layout
@@ -1293,15 +1283,19 @@ for ASCII labels with a null-terminated value read:
All five fields reflect the **setup at event-record time**, not the current device config.
#### 7.8.4 End-of-Stream Behaviour and Chunk Timing
#### 7.8.4 End-of-Stream Behaviour and Chunk Timing — REINTERPRETED 2026-05-01
> ✅ **Confirmed 2026-04-06** — empirical observation on BE11529 (S338.17) over TCP/cellular.
> The "1 raw byte then silence" pattern documented below was originally interpreted as
> "the device's natural end-of-event signal." The 5-1-26 captures show this is actually
> the device's response when the requester has walked **past** the addressable buffer
> region (i.e. ~5× past the actual event end under the deprecated 0x0400-step walk).
> Under the corrected STRT-bounded walk (§7.8.5), the stream ends cleanly with the TERM
> frame's response — no timeout, no 1-byte teaser. The fallback below remains useful as
> defensive handling for malformed events but should not be the primary loop-exit.
**End-of-stream signal:** After sending all waveform chunks, the device sends exactly **1 raw byte** in response to the next chunk request, then goes silent. This byte is not a complete DLE-framed A5 response — `S3FrameParser.bytes_fed` reports 1 and no frame is ever assembled. This is the device's natural end-of-stream indicator.
Handling logic in `read_bulk_waveform_stream`:
**Defensive fallback handling in `read_bulk_waveform_stream`:**
```
TimeoutError caught:
TimeoutError caught (rare under corrected walk):
if bytes_fed > 0 AND frames already collected:
→ graceful end-of-stream; break loop; proceed to termination frame
else (bytes_fed == 0, no prior frames):
@@ -1313,14 +1307,15 @@ TimeoutError caught:
| Metric | Observed value |
|---|---|
| Chunk response time | ~1 s per chunk |
| Chunks for a 9,306-sample event | 35 chunks |
| Data per chunk (active signal) | 1,036–1,123 bytes |
| Data per chunk (post-event silence) | 1,036 bytes (uniform) |
| Chunks for a 2-sec event (corrected walk) | 14 (12 sample chunks + 2 metadata pages) + TERM |
| Chunks for a 3-sec event (corrected walk) | 18 (16 sample chunks + 2 metadata pages) + TERM |
| Chunks for a continuation event (corrected walk) | ~15 sample chunks + TERM (no metadata reread) |
| Chunks under deprecated walk for 2-3 sec event | 37 (over-reads ~5×) |
| Data per chunk (corrected, 0x0200 size) | ~540–575 bytes wire (= 0x0200 payload + framing) |
| Data per chunk (deprecated 0x0400 step) | 1,036–1,123 bytes wire (= 0x0400 payload + framing) |
| Safe recv timeout per chunk | **10 s** (10× typical) |
| Default transport timeout | 120 s → ~2-min stall at end-of-stream |
Chunks with uniform 1,036-byte payload (chunks 17–35 in the observed event) contain all-zero ADC samples — the device continues recording silence until the configured record time expires before terminating the stream.
**ADC count-to-physical conversion — ✅ CONFIRMED 2026-04-17:**
Raw samples are signed 16-bit integers (−32,768 to +32,767). Source: Interface Handbook §4.5.
@@ -1339,6 +1334,186 @@ where `geo_range = 1.61133 V × 6.206053 = 10.000 in/s` is the Normal (Gain=1) f
`_decode_a5_waveform()` contains `elif fi == 9: continue` from an earlier assumption that frame index 9 is always the device terminator. For streams with more than 9 frames, frame 9 is live waveform data. The skip discards ~1,070 bytes (~133 sample-sets) per event. Terminator detection should use `page_key == 0x0000`, not frame index. This skip should be removed.
#### 7.8.5 Chunk addressing and the STRT end_offset (NEW 2026-05-01) ✅
> ✅ Confirmed across 3 events (4-27-26 + 5-1-26 captures).
`params[0]` is always `0x00`. `params[1:5]` is a 4-byte absolute device flash-buffer
address — equivalently, "the key of the page being requested." The device returns 0x0200
(= 512) bytes starting at that address. Increments between consecutive sample chunks are
**0x0200, NOT 0x0400** (the previous 0x0400 figure was a Blastware-side artifact / our
implementation's bug — see §7.8.2).
##### STRT record (data layout in the first A5 response)
The first A5 response (the probe response, or the first chunk for continuation events)
contains a **STRT record** at byte offset 17 of `data`:
```
data[ 0:14] echoes request: [chunk_size_hi=0x02 / 0x04 ...] [00] [01 11] [counter_hi counter_lo] [00 × 8] [00 12]
data[14:17] 10 03 00 ← inner DLE+ETX frame separator (preserved literally)
data[17:21] "STRT" ← magic
data[21:23] ff fe ← sentinel
data[23:27] end_key ← 4-byte key of where this event ENDS
data[27:31] start_key ← 4-byte key of where this event STARTS
data[31:33] uint16 BE ← ?? sample count or byte count, varies (not yet decoded)
data[33:35] uint16 BE ← ??
data[35] 0x46 ← record type marker (waveform full record)
data[36:] additional pointers / first sample bytes — content varies by event
```
`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**.
Use it to bound the chunk loop and to compute the TERM frame.
##### Chunk pattern by event location in buffer
**Event 1 / start_key[2:4] = 0x0000** (first event after erase or wrap):
```
1. Probe at counter = 0x0000 (params[1:5] = full key)
2. Read fixed metadata pages counter = 0x1002, then 0x1004
3. Walk sample chunks counter = 0x0600, 0x0800, …, by 0x0200,
up to but not including end_offset & 0xFE00
4. TERM (see §7.8.6)
```
The range `[0x0046, 0x0600)` is skipped — likely some pre-event firmware-reserved area for
the first slot in a freshly-erased buffer. Harmless to skip; BW does the same.
**Event 2+ / start_key[2:4] != 0x0000** (continuation events in a populated buffer):
```
1. First chunk at counter = start_key[2:4] + 0x0046 ← acts as both probe and first
sample chunk; response carries STRT
2. Walk sample chunks counter += 0x0200 each
3. TERM
```
**No metadata-page reads.** Pages 0x1002/0x1004 are session-global and were already read
during event 1 in the same Blastware session. In SFM, treat metadata pages as a once-
per-`MiniMateClient.connect()` (or once-per-call-home) read, not per-event.
##### Verified end_offset values
| Capture | start_key | end_key | end_offset | event size | sample-chunk start |
|---|---|---|---|---|---|
| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B | 0x0600 (event-1 case) |
| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B | 0x0600 (event-1 case) |
| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | event 2 size = 0x1F8C = 8,076 B | 0x2238 (= 0x21F2 + 0x46) |
#### 7.8.6 TERM Frame Formula (NEW 2026-05-01) ✅
> ✅ Confirmed across 3 events. Replaces the deprecated `offset_word=0x005A` / `params[2] = key4[2]` formula in §7.8.2.
The TERM frame fetches the partial last chunk and the file footer. Its response payload
contains the bytes between the last full 0x0200-aligned chunk and `end_offset` — typically
20–520 B — and is **required for reconstructing the Blastware waveform file**. Append the
TERM response data to the chunk stream like any other A5 frame.
```
last_chunk_counter = address of last full 0x0200-byte chunk read
next_boundary = last_chunk_counter + 0x0200
TERM offset_word = end_offset - next_boundary
TERM params[0] = key[0] (= 0x01 on every observed device)
TERM params[1] = key[1] (= 0x11)
TERM params[2] = (next_boundary >> 8) & 0xFF
TERM params[3] = next_boundary & 0xFF
TERM params[4:10] = zeros ← 10-byte params (not 11)
Frame = build_5a_frame(offset_word, params)
```
The device receives `requested_address = (params[2] << 8) | offset_word` (where offset_word
contains both `offset_hi` and `offset_lo` of the 5A frame, with the high bit of offset_hi
being effectively `bit 0 of (end_offset >> 8)`). It reconstructs `end_offset` and replies
with `(end_offset - next_boundary)` bytes of waveform tail starting at `next_boundary`.
##### Verification
| Event | end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] | TERM response size |
|---|---|---|---|---|---|---|
| 2-sec | `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ | 208 B |
| 3-sec | `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ | 520 B |
| Event-2 | `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ | (not measured directly; same pattern) |
Equivalent way to write the formula:
- `offset_word = end_offset & 0x01FF` — low 9 bits of end_offset
- `params[2:4] = (end_offset & 0xFE00) BE` — high 7 bits of end_offset, low byte zeroed
(The two forms are arithmetically identical to `end_offset - next_boundary` and
`next_boundary` because `next_boundary = end_offset & 0xFE00` whenever the chunk loop
stopped at the last full 0x0200 boundary below end_offset.)
#### 7.8.7 Fixed Metadata Pages 0x1002 / 0x1004 (NEW 2026-05-01) 🔶
> 🔶 Inferred — observed in BW captures but page contents not yet byte-decoded.
Two chunk addresses are **GLOBAL** device/session metadata, not event-specific:
- `counter = 0x1002` — first metadata page
- `counter = 0x1004` — second metadata page
These are at fixed absolute addresses in the device's flash buffer. They contain the
session-start compliance-setup ASCII strings — **Project**, **Client**, **User Name**,
**Seis Loc**, **Extended Notes** — that under the deprecated 0x0400-step walk used to be
discoverable in the sample-chunk stream as "A5 frame 7" content. Under the corrected
0x0200-step walk these strings come exclusively from the dedicated metadata-page reads,
not from sample chunks.
##### Caching strategy
BW reads them ONCE per Blastware session, during event 1's download, and caches them.
For SFM:
- Read once per `MiniMateClient.connect()` / once per call-home session.
- Subsequent events in the same session don't need to re-fetch them.
- Their content does not change while iterating events. They DO change when the user
applies a new compliance setup (SUB 71 write) — invalidate the cache then.
##### TODO — content layout
The byte-for-byte layout of pages 0x1002 and 0x1004 has not been decoded. First task on
the implementation side: dump both pages from a fresh capture and verify they include all
the strings currently extracted from the deprecated A5 frame 7 of the chunk stream.
Compare to the existing `_decode_a5_metadata_into` parser — same string-search anchors
(`b"Project:"`, `b"Client:"`, `b"User Name:"`, `b"Seis Loc:"`, `b"Extended Notes"`) likely
apply directly.
#### 7.8.8 "Download All" Sequence (NEW 2026-05-01) ✅
> ✅ Confirmed from 5-1-26 "Download All" capture (`raw_*_171216_download_all_2events.bin`).
Before any 5A traffic, BW's "Download All" pre-walks the entire event chain to map keys
and event boundaries:
```
SERIAL × 2 → CHCFG → EVT_KEY (1E, all-zero) → key0
→ WAVEHDR (0A, key0) → off=0x46 (real event start)
→ EVT_NEXT (1F, all-zero) → key1
→ WAVEHDR (0A, key1) → off=0x2C (boundary)
→ EVT_NEXT → key2
→ WAVEHDR (0A, key2) → off=0x46 (real event start)
→ EVT_NEXT → key3
→ WAVEHDR (0A, key3) → off=0x2C (boundary)
→ EVT_NEXT → null sentinel
```
The DATA_LENGTH at `data_rsp.data[5]` (echoed BW offset for the data fetch step)
disambiguates real events from boundary markers:
| WAVEHDR offset | Meaning |
|---|---|
| `0x46` (= 70) | Real event start key — this key has event data behind it |
| `0x2C` (= 44) | Boundary marker — this key is the END of the previous event AND the start of the empty/header gap before the next event |
Pairs: each real event spans `[real_key, next_real_key)` in the buffer. In the example
above: event 1 = `[01110000, 011121F2)`, event 2 = `[01112238, 0111417E)`. Note that the
"end of event 1" key (`011121F2`) is also the "boundary key" that comes BEFORE event 2's
real start key (`01112238`) — they differ by exactly 0x46 bytes (the event header size).
After the pre-walk completes, BW downloads each `0x46`-keyed event in turn using the 5A
bulk stream protocol from §7.8.5. Use the `0x46` keys, not the `0x2C` keys, as input to
`read_bulk_waveform_stream`.
---
## 7.9 Compliance Config Field Inventory (Blastware UI, 2026-04-08) ✅