serversdown/seismo-relay
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Fix: Removed duplicates from merge botch. Stable version of seismo_lab.py

Browse Source
This commit is contained in:
Brian Harrison
2026-05-01 17:34:41 -04:00
parent 738b39f3cb
commit 1ef55521b1
3 changed files with 458 additions and 237 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+228 -33
View File
@@ -41,7 +41,7 @@ Full read pipeline + write pipeline + erase pipeline + monitor log + call home c
| Event header / first key | 1E | ✅ |
| Waveform header | 0A | ✅ |
| Waveform record (peaks, timestamp, project) | 0C | ✅ |
| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ new v0.6.0 |
| **Bulk waveform stream (event-time metadata)** | **5A** | ⚠️ partial — over-reads ~5× past event end for ≥2-sec events; corrected algorithm documented but not yet implemented (see "SUB 5A — chunk counter formula" section, dated 2026-05-01) |
| Event advance / next key | 1F | ✅ |
| **Write commands (push config to device)** | **68–83** | ✅ new v0.8.0 |
| **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 |
@@ -118,29 +118,156 @@ S3→BW (response):
Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26
BW TX capture. All 10 frames verified.
### SUB 5A — chunk counter formula (FINAL CORRECTION 2026-04-26)
### SUB 5A — chunk counter formula (REWRITTEN 2026-05-01 — see 5-1-26 captures)
**Chunk counter = `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` for ALL chunks.**
> ⚠️ **Everything that came before this rewrite was WRONG in important ways.** The previous
> formula `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` happened to *work* for events
> at start_key=0 because the device responds to whatever counter you ask for — but it caused
> a 5× over-read past the actual event, picking up post-event circular-buffer garbage that
> corrupts the reconstructed file for any event > ~1 sec of waveform. The captures in
> `bridges/captures/4-27-26/` and `5-1-26/comcheck/` show BW reads only ~12-16 chunks for
> the same events SFM was reading 37+ chunks for. See "TERM frame" and "STRT end_offset"
> sections below for the actual mechanism.
where `key4[2:4] = (key4[2] << 8) | key4[3]` is the event's circular-buffer base offset.
**Chunk addressing is just absolute device-buffer addresses.**
The `max(..., 0x0400)` guard is critical for events at the start of the circular buffer
(key4[2:4] == 0x0000, e.g. key `01110000`). Without it, chunk 1 gets counter=0x0000, which
is the same address as the probe frame — the device re-returns the STRT record data instead
of waveform payload. With the guard, chunk 1 gets counter=0x0400, which is confirmed correct
from the empirical live-device test 2026-04-06 (`counter=0x0400 → responds immediately and
streams all frames correctly`).
`params[0]=0x00`, `params[1:5]` is a 4-byte absolute device flash-buffer address (= the
"key" of that location), `params[5:11]` are zeros. The device returns 0x0200 (= 512) bytes
starting at that address. Increments between consecutive chunks are **0x0200 (NOT 0x0400)**
— this matches the chunk payload size. The previous "0x0400 step" worked by accident: BW
asks for half-size chunks; SFM was asking for double-size chunks, both with the same-named
"counter" field, but the value is just an address pointer the device honors as-is.
The 4-3-26 capture confirms the pattern for a second event (key `0111245a`, key4[2:4]=0x245a):
chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400).
`max(0x245a, 0x0400) = 0x245a` → formula works correctly for non-zero base offset too.
**The chunk pattern depends on whether the event sits at start_key=0 or not.**
#### Event 1 case — start_key[2:4] == 0x0000 (first event after erase / wrap)
```
1. Probe at counter=0x0000 (params[1:5] = full key, returns STRT record)
2. Read 2 fixed metadata pages: counter=0x1002, counter=0x1004
(these are GLOBAL session metadata — read ONCE per
Blastware session, not per event; contain the
Project/Client/User Name/Seis Loc strings)
3. Sample chunks: counter=0x0600, 0x0800, …, by 0x0200 increment,
up to but not including end_offset (rounded down to
0x0200 boundary)
4. TERM frame (see TERM formula below)
```
The reason `0x0046..0x0600` is skipped for event 1 is unknown — likely some pre-event
firmware reserved area for the first slot in a freshly-erased buffer. Harmless to skip.
#### Event 2+ case — start_key[2:4] != 0x0000 (continuation events)
```
1. First chunk at counter = start_key[2:4] + 0x0046 (this IS the probe — response
contains STRT)
2. Sample chunks: counter += 0x0200 each, up to but
not including end_offset
3. TERM frame
```
No metadata pages — those have already been read during event 1 in the same Blastware
session, and BW caches them. Note that the metadata-page reads happen ONCE per
Blastware-session-on-the-device, not once per event, so an SFM session that downloads
several events should read 0x1002/0x1004 only once at the start.
#### History (do not re-derive)
**History:**
- Original: `_CHUNK1_COUNTER = 0x1004` hardcoded (Blastware capture artifact — WRONG).
- 2026-04-06: Corrected to `chunk_num * 0x0400` (worked for key 01110000 only).
- 2026-04-24: Corrected to `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets,
but accidentally broke key 01110000 — counter=0x0000 sends probe address again).
- 2026-04-26: Final formula: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400`.
- 2026-04-06: `chunk_num * 0x0400` (worked for key 01110000 only).
- 2026-04-24: `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets, broke key 01110000).
- 2026-04-26: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400` (broken — over-read past event end).
- 2026-05-01: Increments are 0x0200 not 0x0400; absolute addresses inside event range; bounded
by STRT end_key, not by `max_chunks` cap or device-side timeout.
### SUB 5A — STRT record encodes end_offset (NEW 2026-05-01)
The first A5 response (probe response, or the first chunk for event 2+) contains a STRT
record at byte offset 17 of the `data` field. Layout:
```
data[17:21] "STRT" magic
data[21:23] ff fe sentinel
data[23:27] end_key ← 4-byte key of where this event ENDS
data[27:31] start_key ← 4-byte key of where this event STARTS
data[31:33] uint16 BE ?? sample-count or total bytes (varies; not yet decoded)
data[33:35] uint16 BE ??
data[35] 0x46 record type (waveform full record)
…
```
`end_offset = (end_key[2] << 8) | end_key[3]` is **the authoritative event-end pointer**.
SFM must extract this from the first A5 response and use it to bound the chunk loop and
encode the TERM frame. The device will happily respond to chunk requests past `end_offset`
(returning post-event circular-buffer contents) — that's the over-read bug.
Verified across 3 events:
| Capture | start_key | end_key | end_offset | event size |
|---|---|---|---|---|
| 4-27-26 "open 2sec" / "copy event to disk" | `01110000` | `01111ABE` | `0x1ABE` | 6,846 B |
| 5-1-26 "copy 3sec" / Download All event 1 | `01110000` | `011121F2` | `0x21F2` | 8,690 B |
| 5-1-26 "copy 2nd address" / DA event 2 | `011121F2` | `0111417E` | `0x417E` (event 2 span 0x1F8C = 8,076 B) |
### SUB 5A — TERM frame formula (FINALIZED 2026-05-01)
The TERM frame fetches the partial last chunk *and* the file footer. It is **not** a simple
"goodbye" frame — its response payload contains the bytes between the last full 0x0200-aligned
chunk and `end_offset`, and is required for reconstructing the Blastware file format.
```
last_chunk_counter = address of last full 0x0200-byte chunk read
next_boundary = last_chunk_counter + 0x0200
TERM offset_word = end_offset - next_boundary
TERM params[0] = key[0] (= 0x01 on every observed device)
TERM params[1] = key[1] (= 0x11)
TERM params[2] = (next_boundary >> 8) & 0xFF
TERM params[3] = next_boundary & 0xFF
TERM params[4:10] = zeros
build_5a_frame(offset_word, params) (10-byte params, NOT 11)
```
The device reconstructs `requested_address = (params[2] << 8) | offset_word = end_offset`
and replies with `(end_offset - next_boundary)` bytes from `next_boundary` — the residual
between the last 0x0200 boundary and the actual event end. Append the TERM response data
to the chunk stream like any other A5 frame; it carries the final waveform tail + footer.
Verified across 3 events:
| end_offset | last chunk | next_boundary | TERM offset_word | TERM params[2:4] |
|---|---|---|---|---|
| `0x1ABE` | `0x1800` | `0x1A00` | `0x00BE` ✓ | `1A 00` ✓ |
| `0x21F2` | `0x1E00` | `0x2000` | `0x01F2` ✓ | `20 00` ✓ |
| `0x417E` | `0x3E38` | `0x4038` | `0x0146` ✓ | `40 38` ✓ |
The previous code's hard-coded `offset_word = 0x005A` and `term_counter = last + 0x0400`
are wrong; the device's response under that path is a tiny 101-byte device-side terminator
(arrived only after we walked the entire post-event buffer), not the proper file footer.
### SUB 5A — fixed metadata pages 0x1002 and 0x1004 (NEW 2026-05-01)
Two chunk addresses are GLOBAL device/session metadata, not event-specific:
- `counter=0x1002` — first metadata page
- `counter=0x1004` — second metadata page
These are at fixed absolute addresses in the device's flash buffer. They contain the
session-start compliance setup (Project/Client/User Name/Seis Loc/Extended Notes ASCII
strings) that A5 frame 7 used to be the source for in the old "0x0400-step" walk. In the
new walk these strings come from the dedicated metadata pages, not from the sample-chunk
stream.
BW reads them ONCE per Blastware session (during event 1's download) and caches them.
For SFM, that means:
- Once per call-home / once per `MiniMateClient.connect()` is enough.
- Subsequent events in the same session don't need to re-fetch them.
- Their content does not change when iterating events; only when the user opens
Compliance Setup → Apply on the device or sends a SUB 71 compliance write.
The contents have not been byte-for-byte decoded yet — first task on the implementation
side is to dump 0x1002 + 0x1004 from a fresh capture and verify they include all the
strings we currently extract from A5[7].
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
@@ -148,10 +275,16 @@ chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400).
confirmed from the BW wire capture. `bulk_waveform_term_params()` returns 10 bytes.
Do not swap them.
### SUB 5A — event-time metadata lives in A5 frame 7
### SUB 5A — event-time metadata source (UPDATED 2026-05-01)
The bulk stream sends 9+ A5 response frames. Frame 7 (0-indexed) contains the compliance
setup as it existed when the event was recorded:
> **Old understanding (deprecated):** the metadata strings live in "A5 frame 7" of the 5A
> bulk stream. This was a side-effect of the old `0x0400`-step walk: the sample-chunk at
> counter ≈ 0x1400 would happen to include the global 0x1002/0x1004 metadata pages because
> the broken counter formula was scanning the wrong region.
>
> **New understanding:** the metadata strings live at fixed counter addresses `0x1002` and
> `0x1004`. See "SUB 5A — fixed metadata pages 0x1002 and 0x1004" above. The 5A
> sample-chunk stream itself does NOT contain these strings any more under the new walk.
```
"Project:" → project description
@@ -171,26 +304,37 @@ used as the authoritative source. `_decode_a5_metadata_into` therefore only set
"Client:", "User Name:", "Seis Loc:", and "Extended Notes" are **NOT** present in the 0C
record — 5A remains the sole source for those fields and they are set unconditionally.
`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears,
then sends the termination frame.
> ⚠️ `stop_after_metadata=True` (which scans for `b"Project:"` in the chunk stream and
> stops one chunk later) is a workaround for the missing end_offset bound — when the new
> STRT-bounded walk lands, this knob becomes obsolete. The proper "stop" condition is
> `next_chunk_counter >= end_offset & 0xFE00`, with the partial tail fetched by the TERM
> frame.
### SUB 5A — end-of-stream signal (confirmed 2026-04-06)
### SUB 5A — end-of-stream — UPDATED 2026-05-01
After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to
the next chunk request, then goes silent. This is the natural end-of-stream indicator — NOT
a complete A5 frame. `S3FrameParser.bytes_fed` will be 1; no frame is assembled.
> **Previous understanding (now known to be a symptom, not a feature):** "After streaming
> all waveform chunks, the device sends exactly **1 raw byte** then goes silent." This was
> not the device's natural end-of-event signal — it was the device's response when SFM had
> walked clean off the end of the addressable buffer region after over-reading by ~5×.
> Under the corrected walk (chunks bounded by `end_offset` from STRT, terminated with the
> proper TERM frame), the stream ends cleanly: TERM request → TERM response (`page=0x0000`,
> sized to the residual `end_offset - next_boundary`). No timeout, no 1-byte teaser.
Handling: on `TimeoutError`, if `bytes_fed > 0` AND frames were already collected, treat as
graceful end-of-stream, break the loop, and proceed to the termination frame. If `bytes_fed
== 0` with no prior frames, it is a genuine transport failure — re-raise.
The `bytes_fed=1 → graceful end` heuristic in `read_bulk_waveform_stream` is still a useful
defence-in-depth fallback for malformed events or unexpected device states, but should not
be the primary loop-exit condition.
**Chunk recv timeout must be 10 s, not the default 120 s.** Chunks arrive within ~1 s each.
Using 120 s causes a ~2-minute stall at every end-of-stream detection. The `_recv_one` call
in the chunk loop passes `timeout=10.0` explicitly.
**Typical chunk count (BE11529, 1024 sps):** A 9,306-sample event produces 35 chunks before
end-of-stream. Chunks with uniform 1,036-byte data are all-zero ADC samples (post-event
silence). Only the initial variable-size chunks contain actual signal.
**Typical chunk count under the corrected walk (BE11529, 1024 sps over TCP/cellular):**
A 2-sec event takes 12 sample chunks + 2 metadata pages (event 1) + TERM = ~15 frames.
A 3-sec event takes 16 sample chunks + 2 metadata pages + TERM = ~19 frames.
An 8 KB event 2 (continuation) takes 15 sample chunks + TERM = ~16 frames.
Compare to the old over-read walk: same 2-sec event was producing 37 chunks, with chunks
17-37 containing post-event circular-buffer garbage that corrupted the file body.
### SUB 5A — fi==9 hardcoded skip (FIXED 2026-04-06)
@@ -303,6 +447,55 @@ sends token=0xFE and is NOT used by any caller.
`advance_event()` returns `(key4, event_data8)`.
Callers (`count_events`, `get_events`) loop while `data8[4:8] != b"\x00\x00\x00\x00"`.
### SUB 0A — WAVEHDR response length distinguishes events from boundaries (NEW 2026-05-01)
When iterating events with the "Download All" pattern (1E → 0A → 1F → 0A → 1F → …), the
DATA_LENGTH at `data_rsp.data[5]` (= the byte BW echoes back as the offset for the data
fetch step) takes one of two values:
| WAVEHDR offset | Meaning |
|---|---|
| `0x46` (= 70) | Real event start key — there is event data at this address |
| `0x2C` (= 44) | Boundary marker between events — this key is the END of the previous event AND the START key for the empty space after it (or is the next event's pre-header) |
Confirmed from the 5-1-26 "Download All" capture:
```
0A(key=01110000) → off=0x46 ← event 1 real start
1F → key=011121F2
0A(key=011121F2) → off=0x2C ← event 1 END / event 2 boundary
1F → key=01112238
0A(key=01112238) → off=0x46 ← event 2 real start (= boundary + 0x46)
1F → key=0111417E
0A(key=0111417E) → off=0x2C ← event 2 END / next-empty marker
1F → null sentinel
```
This is why event 2's first 5A chunk is at `start_key + 0x46` — that's the address of the
"real start" 0x46-record, distinct from the `0x2C`-record at the raw boundary. Use the
`0x46` keys as the input to `read_bulk_waveform_stream`, not the `0x2C` keys.
For event 1 only (start_key[2:4] = 0x0000) BW probes at counter=0x0000 directly, which is
the `0x46`-keyed start record. Subsequent events use `start_key + 0x46`.
**Practical iteration pattern (replaces the old 1E/1F walk for downloads):**
```
Setup: SERIAL × 2 → CHCFG → 1E (token=0x00) → key0
For each event:
0A(cur_key) → DATA_LENGTH = 0x46 (real) or 0x2C (boundary)
1F (token=0x00) → next_key
if length was 0x46: → cur_key is a real event; queue it for download
cur_key = next_key
if next_key all-zero null sentinel: stop
Then for each queued real-event key:
download_event(key) → 5A bulk stream with STRT-bounded chunk walk
```
This is what BW does in the 5-1-26 "Download All" capture — it walks the full event chain
collecting `(key, length)` tuples first, *then* downloads each event using the `0x46` keys.
### SUB 1A — compliance config — orphaned send bug (FIXED, do not re-introduce)
`read_compliance_config()` sends a 4-frame sequence (A, B, C, D) where:
@@ -527,6 +720,8 @@ All DB endpoints are read-only except `PATCH /db/events/{id}/false_trigger`.
| 3-11-26 | `bridges/captures/3-11-26/` | Full compliance setup write, Aux Trigger capture |
| 3-31-26 | `bridges/captures/3-31-26/` | Complete event download cycle (148 BW / 147 S3 frames) — confirmed 1E/0A/0C/1F sequence; only 1 event stored so token=0xFE appeared to work |
| 4-3-26 | `bridges/captures/4-3-26/` | Browse-mode S3 capture with 2+ events — confirmed all-zero params for 1F, 1F response layout, null sentinel, 0A context requirement |
| 4-27-26 | `bridges/captures/4-27-26/` | BW "open 2sec waveform" + "copy event to disk" + paired SFM "seismo_dl" — first proof that SFM was over-reading 5× past event end. BW reads 14 chunks at 0x0200 increments + TERM at end_offset; SFM was reading 37 chunks at 0x0400 increments. STRT end_key field located. |
| 5-1-26 | `bridges/captures/5-1-26/comcheck/` | Three sub-captures: SFM 3-sec download (`seismo_dl_…`), BW comms-check + 3-sec download (`bwcap3sec/`), BW second-event download + "Download All" (`raw_*_170945`/`_171216`). Confirmed: TERM frame formula across 3 events; metadata pages 0x1002/0x1004 are global (read once per session); event-1 vs event-N chunk-pattern split; WAVEHDR length 0x46 vs 0x2C disambiguates real events from boundaries. |
---