Compare commits
7 Commits
dev
...
c5a7914032
| Author | SHA1 | Date | |
|---|---|---|---|
| c5a7914032 | |||
| dbb9febe2c | |||
| 9ae968b108 | |||
| 171dc2551c | |||
| 4f4c1a8f64 | |||
| 0da88ec6aa | |||
| edb4698bfb |
@@ -163,6 +163,83 @@ record — 5A remains the sole source for those fields and they are set uncondit
|
||||
`stop_after_metadata=True` (default) stops the 5A loop as soon as `b"Project:"` appears,
|
||||
then sends the termination frame.
|
||||
|
||||
### SUB 5A — STRT record layout and rectime_seconds (CORRECTED 2026-04-14)
|
||||
|
||||
The STRT record is 21 bytes embedded at the start of A5[0] data. Offsets relative to
|
||||
the `b'STRT'` magic bytes:
|
||||
|
||||
```
|
||||
+0..3 b'STRT' magic
|
||||
+4..5 flags 0xFF 0xFE (single-shot) or 0xFF 0xFD (continuous)
|
||||
+6..9 next_key4 ← key of the NEXT stored event (NOT the current event) ← confirmed 2026-04-14
|
||||
+10..13 prev_key4 ← key of the PREVIOUS stored event ← confirmed 2026-04-14
|
||||
+14..15 UNKNOWN (values seen: 0xDA63=55907, 0xF38F=62351, 0x5685=22149) — NOT total_samples
|
||||
+16..17 UNKNOWN (values seen: 0x0122=290, 0x011A=282, 0x00FA=250) — NOT pretrig_samples
|
||||
+18 uint8 record-MODE byte — NOT rectime in seconds
|
||||
+19..20 0x00 0x00
|
||||
```
|
||||
|
||||
**CONFIRMED field values (2026-04-14) from 3 desk-thump events, firmware S338.17:**
|
||||
|
||||
| Field | What it is | Confirmed |
|
||||
|---|---|---|
|
||||
| +4..5 | 0xFFFE single-shot / 0xFFFD continuous | ✅ |
|
||||
| +6..9 | next_event_key (NOT current) | ✅ 3 events |
|
||||
| +10..13 | prev_event_key | ✅ 3 events |
|
||||
| +18 | mode byte: 0x46 ('F') = single-shot, 0x0E = continuous | ✅ |
|
||||
|
||||
**CONFIRMED (2026-04-14) — total_samples and pretrig_samples are NOT stored in the STRT record.**
|
||||
The prior documented offsets (+14..15 for total_samples, +16..17 for pretrig_samples) were
|
||||
WRONG — confirmed by cross-checking STRT-derived rectime against compliance record_time
|
||||
(4-14-26): all 4 events give STRT-derived rectime of 21–61 s vs actual 3.0 s (ratio 7–20×).
|
||||
Extending the STRT dump to 32 bytes confirmed that bytes 21+ are the start of the raw ADC
|
||||
waveform samples, not more STRT fields. Blastware itself derives total_samples and
|
||||
pretrig_samples from the compliance config — exactly what our fallback does.
|
||||
|
||||
**The compliance-config fallback IS the correct permanent solution, not a workaround.**
|
||||
`_decode_a5_waveform` uses:
|
||||
- `pretrig_samples = round(0.25 × sample_rate)` (compliance monitoring standard)
|
||||
- `total_samples = pretrig_samples + round(record_time × sample_rate)`
|
||||
|
||||
**CONFIRMED (2026-04-14) — waveform starts at strt_pos + 21 (no preamble).**
|
||||
The original `sp + 27` skip (STRT 21B + null-pad 2B + 0xFF-sentinel 4B) was WRONG.
|
||||
The 6-byte "preamble" in the 4-2-26 blast capture (`00 00 ff ff ff ff`) was actually the
|
||||
first ~0.75 sample-sets of quiet pre-trigger ADC data misread as padding. Desk-thump
|
||||
events show different bytes at positions 21-26 (e.g. `00 10 02 00 ff fc`) — they are real
|
||||
ADC readings, not a fixed preamble. The `sp + 27` skip discarded 6 bytes of real waveform
|
||||
data and misaligned the channel decode for all subsequent frames. Fixed: `wave = w[sp+21:]`.
|
||||
|
||||
The +6..9 next_key and +10..13 prev_key fields are confirmed across 4 events including the
|
||||
first-event-after-erase case (prev_key = self-reference `01110000`; next_key = device
|
||||
pre-allocates the predicted next slot even before any second event exists).
|
||||
|
||||
**CRITICAL — strt[18] is a record-mode byte, NOT rectime_seconds (confirmed 2026-04-14):**
|
||||
Analysis of 15 distinct STRT records across the 4-9-26 ACH capture shows:
|
||||
- `flags=0xFFFE` (single-shot) → `strt[18] = 0x46` ('F') for EVERY event regardless of duration
|
||||
- `flags=0xFFFD` (continuous) → `strt[18] = 0x0E` for EVERY event regardless of duration
|
||||
|
||||
Do NOT use `strt[18]` for rectime.
|
||||
|
||||
**Pre-trigger time is separate from record_time (confirmed 2026-04-14):**
|
||||
Blastware documentation states: "The default Time Scale is -0.25 second to 1 second — this
|
||||
negative number accounts for the pre-trigger set for compliance monitoring." Therefore:
|
||||
- `record_time` (3.0 s) is POST-TRIGGER duration only
|
||||
- Pre-trigger = 0.25 s = 256 samples at 1024 sps (compliance monitoring standard default)
|
||||
- The pre-trigger field has NOT yet been located in the raw compliance config bytes
|
||||
- `_decode_a5_waveform` falls back to pretrig = 0.25 × sr from compliance standard
|
||||
- TODO: locate pretrig_time offset in ComplianceConfig — search around anchor or channel blocks
|
||||
|
||||
The device bulk-streams zero-padded frames BEYOND the configured record window. The
|
||||
viewer clips `displayCount = total_samples = pretrig + post_trig` to exclude this padding.
|
||||
|
||||
**Validity checks in `_decode_a5_waveform`:**
|
||||
Check 1: `pretrig_samples >= total_samples` → invalid (original check).
|
||||
Check 2: STRT-derived rectime differs from `compliance_config.record_time` by more than 2×
|
||||
→ invalid. Both failures fall back to the compliance-config derived values.
|
||||
`_decode_a5_waveform` logs `raw strt[0:21]` at WARNING level on any failure.
|
||||
Observed once (2026-04-14) with `strt[16:18] = 0x41 0x01` → pretrig=16641 (impossible).
|
||||
Root cause not yet identified — capture the warning log hex dump to diagnose.
|
||||
|
||||
### SUB 5A — end-of-stream signal (confirmed 2026-04-06)
|
||||
|
||||
After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to
|
||||
|
||||
+59
-1
@@ -371,6 +371,7 @@ class AchSession:
|
||||
full_waveform=True,
|
||||
stop_after_index=stop_idx,
|
||||
skip_waveform_for_keys=seen_keys if seen_keys else None,
|
||||
compliance_config=device_info.compliance_config if device_info else None,
|
||||
)
|
||||
|
||||
# Filter to events whose keys we haven't saved before.
|
||||
@@ -445,8 +446,19 @@ class AchSession:
|
||||
# ── Persist to SQLite DB ─────────────────────────────────────
|
||||
_session_start = datetime.datetime.now()
|
||||
try:
|
||||
# Build waveform blobs for events that have full raw_samples
|
||||
_waveform_blobs = {}
|
||||
for _ev in new_events:
|
||||
if _ev._waveform_key and _ev.raw_samples:
|
||||
_blob = _build_waveform_blob(_ev)
|
||||
if _blob:
|
||||
_waveform_blobs[_ev._waveform_key.hex()] = _blob
|
||||
if _waveform_blobs:
|
||||
log.info(" [DB] waveform blobs prepared: %d", len(_waveform_blobs))
|
||||
|
||||
_ev_ins, _ev_skip = self.db.insert_events(
|
||||
new_events, serial=serial or self.peer, session_id=None
|
||||
new_events, serial=serial or self.peer, session_id=None,
|
||||
waveform_blobs=_waveform_blobs,
|
||||
)
|
||||
_ml_ins, _ml_skip = self.db.insert_monitor_log(
|
||||
new_monitor_entries, session_id=None
|
||||
@@ -599,6 +611,52 @@ def _event_to_dict(e: Event) -> dict:
|
||||
}
|
||||
|
||||
|
||||
def _build_waveform_blob(e: Event) -> Optional[str]:
|
||||
"""
|
||||
Serialise a downloaded event's full waveform data as a JSON string for
|
||||
storage in the DB waveform_blob column.
|
||||
|
||||
Returns the same shape as GET /device/event/{index}/waveform so the
|
||||
waveform viewer can consume either source without modification.
|
||||
Returns None if the event has no raw_samples (e.g. metadata-only download).
|
||||
"""
|
||||
raw = e.raw_samples or {}
|
||||
if not raw:
|
||||
return None
|
||||
|
||||
pv = e.peak_values
|
||||
peak_values = None
|
||||
if pv:
|
||||
peak_values = {
|
||||
"tran": pv.tran,
|
||||
"vert": pv.vert,
|
||||
"long": pv.long,
|
||||
"micl_psi": pv.micl,
|
||||
"peak_vector_sum": pv.peak_vector_sum,
|
||||
}
|
||||
|
||||
ts = e.timestamp
|
||||
timestamp_str = (
|
||||
f"{ts.year:04d}-{ts.month:02d}-{ts.day:02d}T"
|
||||
f"{ts.hour:02d}:{ts.minute:02d}:{ts.second:02d}"
|
||||
if ts else None
|
||||
)
|
||||
|
||||
blob = {
|
||||
"index": e.index,
|
||||
"record_type": e.record_type,
|
||||
"timestamp": timestamp_str,
|
||||
"total_samples": e.total_samples,
|
||||
"pretrig_samples": e.pretrig_samples,
|
||||
"rectime_seconds": e.rectime_seconds,
|
||||
"samples_decoded": len(raw.get("Tran", [])),
|
||||
"sample_rate": e.sample_rate,
|
||||
"peak_values": peak_values,
|
||||
"channels": raw,
|
||||
}
|
||||
return json.dumps(blob)
|
||||
|
||||
|
||||
def _monitor_log_entry_to_dict(e: MonitorLogEntry) -> dict:
|
||||
return {
|
||||
"key": e.key,
|
||||
|
||||
+143
-29
@@ -448,7 +448,7 @@ class MiniMateClient:
|
||||
proto.confirm_erase_all()
|
||||
log.info("delete_all_events: erase confirmed — device memory cleared")
|
||||
|
||||
def get_events(self, full_waveform: bool = False, debug: bool = False, stop_after_index: Optional[int] = None, skip_waveform_for_keys: Optional[set] = None) -> list[Event]:
|
||||
def get_events(self, full_waveform: bool = False, debug: bool = False, stop_after_index: Optional[int] = None, skip_waveform_for_keys: Optional[set] = None, compliance_config: Optional["ComplianceConfig"] = None) -> list[Event]:
|
||||
"""
|
||||
Download all stored events from the device using the confirmed
|
||||
1E → 0A → 0C → 5A → 1F event-iterator protocol.
|
||||
@@ -479,6 +479,7 @@ class MiniMateClient:
|
||||
ProtocolError: on unrecoverable communication failure.
|
||||
"""
|
||||
proto = self._require_proto()
|
||||
_compliance_config = compliance_config # passed through to _decode_a5_waveform
|
||||
|
||||
log.info("get_events: requesting first event (SUB 1E)")
|
||||
try:
|
||||
@@ -603,12 +604,12 @@ class MiniMateClient:
|
||||
"get_events: 5A full waveform download for key=%s", cur_key.hex()
|
||||
)
|
||||
a5_frames = proto.read_bulk_waveform_stream(
|
||||
cur_key, stop_after_metadata=False, max_chunks=128
|
||||
cur_key, stop_after_metadata=False, max_chunks=2048
|
||||
)
|
||||
if a5_frames:
|
||||
a5_ok = True
|
||||
_decode_a5_metadata_into(a5_frames, ev)
|
||||
_decode_a5_waveform(a5_frames, ev)
|
||||
_decode_a5_waveform(a5_frames, ev, compliance_config=_compliance_config)
|
||||
log.info(
|
||||
"get_events: 5A decoded %d sample-sets",
|
||||
len((ev.raw_samples or {}).get("Tran", [])),
|
||||
@@ -1311,6 +1312,7 @@ def _decode_a5_metadata_into(frames_data: list[bytes], event: Event) -> None:
|
||||
def _decode_a5_waveform(
|
||||
frames_data: list[bytes],
|
||||
event: Event,
|
||||
compliance_config: Optional["ComplianceConfig"] = None,
|
||||
) -> None:
|
||||
"""
|
||||
Decode the raw 4-channel ADC waveform from a complete set of SUB 5A
|
||||
@@ -1336,15 +1338,22 @@ def _decode_a5_waveform(
|
||||
|
||||
── Frame structure ──────────────────────────────────────────────────────────
|
||||
A5[0] (probe response):
|
||||
db[7:] = [11-byte header] [21-byte STRT record] [6-byte preamble] [waveform ...]
|
||||
db[7:] = [11-byte header] [21-byte STRT record] [waveform ...]
|
||||
STRT: b'STRT' at offset 11, total 21 bytes
|
||||
+8 uint16 BE: total_samples (expected full-record sample-sets)
|
||||
+16 uint16 BE: pretrig_samples (pre-trigger sample count)
|
||||
+18 uint8: rectime_seconds (record duration)
|
||||
Preamble: 6 bytes after the STRT record (confirmed from 4-2-26 blast capture):
|
||||
bytes 21-22: 0x00 0x00 (null padding)
|
||||
bytes 23-26: 0xFF × 4 (sync sentinel / alignment marker)
|
||||
Waveform starts at strt_pos + 27 within db[7:].
|
||||
+4..5 0xFF 0xFE (single-shot) or 0xFF 0xFD (continuous)
|
||||
+6..9 next_event_key4 (device pre-allocates next slot)
|
||||
+10..13 prev_event_key4
|
||||
+14..15 UNKNOWN (not total_samples)
|
||||
+16..17 UNKNOWN (not pretrig_samples)
|
||||
+18 mode byte: 0x46 single-shot, 0x0E continuous
|
||||
+19..20 0x00 0x00
|
||||
Waveform starts immediately at strt_pos + 21 (no preamble).
|
||||
NOTE: The original 4-2-26 blast capture appeared to show a 6-byte
|
||||
"preamble" (00 00 ff ff ff ff) after the STRT record, but this was
|
||||
actually the first ~0.75 sample-sets of quiet pre-trigger ADC data
|
||||
misread as padding. Confirmed 2026-04-14: bytes 21+ are raw waveform.
|
||||
total_samples and pretrig_samples are NOT stored in the STRT record;
|
||||
they are derived from the compliance config (the correct permanent source).
|
||||
|
||||
A5[1..N] (chunk responses):
|
||||
db[7:] = [8-byte per-frame header] [waveform bytes ...]
|
||||
@@ -1378,27 +1387,48 @@ def _decode_a5_waveform(
|
||||
|
||||
# STRT record layout (21 bytes, offsets relative to b'STRT'):
|
||||
# +0..3 magic b'STRT'
|
||||
# +8..9 uint16 BE total_samples (full-record expected sample-set count)
|
||||
# +16..17 uint16 BE pretrig_samples
|
||||
# +18 uint8 rectime_seconds
|
||||
# +4..5 0xFF 0xFE (single-shot) or 0xFF 0xFD (continuous)
|
||||
# +6..9 next_event_key4 (NOT current key — device pre-allocates next)
|
||||
# +10..13 prev_event_key4
|
||||
# +14..15 UNKNOWN — confirmed NOT total_samples (confirmed 2026-04-14)
|
||||
# +16..17 UNKNOWN — confirmed NOT pretrig_samples (confirmed 2026-04-14)
|
||||
# +18 mode byte: 0x46='F' single-shot, 0x0E continuous — NOT rectime
|
||||
# +19..20 0x00 0x00
|
||||
# Bytes 21+ are raw ADC waveform samples — no preamble.
|
||||
# total_samples / pretrig_samples are NOT stored in STRT at all.
|
||||
# The compliance config fallback is the correct permanent source.
|
||||
strt = w0[strt_pos : strt_pos + 21]
|
||||
if len(strt) < 21:
|
||||
log.warning("_decode_a5_waveform: STRT record truncated (%dB)", len(strt))
|
||||
return
|
||||
|
||||
total_samples = struct.unpack_from(">H", strt, 8)[0]
|
||||
pretrig_samples = struct.unpack_from(">H", strt, 16)[0]
|
||||
rectime_seconds = strt[18]
|
||||
|
||||
event.total_samples = total_samples
|
||||
event.pretrig_samples = pretrig_samples
|
||||
event.rectime_seconds = rectime_seconds
|
||||
|
||||
log.debug(
|
||||
"_decode_a5_waveform: STRT total_samples=%d pretrig=%d rectime=%ds",
|
||||
total_samples, pretrig_samples, rectime_seconds,
|
||||
log.info(
|
||||
"_decode_a5_waveform: STRT raw[0:21]: %s",
|
||||
strt.hex(' '),
|
||||
)
|
||||
|
||||
# STRT bytes +14..17 are unknown fields — confirmed NOT total/pretrig_samples
|
||||
# (2026-04-14). total_samples and pretrig_samples are derived from the
|
||||
# compliance config, which is the correct permanent source.
|
||||
_strt_invalid = True
|
||||
_sample_rate_default = 1024
|
||||
total_samples = 0
|
||||
pretrig_samples = 0
|
||||
rectime_seconds = 0
|
||||
|
||||
log.info(
|
||||
"_decode_a5_waveform: STRT flags=0x%04X next_key=%s prev_key=%s "
|
||||
"mode=0x%02X → using compliance-config for total/pretrig",
|
||||
struct.unpack_from(">H", strt, 4)[0],
|
||||
strt[6:10].hex(),
|
||||
strt[10:14].hex(),
|
||||
strt[18],
|
||||
)
|
||||
|
||||
event.total_samples = 0 # will be overwritten by compliance-config fallback below
|
||||
event.pretrig_samples = 0
|
||||
event.rectime_seconds = 0
|
||||
|
||||
# ── Collect per-frame waveform bytes with global offset tracking ─────────
|
||||
# global_offset is the cumulative byte count across all frames, used to
|
||||
# compute the channel alignment at each frame boundary.
|
||||
@@ -1408,13 +1438,21 @@ def _decode_a5_waveform(
|
||||
for fi, db in enumerate(frames_data):
|
||||
w = db[7:]
|
||||
|
||||
# A5[0]: waveform begins after the 21-byte STRT record and 6-byte preamble.
|
||||
# Layout: STRT(21B) + null-pad(2B) + 0xFF sentinel(4B) = 27 bytes total.
|
||||
# A5[0]: waveform begins immediately after the 21-byte STRT record.
|
||||
# Confirmed 2026-04-14: there is NO preamble after STRT — bytes 21+
|
||||
# are raw ADC sample data. The earlier sp+27 skip was eating 6 bytes
|
||||
# of real waveform, misaligning the channel decode for all subsequent
|
||||
# frames.
|
||||
if fi == 0:
|
||||
sp = w.find(b"STRT")
|
||||
if sp < 0:
|
||||
continue
|
||||
wave = w[sp + 27 :]
|
||||
wave = w[sp + 21 :]
|
||||
log.info(
|
||||
"_decode_a5_waveform: A5[0] waveform starts at sp+21; "
|
||||
"first 24 wave bytes: %s",
|
||||
wave[:24].hex(' '),
|
||||
)
|
||||
|
||||
# Frame 7 carries event-time metadata strings ("Project:", "Client:", …)
|
||||
# and no waveform ADC data.
|
||||
@@ -1490,10 +1528,28 @@ def _decode_a5_waveform(
|
||||
|
||||
running_offset += len(wave)
|
||||
|
||||
n_decoded_total = len(tran)
|
||||
log.debug(
|
||||
"_decode_a5_waveform: decoded %d alignment-corrected sample-sets "
|
||||
"(skipped %d due to frame boundary misalignment)",
|
||||
len(tran), n_sets - len(tran),
|
||||
n_decoded_total, n_sets - n_decoded_total,
|
||||
)
|
||||
# Log first 16 and last 8 samples for every channel — essential for
|
||||
# validating decoder output and diagnosing flatline / misalignment issues.
|
||||
_N = min(16, n_decoded_total)
|
||||
_L = max(0, n_decoded_total - 8)
|
||||
log.info(
|
||||
"_decode_a5_waveform: first %d samples — "
|
||||
"Tran=%s Vert=%s Long=%s Mic=%s",
|
||||
_N,
|
||||
tran[:_N], vert[:_N], long_[:_N], mic[:_N],
|
||||
)
|
||||
if n_decoded_total > 16:
|
||||
log.info(
|
||||
"_decode_a5_waveform: last 8 samples (idx %d–%d) — "
|
||||
"Tran=%s Vert=%s Long=%s Mic=%s",
|
||||
_L, n_decoded_total - 1,
|
||||
tran[_L:], vert[_L:], long_[_L:], mic[_L:],
|
||||
)
|
||||
|
||||
event.raw_samples = {
|
||||
@@ -1503,6 +1559,64 @@ def _decode_a5_waveform(
|
||||
"Mic": mic,
|
||||
}
|
||||
|
||||
# ── Derive pretrig/total from compliance config (always — STRT doesn't store them) ──
|
||||
# total_samples and pretrig_samples are not stored in the STRT record (confirmed
|
||||
# 2026-04-14). They are derived from compliance config record_time × sample_rate.
|
||||
# _strt_invalid is always True; this block always runs.
|
||||
#
|
||||
# Formula:
|
||||
# post_trig = record_time (s) × sample_rate (sps)
|
||||
# pretrig = decoded_samples − post_trig
|
||||
#
|
||||
# This gives the pre-trigger window length, which correctly places t=0 in the
|
||||
# waveform. If compliance_config is not available, leave pretrig=0 (viewer shows
|
||||
# full waveform starting at t=0 — better than a crash or garbage).
|
||||
n_decoded = len(tran)
|
||||
if _strt_invalid:
|
||||
if compliance_config is not None:
|
||||
cc_sr = compliance_config.sample_rate or 1024
|
||||
cc_rt = compliance_config.record_time
|
||||
# Pre-trigger time is a separate device setting from Record Time.
|
||||
# Blastware documentation confirms the compliance monitoring standard
|
||||
# is 0.25 seconds pre-trigger ("the default Time Scale is -0.25 to 1
|
||||
# second — this negative number accounts for the pre-trigger set for
|
||||
# compliance monitoring").
|
||||
# The pre-trigger field has not yet been located in the raw compliance
|
||||
# config bytes; 0.25 s is used as the best-known default until it is
|
||||
# decoded. TODO: locate pretrig_time in ComplianceConfig bytes.
|
||||
_PRETRIG_SECONDS_DEFAULT = 0.25
|
||||
derived_pretrig = int(round(_PRETRIG_SECONDS_DEFAULT * cc_sr))
|
||||
if cc_rt and cc_rt > 0:
|
||||
post_trig_samples = int(round(cc_rt * cc_sr))
|
||||
# Clip total to pretrig + post_trig so the viewer doesn't show the
|
||||
# zero-padded tail frames the device appends beyond the record window.
|
||||
event.total_samples = derived_pretrig + post_trig_samples
|
||||
event.pretrig_samples = derived_pretrig
|
||||
event.rectime_seconds = int(round(cc_rt))
|
||||
log.info(
|
||||
"_decode_a5_waveform: pretrig=%d (%.2fs compliance default) "
|
||||
"post_trig=%d total=%d record_time=%.1fs "
|
||||
"sr=%d sps decoded=%d samples",
|
||||
derived_pretrig, _PRETRIG_SECONDS_DEFAULT,
|
||||
post_trig_samples, event.total_samples,
|
||||
cc_rt, cc_sr, n_decoded,
|
||||
)
|
||||
else:
|
||||
event.total_samples = n_decoded
|
||||
event.pretrig_samples = derived_pretrig
|
||||
log.warning(
|
||||
"_decode_a5_waveform: STRT invalid, compliance config missing "
|
||||
"record_time — pretrig=%d (%.2fs default), total=decoded %d",
|
||||
derived_pretrig, _PRETRIG_SECONDS_DEFAULT, n_decoded,
|
||||
)
|
||||
else:
|
||||
event.total_samples = n_decoded
|
||||
log.warning(
|
||||
"_decode_a5_waveform: STRT invalid, no compliance config available "
|
||||
"— pretrig left as 0, total set to decoded count %d",
|
||||
n_decoded,
|
||||
)
|
||||
|
||||
|
||||
def _extract_record_type(data: bytes) -> Optional[str]:
|
||||
"""
|
||||
|
||||
+40
-2
@@ -81,6 +81,7 @@ CREATE TABLE IF NOT EXISTS events (
|
||||
sample_rate INTEGER,
|
||||
record_type TEXT, -- "single_shot" | "continuous"
|
||||
false_trigger INTEGER NOT NULL DEFAULT 0, -- 0=no, 1=yes (manual flag)
|
||||
waveform_blob TEXT, -- JSON waveform response (channels + metadata)
|
||||
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
|
||||
UNIQUE(serial, timestamp)
|
||||
);
|
||||
@@ -216,6 +217,17 @@ class SeismoDb:
|
||||
""")
|
||||
log.info("_migrate: monitor_log table rebuilt OK")
|
||||
|
||||
# Migration 3: add waveform_blob column to events (nullable TEXT).
|
||||
# ALter TABLE ADD COLUMN is safe in SQLite for nullable columns — no rebuild needed.
|
||||
col_names = {
|
||||
row[1]
|
||||
for row in conn.execute("PRAGMA table_info(events)").fetchall()
|
||||
}
|
||||
if "waveform_blob" not in col_names:
|
||||
log.info("_migrate: adding waveform_blob column to events")
|
||||
conn.execute("ALTER TABLE events ADD COLUMN waveform_blob TEXT")
|
||||
log.info("_migrate: waveform_blob column added OK")
|
||||
|
||||
@staticmethod
|
||||
def _iso(dt: Optional[datetime.datetime]) -> Optional[str]:
|
||||
return dt.isoformat() if dt is not None else None
|
||||
@@ -282,12 +294,19 @@ class SeismoDb:
|
||||
*,
|
||||
serial: str,
|
||||
session_id: Optional[str] = None,
|
||||
waveform_blobs: Optional[dict[str, str]] = None,
|
||||
) -> tuple[int, int]:
|
||||
"""
|
||||
Insert triggered events. Silently skips duplicates (serial+timestamp).
|
||||
Returns (inserted, skipped).
|
||||
|
||||
waveform_blobs: optional mapping of waveform_key (hex str) → JSON string
|
||||
containing the full waveform response (channels + metadata). When provided,
|
||||
the blob is stored alongside the event row and is retrievable via
|
||||
GET /db/events/{id}/waveform.
|
||||
"""
|
||||
inserted = skipped = 0
|
||||
blobs = waveform_blobs or {}
|
||||
with self._connect() as conn:
|
||||
for ev in events:
|
||||
key = ev._waveform_key.hex() if ev._waveform_key else None
|
||||
@@ -307,6 +326,7 @@ class SeismoDb:
|
||||
|
||||
pv = ev.peak_values
|
||||
pi = ev.project_info
|
||||
blob = blobs.get(key)
|
||||
|
||||
try:
|
||||
conn.execute(
|
||||
@@ -315,8 +335,8 @@ class SeismoDb:
|
||||
(id, serial, waveform_key, session_id, timestamp,
|
||||
tran_ppv, vert_ppv, long_ppv, peak_vector_sum, mic_ppv,
|
||||
project, client, operator, sensor_location,
|
||||
sample_rate, record_type)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
sample_rate, record_type, waveform_blob)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
self._new_id(), serial, key, session_id, ts,
|
||||
@@ -331,6 +351,7 @@ class SeismoDb:
|
||||
pi.sensor_location if pi else None,
|
||||
ev.sample_rate,
|
||||
ev.record_type,
|
||||
blob,
|
||||
),
|
||||
)
|
||||
inserted += 1
|
||||
@@ -387,6 +408,23 @@ class SeismoDb:
|
||||
)
|
||||
return cur.rowcount > 0
|
||||
|
||||
def get_event_waveform(self, event_id: str) -> tuple[bool, Optional[str]]:
|
||||
"""
|
||||
Return (found, waveform_blob) for a given event UUID.
|
||||
|
||||
found=False means the event row doesn't exist.
|
||||
found=True, blob=None means the event exists but has no stored waveform
|
||||
(e.g. downloaded before waveform storage was implemented).
|
||||
found=True, blob=<str> means the full waveform JSON is available.
|
||||
"""
|
||||
with self._connect() as conn:
|
||||
row = conn.execute(
|
||||
"SELECT waveform_blob FROM events WHERE id = ?", (event_id,)
|
||||
).fetchone()
|
||||
if row is None:
|
||||
return False, None
|
||||
return True, row["waveform_blob"]
|
||||
|
||||
# ── Monitor log ───────────────────────────────────────────────────────────
|
||||
|
||||
def insert_monitor_log(
|
||||
|
||||
+50
-2
@@ -662,7 +662,11 @@ def device_event_waveform(
|
||||
with _build_client(port, baud, host, tcp_port, timeout=120.0) as client:
|
||||
info = client.connect()
|
||||
# stop_after_index avoids downloading events beyond the one requested.
|
||||
events = client.get_events(full_waveform=True, stop_after_index=index)
|
||||
events = client.get_events(
|
||||
full_waveform=True,
|
||||
stop_after_index=index,
|
||||
compliance_config=info.compliance_config if info else None,
|
||||
)
|
||||
matching = [ev for ev in events if ev.index == index]
|
||||
return matching[0] if matching else None, info
|
||||
ev, info = _run_with_retry(_do, is_tcp=_is_tcp(host))
|
||||
@@ -689,13 +693,22 @@ def device_event_waveform(
|
||||
if sample_rate is None and info.compliance_config:
|
||||
sample_rate = info.compliance_config.sample_rate
|
||||
|
||||
# Recompute rectime_seconds using the actual sample rate now that we have it.
|
||||
# _decode_a5_waveform used 1024 sps as default; override if device says otherwise.
|
||||
# strt[18] is a record-mode byte (0x46 / 0x0E), NOT rectime in seconds.
|
||||
rectime_seconds = ev.rectime_seconds
|
||||
if (ev.total_samples is not None and ev.pretrig_samples is not None
|
||||
and sample_rate and sample_rate > 0):
|
||||
post_trig = max(0, ev.total_samples - ev.pretrig_samples)
|
||||
rectime_seconds = round(post_trig / sample_rate, 2)
|
||||
|
||||
result = {
|
||||
"index": ev.index,
|
||||
"record_type": ev.record_type,
|
||||
"timestamp": _serialise_timestamp(ev.timestamp),
|
||||
"total_samples": ev.total_samples,
|
||||
"pretrig_samples": ev.pretrig_samples,
|
||||
"rectime_seconds": ev.rectime_seconds,
|
||||
"rectime_seconds": rectime_seconds,
|
||||
"samples_decoded": samples_decoded,
|
||||
"sample_rate": sample_rate,
|
||||
"peak_values": _serialise_peak_values(ev.peak_values),
|
||||
@@ -1006,6 +1019,41 @@ def db_set_false_trigger(
|
||||
return {"status": "ok", "event_id": event_id, "false_trigger": value}
|
||||
|
||||
|
||||
@app.get("/db/events/{event_id}/waveform")
|
||||
def db_event_waveform(event_id: str) -> dict:
|
||||
"""
|
||||
Return the stored waveform blob for a DB event.
|
||||
|
||||
The response shape is identical to GET /device/event/{index}/waveform so the
|
||||
waveform viewer can consume either source without modification:
|
||||
- total_samples, pretrig_samples, rectime_seconds, samples_decoded
|
||||
- sample_rate
|
||||
- peak_values (tran, vert, long, micl_psi, peak_vector_sum)
|
||||
- channels ({"Tran": [...], "Vert": [...], "Long": [...], "Mic": [...]})
|
||||
|
||||
Returns 404 if the event doesn't exist, 422 if the event exists but has no
|
||||
stored waveform (downloaded before waveform storage was implemented).
|
||||
"""
|
||||
import json as _json
|
||||
db = _get_db()
|
||||
found, blob_str = db.get_event_waveform(event_id)
|
||||
if not found:
|
||||
raise HTTPException(status_code=404, detail=f"Event {event_id} not found")
|
||||
if blob_str is None:
|
||||
raise HTTPException(
|
||||
status_code=422,
|
||||
detail=(
|
||||
f"Event {event_id} has no stored waveform. "
|
||||
"Waveform storage requires ACH server v0.11+. "
|
||||
"Re-download the event from the device to backfill."
|
||||
),
|
||||
)
|
||||
try:
|
||||
return _json.loads(blob_str)
|
||||
except Exception as exc:
|
||||
raise HTTPException(status_code=500, detail=f"Waveform blob corrupt: {exc}") from exc
|
||||
|
||||
|
||||
@app.get("/db/monitor_log")
|
||||
def db_monitor_log(
|
||||
serial: Optional[str] = Query(None, description="Filter by unit serial"),
|
||||
|
||||
@@ -548,6 +548,18 @@
|
||||
.ft-toggle-btn:hover { border-color: var(--red); color: var(--red); }
|
||||
.ft-toggle-btn.flagged { border-color: var(--red); color: var(--red); background: rgba(248,81,73,0.1); }
|
||||
|
||||
.wf-btn {
|
||||
background: none;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
color: var(--accent);
|
||||
cursor: pointer;
|
||||
font-size: 13px;
|
||||
padding: 1px 6px;
|
||||
line-height: 1;
|
||||
}
|
||||
.wf-btn:hover { background: rgba(56,139,253,0.15); border-color: var(--accent); }
|
||||
|
||||
.db-empty {
|
||||
color: var(--text-mute);
|
||||
font-size: 13px;
|
||||
@@ -921,6 +933,7 @@
|
||||
<table class="db-table" id="hist-table">
|
||||
<thead>
|
||||
<tr>
|
||||
<th></th>
|
||||
<th>Timestamp</th>
|
||||
<th>Serial</th>
|
||||
<th>Tran (in/s)</th>
|
||||
@@ -1744,7 +1757,9 @@ async function loadHistory() {
|
||||
const tr = document.createElement('tr');
|
||||
const pvs = ev.peak_vector_sum;
|
||||
const maxPPV = Math.max(ev.tran_ppv ?? 0, ev.vert_ppv ?? 0, ev.long_ppv ?? 0);
|
||||
const waveformUrl = `${api()}/waveform?db_id=${encodeURIComponent(ev.id)}&api_base=${encodeURIComponent(api())}`;
|
||||
tr.innerHTML = `
|
||||
<td><button class="wf-btn" onclick="window.open('${waveformUrl}','_blank')" title="View waveform">〜</button></td>
|
||||
<td>${_fmtTs(ev.timestamp)}</td>
|
||||
<td class="td-key">${ev.serial ?? '—'}</td>
|
||||
<td class="${_ppvClass(ev.tran_ppv)}">${_ppvFmt(ev.tran_ppv)}</td>
|
||||
|
||||
+115
-16
@@ -193,6 +193,10 @@
|
||||
</div>
|
||||
<button id="connect-btn" onclick="connectUnit()">Connect</button>
|
||||
<button id="load-btn" onclick="loadWaveform()" disabled>Load Waveform</button>
|
||||
<label style="display:flex;align-items:center;gap:4px;color:#8b949e;font-size:12px;cursor:pointer">
|
||||
<input type="checkbox" id="force-reload" style="accent-color:#1f6feb" />
|
||||
Force reload
|
||||
</label>
|
||||
<button id="prev-btn" onclick="stepEvent(-1)" disabled>◀ Prev</button>
|
||||
<button id="next-btn" onclick="stepEvent(+1)" disabled>Next ▶</button>
|
||||
</header>
|
||||
@@ -244,6 +248,46 @@
|
||||
let eventList = []; // populated from /device/events after connect
|
||||
let currentEventIndex = 0;
|
||||
|
||||
// ── DB mode: opened via ?db_id=<uuid>&api_base=<url> from History tab ────────
|
||||
const _urlParams = new URLSearchParams(window.location.search);
|
||||
const _dbId = _urlParams.get('db_id');
|
||||
const _dbApiBase = (_urlParams.get('api_base') || '').replace(/\/$/, '');
|
||||
|
||||
async function _loadFromDb() {
|
||||
const apiBase = _dbApiBase || document.getElementById('api-base').value.replace(/\/$/, '');
|
||||
setStatus('Loading waveform from database…', 'loading');
|
||||
document.getElementById('unit-bar').style.display = 'none';
|
||||
// Hide live-device controls — not relevant in DB mode
|
||||
document.querySelector('header .conn-group').style.display = 'none';
|
||||
|
||||
const url = `${apiBase}/db/events/${encodeURIComponent(_dbId)}/waveform`;
|
||||
let data;
|
||||
try {
|
||||
const resp = await fetch(url);
|
||||
if (!resp.ok) {
|
||||
const err = await resp.json().catch(() => ({ detail: resp.statusText }));
|
||||
throw new Error(err.detail || resp.statusText);
|
||||
}
|
||||
data = await resp.json();
|
||||
} catch (e) {
|
||||
setStatus(`Error: ${e.message}`, 'error');
|
||||
return;
|
||||
}
|
||||
lastData = data;
|
||||
renderWaveform(data);
|
||||
}
|
||||
|
||||
// Auto-load when opened with db_id param
|
||||
window.addEventListener('DOMContentLoaded', () => {
|
||||
if (_dbId) {
|
||||
// Pre-fill api-base if provided
|
||||
if (_dbApiBase) {
|
||||
document.getElementById('api-base').value = _dbApiBase;
|
||||
}
|
||||
_loadFromDb();
|
||||
}
|
||||
});
|
||||
|
||||
function setStatus(msg, cls = '') {
|
||||
const bar = document.getElementById('status-bar');
|
||||
bar.textContent = msg;
|
||||
@@ -364,7 +408,8 @@
|
||||
btn.disabled = true;
|
||||
setStatus('Fetching waveform…', 'loading');
|
||||
|
||||
const url = `${apiBase}/device/event/${evIndex}/waveform?host=${encodeURIComponent(devHost)}&tcp_port=${tcpPort}`;
|
||||
const force = document.getElementById('force-reload')?.checked ? '&force=true' : '';
|
||||
const url = `${apiBase}/device/event/${evIndex}/waveform?host=${encodeURIComponent(devHost)}&tcp_port=${tcpPort}${force}`;
|
||||
|
||||
let data;
|
||||
try {
|
||||
@@ -404,8 +449,11 @@
|
||||
bar.innerHTML = '';
|
||||
bar.className = 'ok';
|
||||
const ts = data.timestamp;
|
||||
if (ts) {
|
||||
bar.textContent = `Event #${data.index} — ${ts.display} `;
|
||||
const tsDisplay = ts
|
||||
? (typeof ts === 'string' ? ts : (ts.display ?? JSON.stringify(ts)))
|
||||
: null;
|
||||
if (tsDisplay) {
|
||||
bar.textContent = `Event #${data.index} — ${tsDisplay} `;
|
||||
} else {
|
||||
bar.textContent = `Event #${data.index} `;
|
||||
}
|
||||
@@ -413,7 +461,14 @@
|
||||
appendMeta('sr', `${sr} sps`);
|
||||
appendMeta('samples', `${decoded.toLocaleString()} / ${total.toLocaleString()}`);
|
||||
appendMeta('pretrig', pretrig);
|
||||
appendMeta('rectime', `${data.rectime_seconds ?? '?'}s`);
|
||||
// rectime_seconds is computed from (total_samples - pretrig_samples) / sr in
|
||||
// _decode_a5_waveform. Also show the compliance config record_time for reference.
|
||||
const cfgRt = unitInfo?.compliance_config?.record_time;
|
||||
const strtRt = data.rectime_seconds;
|
||||
const rtStr = (strtRt !== null && strtRt !== undefined)
|
||||
? `${strtRt}s (stored)` + (cfgRt !== null && cfgRt !== undefined ? ` / ${cfgRt}s (cfg)` : '')
|
||||
: (cfgRt !== null && cfgRt !== undefined ? `${cfgRt}s (cfg)` : '?');
|
||||
appendMeta('rectime', rtStr);
|
||||
|
||||
// No waveform data — show a clear reason instead of empty charts
|
||||
if (decoded === 0) {
|
||||
@@ -428,9 +483,14 @@
|
||||
return;
|
||||
}
|
||||
|
||||
// Build time axis (ms)
|
||||
const times = Array.from({ length: decoded }, (_, i) =>
|
||||
((i - pretrig) / sr * 1000).toFixed(2)
|
||||
// Clip to total_samples to exclude zero-padding the device appends beyond
|
||||
// the configured record window. total = pretrig + post_trig (e.g. 256+3072=3328).
|
||||
// decoded may be larger (e.g. 4495) due to trailing zero-padded bulk-stream frames.
|
||||
const displayCount = (total > 0 && total < decoded) ? total : decoded;
|
||||
|
||||
// Build time axis in seconds (matching Blastware event report layout).
|
||||
const times = Array.from({ length: displayCount }, (_, i) =>
|
||||
((i - pretrig) / sr).toFixed(3)
|
||||
);
|
||||
|
||||
// Show charts area
|
||||
@@ -447,6 +507,13 @@
|
||||
const micPeakPsi = data.peak_values?.micl_psi ?? null;
|
||||
const DBL_REF_PSI = 2.9e-9; // 20 µPa in psi
|
||||
|
||||
// 0C record peak values (device-computed, authoritative) per channel
|
||||
const peakValues0C = {
|
||||
Tran: data.peak_values?.tran_in_s ?? null,
|
||||
Vert: data.peak_values?.vert_in_s ?? null,
|
||||
Long: data.peak_values?.long_in_s ?? null,
|
||||
};
|
||||
|
||||
for (const [ch, color] of Object.entries(CHANNEL_COLORS)) {
|
||||
const samples = channels[ch];
|
||||
if (!samples || samples.length === 0) continue;
|
||||
@@ -455,22 +522,38 @@
|
||||
const isGeo = ch !== 'Mic';
|
||||
let plotSamples, peakLabel, yUnit, tooltipFmt, tickFmt;
|
||||
|
||||
// Clip channel samples to displayCount (same as time axis)
|
||||
const clippedSamples = samples.length > displayCount
|
||||
? samples.slice(0, displayCount)
|
||||
: samples;
|
||||
|
||||
// peak0C declared here (function scope) so it is visible in the Chart.js
|
||||
// config block below (which lives outside the if(isGeo) block).
|
||||
let peak0C = null;
|
||||
|
||||
if (isGeo) {
|
||||
// Geo channels: counts × (range / 32767) → in/s
|
||||
// Scale factor for the waveform shape (may need calibration per unit)
|
||||
const scale = geoRange / 32767;
|
||||
plotSamples = samples.map(c => c * scale);
|
||||
const peakIns = Math.max(...plotSamples.map(Math.abs));
|
||||
plotSamples = clippedSamples.map(c => c * scale);
|
||||
|
||||
// Use the device-computed 0C record peak for the label (authoritative).
|
||||
// The raw-sample-computed peak can be inflated by frame-boundary artifacts.
|
||||
peak0C = peakValues0C[ch];
|
||||
const peakIns = (peak0C !== null && peak0C !== undefined)
|
||||
? peak0C
|
||||
: Math.max(...plotSamples.map(Math.abs));
|
||||
peakLabel = `${peakIns.toFixed(5)} in/s`;
|
||||
yUnit = 'in/s';
|
||||
tooltipFmt = v => `${ch}: ${v.toFixed(5)} in/s`;
|
||||
tickFmt = v => v.toFixed(4);
|
||||
} else {
|
||||
// Mic: derive psi/count scale from the 0C peak value, display as psi; show dBL in header
|
||||
const peakCounts = Math.max(...samples.map(Math.abs));
|
||||
const peakCounts = Math.max(...clippedSamples.map(Math.abs));
|
||||
const micScale = (micPeakPsi !== null && peakCounts > 0)
|
||||
? Math.abs(micPeakPsi) / peakCounts
|
||||
: 1.0;
|
||||
plotSamples = samples.map(c => c * micScale);
|
||||
plotSamples = clippedSamples.map(c => c * micScale);
|
||||
const peakPsi = Math.max(...plotSamples.map(Math.abs));
|
||||
const peakDbl = peakPsi > 0 ? 20 * Math.log10(peakPsi / DBL_REF_PSI) : -Infinity;
|
||||
peakLabel = `${peakDbl.toFixed(1)} dBL (${peakPsi.toExponential(3)} psi)`;
|
||||
@@ -504,7 +587,9 @@
|
||||
renderData = plotSamples.filter((_, i) => i % step === 0);
|
||||
}
|
||||
|
||||
const chart = new Chart(canvas, {
|
||||
let chart;
|
||||
try {
|
||||
chart = new Chart(canvas, {
|
||||
type: 'line',
|
||||
data: {
|
||||
labels: renderTimes,
|
||||
@@ -526,7 +611,7 @@
|
||||
mode: 'index',
|
||||
intersect: false,
|
||||
callbacks: {
|
||||
title: items => `t = ${items[0].label} ms`,
|
||||
title: items => `t = ${items[0].label} s`,
|
||||
label: item => tooltipFmt(item.raw),
|
||||
},
|
||||
},
|
||||
@@ -538,11 +623,21 @@
|
||||
color: '#484f58',
|
||||
maxTicksLimit: 10,
|
||||
maxRotation: 0,
|
||||
callback: (val, i) => renderTimes[i] + ' ms',
|
||||
callback: (val, i) => renderTimes[i] + ' s',
|
||||
},
|
||||
grid: { color: '#21262d' },
|
||||
},
|
||||
y: {
|
||||
// Clamp geo-channel y-axis to ±(0C peak × 1.4) so near-saturation
|
||||
// decode artifacts (which inflate autoscale to full range) don't
|
||||
// squash the actual blast signal into an invisible flat line.
|
||||
// The 0C peak value is authoritative for the true signal amplitude.
|
||||
// Guard: only apply if peak0C is a valid finite positive number.
|
||||
...(isGeo && peak0C !== null && peak0C !== undefined
|
||||
&& isFinite(peak0C) && peak0C > 0 ? {
|
||||
min: -(peak0C * 1.4),
|
||||
max: (peak0C * 1.4),
|
||||
} : {}),
|
||||
ticks: {
|
||||
color: '#484f58',
|
||||
maxTicksLimit: 5,
|
||||
@@ -566,7 +661,7 @@
|
||||
const xAxis = chart.scales.x;
|
||||
const yAxis = chart.scales.y;
|
||||
|
||||
// Find index of t=0
|
||||
// Find index of the trigger point (t ≥ 0.000 s)
|
||||
const zeroIdx = renderTimes.findIndex(t => parseFloat(t) >= 0);
|
||||
if (zeroIdx < 0) return;
|
||||
|
||||
@@ -583,8 +678,12 @@
|
||||
},
|
||||
}],
|
||||
});
|
||||
} catch (err) {
|
||||
console.error(`Chart.js error for channel ${ch}:`, err);
|
||||
canvasWrap.innerHTML = `<p style="color:#f85149;padding:8px;font-size:11px;">Chart error: ${err.message}</p>`;
|
||||
}
|
||||
|
||||
charts[ch] = chart;
|
||||
if (chart) charts[ch] = chart;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user