fix(import): overlay BW report onto Event + upsert DB row on re-import
Two compounding bugs caused forwarded events to land in the DB with
broken-codec peak values (~10 in/s saturation on every channel) and
no project info, even when the watcher correctly paired a BW ASCII
report with the binary.
Bug 1: save_imported_bw built the sidecar JSON with the report's
authoritative peak / project values via event_to_sidecar_dict(
bw_report=...), but never overlaid those onto the in-memory Event
that flows to db.insert_events(). So the DB row got peak_values
from read_blastware_file()._peaks_from_samples() — which runs the
still-undecoded waveform body codec assuming raw int16 LE and
produces ±32K-shaped noise (= ±10 in/s at Normal range) regardless
of the actual signal. The sidecar JSON had the truth but the DB
columns (which the webapp queries for fast filter/sort) lied.
Bug 2: insert_events' IntegrityError handler only refreshed the
filename/filesize/a5_pickle/sidecar columns when a duplicate
(serial, timestamp) was seen. Peak values, project info,
sample_rate, record_type stayed locked in at whatever the FIRST
insert wrote. So even after Bug 1 was fixed, the historical
events in the DB (already inserted with broken-codec peaks) would
never get their values corrected, because a re-forward would just
hit IntegrityError and skip the field refresh.
Fix 1 (minimateplus/event_file_io.py + sfm/waveform_store.py):
- New apply_report_to_event(event, report) helper folds the BW
report's device-authoritative fields onto the Event in-place:
per-channel PPV, peak vector sum, mic PSPL→psi, project /
client / operator / sensor_location, sample_rate, record_time.
- save_imported_bw() calls the helper right after parsing the
report. The Event that flows to insert_events() now carries
correct values.
Fix 2 (sfm/database.py):
- insert_events()'s IntegrityError UPDATE now refreshes every
device-authoritative column from the new data: tran_ppv,
vert_ppv, long_ppv, peak_vector_sum, mic_ppv, project, client,
operator, sensor_location, sample_rate, record_type, plus
the existing filename/filesize/a5_pickle/sidecar fields.
- Preserves: id, waveform_key, session_id, created_at (immutable
/ FK fields), and false_trigger (operator review state).
End-to-end simulation verified:
- Step 1: import without report → DB has ±10 in/s peaks, no project
- Step 2: re-import WITH report → upsert path fires, DB now has
device-authoritative 0.005 in/s peaks + sensor_location
- Step 3: operator sets false_trigger=1, re-import again → flag
preserved, peaks remain correct
For the user's situation: deleting the watcher state file forces a
re-forward of all events. Each re-forward now pairs with its
_ASCII.TXT, applies the report onto the Event, and the upsert
refreshes the DB row. No DB nuke needed.
Full SFM suite: 62 passed, 44 skipped.
This commit is contained in:
+58
-22
@@ -374,28 +374,64 @@ class SeismoDb:
|
||||
inserted += 1
|
||||
except sqlite3.IntegrityError:
|
||||
skipped += 1
|
||||
# Upsert waveform fields onto the existing dedup row so a
|
||||
# re-download via the live endpoint refreshes filename /
|
||||
# size / sidecar without churning the rest of the row.
|
||||
if rec and ts:
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE events
|
||||
SET blastware_filename = ?,
|
||||
blastware_filesize = ?,
|
||||
a5_pickle_filename = ?,
|
||||
sidecar_filename = ?
|
||||
WHERE serial = ? AND timestamp = ?
|
||||
""",
|
||||
(
|
||||
rec.get("filename"),
|
||||
rec.get("filesize"),
|
||||
rec.get("a5_pickle_filename"),
|
||||
rec.get("sidecar_filename"),
|
||||
serial,
|
||||
ts,
|
||||
),
|
||||
)
|
||||
# UPSERT path: a row for this (serial, timestamp) already
|
||||
# exists. Refresh every device-authoritative field from
|
||||
# the new data so that a re-import with better data (e.g.
|
||||
# a watcher re-forward where the previous attempt missed
|
||||
# the paired BW ASCII report) replaces stale peaks /
|
||||
# project info / sample_rate.
|
||||
#
|
||||
# Preserved (not in this UPDATE):
|
||||
# id, waveform_key, session_id, created_at — immutable / FK
|
||||
# false_trigger — operator review state
|
||||
#
|
||||
# Behaviour change vs prior versions: this UPDATE used
|
||||
# to only refresh filename / filesize / a5_pickle /
|
||||
# sidecar fields. As a result, the first insert's
|
||||
# broken-codec peak values were locked in forever even
|
||||
# if subsequent re-forwards arrived with correct
|
||||
# report-derived values. Now every re-import lifts the
|
||||
# DB row up to whatever the latest Event carries.
|
||||
conn.execute(
|
||||
"""
|
||||
UPDATE events
|
||||
SET tran_ppv = ?,
|
||||
vert_ppv = ?,
|
||||
long_ppv = ?,
|
||||
peak_vector_sum = ?,
|
||||
mic_ppv = ?,
|
||||
project = ?,
|
||||
client = ?,
|
||||
operator = ?,
|
||||
sensor_location = ?,
|
||||
sample_rate = ?,
|
||||
record_type = ?,
|
||||
blastware_filename = ?,
|
||||
blastware_filesize = ?,
|
||||
a5_pickle_filename = ?,
|
||||
sidecar_filename = ?
|
||||
WHERE serial = ? AND timestamp = ?
|
||||
""",
|
||||
(
|
||||
pv.tran if pv else None,
|
||||
pv.vert if pv else None,
|
||||
pv.long if pv else None,
|
||||
pv.peak_vector_sum if pv else None,
|
||||
pv.micl if pv else None,
|
||||
pi.project if pi else None,
|
||||
pi.client if pi else None,
|
||||
pi.operator if pi else None,
|
||||
pi.sensor_location if pi else None,
|
||||
ev.sample_rate,
|
||||
ev.record_type,
|
||||
rec.get("filename") if rec else None,
|
||||
rec.get("filesize") if rec else None,
|
||||
rec.get("a5_pickle_filename") if rec else None,
|
||||
rec.get("sidecar_filename") if rec else None,
|
||||
serial,
|
||||
ts,
|
||||
),
|
||||
)
|
||||
|
||||
log.debug("insert_events serial=%s inserted=%d skipped=%d",
|
||||
serial, inserted, skipped)
|
||||
|
||||
Reference in New Issue
Block a user