backfill: overlay bw_report onto Event before DB upsert
Mirror what the ingest path does: BW's reported peaks (and sample_rate
/ record_time) take precedence over codec output where present.
Without this, --force backfill silently overwrites bw_report-overlaid
DB columns with codec-derived peaks. Wrong for events where the codec
doesn't fully decode (waveform walker edge cases on SP0/SS0/SV0-style
events, histogram byte[5]!=0 sub-format that isn't yet RE'd), producing
PVS=0 on real high-amplitude events. Bit on prod 2026-05-22 with
three top-10 waveform events ending up at PVS=0 (rolled back same day,
this fix is the proper resolution).
New helper minimateplus.event_file_io.apply_bw_report_dict_to_event
operates on the projected sidecar dict shape (the structure
_bw_report_to_dict produces, which is what gets preserved in the
sidecar). Mirrors apply_report_to_event's semantics: only writes
fields where bw_report has a non-None value, no-ops cleanly on
empty / None input.
Dev validation against prod snapshot:
pre : 1839.7315 pvs_sum 356 events with DB PVS ≠ sidecar bw_report
post : 2016.4902 pvs_sum 2 events still mismatched (both have NULL
timestamp + duplicate rows, edge case)
Both edge-case events DO get the correct value written by the new
backfill — their stale rows from prior backfills remain because
UNIQUE(serial, timestamp) doesn't fire on NULL. Separate dedup
cleanup needed for those 2 events (0.014% of corpus); not blocking.
Backfill remains idempotent + bw_report preservation still passes
(0 WIPED, 0 CHANGED on the 3rd consecutive run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -254,6 +254,60 @@ def apply_report_to_event(event: Event, report: BwAsciiReport) -> None:
|
||||
event.rectime_seconds = report.record_time_s
|
||||
|
||||
|
||||
def apply_bw_report_dict_to_event(event: Event, bw_report: dict) -> None:
|
||||
"""Mirror of ``apply_report_to_event`` for the projected sidecar
|
||||
dict shape (as produced by ``_bw_report_to_dict``).
|
||||
|
||||
Why this exists
|
||||
───────────────
|
||||
The ingest path holds a live ``BwAsciiReport`` parsed straight from
|
||||
the ``_ASCII.TXT`` and uses ``apply_report_to_event`` to overlay
|
||||
device-authoritative peaks onto the codec output before insert.
|
||||
|
||||
The backfill path doesn't have the original ``.TXT`` (it's not
|
||||
retained in the waveform store), but it does have the preserved
|
||||
``bw_report`` block from the sidecar — which contains the same
|
||||
projected fields. Re-overlaying those during a backfill keeps the
|
||||
DB peak columns aligned with what BW reports rather than letting
|
||||
the codec output (which may be incomplete for unhandled formats or
|
||||
walker edge cases) win by default.
|
||||
|
||||
No-ops cleanly when ``bw_report`` is ``None``, empty, or missing
|
||||
any particular sub-field — only fields with a concrete value get
|
||||
written. Mirrors ``apply_report_to_event``'s "report wins where
|
||||
present" semantics.
|
||||
"""
|
||||
if not bw_report:
|
||||
return
|
||||
if event.peak_values is None:
|
||||
event.peak_values = PeakValues()
|
||||
pv = event.peak_values
|
||||
|
||||
peaks = bw_report.get("peaks") or {}
|
||||
tran = (peaks.get("tran") or {}).get("ppv_ips")
|
||||
vert = (peaks.get("vert") or {}).get("ppv_ips")
|
||||
long = (peaks.get("long") or {}).get("ppv_ips")
|
||||
if tran is not None: pv.tran = tran
|
||||
if vert is not None: pv.vert = vert
|
||||
if long is not None: pv.long = long
|
||||
vs_ips = (peaks.get("vector_sum") or {}).get("ips")
|
||||
if vs_ips is not None:
|
||||
pv.peak_vector_sum = vs_ips
|
||||
|
||||
mic = bw_report.get("mic") or {}
|
||||
pspl = mic.get("pspl_dbl")
|
||||
if pspl is not None and pspl > 0:
|
||||
pv.micl = _dbl_to_psi(pspl)
|
||||
|
||||
rec = bw_report.get("recording") or {}
|
||||
sr = rec.get("sample_rate_sps")
|
||||
if sr:
|
||||
event.sample_rate = sr
|
||||
rt = rec.get("record_time_s")
|
||||
if rt is not None:
|
||||
event.rectime_seconds = rt
|
||||
|
||||
|
||||
def _project_info_to_dict(pi: Optional[ProjectInfo]) -> dict:
|
||||
if pi is None:
|
||||
return {
|
||||
|
||||
Reference in New Issue
Block a user