serversdown/seismo-relay
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(twins): interval-based histogram/waveform matching in find_twins (#102 sub-task 2)

Browse Source 
A real trigger is recorded twice — as a triggered waveform (stamped at the
trigger instant) and inside the scheduled histogram whose interval contains it
(stamped at the 7am/7pm interval start). The two twins routinely differ by
HOURS, so the old ±5-minute window in find_twins silently missed them — which
broke review propagation (flagging one twin left its twin unflagged).

Twins are now matched by: same serial + identical peak_vector_sum + OPPOSITE
record type + the waveform's timestamp falling within the histogram's interval
(bounded by the next same-serial histogram). Matching keys off record timestamps
(not call-in/received times, which drift with field connectivity). window_seconds
is retained but ignored.

Rewrote test_find_twins + test_twin_propagation for the new contract (incl. the
75-min-apart UM12947 case, cross-type exclusion, containing-interval selection,
open-ended latest interval). Full suite: 264 passed; the 16 failures are
pre-existing (missing gitignored fixtures + a v0.26.0 codec case), unchanged
from baseline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
This commit is contained in:
serversdown
2026-08-28 05:23:30 +00:00
co-authored by Claude Opus 4.8
parent dedf1f02c9
commit 75ac610c61
4 changed files with 170 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sfm/database.py
+83 -32
View File
@@ -603,56 +603,107 @@ class SeismoDb:
).fetchall()
return [dict(r) for r in rows]
def find_twins(self, event_id: str, *, window_seconds: int = 300) -> list[dict]:
def find_twins(self, event_id: str, *, window_seconds: int | None = None) -> list[dict]:
"""
Find this event's histogram/waveform twin(s): rows sharing the same
serial and an identical peak_vector_sum, whose timestamp falls
within ``window_seconds`` of this event's timestamp. Excludes the
event itself. Returns [] if the event or any required field
(serial / peak_vector_sum / timestamp) is missing.
Find this event's histogram/waveform twin(s): the SAME physical event
recorded both as a scheduled histogram and as a triggered waveform.
Caveat: identical-PVS matching is a proxy for "same physical event
recorded twice," not a guarantee. In the rare case where the device
clamps/saturates PVS (clamped to sqrt(3) * geo_range), two distinct
saturated events on the same serial within the window can share the
same clamped PVS value and be matched as twins even though they are
different events. This is harmless in practice — false_trigger/
reviewed_real are derived/index columns re-derivable from the
sidecar source of truth — but worth knowing if twin counts look
surprising on a saturated/clamped run.
A real trigger is captured twice — once as a triggered waveform (stamped
at the trigger instant) and once inside the scheduled histogram whose
interval contains it (stamped at the histogram's interval start, e.g. the
7am/7pm call-in). The two can be HOURS apart in time yet report the same
serial and identical peak_vector_sum. Twins are therefore matched by:
* same serial,
* identical peak_vector_sum,
* OPPOSITE record type (one histogram, one waveform), and
* the waveform's timestamp falls within the histogram's interval —
from a histogram's timestamp up to the next histogram (same serial).
This replaces the old ±``window_seconds`` heuristic, which silently
missed twins more than a few minutes apart (a histogram's interval-start
stamp and the trigger instant routinely differ by hours). ``window_seconds``
is still accepted for backward compatibility but is ignored.
Returns [] if the event or a required field (serial / peak_vector_sum /
timestamp) is missing.
Caveat: identical-PVS matching remains a proxy for "same physical event"
— if the device clamps/saturates PVS (to sqrt(3) * geo_range), two
distinct saturated events could share a PVS. The added opposite-type and
interval constraints make a false pairing far less likely than the old
time-window match, and false_trigger/reviewed_real stay re-derivable from
the sidecar source of truth.
"""
def _parse(ts):
if not ts:
return None
try:
return datetime.datetime.fromisoformat(str(ts).replace(" ", "T"))
except ValueError:
return None
def _is_hist(rt):
return str(rt or "").lower().startswith("hist")
row = self.get_event(event_id)
if not row:
return []
serial = row.get("serial"); pvs = row.get("peak_vector_sum"); ts = row.get("timestamp")
if serial is None or pvs is None or not ts:
serial = row.get("serial"); pvs = row.get("peak_vector_sum")
t_target = _parse(row.get("timestamp"))
if serial is None or pvs is None or t_target is None:
return []
try:
t = datetime.datetime.fromisoformat(ts.replace(" ", "T"))
except ValueError:
return []
lo = (t - datetime.timedelta(seconds=window_seconds)).isoformat()
hi = (t + datetime.timedelta(seconds=window_seconds)).isoformat()
with self._connect() as conn:
rows = conn.execute(
"SELECT * FROM events WHERE serial=? AND id!=? AND peak_vector_sum=? "
"AND timestamp BETWEEN ? AND ?",
(serial, event_id, pvs, lo, hi),
).fetchall()
return [dict(r) for r in rows]
target_hist = _is_hist(row.get("record_type"))
def propagate_review_to_twins(self, event_id: str, *, window_seconds: int = 300) -> list[str]:
with self._connect() as conn:
cand_rows = [dict(r) for r in conn.execute(
"SELECT * FROM events WHERE serial=? AND id!=? AND peak_vector_sum=?",
(serial, event_id, pvs)).fetchall()]
hist_ts = [r["timestamp"] for r in conn.execute(
"SELECT timestamp FROM events WHERE serial=? AND lower(record_type) LIKE 'hist%'",
(serial,)).fetchall()]
# Histogram interval-start times for this serial, sorted, to bound intervals.
starts = sorted(x for x in (_parse(t) for t in hist_ts) if x is not None)
def _interval_end(h_start):
# The next histogram strictly after h_start bounds the interval; else open-ended.
for x in starts:
if x > h_start:
return x
return None
def _covers(h_start, w_time):
end = _interval_end(h_start)
return h_start <= w_time and (end is None or w_time < end)
twins = []
for c in cand_rows:
if _is_hist(c.get("record_type")) == target_hist:
continue # twins are strictly cross-type (one histogram, one waveform)
c_time = _parse(c.get("timestamp"))
if c_time is None:
continue
h_start, w_time = (t_target, c_time) if target_hist else (c_time, t_target)
if _covers(h_start, w_time):
twins.append(c)
return twins
def propagate_review_to_twins(self, event_id: str, *, window_seconds: int | None = None) -> list[str]:
"""
Copy this event's `false_trigger`/`reviewed_real` columns onto each
of its histogram/waveform twins (see `find_twins`), so flagging one
twin flags both. Returns the list of twin ids updated.
``window_seconds`` is accepted for backward compatibility but ignored;
twin matching is now interval-based (see `find_twins`).
"""
row = self.get_event(event_id)
if not row:
return []
ft = 1 if row.get("false_trigger") else 0
real = 1 if row.get("reviewed_real") else 0
twins = self.find_twins(event_id, window_seconds=window_seconds)
twins = self.find_twins(event_id)
moved = []
with self._connect() as conn:
for tw in twins: