bw_ascii_report: parse OORANGE saturation marker + TimeSum typo
BW writes "OORANGE" (truncation of "Out Of Range") when a channel exceeds its full-scale, and uses a typo'd label "Peak Vector Sum TimeSum" for the PVS time field. Both confirmed against real ASCII files pulled from a Windows watcher PC 2026-05-27: T190LD5Q.LK0W Vert PPV = OORANGE (Normal range, 10 in/s exceeded) T438L713.RY0W All three PPVs OORANGE (Sensitive range, 1.25 in/s) K557L3YM.OE0W Tran+Vert PPV OORANGE + MicL PSPL OORANGE Previously our _parse_number() returned None for OORANGE → DB columns ended up NULL → events vanished from filters / sorts / dashboards despite being legitimate high-amplitude events. New behavior — substitute a conservative bound + set a saturation flag: - Channel PPV → geo_range_ips + ChannelStats.ppv_saturated - Peak Vector Sum → sqrt(3) * geo_range_ips + peak_vector_sum_saturated - MicL PSPL → 140 dB(L) + MicStats.pspl_saturated Flags propagate to the sidecar's bw_report block so the SFM UI can render "> 10 in/s" / "> 140 dBL" rather than treating the substituted value as exact. Same commit also accepts "Peak Vector Sum TimeSum" as an alias for "Peak Vector Sum Time" (BW always writes the typo on OORANGE PVS lines — every example file confirms it). Tests: new test_oorange_marker_treated_as_saturation (synthetic) + test_real_oorange_event_t190_parses (skips if real fixture absent). 177/177 tests pass; 16 pre-existing missing-fixture skips unchanged. Five events on prod (T190, T438, K557, plus 2 others matching the same fault pattern) will pick up correct peaks + saturation flags once watchers re-forward. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -6,6 +6,16 @@ All notable changes to seismo-relay are documented here.
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **bw_ascii_report parser now handles `OORANGE` saturation marker.** BW writes `"OORANGE"` (truncation of "Out Of Range") in PPV / PVS / MicL PSPL fields when the underlying measurement exceeded the channel's full-scale. Previously our `_parse_number()` returned None → DB ended up with NULL peaks for legitimate high-amplitude events. Confirmed on real ASCII files pulled 2026-05-27 from the Windows watcher PC: T190LD5Q.LK0W (Vert saturated at Normal range 10 in/s), T438L713.RY0W (all three channels saturated at Sensitive range 1.25 in/s), K557L3YM.OE0W (Tran+Vert saturated + Mic PSPL OORANGE). New behavior:
|
||||||
|
- Per-channel PPV: substitute `geo_range_ips` as a conservative lower bound + set `ppv_saturated` flag
|
||||||
|
- Peak Vector Sum: substitute `sqrt(3) * geo_range_ips` (the theoretical max when all 3 channels are simultaneously at full-scale) + `peak_vector_sum_saturated` flag
|
||||||
|
- MicL PSPL: substitute 140 dB(L) (conservative NL-43 max) + `pspl_saturated` flag
|
||||||
|
- Saturation flags are propagated into the sidecar's `bw_report` block for downstream UI rendering (`> 10 in/s` or similar)
|
||||||
|
- Five events on prod (T190 / T438 / K557 + 2 others matching the same fault pattern) will pick up correct DB peaks + saturation flags once re-forwarded
|
||||||
|
- **bw_ascii_report parser handles `Peak Vector Sum TimeSum` typo'd label.** Real BW output uses this misspelled label (Sum appended twice instead of "Peak Vector Sum Time"). Now accepted as an alias. Confirmed against all three OORANGE example files — every one has the typo.
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
- **Histogram per-interval aggregation in `waveform.json`.** Histogram events now render with one bar per BW-reported interval (matching the Blastware printout) instead of ~200 bars per event (the raw codec output). When the sidecar's `bw_report.histogram.n_intervals` is populated (events ingested with the new parser, see next bullet), the `/db/events/{id}/waveform.json` endpoint groups the codec samples into N intervals via max-per-group and returns the aggregated array. `time_axis` gains `histogram_aggregated: true`, `n_intervals`, `interval_size_s`, and `interval_times` (HH:MM:SS strings). Both the modal chart and the standalone event browser use those interval timestamps as x-axis labels when present. Defensive: no-op for events ingested before the parser extension landed (their sidecars lack `histogram.n_intervals`) — those continue to render with raw codec output.
|
- **Histogram per-interval aggregation in `waveform.json`.** Histogram events now render with one bar per BW-reported interval (matching the Blastware printout) instead of ~200 bars per event (the raw codec output). When the sidecar's `bw_report.histogram.n_intervals` is populated (events ingested with the new parser, see next bullet), the `/db/events/{id}/waveform.json` endpoint groups the codec samples into N intervals via max-per-group and returns the aggregated array. `time_axis` gains `histogram_aggregated: true`, `n_intervals`, `interval_size_s`, and `interval_times` (HH:MM:SS strings). Both the modal chart and the standalone event browser use those interval timestamps as x-axis labels when present. Defensive: no-op for events ingested before the parser extension landed (their sidecars lack `histogram.n_intervals`) — those continue to render with raw codec output.
|
||||||
|
|||||||
@@ -60,6 +60,13 @@ class ChannelStats:
|
|||||||
time_of_peak_s: Optional[float] = None # seconds (relative to trigger; can be negative)
|
time_of_peak_s: Optional[float] = None # seconds (relative to trigger; can be negative)
|
||||||
peak_accel_g: Optional[float] = None # g (geo channels only)
|
peak_accel_g: Optional[float] = None # g (geo channels only)
|
||||||
peak_disp_in: Optional[float] = None # in (geo channels only)
|
peak_disp_in: Optional[float] = None # in (geo channels only)
|
||||||
|
# When BW writes "OORANGE" (Out Of Range — truncated) for a PPV
|
||||||
|
# value, the true peak exceeded the channel's full-scale range.
|
||||||
|
# We substitute the range max (e.g. 10.000 in/s for Normal range)
|
||||||
|
# as a lower bound, and flag here so downstream UI / alerts know
|
||||||
|
# to render "> 10 in/s" or "saturated" instead of trusting the
|
||||||
|
# value as an exact measurement.
|
||||||
|
ppv_saturated: bool = False
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -69,6 +76,11 @@ class MicStats:
|
|||||||
pspl_dbl: Optional[float] = None # dB(L)
|
pspl_dbl: Optional[float] = None # dB(L)
|
||||||
zc_freq_hz: Optional[float] = None
|
zc_freq_hz: Optional[float] = None
|
||||||
time_of_peak_s: Optional[float] = None
|
time_of_peak_s: Optional[float] = None
|
||||||
|
# Set when BW writes "OORANGE" for PSPL — mic exceeded its
|
||||||
|
# measurement range. pspl_dbl gets the conservative upper bound
|
||||||
|
# 140 dBL (typical NL-43 max; some units cap at 148). Consumers
|
||||||
|
# should render "> 140 dB(L)" or similar when this flag is set.
|
||||||
|
pspl_saturated: bool = False
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -92,6 +104,21 @@ class MonitorLogEntry:
|
|||||||
description: Optional[str] = None
|
description: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
# BW saturation marker — appears in PPV / Peak Vector Sum / similar
|
||||||
|
# numeric fields when the underlying measurement exceeded the
|
||||||
|
# channel's full-scale range (e.g., a geophone reading > 10 in/s at
|
||||||
|
# Normal range, or a mic exceeding its sensitivity ceiling). Treated
|
||||||
|
# as "≥ range_max" + a saturated flag rather than discarded.
|
||||||
|
# Appears as: ``"Tran PPV : OORANGE in/s"``
|
||||||
|
_OORANGE_MARKERS = ("OORANGE", "OUT OF RANGE")
|
||||||
|
|
||||||
|
|
||||||
|
def _is_oorange(value: str) -> bool:
|
||||||
|
"""True when a BW numeric field is an Out-Of-Range saturation marker."""
|
||||||
|
s = value.strip().upper()
|
||||||
|
return any(m in s for m in _OORANGE_MARKERS)
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class BwAsciiReport:
|
class BwAsciiReport:
|
||||||
"""Structured representation of one BW per-event ASCII export."""
|
"""Structured representation of one BW per-event ASCII export."""
|
||||||
@@ -144,6 +171,12 @@ class BwAsciiReport:
|
|||||||
# ── Vector sum ──────────────────────────────────────────────────────────
|
# ── Vector sum ──────────────────────────────────────────────────────────
|
||||||
peak_vector_sum_ips: Optional[float] = None
|
peak_vector_sum_ips: Optional[float] = None
|
||||||
peak_vector_sum_time_s: Optional[float] = None
|
peak_vector_sum_time_s: Optional[float] = None
|
||||||
|
# Saturation flag — set when BW writes "OORANGE" for the PVS. We
|
||||||
|
# then substitute sqrt(3) * geo_range_ips as a conservative upper
|
||||||
|
# bound (the theoretical maximum PVS when all 3 geo channels are
|
||||||
|
# simultaneously at full-scale). Consumers should display this as
|
||||||
|
# ">{value} in/s" or similar.
|
||||||
|
peak_vector_sum_saturated: bool = False
|
||||||
# Histograms additionally have an absolute date+time for the PVS
|
# Histograms additionally have an absolute date+time for the PVS
|
||||||
# (it occurred at a specific interval). Waveform reports show
|
# (it occurred at a specific interval). Waveform reports show
|
||||||
# only the relative-time value above.
|
# only the relative-time value above.
|
||||||
@@ -486,12 +519,20 @@ def parse_report(text: Union[str, bytes], *, parse_samples: bool = False) -> BwA
|
|||||||
):
|
):
|
||||||
ch_name, stat = key.split(" ", 1)
|
ch_name, stat = key.split(" ", 1)
|
||||||
cs = report.channels.setdefault(ch_name, ChannelStats())
|
cs = report.channels.setdefault(ch_name, ChannelStats())
|
||||||
num = _parse_number(value)
|
if stat == "PPV":
|
||||||
if stat == "PPV": cs.ppv_ips = num
|
if _is_oorange(value):
|
||||||
elif stat == "ZC Freq": cs.zc_freq_hz = num
|
# Channel saturated — substitute range max as lower
|
||||||
elif stat == "Time of Peak": cs.time_of_peak_s = num
|
# bound; flag so downstream UI can render "> 10 in/s".
|
||||||
elif stat == "Peak Acceleration": cs.peak_accel_g = num
|
cs.ppv_ips = report.geo_range_ips
|
||||||
elif stat == "Peak Displacement": cs.peak_disp_in = num
|
cs.ppv_saturated = True
|
||||||
|
else:
|
||||||
|
cs.ppv_ips = _parse_number(value)
|
||||||
|
else:
|
||||||
|
num = _parse_number(value)
|
||||||
|
if stat == "ZC Freq": cs.zc_freq_hz = num
|
||||||
|
elif stat == "Time of Peak": cs.time_of_peak_s = num
|
||||||
|
elif stat == "Peak Acceleration": cs.peak_accel_g = num
|
||||||
|
elif stat == "Peak Displacement": cs.peak_disp_in = num
|
||||||
|
|
||||||
# ── Histogram-specific fields ────────────────────────────────────────
|
# ── Histogram-specific fields ────────────────────────────────────────
|
||||||
# Histograms have Start/Stop time+date pairs + an interval count
|
# Histograms have Start/Stop time+date pairs + an interval count
|
||||||
@@ -532,8 +573,22 @@ def parse_report(text: Union[str, bytes], *, parse_samples: bool = False) -> BwA
|
|||||||
|
|
||||||
# ── Vector Sum ───────────────────────────────────────────────────────
|
# ── Vector Sum ───────────────────────────────────────────────────────
|
||||||
elif key == "Peak Vector Sum":
|
elif key == "Peak Vector Sum":
|
||||||
report.peak_vector_sum_ips = _parse_number(value)
|
if _is_oorange(value):
|
||||||
elif key == "Peak Vector Sum Time":
|
# PVS saturated — conservative upper bound is
|
||||||
|
# sqrt(3) * geo_range_ips (all 3 channels at full-scale).
|
||||||
|
# Real PVS could be lower (channels rarely peak
|
||||||
|
# simultaneously) but never higher within the range.
|
||||||
|
if report.geo_range_ips is not None:
|
||||||
|
import math as _math
|
||||||
|
report.peak_vector_sum_ips = _math.sqrt(3) * report.geo_range_ips
|
||||||
|
report.peak_vector_sum_saturated = True
|
||||||
|
else:
|
||||||
|
report.peak_vector_sum_ips = _parse_number(value)
|
||||||
|
# BW writes the PVS-time label with a typo: "Peak Vector Sum TimeSum"
|
||||||
|
# (looks like Sum got appended twice). Accept both forms. Confirmed
|
||||||
|
# against actual BW output on 2026-05-27 — every PVS-time line in
|
||||||
|
# the field examples (T190, T438, K557) uses the typo'd label.
|
||||||
|
elif key in ("Peak Vector Sum Time", "Peak Vector Sum TimeSum"):
|
||||||
report.peak_vector_sum_time_s = _parse_number(value)
|
report.peak_vector_sum_time_s = _parse_number(value)
|
||||||
_pvs_time_raw = value
|
_pvs_time_raw = value
|
||||||
elif key == "Peak Vector Sum Date":
|
elif key == "Peak Vector Sum Date":
|
||||||
@@ -558,7 +613,12 @@ def parse_report(text: Union[str, bytes], *, parse_samples: bool = False) -> BwA
|
|||||||
elif key == "Microphone":
|
elif key == "Microphone":
|
||||||
report.mic.weighting = value
|
report.mic.weighting = value
|
||||||
elif key == "MicL PSPL":
|
elif key == "MicL PSPL":
|
||||||
report.mic.pspl_dbl = _parse_number(value)
|
if _is_oorange(value):
|
||||||
|
# Mic saturated — substitute conservative upper bound 140 dBL.
|
||||||
|
report.mic.pspl_dbl = 140.0
|
||||||
|
report.mic.pspl_saturated = True
|
||||||
|
else:
|
||||||
|
report.mic.pspl_dbl = _parse_number(value)
|
||||||
# Mirror onto the "MicL" entry in channels so callers querying
|
# Mirror onto the "MicL" entry in channels so callers querying
|
||||||
# `channels["MicL"].ppv_ips` see something — but it's dB(L), not
|
# `channels["MicL"].ppv_ips` see something — but it's dB(L), not
|
||||||
# in/s, so we store as-is in the MicStats and mark the channel.
|
# in/s, so we store as-is in the MicStats and mark the channel.
|
||||||
|
|||||||
@@ -120,7 +120,12 @@ def _bw_report_to_dict(report: BwAsciiReport) -> dict:
|
|||||||
"peak_disp_in": cs.peak_disp_in,
|
"peak_disp_in": cs.peak_disp_in,
|
||||||
}
|
}
|
||||||
# Drop all-None entries — keeps the JSON tidy for partial reports.
|
# Drop all-None entries — keeps the JSON tidy for partial reports.
|
||||||
return {k: v for k, v in out.items() if v is not None}
|
out = {k: v for k, v in out.items() if v is not None}
|
||||||
|
# Saturation flag (only present when True) — signals that ppv_ips
|
||||||
|
# is the channel range max (a lower bound), not an exact reading.
|
||||||
|
if getattr(cs, "ppv_saturated", False):
|
||||||
|
out["ppv_saturated"] = True
|
||||||
|
return out
|
||||||
|
|
||||||
def _sc(ch_name: str) -> dict:
|
def _sc(ch_name: str) -> dict:
|
||||||
sc = report.sensor_check.get(ch_name)
|
sc = report.sensor_check.get(ch_name)
|
||||||
@@ -169,17 +174,22 @@ def _bw_report_to_dict(report: BwAsciiReport) -> dict:
|
|||||||
"vert": _ch("Vert"),
|
"vert": _ch("Vert"),
|
||||||
"long": _ch("Long"),
|
"long": _ch("Long"),
|
||||||
"vector_sum": {
|
"vector_sum": {
|
||||||
"ips": report.peak_vector_sum_ips,
|
"ips": report.peak_vector_sum_ips,
|
||||||
"time_s": report.peak_vector_sum_time_s,
|
"time_s": report.peak_vector_sum_time_s,
|
||||||
# Histogram events have an absolute date+time for the PVS
|
# Histogram events have an absolute date+time for the PVS
|
||||||
# (the interval at which it occurred); waveform events
|
# (the interval at which it occurred); waveform events
|
||||||
# only have the time_s offset.
|
# only have the time_s offset.
|
||||||
"when": report.peak_vector_sum_when.isoformat() if report.peak_vector_sum_when else None,
|
"when": report.peak_vector_sum_when.isoformat() if report.peak_vector_sum_when else None,
|
||||||
|
# Set when BW reported the PVS as OORANGE — value is the
|
||||||
|
# conservative upper bound sqrt(3) * geo_range_ips, not
|
||||||
|
# an exact peak.
|
||||||
|
"saturated": bool(getattr(report, "peak_vector_sum_saturated", False)),
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
"mic": {
|
"mic": {
|
||||||
"weighting": report.mic.weighting,
|
"weighting": report.mic.weighting,
|
||||||
"pspl_dbl": report.mic.pspl_dbl,
|
"pspl_dbl": report.mic.pspl_dbl,
|
||||||
|
"pspl_saturated": bool(getattr(report.mic, "pspl_saturated", False)),
|
||||||
"zc_freq_hz": report.mic.zc_freq_hz,
|
"zc_freq_hz": report.mic.zc_freq_hz,
|
||||||
"time_of_peak_s": report.mic.time_of_peak_s,
|
"time_of_peak_s": report.mic.time_of_peak_s,
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -385,6 +385,64 @@ def test_user_notes_extra_lines_beyond_four_are_dropped():
|
|||||||
assert "L5" not in r.user_note_labels.values()
|
assert "L5" not in r.user_note_labels.values()
|
||||||
|
|
||||||
|
|
||||||
|
def test_oorange_marker_treated_as_saturation():
|
||||||
|
"""BW writes 'OORANGE' (Out Of Range — truncated) when a channel
|
||||||
|
exceeds its full-scale. Verify ppv_ips falls back to geo_range_ips
|
||||||
|
+ saturated flag is set, mirroring the real T190LD5Q.LK0W,
|
||||||
|
T438L713.RY0W, and K557L3YM.OE0W events from prod 2026-05-27.
|
||||||
|
"""
|
||||||
|
txt = """\
|
||||||
|
"Event Type : Full Waveform"
|
||||||
|
"Serial Number : BE18190"
|
||||||
|
"Geo Range : 10.000 in/s"
|
||||||
|
"Tran PPV : 2.140 in/s"
|
||||||
|
"Vert PPV : OORANGE in/s"
|
||||||
|
"Long PPV : 2.830 in/s"
|
||||||
|
"Peak Vector Sum : OORANGE in/s"
|
||||||
|
"Peak Vector Sum TimeSum : 0.007 s"
|
||||||
|
"MicL PSPL : OORANGE "
|
||||||
|
"""
|
||||||
|
r = parse_report(txt)
|
||||||
|
# Tran/Long parse normally
|
||||||
|
assert r.channels["Tran"].ppv_ips == 2.14
|
||||||
|
assert r.channels["Tran"].ppv_saturated is False
|
||||||
|
assert r.channels["Long"].ppv_ips == 2.83
|
||||||
|
# Vert saturated → range max + flag
|
||||||
|
assert r.channels["Vert"].ppv_ips == 10.0
|
||||||
|
assert r.channels["Vert"].ppv_saturated is True
|
||||||
|
# PVS saturated → sqrt(3) * range_max as upper bound + flag
|
||||||
|
import math
|
||||||
|
assert r.peak_vector_sum_ips == pytest.approx(math.sqrt(3) * 10.0)
|
||||||
|
assert r.peak_vector_sum_saturated is True
|
||||||
|
# Mic saturated → 140 dBL conservative upper bound + flag
|
||||||
|
assert r.mic.pspl_dbl == 140.0
|
||||||
|
assert r.mic.pspl_saturated is True
|
||||||
|
# PVS time still parses despite the BW typo'd label "TimeSum"
|
||||||
|
assert r.peak_vector_sum_time_s == pytest.approx(0.007)
|
||||||
|
|
||||||
|
|
||||||
|
def test_real_oorange_event_t190_parses():
|
||||||
|
"""End-to-end against the real T190LD5Q.LK0W ASCII file pulled from
|
||||||
|
a Windows watcher PC on 2026-05-27. This is the canonical example
|
||||||
|
of the parser-PPV-miss bug we fixed in this iteration."""
|
||||||
|
fixture_path = (
|
||||||
|
Path(__file__).parent.parent / "example-events" /
|
||||||
|
"ascii-5-27-26" / "T190LD5Q_LK0W_ASCII.TXT"
|
||||||
|
)
|
||||||
|
if not fixture_path.exists():
|
||||||
|
pytest.skip("real ASCII fixture not present (local-only)")
|
||||||
|
r = parse_report_file(fixture_path)
|
||||||
|
assert r.serial == "BE18190"
|
||||||
|
assert r.geo_range_ips == 10.0
|
||||||
|
# Tran reads cleanly, Vert was OORANGE
|
||||||
|
assert r.channels["Tran"].ppv_ips == pytest.approx(2.14)
|
||||||
|
assert r.channels["Vert"].ppv_ips == 10.0
|
||||||
|
assert r.channels["Vert"].ppv_saturated is True
|
||||||
|
assert r.channels["Long"].ppv_ips == pytest.approx(2.83)
|
||||||
|
assert r.peak_vector_sum_saturated is True
|
||||||
|
assert r.peak_vector_sum_time_s == pytest.approx(0.007)
|
||||||
|
|
||||||
|
|
||||||
def test_real_histogram_fixture_populates_sensor_location():
|
def test_real_histogram_fixture_populates_sensor_location():
|
||||||
"""End-to-end: the histogram fixture uses 'Seis. Location:' — must
|
"""End-to-end: the histogram fixture uses 'Seis. Location:' — must
|
||||||
successfully populate sensor_location via position-based parsing."""
|
successfully populate sensor_location via position-based parsing."""
|
||||||
|
|||||||
Reference in New Issue
Block a user