BW writes "OORANGE" (truncation of "Out Of Range") when a channel
exceeds its full-scale, and uses a typo'd label "Peak Vector Sum
TimeSum" for the PVS time field. Both confirmed against real ASCII
files pulled from a Windows watcher PC 2026-05-27:
T190LD5Q.LK0W Vert PPV = OORANGE (Normal range, 10 in/s exceeded)
T438L713.RY0W All three PPVs OORANGE (Sensitive range, 1.25 in/s)
K557L3YM.OE0W Tran+Vert PPV OORANGE + MicL PSPL OORANGE
Previously our _parse_number() returned None for OORANGE → DB columns
ended up NULL → events vanished from filters / sorts / dashboards
despite being legitimate high-amplitude events.
New behavior — substitute a conservative bound + set a saturation flag:
- Channel PPV → geo_range_ips + ChannelStats.ppv_saturated
- Peak Vector Sum → sqrt(3) * geo_range_ips + peak_vector_sum_saturated
- MicL PSPL → 140 dB(L) + MicStats.pspl_saturated
Flags propagate to the sidecar's bw_report block so the SFM UI can
render "> 10 in/s" / "> 140 dBL" rather than treating the substituted
value as exact.
Same commit also accepts "Peak Vector Sum TimeSum" as an alias for
"Peak Vector Sum Time" (BW always writes the typo on OORANGE PVS
lines — every example file confirms it).
Tests: new test_oorange_marker_treated_as_saturation (synthetic) +
test_real_oorange_event_t190_parses (skips if real fixture absent).
177/177 tests pass; 16 pre-existing missing-fixture skips unchanged.
Five events on prod (T190, T438, K557, plus 2 others matching the
same fault pattern) will pick up correct peaks + saturation flags
once watchers re-forward.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three layered changes that together make histogram charts visually
match BW's printout (one bar per interval, not per codec block):
1. bw_ascii_report parser captures histogram fields it previously
dropped:
- Histogram Start/Stop Time + Date → datetime
- Number of Intervals + Interval Size (string + parsed seconds)
- <Channel> Peak Time + Peak Date → datetime (per-channel)
- Peak Vector Sum Date (combined with PVS Time → datetime;
clears the bogus seconds parse that interpreted "22:33:52"
as 22.0)
New _parse_iso_date() handles BW's ISO format for histograms
(waveforms use "May 8, 2026" long form). New _parse_interval_size()
handles "1 minute" / "5 minutes" / "15 seconds" etc.
2. _bw_report_to_dict() projects the new fields into a new
bw_report.histogram block in the sidecar.
3. /db/events/{id}/waveform.json wraps the existing path 1 (HDF5)
output with _maybe_aggregate_histogram(): when the event is a
histogram AND the sidecar has bw_report.histogram.n_intervals,
group the codec's per-block samples into N intervals via
max-per-group and return the aggregated array. time_axis gains
histogram_aggregated / n_intervals / interval_size_s / interval_times
fields.
Frontend (both modal chart in sfm_webapp.html + standalone event
browser) uses interval_times as x-axis labels when provided (BW-style
HH:MM:SS), falls back to interval index.
Defensive: aggregation is no-op when the sidecar lacks the histogram
block (events ingested before this change). Activates automatically
on prod once a watcher re-forward populates new sidecars.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the .TXT was parsed into the sidecar's bw_report projection
and then discarded at ingest time. Now save_imported_bw() writes it
to <store>/<serial>/<filename>_ASCII.TXT permanently.
Rationale: with BW Mail / Forwarding Agent being phased out of the
operator workflow, the XML/PDF/WMF those tools produce won't be
available — the binary + .TXT (created by BW ACH itself) are our
only authoritative inputs going forward. Keeping the raw .TXT
unlocks:
- Parser bug fixes can be applied RETROACTIVELY by re-parsing the
stored .TXT, instead of requiring a re-forward from the watcher
PC (which lost the .TXT after BW ACH cleanup).
- Audit trail of what BW actually sent us, for debugging.
- The five known parser-PPV-miss events will be re-parseable once
the regex fix lands (instead of staying broken indefinitely).
Storage cost: ~15 KB per event × 14k events = ~210 MB on the
existing prod corpus. Negligible.
Implementation:
- WaveformStore gains txt_path_for() + open_txt()
- save_imported_bw() writes the .TXT when bw_report_text is supplied
- sidecar source block records the txt_filename
- backfill_sidecars.py preserves txt_filename across regens
- New GET /db/events/{id}/ascii_report.txt endpoint serves it
- Returns 404 for events ingested before this change (no .TXT in
the store yet) — re-forward to populate
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reviewed against real Blastware Event Report PDFs (uploaded to
example-events/pdfsnstuff/) for K558LLB7.V20H (histogram) and
K558LLB8.0E0W (waveform). Each event type has its own layout because
BW's printouts genuinely differ:
Waveform header: Date/Time, Trigger Source, Range, Sample Rate
Histogram header: Start, Finish, Intervals At Size, Range, Sample Rate
(no trigger field — histograms aren't triggered)
Waveform stats: PPV, ZC Freq, Time (Rel. to Trig),
Peak Acceleration, Peak Displacement, Sensor Check
Histogram stats: PPV, ZC Freq, Date, Time (of peak), Sensor Check
Waveform plot: 4-channel stacked line, x-axis in SECONDS,
trigger triangle + window markers, symmetric Y
for geo, zero-anchored mic, "0.0" baseline label
on right edge per BW convention
Histogram plot: 4-channel stacked bars, Y-axis 0-to-peak only
(never negative — peaks are magnitudes), 0.0
baseline at the bottom
Waveform footer: USBM chart placeholder upper-right;
"Time X sec/div Amplitude Geo: Y in/s/div Mic: 0.001 psi(L)/div"
"Trigger = ▶━━◀"
Histogram footer: No USBM chart; same scale-info footer with
interval-size as the time unit
Other fixes from the first-pass screenshot review:
- Channel labels (MicL/Long/Vert/Tran) no longer cut off (wider
left margin)
- Histogram bars rise from zero baseline (abs of any signed values)
- ISO timestamp "2026-05-16T22:33:50" → "22:33:50 May 16, 2026"
matching BW's display format
Known gaps (separate work):
- Histogram codec returns per-block granularity (~200 bars for
BW's 4-interval display). XML-driven data source is the planned
fix; the structured BW XML has the per-interval aggregates.
- USBM RI8507 / OSMRE compliance chart still placeholder
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New endpoint GET /db/events/{id}/report.pdf returns a single-page
letter-portrait PDF for any event with waveform data on disk.
Architecture:
sfm/report_pdf.py — gather_report_data() assembles fields from
SeismoDb row + .sfm.json sidecar (bw_report block) + .h5 samples;
render_event_report_pdf() turns that into PDF bytes via matplotlib.
sfm/server.py — new endpoint wires them together, streams PDF back
with Content-Disposition: inline so the browser displays it.
sfm_webapp.html — new "Download PDF" button in the event modal
footer that opens the endpoint in a new tab.
Fields surfaced — same coverage as a Blastware Event Report:
Header metadata (date/time, trigger source, range, sample rate,
project, client, operator, location, serial+firmware,
battery, calibration, file name)
Microphone block (PSPL in dB(L) + psi, ZC freq, channel test)
Per-channel stats (PPV, ZC Freq, Time of Peak, Peak Accel,
Peak Disp, Sensor Check) for Tran/Vert/Long
Peak Vector Sum
Waveform plot (MicL/Long/Vert/Tran stacked, shared time axis,
trigger marker, symmetric Y for geo, zero-anchored
mic) — OR per-interval bar chart for histograms.
Rendering pipeline = matplotlib only (vector PDF, no headless-browser
dep). Adds matplotlib>=3.8 to deps.
Visual layout is approximate until reference PDFs from Instantel land
at docs/reference/instantel/ for iteration. USBM RI8507 / OSMRE
compliance chart is stubbed (placeholder rectangle) — separate work
item.
Smoke-tested on a K558 waveform event: 77 KB valid PDF, all fields
populated correctly from the snapshot DB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three UX upgrades to the main SFM webapp at /, all reinforcing the
'browse stored events' flow as the primary entry point:
1. Default section is now Database, not Live Device. Most users land
here to look at stored events; Live Device is opt-in (click the tab
to talk to a unit). Initial history + units fetch fires on first
paint so the table is populated when the page loads.
2. History table columns are sortable. Click any header to sort:
timestamp, serial, per-channel PPV (Tran/Vert/Long), PVS, mic dB(L),
project, client, type, key. Default direction varies by column type
(desc for numbers + timestamps, asc for text). Sort arrows appear
in the active column header. Headers are sticky so they stay
visible while scrolling.
3. Click-event-to-see-waveform. The existing sidecar review modal now
renders the 4-channel waveform plot inline at the top, fetched from
/db/events/{id}/waveform.json in parallel with the sidecar fetch.
Channels stacked MicL / Long / Vert / Tran (Instantel printout
order), shared bottom time axis, dashed trigger line + triangle
markers at t=0, zero baseline with "0.0" label on the right edge,
peak callouts per channel. Charts cleaned up on modal close.
Resolves the "where is the viewer" surprise — operators no longer need
to know about the /events route to see waveforms.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New standalone HTML page (sfm/event_browser.html, ~470 lines, Chart.js)
that lets you browse persisted events from the SeismoDb + WaveformStore.
Companion to the existing live-device viewer at /waveform:
/waveform — connect to a unit and pull events in real time
/events — browse events already stored in the DB
Flow:
1. Page loads → GET /db/units → populate serial dropdown
2. Select serial → GET /db/events?serial=X&limit=500 → event list
3. Click event → GET /db/events/{id}/waveform.json → render
Layout is Instantel-printout-ready: channels stacked vertically in
Tran / Vert / Long / MicL order, trigger line at t=0, peak labels,
clean dark theme. Frames the future PDF-export feature without
needing extra layout work.
Smoke-tested against the dev prod-snapshot — 4 channels render with
correct peaks for K558 events (L=0.3 in/s = the offset-fault peak
we've been chasing all week).
CHANGELOG entry added under [Unreleased] per the v0.20.0 release plan.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten the Series III / Series IV boundary so UI and storage dispatch
on a clean signal instead of sniffing filenames or applying magnitude
heuristics.
Phase 1 — events.device_family column ("series3" | "series4"):
self-applying migration with filename-based backfill of existing rows
(1,132 backfilled on prod 2026-05-20); plumbed through every import
path (BW endpoint, IDF endpoint, ACH server, BW CLI, sidecar
backfill); UPSERT preserves via COALESCE; UI dispatches on it.
Phase 2 — extract micromate/ package alongside minimateplus/:
native IdfEvent / IdfReport / IdfPeaks / IdfProjectInfo /
IdfSensorCheck (mic in dB(L), not pseudo-psi); moved
idf_ascii_report.py from sfm/ to micromate/; refactored
save_imported_idf to use IdfEvent and bridge to minimateplus.Event at
the SQL-insert boundary; idf_file.py stub for the future binary codec.
Phase 3 prep — docs/idf_protocol_reference.md captures the two
observed Thor binary header signatures (1,012 newer-firmware files vs
2 old files whose layout is byte-for-byte BW-STRT-compatible), file-size
hints suggesting int8 sample encoding, open questions in dependency
order, and a concrete first-session plan for cracking the codec.
Also rolled in the v0.18.1 hotfixes that motivated this work:
- idf_ascii_report parser now handles "<0.005 in/s" (below-threshold)
and "N/A" markers without leaving raw strings in numeric DB columns.
- sfm_webapp.html: defensive _ppvFmt / mic formatter so future
data-shape drift can't kill the whole events table render.
All 1,014 example-data sidecars round-trip through the new package.
See CHANGELOG.md for full notes.
The BW ACH ingest path was inserting every event with
record_type="Waveform" regardless of the actual type because
read_blastware_file() had `ev.record_type = "Waveform"` hardcoded, and
the live watcher-forward path parses files from a tmp path (suffix
".bw") that doesn't carry the original extension.
V10.72+ MiniMate Plus firmware encodes the event type as the last
character of the AB0T extension scheme (H=Histogram, W=Waveform,
M=Manual, E=Event, C=Combo). This change:
1. Adds derive_record_type_from_filename() public helper in
minimateplus/event_file_io.py
2. Uses it inside read_blastware_file() so direct callers (the
--dry-run path of scripts/import_bw.py, tests, ad-hoc scripts)
get correct types automatically
3. Overrides ev.record_type in WaveformStore.save_imported_bw()
using the ORIGINAL filename (source_path.name) — required
because the parser sees only the tmp file
Old S338 firmware (3-char extensions ending in `0`) and any
unrecognized suffix fall back to "Waveform".
Existing DB rows ingested before this fix are stuck with
record_type="Waveform" — a one-off SQL backfill would fix them
retroactively if desired. Terra-view's event modal also derives
client-side from the filename, so the UI already shows the correct
type for old events even without the backfill.
Version bumped to 0.16.1 in pyproject.toml, event_file_io.py
TOOL_VERSION, sfm/server.py FastAPI version, and CHANGELOG.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The "BW ACH ingestion" release. Paired with series3-watcher v1.5.0,
every Blastware ACH event (binary + _ASCII.TXT report) lands in
SeismoDb with device-authoritative peaks, project metadata, sensor
self-check, and ZC/Time-of-Peak data — without depending on the
still-undecoded waveform body codec.
Bumps pyproject.toml + minimateplus/event_file_io.py TOOL_VERSION
to 0.16.0. README banner + CHANGELOG entry summarise the work
that landed across commits cdfe4ad..f83993a on this branch.
### Added
- **Layered event storage architecture.** Each event now lands as four
files in the per-serial waveform store, each with a clear role:
- `<filename>` — the Blastware-readable binary (BW file). Untouched.
- `<filename>.a5.pkl` — the raw 5A frames (regenerative source).
- `<filename>.h5` — clean per-channel waveform arrays in physical
units (in/s for geo, psi for mic) plus event metadata (HDF5 with
gzip compression). This is the canonical format for downstream
analysis tools.
- `<filename>.sfm.json` — the modern review/metadata sidecar (peaks,
project, source provenance, review state, extensions).
SQLite (`seismo_relay.db`) is the searchable index over all four.
- **Plot-ready waveform JSON (`sfm.plot.v1`).** The `/device/event/{idx}/waveform`
and `/db/events/{id}/waveform.json` endpoints now return samples in
physical units with explicit time-axis metadata, peak markers, and
per-channel unit hints — no more guessing the ADC-to-velocity scale
client-side. The webapp waveform viewer was rewritten to consume
this shape.
- **In-app waveform viewer accuracy fix.** The standalone SFM webapp
viewer was scaling geophone amplitudes by `geoAdcScale / 32767`
(≈ 6.206 / 32767), where `geoAdcScale = 6.206053` is the device's
*in/s per V* hardware constant — not the ADC-counts-to-velocity
factor. This silently scaled every plot ~38% too low for Normal-range
geophones (the correct full-scale is 10.0 in/s, or 1.25 in/s for
Sensitive). Conversion is now done server-side using the geo_range
from compliance config; the client just plots.
- New `sfm/event_hdf5.py` module: `write_event_hdf5()`,
`read_event_hdf5()`, plus a plot-JSON helper.
- Backfill script extended to also emit `.h5` for existing events.
### Dependencies
- Added `h5py>=3.10` and `numpy>=1.24` for the HDF5 storage layer.
- Added `python-multipart>=0.0.7` (required by FastAPI for the
`/db/import/blastware_file` endpoint introduced in this release).
test: add regression tests for v0.14.x SUB 5A protocol fixes
refactor(logging): change warning logs to debug for less verbosity in write_blastware_file
## v0.13.0 — 2026-05-01
### Fixed
- **SUB 5A bulk waveform stream — over-read bug for events ≥ 2 sec.**
`read_bulk_waveform_stream` was walking the chunk counter past the actual
end of the event, picking up post-event circular-buffer garbage that
corrupted reconstructed Blastware files for any waveform > ~1 sec. The
loop now extracts the event's `end_offset` from the STRT record at
`data[23:27]` of the probe response and stops the chunk walk when the next
counter would step past it. Verified against three BW MITM captures
(4-27-26 + 5-1-26): 2-sec event drops from 37 over-read chunks to 7
bounded chunks; 3-sec drops to 9; non-zero-start "event 2" drops to 9.
### Added
- `framing.bulk_waveform_term_v2(key4, end_offset, last_chunk_counter)` —
computes the corrected SUB 5A TERM frame's `(offset_word, params)` per the
formula confirmed across all 3 BW captures. Not yet wired into
`read_bulk_waveform_stream` (the legacy TERM is still used to preserve the
existing `blastware_file.write_blastware_file` frame-structure expectations);
available for the next iteration that switches to BW's 0x0200 chunk step.
- `framing.parse_strt_end_offset(a5_data)` — extracts the event-end pointer
from the STRT record in an A5 response payload.
- Update `s3_bridge.py` to default raw capture file paths to "auto" for timestamped naming.
- Modify `gui_bridge.py` to pre-check raw capture options and streamline path handling.
- Extend `ach_server.py` to save both incoming and outgoing raw bytes for analysis.
- Revise `CHANGELOG.md` and `instantel_protocol_reference.md` to reflect changes in recording mode handling and compliance data encoding.
- Added `CallHomeConfig` model to represent the Auto Call Home settings.
- Introduced methods in `MiniMateClient` for reading (`get_call_home_config`) and writing (`set_call_home_config`) the call home configuration.
- Updated `MiniMateProtocol` with new commands for call home operations (SUB 0x2C for read, SUB 0x7E for write, and SUB 0x7F for confirm).
- Created API endpoints for retrieving and updating call home settings in the server.
- Enhanced the web interface with a new "Call Home" tab for user interaction with call home settings.
- Implemented JavaScript functions for reading and writing call home configurations from the web app.
Ports the intelligent-caching branch concept to a plain Python in-memory
implementation — no SQLAlchemy, no extra DB table, no new dependencies.
_LiveCache (threading.Lock + dicts) caches:
- device info: indefinite, invalidated by POST /device/config
- events: keyed by (conn_key, device_event_count); count-probe fast path
(~2s poll+count_events) avoids full downloads when nothing is new
- monitor status: 30-second TTL, invalidated by monitor start/stop
- waveforms: permanent per (conn_key, event_index)
All four cached endpoints accept ?force=true to bypass the cache.
Removes sfm/cache.py (SQLAlchemy experiment, now superseded).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sfm/database.py (new)
- SeismoDb class: three tables keyed by unit serial number
- ach_sessions: one row per ACH call-home
- events: one row per triggered event, deduped by (serial, waveform_key)
- monitor_log: one row per monitoring interval, deduped by (serial, waveform_key)
- WAL mode, per-request connections, silent dedup via UNIQUE constraint
- Query helpers: query_events(), query_monitor_log(), get_sessions(), query_units()
- false_trigger flag on events for future review UI / report filtering
bridges/ach_server.py
- Import SeismoDb; create shared instance at startup pointed at
bridges/captures/seismo_relay.db
- After each call-home: insert_events() + insert_monitor_log() + insert_ach_session()
- DB failures logged as warnings, never abort the session
sfm/server.py
- Import SeismoDb; lazy singleton via _get_db()
- New DB read endpoints: GET /db/units, /db/events, /db/monitor_log, /db/sessions
- PATCH /db/events/{id}/false_trigger for manual review flagging
CLAUDE.md / CHANGELOG.md
- Document DB schema, SFM DB endpoints, architecture decision (unit-keyed only)
- Version bump to v0.11.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add full decode pipeline for 0x2C partial records from the device's event
list, representing continuous monitoring intervals where no threshold was
crossed. These records appear interleaved with full triggered events in the
browse walk and were previously ignored.
minimateplus/models.py
- Add MonitorLogEntry dataclass: key, start_time, stop_time, serial,
geo_threshold_ips, raw_header, duration_seconds property
minimateplus/protocol.py
- read_waveform_header() now returns (data_rsp.data, length) — full payload
including the record-type byte at position 0 — instead of the sliced header.
Callers that need the old slice use raw_data[11:11+length] as before.
minimateplus/client.py
- Add _decode_0a_partial_header(): auto-detects 9-byte (sub_code=0x10) vs
10-byte (sub_code=0x03) timestamp format, handles 1-byte inter-timestamp
gap, extracts serial via BE anchor and geo threshold via Geo: anchor.
- Add get_monitor_log_entries(skip_keys=None): browse walk (1E → 0A → 1F),
decodes partial records, skips full records and already-seen keys.
minimateplus/__init__.py
- Export MonitorLogEntry
bridges/ach_server.py
- After get_events(), call get_monitor_log_entries(skip_keys=seen_keys) and
save new entries to monitor_log.json in the session directory.
- Add _monitor_log_entry_to_dict() helper.
- Include monitor log keys in downloaded_keys for state persistence.
CLAUDE.md / CHANGELOG.md
- Document 0x2C partial record layout (timestamp format, ASCII metadata
region, 1-byte gap edge case) confirmed from 4-11-26 MITM capture.
- Version bump to v0.10.0; update What's next.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CHANGELOG.md:
- New v0.9.0 section covering erase-all protocol, browse helpers,
delete_all_events(), ach_mitm.py, and ACH server overhaul
- Back-filled v0.8.0 section (write pipeline, monitoring, ACH server)
that was missing from the previous release notes
CLAUDE.md:
- Bump version to v0.9.0
- Add erase-all protocol section with full wire sequence, SUB 0x06
storage range response layout, and post-erase key counter reset notes
- Document ACH server state format (ach_state.json v0.9.0 schema with
downloaded_keys + max_downloaded_key)
- Add RV55 DCD/DTR issue to What's next
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Added `_decode_a5_waveform()` to parse SUB 5A frames into per-channel time-series data.
- Introduced `download_waveform(event)` method in `MiniMateClient` to fetch full waveform data.
- Updated `Event` model to include new fields: `total_samples`, `pretrig_samples`, `rectime_seconds`, and `_waveform_key`.
- Enhanced documentation in `CHANGELOG.md` and `instantel_protocol_reference.md` to reflect new features and confirmed protocol details.
- Establish v0.5.0 as first versioned release
- README rewritten to reflect current scope: Blastware replacement in
progress, not just a reverse-engineering capture tool
- Documents all current components: seismo_lab.py, minimateplus,
sfm/server.py, Console tab, TCP/cellular transport
- Adds ACEmanager required settings table (Quiet Mode etc.)
- Adds roadmap section
- CHANGELOG.md created with entries from v0.1.0 through v0.5.0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
serversdown
claude