Documents two commits that landed on dev since v0.20.0:
9b71ead series 4 codec work, initial decode success
micromate/idf_file.read_idf_file() decodes both IDFW
(waveform; 87-99% sample fidelity reusing
decode_waveform_v2 at offset 0x0f1f) and IDFH (histogram;
dedicated segment-based decoder, all 859 corpus files
decode, 181,071 intervals total).
9fd52dd feat: add thor report generation, pdf generation
micromate/idf_to_bw_report.py adapter projects parsed
Thor data into the bw_report sidecar shape so Thor
events flow through sfm/report_pdf.py without a
separate renderer. Wired into save_imported_idf.
Net effect: a Thor event ingested via /db/import/idf_file now
lands with the same fidelity as a BW event, gets a per-event PDF
on demand, and renders in Terra-View's modal chart using the same
plotting code as a BW event.
Roadmap items closed:
- Binary .IDFW / .IDFH codec (was pending)
- Series IV (Thor IDF) binary codec reverse-engineering
Companion: Terra-View v0.13.0 ships in parallel and closes Phase 1
of the SFM integration. No API changes in seismo-relay for that
piece — Terra-View just consumes existing endpoints better.
Bumps:
- pyproject.toml 0.20.0 → 0.21.0
- minimateplus.event_file_io.TOOL_VERSION 0.20.0 → 0.21.0
(any subsequent backfill_sidecars.py --force will re-stamp
existing sidecars; expected + harmless)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes out the Event-Report PDF iteration started in v0.17.x and ships
the parser fixes the real-world events were tripping over.
Today's additions on top of the pre-v0.20 unreleased body:
- Server-wide display TZ via the TZ env var (default America/New_York
on prod). Affects server logs, the PDF report's "Created" footer,
matplotlib datetime axes. DB columns stay UTC. Dockerfile now
installs tzdata.
- ZC Freq "above-range" handling — parser stores 100.0 +
zc_freq_above_range flag for BW's ">100 Hz" marker. Renders as
>100 in the PDF stats table, both modals (inline on webapp Peaks,
new column on event-browser table).
- scripts/backfill_sidecars.py --reparse-txt — re-runs the current
parser against the preserved _ASCII.TXT and overwrites the
sidecar's bw_report block. Lets parser fixes reach old events
without re-forwarding. Validated end-to-end against ~10k prod
events.
Fixes shipped today:
- histogram_interval_size_s missing from ReportData → every
histogram PDF render 500'd.
- Histogram PDF geo channels now share a nice-quantized y-axis
(0.005-LSB-aware 1-2-5 step sequence) instead of auto-scaling
per channel + inventing sub-LSB "0.003 in/s/div" footer labels.
Roadmap delta: closes the BW ASCII parser "PPV-miss on some TXT
formats", "histogram-specific structural fields", and ">100 Hz value
parsing" items. Adds a new entry for the byte[5]==0 histogram body
sub-format observed on S353 events.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New endpoint GET /db/events/{id}/report.pdf returns a single-page
letter-portrait PDF for any event with waveform data on disk.
Architecture:
sfm/report_pdf.py — gather_report_data() assembles fields from
SeismoDb row + .sfm.json sidecar (bw_report block) + .h5 samples;
render_event_report_pdf() turns that into PDF bytes via matplotlib.
sfm/server.py — new endpoint wires them together, streams PDF back
with Content-Disposition: inline so the browser displays it.
sfm_webapp.html — new "Download PDF" button in the event modal
footer that opens the endpoint in a new tab.
Fields surfaced — same coverage as a Blastware Event Report:
Header metadata (date/time, trigger source, range, sample rate,
project, client, operator, location, serial+firmware,
battery, calibration, file name)
Microphone block (PSPL in dB(L) + psi, ZC freq, channel test)
Per-channel stats (PPV, ZC Freq, Time of Peak, Peak Accel,
Peak Disp, Sensor Check) for Tran/Vert/Long
Peak Vector Sum
Waveform plot (MicL/Long/Vert/Tran stacked, shared time axis,
trigger marker, symmetric Y for geo, zero-anchored
mic) — OR per-interval bar chart for histograms.
Rendering pipeline = matplotlib only (vector PDF, no headless-browser
dep). Adds matplotlib>=3.8 to deps.
Visual layout is approximate until reference PDFs from Instantel land
at docs/reference/instantel/ for iteration. USBM RI8507 / OSMRE
compliance chart is stubbed (placeholder rectangle) — separate work
item.
Smoke-tested on a K558 waveform event: 77 KB valid PDF, all fields
populated correctly from the snapshot DB.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten the Series III / Series IV boundary so UI and storage dispatch
on a clean signal instead of sniffing filenames or applying magnitude
heuristics.
Phase 1 — events.device_family column ("series3" | "series4"):
self-applying migration with filename-based backfill of existing rows
(1,132 backfilled on prod 2026-05-20); plumbed through every import
path (BW endpoint, IDF endpoint, ACH server, BW CLI, sidecar
backfill); UPSERT preserves via COALESCE; UI dispatches on it.
Phase 2 — extract micromate/ package alongside minimateplus/:
native IdfEvent / IdfReport / IdfPeaks / IdfProjectInfo /
IdfSensorCheck (mic in dB(L), not pseudo-psi); moved
idf_ascii_report.py from sfm/ to micromate/; refactored
save_imported_idf to use IdfEvent and bridge to minimateplus.Event at
the SQL-insert boundary; idf_file.py stub for the future binary codec.
Phase 3 prep — docs/idf_protocol_reference.md captures the two
observed Thor binary header signatures (1,012 newer-firmware files vs
2 old files whose layout is byte-for-byte BW-STRT-compatible), file-size
hints suggesting int8 sample encoding, open questions in dependency
order, and a concrete first-session plan for cracking the codec.
Also rolled in the v0.18.1 hotfixes that motivated this work:
- idf_ascii_report parser now handles "<0.005 in/s" (below-threshold)
and "N/A" markers without leaving raw strings in numeric DB columns.
- sfm_webapp.html: defensive _ppvFmt / mic formatter so future
data-shape drift can't kill the whole events table render.
All 1,014 example-data sidecars round-trip through the new package.
See CHANGELOG.md for full notes.
The BW ACH ingest path was inserting every event with
record_type="Waveform" regardless of the actual type because
read_blastware_file() had `ev.record_type = "Waveform"` hardcoded, and
the live watcher-forward path parses files from a tmp path (suffix
".bw") that doesn't carry the original extension.
V10.72+ MiniMate Plus firmware encodes the event type as the last
character of the AB0T extension scheme (H=Histogram, W=Waveform,
M=Manual, E=Event, C=Combo). This change:
1. Adds derive_record_type_from_filename() public helper in
minimateplus/event_file_io.py
2. Uses it inside read_blastware_file() so direct callers (the
--dry-run path of scripts/import_bw.py, tests, ad-hoc scripts)
get correct types automatically
3. Overrides ev.record_type in WaveformStore.save_imported_bw()
using the ORIGINAL filename (source_path.name) — required
because the parser sees only the tmp file
Old S338 firmware (3-char extensions ending in `0`) and any
unrecognized suffix fall back to "Waveform".
Existing DB rows ingested before this fix are stuck with
record_type="Waveform" — a one-off SQL backfill would fix them
retroactively if desired. Terra-view's event modal also derives
client-side from the filename, so the UI already shows the correct
type for old events even without the backfill.
Version bumped to 0.16.1 in pyproject.toml, event_file_io.py
TOOL_VERSION, sfm/server.py FastAPI version, and CHANGELOG.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The "BW ACH ingestion" release. Paired with series3-watcher v1.5.0,
every Blastware ACH event (binary + _ASCII.TXT report) lands in
SeismoDb with device-authoritative peaks, project metadata, sensor
self-check, and ZC/Time-of-Peak data — without depending on the
still-undecoded waveform body codec.
Bumps pyproject.toml + minimateplus/event_file_io.py TOOL_VERSION
to 0.16.0. README banner + CHANGELOG entry summarise the work
that landed across commits cdfe4ad..f83993a on this branch.
### Added
- **Layered event storage architecture.** Each event now lands as four
files in the per-serial waveform store, each with a clear role:
- `<filename>` — the Blastware-readable binary (BW file). Untouched.
- `<filename>.a5.pkl` — the raw 5A frames (regenerative source).
- `<filename>.h5` — clean per-channel waveform arrays in physical
units (in/s for geo, psi for mic) plus event metadata (HDF5 with
gzip compression). This is the canonical format for downstream
analysis tools.
- `<filename>.sfm.json` — the modern review/metadata sidecar (peaks,
project, source provenance, review state, extensions).
SQLite (`seismo_relay.db`) is the searchable index over all four.
- **Plot-ready waveform JSON (`sfm.plot.v1`).** The `/device/event/{idx}/waveform`
and `/db/events/{id}/waveform.json` endpoints now return samples in
physical units with explicit time-axis metadata, peak markers, and
per-channel unit hints — no more guessing the ADC-to-velocity scale
client-side. The webapp waveform viewer was rewritten to consume
this shape.
- **In-app waveform viewer accuracy fix.** The standalone SFM webapp
viewer was scaling geophone amplitudes by `geoAdcScale / 32767`
(≈ 6.206 / 32767), where `geoAdcScale = 6.206053` is the device's
*in/s per V* hardware constant — not the ADC-counts-to-velocity
factor. This silently scaled every plot ~38% too low for Normal-range
geophones (the correct full-scale is 10.0 in/s, or 1.25 in/s for
Sensitive). Conversion is now done server-side using the geo_range
from compliance config; the client just plots.
- New `sfm/event_hdf5.py` module: `write_event_hdf5()`,
`read_event_hdf5()`, plus a plot-JSON helper.
- Backfill script extended to also emit `.h5` for existing events.
### Dependencies
- Added `h5py>=3.10` and `numpy>=1.24` for the HDF5 storage layer.
- Added `python-multipart>=0.0.7` (required by FastAPI for the
`/db/import/blastware_file` endpoint introduced in this release).
serversdown
claude