serversdown/seismo-relay

Fork 0

Files

T

serversdown 6cb619ecc4 version bump - 0.21.1

2026-06-01 19:33:44 +00:00

91 KiB

Raw Blame History

Changelog

All notable changes to seismo-relay are documented here.

[Unreleased]

v0.21.1 — 2026-06-01

Bug fixes against v0.21.0 surfaced after the first prod redeploy. Three production-visible symptoms — blank waveform charts on most Thor events, blank histogram charts on all Thor events, and a mic chart that auto-scaled against a dB(L) value treated as psi — all root-caused and fixed.

Fixed

Dynamic IDFW body offset. The v0.21.0 codec hardcoded the body at file offset 0x0f1f based on the example corpus, but only ~52% of production IDFW events use that offset; the rest sit at offsets from 0x1033 up to 0x3082 depending on header padding. At 0x0f1f the codec would find a coincidentally-matching 00 02 00 magic, read the 2-byte Tran preamble, and return empty V/L/M arrays — producing near-empty .h5 files and blank charts. micromate.idf_file._find_waveform_body_offset() now scans every 00 02 00 magic position past 0x0E00, trial-decodes each one, and picks the offset with the most samples. Validated across 483 prod IDFW files: 0 preamble-only events (was ~50%), 355/483 fully decode, 126/483 partial (BW codec walker-stops-early on loud events — pre-existing limitation, samples reached are correct).
IDFH histograms now render bar charts. Histograms previously skipped the .h5 write because there are no per-sample arrays, but the renderer drives the per-interval bar chart from .h5 channel data + bw_report.histogram.n_intervals. save_imported_idf now synthesizes a 1-sample-per-interval array from the decoded IdfhInterval peak counts and writes an .h5 so the existing renderer works unchanged — each "sample" is the per-interval peak ADC count, so the writer's count × geo_fs/32768 conversion yields the right bar height.
Mic chart scaling on Thor events. PeakValues.micl (consumed by the h5 writer's per-count mic scale factor) expects psi, but the Thor bridge was stuffing the dB(L) value (~99.4) into it, producing a per-count factor 5+ orders of magnitude too large and a flat-looking mic chart. Fixed by adding IdfPeaks.mic_pspl_psi alongside mic_pspl_dbl; read_idf_file() computes it from binary mic counts (max(|MicL|) × 2.14e-6 psi/count) for both IDFW and IDFH paths; save_imported_idf merges it onto the typed event after IdfEvent.from_report; the bridge feeds psi to PeakValues.micl with a dB(L)→psi formula fallback when only the dB(L) value is available. dB(L) for the report header still flows through bw_report.mic.pspl_dbl unchanged.

Operator

After deploy, run python scripts/backfill_thor_events.py to refresh every existing Thor event's sidecar + .h5 with the corrected codec output. The script auto-skips events already at the current TOOL_VERSION, so the bump from 0.21.0 → 0.21.1 is what triggers the refresh.

v0.21.0 — 2026-05-29

The "Thor / Series IV codec" release. Two big pieces landed: (1) the IDF binary codec actually decodes now, both IDFW and IDFH, and (2) a Thor→BW adapter lets Thor events flow through the existing Series III Event Report PDF pipeline. Combined effect: a Thor event ingested via /db/import/idf_file now lands in the DB with the same fidelity as a Blastware event, gets a per-event PDF on demand, and renders in Terra-View's modal chart with the same plotting code as a BW event.

Added — Thor IDF binary codec (`micromate/idf_file.read_idf_file`)

IDFW (waveform) — body sits at fixed file offset 0x0f1f; reuses the verified decode_waveform_v2() walker from minimateplus.waveform_codec. Sample fidelity is 87–99% byte-exact against the ASCII-sidecar reference values on quiet events; loud events hit the same walker-stops-early limitation as the BW codec on SP0/SS0/SV0-style events.
IDFH (histogram) — dedicated segment-based decoder for the Thor histogram body format: [len_be][0a 00 00 00][00 NN][05 3f] framing plus N × 72-byte interval records (4 × 16-byte per-channel min/max/halfp). All 859 Thor IDFH corpus files decode, totalling 181,071 intervals; per-channel peaks match the sidecar within ~1.8% (ADC quantization).
BW-aliased binary detection — a small number of corpus files (e.g. BE9439_*.IDFW/IDFH) are actually Series III Blastware binaries that share the IDF filename convention by accident. read_idf_file() detects them via their BW STRT signature and raises NotImplementedError pointing the caller at read_blastware_file() instead of trying to decode them as IDF.
Full field layouts in docs/idf_protocol_reference.md; supporting analysis scripts in analysis_idf/ (decode validators, per-file detail dumps, corpus accuracy reports).

Added — Thor → BW report adapter (`micromate/idf_to_bw_report.py`)

build_bw_report_from_idf(report_dict, binary_md=, intervals=, is_histogram=) projects a parsed Thor IdfReport plus binary-extracted metadata plus decoded IDFH intervals into the bw_report-shaped dict that sfm.report_pdf.gather_report_data consumes. No need to duplicate the renderer — Thor data is ~95% the same metric set as BW; the adapter handles the field-name mapping (MicPSPL → pspl_dbl, >100 sentinel → zc_freq_above_range, free-form Calibration : Nov 22, 2023 by Instantel → calibration_date + calibration_by, etc.).
For IDFH events the adapter derives histogram.interval_times by stepping IntervalSize from HistogramStartTime, matching what the BW pipeline expects from a histogram-mode event.
Wired into WaveformStore.save_imported_idf — every Thor event ingested via /db/import/idf_file now gets a bw_report block in its sidecar in addition to the existing extensions.idf_report (the raw parsed Thor payload). Falls back gracefully (PDF renders from DB-only fields) if the adapter raises — logged as a warning rather than failing the ingest.

Companion releases

Terra-View v0.13.0 ships in parallel — closes Phase 1 of the SFM integration. The shared event-detail modal now renders the SFM event story (Chart.js waveform/histogram chart, inline PDF preview, .TXT download, FT/reviewer/notes review form) without operators needing to bounce to the standalone SFM webapp on port 8200. Uses only existing seismo-relay endpoints — no API changes here, just better consumption.

Migration / Operations

No DB migration needed. Existing Thor events already in the store don't automatically pick up the new bw_report block — they'd need a re-ingest (post the IDF binary + paired .TXT back to /db/import/idf_file) for the adapter to run. Alternatively, run scripts/backfill_sidecars.py --reparse-txt after a small adapter change (the script currently only re-runs the BW ASCII parser; extending it to handle Thor would be a small follow-up).

cd /home/serversdown/terra-view
docker compose build sfm && docker compose up -d sfm

The bumped TOOL_VERSION = "0.21.0" in minimateplus/event_file_io.py means any subsequent backfill_sidecars.py --force pass will re-write sidecars with the new version stamp; that's expected and harmless.

v0.20.0 — 2026-05-28

The "PDF + parser polish" release. Closes out the Event-Report PDF iteration started in v0.17.x: histogram layouts now render correctly against BW reference PDFs, the ASCII parser handles the real-world edge cases production events were tripping over (OORANGE, >100 Hz, histogram timestamps), and the .TXT preservation rollout lets parser fixes be applied retroactively to ingested events. Adds server-wide timezone support so operator-visible timestamps no longer drift into UTC. Rolls up the substantial "pre-v0.20" body of work that had accumulated under [Unreleased] (PDF generation, histogram codec fix, histogram parser fields, .TXT preservation, backfill safety) — see the trailing "pre-v0.20.0 work" section below for the full list.

Added (2026-05-28)

Server-wide display timezone via TZ env var. Both seismo-relay and terra-view now respect a TZ environment variable (default America/New_York on prod). Affects server log timestamps, the PDF report renderer's UTC→local conversions on the "Created" footer line, matplotlib's datetime axes, and any other naïve-vs-aware datetime rendering. DB columns (created_at, etc.) stay UTC regardless — this is a display-side fix, not a storage-side one. Dockerfile now installs tzdata (required for the env var to take effect under python:slim). Override per-deployment via the TZ line in docker-compose.yml.
ZC Freq "above-range" handling — render >100 Hz instead of —. BW writes ">100 Hz" literally when the zero-crossing algorithm sees a peak too fast to count (device cuts off at 100 Hz on V10.72). Previously _parse_number(">100") returned None and the PDF stats table rendered —. Now the parser mirrors the OORANGE pattern: stores 100.0 on zc_freq_hz and sets a new zc_freq_above_range flag. Flag rides through the sidecar's bw_report block. Renders as >100 in the PDF (per-channel + mic block), as · >100 Hz inline on the event modal's Peaks section, and as a dedicated column on the event-browser stats table. Verified against the real T190LD5Q.LK0W fixture from 2026-05-27 plus a synthetic test case.
Per-channel ZC Freq surfaced in event modals. Neither the main webapp modal (sfm_webapp.html) nor the standalone event browser (event_browser.html) previously exposed ZC Freq. Now both do — webapp shows it inline alongside PPV (0.04500 in/s · 47 Hz); event-browser gets a dedicated column on its per-channel stats table. Required wiring a parallel sidecar fetch into the event-browser's loadEvent() (it was only fetching waveform.json). Falls back to — for events without a preserved .TXT (pre-2026-05-27 ingests).
scripts/backfill_sidecars.py --reparse-txt flag. Before this, the backfill script preserved the bw_report block from existing sidecars verbatim — so parser-side fixes (like the >100 Hz addition above) couldn't reach old events. The new flag re-runs the current parser against the preserved <serial>/<filename>_ASCII.TXT, overwrites the bw_report block, and cascade-regenerates the sidecar. Implies sidecar regeneration on every event (bypasses the sha/version skip). No-op for events without a preserved .TXT (legacy ingests pre-2026-05-27 .TXT-preservation rollout). Idempotent. Run with --skip-hdf5 to skip waveform regen — recommended when only the bw_report needs refreshing. Validated end-to-end on prod: 9,999 events refreshed cleanly, ZC Freq + OORANGE flags now populated where the original .TXT had them.

Fixed (2026-05-28)

Histogram PDFs no longer 500 on the missing histogram_interval_size_s attribute. The histogram-interval-times derivation block in gather_report_data referenced rd.histogram_interval_size_s, but the field was never declared on the ReportData dataclass nor read from the sidecar projection (it was inlined into gather_report_data without the seconds-numeric counterpart making it onto the dataclass). Every histogram PDF render raised AttributeError → 500. Waveform PDFs were unaffected. Fix: add the field, read it from the projection's existing bw_report.histogram.interval_size_s key.
Histogram PDF geo channels now share a single nice-quantized y-axis. Previously each geo subplot auto-scaled independently — Tran, Vert, and Long all showed different per-channel maxes, so bar heights weren't directly comparable across channels. The footer "Amplitude Geo: X in/s/div" label was also computed as max(first_geo_channel) / 5 with no LSB quantization, producing nonsense values like 0.003 in/s/div when the geophone LSB is 0.005. Fix: compute a single shared geo y-axis range from max(Tran, Vert, Long), quantize the per-division step to BW's 1-2-5 sequence rounded to the 0.005 in/s LSB (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, ...), apply the same ylim + ticks to all three subplots, and use that step for the footer label. MicL stays on its own auto-scale (different units). Matches BW's chart styling.

Docs (2026-05-28)

Roadmap entry for a second undecoded histogram body sub-format. BE17353 (S353) events observed on 2026-05-28 use a histogram body where byte[5] = 0x00 (looks like a valid block header by every prior signal) but the walker finds zero data blocks. Different from the existing byte[5] != 0 roadmap entry (T190 / O121). Operationally identical impact — ingestion succeeds, DB peaks come from the bw_report overlay, only the chart is empty. Sample events captured in the roadmap entry for future RE work.

Migration / Operations

Re-parse existing events to pick up the new parser fields. Run on whichever box hosts the live waveform store:

docker exec terra-view-sfm-1 python /app/scripts/backfill_sidecars.py \
    --reparse-txt --skip-hdf5 --dry-run -v | tail
# Looks reasonable?  Run for real:
docker exec terra-view-sfm-1 python /app/scripts/backfill_sidecars.py \
    --reparse-txt --skip-hdf5 -v | tee /tmp/reparse.log | tail -30

Idempotent; safe to re-run. Only touches sidecars on disk — no DB writes.

terra-view docker-compose.yml: add TZ=America/New_York (or your deployment's zone) to both the terra-view and sfm service environment: blocks. Without this, server-rendered timestamps stay in UTC even on the rebuilt SFM image.

Pre-v0.20.0 work (rolled into this release)

The bullets below accumulated under [Unreleased] between v0.19.0 and v0.20.0; kept here so the historical narrative isn't lost.

Fixed

bw_ascii_report parser now handles OORANGE saturation marker. BW writes "OORANGE" (truncation of "Out Of Range") in PPV / PVS / MicL PSPL fields when the underlying measurement exceeded the channel's full-scale. Previously our _parse_number() returned None → DB ended up with NULL peaks for legitimate high-amplitude events. Confirmed on real ASCII files pulled 2026-05-27 from the Windows watcher PC: T190LD5Q.LK0W (Vert saturated at Normal range 10 in/s), T438L713.RY0W (all three channels saturated at Sensitive range 1.25 in/s), K557L3YM.OE0W (Tran+Vert saturated + Mic PSPL OORANGE). New behavior:
- Per-channel PPV: substitute geo_range_ips as a conservative lower bound + set ppv_saturated flag
- Peak Vector Sum: substitute sqrt(3) * geo_range_ips (the theoretical max when all 3 channels are simultaneously at full-scale) + peak_vector_sum_saturated flag
- MicL PSPL: substitute 140 dB(L) (conservative NL-43 max) + pspl_saturated flag
- Saturation flags are propagated into the sidecar's bw_report block for downstream UI rendering (> 10 in/s or similar)
- Five events on prod (T190 / T438 / K557 + 2 others matching the same fault pattern) will pick up correct DB peaks + saturation flags once re-forwarded
bw_ascii_report parser handles Peak Vector Sum TimeSum typo'd label. Real BW output uses this misspelled label (Sum appended twice instead of "Peak Vector Sum Time"). Now accepted as an alias. Confirmed against all three OORANGE example files — every one has the typo.

Added

Histogram per-interval aggregation in waveform.json. Histogram events now render with one bar per BW-reported interval (matching the Blastware printout) instead of ~200 bars per event (the raw codec output). When the sidecar's bw_report.histogram.n_intervals is populated (events ingested with the new parser, see next bullet), the /db/events/{id}/waveform.json endpoint groups the codec samples into N intervals via max-per-group and returns the aggregated array. time_axis gains histogram_aggregated: true, n_intervals, interval_size_s, and interval_times (HH:MM:SS strings). Both the modal chart and the standalone event browser use those interval timestamps as x-axis labels when present. Defensive: no-op for events ingested before the parser extension landed (their sidecars lack histogram.n_intervals) — those continue to render with raw codec output.
bw_ascii_report parser now captures histogram-specific fields. Previously the parser dropped these fields silently (Roadmap item closed):
- Histogram Start Time / Histogram Start Date (combined into histogram_start: datetime)
- Histogram Stop Time / Histogram Stop Date (combined into histogram_stop: datetime)
- Number of Intervals (histogram_n_intervals: int)
- Interval Size ("1 minute" string + parsed seconds: histogram_interval_size_str, histogram_interval_size_s)
- <Channel> Peak Time + <Channel> Peak Date for histogram events (combined into channel_peak_when: dict; waveforms continue to use time_of_peak_s relative)
- Peak Vector Sum Date (combined with PVS Time into peak_vector_sum_when: datetime; clears the previous bogus peak_vector_sum_time_s parse that interpreted "22:33:52" as 22.0 seconds)
- All new fields land in the sidecar's bw_report.histogram block via _bw_report_to_dict. Tested against synthetic K558LLB7.V20H-shaped input.
Raw BW ASCII report (.TXT) preservation. save_imported_bw now writes the paired _ASCII.TXT to <store>/<serial>/<filename>_ASCII.TXT alongside the binary at ingest time. Previously the .TXT was parsed into the sidecar's bw_report projection and then discarded — meaning parser bug fixes couldn't be applied retroactively without re-forwarding from the watcher PC. Now the raw .TXT lives in the waveform store permanently (~15 KB per event; ~210 MB total for a 14k-event store; negligible). Sidecar's source.txt_filename field records the saved path; backfill_sidecars preserves it across regens. New GET /db/events/{id}/ascii_report.txt endpoint serves the raw .TXT for any event ingested after this change. Events ingested before today still return 404 from that endpoint until re-forwarded. Architectural rationale: with BW Mail / Forwarding Agent being phased out of the operator workflow, the XML/PDF/WMF that those tools produced are no longer available — the binary + .TXT (created by BW ACH itself) are our authoritative source for everything going forward.
Event Report PDF generation — GET /db/events/{id}/report.pdf returns a single-page letter-portrait PDF for any event with waveform data on disk. Covers every field a Blastware Event Report includes: header metadata (date/time, trigger source, range, sample rate, project/client/operator/location, serial+firmware, battery, calibration, file name), microphone block (PSPL in dB(L) + psi, ZC freq, channel test), per-channel stats table (rows differ for waveform vs histogram), Peak Vector Sum, and the 4-channel plot. Iterated against real Blastware reference PDFs (uploaded to example-events/pdfsnstuff/):
- Waveform layout: header shows Date/Time, Trigger Source, Range, Sample Rate; stats table has PPV / ZC Freq / Time (Rel. to Trig) / Peak Accel / Peak Disp / Sensor Check; bottom plot is 4-channel line waveform (MicL top → Tran bottom), shared time axis in seconds, dashed trigger line + triangle marker at t=0, symmetric Y on geo channels, zero-anchored on mic, "0.0" baseline label on right per BW convention; footer shows Time X sec/div Amplitude Geo: Y in/s/div Mic: 0.001 psi(L)/div and the trigger window ▶━━◀ marker. USBM RI8507/OSMRE compliance chart placeholder upper-right.
- Histogram layout: header shows Start / Finish / Intervals At Size / Range / Sample Rate (no Trigger Source — histograms aren't triggered); NO USBM chart; stats table has PPV / ZC Freq / Date / Time / Sensor Check; bottom plot is per-interval bar chart, Y-axis 0-to-peak (never negative), 0.0 baseline at the bottom; footer shows Time INTERVAL_SIZE /div Amplitude Geo: Y in/s/div Mic: 0.001 psi(L)/div.
- Backed by matplotlib (vector PDF, no headless-browser dep). Adds matplotlib>=3.8 to deps.
- Known gap: histogram codec returns per-block granularity (~200 bars for a 4-interval event) instead of BW's per-interval aggregation. Visual difference vs BW's 4-bar display. XML-driven data source (parsing the structured _XML.XML files BW also exports) is the planned fix; that route also resolves the bw_ascii_report PPV-miss bug.
- Stubbed: USBM RI8507 / OSMRE compliance chart curves (separate work item; requires coding the regulatory piecewise functions).
"Download PDF" button in the event modal's footer — triggers the new endpoint; opens in a new tab so the browser handles save-or-display + surfaces any 404 / server errors visibly.
SFM webapp now opens to Database view by default and the History table is fully interactive. Click any column header to sort ascending / descending (timestamp, serial, per-channel PPV, PVS, mic dB(L), project, client, record type, key — all sortable). Click any event row to open the event modal, which now renders a 4-channel waveform plot inline (MicL / Long / Vert / Tran stacked, Instantel-printout order) alongside the existing sidecar review fields. Headers are sticky so the columns stay visible while scrolling long event lists. No more "where is the viewer" — pick a unit from the filter dropdown, scan the table, click the event, see the waveform.
Stored-event browser — new standalone HTML page at GET /events (sfm/event_browser.html). Pick a serial from the unit dropdown, scroll through that unit's events (newest-first), click any event to render its decoded waveform via the existing /db/events/{id}/waveform.json endpoint. Dark-themed Chart.js viewer, channels stacked vertically (MicL / Long / Vert / Tran — Instantel printout order, designed PDF-export-ready), trigger line at t=0, peak labels, search/filter, false-trigger flag honored. Companion to the existing live-device viewer at /waveform; the two routes are now clearly delineated in their docstrings. The webapp's inline plot at / is the primary path; /events remains a useful diagnostic when you want just a viewer.
Histogram body codec — uint8 peak count fix. Per-channel peak fields at block[6]/[10]/[14]/[18] are uint8, not uint16 LE spanning block[6:8] etc. The original interpretation was byte-exact on the N844 fixture corpus only because every annotation byte (block[7]/[11]/[15]/[19]) in those fixtures was zero. On non-N844 events with non-zero annotation bytes (observed across BE9558 Tran-drift and BE18003 Histogram+Continuous units), the old interpretation produced peaks up to 268 in/s per channel and 35× inflated PVS sums when first deployed to prod (rolled back same day; properly fixed in this release). Cross-correlated against BW's per-interval ASCII export on K558 / T003 / N599 / N844 corpora — 100% byte-exact on T/V/L, 99%+ on M (sub-precision rounding). Annotation byte preserved on each record as record["annotations"] for future RE. Verified against ~3,500 blocks across 5 in-repo fixtures + a synthetic K558 interval-12 regression block.
apply_bw_report_dict_to_event helper in minimateplus.event_file_io. Mirror of apply_report_to_event for the projected sidecar dict shape — used by the backfill path, which has the preserved bw_report block but not the original .TXT file. BW's reported peaks (and sample_rate / record_time) now win over codec output during --force backfill, matching ingest-path behavior.
scripts/check_bw_report_preservation.py — two-step snapshot/diff tool to verify that backfill_sidecars.py doesn't wipe the bw_report block from existing sidecars. Classifies every sidecar as PRESERVED / CHANGED / WIPED / STILL_MISSING / NEW / ADDED / REMOVED. Exit code 1 if any WIPED or CHANGED entries are found, so it can gate a CI step or deploy script.

Fixed

scripts/backfill_sidecars.py no longer wipes bw_report. Before this fix, event_to_sidecar_dict silently dropped the preserved bw_report block during every backfill, since the function only emits a bw_report when called with a live BwAsciiReport dataclass (which the backfill doesn't have — only the projected sidecar dict). Now we read the existing sidecar's bw_report and overlay it onto the regenerated sidecar, alongside the existing review and extensions preservation.
scripts/backfill_sidecars.py --force no longer overwrites BW-overlaid DB peaks with codec output. The backfill path now calls apply_bw_report_dict_to_event before the DB upsert, mirroring what the ingest path does (/db/import/blastware_file parses the .TXT into a BwAsciiReport, calls apply_report_to_event, then upserts). Without this, events where the codec doesn't fully decode (waveform walker edge cases on SP0/SS0/SV0-style events, histogram byte[5]!=0 sub-format) ended up with PVS=0 in the DB after a --force backfill; bit on prod 2026-05-22, rolled back the same day.
Thor IDF files no longer attempted as BW events in backfill. scripts/backfill_sidecars.py now filters out .IDFW / .IDFH files in _looks_like_event_file(); they share the .X0W / .X0H suffix shape but use a separate ingest path (WaveformStore.save_imported_idf) and aren't decodable by event_file_io.read_blastware_file.

Docs

CLAUDE.md — added a three-tier conceptual architecture model (SFM / SDM / shared codec library) near the top of the file, with a placement rule for where new code goes. Documents that what is conceptually SDM (database, waveform store, ingest, /db/* endpoints) still lives under sfm/ for historical reasons; rename deferred until the codebase is quiet enough for a clean refactor.
README.md — added a "Strategic direction" lead-in to the Roadmap that frames seismo-relay as a suite of cooperating components (not a single app), and an explicit "Terra-View ↔ SFM device control" roadmap section with a concrete implementation checklist (auth as hard prerequisite, embedded live-monitor view, action history, Series IV live-device support).
docs/histogram_codec_re_status.md updated with the uint8 retraction and the annotation-byte status.
Three known issues recorded in the Roadmap that were discovered during prod validation: (1) bw_ascii_report parser misses PPV / vector_sum on some .TXT formats (5 events on prod); (2) NULL-timestamp duplicate-row dedup needed (2 events on prod); (3) histogram body sub-format with byte[5] != 0 not yet decoded (~3 events on prod with empty .h5 plots).

v0.19.0 — 2026-05-20

The "device-family separation" release. Tightens the boundary between Series III (MiniMate Plus / Blastware) and Series IV (Micromate / Thor) so the UI and storage layer dispatch deterministically by family instead of sniffing filename extensions or magnitude heuristics.

Added — Phase 1: `device_family` column on `events`

events.device_family TEXT — new column carrying "series3" or "series4". Populated by every import path (/db/import/blastware_file, /db/import/idf_file, ACH server, BW CLI, sidecar backfill script). Returned through /db/events since query_events uses SELECT *.
Self-applying migration — on startup, ALTER TABLE ... ADD COLUMN lands the new column; a follow-on UPDATE backfills existing rows from the binary filename extension (.IDFH/.IDFW → series4, everything else → series3). No manual SQL needed.
UPSERT preserves family — re-imports without an explicit family don't blank existing rows (COALESCE(?, device_family)).
UI dispatches on the column — sfm_webapp.html events-table mic formatter now branches on ev.device_family === 'series4' (Thor stores native dB(L); BW stores psi). Modal uses source.kind === 'idf-import' from the sidecar (sidecars don't carry the DB column). Source-files section labels changed from "BW filename / BW filesize / BW sha256" to format-neutral "Event file / File size / File sha256".

Added — Phase 2: `micromate/` package alongside `minimateplus/`

micromate/ — new sibling package for the Thor / Micromate Series IV device. Currently scoped to offline-file ingest; live-device support (TCP transport, framing, protocol, client) will land here when reverse-engineering happens.
- micromate/idf_ascii_report.py — moved from sfm/idf_ascii_report.py. No behaviour change.
- micromate/models.py — typed IdfReport, IdfEvent, IdfPeaks, IdfProjectInfo, IdfSensorCheck. Stores mic in native mic_pspl_dbl (dB(L)) instead of the pseudo-psi shoehorn that the BW-shaped model uses. IdfEvent.from_report() constructs from a parsed dict + filename; IdfEvent.to_minimateplus_event(waveform_key) bridges to the existing sidecar / DB-insert machinery.
- micromate/idf_file.py — placeholder for the binary codec (.IDFH / .IDFW). Stubbed read_idf_file() raises NotImplementedError; documents the planned reverse-engineering path.
WaveformStore.save_imported_idf refactored to use the native IdfEvent and bridge at the SQL-insert boundary. Cleaner separation of "parse a Thor event" (in micromate/) from "store it on disk + write a sidecar" (in sfm/waveform_store.py).
Tests — tests/test_idf_ascii_report.py imports updated to micromate.idf_ascii_report. All 1,014 example-data sidecars round-trip through IdfEvent.from_report() without errors.

Companion releases

thor-watcher unaffected — it talks to the relay over HTTP only. No version bump needed.
terra-view unaffected today; can use device_family in its event-detail rendering when convenient.

v0.18.0 — 2026-05-19

The "Thor / Series IV ingest adapter" release. Seismo-relay can now accept event files from Instantel Micromate Series IV (Thor) units alongside the existing MiniMate Plus (Series III) Blastware pipeline.

Added — Thor (Series IV) IDF ingest

POST /db/import/idf_file (sfm/server.py) — multipart upload endpoint for .IDFH (histogram) and .IDFW (waveform) event files plus their .IDFH.txt / .IDFW.txt ASCII sidecars. Mirrors the shape of /db/import/blastware_file: pairing by filename, optional serial query hint, per-file outcome reporting.
sfm/idf_ascii_report.py — parser for Thor's TXT sidecars (verified against 1,014 real-world samples). Extracts device-authoritative PPV, ZC Freq, Peak Vector Sum, Mic PSPL, calibration date, firmware version, sensor self-check results, and project/client/operator strings.
WaveformStore.save_imported_idf() (sfm/waveform_store.py) — stores Thor binaries verbatim in <root>/<serial>/<filename>, writes a .sfm.json sidecar with source.kind = "idf-import" and the full parsed report under extensions.idf_report. Reuses the existing events table — Thor events dedupe on (serial, timestamp) and surface in /db/events alongside BW events.
tests/test_idf_ascii_report.py — parser tests against the thor-watcher/example-data/ corpus.

Changed

event_to_sidecar_dict() (minimateplus/event_file_io.py) allow-list for source_kind now includes "idf-import" so the existing sidecar machinery can carry Thor imports.
Bumped pyproject.toml version to 0.18.0.

Companion release

This release ships alongside thor-watcher v0.3.0, which adds the SFM forwarder that targets the new /db/import/idf_file endpoint. Operators flip the switch in thor-watcher's new "SFM Forward" Settings tab; events POST to seismo-relay just like the series3-watcher BW forwarder does today.

v0.17.0 — 2026-05-17

The "field rescue + DB management" release. Hardened against units that are stuck in a runaway call-home loop, and added an operator-facing path for purging bogus events that those same units dump into the DB before recovery. All work in this release was driven by the BE9558H incident (full incident log + recovery procedure at docs/runbooks/wedged_unit_recovery.md).

Added — wedged-unit recovery toolkit

A toolkit for breaking the call-home loop on a misbehaving unit whose firmware is too busy to keep up with normal request/response handshakes. Tested in production against BE9558H (16 May 2026) — a unit with a stuck-triggered Long-axis geophone that had been call-homing the office BW ACH server every 30 seconds for hours. Endpoints layered from "single attempt" to "siege mode" to suit different contention levels:

GET /device/events/storage_range — SUB 0x06 probe. POLL + one read; ~2s. Returns first/last event keys and an is_empty flag. Use to triage whether a unit has stored events without invoking the slow count_events() 1E/1F chain (which choked on BE9558H's corrupted event chain).
GET /device/events/index — SUB 0x08 probe. POLL + one read; ~2s. Returns the lifetime event counter (does NOT decrement on erase — use storage_range for "right now" state).
POST /device/events/erase — full erase sequence 0xA3 → 0x1C → 0x06 → 0xA2 (confirmed 2026-04-11, see the protocol reference). Resets event keys to 0x01110000. Caller's responsibility to disable ACH first if the underlying trigger condition will re-fill the buffer.
POST /device/rescue — one TCP session, short connect+recv timeouts: POLL → disable ACH (compliance config write) → erase events → close. Designed for race-loop usage when the device is busy in another session. 503 on connect-refused, 502 on protocol failure, 200 on full sequence success.
POST /device/stop_monitoring_blind — fire-and-forget Stop Monitoring (SUB 0x97), TCP-only. Dumps SESSION_RESET + POLL_PROBE + SESSION_RESET + POLL_DATA + 0x97 × repeat and closes without reading any S3 response. The full POLL preamble is required — write commands without it are silently ignored by the device's protocol parser (false-positive surface area that bit the first version of this endpoint). Use when the device's firmware can't keep up with full request/response but might process inbound bytes at its own pace.
POST /device/stop_monitoring_spam — server-side hammer loop, duration-bounded. Open TCP → write the same blind payload → close → repeat as fast as possible until duration_s elapses. Configurable connect_timeout (default 500ms) and repeat (frames per session). Reports sent_ok, connect_failed, write_failed, rate_attempts_per_s. Clamped to 5min duration.
POST /device/stop_monitoring_slow_drip — opposite of spam. Open ONE TCP session, drip the wake handshake + stop frames at interval_s (default 3s) for duration_s (default 120s, max 10min). Each drip is ~23 bytes — well under any UART FIFO size. Opportunistically drains any inbound bytes the device sends back; bytes_received > 0 in the response strongly suggests the device has started talking and the session is healthy. This is the endpoint that saved BE9558H. Spam mode had been overrunning the device's UART FIFO; slow drip stayed under it.
Six rescue scripts under scripts/ — thin bash wrappers around the endpoints, default SFM_BASE_URL=http://localhost:8200 (direct, not via Terra-View proxy whose 60s timeout would cut off the longer endpoints):
- rescue_device.sh — race-loop wrapper for /device/rescue
- blind_stop.sh — race-loop wrapper for /device/stop_monitoring_blind
- spam_stop.sh — single-call burst hammer
- slow_drip.sh — single-call held-session drip
- watch_unit.sh — passive periodic reachability check (every N min, logs to file), useful for unattended overnight monitoring of a wedged unit
docs/runbooks/wedged_unit_recovery.md — symptoms, quick-reference recovery procedure, the modem-layer mechanism (Sierra Wireless serial-port mode-flipping is the real failure mode — not the device firmware), and a table of "why simpler approaches don't work" so the next incident skips the dead ends.

Added — operator event DB management

Endpoints powering Terra-View's new /admin/events page (v0.12.0). Designed for purging bogus events from a unit that's been forwarding them in bulk (e.g. a stuck-triggered seismograph dumping hundreds of junk events before it's recovered).

DELETE /db/events/{event_id} — hard-delete one event row. Also unlinks the associated blastware binary (.AB0*), .a5.pkl, .sfm.json sidecar, and .h5 clean-waveform files via the WaveformStore. Returns the per-file removal status. 404 if the event doesn't exist.
POST /db/events/delete_bulk — filter-based or id-list-based bulk delete with safety rails:
- Filters (serial, from_dt, to_dt, false_trigger) combine with AND; same semantics as GET /db/events. ids is an additional inclusion list. Refuses to run with no filters (would wipe the whole table — raises 422).
- confirm must be true to actually delete. Otherwise returns a dry-run summary (status: "dry_run", matched: N, sample_serials: [...]).
- max_rows (default 10,000) caps how many rows can be deleted by-filter in one call. If exceeded, returns status: "too_many" with a hint to narrow or raise the cap. Bypassed when only ids is supplied.
_cleanup_event_files(row) helper in sfm/server.py — best-effort unlink() of all four sidecar paths derived from the row's blastware_filename. Logged at WARN if a path exists but unlink fails; the DB row deletion still proceeds.
SeismoDb.delete_event(id) and SeismoDb.delete_events_bulk(...) in sfm/database.py — both return the deleted row dict(s) so callers can do file cleanup. delete_events_bulk raises ValueError if no filters are supplied.

Changed

Default protocol recv timeout dropped from 30s → 10s in _build_client(). The unit usually responds in well under a second over cellular; 10s leaves comfortable headroom for retransmits while failing reasonably fast when a unit is wedged. The two endpoints that perform full 5A waveform downloads still pass timeout=120.0 explicitly so multi-minute event transfers are unaffected.
_build_client() now accepts an optional connect_timeout (TCP-only) so rescue / race-loop endpoints can fail fast on busy modems without affecting the protocol-level recv timeout.

Fixed

GET /device/monitor/status returned HTTP 500 + uncaught traceback when the device was unresponsive. The retry-on-Exception inner block let the second client.poll()'s ProtocolError propagate out of the handler. Now wrapped in proper try/except — returns 502 with {"detail": "Protocol error: No S3 frame received within 10.0s ..."} on timeout, 502 on connection errors, 500 only for genuinely unexpected exceptions.

Migration

No schema changes. No data migration required.

If you've been running a previous version against a wedged unit and accumulated bogus events, the new /admin/events page in Terra-View v0.12.0 (or direct POST /db/events/delete_bulk with confirm: true) is the cleanup tool. Watcher state on the upstream DL2 PC does NOT need separate cleaning — the watcher's sfm_forwarded.json keys on file sha256 and won't re-forward the same files.

Pairing

This release pairs with Terra-View v0.12.0, which adds the /admin/events UI that consumes the new bulk-delete endpoints, the bulk false-trigger flagging on /unit/{id}, and the field-deployment workflow that uses the same series3-watcher → SFM ingest path as before.

v0.16.1 — 2026-05-14

Fixed

record_type always "Waveform" for forwarded events. read_blastware_file() hardcoded ev.record_type = "Waveform" regardless of the file's actual type. The watcher-forward pipeline (the main BW ACH ingest path) compounds this by parsing files from a tmp path with a .bw suffix, so even a filename-based fallback inside the parser still wouldn't see the original extension. Now:
1. New derive_record_type_from_filename(filename) helper in minimateplus/event_file_io.py derives the type from the LAST character of the filename's extension (V10.72+ AB0T scheme: H=Histogram, W=Waveform, M=Manual, E=Event, C=Combo). Falls back to "Waveform" for old S338 firmware (3-char extensions ending in 0) and any unrecognized suffix.
2. read_blastware_file() now calls the helper with its path.name so direct callers (the --dry-run path in scripts/import_bw.py, tests, ad-hoc scripts) get the right value automatically.
3. WaveformStore.save_imported_bw() overrides ev.record_type with the original filename's derived type after parsing (the tmp file inside the parser doesn't carry the original extension). This is the path the live watcher-forwarder hits, so the DB column now reflects the actual event type going forward.
Events ingested before this fix are stuck with record_type="Waveform" in the DB; a one-off backfill (UPDATE events SET record_type = ... WHERE blastware_filename LIKE '%H') would fix them retroactively if desired. Terra-view's event modal also derives client-side from the filename, so the UI already shows the correct type for old events even without the backfill.

v0.16.0 — 2026-05-11

The "BW ACH ingestion" release. When paired with series3-watcher v1.5.0, every Blastware ACH event (binary + _ASCII.TXT report) lands in SeismoDb with device-authoritative peaks, project metadata, sensor self-check, and ZC/Time-of-Peak data — without depending on the still-undecoded waveform body codec. This is the end-to-end product win discussed in v0.15.0's "out of scope" notes: sortable / filterable monthly-summary review of historical events, populated from the BW ASCII export rather than re-decoded samples.

Added — `/db/import/blastware_file` rich-metadata ingestion

Paired BW ASCII reports. The endpoint now accepts the <binary>_<ext>_ASCII.TXT partner BW writes alongside each event. Pairing handles both filename conventions: ACH (M529LK44_AB0_ASCII.TXT) and manual-export (M529LK44.AB0.TXT). When both present, ACH wins.
minimateplus/bw_ascii_report.py (new) — parser + BwAsciiReport dataclass for BW's per-event ASCII export. Handles every field BW writes: identity, trigger config, per-channel PPV / ZC Freq / Time of Peak / Peak Acceleration / Peak Displacement, Peak Vector Sum + time, MicL PSPL / Time of Peak / ZC Freq, sensor self-check (Test Freq / Test Ratio / Test Amplitude / Pass-Fail per channel), monitor log, PC SW version.
Position-based user-notes parsing. BW's Compliance Setup → Notes tab labels (Project / Client / User Name / Seis Loc) are operator-editable — an operator can rename them to "Building:", "Site Address:", etc. Rather than maintain a label-spelling map, the parser uses positional matching between the Units : and Geo Range : anchors in the ASCII output. The four canonical slots (project / client / operator / sensor_location) populate by position regardless of label; the original labels BW wrote are preserved in report.user_note_labels for downstream UIs (terra-view) to display verbatim.
bw_report sidecar block. New top-level block in .sfm.json carrying the parsed BW report (trigger config, peaks with per-channel stats, mic block, sensor_check, monitor_log, PC SW version, operator-label labels).
apply_report_to_event(event, report) helper. Overlays the report's device-authoritative fields onto an in-memory Event so SeismoDb.insert_events() writes correct DB columns instead of the broken-codec values from _peaks_from_samples().

Fixed — three compounding bugs that left forwarded events with garbage data

Import endpoint inserted under serial="UNKNOWN". _serial_from_event(ev) was a stub that always returned None; the BW-filename-decoded serial that WaveformStore had already resolved was never surfaced to db.insert_events. Now uses rec["serial"] as the authoritative source. scripts/repair_unknown_serials.py repairs existing DB rows.
/db/units ignored events from non-ACH ingest paths. query_units() only aggregated from ach_sessions — events that arrived via save_imported_bw() were never visible in the fleet overview even though they populated events correctly. Now unions both tables.
Re-imports left stale DB rows. The IntegrityError handler in insert_events() only refreshed filename / sidecar columns when a duplicate (serial, timestamp) arrived. Peak values, project info, sample_rate, record_type stayed locked at whatever the first (often broken-codec) insert wrote. Now the upsert path refreshes every device-authoritative column from the new data while preserving false_trigger and immutable fields (id, created_at).
Server-side TXT pairing only knew the legacy convention. The endpoint stripped .TXT and looked up <binary> — which works for manual exports (<binary>.TXT) but not BW ACH (<stem>_<ext>_ASCII.TXT). Reports were arriving in the multipart but silently dropped. Now recognises both conventions and registers each report under all matching binary names.

Migration

For existing deployments where events were forwarded by an older watcher (broken pairing) or imported during the UNKNOWN-bucketing window:

python -m scripts.repair_unknown_serials --db <path> --apply to re-attribute serial="UNKNOWN" rows.
Delete the watcher's sfm_forwarded.json state file and let it re-forward. The server's upsert path will refresh the existing DB rows with the report's authoritative values.
Operator review state (false_trigger, sidecar review block) is preserved across the re-import.

v0.15.0 — 2026-05-07

Added

Layered event storage architecture. Each event now lands as four files in the per-serial waveform store, each with a clear role:
- <filename> — the Blastware-readable binary (BW file). Untouched.
- <filename>.a5.pkl — the raw 5A frames (regenerative source).
- <filename>.h5 — clean per-channel waveform arrays in physical units (in/s for geo, psi for mic) plus event metadata (HDF5 with gzip compression). This is the canonical format for downstream analysis tools.
- <filename>.sfm.json — the modern review/metadata sidecar (peaks, project, source provenance, review state, extensions).
SQLite (seismo_relay.db) is the searchable index over all four.
Plot-ready waveform JSON (sfm.plot.v1). The /device/event/{idx}/waveform and /db/events/{id}/waveform.json endpoints now return samples in physical units with explicit time-axis metadata, peak markers, and per-channel unit hints — no more guessing the ADC-to-velocity scale client-side. The webapp waveform viewer was rewritten to consume this shape.
In-app waveform viewer accuracy fix. The standalone SFM webapp viewer was scaling geophone amplitudes by geoAdcScale / 32767 (≈ 6.206 / 32767), where geoAdcScale = 6.206053 is the device's in/s per V hardware constant — not the ADC-counts-to-velocity factor. This silently scaled every plot ~38% too low for Normal-range geophones (the correct full-scale is 10.0 in/s, or 1.25 in/s for Sensitive). Conversion is now done server-side using the geo_range from compliance config; the client just plots.
New sfm/event_hdf5.py module: write_event_hdf5(), read_event_hdf5(), plus a plot-JSON helper.
Backfill script extended to also emit .h5 for existing events.

Dependencies

Added h5py>=3.10 and numpy>=1.24 for the HDF5 storage layer.
Added python-multipart>=0.0.7 (required by FastAPI for the /db/import/blastware_file endpoint introduced in this release).

v0.14.3 — 2026-05-05

Fixed

build_5a_frame — DLE-stuffing rule for 0x10 bytes in params (the long-standing >1-sec event 0 "won't open in BW" bug).

Previously build_5a_frame wrote params bytes RAW with no DLE stuffing, based on the incorrect assumption that the device handled all 0x10 bytes in params literally. It does not. The device's actual de-stuffing rule for the params region is:
- 10 10 → de-stuffs to 10
- 10 02/03/04 → kept literal (inner-frame markers)
- 10 X for other X → de-stuffs to just X (drops the 0x10)
When the counter passed in params has 0x10 in the high byte (e.g. counter=0x1000 produces params bytes ... 10 00 ...), the device silently corrupts the request to counter=0x__00 and responds with whatever lives at that wrong address. For counter=0x1000 the wrong address was 0x0000, so the response was a copy of the file header + STRT record. That STRT block then got embedded in the assembled body at file offset 0x1016, and Blastware refused to open the file (interprets the second STRT as a malformed multi-event file).

This explains the entire >1-sec event-0 failure pattern:
- 1-sec events have end_offset < 0x1000, so the chunk walk never requests counter 0x10__ and the bug never triggers.
- 2-sec / 3-sec / longer events all need a chunk at counter 0x1000 (and longer events also need 0x1200, 0x1400, etc., none of which have 0x10 in the high byte except 0x1000). Just one corrupted response is enough to embed STRT in the body and break the file.
Verified against BW 5-1-26 "copy 3sec" capture: all 17 5A request frames (probe + 2 metadata pages + 13 sample chunks + TERM) now match BW's wire output byte-for-byte, including the doubled 10 10 00 for counter=0x1000.

Notes

0x10 bytes in offset_hi (the standalone offset field at body[5]) are still written RAW — confirmed correct per the 1-2-26 capture.
BW's actual encoding of 10 02 / 10 04 for meta pages 0x1002 / 0x1004 is not doubled — it relies on the device keeping 10 02 and 10 04 as literal pairs. This is preserved by the fix.

v0.14.2 — 2026-05-04

Fixed

blastware_file.py — removed harmful "duplicate header+STRT" strip. The v0.13.x strip logic was matching the byte sequence 00 12 03 00 STRT in legitimate waveform data — sample chunks at counter 0x1000 and beyond often contain those bytes coincidentally — and zeroing 25 bytes of valid samples per match. This is why event 0 (event-1 case in the protocol) downloads of >1-sec recordings always failed in BW: the strip destroyed real data at body offset 0x1012..0x102B and propagated alignment differences through the rest of the body. Sub-1-sec events worked because their end_offset was below 0x1002, so no sample chunks landed in the metadata-page region and the strip's needle never matched. Verified fix by re-feeding the BW 5-1-26 "copy 3sec" capture's A5 frames into the file builder: output is now byte-identical to BW's saved M529LKIQ.G10 reference (8708 bytes, 0 differences).
BW already concatenates frame contributions in stream order without any de-duplication; SFM now does the same.

v0.14.0 — 2026-05-02

Changed (major rewrite)

read_bulk_waveform_stream — STRT-bounded chunk walk. Replaces the earlier 0x0400-step / max(key4[2:4], 0x0400) chunk-counter formula, which over-read ~5× past the actual event end into post-event circular- buffer garbage. The new walk:
1. Probe at counter = start_offset (event 1: 0x0000; event N: cur_key[2:4]).
2. Parse end_offset from the STRT record at data[17] of the probe response (end_key[2:4] field).
3. For event 1 only, read the two fixed metadata pages at counter 0x1002 and 0x1004 — these contain the global session-start compliance setup (Project / Client / User Name / Seis Loc / Extended Notes ASCII strings). Continuation events skip these (BW caches them across the session).
4. Walk sample chunks at 0x0200 increments (NOT 0x0400), bounded by end_offset — the loop exits when next_chunk_counter + 0x0200 > end_offset.
5. Send the proper TERM frame (see new bulk_waveform_term_v2()) with offset_word = end_offset - next_boundary and params[2:4] = next_boundary BE. The TERM response carries the partial last chunk + 26-byte file footer.
New helpers: bulk_waveform_term_v2(key4, end_offset, last_chunk_counter) and parse_strt_end_offset(a5_data) in minimateplus.framing.
stop_after_metadata / extra_chunks_after_metadata kwargs are now no-ops under the v0.14.x walk. They are retained on the read_bulk_waveform_stream signature for backward compatibility but log a DEBUG line when set. The old "scan for b'Project:' and stop one chunk later" workaround is obsolete — the loop is deterministically bounded by the STRT-derived end_offset.
Project / Client / User Name / Seis Loc string source corrected. These come from the dedicated metadata pages at counter 0x1002 / 0x1004, not from "A5 frame 7" of the sample-chunk stream. The earlier "A5 frame 7" claim was an artifact of the broken 0x0400-step walk where the bad counter formula coincidentally landed sample-chunk fi=7 on top of the 0x1002 metadata page.

Verified

Three independent BW MITM captures (4-27-26 + 5-1-26 + 5-4-26) confirm the new walk matches BW's behaviour event-for-event.
end_offset values verified across 3 events: 0x1ABE (4-27-26 2-sec), 0x21F2 (5-1-26 3-sec), 0x417E (5-1-26 event-2).

Notes

Earlier v0.13.0 / v0.13.1 / v0.13.2 entries describe partial steps along the way (some of the file builder fixes, filename bugs, etc.) that were superseded by the full rewrite. Treat this v0.14.0 entry as the definitive landing point for the corrected SUB 5A protocol.

v0.14.1 — 2026-05-04

Fixed

read_bulk_waveform_stream — event-N probe counter off-by-0x46. Continuation events (start_key[2:4] != 0) were being probed at counter start_offset + 0x0046 instead of just start_offset. In the iteration walk, cur_key from 1F is already the off=0x46 WAVEHDR record key, so the earlier formula effectively double-counted the WAVEHDR offset. The probe landed one WAVEHDR past the actual event start, the response no longer contained the STRT record at byte 17, parse_strt_end_offset returned None, and the chunk loop fell back to the max_chunks=128 cap — walking ~110 chunks of post-event circular-buffer garbage. Verified against the 5-1-26 "copy 2nd address" and 5-4-26 BW 2-sec event captures: BW probes counter=0x2238 with key=01112238 and STRT is present at byte 17 of the response (end_offset=0x417E).
CLAUDE.md / docs/instantel_protocol_reference.md — corrected the event-N section to clarify that start_key in those formulas is the off=0x46 key, not the off=0x2C boundary key, and removed the spurious +0x46 from the chunk-walk pseudocode.

v0.13.2 — 2026-05-01

Fixed

_extract_record_type — third 0C-record header format ("short", 8 bytes). A live SFM download against BE11529 produced files named M5290000.000 (zero-stamped) because the 0C waveform record's first bytes were 01 05 07 ea ... — neither the 9-byte single-shot layout (0x10 at byte 1) nor the 10-byte continuous layout (0x10 at bytes 0 and 2). Investigation showed this is a third format observed in the wild: an 8-byte header with no marker bytes at all ([day][month][year_BE:2][unknown][hour][min][sec]). The detection logic now scans the year (uint16 BE) at byte 2 / byte 3 / byte 4 and picks whichever offset returns a sensible year (2015–2050) — each format has the year at a unique position so this disambiguates cleanly.
- New format → event.record_type = "Waveform (Short)", Timestamp.from_short_record().
- Existing single-shot and continuous parsers unchanged.
- The user's event from May 1, 2026 13:21:37 now correctly resolves to a filename like M529LKIQ.G10 instead of M5290000.000.

Added

Timestamp.from_short_record(data) — decodes the 8-byte header.
_detect_record_format(data) — internal helper returning "single_shot" / "continuous" / "short" / None via year-position scan.

v0.13.1 — 2026-05-01

Fixed

_extract_record_type — Continuous-mode record headers misclassified as Unknown. In single-shot mode the 0C waveform record's 9-byte header puts the sub_code marker 0x10 at byte 1, with the day at byte 0. In Continuous mode the header is 10 bytes with the marker at byte 0 and byte 2, and the day at byte 1. Previous logic only inspected byte 1 and treated any value other than 0x10 / 0x03 as "Unknown", which prevented event.timestamp from being populated for any continuous-mode event whose day-of-month wasn't exactly 3 or 16. As a downstream effect, blastware_filename() saw event.timestamp == None, fell back to stem="0000" / ab="00", and produced filenames like M5290000.000. Discovered from a live SFM run on BE11529 in continuous mode (day-of-month = 5). Now disambiguates by checking BOTH byte 0 and byte 2: if both are 0x10, it's the 10-byte continuous header; else if byte 1 is 0x10, it's the 9-byte single-shot header. Day-of-month no longer matters.

Superseded by v0.13.2 — the user's actual record uses a third 8-byte format with no 0x10 markers, which v0.13.1 still misclassified.

v0.13.0 — 2026-05-01

Fixed

SUB 5A bulk waveform stream — over-read bug for events ≥ 2 sec. read_bulk_waveform_stream was walking the chunk counter past the actual end of the event, picking up post-event circular-buffer garbage that corrupted reconstructed Blastware files for any waveform > ~1 sec. The loop now extracts the event's end_offset from the STRT record at data[23:27] of the probe response and stops the chunk walk when the next counter would step past it. Verified against three BW MITM captures (4-27-26 + 5-1-26): 2-sec event drops from 37 over-read chunks to 7 bounded chunks; 3-sec drops to 9; non-zero-start "event 2" drops to 9.

Added

framing.bulk_waveform_term_v2(key4, end_offset, last_chunk_counter) — computes the corrected SUB 5A TERM frame's (offset_word, params) per the formula confirmed across all 3 BW captures. Not yet wired into read_bulk_waveform_stream (the legacy TERM is still used to preserve the existing blastware_file.write_blastware_file frame-structure expectations); available for the next iteration that switches to BW's 0x0200 chunk step.
framing.parse_strt_end_offset(a5_data) — extracts the event-end pointer from the STRT record in an A5 response payload.

Documentation

CLAUDE.md and docs/instantel_protocol_reference.md extensively rewritten to reflect the corrected SUB 5A protocol. See:
- CLAUDE.md "SUB 5A — chunk counter formula (REWRITTEN 2026-05-01)"
- CLAUDE.md "SUB 5A — STRT record encodes end_offset"
- CLAUDE.md "SUB 5A — TERM frame formula"
- CLAUDE.md "SUB 5A — fixed metadata pages 0x1002 and 0x1004"
- CLAUDE.md "SUB 0A — WAVEHDR response length distinguishes events from boundaries" (0x46 = real event, 0x2C = boundary marker)
- protocol reference §7.8.5 / §7.8.6 / §7.8.7 / §7.8.8
The previous chunk-counter formula (max(key4[2:4], 0x0400) + (chunk-1) * 0x0400) is now marked DEPRECATED and explicitly tagged WRONG with pointers to the new sections, so future work doesn't re-derive it.

Known minor diffs vs Blastware (deferred to a follow-up)

We still use the OLD 0x0400 chunk step rather than BW's 0x0200; switching also requires updating blastware_file.write_blastware_file's skip values and "extra chunk after metadata" logic, which depends on a fresh capture to verify.
We still use the legacy fixed offset_word=0x005A TERM frame rather than BW's end_offset - next_boundary formula, for the same reason.
Two fixed metadata pages at counter 0x1002 and 0x1004 are not yet read explicitly; under the current 0x0400 walk their content is reachable via the sample chunk that covers buffer addresses [0x1000, 0x1400).

v0.12.6 — 2026-05-01

Fixed

blastware_file.py — waveform frame classification — A5 frame classification for waveform-only vs header-only frames now uses frame.record_type instead of frame index. Only waveform frames (0x46) are written to the file body; metadata frames are skipped. Fixes spurious data corruption from incorrectly classified frames.
s3_analyzer.py — A5/5A frame naming — Bulk waveform stream frames (SUB 5A response) are now correctly labeled "A5" in analyzer output instead of being conflated with other multi-frame responses (SUB A4, E5, etc.).
S3FrameParser — frame terminator detection — Corrected the bare ETX terminator detection. Frame termination is now correctly identified by a standalone ETX=0x03 byte, not by the DLE+ETX sequence (which is part of the payload when it appears within a frame).

v0.12.5 — 2026-04-21

Added

seismo_lab.py — Download tab — New fourth tab for live wire-byte capture during event downloads. Captures both BW→device and device→S3 frames in real time, allowing inspection of the 5A bulk stream chunk sequence and frame-by-frame analysis without needing a bridge or MITM proxy. Files are saved with user-specified labels for easy tracking.

Changed

s3_bridge.py — raw captures always-on by default — --raw-bw and --raw-s3 now default to "auto" instead of None. Every bridge session automatically generates timestamped raw_bw_<ts>.bin and raw_s3_<ts>.bin files alongside the .bin/.log session files. Pass --raw-bw "" (explicit empty string) to disable if needed.
gui_bridge.py — raw capture checkboxes pre-checked — Both "BW→S3 raw" and "S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names the files). Unchecking a box passes --raw-bw "" to explicitly disable capture.
Bridge tab — TCP mode added — Serial/TCP radio toggle allows connection via cellular modem (RV50/RV55) instead of direct RS-232. Supports multi-capture design (simultaneous Bridge + Analyzer + Download sessions).
ach_server.py — TX capture added (raw_tx_<ts>.bin) — Every ACH inbound session now saves both directions: raw_rx_<ts>.bin (device → us, S3 side, as before) and raw_tx_<ts>.bin (us → device, BW side). Both files are usable in the Analyzer. TX bytes are buffered in memory until startup handshake succeeds (same as RX), preventing scanner probes from creating empty files.

v0.12.4 — 2026-04-21 (protocol analysis / docs only — no code changes)

Discovered

compliance_raw is wire-encoded, not logical bytes — read_compliance_config() returns bytes that include DLE prefix bytes (0x10) before any 0x03 values (because S3FrameParser preserves DLE+ETX inner-frame pairs as two literal bytes). The previous CLAUDE.md claim that "S3FrameParser handles this transparently so compliance_raw contains logical bytes" was wrong.
anchor-9 behavior per recording mode (confirmed from 4-20-26 BW write captures):
- Single Shot (0x00) / Continuous (0x01): anchor-9 = 0x00
- Histogram (0x03): anchor-9 = 0x10 — the E5 DLE prefix for the 0x03 recording_mode byte
- Histogram+Continuous (0x04): anchor-9 = 0x10 — an actual stored config byte for this mode Anchor position shifts by ±1 when recording_mode = 0x03 due to the extra DLE byte; the dynamic anchor search (buf.find(ANCHOR, 0, 150)) handles this correctly without code changes.
Write frame ETX escaping — BW escapes 0x03 bytes in write frame data as 0x10 0x03 on the wire. Our build_bw_write_frame sends data bytes raw without ETX escaping. Device accepts our raw writes for all tested modes. Hypothesis: device write parser uses the offset/length field for frame boundaries, not ETX scanning, making ETX escaping optional. Histogram mode (recording_mode = 0x03) write via SFM from a non-Histogram starting state not yet tested.
BW write payload vs E5 read payload are byte-identical around the anchor region (confirmed by comparing 3-11-26 BW TX and S3 captures). BW does NOT strip DLE prefix bytes before writing; it round-trips the wire-encoded bytes verbatim with only the modified fields changed.
Capture folder content catalogued — see CLAUDE.md "BW capture reference" table for a summary of all available protocol captures and their contents.

v0.12.3 — 2026-04-20

Added

Auto Call Home config protocol — Full read/write/decode/encode pipeline for the device's Remote Access → Setup Unit ACH settings, confirmed from 4-20-26 call home settings captures.

Protocol (new):
- SUB 0x2C — Call Home Config READ (response 0xD3); two-step read; data offset 0x7C = 124; raw payload 125 bytes (1-byte longer than DATA_LENGTH due to DLE-escaped \x10\x03 at raw[117:119] representing num_retries = 3)
- SUB 0x7E — Call Home Config WRITE (response 0x81); 127-byte payload (125-byte read payload + \x00\x00); offset = data[1]+2 = 0x7E; write format (DLE-aware checksum)
- SUB 0x7F — Call Home WRITE CONFIRM (response 0x80); no data
Field map (confirmed from 10-frame BW TX diff):
- raw[5] — auto_call_home_enabled (bool)
- raw[6:46] — dial_string (40-byte null-padded ASCII)
- raw[87] — after_event_recorded (bool)
- raw[91] — at_specified_times (bool)
- raw[93] — time1_enabled / raw[101] — time1_hour / raw[102] — time1_min
- raw[95] — time2_enabled / raw[105] — time2_hour / raw[106] — time2_min
- raw[117:119] — \x10\x03 (DLE-escaped 0x03 = num_retries value 3)
- raw[120] — time_between_retries_sec / raw[122] — wait_for_connection_sec / raw[124] — warm_up_time_sec
Library (minimateplus/):
- models.py — CallHomeConfig dataclass (14 fields; raw bytes preserved for round-trip writes)
- protocol.py — SUB_CALL_HOME = 0x2C, SUB_CALL_HOME_WRITE = 0x7E, SUB_CALL_HOME_CONFIRM = 0x7F; read_call_home_config(), write_call_home_config()
- client.py — get_call_home_config(), set_call_home_config(), _decode_call_home_config() (handles DLE prefix at raw[117]), _encode_call_home_config() (patches in-place; raises ValueError if hour/min = 3)
REST API (sfm/server.py):
- GET /device/call_home — reads and decodes call home config from device
- POST /device/call_home — reads, patches specified fields, writes back to device
- CallHomeConfigBody Pydantic model with 9 optional writable fields
Web UI (sfm/sfm_webapp.html):
- New "Call Home" tab with enable flag, dial string (read-only), after-event trigger, at-specified-times flag, two time slots (enable + HH:MM each), and read-only retry settings (num_retries, time_between_retries_sec, wait_for_connection_sec, warm_up_time_sec)
- "Read from Device", "Write to Device", "Clear Form" action buttons
- Client-side guard: rejects hour or minute value equal to 3 with a clear message explaining the DLE-encoding limitation

v0.12.2 — 2026-04-20

Added / Fixed

Geophone sensitivity / maximum range field confirmed — 4-20-26 geo sensitivity captures (1.25 in/s vs 10 in/s) diffed across all three SUB 71 write chunks and both E5 read payloads. The geo_range uint8 field per channel is now fully confirmed:
- E5 read offset: channel_label + 33; SUB 71 write offset: channel_label + 29
- 0x00 = Normal 10.000 in/s (standard gain); 0x01 = Sensitive 1.250 in/s (high gain)
- Correction: previous hypothesis (channel_label+20, 0x01=Normal) was wrong. channel_label+20 reads 0x01 on ALL captures regardless of range — not this field.
- _decode_compliance_config_into: read offset corrected from tran_pos+20 → tran_pos+33
- _encode_compliance_config: added geo_range parameter; writes to Tran/Vert/Long at +29
- apply_config: added geo_range parameter
- POST /device/config: added geo_range to DeviceConfigBody
- Web UI Config tab: added "Maximum Range — Geo" select (Normal / Sensitive)
- Web UI Device tab: added "Max Range (geo)" row to compliance table
recording_mode + histogram_interval_sec confirmed and implemented (4-20-26 captures)
- recording_mode: uint8 at anchor−8 (E5 read) / anchor−7 (write); enum: 0x00=Single Shot, 0x01=Continuous, 0x03=Histogram, 0x04=Histogram+Continuous
- histogram_interval_sec: uint16 BE seconds at anchor−4; same offset in read & write; valid: 2, 5, 15, 60, 300, 900 (matching Blastware dropdown: 2s, 5s, 15s, 1m, 5m, 15m)
- Both fields added to ComplianceConfig, _decode_compliance_config_into, _encode_compliance_config, apply_config, REST API body, and web UI

v0.12.1 — 2026-04-16

Added

sfm/server.py — _LiveCache — in-memory live device cache that eliminates redundant TCP round-trips between web requests. Plain Python dict + threading.Lock, no extra dependencies.

Cache strategy per endpoint:

Endpoint	Strategy
`GET /device/info`	Indefinite; invalidated by `POST /device/config`
`GET /device/events`	Count-probe fast path — `poll()+count_events()` (~2 s); returns cached data if event count is unchanged; full download only when new events are detected
`GET /device/monitor/status`	30-second TTL; invalidated immediately on monitor start/stop
`GET /device/event/{idx}/waveform`	Permanent per-index (waveforms are immutable once recorded)

?force=true query param on all cached endpoints — bypasses cache and forces a fresh read from the device.
Cache invalidation hooks — POST /device/config marks device info and events stale; POST /device/monitor/start and /stop evict the monitor status entry immediately so the next status poll reflects the actual device state.

v0.12.0 — 2026-04-13

Added

sfm/server.py — _LiveCache — in-memory live device cache, eliminating redundant TCP round-trips between requests. No extra dependencies (plain Python dict + threading.Lock). Replaces the SQLAlchemy-based sfm/cache.py experiment from the feature/intelligent-caching branch.

Cache behaviour by endpoint:

Endpoint	Cache strategy
`GET /device/info`	Indefinite; invalidated by `POST /device/config`
`GET /device/events`	Count-probe fast path: quick `poll()+count_events()` (~2s); return cache if count matches; full download only when new events detected
`GET /device/monitor/status`	30-second TTL; invalidated by monitor start/stop
`GET /device/event/{idx}/waveform`	Permanent per-index (waveforms are immutable)

?force=true param on all four cached endpoints — bypasses cache and re-reads from device.
POST /device/config cache invalidation — marks device info + events dirty so the next read reflects the new compliance config.
POST /device/monitor/start / stop cache invalidation — evicts the monitor status cache entry immediately so the next poll returns the updated state.

Removed

sfm/cache.py — SQLAlchemy-based cache from the experimental caching branch. Its logic has been ported to the sqlite3-native _LiveCache class above. sqlalchemy is no longer a dependency.

v0.11.0 — 2026-04-13

Added

sfm/database.py — SeismoDb — SQLite persistence layer for all ACH data. Three tables, all unit-keyed by serial number:
- ach_sessions — one row per inbound call-home: serial, timestamp, peer IP, events_downloaded, monitor_entries, duration_seconds
- events — one row per triggered waveform event: serial, waveform_key (dedup), timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location strings, sample_rate, record_type, false_trigger flag
- monitor_log — one row per monitoring interval: serial, waveform_key (dedup), start_time, stop_time, duration_seconds, geo_threshold_ips
- WAL mode, per-request connections — safe for the single-writer / occasional-reader ACH server pattern
- Deduplication by (serial, waveform_key) UNIQUE constraint — re-runs and repeat call-homes never produce duplicate rows
ach_server.py — DB integration — after each successful call-home, writes new events and monitor log entries to seismo_relay.db then records the session in ach_sessions. DB write failures are logged as warnings and do not abort the session.
sfm/server.py — DB read endpoints:
- GET /db/units — distinct serials with last_seen, total_events, total_monitor_entries
- GET /db/events — query events with serial / date range / false_trigger filters
- GET /db/monitor_log — query monitoring intervals
- GET /db/sessions — query ACH call-home sessions
- PATCH /db/events/{id}/false_trigger — flag/unflag false triggers (for review UI)

Architecture

seismo-relay DB is unit-keyed only — no project concepts. Project aggregation is terra-view's responsibility via UnitAssignment / DeploymentRecord + date range queries against the SFM DB endpoints.
DB file lives at bridges/captures/seismo_relay.db by default.

v0.10.0 — 2026-04-11

Added

MiniMateClient.get_monitor_log_entries(skip_keys=None) — browse-mode walk (1E → 0A → 1F) that collects partial records (0x2C record type) from the device's event list without triggering a full waveform download (no 0C or 5A). Returns list[MonitorLogEntry]. Each entry represents one continuous monitoring interval where no threshold was exceeded.
_decode_0a_partial_header(raw_data, index, key4) in client.py — decodes a SUB 0x0A response payload whose record type is 0x2C. Extracts:
- start_time / stop_time — two consecutive timestamps; auto-detects 9-byte (sub_code=0x10, single-shot) vs 10-byte (sub_code=0x03, continuous) format from raw_data[11]. Handles a 1-byte gap between the two timestamps that occurs when ts1 and ts2 share the same minute:second.
- serial — device serial string found via b"BE" anchor scan.
- geo_threshold_ips — trigger level found via b"Geo: " anchor scan.
MonitorLogEntry dataclass in models.py — new model for partial records: index, key, start_time, stop_time, serial, geo_threshold_ips, raw_header, and a duration_seconds property.
read_waveform_header() return value extended — now returns (data_rsp.data, length) (full payload) instead of (data_rsp.data[11:11+length], length). Callers get the complete payload including the record-type byte at position 0. Full records use raw_data[11:11+length] as before; partial records are detected by raw_data[0] == 0x2C.
ACH server: monitor log collection — after get_events(), calls get_monitor_log_entries(skip_keys=seen_keys) and saves new entries to monitor_log.json in the session directory. Monitor log keys are included in downloaded_keys for state persistence (no re-processing on next call-home).
_monitor_log_entry_to_dict() in ach_server.py — serialises a MonitorLogEntry to a JSON-compatible dict with ISO-format timestamps.

Protocol / Documentation

SUB 0x0A partial record (0x2C) format confirmed (✅ 4-11-26 MITM capture, 12 frames):
- Record type 0x2C at raw_data[0]; length < 64 bytes.
- Two timestamps at raw_data[11:] — start and stop of the monitoring interval.
- ASCII metadata region after timestamps: BE<serial>\x00Geo: <float> in/s.
- Edge case: 1-byte separator between timestamps when ts1 and ts2 share minute:second.
- 10-byte timestamp format (sub_code=0x03) signalled by raw_data[11] == 0x10.
Key reuse detection for monitor log entries — monitor log keys are tracked alongside event keys in ach_state.json so the ACH server does not re-process them after a call-home cycle.

v0.9.0 — 2026-04-11

Added

MiniMateClient.list_event_keys() — fast browse-mode walk (1E → 0A → 1F, no waveform download) that returns the list of event key hex strings currently stored on the device. Used by the ACH server as a cheap pre-check before deciding whether to call get_events().
get_events(skip_waveform_for_keys=set(...)) — new optional parameter. For any key in the set the function performs only 0A + 1F(browse) instead of the full 1E-arm → 0C → POLL×3 → 5A sequence. Eliminates redundant waveform downloads on repeat call-homes when the device still holds previously downloaded events.
MiniMateClient.delete_all_events() — erases all events from device memory using the confirmed 4-step sequence:
- SUB 0xA3 begin_erase_all — initiate erase (token=0xFE) → ack 0x5C
- SUB 0x1C read_monitor_status — intermediate status read (Blastware-required)
- SUB 0x06 read_event_storage_range — verify storage state (token=0xFE) → 36-byte response
- SUB 0xA2 confirm_erase_all — commit erase (token=0xFE) → ack 0x5D
All four steps confirmed from 4-11-26 MITM capture of a live Blastware ACH session. After a successful call, the device's event counter resets to 0x01110000.
MiniMateProtocol erase methods: begin_erase_all(), confirm_erase_all(), read_event_storage_range() added to protocol.py with documented SUB constants SUB_ERASE_ALL_BEGIN = 0xA3 and SUB_ERASE_ALL_CONFIRM = 0xA2.
bridges/ach_mitm.py — transparent TCP-to-TCP MITM proxy. Listens for inbound unit connections, connects upstream to a real Blastware ACH server, and saves both directions to raw_bw_<ts>.bin / raw_s3_<ts>.bin files matching the existing capture format. Used to capture the 4-11-26 Blastware ACH session including event deletion. Usage: python bridges/ach_mitm.py --bw-host 127.0.0.1 --bw-port 9999 --listen-port 9998
ACH server: key-based state tracking — ach_state.json now stores downloaded_keys: [hex_strings] and max_downloaded_key: hex_string per unit instead of event_count: N. This correctly handles the standard workflow where events are deleted from the device after upload — a count-based approach would see count=0 on the next call-home and silently skip new events.
ACH server: --clear-after-download flag — after a successful download (at least one new event saved), erases all events from the device using delete_all_events(). Mirrors the standard Blastware ACH workflow. On success, downloaded_keys and max_downloaded_key are reset to empty so the next session starts fresh.
ACH server: post-erase key-reuse detection — after an external erase (Blastware or manual), device keys restart from 0x01110000, colliding with previously downloaded keys. On each browse walk, if max(device_keys) < max_downloaded_key (device counter rolled back), all device keys are treated as new regardless of seen_keys. This also catches erases performed by Blastware between our sessions.

Protocol / Documentation

SUB 0xA3 / SUB 0xA2 — erase-all sequence confirmed (✅ 4-11-26 MITM capture): Both frames use token=0xFE at params[7] and are standard build_bw_frame requests (not write-format). Response SUBs follow the standard formula: 0x5C and 0x5D. The intermediate 0x1C + 0x06 reads between them are required by Blastware.
SUB 0x06 — event storage range read confirmed (✅ 4-11-26 MITM capture): Two-step read, data offset = 0x24 (36 bytes). The last 8 bytes of the response contain the first and last stored event keys (4 bytes each). After a successful erase, both keys read as 01110000 (device-empty state).
Event key counter resets to 0x01110000 after erase — confirmed by observing key 01110000 on the device immediately after the MITM erase session.

v0.8.0 — 2026-04-07

Added

Write pipeline end-to-end — push_config_raw(event_index_data, compliance_data, trigger_data, waveform_data) on MiniMateClient orchestrates the full 68→73 | 71×3→72 | 82→83 | 69→74→72 write sequence.
build_bw_write_frame(sub, data, *, offset, params) in framing.py — dedicated frame builder for write commands (SUBs 0x68–0x83). Doubles only the BW_CMD byte; all other bytes including offset, params, data, and checksum are written raw. Uses the large-frame DLE-aware checksum (sum(b for b in payload[2:] if b != 0x10) + 0x10) & 0xFF).
MiniMateProtocol write methods — write_event_index(), write_compliance(), write_trigger_config(), write_waveform_data(), write_confirm(), start_monitoring(), stop_monitoring().
AchSession inbound server (bridges/ach_server.py) — accepts call-home TCP connections, runs the full handshake + device-info + event-download sequence, saves device_info.json + events.json per session.

Protocol / Documentation

Write frame format confirmed (✅ 3-11-26 BW TX capture, all 11 frames): only BW_CMD byte 0x10 is doubled; all other bytes sent raw. Standard build_bw_frame DLE-stuffing is incorrect for write commands.
Write ack responses confirmed as 17-byte zero-data S3 frames.
Monitoring SUBs 0x96/0x97 confirmed from 4-8-26 capture.
SESSION_RESET signal (41 03) required before POLL for monitoring units.
SUB 0x1C monitoring flag at section[1]: 0x00 = idle, 0x10 = monitoring. Confirmed by byte-diff of all 144 data frames in 4-8-26/2ndtry capture.

v0.7.0 — 2026-04-03

Added

Raw ADC waveform decode — _decode_a5_waveform(frames_data, event) in client.py. Parses the complete set of SUB 5A A5 response frames into per-channel time-series:
- Reads the STRT record from A5[0] (bytes 7+): extracts total_samples (BE uint16 at +8), pretrig_samples (BE uint16 at +16), and rectime_seconds (uint8 at +18) into event.total_samples / pretrig_samples / rectime_seconds.
- Skips the 6-byte preamble (00 00 ff ff ff ff) that follows the 21-byte STRT header; waveform data begins at strt_pos + 27.
- Strips the 8-byte per-frame counter header from A5[1–6, 8] before appending waveform bytes.
- Skips A5[7] (metadata-only) and A5[9] (terminator).
- Cross-frame alignment correction: accumulates running_offset % 8 across all frames and discards (8 − align) % 8 leading bytes per frame to re-align to a T/V/L/M boundary. Required because individual frame waveform payloads are not always multiples of 8 bytes.
- Decodes as 4-channel interleaved signed 16-bit LE at 8 bytes per sample-set: bytes 0–1 = Tran, 2–3 = Vert, 4–5 = Long, 6–7 = Mic.
- Stores result in event.raw_samples = {"Tran": [...], "Vert": [...], "Long": [...], "Mic": [...]}.
download_waveform(event) public method on MiniMateClient. Issues a full SUB 5A stream with stop_after_metadata=False, then calls _decode_a5_waveform() to populate event.raw_samples and event.total_samples / pretrig_samples / rectime_seconds. Previously only metadata frames were fetched during get_events(); raw waveform data is now available on demand.
Event model new fields (models.py): total_samples, pretrig_samples, rectime_seconds (from STRT record), and _waveform_key (4-byte key stored during get_events() for later use by download_waveform()).

Protocol / Documentation

SUB 5A A5[0] STRT record layout confirmed (✅ 2026-04-03, 4-2-26 blast capture):
- STRT header is 21 bytes: b"STRT" + length fields + total_samples (BE uint16 at +8) + pretrig_samples (BE uint16 at +16) + rectime_seconds (uint8 at +18).
- Followed by 6-byte preamble: 00 00 ff ff ff ff. Waveform begins at strt_pos + 27.
- Confirmed: 4-2-26 blast → total_samples=9306, pretrig_samples=298, rectime_seconds=70.
Blast/waveform mode A5 format confirmed (✅ 2026-04-03, 4-2-26 blast capture): 4-channel interleaved int16 LE at 8 bytes per sample-set; cross-frame alignment correction required. 948 of 9306 total sample-sets captured via stop_after_metadata=True (10 frames).
Noise/histogram mode A5 format — endianness corrected (✅ 2026-04-03, 3-31-26 capture): 32-byte block samples are signed 16-bit little-endian (previously documented as BE). 0a 00 → LE int16 = 10 (correct noise floor); BE would give 2560 (wrong).
Protocol reference §7.6 rewritten — split into §7.6.1 (Blast/Waveform mode) and §7.6.2 (Noise/Histogram mode), each with confirmed field layouts and open questions noted.

v0.6.0 — 2026-04-02

Added

True event-time metadata via SUB 5A bulk waveform stream — get_events() now issues a SUB 5A request after each SUB 0C download, reads the A5 response frames, and extracts the Client:, User Name:, and Seis Loc: fields as they existed at the moment the event was recorded. Previously these fields were backfilled from the current compliance config (SUB 1A), which reflects today's setup, not the setup active when the event triggered.
- build_5a_frame(offset_word, raw_params) in framing.py — reproduces Blastware's exact wire format for SUB 5A requests: raw (non-DLE-stuffed) offset_hi, DLE-stuffed params, and a DLE-aware checksum where 10 XX pairs count only XX.
- bulk_waveform_params() returns 11 bytes (extra trailing 0x00 confirmed from 1-2-26 BW wire capture).
- read_bulk_waveform_stream(key4, *, stop_after_metadata=True, max_chunks=32) in protocol.py — loops sending chunk requests (counter increments 0x0400 per chunk), stops early when b"Project:" is found, then sends a termination frame.
- _decode_a5_metadata_into(frames_data, event) in client.py — needle-searches A5 frame data for Project:, Client:, User Name:, Seis Loc:, Extended Notes and overwrites event.project_info.
get_events() sequence extended — now 1E → 0A → 0C → 5A → 1F per event.

Fixed

Compliance config (SUB 1A) channel block missing — orphaned self._send(build_bw_frame(SUB_COMPLIANCE, 0x2A, _DATA_PARAMS)) before the B/C/D receive loop had no corresponding recv_one(), shifting all subsequent receives one step behind and leaving frame D's channel-block data (trigger_level_geo, alarm_level_geo, max_range_geo) unread. Removed the orphaned send. Total config bytes received now correctly ~2126 (was ~1071).
Compliance config anchor search range — _decode_compliance_config_into() searched cfg[40:100] for the sample-rate/record-time anchor. With the orphaned-send bug fixed the 44-byte padding it had been adding is gone, and the anchor now appears at cfg[11]. Search widened to cfg[0:150] to be robust to future layout shifts.
Removed byte-content deduplication from read_compliance_config() — was masking the real receive-ordering bug.

Protocol / Documentation

SUB 5A frame format confirmed — offset_hi byte (0x10) must be sent raw (not DLE-stuffed); checksum is DLE-aware (only the second byte of a 10 XX pair is summed). Standard build_bw_frame DLE-stuffs 0x10 incorrectly for 5A — a dedicated build_5a_frame is required.
Event-time metadata source confirmed — Client:, User Name:, and Seis Loc: strings are present in A5 frame 7 of the bulk waveform stream (SUB 5A), not in the 210-byte SUB 0C waveform record. They reflect the compliance setup as it was when the event was stored on the device.

v0.5.0 — 2026-03-31

Added

Console tab in seismo_lab.py — direct device connection without the bridge subprocess.
- Serial and TCP transport selectable via radio buttons.
- Four one-click commands: POLL, Serial #, Full Config, Event Index.
- Colour-coded scrolling output: TX (blue), RX raw hex (teal), parsed/decoded (green), errors (red).
- Save Log and Send to Analyzer buttons; logs auto-saved to bridges/captures/console_<ts>.log.
- Queue/after(100) pattern — no UI blocking or performance impact.
minimateplus package — clean Python client library for the MiniMate Plus S3 protocol.
- SerialTransport and TcpTransport (for Sierra Wireless RV50/RV55 cellular modems).
- MiniMateProtocol — DLE frame parser/builder, two-step paged reads, checksum validation.
- MiniMateClient — high-level client: connect(), get_serial(), get_config(), get_events().
TCP/cellular transport (TcpTransport) — connect to field units via Sierra Wireless RV50/RV55 modems over cellular.
- read_until_idle(idle_gap=1.5s) to handle modem data-forwarding buffer delay.
- Confirmed working end-to-end: TCP → RV50/RV55 → RS-232 → MiniMate Plus.
bridges/tcp_serial_bridge.py — local TCP-to-serial bridge for bench testing TcpTransport without a cellular modem.
SFM REST server (sfm/server.py) — FastAPI server with device info, event list, and event record endpoints over both serial and TCP.

Fixed

protocol.py startup() was using a hardcoded POLL_RECV_TIMEOUT = 10.0 constant, ignoring the configurable self._recv_timeout. Fixed to use self._recv_timeout throughout.
sfm/server.py now retries once on ProtocolError for TCP connections to handle cold-boot timing on first connect.

Protocol / Documentation

Sierra Wireless RV50/RV55 modem config — confirmed required ACEmanager settings: Quiet Mode = Enable, Data Forwarding Timeout = 1, TCP Connect Response Delay = 0. Quiet Mode disabled causes modem to inject RING\r\nCONNECT\r\n onto the serial line, breaking the S3 handshake.
Calibration year confirmed at SUB FE (Full Config) destuffed payload offset 0x56–0x57 (uint16 BE). 0x07E7 = 2023, 0x07E9 = 2025.
"Operating System" boot string — 16-byte UART boot message captured on cold-start before unit enters DLE-framed mode. Parser handles correctly by scanning for DLE+STX.
RV50/RV55 sends RING/CONNECT over TCP to the calling client even with Quiet Mode enabled — this is normal behaviour, parser discards it.

v0.4.0 — 2026-03-12

Added

seismo_lab.py — combined Bridge + Analyzer GUI. Single window with two tabs; bridge start auto-wires live mode in the Analyzer.
frame_db.py — SQLite frame database. Captures accumulate over time; Query DB tab searches across all sessions.
bridges/s3-bridge/proxy.py — bridge proxy module.
Large BW→S3 write frame checksum algorithm confirmed and implemented (SUM8 of payload [2:-1] skipping 0x10 bytes, plus constant 0x10, mod 256).
SUB A4 identified as composite container frame with embedded inner frames; _extract_a4_inner_frames() and _diff_a4_payloads() reduce diff noise from 2300 → 17 meaningful entries.

Fixed

BAD CHK false positives on BW POLL frames — BW frame terminator 03 41 was being included in the de-stuffed payload. Fixed to strip correctly.
Aux Trigger read location confirmed at SUB FE offset 0x0109.

v0.3.0 — 2026-03-09

Added

Record time confirmed at SUB E5 page2 offset +0x28 as float32 BE.
Trigger Sample Width confirmed at BW→S3 write frame SUB 0x82, destuffed payload offset [22].
Mode-gating documented: several settings only appear on the wire when the appropriate mode is active.

Fixed

0x082A mystery resolved — fixed-size E5 payload length (2090 bytes), not a record-time field.

v0.2.0 — 2026-03-01

Added

Channel config float layout fully confirmed: trigger level, alarm level, and unit string per channel (IEEE 754 BE floats).
Blastware .set file format decoded — little-endian binary struct mirroring the wire payload.
Operator manual (716U0101 Rev 15) added as cross-reference source.

v0.1.0 — 2026-02-26

Added

Initial s3_bridge.py serial bridge — transparent RS-232 tap between Blastware and MiniMate Plus.
s3_parser.py — deterministic DLE state machine frame extractor.
s3_analyzer.py — session parser, frame differ, Claude export.
gui_bridge.py and gui_analyzer.py — Tkinter GUIs.
DLE framing confirmed: DLE+STX / DLE+ETX, 0x41 = ACK (not STX), DLE stuffing rule.
Response SUB rule confirmed: response_SUB = 0xFF - request_SUB.
Year 0x07CB = 1995 confirmed as MiniMate factory RTC default.
Full write command family documented (SUBs 68–83).

91 KiB Raw Blame History Unescape Escape

Changelog

[Unreleased]

v0.21.1 — 2026-06-01

Fixed

Operator

v0.21.0 — 2026-05-29

Added — Thor IDF binary codec (micromate/idf_file.read_idf_file)

Added — Thor → BW report adapter (micromate/idf_to_bw_report.py)

Companion releases

Migration / Operations

v0.20.0 — 2026-05-28

Added (2026-05-28)

Fixed (2026-05-28)

Docs (2026-05-28)

Migration / Operations

Pre-v0.20.0 work (rolled into this release)

Fixed

Added

Fixed

Docs

v0.19.0 — 2026-05-20

Added — Phase 1: device_family column on events

Added — Phase 2: micromate/ package alongside minimateplus/

Companion releases

v0.18.0 — 2026-05-19

Added — Thor (Series IV) IDF ingest

Changed

Companion release

v0.17.0 — 2026-05-17

Added — wedged-unit recovery toolkit

Added — operator event DB management

Changed

Fixed

Migration

Pairing

v0.16.1 — 2026-05-14

Fixed

v0.16.0 — 2026-05-11

Added — /db/import/blastware_file rich-metadata ingestion

Fixed — three compounding bugs that left forwarded events with garbage data

Migration

v0.15.0 — 2026-05-07

Added

Dependencies

v0.14.3 — 2026-05-05

Fixed

Notes

v0.14.2 — 2026-05-04

Fixed

v0.14.0 — 2026-05-02

Changed (major rewrite)

Verified

Notes

v0.14.1 — 2026-05-04

Fixed

v0.13.2 — 2026-05-01

Fixed

Added

v0.13.1 — 2026-05-01

Fixed

v0.13.0 — 2026-05-01

Fixed

Added

Documentation

Known minor diffs vs Blastware (deferred to a follow-up)

v0.12.6 — 2026-05-01

Fixed

v0.12.5 — 2026-04-21

Added

Changed

v0.12.4 — 2026-04-21 (protocol analysis / docs only — no code changes)

Discovered

v0.12.3 — 2026-04-20

Added

v0.12.2 — 2026-04-20

Added / Fixed

v0.12.1 — 2026-04-16

Added

v0.12.0 — 2026-04-13

91 KiB

Raw Blame History

Added — Thor IDF binary codec (`micromate/idf_file.read_idf_file`)

Added — Thor → BW report adapter (`micromate/idf_to_bw_report.py`)

Added — Phase 1: `device_family` column on `events`

Added — Phase 2: `micromate/` package alongside `minimateplus/`

Added — `/db/import/blastware_file` rich-metadata ingestion