feat(forward): SFM event forwarder (v1.5.0) #8
Assignees
serversdown
Clear assignees
serversdown
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: serversdown/series3-watcher#8
Reference in New Issue
Block a user
Delete Branch "sfm-event-forwarder"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
When SFM_FORWARD_ENABLED=true and SFM_URL is set, every Blastware event binary in the ACH watch folder is forwarded to an SFM server's /db/import/blastware_file endpoint as a multipart POST. The paired <binary>.TXT ASCII report (which Blastware's ACH writes alongside each event) is shipped in the same request, letting the SFM server index the full per-channel stats — PPV, ZC Freq, Time of Peak, Peak Acceleration / Displacement, Peak Vector Sum + time, sensor self-check Pass/Fail per channel, and monitor-log timestamps — without depending on the still-undecoded BW waveform body codec. New module event_forwarder.py: - is_event_binary() filename matcher (BW's <P><serial3><stem>.<ext> scheme; rejects .MLG, .TXT, .log, .ini, .h5, etc.) - ForwardState (.json file keyed by sha256 — idempotent across restarts and auto-updates) - find_pending_events() with quiescence + grace-period guards - Hand-rolled multipart encoder (stdlib-only) - forward_event_pair() / forward_pending() — POST loop with structured per-event outcomes Wired into series3_watcher.run_watcher() on its own cadence (SFM_FORWARD_INTERVAL_SECONDS, default 60s) so it doesn't slow the existing 5-min heartbeat scan. Default-off: existing 1.4.x deployments keep their old behaviour after auto-updating until an operator sets SFM_URL + SFM_FORWARD_ENABLED=true and restarts. 17 unit tests in test_event_forwarder.py cover filename matching, state idempotency, scan logic (quiescence, grace, max age, already-forwarded, .TXT pairing), multipart byte shape, and an end-to-end POST against a tiny stdlib http.server fake. Bumps version 1.4.4 → 1.5.0 (minor — additive feature, no API break). Requires SFM server v0.16+ for the paired-.TXT import endpoint.v1.5.0 shipped the forwarder module + INI keys but the settings dialog wasn't extended, so the only way operators could enable forwarding was hand-editing config.ini. This adds a sixth tab ("SFM Forward") with: - Forward events to SFM checkbox - SFM Server URL entry + Test button (GETs /health) - Forward Interval (sec) spinbox - Quiescence (sec) spinbox - Missing-Report Grace spinbox - HTTP Timeout spinbox - State File entry + Browse... Save-time guard: enabling the forwarder without a URL raises a validation error rather than silently saving a non-functional config. Patch release — same on-disk INI schema, no config migration.Two safety nets for first-deploy on Blastware ACH machines that have accumulated tens or hundreds of thousands of historical events in the watch folder. 1. SFM_MAX_FORWARDS_PER_PASS (default 500, 0=unlimited) --------------------------------------------------- Cap on the number of events forwarded per scan tick. At the 60-second default interval that's ~30K events/hour throughput — the SFM server gets a steady drip instead of one giant burst. Scan now sorts by mtime ascending so backfill advances chronologically (oldest first) and successive scans always make progress instead of re-considering the same N newest files. Wired into: - event_forwarder.find_pending_events / forward_pending - series3_watcher.run_watcher loop - config-template.ini - settings_dialog SFM Forward tab (new "Max Events Per Pass" spinbox, validated in _on_save) 2. event_forwarder.py --seed-state CLI ----------------------------------- One-shot mode that walks the watch folder, sha256s every in-window event binary, and marks them all as already-forwarded WITHOUT POSTing anything. Run before flipping SFM_FORWARD_ENABLED=true to skip the historical backfill entirely — the watcher then only forwards events that appear AFTER the seed. Usage: python event_forwarder.py --seed-state \ --watch "C:\Blastware 10\Event\autocall home" \ --state "C:\...\sfm_forwarded.json" \ [--max-age-days 365] 7 new unit tests: - max_per_pass cap enforcement (=N, =0 unlimited, oldest-first ordering) - seed-state mode (in-window seeding, max-age skip, end-to-end skip-after-seed, idempotent re-runs) README adds a "First-time deployment" section walking through both options. Bumps to v1.5.2.On machines running histogram-mode units (extensions ending in H, e.g. H907L1R7.PG0H), every forwarded event was logging "no report" even though histograms never get auto-exported ASCII reports from Blastware — making the log look like every forward was misconfigured when in fact things were working correctly. Three log states now: - Waveform + paired TXT → "+ <txt> attached" - Waveform without TXT (likely BW config issue) → "no report ⚠" - Histogram (any flavour) → "(histogram, no report expected)" New is_histogram_event() helper classifies by BW filename extension: 4-char ext ending in H = histogram; old-firmware 3-char extensions default to non-histogram (safe default — we'd rather flag a missing report than silently suppress the warning on a real waveform event). Forwarding logic itself is unchanged — this is purely log clarity.Blastware's official Auto Call Home server writes per-event ASCII reports as <stem>_<ext>_ASCII.TXT (e.g. N844L20G_630H_ASCII.TXT), not <binary>.TXT (e.g. N844L20G.630H.TXT). Versions v1.5.0–v1.5.3 only looked for the latter and silently shipped every binary alone, so the SFM database lost the per-event Peak Acceleration / Peak Displacement / ZC Freq / Time of Peak / Peak Vector Sum + time / sensor self-check fields on every forwarded event. Fix: pair-finding logic now tries the ACH-convention filename first and falls back to <binary>.TXT for compatibility with operator-saved manual exports and existing test fixtures. ach_report_name("M529LK44.AB0") → "M529LK44_AB0_ASCII.TXT" legacy_report_name("M529LK44.AB0") → "M529LK44.AB0.TXT" When both files exist (operator manually saved + ACH auto-exported), ACH wins because that's the canonical name on modern BW deployments. Both candidates checked case-insensitively against the cached directory listing — no extra stat() calls. 6 new unit tests cover the new pairing logic, helper-function correctness, and the precedence rule. Total now 31 tests, all green. Field-deploy note: re-running v1.5.4 on a folder where v1.5.0–v1.5.3 already ran will NOT re-forward historical events — the sfm_forwarded.json state file remembers them by sha256. To re-forward historical events to populate SFM with the now-correctly-paired reports, delete the state file before starting v1.5.4.