serversdown/series3-watcher
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity

5 Releases 5 Tags

RSS Feed
  • v1.5.0 a457619158
    Compare

    v1.5.0 - SFM forwarder. Stable

    serversdown released this 2026-05-11 12:37:32 -04:00 | 0 commits to main since this release

    [5-11-26] — v1.5.0

    First release of the SFM event forwarder.

    Added — SFM event forwarder

    • Forward Blastware event binaries (+ paired BW ACH ASCII reports) to an SFM server. When SFM_FORWARD_ENABLED=true and SFM_URL is set, every event binary in the BW ACH watch folder is POSTed as multipart to /db/import/blastware_file along with its <stem>_<ext>_ASCII.TXT partner report (BW ACH convention; manual-export <binary>.TXT is also supported as a fallback). SFM parses the report and indexes the full per-channel stats (PPV, ZC Freq, Time of Peak, Peak Acceleration / Displacement, Peak Vector Sum + time, sensor self-check Pass/Fail, monitor-log timestamps) into a searchable database — no codec decoding required.
    • Idempotent forwarding. Forwarded files are tracked by sha256 in a JSON state file (default <log dir>/sfm_forwarded.json, override via SFM_STATE_FILE). Re-scans don't re-POST and the state survives restarts / auto-updates.
    • Re-pair on late-arriving TXT. When the watcher forwards a binary alone (_ASCII.TXT partner didn't appear within SFM_MISSING_REPORT_GRACE_SECONDS), the state file records had_report: false. On subsequent scans, the watcher re-checks whether the TXT has since arrived. If yes, the event is re-forwarded with the TXT attached — the SFM server's upsert path refreshes the DB row with the report's device-authoritative peak / project values. Without this, slow-disk or AV-interrupted TXT writes would permanently leave that event with broken-codec peaks in the SFM database. Legacy state-file entries (without the had_report field) default to had_report: true so an upgrade doesn't unexpectedly re-forward existing entries.
    • Quiescence + grace-period guards. Files modified within SFM_QUIESCENCE_SECONDS (default 5s) are skipped to avoid forwarding mid-write. If a binary's report partner hasn't appeared after SFM_MISSING_REPORT_GRACE_SECONDS (default 60s), the binary is forwarded alone rather than blocking forever.
    • Per-pass rate cap. SFM_MAX_FORWARDS_PER_PASS (default 500) drips first-deploy backfill instead of hammering the SFM server in one burst. At 60-second SFM_FORWARD_INTERVAL_SECONDS cadence that's ~30K events/hour throughput. Set to 0 for unlimited. Scan walks oldest-first so backfill advances chronologically and successive scans reliably progress.
    • event_forwarder.py --seed-state CLI mode. Walks the watch folder once, sha256s every in-window event, and marks them all as already-forwarded without POSTing anything. Recommended pre-deploy workflow on machines with a large historical archive — flip SFM_FORWARD_ENABLED=true after seeding and only events that appear from then on get forwarded.
    • SFM Forward tab in the Settings dialog with: Forward checkbox, SFM URL + Test button (GETs /health), Forward Interval / Quiescence / Missing-Report Grace / HTTP Timeout / Max Events Per Pass spinboxes, State File entry with Browse... Save-time guard: enabling forwarding without a URL shows a validation error.
    • Histogram-aware log clarity. Histogram events (extensions ending in H) don't get auto-exported reports from BW; the log distinguishes that case ((histogram, no report expected)) from a waveform with unexpectedly missing report (no report ⚠).
    • README "First-time deployment" section documenting the seed-state workflow + the rate cap as belt-and-suspenders for safe rollout on machines with hundreds of thousands of historical events.
    • 31 new unit tests in test_event_forwarder.py covering filename matching, state idempotency, scan logic (quiescence / grace period / max age / already-forwarded / TXT pairing), multipart byte shape, rate cap (oldest-first, cap=0 unlimited, cap=N enforcement), seed-state mode (in-window seeding / max-age skip / end-to-end skip-after-seed / idempotent re-runs), histogram classification, and an end-to-end POST against a stdlib fake server.

    Configuration

    New [agent] keys (all default-off — existing 1.4.x deployments don't change behaviour on auto-update):

    Key Default Notes
    SFM_FORWARD_ENABLED false Master toggle for the forwarder
    SFM_URL empty e.g. http://10.0.0.44:8200
    SFM_FORWARD_INTERVAL_SECONDS 60 Scan-and-forward cadence
    SFM_QUIESCENCE_SECONDS 5 Skip files modified in the last N seconds
    SFM_MISSING_REPORT_GRACE_SECONDS 60 Forward without TXT after this delay
    SFM_HTTP_TIMEOUT 60 Per-request HTTP timeout
    SFM_STATE_FILE <log dir>/sfm_forwarded.json Override location of the forwarded-sha256 state file
    SFM_MAX_FORWARDS_PER_PASS 500 Per-scan cap (0 = unlimited)

    Compatibility

    • Requires SFM server v0.16+ (the /db/import/blastware_file endpoint that accepts paired _ASCII.TXT reports + the BW-report label normalisation — released alongside this watcher version on the seismo-relay side).
    Downloads