feat(forward): SFM event forwarder (v1.5.0) #8

Merged
serversdown merged 7 commits from sfm-event-forwarder into dev 2026-05-11 12:10:45 -04:00
6 changed files with 30 additions and 64 deletions
Showing only changes of commit 19548466ad - Show all commits
+23 -57
View File
@@ -6,73 +6,39 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
---
## [1.5.4] - 2026-05-10
## [Unreleased] — v1.5.0
### Fixed
- **CRITICAL: Pair BW ACH ASCII reports using the `_ASCII.TXT` convention.** Blastware's official Auto Call Home server writes per-event ASCII reports as `<stem>_<ext>_ASCII.TXT` (e.g. `N844L20G_630H_ASCII.TXT`), not `<binary>.TXT` (e.g. `N844L20G.630H.TXT`). Versions v1.5.0v1.5.3 only looked for the latter and silently shipped every binary alone, so the SFM database lost the per-event Peak Acceleration / Peak Displacement / ZC Freq / Time of Peak / Peak Vector Sum + time / sensor self-check fields on every forwarded event. After this fix the watcher tries the ACH-convention filename first and falls back to the manual-export `<binary>.TXT` for compatibility with operator-saved exports + existing test fixtures.
First release of the SFM event forwarder.
### Changed
- New helper functions `ach_report_name()` and `legacy_report_name()` make the two filename conventions explicit and testable.
- 6 new unit tests covering both pairing conventions, the precedence-when-both-present rule (ACH wins), and helper-function correctness.
### Field-deploy note
Re-running v1.5.4 on a folder where v1.5.0v1.5.3 already ran will NOT re-forward historical events to pick up the rich metadata — the `sfm_forwarded.json` state file remembers them by sha256 and still considers them forwarded. If you want to re-forward to populate the SFM database with the now-correctly-paired reports for the historical archive, delete the state file before starting v1.5.4. Otherwise the fix only affects events appearing from v1.5.4 onward.
## [1.5.3] - 2026-05-10
### Changed
- **Forward log lines now distinguish histogram events from waveform-without-report.** Previously every event without a paired `.TXT` report logged "no report", which on machines running histogram-mode units (extensions ending in `H`, e.g. `H907L1R7.PG0H`) generated alarming-looking lines on every single event when the lack of report was actually completely normal — Blastware doesn't auto-export ASCII reports for histograms. Three log states now:
- **Waveform + paired TXT**: `+ <txt> attached`
- **Waveform without TXT**: `no report ⚠` (suggests checking BW's "Save Event Report" setting)
- **Histogram (any flavour)**: `(histogram, no report expected)`
- New `is_histogram_event(filename)` helper classifies BW filenames by extension (4-char ext ending in `H` = histogram; old-firmware 3-char extensions remain unclassifiable and default to non-histogram for safe defaults).
- 1 new unit test for histogram classification.
## [1.5.2] - 2026-05-10
### Added
- **`SFM_MAX_FORWARDS_PER_PASS` rate cap.** Default 500 events per scan tick (60-second interval = ~30K events/hour). Drips first-deploy backfill instead of hammering the SFM server with one giant burst on machines that have hundreds of thousands of historical events in the BW ACH folder. `0` = unlimited (preserves the 1.5.0 behaviour for ops who want it).
- **`event_forwarder.py --seed-state` CLI mode.** Walks the watch folder once, sha256s every in-window event, and marks them all as already-forwarded *without* POSTing anything. The recommended pre-deploy workflow on machines with a large historical archive — flip `SFM_FORWARD_ENABLED=true` after seeding and only events that appear *from then on* get forwarded.
- New "Max Events Per Pass" spinbox in the Settings dialog's SFM Forward tab.
- README "First-time deployment" section documenting both options.
- 7 new unit tests covering the cap (oldest-first ordering, cap=0 unlimited, cap=N enforcement) and the seed-state mode (skips out-of-window files, idempotent across re-runs, end-to-end skip-after-seed).
### Behaviour change
The scan loop now sorts entries by mtime ascending before walking, so backfill always advances chronologically (oldest qualifying event first). Without the cap the visible behaviour is identical; with the cap it means each scan reliably advances and we never get stuck re-considering the same N newest files.
## [1.5.1] - 2026-05-10
### Added
- **SFM Forward tab in the Settings dialog.** v1.5.0 shipped the `event_forwarder.py` module + INI keys but missed the GUI; operators had to edit `config.ini` by hand to enable forwarding. The settings dialog now exposes:
- **Forward events to SFM** checkbox
- **SFM Server URL** entry with a **Test** button (mirrors the Connection tab — GETs `/health` and shows the result)
- **Forward Interval / Quiescence / Missing-Report Grace / HTTP Timeout** spinboxes
- **State File** entry with a Browse... button (defaults to `<log dir>/sfm_forwarded.json` when blank)
- Save-time guard: enabling SFM Forward without filling in the URL shows a validation error rather than silently saving a non-functional config.
## [1.5.0] - 2026-05-09
### Added
- **SFM event forwarder.** When `SFM_FORWARD_ENABLED=true` and `SFM_URL` is set, every Blastware event binary is forwarded to an SFM server's `/db/import/blastware_file` endpoint as a multipart POST. The corresponding `<binary>.TXT` ASCII report (which Blastware's ACH writes alongside each event) is paired by filename and shipped in the same request, letting the SFM server index the full per-channel stats (PPV, ZC Freq, Time of Peak, Peak Acceleration / Displacement, Peak Vector Sum + time, sensor self-check Pass/Fail, monitor-log timestamps) without depending on the still-undecoded Blastware waveform body codec.
### Added — SFM event forwarder
- **Forward Blastware event binaries (+ paired BW ACH ASCII reports) to an SFM server.** When `SFM_FORWARD_ENABLED=true` and `SFM_URL` is set, every event binary in the BW ACH watch folder is POSTed as multipart to `/db/import/blastware_file` along with its `<stem>_<ext>_ASCII.TXT` partner report (BW ACH convention; manual-export `<binary>.TXT` is also supported as a fallback). SFM parses the report and indexes the full per-channel stats (PPV, ZC Freq, Time of Peak, Peak Acceleration / Displacement, Peak Vector Sum + time, sensor self-check Pass/Fail, monitor-log timestamps) into a searchable database — no codec decoding required.
- **Idempotent forwarding.** Forwarded files are tracked by sha256 in a JSON state file (default `<log dir>/sfm_forwarded.json`, override via `SFM_STATE_FILE`). Re-scans don't re-POST and the state survives restarts / auto-updates.
- **Quiescence + grace-period guards.** Files modified within `SFM_QUIESCENCE_SECONDS` (default 5s) are skipped to avoid forwarding mid-write. If a binary's `.TXT` partner hasn't appeared after `SFM_MISSING_REPORT_GRACE_SECONDS` (default 60s), the binary is forwarded alone rather than blocking forever.
- New `event_forwarder.py` module + 17 unit tests in `test_event_forwarder.py` covering filename matching, state idempotency, scan logic, multipart encoding, and a fake-server end-to-end POST.
- **Quiescence + grace-period guards.** Files modified within `SFM_QUIESCENCE_SECONDS` (default 5s) are skipped to avoid forwarding mid-write. If a binary's report partner hasn't appeared after `SFM_MISSING_REPORT_GRACE_SECONDS` (default 60s), the binary is forwarded alone rather than blocking forever.
- **Per-pass rate cap.** `SFM_MAX_FORWARDS_PER_PASS` (default 500) drips first-deploy backfill instead of hammering the SFM server in one burst. At 60-second `SFM_FORWARD_INTERVAL_SECONDS` cadence that's ~30K events/hour throughput. Set to `0` for unlimited. Scan walks oldest-first so backfill advances chronologically and successive scans reliably progress.
- **`event_forwarder.py --seed-state` CLI mode.** Walks the watch folder once, sha256s every in-window event, and marks them all as already-forwarded *without* POSTing anything. Recommended pre-deploy workflow on machines with a large historical archive — flip `SFM_FORWARD_ENABLED=true` after seeding and only events that appear from then on get forwarded.
- **SFM Forward tab in the Settings dialog** with: Forward checkbox, SFM URL + Test button (GETs `/health`), Forward Interval / Quiescence / Missing-Report Grace / HTTP Timeout / Max Events Per Pass spinboxes, State File entry with Browse... Save-time guard: enabling forwarding without a URL shows a validation error.
- **Histogram-aware log clarity.** Histogram events (extensions ending in `H`) don't get auto-exported reports from BW; the log distinguishes that case (`(histogram, no report expected)`) from a waveform with unexpectedly missing report (`no report ⚠`).
- **README "First-time deployment" section** documenting the seed-state workflow + the rate cap as belt-and-suspenders for safe rollout on machines with hundreds of thousands of historical events.
- 31 new unit tests in `test_event_forwarder.py` covering filename matching, state idempotency, scan logic (quiescence / grace period / max age / already-forwarded / TXT pairing), multipart byte shape, rate cap (oldest-first, cap=0 unlimited, cap=N enforcement), seed-state mode (in-window seeding / max-age skip / end-to-end skip-after-seed / idempotent re-runs), histogram classification, and an end-to-end POST against a stdlib fake server.
### Configuration
New `[agent]` keys (all default-off — existing 1.4.x deployments don't change behaviour on auto-update):
- `SFM_FORWARD_ENABLED` (default `false`)
- `SFM_URL` (e.g. `http://10.0.0.44:8200`)
- `SFM_FORWARD_INTERVAL_SECONDS` (default `60`)
- `SFM_QUIESCENCE_SECONDS` (default `5`)
- `SFM_MISSING_REPORT_GRACE_SECONDS` (default `60`)
- `SFM_HTTP_TIMEOUT` (default `60`)
- `SFM_STATE_FILE` (default: `<log dir>/sfm_forwarded.json`)
| Key | Default | Notes |
|---|---|---|
| `SFM_FORWARD_ENABLED` | `false` | Master toggle for the forwarder |
| `SFM_URL` | empty | e.g. `http://10.0.0.44:8200` |
| `SFM_FORWARD_INTERVAL_SECONDS` | `60` | Scan-and-forward cadence |
| `SFM_QUIESCENCE_SECONDS` | `5` | Skip files modified in the last N seconds |
| `SFM_MISSING_REPORT_GRACE_SECONDS` | `60` | Forward without TXT after this delay |
| `SFM_HTTP_TIMEOUT` | `60` | Per-request HTTP timeout |
| `SFM_STATE_FILE` | `<log dir>/sfm_forwarded.json` | Override location of the forwarded-sha256 state file |
| `SFM_MAX_FORWARDS_PER_PASS` | `500` | Per-scan cap (`0` = unlimited) |
### Compatibility
- Requires SFM server v0.16+ (the `/db/import/blastware_file` endpoint that accepts paired `.TXT` reports — released alongside this watcher version on the seismo-relay side).
- Requires SFM server v0.16+ (the `/db/import/blastware_file` endpoint that accepts paired `_ASCII.TXT` reports + the BW-report label normalisation — released alongside this watcher version on the seismo-relay side).
## [1.4.4] - 2026-03-17
+3 -3
View File
@@ -1,4 +1,4 @@
# Series 3 Watcher v1.5.4
# Series 3 Watcher v1.5.0
Monitors Instantel **Series 3 (Minimate)** call-in activity on a Blastware server. Runs as a **system tray app** that starts automatically on login, reports heartbeats to terra-view, and self-updates from Gitea.
@@ -88,7 +88,7 @@ All settings live in `config.ini`. The Setup Wizard covers every field, but here
| `UPDATE_SOURCE` | `gitea` (default) or `url` — where to check for updates |
| `UPDATE_URL` | Base URL of the update server when `UPDATE_SOURCE = url` (e.g. terra-view URL). The watcher fetches `/api/updates/series3-watcher/version.txt` and `/api/updates/series3-watcher/series3-watcher.exe` from this base. |
### SFM Event Forwarder (v1.5.4+)
### SFM Event Forwarder (v1.5.0+)
Forwards each Blastware event binary (and its paired `<binary>.TXT` ASCII report when present) to an SFM server's `/db/import/blastware_file` endpoint, where the report is parsed and the rich per-channel stats (PPV, ZC Freq, Time of Peak, Peak Acceleration / Displacement, sensor self-check) land in a searchable database. **Default-off** — existing deployments keep their old behaviour after auto-updating until the operator opts in.
@@ -155,7 +155,7 @@ To view connected watchers: **Settings → Developer → Watcher Manager**.
## Versioning
Follows **Semantic Versioning**. Current release: **v1.5.4**.
Follows **Semantic Versioning**. Current release: **v1.5.0**.
See `CHANGELOG.md` for full history.
---
+1 -1
View File
@@ -3,7 +3,7 @@
[Setup]
AppName=Series 3 Watcher
AppVersion=1.5.4
AppVersion=1.5.0
AppPublisher=Terra-Mechanics Inc.
DefaultDirName={pf}\Series3Watcher
DefaultGroupName=Series 3 Watcher
+1 -1
View File
@@ -1,5 +1,5 @@
"""
Series 3 Watcher — System Tray Launcher v1.5.4
Series 3 Watcher — System Tray Launcher v1.5.0
Requires: pystray, Pillow, tkinter (stdlib)
Run with: pythonw series3_tray.py (no console window)
+1 -1
View File
@@ -247,7 +247,7 @@ def scan_latest(
# --- API heartbeat / SFM telemetry helpers ---
VERSION = "1.5.4"
VERSION = "1.5.0"
def _read_log_tail(log_file: str, n: int = 25) -> Optional[list]:
+1 -1
View File
@@ -1,5 +1,5 @@
"""
Series 3 Watcher Settings Dialog v1.5.4
Series 3 Watcher Settings Dialog v1.5.0
Provides a Tkinter settings dialog that doubles as a first-run wizard.