serversdown/series3-watcher
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 5 Wiki Activity

feat(forward): rate cap + seed-state mode for safe backfill (v1.5.2)

Browse Source 
Two safety nets for first-deploy on Blastware ACH machines that
have accumulated tens or hundreds of thousands of historical events
in the watch folder.

1. SFM_MAX_FORWARDS_PER_PASS (default 500, 0=unlimited)
   ---------------------------------------------------
   Cap on the number of events forwarded per scan tick.  At the
   60-second default interval that's ~30K events/hour throughput —
   the SFM server gets a steady drip instead of one giant burst.
   Scan now sorts by mtime ascending so backfill advances
   chronologically (oldest first) and successive scans always
   make progress instead of re-considering the same N newest files.

   Wired into:
     - event_forwarder.find_pending_events / forward_pending
     - series3_watcher.run_watcher loop
     - config-template.ini
     - settings_dialog SFM Forward tab (new "Max Events Per Pass"
       spinbox, validated in _on_save)

2. event_forwarder.py --seed-state CLI
   -----------------------------------
   One-shot mode that walks the watch folder, sha256s every in-window
   event binary, and marks them all as already-forwarded WITHOUT
   POSTing anything.  Run before flipping SFM_FORWARD_ENABLED=true
   to skip the historical backfill entirely — the watcher then only
   forwards events that appear AFTER the seed.

   Usage:
       python event_forwarder.py --seed-state \
           --watch "C:\Blastware 10\Event\autocall home" \
           --state "C:\...\sfm_forwarded.json" \
           [--max-age-days 365]

7 new unit tests:
  - max_per_pass cap enforcement (=N, =0 unlimited, oldest-first
    ordering)
  - seed-state mode (in-window seeding, max-age skip,
    end-to-end skip-after-seed, idempotent re-runs)

README adds a "First-time deployment" section walking through both
options.  Bumps to v1.5.2.
This commit is contained in:
serversdown
2026-05-10 00:20:10 +00:00
parent 3ee0cae31e
commit 815c643fb2
9 changed files with 390 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+13
View File
@@ -6,6 +6,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
---
## [1.5.2] - 2026-05-10
### Added
- **`SFM_MAX_FORWARDS_PER_PASS` rate cap.** Default 500 events per scan tick (60-second interval = ~30K events/hour). Drips first-deploy backfill instead of hammering the SFM server with one giant burst on machines that have hundreds of thousands of historical events in the BW ACH folder. `0` = unlimited (preserves the 1.5.0 behaviour for ops who want it).
- **`event_forwarder.py --seed-state` CLI mode.** Walks the watch folder once, sha256s every in-window event, and marks them all as already-forwarded *without* POSTing anything. The recommended pre-deploy workflow on machines with a large historical archive — flip `SFM_FORWARD_ENABLED=true` after seeding and only events that appear *from then on* get forwarded.
- New "Max Events Per Pass" spinbox in the Settings dialog's SFM Forward tab.
- README "First-time deployment" section documenting both options.
- 7 new unit tests covering the cap (oldest-first ordering, cap=0 unlimited, cap=N enforcement) and the seed-state mode (skips out-of-window files, idempotent across re-runs, end-to-end skip-after-seed).
### Behaviour change
The scan loop now sorts entries by mtime ascending before walking, so backfill always advances chronologically (oldest qualifying event first). Without the cap the visible behaviour is identical; with the cap it means each scan reliably advances and we never get stuck re-considering the same N newest files.
## [1.5.1] - 2026-05-10
### Added