feat(forward): rate cap + seed-state mode for safe backfill (v1.5.2)
Two safety nets for first-deploy on Blastware ACH machines that
have accumulated tens or hundreds of thousands of historical events
in the watch folder.
1. SFM_MAX_FORWARDS_PER_PASS (default 500, 0=unlimited)
---------------------------------------------------
Cap on the number of events forwarded per scan tick. At the
60-second default interval that's ~30K events/hour throughput —
the SFM server gets a steady drip instead of one giant burst.
Scan now sorts by mtime ascending so backfill advances
chronologically (oldest first) and successive scans always
make progress instead of re-considering the same N newest files.
Wired into:
- event_forwarder.find_pending_events / forward_pending
- series3_watcher.run_watcher loop
- config-template.ini
- settings_dialog SFM Forward tab (new "Max Events Per Pass"
spinbox, validated in _on_save)
2. event_forwarder.py --seed-state CLI
-----------------------------------
One-shot mode that walks the watch folder, sha256s every in-window
event binary, and marks them all as already-forwarded WITHOUT
POSTing anything. Run before flipping SFM_FORWARD_ENABLED=true
to skip the historical backfill entirely — the watcher then only
forwards events that appear AFTER the seed.
Usage:
python event_forwarder.py --seed-state \
--watch "C:\Blastware 10\Event\autocall home" \
--state "C:\...\sfm_forwarded.json" \
[--max-age-days 365]
7 new unit tests:
- max_per_pass cap enforcement (=N, =0 unlimited, oldest-first
ordering)
- seed-state mode (in-window seeding, max-age skip,
end-to-end skip-after-seed, idempotent re-runs)
README adds a "First-time deployment" section walking through both
options. Bumps to v1.5.2.
This commit is contained in:
+8
-1
@@ -104,6 +104,12 @@ def load_config(path: str) -> Dict[str, Any]:
|
||||
# State file for forwarded-sha256 idempotency tracking.
|
||||
# Defaults next to the log file for easy operator access.
|
||||
"SFM_STATE_FILE": get_str("SFM_STATE_FILE", ""),
|
||||
# Per-pass cap — forward at most N events per scan tick.
|
||||
# 0 = unlimited. Default 500 as a safety against accidentally
|
||||
# backfilling tens of thousands of events in one burst on
|
||||
# first deploy in a folder that's been accumulating for years.
|
||||
# See README "First-time deployment" section.
|
||||
"SFM_MAX_FORWARDS_PER_PASS": get_int("SFM_MAX_FORWARDS_PER_PASS", 500),
|
||||
}
|
||||
|
||||
|
||||
@@ -241,7 +247,7 @@ def scan_latest(
|
||||
|
||||
|
||||
# --- API heartbeat / SFM telemetry helpers ---
|
||||
VERSION = "1.5.1"
|
||||
VERSION = "1.5.2"
|
||||
|
||||
|
||||
def _read_log_tail(log_file: str, n: int = 25) -> Optional[list]:
|
||||
@@ -522,6 +528,7 @@ def run_watcher(state: Dict[str, Any], stop_event: threading.Event) -> None:
|
||||
cfg.get("SFM_MISSING_REPORT_GRACE_SECONDS", 60)
|
||||
),
|
||||
timeout=int(cfg.get("SFM_HTTP_TIMEOUT", 60)),
|
||||
max_per_pass=int(cfg.get("SFM_MAX_FORWARDS_PER_PASS", 500)),
|
||||
logger=lambda m: log_message(LOG_FILE, ENABLE_LOGGING, m),
|
||||
)
|
||||
last_forward_ts = now_ts
|
||||
|
||||
Reference in New Issue
Block a user