3.9 KiB
Watcher Mirror (Dual-Send) — Design
Date: 2026-06-24 · Branch: feat/mirror-dual-send (both thor-watcher + series3-watcher) · Status: approved, implementing
Goal
Let each watcher post every heartbeat and event to a second ("mirror") server in addition to the primary, so the office NAS stays a continuous replica of prod during the migration. The eventual cutover becomes a non-event. Default off → existing installs unchanged.
Pattern
Classic dual-write migration. The primary path stays exactly as reliable as today; the mirror is pure best-effort bonus that can never delay or fail the primary.
The two watchers (parallel changes)
| thor-watcher (series4) | series3-watcher (s3) | |
|---|---|---|
| Config | config.json (JSON) |
config.ini ([agent], INI) |
| Heartbeat fn | send_api_payload(payload, api_url, timeout) |
send_api_payload(payload, api_url) |
| Event fn | event_forwarder.forward_pending(root, sfm_url, state, …) |
same (event_forwarder is a sibling port) |
| Loop | series4_ingest.run_watcher |
series3_watcher.run_watcher |
| Settings GUI | thor_settings_dialog.py |
settings_dialog.py |
Design (Approach A — reliable event mirror, best-effort heartbeat)
Config additions (default empty = off)
- thor (
config.json):mirror_api_url,mirror_sfm_url,mirror_sfm_state_file(blank →<log_dir>/thor_forwarded_mirror.json). - s3 (
config.ini):MIRROR_API_URL,MIRROR_SFM_URL,MIRROR_SFM_STATE_FILE(blank →sfm_forwarded_mirror.json).
The mirror rides along with the primary: heartbeat-mirror active iff primary heartbeat on and mirror_api_url set; event-mirror active iff primary forwarding on and mirror_sfm_url set. No separate enable flag.
Heartbeat mirror (best-effort)
After the primary send_api_payload, if mirror_api_url is set, fire one more send_api_payload(payload, mirror_api_url, …). Wrapped so it can never raise into the loop; result ignored except a debug log. Bounded by the existing short API timeout (~5 s).
Event mirror (reliable, isolated)
- Init a second
ForwardState(mirror_state_path)alongside the primary state. - Each forward tick, after the primary
forward_pending, run a secondforward_pending(root, mirror_sfm_url, mirror_state, …)in its own try/except. - Reachability guard: before the mirror forward, a quick TCP/HTTP probe of
mirror_sfm_url(~3 s). If unreachable → skip this pass, log, retry next tick. Prevents a down NAS from stalling the loop on per-event timeouts. The mirror state file means skipped events stay pending → delivered when the NAS returns. No data loss. - Reliable + idempotent via its own sha256 state, fully independent of the primary's state.
Isolation invariant (the one rule)
Nothing on the mirror path can delay or fail the primary: separate state file, its own try/except, the reachability guard + bounded timeouts, all exceptions swallowed-and-logged.
Settings dialog
Add Mirror API URL + Mirror SFM URL fields (blank = off) to each watcher's settings dialog.
Seeding (operational, for this migration)
The mirror state file should reflect what the NAS already has (everything ≤ the migration snapshot, ~16:23 on 2026-06-24). Recipe: copy the primary state file → mirror state file, then drop entries whose forwarded_at is after the snapshot, so the mirror re-delivers exactly the gap. SFM-side dedup covers any overlap. (If a watcher had no events in the gap, a straight copy suffices.)
Testing
- thor
test_event_forwarder.py: mirror uses a separate state file (idempotent, independent); a mirror failure leaves the primary's result + state untouched; reachability guard skips cleanly when the mirror is down. - s3: add a minimal test for the same isolation/idempotency.
Out of scope / handoff
Rebuilding the Windows installers + redeploying (s3 on the Win7 box, thor on the Win10 box) is the operator's job — this change is code + tests only.