docs: mirror dual-send design spec

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-24 23:02:47 +00:00
parent 997b5f053c
commit dae67afeb9
1 changed files with 51 additions and 0 deletions
@@ -0,0 +1,51 @@
+# Watcher Mirror (Dual-Send) — Design
+
+**Date:** 2026-06-24 · **Branch:** `feat/mirror-dual-send` (both `thor-watcher` + `series3-watcher`) · **Status:** approved, implementing
+
+## Goal
+Let each watcher post every heartbeat and event to a **second ("mirror") server** in addition to the primary, so the office NAS stays a continuous replica of prod during the migration. The eventual cutover becomes a non-event. **Default off** → existing installs unchanged.
+
+## Pattern
+Classic dual-write migration. The primary path stays **exactly as reliable as today**; the mirror is pure best-effort bonus that can **never delay or fail** the primary.
+
+## The two watchers (parallel changes)
+| | thor-watcher (series4) | series3-watcher (s3) |
+|---|---|---|
+| Config | `config.json` (JSON) | `config.ini` (`[agent]`, INI) |
+| Heartbeat fn | `send_api_payload(payload, api_url, timeout)` | `send_api_payload(payload, api_url)` |
+| Event fn | `event_forwarder.forward_pending(root, sfm_url, state, …)` | same (`event_forwarder` is a sibling port) |
+| Loop | `series4_ingest.run_watcher` | `series3_watcher.run_watcher` |
+| Settings GUI | `thor_settings_dialog.py` | `settings_dialog.py` |
+
+## Design (Approach A — reliable event mirror, best-effort heartbeat)
+
+### Config additions (default empty = off)
+- thor (`config.json`): `mirror_api_url`, `mirror_sfm_url`, `mirror_sfm_state_file` (blank → `<log_dir>/thor_forwarded_mirror.json`).
+- s3 (`config.ini`): `MIRROR_API_URL`, `MIRROR_SFM_URL`, `MIRROR_SFM_STATE_FILE` (blank → `sfm_forwarded_mirror.json`).
+
+The mirror **rides along** with the primary: heartbeat-mirror active iff primary heartbeat on **and** `mirror_api_url` set; event-mirror active iff primary forwarding on **and** `mirror_sfm_url` set. No separate enable flag.
+
+### Heartbeat mirror (best-effort)
+After the primary `send_api_payload`, if `mirror_api_url` is set, fire one more `send_api_payload(payload, mirror_api_url, …)`. Wrapped so it can never raise into the loop; result ignored except a debug log. Bounded by the existing short API timeout (~5 s).
+
+### Event mirror (reliable, isolated)
+- Init a **second** `ForwardState(mirror_state_path)` alongside the primary state.
+- Each forward tick, **after** the primary `forward_pending`, run a second `forward_pending(root, mirror_sfm_url, mirror_state, …)` in its **own** try/except.
+- **Reachability guard:** before the mirror forward, a quick TCP/HTTP probe of `mirror_sfm_url` (~3 s). If unreachable → skip this pass, log, retry next tick. Prevents a down NAS from stalling the loop on per-event timeouts. The mirror state file means skipped events stay pending → delivered when the NAS returns. **No data loss.**
+- Reliable + idempotent via its own sha256 state, fully independent of the primary's state.
+
+### Isolation invariant (the one rule)
+Nothing on the mirror path can delay or fail the primary: separate state file, its own try/except, the reachability guard + bounded timeouts, all exceptions swallowed-and-logged.
+
+### Settings dialog
+Add `Mirror API URL` + `Mirror SFM URL` fields (blank = off) to each watcher's settings dialog.
+
+## Seeding (operational, for this migration)
+The mirror state file should reflect what the NAS already has (everything ≤ the migration snapshot, ~16:23 on 2026-06-24). Recipe: copy the primary state file → mirror state file, then drop entries whose `forwarded_at` is after the snapshot, so the mirror re-delivers exactly the gap. SFM-side dedup covers any overlap. (If a watcher had no events in the gap, a straight copy suffices.)
+
+## Testing
+- thor `test_event_forwarder.py`: mirror uses a separate state file (idempotent, independent); a mirror failure leaves the primary's result + state untouched; reachability guard skips cleanly when the mirror is down.
+- s3: add a minimal test for the same isolation/idempotency.
+
+## Out of scope / handoff
+Rebuilding the Windows installers + redeploying (s3 on the Win7 box, thor on the Win10 box) is the operator's job — this change is **code + tests only**.