v1.6.0 - Optional dual-send mirror so a standby server stays a live replica. #12

Merged
serversdown merged 4 commits from dev into main 2026-06-25 10:53:30 -04:00
6 changed files with 53 additions and 6 deletions
Showing only changes of commit e6e4c94f19 - Show all commits
+26
View File
@@ -6,6 +6,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
--- ---
## [6-25-26] — v1.6.0
Optional dual-send mirror so a standby server stays a live replica.
### Added — Mirror (dual-send)
- **Post every heartbeat and event to a second ("mirror") server in addition to the primary.** When `MIRROR_API_URL` and/or `MIRROR_SFM_URL` are set, each heartbeat POST and each forwarded Blastware event is *also* sent to the mirror destination — letting a standby box (e.g. the office NAS) stay a continuous replica of prod during a migration so the eventual cutover is a non-event. **Default off** — blank URLs mean existing 1.5.x installs don't change behaviour after an auto-update.
- **Heartbeat mirror (best-effort).** After the primary `send_api_payload`, the same payload is fired at `MIRROR_API_URL` when set. It is fully wrapped and bounded by the existing API timeout — the mirror can **never** delay or fail the primary heartbeat; its outcome is ignored except for a debug log line.
- **Event mirror (reliable, isolated).** Events are re-forwarded to `MIRROR_SFM_URL` through a *separate* sha256 state file (`MIRROR_SFM_STATE_FILE`, blank → `<log dir>/sfm_forwarded_mirror.json`) so the mirror tracks its own delivery independently of the primary forwarder. Before each mirror pass a quick reachability probe (~3 s) checks the mirror server; if it's down the pass is skipped and retried next tick rather than blocking the loop on per-event timeouts. Skipped events stay pending in the mirror state and deliver once the mirror returns — **no data loss**.
- **Isolation invariant.** Nothing on the mirror path can delay or fail the primary: its own state file, its own try/except, the reachability guard + bounded timeouts, and all exceptions swallowed-and-logged. The primary path stays exactly as reliable as before. The mirror rides along with the primary — there is no separate enable flag (heartbeat-mirror active iff the primary heartbeat is on *and* `MIRROR_API_URL` is set; event-mirror active iff primary forwarding is on *and* `MIRROR_SFM_URL` is set).
- **Mirror fields in the Settings dialog**: `Mirror API URL` + `Mirror SFM URL` (blank = off).
- New unit tests in `test_event_forwarder.py` covering mirror reachability, the down-mirror skip, separate-state idempotency, and the isolation invariant (a mirror failure leaves the primary's result + state untouched).
### Configuration
New `[agent]` keys (all default-off — existing 1.5.x deployments don't change behaviour on auto-update):
| Key | Default | Notes |
|---|---|---|
| `MIRROR_API_URL` | empty | Second heartbeat base URL, e.g. `http://10.0.0.x:8001` (blank = off) |
| `MIRROR_SFM_URL` | empty | Second SFM base URL, e.g. `http://10.0.0.x:8200` (blank = off) |
| `MIRROR_SFM_STATE_FILE` | `<log dir>/sfm_forwarded_mirror.json` | Override location of the mirror's forwarded-sha256 state file |
### Operational (migration seeding)
To make the mirror re-deliver only the gap since a one-time replica snapshot, copy the primary state file to the mirror state file and drop entries whose `forwarded_at` is after the snapshot. SFM-side dedup covers any overlap. See `docs/mirror-dual-send-design.md`.
## [5-11-26] — v1.5.0 ## [5-11-26] — v1.5.0
First release of the SFM event forwarder. First release of the SFM event forwarder.
+23 -2
View File
@@ -1,4 +1,4 @@
# Series 3 Watcher v1.5.0 # Series 3 Watcher v1.6.0
Monitors Instantel **Series 3 (Minimate)** call-in activity on a Blastware server. Runs as a **system tray app** that starts automatically on login, reports heartbeats to terra-view, and self-updates from Gitea. Monitors Instantel **Series 3 (Minimate)** call-in activity on a Blastware server. Runs as a **system tray app** that starts automatically on login, reports heartbeats to terra-view, and self-updates from Gitea.
@@ -124,6 +124,27 @@ Combine both for a fully controlled rollout: seed-state to skip the deep archive
--- ---
## Mirror (Dual-Send) — v1.6.0+
The watcher can post every heartbeat and event to a **second ("mirror") server** in addition to the primary — useful for keeping a standby box (e.g. the office NAS) as a continuous replica during a server migration, so the eventual cutover is a non-event. **Default off.**
The mirror rides along with the primary — there is no separate enable flag:
- **Heartbeat mirror** is active when `API_ENABLED = true` **and** `MIRROR_API_URL` is set. After the primary heartbeat POST, the same payload is fired at the mirror.
- **Event mirror** is active when `SFM_FORWARD_ENABLED = true` **and** `MIRROR_SFM_URL` is set. Events are re-forwarded to the mirror using a **separate** sha256 state file (`MIRROR_SFM_STATE_FILE`, blank → `<log dir>/sfm_forwarded_mirror.json`).
**The mirror can never delay or fail the primary.** The heartbeat mirror is fully wrapped and bounded by the API timeout. The event mirror runs a quick reachability probe (~3 s) first and skips the pass if the mirror server is down — skipped events stay pending in the mirror state and deliver when it returns, so nothing is lost. All mirror exceptions are swallowed-and-logged; the primary path stays exactly as reliable as before.
| Key | Description |
|-----|-------------|
| `MIRROR_API_URL` | Second heartbeat base URL, e.g. `http://10.0.0.x:8001` (blank = off) |
| `MIRROR_SFM_URL` | Second SFM base URL, e.g. `http://10.0.0.x:8200` (blank = off) |
| `MIRROR_SFM_STATE_FILE` | Path to the mirror's sha256 state file. Blank → `<log dir>/sfm_forwarded_mirror.json` |
**Seeding for a migration.** To re-deliver only the gap since a one-time replica snapshot, copy the primary state file to the mirror state file and drop entries whose `forwarded_at` is after the snapshot. SFM-side dedup covers any overlap. See [`docs/mirror-dual-send-design.md`](docs/mirror-dual-send-design.md).
---
## Tray Icon ## Tray Icon
| Colour | Meaning | | Colour | Meaning |
@@ -167,7 +188,7 @@ where the corresponding server-side work lives.
## Versioning ## Versioning
Follows **Semantic Versioning**. Current release: **v1.5.0**. Follows **Semantic Versioning**. Current release: **v1.6.0**.
See `CHANGELOG.md` for full history. See `CHANGELOG.md` for full history.
--- ---
+1 -1
View File
@@ -3,7 +3,7 @@
[Setup] [Setup]
AppName=Series 3 Watcher AppName=Series 3 Watcher
AppVersion=1.5.0 AppVersion=1.6.0
AppPublisher=Terra-Mechanics Inc. AppPublisher=Terra-Mechanics Inc.
DefaultDirName={pf}\Series3Watcher DefaultDirName={pf}\Series3Watcher
DefaultGroupName=Series 3 Watcher DefaultGroupName=Series 3 Watcher
+1 -1
View File
@@ -1,5 +1,5 @@
""" """
Series 3 Watcher — System Tray Launcher v1.5.0 Series 3 Watcher — System Tray Launcher v1.6.0
Requires: pystray, Pillow, tkinter (stdlib) Requires: pystray, Pillow, tkinter (stdlib)
Run with: pythonw series3_tray.py (no console window) Run with: pythonw series3_tray.py (no console window)
+1 -1
View File
@@ -253,7 +253,7 @@ def scan_latest(
# --- API heartbeat / SFM telemetry helpers --- # --- API heartbeat / SFM telemetry helpers ---
VERSION = "1.5.0" VERSION = "1.6.0"
def _read_log_tail(log_file: str, n: int = 25) -> Optional[list]: def _read_log_tail(log_file: str, n: int = 25) -> Optional[list]:
+1 -1
View File
@@ -1,5 +1,5 @@
""" """
Series 3 Watcher — Settings Dialog v1.5.0 Series 3 Watcher — Settings Dialog v1.6.0
Provides a Tkinter settings dialog that doubles as a first-run wizard. Provides a Tkinter settings dialog that doubles as a first-run wizard.