v0.4.0 - dual send mirror implementation #6

Merged
serversdown merged 9 commits from dev into main 2026-06-25 12:36:43 -04:00
6 changed files with 53 additions and 9 deletions
Showing only changes of commit 3bef58b8dd - Show all commits
+26
View File
@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.4.0] - 2026-06-25
### Added — Mirror (dual-send)
- **Optional best-effort mirror server.** Each heartbeat and each forwarded event can now be sent to a *second* ("mirror") server in addition to the primary, so a standby box (e.g. the office NAS) stays a continuous replica during a migration — making the eventual cutover a non-event. **Default off** — blank `mirror_api_url`/`mirror_sfm_url` means existing installs are unchanged after auto-update.
- **Heartbeat mirror (best-effort).** After the primary heartbeat POST, the same payload is fired at `mirror_api_url` when set. Bounded by the existing API timeout and fully wrapped — it can never delay or fail the primary heartbeat; its result is ignored except for a debug log line.
- **Event mirror (reliable, isolated).** Events are re-forwarded to `mirror_sfm_url` using a *separate* sha256 state file (`mirror_sfm_state_file`, blank → `<log_dir>/thor_forwarded_mirror.json`), so the mirror tracks its own delivery independently of the primary. A quick reachability probe (~3 s) skips the mirror pass when the mirror server is down, so a dead mirror never stalls the loop on per-event timeouts; skipped events stay pending in the mirror state and deliver when the mirror returns — **no data loss**.
- **Isolation invariant (the one rule).** Nothing on the mirror path can delay or fail the primary: its own state file, its own try/except, the reachability guard + bounded timeouts, all exceptions swallowed-and-logged. The primary path stays exactly as reliable as before.
- **Mirror tab in the Settings dialog** with `Mirror API URL` + `Mirror SFM URL` fields (blank = off). The mirror rides along with the primary — there is no separate enable flag (heartbeat-mirror active iff the primary heartbeat is on *and* `mirror_api_url` is set; event-mirror active iff primary forwarding is on *and* `mirror_sfm_url` is set).
- New unit tests in `test_event_forwarder.py` covering mirror reachability, the down-mirror skip, separate-state idempotency, and the isolation invariant (a mirror failure leaves the primary's result + state untouched).
### Configuration
New `config.json` keys (all default-empty = off; existing 0.3.x deployments don't change behaviour on auto-update):
| Key | Default | Notes |
|---|---|---|
| `mirror_api_url` | `""` | Second heartbeat destination — same URL form as `api_url` (blank = off) |
| `mirror_sfm_url` | `""` | Second SFM base URL, e.g. `http://10.0.0.x:8200` (blank = off) |
| `mirror_sfm_state_file` | `""` | Override the mirror's forwarded-sha256 state file. Blank → `<log_dir>\thor_forwarded_mirror.json` |
### Changed
- Bumped `VERSION` to `0.4.0`
> **Operational note (migration seeding).** To make the mirror re-deliver only the gap since a one-time replica snapshot, copy the primary state file to the mirror state file and drop entries whose `forwarded_at` is after the snapshot. SFM-side dedup covers any overlap. See `docs/mirror-dual-send-design.md`.
## [0.3.1] - 2026-05-20
### Hot fix
+22 -4
View File
@@ -1,8 +1,8 @@
# Thor Watcher
**Version:** 0.3.1
**Version:** 0.4.0
Micromate (Series 4) watcher agent for Terra-View fleet management. Runs as a Windows system tray application, scans THORDATA for Micromate unit activity, sends heartbeat data to Terra-View, and (optionally) forwards `.IDFH`/`.IDFW` event files to a seismo-relay SFM server.
Micromate (Series 4) watcher agent for Terra-View fleet management. Runs as a Windows system tray application, scans THORDATA for Micromate unit activity, sends heartbeat data to Terra-View, and (optionally) forwards `.IDFH`/`.IDFW` event files to a seismo-relay SFM server. Heartbeats and events can optionally be **mirrored** to a second server (e.g. a standby/NAS) in addition to the primary.
---
@@ -29,7 +29,7 @@ build.bat
```
Produces:
- `dist\thor-watcher-0.3.1.exe` — upload to Gitea release
- `dist\thor-watcher-0.4.0.exe` — upload to Gitea release
- `dist\thor-watcher.exe` — use with Inno Setup
Then run Inno Setup Compiler on `installer.iss` to produce `thor-watcher-setup.exe`.
@@ -71,6 +71,9 @@ Managed through the Settings dialog (right-click tray icon → Settings). A `con
| `sfm_state_file` | string | `""` | Path to the sha256-keyed state file. Blank → `<log_dir>\thor_forwarded.json` |
| `sfm_max_forwards_per_pass` | integer | `500` | Cap per pass to drip-feed large backfills |
| `sfm_max_event_age_days` | integer | `365` | Skip event files older than this many days |
| `mirror_api_url` | string | `""` | Second heartbeat destination (same URL form as `api_url`). Blank → heartbeat mirror off |
| `mirror_sfm_url` | string | `""` | Second SFM base URL (e.g. `http://10.0.0.x:8200`). Blank → event mirror off |
| `mirror_sfm_state_file` | string | `""` | Path to the mirror's sha256 state file. Blank → `<log_dir>\thor_forwarded_mirror.json` |
---
@@ -86,6 +89,21 @@ When `sfm_forward_enabled` is true and `sfm_url` is set, Thor Watcher walks the
---
## Mirror (Dual-Send)
Thor Watcher can post every heartbeat and event to a **second ("mirror") server** in addition to the primary — useful for keeping a standby box (e.g. the office NAS) as a continuous replica during a server migration, so the eventual cutover is a non-event. **Default off.**
The mirror rides along with the primary — there is no separate enable flag:
- **Heartbeat mirror** is active when the API heartbeat is enabled **and** `mirror_api_url` is set. After the primary heartbeat POST, the same payload is fired at `mirror_api_url`.
- **Event mirror** is active when SFM forwarding is enabled **and** `mirror_sfm_url` is set. Events are re-forwarded to `mirror_sfm_url` using a **separate** sha256 state file (`mirror_sfm_state_file`, blank → `<log_dir>\thor_forwarded_mirror.json`).
**The mirror can never delay or fail the primary.** The heartbeat mirror is fully wrapped and bounded by the API timeout. The event mirror runs a quick reachability probe (~3 s) first and skips the pass if the mirror server is down — skipped events stay pending in the mirror state and deliver when it returns, so nothing is lost. All mirror exceptions are swallowed-and-logged; the primary path stays exactly as reliable as before.
- **Seeding for a migration.** To re-deliver only the gap since a one-time replica snapshot, copy the primary state file to the mirror state file and drop entries whose `forwarded_at` is after the snapshot. SFM-side dedup covers any overlap. See [`docs/mirror-dual-send-design.md`](docs/mirror-dual-send-design.md).
---
## Tray Icon Colors
| Color | Meaning |
@@ -121,7 +139,7 @@ Posted to `api_url` on each API interval:
{
"source_id": "THOR-PC",
"source_type": "series4_watcher",
"version": "0.3.1",
"version": "0.4.0",
"generated_at": "2026-03-20T14:30:00Z",
"log_tail": ["...last 25 log lines..."],
"units": [
+1 -1
View File
@@ -3,7 +3,7 @@
[Setup]
AppName=Thor Watcher
AppVersion=0.3.1
AppVersion=0.4.0
AppPublisher=Terra-Mechanics Inc.
DefaultDirName={pf}\ThorWatcher
DefaultGroupName=Thor Watcher
+2 -2
View File
@@ -1,5 +1,5 @@
"""
Thor Watcher — Series 4 Ingest Agent v0.3.1
Thor Watcher — Series 4 Ingest Agent v0.4.0
Micromate (Series 4) ingest agent for Terra-View.
@@ -29,7 +29,7 @@ import event_forwarder
# ── Version ───────────────────────────────────────────────────────────────────
VERSION = "0.3.1"
VERSION = "0.4.0"
# ── Config ────────────────────────────────────────────────────────────────────
+1 -1
View File
@@ -1,5 +1,5 @@
"""
Thor Watcher — Settings Dialog v0.3.1
Thor Watcher — Settings Dialog v0.4.0
Provides a Tkinter settings dialog that doubles as a first-run wizard.
+1 -1
View File
@@ -1,5 +1,5 @@
"""
Thor Watcher — System Tray Launcher v0.3.1
Thor Watcher — System Tray Launcher v0.4.0
Requires: pystray, Pillow, tkinter (stdlib)
Run with: pythonw thor_tray.py (no console window)