Compare commits
5 Commits
v1.5.0
..
fb1edfbad4
| Author | SHA1 | Date | |
|---|---|---|---|
| fb1edfbad4 | |||
| e6e4c94f19 | |||
| 92ebbd6537 | |||
| 4adaa09d75 | |||
| 730f215e23 |
@@ -6,6 +6,32 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## [6-25-26] — v1.6.0
|
||||||
|
|
||||||
|
Optional dual-send mirror so a standby server stays a live replica.
|
||||||
|
|
||||||
|
### Added — Mirror (dual-send)
|
||||||
|
- **Post every heartbeat and event to a second ("mirror") server in addition to the primary.** When `MIRROR_API_URL` and/or `MIRROR_SFM_URL` are set, each heartbeat POST and each forwarded Blastware event is *also* sent to the mirror destination — letting a standby box (e.g. the office NAS) stay a continuous replica of prod during a migration so the eventual cutover is a non-event. **Default off** — blank URLs mean existing 1.5.x installs don't change behaviour after an auto-update.
|
||||||
|
- **Heartbeat mirror (best-effort).** After the primary `send_api_payload`, the same payload is fired at `MIRROR_API_URL` when set. It is fully wrapped and bounded by the existing API timeout — the mirror can **never** delay or fail the primary heartbeat; its outcome is ignored except for a debug log line.
|
||||||
|
- **Event mirror (reliable, isolated).** Events are re-forwarded to `MIRROR_SFM_URL` through a *separate* sha256 state file (`MIRROR_SFM_STATE_FILE`, blank → `<log dir>/sfm_forwarded_mirror.json`) so the mirror tracks its own delivery independently of the primary forwarder. Before each mirror pass a quick reachability probe (~3 s) checks the mirror server; if it's down the pass is skipped and retried next tick rather than blocking the loop on per-event timeouts. Skipped events stay pending in the mirror state and deliver once the mirror returns — **no data loss**.
|
||||||
|
- **Isolation invariant.** Nothing on the mirror path can delay or fail the primary: its own state file, its own try/except, the reachability guard + bounded timeouts, and all exceptions swallowed-and-logged. The primary path stays exactly as reliable as before. The mirror rides along with the primary — there is no separate enable flag (heartbeat-mirror active iff the primary heartbeat is on *and* `MIRROR_API_URL` is set; event-mirror active iff primary forwarding is on *and* `MIRROR_SFM_URL` is set).
|
||||||
|
- **Mirror fields in the Settings dialog**: `Mirror API URL` + `Mirror SFM URL` (blank = off).
|
||||||
|
- New unit tests in `test_event_forwarder.py` covering mirror reachability, the down-mirror skip, separate-state idempotency, and the isolation invariant (a mirror failure leaves the primary's result + state untouched).
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
New `[agent]` keys (all default-off — existing 1.5.x deployments don't change behaviour on auto-update):
|
||||||
|
|
||||||
|
| Key | Default | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `MIRROR_API_URL` | empty | Second heartbeat base URL, e.g. `http://10.0.0.x:8001` (blank = off) |
|
||||||
|
| `MIRROR_SFM_URL` | empty | Second SFM base URL, e.g. `http://10.0.0.x:8200` (blank = off) |
|
||||||
|
| `MIRROR_SFM_STATE_FILE` | `<log dir>/sfm_forwarded_mirror.json` | Override location of the mirror's forwarded-sha256 state file |
|
||||||
|
|
||||||
|
### Operational (migration seeding)
|
||||||
|
|
||||||
|
To make the mirror re-deliver only the gap since a one-time replica snapshot, copy the primary state file to the mirror state file and drop entries whose `forwarded_at` is after the snapshot. SFM-side dedup covers any overlap. See `docs/mirror-dual-send-design.md`.
|
||||||
|
|
||||||
## [5-11-26] — v1.5.0
|
## [5-11-26] — v1.5.0
|
||||||
|
|
||||||
First release of the SFM event forwarder.
|
First release of the SFM event forwarder.
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
# Series 3 Watcher v1.5.0
|
# Series 3 Watcher v1.6.0
|
||||||
|
|
||||||
Monitors Instantel **Series 3 (Minimate)** call-in activity on a Blastware server. Runs as a **system tray app** that starts automatically on login, reports heartbeats to terra-view, and self-updates from Gitea.
|
Monitors Instantel **Series 3 (Minimate)** call-in activity on a Blastware server. Runs as a **system tray app** that starts automatically on login, reports heartbeats to terra-view, and self-updates from Gitea.
|
||||||
|
|
||||||
@@ -124,6 +124,27 @@ Combine both for a fully controlled rollout: seed-state to skip the deep archive
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Mirror (Dual-Send) — v1.6.0+
|
||||||
|
|
||||||
|
The watcher can post every heartbeat and event to a **second ("mirror") server** in addition to the primary — useful for keeping a standby box (e.g. the office NAS) as a continuous replica during a server migration, so the eventual cutover is a non-event. **Default off.**
|
||||||
|
|
||||||
|
The mirror rides along with the primary — there is no separate enable flag:
|
||||||
|
|
||||||
|
- **Heartbeat mirror** is active when `API_ENABLED = true` **and** `MIRROR_API_URL` is set. After the primary heartbeat POST, the same payload is fired at the mirror.
|
||||||
|
- **Event mirror** is active when `SFM_FORWARD_ENABLED = true` **and** `MIRROR_SFM_URL` is set. Events are re-forwarded to the mirror using a **separate** sha256 state file (`MIRROR_SFM_STATE_FILE`, blank → `<log dir>/sfm_forwarded_mirror.json`).
|
||||||
|
|
||||||
|
**The mirror can never delay or fail the primary.** The heartbeat mirror is fully wrapped and bounded by the API timeout. The event mirror runs a quick reachability probe (~3 s) first and skips the pass if the mirror server is down — skipped events stay pending in the mirror state and deliver when it returns, so nothing is lost. All mirror exceptions are swallowed-and-logged; the primary path stays exactly as reliable as before.
|
||||||
|
|
||||||
|
| Key | Description |
|
||||||
|
|-----|-------------|
|
||||||
|
| `MIRROR_API_URL` | Second heartbeat base URL, e.g. `http://10.0.0.x:8001` (blank = off) |
|
||||||
|
| `MIRROR_SFM_URL` | Second SFM base URL, e.g. `http://10.0.0.x:8200` (blank = off) |
|
||||||
|
| `MIRROR_SFM_STATE_FILE` | Path to the mirror's sha256 state file. Blank → `<log dir>/sfm_forwarded_mirror.json` |
|
||||||
|
|
||||||
|
**Seeding for a migration.** To re-deliver only the gap since a one-time replica snapshot, copy the primary state file to the mirror state file and drop entries whose `forwarded_at` is after the snapshot. SFM-side dedup covers any overlap. See [`docs/mirror-dual-send-design.md`](docs/mirror-dual-send-design.md).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Tray Icon
|
## Tray Icon
|
||||||
|
|
||||||
| Colour | Meaning |
|
| Colour | Meaning |
|
||||||
@@ -167,7 +188,7 @@ where the corresponding server-side work lives.
|
|||||||
|
|
||||||
## Versioning
|
## Versioning
|
||||||
|
|
||||||
Follows **Semantic Versioning**. Current release: **v1.5.0**.
|
Follows **Semantic Versioning**. Current release: **v1.6.0**.
|
||||||
See `CHANGELOG.md` for full history.
|
See `CHANGELOG.md` for full history.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -72,3 +72,14 @@ SFM_STATE_FILE =
|
|||||||
# `--seed-state` workflow that skips the historical backfill entirely.
|
# `--seed-state` workflow that skips the historical backfill entirely.
|
||||||
SFM_MAX_FORWARDS_PER_PASS = 500
|
SFM_MAX_FORWARDS_PER_PASS = 500
|
||||||
|
|
||||||
|
# --- Mirror (dual-send) ---
|
||||||
|
# Optional best-effort second destination: post each heartbeat and event
|
||||||
|
# to a mirror server (e.g. the office NAS) IN ADDITION to the primary
|
||||||
|
# above. Default off (blank URLs). The mirror can never delay or fail
|
||||||
|
# the primary; the event mirror keeps its own state file so nothing is
|
||||||
|
# lost while it's down. See docs/mirror-dual-send-design.md.
|
||||||
|
MIRROR_API_URL = ; base URL e.g. http://10.0.0.x:8001 (blank = off)
|
||||||
|
MIRROR_SFM_URL = ; base URL e.g. http://10.0.0.x:8200 (blank = off)
|
||||||
|
# Blank → defaults to <log dir>/sfm_forwarded_mirror.json
|
||||||
|
MIRROR_SFM_STATE_FILE =
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# Watcher Mirror (Dual-Send) — Design
|
||||||
|
|
||||||
|
**Date:** 2026-06-24 · **Branch:** `feat/mirror-dual-send` (both `thor-watcher` + `series3-watcher`) · **Status:** approved, implementing
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
Let each watcher post every heartbeat and event to a **second ("mirror") server** in addition to the primary, so the office NAS stays a continuous replica of prod during the migration. The eventual cutover becomes a non-event. **Default off** → existing installs unchanged.
|
||||||
|
|
||||||
|
## Pattern
|
||||||
|
Classic dual-write migration. The primary path stays **exactly as reliable as today**; the mirror is pure best-effort bonus that can **never delay or fail** the primary.
|
||||||
|
|
||||||
|
## The two watchers (parallel changes)
|
||||||
|
| | thor-watcher (series4) | series3-watcher (s3) |
|
||||||
|
|---|---|---|
|
||||||
|
| Config | `config.json` (JSON) | `config.ini` (`[agent]`, INI) |
|
||||||
|
| Heartbeat fn | `send_api_payload(payload, api_url, timeout)` | `send_api_payload(payload, api_url)` |
|
||||||
|
| Event fn | `event_forwarder.forward_pending(root, sfm_url, state, …)` | same (`event_forwarder` is a sibling port) |
|
||||||
|
| Loop | `series4_ingest.run_watcher` | `series3_watcher.run_watcher` |
|
||||||
|
| Settings GUI | `thor_settings_dialog.py` | `settings_dialog.py` |
|
||||||
|
|
||||||
|
## Design (Approach A — reliable event mirror, best-effort heartbeat)
|
||||||
|
|
||||||
|
### Config additions (default empty = off)
|
||||||
|
- thor (`config.json`): `mirror_api_url`, `mirror_sfm_url`, `mirror_sfm_state_file` (blank → `<log_dir>/thor_forwarded_mirror.json`).
|
||||||
|
- s3 (`config.ini`): `MIRROR_API_URL`, `MIRROR_SFM_URL`, `MIRROR_SFM_STATE_FILE` (blank → `sfm_forwarded_mirror.json`).
|
||||||
|
|
||||||
|
The mirror **rides along** with the primary: heartbeat-mirror active iff primary heartbeat on **and** `mirror_api_url` set; event-mirror active iff primary forwarding on **and** `mirror_sfm_url` set. No separate enable flag.
|
||||||
|
|
||||||
|
### Heartbeat mirror (best-effort)
|
||||||
|
After the primary `send_api_payload`, if `mirror_api_url` is set, fire one more `send_api_payload(payload, mirror_api_url, …)`. Wrapped so it can never raise into the loop; result ignored except a debug log. Bounded by the existing short API timeout (~5 s).
|
||||||
|
|
||||||
|
### Event mirror (reliable, isolated)
|
||||||
|
- Init a **second** `ForwardState(mirror_state_path)` alongside the primary state.
|
||||||
|
- Each forward tick, **after** the primary `forward_pending`, run a second `forward_pending(root, mirror_sfm_url, mirror_state, …)` in its **own** try/except.
|
||||||
|
- **Reachability guard:** before the mirror forward, a quick TCP/HTTP probe of `mirror_sfm_url` (~3 s). If unreachable → skip this pass, log, retry next tick. Prevents a down NAS from stalling the loop on per-event timeouts. The mirror state file means skipped events stay pending → delivered when the NAS returns. **No data loss.**
|
||||||
|
- Reliable + idempotent via its own sha256 state, fully independent of the primary's state.
|
||||||
|
|
||||||
|
### Isolation invariant (the one rule)
|
||||||
|
Nothing on the mirror path can delay or fail the primary: separate state file, its own try/except, the reachability guard + bounded timeouts, all exceptions swallowed-and-logged.
|
||||||
|
|
||||||
|
### Settings dialog
|
||||||
|
Add `Mirror API URL` + `Mirror SFM URL` fields (blank = off) to each watcher's settings dialog.
|
||||||
|
|
||||||
|
## Seeding (operational, for this migration)
|
||||||
|
The mirror state file should reflect what the NAS already has (everything ≤ the migration snapshot, ~16:23 on 2026-06-24). Recipe: copy the primary state file → mirror state file, then drop entries whose `forwarded_at` is after the snapshot, so the mirror re-delivers exactly the gap. SFM-side dedup covers any overlap. (If a watcher had no events in the gap, a straight copy suffices.)
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
- thor `test_event_forwarder.py`: mirror uses a separate state file (idempotent, independent); a mirror failure leaves the primary's result + state untouched; reachability guard skips cleanly when the mirror is down.
|
||||||
|
- s3: add a minimal test for the same isolation/idempotency.
|
||||||
|
|
||||||
|
## Out of scope / handoff
|
||||||
|
Rebuilding the Windows installers + redeploying (s3 on the Win7 box, thor on the Win10 box) is the operator's job — this change is **code + tests only**.
|
||||||
@@ -684,6 +684,74 @@ def forward_pending(
|
|||||||
return counts
|
return counts
|
||||||
|
|
||||||
|
|
||||||
|
# ── Mirror (dual-send) ────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def mirror_reachable(base_url: str, timeout: float = 3.0) -> bool:
|
||||||
|
"""Quick liveness probe of a mirror SFM base URL.
|
||||||
|
|
||||||
|
Run before a mirror forward pass so an unreachable mirror can't stall
|
||||||
|
the watcher loop on a per-event HTTP timeout for every pending file.
|
||||||
|
Probes ``<base>/health`` (the SFM server exposes it). ANY HTTP
|
||||||
|
response — even 404/500 — means the server is up, so proceed; only a
|
||||||
|
connection-level failure (refused / timed out / DNS) counts as down.
|
||||||
|
Best-effort: never raises.
|
||||||
|
"""
|
||||||
|
if not base_url:
|
||||||
|
return False
|
||||||
|
url = base_url.rstrip("/") + "/health"
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(
|
||||||
|
urllib.request.Request(url, method="GET"), timeout=timeout
|
||||||
|
):
|
||||||
|
return True
|
||||||
|
except urllib.error.HTTPError:
|
||||||
|
return True # got a status back → server is alive
|
||||||
|
except Exception:
|
||||||
|
return False # refused / timeout / DNS / socket → down
|
||||||
|
|
||||||
|
|
||||||
|
def mirror_forward_pass(
|
||||||
|
watch_dir: str,
|
||||||
|
mirror_url: str,
|
||||||
|
mirror_state: ForwardState,
|
||||||
|
*,
|
||||||
|
reachable_fn=mirror_reachable,
|
||||||
|
reachable_timeout: float = 3.0,
|
||||||
|
**forward_kwargs: Any,
|
||||||
|
) -> Optional[Dict[str, int]]:
|
||||||
|
"""One best-effort forwarding pass against a *mirror* SFM server.
|
||||||
|
|
||||||
|
The dual-send entry point. Wraps :func:`forward_pending` with two
|
||||||
|
properties that keep the mirror from ever harming the primary path:
|
||||||
|
|
||||||
|
1. **Fast-fail reachability guard** — probe ``mirror_url`` first; if
|
||||||
|
it's down, return ``None`` immediately rather than let
|
||||||
|
:func:`forward_pending` block on a per-event timeout for every
|
||||||
|
pending file.
|
||||||
|
2. **Total exception isolation** — any error (probe, forward, state
|
||||||
|
I/O) is swallowed and reported as ``None``. This function NEVER
|
||||||
|
raises, so the caller's already-completed primary forward is
|
||||||
|
untouched.
|
||||||
|
|
||||||
|
The mirror keeps its OWN ``ForwardState`` file, so its idempotency
|
||||||
|
and retry are independent of the primary's. Nothing is lost while
|
||||||
|
the mirror is down — skipped events stay pending and are delivered
|
||||||
|
on a later pass once it returns.
|
||||||
|
|
||||||
|
Returns the :func:`forward_pending` counts dict on a completed pass,
|
||||||
|
or ``None`` if the mirror was unreachable or the pass raised.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
if not mirror_url:
|
||||||
|
return None
|
||||||
|
if not reachable_fn(mirror_url, reachable_timeout):
|
||||||
|
return None
|
||||||
|
return forward_pending(watch_dir, mirror_url, mirror_state, **forward_kwargs)
|
||||||
|
except Exception:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
# ── Seed-state mode (skip historical backfill on first deploy) ────────────────
|
# ── Seed-state mode (skip historical backfill on first deploy) ────────────────
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -3,7 +3,7 @@
|
|||||||
|
|
||||||
[Setup]
|
[Setup]
|
||||||
AppName=Series 3 Watcher
|
AppName=Series 3 Watcher
|
||||||
AppVersion=1.5.0
|
AppVersion=1.6.0
|
||||||
AppPublisher=Terra-Mechanics Inc.
|
AppPublisher=Terra-Mechanics Inc.
|
||||||
DefaultDirName={pf}\Series3Watcher
|
DefaultDirName={pf}\Series3Watcher
|
||||||
DefaultGroupName=Series 3 Watcher
|
DefaultGroupName=Series 3 Watcher
|
||||||
|
|||||||
+1
-1
@@ -1,5 +1,5 @@
|
|||||||
"""
|
"""
|
||||||
Series 3 Watcher — System Tray Launcher v1.5.0
|
Series 3 Watcher — System Tray Launcher v1.6.0
|
||||||
Requires: pystray, Pillow, tkinter (stdlib)
|
Requires: pystray, Pillow, tkinter (stdlib)
|
||||||
|
|
||||||
Run with: pythonw series3_tray.py (no console window)
|
Run with: pythonw series3_tray.py (no console window)
|
||||||
|
|||||||
+61
-1
@@ -110,6 +110,12 @@ def load_config(path: str) -> Dict[str, Any]:
|
|||||||
# first deploy in a folder that's been accumulating for years.
|
# first deploy in a folder that's been accumulating for years.
|
||||||
# See README "First-time deployment" section.
|
# See README "First-time deployment" section.
|
||||||
"SFM_MAX_FORWARDS_PER_PASS": get_int("SFM_MAX_FORWARDS_PER_PASS", 500),
|
"SFM_MAX_FORWARDS_PER_PASS": get_int("SFM_MAX_FORWARDS_PER_PASS", 500),
|
||||||
|
|
||||||
|
# Mirror (dual-send) — best-effort second destination, default OFF.
|
||||||
|
# See docs/mirror-dual-send-design.md.
|
||||||
|
"MIRROR_API_URL": get_str("MIRROR_API_URL", ""),
|
||||||
|
"MIRROR_SFM_URL": get_str("MIRROR_SFM_URL", ""),
|
||||||
|
"MIRROR_SFM_STATE_FILE": get_str("MIRROR_SFM_STATE_FILE", ""),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -247,7 +253,7 @@ def scan_latest(
|
|||||||
|
|
||||||
|
|
||||||
# --- API heartbeat / SFM telemetry helpers ---
|
# --- API heartbeat / SFM telemetry helpers ---
|
||||||
VERSION = "1.5.0"
|
VERSION = "1.6.0"
|
||||||
|
|
||||||
|
|
||||||
def _read_log_tail(log_file: str, n: int = 25) -> Optional[list]:
|
def _read_log_tail(log_file: str, n: int = 25) -> Optional[list]:
|
||||||
@@ -427,6 +433,33 @@ def run_watcher(state: Dict[str, Any], stop_event: threading.Event) -> None:
|
|||||||
else:
|
else:
|
||||||
print("[CFG] SFM_FORWARD_ENABLED=false (event forwarding disabled)")
|
print("[CFG] SFM_FORWARD_ENABLED=false (event forwarding disabled)")
|
||||||
|
|
||||||
|
# ---- Mirror (dual-send) setup ----
|
||||||
|
# Best-effort second destination. The event mirror gets its OWN state
|
||||||
|
# file so its idempotency is independent of the primary's.
|
||||||
|
MIRROR_API_URL = cfg.get("MIRROR_API_URL", "")
|
||||||
|
MIRROR_SFM_URL = cfg.get("MIRROR_SFM_URL", "")
|
||||||
|
mirror_state = None
|
||||||
|
if sfm_state is not None and MIRROR_SFM_URL:
|
||||||
|
try:
|
||||||
|
from event_forwarder import ForwardState
|
||||||
|
mirror_state_file = cfg.get("MIRROR_SFM_STATE_FILE") or os.path.join(
|
||||||
|
os.path.dirname(LOG_FILE) or here, "sfm_forwarded_mirror.json"
|
||||||
|
)
|
||||||
|
mirror_state = ForwardState(mirror_state_file)
|
||||||
|
print("[CFG] mirror event forwarding → {} state={} ({} known)".format(
|
||||||
|
MIRROR_SFM_URL, mirror_state_file, mirror_state.count()))
|
||||||
|
log_message(LOG_FILE, ENABLE_LOGGING,
|
||||||
|
"[mirror] event mirror enabled url={} state={} known={}".format(
|
||||||
|
MIRROR_SFM_URL, mirror_state_file, mirror_state.count()))
|
||||||
|
except Exception as e:
|
||||||
|
print("[WARN] mirror forwarder init failed: {}".format(e))
|
||||||
|
log_message(LOG_FILE, ENABLE_LOGGING,
|
||||||
|
"[warn] mirror forwarder init failed: {}".format(e))
|
||||||
|
mirror_state = None
|
||||||
|
if MIRROR_API_URL:
|
||||||
|
log_message(LOG_FILE, ENABLE_LOGGING,
|
||||||
|
"[mirror] heartbeat mirror enabled url={}".format(MIRROR_API_URL))
|
||||||
|
|
||||||
while not stop_event.is_set():
|
while not stop_event.is_set():
|
||||||
try:
|
try:
|
||||||
now_local = datetime.now().isoformat()
|
now_local = datetime.now().isoformat()
|
||||||
@@ -497,6 +530,13 @@ def run_watcher(state: Dict[str, Any], stop_event: threading.Event) -> None:
|
|||||||
hb_payload["log_tail"] = _read_log_tail(cfg.get("LOG_FILE", ""), 25)
|
hb_payload["log_tail"] = _read_log_tail(cfg.get("LOG_FILE", ""), 25)
|
||||||
response = send_api_payload(hb_payload, cfg.get("API_URL", ""))
|
response = send_api_payload(hb_payload, cfg.get("API_URL", ""))
|
||||||
last_api_ts = now_ts
|
last_api_ts = now_ts
|
||||||
|
# Best-effort heartbeat mirror — never affects primary.
|
||||||
|
if MIRROR_API_URL:
|
||||||
|
try:
|
||||||
|
send_api_payload(hb_payload, MIRROR_API_URL)
|
||||||
|
except Exception as e:
|
||||||
|
log_message(LOG_FILE, ENABLE_LOGGING,
|
||||||
|
"[mirror] heartbeat post failed: {}".format(e))
|
||||||
if response is not None:
|
if response is not None:
|
||||||
state["api_status"] = "ok"
|
state["api_status"] = "ok"
|
||||||
state["last_api"] = datetime.now()
|
state["last_api"] = datetime.now()
|
||||||
@@ -550,6 +590,26 @@ def run_watcher(state: Dict[str, Any], stop_event: threading.Event) -> None:
|
|||||||
print(err)
|
print(err)
|
||||||
log_message(LOG_FILE, ENABLE_LOGGING, err)
|
log_message(LOG_FILE, ENABLE_LOGGING, err)
|
||||||
state["sfm_status"] = "fail"
|
state["sfm_status"] = "fail"
|
||||||
|
|
||||||
|
# Best-effort event mirror — own state, fast-fail guard,
|
||||||
|
# fully isolated. Runs after the primary each tick.
|
||||||
|
if mirror_state is not None:
|
||||||
|
from event_forwarder import mirror_forward_pass
|
||||||
|
m_counts = mirror_forward_pass(
|
||||||
|
WATCH_PATH, MIRROR_SFM_URL, mirror_state,
|
||||||
|
max_age_days=MAX_EVENT_AGE_DAYS,
|
||||||
|
quiescence_seconds=int(cfg.get("SFM_QUIESCENCE_SECONDS", 5)),
|
||||||
|
missing_report_grace_seconds=int(cfg.get("SFM_MISSING_REPORT_GRACE_SECONDS", 60)),
|
||||||
|
timeout=int(cfg.get("SFM_HTTP_TIMEOUT", 60)),
|
||||||
|
max_per_pass=int(cfg.get("SFM_MAX_FORWARDS_PER_PASS", 500)),
|
||||||
|
logger=lambda m: log_message(LOG_FILE, ENABLE_LOGGING, "[mirror] " + m),
|
||||||
|
)
|
||||||
|
if m_counts is None:
|
||||||
|
log_message(LOG_FILE, ENABLE_LOGGING,
|
||||||
|
"[mirror] forward skipped (unreachable/error): {}".format(MIRROR_SFM_URL))
|
||||||
|
else:
|
||||||
|
log_message(LOG_FILE, ENABLE_LOGGING,
|
||||||
|
"[mirror] pass forwarded={forwarded} errors={errors}".format(**m_counts))
|
||||||
else:
|
else:
|
||||||
state["sfm_status"] = "disabled"
|
state["sfm_status"] = "disabled"
|
||||||
|
|
||||||
|
|||||||
+77
-1
@@ -1,5 +1,5 @@
|
|||||||
"""
|
"""
|
||||||
Series 3 Watcher — Settings Dialog v1.5.0
|
Series 3 Watcher — Settings Dialog v1.6.0
|
||||||
|
|
||||||
Provides a Tkinter settings dialog that doubles as a first-run wizard.
|
Provides a Tkinter settings dialog that doubles as a first-run wizard.
|
||||||
|
|
||||||
@@ -53,6 +53,11 @@ DEFAULTS = {
|
|||||||
"SFM_HTTP_TIMEOUT": "60",
|
"SFM_HTTP_TIMEOUT": "60",
|
||||||
"SFM_STATE_FILE": "",
|
"SFM_STATE_FILE": "",
|
||||||
"SFM_MAX_FORWARDS_PER_PASS": "500",
|
"SFM_MAX_FORWARDS_PER_PASS": "500",
|
||||||
|
|
||||||
|
# Mirror (dual-send) — best-effort second destination, default OFF.
|
||||||
|
"MIRROR_API_URL": "",
|
||||||
|
"MIRROR_SFM_URL": "",
|
||||||
|
"MIRROR_SFM_STATE_FILE": "",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -260,6 +265,14 @@ class SettingsDialog:
|
|||||||
self.var_sfm_state_file = tk.StringVar(value=v["SFM_STATE_FILE"])
|
self.var_sfm_state_file = tk.StringVar(value=v["SFM_STATE_FILE"])
|
||||||
self.var_sfm_max_per_pass = tk.StringVar(value=v["SFM_MAX_FORWARDS_PER_PASS"])
|
self.var_sfm_max_per_pass = tk.StringVar(value=v["SFM_MAX_FORWARDS_PER_PASS"])
|
||||||
|
|
||||||
|
# Mirror (dual-send) — best-effort second destination
|
||||||
|
_raw_mirror = v["MIRROR_API_URL"]
|
||||||
|
if _raw_mirror.endswith(_suffix):
|
||||||
|
_raw_mirror = _raw_mirror[:-len(_suffix)]
|
||||||
|
self.var_mirror_api_url = tk.StringVar(value=_raw_mirror)
|
||||||
|
self.var_mirror_sfm_url = tk.StringVar(value=v["MIRROR_SFM_URL"])
|
||||||
|
self.var_mirror_sfm_state_file = tk.StringVar(value=v["MIRROR_SFM_STATE_FILE"])
|
||||||
|
|
||||||
# --- UI construction ---
|
# --- UI construction ---
|
||||||
|
|
||||||
def _build_ui(self):
|
def _build_ui(self):
|
||||||
@@ -288,6 +301,7 @@ class SettingsDialog:
|
|||||||
self._build_tab_logging(nb)
|
self._build_tab_logging(nb)
|
||||||
self._build_tab_updates(nb)
|
self._build_tab_updates(nb)
|
||||||
self._build_tab_sfm(nb)
|
self._build_tab_sfm(nb)
|
||||||
|
self._build_tab_mirror(nb)
|
||||||
|
|
||||||
# Buttons
|
# Buttons
|
||||||
btn_frame = tk.Frame(outer)
|
btn_frame = tk.Frame(outer)
|
||||||
@@ -566,6 +580,58 @@ class SettingsDialog:
|
|||||||
row=8, column=0, columnspan=2, sticky="w", padx=(8, 8), pady=(8, 4)
|
row=8, column=0, columnspan=2, sticky="w", padx=(8, 8), pady=(8, 4)
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def _build_tab_mirror(self, nb):
|
||||||
|
"""Optional dual-send mirror: a second heartbeat + event destination."""
|
||||||
|
f = self._tab_frame(nb, "Mirror")
|
||||||
|
|
||||||
|
intro = (
|
||||||
|
"Optional dual-send: post each heartbeat and event to a SECOND\n"
|
||||||
|
"(\"mirror\") server in addition to the primary. Best-effort — the\n"
|
||||||
|
"mirror can never delay or fail the primary. Leave blank to disable."
|
||||||
|
)
|
||||||
|
tk.Label(f, text=intro, justify="left", fg="#1a5276", wraplength=420).grid(
|
||||||
|
row=0, column=0, columnspan=2, sticky="w", padx=(8, 8), pady=(6, 8)
|
||||||
|
)
|
||||||
|
|
||||||
|
_add_label_entry(f, 1, "Mirror Terra-View URL", self.var_mirror_api_url)
|
||||||
|
_add_label_entry(f, 2, "Mirror SFM URL", self.var_mirror_sfm_url)
|
||||||
|
|
||||||
|
tk.Label(f, text="Mirror State File", anchor="w").grid(
|
||||||
|
row=3, column=0, sticky="w", padx=(8, 4), pady=4
|
||||||
|
)
|
||||||
|
state_frame = tk.Frame(f)
|
||||||
|
state_frame.grid(row=3, column=1, sticky="ew", padx=(0, 8), pady=4)
|
||||||
|
state_frame.columnconfigure(0, weight=1)
|
||||||
|
ttk.Entry(state_frame, textvariable=self.var_mirror_sfm_state_file, width=32).grid(
|
||||||
|
row=0, column=0, sticky="ew"
|
||||||
|
)
|
||||||
|
|
||||||
|
def _browse_mirror_state():
|
||||||
|
path = filedialog.asksaveasfilename(
|
||||||
|
title="Mirror forward-state file",
|
||||||
|
defaultextension=".json",
|
||||||
|
filetypes=[("JSON", "*.json"), ("All Files", "*.*")],
|
||||||
|
initialfile="sfm_forwarded_mirror.json",
|
||||||
|
)
|
||||||
|
if path:
|
||||||
|
self.var_mirror_sfm_state_file.set(path)
|
||||||
|
|
||||||
|
ttk.Button(state_frame, text="Browse...", width=10,
|
||||||
|
command=_browse_mirror_state).grid(row=0, column=1, padx=(4, 0))
|
||||||
|
|
||||||
|
hint_text = (
|
||||||
|
"Mirror Terra-View URL → base like http://10.0.0.x:8001 (heartbeats).\n"
|
||||||
|
"Mirror SFM URL → base like http://10.0.0.x:8200 (events).\n\n"
|
||||||
|
"The event mirror keeps its OWN state file, so nothing is lost while\n"
|
||||||
|
"the mirror is down — pending events are delivered once it returns.\n"
|
||||||
|
"A quick reachability check skips the mirror cleanly when unreachable,\n"
|
||||||
|
"so the primary is never slowed.\n"
|
||||||
|
"State File blank → defaults to <log dir>/sfm_forwarded_mirror.json."
|
||||||
|
)
|
||||||
|
tk.Label(f, text=hint_text, justify="left", fg="#555555", wraplength=420).grid(
|
||||||
|
row=4, column=0, columnspan=2, sticky="w", padx=(8, 8), pady=(8, 4)
|
||||||
|
)
|
||||||
|
|
||||||
def _test_sfm_connection(self):
|
def _test_sfm_connection(self):
|
||||||
"""GET <sfm_url>/health and show the result."""
|
"""GET <sfm_url>/health and show the result."""
|
||||||
import urllib.request
|
import urllib.request
|
||||||
@@ -673,6 +739,12 @@ class SettingsDialog:
|
|||||||
else:
|
else:
|
||||||
api_url = api_url.rstrip("/") + "/api/series3/heartbeat"
|
api_url = api_url.rstrip("/") + "/api/series3/heartbeat"
|
||||||
|
|
||||||
|
# Mirror (dual-send) — best-effort second destination.
|
||||||
|
mirror_api_url = self.var_mirror_api_url.get().strip()
|
||||||
|
if mirror_api_url:
|
||||||
|
mirror_api_url = mirror_api_url.rstrip("/") + "/api/series3/heartbeat"
|
||||||
|
mirror_sfm_url = self.var_mirror_sfm_url.get().strip().rstrip("/")
|
||||||
|
|
||||||
values = {
|
values = {
|
||||||
"API_ENABLED": "true" if self.var_api_enabled.get() else "false",
|
"API_ENABLED": "true" if self.var_api_enabled.get() else "false",
|
||||||
"API_URL": api_url,
|
"API_URL": api_url,
|
||||||
@@ -698,6 +770,10 @@ class SettingsDialog:
|
|||||||
"SFM_HTTP_TIMEOUT": str(int_values["SFM HTTP Timeout"]),
|
"SFM_HTTP_TIMEOUT": str(int_values["SFM HTTP Timeout"]),
|
||||||
"SFM_STATE_FILE": self.var_sfm_state_file.get().strip(),
|
"SFM_STATE_FILE": self.var_sfm_state_file.get().strip(),
|
||||||
"SFM_MAX_FORWARDS_PER_PASS": str(int_values["SFM Max Events Per Pass"]),
|
"SFM_MAX_FORWARDS_PER_PASS": str(int_values["SFM Max Events Per Pass"]),
|
||||||
|
|
||||||
|
"MIRROR_API_URL": mirror_api_url,
|
||||||
|
"MIRROR_SFM_URL": mirror_sfm_url,
|
||||||
|
"MIRROR_SFM_STATE_FILE": self.var_mirror_sfm_state_file.get().strip(),
|
||||||
}
|
}
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -714,5 +714,127 @@ class TestForwardEventPair(unittest.TestCase):
|
|||||||
self.assertIn("serial=BE11529", req["path"])
|
self.assertIn("serial=BE11529", req["path"])
|
||||||
|
|
||||||
|
|
||||||
|
# ── Mirror (dual-send) ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def _mk_mirror_event(watch_dir, name="M529LK44.AB0", age_seconds=200, content=b"binary"):
|
||||||
|
"""Create a Blastware-shaped event binary with a controlled mtime."""
|
||||||
|
p = Path(watch_dir) / name
|
||||||
|
p.write_bytes(content)
|
||||||
|
target = time.time() - age_seconds
|
||||||
|
os.utime(str(p), (target, target))
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
class TestMirrorReachable(unittest.TestCase):
|
||||||
|
|
||||||
|
def test_empty_url_is_unreachable(self):
|
||||||
|
self.assertFalse(ef.mirror_reachable("", timeout=1.0))
|
||||||
|
|
||||||
|
def test_dead_port_is_unreachable_and_fast(self):
|
||||||
|
t0 = time.time()
|
||||||
|
self.assertFalse(ef.mirror_reachable("http://127.0.0.1:1", timeout=2.0))
|
||||||
|
self.assertLess(time.time() - t0, 2.5)
|
||||||
|
|
||||||
|
def test_any_http_response_counts_as_reachable(self):
|
||||||
|
server, base = _start_fake_server()
|
||||||
|
try:
|
||||||
|
self.assertTrue(ef.mirror_reachable(base, timeout=2.0))
|
||||||
|
finally:
|
||||||
|
server.shutdown()
|
||||||
|
server.server_close()
|
||||||
|
|
||||||
|
|
||||||
|
class TestMirrorForwardPass(unittest.TestCase):
|
||||||
|
"""The dual-send entry point: best-effort, isolated, own state."""
|
||||||
|
|
||||||
|
def setUp(self):
|
||||||
|
_FakeImportHandler.received = []
|
||||||
|
self.server, self.base_url = _start_fake_server()
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.server.shutdown()
|
||||||
|
self.server.server_close()
|
||||||
|
|
||||||
|
def test_empty_mirror_url_is_noop(self):
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
_mk_mirror_event(tmp)
|
||||||
|
mstate = ef.ForwardState(os.path.join(tmp, "mirror.json"))
|
||||||
|
self.assertIsNone(ef.mirror_forward_pass(tmp, "", mstate, max_age_days=30))
|
||||||
|
self.assertEqual(len(_FakeImportHandler.received), 0)
|
||||||
|
|
||||||
|
def test_skips_when_unreachable_without_posting(self):
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
_mk_mirror_event(tmp)
|
||||||
|
mstate = ef.ForwardState(os.path.join(tmp, "mirror.json"))
|
||||||
|
result = ef.mirror_forward_pass(
|
||||||
|
tmp, self.base_url, mstate,
|
||||||
|
reachable_fn=lambda url, timeout=3.0: False,
|
||||||
|
max_age_days=30, quiescence_seconds=5,
|
||||||
|
missing_report_grace_seconds=60, timeout=5.0,
|
||||||
|
)
|
||||||
|
self.assertIsNone(result)
|
||||||
|
self.assertEqual(mstate.count(), 0)
|
||||||
|
self.assertEqual(len(_FakeImportHandler.received), 0)
|
||||||
|
|
||||||
|
def test_forwards_when_reachable_using_its_own_state(self):
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
_mk_mirror_event(tmp)
|
||||||
|
mstate = ef.ForwardState(os.path.join(tmp, "mirror.json"))
|
||||||
|
counts = ef.mirror_forward_pass(
|
||||||
|
tmp, self.base_url, mstate,
|
||||||
|
reachable_fn=lambda url, timeout=3.0: True,
|
||||||
|
max_age_days=30, quiescence_seconds=5,
|
||||||
|
missing_report_grace_seconds=60, timeout=5.0,
|
||||||
|
)
|
||||||
|
self.assertIsNotNone(counts)
|
||||||
|
self.assertEqual(counts["forwarded"], 1)
|
||||||
|
self.assertEqual(mstate.count(), 1)
|
||||||
|
self.assertEqual(len(_FakeImportHandler.received), 1)
|
||||||
|
|
||||||
|
def test_never_raises_when_forward_blows_up(self):
|
||||||
|
def _boom(*a, **k):
|
||||||
|
raise RuntimeError("boom")
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
_mk_mirror_event(tmp)
|
||||||
|
mstate = ef.ForwardState(os.path.join(tmp, "mirror.json"))
|
||||||
|
orig = ef.forward_pending
|
||||||
|
ef.forward_pending = _boom
|
||||||
|
try:
|
||||||
|
result = ef.mirror_forward_pass(
|
||||||
|
tmp, self.base_url, mstate,
|
||||||
|
reachable_fn=lambda url, timeout=3.0: True,
|
||||||
|
max_age_days=30,
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
ef.forward_pending = orig
|
||||||
|
self.assertIsNone(result)
|
||||||
|
|
||||||
|
def test_down_mirror_leaves_primary_state_untouched(self):
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
_mk_mirror_event(tmp)
|
||||||
|
|
||||||
|
primary_state = ef.ForwardState(os.path.join(tmp, "primary.json"))
|
||||||
|
pcounts = ef.forward_pending(
|
||||||
|
tmp, self.base_url, primary_state,
|
||||||
|
max_age_days=30, quiescence_seconds=5,
|
||||||
|
missing_report_grace_seconds=60, timeout=5.0,
|
||||||
|
)
|
||||||
|
self.assertEqual(pcounts["forwarded"], 1)
|
||||||
|
primary_snapshot = primary_state.count()
|
||||||
|
|
||||||
|
mstate = ef.ForwardState(os.path.join(tmp, "mirror.json"))
|
||||||
|
t0 = time.time()
|
||||||
|
result = ef.mirror_forward_pass(
|
||||||
|
tmp, "http://127.0.0.1:1", mstate, # nothing listens
|
||||||
|
max_age_days=30, quiescence_seconds=5,
|
||||||
|
missing_report_grace_seconds=60, timeout=5.0,
|
||||||
|
)
|
||||||
|
self.assertIsNone(result)
|
||||||
|
self.assertLess(time.time() - t0, 3.5)
|
||||||
|
self.assertEqual(mstate.count(), 0)
|
||||||
|
self.assertEqual(primary_state.count(), primary_snapshot)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
unittest.main()
|
unittest.main()
|
||||||
|
|||||||
Reference in New Issue
Block a user