Merge pull request 'fix: updater survives Settings saves + cache unsniffable MLGs (v1.6.1)' (#13) from dev into main

Reviewed-on: #13

CHANGELOG.md

@@ -6,6 +6,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ---
 
+## [6-25-26] — v1.6.1
+
+Reliability fixes for the tray app's auto-updater and scan logging.
+
+### Fixed
+- **Auto-updater no longer dies when Settings are saved.**  Saving the Settings dialog restarts the watcher thread via `_restart_watcher`, which sets and then replaces the shared `stop_event`.  Because the updater loop keyed off that *same* event, saving settings would terminate the auto-updater thread (a race — it could survive one save and die on the next).  The watcher kept running and reporting heartbeats, but the agent silently stopped checking for / applying updates until the tray app was relaunched.  The updater now keys off its own app-lifetime event (`_app_stop`) that watcher restarts never touch; it stops only on Exit or when applying an update.  (Same fix as thor-watcher 0.4.1.)
+- **Unsniffable `.MLG` files are logged once, not every scan.**  A `.MLG` file the watcher couldn't extract a unit ID from was never cached, so every scan (~every 5 min) re-sniffed *and* re-emitted its `[unsniffable-recent]` log line.  With a few hundred such files in a long-lived Blastware autocall folder that meant tens of thousands of duplicate lines per day (a 10-day-old log had grown to ~600K lines).  The scan now caches the unsniffable result, so each file is sniffed and logged once per session — and the redundant per-scan header reads are gone.
+- Added `test_scan_latest.py` (unsniffable cached / logged-once + happy-path sniff) and `test_series3_tray.py` (updater survives a watcher restart; stops on exit).
+
+## [6-25-26] — v1.6.0
+
+Optional dual-send mirror so a standby server stays a live replica.
+
+### Added — Mirror (dual-send)
+- **Post every heartbeat and event to a second ("mirror") server in addition to the primary.**  When `MIRROR_API_URL` and/or `MIRROR_SFM_URL` are set, each heartbeat POST and each forwarded Blastware event is *also* sent to the mirror destination — letting a standby box (e.g. the office NAS) stay a continuous replica of prod during a migration so the eventual cutover is a non-event.  **Default off** — blank URLs mean existing 1.5.x installs don't change behaviour after an auto-update.
+- **Heartbeat mirror (best-effort).**  After the primary `send_api_payload`, the same payload is fired at `MIRROR_API_URL` when set.  It is fully wrapped and bounded by the existing API timeout — the mirror can **never** delay or fail the primary heartbeat; its outcome is ignored except for a debug log line.
+- **Event mirror (reliable, isolated).**  Events are re-forwarded to `MIRROR_SFM_URL` through a *separate* sha256 state file (`MIRROR_SFM_STATE_FILE`, blank → `<log dir>/sfm_forwarded_mirror.json`) so the mirror tracks its own delivery independently of the primary forwarder.  Before each mirror pass a quick reachability probe (~3 s) checks the mirror server; if it's down the pass is skipped and retried next tick rather than blocking the loop on per-event timeouts.  Skipped events stay pending in the mirror state and deliver once the mirror returns — **no data loss**.
+- **Isolation invariant.**  Nothing on the mirror path can delay or fail the primary: its own state file, its own try/except, the reachability guard + bounded timeouts, and all exceptions swallowed-and-logged.  The primary path stays exactly as reliable as before.  The mirror rides along with the primary — there is no separate enable flag (heartbeat-mirror active iff the primary heartbeat is on *and* `MIRROR_API_URL` is set; event-mirror active iff primary forwarding is on *and* `MIRROR_SFM_URL` is set).
+- **Mirror fields in the Settings dialog**: `Mirror API URL` + `Mirror SFM URL` (blank = off).
+- New unit tests in `test_event_forwarder.py` covering mirror reachability, the down-mirror skip, separate-state idempotency, and the isolation invariant (a mirror failure leaves the primary's result + state untouched).
+
+### Configuration
+
+New `[agent]` keys (all default-off — existing 1.5.x deployments don't change behaviour on auto-update):
+
+| Key | Default | Notes |
+|---|---|---|
+| `MIRROR_API_URL` | empty | Second heartbeat base URL, e.g. `http://10.0.0.x:8001` (blank = off) |
+| `MIRROR_SFM_URL` | empty | Second SFM base URL, e.g. `http://10.0.0.x:8200` (blank = off) |
+| `MIRROR_SFM_STATE_FILE` | `<log dir>/sfm_forwarded_mirror.json` | Override location of the mirror's forwarded-sha256 state file |
+
+### Operational (migration seeding)
+
+To make the mirror re-deliver only the gap since a one-time replica snapshot, copy the primary state file to the mirror state file and drop entries whose `forwarded_at` is after the snapshot.  SFM-side dedup covers any overlap.  See `docs/mirror-dual-send-design.md`.
+
 ## [5-11-26] — v1.5.0
 
 First release of the SFM event forwarder.

README.md

@@ -1,4 +1,4 @@
-# Series 3 Watcher v1.5.0
+# Series 3 Watcher v1.6.1
 
 Monitors Instantel **Series 3 (Minimate)** call-in activity on a Blastware server. Runs as a **system tray app** that starts automatically on login, reports heartbeats to terra-view, and self-updates from Gitea.
 
@@ -124,6 +124,27 @@ Combine both for a fully controlled rollout: seed-state to skip the deep archive
 
 ---
 
+## Mirror (Dual-Send) — v1.6.0+
+
+The watcher can post every heartbeat and event to a **second ("mirror") server** in addition to the primary — useful for keeping a standby box (e.g. the office NAS) as a continuous replica during a server migration, so the eventual cutover is a non-event. **Default off.**
+
+The mirror rides along with the primary — there is no separate enable flag:
+
+- **Heartbeat mirror** is active when `API_ENABLED = true` **and** `MIRROR_API_URL` is set. After the primary heartbeat POST, the same payload is fired at the mirror.
+- **Event mirror** is active when `SFM_FORWARD_ENABLED = true` **and** `MIRROR_SFM_URL` is set. Events are re-forwarded to the mirror using a **separate** sha256 state file (`MIRROR_SFM_STATE_FILE`, blank → `<log dir>/sfm_forwarded_mirror.json`).
+
+**The mirror can never delay or fail the primary.** The heartbeat mirror is fully wrapped and bounded by the API timeout. The event mirror runs a quick reachability probe (~3 s) first and skips the pass if the mirror server is down — skipped events stay pending in the mirror state and deliver when it returns, so nothing is lost. All mirror exceptions are swallowed-and-logged; the primary path stays exactly as reliable as before.
+
+| Key | Description |
+|-----|-------------|
+| `MIRROR_API_URL` | Second heartbeat base URL, e.g. `http://10.0.0.x:8001` (blank = off) |
+| `MIRROR_SFM_URL` | Second SFM base URL, e.g. `http://10.0.0.x:8200` (blank = off) |
+| `MIRROR_SFM_STATE_FILE` | Path to the mirror's sha256 state file. Blank → `<log dir>/sfm_forwarded_mirror.json` |
+
+**Seeding for a migration.** To re-deliver only the gap since a one-time replica snapshot, copy the primary state file to the mirror state file and drop entries whose `forwarded_at` is after the snapshot. SFM-side dedup covers any overlap. See [`docs/mirror-dual-send-design.md`](docs/mirror-dual-send-design.md).
+
+---
+
 ## Tray Icon
 
 | Colour | Meaning |
@@ -167,7 +188,7 @@ where the corresponding server-side work lives.
 
 ## Versioning
 
-Follows **Semantic Versioning**. Current release: **v1.5.0**.
+Follows **Semantic Versioning**. Current release: **v1.6.1**.
 See `CHANGELOG.md` for full history.
 
 ---

config-template.ini

@@ -72,3 +72,14 @@ SFM_STATE_FILE =
 # `--seed-state` workflow that skips the historical backfill entirely.
 SFM_MAX_FORWARDS_PER_PASS = 500
 
+# --- Mirror (dual-send) ---
+# Optional best-effort second destination: post each heartbeat and event
+# to a mirror server (e.g. the office NAS) IN ADDITION to the primary
+# above.  Default off (blank URLs).  The mirror can never delay or fail
+# the primary; the event mirror keeps its own state file so nothing is
+# lost while it's down.  See docs/mirror-dual-send-design.md.
+MIRROR_API_URL =                          ; base URL e.g. http://10.0.0.x:8001 (blank = off)
+MIRROR_SFM_URL =                          ; base URL e.g. http://10.0.0.x:8200 (blank = off)
+# Blank → defaults to <log dir>/sfm_forwarded_mirror.json
+MIRROR_SFM_STATE_FILE =
+

docs/mirror-dual-send-design.md

@@ -0,0 +1,51 @@
+# Watcher Mirror (Dual-Send) — Design
+
+**Date:** 2026-06-24 · **Branch:** `feat/mirror-dual-send` (both `thor-watcher` + `series3-watcher`) · **Status:** approved, implementing
+
+## Goal
+Let each watcher post every heartbeat and event to a **second ("mirror") server** in addition to the primary, so the office NAS stays a continuous replica of prod during the migration. The eventual cutover becomes a non-event. **Default off** → existing installs unchanged.
+
+## Pattern
+Classic dual-write migration. The primary path stays **exactly as reliable as today**; the mirror is pure best-effort bonus that can **never delay or fail** the primary.
+
+## The two watchers (parallel changes)
+| | thor-watcher (series4) | series3-watcher (s3) |
+|---|---|---|
+| Config | `config.json` (JSON) | `config.ini` (`[agent]`, INI) |
+| Heartbeat fn | `send_api_payload(payload, api_url, timeout)` | `send_api_payload(payload, api_url)` |
+| Event fn | `event_forwarder.forward_pending(root, sfm_url, state, …)` | same (`event_forwarder` is a sibling port) |
+| Loop | `series4_ingest.run_watcher` | `series3_watcher.run_watcher` |
+| Settings GUI | `thor_settings_dialog.py` | `settings_dialog.py` |
+
+## Design (Approach A — reliable event mirror, best-effort heartbeat)
+
+### Config additions (default empty = off)
+- thor (`config.json`): `mirror_api_url`, `mirror_sfm_url`, `mirror_sfm_state_file` (blank → `<log_dir>/thor_forwarded_mirror.json`).
+- s3 (`config.ini`): `MIRROR_API_URL`, `MIRROR_SFM_URL`, `MIRROR_SFM_STATE_FILE` (blank → `sfm_forwarded_mirror.json`).
+
+The mirror **rides along** with the primary: heartbeat-mirror active iff primary heartbeat on **and** `mirror_api_url` set; event-mirror active iff primary forwarding on **and** `mirror_sfm_url` set. No separate enable flag.
+
+### Heartbeat mirror (best-effort)
+After the primary `send_api_payload`, if `mirror_api_url` is set, fire one more `send_api_payload(payload, mirror_api_url, …)`. Wrapped so it can never raise into the loop; result ignored except a debug log. Bounded by the existing short API timeout (~5 s).
+
+### Event mirror (reliable, isolated)
+- Init a **second** `ForwardState(mirror_state_path)` alongside the primary state.
+- Each forward tick, **after** the primary `forward_pending`, run a second `forward_pending(root, mirror_sfm_url, mirror_state, …)` in its **own** try/except.
+- **Reachability guard:** before the mirror forward, a quick TCP/HTTP probe of `mirror_sfm_url` (~3 s). If unreachable → skip this pass, log, retry next tick. Prevents a down NAS from stalling the loop on per-event timeouts. The mirror state file means skipped events stay pending → delivered when the NAS returns. **No data loss.**
+- Reliable + idempotent via its own sha256 state, fully independent of the primary's state.
+
+### Isolation invariant (the one rule)
+Nothing on the mirror path can delay or fail the primary: separate state file, its own try/except, the reachability guard + bounded timeouts, all exceptions swallowed-and-logged.
+
+### Settings dialog
+Add `Mirror API URL` + `Mirror SFM URL` fields (blank = off) to each watcher's settings dialog.
+
+## Seeding (operational, for this migration)
+The mirror state file should reflect what the NAS already has (everything ≤ the migration snapshot, ~16:23 on 2026-06-24). Recipe: copy the primary state file → mirror state file, then drop entries whose `forwarded_at` is after the snapshot, so the mirror re-delivers exactly the gap. SFM-side dedup covers any overlap. (If a watcher had no events in the gap, a straight copy suffices.)
+
+## Testing
+- thor `test_event_forwarder.py`: mirror uses a separate state file (idempotent, independent); a mirror failure leaves the primary's result + state untouched; reachability guard skips cleanly when the mirror is down.
+- s3: add a minimal test for the same isolation/idempotency.
+
+## Out of scope / handoff
+Rebuilding the Windows installers + redeploying (s3 on the Win7 box, thor on the Win10 box) is the operator's job — this change is **code + tests only**.

event_forwarder.py

@@ -684,6 +684,74 @@ def forward_pending(
     return counts
 
 
+# ── Mirror (dual-send) ────────────────────────────────────────────────────────
+
+
+def mirror_reachable(base_url: str, timeout: float = 3.0) -> bool:
+    """Quick liveness probe of a mirror SFM base URL.
+
+    Run before a mirror forward pass so an unreachable mirror can't stall
+    the watcher loop on a per-event HTTP timeout for every pending file.
+    Probes ``<base>/health`` (the SFM server exposes it).  ANY HTTP
+    response — even 404/500 — means the server is up, so proceed; only a
+    connection-level failure (refused / timed out / DNS) counts as down.
+    Best-effort: never raises.
+    """
+    if not base_url:
+        return False
+    url = base_url.rstrip("/") + "/health"
+    try:
+        with urllib.request.urlopen(
+            urllib.request.Request(url, method="GET"), timeout=timeout
+        ):
+            return True
+    except urllib.error.HTTPError:
+        return True   # got a status back → server is alive
+    except Exception:
+        return False  # refused / timeout / DNS / socket → down
+
+
+def mirror_forward_pass(
+    watch_dir: str,
+    mirror_url: str,
+    mirror_state: ForwardState,
+    *,
+    reachable_fn=mirror_reachable,
+    reachable_timeout: float = 3.0,
+    **forward_kwargs: Any,
+) -> Optional[Dict[str, int]]:
+    """One best-effort forwarding pass against a *mirror* SFM server.
+
+    The dual-send entry point.  Wraps :func:`forward_pending` with two
+    properties that keep the mirror from ever harming the primary path:
+
+    1. **Fast-fail reachability guard** — probe ``mirror_url`` first; if
+       it's down, return ``None`` immediately rather than let
+       :func:`forward_pending` block on a per-event timeout for every
+       pending file.
+    2. **Total exception isolation** — any error (probe, forward, state
+       I/O) is swallowed and reported as ``None``.  This function NEVER
+       raises, so the caller's already-completed primary forward is
+       untouched.
+
+    The mirror keeps its OWN ``ForwardState`` file, so its idempotency
+    and retry are independent of the primary's.  Nothing is lost while
+    the mirror is down — skipped events stay pending and are delivered
+    on a later pass once it returns.
+
+    Returns the :func:`forward_pending` counts dict on a completed pass,
+    or ``None`` if the mirror was unreachable or the pass raised.
+    """
+    try:
+        if not mirror_url:
+            return None
+        if not reachable_fn(mirror_url, reachable_timeout):
+            return None
+        return forward_pending(watch_dir, mirror_url, mirror_state, **forward_kwargs)
+    except Exception:
+        return None
+
+
 # ── Seed-state mode (skip historical backfill on first deploy) ────────────────
 
 

installer.iss

@@ -3,7 +3,7 @@
 
 [Setup]
 AppName=Series 3 Watcher
-AppVersion=1.5.0
+AppVersion=1.6.1
 AppPublisher=Terra-Mechanics Inc.
 DefaultDirName={pf}\Series3Watcher
 DefaultGroupName=Series 3 Watcher

series3_tray.py

@@ -1,5 +1,5 @@
 """
-Series 3 Watcher — System Tray Launcher  v1.5.0
+Series 3 Watcher — System Tray Launcher  v1.6.1
 Requires: pystray, Pillow, tkinter (stdlib)
 
 Run with:  pythonw series3_tray.py   (no console window)
@@ -335,7 +335,8 @@ def _show_cancel_message():
 class WatcherTray:
     def __init__(self):
         self.state = {}
-        self.stop_event = threading.Event()
+        self.stop_event = threading.Event()   # watcher lifecycle; replaced on restart
+        self._app_stop = threading.Event()    # app lifetime; only set on exit/update
         self._watcher_thread = None
         self._icon = None
         # Lock guards _rebuild_menu calls from the updater thread
@@ -394,6 +395,7 @@ class WatcherTray:
                 subprocess.Popen(["explorer", HERE])
 
     def _exit(self, icon, item):
+        self._app_stop.set()
         self.stop_event.set()
         icon.stop()
 
@@ -459,7 +461,7 @@ class WatcherTray:
         last_status = None
         update_check_counter = 0  # check for updates every ~5 min (30 * 10s ticks)
 
-        while not self.stop_event.is_set():
+        while not self._app_stop.is_set():
             icon_status = self._tray_status()
 
             if self._icon is not None:
@@ -487,7 +489,7 @@ class WatcherTray:
                     self._do_update(url)
                     return  # exit loop; swap bat will relaunch
 
-            self.stop_event.wait(timeout=10)
+            self._app_stop.wait(timeout=10)
 
     def _do_update(self, download_url=None):
         """Notify tray icon then apply update. If url is None, fetch it first."""
@@ -502,6 +504,7 @@ class WatcherTray:
 
         success = apply_update(download_url)
         if success:
+            self._app_stop.set()
             self.stop_event.set()
             if self._icon is not None:
                 self._icon.stop()

series3_watcher.py

@@ -110,6 +110,12 @@ def load_config(path: str) -> Dict[str, Any]:
         # first deploy in a folder that's been accumulating for years.
         # See README "First-time deployment" section.
         "SFM_MAX_FORWARDS_PER_PASS":   get_int("SFM_MAX_FORWARDS_PER_PASS", 500),
+
+        # Mirror (dual-send) — best-effort second destination, default OFF.
+        # See docs/mirror-dual-send-design.md.
+        "MIRROR_API_URL":        get_str("MIRROR_API_URL", ""),
+        "MIRROR_SFM_URL":        get_str("MIRROR_SFM_URL", ""),
+        "MIRROR_SFM_STATE_FILE": get_str("MIRROR_SFM_STATE_FILE", ""),
     }
 
 
@@ -231,13 +237,16 @@ def scan_latest(
                     uid = cached[1]
                 else:
                     uid = sniff_unit_from_mlg(fpath, header_bytes)
-                    if not uid:
-                        # If unsniffable but very recent, log for later inspection
-                        if (recent_cutoff is not None) and (mtime >= recent_cutoff):
-                            if logger:
-                                logger("[unsniffable-recent] {}".format(fpath))
-                        continue  # skip file if no unit ID found in header
+                    # Cache the result either way — including unsniffable (uid=None) —
+                    # so the same file isn't re-sniffed and re-logged on every scan.
                     cache[fpath] = (mtime, uid)
+                    if (not uid) and (recent_cutoff is not None) and (mtime >= recent_cutoff):
+                        # Log once, on first sight, for later inspection.
+                        if logger:
+                            logger("[unsniffable-recent] {}".format(fpath))
+
+                if not uid:
+                    continue  # no unit ID in header — skip (cached above, won't re-log)
 
                 if (uid not in latest) or (mtime > latest[uid]["mtime"]):
                     latest[uid] = {"mtime": mtime, "fname": e.name, "path": fpath}
@@ -247,7 +256,7 @@ def scan_latest(
 
 
 # --- API heartbeat / SFM telemetry helpers ---
-VERSION = "1.5.0"
+VERSION = "1.6.1"
 
 
 def _read_log_tail(log_file: str, n: int = 25) -> Optional[list]:
@@ -427,6 +436,33 @@ def run_watcher(state: Dict[str, Any], stop_event: threading.Event) -> None:
     else:
         print("[CFG] SFM_FORWARD_ENABLED=false (event forwarding disabled)")
 
+    # ---- Mirror (dual-send) setup ----
+    # Best-effort second destination.  The event mirror gets its OWN state
+    # file so its idempotency is independent of the primary's.
+    MIRROR_API_URL = cfg.get("MIRROR_API_URL", "")
+    MIRROR_SFM_URL = cfg.get("MIRROR_SFM_URL", "")
+    mirror_state = None
+    if sfm_state is not None and MIRROR_SFM_URL:
+        try:
+            from event_forwarder import ForwardState
+            mirror_state_file = cfg.get("MIRROR_SFM_STATE_FILE") or os.path.join(
+                os.path.dirname(LOG_FILE) or here, "sfm_forwarded_mirror.json"
+            )
+            mirror_state = ForwardState(mirror_state_file)
+            print("[CFG] mirror event forwarding → {}  state={} ({} known)".format(
+                MIRROR_SFM_URL, mirror_state_file, mirror_state.count()))
+            log_message(LOG_FILE, ENABLE_LOGGING,
+                "[mirror] event mirror enabled  url={}  state={}  known={}".format(
+                    MIRROR_SFM_URL, mirror_state_file, mirror_state.count()))
+        except Exception as e:
+            print("[WARN] mirror forwarder init failed: {}".format(e))
+            log_message(LOG_FILE, ENABLE_LOGGING,
+                "[warn] mirror forwarder init failed: {}".format(e))
+            mirror_state = None
+    if MIRROR_API_URL:
+        log_message(LOG_FILE, ENABLE_LOGGING,
+            "[mirror] heartbeat mirror enabled  url={}".format(MIRROR_API_URL))
+
     while not stop_event.is_set():
         try:
             now_local = datetime.now().isoformat()
@@ -497,6 +533,13 @@ def run_watcher(state: Dict[str, Any], stop_event: threading.Event) -> None:
                     hb_payload["log_tail"] = _read_log_tail(cfg.get("LOG_FILE", ""), 25)
                     response = send_api_payload(hb_payload, cfg.get("API_URL", ""))
                     last_api_ts = now_ts
+                    # Best-effort heartbeat mirror — never affects primary.
+                    if MIRROR_API_URL:
+                        try:
+                            send_api_payload(hb_payload, MIRROR_API_URL)
+                        except Exception as e:
+                            log_message(LOG_FILE, ENABLE_LOGGING,
+                                "[mirror] heartbeat post failed: {}".format(e))
                     if response is not None:
                         state["api_status"] = "ok"
                         state["last_api"] = datetime.now()
@@ -550,6 +593,26 @@ def run_watcher(state: Dict[str, Any], stop_event: threading.Event) -> None:
                         print(err)
                         log_message(LOG_FILE, ENABLE_LOGGING, err)
                         state["sfm_status"] = "fail"
+
+                    # Best-effort event mirror — own state, fast-fail guard,
+                    # fully isolated.  Runs after the primary each tick.
+                    if mirror_state is not None:
+                        from event_forwarder import mirror_forward_pass
+                        m_counts = mirror_forward_pass(
+                            WATCH_PATH, MIRROR_SFM_URL, mirror_state,
+                            max_age_days=MAX_EVENT_AGE_DAYS,
+                            quiescence_seconds=int(cfg.get("SFM_QUIESCENCE_SECONDS", 5)),
+                            missing_report_grace_seconds=int(cfg.get("SFM_MISSING_REPORT_GRACE_SECONDS", 60)),
+                            timeout=int(cfg.get("SFM_HTTP_TIMEOUT", 60)),
+                            max_per_pass=int(cfg.get("SFM_MAX_FORWARDS_PER_PASS", 500)),
+                            logger=lambda m: log_message(LOG_FILE, ENABLE_LOGGING, "[mirror] " + m),
+                        )
+                        if m_counts is None:
+                            log_message(LOG_FILE, ENABLE_LOGGING,
+                                "[mirror] forward skipped (unreachable/error): {}".format(MIRROR_SFM_URL))
+                        else:
+                            log_message(LOG_FILE, ENABLE_LOGGING,
+                                "[mirror] pass forwarded={forwarded} errors={errors}".format(**m_counts))
             else:
                 state["sfm_status"] = "disabled"
 

settings_dialog.py

@@ -1,5 +1,5 @@
 """
-Series 3 Watcher — Settings Dialog  v1.5.0
+Series 3 Watcher — Settings Dialog  v1.6.1
 
 Provides a Tkinter settings dialog that doubles as a first-run wizard.
 
@@ -53,6 +53,11 @@ DEFAULTS = {
     "SFM_HTTP_TIMEOUT":                 "60",
     "SFM_STATE_FILE":                   "",
     "SFM_MAX_FORWARDS_PER_PASS":        "500",
+
+    # Mirror (dual-send) — best-effort second destination, default OFF.
+    "MIRROR_API_URL":        "",
+    "MIRROR_SFM_URL":        "",
+    "MIRROR_SFM_STATE_FILE": "",
 }
 
 
@@ -260,6 +265,14 @@ class SettingsDialog:
         self.var_sfm_state_file              = tk.StringVar(value=v["SFM_STATE_FILE"])
         self.var_sfm_max_per_pass            = tk.StringVar(value=v["SFM_MAX_FORWARDS_PER_PASS"])
 
+        # Mirror (dual-send) — best-effort second destination
+        _raw_mirror = v["MIRROR_API_URL"]
+        if _raw_mirror.endswith(_suffix):
+            _raw_mirror = _raw_mirror[:-len(_suffix)]
+        self.var_mirror_api_url        = tk.StringVar(value=_raw_mirror)
+        self.var_mirror_sfm_url        = tk.StringVar(value=v["MIRROR_SFM_URL"])
+        self.var_mirror_sfm_state_file = tk.StringVar(value=v["MIRROR_SFM_STATE_FILE"])
+
     # --- UI construction ---
 
     def _build_ui(self):
@@ -288,6 +301,7 @@ class SettingsDialog:
         self._build_tab_logging(nb)
         self._build_tab_updates(nb)
         self._build_tab_sfm(nb)
+        self._build_tab_mirror(nb)
 
         # Buttons
         btn_frame = tk.Frame(outer)
@@ -566,6 +580,58 @@ class SettingsDialog:
             row=8, column=0, columnspan=2, sticky="w", padx=(8, 8), pady=(8, 4)
         )
 
+    def _build_tab_mirror(self, nb):
+        """Optional dual-send mirror: a second heartbeat + event destination."""
+        f = self._tab_frame(nb, "Mirror")
+
+        intro = (
+            "Optional dual-send: post each heartbeat and event to a SECOND\n"
+            "(\"mirror\") server in addition to the primary. Best-effort — the\n"
+            "mirror can never delay or fail the primary. Leave blank to disable."
+        )
+        tk.Label(f, text=intro, justify="left", fg="#1a5276", wraplength=420).grid(
+            row=0, column=0, columnspan=2, sticky="w", padx=(8, 8), pady=(6, 8)
+        )
+
+        _add_label_entry(f, 1, "Mirror Terra-View URL", self.var_mirror_api_url)
+        _add_label_entry(f, 2, "Mirror SFM URL", self.var_mirror_sfm_url)
+
+        tk.Label(f, text="Mirror State File", anchor="w").grid(
+            row=3, column=0, sticky="w", padx=(8, 4), pady=4
+        )
+        state_frame = tk.Frame(f)
+        state_frame.grid(row=3, column=1, sticky="ew", padx=(0, 8), pady=4)
+        state_frame.columnconfigure(0, weight=1)
+        ttk.Entry(state_frame, textvariable=self.var_mirror_sfm_state_file, width=32).grid(
+            row=0, column=0, sticky="ew"
+        )
+
+        def _browse_mirror_state():
+            path = filedialog.asksaveasfilename(
+                title="Mirror forward-state file",
+                defaultextension=".json",
+                filetypes=[("JSON", "*.json"), ("All Files", "*.*")],
+                initialfile="sfm_forwarded_mirror.json",
+            )
+            if path:
+                self.var_mirror_sfm_state_file.set(path)
+
+        ttk.Button(state_frame, text="Browse...", width=10,
+                   command=_browse_mirror_state).grid(row=0, column=1, padx=(4, 0))
+
+        hint_text = (
+            "Mirror Terra-View URL → base like http://10.0.0.x:8001 (heartbeats).\n"
+            "Mirror SFM URL → base like http://10.0.0.x:8200 (events).\n\n"
+            "The event mirror keeps its OWN state file, so nothing is lost while\n"
+            "the mirror is down — pending events are delivered once it returns.\n"
+            "A quick reachability check skips the mirror cleanly when unreachable,\n"
+            "so the primary is never slowed.\n"
+            "State File blank → defaults to <log dir>/sfm_forwarded_mirror.json."
+        )
+        tk.Label(f, text=hint_text, justify="left", fg="#555555", wraplength=420).grid(
+            row=4, column=0, columnspan=2, sticky="w", padx=(8, 8), pady=(8, 4)
+        )
+
     def _test_sfm_connection(self):
         """GET <sfm_url>/health and show the result."""
         import urllib.request
@@ -673,6 +739,12 @@ class SettingsDialog:
         else:
             api_url = api_url.rstrip("/") + "/api/series3/heartbeat"
 
+        # Mirror (dual-send) — best-effort second destination.
+        mirror_api_url = self.var_mirror_api_url.get().strip()
+        if mirror_api_url:
+            mirror_api_url = mirror_api_url.rstrip("/") + "/api/series3/heartbeat"
+        mirror_sfm_url = self.var_mirror_sfm_url.get().strip().rstrip("/")
+
         values = {
             "API_ENABLED":          "true" if self.var_api_enabled.get() else "false",
             "API_URL":              api_url,
@@ -698,6 +770,10 @@ class SettingsDialog:
             "SFM_HTTP_TIMEOUT":                 str(int_values["SFM HTTP Timeout"]),
             "SFM_STATE_FILE":                   self.var_sfm_state_file.get().strip(),
             "SFM_MAX_FORWARDS_PER_PASS":        str(int_values["SFM Max Events Per Pass"]),
+
+            "MIRROR_API_URL":        mirror_api_url,
+            "MIRROR_SFM_URL":         mirror_sfm_url,
+            "MIRROR_SFM_STATE_FILE":  self.var_mirror_sfm_state_file.get().strip(),
         }
 
         try:

test_event_forwarder.py

@@ -714,5 +714,127 @@ class TestForwardEventPair(unittest.TestCase):
             self.assertIn("serial=BE11529", req["path"])
 
 
+# ── Mirror (dual-send) ───────────────────────────────────────────────────────
+
+
+def _mk_mirror_event(watch_dir, name="M529LK44.AB0", age_seconds=200, content=b"binary"):
+    """Create a Blastware-shaped event binary with a controlled mtime."""
+    p = Path(watch_dir) / name
+    p.write_bytes(content)
+    target = time.time() - age_seconds
+    os.utime(str(p), (target, target))
+    return p
+
+
+class TestMirrorReachable(unittest.TestCase):
+
+    def test_empty_url_is_unreachable(self):
+        self.assertFalse(ef.mirror_reachable("", timeout=1.0))
+
+    def test_dead_port_is_unreachable_and_fast(self):
+        t0 = time.time()
+        self.assertFalse(ef.mirror_reachable("http://127.0.0.1:1", timeout=2.0))
+        self.assertLess(time.time() - t0, 2.5)
+
+    def test_any_http_response_counts_as_reachable(self):
+        server, base = _start_fake_server()
+        try:
+            self.assertTrue(ef.mirror_reachable(base, timeout=2.0))
+        finally:
+            server.shutdown()
+            server.server_close()
+
+
+class TestMirrorForwardPass(unittest.TestCase):
+    """The dual-send entry point: best-effort, isolated, own state."""
+
+    def setUp(self):
+        _FakeImportHandler.received = []
+        self.server, self.base_url = _start_fake_server()
+
+    def tearDown(self):
+        self.server.shutdown()
+        self.server.server_close()
+
+    def test_empty_mirror_url_is_noop(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            _mk_mirror_event(tmp)
+            mstate = ef.ForwardState(os.path.join(tmp, "mirror.json"))
+            self.assertIsNone(ef.mirror_forward_pass(tmp, "", mstate, max_age_days=30))
+            self.assertEqual(len(_FakeImportHandler.received), 0)
+
+    def test_skips_when_unreachable_without_posting(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            _mk_mirror_event(tmp)
+            mstate = ef.ForwardState(os.path.join(tmp, "mirror.json"))
+            result = ef.mirror_forward_pass(
+                tmp, self.base_url, mstate,
+                reachable_fn=lambda url, timeout=3.0: False,
+                max_age_days=30, quiescence_seconds=5,
+                missing_report_grace_seconds=60, timeout=5.0,
+            )
+            self.assertIsNone(result)
+            self.assertEqual(mstate.count(), 0)
+            self.assertEqual(len(_FakeImportHandler.received), 0)
+
+    def test_forwards_when_reachable_using_its_own_state(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            _mk_mirror_event(tmp)
+            mstate = ef.ForwardState(os.path.join(tmp, "mirror.json"))
+            counts = ef.mirror_forward_pass(
+                tmp, self.base_url, mstate,
+                reachable_fn=lambda url, timeout=3.0: True,
+                max_age_days=30, quiescence_seconds=5,
+                missing_report_grace_seconds=60, timeout=5.0,
+            )
+            self.assertIsNotNone(counts)
+            self.assertEqual(counts["forwarded"], 1)
+            self.assertEqual(mstate.count(), 1)
+            self.assertEqual(len(_FakeImportHandler.received), 1)
+
+    def test_never_raises_when_forward_blows_up(self):
+        def _boom(*a, **k):
+            raise RuntimeError("boom")
+        with tempfile.TemporaryDirectory() as tmp:
+            _mk_mirror_event(tmp)
+            mstate = ef.ForwardState(os.path.join(tmp, "mirror.json"))
+            orig = ef.forward_pending
+            ef.forward_pending = _boom
+            try:
+                result = ef.mirror_forward_pass(
+                    tmp, self.base_url, mstate,
+                    reachable_fn=lambda url, timeout=3.0: True,
+                    max_age_days=30,
+                )
+            finally:
+                ef.forward_pending = orig
+            self.assertIsNone(result)
+
+    def test_down_mirror_leaves_primary_state_untouched(self):
+        with tempfile.TemporaryDirectory() as tmp:
+            _mk_mirror_event(tmp)
+
+            primary_state = ef.ForwardState(os.path.join(tmp, "primary.json"))
+            pcounts = ef.forward_pending(
+                tmp, self.base_url, primary_state,
+                max_age_days=30, quiescence_seconds=5,
+                missing_report_grace_seconds=60, timeout=5.0,
+            )
+            self.assertEqual(pcounts["forwarded"], 1)
+            primary_snapshot = primary_state.count()
+
+            mstate = ef.ForwardState(os.path.join(tmp, "mirror.json"))
+            t0 = time.time()
+            result = ef.mirror_forward_pass(
+                tmp, "http://127.0.0.1:1", mstate,   # nothing listens
+                max_age_days=30, quiescence_seconds=5,
+                missing_report_grace_seconds=60, timeout=5.0,
+            )
+            self.assertIsNone(result)
+            self.assertLess(time.time() - t0, 3.5)
+            self.assertEqual(mstate.count(), 0)
+            self.assertEqual(primary_state.count(), primary_snapshot)
+
+
 if __name__ == "__main__":
     unittest.main()

test_scan_latest.py

@@ -0,0 +1,62 @@
+"""
+Tests for scan_latest's handling of unsniffable .MLG files.
+
+Regression guard: a .MLG file the watcher can't extract a unit ID from must
+be cached (as "no id") so it isn't re-sniffed and re-logged on every scan.
+Previously the unsniffable result was never cached, so the
+`[unsniffable-recent]` warning was re-emitted every scan cycle — ~200 files
+x 288 scans/day flooded the log with hundreds of thousands of duplicate lines.
+"""
+import os
+import time
+import tempfile
+import unittest
+
+import series3_watcher
+
+
+class UnsniffableCaching(unittest.TestCase):
+    def test_unsniffable_recent_file_logged_once_across_scans(self):
+        with tempfile.TemporaryDirectory() as d:
+            # A .MLG file with no BE####/BA#### pattern → sniff returns None.
+            path = os.path.join(d, "junk001.MLG")
+            with open(path, "wb") as f:
+                f.write(b"\x00" * 512)
+
+            calls = []
+            cache = {}
+            recent_cutoff = time.time() - 86400.0  # file (mtime≈now) counts as recent
+
+            for _ in range(3):
+                series3_watcher.scan_latest(
+                    watch=d,
+                    header_bytes=2048,
+                    cache=cache,
+                    recent_cutoff=recent_cutoff,
+                    max_age_days=365,
+                    logger=calls.append,
+                )
+
+            unsniff = [c for c in calls if "[unsniffable-recent]" in c]
+            self.assertEqual(
+                len(unsniff), 1,
+                "unsniffable file should log once across scans, got {}: {}".format(
+                    len(unsniff), unsniff),
+            )
+
+    def test_sniffable_file_still_returned(self):
+        # Sanity: a file WITH a unit id is still detected (happy path intact).
+        with tempfile.TemporaryDirectory() as d:
+            path = os.path.join(d, "evt001.MLG")
+            with open(path, "wb") as f:
+                f.write(b"hdr\x00BE12599\x00hdr")  # contains a sniffable unit id
+            cache = {}
+            result = series3_watcher.scan_latest(
+                watch=d, header_bytes=2048, cache=cache,
+                recent_cutoff=time.time() - 86400.0, max_age_days=365, logger=None,
+            )
+            self.assertIn("BE12599", result)
+
+
+if __name__ == "__main__":
+    unittest.main()

test_series3_tray.py

@@ -0,0 +1,78 @@
+"""
+Tests for series3_tray's thread lifecycle.
+
+Regression guard: saving the Settings dialog restarts the watcher thread via
+``_restart_watcher``, which sets and replaces the shared ``stop_event``. The
+auto-updater must key off its own lifetime event (``_app_stop``) so a watcher
+restart never kills it; it should only stop when the app actually exits.
+"""
+import sys
+import time
+import threading
+import unittest
+from unittest import mock
+
+# Series3's tray pulls in Windows-only GUI deps at import time; stub them.
+for _name in ("pystray", "PIL", "PIL.Image", "PIL.ImageDraw"):
+    sys.modules.setdefault(_name, mock.MagicMock())
+
+import series3_tray  # noqa: E402
+
+
+class TrayThreadLifecycle(unittest.TestCase):
+    def setUp(self):
+        # No-op watcher loop so restarting the watcher doesn't spin up the real
+        # Blastware scanner / network heartbeat.
+        patcher = mock.patch.object(
+            series3_tray.watcher, "run_watcher", lambda state, stop_event: None
+        )
+        patcher.start()
+        self.addCleanup(patcher.stop)
+
+    def test_app_stop_is_separate_lifetime_event(self):
+        app = series3_tray.WatcherTray()
+        self.assertIsInstance(app._app_stop, threading.Event)
+        self.assertIsNot(app._app_stop, app.stop_event)
+        self.assertFalse(app._app_stop.is_set())
+
+    def test_exit_signals_app_stop(self):
+        app = series3_tray.WatcherTray()
+        icon = mock.MagicMock()
+        app._exit(icon, None)
+        self.assertTrue(app._app_stop.is_set())
+        icon.stop.assert_called_once()
+
+    def test_restart_watcher_leaves_watcher_stop_unset(self):
+        app = series3_tray.WatcherTray()
+        app._restart_watcher()
+        self.assertFalse(app.stop_event.is_set())
+
+    def test_updater_survives_watcher_restart_then_stops_on_exit(self):
+        app = series3_tray.WatcherTray()
+        app._icon = None
+        with mock.patch.object(series3_tray, "check_for_update", return_value=(None, None)), \
+                mock.patch.object(app, "_tray_status", return_value="ok"):
+            t = threading.Thread(
+                target=app._icon_updater, daemon=True, name="test-updater"
+            )
+            t.start()
+            time.sleep(0.1)
+            self.assertTrue(t.is_alive(), "updater failed to start")
+
+            # A settings save restarts the watcher — it must NOT kill the updater.
+            app._restart_watcher()
+            time.sleep(0.2)
+            self.assertTrue(
+                t.is_alive(), "updater thread died on watcher restart (the bug)"
+            )
+
+            # Real app exit DOES stop the updater.
+            app._app_stop.set()
+            t.join(timeout=2)
+            self.assertFalse(
+                t.is_alive(), "updater thread did not stop on app exit"
+            )
+
+
+if __name__ == "__main__":
+    unittest.main()