feat: host-side MI50 runaway watchdog (guard A, staged for install)

Independent Proxmox-host backstop to the in-app dream budget: a systemd timer
runs every ~2 min and stops lyra-brain if the MI50 is busy >=1hr continuously OR
junction >=97C for ~6 min, then pings Brian via ntfy. Trips on duration only
after a full hour so a legit ~40-min manual workload runs untouched. GPU temp/use
read from host rocm-smi; stop via 'pct exec 202 -- docker stop'.

Parsing + duration/temp decision logic dry-run-verified locally against real
rocm-smi output format (4 scenarios). NOT yet installed/live-verified — card is
off and Brian's away; install + trip-test per deploy/mi50-watchdog/README.md when
it's back.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015yrEb5qpPGv2FjyxrB7LLk

deploy/mi50-watchdog/README.md

@@ -0,0 +1,54 @@
+# MI50 runaway watchdog (fallback layer "A")
+
+Independent host-side backstop to Lyra's in-app dream-cycle budget (layer "C",
+`lyra/dream.py`). Stops the llama.cpp backend if the MI50 is busy too long or too
+hot, and pings Brian. See
+`docs/superpowers/specs/2026-07-04-mi50-runaway-guards-design.md`.
+
+## What it does
+
+Runs on the **Proxmox host** (`10.0.0.4`) via a systemd timer, every ~2 min:
+
+- **Duration:** if the GPU is busy (`rocm-smi` use% > 0) for **3600s continuously**,
+  it stops the container. Any idle read resets the streak, so a legitimate ~40-min
+  manual workload never trips it.
+- **Temperature:** if junction ≥ **97°C** for **3 consecutive checks (~6 min)**, it
+  stops the container — independent of duration.
+- On either trip: `pct exec 202 -- docker stop lyra-brain`, clear state, `logger` a
+  line, and POST to your ntfy topic.
+
+All thresholds are `Environment=` overrides in the `.service`.
+
+## Install (on the Proxmox host, as root)
+
+```sh
+# copy the three files up (from the repo, on lyra-cortex):
+scp -i ~/.ssh/id_lyra_proxmox deploy/mi50-watchdog/mi50-watchdog.sh \
+    root@10.0.0.4:/usr/local/sbin/mi50-watchdog.sh
+scp -i ~/.ssh/id_lyra_proxmox deploy/mi50-watchdog/mi50-watchdog.{service,timer} \
+    root@10.0.0.4:/etc/systemd/system/
+
+# on the host:
+chmod +x /usr/local/sbin/mi50-watchdog.sh
+# set your ntfy topic (same one Lyra uses) in the service:
+sed -i 's/CHANGE_ME/YOUR_NTFY_TOPIC/' /etc/systemd/system/mi50-watchdog.service
+systemctl daemon-reload
+systemctl enable --now mi50-watchdog.timer
+```
+
+## Verify (when the card is back and healthy)
+
+```sh
+# dry run once, watch what it decides:
+NTFY_URL= /usr/local/sbin/mi50-watchdog.sh; echo "exit $?"
+journalctl -t mi50-watchdog -n 20 --no-pager
+
+# force a trip test with tiny thresholds (won't touch a healthy idle card unless busy):
+MAX_BUSY_SEC=60 TEMP_KILL_C=40 TEMP_KILL_STREAK=1 /usr/local/sbin/mi50-watchdog.sh
+# confirm it stopped lyra-brain + sent the ntfy, then restart the container.
+
+systemctl list-timers mi50-watchdog.timer   # confirm it's scheduled
+```
+
+**Not yet installed / live-verified** — staged here on 2026-07-04 while the card is
+off and Brian is away. Install + trip-test when the MI50 is back.

deploy/mi50-watchdog/mi50-watchdog.service

@@ -0,0 +1,16 @@
+[Unit]
+Description=MI50 runaway watchdog (stop the llama.cpp backend if the GPU is busy too long or too hot)
+After=network-online.target
+
+[Service]
+Type=oneshot
+# Fill in your ntfy topic so it can ping Brian when it trips (leave URL empty to log only).
+Environment=NTFY_URL=https://ntfy.sh
+Environment=NTFY_TOPIC=CHANGE_ME
+# Optional overrides (defaults shown):
+# Environment=MAX_BUSY_SEC=3600
+# Environment=TEMP_KILL_C=97
+# Environment=TEMP_KILL_STREAK=3
+# Environment=CTID=202
+# Environment=CONTAINER=lyra-brain
+ExecStart=/usr/local/sbin/mi50-watchdog.sh

deploy/mi50-watchdog/mi50-watchdog.sh

@@ -0,0 +1,82 @@
+#!/usr/bin/env bash
+# MI50 runaway watchdog — fallback layer "A".
+#
+# Runs on the Proxmox HOST (10.0.0.4) via a systemd timer (every ~2 min). It is the
+# independent backstop to Lyra's own in-app dream-cycle budget ("C", in lyra/dream.py):
+# if the MI50 is busy too LONG or runs too HOT, it stops the llama.cpp backend and
+# pings Brian — regardless of what caused it. Trips on duration only after a full hour
+# of *continuous* busy, so a legitimate ~40-min manual workload runs untouched.
+#
+# The GPU lives on the host; the llama.cpp container ("lyra-brain") runs inside LXC
+# CT202. So temp/use come from host rocm-smi, and the stop goes via `pct exec`.
+#
+# See docs/superpowers/specs/2026-07-04-mi50-runaway-guards-design.md
+set -uo pipefail
+
+# --- tunables (override in the .service via Environment=) ---
+CTID="${CTID:-202}"                          # LXC holding the docker container
+CONTAINER="${CONTAINER:-lyra-brain}"
+MAX_BUSY_SEC="${MAX_BUSY_SEC:-3600}"         # 1 hr continuous busy -> stop
+TEMP_KILL_C="${TEMP_KILL_C:-97}"             # junction >= this ...
+TEMP_KILL_STREAK="${TEMP_KILL_STREAK:-3}"    # ... for this many consecutive checks (~6 min)
+NTFY_URL="${NTFY_URL:-}"                      # e.g. https://ntfy.sh  (empty => log only)
+NTFY_TOPIC="${NTFY_TOPIC:-}"
+BUSY_STATE="${BUSY_STATE:-/run/mi50-watchdog.busy_since}"
+HOT_STATE="${HOT_STATE:-/run/mi50-watchdog.hot_streak}"
+
+now="$(date +%s)"
+
+alert() {  # $1 title, $2 message
+  logger -t mi50-watchdog "$2"
+  if [[ -n "$NTFY_URL" && -n "$NTFY_TOPIC" ]]; then
+    curl -s -m 8 -H "Title: $1" -H "Priority: urgent" -H "Tags: warning" \
+      -d "$2" "$NTFY_URL/$NTFY_TOPIC" >/dev/null 2>&1 || true
+  fi
+}
+
+stop_backend() {  # $1 reason
+  pct exec "$CTID" -- docker stop "$CONTAINER" >/dev/null 2>&1 || true
+  rm -f "$BUSY_STATE" "$HOT_STATE"
+  alert "MI50 watchdog stopped the card" "$1"
+}
+
+# Nothing to guard if the backend isn't even running.
+running="$(pct exec "$CTID" -- docker inspect -f '{{.State.Running}}' "$CONTAINER" 2>/dev/null || echo false)"
+if [[ "$running" != "true" ]]; then
+  rm -f "$BUSY_STATE" "$HOT_STATE"
+  exit 0
+fi
+
+use="$(rocm-smi --showuse  2>/dev/null | awk -F: '/GPU use \(%\)/  {gsub(/[^0-9]/,   "", $NF); print $NF; exit}')"
+junction="$(rocm-smi --showtemp 2>/dev/null | awk -F: '/junction/       {gsub(/[^0-9.]/, "", $NF); print $NF; exit}')"
+
+# --- duration rule: accumulate continuous busy time in a state file ---
+busy=0
+[[ "${use:-}" =~ ^[0-9]+$ ]] && (( use > 0 )) && busy=1
+if (( busy )); then
+  [[ -f "$BUSY_STATE" ]] || echo "$now" > "$BUSY_STATE"
+  since="$(cat "$BUSY_STATE" 2>/dev/null || echo "$now")"
+  elapsed=$(( now - since ))
+  if (( elapsed >= MAX_BUSY_SEC )); then
+    stop_backend "MI50 busy ${elapsed}s continuously (>= ${MAX_BUSY_SEC}s) — stopped ${CONTAINER}."
+    exit 0
+  fi
+else
+  rm -f "$BUSY_STATE"   # idle breaks the streak
+fi
+
+# --- temperature rule: independent of duration ---
+if [[ "${junction:-}" =~ ^[0-9.]+$ ]]; then
+  jint="${junction%.*}"
+  if (( jint >= TEMP_KILL_C )); then
+    streak=$(( $(cat "$HOT_STATE" 2>/dev/null || echo 0) + 1 ))
+    echo "$streak" > "$HOT_STATE"
+    if (( streak >= TEMP_KILL_STREAK )); then
+      stop_backend "MI50 junction ${jint}C >= ${TEMP_KILL_C}C for ${streak} checks — stopped ${CONTAINER}."
+      exit 0
+    fi
+  else
+    rm -f "$HOT_STATE"   # cooled off, reset the streak
+  fi
+fi
+exit 0

deploy/mi50-watchdog/mi50-watchdog.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Run the MI50 runaway watchdog every 2 minutes
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=2min
+AccuracySec=15s
+
+[Install]
+WantedBy=timers.target

docs/superpowers/specs/2026-07-04-mi50-runaway-guards-design.md

@@ -0,0 +1,79 @@
+# MI50 runaway guards: dream-cycle budget + host watchdog
+
+**Date:** 2026-07-04
+**Branch:** `fix/mi50-summary-cap-fallback`
+**Follows:** the summary cap/fallback fix (same branch). This adds general
+"never run unchecked again" protection on top of the specific summary fix.
+
+## Problem
+
+The summary fix stops the *known* runaway (uncapped summaries). But the operator
+wants a guarantee that *no* cause — known or future — can peg the MI50 for hours
+unattended. Two independent layers, per operator decision:
+
+- **C (in-app, primary):** Lyra's own dream cycle bounds itself.
+- **A (host, fallback):** a watchdog on the always-on Proxmox host kills the
+  backend if the GPU runs too long or too hot, regardless of cause. Trips only
+  after **1 hr** of continuous busy so legitimate manual workloads (~40 min) run
+  untouched.
+
+## Design
+
+### C — dream-cycle time budget (`lyra/`)
+
+1. **Per-call ceiling.** `llm.complete()` currently sets a timeout only when one
+   is passed; otherwise it inherits the OpenAI SDK default (600s × 2 retries ≈
+   30 min). Change the default: when no `timeout` is given, the cloud/mi50 paths
+   use **300s + `max_retries=0`**. This bounds *every* consolidation/introspection
+   call (`profile`, `era`, `narrative`, `reflect`, `think`) — not just summaries —
+   with one change. Live chat uses `chat_call*`, a different path, unaffected.
+
+2. **Cycle deadline.** `dream_cycle()` sets `deadline = now + DREAM_CYCLE_BUDGET`
+   (**20 min**) before its heavy stages and checks it between them (continuity →
+   coherence → curiosity). Once past the deadline, remaining stages are skipped,
+   the cycle logs `dream cycle over budget — stopped early`, appends a
+   `stopped early (over budget)` action, and `notify.push()` pings Brian. A hung
+   single call can't blow past ~300s (step 1), so the between-stage checks keep a
+   pass bounded to roughly the budget.
+
+### A — host watchdog (`deploy/mi50-watchdog/`)
+
+A bash script + systemd timer installed on the Proxmox host (`10.0.0.4`), which
+has `rocm-smi` + `docker` and is always on. Runs every 2 min:
+
+- **Duration rule:** track continuous busy time in a state file (`GPU use % > 0`).
+  If busy ≥ **3600s** straight → `docker stop lyra-brain`. Idle clears the timer,
+  so a 40-min job never trips it.
+- **Temp rule (independent):** if junction ≥ **97°C** for **3 consecutive checks
+  (~6 min)** → stop. A normal-temp long workload won't trip this; only a genuinely
+  overheating one.
+- On either trip: stop the container, clear state, `logger` a line, and POST to
+  the ntfy topic so Brian is told. Thresholds are unit-file env vars (tunable).
+
+Files: `mi50-watchdog.sh`, `mi50-watchdog.service`, `mi50-watchdog.timer`,
+`README.md` (install: copy to host, set ntfy env, `systemctl enable --now`).
+
+## Testing
+
+- **C step 1:** `llm.complete()` with no timeout builds the client with
+  `timeout=300, max_retries=0` and still no `max_tokens` (update existing
+  `test_llm_bounds` default test).
+- **C step 2:** a dream pass that goes over budget skips later stages, records the
+  `stopped early` action, and calls `notify.push` (stub the clock/operations in
+  `test_dream`).
+- **A:** decision logic dry-run locally against sample `rocm-smi` output (busy /
+  idle / hot). Cannot be live-verified now (card is off, operator away) — install
+  + real trip test deferred to when the card is back.
+
+## Verification
+
+C is repo code and ships live the moment `lyra-dream` restarts. A is staged in the
+repo for host install; verify on the host when the card returns (force a long/hot
+condition or lower thresholds temporarily and confirm it stops the container +
+pings).
+
+## Out of scope (YAGNI)
+
+- No power cap (option B) — deferred; C+A cover the "unchecked" concern and the
+  electricity cost of one event is trivial (~$0.10).
+- No change to live chat, `chat_call*`, or `config.summary_backend`.

lyra/dream.py

@@ -26,7 +26,8 @@ import time
 from datetime import datetime, timezone
 
 from lyra import (
-    config, era, feeds, logbus, memory, narrative, poker, profile, self_state, summary, thoughts,
+    config, era, feeds, logbus, memory, narrative, notify, poker, profile, self_state,
+    summary, thoughts,
 )
 from lyra.llm import Backend
 from lyra.summary import SUMMARIZE_AFTER
@@ -34,6 +35,17 @@ from lyra.summary import SUMMARIZE_AFTER
 # A drive at/above this has built up enough to act on.
 THRESHOLD = 0.6
 
+# Wall-clock ceiling for a single pass. Every consolidation/introspection call is
+# individually bounded (llm.complete's default timeout), but this caps the whole
+# pass: once exceeded, remaining stages are skipped and Brian is pinged — so a slow
+# or wedged MI50 can never grind for hours unattended. The host watchdog (A) is the
+# independent fallback if this ever fails to fire.
+DREAM_CYCLE_BUDGET_SEC = 20 * 60
+
+
+def _over_budget(deadline: float) -> bool:
+    return time.monotonic() > deadline
+
 # How much backlog saturates each pressure (the drive reaches ~1.0 at this level).
 CONTINUITY_FULL = 4   # ripe (summary-needing) sessions
 COHERENCE_FULL = 10   # gists not yet folded into the profile
@@ -96,9 +108,12 @@ def dream_cycle(backend: Backend | None = None, force: bool = False) -> dict:
         logbus.log("error", "daily digest failed", error=str(exc)[:160])
 
     actions: list[str] = []
+    # Cap the whole pass: skip any stage we reach after the deadline (checked
+    # between stages; each call is already individually bounded).
+    deadline = time.monotonic() + DREAM_CYCLE_BUDGET_SEC
 
     # --- continuity: compact raw sessions into gists ---
-    if force or drives["continuity"] >= THRESHOLD:
+    if (force or drives["continuity"] >= THRESHOLD) and not _over_budget(deadline):
         report = summary.summarize_all(backend=backend)
         actions.append(f"consolidated {report['summarized']} sessions")
         drives["continuity"] = 0.0
@@ -108,7 +123,7 @@ def dream_cycle(backend: Backend | None = None, force: bool = False) -> dict:
         drives["coherence"] = _clamp(profile_lag / COHERENCE_FULL)
 
     # --- coherence: fold gists up into profile / eras / narrative ---
-    if force or drives["coherence"] >= THRESHOLD:
+    if (force or drives["coherence"] >= THRESHOLD) and not _over_budget(deadline):
         profile.rebuild_profile(backend=backend)
         era.rebuild_eras(backend=backend)
         narrative.rebuild_narrative(backend=backend)
@@ -124,7 +139,7 @@ def dream_cycle(backend: Backend | None = None, force: bool = False) -> dict:
             logbus.log("error", "villain merge scan failed", error=str(exc)[:200])
 
     # --- curiosity: reflect and evolve the self, then advance the thought loop ---
-    if force or drives["curiosity"] >= THRESHOLD:
+    if (force or drives["curiosity"] >= THRESHOLD) and not _over_budget(deadline):
         # reflect()/think() self-resolve to the *introspection* backend (her voice),
         # which can differ from the consolidation backend above — don't pass `backend`.
         self_state.reflect(source="dream")  # writes state + journal itself
@@ -139,6 +154,17 @@ def dream_cycle(backend: Backend | None = None, force: bool = False) -> dict:
             logbus.log("error", "thought loop failed", error=str(exc)[:200])
         drives["curiosity"] = CURIOSITY_FLOOR
 
+    if _over_budget(deadline):
+        logbus.log("error", "dream cycle over budget — stopped early",
+                   budget_min=DREAM_CYCLE_BUDGET_SEC // 60, done=actions)
+        actions.append("stopped early (over budget)")
+        notify.push(
+            "Lyra — dream cycle over budget",
+            f"A dream pass ran past {DREAM_CYCLE_BUDGET_SEC // 60} min and stopped early. "
+            "The MI50 backend may be slow or wedged — worth a look.",
+            tags="warning",
+        )
+
     if not actions:
         actions.append("rested (nothing past threshold)")
 

lyra/llm.py

@@ -19,6 +19,11 @@ class Message(TypedDict):
 
 Backend = Literal["local", "cloud", "mi50"]
 
+# Hard ceiling on any single completion so a slow/stuck backend can't hang a call
+# for the SDK's 600s x2-retry default (~30 min). Callers pass an explicit timeout
+# to override (e.g. summary.py's tighter fast-fail).
+_DEFAULT_TIMEOUT = 300.0
+
 
 def _approx_tok(messages: list) -> int:
     """Rough prompt size (chars/4) — enough to see what's loading a backend."""
@@ -59,9 +64,11 @@ def complete(messages: list[Message], backend: Backend = "local", model: str | N
         else:
             # MI50 box runs an OpenAI-compatible llama.cpp server; key is unused.
             client_kwargs = {"api_key": "not-needed", "base_url": cfg.mi50_base_url}
-        if timeout is not None:
-            client_kwargs["timeout"] = timeout
-            client_kwargs["max_retries"] = 0  # caller owns retries (see summary.py)
+        # Always bound the request: default 300s (vs the SDK's 600s x2 retries ≈
+        # 30 min that let a stuck MI50 call hang for half an hour), and disable the
+        # SDK's own retries so the caller owns retry/fallback policy.
+        client_kwargs["timeout"] = timeout if timeout is not None else _DEFAULT_TIMEOUT
+        client_kwargs["max_retries"] = 0
         client = OpenAI(**client_kwargs)
         create_kwargs: dict = {"model": mdl, "messages": messages}
         if max_tokens is not None:

tests/test_dream.py

@@ -77,3 +77,31 @@ def test_dream_cycle_consolidates_and_persists(lyra):
     state2 = dream.dream_cycle(force=False)
     assert state2["dream"]["cycle_count"] == 2
     assert state2["drives"]["continuity"] == 0.0
+
+
+def test_dream_cycle_stops_when_over_budget(lyra, monkeypatch):
+    memory = lyra
+    from lyra import dream, notify
+
+    for k in range(7):
+        _seed(memory, f"s{k}", 4)
+
+    # Go over budget right after the first heavy stage: first check passes
+    # (summarize runs), every check after trips.
+    checks = {"n": 0}
+
+    def fake_over(deadline):
+        checks["n"] += 1
+        return checks["n"] > 1
+    monkeypatch.setattr(dream, "_over_budget", fake_over)
+
+    pings: list = []
+    monkeypatch.setattr(notify, "push",
+                        lambda title, message, **k: pings.append((title, message)) or True)
+
+    state = dream.dream_cycle(force=True)
+    acts = state["dream"]["last_actions"]
+
+    assert any("stopped early" in a for a in acts)   # bailed
+    assert not any("reflected" in a for a in acts)   # later stage skipped
+    assert pings, "expected an over-budget ntfy push"

tests/test_llm_bounds.py

@@ -55,9 +55,12 @@ def test_cloud_threads_max_tokens_and_timeout(fake_openai):
     assert fake_openai["client"]["max_retries"] == 0
 
 
-def test_defaults_omit_cap_and_keep_current_behavior(fake_openai):
-    # No cap / timeout passed -> create() gets no max_tokens, client unbounded.
+def test_default_bounds_calls_even_without_explicit_timeout(fake_openai):
+    # No cap / timeout passed -> still bounded: 300s default + no SDK retries, so
+    # no call can silently inherit the SDK's 600s x2 (~30 min). No length cap
+    # unless asked, though.
     llm.complete([{"role": "user", "content": "hi"}], backend="mi50")
 
     assert "max_tokens" not in fake_openai["create"]
-    assert "timeout" not in fake_openai["client"]
+    assert fake_openai["client"]["timeout"] == 300
+    assert fake_openai["client"]["max_retries"] == 0