docs: spec for MI50 runaway guards (dream budget + host watchdog)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015yrEb5qpPGv2FjyxrB7LLk
2026-07-04 19:05:32 +00:00
parent e631797187
commit af778ef327
1 changed files with 79 additions and 0 deletions
@@ -0,0 +1,79 @@
+# MI50 runaway guards: dream-cycle budget + host watchdog
+
+**Date:** 2026-07-04
+**Branch:** `fix/mi50-summary-cap-fallback`
+**Follows:** the summary cap/fallback fix (same branch). This adds general
+"never run unchecked again" protection on top of the specific summary fix.
+
+## Problem
+
+The summary fix stops the *known* runaway (uncapped summaries). But the operator
+wants a guarantee that *no* cause — known or future — can peg the MI50 for hours
+unattended. Two independent layers, per operator decision:
+
+- **C (in-app, primary):** Lyra's own dream cycle bounds itself.
+- **A (host, fallback):** a watchdog on the always-on Proxmox host kills the
+  backend if the GPU runs too long or too hot, regardless of cause. Trips only
+  after **1 hr** of continuous busy so legitimate manual workloads (~40 min) run
+  untouched.
+
+## Design
+
+### C — dream-cycle time budget (`lyra/`)
+
+1. **Per-call ceiling.** `llm.complete()` currently sets a timeout only when one
+   is passed; otherwise it inherits the OpenAI SDK default (600s × 2 retries ≈
+   30 min). Change the default: when no `timeout` is given, the cloud/mi50 paths
+   use **300s + `max_retries=0`**. This bounds *every* consolidation/introspection
+   call (`profile`, `era`, `narrative`, `reflect`, `think`) — not just summaries —
+   with one change. Live chat uses `chat_call*`, a different path, unaffected.
+
+2. **Cycle deadline.** `dream_cycle()` sets `deadline = now + DREAM_CYCLE_BUDGET`
+   (**20 min**) before its heavy stages and checks it between them (continuity →
+   coherence → curiosity). Once past the deadline, remaining stages are skipped,
+   the cycle logs `dream cycle over budget — stopped early`, appends a
+   `stopped early (over budget)` action, and `notify.push()` pings Brian. A hung
+   single call can't blow past ~300s (step 1), so the between-stage checks keep a
+   pass bounded to roughly the budget.
+
+### A — host watchdog (`deploy/mi50-watchdog/`)
+
+A bash script + systemd timer installed on the Proxmox host (`10.0.0.4`), which
+has `rocm-smi` + `docker` and is always on. Runs every 2 min:
+
+- **Duration rule:** track continuous busy time in a state file (`GPU use % > 0`).
+  If busy ≥ **3600s** straight → `docker stop lyra-brain`. Idle clears the timer,
+  so a 40-min job never trips it.
+- **Temp rule (independent):** if junction ≥ **97°C** for **3 consecutive checks
+  (~6 min)** → stop. A normal-temp long workload won't trip this; only a genuinely
+  overheating one.
+- On either trip: stop the container, clear state, `logger` a line, and POST to
+  the ntfy topic so Brian is told. Thresholds are unit-file env vars (tunable).
+
+Files: `mi50-watchdog.sh`, `mi50-watchdog.service`, `mi50-watchdog.timer`,
+`README.md` (install: copy to host, set ntfy env, `systemctl enable --now`).
+
+## Testing
+
+- **C step 1:** `llm.complete()` with no timeout builds the client with
+  `timeout=300, max_retries=0` and still no `max_tokens` (update existing
+  `test_llm_bounds` default test).
+- **C step 2:** a dream pass that goes over budget skips later stages, records the
+  `stopped early` action, and calls `notify.push` (stub the clock/operations in
+  `test_dream`).
+- **A:** decision logic dry-run locally against sample `rocm-smi` output (busy /
+  idle / hot). Cannot be live-verified now (card is off, operator away) — install
+  + real trip test deferred to when the card is back.
+
+## Verification
+
+C is repo code and ships live the moment `lyra-dream` restarts. A is staged in the
+repo for host install; verify on the host when the card returns (force a long/hot
+condition or lower thresholds temporarily and confirm it stops the container +
+pings).
+
+## Out of scope (YAGNI)
+
+- No power cap (option B) — deferred; C+A cover the "unchecked" concern and the
+  electricity cost of one event is trivial (~$0.10).
+- No change to live chat, `chat_call*`, or `config.summary_backend`.