project-lyra/docs/superpowers/specs/2026-07-04-mi50-runaway-guards-design.md

MI50 runaway guards: dream-cycle budget + host watchdog

Date: 2026-07-04 Branch: fix/mi50-summary-cap-fallback Follows: the summary cap/fallback fix (same branch). This adds general "never run unchecked again" protection on top of the specific summary fix.

Problem

The summary fix stops the known runaway (uncapped summaries). But the operator wants a guarantee that no cause — known or future — can peg the MI50 for hours unattended. Two independent layers, per operator decision:

• C (in-app, primary): Lyra's own dream cycle bounds itself.
• A (host, fallback): a watchdog on the always-on Proxmox host kills the backend if the GPU runs too long or too hot, regardless of cause. Trips only after 1 hr of continuous busy so legitimate manual workloads (~40 min) run untouched.

Design

C — dream-cycle time budget (lyra/)

1. Per-call ceiling. llm.complete() currently sets a timeout only when one is passed; otherwise it inherits the OpenAI SDK default (600s × 2 retries ≈ 30 min). Change the default: when no timeout is given, the cloud/mi50 paths use 300s + max_retries=0. This bounds every consolidation/introspection call (profile, era, narrative, reflect, think) — not just summaries — with one change. Live chat uses chat_call*, a different path, unaffected.

2. Cycle deadline. dream_cycle() sets deadline = now + DREAM_CYCLE_BUDGET (20 min) before its heavy stages and checks it between them (continuity → coherence → curiosity). Once past the deadline, remaining stages are skipped, the cycle logs dream cycle over budget — stopped early, appends a stopped early (over budget) action, and notify.push() pings Brian. A hung single call can't blow past ~300s (step 1), so the between-stage checks keep a pass bounded to roughly the budget.

A — host watchdog (deploy/mi50-watchdog/)

A bash script + systemd timer installed on the Proxmox host (10.0.0.4), which has rocm-smi + docker and is always on. Runs every 2 min:

• Duration rule: track continuous busy time in a state file (GPU use % > 0). If busy ≥ 3600s straight → docker stop lyra-brain. Idle clears the timer, so a 40-min job never trips it.
• Temp rule (independent): if junction ≥ 97°C for 3 consecutive checks (~6 min) → stop. A normal-temp long workload won't trip this; only a genuinely overheating one.
• On either trip: stop the container, clear state, logger a line, and POST to the ntfy topic so Brian is told. Thresholds are unit-file env vars (tunable).

Files: mi50-watchdog.sh, mi50-watchdog.service, mi50-watchdog.timer, README.md (install: copy to host, set ntfy env, systemctl enable --now).

Testing

• C step 1: llm.complete() with no timeout builds the client with timeout=300, max_retries=0 and still no max_tokens (update existing test_llm_bounds default test).
• C step 2: a dream pass that goes over budget skips later stages, records the stopped early action, and calls notify.push (stub the clock/operations in test_dream).
• A: decision logic dry-run locally against sample rocm-smi output (busy / idle / hot). Cannot be live-verified now (card is off, operator away) — install
  • real trip test deferred to when the card is back.

Verification

C is repo code and ships live the moment lyra-dream restarts. A is staged in the repo for host install; verify on the host when the card returns (force a long/hot condition or lower thresholds temporarily and confirm it stops the container + pings).

Out of scope (YAGNI)

• No power cap (option B) — deferred; C+A cover the "unchecked" concern and the electricity cost of one event is trivial (~$0.10).
• No change to live chat, chat_call*, or config.summary_backend.