project-lyra/docs/superpowers/specs/2026-07-04-mi50-summary-cap-fallback-design.md

Bounded MI50 summaries with cloud fallback

Date: 2026-07-04 Branch: fix/mi50-summary-cap-fallback

Problem

The dream cycle's summarize_all runs against the MI50 (backend=mi50). Each summary call to llm.complete() on the mi50 path hands the OpenAI SDK no max_tokens and no timeout, so it inherits SDK defaults — a 600s request timeout with 2 internal retries, i.e. ~30 minutes per call before it raises "Request timed out." On top of that, summary.py had its own 4-attempt retry loop, so a single unsummarizable session could keep the GPU pegged for hours.

Observed live (2026-07-04, ~01:00–02:00): the dream service looped summarize-all … backend=mi50 since 23:02, every call timing out, nothing written to the DB since 00:56, the MI50 generating 7,000–8,000-token completions (a gist needs <200), all four llama.cpp slots busy, fans blaring.

This is not context overflow — the server log showed context shift = 0, truncated = 1 = 0. The prompts are small (~900–1,500 tokens). The failure is purely unbounded generation length on a slow backend → timeout → retry loop.

Goals

• Keep the MI50 as the primary summary backend (Brian's preference, gaming-safe).
• Cap each summary generation so it finishes fast and can never run away.
• Make a stuck MI50 call fail fast and fall back to cloud, instead of looping all night.
• Change nothing about live chat, reflect, or think.

Design

1. lyra/llm.py — complete() gains two optional params

def complete(messages, backend="local", model=None,
             max_tokens: int | None = None, timeout: float | None = None) -> str

• max_tokens (when set): passed to the create() call — max_tokens= for the cloud/mi50 OpenAI paths, options={"num_predict": …} for the local Ollama path.
• timeout (when set): for the cloud/mi50 OpenAI clients, build the client with timeout=<t>, max_retries=0 so the call bails quickly and we own the retry policy (eliminates the hidden 3×600s). For local, use it as the httpx timeout.
• Both default to None → behavior identical to today for every other caller (chat_call, reflect, think, etc.). Backward compatible.

2. lyra/summary.py — capped, fast-fail, cloud fallback

Constants:

SUMMARY_MAX_TOKENS = 768   # ~3× the longest real gist; bounds gen to ~1 min on MI50
MI50_ATTEMPTS      = 2      # attempts on the primary backend before falling back
SUMMARY_TIMEOUT    = 150    # seconds/call — capped 768-tok gist finishes in ~60-90s

Rewrite _summarize_text(text, backend):

1. Try backend up to MI50_ATTEMPTS times, each: llm.complete(messages, backend=backend, max_tokens=SUMMARY_MAX_TOKENS, timeout=SUMMARY_TIMEOUT), with a short backoff between attempts.
2. If all primary attempts fail and backend != "cloud" and an OpenAI key is configured → one final cloud attempt (same cap/timeout), logged as summary fell back to cloud.
3. If cloud also fails or is unavailable → raise.

Fallback is per-_summarize_text call (i.e. per chunk), so the long-session chunk/merge path in _summarize_transcript is unaffected. The old _RETRIES = 4 loop is replaced by this structure.

3. Degenerate-output guard (added 2026-07-04)

A wedged local backend — observed live when the MI50 overheated to 99°C junction — returns a single character repeated ("?????") as a successful 200 response, which neither the timeout nor the exception path catches. So each _call() validates its output: _looks_degenerate(text) flags output (≥24 non-space chars) whose most-common non-whitespace character exceeds 50% of the text, and raises DegenerateOutput — which the retry/fallback loop treats exactly like any other failure (retry the primary, then fall back to cloud). Real gists are diverse prose (top char well under 20%), so the threshold won't false-positive; short outputs are exempt. If cloud also returns junk, it raises and stops — no infinite loop.

Testing

Unit (pytest, tests/test_summary_fallback.py), monkeypatching llm.complete:

• Fallback fires: mi50 raises on every call → after MI50_ATTEMPTS the cloud attempt runs and its result is returned; a fell back to cloud log is emitted.
• No fallback when primary is already cloud (retries, then raises).
• No fallback when no OpenAI key (raises after primary attempts).
• max_tokens and timeout are threaded into every complete() call.

Plus a light llm.complete test that max_tokens/timeout reach the client kwargs (monkeypatch the OpenAI client).

Verification (real)

After deploy (systemctl --user restart lyra-dream lyra-web — editable install): watch journalctl --user -fu lyra-dream through a summarize cycle and confirm llm done … out≈768 completing in ~1 min, an actual summarized session row written (DB summary count rises), and no "Request timed out". Confirm the llama.cpp slot shows bounded n_decoded ≈ 768.

Out of scope (YAGNI)

• The degenerate-output guard (§3) targets the observed failure — one char repeated. It does not try to detect subtler degeneration (repeated phrases, off-topic rambling); that's fuzzy and unmotivated until seen.
• No change to chat_call/reflect/think or config.summary_backend.
• No change to profile/era/narrative rebuild calls (separate, and not the loop culprit); can adopt the same max_tokens later if they show the same rambling.