serversdown
e6134cf535
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015yrEb5qpPGv2FjyxrB7LLk
101 lines
4.4 KiB
Markdown
101 lines
4.4 KiB
Markdown
# Bounded MI50 summaries with cloud fallback
|
||
|
||
**Date:** 2026-07-04
|
||
**Branch:** `fix/mi50-summary-cap-fallback`
|
||
|
||
## Problem
|
||
|
||
The dream cycle's `summarize_all` runs against the MI50 (`backend=mi50`). Each
|
||
summary call to `llm.complete()` on the `mi50` path hands the OpenAI SDK **no
|
||
`max_tokens` and no timeout**, so it inherits SDK defaults — a 600s request
|
||
timeout with 2 internal retries, i.e. **~30 minutes per call before it raises
|
||
"Request timed out."** On top of that, `summary.py` had its own 4-attempt retry
|
||
loop, so a single unsummarizable session could keep the GPU pegged for hours.
|
||
|
||
Observed live (2026-07-04, ~01:00–02:00): the dream service looped
|
||
`summarize-all … backend=mi50` since 23:02, every call timing out, nothing
|
||
written to the DB since 00:56, the MI50 generating **7,000–8,000-token**
|
||
completions (a gist needs <200), all four llama.cpp slots busy, fans blaring.
|
||
|
||
This is **not** context overflow — the server log showed `context shift = 0`,
|
||
`truncated = 1 = 0`. The prompts are small (~900–1,500 tokens). The failure is
|
||
purely **unbounded generation length on a slow backend → timeout → retry loop.**
|
||
|
||
## Goals
|
||
|
||
- Keep the MI50 as the primary summary backend (Brian's preference, gaming-safe).
|
||
- Cap each summary generation so it finishes fast and can never run away.
|
||
- Make a stuck MI50 call **fail fast** and fall back to cloud, instead of looping
|
||
all night.
|
||
- Change nothing about live chat, reflect, or think.
|
||
|
||
## Design
|
||
|
||
### 1. `lyra/llm.py` — `complete()` gains two optional params
|
||
|
||
```
|
||
def complete(messages, backend="local", model=None,
|
||
max_tokens: int | None = None, timeout: float | None = None) -> str
|
||
```
|
||
|
||
- `max_tokens` (when set): passed to the create() call —
|
||
`max_tokens=` for the `cloud`/`mi50` OpenAI paths, `options={"num_predict": …}`
|
||
for the `local` Ollama path.
|
||
- `timeout` (when set): for the `cloud`/`mi50` OpenAI clients, build the client
|
||
with `timeout=<t>, max_retries=0` so the call bails quickly and *we* own the
|
||
retry policy (eliminates the hidden 3×600s). For `local`, use it as the httpx
|
||
timeout.
|
||
- Both default to `None` → **behavior identical to today** for every other
|
||
caller (chat_call, reflect, think, etc.). Backward compatible.
|
||
|
||
### 2. `lyra/summary.py` — capped, fast-fail, cloud fallback
|
||
|
||
Constants:
|
||
|
||
```
|
||
SUMMARY_MAX_TOKENS = 768 # ~3× the longest real gist; bounds gen to ~1 min on MI50
|
||
MI50_ATTEMPTS = 2 # attempts on the primary backend before falling back
|
||
SUMMARY_TIMEOUT = 150 # seconds/call — capped 768-tok gist finishes in ~60-90s
|
||
```
|
||
|
||
Rewrite `_summarize_text(text, backend)`:
|
||
|
||
1. Try `backend` up to `MI50_ATTEMPTS` times, each:
|
||
`llm.complete(messages, backend=backend, max_tokens=SUMMARY_MAX_TOKENS, timeout=SUMMARY_TIMEOUT)`,
|
||
with a short backoff between attempts.
|
||
2. If all primary attempts fail **and** `backend != "cloud"` **and** an OpenAI
|
||
key is configured → one final cloud attempt (same cap/timeout), logged as
|
||
`summary fell back to cloud`.
|
||
3. If cloud also fails or is unavailable → raise.
|
||
|
||
Fallback is per-`_summarize_text` call (i.e. per chunk), so the long-session
|
||
chunk/merge path in `_summarize_transcript` is unaffected. The old `_RETRIES = 4`
|
||
loop is replaced by this structure.
|
||
|
||
## Testing
|
||
|
||
Unit (pytest, `tests/test_summary_fallback.py`), monkeypatching `llm.complete`:
|
||
|
||
- Fallback fires: `mi50` raises on every call → after `MI50_ATTEMPTS` the cloud
|
||
attempt runs and its result is returned; a `fell back to cloud` log is emitted.
|
||
- No fallback when primary is already `cloud` (retries, then raises).
|
||
- No fallback when no OpenAI key (raises after primary attempts).
|
||
- `max_tokens` and `timeout` are threaded into every `complete()` call.
|
||
|
||
Plus a light `llm.complete` test that `max_tokens`/`timeout` reach the client
|
||
kwargs (monkeypatch the OpenAI client).
|
||
|
||
## Verification (real)
|
||
|
||
After deploy (`systemctl --user restart lyra-dream lyra-web` — editable install):
|
||
watch `journalctl --user -fu lyra-dream` through a summarize cycle and confirm
|
||
`llm done … out≈768` completing in ~1 min, an actual `summarized session` row
|
||
written (DB summary count rises), and **no** "Request timed out". Confirm the
|
||
llama.cpp slot shows bounded `n_decoded ≈ 768`.
|
||
|
||
## Out of scope (YAGNI)
|
||
|
||
- No change to `chat_call`/reflect/think or `config.summary_backend`.
|
||
- No change to profile/era/narrative rebuild calls (separate, and not the loop
|
||
culprit); can adopt the same `max_tokens` later if they show the same rambling.
|