serversdown
29a4d59661
The dream cycle's summarize_all ran uncapped against the MI50: no max_tokens and no timeout, so the OpenAI SDK's 600s x2-retry default meant ~30 min per call. Combined with summary.py's own retry loop, one unsummarizable session pegged the GPU for hours (observed 2026-07-04: stuck since 23:02, nothing saved since 00:56, 7-8k-token runaway generations, all 4 llama.cpp slots busy). Not context overflow (0 shifts/truncations) - purely unbounded length on a slow backend timing out and retrying. - llm.complete(): add optional max_tokens (caps generation; num_predict for Ollama) and timeout (bounds the request and sets max_retries=0 so the caller owns retry policy). Both default None -> unchanged for every existing caller. - summary.py: cap gists at 768 tokens, 150s/call fast-fail, 2 MI50 attempts then one cloud fallback (when primary isn't already cloud and a key exists). Known limitation (scoped out per decision): the fallback triggers on timeouts/exceptions, not on a degraded backend returning garbage as a 200. Tests: fallback fires after 2 MI50 failures; no fallback when primary is cloud or no key; cap+timeout threaded into every complete() call; llm bounds tests. 172 pass, ruff clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015yrEb5qpPGv2FjyxrB7LLk
64 lines
2.2 KiB
Python
64 lines
2.2 KiB
Python
"""llm.complete: `max_tokens` and `timeout` are threaded into the backend call.
|
|
|
|
The OpenAI client is faked so nothing hits a network. We assert the generation
|
|
cap reaches the create() call and the fast-fail timeout reaches the client (with
|
|
max_retries=0 so summary.py owns the retry policy, not the SDK).
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import types
|
|
|
|
import pytest
|
|
|
|
from lyra import llm
|
|
|
|
|
|
@pytest.fixture
|
|
def fake_openai(monkeypatch):
|
|
recorded: dict = {}
|
|
|
|
class FakeCompletions:
|
|
def create(self, **kwargs):
|
|
recorded["create"] = kwargs
|
|
msg = types.SimpleNamespace(content="ok")
|
|
return types.SimpleNamespace(choices=[types.SimpleNamespace(message=msg)])
|
|
|
|
class FakeClient:
|
|
def __init__(self, **kwargs):
|
|
recorded["client"] = kwargs
|
|
self.chat = types.SimpleNamespace(completions=FakeCompletions())
|
|
|
|
monkeypatch.setattr(llm, "OpenAI", FakeClient)
|
|
monkeypatch.setattr(llm, "load", lambda: types.SimpleNamespace(
|
|
mi50_base_url="http://mi50/v1", mi50_model="local-gpu",
|
|
cloud_model="gpt-4o-mini", openai_api_key="sk-test", local_model="l",
|
|
))
|
|
return recorded
|
|
|
|
|
|
def test_mi50_threads_max_tokens_and_timeout(fake_openai):
|
|
out = llm.complete([{"role": "user", "content": "hi"}],
|
|
backend="mi50", max_tokens=768, timeout=150)
|
|
|
|
assert out == "ok"
|
|
assert fake_openai["create"]["max_tokens"] == 768
|
|
assert fake_openai["client"]["timeout"] == 150
|
|
assert fake_openai["client"]["max_retries"] == 0
|
|
|
|
|
|
def test_cloud_threads_max_tokens_and_timeout(fake_openai):
|
|
llm.complete([{"role": "user", "content": "hi"}],
|
|
backend="cloud", max_tokens=768, timeout=150)
|
|
|
|
assert fake_openai["create"]["max_tokens"] == 768
|
|
assert fake_openai["client"]["timeout"] == 150
|
|
assert fake_openai["client"]["max_retries"] == 0
|
|
|
|
|
|
def test_defaults_omit_cap_and_keep_current_behavior(fake_openai):
|
|
# No cap / timeout passed -> create() gets no max_tokens, client unbounded.
|
|
llm.complete([{"role": "user", "content": "hi"}], backend="mi50")
|
|
|
|
assert "max_tokens" not in fake_openai["create"]
|
|
assert "timeout" not in fake_openai["client"]
|