fix: serial consolidation on GPU backends (was timing out the MI50)

summarize_all fanned out 8 concurrent workers, but the MI50 llama.cpp server runs
a single slot (--parallel 1). Firing 8 at once queued them, blew the client timeout
('summary retry … Request timed out'), and thrashed/cancelled the KV cache — wasted
compute and heat. Concurrency is now backend-aware: 8 for cloud, 1 for local/MI50.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-27 09:26:25 +00:00
parent 44bb8687f7
commit abac42c344
+9 -5
View File
@@ -129,16 +129,20 @@ def maybe_summarize_async(session_id: str, backend: Backend | None = None) -> No
def summarize_all(
backend: Backend | None = None, limit: int | None = None, workers: int = 8
backend: Backend | None = None, limit: int | None = None, workers: int | None = None
) -> dict:
"""Summarize every session that needs it. Idempotent and resumable.
LLM summarization runs concurrently across `workers` threads (great for a
cloud backend). DB reads (loading transcripts) and writes (store_summary,
which also embeds) happen on the main thread, so the single SQLite
connection is never touched from multiple threads.
Concurrency is backend-aware: the cloud API parallelizes happily, but the
local/MI50 GPU servers run a single slot (llama.cpp --parallel 1) — firing N
requests at them just queues, blows the client timeout, and thrashes the KV
cache (wasted compute + heat). So GPU backends run serially unless overridden.
DB reads/writes (store_summary embeds) stay on the main thread, so the single
SQLite connection is never touched from multiple threads.
"""
backend = backend or config.load().summary_backend
if workers is None:
workers = 8 if backend == "cloud" else 1
# Main thread: collect the work (transcripts) for sessions needing a summary.
todo: list[tuple[str, str, int]] = []