serversdown/project-lyra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix: serial consolidation on GPU backends (was timing out the MI50)

Browse Source 
summarize_all fanned out 8 concurrent workers, but the MI50 llama.cpp server runs
a single slot (--parallel 1). Firing 8 at once queued them, blew the client timeout
('summary retry … Request timed out'), and thrashed/cancelled the KV cache — wasted
compute and heat. Concurrency is now backend-aware: 8 for cloud, 1 for local/MI50.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
serversdown
2026-06-27 09:26:25 +00:00
parent 44bb8687f7
commit abac42c344
1 changed files with 9 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lyra/summary.py
+9 -5
View File
@@ -129,16 +129,20 @@ def maybe_summarize_async(session_id: str, backend: Backend | None = None) -> No
def summarize_all( def summarize_all(
backend: Backend | None = None, limit: int | None = None, workers: int = 8 backend: Backend | None = None, limit: int | None = None, workers: int | None = None
) -> dict: ) -> dict:
"""Summarize every session that needs it. Idempotent and resumable. """Summarize every session that needs it. Idempotent and resumable.
LLM summarization runs concurrently across `workers` threads (great for a Concurrency is backend-aware: the cloud API parallelizes happily, but the
cloud backend). DB reads (loading transcripts) and writes (store_summary, local/MI50 GPU servers run a single slot (llama.cpp --parallel 1) — firing N
which also embeds) happen on the main thread, so the single SQLite requests at them just queues, blows the client timeout, and thrashes the KV
connection is never touched from multiple threads. cache (wasted compute + heat). So GPU backends run serially unless overridden.
DB reads/writes (store_summary embeds) stay on the main thread, so the single
SQLite connection is never touched from multiple threads.
""" """
backend = backend or config.load().summary_backend backend = backend or config.load().summary_backend
if workers is None:
workers = 8 if backend == "cloud" else 1
# Main thread: collect the work (transcripts) for sessions needing a summary. # Main thread: collect the work (transcripts) for sessions needing a summary.
todo: list[tuple[str, str, int]] = [] todo: list[tuple[str, str, int]] = []