feat: MI50 as a Lyra backend (OpenAI-compatible local GPU)
The MI50 box (CT202) runs an OpenAI-compatible llama.cpp server on 10.0.0.44:8080. Wire it in as a third backend: - llm.complete gains backend="mi50" (OpenAI client pointed at MI50_BASE_URL) - config: MI50_BASE_URL (default http://10.0.0.44:8080/v1) + MI50_MODEL - chat.respond labels the model per backend; web _backend_for maps "mi50" - UI backend selector adds "MI50 — local GPU" Verified end-to-end: llm.complete(backend="mi50") returns from the live server. See homelab-inference memory for the box topology. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
+7
-1
@@ -14,7 +14,7 @@ class Message(TypedDict):
|
||||
content: str
|
||||
|
||||
|
||||
Backend = Literal["local", "cloud"]
|
||||
Backend = Literal["local", "cloud", "mi50"]
|
||||
|
||||
|
||||
def complete(messages: list[Message], backend: Backend = "local") -> str:
|
||||
@@ -26,6 +26,12 @@ def complete(messages: list[Message], backend: Backend = "local") -> str:
|
||||
resp = client.chat.completions.create(model=cfg.cloud_model, messages=messages)
|
||||
return resp.choices[0].message.content or ""
|
||||
|
||||
if backend == "mi50":
|
||||
# MI50 box runs an OpenAI-compatible llama.cpp server; key is unused.
|
||||
client = OpenAI(api_key="not-needed", base_url=cfg.mi50_base_url)
|
||||
resp = client.chat.completions.create(model=cfg.mi50_model, messages=messages)
|
||||
return resp.choices[0].message.content or ""
|
||||
|
||||
resp = httpx.post(
|
||||
f"{cfg.local_base_url}/api/chat",
|
||||
json={"model": cfg.local_model, "messages": messages, "stream": False},
|
||||
|
||||
Reference in New Issue
Block a user