feat: MI50 as a Lyra backend (OpenAI-compatible local GPU)
The MI50 box (CT202) runs an OpenAI-compatible llama.cpp server on 10.0.0.44:8080. Wire it in as a third backend: - llm.complete gains backend="mi50" (OpenAI client pointed at MI50_BASE_URL) - config: MI50_BASE_URL (default http://10.0.0.44:8080/v1) + MI50_MODEL - chat.respond labels the model per backend; web _backend_for maps "mi50" - UI backend selector adds "MI50 — local GPU" Verified end-to-end: llm.complete(backend="mi50") returns from the live server. See homelab-inference memory for the box topology. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -14,6 +14,8 @@ load_dotenv()
|
||||
class Config:
|
||||
local_base_url: str
|
||||
local_model: str
|
||||
mi50_base_url: str # OpenAI-compatible llama.cpp server on the MI50 box
|
||||
mi50_model: str
|
||||
openai_api_key: str
|
||||
cloud_model: str
|
||||
embed_backend: str # "cloud" (OpenAI) or "local" (Ollama)
|
||||
@@ -27,6 +29,8 @@ def load() -> Config:
|
||||
return Config(
|
||||
local_base_url=os.getenv("LOCAL_BASE_URL", "http://localhost:11434"),
|
||||
local_model=os.getenv("LOCAL_MODEL", "qwen2.5:7b-instruct"),
|
||||
mi50_base_url=os.getenv("MI50_BASE_URL", "http://10.0.0.44:8080/v1"),
|
||||
mi50_model=os.getenv("MI50_MODEL", "local-gpu"),
|
||||
openai_api_key=os.getenv("OPENAI_API_KEY", ""),
|
||||
cloud_model=os.getenv("CLOUD_MODEL", "gpt-4o-mini"),
|
||||
embed_backend=os.getenv("EMBED_BACKEND", "cloud").lower(),
|
||||
|
||||
Reference in New Issue
Block a user