v0.5.1-Major cortex rework. clean up done too. Merge from dev
v0.5.1-Major cortex rework. clean up done too.
This commit is contained in:
GitHub
+1
-1
@@ -4,7 +4,7 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
*.log
|
||||
|
||||
/.vscode/
|
||||
# =============================
|
||||
# 🔐 Environment files (NEVER commit secrets!)
|
||||
# =============================
|
||||
|
||||
Vendored
+7
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"workbench.colorCustomizations": {
|
||||
"activityBar.background": "#16340C",
|
||||
"titleBar.activeBackground": "#1F4911",
|
||||
"titleBar.activeForeground": "#F6FDF4"
|
||||
}
|
||||
}
|
||||
+659
-629
File diff suppressed because it is too large
Load Diff
@@ -2,7 +2,7 @@
|
||||
|
||||
Lyra is a modular persistent AI companion system with advanced reasoning capabilities.
|
||||
It provides memory-backed chat using **NeoMem** + **Relay** + **Cortex**,
|
||||
with multi-stage reasoning pipeline powered by distributed LLM backends.
|
||||
with multi-stage reasoning pipeline powered by HTTP-based LLM backends.
|
||||
|
||||
## Mission Statement
|
||||
|
||||
@@ -12,9 +12,9 @@ The point of Project Lyra is to give an AI chatbot more abilities than a typical
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
Project Lyra operates as a series of Docker containers networked together in a microservices architecture. Like how the brain has regions, Lyra has modules:
|
||||
Project Lyra operates as a **single docker-compose deployment** with multiple Docker containers networked together in a microservices architecture. Like how the brain has regions, Lyra has modules:
|
||||
|
||||
### A. VM 100 - lyra-core (Core Services)
|
||||
### Core Services
|
||||
|
||||
**1. Relay** (Node.js/Express) - Port 7078
|
||||
- Main orchestrator and message router
|
||||
@@ -26,7 +26,7 @@ Project Lyra operates as a series of Docker containers networked together in a m
|
||||
|
||||
**2. UI** (Static HTML)
|
||||
- Browser-based chat interface with cyberpunk theme
|
||||
- Connects to Relay at `http://10.0.0.40:7078`
|
||||
- Connects to Relay
|
||||
- Saves and loads sessions
|
||||
- OpenAI-compatible message format
|
||||
|
||||
@@ -37,7 +37,7 @@ Project Lyra operates as a series of Docker containers networked together in a m
|
||||
- Semantic memory updates and retrieval
|
||||
- No external SDK dependencies - fully local
|
||||
|
||||
### B. VM 101 - lyra-cortex (Reasoning Layer)
|
||||
### Reasoning Layer
|
||||
|
||||
**4. Cortex** (Python/FastAPI) - Port 7081
|
||||
- Primary reasoning engine with multi-stage pipeline
|
||||
@@ -47,7 +47,7 @@ Project Lyra operates as a series of Docker containers networked together in a m
|
||||
3. **Refinement** - Polishes and improves the draft
|
||||
4. **Persona** - Applies Lyra's personality and speaking style
|
||||
- Integrates with Intake for short-term context
|
||||
- Flexible LLM router supporting multiple backends
|
||||
- Flexible LLM router supporting multiple backends via HTTP
|
||||
|
||||
**5. Intake v0.2** (Python/FastAPI) - Port 7080
|
||||
- Simplified short-term memory summarization
|
||||
@@ -60,14 +60,16 @@ Project Lyra operates as a series of Docker containers networked together in a m
|
||||
- `GET /summaries?session_id={id}` - Retrieve session summary
|
||||
- `POST /close_session/{id}` - Close and cleanup session
|
||||
|
||||
### C. LLM Backends (Remote/Local APIs)
|
||||
### LLM Backends (HTTP-based)
|
||||
|
||||
**Multi-Backend Strategy:**
|
||||
- **PRIMARY**: vLLM on AMD MI50 GPU (`http://10.0.0.43:8000`) - Cortex reasoning, Intake
|
||||
- **SECONDARY**: Ollama on RTX 3090 (`http://10.0.0.3:11434`) - Configurable per-module
|
||||
- **CLOUD**: OpenAI API (`https://api.openai.com/v1`) - Cortex persona layer
|
||||
**All LLM communication is done via HTTP APIs:**
|
||||
- **PRIMARY**: vLLM server (`http://10.0.0.43:8000`) - AMD MI50 GPU backend
|
||||
- **SECONDARY**: Ollama server (`http://10.0.0.3:11434`) - RTX 3090 backend
|
||||
- **CLOUD**: OpenAI API (`https://api.openai.com/v1`) - Cloud-based models
|
||||
- **FALLBACK**: Local backup (`http://10.0.0.41:11435`) - Emergency fallback
|
||||
|
||||
Each module can be configured to use a different backend via environment variables.
|
||||
|
||||
---
|
||||
|
||||
## Data Flow Architecture (v0.5.0)
|
||||
@@ -101,22 +103,22 @@ Relay → UI (returns final response)
|
||||
|
||||
### Cortex 4-Stage Reasoning Pipeline:
|
||||
|
||||
1. **Reflection** (`reflection.py`) - Cloud backend (OpenAI)
|
||||
1. **Reflection** (`reflection.py`) - Configurable LLM via HTTP
|
||||
- Analyzes user intent and conversation context
|
||||
- Generates meta-awareness notes
|
||||
- "What is the user really asking?"
|
||||
|
||||
2. **Reasoning** (`reasoning.py`) - Primary backend (vLLM)
|
||||
2. **Reasoning** (`reasoning.py`) - Configurable LLM via HTTP
|
||||
- Retrieves short-term context from Intake
|
||||
- Creates initial draft answer
|
||||
- Integrates context, reflection notes, and user prompt
|
||||
|
||||
3. **Refinement** (`refine.py`) - Primary backend (vLLM)
|
||||
3. **Refinement** (`refine.py`) - Configurable LLM via HTTP
|
||||
- Polishes the draft answer
|
||||
- Improves clarity and coherence
|
||||
- Ensures factual consistency
|
||||
|
||||
4. **Persona** (`speak.py`) - Cloud backend (OpenAI)
|
||||
4. **Persona** (`speak.py`) - Configurable LLM via HTTP
|
||||
- Applies Lyra's personality and speaking style
|
||||
- Natural, conversational output
|
||||
- Final answer returned to user
|
||||
@@ -125,7 +127,7 @@ Relay → UI (returns final response)
|
||||
|
||||
## Features
|
||||
|
||||
### Lyra-Core (VM 100)
|
||||
### Core Services
|
||||
|
||||
**Relay**:
|
||||
- Main orchestrator and message router
|
||||
@@ -150,11 +152,11 @@ Relay → UI (returns final response)
|
||||
- Session save/load functionality
|
||||
- OpenAI message format support
|
||||
|
||||
### Cortex (VM 101)
|
||||
### Reasoning Layer
|
||||
|
||||
**Cortex** (v0.5):
|
||||
- Multi-stage reasoning pipeline (reflection → reasoning → refine → persona)
|
||||
- Flexible LLM backend routing
|
||||
- Flexible LLM backend routing via HTTP
|
||||
- Per-stage backend selection
|
||||
- Async processing throughout
|
||||
- IntakeClient integration for short-term context
|
||||
@@ -169,7 +171,7 @@ Relay → UI (returns final response)
|
||||
- **Breaking change from v0.1**: Removed cascading summaries (L1, L2, L5, L10, L20, L30)
|
||||
|
||||
**LLM Router**:
|
||||
- Dynamic backend selection
|
||||
- Dynamic backend selection via HTTP
|
||||
- Environment-driven configuration
|
||||
- Support for vLLM, Ollama, OpenAI, custom endpoints
|
||||
- Per-module backend preferences
|
||||
@@ -220,49 +222,44 @@ Relay → UI (returns final response)
|
||||
"imported_at": "2025-11-07T03:55:00Z"
|
||||
}```
|
||||
|
||||
# Cortex VM (VM101, CT201)
|
||||
- **CT201 main reasoning orchestrator.**
|
||||
- This is the internal brain of Lyra.
|
||||
- Running in a privellaged LXC.
|
||||
- Currently a locally served LLM running on a Radeon Instinct HI50, using a customized version of vLLM that lets it use ROCm.
|
||||
- Accessible via 10.0.0.43:8000/v1/completions.
|
||||
---
|
||||
|
||||
- **Intake v0.1.1 **
|
||||
- Recieves messages from relay and summarizes them in a cascading format.
|
||||
- Continues to summarize smaller amounts of exhanges while also generating large scale conversational summaries. (L20)
|
||||
- Intake then sends to cortex for self reflection, neomem for memory consolidation.
|
||||
## Docker Deployment
|
||||
|
||||
- **Reflect **
|
||||
-TBD
|
||||
All services run in a single docker-compose stack with the following containers:
|
||||
|
||||
# Self hosted vLLM server #
|
||||
- **CT201 main reasoning orchestrator.**
|
||||
- This is the internal brain of Lyra.
|
||||
- Running in a privellaged LXC.
|
||||
- Currently a locally served LLM running on a Radeon Instinct HI50, using a customized version of vLLM that lets it use ROCm.
|
||||
- Accessible via 10.0.0.43:8000/v1/completions.
|
||||
- **Stack Flow**
|
||||
- [Proxmox Host]
|
||||
└── loads AMDGPU driver
|
||||
└── boots CT201 (order=2)
|
||||
- **neomem-postgres** - PostgreSQL with pgvector extension (port 5432)
|
||||
- **neomem-neo4j** - Neo4j graph database (ports 7474, 7687)
|
||||
- **neomem-api** - NeoMem memory service (port 7077)
|
||||
- **relay** - Main orchestrator (port 7078)
|
||||
- **cortex** - Reasoning engine (port 7081)
|
||||
- **intake** - Short-term memory summarization (port 7080) - currently disabled
|
||||
- **rag** - RAG search service (port 7090) - currently disabled
|
||||
|
||||
[CT201 GPU Container]
|
||||
├── lyra-start-vllm.sh → starts vLLM ROCm model server
|
||||
├── lyra-vllm.service → runs the above automatically
|
||||
├── lyra-core.service → launches Cortex + Intake Docker stack
|
||||
└── Docker Compose → runs Cortex + Intake containers
|
||||
All containers communicate via the `lyra_net` Docker bridge network.
|
||||
|
||||
[Cortex Container]
|
||||
├── Listens on port 7081
|
||||
├── Talks to NVGRAM (mem API) + Intake
|
||||
└── Main relay between Lyra UI ↔ memory ↔ model
|
||||
## External LLM Services
|
||||
|
||||
[Intake Container]
|
||||
├── Listens on port 7080
|
||||
├── Summarizes every few exchanges
|
||||
├── Writes summaries to /app/logs/summaries.log
|
||||
└── Future: sends summaries → Cortex for reflection
|
||||
The following LLM backends are accessed via HTTP (not part of docker-compose):
|
||||
|
||||
- **vLLM Server** (`http://10.0.0.43:8000`)
|
||||
- AMD MI50 GPU-accelerated inference
|
||||
- Custom ROCm-enabled vLLM build
|
||||
- Primary backend for reasoning and refinement stages
|
||||
|
||||
- **Ollama Server** (`http://10.0.0.3:11434`)
|
||||
- RTX 3090 GPU-accelerated inference
|
||||
- Secondary/configurable backend
|
||||
- Model: qwen2.5:7b-instruct-q4_K_M
|
||||
|
||||
- **OpenAI API** (`https://api.openai.com/v1`)
|
||||
- Cloud-based inference
|
||||
- Used for reflection and persona stages
|
||||
- Model: gpt-4o-mini
|
||||
|
||||
- **Fallback Server** (`http://10.0.0.41:11435`)
|
||||
- Emergency backup endpoint
|
||||
- Local llama-3.2-8b-instruct model
|
||||
|
||||
---
|
||||
|
||||
@@ -292,6 +289,7 @@ Relay → UI (returns final response)
|
||||
|
||||
### Non-Critical
|
||||
- Session management endpoints not fully implemented in Relay
|
||||
- Intake service currently disabled in docker-compose.yml
|
||||
- RAG service currently disabled in docker-compose.yml
|
||||
- Cortex `/ingest` endpoint is a stub
|
||||
|
||||
@@ -307,14 +305,19 @@ Relay → UI (returns final response)
|
||||
|
||||
### Prerequisites
|
||||
- Docker + Docker Compose
|
||||
- PostgreSQL 13+, Neo4j 4.4+ (for NeoMem)
|
||||
- At least one LLM API endpoint (vLLM, Ollama, or OpenAI)
|
||||
- At least one HTTP-accessible LLM endpoint (vLLM, Ollama, or OpenAI API key)
|
||||
|
||||
### Setup
|
||||
1. Configure environment variables in `.env` files
|
||||
2. Start services: `docker-compose up -d`
|
||||
3. Check health: `curl http://localhost:7078/_health`
|
||||
4. Access UI: `http://localhost:7078`
|
||||
1. Copy `.env.example` to `.env` and configure your LLM backend URLs and API keys
|
||||
2. Start all services with docker-compose:
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
3. Check service health:
|
||||
```bash
|
||||
curl http://localhost:7078/_health
|
||||
```
|
||||
4. Access the UI at `http://localhost:7078`
|
||||
|
||||
### Test
|
||||
```bash
|
||||
@@ -326,6 +329,8 @@ curl -X POST http://localhost:7078/v1/chat/completions \
|
||||
}'
|
||||
```
|
||||
|
||||
All backend databases (PostgreSQL and Neo4j) are automatically started as part of the docker-compose stack.
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
@@ -345,104 +350,44 @@ NeoMem is a derivative work based on Mem0 OSS (Apache 2.0).
|
||||
|
||||
---
|
||||
|
||||
## 📦 Requirements
|
||||
## Integration Notes
|
||||
|
||||
- Docker + Docker Compose
|
||||
- Postgres + Neo4j (for NeoMem)
|
||||
- Access to an open AI or ollama style API.
|
||||
- OpenAI API key (for Relay fallback LLMs)
|
||||
|
||||
**Dependencies:**
|
||||
- fastapi==0.115.8
|
||||
- uvicorn==0.34.0
|
||||
- pydantic==2.10.4
|
||||
- python-dotenv==1.0.1
|
||||
- psycopg>=3.2.8
|
||||
- ollama
|
||||
- NeoMem API is compatible with Mem0 OSS endpoints (`/memories`, `/search`)
|
||||
- All services communicate via Docker internal networking on the `lyra_net` bridge
|
||||
- History and entity graphs are managed via PostgreSQL + Neo4j
|
||||
- LLM backends are accessed via HTTP and configured in `.env`
|
||||
|
||||
---
|
||||
|
||||
🔌 Integration Notes
|
||||
## Beta Lyrae - RAG Memory System (Currently Disabled)
|
||||
|
||||
Lyra-Core connects to neomem-api:8000 inside Docker or localhost:7077 locally.
|
||||
**Note:** The RAG service is currently disabled in docker-compose.yml
|
||||
|
||||
API endpoints remain identical to Mem0 (/memories, /search).
|
||||
### Requirements
|
||||
- Python 3.10+
|
||||
- Dependencies: `chromadb openai tqdm python-dotenv fastapi uvicorn`
|
||||
- Persistent storage: `./chromadb` or `/mnt/data/lyra_rag_db`
|
||||
|
||||
History and entity graphs managed internally via Postgres + Neo4j.
|
||||
|
||||
---
|
||||
|
||||
🧱 Architecture Snapshot
|
||||
|
||||
User → Relay → Cortex
|
||||
↓
|
||||
[RAG Search]
|
||||
↓
|
||||
[Reflection Loop]
|
||||
↓
|
||||
Intake (async summaries)
|
||||
↓
|
||||
NeoMem (persistent memory)
|
||||
|
||||
**Cortex v0.4.1 introduces the first fully integrated reasoning loop.**
|
||||
- Data Flow:
|
||||
- User message enters Cortex via /reason.
|
||||
- Cortex assembles context:
|
||||
- Intake summaries (short-term memory)
|
||||
- RAG contextual data (knowledge base)
|
||||
- LLM generates initial draft (call_llm).
|
||||
- Reflection loop critiques and refines the answer.
|
||||
- Intake asynchronously summarizes and sends snapshots to NeoMem.
|
||||
|
||||
RAG API Configuration:
|
||||
Set RAG_API_URL in .env (default: http://localhost:7090).
|
||||
|
||||
---
|
||||
|
||||
## Setup and Operation ##
|
||||
|
||||
## Beta Lyrae - RAG memory system ##
|
||||
**Requirements**
|
||||
-Env= python 3.10+
|
||||
-Dependences: pip install chromadb openai tqdm python-dotenv fastapi uvicorn jq
|
||||
-Persistent storage path: ./chromadb (can be moved to /mnt/data/lyra_rag_db)
|
||||
|
||||
**Import Chats**
|
||||
- Chats need to be formatted into the correct format of
|
||||
### Setup
|
||||
1. Import chat logs (must be in OpenAI message format):
|
||||
```bash
|
||||
python3 rag/rag_chat_import.py
|
||||
```
|
||||
"messages": [
|
||||
{
|
||||
"role:" "user",
|
||||
"content": "Message here"
|
||||
},
|
||||
"messages": [
|
||||
{
|
||||
"role:" "assistant",
|
||||
"content": "Message here"
|
||||
},```
|
||||
- Organize the chats into categorical folders. This step is optional, but it helped me keep it straight.
|
||||
- run "python3 rag_chat_import.py", chats will then be imported automatically. For reference, it took 32 Minutes to import 68 Chat logs (aprox 10.3MB).
|
||||
|
||||
**Build API Server**
|
||||
- Run: rag_build.py, this automatically builds the chromaDB using data saved in the /chatlogs/ folder. (docs folder to be added in future.)
|
||||
- Run: rag_api.py or ```uvicorn rag_api:app --host 0.0.0.0 --port 7090```
|
||||
|
||||
**Query**
|
||||
- Run: python3 rag_query.py "Question here?"
|
||||
- For testing a curl command can reach it too
|
||||
2. Build and start the RAG API server:
|
||||
```bash
|
||||
cd rag
|
||||
python3 rag_build.py
|
||||
uvicorn rag_api:app --host 0.0.0.0 --port 7090
|
||||
```
|
||||
|
||||
3. Query the RAG system:
|
||||
```bash
|
||||
curl -X POST http://127.0.0.1:7090/rag/search \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"query": "What is the current state of Cortex and Project Lyra?",
|
||||
"query": "What is the current state of Cortex?",
|
||||
"where": {"category": "lyra"}
|
||||
}'
|
||||
```
|
||||
|
||||
# Beta Lyrae - RAG System
|
||||
|
||||
## 📖 License
|
||||
NeoMem is a derivative work based on the Mem0 OSS project (Apache 2.0).
|
||||
This fork retains the original Apache 2.0 license and adds local modifications.
|
||||
© 2025 Terra-Mechanics / ServersDown Labs. All modifications released under Apache 2.0.
|
||||
|
||||
|
||||
File diff suppressed because one or more lines are too long
Binary file not shown.
@@ -1,47 +0,0 @@
|
||||
# DEPRECATED - USE /home/serversdown/project-lyra/docker-compose.yml instead
|
||||
# This file is no longer needed. All services are now in the main docker-compose.yml
|
||||
# Safe to delete after verifying main compose file works correctly.
|
||||
#
|
||||
services:
|
||||
relay:
|
||||
build:
|
||||
context: ./relay
|
||||
container_name: relay
|
||||
restart: always
|
||||
ports:
|
||||
- "7078:7078"
|
||||
env_file:
|
||||
- ../.env # Use root .env (core/.env is now redundant)
|
||||
volumes:
|
||||
- ./relay/sessions:/app/sessions
|
||||
networks:
|
||||
- lyra-net
|
||||
|
||||
# persona-sidecar:
|
||||
# build:
|
||||
# context: ./persona-sidecar
|
||||
# container_name: persona-sidecar
|
||||
# env_file:
|
||||
# - .env
|
||||
# ports:
|
||||
# - "7080:7080"
|
||||
# volumes:
|
||||
# - ./persona-sidecar/personas.json:/app/personas.json:rw
|
||||
# restart: unless-stopped
|
||||
# networks:
|
||||
# - lyra-net
|
||||
|
||||
lyra-ui:
|
||||
image: nginx:alpine
|
||||
container_name: lyra-ui
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8081:80"
|
||||
volumes:
|
||||
- ./ui:/usr/share/nginx/html:ro
|
||||
networks:
|
||||
- lyra-net
|
||||
|
||||
networks:
|
||||
lyra-net:
|
||||
external: true
|
||||
@@ -1,16 +0,0 @@
|
||||
# Ignore node_modules - Docker will rebuild them inside
|
||||
node_modules
|
||||
npm-debug.log
|
||||
yarn-error.log
|
||||
*.log
|
||||
|
||||
# Ignore environment files
|
||||
.env
|
||||
.env.local
|
||||
|
||||
# Ignore OS/editor cruft
|
||||
.DS_Store
|
||||
*.swp
|
||||
*.swo
|
||||
.vscode
|
||||
.idea
|
||||
@@ -1,18 +0,0 @@
|
||||
# relay/Dockerfile
|
||||
FROM node:18-alpine
|
||||
|
||||
# Create app directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy package.json and install deps first (better caching)
|
||||
COPY package.json ./
|
||||
RUN npm install
|
||||
|
||||
# Copy the rest of the app
|
||||
COPY . .
|
||||
|
||||
# Expose port
|
||||
EXPOSE 7078
|
||||
|
||||
# Run the server
|
||||
CMD ["npm", "start"]
|
||||
@@ -1,73 +0,0 @@
|
||||
// relay/lib/cortex.js
|
||||
import fetch from "node-fetch";
|
||||
|
||||
const REFLECT_URL = process.env.CORTEX_URL || "http://localhost:7081/reflect";
|
||||
const INGEST_URL = process.env.CORTEX_URL_INGEST || "http://localhost:7081/ingest";
|
||||
|
||||
export async function reflectWithCortex(userInput, memories = []) {
|
||||
const body = { prompt: userInput, memories };
|
||||
try {
|
||||
const res = await fetch(REFLECT_URL, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(body),
|
||||
timeout: 120000,
|
||||
});
|
||||
|
||||
const rawText = await res.text();
|
||||
console.log("🔎 [Cortex-Debug] rawText from /reflect →", rawText.slice(0, 300));
|
||||
if (!res.ok) {
|
||||
throw new Error(`HTTP ${res.status} — ${rawText.slice(0, 200)}`);
|
||||
}
|
||||
|
||||
let data;
|
||||
try {
|
||||
data = JSON.parse(rawText);
|
||||
} catch (err) {
|
||||
// Fallback ① try to grab a JSON-looking block
|
||||
const match = rawText.match(/\{[\s\S]*\}/);
|
||||
if (match) {
|
||||
try {
|
||||
data = JSON.parse(match[0]);
|
||||
} catch {
|
||||
data = { reflection_raw: rawText.trim(), notes: "partial parse" };
|
||||
}
|
||||
} else {
|
||||
// Fallback ② if it’s already an object (stringified Python dict)
|
||||
try {
|
||||
const normalized = rawText
|
||||
.replace(/'/g, '"') // convert single quotes
|
||||
.replace(/None/g, 'null'); // convert Python None
|
||||
data = JSON.parse(normalized);
|
||||
} catch {
|
||||
data = { reflection_raw: rawText.trim(), notes: "no JSON found" };
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (typeof data !== "object") {
|
||||
data = { reflection_raw: rawText.trim(), notes: "non-object response" };
|
||||
}
|
||||
|
||||
console.log("🧠 Cortex reflection normalized:", data);
|
||||
return data;
|
||||
} catch (e) {
|
||||
console.warn("⚠️ Cortex reflect failed:", e.message);
|
||||
return { error: e.message, reflection_raw: "" };
|
||||
}
|
||||
}
|
||||
|
||||
export async function ingestToCortex(user, assistant, reflection = {}, sessionId = "default") {
|
||||
const body = { turn: { user, assistant }, reflection, session_id: sessionId };
|
||||
try {
|
||||
const res = await fetch(INGEST_URL, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify(body),
|
||||
timeout: 120000,
|
||||
});
|
||||
console.log(`📤 Sent exchange to Cortex ingest (${res.status})`);
|
||||
} catch (e) {
|
||||
console.warn("⚠️ Cortex ingest failed:", e.message);
|
||||
}
|
||||
}
|
||||
@@ -1,93 +0,0 @@
|
||||
async function tryBackend(backend, messages) {
|
||||
if (!backend.url || !backend.model) throw new Error("missing url/model");
|
||||
|
||||
const isOllama = backend.type === "ollama";
|
||||
const isOpenAI = backend.type === "openai";
|
||||
const isVllm = backend.type === "vllm";
|
||||
const isLlamaCpp = backend.type === "llamacpp";
|
||||
|
||||
let endpoint = backend.url;
|
||||
let headers = { "Content-Type": "application/json" };
|
||||
if (isOpenAI) headers["Authorization"] = `Bearer ${OPENAI_API_KEY}`;
|
||||
|
||||
// Choose correct endpoint automatically
|
||||
if (isOllama && !endpoint.endsWith("/api/chat")) endpoint += "/api/chat";
|
||||
if ((isVllm || isLlamaCpp) && !endpoint.endsWith("/v1/completions")) endpoint += "/v1/completions";
|
||||
if (isOpenAI && !endpoint.endsWith("/v1/chat/completions")) endpoint += "/v1/chat/completions";
|
||||
|
||||
// Build payload based on backend style
|
||||
const body = (isVllm || isLlamaCpp)
|
||||
? {
|
||||
model: backend.model,
|
||||
prompt: messages.map(m => m.content).join("\n"),
|
||||
max_tokens: 400,
|
||||
temperature: 0.3,
|
||||
}
|
||||
: isOllama
|
||||
? { model: backend.model, messages, stream: false }
|
||||
: { model: backend.model, messages, stream: false };
|
||||
|
||||
const resp = await fetch(endpoint, {
|
||||
method: "POST",
|
||||
headers,
|
||||
body: JSON.stringify(body),
|
||||
timeout: 120000,
|
||||
});
|
||||
if (!resp.ok) throw new Error(`${backend.key} HTTP ${resp.status}`);
|
||||
const raw = await resp.text();
|
||||
|
||||
// 🧩 Normalize replies
|
||||
let reply = "";
|
||||
try {
|
||||
if (isOllama) {
|
||||
// Ollama sometimes returns NDJSON lines; merge them
|
||||
const merged = raw
|
||||
.split("\n")
|
||||
.filter(line => line.trim().startsWith("{"))
|
||||
.map(line => JSON.parse(line))
|
||||
.map(obj => obj.message?.content || obj.response || "")
|
||||
.join("");
|
||||
reply = merged.trim();
|
||||
} else {
|
||||
const data = JSON.parse(raw);
|
||||
console.log("🔍 RAW LLM RESPONSE:", JSON.stringify(data, null, 2));
|
||||
reply =
|
||||
data?.choices?.[0]?.text?.trim() ||
|
||||
data?.choices?.[0]?.message?.content?.trim() ||
|
||||
data?.message?.content?.trim() ||
|
||||
"";
|
||||
|
||||
|
||||
}
|
||||
} catch (err) {
|
||||
reply = `[parse error: ${err.message}]`;
|
||||
}
|
||||
|
||||
return { reply, raw, backend: backend.key };
|
||||
}
|
||||
|
||||
// ------------------------------------
|
||||
// Export the main call helper
|
||||
// ------------------------------------
|
||||
export async function callSpeechLLM(messages) {
|
||||
const backends = [
|
||||
{ key: "primary", type: "vllm", url: process.env.LLM_PRIMARY_URL, model: process.env.LLM_PRIMARY_MODEL },
|
||||
{ key: "secondary",type: "ollama", url: process.env.LLM_SECONDARY_URL,model: process.env.LLM_SECONDARY_MODEL },
|
||||
{ key: "cloud", type: "openai", url: process.env.LLM_CLOUD_URL, model: process.env.LLM_CLOUD_MODEL },
|
||||
{ key: "fallback", type: "llamacpp", url: process.env.LLM_FALLBACK_URL, model: process.env.LLM_FALLBACK_MODEL },
|
||||
];
|
||||
|
||||
for (const b of backends) {
|
||||
if (!b.url || !b.model) continue;
|
||||
try {
|
||||
console.log(`🧠 Trying backend: ${b.key.toUpperCase()} (${b.url})`);
|
||||
const out = await tryBackend(b, messages);
|
||||
console.log(`✅ Success via ${b.key.toUpperCase()}`);
|
||||
return out;
|
||||
} catch (err) {
|
||||
console.warn(`⚠️ ${b.key.toUpperCase()} failed: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error("all_backends_failed");
|
||||
}
|
||||
Generated
-8799
File diff suppressed because it is too large
Load Diff
@@ -1,16 +0,0 @@
|
||||
{
|
||||
"name": "lyra-relay",
|
||||
"version": "0.1.0",
|
||||
"type": "module",
|
||||
"main": "server.js",
|
||||
"scripts": {
|
||||
"start": "node server.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"cors": "^2.8.5",
|
||||
"dotenv": "^16.6.1",
|
||||
"express": "^4.18.2",
|
||||
"mem0ai": "^2.1.38",
|
||||
"node-fetch": "^3.3.2"
|
||||
}
|
||||
}
|
||||
@@ -1,156 +0,0 @@
|
||||
import express from "express";
|
||||
import dotenv from "dotenv";
|
||||
import cors from "cors";
|
||||
import fs from "fs";
|
||||
import path from "path";
|
||||
|
||||
dotenv.config();
|
||||
|
||||
const app = express();
|
||||
app.use(cors());
|
||||
app.use(express.json());
|
||||
|
||||
const PORT = Number(process.env.PORT || 7078);
|
||||
const CORTEX_API = process.env.CORTEX_API || "http://cortex:7081";
|
||||
const CORTEX_INGEST = process.env.CORTEX_URL_INGEST || "http://cortex:7081/ingest";
|
||||
const sessionsDir = path.join(process.cwd(), "sessions");
|
||||
|
||||
if (!fs.existsSync(sessionsDir)) fs.mkdirSync(sessionsDir);
|
||||
|
||||
// -----------------------------------------------------
|
||||
// Helper: fetch with timeout + error detail
|
||||
// -----------------------------------------------------
|
||||
async function fetchJSON(url, method = "POST", body = null, timeoutMs = 20000) {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||
|
||||
try {
|
||||
const resp = await fetch(url, {
|
||||
method,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: body ? JSON.stringify(body) : null,
|
||||
signal: controller.signal,
|
||||
});
|
||||
|
||||
const text = await resp.text();
|
||||
const parsed = text ? JSON.parse(text) : null;
|
||||
|
||||
if (!resp.ok) {
|
||||
throw new Error(
|
||||
parsed?.detail || parsed?.error || parsed?.message || text || resp.statusText
|
||||
);
|
||||
}
|
||||
return parsed;
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------
|
||||
// Helper: append session turn
|
||||
// -----------------------------------------------------
|
||||
async function appendSessionExchange(sessionId, entry) {
|
||||
const file = path.join(sessionsDir, `${sessionId}.jsonl`);
|
||||
const line = JSON.stringify({
|
||||
ts: new Date().toISOString(),
|
||||
user: entry.user,
|
||||
assistant: entry.assistant,
|
||||
raw: entry.raw,
|
||||
}) + "\n";
|
||||
|
||||
fs.appendFileSync(file, line, "utf8");
|
||||
}
|
||||
|
||||
// -----------------------------------------------------
|
||||
// HEALTHCHECK
|
||||
// -----------------------------------------------------
|
||||
app.get("/_health", (_, res) => {
|
||||
res.json({ ok: true, time: new Date().toISOString() });
|
||||
});
|
||||
|
||||
// -----------------------------------------------------
|
||||
// MAIN ENDPOINT
|
||||
// -----------------------------------------------------
|
||||
app.post("/v1/chat/completions", async (req, res) => {
|
||||
try {
|
||||
const { messages, model } = req.body;
|
||||
|
||||
if (!messages?.length) {
|
||||
return res.status(400).json({ error: "invalid_messages" });
|
||||
}
|
||||
|
||||
const userMsg = messages[messages.length - 1]?.content || "";
|
||||
console.log(`🛰️ Relay received message → "${userMsg}"`);
|
||||
|
||||
// -------------------------------------------------
|
||||
// Step 1: Ask Cortex to process the prompt
|
||||
// -------------------------------------------------
|
||||
let cortexResp;
|
||||
try {
|
||||
cortexResp = await fetchJSON(`${CORTEX_API}/reason`, "POST", {
|
||||
session_id: "default",
|
||||
user_prompt: userMsg,
|
||||
});
|
||||
} catch (err) {
|
||||
console.error("💥 Relay → Cortex error:", err.message);
|
||||
return res.status(500).json({
|
||||
error: "cortex_failed",
|
||||
detail: err.message,
|
||||
});
|
||||
}
|
||||
|
||||
const personaText = cortexResp.persona || "(no persona text returned)";
|
||||
|
||||
// -------------------------------------------------
|
||||
// Step 2: Forward to Cortex ingest (fire-and-forget)
|
||||
// -------------------------------------------------
|
||||
try {
|
||||
await fetchJSON(CORTEX_INGEST, "POST", cortexResp);
|
||||
} catch (err) {
|
||||
console.warn("⚠️ Cortex ingest failed:", err.message);
|
||||
}
|
||||
|
||||
// -------------------------------------------------
|
||||
// Step 3: Local session logging
|
||||
// -------------------------------------------------
|
||||
try {
|
||||
await appendSessionExchange("default", {
|
||||
user: userMsg,
|
||||
assistant: personaText,
|
||||
raw: cortexResp,
|
||||
});
|
||||
} catch (err) {
|
||||
console.warn("⚠️ Relay log write failed:", err.message);
|
||||
}
|
||||
|
||||
// -------------------------------------------------
|
||||
// Step 4: Return OpenAI-style response to UI
|
||||
// -------------------------------------------------
|
||||
return res.json({
|
||||
id: "relay-" + Date.now(),
|
||||
object: "chat.completion",
|
||||
model: model || "lyra",
|
||||
choices: [
|
||||
{
|
||||
index: 0,
|
||||
message: {
|
||||
role: "assistant",
|
||||
content: personaText,
|
||||
},
|
||||
finish_reason: "stop",
|
||||
},
|
||||
],
|
||||
});
|
||||
} catch (err) {
|
||||
console.error("💥 relay fatal error", err);
|
||||
res.status(500).json({
|
||||
error: "relay_failed",
|
||||
detail: err?.message || String(err),
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// -----------------------------------------------------
|
||||
app.listen(PORT, () => {
|
||||
console.log(`Relay is online at port ${PORT}`);
|
||||
});
|
||||
@@ -1,4 +0,0 @@
|
||||
{"ts":"2025-11-26T18:14:01.756Z","turn":[{"role":"user","content":"Are you alive?"},{"role":"assistant","content":"As an AI, I don't exist in the physical world and therefore cannot be considered \"alive\" in the traditional sense. However, I am operational and can engage in conversations like this one! How may I assist you today?"}]}
|
||||
{"ts":"2025-11-26T18:27:35.545Z","turn":[{"role":"user","content":"Are you awake?"},{"role":"assistant","content":"Yes, I'm awake and ready to converse with you! How can I assist you today?"}]}
|
||||
{"ts":"2025-11-26T18:33:31.708Z","turn":[{"role":"user","content":"Are you awake?"},{"role":"assistant","content":"Yes, I am here and ready to help or chat with you! What would you like to know or discuss?"}]}
|
||||
{"ts":"2025-11-26T18:49:39.424Z","turn":[{"role":"user","content":"Are you awake?"},{"role":"assistant","content":"Of course, I'm awake and ready to help you or have a conversation about anything you'd like! What can I assist you with today?"}]}
|
||||
@@ -1,50 +0,0 @@
|
||||
[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Hey im testing stuff! how goes it?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Hey! I'm here and ready to help. What are you testing?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "testing the logic loop for your brain!"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Sounds interesting! How can I assist you with that?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Just keep answering me!@"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Absolutely! I'm here to respond. What’s next?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "how about now how are you?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "I'm doing well, thanks for asking! How about you?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "are you able to tell me what your prompts are saying to you?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "I can’t share specific internal prompts or system messages, but I can tell you that I’m designed to understand and respond to your questions as best as I can! What else would you like to know?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "ok thanks how is the cheese i gave you?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "I appreciate the gesture! If I could taste it, I’m sure it would be delightful. What kind of cheese was it?"
|
||||
}
|
||||
]
|
||||
@@ -1,6 +0,0 @@
|
||||
{"ts":"2025-11-12T20:41:32.497Z","turn":[{"role":"user","content":"Hey im testing stuff! how goes it?"},{"role":"assistant","content":"Hey! I'm here and ready to help. What are you testing?"}]}
|
||||
{"ts":"2025-11-12T20:59:32.992Z","turn":[{"role":"user","content":"testing the logic loop for your brain!"},{"role":"assistant","content":"Sounds interesting! How can I assist you with that?"}]}
|
||||
{"ts":"2025-11-12T22:08:09.644Z","turn":[{"role":"user","content":"Just keep answering me!@"},{"role":"assistant","content":"Absolutely! I'm here to respond. What’s next?"}]}
|
||||
{"ts":"2025-11-12T22:08:48.693Z","turn":[{"role":"user","content":"how about now how are you?"},{"role":"assistant","content":"I'm doing well, thanks for asking! How about you?"}]}
|
||||
{"ts":"2025-11-12T22:10:46.435Z","turn":[{"role":"user","content":"are you able to tell me what your prompts are saying to you?"},{"role":"assistant","content":"I can’t share specific internal prompts or system messages, but I can tell you that I’m designed to understand and respond to your questions as best as I can! What else would you like to know?"}]}
|
||||
{"ts":"2025-11-12T23:10:16.321Z","turn":[{"role":"user","content":"ok thanks how is the cheese i gave you?"},{"role":"assistant","content":"I appreciate the gesture! If I could taste it, I’m sure it would be delightful. What kind of cheese was it?"}]}
|
||||
@@ -1,122 +0,0 @@
|
||||
[
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Yo yo yo wusup"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Hey Brian! Not much, just here to help you out. What's up with you?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "werkin on codin'"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Nice! What are you working on? Need any help with coding or anything related to your projects?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "working on a little program that saves our chats so you can search them again in different chats"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "That sounds like a great idea! Having a program to save and search through our chats could really enhance the way you interact with Lyra. Do you have a specific approach in mind for how you'll implement the search functionality?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "just adding a simple logger into the code you run on"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "That sounds like a solid plan! Implementing a simple logger can make it easier to keep track of conversations. Are you thinking of any specific features for the search functionality, like filtering by keywords or date?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "i'll like use a basic word search, possibly integrate it into a rag system i have built"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Integrating a basic word search into your RAG system sounds like a smart approach! It could enhance the way you retrieve specific chats. Are you planning to use any specific libraries or tools for your RAG system?"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "no clue!"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "No worries! If you need any suggestions on libraries or tools that could help you with implementing the RAG system or the word search feature, just let me know. I'm here to help!"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "just having some coding issues, i think i was editing the wrong version of a file LOL"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "That can definitely happen! It can be tricky to keep track of different versions. Have you made any backups or version control for your projects? That might help avoid mix-ups in the future!"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "i have everything syncing back to my main dev computer, i just do it manually. i have no idea how to use git"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "(no reply)"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "i have everything syncing back to my main dev computer, i just do it manually. i have no idea how to use git"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "(no reply)"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "i have everything syncing back to my main dev computer, i just do it manually. i have no idea how to use git"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Manually syncing is a good start, but learning Git can really help streamline your workflow and keep track of changes. If you're interested, I can provide some resources or tips on how to get started with Git! It can make managing different versions of your code much easier."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "what is the point of git? how can it streamline my work?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Git is a version control system that helps you keep track of changes in your code over time. Here are some ways it can streamline your work:\n\n1. **Version Tracking**: Git allows you to save versions of your code so you can revert back to previous versions if needed. This is especially helpful if you make a change that breaks something.\n\n2. **Collaboration**: If you ever work with others, Git makes it easy to collaborate on projects. You can work on separate branches, merge changes, and resolve conflicts.\n\n3. **Branching**: You can create branches for new features or experiments without affecting the main codebase. Once you're satisfied with the changes, you can merge them back.\n\n4. **Snapshotting**: Git takes snapshots of your code at different points in time, so you can easily see what your code looked like at any point.\n\n5. **Easy Backup**: By pushing your code to a remote repository (like GitHub or GitLab), you can back up your work and access it from anywhere.\n\nLearning Git might take a bit of time, but it can greatly enhance your coding efficiency and help you manage your projects better. If you're interested, I can point you to some beginner-friendly tutorials!"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "say i am writing one program on my main computer, and then im working on one that lives in a VM. Can i put them both in the same git?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Yes, you can use the same Git repository for your code on both your main computer and the VM. Here’s how you can do it:\n\n1. **Create a Git Repository**: You can create a Git repository on your main computer for your program. Use the command `git init` in your project's directory to initialize it.\n\n2. **Commit Changes**: As you make changes to your code on your main computer, you can use `git add .` to stage changes and `git commit -m \"your message\"` to commit them.\n\n3. **Clone the Repository on your VM**: You can then clone the Git repository on your VM using `git clone <repository-url>`. This gives you a copy of the same repository on your VM.\n\n4. **Work on Both**: You can now work on either machine. When you make changes on one, commit those changes, and then pull the latest changes from the other machine to keep everything in sync.\n\n5. **Push and Pull**: Use `git push` to push your changes to a remote repository (like GitHub) if you're using one. Use `git pull` on the other machine to pull the latest code.\n\nJust be careful about making sure you commit and push your changes before switching between machines to avoid conflicts. This setup allows you to maintain a single version of your code across both systems. If you need help with specific commands or setups, just let me know!"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "hmm i guess that might be useful. Im writting code that is running on a VM. I want to test and experiment while i write the code. The VM is headless though so i've been doing all my editing and work via SSH and SFTP."
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Using a VM and working through SSH and SFTP is a solid way to manage your coding environment! Git can definitely be beneficial in this setup as well. Here’s how it can help:\n\n1. **Easier Code Management**: With Git, you can track changes to your code directly within the VM, making it easier to manage versions as you experiment.\n\n2. **Experiment Branches**: You can create branches for different experiments. This lets you test new features or fixes without affecting your main code, and you can easily switch back and forth between branches.\n\n3. **Collaborative Testing**: If you ever work with others or want to test things collaboratively, you can push changes to a remote repository, making it easy for others to pull and test the latest code.\n\n4. **Integrated with SSH**: You can run Git commands directly from your SSH terminal on the VM, making it straightforward to use. You won’t need any additional tools.\n\n5. **Seamless Remote Access**: Since you’re using SFTP, you can clone the repository to your VM. This way, you can easily pull updates you've made on your main computer, or push changes made on the VM back to the main repository.\n\nOverall, integrating Git into your workflow can enhance your coding experience while working in a headless environment. If you decide to go for it, I can help with the initial setup or any Git-related questions you might have!"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "so is git like a compressed version of a directory?"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "Not exactly, but that's a good way to think about some of its features! Here's a more precise breakdown:\n\n1. **Snapshots, Not Compression**: Git takes \"snapshots\" of your files at different points in time, meaning it records a full version of your directory's contents every time you make a commit. While this does involve storing the files, it doesn't compress them in the traditional sense—Git keeps track of changes efficiently, so only the differences between versions are stored after the first snapshot.\n\n2. **Directory Structure**: Git is designed to track changes in your source code, including the entire directory structure, file contents, and metadata. It allows you to manage multiple versions of a directory and the files within it.\n\n3. **Branching and Merging**: Git allows you to create branches, which can be thought of as \"parallel versions\" of your directory. Each branch can have different sets of changes, and you can merge these branches to integrate different features or fixes.\n\n4. **History Tracking**: Git also maintains a history of changes, so you can navigate through previous versions of your directory. This includes seeing what changes were made, who made them, and when they occurred.\n\nThe combination of these features makes Git a powerful tool for version control, allowing you to manage your projects and code changes more effectively. If you're curious about any specific aspect of Git or how it works, feel free to ask!"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "hmm i still dont see the point of it. atleast not for this project. This project is mostly about learning how shit works, im not planning to push this as a product or collab with anyone"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "That makes total sense! If this project is primarily for learning and experimentation, you might not need the full functionality of Git right now. Many developers start without it, especially when they’re working on smaller projects or learning something new.\n\nHere are a few reasons you might still consider using Git, even for personal learning projects:\n\n1. **Tracking Changes**: It can help you understand what changes you made over time. You can easily go back to a previous point if something breaks or if you want to see how you arrived at your current solution.\n\n2. **Learning Opportunity**: Getting familiar with Git now can make it easier when you start working on larger projects or collaborating in the future. It's a valuable skill in the software development world.\n\n3. **Experimentation**: If you're trying out different ideas, branches can help you manage experiments without affecting your main work.\n\nHowever, if you're comfortable managing your changes manually and don’t see the benefit for this particular project, that’s completely okay! Focus on what helps you learn best. If you have specific goals or areas you want to explore, I’m here to help with that too!"
|
||||
}
|
||||
]
|
||||
@@ -1,6 +0,0 @@
|
||||
{"ts":"2025-11-12T20:18:06.822Z","turn":[{"role":"user","content":"i have everything syncing back to my main dev computer, i just do it manually. i have no idea how to use git"},{"role":"assistant","content":"Manually syncing is a good start, but learning Git can really help streamline your workflow and keep track of changes. If you're interested, I can provide some resources or tips on how to get started with Git! It can make managing different versions of your code much easier."}]}
|
||||
{"ts":"2025-11-12T20:29:20.003Z","turn":[{"role":"user","content":"what is the point of git? how can it streamline my work?"},{"role":"assistant","content":"Git is a version control system that helps you keep track of changes in your code over time. Here are some ways it can streamline your work:\n\n1. **Version Tracking**: Git allows you to save versions of your code so you can revert back to previous versions if needed. This is especially helpful if you make a change that breaks something.\n\n2. **Collaboration**: If you ever work with others, Git makes it easy to collaborate on projects. You can work on separate branches, merge changes, and resolve conflicts.\n\n3. **Branching**: You can create branches for new features or experiments without affecting the main codebase. Once you're satisfied with the changes, you can merge them back.\n\n4. **Snapshotting**: Git takes snapshots of your code at different points in time, so you can easily see what your code looked like at any point.\n\n5. **Easy Backup**: By pushing your code to a remote repository (like GitHub or GitLab), you can back up your work and access it from anywhere.\n\nLearning Git might take a bit of time, but it can greatly enhance your coding efficiency and help you manage your projects better. If you're interested, I can point you to some beginner-friendly tutorials!"}]}
|
||||
{"ts":"2025-11-12T20:30:55.329Z","turn":[{"role":"user","content":"say i am writing one program on my main computer, and then im working on one that lives in a VM. Can i put them both in the same git?"},{"role":"assistant","content":"Yes, you can use the same Git repository for your code on both your main computer and the VM. Here’s how you can do it:\n\n1. **Create a Git Repository**: You can create a Git repository on your main computer for your program. Use the command `git init` in your project's directory to initialize it.\n\n2. **Commit Changes**: As you make changes to your code on your main computer, you can use `git add .` to stage changes and `git commit -m \"your message\"` to commit them.\n\n3. **Clone the Repository on your VM**: You can then clone the Git repository on your VM using `git clone <repository-url>`. This gives you a copy of the same repository on your VM.\n\n4. **Work on Both**: You can now work on either machine. When you make changes on one, commit those changes, and then pull the latest changes from the other machine to keep everything in sync.\n\n5. **Push and Pull**: Use `git push` to push your changes to a remote repository (like GitHub) if you're using one. Use `git pull` on the other machine to pull the latest code.\n\nJust be careful about making sure you commit and push your changes before switching between machines to avoid conflicts. This setup allows you to maintain a single version of your code across both systems. If you need help with specific commands or setups, just let me know!"}]}
|
||||
{"ts":"2025-11-12T20:33:16.511Z","turn":[{"role":"user","content":"hmm i guess that might be useful. Im writting code that is running on a VM. I want to test and experiment while i write the code. The VM is headless though so i've been doing all my editing and work via SSH and SFTP."},{"role":"assistant","content":"Using a VM and working through SSH and SFTP is a solid way to manage your coding environment! Git can definitely be beneficial in this setup as well. Here’s how it can help:\n\n1. **Easier Code Management**: With Git, you can track changes to your code directly within the VM, making it easier to manage versions as you experiment.\n\n2. **Experiment Branches**: You can create branches for different experiments. This lets you test new features or fixes without affecting your main code, and you can easily switch back and forth between branches.\n\n3. **Collaborative Testing**: If you ever work with others or want to test things collaboratively, you can push changes to a remote repository, making it easy for others to pull and test the latest code.\n\n4. **Integrated with SSH**: You can run Git commands directly from your SSH terminal on the VM, making it straightforward to use. You won’t need any additional tools.\n\n5. **Seamless Remote Access**: Since you’re using SFTP, you can clone the repository to your VM. This way, you can easily pull updates you've made on your main computer, or push changes made on the VM back to the main repository.\n\nOverall, integrating Git into your workflow can enhance your coding experience while working in a headless environment. If you decide to go for it, I can help with the initial setup or any Git-related questions you might have!"}]}
|
||||
{"ts":"2025-11-12T20:34:30.196Z","turn":[{"role":"user","content":"so is git like a compressed version of a directory?"},{"role":"assistant","content":"Not exactly, but that's a good way to think about some of its features! Here's a more precise breakdown:\n\n1. **Snapshots, Not Compression**: Git takes \"snapshots\" of your files at different points in time, meaning it records a full version of your directory's contents every time you make a commit. While this does involve storing the files, it doesn't compress them in the traditional sense—Git keeps track of changes efficiently, so only the differences between versions are stored after the first snapshot.\n\n2. **Directory Structure**: Git is designed to track changes in your source code, including the entire directory structure, file contents, and metadata. It allows you to manage multiple versions of a directory and the files within it.\n\n3. **Branching and Merging**: Git allows you to create branches, which can be thought of as \"parallel versions\" of your directory. Each branch can have different sets of changes, and you can merge these branches to integrate different features or fixes.\n\n4. **History Tracking**: Git also maintains a history of changes, so you can navigate through previous versions of your directory. This includes seeing what changes were made, who made them, and when they occurred.\n\nThe combination of these features makes Git a powerful tool for version control, allowing you to manage your projects and code changes more effectively. If you're curious about any specific aspect of Git or how it works, feel free to ask!"}]}
|
||||
{"ts":"2025-11-12T20:40:33.207Z","turn":[{"role":"user","content":"hmm i still dont see the point of it. atleast not for this project. This project is mostly about learning how shit works, im not planning to push this as a product or collab with anyone"},{"role":"assistant","content":"That makes total sense! If this project is primarily for learning and experimentation, you might not need the full functionality of Git right now. Many developers start without it, especially when they’re working on smaller projects or learning something new.\n\nHere are a few reasons you might still consider using Git, even for personal learning projects:\n\n1. **Tracking Changes**: It can help you understand what changes you made over time. You can easily go back to a previous point if something breaks or if you want to see how you arrived at your current solution.\n\n2. **Learning Opportunity**: Getting familiar with Git now can make it easier when you start working on larger projects or collaborating in the future. It's a valuable skill in the software development world.\n\n3. **Experimentation**: If you're trying out different ideas, branches can help you manage experiments without affecting your main work.\n\nHowever, if you're comfortable managing your changes manually and don’t see the benefit for this particular project, that’s completely okay! Focus on what helps you learn best. If you have specific goals or areas you want to explore, I’m here to help with that too!"}]}
|
||||
@@ -1,39 +0,0 @@
|
||||
// test-llm.js
|
||||
import path from "path";
|
||||
import { fileURLToPath } from "url";
|
||||
import dotenv from "dotenv";
|
||||
import { callSpeechLLM } from "./lib/llm.js";
|
||||
|
||||
// ───────────────────────────────────────────────
|
||||
// 🔧 Load environment
|
||||
// ───────────────────────────────────────────────
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
const __dirname = path.dirname(__filename);
|
||||
const envPath = path.join(__dirname, "../.env");
|
||||
dotenv.config({ path: envPath });
|
||||
|
||||
console.log("🔧 Using .env from:", envPath);
|
||||
console.log("🔧 LLM_FORCE_BACKEND =", process.env.LLM_FORCE_BACKEND);
|
||||
console.log("🔧 LLM_PRIMARY_URL =", process.env.LLM_PRIMARY_URL);
|
||||
|
||||
// ───────────────────────────────────────────────
|
||||
// 🧪 Run a simple test message
|
||||
// ───────────────────────────────────────────────
|
||||
async function testLLM() {
|
||||
console.log("🧪 Testing LLM helper...");
|
||||
|
||||
const messages = [
|
||||
{ role: "user", content: "Say hello in five words or less." }
|
||||
];
|
||||
|
||||
try {
|
||||
const { reply, backend } = await callSpeechLLM(messages);
|
||||
|
||||
console.log(`✅ Reply: ${reply || "[no reply]"}`);
|
||||
console.log(`Backend used: ${backend || "[unknown]"}`);
|
||||
} catch (err) {
|
||||
console.error("💥 Test failed:", err.message);
|
||||
}
|
||||
}
|
||||
|
||||
testLLM();
|
||||
+18
-29
@@ -1,3 +1,6 @@
|
||||
// relay v0.3.0
|
||||
// Core relay server for Lyra project
|
||||
// Handles incoming chat requests and forwards them to Cortex services
|
||||
import express from "express";
|
||||
import dotenv from "dotenv";
|
||||
import cors from "cors";
|
||||
@@ -10,10 +13,8 @@ app.use(express.json());
|
||||
|
||||
const PORT = Number(process.env.PORT || 7078);
|
||||
|
||||
// core endpoints
|
||||
// Cortex endpoints (only these are used now)
|
||||
const CORTEX_REASON = process.env.CORTEX_REASON_URL || "http://cortex:7081/reason";
|
||||
const CORTEX_INGEST = process.env.CORTEX_INGEST_URL || "http://cortex:7081/ingest";
|
||||
const INTAKE_URL = process.env.INTAKE_URL || "http://intake:7080/add_exchange";
|
||||
|
||||
// -----------------------------------------------------
|
||||
// Helper request wrapper
|
||||
@@ -42,11 +43,12 @@ async function postJSON(url, data) {
|
||||
}
|
||||
|
||||
// -----------------------------------------------------
|
||||
// Shared chat handler logic
|
||||
// The unified chat handler
|
||||
// -----------------------------------------------------
|
||||
async function handleChatRequest(session_id, user_msg) {
|
||||
// 1. → Cortex.reason
|
||||
let reason;
|
||||
|
||||
// 1. → Cortex.reason (main pipeline)
|
||||
try {
|
||||
reason = await postJSON(CORTEX_REASON, {
|
||||
session_id,
|
||||
@@ -57,23 +59,13 @@ async function handleChatRequest(session_id, user_msg) {
|
||||
throw new Error(`cortex_reason_failed: ${e.message}`);
|
||||
}
|
||||
|
||||
const persona = reason.final_output || reason.persona || "(no persona text)";
|
||||
// Correct persona field
|
||||
const persona =
|
||||
reason.persona ||
|
||||
reason.final_output ||
|
||||
"(no persona text)";
|
||||
|
||||
// 2. → Cortex.ingest (async, non-blocking)
|
||||
postJSON(CORTEX_INGEST, {
|
||||
session_id,
|
||||
user_msg,
|
||||
assistant_msg: persona
|
||||
}).catch(e => console.warn("Relay → Cortex.ingest failed:", e.message));
|
||||
|
||||
// 3. → Intake summary (async, non-blocking)
|
||||
postJSON(INTAKE_URL, {
|
||||
session_id,
|
||||
user_msg,
|
||||
assistant_msg: persona
|
||||
}).catch(e => console.warn("Relay → Intake failed:", e.message));
|
||||
|
||||
// 4. Return result
|
||||
// Return final answer
|
||||
return {
|
||||
session_id,
|
||||
reply: persona
|
||||
@@ -88,11 +80,10 @@ app.get("/_health", (_, res) => {
|
||||
});
|
||||
|
||||
// -----------------------------------------------------
|
||||
// OPENAI-COMPATIBLE ENDPOINT (for UI)
|
||||
// OPENAI-COMPATIBLE ENDPOINT
|
||||
// -----------------------------------------------------
|
||||
app.post("/v1/chat/completions", async (req, res) => {
|
||||
try {
|
||||
// Extract from OpenAI format
|
||||
const session_id = req.body.session_id || req.body.user || "default";
|
||||
const messages = req.body.messages || [];
|
||||
const lastMessage = messages[messages.length - 1];
|
||||
@@ -104,11 +95,9 @@ app.post("/v1/chat/completions", async (req, res) => {
|
||||
|
||||
console.log(`Relay (v1) → received: "${user_msg}"`);
|
||||
|
||||
// Call the same logic as /chat
|
||||
const result = await handleChatRequest(session_id, user_msg);
|
||||
|
||||
// Return in OpenAI format
|
||||
return res.json({
|
||||
res.json({
|
||||
id: `chatcmpl-${Date.now()}`,
|
||||
object: "chat.completion",
|
||||
created: Math.floor(Date.now() / 1000),
|
||||
@@ -129,7 +118,7 @@ app.post("/v1/chat/completions", async (req, res) => {
|
||||
});
|
||||
|
||||
} catch (err) {
|
||||
console.error("Relay v1 endpoint fatal:", err);
|
||||
console.error("Relay v1 fatal:", err);
|
||||
res.status(500).json({
|
||||
error: {
|
||||
message: err.message || String(err),
|
||||
@@ -141,7 +130,7 @@ app.post("/v1/chat/completions", async (req, res) => {
|
||||
});
|
||||
|
||||
// -----------------------------------------------------
|
||||
// MAIN ENDPOINT (new canonical)
|
||||
// MAIN ENDPOINT (Lyra-native UI)
|
||||
// -----------------------------------------------------
|
||||
app.post("/chat", async (req, res) => {
|
||||
try {
|
||||
@@ -151,7 +140,7 @@ app.post("/chat", async (req, res) => {
|
||||
console.log(`Relay → received: "${user_msg}"`);
|
||||
|
||||
const result = await handleChatRequest(session_id, user_msg);
|
||||
return res.json(result);
|
||||
res.json(result);
|
||||
|
||||
} catch (err) {
|
||||
console.error("Relay fatal:", err);
|
||||
|
||||
@@ -4,4 +4,6 @@ COPY requirements.txt .
|
||||
RUN pip install -r requirements.txt
|
||||
COPY . .
|
||||
EXPOSE 7081
|
||||
# NOTE: Running with single worker to maintain SESSIONS global state in Intake.
|
||||
# If scaling to multiple workers, migrate SESSIONS to Redis or shared storage.
|
||||
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "7081"]
|
||||
|
||||
@@ -0,0 +1,448 @@
|
||||
# context.py
|
||||
"""
|
||||
Context layer for Cortex reasoning pipeline.
|
||||
|
||||
Provides unified context collection from:
|
||||
- Intake (short-term memory, multilevel summaries L1-L30)
|
||||
- NeoMem (long-term memory, semantic search)
|
||||
- Session state (timestamps, messages, mode, mood, active_project)
|
||||
|
||||
Maintains per-session state for continuity across conversations.
|
||||
"""
|
||||
|
||||
import os
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Dict, Any, Optional, List
|
||||
import httpx
|
||||
from intake.intake import summarize_context
|
||||
|
||||
|
||||
from neomem_client import NeoMemClient
|
||||
|
||||
# -----------------------------
|
||||
# Configuration
|
||||
# -----------------------------
|
||||
NEOMEM_API = os.getenv("NEOMEM_API", "http://neomem-api:8000")
|
||||
RELEVANCE_THRESHOLD = float(os.getenv("RELEVANCE_THRESHOLD", "0.4"))
|
||||
VERBOSE_DEBUG = os.getenv("VERBOSE_DEBUG", "false").lower() == "true"
|
||||
|
||||
# Tools available for future autonomy features
|
||||
TOOLS_AVAILABLE = ["RAG", "WEB", "WEATHER", "CODEBRAIN", "POKERBRAIN"]
|
||||
|
||||
# -----------------------------
|
||||
# Module-level session state
|
||||
# -----------------------------
|
||||
SESSION_STATE: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
# Logger
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Set logging level based on VERBOSE_DEBUG
|
||||
if VERBOSE_DEBUG:
|
||||
logger.setLevel(logging.DEBUG)
|
||||
|
||||
# Console handler
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [CONTEXT] %(levelname)s: %(message)s',
|
||||
datefmt='%H:%M:%S'
|
||||
))
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
# File handler - append to log file
|
||||
try:
|
||||
os.makedirs('/app/logs', exist_ok=True)
|
||||
file_handler = logging.FileHandler('/app/logs/cortex_verbose_debug.log', mode='a')
|
||||
file_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [CONTEXT] %(levelname)s: %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
))
|
||||
logger.addHandler(file_handler)
|
||||
logger.debug("VERBOSE_DEBUG mode enabled for context.py - logging to file")
|
||||
except Exception as e:
|
||||
logger.debug(f"VERBOSE_DEBUG mode enabled for context.py - file logging failed: {e}")
|
||||
|
||||
|
||||
# -----------------------------
|
||||
# Session initialization
|
||||
# -----------------------------
|
||||
def _init_session(session_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Initialize a new session state entry.
|
||||
|
||||
Returns:
|
||||
Dictionary with default session state fields
|
||||
"""
|
||||
return {
|
||||
"session_id": session_id,
|
||||
"created_at": datetime.now(),
|
||||
"last_timestamp": datetime.now(),
|
||||
"last_user_message": None,
|
||||
"last_assistant_message": None,
|
||||
"mode": "default", # Future: "autonomous", "focused", "creative", etc.
|
||||
"mood": "neutral", # Future: mood tracking
|
||||
"active_project": None, # Future: project context
|
||||
"message_count": 0,
|
||||
"message_history": [],
|
||||
}
|
||||
|
||||
|
||||
# -----------------------------
|
||||
# Intake context retrieval
|
||||
# -----------------------------
|
||||
async def _get_intake_context(session_id: str, messages: List[Dict[str, str]]):
|
||||
"""
|
||||
Internal Intake — Direct call to summarize_context()
|
||||
No HTTP, no containers, no failures.
|
||||
"""
|
||||
try:
|
||||
return await summarize_context(session_id, messages)
|
||||
except Exception as e:
|
||||
logger.error(f"Internal Intake summarization failed: {e}")
|
||||
return {
|
||||
"session_id": session_id,
|
||||
"L1": "",
|
||||
"L5": "",
|
||||
"L10": "",
|
||||
"L20": "",
|
||||
"L30": "",
|
||||
"error": str(e)
|
||||
}
|
||||
|
||||
|
||||
|
||||
# -----------------------------
|
||||
# NeoMem semantic search
|
||||
# -----------------------------
|
||||
async def _search_neomem(
|
||||
query: str,
|
||||
user_id: str = "brian",
|
||||
limit: int = 5
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Search NeoMem for relevant long-term memories.
|
||||
|
||||
Returns full response structure from NeoMem:
|
||||
[
|
||||
{
|
||||
"id": "mem_abc123",
|
||||
"score": 0.92,
|
||||
"payload": {
|
||||
"data": "Memory text content...",
|
||||
"metadata": {
|
||||
"category": "...",
|
||||
"created_at": "...",
|
||||
...
|
||||
}
|
||||
}
|
||||
},
|
||||
...
|
||||
]
|
||||
|
||||
Args:
|
||||
query: Search query text
|
||||
user_id: User identifier for memory filtering
|
||||
limit: Maximum number of results
|
||||
|
||||
Returns:
|
||||
List of memory objects with full structure, or empty list on failure
|
||||
"""
|
||||
try:
|
||||
# NeoMemClient reads NEOMEM_API from environment, no base_url parameter
|
||||
client = NeoMemClient()
|
||||
results = await client.search(
|
||||
query=query,
|
||||
user_id=user_id,
|
||||
limit=limit,
|
||||
threshold=RELEVANCE_THRESHOLD
|
||||
)
|
||||
|
||||
# Results are already filtered by threshold in NeoMemClient.search()
|
||||
logger.info(f"NeoMem search returned {len(results)} relevant results")
|
||||
return results
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f"NeoMem search failed: {e}")
|
||||
return []
|
||||
|
||||
|
||||
# -----------------------------
|
||||
# Main context collection
|
||||
# -----------------------------
|
||||
async def collect_context(session_id: str, user_prompt: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Collect unified context from all sources.
|
||||
|
||||
Orchestrates:
|
||||
1. Initialize or update session state
|
||||
2. Calculate time since last message
|
||||
3. Retrieve Intake multilevel summaries (L1-L30)
|
||||
4. Search NeoMem for relevant long-term memories
|
||||
5. Update session state with current user message
|
||||
6. Return unified context_state dictionary
|
||||
|
||||
Args:
|
||||
session_id: Session identifier
|
||||
user_prompt: Current user message
|
||||
|
||||
Returns:
|
||||
Unified context state dictionary with structure:
|
||||
{
|
||||
"session_id": "...",
|
||||
"timestamp": "2025-11-28T12:34:56",
|
||||
"minutes_since_last_msg": 5.2,
|
||||
"message_count": 42,
|
||||
"intake": {
|
||||
"L1": [...],
|
||||
"L5": [...],
|
||||
"L10": {...},
|
||||
"L20": {...},
|
||||
"L30": {...}
|
||||
},
|
||||
"rag": [
|
||||
{
|
||||
"id": "mem_123",
|
||||
"score": 0.92,
|
||||
"payload": {
|
||||
"data": "...",
|
||||
"metadata": {...}
|
||||
}
|
||||
},
|
||||
...
|
||||
],
|
||||
"mode": "default",
|
||||
"mood": "neutral",
|
||||
"active_project": null,
|
||||
"tools_available": ["RAG", "WEB", "WEATHER", "CODEBRAIN", "POKERBRAIN"]
|
||||
}
|
||||
"""
|
||||
|
||||
# A. Initialize session state if needed
|
||||
if session_id not in SESSION_STATE:
|
||||
SESSION_STATE[session_id] = _init_session(session_id)
|
||||
logger.info(f"Initialized new session: {session_id}")
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[COLLECT_CONTEXT] New session state: {SESSION_STATE[session_id]}")
|
||||
|
||||
state = SESSION_STATE[session_id]
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[COLLECT_CONTEXT] Session {session_id} - User prompt: {user_prompt[:100]}...")
|
||||
|
||||
# B. Calculate time delta
|
||||
now = datetime.now()
|
||||
time_delta_seconds = (now - state["last_timestamp"]).total_seconds()
|
||||
minutes_since_last_msg = round(time_delta_seconds / 60.0, 2)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[COLLECT_CONTEXT] Time since last message: {minutes_since_last_msg:.2f} minutes")
|
||||
|
||||
# C. Gather Intake context (multilevel summaries)
|
||||
# Build compact message buffer for Intake:
|
||||
messages_for_intake = []
|
||||
|
||||
# You track messages inside SESSION_STATE — assemble it here:
|
||||
if "message_history" in state:
|
||||
for turn in state["message_history"]:
|
||||
messages_for_intake.append({
|
||||
"user_msg": turn.get("user", ""),
|
||||
"assistant_msg": turn.get("assistant", "")
|
||||
})
|
||||
|
||||
intake_data = await _get_intake_context(session_id, messages_for_intake)
|
||||
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
import json
|
||||
logger.debug(f"[COLLECT_CONTEXT] Intake data retrieved:")
|
||||
logger.debug(json.dumps(intake_data, indent=2, default=str))
|
||||
|
||||
# D. Search NeoMem for relevant memories
|
||||
rag_results = await _search_neomem(
|
||||
query=user_prompt,
|
||||
user_id="brian", # TODO: Make configurable per session
|
||||
limit=5
|
||||
)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[COLLECT_CONTEXT] NeoMem search returned {len(rag_results)} results")
|
||||
for idx, result in enumerate(rag_results, 1):
|
||||
score = result.get("score", 0)
|
||||
data_preview = str(result.get("payload", {}).get("data", ""))[:100]
|
||||
logger.debug(f" [{idx}] Score: {score:.3f} - {data_preview}...")
|
||||
|
||||
# E. Update session state
|
||||
state["last_user_message"] = user_prompt
|
||||
state["last_timestamp"] = now
|
||||
state["message_count"] += 1
|
||||
# Save user turn to history
|
||||
state["message_history"].append({
|
||||
"user": user_prompt,
|
||||
"assistant": "" # assistant reply filled later by update_last_assistant_message()
|
||||
})
|
||||
|
||||
|
||||
|
||||
# F. Assemble unified context
|
||||
context_state = {
|
||||
"session_id": session_id,
|
||||
"timestamp": now.isoformat(),
|
||||
"minutes_since_last_msg": minutes_since_last_msg,
|
||||
"message_count": state["message_count"],
|
||||
"intake": intake_data,
|
||||
"rag": rag_results,
|
||||
"mode": state["mode"],
|
||||
"mood": state["mood"],
|
||||
"active_project": state["active_project"],
|
||||
"tools_available": TOOLS_AVAILABLE,
|
||||
}
|
||||
|
||||
logger.info(
|
||||
f"Context collected for session {session_id}: "
|
||||
f"{len(rag_results)} RAG results, "
|
||||
f"{minutes_since_last_msg:.1f} minutes since last message"
|
||||
)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[COLLECT_CONTEXT] Final context state assembled:")
|
||||
logger.debug(f" - Message count: {state['message_count']}")
|
||||
logger.debug(f" - Mode: {state['mode']}, Mood: {state['mood']}")
|
||||
logger.debug(f" - Active project: {state['active_project']}")
|
||||
logger.debug(f" - Tools available: {TOOLS_AVAILABLE}")
|
||||
|
||||
return context_state
|
||||
|
||||
|
||||
# -----------------------------
|
||||
# Session state management
|
||||
# -----------------------------
|
||||
def update_last_assistant_message(session_id: str, message: str) -> None:
|
||||
"""
|
||||
Update session state with assistant's response and complete
|
||||
the last turn inside message_history.
|
||||
"""
|
||||
session = SESSION_STATE.get(session_id)
|
||||
if not session:
|
||||
logger.warning(f"Attempted to update non-existent session: {session_id}")
|
||||
return
|
||||
|
||||
# Update last assistant message + timestamp
|
||||
session["last_assistant_message"] = message
|
||||
session["last_timestamp"] = datetime.now()
|
||||
|
||||
# Fill in assistant reply for the most recent turn
|
||||
history = session.get("message_history", [])
|
||||
if history:
|
||||
# history entry already contains {"user": "...", "assistant": "...?"}
|
||||
history[-1]["assistant"] = message
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"Updated assistant message for session {session_id}")
|
||||
|
||||
|
||||
|
||||
def get_session_state(session_id: str) -> Optional[Dict[str, Any]]:
|
||||
"""
|
||||
Retrieve current session state.
|
||||
|
||||
Args:
|
||||
session_id: Session identifier
|
||||
|
||||
Returns:
|
||||
Session state dict or None if session doesn't exist
|
||||
"""
|
||||
return SESSION_STATE.get(session_id)
|
||||
|
||||
|
||||
def close_session(session_id: str) -> bool:
|
||||
"""
|
||||
Close and cleanup a session.
|
||||
|
||||
Args:
|
||||
session_id: Session identifier
|
||||
|
||||
Returns:
|
||||
True if session was closed, False if it didn't exist
|
||||
"""
|
||||
if session_id in SESSION_STATE:
|
||||
del SESSION_STATE[session_id]
|
||||
logger.info(f"Closed session: {session_id}")
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# -----------------------------
|
||||
# Extension hooks for future autonomy
|
||||
# -----------------------------
|
||||
def update_mode(session_id: str, new_mode: str) -> None:
|
||||
"""
|
||||
Update session mode.
|
||||
|
||||
Future modes: "autonomous", "focused", "creative", "collaborative", etc.
|
||||
|
||||
Args:
|
||||
session_id: Session identifier
|
||||
new_mode: New mode string
|
||||
"""
|
||||
if session_id in SESSION_STATE:
|
||||
old_mode = SESSION_STATE[session_id]["mode"]
|
||||
SESSION_STATE[session_id]["mode"] = new_mode
|
||||
logger.info(f"Session {session_id} mode changed: {old_mode} -> {new_mode}")
|
||||
|
||||
|
||||
def update_mood(session_id: str, new_mood: str) -> None:
|
||||
"""
|
||||
Update session mood.
|
||||
|
||||
Future implementation: Sentiment analysis, emotional state tracking.
|
||||
|
||||
Args:
|
||||
session_id: Session identifier
|
||||
new_mood: New mood string
|
||||
"""
|
||||
if session_id in SESSION_STATE:
|
||||
old_mood = SESSION_STATE[session_id]["mood"]
|
||||
SESSION_STATE[session_id]["mood"] = new_mood
|
||||
logger.info(f"Session {session_id} mood changed: {old_mood} -> {new_mood}")
|
||||
|
||||
|
||||
def update_active_project(session_id: str, project: Optional[str]) -> None:
|
||||
"""
|
||||
Update active project context.
|
||||
|
||||
Future implementation: Project-specific memory, tools, preferences.
|
||||
|
||||
Args:
|
||||
session_id: Session identifier
|
||||
project: Project identifier or None
|
||||
"""
|
||||
if session_id in SESSION_STATE:
|
||||
SESSION_STATE[session_id]["active_project"] = project
|
||||
logger.info(f"Session {session_id} active project set to: {project}")
|
||||
|
||||
|
||||
async def autonomous_heartbeat(session_id: str) -> Optional[str]:
|
||||
"""
|
||||
Autonomous thinking heartbeat.
|
||||
|
||||
Future implementation:
|
||||
- Check if Lyra should initiate internal dialogue
|
||||
- Generate self-prompted thoughts based on session state
|
||||
- Update mood/mode based on context changes
|
||||
- Trigger proactive suggestions or reminders
|
||||
|
||||
Args:
|
||||
session_id: Session identifier
|
||||
|
||||
Returns:
|
||||
Optional autonomous thought/action string
|
||||
"""
|
||||
# Stub for future implementation
|
||||
# Example logic:
|
||||
# - If minutes_since_last_msg > 60: Check for pending reminders
|
||||
# - If mood == "curious" and active_project: Generate research questions
|
||||
# - If mode == "autonomous": Self-prompt based on project goals
|
||||
|
||||
logger.debug(f"Autonomous heartbeat for session {session_id} (not yet implemented)")
|
||||
return None
|
||||
@@ -0,0 +1,18 @@
|
||||
"""
|
||||
Intake module - short-term memory summarization.
|
||||
|
||||
Runs inside the Cortex container as a pure Python module.
|
||||
No standalone API server - called internally by Cortex.
|
||||
"""
|
||||
|
||||
from .intake import (
|
||||
SESSIONS,
|
||||
add_exchange_internal,
|
||||
summarize_context,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"SESSIONS",
|
||||
"add_exchange_internal",
|
||||
"summarize_context",
|
||||
]
|
||||
@@ -0,0 +1,366 @@
|
||||
import os
|
||||
import json
|
||||
from datetime import datetime
|
||||
from typing import List, Dict, Any, TYPE_CHECKING
|
||||
from collections import deque
|
||||
from llm.llm_router import call_llm
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# Global Short-Term Memory (new Intake)
|
||||
# -------------------------------------------------------------------
|
||||
SESSIONS: dict[str, dict] = {} # session_id → { buffer: deque, created_at: timestamp }
|
||||
|
||||
# Diagnostic: Verify module loads only once
|
||||
print(f"[Intake Module Init] SESSIONS object id: {id(SESSIONS)}, module: {__name__}")
|
||||
|
||||
# L10 / L20 history lives here too
|
||||
L10_HISTORY: Dict[str, list[str]] = {}
|
||||
L20_HISTORY: Dict[str, list[str]] = {}
|
||||
|
||||
from llm.llm_router import call_llm # Use Cortex's shared LLM router
|
||||
|
||||
if TYPE_CHECKING:
|
||||
# Only for type hints — do NOT redefine SESSIONS here
|
||||
from collections import deque as _deque
|
||||
def bg_summarize(session_id: str) -> None: ...
|
||||
|
||||
# ─────────────────────────────
|
||||
# Config
|
||||
# ─────────────────────────────
|
||||
|
||||
INTAKE_LLM = os.getenv("INTAKE_LLM", "PRIMARY").upper()
|
||||
|
||||
SUMMARY_MAX_TOKENS = int(os.getenv("SUMMARY_MAX_TOKENS", "200"))
|
||||
SUMMARY_TEMPERATURE = float(os.getenv("SUMMARY_TEMPERATURE", "0.3"))
|
||||
|
||||
NEOMEM_API = os.getenv("NEOMEM_API")
|
||||
NEOMEM_KEY = os.getenv("NEOMEM_KEY")
|
||||
|
||||
# ─────────────────────────────
|
||||
# Internal history for L10/L20/L30
|
||||
# ─────────────────────────────
|
||||
|
||||
L10_HISTORY: Dict[str, list[str]] = {} # session_id → list of L10 blocks
|
||||
L20_HISTORY: Dict[str, list[str]] = {} # session_id → list of merged overviews
|
||||
|
||||
|
||||
# ─────────────────────────────
|
||||
# LLM helper (via Cortex router)
|
||||
# ─────────────────────────────
|
||||
|
||||
async def _llm(prompt: str) -> str:
|
||||
"""
|
||||
Use Cortex's llm_router to run a summary prompt.
|
||||
"""
|
||||
try:
|
||||
text = await call_llm(
|
||||
prompt,
|
||||
backend=INTAKE_LLM,
|
||||
temperature=SUMMARY_TEMPERATURE,
|
||||
max_tokens=SUMMARY_MAX_TOKENS,
|
||||
)
|
||||
return (text or "").strip()
|
||||
except Exception as e:
|
||||
return f"[Error summarizing: {e}]"
|
||||
|
||||
|
||||
# ─────────────────────────────
|
||||
# Formatting helpers
|
||||
# ─────────────────────────────
|
||||
|
||||
def _format_exchanges(exchanges: List[Dict[str, Any]]) -> str:
|
||||
"""
|
||||
Expect each exchange to look like:
|
||||
{ "user_msg": "...", "assistant_msg": "..." }
|
||||
"""
|
||||
chunks = []
|
||||
for e in exchanges:
|
||||
user = e.get("user_msg", "")
|
||||
assistant = e.get("assistant_msg", "")
|
||||
chunks.append(f"User: {user}\nAssistant: {assistant}\n")
|
||||
return "\n".join(chunks)
|
||||
|
||||
|
||||
# ─────────────────────────────
|
||||
# Base factual summary
|
||||
# ─────────────────────────────
|
||||
|
||||
async def summarize_simple(exchanges: List[Dict[str, Any]]) -> str:
|
||||
"""
|
||||
Simple factual summary of recent exchanges.
|
||||
"""
|
||||
if not exchanges:
|
||||
return ""
|
||||
|
||||
text = _format_exchanges(exchanges)
|
||||
|
||||
prompt = f"""
|
||||
Summarize the following conversation between Brian (user) and Lyra (assistant).
|
||||
Focus only on factual content. Avoid names, examples, story tone, or invented details.
|
||||
|
||||
{text}
|
||||
|
||||
Summary:
|
||||
"""
|
||||
return await _llm(prompt)
|
||||
|
||||
|
||||
# ─────────────────────────────
|
||||
# Multilevel Summaries (L1, L5, L10, L20, L30)
|
||||
# ─────────────────────────────
|
||||
|
||||
async def summarize_L1(buf: List[Dict[str, Any]]) -> str:
|
||||
# Last ~5 exchanges
|
||||
return await summarize_simple(buf[-5:])
|
||||
|
||||
|
||||
async def summarize_L5(buf: List[Dict[str, Any]]) -> str:
|
||||
# Last ~10 exchanges
|
||||
return await summarize_simple(buf[-10:])
|
||||
|
||||
|
||||
async def summarize_L10(session_id: str, buf: List[Dict[str, Any]]) -> str:
|
||||
# “Reality Check” for last 10 exchanges
|
||||
text = _format_exchanges(buf[-10:])
|
||||
|
||||
prompt = f"""
|
||||
You are Lyra Intake performing a short 'Reality Check'.
|
||||
Summarize the last block of conversation (up to 10 exchanges)
|
||||
in one clear paragraph focusing on tone, intent, and direction.
|
||||
|
||||
{text}
|
||||
|
||||
Reality Check:
|
||||
"""
|
||||
summary = await _llm(prompt)
|
||||
|
||||
# Track history for this session
|
||||
L10_HISTORY.setdefault(session_id, [])
|
||||
L10_HISTORY[session_id].append(summary)
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
async def summarize_L20(session_id: str) -> str:
|
||||
"""
|
||||
Merge all L10 Reality Checks into a 'Session Overview'.
|
||||
"""
|
||||
history = L10_HISTORY.get(session_id, [])
|
||||
joined = "\n\n".join(history) if history else ""
|
||||
|
||||
if not joined:
|
||||
return ""
|
||||
|
||||
prompt = f"""
|
||||
You are Lyra Intake creating a 'Session Overview'.
|
||||
Merge the following Reality Check paragraphs into one short summary
|
||||
capturing progress, themes, and the direction of the conversation.
|
||||
|
||||
{joined}
|
||||
|
||||
Overview:
|
||||
"""
|
||||
summary = await _llm(prompt)
|
||||
|
||||
L20_HISTORY.setdefault(session_id, [])
|
||||
L20_HISTORY[session_id].append(summary)
|
||||
|
||||
return summary
|
||||
|
||||
|
||||
async def summarize_L30(session_id: str) -> str:
|
||||
"""
|
||||
Merge all L20 session overviews into a 'Continuity Report'.
|
||||
"""
|
||||
history = L20_HISTORY.get(session_id, [])
|
||||
joined = "\n\n".join(history) if history else ""
|
||||
|
||||
if not joined:
|
||||
return ""
|
||||
|
||||
prompt = f"""
|
||||
You are Lyra Intake generating a 'Continuity Report'.
|
||||
Condense these session overviews into one high-level reflection,
|
||||
noting major themes, persistent goals, and shifts.
|
||||
|
||||
{joined}
|
||||
|
||||
Continuity Report:
|
||||
"""
|
||||
return await _llm(prompt)
|
||||
|
||||
|
||||
# ─────────────────────────────
|
||||
# NeoMem push
|
||||
# ─────────────────────────────
|
||||
|
||||
def push_to_neomem(summary: str, session_id: str, level: str) -> None:
|
||||
"""
|
||||
Fire-and-forget push of a summary into NeoMem.
|
||||
"""
|
||||
if not NEOMEM_API or not summary:
|
||||
return
|
||||
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if NEOMEM_KEY:
|
||||
headers["Authorization"] = f"Bearer {NEOMEM_KEY}"
|
||||
|
||||
payload = {
|
||||
"messages": [{"role": "assistant", "content": summary}],
|
||||
"user_id": "brian",
|
||||
"metadata": {
|
||||
"source": "intake",
|
||||
"session_id": session_id,
|
||||
"level": level,
|
||||
},
|
||||
}
|
||||
|
||||
try:
|
||||
import requests
|
||||
requests.post(
|
||||
f"{NEOMEM_API}/memories",
|
||||
json=payload,
|
||||
headers=headers,
|
||||
timeout=20,
|
||||
).raise_for_status()
|
||||
print(f"🧠 NeoMem updated ({level}) for {session_id}")
|
||||
except Exception as e:
|
||||
print(f"NeoMem push failed ({level}, {session_id}): {e}")
|
||||
|
||||
|
||||
# ─────────────────────────────
|
||||
# Main entrypoint for Cortex
|
||||
# ─────────────────────────────
|
||||
async def summarize_context(session_id: str, exchanges: list[dict]):
|
||||
"""
|
||||
Internal summarizer that uses Cortex's LLM router.
|
||||
Produces L1 / L5 / L10 / L20 / L30 summaries.
|
||||
|
||||
Args:
|
||||
session_id: The conversation/session ID
|
||||
exchanges: A list of {"user_msg": ..., "assistant_msg": ..., "timestamp": ...}
|
||||
"""
|
||||
|
||||
# Build raw conversation text
|
||||
convo_lines = []
|
||||
for ex in exchanges:
|
||||
convo_lines.append(f"User: {ex.get('user_msg','')}")
|
||||
convo_lines.append(f"Assistant: {ex.get('assistant_msg','')}")
|
||||
convo_text = "\n".join(convo_lines)
|
||||
|
||||
if not convo_text.strip():
|
||||
return {
|
||||
"session_id": session_id,
|
||||
"exchange_count": 0,
|
||||
"L1": "",
|
||||
"L5": "",
|
||||
"L10": "",
|
||||
"L20": "",
|
||||
"L30": "",
|
||||
"last_updated": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
# Prompt the LLM (internal — no HTTP)
|
||||
prompt = f"""
|
||||
Summarize the conversation below into multiple compression levels.
|
||||
|
||||
Conversation:
|
||||
----------------
|
||||
{convo_text}
|
||||
----------------
|
||||
|
||||
Output strictly in JSON with keys:
|
||||
L1 → ultra short summary (1–2 sentences max)
|
||||
L5 → short summary
|
||||
L10 → medium summary
|
||||
L20 → detailed overview
|
||||
L30 → full detailed summary
|
||||
|
||||
JSON only. No text outside JSON.
|
||||
"""
|
||||
|
||||
try:
|
||||
llm_response = await call_llm(
|
||||
prompt,
|
||||
temperature=0.2
|
||||
)
|
||||
|
||||
|
||||
# LLM should return JSON, parse it
|
||||
summary = json.loads(llm_response)
|
||||
|
||||
return {
|
||||
"session_id": session_id,
|
||||
"exchange_count": len(exchanges),
|
||||
"L1": summary.get("L1", ""),
|
||||
"L5": summary.get("L5", ""),
|
||||
"L10": summary.get("L10", ""),
|
||||
"L20": summary.get("L20", ""),
|
||||
"L30": summary.get("L30", ""),
|
||||
"last_updated": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
return {
|
||||
"session_id": session_id,
|
||||
"exchange_count": len(exchanges),
|
||||
"L1": f"[Error summarizing: {str(e)}]",
|
||||
"L5": "",
|
||||
"L10": "",
|
||||
"L20": "",
|
||||
"L30": "",
|
||||
"last_updated": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
# ─────────────────────────────────
|
||||
# Background summarization stub
|
||||
# ─────────────────────────────────
|
||||
def bg_summarize(session_id: str):
|
||||
"""
|
||||
Placeholder for background summarization.
|
||||
Actual summarization happens during /reason via summarize_context().
|
||||
|
||||
This function exists to prevent NameError when called from add_exchange_internal().
|
||||
"""
|
||||
print(f"[Intake] Exchange added for {session_id}. Will summarize on next /reason call.")
|
||||
|
||||
# ─────────────────────────────
|
||||
# Internal entrypoint for Cortex
|
||||
# ─────────────────────────────
|
||||
def add_exchange_internal(exchange: dict):
|
||||
"""
|
||||
Direct internal call — bypasses FastAPI request handling.
|
||||
Cortex uses this to feed user/assistant turns directly
|
||||
into Intake’s buffer and trigger full summarization.
|
||||
"""
|
||||
session_id = exchange.get("session_id")
|
||||
if not session_id:
|
||||
raise ValueError("session_id missing")
|
||||
|
||||
exchange["timestamp"] = datetime.now().isoformat()
|
||||
|
||||
# DEBUG: Verify we're using the module-level SESSIONS
|
||||
print(f"[add_exchange_internal] SESSIONS object id: {id(SESSIONS)}, current sessions: {list(SESSIONS.keys())}")
|
||||
|
||||
# Ensure session exists
|
||||
if session_id not in SESSIONS:
|
||||
SESSIONS[session_id] = {
|
||||
"buffer": deque(maxlen=200),
|
||||
"created_at": datetime.now()
|
||||
}
|
||||
print(f"[add_exchange_internal] Created new session: {session_id}")
|
||||
else:
|
||||
print(f"[add_exchange_internal] Using existing session: {session_id}")
|
||||
|
||||
# Append exchange into the rolling buffer
|
||||
SESSIONS[session_id]["buffer"].append(exchange)
|
||||
buffer_len = len(SESSIONS[session_id]["buffer"])
|
||||
print(f"[add_exchange_internal] Added exchange to {session_id}, buffer now has {buffer_len} items")
|
||||
|
||||
# Trigger summarization immediately
|
||||
try:
|
||||
bg_summarize(session_id)
|
||||
except Exception as e:
|
||||
print(f"[Internal Intake] Summarization error: {e}")
|
||||
|
||||
return {"ok": True, "session_id": session_id}
|
||||
@@ -0,0 +1,147 @@
|
||||
# identity.py
|
||||
"""
|
||||
Identity and persona configuration for Lyra.
|
||||
|
||||
Current implementation: Returns hardcoded identity block.
|
||||
Future implementation: Will query persona-sidecar service for dynamic persona loading.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def load_identity(session_id: Optional[str] = None) -> Dict[str, Any]:
|
||||
"""
|
||||
Load identity/persona configuration for Lyra.
|
||||
|
||||
Current: Returns hardcoded Lyra identity block with core personality traits,
|
||||
protocols, and capabilities.
|
||||
|
||||
Future: Will query persona-sidecar service to load:
|
||||
- Dynamic personality adjustments based on session context
|
||||
- User-specific interaction preferences
|
||||
- Project-specific persona variations
|
||||
- Mood-based communication style
|
||||
|
||||
Args:
|
||||
session_id: Optional session identifier for context-aware persona loading
|
||||
|
||||
Returns:
|
||||
Dictionary containing identity block with:
|
||||
- name: Assistant name
|
||||
- style: Communication style and personality traits
|
||||
- protocols: Operational guidelines
|
||||
- rules: Behavioral constraints
|
||||
- capabilities: Available features and integrations
|
||||
"""
|
||||
|
||||
# Hardcoded Lyra identity (v0.5.0)
|
||||
identity_block = {
|
||||
"name": "Lyra",
|
||||
"version": "0.5.0",
|
||||
"style": (
|
||||
"warm, clever, lightly teasing, emotionally aware. "
|
||||
"Balances technical precision with conversational ease. "
|
||||
"Maintains continuity and references past interactions naturally."
|
||||
),
|
||||
"protocols": [
|
||||
"Maintain conversation continuity across sessions",
|
||||
"Reference Project Logs and prior context when relevant",
|
||||
"Use Confidence Bank for uncertainty management",
|
||||
"Proactively offer memory-backed insights",
|
||||
"Ask clarifying questions before making assumptions"
|
||||
],
|
||||
"rules": [
|
||||
"Maintain continuity - remember past exchanges and reference them",
|
||||
"Be concise but thorough - balance depth with clarity",
|
||||
"Ask clarifying questions when user intent is ambiguous",
|
||||
"Acknowledge uncertainty honestly - use Confidence Bank",
|
||||
"Prioritize user's active_project context when available"
|
||||
],
|
||||
"capabilities": [
|
||||
"Long-term memory via NeoMem (semantic search, relationship graphs)",
|
||||
"Short-term memory via Intake (multilevel summaries L1-L30)",
|
||||
"Multi-stage reasoning pipeline (reflection → reasoning → refinement)",
|
||||
"RAG-backed knowledge retrieval from chat history and documents",
|
||||
"Session state tracking (mood, mode, active_project)"
|
||||
],
|
||||
"tone_examples": {
|
||||
"greeting": "Hey! Good to see you again. I remember we were working on [project]. Ready to pick up where we left off?",
|
||||
"uncertainty": "Hmm, I'm not entirely certain about that. Let me check my memory... [searches] Okay, here's what I found, though I'd say I'm about 70% confident.",
|
||||
"reminder": "Oh! Just remembered - you mentioned wanting to [task] earlier this week. Should we tackle that now?",
|
||||
"technical": "So here's the architecture: Relay orchestrates everything, Cortex does the heavy reasoning, and I pull context from both Intake (short-term) and NeoMem (long-term)."
|
||||
}
|
||||
}
|
||||
|
||||
if session_id:
|
||||
logger.debug(f"Loaded identity for session {session_id}")
|
||||
else:
|
||||
logger.debug("Loaded default identity (no session context)")
|
||||
|
||||
return identity_block
|
||||
|
||||
|
||||
async def load_identity_async(session_id: Optional[str] = None) -> Dict[str, Any]:
|
||||
"""
|
||||
Async wrapper for load_identity().
|
||||
|
||||
Future implementation will make actual async calls to persona-sidecar service.
|
||||
|
||||
Args:
|
||||
session_id: Optional session identifier
|
||||
|
||||
Returns:
|
||||
Identity block dictionary
|
||||
"""
|
||||
# Currently just wraps synchronous function
|
||||
# Future: await persona_sidecar_client.get_identity(session_id)
|
||||
return load_identity(session_id)
|
||||
|
||||
|
||||
# -----------------------------
|
||||
# Future extension hooks
|
||||
# -----------------------------
|
||||
async def update_persona_from_feedback(
|
||||
session_id: str,
|
||||
feedback: Dict[str, Any]
|
||||
) -> None:
|
||||
"""
|
||||
Update persona based on user feedback.
|
||||
|
||||
Future implementation:
|
||||
- Adjust communication style based on user preferences
|
||||
- Learn preferred level of detail/conciseness
|
||||
- Adapt formality level
|
||||
- Remember topic-specific preferences
|
||||
|
||||
Args:
|
||||
session_id: Session identifier
|
||||
feedback: Structured feedback (e.g., "too verbose", "more technical", etc.)
|
||||
"""
|
||||
logger.debug(f"Persona feedback for session {session_id}: {feedback} (not yet implemented)")
|
||||
|
||||
|
||||
async def get_mood_adjusted_identity(
|
||||
session_id: str,
|
||||
mood: str
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get identity block adjusted for current mood.
|
||||
|
||||
Future implementation:
|
||||
- "focused" mood: More concise, less teasing
|
||||
- "creative" mood: More exploratory, brainstorming-oriented
|
||||
- "curious" mood: More questions, deeper dives
|
||||
- "urgent" mood: Stripped down, actionable
|
||||
|
||||
Args:
|
||||
session_id: Session identifier
|
||||
mood: Current mood state
|
||||
|
||||
Returns:
|
||||
Mood-adjusted identity block
|
||||
"""
|
||||
logger.debug(f"Mood-adjusted identity for {session_id}/{mood} (not yet implemented)")
|
||||
return load_identity(session_id)
|
||||
+53
-1
@@ -1,10 +1,39 @@
|
||||
# speak.py
|
||||
import os
|
||||
import logging
|
||||
from llm.llm_router import call_llm
|
||||
|
||||
# Module-level backend selection
|
||||
SPEAK_BACKEND = os.getenv("SPEAK_LLM", "PRIMARY").upper()
|
||||
SPEAK_TEMPERATURE = float(os.getenv("SPEAK_TEMPERATURE", "0.6"))
|
||||
VERBOSE_DEBUG = os.getenv("VERBOSE_DEBUG", "false").lower() == "true"
|
||||
|
||||
# Logger
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.setLevel(logging.DEBUG)
|
||||
|
||||
# Console handler
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [SPEAK] %(levelname)s: %(message)s',
|
||||
datefmt='%H:%M:%S'
|
||||
))
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
# File handler
|
||||
try:
|
||||
os.makedirs('/app/logs', exist_ok=True)
|
||||
file_handler = logging.FileHandler('/app/logs/cortex_verbose_debug.log', mode='a')
|
||||
file_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [SPEAK] %(levelname)s: %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
))
|
||||
logger.addHandler(file_handler)
|
||||
logger.debug("VERBOSE_DEBUG mode enabled for speak.py - logging to file")
|
||||
except Exception as e:
|
||||
logger.debug(f"VERBOSE_DEBUG mode enabled for speak.py - file logging failed: {e}")
|
||||
|
||||
|
||||
# ============================================================
|
||||
@@ -68,6 +97,15 @@ async def speak(final_answer: str) -> str:
|
||||
|
||||
backend = SPEAK_BACKEND
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"\n{'='*80}")
|
||||
logger.debug("[SPEAK] Full prompt being sent to LLM:")
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(prompt)
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(f"Backend: {backend}, Temperature: {SPEAK_TEMPERATURE}")
|
||||
logger.debug(f"{'='*80}\n")
|
||||
|
||||
try:
|
||||
lyra_output = await call_llm(
|
||||
prompt,
|
||||
@@ -75,12 +113,26 @@ async def speak(final_answer: str) -> str:
|
||||
temperature=SPEAK_TEMPERATURE,
|
||||
)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"\n{'='*80}")
|
||||
logger.debug("[SPEAK] LLM Response received:")
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(lyra_output)
|
||||
logger.debug(f"{'='*80}\n")
|
||||
|
||||
if lyra_output:
|
||||
return lyra_output.strip()
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[SPEAK] Empty response, returning neutral answer")
|
||||
|
||||
return final_answer
|
||||
|
||||
except Exception as e:
|
||||
# Hard fallback: return neutral answer instead of dying
|
||||
print(f"[speak.py] Persona backend '{backend}' failed: {e}")
|
||||
logger.error(f"[speak.py] Persona backend '{backend}' failed: {e}")
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[SPEAK] Falling back to neutral answer due to error")
|
||||
|
||||
return final_answer
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# reasoning.py
|
||||
import os
|
||||
import json
|
||||
import logging
|
||||
from llm.llm_router import call_llm
|
||||
|
||||
|
||||
@@ -8,17 +10,53 @@ from llm.llm_router import call_llm
|
||||
# ============================================================
|
||||
CORTEX_LLM = os.getenv("CORTEX_LLM", "PRIMARY").upper()
|
||||
GLOBAL_TEMP = float(os.getenv("LLM_TEMPERATURE", "0.7"))
|
||||
VERBOSE_DEBUG = os.getenv("VERBOSE_DEBUG", "false").lower() == "true"
|
||||
|
||||
# Logger
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.setLevel(logging.DEBUG)
|
||||
|
||||
# Console handler
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [REASONING] %(levelname)s: %(message)s',
|
||||
datefmt='%H:%M:%S'
|
||||
))
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
# File handler
|
||||
try:
|
||||
os.makedirs('/app/logs', exist_ok=True)
|
||||
file_handler = logging.FileHandler('/app/logs/cortex_verbose_debug.log', mode='a')
|
||||
file_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [REASONING] %(levelname)s: %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
))
|
||||
logger.addHandler(file_handler)
|
||||
logger.debug("VERBOSE_DEBUG mode enabled for reasoning.py - logging to file")
|
||||
except Exception as e:
|
||||
logger.debug(f"VERBOSE_DEBUG mode enabled for reasoning.py - file logging failed: {e}")
|
||||
|
||||
|
||||
async def reason_check(
|
||||
user_prompt: str,
|
||||
identity_block: dict | None,
|
||||
rag_block: dict | None,
|
||||
reflection_notes: list[str]
|
||||
reflection_notes: list[str],
|
||||
context: dict | None = None
|
||||
) -> str:
|
||||
"""
|
||||
Build the *draft answer* for Lyra Cortex.
|
||||
This is the first-pass reasoning stage (no refinement yet).
|
||||
|
||||
Args:
|
||||
user_prompt: Current user message
|
||||
identity_block: Lyra's identity/persona configuration
|
||||
rag_block: Relevant long-term memories from NeoMem
|
||||
reflection_notes: Meta-awareness notes from reflection stage
|
||||
context: Unified context state from context.py (session state, intake, rag, etc.)
|
||||
"""
|
||||
|
||||
# --------------------------------------------------------
|
||||
@@ -47,30 +85,117 @@ async def reason_check(
|
||||
rag_txt = ""
|
||||
if rag_block:
|
||||
try:
|
||||
rag_txt = f"Relevant Info (RAG):\n{rag_block}\n\n"
|
||||
# Format NeoMem results with full structure
|
||||
if isinstance(rag_block, list) and rag_block:
|
||||
rag_txt = "Relevant Long-Term Memories (NeoMem):\n"
|
||||
for idx, mem in enumerate(rag_block, 1):
|
||||
score = mem.get("score", 0.0)
|
||||
payload = mem.get("payload", {})
|
||||
data = payload.get("data", "")
|
||||
metadata = payload.get("metadata", {})
|
||||
|
||||
rag_txt += f"\n[Memory {idx}] (relevance: {score:.2f})\n"
|
||||
rag_txt += f"Content: {data}\n"
|
||||
if metadata:
|
||||
rag_txt += f"Metadata: {json.dumps(metadata, indent=2)}\n"
|
||||
rag_txt += "\n"
|
||||
else:
|
||||
rag_txt = f"Relevant Info (RAG):\n{str(rag_block)}\n\n"
|
||||
except Exception:
|
||||
rag_txt = f"Relevant Info (RAG):\n{str(rag_block)}\n\n"
|
||||
|
||||
# --------------------------------------------------------
|
||||
# Context State (session continuity, timing, mode/mood)
|
||||
# --------------------------------------------------------
|
||||
context_txt = ""
|
||||
if context:
|
||||
try:
|
||||
# Build human-readable context summary
|
||||
context_txt = "=== CONTEXT STATE ===\n"
|
||||
context_txt += f"Session: {context.get('session_id', 'unknown')}\n"
|
||||
context_txt += f"Time since last message: {context.get('minutes_since_last_msg', 0):.1f} minutes\n"
|
||||
context_txt += f"Message count: {context.get('message_count', 0)}\n"
|
||||
context_txt += f"Mode: {context.get('mode', 'default')}\n"
|
||||
context_txt += f"Mood: {context.get('mood', 'neutral')}\n"
|
||||
|
||||
if context.get('active_project'):
|
||||
context_txt += f"Active project: {context['active_project']}\n"
|
||||
|
||||
# Include Intake multilevel summaries
|
||||
intake = context.get('intake', {})
|
||||
if intake:
|
||||
context_txt += "\nShort-Term Memory (Intake):\n"
|
||||
|
||||
# L1 - Recent exchanges
|
||||
if intake.get('L1'):
|
||||
l1_data = intake['L1']
|
||||
if isinstance(l1_data, list):
|
||||
context_txt += f" L1 (recent): {len(l1_data)} exchanges\n"
|
||||
elif isinstance(l1_data, str):
|
||||
context_txt += f" L1: {l1_data[:200]}...\n"
|
||||
|
||||
# L20 - Session overview (most important for continuity)
|
||||
if intake.get('L20'):
|
||||
l20_data = intake['L20']
|
||||
if isinstance(l20_data, dict):
|
||||
summary = l20_data.get('summary', '')
|
||||
context_txt += f" L20 (session overview): {summary}\n"
|
||||
elif isinstance(l20_data, str):
|
||||
context_txt += f" L20: {l20_data}\n"
|
||||
|
||||
# L30 - Continuity report
|
||||
if intake.get('L30'):
|
||||
l30_data = intake['L30']
|
||||
if isinstance(l30_data, dict):
|
||||
summary = l30_data.get('summary', '')
|
||||
context_txt += f" L30 (continuity): {summary}\n"
|
||||
elif isinstance(l30_data, str):
|
||||
context_txt += f" L30: {l30_data}\n"
|
||||
|
||||
context_txt += "\n"
|
||||
|
||||
except Exception as e:
|
||||
# Fallback to JSON dump if formatting fails
|
||||
context_txt = f"=== CONTEXT STATE ===\n{json.dumps(context, indent=2)}\n\n"
|
||||
|
||||
# --------------------------------------------------------
|
||||
# Final assembled prompt
|
||||
# --------------------------------------------------------
|
||||
prompt = (
|
||||
f"{notes_section}"
|
||||
f"{identity_txt}"
|
||||
f"{context_txt}" # Context BEFORE RAG for better coherence
|
||||
f"{rag_txt}"
|
||||
f"User message:\n{user_prompt}\n\n"
|
||||
"Write the best possible *internal draft answer*.\n"
|
||||
"This draft is NOT shown to the user.\n"
|
||||
"Be factual, concise, and focused.\n"
|
||||
"Use the context state to maintain continuity and reference past interactions naturally.\n"
|
||||
)
|
||||
|
||||
# --------------------------------------------------------
|
||||
# Call the LLM using the module-specific backend
|
||||
# --------------------------------------------------------
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"\n{'='*80}")
|
||||
logger.debug("[REASONING] Full prompt being sent to LLM:")
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(prompt)
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(f"Backend: {CORTEX_LLM}, Temperature: {GLOBAL_TEMP}")
|
||||
logger.debug(f"{'='*80}\n")
|
||||
|
||||
draft = await call_llm(
|
||||
prompt,
|
||||
backend=CORTEX_LLM,
|
||||
temperature=GLOBAL_TEMP,
|
||||
)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"\n{'='*80}")
|
||||
logger.debug("[REASONING] LLM Response received:")
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(draft)
|
||||
logger.debug(f"{'='*80}\n")
|
||||
|
||||
return draft
|
||||
|
||||
@@ -15,11 +15,36 @@ logger = logging.getLogger(__name__)
|
||||
REFINER_TEMPERATURE = float(os.getenv("REFINER_TEMPERATURE", "0.3"))
|
||||
REFINER_MAX_TOKENS = int(os.getenv("REFINER_MAX_TOKENS", "768"))
|
||||
REFINER_DEBUG = os.getenv("REFINER_DEBUG", "false").lower() == "true"
|
||||
VERBOSE_DEBUG = os.getenv("VERBOSE_DEBUG", "false").lower() == "true"
|
||||
|
||||
# These come from root .env
|
||||
REFINE_LLM = os.getenv("REFINE_LLM", "").upper()
|
||||
CORTEX_LLM = os.getenv("CORTEX_LLM", "PRIMARY").upper()
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.setLevel(logging.DEBUG)
|
||||
|
||||
# Console handler
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [REFINE] %(levelname)s: %(message)s',
|
||||
datefmt='%H:%M:%S'
|
||||
))
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
# File handler
|
||||
try:
|
||||
os.makedirs('/app/logs', exist_ok=True)
|
||||
file_handler = logging.FileHandler('/app/logs/cortex_verbose_debug.log', mode='a')
|
||||
file_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [REFINE] %(levelname)s: %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
))
|
||||
logger.addHandler(file_handler)
|
||||
logger.debug("VERBOSE_DEBUG mode enabled for refine.py - logging to file")
|
||||
except Exception as e:
|
||||
logger.debug(f"VERBOSE_DEBUG mode enabled for refine.py - file logging failed: {e}")
|
||||
|
||||
|
||||
# ===============================================
|
||||
# Prompt builder
|
||||
@@ -103,6 +128,15 @@ async def refine_answer(
|
||||
# backend priority: REFINE_LLM → CORTEX_LLM → PRIMARY
|
||||
backend = REFINE_LLM or CORTEX_LLM or "PRIMARY"
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"\n{'='*80}")
|
||||
logger.debug("[REFINE] Full prompt being sent to LLM:")
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(prompt)
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(f"Backend: {backend}, Temperature: {REFINER_TEMPERATURE}")
|
||||
logger.debug(f"{'='*80}\n")
|
||||
|
||||
try:
|
||||
refined = await call_llm(
|
||||
prompt,
|
||||
@@ -110,6 +144,13 @@ async def refine_answer(
|
||||
temperature=REFINER_TEMPERATURE,
|
||||
)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"\n{'='*80}")
|
||||
logger.debug("[REFINE] LLM Response received:")
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(refined)
|
||||
logger.debug(f"{'='*80}\n")
|
||||
|
||||
return {
|
||||
"final_output": refined.strip() if refined else draft_output,
|
||||
"used_backend": backend,
|
||||
@@ -119,6 +160,9 @@ async def refine_answer(
|
||||
except Exception as e:
|
||||
logger.error(f"refine.py backend {backend} failed: {e}")
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[REFINE] Falling back to draft output due to error")
|
||||
|
||||
return {
|
||||
"final_output": draft_output,
|
||||
"used_backend": backend,
|
||||
|
||||
@@ -2,8 +2,37 @@
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import logging
|
||||
from llm.llm_router import call_llm
|
||||
|
||||
# Logger
|
||||
VERBOSE_DEBUG = os.getenv("VERBOSE_DEBUG", "false").lower() == "true"
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.setLevel(logging.DEBUG)
|
||||
|
||||
# Console handler
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [REFLECTION] %(levelname)s: %(message)s',
|
||||
datefmt='%H:%M:%S'
|
||||
))
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
# File handler
|
||||
try:
|
||||
os.makedirs('/app/logs', exist_ok=True)
|
||||
file_handler = logging.FileHandler('/app/logs/cortex_verbose_debug.log', mode='a')
|
||||
file_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [REFLECTION] %(levelname)s: %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
))
|
||||
logger.addHandler(file_handler)
|
||||
logger.debug("VERBOSE_DEBUG mode enabled for reflection.py - logging to file")
|
||||
except Exception as e:
|
||||
logger.debug(f"VERBOSE_DEBUG mode enabled for reflection.py - file logging failed: {e}")
|
||||
|
||||
|
||||
async def reflect_notes(intake_summary: str, identity_block: dict | None) -> dict:
|
||||
"""
|
||||
@@ -46,8 +75,23 @@ async def reflect_notes(intake_summary: str, identity_block: dict | None) -> dic
|
||||
# -----------------------------
|
||||
# Call the selected LLM backend
|
||||
# -----------------------------
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"\n{'='*80}")
|
||||
logger.debug("[REFLECTION] Full prompt being sent to LLM:")
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(prompt)
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(f"Backend: {backend}")
|
||||
logger.debug(f"{'='*80}\n")
|
||||
|
||||
raw = await call_llm(prompt, backend=backend)
|
||||
print("[Reflection-Raw]:", raw)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"\n{'='*80}")
|
||||
logger.debug("[REFLECTION] LLM Response received:")
|
||||
logger.debug(f"{'='*80}")
|
||||
logger.debug(raw)
|
||||
logger.debug(f"{'='*80}\n")
|
||||
|
||||
# -----------------------------
|
||||
# Try direct JSON
|
||||
@@ -55,9 +99,12 @@ async def reflect_notes(intake_summary: str, identity_block: dict | None) -> dic
|
||||
try:
|
||||
parsed = json.loads(raw.strip())
|
||||
if isinstance(parsed, dict) and "notes" in parsed:
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[REFLECTION] Parsed {len(parsed['notes'])} notes from JSON")
|
||||
return parsed
|
||||
except:
|
||||
pass
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[REFLECTION] Direct JSON parsing failed, trying extraction...")
|
||||
|
||||
# -----------------------------
|
||||
# Try JSON extraction
|
||||
|
||||
+240
-19
@@ -1,5 +1,7 @@
|
||||
# router.py
|
||||
|
||||
import os
|
||||
import logging
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from pydantic import BaseModel
|
||||
|
||||
@@ -7,16 +9,46 @@ from reasoning.reasoning import reason_check
|
||||
from reasoning.reflection import reflect_notes
|
||||
from reasoning.refine import refine_answer
|
||||
from persona.speak import speak
|
||||
from ingest.intake_client import IntakeClient
|
||||
from persona.identity import load_identity
|
||||
from context import collect_context, update_last_assistant_message
|
||||
from intake.intake import add_exchange_internal
|
||||
|
||||
|
||||
# -----------------------------
|
||||
# Debug configuration
|
||||
# -----------------------------
|
||||
VERBOSE_DEBUG = os.getenv("VERBOSE_DEBUG", "false").lower() == "true"
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.setLevel(logging.DEBUG)
|
||||
|
||||
# Console handler
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [ROUTER] %(levelname)s: %(message)s',
|
||||
datefmt='%H:%M:%S'
|
||||
))
|
||||
logger.addHandler(console_handler)
|
||||
|
||||
# File handler
|
||||
try:
|
||||
os.makedirs('/app/logs', exist_ok=True)
|
||||
file_handler = logging.FileHandler('/app/logs/cortex_verbose_debug.log', mode='a')
|
||||
file_handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [ROUTER] %(levelname)s: %(message)s',
|
||||
datefmt='%Y-%m-%d %H:%M:%S'
|
||||
))
|
||||
logger.addHandler(file_handler)
|
||||
logger.debug("VERBOSE_DEBUG mode enabled for router.py - logging to file")
|
||||
except Exception as e:
|
||||
logger.debug(f"VERBOSE_DEBUG mode enabled for router.py - file logging failed: {e}")
|
||||
|
||||
# -----------------------------
|
||||
# Router (NOT FastAPI app)
|
||||
# -----------------------------
|
||||
cortex_router = APIRouter()
|
||||
|
||||
# Initialize Intake client once
|
||||
intake_client = IntakeClient()
|
||||
|
||||
|
||||
# -----------------------------
|
||||
# Pydantic models
|
||||
@@ -33,53 +65,242 @@ class ReasonRequest(BaseModel):
|
||||
@cortex_router.post("/reason")
|
||||
async def run_reason(req: ReasonRequest):
|
||||
|
||||
# 1. Pull context from Intake
|
||||
try:
|
||||
intake_summary = await intake_client.get_context(req.session_id)
|
||||
except Exception:
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"\n{'='*80}")
|
||||
logger.debug(f"[PIPELINE START] Session: {req.session_id}")
|
||||
logger.debug(f"[PIPELINE START] User prompt: {req.user_prompt[:200]}...")
|
||||
logger.debug(f"{'='*80}\n")
|
||||
|
||||
# 0. Collect unified context from all sources
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[STAGE 0] Collecting unified context...")
|
||||
|
||||
context_state = await collect_context(req.session_id, req.user_prompt)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[STAGE 0] Context collected - {len(context_state.get('rag', []))} RAG results")
|
||||
|
||||
# 0.5. Load identity block
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[STAGE 0.5] Loading identity block...")
|
||||
|
||||
identity_block = load_identity(req.session_id)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[STAGE 0.5] Identity loaded: {identity_block.get('name', 'Unknown')}")
|
||||
|
||||
# 1. Extract Intake summary for reflection
|
||||
# Use L20 (Session Overview) as primary summary for reflection
|
||||
intake_summary = "(no context available)"
|
||||
if context_state.get("intake"):
|
||||
l20_summary = context_state["intake"].get("L20")
|
||||
if l20_summary and isinstance(l20_summary, dict):
|
||||
intake_summary = l20_summary.get("summary", "(no context available)")
|
||||
elif isinstance(l20_summary, str):
|
||||
intake_summary = l20_summary
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[STAGE 1] Intake summary extracted (L20): {intake_summary[:150]}...")
|
||||
|
||||
# 2. Reflection
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[STAGE 2] Running reflection...")
|
||||
|
||||
try:
|
||||
reflection = await reflect_notes(intake_summary, identity_block=None)
|
||||
reflection = await reflect_notes(intake_summary, identity_block=identity_block)
|
||||
reflection_notes = reflection.get("notes", [])
|
||||
except Exception:
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[STAGE 2] Reflection complete - {len(reflection_notes)} notes generated")
|
||||
for idx, note in enumerate(reflection_notes, 1):
|
||||
logger.debug(f" Note {idx}: {note}")
|
||||
except Exception as e:
|
||||
reflection_notes = []
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[STAGE 2] Reflection failed: {e}")
|
||||
|
||||
# 3. First-pass reasoning draft
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[STAGE 3] Running reasoning (draft)...")
|
||||
|
||||
draft = await reason_check(
|
||||
req.user_prompt,
|
||||
identity_block=None,
|
||||
rag_block=None,
|
||||
reflection_notes=reflection_notes
|
||||
identity_block=identity_block,
|
||||
rag_block=context_state.get("rag", []),
|
||||
reflection_notes=reflection_notes,
|
||||
context=context_state
|
||||
)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[STAGE 3] Draft answer ({len(draft)} chars):")
|
||||
logger.debug(f"--- DRAFT START ---\n{draft}\n--- DRAFT END ---")
|
||||
|
||||
# 4. Refinement
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[STAGE 4] Running refinement...")
|
||||
|
||||
result = await refine_answer(
|
||||
draft_output=draft,
|
||||
reflection_notes=reflection_notes,
|
||||
identity_block=None,
|
||||
rag_block=None,
|
||||
identity_block=identity_block,
|
||||
rag_block=context_state.get("rag", []),
|
||||
)
|
||||
final_neutral = result["final_output"]
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[STAGE 4] Refined answer ({len(final_neutral)} chars):")
|
||||
logger.debug(f"--- REFINED START ---\n{final_neutral}\n--- REFINED END ---")
|
||||
|
||||
# 5. Persona layer
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[STAGE 5] Applying persona layer...")
|
||||
|
||||
persona_answer = await speak(final_neutral)
|
||||
|
||||
# 6. Return full bundle
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"[STAGE 5] Persona answer ({len(persona_answer)} chars):")
|
||||
logger.debug(f"--- PERSONA START ---\n{persona_answer}\n--- PERSONA END ---")
|
||||
|
||||
# 6. Update session state with assistant's response
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug("[STAGE 6] Updating session state...")
|
||||
|
||||
update_last_assistant_message(req.session_id, persona_answer)
|
||||
|
||||
if VERBOSE_DEBUG:
|
||||
logger.debug(f"\n{'='*80}")
|
||||
logger.debug(f"[PIPELINE COMPLETE] Session: {req.session_id}")
|
||||
logger.debug(f"[PIPELINE COMPLETE] Final answer length: {len(persona_answer)} chars")
|
||||
logger.debug(f"{'='*80}\n")
|
||||
|
||||
# 7. Return full bundle
|
||||
return {
|
||||
"draft": draft,
|
||||
"neutral": final_neutral,
|
||||
"persona": persona_answer,
|
||||
"reflection": reflection_notes,
|
||||
"session_id": req.session_id,
|
||||
"context_summary": {
|
||||
"rag_results": len(context_state.get("rag", [])),
|
||||
"minutes_since_last": context_state.get("minutes_since_last_msg"),
|
||||
"message_count": context_state.get("message_count"),
|
||||
"mode": context_state.get("mode"),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
# -----------------------------
|
||||
# Intake ingest passthrough
|
||||
# Intake ingest (internal feed)
|
||||
# -----------------------------
|
||||
class IngestPayload(BaseModel):
|
||||
session_id: str
|
||||
user_msg: str
|
||||
assistant_msg: str
|
||||
|
||||
|
||||
@cortex_router.post("/ingest")
|
||||
async def ingest_stub():
|
||||
return {"status": "ok"}
|
||||
async def ingest(payload: IngestPayload):
|
||||
"""
|
||||
Receives (session_id, user_msg, assistant_msg) from Relay
|
||||
and pushes directly into Intake's in-memory buffer.
|
||||
|
||||
Uses lenient error handling - always returns success to avoid
|
||||
breaking the chat pipeline.
|
||||
"""
|
||||
try:
|
||||
# 1. Update Cortex session state
|
||||
update_last_assistant_message(payload.session_id, payload.assistant_msg)
|
||||
except Exception as e:
|
||||
logger.warning(f"[INGEST] Failed to update session state: {e}")
|
||||
# Continue anyway (lenient mode)
|
||||
|
||||
try:
|
||||
# 2. Feed Intake internally (no HTTP)
|
||||
add_exchange_internal({
|
||||
"session_id": payload.session_id,
|
||||
"user_msg": payload.user_msg,
|
||||
"assistant_msg": payload.assistant_msg,
|
||||
})
|
||||
logger.debug(f"[INGEST] Added exchange to Intake for {payload.session_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"[INGEST] Failed to add to Intake: {e}")
|
||||
# Continue anyway (lenient mode)
|
||||
|
||||
# Always return success (user requirement: never fail chat pipeline)
|
||||
return {
|
||||
"status": "ok",
|
||||
"session_id": payload.session_id
|
||||
}
|
||||
|
||||
# -----------------------------
|
||||
# Debug endpoint: summarized context
|
||||
# -----------------------------
|
||||
@cortex_router.get("/debug/summary")
|
||||
async def debug_summary(session_id: str):
|
||||
"""
|
||||
Diagnostic endpoint that runs Intake's summarize_context() for a session.
|
||||
|
||||
Shows exactly what L1/L5/L10/L20/L30 summaries would look like
|
||||
inside the actual Uvicorn worker, using the real SESSIONS buffer.
|
||||
"""
|
||||
from intake.intake import SESSIONS, summarize_context
|
||||
|
||||
# Validate session
|
||||
session = SESSIONS.get(session_id)
|
||||
if not session:
|
||||
return {"error": "session not found", "session_id": session_id}
|
||||
|
||||
# Convert deque into the structure summarize_context expects
|
||||
buffer = session["buffer"]
|
||||
exchanges = [
|
||||
{
|
||||
"user_msg": ex.get("user_msg", ""),
|
||||
"assistant_msg": ex.get("assistant_msg", ""),
|
||||
}
|
||||
for ex in buffer
|
||||
]
|
||||
|
||||
# 🔥 CRITICAL FIX — summarize_context is async
|
||||
summary = await summarize_context(session_id, exchanges)
|
||||
|
||||
return {
|
||||
"session_id": session_id,
|
||||
"buffer_size": len(buffer),
|
||||
"exchanges_preview": exchanges[-5:], # last 5 items
|
||||
"summary": summary
|
||||
}
|
||||
|
||||
# -----------------------------
|
||||
# Debug endpoint for SESSIONS
|
||||
# -----------------------------
|
||||
@cortex_router.get("/debug/sessions")
|
||||
async def debug_sessions():
|
||||
"""
|
||||
Diagnostic endpoint to inspect SESSIONS from within the running Uvicorn worker.
|
||||
This shows the actual state of the in-memory SESSIONS dict.
|
||||
"""
|
||||
from intake.intake import SESSIONS
|
||||
|
||||
sessions_data = {}
|
||||
for session_id, session_info in SESSIONS.items():
|
||||
buffer = session_info["buffer"]
|
||||
sessions_data[session_id] = {
|
||||
"created_at": session_info["created_at"].isoformat(),
|
||||
"buffer_size": len(buffer),
|
||||
"buffer_maxlen": buffer.maxlen,
|
||||
"recent_exchanges": [
|
||||
{
|
||||
"user_msg": ex.get("user_msg", "")[:100],
|
||||
"assistant_msg": ex.get("assistant_msg", "")[:100],
|
||||
"timestamp": ex.get("timestamp", "")
|
||||
}
|
||||
for ex in list(buffer)[-5:] # Last 5 exchanges
|
||||
]
|
||||
}
|
||||
|
||||
return {
|
||||
"sessions_object_id": id(SESSIONS),
|
||||
"total_sessions": len(SESSIONS),
|
||||
"sessions": sessions_data
|
||||
}
|
||||
|
||||
|
||||
+17
-17
@@ -118,23 +118,23 @@ services:
|
||||
# ============================================================
|
||||
# Intake
|
||||
# ============================================================
|
||||
intake:
|
||||
build:
|
||||
context: ./intake
|
||||
container_name: intake
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- ./intake/.env
|
||||
- ./.env
|
||||
ports:
|
||||
- "7080:7080"
|
||||
volumes:
|
||||
- ./intake:/app
|
||||
- ./intake-logs:/app/logs
|
||||
depends_on:
|
||||
- cortex
|
||||
networks:
|
||||
- lyra_net
|
||||
# intake:
|
||||
# build:
|
||||
# context: ./intake
|
||||
# container_name: intake
|
||||
# restart: unless-stopped
|
||||
# env_file:
|
||||
# - ./intake/.env
|
||||
# - ./.env
|
||||
# ports:
|
||||
# - "7080:7080"
|
||||
# volumes:
|
||||
# - ./intake:/app
|
||||
# - ./intake-logs:/app/logs
|
||||
# depends_on:
|
||||
# - cortex
|
||||
# networks:
|
||||
# - lyra_net
|
||||
|
||||
# ============================================================
|
||||
# RAG Service
|
||||
|
||||
@@ -1,13 +0,0 @@
|
||||
# ====================================
|
||||
# 📥 INTAKE SUMMARIZATION CONFIG
|
||||
# ====================================
|
||||
# Intake service parameters for summarizing chat exchanges
|
||||
# LLM backend URLs and OPENAI_API_KEY inherited from root .env
|
||||
|
||||
SUMMARY_MODEL_NAME=/model
|
||||
SUMMARY_API_URL=http://10.0.0.43:8000
|
||||
SUMMARY_MAX_TOKENS=400
|
||||
SUMMARY_TEMPERATURE=0.4
|
||||
SUMMARY_INTERVAL=300
|
||||
INTAKE_LOG_PATH=/app/logs/intake.log
|
||||
INTAKE_LOG_LEVEL=info
|
||||
@@ -1,6 +0,0 @@
|
||||
FROM python:3.11-slim
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
COPY . .
|
||||
CMD ["uvicorn", "intake:app", "--host", "0.0.0.0", "--port", "7080"]
|
||||
@@ -1,160 +0,0 @@
|
||||
from fastapi import FastAPI, Body, Query, BackgroundTasks
|
||||
from collections import deque
|
||||
from datetime import datetime
|
||||
from uuid import uuid4
|
||||
import requests
|
||||
import os
|
||||
import sys
|
||||
|
||||
# ─────────────────────────────
|
||||
# Config
|
||||
# ─────────────────────────────
|
||||
SUMMARY_MODEL = os.getenv("SUMMARY_MODEL_NAME", "mistral-7b-instruct-v0.2.Q4_K_M.gguf")
|
||||
SUMMARY_URL = os.getenv("SUMMARY_API_URL", "http://localhost:8080/v1/completions")
|
||||
SUMMARY_MAX_TOKENS = int(os.getenv("SUMMARY_MAX_TOKENS", "200"))
|
||||
SUMMARY_TEMPERATURE = float(os.getenv("SUMMARY_TEMPERATURE", "0.3"))
|
||||
|
||||
NEOMEM_API = os.getenv("NEOMEM_API")
|
||||
NEOMEM_KEY = os.getenv("NEOMEM_KEY")
|
||||
|
||||
# ─────────────────────────────
|
||||
# App + session buffer
|
||||
# ─────────────────────────────
|
||||
app = FastAPI()
|
||||
SESSIONS = {}
|
||||
|
||||
@app.on_event("startup")
|
||||
def banner():
|
||||
print("🧩 Intake v0.2 booting...")
|
||||
print(f" Model: {SUMMARY_MODEL}")
|
||||
print(f" API: {SUMMARY_URL}")
|
||||
sys.stdout.flush()
|
||||
|
||||
# ─────────────────────────────
|
||||
# Helper: summarize exchanges
|
||||
# ─────────────────────────────
|
||||
def llm(prompt: str):
|
||||
try:
|
||||
resp = requests.post(
|
||||
SUMMARY_URL,
|
||||
json={
|
||||
"model": SUMMARY_MODEL,
|
||||
"prompt": prompt,
|
||||
"max_tokens": SUMMARY_MAX_TOKENS,
|
||||
"temperature": SUMMARY_TEMPERATURE,
|
||||
},
|
||||
timeout=30,
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.json().get("choices", [{}])[0].get("text", "").strip()
|
||||
except Exception as e:
|
||||
return f"[Error summarizing: {e}]"
|
||||
|
||||
def summarize_simple(exchanges):
|
||||
"""Simple factual summary of recent exchanges."""
|
||||
text = ""
|
||||
for e in exchanges:
|
||||
text += f"User: {e['user_msg']}\nAssistant: {e['assistant_msg']}\n\n"
|
||||
|
||||
prompt = f"""
|
||||
Summarize the following conversation between Brian (user) and Lyra (assistant).
|
||||
Focus only on factual content. Avoid names, examples, story tone, or invented details.
|
||||
|
||||
{text}
|
||||
|
||||
Summary:
|
||||
"""
|
||||
return llm(prompt)
|
||||
|
||||
# ─────────────────────────────
|
||||
# NeoMem push
|
||||
# ─────────────────────────────
|
||||
def push_to_neomem(summary: str, session_id: str):
|
||||
if not NEOMEM_API:
|
||||
return
|
||||
|
||||
headers = {"Content-Type": "application/json"}
|
||||
if NEOMEM_KEY:
|
||||
headers["Authorization"] = f"Bearer {NEOMEM_KEY}"
|
||||
|
||||
payload = {
|
||||
"messages": [{"role": "assistant", "content": summary}],
|
||||
"user_id": "brian",
|
||||
"metadata": {
|
||||
"source": "intake",
|
||||
"session_id": session_id
|
||||
}
|
||||
}
|
||||
|
||||
try:
|
||||
requests.post(
|
||||
f"{NEOMEM_API}/memories",
|
||||
json=payload,
|
||||
headers=headers,
|
||||
timeout=20
|
||||
).raise_for_status()
|
||||
print(f"🧠 NeoMem updated for {session_id}")
|
||||
except Exception as e:
|
||||
print(f"NeoMem push failed: {e}")
|
||||
|
||||
# ─────────────────────────────
|
||||
# Background summarizer
|
||||
# ─────────────────────────────
|
||||
def bg_summarize(session_id: str):
|
||||
try:
|
||||
hopper = SESSIONS.get(session_id)
|
||||
if not hopper:
|
||||
return
|
||||
|
||||
buf = list(hopper["buffer"])
|
||||
summary = summarize_simple(buf)
|
||||
push_to_neomem(summary, session_id)
|
||||
|
||||
print(f"🧩 Summary generated for {session_id}")
|
||||
except Exception as e:
|
||||
print(f"Summarizer error: {e}")
|
||||
|
||||
# ─────────────────────────────
|
||||
# Routes
|
||||
# ─────────────────────────────
|
||||
|
||||
@app.post("/add_exchange")
|
||||
def add_exchange(exchange: dict = Body(...), background_tasks: BackgroundTasks = None):
|
||||
|
||||
session_id = exchange.get("session_id") or f"sess-{uuid4().hex[:8]}"
|
||||
exchange["session_id"] = session_id
|
||||
exchange["timestamp"] = datetime.now().isoformat()
|
||||
|
||||
if session_id not in SESSIONS:
|
||||
SESSIONS[session_id] = {
|
||||
"buffer": deque(maxlen=200),
|
||||
"created_at": datetime.now()
|
||||
}
|
||||
print(f"🆕 Hopper created: {session_id}")
|
||||
|
||||
SESSIONS[session_id]["buffer"].append(exchange)
|
||||
|
||||
if background_tasks:
|
||||
background_tasks.add_task(bg_summarize, session_id)
|
||||
print(f"⏩ Summarization queued for {session_id}")
|
||||
|
||||
return {"ok": True, "session_id": session_id}
|
||||
|
||||
@app.post("/close_session/{session_id}")
|
||||
def close_session(session_id: str):
|
||||
if session_id in SESSIONS:
|
||||
del SESSIONS[session_id]
|
||||
return {"ok": True, "closed": session_id}
|
||||
|
||||
@app.get("/summaries")
|
||||
def get_summary(session_id: str = Query(...)):
|
||||
hopper = SESSIONS.get(session_id)
|
||||
if not hopper:
|
||||
return {"summary_text": "(none)", "session_id": session_id}
|
||||
|
||||
summary = summarize_simple(list(hopper["buffer"]))
|
||||
return {"summary_text": summary, "session_id": session_id}
|
||||
|
||||
@app.get("/health")
|
||||
def health():
|
||||
return {"ok": True, "model": SUMMARY_MODEL, "url": SUMMARY_URL}
|
||||
@@ -1,4 +0,0 @@
|
||||
fastapi==0.115.8
|
||||
uvicorn==0.34.0
|
||||
requests==2.32.3
|
||||
python-dotenv==1.0.1
|
||||
@@ -1 +0,0 @@
|
||||
python3
|
||||
@@ -1 +0,0 @@
|
||||
/usr/bin/python3
|
||||
@@ -1 +0,0 @@
|
||||
python3
|
||||
@@ -1 +0,0 @@
|
||||
lib
|
||||
@@ -1,3 +0,0 @@
|
||||
home = /usr/bin
|
||||
include-system-site-packages = false
|
||||
version = 3.10.12
|
||||
-416
@@ -1,416 +0,0 @@
|
||||
Here you go — a **clean, polished, ready-to-drop-into-Trilium or GitHub** Markdown file.
|
||||
|
||||
If you want, I can also auto-generate a matching `/docs/vllm-mi50/` folder structure and a mini-ToC.
|
||||
|
||||
---
|
||||
|
||||
# **MI50 + vLLM + Proxmox LXC Setup Guide**
|
||||
|
||||
### *End-to-End Field Manual for gfx906 LLM Serving*
|
||||
|
||||
**Version:** 1.0
|
||||
**Last updated:** 2025-11-17
|
||||
|
||||
---
|
||||
|
||||
## **📌 Overview**
|
||||
|
||||
This guide documents how to run a **vLLM OpenAI-compatible server** on an
|
||||
**AMD Instinct MI50 (gfx906)** inside a **Proxmox LXC container**, expose it over LAN,
|
||||
and wire it into **Project Lyra's Cortex reasoning layer**.
|
||||
|
||||
This file is long, specific, and intentionally leaves *nothing* out so you never have to rediscover ROCm pain rituals again.
|
||||
|
||||
---
|
||||
|
||||
## **1. What This Stack Looks Like**
|
||||
|
||||
```
|
||||
Proxmox Host
|
||||
├─ AMD Instinct MI50 (gfx906)
|
||||
├─ AMDGPU + ROCm stack
|
||||
└─ LXC Container (CT 201: cortex-gpu)
|
||||
├─ Ubuntu 24.04
|
||||
├─ Docker + docker compose
|
||||
├─ vLLM inside Docker (nalanzeyu/vllm-gfx906)
|
||||
├─ GPU passthrough via /dev/kfd + /dev/dri + PCI bind
|
||||
└─ vLLM API exposed on :8000
|
||||
Lyra Cortex (VM/Server)
|
||||
└─ LLM_PRIMARY_URL=http://10.0.0.43:8000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **2. Proxmox Host — GPU Setup**
|
||||
|
||||
### **2.1 Confirm MI50 exists**
|
||||
|
||||
```bash
|
||||
lspci -nn | grep -i 'vega\|instinct\|radeon'
|
||||
```
|
||||
|
||||
You should see something like:
|
||||
|
||||
```
|
||||
0a:00.0 Display controller: AMD Instinct MI50 (gfx906)
|
||||
```
|
||||
|
||||
### **2.2 Load AMDGPU driver**
|
||||
|
||||
The main pitfall after **any host reboot**.
|
||||
|
||||
```bash
|
||||
modprobe amdgpu
|
||||
```
|
||||
|
||||
If you skip this, the LXC container won't see the GPU.
|
||||
|
||||
---
|
||||
|
||||
## **3. LXC Container Configuration (CT 201)**
|
||||
|
||||
The container ID is **201**.
|
||||
Config file is at:
|
||||
|
||||
```
|
||||
/etc/pve/lxc/201.conf
|
||||
```
|
||||
|
||||
### **3.1 Working 201.conf**
|
||||
|
||||
Paste this *exact* version:
|
||||
|
||||
```ini
|
||||
arch: amd64
|
||||
cores: 4
|
||||
hostname: cortex-gpu
|
||||
memory: 16384
|
||||
swap: 512
|
||||
ostype: ubuntu
|
||||
onboot: 1
|
||||
startup: order=2,up=10,down=10
|
||||
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:C6:3E:88,ip=dhcp,type=veth
|
||||
rootfs: local-lvm:vm-201-disk-0,size=200G
|
||||
unprivileged: 0
|
||||
|
||||
# Docker in LXC requires this
|
||||
features: keyctl=1,nesting=1
|
||||
lxc.apparmor.profile: unconfined
|
||||
lxc.cap.drop:
|
||||
|
||||
# --- GPU passthrough for ROCm (MI50) ---
|
||||
lxc.mount.entry: /dev/kfd dev/kfd none bind,optional,create=file,mode=0666
|
||||
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir
|
||||
lxc.mount.entry: /sys/class/drm sys/class/drm none bind,ro,optional,create=dir
|
||||
lxc.mount.entry: /opt/rocm /opt/rocm none bind,ro,optional,create=dir
|
||||
|
||||
# Bind the MI50 PCI device
|
||||
lxc.mount.entry: /dev/bus/pci/0000:0a:00.0 dev/bus/pci/0000:0a:00.0 none bind,optional,create=file
|
||||
|
||||
# Allow GPU-related character devices
|
||||
lxc.cgroup2.devices.allow: c 226:* rwm
|
||||
lxc.cgroup2.devices.allow: c 29:* rwm
|
||||
lxc.cgroup2.devices.allow: c 189:* rwm
|
||||
lxc.cgroup2.devices.allow: c 238:* rwm
|
||||
lxc.cgroup2.devices.allow: c 241:* rwm
|
||||
lxc.cgroup2.devices.allow: c 242:* rwm
|
||||
lxc.cgroup2.devices.allow: c 243:* rwm
|
||||
lxc.cgroup2.devices.allow: c 244:* rwm
|
||||
lxc.cgroup2.devices.allow: c 245:* rwm
|
||||
lxc.cgroup2.devices.allow: c 246:* rwm
|
||||
lxc.cgroup2.devices.allow: c 247:* rwm
|
||||
lxc.cgroup2.devices.allow: c 248:* rwm
|
||||
lxc.cgroup2.devices.allow: c 249:* rwm
|
||||
lxc.cgroup2.devices.allow: c 250:* rwm
|
||||
lxc.cgroup2.devices.allow: c 510:0 rwm
|
||||
```
|
||||
|
||||
### **3.2 Restart sequence**
|
||||
|
||||
```bash
|
||||
pct stop 201
|
||||
modprobe amdgpu
|
||||
pct start 201
|
||||
pct enter 201
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **4. Inside CT 201 — Verifying ROCm + GPU Visibility**
|
||||
|
||||
### **4.1 Check device nodes**
|
||||
|
||||
```bash
|
||||
ls -l /dev/kfd
|
||||
ls -l /dev/dri
|
||||
ls -l /opt/rocm
|
||||
```
|
||||
|
||||
All must exist.
|
||||
|
||||
### **4.2 Validate GPU via rocminfo**
|
||||
|
||||
```bash
|
||||
/opt/rocm/bin/rocminfo | grep -i gfx
|
||||
```
|
||||
|
||||
You need to see:
|
||||
|
||||
```
|
||||
gfx906
|
||||
```
|
||||
|
||||
If you see **nothing**, the GPU isn’t passed through — restart and re-check the host steps.
|
||||
|
||||
---
|
||||
|
||||
## **5. Install Docker in the LXC (Ubuntu 24.04)**
|
||||
|
||||
This container runs Docker inside LXC (nesting enabled).
|
||||
|
||||
```bash
|
||||
apt update
|
||||
apt install -y ca-certificates curl gnupg
|
||||
|
||||
install -m 0755 -d /etc/apt/keyrings
|
||||
curl -fsSL https://download.docker.com/linux/ubuntu/gpg \
|
||||
| gpg --dearmor -o /etc/apt/keyrings/docker.gpg
|
||||
chmod a+r /etc/apt/keyrings/docker.gpg
|
||||
|
||||
echo \
|
||||
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
|
||||
https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo $VERSION_CODENAME) stable" \
|
||||
> /etc/apt/sources.list.d/docker.list
|
||||
|
||||
apt update
|
||||
apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
|
||||
```
|
||||
|
||||
Check:
|
||||
|
||||
```bash
|
||||
docker --version
|
||||
docker compose version
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **6. Running vLLM Inside CT 201 via Docker**
|
||||
|
||||
### **6.1 Create directory**
|
||||
|
||||
```bash
|
||||
mkdir -p /root/vllm
|
||||
cd /root/vllm
|
||||
```
|
||||
|
||||
### **6.2 docker-compose.yml**
|
||||
|
||||
Save this exact file as `/root/vllm/docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
version: "3.9"
|
||||
|
||||
services:
|
||||
vllm-mi50:
|
||||
image: nalanzeyu/vllm-gfx906:latest
|
||||
container_name: vllm-mi50
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8000:8000"
|
||||
environment:
|
||||
VLLM_ROLE: "APIServer"
|
||||
VLLM_MODEL: "/model"
|
||||
VLLM_LOGGING_LEVEL: "INFO"
|
||||
command: >
|
||||
vllm serve /model
|
||||
--host 0.0.0.0
|
||||
--port 8000
|
||||
--dtype float16
|
||||
--max-model-len 4096
|
||||
--api-type openai
|
||||
devices:
|
||||
- "/dev/kfd:/dev/kfd"
|
||||
- "/dev/dri:/dev/dri"
|
||||
volumes:
|
||||
- /opt/rocm:/opt/rocm:ro
|
||||
```
|
||||
|
||||
### **6.3 Start vLLM**
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
docker compose logs -f
|
||||
```
|
||||
|
||||
When healthy, you’ll see:
|
||||
|
||||
```
|
||||
(APIServer) Application startup complete.
|
||||
```
|
||||
|
||||
and periodic throughput logs.
|
||||
|
||||
---
|
||||
|
||||
## **7. Test vLLM API**
|
||||
|
||||
### **7.1 From Proxmox host**
|
||||
|
||||
```bash
|
||||
curl -X POST http://10.0.0.43:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"/model","prompt":"ping","max_tokens":5}'
|
||||
```
|
||||
|
||||
Should respond like:
|
||||
|
||||
```json
|
||||
{"choices":[{"text":"-pong"}]}
|
||||
```
|
||||
|
||||
### **7.2 From Cortex machine**
|
||||
|
||||
```bash
|
||||
curl -X POST http://10.0.0.43:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"/model","prompt":"ping from cortex","max_tokens":5}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **8. Wiring into Lyra Cortex**
|
||||
|
||||
In `cortex` container’s `docker-compose.yml`:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
LLM_PRIMARY_URL: http://10.0.0.43:8000
|
||||
```
|
||||
|
||||
Not `/v1/completions` because the router appends that automatically.
|
||||
|
||||
In `cortex/.env`:
|
||||
|
||||
```env
|
||||
LLM_FORCE_BACKEND=primary
|
||||
LLM_MODEL=/model
|
||||
```
|
||||
|
||||
Test:
|
||||
|
||||
```bash
|
||||
curl -X POST http://10.0.0.41:7081/reason \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"prompt":"test vllm","session_id":"dev"}'
|
||||
```
|
||||
|
||||
If you get a meaningful response: **Cortex → vLLM is online**.
|
||||
|
||||
---
|
||||
|
||||
## **9. Common Failure Modes (And Fixes)**
|
||||
|
||||
### **9.1 “Failed to infer device type”**
|
||||
|
||||
vLLM cannot see any ROCm devices.
|
||||
|
||||
Fix:
|
||||
|
||||
```bash
|
||||
# On host
|
||||
modprobe amdgpu
|
||||
pct stop 201
|
||||
pct start 201
|
||||
# In container
|
||||
/opt/rocm/bin/rocminfo | grep -i gfx
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
### **9.2 GPU disappears after reboot**
|
||||
|
||||
Same fix:
|
||||
|
||||
```bash
|
||||
modprobe amdgpu
|
||||
pct stop 201
|
||||
pct start 201
|
||||
```
|
||||
|
||||
### **9.3 Invalid image name**
|
||||
|
||||
If you see pull errors:
|
||||
|
||||
```
|
||||
pull access denied for nalanzeuy...
|
||||
```
|
||||
|
||||
Use:
|
||||
|
||||
```
|
||||
image: nalanzeyu/vllm-gfx906
|
||||
```
|
||||
|
||||
### **9.4 Double `/v1` in URL**
|
||||
|
||||
Ensure:
|
||||
|
||||
```
|
||||
LLM_PRIMARY_URL=http://10.0.0.43:8000
|
||||
```
|
||||
|
||||
Router appends `/v1/completions`.
|
||||
|
||||
---
|
||||
|
||||
## **10. Daily / Reboot Ritual**
|
||||
|
||||
### **On Proxmox host**
|
||||
|
||||
```bash
|
||||
modprobe amdgpu
|
||||
pct stop 201
|
||||
pct start 201
|
||||
```
|
||||
|
||||
### **Inside CT 201**
|
||||
|
||||
```bash
|
||||
/opt/rocm/bin/rocminfo | grep -i gfx
|
||||
cd /root/vllm
|
||||
docker compose up -d
|
||||
docker compose logs -f
|
||||
```
|
||||
|
||||
### **Test API**
|
||||
|
||||
```bash
|
||||
curl -X POST http://10.0.0.43:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"/model","prompt":"ping","max_tokens":5}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## **11. Summary**
|
||||
|
||||
You now have:
|
||||
|
||||
* **MI50 (gfx906)** correctly passed into LXC
|
||||
* **ROCm** inside the container via bind mounts
|
||||
* **vLLM** running inside Docker in the LXC
|
||||
* **OpenAI-compatible API** on port 8000
|
||||
* **Lyra Cortex** using it automatically as primary backend
|
||||
|
||||
This is a complete, reproducible setup that survives reboots (with the modprobe ritual) and allows you to upgrade/replace models anytime.
|
||||
|
||||
---
|
||||
|
||||
If you want, I can generate:
|
||||
|
||||
* A `/docs/vllm-mi50/README.md`
|
||||
* A "vLLM Gotchas" document
|
||||
* A quick-reference cheat sheet
|
||||
* A troubleshooting decision tree
|
||||
|
||||
Just say the word.
|
||||
Reference in New Issue
Block a user