Update to v0.9.1 #1

Merged
serversdown merged 44 commits from dev into main 2026-01-18 02:46:25 -05:00
10 changed files with 567 additions and 10 deletions
Showing only changes of commit 0a091fc42c - Show all commits

87
.env.example Normal file
View File

@@ -0,0 +1,87 @@
# ====================================
# 🌌 GLOBAL LYRA CONFIG
# ====================================
LOCAL_TZ_LABEL=America/New_York
DEFAULT_SESSION_ID=default
# ====================================
# 🤖 LLM BACKEND OPTIONS
# ====================================
# Services choose which backend to use from these options
# Primary: vLLM on MI50 GPU
LLM_PRIMARY_PROVIDER=vllm
LLM_PRIMARY_URL=http://10.0.0.43:8000
LLM_PRIMARY_MODEL=/model
# Secondary: Ollama on 3090 GPU
LLM_SECONDARY_PROVIDER=ollama
LLM_SECONDARY_URL=http://10.0.0.3:11434
LLM_SECONDARY_MODEL=qwen2.5:7b-instruct-q4_K_M
# Cloud: OpenAI
LLM_CLOUD_PROVIDER=openai_chat
LLM_CLOUD_URL=https://api.openai.com/v1
LLM_CLOUD_MODEL=gpt-4o-mini
OPENAI_API_KEY=sk-proj-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
# Local Fallback: llama.cpp or LM Studio
LLM_FALLBACK_PROVIDER=openai_completions
LLM_FALLBACK_URL=http://10.0.0.41:11435
LLM_FALLBACK_MODEL=llama-3.2-8b-instruct
# Global LLM controls
LLM_TEMPERATURE=0.7
# ====================================
# 🗄️ DATABASE CONFIGURATION
# ====================================
# Postgres (pgvector for NeoMem)
POSTGRES_USER=neomem
POSTGRES_PASSWORD=change_me_in_production
POSTGRES_DB=neomem
POSTGRES_HOST=neomem-postgres
POSTGRES_PORT=5432
# Neo4j Graph Database
NEO4J_URI=bolt://neomem-neo4j:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=change_me_in_production
NEO4J_AUTH=neo4j/change_me_in_production
# ====================================
# 🧠 MEMORY SERVICES (NEOMEM)
# ====================================
NEOMEM_API=http://neomem-api:7077
NEOMEM_API_KEY=generate_secure_random_token_here
NEOMEM_HISTORY_DB=postgresql://neomem:change_me_in_production@neomem-postgres:5432/neomem
# Embeddings configuration (used by NeoMem)
EMBEDDER_PROVIDER=openai
EMBEDDER_MODEL=text-embedding-3-small
# ====================================
# 🔌 INTERNAL SERVICE URLS
# ====================================
# Using container names for Docker network communication
INTAKE_API_URL=http://intake:7080
CORTEX_API=http://cortex:7081
CORTEX_URL=http://cortex:7081/reflect
CORTEX_URL_INGEST=http://cortex:7081/ingest
RAG_API_URL=http://rag:7090
RELAY_URL=http://relay:7078
# Persona service (optional)
PERSONA_URL=http://persona-sidecar:7080/current
# ====================================
# 🔧 FEATURE FLAGS
# ====================================
CORTEX_ENABLED=true
MEMORY_ENABLED=true
PERSONA_ENABLED=false
DEBUG_PROMPT=true

16
.gitignore vendored
View File

@@ -5,8 +5,22 @@ __pycache__/
*.pyc
*.log
# Env files (store secrets locally)
# =============================
# 🔐 Environment files (NEVER commit secrets!)
# =============================
# Ignore all .env files
.env
.env.local
.env.*.local
**/.env
**/.env.local
# BUT track .env.example templates (safe to commit)
!.env.example
!**/.env.example
# Ignore backup directory
.env-backups/
# =============================
# 🐳 Docker volumes (HUGE)

View File

@@ -2,11 +2,78 @@
All notable changes to Project Lyra are organized by component.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
and adheres to [Semantic Versioning](https://semver.org/).
# Last Updated: 11-13-25
# Last Updated: 11-26-25
---
## 🧠 Lyra-Core ##############################################################################
## [Infrastructure v1.0.0] - 2025-11-26
### Changed
- **Environment Variable Consolidation** - Major reorganization to eliminate duplication and improve maintainability
- Consolidated 9 scattered `.env` files into single source of truth architecture
- Root `.env` now contains all shared infrastructure (LLM backends, databases, API keys, service URLs)
- Service-specific `.env` files minimized to only essential overrides:
- `cortex/.env`: Reduced from 42 to 22 lines (operational parameters only)
- `neomem/.env`: Reduced from 26 to 14 lines (LLM naming conventions only)
- `intake/.env`: Kept at 8 lines (already minimal)
- **Result**: ~24% reduction in total configuration lines (197 → ~150)
- **Docker Compose Consolidation**
- All services now defined in single root `docker-compose.yml`
- Relay service updated with complete configuration (env_file, volumes)
- Removed redundant `core/docker-compose.yml` (marked as DEPRECATED)
- Standardized network communication to use Docker container names
- **Service URL Standardization**
- Internal services use container names: `http://neomem-api:7077`, `http://cortex:7081`
- External services use IP addresses: `http://10.0.0.43:8000` (vLLM), `http://10.0.0.3:11434` (Ollama)
- Removed IP/container name inconsistencies across files
### Added
- **Security Templates** - Created `.env.example` files for all services
- Root `.env.example` with sanitized credentials
- Service-specific templates: `cortex/.env.example`, `neomem/.env.example`, `intake/.env.example`, `rag/.env.example`
- All `.env.example` files safe to commit to version control
- **Documentation**
- `ENVIRONMENT_VARIABLES.md`: Comprehensive reference for all environment variables
- Variable descriptions, defaults, and usage examples
- Multi-backend LLM strategy documentation
- Troubleshooting guide
- Security best practices
- `DEPRECATED_FILES.md`: Deletion guide for deprecated files with verification steps
- **Enhanced .gitignore**
- Ignores all `.env` files (including subdirectories)
- Tracks `.env.example` templates for documentation
- Ignores `.env-backups/` directory
### Removed
- `core/.env` - Redundant with root `.env`, now deleted
- `core/docker-compose.yml` - Consolidated into main compose file (marked DEPRECATED)
### Fixed
- Eliminated duplicate `OPENAI_API_KEY` across 5+ files
- Eliminated duplicate LLM backend URLs across 4+ files
- Eliminated duplicate database credentials across 3+ files
- Resolved Cortex `environment:` section override in docker-compose (now uses env_file)
### Architecture
- **Multi-Backend LLM Strategy**: Root `.env` provides all backend OPTIONS (PRIMARY, SECONDARY, CLOUD, FALLBACK), services choose which to USE
- Cortex → vLLM (PRIMARY) for autonomous reasoning
- NeoMem → Ollama (SECONDARY) + OpenAI embeddings
- Intake → vLLM (PRIMARY) for summarization
- Relay → Fallback chain with user preference
- Preserves per-service flexibility while eliminating URL duplication
### Migration
- All original `.env` files backed up to `.env-backups/` with timestamp `20251126_025334`
- Rollback plan documented in `ENVIRONMENT_VARIABLES.md`
- Verification steps provided in `DEPRECATED_FILES.md`
---
## [Lyra_RAG v0.1.0] 2025-11-07
### Added
- Initial standalone RAG module for Project Lyra.

91
DEPRECATED_FILES.md Normal file
View File

@@ -0,0 +1,91 @@
# Deprecated Files - Safe to Delete
This file lists all deprecated files that can be safely deleted after verification.
## Files Marked for Deletion
### Docker Compose Files
#### `/core/docker-compose.yml.DEPRECATED`
- **Status**: DEPRECATED
- **Reason**: All services consolidated into main `/docker-compose.yml`
- **Replaced by**: `/docker-compose.yml` (relay service now has complete config)
- **Safe to delete**: Yes, after verifying main docker-compose works
### Environment Files
All original `.env` files have been consolidated. Backups exist in `.env-backups/` directory.
#### Previously Deleted (Already Done)
-`/core/.env` - Deleted (redundant with root .env)
### Experimental/Orphaned Files
#### `/core/env experiments/` (entire directory)
- **Status**: User will handle separately
- **Contains**: `.env`, `.env.local`, `.env.openai`
- **Action**: User to review and clean up
## Verification Steps Before Deleting
Before deleting the deprecated files, verify:
1. **Test main docker-compose.yml works:**
```bash
cd /home/serversdown/project-lyra
docker-compose down
docker-compose up -d
docker-compose ps # All services should be running
```
2. **Verify relay service has correct config:**
```bash
docker exec relay env | grep -E "LLM_|NEOMEM_|OPENAI"
docker exec relay ls -la /app/sessions # Sessions volume mounted
```
3. **Test relay functionality:**
- Send a test message through relay
- Verify memory storage works
- Confirm LLM backend connections work
## Deletion Commands
After successful verification, run:
```bash
cd /home/serversdown/project-lyra
# Delete deprecated docker-compose file
rm core/docker-compose.yml.DEPRECATED
# Optionally clean up backup directory after confirming everything works
# (Keep backups for at least a few days/weeks)
# rm -rf .env-backups/
```
## Files to Keep
These files should **NOT** be deleted:
- ✅ `.env` (root) - Single source of truth
- ✅ `.env.example` (root) - Security template (commit to git)
- ✅ `cortex/.env` - Service-specific config
- ✅ `cortex/.env.example` - Security template (commit to git)
- ✅ `neomem/.env` - Service-specific config
- ✅ `neomem/.env.example` - Security template (commit to git)
- ✅ `intake/.env` - Service-specific config
- ✅ `intake/.env.example` - Security template (commit to git)
- ✅ `rag/.env.example` - Security template (commit to git)
- ✅ `docker-compose.yml` - Main orchestration file
- ✅ `ENVIRONMENT_VARIABLES.md` - Documentation
- ✅ `.gitignore` - Git configuration
## Backup Information
All original `.env` files backed up to:
- Location: `/home/serversdown/project-lyra/.env-backups/`
- Timestamp: `20251126_025334`
- Files: 6 original .env files
Keep backups until you're confident the new setup is stable (recommended: 2-4 weeks).

250
ENVIRONMENT_VARIABLES.md Normal file
View File

@@ -0,0 +1,250 @@
# Environment Variables Reference
This document describes all environment variables used across Project Lyra services.
## Quick Start
1. Copy environment templates:
```bash
cp .env.example .env
cp cortex/.env.example cortex/.env
cp neomem/.env.example neomem/.env
cp intake/.env.example intake/.env
```
2. Edit `.env` and add your credentials:
- `OPENAI_API_KEY`: Your OpenAI API key
- `POSTGRES_PASSWORD`: Database password
- `NEO4J_PASSWORD`: Graph database password
- `NEOMEM_API_KEY`: Generate a secure token
3. Update service URLs if your infrastructure differs from defaults
## File Structure
### Root `.env` - Shared Infrastructure
Contains all shared configuration used by multiple services:
- LLM backend options (PRIMARY, SECONDARY, CLOUD, FALLBACK)
- Database credentials (Postgres, Neo4j)
- API keys (OpenAI)
- Internal service URLs
- Feature flags
### Service-Specific `.env` Files
Each service has minimal overrides for service-specific parameters:
- **`cortex/.env`**: Cortex operational parameters
- **`neomem/.env`**: NeoMem LLM naming convention mappings
- **`intake/.env`**: Intake summarization parameters
## Environment Loading Order
Docker Compose loads environment files in this order (later overrides earlier):
1. Service-specific `.env` (e.g., `cortex/.env`)
2. Root `.env`
This means service-specific files can override root values when needed.
## Global Variables (Root `.env`)
### Global Configuration
| Variable | Default | Description |
|----------|---------|-------------|
| `LOCAL_TZ_LABEL` | `America/New_York` | Timezone for logs and timestamps |
| `DEFAULT_SESSION_ID` | `default` | Default chat session identifier |
### LLM Backend Options
Each service chooses which backend to use from these available options.
#### Primary Backend (vLLM on MI50 GPU)
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_PRIMARY_PROVIDER` | `vllm` | Provider type |
| `LLM_PRIMARY_URL` | `http://10.0.0.43:8000` | vLLM server endpoint |
| `LLM_PRIMARY_MODEL` | `/model` | Model path for vLLM |
#### Secondary Backend (Ollama on 3090 GPU)
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_SECONDARY_PROVIDER` | `ollama` | Provider type |
| `LLM_SECONDARY_URL` | `http://10.0.0.3:11434` | Ollama server endpoint |
| `LLM_SECONDARY_MODEL` | `qwen2.5:7b-instruct-q4_K_M` | Ollama model name |
#### Cloud Backend (OpenAI)
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_CLOUD_PROVIDER` | `openai_chat` | Provider type |
| `LLM_CLOUD_URL` | `https://api.openai.com/v1` | OpenAI API endpoint |
| `LLM_CLOUD_MODEL` | `gpt-4o-mini` | OpenAI model to use |
| `OPENAI_API_KEY` | *required* | OpenAI API authentication key |
#### Fallback Backend (llama.cpp/LM Studio)
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_FALLBACK_PROVIDER` | `openai_completions` | Provider type (llama.cpp mimics OpenAI) |
| `LLM_FALLBACK_URL` | `http://10.0.0.41:11435` | Fallback server endpoint |
| `LLM_FALLBACK_MODEL` | `llama-3.2-8b-instruct` | Fallback model name |
#### LLM Global Settings
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_TEMPERATURE` | `0.7` | Sampling temperature (0.0-2.0) |
### Database Configuration
#### PostgreSQL (with pgvector)
| Variable | Default | Description |
|----------|---------|-------------|
| `POSTGRES_USER` | `neomem` | PostgreSQL username |
| `POSTGRES_PASSWORD` | *required* | PostgreSQL password |
| `POSTGRES_DB` | `neomem` | Database name |
| `POSTGRES_HOST` | `neomem-postgres` | Container name/hostname |
| `POSTGRES_PORT` | `5432` | PostgreSQL port |
#### Neo4j Graph Database
| Variable | Default | Description |
|----------|---------|-------------|
| `NEO4J_URI` | `bolt://neomem-neo4j:7687` | Neo4j connection URI |
| `NEO4J_USERNAME` | `neo4j` | Neo4j username |
| `NEO4J_PASSWORD` | *required* | Neo4j password |
| `NEO4J_AUTH` | `neo4j/<password>` | Neo4j auth string |
### Memory Services (NeoMem)
| Variable | Default | Description |
|----------|---------|-------------|
| `NEOMEM_API` | `http://neomem-api:7077` | NeoMem API endpoint |
| `NEOMEM_API_KEY` | *required* | NeoMem API authentication token |
| `NEOMEM_HISTORY_DB` | `postgresql://...` | PostgreSQL connection string for history |
| `EMBEDDER_PROVIDER` | `openai` | Embedding provider (used by NeoMem) |
| `EMBEDDER_MODEL` | `text-embedding-3-small` | Embedding model name |
### Internal Service URLs
All using Docker container names for network communication:
| Variable | Default | Description |
|----------|---------|-------------|
| `INTAKE_API_URL` | `http://intake:7080` | Intake summarizer service |
| `CORTEX_API` | `http://cortex:7081` | Cortex reasoning service |
| `CORTEX_URL` | `http://cortex:7081/reflect` | Cortex reflection endpoint |
| `CORTEX_URL_INGEST` | `http://cortex:7081/ingest` | Cortex ingest endpoint |
| `RAG_API_URL` | `http://rag:7090` | RAG service (if enabled) |
| `RELAY_URL` | `http://relay:7078` | Relay orchestration service |
| `PERSONA_URL` | `http://persona-sidecar:7080/current` | Persona service (optional) |
### Feature Flags
| Variable | Default | Description |
|----------|---------|-------------|
| `CORTEX_ENABLED` | `true` | Enable Cortex autonomous reflection |
| `MEMORY_ENABLED` | `true` | Enable NeoMem long-term memory |
| `PERSONA_ENABLED` | `false` | Enable persona sidecar |
| `DEBUG_PROMPT` | `true` | Enable debug logging for prompts |
## Service-Specific Variables
### Cortex (`cortex/.env`)
Cortex operational parameters:
| Variable | Default | Description |
|----------|---------|-------------|
| `CORTEX_MODE` | `autonomous` | Operation mode (autonomous/manual) |
| `CORTEX_LOOP_INTERVAL` | `300` | Seconds between reflection loops |
| `CORTEX_REFLECTION_INTERVAL` | `86400` | Seconds between deep reflections (24h) |
| `CORTEX_LOG_LEVEL` | `debug` | Logging verbosity |
| `NEOMEM_HEALTH_CHECK_INTERVAL` | `300` | NeoMem health check frequency |
| `REFLECTION_NOTE_TARGET` | `trilium` | Where to store reflection notes |
| `REFLECTION_NOTE_PATH` | `/app/logs/reflections.log` | Reflection output path |
| `RELEVANCE_THRESHOLD` | `0.78` | Memory retrieval relevance threshold |
**Note**: Cortex uses `LLM_PRIMARY` (vLLM on MI50) by default from root `.env`.
### NeoMem (`neomem/.env`)
NeoMem uses different variable naming conventions:
| Variable | Default | Description |
|----------|---------|-------------|
| `LLM_PROVIDER` | `ollama` | NeoMem's LLM provider name |
| `LLM_MODEL` | `qwen2.5:7b-instruct-q4_K_M` | NeoMem's LLM model |
| `LLM_API_BASE` | `http://10.0.0.3:11434` | NeoMem's LLM endpoint (Ollama) |
**Note**: NeoMem uses Ollama (SECONDARY) for reasoning and OpenAI for embeddings. Database credentials and `OPENAI_API_KEY` inherited from root `.env`.
### Intake (`intake/.env`)
Intake summarization parameters:
| Variable | Default | Description |
|----------|---------|-------------|
| `SUMMARY_MODEL_NAME` | `/model` | Model path for summarization |
| `SUMMARY_API_URL` | `http://10.0.0.43:8000` | LLM endpoint for summaries |
| `SUMMARY_MAX_TOKENS` | `400` | Max tokens for summary generation |
| `SUMMARY_TEMPERATURE` | `0.4` | Temperature for summaries (lower = more focused) |
| `SUMMARY_INTERVAL` | `300` | Seconds between summary checks |
| `INTAKE_LOG_PATH` | `/app/logs/intake.log` | Log file location |
| `INTAKE_LOG_LEVEL` | `info` | Logging verbosity |
**Note**: Intake uses `LLM_PRIMARY` (vLLM) by default.
## Multi-Backend LLM Strategy
Project Lyra supports flexible backend selection per service:
**Root `.env` provides backend OPTIONS**:
- PRIMARY: vLLM on MI50 GPU (high performance)
- SECONDARY: Ollama on 3090 GPU (local inference)
- CLOUD: OpenAI API (cloud fallback)
- FALLBACK: llama.cpp/LM Studio (CPU-only)
**Services choose which backend to USE**:
- **Cortex** → vLLM (PRIMARY) for autonomous reasoning
- **NeoMem** → Ollama (SECONDARY) + OpenAI embeddings
- **Intake** → vLLM (PRIMARY) for summarization
- **Relay** → Implements fallback cascade with user preference
This design eliminates URL duplication while preserving per-service flexibility.
## Security Best Practices
1. **Never commit `.env` files to git** - they contain secrets
2. **Use `.env.example` templates** for documentation and onboarding
3. **Rotate credentials regularly**, especially:
- `OPENAI_API_KEY`
- `NEOMEM_API_KEY`
- Database passwords
4. **Use strong passwords** for production databases
5. **Restrict network access** to LLM backends and databases
## Troubleshooting
### Services can't connect to each other
- Verify container names match in service URLs
- Check all services are on the `lyra_net` Docker network
- Use `docker-compose ps` to verify all services are running
### LLM calls failing
- Verify backend URLs are correct for your infrastructure
- Check if LLM servers are running and accessible
- Test with `curl <LLM_URL>/v1/models` (OpenAI-compatible APIs)
### Database connection errors
- Verify database credentials match in all locations
- Check if database containers are healthy: `docker-compose ps`
- Review database logs: `docker-compose logs neomem-postgres`
### Environment variables not loading
- Verify env_file paths in docker-compose.yml
- Check file permissions: `.env` files must be readable
- Remember loading order: service `.env` overrides root `.env`
## Migration from Old Setup
If you have the old multi-file setup with duplicated variables:
1. **Backup existing files**: All original `.env` files are in `.env-backups/`
2. **Copy new templates**: Use `.env.example` files as base
3. **Merge credentials**: Transfer your actual keys/passwords to new root `.env`
4. **Test thoroughly**: Verify all services start and communicate correctly
## Support
For issues or questions:
- Check logs: `docker-compose logs <service>`
- Verify configuration: `docker exec <container> env | grep <VAR>`
- Review this documentation for variable descriptions

View File

@@ -1,3 +1,7 @@
# DEPRECATED - USE /home/serversdown/project-lyra/docker-compose.yml instead
# This file is no longer needed. All services are now in the main docker-compose.yml
# Safe to delete after verifying main compose file works correctly.
#
services:
relay:
build:
@@ -7,7 +11,7 @@ services:
ports:
- "7078:7078"
env_file:
- .env
- ../.env # Use root .env (core/.env is now redundant)
volumes:
- ./relay/sessions:/app/sessions
networks:

21
cortex/.env.example Normal file
View File

@@ -0,0 +1,21 @@
# ====================================
# 🧠 CORTEX OPERATIONAL CONFIG
# ====================================
# Cortex-specific parameters (all other config inherited from root .env)
CORTEX_MODE=autonomous
CORTEX_LOOP_INTERVAL=300
CORTEX_REFLECTION_INTERVAL=86400
CORTEX_LOG_LEVEL=debug
NEOMEM_HEALTH_CHECK_INTERVAL=300
# Reflection output configuration
REFLECTION_NOTE_TARGET=trilium
REFLECTION_NOTE_PATH=/app/logs/reflections.log
# Memory retrieval tuning
RELEVANCE_THRESHOLD=0.78
# NOTE: LLM backend URLs, OPENAI_API_KEY, database credentials,
# and service URLs are all inherited from root .env
# Cortex uses LLM_PRIMARY (vLLM on MI50) by default

View File

@@ -87,6 +87,10 @@ services:
context: ./core/relay
container_name: relay
restart: unless-stopped
env_file:
- ./.env
volumes:
- ./core/relay/sessions:/app/sessions
ports:
- "7078:7078"
networks:
@@ -107,11 +111,6 @@ services:
- ./cortex:/app
ports:
- "7081:7081"
environment:
LLM_PRIMARY_URL: http://10.0.0.43:8000/
NEOMEM_URL: http://neomem-api:7077
RAG_URL: http://rag:7090
RELAY_URL: http://relay:7078
networks:
- lyra_net

13
intake/.env.example Normal file
View File

@@ -0,0 +1,13 @@
# ====================================
# 📥 INTAKE SUMMARIZATION CONFIG
# ====================================
# Intake service parameters for summarizing chat exchanges
# LLM backend URLs and OPENAI_API_KEY inherited from root .env
SUMMARY_MODEL_NAME=/model
SUMMARY_API_URL=http://10.0.0.43:8000
SUMMARY_MAX_TOKENS=400
SUMMARY_TEMPERATURE=0.4
SUMMARY_INTERVAL=300
INTAKE_LOG_PATH=/app/logs/intake.log
INTAKE_LOG_LEVEL=info

11
rag/.env.example Normal file
View File

@@ -0,0 +1,11 @@
# ====================================
# 📚 RAG SERVICE CONFIG
# ====================================
# Retrieval-Augmented Generation service (Beta Lyrae)
# Currently not wired into the system - for future activation
# OPENAI_API_KEY and other shared config inherited from root .env
# RAG-specific configuration will go here when service is activated
# ChromaDB configuration
# Vector store settings
# Retrieval parameters