179 lines
8.3 KiB
Markdown
179 lines
8.3 KiB
Markdown
# Logging System Migration Complete
|
|
|
|
## ✅ What Changed
|
|
|
|
The old `VERBOSE_DEBUG` logging system has been completely replaced with the new structured `LOG_DETAIL_LEVEL` system.
|
|
|
|
### Files Modified
|
|
|
|
1. **[.env](.env)** - Removed `VERBOSE_DEBUG`, cleaned up duplicate `LOG_DETAIL_LEVEL` settings
|
|
2. **[cortex/.env](cortex/.env)** - Removed `VERBOSE_DEBUG` from cortex config
|
|
3. **[cortex/router.py](cortex/router.py)** - Replaced `VERBOSE_DEBUG` checks with `LOG_DETAIL_LEVEL`
|
|
4. **[cortex/context.py](cortex/context.py)** - Replaced `VERBOSE_DEBUG` with `LOG_DETAIL_LEVEL`, removed verbose file logging setup
|
|
|
|
## 🎯 New Logging Configuration
|
|
|
|
### Single Environment Variable
|
|
|
|
Set `LOG_DETAIL_LEVEL` in your `.env` file:
|
|
|
|
```bash
|
|
LOG_DETAIL_LEVEL=detailed
|
|
```
|
|
|
|
### Logging Levels
|
|
|
|
| Level | Lines/Message | What You See |
|
|
|-------|---------------|--------------|
|
|
| **minimal** | 1-2 | Only errors and critical events |
|
|
| **summary** | 5-7 | Pipeline completion, errors, warnings (production mode) |
|
|
| **detailed** | 30-50 | LLM outputs, timing breakdowns, context (debugging mode) |
|
|
| **verbose** | 100+ | Everything including raw JSON dumps (deep debugging) |
|
|
|
|
## 📊 What You Get at Each Level
|
|
|
|
### Summary Mode (Production)
|
|
```
|
|
📊 Context | Session: abc123 | Messages: 42 | Last: 5.2min | RAG: 3 results
|
|
🧠 Monologue | question | Tone: curious
|
|
|
|
====================================================================================================
|
|
✨ PIPELINE COMPLETE | Session: abc123 | Total: 1250ms
|
|
====================================================================================================
|
|
📤 Output: 342 characters
|
|
====================================================================================================
|
|
```
|
|
|
|
### Detailed Mode (Debugging - RECOMMENDED)
|
|
```
|
|
====================================================================================================
|
|
🚀 PIPELINE START | Session: abc123 | 14:23:45.123
|
|
====================================================================================================
|
|
📝 User: What is the meaning of life?
|
|
────────────────────────────────────────────────────────────────────────────────────────────────────
|
|
|
|
────────────────────────────────────────────────────────────────────────────────────────────────────
|
|
🧠 LLM CALL | Backend: PRIMARY | 14:23:45.234
|
|
────────────────────────────────────────────────────────────────────────────────────────────────────
|
|
📝 Prompt: You are Lyra, analyzing the user's question...
|
|
💬 Reply: Based on the context provided, here's my analysis...
|
|
────────────────────────────────────────────────────────────────────────────────────────────────────
|
|
|
|
📊 Context | Session: abc123 | Messages: 42 | Last: 5.2min | RAG: 3 results
|
|
────────────────────────────────────────────────────────────────────────────────────────────────────
|
|
[CONTEXT] Session abc123 | User: What is the meaning of life?
|
|
────────────────────────────────────────────────────────────────────────────────────────────────────
|
|
Mode: default | Mood: neutral | Project: None
|
|
Tools: RAG, WEB, WEATHER, CODEBRAIN, POKERBRAIN
|
|
|
|
╭─ INTAKE SUMMARIES ────────────────────────────────────────────────
|
|
│ L1 : Last message discussed philosophy...
|
|
│ L5 : Recent 5 messages covered existential topics...
|
|
│ L10 : Past 10 messages showed curiosity pattern...
|
|
╰───────────────────────────────────────────────────────────────────
|
|
|
|
╭─ RAG RESULTS (3) ──────────────────────────────────────────────
|
|
│ [1] 0.923 | Previous discussion about purpose...
|
|
│ [2] 0.891 | Note about existential philosophy...
|
|
│ [3] 0.867 | Memory of Viktor Frankl discussion...
|
|
╰───────────────────────────────────────────────────────────────────
|
|
────────────────────────────────────────────────────────────────────────────────────────────────────
|
|
|
|
🧠 Monologue | question | Tone: curious
|
|
|
|
====================================================================================================
|
|
✨ PIPELINE COMPLETE | Session: abc123 | Total: 1250ms
|
|
====================================================================================================
|
|
⏱️ Stage Timings:
|
|
context : 150ms ( 12.0%)
|
|
identity : 10ms ( 0.8%)
|
|
monologue : 200ms ( 16.0%)
|
|
tools : 0ms ( 0.0%)
|
|
reflection : 50ms ( 4.0%)
|
|
reasoning : 450ms ( 36.0%) ← BOTTLENECK!
|
|
refinement : 300ms ( 24.0%)
|
|
persona : 140ms ( 11.2%)
|
|
learning : 50ms ( 4.0%)
|
|
📤 Output: 342 characters
|
|
====================================================================================================
|
|
```
|
|
|
|
### Verbose Mode (Maximum Debug)
|
|
Same as detailed, plus:
|
|
- Full raw JSON responses from LLMs (50-line boxes)
|
|
- Complete intake data structures
|
|
- Stack traces on errors
|
|
|
|
## 🚀 How to Use
|
|
|
|
### For Finding Weak Links (Your Use Case)
|
|
```bash
|
|
# In .env:
|
|
LOG_DETAIL_LEVEL=detailed
|
|
|
|
# Restart services:
|
|
docker-compose restart cortex relay
|
|
```
|
|
|
|
You'll now see:
|
|
- ✅ Which LLM backend is used
|
|
- ✅ What prompts are sent to each LLM
|
|
- ✅ What each LLM responds with
|
|
- ✅ Timing breakdown showing which stage is slow
|
|
- ✅ Context being used (RAG, intake summaries)
|
|
- ✅ Clean, hierarchical structure
|
|
|
|
### For Production
|
|
```bash
|
|
LOG_DETAIL_LEVEL=summary
|
|
```
|
|
|
|
### For Deep Debugging
|
|
```bash
|
|
LOG_DETAIL_LEVEL=verbose
|
|
```
|
|
|
|
## 🔍 Finding Performance Bottlenecks
|
|
|
|
With `detailed` mode, look for:
|
|
|
|
1. **Slow stages in timing breakdown:**
|
|
```
|
|
reasoning : 3450ms ( 76.0%) ← THIS IS YOUR BOTTLENECK!
|
|
```
|
|
|
|
2. **Backend failures:**
|
|
```
|
|
⚠️ [LLM] PRIMARY failed | 14:23:45.234 | Connection timeout
|
|
✅ [LLM] SECONDARY | Reply: Based on... ← Fell back to secondary
|
|
```
|
|
|
|
3. **Loop detection:**
|
|
```
|
|
⚠️ DUPLICATE MESSAGE DETECTED | Session: abc123
|
|
🔁 LOOP DETECTED - Returning cached context
|
|
```
|
|
|
|
## 📁 Removed Features
|
|
|
|
The following old logging features have been removed:
|
|
|
|
- ❌ `VERBOSE_DEBUG` environment variable (replaced with `LOG_DETAIL_LEVEL`)
|
|
- ❌ File logging to `/app/logs/cortex_verbose_debug.log` (use `docker logs` instead)
|
|
- ❌ Separate verbose handlers in Python logging
|
|
- ❌ Per-module verbose flags
|
|
|
|
## ✨ New Features
|
|
|
|
- ✅ Single unified logging configuration
|
|
- ✅ Hierarchical, scannable output
|
|
- ✅ Collapsible data sections (boxes)
|
|
- ✅ Stage timing always shown in detailed mode
|
|
- ✅ Performance profiling built-in
|
|
- ✅ Loop detection and warnings
|
|
- ✅ Clean error formatting
|
|
|
|
---
|
|
|
|
**The logging is now clean, concise, and gives you exactly what you need to find weak links!** 🎯
|