Cortex debugging logs cleaned up
This commit is contained in:
serversdwn
178
LOGGING_MIGRATION.md
Normal file
178
LOGGING_MIGRATION.md
Normal file
@@ -0,0 +1,178 @@
|
||||
# Logging System Migration Complete
|
||||
|
||||
## ✅ What Changed
|
||||
|
||||
The old `VERBOSE_DEBUG` logging system has been completely replaced with the new structured `LOG_DETAIL_LEVEL` system.
|
||||
|
||||
### Files Modified
|
||||
|
||||
1. **[.env](.env)** - Removed `VERBOSE_DEBUG`, cleaned up duplicate `LOG_DETAIL_LEVEL` settings
|
||||
2. **[cortex/.env](cortex/.env)** - Removed `VERBOSE_DEBUG` from cortex config
|
||||
3. **[cortex/router.py](cortex/router.py)** - Replaced `VERBOSE_DEBUG` checks with `LOG_DETAIL_LEVEL`
|
||||
4. **[cortex/context.py](cortex/context.py)** - Replaced `VERBOSE_DEBUG` with `LOG_DETAIL_LEVEL`, removed verbose file logging setup
|
||||
|
||||
## 🎯 New Logging Configuration
|
||||
|
||||
### Single Environment Variable
|
||||
|
||||
Set `LOG_DETAIL_LEVEL` in your `.env` file:
|
||||
|
||||
```bash
|
||||
LOG_DETAIL_LEVEL=detailed
|
||||
```
|
||||
|
||||
### Logging Levels
|
||||
|
||||
| Level | Lines/Message | What You See |
|
||||
|-------|---------------|--------------|
|
||||
| **minimal** | 1-2 | Only errors and critical events |
|
||||
| **summary** | 5-7 | Pipeline completion, errors, warnings (production mode) |
|
||||
| **detailed** | 30-50 | LLM outputs, timing breakdowns, context (debugging mode) |
|
||||
| **verbose** | 100+ | Everything including raw JSON dumps (deep debugging) |
|
||||
|
||||
## 📊 What You Get at Each Level
|
||||
|
||||
### Summary Mode (Production)
|
||||
```
|
||||
📊 Context | Session: abc123 | Messages: 42 | Last: 5.2min | RAG: 3 results
|
||||
🧠 Monologue | question | Tone: curious
|
||||
|
||||
====================================================================================================
|
||||
✨ PIPELINE COMPLETE | Session: abc123 | Total: 1250ms
|
||||
====================================================================================================
|
||||
📤 Output: 342 characters
|
||||
====================================================================================================
|
||||
```
|
||||
|
||||
### Detailed Mode (Debugging - RECOMMENDED)
|
||||
```
|
||||
====================================================================================================
|
||||
🚀 PIPELINE START | Session: abc123 | 14:23:45.123
|
||||
====================================================================================================
|
||||
📝 User: What is the meaning of life?
|
||||
────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
🧠 LLM CALL | Backend: PRIMARY | 14:23:45.234
|
||||
────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
📝 Prompt: You are Lyra, analyzing the user's question...
|
||||
💬 Reply: Based on the context provided, here's my analysis...
|
||||
────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
📊 Context | Session: abc123 | Messages: 42 | Last: 5.2min | RAG: 3 results
|
||||
────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
[CONTEXT] Session abc123 | User: What is the meaning of life?
|
||||
────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
Mode: default | Mood: neutral | Project: None
|
||||
Tools: RAG, WEB, WEATHER, CODEBRAIN, POKERBRAIN
|
||||
|
||||
╭─ INTAKE SUMMARIES ────────────────────────────────────────────────
|
||||
│ L1 : Last message discussed philosophy...
|
||||
│ L5 : Recent 5 messages covered existential topics...
|
||||
│ L10 : Past 10 messages showed curiosity pattern...
|
||||
╰───────────────────────────────────────────────────────────────────
|
||||
|
||||
╭─ RAG RESULTS (3) ──────────────────────────────────────────────
|
||||
│ [1] 0.923 | Previous discussion about purpose...
|
||||
│ [2] 0.891 | Note about existential philosophy...
|
||||
│ [3] 0.867 | Memory of Viktor Frankl discussion...
|
||||
╰───────────────────────────────────────────────────────────────────
|
||||
────────────────────────────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
🧠 Monologue | question | Tone: curious
|
||||
|
||||
====================================================================================================
|
||||
✨ PIPELINE COMPLETE | Session: abc123 | Total: 1250ms
|
||||
====================================================================================================
|
||||
⏱️ Stage Timings:
|
||||
context : 150ms ( 12.0%)
|
||||
identity : 10ms ( 0.8%)
|
||||
monologue : 200ms ( 16.0%)
|
||||
tools : 0ms ( 0.0%)
|
||||
reflection : 50ms ( 4.0%)
|
||||
reasoning : 450ms ( 36.0%) ← BOTTLENECK!
|
||||
refinement : 300ms ( 24.0%)
|
||||
persona : 140ms ( 11.2%)
|
||||
learning : 50ms ( 4.0%)
|
||||
📤 Output: 342 characters
|
||||
====================================================================================================
|
||||
```
|
||||
|
||||
### Verbose Mode (Maximum Debug)
|
||||
Same as detailed, plus:
|
||||
- Full raw JSON responses from LLMs (50-line boxes)
|
||||
- Complete intake data structures
|
||||
- Stack traces on errors
|
||||
|
||||
## 🚀 How to Use
|
||||
|
||||
### For Finding Weak Links (Your Use Case)
|
||||
```bash
|
||||
# In .env:
|
||||
LOG_DETAIL_LEVEL=detailed
|
||||
|
||||
# Restart services:
|
||||
docker-compose restart cortex relay
|
||||
```
|
||||
|
||||
You'll now see:
|
||||
- ✅ Which LLM backend is used
|
||||
- ✅ What prompts are sent to each LLM
|
||||
- ✅ What each LLM responds with
|
||||
- ✅ Timing breakdown showing which stage is slow
|
||||
- ✅ Context being used (RAG, intake summaries)
|
||||
- ✅ Clean, hierarchical structure
|
||||
|
||||
### For Production
|
||||
```bash
|
||||
LOG_DETAIL_LEVEL=summary
|
||||
```
|
||||
|
||||
### For Deep Debugging
|
||||
```bash
|
||||
LOG_DETAIL_LEVEL=verbose
|
||||
```
|
||||
|
||||
## 🔍 Finding Performance Bottlenecks
|
||||
|
||||
With `detailed` mode, look for:
|
||||
|
||||
1. **Slow stages in timing breakdown:**
|
||||
```
|
||||
reasoning : 3450ms ( 76.0%) ← THIS IS YOUR BOTTLENECK!
|
||||
```
|
||||
|
||||
2. **Backend failures:**
|
||||
```
|
||||
⚠️ [LLM] PRIMARY failed | 14:23:45.234 | Connection timeout
|
||||
✅ [LLM] SECONDARY | Reply: Based on... ← Fell back to secondary
|
||||
```
|
||||
|
||||
3. **Loop detection:**
|
||||
```
|
||||
⚠️ DUPLICATE MESSAGE DETECTED | Session: abc123
|
||||
🔁 LOOP DETECTED - Returning cached context
|
||||
```
|
||||
|
||||
## 📁 Removed Features
|
||||
|
||||
The following old logging features have been removed:
|
||||
|
||||
- ❌ `VERBOSE_DEBUG` environment variable (replaced with `LOG_DETAIL_LEVEL`)
|
||||
- ❌ File logging to `/app/logs/cortex_verbose_debug.log` (use `docker logs` instead)
|
||||
- ❌ Separate verbose handlers in Python logging
|
||||
- ❌ Per-module verbose flags
|
||||
|
||||
## ✨ New Features
|
||||
|
||||
- ✅ Single unified logging configuration
|
||||
- ✅ Hierarchical, scannable output
|
||||
- ✅ Collapsible data sections (boxes)
|
||||
- ✅ Stage timing always shown in detailed mode
|
||||
- ✅ Performance profiling built-in
|
||||
- ✅ Loop detection and warnings
|
||||
- ✅ Clean error formatting
|
||||
|
||||
---
|
||||
|
||||
**The logging is now clean, concise, and gives you exactly what you need to find weak links!** 🎯
|
||||
Reference in New Issue
Block a user