project-lyra/LOGGING_MIGRATION.md

Logging System Migration Complete

✅ What Changed

The old VERBOSE_DEBUG logging system has been completely replaced with the new structured LOG_DETAIL_LEVEL system.

Files Modified

1. .env - Removed VERBOSE_DEBUG, cleaned up duplicate LOG_DETAIL_LEVEL settings
2. cortex/.env - Removed VERBOSE_DEBUG from cortex config
3. cortex/router.py - Replaced VERBOSE_DEBUG checks with LOG_DETAIL_LEVEL
4. cortex/context.py - Replaced VERBOSE_DEBUG with LOG_DETAIL_LEVEL, removed verbose file logging setup

🎯 New Logging Configuration

Single Environment Variable

Set LOG_DETAIL_LEVEL in your .env file:

LOG_DETAIL_LEVEL=detailed

Logging Levels

Level	Lines/Message	What You See
minimal	1-2	Only errors and critical events
summary	5-7	Pipeline completion, errors, warnings (production mode)
detailed	30-50	LLM outputs, timing breakdowns, context (debugging mode)
verbose	100+	Everything including raw JSON dumps (deep debugging)

📊 What You Get at Each Level

Summary Mode (Production)

📊 Context | Session: abc123 | Messages: 42 | Last: 5.2min | RAG: 3 results
🧠 Monologue | question | Tone: curious

====================================================================================================
✨ PIPELINE COMPLETE | Session: abc123 | Total: 1250ms
====================================================================================================
📤 Output: 342 characters
====================================================================================================

Detailed Mode (Debugging - RECOMMENDED)

====================================================================================================
🚀 PIPELINE START | Session: abc123 | 14:23:45.123
====================================================================================================
📝 User: What is the meaning of life?
────────────────────────────────────────────────────────────────────────────────────────────────────

────────────────────────────────────────────────────────────────────────────────────────────────────
🧠 LLM CALL | Backend: PRIMARY | 14:23:45.234
────────────────────────────────────────────────────────────────────────────────────────────────────
📝 Prompt: You are Lyra, analyzing the user's question...
💬 Reply: Based on the context provided, here's my analysis...
────────────────────────────────────────────────────────────────────────────────────────────────────

📊 Context | Session: abc123 | Messages: 42 | Last: 5.2min | RAG: 3 results
────────────────────────────────────────────────────────────────────────────────────────────────────
[CONTEXT] Session abc123 | User: What is the meaning of life?
────────────────────────────────────────────────────────────────────────────────────────────────────
  Mode: default | Mood: neutral | Project: None
  Tools: RAG, WEB, WEATHER, CODEBRAIN, POKERBRAIN

  ╭─ INTAKE SUMMARIES ────────────────────────────────────────────────
  │ L1  : Last message discussed philosophy...
  │ L5  : Recent 5 messages covered existential topics...
  │ L10 : Past 10 messages showed curiosity pattern...
  ╰───────────────────────────────────────────────────────────────────

  ╭─ RAG RESULTS (3) ──────────────────────────────────────────────
  │ [1] 0.923 | Previous discussion about purpose...
  │ [2] 0.891 | Note about existential philosophy...
  │ [3] 0.867 | Memory of Viktor Frankl discussion...
  ╰───────────────────────────────────────────────────────────────────
────────────────────────────────────────────────────────────────────────────────────────────────────

🧠 Monologue | question | Tone: curious

====================================================================================================
✨ PIPELINE COMPLETE | Session: abc123 | Total: 1250ms
====================================================================================================
⏱️  Stage Timings:
   context        :   150ms ( 12.0%)
   identity       :    10ms (  0.8%)
   monologue      :   200ms ( 16.0%)
   tools          :     0ms (  0.0%)
   reflection     :    50ms (  4.0%)
   reasoning      :   450ms ( 36.0%)  ← BOTTLENECK!
   refinement     :   300ms ( 24.0%)
   persona        :   140ms ( 11.2%)
   learning       :    50ms (  4.0%)
📤 Output: 342 characters
====================================================================================================

Verbose Mode (Maximum Debug)

Same as detailed, plus:

• Full raw JSON responses from LLMs (50-line boxes)
• Complete intake data structures
• Stack traces on errors

🚀 How to Use

For Finding Weak Links (Your Use Case)

# In .env:
LOG_DETAIL_LEVEL=detailed

# Restart services:
docker-compose restart cortex relay

You'll now see:

• ✅ Which LLM backend is used
• ✅ What prompts are sent to each LLM
• ✅ What each LLM responds with
• ✅ Timing breakdown showing which stage is slow
• ✅ Context being used (RAG, intake summaries)
• ✅ Clean, hierarchical structure

For Production

LOG_DETAIL_LEVEL=summary

For Deep Debugging

LOG_DETAIL_LEVEL=verbose

🔍 Finding Performance Bottlenecks

With detailed mode, look for:

1. Slow stages in timing breakdown:

   reasoning      :  3450ms ( 76.0%)  ← THIS IS YOUR BOTTLENECK!
   

2. Backend failures:

   ⚠️  [LLM] PRIMARY failed | 14:23:45.234 | Connection timeout
   ✅ [LLM] SECONDARY | Reply: Based on...  ← Fell back to secondary
   

3. Loop detection:

   ⚠️  DUPLICATE MESSAGE DETECTED | Session: abc123
   🔁 LOOP DETECTED - Returning cached context
   

📁 Removed Features

The following old logging features have been removed:

• ❌ VERBOSE_DEBUG environment variable (replaced with LOG_DETAIL_LEVEL)
• ❌ File logging to /app/logs/cortex_verbose_debug.log (use docker logs instead)
• ❌ Separate verbose handlers in Python logging
• ❌ Per-module verbose flags

✨ New Features

• ✅ Single unified logging configuration
• ✅ Hierarchical, scannable output
• ✅ Collapsible data sections (boxes)
• ✅ Stage timing always shown in detailed mode
• ✅ Performance profiling built-in
• ✅ Loop detection and warnings
• ✅ Clean error formatting

---

The logging is now clean, concise, and gives you exactly what you need to find weak links! 🎯