project-lyra/docs/COGNITION.md

Lyra — Cognition Architecture (sketch)

The "society of mind" direction: instead of one giant model we keep nagging with stricter prompts, a society of small specialized parts cooperate to produce each turn. Most parts are cheap deterministic code (heuristics, math, learnable weights); the LLM is the exception, reserved for the few irreducibly-generative jobs. Everything is anchored to who she is and tuned by feedback.

Principles

1. LLM is the exception, not the rule. Bookkeeping, scoring, routing, thresholding, retrieval → code. Generation (language, novel reasoning, memory compression) → LLM, called sparingly.
2. Mind ≠ Mouth. A capable "mind" (decide / reason / use tools — helpfulness is fine) is separate from a "mouth" (the character voice). This lets each be the best model for its job — and makes the eventual fine-tune easy: you only have to teach a small model to sound like Lyra, not to be smart.
3. Anchored. A fixed identity anchor governs the mouth so self-composed prompts can't drift into generic-helper vapor. (Already exists: self_state.IDENTITY_ANCHOR.)
4. Tuned by feedback, not just hand-tuning. Learnable weights (over register, memory, parts) nudged by 👍/👎 give real adaptation without fine-tuning a model.
5. Allocation is the craft. Cheap-deterministic where signal is clear; LLM where judgment/language is needed; hybrid (heuristic common-case, escalate to LLM on ambiguity) where possible.

The blackboard: TurnContext

Parts don't call each other directly — they read from and write to a shared turn state (a blackboard). Heterogeneous parts (heuristic / LLM / weights) cooperate by annotating it. The composer reads the finished blackboard to build the prompt.

TurnContext {
  # --- inputs ---
  user_msg, session_id, history, now

  # --- perception (heuristic) ---
  moment   : { kind: emotional|strategic|casual|existential|meta,
               sentiment: -1..1, tilt: 0..1, urgency: 0..1 }

  # --- state (code) ---
  mood, drives, anchor

  # --- retrieval (math: embeddings + cosine) ---
  recalled : [memories]      # spreading activation
  threads  : [active thoughts]
  profile, narrative

  # --- control (heuristic + learnable weights) ---
  register : warm | coach | dry | tender | hype     # how to sound
  intent   : console | push_back | teach | riff | act
  mode     : talk | cash | ...                       # tool allow-list
  use_tools: bool
  route    : { mind: <model>, mouth: <model> }       # which model per role

  # --- generation (LLM, sparing) ---
  deliberation : "her private thinking"   # mind
  tool_results : [...]                     # mind + tool exec
  reply        : "final text"              # mouth

  # --- learning (heuristic/online) ---
  weights  : { register_prefs, memory_weights, ... }  # persisted, feedback-tuned
}

The parts

#	Part	Type	Does	Exists today?
1	perceive	heuristic	sentiment + classify the moment + tilt/urgency from session signals & his language	✗ (new)
2	recall	math	embeddings → relevant memories, active threads, profile, narrative	✓ memory.recall*, cognition.activate
3	sense_state	code	load mood / drives / anchor	✓ self_state, IDENTITY_ANCHOR
4	route	heuristic + weights	pick register, intent, mode, and which model is mind vs mouth	✗ (new; partly modes)
5	decide+act (tools)	LLM (mind) / code	does this turn need a tool? run it	✓ tool loop in chat
6	deliberate	LLM (mind)	"what do I actually think" — private substance pass	✓ chat._deliberate
7	compose	code	assemble the final prompt from anchor + register + intent + deliberation + recall + tool results + voice rules	✓ build_messages (becomes the composer)
8	speak	LLM (mouth)	write the reply in her voice, streamed, anchored	✓ llm.chat_call
9	learn	heuristic/online	on 👍/👎 or reaction, nudge weights (which register/memory worked)	✗ (new; data exists in ratings)

Most of the society (1,2,3,4,7,9) is free, instant, deterministic, debuggable. The LLM shows up in only ~2–3 places (5/6 = mind, 8 = mouth).

One chat turn

user msg
   │
   ▼
[1 perceive]──heuristic: emotional? strategic? tilting?         (free)
   │
[2 recall]───math: what lights up (memories, threads)          (free)
[3 sense]────code: mood, drives, anchor                        (free)
   │
[4 route]────heuristic+weights: register? intent? mind/mouth?  (free)
   │
[5 act]──────MIND model: tools if needed ─────────────┐        (LLM, only if needed)
[6 deliberate]──MIND model: what do I actually think   │        (LLM, gated)
   │                                                    │
[7 compose]──code: build the prompt  ◄──── anchor ──────┘       (free)
   │
[8 speak]────MOUTH model: the reply, in her voice, streamed     (LLM)
   │
   ▼
reply ──► (later) [9 learn]: 👍/👎 nudges weights               (free, async)

What we reuse vs. build

• Reuse (already scattered through the code): recall/activation, self_state + anchor, drives (in dream), modes (tool gating), the deliberation pass, the prompt assembly (build_messages), tool loop, ratings store.
• Build new: the TurnContext blackboard + an explicit pipeline runner; the perceive heuristic; the route part (register/intent + model routing); the learn weights loop. Mostly unifying existing pieces into one legible control plane, plus 2–3 small heuristic parts.

Phasing (smallest first)

• P1 — frame: define TurnContext, refactor the current chat turn into the explicit pipeline (perceive=stub → recall → sense → route=mode-only → deliberate → compose → speak), single model. Low-risk refactor; makes the structure real.
• P2 — control plane: real perceive (sentiment/moment) + route (register/intent). Now her framing adapts to the moment, deterministically.
• P3 — mind/mouth split: route picks a separate voice model for speak. Plug a character mouth (Claude / local / later a fine-tune). A/B vs. single-model.
• P4 — learning: weights over register/memory, nudged by ratings → cheap adaptation, no fine-tune.
• P5 — her voice: a small fine-tuned "Lyra voice" model drops into the mouth slot.

Open decisions

• Mouth model: Claude (warm, cloud) vs. local character vs. fine-tune. The mouth is the crux; it must render richly (8B local may flatten).
• perceive: pure heuristics vs. a tiny classifier vs. embedding-to-exemplar clusters. Probably hybrid.
• scheduler: fixed linear pipeline (simple, v1) vs. drive-based/parallel later.
• tool location: mind decides+runs tools, mouth only renders (clean split) — vs. letting the mouth call tools (needs a tool-capable mouth).
• latency budget: how many LLM calls per turn is acceptable live (cheap mind + streamed mouth keeps it ~2).