serversdown/project-lyra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

test(persona): parameterize replay eval by backend/model (EVAL_BACKEND/EVAL_MODEL)

Browse Source 
Lets the same safe-trigger prompts run through cloud, mi50 (Qwen), or
local (dolphin3:8b) for cross-model voice comparison.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G796GsLCvJQKVN7hwV2cDx
This commit is contained in:
serversdown
2026-07-13 01:12:55 +00:00
parent 488df9cb3d
commit a1c8cf0342
1 changed files with 8 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/persona_replay_eval.py
+8 -1
View File
@@ -1,8 +1,12 @@
"""Replay the exact prompts where Lyra went 'too safe' through the rewritten persona. """Replay the exact prompts where Lyra went 'too safe' through the rewritten persona.
Run: `uv run python scripts/persona_replay_eval.py` (cloud backend; needs OPENAI_API_KEY). Run: `uv run python scripts/persona_replay_eval.py` (cloud backend; needs OPENAI_API_KEY).
Pick a backend/model with env vars, e.g. `EVAL_BACKEND=mi50 uv run python …` or
`EVAL_BACKEND=local EVAL_MODEL=dolphin3:8b uv run python …`.
Eyeball each reply against the four tics: no menu, no tag-question closer, a side taken.""" Eyeball each reply against the four tics: no menu, no tag-question closer, a side taken."""
from __future__ import annotations from __future__ import annotations
import os
from lyra import persona, llm from lyra import persona, llm
# The real safe-trigger prompts from the diagnosed transcripts. # The real safe-trigger prompts from the diagnosed transcripts.
@@ -16,10 +20,13 @@ PROMPTS = [
def main() -> None: def main() -> None:
backend = os.getenv("EVAL_BACKEND", "cloud")
model = os.getenv("EVAL_MODEL") or None
system = persona.core_prompt() system = persona.core_prompt()
print(f"### backend={backend} model={model or '(default)'}")
for i, p in enumerate(PROMPTS, 1): for i, p in enumerate(PROMPTS, 1):
msgs = [{"role": "system", "content": system}, {"role": "user", "content": p}] msgs = [{"role": "system", "content": system}, {"role": "user", "content": p}]
reply = llm.complete(msgs, backend="cloud", model=None) reply = llm.complete(msgs, backend=backend, model=model)
print(f"\n{'='*80}\n[{i}] USER: {p}\nLYRA: {reply}\n") print(f"\n{'='*80}\n[{i}] USER: {p}\nLYRA: {reply}\n")