# MI50 runaway watchdog (fallback layer "A")

Independent host-side backstop to Lyra's in-app dream-cycle budget (layer "C",
`lyra/dream.py`). Stops the llama.cpp backend if the MI50 is busy too long or too
hot, and pings Brian. See
`docs/superpowers/specs/2026-07-04-mi50-runaway-guards-design.md`.

## What it does

Runs on the **Proxmox host** (`10.0.0.4`) via a systemd timer, every ~2 min:

- **Duration:** if the GPU is busy (`rocm-smi` use% > 0) for **3600s continuously**,
  it stops the container. Any idle read resets the streak, so a legitimate ~40-min
  manual workload never trips it.
- **Temperature:** if junction ≥ **97°C** for **3 consecutive checks (~6 min)**, it
  stops the container — independent of duration.
- On either trip: `pct exec 202 -- docker stop lyra-brain`, clear state, `logger` a
  line, and POST to your ntfy topic.

All thresholds are `Environment=` overrides in the `.service`.

## Install (on the Proxmox host, as root)

```sh
# copy the three files up (from the repo, on lyra-cortex):
scp -i ~/.ssh/id_lyra_proxmox deploy/mi50-watchdog/mi50-watchdog.sh \
    root@10.0.0.4:/usr/local/sbin/mi50-watchdog.sh
scp -i ~/.ssh/id_lyra_proxmox deploy/mi50-watchdog/mi50-watchdog.{service,timer} \
    root@10.0.0.4:/etc/systemd/system/

# on the host:
chmod +x /usr/local/sbin/mi50-watchdog.sh
# set your ntfy topic (same one Lyra uses) in the service:
sed -i 's/CHANGE_ME/YOUR_NTFY_TOPIC/' /etc/systemd/system/mi50-watchdog.service
systemctl daemon-reload
systemctl enable --now mi50-watchdog.timer
```

## Verify (when the card is back and healthy)

```sh
# dry run once, watch what it decides:
NTFY_URL= /usr/local/sbin/mi50-watchdog.sh; echo "exit $?"
journalctl -t mi50-watchdog -n 20 --no-pager

# force a trip test with tiny thresholds (won't touch a healthy idle card unless busy):
MAX_BUSY_SEC=60 TEMP_KILL_C=40 TEMP_KILL_STREAK=1 /usr/local/sbin/mi50-watchdog.sh
# confirm it stopped lyra-brain + sent the ntfy, then restart the container.

systemctl list-timers mi50-watchdog.timer   # confirm it's scheduled
```

**Not yet installed / live-verified** — staged here on 2026-07-04 while the card is
off and Brian is away. Install + trip-test when the MI50 is back.