serversdown
c212099738
Independent Proxmox-host backstop to the in-app dream budget: a systemd timer runs every ~2 min and stops lyra-brain if the MI50 is busy >=1hr continuously OR junction >=97C for ~6 min, then pings Brian via ntfy. Trips on duration only after a full hour so a legit ~40-min manual workload runs untouched. GPU temp/use read from host rocm-smi; stop via 'pct exec 202 -- docker stop'. Parsing + duration/temp decision logic dry-run-verified locally against real rocm-smi output format (4 scenarios). NOT yet installed/live-verified — card is off and Brian's away; install + trip-test per deploy/mi50-watchdog/README.md when it's back. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015yrEb5qpPGv2FjyxrB7LLk
55 lines
2.2 KiB
Markdown
55 lines
2.2 KiB
Markdown
# MI50 runaway watchdog (fallback layer "A")
|
|
|
|
Independent host-side backstop to Lyra's in-app dream-cycle budget (layer "C",
|
|
`lyra/dream.py`). Stops the llama.cpp backend if the MI50 is busy too long or too
|
|
hot, and pings Brian. See
|
|
`docs/superpowers/specs/2026-07-04-mi50-runaway-guards-design.md`.
|
|
|
|
## What it does
|
|
|
|
Runs on the **Proxmox host** (`10.0.0.4`) via a systemd timer, every ~2 min:
|
|
|
|
- **Duration:** if the GPU is busy (`rocm-smi` use% > 0) for **3600s continuously**,
|
|
it stops the container. Any idle read resets the streak, so a legitimate ~40-min
|
|
manual workload never trips it.
|
|
- **Temperature:** if junction ≥ **97°C** for **3 consecutive checks (~6 min)**, it
|
|
stops the container — independent of duration.
|
|
- On either trip: `pct exec 202 -- docker stop lyra-brain`, clear state, `logger` a
|
|
line, and POST to your ntfy topic.
|
|
|
|
All thresholds are `Environment=` overrides in the `.service`.
|
|
|
|
## Install (on the Proxmox host, as root)
|
|
|
|
```sh
|
|
# copy the three files up (from the repo, on lyra-cortex):
|
|
scp -i ~/.ssh/id_lyra_proxmox deploy/mi50-watchdog/mi50-watchdog.sh \
|
|
root@10.0.0.4:/usr/local/sbin/mi50-watchdog.sh
|
|
scp -i ~/.ssh/id_lyra_proxmox deploy/mi50-watchdog/mi50-watchdog.{service,timer} \
|
|
root@10.0.0.4:/etc/systemd/system/
|
|
|
|
# on the host:
|
|
chmod +x /usr/local/sbin/mi50-watchdog.sh
|
|
# set your ntfy topic (same one Lyra uses) in the service:
|
|
sed -i 's/CHANGE_ME/YOUR_NTFY_TOPIC/' /etc/systemd/system/mi50-watchdog.service
|
|
systemctl daemon-reload
|
|
systemctl enable --now mi50-watchdog.timer
|
|
```
|
|
|
|
## Verify (when the card is back and healthy)
|
|
|
|
```sh
|
|
# dry run once, watch what it decides:
|
|
NTFY_URL= /usr/local/sbin/mi50-watchdog.sh; echo "exit $?"
|
|
journalctl -t mi50-watchdog -n 20 --no-pager
|
|
|
|
# force a trip test with tiny thresholds (won't touch a healthy idle card unless busy):
|
|
MAX_BUSY_SEC=60 TEMP_KILL_C=40 TEMP_KILL_STREAK=1 /usr/local/sbin/mi50-watchdog.sh
|
|
# confirm it stopped lyra-brain + sent the ntfy, then restart the container.
|
|
|
|
systemctl list-timers mi50-watchdog.timer # confirm it's scheduled
|
|
```
|
|
|
|
**Not yet installed / live-verified** — staged here on 2026-07-04 while the card is
|
|
off and Brian is away. Install + trip-test when the MI50 is back.
|