serversdown/project-lyra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat: host-side MI50 runaway watchdog (guard A, staged for install)

Browse Source 
Independent Proxmox-host backstop to the in-app dream budget: a systemd timer
runs every ~2 min and stops lyra-brain if the MI50 is busy >=1hr continuously OR
junction >=97C for ~6 min, then pings Brian via ntfy. Trips on duration only
after a full hour so a legit ~40-min manual workload runs untouched. GPU temp/use
read from host rocm-smi; stop via 'pct exec 202 -- docker stop'.

Parsing + duration/temp decision logic dry-run-verified locally against real
rocm-smi output format (4 scenarios). NOT yet installed/live-verified — card is
off and Brian's away; install + trip-test per deploy/mi50-watchdog/README.md when
it's back.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015yrEb5qpPGv2FjyxrB7LLk
This commit is contained in:
serversdown
2026-07-04 19:05:32 +00:00
parent 3573ac8d79
commit c212099738
4 changed files with 162 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/mi50-watchdog/mi50-watchdog.service
+16
View File
@@ -0,0 +1,16 @@
[Unit]
Description=MI50 runaway watchdog (stop the llama.cpp backend if the GPU is busy too long or too hot)
After=network-online.target
[Service]
Type=oneshot
# Fill in your ntfy topic so it can ping Brian when it trips (leave URL empty to log only).
Environment=NTFY_URL=https://ntfy.sh
Environment=NTFY_TOPIC=CHANGE_ME
# Optional overrides (defaults shown):
# Environment=MAX_BUSY_SEC=3600
# Environment=TEMP_KILL_C=97
# Environment=TEMP_KILL_STREAK=3
# Environment=CTID=202
# Environment=CONTAINER=lyra-brain
ExecStart=/usr/local/sbin/mi50-watchdog.sh