serversdown
c212099738
Independent Proxmox-host backstop to the in-app dream budget: a systemd timer runs every ~2 min and stops lyra-brain if the MI50 is busy >=1hr continuously OR junction >=97C for ~6 min, then pings Brian via ntfy. Trips on duration only after a full hour so a legit ~40-min manual workload runs untouched. GPU temp/use read from host rocm-smi; stop via 'pct exec 202 -- docker stop'. Parsing + duration/temp decision logic dry-run-verified locally against real rocm-smi output format (4 scenarios). NOT yet installed/live-verified — card is off and Brian's away; install + trip-test per deploy/mi50-watchdog/README.md when it's back. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015yrEb5qpPGv2FjyxrB7LLk
17 lines
557 B
Desktop File
17 lines
557 B
Desktop File
[Unit]
|
|
Description=MI50 runaway watchdog (stop the llama.cpp backend if the GPU is busy too long or too hot)
|
|
After=network-online.target
|
|
|
|
[Service]
|
|
Type=oneshot
|
|
# Fill in your ntfy topic so it can ping Brian when it trips (leave URL empty to log only).
|
|
Environment=NTFY_URL=https://ntfy.sh
|
|
Environment=NTFY_TOPIC=CHANGE_ME
|
|
# Optional overrides (defaults shown):
|
|
# Environment=MAX_BUSY_SEC=3600
|
|
# Environment=TEMP_KILL_C=97
|
|
# Environment=TEMP_KILL_STREAK=3
|
|
# Environment=CTID=202
|
|
# Environment=CONTAINER=lyra-brain
|
|
ExecStart=/usr/local/sbin/mi50-watchdog.sh
|