project-lyra/deploy/mi50-watchdog/mi50-watchdog.timer at dev - project-lyra - Serversdown Labs

serversdown/project-lyra

Files

T

serversdown c212099738 feat: host-side MI50 runaway watchdog (guard A, staged for install)

Independent Proxmox-host backstop to the in-app dream budget: a systemd timer
runs every ~2 min and stops lyra-brain if the MI50 is busy >=1hr continuously OR
junction >=97C for ~6 min, then pings Brian via ntfy. Trips on duration only
after a full hour so a legit ~40-min manual workload runs untouched. GPU temp/use
read from host rocm-smi; stop via 'pct exec 202 -- docker stop'.

Parsing + duration/temp decision logic dry-run-verified locally against real
rocm-smi output format (4 scenarios). NOT yet installed/live-verified — card is
off and Brian's away; install + trip-test per deploy/mi50-watchdog/README.md when
it's back.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015yrEb5qpPGv2FjyxrB7LLk

2026-07-04 19:05:32 +00:00

11 lines

160 B

SYSTEMD

Raw Permalink Blame History

 [Unit]
 Description=Run the MI50 runaway watchdog every 2 minutes
 [Timer]
 OnBootSec=2min
 OnUnitActiveSec=2min
 AccuracySec=15s
 [Install]
 WantedBy=timers.target