feat: host-side MI50 runaway watchdog (guard A, staged for install)

Independent Proxmox-host backstop to the in-app dream budget: a systemd timer
runs every ~2 min and stops lyra-brain if the MI50 is busy >=1hr continuously OR
junction >=97C for ~6 min, then pings Brian via ntfy. Trips on duration only
after a full hour so a legit ~40-min manual workload runs untouched. GPU temp/use
read from host rocm-smi; stop via 'pct exec 202 -- docker stop'.

Parsing + duration/temp decision logic dry-run-verified locally against real
rocm-smi output format (4 scenarios). NOT yet installed/live-verified — card is
off and Brian's away; install + trip-test per deploy/mi50-watchdog/README.md when
it's back.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015yrEb5qpPGv2FjyxrB7LLk

This commit is contained in:

serversdown

2026-07-04 19:05:32 +00:00

parent 3573ac8d79

commit c212099738

4 changed files with 162 additions and 0 deletions

									
										deploy/mi50-watchdog/mi50-watchdog.timer
									
		+10
		
												View File
												
				@@ -0,0 +1,10 @@

				[Unit]

				Description=Run the MI50 runaway watchdog every 2 minutes

				[Timer]

				OnBootSec=2min

				OnUnitActiveSec=2min

				AccuracySec=15s

				[Install]

				WantedBy=timers.target