serversdown
3c22f1d70c
feat: wedge detection and automatic recovery orchestration
...
Adds the recovery layer for the NL-43 TCP wedge failure mode (control
listener on 2255 dies while device/FTP stay alive — see
SLM-stress-test/NL43_RX55_TCP_Wedge_Investigation_2026-02-18.md).
- Failure classification: poll failures probed raw and classified as
wedged (refused = stack alive, listener gone) vs offline (timeout).
FTP-port probe upgrades wedge confidence to definitive.
- RecoveryManager state machine: confirm wedge -> trigger reset ->
wait boot -> reconnect -> resume measurement via start_cycle (keeps
overwrite protection) -> full incident trail in device logs.
- Pluggable reset backends: manual (notify, default) and webhook
(HTTP to relay controller: Pi GPIO, Shelly, Tasmota, etc.).
- Safety: auto_recovery_enabled off by default, 2-failure confirmation
threshold, windowed attempt limit (2 per 6h), one recovery per device.
- Endpoints: GET recovery/status, POST recovery/probe (diagnostic),
POST recovery/trigger (manual, bypasses threshold).
- Migration: migrate_add_recovery_fields.py
- Tests: test_wedge_recovery.py — fake wedge-able NL-43 + fake relay
webhook; 23 assertions covering classification, end-to-end recovery
with resume, and gating. All passing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-06-05 05:03:25 +00:00
serversdown
claude