feat: wedge detection and automatic recovery orchestration

Adds the recovery layer for the NL-43 TCP wedge failure mode (control
listener on 2255 dies while device/FTP stay alive — see
SLM-stress-test/NL43_RX55_TCP_Wedge_Investigation_2026-02-18.md).

- Failure classification: poll failures probed raw and classified as
  wedged (refused = stack alive, listener gone) vs offline (timeout).
  FTP-port probe upgrades wedge confidence to definitive.
- RecoveryManager state machine: confirm wedge -> trigger reset ->
  wait boot -> reconnect -> resume measurement via start_cycle (keeps
  overwrite protection) -> full incident trail in device logs.
- Pluggable reset backends: manual (notify, default) and webhook
  (HTTP to relay controller: Pi GPIO, Shelly, Tasmota, etc.).
- Safety: auto_recovery_enabled off by default, 2-failure confirmation
  threshold, windowed attempt limit (2 per 6h), one recovery per device.
- Endpoints: GET recovery/status, POST recovery/probe (diagnostic),
  POST recovery/trigger (manual, bypasses threshold).
- Migration: migrate_add_recovery_fields.py
- Tests: test_wedge_recovery.py — fake wedge-able NL-43 + fake relay
  webhook; 23 assertions covering classification, end-to-end recovery
  with resume, and gating. All passing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-06-05 05:03:25 +00:00
parent 450509d210
commit 3c22f1d70c
7 changed files with 1205 additions and 0 deletions
+19
View File
@@ -23,6 +23,17 @@ class NL43Config(Base):
poll_interval_seconds = Column(Integer, nullable=True, default=60) # Polling interval (10-3600 seconds)
poll_enabled = Column(Boolean, default=True) # Enable/disable background polling for this device
# Wedge recovery configuration
auto_recovery_enabled = Column(Boolean, default=False) # Master switch for automatic wedge recovery
reset_backend = Column(String, default="manual") # Reset actuation: "manual" (notify only) or "webhook"
reset_webhook_url = Column(String, nullable=True) # URL that triggers a power cycle (relay controller)
reset_webhook_method = Column(String, default="POST") # HTTP method for webhook (GET or POST)
recovery_boot_wait_seconds = Column(Integer, default=90) # Wait after reset before reconnect attempts
recovery_reconnect_timeout = Column(Integer, default=180) # Max seconds to wait for device after boot wait
recovery_auto_resume = Column(Boolean, default=True) # Restart measurement if it was running pre-wedge
recovery_max_attempts = Column(Integer, default=2) # Max recovery attempts per window
recovery_window_minutes = Column(Integer, default=360) # Attempt-limit window (default 6 hours)
class NL43Status(Base):
"""
@@ -57,6 +68,14 @@ class NL43Status(Base):
# FTP start time sync tracking
start_time_sync_attempted = Column(Boolean, default=False) # True if FTP sync was attempted for current measurement
# Wedge detection and recovery tracking
connection_state = Column(String, default="ok") # ok | wedged | degraded | offline
last_wedge_at = Column(DateTime, nullable=True) # When the current/most recent wedge was detected
wedge_count = Column(Integer, default=0) # Lifetime count of detected wedges
recovery_state = Column(String, default="idle") # idle | resetting | waiting_boot | reconnecting | resuming | awaiting_manual_reset | failed
last_recovery_at = Column(DateTime, nullable=True) # When the last recovery attempt started
last_recovery_result = Column(Text, nullable=True) # Outcome summary of the last recovery attempt
class DeviceLog(Base):
"""