feat: wedge detection and automatic recovery orchestration
Adds the recovery layer for the NL-43 TCP wedge failure mode (control listener on 2255 dies while device/FTP stay alive — see SLM-stress-test/NL43_RX55_TCP_Wedge_Investigation_2026-02-18.md). - Failure classification: poll failures probed raw and classified as wedged (refused = stack alive, listener gone) vs offline (timeout). FTP-port probe upgrades wedge confidence to definitive. - RecoveryManager state machine: confirm wedge -> trigger reset -> wait boot -> reconnect -> resume measurement via start_cycle (keeps overwrite protection) -> full incident trail in device logs. - Pluggable reset backends: manual (notify, default) and webhook (HTTP to relay controller: Pi GPIO, Shelly, Tasmota, etc.). - Safety: auto_recovery_enabled off by default, 2-failure confirmation threshold, windowed attempt limit (2 per 6h), one recovery per device. - Endpoints: GET recovery/status, POST recovery/probe (diagnostic), POST recovery/trigger (manual, bypasses threshold). - Migration: migrate_add_recovery_fields.py - Tests: test_wedge_recovery.py — fake wedge-able NL-43 + fake relay webhook; 23 assertions covering classification, end-to-end recovery with resume, and gating. All passing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -23,6 +23,17 @@ class NL43Config(Base):
|
||||
poll_interval_seconds = Column(Integer, nullable=True, default=60) # Polling interval (10-3600 seconds)
|
||||
poll_enabled = Column(Boolean, default=True) # Enable/disable background polling for this device
|
||||
|
||||
# Wedge recovery configuration
|
||||
auto_recovery_enabled = Column(Boolean, default=False) # Master switch for automatic wedge recovery
|
||||
reset_backend = Column(String, default="manual") # Reset actuation: "manual" (notify only) or "webhook"
|
||||
reset_webhook_url = Column(String, nullable=True) # URL that triggers a power cycle (relay controller)
|
||||
reset_webhook_method = Column(String, default="POST") # HTTP method for webhook (GET or POST)
|
||||
recovery_boot_wait_seconds = Column(Integer, default=90) # Wait after reset before reconnect attempts
|
||||
recovery_reconnect_timeout = Column(Integer, default=180) # Max seconds to wait for device after boot wait
|
||||
recovery_auto_resume = Column(Boolean, default=True) # Restart measurement if it was running pre-wedge
|
||||
recovery_max_attempts = Column(Integer, default=2) # Max recovery attempts per window
|
||||
recovery_window_minutes = Column(Integer, default=360) # Attempt-limit window (default 6 hours)
|
||||
|
||||
|
||||
class NL43Status(Base):
|
||||
"""
|
||||
@@ -57,6 +68,14 @@ class NL43Status(Base):
|
||||
# FTP start time sync tracking
|
||||
start_time_sync_attempted = Column(Boolean, default=False) # True if FTP sync was attempted for current measurement
|
||||
|
||||
# Wedge detection and recovery tracking
|
||||
connection_state = Column(String, default="ok") # ok | wedged | degraded | offline
|
||||
last_wedge_at = Column(DateTime, nullable=True) # When the current/most recent wedge was detected
|
||||
wedge_count = Column(Integer, default=0) # Lifetime count of detected wedges
|
||||
recovery_state = Column(String, default="idle") # idle | resetting | waiting_boot | reconnecting | resuming | awaiting_manual_reset | failed
|
||||
last_recovery_at = Column(DateTime, nullable=True) # When the last recovery attempt started
|
||||
last_recovery_result = Column(Text, nullable=True) # Outcome summary of the last recovery attempt
|
||||
|
||||
|
||||
class DeviceLog(Base):
|
||||
"""
|
||||
|
||||
Reference in New Issue
Block a user