Add runbook for recovering wedged units and new scripts for device management
- Created a comprehensive runbook (`wedged_unit_recovery.md`) detailing the recovery process for units stuck in a call-home loop, including symptoms, recovery steps, and explanations of the failure mode. - Added `blind_stop.sh` script to send stop-monitoring commands in a tight loop for unresponsive devices. - Introduced `rescue_device.sh` script to disable Auto Call Home and erase events from a busy device. - Implemented `slow_drip.sh` script to send stop-monitoring frames at a slow rate to prevent UART overrun. - Developed `spam_stop.sh` script to rapidly send stop-monitoring commands to a device. - Created `watch_unit.sh` script for passive monitoring of device reachability, logging results over time.
This commit is contained in:
Executable
+48
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# Hammer a device with blind stop-monitoring sessions as fast as possible.
|
||||
# Single HTTP call kicks off the burst inside SFM (no per-attempt HTTP
|
||||
# overhead). Default: 10 seconds, ~500 ms per attempt = ~20 attempts/sec.
|
||||
#
|
||||
# Usage:
|
||||
# ./spam_stop.sh <host> [tcp_port] [duration_s]
|
||||
#
|
||||
# Examples:
|
||||
# ./spam_stop.sh 166.246.130.1 # 10s burst
|
||||
# ./spam_stop.sh 166.246.130.1 9034 30 # 30s burst
|
||||
# DURATION=60 CONNECT_TIMEOUT=0.2 ./spam_stop.sh 166.246.130.1
|
||||
#
|
||||
# Env:
|
||||
# SFM_BASE_URL Default: http://localhost:8200 (SFM direct).
|
||||
# Set to http://localhost:8001/api/sfm to route through
|
||||
# Terra-View's proxy — but note the proxy has a 60s
|
||||
# timeout, so long bursts need direct mode.
|
||||
# DURATION Default: 10 (seconds; arg 3 overrides)
|
||||
# CONNECT_TIMEOUT Default: 0.5 (seconds)
|
||||
# REPEAT Default: 3 (stop frames per TCP session)
|
||||
|
||||
set -u
|
||||
|
||||
host="${1:-}"
|
||||
tcp_port="${2:-9034}"
|
||||
duration="${3:-${DURATION:-10}}"
|
||||
|
||||
if [[ -z "$host" ]]; then
|
||||
echo "usage: $0 <host> [tcp_port] [duration_s]" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
base="${SFM_BASE_URL:-http://localhost:8200}"
|
||||
connect_timeout="${CONNECT_TIMEOUT:-0.5}"
|
||||
repeat="${REPEAT:-3}"
|
||||
|
||||
url="${base}/device/stop_monitoring_spam?host=${host}&tcp_port=${tcp_port}&duration_s=${duration}&connect_timeout=${connect_timeout}&repeat=${repeat}"
|
||||
|
||||
echo "spam_stop: target ${host}:${tcp_port} duration=${duration}s connect_timeout=${connect_timeout}s repeat=${repeat}"
|
||||
echo "spam_stop: POST ${url}"
|
||||
echo
|
||||
|
||||
# Give curl enough slack to wait out the duration plus a buffer
|
||||
max_time=$(awk -v d="$duration" 'BEGIN { printf "%d", d + 10 }')
|
||||
|
||||
curl -sS --max-time "$max_time" -X POST "$url"
|
||||
echo
|
||||
Reference in New Issue
Block a user