Merge pull request 'v0.3.0, persistent polling update. Persistent TCP connection pool with all features Connection pool diagnostics (API + UI) All 6 new environment variables Changes to health check, diagnostics, and DRD streaming Technical architecture details and cellular' (#2) from dev-persistent into main

Reviewed-on: #2

.gitignore

@@ -1,5 +1,7 @@
 /manuals/
 /data/
+/SLM-stress-test/stress_test_logs/
+/SLM-stress-test/tcpdump-runs/
 
 # Python cache
 __pycache__/
@@ -12,3 +14,5 @@ __pycache__/
 *.egg-info/
 dist/
 build/
+
+*.pcap

CHANGELOG.md

@@ -5,6 +5,70 @@ All notable changes to SLMM (Sound Level Meter Manager) will be documented in th
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.0] - 2026-02-17
+
+### Added
+
+#### Persistent TCP Connection Pool
+- **Connection reuse** - TCP connections are cached per device and reused across commands, eliminating repeated TCP handshakes over cellular modems
+- **OS-level TCP keepalive** - Configurable keepalive probes keep cellular NAT tables alive and detect dead connections early (default: probe after 15s idle, every 10s, 3 failures = dead)
+- **Transparent retry** - If a cached connection goes stale, the system automatically retries with a fresh connection so failures are never visible to the caller
+- **Stale connection detection** - Multi-layer detection via idle TTL, max age, transport state, and reader EOF checks
+- **Background cleanup** - Periodic task (every 30s) evicts expired connections from the pool
+- **Master switch** - Set `TCP_PERSISTENT_ENABLED=false` to revert to per-request connection behavior
+
+#### Connection Pool Diagnostics
+- `GET /api/nl43/_connections/status` - View pool configuration, active connections, age/idle times, and keepalive settings
+- `POST /api/nl43/_connections/flush` - Force-close all cached connections (useful for debugging)
+- **Connections tab on roster page** - Live UI showing pool config, active connections with age/idle/alive status, auto-refreshes every 5s, and flush button
+
+#### Environment Variables
+- `TCP_PERSISTENT_ENABLED` (default: `true`) - Master switch for persistent connections
+- `TCP_IDLE_TTL` (default: `300`) - Close idle connections after N seconds
+- `TCP_MAX_AGE` (default: `1800`) - Force reconnect after N seconds
+- `TCP_KEEPALIVE_IDLE` (default: `15`) - Seconds idle before keepalive probes start
+- `TCP_KEEPALIVE_INTERVAL` (default: `10`) - Seconds between keepalive probes
+- `TCP_KEEPALIVE_COUNT` (default: `3`) - Failed probes before declaring connection dead
+
+### Changed
+- **Health check endpoint** (`/health/devices`) - Now uses connection pool instead of opening throwaway TCP connections; checks for existing live connections first (zero-cost), only opens new connection through pool if needed
+- **Diagnostics endpoint** - Removed separate port 443 modem check (extra handshake waste); TCP reachability test now uses connection pool
+- **DRD streaming** - Streaming connections now get TCP keepalive options set; cached connections are evicted before opening dedicated streaming socket
+- **Default timeouts tuned for cellular** - Idle TTL raised to 300s (5 min), max age raised to 1800s (30 min) to survive typical polling intervals over cellular links
+
+### Technical Details
+
+#### Architecture
+- `ConnectionPool` class in `services.py` manages a single cached connection per device key (NL-43 only supports one TCP connection at a time)
+- Uses existing per-device asyncio locks and rate limiting — no changes to concurrency model
+- Pool is a module-level singleton initialized from environment variables at import time
+- Lifecycle managed via FastAPI lifespan: cleanup task starts on startup, all connections closed on shutdown
+- `_send_command_unlocked()` refactored to use acquire/release/discard pattern with single-retry fallback
+- Command parsing extracted to `_execute_command()` method for reuse between primary and retry paths
+
+#### Cellular Modem Optimizations
+- Keepalive probes at 15s prevent cellular NAT tables from expiring (typically 30-60s timeout)
+- 300s idle TTL ensures connections survive between polling cycles (default 60s interval)
+- 1800s max age allows a single socket to serve ~30 minutes of polling before forced reconnect
+- Health checks and diagnostics produce zero additional TCP handshakes when a pooled connection exists
+- Stale `$` prompt bytes drained from idle connections before command reuse
+
+### Breaking Changes
+None. This release is fully backward-compatible with v0.2.x. Set `TCP_PERSISTENT_ENABLED=false` for identical behavior to previous versions.
+
+---
+
+## [0.2.1] - 2026-01-23
+
+### Added
+- **Roster management**: UI and API endpoints for managing device rosters.
+- **Delete config endpoint**: Remove device configuration alongside cached status data.
+- **Scheduler hooks**: `start_cycle` and `stop_cycle` helpers for Terra-View scheduling integration.
+
+### Changed
+- **FTP logging**: Connection, authentication, and transfer phases now log explicitly.
+- **Documentation**: Reorganized docs/scripts and updated API notes for FTP/TCP verification.
+
 ## [0.2.0] - 2026-01-15
 
 ### Added
@@ -135,5 +199,7 @@ None. This release is fully backward-compatible with v0.1.x. All existing endpoi
 
 ## Version History Summary
 
+- **v0.3.0** (2026-02-17) - Persistent TCP connections with keepalive for cellular modem reliability
+- **v0.2.1** (2026-01-23) - Roster management, scheduler hooks, FTP logging, doc cleanup
 - **v0.2.0** (2026-01-15) - Background Polling System
 - **v0.1.0** (2025-12-XX) - Initial Release

README.md

@@ -1,6 +1,6 @@
 # SLMM - Sound Level Meter Manager
 
-**Version 0.2.0**
+**Version 0.3.0**
 
 Backend API service for controlling and monitoring Rion NL-43/NL-53 Sound Level Meters via TCP and FTP protocols.
 
@@ -8,12 +8,13 @@ Backend API service for controlling and monitoring Rion NL-43/NL-53 Sound Level
 
 SLMM is a standalone backend module that provides REST API routing and command translation for NL43/NL53 sound level meters. This service acts as a bridge between the hardware devices and frontend applications, handling all device communication, data persistence, and protocol management.
 
-**Note:** This is a backend-only service. Actual user interfacing is done via [SFM/Terra-View](https://github.com/your-org/terra-view) frontend applications.
+**Note:** This is a backend-only service. Actual user interfacing is done via customized front ends or cli.
 
 ## Features
 
-- **Background Polling** ⭐ NEW: Continuous automatic polling of devices with configurable intervals
-- **Offline Detection** ⭐ NEW: Automatic device reachability tracking with failure counters
+- **Persistent TCP Connections**: Cached per-device connections with OS-level keepalive, tuned for cellular modem reliability
+- **Background Polling**: Continuous automatic polling of devices with configurable intervals
+- **Offline Detection**: Automatic device reachability tracking with failure counters
 - **Device Management**: Configure and manage multiple NL43/NL53 devices
 - **Real-time Monitoring**: Stream live measurement data via WebSocket
 - **Measurement Control**: Start, stop, pause, resume, and reset measurements
@@ -22,36 +23,47 @@ SLMM is a standalone backend module that provides REST API routing and command t
 - **Device Configuration**: Manage frequency/time weighting, clock sync, and more
 - **Rate Limiting**: Automatic 1-second delay enforcement between device commands
 - **Persistent Storage**: SQLite database for device configs and measurement cache
+- **Connection Diagnostics**: Live UI and API endpoints for monitoring TCP connection pool status
 
 ## Architecture
 
 ```
 ┌─────────────────┐         ┌──────────────────────────────┐         ┌─────────────────┐
-│  Terra-View UI  │◄───────►│  SLMM API                    │◄───────►│  NL43/NL53      │
+│                 │◄───────►│  SLMM API                    │◄───────►│  NL43/NL53      │
 │  (Frontend)     │  HTTP   │  • REST Endpoints            │  TCP    │  Sound Meters   │
-└─────────────────┘         │  • WebSocket Streaming       │         └─────────────────┘
-                            │  • Background Poller ⭐ NEW  │                ▲
-                            └──────────────────────────────┘                │
-                                          │                         Continuous
-                                          ▼                          Polling
-                                  ┌──────────────┐                      │
-                                  │  SQLite DB   │◄─────────────────────┘
+└─────────────────┘         │  • WebSocket Streaming       │  (kept  │  (via cellular  │
+                            │  • Background Poller         │  alive) │   modem)        │
+                            │  • Connection Pool (v0.3)    │         └─────────────────┘
+                            └──────────────────────────────┘
+                                          │
+                                          ▼
+                                  ┌──────────────┐
+                                  │  SQLite DB   │
                                   │  • Config    │
                                   │  • Status    │
                                   └──────────────┘
 ```
 
+### Persistent TCP Connection Pool (v0.3.0)
+
+SLMM maintains persistent TCP connections to devices with OS-level keepalive, designed for reliable operation over cellular modems:
+
+- **Connection Reuse**: One cached TCP socket per device, reused across all commands (no repeated handshakes)
+- **TCP Keepalive**: Probes keep cellular NAT tables alive and detect dead connections early
+- **Transparent Retry**: Stale cached connections automatically retry with a fresh socket
+- **Configurable**: Idle TTL (300s), max age (1800s), and keepalive timing via environment variables
+- **Diagnostics**: Live UI on the roster page and API endpoints for monitoring pool status
+
 ### Background Polling (v0.2.0)
 
-SLMM now includes a background polling service that continuously queries devices and updates the status cache:
+Background polling service continuously queries devices and updates the status cache:
 
 - **Automatic Updates**: Devices are polled at configurable intervals (10-3600 seconds)
 - **Offline Detection**: Devices marked unreachable after 3 consecutive failures
 - **Per-Device Configuration**: Each device can have a custom polling interval
 - **Resource Efficient**: Dynamic sleep intervals and smart scheduling
-- **Graceful Shutdown**: Background task stops cleanly on service shutdown
 
-This makes Terra-View significantly more responsive - status requests return cached data instantly (<100ms) instead of waiting for device queries (1-2 seconds).
+Status requests return cached data instantly (<100ms) instead of waiting for device queries (1-2 seconds).
 
 ## Quick Start
 
@@ -96,9 +108,18 @@ Once running, visit:
 
 ### Environment Variables
 
+**Server:**
 - `PORT`: Server port (default: 8100)
 - `CORS_ORIGINS`: Comma-separated list of allowed origins (default: "*")
 
+**TCP Connection Pool:**
+- `TCP_PERSISTENT_ENABLED`: Enable persistent connections (default: "true")
+- `TCP_IDLE_TTL`: Close idle connections after N seconds (default: 300)
+- `TCP_MAX_AGE`: Force reconnect after N seconds (default: 1800)
+- `TCP_KEEPALIVE_IDLE`: Seconds idle before keepalive probes (default: 15)
+- `TCP_KEEPALIVE_INTERVAL`: Seconds between keepalive probes (default: 10)
+- `TCP_KEEPALIVE_COUNT`: Failed probes before declaring dead (default: 3)
+
 ### Database
 
 The SQLite database is automatically created at [data/slmm.db](data/slmm.db) on first run.
@@ -126,7 +147,7 @@ Logs are written to:
 | GET | `/api/nl43/{unit_id}/live` | Request fresh DOD data from device (bypasses cache) |
 | WS | `/api/nl43/{unit_id}/stream` | WebSocket stream for real-time DRD data |
 
-### Background Polling Configuration ⭐ NEW
+### Background Polling
 
 | Method | Endpoint | Description |
 |--------|----------|-------------|
@@ -134,6 +155,13 @@ Logs are written to:
 | PUT | `/api/nl43/{unit_id}/polling/config` | Update polling interval and enable/disable polling |
 | GET | `/api/nl43/_polling/status` | Get global polling status for all devices |
 
+### Connection Pool
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/api/nl43/_connections/status` | Get pool config, active connections, age/idle times |
+| POST | `/api/nl43/_connections/flush` | Force-close all cached TCP connections |
+
 ### Measurement Control
 
 | Method | Endpoint | Description |
@@ -255,6 +283,9 @@ Caches latest measurement snapshot:
 
 ### TCP Communication
 - Uses ASCII command protocol over TCP
+- Persistent connections with OS-level keepalive (tuned for cellular modems)
+- Connections cached per device and reused across commands
+- Transparent retry on stale connections
 - Enforces ≥1 second delay between commands to same device
 - Two-line response format:
   - Line 1: Result code (R+0000 for success)
@@ -320,6 +351,16 @@ curl http://localhost:8100/api/nl43/meter-001/polling/config
 curl http://localhost:8100/api/nl43/_polling/status
 ```
 
+### Check Connection Pool Status
+```bash
+curl http://localhost:8100/api/nl43/_connections/status | jq '.'
+```
+
+### Flush All Cached Connections
+```bash
+curl -X POST http://localhost:8100/api/nl43/_connections/flush
+```
+
 ### Verify Device Settings
 ```bash
 curl http://localhost:8100/api/nl43/meter-001/settings
@@ -388,11 +429,19 @@ See [API.md](API.md) for detailed integration examples.
 ## Troubleshooting
 
 ### Connection Issues
+- Check connection pool status: `curl http://localhost:8100/api/nl43/_connections/status`
+- Flush stale connections: `curl -X POST http://localhost:8100/api/nl43/_connections/flush`
 - Verify device IP address and port in configuration
 - Ensure device is on the same network
 - Check firewall rules allow TCP/FTP connections
 - Verify RX55 network adapter is properly configured on device
 
+### Cellular Modem Issues
+- If modem wedges from too many handshakes, ensure `TCP_PERSISTENT_ENABLED=true` (default)
+- Increase `TCP_IDLE_TTL` if connections expire between poll cycles
+- Keepalive probes (default: every 15s) keep NAT tables alive — adjust `TCP_KEEPALIVE_IDLE` if needed
+- Set `TCP_PERSISTENT_ENABLED=false` to disable pooling for debugging
+
 ### Rate Limiting
 - API automatically enforces 1-second delay between commands
 - If experiencing delays, this is normal device behavior
@@ -432,8 +481,17 @@ python set_ftp_credentials.py <unit_id> <username> <password>
 ```bash
 # Run comprehensive polling tests
 ./test_polling.sh [unit_id]
+
+# Test settings endpoint
+python3 test_settings_endpoint.py <unit_id>
+
+# Test sleep mode auto-disable
+python3 test_sleep_mode_auto_disable.py <unit_id>
 ```
 
+### Legacy Scripts
+Old migration scripts and manual polling tools have been moved to `archive/` for reference. See [archive/README.md](archive/README.md) for details.
+
 ## Contributing
 
 This is a standalone module kept separate from the SFM/Terra-View codebase. When contributing:

app/background_poller.py

@@ -15,7 +15,8 @@ from sqlalchemy.orm import Session
 
 from app.database import SessionLocal
 from app.models import NL43Config, NL43Status
-from app.services import NL43Client, persist_snapshot
+from app.services import NL43Client, persist_snapshot, sync_measurement_start_time_from_ftp
+from app.device_logger import log_device_event, cleanup_old_logs
 
 logger = logging.getLogger(__name__)
 
@@ -25,7 +26,7 @@ class BackgroundPoller:
     Background task that continuously polls NL43 devices and updates status cache.
 
     Features:
-    - Per-device configurable poll intervals (10-3600 seconds)
+    - Per-device configurable poll intervals (30 seconds to 6 hours)
     - Automatic offline detection (marks unreachable after 3 consecutive failures)
     - Dynamic sleep intervals based on device configurations
     - Graceful shutdown on application stop
@@ -36,6 +37,7 @@ class BackgroundPoller:
         self._task: Optional[asyncio.Task] = None
         self._running = False
         self._logger = logger
+        self._last_cleanup = None  # Track last log cleanup time
 
     async def start(self):
         """Start the background polling task."""
@@ -78,6 +80,15 @@ class BackgroundPoller:
             except Exception as e:
                 self._logger.error(f"Error in poll loop: {e}", exc_info=True)
 
+            # Run log cleanup once per hour
+            try:
+                now = datetime.utcnow()
+                if self._last_cleanup is None or (now - self._last_cleanup).total_seconds() > 3600:
+                    cleanup_old_logs()
+                    self._last_cleanup = now
+            except Exception as e:
+                self._logger.warning(f"Log cleanup failed: {e}")
+
             # Calculate dynamic sleep interval
             sleep_time = self._calculate_sleep_interval()
             self._logger.debug(f"Sleeping for {sleep_time} seconds until next poll cycle")
@@ -205,6 +216,71 @@ class BackgroundPoller:
             db.commit()
             self._logger.info(f"✓ Successfully polled {unit_id}")
 
+            # Log to device log
+            log_device_event(
+                unit_id, "INFO", "POLL",
+                f"Poll success: state={snap.measurement_state}, Leq={snap.leq}, Lp={snap.lp}",
+                db
+            )
+
+            # Check if device is measuring but has no start time recorded
+            # This happens if measurement was started before SLMM began polling
+            # or after a service restart
+            status = db.query(NL43Status).filter_by(unit_id=unit_id).first()
+
+            # Reset the sync flag when measurement stops (so next measurement can sync)
+            if status and status.measurement_state != "Start":
+                if status.start_time_sync_attempted:
+                    status.start_time_sync_attempted = False
+                    db.commit()
+                    self._logger.debug(f"Reset FTP sync flag for {unit_id} (measurement stopped)")
+                    log_device_event(unit_id, "DEBUG", "STATE", "Measurement stopped, reset FTP sync flag", db)
+
+            # Attempt FTP sync if:
+            # - Device is measuring
+            # - No start time recorded
+            # - FTP sync not already attempted for this measurement
+            # - FTP is configured
+            if (status and
+                status.measurement_state == "Start" and
+                status.measurement_start_time is None and
+                not status.start_time_sync_attempted and
+                cfg.ftp_enabled and
+                cfg.ftp_username and
+                cfg.ftp_password):
+
+                self._logger.info(
+                    f"Device {unit_id} is measuring but has no start time - "
+                    f"attempting FTP sync"
+                )
+                log_device_event(unit_id, "INFO", "SYNC", "Attempting FTP sync for measurement start time", db)
+
+                # Mark that we attempted sync (prevents repeated attempts on failure)
+                status.start_time_sync_attempted = True
+                db.commit()
+
+                try:
+                    synced = await sync_measurement_start_time_from_ftp(
+                        unit_id=unit_id,
+                        host=cfg.host,
+                        tcp_port=cfg.tcp_port,
+                        ftp_port=cfg.ftp_port or 21,
+                        ftp_username=cfg.ftp_username,
+                        ftp_password=cfg.ftp_password,
+                        db=db
+                    )
+                    if synced:
+                        self._logger.info(f"✓ FTP sync succeeded for {unit_id}")
+                        log_device_event(unit_id, "INFO", "SYNC", "FTP sync succeeded - measurement start time updated", db)
+                    else:
+                        self._logger.warning(f"FTP sync returned False for {unit_id}")
+                        log_device_event(unit_id, "WARNING", "SYNC", "FTP sync returned False", db)
+                except Exception as sync_err:
+                    self._logger.warning(
+                        f"FTP sync failed for {unit_id}: {sync_err}"
+                    )
+                    log_device_event(unit_id, "ERROR", "SYNC", f"FTP sync failed: {sync_err}", db)
+
         except Exception as e:
             # Failure - increment counter and potentially mark offline
             status.consecutive_failures += 1
@@ -217,11 +293,13 @@ class BackgroundPoller:
                     self._logger.warning(
                         f"Device {unit_id} marked unreachable after {status.consecutive_failures} failures: {error_msg}"
                     )
+                    log_device_event(unit_id, "ERROR", "POLL", f"Device marked UNREACHABLE after {status.consecutive_failures} failures: {error_msg}", db)
                 status.is_reachable = False
             else:
                 self._logger.warning(
                     f"Poll failed for {unit_id} (attempt {status.consecutive_failures}/3): {error_msg}"
                 )
+                log_device_event(unit_id, "WARNING", "POLL", f"Poll failed (attempt {status.consecutive_failures}/3): {error_msg}", db)
 
             db.commit()
 
@@ -230,8 +308,8 @@ class BackgroundPoller:
         Calculate the next sleep interval based on all device poll intervals.
 
         Returns a dynamic sleep time that ensures responsive polling:
-        - Minimum 10 seconds (prevents tight loops)
-        - Maximum 30 seconds (ensures responsiveness)
+        - Minimum 30 seconds (prevents tight loops)
+        - Maximum 300 seconds / 5 minutes (ensures reasonable responsiveness for long intervals)
         - Generally half the minimum device interval
 
         Returns:
@@ -245,14 +323,15 @@ class BackgroundPoller:
             ).all()
 
             if not configs:
-                return 30  # Default sleep when no devices configured
+                return 60  # Default sleep when no devices configured
 
             # Get all intervals
             intervals = [cfg.poll_interval_seconds or 60 for cfg in configs]
             min_interval = min(intervals)
 
-            # Use half the minimum interval, but cap between 10-30 seconds
-            sleep_time = max(10, min(30, min_interval // 2))
+            # Use half the minimum interval, but cap between 30-300 seconds
+            # This allows longer sleep times when polling intervals are long (e.g., hourly)
+            sleep_time = max(30, min(300, min_interval // 2))
 
             return sleep_time
 

app/device_logger.py

@@ -0,0 +1,277 @@
+"""
+Per-device logging system.
+
+Provides dual output: database entries for structured queries and file logs for backup.
+Each device gets its own log file in data/logs/{unit_id}.log with rotation.
+"""
+
+import logging
+import os
+from datetime import datetime, timedelta
+from logging.handlers import RotatingFileHandler
+from pathlib import Path
+from typing import Optional
+
+from sqlalchemy.orm import Session
+
+from app.database import SessionLocal
+from app.models import DeviceLog
+
+# Configure base logger
+logger = logging.getLogger(__name__)
+
+# Log directory (persisted in Docker volume)
+LOG_DIR = Path(os.path.dirname(os.path.dirname(__file__))) / "data" / "logs"
+LOG_DIR.mkdir(parents=True, exist_ok=True)
+
+# Per-device file loggers (cached)
+_device_file_loggers: dict = {}
+
+# Log retention (days)
+LOG_RETENTION_DAYS = int(os.getenv("LOG_RETENTION_DAYS", "7"))
+
+
+def _get_file_logger(unit_id: str) -> logging.Logger:
+    """Get or create a file logger for a specific device."""
+    if unit_id in _device_file_loggers:
+        return _device_file_loggers[unit_id]
+
+    # Create device-specific logger
+    device_logger = logging.getLogger(f"device.{unit_id}")
+    device_logger.setLevel(logging.DEBUG)
+
+    # Avoid duplicate handlers
+    if not device_logger.handlers:
+        # Create rotating file handler (5 MB max, keep 3 backups)
+        log_file = LOG_DIR / f"{unit_id}.log"
+        handler = RotatingFileHandler(
+            log_file,
+            maxBytes=5 * 1024 * 1024,  # 5 MB
+            backupCount=3,
+            encoding="utf-8"
+        )
+        handler.setLevel(logging.DEBUG)
+
+        # Format: timestamp [LEVEL] [CATEGORY] message
+        formatter = logging.Formatter(
+            "%(asctime)s [%(levelname)s] [%(category)s] %(message)s",
+            datefmt="%Y-%m-%d %H:%M:%S"
+        )
+        handler.setFormatter(formatter)
+        device_logger.addHandler(handler)
+
+        # Don't propagate to root logger
+        device_logger.propagate = False
+
+    _device_file_loggers[unit_id] = device_logger
+    return device_logger
+
+
+def log_device_event(
+    unit_id: str,
+    level: str,
+    category: str,
+    message: str,
+    db: Optional[Session] = None
+):
+    """
+    Log an event for a specific device.
+
+    Writes to both:
+    1. Database (DeviceLog table) for structured queries
+    2. File (data/logs/{unit_id}.log) for backup/debugging
+
+    Args:
+        unit_id: Device identifier
+        level: Log level (DEBUG, INFO, WARNING, ERROR)
+        category: Event category (TCP, FTP, POLL, COMMAND, STATE, SYNC)
+        message: Log message
+        db: Optional database session (creates one if not provided)
+    """
+    timestamp = datetime.utcnow()
+
+    # Write to file log
+    try:
+        file_logger = _get_file_logger(unit_id)
+        log_func = getattr(file_logger, level.lower(), file_logger.info)
+        # Pass category as extra for formatter
+        log_func(message, extra={"category": category})
+    except Exception as e:
+        logger.warning(f"Failed to write file log for {unit_id}: {e}")
+
+    # Write to database
+    close_db = False
+    try:
+        if db is None:
+            db = SessionLocal()
+            close_db = True
+
+        log_entry = DeviceLog(
+            unit_id=unit_id,
+            timestamp=timestamp,
+            level=level.upper(),
+            category=category.upper(),
+            message=message
+        )
+        db.add(log_entry)
+        db.commit()
+
+    except Exception as e:
+        logger.warning(f"Failed to write DB log for {unit_id}: {e}")
+        if db:
+            db.rollback()
+    finally:
+        if close_db and db:
+            db.close()
+
+
+def cleanup_old_logs(retention_days: Optional[int] = None, db: Optional[Session] = None):
+    """
+    Delete log entries older than retention period.
+
+    Args:
+        retention_days: Days to retain (default: LOG_RETENTION_DAYS env var or 7)
+        db: Optional database session
+    """
+    if retention_days is None:
+        retention_days = LOG_RETENTION_DAYS
+
+    cutoff = datetime.utcnow() - timedelta(days=retention_days)
+
+    close_db = False
+    try:
+        if db is None:
+            db = SessionLocal()
+            close_db = True
+
+        deleted = db.query(DeviceLog).filter(DeviceLog.timestamp < cutoff).delete()
+        db.commit()
+
+        if deleted > 0:
+            logger.info(f"Cleaned up {deleted} log entries older than {retention_days} days")
+
+    except Exception as e:
+        logger.error(f"Failed to cleanup old logs: {e}")
+        if db:
+            db.rollback()
+    finally:
+        if close_db and db:
+            db.close()
+
+
+def get_device_logs(
+    unit_id: str,
+    limit: int = 100,
+    offset: int = 0,
+    level: Optional[str] = None,
+    category: Optional[str] = None,
+    since: Optional[datetime] = None,
+    db: Optional[Session] = None
+) -> list:
+    """
+    Query log entries for a specific device.
+
+    Args:
+        unit_id: Device identifier
+        limit: Max entries to return (default: 100)
+        offset: Number of entries to skip (default: 0)
+        level: Filter by level (DEBUG, INFO, WARNING, ERROR)
+        category: Filter by category (TCP, FTP, POLL, COMMAND, STATE, SYNC)
+        since: Filter entries after this timestamp
+        db: Optional database session
+
+    Returns:
+        List of log entries as dicts
+    """
+    close_db = False
+    try:
+        if db is None:
+            db = SessionLocal()
+            close_db = True
+
+        query = db.query(DeviceLog).filter(DeviceLog.unit_id == unit_id)
+
+        if level:
+            query = query.filter(DeviceLog.level == level.upper())
+        if category:
+            query = query.filter(DeviceLog.category == category.upper())
+        if since:
+            query = query.filter(DeviceLog.timestamp >= since)
+
+        # Order by newest first
+        query = query.order_by(DeviceLog.timestamp.desc())
+
+        # Apply pagination
+        entries = query.offset(offset).limit(limit).all()
+
+        return [
+            {
+                "id": e.id,
+                "timestamp": e.timestamp.isoformat() + "Z",
+                "level": e.level,
+                "category": e.category,
+                "message": e.message
+            }
+            for e in entries
+        ]
+
+    finally:
+        if close_db and db:
+            db.close()
+
+
+def get_log_stats(unit_id: str, db: Optional[Session] = None) -> dict:
+    """
+    Get log statistics for a device.
+
+    Returns:
+        Dict with counts by level and category
+    """
+    close_db = False
+    try:
+        if db is None:
+            db = SessionLocal()
+            close_db = True
+
+        total = db.query(DeviceLog).filter(DeviceLog.unit_id == unit_id).count()
+
+        # Count by level
+        level_counts = {}
+        for level in ["DEBUG", "INFO", "WARNING", "ERROR"]:
+            count = db.query(DeviceLog).filter(
+                DeviceLog.unit_id == unit_id,
+                DeviceLog.level == level
+            ).count()
+            if count > 0:
+                level_counts[level] = count
+
+        # Count by category
+        category_counts = {}
+        for category in ["TCP", "FTP", "POLL", "COMMAND", "STATE", "SYNC", "GENERAL"]:
+            count = db.query(DeviceLog).filter(
+                DeviceLog.unit_id == unit_id,
+                DeviceLog.category == category
+            ).count()
+            if count > 0:
+                category_counts[category] = count
+
+        # Get oldest and newest
+        oldest = db.query(DeviceLog).filter(
+            DeviceLog.unit_id == unit_id
+        ).order_by(DeviceLog.timestamp.asc()).first()
+
+        newest = db.query(DeviceLog).filter(
+            DeviceLog.unit_id == unit_id
+        ).order_by(DeviceLog.timestamp.desc()).first()
+
+        return {
+            "total": total,
+            "by_level": level_counts,
+            "by_category": category_counts,
+            "oldest": oldest.timestamp.isoformat() + "Z" if oldest else None,
+            "newest": newest.timestamp.isoformat() + "Z" if newest else None
+        }
+
+    finally:
+        if close_db and db:
+            db.close()

app/main.py

@@ -29,7 +29,11 @@ logger.info("Database tables initialized")
 @asynccontextmanager
 async def lifespan(app: FastAPI):
     """Manage application lifecycle - startup and shutdown events."""
+    from app.services import _connection_pool
+
     # Startup
+    logger.info("Starting TCP connection pool cleanup task...")
+    _connection_pool.start_cleanup()
     logger.info("Starting background poller...")
     await poller.start()
     logger.info("Background poller started")
@@ -40,12 +44,15 @@ async def lifespan(app: FastAPI):
     logger.info("Stopping background poller...")
     await poller.stop()
     logger.info("Background poller stopped")
+    logger.info("Closing TCP connection pool...")
+    await _connection_pool.close_all()
+    logger.info("TCP connection pool closed")
 
 
 app = FastAPI(
     title="SLMM NL43 Addon",
     description="Standalone module for NL43 configuration and status APIs with background polling",
-    version="0.2.0",
+    version="0.3.0",
     lifespan=lifespan,
 )
 
@@ -72,6 +79,11 @@ def index(request: Request):
     return templates.TemplateResponse("index.html", {"request": request})
 
 
+@app.get("/roster", response_class=HTMLResponse)
+def roster(request: Request):
+    return templates.TemplateResponse("roster.html", {"request": request})
+
+
 @app.get("/health")
 async def health():
     """Basic health check endpoint."""
@@ -80,10 +92,14 @@ async def health():
 
 @app.get("/health/devices")
 async def health_devices():
-    """Enhanced health check that tests device connectivity."""
+    """Enhanced health check that tests device connectivity.
+
+    Uses the connection pool to avoid unnecessary TCP handshakes — if a
+    cached connection exists and is alive, the device is reachable.
+    """
     from sqlalchemy.orm import Session
     from app.database import SessionLocal
-    from app.services import NL43Client
+    from app.services import _connection_pool
     from app.models import NL43Config
 
     db: Session = SessionLocal()
@@ -93,7 +109,7 @@ async def health_devices():
         configs = db.query(NL43Config).filter_by(tcp_enabled=True).all()
 
         for cfg in configs:
-            client = NL43Client(cfg.host, cfg.tcp_port, timeout=2.0, ftp_username=cfg.ftp_username, ftp_password=cfg.ftp_password)
+            device_key = f"{cfg.host}:{cfg.tcp_port}"
             status = {
                 "unit_id": cfg.unit_id,
                 "host": cfg.host,
@@ -103,14 +119,22 @@ async def health_devices():
             }
 
             try:
-                # Try to connect (don't send command to avoid rate limiting issues)
-                import asyncio
-                reader, writer = await asyncio.wait_for(
-                    asyncio.open_connection(cfg.host, cfg.tcp_port), timeout=2.0
-                )
-                writer.close()
-                await writer.wait_closed()
+                # Check if pool already has a live connection (zero-cost check)
+                pool_stats = _connection_pool.get_stats()
+                conn_info = pool_stats["connections"].get(device_key)
+                if conn_info and conn_info["alive"]:
                     status["reachable"] = True
+                    status["source"] = "pool"
+                else:
+                    # No cached connection — do a lightweight acquire/release
+                    # This opens a connection if needed but keeps it in the pool
+                    import asyncio
+                    reader, writer, from_cache = await _connection_pool.acquire(
+                        device_key, cfg.host, cfg.tcp_port, timeout=2.0
+                    )
+                    await _connection_pool.release(device_key, reader, writer, cfg.host, cfg.tcp_port)
+                    status["reachable"] = True
+                    status["source"] = "cached" if from_cache else "new"
             except Exception as e:
                 status["error"] = str(type(e).__name__)
                 logger.warning(f"Device {cfg.unit_id} health check failed: {e}")

app/models.py

@@ -53,3 +53,22 @@ class NL43Status(Base):
     last_poll_attempt = Column(DateTime, nullable=True)  # Last time background poller attempted to poll
     last_success = Column(DateTime, nullable=True)  # Last successful poll timestamp
     last_error = Column(Text, nullable=True)  # Last error message (truncated to 500 chars)
+
+    # FTP start time sync tracking
+    start_time_sync_attempted = Column(Boolean, default=False)  # True if FTP sync was attempted for current measurement
+
+
+class DeviceLog(Base):
+    """
+    Per-device log entries for debugging and audit trail.
+    Stores events like commands, state changes, errors, and FTP operations.
+    """
+
+    __tablename__ = "device_logs"
+
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    unit_id = Column(String, index=True, nullable=False)
+    timestamp = Column(DateTime, default=func.now(), index=True)
+    level = Column(String, default="INFO")  # DEBUG, INFO, WARNING, ERROR
+    category = Column(String, default="GENERAL")  # TCP, FTP, POLL, COMMAND, STATE, SYNC
+    message = Column(Text, nullable=False)

app/routers.py

@@ -3,6 +3,7 @@ from fastapi.responses import FileResponse
 from sqlalchemy.orm import Session
 from datetime import datetime
 from pydantic import BaseModel, field_validator, Field
+from typing import Optional
 import logging
 import ipaddress
 import json
@@ -49,6 +50,8 @@ class ConfigPayload(BaseModel):
     ftp_username: str | None = None
     ftp_password: str | None = None
     web_enabled: bool | None = None
+    poll_enabled: bool | None = None
+    poll_interval_seconds: int | None = None
 
     @field_validator("host")
     @classmethod
@@ -76,13 +79,48 @@ class ConfigPayload(BaseModel):
             raise ValueError("Port must be between 1 and 65535")
         return v
 
+    @field_validator("poll_interval_seconds")
+    @classmethod
+    def validate_poll_interval(cls, v):
+        if v is not None and not (30 <= v <= 21600):
+            raise ValueError("Poll interval must be between 30 and 21600 seconds (30s to 6 hours)")
+        return v
+
 
 class PollingConfigPayload(BaseModel):
     """Payload for updating device polling configuration."""
-    poll_interval_seconds: int | None = Field(None, ge=10, le=3600, description="Polling interval in seconds (10-3600)")
+    poll_interval_seconds: int | None = Field(None, ge=30, le=21600, description="Polling interval in seconds (30s to 6 hours)")
     poll_enabled: bool | None = Field(None, description="Enable or disable background polling for this device")
 
 
+# ============================================================================
+# TCP CONNECTION POOL ENDPOINTS (must be before /{unit_id} routes)
+# ============================================================================
+
+@router.get("/_connections/status")
+async def get_connection_pool_status():
+    """Get status of the persistent TCP connection pool.
+
+    Returns information about cached connections, keepalive settings,
+    and per-device connection age/idle times.
+    """
+    from app.services import _connection_pool
+    return {"status": "ok", "pool": _connection_pool.get_stats()}
+
+
+@router.post("/_connections/flush")
+async def flush_connection_pool():
+    """Close all cached TCP connections.
+
+    Useful for debugging or forcing fresh connections to all devices.
+    """
+    from app.services import _connection_pool
+    await _connection_pool.close_all()
+    # Restart cleanup task since close_all cancels it
+    _connection_pool.start_cleanup()
+    return {"status": "ok", "message": "All cached connections closed"}
+
+
 # ============================================================================
 # GLOBAL POLLING STATUS ENDPOINT (must be before /{unit_id} routes)
 # ============================================================================
@@ -131,6 +169,164 @@ def get_global_polling_status(db: Session = Depends(get_db)):
     }
 
 
+@router.get("/roster")
+def get_roster(db: Session = Depends(get_db)):
+    """
+    Get list of all configured devices with their status.
+
+    Returns all NL43Config entries along with their associated status information.
+    Used by the roster page to display all devices in a table.
+
+    Note: Must be defined before /{unit_id} routes to avoid routing conflicts.
+    """
+    configs = db.query(NL43Config).all()
+
+    devices = []
+    for cfg in configs:
+        status = db.query(NL43Status).filter_by(unit_id=cfg.unit_id).first()
+
+        device_data = {
+            "unit_id": cfg.unit_id,
+            "host": cfg.host,
+            "tcp_port": cfg.tcp_port,
+            "ftp_port": cfg.ftp_port,
+            "tcp_enabled": cfg.tcp_enabled,
+            "ftp_enabled": cfg.ftp_enabled,
+            "ftp_username": cfg.ftp_username,
+            "ftp_password": cfg.ftp_password,
+            "web_enabled": cfg.web_enabled,
+            "poll_enabled": cfg.poll_enabled,
+            "poll_interval_seconds": cfg.poll_interval_seconds,
+            "status": None
+        }
+
+        if status:
+            device_data["status"] = {
+                "last_seen": status.last_seen.isoformat() if status.last_seen else None,
+                "measurement_state": status.measurement_state,
+                "is_reachable": status.is_reachable,
+                "consecutive_failures": status.consecutive_failures,
+                "last_success": status.last_success.isoformat() if status.last_success else None,
+                "last_error": status.last_error
+            }
+
+        devices.append(device_data)
+
+    return {
+        "status": "ok",
+        "devices": devices,
+        "total": len(devices)
+    }
+
+
+class RosterCreatePayload(BaseModel):
+    """Payload for creating a new device via roster."""
+    unit_id: str
+    host: str
+    tcp_port: int = 2255
+    ftp_port: int = 21
+    tcp_enabled: bool = True
+    ftp_enabled: bool = False
+    ftp_username: str | None = None
+    ftp_password: str | None = None
+    web_enabled: bool = False
+    poll_enabled: bool = True
+    poll_interval_seconds: int = 60
+
+    @field_validator("host")
+    @classmethod
+    def validate_host(cls, v):
+        if v is None:
+            return v
+        # Try to parse as IP address or hostname
+        try:
+            ipaddress.ip_address(v)
+        except ValueError:
+            # Not an IP, check if it's a valid hostname format
+            if not v or len(v) > 253:
+                raise ValueError("Invalid hostname length")
+            # Allow hostnames (basic validation)
+            if not all(c.isalnum() or c in ".-" for c in v):
+                raise ValueError("Host must be a valid IP address or hostname")
+        return v
+
+    @field_validator("tcp_port", "ftp_port")
+    @classmethod
+    def validate_port(cls, v):
+        if v is None:
+            return v
+        if not (1 <= v <= 65535):
+            raise ValueError("Port must be between 1 and 65535")
+        return v
+
+    @field_validator("poll_interval_seconds")
+    @classmethod
+    def validate_poll_interval(cls, v):
+        if v is not None and not (30 <= v <= 21600):
+            raise ValueError("Poll interval must be between 30 and 21600 seconds (30s to 6 hours)")
+        return v
+
+
+@router.post("/roster")
+async def create_device(payload: RosterCreatePayload, db: Session = Depends(get_db)):
+    """
+    Create a new device configuration via roster.
+
+    This endpoint allows creating a new device with all configuration options.
+    If a device with the same unit_id already exists, returns a 409 conflict.
+
+    Note: Must be defined before /{unit_id} routes to avoid routing conflicts.
+    """
+    # Check if device already exists
+    existing = db.query(NL43Config).filter_by(unit_id=payload.unit_id).first()
+    if existing:
+        raise HTTPException(
+            status_code=409,
+            detail=f"Device with unit_id '{payload.unit_id}' already exists. Use PUT /{payload.unit_id}/config to update."
+        )
+
+    # Create new config
+    cfg = NL43Config(
+        unit_id=payload.unit_id,
+        host=payload.host,
+        tcp_port=payload.tcp_port,
+        ftp_port=payload.ftp_port,
+        tcp_enabled=payload.tcp_enabled,
+        ftp_enabled=payload.ftp_enabled,
+        ftp_username=payload.ftp_username,
+        ftp_password=payload.ftp_password,
+        web_enabled=payload.web_enabled,
+        poll_enabled=payload.poll_enabled,
+        poll_interval_seconds=payload.poll_interval_seconds
+    )
+
+    db.add(cfg)
+    db.commit()
+    db.refresh(cfg)
+
+    logger.info(f"Created new device config for {payload.unit_id}")
+
+    # If TCP is enabled, automatically disable sleep mode
+    if cfg.tcp_enabled and cfg.host and cfg.tcp_port:
+        logger.info(f"TCP enabled for {payload.unit_id}, ensuring sleep mode is disabled")
+        client = NL43Client(cfg.host, cfg.tcp_port, ftp_username=cfg.ftp_username, ftp_password=cfg.ftp_password, ftp_port=cfg.ftp_port)
+        await ensure_sleep_mode_disabled(client, payload.unit_id)
+
+    return {
+        "status": "ok",
+        "message": f"Device {payload.unit_id} created successfully",
+        "data": {
+            "unit_id": cfg.unit_id,
+            "host": cfg.host,
+            "tcp_port": cfg.tcp_port,
+            "tcp_enabled": cfg.tcp_enabled,
+            "ftp_enabled": cfg.ftp_enabled,
+            "poll_enabled": cfg.poll_enabled,
+            "poll_interval_seconds": cfg.poll_interval_seconds
+        }
+    }
+
+
 # ============================================================================
 # DEVICE-SPECIFIC ENDPOINTS
 # ============================================================================
@@ -207,6 +403,10 @@ async def upsert_config(unit_id: str, payload: ConfigPayload, db: Session = Depe
         cfg.ftp_password = payload.ftp_password
     if payload.web_enabled is not None:
         cfg.web_enabled = payload.web_enabled
+    if payload.poll_enabled is not None:
+        cfg.poll_enabled = payload.poll_enabled
+    if payload.poll_interval_seconds is not None:
+        cfg.poll_interval_seconds = payload.poll_interval_seconds
 
     db.commit()
     db.refresh(cfg)
@@ -228,6 +428,8 @@ async def upsert_config(unit_id: str, payload: ConfigPayload, db: Session = Depe
             "tcp_enabled": cfg.tcp_enabled,
             "ftp_enabled": cfg.ftp_enabled,
             "web_enabled": cfg.web_enabled,
+            "poll_enabled": cfg.poll_enabled,
+            "poll_interval_seconds": cfg.poll_interval_seconds,
         },
     }
 
@@ -371,12 +573,6 @@ async def stop_measurement(unit_id: str, db: Session = Depends(get_db)):
     try:
         await client.stop()
         logger.info(f"Stopped measurement on unit {unit_id}")
-
-        # Query device status to update database with "Stop" state
-        snap = await client.request_dod()
-        snap.unit_id = unit_id
-        persist_snapshot(snap, db)
-
     except ConnectionError as e:
         logger.error(f"Failed to stop measurement on {unit_id}: {e}")
         raise HTTPException(status_code=502, detail="Failed to communicate with device")
@@ -386,9 +582,117 @@ async def stop_measurement(unit_id: str, db: Session = Depends(get_db)):
     except Exception as e:
         logger.error(f"Unexpected error stopping measurement on {unit_id}: {e}")
         raise HTTPException(status_code=500, detail="Internal server error")
+
+    # Query device status to update database — non-fatal if this fails
+    try:
+        snap = await client.request_dod()
+        snap.unit_id = unit_id
+        persist_snapshot(snap, db)
+    except Exception as e:
+        logger.warning(f"Stop succeeded but failed to update status for {unit_id}: {e}")
+
     return {"status": "ok", "message": "Measurement stopped"}
 
 
+# ============================================================================
+# CYCLE COMMANDS (for scheduled automation)
+# ============================================================================
+
+class StartCyclePayload(BaseModel):
+    """Payload for start_cycle endpoint."""
+    sync_clock: bool = Field(True, description="Whether to sync device clock to server time")
+
+
+class StopCyclePayload(BaseModel):
+    """Payload for stop_cycle endpoint."""
+    download: bool = Field(True, description="Whether to download measurement data")
+    download_path: str | None = Field(None, description="Custom path for ZIP file (optional)")
+
+
+@router.post("/{unit_id}/start-cycle")
+async def start_cycle(unit_id: str, payload: StartCyclePayload = None, db: Session = Depends(get_db)):
+    """
+    Execute complete start cycle for scheduled automation:
+    1. Sync device clock to server time (if sync_clock=True)
+    2. Find next safe index (increment, check overwrite, repeat if needed)
+    3. Start measurement
+
+    Use this instead of /start when automating scheduled measurements.
+    This ensures the device is properly prepared before recording begins.
+    """
+    cfg = db.query(NL43Config).filter_by(unit_id=unit_id).first()
+    if not cfg:
+        raise HTTPException(status_code=404, detail="NL43 config not found")
+
+    if not cfg.tcp_enabled:
+        raise HTTPException(status_code=403, detail="TCP communication is disabled for this device")
+
+    payload = payload or StartCyclePayload()
+    client = NL43Client(cfg.host, cfg.tcp_port, ftp_username=cfg.ftp_username, ftp_password=cfg.ftp_password, ftp_port=cfg.ftp_port or 21)
+
+    try:
+        # Ensure sleep mode is disabled before starting
+        await ensure_sleep_mode_disabled(client, unit_id)
+
+        # Execute the full start cycle
+        result = await client.start_cycle(sync_clock=payload.sync_clock)
+
+        # Update status in database
+        snap = await client.request_dod()
+        snap.unit_id = unit_id
+        persist_snapshot(snap, db)
+
+        logger.info(f"Start cycle completed for {unit_id}: index {result['old_index']} -> {result['new_index']}")
+        return {"status": "ok", "unit_id": unit_id, **result}
+
+    except Exception as e:
+        logger.error(f"Start cycle failed for {unit_id}: {e}")
+        raise HTTPException(status_code=502, detail=str(e))
+
+
+@router.post("/{unit_id}/stop-cycle")
+async def stop_cycle(unit_id: str, payload: StopCyclePayload = None, db: Session = Depends(get_db)):
+    """
+    Execute complete stop cycle for scheduled automation:
+    1. Stop measurement
+    2. Enable FTP
+    3. Download measurement folder (matching current index)
+    4. Verify download succeeded
+
+    Use this instead of /stop when automating scheduled measurements.
+    This ensures data is properly saved and downloaded before the next session.
+    """
+    cfg = db.query(NL43Config).filter_by(unit_id=unit_id).first()
+    if not cfg:
+        raise HTTPException(status_code=404, detail="NL43 config not found")
+
+    if not cfg.tcp_enabled:
+        raise HTTPException(status_code=403, detail="TCP communication is disabled for this device")
+
+    payload = payload or StopCyclePayload()
+    client = NL43Client(cfg.host, cfg.tcp_port, ftp_username=cfg.ftp_username, ftp_password=cfg.ftp_password, ftp_port=cfg.ftp_port or 21)
+
+    try:
+        # Execute the full stop cycle
+        result = await client.stop_cycle(
+            download=payload.download,
+            download_path=payload.download_path,
+        )
+
+        # Update status in database
+        snap = await client.request_dod()
+        snap.unit_id = unit_id
+        persist_snapshot(snap, db)
+
+        logger.info(f"Stop cycle completed for {unit_id}: folder={result.get('downloaded_folder')}, success={result.get('download_success')}")
+        return {"status": "ok", "unit_id": unit_id, **result}
+
+    except Exception as e:
+        error_msg = str(e) if str(e) else f"{type(e).__name__}: No details available"
+        logger.error(f"Stop cycle failed for {unit_id}: {error_msg}")
+        raise HTTPException(status_code=502, detail=error_msg)
+
+
 @router.post("/{unit_id}/store")
 async def manual_store(unit_id: str, db: Session = Depends(get_db)):
     """Manually store measurement data to SD card."""
@@ -1451,74 +1755,38 @@ async def run_diagnostics(unit_id: str, db: Session = Depends(get_db)):
         "message": "TCP communication enabled"
     }
 
-    # Test 3: Modem/Router reachable (check port 443 HTTPS)
+    # Test 3: TCP connection reachable (device port) — uses connection pool
+    # This avoids extra TCP handshakes over cellular. If a cached connection
+    # exists and is alive, we skip the handshake entirely.
+    from app.services import _connection_pool
+    device_key = f"{cfg.host}:{cfg.tcp_port}"
     try:
-        reader, writer = await asyncio.wait_for(
-            asyncio.open_connection(cfg.host, 443), timeout=3.0
-        )
-        writer.close()
-        await writer.wait_closed()
-        diagnostics["tests"]["modem_reachable"] = {
+        pool_stats = _connection_pool.get_stats()
+        conn_info = pool_stats["connections"].get(device_key)
+        if conn_info and conn_info["alive"]:
+            # Pool already has a live connection — device is reachable
+            diagnostics["tests"]["tcp_connection"] = {
                 "status": "pass",
-            "message": f"Modem/router reachable at {cfg.host}"
+                "message": f"TCP connection alive in pool for {cfg.host}:{cfg.tcp_port}"
             }
-    except asyncio.TimeoutError:
-        diagnostics["tests"]["modem_reachable"] = {
-            "status": "fail",
-            "message": f"Modem/router timeout at {cfg.host} (network issue)"
-        }
-        diagnostics["overall_status"] = "fail"
-        return diagnostics
-    except ConnectionRefusedError:
-        # Connection refused means host is up but port 443 closed - that's ok
-        diagnostics["tests"]["modem_reachable"] = {
-            "status": "pass",
-            "message": f"Modem/router reachable at {cfg.host} (HTTPS closed)"
-        }
-    except Exception as e:
-        diagnostics["tests"]["modem_reachable"] = {
-            "status": "fail",
-            "message": f"Cannot reach modem/router at {cfg.host}: {str(e)}"
-        }
-        diagnostics["overall_status"] = "fail"
-        return diagnostics
-
-    # Test 4: TCP connection reachable (device port)
-    try:
-        reader, writer = await asyncio.wait_for(
-            asyncio.open_connection(cfg.host, cfg.tcp_port), timeout=3.0
+        else:
+            # Acquire through the pool (opens new if needed, keeps it cached)
+            reader, writer, from_cache = await _connection_pool.acquire(
+                device_key, cfg.host, cfg.tcp_port, timeout=3.0
             )
-        writer.close()
-        await writer.wait_closed()
+            await _connection_pool.release(device_key, reader, writer, cfg.host, cfg.tcp_port)
             diagnostics["tests"]["tcp_connection"] = {
                 "status": "pass",
                 "message": f"TCP connection successful to {cfg.host}:{cfg.tcp_port}"
             }
-    except asyncio.TimeoutError:
-        diagnostics["tests"]["tcp_connection"] = {
-            "status": "fail",
-            "message": f"Connection timeout to {cfg.host}:{cfg.tcp_port}"
-        }
-        diagnostics["overall_status"] = "fail"
-        return diagnostics
-    except ConnectionRefusedError:
-        diagnostics["tests"]["tcp_connection"] = {
-            "status": "fail",
-            "message": f"Connection refused by {cfg.host}:{cfg.tcp_port}"
-        }
-        diagnostics["overall_status"] = "fail"
-        return diagnostics
     except Exception as e:
         diagnostics["tests"]["tcp_connection"] = {
             "status": "fail",
-            "message": f"Connection error: {str(e)}"
+            "message": f"Connection error to {cfg.host}:{cfg.tcp_port}: {str(e)}"
         }
         diagnostics["overall_status"] = "fail"
         return diagnostics
 
-    # Wait a bit after connection test to let device settle
-    await asyncio.sleep(1.5)
-
     # Test 5: Device responds to commands
     # Use longer timeout to account for rate limiting (device requires ≥1s between commands)
     client = NL43Client(cfg.host, cfg.tcp_port, timeout=10.0, ftp_username=cfg.ftp_username, ftp_password=cfg.ftp_password)
@@ -1571,9 +1839,134 @@ async def run_diagnostics(unit_id: str, db: Session = Depends(get_db)):
 
     # All tests passed
     diagnostics["overall_status"] = "pass"
+
+    # Add database dump: config and status cache
+    diagnostics["database_dump"] = {
+        "config": {
+            "unit_id": cfg.unit_id,
+            "host": cfg.host,
+            "tcp_port": cfg.tcp_port,
+            "tcp_enabled": cfg.tcp_enabled,
+            "ftp_enabled": cfg.ftp_enabled,
+            "ftp_port": cfg.ftp_port,
+            "ftp_username": cfg.ftp_username,
+            "ftp_password": "***" if cfg.ftp_password else None,  # Mask password
+            "web_enabled": cfg.web_enabled,
+            "poll_interval_seconds": cfg.poll_interval_seconds,
+            "poll_enabled": cfg.poll_enabled
+        },
+        "status_cache": None
+    }
+
+    # Get cached status if available
+    status = db.query(NL43Status).filter_by(unit_id=unit_id).first()
+    if status:
+        # Helper to format datetime as ISO with Z suffix to indicate UTC
+        def to_utc_iso(dt):
+            return dt.isoformat() + 'Z' if dt else None
+
+        diagnostics["database_dump"]["status_cache"] = {
+            "unit_id": status.unit_id,
+            "last_seen": to_utc_iso(status.last_seen),
+            "measurement_state": status.measurement_state,
+            "measurement_start_time": to_utc_iso(status.measurement_start_time),
+            "counter": status.counter,
+            "lp": status.lp,
+            "leq": status.leq,
+            "lmax": status.lmax,
+            "lmin": status.lmin,
+            "lpeak": status.lpeak,
+            "battery_level": status.battery_level,
+            "power_source": status.power_source,
+            "sd_remaining_mb": status.sd_remaining_mb,
+            "sd_free_ratio": status.sd_free_ratio,
+            "is_reachable": status.is_reachable,
+            "consecutive_failures": status.consecutive_failures,
+            "last_poll_attempt": to_utc_iso(status.last_poll_attempt),
+            "last_success": to_utc_iso(status.last_success),
+            "last_error": status.last_error,
+            "raw_payload": status.raw_payload
+        }
+
     return diagnostics
 
 
+# ============================================================================
+# DEVICE LOGS ENDPOINTS
+# ============================================================================
+
+@router.get("/{unit_id}/logs")
+def get_device_logs(
+    unit_id: str,
+    limit: int = 100,
+    offset: int = 0,
+    level: Optional[str] = None,
+    category: Optional[str] = None,
+    db: Session = Depends(get_db)
+):
+    """
+    Get log entries for a specific device.
+
+    Query parameters:
+    - limit: Max entries to return (default: 100, max: 1000)
+    - offset: Number of entries to skip (for pagination)
+    - level: Filter by level (DEBUG, INFO, WARNING, ERROR)
+    - category: Filter by category (TCP, FTP, POLL, COMMAND, STATE, SYNC)
+
+    Returns newest entries first.
+    """
+    from app.device_logger import get_device_logs as fetch_logs, get_log_stats
+
+    # Validate limit
+    limit = min(limit, 1000)
+
+    logs = fetch_logs(
+        unit_id=unit_id,
+        limit=limit,
+        offset=offset,
+        level=level,
+        category=category,
+        db=db
+    )
+
+    stats = get_log_stats(unit_id, db)
+
+    return {
+        "status": "ok",
+        "unit_id": unit_id,
+        "logs": logs,
+        "count": len(logs),
+        "stats": stats,
+        "filters": {
+            "level": level,
+            "category": category
+        },
+        "pagination": {
+            "limit": limit,
+            "offset": offset
+        }
+    }
+
+
+@router.delete("/{unit_id}/logs")
+def clear_device_logs(unit_id: str, db: Session = Depends(get_db)):
+    """
+    Clear all log entries for a specific device.
+    """
+    from app.models import DeviceLog
+
+    deleted = db.query(DeviceLog).filter(DeviceLog.unit_id == unit_id).delete()
+    db.commit()
+
+    logger.info(f"Cleared {deleted} log entries for device {unit_id}")
+
+    return {
+        "status": "ok",
+        "message": f"Cleared {deleted} log entries for {unit_id}",
+        "deleted_count": deleted
+    }
+
+
 # ============================================================================
 # BACKGROUND POLLING CONFIGURATION ENDPOINTS
 # ============================================================================
@@ -1609,7 +2002,7 @@ def update_polling_config(
     """
     Update background polling configuration for a device.
 
-    Allows configuring the polling interval (10-3600 seconds) and
+    Allows configuring the polling interval (30-21600 seconds, i.e. 30s to 6 hours) and
     enabling/disabling automatic background polling per device.
 
     Changes take effect on the next polling cycle.
@@ -1620,10 +2013,15 @@ def update_polling_config(
 
     # Update interval if provided
     if payload.poll_interval_seconds is not None:
-        if payload.poll_interval_seconds < 10:
+        if payload.poll_interval_seconds < 30:
             raise HTTPException(
                 status_code=400,
-                detail="Polling interval must be at least 10 seconds"
+                detail="Polling interval must be at least 30 seconds"
+            )
+        if payload.poll_interval_seconds > 21600:
+            raise HTTPException(
+                status_code=400,
+                detail="Polling interval must be at most 21600 seconds (6 hours)"
             )
         cfg.poll_interval_seconds = payload.poll_interval_seconds
 

archive/README.md

@@ -0,0 +1,67 @@
+# SLMM Archive
+
+This directory contains legacy scripts that are no longer needed for normal operation but are preserved for reference.
+
+## Legacy Migrations (`legacy_migrations/`)
+
+These migration scripts were used during SLMM development (v0.1.x) to incrementally add database fields. They are **no longer needed** because:
+
+1. **Fresh databases** get the complete schema automatically from `app/models.py`
+2. **Existing databases** should already have these fields from previous runs
+3. **Current migration** is `migrate_add_polling_fields.py` (v0.2.0) in the parent directory
+
+### Archived Migration Files
+
+- `migrate_add_counter.py` - Added `counter` field to NL43Status
+- `migrate_add_measurement_start_time.py` - Added `measurement_start_time` field
+- `migrate_add_ftp_port.py` - Added `ftp_port` field to NL43Config
+- `migrate_field_names.py` - Renamed fields for consistency (one-time fix)
+- `migrate_revert_field_names.py` - Rollback for the rename migration
+
+**Do not delete** - These provide historical context for database schema evolution.
+
+---
+
+## Legacy Tools
+
+### `nl43_dod_poll.py`
+
+Manual polling script that queries a single NL-43 device for DOD (Device On-Demand) data.
+
+**Status**: Replaced by background polling system in v0.2.0
+
+**Why archived**:
+- Background poller (`app/background_poller.py`) now handles continuous polling automatically
+- No need for manual polling scripts
+- Kept for reference in case manual querying is needed for debugging
+
+**How to use** (if needed):
+```bash
+cd /home/serversdown/tmi/slmm/archive
+python3 nl43_dod_poll.py <host> <port> <unit_id>
+```
+
+---
+
+## Active Scripts (Still in Parent Directory)
+
+These scripts are **actively used** and documented in the main README:
+
+### Migrations
+- `migrate_add_polling_fields.py` - **v0.2.0 migration** - Adds background polling fields
+- `migrate_add_ftp_credentials.py` - **Legacy FTP migration** - Adds FTP auth fields
+
+### Testing
+- `test_polling.sh` - Comprehensive test suite for background polling features
+- `test_settings_endpoint.py` - Tests device settings API
+- `test_sleep_mode_auto_disable.py` - Tests automatic sleep mode handling
+
+### Utilities
+- `set_ftp_credentials.py` - Command-line tool to set FTP credentials for a device
+
+---
+
+## Version History
+
+- **v0.2.0** (2026-01-15) - Background polling system added, manual polling scripts archived
+- **v0.1.0** (2025-12-XX) - Initial release with incremental migrations

docs/API.md

@@ -483,7 +483,7 @@ POST /{unit_id}/ftp/enable
 ```
 Enables FTP server on the device.
 
-**Note:** FTP and TCP are mutually exclusive. Enabling FTP will temporarily disable TCP control.
+**Note:** ~~FTP and TCP are mutually exclusive. Enabling FTP will temporarily disable TCP control.~~ As of v0.2.0, FTP and TCP are working fine in tandem. Just dont spam them a bunch.
 
 ### Disable FTP
 ```

docs/ROSTER.md

@@ -0,0 +1,246 @@
+# SLMM Roster Management
+
+The SLMM standalone application now includes a roster management interface for viewing and configuring all Sound Level Meter devices.
+
+## Features
+
+### Web Interface
+
+Access the roster at: **http://localhost:8100/roster**
+
+The roster page provides:
+
+- **Device List Table**: View all configured SLMs with their connection details
+- **Real-time Status**: See device connectivity status (Online/Offline/Stale)
+- **Add Device**: Create new device configurations with a user-friendly modal form
+- **Edit Device**: Modify existing device configurations
+- **Delete Device**: Remove device configurations (does not affect physical devices)
+- **Test Connection**: Run diagnostics on individual devices
+
+### Table Columns
+
+| Column | Description |
+|--------|-------------|
+| Unit ID | Unique identifier for the device |
+| Host / IP | Device IP address or hostname |
+| TCP Port | TCP control port (default: 2255) |
+| FTP Port | FTP file transfer port (default: 21) |
+| TCP | Whether TCP control is enabled |
+| FTP | Whether FTP file transfer is enabled |
+| Polling | Whether background polling is enabled |
+| Status | Device connectivity status (Online/Offline/Stale) |
+| Actions | Test, Edit, Delete buttons |
+
+### Status Indicators
+
+- **Online** (green): Device responded within the last 5 minutes
+- **Stale** (yellow): Device hasn't responded recently but was seen before
+- **Offline** (red): Device is unreachable or has consecutive failures
+- **Unknown** (gray): No status data available yet
+
+## API Endpoints
+
+### List All Devices
+
+```bash
+GET /api/nl43/roster
+```
+
+Returns all configured devices with their status information.
+
+**Response:**
+```json
+{
+  "status": "ok",
+  "devices": [
+    {
+      "unit_id": "SLM-43-01",
+      "host": "192.168.1.100",
+      "tcp_port": 2255,
+      "ftp_port": 21,
+      "tcp_enabled": true,
+      "ftp_enabled": true,
+      "ftp_username": "USER",
+      "ftp_password": "0000",
+      "web_enabled": false,
+      "poll_enabled": true,
+      "poll_interval_seconds": 60,
+      "status": {
+        "last_seen": "2026-01-16T20:00:00",
+        "measurement_state": "Start",
+        "is_reachable": true,
+        "consecutive_failures": 0,
+        "last_success": "2026-01-16T20:00:00",
+        "last_error": null
+      }
+    }
+  ],
+  "total": 1
+}
+```
+
+### Create New Device
+
+```bash
+POST /api/nl43/roster
+Content-Type: application/json
+
+{
+  "unit_id": "SLM-43-01",
+  "host": "192.168.1.100",
+  "tcp_port": 2255,
+  "ftp_port": 21,
+  "tcp_enabled": true,
+  "ftp_enabled": false,
+  "poll_enabled": true,
+  "poll_interval_seconds": 60
+}
+```
+
+**Required Fields:**
+- `unit_id`: Unique device identifier
+- `host`: IP address or hostname
+
+**Optional Fields:**
+- `tcp_port`: TCP control port (default: 2255)
+- `ftp_port`: FTP port (default: 21)
+- `tcp_enabled`: Enable TCP control (default: true)
+- `ftp_enabled`: Enable FTP transfers (default: false)
+- `ftp_username`: FTP username (only if ftp_enabled)
+- `ftp_password`: FTP password (only if ftp_enabled)
+- `poll_enabled`: Enable background polling (default: true)
+- `poll_interval_seconds`: Polling interval 10-3600 seconds (default: 60)
+
+**Response:**
+```json
+{
+  "status": "ok",
+  "message": "Device SLM-43-01 created successfully",
+  "data": {
+    "unit_id": "SLM-43-01",
+    "host": "192.168.1.100",
+    "tcp_port": 2255,
+    "tcp_enabled": true,
+    "ftp_enabled": false,
+    "poll_enabled": true,
+    "poll_interval_seconds": 60
+  }
+}
+```
+
+### Update Device
+
+```bash
+PUT /api/nl43/{unit_id}/config
+Content-Type: application/json
+
+{
+  "host": "192.168.1.101",
+  "tcp_port": 2255,
+  "poll_interval_seconds": 120
+}
+```
+
+All fields are optional. Only include fields you want to update.
+
+### Delete Device
+
+```bash
+DELETE /api/nl43/{unit_id}/config
+```
+
+Removes the device configuration and associated status data. Does not affect the physical device.
+
+**Response:**
+```json
+{
+  "status": "ok",
+  "message": "Deleted device SLM-43-01"
+}
+```
+
+## Usage Examples
+
+### Via Web Interface
+
+1. Navigate to http://localhost:8100/roster
+2. Click "Add Device" to create a new configuration
+3. Fill in the device details (unit ID, IP address, ports)
+4. Configure TCP, FTP, and polling settings
+5. Click "Save Device"
+6. Use "Test" button to verify connectivity
+7. Edit or delete devices as needed
+
+### Via API (curl)
+
+**Add a new device:**
+```bash
+curl -X POST http://localhost:8100/api/nl43/roster \
+  -H "Content-Type: application/json" \
+  -d '{
+    "unit_id": "slm-site-a",
+    "host": "192.168.1.100",
+    "tcp_port": 2255,
+    "tcp_enabled": true,
+    "ftp_enabled": true,
+    "ftp_username": "USER",
+    "ftp_password": "0000",
+    "poll_enabled": true,
+    "poll_interval_seconds": 60
+  }'
+```
+
+**Update device host:**
+```bash
+curl -X PUT http://localhost:8100/api/nl43/slm-site-a/config \
+  -H "Content-Type: application/json" \
+  -d '{"host": "192.168.1.101"}'
+```
+
+**Delete device:**
+```bash
+curl -X DELETE http://localhost:8100/api/nl43/slm-site-a/config
+```
+
+**List all devices:**
+```bash
+curl http://localhost:8100/api/nl43/roster | python3 -m json.tool
+```
+
+## Integration with Terra-View
+
+When SLMM is used as a module within Terra-View:
+
+1. Terra-View manages device configurations in its own database
+2. Terra-View syncs configurations to SLMM via `PUT /api/nl43/{unit_id}/config`
+3. Terra-View can query device status via `GET /api/nl43/{unit_id}/status`
+4. SLMM's roster page can be used for standalone testing and diagnostics
+
+## Background Polling
+
+Devices with `poll_enabled: true` are automatically polled at their configured interval:
+
+- Polls device status every `poll_interval_seconds` (10-3600 seconds)
+- Updates `NL43Status` table with latest measurements
+- Tracks device reachability and failure counts
+- Provides real-time status updates in the roster
+
+**Note**: Polling respects the NL43 protocol's 1-second rate limit between commands.
+
+## Validation
+
+The roster system validates:
+
+- **Unit ID**: Must be unique across all devices
+- **Host**: Valid IP address or hostname format
+- **Ports**: Must be between 1-65535
+- **Poll Interval**: Must be between 10-3600 seconds
+- **Duplicate Check**: Returns 409 Conflict if unit_id already exists
+
+## Notes
+
+- Deleting a device from the roster does NOT affect the physical device
+- Device configurations are stored in the SLMM database (`data/slmm.db`)
+- Status information is updated by the background polling system
+- The roster page auto-refreshes status indicators
+- Test button runs full diagnostics (connectivity, TCP, FTP if enabled)

docs/features/README.md

@@ -0,0 +1,26 @@
+# SLMM Feature Documentation
+
+This directory contains detailed documentation for specific SLMM features and enhancements.
+
+## Feature Documents
+
+### FEATURE_SUMMARY.md
+Overview of all major features in SLMM.
+
+### SETTINGS_ENDPOINT.md
+Documentation of the device settings endpoint and verification system.
+
+### TIMEZONE_CONFIGURATION.md
+Timezone handling and configuration for SLMM timestamps.
+
+### SLEEP_MODE_AUTO_DISABLE.md  
+Automatic sleep mode wake-up system for background polling.
+
+### UI_UPDATE.md
+UI/UX improvements and interface updates.
+
+## Related Documentation
+
+- [../README.md](../../README.md) - Main SLMM documentation
+- [../CHANGELOG.md](../../CHANGELOG.md) - Version history
+- [../API.md](../../API.md) - Complete API reference

migrate_add_device_logs.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+"""
+Database migration: Add device_logs table.
+
+This table stores per-device log entries for debugging and audit trail.
+
+Run this once to add the new table.
+"""
+
+import sqlite3
+import os
+
+# Path to the SLMM database
+DB_PATH = os.path.join(os.path.dirname(__file__), "data", "slmm.db")
+
+
+def migrate():
+    print(f"Adding device_logs table to: {DB_PATH}")
+
+    if not os.path.exists(DB_PATH):
+        print("Database does not exist yet. Table will be created automatically on first run.")
+        return
+
+    conn = sqlite3.connect(DB_PATH)
+    cursor = conn.cursor()
+
+    try:
+        # Check if table already exists
+        cursor.execute("""
+            SELECT name FROM sqlite_master
+            WHERE type='table' AND name='device_logs'
+        """)
+        if cursor.fetchone():
+            print("✓ device_logs table already exists, no migration needed")
+            return
+
+        # Create the table
+        print("Creating device_logs table...")
+        cursor.execute("""
+            CREATE TABLE device_logs (
+                id INTEGER PRIMARY KEY AUTOINCREMENT,
+                unit_id VARCHAR NOT NULL,
+                timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
+                level VARCHAR DEFAULT 'INFO',
+                category VARCHAR DEFAULT 'GENERAL',
+                message TEXT NOT NULL
+            )
+        """)
+
+        # Create indexes for efficient querying
+        print("Creating indexes...")
+        cursor.execute("CREATE INDEX ix_device_logs_unit_id ON device_logs (unit_id)")
+        cursor.execute("CREATE INDEX ix_device_logs_timestamp ON device_logs (timestamp)")
+
+        conn.commit()
+        print("✓ Created device_logs table with indexes")
+
+        # Verify
+        cursor.execute("""
+            SELECT name FROM sqlite_master
+            WHERE type='table' AND name='device_logs'
+        """)
+        if not cursor.fetchone():
+            raise Exception("device_logs table was not created successfully")
+
+        print("✓ Migration completed successfully")
+
+    finally:
+        conn.close()
+
+
+if __name__ == "__main__":
+    migrate()

migrate_add_start_time_sync_flag.py

@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+"""
+Database migration: Add start_time_sync_attempted field to nl43_status table.
+
+This field tracks whether FTP sync has been attempted for the current measurement,
+preventing repeated sync attempts when FTP fails.
+
+Run this once to add the new column.
+"""
+
+import sqlite3
+import os
+
+# Path to the SLMM database
+DB_PATH = os.path.join(os.path.dirname(__file__), "data", "slmm.db")
+
+
+def migrate():
+    print(f"Adding start_time_sync_attempted field to: {DB_PATH}")
+
+    if not os.path.exists(DB_PATH):
+        print("Database does not exist yet. Column will be created automatically.")
+        return
+
+    conn = sqlite3.connect(DB_PATH)
+    cursor = conn.cursor()
+
+    try:
+        # Check if column already exists
+        cursor.execute("PRAGMA table_info(nl43_status)")
+        columns = [col[1] for col in cursor.fetchall()]
+
+        if 'start_time_sync_attempted' in columns:
+            print("✓ start_time_sync_attempted column already exists, no migration needed")
+            return
+
+        # Add the column
+        print("Adding start_time_sync_attempted column...")
+        cursor.execute("""
+            ALTER TABLE nl43_status
+            ADD COLUMN start_time_sync_attempted BOOLEAN DEFAULT 0
+        """)
+        conn.commit()
+        print("✓ Added start_time_sync_attempted column")
+
+        # Verify
+        cursor.execute("PRAGMA table_info(nl43_status)")
+        columns = [col[1] for col in cursor.fetchall()]
+
+        if 'start_time_sync_attempted' not in columns:
+            raise Exception("start_time_sync_attempted column was not added successfully")
+
+        print("✓ Migration completed successfully")
+
+    finally:
+        conn.close()
+
+
+if __name__ == "__main__":
+    migrate()

templates/index.html

@@ -31,6 +31,11 @@
 <body>
   <h1>SLMM NL43 Standalone</h1>
   <p>Configure a unit (host/port), then use controls to Start/Stop and fetch live status.</p>
+  <p style="margin-bottom: 16px;">
+    <a href="/roster" style="color: #0969da; text-decoration: none; font-weight: 600;">📊 View Device Roster</a>
+    <span style="margin: 0 8px; color: #d0d7de;">|</span>
+    <a href="/docs" style="color: #0969da; text-decoration: none;">API Documentation</a>
+  </p>
 
   <fieldset>
     <legend>🔍 Connection Diagnostics</legend>
@@ -40,13 +45,34 @@
   </fieldset>
 
   <fieldset>
-    <legend>Unit Config</legend>
+    <legend>Unit Selection & Config</legend>
+
+    <div style="display: flex; gap: 8px; align-items: flex-end; margin-bottom: 12px;">
+      <div style="flex: 1;">
+        <label>Select Device</label>
+        <select id="deviceSelector" onchange="loadSelectedDevice()" style="width: 100%; padding: 8px; margin-bottom: 0;">
+          <option value="">-- Select a device --</option>
+        </select>
+      </div>
+      <button onclick="refreshDeviceList()" style="padding: 8px 12px;">↻ Refresh</button>
+    </div>
+
+    <div style="padding: 12px; background: #f6f8fa; border: 1px solid #d0d7de; border-radius: 4px; margin-bottom: 12px;">
+      <div style="display: flex; gap: 16px;">
+        <div style="flex: 1;">
           <label>Unit ID</label>
           <input id="unitId" value="nl43-1" />
+        </div>
+        <div style="flex: 2;">
           <label>Host</label>
           <input id="host" value="127.0.0.1" />
-    <label>Port</label>
-    <input id="port" type="number" value="80" />
+        </div>
+        <div style="flex: 1;">
+          <label>TCP Port</label>
+          <input id="port" type="number" value="2255" />
+        </div>
+      </div>
+    </div>
 
     <div style="margin: 12px 0;">
       <label style="display: inline-flex; align-items: center; margin-right: 16px;">
@@ -66,8 +92,10 @@
       <input id="ftpPassword" type="password" value="0000" />
     </div>
 
-    <button onclick="saveConfig()" style="margin-top: 12px;">Save Config</button>
+    <div style="margin-top: 12px;">
+      <button onclick="saveConfig()">Save Config</button>
       <button onclick="loadConfig()">Load Config</button>
+    </div>
   </fieldset>
 
   <fieldset>
@@ -148,6 +176,7 @@
 
     let ws = null;
     let streamUpdateCount = 0;
+    let availableDevices = [];
 
     function log(msg) {
       logEl.textContent += msg + "\n";
@@ -160,9 +189,97 @@
       ftpCredentials.style.display = ftpEnabled ? 'block' : 'none';
     }
 
-    // Add event listener for FTP checkbox
+    // Load device list from roster
+    async function refreshDeviceList() {
+      try {
+        const res = await fetch('/api/nl43/roster');
+        const data = await res.json();
+
+        if (!res.ok) {
+          log('Failed to load device list');
+          return;
+        }
+
+        availableDevices = data.devices || [];
+        const selector = document.getElementById('deviceSelector');
+
+        // Save current selection
+        const currentSelection = selector.value;
+
+        // Clear and rebuild options
+        selector.innerHTML = '<option value="">-- Select a device --</option>';
+
+        availableDevices.forEach(device => {
+          const option = document.createElement('option');
+          option.value = device.unit_id;
+
+          // Add status indicator
+          let statusIcon = '⚪';
+          if (device.status) {
+            if (device.status.is_reachable === false) {
+              statusIcon = '🔴';
+            } else if (device.status.last_success) {
+              const lastSeen = new Date(device.status.last_success);
+              const ageMinutes = Math.floor((Date.now() - lastSeen) / 60000);
+              statusIcon = ageMinutes < 5 ? '🟢' : '🟡';
+            }
+          }
+
+          option.textContent = `${statusIcon} ${device.unit_id} (${device.host})`;
+          selector.appendChild(option);
+        });
+
+        // Restore selection if it still exists
+        if (currentSelection && availableDevices.find(d => d.unit_id === currentSelection)) {
+          selector.value = currentSelection;
+        }
+
+        log(`Loaded ${availableDevices.length} device(s) from roster`);
+      } catch (err) {
+        log(`Error loading device list: ${err.message}`);
+      }
+    }
+
+    // Load selected device configuration
+    function loadSelectedDevice() {
+      const selector = document.getElementById('deviceSelector');
+      const unitId = selector.value;
+
+      if (!unitId) {
+        return;
+      }
+
+      const device = availableDevices.find(d => d.unit_id === unitId);
+      if (!device) {
+        log(`Device ${unitId} not found in list`);
+        return;
+      }
+
+      // Populate form fields
+      document.getElementById('unitId').value = device.unit_id;
+      document.getElementById('host').value = device.host;
+      document.getElementById('port').value = device.tcp_port || 2255;
+      document.getElementById('tcpEnabled').checked = device.tcp_enabled || false;
+      document.getElementById('ftpEnabled').checked = device.ftp_enabled || false;
+
+      if (device.ftp_username) {
+        document.getElementById('ftpUsername').value = device.ftp_username;
+      }
+      if (device.ftp_password) {
+        document.getElementById('ftpPassword').value = device.ftp_password;
+      }
+
+      toggleFtpCredentials();
+
+      log(`Loaded configuration for ${device.unit_id}`);
+    }
+
+    // Add event listeners
     document.addEventListener('DOMContentLoaded', function() {
       document.getElementById('ftpEnabled').addEventListener('change', toggleFtpCredentials);
+
+      // Load device list on page load
+      refreshDeviceList();
     });
 
     async function runDiagnostics() {
@@ -216,6 +333,134 @@
 
         html += `<p style="margin-top: 12px; font-size: 0.9em; color: #666;">Last run: ${new Date(data.timestamp).toLocaleString()}</p>`;
 
+        // Add database dump section if available
+        if (data.database_dump) {
+          html += `<div style="margin-top: 16px; border-top: 1px solid #d0d7de; padding-top: 12px;">`;
+          html += `<h4 style="margin: 0 0 12px 0;">📦 Database Dump</h4>`;
+
+          // Config section
+          if (data.database_dump.config) {
+            const cfg = data.database_dump.config;
+            html += `<div style="background: #f0f4f8; padding: 12px; border-radius: 4px; margin-bottom: 12px;">`;
+            html += `<strong>Configuration (nl43_config)</strong>`;
+            html += `<table style="width: 100%; margin-top: 8px; font-size: 0.9em;">`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Host</td><td>${cfg.host}:${cfg.tcp_port}</td></tr>`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">TCP Enabled</td><td>${cfg.tcp_enabled ? '✓' : '✗'}</td></tr>`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">FTP Enabled</td><td>${cfg.ftp_enabled ? '✓' : '✗'}${cfg.ftp_enabled ? ` (port ${cfg.ftp_port}, user: ${cfg.ftp_username || 'none'})` : ''}</td></tr>`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Background Polling</td><td>${cfg.poll_enabled ? `✓ every ${cfg.poll_interval_seconds}s` : '✗ disabled'}</td></tr>`;
+            html += `</table></div>`;
+          }
+
+          // Status cache section
+          if (data.database_dump.status_cache) {
+            const cache = data.database_dump.status_cache;
+            html += `<div style="background: #f0f8f4; padding: 12px; border-radius: 4px; margin-bottom: 12px;">`;
+            html += `<strong>Status Cache (nl43_status)</strong>`;
+            html += `<table style="width: 100%; margin-top: 8px; font-size: 0.9em;">`;
+
+            // Measurement state and timing
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Measurement State</td><td><strong>${cache.measurement_state || 'unknown'}</strong></td></tr>`;
+            if (cache.measurement_start_time) {
+              const startTime = new Date(cache.measurement_start_time);
+              const elapsed = Math.floor((Date.now() - startTime) / 1000);
+              const elapsedStr = elapsed > 3600 ? `${Math.floor(elapsed/3600)}h ${Math.floor((elapsed%3600)/60)}m` : elapsed > 60 ? `${Math.floor(elapsed/60)}m ${elapsed%60}s` : `${elapsed}s`;
+              html += `<tr><td style="padding: 2px 8px; color: #666;">Measurement Started</td><td>${startTime.toLocaleString()} (${elapsedStr} ago)</td></tr>`;
+            }
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Counter (d0)</td><td>${cache.counter || 'N/A'}</td></tr>`;
+
+            // Sound levels
+            html += `<tr><td colspan="2" style="padding: 8px 8px 2px 8px; font-weight: 600; border-top: 1px solid #d0d7de;">Sound Levels (dB)</td></tr>`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Lp (Instantaneous)</td><td>${cache.lp || 'N/A'}</td></tr>`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Leq (Equivalent)</td><td>${cache.leq || 'N/A'}</td></tr>`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Lmax / Lmin</td><td>${cache.lmax || 'N/A'} / ${cache.lmin || 'N/A'}</td></tr>`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Lpeak</td><td>${cache.lpeak || 'N/A'}</td></tr>`;
+
+            // Device status
+            html += `<tr><td colspan="2" style="padding: 8px 8px 2px 8px; font-weight: 600; border-top: 1px solid #d0d7de;">Device Status</td></tr>`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Battery</td><td>${cache.battery_level || 'N/A'}${cache.power_source ? ` (${cache.power_source})` : ''}</td></tr>`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">SD Card</td><td>${cache.sd_remaining_mb ? `${cache.sd_remaining_mb} MB` : 'N/A'}${cache.sd_free_ratio ? ` (${cache.sd_free_ratio} free)` : ''}</td></tr>`;
+
+            // Polling status
+            html += `<tr><td colspan="2" style="padding: 8px 8px 2px 8px; font-weight: 600; border-top: 1px solid #d0d7de;">Polling Status</td></tr>`;
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Reachable</td><td>${cache.is_reachable ? '🟢 Yes' : '🔴 No'}</td></tr>`;
+            if (cache.last_seen) {
+              html += `<tr><td style="padding: 2px 8px; color: #666;">Last Seen</td><td>${new Date(cache.last_seen).toLocaleString()}</td></tr>`;
+            }
+            if (cache.last_success) {
+              html += `<tr><td style="padding: 2px 8px; color: #666;">Last Success</td><td>${new Date(cache.last_success).toLocaleString()}</td></tr>`;
+            }
+            if (cache.last_poll_attempt) {
+              html += `<tr><td style="padding: 2px 8px; color: #666;">Last Poll Attempt</td><td>${new Date(cache.last_poll_attempt).toLocaleString()}</td></tr>`;
+            }
+            html += `<tr><td style="padding: 2px 8px; color: #666;">Consecutive Failures</td><td>${cache.consecutive_failures || 0}</td></tr>`;
+            if (cache.last_error) {
+              html += `<tr><td style="padding: 2px 8px; color: #666;">Last Error</td><td style="color: #d00; font-size: 0.85em;">${cache.last_error}</td></tr>`;
+            }
+
+            html += `</table></div>`;
+
+            // Raw payload (collapsible)
+            if (cache.raw_payload) {
+              html += `<details style="margin-top: 8px;"><summary style="cursor: pointer; color: #666; font-size: 0.9em;">📄 Raw Payload</summary>`;
+              html += `<pre style="background: #f6f8fa; padding: 8px; border-radius: 4px; font-size: 0.8em; overflow-x: auto; margin-top: 8px;">${cache.raw_payload}</pre></details>`;
+            }
+          } else {
+            html += `<p style="color: #888; font-style: italic;">No cached status available for this unit.</p>`;
+          }
+
+          html += `</div>`;
+        }
+
+        // Fetch and display device logs
+        try {
+          const logsRes = await fetch(`/api/nl43/${unitId}/logs?limit=50`);
+          if (logsRes.ok) {
+            const logsData = await logsRes.json();
+            if (logsData.logs && logsData.logs.length > 0) {
+              html += `<div style="margin-top: 16px; border-top: 1px solid #d0d7de; padding-top: 12px;">`;
+              html += `<h4 style="margin: 0 0 12px 0;">📋 Device Logs (${logsData.stats.total} total)</h4>`;
+
+              // Stats summary
+              if (logsData.stats.by_level) {
+                html += `<div style="margin-bottom: 8px; font-size: 0.85em; color: #666;">`;
+                const levels = logsData.stats.by_level;
+                const parts = [];
+                if (levels.ERROR) parts.push(`<span style="color: #d00;">${levels.ERROR} errors</span>`);
+                if (levels.WARNING) parts.push(`<span style="color: #fa0;">${levels.WARNING} warnings</span>`);
+                if (levels.INFO) parts.push(`${levels.INFO} info`);
+                html += parts.join(' · ');
+                html += `</div>`;
+              }
+
+              // Log entries (collapsible)
+              html += `<details open><summary style="cursor: pointer; font-size: 0.9em; margin-bottom: 8px;">Recent entries (${logsData.logs.length})</summary>`;
+              html += `<div style="max-height: 300px; overflow-y: auto; background: #f6f8fa; border: 1px solid #d0d7de; border-radius: 4px; padding: 8px; font-size: 0.8em; font-family: monospace;">`;
+
+              logsData.logs.forEach(entry => {
+                const levelColor = {
+                  'ERROR': '#d00',
+                  'WARNING': '#b86e00',
+                  'INFO': '#0969da',
+                  'DEBUG': '#888'
+                }[entry.level] || '#666';
+
+                const time = new Date(entry.timestamp).toLocaleString();
+                html += `<div style="margin-bottom: 4px; border-bottom: 1px solid #eee; padding-bottom: 4px;">`;
+                html += `<span style="color: #888;">${time}</span> `;
+                html += `<span style="color: ${levelColor}; font-weight: 600;">[${entry.level}]</span> `;
+                html += `<span style="color: #666;">[${entry.category}]</span> `;
+                html += `${entry.message}`;
+                html += `</div>`;
+              });
+
+              html += `</div></details>`;
+              html += `</div>`;
+            }
+          }
+        } catch (logErr) {
+          console.log('Could not fetch device logs:', logErr);
+        }
+
         resultsEl.innerHTML = html;
         log(`Diagnostics complete: ${data.overall_status}`);
 

templates/roster.html

@@ -0,0 +1,901 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+  <title>SLMM - Device Roster &amp; Connections</title>
+  <style>
+    * { box-sizing: border-box; }
+    body {
+      font-family: system-ui, -apple-system, sans-serif;
+      margin: 0;
+      padding: 24px;
+      background: #f6f8fa;
+    }
+    .container { max-width: 1400px; margin: 0 auto; }
+    .header {
+      display: flex;
+      justify-content: space-between;
+      align-items: center;
+      margin-bottom: 24px;
+      padding: 16px;
+      background: white;
+      border-radius: 6px;
+      box-shadow: 0 1px 3px rgba(0,0,0,0.1);
+    }
+    h1 { margin: 0; font-size: 24px; }
+    .nav { display: flex; gap: 12px; }
+    .btn {
+      padding: 8px 16px;
+      border: 1px solid #d0d7de;
+      background: white;
+      border-radius: 6px;
+      cursor: pointer;
+      text-decoration: none;
+      color: #24292f;
+      font-size: 14px;
+      transition: background 0.2s;
+    }
+    .btn:hover { background: #f6f8fa; }
+    .btn-primary {
+      background: #2da44e;
+      color: white;
+      border-color: #2da44e;
+    }
+    .btn-primary:hover { background: #2c974b; }
+    .btn-danger {
+      background: #cf222e;
+      color: white;
+      border-color: #cf222e;
+    }
+    .btn-danger:hover { background: #a40e26; }
+    .btn-small {
+      padding: 4px 8px;
+      font-size: 12px;
+      margin-right: 4px;
+    }
+    .table-container {
+      background: white;
+      border-radius: 6px;
+      box-shadow: 0 1px 3px rgba(0,0,0,0.1);
+      overflow-x: auto;
+    }
+    table {
+      width: 100%;
+      border-collapse: collapse;
+    }
+    th {
+      background: #f6f8fa;
+      padding: 12px;
+      text-align: left;
+      font-weight: 600;
+      border-bottom: 2px solid #d0d7de;
+      font-size: 13px;
+      white-space: nowrap;
+    }
+    td {
+      padding: 12px;
+      border-bottom: 1px solid #d0d7de;
+      font-size: 13px;
+    }
+    tr:hover { background: #f6f8fa; }
+    .status-badge {
+      display: inline-block;
+      padding: 2px 8px;
+      border-radius: 12px;
+      font-size: 11px;
+      font-weight: 600;
+      text-transform: uppercase;
+    }
+    .status-ok {
+      background: #dafbe1;
+      color: #1a7f37;
+    }
+    .status-unknown {
+      background: #eaeef2;
+      color: #57606a;
+    }
+    .status-error {
+      background: #ffebe9;
+      color: #cf222e;
+    }
+    .checkbox-cell {
+      text-align: center;
+      width: 80px;
+    }
+    .checkbox-cell input[type="checkbox"] {
+      cursor: pointer;
+      width: 16px;
+      height: 16px;
+    }
+    .actions-cell {
+      white-space: nowrap;
+      width: 200px;
+    }
+    .empty-state {
+      text-align: center;
+      padding: 48px;
+      color: #57606a;
+    }
+    .empty-state-icon {
+      font-size: 48px;
+      margin-bottom: 16px;
+    }
+    .modal {
+      display: none;
+      position: fixed;
+      top: 0;
+      left: 0;
+      width: 100%;
+      height: 100%;
+      background: rgba(0,0,0,0.5);
+      z-index: 1000;
+      align-items: center;
+      justify-content: center;
+    }
+    .modal.active { display: flex; }
+    .modal-content {
+      background: white;
+      padding: 24px;
+      border-radius: 6px;
+      max-width: 600px;
+      width: 90%;
+      max-height: 80vh;
+      overflow-y: auto;
+    }
+    .modal-header {
+      display: flex;
+      justify-content: space-between;
+      align-items: center;
+      margin-bottom: 16px;
+    }
+    .modal-header h2 {
+      margin: 0;
+      font-size: 20px;
+    }
+    .close-btn {
+      background: none;
+      border: none;
+      font-size: 24px;
+      cursor: pointer;
+      color: #57606a;
+      padding: 0;
+      width: 32px;
+      height: 32px;
+    }
+    .close-btn:hover { color: #24292f; }
+    .form-group {
+      margin-bottom: 16px;
+    }
+    .form-group label {
+      display: block;
+      margin-bottom: 6px;
+      font-weight: 600;
+      font-size: 14px;
+    }
+    .form-group input[type="text"],
+    .form-group input[type="number"],
+    .form-group input[type="password"] {
+      width: 100%;
+      padding: 8px 12px;
+      border: 1px solid #d0d7de;
+      border-radius: 6px;
+      font-size: 14px;
+    }
+    .form-group input[type="checkbox"] {
+      width: auto;
+      margin-right: 8px;
+    }
+    .checkbox-label {
+      display: flex;
+      align-items: center;
+      font-weight: normal;
+      cursor: pointer;
+    }
+    .form-actions {
+      display: flex;
+      justify-content: flex-end;
+      gap: 8px;
+      margin-top: 24px;
+    }
+    .toast {
+      position: fixed;
+      top: 24px;
+      right: 24px;
+      padding: 12px 16px;
+      background: #24292f;
+      color: white;
+      border-radius: 6px;
+      box-shadow: 0 4px 12px rgba(0,0,0,0.15);
+      z-index: 2000;
+      display: none;
+      min-width: 300px;
+    }
+    .toast.active {
+      display: block;
+      animation: slideIn 0.3s ease-out;
+    }
+    @keyframes slideIn {
+      from {
+        transform: translateX(400px);
+        opacity: 0;
+      }
+      to {
+        transform: translateX(0);
+        opacity: 1;
+      }
+    }
+    .toast-success { background: #2da44e; }
+    .toast-error { background: #cf222e; }
+
+    /* Tabs */
+    .tabs {
+      display: flex;
+      gap: 0;
+      margin-bottom: 0;
+      border-bottom: 2px solid #d0d7de;
+    }
+    .tab-btn {
+      padding: 10px 20px;
+      border: none;
+      background: none;
+      cursor: pointer;
+      font-size: 14px;
+      font-weight: 600;
+      color: #57606a;
+      border-bottom: 2px solid transparent;
+      margin-bottom: -2px;
+      transition: color 0.2s, border-color 0.2s;
+    }
+    .tab-btn:hover { color: #24292f; }
+    .tab-btn.active {
+      color: #24292f;
+      border-bottom-color: #fd8c73;
+    }
+    .tab-panel { display: none; }
+    .tab-panel.active { display: block; }
+
+    /* Connection pool panel */
+    .pool-config {
+      display: grid;
+      grid-template-columns: repeat(auto-fill, minmax(180px, 1fr));
+      gap: 12px;
+      margin-bottom: 20px;
+    }
+    .pool-config-card {
+      background: #f6f8fa;
+      border: 1px solid #d0d7de;
+      border-radius: 6px;
+      padding: 12px;
+    }
+    .pool-config-card .label {
+      font-size: 11px;
+      color: #57606a;
+      text-transform: uppercase;
+      font-weight: 600;
+      margin-bottom: 4px;
+    }
+    .pool-config-card .value {
+      font-size: 18px;
+      font-weight: 600;
+      color: #24292f;
+    }
+    .conn-card {
+      background: white;
+      border: 1px solid #d0d7de;
+      border-radius: 6px;
+      padding: 16px;
+      margin-bottom: 12px;
+    }
+    .conn-card-header {
+      display: flex;
+      justify-content: space-between;
+      align-items: center;
+      margin-bottom: 12px;
+    }
+    .conn-card-header strong { font-size: 15px; }
+    .conn-card-grid {
+      display: grid;
+      grid-template-columns: repeat(auto-fill, minmax(140px, 1fr));
+      gap: 8px;
+    }
+    .conn-stat .label {
+      font-size: 11px;
+      color: #57606a;
+      text-transform: uppercase;
+      font-weight: 600;
+    }
+    .conn-stat .value {
+      font-size: 14px;
+      font-weight: 600;
+      color: #24292f;
+    }
+    .conn-empty {
+      text-align: center;
+      padding: 32px;
+      color: #57606a;
+    }
+    .pool-actions {
+      display: flex;
+      gap: 8px;
+      margin-bottom: 16px;
+    }
+  </style>
+</head>
+<body>
+  <div class="container">
+    <div class="header">
+      <h1>SLMM - Roster &amp; Connections</h1>
+      <div class="nav">
+        <a href="/" class="btn">&larr; Back to Control Panel</a>
+        <button class="btn btn-primary" onclick="openAddModal()">+ Add Device</button>
+      </div>
+    </div>
+
+    <div class="tabs">
+      <button class="tab-btn active" onclick="switchTab('roster')">Device Roster</button>
+      <button class="tab-btn" onclick="switchTab('connections')">Connections</button>
+    </div>
+
+    <!-- Roster Tab -->
+    <div id="tab-roster" class="tab-panel active">
+      <div class="table-container" style="border-top-left-radius: 0; border-top-right-radius: 0;">
+        <table id="rosterTable">
+          <thead>
+            <tr>
+              <th>Unit ID</th>
+              <th>Host / IP</th>
+              <th>TCP Port</th>
+              <th>FTP Port</th>
+              <th class="checkbox-cell">TCP</th>
+              <th class="checkbox-cell">FTP</th>
+              <th class="checkbox-cell">Polling</th>
+              <th>Status</th>
+              <th class="actions-cell">Actions</th>
+            </tr>
+          </thead>
+          <tbody id="rosterBody">
+            <tr>
+              <td colspan="9" style="text-align: center; padding: 24px;">
+                Loading...
+              </td>
+            </tr>
+          </tbody>
+        </table>
+      </div>
+    </div>
+
+    <!-- Connections Tab -->
+    <div id="tab-connections" class="tab-panel">
+      <div class="table-container" style="padding: 20px; border-top-left-radius: 0; border-top-right-radius: 0;">
+        <div class="pool-actions">
+          <button class="btn" onclick="loadConnections()">Refresh</button>
+          <button class="btn btn-danger" onclick="flushConnections()">Flush All Connections</button>
+        </div>
+
+        <h3 style="margin: 0 0 12px 0; font-size: 16px;">Pool Configuration</h3>
+        <div id="poolConfig" class="pool-config">
+          <div class="pool-config-card">
+            <div class="label">Status</div>
+            <div class="value" id="poolEnabled">--</div>
+          </div>
+        </div>
+
+        <h3 style="margin: 20px 0 12px 0; font-size: 16px;">Active Connections</h3>
+        <div id="connectionsList">
+          <div class="conn-empty">Loading...</div>
+        </div>
+      </div>
+    </div>
+  </div>
+
+  <!-- Add/Edit Modal -->
+  <div id="deviceModal" class="modal">
+    <div class="modal-content">
+      <div class="modal-header">
+        <h2 id="modalTitle">Add Device</h2>
+        <button class="close-btn" onclick="closeModal()">&times;</button>
+      </div>
+
+      <form id="deviceForm" onsubmit="saveDevice(event)">
+        <div class="form-group">
+          <label for="unitId">Unit ID *</label>
+          <input type="text" id="unitId" required placeholder="e.g., nl43-1, slm-site-a" />
+        </div>
+
+        <div class="form-group">
+          <label for="host">Host / IP Address *</label>
+          <input type="text" id="host" required placeholder="e.g., 192.168.1.100" />
+        </div>
+
+        <div class="form-group">
+          <label for="tcpPort">TCP Port *</label>
+          <input type="number" id="tcpPort" required value="2255" min="1" max="65535" />
+        </div>
+
+        <div class="form-group">
+          <label for="ftpPort">FTP Port</label>
+          <input type="number" id="ftpPort" value="21" min="1" max="65535" />
+        </div>
+
+        <div class="form-group">
+          <label class="checkbox-label">
+            <input type="checkbox" id="tcpEnabled" checked />
+            TCP Enabled (required for remote control)
+          </label>
+        </div>
+
+        <div class="form-group">
+          <label class="checkbox-label">
+            <input type="checkbox" id="ftpEnabled" onchange="toggleFtpCredentials()" />
+            FTP Enabled (for file downloads)
+          </label>
+        </div>
+
+        <div id="ftpCredentialsSection" style="display: none; padding: 12px; background: #f6f8fa; border-radius: 6px; margin-bottom: 16px;">
+          <div class="form-group">
+            <label for="ftpUsername">FTP Username</label>
+            <input type="text" id="ftpUsername" placeholder="Default: USER" />
+          </div>
+
+          <div class="form-group">
+            <label for="ftpPassword">FTP Password</label>
+            <input type="password" id="ftpPassword" placeholder="Default: 0000" />
+          </div>
+        </div>
+
+        <div class="form-group">
+          <label class="checkbox-label">
+            <input type="checkbox" id="pollEnabled" checked />
+            Enable background polling (status updates)
+          </label>
+        </div>
+
+        <div class="form-group">
+          <label for="pollInterval">Polling Interval (seconds)</label>
+          <input type="number" id="pollInterval" value="60" min="10" max="3600" />
+        </div>
+
+        <div class="form-actions">
+          <button type="button" class="btn" onclick="closeModal()">Cancel</button>
+          <button type="submit" class="btn btn-primary">Save Device</button>
+        </div>
+      </form>
+    </div>
+  </div>
+
+  <!-- Toast Notification -->
+  <div id="toast" class="toast"></div>
+
+  <script>
+    let devices = [];
+    let editingDeviceId = null;
+
+    // Load roster on page load
+    document.addEventListener('DOMContentLoaded', () => {
+      loadRoster();
+    });
+
+    async function loadRoster() {
+      try {
+        const res = await fetch('/api/nl43/roster');
+        const data = await res.json();
+
+        if (!res.ok) {
+          showToast('Failed to load roster', 'error');
+          return;
+        }
+
+        devices = data.devices || [];
+        renderRoster();
+      } catch (err) {
+        showToast('Error loading roster: ' + err.message, 'error');
+        console.error('Load roster error:', err);
+      }
+    }
+
+    function renderRoster() {
+      const tbody = document.getElementById('rosterBody');
+
+      if (devices.length === 0) {
+        tbody.innerHTML = `
+          <tr>
+            <td colspan="9" class="empty-state">
+              <div class="empty-state-icon">📭</div>
+              <div><strong>No devices configured</strong></div>
+              <div style="margin-top: 8px; font-size: 14px;">Click "Add Device" to configure your first sound level meter</div>
+            </td>
+          </tr>
+        `;
+        return;
+      }
+
+      tbody.innerHTML = devices.map(device => `
+        <tr>
+          <td><strong>${escapeHtml(device.unit_id)}</strong></td>
+          <td>${escapeHtml(device.host)}</td>
+          <td>${device.tcp_port}</td>
+          <td>${device.ftp_port || 21}</td>
+          <td class="checkbox-cell">
+            <input type="checkbox" ${device.tcp_enabled ? 'checked' : ''} disabled />
+          </td>
+          <td class="checkbox-cell">
+            <input type="checkbox" ${device.ftp_enabled ? 'checked' : ''} disabled />
+          </td>
+          <td class="checkbox-cell">
+            <input type="checkbox" ${device.poll_enabled ? 'checked' : ''} disabled />
+          </td>
+          <td>
+            ${getStatusBadge(device)}
+          </td>
+          <td class="actions-cell">
+            <button class="btn btn-small" onclick="testDevice('${escapeHtml(device.unit_id)}')">Test</button>
+            <button class="btn btn-small" onclick="openEditModal('${escapeHtml(device.unit_id)}')">Edit</button>
+            <button class="btn btn-small btn-danger" onclick="deleteDevice('${escapeHtml(device.unit_id)}')">Delete</button>
+          </td>
+        </tr>
+      `).join('');
+    }
+
+    function getStatusBadge(device) {
+      if (!device.status) {
+        return '<span class="status-badge status-unknown">Unknown</span>';
+      }
+
+      if (device.status.is_reachable === false) {
+        return '<span class="status-badge status-error">Offline</span>';
+      }
+
+      if (device.status.last_success) {
+        const lastSeen = new Date(device.status.last_success);
+        const ago = Math.floor((Date.now() - lastSeen) / 1000);
+        if (ago < 300) { // Less than 5 minutes
+          return '<span class="status-badge status-ok">Online</span>';
+        } else {
+          return `<span class="status-badge status-unknown">Stale (${Math.floor(ago / 60)}m ago)</span>`;
+        }
+      }
+
+      return '<span class="status-badge status-unknown">Unknown</span>';
+    }
+
+    function escapeHtml(text) {
+      const map = {
+        '&': '&amp;',
+        '<': '&lt;',
+        '>': '&gt;',
+        '"': '&quot;',
+        "'": '&#039;'
+      };
+      return String(text).replace(/[&<>"']/g, m => map[m]);
+    }
+
+    function openAddModal() {
+      editingDeviceId = null;
+      document.getElementById('modalTitle').textContent = 'Add Device';
+      document.getElementById('deviceForm').reset();
+      document.getElementById('unitId').disabled = false;
+      document.getElementById('tcpEnabled').checked = true;
+      document.getElementById('ftpEnabled').checked = false;
+      document.getElementById('pollEnabled').checked = true;
+      document.getElementById('tcpPort').value = 2255;
+      document.getElementById('ftpPort').value = 21;
+      document.getElementById('pollInterval').value = 60;
+      toggleFtpCredentials();
+      document.getElementById('deviceModal').classList.add('active');
+    }
+
+    function openEditModal(unitId) {
+      const device = devices.find(d => d.unit_id === unitId);
+      if (!device) {
+        showToast('Device not found', 'error');
+        return;
+      }
+
+      editingDeviceId = unitId;
+      document.getElementById('modalTitle').textContent = 'Edit Device';
+      document.getElementById('unitId').value = device.unit_id;
+      document.getElementById('unitId').disabled = true;
+      document.getElementById('host').value = device.host;
+      document.getElementById('tcpPort').value = device.tcp_port;
+      document.getElementById('ftpPort').value = device.ftp_port || 21;
+      document.getElementById('tcpEnabled').checked = device.tcp_enabled;
+      document.getElementById('ftpEnabled').checked = device.ftp_enabled;
+      document.getElementById('ftpUsername').value = device.ftp_username || '';
+      document.getElementById('ftpPassword').value = device.ftp_password || '';
+      document.getElementById('pollEnabled').checked = device.poll_enabled;
+      document.getElementById('pollInterval').value = device.poll_interval_seconds || 60;
+      toggleFtpCredentials();
+      document.getElementById('deviceModal').classList.add('active');
+    }
+
+    function closeModal() {
+      document.getElementById('deviceModal').classList.remove('active');
+      editingDeviceId = null;
+    }
+
+    function toggleFtpCredentials() {
+      const ftpEnabled = document.getElementById('ftpEnabled').checked;
+      document.getElementById('ftpCredentialsSection').style.display = ftpEnabled ? 'block' : 'none';
+    }
+
+    async function saveDevice(event) {
+      event.preventDefault();
+
+      const unitId = document.getElementById('unitId').value.trim();
+      const payload = {
+        host: document.getElementById('host').value.trim(),
+        tcp_port: parseInt(document.getElementById('tcpPort').value),
+        ftp_port: parseInt(document.getElementById('ftpPort').value),
+        tcp_enabled: document.getElementById('tcpEnabled').checked,
+        ftp_enabled: document.getElementById('ftpEnabled').checked,
+        poll_enabled: document.getElementById('pollEnabled').checked,
+        poll_interval_seconds: parseInt(document.getElementById('pollInterval').value)
+      };
+
+      if (payload.ftp_enabled) {
+        const username = document.getElementById('ftpUsername').value.trim();
+        const password = document.getElementById('ftpPassword').value.trim();
+        if (username) payload.ftp_username = username;
+        if (password) payload.ftp_password = password;
+      }
+
+      try {
+        const url = editingDeviceId
+          ? `/api/nl43/${editingDeviceId}/config`
+          : `/api/nl43/roster`;
+
+        const method = editingDeviceId ? 'PUT' : 'POST';
+
+        const body = editingDeviceId
+          ? payload
+          : { unit_id: unitId, ...payload };
+
+        const res = await fetch(url, {
+          method,
+          headers: { 'Content-Type': 'application/json' },
+          body: JSON.stringify(body)
+        });
+
+        const data = await res.json();
+
+        if (!res.ok) {
+          showToast(data.detail || 'Failed to save device', 'error');
+          return;
+        }
+
+        showToast(editingDeviceId ? 'Device updated successfully' : 'Device added successfully', 'success');
+        closeModal();
+        await loadRoster();
+      } catch (err) {
+        showToast('Error saving device: ' + err.message, 'error');
+        console.error('Save device error:', err);
+      }
+    }
+
+    async function deleteDevice(unitId) {
+      if (!confirm(`Are you sure you want to delete "${unitId}"?\n\nThis will remove the device configuration but will not affect the physical device.`)) {
+        return;
+      }
+
+      try {
+        const res = await fetch(`/api/nl43/${unitId}/config`, {
+          method: 'DELETE'
+        });
+
+        const data = await res.json();
+
+        if (!res.ok) {
+          showToast(data.detail || 'Failed to delete device', 'error');
+          return;
+        }
+
+        showToast('Device deleted successfully', 'success');
+        await loadRoster();
+      } catch (err) {
+        showToast('Error deleting device: ' + err.message, 'error');
+        console.error('Delete device error:', err);
+      }
+    }
+
+    async function testDevice(unitId) {
+      showToast('Testing device connection...', 'success');
+
+      try {
+        const res = await fetch(`/api/nl43/${unitId}/diagnostics`);
+        const data = await res.json();
+
+        if (!res.ok) {
+          showToast('Device test failed', 'error');
+          return;
+        }
+
+        const statusText = {
+          'pass': 'All systems operational ✓',
+          'fail': 'Connection failed ✗',
+          'degraded': 'Partial connectivity ⚠'
+        };
+
+        showToast(statusText[data.overall_status] || 'Test complete',
+                  data.overall_status === 'pass' ? 'success' : 'error');
+
+        // Reload to update status
+        await loadRoster();
+      } catch (err) {
+        showToast('Error testing device: ' + err.message, 'error');
+        console.error('Test device error:', err);
+      }
+    }
+
+    function showToast(message, type = 'success') {
+      const toast = document.getElementById('toast');
+      toast.textContent = message;
+      toast.className = `toast toast-${type} active`;
+
+      setTimeout(() => {
+        toast.classList.remove('active');
+      }, 3000);
+    }
+
+    // Close modal when clicking outside
+    document.getElementById('deviceModal').addEventListener('click', (e) => {
+      if (e.target.id === 'deviceModal') {
+        closeModal();
+      }
+    });
+
+    // ========== Tab Switching ==========
+
+    function switchTab(tabName) {
+      document.querySelectorAll('.tab-btn').forEach(btn => btn.classList.remove('active'));
+      document.querySelectorAll('.tab-panel').forEach(panel => panel.classList.remove('active'));
+
+      document.querySelector(`.tab-btn[onclick="switchTab('${tabName}')"]`).classList.add('active');
+      document.getElementById(`tab-${tabName}`).classList.add('active');
+
+      if (tabName === 'connections') {
+        loadConnections();
+      }
+    }
+
+    // ========== Connection Pool ==========
+
+    let connectionsRefreshTimer = null;
+
+    async function loadConnections() {
+      try {
+        const res = await fetch('/api/nl43/_connections/status');
+        const data = await res.json();
+
+        if (!res.ok) {
+          showToast('Failed to load connection pool status', 'error');
+          return;
+        }
+
+        const pool = data.pool;
+        renderPoolConfig(pool);
+        renderConnections(pool.connections);
+
+        // Auto-refresh while tab is active
+        clearTimeout(connectionsRefreshTimer);
+        if (document.getElementById('tab-connections').classList.contains('active')) {
+          connectionsRefreshTimer = setTimeout(loadConnections, 5000);
+        }
+      } catch (err) {
+        showToast('Error loading connections: ' + err.message, 'error');
+        console.error('Load connections error:', err);
+      }
+    }
+
+    function renderPoolConfig(pool) {
+      document.getElementById('poolConfig').innerHTML = `
+        <div class="pool-config-card">
+          <div class="label">Persistent</div>
+          <div class="value" style="color: ${pool.enabled ? '#1a7f37' : '#cf222e'}">${pool.enabled ? 'Enabled' : 'Disabled'}</div>
+        </div>
+        <div class="pool-config-card">
+          <div class="label">Active</div>
+          <div class="value">${pool.active_connections}</div>
+        </div>
+        <div class="pool-config-card">
+          <div class="label">Idle TTL</div>
+          <div class="value">${pool.idle_ttl}s</div>
+        </div>
+        <div class="pool-config-card">
+          <div class="label">Max Age</div>
+          <div class="value">${pool.max_age}s</div>
+        </div>
+        <div class="pool-config-card">
+          <div class="label">KA Idle</div>
+          <div class="value">${pool.keepalive_idle}s</div>
+        </div>
+        <div class="pool-config-card">
+          <div class="label">KA Interval</div>
+          <div class="value">${pool.keepalive_interval}s</div>
+        </div>
+        <div class="pool-config-card">
+          <div class="label">KA Probes</div>
+          <div class="value">${pool.keepalive_count}</div>
+        </div>
+      `;
+    }
+
+    function renderConnections(connections) {
+      const container = document.getElementById('connectionsList');
+      const keys = Object.keys(connections);
+
+      if (keys.length === 0) {
+        container.innerHTML = `
+          <div class="conn-empty">
+            <div style="font-size: 32px; margin-bottom: 8px;">~</div>
+            <div><strong>No active connections</strong></div>
+            <div style="margin-top: 4px; font-size: 13px;">
+              Connections appear here when devices are actively being polled and the connection is cached between commands.
+            </div>
+          </div>
+        `;
+        return;
+      }
+
+      container.innerHTML = keys.map(key => {
+        const conn = connections[key];
+        const aliveColor = conn.alive ? '#1a7f37' : '#cf222e';
+        const aliveText = conn.alive ? 'Alive' : 'Stale';
+        return `
+          <div class="conn-card">
+            <div class="conn-card-header">
+              <strong>${escapeHtml(key)}</strong>
+              <span class="status-badge ${conn.alive ? 'status-ok' : 'status-error'}">${aliveText}</span>
+            </div>
+            <div class="conn-card-grid">
+              <div class="conn-stat">
+                <div class="label">Host</div>
+                <div class="value">${escapeHtml(conn.host)}</div>
+              </div>
+              <div class="conn-stat">
+                <div class="label">Port</div>
+                <div class="value">${conn.port}</div>
+              </div>
+              <div class="conn-stat">
+                <div class="label">Age</div>
+                <div class="value">${formatSeconds(conn.age_seconds)}</div>
+              </div>
+              <div class="conn-stat">
+                <div class="label">Idle</div>
+                <div class="value">${formatSeconds(conn.idle_seconds)}</div>
+              </div>
+            </div>
+          </div>
+        `;
+      }).join('');
+    }
+
+    function formatSeconds(s) {
+      if (s < 60) return Math.round(s) + 's';
+      if (s < 3600) return Math.floor(s / 60) + 'm ' + Math.round(s % 60) + 's';
+      return Math.floor(s / 3600) + 'h ' + Math.floor((s % 3600) / 60) + 'm';
+    }
+
+    async function flushConnections() {
+      if (!confirm('Close all cached TCP connections?\n\nDevices will reconnect on the next poll cycle.')) {
+        return;
+      }
+
+      try {
+        const res = await fetch('/api/nl43/_connections/flush', { method: 'POST' });
+        const data = await res.json();
+
+        if (!res.ok) {
+          showToast(data.detail || 'Failed to flush connections', 'error');
+          return;
+        }
+
+        showToast('All connections flushed', 'success');
+        await loadConnections();
+      } catch (err) {
+        showToast('Error flushing connections: ' + err.message, 'error');
+      }
+    }
+  </script>
+</body>
+</html>

test_sleep_mode_auto_disable.py

@@ -1,128 +0,0 @@
-#!/usr/bin/env python3
-"""
-Test script to verify that sleep mode is automatically disabled when:
-1. Device configuration is created/updated with TCP enabled
-2. Measurements are started
-
-This script tests the API endpoints, not the actual device communication.
-"""
-
-import requests
-import json
-
-BASE_URL = "http://localhost:8100/api/nl43"
-UNIT_ID = "test-nl43-001"
-
-def test_config_update():
-    """Test that config update works (actual sleep mode disable requires real device)"""
-    print("\n=== Testing Config Update ===")
-
-    # Create/update a device config
-    config_data = {
-        "host": "192.168.1.100",
-        "tcp_port": 2255,
-        "tcp_enabled": True,
-        "ftp_enabled": False,
-        "ftp_username": "admin",
-        "ftp_password": "password"
-    }
-
-    print(f"Updating config for {UNIT_ID}...")
-    response = requests.put(f"{BASE_URL}/{UNIT_ID}/config", json=config_data)
-
-    if response.status_code == 200:
-        print("✓ Config updated successfully")
-        print(f"Response: {json.dumps(response.json(), indent=2)}")
-        print("\nNote: Sleep mode disable was attempted (will succeed if device is reachable)")
-        return True
-    else:
-        print(f"✗ Config update failed: {response.status_code}")
-        print(f"Error: {response.text}")
-        return False
-
-def test_get_config():
-    """Test retrieving the config"""
-    print("\n=== Testing Get Config ===")
-
-    response = requests.get(f"{BASE_URL}/{UNIT_ID}/config")
-
-    if response.status_code == 200:
-        print("✓ Config retrieved successfully")
-        print(f"Response: {json.dumps(response.json(), indent=2)}")
-        return True
-    elif response.status_code == 404:
-        print("✗ Config not found (create one first)")
-        return False
-    else:
-        print(f"✗ Request failed: {response.status_code}")
-        print(f"Error: {response.text}")
-        return False
-
-def test_start_measurement():
-    """Test that start measurement attempts to disable sleep mode"""
-    print("\n=== Testing Start Measurement ===")
-
-    print(f"Attempting to start measurement on {UNIT_ID}...")
-    response = requests.post(f"{BASE_URL}/{UNIT_ID}/start")
-
-    if response.status_code == 200:
-        print("✓ Start command accepted")
-        print(f"Response: {json.dumps(response.json(), indent=2)}")
-        print("\nNote: Sleep mode was disabled before starting measurement")
-        return True
-    elif response.status_code == 404:
-        print("✗ Device config not found (create config first)")
-        return False
-    elif response.status_code == 502:
-        print("✗ Device not reachable (expected if no physical device)")
-        print(f"Response: {response.text}")
-        print("\nNote: This is expected behavior when testing without a physical device")
-        return True  # This is actually success - the endpoint tried to communicate
-    else:
-        print(f"✗ Request failed: {response.status_code}")
-        print(f"Error: {response.text}")
-        return False
-
-def main():
-    print("=" * 60)
-    print("Sleep Mode Auto-Disable Test")
-    print("=" * 60)
-    print("\nThis test verifies that sleep mode is automatically disabled")
-    print("when device configs are updated or measurements are started.")
-    print("\nNote: Without a physical device, some operations will fail at")
-    print("the device communication level, but the API logic will execute.")
-
-    # Run tests
-    results = []
-
-    # Test 1: Update config (should attempt to disable sleep mode)
-    results.append(("Config Update", test_config_update()))
-
-    # Test 2: Get config
-    results.append(("Get Config", test_get_config()))
-
-    # Test 3: Start measurement (should attempt to disable sleep mode)
-    results.append(("Start Measurement", test_start_measurement()))
-
-    # Summary
-    print("\n" + "=" * 60)
-    print("Test Summary")
-    print("=" * 60)
-
-    for test_name, result in results:
-        status = "✓ PASS" if result else "✗ FAIL"
-        print(f"{status}: {test_name}")
-
-    print("\n" + "=" * 60)
-    print("Implementation Details:")
-    print("=" * 60)
-    print("1. Config endpoint is now async and calls ensure_sleep_mode_disabled()")
-    print("   when TCP is enabled")
-    print("2. Start measurement endpoint calls ensure_sleep_mode_disabled()")
-    print("   before starting the measurement")
-    print("3. Sleep mode check is non-blocking - config/start will succeed")
-    print("   even if the device is unreachable")
-    print("=" * 60)
-
-if __name__ == "__main__":
-    main()