terra-view/CHANGELOG.md

# Changelog

All notable changes to Seismo Fleet Manager will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.4.2] - 2026-01-05

### Added
- **SLM Configuration Interface**: Sound Level Meters can now be configured directly from the SLM dashboard
  - Configuration modal with comprehensive SLM parameter editing
  - TCP port configuration for SLM control connections (default: 2255)
  - FTP port configuration for SLM data retrieval (default: 21)
  - Modem assignment for network access or direct IP connection support
  - Test Modem button with ping-based connectivity verification (shows IP and response time)
  - Test SLM Connection button for end-to-end connectivity validation
  - Dynamic form fields that hide/show based on modem selection
- **SLM Dashboard Endpoints**: New API routes for SLM management
  - `GET /api/slm-dashboard/config/{unit_id}` - Load SLM configuration form
  - `POST /api/slm-dashboard/config/{unit_id}` - Save SLM configuration
  - `GET /api/slm-dashboard/test-modem/{modem_id}` - Ping modem for connectivity test
- **Database Schema Updates**: Added `slm_ftp_port` column to roster table
  - Migration script: `scripts/add_slm_ftp_port.py`
  - Supports both TCP (control) and FTP (data) port configuration per SLM unit
- **Docker Environment Enhancements**:
  - Added `iputils-ping` and `curl` packages to Docker image for network diagnostics
  - Health check endpoint support via curl

### Fixed
- **Form Validation**: Fixed 400 Bad Request error when adding modem units
  - Form fields for device-specific parameters now properly disabled when hidden
  - Empty string values for integer fields no longer cause validation failures
  - JavaScript now disables hidden form sections to prevent unwanted data submission
- **Unit Status Accuracy**: Fixed issue where unit status was loading from a saved cache instead of actual last-heard time
  - Unit status now accurately reflects real-time connectivity
  - Status determination based on actual `slm_last_check` timestamp

### Changed
- **Roster Form Behavior**: Device-specific form fields are now disabled (not just hidden) when not applicable
  - Prevents SLM fields from submitting when adding modems
  - Prevents modem fields from submitting when adding SLMs
  - Cleaner form submissions with only relevant data
- **Port Field Handling**: Backend now accepts port fields as strings and converts to integers
  - Handles empty string values gracefully
  - Proper type conversion with None fallback for empty values

### Technical Details
- Added `setFieldsDisabled()` helper function for managing form field state
- Updated `toggleDeviceFields()` and `toggleEditDeviceFields()` to disable/enable fields
- Backend type conversion: `slm_tcp_port` and `slm_ftp_port` accept strings, convert to int with empty string handling
- Modem ping uses subprocess with 1 packet, 2-second timeout, returns response time in milliseconds
- Configuration form uses 3-column grid layout for TCP Port, FTP Port, and Direct IP fields

## [0.4.1] - 2026-01-05
### Added
- **SLM Integration**: Sound Level Meters are now manageable in SFM

### Fixed
- Fixed an issue where unit status was loading from a saved cache and not based on when it was actually heard from last. Unit status is now accurate.


## [0.4.0] - 2025-12-16

### Added
- **Database Management System**: Comprehensive backup and restore capabilities
  - **Manual Snapshots**: Create on-demand backups of the entire database with optional descriptions
  - **Restore from Snapshot**: Restore database from any snapshot with automatic safety backup
  - **Upload/Download Snapshots**: Transfer database snapshots to/from the server
  - **Database Tab**: New dedicated tab in Settings for all database management operations
  - **Database Statistics**: View database size, row counts by table, and last modified time
  - **Snapshot Metadata**: Each snapshot includes creation time, description, size, and type (manual/automatic)
  - **Safety Backups**: Automatic backup created before any restore operation
- **Remote Database Cloning**: Dev tools for cloning production database to remote development servers
  - **Clone Script**: `scripts/clone_db_to_dev.py` for copying database over WAN
  - **Network Upload**: Upload snapshots via HTTP to remote servers
  - **Auto-restore**: Automatically restore uploaded database on target server
  - **Authentication Support**: Optional token-based authentication for secure transfers
- **Automatic Backup Scheduler**: Background service for automated database backups
  - **Configurable Intervals**: Set backup frequency (default: 24 hours)
  - **Retention Management**: Automatically delete old backups (configurable keep count)
  - **Manual Trigger**: Force immediate backup via API
  - **Status Monitoring**: Check scheduler status and next scheduled run time
  - **Background Thread**: Non-blocking operation using Python threading
- **Settings Reorganization**: Improved tab structure for better organization
  - Renamed "Data Management" tab to "Roster Management"
  - Moved CSV Replace Mode from Advanced tab to Roster Management tab
  - Created dedicated Database tab for all backup/restore operations
- **Comprehensive Documentation**: New `docs/DATABASE_MANAGEMENT.md` guide covering:
  - Manual snapshot creation and restoration workflows
  - Download/upload procedures for off-site backups
  - Remote database cloning setup and usage
  - Automatic backup configuration and integration
  - API reference for all database endpoints
  - Best practices and troubleshooting guide

### Changed
- **Settings Tab Organization**: Restructured for better logical grouping
  - **General**: Display preferences (timezone, theme, auto-refresh)
  - **Roster Management**: CSV operations and roster table (now includes Replace Mode)
  - **Database**: All backup/restore operations (NEW)
  - **Advanced**: Power user settings (calibration, thresholds)
  - **Danger Zone**: Destructive operations
- CSV Replace Mode warnings enhanced and moved to Roster Management context

### Technical Details
- **SQLite Backup API**: Uses native SQLite backup API for concurrent-safe snapshots
- **Metadata Tracking**: JSON sidecar files store snapshot metadata alongside database files
- **Atomic Operations**: Database restoration is atomic with automatic rollback on failure
- **File Structure**: Snapshots stored in `./data/backups/` with timestamped filenames
- **API Endpoints**: 7 new endpoints for database management operations
- **Backup Service**: `backend/services/database_backup.py` - Core backup/restore logic
- **Scheduler Service**: `backend/services/backup_scheduler.py` - Automatic backup automation
- **Clone Utility**: `scripts/clone_db_to_dev.py` - Remote database synchronization tool

### Security Considerations
- Snapshots contain full database data and should be secured appropriately
- Remote cloning supports optional authentication tokens
- Restore operations require safety backup creation by default
- All destructive operations remain in Danger Zone with warnings

### Migration Notes
No database migration required for v0.4.0. All new features use existing database structure and add new backup management capabilities without modifying the core schema.

## [0.3.3] - 2025-12-12

### Changed
- **Mobile Navigation**: Moved hamburger menu button from floating top-right to bottom navigation bar
  - Bottom nav now shows: Menu (hamburger), Dashboard, Roster, Settings
  - Removed "Add Unit" from bottom nav (still accessible via sidebar menu)
  - Hamburger no longer floats over content on mobile
- **Status Dot Visibility**: Increased status dot size from 12px to 16px (w-3/h-3 → w-4/h-4) in dashboard fleet overview for better at-a-glance visibility
  - Affects both Active and Benched tabs in dashboard
  - Makes status colors (green/yellow/red) easier to spot during quick scroll

### Fixed
- **Location Navigation**: Moved tap-to-navigate functionality from roster card view to unit detail modal only
  - Roster cards now show simple location text with pin emoji
  - Navigation links (opening Maps app) only appear in the modal when tapping a unit
  - Reduces visual clutter and accidental navigation triggers

### Technical Details
- Bottom navigation remains at 4 buttons, first button now triggers sidebar menu
- Removed standalone hamburger button element and associated CSS
- Modal already had navigation links, no changes needed there

## [0.3.2] - 2025-12-12

### Added
- **Progressive Web App (PWA) Mobile Optimization**: Complete mobile-first redesign for field deployment usage
  - **Responsive Navigation**: Hamburger menu with slide-in sidebar for mobile, always-visible sidebar for desktop
  - **Bottom Navigation Bar**: Quick access to Dashboard, Roster, Add Unit, and Settings (mobile only)
  - **Mobile Card View**: Compact card layout for roster units with status dots, location, and project ID
  - **Tap-to-Navigate**: Location addresses and coordinates are clickable and open in user's default navigation app (Google Maps, Apple Maps, Waze, etc.)
  - **Unit Detail Modal**: Bottom sheet modal showing full unit details with edit capabilities (tap any unit card to open)
  - **Touch Optimization**: 44x44px minimum button targets following iOS/Android accessibility guidelines
  - **Service Worker**: Network-first caching strategy for offline-capable operation
  - **IndexedDB Storage**: Offline data persistence for unit information and pending edits
  - **Background Sync**: Queues edits made while offline and syncs automatically when connection returns
  - **Offline Indicator**: Visual banner showing offline status with manual sync button
  - **PWA Manifest**: Installable as a standalone app on mobile devices with custom icons
  - **Hard Reload Button**: "Clear Cache & Reload" utility in sidebar menu to force fresh JavaScript/CSS
- **Mobile-Specific Files**:
  - `backend/static/mobile.css` - Mobile UI styles, hamburger menu, bottom nav, cards, modals
  - `backend/static/mobile.js` - Mobile interactions, offline sync, modal management
  - `backend/static/sw.js` - Service worker for PWA functionality
  - `backend/static/offline-db.js` - IndexedDB wrapper for offline storage
  - `backend/static/manifest.json` - PWA configuration
  - `backend/static/icons/` - 8 PWA icon sizes (72px-512px)

### Changed
- **Dashboard Alerts**: Only show Missing units in notifications (Pending units no longer appear in alerts)
- **Roster Template**: Mobile card view shows status from server-side render instead of fetching separately
- **Mobile Status Display**: Benched units show "Benched" label instead of "Unknown" or "N/A"
- **Base Template**: Added cache-busting query parameters to JavaScript files (e.g., `mobile.js?v=0.3.2`)
- **Sidebar Menu**: Added utility section with "Toggle theme" and "Clear Cache & Reload" buttons

### Fixed
- **Modal Status Display**: Fixed unit detail modal showing "Unknown" status by passing status data from card to modal
- **Mobile Card Status**: Fixed grey dot with "Unknown" label for benched units - now properly shows deployment state
- **Status Data Passing**: Roster cards now pass status and age to modal via function parameters and global status map
- **Service Worker Caching**: Aggressive browser caching issue resolved with version query parameters and hard reload function

### Technical Details
- Mobile breakpoint at 768px (`md:` prefix in TailwindCSS)
- PWA installable via Add to Home Screen on iOS/Android
- Service worker caches all static assets with network-first strategy
- Google Maps search API used for universal navigation links (works across all map apps)
- Status map stored in `window.rosterStatusMap` from server-side rendered data
- Hard reload function clears service worker caches, unregisters workers, and deletes IndexedDB

## [0.3.1] - 2025-12-12

### Fixed
- **Dashboard Notifications**: Removed Pending units from alert list - only Missing units now trigger notifications
- **Status Dots**: Verified deployed units display correct status dots (OK=green, Pending=yellow, Missing=red) in both active and benched tables
- **Mobile Card View**: Fixed roster cards showing "Unknown" status by using `.get()` with defaults in backend routes
- **Backend Status Handling**: Added default values for status, age, last_seen fields to prevent KeyError exceptions

### Changed
- Backend roster partial routes (`/partials/roster-deployed`, `/partials/roster-benched`) now use `.get()` method with sensible defaults
- Deployed units default to "Unknown" status when data unavailable
- Benched units default to "N/A" status when data unavailable

## [0.3.0] - 2025-12-09

### Added
- **Series 4 (Micromate) Support**: New `/api/series4/heartbeat` endpoint for receiving telemetry from Series 4 Micromate units
  - Auto-detection of Series 4 units via UM##### ID pattern
  - Stores project hints from emitter payload in unit notes
  - Automatic unit type classification across both Series 3 and Series 4 endpoints
- **Development Environment Labels**: Visual indicators to distinguish dev from production deployments
  - Yellow "DEV" badge in sidebar navigation
  - "[DEV]" prefix in browser title
  - Yellow banner on dashboard when running in development mode
  - Environment variable support in docker-compose.yml (ENVIRONMENT=production|development)
- **Quality of Life Improvements**:
  - Human-readable relative timestamps (e.g., "2h 15m ago", "3d ago") with full date in tooltips
  - "Last Updated" timestamp indicator on dashboard
  - Status icons for colorblind accessibility (checkmark for OK, clock for Pending, X for Missing)
  - Breadcrumb navigation on unit detail pages
  - Copy-to-clipboard buttons for unit IDs
  - Search/filter functionality for fleet roster table
  - Improved empty state messages with icons
- **Timezone Support**: Comprehensive timezone handling across the application
  - Timezone selector in Settings (defaults to America/New_York EST)
  - Human-readable timestamp format (e.g., "9/10/2020 8:00 AM EST")
  - Timezone-aware display for all timestamps site-wide
  - Settings stored in localStorage for immediate effect
- **Settings Page Redesign**: Complete overhaul with tabbed interface and persistent preferences
  - **General Tab**: Display preferences (timezone, theme, auto-refresh interval)
  - **Data Management Tab**: Safe operations (CSV export, merge import, roster table)
  - **Advanced Tab**: Power user settings (replace mode import, calibration defaults, status thresholds)
  - **Danger Zone Tab**: Destructive operations isolated with enhanced warnings
  - Backend preferences storage via new UserPreferences model
  - Tab state persistence in localStorage
  - Smooth animations and consistent styling with existing pages
- **User Preferences API**: New backend endpoints for persistent settings storage
  - `GET /api/settings/preferences` - Retrieve all user preferences
  - `PUT /api/settings/preferences` - Update preferences (supports partial updates)
  - Database-backed storage for cross-device preference sync
  - Migration script: `backend/migrate_add_user_preferences.py`

### Changed
- Timestamps now display in user-selected timezone with human-readable format throughout the application
- Settings page reorganized from 644-line flat layout to clean 4-tab interface
- CSV Replace Mode moved from Data Management to Advanced tab with additional warnings
- Import operations separated: safe merge in Data Management tab, destructive replace in Advanced tab
- Page title changed from "Roster Manager" to "Settings" for better clarity
- All preferences now persist to backend database instead of relying solely on localStorage

### Fixed
- Unit type classification now consistent across Series 3 and Series 4 heartbeat endpoints
- Auto-correction of misclassified unit types when they report to wrong endpoint

### Technical Details
- New `detect_unit_type()` helper function for pattern-based unit classification
- UserPreferences model with single-row table pattern (id=1) for global settings
- Series 4 units identified by UM prefix followed by digits (e.g., UM11719)
- JavaScript Intl API used for client-side timezone conversion
- Pydantic schema for partial preference updates (PreferencesUpdate model)
- Environment context injection via custom FastAPI template response wrapper

## [0.2.1] - 2025-12-03

### Added
- `/settings` roster manager page with CSV export/import, live stats, and danger-zone reset controls.
- `/api/settings` router that exposes `export-csv`, `stats`, `roster-units`, `import-csv-replace`, and the clear-* endpoints backing the UI.
- Dedicated HTMX partials/tabs for deployed, benched, retired, and ignored units plus new ignored-table UI to unignore or delete entries.

### Changed
- Roster and unit detail templates now display device-type specific metadata (calibration windows, modem pairings, IP/phone fields) alongside inline actions.
- Base navigation highlights the new settings workflow and routes retired/ignored buckets through dedicated endpoints + partials.

### Fixed
- Snapshot summary counts only consider deployed units, preventing dashboard alerts from including benched hardware.
- Snapshot payloads now include address/coordinate metadata so map widgets and CSV exports stay accurate.

## [0.2.0] - 2025-12-03

### Added
- Device-type aware roster schema (seismographs vs modems) with new metadata columns plus `backend/migrate_add_device_types.py` for upgrading existing SQLite files.
- `create_test_db.py` helper that generates a ready-to-use demo database with sample seismographs, modems, and emitter rows.
- Ignore list persistence/API so noisy legacy emitters can be quarantined via `/api/roster/ignore` and surfaced in the UI.
- Roster page enhancements: Add Unit modal, CSV import modal, and HTMX-powered table fed by `/partials/roster-table`.
- Unit detail view rewritten to fetch data via API, expose deployment status, and allow edits to all metadata.

### Changed
- Snapshot service now merges roster + emitter data into active/benched/retired/unknown buckets and includes device-specific metadata in each record.
- Roster edit endpoints parse date fields, manage modem/seismograph specific attributes, and guarantee records exist when toggling deployed/retired states.
- Dashboard partial endpoints are grouped under `/dashboard/*` so HTMX tabs stay in sync with the consolidated snapshot payload.

### Fixed
- Toggling deployed/retired flags no longer fails when a unit does not exist because the router now auto-creates placeholder roster rows.
- CSV import applies address/coordinate updates instead of silently dropping unknown columns.

## [0.1.1] - 2025-12-02

### Added
- **Roster Editing API**: Full CRUD operations for roster management
  - `POST /api/roster/add` - Add new units to roster
  - `POST /api/roster/set-deployed/{unit_id}` - Toggle deployment status
  - `POST /api/roster/set-retired/{unit_id}` - Toggle retired status
  - `POST /api/roster/set-note/{unit_id}` - Update unit notes
- **CSV Import**: Bulk roster import functionality
  - `POST /api/roster/import-csv` - Import units from CSV file
  - Support for all roster fields: unit_id, unit_type, deployed, retired, note, project_id, location
  - Optional update_existing parameter to control duplicate handling
  - Detailed import summary with added/updated/skipped/error counts
- **Enhanced Database Models**:
  - Added `project_id` field to RosterUnit model
  - Added `location` field to RosterUnit model
  - Added `last_updated` timestamp tracking
- **Dashboard Enhancements**:
  - Separate views for Active, Benched, and Retired units
  - New endpoints: `/dashboard/active` and `/dashboard/benched`

### Fixed
- Database session management bug in `emit_status_snapshot()`
  - Added `get_db_session()` helper function for direct session access
  - Implemented proper session cleanup with try/finally blocks
- Database schema synchronization issues
  - Database now properly recreates when model changes are detected

### Changed
- Updated RosterUnit model to include additional metadata fields
- Improved error handling in CSV import with row-level error reporting
- Enhanced snapshot service to properly manage database connections

### Technical Details
- All roster editing endpoints use Form data for better HTML form compatibility
- CSV import uses multipart/form-data for file uploads
- Boolean fields in CSV accept: 'true', '1', 'yes' (case-insensitive)
- Database sessions now properly closed to prevent connection leaks

## [0.1.0] - 2024-11-20

### Added
- Initial release of Seismo Fleet Manager
- FastAPI-based REST API for fleet management
- SQLite database with SQLAlchemy ORM
- Emitter reporting endpoints
- Basic fleet status monitoring
- Docker and Docker Compose support
- Web-based dashboard with HTMX
- Dark/light mode toggle
- Interactive maps with Leaflet
- Photo management per unit
- Automated status categorization (OK/Pending/Missing)

[0.4.2]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.4.1...v0.4.2
[0.4.1]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.4.0...v0.4.1
[0.4.0]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.3.3...v0.4.0
[0.3.3]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.3.2...v0.3.3
[0.3.2]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.3.1...v0.3.2
[0.3.1]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.3.0...v0.3.1
[0.3.0]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.2.1...v0.3.0
[0.2.1]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.2.0...v0.2.1
[0.2.0]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.1.1...v0.2.0
[0.1.1]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/serversdwn/seismo-fleet-manager/releases/tag/v0.1.0