serversdown/terra-view
1
0
Fork 0
Code Issues 10 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5b907c0cd7279c5a4f86d73bf00e21f3060f94e0
terra-view/CHANGELOG.md

451 lines
26 KiB
Markdown
Raw Blame History

# Changelog
All notable changes to Terra-View will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.5.0] - 2026-01-09
### Added
- **Unified Modular Monolith Architecture**: Complete architectural refactoring to modular monolith pattern
- **Three Feature Modules**: Seismo (seismograph fleet), SLM (sound level meters), UI (shared templates/static)
- **Module Isolation**: Each module has its own database, models, services, and routers
- **Shared Infrastructure**: Common utilities and API aggregation layer
- **Multi-Container Deployment**: Three Docker containers (terra-view, sfm, slmm) built from single codebase
- **SLMM Integration**: Sound Level Meter Manager fully integrated as `app/slm/` module
- Migrated from separate repository to unified codebase
- Complete NL43 device management API (`/api/nl43/*`)
- Database models for NL43Config and NL43Status
- NL43Client service for device communication
- FTP, TCP, and web interface support for NL43 devices
- **SLM Dashboard API Layer**: New dashboard endpoints bridge UI and device APIs
- `GET /api/slm-dashboard/stats` - Aggregate statistics (total units, online/offline, measuring/idle)
- `GET /api/slm-dashboard/units` - List all units with latest status
- `GET /api/slm-dashboard/live-view/{unit_id}` - Real-time measurement data
- `GET /api/slm-dashboard/config/{unit_id}` - Retrieve unit configuration
- `POST /api/slm-dashboard/config/{unit_id}` - Update unit configuration
- `POST /api/slm-dashboard/control/{unit_id}/{action}` - Send control commands (start, stop, pause, resume, reset, sleep, wake)
- `GET /api/slm-dashboard/test-modem/{unit_id}` - Test device connectivity
- **Repository Rebranding**: Renamed from `seismo-fleet-manager` to `terra-view`
- Reflects unified platform nature (seismo + SLM + future modules)
- Git remote updated to `terra-view.git`
- All references updated throughout codebase
### Changed
- **Project Structure**: Complete reorganization following modular monolith pattern
- `app/seismo/` - Seismograph fleet module (formerly `backend/`)
- `app/slm/` - Sound level meter module (integrated from SLMM)
- `app/ui/` - Shared templates and static assets
- `app/api/` - Cross-module API aggregation layer
- Removed `backend/` and `templates/` directories
- **Import Paths**: All imports updated from `backend.*` to `app.seismo.*` or `app.slm.*`
- **Database Initialization**: Each module initializes its own database tables
- Seismo database: `app/seismo/database.py`
- SLM database: `app/slm/database.py`
- **Docker Architecture**: Three-container deployment from single codebase
- `terra-view` (port 8001): Main UI/orchestrator with all modules
- `sfm` (port 8002): Seismograph Fleet Module API
- `slmm` (port 8100): Sound Level Meter Manager API
- All containers built from same unified codebase with different entry points
### Fixed
- **Template Path Issues**: Fixed seismo dashboard template references
- Updated `app/seismo/routers/dashboard.py` to use `app/ui/templates` directory
- Resolved 404 errors for `partials/benched_table.html` and `partials/active_table.html`
- **Module Import Errors**: Corrected SLMM module structure
- Fixed `app/slm/main.py` to import from `app.slm.routers` instead of `app.routers`
- Updated all SLMM internal imports to use `app.slm.*` namespace
- **Docker Build Issues**: Resolved file permission problems
- Fixed dashboard.py permissions for Docker COPY operations
- Ensured all source files readable during container builds
### Technical Details
- **Modular Monolith Benefits**:
- Single repository for easier development and deployment
- Module boundaries enforced through folder structure
- Shared dependencies managed in single requirements.txt
- Independent database schemas per module
- Clean separation of concerns with explicit module APIs
- **Migration Path**: Existing installations automatically migrate
- Import path updates applied programmatically
- Database schemas remain compatible
- No data migration required
- **Module Structure**: Each module follows consistent pattern
- `database.py` - SQLAlchemy models and session management
- `models.py` - Pydantic schemas and database models
- `routers.py` - FastAPI route definitions
- `services.py` - Business logic and external integrations
- **Container Communication**: Containers use host networking
- terra-view proxies to sfm and slmm containers
- Environment variables configure API URLs
- Health checks ensure container availability
### Migration Notes
- **Breaking Changes**: Import paths changed for all modules
- Old: `from backend.models import RosterUnit`
- New: `from app.seismo.models import RosterUnit`
- **Configuration Updates**: Environment variables for multi-container setup
- `SFM_API_URL=http://localhost:8002` - SFM backend endpoint
- `SLMM_API_URL=http://localhost:8100` - SLMM backend endpoint
- `MODULE_MODE=sfm|slmm` - Future flag for API-only containers
- **Repository Migration**: Update git remotes for renamed repository
```bash
git remote set-url origin ssh://git@10.0.0.2:2222/serversdown/terra-view.git
```
## [0.4.2] - 2026-01-05
### Added
- **SLM Configuration Interface**: Sound Level Meters can now be configured directly from the SLM dashboard
- Configuration modal with comprehensive SLM parameter editing
- TCP port configuration for SLM control connections (default: 2255)
- FTP port configuration for SLM data retrieval (default: 21)
- Modem assignment for network access or direct IP connection support
- Test Modem button with ping-based connectivity verification (shows IP and response time)
- Test SLM Connection button for end-to-end connectivity validation
- Dynamic form fields that hide/show based on modem selection
- **SLM Dashboard Endpoints**: New API routes for SLM management
- `GET /api/slm-dashboard/config/{unit_id}` - Load SLM configuration form
- `POST /api/slm-dashboard/config/{unit_id}` - Save SLM configuration
- `GET /api/slm-dashboard/test-modem/{modem_id}` - Ping modem for connectivity test
- **Database Schema Updates**: Added `slm_ftp_port` column to roster table
- Migration script: `scripts/add_slm_ftp_port.py`
- Supports both TCP (control) and FTP (data) port configuration per SLM unit
- **Docker Environment Enhancements**:
- Added `iputils-ping` and `curl` packages to Docker image for network diagnostics
- Health check endpoint support via curl
### Fixed
- **Form Validation**: Fixed 400 Bad Request error when adding modem units
- Form fields for device-specific parameters now properly disabled when hidden
- Empty string values for integer fields no longer cause validation failures
- JavaScript now disables hidden form sections to prevent unwanted data submission
- **Unit Status Accuracy**: Fixed issue where unit status was loading from a saved cache instead of actual last-heard time
- Unit status now accurately reflects real-time connectivity
- Status determination based on actual `slm_last_check` timestamp
### Changed
- **Roster Form Behavior**: Device-specific form fields are now disabled (not just hidden) when not applicable
- Prevents SLM fields from submitting when adding modems
- Prevents modem fields from submitting when adding SLMs
- Cleaner form submissions with only relevant data
- **Port Field Handling**: Backend now accepts port fields as strings and converts to integers
- Handles empty string values gracefully
- Proper type conversion with None fallback for empty values
### Technical Details
- Added `setFieldsDisabled()` helper function for managing form field state
- Updated `toggleDeviceFields()` and `toggleEditDeviceFields()` to disable/enable fields
- Backend type conversion: `slm_tcp_port` and `slm_ftp_port` accept strings, convert to int with empty string handling
- Modem ping uses subprocess with 1 packet, 2-second timeout, returns response time in milliseconds
- Configuration form uses 3-column grid layout for TCP Port, FTP Port, and Direct IP fields
## [0.4.1] - 2026-01-05
### Added
- **SLM Integration**: Sound Level Meters are now manageable in SFM
### Fixed
- Fixed an issue where unit status was loading from a saved cache and not based on when it was actually heard from last. Unit status is now accurate.
## [0.4.0] - 2025-12-16
### Added
- **Database Management System**: Comprehensive backup and restore capabilities
- **Manual Snapshots**: Create on-demand backups of the entire database with optional descriptions
- **Restore from Snapshot**: Restore database from any snapshot with automatic safety backup
- **Upload/Download Snapshots**: Transfer database snapshots to/from the server
- **Database Tab**: New dedicated tab in Settings for all database management operations
- **Database Statistics**: View database size, row counts by table, and last modified time
- **Snapshot Metadata**: Each snapshot includes creation time, description, size, and type (manual/automatic)
- **Safety Backups**: Automatic backup created before any restore operation
- **Remote Database Cloning**: Dev tools for cloning production database to remote development servers
- **Clone Script**: `scripts/clone_db_to_dev.py` for copying database over WAN
- **Network Upload**: Upload snapshots via HTTP to remote servers
- **Auto-restore**: Automatically restore uploaded database on target server
- **Authentication Support**: Optional token-based authentication for secure transfers
- **Automatic Backup Scheduler**: Background service for automated database backups
- **Configurable Intervals**: Set backup frequency (default: 24 hours)
- **Retention Management**: Automatically delete old backups (configurable keep count)
- **Manual Trigger**: Force immediate backup via API
- **Status Monitoring**: Check scheduler status and next scheduled run time
- **Background Thread**: Non-blocking operation using Python threading
- **Settings Reorganization**: Improved tab structure for better organization
- Renamed "Data Management" tab to "Roster Management"
- Moved CSV Replace Mode from Advanced tab to Roster Management tab
- Created dedicated Database tab for all backup/restore operations
- **Comprehensive Documentation**: New `docs/DATABASE_MANAGEMENT.md` guide covering:
- Manual snapshot creation and restoration workflows
- Download/upload procedures for off-site backups
- Remote database cloning setup and usage
- Automatic backup configuration and integration
- API reference for all database endpoints
- Best practices and troubleshooting guide
### Changed
- **Settings Tab Organization**: Restructured for better logical grouping
- **General**: Display preferences (timezone, theme, auto-refresh)
- **Roster Management**: CSV operations and roster table (now includes Replace Mode)
- **Database**: All backup/restore operations (NEW)
- **Advanced**: Power user settings (calibration, thresholds)
- **Danger Zone**: Destructive operations
- CSV Replace Mode warnings enhanced and moved to Roster Management context
### Technical Details
- **SQLite Backup API**: Uses native SQLite backup API for concurrent-safe snapshots
- **Metadata Tracking**: JSON sidecar files store snapshot metadata alongside database files
- **Atomic Operations**: Database restoration is atomic with automatic rollback on failure
- **File Structure**: Snapshots stored in `./data/backups/` with timestamped filenames
- **API Endpoints**: 7 new endpoints for database management operations
- **Backup Service**: `backend/services/database_backup.py` - Core backup/restore logic
- **Scheduler Service**: `backend/services/backup_scheduler.py` - Automatic backup automation
- **Clone Utility**: `scripts/clone_db_to_dev.py` - Remote database synchronization tool
### Security Considerations
- Snapshots contain full database data and should be secured appropriately
- Remote cloning supports optional authentication tokens
- Restore operations require safety backup creation by default
- All destructive operations remain in Danger Zone with warnings
### Migration Notes
No database migration required for v0.4.0. All new features use existing database structure and add new backup management capabilities without modifying the core schema.
## [0.3.3] - 2025-12-12
### Changed
- **Mobile Navigation**: Moved hamburger menu button from floating top-right to bottom navigation bar
- Bottom nav now shows: Menu (hamburger), Dashboard, Roster, Settings
- Removed "Add Unit" from bottom nav (still accessible via sidebar menu)
- Hamburger no longer floats over content on mobile
- **Status Dot Visibility**: Increased status dot size from 12px to 16px (w-3/h-3 → w-4/h-4) in dashboard fleet overview for better at-a-glance visibility
- Affects both Active and Benched tabs in dashboard
- Makes status colors (green/yellow/red) easier to spot during quick scroll
### Fixed
- **Location Navigation**: Moved tap-to-navigate functionality from roster card view to unit detail modal only
- Roster cards now show simple location text with pin emoji
- Navigation links (opening Maps app) only appear in the modal when tapping a unit
- Reduces visual clutter and accidental navigation triggers
### Technical Details
- Bottom navigation remains at 4 buttons, first button now triggers sidebar menu
- Removed standalone hamburger button element and associated CSS
- Modal already had navigation links, no changes needed there
## [0.3.2] - 2025-12-12
### Added
- **Progressive Web App (PWA) Mobile Optimization**: Complete mobile-first redesign for field deployment usage
- **Responsive Navigation**: Hamburger menu with slide-in sidebar for mobile, always-visible sidebar for desktop
- **Bottom Navigation Bar**: Quick access to Dashboard, Roster, Add Unit, and Settings (mobile only)
- **Mobile Card View**: Compact card layout for roster units with status dots, location, and project ID
- **Tap-to-Navigate**: Location addresses and coordinates are clickable and open in user's default navigation app (Google Maps, Apple Maps, Waze, etc.)
- **Unit Detail Modal**: Bottom sheet modal showing full unit details with edit capabilities (tap any unit card to open)
- **Touch Optimization**: 44x44px minimum button targets following iOS/Android accessibility guidelines
- **Service Worker**: Network-first caching strategy for offline-capable operation
- **IndexedDB Storage**: Offline data persistence for unit information and pending edits
- **Background Sync**: Queues edits made while offline and syncs automatically when connection returns
- **Offline Indicator**: Visual banner showing offline status with manual sync button
- **PWA Manifest**: Installable as a standalone app on mobile devices with custom icons
- **Hard Reload Button**: "Clear Cache & Reload" utility in sidebar menu to force fresh JavaScript/CSS
- **Mobile-Specific Files**:
- `backend/static/mobile.css` - Mobile UI styles, hamburger menu, bottom nav, cards, modals
- `backend/static/mobile.js` - Mobile interactions, offline sync, modal management
- `backend/static/sw.js` - Service worker for PWA functionality
- `backend/static/offline-db.js` - IndexedDB wrapper for offline storage
- `backend/static/manifest.json` - PWA configuration
- `backend/static/icons/` - 8 PWA icon sizes (72px-512px)
### Changed
- **Dashboard Alerts**: Only show Missing units in notifications (Pending units no longer appear in alerts)
- **Roster Template**: Mobile card view shows status from server-side render instead of fetching separately
- **Mobile Status Display**: Benched units show "Benched" label instead of "Unknown" or "N/A"
- **Base Template**: Added cache-busting query parameters to JavaScript files (e.g., `mobile.js?v=0.3.2`)
- **Sidebar Menu**: Added utility section with "Toggle theme" and "Clear Cache & Reload" buttons
### Fixed
- **Modal Status Display**: Fixed unit detail modal showing "Unknown" status by passing status data from card to modal
- **Mobile Card Status**: Fixed grey dot with "Unknown" label for benched units - now properly shows deployment state
- **Status Data Passing**: Roster cards now pass status and age to modal via function parameters and global status map
- **Service Worker Caching**: Aggressive browser caching issue resolved with version query parameters and hard reload function
### Technical Details
- Mobile breakpoint at 768px (`md:` prefix in TailwindCSS)
- PWA installable via Add to Home Screen on iOS/Android
- Service worker caches all static assets with network-first strategy
- Google Maps search API used for universal navigation links (works across all map apps)
- Status map stored in `window.rosterStatusMap` from server-side rendered data
- Hard reload function clears service worker caches, unregisters workers, and deletes IndexedDB
## [0.3.1] - 2025-12-12
### Fixed
- **Dashboard Notifications**: Removed Pending units from alert list - only Missing units now trigger notifications
- **Status Dots**: Verified deployed units display correct status dots (OK=green, Pending=yellow, Missing=red) in both active and benched tables
- **Mobile Card View**: Fixed roster cards showing "Unknown" status by using `.get()` with defaults in backend routes
- **Backend Status Handling**: Added default values for status, age, last_seen fields to prevent KeyError exceptions
### Changed
- Backend roster partial routes (`/partials/roster-deployed`, `/partials/roster-benched`) now use `.get()` method with sensible defaults
- Deployed units default to "Unknown" status when data unavailable
- Benched units default to "N/A" status when data unavailable
## [0.3.0] - 2025-12-09
### Added
- **Series 4 (Micromate) Support**: New `/api/series4/heartbeat` endpoint for receiving telemetry from Series 4 Micromate units
- Auto-detection of Series 4 units via UM##### ID pattern
- Stores project hints from emitter payload in unit notes
- Automatic unit type classification across both Series 3 and Series 4 endpoints
- **Development Environment Labels**: Visual indicators to distinguish dev from production deployments
- Yellow "DEV" badge in sidebar navigation
- "[DEV]" prefix in browser title
- Yellow banner on dashboard when running in development mode
- Environment variable support in docker-compose.yml (ENVIRONMENT=production|development)
- **Quality of Life Improvements**:
- Human-readable relative timestamps (e.g., "2h 15m ago", "3d ago") with full date in tooltips
- "Last Updated" timestamp indicator on dashboard
- Status icons for colorblind accessibility (checkmark for OK, clock for Pending, X for Missing)
- Breadcrumb navigation on unit detail pages
- Copy-to-clipboard buttons for unit IDs
- Search/filter functionality for fleet roster table
- Improved empty state messages with icons
- **Timezone Support**: Comprehensive timezone handling across the application
- Timezone selector in Settings (defaults to America/New_York EST)
- Human-readable timestamp format (e.g., "9/10/2020 8:00 AM EST")
- Timezone-aware display for all timestamps site-wide
- Settings stored in localStorage for immediate effect
- **Settings Page Redesign**: Complete overhaul with tabbed interface and persistent preferences
- **General Tab**: Display preferences (timezone, theme, auto-refresh interval)
- **Data Management Tab**: Safe operations (CSV export, merge import, roster table)
- **Advanced Tab**: Power user settings (replace mode import, calibration defaults, status thresholds)
- **Danger Zone Tab**: Destructive operations isolated with enhanced warnings
- Backend preferences storage via new UserPreferences model
- Tab state persistence in localStorage
- Smooth animations and consistent styling with existing pages
- **User Preferences API**: New backend endpoints for persistent settings storage
- `GET /api/settings/preferences` - Retrieve all user preferences
- `PUT /api/settings/preferences` - Update preferences (supports partial updates)
- Database-backed storage for cross-device preference sync
- Migration script: `backend/migrate_add_user_preferences.py`
### Changed
- Timestamps now display in user-selected timezone with human-readable format throughout the application
- Settings page reorganized from 644-line flat layout to clean 4-tab interface
- CSV Replace Mode moved from Data Management to Advanced tab with additional warnings
- Import operations separated: safe merge in Data Management tab, destructive replace in Advanced tab
- Page title changed from "Roster Manager" to "Settings" for better clarity
- All preferences now persist to backend database instead of relying solely on localStorage
### Fixed
- Unit type classification now consistent across Series 3 and Series 4 heartbeat endpoints
- Auto-correction of misclassified unit types when they report to wrong endpoint
### Technical Details
- New `detect_unit_type()` helper function for pattern-based unit classification
- UserPreferences model with single-row table pattern (id=1) for global settings
- Series 4 units identified by UM prefix followed by digits (e.g., UM11719)
- JavaScript Intl API used for client-side timezone conversion
- Pydantic schema for partial preference updates (PreferencesUpdate model)
- Environment context injection via custom FastAPI template response wrapper
## [0.2.1] - 2025-12-03
### Added
- `/settings` roster manager page with CSV export/import, live stats, and danger-zone reset controls.
- `/api/settings` router that exposes `export-csv`, `stats`, `roster-units`, `import-csv-replace`, and the clear-* endpoints backing the UI.
- Dedicated HTMX partials/tabs for deployed, benched, retired, and ignored units plus new ignored-table UI to unignore or delete entries.
### Changed
- Roster and unit detail templates now display device-type specific metadata (calibration windows, modem pairings, IP/phone fields) alongside inline actions.
- Base navigation highlights the new settings workflow and routes retired/ignored buckets through dedicated endpoints + partials.
### Fixed
- Snapshot summary counts only consider deployed units, preventing dashboard alerts from including benched hardware.
- Snapshot payloads now include address/coordinate metadata so map widgets and CSV exports stay accurate.
## [0.2.0] - 2025-12-03
### Added
- Device-type aware roster schema (seismographs vs modems) with new metadata columns plus `backend/migrate_add_device_types.py` for upgrading existing SQLite files.
- `create_test_db.py` helper that generates a ready-to-use demo database with sample seismographs, modems, and emitter rows.
- Ignore list persistence/API so noisy legacy emitters can be quarantined via `/api/roster/ignore` and surfaced in the UI.
- Roster page enhancements: Add Unit modal, CSV import modal, and HTMX-powered table fed by `/partials/roster-table`.
- Unit detail view rewritten to fetch data via API, expose deployment status, and allow edits to all metadata.
### Changed
- Snapshot service now merges roster + emitter data into active/benched/retired/unknown buckets and includes device-specific metadata in each record.
- Roster edit endpoints parse date fields, manage modem/seismograph specific attributes, and guarantee records exist when toggling deployed/retired states.
- Dashboard partial endpoints are grouped under `/dashboard/*` so HTMX tabs stay in sync with the consolidated snapshot payload.
### Fixed
- Toggling deployed/retired flags no longer fails when a unit does not exist because the router now auto-creates placeholder roster rows.
- CSV import applies address/coordinate updates instead of silently dropping unknown columns.
## [0.1.1] - 2025-12-02
### Added
- **Roster Editing API**: Full CRUD operations for roster management
- `POST /api/roster/add` - Add new units to roster
- `POST /api/roster/set-deployed/{unit_id}` - Toggle deployment status
- `POST /api/roster/set-retired/{unit_id}` - Toggle retired status
- `POST /api/roster/set-note/{unit_id}` - Update unit notes
- **CSV Import**: Bulk roster import functionality
- `POST /api/roster/import-csv` - Import units from CSV file
- Support for all roster fields: unit_id, unit_type, deployed, retired, note, project_id, location
- Optional update_existing parameter to control duplicate handling
- Detailed import summary with added/updated/skipped/error counts
- **Enhanced Database Models**:
- Added `project_id` field to RosterUnit model
- Added `location` field to RosterUnit model
- Added `last_updated` timestamp tracking
- **Dashboard Enhancements**:
- Separate views for Active, Benched, and Retired units
- New endpoints: `/dashboard/active` and `/dashboard/benched`
### Fixed
- Database session management bug in `emit_status_snapshot()`
- Added `get_db_session()` helper function for direct session access
- Implemented proper session cleanup with try/finally blocks
- Database schema synchronization issues
- Database now properly recreates when model changes are detected
### Changed
- Updated RosterUnit model to include additional metadata fields
- Improved error handling in CSV import with row-level error reporting
- Enhanced snapshot service to properly manage database connections
### Technical Details
- All roster editing endpoints use Form data for better HTML form compatibility
- CSV import uses multipart/form-data for file uploads
- Boolean fields in CSV accept: 'true', '1', 'yes' (case-insensitive)
- Database sessions now properly closed to prevent connection leaks
## [0.1.0] - 2024-11-20
### Added
- Initial release of Seismo Fleet Manager
- FastAPI-based REST API for fleet management
- SQLite database with SQLAlchemy ORM
- Emitter reporting endpoints
- Basic fleet status monitoring
- Docker and Docker Compose support
- Web-based dashboard with HTMX
- Dark/light mode toggle
- Interactive maps with Leaflet
- Photo management per unit
- Automated status categorization (OK/Pending/Missing)
[0.4.2]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.4.1...v0.4.2
[0.4.1]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.4.0...v0.4.1
[0.4.0]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.3.3...v0.4.0
[0.3.3]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.3.2...v0.3.3
[0.3.2]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.3.1...v0.3.2
[0.3.1]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.3.0...v0.3.1
[0.3.0]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.2.1...v0.3.0
[0.2.1]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.2.0...v0.2.1
[0.2.0]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.1.1...v0.2.0
[0.1.1]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.1.0...v0.1.1
[0.1.0]: https://github.com/serversdwn/seismo-fleet-manager/releases/tag/v0.1.0
Reference in New Issue View Git Blame Copy Permalink