terra-view/README.md

# Terra-View v0.5.0
Unified platform for managing seismograph fleets and sound level meter deployments. Built as a modular monolith with independent feature modules (Seismo, SLM) sharing a common UI layer. Track deployments, monitor health in real time, merge roster intent with incoming telemetry, and control your entire fleet through a unified database and dashboard.

## Architecture

Terra-View follows a **modular monolith** architecture with independent feature modules in a single codebase:

- **app/seismo/** - Seismograph Fleet Module (SFM)
  - Device roster and deployment tracking
  - Series 3/4 telemetry ingestion
  - Status monitoring (OK/Pending/Missing)
  - Photo management and location tracking
- **app/slm/** - Sound Level Meter Manager (SLMM)
  - NL43 device configuration and control
  - Real-time measurement monitoring
  - TCP/FTP/Web interface support
  - Dashboard statistics and unit management
- **app/ui/** - Shared UI layer
  - Templates, static assets, and common components
  - Progressive Web App (PWA) support
- **app/api/** - API aggregation layer
  - Cross-module endpoints
  - Future unified dashboard APIs

**Multi-Container Deployment**: Three Docker containers built from the same codebase:
- `terra-view` (port 8001) - Main UI with all modules integrated
- `sfm` (port 8002) - Seismo API backend
- `slmm` (port 8100) - SLM API backend

## Features

- **Progressive Web App (PWA)**: Mobile-first responsive design optimized for field deployment operations
  - **Install as App**: Add to home screen on iOS/Android for native app experience
  - **Offline Capable**: Service worker caching with IndexedDB storage for offline operation
  - **Touch Optimized**: 44x44px minimum touch targets, hamburger menu, bottom navigation bar
  - **Mobile Card View**: Compact unit cards with status dots, tap-to-navigate locations, and detail modals
  - **Background Sync**: Queue edits while offline and automatically sync when connection returns
- **Web Dashboard**: Modern, responsive UI with dark/light mode, live HTMX updates, and integrated fleet map
- **Fleet Monitoring**: Track deployed, benched, retired, and ignored units in separate buckets with unknown-emitter triage
- **Roster Management**: Full CRUD + CSV import/export, device-type aware metadata, and inline actions from the roster tables
- **Settings & Safeguards**: `/settings` page exposes roster stats, exports, replace-all imports, and danger-zone reset tools
- **Device & Modem Metadata**: Capture calibration windows, modem pairings, phone/IP details, and addresses per unit
- **Status Management**: Automatically mark deployed units as OK, Pending (>12h), or Missing (>24h) based on recent telemetry
- **Data Ingestion**: Accept reports from emitter scripts via REST API
- **Photo Management**: Upload and view photos for each unit
- **Interactive Maps**: Leaflet-based maps showing unit locations with tap-to-navigate for mobile
- **SQLite Storage**: Lightweight, file-based database for easy deployment
- **Database Management**: Comprehensive backup and restore system
  - **Manual Snapshots**: Create on-demand backups with descriptions
  - **Restore from Snapshot**: Restore database with automatic safety backups
  - **Upload/Download**: Transfer database snapshots for off-site storage
  - **Remote Cloning**: Copy production database to remote dev servers over WAN
  - **Automatic Backups**: Scheduled background backups with configurable retention

## Roster Manager & Settings

Visit [`/settings`](http://localhost:8001/settings) to perform bulk roster operations with guardrails:

- **CSV export/import**: Download the entire roster, merge updates, or replace all units in one transaction.
- **Live roster table**: Fetch every unit via HTMX, edit metadata, toggle deployed/retired states, move emitters to the ignore list, or delete records in-place.
- **Database backups**: Create snapshots, restore from backups, upload/download database files, view database statistics.
- **Remote cloning**: Clone production database to remote development servers over the network (see `scripts/clone_db_to_dev.py`).
- **Stats at a glance**: View counts for the roster, emitters, and ignored units to confirm import/cleanup operations worked.
- **Danger zone controls**: Clear specific tables or wipe all fleet data when resetting a lab/demo environment.

All UI actions call `GET/POST /api/settings/*` endpoints so you can automate the same workflows from scripts. See [docs/DATABASE_MANAGEMENT.md](docs/DATABASE_MANAGEMENT.md) for comprehensive database backup and restore documentation.

## Tech Stack

- **FastAPI**: Modern, fast web framework
- **SQLAlchemy**: SQL toolkit and ORM
- **SQLite**: Lightweight database
- **HTMX**: Dynamic updates without heavy JavaScript frameworks
- **TailwindCSS**: Utility-first CSS framework
- **Leaflet**: Interactive maps
- **Jinja2**: Server-side templating
- **uvicorn**: ASGI server
- **Docker**: Containerization for easy deployment

## Quick Start with Docker Compose (Recommended)

### Prerequisites
- Docker and Docker Compose installed

### Running the Application

1. **Start the service:**
   ```bash
   docker compose up -d
   ```

2. **Check logs:**
   ```bash
   docker compose logs -f
   ```

3. **Stop the service:**
   ```bash
   docker compose down
   ```

The application will be available at:
- **Web Interface**: http://localhost:8001
- **API Documentation**: http://localhost:8001/docs
- **Health Check**: http://localhost:8001/health

### Data Persistence

The SQLite database and photos are stored in the `./data` directory, which is mounted as a volume. Your data will persist even if you restart or rebuild the container.

## Local Development (Without Docker)

### Prerequisites
- Python 3.11+
- pip

### Setup

1. **Install dependencies:**
   ```bash
   pip install -r requirements.txt
   ```

2. **Run the server:**
   ```bash
   uvicorn backend.main:app --host 0.0.0.0 --port 8001 --reload
   ```

The application will be available at http://localhost:8001

### Optional: Generate Sample Data

Need realistic data quickly? Run:

```bash
python create_test_db.py
cp /tmp/sfm_test.db data/seismo_fleet.db
```

The helper script creates a modem/seismograph mix so you can exercise the dashboard, roster tabs, and unit detail screens immediately.

## Upgrading from Previous Versions

### From v0.2.x to v0.3.0

Version 0.3.0 introduces user preferences storage. Run the migration once per database file:

```bash
python backend/migrate_add_user_preferences.py
```

This creates the `user_preferences` table for persistent settings storage (timezone, theme, auto-refresh interval, calibration defaults, status thresholds).

### From v0.1.x to v0.2.x or later

Versions ≥0.2 introduce new roster columns (device_type, calibration dates, modem metadata, addresses, etc.). Run the migration once per database file:

```bash
python backend/migrate_add_device_types.py
```

Both migration scripts are idempotent—if the columns/tables already exist, they simply exit.

## API Endpoints

### Web Pages
- **GET** `/` - Dashboard home page
- **GET** `/roster` - Fleet roster page
- **GET** `/unit/{unit_id}` - Unit detail page
- **GET** `/settings` - Roster manager, CSV import/export, and danger-zone utilities

### Fleet Status & Monitoring
- **GET** `/api/status-snapshot` - Complete fleet status snapshot
- **GET** `/api/roster` - List of all units with metadata
- **GET** `/api/unit/{unit_id}` - Detailed unit information
- **GET** `/health` - Health check endpoint

### Roster Management
- **POST** `/api/roster/add` - Add new unit to roster
  ```bash
  curl -X POST http://localhost:8001/api/roster/add \
    -F "id=BE1234" \
    -F "device_type=seismograph" \
    -F "unit_type=series3" \
    -F "project_id=PROJ-001" \
    -F "deployed=true" \
    -F "note=Main site sensor"
  ```
- **GET** `/api/roster/{unit_id}` - Fetch a single roster entry for editing
- **POST** `/api/roster/edit/{unit_id}` - Update all metadata (device type, calibration dates, modem fields, etc.)
- **POST** `/api/roster/set-deployed/{unit_id}` - Toggle deployment status
- **POST** `/api/roster/set-retired/{unit_id}` - Toggle retired status
- **POST** `/api/roster/set-note/{unit_id}` - Update unit notes
- **DELETE** `/api/roster/{unit_id}` - Remove a roster/emitter pair entirely
- **POST** `/api/roster/import-csv` - Bulk import from CSV (merge/update mode)
  ```bash
  curl -X POST http://localhost:8001/api/roster/import-csv \
    -F "file=@roster.csv" \
    -F "update_existing=true"
  ```
- **POST** `/api/roster/ignore/{unit_id}` - Move an unknown emitter to the ignore list
- **DELETE** `/api/roster/ignore/{unit_id}` - Remove a unit from the ignore list
- **GET** `/api/roster/ignored` - List all ignored units with reasons

### Settings & Data Management
- **GET** `/api/settings/export-csv` - Download the entire roster as CSV
- **GET** `/api/settings/stats` - Counts for roster, emitters, and ignored tables
- **GET** `/api/settings/roster-units` - Raw roster dump for the settings data grid
- **POST** `/api/settings/import-csv-replace` - Replace the entire roster in one atomic transaction
- **GET** `/api/settings/preferences` - Get user preferences (timezone, theme, calibration defaults, etc.)
- **PUT** `/api/settings/preferences` - Update user preferences (supports partial updates)
- **POST** `/api/settings/clear-all` - Danger-zone action that wipes roster, emitters, and ignored tables
- **POST** `/api/settings/clear-roster` - Delete only roster entries
- **POST** `/api/settings/clear-emitters` - Delete auto-discovered emitters
- **POST** `/api/settings/clear-ignored` - Reset ignore list

### Database Management
- **GET** `/api/settings/database/stats` - Database size, row counts, and last modified time
- **POST** `/api/settings/database/snapshot` - Create manual database snapshot with optional description
- **GET** `/api/settings/database/snapshots` - List all available snapshots with metadata
- **GET** `/api/settings/database/snapshot/{filename}` - Download a specific snapshot file
- **DELETE** `/api/settings/database/snapshot/{filename}` - Delete a snapshot
- **POST** `/api/settings/database/restore` - Restore database from snapshot (creates safety backup)
- **POST** `/api/settings/database/upload-snapshot` - Upload snapshot file to server

See [docs/DATABASE_MANAGEMENT.md](docs/DATABASE_MANAGEMENT.md) for detailed documentation and examples.

### CSV Import Format
Create a CSV file with the following columns (only `unit_id` is required, everything else is optional):

```
unit_id,unit_type,device_type,deployed,retired,note,project_id,location,address,coordinates,last_calibrated,next_calibration_due,deployed_with_modem_id,ip_address,phone_number,hardware_model
```

Boolean columns accept `true/false`, `1/0`, or `yes/no` (case-insensitive). Date columns expect `YYYY-MM-DD`. Use the same schema whether you merge via `/api/roster/import-csv` or replace everything with `/api/settings/import-csv-replace`.

#### Example

```csv
unit_id,unit_type,device_type,deployed,retired,note,project_id,location,address,coordinates,last_calibrated,next_calibration_due,deployed_with_modem_id,ip_address,phone_number,hardware_model
BE1234,series3,seismograph,true,false,Primary sensor,PROJ-001,"Station A","123 Market St, San Francisco, CA","37.7937,-122.3965",2025-01-15,2026-01-15,MDM001,,,
MDM001,modem,modem,true,false,Field modem,PROJ-001,"Station A","123 Market St, San Francisco, CA","37.7937,-122.3965",,,,"192.0.2.10","+1-555-0100","Raven XTV"
```

See [sample_roster.csv](sample_roster.csv) for a minimal working example.

### Emitter Reporting
- **POST** `/emitters/report` - Submit status report from a seismograph unit
- **POST** `/api/series3/heartbeat` - Series 3 multi-unit telemetry payload
- **POST** `/api/series4/heartbeat` - Series 4 (Micromate) multi-unit telemetry payload
- **GET** `/fleet/status` - Retrieve status of all seismograph units (legacy)

### Photo Management
- **GET** `/api/unit/{unit_id}/photos` - List photos for a unit
- **GET** `/api/unit/{unit_id}/photo/{filename}` - Serve specific photo file

## API Documentation

Once running, interactive API documentation is available at:
- **Swagger UI**: http://localhost:8001/docs
- **ReDoc**: http://localhost:8001/redoc

## Testing the API

### Using curl

**Submit a report:**
```bash
curl -X POST http://localhost:8001/emitters/report \
  -H "Content-Type: application/json" \
  -d '{
    "unit": "SEISMO-001",
    "unit_type": "series3",
    "timestamp": "2025-11-20T10:30:00",
    "file": "event_20251120_103000.dat",
    "status": "OK"
  }'
```

**Get fleet status:**
```bash
curl http://localhost:8001/api/roster
```

**Import roster from CSV:**
```bash
curl -X POST http://localhost:8001/api/roster/import-csv \
  -F "file=@sample_roster.csv" \
  -F "update_existing=true"
```

### Using Python

```python
import requests
from datetime import datetime

# Submit report
response = requests.post(
    "http://localhost:8001/emitters/report",
    json={
        "unit": "SEISMO-001",
        "unit_type": "series3",
        "timestamp": datetime.utcnow().isoformat(),
        "file": "event_20251120_103000.dat",
        "status": "OK"
    }
)
print(response.json())

# Get fleet status
response = requests.get("http://localhost:8001/api/roster")
print(response.json())

# Import CSV
with open('roster.csv', 'rb') as f:
    files = {'file': f}
    data = {'update_existing': 'true'}
    response = requests.post(
        "http://localhost:8001/api/roster/import-csv",
        files=files,
        data=data
    )
print(response.json())
```

## Data Model

### RosterUnit Table (Fleet Roster)

**Common fields**

| Field | Type | Description |
|-------|------|-------------|
| id | string | Unit identifier (primary key) |
| unit_type | string | Hardware model name (default: `series3`) |
| device_type | string | `seismograph` or `modem` discriminator |
| deployed | boolean | Whether the unit is in the field |
| retired | boolean | Removes the unit from deployments but preserves history |
| note | string | Notes about the unit |
| project_id | string | Associated project identifier |
| location | string | Legacy location label |
| address | string | Human-readable address |
| coordinates | string | `lat,lon` pair used by the map |
| last_updated | datetime | Last modification timestamp |

**Seismograph-only fields**

| Field | Type | Description |
|-------|------|-------------|
| last_calibrated | date | Last calibration date |
| next_calibration_due | date | Next calibration date |
| deployed_with_modem_id | string | Which modem is paired during deployment |

**Modem-only fields**

| Field | Type | Description |
|-------|------|-------------|
| ip_address | string | Assigned IP (static or DHCP) |
| phone_number | string | Cellular number for the modem |
| hardware_model | string | Modem hardware reference |

### Emitter Table (Device Check-ins)

| Field | Type | Description |
|-------|------|-------------|
| id | string | Unit identifier (primary key) |
| unit_type | string | Reported device type/model |
| last_seen | datetime | Last report timestamp |
| last_file | string | Last file processed |
| status | string | Current status: OK, Pending, Missing |
| notes | string | Optional notes (nullable) |

### IgnoredUnit Table (Noise Management)

| Field | Type | Description |
|-------|------|-------------|
| id | string | Unit identifier (primary key) |
| reason | string | Optional context for ignoring |
| ignored_at | datetime | When the ignore action occurred |

### UserPreferences Table (Settings Storage)

| Field | Type | Description |
|-------|------|-------------|
| id | integer | Always 1 (single-row table) |
| timezone | string | Display timezone (default: America/New_York) |
| theme | string | UI theme: auto, light, or dark |
| auto_refresh_interval | integer | Dashboard refresh interval in seconds |
| date_format | string | Date format preference |
| table_rows_per_page | integer | Default pagination size |
| calibration_interval_days | integer | Default days between calibrations |
| calibration_warning_days | integer | Warning threshold before calibration due |
| status_ok_threshold_hours | integer | Hours for OK status threshold |
| status_pending_threshold_hours | integer | Hours for Pending status threshold |
| updated_at | datetime | Last preference update timestamp |

## Project Structure

```
seismo-fleet-manager/
├── backend/
│   ├── main.py              # FastAPI app entry point
│   ├── database.py          # SQLAlchemy database configuration
│   ├── models.py            # Database models (RosterUnit, Emitter, IgnoredUnit, UserPreferences)
│   ├── routes.py            # Legacy API endpoints + Series 3/4 heartbeat endpoints
│   ├── routers/             # Modular API routers
│   │   ├── roster.py        # Fleet status endpoints
│   │   ├── roster_edit.py   # Roster management & CSV import
│   │   ├── units.py         # Unit detail endpoints
│   │   ├── photos.py        # Photo management
│   │   ├── dashboard.py     # Dashboard partials
│   │   ├── dashboard_tabs.py # Dashboard tab endpoints
│   │   └── settings.py      # Settings, preferences, and data management
│   ├── services/
│   │   ├── snapshot.py      # Fleet status snapshot logic
│   │   ├── database_backup.py # Database backup and restore service
│   │   └── backup_scheduler.py # Automatic backup scheduler
│   ├── migrate_add_device_types.py # SQLite migration for v0.2 schema
│   ├── migrate_add_user_preferences.py # SQLite migration for v0.3 schema
│   └── static/              # Static assets (CSS, etc.)
├── create_test_db.py        # Generate a sample SQLite DB with mixed devices
├── templates/               # Jinja2 HTML templates
│   ├── base.html            # Base layout with sidebar
│   ├── dashboard.html       # Main dashboard
│   ├── roster.html          # Fleet roster table
│   ├── unit_detail.html     # Unit detail page
│   ├── settings.html        # Roster manager UI
│   └── partials/            # HTMX partial templates
│       ├── roster_table.html
│       ├── retired_table.html
│       ├── ignored_table.html
│       └── unknown_emitters.html
├── data/                    # SQLite database & photos (persisted)
│   └── backups/             # Database snapshots directory
├── scripts/
│   └── clone_db_to_dev.py   # Remote database cloning utility
├── docs/
│   └── DATABASE_MANAGEMENT.md # Database backup/restore guide
├── requirements.txt         # Python dependencies
├── Dockerfile               # Docker container definition
├── docker-compose.yml       # Docker Compose configuration
├── CHANGELOG.md             # Version history
├── FRONTEND_README.md       # Frontend documentation
└── README.md                # This file
```

## Docker Commands

**Build the image:**
```bash
docker compose build
```

**Start in foreground:**
```bash
docker compose up
```

**Start in background:**
```bash
docker compose up -d
```

**View logs:**
```bash
docker compose logs -f seismo-backend
```

**Restart service:**
```bash
docker compose restart
```

**Rebuild and restart:**
```bash
docker compose up -d --build
```

**Stop and remove containers:**
```bash
docker compose down
```

**Remove containers and volumes:**
```bash
docker compose down -v
```

## Release Highlights

### v0.4.0 — 2025-12-16
- **Database Management System**: Complete backup and restore functionality with manual snapshots, restore operations, and upload/download capabilities
- **Remote Database Cloning**: New `clone_db_to_dev.py` script for copying production database to remote dev servers over WAN
- **Automatic Backup Scheduler**: Background service for scheduled backups with configurable retention management
- **Database Tab**: New dedicated tab in Settings for all database operations with real-time statistics
- **Settings Reorganization**: Improved tab structure - renamed "Data Management" to "Roster Management", moved CSV Replace Mode, created Database tab
- **Comprehensive Documentation**: New `docs/DATABASE_MANAGEMENT.md` with complete guide to backup/restore workflows, API reference, and best practices

### v0.3.3 — 2025-12-12
- **Improved Mobile Navigation**: Hamburger menu moved to bottom nav bar (no more floating button covering content)
- **Better Status Visibility**: Larger status dots (16px) in dashboard fleet overview for easier at-a-glance status checks
- **Cleaner Roster Cards**: Location navigation links moved to detail modal only, reducing clutter in card view

### v0.3.2 — 2025-12-12
- **Progressive Web App (PWA)**: Complete mobile optimization with offline support, installable as standalone app
- **Mobile-First UI**: Hamburger menu, bottom navigation bar, card-based roster view optimized for touch
- **Tap-to-Navigate**: Location links open in user's preferred navigation app (Google Maps, Apple Maps, Waze)
- **Offline Editing**: Service worker + IndexedDB for offline operation with automatic sync when online
- **Unit Detail Modals**: Bottom sheet modals for quick unit info access with full edit capabilities
- **Hard Reload Utility**: "Clear Cache & Reload" button to force fresh assets (helpful for development)

### v0.3.1 — 2025-12-12
- **Dashboard Alerts**: Only Missing units show in notifications (Pending units no longer alert)
- **Status Fixes**: Fixed "Unknown" status issues in mobile card views and detail modals
- **Backend Improvements**: Safer data access with `.get()` defaults to prevent errors

### v0.3.0 — 2025-12-09
- **Series 4 Support**: New `/api/series4/heartbeat` endpoint with auto-detection for Micromate units (UM##### pattern)
- **Settings Redesign**: Completely redesigned Settings page with 4-tab interface (General, Data Management, Advanced, Danger Zone)
- **User Preferences**: Backend storage for timezone, theme, auto-refresh interval, calibration defaults, and status thresholds
- **Development Labels**: Visual indicators to distinguish dev from production environments
- **Timezone Support**: Comprehensive timezone handling with human-readable timestamps site-wide
- **Quality of Life**: Relative timestamps, status icons for accessibility, breadcrumb navigation, copy-to-clipboard, search functionality

### v0.2.1 — 2025-12-03
- Added the `/settings` roster manager with CSV export/import, live stats, and danger-zone table reset actions
- Deployed/Benched/Retired/Ignored tabs now have dedicated HTMX partials, sorting, and inline actions
- Unit detail pages expose device-type specific metadata (calibration windows, modem pairing, IP/phone fields)
- Snapshot summary and dashboard counts now focus on deployed units and include address/coordinate data

### v0.2.0 — 2025-12-03
- Introduced device-type aware roster schema (seismograph vs modem) plus migration + `create_test_db.py` helper
- Added Ignore list model/endpoints to quarantine noisy emitters directly from the roster
- Roster page gained Add Unit + CSV Import modals, HTMX-driven updates, and unknown emitter callouts
- Snapshot service now returns active/benched/retired/unknown buckets containing richer metadata

### v0.1.1 — 2025-12-02
- **Roster Editing API**: Full CRUD operations for managing your fleet roster
- **CSV Import**: Bulk upload roster data from CSV files
- **Enhanced Data Model**: Added project_id and location fields to roster
- **Bug Fixes**: Improved database session management and error handling

See [CHANGELOG.md](CHANGELOG.md) for the full release notes.

## Future Enhancements

- Email/SMS alerts for missing units
- Historical data tracking and reporting
- Multi-user authentication
- PostgreSQL support for larger deployments
- Advanced filtering and search
- Export roster to various formats

## License

MIT

## Version

**Current: 0.4.0** — Database management system with backup/restore and remote cloning (2025-12-16)

Previous: 0.3.3 — Mobile navigation improvements and better status visibility (2025-12-12)

0.3.2 — Progressive Web App with mobile optimization (2025-12-12)

0.3.1 — Dashboard alerts and status fixes (2025-12-12)

0.3.0 — Series 4 support, settings redesign, user preferences (2025-12-09)

0.2.1 — Settings & roster manager refresh (2025-12-03)

0.2.0 — Device-type aware roster + ignore list (2025-12-03)

0.1.1 — Roster Management & CSV Import (2025-12-02)

0.1.0 — Initial Release (2024-11-20)