terra-view/docs/DATABASE_MANAGEMENT.md

Database Management Guide

This guide covers the comprehensive database management features available in the Seismo Fleet Manager, including manual snapshots, restoration, remote cloning, and automatic backups.

Table of Contents

1. Manual Database Snapshots
2. Restore from Snapshot
3. Download and Upload Snapshots
4. Clone Database to Dev Server
5. Automatic Backup Service
6. API Reference

---

Manual Database Snapshots

Creating a Snapshot via UI

1. Navigate to Settings → Danger Zone tab
2. Scroll to the Database Management section
3. Click "Create Snapshot"
4. Optionally enter a description
5. The snapshot will be created and appear in the "Available Snapshots" list

Creating a Snapshot via API

curl -X POST http://localhost:8000/api/settings/database/snapshot \
  -H "Content-Type: application/json" \
  -d '{"description": "Pre-deployment backup"}'

What Happens

• A full copy of the SQLite database is created using the SQLite backup API
• The snapshot is stored in ./data/backups/ directory
• A metadata JSON file is created alongside the snapshot
• No downtime or interruption to the running application

Snapshot Files

Snapshots are stored as:

• Database file: snapshot_YYYYMMDD_HHMMSS.db
• Metadata file: snapshot_YYYYMMDD_HHMMSS.db.meta.json

Example:

data/backups/
├── snapshot_20250101_143022.db
├── snapshot_20250101_143022.db.meta.json
├── snapshot_20250102_080000.db
└── snapshot_20250102_080000.db.meta.json

---

Restore from Snapshot

Restoring via UI

1. Navigate to Settings → Danger Zone tab
2. In the Available Snapshots section, find the snapshot you want to restore
3. Click the restore icon (circular arrow) next to the snapshot
4. Confirm the restoration warning
5. A safety backup of the current database is automatically created
6. The database is replaced with the snapshot
7. The page reloads automatically

Restoring via API

curl -X POST http://localhost:8000/api/settings/database/restore \
  -H "Content-Type: application/json" \
  -d '{
    "filename": "snapshot_20250101_143022.db",
    "create_backup": true
  }'

Important Notes

• Always creates a safety backup before restoring (unless explicitly disabled)
• Application reload required - Users should refresh their browsers
• Atomic operation - The entire database is replaced at once
• Cannot be undone - But you'll have the safety backup

---

Download and Upload Snapshots

Download a Snapshot

Via UI: Click the download icon next to any snapshot in the list

Via Browser:

http://localhost:8000/api/settings/database/snapshot/snapshot_20250101_143022.db

Via Command Line:

curl -o backup.db http://localhost:8000/api/settings/database/snapshot/snapshot_20250101_143022.db

Upload a Snapshot

Via UI:

1. Navigate to Settings → Danger Zone tab
2. Find the Upload Snapshot section
3. Click "Choose File" and select a .db file
4. Click "Upload Snapshot"

Via Command Line:

curl -X POST http://localhost:8000/api/settings/database/upload-snapshot \
  -F "file=@/path/to/your/backup.db"

---

Clone Database to Dev Server

The clone tool allows you to copy the production database to a remote development server over the network.

Prerequisites

• Remote dev server must have the same Seismo Fleet Manager installation
• Network connectivity between production and dev servers
• Python 3 and requests library installed

Basic Usage

# Clone current database to dev server
python3 scripts/clone_db_to_dev.py --url https://dev.example.com

# Clone using existing snapshot
python3 scripts/clone_db_to_dev.py \
  --url https://dev.example.com \
  --snapshot snapshot_20250101_143022.db

# Clone with authentication token
python3 scripts/clone_db_to_dev.py \
  --url https://dev.example.com \
  --token YOUR_AUTH_TOKEN

What Happens

1. Creates a snapshot of the production database (or uses existing one)
2. Uploads the snapshot to the remote dev server
3. Automatically restores the snapshot on the dev server
4. Creates a safety backup on the dev server before restoring

Remote Server Setup

The remote dev server needs no special setup - it just needs to be running the same Seismo Fleet Manager application with the database management endpoints enabled.

Use Cases

• Testing: Test changes against production data in a dev environment
• Debugging: Investigate production issues with real data safely
• Training: Provide realistic data for user training
• Development: Build new features with realistic data

---

Automatic Backup Service

The automatic backup service runs scheduled backups in the background and manages backup retention.

Configuration

The backup scheduler can be configured programmatically or via environment variables.

Programmatic Configuration:

from backend.services.backup_scheduler import get_backup_scheduler

scheduler = get_backup_scheduler()
scheduler.configure(
    interval_hours=24,  # Backup every 24 hours
    keep_count=10,      # Keep last 10 backups
    enabled=True        # Enable automatic backups
)
scheduler.start()

Environment Variables (add to your .env or deployment config):

AUTO_BACKUP_ENABLED=true
AUTO_BACKUP_INTERVAL_HOURS=24
AUTO_BACKUP_KEEP_COUNT=10

Integration with Application Startup

Add to backend/main.py:

from backend.services.backup_scheduler import get_backup_scheduler

@app.on_event("startup")
async def startup_event():
    # Start automatic backup scheduler
    scheduler = get_backup_scheduler()
    scheduler.configure(
        interval_hours=24,  # Daily backups
        keep_count=10,      # Keep 10 most recent
        enabled=True
    )
    scheduler.start()

@app.on_event("shutdown")
async def shutdown_event():
    # Stop backup scheduler gracefully
    scheduler = get_backup_scheduler()
    scheduler.stop()

Manual Control

from backend.services.backup_scheduler import get_backup_scheduler

scheduler = get_backup_scheduler()

# Get current status
status = scheduler.get_status()
print(status)
# {'enabled': True, 'running': True, 'interval_hours': 24, 'keep_count': 10, 'next_run': '2025-01-02T14:00:00'}

# Create backup immediately
scheduler.create_automatic_backup()

# Stop scheduler
scheduler.stop()

# Start scheduler
scheduler.start()

Backup Retention

The scheduler automatically deletes old backups based on the keep_count setting. For example, if keep_count=10, only the 10 most recent backups are kept, and older ones are automatically deleted.

---

API Reference

Database Statistics

GET /api/settings/database/stats

Returns database size, row counts, and last modified time.

Response:

{
  "database_path": "./data/seismo_fleet.db",
  "size_bytes": 1048576,
  "size_mb": 1.0,
  "total_rows": 1250,
  "tables": {
    "roster": 450,
    "emitters": 600,
    "ignored_units": 50,
    "unit_history": 150
  },
  "last_modified": "2025-01-01T14:30:22"
}

Create Snapshot

POST /api/settings/database/snapshot
Content-Type: application/json

{
  "description": "Optional description"
}

Response:

{
  "message": "Snapshot created successfully",
  "snapshot": {
    "filename": "snapshot_20250101_143022.db",
    "created_at": "20250101_143022",
    "created_at_iso": "2025-01-01T14:30:22",
    "description": "Optional description",
    "size_bytes": 1048576,
    "size_mb": 1.0,
    "type": "manual"
  }
}

List Snapshots

GET /api/settings/database/snapshots

Response:

{
  "snapshots": [
    {
      "filename": "snapshot_20250101_143022.db",
      "created_at": "20250101_143022",
      "created_at_iso": "2025-01-01T14:30:22",
      "description": "Manual backup",
      "size_mb": 1.0,
      "type": "manual"
    }
  ],
  "count": 1
}

Download Snapshot

GET /api/settings/database/snapshot/{filename}

Returns the snapshot file as a download.

Delete Snapshot

DELETE /api/settings/database/snapshot/{filename}

Restore Database

POST /api/settings/database/restore
Content-Type: application/json

{
  "filename": "snapshot_20250101_143022.db",
  "create_backup": true
}

Response:

{
  "message": "Database restored successfully",
  "restored_from": "snapshot_20250101_143022.db",
  "restored_at": "2025-01-01T15:00:00",
  "backup_created": "snapshot_20250101_150000.db"
}

Upload Snapshot

POST /api/settings/database/upload-snapshot
Content-Type: multipart/form-data

file: <binary data>

---

Best Practices

1. Regular Backups

• Enable automatic backups with a 24-hour interval
• Keep at least 7-10 backups for historical coverage
• Create manual snapshots before major changes

2. Before Major Operations

Always create a snapshot before:

• Software upgrades
• Bulk data imports
• Database schema changes
• Testing destructive operations

3. Testing Restores

Periodically test your restore process:

1. Download a snapshot
2. Test restoration on a dev environment
3. Verify data integrity

4. Off-Site Backups

For production systems:

• Download snapshots to external storage regularly
• Use the clone tool to sync to remote servers
• Store backups in multiple geographic locations

5. Snapshot Management

• Delete old snapshots when no longer needed
• Use descriptive names/descriptions for manual snapshots
• Keep pre-deployment snapshots separate

---

Troubleshooting

Snapshot Creation Fails

Problem: "Database is locked" error

Solution: The database is being written to. Wait a moment and try again. The SQLite backup API handles most locking automatically.

Restore Doesn't Complete

Problem: Restore appears to hang

Solution:

• Check server logs for errors
• Ensure sufficient disk space
• Verify the snapshot file isn't corrupted

Upload Fails on Dev Server

Problem: "Permission denied" or "File too large"

Solutions:

• Check file upload size limits in your web server config (nginx/apache)
• Verify write permissions on ./data/backups/ directory
• Ensure sufficient disk space

Automatic Backups Not Running

Problem: No automatic backups being created

Solutions:

1. Check if scheduler is enabled: scheduler.get_status()
2. Check application logs for scheduler errors
3. Ensure schedule library is installed: pip install schedule
4. Verify scheduler was started in application startup

---

Security Considerations

1. Access Control: Restrict access to the Settings → Danger Zone to administrators only
2. Backup Storage: Store backups in a secure location with proper permissions
3. Remote Cloning: Use authentication tokens when cloning to remote servers
4. Data Sensitivity: Remember that snapshots contain all database data - treat them with the same security as the live database

---

File Locations

• Database: ./data/seismo_fleet.db
• Backups Directory: ./data/backups/
• Clone Script: ./scripts/clone_db_to_dev.py
• Backup Service: ./backend/services/database_backup.py
• Scheduler Service: ./backend/services/backup_scheduler.py

---

Support

For issues or questions:

1. Check application logs in ./logs/
2. Review this documentation
3. Test with a small database first
4. Contact your system administrator