db management system added
This commit is contained in:
serversdwn
477
docs/DATABASE_MANAGEMENT.md
Normal file
477
docs/DATABASE_MANAGEMENT.md
Normal file
@@ -0,0 +1,477 @@
|
||||
# Database Management Guide
|
||||
|
||||
This guide covers the comprehensive database management features available in the Seismo Fleet Manager, including manual snapshots, restoration, remote cloning, and automatic backups.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Manual Database Snapshots](#manual-database-snapshots)
|
||||
2. [Restore from Snapshot](#restore-from-snapshot)
|
||||
3. [Download and Upload Snapshots](#download-and-upload-snapshots)
|
||||
4. [Clone Database to Dev Server](#clone-database-to-dev-server)
|
||||
5. [Automatic Backup Service](#automatic-backup-service)
|
||||
6. [API Reference](#api-reference)
|
||||
|
||||
---
|
||||
|
||||
## Manual Database Snapshots
|
||||
|
||||
### Creating a Snapshot via UI
|
||||
|
||||
1. Navigate to **Settings** → **Danger Zone** tab
|
||||
2. Scroll to the **Database Management** section
|
||||
3. Click **"Create Snapshot"**
|
||||
4. Optionally enter a description
|
||||
5. The snapshot will be created and appear in the "Available Snapshots" list
|
||||
|
||||
### Creating a Snapshot via API
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/settings/database/snapshot \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"description": "Pre-deployment backup"}'
|
||||
```
|
||||
|
||||
### What Happens
|
||||
|
||||
- A full copy of the SQLite database is created using the SQLite backup API
|
||||
- The snapshot is stored in `./data/backups/` directory
|
||||
- A metadata JSON file is created alongside the snapshot
|
||||
- No downtime or interruption to the running application
|
||||
|
||||
### Snapshot Files
|
||||
|
||||
Snapshots are stored as:
|
||||
- **Database file**: `snapshot_YYYYMMDD_HHMMSS.db`
|
||||
- **Metadata file**: `snapshot_YYYYMMDD_HHMMSS.db.meta.json`
|
||||
|
||||
Example:
|
||||
```
|
||||
data/backups/
|
||||
├── snapshot_20250101_143022.db
|
||||
├── snapshot_20250101_143022.db.meta.json
|
||||
├── snapshot_20250102_080000.db
|
||||
└── snapshot_20250102_080000.db.meta.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Restore from Snapshot
|
||||
|
||||
### Restoring via UI
|
||||
|
||||
1. Navigate to **Settings** → **Danger Zone** tab
|
||||
2. In the **Available Snapshots** section, find the snapshot you want to restore
|
||||
3. Click the **restore icon** (circular arrow) next to the snapshot
|
||||
4. Confirm the restoration warning
|
||||
5. A safety backup of the current database is automatically created
|
||||
6. The database is replaced with the snapshot
|
||||
7. The page reloads automatically
|
||||
|
||||
### Restoring via API
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/settings/database/restore \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"filename": "snapshot_20250101_143022.db",
|
||||
"create_backup": true
|
||||
}'
|
||||
```
|
||||
|
||||
### Important Notes
|
||||
|
||||
- **Always creates a safety backup** before restoring (unless explicitly disabled)
|
||||
- **Application reload required** - Users should refresh their browsers
|
||||
- **Atomic operation** - The entire database is replaced at once
|
||||
- **Cannot be undone** - But you'll have the safety backup
|
||||
|
||||
---
|
||||
|
||||
## Download and Upload Snapshots
|
||||
|
||||
### Download a Snapshot
|
||||
|
||||
**Via UI**: Click the download icon next to any snapshot in the list
|
||||
|
||||
**Via Browser**:
|
||||
```
|
||||
http://localhost:8000/api/settings/database/snapshot/snapshot_20250101_143022.db
|
||||
```
|
||||
|
||||
**Via Command Line**:
|
||||
```bash
|
||||
curl -o backup.db http://localhost:8000/api/settings/database/snapshot/snapshot_20250101_143022.db
|
||||
```
|
||||
|
||||
### Upload a Snapshot
|
||||
|
||||
**Via UI**:
|
||||
1. Navigate to **Settings** → **Danger Zone** tab
|
||||
2. Find the **Upload Snapshot** section
|
||||
3. Click **"Choose File"** and select a `.db` file
|
||||
4. Click **"Upload Snapshot"**
|
||||
|
||||
**Via Command Line**:
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/api/settings/database/upload-snapshot \
|
||||
-F "file=@/path/to/your/backup.db"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Clone Database to Dev Server
|
||||
|
||||
The clone tool allows you to copy the production database to a remote development server over the network.
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Remote dev server must have the same Seismo Fleet Manager installation
|
||||
- Network connectivity between production and dev servers
|
||||
- Python 3 and `requests` library installed
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# Clone current database to dev server
|
||||
python3 scripts/clone_db_to_dev.py --url https://dev.example.com
|
||||
|
||||
# Clone using existing snapshot
|
||||
python3 scripts/clone_db_to_dev.py \
|
||||
--url https://dev.example.com \
|
||||
--snapshot snapshot_20250101_143022.db
|
||||
|
||||
# Clone with authentication token
|
||||
python3 scripts/clone_db_to_dev.py \
|
||||
--url https://dev.example.com \
|
||||
--token YOUR_AUTH_TOKEN
|
||||
```
|
||||
|
||||
### What Happens
|
||||
|
||||
1. Creates a snapshot of the production database (or uses existing one)
|
||||
2. Uploads the snapshot to the remote dev server
|
||||
3. Automatically restores the snapshot on the dev server
|
||||
4. Creates a safety backup on the dev server before restoring
|
||||
|
||||
### Remote Server Setup
|
||||
|
||||
The remote dev server needs no special setup - it just needs to be running the same Seismo Fleet Manager application with the database management endpoints enabled.
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **Testing**: Test changes against production data in a dev environment
|
||||
- **Debugging**: Investigate production issues with real data safely
|
||||
- **Training**: Provide realistic data for user training
|
||||
- **Development**: Build new features with realistic data
|
||||
|
||||
---
|
||||
|
||||
## Automatic Backup Service
|
||||
|
||||
The automatic backup service runs scheduled backups in the background and manages backup retention.
|
||||
|
||||
### Configuration
|
||||
|
||||
The backup scheduler can be configured programmatically or via environment variables.
|
||||
|
||||
**Programmatic Configuration**:
|
||||
|
||||
```python
|
||||
from backend.services.backup_scheduler import get_backup_scheduler
|
||||
|
||||
scheduler = get_backup_scheduler()
|
||||
scheduler.configure(
|
||||
interval_hours=24, # Backup every 24 hours
|
||||
keep_count=10, # Keep last 10 backups
|
||||
enabled=True # Enable automatic backups
|
||||
)
|
||||
scheduler.start()
|
||||
```
|
||||
|
||||
**Environment Variables** (add to your `.env` or deployment config):
|
||||
|
||||
```bash
|
||||
AUTO_BACKUP_ENABLED=true
|
||||
AUTO_BACKUP_INTERVAL_HOURS=24
|
||||
AUTO_BACKUP_KEEP_COUNT=10
|
||||
```
|
||||
|
||||
### Integration with Application Startup
|
||||
|
||||
Add to `backend/main.py`:
|
||||
|
||||
```python
|
||||
from backend.services.backup_scheduler import get_backup_scheduler
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup_event():
|
||||
# Start automatic backup scheduler
|
||||
scheduler = get_backup_scheduler()
|
||||
scheduler.configure(
|
||||
interval_hours=24, # Daily backups
|
||||
keep_count=10, # Keep 10 most recent
|
||||
enabled=True
|
||||
)
|
||||
scheduler.start()
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def shutdown_event():
|
||||
# Stop backup scheduler gracefully
|
||||
scheduler = get_backup_scheduler()
|
||||
scheduler.stop()
|
||||
```
|
||||
|
||||
### Manual Control
|
||||
|
||||
```python
|
||||
from backend.services.backup_scheduler import get_backup_scheduler
|
||||
|
||||
scheduler = get_backup_scheduler()
|
||||
|
||||
# Get current status
|
||||
status = scheduler.get_status()
|
||||
print(status)
|
||||
# {'enabled': True, 'running': True, 'interval_hours': 24, 'keep_count': 10, 'next_run': '2025-01-02T14:00:00'}
|
||||
|
||||
# Create backup immediately
|
||||
scheduler.create_automatic_backup()
|
||||
|
||||
# Stop scheduler
|
||||
scheduler.stop()
|
||||
|
||||
# Start scheduler
|
||||
scheduler.start()
|
||||
```
|
||||
|
||||
### Backup Retention
|
||||
|
||||
The scheduler automatically deletes old backups based on the `keep_count` setting. For example, if `keep_count=10`, only the 10 most recent backups are kept, and older ones are automatically deleted.
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Database Statistics
|
||||
|
||||
```http
|
||||
GET /api/settings/database/stats
|
||||
```
|
||||
|
||||
Returns database size, row counts, and last modified time.
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"database_path": "./data/seismo_fleet.db",
|
||||
"size_bytes": 1048576,
|
||||
"size_mb": 1.0,
|
||||
"total_rows": 1250,
|
||||
"tables": {
|
||||
"roster": 450,
|
||||
"emitters": 600,
|
||||
"ignored_units": 50,
|
||||
"unit_history": 150
|
||||
},
|
||||
"last_modified": "2025-01-01T14:30:22"
|
||||
}
|
||||
```
|
||||
|
||||
### Create Snapshot
|
||||
|
||||
```http
|
||||
POST /api/settings/database/snapshot
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"description": "Optional description"
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"message": "Snapshot created successfully",
|
||||
"snapshot": {
|
||||
"filename": "snapshot_20250101_143022.db",
|
||||
"created_at": "20250101_143022",
|
||||
"created_at_iso": "2025-01-01T14:30:22",
|
||||
"description": "Optional description",
|
||||
"size_bytes": 1048576,
|
||||
"size_mb": 1.0,
|
||||
"type": "manual"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### List Snapshots
|
||||
|
||||
```http
|
||||
GET /api/settings/database/snapshots
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"snapshots": [
|
||||
{
|
||||
"filename": "snapshot_20250101_143022.db",
|
||||
"created_at": "20250101_143022",
|
||||
"created_at_iso": "2025-01-01T14:30:22",
|
||||
"description": "Manual backup",
|
||||
"size_mb": 1.0,
|
||||
"type": "manual"
|
||||
}
|
||||
],
|
||||
"count": 1
|
||||
}
|
||||
```
|
||||
|
||||
### Download Snapshot
|
||||
|
||||
```http
|
||||
GET /api/settings/database/snapshot/{filename}
|
||||
```
|
||||
|
||||
Returns the snapshot file as a download.
|
||||
|
||||
### Delete Snapshot
|
||||
|
||||
```http
|
||||
DELETE /api/settings/database/snapshot/{filename}
|
||||
```
|
||||
|
||||
### Restore Database
|
||||
|
||||
```http
|
||||
POST /api/settings/database/restore
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"filename": "snapshot_20250101_143022.db",
|
||||
"create_backup": true
|
||||
}
|
||||
```
|
||||
|
||||
**Response**:
|
||||
```json
|
||||
{
|
||||
"message": "Database restored successfully",
|
||||
"restored_from": "snapshot_20250101_143022.db",
|
||||
"restored_at": "2025-01-01T15:00:00",
|
||||
"backup_created": "snapshot_20250101_150000.db"
|
||||
}
|
||||
```
|
||||
|
||||
### Upload Snapshot
|
||||
|
||||
```http
|
||||
POST /api/settings/database/upload-snapshot
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
file: <binary data>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Regular Backups
|
||||
|
||||
- **Enable automatic backups** with a 24-hour interval
|
||||
- **Keep at least 7-10 backups** for historical coverage
|
||||
- **Create manual snapshots** before major changes
|
||||
|
||||
### 2. Before Major Operations
|
||||
|
||||
Always create a snapshot before:
|
||||
- Software upgrades
|
||||
- Bulk data imports
|
||||
- Database schema changes
|
||||
- Testing destructive operations
|
||||
|
||||
### 3. Testing Restores
|
||||
|
||||
Periodically test your restore process:
|
||||
1. Download a snapshot
|
||||
2. Test restoration on a dev environment
|
||||
3. Verify data integrity
|
||||
|
||||
### 4. Off-Site Backups
|
||||
|
||||
For production systems:
|
||||
- **Download snapshots** to external storage regularly
|
||||
- Use the clone tool to **sync to remote servers**
|
||||
- Store backups in **multiple geographic locations**
|
||||
|
||||
### 5. Snapshot Management
|
||||
|
||||
- Delete old snapshots when no longer needed
|
||||
- Use descriptive names/descriptions for manual snapshots
|
||||
- Keep pre-deployment snapshots separate
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Snapshot Creation Fails
|
||||
|
||||
**Problem**: "Database is locked" error
|
||||
|
||||
**Solution**: The database is being written to. Wait a moment and try again. The SQLite backup API handles most locking automatically.
|
||||
|
||||
### Restore Doesn't Complete
|
||||
|
||||
**Problem**: Restore appears to hang
|
||||
|
||||
**Solution**:
|
||||
- Check server logs for errors
|
||||
- Ensure sufficient disk space
|
||||
- Verify the snapshot file isn't corrupted
|
||||
|
||||
### Upload Fails on Dev Server
|
||||
|
||||
**Problem**: "Permission denied" or "File too large"
|
||||
|
||||
**Solutions**:
|
||||
- Check file upload size limits in your web server config (nginx/apache)
|
||||
- Verify write permissions on `./data/backups/` directory
|
||||
- Ensure sufficient disk space
|
||||
|
||||
### Automatic Backups Not Running
|
||||
|
||||
**Problem**: No automatic backups being created
|
||||
|
||||
**Solutions**:
|
||||
1. Check if scheduler is enabled: `scheduler.get_status()`
|
||||
2. Check application logs for scheduler errors
|
||||
3. Ensure `schedule` library is installed: `pip install schedule`
|
||||
4. Verify scheduler was started in application startup
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Access Control**: Restrict access to the Settings → Danger Zone to administrators only
|
||||
2. **Backup Storage**: Store backups in a secure location with proper permissions
|
||||
3. **Remote Cloning**: Use authentication tokens when cloning to remote servers
|
||||
4. **Data Sensitivity**: Remember that snapshots contain all database data - treat them with the same security as the live database
|
||||
|
||||
---
|
||||
|
||||
## File Locations
|
||||
|
||||
- **Database**: `./data/seismo_fleet.db`
|
||||
- **Backups Directory**: `./data/backups/`
|
||||
- **Clone Script**: `./scripts/clone_db_to_dev.py`
|
||||
- **Backup Service**: `./backend/services/database_backup.py`
|
||||
- **Scheduler Service**: `./backend/services/backup_scheduler.py`
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
1. Check application logs in `./logs/`
|
||||
2. Review this documentation
|
||||
3. Test with a small database first
|
||||
4. Contact your system administrator
|
||||
Reference in New Issue
Block a user