Move SLM control center groundwork onto dev

2026-01-12 18:07:26 +00:00
parent 893cb96e8d
commit 8a5fadb5df
22 changed files with 3960 additions and 29 deletions
--- a/PROJECTS_SYSTEM_IMPLEMENTATION.md
+++ b/PROJECTS_SYSTEM_IMPLEMENTATION.md
@@ -0,0 +1,546 @@
+# Projects System Implementation - Terra-View
+
+## Overview
+
+The Projects system has been successfully scaffolded in Terra-View. This document provides a complete overview of what has been built, how it works, and what needs to be completed.
+
+## ✅ Completed Components
+
+### 1. Database Schema
+
+**Location**: `/backend/models.py`
+
+Seven new tables have been added:
+
+- **ProjectType**: Template definitions for project types (Sound, Vibration, Combined)
+- **Project**: Top-level project organization with type reference
+- **MonitoringLocation**: Generic locations (NRLs for sound, monitoring points for vibration)
+- **UnitAssignment**: Links devices to locations
+- **ScheduledAction**: Automated recording control schedules
+- **RecordingSession**: Tracks actual recording/monitoring sessions
+- **DataFile**: File references for downloaded data
+
+**Key Features**:
+- Type-aware design (project_type_id determines features)
+- Flexible metadata fields (JSON columns for type-specific data)
+- Denormalized fields for efficient queries
+- Proper indexing on foreign keys
+
+### 2. Service Layer
+
+#### SLMM Client (`/backend/services/slmm_client.py`)
+- Clean wrapper for all SLMM API operations
+- Methods for: start/stop/pause/resume recording, get status, configure devices
+- Error handling with custom exceptions
+- Singleton pattern for easy access
+
+#### Device Controller (`/backend/services/device_controller.py`)
+- Routes commands to appropriate backend (SLMM for SLMs, SFM for seismographs)
+- Unified interface across device types
+- Ready for future SFM implementation
+
+#### Scheduler Service (`/backend/services/scheduler.py`)
+- Background task that checks for pending scheduled actions every 60 seconds
+- Executes actions by calling device controller
+- Creates/updates recording sessions
+- Tracks execution status and errors
+- Manual execution support for testing
+
+### 3. API Routers
+
+#### Projects Router (`/backend/routers/projects.py`)
+Endpoints:
+- `GET /api/projects/list` - Project list with stats
+- `GET /api/projects/stats` - Overview statistics
+- `POST /api/projects/create` - Create new project
+- `GET /api/projects/{id}` - Get project details
+- `PUT /api/projects/{id}` - Update project
+- `DELETE /api/projects/{id}` - Archive project
+- `GET /api/projects/{id}/dashboard` - Project dashboard data
+- `GET /api/projects/types/list` - Get project type templates
+
+#### Project Locations Router (`/backend/routers/project_locations.py`)
+Endpoints:
+- `GET /api/projects/{id}/locations` - List locations
+- `POST /api/projects/{id}/locations/create` - Create location
+- `PUT /api/projects/{id}/locations/{location_id}` - Update location
+- `DELETE /api/projects/{id}/locations/{location_id}` - Delete location
+- `GET /api/projects/{id}/assignments` - List unit assignments
+- `POST /api/projects/{id}/locations/{location_id}/assign` - Assign unit
+- `POST /api/projects/{id}/assignments/{assignment_id}/unassign` - Unassign unit
+- `GET /api/projects/{id}/available-units` - Get units available for assignment
+
+#### Scheduler Router (`/backend/routers/scheduler.py`)
+Endpoints:
+- `GET /api/projects/{id}/scheduler/actions` - List scheduled actions
+- `POST /api/projects/{id}/scheduler/actions/create` - Create action
+- `POST /api/projects/{id}/scheduler/schedule-session` - Schedule recording session
+- `PUT /api/projects/{id}/scheduler/actions/{action_id}` - Update action
+- `POST /api/projects/{id}/scheduler/actions/{action_id}/cancel` - Cancel action
+- `DELETE /api/projects/{id}/scheduler/actions/{action_id}` - Delete action
+- `POST /api/projects/{id}/scheduler/actions/{action_id}/execute` - Manual execution
+- `GET /api/projects/{id}/scheduler/status` - Scheduler status
+- `POST /api/projects/{id}/scheduler/execute-pending` - Trigger pending executions
+
+### 4. Frontend
+
+#### Main Page
+**Location**: `/templates/projects/overview.html`
+
+Features:
+- Summary statistics cards (projects, locations, assignments, sessions)
+- Tabbed interface (All, Active, Completed, Archived)
+- Project cards grid layout
+- Create project modal with two-step flow:
+  1. Select project type (Sound/Vibration/Combined)
+  2. Fill project details
+- HTMX-powered dynamic updates
+
+#### Navigation
+**Location**: `/templates/base.html` (updated)
+- "Projects" link added to sidebar
+- Active state highlighting
+
+### 5. Application Integration
+
+**Location**: `/backend/main.py`
+
+- Routers registered
+- Page route added (`/projects`)
+- Scheduler service starts on application startup
+- Scheduler stops on application shutdown
+
+### 6. Database Initialization
+
+**Script**: `/backend/init_projects_db.py`
+
+- Creates all project tables
+- Populates ProjectType with default templates
+- ✅ Successfully executed - database is ready
+
+---
+
+## 📁 File Organization
+
+```
+terra-view/
+├── backend/
+│   ├── models.py                              [✅ Updated]
+│   ├── init_projects_db.py                    [✅ Created]
+│   ├── main.py                                [✅ Updated]
+│   ├── routers/
+│   │   ├── projects.py                        [✅ Created]
+│   │   ├── project_locations.py               [✅ Created]
+│   │   └── scheduler.py                       [✅ Created]
+│   └── services/
+│       ├── slmm_client.py                     [✅ Created]
+│       ├── device_controller.py               [✅ Created]
+│       └── scheduler.py                       [✅ Created]
+├── templates/
+│   ├── base.html                              [✅ Updated]
+│   ├── projects/
+│   │   └── overview.html                      [✅ Created]
+│   └── partials/
+│       └── projects/                          [📁 Created, empty]
+└── data/
+    └── seismo_fleet.db                        [✅ Tables created]
+```
+
+---
+
+## 🔨 What Still Needs to be Built
+
+### 1. Frontend Templates (Partials)
+
+**Directory**: `/templates/partials/projects/`
+
+**Required Files**:
+
+#### `project_stats.html`
+Stats cards for overview page:
+- Total/Active/Completed projects
+- Total locations
+- Assigned units
+- Active sessions
+
+#### `project_list.html`
+Project cards grid:
+- Project name, type, status
+- Location count, unit count
+- Active session indicator
+- Link to project dashboard
+
+#### `project_dashboard.html`
+Main project dashboard panel with tabs:
+- Summary stats
+- Active locations and assignments
+- Upcoming scheduled actions
+- Recent sessions
+
+#### `location_list.html`
+Location cards/table:
+- Location name, type, coordinates
+- Assigned unit (if any)
+- Session count
+- Assign/unassign button
+
+#### `assignment_list.html`
+Unit assignment table:
+- Unit ID, device type
+- Location name
+- Assignment dates
+- Status
+- Unassign button
+
+#### `scheduler_agenda.html`
+Calendar/agenda view:
+- Scheduled actions sorted by time
+- Action type (start/stop/download)
+- Location and unit
+- Status indicator
+- Cancel/execute buttons
+
+### 2. Project Dashboard Page
+
+**Location**: `/templates/projects/project_dashboard.html`
+
+Full project detail page with:
+- Header with project name, type, status
+- Tab navigation (Dashboard, Scheduler, Locations, Units, Data, Settings)
+- Tab content areas
+- Modals for adding locations, scheduling sessions
+
+### 3. Additional UI Components
+
+- Project type selection cards (with icons)
+- Location creation modal
+- Unit assignment modal
+- Schedule session modal (with date/time picker)
+- Data file browser
+
+### 4. SLMM Enhancements
+
+**Location**: `/slmm/app/routers.py` (SLMM repo)
+
+New endpoint needed:
+```python
+POST /api/nl43/{unit_id}/ftp/download
+```
+
+This should:
+- Accept destination_path and files list
+- Connect to SLM via FTP
+- Download specified files
+- Save to Terra-View's `data/Projects/` directory
+- Return file list with metadata
+
+### 5. SFM Client (Future)
+
+**Location**: `/backend/services/sfm_client.py` (to be created)
+
+Similar to SLMM client, but for seismographs:
+- Get seismograph status
+- Start/stop recording
+- Download data files
+- Integrate with device controller
+
+---
+
+## 🚀 Testing the System
+
+### 1. Start Terra-View
+
+```bash
+cd /home/serversdown/tmi/terra-view
+# Start Terra-View (however you normally start it)
+```
+
+Verify in logs:
+```
+Starting scheduler service...
+Scheduler service started
+```
+
+### 2. Navigate to Projects
+
+Open browser: `http://localhost:8001/projects`
+
+You should see:
+- Summary stats cards (all zeros initially)
+- Tabs (All Projects, Active, Completed, Archived)
+- "New Project" button
+
+### 3. Create a Project
+
+1. Click "New Project"
+2. Select a project type (e.g., "Sound Monitoring")
+3. Fill in details:
+   - Name: "Test Sound Project"
+   - Client: "Test Client"
+   - Start Date: Today
+4. Submit
+
+### 4. Test API Endpoints
+
+```bash
+# Get project types
+curl http://localhost:8001/api/projects/types/list
+
+# Get projects list
+curl http://localhost:8001/api/projects/list
+
+# Get project stats
+curl http://localhost:8001/api/projects/stats
+```
+
+### 5. Test Scheduler Status
+
+```bash
+curl http://localhost:8001/api/projects/{project_id}/scheduler/status
+```
+
+---
+
+## 📋 Dataflow Examples
+
+### Creating and Scheduling a Recording Session
+
+1. **User creates project** → Project record in DB
+2. **User adds NRL** → MonitoringLocation record
+3. **User assigns SLM to NRL** → UnitAssignment record
+4. **User schedules recording** → 2 ScheduledAction records (start + stop)
+5. **Scheduler runs every minute** → Checks for pending actions
+6. **Start action time arrives** → Scheduler calls SLMM via device controller
+7. **SLMM sends TCP command to SLM** → Recording starts
+8. **RecordingSession created** → Tracks the session
+9. **Stop action time arrives** → Scheduler stops recording
+10. **Session updated** → stopped_at, duration_seconds filled
+11. **User triggers download** → Files copied to `data/Projects/{project_id}/sound/{nrl_name}/`
+12. **DataFile records created** → Track file references
+
+---
+
+## 🎨 UI Design Patterns
+
+### Established Patterns (from SLM dashboard):
+
+1. **Stats Cards**: 4-column grid, auto-refresh every 30s
+2. **Sidebar Lists**: Searchable, filterable, auto-refresh
+3. **Main Panel**: Large central area for details
+4. **Modals**: Centered, overlay background
+5. **HTMX**: All dynamic updates, minimal JavaScript
+6. **Tailwind**: Consistent styling with dark mode support
+
+### Color Scheme:
+
+- Primary: `seismo-orange` (#f48b1c)
+- Secondary: `seismo-navy` (#142a66)
+- Accent: `seismo-burgundy` (#7d234d)
+
+---
+
+## 🔧 Configuration
+
+### Environment Variables
+
+- `SLMM_BASE_URL`: SLMM backend URL (default: http://localhost:8100)
+- `ENVIRONMENT`: "development" or "production"
+
+### Scheduler Settings
+
+Located in `/backend/services/scheduler.py`:
+- `check_interval`: 60 seconds (adjust as needed)
+
+---
+
+## 📚 Next Steps
+
+### Immediate (Get Basic UI Working):
+1. Create partial templates (stats, lists)
+2. Test creating projects via UI
+3. Implement project dashboard page
+
+### Short-term (Core Features):
+4. Add location management UI
+5. Add unit assignment UI
+6. Add scheduler UI (agenda view)
+
+### Medium-term (Data Flow):
+7. Implement SLMM download endpoint
+8. Test full recording workflow
+9. Add file browser for downloaded data
+
+### Long-term (Complete System):
+10. Implement SFM client for seismographs
+11. Add data visualization
+12. Add project reporting
+13. Add user authentication
+
+---
+
+## 🐛 Known Issues / TODOs
+
+1. **Partial templates missing**: Need to create HTML templates for all partials
+2. **SLMM download endpoint**: Needs implementation in SLMM backend
+3. **Project dashboard page**: Not yet created
+4. **SFM integration**: Placeholder only, needs real implementation
+5. **File download tracking**: DataFile records not yet created after downloads
+6. **Error handling**: Need better user-facing error messages
+7. **Validation**: Form validation could be improved
+8. **Testing**: No automated tests yet
+
+---
+
+## 📖 API Documentation
+
+### Project Type Object
+```json
+{
+  "id": "sound_monitoring",
+  "name": "Sound Monitoring",
+  "description": "...",
+  "icon": "volume-2",
+  "supports_sound": true,
+  "supports_vibration": false
+}
+```
+
+### Project Object
+```json
+{
+  "id": "uuid",
+  "name": "Project Name",
+  "description": "...",
+  "project_type_id": "sound_monitoring",
+  "status": "active",
+  "client_name": "Client Inc",
+  "site_address": "123 Main St",
+  "site_coordinates": "40.7128,-74.0060",
+  "start_date": "2024-01-15",
+  "end_date": null,
+  "created_at": "2024-01-15T10:00:00",
+  "updated_at": "2024-01-15T10:00:00"
+}
+```
+
+### MonitoringLocation Object
+```json
+{
+  "id": "uuid",
+  "project_id": "uuid",
+  "location_type": "sound",
+  "name": "NRL-001",
+  "description": "...",
+  "coordinates": "40.7128,-74.0060",
+  "address": "123 Main St",
+  "location_metadata": "{...}",
+  "created_at": "2024-01-15T10:00:00"
+}
+```
+
+### UnitAssignment Object
+```json
+{
+  "id": "uuid",
+  "unit_id": "nl43-001",
+  "location_id": "uuid",
+  "project_id": "uuid",
+  "device_type": "sound_level_meter",
+  "assigned_at": "2024-01-15T10:00:00",
+  "assigned_until": null,
+  "status": "active",
+  "notes": "..."
+}
+```
+
+### ScheduledAction Object
+```json
+{
+  "id": "uuid",
+  "project_id": "uuid",
+  "location_id": "uuid",
+  "unit_id": "nl43-001",
+  "action_type": "start",
+  "device_type": "sound_level_meter",
+  "scheduled_time": "2024-01-16T08:00:00",
+  "executed_at": null,
+  "execution_status": "pending",
+  "module_response": null,
+  "error_message": null
+}
+```
+
+---
+
+## 🎓 Architecture Decisions
+
+### Why Project Types?
+Allows the system to scale to different monitoring scenarios (air quality, multi-hazard, etc.) without code changes. Just add a new ProjectType record and the UI adapts.
+
+### Why Generic MonitoringLocation?
+Instead of separate NRL and MonitoringPoint tables, one table with a `location_type` discriminator keeps the schema clean and allows for combined projects.
+
+### Why Denormalized Fields?
+Fields like `project_id` in UnitAssignment (already have via location) enable faster queries without joins.
+
+### Why Scheduler in Terra-View?
+Terra-View is the orchestration layer. SLMM only handles device communication. Keeping scheduling logic in Terra-View allows for complex workflows across multiple device types.
+
+### Why JSON Metadata Columns?
+Type-specific fields (like ambient_conditions for sound projects) don't apply to all location types. JSON columns provide flexibility without cluttering the schema.
+
+---
+
+## 💡 Tips for Continuing Development
+
+1. **Follow Existing Patterns**: Look at the SLM dashboard code for reference
+2. **Use HTMX Aggressively**: Minimize JavaScript, let HTMX handle updates
+3. **Keep Routers Thin**: Move business logic to service layer
+4. **Return HTML Partials**: Most endpoints should return HTML, not JSON
+5. **Test Incrementally**: Build one partial at a time and test in browser
+6. **Check Logs**: Scheduler logs execution attempts
+7. **Use Browser DevTools**: Network tab shows HTMX requests
+
+---
+
+## 📞 Support
+
+For questions or issues:
+1. Check this document first
+2. Review existing dashboards (SLM, Seismographs) for patterns
+3. Check logs for scheduler execution details
+4. Test API endpoints with curl to isolate issues
+
+---
+
+## ✅ Checklist for Completion
+
+- [x] Database schema designed
+- [x] Models created
+- [x] Migration script run successfully
+- [x] Service layer complete (SLMM client, device controller, scheduler)
+- [x] API routers created (projects, locations, scheduler)
+- [x] Navigation updated
+- [x] Main overview page created
+- [x] Routes registered in main.py
+- [x] Scheduler service integrated
+- [ ] Partial templates created
+- [ ] Project dashboard page created
+- [ ] Location management UI
+- [ ] Unit assignment UI
+- [ ] Scheduler UI (agenda view)
+- [ ] SLMM download endpoint implemented
+- [ ] Full workflow tested end-to-end
+- [ ] SFM client implemented (future)
+
+---
+
+**Last Updated**: 2026-01-12
+
+**Database Status**: ✅ Initialized
+
+**Backend Status**: ✅ Complete
+
+**Frontend Status**: 🟡 Partial (overview page only)
+
+**Ready for Testing**: ✅ Yes (basic functionality)