terra-view/PROJECTS_SYSTEM_IMPLEMENTATION.md

# Projects System Implementation - Terra-View

## Overview

The Projects system has been successfully scaffolded in Terra-View. This document provides a complete overview of what has been built, how it works, and what needs to be completed.

## ✅ Completed Components

### 1. Database Schema

**Location**: `/backend/models.py`

Seven new tables have been added:

- **ProjectType**: Template definitions for project types (Sound, Vibration, Combined)
- **Project**: Top-level project organization with type reference
- **MonitoringLocation**: Generic locations (NRLs for sound, monitoring points for vibration)
- **UnitAssignment**: Links devices to locations
- **ScheduledAction**: Automated recording control schedules
- **RecordingSession**: Tracks actual recording/monitoring sessions
- **DataFile**: File references for downloaded data

**Key Features**:
- Type-aware design (project_type_id determines features)
- Flexible metadata fields (JSON columns for type-specific data)
- Denormalized fields for efficient queries
- Proper indexing on foreign keys

### 2. Service Layer

#### SLMM Client (`/backend/services/slmm_client.py`)
- Clean wrapper for all SLMM API operations
- Methods for: start/stop/pause/resume recording, get status, configure devices
- Error handling with custom exceptions
- Singleton pattern for easy access

#### Device Controller (`/backend/services/device_controller.py`)
- Routes commands to appropriate backend (SLMM for SLMs, SFM for seismographs)
- Unified interface across device types
- Ready for future SFM implementation

#### Scheduler Service (`/backend/services/scheduler.py`)
- Background task that checks for pending scheduled actions every 60 seconds
- Executes actions by calling device controller
- Creates/updates recording sessions
- Tracks execution status and errors
- Manual execution support for testing

### 3. API Routers

#### Projects Router (`/backend/routers/projects.py`)
Endpoints:
- `GET /api/projects/list` - Project list with stats
- `GET /api/projects/stats` - Overview statistics
- `POST /api/projects/create` - Create new project
- `GET /api/projects/{id}` - Get project details
- `PUT /api/projects/{id}` - Update project
- `DELETE /api/projects/{id}` - Archive project
- `GET /api/projects/{id}/dashboard` - Project dashboard data
- `GET /api/projects/types/list` - Get project type templates

#### Project Locations Router (`/backend/routers/project_locations.py`)
Endpoints:
- `GET /api/projects/{id}/locations` - List locations
- `POST /api/projects/{id}/locations/create` - Create location
- `PUT /api/projects/{id}/locations/{location_id}` - Update location
- `DELETE /api/projects/{id}/locations/{location_id}` - Delete location
- `GET /api/projects/{id}/assignments` - List unit assignments
- `POST /api/projects/{id}/locations/{location_id}/assign` - Assign unit
- `POST /api/projects/{id}/assignments/{assignment_id}/unassign` - Unassign unit
- `GET /api/projects/{id}/available-units` - Get units available for assignment

#### Scheduler Router (`/backend/routers/scheduler.py`)
Endpoints:
- `GET /api/projects/{id}/scheduler/actions` - List scheduled actions
- `POST /api/projects/{id}/scheduler/actions/create` - Create action
- `POST /api/projects/{id}/scheduler/schedule-session` - Schedule recording session
- `PUT /api/projects/{id}/scheduler/actions/{action_id}` - Update action
- `POST /api/projects/{id}/scheduler/actions/{action_id}/cancel` - Cancel action
- `DELETE /api/projects/{id}/scheduler/actions/{action_id}` - Delete action
- `POST /api/projects/{id}/scheduler/actions/{action_id}/execute` - Manual execution
- `GET /api/projects/{id}/scheduler/status` - Scheduler status
- `POST /api/projects/{id}/scheduler/execute-pending` - Trigger pending executions

### 4. Frontend

#### Main Page
**Location**: `/templates/projects/overview.html`

Features:
- Summary statistics cards (projects, locations, assignments, sessions)
- Tabbed interface (All, Active, Completed, Archived)
- Project cards grid layout
- Create project modal with two-step flow:
  1. Select project type (Sound/Vibration/Combined)
  2. Fill project details
- HTMX-powered dynamic updates

#### Navigation
**Location**: `/templates/base.html` (updated)
- "Projects" link added to sidebar
- Active state highlighting

### 5. Application Integration

**Location**: `/backend/main.py`

- Routers registered
- Page route added (`/projects`)
- Scheduler service starts on application startup
- Scheduler stops on application shutdown

### 6. Database Initialization

**Script**: `/backend/init_projects_db.py`

- Creates all project tables
- Populates ProjectType with default templates
- ✅ Successfully executed - database is ready

---

## 📁 File Organization

```
terra-view/
├── backend/
│   ├── models.py                              [✅ Updated]
│   ├── init_projects_db.py                    [✅ Created]
│   ├── main.py                                [✅ Updated]
│   ├── routers/
│   │   ├── projects.py                        [✅ Created]
│   │   ├── project_locations.py               [✅ Created]
│   │   └── scheduler.py                       [✅ Created]
│   └── services/
│       ├── slmm_client.py                     [✅ Created]
│       ├── device_controller.py               [✅ Created]
│       └── scheduler.py                       [✅ Created]
├── templates/
│   ├── base.html                              [✅ Updated]
│   ├── projects/
│   │   └── overview.html                      [✅ Created]
│   └── partials/
│       └── projects/                          [📁 Created, empty]
└── data/
    └── seismo_fleet.db                        [✅ Tables created]
```

---

## 🔨 What Still Needs to be Built

### 1. Frontend Templates (Partials)

**Directory**: `/templates/partials/projects/`

**Required Files**:

#### `project_stats.html`
Stats cards for overview page:
- Total/Active/Completed projects
- Total locations
- Assigned units
- Active sessions

#### `project_list.html`
Project cards grid:
- Project name, type, status
- Location count, unit count
- Active session indicator
- Link to project dashboard

#### `project_dashboard.html`
Main project dashboard panel with tabs:
- Summary stats
- Active locations and assignments
- Upcoming scheduled actions
- Recent sessions

#### `location_list.html`
Location cards/table:
- Location name, type, coordinates
- Assigned unit (if any)
- Session count
- Assign/unassign button

#### `assignment_list.html`
Unit assignment table:
- Unit ID, device type
- Location name
- Assignment dates
- Status
- Unassign button

#### `scheduler_agenda.html`
Calendar/agenda view:
- Scheduled actions sorted by time
- Action type (start/stop/download)
- Location and unit
- Status indicator
- Cancel/execute buttons

### 2. Project Dashboard Page

**Location**: `/templates/projects/project_dashboard.html`

Full project detail page with:
- Header with project name, type, status
- Tab navigation (Dashboard, Scheduler, Locations, Units, Data, Settings)
- Tab content areas
- Modals for adding locations, scheduling sessions

### 3. Additional UI Components

- Project type selection cards (with icons)
- Location creation modal
- Unit assignment modal
- Schedule session modal (with date/time picker)
- Data file browser

### 4. SLMM Enhancements

**Location**: `/slmm/app/routers.py` (SLMM repo)

New endpoint needed:
```python
POST /api/nl43/{unit_id}/ftp/download
```

This should:
- Accept destination_path and files list
- Connect to SLM via FTP
- Download specified files
- Save to Terra-View's `data/Projects/` directory
- Return file list with metadata

### 5. SFM Client (Future)

**Location**: `/backend/services/sfm_client.py` (to be created)

Similar to SLMM client, but for seismographs:
- Get seismograph status
- Start/stop recording
- Download data files
- Integrate with device controller

---

## 🚀 Testing the System

### 1. Start Terra-View

```bash
cd /home/serversdown/tmi/terra-view
# Start Terra-View (however you normally start it)
```

Verify in logs:
```
Starting scheduler service...
Scheduler service started
```

### 2. Navigate to Projects

Open browser: `http://localhost:8001/projects`

You should see:
- Summary stats cards (all zeros initially)
- Tabs (All Projects, Active, Completed, Archived)
- "New Project" button

### 3. Create a Project

1. Click "New Project"
2. Select a project type (e.g., "Sound Monitoring")
3. Fill in details:
   - Name: "Test Sound Project"
   - Client: "Test Client"
   - Start Date: Today
4. Submit

### 4. Test API Endpoints

```bash
# Get project types
curl http://localhost:8001/api/projects/types/list

# Get projects list
curl http://localhost:8001/api/projects/list

# Get project stats
curl http://localhost:8001/api/projects/stats
```

### 5. Test Scheduler Status

```bash
curl http://localhost:8001/api/projects/{project_id}/scheduler/status
```

---

## 📋 Dataflow Examples

### Creating and Scheduling a Recording Session

1. **User creates project** → Project record in DB
2. **User adds NRL** → MonitoringLocation record
3. **User assigns SLM to NRL** → UnitAssignment record
4. **User schedules recording** → 2 ScheduledAction records (start + stop)
5. **Scheduler runs every minute** → Checks for pending actions
6. **Start action time arrives** → Scheduler calls SLMM via device controller
7. **SLMM sends TCP command to SLM** → Recording starts
8. **RecordingSession created** → Tracks the session
9. **Stop action time arrives** → Scheduler stops recording
10. **Session updated** → stopped_at, duration_seconds filled
11. **User triggers download** → Files copied to `data/Projects/{project_id}/sound/{nrl_name}/`
12. **DataFile records created** → Track file references

---

## 🎨 UI Design Patterns

### Established Patterns (from SLM dashboard):

1. **Stats Cards**: 4-column grid, auto-refresh every 30s
2. **Sidebar Lists**: Searchable, filterable, auto-refresh
3. **Main Panel**: Large central area for details
4. **Modals**: Centered, overlay background
5. **HTMX**: All dynamic updates, minimal JavaScript
6. **Tailwind**: Consistent styling with dark mode support

### Color Scheme:

- Primary: `seismo-orange` (#f48b1c)
- Secondary: `seismo-navy` (#142a66)
- Accent: `seismo-burgundy` (#7d234d)

---

## 🔧 Configuration

### Environment Variables

- `SLMM_BASE_URL`: SLMM backend URL (default: http://localhost:8100)
- `ENVIRONMENT`: "development" or "production"

### Scheduler Settings

Located in `/backend/services/scheduler.py`:
- `check_interval`: 60 seconds (adjust as needed)

---

## 📚 Next Steps

### Immediate (Get Basic UI Working):
1. Create partial templates (stats, lists)
2. Test creating projects via UI
3. Implement project dashboard page

### Short-term (Core Features):
4. Add location management UI
5. Add unit assignment UI
6. Add scheduler UI (agenda view)

### Medium-term (Data Flow):
7. Implement SLMM download endpoint
8. Test full recording workflow
9. Add file browser for downloaded data

### Long-term (Complete System):
10. Implement SFM client for seismographs
11. Add data visualization
12. Add project reporting
13. Add user authentication

---

## 🐛 Known Issues / TODOs

1. **Partial templates missing**: Need to create HTML templates for all partials
2. **SLMM download endpoint**: Needs implementation in SLMM backend
3. **Project dashboard page**: Not yet created
4. **SFM integration**: Placeholder only, needs real implementation
5. **File download tracking**: DataFile records not yet created after downloads
6. **Error handling**: Need better user-facing error messages
7. **Validation**: Form validation could be improved
8. **Testing**: No automated tests yet

---

## 📖 API Documentation

### Project Type Object
```json
{
  "id": "sound_monitoring",
  "name": "Sound Monitoring",
  "description": "...",
  "icon": "volume-2",
  "supports_sound": true,
  "supports_vibration": false
}
```

### Project Object
```json
{
  "id": "uuid",
  "name": "Project Name",
  "description": "...",
  "project_type_id": "sound_monitoring",
  "status": "active",
  "client_name": "Client Inc",
  "site_address": "123 Main St",
  "site_coordinates": "40.7128,-74.0060",
  "start_date": "2024-01-15",
  "end_date": null,
  "created_at": "2024-01-15T10:00:00",
  "updated_at": "2024-01-15T10:00:00"
}
```

### MonitoringLocation Object
```json
{
  "id": "uuid",
  "project_id": "uuid",
  "location_type": "sound",
  "name": "NRL-001",
  "description": "...",
  "coordinates": "40.7128,-74.0060",
  "address": "123 Main St",
  "location_metadata": "{...}",
  "created_at": "2024-01-15T10:00:00"
}
```

### UnitAssignment Object
```json
{
  "id": "uuid",
  "unit_id": "nl43-001",
  "location_id": "uuid",
  "project_id": "uuid",
  "device_type": "sound_level_meter",
  "assigned_at": "2024-01-15T10:00:00",
  "assigned_until": null,
  "status": "active",
  "notes": "..."
}
```

### ScheduledAction Object
```json
{
  "id": "uuid",
  "project_id": "uuid",
  "location_id": "uuid",
  "unit_id": "nl43-001",
  "action_type": "start",
  "device_type": "sound_level_meter",
  "scheduled_time": "2024-01-16T08:00:00",
  "executed_at": null,
  "execution_status": "pending",
  "module_response": null,
  "error_message": null
}
```

---

## 🎓 Architecture Decisions

### Why Project Types?
Allows the system to scale to different monitoring scenarios (air quality, multi-hazard, etc.) without code changes. Just add a new ProjectType record and the UI adapts.

### Why Generic MonitoringLocation?
Instead of separate NRL and MonitoringPoint tables, one table with a `location_type` discriminator keeps the schema clean and allows for combined projects.

### Why Denormalized Fields?
Fields like `project_id` in UnitAssignment (already have via location) enable faster queries without joins.

### Why Scheduler in Terra-View?
Terra-View is the orchestration layer. SLMM only handles device communication. Keeping scheduling logic in Terra-View allows for complex workflows across multiple device types.

### Why JSON Metadata Columns?
Type-specific fields (like ambient_conditions for sound projects) don't apply to all location types. JSON columns provide flexibility without cluttering the schema.

---

## 💡 Tips for Continuing Development

1. **Follow Existing Patterns**: Look at the SLM dashboard code for reference
2. **Use HTMX Aggressively**: Minimize JavaScript, let HTMX handle updates
3. **Keep Routers Thin**: Move business logic to service layer
4. **Return HTML Partials**: Most endpoints should return HTML, not JSON
5. **Test Incrementally**: Build one partial at a time and test in browser
6. **Check Logs**: Scheduler logs execution attempts
7. **Use Browser DevTools**: Network tab shows HTMX requests

---

## 📞 Support

For questions or issues:
1. Check this document first
2. Review existing dashboards (SLM, Seismographs) for patterns
3. Check logs for scheduler execution details
4. Test API endpoints with curl to isolate issues

---

## ✅ Checklist for Completion

- [x] Database schema designed
- [x] Models created
- [x] Migration script run successfully
- [x] Service layer complete (SLMM client, device controller, scheduler)
- [x] API routers created (projects, locations, scheduler)
- [x] Navigation updated
- [x] Main overview page created
- [x] Routes registered in main.py
- [x] Scheduler service integrated
- [ ] Partial templates created
- [ ] Project dashboard page created
- [ ] Location management UI
- [ ] Unit assignment UI
- [ ] Scheduler UI (agenda view)
- [ ] SLMM download endpoint implemented
- [ ] Full workflow tested end-to-end
- [ ] SFM client implemented (future)

---

**Last Updated**: 2026-01-12

**Database Status**: ✅ Initialized

**Backend Status**: ✅ Complete

**Frontend Status**: 🟡 Partial (overview page only)

**Ready for Testing**: ✅ Yes (basic functionality)