terra-view/PROJECTS_SYSTEM_IMPLEMENTATION.md

Projects System Implementation - Terra-View

Overview

The Projects system has been successfully scaffolded in Terra-View. This document provides a complete overview of what has been built, how it works, and what needs to be completed.

✅ Completed Components

1. Database Schema

Location: /backend/models.py

Seven new tables have been added:

• ProjectType: Template definitions for project types (Sound, Vibration, Combined)
• Project: Top-level project organization with type reference
• MonitoringLocation: Generic locations (NRLs for sound, monitoring points for vibration)
• UnitAssignment: Links devices to locations
• ScheduledAction: Automated recording control schedules
• RecordingSession: Tracks actual recording/monitoring sessions
• DataFile: File references for downloaded data

Key Features:

• Type-aware design (project_type_id determines features)
• Flexible metadata fields (JSON columns for type-specific data)
• Denormalized fields for efficient queries
• Proper indexing on foreign keys

2. Service Layer

SLMM Client (/backend/services/slmm_client.py)

• Clean wrapper for all SLMM API operations
• Methods for: start/stop/pause/resume recording, get status, configure devices
• Error handling with custom exceptions
• Singleton pattern for easy access

Device Controller (/backend/services/device_controller.py)

• Routes commands to appropriate backend (SLMM for SLMs, SFM for seismographs)
• Unified interface across device types
• Ready for future SFM implementation

Scheduler Service (/backend/services/scheduler.py)

• Background task that checks for pending scheduled actions every 60 seconds
• Executes actions by calling device controller
• Creates/updates recording sessions
• Tracks execution status and errors
• Manual execution support for testing

3. API Routers

Projects Router (/backend/routers/projects.py)

Endpoints:

• GET /api/projects/list - Project list with stats
• GET /api/projects/stats - Overview statistics
• POST /api/projects/create - Create new project
• GET /api/projects/{id} - Get project details
• PUT /api/projects/{id} - Update project
• DELETE /api/projects/{id} - Archive project
• GET /api/projects/{id}/dashboard - Project dashboard data
• GET /api/projects/types/list - Get project type templates

Project Locations Router (/backend/routers/project_locations.py)

Endpoints:

• GET /api/projects/{id}/locations - List locations
• POST /api/projects/{id}/locations/create - Create location
• PUT /api/projects/{id}/locations/{location_id} - Update location
• DELETE /api/projects/{id}/locations/{location_id} - Delete location
• GET /api/projects/{id}/assignments - List unit assignments
• POST /api/projects/{id}/locations/{location_id}/assign - Assign unit
• POST /api/projects/{id}/assignments/{assignment_id}/unassign - Unassign unit
• GET /api/projects/{id}/available-units - Get units available for assignment

Scheduler Router (/backend/routers/scheduler.py)

Endpoints:

• GET /api/projects/{id}/scheduler/actions - List scheduled actions
• POST /api/projects/{id}/scheduler/actions/create - Create action
• POST /api/projects/{id}/scheduler/schedule-session - Schedule recording session
• PUT /api/projects/{id}/scheduler/actions/{action_id} - Update action
• POST /api/projects/{id}/scheduler/actions/{action_id}/cancel - Cancel action
• DELETE /api/projects/{id}/scheduler/actions/{action_id} - Delete action
• POST /api/projects/{id}/scheduler/actions/{action_id}/execute - Manual execution
• GET /api/projects/{id}/scheduler/status - Scheduler status
• POST /api/projects/{id}/scheduler/execute-pending - Trigger pending executions

4. Frontend

Main Page

Location: /templates/projects/overview.html

Features:

• Summary statistics cards (projects, locations, assignments, sessions)
• Tabbed interface (All, Active, Completed, Archived)
• Project cards grid layout
• Create project modal with two-step flow:
  1. Select project type (Sound/Vibration/Combined)
  2. Fill project details
• HTMX-powered dynamic updates

Navigation

Location: /templates/base.html (updated)

• "Projects" link added to sidebar
• Active state highlighting

5. Application Integration

Location: /backend/main.py

• Routers registered
• Page route added (/projects)
• Scheduler service starts on application startup
• Scheduler stops on application shutdown

6. Database Initialization

Script: /backend/init_projects_db.py

• Creates all project tables
• Populates ProjectType with default templates
• ✅ Successfully executed - database is ready

---

📁 File Organization

terra-view/
├── backend/
│   ├── models.py                              [✅ Updated]
│   ├── init_projects_db.py                    [✅ Created]
│   ├── main.py                                [✅ Updated]
│   ├── routers/
│   │   ├── projects.py                        [✅ Created]
│   │   ├── project_locations.py               [✅ Created]
│   │   └── scheduler.py                       [✅ Created]
│   └── services/
│       ├── slmm_client.py                     [✅ Created]
│       ├── device_controller.py               [✅ Created]
│       └── scheduler.py                       [✅ Created]
├── templates/
│   ├── base.html                              [✅ Updated]
│   ├── projects/
│   │   └── overview.html                      [✅ Created]
│   └── partials/
│       └── projects/                          [📁 Created, empty]
└── data/
    └── seismo_fleet.db                        [✅ Tables created]

---

🔨 What Still Needs to be Built

1. Frontend Templates (Partials)

Directory: /templates/partials/projects/

Required Files:

project_stats.html

Stats cards for overview page:

• Total/Active/Completed projects
• Total locations
• Assigned units
• Active sessions

project_list.html

Project cards grid:

• Project name, type, status
• Location count, unit count
• Active session indicator
• Link to project dashboard

project_dashboard.html

Main project dashboard panel with tabs:

• Summary stats
• Active locations and assignments
• Upcoming scheduled actions
• Recent sessions

location_list.html

Location cards/table:

• Location name, type, coordinates
• Assigned unit (if any)
• Session count
• Assign/unassign button

assignment_list.html

Unit assignment table:

• Unit ID, device type
• Location name
• Assignment dates
• Status
• Unassign button

scheduler_agenda.html

Calendar/agenda view:

• Scheduled actions sorted by time
• Action type (start/stop/download)
• Location and unit
• Status indicator
• Cancel/execute buttons

2. Project Dashboard Page

Location: /templates/projects/project_dashboard.html

Full project detail page with:

• Header with project name, type, status
• Tab navigation (Dashboard, Scheduler, Locations, Units, Data, Settings)
• Tab content areas
• Modals for adding locations, scheduling sessions

3. Additional UI Components

• Project type selection cards (with icons)
• Location creation modal
• Unit assignment modal
• Schedule session modal (with date/time picker)
• Data file browser

4. SLMM Enhancements

Location: /slmm/app/routers.py (SLMM repo)

New endpoint needed:

POST /api/nl43/{unit_id}/ftp/download

This should:

• Accept destination_path and files list
• Connect to SLM via FTP
• Download specified files
• Save to Terra-View's data/Projects/ directory
• Return file list with metadata

5. SFM Client (Future)

Location: /backend/services/sfm_client.py (to be created)

Similar to SLMM client, but for seismographs:

• Get seismograph status
• Start/stop recording
• Download data files
• Integrate with device controller

---

🚀 Testing the System

1. Start Terra-View

cd /home/serversdown/tmi/terra-view
# Start Terra-View (however you normally start it)

Verify in logs:

Starting scheduler service...
Scheduler service started

2. Navigate to Projects

Open browser: http://localhost:8001/projects

You should see:

• Summary stats cards (all zeros initially)
• Tabs (All Projects, Active, Completed, Archived)
• "New Project" button

3. Create a Project

1. Click "New Project"
2. Select a project type (e.g., "Sound Monitoring")
3. Fill in details:
   • Name: "Test Sound Project"
   • Client: "Test Client"
   • Start Date: Today
4. Submit

4. Test API Endpoints

# Get project types
curl http://localhost:8001/api/projects/types/list

# Get projects list
curl http://localhost:8001/api/projects/list

# Get project stats
curl http://localhost:8001/api/projects/stats

5. Test Scheduler Status

curl http://localhost:8001/api/projects/{project_id}/scheduler/status

---

📋 Dataflow Examples

Creating and Scheduling a Recording Session

1. User creates project → Project record in DB
2. User adds NRL → MonitoringLocation record
3. User assigns SLM to NRL → UnitAssignment record
4. User schedules recording → 2 ScheduledAction records (start + stop)
5. Scheduler runs every minute → Checks for pending actions
6. Start action time arrives → Scheduler calls SLMM via device controller
7. SLMM sends TCP command to SLM → Recording starts
8. RecordingSession created → Tracks the session
9. Stop action time arrives → Scheduler stops recording
10. Session updated → stopped_at, duration_seconds filled
11. User triggers download → Files copied to data/Projects/{project_id}/sound/{nrl_name}/
12. DataFile records created → Track file references

---

🎨 UI Design Patterns

Established Patterns (from SLM dashboard):

1. Stats Cards: 4-column grid, auto-refresh every 30s
2. Sidebar Lists: Searchable, filterable, auto-refresh
3. Main Panel: Large central area for details
4. Modals: Centered, overlay background
5. HTMX: All dynamic updates, minimal JavaScript
6. Tailwind: Consistent styling with dark mode support

Color Scheme:

• Primary: seismo-orange (#f48b1c)
• Secondary: seismo-navy (#142a66)
• Accent: seismo-burgundy (#7d234d)

---

🔧 Configuration

Environment Variables

• SLMM_BASE_URL: SLMM backend URL (default: http://localhost:8100)
• ENVIRONMENT: "development" or "production"

Scheduler Settings

Located in /backend/services/scheduler.py:

• check_interval: 60 seconds (adjust as needed)

---

📚 Next Steps

Immediate (Get Basic UI Working):

1. Create partial templates (stats, lists)
2. Test creating projects via UI
3. Implement project dashboard page

Short-term (Core Features):

4. Add location management UI
5. Add unit assignment UI
6. Add scheduler UI (agenda view)

Medium-term (Data Flow):

7. Implement SLMM download endpoint
8. Test full recording workflow
9. Add file browser for downloaded data

Long-term (Complete System):

10. Implement SFM client for seismographs
11. Add data visualization
12. Add project reporting
13. Add user authentication

---

🐛 Known Issues / TODOs

1. Partial templates missing: Need to create HTML templates for all partials
2. SLMM download endpoint: Needs implementation in SLMM backend
3. Project dashboard page: Not yet created
4. SFM integration: Placeholder only, needs real implementation
5. File download tracking: DataFile records not yet created after downloads
6. Error handling: Need better user-facing error messages
7. Validation: Form validation could be improved
8. Testing: No automated tests yet

---

📖 API Documentation

Project Type Object

{
  "id": "sound_monitoring",
  "name": "Sound Monitoring",
  "description": "...",
  "icon": "volume-2",
  "supports_sound": true,
  "supports_vibration": false
}

Project Object

{
  "id": "uuid",
  "name": "Project Name",
  "description": "...",
  "project_type_id": "sound_monitoring",
  "status": "active",
  "client_name": "Client Inc",
  "site_address": "123 Main St",
  "site_coordinates": "40.7128,-74.0060",
  "start_date": "2024-01-15",
  "end_date": null,
  "created_at": "2024-01-15T10:00:00",
  "updated_at": "2024-01-15T10:00:00"
}

MonitoringLocation Object

{
  "id": "uuid",
  "project_id": "uuid",
  "location_type": "sound",
  "name": "NRL-001",
  "description": "...",
  "coordinates": "40.7128,-74.0060",
  "address": "123 Main St",
  "location_metadata": "{...}",
  "created_at": "2024-01-15T10:00:00"
}

UnitAssignment Object

{
  "id": "uuid",
  "unit_id": "nl43-001",
  "location_id": "uuid",
  "project_id": "uuid",
  "device_type": "sound_level_meter",
  "assigned_at": "2024-01-15T10:00:00",
  "assigned_until": null,
  "status": "active",
  "notes": "..."
}

ScheduledAction Object

{
  "id": "uuid",
  "project_id": "uuid",
  "location_id": "uuid",
  "unit_id": "nl43-001",
  "action_type": "start",
  "device_type": "sound_level_meter",
  "scheduled_time": "2024-01-16T08:00:00",
  "executed_at": null,
  "execution_status": "pending",
  "module_response": null,
  "error_message": null
}

---

🎓 Architecture Decisions

Why Project Types?

Allows the system to scale to different monitoring scenarios (air quality, multi-hazard, etc.) without code changes. Just add a new ProjectType record and the UI adapts.

Why Generic MonitoringLocation?

Instead of separate NRL and MonitoringPoint tables, one table with a location_type discriminator keeps the schema clean and allows for combined projects.

Why Denormalized Fields?

Fields like project_id in UnitAssignment (already have via location) enable faster queries without joins.

Why Scheduler in Terra-View?

Terra-View is the orchestration layer. SLMM only handles device communication. Keeping scheduling logic in Terra-View allows for complex workflows across multiple device types.

Why JSON Metadata Columns?

Type-specific fields (like ambient_conditions for sound projects) don't apply to all location types. JSON columns provide flexibility without cluttering the schema.

---

💡 Tips for Continuing Development

1. Follow Existing Patterns: Look at the SLM dashboard code for reference
2. Use HTMX Aggressively: Minimize JavaScript, let HTMX handle updates
3. Keep Routers Thin: Move business logic to service layer
4. Return HTML Partials: Most endpoints should return HTML, not JSON
5. Test Incrementally: Build one partial at a time and test in browser
6. Check Logs: Scheduler logs execution attempts
7. Use Browser DevTools: Network tab shows HTMX requests

---

📞 Support

For questions or issues:

1. Check this document first
2. Review existing dashboards (SLM, Seismographs) for patterns
3. Check logs for scheduler execution details
4. Test API endpoints with curl to isolate issues

---

✅ Checklist for Completion

• Database schema designed
• Models created
• Migration script run successfully
• Service layer complete (SLMM client, device controller, scheduler)
• API routers created (projects, locations, scheduler)
• Navigation updated
• Main overview page created
• Routes registered in main.py
• Scheduler service integrated
• Partial templates created
• Project dashboard page created
• Location management UI
• Unit assignment UI
• Scheduler UI (agenda view)
• SLMM download endpoint implemented
• Full workflow tested end-to-end
• SFM client implemented (future)

---

Last Updated: 2026-01-12

Database Status: ✅ Initialized

Backend Status: ✅ Complete

Frontend Status: 🟡 Partial (overview page only)

Ready for Testing: ✅ Yes (basic functionality)