Compare commits
153 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| fd37425f1c | |||
| 4378290c9c | |||
| 9775dca114 | |||
| 904ff04440 | |||
| 155f0b007a | |||
| 583af1948e | |||
| 449e031589 | |||
| 18fd0472a5 | |||
| e15481884a | |||
| 737901c962 | |||
| 2cf5bf47d3 | |||
| 77483c2186 | |||
| b1c2a1d778 | |||
| d3b5a3fd26 | |||
| d46f9fccf8 | |||
| 6ebbe28308 | |||
| 42de06f441 | |||
| 21844b4d65 | |||
| 80fa76208a | |||
| f1f3da8e61 | |||
| 63bd6ad8a2 | |||
| bc5a151faa | |||
| 09db988a35 | |||
| df771a87de | |||
| a71e6f5efd | |||
| ec661ee079 | |||
| 32d2a57bc9 | |||
| 63ba63edaf | |||
| 2ba20c7809 | |||
| f84d0818d2 | |||
| 3e0d20d62d | |||
| f50cf2b7f6 | |||
| 20e180644e | |||
| 73a6ff4d20 | |||
| 0f582a8a17 | |||
| 184f0ddd13 | |||
| e7bd09418b | |||
| 27eeb0fae6 | |||
| 192e15f238 | |||
| 49bc625c1a | |||
| 95fedca8c9 | |||
| e8e155556a | |||
| 33e962e73d | |||
| ac48fb2977 | |||
| 5e9cc32fdc | |||
| 3c4b81cf78 | |||
| d135727ebd | |||
| 64d4423308 | |||
| 4f71d528ce | |||
| 4f56dea4f3 | |||
| 40359db066 | |||
| 57a85f565b | |||
| e6555ba924 | |||
| 3d5b2fddef | |||
| 8694282dd0 | |||
| bc02dc9564 | |||
| 0d01715f81 | |||
| b3ec249c5e | |||
| b6e74258f1 | |||
| 5ea64c3561 | |||
| 1a87ff13c9 | |||
| 22c62c0729 | |||
| 0f47b69c92 | |||
| 76667454b3 | |||
| 0e3f512203 | |||
| 15d962ba42 | |||
| e4d1f0d684 | |||
| b571dc29bc | |||
| e2c841d5d7 | |||
| cc94493331 | |||
| 5a5426cceb | |||
| 66eddd6fe2 | |||
| c77794787c | |||
| 61c84bc71d | |||
| fbf7f2a65d | |||
| 202fcaf91c | |||
| 3a411d0a89 | |||
| 0c2186f5d8 | |||
| c138e8c6a0 | |||
| 1dd396acd8 | |||
| e89a04f58c | |||
| e4ef065db8 | |||
| 86010de60c | |||
| f89f04cd6f | |||
| 67a2faa2d3 | |||
| 14856e61ef | |||
| 2b69518b33 | |||
| 6070d03e83 | |||
| 240552751c | |||
| 015ce0a254 | |||
| ef8c046f31 | |||
| 3637cf5af8 | |||
| 7fde14d882 | |||
| bd3d937a82 | |||
| 291fa8e862 | |||
| 8e292b1aca | |||
| 7516bbea70 | |||
| da4e5f66c5 | |||
| dae2595303 | |||
| 0c4e7aa5e6 | |||
| 229499ccf6 | |||
| fdc4adeaee | |||
| b3bf91880a | |||
| 17b3f91dfc | |||
| 6c1d0bc467 | |||
| abd059983f | |||
| 0f17841218 | |||
| 65362bab21 | |||
| dc77a362ce | |||
| 28942600ab | |||
| 80861997af | |||
| b15d434fce | |||
| 70ef43de11 | |||
| 7b4e12c127 | |||
| 24473c9ca3 | |||
| caabfd0c42 | |||
| ebe60d2b7d | |||
| 842e9d6f61 | |||
| 742a98a8ed | |||
| 3b29c4d645 | |||
| 63d9c59873 | |||
| 794bfc00dc | |||
| 89662d2fa5 | |||
| eb0a99796d | |||
| b47e69e609 | |||
| 1cb25b6c17 | |||
| e515bff1a9 | |||
| f296806fd1 | |||
| 24da5ab79f | |||
| 305540f564 | |||
| 639b485c28 | |||
| d78bafb76e | |||
| 8373cff10d | |||
| 4957a08198 | |||
| 05482bd903 | |||
| 5ee6f5eb28 | |||
| 7ce0f6115d | |||
| 6492fdff82 | |||
| 44d7841852 | |||
| 38c600aca3 | |||
| eeda94926f | |||
| 57be9bf1f1 | |||
| 8431784708 | |||
| c771a86675 | |||
| 65ea0920db | |||
| 1f3fa7a718 | |||
| a9c9b1fd48 | |||
| 4c213c96ee | |||
| ff38b74548 | |||
| c8a030a3ba | |||
| d8a8330427 | |||
| 1ef0557ccb | |||
| 6c7ce5aad0 |
@@ -1,3 +1,5 @@
|
||||
docker-compose.override.yml
|
||||
|
||||
# Python cache / compiled
|
||||
__pycache__
|
||||
*.pyc
|
||||
@@ -28,6 +30,7 @@ ENV/
|
||||
|
||||
# Runtime data (mounted volumes)
|
||||
data/
|
||||
data-dev/
|
||||
|
||||
# Editors / OS junk
|
||||
.vscode/
|
||||
|
||||
@@ -1,3 +1,17 @@
|
||||
# Terra-View Specifics
|
||||
# Dev build counter (local only, never commit)
|
||||
build_number.txt
|
||||
docker-compose.override.yml
|
||||
|
||||
# SQLite database files
|
||||
*.db
|
||||
*.db-journal
|
||||
data/
|
||||
data-dev/
|
||||
.aider*
|
||||
.aider*
|
||||
|
||||
|
||||
# Byte-compiled / optimized / DLL files
|
||||
__pycache__/
|
||||
*.py[codz]
|
||||
@@ -206,10 +220,14 @@ marimo/_static/
|
||||
marimo/_lsp/
|
||||
__marimo__/
|
||||
|
||||
<<<<<<< HEAD
|
||||
# Seismo Fleet Manager
|
||||
# SQLite database files
|
||||
*.db
|
||||
*.db-journal
|
||||
data/
|
||||
/data/
|
||||
/data-dev/
|
||||
.aider*
|
||||
.aider*
|
||||
=======
|
||||
>>>>>>> 0c2186f5d89d948b0357d674c0773a67a67d8027
|
||||
|
||||
@@ -1,10 +1,360 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes to Seismo Fleet Manager will be documented in this file.
|
||||
All notable changes to Terra-View will be documented in this file.
|
||||
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [0.10.0] - 2026-05-14
|
||||
|
||||
This release brings terra-view onto the SFM (Seismograph Field Module) event pipeline. Triggered events forwarded by series3-watcher now land in SFM, and terra-view reads from that store as the authoritative source for vibration data. The watcher heartbeat is preserved as a transparent fallback signal.
|
||||
|
||||
### Added
|
||||
- **SFM Integration**: New fleet-wide events page at `/sfm` listing every event ingested by SFM, with filters for serial, date range, false-trigger flag, and limit. Unit detail pages and project-location pages show their own attributed subsets of the same event stream.
|
||||
- **Event Detail Modal**: Shared across `/sfm`, unit detail, and project-location pages — clicking any event opens a rich modal showing peaks per channel (PVS color-coded by magnitude), microphone dB(L) + ZC frequency + time of peak, sensor self-check table with pass/fail per channel, device/recording metadata (firmware, battery, calibration date, geo range), and download buttons for the original Blastware binary and the sidecar JSON. Includes an inline pretty-printed JSON viewer with copy-to-clipboard.
|
||||
- **Events Attribution Engine** (`backend/services/sfm_events.py`): Per-event attribution against `UnitAssignment` time windows. Events outside any assignment window surface in an "Unattributed" bucket with the nearest-assignment diagnostic (which location, signed delta in days).
|
||||
- **Metadata Backfill Tool** (`/tools` → Backfill from event metadata): Scans operator-typed `project` and `sensor_location` strings in event sidecars, fuzzy-clusters them via `rapidfuzz.WRatio`, and proposes retroactive `UnitAssignment` records to attribute orphan events. Tracks operator decisions per cluster across re-scans.
|
||||
- **Project Tidy Tool** (`/tools` → Project Tidy): Fuzzy-detect duplicate projects and bulk-merge them with a single click. Source projects soft-deleted with full audit trail.
|
||||
- **Vibration Summary on Project Pages**: New roll-up card on vibration project detail pages showing per-location event counts, the project's "Overall Peak" PVS (false triggers excluded), last event timestamp, and a Top Locations by Activity list.
|
||||
- **SFM-Primary Seismograph Status**: `emit_status_snapshot()` now consults SFM's `/db/units` (cached 15s) before falling back to `Emitter.last_seen` for each seismograph. The fresher signal wins; the choice is recorded in a new per-unit `last_seen_source` field. A small `SFM` (orange) or `HB` (gray) badge on each unit's active-table row shows which path is currently driving the status.
|
||||
- **Dashboard Rework**: Top row reordered to Recent Alerts → Recent Call-Ins (double-wide) → Fleet Summary. Today's Schedule moved to a horizontal collapsible card below the Fleet Map, auto-expanding only when pending actions exist. Recent Call-Ins now sources from a new `/api/recent-event-callins` endpoint backed by SFM event forwards instead of the watcher-heartbeat endpoint.
|
||||
- **Sortable Events Tables**: `/sfm` and unit-detail SFM Events tables now have clickable column headers with ↕/↓/↑ indicators. Default sort is Timestamp DESC. Click same column to toggle direction; click different column to switch and reset to DESC. Pure client-side over cached rows — no re-fetches.
|
||||
- **Developer → SFM Admin** (`/admin/sfm`): Health banner with reachability indicator, terra-view↔SFM connection panel, 4 KPI tiles (known units, total events, stale `monitor_log` rows, stale `ach_sessions` rows), per-unit roll-up table, recent-events table with color-coded forwarding latency (so stale watcher forwards stand out), and a raw API tester for any `/api/sfm/*` path.
|
||||
- **Developer → SLMM Admin** (`/admin/slmm`): Stripped-down companion page — health, connection info, raw API tester.
|
||||
- **Tools Workflow Hub** (`/tools`): New top-level sidebar entry consolidating Pair Devices, Project Tidy, Metadata Backfill, Reports (info card), and Swap Detection (placeholder).
|
||||
- **Sidebar Reorganization**: Devices → Projects → Events → Tools → Job Planner → Settings. Devices is now a single entry with internal tabs (All Devices / Seismographs / Sound Level Meters / Modems / Pair Devices) replacing five separate sidebar items.
|
||||
- **Synology Deployment Doc** (`docs/SYNOLOGY_DEPLOYMENT.md`): End-to-end playbook for migrating the stack to an always-on office NAS — phased rollout (pre-stage, data rsync, watcher repoint, external access, decommission), Tailscale vs reverse-proxy options, rollback plan, and gotchas.
|
||||
|
||||
### Changed
|
||||
- **Overall Peak excludes false triggers**: The project-level "Overall Peak" KPI tile (and the underlying `_compute_stats()` function in `sfm_events.py`) now skip events flagged as false triggers when computing the highest PVS, so operators see the highest real event rather than the biggest sensor glitch. `false_trigger_count` still includes flagged events so operators can see how many were filtered out.
|
||||
- **`RosterUnit.note` Editing**: Inline edit on seismograph cards is more forgiving and now auto-saves on blur.
|
||||
- **Sidebar Nav Renamed**: Old "Fleet" sidebar entry → "Devices" (renamed because it always meant the device list, not the broader fleet view).
|
||||
|
||||
### Fixed
|
||||
- **Status drift between watcher heartbeat and actual event arrivals**: Seismographs are now reported with whichever signal is more recent — eliminates the case where a unit had recent SFM events but a stale heartbeat (or vice-versa) showed the wrong status.
|
||||
- **Event modal: Record Type always showed "Waveform"**: Workaround client-side — Record Type now derived from the Blastware filename's last-char code (`H`=Histogram, `W`=Waveform, `M`=Manual, `E`=Event, `C`=Combo). The proper fix lives in SFM's sidecar parser; tracked separately.
|
||||
- **Event modal: Mic PSI tile removed**: Operators only care about dB(L); the redundant PSI tile was dropped.
|
||||
|
||||
### Migration Notes
|
||||
Run on each database before deploying. Every migration is idempotent.
|
||||
|
||||
```bash
|
||||
# Cleanest: re-run all migrations in chronological order.
|
||||
# Already-applied migrations no-op safely.
|
||||
for f in backend/migrate_*.py; do
|
||||
docker exec terra-view-terra-view-1 python3 "/app/backend/$(basename $f)"
|
||||
done
|
||||
```
|
||||
|
||||
Migrations new in this release:
|
||||
- `migrate_add_metadata_backfill.py` — adds `unit_assignments.source` column and `metadata_backfill_decisions` table for the Metadata Backfill tool
|
||||
|
||||
### Deployment Notes
|
||||
- **`SFM_BASE_URL`**: Confirm prod's `docker-compose.yml` sets this for the terra-view service (typically `http://sfm:8200` for the in-stack SFM container, or an external URL if SFM lives elsewhere).
|
||||
- **Watcher repoint**: series3-watcher's `sfm_forward_url` should point at `https://<your-terra-view-host>/api/sfm` (proxy-based — no second port forward needed). Watcher composes the full path `/db/import/blastware_file` itself.
|
||||
|
||||
---
|
||||
|
||||
## [0.9.4] - 2026-04-06
|
||||
|
||||
### Added
|
||||
- **Modular Project Types**: Projects now support optional modules (Sound Monitoring, Vibration Monitoring) selectable at creation time. The project header and dashboard dynamically show/hide tabs and actions based on which modules are enabled, and modules can be added or removed after creation.
|
||||
- **Deleted Project Management**: Settings page now includes a section for soft-deleted projects with options to restore or permanently delete each one. Deleted projects load automatically when the Data tab is opened.
|
||||
|
||||
### Changed
|
||||
- **Swap Modal Search**: The unit/modem swap modal on vibration location detail pages now includes live search filtering for both seismographs and modems, making it easier to find the right unit in large fleets.
|
||||
|
||||
### Fixed
|
||||
- **Roster Auto-Refresh No Longer Disrupts Scroll/Sort**: The roster page's 30-second background refresh now updates status, age, and last-seen values in-place via a lightweight JSON poll instead of replacing the entire table HTML. Sort order, scroll position, and active filters are all preserved across refreshes.
|
||||
|
||||
### Migration Notes
|
||||
Run on each database before deploying:
|
||||
```bash
|
||||
docker compose exec terra-view python3 backend/migrate_add_project_modules.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [0.9.3] - 2026-03-28
|
||||
|
||||
### Added
|
||||
- **Monitoring Session Detail Page**: New dedicated page for each session showing session info, data files (with View/Report/Download actions), an editable session panel, and report actions.
|
||||
- **Session Calendar with Gantt Bars**: Monthly calendar view below the session list, showing each session as a Gantt-style bar. The dim bar represents the full device on/off window; the bright bar highlights the effective recording window. Bars extend edge-to-edge across day cells for sessions spanning midnight.
|
||||
- **Configurable Period Windows**: Sessions now store `period_start_hour` and `period_end_hour` to define the exact hours that count toward reports, replacing hardcoded day/night defaults. The session edit panel shows a "Required Recording Window" section with a live preview (e.g. "7:00 AM → 7:00 PM") and a Defaults button that auto-fills based on period type.
|
||||
- **Report Date Field**: Sessions can now store an explicit `report_date` to override the automatic target-date heuristic — useful when a device ran across multiple days but only one specific day's data is needed for the report.
|
||||
- **Effective Window on Session Info**: Session detail and session cards now show an "Effective" row displaying the computed recording window dates and times in local time.
|
||||
- **Vibration Project Redesign**: Vibration project detail page is stripped back to project details and monitoring locations only. Each location supports assigning a seismograph and optional modem. Sound-specific tabs (Schedules, Sessions, Data Files, Assigned Units) are hidden for vibration projects.
|
||||
- **Modem Assignment on Locations**: Vibration monitoring locations now support an optional paired modem alongside the seismograph. The swap endpoint handles both assignments atomically, updating bidirectional pairing fields on both units.
|
||||
- **Available Modems Endpoint**: New `GET /api/projects/{project_id}/available-modems` endpoint returning all deployed, non-retired modems for use in assignment dropdowns.
|
||||
|
||||
### Fixed
|
||||
- **Active Assignment Checks**: Unified all `UnitAssignment` "active" checks from `status == "active"` to `assigned_until IS NULL` throughout `project_locations.py` and `projects.py` for consistency with the canonical active definition.
|
||||
|
||||
### Changed
|
||||
- **Sound-Only Endpoint Guards**: FTP browser, RND viewer, Excel report generation, combined report wizard, and data upload endpoints now return HTTP 400 if called on a non-sound-monitoring project.
|
||||
|
||||
### Migration Notes
|
||||
Run on each database before deploying:
|
||||
```bash
|
||||
docker compose exec terra-view python3 backend/migrate_add_session_period_hours.py
|
||||
docker compose exec terra-view python3 backend/migrate_add_session_report_date.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [0.9.2] - 2026-03-27
|
||||
|
||||
### Added
|
||||
- **Deployment Records**: Seismographs now track a full deployment history (location, project, dates). Each deployment is logged on the unit detail page with start/end dates, and the fleet calendar service uses this history for availability calculations.
|
||||
- **Allocated Unit Status**: New `allocated` status for units reserved for an upcoming job but not yet deployed. Allocated units appear in the dashboard summary, roster filters, and devices table with visual indicators.
|
||||
- **Project Allocation**: Units can be linked to a project via `allocated_to_project_id`. Allocation is shown on the unit detail page and in a new quick-info modal accessible from the fleet calendar and roster.
|
||||
- **Quick-Info Unit Modal**: Click any unit in the fleet calendar or roster to open a modal showing cal status, project allocation, upcoming jobs, and deployment state — without leaving the page.
|
||||
- **Cal Date in Planner**: When a unit is selected for a monitoring location slot in the Job Planner, its calibration expiry date is now shown inline so you can spot near-expiry units before committing.
|
||||
- **Inline Seismograph Editing**: Unit rows in the seismograph dashboard now support inline editing of cal date, notes, and deployment status without navigating to the full detail page.
|
||||
|
||||
### Migration Notes
|
||||
Run on each database before deploying:
|
||||
```bash
|
||||
docker compose exec terra-view python3 backend/migrate_add_allocated.py
|
||||
docker compose exec terra-view python3 backend/migrate_add_deployment_records.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [0.9.1] - 2026-03-20
|
||||
|
||||
### Fixed
|
||||
- **Location slots not persisting**: Empty monitoring location slots (no unit assigned yet) were lost on save/reload. Added `location_slots` JSON column to `job_reservations` to store the full slot list including empty slots.
|
||||
- **Modems in Recent Alerts**: Modems no longer appear in the dashboard Recent Alerts panel — alerts are for seismographs and SLMs only. Modem status is still tracked internally via paired device inheritance.
|
||||
- **Series 4 heartbeat `source_id`**: Updated heartbeat endpoint to accept the new `source_id` field from Series 4 units with fallback to the legacy field for backwards compatibility.
|
||||
|
||||
### Migration Notes
|
||||
Run on each database before deploying:
|
||||
```bash
|
||||
docker compose exec terra-view python3 backend/migrate_add_location_slots.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [0.9.0] - 2026-03-19
|
||||
|
||||
### Added
|
||||
- **Job Planner**: Full redesign of the Fleet Calendar into a two-tab Job Planner / Calendar interface
|
||||
- **Planner tab**: Create and manage job reservations with name, device type, dates, color, estimated units, and monitoring locations
|
||||
- **Calendar tab**: 12-month rolling heatmap with colored job bars per day; confirmed jobs solid, planned jobs dashed
|
||||
- **Monitoring Locations**: Each job has named location slots (filled = unit assigned, empty = needs a unit); progress shown as `2/5` with colored squares that fill as units are assigned
|
||||
- **Estimated Units**: Separate planning number independent of actual location count; shown prominently on job cards
|
||||
- **Fleet Summary panel**: Unit counts as clickable filter buttons; unit list shows reservation badges with job name, dates, and color
|
||||
- **Available Units panel**: Shows units available for the job's date range when assigning
|
||||
- **Smart color picker**: 18-swatch palette + custom color wheel; new jobs auto-pick a color maximally distant in hue from existing jobs
|
||||
- **Job card progress**: `est. N · X/Y (Z more)` with filled/empty squares; amber → green when fully assigned
|
||||
- **Promote to Project**: Promote a planned job to a tracked project directly from the planner form
|
||||
- **Collapsible job details**: Name, dates, device type, color, project link, and estimated units collapse into a summary header
|
||||
- **Calendar bar tooltips**: Hover any job bar to see job name and date range
|
||||
- **Hash-based tab persistence**: `#cal` in URL restores Calendar tab on refresh; device type toggle preserves active tab
|
||||
- **Auto-scroll to today**: Switching to Calendar tab smooth-scrolls to the current month
|
||||
- **Upcoming project status**: New `upcoming` status for projects promoted from reservations
|
||||
- **Job device type**: Reservations carry a device type so they only appear on the correct calendar
|
||||
- **Project filtering by device type**: Projects only appear on the calendar matching their type (vibration → seismograph, sound → SLM, combined → both)
|
||||
- **Confirmed/Planned toggles**: Independent show/hide toggles for job bar layers on the calendar
|
||||
- **Cal expire dots toggle**: Calibration expiry dots off by default, togglable
|
||||
|
||||
### Changed
|
||||
- **Renamed**: "Fleet Calendar" / "Reservation Planner" → **"Job Planner"** throughout UI and sidebar
|
||||
- **Project status dropdown**: Inline `<select>` in project header for quick status changes
|
||||
- **"All Projects" tab**: Shows everything except deleted; default view excludes archived/completed
|
||||
- **Toast notifications**: All `alert()` dialogs replaced with non-blocking toasts (green = success, red = error)
|
||||
|
||||
### Migration Notes
|
||||
Run on each database before deploying:
|
||||
```bash
|
||||
docker compose exec terra-view python3 -c "
|
||||
import sqlite3
|
||||
conn = sqlite3.connect('/app/data/seismo_fleet.db')
|
||||
conn.execute('ALTER TABLE job_reservations ADD COLUMN estimated_units INTEGER')
|
||||
conn.commit()
|
||||
conn.close()
|
||||
"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [0.8.0] - 2026-03-18
|
||||
|
||||
### Added
|
||||
- **Watcher Manager**: New admin page (`/admin/watchers`) for monitoring field watcher agents
|
||||
- Live status cards per agent showing connectivity, version, IP, last-seen age, and log tail
|
||||
- Trigger Update button to queue a self-update on the agent's next heartbeat
|
||||
- Expand/collapse log tail with full-log expand mode
|
||||
- Live surgical refresh every 30 seconds via `/api/admin/watchers` — no full page reload, open logs stay open
|
||||
|
||||
### Changed
|
||||
- **Watcher status logic**: Agent status now reflects whether Terra-View is hearing from the watcher (ok if seen within 60 minutes, missing otherwise) — previously reflected the worst unit status from the last heartbeat payload, which caused false alarms when units went missing
|
||||
|
||||
### Fixed
|
||||
- **Watcher Manager meta row**: Dark mode background was white due to invalid `dark:bg-slate-850` Tailwind class; corrected to `dark:bg-slate-800`
|
||||
|
||||
---
|
||||
|
||||
## [0.7.1] - 2026-03-12
|
||||
|
||||
### Added
|
||||
- **"Out for Calibration" Unit Status**: New `out_for_cal` status for units currently away for calibration, with visual indicators in the roster, unit list, and seismograph stats panel
|
||||
- **Reservation Modal**: Fleet calendar reservation modal is now fully functional for creating and managing device reservations
|
||||
|
||||
### Changed
|
||||
- **Retire Unit Button**: Redesigned to be more visually prominent/destructive to reduce accidental clicks
|
||||
|
||||
### Fixed
|
||||
- **Migration Scripts**: Fixed database path references in several migration scripts
|
||||
- **Docker Compose**: Removed dev override file from the repository; dev environment config kept separate
|
||||
|
||||
### Migration Notes
|
||||
Run the following migration script once per database before deploying:
|
||||
```bash
|
||||
python backend/migrate_add_out_for_calibration.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [0.7.0] - 2026-03-07
|
||||
|
||||
### Added
|
||||
- **Project Status Management**: Projects can now be placed `on_hold` or `archived`, with automatic cancellation of pending scheduled actions
|
||||
- **Hard Delete Projects**: Support for permanently deleting projects, in addition to soft-delete with auto-pruning
|
||||
- **Vibration Location Detail**: New dedicated template for vibration project location detail views
|
||||
- **Vibration Project Isolation**: Vibration projects no longer show SLM-specific project tabs
|
||||
- **Manual SD Card Data Upload**: Upload offline NRL data directly from SD card via ZIP or multi-file select
|
||||
- Accepts `.rnd`/`.rnh` files; parses `.rnh` metadata for session start/stop times, serial number, and store name
|
||||
- Creates `MonitoringSession` and `DataFile` records automatically; no unit assignment required
|
||||
- Upload panel on NRL detail Data Files tab with inline feedback and auto-refresh via HTMX
|
||||
- **Standalone SLM Type**: New SLM device mode that operates without a modem (direct IP connection)
|
||||
- **NL32 Data Support**: Report generator and web viewer now support NL32 measurement data format
|
||||
- **Combined Report Wizard**: Multi-session combined Excel report generation tool
|
||||
- Wizard UI grouped by location with period type badges (day/night)
|
||||
- Each selected session produces one `.xlsx` in a ZIP archive
|
||||
- Period type filtering: day sessions keep last calendar date (7AM–6:59PM); night sessions span both days (7PM–6:59AM)
|
||||
- **Combined Report Preview**: Interactive spreadsheet-style preview before generating combined reports
|
||||
- **Chart Preview**: Live chart preview in the report generator matching final report styling
|
||||
- **SLM Model Schemas**: Per-model configuration schemas for NL32, NL43, NL53 devices
|
||||
- **Data Collection Mode**: Projects now store a data collection mode field with UI controls and migration
|
||||
|
||||
### Changed
|
||||
- **MonitoringSession rename**: `RecordingSession` renamed to `MonitoringSession` throughout codebase; DB table renamed from `recording_sessions` to `monitoring_sessions`
|
||||
- Migration: `backend/migrate_rename_recording_to_monitoring_sessions.py`
|
||||
- **Combined Report Split Logic**: Separate days now generate separate `.xlsx` files; NRLs remain one per sheet
|
||||
- **Mass Upload Parsing**: Smarter file filtering — no longer imports unneeded Lp files or `.xlsx` files
|
||||
- **SLM Start Time Grace Period**: 15-minute grace window added so data starting at session start time is included
|
||||
- **NL32 Date Parsing**: Date now read from `start_time` field instead of file metadata
|
||||
- **Project Data Labels**: Improved Jinja filters and UI label clarity for project data views
|
||||
|
||||
### Fixed
|
||||
- **Dev/Prod Separation**: Dev server now uses Docker Compose override; production deployment no longer affected by dev config
|
||||
- **SLM Modal**: Bench/deploy toggle now correctly shown in SLM unit modal
|
||||
- **Auto-Downloaded Files**: Files downloaded by scheduler now appear in project file listings
|
||||
- **Duplicate Download**: Removed duplicate file download that occurred following a scheduled stop
|
||||
- **SLMM Environment Variables**: `TCP_IDLE_TTL` and `TCP_MAX_AGE` now correctly passed to SLMM service via docker-compose
|
||||
|
||||
### Technical Details
|
||||
- `session_label` and `period_type` stored on `monitoring_sessions` table (migration: `migrate_add_session_period_type.py`)
|
||||
- `device_model` stored on `monitoring_sessions` table (migration: `migrate_add_session_device_model.py`)
|
||||
- Upload endpoint: `POST /api/projects/{project_id}/nrl/{location_id}/upload-data`
|
||||
- ZIP filename format: `{session_label}_{project_name}_report.xlsx` (label first)
|
||||
|
||||
### Migration Notes
|
||||
Run the following migration scripts once per database before deploying:
|
||||
```bash
|
||||
python backend/migrate_rename_recording_to_monitoring_sessions.py
|
||||
python backend/migrate_add_session_period_type.py
|
||||
python backend/migrate_add_session_device_model.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## [0.6.1] - 2026-02-16
|
||||
|
||||
### Added
|
||||
- **One-Off Recording Schedules**: Support for scheduling single recordings with specific start and end datetimes
|
||||
- **Bidirectional Pairing Sync**: Pairing a device with a modem now automatically updates both sides, clearing stale pairings when reassigned
|
||||
- **Auto-Fill Notes from Modem**: Notes are now copied from modem to paired device when fields are empty
|
||||
- **SLMM Download Requests**: New `_download_request` method in SLMM client for binary file downloads with local save
|
||||
|
||||
### Fixed
|
||||
- **Scheduler Timezone**: One-off scheduler times now use local time instead of UTC
|
||||
- **Pairing Consistency**: Old device references are properly cleared when a modem is re-paired to a new device
|
||||
|
||||
## [0.6.0] - 2026-02-06
|
||||
|
||||
### Added
|
||||
- **Calendar & Reservation Mode**: Fleet calendar view with reservation system for scheduling device deployments
|
||||
- **Device Pairing Interface**: New two-column pairing page (`/pair-devices`) for linking recorders (seismographs/SLMs) with modems
|
||||
- Visual pairing interface with drag-and-drop style interactions
|
||||
- Fuzzy-search modem pairing for SLMs
|
||||
- Pairing options now accessible from modem page
|
||||
- Improved pair status sharing across views
|
||||
- **Modem Dashboard Enhancements**:
|
||||
- Modem model number now a dedicated configuration field with per-model options
|
||||
- Direct link to modem login page from unit detail view
|
||||
- Modem view converted to list format
|
||||
- **Seismograph List Improvements**:
|
||||
- Enhanced visibility with better filtering and sorting
|
||||
- Calibration dates now color-coded for quick status assessment
|
||||
- User sets date of previous calibration (not expiry) for clearer workflow
|
||||
- **SLMM Device Control Lock**: Prevents command flooding to NL-43 devices
|
||||
|
||||
### Changed
|
||||
- **Calibration Date UX**: Users now set the date of the previous calibration rather than upcoming expiry dates - more intuitive workflow
|
||||
- **Settings Persistence**: Settings save no longer reloads the page
|
||||
- **Tab State**: Tab state now persists in URL hash for better navigation
|
||||
- **Scheduler Management**: Schedule changes now cascade to individual events
|
||||
- **Dashboard Filtering**: Enhanced dashboard with additional filtering options and SLM status sync
|
||||
- **SLMM Polling Intervals**: Fixed and improved polling intervals for better responsiveness
|
||||
- **24-Hour Scheduler Cycle**: Improved cycle handling to prevent issues with scheduled downloads
|
||||
|
||||
### Fixed
|
||||
- **SLM Modal Fields**: Modal now only contains correct device-specific fields
|
||||
- **IP Address Handling**: IP address correctly passed via modem pairing
|
||||
- **Mobile Type Display**: Fixed incorrect device type display in roster and device tables
|
||||
- **SLMM Scheduled Downloads**: Fixed issues with scheduled download operations
|
||||
|
||||
## [0.5.1] - 2026-01-27
|
||||
|
||||
### Added
|
||||
- **Dashboard Schedule View**: Today's scheduled actions now display directly on the main dashboard
|
||||
- New "Today's Actions" panel showing upcoming and past scheduled events
|
||||
- Schedule list partial for project-specific schedule views
|
||||
- API endpoint for fetching today's schedule data
|
||||
- **New Branding Assets**: Complete logo rework for Terra-View
|
||||
- New Terra-View logos for light and dark themes
|
||||
- Retina-ready (@2x) logo variants
|
||||
- Updated favicons (16px and 32px)
|
||||
- Refreshed PWA icons (72px through 512px)
|
||||
|
||||
### Changed
|
||||
- **Dashboard Layout**: Reorganized to include schedule information panel
|
||||
- **Base Template**: Updated to use new Terra-View logos with theme-aware switching
|
||||
|
||||
## [0.5.0] - 2026-01-23
|
||||
|
||||
_Note: This version was not formally released; changes were included in v0.5.1._
|
||||
|
||||
## [0.4.4] - 2026-01-23
|
||||
|
||||
### Added
|
||||
- **Recurring schedules**: New scheduler service, recurring schedule APIs, and schedule templates (calendar/interval/list).
|
||||
- **Alerts UI + backend**: Alerting service plus dropdown/list templates for surfacing notifications.
|
||||
- **Report templates + viewers**: CRUD API for report templates, report preview screen, and RND file viewer.
|
||||
- **SLM tooling**: SLM settings modal and SLM project report generator workflow.
|
||||
|
||||
### Changed
|
||||
- **Project data management**: Unified files view, refreshed FTP browser, and new project header/templates for file/session/unit/assignment lists.
|
||||
- **Device/SLM sync**: Standardized SLM device types and tightened SLMM sync paths.
|
||||
- **Docs/scripts**: Cleanup pass and expanded device-type documentation.
|
||||
|
||||
### Fixed
|
||||
- **Scheduler actions**: Strict command definitions so actions run reliably.
|
||||
- **Project view title**: Resolved JSON string rendering in project headers.
|
||||
|
||||
## [0.4.3] - 2026-01-14
|
||||
|
||||
### Added
|
||||
@@ -361,6 +711,11 @@ No database migration required for v0.4.0. All new features use existing databas
|
||||
- Photo management per unit
|
||||
- Automated status categorization (OK/Pending/Missing)
|
||||
|
||||
[0.7.0]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.6.1...v0.7.0
|
||||
[0.6.0]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.5.1...v0.6.0
|
||||
[0.5.1]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.5.0...v0.5.1
|
||||
[0.5.0]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.4.4...v0.5.0
|
||||
[0.4.4]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.4.3...v0.4.4
|
||||
[0.4.3]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.4.2...v0.4.3
|
||||
[0.4.2]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.4.1...v0.4.2
|
||||
[0.4.1]: https://github.com/serversdwn/seismo-fleet-manager/compare/v0.4.0...v0.4.1
|
||||
|
||||
@@ -1,5 +1,9 @@
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Build number for dev builds (injected via --build-arg)
|
||||
ARG BUILD_NUMBER=0
|
||||
ENV BUILD_NUMBER=${BUILD_NUMBER}
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Seismo Fleet Manager v0.4.3
|
||||
# Terra-View v0.10.0
|
||||
Backend API and HTMX-powered web interface for managing a mixed fleet of seismographs and field modems. Track deployments, monitor health in real time, merge roster intent with incoming telemetry, and control your fleet through a unified database and dashboard.
|
||||
|
||||
## Features
|
||||
@@ -308,7 +308,7 @@ print(response.json())
|
||||
|-------|------|-------------|
|
||||
| id | string | Unit identifier (primary key) |
|
||||
| unit_type | string | Hardware model name (default: `series3`) |
|
||||
| device_type | string | `seismograph` or `modem` discriminator |
|
||||
| device_type | string | Device type: `"seismograph"`, `"modem"`, or `"slm"` (sound level meter) |
|
||||
| deployed | boolean | Whether the unit is in the field |
|
||||
| retired | boolean | Removes the unit from deployments but preserves history |
|
||||
| note | string | Notes about the unit |
|
||||
@@ -334,6 +334,39 @@ print(response.json())
|
||||
| phone_number | string | Cellular number for the modem |
|
||||
| hardware_model | string | Modem hardware reference |
|
||||
|
||||
**Sound Level Meter (SLM) fields**
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| slm_host | string | Direct IP address for SLM (if not using modem) |
|
||||
| slm_tcp_port | integer | TCP control port (default: 2255) |
|
||||
| slm_ftp_port | integer | FTP file transfer port (default: 21) |
|
||||
| slm_model | string | Device model (NL-43, NL-53) |
|
||||
| slm_serial_number | string | Manufacturer serial number |
|
||||
| slm_frequency_weighting | string | Frequency weighting setting (A, C, Z) |
|
||||
| slm_time_weighting | string | Time weighting setting (F=Fast, S=Slow) |
|
||||
| slm_measurement_range | string | Measurement range setting |
|
||||
| slm_last_check | datetime | Last status check timestamp |
|
||||
| deployed_with_modem_id | string | Modem pairing (shared with seismographs) |
|
||||
|
||||
### Device Type Schema
|
||||
|
||||
Terra-View supports three device types with the following standardized `device_type` values:
|
||||
|
||||
- **`"seismograph"`** (default) - Seismic monitoring devices (Series 3, Series 4, Micromate)
|
||||
- Uses: calibration dates, modem pairing
|
||||
- Examples: BE1234, UM12345 (Series 3/4 units)
|
||||
|
||||
- **`"modem"`** - Field modems and network equipment
|
||||
- Uses: IP address, phone number, hardware model
|
||||
- Examples: MDM001, MODEM-2025-01
|
||||
|
||||
- **`"slm"`** - Sound level meters (Rion NL-43/NL-53)
|
||||
- Uses: TCP/FTP configuration, measurement settings, modem pairing
|
||||
- Examples: SLM-43-01, NL43-001
|
||||
|
||||
**Important**: All `device_type` values must be lowercase. The legacy value `"sound_level_meter"` has been deprecated in favor of the shorter `"slm"`. Run `backend/migrate_standardize_device_types.py` to update existing databases.
|
||||
|
||||
### Emitter Table (Device Check-ins)
|
||||
|
||||
| Field | Type | Description |
|
||||
@@ -463,6 +496,47 @@ docker compose down -v
|
||||
|
||||
## Release Highlights
|
||||
|
||||
### v0.10.0 — 2026-05-14
|
||||
- **SFM Integration**: terra-view now consumes events from the SFM (Seismograph Field Module) backend in real time, with a fleet-wide events page at `/sfm`, per-unit attribution against project assignment windows, and a project-level vibration roll-up that uses SFM data as the single source of truth.
|
||||
- **SFM-Primary Seismograph Status**: Deployed seismograph status (OK/Pending/Missing) now flows from SFM event forwards first; the watcher heartbeat stays as a transparent backup. Each unit's active table row shows a small `SFM` or `HB` badge so operators can see at a glance which signal is currently driving the status.
|
||||
- **Dashboard Rework**: Top row reordered to Recent Alerts → Recent Call-Ins (double-wide) → Fleet Summary. Today's Schedule moves to a horizontal collapsible card below the Fleet Map, auto-expanding only when there's a pending action. Recent Call-Ins now sources from SFM event forwards instead of the legacy watcher-heartbeat endpoint.
|
||||
- **Event Detail Modal**: Click any event anywhere in the app to open a rich detail modal showing peak particle velocity per channel, microphone dB(L), sensor self-check results, device/recording metadata, and download buttons for the original Blastware binary and sidecar JSON. Includes an inline JSON viewer with one-click copy.
|
||||
- **Sortable Events Tables**: Every events table (project events, unit-detail events, fleet-wide /sfm) now supports clickable column-header sorting with directional indicators. Defaults to newest-first.
|
||||
- **Events Attribution & Backfill**: Each SFM event is automatically attributed to a project/location based on `UnitAssignment` time windows. Unattributed events get a diagnostic showing the nearest assignment and a delta-days gap. The metadata-backfill tool in `/tools` scans operator-typed project/sensor-location strings in event sidecars and clusters them via fuzzy matching to propose new assignment retroactives.
|
||||
- **Projects Tools**: New `/tools` workflow hub consolidates Pair Devices, Project Tidy (fuzzy-detect + merge duplicate projects), Metadata Backfill, Reports, and Swap Detection (placeholder).
|
||||
- **Sidebar Reorganization**: Devices → Projects → Events → Tools → Job Planner → Settings. Devices is now a single entry with internal tabs (All Devices / Seismographs / Sound Level Meters / Modems / Pair Devices).
|
||||
- **Developer → SFM Admin**: New `/admin/sfm` page surfacing SFM health, per-unit roll-up from `/db/units`, recent-events table with forwarding latency (so operators can spot stale watcher forwards), stale-table counts, and a raw API tester. Companion `/admin/slmm` page covers SLMM health + raw API.
|
||||
- **"Overall Peak" excludes False Triggers**: The project-level Overall Peak KPI tile now excludes events flagged as false triggers — operators see the highest real event, not the biggest sensor glitch.
|
||||
- **Synology Deployment Doc**: New `docs/SYNOLOGY_DEPLOYMENT.md` covers migrating the stack to an always-on office NAS, including phased rollout, data rsync, watcher repoint, external-access (Tailscale or reverse-proxy), and rollback plan.
|
||||
|
||||
### v0.8.0 — 2026-03-18
|
||||
- **Watcher Manager**: Admin page for monitoring field watcher agents with live status cards, log tails, and one-click update triggering
|
||||
- **Watcher Status Fix**: Agent status now reflects heartbeat connectivity (missing if not heard from in >60 min) rather than unit-level data staleness
|
||||
- **Live Refresh**: Watcher Manager surgically patches status, last-seen, and pending indicators every 30s without a full page reload
|
||||
|
||||
### v0.7.0 — 2026-03-07
|
||||
- **Project Status Management**: On-hold and archived project states with automatic cancellation of pending actions
|
||||
- **Manual SD Card Upload**: Upload offline NRL/SLM data directly from SD card (ZIP or multi-file); auto-creates monitoring sessions from `.rnh` metadata
|
||||
- **Combined Report Wizard**: Multi-session Excel report generation with location grouping, period type filtering, and ZIP download
|
||||
- **NL32 Support**: Report generator and web viewer now handle NL32 measurement data
|
||||
- **Chart Preview**: Live chart preview in the report generator matching final output styling
|
||||
- **Standalone SLM Mode**: SLMs can now be configured without a paired modem (direct IP)
|
||||
- **Vibration Project Isolation**: Vibration project views no longer show SLM-specific tabs
|
||||
- **MonitoringSession Rename**: `RecordingSession` renamed to `MonitoringSession` throughout; run migration before deploying
|
||||
|
||||
### v0.6.1 — 2026-02-16
|
||||
- **One-Off Recording Schedules**: Schedule single recordings with specific start/end datetimes
|
||||
- **Bidirectional Pairing Sync**: Device-modem pairing now updates both sides automatically
|
||||
- **Scheduler Timezone Fix**: One-off schedule times use local time instead of UTC
|
||||
|
||||
### v0.6.0 — 2026-02-06
|
||||
- **Calendar & Reservation Mode**: Fleet calendar view with device deployment scheduling and reservation system
|
||||
- **Device Pairing Interface**: New `/pair-devices` page with two-column layout for linking recorders with modems, fuzzy-search, and visual pairing workflow
|
||||
- **Calibration UX Overhaul**: Users now set date of previous calibration (not expiry); seismograph list enhanced with color-coded calibration status, filtering, and sorting
|
||||
- **Modem Dashboard**: Model number as dedicated config, modem login links, list view format, and pairing options accessible from modem page
|
||||
- **SLMM Improvements**: Device control lock prevents command flooding, fixed polling intervals and scheduled downloads
|
||||
- **UI Polish**: Tab state persists in URL hash, settings save without reload, scheduler changes cascade to events, fixed mobile type display
|
||||
|
||||
### v0.4.3 — 2026-01-14
|
||||
- **Sound Level Meter workflow**: Roster manager surfaces SLM metadata, supports rename actions, and adds return-to-project navigation plus schedule/unit templates for project planning.
|
||||
- **Project insight panels**: Project dashboards now expose file and session lists so teams can see what each project stores before diving into units.
|
||||
@@ -538,9 +612,35 @@ MIT
|
||||
|
||||
## Version
|
||||
|
||||
**Current: 0.4.3** — SLM roster/project view refresh, project insight panels, FTP browser folder downloads, and SLMM sync (2026-01-14)
|
||||
**Current: 0.10.0** — SFM integration, SFM-primary seismograph status, dashboard rework, sortable events tables, event detail modal, /admin/sfm + /admin/slmm diagnostic pages, Tools workflow hub (2026-05-14)
|
||||
|
||||
Previous: 0.4.2 — SLM configuration interface with TCP/FTP controls, modem diagnostics, and dashboard endpoints for Sound Level Meters (2026-01-05)
|
||||
Previous: 0.9.4 — Modular project types, deleted project management, swap modal search, roster auto-refresh fix (2026-04-06)
|
||||
|
||||
0.9.3 — Monitoring session detail page, configurable period windows, vibration project redesign, modem assignment on locations (2026-03-28)
|
||||
|
||||
0.9.2 — Deployment records, allocated status, quick-info unit modal, inline seismograph editing (2026-03-27)
|
||||
|
||||
0.9.1 — Fix location slots not persisting on save/reload (2026-03-20)
|
||||
|
||||
0.9.0 — Job Planner redesign, monitoring locations, estimated units, smart color picker, calendar bar tooltips, toast notifications (2026-03-19)
|
||||
|
||||
0.8.0 — Watcher Manager admin page, live agent status refresh, watcher connectivity-based status (2026-03-18)
|
||||
|
||||
0.7.1 — Out-for-calibration status, reservation modal, migration fixes (2026-03-12)
|
||||
|
||||
0.7.0 — Project status management, manual SD card upload, combined report wizard, NL32 support, MonitoringSession rename (2026-03-07)
|
||||
|
||||
0.6.1 — One-off recording schedules, bidirectional pairing sync, scheduler timezone fix (2026-02-16)
|
||||
|
||||
0.6.0 — Calendar & reservation mode, device pairing interface, calibration UX overhaul, modem dashboard enhancements (2026-02-06)
|
||||
|
||||
0.5.1 — Dashboard schedule view with today's actions panel, new Terra-View branding and logo rework (2026-01-27)
|
||||
|
||||
0.4.4 — Recurring schedules, alerting UI, report templates + RND viewer, and SLM workflow polish (2026-01-23)
|
||||
|
||||
0.4.3 — SLM roster/project view refresh, project insight panels, FTP browser folder downloads, and SLMM sync (2026-01-14)
|
||||
|
||||
0.4.2 — SLM configuration interface with TCP/FTP controls, modem diagnostics, and dashboard endpoints for Sound Level Meters (2026-01-05)
|
||||
|
||||
0.4.1 — Sound Level Meter integration with full management UI for SLM units (2026-01-05)
|
||||
|
||||
|
||||
|
After Width: | Height: | Size: 36 KiB |
@@ -18,7 +18,7 @@ from backend.models import (
|
||||
MonitoringLocation,
|
||||
UnitAssignment,
|
||||
ScheduledAction,
|
||||
RecordingSession,
|
||||
MonitoringSession,
|
||||
DataFile,
|
||||
)
|
||||
from datetime import datetime
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import os
|
||||
import logging
|
||||
from fastapi import FastAPI, Request, Depends
|
||||
from fastapi import FastAPI, Request, Depends, HTTPException
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.templating import Jinja2Templates
|
||||
@@ -18,9 +18,10 @@ logging.basicConfig(
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
from backend.database import engine, Base, get_db
|
||||
from backend.routers import roster, units, photos, roster_edit, roster_rename, dashboard, dashboard_tabs, activity, slmm, slm_ui, slm_dashboard, seismo_dashboard, projects, project_locations, scheduler
|
||||
from backend.routers import roster, units, photos, roster_edit, roster_rename, dashboard, dashboard_tabs, activity, slmm, slm_ui, slm_dashboard, seismo_dashboard, sfm, projects, project_locations, scheduler, modem_dashboard
|
||||
from backend.services.snapshot import emit_status_snapshot
|
||||
from backend.models import IgnoredUnit
|
||||
from backend.utils.timezone import get_user_timezone
|
||||
|
||||
# Create database tables
|
||||
Base.metadata.create_all(bind=engine)
|
||||
@@ -29,7 +30,11 @@ Base.metadata.create_all(bind=engine)
|
||||
ENVIRONMENT = os.getenv("ENVIRONMENT", "production")
|
||||
|
||||
# Initialize FastAPI app
|
||||
VERSION = "0.4.3"
|
||||
VERSION = "0.10.0"
|
||||
if ENVIRONMENT == "development":
|
||||
_build = os.getenv("BUILD_NUMBER", "0")
|
||||
if _build and _build != "0":
|
||||
VERSION = f"{VERSION}-{_build}"
|
||||
app = FastAPI(
|
||||
title="Seismo Fleet Manager",
|
||||
description="Backend API for managing seismograph fleet status",
|
||||
@@ -58,8 +63,8 @@ app.add_middleware(
|
||||
# Mount static files
|
||||
app.mount("/static", StaticFiles(directory="backend/static"), name="static")
|
||||
|
||||
# Setup Jinja2 templates
|
||||
templates = Jinja2Templates(directory="templates")
|
||||
# Use shared templates configuration with timezone filters
|
||||
from backend.templates_config import templates
|
||||
|
||||
# Add custom context processor to inject environment variable into all templates
|
||||
@app.middleware("http")
|
||||
@@ -92,17 +97,50 @@ app.include_router(slmm.router)
|
||||
app.include_router(slm_ui.router)
|
||||
app.include_router(slm_dashboard.router)
|
||||
app.include_router(seismo_dashboard.router)
|
||||
app.include_router(sfm.router)
|
||||
app.include_router(modem_dashboard.router)
|
||||
|
||||
from backend.routers import settings
|
||||
app.include_router(settings.router)
|
||||
|
||||
from backend.routers import watcher_manager
|
||||
app.include_router(watcher_manager.router)
|
||||
|
||||
from backend.routers import admin_modules
|
||||
app.include_router(admin_modules.router)
|
||||
|
||||
# Projects system routers
|
||||
app.include_router(projects.router)
|
||||
app.include_router(project_locations.router)
|
||||
app.include_router(scheduler.router)
|
||||
|
||||
# Start scheduler service on application startup
|
||||
# Report templates router
|
||||
from backend.routers import report_templates
|
||||
app.include_router(report_templates.router)
|
||||
|
||||
# Metadata-backfill admin router (Phase 5a)
|
||||
from backend.routers import metadata_backfill
|
||||
app.include_router(metadata_backfill.router)
|
||||
|
||||
# Alerts router
|
||||
from backend.routers import alerts
|
||||
app.include_router(alerts.router)
|
||||
|
||||
# Recurring schedules router
|
||||
from backend.routers import recurring_schedules
|
||||
app.include_router(recurring_schedules.router)
|
||||
|
||||
# Fleet Calendar router
|
||||
from backend.routers import fleet_calendar
|
||||
app.include_router(fleet_calendar.router)
|
||||
|
||||
# Deployment Records router
|
||||
from backend.routers import deployments
|
||||
app.include_router(deployments.router)
|
||||
|
||||
# Start scheduler service and device status monitor on application startup
|
||||
from backend.services.scheduler import start_scheduler, stop_scheduler
|
||||
from backend.services.device_status_monitor import start_device_status_monitor, stop_device_status_monitor
|
||||
|
||||
@app.on_event("startup")
|
||||
async def startup_event():
|
||||
@@ -111,9 +149,17 @@ async def startup_event():
|
||||
await start_scheduler()
|
||||
logger.info("Scheduler service started")
|
||||
|
||||
logger.info("Starting device status monitor...")
|
||||
await start_device_status_monitor()
|
||||
logger.info("Device status monitor started")
|
||||
|
||||
@app.on_event("shutdown")
|
||||
def shutdown_event():
|
||||
"""Clean up services on app shutdown"""
|
||||
logger.info("Stopping device status monitor...")
|
||||
stop_device_status_monitor()
|
||||
logger.info("Device status monitor stopped")
|
||||
|
||||
logger.info("Stopping scheduler service...")
|
||||
stop_scheduler()
|
||||
logger.info("Scheduler service stopped")
|
||||
@@ -195,6 +241,101 @@ async def seismographs_page(request: Request):
|
||||
return templates.TemplateResponse("seismographs.html", {"request": request})
|
||||
|
||||
|
||||
@app.get("/sfm", response_class=HTMLResponse)
|
||||
async def sfm_page(request: Request):
|
||||
"""SFM live event data and device control dashboard"""
|
||||
return templates.TemplateResponse("sfm.html", {"request": request})
|
||||
|
||||
|
||||
@app.get("/settings/developer/metadata-backfill", response_class=HTMLResponse)
|
||||
async def metadata_backfill_wizard_page(request: Request):
|
||||
"""Wizard for auto-creating projects/locations/assignments from
|
||||
operator-typed BW event metadata (Phase 5a)."""
|
||||
return templates.TemplateResponse("admin/metadata_backfill.html", {"request": request})
|
||||
|
||||
|
||||
@app.get("/settings/developer/project-tidy", response_class=HTMLResponse)
|
||||
async def project_tidy_page(request: Request):
|
||||
"""Tidy duplicate-looking projects: detect by fuzzy name match, merge
|
||||
by clicking through pairs (Phase 5b)."""
|
||||
return templates.TemplateResponse("admin/project_tidy.html", {"request": request})
|
||||
|
||||
|
||||
@app.get("/tools", response_class=HTMLResponse)
|
||||
async def tools_page(request: Request):
|
||||
"""Tools / workflow hub. Active operator workflows (device pairing,
|
||||
project tidy, metadata backfill, future swap detection, report
|
||||
generators) all live here in card form."""
|
||||
return templates.TemplateResponse("tools.html", {"request": request})
|
||||
|
||||
|
||||
@app.get("/modems", response_class=HTMLResponse)
|
||||
async def modems_page(request: Request):
|
||||
"""Field modems management dashboard"""
|
||||
return templates.TemplateResponse("modems.html", {"request": request})
|
||||
|
||||
|
||||
@app.get("/pair-devices", response_class=HTMLResponse)
|
||||
async def pair_devices_page(request: Request, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Device pairing page - two-column layout for pairing recorders with modems.
|
||||
"""
|
||||
from backend.models import RosterUnit
|
||||
|
||||
# Get all non-retired recorders (seismographs and SLMs)
|
||||
recorders = db.query(RosterUnit).filter(
|
||||
RosterUnit.retired == False,
|
||||
RosterUnit.device_type.in_(["seismograph", "slm", None]) # None defaults to seismograph
|
||||
).order_by(RosterUnit.id).all()
|
||||
|
||||
# Get all non-retired modems
|
||||
modems = db.query(RosterUnit).filter(
|
||||
RosterUnit.retired == False,
|
||||
RosterUnit.device_type == "modem"
|
||||
).order_by(RosterUnit.id).all()
|
||||
|
||||
# Build existing pairings list
|
||||
pairings = []
|
||||
for recorder in recorders:
|
||||
if recorder.deployed_with_modem_id:
|
||||
modem = next((m for m in modems if m.id == recorder.deployed_with_modem_id), None)
|
||||
pairings.append({
|
||||
"recorder_id": recorder.id,
|
||||
"recorder_type": (recorder.device_type or "seismograph").upper(),
|
||||
"modem_id": recorder.deployed_with_modem_id,
|
||||
"modem_ip": modem.ip_address if modem else None
|
||||
})
|
||||
|
||||
# Convert to dicts for template
|
||||
recorders_data = [
|
||||
{
|
||||
"id": r.id,
|
||||
"device_type": r.device_type or "seismograph",
|
||||
"deployed": r.deployed,
|
||||
"deployed_with_modem_id": r.deployed_with_modem_id
|
||||
}
|
||||
for r in recorders
|
||||
]
|
||||
|
||||
modems_data = [
|
||||
{
|
||||
"id": m.id,
|
||||
"deployed": m.deployed,
|
||||
"deployed_with_unit_id": m.deployed_with_unit_id,
|
||||
"ip_address": m.ip_address,
|
||||
"phone_number": m.phone_number
|
||||
}
|
||||
for m in modems
|
||||
]
|
||||
|
||||
return templates.TemplateResponse("pair_devices.html", {
|
||||
"request": request,
|
||||
"recorders": recorders_data,
|
||||
"modems": modems_data,
|
||||
"pairings": pairings
|
||||
})
|
||||
|
||||
|
||||
@app.get("/projects", response_class=HTMLResponse)
|
||||
async def projects_page(request: Request):
|
||||
"""Projects management and overview"""
|
||||
@@ -218,7 +359,7 @@ async def nrl_detail_page(
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""NRL (Noise Recording Location) detail page with tabs"""
|
||||
from backend.models import Project, MonitoringLocation, UnitAssignment, RosterUnit, RecordingSession, DataFile
|
||||
from backend.models import Project, MonitoringLocation, UnitAssignment, RosterUnit, MonitoringSession, DataFile
|
||||
from sqlalchemy import and_
|
||||
|
||||
# Get project
|
||||
@@ -250,27 +391,40 @@ async def nrl_detail_page(
|
||||
).first()
|
||||
|
||||
assigned_unit = None
|
||||
assigned_modem = None
|
||||
if assignment:
|
||||
assigned_unit = db.query(RosterUnit).filter_by(id=assignment.unit_id).first()
|
||||
if assigned_unit and assigned_unit.deployed_with_modem_id:
|
||||
assigned_modem = db.query(RosterUnit).filter_by(id=assigned_unit.deployed_with_modem_id).first()
|
||||
|
||||
# Get session count
|
||||
session_count = db.query(RecordingSession).filter_by(location_id=location_id).count()
|
||||
session_count = db.query(MonitoringSession).filter_by(location_id=location_id).count()
|
||||
|
||||
# Get file count (DataFile links to session, not directly to location)
|
||||
file_count = db.query(DataFile).join(
|
||||
RecordingSession,
|
||||
DataFile.session_id == RecordingSession.id
|
||||
).filter(RecordingSession.location_id == location_id).count()
|
||||
MonitoringSession,
|
||||
DataFile.session_id == MonitoringSession.id
|
||||
).filter(MonitoringSession.location_id == location_id).count()
|
||||
|
||||
# Check for active session
|
||||
active_session = db.query(RecordingSession).filter(
|
||||
active_session = db.query(MonitoringSession).filter(
|
||||
and_(
|
||||
RecordingSession.location_id == location_id,
|
||||
RecordingSession.status == "recording"
|
||||
MonitoringSession.location_id == location_id,
|
||||
MonitoringSession.status == "recording"
|
||||
)
|
||||
).first()
|
||||
|
||||
return templates.TemplateResponse("nrl_detail.html", {
|
||||
# Parse connection_mode from location_metadata JSON
|
||||
import json as _json
|
||||
connection_mode = "connected"
|
||||
try:
|
||||
meta = _json.loads(location.location_metadata or "{}")
|
||||
connection_mode = meta.get("connection_mode", "connected")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
template = "vibration_location_detail.html" if location.location_type == "vibration" else "nrl_detail.html"
|
||||
return templates.TemplateResponse(template, {
|
||||
"request": request,
|
||||
"project_id": project_id,
|
||||
"location_id": location_id,
|
||||
@@ -278,9 +432,11 @@ async def nrl_detail_page(
|
||||
"location": location,
|
||||
"assignment": assignment,
|
||||
"assigned_unit": assigned_unit,
|
||||
"assigned_modem": assigned_modem,
|
||||
"session_count": session_count,
|
||||
"file_count": file_count,
|
||||
"active_session": active_session,
|
||||
"connection_mode": connection_mode,
|
||||
})
|
||||
|
||||
|
||||
@@ -550,6 +706,7 @@ async def devices_all_partial(request: Request):
|
||||
"last_seen": unit_data.get("last", "Never"),
|
||||
"deployed": True,
|
||||
"retired": False,
|
||||
"out_for_calibration": False,
|
||||
"ignored": False,
|
||||
"note": unit_data.get("note", ""),
|
||||
"device_type": unit_data.get("device_type", "seismograph"),
|
||||
@@ -559,6 +716,7 @@ async def devices_all_partial(request: Request):
|
||||
"last_calibrated": unit_data.get("last_calibrated"),
|
||||
"next_calibration_due": unit_data.get("next_calibration_due"),
|
||||
"deployed_with_modem_id": unit_data.get("deployed_with_modem_id"),
|
||||
"deployed_with_unit_id": unit_data.get("deployed_with_unit_id"),
|
||||
"ip_address": unit_data.get("ip_address"),
|
||||
"phone_number": unit_data.get("phone_number"),
|
||||
"hardware_model": unit_data.get("hardware_model"),
|
||||
@@ -573,6 +731,7 @@ async def devices_all_partial(request: Request):
|
||||
"last_seen": unit_data.get("last", "Never"),
|
||||
"deployed": False,
|
||||
"retired": False,
|
||||
"out_for_calibration": False,
|
||||
"ignored": False,
|
||||
"note": unit_data.get("note", ""),
|
||||
"device_type": unit_data.get("device_type", "seismograph"),
|
||||
@@ -582,6 +741,59 @@ async def devices_all_partial(request: Request):
|
||||
"last_calibrated": unit_data.get("last_calibrated"),
|
||||
"next_calibration_due": unit_data.get("next_calibration_due"),
|
||||
"deployed_with_modem_id": unit_data.get("deployed_with_modem_id"),
|
||||
"deployed_with_unit_id": unit_data.get("deployed_with_unit_id"),
|
||||
"ip_address": unit_data.get("ip_address"),
|
||||
"phone_number": unit_data.get("phone_number"),
|
||||
"hardware_model": unit_data.get("hardware_model"),
|
||||
})
|
||||
|
||||
# Add allocated units
|
||||
for unit_id, unit_data in snapshot.get("allocated", {}).items():
|
||||
units_list.append({
|
||||
"id": unit_id,
|
||||
"status": "Allocated",
|
||||
"age": "N/A",
|
||||
"last_seen": "N/A",
|
||||
"deployed": False,
|
||||
"retired": False,
|
||||
"out_for_calibration": False,
|
||||
"allocated": True,
|
||||
"allocated_to_project_id": unit_data.get("allocated_to_project_id", ""),
|
||||
"ignored": False,
|
||||
"note": unit_data.get("note", ""),
|
||||
"device_type": unit_data.get("device_type", "seismograph"),
|
||||
"address": unit_data.get("address", ""),
|
||||
"coordinates": unit_data.get("coordinates", ""),
|
||||
"project_id": unit_data.get("project_id", ""),
|
||||
"last_calibrated": unit_data.get("last_calibrated"),
|
||||
"next_calibration_due": unit_data.get("next_calibration_due"),
|
||||
"deployed_with_modem_id": unit_data.get("deployed_with_modem_id"),
|
||||
"deployed_with_unit_id": unit_data.get("deployed_with_unit_id"),
|
||||
"ip_address": unit_data.get("ip_address"),
|
||||
"phone_number": unit_data.get("phone_number"),
|
||||
"hardware_model": unit_data.get("hardware_model"),
|
||||
})
|
||||
|
||||
# Add out-for-calibration units
|
||||
for unit_id, unit_data in snapshot["out_for_calibration"].items():
|
||||
units_list.append({
|
||||
"id": unit_id,
|
||||
"status": "Out for Calibration",
|
||||
"age": "N/A",
|
||||
"last_seen": "N/A",
|
||||
"deployed": False,
|
||||
"retired": False,
|
||||
"out_for_calibration": True,
|
||||
"ignored": False,
|
||||
"note": unit_data.get("note", ""),
|
||||
"device_type": unit_data.get("device_type", "seismograph"),
|
||||
"address": unit_data.get("address", ""),
|
||||
"coordinates": unit_data.get("coordinates", ""),
|
||||
"project_id": unit_data.get("project_id", ""),
|
||||
"last_calibrated": unit_data.get("last_calibrated"),
|
||||
"next_calibration_due": unit_data.get("next_calibration_due"),
|
||||
"deployed_with_modem_id": unit_data.get("deployed_with_modem_id"),
|
||||
"deployed_with_unit_id": unit_data.get("deployed_with_unit_id"),
|
||||
"ip_address": unit_data.get("ip_address"),
|
||||
"phone_number": unit_data.get("phone_number"),
|
||||
"hardware_model": unit_data.get("hardware_model"),
|
||||
@@ -596,6 +808,7 @@ async def devices_all_partial(request: Request):
|
||||
"last_seen": "N/A",
|
||||
"deployed": False,
|
||||
"retired": True,
|
||||
"out_for_calibration": False,
|
||||
"ignored": False,
|
||||
"note": unit_data.get("note", ""),
|
||||
"device_type": unit_data.get("device_type", "seismograph"),
|
||||
@@ -605,6 +818,7 @@ async def devices_all_partial(request: Request):
|
||||
"last_calibrated": unit_data.get("last_calibrated"),
|
||||
"next_calibration_due": unit_data.get("next_calibration_due"),
|
||||
"deployed_with_modem_id": unit_data.get("deployed_with_modem_id"),
|
||||
"deployed_with_unit_id": unit_data.get("deployed_with_unit_id"),
|
||||
"ip_address": unit_data.get("ip_address"),
|
||||
"phone_number": unit_data.get("phone_number"),
|
||||
"hardware_model": unit_data.get("hardware_model"),
|
||||
@@ -619,6 +833,7 @@ async def devices_all_partial(request: Request):
|
||||
"last_seen": "N/A",
|
||||
"deployed": False,
|
||||
"retired": False,
|
||||
"out_for_calibration": False,
|
||||
"ignored": True,
|
||||
"note": unit_data.get("note", unit_data.get("reason", "")),
|
||||
"device_type": unit_data.get("device_type", "unknown"),
|
||||
@@ -628,6 +843,7 @@ async def devices_all_partial(request: Request):
|
||||
"last_calibrated": None,
|
||||
"next_calibration_due": None,
|
||||
"deployed_with_modem_id": None,
|
||||
"deployed_with_unit_id": None,
|
||||
"ip_address": None,
|
||||
"phone_number": None,
|
||||
"hardware_model": None,
|
||||
@@ -635,22 +851,27 @@ async def devices_all_partial(request: Request):
|
||||
|
||||
# Sort by status category, then by ID
|
||||
def sort_key(unit):
|
||||
# Priority: deployed (active) -> benched -> retired -> ignored
|
||||
# Priority: deployed (active) -> allocated -> benched -> out_for_calibration -> retired -> ignored
|
||||
if unit["deployed"]:
|
||||
return (0, unit["id"])
|
||||
elif not unit["retired"] and not unit["ignored"]:
|
||||
elif unit.get("allocated"):
|
||||
return (1, unit["id"])
|
||||
elif unit["retired"]:
|
||||
elif not unit["retired"] and not unit["out_for_calibration"] and not unit["ignored"]:
|
||||
return (2, unit["id"])
|
||||
else:
|
||||
elif unit["out_for_calibration"]:
|
||||
return (3, unit["id"])
|
||||
elif unit["retired"]:
|
||||
return (4, unit["id"])
|
||||
else:
|
||||
return (5, unit["id"])
|
||||
|
||||
units_list.sort(key=sort_key)
|
||||
|
||||
return templates.TemplateResponse("partials/devices_table.html", {
|
||||
"request": request,
|
||||
"units": units_list,
|
||||
"timestamp": datetime.now().strftime("%H:%M:%S")
|
||||
"timestamp": datetime.now().strftime("%H:%M:%S"),
|
||||
"user_timezone": get_user_timezone()
|
||||
})
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
"""
|
||||
Migration: Add allocated and allocated_to_project_id columns to roster table.
|
||||
Run once: python backend/migrate_add_allocated.py
|
||||
"""
|
||||
import sqlite3
|
||||
import os
|
||||
|
||||
DB_PATH = os.path.join(os.path.dirname(__file__), '..', 'data', 'seismo_fleet.db')
|
||||
|
||||
def run():
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
cur = conn.cursor()
|
||||
|
||||
# Check existing columns
|
||||
cur.execute("PRAGMA table_info(roster)")
|
||||
cols = {row[1] for row in cur.fetchall()}
|
||||
|
||||
if 'allocated' not in cols:
|
||||
cur.execute("ALTER TABLE roster ADD COLUMN allocated BOOLEAN DEFAULT 0 NOT NULL")
|
||||
print("Added column: allocated")
|
||||
else:
|
||||
print("Column already exists: allocated")
|
||||
|
||||
if 'allocated_to_project_id' not in cols:
|
||||
cur.execute("ALTER TABLE roster ADD COLUMN allocated_to_project_id VARCHAR")
|
||||
print("Added column: allocated_to_project_id")
|
||||
else:
|
||||
print("Column already exists: allocated_to_project_id")
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
print("Migration complete.")
|
||||
|
||||
if __name__ == '__main__':
|
||||
run()
|
||||
@@ -0,0 +1,67 @@
|
||||
"""
|
||||
Migration: Add auto_increment_index column to recurring_schedules table
|
||||
|
||||
This migration adds the auto_increment_index column that controls whether
|
||||
the scheduler should automatically find an unused store index before starting
|
||||
a new measurement.
|
||||
|
||||
Run this script once to update existing databases:
|
||||
python -m backend.migrate_add_auto_increment_index
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import os
|
||||
|
||||
DB_PATH = "data/seismo_fleet.db"
|
||||
|
||||
|
||||
def migrate():
|
||||
"""Add auto_increment_index column to recurring_schedules table."""
|
||||
if not os.path.exists(DB_PATH):
|
||||
print(f"Database not found at {DB_PATH}")
|
||||
return False
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
|
||||
try:
|
||||
# Check if recurring_schedules table exists
|
||||
cursor.execute("""
|
||||
SELECT name FROM sqlite_master
|
||||
WHERE type='table' AND name='recurring_schedules'
|
||||
""")
|
||||
if not cursor.fetchone():
|
||||
print("recurring_schedules table does not exist yet. Will be created on app startup.")
|
||||
conn.close()
|
||||
return True
|
||||
|
||||
# Check if auto_increment_index column already exists
|
||||
cursor.execute("PRAGMA table_info(recurring_schedules)")
|
||||
columns = [row[1] for row in cursor.fetchall()]
|
||||
|
||||
if "auto_increment_index" in columns:
|
||||
print("auto_increment_index column already exists in recurring_schedules table.")
|
||||
conn.close()
|
||||
return True
|
||||
|
||||
# Add the column
|
||||
print("Adding auto_increment_index column to recurring_schedules table...")
|
||||
cursor.execute("""
|
||||
ALTER TABLE recurring_schedules
|
||||
ADD COLUMN auto_increment_index BOOLEAN DEFAULT 1
|
||||
""")
|
||||
conn.commit()
|
||||
print("Successfully added auto_increment_index column.")
|
||||
|
||||
conn.close()
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"Migration failed: {e}")
|
||||
conn.close()
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = migrate()
|
||||
exit(0 if success else 1)
|
||||
@@ -0,0 +1,79 @@
|
||||
"""
|
||||
Migration: Add deployment_records table.
|
||||
|
||||
Tracks each time a unit is sent to the field and returned.
|
||||
The active deployment is the row with actual_removal_date IS NULL.
|
||||
|
||||
Run once per database:
|
||||
python backend/migrate_add_deployment_records.py
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import os
|
||||
|
||||
DB_PATH = "./data/seismo_fleet.db"
|
||||
|
||||
|
||||
def migrate_database():
|
||||
if not os.path.exists(DB_PATH):
|
||||
print(f"Database not found at {DB_PATH}")
|
||||
return
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
|
||||
try:
|
||||
# Check if table already exists
|
||||
cursor.execute("""
|
||||
SELECT name FROM sqlite_master
|
||||
WHERE type='table' AND name='deployment_records'
|
||||
""")
|
||||
if cursor.fetchone():
|
||||
print("✓ deployment_records table already exists, skipping")
|
||||
return
|
||||
|
||||
print("Creating deployment_records table...")
|
||||
cursor.execute("""
|
||||
CREATE TABLE deployment_records (
|
||||
id TEXT PRIMARY KEY,
|
||||
unit_id TEXT NOT NULL,
|
||||
deployed_date DATE,
|
||||
estimated_removal_date DATE,
|
||||
actual_removal_date DATE,
|
||||
project_ref TEXT,
|
||||
project_id TEXT,
|
||||
location_name TEXT,
|
||||
notes TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
""")
|
||||
|
||||
cursor.execute("""
|
||||
CREATE INDEX idx_deployment_records_unit_id
|
||||
ON deployment_records(unit_id)
|
||||
""")
|
||||
cursor.execute("""
|
||||
CREATE INDEX idx_deployment_records_project_id
|
||||
ON deployment_records(project_id)
|
||||
""")
|
||||
# Index for finding active deployments quickly
|
||||
cursor.execute("""
|
||||
CREATE INDEX idx_deployment_records_active
|
||||
ON deployment_records(unit_id, actual_removal_date)
|
||||
""")
|
||||
|
||||
conn.commit()
|
||||
print("✓ deployment_records table created successfully")
|
||||
print("✓ Indexes created")
|
||||
|
||||
except Exception as e:
|
||||
conn.rollback()
|
||||
print(f"✗ Migration failed: {e}")
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate_database()
|
||||
@@ -0,0 +1,84 @@
|
||||
"""
|
||||
Migration script to add deployment_type and deployed_with_unit_id fields to roster table.
|
||||
|
||||
deployment_type: tracks what type of device a modem is deployed with:
|
||||
- "seismograph" - Modem is connected to a seismograph
|
||||
- "slm" - Modem is connected to a sound level meter
|
||||
- NULL/empty - Not assigned or unknown
|
||||
|
||||
deployed_with_unit_id: stores the ID of the seismograph/SLM this modem is deployed with
|
||||
(reverse relationship of deployed_with_modem_id)
|
||||
|
||||
Run this script once to migrate an existing database.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import os
|
||||
|
||||
# Database path
|
||||
DB_PATH = "./data/seismo_fleet.db"
|
||||
|
||||
|
||||
def migrate_database():
|
||||
"""Add deployment_type and deployed_with_unit_id columns to roster table"""
|
||||
|
||||
if not os.path.exists(DB_PATH):
|
||||
print(f"Database not found at {DB_PATH}")
|
||||
print("The database will be created automatically when you run the application.")
|
||||
return
|
||||
|
||||
print(f"Migrating database: {DB_PATH}")
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Check if roster table exists
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='roster'")
|
||||
table_exists = cursor.fetchone()
|
||||
|
||||
if not table_exists:
|
||||
print("Roster table does not exist yet - will be created when app runs")
|
||||
conn.close()
|
||||
return
|
||||
|
||||
# Check existing columns
|
||||
cursor.execute("PRAGMA table_info(roster)")
|
||||
columns = [col[1] for col in cursor.fetchall()]
|
||||
|
||||
try:
|
||||
# Add deployment_type if not exists
|
||||
if 'deployment_type' not in columns:
|
||||
print("Adding deployment_type column to roster table...")
|
||||
cursor.execute("ALTER TABLE roster ADD COLUMN deployment_type TEXT")
|
||||
print(" Added deployment_type column")
|
||||
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS ix_roster_deployment_type ON roster(deployment_type)")
|
||||
print(" Created index on deployment_type")
|
||||
else:
|
||||
print("deployment_type column already exists")
|
||||
|
||||
# Add deployed_with_unit_id if not exists
|
||||
if 'deployed_with_unit_id' not in columns:
|
||||
print("Adding deployed_with_unit_id column to roster table...")
|
||||
cursor.execute("ALTER TABLE roster ADD COLUMN deployed_with_unit_id TEXT")
|
||||
print(" Added deployed_with_unit_id column")
|
||||
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS ix_roster_deployed_with_unit_id ON roster(deployed_with_unit_id)")
|
||||
print(" Created index on deployed_with_unit_id")
|
||||
else:
|
||||
print("deployed_with_unit_id column already exists")
|
||||
|
||||
conn.commit()
|
||||
print("\nMigration completed successfully!")
|
||||
|
||||
except sqlite3.Error as e:
|
||||
print(f"\nError during migration: {e}")
|
||||
conn.rollback()
|
||||
raise
|
||||
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate_database()
|
||||
@@ -0,0 +1,62 @@
|
||||
"""
|
||||
Migration: Add estimated_units to job_reservations
|
||||
|
||||
Adds column:
|
||||
- job_reservations.estimated_units: Estimated number of units for the reservation (nullable integer)
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Default database path (matches production pattern)
|
||||
DB_PATH = "./data/seismo_fleet.db"
|
||||
|
||||
|
||||
def migrate(db_path: str):
|
||||
"""Run the migration."""
|
||||
print(f"Migrating database: {db_path}")
|
||||
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
|
||||
try:
|
||||
# Check if job_reservations table exists
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='job_reservations'")
|
||||
if not cursor.fetchone():
|
||||
print("job_reservations table does not exist. Skipping migration.")
|
||||
return
|
||||
|
||||
# Get existing columns in job_reservations
|
||||
cursor.execute("PRAGMA table_info(job_reservations)")
|
||||
existing_cols = {row[1] for row in cursor.fetchall()}
|
||||
|
||||
# Add estimated_units column if it doesn't exist
|
||||
if 'estimated_units' not in existing_cols:
|
||||
print("Adding estimated_units column to job_reservations...")
|
||||
cursor.execute("ALTER TABLE job_reservations ADD COLUMN estimated_units INTEGER")
|
||||
else:
|
||||
print("estimated_units column already exists. Skipping.")
|
||||
|
||||
conn.commit()
|
||||
print("Migration completed successfully!")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Migration failed: {e}")
|
||||
conn.rollback()
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
db_path = DB_PATH
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
db_path = sys.argv[1]
|
||||
|
||||
if not Path(db_path).exists():
|
||||
print(f"Database not found: {db_path}")
|
||||
sys.exit(1)
|
||||
|
||||
migrate(db_path)
|
||||
@@ -0,0 +1,103 @@
|
||||
"""
|
||||
Migration script to add job reservations for the Fleet Calendar feature.
|
||||
|
||||
This creates two tables:
|
||||
- job_reservations: Track future unit assignments for jobs/projects
|
||||
- job_reservation_units: Link specific units to reservations
|
||||
|
||||
Run this script once to migrate an existing database.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import os
|
||||
|
||||
# Database path
|
||||
DB_PATH = "./data/seismo_fleet.db"
|
||||
|
||||
|
||||
def migrate_database():
|
||||
"""Create the job_reservations and job_reservation_units tables"""
|
||||
|
||||
if not os.path.exists(DB_PATH):
|
||||
print(f"Database not found at {DB_PATH}")
|
||||
print("The database will be created automatically when you run the application.")
|
||||
return
|
||||
|
||||
print(f"Migrating database: {DB_PATH}")
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Check if job_reservations table already exists
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='job_reservations'")
|
||||
if cursor.fetchone():
|
||||
print("Migration already applied - job_reservations table exists")
|
||||
conn.close()
|
||||
return
|
||||
|
||||
print("Creating job_reservations table...")
|
||||
|
||||
try:
|
||||
# Create job_reservations table
|
||||
cursor.execute("""
|
||||
CREATE TABLE job_reservations (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
project_id TEXT,
|
||||
start_date DATE NOT NULL,
|
||||
end_date DATE NOT NULL,
|
||||
assignment_type TEXT NOT NULL DEFAULT 'quantity',
|
||||
device_type TEXT DEFAULT 'seismograph',
|
||||
quantity_needed INTEGER,
|
||||
notes TEXT,
|
||||
color TEXT DEFAULT '#3B82F6',
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
""")
|
||||
print(" Created job_reservations table")
|
||||
|
||||
# Create indexes for job_reservations
|
||||
cursor.execute("CREATE INDEX idx_job_reservations_project_id ON job_reservations(project_id)")
|
||||
print(" Created index on project_id")
|
||||
|
||||
cursor.execute("CREATE INDEX idx_job_reservations_dates ON job_reservations(start_date, end_date)")
|
||||
print(" Created index on dates")
|
||||
|
||||
# Create job_reservation_units table
|
||||
print("Creating job_reservation_units table...")
|
||||
cursor.execute("""
|
||||
CREATE TABLE job_reservation_units (
|
||||
id TEXT PRIMARY KEY,
|
||||
reservation_id TEXT NOT NULL,
|
||||
unit_id TEXT NOT NULL,
|
||||
assignment_source TEXT DEFAULT 'specific',
|
||||
assigned_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
FOREIGN KEY (reservation_id) REFERENCES job_reservations(id),
|
||||
FOREIGN KEY (unit_id) REFERENCES roster(id)
|
||||
)
|
||||
""")
|
||||
print(" Created job_reservation_units table")
|
||||
|
||||
# Create indexes for job_reservation_units
|
||||
cursor.execute("CREATE INDEX idx_job_reservation_units_reservation_id ON job_reservation_units(reservation_id)")
|
||||
print(" Created index on reservation_id")
|
||||
|
||||
cursor.execute("CREATE INDEX idx_job_reservation_units_unit_id ON job_reservation_units(unit_id)")
|
||||
print(" Created index on unit_id")
|
||||
|
||||
conn.commit()
|
||||
print("\nMigration completed successfully!")
|
||||
print("You can now use the Fleet Calendar to manage unit reservations.")
|
||||
|
||||
except sqlite3.Error as e:
|
||||
print(f"\nError during migration: {e}")
|
||||
conn.rollback()
|
||||
raise
|
||||
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate_database()
|
||||
@@ -0,0 +1,24 @@
|
||||
"""
|
||||
Migration: Add location_slots column to job_reservations table.
|
||||
Stores the full ordered slot list (including empty/unassigned slots) as JSON.
|
||||
Run once per database.
|
||||
"""
|
||||
import sqlite3
|
||||
import os
|
||||
|
||||
DB_PATH = os.environ.get("DB_PATH", "/app/data/seismo_fleet.db")
|
||||
|
||||
def run():
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
existing = [r[1] for r in cursor.execute("PRAGMA table_info(job_reservations)").fetchall()]
|
||||
if "location_slots" not in existing:
|
||||
cursor.execute("ALTER TABLE job_reservations ADD COLUMN location_slots TEXT")
|
||||
conn.commit()
|
||||
print("Added location_slots column to job_reservations.")
|
||||
else:
|
||||
print("location_slots column already exists, skipping.")
|
||||
conn.close()
|
||||
|
||||
if __name__ == "__main__":
|
||||
run()
|
||||
@@ -0,0 +1,94 @@
|
||||
"""
|
||||
Migration: add metadata-backfill support.
|
||||
|
||||
Adds:
|
||||
1. `unit_assignments.source` column (TEXT, default 'manual').
|
||||
Lets us audit which assignments were created by the metadata-backfill
|
||||
parser vs by a human, and bulk-undo parser actions if needed.
|
||||
|
||||
2. `metadata_backfill_decisions` table. Tracks operator decisions per
|
||||
cluster_id so the wizard remembers what's been skipped, what's
|
||||
been applied, and what's pending across re-scans.
|
||||
|
||||
Idempotent — safe to re-run.
|
||||
Non-destructive — adds only.
|
||||
|
||||
Run with:
|
||||
docker exec terra-view-terra-view-1 python3 /app/backend/migrate_add_metadata_backfill.py
|
||||
"""
|
||||
|
||||
import os
|
||||
import sqlite3
|
||||
|
||||
DB_PATH = "./data/seismo_fleet.db"
|
||||
|
||||
|
||||
def migrate_database():
|
||||
if not os.path.exists(DB_PATH):
|
||||
print(f"Database not found at {DB_PATH}")
|
||||
return
|
||||
|
||||
print(f"Migrating database: {DB_PATH}")
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
cur = conn.cursor()
|
||||
|
||||
# ── 1. unit_assignments.source column ──────────────────────────────────
|
||||
cur.execute("PRAGMA table_info(unit_assignments)")
|
||||
cols = {row[1] for row in cur.fetchall()}
|
||||
if "source" not in cols:
|
||||
print("Adding unit_assignments.source column (default 'manual') ...")
|
||||
cur.execute(
|
||||
"ALTER TABLE unit_assignments ADD COLUMN source TEXT DEFAULT 'manual'"
|
||||
)
|
||||
# Backfill: any existing row gets source='manual'
|
||||
cur.execute("UPDATE unit_assignments SET source='manual' WHERE source IS NULL")
|
||||
conn.commit()
|
||||
print(" Done.")
|
||||
else:
|
||||
print("unit_assignments.source already exists — skipping")
|
||||
|
||||
# ── 2. metadata_backfill_decisions table ──────────────────────────────
|
||||
cur.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name='metadata_backfill_decisions'"
|
||||
)
|
||||
if cur.fetchone() is None:
|
||||
print("Creating metadata_backfill_decisions table ...")
|
||||
cur.execute("""
|
||||
CREATE TABLE metadata_backfill_decisions (
|
||||
cluster_id TEXT PRIMARY KEY, -- deterministic hash
|
||||
status TEXT NOT NULL, -- pending | applied | skipped | conflict
|
||||
confidence TEXT NOT NULL, -- high | medium | low (at time of decision)
|
||||
decided_at TEXT, -- when applied/skipped
|
||||
decided_by TEXT, -- 'background' | 'operator' | 'auto-high'
|
||||
applied_assignment_id TEXT, -- FK to unit_assignments (if applied)
|
||||
notes TEXT,
|
||||
first_seen_at TEXT NOT NULL,
|
||||
last_seen_at TEXT NOT NULL,
|
||||
serial TEXT NOT NULL,
|
||||
project_raw TEXT,
|
||||
location_raw TEXT,
|
||||
first_event_ts TEXT,
|
||||
last_event_ts TEXT,
|
||||
event_count INTEGER NOT NULL DEFAULT 0
|
||||
)
|
||||
""")
|
||||
cur.execute(
|
||||
"CREATE INDEX idx_mbd_status ON metadata_backfill_decisions(status)"
|
||||
)
|
||||
cur.execute(
|
||||
"CREATE INDEX idx_mbd_last_seen ON metadata_backfill_decisions(last_seen_at)"
|
||||
)
|
||||
cur.execute(
|
||||
"CREATE INDEX idx_mbd_serial ON metadata_backfill_decisions(serial)"
|
||||
)
|
||||
conn.commit()
|
||||
print(" Done.")
|
||||
else:
|
||||
print("metadata_backfill_decisions table already exists — skipping")
|
||||
|
||||
conn.close()
|
||||
print("\nMigration complete.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate_database()
|
||||
@@ -0,0 +1,73 @@
|
||||
"""
|
||||
Migration: Add one-off schedule fields to recurring_schedules table
|
||||
|
||||
Adds start_datetime and end_datetime columns for one-off recording schedules.
|
||||
|
||||
Run this script once to update existing databases:
|
||||
python -m backend.migrate_add_oneoff_schedule_fields
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import os
|
||||
|
||||
DB_PATH = "data/seismo_fleet.db"
|
||||
|
||||
|
||||
def migrate():
|
||||
"""Add one-off schedule columns to recurring_schedules table."""
|
||||
if not os.path.exists(DB_PATH):
|
||||
print(f"Database not found at {DB_PATH}")
|
||||
return False
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
|
||||
try:
|
||||
cursor.execute("""
|
||||
SELECT name FROM sqlite_master
|
||||
WHERE type='table' AND name='recurring_schedules'
|
||||
""")
|
||||
if not cursor.fetchone():
|
||||
print("recurring_schedules table does not exist yet. Will be created on app startup.")
|
||||
conn.close()
|
||||
return True
|
||||
|
||||
cursor.execute("PRAGMA table_info(recurring_schedules)")
|
||||
columns = [row[1] for row in cursor.fetchall()]
|
||||
|
||||
added = False
|
||||
|
||||
if "start_datetime" not in columns:
|
||||
print("Adding start_datetime column to recurring_schedules table...")
|
||||
cursor.execute("""
|
||||
ALTER TABLE recurring_schedules
|
||||
ADD COLUMN start_datetime DATETIME NULL
|
||||
""")
|
||||
added = True
|
||||
|
||||
if "end_datetime" not in columns:
|
||||
print("Adding end_datetime column to recurring_schedules table...")
|
||||
cursor.execute("""
|
||||
ALTER TABLE recurring_schedules
|
||||
ADD COLUMN end_datetime DATETIME NULL
|
||||
""")
|
||||
added = True
|
||||
|
||||
if added:
|
||||
conn.commit()
|
||||
print("Successfully added one-off schedule columns.")
|
||||
else:
|
||||
print("One-off schedule columns already exist.")
|
||||
|
||||
conn.close()
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"Migration failed: {e}")
|
||||
conn.close()
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = migrate()
|
||||
exit(0 if success else 1)
|
||||
@@ -0,0 +1,54 @@
|
||||
"""
|
||||
Database Migration: Add out_for_calibration field to roster table
|
||||
|
||||
Changes:
|
||||
- Adds out_for_calibration BOOLEAN column (default FALSE) to roster table
|
||||
- Safe to run multiple times (idempotent)
|
||||
- No data loss
|
||||
|
||||
Usage:
|
||||
python backend/migrate_add_out_for_calibration.py
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
SQLALCHEMY_DATABASE_URL = "sqlite:///./data/seismo_fleet.db"
|
||||
engine = create_engine(SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False})
|
||||
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
|
||||
|
||||
|
||||
def migrate():
|
||||
db = SessionLocal()
|
||||
try:
|
||||
print("=" * 60)
|
||||
print("Migration: Add out_for_calibration to roster")
|
||||
print("=" * 60)
|
||||
|
||||
# Check if column already exists
|
||||
result = db.execute(text("PRAGMA table_info(roster)")).fetchall()
|
||||
columns = [row[1] for row in result]
|
||||
|
||||
if "out_for_calibration" in columns:
|
||||
print("Column out_for_calibration already exists. Skipping.")
|
||||
else:
|
||||
db.execute(text("ALTER TABLE roster ADD COLUMN out_for_calibration BOOLEAN DEFAULT FALSE"))
|
||||
db.commit()
|
||||
print("Added out_for_calibration column to roster table.")
|
||||
|
||||
print("Migration complete.")
|
||||
except Exception as e:
|
||||
db.rollback()
|
||||
print(f"Error: {e}")
|
||||
raise
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate()
|
||||
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Migration: Add data_collection_mode column to projects table.
|
||||
|
||||
Values:
|
||||
"remote" — units have modems; data pulled via FTP/scheduler automatically
|
||||
"manual" — no modem; SD cards retrieved daily and uploaded by hand
|
||||
|
||||
All existing projects are backfilled to "manual" (safe conservative default).
|
||||
|
||||
Run once inside the Docker container:
|
||||
docker exec terra-view python3 backend/migrate_add_project_data_collection_mode.py
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
DB_PATH = Path("data/seismo_fleet.db")
|
||||
|
||||
|
||||
def migrate():
|
||||
import sqlite3
|
||||
|
||||
if not DB_PATH.exists():
|
||||
print(f"Database not found at {DB_PATH}. Are you running from /home/serversdown/terra-view?")
|
||||
return
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
cur = conn.cursor()
|
||||
|
||||
# ── 1. Add column (idempotent) ───────────────────────────────────────────
|
||||
cur.execute("PRAGMA table_info(projects)")
|
||||
existing_cols = {row["name"] for row in cur.fetchall()}
|
||||
|
||||
if "data_collection_mode" not in existing_cols:
|
||||
cur.execute("ALTER TABLE projects ADD COLUMN data_collection_mode TEXT DEFAULT 'manual'")
|
||||
conn.commit()
|
||||
print("✓ Added column data_collection_mode to projects")
|
||||
else:
|
||||
print("○ Column data_collection_mode already exists — skipping ALTER TABLE")
|
||||
|
||||
# ── 2. Backfill NULLs to 'manual' ────────────────────────────────────────
|
||||
cur.execute("UPDATE projects SET data_collection_mode = 'manual' WHERE data_collection_mode IS NULL")
|
||||
updated = cur.rowcount
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
if updated:
|
||||
print(f"✓ Backfilled {updated} project(s) to data_collection_mode='manual'.")
|
||||
print("Migration complete.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate()
|
||||
@@ -0,0 +1,56 @@
|
||||
"""
|
||||
Migration: Add deleted_at column to projects table
|
||||
|
||||
Adds columns:
|
||||
- projects.deleted_at: Timestamp set when status='deleted'; data hard-deleted after 60 days
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def migrate(db_path: str):
|
||||
"""Run the migration."""
|
||||
print(f"Migrating database: {db_path}")
|
||||
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
|
||||
try:
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='projects'")
|
||||
if not cursor.fetchone():
|
||||
print("projects table does not exist. Skipping migration.")
|
||||
return
|
||||
|
||||
cursor.execute("PRAGMA table_info(projects)")
|
||||
existing_cols = {row[1] for row in cursor.fetchall()}
|
||||
|
||||
if 'deleted_at' not in existing_cols:
|
||||
print("Adding deleted_at column to projects...")
|
||||
cursor.execute("ALTER TABLE projects ADD COLUMN deleted_at DATETIME")
|
||||
else:
|
||||
print("deleted_at column already exists. Skipping.")
|
||||
|
||||
conn.commit()
|
||||
print("Migration completed successfully!")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Migration failed: {e}")
|
||||
conn.rollback()
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
db_path = "./data/seismo_fleet.db"
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
db_path = sys.argv[1]
|
||||
|
||||
if not Path(db_path).exists():
|
||||
print(f"Database not found: {db_path}")
|
||||
sys.exit(1)
|
||||
|
||||
migrate(db_path)
|
||||
@@ -0,0 +1,71 @@
|
||||
"""
|
||||
Migration: Add project_modules table and seed from existing project_type_id values.
|
||||
|
||||
Safe to run multiple times — idempotent.
|
||||
"""
|
||||
import sqlite3
|
||||
import uuid
|
||||
import os
|
||||
|
||||
DB_PATH = os.path.join(os.path.dirname(__file__), "..", "data", "seismo_fleet.db")
|
||||
DB_PATH = os.path.abspath(DB_PATH)
|
||||
|
||||
|
||||
def run():
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
cur = conn.cursor()
|
||||
|
||||
# 1. Create project_modules table if not exists
|
||||
cur.execute("""
|
||||
CREATE TABLE IF NOT EXISTS project_modules (
|
||||
id TEXT PRIMARY KEY,
|
||||
project_id TEXT NOT NULL,
|
||||
module_type TEXT NOT NULL,
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
created_at TEXT NOT NULL DEFAULT (datetime('now')),
|
||||
UNIQUE(project_id, module_type)
|
||||
)
|
||||
""")
|
||||
print(" Table 'project_modules' ready.")
|
||||
|
||||
# 2. Seed modules from existing project_type_id values
|
||||
cur.execute("SELECT id, project_type_id FROM projects WHERE project_type_id IS NOT NULL")
|
||||
projects = cur.fetchall()
|
||||
|
||||
seeded = 0
|
||||
for p in projects:
|
||||
pid = p["id"]
|
||||
ptype = p["project_type_id"]
|
||||
|
||||
modules_to_add = []
|
||||
if ptype == "sound_monitoring":
|
||||
modules_to_add = ["sound_monitoring"]
|
||||
elif ptype == "vibration_monitoring":
|
||||
modules_to_add = ["vibration_monitoring"]
|
||||
elif ptype == "combined":
|
||||
modules_to_add = ["sound_monitoring", "vibration_monitoring"]
|
||||
|
||||
for module_type in modules_to_add:
|
||||
# INSERT OR IGNORE — skip if already exists
|
||||
cur.execute("""
|
||||
INSERT OR IGNORE INTO project_modules (id, project_id, module_type, enabled)
|
||||
VALUES (?, ?, ?, 1)
|
||||
""", (str(uuid.uuid4()), pid, module_type))
|
||||
if cur.rowcount > 0:
|
||||
seeded += 1
|
||||
|
||||
conn.commit()
|
||||
print(f" Seeded {seeded} module record(s) from existing project_type_id values.")
|
||||
|
||||
# 3. Make project_type_id nullable (SQLite doesn't support ALTER COLUMN,
|
||||
# but since we're just loosening a constraint this is a no-op in SQLite —
|
||||
# the column already accepts NULL in practice. Nothing to do.)
|
||||
print(" project_type_id column is now treated as nullable (legacy field).")
|
||||
|
||||
conn.close()
|
||||
print("Migration complete.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run()
|
||||
@@ -0,0 +1,80 @@
|
||||
"""
|
||||
Migration script to add project_number field to projects table.
|
||||
|
||||
This adds a new column for TMI internal project numbering:
|
||||
- Format: xxxx-YY (e.g., "2567-23")
|
||||
- xxxx = incremental project number
|
||||
- YY = year project was started
|
||||
|
||||
Combined with client_name and name (project/site name), this enables
|
||||
smart searching across all project identifiers.
|
||||
|
||||
Run this script once to migrate an existing database.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import os
|
||||
|
||||
# Database path
|
||||
DB_PATH = "./data/seismo_fleet.db"
|
||||
|
||||
|
||||
def migrate_database():
|
||||
"""Add project_number column to projects table"""
|
||||
|
||||
if not os.path.exists(DB_PATH):
|
||||
print(f"Database not found at {DB_PATH}")
|
||||
print("The database will be created automatically when you run the application.")
|
||||
return
|
||||
|
||||
print(f"Migrating database: {DB_PATH}")
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Check if projects table exists
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='projects'")
|
||||
table_exists = cursor.fetchone()
|
||||
|
||||
if not table_exists:
|
||||
print("Projects table does not exist yet - will be created when app runs")
|
||||
conn.close()
|
||||
return
|
||||
|
||||
# Check if project_number column already exists
|
||||
cursor.execute("PRAGMA table_info(projects)")
|
||||
columns = [col[1] for col in cursor.fetchall()]
|
||||
|
||||
if 'project_number' in columns:
|
||||
print("Migration already applied - project_number column exists")
|
||||
conn.close()
|
||||
return
|
||||
|
||||
print("Adding project_number column to projects table...")
|
||||
|
||||
try:
|
||||
cursor.execute("ALTER TABLE projects ADD COLUMN project_number TEXT")
|
||||
print(" Added project_number column")
|
||||
|
||||
# Create index for faster searching
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS ix_projects_project_number ON projects(project_number)")
|
||||
print(" Created index on project_number")
|
||||
|
||||
# Also add index on client_name if it doesn't exist
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS ix_projects_client_name ON projects(client_name)")
|
||||
print(" Created index on client_name")
|
||||
|
||||
conn.commit()
|
||||
print("\nMigration completed successfully!")
|
||||
|
||||
except sqlite3.Error as e:
|
||||
print(f"\nError during migration: {e}")
|
||||
conn.rollback()
|
||||
raise
|
||||
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate_database()
|
||||
@@ -0,0 +1,88 @@
|
||||
"""
|
||||
Migration script to add report_templates table.
|
||||
|
||||
This creates a new table for storing report generation configurations:
|
||||
- Template name and project association
|
||||
- Time filtering settings (start/end time)
|
||||
- Date range filtering (optional)
|
||||
- Report title defaults
|
||||
|
||||
Run this script once to migrate an existing database.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import os
|
||||
|
||||
# Database path
|
||||
DB_PATH = "./data/seismo_fleet.db"
|
||||
|
||||
def migrate_database():
|
||||
"""Create report_templates table"""
|
||||
|
||||
if not os.path.exists(DB_PATH):
|
||||
print(f"Database not found at {DB_PATH}")
|
||||
print("The database will be created automatically when you run the application.")
|
||||
return
|
||||
|
||||
print(f"Migrating database: {DB_PATH}")
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Check if report_templates table already exists
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='report_templates'")
|
||||
table_exists = cursor.fetchone()
|
||||
|
||||
if table_exists:
|
||||
print("Migration already applied - report_templates table exists")
|
||||
conn.close()
|
||||
return
|
||||
|
||||
print("Creating report_templates table...")
|
||||
|
||||
try:
|
||||
cursor.execute("""
|
||||
CREATE TABLE report_templates (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
project_id TEXT,
|
||||
report_title TEXT DEFAULT 'Background Noise Study',
|
||||
start_time TEXT,
|
||||
end_time TEXT,
|
||||
start_date TEXT,
|
||||
end_date TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
)
|
||||
""")
|
||||
print(" ✓ Created report_templates table")
|
||||
|
||||
# Insert default templates
|
||||
import uuid
|
||||
|
||||
default_templates = [
|
||||
(str(uuid.uuid4()), "Nighttime (7PM-7AM)", None, "Background Noise Study", "19:00", "07:00", None, None),
|
||||
(str(uuid.uuid4()), "Daytime (7AM-7PM)", None, "Background Noise Study", "07:00", "19:00", None, None),
|
||||
(str(uuid.uuid4()), "Full Day (All Data)", None, "Background Noise Study", None, None, None, None),
|
||||
]
|
||||
|
||||
cursor.executemany("""
|
||||
INSERT INTO report_templates (id, name, project_id, report_title, start_time, end_time, start_date, end_date)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", default_templates)
|
||||
print(" ✓ Inserted default templates (Nighttime, Daytime, Full Day)")
|
||||
|
||||
conn.commit()
|
||||
print("\nMigration completed successfully!")
|
||||
|
||||
except sqlite3.Error as e:
|
||||
print(f"\nError during migration: {e}")
|
||||
conn.rollback()
|
||||
raise
|
||||
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate_database()
|
||||
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Migration: Add device_model column to monitoring_sessions table.
|
||||
|
||||
Records which physical SLM model produced each session's data (e.g. "NL-43",
|
||||
"NL-53", "NL-32"). Used by report generation to apply the correct parsing
|
||||
logic without re-opening files to detect format.
|
||||
|
||||
Run once inside the Docker container:
|
||||
docker exec terra-view python3 backend/migrate_add_session_device_model.py
|
||||
|
||||
Backfill strategy for existing rows:
|
||||
1. If session.unit_id is set, use roster.slm_model for that unit.
|
||||
2. Else, peek at the first .rnd file in the session: presence of the 'LAeq'
|
||||
column header identifies AU2 / NL-32 format.
|
||||
Sessions where neither hint is available remain NULL — the file-content
|
||||
fallback in report code handles them transparently.
|
||||
"""
|
||||
import csv
|
||||
import io
|
||||
from pathlib import Path
|
||||
|
||||
DB_PATH = Path("data/seismo_fleet.db")
|
||||
|
||||
|
||||
def _peek_first_row(abs_path: Path) -> dict:
|
||||
"""Read only the header + first data row of an RND file. Very cheap."""
|
||||
try:
|
||||
with open(abs_path, "r", encoding="utf-8", errors="replace") as f:
|
||||
reader = csv.DictReader(f)
|
||||
return next(reader, None) or {}
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _detect_model_from_rnd(abs_path: Path) -> str | None:
|
||||
"""Return 'NL-32' if file uses AU2 column format, else None."""
|
||||
row = _peek_first_row(abs_path)
|
||||
if "LAeq" in row:
|
||||
return "NL-32"
|
||||
return None
|
||||
|
||||
|
||||
def migrate():
|
||||
import sqlite3
|
||||
|
||||
if not DB_PATH.exists():
|
||||
print(f"Database not found at {DB_PATH}. Are you running from /home/serversdown/terra-view?")
|
||||
return
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
cur = conn.cursor()
|
||||
|
||||
# ── 1. Add column (idempotent) ───────────────────────────────────────────
|
||||
cur.execute("PRAGMA table_info(monitoring_sessions)")
|
||||
existing_cols = {row["name"] for row in cur.fetchall()}
|
||||
|
||||
if "device_model" not in existing_cols:
|
||||
cur.execute("ALTER TABLE monitoring_sessions ADD COLUMN device_model TEXT")
|
||||
conn.commit()
|
||||
print("✓ Added column device_model to monitoring_sessions")
|
||||
else:
|
||||
print("○ Column device_model already exists — skipping ALTER TABLE")
|
||||
|
||||
# ── 2. Backfill existing NULL rows ───────────────────────────────────────
|
||||
cur.execute(
|
||||
"SELECT id, unit_id FROM monitoring_sessions WHERE device_model IS NULL"
|
||||
)
|
||||
sessions = cur.fetchall()
|
||||
print(f"Backfilling {len(sessions)} session(s) with device_model=NULL...")
|
||||
|
||||
updated = skipped = 0
|
||||
for row in sessions:
|
||||
session_id = row["id"]
|
||||
unit_id = row["unit_id"]
|
||||
device_model = None
|
||||
|
||||
# Strategy A: look up unit's slm_model from the roster
|
||||
if unit_id:
|
||||
cur.execute(
|
||||
"SELECT slm_model FROM roster WHERE id = ?", (unit_id,)
|
||||
)
|
||||
unit_row = cur.fetchone()
|
||||
if unit_row and unit_row["slm_model"]:
|
||||
device_model = unit_row["slm_model"]
|
||||
|
||||
# Strategy B: detect from first .rnd file in the session
|
||||
if device_model is None:
|
||||
cur.execute(
|
||||
"""SELECT file_path FROM data_files
|
||||
WHERE session_id = ?
|
||||
AND lower(file_path) LIKE '%.rnd'
|
||||
LIMIT 1""",
|
||||
(session_id,),
|
||||
)
|
||||
file_row = cur.fetchone()
|
||||
if file_row:
|
||||
abs_path = Path("data") / file_row["file_path"]
|
||||
device_model = _detect_model_from_rnd(abs_path)
|
||||
# None here means NL-43/NL-53 format (or unreadable file) —
|
||||
# leave as NULL so the existing fallback applies.
|
||||
|
||||
if device_model:
|
||||
cur.execute(
|
||||
"UPDATE monitoring_sessions SET device_model = ? WHERE id = ?",
|
||||
(device_model, session_id),
|
||||
)
|
||||
updated += 1
|
||||
else:
|
||||
skipped += 1
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
print(f"✓ Backfilled {updated} session(s) with a device_model.")
|
||||
if skipped:
|
||||
print(
|
||||
f" {skipped} session(s) left as NULL "
|
||||
"(no unit link and no AU2 file hint — NL-43/NL-53 or unknown; "
|
||||
"file-content detection applies at report time)."
|
||||
)
|
||||
print("Migration complete.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate()
|
||||
@@ -0,0 +1,42 @@
|
||||
"""
|
||||
Migration: add period_start_hour and period_end_hour to monitoring_sessions.
|
||||
|
||||
Run once:
|
||||
python backend/migrate_add_session_period_hours.py
|
||||
|
||||
Or inside the container:
|
||||
docker exec terra-view python3 backend/migrate_add_session_period_hours.py
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from backend.database import engine
|
||||
from sqlalchemy import text
|
||||
|
||||
def run():
|
||||
with engine.connect() as conn:
|
||||
# Check which columns already exist
|
||||
result = conn.execute(text("PRAGMA table_info(monitoring_sessions)"))
|
||||
existing = {row[1] for row in result}
|
||||
|
||||
added = []
|
||||
for col, definition in [
|
||||
("period_start_hour", "INTEGER"),
|
||||
("period_end_hour", "INTEGER"),
|
||||
]:
|
||||
if col not in existing:
|
||||
conn.execute(text(f"ALTER TABLE monitoring_sessions ADD COLUMN {col} {definition}"))
|
||||
added.append(col)
|
||||
else:
|
||||
print(f" Column '{col}' already exists — skipping.")
|
||||
|
||||
conn.commit()
|
||||
|
||||
if added:
|
||||
print(f" Added columns: {', '.join(added)}")
|
||||
print("Migration complete.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
run()
|
||||
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Migration: Add session_label and period_type columns to monitoring_sessions.
|
||||
|
||||
session_label - user-editable display name, e.g. "NRL-1 Sun 2/23 Night"
|
||||
period_type - one of: weekday_day | weekday_night | weekend_day | weekend_night
|
||||
Auto-derived from started_at when NULL.
|
||||
|
||||
Period definitions (used in report stats table):
|
||||
weekday_day Mon-Fri 07:00-22:00 -> Daytime (7AM-10PM)
|
||||
weekday_night Mon-Fri 22:00-07:00 -> Nighttime (10PM-7AM)
|
||||
weekend_day Sat-Sun 07:00-22:00 -> Daytime (7AM-10PM)
|
||||
weekend_night Sat-Sun 22:00-07:00 -> Nighttime (10PM-7AM)
|
||||
|
||||
Run once inside the Docker container:
|
||||
docker exec terra-view python3 backend/migrate_add_session_period_type.py
|
||||
"""
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
|
||||
DB_PATH = Path("data/seismo_fleet.db")
|
||||
|
||||
|
||||
def _derive_period_type(started_at_str: str) -> str | None:
|
||||
"""Derive period_type from a started_at ISO datetime string."""
|
||||
if not started_at_str:
|
||||
return None
|
||||
try:
|
||||
dt = datetime.fromisoformat(started_at_str)
|
||||
except ValueError:
|
||||
return None
|
||||
is_weekend = dt.weekday() >= 5 # 5=Sat, 6=Sun
|
||||
is_night = dt.hour >= 22 or dt.hour < 7
|
||||
if is_weekend:
|
||||
return "weekend_night" if is_night else "weekend_day"
|
||||
else:
|
||||
return "weekday_night" if is_night else "weekday_day"
|
||||
|
||||
|
||||
def _build_label(started_at_str: str, location_name: str | None, period_type: str | None) -> str | None:
|
||||
"""Build a human-readable session label."""
|
||||
if not started_at_str:
|
||||
return None
|
||||
try:
|
||||
dt = datetime.fromisoformat(started_at_str)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
day_abbr = dt.strftime("%a") # Mon, Tue, Sun, etc.
|
||||
date_str = dt.strftime("%-m/%-d") # 2/23
|
||||
|
||||
period_labels = {
|
||||
"weekday_day": "Day",
|
||||
"weekday_night": "Night",
|
||||
"weekend_day": "Day",
|
||||
"weekend_night": "Night",
|
||||
}
|
||||
period_str = period_labels.get(period_type or "", "")
|
||||
|
||||
parts = []
|
||||
if location_name:
|
||||
parts.append(location_name)
|
||||
parts.append(f"{day_abbr} {date_str}")
|
||||
if period_str:
|
||||
parts.append(period_str)
|
||||
return " — ".join(parts)
|
||||
|
||||
|
||||
def migrate():
|
||||
import sqlite3
|
||||
|
||||
if not DB_PATH.exists():
|
||||
print(f"Database not found at {DB_PATH}. Are you running from /home/serversdown/terra-view?")
|
||||
return
|
||||
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
cur = conn.cursor()
|
||||
|
||||
# 1. Add columns (idempotent)
|
||||
cur.execute("PRAGMA table_info(monitoring_sessions)")
|
||||
existing_cols = {row["name"] for row in cur.fetchall()}
|
||||
|
||||
for col, typedef in [("session_label", "TEXT"), ("period_type", "TEXT")]:
|
||||
if col not in existing_cols:
|
||||
cur.execute(f"ALTER TABLE monitoring_sessions ADD COLUMN {col} {typedef}")
|
||||
conn.commit()
|
||||
print(f"✓ Added column {col} to monitoring_sessions")
|
||||
else:
|
||||
print(f"○ Column {col} already exists — skipping ALTER TABLE")
|
||||
|
||||
# 2. Backfill existing rows
|
||||
cur.execute(
|
||||
"""SELECT ms.id, ms.started_at, ms.location_id
|
||||
FROM monitoring_sessions ms
|
||||
WHERE ms.period_type IS NULL OR ms.session_label IS NULL"""
|
||||
)
|
||||
sessions = cur.fetchall()
|
||||
print(f"Backfilling {len(sessions)} session(s)...")
|
||||
|
||||
updated = 0
|
||||
for row in sessions:
|
||||
session_id = row["id"]
|
||||
started_at = row["started_at"]
|
||||
location_id = row["location_id"]
|
||||
|
||||
# Look up location name
|
||||
location_name = None
|
||||
if location_id:
|
||||
cur.execute("SELECT name FROM monitoring_locations WHERE id = ?", (location_id,))
|
||||
loc_row = cur.fetchone()
|
||||
if loc_row:
|
||||
location_name = loc_row["name"]
|
||||
|
||||
period_type = _derive_period_type(started_at)
|
||||
label = _build_label(started_at, location_name, period_type)
|
||||
|
||||
cur.execute(
|
||||
"UPDATE monitoring_sessions SET period_type = ?, session_label = ? WHERE id = ?",
|
||||
(period_type, label, session_id),
|
||||
)
|
||||
updated += 1
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
print(f"✓ Backfilled {updated} session(s).")
|
||||
print("Migration complete.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate()
|
||||
@@ -0,0 +1,41 @@
|
||||
"""
|
||||
Migration: add report_date to monitoring_sessions.
|
||||
|
||||
Run once:
|
||||
python backend/migrate_add_session_report_date.py
|
||||
|
||||
Or inside the container:
|
||||
docker exec terra-view-terra-view-1 python3 backend/migrate_add_session_report_date.py
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from backend.database import engine
|
||||
from sqlalchemy import text
|
||||
|
||||
def run():
|
||||
with engine.connect() as conn:
|
||||
# Check which columns already exist
|
||||
result = conn.execute(text("PRAGMA table_info(monitoring_sessions)"))
|
||||
existing = {row[1] for row in result}
|
||||
|
||||
added = []
|
||||
for col, definition in [
|
||||
("report_date", "DATE"),
|
||||
]:
|
||||
if col not in existing:
|
||||
conn.execute(text(f"ALTER TABLE monitoring_sessions ADD COLUMN {col} {definition}"))
|
||||
added.append(col)
|
||||
else:
|
||||
print(f" Column '{col}' already exists — skipping.")
|
||||
|
||||
conn.commit()
|
||||
|
||||
if added:
|
||||
print(f" Added columns: {', '.join(added)}")
|
||||
print("Migration complete.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
run()
|
||||
@@ -71,7 +71,7 @@ def migrate():
|
||||
print("\n○ No migration needed - all columns already exist.")
|
||||
|
||||
print("\nSound level meter fields are now available in the roster table.")
|
||||
print("You can now set device_type='sound_level_meter' for SLM devices.")
|
||||
print("Note: Use device_type='slm' for Sound Level Meters. Legacy 'sound_level_meter' has been deprecated.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
"""
|
||||
Migration: Add TBD date support to job reservations
|
||||
|
||||
Adds columns:
|
||||
- job_reservations.estimated_end_date: For planning when end is TBD
|
||||
- job_reservations.end_date_tbd: Boolean flag for TBD end dates
|
||||
- job_reservation_units.unit_start_date: Unit-specific start (for swaps)
|
||||
- job_reservation_units.unit_end_date: Unit-specific end (for swaps)
|
||||
- job_reservation_units.unit_end_tbd: Unit-specific TBD flag
|
||||
- job_reservation_units.notes: Notes for the assignment
|
||||
|
||||
Also makes job_reservations.end_date nullable.
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
def migrate(db_path: str):
|
||||
"""Run the migration."""
|
||||
print(f"Migrating database: {db_path}")
|
||||
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
|
||||
try:
|
||||
# Check if job_reservations table exists
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='job_reservations'")
|
||||
if not cursor.fetchone():
|
||||
print("job_reservations table does not exist. Skipping migration.")
|
||||
return
|
||||
|
||||
# Get existing columns in job_reservations
|
||||
cursor.execute("PRAGMA table_info(job_reservations)")
|
||||
existing_cols = {row[1] for row in cursor.fetchall()}
|
||||
|
||||
# Add new columns to job_reservations if they don't exist
|
||||
if 'estimated_end_date' not in existing_cols:
|
||||
print("Adding estimated_end_date column to job_reservations...")
|
||||
cursor.execute("ALTER TABLE job_reservations ADD COLUMN estimated_end_date DATE")
|
||||
|
||||
if 'end_date_tbd' not in existing_cols:
|
||||
print("Adding end_date_tbd column to job_reservations...")
|
||||
cursor.execute("ALTER TABLE job_reservations ADD COLUMN end_date_tbd BOOLEAN DEFAULT 0")
|
||||
|
||||
# Get existing columns in job_reservation_units
|
||||
cursor.execute("PRAGMA table_info(job_reservation_units)")
|
||||
unit_cols = {row[1] for row in cursor.fetchall()}
|
||||
|
||||
# Add new columns to job_reservation_units if they don't exist
|
||||
if 'unit_start_date' not in unit_cols:
|
||||
print("Adding unit_start_date column to job_reservation_units...")
|
||||
cursor.execute("ALTER TABLE job_reservation_units ADD COLUMN unit_start_date DATE")
|
||||
|
||||
if 'unit_end_date' not in unit_cols:
|
||||
print("Adding unit_end_date column to job_reservation_units...")
|
||||
cursor.execute("ALTER TABLE job_reservation_units ADD COLUMN unit_end_date DATE")
|
||||
|
||||
if 'unit_end_tbd' not in unit_cols:
|
||||
print("Adding unit_end_tbd column to job_reservation_units...")
|
||||
cursor.execute("ALTER TABLE job_reservation_units ADD COLUMN unit_end_tbd BOOLEAN DEFAULT 0")
|
||||
|
||||
if 'notes' not in unit_cols:
|
||||
print("Adding notes column to job_reservation_units...")
|
||||
cursor.execute("ALTER TABLE job_reservation_units ADD COLUMN notes TEXT")
|
||||
|
||||
conn.commit()
|
||||
print("Migration completed successfully!")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Migration failed: {e}")
|
||||
conn.rollback()
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Default to dev database
|
||||
db_path = "./data-dev/seismo_fleet.db"
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
db_path = sys.argv[1]
|
||||
|
||||
if not Path(db_path).exists():
|
||||
print(f"Database not found: {db_path}")
|
||||
sys.exit(1)
|
||||
|
||||
migrate(db_path)
|
||||
@@ -0,0 +1,209 @@
|
||||
"""
|
||||
Migration: deprecate the `deployment_records` table.
|
||||
|
||||
Why:
|
||||
The deployment-history view on the unit detail page used to render
|
||||
from `deployment_records` — a manually-maintained table that drifted
|
||||
out of sync with `unit_assignments` (the auto-written project/location
|
||||
assignment table). That caused the "wonky timeline" symptom: missing
|
||||
entries, duplicate / contradictory rows, and a UI that couldn't tell
|
||||
the operator what the unit was actually doing during each window.
|
||||
|
||||
Phase 4 of the SFM integration replaces the deployment-history view
|
||||
with a derived timeline computed from `unit_assignments` +
|
||||
`unit_history` + SFM event overlay. This migration is the cleanup:
|
||||
|
||||
1. Adds a `deprecated_at` timestamp column to `deployment_records` so
|
||||
we can mark rows that have been migrated.
|
||||
2. For every `deployment_records` row that does NOT have a matching
|
||||
`unit_assignments` row (matched by unit_id + overlapping date
|
||||
range), synthesizes a best-effort UnitAssignment row. The
|
||||
free-text `location_name` from the legacy table is preserved on
|
||||
the new row's `notes` field (we do NOT try to fuzzy-match it to a
|
||||
MonitoringLocation id; too error-prone — operators will need to
|
||||
reattach those manually if they want).
|
||||
3. Marks every migrated deployment_records row with `deprecated_at`.
|
||||
|
||||
This migration is non-destructive: deployment_records rows stay in
|
||||
the DB. The actual `DROP TABLE` happens in a follow-up release after
|
||||
one operator cycle confirms nothing relies on the legacy data.
|
||||
|
||||
Idempotent: re-running the script is a no-op if the column already
|
||||
exists and all migratable rows have already been processed.
|
||||
|
||||
Run with:
|
||||
docker exec terra-view-terra-view-1 python3 /app/backend/migrate_deprecate_deployment_records.py
|
||||
"""
|
||||
|
||||
import os
|
||||
import sqlite3
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
|
||||
DB_PATH = "./data/seismo_fleet.db"
|
||||
|
||||
|
||||
def migrate_database():
|
||||
if not os.path.exists(DB_PATH):
|
||||
print(f"Database not found at {DB_PATH}")
|
||||
return
|
||||
|
||||
print(f"Migrating database: {DB_PATH}")
|
||||
conn = sqlite3.connect(DB_PATH)
|
||||
conn.row_factory = sqlite3.Row
|
||||
cur = conn.cursor()
|
||||
|
||||
# 1. Add deprecated_at column if not present.
|
||||
cur.execute("PRAGMA table_info(deployment_records)")
|
||||
cols = {row["name"] for row in cur.fetchall()}
|
||||
if "deprecated_at" not in cols:
|
||||
print("Adding deployment_records.deprecated_at column ...")
|
||||
cur.execute("ALTER TABLE deployment_records ADD COLUMN deprecated_at TEXT")
|
||||
conn.commit()
|
||||
else:
|
||||
print("deployment_records.deprecated_at column already exists — skipping ADD COLUMN")
|
||||
|
||||
# 2. Find candidate rows: not-yet-deprecated deployment_records that
|
||||
# have no matching unit_assignments row.
|
||||
cur.execute("""
|
||||
SELECT id, unit_id, deployed_date, estimated_removal_date,
|
||||
actual_removal_date, project_id, project_ref, location_name, notes
|
||||
FROM deployment_records
|
||||
WHERE deprecated_at IS NULL
|
||||
""")
|
||||
rows = cur.fetchall()
|
||||
print(f"\nFound {len(rows)} deployment_records rows not yet deprecated.")
|
||||
|
||||
backfilled = 0
|
||||
skipped_no_match_attempted = 0
|
||||
skipped_already_in_assignments = 0
|
||||
skipped_missing_unit = 0
|
||||
|
||||
for row in rows:
|
||||
unit_id = row["unit_id"]
|
||||
if not unit_id:
|
||||
print(f" ⚠ row {row['id']!r}: no unit_id, marking deprecated without backfill")
|
||||
cur.execute(
|
||||
"UPDATE deployment_records SET deprecated_at=? WHERE id=?",
|
||||
(datetime.utcnow().isoformat(), row["id"]),
|
||||
)
|
||||
skipped_missing_unit += 1
|
||||
continue
|
||||
|
||||
# Does the unit still exist? If not, skip — we don't synthesize
|
||||
# assignments for ghost units.
|
||||
cur.execute("SELECT id, device_type FROM roster WHERE id=?", (unit_id,))
|
||||
roster = cur.fetchone()
|
||||
if not roster:
|
||||
print(f" ⚠ row {row['id']!r}: unit_id {unit_id!r} not in roster, marking deprecated without backfill")
|
||||
cur.execute(
|
||||
"UPDATE deployment_records SET deprecated_at=? WHERE id=?",
|
||||
(datetime.utcnow().isoformat(), row["id"]),
|
||||
)
|
||||
skipped_missing_unit += 1
|
||||
continue
|
||||
|
||||
# Check if a UnitAssignment already covers this window (any overlap).
|
||||
# We don't try to be clever — just see if a row exists for this unit
|
||||
# whose [assigned_at, assigned_until] overlaps the deployment window.
|
||||
cur.execute("""
|
||||
SELECT id FROM unit_assignments
|
||||
WHERE unit_id=?
|
||||
AND (assigned_at <= COALESCE(?, '9999')
|
||||
AND COALESCE(assigned_until, '9999') >= COALESCE(?, '0000'))
|
||||
LIMIT 1
|
||||
""", (
|
||||
unit_id,
|
||||
row["actual_removal_date"] or row["estimated_removal_date"] or row["deployed_date"],
|
||||
row["deployed_date"],
|
||||
))
|
||||
if cur.fetchone():
|
||||
cur.execute(
|
||||
"UPDATE deployment_records SET deprecated_at=? WHERE id=?",
|
||||
(datetime.utcnow().isoformat(), row["id"]),
|
||||
)
|
||||
skipped_already_in_assignments += 1
|
||||
continue
|
||||
|
||||
# No matching UnitAssignment — synthesize one. We can't FK to a
|
||||
# MonitoringLocation because the legacy `location_name` is free
|
||||
# text. Backfilled rows go in with location_id = "" (empty) and
|
||||
# the original location_name dropped into notes for operator
|
||||
# context.
|
||||
if not row["project_id"]:
|
||||
print(f" ⚠ row {row['id']!r}: no project_id, can't synthesize unit_assignment, marking deprecated")
|
||||
cur.execute(
|
||||
"UPDATE deployment_records SET deprecated_at=? WHERE id=?",
|
||||
(datetime.utcnow().isoformat(), row["id"]),
|
||||
)
|
||||
skipped_no_match_attempted += 1
|
||||
continue
|
||||
|
||||
synthesized_id = str(uuid.uuid4())
|
||||
synth_notes_parts = []
|
||||
if row["location_name"]:
|
||||
synth_notes_parts.append(f"Legacy location: {row['location_name']}")
|
||||
if row["project_ref"]:
|
||||
synth_notes_parts.append(f"Legacy project_ref: {row['project_ref']}")
|
||||
if row["notes"]:
|
||||
synth_notes_parts.append(f"Original notes: {row['notes']}")
|
||||
synth_notes_parts.append(f"(Synthesized from deployment_records row {row['id']})")
|
||||
synth_notes = " | ".join(synth_notes_parts)
|
||||
|
||||
assigned_until = row["actual_removal_date"]
|
||||
# Don't auto-close active deployments based on estimated_removal_date.
|
||||
status = "completed" if assigned_until else "active"
|
||||
|
||||
# Need a location_id to satisfy NOT NULL constraint. Use a
|
||||
# placeholder UUID so the FK can be cleaned up later if the
|
||||
# operator decides to retarget the assignment to a real location.
|
||||
# We tag this with the synthesized notes so it's discoverable.
|
||||
placeholder_loc_id = ""
|
||||
|
||||
try:
|
||||
cur.execute("""
|
||||
INSERT INTO unit_assignments (
|
||||
id, unit_id, location_id, project_id, device_type,
|
||||
assigned_at, assigned_until, status, notes, created_at
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""", (
|
||||
synthesized_id,
|
||||
unit_id,
|
||||
placeholder_loc_id,
|
||||
row["project_id"],
|
||||
roster["device_type"] or "seismograph",
|
||||
row["deployed_date"] or datetime.utcnow().isoformat(),
|
||||
assigned_until,
|
||||
status,
|
||||
synth_notes,
|
||||
datetime.utcnow().isoformat(),
|
||||
))
|
||||
cur.execute(
|
||||
"UPDATE deployment_records SET deprecated_at=? WHERE id=?",
|
||||
(datetime.utcnow().isoformat(), row["id"]),
|
||||
)
|
||||
backfilled += 1
|
||||
print(
|
||||
f" ✓ row {row['id']!r}: synthesized unit_assignment {synthesized_id} "
|
||||
f"for unit={unit_id} project={row['project_id'][:8]}… "
|
||||
f"({row['deployed_date']} → {assigned_until or 'present'})"
|
||||
)
|
||||
except Exception as e:
|
||||
print(f" ✗ row {row['id']!r}: failed to synthesize — {e}")
|
||||
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
print("\n────────────────────────────────────────────────────────")
|
||||
print(f"Backfilled new unit_assignments: {backfilled}")
|
||||
print(f"Already covered (deprecated only): {skipped_already_in_assignments}")
|
||||
print(f"No project_id (deprecated only): {skipped_no_match_attempted}")
|
||||
print(f"Missing/orphaned unit (deprecated): {skipped_missing_unit}")
|
||||
print(f"\nNOTE: synthesized rows have an empty location_id and the legacy")
|
||||
print(f" free-text location is preserved in notes. An operator should")
|
||||
print(f" retarget them to real MonitoringLocation rows if they want")
|
||||
print(f" events to show up on a location detail page.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate_database()
|
||||
@@ -0,0 +1,105 @@
|
||||
"""
|
||||
Migration: Make job_reservations.end_date nullable for TBD support
|
||||
|
||||
SQLite doesn't support ALTER COLUMN, so we need to:
|
||||
1. Create a new table with the correct schema
|
||||
2. Copy data
|
||||
3. Drop old table
|
||||
4. Rename new table
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
def migrate(db_path: str):
|
||||
"""Run the migration."""
|
||||
print(f"Migrating database: {db_path}")
|
||||
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
|
||||
try:
|
||||
# Check if job_reservations table exists
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='job_reservations'")
|
||||
if not cursor.fetchone():
|
||||
print("job_reservations table does not exist. Skipping migration.")
|
||||
return
|
||||
|
||||
# Check current schema
|
||||
cursor.execute("PRAGMA table_info(job_reservations)")
|
||||
columns = cursor.fetchall()
|
||||
col_info = {row[1]: row for row in columns}
|
||||
|
||||
# Check if end_date is already nullable (notnull=0)
|
||||
if 'end_date' in col_info and col_info['end_date'][3] == 0:
|
||||
print("end_date is already nullable. Skipping table recreation.")
|
||||
return
|
||||
|
||||
print("Recreating job_reservations table with nullable end_date...")
|
||||
|
||||
# Create new table with correct schema
|
||||
cursor.execute("""
|
||||
CREATE TABLE job_reservations_new (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
project_id TEXT,
|
||||
start_date DATE NOT NULL,
|
||||
end_date DATE,
|
||||
estimated_end_date DATE,
|
||||
end_date_tbd BOOLEAN DEFAULT 0,
|
||||
assignment_type TEXT NOT NULL DEFAULT 'quantity',
|
||||
device_type TEXT DEFAULT 'seismograph',
|
||||
quantity_needed INTEGER,
|
||||
notes TEXT,
|
||||
color TEXT DEFAULT '#3B82F6',
|
||||
created_at DATETIME,
|
||||
updated_at DATETIME
|
||||
)
|
||||
""")
|
||||
|
||||
# Copy existing data
|
||||
cursor.execute("""
|
||||
INSERT INTO job_reservations_new
|
||||
SELECT
|
||||
id, name, project_id, start_date, end_date,
|
||||
COALESCE(estimated_end_date, NULL) as estimated_end_date,
|
||||
COALESCE(end_date_tbd, 0) as end_date_tbd,
|
||||
assignment_type, device_type, quantity_needed, notes, color,
|
||||
created_at, updated_at
|
||||
FROM job_reservations
|
||||
""")
|
||||
|
||||
# Drop old table
|
||||
cursor.execute("DROP TABLE job_reservations")
|
||||
|
||||
# Rename new table
|
||||
cursor.execute("ALTER TABLE job_reservations_new RENAME TO job_reservations")
|
||||
|
||||
# Recreate index
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS ix_job_reservations_id ON job_reservations (id)")
|
||||
cursor.execute("CREATE INDEX IF NOT EXISTS ix_job_reservations_project_id ON job_reservations (project_id)")
|
||||
|
||||
conn.commit()
|
||||
print("Migration completed successfully!")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Migration failed: {e}")
|
||||
conn.rollback()
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Default to dev database
|
||||
db_path = "./data-dev/seismo_fleet.db"
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
db_path = sys.argv[1]
|
||||
|
||||
if not Path(db_path).exists():
|
||||
print(f"Database not found: {db_path}")
|
||||
sys.exit(1)
|
||||
|
||||
migrate(db_path)
|
||||
@@ -0,0 +1,54 @@
|
||||
"""
|
||||
Migration: Rename recording_sessions table to monitoring_sessions
|
||||
|
||||
Renames the table and updates the model name from RecordingSession to MonitoringSession.
|
||||
Run once per database: python backend/migrate_rename_recording_to_monitoring_sessions.py
|
||||
"""
|
||||
|
||||
import sqlite3
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
def migrate(db_path: str):
|
||||
"""Run the migration."""
|
||||
print(f"Migrating database: {db_path}")
|
||||
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.cursor()
|
||||
|
||||
try:
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='recording_sessions'")
|
||||
if not cursor.fetchone():
|
||||
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='monitoring_sessions'")
|
||||
if cursor.fetchone():
|
||||
print("monitoring_sessions table already exists. Skipping migration.")
|
||||
else:
|
||||
print("recording_sessions table does not exist. Skipping migration.")
|
||||
return
|
||||
|
||||
print("Renaming recording_sessions -> monitoring_sessions...")
|
||||
cursor.execute("ALTER TABLE recording_sessions RENAME TO monitoring_sessions")
|
||||
|
||||
conn.commit()
|
||||
print("Migration completed successfully!")
|
||||
|
||||
except Exception as e:
|
||||
print(f"Migration failed: {e}")
|
||||
conn.rollback()
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
db_path = "./data/seismo_fleet.db"
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
db_path = sys.argv[1]
|
||||
|
||||
if not Path(db_path).exists():
|
||||
print(f"Database not found: {db_path}")
|
||||
sys.exit(1)
|
||||
|
||||
migrate(db_path)
|
||||
@@ -0,0 +1,106 @@
|
||||
"""
|
||||
Database Migration: Standardize device_type values
|
||||
|
||||
This migration ensures all device_type values follow the official schema:
|
||||
- "seismograph" - Seismic monitoring devices
|
||||
- "modem" - Field modems and network equipment
|
||||
- "slm" - Sound level meters (NL-43/NL-53)
|
||||
|
||||
Changes:
|
||||
- Converts "sound_level_meter" → "slm"
|
||||
- Safe to run multiple times (idempotent)
|
||||
- No data loss
|
||||
|
||||
Usage:
|
||||
python backend/migrate_standardize_device_types.py
|
||||
"""
|
||||
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Add parent directory to path so we can import backend modules
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
from sqlalchemy import create_engine, text
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
# Database configuration
|
||||
SQLALCHEMY_DATABASE_URL = "sqlite:///./data/seismo_fleet.db"
|
||||
engine = create_engine(SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False})
|
||||
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
|
||||
|
||||
|
||||
def migrate():
|
||||
"""Standardize device_type values in the database"""
|
||||
db = SessionLocal()
|
||||
|
||||
try:
|
||||
print("=" * 70)
|
||||
print("Database Migration: Standardize device_type values")
|
||||
print("=" * 70)
|
||||
print()
|
||||
|
||||
# Check for existing "sound_level_meter" values
|
||||
result = db.execute(
|
||||
text("SELECT COUNT(*) as count FROM roster WHERE device_type = 'sound_level_meter'")
|
||||
).fetchone()
|
||||
|
||||
count_to_migrate = result[0] if result else 0
|
||||
|
||||
if count_to_migrate == 0:
|
||||
print("✓ No records need migration - all device_type values are already standardized")
|
||||
print()
|
||||
print("Current device_type distribution:")
|
||||
|
||||
# Show distribution
|
||||
distribution = db.execute(
|
||||
text("SELECT device_type, COUNT(*) as count FROM roster GROUP BY device_type ORDER BY count DESC")
|
||||
).fetchall()
|
||||
|
||||
for row in distribution:
|
||||
device_type, count = row
|
||||
print(f" - {device_type}: {count} units")
|
||||
|
||||
print()
|
||||
print("Migration not needed.")
|
||||
return
|
||||
|
||||
print(f"Found {count_to_migrate} record(s) with device_type='sound_level_meter'")
|
||||
print()
|
||||
print("Converting 'sound_level_meter' → 'slm'...")
|
||||
|
||||
# Perform the migration
|
||||
db.execute(
|
||||
text("UPDATE roster SET device_type = 'slm' WHERE device_type = 'sound_level_meter'")
|
||||
)
|
||||
db.commit()
|
||||
|
||||
print(f"✓ Successfully migrated {count_to_migrate} record(s)")
|
||||
print()
|
||||
|
||||
# Show final distribution
|
||||
print("Updated device_type distribution:")
|
||||
distribution = db.execute(
|
||||
text("SELECT device_type, COUNT(*) as count FROM roster GROUP BY device_type ORDER BY count DESC")
|
||||
).fetchall()
|
||||
|
||||
for row in distribution:
|
||||
device_type, count = row
|
||||
print(f" - {device_type}: {count} units")
|
||||
|
||||
print()
|
||||
print("=" * 70)
|
||||
print("Migration completed successfully!")
|
||||
print("=" * 70)
|
||||
|
||||
except Exception as e:
|
||||
db.rollback()
|
||||
print(f"\n❌ Error during migration: {e}")
|
||||
print("\nRolling back changes...")
|
||||
raise
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
migrate()
|
||||
@@ -1,4 +1,4 @@
|
||||
from sqlalchemy import Column, String, DateTime, Boolean, Text, Date, Integer
|
||||
from sqlalchemy import Column, String, DateTime, Boolean, Text, Date, Integer, UniqueConstraint
|
||||
from datetime import datetime
|
||||
from backend.database import Base
|
||||
|
||||
@@ -19,16 +19,22 @@ class RosterUnit(Base):
|
||||
Roster table: represents our *intended assignment* of a unit.
|
||||
This is editable from the GUI.
|
||||
|
||||
Supports multiple device types (seismograph, modem, sound_level_meter) with type-specific fields.
|
||||
Supports multiple device types with type-specific fields:
|
||||
- "seismograph" - Seismic monitoring devices (default)
|
||||
- "modem" - Field modems and network equipment
|
||||
- "slm" - Sound level meters (NL-43/NL-53)
|
||||
"""
|
||||
__tablename__ = "roster"
|
||||
|
||||
# Core fields (all device types)
|
||||
id = Column(String, primary_key=True, index=True)
|
||||
unit_type = Column(String, default="series3") # Backward compatibility
|
||||
device_type = Column(String, default="seismograph") # "seismograph" | "modem" | "sound_level_meter"
|
||||
device_type = Column(String, default="seismograph") # "seismograph" | "modem" | "slm"
|
||||
deployed = Column(Boolean, default=True)
|
||||
retired = Column(Boolean, default=False)
|
||||
out_for_calibration = Column(Boolean, default=False)
|
||||
allocated = Column(Boolean, default=False) # Staged for an upcoming job, not yet deployed
|
||||
allocated_to_project_id = Column(String, nullable=True) # Which project it's allocated to
|
||||
note = Column(String, nullable=True)
|
||||
project_id = Column(String, nullable=True)
|
||||
location = Column(String, nullable=True) # Legacy field - use address/coordinates instead
|
||||
@@ -47,6 +53,8 @@ class RosterUnit(Base):
|
||||
ip_address = Column(String, nullable=True)
|
||||
phone_number = Column(String, nullable=True)
|
||||
hardware_model = Column(String, nullable=True)
|
||||
deployment_type = Column(String, nullable=True) # "seismograph" | "slm" - what type of device this modem is deployed with
|
||||
deployed_with_unit_id = Column(String, nullable=True) # ID of seismograph/SLM this modem is deployed with
|
||||
|
||||
# Sound Level Meter-specific fields (nullable for seismographs and modems)
|
||||
slm_host = Column(String, nullable=True) # Device IP or hostname
|
||||
@@ -60,6 +68,26 @@ class RosterUnit(Base):
|
||||
slm_last_check = Column(DateTime, nullable=True) # Last communication check
|
||||
|
||||
|
||||
class WatcherAgent(Base):
|
||||
"""
|
||||
Watcher agents: tracks the watcher processes (series3-watcher, thor-watcher)
|
||||
that run on field machines and report unit heartbeats.
|
||||
|
||||
Updated on every heartbeat received from each source_id.
|
||||
"""
|
||||
__tablename__ = "watcher_agents"
|
||||
|
||||
id = Column(String, primary_key=True, index=True) # source_id (hostname)
|
||||
source_type = Column(String, nullable=False) # series3_watcher | series4_watcher
|
||||
version = Column(String, nullable=True) # e.g. "1.4.0"
|
||||
last_seen = Column(DateTime, default=datetime.utcnow)
|
||||
status = Column(String, nullable=False, default="unknown") # ok | pending | missing | error | unknown
|
||||
ip_address = Column(String, nullable=True)
|
||||
log_tail = Column(Text, nullable=True) # last N log lines (JSON array of strings)
|
||||
update_pending = Column(Boolean, default=False) # set True to trigger remote update
|
||||
update_version = Column(String, nullable=True) # target version to update to
|
||||
|
||||
|
||||
class IgnoredUnit(Base):
|
||||
"""
|
||||
Ignored units: units that report but should be filtered out from unknown emitters.
|
||||
@@ -134,17 +162,31 @@ class Project(Base):
|
||||
"""
|
||||
Projects: top-level organization for monitoring work.
|
||||
Type-aware to enable/disable features based on project_type_id.
|
||||
|
||||
Project naming convention:
|
||||
- project_number: TMI internal ID format xxxx-YY (e.g., "2567-23")
|
||||
- client_name: Client/contractor name (e.g., "PJ Dick")
|
||||
- name: Project/site name (e.g., "RKM Hall", "CMU Campus")
|
||||
|
||||
Display format: "2567-23 - PJ Dick - RKM Hall"
|
||||
Users can search by any of these fields.
|
||||
"""
|
||||
__tablename__ = "projects"
|
||||
|
||||
id = Column(String, primary_key=True, index=True) # UUID
|
||||
name = Column(String, nullable=False, unique=True)
|
||||
project_number = Column(String, nullable=True, index=True) # TMI ID: xxxx-YY format (e.g., "2567-23")
|
||||
name = Column(String, nullable=False, unique=True) # Project/site name (e.g., "RKM Hall")
|
||||
description = Column(Text, nullable=True)
|
||||
project_type_id = Column(String, nullable=False) # FK to ProjectType.id
|
||||
status = Column(String, default="active") # active, completed, archived
|
||||
project_type_id = Column(String, nullable=True) # Legacy FK to ProjectType.id; use ProjectModule for feature flags
|
||||
status = Column(String, default="active") # active, on_hold, completed, archived, deleted
|
||||
|
||||
# Data collection mode: how field data reaches Terra-View.
|
||||
# "remote" — units have modems; data pulled via FTP/scheduler automatically
|
||||
# "manual" — no modem; SD cards retrieved daily and uploaded by hand
|
||||
data_collection_mode = Column(String, default="manual") # remote | manual
|
||||
|
||||
# Project metadata
|
||||
client_name = Column(String, nullable=True)
|
||||
client_name = Column(String, nullable=True, index=True) # Client name (e.g., "PJ Dick")
|
||||
site_address = Column(String, nullable=True)
|
||||
site_coordinates = Column(String, nullable=True) # "lat,lon"
|
||||
start_date = Column(Date, nullable=True)
|
||||
@@ -152,6 +194,23 @@ class Project(Base):
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
deleted_at = Column(DateTime, nullable=True) # Set when status='deleted'; hard delete scheduled after 60 days
|
||||
|
||||
|
||||
class ProjectModule(Base):
|
||||
"""
|
||||
Modules enabled on a project. Each module unlocks a set of features/tabs.
|
||||
A project can have zero or more modules (sound_monitoring, vibration_monitoring, etc.).
|
||||
"""
|
||||
__tablename__ = "project_modules"
|
||||
|
||||
id = Column(String, primary_key=True, default=lambda: __import__('uuid').uuid4().__str__())
|
||||
project_id = Column(String, nullable=False, index=True) # FK to projects.id
|
||||
module_type = Column(String, nullable=False) # sound_monitoring | vibration_monitoring | ...
|
||||
enabled = Column(Boolean, default=True, nullable=False)
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
__table_args__ = (UniqueConstraint("project_id", "module_type", name="uq_project_module"),)
|
||||
|
||||
|
||||
class MonitoringLocation(Base):
|
||||
@@ -197,12 +256,51 @@ class UnitAssignment(Base):
|
||||
notes = Column(Text, nullable=True)
|
||||
|
||||
# Denormalized for efficient queries
|
||||
device_type = Column(String, nullable=False) # sound_level_meter | seismograph
|
||||
device_type = Column(String, nullable=False) # "slm" | "seismograph"
|
||||
project_id = Column(String, nullable=False, index=True) # FK to Project.id
|
||||
|
||||
# Provenance: how was this assignment created? Used for auditing,
|
||||
# bulk-undo of parser actions, and the Phase 4 deployment timeline.
|
||||
# "manual" — operator created via UI
|
||||
# "metadata_backfill" — auto-created by the metadata parser
|
||||
# from operator-typed BW event metadata
|
||||
# (bulk backfill workflow)
|
||||
# "metadata_backfill_swap" — auto-created by swap-detection
|
||||
# background job
|
||||
source = Column(String, nullable=False, default="manual")
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class MetadataBackfillDecision(Base):
|
||||
"""
|
||||
Per-cluster decisions tracked by the metadata-backfill parser.
|
||||
|
||||
`cluster_id` is the deterministic SHA1 hash of
|
||||
(serial, first_event_date, last_event_date), so the same cluster
|
||||
produces the same id across re-scans. The decisions table lets the
|
||||
parser remember "I already applied this" or "operator skipped this"
|
||||
across scan invocations.
|
||||
"""
|
||||
__tablename__ = "metadata_backfill_decisions"
|
||||
|
||||
cluster_id = Column(String, primary_key=True)
|
||||
status = Column(String, nullable=False) # pending | applied | skipped | conflict
|
||||
confidence = Column(String, nullable=False) # high | medium | low
|
||||
decided_at = Column(DateTime, nullable=True)
|
||||
decided_by = Column(String, nullable=True) # background | operator | auto-high
|
||||
applied_assignment_id = Column(String, nullable=True) # FK to unit_assignments.id
|
||||
notes = Column(Text, nullable=True)
|
||||
first_seen_at = Column(DateTime, nullable=False, default=datetime.utcnow)
|
||||
last_seen_at = Column(DateTime, nullable=False, default=datetime.utcnow)
|
||||
serial = Column(String, nullable=False, index=True)
|
||||
project_raw = Column(String, nullable=True)
|
||||
location_raw = Column(String, nullable=True)
|
||||
first_event_ts = Column(DateTime, nullable=True)
|
||||
last_event_ts = Column(DateTime, nullable=True)
|
||||
event_count = Column(Integer, nullable=False, default=0)
|
||||
|
||||
|
||||
class ScheduledAction(Base):
|
||||
"""
|
||||
Scheduled actions: automation for recording start/stop/download.
|
||||
@@ -215,8 +313,8 @@ class ScheduledAction(Base):
|
||||
location_id = Column(String, nullable=False, index=True) # FK to MonitoringLocation.id
|
||||
unit_id = Column(String, nullable=True, index=True) # FK to RosterUnit.id (nullable if location-based)
|
||||
|
||||
action_type = Column(String, nullable=False) # start, stop, download, calibrate
|
||||
device_type = Column(String, nullable=False) # sound_level_meter | seismograph
|
||||
action_type = Column(String, nullable=False) # start, stop, download, cycle, calibrate
|
||||
device_type = Column(String, nullable=False) # "slm" | "seismograph"
|
||||
|
||||
scheduled_time = Column(DateTime, nullable=False, index=True)
|
||||
executed_at = Column(DateTime, nullable=True)
|
||||
@@ -230,17 +328,21 @@ class ScheduledAction(Base):
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class RecordingSession(Base):
|
||||
class MonitoringSession(Base):
|
||||
"""
|
||||
Recording sessions: tracks actual monitoring sessions.
|
||||
Created when recording starts, updated when it stops.
|
||||
Monitoring sessions: tracks actual monitoring sessions.
|
||||
Created when monitoring starts, updated when it stops.
|
||||
"""
|
||||
__tablename__ = "recording_sessions"
|
||||
__tablename__ = "monitoring_sessions"
|
||||
|
||||
id = Column(String, primary_key=True, index=True) # UUID
|
||||
project_id = Column(String, nullable=False, index=True) # FK to Project.id
|
||||
location_id = Column(String, nullable=False, index=True) # FK to MonitoringLocation.id
|
||||
unit_id = Column(String, nullable=False, index=True) # FK to RosterUnit.id
|
||||
unit_id = Column(String, nullable=True, index=True) # FK to RosterUnit.id (nullable for offline uploads)
|
||||
|
||||
# Physical device model that produced this session's data (e.g. "NL-43", "NL-53", "NL-32").
|
||||
# Null for older records; report code falls back to file-content detection when null.
|
||||
device_model = Column(String, nullable=True)
|
||||
|
||||
session_type = Column(String, nullable=False) # sound | vibration
|
||||
started_at = Column(DateTime, nullable=False)
|
||||
@@ -248,6 +350,25 @@ class RecordingSession(Base):
|
||||
duration_seconds = Column(Integer, nullable=True)
|
||||
status = Column(String, default="recording") # recording, completed, failed
|
||||
|
||||
# Human-readable label auto-derived from date/location, editable by user.
|
||||
# e.g. "NRL-1 — Sun 2/23 — Night"
|
||||
session_label = Column(String, nullable=True)
|
||||
|
||||
# Period classification for report stats columns.
|
||||
# weekday_day | weekday_night | weekend_day | weekend_night
|
||||
period_type = Column(String, nullable=True)
|
||||
|
||||
# Effective monitoring window (hours 0–23). Night sessions cross midnight
|
||||
# (period_end_hour < period_start_hour). NULL = no filtering applied.
|
||||
# e.g. Day: start=7, end=19 Night: start=19, end=7
|
||||
period_start_hour = Column(Integer, nullable=True)
|
||||
period_end_hour = Column(Integer, nullable=True)
|
||||
|
||||
# For day sessions: the specific calendar date to use for report filtering.
|
||||
# Overrides the automatic "last date with daytime rows" heuristic.
|
||||
# Null = use heuristic.
|
||||
report_date = Column(Date, nullable=True)
|
||||
|
||||
# Snapshot of device configuration at recording time
|
||||
session_metadata = Column(Text, nullable=True) # JSON
|
||||
|
||||
@@ -263,7 +384,7 @@ class DataFile(Base):
|
||||
__tablename__ = "data_files"
|
||||
|
||||
id = Column(String, primary_key=True, index=True) # UUID
|
||||
session_id = Column(String, nullable=False, index=True) # FK to RecordingSession.id
|
||||
session_id = Column(String, nullable=False, index=True) # FK to MonitoringSession.id
|
||||
|
||||
file_path = Column(String, nullable=False) # Relative to data/Projects/
|
||||
file_type = Column(String, nullable=False) # wav, csv, mseed, json
|
||||
@@ -275,3 +396,237 @@ class DataFile(Base):
|
||||
file_metadata = Column(Text, nullable=True) # JSON
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
|
||||
|
||||
class ReportTemplate(Base):
|
||||
"""
|
||||
Report templates: saved configurations for generating Excel reports.
|
||||
Allows users to save time filter presets, titles, etc. for reuse.
|
||||
"""
|
||||
__tablename__ = "report_templates"
|
||||
|
||||
id = Column(String, primary_key=True, index=True) # UUID
|
||||
name = Column(String, nullable=False) # "Nighttime Report", "Full Day Report"
|
||||
project_id = Column(String, nullable=True) # Optional: project-specific template
|
||||
|
||||
# Template settings
|
||||
report_title = Column(String, default="Background Noise Study")
|
||||
start_time = Column(String, nullable=True) # "19:00" format
|
||||
end_time = Column(String, nullable=True) # "07:00" format
|
||||
start_date = Column(String, nullable=True) # "2025-01-15" format (optional)
|
||||
end_date = Column(String, nullable=True) # "2025-01-20" format (optional)
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Sound Monitoring Scheduler
|
||||
# ============================================================================
|
||||
|
||||
class RecurringSchedule(Base):
|
||||
"""
|
||||
Recurring schedule definitions for automated sound monitoring.
|
||||
|
||||
Supports three schedule types:
|
||||
- "weekly_calendar": Select specific days with start/end times (e.g., Mon/Wed/Fri 7pm-7am)
|
||||
- "simple_interval": For 24/7 monitoring with daily stop/download/restart cycles
|
||||
- "one_off": Single recording session with specific start and end date/time
|
||||
"""
|
||||
__tablename__ = "recurring_schedules"
|
||||
|
||||
id = Column(String, primary_key=True, index=True) # UUID
|
||||
project_id = Column(String, nullable=False, index=True) # FK to Project.id
|
||||
location_id = Column(String, nullable=False, index=True) # FK to MonitoringLocation.id
|
||||
unit_id = Column(String, nullable=True, index=True) # FK to RosterUnit.id (optional, can use assignment)
|
||||
|
||||
name = Column(String, nullable=False) # "Weeknight Monitoring", "24/7 Continuous"
|
||||
schedule_type = Column(String, nullable=False) # "weekly_calendar" | "simple_interval" | "one_off"
|
||||
device_type = Column(String, nullable=False) # "slm" | "seismograph"
|
||||
|
||||
# Weekly Calendar fields (schedule_type = "weekly_calendar")
|
||||
# JSON format: {
|
||||
# "monday": {"enabled": true, "start": "19:00", "end": "07:00"},
|
||||
# "tuesday": {"enabled": false},
|
||||
# ...
|
||||
# }
|
||||
weekly_pattern = Column(Text, nullable=True)
|
||||
|
||||
# Simple Interval fields (schedule_type = "simple_interval")
|
||||
interval_type = Column(String, nullable=True) # "daily" | "hourly"
|
||||
cycle_time = Column(String, nullable=True) # "00:00" - time to run stop/download/restart
|
||||
include_download = Column(Boolean, default=True) # Download data before restart
|
||||
|
||||
# One-Off fields (schedule_type = "one_off")
|
||||
start_datetime = Column(DateTime, nullable=True) # Exact start date+time (stored as UTC)
|
||||
end_datetime = Column(DateTime, nullable=True) # Exact end date+time (stored as UTC)
|
||||
|
||||
# Automation options (applies to all schedule types)
|
||||
auto_increment_index = Column(Boolean, default=True) # Auto-increment store/index number before start
|
||||
# When True: prevents "overwrite data?" prompts by using a new index each time
|
||||
|
||||
# Shared configuration
|
||||
enabled = Column(Boolean, default=True)
|
||||
timezone = Column(String, default="America/New_York")
|
||||
|
||||
# Tracking
|
||||
last_generated_at = Column(DateTime, nullable=True) # When actions were last generated
|
||||
next_occurrence = Column(DateTime, nullable=True) # Computed next action time
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
|
||||
class Alert(Base):
|
||||
"""
|
||||
In-app alerts for device status changes and system events.
|
||||
|
||||
Designed for future expansion to email/webhook notifications.
|
||||
Currently supports:
|
||||
- device_offline: Device became unreachable
|
||||
- device_online: Device came back online
|
||||
- schedule_failed: Scheduled action failed to execute
|
||||
- schedule_completed: Scheduled action completed successfully
|
||||
"""
|
||||
__tablename__ = "alerts"
|
||||
|
||||
id = Column(String, primary_key=True, index=True) # UUID
|
||||
|
||||
# Alert classification
|
||||
alert_type = Column(String, nullable=False) # "device_offline" | "device_online" | "schedule_failed" | "schedule_completed"
|
||||
severity = Column(String, default="warning") # "info" | "warning" | "critical"
|
||||
|
||||
# Related entities (nullable - may not all apply)
|
||||
project_id = Column(String, nullable=True, index=True)
|
||||
location_id = Column(String, nullable=True, index=True)
|
||||
unit_id = Column(String, nullable=True, index=True)
|
||||
schedule_id = Column(String, nullable=True) # RecurringSchedule or ScheduledAction id
|
||||
|
||||
# Alert content
|
||||
title = Column(String, nullable=False) # "NRL-001 Device Offline"
|
||||
message = Column(Text, nullable=True) # Detailed description
|
||||
alert_metadata = Column(Text, nullable=True) # JSON: additional context data
|
||||
|
||||
# Status tracking
|
||||
status = Column(String, default="active") # "active" | "acknowledged" | "resolved" | "dismissed"
|
||||
acknowledged_at = Column(DateTime, nullable=True)
|
||||
resolved_at = Column(DateTime, nullable=True)
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
expires_at = Column(DateTime, nullable=True) # Auto-dismiss after this time
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Deployment Records
|
||||
# ============================================================================
|
||||
|
||||
class DeploymentRecord(Base):
|
||||
"""
|
||||
Deployment records: tracks each time a unit is sent to the field and returned.
|
||||
|
||||
Each row represents one deployment. The active deployment is the record
|
||||
with actual_removal_date IS NULL. The fleet calendar uses this to show
|
||||
units as "In Field" and surface their expected return date.
|
||||
|
||||
project_ref is a freeform string for legacy/vibration jobs like "Fay I-80".
|
||||
project_id will be populated once those jobs are migrated to proper Project records.
|
||||
"""
|
||||
__tablename__ = "deployment_records"
|
||||
|
||||
id = Column(String, primary_key=True, index=True) # UUID
|
||||
unit_id = Column(String, nullable=False, index=True) # FK to RosterUnit.id
|
||||
|
||||
deployed_date = Column(Date, nullable=True) # When unit left the yard
|
||||
estimated_removal_date = Column(Date, nullable=True) # Expected return date
|
||||
actual_removal_date = Column(Date, nullable=True) # Filled in when returned; NULL = still out
|
||||
|
||||
# Project linkage: freeform for legacy jobs, FK for proper project records
|
||||
project_ref = Column(String, nullable=True) # e.g. "Fay I-80" (vibration jobs)
|
||||
project_id = Column(String, nullable=True, index=True) # FK to Project.id (when available)
|
||||
|
||||
location_name = Column(String, nullable=True) # e.g. "North Gate", "VP-001"
|
||||
notes = Column(Text, nullable=True)
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Fleet Calendar & Job Reservations
|
||||
# ============================================================================
|
||||
|
||||
class JobReservation(Base):
|
||||
"""
|
||||
Job reservations: reserve units for future jobs/projects.
|
||||
|
||||
Supports two assignment modes:
|
||||
- "specific": Pick exact units (SN-001, SN-002, etc.)
|
||||
- "quantity": Reserve a number of units (e.g., "need 8 seismographs")
|
||||
|
||||
Used by the Fleet Calendar to visualize unit availability over time.
|
||||
"""
|
||||
__tablename__ = "job_reservations"
|
||||
|
||||
id = Column(String, primary_key=True, index=True) # UUID
|
||||
name = Column(String, nullable=False) # "Job A - March deployment"
|
||||
project_id = Column(String, nullable=True, index=True) # Optional FK to Project
|
||||
|
||||
# Date range for the reservation
|
||||
start_date = Column(Date, nullable=False)
|
||||
end_date = Column(Date, nullable=True) # Nullable = TBD / ongoing
|
||||
estimated_end_date = Column(Date, nullable=True) # For planning when end is TBD
|
||||
end_date_tbd = Column(Boolean, default=False) # True = end date unknown
|
||||
|
||||
# Assignment type: "specific" or "quantity"
|
||||
assignment_type = Column(String, nullable=False, default="quantity")
|
||||
|
||||
# For quantity reservations
|
||||
device_type = Column(String, default="seismograph") # seismograph | slm
|
||||
quantity_needed = Column(Integer, nullable=True) # e.g., 8 units
|
||||
estimated_units = Column(Integer, nullable=True)
|
||||
|
||||
# Full slot list as JSON: [{"location_name": "North Gate", "unit_id": null}, ...]
|
||||
# Includes empty slots (no unit assigned yet). Filled slots are authoritative in JobReservationUnit.
|
||||
location_slots = Column(Text, nullable=True)
|
||||
|
||||
# Metadata
|
||||
notes = Column(Text, nullable=True)
|
||||
color = Column(String, default="#3B82F6") # For calendar display (blue default)
|
||||
|
||||
created_at = Column(DateTime, default=datetime.utcnow)
|
||||
updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow)
|
||||
|
||||
|
||||
class JobReservationUnit(Base):
|
||||
"""
|
||||
Links specific units to job reservations.
|
||||
|
||||
Used when:
|
||||
- assignment_type="specific": Units are directly assigned
|
||||
- assignment_type="quantity": Units can be filled in later
|
||||
|
||||
Supports unit swaps: same reservation can have multiple units with
|
||||
different date ranges (e.g., BE17353 Feb-Jun, then BE18438 Jun-Nov).
|
||||
"""
|
||||
__tablename__ = "job_reservation_units"
|
||||
|
||||
id = Column(String, primary_key=True, index=True) # UUID
|
||||
reservation_id = Column(String, nullable=False, index=True) # FK to JobReservation
|
||||
unit_id = Column(String, nullable=False, index=True) # FK to RosterUnit
|
||||
|
||||
# Unit-specific date range (for swaps) - defaults to reservation dates if null
|
||||
unit_start_date = Column(Date, nullable=True) # When this specific unit starts
|
||||
unit_end_date = Column(Date, nullable=True) # When this unit ends (swap out date)
|
||||
unit_end_tbd = Column(Boolean, default=False) # True = end unknown (until cal expires or job ends)
|
||||
|
||||
# Track how this assignment was made
|
||||
assignment_source = Column(String, default="specific") # "specific" | "filled" | "swap"
|
||||
assigned_at = Column(DateTime, default=datetime.utcnow)
|
||||
notes = Column(Text, nullable=True) # "Replacing BE17353" etc.
|
||||
|
||||
# Power requirements for this deployment slot
|
||||
power_type = Column(String, nullable=True) # "ac" | "solar" | None
|
||||
|
||||
# Location identity
|
||||
location_name = Column(String, nullable=True) # e.g. "North Gate", "Main Entrance"
|
||||
slot_index = Column(Integer, nullable=True) # Order within reservation (0-based)
|
||||
|
||||
@@ -4,11 +4,29 @@ from sqlalchemy import desc
|
||||
from pathlib import Path
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from typing import List, Dict, Any
|
||||
import os
|
||||
import logging
|
||||
import httpx
|
||||
from backend.database import get_db
|
||||
from backend.models import UnitHistory, Emitter, RosterUnit
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["activity"])
|
||||
|
||||
SFM_BASE_URL = os.getenv("SFM_BASE_URL", "http://localhost:8200")
|
||||
|
||||
|
||||
def _humanize_age(seconds: float) -> str:
|
||||
if seconds < 60:
|
||||
return "just now"
|
||||
if seconds < 3600:
|
||||
return f"{int(seconds / 60)}m ago"
|
||||
if seconds < 86400:
|
||||
hrs = seconds / 3600
|
||||
return f"{int(hrs)}h {int((hrs % 1) * 60)}m ago"
|
||||
return f"{int(seconds / 86400)}d ago"
|
||||
|
||||
PHOTOS_BASE_DIR = Path("data/photos")
|
||||
|
||||
|
||||
@@ -144,3 +162,86 @@ def get_recent_callins(hours: int = 6, limit: int = None, db: Session = Depends(
|
||||
"hours": hours,
|
||||
"time_threshold": time_threshold.isoformat()
|
||||
}
|
||||
|
||||
|
||||
@router.get("/recent-event-callins")
|
||||
async def get_recent_event_callins(limit: int = 10, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Recent unit call-ins derived from SFM event forwards.
|
||||
|
||||
Architecture context: the live ACH replacement is on hold, so call-homes
|
||||
arrive as Blastware ACH event files forwarded by series3-watcher and
|
||||
landed in the SFM events store. One event ≈ one call-in. This is the
|
||||
forward-looking source of "recent call-ins" that will eventually replace
|
||||
the heartbeat-based /recent-callins endpoint entirely.
|
||||
|
||||
Each row represents one event; multiple consecutive events from the same
|
||||
serial are intentionally NOT collapsed — each one is a distinct call-home.
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=10.0) as client:
|
||||
resp = await client.get(
|
||||
f"{SFM_BASE_URL}/db/events",
|
||||
params={"limit": limit},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
payload = resp.json()
|
||||
except httpx.HTTPError as e:
|
||||
log.warning("SFM /db/events failed for recent-event-callins: %s", e)
|
||||
return {"call_ins": [], "total": 0, "error": str(e)}
|
||||
|
||||
events = payload.get("events", []) or []
|
||||
|
||||
# Bulk-resolve serials → roster (single query, no N+1)
|
||||
serials = list({ev.get("serial") for ev in events if ev.get("serial")})
|
||||
roster_map: Dict[str, RosterUnit] = {}
|
||||
if serials:
|
||||
roster_map = {
|
||||
r.id: r
|
||||
for r in db.query(RosterUnit).filter(RosterUnit.id.in_(serials)).all()
|
||||
}
|
||||
|
||||
now = datetime.now(timezone.utc)
|
||||
call_ins: List[Dict[str, Any]] = []
|
||||
|
||||
for ev in events:
|
||||
serial = ev.get("serial")
|
||||
if not serial:
|
||||
continue
|
||||
|
||||
roster = roster_map.get(serial)
|
||||
|
||||
# created_at = when SFM received the forward. Falls back to the event
|
||||
# timestamp if the SFM payload didn't carry created_at (older rows).
|
||||
created_at_str = ev.get("created_at") or ev.get("timestamp")
|
||||
time_ago = "—"
|
||||
if created_at_str:
|
||||
try:
|
||||
ts = datetime.fromisoformat(created_at_str.replace("Z", "+00:00"))
|
||||
if ts.tzinfo is None:
|
||||
ts = ts.replace(tzinfo=timezone.utc)
|
||||
time_ago = _humanize_age((now - ts).total_seconds())
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
call_ins.append({
|
||||
"unit_id": serial,
|
||||
"serial": serial,
|
||||
"event_id": ev.get("id"),
|
||||
"event_timestamp": ev.get("timestamp"),
|
||||
"created_at": ev.get("created_at"),
|
||||
"time_ago": time_ago,
|
||||
"peak_vector_sum": ev.get("peak_vector_sum"),
|
||||
"false_trigger": bool(ev.get("false_trigger")),
|
||||
"sensor_location": ev.get("sensor_location") or "",
|
||||
"project": ev.get("project") or "",
|
||||
"device_type": roster.device_type if roster else "seismograph",
|
||||
"in_roster": roster is not None,
|
||||
"note": (roster.note if roster else "") or "",
|
||||
})
|
||||
|
||||
return {
|
||||
"call_ins": call_ins,
|
||||
"total": len(call_ins),
|
||||
"source": "sfm-events",
|
||||
}
|
||||
|
||||
@@ -0,0 +1,209 @@
|
||||
"""
|
||||
Admin / diagnostic pages for the device modules (SFM, SLMM).
|
||||
|
||||
These pages live under /admin/{module} and exist purely so an operator can
|
||||
peek under the hood and confirm the module is reachable, what data it's
|
||||
holding, and whether the proxy from terra-view is healthy.
|
||||
|
||||
Routes:
|
||||
GET /admin/sfm — SFM diagnostic page
|
||||
GET /admin/slmm — SLMM diagnostic page
|
||||
|
||||
API helpers (called by the HTML pages via fetch):
|
||||
GET /api/admin/sfm/overview — aggregated SFM health + db stats in one call
|
||||
GET /api/admin/slmm/overview — aggregated SLMM health + device count
|
||||
|
||||
The pages are intentionally read-only. Any actual administration of SFM
|
||||
or SLMM happens in those modules directly.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any, Dict
|
||||
|
||||
import httpx
|
||||
from fastapi import APIRouter, Depends, Request
|
||||
from fastapi.responses import HTMLResponse, JSONResponse
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.templates_config import templates
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
SFM_BASE_URL = os.getenv("SFM_BASE_URL", "http://localhost:8200")
|
||||
SLMM_BASE_URL = os.getenv("SLMM_BASE_URL", "http://localhost:8100")
|
||||
|
||||
|
||||
# ── SFM ───────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/admin/sfm", response_class=HTMLResponse)
|
||||
def admin_sfm_page(request: Request):
|
||||
return templates.TemplateResponse("admin_sfm.html", {
|
||||
"request": request,
|
||||
"sfm_base_url": SFM_BASE_URL,
|
||||
})
|
||||
|
||||
|
||||
@router.get("/api/admin/sfm/overview")
|
||||
async def admin_sfm_overview() -> JSONResponse:
|
||||
"""Aggregated SFM diagnostic snapshot.
|
||||
|
||||
Returns health, db stats, stale-table counts, per-unit summary, and
|
||||
recent events with forwarding latency. Tolerant of partial failures:
|
||||
any individual sub-fetch error is captured into its section, so a flaky
|
||||
sub-endpoint doesn't break the whole page.
|
||||
"""
|
||||
overview: Dict[str, Any] = {
|
||||
"sfm_base_url": SFM_BASE_URL,
|
||||
"checked_at": datetime.now(timezone.utc).isoformat(),
|
||||
"health": None,
|
||||
"reachable": False,
|
||||
"units": [],
|
||||
"events": [],
|
||||
"stale": {
|
||||
"monitor_log": None,
|
||||
"sessions": None,
|
||||
},
|
||||
"cache_stats": None,
|
||||
"errors": {},
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
# Health
|
||||
try:
|
||||
r = await client.get(f"{SFM_BASE_URL}/health")
|
||||
r.raise_for_status()
|
||||
overview["health"] = r.json()
|
||||
overview["reachable"] = overview["health"].get("status") == "ok"
|
||||
except Exception as e: # noqa: BLE001
|
||||
overview["errors"]["health"] = str(e)
|
||||
overview["reachable"] = False
|
||||
|
||||
# If SFM is down, no point hitting the rest.
|
||||
if not overview["reachable"]:
|
||||
return JSONResponse(overview)
|
||||
|
||||
# Units
|
||||
try:
|
||||
r = await client.get(f"{SFM_BASE_URL}/db/units")
|
||||
r.raise_for_status()
|
||||
overview["units"] = r.json() or []
|
||||
except Exception as e: # noqa: BLE001
|
||||
overview["errors"]["units"] = str(e)
|
||||
|
||||
# Recent events (newest 25 — bigger sample of the call-home stream)
|
||||
try:
|
||||
r = await client.get(f"{SFM_BASE_URL}/db/events", params={"limit": 25})
|
||||
r.raise_for_status()
|
||||
payload = r.json() or {}
|
||||
events = payload.get("events", []) or []
|
||||
# Compute forwarding latency: created_at (SFM ingest) − timestamp (event).
|
||||
now = datetime.now(timezone.utc)
|
||||
for ev in events:
|
||||
ev.pop("waveform_blob", None)
|
||||
ev.pop("a5_pickle_filename", None)
|
||||
ts_str = ev.get("timestamp")
|
||||
ca_str = ev.get("created_at")
|
||||
latency_seconds = None
|
||||
try:
|
||||
if ts_str and ca_str:
|
||||
ts = datetime.fromisoformat(ts_str.replace("Z", "+00:00"))
|
||||
ca = datetime.fromisoformat(ca_str.replace("Z", "+00:00"))
|
||||
if ts.tzinfo is None: ts = ts.replace(tzinfo=timezone.utc)
|
||||
if ca.tzinfo is None: ca = ca.replace(tzinfo=timezone.utc)
|
||||
latency_seconds = (ca - ts).total_seconds()
|
||||
except ValueError:
|
||||
pass
|
||||
ev["forwarding_latency_seconds"] = latency_seconds
|
||||
overview["events"] = events
|
||||
except Exception as e: # noqa: BLE001
|
||||
overview["errors"]["events"] = str(e)
|
||||
|
||||
# Stale tables (deprecated by the watcher-forward pipeline but still
|
||||
# present in SFM's SQLite). Surface as counts only.
|
||||
for key, path in (("monitor_log", "/db/monitor_log"),
|
||||
("sessions", "/db/sessions")):
|
||||
try:
|
||||
r = await client.get(f"{SFM_BASE_URL}{path}", params={"limit": 1})
|
||||
r.raise_for_status()
|
||||
payload = r.json() or {}
|
||||
# SFM returns count = total when limit covers all rows; we
|
||||
# query with limit=1 just to be polite, then ask again with
|
||||
# a high limit if we need the real total.
|
||||
first_count = payload.get("count")
|
||||
if first_count is None:
|
||||
overview["stale"][key] = None
|
||||
continue
|
||||
# Re-query with high limit to get the true total.
|
||||
r2 = await client.get(f"{SFM_BASE_URL}{path}", params={"limit": 100000})
|
||||
r2.raise_for_status()
|
||||
overview["stale"][key] = (r2.json() or {}).get("count")
|
||||
except Exception as e: # noqa: BLE001
|
||||
overview["errors"][f"stale_{key}"] = str(e)
|
||||
|
||||
# Cache stats (in-memory device cache on SFM)
|
||||
try:
|
||||
r = await client.get(f"{SFM_BASE_URL}/cache/stats")
|
||||
r.raise_for_status()
|
||||
overview["cache_stats"] = r.json()
|
||||
except Exception as e: # noqa: BLE001
|
||||
overview["errors"]["cache_stats"] = str(e)
|
||||
|
||||
# Aggregate counts the UI can render without re-walking arrays
|
||||
overview["totals"] = {
|
||||
"units": len(overview["units"]),
|
||||
"events_total": sum(u.get("total_events", 0) for u in overview["units"]),
|
||||
"stale_monitor_log": overview["stale"]["monitor_log"],
|
||||
"stale_sessions": overview["stale"]["sessions"],
|
||||
}
|
||||
|
||||
return JSONResponse(overview)
|
||||
|
||||
|
||||
# ── SLMM ──────────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@router.get("/admin/slmm", response_class=HTMLResponse)
|
||||
def admin_slmm_page(request: Request):
|
||||
return templates.TemplateResponse("admin_slmm.html", {
|
||||
"request": request,
|
||||
"slmm_base_url": SLMM_BASE_URL,
|
||||
})
|
||||
|
||||
|
||||
@router.get("/api/admin/slmm/overview")
|
||||
async def admin_slmm_overview() -> JSONResponse:
|
||||
"""Aggregated SLMM diagnostic snapshot."""
|
||||
overview: Dict[str, Any] = {
|
||||
"slmm_base_url": SLMM_BASE_URL,
|
||||
"checked_at": datetime.now(timezone.utc).isoformat(),
|
||||
"health": None,
|
||||
"reachable": False,
|
||||
"devices": [],
|
||||
"errors": {},
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
try:
|
||||
r = await client.get(f"{SLMM_BASE_URL}/health")
|
||||
r.raise_for_status()
|
||||
overview["health"] = r.json()
|
||||
overview["reachable"] = True
|
||||
except Exception as e: # noqa: BLE001
|
||||
overview["errors"]["health"] = str(e)
|
||||
return JSONResponse(overview)
|
||||
|
||||
# Pull a roster of configured devices (SLMM exposes per-unit
|
||||
# config + status under /api/nl43/*). This is a best-effort probe
|
||||
# — SLMM doesn't expose a "list all devices" endpoint, so we ask
|
||||
# terra-view's RosterUnit table what serials it knows about for
|
||||
# SLMs and just check each one. For now, just surface the health
|
||||
# payload and let the operator click through to /sound-level-meters
|
||||
# for the per-device details.
|
||||
|
||||
return JSONResponse(overview)
|
||||
@@ -0,0 +1,326 @@
|
||||
"""
|
||||
Alerts Router
|
||||
|
||||
API endpoints for managing in-app alerts.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Request, Depends, HTTPException, Query
|
||||
from fastapi.responses import HTMLResponse, JSONResponse
|
||||
from sqlalchemy.orm import Session
|
||||
from typing import Optional
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import Alert, RosterUnit
|
||||
from backend.services.alert_service import get_alert_service
|
||||
from backend.templates_config import templates
|
||||
|
||||
router = APIRouter(prefix="/api/alerts", tags=["alerts"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Alert List and Count
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/")
|
||||
async def list_alerts(
|
||||
db: Session = Depends(get_db),
|
||||
status: Optional[str] = Query(None, description="Filter by status: active, acknowledged, resolved, dismissed"),
|
||||
project_id: Optional[str] = Query(None),
|
||||
unit_id: Optional[str] = Query(None),
|
||||
alert_type: Optional[str] = Query(None, description="Filter by type: device_offline, device_online, schedule_failed"),
|
||||
limit: int = Query(50, le=100),
|
||||
offset: int = Query(0, ge=0),
|
||||
):
|
||||
"""
|
||||
List alerts with optional filters.
|
||||
"""
|
||||
alert_service = get_alert_service(db)
|
||||
|
||||
alerts = alert_service.get_all_alerts(
|
||||
status=status,
|
||||
project_id=project_id,
|
||||
unit_id=unit_id,
|
||||
alert_type=alert_type,
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
)
|
||||
|
||||
return {
|
||||
"alerts": [
|
||||
{
|
||||
"id": a.id,
|
||||
"alert_type": a.alert_type,
|
||||
"severity": a.severity,
|
||||
"title": a.title,
|
||||
"message": a.message,
|
||||
"status": a.status,
|
||||
"unit_id": a.unit_id,
|
||||
"project_id": a.project_id,
|
||||
"location_id": a.location_id,
|
||||
"created_at": a.created_at.isoformat() if a.created_at else None,
|
||||
"acknowledged_at": a.acknowledged_at.isoformat() if a.acknowledged_at else None,
|
||||
"resolved_at": a.resolved_at.isoformat() if a.resolved_at else None,
|
||||
}
|
||||
for a in alerts
|
||||
],
|
||||
"count": len(alerts),
|
||||
"limit": limit,
|
||||
"offset": offset,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/active")
|
||||
async def list_active_alerts(
|
||||
db: Session = Depends(get_db),
|
||||
project_id: Optional[str] = Query(None),
|
||||
unit_id: Optional[str] = Query(None),
|
||||
alert_type: Optional[str] = Query(None),
|
||||
min_severity: Optional[str] = Query(None, description="Minimum severity: info, warning, critical"),
|
||||
limit: int = Query(50, le=100),
|
||||
):
|
||||
"""
|
||||
List only active alerts.
|
||||
"""
|
||||
alert_service = get_alert_service(db)
|
||||
|
||||
alerts = alert_service.get_active_alerts(
|
||||
project_id=project_id,
|
||||
unit_id=unit_id,
|
||||
alert_type=alert_type,
|
||||
min_severity=min_severity,
|
||||
limit=limit,
|
||||
)
|
||||
|
||||
return {
|
||||
"alerts": [
|
||||
{
|
||||
"id": a.id,
|
||||
"alert_type": a.alert_type,
|
||||
"severity": a.severity,
|
||||
"title": a.title,
|
||||
"message": a.message,
|
||||
"unit_id": a.unit_id,
|
||||
"project_id": a.project_id,
|
||||
"created_at": a.created_at.isoformat() if a.created_at else None,
|
||||
}
|
||||
for a in alerts
|
||||
],
|
||||
"count": len(alerts),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/active/count")
|
||||
async def get_active_alert_count(db: Session = Depends(get_db)):
|
||||
"""
|
||||
Get count of active alerts (for navbar badge).
|
||||
"""
|
||||
alert_service = get_alert_service(db)
|
||||
count = alert_service.get_active_alert_count()
|
||||
return {"count": count}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Single Alert Operations
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/{alert_id}")
|
||||
async def get_alert(
|
||||
alert_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Get a specific alert.
|
||||
"""
|
||||
alert = db.query(Alert).filter_by(id=alert_id).first()
|
||||
if not alert:
|
||||
raise HTTPException(status_code=404, detail="Alert not found")
|
||||
|
||||
# Get related unit info
|
||||
unit = None
|
||||
if alert.unit_id:
|
||||
unit = db.query(RosterUnit).filter_by(id=alert.unit_id).first()
|
||||
|
||||
return {
|
||||
"id": alert.id,
|
||||
"alert_type": alert.alert_type,
|
||||
"severity": alert.severity,
|
||||
"title": alert.title,
|
||||
"message": alert.message,
|
||||
"metadata": alert.alert_metadata,
|
||||
"status": alert.status,
|
||||
"unit_id": alert.unit_id,
|
||||
"unit_name": unit.id if unit else None,
|
||||
"project_id": alert.project_id,
|
||||
"location_id": alert.location_id,
|
||||
"schedule_id": alert.schedule_id,
|
||||
"created_at": alert.created_at.isoformat() if alert.created_at else None,
|
||||
"acknowledged_at": alert.acknowledged_at.isoformat() if alert.acknowledged_at else None,
|
||||
"resolved_at": alert.resolved_at.isoformat() if alert.resolved_at else None,
|
||||
"expires_at": alert.expires_at.isoformat() if alert.expires_at else None,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{alert_id}/acknowledge")
|
||||
async def acknowledge_alert(
|
||||
alert_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Mark alert as acknowledged.
|
||||
"""
|
||||
alert_service = get_alert_service(db)
|
||||
alert = alert_service.acknowledge_alert(alert_id)
|
||||
|
||||
if not alert:
|
||||
raise HTTPException(status_code=404, detail="Alert not found")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"alert_id": alert.id,
|
||||
"status": alert.status,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{alert_id}/dismiss")
|
||||
async def dismiss_alert(
|
||||
alert_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Dismiss alert.
|
||||
"""
|
||||
alert_service = get_alert_service(db)
|
||||
alert = alert_service.dismiss_alert(alert_id)
|
||||
|
||||
if not alert:
|
||||
raise HTTPException(status_code=404, detail="Alert not found")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"alert_id": alert.id,
|
||||
"status": alert.status,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{alert_id}/resolve")
|
||||
async def resolve_alert(
|
||||
alert_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Manually resolve an alert.
|
||||
"""
|
||||
alert_service = get_alert_service(db)
|
||||
alert = alert_service.resolve_alert(alert_id)
|
||||
|
||||
if not alert:
|
||||
raise HTTPException(status_code=404, detail="Alert not found")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"alert_id": alert.id,
|
||||
"status": alert.status,
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# HTML Partials for HTMX
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/partials/dropdown", response_class=HTMLResponse)
|
||||
async def get_alert_dropdown(
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Return HTML partial for alert dropdown in navbar.
|
||||
"""
|
||||
alert_service = get_alert_service(db)
|
||||
alerts = alert_service.get_active_alerts(limit=10)
|
||||
|
||||
# Calculate relative time for each alert
|
||||
now = datetime.utcnow()
|
||||
alerts_data = []
|
||||
for alert in alerts:
|
||||
delta = now - alert.created_at
|
||||
if delta.days > 0:
|
||||
time_ago = f"{delta.days}d ago"
|
||||
elif delta.seconds >= 3600:
|
||||
time_ago = f"{delta.seconds // 3600}h ago"
|
||||
elif delta.seconds >= 60:
|
||||
time_ago = f"{delta.seconds // 60}m ago"
|
||||
else:
|
||||
time_ago = "just now"
|
||||
|
||||
alerts_data.append({
|
||||
"alert": alert,
|
||||
"time_ago": time_ago,
|
||||
})
|
||||
|
||||
return templates.TemplateResponse("partials/alerts/alert_dropdown.html", {
|
||||
"request": request,
|
||||
"alerts": alerts_data,
|
||||
"total_count": alert_service.get_active_alert_count(),
|
||||
})
|
||||
|
||||
|
||||
@router.get("/partials/list", response_class=HTMLResponse)
|
||||
async def get_alert_list(
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
status: Optional[str] = Query(None),
|
||||
limit: int = Query(20),
|
||||
):
|
||||
"""
|
||||
Return HTML partial for alert list page.
|
||||
"""
|
||||
alert_service = get_alert_service(db)
|
||||
|
||||
if status:
|
||||
alerts = alert_service.get_all_alerts(status=status, limit=limit)
|
||||
else:
|
||||
alerts = alert_service.get_all_alerts(limit=limit)
|
||||
|
||||
# Calculate relative time for each alert
|
||||
now = datetime.utcnow()
|
||||
alerts_data = []
|
||||
for alert in alerts:
|
||||
delta = now - alert.created_at
|
||||
if delta.days > 0:
|
||||
time_ago = f"{delta.days}d ago"
|
||||
elif delta.seconds >= 3600:
|
||||
time_ago = f"{delta.seconds // 3600}h ago"
|
||||
elif delta.seconds >= 60:
|
||||
time_ago = f"{delta.seconds // 60}m ago"
|
||||
else:
|
||||
time_ago = "just now"
|
||||
|
||||
alerts_data.append({
|
||||
"alert": alert,
|
||||
"time_ago": time_ago,
|
||||
})
|
||||
|
||||
return templates.TemplateResponse("partials/alerts/alert_list.html", {
|
||||
"request": request,
|
||||
"alerts": alerts_data,
|
||||
"status_filter": status,
|
||||
})
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Cleanup
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/cleanup-expired")
|
||||
async def cleanup_expired_alerts(db: Session = Depends(get_db)):
|
||||
"""
|
||||
Cleanup expired alerts (admin/maintenance endpoint).
|
||||
"""
|
||||
alert_service = get_alert_service(db)
|
||||
count = alert_service.cleanup_expired_alerts()
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"cleaned_up": count,
|
||||
}
|
||||
@@ -1,10 +1,15 @@
|
||||
from fastapi import APIRouter, Request, Depends
|
||||
from fastapi.templating import Jinja2Templates
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import and_
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import ScheduledAction, MonitoringLocation, Project
|
||||
from backend.services.snapshot import emit_status_snapshot
|
||||
from backend.templates_config import templates
|
||||
from backend.utils.timezone import utc_to_local, local_to_utc, get_user_timezone
|
||||
|
||||
router = APIRouter()
|
||||
templates = Jinja2Templates(directory="templates")
|
||||
|
||||
|
||||
@router.get("/dashboard/active")
|
||||
@@ -23,3 +28,79 @@ def dashboard_benched(request: Request):
|
||||
"partials/benched_table.html",
|
||||
{"request": request, "units": snapshot["benched"]}
|
||||
)
|
||||
|
||||
|
||||
@router.get("/dashboard/todays-actions")
|
||||
def dashboard_todays_actions(request: Request, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Get today's scheduled actions for the dashboard card.
|
||||
Shows upcoming, completed, and failed actions for today.
|
||||
"""
|
||||
import json
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
# Get today's date range in local timezone
|
||||
tz = ZoneInfo(get_user_timezone())
|
||||
now_local = datetime.now(tz)
|
||||
today_start_local = now_local.replace(hour=0, minute=0, second=0, microsecond=0)
|
||||
today_end_local = today_start_local + timedelta(days=1)
|
||||
|
||||
# Convert to UTC for database query
|
||||
today_start_utc = today_start_local.astimezone(ZoneInfo("UTC")).replace(tzinfo=None)
|
||||
today_end_utc = today_end_local.astimezone(ZoneInfo("UTC")).replace(tzinfo=None)
|
||||
|
||||
# Exclude actions from paused/removed projects
|
||||
paused_project_ids = [
|
||||
p.id for p in db.query(Project.id).filter(
|
||||
Project.status.in_(["on_hold", "archived", "deleted"])
|
||||
).all()
|
||||
]
|
||||
|
||||
# Query today's actions
|
||||
actions = db.query(ScheduledAction).filter(
|
||||
ScheduledAction.scheduled_time >= today_start_utc,
|
||||
ScheduledAction.scheduled_time < today_end_utc,
|
||||
ScheduledAction.project_id.notin_(paused_project_ids),
|
||||
).order_by(ScheduledAction.scheduled_time.asc()).all()
|
||||
|
||||
# Enrich with location/project info and parse results
|
||||
enriched_actions = []
|
||||
for action in actions:
|
||||
location = None
|
||||
project = None
|
||||
if action.location_id:
|
||||
location = db.query(MonitoringLocation).filter_by(id=action.location_id).first()
|
||||
if action.project_id:
|
||||
project = db.query(Project).filter_by(id=action.project_id).first()
|
||||
|
||||
# Parse module_response for result details
|
||||
result_data = None
|
||||
if action.module_response:
|
||||
try:
|
||||
result_data = json.loads(action.module_response)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
enriched_actions.append({
|
||||
"action": action,
|
||||
"location": location,
|
||||
"project": project,
|
||||
"result": result_data,
|
||||
})
|
||||
|
||||
# Count by status
|
||||
pending_count = sum(1 for a in actions if a.execution_status == "pending")
|
||||
completed_count = sum(1 for a in actions if a.execution_status == "completed")
|
||||
failed_count = sum(1 for a in actions if a.execution_status == "failed")
|
||||
|
||||
return templates.TemplateResponse(
|
||||
"partials/dashboard/todays_actions.html",
|
||||
{
|
||||
"request": request,
|
||||
"actions": enriched_actions,
|
||||
"pending_count": pending_count,
|
||||
"completed_count": completed_count,
|
||||
"failed_count": failed_count,
|
||||
"total_count": len(actions),
|
||||
}
|
||||
)
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from sqlalchemy.orm import Session
|
||||
from datetime import datetime, date
|
||||
from typing import Optional
|
||||
import uuid
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import DeploymentRecord, RosterUnit
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["deployments"])
|
||||
|
||||
|
||||
def _serialize(record: DeploymentRecord) -> dict:
|
||||
return {
|
||||
"id": record.id,
|
||||
"unit_id": record.unit_id,
|
||||
"deployed_date": record.deployed_date.isoformat() if record.deployed_date else None,
|
||||
"estimated_removal_date": record.estimated_removal_date.isoformat() if record.estimated_removal_date else None,
|
||||
"actual_removal_date": record.actual_removal_date.isoformat() if record.actual_removal_date else None,
|
||||
"project_ref": record.project_ref,
|
||||
"project_id": record.project_id,
|
||||
"location_name": record.location_name,
|
||||
"notes": record.notes,
|
||||
"created_at": record.created_at.isoformat() if record.created_at else None,
|
||||
"updated_at": record.updated_at.isoformat() if record.updated_at else None,
|
||||
"is_active": record.actual_removal_date is None,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/deployments/{unit_id}")
|
||||
def get_deployments(unit_id: str, db: Session = Depends(get_db)):
|
||||
"""Get all deployment records for a unit, newest first."""
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id).first()
|
||||
if not unit:
|
||||
raise HTTPException(status_code=404, detail=f"Unit {unit_id} not found")
|
||||
|
||||
records = (
|
||||
db.query(DeploymentRecord)
|
||||
.filter_by(unit_id=unit_id)
|
||||
.order_by(DeploymentRecord.deployed_date.desc(), DeploymentRecord.created_at.desc())
|
||||
.all()
|
||||
)
|
||||
return {"deployments": [_serialize(r) for r in records]}
|
||||
|
||||
|
||||
@router.get("/deployments/{unit_id}/active")
|
||||
def get_active_deployment(unit_id: str, db: Session = Depends(get_db)):
|
||||
"""Get the current active deployment (actual_removal_date is NULL), or null."""
|
||||
record = (
|
||||
db.query(DeploymentRecord)
|
||||
.filter(
|
||||
DeploymentRecord.unit_id == unit_id,
|
||||
DeploymentRecord.actual_removal_date == None
|
||||
)
|
||||
.order_by(DeploymentRecord.created_at.desc())
|
||||
.first()
|
||||
)
|
||||
return {"deployment": _serialize(record) if record else None}
|
||||
|
||||
|
||||
@router.post("/deployments/{unit_id}")
|
||||
def create_deployment(unit_id: str, payload: dict, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Create a new deployment record for a unit.
|
||||
|
||||
Body fields (all optional):
|
||||
deployed_date (YYYY-MM-DD)
|
||||
estimated_removal_date (YYYY-MM-DD)
|
||||
project_ref (freeform string)
|
||||
project_id (UUID if linked to Project)
|
||||
location_name
|
||||
notes
|
||||
"""
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id).first()
|
||||
if not unit:
|
||||
raise HTTPException(status_code=404, detail=f"Unit {unit_id} not found")
|
||||
|
||||
def parse_date(val) -> Optional[date]:
|
||||
if not val:
|
||||
return None
|
||||
if isinstance(val, date):
|
||||
return val
|
||||
return date.fromisoformat(str(val))
|
||||
|
||||
record = DeploymentRecord(
|
||||
id=str(uuid.uuid4()),
|
||||
unit_id=unit_id,
|
||||
deployed_date=parse_date(payload.get("deployed_date")),
|
||||
estimated_removal_date=parse_date(payload.get("estimated_removal_date")),
|
||||
actual_removal_date=None,
|
||||
project_ref=payload.get("project_ref"),
|
||||
project_id=payload.get("project_id"),
|
||||
location_name=payload.get("location_name"),
|
||||
notes=payload.get("notes"),
|
||||
created_at=datetime.utcnow(),
|
||||
updated_at=datetime.utcnow(),
|
||||
)
|
||||
db.add(record)
|
||||
db.commit()
|
||||
db.refresh(record)
|
||||
return _serialize(record)
|
||||
|
||||
|
||||
@router.put("/deployments/{unit_id}/{deployment_id}")
|
||||
def update_deployment(unit_id: str, deployment_id: str, payload: dict, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Update a deployment record. Used for:
|
||||
- Setting/changing estimated_removal_date
|
||||
- Closing a deployment (set actual_removal_date to mark unit returned)
|
||||
- Editing project_ref, location_name, notes
|
||||
"""
|
||||
record = db.query(DeploymentRecord).filter_by(id=deployment_id, unit_id=unit_id).first()
|
||||
if not record:
|
||||
raise HTTPException(status_code=404, detail="Deployment record not found")
|
||||
|
||||
def parse_date(val) -> Optional[date]:
|
||||
if val is None:
|
||||
return None
|
||||
if val == "":
|
||||
return None
|
||||
if isinstance(val, date):
|
||||
return val
|
||||
return date.fromisoformat(str(val))
|
||||
|
||||
if "deployed_date" in payload:
|
||||
record.deployed_date = parse_date(payload["deployed_date"])
|
||||
if "estimated_removal_date" in payload:
|
||||
record.estimated_removal_date = parse_date(payload["estimated_removal_date"])
|
||||
if "actual_removal_date" in payload:
|
||||
record.actual_removal_date = parse_date(payload["actual_removal_date"])
|
||||
if "project_ref" in payload:
|
||||
record.project_ref = payload["project_ref"]
|
||||
if "project_id" in payload:
|
||||
record.project_id = payload["project_id"]
|
||||
if "location_name" in payload:
|
||||
record.location_name = payload["location_name"]
|
||||
if "notes" in payload:
|
||||
record.notes = payload["notes"]
|
||||
|
||||
record.updated_at = datetime.utcnow()
|
||||
db.commit()
|
||||
db.refresh(record)
|
||||
return _serialize(record)
|
||||
|
||||
|
||||
@router.delete("/deployments/{unit_id}/{deployment_id}")
|
||||
def delete_deployment(unit_id: str, deployment_id: str, db: Session = Depends(get_db)):
|
||||
"""Delete a deployment record."""
|
||||
record = db.query(DeploymentRecord).filter_by(id=deployment_id, unit_id=unit_id).first()
|
||||
if not record:
|
||||
raise HTTPException(status_code=404, detail="Deployment record not found")
|
||||
db.delete(record)
|
||||
db.commit()
|
||||
return {"ok": True}
|
||||
@@ -0,0 +1,928 @@
|
||||
"""
|
||||
Fleet Calendar Router
|
||||
|
||||
API endpoints for the Fleet Calendar feature:
|
||||
- Calendar page and data
|
||||
- Job reservation CRUD
|
||||
- Unit assignment management
|
||||
- Availability checking
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Request, Depends, HTTPException, Query
|
||||
from fastapi.responses import HTMLResponse, JSONResponse
|
||||
from sqlalchemy.orm import Session
|
||||
from datetime import datetime, date, timedelta
|
||||
from typing import Optional, List
|
||||
import uuid
|
||||
import logging
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import (
|
||||
RosterUnit, JobReservation, JobReservationUnit,
|
||||
UserPreferences, Project, MonitoringLocation, UnitAssignment
|
||||
)
|
||||
from backend.templates_config import templates
|
||||
from backend.services.fleet_calendar_service import (
|
||||
get_day_summary,
|
||||
get_calendar_year_data,
|
||||
get_rolling_calendar_data,
|
||||
check_calibration_conflicts,
|
||||
get_available_units_for_period,
|
||||
get_calibration_status
|
||||
)
|
||||
|
||||
router = APIRouter(tags=["fleet-calendar"])
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Calendar Page
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/fleet-calendar", response_class=HTMLResponse)
|
||||
async def fleet_calendar_page(
|
||||
request: Request,
|
||||
year: Optional[int] = None,
|
||||
month: Optional[int] = None,
|
||||
device_type: str = "seismograph",
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Main Fleet Calendar page with rolling 12-month view."""
|
||||
today = date.today()
|
||||
|
||||
# Default to current month as the start
|
||||
if year is None:
|
||||
year = today.year
|
||||
if month is None:
|
||||
month = today.month
|
||||
|
||||
# Get calendar data for 12 months starting from year/month
|
||||
calendar_data = get_rolling_calendar_data(db, year, month, device_type)
|
||||
|
||||
# Get projects for the reservation form dropdown
|
||||
projects = db.query(Project).filter(
|
||||
Project.status.in_(["active", "upcoming", "on_hold"])
|
||||
).order_by(Project.name).all()
|
||||
|
||||
# Build a serializable list of items with dates for calendar bars
|
||||
# Includes both tracked Projects (with dates) and Job Reservations (matching device_type)
|
||||
project_colors = ['#3B82F6', '#10B981', '#F59E0B', '#EF4444', '#8B5CF6', '#EC4899', '#06B6D4', '#F97316']
|
||||
# Map calendar device_type to project_type_ids
|
||||
device_type_to_project_types = {
|
||||
"seismograph": ["vibration_monitoring", "combined"],
|
||||
"slm": ["sound_monitoring", "combined"],
|
||||
}
|
||||
relevant_project_types = device_type_to_project_types.get(device_type, [])
|
||||
|
||||
calendar_projects = []
|
||||
for i, p in enumerate(projects):
|
||||
if p.start_date and p.project_type_id in relevant_project_types:
|
||||
calendar_projects.append({
|
||||
"id": p.id,
|
||||
"name": p.name,
|
||||
"start_date": p.start_date.isoformat(),
|
||||
"end_date": p.end_date.isoformat() if p.end_date else None,
|
||||
"color": project_colors[i % len(project_colors)],
|
||||
"confirmed": True,
|
||||
})
|
||||
|
||||
# Add job reservations for this device_type as bars
|
||||
from sqlalchemy import or_ as _or
|
||||
cal_window_end = date(year + ((month + 10) // 12), ((month + 10) % 12) + 1, 1)
|
||||
reservations_for_cal = db.query(JobReservation).filter(
|
||||
JobReservation.device_type == device_type,
|
||||
JobReservation.start_date <= cal_window_end,
|
||||
_or(
|
||||
JobReservation.end_date >= date(year, month, 1),
|
||||
JobReservation.end_date == None,
|
||||
)
|
||||
).all()
|
||||
for res in reservations_for_cal:
|
||||
end = res.end_date or res.estimated_end_date
|
||||
calendar_projects.append({
|
||||
"id": res.id,
|
||||
"name": res.name,
|
||||
"start_date": res.start_date.isoformat(),
|
||||
"end_date": end.isoformat() if end else None,
|
||||
"color": res.color,
|
||||
"confirmed": bool(res.project_id),
|
||||
})
|
||||
|
||||
# Calculate prev/next month navigation
|
||||
prev_year, prev_month = (year - 1, 12) if month == 1 else (year, month - 1)
|
||||
next_year, next_month = (year + 1, 1) if month == 12 else (year, month + 1)
|
||||
|
||||
return templates.TemplateResponse(
|
||||
"fleet_calendar.html",
|
||||
{
|
||||
"request": request,
|
||||
"start_year": year,
|
||||
"start_month": month,
|
||||
"prev_year": prev_year,
|
||||
"prev_month": prev_month,
|
||||
"next_year": next_year,
|
||||
"next_month": next_month,
|
||||
"device_type": device_type,
|
||||
"calendar_data": calendar_data,
|
||||
"projects": projects,
|
||||
"calendar_projects": calendar_projects,
|
||||
"today": today.isoformat()
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Calendar Data API
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/api/fleet-calendar/data", response_class=JSONResponse)
|
||||
async def get_calendar_data(
|
||||
year: int,
|
||||
device_type: str = "seismograph",
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Get calendar data for a specific year."""
|
||||
return get_calendar_year_data(db, year, device_type)
|
||||
|
||||
|
||||
@router.get("/api/fleet-calendar/day/{date_str}", response_class=HTMLResponse)
|
||||
async def get_day_detail(
|
||||
request: Request,
|
||||
date_str: str,
|
||||
device_type: str = "seismograph",
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Get detailed view for a specific day (HTMX partial)."""
|
||||
try:
|
||||
check_date = date.fromisoformat(date_str)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid date format. Use YYYY-MM-DD")
|
||||
|
||||
day_data = get_day_summary(db, check_date, device_type)
|
||||
|
||||
# Get projects for display names
|
||||
projects = {p.id: p for p in db.query(Project).all()}
|
||||
|
||||
return templates.TemplateResponse(
|
||||
"partials/fleet_calendar/day_detail.html",
|
||||
{
|
||||
"request": request,
|
||||
"day_data": day_data,
|
||||
"date_str": date_str,
|
||||
"date_display": check_date.strftime("%B %d, %Y"),
|
||||
"device_type": device_type,
|
||||
"projects": projects
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Reservation CRUD
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/api/fleet-calendar/reservations", response_class=JSONResponse)
|
||||
async def create_reservation(
|
||||
request: Request,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Create a new job reservation."""
|
||||
data = await request.json()
|
||||
|
||||
# Validate required fields
|
||||
required = ["name", "start_date", "assignment_type"]
|
||||
for field in required:
|
||||
if field not in data:
|
||||
raise HTTPException(status_code=400, detail=f"Missing required field: {field}")
|
||||
|
||||
# Need either end_date or end_date_tbd
|
||||
end_date_tbd = data.get("end_date_tbd", False)
|
||||
if not end_date_tbd and not data.get("end_date"):
|
||||
raise HTTPException(status_code=400, detail="End date is required unless marked as TBD")
|
||||
|
||||
try:
|
||||
start_date = date.fromisoformat(data["start_date"])
|
||||
end_date = date.fromisoformat(data["end_date"]) if data.get("end_date") else None
|
||||
estimated_end_date = date.fromisoformat(data["estimated_end_date"]) if data.get("estimated_end_date") else None
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid date format. Use YYYY-MM-DD")
|
||||
|
||||
if end_date and end_date < start_date:
|
||||
raise HTTPException(status_code=400, detail="End date must be after start date")
|
||||
|
||||
if estimated_end_date and estimated_end_date < start_date:
|
||||
raise HTTPException(status_code=400, detail="Estimated end date must be after start date")
|
||||
|
||||
import json as _json
|
||||
reservation = JobReservation(
|
||||
id=str(uuid.uuid4()),
|
||||
name=data["name"],
|
||||
project_id=data.get("project_id"),
|
||||
start_date=start_date,
|
||||
end_date=end_date,
|
||||
estimated_end_date=estimated_end_date,
|
||||
end_date_tbd=end_date_tbd,
|
||||
assignment_type=data["assignment_type"],
|
||||
device_type=data.get("device_type", "seismograph"),
|
||||
quantity_needed=data.get("quantity_needed"),
|
||||
estimated_units=data.get("estimated_units"),
|
||||
location_slots=_json.dumps(data["location_slots"]) if data.get("location_slots") is not None else None,
|
||||
notes=data.get("notes"),
|
||||
color=data.get("color", "#3B82F6")
|
||||
)
|
||||
|
||||
db.add(reservation)
|
||||
|
||||
# If specific units were provided, assign them
|
||||
if data.get("unit_ids") and data["assignment_type"] == "specific":
|
||||
for unit_id in data["unit_ids"]:
|
||||
assignment = JobReservationUnit(
|
||||
id=str(uuid.uuid4()),
|
||||
reservation_id=reservation.id,
|
||||
unit_id=unit_id,
|
||||
assignment_source="specific"
|
||||
)
|
||||
db.add(assignment)
|
||||
|
||||
db.commit()
|
||||
|
||||
logger.info(f"Created reservation: {reservation.name} ({reservation.id})")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"reservation_id": reservation.id,
|
||||
"message": f"Created reservation: {reservation.name}"
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/fleet-calendar/reservations/{reservation_id}", response_class=JSONResponse)
|
||||
async def get_reservation(
|
||||
reservation_id: str,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Get a specific reservation with its assigned units."""
|
||||
reservation = db.query(JobReservation).filter_by(id=reservation_id).first()
|
||||
if not reservation:
|
||||
raise HTTPException(status_code=404, detail="Reservation not found")
|
||||
|
||||
# Get assigned units
|
||||
assignments = db.query(JobReservationUnit).filter_by(
|
||||
reservation_id=reservation_id
|
||||
).all()
|
||||
|
||||
# Sort assignments by slot_index so order is preserved
|
||||
assignments_sorted = sorted(assignments, key=lambda a: (a.slot_index if a.slot_index is not None else 999))
|
||||
unit_ids = [a.unit_id for a in assignments_sorted]
|
||||
units = db.query(RosterUnit).filter(RosterUnit.id.in_(unit_ids)).all() if unit_ids else []
|
||||
units_by_id = {u.id: u for u in units}
|
||||
# Build per-unit lookups from assignments
|
||||
assignment_map = {a.unit_id: a for a in assignments_sorted}
|
||||
|
||||
import json as _json
|
||||
stored_slots = _json.loads(reservation.location_slots) if reservation.location_slots else None
|
||||
|
||||
return {
|
||||
"id": reservation.id,
|
||||
"name": reservation.name,
|
||||
"project_id": reservation.project_id,
|
||||
"start_date": reservation.start_date.isoformat(),
|
||||
"end_date": reservation.end_date.isoformat() if reservation.end_date else None,
|
||||
"estimated_end_date": reservation.estimated_end_date.isoformat() if reservation.estimated_end_date else None,
|
||||
"end_date_tbd": reservation.end_date_tbd,
|
||||
"assignment_type": reservation.assignment_type,
|
||||
"device_type": reservation.device_type,
|
||||
"quantity_needed": reservation.quantity_needed,
|
||||
"estimated_units": reservation.estimated_units,
|
||||
"location_slots": stored_slots,
|
||||
"notes": reservation.notes,
|
||||
"color": reservation.color,
|
||||
"assigned_units": [
|
||||
{
|
||||
"id": uid,
|
||||
"last_calibrated": units_by_id[uid].last_calibrated.isoformat() if uid in units_by_id and units_by_id[uid].last_calibrated else None,
|
||||
"deployed": units_by_id[uid].deployed if uid in units_by_id else False,
|
||||
"power_type": assignment_map[uid].power_type,
|
||||
"notes": assignment_map[uid].notes,
|
||||
"location_name": assignment_map[uid].location_name,
|
||||
"slot_index": assignment_map[uid].slot_index,
|
||||
}
|
||||
for uid in unit_ids
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@router.put("/api/fleet-calendar/reservations/{reservation_id}", response_class=JSONResponse)
|
||||
async def update_reservation(
|
||||
reservation_id: str,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Update an existing reservation."""
|
||||
reservation = db.query(JobReservation).filter_by(id=reservation_id).first()
|
||||
if not reservation:
|
||||
raise HTTPException(status_code=404, detail="Reservation not found")
|
||||
|
||||
data = await request.json()
|
||||
|
||||
# Update fields if provided
|
||||
if "name" in data:
|
||||
reservation.name = data["name"]
|
||||
if "project_id" in data:
|
||||
reservation.project_id = data["project_id"]
|
||||
if "start_date" in data:
|
||||
reservation.start_date = date.fromisoformat(data["start_date"])
|
||||
if "end_date" in data:
|
||||
reservation.end_date = date.fromisoformat(data["end_date"]) if data["end_date"] else None
|
||||
if "estimated_end_date" in data:
|
||||
reservation.estimated_end_date = date.fromisoformat(data["estimated_end_date"]) if data["estimated_end_date"] else None
|
||||
if "end_date_tbd" in data:
|
||||
reservation.end_date_tbd = data["end_date_tbd"]
|
||||
if "assignment_type" in data:
|
||||
reservation.assignment_type = data["assignment_type"]
|
||||
if "quantity_needed" in data:
|
||||
reservation.quantity_needed = data["quantity_needed"]
|
||||
if "estimated_units" in data:
|
||||
reservation.estimated_units = data["estimated_units"]
|
||||
if "location_slots" in data:
|
||||
import json as _json
|
||||
reservation.location_slots = _json.dumps(data["location_slots"]) if data["location_slots"] is not None else None
|
||||
if "notes" in data:
|
||||
reservation.notes = data["notes"]
|
||||
if "color" in data:
|
||||
reservation.color = data["color"]
|
||||
|
||||
reservation.updated_at = datetime.utcnow()
|
||||
|
||||
db.commit()
|
||||
|
||||
logger.info(f"Updated reservation: {reservation.name} ({reservation.id})")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"message": f"Updated reservation: {reservation.name}"
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/api/fleet-calendar/reservations/{reservation_id}", response_class=JSONResponse)
|
||||
async def delete_reservation(
|
||||
reservation_id: str,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Delete a reservation and its unit assignments."""
|
||||
reservation = db.query(JobReservation).filter_by(id=reservation_id).first()
|
||||
if not reservation:
|
||||
raise HTTPException(status_code=404, detail="Reservation not found")
|
||||
|
||||
# Delete unit assignments first
|
||||
db.query(JobReservationUnit).filter_by(reservation_id=reservation_id).delete()
|
||||
|
||||
# Delete the reservation
|
||||
db.delete(reservation)
|
||||
db.commit()
|
||||
|
||||
logger.info(f"Deleted reservation: {reservation.name} ({reservation_id})")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"message": "Reservation deleted"
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Unit Assignment
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/api/fleet-calendar/reservations/{reservation_id}/assign-units", response_class=JSONResponse)
|
||||
async def assign_units_to_reservation(
|
||||
reservation_id: str,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Assign specific units to a reservation."""
|
||||
reservation = db.query(JobReservation).filter_by(id=reservation_id).first()
|
||||
if not reservation:
|
||||
raise HTTPException(status_code=404, detail="Reservation not found")
|
||||
|
||||
data = await request.json()
|
||||
unit_ids = data.get("unit_ids", [])
|
||||
# Optional per-unit dicts keyed by unit_id
|
||||
power_types = data.get("power_types", {})
|
||||
location_notes = data.get("location_notes", {})
|
||||
location_names = data.get("location_names", {})
|
||||
# slot_indices: {"BE17354": 0, "BE9441": 1, ...}
|
||||
slot_indices = data.get("slot_indices", {})
|
||||
|
||||
# Verify units exist (allow empty list to clear all assignments)
|
||||
if unit_ids:
|
||||
units = db.query(RosterUnit).filter(RosterUnit.id.in_(unit_ids)).all()
|
||||
found_ids = {u.id for u in units}
|
||||
missing = set(unit_ids) - found_ids
|
||||
if missing:
|
||||
raise HTTPException(status_code=404, detail=f"Units not found: {', '.join(missing)}")
|
||||
|
||||
# Full replace: delete all existing assignments for this reservation first
|
||||
db.query(JobReservationUnit).filter_by(reservation_id=reservation_id).delete()
|
||||
db.flush()
|
||||
|
||||
# Check for conflicts with other reservations and insert new assignments
|
||||
conflicts = []
|
||||
for unit_id in unit_ids:
|
||||
# Check overlapping reservations
|
||||
if reservation.end_date:
|
||||
overlapping = db.query(JobReservation).join(
|
||||
JobReservationUnit, JobReservation.id == JobReservationUnit.reservation_id
|
||||
).filter(
|
||||
JobReservationUnit.unit_id == unit_id,
|
||||
JobReservation.id != reservation_id,
|
||||
JobReservation.start_date <= reservation.end_date,
|
||||
JobReservation.end_date >= reservation.start_date
|
||||
).first()
|
||||
|
||||
if overlapping:
|
||||
conflicts.append({
|
||||
"unit_id": unit_id,
|
||||
"conflict_reservation": overlapping.name,
|
||||
"conflict_dates": f"{overlapping.start_date} - {overlapping.end_date}"
|
||||
})
|
||||
continue
|
||||
|
||||
# Add assignment
|
||||
assignment = JobReservationUnit(
|
||||
id=str(uuid.uuid4()),
|
||||
reservation_id=reservation_id,
|
||||
unit_id=unit_id,
|
||||
assignment_source="filled" if reservation.assignment_type == "quantity" else "specific",
|
||||
power_type=power_types.get(unit_id),
|
||||
notes=location_notes.get(unit_id),
|
||||
location_name=location_names.get(unit_id),
|
||||
slot_index=slot_indices.get(unit_id),
|
||||
)
|
||||
db.add(assignment)
|
||||
|
||||
db.commit()
|
||||
|
||||
# Check for calibration conflicts
|
||||
cal_conflicts = check_calibration_conflicts(db, reservation_id)
|
||||
|
||||
assigned_count = db.query(JobReservationUnit).filter_by(
|
||||
reservation_id=reservation_id
|
||||
).count()
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"assigned_count": assigned_count,
|
||||
"conflicts": conflicts,
|
||||
"calibration_warnings": cal_conflicts,
|
||||
"message": f"Assigned {len(unit_ids) - len(conflicts)} units"
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/api/fleet-calendar/reservations/{reservation_id}/units/{unit_id}", response_class=JSONResponse)
|
||||
async def remove_unit_from_reservation(
|
||||
reservation_id: str,
|
||||
unit_id: str,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Remove a unit from a reservation."""
|
||||
assignment = db.query(JobReservationUnit).filter_by(
|
||||
reservation_id=reservation_id,
|
||||
unit_id=unit_id
|
||||
).first()
|
||||
|
||||
if not assignment:
|
||||
raise HTTPException(status_code=404, detail="Unit assignment not found")
|
||||
|
||||
db.delete(assignment)
|
||||
db.commit()
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"message": f"Removed {unit_id} from reservation"
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Availability & Conflicts
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/api/fleet-calendar/availability", response_class=JSONResponse)
|
||||
async def check_availability(
|
||||
start_date: str,
|
||||
end_date: str,
|
||||
device_type: str = "seismograph",
|
||||
exclude_reservation_id: Optional[str] = None,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Get units available for a specific date range."""
|
||||
try:
|
||||
start = date.fromisoformat(start_date)
|
||||
end = date.fromisoformat(end_date)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid date format. Use YYYY-MM-DD")
|
||||
|
||||
available = get_available_units_for_period(
|
||||
db, start, end, device_type, exclude_reservation_id
|
||||
)
|
||||
|
||||
return {
|
||||
"start_date": start_date,
|
||||
"end_date": end_date,
|
||||
"device_type": device_type,
|
||||
"available_units": available,
|
||||
"count": len(available)
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/fleet-calendar/reservations/{reservation_id}/conflicts", response_class=JSONResponse)
|
||||
async def get_reservation_conflicts(
|
||||
reservation_id: str,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Check for calibration conflicts in a reservation."""
|
||||
reservation = db.query(JobReservation).filter_by(id=reservation_id).first()
|
||||
if not reservation:
|
||||
raise HTTPException(status_code=404, detail="Reservation not found")
|
||||
|
||||
conflicts = check_calibration_conflicts(db, reservation_id)
|
||||
|
||||
return {
|
||||
"reservation_id": reservation_id,
|
||||
"reservation_name": reservation.name,
|
||||
"conflicts": conflicts,
|
||||
"has_conflicts": len(conflicts) > 0
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# HTMX Partials
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/api/fleet-calendar/reservations-list", response_class=HTMLResponse)
|
||||
async def get_reservations_list(
|
||||
request: Request,
|
||||
year: Optional[int] = None,
|
||||
month: Optional[int] = None,
|
||||
device_type: str = "seismograph",
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Get list of reservations as HTMX partial."""
|
||||
from sqlalchemy import or_
|
||||
|
||||
today = date.today()
|
||||
if year is None:
|
||||
year = today.year
|
||||
if month is None:
|
||||
month = today.month
|
||||
|
||||
# Calculate 12-month window
|
||||
start_date = date(year, month, 1)
|
||||
# End date is 12 months later
|
||||
end_year = year + ((month + 10) // 12)
|
||||
end_month = ((month + 10) % 12) + 1
|
||||
if end_month == 12:
|
||||
end_date = date(end_year, 12, 31)
|
||||
else:
|
||||
end_date = date(end_year, end_month + 1, 1) - timedelta(days=1)
|
||||
|
||||
# Filter by device_type and date window
|
||||
reservations = db.query(JobReservation).filter(
|
||||
JobReservation.device_type == device_type,
|
||||
JobReservation.start_date <= end_date,
|
||||
or_(
|
||||
JobReservation.end_date >= start_date,
|
||||
JobReservation.end_date == None # TBD reservations
|
||||
)
|
||||
).order_by(JobReservation.start_date).all()
|
||||
|
||||
# Get assignment counts
|
||||
reservation_data = []
|
||||
for res in reservations:
|
||||
assignments = db.query(JobReservationUnit).filter_by(
|
||||
reservation_id=res.id
|
||||
).all()
|
||||
assigned_count = len(assignments)
|
||||
|
||||
# Enrich assignments with unit details, sorted by slot_index
|
||||
assignments_sorted = sorted(assignments, key=lambda a: (a.slot_index if a.slot_index is not None else 999))
|
||||
unit_ids = [a.unit_id for a in assignments_sorted]
|
||||
units = db.query(RosterUnit).filter(RosterUnit.id.in_(unit_ids)).all() if unit_ids else []
|
||||
units_by_id = {u.id: u for u in units}
|
||||
assigned_units = [
|
||||
{
|
||||
"id": a.unit_id,
|
||||
"power_type": a.power_type,
|
||||
"notes": a.notes,
|
||||
"location_name": a.location_name,
|
||||
"slot_index": a.slot_index,
|
||||
"deployed": units_by_id[a.unit_id].deployed if a.unit_id in units_by_id else False,
|
||||
"last_calibrated": units_by_id[a.unit_id].last_calibrated if a.unit_id in units_by_id else None,
|
||||
}
|
||||
for a in assignments_sorted
|
||||
]
|
||||
|
||||
# Check for calibration conflicts
|
||||
conflicts = check_calibration_conflicts(db, res.id)
|
||||
|
||||
location_count = res.quantity_needed or assigned_count
|
||||
reservation_data.append({
|
||||
"reservation": res,
|
||||
"assigned_count": assigned_count,
|
||||
"location_count": location_count,
|
||||
"assigned_units": assigned_units,
|
||||
"has_conflicts": len(conflicts) > 0,
|
||||
"conflict_count": len(conflicts)
|
||||
})
|
||||
|
||||
return templates.TemplateResponse(
|
||||
"partials/fleet_calendar/reservations_list.html",
|
||||
{
|
||||
"request": request,
|
||||
"reservations": reservation_data,
|
||||
"year": year,
|
||||
"device_type": device_type
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.get("/api/fleet-calendar/planner-availability", response_class=JSONResponse)
|
||||
async def get_planner_availability(
|
||||
device_type: str = "seismograph",
|
||||
start_date: Optional[str] = None,
|
||||
end_date: Optional[str] = None,
|
||||
exclude_reservation_id: Optional[str] = None,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Get available units for the reservation planner split-panel UI.
|
||||
Dates are optional — if omitted, returns all non-retired units regardless of reservations.
|
||||
"""
|
||||
if start_date and end_date:
|
||||
try:
|
||||
start = date.fromisoformat(start_date)
|
||||
end = date.fromisoformat(end_date)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid date format. Use YYYY-MM-DD")
|
||||
units = get_available_units_for_period(db, start, end, device_type, exclude_reservation_id)
|
||||
else:
|
||||
# No dates: return all non-retired units of this type, with current reservation info
|
||||
from backend.models import RosterUnit as RU
|
||||
from datetime import timedelta
|
||||
today = date.today()
|
||||
all_units = db.query(RU).filter(
|
||||
RU.device_type == device_type,
|
||||
RU.retired == False
|
||||
).all()
|
||||
|
||||
# Build a map: unit_id -> list of active/upcoming reservations
|
||||
active_assignments = db.query(JobReservationUnit).join(
|
||||
JobReservation, JobReservationUnit.reservation_id == JobReservation.id
|
||||
).filter(
|
||||
JobReservation.device_type == device_type,
|
||||
JobReservation.end_date >= today
|
||||
).all()
|
||||
unit_reservations = {}
|
||||
for assignment in active_assignments:
|
||||
res = db.query(JobReservation).filter(JobReservation.id == assignment.reservation_id).first()
|
||||
if not res:
|
||||
continue
|
||||
unit_reservations.setdefault(assignment.unit_id, []).append({
|
||||
"reservation_id": res.id,
|
||||
"reservation_name": res.name,
|
||||
"start_date": res.start_date.isoformat() if res.start_date else None,
|
||||
"end_date": res.end_date.isoformat() if res.end_date else None,
|
||||
"color": res.color or "#3B82F6"
|
||||
})
|
||||
|
||||
units = []
|
||||
for u in all_units:
|
||||
expiry = (u.last_calibrated + timedelta(days=365)) if u.last_calibrated else None
|
||||
units.append({
|
||||
"id": u.id,
|
||||
"last_calibrated": u.last_calibrated.isoformat() if u.last_calibrated else None,
|
||||
"expiry_date": expiry.isoformat() if expiry else None,
|
||||
"calibration_status": "needs_calibration" if not u.last_calibrated else "valid",
|
||||
"deployed": u.deployed,
|
||||
"out_for_calibration": u.out_for_calibration or False,
|
||||
"allocated": getattr(u, 'allocated', False) or False,
|
||||
"allocated_to_project_id": getattr(u, 'allocated_to_project_id', None) or "",
|
||||
"note": u.note or "",
|
||||
"reservations": unit_reservations.get(u.id, [])
|
||||
})
|
||||
|
||||
# Sort: benched first (easier to assign), then deployed, then by ID
|
||||
units.sort(key=lambda u: (1 if u["deployed"] else 0, u["id"]))
|
||||
|
||||
return {
|
||||
"units": units,
|
||||
"start_date": start_date,
|
||||
"end_date": end_date,
|
||||
"count": len(units)
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/fleet-calendar/unit-quick-info/{unit_id}", response_class=JSONResponse)
|
||||
async def get_unit_quick_info(unit_id: str, db: Session = Depends(get_db)):
|
||||
"""Return at-a-glance info for the planner quick-view modal."""
|
||||
from backend.models import Emitter
|
||||
u = db.query(RosterUnit).filter(RosterUnit.id == unit_id).first()
|
||||
if not u:
|
||||
raise HTTPException(status_code=404, detail="Unit not found")
|
||||
|
||||
today = date.today()
|
||||
expiry = (u.last_calibrated + timedelta(days=365)) if u.last_calibrated else None
|
||||
|
||||
# Active/upcoming reservations
|
||||
assignments = db.query(JobReservationUnit).filter(JobReservationUnit.unit_id == unit_id).all()
|
||||
reservations = []
|
||||
for a in assignments:
|
||||
res = db.query(JobReservation).filter(
|
||||
JobReservation.id == a.reservation_id,
|
||||
JobReservation.end_date >= today
|
||||
).first()
|
||||
if res:
|
||||
reservations.append({
|
||||
"name": res.name,
|
||||
"start_date": res.start_date.isoformat() if res.start_date else None,
|
||||
"end_date": res.end_date.isoformat() if res.end_date else None,
|
||||
"end_date_tbd": res.end_date_tbd,
|
||||
"color": res.color or "#3B82F6",
|
||||
"location_name": a.location_name,
|
||||
})
|
||||
|
||||
# Last seen from emitter
|
||||
emitter = db.query(Emitter).filter(Emitter.unit_type == unit_id).first()
|
||||
|
||||
return {
|
||||
"id": u.id,
|
||||
"unit_type": u.unit_type,
|
||||
"deployed": u.deployed,
|
||||
"out_for_calibration": u.out_for_calibration or False,
|
||||
"note": u.note or "",
|
||||
"project_id": u.project_id or "",
|
||||
"address": u.address or u.location or "",
|
||||
"coordinates": u.coordinates or "",
|
||||
"deployed_with_modem_id": u.deployed_with_modem_id or "",
|
||||
"last_calibrated": u.last_calibrated.isoformat() if u.last_calibrated else None,
|
||||
"next_calibration_due": u.next_calibration_due.isoformat() if u.next_calibration_due else (expiry.isoformat() if expiry else None),
|
||||
"cal_expired": not u.last_calibrated or (expiry and expiry < today),
|
||||
"last_seen": emitter.last_seen.isoformat() if emitter and emitter.last_seen else None,
|
||||
"reservations": reservations,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/api/fleet-calendar/available-units", response_class=HTMLResponse)
|
||||
async def get_available_units_partial(
|
||||
request: Request,
|
||||
start_date: str,
|
||||
end_date: str,
|
||||
device_type: str = "seismograph",
|
||||
reservation_id: Optional[str] = None,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Get available units as HTMX partial for the assignment modal."""
|
||||
try:
|
||||
start = date.fromisoformat(start_date)
|
||||
end = date.fromisoformat(end_date)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid date format")
|
||||
|
||||
available = get_available_units_for_period(
|
||||
db, start, end, device_type, reservation_id
|
||||
)
|
||||
|
||||
return templates.TemplateResponse(
|
||||
"partials/fleet_calendar/available_units.html",
|
||||
{
|
||||
"request": request,
|
||||
"units": available,
|
||||
"start_date": start_date,
|
||||
"end_date": end_date,
|
||||
"device_type": device_type,
|
||||
"reservation_id": reservation_id
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
@router.get("/api/fleet-calendar/month/{year}/{month}", response_class=HTMLResponse)
|
||||
async def get_month_partial(
|
||||
request: Request,
|
||||
year: int,
|
||||
month: int,
|
||||
device_type: str = "seismograph",
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""Get a single month calendar as HTMX partial."""
|
||||
calendar_data = get_calendar_year_data(db, year, device_type)
|
||||
month_data = calendar_data["months"].get(month)
|
||||
|
||||
if not month_data:
|
||||
raise HTTPException(status_code=404, detail="Invalid month")
|
||||
|
||||
return templates.TemplateResponse(
|
||||
"partials/fleet_calendar/month_grid.html",
|
||||
{
|
||||
"request": request,
|
||||
"year": year,
|
||||
"month": month,
|
||||
"month_data": month_data,
|
||||
"device_type": device_type,
|
||||
"today": date.today().isoformat()
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Promote Reservation to Project
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/api/fleet-calendar/reservations/{reservation_id}/promote-to-project", response_class=JSONResponse)
|
||||
async def promote_reservation_to_project(
|
||||
reservation_id: str,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db)
|
||||
):
|
||||
"""
|
||||
Promote a job reservation to a full project in the projects DB.
|
||||
Creates: Project + MonitoringLocations + UnitAssignments.
|
||||
"""
|
||||
reservation = db.query(JobReservation).filter_by(id=reservation_id).first()
|
||||
if not reservation:
|
||||
raise HTTPException(status_code=404, detail="Reservation not found")
|
||||
|
||||
data = await request.json()
|
||||
project_number = data.get("project_number") or None
|
||||
client_name = data.get("client_name") or None
|
||||
|
||||
# Map device_type to project_type_id
|
||||
if reservation.device_type == "slm":
|
||||
project_type_id = "sound_monitoring"
|
||||
location_type = "sound"
|
||||
else:
|
||||
project_type_id = "vibration_monitoring"
|
||||
location_type = "vibration"
|
||||
|
||||
# Check for duplicate project name
|
||||
existing = db.query(Project).filter_by(name=reservation.name).first()
|
||||
if existing:
|
||||
raise HTTPException(status_code=409, detail=f"A project named '{reservation.name}' already exists.")
|
||||
|
||||
# Create the project
|
||||
project_id = str(uuid.uuid4())
|
||||
project = Project(
|
||||
id=project_id,
|
||||
name=reservation.name,
|
||||
project_number=project_number,
|
||||
client_name=client_name,
|
||||
project_type_id=project_type_id,
|
||||
status="upcoming",
|
||||
start_date=reservation.start_date,
|
||||
end_date=reservation.end_date,
|
||||
description=reservation.notes,
|
||||
)
|
||||
db.add(project)
|
||||
db.flush()
|
||||
|
||||
# Load assignments sorted by slot_index
|
||||
assignments = db.query(JobReservationUnit).filter_by(reservation_id=reservation_id).all()
|
||||
assignments_sorted = sorted(assignments, key=lambda a: (a.slot_index if a.slot_index is not None else 999))
|
||||
|
||||
locations_created = 0
|
||||
units_assigned = 0
|
||||
|
||||
for i, assignment in enumerate(assignments_sorted):
|
||||
loc_num = str(i + 1).zfill(3)
|
||||
loc_name = assignment.location_name or f"Location {i + 1}"
|
||||
|
||||
location = MonitoringLocation(
|
||||
id=str(uuid.uuid4()),
|
||||
project_id=project_id,
|
||||
location_type=location_type,
|
||||
name=loc_name,
|
||||
description=assignment.notes,
|
||||
)
|
||||
db.add(location)
|
||||
db.flush()
|
||||
locations_created += 1
|
||||
|
||||
if assignment.unit_id:
|
||||
unit_assignment = UnitAssignment(
|
||||
id=str(uuid.uuid4()),
|
||||
unit_id=assignment.unit_id,
|
||||
location_id=location.id,
|
||||
project_id=project_id,
|
||||
device_type=reservation.device_type or "seismograph",
|
||||
status="active",
|
||||
notes=f"Power: {assignment.power_type}" if assignment.power_type else None,
|
||||
)
|
||||
db.add(unit_assignment)
|
||||
units_assigned += 1
|
||||
|
||||
db.commit()
|
||||
|
||||
logger.info(f"Promoted reservation '{reservation.name}' to project {project_id}")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"project_id": project_id,
|
||||
"project_name": reservation.name,
|
||||
"locations_created": locations_created,
|
||||
"units_assigned": units_assigned,
|
||||
}
|
||||
@@ -0,0 +1,394 @@
|
||||
"""
|
||||
Metadata-backfill admin router.
|
||||
|
||||
Endpoints under /api/admin/metadata_backfill:
|
||||
|
||||
GET /scan — run the scan; return clusters + suggestions (JSON).
|
||||
Cached 5 minutes so the wizard doesn't re-scan on
|
||||
every page render.
|
||||
POST /apply — apply a list of cluster_ids; body specifies which to
|
||||
accept and optional per-cluster overrides.
|
||||
POST /skip — mark cluster_ids as skipped (won't reappear).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from fastapi.responses import JSONResponse
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import Project, MonitoringLocation
|
||||
from backend.services import metadata_backfill as svc
|
||||
|
||||
router = APIRouter(prefix="/api/admin/metadata_backfill", tags=["metadata-backfill"])
|
||||
|
||||
SFM_BASE_URL = os.getenv("SFM_BASE_URL", "http://localhost:8200")
|
||||
|
||||
# In-process scan cache. Trades memory for not re-hammering SFM on every
|
||||
# wizard render. TTL: 5 minutes. Singleton per-process; fine for a
|
||||
# single-worker uvicorn dev setup. For prod multi-worker we'd want to put
|
||||
# this in the DB or Redis; deferred.
|
||||
_SCAN_CACHE: dict = {"at": 0.0, "result": None}
|
||||
_SCAN_CACHE_TTL_SECONDS = 300.0
|
||||
|
||||
|
||||
def _serialise_suggestion(s: svc.Suggestion) -> dict:
|
||||
c = s.cluster
|
||||
return {
|
||||
"cluster_id": c.cluster_id,
|
||||
"serial": c.serial,
|
||||
"first_event_ts": c.first_event_ts.isoformat(),
|
||||
"last_event_ts": c.last_event_ts.isoformat(),
|
||||
"event_count": c.event_count,
|
||||
"sample_event_id": c.sample_event_id,
|
||||
"project_raw": c.project_raw,
|
||||
"project_root": c.project_root,
|
||||
"location_raw": c.location_raw,
|
||||
"client_raw": c.client_raw,
|
||||
"operator_raw": c.operator_raw,
|
||||
"is_blank_meta": c.is_blank_meta,
|
||||
"metadata_consistency": c.metadata_consistency,
|
||||
|
||||
"project_match": s.project_match,
|
||||
"project_existing_id": s.project_existing_id,
|
||||
"project_existing_name": s.project_existing_name,
|
||||
"project_match_score": s.project_match_score,
|
||||
"project_suggested_name": s.project_suggested_name,
|
||||
|
||||
"location_match": s.location_match,
|
||||
"location_existing_id": s.location_existing_id,
|
||||
"location_existing_name": s.location_existing_name,
|
||||
"location_match_score": s.location_match_score,
|
||||
"location_suggested_name": s.location_suggested_name,
|
||||
|
||||
"proposed_assigned_at": s.proposed_assigned_at.isoformat(),
|
||||
"proposed_assigned_until": s.proposed_assigned_until.isoformat() if s.proposed_assigned_until else None,
|
||||
|
||||
"confidence": s.confidence,
|
||||
"blocking_conflict": s.blocking_conflict,
|
||||
"conflicts": [
|
||||
{
|
||||
"existing_assignment_id": cf.existing_assignment_id,
|
||||
"other_location_id": cf.other_location_id,
|
||||
"other_location_name": cf.other_location_name,
|
||||
"other_project_id": cf.other_project_id,
|
||||
"other_project_name": cf.other_project_name,
|
||||
}
|
||||
for cf in s.conflicts
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@router.get("/scan")
|
||||
async def scan(
|
||||
force: bool = False,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Run a scan and return clusters + suggestions.
|
||||
|
||||
Set force=true to bypass the 5-minute cache.
|
||||
"""
|
||||
now = time.time()
|
||||
if not force and _SCAN_CACHE["result"] is not None \
|
||||
and (now - _SCAN_CACHE["at"]) < _SCAN_CACHE_TTL_SECONDS:
|
||||
return _SCAN_CACHE["result"]
|
||||
|
||||
result = await svc.scan_clusters_and_build_suggestions(db, SFM_BASE_URL)
|
||||
|
||||
# Group suggestions for the wizard UI.
|
||||
by_confidence = {"high": [], "medium": [], "low": []}
|
||||
blocking_conflict_count = 0
|
||||
for s in result.suggestions:
|
||||
by_confidence[s.confidence].append(_serialise_suggestion(s))
|
||||
if s.blocking_conflict:
|
||||
blocking_conflict_count += 1
|
||||
|
||||
payload = {
|
||||
"scanned_event_count": result.scanned_event_count,
|
||||
"cluster_count": result.cluster_count,
|
||||
"already_attributed": result.already_attributed,
|
||||
"skipped_orphans": result.skipped_orphans,
|
||||
"pending_count": len(result.suggestions),
|
||||
"blocking_conflict_count": blocking_conflict_count,
|
||||
"by_confidence": {
|
||||
"high": by_confidence["high"],
|
||||
"medium": by_confidence["medium"],
|
||||
"low": by_confidence["low"],
|
||||
},
|
||||
"scanned_at": now,
|
||||
}
|
||||
_SCAN_CACHE["result"] = payload
|
||||
_SCAN_CACHE["at"] = now
|
||||
return payload
|
||||
|
||||
|
||||
@router.post("/apply")
|
||||
async def apply(
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Apply a list of clusters.
|
||||
|
||||
Body:
|
||||
{
|
||||
"cluster_ids": ["abc...", "def..."],
|
||||
"overrides": { "abc...": { "project_name": "...", "location_name": "..." } }
|
||||
}
|
||||
|
||||
To accept ALL non-conflict suggestions in one shot, the UI sends every
|
||||
pending cluster_id with no overrides.
|
||||
"""
|
||||
try:
|
||||
body = await request.json()
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail="Invalid JSON body")
|
||||
|
||||
cluster_ids = body.get("cluster_ids") or []
|
||||
overrides = body.get("overrides") or {}
|
||||
if not isinstance(cluster_ids, list) or not cluster_ids:
|
||||
raise HTTPException(status_code=400, detail="cluster_ids must be a non-empty list")
|
||||
|
||||
# Re-scan to get current suggestions. We don't trust the cached scan
|
||||
# blindly — the operator might have manually created projects in
|
||||
# between scan and apply.
|
||||
scan_result = await svc.scan_clusters_and_build_suggestions(db, SFM_BASE_URL)
|
||||
suggestions_by_id = {s.cluster.cluster_id: s for s in scan_result.suggestions}
|
||||
|
||||
selected: list[svc.Suggestion] = []
|
||||
not_found: list[str] = []
|
||||
for cid in cluster_ids:
|
||||
s = suggestions_by_id.get(cid)
|
||||
if s is None:
|
||||
not_found.append(cid)
|
||||
continue
|
||||
# Apply overrides. Per-cluster overrides take precedence over the
|
||||
# parser's suggested match. Four override fields supported:
|
||||
# project_id — attach to an existing Project (operator picked
|
||||
# from the typeahead)
|
||||
# project_name — create new project with this name (operator
|
||||
# typed a custom name not matching anything)
|
||||
# location_id — attach to an existing MonitoringLocation
|
||||
# location_name — create new location with this name
|
||||
# project_id + location_id pairings: location_id is only honored
|
||||
# if its project_id matches the chosen project (otherwise treated
|
||||
# as a create-new).
|
||||
ov = overrides.get(cid) or {}
|
||||
|
||||
if ov.get("project_id"):
|
||||
target_id = ov["project_id"]
|
||||
existing = db.query(svc.Project).filter_by(id=target_id).first()
|
||||
if existing is not None:
|
||||
s.project_existing_id = existing.id
|
||||
s.project_existing_name = existing.name
|
||||
s.project_suggested_name = existing.name
|
||||
s.project_match = "exact"
|
||||
else:
|
||||
# Stale ID — treat as create_new with the cluster's typed name.
|
||||
s.project_existing_id = None
|
||||
s.project_match = "create_new"
|
||||
elif "project_name" in ov:
|
||||
new_name = (ov["project_name"] or "").strip()
|
||||
if new_name:
|
||||
s.project_suggested_name = new_name
|
||||
s.project_existing_id = None
|
||||
s.project_existing_name = None
|
||||
s.project_match = "create_new"
|
||||
|
||||
if ov.get("location_id"):
|
||||
target_id = ov["location_id"]
|
||||
existing = db.query(svc.MonitoringLocation).filter_by(id=target_id).first()
|
||||
# Only attach if the location belongs to the (now chosen) project.
|
||||
chosen_project_id = s.project_existing_id
|
||||
if existing is not None and (
|
||||
chosen_project_id is None or existing.project_id == chosen_project_id
|
||||
):
|
||||
s.location_existing_id = existing.id
|
||||
s.location_existing_name = existing.name
|
||||
s.location_suggested_name = existing.name
|
||||
s.location_match = "exact"
|
||||
else:
|
||||
s.location_existing_id = None
|
||||
s.location_match = "create_new"
|
||||
elif "location_name" in ov:
|
||||
new_name = (ov["location_name"] or "").strip()
|
||||
if new_name:
|
||||
s.location_suggested_name = new_name
|
||||
s.location_existing_id = None
|
||||
s.location_existing_name = None
|
||||
s.location_match = "create_new"
|
||||
|
||||
selected.append(s)
|
||||
|
||||
apply_result = svc.apply_suggestions(db, selected, decided_by="operator")
|
||||
|
||||
# Invalidate the scan cache so the next /scan picks up the new state.
|
||||
_SCAN_CACHE["at"] = 0.0
|
||||
_SCAN_CACHE["result"] = None
|
||||
|
||||
return {
|
||||
"applied": apply_result.applied,
|
||||
"failed": [{"cluster_id": cid, "reason": r} for cid, r in apply_result.failed],
|
||||
"not_found": not_found,
|
||||
"project_ids_created": apply_result.project_ids_created,
|
||||
"location_ids_created": apply_result.location_ids_created,
|
||||
"assignment_ids_created": apply_result.assignment_ids_created,
|
||||
}
|
||||
|
||||
|
||||
@router.post("/skip")
|
||||
async def skip(
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Mark cluster_ids as skipped — they won't reappear in future scans."""
|
||||
try:
|
||||
body = await request.json()
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail="Invalid JSON body")
|
||||
|
||||
cluster_ids = body.get("cluster_ids") or []
|
||||
if not isinstance(cluster_ids, list):
|
||||
raise HTTPException(status_code=400, detail="cluster_ids must be a list")
|
||||
|
||||
n = svc.skip_clusters(db, cluster_ids, decided_by="operator")
|
||||
|
||||
_SCAN_CACHE["at"] = 0.0
|
||||
_SCAN_CACHE["result"] = None
|
||||
|
||||
return {"skipped": n}
|
||||
|
||||
|
||||
@router.get("/projects_search")
|
||||
def projects_search(
|
||||
q: str = "",
|
||||
limit: int = 10,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Typeahead search of existing projects for the wizard's per-cluster
|
||||
override inputs. Combines case-insensitive substring match with
|
||||
rapidfuzz scoring so partial typing and slight typos both surface
|
||||
candidates. Always returns a 'Create new' option at the end so the
|
||||
operator can confirm they want to create rather than match.
|
||||
|
||||
Returns:
|
||||
{
|
||||
"matches": [
|
||||
{"id": "...", "name": "...", "score": 0.91, "location_count": 3},
|
||||
...
|
||||
],
|
||||
"create_new": {"label": "Create new: \"<q>\""}
|
||||
}
|
||||
"""
|
||||
q_clean = (q or "").strip()
|
||||
q_norm = svc._normalise(q_clean)
|
||||
|
||||
projects = (
|
||||
db.query(Project)
|
||||
.filter(Project.status != "deleted")
|
||||
.all()
|
||||
)
|
||||
|
||||
scored: list[tuple[Project, float]] = []
|
||||
for p in projects:
|
||||
p_norm = svc._normalise(p.name)
|
||||
if not q_norm:
|
||||
# Empty query → return top projects by latest activity
|
||||
# (cheap heuristic: keep them all and sort by name).
|
||||
scored.append((p, 0.0))
|
||||
continue
|
||||
# Cheap substring boost: if the normalised query is a substring,
|
||||
# treat that as 1.0 regardless of WRatio.
|
||||
if q_norm in p_norm:
|
||||
scored.append((p, 1.0))
|
||||
continue
|
||||
score = svc.similarity(q_norm, p_norm)
|
||||
if score >= 0.50: # surfacing threshold; not the match threshold
|
||||
scored.append((p, score))
|
||||
|
||||
# Sort: score desc, then name asc.
|
||||
scored.sort(key=lambda t: (-t[1], t[0].name.lower()))
|
||||
scored = scored[:limit]
|
||||
|
||||
# Compute location counts in one batch query.
|
||||
loc_counts: dict[str, int] = {}
|
||||
if scored:
|
||||
from sqlalchemy import func
|
||||
ids = [p.id for p, _ in scored]
|
||||
rows = (
|
||||
db.query(MonitoringLocation.project_id, func.count(MonitoringLocation.id))
|
||||
.filter(MonitoringLocation.project_id.in_(ids))
|
||||
.group_by(MonitoringLocation.project_id)
|
||||
.all()
|
||||
)
|
||||
loc_counts = {pid: cnt for pid, cnt in rows}
|
||||
|
||||
return {
|
||||
"matches": [
|
||||
{
|
||||
"id": p.id,
|
||||
"name": p.name,
|
||||
"project_number": p.project_number,
|
||||
"client_name": p.client_name,
|
||||
"score": round(score, 3),
|
||||
"location_count": loc_counts.get(p.id, 0),
|
||||
}
|
||||
for p, score in scored
|
||||
],
|
||||
"create_new": {"label": f'Create new: "{q_clean}"' if q_clean else None},
|
||||
}
|
||||
|
||||
|
||||
@router.get("/locations_search")
|
||||
def locations_search(
|
||||
project_id: str,
|
||||
q: str = "",
|
||||
limit: int = 10,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Typeahead search of existing locations within a project."""
|
||||
if not project_id:
|
||||
raise HTTPException(status_code=400, detail="project_id required")
|
||||
|
||||
q_clean = (q or "").strip()
|
||||
q_norm = svc._normalise(q_clean)
|
||||
|
||||
locations = (
|
||||
db.query(MonitoringLocation)
|
||||
.filter(MonitoringLocation.project_id == project_id)
|
||||
.filter(MonitoringLocation.location_type == "vibration")
|
||||
.all()
|
||||
)
|
||||
|
||||
scored: list[tuple[MonitoringLocation, float]] = []
|
||||
for l in locations:
|
||||
l_norm = svc._normalise(l.name)
|
||||
if not q_norm:
|
||||
scored.append((l, 0.0))
|
||||
continue
|
||||
if q_norm in l_norm:
|
||||
scored.append((l, 1.0))
|
||||
continue
|
||||
score = svc.similarity(q_norm, l_norm)
|
||||
if score >= 0.50:
|
||||
scored.append((l, score))
|
||||
|
||||
scored.sort(key=lambda t: (-t[1], t[0].name.lower()))
|
||||
scored = scored[:limit]
|
||||
|
||||
return {
|
||||
"matches": [
|
||||
{
|
||||
"id": l.id,
|
||||
"name": l.name,
|
||||
"address": l.address,
|
||||
"score": round(score, 3),
|
||||
}
|
||||
for l, score in scored
|
||||
],
|
||||
"create_new": {"label": f'Create new: "{q_clean}"' if q_clean else None},
|
||||
}
|
||||
@@ -0,0 +1,429 @@
|
||||
"""
|
||||
Modem Dashboard Router
|
||||
|
||||
Provides API endpoints for the Field Modems management page.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Request, Depends, Query
|
||||
from fastapi.responses import HTMLResponse
|
||||
from sqlalchemy.orm import Session
|
||||
from datetime import datetime
|
||||
import subprocess
|
||||
import time
|
||||
import logging
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import RosterUnit
|
||||
from backend.templates_config import templates
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/modem-dashboard", tags=["modem-dashboard"])
|
||||
|
||||
|
||||
@router.get("/stats", response_class=HTMLResponse)
|
||||
async def get_modem_stats(request: Request, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Get summary statistics for modem dashboard.
|
||||
Returns HTML partial with stat cards.
|
||||
"""
|
||||
# Query all modems
|
||||
all_modems = db.query(RosterUnit).filter_by(device_type="modem").all()
|
||||
|
||||
# Get IDs of modems that have devices paired to them
|
||||
paired_modem_ids = set()
|
||||
devices_with_modems = db.query(RosterUnit).filter(
|
||||
RosterUnit.deployed_with_modem_id.isnot(None),
|
||||
RosterUnit.retired == False
|
||||
).all()
|
||||
for device in devices_with_modems:
|
||||
if device.deployed_with_modem_id:
|
||||
paired_modem_ids.add(device.deployed_with_modem_id)
|
||||
|
||||
# Count categories
|
||||
total_count = len(all_modems)
|
||||
retired_count = sum(1 for m in all_modems if m.retired)
|
||||
|
||||
# In use = deployed AND paired with a device
|
||||
in_use_count = sum(1 for m in all_modems
|
||||
if m.deployed and not m.retired and m.id in paired_modem_ids)
|
||||
|
||||
# Spare = deployed but NOT paired (available for assignment)
|
||||
spare_count = sum(1 for m in all_modems
|
||||
if m.deployed and not m.retired and m.id not in paired_modem_ids)
|
||||
|
||||
# Benched = not deployed and not retired
|
||||
benched_count = sum(1 for m in all_modems if not m.deployed and not m.retired)
|
||||
|
||||
return templates.TemplateResponse("partials/modem_stats.html", {
|
||||
"request": request,
|
||||
"total_count": total_count,
|
||||
"in_use_count": in_use_count,
|
||||
"spare_count": spare_count,
|
||||
"benched_count": benched_count,
|
||||
"retired_count": retired_count
|
||||
})
|
||||
|
||||
|
||||
@router.get("/units", response_class=HTMLResponse)
|
||||
async def get_modem_units(
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
search: str = Query(None),
|
||||
filter_status: str = Query(None), # "in_use", "spare", "benched", "retired"
|
||||
):
|
||||
"""
|
||||
Get list of modem units for the dashboard.
|
||||
Returns HTML partial with modem cards.
|
||||
"""
|
||||
query = db.query(RosterUnit).filter_by(device_type="modem")
|
||||
|
||||
# Filter by search term if provided
|
||||
if search:
|
||||
search_term = f"%{search}%"
|
||||
query = query.filter(
|
||||
(RosterUnit.id.ilike(search_term)) |
|
||||
(RosterUnit.ip_address.ilike(search_term)) |
|
||||
(RosterUnit.hardware_model.ilike(search_term)) |
|
||||
(RosterUnit.phone_number.ilike(search_term)) |
|
||||
(RosterUnit.location.ilike(search_term))
|
||||
)
|
||||
|
||||
modems = query.order_by(
|
||||
RosterUnit.retired.asc(),
|
||||
RosterUnit.deployed.desc(),
|
||||
RosterUnit.id.asc()
|
||||
).all()
|
||||
|
||||
# Get paired device info for each modem
|
||||
paired_devices = {}
|
||||
devices_with_modems = db.query(RosterUnit).filter(
|
||||
RosterUnit.deployed_with_modem_id.isnot(None),
|
||||
RosterUnit.retired == False
|
||||
).all()
|
||||
for device in devices_with_modems:
|
||||
if device.deployed_with_modem_id:
|
||||
paired_devices[device.deployed_with_modem_id] = {
|
||||
"id": device.id,
|
||||
"device_type": device.device_type,
|
||||
"deployed": device.deployed
|
||||
}
|
||||
|
||||
# Annotate modems with paired device info
|
||||
modem_list = []
|
||||
for modem in modems:
|
||||
paired = paired_devices.get(modem.id)
|
||||
|
||||
# Determine status category
|
||||
if modem.retired:
|
||||
status = "retired"
|
||||
elif not modem.deployed:
|
||||
status = "benched"
|
||||
elif paired:
|
||||
status = "in_use"
|
||||
else:
|
||||
status = "spare"
|
||||
|
||||
# Apply filter if specified
|
||||
if filter_status and status != filter_status:
|
||||
continue
|
||||
|
||||
modem_list.append({
|
||||
"id": modem.id,
|
||||
"ip_address": modem.ip_address,
|
||||
"phone_number": modem.phone_number,
|
||||
"hardware_model": modem.hardware_model,
|
||||
"deployed": modem.deployed,
|
||||
"retired": modem.retired,
|
||||
"location": modem.location,
|
||||
"project_id": modem.project_id,
|
||||
"paired_device": paired,
|
||||
"status": status
|
||||
})
|
||||
|
||||
return templates.TemplateResponse("partials/modem_list.html", {
|
||||
"request": request,
|
||||
"modems": modem_list
|
||||
})
|
||||
|
||||
|
||||
@router.get("/{modem_id}/paired-device")
|
||||
async def get_paired_device(modem_id: str, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Get the device (SLM/seismograph) that is paired with this modem.
|
||||
Returns JSON with device info or null if not paired.
|
||||
"""
|
||||
# Check modem exists
|
||||
modem = db.query(RosterUnit).filter_by(id=modem_id, device_type="modem").first()
|
||||
if not modem:
|
||||
return {"status": "error", "detail": f"Modem {modem_id} not found"}
|
||||
|
||||
# Find device paired with this modem
|
||||
device = db.query(RosterUnit).filter(
|
||||
RosterUnit.deployed_with_modem_id == modem_id,
|
||||
RosterUnit.retired == False
|
||||
).first()
|
||||
|
||||
if device:
|
||||
return {
|
||||
"paired": True,
|
||||
"device": {
|
||||
"id": device.id,
|
||||
"device_type": device.device_type,
|
||||
"deployed": device.deployed,
|
||||
"project_id": device.project_id,
|
||||
"location": device.location or device.address
|
||||
}
|
||||
}
|
||||
|
||||
return {"paired": False, "device": None}
|
||||
|
||||
|
||||
@router.get("/{modem_id}/paired-device-html", response_class=HTMLResponse)
|
||||
async def get_paired_device_html(modem_id: str, request: Request, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Get HTML partial showing the device paired with this modem.
|
||||
Used by unit_detail.html for modems.
|
||||
"""
|
||||
# Check modem exists
|
||||
modem = db.query(RosterUnit).filter_by(id=modem_id, device_type="modem").first()
|
||||
if not modem:
|
||||
return HTMLResponse('<p class="text-red-500">Modem not found</p>')
|
||||
|
||||
# Find device paired with this modem
|
||||
device = db.query(RosterUnit).filter(
|
||||
RosterUnit.deployed_with_modem_id == modem_id,
|
||||
RosterUnit.retired == False
|
||||
).first()
|
||||
|
||||
return templates.TemplateResponse("partials/modem_paired_device.html", {
|
||||
"request": request,
|
||||
"modem_id": modem_id,
|
||||
"device": device
|
||||
})
|
||||
|
||||
|
||||
@router.get("/{modem_id}/ping")
|
||||
async def ping_modem(modem_id: str, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Test modem connectivity with a simple ping.
|
||||
Returns response time and connection status.
|
||||
"""
|
||||
# Get modem from database
|
||||
modem = db.query(RosterUnit).filter_by(id=modem_id, device_type="modem").first()
|
||||
|
||||
if not modem:
|
||||
return {"status": "error", "detail": f"Modem {modem_id} not found"}
|
||||
|
||||
if not modem.ip_address:
|
||||
return {"status": "error", "detail": f"Modem {modem_id} has no IP address configured"}
|
||||
|
||||
try:
|
||||
# Ping the modem (1 packet, 2 second timeout)
|
||||
start_time = time.time()
|
||||
result = subprocess.run(
|
||||
["ping", "-c", "1", "-W", "2", modem.ip_address],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=3
|
||||
)
|
||||
response_time = int((time.time() - start_time) * 1000) # Convert to milliseconds
|
||||
|
||||
if result.returncode == 0:
|
||||
return {
|
||||
"status": "success",
|
||||
"modem_id": modem_id,
|
||||
"ip_address": modem.ip_address,
|
||||
"response_time_ms": response_time,
|
||||
"message": "Modem is responding"
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"status": "error",
|
||||
"modem_id": modem_id,
|
||||
"ip_address": modem.ip_address,
|
||||
"detail": "Modem not responding to ping"
|
||||
}
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
return {
|
||||
"status": "error",
|
||||
"modem_id": modem_id,
|
||||
"ip_address": modem.ip_address,
|
||||
"detail": "Ping timeout"
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to ping modem {modem_id}: {e}")
|
||||
return {
|
||||
"status": "error",
|
||||
"modem_id": modem_id,
|
||||
"detail": str(e)
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{modem_id}/diagnostics")
|
||||
async def get_modem_diagnostics(modem_id: str, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Get modem diagnostics (signal strength, data usage, uptime).
|
||||
|
||||
Currently returns placeholders. When ModemManager is available,
|
||||
this endpoint will query it for real diagnostics.
|
||||
"""
|
||||
modem = db.query(RosterUnit).filter_by(id=modem_id, device_type="modem").first()
|
||||
if not modem:
|
||||
return {"status": "error", "detail": f"Modem {modem_id} not found"}
|
||||
|
||||
# TODO: Query ModemManager backend when available
|
||||
return {
|
||||
"status": "unavailable",
|
||||
"message": "ModemManager integration not yet available",
|
||||
"modem_id": modem_id,
|
||||
"signal_strength_dbm": None,
|
||||
"data_usage_mb": None,
|
||||
"uptime_seconds": None,
|
||||
"carrier": None,
|
||||
"connection_type": None # LTE, 5G, etc.
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{modem_id}/pairable-devices")
|
||||
async def get_pairable_devices(
|
||||
modem_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
search: str = Query(None),
|
||||
hide_paired: bool = Query(True)
|
||||
):
|
||||
"""
|
||||
Get list of devices (seismographs and SLMs) that can be paired with this modem.
|
||||
Used by the device picker modal in unit_detail.html.
|
||||
"""
|
||||
# Check modem exists
|
||||
modem = db.query(RosterUnit).filter_by(id=modem_id, device_type="modem").first()
|
||||
if not modem:
|
||||
return {"status": "error", "detail": f"Modem {modem_id} not found"}
|
||||
|
||||
# Query seismographs and SLMs
|
||||
query = db.query(RosterUnit).filter(
|
||||
RosterUnit.device_type.in_(["seismograph", "sound_level_meter"]),
|
||||
RosterUnit.retired == False
|
||||
)
|
||||
|
||||
# Filter by search term if provided
|
||||
if search:
|
||||
search_term = f"%{search}%"
|
||||
query = query.filter(
|
||||
(RosterUnit.id.ilike(search_term)) |
|
||||
(RosterUnit.project_id.ilike(search_term)) |
|
||||
(RosterUnit.location.ilike(search_term)) |
|
||||
(RosterUnit.address.ilike(search_term)) |
|
||||
(RosterUnit.note.ilike(search_term))
|
||||
)
|
||||
|
||||
devices = query.order_by(
|
||||
RosterUnit.deployed.desc(),
|
||||
RosterUnit.device_type.asc(),
|
||||
RosterUnit.id.asc()
|
||||
).all()
|
||||
|
||||
# Build device list
|
||||
device_list = []
|
||||
for device in devices:
|
||||
# Skip already paired devices if hide_paired is True
|
||||
is_paired_to_other = (
|
||||
device.deployed_with_modem_id is not None and
|
||||
device.deployed_with_modem_id != modem_id
|
||||
)
|
||||
is_paired_to_this = device.deployed_with_modem_id == modem_id
|
||||
|
||||
if hide_paired and is_paired_to_other:
|
||||
continue
|
||||
|
||||
device_list.append({
|
||||
"id": device.id,
|
||||
"device_type": device.device_type,
|
||||
"deployed": device.deployed,
|
||||
"project_id": device.project_id,
|
||||
"location": device.location or device.address,
|
||||
"note": device.note,
|
||||
"paired_modem_id": device.deployed_with_modem_id,
|
||||
"is_paired_to_this": is_paired_to_this,
|
||||
"is_paired_to_other": is_paired_to_other
|
||||
})
|
||||
|
||||
return {"devices": device_list, "modem_id": modem_id}
|
||||
|
||||
|
||||
@router.post("/{modem_id}/pair")
|
||||
async def pair_device_to_modem(
|
||||
modem_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
device_id: str = Query(..., description="ID of the device to pair")
|
||||
):
|
||||
"""
|
||||
Pair a device (seismograph or SLM) to this modem.
|
||||
Updates the device's deployed_with_modem_id field.
|
||||
"""
|
||||
# Check modem exists
|
||||
modem = db.query(RosterUnit).filter_by(id=modem_id, device_type="modem").first()
|
||||
if not modem:
|
||||
return {"status": "error", "detail": f"Modem {modem_id} not found"}
|
||||
|
||||
# Find the device
|
||||
device = db.query(RosterUnit).filter(
|
||||
RosterUnit.id == device_id,
|
||||
RosterUnit.device_type.in_(["seismograph", "sound_level_meter"]),
|
||||
RosterUnit.retired == False
|
||||
).first()
|
||||
if not device:
|
||||
return {"status": "error", "detail": f"Device {device_id} not found"}
|
||||
|
||||
# Unpair any device currently paired to this modem
|
||||
currently_paired = db.query(RosterUnit).filter(
|
||||
RosterUnit.deployed_with_modem_id == modem_id
|
||||
).all()
|
||||
for paired_device in currently_paired:
|
||||
paired_device.deployed_with_modem_id = None
|
||||
|
||||
# Pair the new device
|
||||
device.deployed_with_modem_id = modem_id
|
||||
db.commit()
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"modem_id": modem_id,
|
||||
"device_id": device_id,
|
||||
"message": f"Device {device_id} paired to modem {modem_id}"
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{modem_id}/unpair")
|
||||
async def unpair_device_from_modem(modem_id: str, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Unpair any device currently paired to this modem.
|
||||
"""
|
||||
# Check modem exists
|
||||
modem = db.query(RosterUnit).filter_by(id=modem_id, device_type="modem").first()
|
||||
if not modem:
|
||||
return {"status": "error", "detail": f"Modem {modem_id} not found"}
|
||||
|
||||
# Find and unpair device
|
||||
device = db.query(RosterUnit).filter(
|
||||
RosterUnit.deployed_with_modem_id == modem_id
|
||||
).first()
|
||||
|
||||
if device:
|
||||
old_device_id = device.id
|
||||
device.deployed_with_modem_id = None
|
||||
db.commit()
|
||||
return {
|
||||
"status": "success",
|
||||
"modem_id": modem_id,
|
||||
"unpaired_device_id": old_device_id,
|
||||
"message": f"Device {old_device_id} unpaired from modem {modem_id}"
|
||||
}
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"modem_id": modem_id,
|
||||
"message": "No device was paired to this modem"
|
||||
}
|
||||
@@ -0,0 +1,522 @@
|
||||
"""
|
||||
Recurring Schedules Router
|
||||
|
||||
API endpoints for managing recurring monitoring schedules.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Request, Depends, HTTPException, Query
|
||||
from fastapi.responses import HTMLResponse, JSONResponse
|
||||
from sqlalchemy.orm import Session
|
||||
from typing import Optional
|
||||
from datetime import datetime
|
||||
import json
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import RecurringSchedule, MonitoringLocation, Project, RosterUnit
|
||||
from backend.services.recurring_schedule_service import get_recurring_schedule_service
|
||||
from backend.templates_config import templates
|
||||
|
||||
router = APIRouter(prefix="/api/projects/{project_id}/recurring-schedules", tags=["recurring-schedules"])
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# List and Get
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/")
|
||||
async def list_recurring_schedules(
|
||||
project_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
enabled_only: bool = Query(False),
|
||||
):
|
||||
"""
|
||||
List all recurring schedules for a project.
|
||||
"""
|
||||
project = db.query(Project).filter_by(id=project_id).first()
|
||||
if not project:
|
||||
raise HTTPException(status_code=404, detail="Project not found")
|
||||
|
||||
query = db.query(RecurringSchedule).filter_by(project_id=project_id)
|
||||
if enabled_only:
|
||||
query = query.filter_by(enabled=True)
|
||||
|
||||
schedules = query.order_by(RecurringSchedule.created_at.desc()).all()
|
||||
|
||||
return {
|
||||
"schedules": [
|
||||
{
|
||||
"id": s.id,
|
||||
"name": s.name,
|
||||
"schedule_type": s.schedule_type,
|
||||
"device_type": s.device_type,
|
||||
"location_id": s.location_id,
|
||||
"unit_id": s.unit_id,
|
||||
"enabled": s.enabled,
|
||||
"weekly_pattern": json.loads(s.weekly_pattern) if s.weekly_pattern else None,
|
||||
"interval_type": s.interval_type,
|
||||
"cycle_time": s.cycle_time,
|
||||
"include_download": s.include_download,
|
||||
"timezone": s.timezone,
|
||||
"next_occurrence": s.next_occurrence.isoformat() if s.next_occurrence else None,
|
||||
"last_generated_at": s.last_generated_at.isoformat() if s.last_generated_at else None,
|
||||
"created_at": s.created_at.isoformat() if s.created_at else None,
|
||||
}
|
||||
for s in schedules
|
||||
],
|
||||
"count": len(schedules),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{schedule_id}")
|
||||
async def get_recurring_schedule(
|
||||
project_id: str,
|
||||
schedule_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Get a specific recurring schedule.
|
||||
"""
|
||||
schedule = db.query(RecurringSchedule).filter_by(
|
||||
id=schedule_id,
|
||||
project_id=project_id,
|
||||
).first()
|
||||
|
||||
if not schedule:
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
|
||||
# Get related location and unit info
|
||||
location = db.query(MonitoringLocation).filter_by(id=schedule.location_id).first()
|
||||
unit = None
|
||||
if schedule.unit_id:
|
||||
unit = db.query(RosterUnit).filter_by(id=schedule.unit_id).first()
|
||||
|
||||
return {
|
||||
"id": schedule.id,
|
||||
"name": schedule.name,
|
||||
"schedule_type": schedule.schedule_type,
|
||||
"device_type": schedule.device_type,
|
||||
"location_id": schedule.location_id,
|
||||
"location_name": location.name if location else None,
|
||||
"unit_id": schedule.unit_id,
|
||||
"unit_name": unit.id if unit else None,
|
||||
"enabled": schedule.enabled,
|
||||
"weekly_pattern": json.loads(schedule.weekly_pattern) if schedule.weekly_pattern else None,
|
||||
"interval_type": schedule.interval_type,
|
||||
"cycle_time": schedule.cycle_time,
|
||||
"include_download": schedule.include_download,
|
||||
"timezone": schedule.timezone,
|
||||
"next_occurrence": schedule.next_occurrence.isoformat() if schedule.next_occurrence else None,
|
||||
"last_generated_at": schedule.last_generated_at.isoformat() if schedule.last_generated_at else None,
|
||||
"created_at": schedule.created_at.isoformat() if schedule.created_at else None,
|
||||
"updated_at": schedule.updated_at.isoformat() if schedule.updated_at else None,
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Create
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/")
|
||||
async def create_recurring_schedule(
|
||||
project_id: str,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Create recurring schedules for one or more locations.
|
||||
|
||||
Body for weekly_calendar (supports multiple locations):
|
||||
{
|
||||
"name": "Weeknight Monitoring",
|
||||
"schedule_type": "weekly_calendar",
|
||||
"location_ids": ["uuid1", "uuid2"], // Array of location IDs
|
||||
"weekly_pattern": {
|
||||
"monday": {"enabled": true, "start": "19:00", "end": "07:00"},
|
||||
"tuesday": {"enabled": false},
|
||||
...
|
||||
},
|
||||
"include_download": true,
|
||||
"auto_increment_index": true,
|
||||
"timezone": "America/New_York"
|
||||
}
|
||||
|
||||
Body for simple_interval (supports multiple locations):
|
||||
{
|
||||
"name": "24/7 Continuous",
|
||||
"schedule_type": "simple_interval",
|
||||
"location_ids": ["uuid1", "uuid2"], // Array of location IDs
|
||||
"interval_type": "daily",
|
||||
"cycle_time": "00:00",
|
||||
"include_download": true,
|
||||
"auto_increment_index": true,
|
||||
"timezone": "America/New_York"
|
||||
}
|
||||
|
||||
Legacy single location support (backwards compatible):
|
||||
{
|
||||
"name": "...",
|
||||
"location_id": "uuid", // Single location ID
|
||||
...
|
||||
}
|
||||
"""
|
||||
project = db.query(Project).filter_by(id=project_id).first()
|
||||
if not project:
|
||||
raise HTTPException(status_code=404, detail="Project not found")
|
||||
|
||||
data = await request.json()
|
||||
|
||||
# Support both location_ids (array) and location_id (single) for backwards compatibility
|
||||
location_ids = data.get("location_ids", [])
|
||||
if not location_ids and data.get("location_id"):
|
||||
location_ids = [data.get("location_id")]
|
||||
|
||||
if not location_ids:
|
||||
raise HTTPException(status_code=400, detail="At least one location is required")
|
||||
|
||||
# Validate all locations exist
|
||||
locations = db.query(MonitoringLocation).filter(
|
||||
MonitoringLocation.id.in_(location_ids),
|
||||
MonitoringLocation.project_id == project_id,
|
||||
).all()
|
||||
|
||||
if len(locations) != len(location_ids):
|
||||
raise HTTPException(status_code=404, detail="One or more locations not found")
|
||||
|
||||
service = get_recurring_schedule_service(db)
|
||||
created_schedules = []
|
||||
base_name = data.get("name", "Unnamed Schedule")
|
||||
|
||||
# Parse one-off datetime fields if applicable
|
||||
one_off_start = None
|
||||
one_off_end = None
|
||||
if data.get("schedule_type") == "one_off":
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
tz = ZoneInfo(data.get("timezone", "America/New_York"))
|
||||
|
||||
start_dt_str = data.get("start_datetime")
|
||||
end_dt_str = data.get("end_datetime")
|
||||
|
||||
if not start_dt_str or not end_dt_str:
|
||||
raise HTTPException(status_code=400, detail="One-off schedules require start and end date/time")
|
||||
|
||||
try:
|
||||
start_local = datetime.fromisoformat(start_dt_str).replace(tzinfo=tz)
|
||||
end_local = datetime.fromisoformat(end_dt_str).replace(tzinfo=tz)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid datetime format")
|
||||
|
||||
duration = end_local - start_local
|
||||
if duration.total_seconds() < 900:
|
||||
raise HTTPException(status_code=400, detail="Duration must be at least 15 minutes")
|
||||
if duration.total_seconds() > 86400:
|
||||
raise HTTPException(status_code=400, detail="Duration cannot exceed 24 hours")
|
||||
|
||||
from datetime import timezone as dt_timezone
|
||||
now_local = datetime.now(tz)
|
||||
if start_local <= now_local:
|
||||
raise HTTPException(status_code=400, detail="Start time must be in the future")
|
||||
|
||||
# Convert to UTC for storage
|
||||
one_off_start = start_local.astimezone(ZoneInfo("UTC")).replace(tzinfo=None)
|
||||
one_off_end = end_local.astimezone(ZoneInfo("UTC")).replace(tzinfo=None)
|
||||
|
||||
# Create a schedule for each location
|
||||
for location in locations:
|
||||
# Determine device type from location
|
||||
device_type = "slm" if location.location_type == "sound" else "seismograph"
|
||||
|
||||
# Append location name if multiple locations
|
||||
schedule_name = f"{base_name} - {location.name}" if len(locations) > 1 else base_name
|
||||
|
||||
schedule = service.create_schedule(
|
||||
project_id=project_id,
|
||||
location_id=location.id,
|
||||
name=schedule_name,
|
||||
schedule_type=data.get("schedule_type", "weekly_calendar"),
|
||||
device_type=device_type,
|
||||
unit_id=data.get("unit_id"),
|
||||
weekly_pattern=data.get("weekly_pattern"),
|
||||
interval_type=data.get("interval_type"),
|
||||
cycle_time=data.get("cycle_time"),
|
||||
include_download=data.get("include_download", True),
|
||||
auto_increment_index=data.get("auto_increment_index", True),
|
||||
timezone=data.get("timezone", "America/New_York"),
|
||||
start_datetime=one_off_start,
|
||||
end_datetime=one_off_end,
|
||||
)
|
||||
|
||||
# Generate actions immediately so they appear right away
|
||||
generated_actions = service.generate_actions_for_schedule(schedule, horizon_days=7)
|
||||
|
||||
created_schedules.append({
|
||||
"schedule_id": schedule.id,
|
||||
"location_id": location.id,
|
||||
"location_name": location.name,
|
||||
"actions_generated": len(generated_actions),
|
||||
})
|
||||
|
||||
total_actions = sum(s.get("actions_generated", 0) for s in created_schedules)
|
||||
|
||||
return JSONResponse({
|
||||
"success": True,
|
||||
"schedules": created_schedules,
|
||||
"count": len(created_schedules),
|
||||
"actions_generated": total_actions,
|
||||
"message": f"Created {len(created_schedules)} recurring schedule(s) with {total_actions} upcoming actions",
|
||||
})
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Update
|
||||
# ============================================================================
|
||||
|
||||
@router.put("/{schedule_id}")
|
||||
async def update_recurring_schedule(
|
||||
project_id: str,
|
||||
schedule_id: str,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Update a recurring schedule.
|
||||
"""
|
||||
schedule = db.query(RecurringSchedule).filter_by(
|
||||
id=schedule_id,
|
||||
project_id=project_id,
|
||||
).first()
|
||||
|
||||
if not schedule:
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
|
||||
data = await request.json()
|
||||
service = get_recurring_schedule_service(db)
|
||||
|
||||
# Build update kwargs
|
||||
update_kwargs = {}
|
||||
for field in ["name", "weekly_pattern", "interval_type", "cycle_time",
|
||||
"include_download", "auto_increment_index", "timezone", "unit_id"]:
|
||||
if field in data:
|
||||
update_kwargs[field] = data[field]
|
||||
|
||||
updated = service.update_schedule(schedule_id, **update_kwargs)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"schedule_id": updated.id,
|
||||
"message": "Schedule updated successfully",
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Delete
|
||||
# ============================================================================
|
||||
|
||||
@router.delete("/{schedule_id}")
|
||||
async def delete_recurring_schedule(
|
||||
project_id: str,
|
||||
schedule_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Delete a recurring schedule.
|
||||
"""
|
||||
service = get_recurring_schedule_service(db)
|
||||
deleted = service.delete_schedule(schedule_id)
|
||||
|
||||
if not deleted:
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"message": "Schedule deleted successfully",
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Enable/Disable
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/{schedule_id}/enable")
|
||||
async def enable_schedule(
|
||||
project_id: str,
|
||||
schedule_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Enable a disabled schedule.
|
||||
"""
|
||||
service = get_recurring_schedule_service(db)
|
||||
schedule = service.enable_schedule(schedule_id)
|
||||
|
||||
if not schedule:
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"schedule_id": schedule.id,
|
||||
"enabled": schedule.enabled,
|
||||
"message": "Schedule enabled",
|
||||
}
|
||||
|
||||
|
||||
@router.post("/{schedule_id}/disable")
|
||||
async def disable_schedule(
|
||||
project_id: str,
|
||||
schedule_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Disable a schedule and cancel all its pending actions.
|
||||
"""
|
||||
service = get_recurring_schedule_service(db)
|
||||
|
||||
# Count pending actions before disabling (for response message)
|
||||
from sqlalchemy import and_
|
||||
from backend.models import ScheduledAction
|
||||
pending_count = db.query(ScheduledAction).filter(
|
||||
and_(
|
||||
ScheduledAction.execution_status == "pending",
|
||||
ScheduledAction.notes.like(f'%"schedule_id": "{schedule_id}"%'),
|
||||
)
|
||||
).count()
|
||||
|
||||
schedule = service.disable_schedule(schedule_id)
|
||||
|
||||
if not schedule:
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
|
||||
message = "Schedule disabled"
|
||||
if pending_count > 0:
|
||||
message += f" and {pending_count} pending action(s) cancelled"
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"schedule_id": schedule.id,
|
||||
"enabled": schedule.enabled,
|
||||
"cancelled_actions": pending_count,
|
||||
"message": message,
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Preview Generated Actions
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/{schedule_id}/generate-preview")
|
||||
async def preview_generated_actions(
|
||||
project_id: str,
|
||||
schedule_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
days: int = Query(7, ge=1, le=30),
|
||||
):
|
||||
"""
|
||||
Preview what actions would be generated without saving them.
|
||||
"""
|
||||
schedule = db.query(RecurringSchedule).filter_by(
|
||||
id=schedule_id,
|
||||
project_id=project_id,
|
||||
).first()
|
||||
|
||||
if not schedule:
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
|
||||
service = get_recurring_schedule_service(db)
|
||||
actions = service.generate_actions_for_schedule(
|
||||
schedule,
|
||||
horizon_days=days,
|
||||
preview_only=True,
|
||||
)
|
||||
|
||||
return {
|
||||
"schedule_id": schedule_id,
|
||||
"schedule_name": schedule.name,
|
||||
"preview_days": days,
|
||||
"actions": [
|
||||
{
|
||||
"action_type": a.action_type,
|
||||
"scheduled_time": a.scheduled_time.isoformat(),
|
||||
"notes": a.notes,
|
||||
}
|
||||
for a in actions
|
||||
],
|
||||
"action_count": len(actions),
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Manual Generation Trigger
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/{schedule_id}/generate")
|
||||
async def generate_actions_now(
|
||||
project_id: str,
|
||||
schedule_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
days: int = Query(7, ge=1, le=30),
|
||||
):
|
||||
"""
|
||||
Manually trigger action generation for a schedule.
|
||||
"""
|
||||
schedule = db.query(RecurringSchedule).filter_by(
|
||||
id=schedule_id,
|
||||
project_id=project_id,
|
||||
).first()
|
||||
|
||||
if not schedule:
|
||||
raise HTTPException(status_code=404, detail="Schedule not found")
|
||||
|
||||
if not schedule.enabled:
|
||||
raise HTTPException(status_code=400, detail="Schedule is disabled")
|
||||
|
||||
service = get_recurring_schedule_service(db)
|
||||
actions = service.generate_actions_for_schedule(
|
||||
schedule,
|
||||
horizon_days=days,
|
||||
preview_only=False,
|
||||
)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"schedule_id": schedule_id,
|
||||
"generated_count": len(actions),
|
||||
"message": f"Generated {len(actions)} scheduled actions",
|
||||
}
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# HTML Partials
|
||||
# ============================================================================
|
||||
|
||||
@router.get("/partials/list", response_class=HTMLResponse)
|
||||
async def get_schedule_list_partial(
|
||||
project_id: str,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Return HTML partial for schedule list.
|
||||
"""
|
||||
project = db.query(Project).filter_by(id=project_id).first()
|
||||
project_status = project.status if project else "active"
|
||||
|
||||
schedules = db.query(RecurringSchedule).filter_by(
|
||||
project_id=project_id
|
||||
).order_by(RecurringSchedule.created_at.desc()).all()
|
||||
|
||||
# Enrich with location info
|
||||
schedule_data = []
|
||||
for s in schedules:
|
||||
location = db.query(MonitoringLocation).filter_by(id=s.location_id).first()
|
||||
schedule_data.append({
|
||||
"schedule": s,
|
||||
"location": location,
|
||||
"pattern": json.loads(s.weekly_pattern) if s.weekly_pattern else None,
|
||||
})
|
||||
|
||||
return templates.TemplateResponse("partials/projects/recurring_schedule_list.html", {
|
||||
"request": request,
|
||||
"project_id": project_id,
|
||||
"schedules": schedule_data,
|
||||
"project_status": project_status,
|
||||
})
|
||||
@@ -0,0 +1,187 @@
|
||||
"""
|
||||
Report Templates Router
|
||||
|
||||
CRUD operations for report template management.
|
||||
Templates store time filter presets and report configuration for reuse.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi.responses import JSONResponse
|
||||
from sqlalchemy.orm import Session
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
import uuid
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import ReportTemplate
|
||||
|
||||
router = APIRouter(prefix="/api/report-templates", tags=["report-templates"])
|
||||
|
||||
|
||||
@router.get("")
|
||||
async def list_templates(
|
||||
project_id: Optional[str] = None,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
List all report templates.
|
||||
Optionally filter by project_id (includes global templates with project_id=None).
|
||||
"""
|
||||
query = db.query(ReportTemplate)
|
||||
|
||||
if project_id:
|
||||
# Include global templates (project_id=None) AND project-specific templates
|
||||
query = query.filter(
|
||||
(ReportTemplate.project_id == None) | (ReportTemplate.project_id == project_id)
|
||||
)
|
||||
|
||||
templates = query.order_by(ReportTemplate.name).all()
|
||||
|
||||
return [
|
||||
{
|
||||
"id": t.id,
|
||||
"name": t.name,
|
||||
"project_id": t.project_id,
|
||||
"report_title": t.report_title,
|
||||
"start_time": t.start_time,
|
||||
"end_time": t.end_time,
|
||||
"start_date": t.start_date,
|
||||
"end_date": t.end_date,
|
||||
"created_at": t.created_at.isoformat() if t.created_at else None,
|
||||
"updated_at": t.updated_at.isoformat() if t.updated_at else None,
|
||||
}
|
||||
for t in templates
|
||||
]
|
||||
|
||||
|
||||
@router.post("")
|
||||
async def create_template(
|
||||
data: dict,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Create a new report template.
|
||||
|
||||
Request body:
|
||||
- name: Template name (required)
|
||||
- project_id: Optional project ID for project-specific template
|
||||
- report_title: Default report title
|
||||
- start_time: Start time filter (HH:MM format)
|
||||
- end_time: End time filter (HH:MM format)
|
||||
- start_date: Start date filter (YYYY-MM-DD format)
|
||||
- end_date: End date filter (YYYY-MM-DD format)
|
||||
"""
|
||||
name = data.get("name")
|
||||
if not name:
|
||||
raise HTTPException(status_code=400, detail="Template name is required")
|
||||
|
||||
template = ReportTemplate(
|
||||
id=str(uuid.uuid4()),
|
||||
name=name,
|
||||
project_id=data.get("project_id"),
|
||||
report_title=data.get("report_title", "Background Noise Study"),
|
||||
start_time=data.get("start_time"),
|
||||
end_time=data.get("end_time"),
|
||||
start_date=data.get("start_date"),
|
||||
end_date=data.get("end_date"),
|
||||
)
|
||||
|
||||
db.add(template)
|
||||
db.commit()
|
||||
db.refresh(template)
|
||||
|
||||
return {
|
||||
"id": template.id,
|
||||
"name": template.name,
|
||||
"project_id": template.project_id,
|
||||
"report_title": template.report_title,
|
||||
"start_time": template.start_time,
|
||||
"end_time": template.end_time,
|
||||
"start_date": template.start_date,
|
||||
"end_date": template.end_date,
|
||||
"created_at": template.created_at.isoformat() if template.created_at else None,
|
||||
}
|
||||
|
||||
|
||||
@router.get("/{template_id}")
|
||||
async def get_template(
|
||||
template_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Get a specific report template by ID."""
|
||||
template = db.query(ReportTemplate).filter_by(id=template_id).first()
|
||||
if not template:
|
||||
raise HTTPException(status_code=404, detail="Template not found")
|
||||
|
||||
return {
|
||||
"id": template.id,
|
||||
"name": template.name,
|
||||
"project_id": template.project_id,
|
||||
"report_title": template.report_title,
|
||||
"start_time": template.start_time,
|
||||
"end_time": template.end_time,
|
||||
"start_date": template.start_date,
|
||||
"end_date": template.end_date,
|
||||
"created_at": template.created_at.isoformat() if template.created_at else None,
|
||||
"updated_at": template.updated_at.isoformat() if template.updated_at else None,
|
||||
}
|
||||
|
||||
|
||||
@router.put("/{template_id}")
|
||||
async def update_template(
|
||||
template_id: str,
|
||||
data: dict,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Update an existing report template."""
|
||||
template = db.query(ReportTemplate).filter_by(id=template_id).first()
|
||||
if not template:
|
||||
raise HTTPException(status_code=404, detail="Template not found")
|
||||
|
||||
# Update fields if provided
|
||||
if "name" in data:
|
||||
template.name = data["name"]
|
||||
if "project_id" in data:
|
||||
template.project_id = data["project_id"]
|
||||
if "report_title" in data:
|
||||
template.report_title = data["report_title"]
|
||||
if "start_time" in data:
|
||||
template.start_time = data["start_time"]
|
||||
if "end_time" in data:
|
||||
template.end_time = data["end_time"]
|
||||
if "start_date" in data:
|
||||
template.start_date = data["start_date"]
|
||||
if "end_date" in data:
|
||||
template.end_date = data["end_date"]
|
||||
|
||||
template.updated_at = datetime.utcnow()
|
||||
db.commit()
|
||||
db.refresh(template)
|
||||
|
||||
return {
|
||||
"id": template.id,
|
||||
"name": template.name,
|
||||
"project_id": template.project_id,
|
||||
"report_title": template.report_title,
|
||||
"start_time": template.start_time,
|
||||
"end_time": template.end_time,
|
||||
"start_date": template.start_date,
|
||||
"end_date": template.end_date,
|
||||
"updated_at": template.updated_at.isoformat() if template.updated_at else None,
|
||||
}
|
||||
|
||||
|
||||
@router.delete("/{template_id}")
|
||||
async def delete_template(
|
||||
template_id: str,
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""Delete a report template."""
|
||||
template = db.query(ReportTemplate).filter_by(id=template_id).first()
|
||||
if not template:
|
||||
raise HTTPException(status_code=404, detail="Template not found")
|
||||
|
||||
db.delete(template)
|
||||
db.commit()
|
||||
|
||||
return JSONResponse({"status": "success", "message": "Template deleted"})
|
||||
@@ -2,20 +2,32 @@ from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.orm import Session
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Dict, Any
|
||||
import asyncio
|
||||
import logging
|
||||
import random
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.services.snapshot import emit_status_snapshot
|
||||
from backend.services.slm_status_sync import sync_slm_status_to_emitters
|
||||
|
||||
router = APIRouter(prefix="/api", tags=["roster"])
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@router.get("/status-snapshot")
|
||||
def get_status_snapshot(db: Session = Depends(get_db)):
|
||||
async def get_status_snapshot(db: Session = Depends(get_db)):
|
||||
"""
|
||||
Calls emit_status_snapshot() to get current fleet status.
|
||||
This will be replaced with real Series3 emitter logic later.
|
||||
Syncs SLM status from SLMM before generating snapshot.
|
||||
"""
|
||||
# Sync SLM status from SLMM (with timeout to prevent blocking)
|
||||
try:
|
||||
await asyncio.wait_for(sync_slm_status_to_emitters(), timeout=2.0)
|
||||
except asyncio.TimeoutError:
|
||||
logger.warning("SLM status sync timed out, using cached data")
|
||||
except Exception as e:
|
||||
logger.warning(f"SLM status sync failed: {e}")
|
||||
|
||||
return emit_status_snapshot()
|
||||
|
||||
|
||||
|
||||
@@ -92,21 +92,21 @@ async def rename_unit(
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not update unit_assignments: {e}")
|
||||
|
||||
# Update recording_sessions table (if exists)
|
||||
# Update monitoring_sessions table (if exists)
|
||||
try:
|
||||
from backend.models import RecordingSession
|
||||
db.query(RecordingSession).filter(RecordingSession.unit_id == old_id).update(
|
||||
from backend.models import MonitoringSession
|
||||
db.query(MonitoringSession).filter(MonitoringSession.unit_id == old_id).update(
|
||||
{"unit_id": new_id},
|
||||
synchronize_session=False
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not update recording_sessions: {e}")
|
||||
logger.warning(f"Could not update monitoring_sessions: {e}")
|
||||
|
||||
# Commit all changes
|
||||
db.commit()
|
||||
|
||||
# If sound level meter, sync updated config to SLMM cache
|
||||
if device_type == "sound_level_meter":
|
||||
if device_type == "slm":
|
||||
logger.info(f"Syncing renamed SLM {new_id} (was {old_id}) config to SLMM cache...")
|
||||
result = await sync_slm_to_slmm_cache(
|
||||
unit_id=new_id,
|
||||
|
||||
@@ -5,7 +5,6 @@ Handles scheduled actions for automated recording control.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Request, Depends, HTTPException, Query
|
||||
from fastapi.templating import Jinja2Templates
|
||||
from fastapi.responses import HTMLResponse, JSONResponse
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import and_, or_
|
||||
@@ -23,9 +22,9 @@ from backend.models import (
|
||||
RosterUnit,
|
||||
)
|
||||
from backend.services.scheduler import get_scheduler
|
||||
from backend.templates_config import templates
|
||||
|
||||
router = APIRouter(prefix="/api/projects/{project_id}/scheduler", tags=["scheduler"])
|
||||
templates = Jinja2Templates(directory="templates")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
@@ -131,7 +130,7 @@ async def create_scheduled_action(
|
||||
raise HTTPException(status_code=404, detail="Location not found")
|
||||
|
||||
# Determine device type from location
|
||||
device_type = "sound_level_meter" if location.location_type == "sound" else "seismograph"
|
||||
device_type = "slm" if location.location_type == "sound" else "seismograph"
|
||||
|
||||
# Get unit_id (optional - can be determined from assignment at execution time)
|
||||
unit_id = form_data.get("unit_id")
|
||||
@@ -188,7 +187,7 @@ async def schedule_recording_session(
|
||||
if not location:
|
||||
raise HTTPException(status_code=404, detail="Location not found")
|
||||
|
||||
device_type = "sound_level_meter" if location.location_type == "sound" else "seismograph"
|
||||
device_type = "slm" if location.location_type == "sound" else "seismograph"
|
||||
unit_id = form_data.get("unit_id")
|
||||
|
||||
start_time = datetime.fromisoformat(form_data.get("start_time"))
|
||||
|
||||
@@ -3,15 +3,16 @@ Seismograph Dashboard API Router
|
||||
Provides endpoints for the seismograph-specific dashboard
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Request, Depends, Query
|
||||
from datetime import date, datetime, timedelta
|
||||
|
||||
from fastapi import APIRouter, Request, Depends, Query, Form, HTTPException
|
||||
from fastapi.responses import HTMLResponse
|
||||
from fastapi.templating import Jinja2Templates
|
||||
from sqlalchemy.orm import Session
|
||||
from backend.database import get_db
|
||||
from backend.models import RosterUnit
|
||||
from backend.models import RosterUnit, UnitHistory, UserPreferences
|
||||
from backend.templates_config import templates
|
||||
|
||||
router = APIRouter(prefix="/api/seismo-dashboard", tags=["seismo-dashboard"])
|
||||
templates = Jinja2Templates(directory="templates")
|
||||
|
||||
|
||||
@router.get("/stats", response_class=HTMLResponse)
|
||||
@@ -27,7 +28,8 @@ async def get_seismo_stats(request: Request, db: Session = Depends(get_db)):
|
||||
|
||||
total = len(seismos)
|
||||
deployed = sum(1 for s in seismos if s.deployed)
|
||||
benched = sum(1 for s in seismos if not s.deployed)
|
||||
benched = sum(1 for s in seismos if not s.deployed and not s.out_for_calibration)
|
||||
out_for_calibration = sum(1 for s in seismos if s.out_for_calibration)
|
||||
|
||||
# Count modems assigned to deployed seismographs
|
||||
with_modem = sum(1 for s in seismos if s.deployed and s.deployed_with_modem_id)
|
||||
@@ -40,6 +42,7 @@ async def get_seismo_stats(request: Request, db: Session = Depends(get_db)):
|
||||
"total": total,
|
||||
"deployed": deployed,
|
||||
"benched": benched,
|
||||
"out_for_calibration": out_for_calibration,
|
||||
"with_modem": with_modem,
|
||||
"without_modem": without_modem
|
||||
}
|
||||
@@ -50,10 +53,14 @@ async def get_seismo_stats(request: Request, db: Session = Depends(get_db)):
|
||||
async def get_seismo_units(
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
search: str = Query(None)
|
||||
search: str = Query(None),
|
||||
sort: str = Query("id"),
|
||||
order: str = Query("asc"),
|
||||
status: str = Query(None),
|
||||
modem: str = Query(None)
|
||||
):
|
||||
"""
|
||||
Returns HTML partial with filterable seismograph unit list
|
||||
Returns HTML partial with filterable and sortable seismograph unit list
|
||||
"""
|
||||
query = db.query(RosterUnit).filter_by(
|
||||
device_type="seismograph",
|
||||
@@ -62,20 +69,160 @@ async def get_seismo_units(
|
||||
|
||||
# Apply search filter
|
||||
if search:
|
||||
search_lower = search.lower()
|
||||
query = query.filter(
|
||||
(RosterUnit.id.ilike(f"%{search}%")) |
|
||||
(RosterUnit.note.ilike(f"%{search}%")) |
|
||||
(RosterUnit.address.ilike(f"%{search}%"))
|
||||
)
|
||||
|
||||
seismos = query.order_by(RosterUnit.id).all()
|
||||
# Apply status filter
|
||||
if status == "deployed":
|
||||
query = query.filter(RosterUnit.deployed == True)
|
||||
elif status == "benched":
|
||||
query = query.filter(RosterUnit.deployed == False, RosterUnit.out_for_calibration == False)
|
||||
elif status == "out_for_calibration":
|
||||
query = query.filter(RosterUnit.out_for_calibration == True)
|
||||
|
||||
# Apply modem filter
|
||||
if modem == "with":
|
||||
query = query.filter(RosterUnit.deployed_with_modem_id.isnot(None))
|
||||
elif modem == "without":
|
||||
query = query.filter(RosterUnit.deployed_with_modem_id.is_(None))
|
||||
|
||||
# Apply sorting
|
||||
sort_column_map = {
|
||||
"id": RosterUnit.id,
|
||||
"status": RosterUnit.deployed,
|
||||
"modem": RosterUnit.deployed_with_modem_id,
|
||||
"location": RosterUnit.address,
|
||||
"last_calibrated": RosterUnit.last_calibrated,
|
||||
"notes": RosterUnit.note
|
||||
}
|
||||
sort_column = sort_column_map.get(sort, RosterUnit.id)
|
||||
|
||||
if order == "desc":
|
||||
query = query.order_by(sort_column.desc())
|
||||
else:
|
||||
query = query.order_by(sort_column.asc())
|
||||
|
||||
seismos = query.all()
|
||||
|
||||
return templates.TemplateResponse(
|
||||
"partials/seismo_unit_list.html",
|
||||
{
|
||||
"request": request,
|
||||
"units": seismos,
|
||||
"search": search or ""
|
||||
"search": search or "",
|
||||
"sort": sort,
|
||||
"order": order,
|
||||
"status": status or "",
|
||||
"modem": modem or "",
|
||||
"today": date.today()
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def _get_calibration_interval(db: Session) -> int:
|
||||
prefs = db.query(UserPreferences).first()
|
||||
if prefs and prefs.calibration_interval_days:
|
||||
return prefs.calibration_interval_days
|
||||
return 365
|
||||
|
||||
|
||||
def _row_context(request: Request, unit: RosterUnit) -> dict:
|
||||
return {"request": request, "unit": unit, "today": date.today()}
|
||||
|
||||
|
||||
@router.get("/unit/{unit_id}/view-row", response_class=HTMLResponse)
|
||||
async def get_seismo_view_row(unit_id: str, request: Request, db: Session = Depends(get_db)):
|
||||
unit = db.query(RosterUnit).filter(RosterUnit.id == unit_id).first()
|
||||
if not unit:
|
||||
raise HTTPException(status_code=404, detail="Unit not found")
|
||||
return templates.TemplateResponse("partials/seismo_row_view.html", _row_context(request, unit))
|
||||
|
||||
|
||||
@router.get("/unit/{unit_id}/edit-row", response_class=HTMLResponse)
|
||||
async def get_seismo_edit_row(unit_id: str, request: Request, db: Session = Depends(get_db)):
|
||||
unit = db.query(RosterUnit).filter(RosterUnit.id == unit_id).first()
|
||||
if not unit:
|
||||
raise HTTPException(status_code=404, detail="Unit not found")
|
||||
return templates.TemplateResponse("partials/seismo_row_edit.html", _row_context(request, unit))
|
||||
|
||||
|
||||
@router.post("/unit/{unit_id}/quick-update", response_class=HTMLResponse)
|
||||
async def quick_update_seismo_unit(
|
||||
unit_id: str,
|
||||
request: Request,
|
||||
db: Session = Depends(get_db),
|
||||
status: str = Form(...),
|
||||
last_calibrated: str = Form(""),
|
||||
note: str = Form(""),
|
||||
):
|
||||
unit = db.query(RosterUnit).filter(RosterUnit.id == unit_id).first()
|
||||
if not unit:
|
||||
raise HTTPException(status_code=404, detail="Unit not found")
|
||||
|
||||
# --- Status ---
|
||||
old_deployed = unit.deployed
|
||||
old_out_for_cal = unit.out_for_calibration
|
||||
if status == "deployed":
|
||||
unit.deployed = True
|
||||
unit.out_for_calibration = False
|
||||
elif status == "out_for_calibration":
|
||||
unit.deployed = False
|
||||
unit.out_for_calibration = True
|
||||
else:
|
||||
unit.deployed = False
|
||||
unit.out_for_calibration = False
|
||||
|
||||
if unit.deployed != old_deployed or unit.out_for_calibration != old_out_for_cal:
|
||||
old_status = "deployed" if old_deployed else ("out_for_calibration" if old_out_for_cal else "benched")
|
||||
db.add(UnitHistory(
|
||||
unit_id=unit_id,
|
||||
change_type="deployed_change",
|
||||
field_name="status",
|
||||
old_value=old_status,
|
||||
new_value=status,
|
||||
source="manual",
|
||||
))
|
||||
|
||||
# --- Last calibrated ---
|
||||
old_cal = unit.last_calibrated
|
||||
if last_calibrated:
|
||||
try:
|
||||
new_cal = datetime.strptime(last_calibrated, "%Y-%m-%d").date()
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=400, detail="Invalid date format. Use YYYY-MM-DD")
|
||||
unit.last_calibrated = new_cal
|
||||
unit.next_calibration_due = new_cal + timedelta(days=_get_calibration_interval(db))
|
||||
else:
|
||||
unit.last_calibrated = None
|
||||
unit.next_calibration_due = None
|
||||
|
||||
if unit.last_calibrated != old_cal:
|
||||
db.add(UnitHistory(
|
||||
unit_id=unit_id,
|
||||
change_type="calibration_status_change",
|
||||
field_name="last_calibrated",
|
||||
old_value=old_cal.strftime("%Y-%m-%d") if old_cal else None,
|
||||
new_value=last_calibrated or None,
|
||||
source="manual",
|
||||
))
|
||||
|
||||
# --- Note ---
|
||||
old_note = unit.note
|
||||
unit.note = note or None
|
||||
if unit.note != old_note:
|
||||
db.add(UnitHistory(
|
||||
unit_id=unit_id,
|
||||
change_type="note_change",
|
||||
field_name="note",
|
||||
old_value=old_note,
|
||||
new_value=unit.note,
|
||||
source="manual",
|
||||
))
|
||||
|
||||
db.commit()
|
||||
db.refresh(unit)
|
||||
|
||||
return templates.TemplateResponse("partials/seismo_row_view.html", _row_context(request, unit))
|
||||
|
||||
@@ -477,3 +477,75 @@ async def upload_snapshot(file: UploadFile = File(...)):
|
||||
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Upload failed: {str(e)}")
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# SLMM SYNC ENDPOINTS
|
||||
# ============================================================================
|
||||
|
||||
@router.post("/slmm/sync-all")
|
||||
async def sync_all_slms(db: Session = Depends(get_db)):
|
||||
"""
|
||||
Manually trigger full sync of all SLM devices from Terra-View roster to SLMM.
|
||||
|
||||
This ensures SLMM database matches Terra-View roster (source of truth).
|
||||
Also cleans up orphaned devices in SLMM that are not in Terra-View.
|
||||
"""
|
||||
from backend.services.slmm_sync import sync_all_slms_to_slmm, cleanup_orphaned_slmm_devices
|
||||
|
||||
try:
|
||||
# Sync all SLMs
|
||||
sync_results = await sync_all_slms_to_slmm(db)
|
||||
|
||||
# Clean up orphaned devices
|
||||
cleanup_results = await cleanup_orphaned_slmm_devices(db)
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"sync": sync_results,
|
||||
"cleanup": cleanup_results
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Sync failed: {str(e)}")
|
||||
|
||||
|
||||
@router.get("/slmm/status")
|
||||
async def get_slmm_sync_status(db: Session = Depends(get_db)):
|
||||
"""
|
||||
Get status of SLMM synchronization.
|
||||
|
||||
Shows which devices are in Terra-View roster vs SLMM database.
|
||||
"""
|
||||
from backend.services.slmm_sync import get_slmm_devices
|
||||
|
||||
try:
|
||||
# Get devices from both systems
|
||||
roster_slms = db.query(RosterUnit).filter_by(device_type="slm").all()
|
||||
slmm_devices = await get_slmm_devices()
|
||||
|
||||
if slmm_devices is None:
|
||||
raise HTTPException(status_code=503, detail="SLMM service unavailable")
|
||||
|
||||
roster_unit_ids = {unit.unit_type for unit in roster_slms}
|
||||
slmm_unit_ids = set(slmm_devices)
|
||||
|
||||
# Find differences
|
||||
in_roster_only = roster_unit_ids - slmm_unit_ids
|
||||
in_slmm_only = slmm_unit_ids - roster_unit_ids
|
||||
in_both = roster_unit_ids & slmm_unit_ids
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"terra_view_total": len(roster_unit_ids),
|
||||
"slmm_total": len(slmm_unit_ids),
|
||||
"synced": len(in_both),
|
||||
"missing_from_slmm": list(in_roster_only),
|
||||
"orphaned_in_slmm": list(in_slmm_only),
|
||||
"in_sync": len(in_roster_only) == 0 and len(in_slmm_only) == 0
|
||||
}
|
||||
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise HTTPException(status_code=500, detail=f"Status check failed: {str(e)}")
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
"""
|
||||
SFM (Seismograph Field Module) Proxy Router
|
||||
|
||||
Proxies requests from terra-view to the standalone SFM backend service.
|
||||
SFM runs on port 8200 and handles MiniMate Plus seismograph communication
|
||||
and event database queries.
|
||||
|
||||
SFM endpoints are at root level (e.g. /db/units, /device/info) — no /api/ prefix.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, HTTPException, Request, Response
|
||||
import httpx
|
||||
import logging
|
||||
import os
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/sfm", tags=["sfm"])
|
||||
|
||||
# SFM backend URL - configurable via environment variable
|
||||
SFM_BASE_URL = os.getenv("SFM_BASE_URL", "http://localhost:8200")
|
||||
|
||||
|
||||
@router.get("/health")
|
||||
async def check_sfm_health():
|
||||
"""
|
||||
Check if the SFM backend service is reachable and healthy.
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
response = await client.get(f"{SFM_BASE_URL}/health")
|
||||
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
return {
|
||||
"status": "ok",
|
||||
"sfm_status": "connected",
|
||||
"sfm_url": SFM_BASE_URL,
|
||||
"sfm_response": data
|
||||
}
|
||||
else:
|
||||
return {
|
||||
"status": "degraded",
|
||||
"sfm_status": "error",
|
||||
"sfm_url": SFM_BASE_URL,
|
||||
"detail": f"SFM returned status {response.status_code}"
|
||||
}
|
||||
|
||||
except httpx.ConnectError:
|
||||
return {
|
||||
"status": "error",
|
||||
"sfm_status": "unreachable",
|
||||
"sfm_url": SFM_BASE_URL,
|
||||
"detail": "Cannot connect to SFM backend. Is it running?"
|
||||
}
|
||||
except Exception as e:
|
||||
return {
|
||||
"status": "error",
|
||||
"sfm_status": "error",
|
||||
"sfm_url": SFM_BASE_URL,
|
||||
"detail": str(e)
|
||||
}
|
||||
|
||||
|
||||
# HTTP catch-all — proxies everything else to SFM backend
|
||||
@router.api_route("/{path:path}", methods=["GET", "POST", "PUT", "DELETE", "PATCH"])
|
||||
async def proxy_to_sfm(path: str, request: Request):
|
||||
"""
|
||||
Proxy all requests to the SFM backend service.
|
||||
|
||||
SFM endpoints have no /api/ prefix — target URL is {SFM_BASE_URL}/{path}.
|
||||
Timeout is 60s to allow for live device round-trips (event downloads can
|
||||
take 30-45s for a full event list).
|
||||
"""
|
||||
# Build target URL — SFM endpoints live at root, not /api/
|
||||
target_url = f"{SFM_BASE_URL}/{path}"
|
||||
|
||||
# Forward query params
|
||||
query_params = dict(request.query_params)
|
||||
|
||||
# Read body for mutation requests
|
||||
body = None
|
||||
if request.method in ["POST", "PUT", "PATCH"]:
|
||||
try:
|
||||
body = await request.body()
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read request body: {e}")
|
||||
body = None
|
||||
|
||||
# Strip hop-by-hop headers
|
||||
headers = dict(request.headers)
|
||||
headers_to_exclude = ["host", "content-length", "transfer-encoding", "connection"]
|
||||
proxy_headers = {k: v for k, v in headers.items() if k.lower() not in headers_to_exclude}
|
||||
|
||||
logger.info(f"Proxying {request.method} {path} → SFM: {target_url}")
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=60.0) as client:
|
||||
response = await client.request(
|
||||
method=request.method,
|
||||
url=target_url,
|
||||
params=query_params,
|
||||
headers=proxy_headers,
|
||||
content=body
|
||||
)
|
||||
return Response(
|
||||
content=response.content,
|
||||
status_code=response.status_code,
|
||||
headers=dict(response.headers),
|
||||
media_type=response.headers.get("content-type")
|
||||
)
|
||||
|
||||
except httpx.ConnectError:
|
||||
logger.error(f"Failed to connect to SFM backend at {SFM_BASE_URL}")
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail=f"SFM backend service unavailable. Is SFM running on {SFM_BASE_URL}?"
|
||||
)
|
||||
except httpx.TimeoutException:
|
||||
logger.error(f"Timeout connecting to SFM backend at {SFM_BASE_URL}")
|
||||
raise HTTPException(
|
||||
status_code=504,
|
||||
detail="SFM backend timeout"
|
||||
)
|
||||
except Exception as e:
|
||||
logger.error(f"Error proxying to SFM: {e}")
|
||||
raise HTTPException(
|
||||
status_code=500,
|
||||
detail=f"Failed to proxy request to SFM: {str(e)}"
|
||||
)
|
||||
@@ -5,7 +5,6 @@ Provides API endpoints for the Sound Level Meters dashboard page.
|
||||
"""
|
||||
|
||||
from fastapi import APIRouter, Request, Depends, Query
|
||||
from fastapi.templating import Jinja2Templates
|
||||
from fastapi.responses import HTMLResponse
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import func
|
||||
@@ -18,11 +17,11 @@ import os
|
||||
from backend.database import get_db
|
||||
from backend.models import RosterUnit
|
||||
from backend.routers.roster_edit import sync_slm_to_slmm_cache
|
||||
from backend.templates_config import templates
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/api/slm-dashboard", tags=["slm-dashboard"])
|
||||
templates = Jinja2Templates(directory="templates")
|
||||
|
||||
# SLMM backend URL - configurable via environment variable
|
||||
SLMM_BASE_URL = os.getenv("SLMM_BASE_URL", "http://localhost:8100")
|
||||
@@ -35,7 +34,7 @@ async def get_slm_stats(request: Request, db: Session = Depends(get_db)):
|
||||
Returns HTML partial with stat cards.
|
||||
"""
|
||||
# Query all SLMs
|
||||
all_slms = db.query(RosterUnit).filter_by(device_type="sound_level_meter").all()
|
||||
all_slms = db.query(RosterUnit).filter_by(device_type="slm").all()
|
||||
|
||||
# Count deployed vs benched
|
||||
deployed_count = sum(1 for slm in all_slms if slm.deployed and not slm.retired)
|
||||
@@ -69,7 +68,7 @@ async def get_slm_units(
|
||||
Get list of SLM units for the sidebar.
|
||||
Returns HTML partial with unit cards.
|
||||
"""
|
||||
query = db.query(RosterUnit).filter_by(device_type="sound_level_meter")
|
||||
query = db.query(RosterUnit).filter_by(device_type="slm")
|
||||
|
||||
# Filter by project if provided
|
||||
if project:
|
||||
@@ -129,7 +128,7 @@ async def get_live_view(request: Request, unit_id: str, db: Session = Depends(ge
|
||||
Returns HTML partial with live metrics and chart.
|
||||
"""
|
||||
# Get unit from database
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id, device_type="sound_level_meter").first()
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id, device_type="slm").first()
|
||||
|
||||
if not unit:
|
||||
return templates.TemplateResponse("partials/slm_live_view_error.html", {
|
||||
@@ -168,23 +167,7 @@ async def get_live_view(request: Request, unit_id: str, db: Session = Depends(ge
|
||||
measurement_state = state_data.get("measurement_state", "Unknown")
|
||||
is_measuring = state_data.get("is_measuring", False)
|
||||
|
||||
# If measuring, sync start time from FTP to database (fixes wrong timestamps)
|
||||
if is_measuring:
|
||||
try:
|
||||
sync_response = await client.post(
|
||||
f"{SLMM_BASE_URL}/api/nl43/{unit_id}/sync-start-time",
|
||||
timeout=10.0
|
||||
)
|
||||
if sync_response.status_code == 200:
|
||||
sync_data = sync_response.json()
|
||||
logger.info(f"Synced start time for {unit_id}: {sync_data.get('message')}")
|
||||
else:
|
||||
logger.warning(f"Failed to sync start time for {unit_id}: {sync_response.status_code}")
|
||||
except Exception as e:
|
||||
# Don't fail the whole request if sync fails
|
||||
logger.warning(f"Could not sync start time for {unit_id}: {e}")
|
||||
|
||||
# Get live status (now with corrected start time)
|
||||
# Get live status (measurement_start_time is already stored in SLMM database)
|
||||
status_response = await client.get(
|
||||
f"{SLMM_BASE_URL}/api/nl43/{unit_id}/live"
|
||||
)
|
||||
@@ -242,7 +225,7 @@ async def get_slm_config(request: Request, unit_id: str, db: Session = Depends(g
|
||||
Get configuration form for a specific SLM unit.
|
||||
Returns HTML partial with configuration form.
|
||||
"""
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id, device_type="sound_level_meter").first()
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id, device_type="slm").first()
|
||||
|
||||
if not unit:
|
||||
return HTMLResponse(
|
||||
@@ -262,7 +245,7 @@ async def save_slm_config(request: Request, unit_id: str, db: Session = Depends(
|
||||
Save SLM configuration.
|
||||
Updates unit parameters in the database.
|
||||
"""
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id, device_type="sound_level_meter").first()
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id, device_type="slm").first()
|
||||
|
||||
if not unit:
|
||||
return {"status": "error", "detail": f"Unit {unit_id} not found"}
|
||||
|
||||
@@ -6,7 +6,6 @@ Provides endpoints for SLM dashboard cards, detail pages, and real-time data.
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from fastapi.responses import HTMLResponse
|
||||
from fastapi.templating import Jinja2Templates
|
||||
from sqlalchemy.orm import Session
|
||||
from datetime import datetime
|
||||
import httpx
|
||||
@@ -15,11 +14,11 @@ import os
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import RosterUnit
|
||||
from backend.templates_config import templates
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter(prefix="/slm", tags=["slm-ui"])
|
||||
templates = Jinja2Templates(directory="templates")
|
||||
|
||||
SLMM_BASE_URL = os.getenv("SLMM_BASE_URL", "http://172.19.0.1:8100")
|
||||
|
||||
@@ -30,7 +29,7 @@ async def slm_detail_page(request: Request, unit_id: str, db: Session = Depends(
|
||||
|
||||
# Get roster unit
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id).first()
|
||||
if not unit or unit.device_type != "sound_level_meter":
|
||||
if not unit or unit.device_type != "slm":
|
||||
raise HTTPException(status_code=404, detail="Sound level meter not found")
|
||||
|
||||
return templates.TemplateResponse("slm_detail.html", {
|
||||
@@ -46,7 +45,7 @@ async def get_slm_summary(unit_id: str, db: Session = Depends(get_db)):
|
||||
|
||||
# Get roster unit
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id).first()
|
||||
if not unit or unit.device_type != "sound_level_meter":
|
||||
if not unit or unit.device_type != "slm":
|
||||
raise HTTPException(status_code=404, detail="Sound level meter not found")
|
||||
|
||||
# Try to get live status from SLMM
|
||||
@@ -61,7 +60,7 @@ async def get_slm_summary(unit_id: str, db: Session = Depends(get_db)):
|
||||
|
||||
return {
|
||||
"unit_id": unit_id,
|
||||
"device_type": "sound_level_meter",
|
||||
"device_type": "slm",
|
||||
"deployed": unit.deployed,
|
||||
"model": unit.slm_model or "NL-43",
|
||||
"location": unit.address or unit.location,
|
||||
@@ -89,7 +88,7 @@ async def slm_controls_partial(request: Request, unit_id: str, db: Session = Dep
|
||||
"""Render SLM control panel partial."""
|
||||
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id).first()
|
||||
if not unit or unit.device_type != "sound_level_meter":
|
||||
if not unit or unit.device_type != "slm":
|
||||
raise HTTPException(status_code=404, detail="Sound level meter not found")
|
||||
|
||||
# Get current status from SLMM
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
from fastapi import APIRouter, Depends, HTTPException
|
||||
from fastapi import APIRouter, Depends, HTTPException, Query
|
||||
from sqlalchemy.orm import Session
|
||||
from datetime import datetime
|
||||
from typing import Dict, Any
|
||||
from typing import Dict, Any, Optional
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.services.snapshot import emit_status_snapshot
|
||||
@@ -72,3 +72,101 @@ def get_unit_by_id(unit_id: str, db: Session = Depends(get_db)):
|
||||
"slm_serial_number": unit.slm_serial_number,
|
||||
"deployed_with_modem_id": unit.deployed_with_modem_id
|
||||
}
|
||||
|
||||
|
||||
@router.get("/units/{unit_id}/events")
|
||||
async def get_unit_events(
|
||||
unit_id: str,
|
||||
bucket: str = Query("all", regex="^(all|attributed|unattributed)$"),
|
||||
from_dt: Optional[datetime] = Query(None),
|
||||
to_dt: Optional[datetime] = Query(None),
|
||||
false_trigger: Optional[bool] = Query(None),
|
||||
limit: int = Query(500, ge=1, le=5000),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Return SFM events for a single unit, annotated with assignment attribution.
|
||||
|
||||
Each event includes an `attribution` object pointing at the project/location
|
||||
it falls into (or null if outside every assignment window). Unattributed
|
||||
events also carry a `nearest_assignment` field with `delta_days` so the
|
||||
operator can see how far off the nearest assignment is — useful for
|
||||
deciding whether to backdate the assignment to absorb the event.
|
||||
|
||||
Bucket filter:
|
||||
- all (default): every event
|
||||
- attributed: only events inside an assignment window
|
||||
- unattributed: only orphan events (the diagnostic bucket)
|
||||
|
||||
Non-seismograph units return an empty events list. The route does not
|
||||
404 for SLMs/modems so the unit detail page can render the section
|
||||
conditionally without depending on the response shape.
|
||||
"""
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id).first()
|
||||
if not unit:
|
||||
raise HTTPException(status_code=404, detail=f"Unit {unit_id} not found")
|
||||
|
||||
if unit.device_type != "seismograph":
|
||||
return {
|
||||
"events": [],
|
||||
"count": 0,
|
||||
"stats": {
|
||||
"event_count": 0,
|
||||
"unattributed_count": 0,
|
||||
"peak_pvs": None,
|
||||
"peak_pvs_at": None,
|
||||
"peak_pvs_serial": None,
|
||||
"last_event": None,
|
||||
"false_trigger_count": 0,
|
||||
},
|
||||
"assignments_total": 0,
|
||||
"device_type": unit.device_type,
|
||||
}
|
||||
|
||||
from backend.services.sfm_events import events_for_unit
|
||||
|
||||
result = await events_for_unit(
|
||||
db,
|
||||
unit_id,
|
||||
bucket=bucket,
|
||||
from_dt=from_dt,
|
||||
to_dt=to_dt,
|
||||
false_trigger=false_trigger,
|
||||
limit=limit,
|
||||
)
|
||||
result["device_type"] = unit.device_type
|
||||
return result
|
||||
|
||||
|
||||
@router.get("/units/{unit_id}/deployment_timeline")
|
||||
async def get_unit_deployment_timeline(
|
||||
unit_id: str,
|
||||
include_events: bool = Query(True),
|
||||
db: Session = Depends(get_db),
|
||||
):
|
||||
"""
|
||||
Return a chronological deployment timeline for a unit.
|
||||
|
||||
Merges three sources:
|
||||
1. unit_assignments — authoritative project/location deployments
|
||||
2. unit_history — state changes (calibration, retirement, etc.)
|
||||
3. SFM events — per-assignment overlay (count + peak PVS + last event)
|
||||
|
||||
Replaces the legacy /api/deployments/{unit_id} (which read the
|
||||
deprecated `deployment_records` table) and the
|
||||
/api/roster/history/{unit_id} timeline endpoint, unifying them into
|
||||
a single derived view.
|
||||
|
||||
Gaps >= 1 day between consecutive assignments are surfaced as
|
||||
synthetic "gap" entries.
|
||||
|
||||
Pass include_events=false to skip the SFM event overlay (saves N
|
||||
HTTP calls; useful for fast text-only history dumps).
|
||||
"""
|
||||
from backend.services.deployment_timeline import deployment_timeline_for_unit
|
||||
|
||||
return await deployment_timeline_for_unit(
|
||||
db,
|
||||
unit_id,
|
||||
include_event_overlay=include_events,
|
||||
)
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
"""
|
||||
Watcher Manager — admin API for series3-watcher and thor-watcher agents.
|
||||
|
||||
Endpoints:
|
||||
GET /api/admin/watchers — list all watcher agents
|
||||
GET /api/admin/watchers/{agent_id} — get single agent detail
|
||||
POST /api/admin/watchers/{agent_id}/trigger-update — flag agent for update
|
||||
POST /api/admin/watchers/{agent_id}/clear-update — clear update flag
|
||||
GET /api/admin/watchers/{agent_id}/update-check — polled by watcher on heartbeat
|
||||
|
||||
Page:
|
||||
GET /admin/watchers — HTML admin page
|
||||
"""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
from fastapi.responses import HTMLResponse
|
||||
from pydantic import BaseModel
|
||||
from sqlalchemy.orm import Session
|
||||
from typing import Optional
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import WatcherAgent
|
||||
from backend.templates_config import templates
|
||||
|
||||
router = APIRouter(tags=["admin"])
|
||||
|
||||
|
||||
# ── helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def _agent_to_dict(agent: WatcherAgent) -> dict:
|
||||
last_seen = agent.last_seen
|
||||
if last_seen:
|
||||
now_utc = datetime.utcnow()
|
||||
age_minutes = int((now_utc - last_seen).total_seconds() // 60)
|
||||
if age_minutes > 60:
|
||||
status = "missing"
|
||||
else:
|
||||
status = "ok"
|
||||
else:
|
||||
age_minutes = None
|
||||
status = "missing"
|
||||
|
||||
return {
|
||||
"id": agent.id,
|
||||
"source_type": agent.source_type,
|
||||
"version": agent.version,
|
||||
"last_seen": last_seen.isoformat() if last_seen else None,
|
||||
"age_minutes": age_minutes,
|
||||
"status": status,
|
||||
"ip_address": agent.ip_address,
|
||||
"log_tail": agent.log_tail,
|
||||
"update_pending": bool(agent.update_pending),
|
||||
"update_version": agent.update_version,
|
||||
}
|
||||
|
||||
|
||||
# ── API routes ────────────────────────────────────────────────────────────────
|
||||
|
||||
@router.get("/api/admin/watchers")
|
||||
def list_watchers(db: Session = Depends(get_db)):
|
||||
agents = db.query(WatcherAgent).order_by(WatcherAgent.last_seen.desc()).all()
|
||||
return [_agent_to_dict(a) for a in agents]
|
||||
|
||||
|
||||
@router.get("/api/admin/watchers/{agent_id}")
|
||||
def get_watcher(agent_id: str, db: Session = Depends(get_db)):
|
||||
agent = db.query(WatcherAgent).filter(WatcherAgent.id == agent_id).first()
|
||||
if not agent:
|
||||
raise HTTPException(status_code=404, detail="Watcher agent not found")
|
||||
return _agent_to_dict(agent)
|
||||
|
||||
|
||||
class TriggerUpdateRequest(BaseModel):
|
||||
version: Optional[str] = None # target version label (informational)
|
||||
|
||||
|
||||
@router.post("/api/admin/watchers/{agent_id}/trigger-update")
|
||||
def trigger_update(agent_id: str, body: TriggerUpdateRequest, db: Session = Depends(get_db)):
|
||||
agent = db.query(WatcherAgent).filter(WatcherAgent.id == agent_id).first()
|
||||
if not agent:
|
||||
raise HTTPException(status_code=404, detail="Watcher agent not found")
|
||||
agent.update_pending = True
|
||||
agent.update_version = body.version
|
||||
db.commit()
|
||||
return {"ok": True, "agent_id": agent_id, "update_pending": True}
|
||||
|
||||
|
||||
@router.post("/api/admin/watchers/{agent_id}/clear-update")
|
||||
def clear_update(agent_id: str, db: Session = Depends(get_db)):
|
||||
agent = db.query(WatcherAgent).filter(WatcherAgent.id == agent_id).first()
|
||||
if not agent:
|
||||
raise HTTPException(status_code=404, detail="Watcher agent not found")
|
||||
agent.update_pending = False
|
||||
agent.update_version = None
|
||||
db.commit()
|
||||
return {"ok": True, "agent_id": agent_id, "update_pending": False}
|
||||
|
||||
|
||||
@router.get("/api/admin/watchers/{agent_id}/update-check")
|
||||
def update_check(agent_id: str, db: Session = Depends(get_db)):
|
||||
"""
|
||||
Polled by watcher agents on each heartbeat cycle.
|
||||
Returns update_available=True when an update has been triggered via the UI.
|
||||
Automatically clears the flag after the watcher acknowledges it.
|
||||
"""
|
||||
agent = db.query(WatcherAgent).filter(WatcherAgent.id == agent_id).first()
|
||||
if not agent:
|
||||
return {"update_available": False}
|
||||
|
||||
pending = bool(agent.update_pending)
|
||||
|
||||
if pending:
|
||||
# Clear the flag — the watcher will now self-update
|
||||
agent.update_pending = False
|
||||
db.commit()
|
||||
|
||||
return {
|
||||
"update_available": pending,
|
||||
"version": agent.update_version,
|
||||
}
|
||||
|
||||
|
||||
# ── HTML page ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@router.get("/admin/watchers", response_class=HTMLResponse)
|
||||
def admin_watchers_page(request: Request, db: Session = Depends(get_db)):
|
||||
agents = db.query(WatcherAgent).order_by(WatcherAgent.last_seen.desc()).all()
|
||||
agents_data = [_agent_to_dict(a) for a in agents]
|
||||
return templates.TemplateResponse("admin_watchers.html", {
|
||||
"request": request,
|
||||
"agents": agents_data,
|
||||
})
|
||||
@@ -5,7 +5,7 @@ from datetime import datetime
|
||||
from typing import Optional, List
|
||||
|
||||
from backend.database import get_db
|
||||
from backend.models import Emitter
|
||||
from backend.models import Emitter, WatcherAgent
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@@ -107,6 +107,35 @@ def get_fleet_status(db: Session = Depends(get_db)):
|
||||
emitters = db.query(Emitter).all()
|
||||
return emitters
|
||||
|
||||
# ── Watcher agent upsert helper ───────────────────────────────────────────────
|
||||
|
||||
def _upsert_watcher_agent(db: Session, source_id: str, source_type: str,
|
||||
version: str, ip_address: str, log_tail: str,
|
||||
status: str) -> None:
|
||||
"""Create or update the WatcherAgent row for a given source_id."""
|
||||
agent = db.query(WatcherAgent).filter(WatcherAgent.id == source_id).first()
|
||||
if agent:
|
||||
agent.source_type = source_type
|
||||
agent.version = version
|
||||
agent.last_seen = datetime.utcnow()
|
||||
agent.status = status
|
||||
if ip_address:
|
||||
agent.ip_address = ip_address
|
||||
if log_tail is not None:
|
||||
agent.log_tail = log_tail
|
||||
else:
|
||||
agent = WatcherAgent(
|
||||
id=source_id,
|
||||
source_type=source_type,
|
||||
version=version,
|
||||
last_seen=datetime.utcnow(),
|
||||
status=status,
|
||||
ip_address=ip_address,
|
||||
log_tail=log_tail,
|
||||
)
|
||||
db.add(agent)
|
||||
|
||||
|
||||
# series3v1.1 Standardized Heartbeat Schema (multi-unit)
|
||||
from fastapi import Request
|
||||
|
||||
@@ -120,6 +149,11 @@ async def series3_heartbeat(request: Request, db: Session = Depends(get_db)):
|
||||
|
||||
source = payload.get("source_id")
|
||||
units = payload.get("units", [])
|
||||
version = payload.get("version")
|
||||
log_tail = payload.get("log_tail") # list of strings or None
|
||||
import json as _json
|
||||
log_tail_str = _json.dumps(log_tail) if log_tail is not None else None
|
||||
client_ip = request.client.host if request.client else None
|
||||
|
||||
print("\n=== Series 3 Heartbeat ===")
|
||||
print("Source:", source)
|
||||
@@ -182,13 +216,27 @@ async def series3_heartbeat(request: Request, db: Session = Depends(get_db)):
|
||||
|
||||
results.append({"unit": uid, "status": status})
|
||||
|
||||
if source:
|
||||
_upsert_watcher_agent(db, source, "series3_watcher", version,
|
||||
client_ip, log_tail_str, "ok")
|
||||
|
||||
db.commit()
|
||||
|
||||
# Check if an update has been triggered for this agent
|
||||
update_available = False
|
||||
if source:
|
||||
agent = db.query(WatcherAgent).filter(WatcherAgent.id == source).first()
|
||||
if agent and agent.update_pending:
|
||||
update_available = True
|
||||
agent.update_pending = False
|
||||
db.commit()
|
||||
|
||||
return {
|
||||
"message": "Heartbeat processed",
|
||||
"source": source,
|
||||
"units_processed": len(results),
|
||||
"results": results
|
||||
"results": results,
|
||||
"update_available": update_available,
|
||||
}
|
||||
|
||||
|
||||
@@ -219,8 +267,14 @@ async def series4_heartbeat(request: Request, db: Session = Depends(get_db)):
|
||||
"""
|
||||
payload = await request.json()
|
||||
|
||||
source = payload.get("source", "series4_emitter")
|
||||
# Accept source_id (new standard field) with fallback to legacy "source" key
|
||||
source = payload.get("source_id") or payload.get("source", "series4_emitter")
|
||||
units = payload.get("units", [])
|
||||
version = payload.get("version")
|
||||
log_tail = payload.get("log_tail")
|
||||
import json as _json
|
||||
log_tail_str = _json.dumps(log_tail) if log_tail is not None else None
|
||||
client_ip = request.client.host if request.client else None
|
||||
|
||||
print("\n=== Series 4 Heartbeat ===")
|
||||
print("Source:", source)
|
||||
@@ -276,11 +330,25 @@ async def series4_heartbeat(request: Request, db: Session = Depends(get_db)):
|
||||
|
||||
results.append({"unit": uid, "status": status})
|
||||
|
||||
if source:
|
||||
_upsert_watcher_agent(db, source, "series4_watcher", version,
|
||||
client_ip, log_tail_str, "ok")
|
||||
|
||||
db.commit()
|
||||
|
||||
# Check if an update has been triggered for this agent
|
||||
update_available = False
|
||||
if source:
|
||||
agent = db.query(WatcherAgent).filter(WatcherAgent.id == source).first()
|
||||
if agent and agent.update_pending:
|
||||
update_available = True
|
||||
agent.update_pending = False
|
||||
db.commit()
|
||||
|
||||
return {
|
||||
"message": "Heartbeat processed",
|
||||
"source": source,
|
||||
"units_processed": len(results),
|
||||
"results": results
|
||||
"results": results,
|
||||
"update_available": update_available,
|
||||
}
|
||||
|
||||
@@ -0,0 +1,462 @@
|
||||
"""
|
||||
Alert Service
|
||||
|
||||
Manages in-app alerts for device status changes and system events.
|
||||
Provides foundation for future notification channels (email, webhook).
|
||||
"""
|
||||
|
||||
import json
|
||||
import uuid
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional, List, Dict, Any
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import and_, or_
|
||||
|
||||
from backend.models import Alert, RosterUnit
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AlertService:
|
||||
"""
|
||||
Service for managing alerts.
|
||||
|
||||
Handles alert lifecycle:
|
||||
- Create alerts from various triggers
|
||||
- Query active alerts
|
||||
- Acknowledge/resolve/dismiss alerts
|
||||
- (Future) Dispatch to notification channels
|
||||
"""
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
|
||||
def create_alert(
|
||||
self,
|
||||
alert_type: str,
|
||||
title: str,
|
||||
message: str = None,
|
||||
severity: str = "warning",
|
||||
unit_id: str = None,
|
||||
project_id: str = None,
|
||||
location_id: str = None,
|
||||
schedule_id: str = None,
|
||||
metadata: dict = None,
|
||||
expires_hours: int = 24,
|
||||
) -> Alert:
|
||||
"""
|
||||
Create a new alert.
|
||||
|
||||
Args:
|
||||
alert_type: Type of alert (device_offline, device_online, schedule_failed)
|
||||
title: Short alert title
|
||||
message: Detailed description
|
||||
severity: info, warning, or critical
|
||||
unit_id: Related unit ID (optional)
|
||||
project_id: Related project ID (optional)
|
||||
location_id: Related location ID (optional)
|
||||
schedule_id: Related schedule ID (optional)
|
||||
metadata: Additional JSON data
|
||||
expires_hours: Hours until auto-expiry (default 24)
|
||||
|
||||
Returns:
|
||||
Created Alert instance
|
||||
"""
|
||||
alert = Alert(
|
||||
id=str(uuid.uuid4()),
|
||||
alert_type=alert_type,
|
||||
title=title,
|
||||
message=message,
|
||||
severity=severity,
|
||||
unit_id=unit_id,
|
||||
project_id=project_id,
|
||||
location_id=location_id,
|
||||
schedule_id=schedule_id,
|
||||
alert_metadata=json.dumps(metadata) if metadata else None,
|
||||
status="active",
|
||||
expires_at=datetime.utcnow() + timedelta(hours=expires_hours),
|
||||
)
|
||||
|
||||
self.db.add(alert)
|
||||
self.db.commit()
|
||||
self.db.refresh(alert)
|
||||
|
||||
logger.info(f"Created alert: {alert.title} ({alert.alert_type})")
|
||||
return alert
|
||||
|
||||
def create_device_offline_alert(
|
||||
self,
|
||||
unit_id: str,
|
||||
consecutive_failures: int = 0,
|
||||
last_error: str = None,
|
||||
) -> Optional[Alert]:
|
||||
"""
|
||||
Create alert when device becomes unreachable.
|
||||
|
||||
Only creates if no active offline alert exists for this device.
|
||||
|
||||
Args:
|
||||
unit_id: The unit that went offline
|
||||
consecutive_failures: Number of consecutive poll failures
|
||||
last_error: Last error message from polling
|
||||
|
||||
Returns:
|
||||
Created Alert or None if alert already exists
|
||||
"""
|
||||
# Check if active offline alert already exists
|
||||
existing = self.db.query(Alert).filter(
|
||||
and_(
|
||||
Alert.unit_id == unit_id,
|
||||
Alert.alert_type == "device_offline",
|
||||
Alert.status == "active",
|
||||
)
|
||||
).first()
|
||||
|
||||
if existing:
|
||||
logger.debug(f"Offline alert already exists for {unit_id}")
|
||||
return None
|
||||
|
||||
# Get unit info for title
|
||||
unit = self.db.query(RosterUnit).filter_by(id=unit_id).first()
|
||||
unit_name = unit.id if unit else unit_id
|
||||
|
||||
# Determine severity based on failure count
|
||||
severity = "critical" if consecutive_failures >= 5 else "warning"
|
||||
|
||||
return self.create_alert(
|
||||
alert_type="device_offline",
|
||||
title=f"{unit_name} is offline",
|
||||
message=f"Device has been unreachable after {consecutive_failures} failed connection attempts."
|
||||
+ (f" Last error: {last_error}" if last_error else ""),
|
||||
severity=severity,
|
||||
unit_id=unit_id,
|
||||
metadata={
|
||||
"consecutive_failures": consecutive_failures,
|
||||
"last_error": last_error,
|
||||
},
|
||||
expires_hours=48, # Offline alerts stay longer
|
||||
)
|
||||
|
||||
def resolve_device_offline_alert(self, unit_id: str) -> Optional[Alert]:
|
||||
"""
|
||||
Auto-resolve offline alert when device comes back online.
|
||||
|
||||
Also creates an "device_online" info alert to notify user.
|
||||
|
||||
Args:
|
||||
unit_id: The unit that came back online
|
||||
|
||||
Returns:
|
||||
The resolved Alert or None if no alert existed
|
||||
"""
|
||||
# Find active offline alert
|
||||
alert = self.db.query(Alert).filter(
|
||||
and_(
|
||||
Alert.unit_id == unit_id,
|
||||
Alert.alert_type == "device_offline",
|
||||
Alert.status == "active",
|
||||
)
|
||||
).first()
|
||||
|
||||
if not alert:
|
||||
return None
|
||||
|
||||
# Resolve the offline alert
|
||||
alert.status = "resolved"
|
||||
alert.resolved_at = datetime.utcnow()
|
||||
self.db.commit()
|
||||
|
||||
logger.info(f"Resolved offline alert for {unit_id}")
|
||||
|
||||
# Create online notification
|
||||
unit = self.db.query(RosterUnit).filter_by(id=unit_id).first()
|
||||
unit_name = unit.id if unit else unit_id
|
||||
|
||||
self.create_alert(
|
||||
alert_type="device_online",
|
||||
title=f"{unit_name} is back online",
|
||||
message="Device connection has been restored.",
|
||||
severity="info",
|
||||
unit_id=unit_id,
|
||||
expires_hours=6, # Info alerts expire quickly
|
||||
)
|
||||
|
||||
return alert
|
||||
|
||||
def create_schedule_failed_alert(
|
||||
self,
|
||||
schedule_id: str,
|
||||
action_type: str,
|
||||
unit_id: str = None,
|
||||
error_message: str = None,
|
||||
project_id: str = None,
|
||||
location_id: str = None,
|
||||
) -> Alert:
|
||||
"""
|
||||
Create alert when a scheduled action fails.
|
||||
|
||||
Args:
|
||||
schedule_id: The ScheduledAction or RecurringSchedule ID
|
||||
action_type: start, stop, download, cycle
|
||||
unit_id: Related unit
|
||||
error_message: Error from execution
|
||||
project_id: Related project
|
||||
location_id: Related location
|
||||
|
||||
Returns:
|
||||
Created Alert
|
||||
"""
|
||||
return self.create_alert(
|
||||
alert_type="schedule_failed",
|
||||
title=f"Scheduled {action_type} failed",
|
||||
message=error_message or f"The scheduled {action_type} action did not complete successfully.",
|
||||
severity="warning",
|
||||
unit_id=unit_id,
|
||||
project_id=project_id,
|
||||
location_id=location_id,
|
||||
schedule_id=schedule_id,
|
||||
metadata={"action_type": action_type},
|
||||
expires_hours=24,
|
||||
)
|
||||
|
||||
def create_schedule_completed_alert(
|
||||
self,
|
||||
schedule_id: str,
|
||||
action_type: str,
|
||||
unit_id: str = None,
|
||||
project_id: str = None,
|
||||
location_id: str = None,
|
||||
metadata: dict = None,
|
||||
) -> Alert:
|
||||
"""
|
||||
Create alert when a scheduled action completes successfully.
|
||||
|
||||
Args:
|
||||
schedule_id: The ScheduledAction ID
|
||||
action_type: start, stop, download, cycle
|
||||
unit_id: Related unit
|
||||
project_id: Related project
|
||||
location_id: Related location
|
||||
metadata: Additional info (e.g., downloaded folder, index numbers)
|
||||
|
||||
Returns:
|
||||
Created Alert
|
||||
"""
|
||||
# Build descriptive message based on action type and metadata
|
||||
if action_type == "stop" and metadata:
|
||||
download_folder = metadata.get("downloaded_folder")
|
||||
download_success = metadata.get("download_success", False)
|
||||
if download_success and download_folder:
|
||||
message = f"Measurement stopped and data downloaded ({download_folder})"
|
||||
elif download_success is False and metadata.get("download_attempted"):
|
||||
message = "Measurement stopped but download failed"
|
||||
else:
|
||||
message = "Measurement stopped successfully"
|
||||
elif action_type == "start" and metadata:
|
||||
new_index = metadata.get("new_index")
|
||||
if new_index is not None:
|
||||
message = f"Measurement started (index {new_index:04d})"
|
||||
else:
|
||||
message = "Measurement started successfully"
|
||||
else:
|
||||
message = f"Scheduled {action_type} completed successfully"
|
||||
|
||||
return self.create_alert(
|
||||
alert_type="schedule_completed",
|
||||
title=f"Scheduled {action_type} completed",
|
||||
message=message,
|
||||
severity="info",
|
||||
unit_id=unit_id,
|
||||
project_id=project_id,
|
||||
location_id=location_id,
|
||||
schedule_id=schedule_id,
|
||||
metadata={"action_type": action_type, **(metadata or {})},
|
||||
expires_hours=12, # Info alerts expire quickly
|
||||
)
|
||||
|
||||
def get_active_alerts(
|
||||
self,
|
||||
project_id: str = None,
|
||||
unit_id: str = None,
|
||||
alert_type: str = None,
|
||||
min_severity: str = None,
|
||||
limit: int = 50,
|
||||
) -> List[Alert]:
|
||||
"""
|
||||
Query active alerts with optional filters.
|
||||
|
||||
Args:
|
||||
project_id: Filter by project
|
||||
unit_id: Filter by unit
|
||||
alert_type: Filter by alert type
|
||||
min_severity: Minimum severity (info, warning, critical)
|
||||
limit: Maximum results
|
||||
|
||||
Returns:
|
||||
List of matching alerts
|
||||
"""
|
||||
query = self.db.query(Alert).filter(Alert.status == "active")
|
||||
|
||||
if project_id:
|
||||
query = query.filter(Alert.project_id == project_id)
|
||||
|
||||
if unit_id:
|
||||
query = query.filter(Alert.unit_id == unit_id)
|
||||
|
||||
if alert_type:
|
||||
query = query.filter(Alert.alert_type == alert_type)
|
||||
|
||||
if min_severity:
|
||||
# Map severity to numeric for comparison
|
||||
severity_levels = {"info": 1, "warning": 2, "critical": 3}
|
||||
min_level = severity_levels.get(min_severity, 1)
|
||||
|
||||
if min_level == 2:
|
||||
query = query.filter(Alert.severity.in_(["warning", "critical"]))
|
||||
elif min_level == 3:
|
||||
query = query.filter(Alert.severity == "critical")
|
||||
|
||||
return query.order_by(Alert.created_at.desc()).limit(limit).all()
|
||||
|
||||
def get_all_alerts(
|
||||
self,
|
||||
status: str = None,
|
||||
project_id: str = None,
|
||||
unit_id: str = None,
|
||||
alert_type: str = None,
|
||||
limit: int = 50,
|
||||
offset: int = 0,
|
||||
) -> List[Alert]:
|
||||
"""
|
||||
Query all alerts with optional filters (includes non-active).
|
||||
|
||||
Args:
|
||||
status: Filter by status (active, acknowledged, resolved, dismissed)
|
||||
project_id: Filter by project
|
||||
unit_id: Filter by unit
|
||||
alert_type: Filter by alert type
|
||||
limit: Maximum results
|
||||
offset: Pagination offset
|
||||
|
||||
Returns:
|
||||
List of matching alerts
|
||||
"""
|
||||
query = self.db.query(Alert)
|
||||
|
||||
if status:
|
||||
query = query.filter(Alert.status == status)
|
||||
|
||||
if project_id:
|
||||
query = query.filter(Alert.project_id == project_id)
|
||||
|
||||
if unit_id:
|
||||
query = query.filter(Alert.unit_id == unit_id)
|
||||
|
||||
if alert_type:
|
||||
query = query.filter(Alert.alert_type == alert_type)
|
||||
|
||||
return (
|
||||
query.order_by(Alert.created_at.desc())
|
||||
.offset(offset)
|
||||
.limit(limit)
|
||||
.all()
|
||||
)
|
||||
|
||||
def get_active_alert_count(self) -> int:
|
||||
"""Get count of active alerts for badge display."""
|
||||
return self.db.query(Alert).filter(Alert.status == "active").count()
|
||||
|
||||
def acknowledge_alert(self, alert_id: str) -> Optional[Alert]:
|
||||
"""
|
||||
Mark alert as acknowledged.
|
||||
|
||||
Args:
|
||||
alert_id: Alert to acknowledge
|
||||
|
||||
Returns:
|
||||
Updated Alert or None if not found
|
||||
"""
|
||||
alert = self.db.query(Alert).filter_by(id=alert_id).first()
|
||||
if not alert:
|
||||
return None
|
||||
|
||||
alert.status = "acknowledged"
|
||||
alert.acknowledged_at = datetime.utcnow()
|
||||
self.db.commit()
|
||||
|
||||
logger.info(f"Acknowledged alert: {alert.title}")
|
||||
return alert
|
||||
|
||||
def dismiss_alert(self, alert_id: str) -> Optional[Alert]:
|
||||
"""
|
||||
Dismiss alert (user chose to ignore).
|
||||
|
||||
Args:
|
||||
alert_id: Alert to dismiss
|
||||
|
||||
Returns:
|
||||
Updated Alert or None if not found
|
||||
"""
|
||||
alert = self.db.query(Alert).filter_by(id=alert_id).first()
|
||||
if not alert:
|
||||
return None
|
||||
|
||||
alert.status = "dismissed"
|
||||
self.db.commit()
|
||||
|
||||
logger.info(f"Dismissed alert: {alert.title}")
|
||||
return alert
|
||||
|
||||
def resolve_alert(self, alert_id: str) -> Optional[Alert]:
|
||||
"""
|
||||
Manually resolve an alert.
|
||||
|
||||
Args:
|
||||
alert_id: Alert to resolve
|
||||
|
||||
Returns:
|
||||
Updated Alert or None if not found
|
||||
"""
|
||||
alert = self.db.query(Alert).filter_by(id=alert_id).first()
|
||||
if not alert:
|
||||
return None
|
||||
|
||||
alert.status = "resolved"
|
||||
alert.resolved_at = datetime.utcnow()
|
||||
self.db.commit()
|
||||
|
||||
logger.info(f"Resolved alert: {alert.title}")
|
||||
return alert
|
||||
|
||||
def cleanup_expired_alerts(self) -> int:
|
||||
"""
|
||||
Remove alerts past their expiration time.
|
||||
|
||||
Returns:
|
||||
Number of alerts cleaned up
|
||||
"""
|
||||
now = datetime.utcnow()
|
||||
expired = self.db.query(Alert).filter(
|
||||
and_(
|
||||
Alert.expires_at.isnot(None),
|
||||
Alert.expires_at < now,
|
||||
Alert.status == "active",
|
||||
)
|
||||
).all()
|
||||
|
||||
count = len(expired)
|
||||
for alert in expired:
|
||||
alert.status = "dismissed"
|
||||
|
||||
if count > 0:
|
||||
self.db.commit()
|
||||
logger.info(f"Cleaned up {count} expired alerts")
|
||||
|
||||
return count
|
||||
|
||||
|
||||
def get_alert_service(db: Session) -> AlertService:
|
||||
"""Get an AlertService instance with the given database session."""
|
||||
return AlertService(db)
|
||||
@@ -0,0 +1,256 @@
|
||||
"""
|
||||
Deployment timeline service — replaces the legacy `deployment_records`-driven
|
||||
timeline on the seismograph unit detail page.
|
||||
|
||||
Architecture:
|
||||
- `unit_assignments` is the authoritative source for "where was this unit"
|
||||
(one row per location/time-window). Auto-written by the project location
|
||||
swap/assign/unassign/update workflows.
|
||||
- `unit_history` is the audit log for non-location state changes
|
||||
(calibration toggles, retirement, allocation, etc.).
|
||||
- SFM events are overlaid per assignment window to show "what was the unit
|
||||
actually doing during this deployment" (count + peak PVS + last-event).
|
||||
|
||||
Gaps between assignments are emitted as synthetic "gap" entries so operators
|
||||
can see when the unit was idle vs out-of-service.
|
||||
|
||||
`deployment_records` is being deprecated; this module does not read it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional
|
||||
|
||||
import httpx
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from backend.models import (
|
||||
UnitAssignment,
|
||||
UnitHistory,
|
||||
MonitoringLocation,
|
||||
Project,
|
||||
RosterUnit,
|
||||
)
|
||||
from backend.services.sfm_events import (
|
||||
SFM_BASE_URL,
|
||||
_fetch_events_for_serial,
|
||||
_iso_utc,
|
||||
)
|
||||
|
||||
log = logging.getLogger("backend.services.deployment_timeline")
|
||||
|
||||
# Don't emit synthetic gap entries shorter than this (seconds). Avoids visual
|
||||
# clutter from a sub-second handoff during a swap workflow.
|
||||
_MIN_GAP_SECONDS = 24 * 3600 # 1 day
|
||||
|
||||
# Per-call timeout when querying SFM for the event overlay.
|
||||
_SFM_TIMEOUT = 10.0
|
||||
_SFM_FETCH_CEILING = 5000
|
||||
|
||||
|
||||
# ── Public API ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def deployment_timeline_for_unit(
|
||||
db: Session,
|
||||
unit_id: str,
|
||||
*,
|
||||
include_event_overlay: bool = True,
|
||||
) -> dict:
|
||||
"""Build a chronological timeline for a unit.
|
||||
|
||||
Returns:
|
||||
{
|
||||
"unit_id": str,
|
||||
"device_type": str,
|
||||
"entries": [
|
||||
{
|
||||
"kind": "assignment" | "gap" | "state_change",
|
||||
"starts_at": ISO timestamp,
|
||||
"ends_at": ISO timestamp | None,
|
||||
"duration_days": float | None,
|
||||
# — assignment-only fields —
|
||||
"assignment_id": str,
|
||||
"location_id": str,
|
||||
"location_name": str,
|
||||
"project_id": str,
|
||||
"project_name": str,
|
||||
"is_active": bool,
|
||||
"event_overlay": {event_count, peak_pvs, peak_pvs_at, last_event}
|
||||
or None if include_event_overlay=False,
|
||||
"notes": str | None,
|
||||
# — gap-only fields —
|
||||
"context": "between assignments" | None,
|
||||
# — state_change-only fields —
|
||||
"change_type": str,
|
||||
"field_name": str | None,
|
||||
"old_value": str | None,
|
||||
"new_value": str | None,
|
||||
"source": str,
|
||||
"history_notes": str | None,
|
||||
},
|
||||
... # newest first
|
||||
],
|
||||
}
|
||||
"""
|
||||
unit = db.query(RosterUnit).filter_by(id=unit_id).first()
|
||||
if not unit:
|
||||
return {"unit_id": unit_id, "device_type": None, "entries": []}
|
||||
|
||||
# 1. Load assignments + their location/project lookups in bulk.
|
||||
assignments = (
|
||||
db.query(UnitAssignment)
|
||||
.filter(UnitAssignment.unit_id == unit_id)
|
||||
.order_by(UnitAssignment.assigned_at.asc())
|
||||
.all()
|
||||
)
|
||||
|
||||
loc_ids = {a.location_id for a in assignments}
|
||||
proj_ids = {a.project_id for a in assignments}
|
||||
loc_map = {
|
||||
l.id: l for l in db.query(MonitoringLocation).filter(
|
||||
MonitoringLocation.id.in_(loc_ids)
|
||||
).all()
|
||||
} if loc_ids else {}
|
||||
proj_map = {
|
||||
p.id: p for p in db.query(Project).filter(
|
||||
Project.id.in_(proj_ids)
|
||||
).all()
|
||||
} if proj_ids else {}
|
||||
|
||||
# 2. Load relevant unit_history rows. We surface state changes that
|
||||
# operators care about on a deployment timeline: calibration status,
|
||||
# retirement, deployed flag, allocation, calibration date, and the
|
||||
# assignment_* events we just added (those are redundant with the
|
||||
# assignment rows themselves, so we skip them to avoid double-rendering).
|
||||
interesting_change_types = (
|
||||
"calibration_status_change",
|
||||
"retired_change",
|
||||
"deployed_change",
|
||||
"allocation_change",
|
||||
"last_calibrated_change",
|
||||
"next_calibration_due_change",
|
||||
)
|
||||
history = (
|
||||
db.query(UnitHistory)
|
||||
.filter(UnitHistory.unit_id == unit_id)
|
||||
.filter(UnitHistory.change_type.in_(interesting_change_types))
|
||||
.order_by(UnitHistory.changed_at.asc())
|
||||
.all()
|
||||
)
|
||||
|
||||
now = datetime.utcnow()
|
||||
|
||||
# 3. Optionally fetch SFM event overlay for each assignment window.
|
||||
# Concurrent fan-out via httpx + asyncio.gather.
|
||||
overlays: dict[str, dict] = {}
|
||||
if include_event_overlay and assignments and unit.device_type == "seismograph":
|
||||
async with httpx.AsyncClient(timeout=_SFM_TIMEOUT) as client:
|
||||
results = await asyncio.gather(
|
||||
*(
|
||||
_fetch_events_for_serial(
|
||||
client,
|
||||
serial=unit_id,
|
||||
from_dt=a.assigned_at,
|
||||
to_dt=a.assigned_until or now,
|
||||
false_trigger=None,
|
||||
limit=_SFM_FETCH_CEILING,
|
||||
)
|
||||
for a in assignments
|
||||
),
|
||||
return_exceptions=False,
|
||||
)
|
||||
for a, events in zip(assignments, results):
|
||||
peak = None
|
||||
peak_at = None
|
||||
last_ev = None
|
||||
for ev in events:
|
||||
pvs = ev.get("peak_vector_sum")
|
||||
if pvs is not None and (peak is None or pvs > peak):
|
||||
peak = pvs
|
||||
peak_at = ev.get("timestamp")
|
||||
ts = ev.get("timestamp")
|
||||
if ts and (last_ev is None or ts > last_ev):
|
||||
last_ev = ts
|
||||
overlays[a.id] = {
|
||||
"event_count": len(events),
|
||||
"peak_pvs": peak,
|
||||
"peak_pvs_at": peak_at,
|
||||
"last_event": last_ev,
|
||||
}
|
||||
|
||||
# 4. Build entries. Start by emitting assignment rows + gap rows between
|
||||
# consecutive assignments, then add state-change rows from unit_history.
|
||||
entries: list[dict] = []
|
||||
|
||||
for idx, a in enumerate(assignments):
|
||||
loc = loc_map.get(a.location_id)
|
||||
proj = proj_map.get(a.project_id)
|
||||
is_active = a.assigned_until is None
|
||||
ends_at = a.assigned_until or now
|
||||
duration_days = (ends_at - a.assigned_at).total_seconds() / 86400 if a.assigned_at else None
|
||||
|
||||
entry = {
|
||||
"kind": "assignment",
|
||||
"starts_at": _iso_utc(a.assigned_at),
|
||||
"ends_at": _iso_utc(a.assigned_until),
|
||||
"duration_days": round(duration_days, 1) if duration_days is not None else None,
|
||||
"assignment_id": a.id,
|
||||
"location_id": a.location_id,
|
||||
"location_name": loc.name if loc else None,
|
||||
"project_id": a.project_id,
|
||||
"project_name": proj.name if proj else None,
|
||||
"is_active": is_active,
|
||||
"notes": a.notes,
|
||||
"event_overlay": overlays.get(a.id),
|
||||
}
|
||||
entries.append(entry)
|
||||
|
||||
# Gap detection: from the end of this assignment to the start of the
|
||||
# next one. Only emit gaps that are at least _MIN_GAP_SECONDS long
|
||||
# so trivial sub-second handoffs during swaps don't clutter the view.
|
||||
if idx + 1 < len(assignments):
|
||||
next_a = assignments[idx + 1]
|
||||
gap_start = a.assigned_until or now
|
||||
gap_end = next_a.assigned_at
|
||||
gap_seconds = (gap_end - gap_start).total_seconds() if gap_end and gap_start else 0
|
||||
if gap_seconds >= _MIN_GAP_SECONDS:
|
||||
entries.append({
|
||||
"kind": "gap",
|
||||
"starts_at": _iso_utc(gap_start),
|
||||
"ends_at": _iso_utc(gap_end),
|
||||
"duration_days": round(gap_seconds / 86400, 1),
|
||||
"context": "between assignments",
|
||||
})
|
||||
|
||||
# 5. State changes — interleaved by timestamp. Skip no-op rows where
|
||||
# old_value == new_value (an artifact of the legacy record_history()
|
||||
# being called on every save regardless of whether the field changed).
|
||||
for h in history:
|
||||
if h.old_value == h.new_value:
|
||||
continue
|
||||
entries.append({
|
||||
"kind": "state_change",
|
||||
"starts_at": _iso_utc(h.changed_at),
|
||||
"ends_at": None,
|
||||
"duration_days": None,
|
||||
"change_type": h.change_type,
|
||||
"field_name": h.field_name,
|
||||
"old_value": h.old_value,
|
||||
"new_value": h.new_value,
|
||||
"source": h.source,
|
||||
"history_notes": h.notes,
|
||||
})
|
||||
|
||||
# 6. Sort newest first. Active assignments (no end) sort by start time,
|
||||
# same as everything else.
|
||||
entries.sort(key=lambda e: e.get("starts_at") or "", reverse=True)
|
||||
|
||||
return {
|
||||
"unit_id": unit.id,
|
||||
"device_type": unit.device_type,
|
||||
"entries": entries,
|
||||
}
|
||||
@@ -31,7 +31,7 @@ class DeviceController:
|
||||
|
||||
Usage:
|
||||
controller = DeviceController()
|
||||
await controller.start_recording("nl43-001", "sound_level_meter", config={})
|
||||
await controller.start_recording("nl43-001", "slm", config={})
|
||||
await controller.stop_recording("seismo-042", "seismograph")
|
||||
"""
|
||||
|
||||
@@ -53,7 +53,7 @@ class DeviceController:
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "sound_level_meter" | "seismograph"
|
||||
device_type: "slm" | "seismograph"
|
||||
config: Device-specific recording configuration
|
||||
|
||||
Returns:
|
||||
@@ -63,7 +63,7 @@ class DeviceController:
|
||||
UnsupportedDeviceTypeError: Device type not supported
|
||||
DeviceControllerError: Operation failed
|
||||
"""
|
||||
if device_type == "sound_level_meter":
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.start_recording(unit_id, config)
|
||||
except SLMMClientError as e:
|
||||
@@ -81,7 +81,7 @@ class DeviceController:
|
||||
else:
|
||||
raise UnsupportedDeviceTypeError(
|
||||
f"Device type '{device_type}' is not supported. "
|
||||
f"Supported types: sound_level_meter, seismograph"
|
||||
f"Supported types: slm, seismograph"
|
||||
)
|
||||
|
||||
async def stop_recording(
|
||||
@@ -94,12 +94,12 @@ class DeviceController:
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "sound_level_meter" | "seismograph"
|
||||
device_type: "slm" | "seismograph"
|
||||
|
||||
Returns:
|
||||
Response dict from device module
|
||||
"""
|
||||
if device_type == "sound_level_meter":
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.stop_recording(unit_id)
|
||||
except SLMMClientError as e:
|
||||
@@ -126,12 +126,12 @@ class DeviceController:
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "sound_level_meter" | "seismograph"
|
||||
device_type: "slm" | "seismograph"
|
||||
|
||||
Returns:
|
||||
Response dict from device module
|
||||
"""
|
||||
if device_type == "sound_level_meter":
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.pause_recording(unit_id)
|
||||
except SLMMClientError as e:
|
||||
@@ -157,12 +157,12 @@ class DeviceController:
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "sound_level_meter" | "seismograph"
|
||||
device_type: "slm" | "seismograph"
|
||||
|
||||
Returns:
|
||||
Response dict from device module
|
||||
"""
|
||||
if device_type == "sound_level_meter":
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.resume_recording(unit_id)
|
||||
except SLMMClientError as e:
|
||||
@@ -192,12 +192,12 @@ class DeviceController:
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "sound_level_meter" | "seismograph"
|
||||
device_type: "slm" | "seismograph"
|
||||
|
||||
Returns:
|
||||
Status dict from device module
|
||||
"""
|
||||
if device_type == "sound_level_meter":
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.get_unit_status(unit_id)
|
||||
except SLMMClientError as e:
|
||||
@@ -224,12 +224,12 @@ class DeviceController:
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "sound_level_meter" | "seismograph"
|
||||
device_type: "slm" | "seismograph"
|
||||
|
||||
Returns:
|
||||
Live data dict from device module
|
||||
"""
|
||||
if device_type == "sound_level_meter":
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.get_live_data(unit_id)
|
||||
except SLMMClientError as e:
|
||||
@@ -261,14 +261,14 @@ class DeviceController:
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "sound_level_meter" | "seismograph"
|
||||
device_type: "slm" | "seismograph"
|
||||
destination_path: Local path to save files
|
||||
files: List of filenames, or None for all
|
||||
|
||||
Returns:
|
||||
Download result with file list
|
||||
"""
|
||||
if device_type == "sound_level_meter":
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.download_files(
|
||||
unit_id,
|
||||
@@ -289,6 +289,74 @@ class DeviceController:
|
||||
else:
|
||||
raise UnsupportedDeviceTypeError(f"Unsupported device type: {device_type}")
|
||||
|
||||
# ========================================================================
|
||||
# FTP Control
|
||||
# ========================================================================
|
||||
|
||||
async def enable_ftp(
|
||||
self,
|
||||
unit_id: str,
|
||||
device_type: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Enable FTP server on device.
|
||||
|
||||
Must be called before downloading files.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "slm" | "seismograph"
|
||||
|
||||
Returns:
|
||||
Response dict with status
|
||||
"""
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.enable_ftp(unit_id)
|
||||
except SLMMClientError as e:
|
||||
raise DeviceControllerError(f"SLMM error: {str(e)}")
|
||||
|
||||
elif device_type == "seismograph":
|
||||
return {
|
||||
"status": "not_implemented",
|
||||
"message": "Seismograph FTP not yet implemented",
|
||||
"unit_id": unit_id,
|
||||
}
|
||||
|
||||
else:
|
||||
raise UnsupportedDeviceTypeError(f"Unsupported device type: {device_type}")
|
||||
|
||||
async def disable_ftp(
|
||||
self,
|
||||
unit_id: str,
|
||||
device_type: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Disable FTP server on device.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "slm" | "seismograph"
|
||||
|
||||
Returns:
|
||||
Response dict with status
|
||||
"""
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.disable_ftp(unit_id)
|
||||
except SLMMClientError as e:
|
||||
raise DeviceControllerError(f"SLMM error: {str(e)}")
|
||||
|
||||
elif device_type == "seismograph":
|
||||
return {
|
||||
"status": "not_implemented",
|
||||
"message": "Seismograph FTP not yet implemented",
|
||||
"unit_id": unit_id,
|
||||
}
|
||||
|
||||
else:
|
||||
raise UnsupportedDeviceTypeError(f"Unsupported device type: {device_type}")
|
||||
|
||||
# ========================================================================
|
||||
# Device Configuration
|
||||
# ========================================================================
|
||||
@@ -304,13 +372,13 @@ class DeviceController:
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "sound_level_meter" | "seismograph"
|
||||
device_type: "slm" | "seismograph"
|
||||
config: Configuration parameters
|
||||
|
||||
Returns:
|
||||
Updated config from device module
|
||||
"""
|
||||
if device_type == "sound_level_meter":
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.update_unit_config(
|
||||
unit_id,
|
||||
@@ -333,6 +401,157 @@ class DeviceController:
|
||||
else:
|
||||
raise UnsupportedDeviceTypeError(f"Unsupported device type: {device_type}")
|
||||
|
||||
# ========================================================================
|
||||
# Store/Index Management
|
||||
# ========================================================================
|
||||
|
||||
async def increment_index(
|
||||
self,
|
||||
unit_id: str,
|
||||
device_type: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Increment the store/index number on a device.
|
||||
|
||||
For SLMs, this increments the store name to prevent "overwrite data?" prompts.
|
||||
Should be called before starting a new measurement if auto_increment_index is enabled.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "slm" | "seismograph"
|
||||
|
||||
Returns:
|
||||
Response dict with old_index and new_index
|
||||
"""
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.increment_index(unit_id)
|
||||
except SLMMClientError as e:
|
||||
raise DeviceControllerError(f"SLMM error: {str(e)}")
|
||||
|
||||
elif device_type == "seismograph":
|
||||
# Seismographs may not have the same concept of store index
|
||||
return {
|
||||
"status": "not_applicable",
|
||||
"message": "Index increment not applicable for seismographs",
|
||||
"unit_id": unit_id,
|
||||
}
|
||||
|
||||
else:
|
||||
raise UnsupportedDeviceTypeError(f"Unsupported device type: {device_type}")
|
||||
|
||||
async def get_index_number(
|
||||
self,
|
||||
unit_id: str,
|
||||
device_type: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Get current store/index number from device.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "slm" | "seismograph"
|
||||
|
||||
Returns:
|
||||
Response dict with current index_number
|
||||
"""
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.get_index_number(unit_id)
|
||||
except SLMMClientError as e:
|
||||
raise DeviceControllerError(f"SLMM error: {str(e)}")
|
||||
|
||||
elif device_type == "seismograph":
|
||||
return {
|
||||
"status": "not_applicable",
|
||||
"message": "Index number not applicable for seismographs",
|
||||
"unit_id": unit_id,
|
||||
}
|
||||
|
||||
else:
|
||||
raise UnsupportedDeviceTypeError(f"Unsupported device type: {device_type}")
|
||||
|
||||
# ========================================================================
|
||||
# Cycle Commands (for scheduled automation)
|
||||
# ========================================================================
|
||||
|
||||
async def start_cycle(
|
||||
self,
|
||||
unit_id: str,
|
||||
device_type: str,
|
||||
sync_clock: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Execute complete start cycle for scheduled automation.
|
||||
|
||||
This handles the full pre-recording workflow:
|
||||
1. Sync device clock to server time
|
||||
2. Find next safe index (with overwrite protection)
|
||||
3. Start measurement
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "slm" | "seismograph"
|
||||
sync_clock: Whether to sync device clock to server time
|
||||
|
||||
Returns:
|
||||
Response dict from device module
|
||||
"""
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.start_cycle(unit_id, sync_clock)
|
||||
except SLMMClientError as e:
|
||||
raise DeviceControllerError(f"SLMM error: {str(e)}")
|
||||
|
||||
elif device_type == "seismograph":
|
||||
return {
|
||||
"status": "not_implemented",
|
||||
"message": "Seismograph start cycle not yet implemented",
|
||||
"unit_id": unit_id,
|
||||
}
|
||||
|
||||
else:
|
||||
raise UnsupportedDeviceTypeError(f"Unsupported device type: {device_type}")
|
||||
|
||||
async def stop_cycle(
|
||||
self,
|
||||
unit_id: str,
|
||||
device_type: str,
|
||||
download: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Execute complete stop cycle for scheduled automation.
|
||||
|
||||
This handles the full post-recording workflow:
|
||||
1. Stop measurement
|
||||
2. Enable FTP
|
||||
3. Download measurement folder
|
||||
4. Verify download
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "slm" | "seismograph"
|
||||
download: Whether to download measurement data
|
||||
|
||||
Returns:
|
||||
Response dict from device module
|
||||
"""
|
||||
if device_type == "slm":
|
||||
try:
|
||||
return await self.slmm_client.stop_cycle(unit_id, download)
|
||||
except SLMMClientError as e:
|
||||
raise DeviceControllerError(f"SLMM error: {str(e)}")
|
||||
|
||||
elif device_type == "seismograph":
|
||||
return {
|
||||
"status": "not_implemented",
|
||||
"message": "Seismograph stop cycle not yet implemented",
|
||||
"unit_id": unit_id,
|
||||
}
|
||||
|
||||
else:
|
||||
raise UnsupportedDeviceTypeError(f"Unsupported device type: {device_type}")
|
||||
|
||||
# ========================================================================
|
||||
# Health Check
|
||||
# ========================================================================
|
||||
@@ -347,12 +566,12 @@ class DeviceController:
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
device_type: "sound_level_meter" | "seismograph"
|
||||
device_type: "slm" | "seismograph"
|
||||
|
||||
Returns:
|
||||
True if device is reachable, False otherwise
|
||||
"""
|
||||
if device_type == "sound_level_meter":
|
||||
if device_type == "slm":
|
||||
try:
|
||||
status = await self.slmm_client.get_unit_status(unit_id)
|
||||
return status.get("last_seen") is not None
|
||||
|
||||
@@ -0,0 +1,184 @@
|
||||
"""
|
||||
Device Status Monitor
|
||||
|
||||
Background task that monitors device reachability via SLMM polling status
|
||||
and triggers alerts when devices go offline or come back online.
|
||||
|
||||
This service bridges SLMM's device polling with Terra-View's alert system.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Optional, Dict
|
||||
|
||||
from backend.database import SessionLocal
|
||||
from backend.services.slmm_client import get_slmm_client, SLMMClientError
|
||||
from backend.services.alert_service import get_alert_service
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class DeviceStatusMonitor:
|
||||
"""
|
||||
Monitors device reachability via SLMM's polling status endpoint.
|
||||
|
||||
Detects state transitions (online→offline, offline→online) and
|
||||
triggers AlertService to create/resolve alerts.
|
||||
|
||||
Usage:
|
||||
monitor = DeviceStatusMonitor()
|
||||
await monitor.start() # Start background monitoring
|
||||
monitor.stop() # Stop monitoring
|
||||
"""
|
||||
|
||||
def __init__(self, check_interval: int = 60):
|
||||
"""
|
||||
Initialize the monitor.
|
||||
|
||||
Args:
|
||||
check_interval: Seconds between status checks (default: 60)
|
||||
"""
|
||||
self.check_interval = check_interval
|
||||
self.running = False
|
||||
self.task: Optional[asyncio.Task] = None
|
||||
self.slmm_client = get_slmm_client()
|
||||
|
||||
# Track previous device states to detect transitions
|
||||
self._device_states: Dict[str, bool] = {}
|
||||
|
||||
async def start(self):
|
||||
"""Start the monitoring background task."""
|
||||
if self.running:
|
||||
logger.warning("DeviceStatusMonitor is already running")
|
||||
return
|
||||
|
||||
self.running = True
|
||||
self.task = asyncio.create_task(self._monitor_loop())
|
||||
logger.info(f"DeviceStatusMonitor started (checking every {self.check_interval}s)")
|
||||
|
||||
def stop(self):
|
||||
"""Stop the monitoring background task."""
|
||||
self.running = False
|
||||
if self.task:
|
||||
self.task.cancel()
|
||||
logger.info("DeviceStatusMonitor stopped")
|
||||
|
||||
async def _monitor_loop(self):
|
||||
"""Main monitoring loop."""
|
||||
while self.running:
|
||||
try:
|
||||
await self._check_all_devices()
|
||||
except Exception as e:
|
||||
logger.error(f"Error in device status monitor: {e}", exc_info=True)
|
||||
|
||||
# Sleep in small intervals for graceful shutdown
|
||||
for _ in range(self.check_interval):
|
||||
if not self.running:
|
||||
break
|
||||
await asyncio.sleep(1)
|
||||
|
||||
logger.info("DeviceStatusMonitor loop exited")
|
||||
|
||||
async def _check_all_devices(self):
|
||||
"""
|
||||
Fetch polling status from SLMM and detect state transitions.
|
||||
|
||||
Uses GET /api/slmm/_polling/status (proxied to SLMM)
|
||||
"""
|
||||
try:
|
||||
# Get status from SLMM
|
||||
status_response = await self.slmm_client.get_polling_status()
|
||||
devices = status_response.get("devices", [])
|
||||
|
||||
if not devices:
|
||||
logger.debug("No devices in polling status response")
|
||||
return
|
||||
|
||||
db = SessionLocal()
|
||||
try:
|
||||
alert_service = get_alert_service(db)
|
||||
|
||||
for device in devices:
|
||||
unit_id = device.get("unit_id")
|
||||
if not unit_id:
|
||||
continue
|
||||
|
||||
is_reachable = device.get("is_reachable", True)
|
||||
previous_reachable = self._device_states.get(unit_id)
|
||||
|
||||
# Skip if this is the first check (no previous state)
|
||||
if previous_reachable is None:
|
||||
self._device_states[unit_id] = is_reachable
|
||||
logger.debug(f"Initial state for {unit_id}: reachable={is_reachable}")
|
||||
continue
|
||||
|
||||
# Detect offline transition (was online, now offline)
|
||||
if previous_reachable and not is_reachable:
|
||||
logger.warning(f"Device {unit_id} went OFFLINE")
|
||||
alert_service.create_device_offline_alert(
|
||||
unit_id=unit_id,
|
||||
consecutive_failures=device.get("consecutive_failures", 0),
|
||||
last_error=device.get("last_error"),
|
||||
)
|
||||
|
||||
# Detect online transition (was offline, now online)
|
||||
elif not previous_reachable and is_reachable:
|
||||
logger.info(f"Device {unit_id} came back ONLINE")
|
||||
alert_service.resolve_device_offline_alert(unit_id)
|
||||
|
||||
# Update tracked state
|
||||
self._device_states[unit_id] = is_reachable
|
||||
|
||||
# Cleanup expired alerts while we're here
|
||||
alert_service.cleanup_expired_alerts()
|
||||
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
except SLMMClientError as e:
|
||||
logger.warning(f"Could not reach SLMM for status check: {e}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error checking device status: {e}", exc_info=True)
|
||||
|
||||
def get_tracked_devices(self) -> Dict[str, bool]:
|
||||
"""
|
||||
Get the current tracked device states.
|
||||
|
||||
Returns:
|
||||
Dict mapping unit_id to is_reachable status
|
||||
"""
|
||||
return dict(self._device_states)
|
||||
|
||||
def clear_tracked_devices(self):
|
||||
"""Clear all tracked device states (useful for testing)."""
|
||||
self._device_states.clear()
|
||||
|
||||
|
||||
# Singleton instance
|
||||
_monitor_instance: Optional[DeviceStatusMonitor] = None
|
||||
|
||||
|
||||
def get_device_status_monitor() -> DeviceStatusMonitor:
|
||||
"""
|
||||
Get the device status monitor singleton instance.
|
||||
|
||||
Returns:
|
||||
DeviceStatusMonitor instance
|
||||
"""
|
||||
global _monitor_instance
|
||||
if _monitor_instance is None:
|
||||
_monitor_instance = DeviceStatusMonitor()
|
||||
return _monitor_instance
|
||||
|
||||
|
||||
async def start_device_status_monitor():
|
||||
"""Start the global device status monitor."""
|
||||
monitor = get_device_status_monitor()
|
||||
await monitor.start()
|
||||
|
||||
|
||||
def stop_device_status_monitor():
|
||||
"""Stop the global device status monitor."""
|
||||
monitor = get_device_status_monitor()
|
||||
monitor.stop()
|
||||
@@ -0,0 +1,725 @@
|
||||
"""
|
||||
Fleet Calendar Service
|
||||
|
||||
Business logic for:
|
||||
- Calculating unit availability on any given date
|
||||
- Calibration status tracking (valid, expiring soon, expired)
|
||||
- Job reservation management
|
||||
- Conflict detection (calibration expires mid-job)
|
||||
"""
|
||||
|
||||
from datetime import date, datetime, timedelta
|
||||
from typing import Dict, List, Optional, Tuple
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import and_, or_
|
||||
|
||||
from backend.models import (
|
||||
RosterUnit, JobReservation, JobReservationUnit,
|
||||
UserPreferences, Project, DeploymentRecord
|
||||
)
|
||||
|
||||
|
||||
def get_calibration_status(
|
||||
unit: RosterUnit,
|
||||
check_date: date,
|
||||
warning_days: int = 30
|
||||
) -> str:
|
||||
"""
|
||||
Determine calibration status for a unit on a specific date.
|
||||
|
||||
Returns:
|
||||
"valid" - Calibration is good on this date
|
||||
"expiring_soon" - Within warning_days of expiry
|
||||
"expired" - Calibration has expired
|
||||
"needs_calibration" - No calibration date set
|
||||
"""
|
||||
if not unit.last_calibrated:
|
||||
return "needs_calibration"
|
||||
|
||||
# Calculate expiry date (1 year from last calibration)
|
||||
expiry_date = unit.last_calibrated + timedelta(days=365)
|
||||
|
||||
if check_date >= expiry_date:
|
||||
return "expired"
|
||||
elif check_date >= expiry_date - timedelta(days=warning_days):
|
||||
return "expiring_soon"
|
||||
else:
|
||||
return "valid"
|
||||
|
||||
|
||||
def get_unit_reservations_on_date(
|
||||
db: Session,
|
||||
unit_id: str,
|
||||
check_date: date
|
||||
) -> List[JobReservation]:
|
||||
"""Get all reservations that include this unit on the given date."""
|
||||
|
||||
# Get reservation IDs that have this unit assigned
|
||||
assigned_reservation_ids = db.query(JobReservationUnit.reservation_id).filter(
|
||||
JobReservationUnit.unit_id == unit_id
|
||||
).subquery()
|
||||
|
||||
# Get reservations that:
|
||||
# 1. Have this unit assigned AND date is within range
|
||||
reservations = db.query(JobReservation).filter(
|
||||
JobReservation.id.in_(assigned_reservation_ids),
|
||||
JobReservation.start_date <= check_date,
|
||||
JobReservation.end_date >= check_date
|
||||
).all()
|
||||
|
||||
return reservations
|
||||
|
||||
|
||||
def get_active_deployment(db: Session, unit_id: str) -> Optional[DeploymentRecord]:
|
||||
"""Return the active (unreturned) deployment record for a unit, or None."""
|
||||
return (
|
||||
db.query(DeploymentRecord)
|
||||
.filter(
|
||||
DeploymentRecord.unit_id == unit_id,
|
||||
DeploymentRecord.actual_removal_date == None
|
||||
)
|
||||
.order_by(DeploymentRecord.created_at.desc())
|
||||
.first()
|
||||
)
|
||||
|
||||
|
||||
def is_unit_available_on_date(
|
||||
db: Session,
|
||||
unit: RosterUnit,
|
||||
check_date: date,
|
||||
warning_days: int = 30
|
||||
) -> Tuple[bool, str, Optional[str]]:
|
||||
"""
|
||||
Check if a unit is available on a specific date.
|
||||
|
||||
Returns:
|
||||
(is_available, status, reservation_name)
|
||||
- is_available: True if unit can be assigned to new work
|
||||
- status: "available", "reserved", "expired", "retired", "needs_calibration", "in_field"
|
||||
- reservation_name: Name of blocking reservation or project ref (if any)
|
||||
"""
|
||||
# Check if retired
|
||||
if unit.retired:
|
||||
return False, "retired", None
|
||||
|
||||
# Check calibration status
|
||||
cal_status = get_calibration_status(unit, check_date, warning_days)
|
||||
if cal_status == "expired":
|
||||
return False, "expired", None
|
||||
if cal_status == "needs_calibration":
|
||||
return False, "needs_calibration", None
|
||||
|
||||
# Check for an active deployment record (unit is physically in the field)
|
||||
active_deployment = get_active_deployment(db, unit.id)
|
||||
if active_deployment:
|
||||
label = active_deployment.project_ref or "Field deployment"
|
||||
return False, "in_field", label
|
||||
|
||||
# Check if already reserved
|
||||
reservations = get_unit_reservations_on_date(db, unit.id, check_date)
|
||||
if reservations:
|
||||
return False, "reserved", reservations[0].name
|
||||
|
||||
# Unit is available (even if expiring soon - that's just a warning)
|
||||
return True, "available", None
|
||||
|
||||
|
||||
def get_day_summary(
|
||||
db: Session,
|
||||
check_date: date,
|
||||
device_type: str = "seismograph"
|
||||
) -> Dict:
|
||||
"""
|
||||
Get a complete summary of fleet status for a specific day.
|
||||
|
||||
Returns dict with:
|
||||
- available_units: List of available unit IDs with calibration info
|
||||
- reserved_units: List of reserved unit IDs with reservation info
|
||||
- expired_units: List of units with expired calibration
|
||||
- expiring_soon_units: List of units expiring within warning period
|
||||
- reservations: List of active reservations on this date
|
||||
- counts: Summary counts
|
||||
"""
|
||||
# Get user preferences for warning days
|
||||
prefs = db.query(UserPreferences).filter_by(id=1).first()
|
||||
warning_days = prefs.calibration_warning_days if prefs else 30
|
||||
|
||||
# Get all non-retired units of the specified device type
|
||||
units = db.query(RosterUnit).filter(
|
||||
RosterUnit.device_type == device_type,
|
||||
RosterUnit.retired == False
|
||||
).all()
|
||||
|
||||
available_units = []
|
||||
reserved_units = []
|
||||
expired_units = []
|
||||
expiring_soon_units = []
|
||||
needs_calibration_units = []
|
||||
in_field_units = []
|
||||
cal_expiring_today = [] # Units whose calibration expires ON this day
|
||||
|
||||
for unit in units:
|
||||
is_avail, status, reservation_name = is_unit_available_on_date(
|
||||
db, unit, check_date, warning_days
|
||||
)
|
||||
|
||||
cal_status = get_calibration_status(unit, check_date, warning_days)
|
||||
expiry_date = None
|
||||
if unit.last_calibrated:
|
||||
expiry_date = (unit.last_calibrated + timedelta(days=365)).isoformat()
|
||||
|
||||
unit_info = {
|
||||
"id": unit.id,
|
||||
"last_calibrated": unit.last_calibrated.isoformat() if unit.last_calibrated else None,
|
||||
"expiry_date": expiry_date,
|
||||
"calibration_status": cal_status,
|
||||
"deployed": unit.deployed,
|
||||
"note": unit.note or ""
|
||||
}
|
||||
|
||||
# Check if calibration expires ON this specific day
|
||||
if unit.last_calibrated:
|
||||
unit_expiry_date = unit.last_calibrated + timedelta(days=365)
|
||||
if unit_expiry_date == check_date:
|
||||
cal_expiring_today.append(unit_info)
|
||||
|
||||
if status == "available":
|
||||
available_units.append(unit_info)
|
||||
if cal_status == "expiring_soon":
|
||||
expiring_soon_units.append(unit_info)
|
||||
elif status == "in_field":
|
||||
unit_info["project_ref"] = reservation_name
|
||||
in_field_units.append(unit_info)
|
||||
elif status == "reserved":
|
||||
unit_info["reservation_name"] = reservation_name
|
||||
reserved_units.append(unit_info)
|
||||
if cal_status == "expiring_soon":
|
||||
expiring_soon_units.append(unit_info)
|
||||
elif status == "expired":
|
||||
expired_units.append(unit_info)
|
||||
elif status == "needs_calibration":
|
||||
needs_calibration_units.append(unit_info)
|
||||
|
||||
# Get active reservations on this date
|
||||
reservations = db.query(JobReservation).filter(
|
||||
JobReservation.device_type == device_type,
|
||||
JobReservation.start_date <= check_date,
|
||||
JobReservation.end_date >= check_date
|
||||
).all()
|
||||
|
||||
reservation_list = []
|
||||
for res in reservations:
|
||||
# Count assigned units for this reservation
|
||||
assigned_count = db.query(JobReservationUnit).filter(
|
||||
JobReservationUnit.reservation_id == res.id
|
||||
).count()
|
||||
|
||||
reservation_list.append({
|
||||
"id": res.id,
|
||||
"name": res.name,
|
||||
"start_date": res.start_date.isoformat(),
|
||||
"end_date": res.end_date.isoformat(),
|
||||
"assignment_type": res.assignment_type,
|
||||
"quantity_needed": res.quantity_needed,
|
||||
"assigned_count": assigned_count,
|
||||
"color": res.color,
|
||||
"project_id": res.project_id
|
||||
})
|
||||
|
||||
return {
|
||||
"date": check_date.isoformat(),
|
||||
"device_type": device_type,
|
||||
"available_units": available_units,
|
||||
"in_field_units": in_field_units,
|
||||
"reserved_units": reserved_units,
|
||||
"expired_units": expired_units,
|
||||
"expiring_soon_units": expiring_soon_units,
|
||||
"needs_calibration_units": needs_calibration_units,
|
||||
"cal_expiring_today": cal_expiring_today,
|
||||
"reservations": reservation_list,
|
||||
"counts": {
|
||||
"available": len(available_units),
|
||||
"in_field": len(in_field_units),
|
||||
"reserved": len(reserved_units),
|
||||
"expired": len(expired_units),
|
||||
"expiring_soon": len(expiring_soon_units),
|
||||
"needs_calibration": len(needs_calibration_units),
|
||||
"cal_expiring_today": len(cal_expiring_today),
|
||||
"total": len(units)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
def get_calendar_year_data(
|
||||
db: Session,
|
||||
year: int,
|
||||
device_type: str = "seismograph"
|
||||
) -> Dict:
|
||||
"""
|
||||
Get calendar data for an entire year.
|
||||
|
||||
For performance, this returns summary counts per day rather than
|
||||
full unit lists. Use get_day_summary() for detailed day data.
|
||||
"""
|
||||
# Get user preferences
|
||||
prefs = db.query(UserPreferences).filter_by(id=1).first()
|
||||
warning_days = prefs.calibration_warning_days if prefs else 30
|
||||
|
||||
# Get all units
|
||||
units = db.query(RosterUnit).filter(
|
||||
RosterUnit.device_type == device_type,
|
||||
RosterUnit.retired == False
|
||||
).all()
|
||||
|
||||
# Get all reservations that overlap with this year
|
||||
# Include TBD reservations (end_date is null) that started before year end
|
||||
year_start = date(year, 1, 1)
|
||||
year_end = date(year, 12, 31)
|
||||
|
||||
reservations = db.query(JobReservation).filter(
|
||||
JobReservation.device_type == device_type,
|
||||
JobReservation.start_date <= year_end,
|
||||
or_(
|
||||
JobReservation.end_date >= year_start,
|
||||
JobReservation.end_date == None # TBD reservations
|
||||
)
|
||||
).all()
|
||||
|
||||
# Get all unit assignments for these reservations
|
||||
reservation_ids = [r.id for r in reservations]
|
||||
assignments = db.query(JobReservationUnit).filter(
|
||||
JobReservationUnit.reservation_id.in_(reservation_ids)
|
||||
).all() if reservation_ids else []
|
||||
|
||||
# Build a lookup: unit_id -> list of (start_date, end_date, reservation_name)
|
||||
# For TBD reservations, use estimated_end_date if available, or a far future date
|
||||
unit_reservations = {}
|
||||
for res in reservations:
|
||||
res_assignments = [a for a in assignments if a.reservation_id == res.id]
|
||||
for assignment in res_assignments:
|
||||
unit_id = assignment.unit_id
|
||||
# Use unit-specific dates if set, otherwise use reservation dates
|
||||
start_d = assignment.unit_start_date or res.start_date
|
||||
if assignment.unit_end_tbd or (assignment.unit_end_date is None and res.end_date_tbd):
|
||||
# TBD: use estimated date or far future for availability calculation
|
||||
end_d = res.estimated_end_date or date(year + 5, 12, 31)
|
||||
else:
|
||||
end_d = assignment.unit_end_date or res.end_date or date(year + 5, 12, 31)
|
||||
|
||||
if unit_id not in unit_reservations:
|
||||
unit_reservations[unit_id] = []
|
||||
unit_reservations[unit_id].append((start_d, end_d, res.name))
|
||||
|
||||
# Build set of unit IDs that have an active deployment record (still in the field)
|
||||
unit_ids = [u.id for u in units]
|
||||
active_deployments = db.query(DeploymentRecord.unit_id).filter(
|
||||
DeploymentRecord.unit_id.in_(unit_ids),
|
||||
DeploymentRecord.actual_removal_date == None
|
||||
).all()
|
||||
unit_in_field = {row.unit_id for row in active_deployments}
|
||||
|
||||
# Generate data for each month
|
||||
months_data = {}
|
||||
|
||||
for month in range(1, 13):
|
||||
# Get first and last day of month
|
||||
first_day = date(year, month, 1)
|
||||
if month == 12:
|
||||
last_day = date(year, 12, 31)
|
||||
else:
|
||||
last_day = date(year, month + 1, 1) - timedelta(days=1)
|
||||
|
||||
days_data = {}
|
||||
current_day = first_day
|
||||
|
||||
while current_day <= last_day:
|
||||
available = 0
|
||||
in_field = 0
|
||||
reserved = 0
|
||||
expired = 0
|
||||
expiring_soon = 0
|
||||
needs_cal = 0
|
||||
cal_expiring_on_day = 0 # Units whose calibration expires ON this day
|
||||
cal_expired_on_day = 0 # Units whose calibration expired ON this day
|
||||
|
||||
for unit in units:
|
||||
# Check calibration
|
||||
cal_status = get_calibration_status(unit, current_day, warning_days)
|
||||
|
||||
# Check if calibration expires/expired ON this specific day
|
||||
if unit.last_calibrated:
|
||||
unit_expiry = unit.last_calibrated + timedelta(days=365)
|
||||
if unit_expiry == current_day:
|
||||
cal_expiring_on_day += 1
|
||||
# Check if expired yesterday (first day of being expired)
|
||||
elif unit_expiry == current_day - timedelta(days=1):
|
||||
cal_expired_on_day += 1
|
||||
|
||||
if cal_status == "expired":
|
||||
expired += 1
|
||||
continue
|
||||
if cal_status == "needs_calibration":
|
||||
needs_cal += 1
|
||||
continue
|
||||
|
||||
# Check active deployment record (in field)
|
||||
if unit.id in unit_in_field:
|
||||
in_field += 1
|
||||
continue
|
||||
|
||||
# Check if reserved
|
||||
is_reserved = False
|
||||
if unit.id in unit_reservations:
|
||||
for start_d, end_d, _ in unit_reservations[unit.id]:
|
||||
if start_d <= current_day <= end_d:
|
||||
is_reserved = True
|
||||
break
|
||||
|
||||
if is_reserved:
|
||||
reserved += 1
|
||||
else:
|
||||
available += 1
|
||||
|
||||
if cal_status == "expiring_soon":
|
||||
expiring_soon += 1
|
||||
|
||||
days_data[current_day.day] = {
|
||||
"available": available,
|
||||
"in_field": in_field,
|
||||
"reserved": reserved,
|
||||
"expired": expired,
|
||||
"expiring_soon": expiring_soon,
|
||||
"needs_calibration": needs_cal,
|
||||
"cal_expiring_on_day": cal_expiring_on_day,
|
||||
"cal_expired_on_day": cal_expired_on_day
|
||||
}
|
||||
|
||||
current_day += timedelta(days=1)
|
||||
|
||||
months_data[month] = {
|
||||
"name": first_day.strftime("%B"),
|
||||
"short_name": first_day.strftime("%b"),
|
||||
"days": days_data,
|
||||
"first_weekday": first_day.weekday(), # 0=Monday, 6=Sunday
|
||||
"num_days": last_day.day
|
||||
}
|
||||
|
||||
# Also include reservation summary for the year
|
||||
reservation_list = []
|
||||
for res in reservations:
|
||||
assigned_count = len([a for a in assignments if a.reservation_id == res.id])
|
||||
reservation_list.append({
|
||||
"id": res.id,
|
||||
"name": res.name,
|
||||
"start_date": res.start_date.isoformat(),
|
||||
"end_date": res.end_date.isoformat(),
|
||||
"quantity_needed": res.quantity_needed,
|
||||
"assigned_count": assigned_count,
|
||||
"color": res.color
|
||||
})
|
||||
|
||||
return {
|
||||
"year": year,
|
||||
"device_type": device_type,
|
||||
"months": months_data,
|
||||
"reservations": reservation_list,
|
||||
"total_units": len(units)
|
||||
}
|
||||
|
||||
|
||||
def get_rolling_calendar_data(
|
||||
db: Session,
|
||||
start_year: int,
|
||||
start_month: int,
|
||||
device_type: str = "seismograph"
|
||||
) -> Dict:
|
||||
"""
|
||||
Get calendar data for 12 months starting from a specific month/year.
|
||||
|
||||
This supports the rolling calendar view where users can scroll through
|
||||
months one at a time, viewing any 12-month window.
|
||||
"""
|
||||
# Get user preferences
|
||||
prefs = db.query(UserPreferences).filter_by(id=1).first()
|
||||
warning_days = prefs.calibration_warning_days if prefs else 30
|
||||
|
||||
# Get all units
|
||||
units = db.query(RosterUnit).filter(
|
||||
RosterUnit.device_type == device_type,
|
||||
RosterUnit.retired == False
|
||||
).all()
|
||||
|
||||
# Calculate the date range for 12 months
|
||||
first_date = date(start_year, start_month, 1)
|
||||
# Calculate end date (12 months later)
|
||||
end_year = start_year + 1 if start_month == 1 else start_year
|
||||
end_month = 12 if start_month == 1 else start_month - 1
|
||||
if start_month == 1:
|
||||
end_year = start_year
|
||||
end_month = 12
|
||||
else:
|
||||
# 12 months from start_month means we end at start_month - 1 next year
|
||||
end_year = start_year + 1
|
||||
end_month = start_month - 1
|
||||
|
||||
# Actually, simpler: go 11 months forward from start
|
||||
end_year = start_year + ((start_month + 10) // 12)
|
||||
end_month = ((start_month + 10) % 12) + 1
|
||||
if end_month == 12:
|
||||
last_date = date(end_year, 12, 31)
|
||||
else:
|
||||
last_date = date(end_year, end_month + 1, 1) - timedelta(days=1)
|
||||
|
||||
# Get all reservations that overlap with this 12-month range
|
||||
reservations = db.query(JobReservation).filter(
|
||||
JobReservation.device_type == device_type,
|
||||
JobReservation.start_date <= last_date,
|
||||
or_(
|
||||
JobReservation.end_date >= first_date,
|
||||
JobReservation.end_date == None # TBD reservations
|
||||
)
|
||||
).all()
|
||||
|
||||
# Get all unit assignments for these reservations
|
||||
reservation_ids = [r.id for r in reservations]
|
||||
assignments = db.query(JobReservationUnit).filter(
|
||||
JobReservationUnit.reservation_id.in_(reservation_ids)
|
||||
).all() if reservation_ids else []
|
||||
|
||||
# Build a lookup: unit_id -> list of (start_date, end_date, reservation_name)
|
||||
unit_reservations = {}
|
||||
for res in reservations:
|
||||
res_assignments = [a for a in assignments if a.reservation_id == res.id]
|
||||
for assignment in res_assignments:
|
||||
unit_id = assignment.unit_id
|
||||
start_d = assignment.unit_start_date or res.start_date
|
||||
if assignment.unit_end_tbd or (assignment.unit_end_date is None and res.end_date_tbd):
|
||||
end_d = res.estimated_end_date or date(start_year + 5, 12, 31)
|
||||
else:
|
||||
end_d = assignment.unit_end_date or res.end_date or date(start_year + 5, 12, 31)
|
||||
|
||||
if unit_id not in unit_reservations:
|
||||
unit_reservations[unit_id] = []
|
||||
unit_reservations[unit_id].append((start_d, end_d, res.name))
|
||||
|
||||
# Build set of unit IDs that have an active deployment record (still in the field)
|
||||
unit_ids = [u.id for u in units]
|
||||
active_deployments = db.query(DeploymentRecord.unit_id).filter(
|
||||
DeploymentRecord.unit_id.in_(unit_ids),
|
||||
DeploymentRecord.actual_removal_date == None
|
||||
).all()
|
||||
unit_in_field = {row.unit_id for row in active_deployments}
|
||||
|
||||
# Generate data for each of the 12 months
|
||||
months_data = []
|
||||
current_year = start_year
|
||||
current_month = start_month
|
||||
|
||||
for i in range(12):
|
||||
# Calculate this month's year and month
|
||||
m_year = start_year + ((start_month - 1 + i) // 12)
|
||||
m_month = ((start_month - 1 + i) % 12) + 1
|
||||
|
||||
first_day = date(m_year, m_month, 1)
|
||||
if m_month == 12:
|
||||
last_day = date(m_year, 12, 31)
|
||||
else:
|
||||
last_day = date(m_year, m_month + 1, 1) - timedelta(days=1)
|
||||
|
||||
days_data = {}
|
||||
current_day = first_day
|
||||
|
||||
while current_day <= last_day:
|
||||
available = 0
|
||||
reserved = 0
|
||||
expired = 0
|
||||
expiring_soon = 0
|
||||
needs_cal = 0
|
||||
cal_expiring_on_day = 0
|
||||
cal_expired_on_day = 0
|
||||
|
||||
for unit in units:
|
||||
cal_status = get_calibration_status(unit, current_day, warning_days)
|
||||
|
||||
if unit.last_calibrated:
|
||||
unit_expiry = unit.last_calibrated + timedelta(days=365)
|
||||
if unit_expiry == current_day:
|
||||
cal_expiring_on_day += 1
|
||||
elif unit_expiry == current_day - timedelta(days=1):
|
||||
cal_expired_on_day += 1
|
||||
|
||||
if cal_status == "expired":
|
||||
expired += 1
|
||||
continue
|
||||
if cal_status == "needs_calibration":
|
||||
needs_cal += 1
|
||||
continue
|
||||
|
||||
is_reserved = False
|
||||
if unit.id in unit_reservations:
|
||||
for start_d, end_d, _ in unit_reservations[unit.id]:
|
||||
if start_d <= current_day <= end_d:
|
||||
is_reserved = True
|
||||
break
|
||||
|
||||
if is_reserved:
|
||||
reserved += 1
|
||||
else:
|
||||
available += 1
|
||||
|
||||
if cal_status == "expiring_soon":
|
||||
expiring_soon += 1
|
||||
|
||||
days_data[current_day.day] = {
|
||||
"available": available,
|
||||
"reserved": reserved,
|
||||
"expired": expired,
|
||||
"expiring_soon": expiring_soon,
|
||||
"needs_calibration": needs_cal,
|
||||
"cal_expiring_on_day": cal_expiring_on_day,
|
||||
"cal_expired_on_day": cal_expired_on_day
|
||||
}
|
||||
|
||||
current_day += timedelta(days=1)
|
||||
|
||||
months_data.append({
|
||||
"year": m_year,
|
||||
"month": m_month,
|
||||
"name": first_day.strftime("%B"),
|
||||
"short_name": first_day.strftime("%b"),
|
||||
"year_short": first_day.strftime("%y"),
|
||||
"days": days_data,
|
||||
"first_weekday": first_day.weekday(),
|
||||
"num_days": last_day.day
|
||||
})
|
||||
|
||||
return {
|
||||
"start_year": start_year,
|
||||
"start_month": start_month,
|
||||
"device_type": device_type,
|
||||
"months": months_data,
|
||||
"total_units": len(units)
|
||||
}
|
||||
|
||||
|
||||
def check_calibration_conflicts(
|
||||
db: Session,
|
||||
reservation_id: str
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
Check if any units assigned to a reservation will have their
|
||||
calibration expire during the reservation period.
|
||||
|
||||
Returns list of conflicts with unit info and expiry date.
|
||||
"""
|
||||
reservation = db.query(JobReservation).filter_by(id=reservation_id).first()
|
||||
if not reservation:
|
||||
return []
|
||||
|
||||
# Get assigned units
|
||||
assigned = db.query(JobReservationUnit).filter_by(
|
||||
reservation_id=reservation_id
|
||||
).all()
|
||||
|
||||
conflicts = []
|
||||
for assignment in assigned:
|
||||
unit = db.query(RosterUnit).filter_by(id=assignment.unit_id).first()
|
||||
if not unit or not unit.last_calibrated:
|
||||
continue
|
||||
|
||||
expiry_date = unit.last_calibrated + timedelta(days=365)
|
||||
|
||||
# Check if expiry falls within reservation period
|
||||
if reservation.start_date < expiry_date <= reservation.end_date:
|
||||
conflicts.append({
|
||||
"unit_id": unit.id,
|
||||
"last_calibrated": unit.last_calibrated.isoformat(),
|
||||
"expiry_date": expiry_date.isoformat(),
|
||||
"reservation_name": reservation.name,
|
||||
"days_into_job": (expiry_date - reservation.start_date).days
|
||||
})
|
||||
|
||||
return conflicts
|
||||
|
||||
|
||||
def get_available_units_for_period(
|
||||
db: Session,
|
||||
start_date: date,
|
||||
end_date: date,
|
||||
device_type: str = "seismograph",
|
||||
exclude_reservation_id: Optional[str] = None
|
||||
) -> List[Dict]:
|
||||
"""
|
||||
Get units that are available for the entire specified period.
|
||||
|
||||
A unit is available if:
|
||||
- Not retired
|
||||
- Calibration is valid through the end date
|
||||
- Not assigned to any other reservation that overlaps the period
|
||||
"""
|
||||
prefs = db.query(UserPreferences).filter_by(id=1).first()
|
||||
warning_days = prefs.calibration_warning_days if prefs else 30
|
||||
|
||||
units = db.query(RosterUnit).filter(
|
||||
RosterUnit.device_type == device_type,
|
||||
RosterUnit.retired == False
|
||||
).all()
|
||||
|
||||
# Get reservations that overlap with this period
|
||||
overlapping_reservations = db.query(JobReservation).filter(
|
||||
JobReservation.device_type == device_type,
|
||||
JobReservation.start_date <= end_date,
|
||||
JobReservation.end_date >= start_date
|
||||
)
|
||||
|
||||
if exclude_reservation_id:
|
||||
overlapping_reservations = overlapping_reservations.filter(
|
||||
JobReservation.id != exclude_reservation_id
|
||||
)
|
||||
|
||||
overlapping_reservations = overlapping_reservations.all()
|
||||
|
||||
# Get all units assigned to overlapping reservations
|
||||
reserved_unit_ids = set()
|
||||
for res in overlapping_reservations:
|
||||
assigned = db.query(JobReservationUnit).filter_by(
|
||||
reservation_id=res.id
|
||||
).all()
|
||||
for a in assigned:
|
||||
reserved_unit_ids.add(a.unit_id)
|
||||
|
||||
# Get units with active deployment records (still in the field)
|
||||
unit_ids = [u.id for u in units]
|
||||
active_deps = db.query(DeploymentRecord.unit_id).filter(
|
||||
DeploymentRecord.unit_id.in_(unit_ids),
|
||||
DeploymentRecord.actual_removal_date == None
|
||||
).all()
|
||||
in_field_unit_ids = {row.unit_id for row in active_deps}
|
||||
|
||||
available_units = []
|
||||
for unit in units:
|
||||
# Check if already reserved
|
||||
if unit.id in reserved_unit_ids:
|
||||
continue
|
||||
# Check if currently in the field
|
||||
if unit.id in in_field_unit_ids:
|
||||
continue
|
||||
|
||||
if unit.last_calibrated:
|
||||
expiry_date = unit.last_calibrated + timedelta(days=365)
|
||||
cal_status = get_calibration_status(unit, end_date, warning_days)
|
||||
else:
|
||||
expiry_date = None
|
||||
cal_status = "needs_calibration"
|
||||
|
||||
available_units.append({
|
||||
"id": unit.id,
|
||||
"last_calibrated": unit.last_calibrated.isoformat() if unit.last_calibrated else None,
|
||||
"expiry_date": expiry_date.isoformat() if expiry_date else None,
|
||||
"calibration_status": cal_status,
|
||||
"deployed": unit.deployed,
|
||||
"out_for_calibration": unit.out_for_calibration or False,
|
||||
"note": unit.note or ""
|
||||
})
|
||||
|
||||
return available_units
|
||||
@@ -0,0 +1,435 @@
|
||||
"""
|
||||
project_merge.py — consolidate a duplicate project into another.
|
||||
|
||||
Use case: the metadata-backfill parser (and operators) create projects with
|
||||
slight name variations ("SR81" vs "SR 81", "Swank-Karns Crossing" vs
|
||||
"Swank-Karns Crossings", "Trumbull-Bryman Mont.Dam" vs
|
||||
"Trumbull-Brayman-Mont Dam"). Operator picks a SOURCE project to merge
|
||||
into a TARGET project; everything attached to source moves to target,
|
||||
same-named locations consolidate, and source is soft-deleted.
|
||||
|
||||
Public API:
|
||||
preview(db, source_id, target_id) → MergePreview
|
||||
execute(db, source_id, target_id, *, decided_by="operator") → MergeResult
|
||||
|
||||
Both raise HTTPException with appropriate 4xx codes for validation failures.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
from fastapi import HTTPException
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from backend.models import (
|
||||
Project,
|
||||
ProjectModule,
|
||||
MonitoringLocation,
|
||||
UnitAssignment,
|
||||
UnitHistory,
|
||||
MonitoringSession,
|
||||
DataFile,
|
||||
)
|
||||
|
||||
log = logging.getLogger("backend.services.project_merge")
|
||||
|
||||
|
||||
# ── Dataclasses ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@dataclass
|
||||
class LocationMergePlan:
|
||||
source_id: str
|
||||
source_name: str
|
||||
target_id: Optional[str] # None = will be inserted as-new under target project
|
||||
target_name: Optional[str] # name in target after merge
|
||||
action: str # "move" | "consolidate"
|
||||
assignments_moving: int
|
||||
sessions_moving: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class MergePreview:
|
||||
source_project_id: str
|
||||
source_project_name: str
|
||||
target_project_id: str
|
||||
target_project_name: str
|
||||
location_plans: list[LocationMergePlan] = field(default_factory=list)
|
||||
total_assignments_moving: int = 0
|
||||
total_sessions_moving: int = 0
|
||||
total_data_files_moving: int = 0
|
||||
modules_to_add: list[str] = field(default_factory=list)
|
||||
warnings: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
@dataclass
|
||||
class MergeResult:
|
||||
source_project_id: str
|
||||
target_project_id: str
|
||||
assignments_moved: int
|
||||
locations_moved: int
|
||||
locations_consolidated: int
|
||||
sessions_moved: int
|
||||
data_files_moved: int
|
||||
modules_added: list[str]
|
||||
audit_rows_written: int
|
||||
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _normalise_name(s: Optional[str]) -> str:
|
||||
"""Case-insensitive, whitespace-collapsing name normalisation.
|
||||
|
||||
Lighter than metadata_backfill._normalise (no punctuation stripping)
|
||||
— for merging we want "Loc 1" and "Loc 1" to match but NOT "Loc 1"
|
||||
and "Loc-1" (those might be intentionally different). If operators
|
||||
DO want loose matching, they can rename one before merging.
|
||||
"""
|
||||
if not s:
|
||||
return ""
|
||||
import re
|
||||
return re.sub(r"\s+", " ", s.strip()).casefold()
|
||||
|
||||
|
||||
def _validate_pair(db: Session, source_id: str, target_id: str) -> tuple[Project, Project]:
|
||||
if source_id == target_id:
|
||||
raise HTTPException(status_code=400, detail="Cannot merge a project into itself.")
|
||||
|
||||
source = db.query(Project).filter_by(id=source_id).first()
|
||||
target = db.query(Project).filter_by(id=target_id).first()
|
||||
if source is None:
|
||||
raise HTTPException(status_code=404, detail=f"Source project not found.")
|
||||
if target is None:
|
||||
raise HTTPException(status_code=404, detail=f"Target project not found.")
|
||||
if source.status == "deleted":
|
||||
raise HTTPException(status_code=400, detail=f"Source project '{source.name}' is already deleted.")
|
||||
if target.status == "deleted":
|
||||
raise HTTPException(status_code=400, detail=f"Target project '{target.name}' is deleted.")
|
||||
|
||||
return source, target
|
||||
|
||||
|
||||
# ── Preview ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def preview(db: Session, source_id: str, target_id: str) -> MergePreview:
|
||||
"""Build a preview of what the merge will do. No writes."""
|
||||
source, target = _validate_pair(db, source_id, target_id)
|
||||
|
||||
# Locations in source vs target.
|
||||
source_locs = (
|
||||
db.query(MonitoringLocation)
|
||||
.filter(MonitoringLocation.project_id == source_id)
|
||||
.all()
|
||||
)
|
||||
target_locs = (
|
||||
db.query(MonitoringLocation)
|
||||
.filter(MonitoringLocation.project_id == target_id)
|
||||
.all()
|
||||
)
|
||||
target_by_norm = {_normalise_name(l.name): l for l in target_locs}
|
||||
|
||||
location_plans: list[LocationMergePlan] = []
|
||||
total_assignments_moving = 0
|
||||
total_sessions_moving = 0
|
||||
|
||||
for sl in source_locs:
|
||||
n = _normalise_name(sl.name)
|
||||
tl = target_by_norm.get(n)
|
||||
|
||||
a_count = (
|
||||
db.query(UnitAssignment)
|
||||
.filter(UnitAssignment.location_id == sl.id)
|
||||
.count()
|
||||
)
|
||||
s_count = (
|
||||
db.query(MonitoringSession)
|
||||
.filter(MonitoringSession.location_id == sl.id)
|
||||
.count()
|
||||
)
|
||||
total_assignments_moving += a_count
|
||||
total_sessions_moving += s_count
|
||||
|
||||
if tl is not None:
|
||||
location_plans.append(LocationMergePlan(
|
||||
source_id = sl.id,
|
||||
source_name = sl.name,
|
||||
target_id = tl.id,
|
||||
target_name = tl.name,
|
||||
action = "consolidate",
|
||||
assignments_moving = a_count,
|
||||
sessions_moving = s_count,
|
||||
))
|
||||
else:
|
||||
location_plans.append(LocationMergePlan(
|
||||
source_id = sl.id,
|
||||
source_name = sl.name,
|
||||
target_id = None,
|
||||
target_name = sl.name,
|
||||
action = "move",
|
||||
assignments_moving = a_count,
|
||||
sessions_moving = s_count,
|
||||
))
|
||||
|
||||
# DataFiles attached to the source project (if the table exists with a
|
||||
# project_id column). Optional — terra-view's DataFile model may not
|
||||
# always FK to project, so handle gracefully.
|
||||
df_count = 0
|
||||
try:
|
||||
df_count = (
|
||||
db.query(DataFile)
|
||||
.filter(DataFile.project_id == source_id)
|
||||
.count()
|
||||
)
|
||||
except Exception:
|
||||
df_count = 0
|
||||
total_data_files_moving = df_count
|
||||
|
||||
# Modules: add anything in source missing from target.
|
||||
src_modules = {
|
||||
m.module_type for m in db.query(ProjectModule)
|
||||
.filter(ProjectModule.project_id == source_id, ProjectModule.enabled.is_(True))
|
||||
.all()
|
||||
}
|
||||
tgt_modules = {
|
||||
m.module_type for m in db.query(ProjectModule)
|
||||
.filter(ProjectModule.project_id == target_id, ProjectModule.enabled.is_(True))
|
||||
.all()
|
||||
}
|
||||
modules_to_add = sorted(src_modules - tgt_modules)
|
||||
|
||||
warnings: list[str] = []
|
||||
# Surface conditions the operator should think about.
|
||||
consolidations = sum(1 for p in location_plans if p.action == "consolidate")
|
||||
if consolidations:
|
||||
warnings.append(
|
||||
f"{consolidations} location(s) with matching names will be consolidated "
|
||||
f"(source assignments will move to the target's existing location). "
|
||||
f"If your same-named locations are actually different sites, rename one first."
|
||||
)
|
||||
if source.client_name and target.client_name and source.client_name.strip().casefold() != target.client_name.strip().casefold():
|
||||
warnings.append(
|
||||
f"Client names differ: source is \"{source.client_name}\", target is "
|
||||
f"\"{target.client_name}\". Target's client name will be kept."
|
||||
)
|
||||
|
||||
return MergePreview(
|
||||
source_project_id = source.id,
|
||||
source_project_name = source.name,
|
||||
target_project_id = target.id,
|
||||
target_project_name = target.name,
|
||||
location_plans = location_plans,
|
||||
total_assignments_moving = total_assignments_moving,
|
||||
total_sessions_moving = total_sessions_moving,
|
||||
total_data_files_moving = total_data_files_moving,
|
||||
modules_to_add = modules_to_add,
|
||||
warnings = warnings,
|
||||
)
|
||||
|
||||
|
||||
# ── Execute ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def execute(
|
||||
db: Session,
|
||||
source_id: str,
|
||||
target_id: str,
|
||||
*,
|
||||
decided_by: str = "operator",
|
||||
) -> MergeResult:
|
||||
"""Perform the merge in a single transaction.
|
||||
|
||||
Steps:
|
||||
1. Re-validate the pair.
|
||||
2. For each location in source:
|
||||
- if a same-name location exists in target → "consolidate" mode:
|
||||
move source's assignments + sessions to target's location id,
|
||||
delete source's location.
|
||||
- else → "move" mode: just re-point the location's project_id.
|
||||
3. Move any remaining direct-to-project FK rows (DataFiles).
|
||||
4. Ensure target has all of source's modules.
|
||||
5. Soft-delete source project.
|
||||
6. Write a UnitHistory row per assignment that was moved
|
||||
(change_type='assignment_merged') so the deployment timeline
|
||||
on each affected unit reflects the merge.
|
||||
7. Commit.
|
||||
"""
|
||||
source, target = _validate_pair(db, source_id, target_id)
|
||||
|
||||
src_modules = {
|
||||
m.module_type for m in db.query(ProjectModule)
|
||||
.filter(ProjectModule.project_id == source_id, ProjectModule.enabled.is_(True))
|
||||
.all()
|
||||
}
|
||||
tgt_modules = {
|
||||
m.module_type for m in db.query(ProjectModule)
|
||||
.filter(ProjectModule.project_id == target_id, ProjectModule.enabled.is_(True))
|
||||
.all()
|
||||
}
|
||||
modules_to_add = sorted(src_modules - tgt_modules)
|
||||
|
||||
# ── 1. Locations + their dependents ───────────────────────────────
|
||||
source_locs = (
|
||||
db.query(MonitoringLocation)
|
||||
.filter(MonitoringLocation.project_id == source_id)
|
||||
.all()
|
||||
)
|
||||
target_locs = (
|
||||
db.query(MonitoringLocation)
|
||||
.filter(MonitoringLocation.project_id == target_id)
|
||||
.all()
|
||||
)
|
||||
target_by_norm = {_normalise_name(l.name): l for l in target_locs}
|
||||
|
||||
assignments_moved = 0
|
||||
sessions_moved = 0
|
||||
locations_moved = 0
|
||||
locations_consolidated = 0
|
||||
audit_rows_written = 0
|
||||
|
||||
for sl in source_locs:
|
||||
n = _normalise_name(sl.name)
|
||||
tl = target_by_norm.get(n)
|
||||
|
||||
# Pull this location's assignments + sessions (we'll re-point them).
|
||||
assignments = (
|
||||
db.query(UnitAssignment)
|
||||
.filter(UnitAssignment.location_id == sl.id)
|
||||
.all()
|
||||
)
|
||||
sessions = (
|
||||
db.query(MonitoringSession)
|
||||
.filter(MonitoringSession.location_id == sl.id)
|
||||
.all()
|
||||
)
|
||||
|
||||
if tl is not None:
|
||||
# Consolidate: move dependents to target's existing location;
|
||||
# then delete the source location.
|
||||
for a in assignments:
|
||||
old_loc_id = a.location_id
|
||||
a.location_id = tl.id
|
||||
a.project_id = target.id
|
||||
|
||||
db.add(UnitHistory(
|
||||
unit_id = a.unit_id,
|
||||
change_type = "assignment_merged",
|
||||
field_name = "unit_assignment.project_id",
|
||||
old_value = f"{source.name} / {sl.name}",
|
||||
new_value = f"{target.name} / {tl.name}",
|
||||
changed_at = datetime.utcnow(),
|
||||
source = "project_merge",
|
||||
notes = (
|
||||
f"Project merge: '{source.name}' → '{target.name}'. "
|
||||
f"Location consolidated by name match. "
|
||||
f"By: {decided_by}."
|
||||
),
|
||||
))
|
||||
audit_rows_written += 1
|
||||
assignments_moved += 1
|
||||
|
||||
for s in sessions:
|
||||
s.location_id = tl.id
|
||||
s.project_id = target.id
|
||||
sessions_moved += 1
|
||||
|
||||
# Delete the now-empty source location.
|
||||
db.delete(sl)
|
||||
locations_consolidated += 1
|
||||
else:
|
||||
# Move: just re-point this location to the target project.
|
||||
sl.project_id = target.id
|
||||
|
||||
for a in assignments:
|
||||
old_proj_id = a.project_id
|
||||
a.project_id = target.id
|
||||
|
||||
db.add(UnitHistory(
|
||||
unit_id = a.unit_id,
|
||||
change_type = "assignment_merged",
|
||||
field_name = "unit_assignment.project_id",
|
||||
old_value = f"{source.name} / {sl.name}",
|
||||
new_value = f"{target.name} / {sl.name}",
|
||||
changed_at = datetime.utcnow(),
|
||||
source = "project_merge",
|
||||
notes = (
|
||||
f"Project merge: '{source.name}' → '{target.name}'. "
|
||||
f"Location moved as-is. By: {decided_by}."
|
||||
),
|
||||
))
|
||||
audit_rows_written += 1
|
||||
assignments_moved += 1
|
||||
|
||||
for s in sessions:
|
||||
s.project_id = target.id
|
||||
sessions_moved += 1
|
||||
|
||||
locations_moved += 1
|
||||
|
||||
# ── 2. Direct-to-project rows (DataFiles, ScheduledActions) ──────
|
||||
data_files_moved = 0
|
||||
try:
|
||||
data_files = (
|
||||
db.query(DataFile)
|
||||
.filter(DataFile.project_id == source_id)
|
||||
.all()
|
||||
)
|
||||
for df in data_files:
|
||||
df.project_id = target.id
|
||||
data_files_moved += 1
|
||||
except Exception as e:
|
||||
log.warning("DataFile move skipped (model may differ): %s", e)
|
||||
|
||||
# ── 3. UnitAssignments that point directly at source.project_id with
|
||||
# no location (shouldn't happen but be defensive) ──────────────
|
||||
orphan_assignments = (
|
||||
db.query(UnitAssignment)
|
||||
.filter(UnitAssignment.project_id == source_id)
|
||||
.all()
|
||||
)
|
||||
for a in orphan_assignments:
|
||||
# Already moved if its location was moved. Catch any stragglers.
|
||||
if a.project_id == source_id:
|
||||
a.project_id = target.id
|
||||
|
||||
# ── 4. Modules ────────────────────────────────────────────────────
|
||||
import uuid
|
||||
for mod_type in modules_to_add:
|
||||
db.add(ProjectModule(
|
||||
id = str(uuid.uuid4()),
|
||||
project_id = target.id,
|
||||
module_type = mod_type,
|
||||
enabled = True,
|
||||
))
|
||||
|
||||
# Disable source's modules (defensive — source is being soft-deleted
|
||||
# but its modules table rows could still be inspected).
|
||||
for m in db.query(ProjectModule).filter(ProjectModule.project_id == source_id).all():
|
||||
m.enabled = False
|
||||
|
||||
# ── 5. Soft-delete source ─────────────────────────────────────────
|
||||
source.status = "deleted"
|
||||
source.deleted_at = datetime.utcnow()
|
||||
|
||||
# Final audit row on the source project itself (operator-facing).
|
||||
# We don't have a Project-level history table, so log on every
|
||||
# affected unit as a marker. Already done per-assignment above.
|
||||
|
||||
db.commit()
|
||||
|
||||
return MergeResult(
|
||||
source_project_id = source.id,
|
||||
target_project_id = target.id,
|
||||
assignments_moved = assignments_moved,
|
||||
locations_moved = locations_moved,
|
||||
locations_consolidated = locations_consolidated,
|
||||
sessions_moved = sessions_moved,
|
||||
data_files_moved = data_files_moved,
|
||||
modules_added = modules_to_add,
|
||||
audit_rows_written = audit_rows_written,
|
||||
)
|
||||
@@ -0,0 +1,235 @@
|
||||
"""
|
||||
project_tidy.py — find duplicate-looking projects + offer bulk merge.
|
||||
|
||||
The metadata-backfill parser is good at clustering events into candidate
|
||||
projects but doesn't compare its proposed project names against EACH OTHER
|
||||
(it only checks against existing terra-view projects). After a bulk
|
||||
apply, you can end up with many near-duplicate projects — typo variants,
|
||||
abbreviation differences, etc. This module surfaces them as pairs the
|
||||
operator can merge.
|
||||
|
||||
Pairs vs clusters: a fully-connected group like (A, B, C) where each pair
|
||||
scores >= threshold becomes 3 pairs. The operator has to do 2 merges to
|
||||
fully consolidate. We don't try to be smarter about transitive grouping —
|
||||
in practice operators want to review the highest-similarity pair first
|
||||
anyway, and the list re-computes after each merge.
|
||||
|
||||
Public API:
|
||||
find_duplicate_pairs(db, *, threshold=0.85, max_pairs=200) → list[DuplicatePair]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
import rapidfuzz
|
||||
from sqlalchemy import func
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from backend.models import (
|
||||
Project,
|
||||
MonitoringLocation,
|
||||
UnitAssignment,
|
||||
)
|
||||
from backend.services.metadata_backfill import _normalise as _meta_normalise
|
||||
|
||||
log = logging.getLogger("backend.services.project_tidy")
|
||||
|
||||
|
||||
DEFAULT_THRESHOLD = 0.85 # WRatio similarity above which we surface a pair
|
||||
DEFAULT_MAX_PAIRS = 200 # Cap the result list to keep response small
|
||||
MIN_NORMALISED_LENGTH = 4 # Skip projects whose normalised name is too short
|
||||
# to fuzzy-match safely (avoids "1" / "1" pairs).
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProjectSummary:
|
||||
id: str
|
||||
name: str
|
||||
project_number: Optional[str]
|
||||
client_name: Optional[str]
|
||||
source: str # 'manual' | 'metadata_backfill' | ...
|
||||
status: str
|
||||
location_count: int
|
||||
assignment_count: int
|
||||
event_count_total: int # approx — sum across assignments
|
||||
|
||||
|
||||
@dataclass
|
||||
class DuplicatePair:
|
||||
a: ProjectSummary
|
||||
b: ProjectSummary
|
||||
score: float
|
||||
suggested_target_id: str # the recommended "keep" side
|
||||
reason: str # why we picked that target
|
||||
|
||||
|
||||
# ── Helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _normalise_project_name(name: str) -> str:
|
||||
"""Project-name normalisation for tidy comparison.
|
||||
|
||||
Reuses the metadata_backfill normaliser (lowercase, punctuation→space,
|
||||
collapse whitespace). Returns "" for None or all-punctuation names.
|
||||
"""
|
||||
return _meta_normalise(name)
|
||||
|
||||
|
||||
def _summarise_projects(db: Session) -> list[ProjectSummary]:
|
||||
"""One row per active project with cached counts. Excludes deleted."""
|
||||
projects = (
|
||||
db.query(Project)
|
||||
.filter(Project.status != "deleted")
|
||||
.all()
|
||||
)
|
||||
|
||||
# Bulk lookup: assignment counts + location counts per project.
|
||||
loc_counts: dict[str, int] = dict(
|
||||
db.query(MonitoringLocation.project_id, func.count(MonitoringLocation.id))
|
||||
.filter(MonitoringLocation.project_id.in_([p.id for p in projects]) if projects else False)
|
||||
.group_by(MonitoringLocation.project_id)
|
||||
.all()
|
||||
)
|
||||
asgn_counts: dict[str, int] = dict(
|
||||
db.query(UnitAssignment.project_id, func.count(UnitAssignment.id))
|
||||
.filter(UnitAssignment.project_id.in_([p.id for p in projects]) if projects else False)
|
||||
.group_by(UnitAssignment.project_id)
|
||||
.all()
|
||||
)
|
||||
|
||||
summaries: list[ProjectSummary] = []
|
||||
for p in projects:
|
||||
summaries.append(ProjectSummary(
|
||||
id = p.id,
|
||||
name = p.name,
|
||||
project_number = p.project_number,
|
||||
client_name = p.client_name,
|
||||
source = None, # filled below per assignment
|
||||
status = p.status or "active",
|
||||
location_count = loc_counts.get(p.id, 0),
|
||||
assignment_count = asgn_counts.get(p.id, 0),
|
||||
event_count_total = 0, # not cheap to compute here; left 0
|
||||
))
|
||||
|
||||
# Determine each project's dominant assignment source. Used to break ties
|
||||
# when picking the "keep" target — prefer manual over parser-created.
|
||||
rows = (
|
||||
db.query(UnitAssignment.project_id, UnitAssignment.source, func.count(UnitAssignment.id))
|
||||
.group_by(UnitAssignment.project_id, UnitAssignment.source)
|
||||
.all()
|
||||
)
|
||||
by_proj_src: dict[str, dict[str, int]] = {}
|
||||
for proj_id, src, cnt in rows:
|
||||
by_proj_src.setdefault(proj_id, {})[src or "manual"] = cnt
|
||||
for s in summaries:
|
||||
src_map = by_proj_src.get(s.id, {})
|
||||
if not src_map:
|
||||
s.source = "manual"
|
||||
else:
|
||||
# Dominant source (most assignments).
|
||||
s.source = max(src_map.items(), key=lambda kv: kv[1])[0]
|
||||
|
||||
return summaries
|
||||
|
||||
|
||||
def _pick_target(a: ProjectSummary, b: ProjectSummary) -> tuple[str, str]:
|
||||
"""Decide which project should be the merge target (the one we keep).
|
||||
|
||||
Priorities (in order):
|
||||
1. The one with `source='manual'` over `source='metadata_backfill'`
|
||||
— operator-curated projects beat parser-created ones.
|
||||
2. The one with a populated `project_number`.
|
||||
3. The one with more locations (more curation history).
|
||||
4. The one with more assignments.
|
||||
5. The one with the shorter, cleaner name (tiebreaker).
|
||||
|
||||
Returns (target_id, reason_string).
|
||||
"""
|
||||
# 1. Source provenance.
|
||||
a_manual = a.source == "manual"
|
||||
b_manual = b.source == "manual"
|
||||
if a_manual and not b_manual:
|
||||
return a.id, "A is manually-created; B is parser-created"
|
||||
if b_manual and not a_manual:
|
||||
return b.id, "B is manually-created; A is parser-created"
|
||||
|
||||
# 2. project_number populated.
|
||||
if a.project_number and not b.project_number:
|
||||
return a.id, "A has a project_number; B doesn't"
|
||||
if b.project_number and not a.project_number:
|
||||
return b.id, "B has a project_number; A doesn't"
|
||||
|
||||
# 3. More locations.
|
||||
if a.location_count > b.location_count:
|
||||
return a.id, f"A has more locations ({a.location_count} vs {b.location_count})"
|
||||
if b.location_count > a.location_count:
|
||||
return b.id, f"B has more locations ({b.location_count} vs {a.location_count})"
|
||||
|
||||
# 4. More assignments.
|
||||
if a.assignment_count > b.assignment_count:
|
||||
return a.id, f"A has more assignments ({a.assignment_count} vs {b.assignment_count})"
|
||||
if b.assignment_count > a.assignment_count:
|
||||
return b.id, f"B has more assignments ({b.assignment_count} vs {a.assignment_count})"
|
||||
|
||||
# 5. Shorter name (less likely to have baked-in junk).
|
||||
if len(a.name) <= len(b.name):
|
||||
return a.id, "A has the shorter / cleaner name"
|
||||
return b.id, "B has the shorter / cleaner name"
|
||||
|
||||
|
||||
# ── Public ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def find_duplicate_pairs(
|
||||
db: Session,
|
||||
*,
|
||||
threshold: float = DEFAULT_THRESHOLD,
|
||||
max_pairs: int = DEFAULT_MAX_PAIRS,
|
||||
) -> list[DuplicatePair]:
|
||||
"""Compute all project-pair similarities above `threshold`.
|
||||
|
||||
O(N^2) over the project count — fine up to ~500 projects; beyond that
|
||||
we'd want a blocked / token-indexed approach. In practice
|
||||
`metadata_backfill` projects tend to share tokens, so a simple
|
||||
pre-filter (skip pairs that share NO tokens) would cheaply cut the
|
||||
inner loop. Deferred until profiling motivates it.
|
||||
"""
|
||||
summaries = _summarise_projects(db)
|
||||
|
||||
# Pre-compute normalised names; skip too-short ones.
|
||||
norm_by_id: dict[str, str] = {}
|
||||
candidates: list[ProjectSummary] = []
|
||||
for s in summaries:
|
||||
n = _normalise_project_name(s.name)
|
||||
if len(n) < MIN_NORMALISED_LENGTH:
|
||||
continue
|
||||
norm_by_id[s.id] = n
|
||||
candidates.append(s)
|
||||
|
||||
pairs: list[DuplicatePair] = []
|
||||
n = len(candidates)
|
||||
for i in range(n):
|
||||
a = candidates[i]
|
||||
a_norm = norm_by_id[a.id]
|
||||
for j in range(i + 1, n):
|
||||
b = candidates[j]
|
||||
b_norm = norm_by_id[b.id]
|
||||
score = rapidfuzz.fuzz.WRatio(a_norm, b_norm) / 100.0
|
||||
if score < threshold:
|
||||
continue
|
||||
target_id, reason = _pick_target(a, b)
|
||||
pairs.append(DuplicatePair(
|
||||
a = a,
|
||||
b = b,
|
||||
score = score,
|
||||
suggested_target_id = target_id,
|
||||
reason = reason,
|
||||
))
|
||||
|
||||
# Sort by score desc, then by total content (more data → review first).
|
||||
pairs.sort(key=lambda p: (-p.score, -(p.a.assignment_count + p.b.assignment_count)))
|
||||
|
||||
return pairs[:max_pairs]
|
||||
@@ -0,0 +1,611 @@
|
||||
"""
|
||||
Recurring Schedule Service
|
||||
|
||||
Manages recurring schedule definitions and generates ScheduledAction
|
||||
instances based on patterns (weekly calendar, simple interval).
|
||||
"""
|
||||
|
||||
import json
|
||||
import uuid
|
||||
import logging
|
||||
from datetime import datetime, timedelta, date, time
|
||||
from typing import Optional, List, Dict, Any, Tuple
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import and_
|
||||
|
||||
from backend.models import RecurringSchedule, ScheduledAction, MonitoringLocation, UnitAssignment, Project
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Day name mapping
|
||||
DAY_NAMES = ["monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday"]
|
||||
|
||||
|
||||
class RecurringScheduleService:
|
||||
"""
|
||||
Service for managing recurring schedules and generating ScheduledActions.
|
||||
|
||||
Supports two schedule types:
|
||||
- weekly_calendar: Specific days with start/end times
|
||||
- simple_interval: Daily stop/download/restart cycles for 24/7 monitoring
|
||||
"""
|
||||
|
||||
def __init__(self, db: Session):
|
||||
self.db = db
|
||||
|
||||
def create_schedule(
|
||||
self,
|
||||
project_id: str,
|
||||
location_id: str,
|
||||
name: str,
|
||||
schedule_type: str,
|
||||
device_type: str = "slm",
|
||||
unit_id: str = None,
|
||||
weekly_pattern: dict = None,
|
||||
interval_type: str = None,
|
||||
cycle_time: str = None,
|
||||
include_download: bool = True,
|
||||
auto_increment_index: bool = True,
|
||||
timezone: str = "America/New_York",
|
||||
start_datetime: datetime = None,
|
||||
end_datetime: datetime = None,
|
||||
) -> RecurringSchedule:
|
||||
"""
|
||||
Create a new recurring schedule.
|
||||
|
||||
Args:
|
||||
project_id: Project ID
|
||||
location_id: Monitoring location ID
|
||||
name: Schedule name
|
||||
schedule_type: "weekly_calendar", "simple_interval", or "one_off"
|
||||
device_type: "slm" or "seismograph"
|
||||
unit_id: Specific unit (optional, can use assignment)
|
||||
weekly_pattern: Dict of day patterns for weekly_calendar
|
||||
interval_type: "daily" or "hourly" for simple_interval
|
||||
cycle_time: Time string "HH:MM" for cycle
|
||||
include_download: Whether to download data on cycle
|
||||
auto_increment_index: Whether to auto-increment store index before start
|
||||
timezone: Timezone for schedule times
|
||||
start_datetime: Start date+time in UTC (one_off only)
|
||||
end_datetime: End date+time in UTC (one_off only)
|
||||
|
||||
Returns:
|
||||
Created RecurringSchedule
|
||||
"""
|
||||
schedule = RecurringSchedule(
|
||||
id=str(uuid.uuid4()),
|
||||
project_id=project_id,
|
||||
location_id=location_id,
|
||||
unit_id=unit_id,
|
||||
name=name,
|
||||
schedule_type=schedule_type,
|
||||
device_type=device_type,
|
||||
weekly_pattern=json.dumps(weekly_pattern) if weekly_pattern else None,
|
||||
interval_type=interval_type,
|
||||
cycle_time=cycle_time,
|
||||
include_download=include_download,
|
||||
auto_increment_index=auto_increment_index,
|
||||
enabled=True,
|
||||
timezone=timezone,
|
||||
start_datetime=start_datetime,
|
||||
end_datetime=end_datetime,
|
||||
)
|
||||
|
||||
# Calculate next occurrence
|
||||
schedule.next_occurrence = self._calculate_next_occurrence(schedule)
|
||||
|
||||
self.db.add(schedule)
|
||||
self.db.commit()
|
||||
self.db.refresh(schedule)
|
||||
|
||||
logger.info(f"Created recurring schedule: {name} ({schedule_type})")
|
||||
return schedule
|
||||
|
||||
def update_schedule(
|
||||
self,
|
||||
schedule_id: str,
|
||||
**kwargs,
|
||||
) -> Optional[RecurringSchedule]:
|
||||
"""
|
||||
Update a recurring schedule.
|
||||
|
||||
Args:
|
||||
schedule_id: Schedule to update
|
||||
**kwargs: Fields to update
|
||||
|
||||
Returns:
|
||||
Updated schedule or None
|
||||
"""
|
||||
schedule = self.db.query(RecurringSchedule).filter_by(id=schedule_id).first()
|
||||
if not schedule:
|
||||
return None
|
||||
|
||||
for key, value in kwargs.items():
|
||||
if hasattr(schedule, key):
|
||||
if key == "weekly_pattern" and isinstance(value, dict):
|
||||
value = json.dumps(value)
|
||||
setattr(schedule, key, value)
|
||||
|
||||
# Recalculate next occurrence
|
||||
schedule.next_occurrence = self._calculate_next_occurrence(schedule)
|
||||
|
||||
self.db.commit()
|
||||
self.db.refresh(schedule)
|
||||
|
||||
logger.info(f"Updated recurring schedule: {schedule.name}")
|
||||
return schedule
|
||||
|
||||
def delete_schedule(self, schedule_id: str) -> bool:
|
||||
"""
|
||||
Delete a recurring schedule and its pending generated actions.
|
||||
|
||||
Args:
|
||||
schedule_id: Schedule to delete
|
||||
|
||||
Returns:
|
||||
True if deleted, False if not found
|
||||
"""
|
||||
schedule = self.db.query(RecurringSchedule).filter_by(id=schedule_id).first()
|
||||
if not schedule:
|
||||
return False
|
||||
|
||||
# Delete pending generated actions for this schedule
|
||||
# The schedule_id is stored in the notes field as JSON
|
||||
pending_actions = self.db.query(ScheduledAction).filter(
|
||||
and_(
|
||||
ScheduledAction.execution_status == "pending",
|
||||
ScheduledAction.notes.like(f'%"schedule_id": "{schedule_id}"%'),
|
||||
)
|
||||
).all()
|
||||
|
||||
deleted_count = len(pending_actions)
|
||||
for action in pending_actions:
|
||||
self.db.delete(action)
|
||||
|
||||
self.db.delete(schedule)
|
||||
self.db.commit()
|
||||
|
||||
logger.info(f"Deleted recurring schedule: {schedule.name} (and {deleted_count} pending actions)")
|
||||
return True
|
||||
|
||||
def enable_schedule(self, schedule_id: str) -> Optional[RecurringSchedule]:
|
||||
"""Enable a disabled schedule."""
|
||||
return self.update_schedule(schedule_id, enabled=True)
|
||||
|
||||
def disable_schedule(self, schedule_id: str) -> Optional[RecurringSchedule]:
|
||||
"""Disable a schedule and cancel its pending actions."""
|
||||
schedule = self.update_schedule(schedule_id, enabled=False)
|
||||
if schedule:
|
||||
# Cancel all pending actions generated by this schedule
|
||||
pending_actions = self.db.query(ScheduledAction).filter(
|
||||
and_(
|
||||
ScheduledAction.execution_status == "pending",
|
||||
ScheduledAction.notes.like(f'%"schedule_id": "{schedule_id}"%'),
|
||||
)
|
||||
).all()
|
||||
|
||||
for action in pending_actions:
|
||||
action.execution_status = "cancelled"
|
||||
|
||||
if pending_actions:
|
||||
self.db.commit()
|
||||
logger.info(f"Cancelled {len(pending_actions)} pending actions for disabled schedule {schedule.name}")
|
||||
|
||||
return schedule
|
||||
|
||||
def generate_actions_for_schedule(
|
||||
self,
|
||||
schedule: RecurringSchedule,
|
||||
horizon_days: int = 7,
|
||||
preview_only: bool = False,
|
||||
) -> List[ScheduledAction]:
|
||||
"""
|
||||
Generate ScheduledAction entries for the next N days based on pattern.
|
||||
|
||||
Args:
|
||||
schedule: The recurring schedule
|
||||
horizon_days: Days ahead to generate
|
||||
preview_only: If True, don't save to DB (for preview)
|
||||
|
||||
Returns:
|
||||
List of generated ScheduledAction instances
|
||||
"""
|
||||
if not schedule.enabled:
|
||||
return []
|
||||
|
||||
if schedule.schedule_type == "weekly_calendar":
|
||||
actions = self._generate_weekly_calendar_actions(schedule, horizon_days)
|
||||
elif schedule.schedule_type == "simple_interval":
|
||||
actions = self._generate_interval_actions(schedule, horizon_days)
|
||||
elif schedule.schedule_type == "one_off":
|
||||
actions = self._generate_one_off_actions(schedule)
|
||||
else:
|
||||
logger.warning(f"Unknown schedule type: {schedule.schedule_type}")
|
||||
return []
|
||||
|
||||
if not preview_only and actions:
|
||||
for action in actions:
|
||||
self.db.add(action)
|
||||
|
||||
schedule.last_generated_at = datetime.utcnow()
|
||||
schedule.next_occurrence = self._calculate_next_occurrence(schedule)
|
||||
|
||||
self.db.commit()
|
||||
logger.info(f"Generated {len(actions)} actions for schedule: {schedule.name}")
|
||||
|
||||
return actions
|
||||
|
||||
def _generate_weekly_calendar_actions(
|
||||
self,
|
||||
schedule: RecurringSchedule,
|
||||
horizon_days: int,
|
||||
) -> List[ScheduledAction]:
|
||||
"""
|
||||
Generate actions from weekly calendar pattern.
|
||||
|
||||
Pattern format:
|
||||
{
|
||||
"monday": {"enabled": true, "start": "19:00", "end": "07:00"},
|
||||
"tuesday": {"enabled": false},
|
||||
...
|
||||
}
|
||||
"""
|
||||
if not schedule.weekly_pattern:
|
||||
return []
|
||||
|
||||
try:
|
||||
pattern = json.loads(schedule.weekly_pattern)
|
||||
except json.JSONDecodeError:
|
||||
logger.error(f"Invalid weekly_pattern JSON for schedule {schedule.id}")
|
||||
return []
|
||||
|
||||
actions = []
|
||||
tz = ZoneInfo(schedule.timezone)
|
||||
now_utc = datetime.utcnow()
|
||||
now_local = now_utc.replace(tzinfo=ZoneInfo("UTC")).astimezone(tz)
|
||||
|
||||
# Get unit_id (from schedule or assignment)
|
||||
unit_id = self._resolve_unit_id(schedule)
|
||||
|
||||
for day_offset in range(horizon_days):
|
||||
check_date = now_local.date() + timedelta(days=day_offset)
|
||||
day_name = DAY_NAMES[check_date.weekday()]
|
||||
day_config = pattern.get(day_name, {})
|
||||
|
||||
if not day_config.get("enabled", False):
|
||||
continue
|
||||
|
||||
start_time_str = day_config.get("start")
|
||||
end_time_str = day_config.get("end")
|
||||
|
||||
if not start_time_str or not end_time_str:
|
||||
continue
|
||||
|
||||
# Parse times
|
||||
start_time = self._parse_time(start_time_str)
|
||||
end_time = self._parse_time(end_time_str)
|
||||
|
||||
if not start_time or not end_time:
|
||||
continue
|
||||
|
||||
# Create start datetime in local timezone
|
||||
start_local = datetime.combine(check_date, start_time, tzinfo=tz)
|
||||
start_utc = start_local.astimezone(ZoneInfo("UTC")).replace(tzinfo=None)
|
||||
|
||||
# Handle overnight schedules (end time is next day)
|
||||
if end_time <= start_time:
|
||||
end_date = check_date + timedelta(days=1)
|
||||
else:
|
||||
end_date = check_date
|
||||
|
||||
end_local = datetime.combine(end_date, end_time, tzinfo=tz)
|
||||
end_utc = end_local.astimezone(ZoneInfo("UTC")).replace(tzinfo=None)
|
||||
|
||||
# Skip if start time has already passed
|
||||
if start_utc <= now_utc:
|
||||
continue
|
||||
|
||||
# Check if action already exists
|
||||
if self._action_exists(schedule.project_id, schedule.location_id, "start", start_utc):
|
||||
continue
|
||||
|
||||
# Build notes with automation metadata
|
||||
start_notes = json.dumps({
|
||||
"schedule_name": schedule.name,
|
||||
"schedule_id": schedule.id,
|
||||
"auto_increment_index": schedule.auto_increment_index,
|
||||
})
|
||||
|
||||
# Create START action
|
||||
start_action = ScheduledAction(
|
||||
id=str(uuid.uuid4()),
|
||||
project_id=schedule.project_id,
|
||||
location_id=schedule.location_id,
|
||||
unit_id=unit_id,
|
||||
action_type="start",
|
||||
device_type=schedule.device_type,
|
||||
scheduled_time=start_utc,
|
||||
execution_status="pending",
|
||||
notes=start_notes,
|
||||
)
|
||||
actions.append(start_action)
|
||||
|
||||
# Create STOP action (stop_cycle handles download when include_download is True)
|
||||
stop_notes = json.dumps({
|
||||
"schedule_name": schedule.name,
|
||||
"schedule_id": schedule.id,
|
||||
"schedule_type": "weekly_calendar",
|
||||
"include_download": schedule.include_download,
|
||||
})
|
||||
stop_action = ScheduledAction(
|
||||
id=str(uuid.uuid4()),
|
||||
project_id=schedule.project_id,
|
||||
location_id=schedule.location_id,
|
||||
unit_id=unit_id,
|
||||
action_type="stop",
|
||||
device_type=schedule.device_type,
|
||||
scheduled_time=end_utc,
|
||||
execution_status="pending",
|
||||
notes=stop_notes,
|
||||
)
|
||||
actions.append(stop_action)
|
||||
|
||||
return actions
|
||||
|
||||
def _generate_interval_actions(
|
||||
self,
|
||||
schedule: RecurringSchedule,
|
||||
horizon_days: int,
|
||||
) -> List[ScheduledAction]:
|
||||
"""
|
||||
Generate actions from simple interval pattern.
|
||||
|
||||
For daily cycles: stop, download (optional), start at cycle_time each day.
|
||||
"""
|
||||
if not schedule.cycle_time:
|
||||
return []
|
||||
|
||||
cycle_time = self._parse_time(schedule.cycle_time)
|
||||
if not cycle_time:
|
||||
return []
|
||||
|
||||
actions = []
|
||||
tz = ZoneInfo(schedule.timezone)
|
||||
now_utc = datetime.utcnow()
|
||||
now_local = now_utc.replace(tzinfo=ZoneInfo("UTC")).astimezone(tz)
|
||||
|
||||
# Get unit_id
|
||||
unit_id = self._resolve_unit_id(schedule)
|
||||
|
||||
for day_offset in range(horizon_days):
|
||||
check_date = now_local.date() + timedelta(days=day_offset)
|
||||
|
||||
# Create cycle datetime in local timezone
|
||||
cycle_local = datetime.combine(check_date, cycle_time, tzinfo=tz)
|
||||
cycle_utc = cycle_local.astimezone(ZoneInfo("UTC")).replace(tzinfo=None)
|
||||
|
||||
# Skip if time has passed
|
||||
if cycle_utc <= now_utc:
|
||||
continue
|
||||
|
||||
# Check if cycle action already exists
|
||||
if self._action_exists(schedule.project_id, schedule.location_id, "cycle", cycle_utc):
|
||||
continue
|
||||
|
||||
# Build notes with metadata for cycle action
|
||||
cycle_notes = json.dumps({
|
||||
"schedule_name": schedule.name,
|
||||
"schedule_id": schedule.id,
|
||||
"cycle_type": "daily",
|
||||
"include_download": schedule.include_download,
|
||||
"auto_increment_index": schedule.auto_increment_index,
|
||||
})
|
||||
|
||||
# Create single CYCLE action that handles stop -> download -> start
|
||||
# The scheduler's _execute_cycle method handles the full workflow with delays
|
||||
cycle_action = ScheduledAction(
|
||||
id=str(uuid.uuid4()),
|
||||
project_id=schedule.project_id,
|
||||
location_id=schedule.location_id,
|
||||
unit_id=unit_id,
|
||||
action_type="cycle",
|
||||
device_type=schedule.device_type,
|
||||
scheduled_time=cycle_utc,
|
||||
execution_status="pending",
|
||||
notes=cycle_notes,
|
||||
)
|
||||
actions.append(cycle_action)
|
||||
|
||||
return actions
|
||||
|
||||
def _generate_one_off_actions(
|
||||
self,
|
||||
schedule: RecurringSchedule,
|
||||
) -> List[ScheduledAction]:
|
||||
"""
|
||||
Generate start and stop actions for a one-off recording.
|
||||
|
||||
Unlike recurring types, this generates exactly one start and one stop action
|
||||
using the schedule's start_datetime and end_datetime directly.
|
||||
"""
|
||||
if not schedule.start_datetime or not schedule.end_datetime:
|
||||
logger.warning(f"One-off schedule {schedule.id} missing start/end datetime")
|
||||
return []
|
||||
|
||||
actions = []
|
||||
now_utc = datetime.utcnow()
|
||||
unit_id = self._resolve_unit_id(schedule)
|
||||
|
||||
# Skip if end time has already passed
|
||||
if schedule.end_datetime <= now_utc:
|
||||
return []
|
||||
|
||||
# Check if actions already exist for this schedule
|
||||
if self._action_exists(schedule.project_id, schedule.location_id, "start", schedule.start_datetime):
|
||||
return []
|
||||
|
||||
# Create START action (only if start time hasn't passed)
|
||||
if schedule.start_datetime > now_utc:
|
||||
start_notes = json.dumps({
|
||||
"schedule_name": schedule.name,
|
||||
"schedule_id": schedule.id,
|
||||
"schedule_type": "one_off",
|
||||
"auto_increment_index": schedule.auto_increment_index,
|
||||
})
|
||||
|
||||
start_action = ScheduledAction(
|
||||
id=str(uuid.uuid4()),
|
||||
project_id=schedule.project_id,
|
||||
location_id=schedule.location_id,
|
||||
unit_id=unit_id,
|
||||
action_type="start",
|
||||
device_type=schedule.device_type,
|
||||
scheduled_time=schedule.start_datetime,
|
||||
execution_status="pending",
|
||||
notes=start_notes,
|
||||
)
|
||||
actions.append(start_action)
|
||||
|
||||
# Create STOP action
|
||||
stop_notes = json.dumps({
|
||||
"schedule_name": schedule.name,
|
||||
"schedule_id": schedule.id,
|
||||
"schedule_type": "one_off",
|
||||
"include_download": schedule.include_download,
|
||||
})
|
||||
|
||||
stop_action = ScheduledAction(
|
||||
id=str(uuid.uuid4()),
|
||||
project_id=schedule.project_id,
|
||||
location_id=schedule.location_id,
|
||||
unit_id=unit_id,
|
||||
action_type="stop",
|
||||
device_type=schedule.device_type,
|
||||
scheduled_time=schedule.end_datetime,
|
||||
execution_status="pending",
|
||||
notes=stop_notes,
|
||||
)
|
||||
actions.append(stop_action)
|
||||
|
||||
return actions
|
||||
|
||||
def _calculate_next_occurrence(self, schedule: RecurringSchedule) -> Optional[datetime]:
|
||||
"""Calculate when the next action should occur."""
|
||||
if not schedule.enabled:
|
||||
return None
|
||||
|
||||
tz = ZoneInfo(schedule.timezone)
|
||||
now_utc = datetime.utcnow()
|
||||
now_local = now_utc.replace(tzinfo=ZoneInfo("UTC")).astimezone(tz)
|
||||
|
||||
if schedule.schedule_type == "weekly_calendar" and schedule.weekly_pattern:
|
||||
try:
|
||||
pattern = json.loads(schedule.weekly_pattern)
|
||||
except:
|
||||
return None
|
||||
|
||||
# Find next enabled day
|
||||
for day_offset in range(8): # Check up to a week ahead
|
||||
check_date = now_local.date() + timedelta(days=day_offset)
|
||||
day_name = DAY_NAMES[check_date.weekday()]
|
||||
day_config = pattern.get(day_name, {})
|
||||
|
||||
if day_config.get("enabled") and day_config.get("start"):
|
||||
start_time = self._parse_time(day_config["start"])
|
||||
if start_time:
|
||||
start_local = datetime.combine(check_date, start_time, tzinfo=tz)
|
||||
start_utc = start_local.astimezone(ZoneInfo("UTC")).replace(tzinfo=None)
|
||||
if start_utc > now_utc:
|
||||
return start_utc
|
||||
|
||||
elif schedule.schedule_type == "simple_interval" and schedule.cycle_time:
|
||||
cycle_time = self._parse_time(schedule.cycle_time)
|
||||
if cycle_time:
|
||||
# Find next cycle time
|
||||
for day_offset in range(2):
|
||||
check_date = now_local.date() + timedelta(days=day_offset)
|
||||
cycle_local = datetime.combine(check_date, cycle_time, tzinfo=tz)
|
||||
cycle_utc = cycle_local.astimezone(ZoneInfo("UTC")).replace(tzinfo=None)
|
||||
if cycle_utc > now_utc:
|
||||
return cycle_utc
|
||||
|
||||
elif schedule.schedule_type == "one_off":
|
||||
if schedule.start_datetime and schedule.start_datetime > now_utc:
|
||||
return schedule.start_datetime
|
||||
elif schedule.end_datetime and schedule.end_datetime > now_utc:
|
||||
return schedule.end_datetime
|
||||
return None
|
||||
|
||||
return None
|
||||
|
||||
def _resolve_unit_id(self, schedule: RecurringSchedule) -> Optional[str]:
|
||||
"""Get unit_id from schedule or active assignment."""
|
||||
if schedule.unit_id:
|
||||
return schedule.unit_id
|
||||
|
||||
# Try to get from active assignment
|
||||
assignment = self.db.query(UnitAssignment).filter(
|
||||
and_(
|
||||
UnitAssignment.location_id == schedule.location_id,
|
||||
UnitAssignment.status == "active",
|
||||
)
|
||||
).first()
|
||||
|
||||
return assignment.unit_id if assignment else None
|
||||
|
||||
def _action_exists(
|
||||
self,
|
||||
project_id: str,
|
||||
location_id: str,
|
||||
action_type: str,
|
||||
scheduled_time: datetime,
|
||||
) -> bool:
|
||||
"""Check if an action already exists for this time slot."""
|
||||
# Allow 5-minute window for duplicate detection
|
||||
time_window_start = scheduled_time - timedelta(minutes=5)
|
||||
time_window_end = scheduled_time + timedelta(minutes=5)
|
||||
|
||||
exists = self.db.query(ScheduledAction).filter(
|
||||
and_(
|
||||
ScheduledAction.project_id == project_id,
|
||||
ScheduledAction.location_id == location_id,
|
||||
ScheduledAction.action_type == action_type,
|
||||
ScheduledAction.scheduled_time >= time_window_start,
|
||||
ScheduledAction.scheduled_time <= time_window_end,
|
||||
ScheduledAction.execution_status == "pending",
|
||||
)
|
||||
).first()
|
||||
|
||||
return exists is not None
|
||||
|
||||
@staticmethod
|
||||
def _parse_time(time_str: str) -> Optional[time]:
|
||||
"""Parse time string "HH:MM" to time object."""
|
||||
try:
|
||||
parts = time_str.split(":")
|
||||
return time(int(parts[0]), int(parts[1]))
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
|
||||
def get_schedules_for_project(self, project_id: str) -> List[RecurringSchedule]:
|
||||
"""Get all recurring schedules for a project."""
|
||||
return self.db.query(RecurringSchedule).filter_by(project_id=project_id).all()
|
||||
|
||||
def get_enabled_schedules(self) -> List[RecurringSchedule]:
|
||||
"""Get all enabled recurring schedules for projects that are not on hold or deleted."""
|
||||
active_project_ids = [
|
||||
p.id for p in self.db.query(Project.id).filter(
|
||||
Project.status.notin_(["on_hold", "archived", "deleted"])
|
||||
).all()
|
||||
]
|
||||
return self.db.query(RecurringSchedule).filter(
|
||||
RecurringSchedule.enabled == True,
|
||||
RecurringSchedule.project_id.in_(active_project_ids),
|
||||
).all()
|
||||
|
||||
|
||||
def get_recurring_schedule_service(db: Session) -> RecurringScheduleService:
|
||||
"""Get a RecurringScheduleService instance."""
|
||||
return RecurringScheduleService(db)
|
||||
@@ -4,22 +4,30 @@ Scheduler Service
|
||||
Executes scheduled actions for Projects system.
|
||||
Monitors pending scheduled actions and executes them by calling device modules (SLMM/SFM).
|
||||
|
||||
Extended to support recurring schedules:
|
||||
- Generates ScheduledActions from RecurringSchedule patterns
|
||||
- Cleans up old completed/failed actions
|
||||
|
||||
This service runs as a background task in FastAPI, checking for pending actions
|
||||
every minute and executing them when their scheduled time arrives.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional, List, Dict, Any
|
||||
from sqlalchemy.orm import Session
|
||||
from sqlalchemy import and_
|
||||
|
||||
from backend.database import SessionLocal
|
||||
from backend.models import ScheduledAction, RecordingSession, MonitoringLocation, Project
|
||||
from backend.models import ScheduledAction, MonitoringSession, MonitoringLocation, Project, RecurringSchedule
|
||||
from backend.services.device_controller import get_device_controller, DeviceControllerError
|
||||
from backend.services.alert_service import get_alert_service
|
||||
import uuid
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SchedulerService:
|
||||
"""
|
||||
@@ -62,11 +70,26 @@ class SchedulerService:
|
||||
|
||||
async def _run_loop(self):
|
||||
"""Main scheduler loop."""
|
||||
# Track when we last generated recurring actions (do this once per hour)
|
||||
last_generation_check = datetime.utcnow() - timedelta(hours=1)
|
||||
|
||||
while self.running:
|
||||
try:
|
||||
# Execute pending actions
|
||||
await self.execute_pending_actions()
|
||||
|
||||
# Generate actions from recurring schedules (every hour)
|
||||
now = datetime.utcnow()
|
||||
if (now - last_generation_check).total_seconds() >= 3600:
|
||||
await self.generate_recurring_actions()
|
||||
last_generation_check = now
|
||||
|
||||
# Cleanup old actions (also every hour, during generation cycle)
|
||||
if (now - last_generation_check).total_seconds() < 60:
|
||||
await self.cleanup_old_actions()
|
||||
|
||||
except Exception as e:
|
||||
print(f"Scheduler error: {e}")
|
||||
logger.error(f"Scheduler error: {e}", exc_info=True)
|
||||
# Continue running even if there's an error
|
||||
|
||||
await asyncio.sleep(self.check_interval)
|
||||
@@ -84,10 +107,19 @@ class SchedulerService:
|
||||
try:
|
||||
# Find pending actions that are due
|
||||
now = datetime.utcnow()
|
||||
|
||||
# Only execute actions for active/completed projects (not on_hold, archived, or deleted)
|
||||
active_project_ids = [
|
||||
p.id for p in db.query(Project.id).filter(
|
||||
Project.status.notin_(["on_hold", "archived", "deleted"])
|
||||
).all()
|
||||
]
|
||||
|
||||
pending_actions = db.query(ScheduledAction).filter(
|
||||
and_(
|
||||
ScheduledAction.execution_status == "pending",
|
||||
ScheduledAction.scheduled_time <= now,
|
||||
ScheduledAction.project_id.in_(active_project_ids),
|
||||
)
|
||||
).order_by(ScheduledAction.scheduled_time).all()
|
||||
|
||||
@@ -162,6 +194,8 @@ class SchedulerService:
|
||||
response = await self._execute_stop(action, unit_id, db)
|
||||
elif action.action_type == "download":
|
||||
response = await self._execute_download(action, unit_id, db)
|
||||
elif action.action_type == "cycle":
|
||||
response = await self._execute_cycle(action, unit_id, db)
|
||||
else:
|
||||
raise Exception(f"Unknown action type: {action.action_type}")
|
||||
|
||||
@@ -175,6 +209,21 @@ class SchedulerService:
|
||||
|
||||
print(f"✓ Action {action.id} completed successfully")
|
||||
|
||||
# Create success alert
|
||||
try:
|
||||
alert_service = get_alert_service(db)
|
||||
alert_metadata = response.get("cycle_response", {}) if isinstance(response, dict) else {}
|
||||
alert_service.create_schedule_completed_alert(
|
||||
schedule_id=action.id,
|
||||
action_type=action.action_type,
|
||||
unit_id=unit_id,
|
||||
project_id=action.project_id,
|
||||
location_id=action.location_id,
|
||||
metadata=alert_metadata,
|
||||
)
|
||||
except Exception as alert_err:
|
||||
logger.warning(f"Failed to create success alert: {alert_err}")
|
||||
|
||||
except Exception as e:
|
||||
# Mark action as failed
|
||||
action.execution_status = "failed"
|
||||
@@ -185,6 +234,20 @@ class SchedulerService:
|
||||
|
||||
print(f"✗ Action {action.id} failed: {e}")
|
||||
|
||||
# Create failure alert
|
||||
try:
|
||||
alert_service = get_alert_service(db)
|
||||
alert_service.create_schedule_failed_alert(
|
||||
schedule_id=action.id,
|
||||
action_type=action.action_type,
|
||||
unit_id=unit_id if 'unit_id' in dir() else action.unit_id,
|
||||
error_message=str(e),
|
||||
project_id=action.project_id,
|
||||
location_id=action.location_id,
|
||||
)
|
||||
except Exception as alert_err:
|
||||
logger.warning(f"Failed to create failure alert: {alert_err}")
|
||||
|
||||
return result
|
||||
|
||||
async def _execute_start(
|
||||
@@ -193,31 +256,41 @@ class SchedulerService:
|
||||
unit_id: str,
|
||||
db: Session,
|
||||
) -> Dict[str, Any]:
|
||||
"""Execute a 'start' action."""
|
||||
# Start recording via device controller
|
||||
response = await self.device_controller.start_recording(
|
||||
"""Execute a 'start' action using the start_cycle command.
|
||||
|
||||
start_cycle handles:
|
||||
1. Sync device clock to server time
|
||||
2. Find next safe index (with overwrite protection)
|
||||
3. Start measurement
|
||||
"""
|
||||
# Execute the full start cycle via device controller
|
||||
# SLMM handles clock sync, index increment, and start
|
||||
cycle_response = await self.device_controller.start_cycle(
|
||||
unit_id,
|
||||
action.device_type,
|
||||
config={}, # TODO: Load config from action.notes or metadata
|
||||
sync_clock=True,
|
||||
)
|
||||
|
||||
# Create recording session
|
||||
session = RecordingSession(
|
||||
session = MonitoringSession(
|
||||
id=str(uuid.uuid4()),
|
||||
project_id=action.project_id,
|
||||
location_id=action.location_id,
|
||||
unit_id=unit_id,
|
||||
session_type="sound" if action.device_type == "sound_level_meter" else "vibration",
|
||||
session_type="sound" if action.device_type == "slm" else "vibration",
|
||||
started_at=datetime.utcnow(),
|
||||
status="recording",
|
||||
session_metadata=json.dumps({"scheduled_action_id": action.id}),
|
||||
session_metadata=json.dumps({
|
||||
"scheduled_action_id": action.id,
|
||||
"cycle_response": cycle_response,
|
||||
}),
|
||||
)
|
||||
db.add(session)
|
||||
|
||||
return {
|
||||
"status": "started",
|
||||
"session_id": session.id,
|
||||
"device_response": response,
|
||||
"cycle_response": cycle_response,
|
||||
}
|
||||
|
||||
async def _execute_stop(
|
||||
@@ -226,19 +299,48 @@ class SchedulerService:
|
||||
unit_id: str,
|
||||
db: Session,
|
||||
) -> Dict[str, Any]:
|
||||
"""Execute a 'stop' action."""
|
||||
# Stop recording via device controller
|
||||
response = await self.device_controller.stop_recording(
|
||||
"""Execute a 'stop' action using the stop_cycle command.
|
||||
|
||||
stop_cycle handles:
|
||||
1. Stop measurement
|
||||
2. Enable FTP
|
||||
3. Download measurement folder to SLMM local storage
|
||||
|
||||
After stop_cycle, if download succeeded, this method fetches the ZIP
|
||||
from SLMM and extracts it into Terra-View's project directory, creating
|
||||
DataFile records for each file.
|
||||
"""
|
||||
import hashlib
|
||||
import io
|
||||
import os
|
||||
import zipfile
|
||||
import httpx
|
||||
from pathlib import Path
|
||||
from backend.models import DataFile
|
||||
|
||||
# Parse notes for download preference
|
||||
include_download = True
|
||||
try:
|
||||
if action.notes:
|
||||
notes_data = json.loads(action.notes)
|
||||
include_download = notes_data.get("include_download", True)
|
||||
except json.JSONDecodeError:
|
||||
pass # Notes is plain text, not JSON
|
||||
|
||||
# Execute the full stop cycle via device controller
|
||||
# SLMM handles stop, FTP enable, and download to SLMM-local storage
|
||||
cycle_response = await self.device_controller.stop_cycle(
|
||||
unit_id,
|
||||
action.device_type,
|
||||
download=include_download,
|
||||
)
|
||||
|
||||
# Find and update the active recording session
|
||||
active_session = db.query(RecordingSession).filter(
|
||||
active_session = db.query(MonitoringSession).filter(
|
||||
and_(
|
||||
RecordingSession.location_id == action.location_id,
|
||||
RecordingSession.unit_id == unit_id,
|
||||
RecordingSession.status == "recording",
|
||||
MonitoringSession.location_id == action.location_id,
|
||||
MonitoringSession.unit_id == unit_id,
|
||||
MonitoringSession.status == "recording",
|
||||
)
|
||||
).first()
|
||||
|
||||
@@ -248,11 +350,91 @@ class SchedulerService:
|
||||
active_session.duration_seconds = int(
|
||||
(active_session.stopped_at - active_session.started_at).total_seconds()
|
||||
)
|
||||
# Store download info in session metadata
|
||||
if cycle_response.get("download_success"):
|
||||
try:
|
||||
metadata = json.loads(active_session.session_metadata or "{}")
|
||||
metadata["downloaded_folder"] = cycle_response.get("downloaded_folder")
|
||||
metadata["local_path"] = cycle_response.get("local_path")
|
||||
active_session.session_metadata = json.dumps(metadata)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
db.commit()
|
||||
|
||||
# If SLMM downloaded the folder successfully, fetch the ZIP from SLMM
|
||||
# and extract it into Terra-View's project directory, creating DataFile records
|
||||
files_created = 0
|
||||
if include_download and cycle_response.get("download_success") and active_session:
|
||||
folder_name = cycle_response.get("downloaded_folder") # e.g. "Auto_0058"
|
||||
remote_path = f"/NL-43/{folder_name}"
|
||||
|
||||
try:
|
||||
SLMM_BASE_URL = os.getenv("SLMM_BASE_URL", "http://localhost:8100")
|
||||
async with httpx.AsyncClient(timeout=600.0) as client:
|
||||
zip_response = await client.post(
|
||||
f"{SLMM_BASE_URL}/api/nl43/{unit_id}/ftp/download-folder",
|
||||
json={"remote_path": remote_path}
|
||||
)
|
||||
|
||||
if zip_response.is_success and len(zip_response.content) > 22:
|
||||
base_dir = Path(f"data/Projects/{action.project_id}/{active_session.id}/{folder_name}")
|
||||
base_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
file_type_map = {
|
||||
'.wav': 'audio', '.mp3': 'audio',
|
||||
'.csv': 'data', '.txt': 'data', '.json': 'data', '.dat': 'data',
|
||||
'.rnd': 'data', '.rnh': 'data',
|
||||
'.log': 'log',
|
||||
'.zip': 'archive',
|
||||
'.jpg': 'image', '.jpeg': 'image', '.png': 'image',
|
||||
'.pdf': 'document',
|
||||
}
|
||||
|
||||
with zipfile.ZipFile(io.BytesIO(zip_response.content)) as zf:
|
||||
for zip_info in zf.filelist:
|
||||
if zip_info.is_dir():
|
||||
continue
|
||||
file_data = zf.read(zip_info.filename)
|
||||
file_path = base_dir / zip_info.filename
|
||||
file_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(file_path, 'wb') as f:
|
||||
f.write(file_data)
|
||||
checksum = hashlib.sha256(file_data).hexdigest()
|
||||
ext = os.path.splitext(zip_info.filename)[1].lower()
|
||||
data_file = DataFile(
|
||||
id=str(uuid.uuid4()),
|
||||
session_id=active_session.id,
|
||||
file_path=str(file_path.relative_to("data")),
|
||||
file_type=file_type_map.get(ext, 'data'),
|
||||
file_size_bytes=len(file_data),
|
||||
downloaded_at=datetime.utcnow(),
|
||||
checksum=checksum,
|
||||
file_metadata=json.dumps({
|
||||
"source": "stop_cycle",
|
||||
"remote_path": remote_path,
|
||||
"unit_id": unit_id,
|
||||
"folder_name": folder_name,
|
||||
"relative_path": zip_info.filename,
|
||||
}),
|
||||
)
|
||||
db.add(data_file)
|
||||
files_created += 1
|
||||
|
||||
db.commit()
|
||||
logger.info(f"Created {files_created} DataFile records for session {active_session.id} from {folder_name}")
|
||||
else:
|
||||
logger.warning(f"ZIP from SLMM for {folder_name} was empty or failed, skipping DataFile creation")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to extract ZIP and create DataFile records for {folder_name}: {e}")
|
||||
# Don't fail the stop action — the device was stopped successfully
|
||||
|
||||
return {
|
||||
"status": "stopped",
|
||||
"session_id": active_session.id if active_session else None,
|
||||
"device_response": response,
|
||||
"cycle_response": cycle_response,
|
||||
"files_created": files_created,
|
||||
}
|
||||
|
||||
async def _execute_download(
|
||||
@@ -261,7 +443,14 @@ class SchedulerService:
|
||||
unit_id: str,
|
||||
db: Session,
|
||||
) -> Dict[str, Any]:
|
||||
"""Execute a 'download' action."""
|
||||
"""Execute a 'download' action.
|
||||
|
||||
This handles standalone download actions (not part of stop_cycle).
|
||||
The workflow is:
|
||||
1. Enable FTP on device
|
||||
2. Download current measurement folder
|
||||
3. (Optionally disable FTP - left enabled for now)
|
||||
"""
|
||||
# Get project and location info for file path
|
||||
location = db.query(MonitoringLocation).filter_by(id=action.location_id).first()
|
||||
project = db.query(Project).filter_by(id=action.project_id).first()
|
||||
@@ -269,22 +458,33 @@ class SchedulerService:
|
||||
if not location or not project:
|
||||
raise Exception("Project or location not found")
|
||||
|
||||
# Build destination path
|
||||
# Example: data/Projects/{project-id}/sound/{location-name}/session-{timestamp}/
|
||||
# Build destination path (for logging/metadata reference)
|
||||
# Actual download location is managed by SLMM (data/downloads/{unit_id}/)
|
||||
session_timestamp = datetime.utcnow().strftime("%Y-%m-%d-%H%M")
|
||||
location_type_dir = "sound" if action.device_type == "sound_level_meter" else "vibration"
|
||||
location_type_dir = "sound" if action.device_type == "slm" else "vibration"
|
||||
|
||||
destination_path = (
|
||||
f"data/Projects/{project.id}/{location_type_dir}/"
|
||||
f"{location.name}/session-{session_timestamp}/"
|
||||
)
|
||||
|
||||
# Download files via device controller
|
||||
# Step 1: Disable FTP first to reset any stale connection state
|
||||
# Then enable FTP on device
|
||||
logger.info(f"Resetting FTP on {unit_id} for download (disable then enable)")
|
||||
try:
|
||||
await self.device_controller.disable_ftp(unit_id, action.device_type)
|
||||
except Exception as e:
|
||||
logger.warning(f"FTP disable failed (may already be off): {e}")
|
||||
await self.device_controller.enable_ftp(unit_id, action.device_type)
|
||||
|
||||
# Step 2: Download current measurement folder
|
||||
# The slmm_client.download_files() now automatically determines the correct
|
||||
# folder based on the device's current index number
|
||||
response = await self.device_controller.download_files(
|
||||
unit_id,
|
||||
action.device_type,
|
||||
destination_path,
|
||||
files=None, # Download all files
|
||||
files=None, # Download all files in current measurement folder
|
||||
)
|
||||
|
||||
# TODO: Create DataFile records for downloaded files
|
||||
@@ -295,6 +495,293 @@ class SchedulerService:
|
||||
"device_response": response,
|
||||
}
|
||||
|
||||
async def _execute_cycle(
|
||||
self,
|
||||
action: ScheduledAction,
|
||||
unit_id: str,
|
||||
db: Session,
|
||||
) -> Dict[str, Any]:
|
||||
"""Execute a full 'cycle' action: stop -> download -> start.
|
||||
|
||||
This combines stop, download, and start into a single action with
|
||||
appropriate delays between steps to ensure device stability.
|
||||
|
||||
Workflow:
|
||||
0. Pause background polling to prevent command conflicts
|
||||
1. Stop measurement (wait 10s)
|
||||
2. Disable FTP to reset state (wait 10s)
|
||||
3. Enable FTP (wait 10s)
|
||||
4. Download current measurement folder
|
||||
5. Wait 30s for device to settle
|
||||
6. Start new measurement cycle
|
||||
7. Re-enable background polling
|
||||
|
||||
Total time: ~70-90 seconds depending on download size
|
||||
"""
|
||||
logger.info(f"[CYCLE] === Starting full cycle for {unit_id} ===")
|
||||
|
||||
result = {
|
||||
"status": "cycle_complete",
|
||||
"steps": {},
|
||||
"old_session_id": None,
|
||||
"new_session_id": None,
|
||||
"polling_paused": False,
|
||||
}
|
||||
|
||||
# Step 0: Pause background polling for this device to prevent command conflicts
|
||||
# NL-43 devices only support one TCP connection at a time
|
||||
logger.info(f"[CYCLE] Step 0: Pausing background polling for {unit_id}")
|
||||
polling_was_enabled = False
|
||||
try:
|
||||
if action.device_type == "slm":
|
||||
# Get current polling state to restore later
|
||||
from backend.services.slmm_client import get_slmm_client
|
||||
slmm = get_slmm_client()
|
||||
try:
|
||||
polling_config = await slmm.get_device_polling_config(unit_id)
|
||||
polling_was_enabled = polling_config.get("poll_enabled", False)
|
||||
except Exception:
|
||||
polling_was_enabled = True # Assume enabled if can't check
|
||||
|
||||
# Disable polling during cycle
|
||||
await slmm.update_device_polling_config(unit_id, poll_enabled=False)
|
||||
result["polling_paused"] = True
|
||||
logger.info(f"[CYCLE] Background polling paused for {unit_id}")
|
||||
except Exception as e:
|
||||
logger.warning(f"[CYCLE] Failed to pause polling (continuing anyway): {e}")
|
||||
|
||||
try:
|
||||
# Step 1: Stop measurement
|
||||
logger.info(f"[CYCLE] Step 1/7: Stopping measurement on {unit_id}")
|
||||
try:
|
||||
stop_response = await self.device_controller.stop_recording(unit_id, action.device_type)
|
||||
result["steps"]["stop"] = {"success": True, "response": stop_response}
|
||||
logger.info(f"[CYCLE] Measurement stopped, waiting 10s...")
|
||||
except Exception as e:
|
||||
logger.warning(f"[CYCLE] Stop failed (may already be stopped): {e}")
|
||||
result["steps"]["stop"] = {"success": False, "error": str(e)}
|
||||
|
||||
await asyncio.sleep(10)
|
||||
|
||||
# Step 2: Disable FTP to reset any stale state
|
||||
logger.info(f"[CYCLE] Step 2/7: Disabling FTP on {unit_id}")
|
||||
try:
|
||||
await self.device_controller.disable_ftp(unit_id, action.device_type)
|
||||
result["steps"]["ftp_disable"] = {"success": True}
|
||||
logger.info(f"[CYCLE] FTP disabled, waiting 10s...")
|
||||
except Exception as e:
|
||||
logger.warning(f"[CYCLE] FTP disable failed (may already be off): {e}")
|
||||
result["steps"]["ftp_disable"] = {"success": False, "error": str(e)}
|
||||
|
||||
await asyncio.sleep(10)
|
||||
|
||||
# Step 3: Enable FTP
|
||||
logger.info(f"[CYCLE] Step 3/7: Enabling FTP on {unit_id}")
|
||||
try:
|
||||
await self.device_controller.enable_ftp(unit_id, action.device_type)
|
||||
result["steps"]["ftp_enable"] = {"success": True}
|
||||
logger.info(f"[CYCLE] FTP enabled, waiting 10s...")
|
||||
except Exception as e:
|
||||
logger.error(f"[CYCLE] FTP enable failed: {e}")
|
||||
result["steps"]["ftp_enable"] = {"success": False, "error": str(e)}
|
||||
# Continue anyway - download will fail but we can still try to start
|
||||
|
||||
await asyncio.sleep(10)
|
||||
|
||||
# Step 4: Download current measurement folder
|
||||
logger.info(f"[CYCLE] Step 4/7: Downloading measurement data from {unit_id}")
|
||||
location = db.query(MonitoringLocation).filter_by(id=action.location_id).first()
|
||||
project = db.query(Project).filter_by(id=action.project_id).first()
|
||||
|
||||
if location and project:
|
||||
session_timestamp = datetime.utcnow().strftime("%Y-%m-%d-%H%M")
|
||||
location_type_dir = "sound" if action.device_type == "slm" else "vibration"
|
||||
destination_path = (
|
||||
f"data/Projects/{project.id}/{location_type_dir}/"
|
||||
f"{location.name}/session-{session_timestamp}/"
|
||||
)
|
||||
|
||||
try:
|
||||
download_response = await self.device_controller.download_files(
|
||||
unit_id,
|
||||
action.device_type,
|
||||
destination_path,
|
||||
files=None,
|
||||
)
|
||||
result["steps"]["download"] = {"success": True, "response": download_response}
|
||||
logger.info(f"[CYCLE] Download complete")
|
||||
except Exception as e:
|
||||
logger.error(f"[CYCLE] Download failed: {e}")
|
||||
result["steps"]["download"] = {"success": False, "error": str(e)}
|
||||
else:
|
||||
result["steps"]["download"] = {"success": False, "error": "Project or location not found"}
|
||||
|
||||
# Close out the old recording session
|
||||
active_session = db.query(MonitoringSession).filter(
|
||||
and_(
|
||||
MonitoringSession.location_id == action.location_id,
|
||||
MonitoringSession.unit_id == unit_id,
|
||||
MonitoringSession.status == "recording",
|
||||
)
|
||||
).first()
|
||||
|
||||
if active_session:
|
||||
active_session.stopped_at = datetime.utcnow()
|
||||
active_session.status = "completed"
|
||||
active_session.duration_seconds = int(
|
||||
(active_session.stopped_at - active_session.started_at).total_seconds()
|
||||
)
|
||||
result["old_session_id"] = active_session.id
|
||||
|
||||
# Step 5: Wait for device to settle before starting new measurement
|
||||
logger.info(f"[CYCLE] Step 5/7: Waiting 30s for device to settle...")
|
||||
await asyncio.sleep(30)
|
||||
|
||||
# Step 6: Start new measurement cycle
|
||||
logger.info(f"[CYCLE] Step 6/7: Starting new measurement on {unit_id}")
|
||||
try:
|
||||
cycle_response = await self.device_controller.start_cycle(
|
||||
unit_id,
|
||||
action.device_type,
|
||||
sync_clock=True,
|
||||
)
|
||||
result["steps"]["start"] = {"success": True, "response": cycle_response}
|
||||
|
||||
# Create new recording session
|
||||
new_session = MonitoringSession(
|
||||
id=str(uuid.uuid4()),
|
||||
project_id=action.project_id,
|
||||
location_id=action.location_id,
|
||||
unit_id=unit_id,
|
||||
session_type="sound" if action.device_type == "slm" else "vibration",
|
||||
started_at=datetime.utcnow(),
|
||||
status="recording",
|
||||
session_metadata=json.dumps({
|
||||
"scheduled_action_id": action.id,
|
||||
"cycle_response": cycle_response,
|
||||
"action_type": "cycle",
|
||||
}),
|
||||
)
|
||||
db.add(new_session)
|
||||
result["new_session_id"] = new_session.id
|
||||
|
||||
logger.info(f"[CYCLE] New measurement started, session {new_session.id}")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"[CYCLE] Start failed: {e}")
|
||||
result["steps"]["start"] = {"success": False, "error": str(e)}
|
||||
raise # Re-raise to mark the action as failed
|
||||
|
||||
finally:
|
||||
# Step 7: Re-enable background polling (always runs, even on failure)
|
||||
if result.get("polling_paused") and polling_was_enabled:
|
||||
logger.info(f"[CYCLE] Step 7/7: Re-enabling background polling for {unit_id}")
|
||||
try:
|
||||
if action.device_type == "slm":
|
||||
from backend.services.slmm_client import get_slmm_client
|
||||
slmm = get_slmm_client()
|
||||
await slmm.update_device_polling_config(unit_id, poll_enabled=True)
|
||||
logger.info(f"[CYCLE] Background polling re-enabled for {unit_id}")
|
||||
except Exception as e:
|
||||
logger.error(f"[CYCLE] Failed to re-enable polling: {e}")
|
||||
# Don't raise - cycle completed, just log the error
|
||||
|
||||
logger.info(f"[CYCLE] === Cycle complete for {unit_id} ===")
|
||||
return result
|
||||
|
||||
# ========================================================================
|
||||
# Recurring Schedule Generation
|
||||
# ========================================================================
|
||||
|
||||
async def generate_recurring_actions(self) -> int:
|
||||
"""
|
||||
Generate ScheduledActions from all enabled recurring schedules.
|
||||
|
||||
Runs once per hour to generate actions for the next 7 days.
|
||||
|
||||
Returns:
|
||||
Total number of actions generated
|
||||
"""
|
||||
db = SessionLocal()
|
||||
total_generated = 0
|
||||
|
||||
try:
|
||||
from backend.services.recurring_schedule_service import get_recurring_schedule_service
|
||||
|
||||
service = get_recurring_schedule_service(db)
|
||||
schedules = service.get_enabled_schedules()
|
||||
|
||||
if not schedules:
|
||||
logger.debug("No enabled recurring schedules found")
|
||||
return 0
|
||||
|
||||
logger.info(f"Generating actions for {len(schedules)} recurring schedule(s)")
|
||||
|
||||
for schedule in schedules:
|
||||
try:
|
||||
# Auto-disable one-off schedules whose end time has passed
|
||||
if schedule.schedule_type == "one_off" and schedule.end_datetime:
|
||||
if schedule.end_datetime <= datetime.utcnow():
|
||||
schedule.enabled = False
|
||||
schedule.next_occurrence = None
|
||||
db.commit()
|
||||
logger.info(f"Auto-disabled completed one-off schedule: {schedule.name}")
|
||||
continue
|
||||
|
||||
actions = service.generate_actions_for_schedule(schedule, horizon_days=7)
|
||||
total_generated += len(actions)
|
||||
except Exception as e:
|
||||
logger.error(f"Error generating actions for schedule {schedule.id}: {e}")
|
||||
|
||||
if total_generated > 0:
|
||||
logger.info(f"Generated {total_generated} scheduled actions from recurring schedules")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error in generate_recurring_actions: {e}", exc_info=True)
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
return total_generated
|
||||
|
||||
async def cleanup_old_actions(self, retention_days: int = 30) -> int:
|
||||
"""
|
||||
Remove old completed/failed actions to prevent database bloat.
|
||||
|
||||
Args:
|
||||
retention_days: Keep actions newer than this many days
|
||||
|
||||
Returns:
|
||||
Number of actions cleaned up
|
||||
"""
|
||||
db = SessionLocal()
|
||||
cleaned = 0
|
||||
|
||||
try:
|
||||
cutoff = datetime.utcnow() - timedelta(days=retention_days)
|
||||
|
||||
old_actions = db.query(ScheduledAction).filter(
|
||||
and_(
|
||||
ScheduledAction.execution_status.in_(["completed", "failed", "cancelled"]),
|
||||
ScheduledAction.executed_at < cutoff,
|
||||
)
|
||||
).all()
|
||||
|
||||
cleaned = len(old_actions)
|
||||
for action in old_actions:
|
||||
db.delete(action)
|
||||
|
||||
if cleaned > 0:
|
||||
db.commit()
|
||||
logger.info(f"Cleaned up {cleaned} old scheduled actions (>{retention_days} days)")
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error cleaning up old actions: {e}")
|
||||
db.rollback()
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
return cleaned
|
||||
|
||||
# ========================================================================
|
||||
# Manual Execution (for testing/debugging)
|
||||
# ========================================================================
|
||||
|
||||
@@ -0,0 +1,592 @@
|
||||
"""
|
||||
SFM events service — bridge between terra-view's UnitAssignment time-windows
|
||||
and the SFM (seismo-relay) events store.
|
||||
|
||||
Architecture:
|
||||
1. Terra-view owns the *assignment graph*: which seismograph was at which
|
||||
monitoring location during which time window (UnitAssignment rows).
|
||||
2. SFM owns the *events store*: triggered waveform events keyed by
|
||||
(serial, timestamp), forwarded from Blastware ACH by series3-watcher.
|
||||
3. This module fans out the assignments for a given location, queries SFM
|
||||
for the events emitted by each (serial, window) pair concurrently, and
|
||||
unions/sorts/paginates the results.
|
||||
|
||||
SFM remains the single source of truth for events. Terra-view does not
|
||||
copy events into its own DB; every query hits SFM live.
|
||||
|
||||
The events_for_location helper is also reused by Phase 3 (project-level
|
||||
roll-up) to aggregate across every location in a project.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
from typing import Optional
|
||||
|
||||
import httpx
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from backend.models import UnitAssignment, RosterUnit, MonitoringLocation, Project
|
||||
|
||||
log = logging.getLogger("backend.services.sfm_events")
|
||||
|
||||
SFM_BASE_URL = os.getenv("SFM_BASE_URL", "http://localhost:8200")
|
||||
|
||||
# Per-request timeout when calling SFM /db/events. SFM is local on the
|
||||
# docker network so this should be fast; bump if you start seeing timeouts.
|
||||
_SFM_TIMEOUT_SECONDS = 10.0
|
||||
|
||||
# Max events we ever fetch per (serial, window) call to SFM. Must match
|
||||
# SFM's own /db/events max limit (currently 5000). The user-facing display
|
||||
# limit is independent — we over-fetch up to this cap so summary stats are
|
||||
# accurate, then trim the displayed list to the requested limit.
|
||||
_SFM_FETCH_CEILING = 5000
|
||||
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _iso_utc(dt: Optional[datetime]) -> Optional[str]:
|
||||
"""Render a datetime in the ISO format SFM /db/events expects."""
|
||||
if dt is None:
|
||||
return None
|
||||
# SFM parses naive ISO strings as UTC; strip tzinfo for consistency.
|
||||
if dt.tzinfo is not None:
|
||||
dt = dt.astimezone(timezone.utc).replace(tzinfo=None)
|
||||
return dt.isoformat(sep=" ", timespec="seconds")
|
||||
|
||||
|
||||
def _intersect_window(
|
||||
assignment_start: datetime,
|
||||
assignment_end: Optional[datetime],
|
||||
filter_from: Optional[datetime],
|
||||
filter_to: Optional[datetime],
|
||||
now: datetime,
|
||||
) -> Optional[tuple[datetime, datetime]]:
|
||||
"""Intersect an assignment window with the requested filter window.
|
||||
|
||||
Returns (effective_start, effective_end) or None if there's no overlap.
|
||||
Open-ended assignments (assigned_until=NULL) are bounded by `now`.
|
||||
"""
|
||||
a_end = assignment_end or now
|
||||
if filter_from and a_end <= filter_from:
|
||||
return None
|
||||
if filter_to and assignment_start >= filter_to:
|
||||
return None
|
||||
start = max(assignment_start, filter_from) if filter_from else assignment_start
|
||||
end = min(a_end, filter_to) if filter_to else a_end
|
||||
if end <= start:
|
||||
return None
|
||||
return (start, end)
|
||||
|
||||
|
||||
async def _fetch_events_for_serial(
|
||||
client: httpx.AsyncClient,
|
||||
serial: str,
|
||||
*,
|
||||
from_dt: datetime,
|
||||
to_dt: datetime,
|
||||
false_trigger: Optional[bool],
|
||||
limit: int,
|
||||
) -> list[dict]:
|
||||
"""Issue one /db/events call to SFM for one (serial, window) pair."""
|
||||
params: dict[str, str] = {
|
||||
"serial": serial,
|
||||
"from_dt": _iso_utc(from_dt) or "",
|
||||
"to_dt": _iso_utc(to_dt) or "",
|
||||
"limit": str(limit),
|
||||
}
|
||||
if false_trigger is not None:
|
||||
params["false_trigger"] = "true" if false_trigger else "false"
|
||||
|
||||
try:
|
||||
resp = await client.get(f"{SFM_BASE_URL}/db/events", params=params)
|
||||
resp.raise_for_status()
|
||||
except httpx.HTTPError as e:
|
||||
log.warning("SFM /db/events failed for serial=%s: %s", serial, e)
|
||||
return []
|
||||
|
||||
payload = resp.json()
|
||||
events = payload.get("events", []) or []
|
||||
# Strip waveform_blob if present — it's the big per-event binary and we
|
||||
# don't render it in the list view. SFM returns it by default.
|
||||
for ev in events:
|
||||
ev.pop("waveform_blob", None)
|
||||
ev.pop("a5_pickle_filename", None)
|
||||
return events
|
||||
|
||||
|
||||
# ── Public API ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
async def events_for_location(
|
||||
db: Session,
|
||||
location_id: str,
|
||||
*,
|
||||
from_dt: Optional[datetime] = None,
|
||||
to_dt: Optional[datetime] = None,
|
||||
false_trigger: Optional[bool] = None,
|
||||
limit: int = 500,
|
||||
) -> dict:
|
||||
"""Fan out UnitAssignment rows for `location_id` and union SFM events.
|
||||
|
||||
Returns:
|
||||
{
|
||||
"events": [merged event dicts, newest first, capped at limit],
|
||||
"count": total events found across all windows (pre-cap),
|
||||
"stats": {event_count, peak_pvs, peak_pvs_at,
|
||||
last_event, false_trigger_count},
|
||||
"assignments_used": [{unit_id, assigned_at, assigned_until,
|
||||
events_in_window}, ...],
|
||||
}
|
||||
|
||||
The "events outside any assignment window" rule (Phase 1 design decision):
|
||||
events whose timestamp falls outside every assignment window are simply
|
||||
not fetched — we only ask SFM for events inside the intersected windows.
|
||||
Those orphan events surface under the per-unit detail page in Phase 2.
|
||||
"""
|
||||
# 1. Fetch all assignments (active + closed) for the location.
|
||||
assignments = (
|
||||
db.query(UnitAssignment)
|
||||
.filter(UnitAssignment.location_id == location_id)
|
||||
.filter(UnitAssignment.device_type == "seismograph")
|
||||
.order_by(UnitAssignment.assigned_at.asc())
|
||||
.all()
|
||||
)
|
||||
|
||||
if not assignments:
|
||||
return {
|
||||
"events": [],
|
||||
"count": 0,
|
||||
"stats": _empty_stats(),
|
||||
"assignments_used": [],
|
||||
}
|
||||
|
||||
now = datetime.utcnow()
|
||||
|
||||
# 2. For each assignment, compute the effective (start, end) window after
|
||||
# intersecting with the requested filter range. Drop assignments that
|
||||
# don't overlap the filter window.
|
||||
fetch_specs: list[tuple[UnitAssignment, datetime, datetime]] = []
|
||||
for a in assignments:
|
||||
window = _intersect_window(a.assigned_at, a.assigned_until, from_dt, to_dt, now)
|
||||
if window is not None:
|
||||
fetch_specs.append((a, window[0], window[1]))
|
||||
|
||||
if not fetch_specs:
|
||||
return {
|
||||
"events": [],
|
||||
"count": 0,
|
||||
"stats": _empty_stats(),
|
||||
"assignments_used": [
|
||||
{
|
||||
"unit_id": a.unit_id,
|
||||
"assigned_at": _iso_utc(a.assigned_at),
|
||||
"assigned_until": _iso_utc(a.assigned_until),
|
||||
"events_in_window": 0,
|
||||
}
|
||||
for a in assignments
|
||||
],
|
||||
}
|
||||
|
||||
# 3. Concurrent SFM fetches. We over-fetch (up to _SFM_FETCH_CEILING per
|
||||
# window) so summary stats reflect the true peak/last/count across the
|
||||
# full filter window, not just what fits in the user's display limit.
|
||||
# The displayed event list is trimmed to `limit` after merge.
|
||||
async with httpx.AsyncClient(timeout=_SFM_TIMEOUT_SECONDS) as client:
|
||||
per_window_lists = await asyncio.gather(
|
||||
*(
|
||||
_fetch_events_for_serial(
|
||||
client,
|
||||
serial=a.unit_id,
|
||||
from_dt=start,
|
||||
to_dt=end,
|
||||
false_trigger=false_trigger,
|
||||
limit=_SFM_FETCH_CEILING,
|
||||
)
|
||||
for a, start, end in fetch_specs
|
||||
),
|
||||
return_exceptions=False,
|
||||
)
|
||||
|
||||
# 4. Build the per-assignment event counts (transparency for the operator).
|
||||
spec_event_counts: dict[str, int] = {}
|
||||
for (a, _start, _end), evs in zip(fetch_specs, per_window_lists):
|
||||
spec_event_counts[a.id] = len(evs)
|
||||
|
||||
# 5. Union, sort newest-first, cap.
|
||||
merged: list[dict] = []
|
||||
for evs in per_window_lists:
|
||||
merged.extend(evs)
|
||||
merged.sort(key=lambda e: e.get("timestamp") or "", reverse=True)
|
||||
total_count = len(merged)
|
||||
capped = merged[:limit]
|
||||
|
||||
# 6. Compute summary stats over the full merged set (not the capped one).
|
||||
stats = _compute_stats(merged)
|
||||
|
||||
# 7. Build the assignments_used report (every assignment, in chronological
|
||||
# order, with its event count — even ones that fell outside the filter
|
||||
# window so the operator sees them but with count=0).
|
||||
assignments_used = []
|
||||
for a in assignments:
|
||||
assignments_used.append(
|
||||
{
|
||||
"unit_id": a.unit_id,
|
||||
"assignment_id": a.id,
|
||||
"assigned_at": _iso_utc(a.assigned_at),
|
||||
"assigned_until": _iso_utc(a.assigned_until),
|
||||
"events_in_window": spec_event_counts.get(a.id, 0),
|
||||
"status": a.status,
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"events": capped,
|
||||
"count": total_count,
|
||||
"stats": stats,
|
||||
"assignments_used": assignments_used,
|
||||
}
|
||||
|
||||
|
||||
# ── Per-unit (cross-project) view ─────────────────────────────────────────────
|
||||
|
||||
|
||||
async def events_for_unit(
|
||||
db: Session,
|
||||
unit_id: str,
|
||||
*,
|
||||
bucket: str = "all", # "all" | "attributed" | "unattributed"
|
||||
from_dt: Optional[datetime] = None,
|
||||
to_dt: Optional[datetime] = None,
|
||||
false_trigger: Optional[bool] = None,
|
||||
limit: int = 500,
|
||||
) -> dict:
|
||||
"""Return events for a unit annotated with their assignment attribution.
|
||||
|
||||
Unlike events_for_location (which queries SFM per assignment window), this
|
||||
helper queries SFM for ALL events for the serial within the optional
|
||||
[from_dt, to_dt] filter, then walks each event against the unit's
|
||||
UnitAssignment intervals to compute attribution.
|
||||
|
||||
Bucket semantics:
|
||||
- "all": every event, attributed or not
|
||||
- "attributed": events that fall inside at least one assignment window
|
||||
- "unattributed": events with no overlapping assignment (the diagnostic
|
||||
bucket — operator should fix assignment dates to
|
||||
attribute these)
|
||||
|
||||
Each event gets an extra `attribution` field:
|
||||
{assignment_id, location_id, location_name, project_id, project_name,
|
||||
assigned_at, assigned_until} or None
|
||||
|
||||
Unattributed events also get a `nearest_assignment` field with the
|
||||
same shape plus `delta_days` (signed; negative = event before assignment).
|
||||
"""
|
||||
# 1. Pull all assignments for this unit (any device_type — caller has
|
||||
# already filtered by seismograph in the route). Order matters: we
|
||||
# want the earliest-start assignment first so attribution prefers the
|
||||
# chronologically-first overlap when there are simultaneous active
|
||||
# assignments at different locations (rare but possible).
|
||||
assignments = (
|
||||
db.query(UnitAssignment)
|
||||
.filter(UnitAssignment.unit_id == unit_id)
|
||||
.order_by(UnitAssignment.assigned_at.asc())
|
||||
.all()
|
||||
)
|
||||
|
||||
# Resolve location + project names once.
|
||||
loc_ids = {a.location_id for a in assignments}
|
||||
proj_ids = {a.project_id for a in assignments}
|
||||
loc_map = {
|
||||
l.id: l for l in db.query(MonitoringLocation).filter(
|
||||
MonitoringLocation.id.in_(loc_ids)
|
||||
).all()
|
||||
} if loc_ids else {}
|
||||
proj_map = {
|
||||
p.id: p for p in db.query(Project).filter(
|
||||
Project.id.in_(proj_ids)
|
||||
).all()
|
||||
} if proj_ids else {}
|
||||
|
||||
now = datetime.utcnow()
|
||||
|
||||
def _attr_dict(a: UnitAssignment) -> dict:
|
||||
loc = loc_map.get(a.location_id)
|
||||
proj = proj_map.get(a.project_id)
|
||||
return {
|
||||
"assignment_id": a.id,
|
||||
"location_id": a.location_id,
|
||||
"location_name": loc.name if loc else None,
|
||||
"project_id": a.project_id,
|
||||
"project_name": proj.name if proj else None,
|
||||
"assigned_at": _iso_utc(a.assigned_at),
|
||||
"assigned_until": _iso_utc(a.assigned_until),
|
||||
}
|
||||
|
||||
# 2. Fetch all events for this serial in one shot.
|
||||
async with httpx.AsyncClient(timeout=_SFM_TIMEOUT_SECONDS) as client:
|
||||
events = await _fetch_events_for_serial(
|
||||
client,
|
||||
serial=unit_id,
|
||||
from_dt=from_dt or datetime(1970, 1, 1),
|
||||
to_dt=to_dt or now,
|
||||
false_trigger=false_trigger,
|
||||
limit=_SFM_FETCH_CEILING,
|
||||
)
|
||||
|
||||
# 3. For each event, walk the assignment list and find the first
|
||||
# overlapping window. O(N * M) but both are small in practice.
|
||||
for ev in events:
|
||||
ts_str = ev.get("timestamp")
|
||||
if not ts_str:
|
||||
ev["attribution"] = None
|
||||
continue
|
||||
try:
|
||||
# SFM returns ISO with "T" separator; tolerate both.
|
||||
ts = datetime.fromisoformat(ts_str.replace(" ", "T"))
|
||||
except ValueError:
|
||||
ev["attribution"] = None
|
||||
continue
|
||||
|
||||
matched: Optional[UnitAssignment] = None
|
||||
for a in assignments:
|
||||
a_end = a.assigned_until or now
|
||||
if a.assigned_at <= ts <= a_end:
|
||||
matched = a
|
||||
break
|
||||
|
||||
if matched is not None:
|
||||
ev["attribution"] = _attr_dict(matched)
|
||||
else:
|
||||
ev["attribution"] = None
|
||||
# Find the nearest assignment (chronologically) for diagnostic.
|
||||
if assignments:
|
||||
nearest = min(
|
||||
assignments,
|
||||
key=lambda a: min(
|
||||
abs((ts - a.assigned_at).total_seconds()),
|
||||
abs((ts - (a.assigned_until or now)).total_seconds()),
|
||||
),
|
||||
)
|
||||
# Signed delta in days from the nearest boundary
|
||||
# (negative = event BEFORE that boundary).
|
||||
if ts < nearest.assigned_at:
|
||||
delta_seconds = (ts - nearest.assigned_at).total_seconds()
|
||||
elif ts > (nearest.assigned_until or now):
|
||||
delta_seconds = (ts - (nearest.assigned_until or now)).total_seconds()
|
||||
else:
|
||||
delta_seconds = 0
|
||||
ev["nearest_assignment"] = {
|
||||
**_attr_dict(nearest),
|
||||
"delta_days": round(delta_seconds / 86400, 1),
|
||||
}
|
||||
|
||||
# 4. Apply bucket filter.
|
||||
if bucket == "attributed":
|
||||
filtered = [e for e in events if e.get("attribution") is not None]
|
||||
elif bucket == "unattributed":
|
||||
filtered = [e for e in events if e.get("attribution") is None]
|
||||
else:
|
||||
filtered = events
|
||||
|
||||
filtered.sort(key=lambda e: e.get("timestamp") or "", reverse=True)
|
||||
total_count = len(filtered)
|
||||
capped = filtered[:limit]
|
||||
|
||||
# 5. Stats: compute over the ENTIRE event set (not the filtered bucket)
|
||||
# so the unattributed_count tile is always meaningful regardless of
|
||||
# which bucket the operator has selected.
|
||||
base_stats = _compute_stats(events)
|
||||
unattributed_count = sum(
|
||||
1 for e in events if e.get("attribution") is None
|
||||
)
|
||||
base_stats["unattributed_count"] = unattributed_count
|
||||
|
||||
return {
|
||||
"events": capped,
|
||||
"count": total_count,
|
||||
"stats": base_stats,
|
||||
"assignments_total": len(assignments),
|
||||
}
|
||||
|
||||
|
||||
# ── Project-level roll-up (aggregates across all vibration locations) ─────────
|
||||
|
||||
|
||||
async def vibration_summary_for_project(
|
||||
db: Session,
|
||||
project_id: str,
|
||||
*,
|
||||
from_dt: Optional[datetime] = None,
|
||||
to_dt: Optional[datetime] = None,
|
||||
) -> dict:
|
||||
"""Aggregate SFM events across every vibration location in a project.
|
||||
|
||||
Returns:
|
||||
{
|
||||
"project_id": str,
|
||||
"total_events": int,
|
||||
"peak_pvs": float | None,
|
||||
"peak_pvs_at": ISO timestamp | None,
|
||||
"peak_pvs_location_id": str | None,
|
||||
"peak_pvs_location_name": str | None,
|
||||
"last_event": ISO timestamp | None,
|
||||
"false_trigger_count": int,
|
||||
"per_location": [
|
||||
{"location_id", "location_name", "event_count",
|
||||
"peak_pvs", "last_event"},
|
||||
... # sorted by event_count DESC
|
||||
],
|
||||
"vibration_location_count": int,
|
||||
}
|
||||
"""
|
||||
locations = (
|
||||
db.query(MonitoringLocation)
|
||||
.filter(MonitoringLocation.project_id == project_id)
|
||||
.filter(MonitoringLocation.location_type == "vibration")
|
||||
.all()
|
||||
)
|
||||
|
||||
if not locations:
|
||||
return {
|
||||
"project_id": project_id,
|
||||
"total_events": 0,
|
||||
"peak_pvs": None,
|
||||
"peak_pvs_at": None,
|
||||
"peak_pvs_location_id": None,
|
||||
"peak_pvs_location_name": None,
|
||||
"last_event": None,
|
||||
"false_trigger_count": 0,
|
||||
"per_location": [],
|
||||
"vibration_location_count": 0,
|
||||
}
|
||||
|
||||
# Fan out across locations. Each call internally fans out across that
|
||||
# location's UnitAssignment rows, so this is a nested fan-out. Both
|
||||
# tiers happen concurrently because asyncio.gather + httpx pool.
|
||||
results = await asyncio.gather(
|
||||
*(
|
||||
events_for_location(
|
||||
db,
|
||||
loc.id,
|
||||
from_dt=from_dt,
|
||||
to_dt=to_dt,
|
||||
false_trigger=None,
|
||||
limit=1, # We only need stats; events list itself is ignored.
|
||||
)
|
||||
for loc in locations
|
||||
),
|
||||
return_exceptions=False,
|
||||
)
|
||||
|
||||
per_location: list[dict] = []
|
||||
total_events = 0
|
||||
peak_pvs = None
|
||||
peak_pvs_at = None
|
||||
peak_pvs_location_id = None
|
||||
peak_pvs_location_name = None
|
||||
last_event = None
|
||||
false_trigger_count = 0
|
||||
|
||||
for loc, res in zip(locations, results):
|
||||
st = res.get("stats", {}) or {}
|
||||
ec = st.get("event_count", 0) or 0
|
||||
total_events += ec
|
||||
false_trigger_count += st.get("false_trigger_count", 0) or 0
|
||||
|
||||
ev_last = st.get("last_event")
|
||||
if ev_last and (last_event is None or ev_last > last_event):
|
||||
last_event = ev_last
|
||||
|
||||
ev_peak = st.get("peak_pvs")
|
||||
if ev_peak is not None and (peak_pvs is None or ev_peak > peak_pvs):
|
||||
peak_pvs = ev_peak
|
||||
peak_pvs_at = st.get("peak_pvs_at")
|
||||
peak_pvs_location_id = loc.id
|
||||
peak_pvs_location_name = loc.name
|
||||
|
||||
per_location.append({
|
||||
"location_id": loc.id,
|
||||
"location_name": loc.name,
|
||||
"event_count": ec,
|
||||
"peak_pvs": ev_peak,
|
||||
"last_event": ev_last,
|
||||
})
|
||||
|
||||
per_location.sort(key=lambda r: r["event_count"], reverse=True)
|
||||
|
||||
return {
|
||||
"project_id": project_id,
|
||||
"total_events": total_events,
|
||||
"peak_pvs": peak_pvs,
|
||||
"peak_pvs_at": peak_pvs_at,
|
||||
"peak_pvs_location_id": peak_pvs_location_id,
|
||||
"peak_pvs_location_name": peak_pvs_location_name,
|
||||
"last_event": last_event,
|
||||
"false_trigger_count": false_trigger_count,
|
||||
"per_location": per_location,
|
||||
"vibration_location_count": len(locations),
|
||||
}
|
||||
|
||||
|
||||
# ── Stats helpers ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _empty_stats() -> dict:
|
||||
return {
|
||||
"event_count": 0,
|
||||
"peak_pvs": None,
|
||||
"peak_pvs_at": None,
|
||||
"peak_pvs_serial": None,
|
||||
"last_event": None,
|
||||
"false_trigger_count": 0,
|
||||
}
|
||||
|
||||
|
||||
def _compute_stats(events: list[dict]) -> dict:
|
||||
"""Roll up summary stats from a merged event list. Cheap O(N) pass.
|
||||
|
||||
The "Overall Peak" stat (peak_pvs) EXCLUDES events flagged as false
|
||||
triggers — operators care about the highest REAL event, not the
|
||||
biggest sensor glitch. false_trigger_count still includes them so
|
||||
operators can see how many were filtered out. last_event uses
|
||||
every event regardless (it's about activity recency, not magnitude).
|
||||
"""
|
||||
if not events:
|
||||
return _empty_stats()
|
||||
|
||||
peak_pvs = None
|
||||
peak_pvs_at = None
|
||||
peak_pvs_serial = None
|
||||
last_event = None
|
||||
false_trigger_count = 0
|
||||
|
||||
for ev in events:
|
||||
is_false_trigger = bool(ev.get("false_trigger"))
|
||||
if is_false_trigger:
|
||||
false_trigger_count += 1
|
||||
|
||||
# Peak calculation: skip flagged false triggers.
|
||||
if not is_false_trigger:
|
||||
pvs = ev.get("peak_vector_sum")
|
||||
if pvs is not None and (peak_pvs is None or pvs > peak_pvs):
|
||||
peak_pvs = pvs
|
||||
peak_pvs_at = ev.get("timestamp")
|
||||
peak_pvs_serial = ev.get("serial")
|
||||
|
||||
ts = ev.get("timestamp")
|
||||
if ts and (last_event is None or ts > last_event):
|
||||
last_event = ts
|
||||
|
||||
return {
|
||||
"event_count": len(events),
|
||||
"peak_pvs": peak_pvs,
|
||||
"peak_pvs_at": peak_pvs_at,
|
||||
"peak_pvs_serial": peak_pvs_serial,
|
||||
"last_event": last_event,
|
||||
"false_trigger_count": false_trigger_count,
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
"""
|
||||
SLM Status Synchronization Service
|
||||
|
||||
Syncs SLM device status from SLMM backend to Terra-View's Emitter table.
|
||||
This bridges SLMM's polling data with Terra-View's status snapshot system.
|
||||
|
||||
SLMM tracks device reachability via background polling. This service
|
||||
fetches that data and creates/updates Emitter records so SLMs appear
|
||||
correctly in the dashboard status snapshot.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, Any
|
||||
|
||||
from backend.database import get_db_session
|
||||
from backend.models import Emitter
|
||||
from backend.services.slmm_client import get_slmm_client, SLMMClientError
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def sync_slm_status_to_emitters() -> Dict[str, Any]:
|
||||
"""
|
||||
Fetch SLM status from SLMM and sync to Terra-View's Emitter table.
|
||||
|
||||
For each device in SLMM's polling status:
|
||||
- If last_success exists, create/update Emitter with that timestamp
|
||||
- If not reachable, update Emitter with last known timestamp (or None)
|
||||
|
||||
Returns:
|
||||
Dict with synced_count, error_count, errors list
|
||||
"""
|
||||
client = get_slmm_client()
|
||||
synced = 0
|
||||
errors = []
|
||||
|
||||
try:
|
||||
# Get polling status from SLMM
|
||||
status_response = await client.get_polling_status()
|
||||
|
||||
# Handle nested response structure
|
||||
data = status_response.get("data", status_response)
|
||||
devices = data.get("devices", [])
|
||||
|
||||
if not devices:
|
||||
logger.debug("No SLM devices in SLMM polling status")
|
||||
return {"synced_count": 0, "error_count": 0, "errors": []}
|
||||
|
||||
db = get_db_session()
|
||||
try:
|
||||
for device in devices:
|
||||
unit_id = device.get("unit_id")
|
||||
if not unit_id:
|
||||
continue
|
||||
|
||||
try:
|
||||
# Get or create Emitter record
|
||||
emitter = db.query(Emitter).filter(Emitter.id == unit_id).first()
|
||||
|
||||
# Determine last_seen from SLMM data
|
||||
last_success_str = device.get("last_success")
|
||||
is_reachable = device.get("is_reachable", False)
|
||||
|
||||
if last_success_str:
|
||||
# Parse ISO format timestamp
|
||||
last_seen = datetime.fromisoformat(
|
||||
last_success_str.replace("Z", "+00:00")
|
||||
)
|
||||
# Convert to naive UTC for consistency with existing code
|
||||
if last_seen.tzinfo:
|
||||
last_seen = last_seen.astimezone(timezone.utc).replace(tzinfo=None)
|
||||
elif is_reachable:
|
||||
# Device is reachable but no last_success yet (first poll or just started)
|
||||
# Use current time so it shows as OK, not Missing
|
||||
last_seen = datetime.utcnow()
|
||||
else:
|
||||
last_seen = None
|
||||
|
||||
# Status will be recalculated by snapshot.py based on time thresholds
|
||||
# Just store a provisional status here
|
||||
status = "OK" if is_reachable else "Missing"
|
||||
|
||||
# Store last error message if available
|
||||
last_error = device.get("last_error") or ""
|
||||
|
||||
if emitter:
|
||||
# Update existing record
|
||||
emitter.last_seen = last_seen
|
||||
emitter.status = status
|
||||
emitter.unit_type = "slm"
|
||||
emitter.last_file = last_error
|
||||
else:
|
||||
# Create new record
|
||||
emitter = Emitter(
|
||||
id=unit_id,
|
||||
unit_type="slm",
|
||||
last_seen=last_seen,
|
||||
last_file=last_error,
|
||||
status=status
|
||||
)
|
||||
db.add(emitter)
|
||||
|
||||
synced += 1
|
||||
|
||||
except Exception as e:
|
||||
errors.append(f"{unit_id}: {str(e)}")
|
||||
logger.error(f"Error syncing SLM {unit_id}: {e}")
|
||||
|
||||
db.commit()
|
||||
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
if synced > 0:
|
||||
logger.info(f"Synced {synced} SLM device(s) to Emitter table")
|
||||
|
||||
except SLMMClientError as e:
|
||||
logger.warning(f"Could not reach SLMM for status sync: {e}")
|
||||
errors.append(f"SLMM unreachable: {str(e)}")
|
||||
except Exception as e:
|
||||
logger.error(f"Error in SLM status sync: {e}", exc_info=True)
|
||||
errors.append(str(e))
|
||||
|
||||
return {
|
||||
"synced_count": synced,
|
||||
"error_count": len(errors),
|
||||
"errors": errors
|
||||
}
|
||||
@@ -9,13 +9,14 @@ that handles TCP/FTP communication with Rion NL-43/NL-53 devices.
|
||||
"""
|
||||
|
||||
import httpx
|
||||
import os
|
||||
from typing import Optional, Dict, Any, List
|
||||
from datetime import datetime
|
||||
import json
|
||||
|
||||
|
||||
# SLMM backend base URLs
|
||||
SLMM_BASE_URL = "http://localhost:8100"
|
||||
# SLMM backend base URLs - use environment variable if set (for Docker)
|
||||
SLMM_BASE_URL = os.environ.get("SLMM_BASE_URL", "http://localhost:8100")
|
||||
SLMM_API_BASE = f"{SLMM_BASE_URL}/api/nl43"
|
||||
|
||||
|
||||
@@ -108,7 +109,71 @@ class SLMMClient:
|
||||
f"SLMM operation failed: {error_detail}"
|
||||
)
|
||||
except Exception as e:
|
||||
raise SLMMClientError(f"Unexpected error: {str(e)}")
|
||||
error_msg = str(e) if str(e) else type(e).__name__
|
||||
raise SLMMClientError(f"Unexpected error: {error_msg}")
|
||||
|
||||
async def _download_request(
|
||||
self,
|
||||
endpoint: str,
|
||||
data: Dict[str, Any],
|
||||
unit_id: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Make a download request to SLMM that returns binary file content (not JSON).
|
||||
|
||||
Saves the file locally and returns metadata about the download.
|
||||
"""
|
||||
url = f"{self.api_base}{endpoint}"
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=httpx.Timeout(300.0)) as client:
|
||||
response = await client.post(url, json=data)
|
||||
response.raise_for_status()
|
||||
|
||||
# Determine filename from Content-Disposition header or generate one
|
||||
content_disp = response.headers.get("content-disposition", "")
|
||||
filename = None
|
||||
if "filename=" in content_disp:
|
||||
filename = content_disp.split("filename=")[-1].strip('" ')
|
||||
|
||||
if not filename:
|
||||
remote_path = data.get("remote_path", "download")
|
||||
base = os.path.basename(remote_path.rstrip("/"))
|
||||
filename = f"{base}.zip" if not base.endswith(".zip") else base
|
||||
|
||||
# Save to local downloads directory
|
||||
download_dir = os.path.join("data", "downloads", unit_id)
|
||||
os.makedirs(download_dir, exist_ok=True)
|
||||
local_path = os.path.join(download_dir, filename)
|
||||
|
||||
with open(local_path, "wb") as f:
|
||||
f.write(response.content)
|
||||
|
||||
return {
|
||||
"success": True,
|
||||
"local_path": local_path,
|
||||
"filename": filename,
|
||||
"size_bytes": len(response.content),
|
||||
}
|
||||
|
||||
except httpx.ConnectError as e:
|
||||
raise SLMMConnectionError(
|
||||
f"Cannot connect to SLMM backend at {self.base_url}. "
|
||||
f"Is SLMM running? Error: {str(e)}"
|
||||
)
|
||||
except httpx.HTTPStatusError as e:
|
||||
error_detail = "Unknown error"
|
||||
try:
|
||||
error_data = e.response.json()
|
||||
error_detail = error_data.get("detail", str(error_data))
|
||||
except Exception:
|
||||
error_detail = e.response.text or str(e)
|
||||
raise SLMMDeviceError(f"SLMM download failed: {error_detail}")
|
||||
except (SLMMConnectionError, SLMMDeviceError):
|
||||
raise
|
||||
except Exception as e:
|
||||
error_msg = str(e) if str(e) else type(e).__name__
|
||||
raise SLMMClientError(f"Download error: {error_msg}")
|
||||
|
||||
# ========================================================================
|
||||
# Unit Management
|
||||
@@ -276,6 +341,124 @@ class SLMMClient:
|
||||
"""
|
||||
return await self._request("POST", f"/{unit_id}/reset")
|
||||
|
||||
# ========================================================================
|
||||
# Store/Index Management
|
||||
# ========================================================================
|
||||
|
||||
async def get_index_number(self, unit_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get current store/index number from device.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
|
||||
Returns:
|
||||
Dict with current index_number (store name)
|
||||
"""
|
||||
return await self._request("GET", f"/{unit_id}/index-number")
|
||||
|
||||
async def set_index_number(
|
||||
self,
|
||||
unit_id: str,
|
||||
index_number: int,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Set store/index number on device.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
index_number: New index number to set
|
||||
|
||||
Returns:
|
||||
Confirmation response
|
||||
"""
|
||||
return await self._request(
|
||||
"PUT",
|
||||
f"/{unit_id}/index-number",
|
||||
data={"index_number": index_number},
|
||||
)
|
||||
|
||||
async def check_overwrite_status(self, unit_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Check if data exists at the current store index.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
|
||||
Returns:
|
||||
Dict with:
|
||||
- overwrite_status: "None" (safe) or "Exist" (would overwrite)
|
||||
- will_overwrite: bool
|
||||
- safe_to_store: bool
|
||||
"""
|
||||
return await self._request("GET", f"/{unit_id}/overwrite-check")
|
||||
|
||||
async def increment_index(self, unit_id: str, max_attempts: int = 100) -> Dict[str, Any]:
|
||||
"""
|
||||
Find and set the next available (unused) store/index number.
|
||||
|
||||
Checks the current index - if it would overwrite existing data,
|
||||
increments until finding an unused index number.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
max_attempts: Maximum number of indices to try before giving up
|
||||
|
||||
Returns:
|
||||
Dict with old_index, new_index, and attempts_made
|
||||
"""
|
||||
# Get current index
|
||||
current = await self.get_index_number(unit_id)
|
||||
old_index = current.get("index_number", 0)
|
||||
|
||||
# Check if current index is safe
|
||||
overwrite_check = await self.check_overwrite_status(unit_id)
|
||||
if overwrite_check.get("safe_to_store", False):
|
||||
# Current index is safe, no need to increment
|
||||
return {
|
||||
"success": True,
|
||||
"old_index": old_index,
|
||||
"new_index": old_index,
|
||||
"unit_id": unit_id,
|
||||
"already_safe": True,
|
||||
"attempts_made": 0,
|
||||
}
|
||||
|
||||
# Need to find an unused index
|
||||
attempts = 0
|
||||
test_index = old_index + 1
|
||||
|
||||
while attempts < max_attempts:
|
||||
# Set the new index
|
||||
await self.set_index_number(unit_id, test_index)
|
||||
|
||||
# Check if this index is safe
|
||||
overwrite_check = await self.check_overwrite_status(unit_id)
|
||||
attempts += 1
|
||||
|
||||
if overwrite_check.get("safe_to_store", False):
|
||||
return {
|
||||
"success": True,
|
||||
"old_index": old_index,
|
||||
"new_index": test_index,
|
||||
"unit_id": unit_id,
|
||||
"already_safe": False,
|
||||
"attempts_made": attempts,
|
||||
}
|
||||
|
||||
# Try next index (wrap around at 9999)
|
||||
test_index = (test_index + 1) % 10000
|
||||
|
||||
# Avoid infinite loops if we've wrapped around
|
||||
if test_index == old_index:
|
||||
break
|
||||
|
||||
# Could not find a safe index
|
||||
raise SLMMDeviceError(
|
||||
f"Could not find unused store index for {unit_id} after {attempts} attempts. "
|
||||
f"Consider downloading and clearing data from the device."
|
||||
)
|
||||
|
||||
# ========================================================================
|
||||
# Device Settings
|
||||
# ========================================================================
|
||||
@@ -359,9 +542,130 @@ class SLMMClient:
|
||||
return await self._request("GET", f"/{unit_id}/settings")
|
||||
|
||||
# ========================================================================
|
||||
# Data Download (Future)
|
||||
# FTP Control
|
||||
# ========================================================================
|
||||
|
||||
async def enable_ftp(self, unit_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Enable FTP server on device.
|
||||
|
||||
Must be called before downloading files. FTP and TCP can work in tandem.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
|
||||
Returns:
|
||||
Dict with status message
|
||||
"""
|
||||
return await self._request("POST", f"/{unit_id}/ftp/enable")
|
||||
|
||||
async def disable_ftp(self, unit_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Disable FTP server on device.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
|
||||
Returns:
|
||||
Dict with status message
|
||||
"""
|
||||
return await self._request("POST", f"/{unit_id}/ftp/disable")
|
||||
|
||||
async def get_ftp_status(self, unit_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get FTP server status on device.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
|
||||
Returns:
|
||||
Dict with ftp_enabled status
|
||||
"""
|
||||
return await self._request("GET", f"/{unit_id}/ftp/status")
|
||||
|
||||
# ========================================================================
|
||||
# Data Download
|
||||
# ========================================================================
|
||||
|
||||
async def download_file(
|
||||
self,
|
||||
unit_id: str,
|
||||
remote_path: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Download a single file from unit via FTP.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
remote_path: Path on device to download (e.g., "/NL43_DATA/measurement.wav")
|
||||
|
||||
Returns:
|
||||
Dict with local_path, filename, size_bytes
|
||||
"""
|
||||
return await self._download_request(
|
||||
f"/{unit_id}/ftp/download",
|
||||
{"remote_path": remote_path},
|
||||
unit_id,
|
||||
)
|
||||
|
||||
async def download_folder(
|
||||
self,
|
||||
unit_id: str,
|
||||
remote_path: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Download an entire folder from unit via FTP as a ZIP archive.
|
||||
|
||||
Useful for downloading complete measurement sessions (e.g., Auto_0000 folders).
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
remote_path: Folder path on device to download (e.g., "/NL43_DATA/Auto_0000")
|
||||
|
||||
Returns:
|
||||
Dict with local_path, folder_name, size_bytes
|
||||
"""
|
||||
return await self._download_request(
|
||||
f"/{unit_id}/ftp/download-folder",
|
||||
{"remote_path": remote_path},
|
||||
unit_id,
|
||||
)
|
||||
|
||||
async def download_current_measurement(
|
||||
self,
|
||||
unit_id: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Download the current measurement folder based on device's index number.
|
||||
|
||||
This is the recommended method for scheduled downloads - it automatically
|
||||
determines which folder to download based on the device's current store index.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
|
||||
Returns:
|
||||
Dict with local_path, folder_name, file_count, zip_size_bytes, index_number
|
||||
"""
|
||||
# Get current index number from device
|
||||
index_info = await self.get_index_number(unit_id)
|
||||
index_number_raw = index_info.get("index_number", 0)
|
||||
|
||||
# Convert to int - device returns string like "0000" or "0001"
|
||||
try:
|
||||
index_number = int(index_number_raw)
|
||||
except (ValueError, TypeError):
|
||||
index_number = 0
|
||||
|
||||
# Format as Auto_XXXX folder name
|
||||
folder_name = f"Auto_{index_number:04d}"
|
||||
remote_path = f"/NL-43/{folder_name}"
|
||||
|
||||
# Download the folder
|
||||
result = await self.download_folder(unit_id, remote_path)
|
||||
result["index_number"] = index_number
|
||||
return result
|
||||
|
||||
async def download_files(
|
||||
self,
|
||||
unit_id: str,
|
||||
@@ -369,23 +673,153 @@ class SLMMClient:
|
||||
files: Optional[List[str]] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Download files from unit via FTP.
|
||||
Download measurement files from unit via FTP.
|
||||
|
||||
NOTE: This endpoint doesn't exist in SLMM yet. Will need to implement.
|
||||
This method automatically determines the current measurement folder and downloads it.
|
||||
The destination_path parameter is logged for reference but actual download location
|
||||
is managed by SLMM (data/downloads/{unit_id}/).
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
destination_path: Local path to save files
|
||||
files: List of filenames to download, or None for all
|
||||
destination_path: Reference path (for logging/metadata, not used by SLMM)
|
||||
files: Ignored - always downloads the current measurement folder
|
||||
|
||||
Returns:
|
||||
Dict with downloaded files list and metadata
|
||||
Dict with download result including local_path, folder_name, etc.
|
||||
"""
|
||||
data = {
|
||||
"destination_path": destination_path,
|
||||
"files": files or "all",
|
||||
}
|
||||
return await self._request("POST", f"/{unit_id}/ftp/download", data=data)
|
||||
# Use the new method that automatically determines what to download
|
||||
result = await self.download_current_measurement(unit_id)
|
||||
result["requested_destination"] = destination_path
|
||||
return result
|
||||
|
||||
# ========================================================================
|
||||
# Cycle Commands (for scheduled automation)
|
||||
# ========================================================================
|
||||
|
||||
async def start_cycle(
|
||||
self,
|
||||
unit_id: str,
|
||||
sync_clock: bool = True,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Execute complete start cycle on device via SLMM.
|
||||
|
||||
This handles the full pre-recording workflow:
|
||||
1. Sync device clock to server time
|
||||
2. Find next safe index (with overwrite protection)
|
||||
3. Start measurement
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
sync_clock: Whether to sync device clock to server time
|
||||
|
||||
Returns:
|
||||
Dict with clock_synced, old_index, new_index, started, etc.
|
||||
"""
|
||||
return await self._request(
|
||||
"POST",
|
||||
f"/{unit_id}/start-cycle",
|
||||
data={"sync_clock": sync_clock},
|
||||
)
|
||||
|
||||
async def stop_cycle(
|
||||
self,
|
||||
unit_id: str,
|
||||
download: bool = True,
|
||||
download_path: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Execute complete stop cycle on device via SLMM.
|
||||
|
||||
This handles the full post-recording workflow:
|
||||
1. Stop measurement
|
||||
2. Enable FTP
|
||||
3. Download measurement folder (if download=True)
|
||||
4. Verify download
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
download: Whether to download measurement data
|
||||
download_path: Custom path for downloaded ZIP (optional)
|
||||
|
||||
Returns:
|
||||
Dict with stopped, ftp_enabled, download_success, local_path, etc.
|
||||
"""
|
||||
data = {"download": download}
|
||||
if download_path:
|
||||
data["download_path"] = download_path
|
||||
return await self._request(
|
||||
"POST",
|
||||
f"/{unit_id}/stop-cycle",
|
||||
data=data,
|
||||
)
|
||||
|
||||
# ========================================================================
|
||||
# Polling Status (for device monitoring/alerts)
|
||||
# ========================================================================
|
||||
|
||||
async def get_polling_status(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Get global polling status from SLMM.
|
||||
|
||||
Returns device reachability information for all polled devices.
|
||||
Used by DeviceStatusMonitor to detect offline/online transitions.
|
||||
|
||||
Returns:
|
||||
Dict with devices list containing:
|
||||
- unit_id
|
||||
- is_reachable
|
||||
- consecutive_failures
|
||||
- last_poll_attempt
|
||||
- last_success
|
||||
- last_error
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=self.timeout) as client:
|
||||
response = await client.get(f"{self.base_url}/api/nl43/_polling/status")
|
||||
response.raise_for_status()
|
||||
return response.json()
|
||||
except httpx.ConnectError:
|
||||
raise SLMMConnectionError("Cannot connect to SLMM for polling status")
|
||||
except Exception as e:
|
||||
raise SLMMClientError(f"Failed to get polling status: {str(e)}")
|
||||
|
||||
async def get_device_polling_config(self, unit_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get polling configuration for a specific device.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
|
||||
Returns:
|
||||
Dict with poll_enabled and poll_interval_seconds
|
||||
"""
|
||||
return await self._request("GET", f"/{unit_id}/polling/config")
|
||||
|
||||
async def update_device_polling_config(
|
||||
self,
|
||||
unit_id: str,
|
||||
poll_enabled: Optional[bool] = None,
|
||||
poll_interval_seconds: Optional[int] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Update polling configuration for a device.
|
||||
|
||||
Args:
|
||||
unit_id: Unit identifier
|
||||
poll_enabled: Enable/disable polling
|
||||
poll_interval_seconds: Polling interval (10-3600)
|
||||
|
||||
Returns:
|
||||
Updated config
|
||||
"""
|
||||
config = {}
|
||||
if poll_enabled is not None:
|
||||
config["poll_enabled"] = poll_enabled
|
||||
if poll_interval_seconds is not None:
|
||||
config["poll_interval_seconds"] = poll_interval_seconds
|
||||
|
||||
return await self._request("PUT", f"/{unit_id}/polling/config", data=config)
|
||||
|
||||
# ========================================================================
|
||||
# Health Check
|
||||
|
||||
@@ -0,0 +1,231 @@
|
||||
"""
|
||||
SLMM Synchronization Service
|
||||
|
||||
This service ensures Terra-View roster is the single source of truth for SLM device configuration.
|
||||
When SLM devices are added, edited, or deleted in Terra-View, changes are automatically synced to SLMM.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import httpx
|
||||
import os
|
||||
from typing import Optional
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from backend.models import RosterUnit
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SLMM_BASE_URL = os.getenv("SLMM_BASE_URL", "http://localhost:8100")
|
||||
|
||||
|
||||
async def sync_slm_to_slmm(unit: RosterUnit) -> bool:
|
||||
"""
|
||||
Sync a single SLM device from Terra-View roster to SLMM.
|
||||
|
||||
Args:
|
||||
unit: RosterUnit with device_type="slm"
|
||||
|
||||
Returns:
|
||||
True if sync successful, False otherwise
|
||||
"""
|
||||
if unit.device_type != "slm":
|
||||
logger.warning(f"Attempted to sync non-SLM unit {unit.id} to SLMM")
|
||||
return False
|
||||
|
||||
if not unit.slm_host:
|
||||
logger.warning(f"SLM {unit.id} has no host configured, skipping SLMM sync")
|
||||
return False
|
||||
|
||||
# Disable polling if unit is benched (deployed=False) or retired
|
||||
# Only actively deployed units should be polled
|
||||
should_poll = unit.deployed and not unit.retired
|
||||
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
response = await client.put(
|
||||
f"{SLMM_BASE_URL}/api/nl43/{unit.id}/config",
|
||||
json={
|
||||
"host": unit.slm_host,
|
||||
"tcp_port": unit.slm_tcp_port or 2255,
|
||||
"tcp_enabled": True,
|
||||
"ftp_enabled": True,
|
||||
"ftp_username": "USER", # Default NL43 credentials
|
||||
"ftp_password": "0000",
|
||||
"poll_enabled": should_poll, # Disable polling for benched or retired units
|
||||
"poll_interval_seconds": 3600, # Default to 1 hour polling
|
||||
}
|
||||
)
|
||||
|
||||
if response.status_code in [200, 201]:
|
||||
logger.info(f"✓ Synced SLM {unit.id} to SLMM at {unit.slm_host}:{unit.slm_tcp_port or 2255}")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"Failed to sync SLM {unit.id} to SLMM: {response.status_code} {response.text}")
|
||||
return False
|
||||
|
||||
except httpx.TimeoutException:
|
||||
logger.error(f"Timeout syncing SLM {unit.id} to SLMM")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Error syncing SLM {unit.id} to SLMM: {e}")
|
||||
return False
|
||||
|
||||
|
||||
async def delete_slm_from_slmm(unit_id: str) -> bool:
|
||||
"""
|
||||
Delete a device from SLMM database.
|
||||
|
||||
Args:
|
||||
unit_id: The unit ID to delete
|
||||
|
||||
Returns:
|
||||
True if deletion successful or device doesn't exist, False on error
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
response = await client.delete(
|
||||
f"{SLMM_BASE_URL}/api/nl43/{unit_id}/config"
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
logger.info(f"✓ Deleted SLM {unit_id} from SLMM")
|
||||
return True
|
||||
elif response.status_code == 404:
|
||||
logger.info(f"SLM {unit_id} not found in SLMM (already deleted)")
|
||||
return True
|
||||
else:
|
||||
logger.error(f"Failed to delete SLM {unit_id} from SLMM: {response.status_code} {response.text}")
|
||||
return False
|
||||
|
||||
except httpx.TimeoutException:
|
||||
logger.error(f"Timeout deleting SLM {unit_id} from SLMM")
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.error(f"Error deleting SLM {unit_id} from SLMM: {e}")
|
||||
return False
|
||||
|
||||
|
||||
async def sync_all_slms_to_slmm(db: Session) -> dict:
|
||||
"""
|
||||
Sync all SLM devices from Terra-View roster to SLMM.
|
||||
|
||||
This ensures SLMM database matches Terra-View roster as the source of truth.
|
||||
Should be called on Terra-View startup and optionally via admin endpoint.
|
||||
|
||||
Args:
|
||||
db: Database session
|
||||
|
||||
Returns:
|
||||
Dictionary with sync results
|
||||
"""
|
||||
logger.info("Starting full SLM sync to SLMM...")
|
||||
|
||||
# Get all SLM units from roster
|
||||
slm_units = db.query(RosterUnit).filter_by(device_type="slm").all()
|
||||
|
||||
results = {
|
||||
"total": len(slm_units),
|
||||
"synced": 0,
|
||||
"skipped": 0,
|
||||
"failed": 0
|
||||
}
|
||||
|
||||
for unit in slm_units:
|
||||
# Skip units without host configured
|
||||
if not unit.slm_host:
|
||||
results["skipped"] += 1
|
||||
logger.debug(f"Skipped {unit.unit_type} - no host configured")
|
||||
continue
|
||||
|
||||
# Sync to SLMM
|
||||
success = await sync_slm_to_slmm(unit)
|
||||
if success:
|
||||
results["synced"] += 1
|
||||
else:
|
||||
results["failed"] += 1
|
||||
|
||||
logger.info(
|
||||
f"SLM sync complete: {results['synced']} synced, "
|
||||
f"{results['skipped']} skipped, {results['failed']} failed"
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
|
||||
async def get_slmm_devices() -> Optional[list]:
|
||||
"""
|
||||
Get list of all devices currently in SLMM database.
|
||||
|
||||
Returns:
|
||||
List of device unit_ids, or None on error
|
||||
"""
|
||||
try:
|
||||
async with httpx.AsyncClient(timeout=5.0) as client:
|
||||
response = await client.get(f"{SLMM_BASE_URL}/api/nl43/_polling/status")
|
||||
|
||||
if response.status_code == 200:
|
||||
data = response.json()
|
||||
return [device["unit_id"] for device in data["data"]["devices"]]
|
||||
else:
|
||||
logger.error(f"Failed to get SLMM devices: {response.status_code}")
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error getting SLMM devices: {e}")
|
||||
return None
|
||||
|
||||
|
||||
async def cleanup_orphaned_slmm_devices(db: Session) -> dict:
|
||||
"""
|
||||
Remove devices from SLMM that are not in Terra-View roster.
|
||||
|
||||
This cleans up orphaned test devices or devices that were manually added to SLMM.
|
||||
|
||||
Args:
|
||||
db: Database session
|
||||
|
||||
Returns:
|
||||
Dictionary with cleanup results
|
||||
"""
|
||||
logger.info("Checking for orphaned devices in SLMM...")
|
||||
|
||||
# Get all device IDs from SLMM
|
||||
slmm_devices = await get_slmm_devices()
|
||||
if slmm_devices is None:
|
||||
return {"error": "Failed to get SLMM device list"}
|
||||
|
||||
# Get all SLM unit IDs from Terra-View roster
|
||||
roster_units = db.query(RosterUnit.id).filter_by(device_type="slm").all()
|
||||
roster_unit_ids = {unit.id for unit in roster_units}
|
||||
|
||||
# Find orphaned devices (in SLMM but not in roster)
|
||||
orphaned = [uid for uid in slmm_devices if uid not in roster_unit_ids]
|
||||
|
||||
results = {
|
||||
"total_in_slmm": len(slmm_devices),
|
||||
"total_in_roster": len(roster_unit_ids),
|
||||
"orphaned": len(orphaned),
|
||||
"deleted": 0,
|
||||
"failed": 0,
|
||||
"orphaned_devices": orphaned
|
||||
}
|
||||
|
||||
if not orphaned:
|
||||
logger.info("No orphaned devices found in SLMM")
|
||||
return results
|
||||
|
||||
logger.info(f"Found {len(orphaned)} orphaned devices in SLMM: {orphaned}")
|
||||
|
||||
# Delete orphaned devices
|
||||
for unit_id in orphaned:
|
||||
success = await delete_slm_from_slmm(unit_id)
|
||||
if success:
|
||||
results["deleted"] += 1
|
||||
else:
|
||||
results["failed"] += 1
|
||||
|
||||
logger.info(
|
||||
f"Cleanup complete: {results['deleted']} deleted, {results['failed']} failed"
|
||||
)
|
||||
|
||||
return results
|
||||
@@ -1,9 +1,77 @@
|
||||
from datetime import datetime, timezone
|
||||
import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from typing import Optional
|
||||
|
||||
import httpx
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from backend.database import get_db_session
|
||||
from backend.models import Emitter, RosterUnit, IgnoredUnit
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
SFM_BASE_URL = os.getenv("SFM_BASE_URL", "http://localhost:8200")
|
||||
|
||||
# Tiny module-level cache: /api/status-snapshot is polled every 10s by the
|
||||
# dashboard, and we don't want to hammer SFM with one /db/units roundtrip per
|
||||
# call. 15s TTL keeps the cache mostly hot, with occasional refreshes.
|
||||
_SFM_CACHE_TTL_SECONDS = 15.0
|
||||
_sfm_cache_lock = threading.Lock()
|
||||
_sfm_cache: dict = {"fetched_at": 0.0, "data": None, "reachable": False}
|
||||
|
||||
|
||||
def _parse_sfm_timestamp(ts_str: Optional[str]) -> Optional[datetime]:
|
||||
"""SFM /db/units returns naive ISO timestamps (no tz suffix). Treat them
|
||||
as UTC, mirroring how the watcher heartbeat stores Emitter.last_seen."""
|
||||
if not ts_str:
|
||||
return None
|
||||
try:
|
||||
ts = datetime.fromisoformat(ts_str.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return None
|
||||
if ts.tzinfo is None:
|
||||
ts = ts.replace(tzinfo=timezone.utc)
|
||||
return ts
|
||||
|
||||
|
||||
def fetch_sfm_unit_last_seen() -> tuple[dict[str, datetime], bool]:
|
||||
"""Return ({serial: last_seen_utc}, sfm_reachable).
|
||||
|
||||
Cached for _SFM_CACHE_TTL_SECONDS. On any HTTP error returns ({}, False)
|
||||
so callers transparently fall back to the watcher-heartbeat path.
|
||||
"""
|
||||
now = time.monotonic()
|
||||
with _sfm_cache_lock:
|
||||
if _sfm_cache["data"] is not None and (now - _sfm_cache["fetched_at"]) < _SFM_CACHE_TTL_SECONDS:
|
||||
return _sfm_cache["data"], _sfm_cache["reachable"]
|
||||
|
||||
data: dict[str, datetime] = {}
|
||||
reachable = False
|
||||
try:
|
||||
with httpx.Client(timeout=4.0) as client:
|
||||
resp = client.get(f"{SFM_BASE_URL}/db/units")
|
||||
resp.raise_for_status()
|
||||
payload = resp.json() or []
|
||||
for row in payload:
|
||||
serial = row.get("serial")
|
||||
ts = _parse_sfm_timestamp(row.get("last_seen"))
|
||||
if serial and ts is not None:
|
||||
data[serial] = ts
|
||||
reachable = True
|
||||
except httpx.HTTPError as e:
|
||||
log.warning("SFM /db/units unreachable for status snapshot: %s", e)
|
||||
except Exception as e: # noqa: BLE001 — defensive against malformed payload
|
||||
log.warning("SFM /db/units parse error: %s", e)
|
||||
|
||||
with _sfm_cache_lock:
|
||||
_sfm_cache["fetched_at"] = now
|
||||
_sfm_cache["data"] = data
|
||||
_sfm_cache["reachable"] = reachable
|
||||
return data, reachable
|
||||
|
||||
|
||||
def ensure_utc(dt):
|
||||
if dt is None:
|
||||
@@ -69,6 +137,11 @@ def emit_status_snapshot():
|
||||
emitters = {e.id: e for e in db.query(Emitter).all()}
|
||||
ignored = {i.id for i in db.query(IgnoredUnit).all()}
|
||||
|
||||
# SFM event-forwards are now the primary "last seen" signal for
|
||||
# seismographs. Watcher heartbeats stay as a backup — if SFM is down
|
||||
# or hasn't seen a serial, we fall back to Emitter.last_seen.
|
||||
sfm_last_seen_map, sfm_reachable = fetch_sfm_unit_last_seen()
|
||||
|
||||
units = {}
|
||||
|
||||
# --- Merge roster entries first ---
|
||||
@@ -80,34 +153,75 @@ def emit_status_snapshot():
|
||||
age = "N/A"
|
||||
last_seen = None
|
||||
fname = ""
|
||||
elif r.out_for_calibration:
|
||||
# Out for calibration units get separated later
|
||||
status = "Out for Calibration"
|
||||
age = "N/A"
|
||||
last_seen = None
|
||||
fname = ""
|
||||
elif getattr(r, 'allocated', False) and not r.deployed:
|
||||
# Allocated: staged for an upcoming job, not yet physically deployed
|
||||
status = "Allocated"
|
||||
age = "N/A"
|
||||
last_seen = None
|
||||
fname = ""
|
||||
else:
|
||||
if e:
|
||||
last_seen = ensure_utc(e.last_seen)
|
||||
# RECALCULATE status based on current time, not stored value
|
||||
device_type = r.device_type or "seismograph"
|
||||
emitter_last_seen = ensure_utc(e.last_seen) if e else None
|
||||
fname = e.last_file if e else ""
|
||||
|
||||
# SFM-primary, heartbeat-backup logic — only for seismographs.
|
||||
# (SLMs / modems aren't forwarded into SFM's events store.)
|
||||
sfm_last_seen = sfm_last_seen_map.get(unit_id) if device_type == "seismograph" else None
|
||||
|
||||
if sfm_last_seen and emitter_last_seen:
|
||||
# Both sources reported — use whichever is more recent.
|
||||
if sfm_last_seen >= emitter_last_seen:
|
||||
last_seen = sfm_last_seen
|
||||
last_seen_source = "sfm"
|
||||
else:
|
||||
last_seen = emitter_last_seen
|
||||
last_seen_source = "heartbeat"
|
||||
elif sfm_last_seen:
|
||||
last_seen = sfm_last_seen
|
||||
last_seen_source = "sfm"
|
||||
elif emitter_last_seen:
|
||||
last_seen = emitter_last_seen
|
||||
# If SFM was reachable but doesn't have this serial, it
|
||||
# means the unit is calling home to the watcher but not
|
||||
# being forwarded — still a working state for now.
|
||||
last_seen_source = "heartbeat"
|
||||
else:
|
||||
last_seen = None
|
||||
last_seen_source = "none"
|
||||
|
||||
if last_seen is not None:
|
||||
status = calculate_status(last_seen, status_ok_threshold, status_pending_threshold)
|
||||
age = format_age(last_seen)
|
||||
fname = e.last_file
|
||||
else:
|
||||
# Rostered but no emitter data
|
||||
status = "Missing"
|
||||
last_seen = None
|
||||
age = "N/A"
|
||||
fname = ""
|
||||
|
||||
units[unit_id] = {
|
||||
"id": unit_id,
|
||||
"status": status,
|
||||
"age": age,
|
||||
"last": last_seen.isoformat() if last_seen else None,
|
||||
"last_seen_source": last_seen_source,
|
||||
"sfm_reachable": sfm_reachable,
|
||||
"fname": fname,
|
||||
"deployed": r.deployed,
|
||||
"note": r.note or "",
|
||||
"retired": r.retired,
|
||||
"out_for_calibration": r.out_for_calibration or False,
|
||||
"allocated": getattr(r, 'allocated', False) or False,
|
||||
"allocated_to_project_id": getattr(r, 'allocated_to_project_id', None) or "",
|
||||
# Device type and type-specific fields
|
||||
"device_type": r.device_type or "seismograph",
|
||||
"last_calibrated": r.last_calibrated.isoformat() if r.last_calibrated else None,
|
||||
"next_calibration_due": r.next_calibration_due.isoformat() if r.next_calibration_due else None,
|
||||
"deployed_with_modem_id": r.deployed_with_modem_id,
|
||||
"deployed_with_unit_id": r.deployed_with_unit_id,
|
||||
"ip_address": r.ip_address,
|
||||
"phone_number": r.phone_number,
|
||||
"hardware_model": r.hardware_model,
|
||||
@@ -120,23 +234,36 @@ def emit_status_snapshot():
|
||||
# --- Add unexpected emitter-only units ---
|
||||
for unit_id, e in emitters.items():
|
||||
if unit_id not in roster:
|
||||
last_seen = ensure_utc(e.last_seen)
|
||||
emitter_last_seen = ensure_utc(e.last_seen)
|
||||
sfm_last_seen = sfm_last_seen_map.get(unit_id)
|
||||
if sfm_last_seen and (not emitter_last_seen or sfm_last_seen >= emitter_last_seen):
|
||||
last_seen = sfm_last_seen
|
||||
last_seen_source = "sfm"
|
||||
else:
|
||||
last_seen = emitter_last_seen
|
||||
last_seen_source = "heartbeat"
|
||||
# RECALCULATE status for unknown units too
|
||||
status = calculate_status(last_seen, status_ok_threshold, status_pending_threshold)
|
||||
units[unit_id] = {
|
||||
"id": unit_id,
|
||||
"status": status,
|
||||
"age": format_age(last_seen),
|
||||
"last": last_seen.isoformat(),
|
||||
"last": last_seen.isoformat() if last_seen else None,
|
||||
"last_seen_source": last_seen_source,
|
||||
"sfm_reachable": sfm_reachable,
|
||||
"fname": e.last_file,
|
||||
"deployed": False, # default
|
||||
"note": "",
|
||||
"retired": False,
|
||||
"out_for_calibration": False,
|
||||
"allocated": False,
|
||||
"allocated_to_project_id": "",
|
||||
# Device type and type-specific fields (defaults for unknown units)
|
||||
"device_type": "seismograph", # default
|
||||
"last_calibrated": None,
|
||||
"next_calibration_due": None,
|
||||
"deployed_with_modem_id": None,
|
||||
"deployed_with_unit_id": None,
|
||||
"ip_address": None,
|
||||
"phone_number": None,
|
||||
"hardware_model": None,
|
||||
@@ -146,15 +273,49 @@ def emit_status_snapshot():
|
||||
"coordinates": "",
|
||||
}
|
||||
|
||||
# --- Derive modem status from paired devices ---
|
||||
# Modems don't have their own check-in system, so we inherit status
|
||||
# from whatever device they're paired with (seismograph or SLM)
|
||||
# Check both directions: modem.deployed_with_unit_id OR device.deployed_with_modem_id
|
||||
for unit_id, unit_data in units.items():
|
||||
if unit_data.get("device_type") == "modem" and not unit_data.get("retired"):
|
||||
paired_unit_id = None
|
||||
roster_unit = roster.get(unit_id)
|
||||
|
||||
# First, check if modem has deployed_with_unit_id set
|
||||
if roster_unit and roster_unit.deployed_with_unit_id:
|
||||
paired_unit_id = roster_unit.deployed_with_unit_id
|
||||
else:
|
||||
# Fallback: check if any device has this modem in deployed_with_modem_id
|
||||
for other_id, other_roster in roster.items():
|
||||
if other_roster.deployed_with_modem_id == unit_id:
|
||||
paired_unit_id = other_id
|
||||
break
|
||||
|
||||
if paired_unit_id:
|
||||
paired_unit = units.get(paired_unit_id)
|
||||
if paired_unit:
|
||||
# Inherit status from paired device
|
||||
unit_data["status"] = paired_unit.get("status", "Missing")
|
||||
unit_data["age"] = paired_unit.get("age", "N/A")
|
||||
unit_data["last"] = paired_unit.get("last")
|
||||
unit_data["last_seen_source"] = paired_unit.get("last_seen_source", "none")
|
||||
unit_data["derived_from"] = paired_unit_id
|
||||
|
||||
# Separate buckets for UI
|
||||
active_units = {
|
||||
uid: u for uid, u in units.items()
|
||||
if not u["retired"] and u["deployed"] and uid not in ignored
|
||||
if not u["retired"] and not u["out_for_calibration"] and u["deployed"] and uid not in ignored
|
||||
}
|
||||
|
||||
benched_units = {
|
||||
uid: u for uid, u in units.items()
|
||||
if not u["retired"] and not u["deployed"] and uid not in ignored
|
||||
if not u["retired"] and not u["out_for_calibration"] and not u["allocated"] and not u["deployed"] and uid not in ignored
|
||||
}
|
||||
|
||||
allocated_units = {
|
||||
uid: u for uid, u in units.items()
|
||||
if not u["retired"] and not u["out_for_calibration"] and u["allocated"] and not u["deployed"] and uid not in ignored
|
||||
}
|
||||
|
||||
retired_units = {
|
||||
@@ -162,6 +323,11 @@ def emit_status_snapshot():
|
||||
if u["retired"]
|
||||
}
|
||||
|
||||
out_for_calibration_units = {
|
||||
uid: u for uid, u in units.items()
|
||||
if u["out_for_calibration"]
|
||||
}
|
||||
|
||||
# Unknown units - emitters that aren't in the roster and aren't ignored
|
||||
unknown_units = {
|
||||
uid: u for uid, u in units.items()
|
||||
@@ -173,13 +339,17 @@ def emit_status_snapshot():
|
||||
"units": units,
|
||||
"active": active_units,
|
||||
"benched": benched_units,
|
||||
"allocated": allocated_units,
|
||||
"retired": retired_units,
|
||||
"out_for_calibration": out_for_calibration_units,
|
||||
"unknown": unknown_units,
|
||||
"summary": {
|
||||
"total": len(active_units) + len(benched_units),
|
||||
"total": len(active_units) + len(benched_units) + len(allocated_units),
|
||||
"active": len(active_units),
|
||||
"benched": len(benched_units),
|
||||
"allocated": len(allocated_units),
|
||||
"retired": len(retired_units),
|
||||
"out_for_calibration": len(out_for_calibration_units),
|
||||
"unknown": len(unknown_units),
|
||||
# Status counts only for deployed units (active_units)
|
||||
"ok": sum(1 for u in active_units.values() if u["status"] == "OK"),
|
||||
|
||||
@@ -0,0 +1,401 @@
|
||||
/* event-modal.js — shared event-detail modal.
|
||||
*
|
||||
* Used by:
|
||||
* - /sfm (admin Events tab)
|
||||
* - /projects/{p}/nrl/{l} (project-location Events tab)
|
||||
* - /unit/{id} (unit-detail SFM Events table)
|
||||
*
|
||||
* Pages must include partials/event_detail_modal.html in the body
|
||||
* before this script is loaded.
|
||||
*
|
||||
* Public API:
|
||||
* showEventDetail(eventId)
|
||||
* Open the modal and fetch /api/sfm/db/events/{id}/sidecar to
|
||||
* populate the rich BW report fields (peaks, ZC freq, sensor
|
||||
* self-check, device info, etc.) into a tabbed/sectioned view.
|
||||
*
|
||||
* closeEventDetailModal()
|
||||
* Close the modal.
|
||||
*
|
||||
* Notes:
|
||||
* - Fetches sidecar live from SFM via terra-view's /api/sfm proxy.
|
||||
* - Renders gracefully when the sidecar lacks a bw_report block
|
||||
* (older events forwarded before the _ASCII.TXT pairing fix).
|
||||
* - All functions are global on window so inline onclick handlers
|
||||
* can reach them across all three host pages.
|
||||
*/
|
||||
|
||||
(function () {
|
||||
const MODAL_ID = 'event-detail-modal';
|
||||
|
||||
function _esc(s) {
|
||||
if (s == null) return '';
|
||||
return String(s).replace(/&/g, '&')
|
||||
.replace(/</g, '<')
|
||||
.replace(/>/g, '>')
|
||||
.replace(/"/g, '"');
|
||||
}
|
||||
|
||||
function _fmt(v, digits = 4, suffix = '') {
|
||||
if (v == null || (typeof v === 'number' && Number.isNaN(v))) return '—';
|
||||
if (typeof v === 'number') {
|
||||
return v.toFixed(digits) + (suffix ? ` ${suffix}` : '');
|
||||
}
|
||||
return _esc(v) + (suffix ? ` ${suffix}` : '');
|
||||
}
|
||||
|
||||
function _ppvClass(v) {
|
||||
if (v == null) return 'text-gray-400';
|
||||
if (v < 0.5) return 'text-green-600 dark:text-green-400';
|
||||
if (v < 2.0) return 'text-amber-600 dark:text-amber-400';
|
||||
return 'text-red-600 dark:text-red-400 font-semibold';
|
||||
}
|
||||
|
||||
function _kvCard(label, value, options = {}) {
|
||||
// Single key-value tile. `value` is pre-rendered HTML (or text).
|
||||
const colorCls = options.colorCls || '';
|
||||
const valCls = `font-mono font-semibold ${colorCls}`;
|
||||
return `<div class="bg-gray-50 dark:bg-slate-900/50 rounded-lg p-3">
|
||||
<div class="text-xs text-gray-500 dark:text-gray-400 uppercase tracking-wider">${_esc(label)}</div>
|
||||
<div class="${valCls} mt-1">${value}</div>
|
||||
${options.sub ? `<div class="text-xs text-gray-500 dark:text-gray-400 mt-1">${options.sub}</div>` : ''}
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function _deriveRecordType(filename, fallback) {
|
||||
// SFM currently hardcodes record_type="Waveform" for every event.
|
||||
// The actual type is encoded in the LAST character of the Blastware
|
||||
// filename's extension (e.g. "O121LL5E.IS0H" → "H" → Histogram).
|
||||
// We derive it client-side until SFM is fixed; if the suffix isn't
|
||||
// a known code we fall back to whatever SFM reported.
|
||||
if (!filename) return fallback || '—';
|
||||
const dotIdx = filename.lastIndexOf('.');
|
||||
if (dotIdx < 0 || dotIdx === filename.length - 1) return fallback || '—';
|
||||
const ext = filename.slice(dotIdx + 1);
|
||||
const lastChar = ext.slice(-1).toUpperCase();
|
||||
const typeMap = {
|
||||
'H': 'Histogram',
|
||||
'W': 'Waveform',
|
||||
'M': 'Manual',
|
||||
'E': 'Event',
|
||||
'C': 'Combo',
|
||||
};
|
||||
return typeMap[lastChar] || (fallback || '—');
|
||||
}
|
||||
|
||||
function _sectionHeader(title, sub) {
|
||||
return `<h4 class="text-xs font-semibold text-gray-500 dark:text-gray-400 uppercase tracking-wider mb-3 mt-5 first:mt-0">
|
||||
${_esc(title)}${sub ? ` <span class="text-xs text-gray-400 normal-case font-normal ml-2">${_esc(sub)}</span>` : ''}
|
||||
</h4>`;
|
||||
}
|
||||
|
||||
// ── Section renderers ────────────────────────────────────────────
|
||||
|
||||
function _renderEventHeader(s) {
|
||||
const ev = s.event || {};
|
||||
const bw = s.blastware || {};
|
||||
const ts = ev.timestamp ? ev.timestamp.replace('T', ' ').slice(0, 19) : '—';
|
||||
const recType = _deriveRecordType(bw.filename || ev.blastware_filename, ev.record_type);
|
||||
return `<div class="grid grid-cols-1 sm:grid-cols-3 gap-x-6 gap-y-2 text-sm">
|
||||
<div><span class="text-gray-500">Serial</span> <span class="font-mono font-semibold text-seismo-orange ml-1">${_esc(ev.serial)}</span></div>
|
||||
<div><span class="text-gray-500">Timestamp</span> <span class="font-medium ml-1">${ts}</span></div>
|
||||
<div><span class="text-gray-500">Record Type</span> <span class="font-medium ml-1">${_esc(recType)}</span></div>
|
||||
<div><span class="text-gray-500">Sample Rate</span> <span class="font-medium ml-1">${ev.sample_rate ?? '—'} sps</span></div>
|
||||
<div><span class="text-gray-500">Rec Time</span> <span class="font-medium ml-1">${ev.rectime_seconds != null ? ev.rectime_seconds + ' s' : '—'}</span></div>
|
||||
<div><span class="text-gray-500">Waveform Key</span> <span class="font-mono text-xs ml-1">${_esc(ev.waveform_key || '—')}</span></div>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function _renderUserNotes(s) {
|
||||
// The "user notes" metadata the operator typed into the BW device.
|
||||
// These are the strings the future metadata-driven parser will use.
|
||||
// NOTE: SFM's sidecar JSON still names this block `project_info` —
|
||||
// we render it as "User Notes" (the actual BW term) but read the
|
||||
// field by its SFM-API name. Rename in SFM is a future cleanup.
|
||||
const p = s.project_info || {};
|
||||
return `<div class="grid grid-cols-1 sm:grid-cols-2 gap-x-6 gap-y-2 text-sm">
|
||||
<div><span class="text-gray-500">Project</span> <span class="font-medium ml-1">${_esc(p.project || '—')}</span></div>
|
||||
<div><span class="text-gray-500">Client</span> <span class="font-medium ml-1">${_esc(p.client || '—')}</span></div>
|
||||
<div><span class="text-gray-500">Operator</span> <span class="font-medium ml-1">${_esc(p.operator || '—')}</span></div>
|
||||
<div><span class="text-gray-500">Sensor Location</span> <span class="font-medium ml-1">${_esc(p.sensor_location || '—')}</span></div>
|
||||
</div>
|
||||
<p class="text-xs text-gray-500 dark:text-gray-400 mt-2 italic">
|
||||
Values are as typed into the seismograph at session start — not the terra-view project/location assignment.
|
||||
</p>`;
|
||||
}
|
||||
|
||||
function _renderPeakValues(s) {
|
||||
// Prefer bw_report.peaks for richer per-channel data; fall back to peak_values.
|
||||
const bwPeaks = (s.bw_report && s.bw_report.peaks) || null;
|
||||
const pv = s.peak_values || {};
|
||||
|
||||
const tran = bwPeaks ? bwPeaks.tran?.ppv_ips : pv.transverse;
|
||||
const vert = bwPeaks ? bwPeaks.vert?.ppv_ips : pv.vertical;
|
||||
const lng = bwPeaks ? bwPeaks.long?.ppv_ips : pv.longitudinal;
|
||||
const pvs = bwPeaks ? bwPeaks.vector_sum?.ips : pv.vector_sum;
|
||||
const pvsAt = bwPeaks ? bwPeaks.vector_sum?.time_s : null;
|
||||
|
||||
return `<div class="grid grid-cols-2 sm:grid-cols-4 gap-3">
|
||||
${_kvCard('Transverse', `<span class="${_ppvClass(tran)}">${_fmt(tran, 4)}</span>`, { sub: 'in/s' })}
|
||||
${_kvCard('Vertical', `<span class="${_ppvClass(vert)}">${_fmt(vert, 4)}</span>`, { sub: 'in/s' })}
|
||||
${_kvCard('Longitudinal', `<span class="${_ppvClass(lng)}">${_fmt(lng, 4)}</span>`, { sub: 'in/s' })}
|
||||
${_kvCard('Peak Vector Sum', `<span class="${_ppvClass(pvs)} text-base">${_fmt(pvs, 4)}</span>`, {
|
||||
sub: pvsAt != null ? `in/s @ t=${_fmt(pvsAt, 2)}s` : 'in/s',
|
||||
})}
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function _renderMic(s) {
|
||||
// Operators only care about dB(L); PSI tile was dropped 2026-05.
|
||||
// We still render the row if any mic data is present so ZC freq /
|
||||
// time-of-peak stay visible even when bw_report.mic is missing.
|
||||
const mic = (s.bw_report && s.bw_report.mic) || null;
|
||||
const pv = s.peak_values || {};
|
||||
|
||||
if (!mic && pv.mic_psi == null) return '';
|
||||
|
||||
const dbl = mic?.pspl_dbl;
|
||||
const zcHz = mic?.zc_freq_hz;
|
||||
const tPk = mic?.time_of_peak_s;
|
||||
const wt = mic?.weighting;
|
||||
|
||||
return `<div class="grid grid-cols-1 sm:grid-cols-3 gap-3">
|
||||
${_kvCard('Peak Mic dB(L)', _fmt(dbl, 1), { sub: wt || '' })}
|
||||
${_kvCard('ZC Frequency', _fmt(zcHz, 1, 'Hz'))}
|
||||
${_kvCard('Time of Peak', tPk != null ? _fmt(tPk, 2, 's') : '—')}
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function _sensorRow(label, ch) {
|
||||
if (!ch) {
|
||||
return `<tr><td class="px-3 py-2 text-sm font-medium">${_esc(label)}</td>
|
||||
<td class="px-3 py-2 text-sm text-gray-400" colspan="3">—</td></tr>`;
|
||||
}
|
||||
const result = ch.result || '—';
|
||||
const resultCls = result === 'Passed'
|
||||
? 'text-green-600 dark:text-green-400'
|
||||
: (result === 'Failed' ? 'text-red-600 dark:text-red-400 font-semibold' : 'text-gray-500');
|
||||
|
||||
// Geo channels have freq + ratio; mic has freq + amplitude.
|
||||
const rightCol = (ch.amplitude_mv != null)
|
||||
? `<td class="px-3 py-2 text-sm font-mono">${_fmt(ch.amplitude_mv, 1, 'mV')}</td>`
|
||||
: `<td class="px-3 py-2 text-sm font-mono">${ch.ratio != null ? ch.ratio.toFixed(1) + ' ratio' : '—'}</td>`;
|
||||
|
||||
return `<tr>
|
||||
<td class="px-3 py-2 text-sm font-medium">${_esc(label)}</td>
|
||||
<td class="px-3 py-2 text-sm font-mono">${_fmt(ch.freq_hz, 1, 'Hz')}</td>
|
||||
${rightCol}
|
||||
<td class="px-3 py-2 text-sm ${resultCls}">${_esc(result)}</td>
|
||||
</tr>`;
|
||||
}
|
||||
|
||||
function _renderSensorCheck(s) {
|
||||
const sc = s.bw_report && s.bw_report.sensor_check;
|
||||
if (!sc) return '';
|
||||
return `<table class="w-full text-left rounded overflow-hidden border border-gray-200 dark:border-gray-700">
|
||||
<thead class="bg-gray-50 dark:bg-slate-700">
|
||||
<tr>
|
||||
<th class="px-3 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Channel</th>
|
||||
<th class="px-3 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Frequency</th>
|
||||
<th class="px-3 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Amplitude/Ratio</th>
|
||||
<th class="px-3 py-2 text-xs font-medium text-gray-600 dark:text-gray-400 uppercase">Result</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody class="divide-y divide-gray-200 dark:divide-gray-700 bg-white dark:bg-slate-800">
|
||||
${_sensorRow('Transverse', sc.tran)}
|
||||
${_sensorRow('Vertical', sc.vert)}
|
||||
${_sensorRow('Longitudinal', sc.long)}
|
||||
${_sensorRow('Microphone', sc.mic)}
|
||||
</tbody>
|
||||
</table>`;
|
||||
}
|
||||
|
||||
function _renderDeviceMetadata(s) {
|
||||
const bw = s.bw_report || {};
|
||||
const dev = bw.device || {};
|
||||
const rec = bw.recording || {};
|
||||
return `<div class="grid grid-cols-2 sm:grid-cols-3 gap-x-6 gap-y-2 text-sm">
|
||||
<div><span class="text-gray-500">Firmware</span> <span class="font-mono text-xs ml-1">${_esc(bw.version || '—')}</span></div>
|
||||
<div><span class="text-gray-500">Battery</span> <span class="font-medium ml-1">${dev.battery_volts != null ? dev.battery_volts.toFixed(2) + ' V' : '—'}</span></div>
|
||||
<div><span class="text-gray-500">Calibrated</span> <span class="font-medium ml-1">${_esc(dev.calibration_date || '—')}${dev.calibration_by ? ' (' + _esc(dev.calibration_by) + ')' : ''}</span></div>
|
||||
<div><span class="text-gray-500">Geo Range</span> <span class="font-medium ml-1">${rec.geo_range_ips != null ? rec.geo_range_ips + ' in/s' : '—'}</span></div>
|
||||
<div><span class="text-gray-500">Stop Mode</span> <span class="font-medium ml-1">${_esc(rec.stop_mode || '—')}</span></div>
|
||||
<div><span class="text-gray-500">Units</span> <span class="font-medium ml-1">${_esc(rec.units || '—')}</span></div>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
function _renderFileInfo(s, eventId) {
|
||||
const bw = s.blastware || {};
|
||||
const src = s.source || {};
|
||||
const sizeKb = bw.filesize ? (bw.filesize / 1024).toFixed(1) : null;
|
||||
const canDownloadBinary = !!(bw.available && bw.filename && eventId);
|
||||
|
||||
const downloadButtons = `
|
||||
<div class="flex flex-wrap gap-2 mb-4">
|
||||
${canDownloadBinary ? `
|
||||
<a href="/api/sfm/db/events/${encodeURIComponent(eventId)}/blastware_file"
|
||||
download="${_esc(bw.filename)}"
|
||||
class="inline-flex items-center gap-2 px-4 py-2 bg-seismo-orange hover:bg-seismo-navy text-white rounded-lg text-sm font-medium transition-colors">
|
||||
<svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 16v1a3 3 0 003 3h10a3 3 0 003-3v-1m-4-4l-4 4m0 0l-4-4m4 4V4"></path>
|
||||
</svg>
|
||||
Download Blastware file
|
||||
<span class="text-xs opacity-80 ml-1">(${_esc(bw.filename)}${sizeKb ? `, ${sizeKb} KB` : ''})</span>
|
||||
</a>
|
||||
` : `
|
||||
<span class="inline-flex items-center gap-2 px-4 py-2 bg-gray-100 dark:bg-gray-800 text-gray-500 dark:text-gray-400 rounded-lg text-sm cursor-not-allowed">
|
||||
<svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 16v1a3 3 0 003 3h10a3 3 0 003-3v-1m-4-4l-4 4m0 0l-4-4m4 4V4"></path>
|
||||
</svg>
|
||||
Blastware file unavailable
|
||||
</span>
|
||||
`}
|
||||
<button type="button"
|
||||
onclick="window.toggleEventJsonViewer()"
|
||||
class="inline-flex items-center gap-2 px-3 py-2 border border-gray-300 dark:border-gray-600 hover:bg-gray-50 dark:hover:bg-gray-700 text-gray-700 dark:text-gray-300 rounded-lg text-sm transition-colors">
|
||||
<svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M10 20l4-16m4 4l4 4-4 4M6 16l-4-4 4-4"></path>
|
||||
</svg>
|
||||
<span id="event-json-toggle-label">View JSON</span>
|
||||
</button>
|
||||
<a href="/api/sfm/db/events/${encodeURIComponent(eventId)}/sidecar"
|
||||
download="${_esc((bw.filename || 'event') + '.sfm.json')}"
|
||||
class="inline-flex items-center gap-2 px-3 py-2 border border-gray-300 dark:border-gray-600 hover:bg-gray-50 dark:hover:bg-gray-700 text-gray-700 dark:text-gray-300 rounded-lg text-sm transition-colors">
|
||||
<svg class="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 12h6m-6 4h6m2 5H7a2 2 0 01-2-2V5a2 2 0 012-2h5.586a1 1 0 01.707.293l5.414 5.414a1 1 0 01.293.707V19a2 2 0 01-2 2z"></path>
|
||||
</svg>
|
||||
Download sidecar JSON
|
||||
</a>
|
||||
</div>
|
||||
<div id="event-json-viewer" class="hidden mb-4">
|
||||
<div class="flex items-center justify-between mb-2">
|
||||
<span class="text-xs text-gray-500 dark:text-gray-400 uppercase tracking-wider">Sidecar JSON</span>
|
||||
<button type="button" onclick="window.copyEventJson()"
|
||||
class="text-xs text-seismo-orange hover:text-seismo-navy">
|
||||
<span id="event-json-copy-label">Copy</span>
|
||||
</button>
|
||||
</div>
|
||||
<pre id="event-json-pre" class="bg-gray-900 dark:bg-black text-gray-200 font-mono text-xs p-4 rounded-lg max-h-96 overflow-auto whitespace-pre">${_esc(JSON.stringify(s, null, 2))}</pre>
|
||||
</div>
|
||||
`;
|
||||
|
||||
return `${downloadButtons}
|
||||
<div class="grid grid-cols-1 sm:grid-cols-2 gap-x-6 gap-y-2 text-sm">
|
||||
<div class="sm:col-span-2"><span class="text-gray-500">Blastware file</span> <span class="font-mono text-xs ml-1">${_esc(bw.filename || '—')}</span> ${sizeKb ? `<span class="text-xs text-gray-500 ml-2">(${sizeKb} KB)</span>` : ''}</div>
|
||||
<div class="sm:col-span-2"><span class="text-gray-500">SHA-256</span> <span class="font-mono text-xs ml-1 break-all">${_esc(bw.sha256 || '—')}</span></div>
|
||||
<div><span class="text-gray-500">Captured at</span> <span class="font-medium ml-1">${_esc(src.captured_at ? src.captured_at.slice(0, 19).replace('T', ' ') : '—')}</span></div>
|
||||
<div><span class="text-gray-500">Tool version</span> <span class="font-mono text-xs ml-1">${_esc(src.tool_version || '—')}</span></div>
|
||||
</div>`;
|
||||
}
|
||||
|
||||
// ── Public API ───────────────────────────────────────────────────
|
||||
|
||||
window.showEventDetail = async function (eventId) {
|
||||
const modal = document.getElementById(MODAL_ID);
|
||||
if (!modal) {
|
||||
console.warn('event-modal: include event_detail_modal.html partial on this page.');
|
||||
return;
|
||||
}
|
||||
modal.classList.remove('hidden');
|
||||
document.getElementById(MODAL_ID + '-title').textContent = 'Event Detail';
|
||||
document.getElementById(MODAL_ID + '-content').innerHTML = `
|
||||
<div class="text-center py-12 text-gray-500 dark:text-gray-400">
|
||||
<div class="animate-spin rounded-full h-8 w-8 border-b-2 border-seismo-orange mx-auto mb-3"></div>
|
||||
Loading event detail…
|
||||
</div>`;
|
||||
|
||||
let s;
|
||||
try {
|
||||
const r = await fetch(`/api/sfm/db/events/${encodeURIComponent(eventId)}/sidecar`);
|
||||
if (!r.ok) {
|
||||
throw new Error('HTTP ' + r.status + ' fetching sidecar');
|
||||
}
|
||||
s = await r.json();
|
||||
} catch (e) {
|
||||
document.getElementById(MODAL_ID + '-content').innerHTML = `
|
||||
<div class="text-center py-8 text-red-500 text-sm">
|
||||
Failed to load event detail: ${_esc(e.message)}
|
||||
</div>`;
|
||||
return;
|
||||
}
|
||||
|
||||
const ev = s.event || {};
|
||||
const ts = ev.timestamp ? ev.timestamp.replace('T', ' ').slice(0, 19) : '';
|
||||
document.getElementById(MODAL_ID + '-title').textContent =
|
||||
`Event — ${ev.serial || '?'} @ ${ts}`;
|
||||
|
||||
const hasReport = !!s.bw_report;
|
||||
const reportNote = hasReport
|
||||
? ''
|
||||
: `<div class="bg-amber-50 dark:bg-amber-900/20 border border-amber-200 dark:border-amber-800 rounded-lg p-3 text-sm text-amber-800 dark:text-amber-300 mb-4">
|
||||
<strong>No BW ASCII report paired with this event.</strong>
|
||||
Older events forwarded before the watcher's <code class="font-mono text-xs">_ASCII.TXT</code> pairing fix landed lack this data.
|
||||
PPV is still available from the binary event file.
|
||||
</div>`;
|
||||
|
||||
document.getElementById(MODAL_ID + '-content').innerHTML = `
|
||||
${reportNote}
|
||||
|
||||
${_sectionHeader('Event')}
|
||||
${_renderEventHeader(s)}
|
||||
|
||||
${_sectionHeader('User Notes')}
|
||||
${_renderUserNotes(s)}
|
||||
|
||||
${_sectionHeader('Peak Particle Velocity')}
|
||||
${_renderPeakValues(s)}
|
||||
|
||||
${(s.bw_report && (s.bw_report.mic || s.peak_values?.mic_psi != null)) ? `
|
||||
${_sectionHeader('Microphone')}
|
||||
${_renderMic(s)}
|
||||
` : ''}
|
||||
|
||||
${hasReport ? `
|
||||
${_sectionHeader('Sensor Self-Check')}
|
||||
${_renderSensorCheck(s)}
|
||||
|
||||
${_sectionHeader('Device & Recording Metadata')}
|
||||
${_renderDeviceMetadata(s)}
|
||||
` : ''}
|
||||
|
||||
${_sectionHeader('Source File')}
|
||||
${_renderFileInfo(s, eventId)}
|
||||
`;
|
||||
};
|
||||
|
||||
window.closeEventDetailModal = function () {
|
||||
const modal = document.getElementById(MODAL_ID);
|
||||
if (modal) modal.classList.add('hidden');
|
||||
};
|
||||
|
||||
window.toggleEventJsonViewer = function () {
|
||||
const viewer = document.getElementById('event-json-viewer');
|
||||
const label = document.getElementById('event-json-toggle-label');
|
||||
if (!viewer) return;
|
||||
const isHidden = viewer.classList.toggle('hidden');
|
||||
if (label) label.textContent = isHidden ? 'View JSON' : 'Hide JSON';
|
||||
};
|
||||
|
||||
window.copyEventJson = function () {
|
||||
const pre = document.getElementById('event-json-pre');
|
||||
const label = document.getElementById('event-json-copy-label');
|
||||
if (!pre) return;
|
||||
navigator.clipboard.writeText(pre.textContent).then(() => {
|
||||
if (label) {
|
||||
label.textContent = 'Copied!';
|
||||
setTimeout(() => { label.textContent = 'Copy'; }, 1500);
|
||||
}
|
||||
}).catch(err => {
|
||||
console.error('clipboard write failed', err);
|
||||
if (label) {
|
||||
label.textContent = 'Failed';
|
||||
setTimeout(() => { label.textContent = 'Copy'; }, 1500);
|
||||
}
|
||||
});
|
||||
};
|
||||
|
||||
// Close on Escape.
|
||||
document.addEventListener('keydown', function (e) {
|
||||
if (e.key === 'Escape') window.closeEventDetailModal();
|
||||
});
|
||||
})();
|
||||
|
After Width: | Height: | Size: 424 B |
|
After Width: | Height: | Size: 1.1 KiB |
|
Before Width: | Height: | Size: 1.9 KiB After Width: | Height: | Size: 7.7 KiB |
|
Before Width: | Height: | Size: 2.2 KiB After Width: | Height: | Size: 9.2 KiB |
|
Before Width: | Height: | Size: 2.2 KiB After Width: | Height: | Size: 10 KiB |
|
Before Width: | Height: | Size: 2.9 KiB After Width: | Height: | Size: 15 KiB |
|
Before Width: | Height: | Size: 5.8 KiB After Width: | Height: | Size: 44 KiB |
|
Before Width: | Height: | Size: 7.8 KiB After Width: | Height: | Size: 68 KiB |
|
Before Width: | Height: | Size: 1.1 KiB After Width: | Height: | Size: 3.2 KiB |
|
Before Width: | Height: | Size: 1.4 KiB After Width: | Height: | Size: 5.0 KiB |
|
After Width: | Height: | Size: 13 KiB |
|
After Width: | Height: | Size: 57 KiB |
|
After Width: | Height: | Size: 14 KiB |
|
After Width: | Height: | Size: 49 KiB |
@@ -0,0 +1,90 @@
|
||||
"""
|
||||
Shared Jinja2 templates configuration.
|
||||
|
||||
All routers should import `templates` from this module to get consistent
|
||||
filter and global function registration.
|
||||
"""
|
||||
|
||||
import json as _json
|
||||
from fastapi.templating import Jinja2Templates
|
||||
|
||||
# Import timezone utilities
|
||||
from backend.utils.timezone import (
|
||||
format_local_datetime, format_local_time,
|
||||
get_user_timezone, get_timezone_abbreviation
|
||||
)
|
||||
|
||||
|
||||
def jinja_local_datetime(dt, fmt="%Y-%m-%d %H:%M"):
|
||||
"""Jinja filter to convert UTC datetime to local timezone."""
|
||||
return format_local_datetime(dt, fmt)
|
||||
|
||||
|
||||
def jinja_local_time(dt):
|
||||
"""Jinja filter to format time in local timezone."""
|
||||
return format_local_time(dt)
|
||||
|
||||
|
||||
def jinja_timezone_abbr():
|
||||
"""Jinja global to get current timezone abbreviation."""
|
||||
return get_timezone_abbreviation()
|
||||
|
||||
|
||||
# Create templates instance
|
||||
templates = Jinja2Templates(directory="templates")
|
||||
|
||||
def jinja_local_date(dt, fmt="%m-%d-%y"):
|
||||
"""Jinja filter: format a UTC datetime as a local date string (e.g. 02-19-26)."""
|
||||
return format_local_datetime(dt, fmt)
|
||||
|
||||
|
||||
def jinja_fromjson(s):
|
||||
"""Jinja filter: parse a JSON string into a dict (returns {} on failure)."""
|
||||
if not s:
|
||||
return {}
|
||||
try:
|
||||
return _json.loads(s)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def jinja_same_date(dt1, dt2) -> bool:
|
||||
"""Jinja global: True if two datetimes fall on the same local date."""
|
||||
if not dt1 or not dt2:
|
||||
return False
|
||||
try:
|
||||
d1 = format_local_datetime(dt1, "%Y-%m-%d")
|
||||
d2 = format_local_datetime(dt2, "%Y-%m-%d")
|
||||
return d1 == d2
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
def jinja_log_tail_display(s):
|
||||
"""Jinja filter: decode a JSON-encoded log tail array into a plain-text string."""
|
||||
if not s:
|
||||
return ""
|
||||
try:
|
||||
lines = _json.loads(s)
|
||||
if isinstance(lines, list):
|
||||
return "\n".join(str(l) for l in lines)
|
||||
return str(s)
|
||||
except Exception:
|
||||
return str(s)
|
||||
|
||||
|
||||
def jinja_local_datetime_input(dt):
|
||||
"""Jinja filter: format UTC datetime as local YYYY-MM-DDTHH:MM for <input type='datetime-local'>."""
|
||||
return format_local_datetime(dt, "%Y-%m-%dT%H:%M")
|
||||
|
||||
|
||||
# Register Jinja filters and globals
|
||||
templates.env.filters["local_datetime"] = jinja_local_datetime
|
||||
templates.env.filters["local_time"] = jinja_local_time
|
||||
templates.env.filters["local_date"] = jinja_local_date
|
||||
templates.env.filters["local_datetime_input"] = jinja_local_datetime_input
|
||||
templates.env.filters["fromjson"] = jinja_fromjson
|
||||
templates.env.globals["timezone_abbr"] = jinja_timezone_abbr
|
||||
templates.env.globals["get_user_timezone"] = get_user_timezone
|
||||
templates.env.globals["same_date"] = jinja_same_date
|
||||
templates.env.filters["log_tail_display"] = jinja_log_tail_display
|
||||
@@ -0,0 +1 @@
|
||||
# Utils package
|
||||
@@ -0,0 +1,173 @@
|
||||
"""
|
||||
Timezone utilities for Terra-View.
|
||||
|
||||
Provides consistent timezone handling throughout the application.
|
||||
All database times are stored in UTC; this module converts for display.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from zoneinfo import ZoneInfo
|
||||
from typing import Optional
|
||||
|
||||
from backend.database import SessionLocal
|
||||
from backend.models import UserPreferences
|
||||
|
||||
|
||||
# Default timezone if none set
|
||||
DEFAULT_TIMEZONE = "America/New_York"
|
||||
|
||||
|
||||
def get_user_timezone() -> str:
|
||||
"""
|
||||
Get the user's configured timezone from preferences.
|
||||
|
||||
Returns:
|
||||
Timezone string (e.g., "America/New_York")
|
||||
"""
|
||||
db = SessionLocal()
|
||||
try:
|
||||
prefs = db.query(UserPreferences).filter_by(id=1).first()
|
||||
if prefs and prefs.timezone:
|
||||
return prefs.timezone
|
||||
return DEFAULT_TIMEZONE
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def get_timezone_info(tz_name: str = None) -> ZoneInfo:
|
||||
"""
|
||||
Get ZoneInfo object for the specified or user's timezone.
|
||||
|
||||
Args:
|
||||
tz_name: Timezone name, or None to use user preference
|
||||
|
||||
Returns:
|
||||
ZoneInfo object
|
||||
"""
|
||||
if tz_name is None:
|
||||
tz_name = get_user_timezone()
|
||||
try:
|
||||
return ZoneInfo(tz_name)
|
||||
except Exception:
|
||||
return ZoneInfo(DEFAULT_TIMEZONE)
|
||||
|
||||
|
||||
def utc_to_local(dt: datetime, tz_name: str = None) -> datetime:
|
||||
"""
|
||||
Convert a UTC datetime to local timezone.
|
||||
|
||||
Args:
|
||||
dt: Datetime in UTC (naive or aware)
|
||||
tz_name: Target timezone, or None to use user preference
|
||||
|
||||
Returns:
|
||||
Datetime in local timezone
|
||||
"""
|
||||
if dt is None:
|
||||
return None
|
||||
|
||||
tz = get_timezone_info(tz_name)
|
||||
|
||||
# Assume naive datetime is UTC
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=ZoneInfo("UTC"))
|
||||
|
||||
return dt.astimezone(tz)
|
||||
|
||||
|
||||
def local_to_utc(dt: datetime, tz_name: str = None) -> datetime:
|
||||
"""
|
||||
Convert a local datetime to UTC.
|
||||
|
||||
Args:
|
||||
dt: Datetime in local timezone (naive or aware)
|
||||
tz_name: Source timezone, or None to use user preference
|
||||
|
||||
Returns:
|
||||
Datetime in UTC (naive, for database storage)
|
||||
"""
|
||||
if dt is None:
|
||||
return None
|
||||
|
||||
tz = get_timezone_info(tz_name)
|
||||
|
||||
# Assume naive datetime is in local timezone
|
||||
if dt.tzinfo is None:
|
||||
dt = dt.replace(tzinfo=tz)
|
||||
|
||||
# Convert to UTC and strip tzinfo for database storage
|
||||
return dt.astimezone(ZoneInfo("UTC")).replace(tzinfo=None)
|
||||
|
||||
|
||||
def format_local_datetime(dt: datetime, fmt: str = "%Y-%m-%d %H:%M", tz_name: str = None) -> str:
|
||||
"""
|
||||
Format a UTC datetime as local time string.
|
||||
|
||||
Args:
|
||||
dt: Datetime in UTC
|
||||
fmt: strftime format string
|
||||
tz_name: Target timezone, or None to use user preference
|
||||
|
||||
Returns:
|
||||
Formatted datetime string in local time
|
||||
"""
|
||||
if dt is None:
|
||||
return "N/A"
|
||||
|
||||
local_dt = utc_to_local(dt, tz_name)
|
||||
return local_dt.strftime(fmt)
|
||||
|
||||
|
||||
def format_local_time(dt: datetime, tz_name: str = None) -> str:
|
||||
"""
|
||||
Format a UTC datetime as local time (HH:MM format).
|
||||
|
||||
Args:
|
||||
dt: Datetime in UTC
|
||||
tz_name: Target timezone
|
||||
|
||||
Returns:
|
||||
Time string in HH:MM format
|
||||
"""
|
||||
return format_local_datetime(dt, "%H:%M", tz_name)
|
||||
|
||||
|
||||
def format_local_date(dt: datetime, tz_name: str = None) -> str:
|
||||
"""
|
||||
Format a UTC datetime as local date (YYYY-MM-DD format).
|
||||
|
||||
Args:
|
||||
dt: Datetime in UTC
|
||||
tz_name: Target timezone
|
||||
|
||||
Returns:
|
||||
Date string
|
||||
"""
|
||||
return format_local_datetime(dt, "%Y-%m-%d", tz_name)
|
||||
|
||||
|
||||
def get_timezone_abbreviation(tz_name: str = None) -> str:
|
||||
"""
|
||||
Get the abbreviation for a timezone (e.g., EST, EDT, PST).
|
||||
|
||||
Args:
|
||||
tz_name: Timezone name, or None to use user preference
|
||||
|
||||
Returns:
|
||||
Timezone abbreviation
|
||||
"""
|
||||
tz = get_timezone_info(tz_name)
|
||||
now = datetime.now(tz)
|
||||
return now.strftime("%Z")
|
||||
|
||||
|
||||
# Common US timezone choices for settings dropdown
|
||||
TIMEZONE_CHOICES = [
|
||||
("America/New_York", "Eastern Time (ET)"),
|
||||
("America/Chicago", "Central Time (CT)"),
|
||||
("America/Denver", "Mountain Time (MT)"),
|
||||
("America/Los_Angeles", "Pacific Time (PT)"),
|
||||
("America/Anchorage", "Alaska Time (AKT)"),
|
||||
("Pacific/Honolulu", "Hawaii Time (HT)"),
|
||||
("UTC", "UTC"),
|
||||
]
|
||||
@@ -1,10 +0,0 @@
|
||||
{
|
||||
"filename": "snapshot_20251216_201738.db",
|
||||
"created_at": "20251216_201738",
|
||||
"created_at_iso": "2025-12-16T20:17:38.638982",
|
||||
"description": "Auto-backup before restore",
|
||||
"size_bytes": 57344,
|
||||
"size_mb": 0.05,
|
||||
"original_db_size_bytes": 57344,
|
||||
"type": "manual"
|
||||
}
|
||||
@@ -1,9 +0,0 @@
|
||||
{
|
||||
"filename": "snapshot_uploaded_20251216_201732.db",
|
||||
"created_at": "20251216_201732",
|
||||
"created_at_iso": "2025-12-16T20:17:32.574205",
|
||||
"description": "Uploaded: snapshot_20251216_200259.db",
|
||||
"size_bytes": 77824,
|
||||
"size_mb": 0.07,
|
||||
"type": "uploaded"
|
||||
}
|
||||
@@ -1,9 +1,7 @@
|
||||
services:
|
||||
|
||||
# --- TERRA-VIEW PRODUCTION ---
|
||||
terra-view-prod:
|
||||
terra-view:
|
||||
build: .
|
||||
container_name: terra-view
|
||||
ports:
|
||||
- "8001:8001"
|
||||
volumes:
|
||||
@@ -12,9 +10,11 @@ services:
|
||||
- PYTHONUNBUFFERED=1
|
||||
- ENVIRONMENT=production
|
||||
- SLMM_BASE_URL=http://host.docker.internal:8100
|
||||
- SFM_BASE_URL=http://sfm:8200
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- slmm
|
||||
- sfm
|
||||
extra_hosts:
|
||||
- "host.docker.internal:host-gateway"
|
||||
healthcheck:
|
||||
@@ -24,34 +24,11 @@ services:
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
|
||||
# --- TERRA-VIEW DEVELOPMENT ---
|
||||
# terra-view-dev:
|
||||
# build: .
|
||||
# container_name: terra-view-dev
|
||||
# ports:
|
||||
# - "1001:8001"
|
||||
# volumes:
|
||||
# - ./data-dev:/app/data
|
||||
# environment:
|
||||
# - PYTHONUNBUFFERED=1
|
||||
# - ENVIRONMENT=development
|
||||
# - SLMM_BASE_URL=http://slmm:8100
|
||||
# restart: unless-stopped
|
||||
# depends_on:
|
||||
# - slmm
|
||||
# healthcheck:
|
||||
# test: ["CMD", "curl", "-f", "http://localhost:8001/health"]
|
||||
# interval: 30s
|
||||
# timeout: 10s
|
||||
# retries: 3
|
||||
# start_period: 40s
|
||||
|
||||
# --- SLMM (Sound Level Meter Manager) ---
|
||||
slmm:
|
||||
build:
|
||||
context: ../slmm
|
||||
dockerfile: Dockerfile
|
||||
container_name: slmm
|
||||
network_mode: host
|
||||
volumes:
|
||||
- ../slmm/data:/app/data
|
||||
@@ -59,6 +36,8 @@ services:
|
||||
- PYTHONUNBUFFERED=1
|
||||
- PORT=8100
|
||||
- CORS_ORIGINS=*
|
||||
- TCP_IDLE_TTL=-1
|
||||
- TCP_MAX_AGE=-1
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8100/health"]
|
||||
@@ -67,6 +46,25 @@ services:
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
# --- SFM (Seismo Fleet Manager) ---
|
||||
sfm:
|
||||
build:
|
||||
context: ../seismo-relay
|
||||
dockerfile: Dockerfile
|
||||
ports:
|
||||
- "8200:8200"
|
||||
volumes:
|
||||
- ../seismo-relay/sfm/data:/app/sfm/data
|
||||
- ../seismo-relay/bridges/captures:/app/bridges/captures
|
||||
environment:
|
||||
- PYTHONUNBUFFERED=1
|
||||
- PORT=8200
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8200/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
volumes:
|
||||
data:
|
||||
data-dev:
|
||||
|
||||
@@ -125,7 +125,7 @@ seismos = db.query(RosterUnit).filter_by(
|
||||
### Sound Level Meters Query
|
||||
```python
|
||||
slms = db.query(RosterUnit).filter_by(
|
||||
device_type="sound_level_meter",
|
||||
device_type="slm",
|
||||
retired=False
|
||||
).all()
|
||||
```
|
||||
|
||||
@@ -0,0 +1,288 @@
|
||||
# Device Type Schema - Terra-View
|
||||
|
||||
## Overview
|
||||
|
||||
Terra-View uses a single roster table to manage three different device types. The `device_type` field is the primary discriminator that determines which fields are relevant for each unit.
|
||||
|
||||
## Official device_type Values
|
||||
|
||||
As of **Terra-View v0.4.3**, the following device_type values are standardized:
|
||||
|
||||
### 1. `"seismograph"` (Default)
|
||||
**Purpose**: Seismic monitoring devices
|
||||
|
||||
**Applicable Fields**:
|
||||
- Common: id, unit_type, deployed, retired, note, project_id, location, address, coordinates
|
||||
- Specific: last_calibrated, next_calibration_due, deployed_with_modem_id
|
||||
|
||||
**Examples**:
|
||||
- `BE1234` - Series 3 seismograph
|
||||
- `UM12345` - Series 4 Micromate unit
|
||||
- `SEISMO-001` - Custom seismograph
|
||||
|
||||
**Unit Type Values**:
|
||||
- `series3` - Series 3 devices (default)
|
||||
- `series4` - Series 4 devices
|
||||
- `micromate` - Micromate devices
|
||||
|
||||
---
|
||||
|
||||
### 2. `"modem"`
|
||||
**Purpose**: Field modems and network equipment
|
||||
|
||||
**Applicable Fields**:
|
||||
- Common: id, unit_type, deployed, retired, note, project_id, location, address, coordinates
|
||||
- Specific: ip_address, phone_number, hardware_model
|
||||
|
||||
**Examples**:
|
||||
- `MDM001` - Field modem
|
||||
- `MODEM-2025-01` - Network modem
|
||||
- `RAVEN-XTV-01` - Specific modem model
|
||||
|
||||
**Unit Type Values**:
|
||||
- `modem` - Generic modem
|
||||
- `raven-xtv` - Raven XTV model
|
||||
- Custom values for specific hardware
|
||||
|
||||
---
|
||||
|
||||
### 3. `"slm"` ⭐
|
||||
**Purpose**: Sound level meters (Rion NL-43/NL-53)
|
||||
|
||||
**Applicable Fields**:
|
||||
- Common: id, unit_type, deployed, retired, note, project_id, location, address, coordinates
|
||||
- Specific: slm_host, slm_tcp_port, slm_ftp_port, slm_model, slm_serial_number, slm_frequency_weighting, slm_time_weighting, slm_measurement_range, slm_last_check, deployed_with_modem_id
|
||||
|
||||
**Examples**:
|
||||
- `SLM-43-01` - NL-43 sound level meter
|
||||
- `NL43-001` - NL-43 unit
|
||||
- `NL53-002` - NL-53 unit
|
||||
|
||||
**Unit Type Values**:
|
||||
- `nl43` - Rion NL-43 model
|
||||
- `nl53` - Rion NL-53 model
|
||||
|
||||
---
|
||||
|
||||
## Migration from Legacy Values
|
||||
|
||||
### Deprecated Values
|
||||
|
||||
The following device_type values have been **deprecated** and should be migrated:
|
||||
|
||||
- ❌ `"sound_level_meter"` → ✅ `"slm"`
|
||||
|
||||
### How to Migrate
|
||||
|
||||
Run the standardization migration script to update existing databases:
|
||||
|
||||
```bash
|
||||
cd /home/serversdown/tmi/terra-view
|
||||
python3 backend/migrate_standardize_device_types.py
|
||||
```
|
||||
|
||||
This script:
|
||||
- Converts all `"sound_level_meter"` values to `"slm"`
|
||||
- Is idempotent (safe to run multiple times)
|
||||
- Shows before/after distribution of device types
|
||||
- No data loss
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
### RosterUnit Model (`backend/models.py`)
|
||||
|
||||
```python
|
||||
class RosterUnit(Base):
|
||||
"""
|
||||
Supports multiple device types:
|
||||
- "seismograph" - Seismic monitoring devices (default)
|
||||
- "modem" - Field modems and network equipment
|
||||
- "slm" - Sound level meters (NL-43/NL-53)
|
||||
"""
|
||||
__tablename__ = "roster"
|
||||
|
||||
# Core fields (all device types)
|
||||
id = Column(String, primary_key=True)
|
||||
unit_type = Column(String, default="series3")
|
||||
device_type = Column(String, default="seismograph") # "seismograph" | "modem" | "slm"
|
||||
deployed = Column(Boolean, default=True)
|
||||
retired = Column(Boolean, default=False)
|
||||
# ... other common fields
|
||||
|
||||
# Seismograph-specific
|
||||
last_calibrated = Column(Date, nullable=True)
|
||||
next_calibration_due = Column(Date, nullable=True)
|
||||
|
||||
# Modem-specific
|
||||
ip_address = Column(String, nullable=True)
|
||||
phone_number = Column(String, nullable=True)
|
||||
hardware_model = Column(String, nullable=True)
|
||||
|
||||
# SLM-specific
|
||||
slm_host = Column(String, nullable=True)
|
||||
slm_tcp_port = Column(Integer, nullable=True)
|
||||
slm_ftp_port = Column(Integer, nullable=True)
|
||||
slm_model = Column(String, nullable=True)
|
||||
slm_serial_number = Column(String, nullable=True)
|
||||
slm_frequency_weighting = Column(String, nullable=True)
|
||||
slm_time_weighting = Column(String, nullable=True)
|
||||
slm_measurement_range = Column(String, nullable=True)
|
||||
slm_last_check = Column(DateTime, nullable=True)
|
||||
|
||||
# Shared fields (seismograph + SLM)
|
||||
deployed_with_modem_id = Column(String, nullable=True) # FK to modem
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Usage
|
||||
|
||||
### Adding a New Unit
|
||||
|
||||
**Seismograph**:
|
||||
```bash
|
||||
curl -X POST http://localhost:8001/api/roster/add \
|
||||
-F "id=BE1234" \
|
||||
-F "device_type=seismograph" \
|
||||
-F "unit_type=series3" \
|
||||
-F "deployed=true"
|
||||
```
|
||||
|
||||
**Modem**:
|
||||
```bash
|
||||
curl -X POST http://localhost:8001/api/roster/add \
|
||||
-F "id=MDM001" \
|
||||
-F "device_type=modem" \
|
||||
-F "ip_address=192.0.2.10" \
|
||||
-F "phone_number=+1-555-0100"
|
||||
```
|
||||
|
||||
**Sound Level Meter**:
|
||||
```bash
|
||||
curl -X POST http://localhost:8001/api/roster/add \
|
||||
-F "id=SLM-43-01" \
|
||||
-F "device_type=slm" \
|
||||
-F "slm_host=63.45.161.30" \
|
||||
-F "slm_tcp_port=2255" \
|
||||
-F "slm_model=NL-43"
|
||||
```
|
||||
|
||||
### CSV Import Format
|
||||
|
||||
```csv
|
||||
unit_id,unit_type,device_type,deployed,slm_host,slm_tcp_port,slm_model
|
||||
SLM-43-01,nl43,slm,true,63.45.161.30,2255,NL-43
|
||||
SLM-43-02,nl43,slm,true,63.45.161.31,2255,NL-43
|
||||
BE1234,series3,seismograph,true,,,
|
||||
MDM001,modem,modem,true,,,
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Frontend Behavior
|
||||
|
||||
### Device Type Selection
|
||||
|
||||
**Templates**: `unit_detail.html`, `roster.html`
|
||||
|
||||
```html
|
||||
<select name="device_type">
|
||||
<option value="seismograph">Seismograph</option>
|
||||
<option value="modem">Modem</option>
|
||||
<option value="slm">Sound Level Meter</option>
|
||||
</select>
|
||||
```
|
||||
|
||||
### Conditional Field Display
|
||||
|
||||
JavaScript functions check `device_type` to show/hide relevant fields:
|
||||
|
||||
```javascript
|
||||
function toggleDetailFields() {
|
||||
const deviceType = document.getElementById('device_type').value;
|
||||
|
||||
if (deviceType === 'seismograph') {
|
||||
// Show calibration fields
|
||||
} else if (deviceType === 'modem') {
|
||||
// Show network fields
|
||||
} else if (deviceType === 'slm') {
|
||||
// Show SLM configuration fields
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Code Conventions
|
||||
|
||||
### Always Use Lowercase
|
||||
|
||||
✅ **Correct**:
|
||||
```python
|
||||
if unit.device_type == "slm":
|
||||
# Handle sound level meter
|
||||
```
|
||||
|
||||
❌ **Incorrect**:
|
||||
```python
|
||||
if unit.device_type == "SLM": # Wrong - case sensitive
|
||||
if unit.device_type == "sound_level_meter": # Deprecated
|
||||
```
|
||||
|
||||
### Query Patterns
|
||||
|
||||
**Filter by device type**:
|
||||
```python
|
||||
# Get all SLMs
|
||||
slms = db.query(RosterUnit).filter_by(device_type="slm").all()
|
||||
|
||||
# Get deployed seismographs
|
||||
seismos = db.query(RosterUnit).filter_by(
|
||||
device_type="seismograph",
|
||||
deployed=True
|
||||
).all()
|
||||
|
||||
# Get all modems
|
||||
modems = db.query(RosterUnit).filter_by(device_type="modem").all()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
### Verify Device Type Distribution
|
||||
|
||||
```bash
|
||||
# Quick check
|
||||
sqlite3 data/seismo_fleet.db "SELECT device_type, COUNT(*) FROM roster GROUP BY device_type;"
|
||||
|
||||
# Detailed view
|
||||
sqlite3 data/seismo_fleet.db "SELECT id, device_type, unit_type, deployed FROM roster ORDER BY device_type, id;"
|
||||
```
|
||||
|
||||
### Check for Legacy Values
|
||||
|
||||
```bash
|
||||
# Should return 0 rows after migration
|
||||
sqlite3 data/seismo_fleet.db "SELECT id FROM roster WHERE device_type = 'sound_level_meter';"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Version History
|
||||
|
||||
- **v0.4.3** (2026-01-16) - Standardized device_type values, deprecated `"sound_level_meter"` → `"slm"`
|
||||
- **v0.4.0** (2026-01-05) - Added SLM support with `"sound_level_meter"` value
|
||||
- **v0.2.0** (2025-12-03) - Added modem device type
|
||||
- **v0.1.0** (2024-11-20) - Initial release with seismograph-only support
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [README.md](../README.md) - Main project documentation with data model
|
||||
- [DEVICE_TYPE_SLM_SUPPORT.md](DEVICE_TYPE_SLM_SUPPORT.md) - Legacy SLM implementation notes
|
||||
- [SOUND_LEVEL_METERS_DASHBOARD.md](SOUND_LEVEL_METERS_DASHBOARD.md) - SLM dashboard features
|
||||
- [SLM_CONFIGURATION.md](SLM_CONFIGURATION.md) - SLM device configuration guide
|
||||
@@ -1,5 +1,7 @@
|
||||
# Sound Level Meter Device Type Support
|
||||
|
||||
**⚠️ IMPORTANT**: This documentation uses the legacy `sound_level_meter` device type value. As of v0.4.3, the standardized value is `"slm"`. Run `backend/migrate_standardize_device_types.py` to update your database.
|
||||
|
||||
## Overview
|
||||
|
||||
Added full support for "Sound Level Meter" as a device type in the roster management system. Users can now create, edit, and manage SLM units through the Fleet Roster interface.
|
||||
@@ -95,7 +97,7 @@ All SLM fields are updated when editing existing unit.
|
||||
|
||||
The database schema already included SLM fields (no changes needed):
|
||||
- All fields are nullable to support multiple device types
|
||||
- Fields are only relevant when `device_type = "sound_level_meter"`
|
||||
- Fields are only relevant when `device_type = "slm"`
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -125,7 +127,7 @@ The form automatically shows/hides relevant fields based on device type:
|
||||
|
||||
## Integration with SLMM Dashboard
|
||||
|
||||
Units with `device_type = "sound_level_meter"` will:
|
||||
Units with `device_type = "slm"` will:
|
||||
- Appear in the Sound Level Meters dashboard (`/sound-level-meters`)
|
||||
- Be available for live monitoring and control
|
||||
- Use the configured `slm_host` and `slm_tcp_port` for device communication
|
||||
|
||||
@@ -300,7 +300,7 @@ slm.deployed_with_modem_id = "modem-001"
|
||||
```json
|
||||
{
|
||||
"id": "nl43-001",
|
||||
"device_type": "sound_level_meter",
|
||||
"device_type": "slm",
|
||||
"deployed_with_modem_id": "modem-001",
|
||||
"slm_tcp_port": 2255,
|
||||
"slm_model": "NL-43",
|
||||
|
||||
@@ -135,7 +135,7 @@ The dashboard communicates with the SLMM backend service running on port 8100:
|
||||
SLM-specific fields in the RosterUnit model:
|
||||
|
||||
```python
|
||||
device_type = "sound_level_meter" # Distinguishes SLMs from seismographs
|
||||
device_type = "slm" # Distinguishes SLMs from seismographs
|
||||
slm_host = String # Device IP or hostname
|
||||
slm_tcp_port = Integer # TCP control port (default 2255)
|
||||
slm_model = String # NL-43, NL-53, etc.
|
||||
|
||||
@@ -0,0 +1,436 @@
|
||||
# Synology NAS Deployment Guide
|
||||
|
||||
This guide covers migrating the terra-view stack from a generic Linux host
|
||||
(currently the home server at `10.0.0.44`) to an always-on Synology NAS in
|
||||
the office, including data migration and the minimal external-access
|
||||
networking layer.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Architecture overview](#architecture-overview)
|
||||
2. [Pre-requisites](#pre-requisites)
|
||||
3. [Phase 1 — Pre-stage on the NAS (no downtime)](#phase-1--pre-stage-on-the-nas-no-downtime)
|
||||
4. [Phase 2 — Data migration (~10 min window)](#phase-2--data-migration-10-min-window)
|
||||
5. [Phase 3 — Repoint the watcher (download2-PC)](#phase-3--repoint-the-watcher-download2-pc)
|
||||
6. [Phase 4 — External access for remote operators](#phase-4--external-access-for-remote-operators)
|
||||
7. [Phase 5 — Decommission home server](#phase-5--decommission-home-server)
|
||||
8. [Verification checklist](#verification-checklist)
|
||||
9. [Rollback plan](#rollback-plan)
|
||||
10. [Gotchas](#gotchas)
|
||||
|
||||
---
|
||||
|
||||
## Architecture overview
|
||||
|
||||
The terra-view stack is three containers:
|
||||
|
||||
| Service | Port | What writes to it | Where it lives |
|
||||
|---------------|-------|-----------------------------|----------------|
|
||||
| terra-view | 8001 | Operators (UI), watchers (heartbeat) | Synology NAS |
|
||||
| SFM | 8200 | Watchers (Blastware ACH forwards) | Synology NAS |
|
||||
| SLMM | 8100 | terra-view (proxied), SLMs on LAN | Synology NAS |
|
||||
|
||||
Everything that **writes** to the stack lives inside the office LAN:
|
||||
|
||||
- **download2-PC** is the series3-watcher host. It has a static office IP and
|
||||
POSTs to terra-view's heartbeat endpoint plus SFM's Blastware import
|
||||
endpoint. Both flows are LAN-internal.
|
||||
- **Sound level meters (NL-43)** sit on the office LAN; SLMM reaches them
|
||||
via `network_mode: host`.
|
||||
|
||||
The **only** thing that needs to cross the office firewall is operator UI
|
||||
access from outside the office (laptops, phones, working from home). That
|
||||
makes the external networking layer trivial — see Phase 4.
|
||||
|
||||
---
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
On the Synology side:
|
||||
|
||||
- **DSM 7.2+** with **Container Manager** installed (Package Center).
|
||||
Older "Docker" package works too — same engine, different menu names.
|
||||
- **x86_64 model** (Plus / Value / XS series). ARM j-series will build but
|
||||
expect a slower first build.
|
||||
- **Static LAN IP** reserved for the NAS in the office router's DHCP table.
|
||||
Devices on the LAN must have a stable target.
|
||||
- **SSH enabled** — Control Panel → Terminal & SNMP → Enable SSH service.
|
||||
- **Shared folder** for the stack — e.g. `/volume1/docker/`.
|
||||
|
||||
On the home server side:
|
||||
|
||||
- Working terra-view / SFM / SLMM stack you want to migrate.
|
||||
- `rsync` available (it almost certainly is).
|
||||
|
||||
You will also need:
|
||||
|
||||
- An admin account on the Synology with sudo privileges.
|
||||
- Network access between the home server and the NAS during the migration
|
||||
window (or USB-drive shuttle if not).
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — Pre-stage on the NAS (no downtime)
|
||||
|
||||
Goal: get the NAS booting an empty stack so you can validate the build and
|
||||
networking *before* touching any production data.
|
||||
|
||||
### 1.1 Clone the repos
|
||||
|
||||
SSH to the NAS as admin:
|
||||
|
||||
```bash
|
||||
sudo mkdir -p /volume1/docker
|
||||
cd /volume1/docker
|
||||
sudo git clone <your-terra-view-remote> terra-view
|
||||
sudo git clone <your-slmm-remote> slmm
|
||||
sudo git clone <your-seismo-relay-remote> seismo-relay
|
||||
cd terra-view
|
||||
sudo git checkout main # or whichever branch you ship from
|
||||
```
|
||||
|
||||
### 1.2 Build images
|
||||
|
||||
```bash
|
||||
cd /volume1/docker/terra-view
|
||||
sudo docker compose build
|
||||
```
|
||||
|
||||
First build takes 5–15 min depending on model.
|
||||
|
||||
### 1.3 Boot the empty stack
|
||||
|
||||
```bash
|
||||
sudo docker compose up -d
|
||||
```
|
||||
|
||||
Hit `http://<nas-lan-ip>:1001` (dev profile) or `:8001` (prod profile) from
|
||||
another office machine. You should see an empty fleet roster. If that
|
||||
works, the NAS can run the stack — proven before any production data is
|
||||
at risk.
|
||||
|
||||
### 1.4 Stop the NAS stack again
|
||||
|
||||
```bash
|
||||
sudo docker compose stop
|
||||
```
|
||||
|
||||
We're ready for the data migration.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Data migration (~10 min window)
|
||||
|
||||
The terra-view stack is stateful in three places. All three must be moved
|
||||
together for consistency.
|
||||
|
||||
| Service | Data location (home server) |
|
||||
|------------|----------------------------------------------|
|
||||
| terra-view | `/home/serversdown/terra-view/data/` |
|
||||
| SLMM | `/home/serversdown/slmm/data/` |
|
||||
| SFM | `/home/serversdown/seismo-relay/data/` |
|
||||
|
||||
### 2.1 Stop writes on both sides
|
||||
|
||||
On the NAS:
|
||||
|
||||
```bash
|
||||
cd /volume1/docker/terra-view
|
||||
sudo docker compose stop
|
||||
```
|
||||
|
||||
On the home server:
|
||||
|
||||
```bash
|
||||
cd /home/serversdown/terra-view
|
||||
docker compose stop terra-view slmm sfm
|
||||
```
|
||||
|
||||
### 2.2 rsync the data dirs
|
||||
|
||||
From the home server (or anywhere with SSH access to both):
|
||||
|
||||
```bash
|
||||
rsync -avh /home/serversdown/terra-view/data/ admin@<nas-lan-ip>:/volume1/docker/terra-view/data/
|
||||
rsync -avh /home/serversdown/slmm/data/ admin@<nas-lan-ip>:/volume1/docker/slmm/data/
|
||||
rsync -avh /home/serversdown/seismo-relay/data/ admin@<nas-lan-ip>:/volume1/docker/seismo-relay/data/
|
||||
```
|
||||
|
||||
### 2.3 Fix ownership on the NAS
|
||||
|
||||
Synology admin is usually UID `1026`, GID `100`. Inside containers running
|
||||
as root, this doesn't matter — but if you've configured `user:` in any
|
||||
compose file it will. Safe default:
|
||||
|
||||
```bash
|
||||
ssh admin@<nas-lan-ip> "sudo chown -R 1026:100 \
|
||||
/volume1/docker/terra-view/data \
|
||||
/volume1/docker/slmm/data \
|
||||
/volume1/docker/seismo-relay/data"
|
||||
```
|
||||
|
||||
### 2.4 Run any pending migrations
|
||||
|
||||
Some earlier feature work added migration scripts that need to run once
|
||||
per database. After the rsync, before starting the stack, check what's
|
||||
pending:
|
||||
|
||||
```bash
|
||||
ssh admin@<nas-lan-ip>
|
||||
cd /volume1/docker/terra-view
|
||||
ls backend/migrate_*.py
|
||||
```
|
||||
|
||||
Run each one inside the container (after starting it temporarily) or apply
|
||||
them on the host with the same Python environment. Idempotent migrations
|
||||
re-run safely.
|
||||
|
||||
### 2.5 Start the NAS stack
|
||||
|
||||
```bash
|
||||
ssh admin@<nas-lan-ip> \
|
||||
"cd /volume1/docker/terra-view && sudo docker compose up -d"
|
||||
```
|
||||
|
||||
### 2.6 Spot-check
|
||||
|
||||
- Dashboard loads with real units
|
||||
- `/sfm` page lists historical events
|
||||
- A photo loads on a unit detail page
|
||||
- SFM/HB badge mix on the active table matches what you saw on the home
|
||||
server
|
||||
|
||||
If anything's off, see [Rollback plan](#rollback-plan).
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Repoint the watcher (download2-PC)
|
||||
|
||||
The download2-PC is the one client we have to reconfigure. It currently
|
||||
POSTs to the home server. Two endpoints to change:
|
||||
|
||||
1. **terra-view heartbeat URL** —
|
||||
`http://<old-home-ip>:8001/api/series3/heartbeat`
|
||||
→ `http://<new-nas-lan-ip>:8001/api/series3/heartbeat`
|
||||
|
||||
2. **SFM Blastware import URL** —
|
||||
`http://<old-home-ip>:8200/db/import/blastware_file`
|
||||
→ `http://<new-nas-lan-ip>:8200/db/import/blastware_file`
|
||||
|
||||
Or, if you want to keep SFM container-internal and not publish 8200 on
|
||||
the LAN at all, point it through terra-view's existing SFM proxy:
|
||||
→ `http://<new-nas-lan-ip>:8001/api/sfm/db/import/blastware_file`
|
||||
|
||||
Update the config, restart the watcher service, and confirm the next
|
||||
heartbeat lands in the NAS DB (check the Recent Call-Ins card on the
|
||||
dashboard).
|
||||
|
||||
> **Tip:** keep the home server running in parallel for 1–2 days. If you
|
||||
> forget to repoint something, it'll still flow into the old DB and you
|
||||
> can resync.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — External access for remote operators
|
||||
|
||||
Only the terra-view UI needs to be reachable from outside the office. Two
|
||||
clean options — pick one.
|
||||
|
||||
### Option A — Tailscale (recommended for small teams)
|
||||
|
||||
Zero port forwards, zero certs, zero public DNS, zero reverse proxy.
|
||||
|
||||
1. Install Tailscale from Synology Package Center, sign in.
|
||||
2. Install Tailscale on each operator's laptop/phone, sign in to the same
|
||||
tailnet.
|
||||
3. Operators access `http://<nas-tailscale-ip>:8001` from anywhere.
|
||||
|
||||
That's the whole setup. The office network has no external exposure at
|
||||
all.
|
||||
|
||||
### Option B — Reverse proxy with Let's Encrypt
|
||||
|
||||
If you want a `https://terraview.yourdomain.com` URL that any browser can
|
||||
reach:
|
||||
|
||||
#### B.1 Port forward on the office router
|
||||
|
||||
```
|
||||
WAN 443 → <nas-lan-ip>:443
|
||||
WAN 80 → <nas-lan-ip>:80 (only needed for Let's Encrypt HTTP-01;
|
||||
skip if you use DNS-01 challenge)
|
||||
```
|
||||
|
||||
Do **not** forward 1001, 8001, 8100, or 8200.
|
||||
|
||||
#### B.2 Public DNS
|
||||
|
||||
- Free: Synology DDNS (Control Panel → External Access → DDNS) — gives
|
||||
you `something.synology.me`.
|
||||
- Better: your own domain with an A record → office WAN IP, or a CNAME →
|
||||
Synology DDNS hostname (handles dynamic IPs automatically).
|
||||
|
||||
#### B.3 Let's Encrypt certificate
|
||||
|
||||
Control Panel → Security → Certificate → Add → "Get a certificate from
|
||||
Let's Encrypt." DSM handles renewal.
|
||||
|
||||
#### B.4 Synology reverse proxy
|
||||
|
||||
Control Panel → Login Portal → Advanced → Reverse Proxy → Create:
|
||||
|
||||
```
|
||||
Source: Hostname terraview.yourdomain.com
|
||||
Protocol HTTPS
|
||||
Port 443
|
||||
Destination: Hostname localhost
|
||||
Protocol HTTP
|
||||
Port 8001
|
||||
```
|
||||
|
||||
Under "Custom Header", add:
|
||||
|
||||
| Header | Value |
|
||||
|---------------------|------------------------------------|
|
||||
| `X-Forwarded-For` | `$proxy_add_x_forwarded_for` |
|
||||
| `X-Forwarded-Proto` | `$scheme` |
|
||||
| `Host` | `$host` |
|
||||
|
||||
Tick the WebSocket support checkbox.
|
||||
|
||||
#### B.5 DSM firewall
|
||||
|
||||
Control Panel → Security → Firewall → enable:
|
||||
|
||||
- 443/TCP from `Anywhere` — allow
|
||||
- 80/TCP from `Anywhere` — allow (cert renewal only)
|
||||
- Everything else from WAN — deny
|
||||
- All from LAN — allow
|
||||
|
||||
Optional: geo-block to your country if your operators are domestic only.
|
||||
Big reduction in scanning noise.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5 — Decommission home server
|
||||
|
||||
After 1–2 weeks of stable NAS operation:
|
||||
|
||||
1. Take a final `docker compose down` on the home server.
|
||||
2. Archive `/home/serversdown/{terra-view,slmm,seismo-relay}/data/` to a
|
||||
backup volume.
|
||||
3. Free the home server hardware.
|
||||
|
||||
---
|
||||
|
||||
## Verification checklist
|
||||
|
||||
After Phase 2 (data migration):
|
||||
|
||||
- [ ] `http://<nas-lan-ip>:8001/` loads dashboard with real units
|
||||
- [ ] Recent Alerts, Call-Ins (2 cols), Fleet Summary across the top
|
||||
- [ ] SFM/HB badge mix on the active table looks sane
|
||||
- [ ] `/sfm` page lists historical events (the same count as before)
|
||||
- [ ] A unit detail page loads with photos rendering
|
||||
- [ ] `/api/recent-event-callins` returns 200 with real data
|
||||
- [ ] `/api/status-snapshot` returns 200, `sfm_reachable: true`
|
||||
|
||||
After Phase 3 (watcher cutover):
|
||||
|
||||
- [ ] Next heartbeat from download2-PC lands in NAS DB
|
||||
- [ ] A new event arrives in `/sfm` page on the NAS within the next
|
||||
Blastware ACH cycle
|
||||
- [ ] No errors in `docker logs terra-view-terra-view-1`
|
||||
|
||||
After Phase 4 (external access):
|
||||
|
||||
- [ ] (Option A) Operator laptop on tailnet can reach
|
||||
`http://<nas-tailscale-ip>:8001`
|
||||
- [ ] (Option B) `https://terraview.yourdomain.com` resolves, cert is
|
||||
valid, dashboard loads
|
||||
- [ ] (Option B) Office DSM admin (5001) is **not** reachable from outside
|
||||
|
||||
---
|
||||
|
||||
## Rollback plan
|
||||
|
||||
The home server stays alive in parallel through Phases 2–3 as a safety
|
||||
net. If anything goes wrong on the NAS:
|
||||
|
||||
1. On the home server:
|
||||
```bash
|
||||
cd /home/serversdown/terra-view
|
||||
docker compose up -d
|
||||
```
|
||||
2. Point download2-PC back at the home server IP.
|
||||
3. NAS data isn't lost — it's just sitting idle. Investigate, fix, retry.
|
||||
|
||||
The "irreversible" point is when you decommission the home server in
|
||||
Phase 5. Until then, you can always fall back.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas
|
||||
|
||||
1. **Synology UID/GID quirks.** Synology admin is usually `1026:100`.
|
||||
Containers running as root inside don't care, but if your compose
|
||||
files set `user:`, mismatched UIDs cause SQLite "readonly database"
|
||||
errors. Easiest fix: omit `user:` and let containers run as root.
|
||||
|
||||
2. **`network_mode: host` for SLMM.** Required for LAN-direct comms with
|
||||
sound level meters. On Synology this binds to the NAS's interface —
|
||||
confirm nothing else on the NAS uses ports 8100 or 21 (FTP).
|
||||
|
||||
3. **Auto-start on boot.** Container Manager → Project → Settings →
|
||||
enable "Auto-restart". Otherwise a DSM update or NAS reboot drops the
|
||||
stack.
|
||||
|
||||
4. **`restart: unless-stopped` in compose.** Verify every service has it.
|
||||
DSM occasionally restarts Docker during DSM updates — this flag
|
||||
ensures everything comes back.
|
||||
|
||||
5. **Hyper Backup.** Schedule a daily snapshot of
|
||||
`/volume1/docker/terra-view/data/` to a USB drive or off-site. SQLite
|
||||
+ small photo dir = trivially small backups. The DB-Management UI's
|
||||
built-in snapshots are an additional layer but not a replacement.
|
||||
|
||||
6. **NAT loopback (Option B only).** If your office router doesn't
|
||||
support hairpinning, machines INSIDE the office can't reach the NAS
|
||||
by its public hostname — they have to use the LAN IP. Most modern
|
||||
routers handle this; some ISP-provided ones don't. Test from a laptop
|
||||
on the office Wi-Fi.
|
||||
|
||||
7. **Let's Encrypt rate limits (Option B only).** 5 issuances per domain
|
||||
per week. Don't fat-finger DNS or you'll be locked out. Test with the
|
||||
staging endpoint first if unsure.
|
||||
|
||||
8. **`host.docker.internal` resolution.** terra-view's
|
||||
`SFM_BASE_URL=http://host.docker.internal:8200` relies on Docker's
|
||||
internal DNS. Works on DSM 7.2+ in bridge mode. If you see "name not
|
||||
resolved" errors, fall back to explicit container names with a custom
|
||||
network in compose.
|
||||
|
||||
9. **SFM stale rows.** The SFM SQLite has a few rows in `monitor_log`
|
||||
and `ach_sessions` from earlier Python-ACH experiments. Harmless
|
||||
to bring over — invisible to terra-view's UI under the
|
||||
watcher-forward pipeline.
|
||||
|
||||
---
|
||||
|
||||
## Suggested timeline
|
||||
|
||||
For a low-risk migration:
|
||||
|
||||
- **Week 1**: Phase 1. Get the NAS booting an empty stack. No production
|
||||
touch.
|
||||
- **Week 2, day 1**: Phase 2. Migrate data. 10-min window. Keep home
|
||||
server alive in parallel.
|
||||
- **Week 2, day 1**: Phase 3. Repoint download2-PC. Watch heartbeats
|
||||
land on the NAS for the rest of the day.
|
||||
- **Week 3**: Phase 4. Add Tailscale or reverse-proxy access for remote
|
||||
operators.
|
||||
- **Week 4–5**: Monitor. Confirm everything's stable. Then Phase 5
|
||||
(decommission home server).
|
||||
|
||||
Splitting "make it work on LAN" from "expose it remotely" means you
|
||||
debug one thing at a time.
|
||||