diff --git a/.gitignore b/.gitignore index 8458942..abd3d05 100644 --- a/.gitignore +++ b/.gitignore @@ -220,7 +220,6 @@ marimo/_static/ marimo/_lsp/ __marimo__/ -<<<<<<< HEAD # Seismo Fleet Manager # SQLite database files *.db @@ -228,6 +227,3 @@ __marimo__/ /data/ /data-dev/ .aider* -.aider* -======= ->>>>>>> 0c2186f5d89d948b0357d674c0773a67a67d8027 diff --git a/CHANGELOG.md b/CHANGELOG.md index edba477..5aa9db9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,168 @@ All notable changes to Terra-View will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +## [0.14.0] - 2026-06-17 + +Rounds out **sound monitoring** and adds a **client-facing portal**, consolidating four threads since 0.13.x: SLM live monitoring (now on SLMM's shared, cached feed), an automated **FTP night-report pipeline**, a read-only **client portal**, and **per-project password auth** for it. Depends on the matching **SLMM `dev`** build — see Upgrade Notes at the end of each section. + +### SLM live monitoring — fan-out feed + cache-first + +SLM live monitoring — fan-out feed + cache-first reads. The throughline: the NL-43 allows exactly **one** TCP connection at a time, so every page that opened its own device stream (or sent its own `Measure?`/DOD on load) was competing for that single connection — a second viewer saw nothing, and dashboard loads stole polling resolution from the live feed. This release moves Terra-View entirely onto SLMM's shared, cached monitoring: one DOD poll loop per device, fanned out to all viewers; dashboards read SLMM's cache (a DB read on SLMM's side) instead of touching the device; and the live panels populate instantly from cache on open, upgrading to the live WS only on demand. Paired with the SLMM-side work (adaptive poll rate, unreachable backoff, device-offline alert) on SLMM branch `dev`. + +#### Added + +- **Fan-out `/monitor` feed consumption.** The unit live view (`partials/slm_live_view.html`) and the dashboard live tile (`sound_level_meters.html`) now subscribe to SLMM's shared per-device monitor over `WS /api/slmm/{unit}/monitor` instead of each opening its own device stream. Any number of clients attach without each consuming the NL-43's single connection — the "second viewer sees nothing" contention is gone. A WS proxy handler for `/monitor` was added to `backend/routers/slmm.py`. +- **L1/L10 percentile lines + cards.** Both the per-unit live chart and the dashboard card chart now plot L1 (purple) and L10 (orange) alongside Lp/Leq, and the KPI cards show L1/L10. Sourced from the DOD feed's `ln1`/`ln2` (DRD streaming can't carry percentiles, DOD can). Missing/`-.-` values leave a gap rather than dropping the line to 0. +- **Live-chart backfill on open.** Charts seed from SLMM's downsampled DOD trail (`GET /api/slmm/{unit}/history?hours=2`) so a viewer sees recent trend immediately instead of a blank chart that fills one point per second. +- **Live Measurements panel auto-populates from cache.** Opening the dashboard panel fills the KPI cards from cached `/status` and backfills the chart from `/history` — pure cache reads, no device hit. Shows a measuring badge (● Measuring / ■ Stopped) and a freshness stamp ("as of 3:48 PM (10s ago)", amber + "cached" when stale). Re-polls the cache every 15s while open; **Start Live Stream** upgrades to the live WS and no longer wipes the backfilled trail (chart point cap raised 60 → 600). +- **Refresh buttons** — one per device-list row, one in the panel header. On-demand, user-initiated single device read via `GET /api/slmm/{unit}/live` (which also refreshes SLMM's cache), with a spinner + success/error toast, then reloads the device list. +- **Per-unit live-monitoring (keepalive) toggle on `/admin/slmm`** — turns a device's server-side keepalive feed on/off (`POST /monitor/start|stop`), so alerting can keep a device's feed running with no browser attached. + +#### Changed + +- **Dashboard device list + command center read SLMM's cache, not the device.** `slm_dashboard.py`'s `get_slm_units` pulls each unit's cached status from SLMM's `/roster` (one call, a SLMM DB read) for the badge + freshness; the command-center `get_live_view` reads cached `/status` instead of sending `Measure?` + a fresh DOD on every load. This stops dashboard loads from stealing the device's single connection from the live monitor. The elapsed-measurement timer still works because `measurement_start_time` is now included in the cached `/status` response. +- **Device-list freshness reflects real monitoring.** The "Last check" line now uses SLMM's cached `last_seen` (which the monitor advances on every successful poll) via `unit.cache_last_seen`, instead of the `slm_last_check` roster field the monitor never updates. The status badge also treats `Measure` as Measuring, matching the panel and SLMM's cache. +- **Status badge relocated** to the card's bottom meta row (next to "Last check"), off the top-right corner where it collided with the chart/gear/refresh action icons. + +#### Fixed + +- **Deploy/bench threw `can't access property "dispatchEvent", e is null`.** `toggleSLMDeployed()` and the save-config path called `htmx.trigger('#slm-list', 'load')` guarded only by `typeof htmx !== 'undefined'`; no page has a `#slm-list`, so htmx resolved null and called `null.dispatchEvent(...)`. The deploy POST had already succeeded, so the operator saw both the green success **and** a red error. Both call sites now guard on the element existing (`slm_settings_modal.html`). +- **Monitor WS proxy leaked `CancelledError` / "task exception never retrieved"** on stream stop — the cleanup awaited pending tasks but only caught `Exception`, missing `CancelledError` (a `BaseException`). +- **"No recent check-in" shown even on an actively-monitored device** — the row read the stale `slm_last_check` roster field instead of SLMM's live cache (see Changed). +- **L1/L10 KPI cards populated but the chart drew no L1/L10 lines** — the card chart only had Lp + Leq datasets. + +#### Upgrade Notes + +Requires the **matching SLMM build (branch `dev`)** — Terra-View now depends on SLMM's fan-out `/monitor` feed, `/history` trail, `/status` carrying `ln1`/`ln2` + `measurement_start_time`, cached `/roster` status, and the `monitor_enabled` keepalive flag. + +```bash +# SLMM (branch dev) — REBUILD + MIGRATE (or you'll get `no such column: nl43_status.ln1` 500s) +cd /home/serversdown/slmm && docker compose build slmm && docker compose up -d slmm +docker exec terra-view-slmm-1 python3 migrate_add_ln_percentiles.py +docker exec terra-view-slmm-1 python3 migrate_add_monitor_enabled.py + +# Terra-View — NO migration; templates are baked into the image, so rebuild (don't just restart) +cd /home/serversdown/terra-view && docker compose build terra-view && docker compose up -d terra-view +``` + +The two builds must ship **together**. Note the `docker-compose.yml` container was renamed for clarity (now `terra-view-terra-view-1`) — adjust any `docker exec` scripts that referenced the old name. + +--- + +### FTP night-report pipeline *(new)* + +Automated daily morning report of last night's noise (7 PM–7 AM) vs a baseline, +per location, for 24/7 remote sound jobs. The meter records to its SD card +regardless of TCP state, so the report pulls the meter's own stored 15-minute +Leq intervals over FTP (via the SLMM proxy) — accurate, and resilient to a +control-path wedge. **Field-tested on a real NL-43.** + +#### Added + +- **Report engine.** Per-location LAmax / LA01 / LA10 / LA90 / LAeq over Evening + (7–10 PM) + Nighttime (10 PM–7 AM); Leq energy-averaged, percentiles/Lmax + arithmetic; the LN→percentile map is read from the device's own `.rnh`. Two + baseline modes: *captured* (weekly average) and *reference* (typed per-location + limits). +- **Renderers.** HTML email body + Excel attachment (per-NRL interval table + + line chart + Last/Base/Δ summary). +- **Capture cycle.** The daily scheduled "24/7 Continuous" cycle stops → + downloads → ingests → re-indexes → restarts the meter, verifies it resumed + measuring via a fresh DOD read, and retries the restart once before alerting. +- **Standardized ingest.** Manual SD upload, manual FTP "Download & Save", and + the scheduled cycle all funnel through one ingest core: keeps the `.rnh` + + 15-minute Leq, drops the 1-second `_Lp_` files, parses the header, dedupes, and + derives the session's real recording window from the Leq rows. +- **UI.** Night Report button/modal (view / run-and-email / recent reports) and a + per-project Settings panel (enable, time, baseline, recipients, test-email); the + per-NRL Data Files tab now matches the project-wide tab. +- **Config-driven SMTP** sender (`REPORT_SMTP_*`), dry-run when unconfigured. + +#### Fixed + +- **NL-43 sessions stamped `now` / zero-duration.** The NL-43 `.rnh` carries no + measurement timestamps, so the session window is now derived from the Leq rows. + Also fixes NL-43 dedupe (it had keyed on an always-empty start time). +- **"Browse Files" did nothing on the NRL Data Files tab** — the FTP-browser + script's global functions collided with the SLM live-view's (both loaded on that + page); it's now namespaced behind `window.FtpBrowser`. + +#### Upgrade Notes + +- **No DB migration** — the `sound_report_configs` table auto-creates on startup. +- Set `REPORT_SMTP_HOST/PORT/SECURITY/USER/PASSWORD/FROM/RECIPIENTS` to send email + (reports build to `data/reports/…` in dry-run until then). +- To automate a job: a **"24/7 Continuous"** recurring schedule (~7:15 AM) + enable + the report (~8:00 AM) + set a baseline. + +--- + +### Client portal *(new — read-only client-facing view)* + +A scoped, read-only portal at **`/portal/*`** where a client sees only *their* +locations, live. Built inside Terra-View (no new service), reusing the cached +SLMM feed; every route resolves the client through one swappable +`get_current_client` gate, so the interim magic/open-link auth can be replaced +(M4) without touching routes or templates. Strictly read-only — no device control. + +#### Added + +- **Per-client scoping + interim auth.** New `Client`, `ClientAccessToken`, and a + `Project.client_id` FK. A signed (HMAC) session cookie carries the access-token + id, re-validated against the DB each request (revoke kills live sessions, with + server-side expiry). Entry via a magic link (`/portal/enter/{token}`) or a + dev-only plain link (`/portal/open/{id}`, `PORTAL_OPEN_LINKS`, **default off**). +- **Live location view.** KPI cards (Lp/Leq/Lmax/L1/L10) + chart populate + instantly from cache, then upgrade to a real **~1 Hz WebSocket stream** scoped to + the client's unit (a scrubbed bridge to the SLMM fan-out feed). The stream + **auto-closes when the tab is hidden** (Page Visibility) and after a 15-min idle + cap, so an abandoned tab can't pin the device at 1 Hz / burn cellular. +- **Locations overview.** Live status map (level-colored dots, dark/light CARTO + tiles) + a status rollup (live/offline counts, "loudest now"). Leq is the + headline metric. +- **Alerts (config → surface → 24/7).** Threshold-rule config on the SLM detail + page (proxying SLMM's alert CRUD); breach **history + ack** internally and a + read-only, scrubbed history + current-alarm banner + **"your alert limits"** panel + in the portal; enabling a rule pins that device's monitor on so alerts evaluate + round-the-clock. +- **Operator sharing tools.** A **"View client portal"** preview button and a + **"Copy client link"** modal (mint / list / revoke magic links) on the project + page, plus a `backend/portal_admin.py` CLI. +- **Field-instrument design.** Distinctive themed portal — Hanken Grotesk UI + + IBM Plex Mono readouts, panel system, pulsing live dot, staggered reveal — with a + **light/dark toggle** (light default, persisted, no-flash). + +#### Security + +- All scoping enforced server-side (404-not-403, no existence leak); client + endpoints return **scrubbed** projections (no device-health/internal ids); WS + frames whitelisted; operator-set strings HTML-escaped before injection (XSS). + Pre-merge code review hardened cookie expiry, open-links default, and the slug + collision. Remaining hardening (reverse proxy, TLS, `SECRET_KEY`, M4 auth) is + tracked in `docs/CLIENT_PORTAL.md` → "Security hardening backlog". + +#### Upgrade Notes + +- **Migration:** `docker compose exec web-app python3 backend/migrate_add_client_portal.py` + (adds `projects.client_id`; the `clients` / `client_access_tokens` tables + auto-create). +- Set a real **`SECRET_KEY`** in any internet-facing env (signs session cookies), + and keep **`PORTAL_OPEN_LINKS=false`** there. +- Portal alerts depend on the **SLMM `dev`** alert engine (rules/events/evaluator + + cooldown + keepalive coupling) — same build pairing as above. + +--- + +### Portal authentication (Phase 1) +- Each project's client portal is now gated by a **secure per-project link + shared password** (argon2-hashed). Operators manage it from the project page's **Portal access** panel (enable, generate password, copy link). +- Per-project session isolation (a session for one project can't read another's data); brute-force lockout (5 tries / 15 min) on the password gate. +- Retired the interim magic-link / `PORTAL_OPEN_LINKS` open links and the `portal_admin.py mint-link` command. +- **Upgrade:** new `argon2-cffi` dependency → **rebuild the image**, then run `python3 backend/migrate_add_project_portal_auth.py` per DB (adds the `projects.portal_*` columns). `SECRET_KEY` and `COOKIE_SECURE` are now passed through in `docker-compose.yml` (settable via a `.env` file) — set a real `SECRET_KEY` (and `COOKIE_SECURE=true` once on HTTPS) before the portal faces the internet. + +--- + ## [0.13.3] - 2026-06-05 Calibration sync from SFM events. Closes the manual data-entry loop on calibration dates — Terra-View now pulls `device.calibration_date` from each seismograph's most recent event sidecar once a day and updates `RosterUnit.last_calibrated` when the device reports something fresher than what's stored. Manual edits still win when they're newer than the latest event; a fresh event arriving later supersedes the manual edit. Adds a "Sync now" button under Settings → Advanced → Calibration Defaults for on-demand runs, and a `docs/ROADMAP.md` to track in-flight + deferred work. diff --git a/README.md b/README.md index df2cbd5..f06d19d 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -# Terra-View v0.13.3 -Backend API and HTMX-powered web interface for managing a mixed fleet of seismographs and field modems. Track deployments, monitor health in real time, merge roster intent with incoming telemetry, and control your fleet through a unified database and dashboard. +# Terra-View v0.14.0 +Backend API and HTMX-powered web interface for managing a mixed fleet of seismographs, sound level meters, and field modems. Track deployments, monitor health in real time, merge roster intent with incoming telemetry, and control your fleet through a unified database and dashboard. ## Features @@ -18,6 +18,9 @@ Backend API and HTMX-powered web interface for managing a mixed fleet of seismog - **Settings & Safeguards**: `/settings` page exposes roster stats, exports, replace-all imports, and danger-zone reset tools - **Device & Modem Metadata**: Capture calibration windows, modem pairings, phone/IP details, and addresses per unit - **Status Management**: Automatically mark deployed units as OK, Pending (>12h), or Missing (>24h) based on recent telemetry +- **Sound Level Meter Monitoring**: Live per-device monitoring through SLMM's shared, cached feed — multiple viewers without contending for the NL-43's single connection — with L1/L10 percentile lines, a measuring/freshness indicator, and on-demand refresh +- **Automated Night Reports**: Daily per-location noise report (last night vs a baseline) for 24/7 remote sound jobs — pulls the meter's 15-minute Leq over FTP and emails an HTML summary + Excel; the meter is auto-cycled (stop → download → ingest → restart, with restart verification) each morning +- **Client Portal** (`/portal/*`): scoped, read-only, client-facing live view of *their* locations only, gated by a per-project link + shared password (argon2-hashed) - **SFM Event DB Manager** (`/admin/events`): cross-unit event browser with bulk false-trigger flagging and admin-only hard-delete (cleans on-disk binaries + sidecars too) for purging bogus events from misbehaving units - **Deployment-History Calendar + Gantt** (`/tools/deployment-history`): fleet-wide 12-month calendar with side-panel day drill-down, plus "Gantt by Project" / "Gantt by Unit" tabs - **Photo Management**: Upload and view photos for each unit diff --git a/REPORT_PIPELINE_BRIEF.md b/REPORT_PIPELINE_BRIEF.md new file mode 100644 index 0000000..78674fc --- /dev/null +++ b/REPORT_PIPELINE_BRIEF.md @@ -0,0 +1,59 @@ +# FTP Report Pipeline — session brief + +**Branch:** `feat/ftp-report-pipeline` (off `dev`), worktree `/home/serversdown/terra-view-reports`. +**Scope:** Terra-View only. Do NOT touch SLMM — the SLMM alert/monitor work is live in a +parallel session on `slmm` branch `feat/drd-fix`. Pull device data through the **existing** +SLMM FTP proxy endpoints; add no SLMM code (for v1). + +See memory note `client_sound_monitoring_job_2026-07` for the client requirements + timeline. + +## Goal +Automated **daily morning report** for the John Myler 3-location sound job: each AM, last +night's noise levels vs the **baseline week**, per location. Data pulled from the meters via +FTP (the meter records 24/7 to SD regardless of TCP wedges). Alerts are a *separate* workstream +(SLMM, real-time DOD) — not in scope here. + +## The big realization (why this is small) +The hard parts already exist: +- **SLMM (use as-is, via the `/api/slmm/...` proxy):** + - `GET /api/slmm/{unit}/ftp/files?path=/NL-43` → list files/folders + - `POST /api/slmm/{unit}/ftp/download-folder` → returns the `Auto_####` folder as a **ZIP** +- **Terra-View ingest (reuse):** `backend/routers/project_locations.py:1743` `upload_nrl_data` + already accepts a **ZIP**, extracts, keeps `.rnh` + `_Leq_ .rnd` (drops `_Lp_`/junk via + `_is_wanted`), runs `_parse_rnh` (line 1687) → creates `MonitoringSession` + `DataFile`. +- **Report generator (reuse, source-agnostic):** `backend/routers/projects.py`. The `.rnd` + file reads funnel through 3 helpers — `_peek_rnd_headers` (~135), `_is_leq_file` (~147), + `_read_rnd_file_rows` (~256). `.rnd` files live on disk under `data/{file_path}` (DataFile + holds the path, not a BLOB). The stats/Excel/formatting logic doesn't care where bytes come from. + +## Build (Terra-View) +1. **Refactor** `upload_nrl_data`'s core into a callable `ingest_nrl_zip(location_id, zip_bytes, db)` + so it can be invoked programmatically (not only via HTTP UploadFile). +2. **Scheduled pull job** (reuse the existing scheduler): per project location/unit → + `GET /ftp/files` to find new `Auto_####` folders → `POST /ftp/download-folder` (zip) → + `ingest_nrl_zip(...)`. **Dedup** so repeated pulls don't duplicate sessions/files + (track ingested folder names per location). +3. **Baseline aggregation:** aggregate the baseline-week `_Leq_` intervals per location → + reference values (nighttime Leq, L90 floor, typical Lmax). +4. **Nightly report + email:** compute last night's metrics per location, compare to baseline + (deltas), render (reuse the Excel/report machinery), email each morning. + +## Data-location decision (light version, agreed) +Keep `MonitoringSession`/`DataFile` **metadata in TV** for now; reuse the existing on-disk file +store. Optional refinement (later): have SLMM keep the pulled files and TV read them through a +SLMM file-serve endpoint (avoids the copy-into-TV step). Don't do that refinement under the +deadline unless trivial — the report logic is identical either way. + +## Open questions to resolve early +1. **What's actually in a `_Leq_ .rnd`** — Leq only, or Leq + Lmax + Ln per 15-min interval? + Decides whether the night-vs-baseline report can show L90/Lmax or just Leq. Inspect a real file. +2. **Session rollover / dedup** — does a 2-week run write one growing `Auto_####` folder or new + folders? Drives the "what's new" logic. +3. **`download-folder` over a multi-day run** — confirm it zips cleanly (size/time). + +## Client params (confirm with Dave before locking) +Threshold/metric + their "night" window; report recipients + format (email body vs PDF/Excel). + +## Timeline +Setup ~7/1–7/2 (baseline week), shutdown week through ~7/17. Reports needed by ~7/8 (before +shutdown). Today is ~3 weeks out — reliability > features. diff --git a/backend/auth_passwords.py b/backend/auth_passwords.py new file mode 100644 index 0000000..39839a6 --- /dev/null +++ b/backend/auth_passwords.py @@ -0,0 +1,26 @@ +"""Password hashing for the client portal — argon2id via argon2-cffi. + +Kept separate from portal_auth (cookie signing) so the future operator auth can +reuse the same hasher. Never store or log raw passwords.""" +import secrets +from argon2 import PasswordHasher + +_ph = PasswordHasher() + + +def hash_password(raw: str) -> str: + """Return an argon2id hash string for a raw password.""" + return _ph.hash(raw) + + +def verify_password(raw: str, hashed: str) -> bool: + """True iff raw matches the stored hash. Never raises.""" + try: + return _ph.verify(hashed, raw) + except Exception: # argon2 raises on mismatch/garbage; treat all as "no match" + return False + + +def generate_password(n_bytes: int = 12) -> str: + """A strong, URL-safe shareable password (~16 chars for n_bytes=12).""" + return secrets.token_urlsafe(n_bytes) diff --git a/backend/main.py b/backend/main.py index 54792ad..e6b5a17 100644 --- a/backend/main.py +++ b/backend/main.py @@ -4,7 +4,7 @@ from fastapi import FastAPI, Request, Depends, HTTPException from fastapi.middleware.cors import CORSMiddleware from fastapi.staticfiles import StaticFiles from fastapi.templating import Jinja2Templates -from fastapi.responses import HTMLResponse, FileResponse, JSONResponse +from fastapi.responses import HTMLResponse, FileResponse, JSONResponse, RedirectResponse from fastapi.exceptions import RequestValidationError from sqlalchemy.orm import Session from typing import List, Dict, Optional @@ -30,7 +30,7 @@ Base.metadata.create_all(bind=engine) ENVIRONMENT = os.getenv("ENVIRONMENT", "production") # Initialize FastAPI app -VERSION = "0.13.3" +VERSION = "0.14.0" if ENVIRONMENT == "development": _build = os.getenv("BUILD_NUMBER", "0") if _build and _build != "0": @@ -66,6 +66,21 @@ app.mount("/static", StaticFiles(directory="backend/static"), name="static") # Use shared templates configuration with timezone filters from backend.templates_config import templates +# Client-portal auth: an unauthenticated portal request renders the access page +# (HTML routes) or returns 401 JSON (/portal/api/* routes). Centralized so every +# portal route can simply Depends(get_current_client). +from backend.portal_auth import PortalAuthError + +@app.exception_handler(PortalAuthError) +async def portal_auth_handler(request: Request, exc: PortalAuthError): + if request.url.path.startswith("/portal/api"): + return JSONResponse(status_code=401, content={"detail": "Not authenticated"}) + return templates.TemplateResponse( + "portal/access_required.html", + {"request": request, "reason": "required"}, + status_code=401, + ) + # Add custom context processor to inject environment variable into all templates @app.middleware("http") async def add_environment_to_context(request: Request, call_next): @@ -97,6 +112,10 @@ app.include_router(slmm.router) app.include_router(slm_ui.router) app.include_router(slm_dashboard.router) app.include_router(seismo_dashboard.router) + +# Client portal (read-only, scoped client view) — see docs/CLIENT_PORTAL.md +from backend.routers import portal +app.include_router(portal.router) app.include_router(sfm.router) app.include_router(modem_dashboard.router) @@ -148,6 +167,10 @@ app.include_router(deployments.router) from backend.routers import calibration app.include_router(calibration.router) +# Nightly sound-report pipeline (manual triggers; scheduled tick reuses run_nightly_report) +from backend.routers import reports +app.include_router(reports.router) + # Start scheduler service and device status monitor on application startup from backend.services.scheduler import start_scheduler, stop_scheduler from backend.services.device_status_monitor import start_device_status_monitor, stop_device_status_monitor @@ -390,10 +413,82 @@ async def project_detail_page(request: Request, project_id: str): """Project detail dashboard""" return templates.TemplateResponse("projects/detail.html", { "request": request, - "project_id": project_id + "project_id": project_id, }) +@app.get("/projects/{project_id}/portal-preview") +async def project_portal_preview(project_id: str, db: Session = Depends(get_db)): + """Operator testing shortcut: open this project's client portal (no CLI).""" + from backend.models import Project + from backend.portal_auth import mint_portal_session, make_session_cookie, COOKIE_NAME, COOKIE_MAX_AGE, COOKIE_SECURE + project = db.query(Project).filter_by(id=project_id).first() + if not project: + return JSONResponse(status_code=404, content={"detail": "Project not found"}) + token_id = mint_portal_session(project, db) + resp = RedirectResponse(url="/portal", status_code=303) + resp.set_cookie(COOKIE_NAME, make_session_cookie(token_id), + max_age=COOKIE_MAX_AGE, httponly=True, samesite="lax", secure=COOKIE_SECURE) + return resp + + +@app.get("/projects/{project_id}/portal-access") +async def project_portal_access_state(project_id: str, request: Request, db: Session = Depends(get_db)): + """Current portal-access state for the operator panel.""" + from backend.models import Project + p = db.query(Project).filter_by(id=project_id).first() + if not p: + return JSONResponse(status_code=404, content={"detail": "Project not found"}) + link_url = (str(request.base_url).rstrip("/") + f"/portal/p/{p.portal_link_token}") \ + if (p.portal_enabled and p.portal_link_token) else None + return {"enabled": bool(p.portal_enabled), "has_password": bool(p.portal_password_hash), + "link_url": link_url} + + +@app.post("/projects/{project_id}/portal-access/enable") +async def project_portal_access_enable(project_id: str, request: Request, db: Session = Depends(get_db)): + """Turn the portal on; mint a link token if one doesn't exist yet.""" + import secrets + from backend.models import Project + p = db.query(Project).filter_by(id=project_id).first() + if not p: + return JSONResponse(status_code=404, content={"detail": "Project not found"}) + if not p.portal_link_token: + p.portal_link_token = secrets.token_urlsafe(24) + p.portal_enabled = True + db.commit() + link_url = str(request.base_url).rstrip("/") + f"/portal/p/{p.portal_link_token}" + return {"enabled": True, "has_password": bool(p.portal_password_hash), "link_url": link_url} + + +@app.post("/projects/{project_id}/portal-access/password") +async def project_portal_access_password(project_id: str, db: Session = Depends(get_db)): + """Generate a fresh strong password, store its hash, return the raw once.""" + from backend.models import Project + from backend.auth_passwords import hash_password, generate_password + p = db.query(Project).filter_by(id=project_id).first() + if not p: + return JSONResponse(status_code=404, content={"detail": "Project not found"}) + raw = generate_password() + p.portal_password_hash = hash_password(raw) + db.commit() + return {"password": raw} + + +@app.post("/projects/{project_id}/portal-access/disable") +async def project_portal_access_disable(project_id: str, db: Session = Depends(get_db)): + """Turn the portal off and rotate the link token (kills the old link).""" + import secrets + from backend.models import Project + p = db.query(Project).filter_by(id=project_id).first() + if not p: + return JSONResponse(status_code=404, content={"detail": "Project not found"}) + p.portal_enabled = False + p.portal_link_token = secrets.token_urlsafe(24) # rotate so the old link 404s + db.commit() + return {"enabled": False} + + @app.get("/projects/{project_id}/nrl/{location_id}", response_class=HTMLResponse) async def nrl_detail_page( request: Request, diff --git a/backend/migrate_add_client_portal.py b/backend/migrate_add_client_portal.py new file mode 100644 index 0000000..bc66f72 --- /dev/null +++ b/backend/migrate_add_client_portal.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python3 +""" +Database migration: Client Portal (M1). + +Adds the authoritative client link to projects: + - projects.client_id (TEXT, nullable) -> clients.id + +The `clients` and `client_access_tokens` tables are created automatically by +SQLAlchemy `create_all` at app startup (they're brand-new tables), so this +migration only handles the column that create_all won't add to an existing +`projects` table. + +Run once per database: + docker exec terra-view-terra-view-1 python3 backend/migrate_add_client_portal.py +""" + +import sqlite3 +from pathlib import Path + + +def migrate(): + possible_paths = [ + Path("data/seismo_fleet.db"), + Path("data/sfm.db"), + Path("data/seismo.db"), + ] + db_path = next((p for p in possible_paths if p.exists()), None) + if db_path is None: + print(f"Database not found in any of: {[str(p) for p in possible_paths]}") + print("A fresh DB created via models.py will include projects.client_id automatically.") + return + + print(f"Using database: {db_path}") + conn = sqlite3.connect(db_path) + cursor = conn.cursor() + + cursor.execute("PRAGMA table_info(projects)") + existing = {row[1] for row in cursor.fetchall()} + + if "client_id" not in existing: + try: + cursor.execute("ALTER TABLE projects ADD COLUMN client_id TEXT") + print("✓ Added column: projects.client_id (TEXT)") + except sqlite3.OperationalError as e: + print(f"✗ Failed to add projects.client_id: {e}") + else: + print("○ Column already exists: projects.client_id") + + conn.commit() + conn.close() + print("\n✓ Client-portal migration complete.") + print(" Note: `clients` + `client_access_tokens` tables auto-create on app startup.") + + +if __name__ == "__main__": + migrate() diff --git a/backend/migrate_add_project_portal_auth.py b/backend/migrate_add_project_portal_auth.py new file mode 100644 index 0000000..68ce2d4 --- /dev/null +++ b/backend/migrate_add_project_portal_auth.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +""" +Database migration: Project portal auth (Phase 1). + +Adds the per-project portal gate columns to `projects`: + - portal_enabled (BOOLEAN, default 0) + - portal_password_hash (TEXT, nullable) + - portal_link_token (TEXT, nullable) [+ unique index] + +Idempotent. Run once per existing DB: + docker exec terra-view-terra-view-1 python3 backend/migrate_add_project_portal_auth.py +""" +import sqlite3 +from pathlib import Path + +_COLUMNS = { + "portal_enabled": "BOOLEAN DEFAULT 0", + "portal_password_hash": "TEXT", + "portal_link_token": "TEXT", +} + + +def migrate(): + possible_paths = [Path("data/seismo_fleet.db"), Path("data/sfm.db"), Path("data/seismo.db")] + db_path = next((p for p in possible_paths if p.exists()), None) + if db_path is None: + print(f"Database not found in any of: {[str(p) for p in possible_paths]}") + print("A fresh DB created via models.py will include these columns automatically.") + return + + print(f"Using database: {db_path}") + conn = sqlite3.connect(db_path) + cursor = conn.cursor() + cursor.execute("PRAGMA table_info(projects)") + existing = {row[1] for row in cursor.fetchall()} + + for col, ddl in _COLUMNS.items(): + if col in existing: + print(f"○ Column already exists: projects.{col}") + continue + try: + cursor.execute(f"ALTER TABLE projects ADD COLUMN {col} {ddl}") + print(f"✓ Added column: projects.{col} ({ddl})") + except sqlite3.OperationalError as e: + print(f"✗ Failed to add projects.{col}: {e}") + + # Unique index on the link token (separate from ADD COLUMN; idempotent via IF NOT EXISTS). + try: + cursor.execute("CREATE UNIQUE INDEX IF NOT EXISTS ix_projects_portal_link_token " + "ON projects (portal_link_token)") + print("✓ Ensured unique index: ix_projects_portal_link_token") + except sqlite3.OperationalError as e: + print(f"✗ Failed to create index: {e}") + + conn.commit() + conn.close() + print("\n✓ Project portal-auth migration complete.") + + +if __name__ == "__main__": + migrate() diff --git a/backend/models.py b/backend/models.py index aae3039..8de63e8 100644 --- a/backend/models.py +++ b/backend/models.py @@ -192,6 +192,11 @@ class Project(Base): # Project metadata client_name = Column(String, nullable=True, index=True) # Client name (e.g., "PJ Dick") + client_id = Column(String, nullable=True, index=True) # FK -> clients.id; authoritative portal link (client_name kept for display) + # --- Client portal (Phase 1: per-project link + password gate) --- + portal_enabled = Column(Boolean, default=False) # is the portal open for this project + portal_password_hash = Column(String, nullable=True) # argon2 hash of the shared password + portal_link_token = Column(String, nullable=True, unique=True, index=True) # unguessable token in the secure link site_address = Column(String, nullable=True) site_coordinates = Column(String, nullable=True) # "lat,lon" start_date = Column(Date, nullable=True) @@ -218,6 +223,35 @@ class ProjectModule(Base): __table_args__ = (UniqueConstraint("project_id", "module_type", name="uq_project_module"),) +class SoundReportConfig(Base): + """ + Per-project configuration for the automated nightly sound report + (FTP report pipeline). One row per project. Read by the morning tick in + SchedulerService and by the manual /reports endpoints (as defaults). + + New table → created by Base.metadata.create_all() on startup; no migration + needed (only a rebuild/restart). + """ + __tablename__ = "sound_report_configs" + + id = Column(String, primary_key=True, default=lambda: __import__('uuid').uuid4().__str__()) + project_id = Column(String, nullable=False, index=True, unique=True) # FK to projects.id + + enabled = Column(Boolean, default=False, nullable=False) # run the daily report? + report_time = Column(String, default="08:00", nullable=False) # local HH:MM to run/send + metric_keys = Column(String, default="lmax,l01,l10,l90", nullable=False) # csv of metric keys + # Baseline source: "captured" = compute from recorded nights in the date range below; + # "reference" = use fixed values typed per location (old-report averages or a spec limit). + baseline_mode = Column(String, default="captured", nullable=False) + baseline_start = Column(Date, nullable=True) # captured-mode range + baseline_end = Column(Date, nullable=True) + recipients = Column(Text, nullable=True) # csv; falls back to REPORT_SMTP_RECIPIENTS env + last_run_date = Column(Date, nullable=True) # evening-date of the last reported night (dedup) + + created_at = Column(DateTime, default=datetime.utcnow) + updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow) + + class MonitoringLocation(Base): """ Monitoring locations: generic location for monitoring activities. @@ -704,3 +738,37 @@ class PendingDeployment(Base): created_at = Column(DateTime, default=datetime.utcnow) updated_at = Column(DateTime, default=datetime.utcnow, onupdate=datetime.utcnow) + + +# ============================================================================ +# CLIENT PORTAL — read-only, scoped client access (see docs/CLIENT_PORTAL.md) +# ============================================================================ + +class Client(Base): + """A portal client (customer org). Owns one or more Projects via + Project.client_id; their portal surfaces only those projects' locations. + Read-only — clients never control devices.""" + __tablename__ = "clients" + + id = Column(String, primary_key=True, index=True) # UUID + name = Column(String, nullable=False) # display name, e.g. "PJ Dick" + slug = Column(String, nullable=False, unique=True, index=True) # URL-safe handle + contact_email = Column(String, nullable=True) # for M4 magic-link + active = Column(Boolean, default=True) # False = portal access off + created_at = Column(DateTime, default=datetime.utcnow) + + +class ClientAccessToken(Base): + """Interim 'magic URL' gate (M1-M3). The raw secret lives in the link and is + shown once on creation; only its sha256 is stored here. Revoke by setting + revoked_at. In M4 this is replaced behind get_current_client() without + touching routes/templates.""" + __tablename__ = "client_access_tokens" + + id = Column(String, primary_key=True, index=True) # UUID + client_id = Column(String, nullable=False, index=True) # FK -> clients.id + token_hash = Column(String, nullable=False, index=True) # sha256 hex of the secret + label = Column(String, nullable=True) # e.g. "Dave's link" + created_at = Column(DateTime, default=datetime.utcnow) + last_used_at = Column(DateTime, nullable=True) + revoked_at = Column(DateTime, nullable=True) # set = link no longer works diff --git a/backend/portal_admin.py b/backend/portal_admin.py new file mode 100644 index 0000000..b4fc468 --- /dev/null +++ b/backend/portal_admin.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +""" +Client-portal admin CLI (M1). Operator tooling — run inside the terra-view +container against the live DB. The raw magic-link token is shown ONCE on mint; +only its hash is stored. + + # create a client + python3 backend/portal_admin.py create-client --name "Myler Co" --slug myler [--email dave@x.com] + + # attach a project to a client (sets Project.client_id) — by id, number, or name + python3 backend/portal_admin.py link-project --slug myler --project-id + python3 backend/portal_admin.py link-project --slug myler --project-number 2567-23 + python3 backend/portal_admin.py link-project --slug myler --project-name "RKM Hall" + + # mint-link is RETIRED — per-client magic URLs (/portal/enter) no longer exist. + # Client access is now per-PROJECT + password: open the project's page in + # Terra-View → "Portal access" to enable it, generate a password, and copy + # the /portal/p/ link. (create-client / link-project / list / revoke + # still operate on the underlying Client/token rows.) + + # list clients, their projects, and active links + python3 backend/portal_admin.py list + + # revoke a link (stops the link AND any live session it minted) + python3 backend/portal_admin.py revoke --token-id +""" + +import os +import sys +import uuid +import argparse +from datetime import datetime + +# Allow `python3 backend/portal_admin.py ...` (which puts backend/ on sys.path[0], +# hiding the `backend` package) in addition to `python3 -m backend.portal_admin`. +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) + +from backend.database import SessionLocal +from backend.models import Client, ClientAccessToken, Project + + +def _get_client(db, slug): + c = db.query(Client).filter_by(slug=slug).first() + if not c: + sys.exit(f"No client with slug '{slug}'. Create it first.") + return c + + +def create_client(args): + db = SessionLocal() + try: + if db.query(Client).filter_by(slug=args.slug).first(): + sys.exit(f"A client with slug '{args.slug}' already exists.") + c = Client(id=str(uuid.uuid4()), name=args.name, slug=args.slug, + contact_email=args.email, active=True) + db.add(c) + db.commit() + print(f"✓ Created client '{c.name}' (slug={c.slug}, id={c.id})") + print(" Next: link-project, then mint-link.") + finally: + db.close() + + +def link_project(args): + db = SessionLocal() + try: + c = _get_client(db, args.slug) + q = db.query(Project) + if args.project_id: + p = q.filter_by(id=args.project_id).first() + elif args.project_number: + p = q.filter_by(project_number=args.project_number).first() + elif args.project_name: + p = q.filter_by(name=args.project_name).first() + else: + sys.exit("Specify --project-id, --project-number, or --project-name.") + if not p: + sys.exit("Project not found.") + p.client_id = c.id + db.commit() + print(f"✓ Linked project '{p.name}' (id={p.id}) -> client '{c.name}'") + finally: + db.close() + + +def mint_link(args): + # Retired: the per-client magic URL (/portal/enter/...) was removed when the + # portal moved to per-project + password access. Minting a token here would + # only produce a dead link. + sys.exit( + "mint-link is retired: per-client magic URLs (/portal/enter/...) no longer exist.\n" + "Client access is now per-project + password. In Terra-View, open the project's page →\n" + "'Portal access' to enable the portal, generate a password, and copy the /portal/p/\n" + "link to send the client." + ) + + +def revoke(args): + db = SessionLocal() + try: + tok = db.query(ClientAccessToken).filter_by(id=args.token_id).first() + if not tok: + sys.exit("No token with that id.") + if tok.revoked_at: + print("○ Already revoked.") + return + tok.revoked_at = datetime.utcnow() + db.commit() + print(f"✓ Revoked token {tok.id} — the link and any live sessions it minted are dead.") + finally: + db.close() + + +def list_all(args): + db = SessionLocal() + try: + clients = db.query(Client).order_by(Client.name).all() + if not clients: + print("No clients yet.") + return + for c in clients: + state = "" if c.active else " [INACTIVE]" + print(f"\n● {c.name} (slug={c.slug}){state}") + projs = db.query(Project).filter_by(client_id=c.id).all() + print(" projects: " + (", ".join(p.name for p in projs) or "(none linked)")) + toks = db.query(ClientAccessToken).filter_by(client_id=c.id).all() + if not toks: + print(" links: (none — run mint-link)") + for t in toks: + status = "revoked" if t.revoked_at else "active" + last = t.last_used_at.strftime("%Y-%m-%d %H:%M") if t.last_used_at else "never used" + print(f" link {t.id} [{status}] {t.label or ''} (last: {last})") + print() + finally: + db.close() + + +def main(): + ap = argparse.ArgumentParser(description="Client-portal admin (M1)") + sub = ap.add_subparsers(dest="cmd", required=True) + + p = sub.add_parser("create-client"); p.add_argument("--name", required=True) + p.add_argument("--slug", required=True); p.add_argument("--email"); p.set_defaults(fn=create_client) + + p = sub.add_parser("link-project"); p.add_argument("--slug", required=True) + p.add_argument("--project-id"); p.add_argument("--project-number"); p.add_argument("--project-name") + p.set_defaults(fn=link_project) + + p = sub.add_parser("mint-link"); p.add_argument("--slug", required=True) + p.add_argument("--label"); p.set_defaults(fn=mint_link) + + p = sub.add_parser("revoke"); p.add_argument("--token-id", required=True); p.set_defaults(fn=revoke) + + p = sub.add_parser("list"); p.set_defaults(fn=list_all) + + args = ap.parse_args() + args.fn(args) + + +if __name__ == "__main__": + main() diff --git a/backend/portal_auth.py b/backend/portal_auth.py new file mode 100644 index 0000000..39c9255 --- /dev/null +++ b/backend/portal_auth.py @@ -0,0 +1,193 @@ +""" +Client-portal auth — the swappable gate (see docs/CLIENT_PORTAL.md). + +M1-M3 ride on an interim signed "magic URL": an unguessable token in the link +mints a signed session cookie. Every portal route depends on get_current_client(); +M4 replaces the backing (magic-link / accounts) without touching routes/templates. + +The cookie carries the ACCESS-TOKEN id (not the client id) and is re-validated +against the DB on every request, so revoking a link (revoked_at) kills its live +sessions on the next request — not just future clicks. + +No new dependency: the cookie is signed with stdlib HMAC-SHA256 over a SECRET_KEY. +""" + +import os +import hmac +import json +import time +import uuid +import base64 +import hashlib +import logging +import secrets + +from fastapi import Request, Depends +from sqlalchemy.orm import Session + +from backend.database import get_db +from backend.models import Client, ClientAccessToken, Project + +logger = logging.getLogger(__name__) + +# Signing secret for portal session cookies. MUST be set to a real secret in prod +# (env). The insecure default only exists so dev/test boots without config. +SECRET_KEY = os.getenv("SECRET_KEY", "dev-insecure-change-me") +if SECRET_KEY == "dev-insecure-change-me": + logger.warning("[PORTAL] SECRET_KEY is the insecure default — set SECRET_KEY in prod.") + +COOKIE_NAME = "portal_session" +COOKIE_MAX_AGE = 60 * 60 * 24 * 30 # 30 days +# Set COOKIE_SECURE=true once the portal is served over HTTPS (TLS terminates at +# the Synology reverse proxy). Default false so plain-HTTP dev still works. +COOKIE_SECURE = os.getenv("COOKIE_SECURE", "false").lower() in ("1", "true", "yes") + + +class PortalAuthError(Exception): + """Raised by get_current_client when there's no valid portal session. + Handled centrally in main.py: HTML routes get the access-required page, + /portal/api/* routes get a 401 JSON.""" + + +# -- token + cookie primitives ---------------------------------------------- + +def hash_token(raw: str) -> str: + """sha256 hex of a raw access-token secret (what we store + look up by).""" + return hashlib.sha256(raw.encode()).hexdigest() + + +def _sign(body: str) -> str: + return hmac.new(SECRET_KEY.encode(), body.encode(), hashlib.sha256).hexdigest() + + +def make_session_cookie(token_id: str) -> str: + body = base64.urlsafe_b64encode( + json.dumps({"tid": token_id, "iat": int(time.time())}).encode() + ).decode() + return f"{body}.{_sign(body)}" + + +def _read_session_cookie(value: str): + """Return the token id from a signed cookie, or None if missing/tampered.""" + try: + body, sig = value.rsplit(".", 1) + except (ValueError, AttributeError): + return None + if not hmac.compare_digest(sig, _sign(body)): + return None + try: + data = json.loads(base64.urlsafe_b64decode(body.encode())) + if not isinstance(data, dict): + return None + # Server-side expiry: a leaked cookie isn't valid forever (max_age is only a + # browser hint). iat is set by make_session_cookie. + iat = data.get("iat") + if not isinstance(iat, (int, float)) or (time.time() - iat) > COOKIE_MAX_AGE: + return None + return data.get("tid") + except Exception: + return None + + +# -- the dependency every portal route uses --------------------------------- + +def client_from_cookie(cookie_value, db: Session): + """Resolve a Client from a raw session-cookie value, or None. Re-validates the + access token against the DB each call, so a revoked link / disabled client + drops immediately. Shared by the HTTP dependency and the WebSocket handler + (which can't use Request-based Depends).""" + token_id = _read_session_cookie(cookie_value) if cookie_value else None + if not token_id: + return None + tok = db.query(ClientAccessToken).filter_by(id=token_id, revoked_at=None).first() + if not tok: + return None + return db.query(Client).filter_by(id=tok.client_id, active=True).first() + + +def get_current_client(request: Request, db: Session = Depends(get_db)) -> Client: + """Resolve the authenticated client, or raise PortalAuthError.""" + client = client_from_cookie(request.cookies.get(COOKIE_NAME), db) + if client is None: + raise PortalAuthError() + return client + + +# --- Phase-1 per-project password gate ------------------------------------------- +# A portal-enabled project gets its OWN dedicated client (slug "portal-") +# owning exactly that project. The project is linked to it via project.client_id so +# the existing client-scoped routes (which resolve projects by Project.client_id == +# client.id) surface exactly this one project for the portal session — per-project +# isolation with no route changes. (Phase 1 repurposes project.client_id for this; a +# real per-client model is the deferred multi-tenant work.) + + +def portal_client_for_project(project, db) -> Client: + """Get-or-create the dedicated 1:1 portal client for a project, and link the + project to it so the client-scoped routes resolve exactly this project.""" + slug = f"portal-{project.id}" + client = db.query(Client).filter_by(slug=slug).first() + if client is None: + client = Client(id=str(uuid.uuid4()), + name=(project.client_name or project.name or "Client"), + slug=slug, active=True) + db.add(client) + db.flush() + if project.client_id != client.id: + project.client_id = client.id # without this, the client owns no projects + db.flush() + return client + + +def mint_portal_session(project, db) -> str: + """Ensure the project's portal client + an access token exist; return the token + id to seal into a session cookie. Reuses an existing token to avoid clutter.""" + client = portal_client_for_project(project, db) + tok = db.query(ClientAccessToken).filter_by(client_id=client.id, revoked_at=None).first() + if tok is None: + tok = ClientAccessToken(id=str(uuid.uuid4()), client_id=client.id, + token_hash=hash_token(secrets.token_urlsafe(32)), + label="portal") + db.add(tok) + db.commit() + return tok.id + + +def resolve_project_by_link_token(link_token: str, db): + """Return the portal-enabled Project for a link token, or None.""" + if not link_token: + return None + return db.query(Project).filter_by( + portal_link_token=link_token, portal_enabled=True).first() + + +# In-memory brute-force lockout, keyed per link_token (the password is shared per +# project, so per-IP granularity buys nothing and an IP term only lets an attacker +# reset the budget by rotating source IPs). Resets on restart; adequate for a +# read-only surface behind the UniFi edge. Single-worker dev; multi-worker would +# need a shared store. +MAX_ATTEMPTS = 5 +LOCK_SECONDS = 15 * 60 +_failures: dict = {} # key -> (count, first_failure_epoch) + + +def is_locked(key: str) -> bool: + rec = _failures.get(key) + if not rec: + return False + count, first = rec + if count < MAX_ATTEMPTS: + return False + if (time.time() - first) > LOCK_SECONDS: + _failures.pop(key, None) # window expired + return False + return True + + +def register_failure(key: str) -> None: + count, first = _failures.get(key, (0, time.time())) + _failures[key] = (count + 1, first) + + +def clear_failures(key: str) -> None: + _failures.pop(key, None) diff --git a/backend/routers/portal.py b/backend/routers/portal.py new file mode 100644 index 0000000..a703f31 --- /dev/null +++ b/backend/routers/portal.py @@ -0,0 +1,375 @@ +""" +Client portal — read-only, scoped client view (see docs/CLIENT_PORTAL.md). + +A client opens a per-project secure link (/portal/p/{link_token}), enters the +shared password, and gets a signed session cookie scoped to that project; they +then see that project's locations (overview) and per-location read-only live +data sourced from SLMM's cache. Every data route re-checks ownership. +""" + +import os +import json +import asyncio +import logging +from datetime import datetime + +import httpx +import websockets +from fastapi import APIRouter, Request, Depends, HTTPException, WebSocket, Form +from fastapi.responses import RedirectResponse +from sqlalchemy import or_ +from sqlalchemy.orm import Session + +from backend.database import get_db, SessionLocal +from backend.models import Client, MonitoringLocation, Project, UnitAssignment +from backend.templates_config import templates +from backend.portal_auth import ( + get_current_client, client_from_cookie, make_session_cookie, + COOKIE_NAME, COOKIE_MAX_AGE, COOKIE_SECURE, + resolve_project_by_link_token, mint_portal_session, + is_locked, register_failure, clear_failures, +) +from backend.auth_passwords import verify_password + +logger = logging.getLogger(__name__) +router = APIRouter(prefix="/portal", tags=["portal"]) + +SLMM_BASE_URL = os.getenv("SLMM_BASE_URL", "http://localhost:8100") +SLMM_WS_BASE_URL = SLMM_BASE_URL.replace("http://", "ws://").replace("https://", "wss://") + +# Whitelist of fields the portal exposes to a client — sound metrics + run state +# only. Internal device health (battery/power/SD/raw_payload) is NOT disclosed. +_PORTAL_LIVE_FIELDS = ("measurement_state", "last_seen", "measurement_start_time", + "lp", "leq", "lmax", "lpeak", "ln1", "ln2") + + +# -- scoping (every data route gates through these) -------------------------- + +def _client_project_ids(client: Client, db: Session) -> list: + return [r[0] for r in db.query(Project.id).filter( + Project.client_id == client.id, Project.status != "deleted").all()] + + +def resolve_client_location(client: Client, location_id: str, db: Session) -> MonitoringLocation: + """Ownership gate: location must be a sound location in one of the client's + active projects. Raises 404 (not 403) for both 'missing' and 'not yours' so + we never leak whether a location exists.""" + loc = db.query(MonitoringLocation).filter_by(id=location_id, removed_at=None).first() + if (not loc or loc.location_type != "sound" + or loc.project_id not in _client_project_ids(client, db)): + raise HTTPException(status_code=404, detail="Location not found") + return loc + + +def active_unit_for_location(location_id: str, db: Session): + """The SLM unit currently assigned to this location, or None.""" + now = datetime.utcnow() + asg = (db.query(UnitAssignment) + .filter(UnitAssignment.location_id == location_id, + UnitAssignment.status == "active", + UnitAssignment.device_type == "slm", + or_(UnitAssignment.assigned_until.is_(None), + UnitAssignment.assigned_until > now)) + .order_by(UnitAssignment.assigned_at.desc()).first()) + return asg.unit_id if asg else None + + +def _client_locations(client: Client, db: Session) -> list: + """The client's active sound locations (for the overview tiles + map).""" + pids = _client_project_ids(client, db) + if not pids: + return [] + projs = {p.id: p.name for p in + db.query(Project.id, Project.name).filter(Project.id.in_(pids)).all()} + locs = (db.query(MonitoringLocation) + .filter(MonitoringLocation.project_id.in_(pids), + MonitoringLocation.location_type == "sound", + MonitoringLocation.removed_at.is_(None)) + .order_by(MonitoringLocation.sort_order, MonitoringLocation.name).all()) + return [{ + "id": loc.id, "name": loc.name, + "address": loc.address, "coordinates": loc.coordinates, + "project_name": projs.get(loc.project_id), + "has_device": active_unit_for_location(loc.id, db) is not None, + } for loc in locs] + + + +@router.get("/logout") +def portal_logout(): + resp = RedirectResponse(url="/portal/access", status_code=303) + resp.delete_cookie(COOKIE_NAME) + return resp + + +@router.get("/access") +def portal_access(request: Request): + """Landing for an unauthenticated visitor (no valid link).""" + return templates.TemplateResponse( + "portal/access_required.html", {"request": request, "reason": "required"} + ) + + +@router.get("/p/{link_token}") +def portal_password_prompt(link_token: str, request: Request, db: Session = Depends(get_db)): + """Secure per-project link: resolve the project from the token, prompt for the + shared password. Generic page if the token is unknown/disabled (no leak).""" + project = resolve_project_by_link_token(link_token, db) + if not project or not project.portal_password_hash: + # unknown token, disabled portal, or enabled-but-no-password-set — all look + # identical to a client (no existence/config leak, no self-lockout on a + # passwordless project). + return templates.TemplateResponse( + "portal/access_required.html", {"request": request, "reason": "invalid"}, + status_code=404) + return templates.TemplateResponse("portal/password.html", { + "request": request, "link_token": link_token, + "project_name": project.name, "error": None}) + + +@router.post("/p/{link_token}") +def portal_password_submit(link_token: str, request: Request, + password: str = Form(...), db: Session = Depends(get_db)): + """Verify the shared password; on success mint a project-scoped session cookie.""" + project = resolve_project_by_link_token(link_token, db) + if not project or not project.portal_password_hash: + # unknown token, disabled portal, or enabled-but-no-password-set — all look + # identical to a client (no existence/config leak, no self-lockout on a + # passwordless project). + return templates.TemplateResponse( + "portal/access_required.html", {"request": request, "reason": "invalid"}, + status_code=404) + + # Shared per-project password → lock per token. (Keying on IP too only enabled a + # bypass via source-IP rotation, and behind the reverse proxy every client shares + # one IP anyway.) + lock_key = link_token + if is_locked(lock_key): + return templates.TemplateResponse("portal/password.html", { + "request": request, "link_token": link_token, "project_name": project.name, + "error": "Too many attempts. Try again in 15 minutes."}, status_code=200) + + if not verify_password(password, project.portal_password_hash): + register_failure(lock_key) + return templates.TemplateResponse("portal/password.html", { + "request": request, "link_token": link_token, "project_name": project.name, + "error": "Incorrect password."}, status_code=200) + + clear_failures(lock_key) + token_id = mint_portal_session(project, db) + resp = RedirectResponse(url="/portal", status_code=303) + resp.set_cookie(COOKIE_NAME, make_session_cookie(token_id), + max_age=COOKIE_MAX_AGE, httponly=True, samesite="lax", secure=COOKIE_SECURE) + logger.info(f"[PORTAL] password ok for project {project.id[:8]} → session opened") + return resp + + +@router.get("") +def portal_home(request: Request, client: Client = Depends(get_current_client), + db: Session = Depends(get_db)): + """Client overview — their active sound locations with live tiles + a map.""" + return templates.TemplateResponse( + "portal/overview.html", + {"request": request, "client": client, + "locations": _client_locations(client, db)}, + ) + + +@router.get("/location/{location_id}") +def portal_location(location_id: str, request: Request, + client: Client = Depends(get_current_client), + db: Session = Depends(get_db)): + """Read-only live view for one of the client's locations (404 if not owned).""" + loc = resolve_client_location(client, location_id, db) + return templates.TemplateResponse("portal/location.html", { + "request": request, "client": client, "location": loc, + "has_device": active_unit_for_location(location_id, db) is not None, + }) + + +# -- scoped data (cache reads only — never hits the device) ------------------ + +@router.get("/api/location/{location_id}/live") +async def portal_location_live(location_id: str, + client: Client = Depends(get_current_client), + db: Session = Depends(get_db)): + """Scrubbed cached live reading for a location the client owns.""" + resolve_client_location(client, location_id, db) + unit_id = active_unit_for_location(location_id, db) + if not unit_id: + return {"status": "ok", "data": None, "reason": "no_device"} + try: + async with httpx.AsyncClient(timeout=5.0) as hc: + r = await hc.get(f"{SLMM_BASE_URL}/api/nl43/{unit_id}/status") + except Exception: + return {"status": "ok", "data": None, "reason": "unreachable"} + if r.status_code != 200: + return {"status": "ok", "data": None, "reason": "no_data"} + full = (r.json() or {}).get("data", {}) or {} + return {"status": "ok", "data": {k: full.get(k) for k in _PORTAL_LIVE_FIELDS}} + + +@router.get("/api/location/{location_id}/history") +async def portal_location_history(location_id: str, hours: float = 2.0, + client: Client = Depends(get_current_client), + db: Session = Depends(get_db)): + """Cached chart trail for a location the client owns. (Trail rows are already + just timestamp + lp/leq/lmax/ln1/ln2 — safe to pass through.)""" + resolve_client_location(client, location_id, db) + unit_id = active_unit_for_location(location_id, db) + if not unit_id: + return {"status": "ok", "readings": []} + hours = max(0.1, min(hours, 48.0)) + try: + async with httpx.AsyncClient(timeout=5.0) as hc: + r = await hc.get(f"{SLMM_BASE_URL}/api/nl43/{unit_id}/history", + params={"hours": hours}) + except Exception: + return {"status": "ok", "readings": []} + if r.status_code != 200: + return {"status": "ok", "readings": []} + raw = (r.json() or {}).get("readings", []) + fields = ("timestamp", "lp", "leq", "lmax", "ln1", "ln2") # whitelist, like the other endpoints + return {"status": "ok", "readings": [{k: x.get(k) for k in fields} for x in raw]} + + +# Whitelist of alert-event fields exposed to a client (no internal ids/ack-by). +_PORTAL_EVENT_FIELDS = ("rule_name", "metric", "threshold_db", "onset_at", + "onset_value", "peak_value", "clear_at", "status") + + +@router.get("/api/location/{location_id}/events") +async def portal_location_events(location_id: str, limit: int = 20, + client: Client = Depends(get_current_client), + db: Session = Depends(get_db)): + """Scrubbed breach history for a location the client owns (read-only).""" + resolve_client_location(client, location_id, db) + unit_id = active_unit_for_location(location_id, db) + if not unit_id: + return {"status": "ok", "events": []} + limit = max(1, min(limit, 100)) + try: + async with httpx.AsyncClient(timeout=5.0) as hc: + r = await hc.get(f"{SLMM_BASE_URL}/api/nl43/{unit_id}/alerts/events", + params={"limit": limit}) + except Exception: + return {"status": "ok", "events": []} + if r.status_code != 200: + return {"status": "ok", "events": []} + raw = (r.json() or {}).get("events", []) + events = [{k: e.get(k) for k in _PORTAL_EVENT_FIELDS} for e in raw] + return {"status": "ok", "events": events, "active": sum(1 for e in events if e.get("status") == "active")} + + +# Whitelist of alert-rule fields shown to a client (the active limits, no cooldown/ +# hysteresis internals). +_PORTAL_RULE_FIELDS = ("name", "metric", "comparison", "threshold_db", "duration_s", + "schedule_start", "schedule_end", "schedule_days") + + +@router.get("/api/location/{location_id}/thresholds") +async def portal_location_thresholds(location_id: str, + client: Client = Depends(get_current_client), + db: Session = Depends(get_db)): + """The active alert limits for a location the client owns (enabled rules only), + so the client can see what they're being alerted on. Read-only, scrubbed.""" + resolve_client_location(client, location_id, db) + unit_id = active_unit_for_location(location_id, db) + if not unit_id: + return {"status": "ok", "rules": []} + try: + async with httpx.AsyncClient(timeout=5.0) as hc: + r = await hc.get(f"{SLMM_BASE_URL}/api/nl43/{unit_id}/alerts/rules") + except Exception: + return {"status": "ok", "rules": []} + if r.status_code != 200: + return {"status": "ok", "rules": []} + raw = (r.json() or {}).get("rules", []) + rules = [{k: x.get(k) for k in _PORTAL_RULE_FIELDS} for x in raw if x.get("enabled")] + return {"status": "ok", "rules": rules} + + +# -- live stream (fan-out feed, scoped + scrubbed) --------------------------- + +def _scrub_frame(raw: str): + """Project a monitor frame down to the portal whitelist. Drops internal fields + (unit_id, raw_payload, lmin) before it reaches a client; passes control fields + (feed_status, heartbeat) + timestamp through. Returns None for a non-JSON frame + so the caller drops it rather than forwarding anything unscrubbed.""" + try: + d = json.loads(raw) + except Exception: + return None + out = {k: d.get(k) for k in _PORTAL_LIVE_FIELDS if k in d} + if "timestamp" in d: + out["timestamp"] = d["timestamp"] + for ctrl in ("feed_status", "heartbeat"): + if ctrl in d: + out[ctrl] = d[ctrl] + return json.dumps(out) + + +@router.websocket("/api/location/{location_id}/stream") +async def portal_location_stream(websocket: WebSocket, location_id: str): + """Live ~1Hz feed for a location the client owns. Auths via the session cookie, + enforces ownership, then bridges the unit's shared SLMM /monitor fan-out feed + to the browser (scrubbed). A viewer is just one more subscriber to the one + device feed — no extra device connection.""" + await websocket.accept() + + # Auth + ownership on a short-lived session, then release it for the long bridge. + db = SessionLocal() + try: + client = client_from_cookie(websocket.cookies.get(COOKIE_NAME), db) + if client is None: + await websocket.close(code=1008) # policy violation (not authenticated) + return + try: + resolve_client_location(client, location_id, db) + except HTTPException: + await websocket.close(code=1008) + return + unit_id = active_unit_for_location(location_id, db) + finally: + db.close() + + if not unit_id: + try: + await websocket.send_json({"feed_status": "no_device"}) + finally: + await websocket.close(code=1000) + return + + target = f"{SLMM_WS_BASE_URL}/api/nl43/{unit_id}/monitor" + backend_ws = None + try: + backend_ws = await websockets.connect(target) + + async def forward_to_client(): + async for message in backend_ws: + frame = _scrub_frame(message) + if frame is not None: + await websocket.send_text(frame) + + async def watch_client(): + while True: + await websocket.receive_text() + + tasks = [asyncio.ensure_future(forward_to_client()), + asyncio.ensure_future(watch_client())] + done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) + for t in pending: + t.cancel() + for t in tasks: + try: + await t + except (asyncio.CancelledError, Exception): + pass + except Exception as e: + logger.warning(f"[PORTAL] stream {location_id}: {e}") + finally: + if backend_ws: + try: + await backend_ws.close() + except Exception: + pass diff --git a/backend/routers/project_locations.py b/backend/routers/project_locations.py index 0b17003..1074a5a 100644 --- a/backend/routers/project_locations.py +++ b/backend/routers/project_locations.py @@ -9,7 +9,7 @@ from fastapi import APIRouter, Request, Depends, HTTPException, Query from fastapi.responses import HTMLResponse, JSONResponse from sqlalchemy.orm import Session from sqlalchemy import and_, or_ -from datetime import datetime +from datetime import datetime, timedelta from zoneinfo import ZoneInfo from typing import Optional import uuid @@ -1712,6 +1712,19 @@ def _parse_rnh(content: bytes) -> dict: result["stop_time_str"] = value elif key == "Total Measurement Time": result["total_time_str"] = value + elif key == "Frequency Weighting (Main)": + result["frequency_weighting"] = value + elif key == "Time Weighting (Main)": + result["time_weighting"] = value + elif key == "Leq Calculation Interval": + result["leq_interval"] = value + elif key.startswith("Percentile "): + # e.g. "Percentile 4,90.0" → percentiles["4"] = "90.0". + # Lets the report label the LN slots (here LN4 = L90) from the + # device's own config instead of hardcoding which slot is which — + # the percentile assignment is reconfigurable per job. + slot = key[len("Percentile "):].strip() + result.setdefault("percentiles", {})[slot] = value except Exception: pass return result @@ -1740,6 +1753,347 @@ def _classify_file(filename: str) -> str: return "data" +def _rnd_interval_seconds(s: Optional[str]) -> Optional[int]: + """Parse an NL-43 interval string ('15m' / '1s' / '1h') into seconds.""" + import re + m = re.match(r"\s*(\d+)\s*([smh])", (s or "").strip().lower()) + if not m: + return None + return int(m.group(1)) * {"s": 1, "m": 60, "h": 3600}[m.group(2)] + + +def _leq_window_local(leq_bytes: bytes): + """Recording window from a Leq .rnd's 'Start Time' column (meter-local time). + + Returns (first_start, last_start, row_count, inferred_interval_seconds). + This is the source of truth for the recording window on NL-43 units, whose + .rnh carries no measurement timestamps. Reuses the report's AU2 normaliser + so NL-43 and AU2 files parse identically. + """ + import csv as _csv + from backend.routers.projects import _normalize_rnd_rows # lazy: avoid import cycle + try: + text = leq_bytes.decode("utf-8", errors="replace") + rows = list(_csv.DictReader(io.StringIO(text))) + except Exception: + return None, None, 0, None + try: + rows, _ = _normalize_rnd_rows(rows) + except Exception: + pass + times = [] + for r in rows: + v = (r.get("Start Time") or "").strip() + try: + times.append(datetime.strptime(v, "%Y/%m/%d %H:%M:%S")) + except (ValueError, TypeError): + continue + if not times: + return None, None, 0, None + times.sort() + inferred = int((times[1] - times[0]).total_seconds()) if len(times) >= 2 else None + return times[0], times[-1], len(times), inferred + + +def _is_wanted_nrl_file(fname: str) -> bool: + """Keep only the files an NRL ingest cares about: .rnh metadata + the + averaged Leq .rnd. Drops the 1-second _Lp_ files and everything else. + + - NL-43 writes two .rnd types: _Leq_ (15-min averages, wanted) and + _Lp_ (1-second granular, skipped). + - AU2 (NL-23/older Rion) writes a single Au2_####.rnd — always keep. + + Note this is purely about which *files* to store, not which *metrics* to + report: the kept Leq file carries every column (Leq, Lmax, L1/L10/L50/ + L90/L95, Lpeak, …), so the report layer can select any metric later. + """ + n = fname.lower() + if n.endswith(".rnh"): + return True + if n.endswith(".rnd"): + if "_leq_" in n: # NL-43 Leq file + return True + if n.startswith("au2_"): # AU2 format (NL-23) — Leq equivalent + return True + if "_lp" not in n and "_leq_" not in n: + # Unknown .rnd format — include it so we don't silently drop data + return True + return False + + +class IngestError(Exception): + """Raised when an NRL upload/ZIP has no usable data or an invalid target. + + Kept HTTP-agnostic so the ingest core can be driven programmatically (the + scheduled FTP pull) as well as from the HTTP upload endpoint. Callers + translate it: the endpoint → HTTP 400, the scheduler → logged failure. + """ + pass + + +def _find_existing_session( + db: Session, + location_id: str, + store_name: str, + started_at, + start_time_str: str, +): + """Return an already-ingested session for this location that represents the + same measurement, or None. + + Used to make FTP re-pulls idempotent: a daily cycle closes one Auto_#### + folder per day, so a session is uniquely identified within a location by + (store_name + measurement start time). Store names recycle across jobs, so + we always match on start time too. + """ + if not store_name and not started_at: + return None + candidates = db.query(MonitoringSession).filter( + MonitoringSession.location_id == location_id, + MonitoringSession.session_type == "sound", + ).all() + for s in candidates: + try: + meta = json.loads(s.session_metadata or "{}") + except (json.JSONDecodeError, TypeError): + meta = {} + if store_name and meta.get("store_name") != store_name: + continue + # Same store_name — confirm it's the same measurement by start time. + if start_time_str and meta.get("start_time_str") == start_time_str: + return s + if not meta.get("start_time_str") and started_at and s.started_at == started_at: + return s + return None + + +def _ingest_file_entries( + location: MonitoringLocation, + file_entries: list[tuple[str, bytes]], + db: Session, + *, + source: str = "manual_upload", + dedupe: bool = False, + unit_id: Optional[str] = None, +) -> dict: + """Core NRL ingest, shared by the HTTP upload and the programmatic FTP pull. + + Takes already-normalized (filename, bytes) entries, keeps the wanted files, + parses the .rnh, and creates a MonitoringSession + DataFile rows under the + location's project. Metric-agnostic: the full Leq file is written to disk + and every column preserved; metric selection happens in the report layer. + + `unit_id` attributes the session to the recording unit when the caller knows + it (manual FTP download / SD upload from a known unit). Left None for paths + that link the unit afterwards (the scheduler's `_ingest_and_link`). + + Raises IngestError if no usable files are present. + """ + # --- Filter to the files we keep (.rnh + Leq .rnd) --- + file_entries = [(f, b) for f, b in file_entries if _is_wanted_nrl_file(f)] + if not file_entries: + raise IngestError( + "No usable .rnd or .rnh files found. Expected NL-43 _Leq_ files or AU2 format .rnd files." + ) + + # --- Parse .rnh metadata (first one wins) --- + rnh_meta = {} + for fname, fbytes in file_entries: + if fname.lower().endswith(".rnh"): + rnh_meta = _parse_rnh(fbytes) + break + + # RNH stores local time (no UTC offset). Use local for period/label, then + # convert to UTC for storage so the local_datetime filter displays correctly. + started_at_local = _parse_rnh_datetime(rnh_meta.get("start_time_str")) or datetime.utcnow() + stopped_at_local = _parse_rnh_datetime(rnh_meta.get("stop_time_str")) + started_at = local_to_utc(started_at_local) + stopped_at = local_to_utc(stopped_at_local) if stopped_at_local else None + duration_seconds = ( + int((stopped_at - started_at).total_seconds()) + if (started_at and stopped_at) else None + ) + + store_name = rnh_meta.get("store_name", "") + serial_number = rnh_meta.get("serial_number", "") + index_number = rnh_meta.get("index_number", "") + start_time_str = rnh_meta.get("start_time_str", "") + + # The NL-43 .rnh has NO measurement timestamps — the real recording window + # lives in the Leq .rnd's "Start Time" column. Whenever the header didn't + # give us a start (and/or stop), derive it from the Leq rows so the session + # gets the true window + duration (and a stable start_time_str for dedupe). + if not start_time_str or stopped_at_local is None: + leq_entry = next( + ((f, b) for f, b in file_entries + if f.lower().endswith(".rnd") and ("_leq_" in f.lower() or f.lower().startswith("au2_"))), + None, + ) + if leq_entry is not None: + first_dt, last_dt, _n, inferred = _leq_window_local(leq_entry[1]) + interval_s = _rnd_interval_seconds(rnh_meta.get("leq_interval")) or inferred or 0 + if first_dt and not start_time_str: + started_at_local = first_dt + start_time_str = first_dt.strftime("%Y/%m/%d %H:%M:%S") + if last_dt and stopped_at_local is None: + stopped_at_local = last_dt + timedelta(seconds=interval_s) + # Recompute UTC + duration from the resolved window. + started_at = local_to_utc(started_at_local) + stopped_at = local_to_utc(stopped_at_local) if stopped_at_local else None + duration_seconds = ( + int((stopped_at - started_at).total_seconds()) + if (started_at and stopped_at) else duration_seconds + ) + + # --- Dedupe: skip if this exact measurement is already ingested --- + if dedupe: + existing = _find_existing_session(db, location.id, store_name, started_at, start_time_str) + if existing: + return { + "success": True, + "deduped": True, + "session_id": existing.id, + "files_imported": 0, + "leq_files": 0, + "lp_files": 0, + "metadata_files": 0, + "store_name": store_name, + "started_at": started_at.isoformat() if started_at else None, + "stopped_at": stopped_at.isoformat() if stopped_at else None, + "duration_seconds": duration_seconds, + } + + # --- Create MonitoringSession (local times drive period/label) --- + period_type = _derive_period_type(started_at_local) if started_at_local else None + session_label = ( + _build_session_label(started_at_local, location.name, period_type) + if started_at_local else None + ) + + session_id = str(uuid.uuid4()) + monitoring_session = MonitoringSession( + id=session_id, + project_id=location.project_id, + location_id=location.id, + unit_id=unit_id, + session_type="sound", + started_at=started_at, + stopped_at=stopped_at, + duration_seconds=duration_seconds, + status="completed", + session_label=session_label, + period_type=period_type, + session_metadata=json.dumps({ + "source": source, + "store_name": store_name, + "serial_number": serial_number, + "index_number": index_number, + "start_time_str": start_time_str, + # Captured from the .rnh so the report can label metrics from the + # device's own config (which LN slot is L90, the weightings, etc.). + "percentiles": rnh_meta.get("percentiles", {}), + "frequency_weighting": rnh_meta.get("frequency_weighting", ""), + "time_weighting": rnh_meta.get("time_weighting", ""), + "leq_interval": rnh_meta.get("leq_interval", ""), + }), + ) + db.add(monitoring_session) + db.commit() + db.refresh(monitoring_session) + + # --- Write files to disk + create DataFile records --- + output_dir = Path("data/Projects") / location.project_id / session_id + output_dir.mkdir(parents=True, exist_ok=True) + + leq_count = lp_count = metadata_count = files_imported = 0 + for fname, fbytes in file_entries: + fname_lower = fname.lower() + if fname_lower.endswith(".rnd"): + if "_leq_" in fname_lower: + leq_count += 1 + elif "_lp" in fname_lower: + lp_count += 1 + elif fname_lower.endswith(".rnh"): + metadata_count += 1 + + dest = output_dir / fname + dest.write_bytes(fbytes) + checksum = hashlib.sha256(fbytes).hexdigest() + rel_path = str(dest.relative_to("data")) + + db.add(DataFile( + id=str(uuid.uuid4()), + session_id=session_id, + file_path=rel_path, + file_type=_classify_file(fname), + file_size_bytes=len(fbytes), + downloaded_at=datetime.utcnow(), + checksum=checksum, + file_metadata=json.dumps({ + "source": source, + "original_filename": fname, + "store_name": store_name, + }), + )) + files_imported += 1 + + db.commit() + + return { + "success": True, + "deduped": False, + "session_id": session_id, + "files_imported": files_imported, + "leq_files": leq_count, + "lp_files": lp_count, + "metadata_files": metadata_count, + "store_name": store_name, + "started_at": started_at.isoformat() if started_at else None, + "stopped_at": stopped_at.isoformat() if stopped_at else None, + "duration_seconds": duration_seconds, + } + + +def ingest_nrl_zip( + location_id: str, + zip_bytes: bytes, + db: Session, + *, + source: str = "ftp_pull", + dedupe: bool = True, + unit_id: Optional[str] = None, +) -> dict: + """Programmatically ingest an Auto_#### ZIP (e.g. a scheduled FTP pull). + + Extracts the ZIP (flattening any nested Auto_Leq/Auto_Lp_ folders), keeps + the .rnh + Leq .rnd, parses the header, and creates a MonitoringSession + + DataFile rows for `location_id`. Defaults to dedupe=True so repeated daily + pulls of the same closed folder don't create duplicate sessions. Pass + `unit_id` to attribute the session to the recording unit at creation. + + Returns the same dict shape as the HTTP upload, plus a `deduped` flag. + Raises IngestError on a bad ZIP, no usable files, or unknown location. + """ + location = db.query(MonitoringLocation).filter_by(id=location_id).first() + if not location: + raise IngestError(f"Location {location_id} not found") + + try: + with zipfile.ZipFile(io.BytesIO(zip_bytes)) as zf: + file_entries: list[tuple[str, bytes]] = [] + for info in zf.infolist(): + if info.is_dir(): + continue + name = Path(info.filename).name # strip nested folder paths + if not name: + continue + file_entries.append((name, zf.read(info))) + except zipfile.BadZipFile: + raise IngestError("Downloaded data is not a valid ZIP archive.") + + return _ingest_file_entries(location, file_entries, db, source=source, dedupe=dedupe, unit_id=unit_id) + + @router.post("/nrl/{location_id}/upload-data") async def upload_nrl_data( project_id: str, @@ -1754,11 +2108,13 @@ async def upload_nrl_data( - A single .zip file (the Auto_#### folder zipped) — auto-extracted - Multiple .rnd / .rnh files selected directly from the SD card folder - Creates a MonitoringSession from .rnh metadata and DataFile records - for each measurement file. No unit assignment required. + Normalizes the upload to (filename, bytes) entries, then hands off to the + shared ingest core (`_ingest_file_entries`) — the same path the scheduled + FTP pull uses via `ingest_nrl_zip`. Creates a MonitoringSession from the + .rnh metadata and DataFile records for each measurement file. No unit + assignment required. dedupe=False here preserves the prior manual-upload + behaviour (re-uploading creates a fresh session). """ - from datetime import datetime - # Verify project and location exist project = db.query(Project).filter_by(id=project_id).first() _require_module(project, "sound_monitoring", db) @@ -1769,7 +2125,7 @@ async def upload_nrl_data( if not location: raise HTTPException(status_code=404, detail="Location not found") - # --- Step 1: Normalize to (filename, bytes) list --- + # --- Normalize upload to (filename, bytes) entries --- file_entries: list[tuple[str, bytes]] = [] if len(files) == 1 and files[0].filename.lower().endswith(".zip"): @@ -1793,145 +2149,11 @@ async def upload_nrl_data( if not file_entries: raise HTTPException(status_code=400, detail="No usable files found in upload.") - # --- Step 1b: Filter to only relevant files --- - # Keep: .rnh (metadata) and measurement .rnd files - # NL-43 generates two .rnd types: _Leq_ (15-min averages, wanted) and _Lp_ (1-sec granular, skip) - # AU2 (NL-23/older Rion) generates a single Au2_####.rnd per session — always keep those - # Drop: _Lp_ .rnd, .xlsx, .mp3, and anything else - def _is_wanted(fname: str) -> bool: - n = fname.lower() - if n.endswith(".rnh"): - return True - if n.endswith(".rnd"): - if "_leq_" in n: # NL-43 Leq file - return True - if n.startswith("au2_"): # AU2 format (NL-23) — always Leq equivalent - return True - if "_lp" not in n and "_leq_" not in n: - # Unknown .rnd format — include it so we don't silently drop data - return True - return False - - file_entries = [(fname, fbytes) for fname, fbytes in file_entries if _is_wanted(fname)] - - if not file_entries: - raise HTTPException(status_code=400, detail="No usable .rnd or .rnh files found. Expected NL-43 _Leq_ files or AU2 format .rnd files.") - - # --- Step 2: Find and parse .rnh metadata --- - rnh_meta = {} - for fname, fbytes in file_entries: - if fname.lower().endswith(".rnh"): - rnh_meta = _parse_rnh(fbytes) - break - - # RNH files store local time (no UTC offset). Use local values for period - # classification / label generation, then convert to UTC for DB storage so - # the local_datetime Jinja filter displays the correct time. - started_at_local = _parse_rnh_datetime(rnh_meta.get("start_time_str")) or datetime.utcnow() - stopped_at_local = _parse_rnh_datetime(rnh_meta.get("stop_time_str")) - - started_at = local_to_utc(started_at_local) - stopped_at = local_to_utc(stopped_at_local) if stopped_at_local else None - - duration_seconds = None - if started_at and stopped_at: - duration_seconds = int((stopped_at - started_at).total_seconds()) - - store_name = rnh_meta.get("store_name", "") - serial_number = rnh_meta.get("serial_number", "") - index_number = rnh_meta.get("index_number", "") - - # --- Step 3: Create MonitoringSession --- - # Use local times for period/label so classification reflects the clock at the site. - period_type = _derive_period_type(started_at_local) if started_at_local else None - session_label = _build_session_label(started_at_local, location.name, period_type) if started_at_local else None - - session_id = str(uuid.uuid4()) - monitoring_session = MonitoringSession( - id=session_id, - project_id=project_id, - location_id=location_id, - unit_id=None, - session_type="sound", - started_at=started_at, - stopped_at=stopped_at, - duration_seconds=duration_seconds, - status="completed", - session_label=session_label, - period_type=period_type, - session_metadata=json.dumps({ - "source": "manual_upload", - "store_name": store_name, - "serial_number": serial_number, - "index_number": index_number, - }), - ) - db.add(monitoring_session) - db.commit() - db.refresh(monitoring_session) - - # --- Step 4: Write files to disk and create DataFile records --- - output_dir = Path("data/Projects") / project_id / session_id - output_dir.mkdir(parents=True, exist_ok=True) - - leq_count = 0 - lp_count = 0 - metadata_count = 0 - files_imported = 0 - - for fname, fbytes in file_entries: - file_type = _classify_file(fname) - fname_lower = fname.lower() - - # Track counts for summary - if fname_lower.endswith(".rnd"): - if "_leq_" in fname_lower: - leq_count += 1 - elif "_lp" in fname_lower: - lp_count += 1 - elif fname_lower.endswith(".rnh"): - metadata_count += 1 - - # Write to disk - dest = output_dir / fname - dest.write_bytes(fbytes) - - # Compute checksum - checksum = hashlib.sha256(fbytes).hexdigest() - - # Store relative path from data/ dir - rel_path = str(dest.relative_to("data")) - - data_file = DataFile( - id=str(uuid.uuid4()), - session_id=session_id, - file_path=rel_path, - file_type=file_type, - file_size_bytes=len(fbytes), - downloaded_at=datetime.utcnow(), - checksum=checksum, - file_metadata=json.dumps({ - "source": "manual_upload", - "original_filename": fname, - "store_name": store_name, - }), - ) - db.add(data_file) - files_imported += 1 - - db.commit() - - return { - "success": True, - "session_id": session_id, - "files_imported": files_imported, - "leq_files": leq_count, - "lp_files": lp_count, - "metadata_files": metadata_count, - "store_name": store_name, - "started_at": started_at.isoformat() if started_at else None, - "stopped_at": stopped_at.isoformat() if stopped_at else None, - } + # --- Hand off to the shared ingest core --- + try: + return _ingest_file_entries(location, file_entries, db, source="manual_upload", dedupe=False) + except IngestError as e: + raise HTTPException(status_code=400, detail=str(e)) # ============================================================================ diff --git a/backend/routers/projects.py b/backend/routers/projects.py index 3c61236..b1407de 100644 --- a/backend/routers/projects.py +++ b/backend/routers/projects.py @@ -1591,24 +1591,32 @@ async def get_sessions_calendar( async def get_ftp_browser( project_id: str, request: Request, + location_id: Optional[str] = None, db: Session = Depends(get_db), ): """ Get FTP browser interface for downloading files from assigned SLMs. Returns HTML partial with FTP browser. Sound Monitoring projects only. + + When `location_id` is given, scope to just the unit(s) assigned to that NRL + (used by the per-NRL Data Files tab, which mirrors the project-wide tab). """ from backend.models import DataFile project = db.query(Project).filter_by(id=project_id).first() _require_module(project, "sound_monitoring", db) - # Get all assignments for this project (active = assigned_until IS NULL) - assignments = db.query(UnitAssignment).filter( + # Active assignments for this project (active = assigned_until IS NULL), + # optionally scoped to a single NRL/location. + q = db.query(UnitAssignment).filter( and_( UnitAssignment.project_id == project_id, UnitAssignment.assigned_until == None, ) - ).all() + ) + if location_id: + q = q.filter(UnitAssignment.location_id == location_id) + assignments = q.all() # Enrich with unit and location details units_data = [] @@ -1638,9 +1646,13 @@ async def ftp_download_to_server( db: Session = Depends(get_db), ): """ - Download a file from an SLM to the server via FTP. - Creates a DataFile record and stores the file in data/Projects/{project_id}/ - Sound Monitoring projects only. + Download a single file from an SLM to the server via FTP. + + NRL measurement files (.rnh / _Leq_ .rnd) are routed through the shared NRL + ingest so the session is parsed and attributed to the unit (a lone .rnh still + yields the real recording window + duration). Any other file type — or a + unit with no location — falls back to a generic stored DataFile, preserving + the original behaviour. Sound Monitoring projects only. """ import httpx import os @@ -1658,7 +1670,55 @@ async def ftp_download_to_server( if not unit_id or not remote_path: raise HTTPException(status_code=400, detail="Missing unit_id or remote_path") - # Get or create active session for this location/unit + filename = os.path.basename(remote_path) + + # Download the file from SLMM + SLMM_BASE_URL = os.getenv("SLMM_BASE_URL", "http://localhost:8100") + try: + async with httpx.AsyncClient(timeout=300.0) as client: + response = await client.post( + f"{SLMM_BASE_URL}/api/nl43/{unit_id}/ftp/download", + json={"remote_path": remote_path} + ) + except httpx.TimeoutException: + raise HTTPException(status_code=504, detail="Timeout downloading file from SLM") + except Exception as e: + logger.error(f"Error reaching SLMM for file download: {e}") + raise HTTPException(status_code=502, detail=f"Failed to reach SLMM: {str(e)}") + + if not response.is_success: + raise HTTPException( + status_code=response.status_code, + detail=f"Failed to download from SLMM: {response.text}", + ) + file_content = response.content + + # NRL measurement file + known location → shared ingest (parsed + attributed). + from backend.routers.project_locations import ( + _ingest_file_entries, IngestError, _is_wanted_nrl_file, + ) + if location_id and _is_wanted_nrl_file(filename): + location = db.query(MonitoringLocation).filter_by(id=location_id).first() + if location: + try: + result = _ingest_file_entries( + location, [(filename, file_content)], db, + source="ftp_manual", dedupe=False, unit_id=unit_id, + ) + except IngestError as e: + raise HTTPException(status_code=400, detail=str(e)) + return { + "success": True, + "message": f"Imported {filename} as NRL measurement data", + "ingested": True, + "session_id": result["session_id"], + "file_size": len(file_content), + "started_at": result["started_at"], + "stopped_at": result["stopped_at"], + "duration_seconds": result["duration_seconds"], + } + + # --- Generic path: any other file type (or no location) — store as-is --- session = db.query(MonitoringSession).filter( and_( MonitoringSession.project_id == project_id, @@ -1668,7 +1728,6 @@ async def ftp_download_to_server( ) ).first() - # If no active session, create one if not session: _ftp_unit = db.query(RosterUnit).filter_by(id=unit_id).first() session = MonitoringSession( @@ -1687,115 +1746,50 @@ async def ftp_download_to_server( db.commit() db.refresh(session) - # Download file from SLMM - SLMM_BASE_URL = os.getenv("SLMM_BASE_URL", "http://localhost:8100") + ext = os.path.splitext(filename)[1].lower() + file_type_map = { + '.wav': 'audio', '.mp3': 'audio', '.flac': 'audio', '.m4a': 'audio', '.aac': 'audio', + '.rnd': 'measurement', + '.csv': 'data', '.txt': 'data', '.json': 'data', '.xml': 'data', '.dat': 'data', + '.log': 'log', + '.zip': 'archive', '.tar': 'archive', '.gz': 'archive', '.7z': 'archive', '.rar': 'archive', + '.jpg': 'image', '.jpeg': 'image', '.png': 'image', '.gif': 'image', + '.pdf': 'document', '.doc': 'document', '.docx': 'document', + } + file_type = file_type_map.get(ext, 'data') - try: - async with httpx.AsyncClient(timeout=300.0) as client: - response = await client.post( - f"{SLMM_BASE_URL}/api/nl43/{unit_id}/ftp/download", - json={"remote_path": remote_path} - ) + project_dir = Path(f"data/Projects/{project_id}/{session.id}") + project_dir.mkdir(parents=True, exist_ok=True) + file_path = project_dir / filename + with open(file_path, 'wb') as f: + f.write(file_content) + checksum = hashlib.sha256(file_content).hexdigest() - if not response.is_success: - raise HTTPException( - status_code=response.status_code, - detail=f"Failed to download from SLMM: {response.text}" - ) + data_file = DataFile( + id=str(uuid.uuid4()), + session_id=session.id, + file_path=str(file_path.relative_to("data")), # Store relative to data/ + file_type=file_type, + file_size_bytes=len(file_content), + downloaded_at=datetime.utcnow(), + checksum=checksum, + file_metadata=json.dumps({ + "source": "ftp", + "remote_path": remote_path, + "unit_id": unit_id, + "location_id": location_id, + }) + ) + db.add(data_file) + db.commit() - # Extract filename from remote_path - filename = os.path.basename(remote_path) - - # Determine file type from extension - ext = os.path.splitext(filename)[1].lower() - file_type_map = { - # Audio files - '.wav': 'audio', - '.mp3': 'audio', - '.flac': 'audio', - '.m4a': 'audio', - '.aac': 'audio', - # Sound level meter measurement files - '.rnd': 'measurement', - # Data files - '.csv': 'data', - '.txt': 'data', - '.json': 'data', - '.xml': 'data', - '.dat': 'data', - # Log files - '.log': 'log', - # Archives - '.zip': 'archive', - '.tar': 'archive', - '.gz': 'archive', - '.7z': 'archive', - '.rar': 'archive', - # Images - '.jpg': 'image', - '.jpeg': 'image', - '.png': 'image', - '.gif': 'image', - # Documents - '.pdf': 'document', - '.doc': 'document', - '.docx': 'document', - } - file_type = file_type_map.get(ext, 'data') - - # Create directory structure: data/Projects/{project_id}/{session_id}/ - project_dir = Path(f"data/Projects/{project_id}/{session.id}") - project_dir.mkdir(parents=True, exist_ok=True) - - # Save file to disk - file_path = project_dir / filename - file_content = response.content - - with open(file_path, 'wb') as f: - f.write(file_content) - - # Calculate checksum - checksum = hashlib.sha256(file_content).hexdigest() - - # Create DataFile record - data_file = DataFile( - id=str(uuid.uuid4()), - session_id=session.id, - file_path=str(file_path.relative_to("data")), # Store relative to data/ - file_type=file_type, - file_size_bytes=len(file_content), - downloaded_at=datetime.utcnow(), - checksum=checksum, - file_metadata=json.dumps({ - "source": "ftp", - "remote_path": remote_path, - "unit_id": unit_id, - "location_id": location_id, - }) - ) - - db.add(data_file) - db.commit() - - return { - "success": True, - "message": f"Downloaded {filename} to server", - "file_id": data_file.id, - "file_path": str(file_path), - "file_size": len(file_content), - } - - except httpx.TimeoutException: - raise HTTPException( - status_code=504, - detail="Timeout downloading file from SLM" - ) - except Exception as e: - logger.error(f"Error downloading file to server: {e}") - raise HTTPException( - status_code=500, - detail=f"Failed to download file to server: {str(e)}" - ) + return { + "success": True, + "message": f"Downloaded {filename} to server", + "file_id": data_file.id, + "file_path": str(file_path), + "file_size": len(file_content), + } @router.post("/{project_id}/ftp-download-folder-to-server") @@ -1805,20 +1799,20 @@ async def ftp_download_folder_to_server( db: Session = Depends(get_db), ): """ - Download an entire folder from an SLM to the server via FTP. - Extracts all files from the ZIP and preserves folder structure. - Creates individual DataFile records for each file. - Sound Monitoring projects only. + Download an entire Auto_#### measurement folder from an SLM to the server. + + Routes the downloaded ZIP through the shared NRL ingest — the same path the + scheduled FTP pull, the daily cycle, and the manual SD-card upload use. That + means: keep the .rnh + Leq .rnd, parse the header (real recording start/stop + + duration, percentile slot map, weightings), drop the 1-second _Lp_ files, + and create one clean MonitoringSession attributed to the unit. Sound + Monitoring projects only. """ import httpx import os - import hashlib - import zipfile - import io _require_module(db.query(Project).filter_by(id=project_id).first(), "sound_monitoring", db) - from pathlib import Path - from backend.models import DataFile + from backend.routers.project_locations import ingest_nrl_zip, IngestError data = await request.json() unit_id = data.get("unit_id") @@ -1827,160 +1821,66 @@ async def ftp_download_folder_to_server( if not unit_id or not remote_path: raise HTTPException(status_code=400, detail="Missing unit_id or remote_path") - - # Get or create active session for this location/unit - session = db.query(MonitoringSession).filter( - and_( - MonitoringSession.project_id == project_id, - MonitoringSession.location_id == location_id, - MonitoringSession.unit_id == unit_id, - MonitoringSession.status.in_(["recording", "paused"]) + if not location_id: + raise HTTPException( + status_code=400, + detail=("This unit isn't assigned to a monitoring location. Assign it to an " + "NRL first so the downloaded measurement attaches to the right location."), ) - ).first() - # If no active session, create one - if not session: - _ftp_unit = db.query(RosterUnit).filter_by(id=unit_id).first() - session = MonitoringSession( - id=str(uuid.uuid4()), - project_id=project_id, - location_id=location_id, - unit_id=unit_id, - session_type="sound", # SLMs are sound monitoring devices - status="completed", - started_at=datetime.utcnow(), - stopped_at=datetime.utcnow(), - device_model=_ftp_unit.slm_model if _ftp_unit else None, - session_metadata='{"source": "ftp_folder_download", "note": "Auto-created for FTP folder download"}' - ) - db.add(session) - db.commit() - db.refresh(session) - - # Download folder from SLMM (returns ZIP) + # Download the folder from SLMM (returns a ZIP of the Auto_#### folder) SLMM_BASE_URL = os.getenv("SLMM_BASE_URL", "http://localhost:8100") - try: - async with httpx.AsyncClient(timeout=600.0) as client: # Longer timeout for folders + async with httpx.AsyncClient(timeout=600.0) as client: # longer timeout for folders response = await client.post( f"{SLMM_BASE_URL}/api/nl43/{unit_id}/ftp/download-folder", json={"remote_path": remote_path} ) - - if not response.is_success: - raise HTTPException( - status_code=response.status_code, - detail=f"Failed to download folder from SLMM: {response.text}" - ) - - # Extract folder name from remote_path - folder_name = os.path.basename(remote_path.rstrip('/')) - - # Create base directory: data/Projects/{project_id}/{session_id}/{folder_name}/ - base_dir = Path(f"data/Projects/{project_id}/{session.id}/{folder_name}") - base_dir.mkdir(parents=True, exist_ok=True) - - # Extract ZIP and save individual files - zip_content = response.content - created_files = [] - total_size = 0 - - # File type mapping for classification - file_type_map = { - # Audio files - '.wav': 'audio', '.mp3': 'audio', '.flac': 'audio', '.m4a': 'audio', '.aac': 'audio', - # Data files - '.csv': 'data', '.txt': 'data', '.json': 'data', '.xml': 'data', '.dat': 'data', - # Log files - '.log': 'log', - # Archives - '.zip': 'archive', '.tar': 'archive', '.gz': 'archive', '.7z': 'archive', '.rar': 'archive', - # Images - '.jpg': 'image', '.jpeg': 'image', '.png': 'image', '.gif': 'image', - # Documents - '.pdf': 'document', '.doc': 'document', '.docx': 'document', - } - - with zipfile.ZipFile(io.BytesIO(zip_content)) as zf: - for zip_info in zf.filelist: - # Skip directories - if zip_info.is_dir(): - continue - - # Read file from ZIP - file_data = zf.read(zip_info.filename) - - # Determine file path (preserve structure within folder) - # zip_info.filename might be like "Auto_0001/measurement.wav" - file_path = base_dir / zip_info.filename - file_path.parent.mkdir(parents=True, exist_ok=True) - - # Write file to disk - with open(file_path, 'wb') as f: - f.write(file_data) - - # Calculate checksum - checksum = hashlib.sha256(file_data).hexdigest() - - # Determine file type - ext = os.path.splitext(zip_info.filename)[1].lower() - file_type = file_type_map.get(ext, 'data') - - # Create DataFile record - data_file = DataFile( - id=str(uuid.uuid4()), - session_id=session.id, - file_path=str(file_path.relative_to("data")), - file_type=file_type, - file_size_bytes=len(file_data), - downloaded_at=datetime.utcnow(), - checksum=checksum, - file_metadata=json.dumps({ - "source": "ftp_folder", - "remote_path": remote_path, - "unit_id": unit_id, - "location_id": location_id, - "folder_name": folder_name, - "relative_path": zip_info.filename, - }) - ) - - db.add(data_file) - created_files.append({ - "filename": zip_info.filename, - "size": len(file_data), - "type": file_type - }) - total_size += len(file_data) - - db.commit() - - return { - "success": True, - "message": f"Downloaded folder {folder_name} with {len(created_files)} files", - "folder_name": folder_name, - "file_count": len(created_files), - "total_size": total_size, - "files": created_files, - } - except httpx.TimeoutException: raise HTTPException( status_code=504, - detail="Timeout downloading folder from SLM (large folders may take a while)" - ) - except zipfile.BadZipFile: - raise HTTPException( - status_code=500, - detail="Downloaded file is not a valid ZIP archive" + detail="Timeout downloading folder from SLM (large folders may take a while)", ) except Exception as e: - logger.error(f"Error downloading folder to server: {e}") + logger.error(f"Error reaching SLMM for folder download: {e}") + raise HTTPException(status_code=502, detail=f"Failed to reach SLMM: {str(e)}") + + if not response.is_success: raise HTTPException( - status_code=500, - detail=f"Failed to download folder to server: {str(e)}" + status_code=response.status_code, + detail=f"Failed to download folder from SLMM: {response.text}", ) + # Ingest through the shared NRL core. dedupe=False so a re-download of a + # still-growing folder captures the latest intervals (matches manual upload). + try: + result = ingest_nrl_zip( + location_id, response.content, db, + source="ftp_manual", dedupe=False, unit_id=unit_id, + ) + except IngestError as e: + # No usable .rnd/.rnh in the folder, or unknown location. + raise HTTPException(status_code=400, detail=str(e)) + except Exception as e: + logger.error(f"Error ingesting downloaded folder: {e}") + raise HTTPException(status_code=500, detail=f"Failed to ingest downloaded folder: {str(e)}") + + folder_name = os.path.basename(remote_path.rstrip('/')) + return { + "success": True, + "message": ( + f"Imported {result['leq_files']} Leq file(s) from {folder_name} " + f"({result['files_imported']} stored; 1-second _Lp_ data skipped)" + ), + "folder_name": folder_name, + "session_id": result["session_id"], + "file_count": result["files_imported"], + "leq_files": result["leq_files"], + "started_at": result["started_at"], + "stopped_at": result["stopped_at"], + "duration_seconds": result["duration_seconds"], + } + # ============================================================================ # Project Types @@ -1990,21 +1890,26 @@ async def ftp_download_folder_to_server( async def get_unified_files( project_id: str, request: Request, + location_id: Optional[str] = None, db: Session = Depends(get_db), ): """ Get unified view of all files in this project. Groups files by recording session with full metadata. Returns HTML partial with hierarchical file listing. + + When `location_id` is given, scope to a single NRL/location (used by the + per-NRL Data Files tab so it mirrors the project-wide tab). """ from backend.models import DataFile from pathlib import Path import json - # Get all sessions for this project - sessions = db.query(MonitoringSession).filter_by( - project_id=project_id - ).order_by(MonitoringSession.started_at.desc()).all() + # Sessions for this project (optionally scoped to one NRL/location) + q = db.query(MonitoringSession).filter_by(project_id=project_id) + if location_id: + q = q.filter(MonitoringSession.location_id == location_id) + sessions = q.order_by(MonitoringSession.started_at.desc()).all() sessions_data = [] for session in sessions: diff --git a/backend/routers/reports.py b/backend/routers/reports.py new file mode 100644 index 0000000..1f2ca49 --- /dev/null +++ b/backend/routers/reports.py @@ -0,0 +1,434 @@ +""" +Nightly Report Router. + +Manual triggers for the night-vs-baseline sound report — the same entry point +the scheduled morning tick will reuse. Two endpoints: + + GET …/reports/nightly/view → render and return the HTML inline (preview). + No write, no email. Browser-friendly. + POST …/reports/nightly/run → full run: build → write report.html/json to + disk → (dry-run) email. Returns JSON result. + +Dates are the *evening* date of the night being reported (the 7/7 in "night of +7/7 → morning 7/8"). Defaults to last night. Baseline is optional; pass the +baseline-week range to populate the comparison. +""" + +from __future__ import annotations + +import json +import logging +import re +import uuid +from datetime import datetime, timedelta, date +from html import escape +from pathlib import Path +from typing import Optional + +from fastapi import APIRouter, Depends, HTTPException, Query, Request +from fastapi.responses import HTMLResponse +from sqlalchemy.orm import Session + +from backend.database import get_db +from backend.models import Project, SoundReportConfig, MonitoringLocation +from backend.services.report_pipeline import ( + METRIC_REGISTRY, DEFAULT_METRICS, DEFAULT_WINDOWS, _location_reference_baseline, +) +from backend.services.report_orchestrator import run_nightly_report +from backend.utils.timezone import utc_to_local + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/projects/{project_id}/reports", tags=["reports"]) + + +def _default_night_date() -> date: + """Last night = yesterday in the user's local timezone.""" + return (utc_to_local(datetime.utcnow()) - timedelta(days=1)).date() + + +def _parse_date(s: Optional[str], field: str) -> Optional[date]: + if not s: + return None + try: + return datetime.strptime(s, "%Y-%m-%d").date() + except ValueError: + raise HTTPException(status_code=400, detail=f"{field} must be YYYY-MM-DD (got {s!r})") + + +def _parse_metrics(s: Optional[str]) -> list[str]: + if not s: + return list(DEFAULT_METRICS) + keys = [k.strip().lower() for k in s.split(",") if k.strip()] + unknown = [k for k in keys if k not in METRIC_REGISTRY] + if unknown: + raise HTTPException( + status_code=400, + detail=f"Unknown metric(s): {unknown}. Known: {sorted(METRIC_REGISTRY)}", + ) + return keys or list(DEFAULT_METRICS) + + +def _validate_hhmm(s) -> str: + """Validate a local HH:MM (24h) time string.""" + try: + hh, mm = str(s).split(":") + h, m = int(hh), int(mm) + if 0 <= h < 24 and 0 <= m < 60: + return f"{h:02d}:{m:02d}" + except (ValueError, AttributeError): + pass + raise HTTPException(status_code=400, detail=f"report_time must be HH:MM 24-hour (got {s!r})") + + +def _config_dict(cfg: Optional[SoundReportConfig], project_id: str) -> dict: + """Serialise a config row (or defaults if none yet) to JSON.""" + return { + "project_id": project_id, + "exists": cfg is not None, + "enabled": cfg.enabled if cfg else False, + "report_time": cfg.report_time if cfg else "08:00", + "metric_keys": cfg.metric_keys if cfg else ",".join(DEFAULT_METRICS), + "baseline_mode": cfg.baseline_mode if cfg else "captured", + "baseline_start": cfg.baseline_start.isoformat() if cfg and cfg.baseline_start else None, + "baseline_end": cfg.baseline_end.isoformat() if cfg and cfg.baseline_end else None, + "recipients": (cfg.recipients if cfg and cfg.recipients else ""), + "last_run_date": cfg.last_run_date.isoformat() if cfg and cfg.last_run_date else None, + } + + +@router.get("/config") +async def get_report_config(project_id: str, db: Session = Depends(get_db)): + """Return the project's nightly-report config (or defaults if not set yet).""" + if not db.query(Project).filter_by(id=project_id).first(): + raise HTTPException(status_code=404, detail="Project not found") + cfg = db.query(SoundReportConfig).filter_by(project_id=project_id).first() + return _config_dict(cfg, project_id) + + +@router.put("/config") +async def put_report_config(project_id: str, request: Request, db: Session = Depends(get_db)): + """Create or update the project's nightly-report config (JSON body).""" + if not db.query(Project).filter_by(id=project_id).first(): + raise HTTPException(status_code=404, detail="Project not found") + data = await request.json() + + cfg = db.query(SoundReportConfig).filter_by(project_id=project_id).first() + created = cfg is None + if cfg is None: + cfg = SoundReportConfig(id=str(uuid.uuid4()), project_id=project_id) + db.add(cfg) + + if "enabled" in data: + cfg.enabled = bool(data["enabled"]) + if "report_time" in data: + cfg.report_time = _validate_hhmm(data["report_time"]) + if "metric_keys" in data: + mk = data["metric_keys"] + mk = mk if isinstance(mk, str) else ",".join(mk or []) + cfg.metric_keys = ",".join(_parse_metrics(mk)) + if "baseline_mode" in data: + bm = str(data["baseline_mode"]).lower() + if bm not in ("captured", "reference"): + raise HTTPException(status_code=400, detail="baseline_mode must be 'captured' or 'reference'") + cfg.baseline_mode = bm + if "baseline_start" in data or "baseline_end" in data: + bs = _parse_date(data.get("baseline_start") or None, "baseline_start") + be = _parse_date(data.get("baseline_end") or None, "baseline_end") + if (bs and not be) or (be and not bs): + raise HTTPException(status_code=400, detail="Provide both baseline dates, or neither.") + if bs and be and bs > be: + raise HTTPException(status_code=400, detail="baseline_start must be on or before baseline_end.") + cfg.baseline_start, cfg.baseline_end = bs, be + if "recipients" in data: + recips = data["recipients"] + if isinstance(recips, list): + recips = ",".join(recips) + cfg.recipients = (recips or "").strip() or None + + db.commit() + db.refresh(cfg) + return {**_config_dict(cfg, project_id), "created": created} + + +def _resolve_params(project_id, db, night_date, baseline_start, baseline_end, metrics): + """Validate inputs and resolve the baseline source. + + Explicit baseline dates in the query override (captured mode with those + dates). Otherwise the project's saved config supplies the baseline (its + mode + dates) and the default metric set — so the manual view/run match + what the scheduled report does. + Returns (night_date, baseline_mode, baseline_start, baseline_end, metric_keys). + """ + if not db.query(Project).filter_by(id=project_id).first(): + raise HTTPException(status_code=404, detail="Project not found") + nd = _parse_date(night_date, "night_date") or _default_night_date() + bs = _parse_date(baseline_start, "baseline_start") + be = _parse_date(baseline_end, "baseline_end") + if (bs and not be) or (be and not bs): + raise HTTPException(status_code=400, detail="Provide both baseline_start and baseline_end, or neither.") + if bs and be and bs > be: + raise HTTPException(status_code=400, detail="baseline_start must be on or before baseline_end.") + + cfg = db.query(SoundReportConfig).filter_by(project_id=project_id).first() + if bs and be: + baseline_mode = "captured" # explicit dates win + elif cfg: + baseline_mode = cfg.baseline_mode # fall back to saved config + bs, be = cfg.baseline_start, cfg.baseline_end + else: + baseline_mode = "captured" + + if metrics: + metric_keys = _parse_metrics(metrics) + elif cfg and cfg.metric_keys: + metric_keys = _parse_metrics(cfg.metric_keys) + else: + metric_keys = list(DEFAULT_METRICS) + + return nd, baseline_mode, bs, be, metric_keys + + +@router.get("/nightly/view", response_class=HTMLResponse) +async def view_nightly_report( + project_id: str, + night_date: Optional[str] = Query(None, description="Evening date of the night (YYYY-MM-DD). Default: last night."), + baseline_start: Optional[str] = Query(None, description="Baseline range start (YYYY-MM-DD)."), + baseline_end: Optional[str] = Query(None, description="Baseline range end (YYYY-MM-DD)."), + metrics: Optional[str] = Query(None, description="Comma list, e.g. lmax,l01,l10,l90. Default: house set."), + db: Session = Depends(get_db), +): + """Render the night report and return the HTML inline (preview — no write, no email).""" + nd, bmode, bs, be, metric_keys = _resolve_params(project_id, db, night_date, baseline_start, baseline_end, metrics) + try: + result = run_nightly_report( + db, project_id, nd, + metric_keys=metric_keys, baseline_mode=bmode, baseline_start=bs, baseline_end=be, + send=False, # preview: no email + ) + except HTTPException: + raise + except Exception as e: # noqa: BLE001 + logger.error("nightly/view failed for %s (%s): %s", project_id, nd, e, exc_info=True) + raise HTTPException(status_code=500, detail=f"Report generation failed: {e}") + return HTMLResponse(result["html"]) + + +@router.post("/nightly/run") +async def run_nightly_report_endpoint( + project_id: str, + night_date: Optional[str] = Query(None, description="Evening date of the night (YYYY-MM-DD). Default: last night."), + baseline_start: Optional[str] = Query(None, description="Baseline range start (YYYY-MM-DD)."), + baseline_end: Optional[str] = Query(None, description="Baseline range end (YYYY-MM-DD)."), + metrics: Optional[str] = Query(None, description="Comma list, e.g. lmax,l01,l10,l90. Default: house set."), + send: bool = Query(True, description="Attempt email (dry-run until SMTP is configured)."), + db: Session = Depends(get_db), +): + """Run the night report: build → write report.html/report.json to disk → email (best-effort). + + This is the same path the scheduled morning tick will call. The `html` field + is omitted from the JSON response (it's large and on disk); use /view to see it. + """ + nd, bmode, bs, be, metric_keys = _resolve_params(project_id, db, night_date, baseline_start, baseline_end, metrics) + try: + result = run_nightly_report( + db, project_id, nd, + metric_keys=metric_keys, baseline_mode=bmode, baseline_start=bs, baseline_end=be, + send=send, + ) + except HTTPException: + raise + except Exception as e: # noqa: BLE001 + logger.error("nightly/run failed for %s (%s): %s", project_id, nd, e, exc_info=True) + raise HTTPException(status_code=500, detail=f"Report generation failed: {e}") + result.pop("html", None) # keep the JSON response lean — view it via /view or the file + result["view_url"] = ( + f"/api/projects/{project_id}/reports/nightly/view" + f"?night_date={nd:%Y-%m-%d}" + + (f"&baseline_start={bs:%Y-%m-%d}&baseline_end={be:%Y-%m-%d}" if bs and be else "") + + (f"&metrics={','.join(metric_keys)}") + ) + return result + + +# ============================================================================ +# Test email + generated-report archive +# ============================================================================ + +_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$") + + +@router.post("/test-email") +async def send_test_email(project_id: str, request: Request, db: Session = Depends(get_db)): + """Send a small test email to verify the SMTP relay (dry-run if unconfigured). + + Recipients: JSON body {"recipients": "..."} overrides; else the project's + configured recipients; else the REPORT_SMTP_RECIPIENTS env default. + """ + project = db.query(Project).filter_by(id=project_id).first() + if not project: + raise HTTPException(status_code=404, detail="Project not found") + try: + data = await request.json() + except Exception: + data = {} + + raw = (data or {}).get("recipients") + if not raw: + cfg = db.query(SoundReportConfig).filter_by(project_id=project_id).first() + raw = cfg.recipients if cfg else None + recipients = None + if raw: + if isinstance(raw, list): + raw = ",".join(raw) + recipients = [r.strip() for r in raw.split(",") if r.strip()] + + from backend.services.report_email import send_report_email + body = ( + "
" + f"Terra-View test email for {escape(project.name)}.
" + "If you got this, the nightly sound-report email path is working.
" + ) + return send_report_email("Terra-View — nightly report test email", body, recipients=recipients) + + +@router.get("/list") +async def list_reports(project_id: str, db: Session = Depends(get_db)): + """List the generated report artifacts on disk for this project (newest first).""" + if not db.query(Project).filter_by(id=project_id).first(): + raise HTTPException(status_code=404, detail="Project not found") + base = Path("data/reports") / project_id + out = [] + if base.exists(): + for d in sorted((p for p in base.iterdir() if p.is_dir()), key=lambda p: p.name, reverse=True): + html_file = d / "report.html" + if html_file.exists(): + st = html_file.stat() + out.append({ + "night_date": d.name, + "view_url": f"/api/projects/{project_id}/reports/archive/{d.name}", + "xlsx_url": (f"/api/projects/{project_id}/reports/archive/{d.name}/xlsx" + if (d / "report.xlsx").exists() else None), + "size_bytes": st.st_size, + "generated_at": datetime.utcfromtimestamp(st.st_mtime).isoformat(), + }) + return {"reports": out, "count": len(out)} + + +@router.get("/archive/{night_date}", response_class=HTMLResponse) +async def view_archived_report(project_id: str, night_date: str, db: Session = Depends(get_db)): + """Serve a previously generated report.html from disk (the actual artifact).""" + if not db.query(Project).filter_by(id=project_id).first(): + raise HTTPException(status_code=404, detail="Project not found") + if not _DATE_RE.match(night_date): + raise HTTPException(status_code=400, detail="Invalid date (YYYY-MM-DD)") + safe = _parse_date(night_date, "night_date") # also guards path traversal + path = Path("data/reports") / project_id / f"{safe:%Y-%m-%d}" / "report.html" + if not path.exists(): + raise HTTPException(status_code=404, detail="No saved report for that date") + return HTMLResponse(path.read_text(encoding="utf-8")) + + +@router.get("/archive/{night_date}/xlsx") +async def download_archived_xlsx(project_id: str, night_date: str, db: Session = Depends(get_db)): + """Download a previously generated report.xlsx from disk.""" + from fastapi.responses import Response + if not db.query(Project).filter_by(id=project_id).first(): + raise HTTPException(status_code=404, detail="Project not found") + if not _DATE_RE.match(night_date): + raise HTTPException(status_code=400, detail="Invalid date (YYYY-MM-DD)") + safe = _parse_date(night_date, "night_date") + path = Path("data/reports") / project_id / f"{safe:%Y-%m-%d}" / "report.xlsx" + if not path.exists(): + raise HTTPException(status_code=404, detail="No saved spreadsheet for that date") + return Response( + content=path.read_bytes(), + media_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + headers={"Content-Disposition": f'attachment; filename="night_report_{safe:%Y-%m-%d}.xlsx"'}, + ) + + +# ============================================================================ +# Reference baseline (fixed values typed per location — limits / prior averages) +# ============================================================================ + +@router.get("/baseline") +async def get_baseline(project_id: str, db: Session = Depends(get_db)): + """Return the baseline mode + per-location reference values + the metric/window + grid to render the editor.""" + if not db.query(Project).filter_by(id=project_id).first(): + raise HTTPException(status_code=404, detail="Project not found") + cfg = db.query(SoundReportConfig).filter_by(project_id=project_id).first() + mode = cfg.baseline_mode if cfg else "captured" + metric_keys = _parse_metrics(cfg.metric_keys) if cfg and cfg.metric_keys else list(DEFAULT_METRICS) + + locations = db.query(MonitoringLocation).filter_by( + project_id=project_id, location_type="sound", + ).order_by(MonitoringLocation.sort_order, MonitoringLocation.name).all() + locations = [l for l in locations if getattr(l, "removed_at", None) is None] + + return { + "mode": mode, + "windows": [{"key": w.key, "label": w.label} for w in DEFAULT_WINDOWS], + "metrics": [{"key": k, "label": METRIC_REGISTRY[k].label} for k in metric_keys], + "locations": [ + {"id": loc.id, "name": loc.name, "values": _location_reference_baseline(loc)} + for loc in locations + ], + } + + +@router.put("/baseline") +async def put_baseline(project_id: str, request: Request, db: Session = Depends(get_db)): + """Save the baseline mode (on config) and per-location reference values + (on each location's metadata). Body: + {"mode": "reference", + "locations": {"": {"nighttime": {"l10": 85}, "evening": {...}}}} + """ + if not db.query(Project).filter_by(id=project_id).first(): + raise HTTPException(status_code=404, detail="Project not found") + data = await request.json() + + if "mode" in data: + bm = str(data["mode"]).lower() + if bm not in ("captured", "reference"): + raise HTTPException(status_code=400, detail="mode must be 'captured' or 'reference'") + cfg = db.query(SoundReportConfig).filter_by(project_id=project_id).first() + if cfg is None: + cfg = SoundReportConfig(id=str(uuid.uuid4()), project_id=project_id) + db.add(cfg) + cfg.baseline_mode = bm + + loc_values = data.get("locations") or {} + updated = 0 + for loc_id, windows in loc_values.items(): + loc = db.query(MonitoringLocation).filter_by(id=loc_id, project_id=project_id).first() + if not loc or not isinstance(windows, dict): + continue + try: + meta = json.loads(loc.location_metadata or "{}") + except (json.JSONDecodeError, TypeError): + meta = {} + clean: dict = {} + for wkey, mvals in windows.items(): + if not isinstance(mvals, dict): + continue + cm = {} + for mkey, val in mvals.items(): + if val in (None, ""): + continue + try: + cm[mkey] = round(float(val), 1) + except (ValueError, TypeError): + continue + if cm: + clean[wkey] = cm + if clean: + meta["report_baseline"] = clean + else: + meta.pop("report_baseline", None) + loc.location_metadata = json.dumps(meta) + updated += 1 + + db.commit() + return {"ok": True, "locations_updated": updated} diff --git a/backend/routers/slm_dashboard.py b/backend/routers/slm_dashboard.py index 3b93488..d35746c 100644 --- a/backend/routers/slm_dashboard.py +++ b/backend/routers/slm_dashboard.py @@ -91,29 +91,43 @@ async def get_slm_units( one_hour_ago = datetime.utcnow() - timedelta(hours=1) for unit in units: + # Legacy default from the roster field; refined from SLMM's cached status below. unit.is_recent = bool(unit.slm_last_check and unit.slm_last_check > one_hour_ago) + unit.measurement_state = None + unit.cache_last_seen = None # SLMM cache last_seen (real monitoring freshness) if include_measurement: - async def fetch_measurement_state(client: httpx.AsyncClient, unit_id: str) -> str | None: - try: - response = await client.get(f"{SLMM_BASE_URL}/api/nl43/{unit_id}/measurement-state") - if response.status_code == 200: - return response.json().get("measurement_state") - except Exception: - return None - return None - - deployed_units = [unit for unit in units if unit.deployed and not unit.retired] - if deployed_units: + # SLMM's /roster carries each unit's CACHED status (last_seen, + # measurement_state) from NL43Status — a DB read on SLMM's side, NOT a device + # call. The live monitor refreshes that cache ~every 1.3s, so this reflects + # real monitoring without sending Measure? to the device (which the old + # /measurement-state did) and competing with DOD polling. One call covers all. + slmm_status = {} + try: async with httpx.AsyncClient(timeout=3.0) as client: - tasks = [fetch_measurement_state(client, unit.id) for unit in deployed_units] - results = await asyncio.gather(*tasks, return_exceptions=True) + r = await client.get(f"{SLMM_BASE_URL}/api/nl43/roster") + if r.status_code == 200: + for dev in (r.json().get("devices") or []): + slmm_status[dev.get("unit_id")] = dev.get("status") or {} + except Exception: + slmm_status = {} - for unit, state in zip(deployed_units, results): - if isinstance(state, Exception): - unit.measurement_state = None - else: - unit.measurement_state = state + # "Recent" = the monitor has a fresh successful read. last_seen only advances + # on a successful poll, so staleness == the device isn't being reached. + recent_cutoff = datetime.utcnow() - timedelta(minutes=5) + for unit in units: + st = slmm_status.get(unit.id) + if not st: + continue + unit.measurement_state = st.get("measurement_state") + last_seen = st.get("last_seen") + if last_seen: + try: + ls = datetime.fromisoformat(last_seen.replace("Z", "")) + unit.is_recent = ls > recent_cutoff + unit.cache_last_seen = ls # the real freshness the monitor updates + except Exception: + pass return templates.TemplateResponse("partials/slm_device_list.html", { "request": request, @@ -157,25 +171,18 @@ async def get_live_view(request: Request, unit_id: str, db: Session = Depends(ge is_measuring = False try: - async with httpx.AsyncClient(timeout=10.0) as client: - # Get measurement state - state_response = await client.get( - f"{SLMM_BASE_URL}/api/nl43/{unit_id}/measurement-state" - ) - if state_response.status_code == 200: - state_data = state_response.json() - measurement_state = state_data.get("measurement_state", "Unknown") - is_measuring = state_data.get("is_measuring", False) - - # Get live status (measurement_start_time is already stored in SLMM database) - status_response = await client.get( - f"{SLMM_BASE_URL}/api/nl43/{unit_id}/live" - ) - if status_response.status_code == 200: - status_data = status_response.json() - current_status = status_data.get("data", {}) + # Read SLMM's CACHED status (NL43Status) — no device call. The live monitor + # keeps it fresh (~1.3s) and the live-stream WS provides ongoing updates, so we + # no longer fire Measure? + a fresh DOD read at the device on every command- + # center load (which competed with DOD polling for the single connection). + async with httpx.AsyncClient(timeout=5.0) as client: + r = await client.get(f"{SLMM_BASE_URL}/api/nl43/{unit_id}/status") + if r.status_code == 200: + current_status = r.json().get("data", {}) + measurement_state = current_status.get("measurement_state") + is_measuring = measurement_state in ("Start", "Measure") except Exception as e: - logger.error(f"Failed to get status for {unit_id}: {e}") + logger.error(f"Failed to get cached status for {unit_id}: {e}") return templates.TemplateResponse("partials/slm_live_view.html", { "request": request, diff --git a/backend/routers/slmm.py b/backend/routers/slmm.py index 1c73f5e..62a0385 100644 --- a/backend/routers/slmm.py +++ b/backend/routers/slmm.py @@ -231,6 +231,76 @@ async def proxy_websocket_live(websocket: WebSocket, unit_id: str): logger.info(f"WebSocket proxy closed for {unit_id} (live)") +@router.websocket("/{unit_id}/monitor") +async def proxy_websocket_monitor(websocket: WebSocket, unit_id: str): + """ + Proxy WebSocket connections to SLMM's /monitor (fan-out DOD feed). + + This is the shared ~1Hz DOD feed: many clients subscribe to one device feed + (no single-connection contention) and it carries L1/L10 (which the DRD + /stream cannot). Preferred over /stream for the live view. + """ + await websocket.accept() + logger.info(f"WebSocket accepted for SLMM unit {unit_id} (monitor)") + + target_ws_url = f"{SLMM_WS_BASE_URL}/api/nl43/{unit_id}/monitor" + backend_ws = None + + try: + backend_ws = await websockets.connect(target_ws_url) + logger.info(f"Connected to SLMM monitor feed for {unit_id}") + + async def forward_to_client(): + """Backend monitor frames -> browser.""" + async for message in backend_ws: + await websocket.send_text(message) + + async def watch_client(): + """Drain client frames; raises WebSocketDisconnect on close so we can + tear the pair down (the monitor feed is server->client only).""" + while True: + await websocket.receive_text() + + # When EITHER side ends (browser disconnects or backend closes), cancel the + # other immediately — avoids sending into a closed socket (the + # "Unexpected ASGI message after close" race that asyncio.gather leaves open). + tasks = [asyncio.ensure_future(forward_to_client()), + asyncio.ensure_future(watch_client())] + done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED) + for t in pending: + t.cancel() + # Await ALL tasks (the done one AND the cancelled one) and swallow both + # the expected WebSocketDisconnect and CancelledError. CancelledError is a + # BaseException, so a bare `except Exception` misses it — that's what leaked + # the traceback on stop; and awaiting only `pending` left the done task's + # exception unretrieved. + for t in tasks: + try: + await t + except (asyncio.CancelledError, Exception): + pass + + except websockets.exceptions.WebSocketException as e: + logger.error(f"WebSocket error connecting to SLMM monitor for {unit_id}: {e}") + try: + await websocket.send_json({"error": "Failed to connect to SLMM monitor", "detail": str(e)}) + except Exception: + pass + except Exception as e: + logger.error(f"Unexpected error in monitor proxy for {unit_id}: {e}") + finally: + if backend_ws: + try: + await backend_ws.close() + except Exception: + pass + try: + await websocket.close() + except Exception: + pass + logger.info(f"WebSocket monitor proxy closed for {unit_id}") + + # HTTP catch-all route MUST come after specific routes (including WebSocket routes) @router.api_route("/{path:path}", methods=["GET", "POST", "PUT", "DELETE", "PATCH"]) async def proxy_to_slmm(path: str, request: Request): diff --git a/backend/services/report_email.py b/backend/services/report_email.py new file mode 100644 index 0000000..9750263 --- /dev/null +++ b/backend/services/report_email.py @@ -0,0 +1,172 @@ +""" +Report email sender — config-driven SMTP via the Python standard library. + +Connection settings come from environment variables so the mail backend +(internal relay / Microsoft 365 / Gmail / SendGrid) can be swapped without code +changes — see the build plan: terra-mechanics.com is on M365 and has a smarthost +relay that already sends the seismograph alerts as remote@terra-mechanics.com; +reuse that relay's settings here. + +DRY-RUN: if SMTP isn't configured (no host/from), the message is built and +logged but NOT sent, and the call still succeeds. This keeps report generation +working before the relay is wired up, and means a missing/incomplete mail config +can never crash the nightly pipeline. + +Env vars +-------- + REPORT_SMTP_HOST e.g. smtp.office365.com (unset → dry-run) + REPORT_SMTP_PORT default 587 + REPORT_SMTP_SECURITY starttls (default) | ssl | none + REPORT_SMTP_USER optional — omit for IP-authenticated relays + REPORT_SMTP_PASSWORD optional + REPORT_SMTP_FROM e.g. "TMI Monitoring " + REPORT_SMTP_RECIPIENTS comma-separated default recipient list + REPORT_SMTP_TIMEOUT seconds, default 30 +""" + +from __future__ import annotations + +import logging +import os +import smtplib +import ssl +from dataclasses import dataclass, field +from email.message import EmailMessage +from typing import Optional + +logger = logging.getLogger(__name__) + +# Convenient MIME type for the Excel attachment. +XLSX_MIME = ("application", "vnd.openxmlformats-officedocument.spreadsheetml.sheet") + + +@dataclass +class Attachment: + filename: str + content: bytes + maintype: str = "application" + subtype: str = "octet-stream" + + +@dataclass +class SMTPConfig: + host: str = "" + port: int = 587 + security: str = "starttls" # "starttls" | "ssl" | "none" + user: str = "" + password: str = "" + sender: str = "" + recipients: list[str] = field(default_factory=list) + timeout: float = 30.0 + + @classmethod + def from_env(cls) -> "SMTPConfig": + rec = os.getenv("REPORT_SMTP_RECIPIENTS", "") + return cls( + host=os.getenv("REPORT_SMTP_HOST", "").strip(), + port=int(os.getenv("REPORT_SMTP_PORT", "587") or 587), + security=os.getenv("REPORT_SMTP_SECURITY", "starttls").strip().lower(), + user=os.getenv("REPORT_SMTP_USER", "").strip(), + password=os.getenv("REPORT_SMTP_PASSWORD", ""), + sender=os.getenv("REPORT_SMTP_FROM", "").strip(), + recipients=[r.strip() for r in rec.split(",") if r.strip()], + timeout=float(os.getenv("REPORT_SMTP_TIMEOUT", "30") or 30), + ) + + @property + def configured(self) -> bool: + """True only when we have enough to actually send (host + from).""" + return bool(self.host and self.sender) + + +def build_message( + cfg: SMTPConfig, + subject: str, + html_body: str, + recipients: list[str], + attachments: Optional[list[Attachment]] = None, + text_body: Optional[str] = None, +) -> EmailMessage: + """Assemble a multipart message: plain-text fallback + HTML + attachments.""" + msg = EmailMessage() + msg["From"] = cfg.sender or "terra-view@localhost" + msg["To"] = ", ".join(recipients) + msg["Subject"] = subject + # Plain-text part first, then the HTML alternative (clients prefer the HTML). + msg.set_content(text_body or "This report is best viewed in an HTML email client.") + msg.add_alternative(html_body, subtype="html") + for att in (attachments or []): + msg.add_attachment( + att.content, maintype=att.maintype, subtype=att.subtype, filename=att.filename, + ) + return msg + + +def send_report_email( + subject: str, + html_body: str, + *, + attachments: Optional[list[Attachment]] = None, + recipients: Optional[list[str]] = None, + text_body: Optional[str] = None, + cfg: Optional[SMTPConfig] = None, +) -> dict: + """Send (or dry-run) the report email. + + Returns a result dict: {sent, dry_run, recipients, error}. Never raises on + a send failure — it logs and returns error, so the orchestrator can record + the failure without aborting the rest of the pipeline. + """ + cfg = cfg or SMTPConfig.from_env() + recipients = recipients if recipients is not None else cfg.recipients + result = {"sent": False, "dry_run": False, "recipients": recipients, "error": None} + + if not recipients: + result["error"] = "No recipients configured" + logger.warning("Report email: no recipients set; skipping send of %r", subject) + return result + + msg = build_message(cfg, subject, html_body, recipients, attachments, text_body) + + if not cfg.configured: + result["dry_run"] = True + logger.info( + "Report email DRY-RUN (SMTP not configured): would send %r to %s with %d attachment(s)", + subject, recipients, len(attachments or []), + ) + return result + + # Validate the security mode: an unrecognized value (typo) must NOT silently + # fall through to a plaintext connection while still sending credentials. + sec = cfg.security if cfg.security in ("ssl", "starttls", "none") else "starttls" + if sec != cfg.security: + logger.warning("Unknown REPORT_SMTP_SECURITY=%r — falling back to 'starttls'", cfg.security) + + try: + if sec == "ssl": + ctx = ssl.create_default_context() + with smtplib.SMTP_SSL(cfg.host, cfg.port, timeout=cfg.timeout, context=ctx) as s: + if cfg.user: + s.login(cfg.user, cfg.password) + s.send_message(msg) + else: + with smtplib.SMTP(cfg.host, cfg.port, timeout=cfg.timeout) as s: + s.ehlo() + if sec == "starttls": + s.starttls(context=ssl.create_default_context()) + s.ehlo() + if cfg.user: + if sec == "none": + logger.warning( + "Sending SMTP credentials over an UNENCRYPTED connection " + "(REPORT_SMTP_SECURITY=none) — set starttls/ssl if the relay supports it." + ) + s.login(cfg.user, cfg.password) + s.send_message(msg) + result["sent"] = True + logger.info("Report email sent: %r to %s", subject, recipients) + except Exception as e: # noqa: BLE001 — surface as result, never abort the pipeline + result["error"] = str(e) + logger.error("Report email send failed: %s", e, exc_info=True) + + return result diff --git a/backend/services/report_orchestrator.py b/backend/services/report_orchestrator.py new file mode 100644 index 0000000..3fca051 --- /dev/null +++ b/backend/services/report_orchestrator.py @@ -0,0 +1,150 @@ +""" +Nightly Report Orchestrator. + +Ties the pieces together: compute → render → write-to-disk → email. + +This is what the daily cycle (or a manual trigger) calls. It ALWAYS writes the +rendered report to disk — `data/reports/{project_id}/{night_date}/report.html` +(+ `report.json` with the raw numbers) — so there's a viewable artifact even +when email is in dry-run (SMTP not configured yet). The email step is +best-effort and never aborts the run. +""" + +from __future__ import annotations + +import json +import logging +from datetime import date +from pathlib import Path +from typing import Optional + +from sqlalchemy.orm import Session + +from backend.services.report_pipeline import ( + ProjectNightReport, build_project_night_report, Window, +) +from backend.services.report_renderers import render_html_summary, render_excel +from backend.services.report_email import send_report_email, Attachment, XLSX_MIME + +logger = logging.getLogger(__name__) + +DEFAULT_OUTPUT_ROOT = "data/reports" + + +def _report_to_dict(report: ProjectNightReport) -> dict: + """Serialise the report data model to plain JSON (for the on-disk record).""" + return { + "project_id": report.project_id, + "project_name": report.project_name, + "night_date": report.night_date.isoformat(), + "metrics": [m.key for m in report.metrics], + "locations": [ + { + "name": loc.location_name, + "night_interval_count": loc.night_interval_count, + "baseline_nights_used": loc.baseline_nights_used, + "notes": loc.notes, + "windows": { + w.key: { + "label": w.label, + "metrics": { + m.key: { + "label": m.label, + "last_night": loc.table[w.key][m.key].last_night, + "baseline": loc.table[w.key][m.key].baseline, + "delta": loc.table[w.key][m.key].delta, + } + for m in loc.metrics + }, + } + for w in loc.windows + }, + } + for loc in report.locations + ], + } + + +def run_nightly_report( + db: Session, + project_id: str, + night_date: date, + *, + metric_keys: Optional[list[str]] = None, + windows: Optional[list[Window]] = None, + baseline_mode: str = "captured", + baseline_start: Optional[date] = None, + baseline_end: Optional[date] = None, + recipients: Optional[list[str]] = None, + output_root: str = DEFAULT_OUTPUT_ROOT, + send: bool = True, +) -> dict: + """Build, persist, and (dry-run) email the night report for a project. + + Returns a result dict with the on-disk artifact paths and the email result. + Designed to be called from the daily cycle or a manual trigger. + """ + report = build_project_night_report( + db, project_id, night_date, + metric_keys=metric_keys, windows=windows, + baseline_mode=baseline_mode, + baseline_start=baseline_start, baseline_end=baseline_end, + ) + + html = render_html_summary(report) + subject = f"{report.project_name} — night report {night_date:%m/%d/%y}" + + # --- Always persist a viewable copy --- + out_dir = Path(output_root) / project_id / f"{night_date:%Y-%m-%d}" + out_dir.mkdir(parents=True, exist_ok=True) + html_path = out_dir / "report.html" + html_path.write_text(html, encoding="utf-8") + json_path = out_dir / "report.json" + json_path.write_text(json.dumps(_report_to_dict(report), indent=2), encoding="utf-8") + + # --- Excel (the email attachment; also written to disk for the archive) --- + attachments: list[Attachment] = [] + xlsx_path = None + try: + xlsx_bytes = render_excel(report) + xlsx_path = out_dir / "report.xlsx" + xlsx_path.write_bytes(xlsx_bytes) + safe_name = "".join(c for c in report.project_name if c.isalnum() or c in " -_").strip().replace(" ", "_") + attachments.append(Attachment( + f"{safe_name or 'report'}_{night_date:%Y-%m-%d}_night_report.xlsx", + xlsx_bytes, *XLSX_MIME, + )) + except Exception as e: # noqa: BLE001 — never let the spreadsheet sink the report + logger.error("Excel render failed for %s (%s): %s", project_id, night_date, e, exc_info=True) + + # --- Email (best-effort; dry-run until SMTP is configured) --- + email_result = {"sent": False, "dry_run": False, "skipped": True, "error": None} + if send: + try: + email_result = send_report_email( + subject, html, attachments=attachments, recipients=recipients, + ) + except Exception as e: # noqa: BLE001 — artifacts are already written; never abort on email + logger.error("send_report_email raised for %s (%s): %s", project_id, night_date, e, exc_info=True) + email_result = {"sent": False, "dry_run": False, "skipped": False, "error": str(e)} + + result = { + "project_id": project_id, + "project_name": report.project_name, + "night_date": night_date.isoformat(), + "subject": subject, + "location_count": len(report.locations), + "html_path": str(html_path), + "json_path": str(json_path), + "xlsx_path": str(xlsx_path) if xlsx_path else None, + "html": html, # for callers that want to display it inline + "email": email_result, + } + logger.info( + "Nightly report for %s (%s): %d location(s) → %s; email=%s", + report.project_name, night_date, len(report.locations), html_path, + "sent" if email_result.get("sent") else + ("dry-run" if email_result.get("dry_run") else + ("skipped" if email_result.get("skipped") else f"error: {email_result.get('error')}")), + ) + return result diff --git a/backend/services/report_pipeline.py b/backend/services/report_pipeline.py new file mode 100644 index 0000000..7448e7f --- /dev/null +++ b/backend/services/report_pipeline.py @@ -0,0 +1,432 @@ +""" +Nightly Report Pipeline — computation core. + +Builds the data model for the John-Myler-style "last night vs. baseline" sound +report. Source-agnostic: it reads the same on-disk Leq `.rnd` files the manual +upload + FTP-pull ingest produce (see `project_locations.ingest_nrl_zip`). + +Design notes +------------ +* **Ingest everything, report selectively.** Ingest preserves every column of + the Leq file; this layer chooses which *metrics* to surface via `metric_keys` + (a future report wizard is just a UI over that list). +* **House format match.** Defaults reproduce the existing Excel report: + LAmax (max of interval maxima), LA01 / LA10 (arithmetic average), split into + Evening (7–10PM) and Nighttime (10PM–7AM) windows. L90 (background) is added + for the baseline comparison. +* **Metric labelling from the device.** The LN→percentile assignment is + reconfigurable per job; we resolve which `LNx(Main)` column is L90/L10/etc. + from the percentile map captured in the session metadata at ingest, falling + back to the NL-43 default order. +* **Correct averaging.** Leq is energy-averaged (logarithmic); percentiles and + Lmax are arithmetic. Baseline references combine the per-night values into a + "typical night" (arithmetic mean of per-night values — so baseline Lmax is the + typical nightly peak, not the worst-of-week). +""" + +from __future__ import annotations + +import json +import logging +import math +from dataclasses import dataclass, field +from datetime import datetime, timedelta, date +from typing import Optional + +from sqlalchemy.orm import Session + +from backend.models import MonitoringSession, DataFile, MonitoringLocation, Project + +logger = logging.getLogger(__name__) + + +# --------------------------------------------------------------------------- +# Metric registry +# --------------------------------------------------------------------------- + +@dataclass(frozen=True) +class Metric: + """A reportable metric. + + `agg` is the *within-night* aggregation used to collapse a window's 15-min + intervals into one value: + - "max" → loudest interval (LAmax) + - "arith" → arithmetic mean (percentiles: L01/L10/L90…) + - "log" → energy/logarithmic mean (Leq only) + `column` pins a fixed .rnd column; `percentile` instead resolves the LNx + column from the session's captured percentile map. + """ + key: str + label: str + agg: str + column: Optional[str] = None + percentile: Optional[float] = None + + +METRIC_REGISTRY: dict[str, Metric] = { + "lmax": Metric("lmax", "LAmax", "max", column="Lmax(Main)"), + "leq": Metric("leq", "LAeq", "log", column="Leq(Main)"), + "lmin": Metric("lmin", "LAmin", "arith", column="Lmin(Main)"), + "l01": Metric("l01", "LA01", "arith", percentile=1.0), + "l10": Metric("l10", "LA10", "arith", percentile=10.0), + "l50": Metric("l50", "LA50", "arith", percentile=50.0), + "l90": Metric("l90", "LA90", "arith", percentile=90.0), + "l95": Metric("l95", "LA95", "arith", percentile=95.0), +} + +# House report metrics + L90 (background) for the baseline comparison. +DEFAULT_METRICS: list[str] = ["lmax", "l01", "l10", "l90"] + +# NL-43 default percentile→slot assignment, used when a session has no captured map. +_DEFAULT_SLOT_FOR_PCT: dict[float, int] = {1.0: 1, 10.0: 2, 50.0: 3, 90.0: 4, 95.0: 5} + + +def _resolve_column(metric: Metric, pct_map: dict) -> Optional[str]: + """Resolve the .rnd column for a metric, using the session's percentile map.""" + if metric.column: + return metric.column + if metric.percentile is None: + return None + # pct_map: {"1": "1.0", "2": "10.0", "4": "90.0", ...} → slot : percentile + if pct_map: + for slot, pval in pct_map.items(): + try: + if float(pval) == metric.percentile: + return f"LN{int(slot)}(Main)" + except (ValueError, TypeError): + continue + slot = _DEFAULT_SLOT_FOR_PCT.get(metric.percentile) + return f"LN{slot}(Main)" if slot else None + + +# --------------------------------------------------------------------------- +# Time windows +# --------------------------------------------------------------------------- + +@dataclass(frozen=True) +class Window: + key: str + label: str + start_hour: int + end_hour: int + + def contains(self, hour: int) -> bool: + if self.start_hour < self.end_hour: + return self.start_hour <= hour < self.end_hour + return hour >= self.start_hour or hour < self.end_hour + + +# Matches the existing Excel report's stats table. +DEFAULT_WINDOWS: list[Window] = [ + Window("evening", "Evening (7PM–10PM)", 19, 22), + Window("nighttime", "Nighttime (10PM–7AM)", 22, 7), +] + +# The full night used to select which intervals belong to "last night". +NIGHT_START_HOUR = 19 +NIGHT_LENGTH_HOURS = 12 + + +# --------------------------------------------------------------------------- +# Aggregation +# --------------------------------------------------------------------------- + +def _aggregate(values: list, method: str) -> Optional[float]: + """Collapse a window's interval values into one number per `method`.""" + vals = [v for v in values if isinstance(v, (int, float))] + if not vals: + return None + if method == "max": + return round(max(vals), 1) + if method == "log": + return round(10 * math.log10(sum(10 ** (v / 10.0) for v in vals) / len(vals)), 1) + return round(sum(vals) / len(vals), 1) # arithmetic + + +def _combine_across_nights(per_night: list, method: str) -> Optional[float]: + """Combine per-night window values into a baseline 'typical night' value. + + Arithmetic mean for max/arith metrics (so baseline Lmax = typical nightly + peak, the agreed default), logarithmic mean for Leq. + """ + vals = [v for v in per_night if v is not None] + if not vals: + return None + if method == "log": + return round(10 * math.log10(sum(10 ** (v / 10.0) for v in vals) / len(vals)), 1) + return round(sum(vals) / len(vals), 1) + + +# --------------------------------------------------------------------------- +# Row gathering +# --------------------------------------------------------------------------- + +def _parse_dt(s: str) -> Optional[datetime]: + try: + return datetime.strptime(s, "%Y/%m/%d %H:%M:%S") + except (ValueError, TypeError): + return None + + +def _location_leq_rows(db: Session, location_id: str) -> list[tuple[datetime, dict, dict]]: + """All Leq intervals at a location as (interval_dt, row, percentile_map). + + Reuses the same .rnd readers as the report endpoints so parsing stays + identical. Times are the meter's local clock (as written in the file). + """ + # Lazy import avoids a service→router import cycle at module load. + from backend.routers.projects import ( + _read_rnd_file_rows, _normalize_rnd_rows, _is_leq_file, _peek_rnd_headers, + ) + from pathlib import Path + + out: list[tuple[datetime, dict, dict]] = [] + sessions = db.query(MonitoringSession).filter_by( + location_id=location_id, session_type="sound", + ).all() + for s in sessions: + try: + meta = json.loads(s.session_metadata or "{}") + except (json.JSONDecodeError, TypeError): + meta = {} + pct_map = meta.get("percentiles", {}) or {} + for f in db.query(DataFile).filter_by(session_id=s.id).all(): + if not f.file_path or not f.file_path.lower().endswith(".rnd"): + continue + peek = _peek_rnd_headers(Path("data") / f.file_path) + if not _is_leq_file(f.file_path, peek): + continue + rows = _read_rnd_file_rows(f.file_path) + rows, _ = _normalize_rnd_rows(rows) + for r in rows: + dt = _parse_dt(r.get("Start Time", "")) + if dt: + out.append((dt, r, pct_map)) + out.sort(key=lambda t: t[0]) + return out + + +def _rows_in_night(rows: list, night_date: date) -> list: + """Rows falling in the night that *starts* on night_date (19:00 → +12h).""" + start = datetime(night_date.year, night_date.month, night_date.day, NIGHT_START_HOUR, 0) + end = start + timedelta(hours=NIGHT_LENGTH_HOURS) + return [(dt, r, p) for (dt, r, p) in rows if start <= dt < end] + + +def _eligible_nights(rows: list, start_date: date, end_date: date) -> list[date]: + """Evening-dates in [start_date, end_date] that actually have night data.""" + nights = [] + cur = start_date + while cur <= end_date: + if _rows_in_night(rows, cur): + nights.append(cur) + cur += timedelta(days=1) + return nights + + +def _window_value(rows: list, metric: Metric, window: Window) -> Optional[float]: + """Single aggregated value for one metric over one window of `rows`.""" + vals = [] + for dt, r, pct_map in rows: + if window.contains(dt.hour): + col = _resolve_column(metric, pct_map) + if col: + vals.append(r.get(col)) + return _aggregate(vals, metric.agg) + + +# --------------------------------------------------------------------------- +# Report data model +# --------------------------------------------------------------------------- + +@dataclass +class CellPair: + last_night: Optional[float] + baseline: Optional[float] + + @property + def delta(self) -> Optional[float]: + if self.last_night is None or self.baseline is None: + return None + return round(self.last_night - self.baseline, 1) + + +@dataclass +class LocationNightReport: + location_id: str + location_name: str + night_date: date + metrics: list[Metric] + windows: list[Window] + # table[window_key][metric_key] = CellPair + table: dict[str, dict[str, CellPair]] + interval_series: list[dict] + night_interval_count: int + baseline_nights_used: int + notes: list[str] = field(default_factory=list) + + +def _location_reference_baseline(loc) -> dict: + """A location's manually-entered reference baseline, from its metadata. + + Shape: {window_key: {metric_key: float}} e.g. {"nighttime": {"l10": 85.0}}. + Used when baseline_mode == "reference" — fixed targets/limits or prior-report + averages typed in, rather than computed from captured nights. + """ + if not loc: + return {} + try: + meta = json.loads(loc.location_metadata or "{}") + except (json.JSONDecodeError, TypeError): + return {} + ref = meta.get("report_baseline") or {} + out: dict[str, dict[str, float]] = {} + if isinstance(ref, dict): + for wkey, mvals in ref.items(): + if not isinstance(mvals, dict): + continue + clean = {} + for mkey, val in mvals.items(): + try: + clean[mkey] = float(val) + except (ValueError, TypeError): + continue + if clean: + out[wkey] = clean + return out + + +def build_location_night_report( + db: Session, + location_id: str, + night_date: date, + *, + metric_keys: Optional[list[str]] = None, + windows: Optional[list[Window]] = None, + baseline_mode: str = "captured", + baseline_start: Optional[date] = None, + baseline_end: Optional[date] = None, +) -> LocationNightReport: + """Build the night-vs-baseline data model for one location. + + `night_date` is the *evening* date of the night being reported (e.g. the + 7/7 in "night of 7/7 → morning 7/8"). Baseline comes from one of: + - "captured": the typical-night value across eligible nights in + [baseline_start, baseline_end] (computed from recorded data); + - "reference": fixed values typed per location (a spec limit like + "L10 = 85", or a prior report's averages). + """ + metric_keys = metric_keys or DEFAULT_METRICS + metrics = [METRIC_REGISTRY[k] for k in metric_keys] + windows = windows or DEFAULT_WINDOWS + + loc = db.query(MonitoringLocation).filter_by(id=location_id).first() + loc_name = loc.name if loc else location_id + + all_rows = _location_leq_rows(db, location_id) + night_rows = _rows_in_night(all_rows, night_date) + + reference = _location_reference_baseline(loc) if baseline_mode == "reference" else {} + + baseline_nights: list[date] = [] + if baseline_mode != "reference" and baseline_start and baseline_end: + baseline_nights = _eligible_nights(all_rows, baseline_start, baseline_end) + # Don't let the reported night double as its own baseline. + baseline_nights = [n for n in baseline_nights if n != night_date] + + table: dict[str, dict[str, CellPair]] = {} + for w in windows: + table[w.key] = {} + for m in metrics: + last_night_val = _window_value(night_rows, m, w) + if baseline_mode == "reference": + baseline_val = reference.get(w.key, {}).get(m.key) + elif baseline_nights: + per_night = [ + _window_value(_rows_in_night(all_rows, nd), m, w) + for nd in baseline_nights + ] + baseline_val = _combine_across_nights(per_night, m.agg) + else: + baseline_val = None + table[w.key][m.key] = CellPair(last_night_val, baseline_val) + + interval_series = [] + for dt, r, pct_map in night_rows: + entry = {"dt": dt, "time": dt.strftime("%H:%M")} + for m in metrics: + col = _resolve_column(m, pct_map) + val = r.get(col) if col else None + entry[m.key] = val if isinstance(val, (int, float)) else None + interval_series.append(entry) + + notes: list[str] = [] + if not night_rows: + notes.append(f"No data found for the night of {night_date:%m/%d/%y}.") + if baseline_mode == "reference": + if not any(reference.values()): + notes.append("Reference-baseline mode is on but no reference values are set for this location.") + elif (baseline_start or baseline_end) and not baseline_nights: + notes.append("No baseline nights with data in the configured range.") + + return LocationNightReport( + location_id=location_id, + location_name=loc_name, + night_date=night_date, + metrics=metrics, + windows=windows, + table=table, + interval_series=interval_series, + night_interval_count=len(night_rows), + baseline_nights_used=len(baseline_nights), + notes=notes, + ) + + +@dataclass +class ProjectNightReport: + project_id: str + project_name: str + night_date: date + metrics: list[Metric] + locations: list[LocationNightReport] + + +def build_project_night_report( + db: Session, + project_id: str, + night_date: date, + *, + metric_keys: Optional[list[str]] = None, + windows: Optional[list[Window]] = None, + baseline_mode: str = "captured", + baseline_start: Optional[date] = None, + baseline_end: Optional[date] = None, +) -> ProjectNightReport: + """Build the night report for every active sound location in a project.""" + metric_keys = metric_keys or DEFAULT_METRICS + project = db.query(Project).filter_by(id=project_id).first() + project_name = project.name if project else project_id + + locations = db.query(MonitoringLocation).filter_by( + project_id=project_id, location_type="sound", + ).order_by(MonitoringLocation.sort_order, MonitoringLocation.name).all() + locations = [l for l in locations if getattr(l, "removed_at", None) is None] + + reports = [ + build_location_night_report( + db, loc.id, night_date, + metric_keys=metric_keys, windows=windows, + baseline_mode=baseline_mode, + baseline_start=baseline_start, baseline_end=baseline_end, + ) + for loc in locations + ] + + return ProjectNightReport( + project_id=project_id, + project_name=project_name, + night_date=night_date, + metrics=[METRIC_REGISTRY[k] for k in metric_keys], + locations=reports, + ) diff --git a/backend/services/report_renderers.py b/backend/services/report_renderers.py new file mode 100644 index 0000000..d1063ae --- /dev/null +++ b/backend/services/report_renderers.py @@ -0,0 +1,240 @@ +""" +Nightly Report Renderers. + +Pluggable renderers over the `report_pipeline` data model. v1 ships the HTML +email body + the Excel attachment; PDF and an inline chart image are v1.1 +(each needs a new dependency). Keeping renderers separate from the compute +core means a future report wizard just toggles metrics/renderers — the data +model is unchanged. + +Email-client constraints: the HTML uses a table layout with **inline styles +only** (no + {% block head %}{% endblock %} + + +
+
+ + + + TMI Monitoring + {% if client %}/ + {{ client.name }}{% endif %} + + +
+ + {% if client %} + Sign out + {% endif %} +
+
+
+ +
+ {% block content %}{% endblock %} +
+ +
+ + Read-only monitoring view · data provided as-is for informational purposes +
+ + + {% block scripts %}{% endblock %} + + diff --git a/templates/portal/location.html b/templates/portal/location.html new file mode 100644 index 0000000..377117f --- /dev/null +++ b/templates/portal/location.html @@ -0,0 +1,335 @@ +{% extends "portal/base.html" %} +{% block title %}{{ location.name }}{% endblock %} +{% block head %} + +{% endblock %} +{% block content %} + + + All locations + +
+

{{ location.name }}

+
+ + +
+
+ +{% if not has_device %} +
No device is currently assigned to this location.
+{% else %} + + + +
+
Leq · average
+
+ -- + dB +
+ +
+
+
Lp · instant
+
--dB
+
+
+
Lmax · peak
+
--dB
+
+
+
L1
+
--dB
+
+
+
L10
+
--dB
+
+
+
+ + +
+
Live trace · last 2h
+
+ + +
+
+ + + + + +
+
Alert history
+
+
+{% endif %} +{% endblock %} + +{% block scripts %} +{% if has_device %} + +{% endif %} +{% endblock %} diff --git a/templates/portal/overview.html b/templates/portal/overview.html new file mode 100644 index 0000000..941063d --- /dev/null +++ b/templates/portal/overview.html @@ -0,0 +1,192 @@ +{% extends "portal/base.html" %} +{% block title %}Your locations{% endblock %} +{% block head %} + +{% endblock %} +{% block content %} +
+
Live monitoring
+

Your locations

+

Real-time sound levels across your active monitoring sites.

+
+ +{% if locations %} + + + + + +
+ {% for loc in locations %} + +
+
+
{{ loc.name }}
+
{{ loc.address or loc.project_name or '' }}
+
+ +
+
+ -- + dB Leq +
+
 
+ + {% endfor %} +
+{% else %} +
No active monitoring locations yet.
+{% endif %} +{% endblock %} + +{% block scripts %} + + +{% endblock %} diff --git a/templates/portal/password.html b/templates/portal/password.html new file mode 100644 index 0000000..93b12a1 --- /dev/null +++ b/templates/portal/password.html @@ -0,0 +1,26 @@ +{% extends "portal/base.html" %} +{% block title %}{{ project_name }}{% endblock %} +{% block content %} +
+
+ + + +
+

{{ project_name }}

+

Enter the password to view this monitoring portal.

+ {% if error %} +

{{ error }}

+ {% endif %} +
+ + + +
+
+{% endblock %} diff --git a/templates/projects/detail.html b/templates/projects/detail.html index ba971f3..b7a35b6 100644 --- a/templates/projects/detail.html +++ b/templates/projects/detail.html @@ -4,7 +4,7 @@ {% block content %} -
+
+ + +
+ + + + + + + Preview + +
@@ -2074,5 +2096,97 @@ document.addEventListener('DOMContentLoaded', function() { } }); + + + + + + {% endblock %} diff --git a/templates/slm_detail.html b/templates/slm_detail.html index 6a17ea8..3aa8fdf 100644 --- a/templates/slm_detail.html +++ b/templates/slm_detail.html @@ -112,4 +112,267 @@
+ + +
+
+
+

Alerts + +

+

Threshold rules evaluated on this device's live feed. An enabled alert keeps the device monitored 24/7.

+
+ +
+ +
+ + + + + +
+
+

History

+ +
+
+
+
+ + {% endblock %} diff --git a/templates/sound_level_meters.html b/templates/sound_level_meters.html index a843612..123e1d8 100644 --- a/templates/sound_level_meters.html +++ b/templates/sound_level_meters.html @@ -51,13 +51,31 @@ @@ -150,9 +168,18 @@ window.selectedUnitId = null; window.dashboardChartData = { timestamps: [], lp: [], - leq: [] + leq: [], + ln1: [], + ln2: [] }; +// Parse a metric to a number, or null (so a missing/"-.-" percentile leaves a gap +// in the line instead of dropping it to 0). +function numOrNull(v) { + const f = parseFloat(v); + return isNaN(f) ? null : f; +} + // Initialize Chart.js function initializeDashboardChart() { if (typeof Chart === 'undefined') { @@ -194,6 +221,26 @@ function initializeDashboardChart() { tension: 0.3, borderWidth: 2, pointRadius: 0 + }, + { + label: 'L1', + data: [], + borderColor: 'rgb(168, 85, 247)', + backgroundColor: 'rgba(168, 85, 247, 0.1)', + tension: 0.3, + borderWidth: 2, + pointRadius: 0, + spanGaps: true + }, + { + label: 'L10', + data: [], + borderColor: 'rgb(249, 115, 22)', + backgroundColor: 'rgba(249, 115, 22, 0.1)', + tension: 0.3, + borderWidth: 2, + pointRadius: 0, + spanGaps: true } ] }, @@ -244,12 +291,24 @@ function showLiveChart(unitId) { initializeDashboardChart(); } - // Reset data - window.dashboardChartData = { - timestamps: [], - lp: [], - leq: [] - }; + // Reset data for the newly-selected unit (clears any prior unit's line) + window.dashboardChartData = { timestamps: [], lp: [], leq: [], ln1: [], ln2: [] }; + if (window.dashboardChart) { + window.dashboardChart.data.labels = []; + window.dashboardChart.data.datasets.forEach(ds => ds.data = []); + window.dashboardChart.update('none'); + } + + // Name the unit; clear stale status until the cache read returns + const unitLabel = document.getElementById('panel-unit-id'); + if (unitLabel) unitLabel.textContent = '· ' + unitId; + setPanelStatus(null, null); + + // Populate immediately from CACHE (no device hit): KPI cards + chart trail. + prefillDashboardPanel(unitId); + backfillDashboardChart(unitId); + // Keep the cards updating from cache (~15s) without opening a device stream. + startPanelCachePolling(unitId); // Scroll to chart panel.scrollIntoView({ behavior: 'smooth', block: 'start' }); @@ -257,6 +316,7 @@ function showLiveChart(unitId) { function closeLiveChart() { stopDashboardStream(); + stopPanelCachePolling(); document.getElementById('live-chart-panel').classList.add('hidden'); window.selectedUnitId = null; } @@ -270,17 +330,12 @@ function startDashboardStream() { window.dashboardWebSocket.close(); } - // Reset chart data - window.dashboardChartData = { timestamps: [], lp: [], leq: [] }; - if (window.dashboardChart) { - window.dashboardChart.data.labels = []; - window.dashboardChart.data.datasets[0].data = []; - window.dashboardChart.data.datasets[1].data = []; - window.dashboardChart.update(); - } + // The live WS takes over from the cache poller; keep the backfilled trail on + // the chart so the live frames continue the line instead of blanking it. + stopPanelCachePolling(); const wsProtocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:'; - const wsUrl = `${wsProtocol}//${window.location.host}/api/slmm/${window.selectedUnitId}/live`; + const wsUrl = `${wsProtocol}//${window.location.host}/api/slmm/${window.selectedUnitId}/monitor`; window.dashboardWebSocket = new WebSocket(wsUrl); @@ -293,6 +348,10 @@ function startDashboardStream() { window.dashboardWebSocket.onmessage = function(event) { try { const data = JSON.parse(event.data); + // /monitor sends keepalive 'heartbeat' frames (no metrics) and a per-frame + // 'feed_status'; skip heartbeats and offline frames so they don't blank the + // metrics or spike the chart with zeros. + if (data.heartbeat || data.feed_status === 'unreachable') return; updateDashboardMetrics(data); updateDashboardChart(data); } catch (error) { @@ -316,37 +375,219 @@ function stopDashboardStream() { window.dashboardWebSocket.close(); window.dashboardWebSocket = null; } + // Fall back to cache polling so the cards keep refreshing while the panel is open. + if (window.selectedUnitId && !document.getElementById('live-chart-panel').classList.contains('hidden')) { + startPanelCachePolling(window.selectedUnitId); + } } function updateDashboardMetrics(data) { document.getElementById('chart-lp').textContent = data.lp || '--'; document.getElementById('chart-leq').textContent = data.leq || '--'; document.getElementById('chart-lmax').textContent = data.lmax || '--'; - document.getElementById('chart-lmin').textContent = data.lmin || '--'; - document.getElementById('chart-lpeak').textContent = data.lpeak || '--'; + // Guard: DRD stream frames omit percentiles, so only overwrite when present + // (else the live stream blanks L1/L10 over the cached DOD snapshot values). + if (data.ln1 != null) document.getElementById('chart-ln1').textContent = data.ln1; + if (data.ln2 != null) document.getElementById('chart-ln2').textContent = data.ln2; + if (data.ln1_label) document.getElementById('chart-ln1-label').textContent = data.ln1_label; + if (data.ln2_label) document.getElementById('chart-ln2-label').textContent = data.ln2_label; } function updateDashboardChart(data) { + const cd = window.dashboardChartData; const now = new Date(); - window.dashboardChartData.timestamps.push(now.toLocaleTimeString()); - window.dashboardChartData.lp.push(parseFloat(data.lp || 0)); - window.dashboardChartData.leq.push(parseFloat(data.leq || 0)); + cd.timestamps.push(now.toLocaleTimeString()); + cd.lp.push(numOrNull(data.lp)); + cd.leq.push(numOrNull(data.leq)); + // /monitor (DOD) frames carry ln1/ln2; a DRD frame would omit them -> null gap. + cd.ln1.push(numOrNull(data.ln1)); + cd.ln2.push(numOrNull(data.ln2)); - // Keep only last 60 data points - if (window.dashboardChartData.timestamps.length > 60) { - window.dashboardChartData.timestamps.shift(); - window.dashboardChartData.lp.shift(); - window.dashboardChartData.leq.shift(); + // Keep a generous window (backfill seeds up to ~120 points from the 2h trail). + if (cd.timestamps.length > 600) { + cd.timestamps.shift(); + cd.lp.shift(); + cd.leq.shift(); + cd.ln1.shift(); + cd.ln2.shift(); } if (window.dashboardChart) { - window.dashboardChart.data.labels = window.dashboardChartData.timestamps; - window.dashboardChart.data.datasets[0].data = window.dashboardChartData.lp; - window.dashboardChart.data.datasets[1].data = window.dashboardChartData.leq; + window.dashboardChart.data.labels = cd.timestamps; + window.dashboardChart.data.datasets[0].data = cd.lp; + window.dashboardChart.data.datasets[1].data = cd.leq; + window.dashboardChart.data.datasets[2].data = cd.ln1; + window.dashboardChart.data.datasets[3].data = cd.ln2; window.dashboardChart.update('none'); } } +// ---- Cached-data panel population (no device hit) ----------------------- + +// Fill the KPI cards + measuring/freshness from the cached NL43Status snapshot. +async function prefillDashboardPanel(unitId) { + try { + const r = await fetch(`/api/slmm/${encodeURIComponent(unitId)}/status`); + if (!r.ok) { // 404 = device has never reported yet + setPanelStatus(null, null); + return; + } + const d = (await r.json()).data || {}; + updateDashboardMetrics(d); // lp/leq/lmax/ln1/ln2 (ln guards keep cached percentiles) + const measuring = d.measurement_state === 'Start' || d.measurement_state === 'Measure'; + setPanelStatus(measuring, d.last_seen); + } catch (e) { + console.warn('Panel cache prefill failed:', e); + } +} + +// Seed the chart from the downsampled DOD trail so it shows recent trend on open. +async function backfillDashboardChart(unitId) { + try { + const r = await fetch(`/api/slmm/${encodeURIComponent(unitId)}/history?hours=2`); + if (!r.ok) return; + const readings = (await r.json()).readings || []; + const cd = window.dashboardChartData; + if (!cd) return; + for (const row of readings) { + // Trail timestamps are naive UTC; append 'Z' to render in local time + // consistently with the live frames (which use local Date.now()). + cd.timestamps.push(row.timestamp ? new Date(row.timestamp + 'Z').toLocaleTimeString() : ''); + cd.lp.push(numOrNull(row.lp)); + cd.leq.push(numOrNull(row.leq)); + cd.ln1.push(numOrNull(row.ln1)); + cd.ln2.push(numOrNull(row.ln2)); + } + if (window.dashboardChart) { + window.dashboardChart.data.labels = cd.timestamps; + window.dashboardChart.data.datasets[0].data = cd.lp; + window.dashboardChart.data.datasets[1].data = cd.leq; + window.dashboardChart.data.datasets[2].data = cd.ln1; + window.dashboardChart.data.datasets[3].data = cd.ln2; + window.dashboardChart.update('none'); + } + } catch (e) { + console.warn('Panel chart backfill failed:', e); + } +} + +// Measuring badge + "as of