0.14.0 update from dev - client portal, SLMM expansion, multistream support. #67
Assignees
serversdown
Clear assignees
serversdown
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: serversdown/terra-view#67
Reference in New Issue
Block a user
Delete Branch "dev"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
SLM live monitoring — fan-out feed + cache-first reads. Targets 0.14.0. The throughline: the NL-43 allows exactly one TCP connection at a time, so every page that opened its own device stream (or sent its own
Measure?/DOD on load) was competing for that single connection — a second viewer saw nothing, and dashboard loads stole polling resolution from the live feed. This release moves Terra-View entirely onto SLMM's shared, cached monitoring: one DOD poll loop per device, fanned out to all viewers; dashboards read SLMM's cache (a DB read on SLMM's side) instead of touching the device; and the live panels populate instantly from cache on open, upgrading to the live WS only on demand. Paired with the SLMM-side work (adaptive poll rate, unreachable backoff, device-offline alert) on SLMM branchdev.Added
/monitorfeed consumption. The unit live view (partials/slm_live_view.html) and the dashboard live tile (sound_level_meters.html) now subscribe to SLMM's shared per-device monitor overWS /api/slmm/{unit}/monitorinstead of each opening its own device stream. Any number of clients attach without each consuming the NL-43's single connection — the "second viewer sees nothing" contention is gone. A WS proxy handler for/monitorwas added tobackend/routers/slmm.py.ln1/ln2(DRD streaming can't carry percentiles, DOD can). Missing/-.-values leave a gap rather than dropping the line to 0.GET /api/slmm/{unit}/history?hours=2) so a viewer sees recent trend immediately instead of a blank chart that fills one point per second./statusand backfills the chart from/history— pure cache reads, no device hit. Shows a measuring badge (● Measuring / ■ Stopped) and a freshness stamp ("as of 3:48 PM (10s ago)", amber + "cached" when stale). Re-polls the cache every 15s while open; Start Live Stream upgrades to the live WS and no longer wipes the backfilled trail (chart point cap raised 60 → 600).GET /api/slmm/{unit}/live(which also refreshes SLMM's cache), with a spinner + success/error toast, then reloads the device list./admin/slmm— turns a device's server-side keepalive feed on/off (POST /monitor/start|stop), so alerting can keep a device's feed running with no browser attached.Changed
slm_dashboard.py'sget_slm_unitspulls each unit's cached status from SLMM's/roster(one call, a SLMM DB read) for the badge + freshness; the command-centerget_live_viewreads cached/statusinstead of sendingMeasure?+ a fresh DOD on every load. This stops dashboard loads from stealing the device's single connection from the live monitor. The elapsed-measurement timer still works becausemeasurement_start_timeis now included in the cached/statusresponse.last_seen(which the monitor advances on every successful poll) viaunit.cache_last_seen, instead of theslm_last_checkroster field the monitor never updates. The status badge also treatsMeasureas Measuring, matching the panel and SLMM's cache.Fixed
can't access property "dispatchEvent", e is null.toggleSLMDeployed()and the save-config path calledhtmx.trigger('#slm-list', 'load')guarded only bytypeof htmx !== 'undefined'; no page has a#slm-list, so htmx resolved null and callednull.dispatchEvent(...). The deploy POST had already succeeded, so the operator saw both the green success and a red error. Both call sites now guard on the element existing (slm_settings_modal.html).CancelledError/ "task exception never retrieved" on stream stop — the cleanup awaited pending tasks but only caughtException, missingCancelledError(aBaseException).slm_last_checkroster field instead of SLMM's live cache (see Changed).Upgrade Notes
Requires the matching SLMM build (branch
dev) — Terra-View now depends on SLMM's fan-out/monitorfeed,/historytrail,/statuscarryingln1/ln2+measurement_start_time, cached/rosterstatus, and themonitor_enabledkeepalive flag.The two builds must ship together. Note the
docker-compose.ymlcontainer was renamed for clarity (nowterra-view-terra-view-1) — adjust anydocker execscripts that referenced the old name.Client portal (new — read-only client-facing view)
A scoped, read-only portal at
/portal/*where a client sees only theirlocations, live. Built inside Terra-View (no new service), reusing the cached
SLMM feed; every route resolves the client through one swappable
get_current_clientgate, so the interim magic/open-link auth can be replaced(M4) without touching routes or templates. Strictly read-only — no device control.
Added
Client,ClientAccessToken, and aProject.client_idFK. A signed (HMAC) session cookie carries the access-tokenid, re-validated against the DB each request (revoke kills live sessions, with
server-side expiry). Entry via a magic link (
/portal/enter/{token}) or adev-only plain link (
/portal/open/{id},PORTAL_OPEN_LINKS, default off).instantly from cache, then upgrade to a real ~1 Hz WebSocket stream scoped to
the client's unit (a scrubbed bridge to the SLMM fan-out feed). The stream
auto-closes when the tab is hidden (Page Visibility) and after a 15-min idle
cap, so an abandoned tab can't pin the device at 1 Hz / burn cellular.
tiles) + a status rollup (live/offline counts, "loudest now"). Leq is the
headline metric.
page (proxying SLMM's alert CRUD); breach history + ack internally and a
read-only, scrubbed history + current-alarm banner + "your alert limits" panel
in the portal; enabling a rule pins that device's monitor on so alerts evaluate
round-the-clock.
"Copy client link" modal (mint / list / revoke magic links) on the project
page, plus a
backend/portal_admin.pyCLI.IBM Plex Mono readouts, panel system, pulsing live dot, staggered reveal — with a
light/dark toggle (light default, persisted, no-flash).
Security
endpoints return scrubbed projections (no device-health/internal ids); WS
frames whitelisted; operator-set strings HTML-escaped before injection (XSS).
Pre-merge code review hardened cookie expiry, open-links default, and the slug
collision. Remaining hardening (reverse proxy, TLS,
SECRET_KEY, M4 auth) istracked in
docs/CLIENT_PORTAL.md→ "Security hardening backlog".Upgrade Notes
docker compose exec web-app python3 backend/migrate_add_client_portal.py(adds
projects.client_id; theclients/client_access_tokenstablesauto-create).
SECRET_KEYin any internet-facing env (signs session cookies),and keep
PORTAL_OPEN_LINKS=falsethere.devalert engine (rules/events/evaluator +cooldown + keepalive coupling) — same build pairing as above.
Portal authentication (Phase 1)
PORTAL_OPEN_LINKSopen links and theportal_admin.py mint-linkcommand.argon2-cffidependency → rebuild the image, then runpython3 backend/migrate_add_project_portal_auth.pyper DB (adds theprojects.portal_*columns).SECRET_KEYandCOOKIE_SECUREare now passed through indocker-compose.yml(settable via a.envfile) — set a realSECRET_KEY(andCOOKIE_SECURE=trueonce on HTTPS) before the portal faces the internet.The SLM live view now consumes SLMM's shared DOD /monitor feed instead of the per-client DRD /stream. This fixes the single-connection contention (many viewers share one device feed) and finally puts L1/L10 in the live chart (DRD couldn't carry percentiles). - New WS proxy handler /api/slmm/{unit}/monitor -> SLMM /api/nl43/{unit}/monitor. Uses asyncio.wait(FIRST_COMPLETED) + cancel-sibling instead of gather(), so it doesn't leave a task sending into a closed socket ("Unexpected ASGI message after close"). - Live view JS points at /monitor; onmessage reflects feed_status and ignores heartbeat / unreachable frames so they don't blank the cards or zero-spike the chart. Adds a small Live/Device-offline badge. Still on the old /live (DRD): the dashboard live tile (sound_level_meters.html) — next slice. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>The /monitor WS proxy cancelled its sibling task on disconnect but then `except Exception` failed to swallow the resulting CancelledError (a BaseException), so stopping the stream raised "Exception in ASGI application". It also only awaited the pending task, leaving the done task's WebSocketDisconnect unretrieved ("Task exception was never retrieved"). Await all tasks and catch (CancelledError, Exception). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Adds a "Live Monitoring (keepalive)" card listing each SLMM device with its monitor_enabled state and an Enable/Disable toggle. Reads from /api/slmm/roster (now includes monitor_enabled) and POSTs to /api/slmm/{unit}/monitor/{start,stop}, which persist the flag in SLMM (survives restarts; auto-started on boot). Shows a reachability dot + 24/7 ON/OFF badge. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>On opening the live view, fetch GET /api/slmm/{unit}/history?hours=2 and seed the chart with the recent trend BEFORE connecting the live socket, so it opens with context instead of blank. Live frames then append in order. - backfillChart() populates all four series (Lp/Leq/L1/L10) from the trail. - initLiveDataStream is async and awaits the backfill before opening the WS. - Chart rolling window raised 60 -> 600 points so the ~2h backfill (1/min) isn't immediately shifted out. - Trail timestamps are naive UTC -> append 'Z' so they localize consistently with the live frames. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>toggleSLMDeployed() and the save-config success path both called htmx.trigger('#slm-list', 'load') guarded only by `typeof htmx !== 'undefined'`. No page actually has a #slm-list element, so htmx resolved the selector to null and called null.dispatchEvent(...) -> "can't access property dispatchEvent, e is null". The deploy POST had already succeeded and the green success message had already rendered, so the user saw both "Unit marked as deployed." and a red error. Guard the trigger on the element existing so it's a harmless no-op. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Live Measurements panel no longer sits blank until you click Start Live Stream: - On open it fills the KPI cards from the cached /status snapshot (lp/leq/lmax/ L1/L10) and backfills the chart from the /history DOD trail — both pure cache reads, no device hit. - Shows measuring state (● Measuring / ■ Stopped) and a freshness stamp ("as of 2:14 PM (12m ago)") that turns amber + "cached" when stale, so a cached value is never mistaken for a live reading. - Polls the cache every 15s while open so the cards stay current without opening a device stream; Start Live Stream takes over (and no longer wipes the backfilled trail). Chart cap raised 60 -> 600 so the 2h backfill isn't truncated. Refresh buttons (on-demand, user-initiated single device read via GET /live, which also updates the cache): - one per device row in the list, and one in the panel header. Spinner while in flight; toast on success/failure; reloads the list so badges + last-check update. Layout fix: the status badge (Measuring/Active/Idle/Benched) was rendered at the top-right of the card, colliding with the absolutely-positioned chart/gear icons. Moved it to the bottom meta row next to "Last check", padded the card content clear of the action icons, and added the refresh icon to that group. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>backend/portal_auth.py: stdlib HMAC-signed session cookie carrying the access- token id (re-validated against the DB each request, so revoke kills live sessions), hash_token, resolve_token, and the get_current_client dependency (raises PortalAuthError). SECRET_KEY env (insecure dev default + warning). routers/portal.py: /portal/enter/{token} mints the cookie -> /portal; /logout; /access; /portal home stub. main.py registers the router + a PortalAuthError handler (HTML access page for pages, 401 JSON for /portal/api/*). Portal shell templates (base, access_required, overview stub), branded dark. Verified: cookie round-trip + tamper/garbage rejection, token resolution (valid/bad), get_current_client (valid/no-cookie/revoked) — 8/8 against a temp DB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>resolve_client_location() enforces ownership (sound location in one of the client's active projects) and 404s everything else — same response for missing and not-yours, so location existence never leaks. active_unit_for_location() resolves the currently-assigned SLM. Scoped GET /portal/api/location/{id}/live and /history: gate -> resolve unit -> read SLMM cache (never the device). /live returns a SCRUBBED projection (sound metrics + run state only; no battery/SD/raw_payload). Both degrade gracefully when there's no device or SLMM is down. Verified: ownership gate (owns / other-client / vibration / deleted-project / removed / nonexistent) + active-vs-completed unit resolution — 8/8 on a temp DB. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>/portal overview: client's active sound locations as live tiles (current Lp + Live/Stopped badge + "updated Xm ago", polled from the scoped cache every 15s) plus a Leaflet map of locations with coordinates. /portal/location/{id}: 404-gated read-only live panel — Lp/Leq/Lmax/L1/L10 cards + a 4-line Chart.js trace (backfilled from /history) + measuring/freshness badge. Cache-only, 15s poll, no device controls, no refresh-from-device. _client_locations() feeds the overview. Verified: portal.py compiles; both inline scripts balance; all four portal templates parse in Jinja2. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Add backend/routers/reports.py (registered in main.py): - GET /api/projects/{id}/reports/nightly/view — render the night report HTML inline (preview; no write, no email) - POST /api/projects/{id}/reports/nightly/run — build -> write report.html/report.json to disk -> dry-run email -> JSON result + view_url Same entry point the scheduled morning tick will reuse. Query params: night_date (default last night, local tz), baseline_start/end, metrics, send. Orchestrator now also returns the rendered html for inline display. Verified via FastAPI TestClient on real meter data (200 HTML with the computed numbers, files written to disk, 400/404 validation paths). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Adds a "View client portal" button on the project detail page that opens the client portal scoped to that project — no CLI. GET /projects/{id}/portal-preview auto-provisions a client + access token for the project (provision_preview_session) and seals a portal session cookie, then redirects to /portal. - Reuses the project's linked client if it has one; otherwise creates/reuses a per-project 'preview-<id>' client. Only sets project.client_id when unset, so it never clobbers a real client link. Idempotent — repeat clicks reuse the same client/token. - Lives under /projects (not /portal), so a future public proxy exposing only /portal/* won't expose this operator shortcut. Verified: provisioning (unlinked creates+links, idempotent, linked-no-clobber) 7/7. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>The portal location view is now genuinely live, not a 15s poll. Scoped WS endpoint /portal/api/location/{id}/stream: authenticates via the session cookie, enforces ownership (resolve_client_location), then bridges the unit's shared SLMM /monitor fan-out feed to the browser — a viewer is just one more subscriber, no extra device connection. Frames are scrubbed to the portal whitelist (drops unit_id, raw_payload, counter, lmin) before reaching the client. location.html: cache prefill for instant first paint, then upgrades to the live socket (cards tick ~1Hz, chart scrolls). Auto-close so an abandoned tab can't pin the device at 1Hz polling (~8x cellular data): - closes when the tab is hidden, reopens when visible (Page Visibility) — the main guard; - hard 15-min cap -> "Live paused — click to resume" overlay. Refactor: client_from_cookie() extracted from get_current_client so the WS handler (no Request-based Depends) can auth the same way. Verified: scrub drops internal fields / keeps metrics + heartbeat (7/7), auth refactor (3/3), portal compiles, location.html JS balances + parses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Add SoundReportConfig (one row per project) + the scheduler tick that runs the nightly report on its own: - model SoundReportConfig (enabled, report_time, metric_keys, baseline range, recipients, last_run_date) — new table, auto-created by create_all (no migration). - GET/PUT /api/projects/{id}/reports/config with validation. - SchedulerService.run_due_reports(): each loop, for every enabled config past its report_time, run last night's report once (dedup via last_run_date), writing the file + emailing (dry-run until SMTP is set). - UI: gear button beside "Night Report" opens a settings modal (enable, time, baseline range, metrics, recipients) that GET/PUTs the config. Verified: table registers + auto-creates, config CRUD + validation, tick runs/dedups, templates render and gate to sound projects. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>No-CLI way to get a real shareable magic link (/portal/enter/<token>) for a project's client. Project page gets a "Copy client link" button next to the preview; opens a modal that lists active links (with revoke), generates a fresh one, and copies it to the clipboard. Backend (operator, internal /projects/*): - POST /projects/{id}/portal-link -> mint a fresh token, return the full URL (built from request.base_url so it uses the operator's host). - GET /projects/{id}/portal-links -> list active links (label/created/last-used). - POST /projects/{id}/portal-link/{tid}/revoke -> revoke one (scoped to the project's client). Refactor: split ensure_project_client() + mint_link_token() out of provision_preview_session() so minting a shareable link and the preview cookie share one provisioning path. Verified: ensure/mint persistence across commits + sessions, minted link resolves, token stored hashed, second mint = distinct active link (4/4); compiles; share script balances; detail.html parses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Backend (reports router): - POST /reports/test-email — send a test email (body/config recipients; dry-run if SMTP unset) to verify the relay. - GET /reports/list — list generated report artifacts on disk (newest first). - GET /reports/archive/{date} — serve a saved report.html (traversal-guarded). Frontend (sound project header modals): - Night Report modal: "Run & Email" button (POST /run) + a "Recent reports" list (GET /list → opens the archived report.html in a new tab). - Settings modal: schedule + last-run status line, and a "Send test email" button. Verified: endpoints (run→list→archive, traversal blocked, test-email recipient fallback) and the template renders with all four wired + gated to sound projects. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>Adds a frictionless shareable link so anyone can open a project's client portal during dev without minting/copying a magic token. GET /portal/open/{project_id} (gated by PORTAL_OPEN_LINKS) provisions the client session and lands on /portal; lives under /portal so it works through a proxy exposing only /portal/*. The project page's "Copy client link" modal now leads with this Quick share link (amber, host taken from window.location.origin so it always matches the host you copied it from — no more LAN-vs-public foot-gun). The token-based generate/list/ revoke stays below for the eventual secure path. PORTAL_OPEN_LINKS defaults ON for the prototype (whole app is open anyway) and logs a warning; set =false before real clients. The get_current_client seam is untouched, so M4 auth still layers in front of the same routes regardless. Verified: compiles, share script balances, detail.html parses, flag default on / =false off. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Adds an "Alerts" card to /slm/{id}: lists rules and a create/edit/delete form (simple-first — "Alert when [Leq] is [above] [65] dB for [N] s", optional time-of-day window + day picker, advanced hysteresis/cooldown collapsed). Talks to the existing SLMM alert CRUD via the proxy (/api/slmm/{unit}/alerts/rules); no SLMM changes. Rule changes invalidate the evaluator's cache server-side. Verified: alerts script JS balances, slm_detail.html parses, and the TV proxy forwards method + JSON body + query params for POST/PUT/DELETE. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Internal (SLM detail page): live alarm-state badge in the Alerts header (● N active / ✓ all clear), a History list of fired events (onset → clear, peak dB, ack status) with an Ack button, refreshed every 20s. Reads the existing SLMM /alerts/events + /ack via the proxy. Portal (client, read-only, scoped): new GET /portal/api/location/{id}/events — ownership-gated, returns a scrubbed projection (rule_name/metric/threshold/onset/ peak/clear/status only; no internal ids or ack-by) plus an `active` count. The location page shows a red "Currently above threshold" banner when active and a read-only breach history, polled every 20s. No ack on the client side. Verified: portal.py compiles; both scripts balance; both templates parse. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>Baseline can now come from fixed values typed per location, not just captured data — for a spec limit ("L10 = 85") or a prior report's averages when the raw data isn't available. - SoundReportConfig.baseline_mode ("captured" | "reference"). - report_pipeline: _location_reference_baseline() reads per-location values from location_metadata; build_*_night_report honor baseline_mode (reference cells use the typed value; unset metrics compare against nothing). - reports router: GET/PUT /reports/baseline (mode on config + per-location values in location_metadata); config carries baseline_mode; manual view/run fall back to the saved config's baseline when no explicit dates are given. - orchestrator + scheduler tick thread baseline_mode through. Verified end-to-end: PUT/GET /baseline, reference deltas (L10 66.6 vs 85 -> -18.4), unset metrics compare against nothing, captured-mode regression intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>render_excel(report): one worksheet per location — interval table, a line chart, and a Last/Base/Δ summary per window. Metric-driven, so it tracks whatever metric set is configured. - orchestrator: render report.xlsx alongside report.html, attach it to the email (dry-run until SMTP set), expose xlsx_path. Never lets a spreadsheet error sink the report. - reports router: /list includes xlsx_url when present; new GET /archive/{date}/xlsx serves the saved spreadsheet. - UI: Recent-reports rows get an "Excel" download link. Verified: real Feb data -> valid .xlsx (sheet per NRL, interval table + chart + summary with real values), attachment path runs, both archive routes registered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>New scoped GET /portal/api/location/{id}/thresholds returns the enabled alert rules (scrubbed: name/metric/comparison/threshold/duration/schedule — no cooldown or hysteresis internals). Location page renders an "Alert limits" panel above the history, e.g. "Night noise · Leq above 65 dB for 60s · 22:00–07:00", hidden when no limits are set. Gives the breach history context. Verified: portal.py compiles; location script balances; template parses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>