Adds operator_gate Starlette HTTP middleware that gates every route
except an explicit allow-list. Flag defaults OFF so all existing
behaviour and tests are unchanged. wire_operator_auth helper in
conftest lets tests monkeypatch the module-global SessionLocal and
flag, keeping the gate's own DB session pointed at the test engine.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add OperatorUser SQLAlchemy model (operator_users table, auto-created by
create_all) with email uniqueness, default active/must_change_password/
failed_login_count, and sessions_valid_from truncated to whole seconds.
Add backend/operator_auth.py with feature flag, cookie constants, _ROLE_RANK
map, role_at_least(), and _norm_email() helpers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
If the post-restart DOD check shows the meter isn't measuring, retry once with start_recording (a plain start that does NOT re-index, unlike start_cycle) and re-verify before raising the schedule-failed alert. Retry fires only on a confident not-measuring reading — never on a failed/inconclusive DOD read — so a flaky read can't disrupt an already-running measurement or split the night across two store folders. Turns a transient restart hiccup into a self-heal instead of a meter left stopped overnight.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- delete dead magic-link helpers (resolve_token, ensure_project_client,
mint_link_token, provision_preview_session) + now-unused datetime import
- key brute-force lockout on link_token alone (IP term only enabled a
source-IP-rotation bypass; behind the proxy all clients share one IP)
- drop unused PORTAL_BASE_URL from the retired CLI
- add WebSocket ownership tests (unauth + cross-project both close 1008)
Caught by adversarial review of the scope test: portal_client_for_project minted a
dedicated client but never set project.client_id, so the client-scoped routes found
no projects — every location 404'd, including the client's own (empty portal). Now
links the project + adds a positive-case test.
The per-NRL Data Files tab now reuses the same FTP browser + unified-files partials as the project-wide tab, scoped to the one NRL: ftp-browser and files-unified take an optional location_id. nrl_detail.html drops the flat file_list view for 'Download Files from SLMs' (Browse Files -> Download & Save) plus the grouped 'Project Files' view (edit times / download-all / delete), keeping the NRL upload and adding a refresh button.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ftp-download-folder-to-server and ftp-download-to-server now route NRL data through the shared ingest (ingest_nrl_zip / _ingest_file_entries) instead of hand-rolling DataFile rows on a now/zero-duration session. Folder save requires the unit be assigned to a location; non-NRL single files keep the generic save path. The FTP browser popup now reports how long the measurement ran.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The NL-43 .rnh carries no measurement timestamps, so _ingest_file_entries was stamping every session with utcnow() and no duration. Derive started_at/stopped_at/duration from the Leq .rnd 'Start Time' column when the header lacks them (interval from the .rnh, else inferred from row spacing). Adds an optional unit_id so callers that know the recording unit attribute the session at creation, and returns duration_seconds.
Side effect: NL-43 dedupe now works (it keyed on a previously-empty start_time_str). Affects all ingest paths: manual upload, FTP cycle, stop, download, and manual FTP download.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_execute_stop and _execute_download no longer hand-roll ZIP extraction; all three actions now call a shared _ingest_and_link helper (ingest via ingest_nrl_zip, link the unit, drop the empty placeholder session). Every capture path produces the same clean, .rnh-parsed, percentile-aware, deduped, Leq-only session. _execute_download previously created no session at all (TODO); it now ingests like the others.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- #1 XSS: escape user-controlled values (location name, baseline values, recent-
report fields, SMTP status message) in the modals via the existing _mergeEsc
helper — they were concatenated raw into innerHTML (stored XSS via location name).
- #2 SMTP: an unrecognized REPORT_SMTP_SECURITY no longer silently downgrades to a
plaintext connection while still calling login() — it falls back to starttls and
warns; warn on intentional security=none + auth.
- #3 scheduler: run the (blocking smtplib + Excel) nightly report in a worker thread
(asyncio.to_thread + its own DB session) so it can't stall the loop that drives
time-sensitive device cycles. New _run_one_report helper.
- #4 cycle ingest: set unit_id on the ingested data session (ingest_nrl_zip leaves
it None) before dropping the empty placeholder, preserving the unit<->session link;
repoint old_session_id at the real row.
- #7 robustness: wrap send_report_email in the orchestrator and run_nightly_report in
/view + /run so a render/SMTP error returns a clean error instead of a raw 500
after artifacts are written.
Verified: SMTP paths (typo->starttls, none, starttls, ssl), off-thread tick stamps
last_run_date + writes the file, /view 200, escaping wired, app imports.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- PORTAL_OPEN_LINKS now defaults OFF — /portal/open/* is an unauthenticated,
proxy-reachable session-minting path (and a linked project's open link grants
the whole client's scope), so it must be explicitly enabled in dev.
- Session cookie: enforce server-side expiry (check iat vs COOKIE_MAX_AGE — was
browser-only) and guard a non-dict signed body (was an uncaught AttributeError →
500, reachable if SECRET_KEY is the insecure default).
- Escape operator-set strings (location/rule/event names) before innerHTML +
Leaflet tooltips — they're client-facing, so a name with markup was stored XSS
in the client's browser. Global esc() helper applied at every injection point.
- WS _scrub_frame drops a non-JSON frame instead of forwarding it raw; /history
rows now whitelisted like the other scoped endpoints.
- Preview-client slug uses the full project id (an 8-char prefix could collide
two projects onto one client).
Verified: cookie reader (fresh/expired/non-dict/missing-iat) + open-links default
off; templates parse; scoped scrubbing intact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New scoped GET /portal/api/location/{id}/thresholds returns the enabled alert
rules (scrubbed: name/metric/comparison/threshold/duration/schedule — no cooldown
or hysteresis internals). Location page renders an "Alert limits" panel above the
history, e.g. "Night noise · Leq above 65 dB for 60s · 22:00–07:00", hidden when
no limits are set. Gives the breach history context.
Verified: portal.py compiles; location script balances; template parses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
render_excel(report): one worksheet per location — interval table, a line chart,
and a Last/Base/Δ summary per window. Metric-driven, so it tracks whatever metric
set is configured.
- orchestrator: render report.xlsx alongside report.html, attach it to the email
(dry-run until SMTP set), expose xlsx_path. Never lets a spreadsheet error sink
the report.
- reports router: /list includes xlsx_url when present; new
GET /archive/{date}/xlsx serves the saved spreadsheet.
- UI: Recent-reports rows get an "Excel" download link.
Verified: real Feb data -> valid .xlsx (sheet per NRL, interval table + chart +
summary with real values), attachment path runs, both archive routes registered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend _execute_cycle (daily stop/download/increment/restart) so the nightly
report's data lands automatically:
- Step 4b: after the device download, fetch the just-finished Auto_#### folder
from SLMM and ingest via ingest_nrl_zip (clean session + DataFiles, Lp filtered,
dedup). Drops the empty "recording" placeholder session once the real data
session exists. New helper _ingest_cycle_folder.
- Step 6b: after restart, verify the meter resumed measuring via a fresh DOD
(measurement_state) — advisory: alerts loudly on failure but doesn't fail the
cycle (keepalive polling re-confirms within ~10s).
Both wrapped defensively so they never break the cycle. Ingest-hook logic verified
with a mocked SLMM (real Feb folder -> session + 2 DataFiles, dedup, empty/HTTP
guards). Device-control paths (restart-verify, live download) are field-untested
— no meter available in dev.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Baseline can now come from fixed values typed per location, not just captured
data — for a spec limit ("L10 = 85") or a prior report's averages when the raw
data isn't available.
- SoundReportConfig.baseline_mode ("captured" | "reference").
- report_pipeline: _location_reference_baseline() reads per-location values from
location_metadata; build_*_night_report honor baseline_mode (reference cells
use the typed value; unset metrics compare against nothing).
- reports router: GET/PUT /reports/baseline (mode on config + per-location values
in location_metadata); config carries baseline_mode; manual view/run fall back
to the saved config's baseline when no explicit dates are given.
- orchestrator + scheduler tick thread baseline_mode through.
Verified end-to-end: PUT/GET /baseline, reference deltas (L10 66.6 vs 85 -> -18.4),
unset metrics compare against nothing, captured-mode regression intact.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Internal (SLM detail page): live alarm-state badge in the Alerts header
(● N active / ✓ all clear), a History list of fired events (onset → clear, peak
dB, ack status) with an Ack button, refreshed every 20s. Reads the existing SLMM
/alerts/events + /ack via the proxy.
Portal (client, read-only, scoped): new GET /portal/api/location/{id}/events —
ownership-gated, returns a scrubbed projection (rule_name/metric/threshold/onset/
peak/clear/status only; no internal ids or ack-by) plus an `active` count. The
location page shows a red "Currently above threshold" banner when active and a
read-only breach history, polled every 20s. No ack on the client side.
Verified: portal.py compiles; both scripts balance; both templates parse.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a frictionless shareable link so anyone can open a project's client portal
during dev without minting/copying a magic token. GET /portal/open/{project_id}
(gated by PORTAL_OPEN_LINKS) provisions the client session and lands on /portal;
lives under /portal so it works through a proxy exposing only /portal/*.
The project page's "Copy client link" modal now leads with this Quick share link
(amber, host taken from window.location.origin so it always matches the host you
copied it from — no more LAN-vs-public foot-gun). The token-based generate/list/
revoke stays below for the eventual secure path.
PORTAL_OPEN_LINKS defaults ON for the prototype (whole app is open anyway) and logs
a warning; set =false before real clients. The get_current_client seam is
untouched, so M4 auth still layers in front of the same routes regardless.
Verified: compiles, share script balances, detail.html parses, flag default
on / =false off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Backend (reports router):
- POST /reports/test-email — send a test email (body/config recipients; dry-run
if SMTP unset) to verify the relay.
- GET /reports/list — list generated report artifacts on disk (newest first).
- GET /reports/archive/{date} — serve a saved report.html (traversal-guarded).
Frontend (sound project header modals):
- Night Report modal: "Run & Email" button (POST /run) + a "Recent reports" list
(GET /list → opens the archived report.html in a new tab).
- Settings modal: schedule + last-run status line, and a "Send test email" button.
Verified: endpoints (run→list→archive, traversal blocked, test-email recipient
fallback) and the template renders with all four wired + gated to sound projects.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
No-CLI way to get a real shareable magic link (/portal/enter/<token>) for a
project's client. Project page gets a "Copy client link" button next to the
preview; opens a modal that lists active links (with revoke), generates a fresh
one, and copies it to the clipboard.
Backend (operator, internal /projects/*):
- POST /projects/{id}/portal-link -> mint a fresh token, return the full URL
(built from request.base_url so it uses the operator's host).
- GET /projects/{id}/portal-links -> list active links (label/created/last-used).
- POST /projects/{id}/portal-link/{tid}/revoke -> revoke one (scoped to the
project's client).
Refactor: split ensure_project_client() + mint_link_token() out of
provision_preview_session() so minting a shareable link and the preview cookie
share one provisioning path.
Verified: ensure/mint persistence across commits + sessions, minted link resolves,
token stored hashed, second mint = distinct active link (4/4); compiles; share
script balances; detail.html parses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add SoundReportConfig (one row per project) + the scheduler tick that runs the
nightly report on its own:
- model SoundReportConfig (enabled, report_time, metric_keys, baseline range,
recipients, last_run_date) — new table, auto-created by create_all (no migration).
- GET/PUT /api/projects/{id}/reports/config with validation.
- SchedulerService.run_due_reports(): each loop, for every enabled config past
its report_time, run last night's report once (dedup via last_run_date),
writing the file + emailing (dry-run until SMTP is set).
- UI: gear button beside "Night Report" opens a settings modal (enable, time,
baseline range, metrics, recipients) that GET/PUTs the config.
Verified: table registers + auto-creates, config CRUD + validation, tick
runs/dedups, templates render and gate to sound projects.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The portal location view is now genuinely live, not a 15s poll. Scoped WS endpoint
/portal/api/location/{id}/stream: authenticates via the session cookie, enforces
ownership (resolve_client_location), then bridges the unit's shared SLMM /monitor
fan-out feed to the browser — a viewer is just one more subscriber, no extra
device connection. Frames are scrubbed to the portal whitelist (drops unit_id,
raw_payload, counter, lmin) before reaching the client.
location.html: cache prefill for instant first paint, then upgrades to the live
socket (cards tick ~1Hz, chart scrolls). Auto-close so an abandoned tab can't pin
the device at 1Hz polling (~8x cellular data):
- closes when the tab is hidden, reopens when visible (Page Visibility) — the main
guard;
- hard 15-min cap -> "Live paused — click to resume" overlay.
Refactor: client_from_cookie() extracted from get_current_client so the WS handler
(no Request-based Depends) can auth the same way.
Verified: scrub drops internal fields / keeps metrics + heartbeat (7/7), auth
refactor (3/3), portal compiles, location.html JS balances + parses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a "View client portal" button on the project detail page that opens the
client portal scoped to that project — no CLI. GET /projects/{id}/portal-preview
auto-provisions a client + access token for the project (provision_preview_session)
and seals a portal session cookie, then redirects to /portal.
- Reuses the project's linked client if it has one; otherwise creates/reuses a
per-project 'preview-<id>' client. Only sets project.client_id when unset, so it
never clobbers a real client link. Idempotent — repeat clicks reuse the same
client/token.
- Lives under /projects (not /portal), so a future public proxy exposing only
/portal/* won't expose this operator shortcut.
Verified: provisioning (unlinked creates+links, idempotent, linked-no-clobber) 7/7.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`python3 backend/portal_admin.py` set sys.path[0] to backend/, hiding the
`backend` package and breaking `from backend.database import ...`. Insert the
project root on sys.path so the documented script invocation works.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add backend/routers/reports.py (registered in main.py):
- GET /api/projects/{id}/reports/nightly/view — render the night report
HTML inline (preview; no write, no email)
- POST /api/projects/{id}/reports/nightly/run — build -> write
report.html/report.json to disk -> dry-run email -> JSON result + view_url
Same entry point the scheduled morning tick will reuse. Query params:
night_date (default last night, local tz), baseline_start/end, metrics, send.
Orchestrator now also returns the rendered html for inline display.
Verified via FastAPI TestClient on real meter data (200 HTML with the computed
numbers, files written to disk, 400/404 validation paths).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
backend/portal_admin.py (run in-container): create-client, link-project (by id/
number/name -> sets Project.client_id), mint-link (prints the full magic URL once,
stores only the hash), list, revoke. PORTAL_BASE_URL controls the printed link base.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
/portal overview: client's active sound locations as live tiles (current Lp +
Live/Stopped badge + "updated Xm ago", polled from the scoped cache every 15s)
plus a Leaflet map of locations with coordinates. /portal/location/{id}: 404-gated
read-only live panel — Lp/Leq/Lmax/L1/L10 cards + a 4-line Chart.js trace
(backfilled from /history) + measuring/freshness badge. Cache-only, 15s poll, no
device controls, no refresh-from-device. _client_locations() feeds the overview.
Verified: portal.py compiles; both inline scripts balance; all four portal
templates parse in Jinja2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
resolve_client_location() enforces ownership (sound location in one of the
client's active projects) and 404s everything else — same response for missing
and not-yours, so location existence never leaks. active_unit_for_location()
resolves the currently-assigned SLM.
Scoped GET /portal/api/location/{id}/live and /history: gate -> resolve unit ->
read SLMM cache (never the device). /live returns a SCRUBBED projection (sound
metrics + run state only; no battery/SD/raw_payload). Both degrade gracefully
when there's no device or SLMM is down.
Verified: ownership gate (owns / other-client / vibration / deleted-project /
removed / nonexistent) + active-vs-completed unit resolution — 8/8 on a temp DB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
backend/portal_auth.py: stdlib HMAC-signed session cookie carrying the access-
token id (re-validated against the DB each request, so revoke kills live
sessions), hash_token, resolve_token, and the get_current_client dependency
(raises PortalAuthError). SECRET_KEY env (insecure dev default + warning).
routers/portal.py: /portal/enter/{token} mints the cookie -> /portal; /logout;
/access; /portal home stub. main.py registers the router + a PortalAuthError
handler (HTML access page for pages, 401 JSON for /portal/api/*).
Portal shell templates (base, access_required, overview stub), branded dark.
Verified: cookie round-trip + tamper/garbage rejection, token resolution
(valid/bad), get_current_client (valid/no-cookie/revoked) — 8/8 against a temp DB.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Client (customer org), ClientAccessToken (interim hashed magic-URL gate), and an
authoritative Project.client_id FK (client_name kept for display). New tables
auto-create via create_all; migrate_add_client_portal.py adds projects.client_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Terra-View side of the daily night-vs-baseline sound report for the John Myler
24/7 job. Engine is built and verified end-to-end against real meter data;
SMTP send + scheduler/capture wiring still pending.
- ingest: refactor upload_nrl_data into a callable ingest_nrl_zip(location_id,
zip_bytes, db) sharing one core with the HTTP endpoint. Capture the .rnh
percentile map + weightings into session metadata; dedup on store-name +
start time. Ingest stays metric-agnostic (every Leq column preserved).
- report_pipeline.py: metric registry, Evening/Nighttime windows, correct
aggregation (Lmax=max, Ln=arithmetic, Leq=logarithmic), baseline = typical
night, per-location + per-project builders.
- report_renderers.py: HTML email-body renderer (Last/Base/delta layout).
- report_email.py: config-driven SMTP via stdlib (env vars) with a dry-run
fallback so the pipeline runs without credentials.
- report_orchestrator.py: compute -> render -> always write report.html +
report.json to disk -> best-effort email.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
serversdown