FTP Night-Report Pipeline — changelog entry

How to use: paste the block below into Terra-View's CHANGELOG.md. The current [Unreleased] section targets 0.14.0 (SLM live monitoring); this is a separate, larger feature, so it's drafted here as 0.15.0 — fold it into [Unreleased] or bump the version as you prefer. Set the release date when you ship.

[0.15.0] - 2026-XX-XX

FTP night-report pipeline. Automated daily morning report of last night's noise (7PM–7AM) versus a baseline, per location, for 24/7 remote sound jobs. The meter records 24/7 to its SD card regardless of TCP state, so the report pulls the meter's own stored 15-minute _Leq_ intervals over FTP (through the existing /api/slmm/.../ftp/... proxy) — accurate Leq/Lmax/Ln straight from the device, and resilient to a TCP-control wedge. The report engine is source-agnostic and metric-driven; delivery is an HTML email body plus an Excel attachment. Built around the existing MonitoringSession/DataFile store and the existing scheduled cycle action — the meter is cycled each morning (stop → download → ingest → increment store index → restart), and the report runs off the just-finished, finalized folder.

Added

Callable ingest — ingest_nrl_zip(location_id, zip_bytes, db) (backend/routers/project_locations.py). The manual SD-card upload (upload_nrl_data) was refactored into a shared core so the same path runs programmatically from the scheduler. Keeps .rnh + the averaged _Leq_ .rnd, drops the 1-second _Lp_ files, parses the header (now also capturing the device's percentile→slot map and weightings into session metadata), and dedups repeated pulls of the same folder by store-name + start time. Metric-agnostic: every column of the Leq file is preserved on disk; metric selection happens in the report layer.
Report compute engine (backend/services/report_pipeline.py). Per-location night model: LAmax / LA01 / LA10 / LA90 / LAeq over Evening (7–10PM) and Nighttime (10PM–7AM) windows, with correct aggregation — Lmax = loudest interval, percentiles = arithmetic mean, Leq = logarithmic (energy) mean. The LN→percentile mapping is read from the device's own .rnh config, not hardcoded.
Two baseline sources. Captured — computed from recorded nights in a configurable date range (the "typical night" = mean of per-night values). Reference — fixed values typed per location, for a spec limit (e.g. "L10 = 85") or a prior report's averages when the raw data isn't in the system. Blank reference cells aren't compared.
Renderers (backend/services/report_renderers.py). HTML email body (per-location Last / Baseline / Δ table, colored louder/quieter) + an Excel attachment — one worksheet per NRL with the 15-minute interval table, a line chart, and a Last/Base/Δ summary per window. Metric-driven, so it tracks whatever metric set is configured.
Config-driven SMTP sender (backend/services/report_email.py). Reads host/port/security/user/password/from/recipients from env (REPORT_SMTP_*); dry-run when unconfigured, so reports still generate and persist without credentials.
Per-project config + automatic morning run. New SoundReportConfig table (enabled, report time, metrics, baseline mode + range, recipients) and a scheduler tick (SchedulerService.run_due_reports) that builds + emails each enabled project's report once per morning, off the event loop. The orchestrator (report_orchestrator.py) always writes report.html / report.json / report.xlsx to data/reports/{project}/{date}/, then emails.
Capture hook in the daily cycle. _execute_cycle now ingests the just-finished Auto_#### folder into Terra-View after the download, and verifies the meter resumed measuring via a fresh DOD (measurement_state) — alerting if not.
UI on the sound project header. A Night Report button (modal: view a night, Run & Email on demand, and a Recent reports list with HTML + Excel links) and a gear → Settings modal (enable/time, baseline source toggle with a per-NRL value editor, metrics, recipients, a Send test email button, and a schedule/last-run status line).
Endpoints (backend/routers/reports.py): GET/PUT …/reports/config, GET/PUT …/reports/baseline, GET …/reports/nightly/view, POST …/reports/nightly/run, POST …/reports/test-email, GET …/reports/list, GET …/reports/archive/{date} (+ /xlsx).

Changed

Manual SD upload now shares the new ingest core. POST …/nrl/{location_id}/upload-data behaves as before (zip or loose files) but routes through _ingest_file_entries, so manually-uploaded sessions also get the captured percentile map.

Security / hardening

HTML modal fields built from user-controlled data (location names, baseline values) are HTML-escaped before insertion (stored-XSS fix).
The SMTP sender refuses to silently downgrade to a plaintext connection on an unrecognized REPORT_SMTP_SECURITY value (falls back to STARTTLS), and warns when credentials would go over an unencrypted link.

Upgrade Notes

No database migration. sound_report_configs is a brand-new table created automatically by create_all on startup (the baseline_mode column lives on it). Templates and Python are baked into the image, so rebuild (don't just restart):
```
cd /home/serversdown/terra-view && docker compose build terra-view && docker compose up -d terra-view
```
To actually send email, set the relay env vars (e.g. on the terra-view service in docker-compose.yml). Until then, reports still build and write to data/reports/… in dry-run:
```
REPORT_SMTP_HOST, REPORT_SMTP_PORT, REPORT_SMTP_SECURITY=starttls|ssl|none,
REPORT_SMTP_USER, REPORT_SMTP_PASSWORD, REPORT_SMTP_FROM, REPORT_SMTP_RECIPIENTS
```
Use Settings → Send test email to verify the relay once set.
To turn on automation for a job: configure a daily cycle recurring schedule per NRL (~7:15 AM, after the night ends) so the meter is stopped/downloaded/ingested/restarted, then enable the report in the gear (report time ~8 AM) and set the baseline (range or fixed values).
Not yet field-tested on a physical meter — the live device-control portion of the cycle hook (download + restart-verify) was validated against a mocked SLMM only.

6.3 KiB Raw Permalink Blame History Unescape Escape