# FTP Night-Report Pipeline — changelog entry

> **How to use:** paste the block below into Terra-View's `CHANGELOG.md`.
> The current `[Unreleased]` section targets **0.14.0** (SLM live monitoring); this
> is a separate, larger feature, so it's drafted here as **0.15.0** — fold it into
> `[Unreleased]` or bump the version as you prefer. Set the release date when you ship.

---

## [0.15.0] - 2026-XX-XX

FTP night-report pipeline.  Automated **daily morning report** of last night's noise (7PM–7AM) versus a baseline, per location, for 24/7 remote sound jobs.  The meter records 24/7 to its SD card regardless of TCP state, so the report pulls the meter's own stored 15-minute `_Leq_` intervals over FTP (through the existing `/api/slmm/.../ftp/...` proxy) — accurate Leq/Lmax/Ln straight from the device, and resilient to a TCP-control wedge.  The report engine is source-agnostic and metric-driven; delivery is an HTML email body plus an Excel attachment.  Built around the existing `MonitoringSession`/`DataFile` store and the existing scheduled `cycle` action — the meter is cycled each morning (stop → download → ingest → increment store index → restart), and the report runs off the just-finished, finalized folder.

### Added

- **Callable ingest — `ingest_nrl_zip(location_id, zip_bytes, db)`** (`backend/routers/project_locations.py`).  The manual SD-card upload (`upload_nrl_data`) was refactored into a shared core so the same path runs programmatically from the scheduler.  Keeps `.rnh` + the averaged `_Leq_ .rnd`, drops the 1-second `_Lp_` files, parses the header (now also capturing the device's **percentile→slot map** and weightings into session metadata), and **dedups** repeated pulls of the same folder by store-name + start time.  Metric-agnostic: every column of the Leq file is preserved on disk; metric selection happens in the report layer.
- **Report compute engine** (`backend/services/report_pipeline.py`).  Per-location night model: **LAmax / LA01 / LA10 / LA90 / LAeq** over **Evening (7–10PM)** and **Nighttime (10PM–7AM)** windows, with correct aggregation — Lmax = loudest interval, percentiles = arithmetic mean, **Leq = logarithmic (energy) mean**.  The LN→percentile mapping is read from the device's own `.rnh` config, not hardcoded.
- **Two baseline sources.**  *Captured* — computed from recorded nights in a configurable date range (the "typical night" = mean of per-night values).  *Reference* — fixed values typed per location, for a spec limit (e.g. *"L10 = 85"*) or a prior report's averages when the raw data isn't in the system.  Blank reference cells aren't compared.
- **Renderers** (`backend/services/report_renderers.py`).  HTML email body (per-location Last / Baseline / Δ table, colored louder/quieter) **+ an Excel attachment** — one worksheet per NRL with the 15-minute interval table, a line chart, and a Last/Base/Δ summary per window.  Metric-driven, so it tracks whatever metric set is configured.
- **Config-driven SMTP sender** (`backend/services/report_email.py`).  Reads host/port/security/user/password/from/recipients from env (`REPORT_SMTP_*`); **dry-run** when unconfigured, so reports still generate and persist without credentials.
- **Per-project config + automatic morning run.**  New `SoundReportConfig` table (enabled, report time, metrics, baseline mode + range, recipients) and a scheduler tick (`SchedulerService.run_due_reports`) that builds + emails each enabled project's report once per morning, off the event loop.  The orchestrator (`report_orchestrator.py`) always writes `report.html` / `report.json` / `report.xlsx` to `data/reports/{project}/{date}/`, then emails.
- **Capture hook in the daily cycle.**  `_execute_cycle` now ingests the just-finished `Auto_####` folder into Terra-View after the download, and verifies the meter resumed measuring via a fresh DOD (`measurement_state`) — alerting if not.
- **UI on the sound project header.**  A **Night Report** button (modal: view a night, *Run & Email* on demand, and a *Recent reports* list with HTML + Excel links) and a **gear → Settings** modal (enable/time, **baseline source toggle** with a per-NRL value editor, metrics, recipients, a **Send test email** button, and a schedule/last-run status line).
- **Endpoints** (`backend/routers/reports.py`): `GET/PUT …/reports/config`, `GET/PUT …/reports/baseline`, `GET …/reports/nightly/view`, `POST …/reports/nightly/run`, `POST …/reports/test-email`, `GET …/reports/list`, `GET …/reports/archive/{date}` (+ `/xlsx`).

### Changed

- **Manual SD upload now shares the new ingest core.**  `POST …/nrl/{location_id}/upload-data` behaves as before (zip or loose files) but routes through `_ingest_file_entries`, so manually-uploaded sessions also get the captured percentile map.

### Security / hardening

- HTML modal fields built from user-controlled data (location names, baseline values) are HTML-escaped before insertion (stored-XSS fix).
- The SMTP sender refuses to silently downgrade to a plaintext connection on an unrecognized `REPORT_SMTP_SECURITY` value (falls back to STARTTLS), and warns when credentials would go over an unencrypted link.

### Upgrade Notes

- **No database migration.**  `sound_report_configs` is a brand-new table created automatically by `create_all` on startup (the `baseline_mode` column lives on it).  Templates and Python are baked into the image, so **rebuild** (don't just restart):

  ```bash
  cd /home/serversdown/terra-view && docker compose build terra-view && docker compose up -d terra-view
  ```

- **To actually send email**, set the relay env vars (e.g. on the `terra-view` service in `docker-compose.yml`).  Until then, reports still build and write to `data/reports/…` in dry-run:

  ```
  REPORT_SMTP_HOST, REPORT_SMTP_PORT, REPORT_SMTP_SECURITY=starttls|ssl|none,
  REPORT_SMTP_USER, REPORT_SMTP_PASSWORD, REPORT_SMTP_FROM, REPORT_SMTP_RECIPIENTS
  ```

  Use **Settings → Send test email** to verify the relay once set.

- **To turn on automation for a job:** configure a daily `cycle` recurring schedule per NRL (~7:15 AM, after the night ends) so the meter is stopped/downloaded/ingested/restarted, then **enable** the report in the gear (report time ~8 AM) and set the baseline (range or fixed values).

- **Not yet field-tested on a physical meter** — the live device-control portion of the cycle hook (download + restart-verify) was validated against a mocked SLMM only.
