55 lines
6.3 KiB
Markdown
55 lines
6.3 KiB
Markdown
# FTP Night-Report Pipeline — changelog entry
|
||
|
||
> **How to use:** paste the block below into Terra-View's `CHANGELOG.md`.
|
||
> The current `[Unreleased]` section targets **0.14.0** (SLM live monitoring); this
|
||
> is a separate, larger feature, so it's drafted here as **0.15.0** — fold it into
|
||
> `[Unreleased]` or bump the version as you prefer. Set the release date when you ship.
|
||
|
||
---
|
||
|
||
## [0.15.0] - 2026-XX-XX
|
||
|
||
FTP night-report pipeline. Automated **daily morning report** of last night's noise (7PM–7AM) versus a baseline, per location, for 24/7 remote sound jobs. The meter records 24/7 to its SD card regardless of TCP state, so the report pulls the meter's own stored 15-minute `_Leq_` intervals over FTP (through the existing `/api/slmm/.../ftp/...` proxy) — accurate Leq/Lmax/Ln straight from the device, and resilient to a TCP-control wedge. The report engine is source-agnostic and metric-driven; delivery is an HTML email body plus an Excel attachment. Built around the existing `MonitoringSession`/`DataFile` store and the existing scheduled `cycle` action — the meter is cycled each morning (stop → download → ingest → increment store index → restart), and the report runs off the just-finished, finalized folder.
|
||
|
||
### Added
|
||
|
||
- **Callable ingest — `ingest_nrl_zip(location_id, zip_bytes, db)`** (`backend/routers/project_locations.py`). The manual SD-card upload (`upload_nrl_data`) was refactored into a shared core so the same path runs programmatically from the scheduler. Keeps `.rnh` + the averaged `_Leq_ .rnd`, drops the 1-second `_Lp_` files, parses the header (now also capturing the device's **percentile→slot map** and weightings into session metadata), and **dedups** repeated pulls of the same folder by store-name + start time. Metric-agnostic: every column of the Leq file is preserved on disk; metric selection happens in the report layer.
|
||
- **Report compute engine** (`backend/services/report_pipeline.py`). Per-location night model: **LAmax / LA01 / LA10 / LA90 / LAeq** over **Evening (7–10PM)** and **Nighttime (10PM–7AM)** windows, with correct aggregation — Lmax = loudest interval, percentiles = arithmetic mean, **Leq = logarithmic (energy) mean**. The LN→percentile mapping is read from the device's own `.rnh` config, not hardcoded.
|
||
- **Two baseline sources.** *Captured* — computed from recorded nights in a configurable date range (the "typical night" = mean of per-night values). *Reference* — fixed values typed per location, for a spec limit (e.g. *"L10 = 85"*) or a prior report's averages when the raw data isn't in the system. Blank reference cells aren't compared.
|
||
- **Renderers** (`backend/services/report_renderers.py`). HTML email body (per-location Last / Baseline / Δ table, colored louder/quieter) **+ an Excel attachment** — one worksheet per NRL with the 15-minute interval table, a line chart, and a Last/Base/Δ summary per window. Metric-driven, so it tracks whatever metric set is configured.
|
||
- **Config-driven SMTP sender** (`backend/services/report_email.py`). Reads host/port/security/user/password/from/recipients from env (`REPORT_SMTP_*`); **dry-run** when unconfigured, so reports still generate and persist without credentials.
|
||
- **Per-project config + automatic morning run.** New `SoundReportConfig` table (enabled, report time, metrics, baseline mode + range, recipients) and a scheduler tick (`SchedulerService.run_due_reports`) that builds + emails each enabled project's report once per morning, off the event loop. The orchestrator (`report_orchestrator.py`) always writes `report.html` / `report.json` / `report.xlsx` to `data/reports/{project}/{date}/`, then emails.
|
||
- **Capture hook in the daily cycle.** `_execute_cycle` now ingests the just-finished `Auto_####` folder into Terra-View after the download, and verifies the meter resumed measuring via a fresh DOD (`measurement_state`) — alerting if not.
|
||
- **UI on the sound project header.** A **Night Report** button (modal: view a night, *Run & Email* on demand, and a *Recent reports* list with HTML + Excel links) and a **gear → Settings** modal (enable/time, **baseline source toggle** with a per-NRL value editor, metrics, recipients, a **Send test email** button, and a schedule/last-run status line).
|
||
- **Endpoints** (`backend/routers/reports.py`): `GET/PUT …/reports/config`, `GET/PUT …/reports/baseline`, `GET …/reports/nightly/view`, `POST …/reports/nightly/run`, `POST …/reports/test-email`, `GET …/reports/list`, `GET …/reports/archive/{date}` (+ `/xlsx`).
|
||
|
||
### Changed
|
||
|
||
- **Manual SD upload now shares the new ingest core.** `POST …/nrl/{location_id}/upload-data` behaves as before (zip or loose files) but routes through `_ingest_file_entries`, so manually-uploaded sessions also get the captured percentile map.
|
||
|
||
### Security / hardening
|
||
|
||
- HTML modal fields built from user-controlled data (location names, baseline values) are HTML-escaped before insertion (stored-XSS fix).
|
||
- The SMTP sender refuses to silently downgrade to a plaintext connection on an unrecognized `REPORT_SMTP_SECURITY` value (falls back to STARTTLS), and warns when credentials would go over an unencrypted link.
|
||
|
||
### Upgrade Notes
|
||
|
||
- **No database migration.** `sound_report_configs` is a brand-new table created automatically by `create_all` on startup (the `baseline_mode` column lives on it). Templates and Python are baked into the image, so **rebuild** (don't just restart):
|
||
|
||
```bash
|
||
cd /home/serversdown/terra-view && docker compose build terra-view && docker compose up -d terra-view
|
||
```
|
||
|
||
- **To actually send email**, set the relay env vars (e.g. on the `terra-view` service in `docker-compose.yml`). Until then, reports still build and write to `data/reports/…` in dry-run:
|
||
|
||
```
|
||
REPORT_SMTP_HOST, REPORT_SMTP_PORT, REPORT_SMTP_SECURITY=starttls|ssl|none,
|
||
REPORT_SMTP_USER, REPORT_SMTP_PASSWORD, REPORT_SMTP_FROM, REPORT_SMTP_RECIPIENTS
|
||
```
|
||
|
||
Use **Settings → Send test email** to verify the relay once set.
|
||
|
||
- **To turn on automation for a job:** configure a daily `cycle` recurring schedule per NRL (~7:15 AM, after the night ends) so the meter is stopped/downloaded/ingested/restarted, then **enable** the report in the gear (report time ~8 AM) and set the baseline (range or fixed values).
|
||
|
||
- **Not yet field-tested on a physical meter** — the live device-control portion of the cycle hook (download + restart-verify) was validated against a mocked SLMM only.
|