From 576e4f89ca7d9b8cf254089f70140f020c66b1ad Mon Sep 17 00:00:00 2001 From: serversdown Date: Fri, 12 Jun 2026 03:26:30 +0000 Subject: [PATCH] doc: changelog entry for reports --- report-changelog.md | 54 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 report-changelog.md diff --git a/report-changelog.md b/report-changelog.md new file mode 100644 index 0000000..df0d666 --- /dev/null +++ b/report-changelog.md @@ -0,0 +1,54 @@ +# FTP Night-Report Pipeline — changelog entry + +> **How to use:** paste the block below into Terra-View's `CHANGELOG.md`. +> The current `[Unreleased]` section targets **0.14.0** (SLM live monitoring); this +> is a separate, larger feature, so it's drafted here as **0.15.0** — fold it into +> `[Unreleased]` or bump the version as you prefer. Set the release date when you ship. + +--- + +## [0.15.0] - 2026-XX-XX + +FTP night-report pipeline. Automated **daily morning report** of last night's noise (7PM–7AM) versus a baseline, per location, for 24/7 remote sound jobs. The meter records 24/7 to its SD card regardless of TCP state, so the report pulls the meter's own stored 15-minute `_Leq_` intervals over FTP (through the existing `/api/slmm/.../ftp/...` proxy) — accurate Leq/Lmax/Ln straight from the device, and resilient to a TCP-control wedge. The report engine is source-agnostic and metric-driven; delivery is an HTML email body plus an Excel attachment. Built around the existing `MonitoringSession`/`DataFile` store and the existing scheduled `cycle` action — the meter is cycled each morning (stop → download → ingest → increment store index → restart), and the report runs off the just-finished, finalized folder. + +### Added + +- **Callable ingest — `ingest_nrl_zip(location_id, zip_bytes, db)`** (`backend/routers/project_locations.py`). The manual SD-card upload (`upload_nrl_data`) was refactored into a shared core so the same path runs programmatically from the scheduler. Keeps `.rnh` + the averaged `_Leq_ .rnd`, drops the 1-second `_Lp_` files, parses the header (now also capturing the device's **percentile→slot map** and weightings into session metadata), and **dedups** repeated pulls of the same folder by store-name + start time. Metric-agnostic: every column of the Leq file is preserved on disk; metric selection happens in the report layer. +- **Report compute engine** (`backend/services/report_pipeline.py`). Per-location night model: **LAmax / LA01 / LA10 / LA90 / LAeq** over **Evening (7–10PM)** and **Nighttime (10PM–7AM)** windows, with correct aggregation — Lmax = loudest interval, percentiles = arithmetic mean, **Leq = logarithmic (energy) mean**. The LN→percentile mapping is read from the device's own `.rnh` config, not hardcoded. +- **Two baseline sources.** *Captured* — computed from recorded nights in a configurable date range (the "typical night" = mean of per-night values). *Reference* — fixed values typed per location, for a spec limit (e.g. *"L10 = 85"*) or a prior report's averages when the raw data isn't in the system. Blank reference cells aren't compared. +- **Renderers** (`backend/services/report_renderers.py`). HTML email body (per-location Last / Baseline / Δ table, colored louder/quieter) **+ an Excel attachment** — one worksheet per NRL with the 15-minute interval table, a line chart, and a Last/Base/Δ summary per window. Metric-driven, so it tracks whatever metric set is configured. +- **Config-driven SMTP sender** (`backend/services/report_email.py`). Reads host/port/security/user/password/from/recipients from env (`REPORT_SMTP_*`); **dry-run** when unconfigured, so reports still generate and persist without credentials. +- **Per-project config + automatic morning run.** New `SoundReportConfig` table (enabled, report time, metrics, baseline mode + range, recipients) and a scheduler tick (`SchedulerService.run_due_reports`) that builds + emails each enabled project's report once per morning, off the event loop. The orchestrator (`report_orchestrator.py`) always writes `report.html` / `report.json` / `report.xlsx` to `data/reports/{project}/{date}/`, then emails. +- **Capture hook in the daily cycle.** `_execute_cycle` now ingests the just-finished `Auto_####` folder into Terra-View after the download, and verifies the meter resumed measuring via a fresh DOD (`measurement_state`) — alerting if not. +- **UI on the sound project header.** A **Night Report** button (modal: view a night, *Run & Email* on demand, and a *Recent reports* list with HTML + Excel links) and a **gear → Settings** modal (enable/time, **baseline source toggle** with a per-NRL value editor, metrics, recipients, a **Send test email** button, and a schedule/last-run status line). +- **Endpoints** (`backend/routers/reports.py`): `GET/PUT …/reports/config`, `GET/PUT …/reports/baseline`, `GET …/reports/nightly/view`, `POST …/reports/nightly/run`, `POST …/reports/test-email`, `GET …/reports/list`, `GET …/reports/archive/{date}` (+ `/xlsx`). + +### Changed + +- **Manual SD upload now shares the new ingest core.** `POST …/nrl/{location_id}/upload-data` behaves as before (zip or loose files) but routes through `_ingest_file_entries`, so manually-uploaded sessions also get the captured percentile map. + +### Security / hardening + +- HTML modal fields built from user-controlled data (location names, baseline values) are HTML-escaped before insertion (stored-XSS fix). +- The SMTP sender refuses to silently downgrade to a plaintext connection on an unrecognized `REPORT_SMTP_SECURITY` value (falls back to STARTTLS), and warns when credentials would go over an unencrypted link. + +### Upgrade Notes + +- **No database migration.** `sound_report_configs` is a brand-new table created automatically by `create_all` on startup (the `baseline_mode` column lives on it). Templates and Python are baked into the image, so **rebuild** (don't just restart): + + ```bash + cd /home/serversdown/terra-view && docker compose build terra-view && docker compose up -d terra-view + ``` + +- **To actually send email**, set the relay env vars (e.g. on the `terra-view` service in `docker-compose.yml`). Until then, reports still build and write to `data/reports/…` in dry-run: + + ``` + REPORT_SMTP_HOST, REPORT_SMTP_PORT, REPORT_SMTP_SECURITY=starttls|ssl|none, + REPORT_SMTP_USER, REPORT_SMTP_PASSWORD, REPORT_SMTP_FROM, REPORT_SMTP_RECIPIENTS + ``` + + Use **Settings → Send test email** to verify the relay once set. + +- **To turn on automation for a job:** configure a daily `cycle` recurring schedule per NRL (~7:15 AM, after the night ends) so the meter is stopped/downloaded/ingested/restarted, then **enable** the report in the gear (report time ~8 AM) and set the baseline (range or fixed values). + +- **Not yet field-tested on a physical meter** — the live device-control portion of the cycle hook (download + restart-verify) was validated against a mocked SLMM only.