Sound night-report pipeline (v1): automated FTP capture → ingest → morning report #66

2026-06-16T20:16:56-04:00

serversdown commented

2026-06-16 20:16:56 -04:00

What this adds

A daily morning report for 24/7 remote sound jobs (built for John Myler): each
morning, per location, email last night's noise (7 PM–7 AM) vs a baseline,
pulling the meter's own stored 15-min Leq intervals over FTP. The meter is
cycled each morning (stop → download → ingest → re-index → restart) and the
report runs off the just-finalized folder.

Highlights

Report engine — per-location LAmax/LA01/LA10/LA90/LAeq over Evening (7–10 PM)
- Nighttime (10 PM–7 AM); Leq energy-averaged, percentiles/Lmax arithmetic;
  LN→percentile map read from the device's own .rnh. Two baseline modes
  (captured = weekly average; reference = typed per-location limits).
Renderers — HTML email body + Excel attachment (interval table + chart).
Config-driven SMTP sender (REPORT_SMTP_*), dry-run when unconfigured.
Automation — per-project SoundReportConfig + a morning scheduler tick.
Capture cycle (_execute_cycle) — stop → download → ingest → restart →
DOD restart-verify, with one safe restart-retry before alerting.
Field-tested on a real NL-43 (clean cycle: 1 Leq / 0 _Lp_ / real ~26 h
duration / restart verified).
Standardized ingest — stop/download/cycle and both manual "Download & Save"
endpoints all funnel through one ingest core (Leq-only, .rnh parsed,
deduped, unit-linked).
UI — Night Report button/modal, gear Settings (enable/time/baseline/
recipients/test-email), per-NRL Data Files tab brought to parity with the
project-wide tab.
ADR 0001 — device data ownership (modules own raw data; Terra-View owns
fleet context).

Notable fixes

The NL-43 .rnh carries no measurement timestamps — session start/stop/
duration are now derived from the Leq rows. (Also fixes NL-43 dedupe.)
FTP browser script namespaced behind window.FtpBrowser — fixes a global
function collision that broke "Browse Files" on the NRL page.

Deploy notes

Rebuild required (templates + Python baked into the image); no DB
migration — sound_report_configs is a new table created on startup.
Set REPORT_SMTP_HOST/PORT/SECURITY/USER/PASSWORD/FROM/RECIPIENTS to send;
until then reports still build to data/reports/… in dry-run.
To automate a job: a "24/7 Continuous" recurring schedule (~7:15 AM) + enable
the report (~8:00 AM) + set the baseline.

Not yet / follow-ups

Report engine not yet validated end-to-end on real ingested data (capture is;
run a Night Report against a real session next); SMTP relay creds pending.
Known UI bug: the NRL "Schedule Session" button calls an undefined handler —
use the project Schedules tab for now.
Deferred review items: missed-night backfill, meter-clock vs server-TZ
windowing, a few input-guard nits.
Report-content redesign (per-15-min baseline-vs-tonight table + 2-line chart)
is parked for a later pass.

Merge note

Portal files show up in this diff only because the branch carried an older
client-portal snapshot (from an earlier dev merge); dev has since landed
Portal Auth Phase 1. Resolve those conflicts to dev's version — net portal
behavior is unchanged from dev; this PR only adds the sound report pipeline.

🤖 Generated with Claude Code

## What this adds A daily morning report for 24/7 remote sound jobs (built for John Myler): each morning, per location, email last night's noise (7 PM–7 AM) vs a baseline, pulling the meter's own stored 15-min Leq intervals over FTP. The meter is cycled each morning (stop → download → ingest → re-index → restart) and the report runs off the just-finalized folder. ## Highlights - **Report engine** — per-location LAmax/LA01/LA10/LA90/LAeq over Evening (7–10 PM) + Nighttime (10 PM–7 AM); Leq energy-averaged, percentiles/Lmax arithmetic; LN→percentile map read from the device's own `.rnh`. Two baseline modes (captured = weekly average; reference = typed per-location limits). - **Renderers** — HTML email body + Excel attachment (interval table + chart). - **Config-driven SMTP** sender (`REPORT_SMTP_*`), dry-run when unconfigured. - **Automation** — per-project `SoundReportConfig` + a morning scheduler tick. - **Capture cycle** (`_execute_cycle`) — stop → download → ingest → restart → DOD restart-verify, with **one safe restart-retry** before alerting. **Field-tested on a real NL-43** (clean cycle: 1 Leq / 0 `_Lp_` / real ~26 h duration / restart verified). - **Standardized ingest** — stop/download/cycle and both manual "Download & Save" endpoints all funnel through one ingest core (Leq-only, `.rnh` parsed, deduped, unit-linked). - **UI** — Night Report button/modal, gear Settings (enable/time/baseline/ recipients/test-email), per-NRL Data Files tab brought to parity with the project-wide tab. - **ADR 0001** — device data ownership (modules own raw data; Terra-View owns fleet context). ## Notable fixes - The NL-43 `.rnh` carries no measurement timestamps — session start/stop/ duration are now derived from the Leq rows. (Also fixes NL-43 dedupe.) - FTP browser script namespaced behind `window.FtpBrowser` — fixes a global function collision that broke "Browse Files" on the NRL page. ## Deploy notes - **Rebuild required** (templates + Python baked into the image); **no DB migration** — `sound_report_configs` is a new table created on startup. - Set `REPORT_SMTP_HOST/PORT/SECURITY/USER/PASSWORD/FROM/RECIPIENTS` to send; until then reports still build to `data/reports/…` in dry-run. - To automate a job: a "24/7 Continuous" recurring schedule (~7:15 AM) + enable the report (~8:00 AM) + set the baseline. ## Not yet / follow-ups - Report engine not yet validated end-to-end on real ingested data (capture is; run a Night Report against a real session next); SMTP relay creds pending. - Known UI bug: the NRL "Schedule Session" button calls an undefined handler — use the project Schedules tab for now. - Deferred review items: missed-night backfill, meter-clock vs server-TZ windowing, a few input-guard nits. - Report-content redesign (per-15-min baseline-vs-tonight table + 2-line chart) is parked for a later pass. ## Merge note Portal files show up in this diff only because the branch carried an older client-portal snapshot (from an earlier `dev` merge); `dev` has since landed Portal Auth Phase 1. **Resolve those conflicts to `dev`'s version** — net portal behavior is unchanged from `dev`; this PR only adds the sound report pipeline. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

serversdown added 10 commits 2026-06-16 20:16:57 -04:00

Merge branch 'dev' into feat/ftp-report-pipeline 5c38f6ec1e

refactor(reports): funnel scheduler stop/download/cycle through one ingest abcfba179f

_execute_stop and _execute_download no longer hand-roll ZIP extraction; all three actions now call a shared _ingest_and_link helper (ingest via ingest_nrl_zip, link the unit, drop the empty placeholder session). Every capture path produces the same clean, .rnh-parsed, percentile-aware, deduped, Leq-only session. _execute_download previously created no session at all (TODO); it now ingests like the others.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(reports): derive session recording window from the Leq rows 2ecf1f54d5

The NL-43 .rnh carries no measurement timestamps, so _ingest_file_entries was stamping every session with utcnow() and no duration. Derive started_at/stopped_at/duration from the Leq .rnd 'Start Time' column when the header lacks them (interval from the .rnh, else inferred from row spacing). Adds an optional unit_id so callers that know the recording unit attribute the session at creation, and returns duration_seconds.

Side effect: NL-43 dedupe now works (it keyed on a previously-empty start_time_str). Affects all ingest paths: manual upload, FTP cycle, stop, download, and manual FTP download.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(reports): manual FTP "Download & Save" saves a parsed session 7716a4b51d

ftp-download-folder-to-server and ftp-download-to-server now route NRL data through the shared ingest (ingest_nrl_zip / _ingest_file_entries) instead of hand-rolling DataFile rows on a now/zero-duration session. Folder save requires the unit be assigned to a location; non-NRL single files keep the generic save path. The FTP browser popup now reports how long the measurement ran.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(reports): per-NRL Data Files tab reaches parity with the project-wide tab aa21c81c2e

The per-NRL Data Files tab now reuses the same FTP browser + unified-files partials as the project-wide tab, scoped to the one NRL: ftp-browser and files-unified take an optional location_id. nrl_detail.html drops the flat file_list view for 'Download Files from SLMs' (Browse Files -> Download & Save) plus the grouped 'Project Files' view (edit times / download-all / delete), keeping the NRL upload and adding a refresh button.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

fix(reports): resolve loadFTPFiles collision breaking Browse Files on the NRL tab 1801d4eb74

ftp_browser.html and slm_live_view.html are both loaded on the per-NRL detail page (Data Files + Command Center tabs) and each defined loadFTPFiles / downloadToServer / downloadFTPFile / enableFTP / formatFileSize as globals — last to load won. 'Browse Files' then called slm_live_view's loadFTPFiles, which renders into the hidden Command Center's #ftp-files-list, so the FTP request fired but nothing appeared. Prefix ftp_browser's five colliding functions with fb* so each partial keeps its own. (Element IDs don't collide: per-unit vs fixed.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

refactor(reports): namespace the FTP browser partial behind window.FtpBrowser 694980c869

Proper fix superseding the fb* prefix band-aid (1801d4e): wrap ftp_browser.html's script in an IIFE and expose only window.FtpBrowser. Its ~11 helpers no longer leak to global scope, so the partial is safe to co-load with other FTP-browsing partials (e.g. slm_live_view's Command Center) without name collisions in either direction. Inline onclick handlers call FtpBrowser.*; showFTPSettings stays global (it's from the included settings modal). Behaviour unchanged — verified full Jinja render + balanced delimiters.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs(adr): establish device data ownership principle (ADR 0001) 76f16a2aba

Modules own raw device data; Terra-View owns fleet/project/session/report context. Documents the SFM (read-through) vs SLMM (Terra-View-stored) asymmetry, the rule new modules must follow, and grandfathers SLMM as a deliberate-future-realignment exception. Establishes the docs/adr/ convention.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(reports): one safe restart-retry in the cycle before alerting 3c5e830f9c

If the post-restart DOD check shows the meter isn't measuring, retry once with start_recording (a plain start that does NOT re-index, unlike start_cycle) and re-verify before raising the schedule-failed alert. Retry fires only on a confident not-measuring reading — never on a failed/inconclusive DOD read — so a flaky read can't disrupt an already-running measurement or split the night across two store folders. Turns a transient restart hiccup into a self-heal instead of a meter left stopped overnight.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Merge branch 'dev' into feat/ftp-report-pipeline b0e5dcdc52

serversdown merged commit 0beba9bfbc into dev

2026-06-16 20:17:10 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: serversdown/terra-view#66