Merge pull request '0.14.0 update from dev - client portal, SLMM expansion, multistream support.' (#67 ) from dev into main

Reviewed-on: #67
docs: consolidate changelog into 0.14.0 + refresh README
2026-06-17 16:41:10 -04:00 · 2026-06-17 01:29:32 +00:00 · 2026-06-05 02:39:05 -04:00 · 2026-06-01 17:41:18 -04:00 · 2026-05-20 11:44:47 -04:00 · 2026-05-17 19:44:56 -04:00
3 changed files with 65 additions and 8 deletions
@@ -7,9 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

 ## [Unreleased]

-SLM live monitoring — fan-out feed + cache-first reads.  Targets **0.14.0**.  The throughline: the NL-43 allows exactly **one** TCP connection at a time, so every page that opened its own device stream (or sent its own `Measure?`/DOD on load) was competing for that single connection — a second viewer saw nothing, and dashboard loads stole polling resolution from the live feed.  This release moves Terra-View entirely onto SLMM's shared, cached monitoring: one DOD poll loop per device, fanned out to all viewers; dashboards read SLMM's cache (a DB read on SLMM's side) instead of touching the device; and the live panels populate instantly from cache on open, upgrading to the live WS only on demand.  Paired with the SLMM-side work (adaptive poll rate, unreachable backoff, device-offline alert) on SLMM branch `dev`.
+## [0.14.0] - 2026-06-17

-### Added
+Rounds out **sound monitoring** and adds a **client-facing portal**, consolidating four threads since 0.13.x: SLM live monitoring (now on SLMM's shared, cached feed), an automated **FTP night-report pipeline**, a read-only **client portal**, and **per-project password auth** for it. Depends on the matching **SLMM `dev`** build — see Upgrade Notes at the end of each section.
+
+### SLM live monitoring — fan-out feed + cache-first
+
+SLM live monitoring — fan-out feed + cache-first reads.  The throughline: the NL-43 allows exactly **one** TCP connection at a time, so every page that opened its own device stream (or sent its own `Measure?`/DOD on load) was competing for that single connection — a second viewer saw nothing, and dashboard loads stole polling resolution from the live feed.  This release moves Terra-View entirely onto SLMM's shared, cached monitoring: one DOD poll loop per device, fanned out to all viewers; dashboards read SLMM's cache (a DB read on SLMM's side) instead of touching the device; and the live panels populate instantly from cache on open, upgrading to the live WS only on demand.  Paired with the SLMM-side work (adaptive poll rate, unreachable backoff, device-offline alert) on SLMM branch `dev`.
+
+#### Added

 - **Fan-out `/monitor` feed consumption.**  The unit live view (`partials/slm_live_view.html`) and the dashboard live tile (`sound_level_meters.html`) now subscribe to SLMM's shared per-device monitor over `WS /api/slmm/{unit}/monitor` instead of each opening its own device stream.  Any number of clients attach without each consuming the NL-43's single connection — the "second viewer sees nothing" contention is gone.  A WS proxy handler for `/monitor` was added to `backend/routers/slmm.py`.
 - **L1/L10 percentile lines + cards.**  Both the per-unit live chart and the dashboard card chart now plot L1 (purple) and L10 (orange) alongside Lp/Leq, and the KPI cards show L1/L10.  Sourced from the DOD feed's `ln1`/`ln2` (DRD streaming can't carry percentiles, DOD can).  Missing/`-.-` values leave a gap rather than dropping the line to 0.
@@ -18,20 +24,20 @@ SLM live monitoring — fan-out feed + cache-first reads.  Targets **0.14.0**.
 - **Refresh buttons** — one per device-list row, one in the panel header.  On-demand, user-initiated single device read via `GET /api/slmm/{unit}/live` (which also refreshes SLMM's cache), with a spinner + success/error toast, then reloads the device list.
 - **Per-unit live-monitoring (keepalive) toggle on `/admin/slmm`** — turns a device's server-side keepalive feed on/off (`POST /monitor/start|stop`), so alerting can keep a device's feed running with no browser attached.

-### Changed
+#### Changed

 - **Dashboard device list + command center read SLMM's cache, not the device.**  `slm_dashboard.py`'s `get_slm_units` pulls each unit's cached status from SLMM's `/roster` (one call, a SLMM DB read) for the badge + freshness; the command-center `get_live_view` reads cached `/status` instead of sending `Measure?` + a fresh DOD on every load.  This stops dashboard loads from stealing the device's single connection from the live monitor.  The elapsed-measurement timer still works because `measurement_start_time` is now included in the cached `/status` response.
 - **Device-list freshness reflects real monitoring.**  The "Last check" line now uses SLMM's cached `last_seen` (which the monitor advances on every successful poll) via `unit.cache_last_seen`, instead of the `slm_last_check` roster field the monitor never updates.  The status badge also treats `Measure` as Measuring, matching the panel and SLMM's cache.
 - **Status badge relocated** to the card's bottom meta row (next to "Last check"), off the top-right corner where it collided with the chart/gear/refresh action icons.

-### Fixed
+#### Fixed

 - **Deploy/bench threw `can't access property "dispatchEvent", e is null`.**  `toggleSLMDeployed()` and the save-config path called `htmx.trigger('#slm-list', 'load')` guarded only by `typeof htmx !== 'undefined'`; no page has a `#slm-list`, so htmx resolved null and called `null.dispatchEvent(...)`.  The deploy POST had already succeeded, so the operator saw both the green success **and** a red error.  Both call sites now guard on the element existing (`slm_settings_modal.html`).
 - **Monitor WS proxy leaked `CancelledError` / "task exception never retrieved"** on stream stop — the cleanup awaited pending tasks but only caught `Exception`, missing `CancelledError` (a `BaseException`).
 - **"No recent check-in" shown even on an actively-monitored device** — the row read the stale `slm_last_check` roster field instead of SLMM's live cache (see Changed).
 - **L1/L10 KPI cards populated but the chart drew no L1/L10 lines** — the card chart only had Lp + Leq datasets.

-### Upgrade Notes
+#### Upgrade Notes

 Requires the **matching SLMM build (branch `dev`)** — Terra-View now depends on SLMM's fan-out `/monitor` feed, `/history` trail, `/status` carrying `ln1`/`ln2` + `measurement_start_time`, cached `/roster` status, and the `monitor_enabled` keepalive flag.

@@ -49,6 +55,54 @@ The two builds must ship **together**.  Note the `docker-compose.yml` container

 ---

+### FTP night-report pipeline *(new)*
+
+Automated daily morning report of last night's noise (7 PM–7 AM) vs a baseline,
+per location, for 24/7 remote sound jobs.  The meter records to its SD card
+regardless of TCP state, so the report pulls the meter's own stored 15-minute
+Leq intervals over FTP (via the SLMM proxy) — accurate, and resilient to a
+control-path wedge.  **Field-tested on a real NL-43.**
+
+#### Added
+
+- **Report engine.**  Per-location LAmax / LA01 / LA10 / LA90 / LAeq over Evening
+  (7–10 PM) + Nighttime (10 PM–7 AM); Leq energy-averaged, percentiles/Lmax
+  arithmetic; the LN→percentile map is read from the device's own `.rnh`.  Two
+  baseline modes: *captured* (weekly average) and *reference* (typed per-location
+  limits).
+- **Renderers.**  HTML email body + Excel attachment (per-NRL interval table +
+  line chart + Last/Base/Δ summary).
+- **Capture cycle.**  The daily scheduled "24/7 Continuous" cycle stops →
+  downloads → ingests → re-indexes → restarts the meter, verifies it resumed
+  measuring via a fresh DOD read, and retries the restart once before alerting.
+- **Standardized ingest.**  Manual SD upload, manual FTP "Download & Save", and
+  the scheduled cycle all funnel through one ingest core: keeps the `.rnh` +
+  15-minute Leq, drops the 1-second `_Lp_` files, parses the header, dedupes, and
+  derives the session's real recording window from the Leq rows.
+- **UI.**  Night Report button/modal (view / run-and-email / recent reports) and a
+  per-project Settings panel (enable, time, baseline, recipients, test-email); the
+  per-NRL Data Files tab now matches the project-wide tab.
+- **Config-driven SMTP** sender (`REPORT_SMTP_*`), dry-run when unconfigured.
+
+#### Fixed
+
+- **NL-43 sessions stamped `now` / zero-duration.**  The NL-43 `.rnh` carries no
+  measurement timestamps, so the session window is now derived from the Leq rows.
+  Also fixes NL-43 dedupe (it had keyed on an always-empty start time).
+- **"Browse Files" did nothing on the NRL Data Files tab** — the FTP-browser
+  script's global functions collided with the SLM live-view's (both loaded on that
+  page); it's now namespaced behind `window.FtpBrowser`.
+
+#### Upgrade Notes
+
+- **No DB migration** — the `sound_report_configs` table auto-creates on startup.
+- Set `REPORT_SMTP_HOST/PORT/SECURITY/USER/PASSWORD/FROM/RECIPIENTS` to send email
+  (reports build to `data/reports/…` in dry-run until then).
+- To automate a job: a **"24/7 Continuous"** recurring schedule (~7:15 AM) + enable
+  the report (~8:00 AM) + set a baseline.
+
+---
+
 ### Client portal *(new — read-only client-facing view)*

 A scoped, read-only portal at **`/portal/*`** where a client sees only *their*
@@ -1,5 +1,5 @@
-# Terra-View v0.13.3
-Backend API and HTMX-powered web interface for managing a mixed fleet of seismographs and field modems. Track deployments, monitor health in real time, merge roster intent with incoming telemetry, and control your fleet through a unified database and dashboard.
+# Terra-View v0.14.0
+Backend API and HTMX-powered web interface for managing a mixed fleet of seismographs, sound level meters, and field modems. Track deployments, monitor health in real time, merge roster intent with incoming telemetry, and control your fleet through a unified database and dashboard.

 ## Features

@@ -18,6 +18,9 @@ Backend API and HTMX-powered web interface for managing a mixed fleet of seismog
 - **Settings & Safeguards**: `/settings` page exposes roster stats, exports, replace-all imports, and danger-zone reset tools
 - **Device & Modem Metadata**: Capture calibration windows, modem pairings, phone/IP details, and addresses per unit
 - **Status Management**: Automatically mark deployed units as OK, Pending (>12h), or Missing (>24h) based on recent telemetry
+- **Sound Level Meter Monitoring**: Live per-device monitoring through SLMM's shared, cached feed — multiple viewers without contending for the NL-43's single connection — with L1/L10 percentile lines, a measuring/freshness indicator, and on-demand refresh
+- **Automated Night Reports**: Daily per-location noise report (last night vs a baseline) for 24/7 remote sound jobs — pulls the meter's 15-minute Leq over FTP and emails an HTML summary + Excel; the meter is auto-cycled (stop → download → ingest → restart, with restart verification) each morning
+- **Client Portal** (`/portal/*`): scoped, read-only, client-facing live view of *their* locations only, gated by a per-project link + shared password (argon2-hashed)
 - **SFM Event DB Manager** (`/admin/events`): cross-unit event browser with bulk false-trigger flagging and admin-only hard-delete (cleans on-disk binaries + sidecars too) for purging bogus events from misbehaving units
 - **Deployment-History Calendar + Gantt** (`/tools/deployment-history`): fleet-wide 12-month calendar with side-panel day drill-down, plus "Gantt by Project" / "Gantt by Unit" tabs
 - **Photo Management**: Upload and view photos for each unit
@@ -30,7 +30,7 @@ Base.metadata.create_all(bind=engine)
 ENVIRONMENT = os.getenv("ENVIRONMENT", "production")

 # Initialize FastAPI app
-VERSION = "0.13.3"
+VERSION = "0.14.0"
 if ENVIRONMENT == "development":
    _build = os.getenv("BUILD_NUMBER", "0")
    if _build and _build != "0":
Author	SHA1	Message	Date
serversdown	3f0b53c46c	Merge pull request '0.14.0 update from dev - client portal, SLMM expansion, multistream support.' (#67 ) from dev into main Reviewed-on: #67	2026-06-17 16:41:10 -04:00
serversdown	abdb3869bd	docs: consolidate changelog into 0.14.0 + refresh README Cut [0.14.0] consolidating SLM live monitoring, the FTP night-report pipeline (was missing from the changelog entirely), the client portal, and portal auth Phase 1 under one entry. Bump VERSION + README to 0.14.0 and add the sound-monitoring / night-report / client-portal features to the README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-17 01:29:32 +00:00
serversdown	ca3035f50a	Merge pull request 'update to 0.13.3' (#57 ) from dev into main ## [0.13.3] - 2026-06-05 Calibration sync from SFM events. Closes the manual data-entry loop on calibration dates — Terra-View now pulls `device.calibration_date` from each seismograph's most recent event sidecar once a day and updates `RosterUnit.last_calibrated` when the device reports something fresher than what's stored. Manual edits still win when they're newer than the latest event; a fresh event arriving later supersedes the manual edit. Adds a "Sync now" button under Settings → Advanced → Calibration Defaults for on-demand runs, and a `docs/ROADMAP.md` to track in-flight + deferred work. ### Added - Calibration sync service (`backend/services/calibration_sync.py`). Per-unit: fetches `/db/events?serial={id}&limit=1` then `/db/events/{event_id}/sidecar` via the SFM proxy, reads `device.calibration_date`, and writes it to `RosterUnit.last_calibrated` with `next_calibration_due` recomputed from `UserPreferences.calibration_interval_days`. Every change is logged in `UnitHistory` with `source='sfm_event'` and `notes="Synced from event {id}"` so the unit detail history timeline reflects auto-sync activity alongside manual edits. - Conflict rule: events-as-truth, manual wins when newer. Three outcomes per unit: - `already_in_sync` — stored date already matches the event's calibration date. - `skipped_manual_newer` — the latest `UnitHistory` change for `last_calibrated` happened after the event's timestamp, so the manual edit is preserved. Only a future event can supersede it. - `updated` — the event is newer (or no manual edit exists), so the stored date is replaced. - Daily background job at 03:15 local via the `schedule` library + a worker thread (modeled on `backup_scheduler.py`). Started in `main.py`'s startup hook, stopped on shutdown. Does not run on boot — first sync after a server start fires at the next 03:15. - `POST /api/calibration/sync` — runs a full sync immediately and returns a summary `{checked, updated, skipped_manual_newer, already_in_sync, no_event, no_sidecar, no_cal_in_sidecar, errors, results: [...]}`. Powers the Settings button. - `GET /api/calibration/sync/status` — returns scheduler state + the last run's summary including per-unit `{unit_id, action, old, new, event_id}` rows. Useful for diagnostics: `curl localhost:8001/api/calibration/sync/status \| jq`. - Settings UI: "Sync from SFM events" section under the Calibration Defaults card (Advanced tab). Click "Sync now" → result line shows counts: `Checked N · Updated N · Already in sync N · Manual kept N · No event N`. - `docs/ROADMAP.md` — first-pass roadmap pulling deferred items from `CLAUDE.md`'s focus block, in-code TODOs (`photos.py` GPS migration → `MonitoringLocation`, `device_controller.py` SFM Phase 2 stubs, `modem_dashboard.py` ModemManager backend, `dashboard.html` geocoding), and the README's long-standing "Future Enhancements" wishlist. Grouped into In Flight / Near-Term / Medium-Term / Wishlist; intended as a living document. ### Fixed - Prod startup crash: `ModuleNotFoundError: No module named 'schedule'`. The `schedule` library wasn't pinned in `requirements.txt` even though `backend/services/backup_scheduler.py` has been using it since v0.4.x — the dev image happened to have it from an earlier manual `pip install`, but a clean prod rebuild dropped it. Added `schedule==1.2.2` so the new calibration scheduler (and the existing backup scheduler) survive a clean rebuild. ### Upgrade Notes No DB migration required — `UnitHistory.source` and `RosterUnit.last_calibrated`/`next_calibration_due` already exist. Rebuild only: ```bash cd /home/serversdown/terra-view docker compose build terra-view && docker compose up -d terra-view ``` After rebuild, Settings → Advanced → "Sync from SFM events" → "Sync now" to backfill in one shot; otherwise wait for the 03:15 job. ---	2026-06-05 02:39:05 -04:00
serversdown	0e2086d6bb	Merge pull request 'v0.13.2 - s4 event pipeline complete' (#56 ) from dev into main Reviewed-on: #56	2026-06-01 17:41:18 -04:00
serversdown	d0685baed5	Merge pull request 'v0.12.1 — Unit Swap wizard, editable timeline, roster/tz fixes' (#54 ) from dev into main Reviewed-on: #54 docs+chore: v0.12.1 — Unit Swap wizard, editable timeline, roster/tz fixes CHANGELOG entry for the five commits that landed after the v0.12.0 tag: two features (Unit Swap wizard at /tools/unit-swap, editable deployment timeline on /unit/{id}) and two correctness fixes (RosterUnit.deployed now flips on swap/unassign/promote; deployment timeline now respects user timezone for both display and edits). No schema migrations. README bumped to v0.12.1 with new bullets for the post-v0.12.0 features and several already-shipped items that were missing from the list (SFM Event DB Manager, Deployment-History calendar + Gantt tabs, reusable location-map partial). backend/main.py VERSION constant bumped too.	2026-05-20 11:44:47 -04:00
serversdown	275a168046	Merge pull request 'merge v0.12.0' (#51 ) from dev into main Reviewed-on: #51	2026-05-17 19:44:56 -04:00
serversdown	f4fd1c943d	Merge pull request 'v0.11.0' (#50 ) from release/0.11.0 into main ## [0.11.0] - 2026-05-15 Operator-facing polish release. All work builds on the v0.10.0 SFM integration foundation — this release is about making the day-to-day workflows (managing locations, cleaning up bad attributions, browsing deployments) faster and less error-prone. ### Added - Soft-remove monitoring locations (`POST /api/projects/{p}/locations/{l}/remove` + `/restore`): mark a location as no longer actively monitored without destroying historical events. Cascade-closes active unit assignments and cancels pending scheduled actions at the location. Restored locations rejoin the active list (assignments are NOT auto-reopened — operator creates new ones if resuming). Project page splits locations into Active and Removed sections; removed cards are greyed out, badged with the removal date + reason, and offer a Restore button. - Per-unit deployment Gantt chart above the existing Deployment Timeline list on every seismograph unit detail page. Plain-SVG rendering, color per location, today marker (orange dashed line), reduced-opacity bars for closed assignments, blue outlines on metadata-backfilled assignments, dashed blue underlines marking mergeable groups. Click a bar to scroll the matching list row into view with a flash highlight. - Merge consecutive same-location assignments (`POST /api/projects/{p}/assignments/merge`): operators often end up with several rows representing one continuous deployment (after remove/restore, or metadata-backfill adjacent to a manual record). Now auto-detected and surfaceable in the timeline header — one click combines them into a single record. Preserves the earliest record's notes + ingest source, writes an `assignment_merged` audit entry, deletes the others. - Delete assignment for mis-clicks (`DELETE /api/projects/{p}/assignments/{a}`): hard-deletes a bogus assignment row that was never a real deployment. Trash icon in each row of the location's Deployment History panel. Refuses the delete if any `MonitoringSession` exists in the assignment's window — those should go through Unassign instead, which preserves audit history. Writes an `assignment_deleted` UnitHistory row. - Drag-to-reorder location cards: each active card has a six-dot drag handle on the left. Drag/drop reorders the DOM and persists via `POST /api/projects/{p}/locations/reorder`. Implementation uses native HTML5 drag-and-drop (no library). New locations land at the end (`sort_order = max + 1`); removed locations stay sorted by removal date. - Three-dot kebab menu on location cards: replaces the four inline pill buttons (Unassign / Edit / Remove / Delete) with a single ⋮ menu. Click ⋮ to open; click outside or Escape to close; only one menu open at a time. - Event count on vibration location cards: vibration cards now show "{N} events" sourced from SFM via concurrent fan-out, instead of "Sessions: 0" (sessions don't exist under the watcher-forward pipeline). Sound locations still show session counts. - Project overview location map: right column of every project's overview replaces the lightly-used Upcoming Actions panel with a Leaflet map. One pin per active monitoring location (parsed from the `coordinates` field). Click pin → scrolls + flashes the matching card. Tooltip on hover. Locations without coordinates surface as an inline hint below the map. If the project has pending scheduled actions, a small "{N} upcoming actions →" link appears in the card header that switches to the Schedules tab. ### Changed - Backfill location fuzzy matcher is now stricter: `rapidfuzz.WRatio` was over-confident on location names because their shared boilerplate vocabulary ("Area", "Loc", numbers) inflated scores. Example false positive that prompted the change: `"Area 2 - Brookville Dam - Loc 2 East"` vs `"Area 1 - Loc 1 - 87 Jenks"` scored 86% via WRatio. Now uses `token_set_ratio` as the base scorer plus a 0.30 penalty when the two strings have disjoint multi-digit numeric tokens. Catches the "same project, different address number" case (`"68 Jenks"` vs `"87 Jenks"`) that pure token-set scoring still rated above 0.90. Project matching keeps WRatio (where its leniency is desirable for typos like `1-80` vs `I-80`). ### Fixed - Three separate JSON.stringify quote-collision bugs: any inline `onclick="...({...} \| tojson)"` or `onclick="...${JSON.stringify(x)}..."` where `x` contained any character that JSON quotes (essentially every real-world string) broke the HTML attribute and silently un-bound the click handler. Surfaced in three places this release; all fixed by switching to `data-` attributes plus a trampoline function reading from `this.dataset`: - Location Remove button* on the project page - Metadata-backfill typeahead dropdown (existing project + location pickers) - Project-merge typeahead dropdown (in the per-project header) - Project-merge modal too short to show typeahead options without scrolling: modal body's `flex-1 overflow-y-auto` collapsed tight; added `min-height: 480px` to the modal container + `min-h-[320px]` to the body so the dropdown always has room. - Project location map covered modals: Leaflet's internal panes carry z-indexes 200–800 by default and the map container didn't establish a stacking context, so those z-indexes leaked into the root and outranked modals' `z-50`. Fixed by adding `isolation: isolate` to the map container. - `delete_assignment` crashed with `AttributeError`: the safety check queried `MonitoringSession.start_time` but the actual column is `started_at`. Every DELETE call to `/assignments/{id}` failed with 500 before doing anything. ### Migration Notes Run on each database before deploying. Both migrations are idempotent and non-destructive. ```bash docker exec terra-view-terra-view-1 python3 /app/backend/migrate_add_location_removed.py docker exec terra-view-terra-view-1 python3 /app/backend/migrate_add_location_sort_order.py ``` Or sweep all migrations at once (safe — already-applied ones no-op): ```bash for f in backend/migrate_.py; do docker exec terra-view-terra-view-1 python3 "/app/backend/$(basename $f)" done ``` New columns added this release: - `monitoring_locations.removed_at` (DATETIME, nullable) — NULL means active - `monitoring_locations.removal_reason` (TEXT, nullable) - `monitoring_locations.sort_order` (INTEGER, default 0) — seeded to alphabetical-index per project on first migration Deploy order matters*: migrations must run BEFORE the new code is up, otherwise the running app will throw 500s on the unrecognized columns. Idempotent migrations make this recoverable but it's better avoided — the v0.11.0 deploy on prod hit this exact window after the v0.10.0 release. ---	2026-05-15 19:16:42 -04:00