Phase 5b first slice. Surfaces near-duplicate projects (typo variants,
abbreviation differences, spacing variations like "SR81" vs "SR 81")
as side-by-side pairs the operator can merge with one click.
Backend (backend/services/project_tidy.py):
- find_duplicate_pairs(db, threshold=0.85) walks all active projects and
computes rapidfuzz.WRatio similarity for every pair. Pre-filters
too-short normalised names (< 4 chars) to avoid noise. Skips
soft-deleted projects. Returns pairs sorted by score desc, then by
total content (more assignments → review first).
- Each pair carries a suggested merge target with a human-readable
reason. Priorities (in order): manual source over parser source,
populated project_number, more locations, more assignments, shorter
name. Operator can override the suggestion by clicking the OTHER
direction button.
- O(N^2) over project count. Fine up to ~500 projects. Token-prefix
blocking is the obvious next optimisation if it becomes slow.
Backend (backend/routers/projects.py):
- GET /api/projects/admin/duplicate_pairs?threshold=&max_pairs= returns
pairs as JSON for the Tidy page.
Frontend (templates/admin/project_tidy.html):
- New admin page at /settings/developer/project-tidy. Threshold selector
(95% / 90% / 85% / 80%) at the top; rescan button next to it; auto-
scans on load.
- Each pair card shows side-by-side project summaries (name, project_
number, client, source-badge, location/assignment counts) with the
suggested target visually highlighted (orange border). Three buttons:
"Merge A → B", "Merge B → A", "Not a dup" (hide locally).
- Click-to-merge opens a native confirm with the preview totals
(assignments/sessions/data files moving, consolidations) — same data
the project_header.html merge modal shows. On confirm, hits the
existing /merge_into endpoint and re-scans automatically.
- Source badges distinguish parser-created (`metadata_backfill`) from
manual projects — at a glance the operator can see "this duplicate is
parser-generated; safe to merge into the manual one".
Frontend (templates/admin/metadata_backfill.html):
- Apply-result handling now surfaces failed[] cluster reasons in a
dedicated failure panel (bottom-left, dismissable). Previously a 200
OK with all-failures showed a misleading "1 cluster applied" success
toast because the count and the failure list weren't being reconciled.
This bit us during the DB-revert recovery earlier — the
project_modules table was missing, every apply silently rolled back,
user saw success toasts. Fixed.
Smoke-verified against current state (10K events, 9 projects, post-
merge): tool correctly finds 0 pairs at threshold 0.85 (data is clean),
1 false-positive at 0.70 (two unrelated projects sharing the token "81"
— example of why the 0.85 default is correct).
Settings link added under Developer → Project Tidy.
Phase 5c (swap-detection daily background job + notification inbox)
remains deferred to the next session.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Operator-facing tool for cleaning up duplicate projects. Common after
the metadata-backfill parser auto-creates near-duplicates from operator
name variations ("SR81" vs "SR 81", "Swank-Karns Crossing" vs
"Swank-Karns Crossings", "Trumbull-Bryman Mont.Dam" vs
"Trumbull-Brayman-Mont Dam", etc.).
Workflow: visit the duplicate project's detail page, click "Merge into…"
in the header, search for the canonical target project from a typeahead,
review the preview (what assignments / locations / sessions will move,
any conflicts), confirm. Source is soft-deleted; everything else
re-points to the target. Smart consolidation: same-named locations in
both projects merge into one (source's assignments move to target's
existing location with the same name; source's empty location is then
deleted). Different-named locations move as-is.
Backend:
- backend/services/project_merge.py (new): preview() and execute()
functions. Transaction-safe. Per-assignment UnitHistory audit row
with change_type='assignment_merged' so the deployment timeline shows
the merge. Source modules disabled; missing modules added to target.
Handles edge cases: same project_id rejected, deleted projects rejected,
orphan project-direct assignments (no location) re-pointed defensively.
- backend/routers/projects.py: new endpoints
GET /api/projects/{source_id}/merge_preview?target_id=...
POST /api/projects/{source_id}/merge_into?target_id=...
Frontend (templates/partials/projects/project_header.html):
- "Merge into…" button in Project Actions area.
- Modal with typeahead (reuses /api/admin/metadata_backfill/projects_search)
scoped to existing projects only (no create-new option). Filters out
the source project from candidates so operator can't accidentally pick
it as target.
- Preview pane shows totals + per-location plan (consolidate vs move) +
warnings (mismatched client names, location consolidation note).
- Red "Merge (permanent)" confirm button only enables after a target is
picked and preview loads.
- On success, browser redirects to target project page.
Smoke verified: "Swank-Karns Crossing" (1 assignment) merged into
"Swank-Karns Crossings"; target now has 2 locations + 2 assignments,
source has 0 dangling rows, 1 project_merge audit entry written.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Introduced a new section for displaying soft-deleted projects.
- Implemented loading of deleted projects via an API call.
- Added restore and permanently delete options for each deleted project.
- Integrated loading of deleted projects when the data tab is shown.
- Updated project creation modal to allow selection of optional modules (Sound and Vibration Monitoring).
- Modified project dashboard and header to display active modules and provide options to add/remove them.
- Enhanced project detail view to dynamically adjust UI based on enabled modules.
- Implemented a new migration script to create a `project_modules` table and seed it based on existing project types.
- Adjusted form submissions to handle module selections and ensure proper API interactions for module management.
- Fix UTC display bug: upload_nrl_data now wraps RNH datetimes with
local_to_utc() before storing, matching patch_session behavior.
Period type and label are derived from local time before conversion.
- Add period_start_hour / period_end_hour to MonitoringSession model
(nullable integers 0–23). Migration: migrate_add_session_period_hours.py
- Update patch_session to accept and store period_start_hour / period_end_hour.
Response now includes both fields.
- Update get_project_sessions to compute "Effective: M/D H:MM AM → M/D H:MM AM"
string from period hours and pass it to session_list.html.
- Rework period edit UI in session_list.html: clicking the period badge now
opens an inline editor with period type selector + start/end hour inputs.
Selecting a period type pre-fills default hours (Day: 7–19, Night: 19–7).
- Wire period hours into _build_location_data_from_sessions: uses
period_start/end_hour when set, falls back to hardcoded defaults.
- RND viewer: inject SESSION_PERIOD_START/END_HOUR from template context.
renderTable() dims rows outside the period window (opacity-40) with a
tooltip; shows "(N in period window)" in the row count.
- New session detail page at /api/projects/{id}/sessions/{id}/detail:
shows breadcrumb, files list with View/Download/Report actions,
editable session info form (label, period type, hours, times).
- Add local_datetime_input Jinja filter for datetime-local input values.
- Monthly calendar view: new get_sessions_calendar endpoint returns
sessions_calendar.html partial; added below sessions list in detail.html.
Color-coded per NRL with legend, HTMX prev/next navigation, session dots
link to detail page.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace all UnitAssignment "active" checks from `status == "active"` to
`assigned_until == None` in both project_locations.py and projects.py.
This aligns with the canonical definition: active = no end date set.
(status field is still set in sync, but is no longer the query criterion)
- Add `_require_sound_project()` helper to both routers and call it at the
top of every sound-monitoring-specific endpoint (FTP browser, FTP downloads,
RND file viewer, all Excel report endpoints, combined report wizard,
upload-all, NRL live status, NRL data upload). Vibration projects hitting
these endpoints now receive a clear 400 instead of silently failing or
returning empty results.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Updated reservation list to display estimated units and improved count display.
- Added "Upcoming" status to project dashboard and header with corresponding styles.
- Implemented a dropdown for quick status updates in project header.
- Modified project list compact view to reflect new status labels.
- Updated project overview to include a tab for upcoming projects.
- Added migration script to introduce estimated_units column in job_reservations table.
- Add POST /api/projects/{project_id}/nrl/{location_id}/upload-data endpoint
accepting a ZIP or multi-file select of .rnd/.rnh files from an SD card.
Parses .rnh metadata for session start/stop times, serial number, and store
name. Creates a MonitoringSession (no unit assignment required) and DataFile
records for each measurement file.
- Add Upload Data button and collapsible upload panel to the NRL detail Data
Files tab, with inline success/error feedback and automatic file list refresh
via HTMX after import.
- Rename RecordingSession -> MonitoringSession throughout the codebase
(models.py, projects.py, project_locations.py, scheduler.py, roster_rename.py,
main.py, init_projects_db.py, scripts/rename_unit.py). DB table renamed from
recording_sessions to monitoring_sessions; old indexes dropped and recreated.
- Update all template UI copy from Recording Sessions to Monitoring Sessions
(nrl_detail, projects/detail, session_list, schedule_oneoff, roster).
- Add backend/migrate_rename_recording_to_monitoring_sessions.py for applying
the table rename on production databases before deploying this build.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Moved Jinja2 template setup to a shared configuration file (templates_config.py) for consistent usage across routers.
- Introduced timezone utilities in a new module (timezone.py) to handle UTC to local time conversions and formatting.
- Updated all relevant routers to use the new shared template configuration and timezone filters.
- Enhanced templates to utilize local time formatting for various datetime fields, improving user experience with timezone awareness.
- Implemented a new API router for managing report templates, including endpoints for listing, creating, retrieving, updating, and deleting templates.
- Added a new HTML partial for a unified SLM settings modal, allowing users to configure SLM settings with dynamic modem selection and FTP credentials.
- Created a report preview page with an editable data table using jspreadsheet, enabling users to modify report details and download the report as an Excel file.
- Updated all instances of device_type from "sound_level_meter" to "slm" across the codebase.
- Enhanced documentation to reflect the new device type standardization.
- Added migration script to convert legacy device types in the database.
- Updated relevant API endpoints, models, and frontend templates to use the new device type.
- Ensured backward compatibility by deprecating the old device type without data loss.
- Created `schedule_list.html` to display scheduled actions with execution status, location, and timestamps.
- Implemented buttons for executing and canceling schedules, along with a details view placeholder.
- Created `unit_list.html` to show assigned units with their status, location, model, and session/file counts.
- Added conditional rendering for active sessions and links to view unit and location details.
serversdown
claude