Version bumped in backend/main.py, README header/highlights/history,
and CHANGELOG.md with a comprehensive 0.10.0 entry covering the SFM
integration work shipped on this branch.
Highlights:
- SFM event store is now the authoritative vibration data source
- SFM-primary seismograph status with heartbeat fallback
- Dashboard rework (top-row reorder, schedule moved, SFM-sourced
Recent Call-Ins)
- Event detail modal + sortable events tables across all event
surfaces
- Events attribution engine + metadata-backfill tool
- /admin/sfm + /admin/slmm diagnostic pages under Developer settings
- Tools workflow hub at /tools (Pair Devices, Project Tidy, Backfill)
- Sidebar reorganization (single Devices entry w/ tabs, new Events
+ Tools entries)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New /admin/sfm page (linked from Settings → Developer):
- Health banner — green/red with version + last-checked timestamp
- Connection panel — shows SFM_BASE_URL terra-view is configured with
- 4 KPI tiles — known units, total events, stale monitor_log rows,
stale ach_sessions rows (the deprecated tables from the paused
Python-ACH experiment, useful for confirming nothing's growing them)
- Per-unit roll-up table — serial, last_seen, event count, stale
per-unit counts, sourced from SFM's /db/units
- Recent events with forwarding latency — color-coded gap between
the event's recorded timestamp and SFM ingest time, so operators
can spot watchers that are forwarding stale files (e.g. after a
jobsite outage)
- Raw API tester — text input + GET button against any /api/sfm/*
path, response rendered as prettified JSON
New /admin/slmm page — same layout, stripped down to health + connection
+ raw API tester. For per-device SLM control the existing
/sound-level-meters dashboard remains the right entry point.
Backend (backend/routers/admin_modules.py):
- GET /admin/sfm, GET /admin/slmm — HTML pages
- GET /api/admin/sfm/overview — single aggregated probe that returns
health, units, last 25 events with computed latency, stale-table
counts, cache stats. Tolerant of partial failures: any sub-fetch
error is captured into errors{} so a flaky SFM endpoint doesn't
break the whole page
- GET /api/admin/slmm/overview — health + connection info only for now
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sidebar evolved from "Fleet defaults to seismograph dashboard" to
"Devices defaults to unified roster" + a new "Tools" entry housing the
active operator workflows.
Sidebar (6 items):
Dashboard · Devices · Projects · Tools · Job Planner · Settings
Changes:
- templates/base.html: renamed Fleet → Devices. Default route changed
from /seismographs to /roster — clicking Devices now lands on the
unified all-devices view, then operators drill into type-specific
layouts via the tab strip. Tools entry added between Projects and
Job Planner; highlights when on /tools or any of its linked workflow
pages.
- templates/partials/fleet_tab_strip.html: reordered tabs so "All
Devices" comes first (matches the new default landing).
Seismographs → SLMs → Modems follow.
- templates/tools.html (new) + /tools route in main.py: card grid hub
for active workflows.
• Pair Devices — links to /pair-devices
• Project Tidy — links to /settings/developer/project-tidy
• Backfill from event metadata — /settings/developer/metadata-backfill
• Reports — info card pointing to project detail pages where
Excel report generation actually lives (per-project context)
• Swap Detection — greyed-out placeholder for Phase 5c
- templates/settings.html: removed Project Tidy + Metadata Backfill
cards from Settings → Developer. They now live in Tools. Settings
→ Developer retains the truly admin/dev surfaces (Watcher Manager,
SFM Admin).
The workflow page URLs (/settings/developer/project-tidy,
/settings/developer/metadata-backfill) stay where they are — only the
nav entry point changes. Bookmarks still work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 5b first slice. Surfaces near-duplicate projects (typo variants,
abbreviation differences, spacing variations like "SR81" vs "SR 81")
as side-by-side pairs the operator can merge with one click.
Backend (backend/services/project_tidy.py):
- find_duplicate_pairs(db, threshold=0.85) walks all active projects and
computes rapidfuzz.WRatio similarity for every pair. Pre-filters
too-short normalised names (< 4 chars) to avoid noise. Skips
soft-deleted projects. Returns pairs sorted by score desc, then by
total content (more assignments → review first).
- Each pair carries a suggested merge target with a human-readable
reason. Priorities (in order): manual source over parser source,
populated project_number, more locations, more assignments, shorter
name. Operator can override the suggestion by clicking the OTHER
direction button.
- O(N^2) over project count. Fine up to ~500 projects. Token-prefix
blocking is the obvious next optimisation if it becomes slow.
Backend (backend/routers/projects.py):
- GET /api/projects/admin/duplicate_pairs?threshold=&max_pairs= returns
pairs as JSON for the Tidy page.
Frontend (templates/admin/project_tidy.html):
- New admin page at /settings/developer/project-tidy. Threshold selector
(95% / 90% / 85% / 80%) at the top; rescan button next to it; auto-
scans on load.
- Each pair card shows side-by-side project summaries (name, project_
number, client, source-badge, location/assignment counts) with the
suggested target visually highlighted (orange border). Three buttons:
"Merge A → B", "Merge B → A", "Not a dup" (hide locally).
- Click-to-merge opens a native confirm with the preview totals
(assignments/sessions/data files moving, consolidations) — same data
the project_header.html merge modal shows. On confirm, hits the
existing /merge_into endpoint and re-scans automatically.
- Source badges distinguish parser-created (`metadata_backfill`) from
manual projects — at a glance the operator can see "this duplicate is
parser-generated; safe to merge into the manual one".
Frontend (templates/admin/metadata_backfill.html):
- Apply-result handling now surfaces failed[] cluster reasons in a
dedicated failure panel (bottom-left, dismissable). Previously a 200
OK with all-failures showed a misleading "1 cluster applied" success
toast because the count and the failure list weren't being reconciled.
This bit us during the DB-revert recovery earlier — the
project_modules table was missing, every apply silently rolled back,
user saw success toasts. Fixed.
Smoke-verified against current state (10K events, 9 projects, post-
merge): tool correctly finds 0 pairs at threshold 0.85 (data is clean),
1 false-positive at 0.70 (two unrelated projects sharing the token "81"
— example of why the 0.85 default is correct).
Settings link added under Developer → Project Tidy.
Phase 5c (swap-detection daily background job + notification inbox)
remains deferred to the next session.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Operator clicks one button. Parser reads SFM's events table (operator-typed
project / client / sensor_location strings), clusters by serial + time +
metadata, fuzzy-matches against existing projects, and proposes
Project / MonitoringLocation / UnitAssignment chains to create.
Auto-applies high-confidence non-conflicting clusters in bulk; queues
medium/low confidence for individual review.
Verified against real data: 10,052 events → 59 clusters → 37 high-
confidence + 14 medium + 8 low. Test-applied one cluster end-to-end;
Project + Module + Location + Assignment + UnitHistory + Decision rows
all created correctly, and Phase 2's attribution walk picked up the
events automatically on the new location's detail page.
Pipeline (backend/services/metadata_backfill.py, ~700 lines):
1. Pull all SFM events via /db/events per serial.
2. Pre-filter: drop events already covered by an existing UnitAssignment
window (Phase 2 handles those automatically).
3. Time-cluster what's left: serial + 7-day gap is the cluster identity.
4. Metadata-split each time-cluster on persistent metadata transitions
(≥ 2 consecutive events) so a single typo doesn't fork the cluster.
5. Match against existing graph (rapidfuzz.WRatio multi-signal scoring,
normalisation that handles abbreviations / reorders / separator
variations). Thresholds: 0.95 exact, 0.80 fuzzy, min-shorter-input
5 chars to guardrail false positives on single common words.
6. Score confidence (high/medium/low) using event count, span,
blank-meta, conflict, ambiguity rules.
7. Detect conflicts: overlap with existing UnitAssignment at a different
location for the same serial → blocking. Operator must reconcile.
8. Apply: ensure auto_imported ProjectType exists, ensure
vibration_monitoring ProjectModule on the project, write
Project / MonitoringLocation / UnitAssignment / UnitHistory all in
one transaction.
Migration (backend/migrate_add_metadata_backfill.py): adds
unit_assignments.source column (default 'manual') and
metadata_backfill_decisions table. Idempotent, non-destructive.
API (backend/routers/metadata_backfill.py):
GET /api/admin/metadata_backfill/scan — clusters + suggestions
POST /api/admin/metadata_backfill/apply — bulk apply by cluster_ids
w/ optional per-cluster
project/location overrides
POST /api/admin/metadata_backfill/skip — mark skipped (persistent)
UI (templates/admin/metadata_backfill.html, accessible at
/settings/developer/metadata-backfill via the Developer tab of Settings):
- One-button "Run scan" entry.
- Summary KPI tiles (scanned / already attributed / pending / conflicts).
- "Apply all high-confidence" bulk button at the top — primary path.
- Per-cluster cards below with Apply / Skip / Preview event actions.
- Blank-meta clusters get inline input fields for operator-typed project +
location names before applying.
- Blocking-conflict clusters render with the conflicting assignment
information and a disabled Apply button.
- Live progress toast during apply.
- Reuses the Phase 1+2+4 event-detail modal for "Preview event" — operator
can sanity-check the BW report data against the cluster's sample event.
Dependencies: rapidfuzz==3.10.1 added to requirements.txt. Pre-built C
wheels for all platforms, ~5s docker build hit.
Phase 5b (deferred to next session): swap-detection daily background job,
notification inbox for auto-applied swaps, recently-applied audit view,
"Tidy" page for renaming/merging auto-created projects.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- backend/routers/sfm.py: HTTP proxy to SFM backend (localhost:8200),
mirrors the SLMM proxy pattern. SFM_BASE_URL env var for docker-compose.
Catch-all /{path} forwards to SFM root (no /api/ prefix). 60s timeout.
- templates/sfm.html: full SFM dashboard with 5 tabs:
Events (DB listing, filters by serial/date/false-trigger, flag/unflag FT),
Units (known serials + stats, filter events by unit),
Monitor Log (continuous monitoring intervals),
ACH Sessions (call-home history),
Live Device (TCP connect, device info cards, start/stop monitoring,
push project config, download events from device, operation log).
- backend/main.py: import sfm router, include router, add GET /sfm route
- templates/base.html: add SFM Live Data nav link under Seismographs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Updated dashboard to display allocated units alongside deployed and benched units.
- Introduced a quick-info modal for units, showing detailed information including calibration status, project allocation, and upcoming jobs.
- Enhanced fleet calendar with a new quick-info modal for units, allowing users to view unit details without navigating away.
- Modified devices table to include allocated status and visual indicators for allocated units.
- Added allocated filter option in the roster view for better unit management.
- Implemented backend migration to add 'allocated' and 'allocated_to_project_id' columns to the roster table.
- Updated unit detail view to reflect allocated status and allow for project allocation input.
- Add POST /api/projects/{project_id}/nrl/{location_id}/upload-data endpoint
accepting a ZIP or multi-file select of .rnd/.rnh files from an SD card.
Parses .rnh metadata for session start/stop times, serial number, and store
name. Creates a MonitoringSession (no unit assignment required) and DataFile
records for each measurement file.
- Add Upload Data button and collapsible upload panel to the NRL detail Data
Files tab, with inline success/error feedback and automatic file list refresh
via HTMX after import.
- Rename RecordingSession -> MonitoringSession throughout the codebase
(models.py, projects.py, project_locations.py, scheduler.py, roster_rename.py,
main.py, init_projects_db.py, scripts/rename_unit.py). DB table renamed from
recording_sessions to monitoring_sessions; old indexes dropped and recreated.
- Update all template UI copy from Recording Sessions to Monitoring Sessions
(nrl_detail, projects/detail, session_list, schedule_oneoff, roster).
- Add backend/migrate_rename_recording_to_monitoring_sessions.py for applying
the table rename on production databases before deploying this build.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- pair_devices.html template for device pairing interface
- SLMM device control lock prevents flooding nl43.
Fix:
- Polling intervals for SLMM.
- modem view now list
- device pairing much improved.
- various other tweaks through out UI.
- SLMM Scheduled downloads fixed.
- Moved Jinja2 template setup to a shared configuration file (templates_config.py) for consistent usage across routers.
- Introduced timezone utilities in a new module (timezone.py) to handle UTC to local time conversions and formatting.
- Updated all relevant routers to use the new shared template configuration and timezone filters.
- Enhanced templates to utilize local time formatting for various datetime fields, improving user experience with timezone awareness.
- Implemented a new API router for managing report templates, including endpoints for listing, creating, retrieving, updating, and deleting templates.
- Added a new HTML partial for a unified SLM settings modal, allowing users to configure SLM settings with dynamic modem selection and FTP credentials.
- Created a report preview page with an editable data table using jspreadsheet, enabling users to modify report details and download the report as an Excel file.
- Updated all instances of device_type from "sound_level_meter" to "slm" across the codebase.
- Enhanced documentation to reflect the new device type standardization.
- Added migration script to convert legacy device types in the database.
- Updated relevant API endpoints, models, and frontend templates to use the new device type.
- Ensured backward compatibility by deprecating the old device type without data loss.
- Implemented a modal for renaming units with validation and confirmation prompts.
- Added JavaScript functions to handle opening, closing, and submitting the rename unit form.
- Enhanced the back navigation in the SLM detail page to check referrer history.
- Updated breadcrumb navigation in the legacy dashboard to accommodate NRL locations.
- Improved the sound level meters page with a more informative header and device list.
- Introduced a live measurement chart with WebSocket support for real-time data streaming.
- Added functionality to manage active devices and projects with auto-refresh capabilities.
- Created `schedule_list.html` to display scheduled actions with execution status, location, and timestamps.
- Implemented buttons for executing and canceling schedules, along with a details view placeholder.
- Created `unit_list.html` to show assigned units with their status, location, model, and session/file counts.
- Added conditional rendering for active sessions and links to view unit and location details.
- Created a new template for displaying a list of data files in `file_list.html`, including file details and actions for downloading and viewing file details.
- Added a new template for displaying recording sessions in `session_list.html`, featuring session status, details, and action buttons for stopping recordings and viewing session details.
- Introduced a legacy dashboard template `slm_legacy_dashboard.html` for sound level meter control, including a live view panel and configuration modal with dynamic content loading.
serversdown
claude