Version bumped in backend/main.py, README header/highlights/history,
and CHANGELOG.md with a comprehensive 0.10.0 entry covering the SFM
integration work shipped on this branch.
Highlights:
- SFM event store is now the authoritative vibration data source
- SFM-primary seismograph status with heartbeat fallback
- Dashboard rework (top-row reorder, schedule moved, SFM-sourced
Recent Call-Ins)
- Event detail modal + sortable events tables across all event
surfaces
- Events attribution engine + metadata-backfill tool
- /admin/sfm + /admin/slmm diagnostic pages under Developer settings
- Tools workflow hub at /tools (Pair Devices, Project Tidy, Backfill)
- Sidebar reorganization (single Devices entry w/ tabs, new Events
+ Tools entries)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New /admin/sfm page (linked from Settings → Developer):
- Health banner — green/red with version + last-checked timestamp
- Connection panel — shows SFM_BASE_URL terra-view is configured with
- 4 KPI tiles — known units, total events, stale monitor_log rows,
stale ach_sessions rows (the deprecated tables from the paused
Python-ACH experiment, useful for confirming nothing's growing them)
- Per-unit roll-up table — serial, last_seen, event count, stale
per-unit counts, sourced from SFM's /db/units
- Recent events with forwarding latency — color-coded gap between
the event's recorded timestamp and SFM ingest time, so operators
can spot watchers that are forwarding stale files (e.g. after a
jobsite outage)
- Raw API tester — text input + GET button against any /api/sfm/*
path, response rendered as prettified JSON
New /admin/slmm page — same layout, stripped down to health + connection
+ raw API tester. For per-device SLM control the existing
/sound-level-meters dashboard remains the right entry point.
Backend (backend/routers/admin_modules.py):
- GET /admin/sfm, GET /admin/slmm — HTML pages
- GET /api/admin/sfm/overview — single aggregated probe that returns
health, units, last 25 events with computed latency, stale-table
counts, cache stats. Tolerant of partial failures: any sub-fetch
error is captured into errors{} so a flaky SFM endpoint doesn't
break the whole page
- GET /api/admin/slmm/overview — health + connection info only for now
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Event modal (event-modal.js):
- Record Type now derived from Blastware filename's last-char code
(H=Histogram, W=Waveform, M=Manual, E=Event, C=Combo). Falls back to
whatever SFM reported if the code isn't recognized. Client-side
workaround — SFM still hardcodes "Waveform" server-side and needs a
proper fix in its sidecar parser.
- PSI mic tile dropped; mic section now renders 3 tiles (dB(L), ZC
Frequency, Time of Peak) instead of 4.
- New "View JSON" toggle exposes a prettified inline JSON viewer with
a Copy-to-clipboard button alongside the existing "Download sidecar
JSON" link.
- "Project Info" section header renamed to "User Notes" to reflect
that these are operator-typed fields, not the terra-view project
assignment.
Sortable tables (sfm.html + unit_detail.html):
- Both Events tables now have clickable column headers with ↕/↓/↑
indicators. Default sort is Timestamp DESC. Clicking the same
column toggles direction; clicking a different column switches and
resets to DESC. Sort is purely client-side over the cached rowset,
so no extra fetches.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
emit_status_snapshot() now consults SFM /db/units (cached 15s) before
falling back to Emitter.last_seen for each seismograph. The fresher of
the two wins and the choice is recorded in a new per-unit
last_seen_source field ("sfm" | "heartbeat" | "none"). sfm_reachable is
exposed alongside so the UI can show degraded state.
Fallback is transparent: if SFM is unreachable or has no record for a
serial, the watcher heartbeat path takes over and the unit just shows
the HB badge instead of SFM. No schema changes; SLMs are untouched
(they don't go through SFM); modems inherit source from their pair.
active_table.html grows a small "SFM" / "HB" badge next to the age
column so operators can see at a glance which path is currently
driving each unit's status.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Top row left→right: Recent Alerts | Recent Call-Ins (2 cols) | Fleet Summary
- Today's Schedule becomes a horizontal collapsible card below Fleet Map.
Collapsed by default; auto-expands when pending actions are detected in
the rendered partial; manual toggle sticks via localStorage.
- New /api/recent-event-callins proxies SFM /db/events and bulk-joins each
serial against RosterUnit for in-roster annotation. Phases the
heartbeat-derived /api/recent-callins out of the UI while keeping it as
a backup endpoint for now.
- Call-ins card renders a dense 2-column grid (last 10 events) showing
PVS, sensor_location, false-trigger badge, event timestamp, and
links to the unit page when rostered.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two related operator-facing improvements after the nav reorg.
1) Events as a top-level sidebar entry.
The /sfm page (fleet-wide event database) was demoted to Settings →
Developer in the previous reorg. Bringing it back to main nav as
"Events" — operators do reach for the cross-project, sortable
event list, so it earns a top-level slot.
Sidebar now (7 items):
Dashboard · Devices · Projects · Events · Tools · Job Planner · Settings
Settings → Developer card pointing at /sfm is removed. /sfm page
title/subtitle updated from "SFM Event Data" to just "Events". URL
unchanged.
2) "Peak PVS" KPI tile becomes "Overall Peak" and excludes false
triggers from the calculation.
When operators ask "what's the biggest event at this location/unit/
project?" they mean the biggest REAL event, not the biggest sensor
glitch. A single mis-flagged false trigger could otherwise dominate
the tile (the 14.13 in/s spike at Loc 1 was a prime example).
backend/services/sfm_events.py:
- _compute_stats() skips false_trigger=True events when computing
peak_pvs / peak_pvs_at / peak_pvs_serial. Continues counting them
in false_trigger_count so the separate "False Triggers" tile still
reflects what got filtered out. last_event unchanged (recency, not
magnitude).
- Same change automatically propagates to events_for_unit() and
vibration_summary_for_project() — both call _compute_stats().
Templates: "Peak PVS" → "Overall Peak" in 3 KPI tile locations
(vibration_location_detail.html, partials/projects/vibration_summary
.html, unit_detail.html). The physical-quantity name "Peak Vector
Sum" in the event-detail modal stays — that's the actual physics
term, not a summary stat.
Verified end-to-end: Overall Peak renders on real data; peak event
false_trigger flag confirmed False.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sidebar evolved from "Fleet defaults to seismograph dashboard" to
"Devices defaults to unified roster" + a new "Tools" entry housing the
active operator workflows.
Sidebar (6 items):
Dashboard · Devices · Projects · Tools · Job Planner · Settings
Changes:
- templates/base.html: renamed Fleet → Devices. Default route changed
from /seismographs to /roster — clicking Devices now lands on the
unified all-devices view, then operators drill into type-specific
layouts via the tab strip. Tools entry added between Projects and
Job Planner; highlights when on /tools or any of its linked workflow
pages.
- templates/partials/fleet_tab_strip.html: reordered tabs so "All
Devices" comes first (matches the new default landing).
Seismographs → SLMs → Modems follow.
- templates/tools.html (new) + /tools route in main.py: card grid hub
for active workflows.
• Pair Devices — links to /pair-devices
• Project Tidy — links to /settings/developer/project-tidy
• Backfill from event metadata — /settings/developer/metadata-backfill
• Reports — info card pointing to project detail pages where
Excel report generation actually lives (per-project context)
• Swap Detection — greyed-out placeholder for Phase 5c
- templates/settings.html: removed Project Tidy + Metadata Backfill
cards from Settings → Developer. They now live in Tools. Settings
→ Developer retains the truly admin/dev surfaces (Watcher Manager,
SFM Admin).
The workflow page URLs (/settings/developer/project-tidy,
/settings/developer/metadata-backfill) stay where they are — only the
nav entry point changes. Bookmarks still work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 5b first slice. Surfaces near-duplicate projects (typo variants,
abbreviation differences, spacing variations like "SR81" vs "SR 81")
as side-by-side pairs the operator can merge with one click.
Backend (backend/services/project_tidy.py):
- find_duplicate_pairs(db, threshold=0.85) walks all active projects and
computes rapidfuzz.WRatio similarity for every pair. Pre-filters
too-short normalised names (< 4 chars) to avoid noise. Skips
soft-deleted projects. Returns pairs sorted by score desc, then by
total content (more assignments → review first).
- Each pair carries a suggested merge target with a human-readable
reason. Priorities (in order): manual source over parser source,
populated project_number, more locations, more assignments, shorter
name. Operator can override the suggestion by clicking the OTHER
direction button.
- O(N^2) over project count. Fine up to ~500 projects. Token-prefix
blocking is the obvious next optimisation if it becomes slow.
Backend (backend/routers/projects.py):
- GET /api/projects/admin/duplicate_pairs?threshold=&max_pairs= returns
pairs as JSON for the Tidy page.
Frontend (templates/admin/project_tidy.html):
- New admin page at /settings/developer/project-tidy. Threshold selector
(95% / 90% / 85% / 80%) at the top; rescan button next to it; auto-
scans on load.
- Each pair card shows side-by-side project summaries (name, project_
number, client, source-badge, location/assignment counts) with the
suggested target visually highlighted (orange border). Three buttons:
"Merge A → B", "Merge B → A", "Not a dup" (hide locally).
- Click-to-merge opens a native confirm with the preview totals
(assignments/sessions/data files moving, consolidations) — same data
the project_header.html merge modal shows. On confirm, hits the
existing /merge_into endpoint and re-scans automatically.
- Source badges distinguish parser-created (`metadata_backfill`) from
manual projects — at a glance the operator can see "this duplicate is
parser-generated; safe to merge into the manual one".
Frontend (templates/admin/metadata_backfill.html):
- Apply-result handling now surfaces failed[] cluster reasons in a
dedicated failure panel (bottom-left, dismissable). Previously a 200
OK with all-failures showed a misleading "1 cluster applied" success
toast because the count and the failure list weren't being reconciled.
This bit us during the DB-revert recovery earlier — the
project_modules table was missing, every apply silently rolled back,
user saw success toasts. Fixed.
Smoke-verified against current state (10K events, 9 projects, post-
merge): tool correctly finds 0 pairs at threshold 0.85 (data is clean),
1 false-positive at 0.70 (two unrelated projects sharing the token "81"
— example of why the 0.85 default is correct).
Settings link added under Developer → Project Tidy.
Phase 5c (swap-detection daily background job + notification inbox)
remains deferred to the next session.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Operator-facing tool for cleaning up duplicate projects. Common after
the metadata-backfill parser auto-creates near-duplicates from operator
name variations ("SR81" vs "SR 81", "Swank-Karns Crossing" vs
"Swank-Karns Crossings", "Trumbull-Bryman Mont.Dam" vs
"Trumbull-Brayman-Mont Dam", etc.).
Workflow: visit the duplicate project's detail page, click "Merge into…"
in the header, search for the canonical target project from a typeahead,
review the preview (what assignments / locations / sessions will move,
any conflicts), confirm. Source is soft-deleted; everything else
re-points to the target. Smart consolidation: same-named locations in
both projects merge into one (source's assignments move to target's
existing location with the same name; source's empty location is then
deleted). Different-named locations move as-is.
Backend:
- backend/services/project_merge.py (new): preview() and execute()
functions. Transaction-safe. Per-assignment UnitHistory audit row
with change_type='assignment_merged' so the deployment timeline shows
the merge. Source modules disabled; missing modules added to target.
Handles edge cases: same project_id rejected, deleted projects rejected,
orphan project-direct assignments (no location) re-pointed defensively.
- backend/routers/projects.py: new endpoints
GET /api/projects/{source_id}/merge_preview?target_id=...
POST /api/projects/{source_id}/merge_into?target_id=...
Frontend (templates/partials/projects/project_header.html):
- "Merge into…" button in Project Actions area.
- Modal with typeahead (reuses /api/admin/metadata_backfill/projects_search)
scoped to existing projects only (no create-new option). Filters out
the source project from candidates so operator can't accidentally pick
it as target.
- Preview pane shows totals + per-location plan (consolidate vs move) +
warnings (mismatched client names, location consolidation note).
- Red "Merge (permanent)" confirm button only enables after a target is
picked and preview loads.
- On success, browser redirects to target project page.
Smoke verified: "Swank-Karns Crossing" (1 assignment) merged into
"Swank-Karns Crossings"; target now has 2 locations + 2 assignments,
source has 0 dangling rows, 1 project_merge audit entry written.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Operator no longer has to accept the parser's suggested project /
location verbatim. Each cluster card now has editable typeahead inputs
that search existing projects (and existing locations within the chosen
project), with a "Create new: <typed>" fallback always available.
Solves the I-80-North-Fork case: of the 20+ cluster variants
("I-80-North Fork Bridges-I80 E. Abutment", "I-80- North Fork
Bridges-543 Plank Rd", etc.), operator types "I-80" in the Project
input, picks the existing project from the dropdown, and the cluster
attaches to it. Repeat for the other variants. No need to pre-create
the canonical project — though pre-creation still works fine if you'd
rather.
Backend (backend/routers/metadata_backfill.py):
- GET /api/admin/metadata_backfill/projects_search?q=&limit=
Returns existing projects matching by case-insensitive substring OR
rapidfuzz WRatio score >= 0.50. Substring matches sort to the top
(treated as exact for ordering). Includes location_count and
project_number/client_name in each result for disambiguation. Always
emits a "Create new: <q>" suggestion alongside the matches.
- GET /api/admin/metadata_backfill/locations_search?project_id=&q=&limit=
Same shape, scoped to a single project's vibration locations.
- POST /api/admin/metadata_backfill/apply now accepts four override
keys per cluster (was previously two):
project_id → attach to existing Project (operator picked from
typeahead)
project_name → create new with this name (operator typed a
custom name; existing project_name behaviour)
location_id → attach to existing MonitoringLocation; validated
against the chosen project_id so a stale location
FK can't sneak in
location_name → create new location with this name
Frontend (templates/admin/metadata_backfill.html):
- Each non-blank-meta cluster card now has two editable typeahead inputs
(Project + Location) pre-populated with the parser's suggested
values. Old static "Project: + Create new: X" / "≈ Fuzzy match" pills
replaced with compact hint lines under the inputs showing what the
current value will do.
- Typeahead dropdown opens on focus, debounced 150ms on type. Shows
matched existing entities with score badges (exact / NN%) plus a
"Create new: <typed>" option at the bottom. Click-to-pick fills the
text input and writes the entity id into a hidden field.
- Picking a new project clears the location id (forces re-pick under
the new project, avoids cross-project location FKs).
- _gatherOverrides re-wired to emit the new project_id / location_id
keys when the operator picked from the dropdown, falling back to
*_name when they typed free-form.
Backward-compatible: blank-meta clusters keep their existing "project_name
/ location_name" plain inputs and the override path still honours them.
Verified end-to-end:
- /projects_search?q=I-80 returns the existing "I-80 - North Fork
Bridge" project (score 1.0, has 4 locations) plus a "Create new"
option.
- /locations_search requires project_id (400 without it).
- Wizard page renders with typeahead wiring confirmed in HTML.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Operators use more separator variations than the original regex caught:
- "Trumbull-Brayman-JV- Mont.Dam.Loc 2-R-25" — period as separator
- "CMU - RKM Hall - Loc No. 3 - 4615 Forbes" — "No." between Loc and digit
Added period to the separator character class and optional "No." token
before the digit. Catches both above patterns plus near-variants
without false-positives on normal project strings.
Real-data impact: 5 more clusters now auto-strip cleanly, including the
1,903-event Trumbull-Brayman-JV- Mont.Dam cluster. Confidence
distribution: 43 → 44 high.
Operators sometimes bake location identifiers into the project string
for email-readability — "Fay - Locks & Dam No3 - Loc 2 - 735 Bunola"
where "Fay - Locks & Dam No3" is the actual project and "- Loc 2 -
735 Bunola" is location info that already lives in sensor_location.
Without stripping, every "- Loc N" variant became a separate project,
fragmenting what should be one project with several locations.
Backend:
- New _extract_project_root() helper. Regex matches " - Loc N" / "-Loc3" /
" - Location #5" / etc. with case-insensitive multi-dash support; strips
from that marker forward and cleans up dangling separators. Strings
without a Loc-marker pass through unchanged.
- Cluster dataclass adds project_root field alongside project_raw.
project_raw stays the operator-typed string for display ("hover to see
what was actually typed"). project_root is what gets normalised for
matching and used as the suggested project name.
- _ensure_project + _ensure_location now do normalisation-aware dedup
before creating: a cluster of "SR81" and a cluster of "SR 81" (which
normalise to the same string) collapse into one project on apply,
even when applied in the same bulk operation. Avoids UNIQUE
constraint collisions and duplicate-named-by-spacing projects.
Frontend:
- Wizard cluster cards show "↳ stripped trailing 'Loc N' suffix; operator
typed: <raw>" when project_root differs from project_raw, so the
operator can see at a glance what the parser did to the string.
Real-data results: against the same 10,055 SFM events, confidence
distribution improved from 37/14/8 (high/med/low) to 43/9/7. "Fay -
Locks & Dam No3" now appears as ONE project across 6 cluster instances
spanning 3 serials and 6 different locations — exactly the
"one project, many locations" model the user described.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Operator clicks one button. Parser reads SFM's events table (operator-typed
project / client / sensor_location strings), clusters by serial + time +
metadata, fuzzy-matches against existing projects, and proposes
Project / MonitoringLocation / UnitAssignment chains to create.
Auto-applies high-confidence non-conflicting clusters in bulk; queues
medium/low confidence for individual review.
Verified against real data: 10,052 events → 59 clusters → 37 high-
confidence + 14 medium + 8 low. Test-applied one cluster end-to-end;
Project + Module + Location + Assignment + UnitHistory + Decision rows
all created correctly, and Phase 2's attribution walk picked up the
events automatically on the new location's detail page.
Pipeline (backend/services/metadata_backfill.py, ~700 lines):
1. Pull all SFM events via /db/events per serial.
2. Pre-filter: drop events already covered by an existing UnitAssignment
window (Phase 2 handles those automatically).
3. Time-cluster what's left: serial + 7-day gap is the cluster identity.
4. Metadata-split each time-cluster on persistent metadata transitions
(≥ 2 consecutive events) so a single typo doesn't fork the cluster.
5. Match against existing graph (rapidfuzz.WRatio multi-signal scoring,
normalisation that handles abbreviations / reorders / separator
variations). Thresholds: 0.95 exact, 0.80 fuzzy, min-shorter-input
5 chars to guardrail false positives on single common words.
6. Score confidence (high/medium/low) using event count, span,
blank-meta, conflict, ambiguity rules.
7. Detect conflicts: overlap with existing UnitAssignment at a different
location for the same serial → blocking. Operator must reconcile.
8. Apply: ensure auto_imported ProjectType exists, ensure
vibration_monitoring ProjectModule on the project, write
Project / MonitoringLocation / UnitAssignment / UnitHistory all in
one transaction.
Migration (backend/migrate_add_metadata_backfill.py): adds
unit_assignments.source column (default 'manual') and
metadata_backfill_decisions table. Idempotent, non-destructive.
API (backend/routers/metadata_backfill.py):
GET /api/admin/metadata_backfill/scan — clusters + suggestions
POST /api/admin/metadata_backfill/apply — bulk apply by cluster_ids
w/ optional per-cluster
project/location overrides
POST /api/admin/metadata_backfill/skip — mark skipped (persistent)
UI (templates/admin/metadata_backfill.html, accessible at
/settings/developer/metadata-backfill via the Developer tab of Settings):
- One-button "Run scan" entry.
- Summary KPI tiles (scanned / already attributed / pending / conflicts).
- "Apply all high-confidence" bulk button at the top — primary path.
- Per-cluster cards below with Apply / Skip / Preview event actions.
- Blank-meta clusters get inline input fields for operator-typed project +
location names before applying.
- Blocking-conflict clusters render with the conflicting assignment
information and a disabled Apply button.
- Live progress toast during apply.
- Reuses the Phase 1+2+4 event-detail modal for "Preview event" — operator
can sanity-check the BW report data against the cluster's sample event.
Dependencies: rapidfuzz==3.10.1 added to requirements.txt. Pre-built C
wheels for all platforms, ~5s docker build hit.
Phase 5b (deferred to next session): swap-detection daily background job,
notification inbox for auto-applied swaps, recently-applied audit view,
"Tidy" page for renaming/merging auto-created projects.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two new action buttons at the top of the Source File section of the
event-detail modal:
1. Download Blastware file — primary orange button. Pulls the raw .AB0
/.G10/.6R0/etc. binary from SFM (/db/events/{id}/blastware_file) via
terra-view's /api/sfm proxy. The browser saves it with the original
on-disk filename (using the HTML5 `download` attribute pointed at
sidecar.blastware.filename). Operator can then open the file
directly in Blastware on a Windows box for full waveform analysis,
archive it, or attach it to a compliance report.
Greyed-out "Blastware file unavailable" placeholder shown when
sidecar.blastware.available is false (rare — would mean SFM stored
the metadata but lost the binary).
2. Download sidecar JSON — secondary outlined button. Pulls the same
.sfm.json the modal renders from. Saved as <binary>.sfm.json.
Useful for ops/diagnostics and for the future metadata-driven
project parser (Phase 5) which can chew on these directly.
End-to-end verified through the proxy: 8882-byte Blastware binary
intact with "Instantel" magic header preserved.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Clicking any event row in any of the three event tables (/sfm Events,
project-location Events tab, unit detail SFM Events) now opens a modal
populated from the SFM .sfm.json sidecar. Previously the /sfm page had
a basic inline modal showing only the columns already in the table;
this rebuilds it as a shared component and exposes the rich fields
that the BW ASCII report unlocks.
Shared component:
- backend/static/event-modal.js — single ~250-line module. Public API:
showEventDetail(eventId) fetches /api/sfm/db/events/{id}/sidecar
live (no extra terra-view caching) and renders sections for:
• Event (serial, timestamp, record type, sample rate, rec time,
waveform key)
• Project Info (operator-typed user notes — project / client /
operator / sensor_location — flagged in the UI as "as typed
into the seismograph at session start", not the terra-view
assignment)
• Peak Particle Velocity (per-channel + vector sum, with the
time-of-vector-sum-peak when bw_report is available)
• Microphone (Peak dB(L) + psi, ZC frequency, time of peak)
• Sensor Self-Check table (per-channel freq + ratio/amplitude +
pass/fail)
• Device & Recording Metadata (firmware, battery, calibration
date + by-whom, geo range, stop mode, units)
• Source File (Blastware filename, size, SHA-256, capture time)
closeEventDetailModal() closes; Escape key also closes.
- templates/partials/event_detail_modal.html — modal shell partial
(sticky title bar, scrollable body, click-outside-to-close).
Wired into three pages:
- templates/sfm.html: removed the old inline modal + showEventDetail /
ppvCard / closeEventModal functions (replaced by the shared module).
Row onclick now passes just the event id instead of the full JSON.
- templates/vibration_location_detail.html: row click on the Events
tab opens the modal. The /unit/{serial} link inside the row has
event.stopPropagation() so the link navigates instead of opening
the modal.
- templates/unit_detail.html: row click on the SFM Events table opens
the modal. The attribution-cell project/location links also got
stopPropagation.
Graceful degradation: older events forwarded before the watcher's
_ASCII.TXT pairing fix don't have a bw_report block in their sidecar.
The modal renders an amber banner explaining that and shows just the
event + project_info + peak_values + source-file sections.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 4. Rebuilds the seismograph "Deployment History" + "Timeline"
sections on the unit detail page as a single derived view computed from
three sources: unit_assignments (authoritative project/location windows),
unit_history (calibration/retirement/deployed state changes), and SFM
events overlaid per assignment window (count + peak PVS + last event).
Fixes the wonky-timeline symptoms: missing entries, duplicate/contradictory
rows, and no visibility into what the unit was actually doing during each
deployment window.
Backend:
- backend/services/deployment_timeline.py: new deployment_timeline_for_unit()
helper. Merges UnitAssignment rows (with SFM event overlay fetched
concurrently via httpx), UnitHistory state-change rows (filtered to
meaningful change_types and de-noised by dropping rows where
old_value == new_value — there's noise in legacy audit log from
record_history() being called on every save), and synthetic "gap"
entries between assignments >= 1 day apart. Sorts newest first.
- backend/routers/units.py: new GET /api/units/{unit_id}/deployment_timeline
endpoint with optional include_events=false flag.
- backend/routers/project_locations.py: assign / unassign / swap /
update endpoints now write UnitHistory rows on every assignment
lifecycle event. New change_types: assignment_created,
assignment_ended, assignment_swapped, assignment_updated. These
surface in the unified timeline (where the assignment row itself
shows the structural data; the audit row is filtered out to avoid
double-rendering). Closes a real gap — assignment changes were
previously invisible to any audit consumer.
- backend/migrate_deprecate_deployment_records.py: non-destructive
migration. Adds deployment_records.deprecated_at column. For each
legacy row without a matching UnitAssignment, best-effort
synthesizes one (with the free-text location_name preserved in
notes). Marks every processed row. Idempotent. DROP TABLE
deferred to a follow-up release.
Frontend (templates/unit_detail.html):
- Removed legacy "Deployment History" card (with Log Deployment button)
and the separate "Timeline" card. Replaced with a single
"Deployment Timeline" section.
- Three entry visual styles: assignment rows (orange dot, location +
project link, event-overlay summary), gap rows (dashed outline, idle
day count), and state_change rows (navy dot, friendly label, old →
new value). Active assignments get a green dot + "active" badge.
- Existing loadUnitHistory() and loadDeploymentHistory() functions kept
as shims that delegate to loadDeploymentTimeline(), so modal-save
callbacks that referenced them still trigger a refresh of the visible
section. Legacy function bodies preserved under _legacy_*_unused
names for archeology; not called by anything.
Verified end-to-end:
- BE11529 timeline now shows 2 entries (active assignment with 24-event
overlay + the deployed→benched state change), compared to the previous
noisy mix that included 6 no-op state-change rows.
- Migration ran against real DB: 1 legacy row processed (had no
project_id, marked deprecated without backfill).
- Assign / unassign / swap / edit now leave a paper trail in
unit_history.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 3 of the SFM integration. Adds a "Project-wide vibration events"
KPI card to the Vibration tab of every project detail page, summarising
event activity across all of that project's vibration MonitoringLocations.
Backend:
- backend/services/sfm_events.py: vibration_summary_for_project() helper.
Concurrently fans out events_for_location() across every vibration
location in the project; aggregates total events, peak PVS (with the
location it occurred at), last-event timestamp, false-trigger count;
and produces a per-location breakdown sorted by event count.
- backend/routers/project_locations.py: new GET /api/projects/{p}/
vibration_summary endpoint returning an HTML partial (HTMX-friendly,
matches the locations-list HTMX pattern already used on this page).
Frontend:
- templates/partials/projects/vibration_summary.html: new partial with
four KPI tiles (total, peak PVS + linked location + date, last event,
false triggers) and a "Top locations by activity" mini-list showing
the top 5 by event count. Empty-state copy when the project has no
vibration locations yet.
- templates/projects/detail.html: HTMX-load the new summary above the
locations list inside the Vibration tab.
Verified against terra-view-alpha: 24 events across "Loc 1 - 78 poop
street", peak PVS 14.1351 in/s.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 2 of the SFM integration. Adds a "SFM Events" section to the
seismograph unit detail page (/unit/{id}). Every event SFM has for the
serial is shown, with each event annotated by which project/location
assignment window it falls into. Events outside every assignment window
get the "⚠ Unattributed" badge plus a "<N>d before/after <nearest location>"
hint — that's the operator's signal that backdating an assignment (Phase 1
edit-pencil) will absorb the orphan events.
Backend:
- backend/services/sfm_events.py: new events_for_unit() helper. Fetches
all events for the serial via SFM /db/events (one call, ceiling 5000),
loads every UnitAssignment for the unit + resolves MonitoringLocation +
Project names, then annotates each event with attribution or
nearest_assignment (signed delta_days). Bucket filter: all /
attributed / unattributed. Stats always reflect the full event set so
the "Unattributed" KPI tile is meaningful regardless of which bucket
is being viewed.
- backend/routers/units.py: new GET /api/units/{unit_id}/events with
bucket / date-range / false_trigger / limit query params. 404s on
unknown unit_id; returns an empty payload for non-seismograph
device_types so the page can render the section conditionally.
Frontend (templates/unit_detail.html):
- New "SFM Events" section between "Deployment History" and "Timeline",
styled to match the existing card pattern (border-t divider, same
heading weight).
- Hidden by default; revealed only when currentUnit.device_type ===
'seismograph' after the unit data loads.
- Four KPI tiles: Total Events / Unattributed (highlighted amber when
> 0) / Peak PVS / Last Event.
- Filters: Bucket (all|attributed|unattributed), From/To, False
Triggers, Limit, + Refresh.
- Event table with Attribution column. Attributed rows link to the
project/location detail page; unattributed rows are tinted amber
and show "<N>d before/after <nearest location>" with a link to the
nearest location.
- Empty-state copy varies by bucket: e.g. unattributed-with-zero shows
"✅ All events for this unit are attributed to a project/location".
Verified end-to-end against BE11529 (81 events total, 24 attributed,
57 unattributed — all 57 unattributed events emitted within hours of
the assignment start, which means backdating the assignment by a day
would attribute every one of them).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Operators couldn't change a unit's assigned_at / assigned_until after
creating the assignment, so a unit physically deployed in December 2025
but only recorded in terra-view today would show "deployed today" and
all its real events would be invisible on the project's location page.
Backend:
- PATCH /api/projects/{project_id}/assignments/{assignment_id}
Accepts JSON body with optional assigned_at, assigned_until, notes.
- assigned_at is required (cannot be cleared)
- assigned_until can be null to mark active / indefinite
- assigned_until must be after assigned_at
- rejects overlaps with other assignments of the same unit at the
same location (different units overlapping is fine — that's a
legitimate swap window)
- assignment.status flips to "active" when assigned_until is cleared,
"completed" when set
- 404 if the assignment doesn't belong to {project_id} (security)
Frontend (vibration_location_detail.html):
- Pencil icon next to each row in the "Seismographs deployed at this
location" card. Click to open a modal with datetime-local inputs for
From + Until (blank = active) and a Notes textarea. Save reloads the
Events tab so KPI tiles and the event table reflect the new window.
- Helper line under the assignment list explains the workflow:
"Click the pencil to backdate a deployment so historical events get
attributed to this location."
Verified end-to-end against real data: backdating BE11529's assignment
on a vibration location from 2026-04-14 to 2025-12-01 surfaced 10
additional events (24 -> 34) that were previously invisible.
Validation suite (all returning correct HTTP codes):
- assigned_until < assigned_at -> 400
- cross-project assignment_id -> 404
- assigned_at cleared -> 400
- notes-only update -> 200
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Phase 1 of the SFM project/location integration. When viewing a vibration
monitoring location, operators now see the events that were actually
recorded there — fanned out across every seismograph that was ever
assigned to that location (handles mid-project unit swaps).
Backend:
- backend/services/sfm_events.py: new events_for_location() async helper.
Walks UnitAssignment rows for the location (active + closed), intersects
each assignment's [assigned_at, assigned_until] window with the requested
filter, and concurrently queries SFM /db/events for each (serial, window)
pair via httpx.AsyncClient. Unions, sorts newest-first, computes summary
stats (event count, peak PVS + when/who, last event, false-trigger count)
over the full set, and trims to the user's display limit. Over-fetches
per-window (up to 5000) so stats stay accurate even with a small display
limit.
- backend/routers/project_locations.py: new GET endpoint
/api/projects/{project_id}/locations/{location_id}/events. Validates
project/location pairing (404 on mismatch). SLM locations return an
empty payload rather than 404 so the frontend can render gracefully.
Frontend:
- templates/vibration_location_detail.html: new "Events" tab on the
location detail page. KPI tiles (total / peak PVS / last event / false
triggers), "Seismographs deployed at this location" assignment list
(transparency: shows each assignment's date range and contributed event
count), date / false-trigger / limit filters, and the paginated event
table. Lazy-loaded on first tab visit; manual refresh button.
Architectural notes:
- SFM remains the single source of truth for events. No event sync; live
HTTP per page load.
- UnitAssignment is the join key (not MonitoringSession).
- Events whose timestamp falls outside every assignment window are NOT
surfaced here. Those orphan events get a dedicated "Unattributed
events" view on the per-unit detail page in Phase 2.
Out of scope (this commit):
- Phase 2 (per-unit history view) and Phase 3 (project-level roll-up)
reuse this helper but ship separately.
- Phase 4 (deprecating deployment_records) is independent.
- Extracting the event-table JS to a shared file is a follow-up.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- backend/routers/sfm.py: HTTP proxy to SFM backend (localhost:8200),
mirrors the SLMM proxy pattern. SFM_BASE_URL env var for docker-compose.
Catch-all /{path} forwards to SFM root (no /api/ prefix). 60s timeout.
- templates/sfm.html: full SFM dashboard with 5 tabs:
Events (DB listing, filters by serial/date/false-trigger, flag/unflag FT),
Units (known serials + stats, filter events by unit),
Monitor Log (continuous monitoring intervals),
ACH Sessions (call-home history),
Live Device (TCP connect, device info cards, start/stop monitoring,
push project config, download events from device, operation log).
- backend/main.py: import sfm router, include router, add GET /sfm route
- templates/base.html: add SFM Live Data nav link under Seismographs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Introduced a new section for displaying soft-deleted projects.
- Implemented loading of deleted projects via an API call.
- Added restore and permanently delete options for each deleted project.
- Integrated loading of deleted projects when the data tab is shown.
- Updated project creation modal to allow selection of optional modules (Sound and Vibration Monitoring).
- Modified project dashboard and header to display active modules and provide options to add/remove them.
- Enhanced project detail view to dynamically adjust UI based on enabled modules.
- Implemented a new migration script to create a `project_modules` table and seed it based on existing project types.
- Adjusted form submissions to handle module selections and ensure proper API interactions for module management.
- Fix UTC display bug: upload_nrl_data now wraps RNH datetimes with
local_to_utc() before storing, matching patch_session behavior.
Period type and label are derived from local time before conversion.
- Add period_start_hour / period_end_hour to MonitoringSession model
(nullable integers 0–23). Migration: migrate_add_session_period_hours.py
- Update patch_session to accept and store period_start_hour / period_end_hour.
Response now includes both fields.
- Update get_project_sessions to compute "Effective: M/D H:MM AM → M/D H:MM AM"
string from period hours and pass it to session_list.html.
- Rework period edit UI in session_list.html: clicking the period badge now
opens an inline editor with period type selector + start/end hour inputs.
Selecting a period type pre-fills default hours (Day: 7–19, Night: 19–7).
- Wire period hours into _build_location_data_from_sessions: uses
period_start/end_hour when set, falls back to hardcoded defaults.
- RND viewer: inject SESSION_PERIOD_START/END_HOUR from template context.
renderTable() dims rows outside the period window (opacity-40) with a
tooltip; shows "(N in period window)" in the row count.
- New session detail page at /api/projects/{id}/sessions/{id}/detail:
shows breadcrumb, files list with View/Download/Report actions,
editable session info form (label, period type, hours, times).
- Add local_datetime_input Jinja filter for datetime-local input values.
- Monthly calendar view: new get_sessions_calendar endpoint returns
sessions_calendar.html partial; added below sessions list in detail.html.
Color-coded per NRL with legend, HTMX prev/next navigation, session dots
link to detail page.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Replace all UnitAssignment "active" checks from `status == "active"` to
`assigned_until == None` in both project_locations.py and projects.py.
This aligns with the canonical definition: active = no end date set.
(status field is still set in sync, but is no longer the query criterion)
- Add `_require_sound_project()` helper to both routers and call it at the
top of every sound-monitoring-specific endpoint (FTP browser, FTP downloads,
RND file viewer, all Excel report endpoints, combined report wizard,
upload-all, NRL live status, NRL data upload). Vibration projects hitting
these endpoints now receive a clear 400 instead of silently failing or
returning empty results.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Updated dashboard to display allocated units alongside deployed and benched units.
- Introduced a quick-info modal for units, showing detailed information including calibration status, project allocation, and upcoming jobs.
- Enhanced fleet calendar with a new quick-info modal for units, allowing users to view unit details without navigating away.
- Modified devices table to include allocated status and visual indicators for allocated units.
- Added allocated filter option in the roster view for better unit management.
- Implemented backend migration to add 'allocated' and 'allocated_to_project_id' columns to the roster table.
- Updated unit detail view to reflect allocated status and allow for project allocation input.
- Updated reservation list to display estimated units and improved count display.
- Added "Upcoming" status to project dashboard and header with corresponding styles.
- Implemented a dropdown for quick status updates in project header.
- Modified project list compact view to reflect new status labels.
- Updated project overview to include a tab for upcoming projects.
- Added migration script to introduce estimated_units column in job_reservations table.
- Each monitoring location slot can now have a named location (e.g. "North Gate")
- Location names and slot order are persisted and restored in the planner
- Location names display in the expanded reservation card view
- Added "Promote to Project" button that converts a reservation into a
tracked project with monitoring locations and unit assignments pre-filled
Requires DB migration on prod:
ALTER TABLE job_reservation_units ADD COLUMN location_name TEXT;
ALTER TABLE job_reservation_units ADD COLUMN slot_index INTEGER;
serversdown
claude