serversdown
77483c2186
Phase 5b first slice. Surfaces near-duplicate projects (typo variants, abbreviation differences, spacing variations like "SR81" vs "SR 81") as side-by-side pairs the operator can merge with one click. Backend (backend/services/project_tidy.py): - find_duplicate_pairs(db, threshold=0.85) walks all active projects and computes rapidfuzz.WRatio similarity for every pair. Pre-filters too-short normalised names (< 4 chars) to avoid noise. Skips soft-deleted projects. Returns pairs sorted by score desc, then by total content (more assignments → review first). - Each pair carries a suggested merge target with a human-readable reason. Priorities (in order): manual source over parser source, populated project_number, more locations, more assignments, shorter name. Operator can override the suggestion by clicking the OTHER direction button. - O(N^2) over project count. Fine up to ~500 projects. Token-prefix blocking is the obvious next optimisation if it becomes slow. Backend (backend/routers/projects.py): - GET /api/projects/admin/duplicate_pairs?threshold=&max_pairs= returns pairs as JSON for the Tidy page. Frontend (templates/admin/project_tidy.html): - New admin page at /settings/developer/project-tidy. Threshold selector (95% / 90% / 85% / 80%) at the top; rescan button next to it; auto- scans on load. - Each pair card shows side-by-side project summaries (name, project_ number, client, source-badge, location/assignment counts) with the suggested target visually highlighted (orange border). Three buttons: "Merge A → B", "Merge B → A", "Not a dup" (hide locally). - Click-to-merge opens a native confirm with the preview totals (assignments/sessions/data files moving, consolidations) — same data the project_header.html merge modal shows. On confirm, hits the existing /merge_into endpoint and re-scans automatically. - Source badges distinguish parser-created (`metadata_backfill`) from manual projects — at a glance the operator can see "this duplicate is parser-generated; safe to merge into the manual one". Frontend (templates/admin/metadata_backfill.html): - Apply-result handling now surfaces failed[] cluster reasons in a dedicated failure panel (bottom-left, dismissable). Previously a 200 OK with all-failures showed a misleading "1 cluster applied" success toast because the count and the failure list weren't being reconciled. This bit us during the DB-revert recovery earlier — the project_modules table was missing, every apply silently rolled back, user saw success toasts. Fixed. Smoke-verified against current state (10K events, 9 projects, post- merge): tool correctly finds 0 pairs at threshold 0.85 (data is clean), 1 false-positive at 0.70 (two unrelated projects sharing the token "81" — example of why the 0.85 default is correct). Settings link added under Developer → Project Tidy. Phase 5c (swap-detection daily background job + notification inbox) remains deferred to the next session. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
693 lines
38 KiB
HTML
693 lines
38 KiB
HTML
{% extends "base.html" %}
|
|
|
|
{% block title %}Metadata Backfill - Seismo Fleet Manager{% endblock %}
|
|
|
|
{% block content %}
|
|
<!-- Breadcrumb -->
|
|
<div class="mb-6">
|
|
<nav class="flex items-center space-x-2 text-sm">
|
|
<a href="/settings" class="text-seismo-orange hover:text-seismo-navy flex items-center">
|
|
<svg class="w-4 h-4 mr-1" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 19l-7-7 7-7"></path>
|
|
</svg>
|
|
Settings
|
|
</a>
|
|
<svg class="w-4 h-4 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 5l7 7-7 7"></path>
|
|
</svg>
|
|
<span class="text-gray-900 dark:text-white font-medium">Metadata Backfill</span>
|
|
</nav>
|
|
</div>
|
|
|
|
<!-- Header -->
|
|
<div class="mb-6">
|
|
<h1 class="text-3xl font-bold text-gray-900 dark:text-white">Backfill from event metadata</h1>
|
|
<p class="text-gray-600 dark:text-gray-400 mt-1">
|
|
Auto-create projects, locations, and unit assignments from operator-typed metadata on Blastware events.
|
|
</p>
|
|
</div>
|
|
|
|
<!-- Summary card (populated after scan) -->
|
|
<div id="summary-card" class="bg-white dark:bg-slate-800 rounded-xl shadow-lg p-6 mb-6">
|
|
<div id="summary-initial">
|
|
<div class="text-center py-8">
|
|
<svg class="w-16 h-16 mx-auto mb-4 text-gray-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M21 21l-6-6m2-5a7 7 0 11-14 0 7 7 0 0114 0z"></path>
|
|
</svg>
|
|
<h2 class="text-xl font-semibold text-gray-900 dark:text-white mb-2">Scan SFM events</h2>
|
|
<p class="text-gray-500 dark:text-gray-400 text-sm mb-6 max-w-xl mx-auto">
|
|
Reads all events from SFM, clusters them by serial & time, matches the
|
|
operator-typed metadata against your existing projects, and proposes
|
|
<strong>Project</strong> / <strong>Location</strong> / <strong>UnitAssignment</strong>
|
|
chains to create.
|
|
</p>
|
|
<button onclick="runScan(false)"
|
|
id="initial-scan-btn"
|
|
class="px-6 py-3 bg-seismo-orange hover:bg-seismo-navy text-white rounded-lg font-medium transition-colors">
|
|
↻ Run scan
|
|
</button>
|
|
</div>
|
|
</div>
|
|
|
|
<div id="summary-results" class="hidden">
|
|
<div class="flex items-start justify-between mb-4 flex-wrap gap-3">
|
|
<div>
|
|
<h2 class="text-xl font-semibold text-gray-900 dark:text-white">Scan summary</h2>
|
|
<p id="summary-scanned-at" class="text-xs text-gray-500 dark:text-gray-400 mt-1"></p>
|
|
</div>
|
|
<button onclick="runScan(true)"
|
|
class="px-3 py-1.5 text-sm border border-gray-300 dark:border-gray-600 hover:bg-gray-50 dark:hover:bg-gray-700 text-gray-700 dark:text-gray-300 rounded-lg">
|
|
↻ Re-scan
|
|
</button>
|
|
</div>
|
|
|
|
<!-- KPI tiles -->
|
|
<div class="grid grid-cols-2 md:grid-cols-4 gap-3 mb-4">
|
|
<div class="bg-gray-50 dark:bg-slate-900/50 rounded-lg p-3 flex flex-col">
|
|
<span class="text-xs text-gray-500 dark:text-gray-400 uppercase tracking-wider">Events scanned</span>
|
|
<span id="kpi-scanned" class="text-2xl font-bold text-gray-900 dark:text-white mt-1">—</span>
|
|
</div>
|
|
<div class="bg-gray-50 dark:bg-slate-900/50 rounded-lg p-3 flex flex-col">
|
|
<span class="text-xs text-gray-500 dark:text-gray-400 uppercase tracking-wider">Already attributed</span>
|
|
<span id="kpi-already" class="text-2xl font-bold text-gray-900 dark:text-white mt-1">—</span>
|
|
<span class="text-xs text-gray-500 dark:text-gray-400 mt-1">inside existing assignments</span>
|
|
</div>
|
|
<div class="bg-gray-50 dark:bg-slate-900/50 rounded-lg p-3 flex flex-col">
|
|
<span class="text-xs text-gray-500 dark:text-gray-400 uppercase tracking-wider">Pending review</span>
|
|
<span id="kpi-pending" class="text-2xl font-bold text-gray-900 dark:text-white mt-1">—</span>
|
|
<span class="text-xs text-gray-500 dark:text-gray-400 mt-1">clusters to attribute</span>
|
|
</div>
|
|
<div class="bg-gray-50 dark:bg-slate-900/50 rounded-lg p-3 flex flex-col">
|
|
<span class="text-xs text-gray-500 dark:text-gray-400 uppercase tracking-wider">Conflicts</span>
|
|
<span id="kpi-conflicts" class="text-2xl font-bold text-gray-900 dark:text-white mt-1">—</span>
|
|
<span class="text-xs text-gray-500 dark:text-gray-400 mt-1">need manual reconciliation</span>
|
|
</div>
|
|
</div>
|
|
|
|
<!-- One-click bulk apply -->
|
|
<div id="bulk-apply-card" class="bg-orange-50 dark:bg-orange-900/20 border border-orange-200 dark:border-orange-800 rounded-lg p-4 mb-4 hidden">
|
|
<div class="flex items-start gap-3">
|
|
<svg class="w-6 h-6 text-seismo-orange shrink-0 mt-0.5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13 10V3L4 14h7v7l9-11h-7z"></path>
|
|
</svg>
|
|
<div class="flex-1">
|
|
<h3 class="font-semibold text-gray-900 dark:text-white mb-1">
|
|
Bulk-apply <span id="bulk-applicable-count">0</span> high-confidence cluster(s)
|
|
</h3>
|
|
<p class="text-sm text-gray-700 dark:text-gray-300 mb-3">
|
|
Apply every cluster scored <strong>high confidence</strong> with no blocking conflicts.
|
|
Will create <span id="bulk-stats" class="font-medium">—</span>.
|
|
Medium and low confidence clusters remain in the list below for individual review.
|
|
</p>
|
|
<button onclick="applyBulkHighConfidence()"
|
|
class="px-5 py-2 bg-seismo-orange hover:bg-seismo-navy text-white rounded-lg font-medium transition-colors">
|
|
Apply all high-confidence
|
|
</button>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<p class="text-xs text-gray-500 dark:text-gray-400 italic">
|
|
Each cluster below shows the operator-typed metadata, what would be created or matched, and the proposed
|
|
assignment date window. Click <em>Apply</em> to attribute that cluster, <em>Skip</em> to ignore it (won't reappear),
|
|
or <em>Edit</em> to rename before applying.
|
|
</p>
|
|
</div>
|
|
</div>
|
|
|
|
<!-- Cluster list -->
|
|
<div id="cluster-list" class="space-y-3"></div>
|
|
|
|
<!-- Apply progress toast -->
|
|
<div id="apply-toast" class="hidden fixed bottom-6 right-6 bg-white dark:bg-slate-800 rounded-xl shadow-2xl border border-gray-200 dark:border-gray-700 p-4 z-50 max-w-md">
|
|
<div class="flex items-center gap-3">
|
|
<div id="toast-icon" class="shrink-0">
|
|
<div class="animate-spin rounded-full h-6 w-6 border-b-2 border-seismo-orange"></div>
|
|
</div>
|
|
<div class="flex-1">
|
|
<p id="toast-message" class="text-sm font-medium text-gray-900 dark:text-white">Applying…</p>
|
|
<p id="toast-sub" class="text-xs text-gray-500 dark:text-gray-400 mt-0.5"></p>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
<!-- Shared event-detail modal (Preview event button uses it) -->
|
|
{% include 'partials/event_detail_modal.html' %}
|
|
<script src="/static/event-modal.js"></script>
|
|
|
|
<script>
|
|
let _scanData = null;
|
|
|
|
function _esc(s) {
|
|
if (s == null) return '';
|
|
return String(s).replace(/&/g, '&').replace(/</g, '<').replace(/>/g, '>').replace(/"/g, '"');
|
|
}
|
|
|
|
function _fmtDate(iso) {
|
|
if (!iso) return '—';
|
|
return iso.slice(0, 10);
|
|
}
|
|
|
|
function _fmtDateTime(iso) {
|
|
if (!iso) return '—';
|
|
return iso.slice(0, 19).replace('T', ' ');
|
|
}
|
|
|
|
function _confidenceBadge(c) {
|
|
const map = {
|
|
high: { cls: 'bg-green-100 text-green-800 dark:bg-green-900/30 dark:text-green-300', icon: '🟢', label: 'high' },
|
|
medium: { cls: 'bg-amber-100 text-amber-800 dark:bg-amber-900/30 dark:text-amber-300', icon: '🟡', label: 'medium' },
|
|
low: { cls: 'bg-red-100 text-red-800 dark:bg-red-900/30 dark:text-red-300', icon: '🔴', label: 'low' },
|
|
};
|
|
const e = map[c] || map.low;
|
|
return `<span class="px-2 py-0.5 rounded text-xs font-medium ${e.cls}">${e.icon} ${e.label}</span>`;
|
|
}
|
|
|
|
function _matchPill(match, score, suggestedName, existingName) {
|
|
if (match === 'exact') {
|
|
return `<span class="font-medium text-green-700 dark:text-green-400">✓ Matches existing: <em>${_esc(existingName || suggestedName)}</em></span>`;
|
|
}
|
|
if (match === 'fuzzy') {
|
|
return `<span class="font-medium text-amber-700 dark:text-amber-400">≈ Fuzzy match (${(score*100).toFixed(0)}%): <em>${_esc(existingName)}</em></span>
|
|
<span class="text-xs text-gray-500 dark:text-gray-400 ml-1">(your value: "${_esc(suggestedName)}")</span>`;
|
|
}
|
|
if (match === 'ambiguous') {
|
|
return `<span class="font-medium text-yellow-700 dark:text-yellow-400">? Ambiguous — multiple matches</span>`;
|
|
}
|
|
return `<span class="font-medium text-seismo-orange">+ Create new: <em>${_esc(suggestedName)}</em></span>`;
|
|
}
|
|
|
|
// Compact "hint" line under each typeahead input, showing what the parser
|
|
// thinks the current value will do (match existing vs create new).
|
|
function _matchHint(match, score, existingName, suggestedName) {
|
|
if (match === 'exact') {
|
|
return `<span class="text-green-700 dark:text-green-400">✓ matches existing</span>`;
|
|
}
|
|
if (match === 'fuzzy') {
|
|
return `<span class="text-amber-700 dark:text-amber-400">≈ fuzzy match to "${_esc(existingName)}" (${(score*100).toFixed(0)}%)</span>`;
|
|
}
|
|
if (match === 'ambiguous') {
|
|
return `<span class="text-yellow-700 dark:text-yellow-400">? ambiguous — pick from dropdown</span>`;
|
|
}
|
|
return `<span class="text-seismo-orange">+ will create new</span>`;
|
|
}
|
|
|
|
// ── Typeahead ────────────────────────────────────────────────────────────
|
|
// Per-cluster project + location inputs with debounced typeahead search.
|
|
// Selecting a result writes the existing entity's id into the hidden
|
|
// project_id / location_id input; clearing-and-typing falls back to
|
|
// "create new" semantics.
|
|
|
|
let _typeaheadDebounce = null;
|
|
|
|
function onTypeaheadInput(e, fieldKind) {
|
|
// fieldKind ∈ {'project', 'location'}
|
|
const inp = e.target;
|
|
const cid = inp.dataset.clusterId;
|
|
// Clear the "id" hidden input — operator is typing freely now.
|
|
const hidden = document.querySelector(`input[type="hidden"][data-cluster-id="${cid}"][data-field="${fieldKind}_id"]`);
|
|
if (hidden) hidden.value = '';
|
|
// Debounce the search.
|
|
if (_typeaheadDebounce) clearTimeout(_typeaheadDebounce);
|
|
_typeaheadDebounce = setTimeout(() => _fetchTypeahead(inp, fieldKind), 150);
|
|
}
|
|
|
|
function onTypeaheadFocus(e, fieldKind) {
|
|
_fetchTypeahead(e.target, fieldKind);
|
|
}
|
|
|
|
function onTypeaheadBlur(e) {
|
|
// Delayed hide so a click on the dropdown can register first.
|
|
const dropdown = e.target.parentElement.querySelector('.typeahead-dropdown');
|
|
if (dropdown) {
|
|
setTimeout(() => dropdown.classList.add('hidden'), 150);
|
|
}
|
|
}
|
|
|
|
async function _fetchTypeahead(input, fieldKind) {
|
|
const dropdown = input.parentElement.querySelector('.typeahead-dropdown');
|
|
if (!dropdown) return;
|
|
const q = input.value.trim();
|
|
const cid = input.dataset.clusterId;
|
|
|
|
let url;
|
|
if (fieldKind === 'project') {
|
|
url = `/api/admin/metadata_backfill/projects_search?q=${encodeURIComponent(q)}`;
|
|
} else {
|
|
// For locations, scope to the currently-chosen project (if any).
|
|
const projectIdInput = document.querySelector(`input[type="hidden"][data-cluster-id="${cid}"][data-field="project_id"]`);
|
|
const projectId = projectIdInput ? projectIdInput.value : '';
|
|
if (!projectId) {
|
|
// Operator hasn't picked an existing project — there are no
|
|
// existing locations to match against (location is implicitly
|
|
// "create new" inside a new project).
|
|
dropdown.innerHTML = `<div class="px-3 py-2 text-sm text-gray-500 dark:text-gray-400 italic">
|
|
${q ? `+ Will create new: <strong>"${_esc(q)}"</strong>` : 'Pick a project first, or type a new location name.'}
|
|
</div>`;
|
|
dropdown.classList.remove('hidden');
|
|
return;
|
|
}
|
|
url = `/api/admin/metadata_backfill/locations_search?project_id=${encodeURIComponent(projectId)}&q=${encodeURIComponent(q)}`;
|
|
}
|
|
|
|
let data;
|
|
try {
|
|
const r = await fetch(url);
|
|
if (!r.ok) throw new Error('HTTP ' + r.status);
|
|
data = await r.json();
|
|
} catch (err) {
|
|
dropdown.innerHTML = `<div class="px-3 py-2 text-sm text-red-500">Search failed: ${_esc(err.message)}</div>`;
|
|
dropdown.classList.remove('hidden');
|
|
return;
|
|
}
|
|
|
|
const items = [];
|
|
for (const m of (data.matches || [])) {
|
|
items.push({ kind: 'match', payload: m });
|
|
}
|
|
if (data.create_new && data.create_new.label) {
|
|
items.push({ kind: 'create_new', label: data.create_new.label, name: q });
|
|
}
|
|
|
|
if (items.length === 0) {
|
|
dropdown.innerHTML = `<div class="px-3 py-2 text-sm text-gray-500 dark:text-gray-400 italic">No matches. Type to create.</div>`;
|
|
dropdown.classList.remove('hidden');
|
|
return;
|
|
}
|
|
|
|
dropdown.innerHTML = items.map((it, idx) => {
|
|
if (it.kind === 'match') {
|
|
const m = it.payload;
|
|
const scoreBadge = m.score >= 0.99
|
|
? '<span class="text-xs text-green-600 dark:text-green-400 ml-2">exact</span>'
|
|
: `<span class="text-xs text-gray-500 dark:text-gray-400 ml-2">${(m.score*100).toFixed(0)}%</span>`;
|
|
const meta = [];
|
|
if (fieldKind === 'project') {
|
|
if (m.project_number) meta.push(_esc(m.project_number));
|
|
if (m.client_name) meta.push(_esc(m.client_name));
|
|
if (m.location_count > 0) meta.push(`${m.location_count} location${m.location_count === 1 ? '' : 's'}`);
|
|
} else {
|
|
if (m.address) meta.push(_esc(m.address));
|
|
}
|
|
const metaLine = meta.length ? `<div class="text-xs text-gray-500 dark:text-gray-400">${meta.join(' · ')}</div>` : '';
|
|
return `<button type="button"
|
|
onmousedown="event.preventDefault()"
|
|
onclick="onTypeaheadPick(event, '${_esc(input.dataset.clusterId)}', '${fieldKind}', '${_esc(m.id)}', ${JSON.stringify(m.name)})"
|
|
class="w-full text-left px-3 py-2 hover:bg-gray-50 dark:hover:bg-slate-700 border-b border-gray-100 dark:border-gray-700 last:border-b-0">
|
|
<div class="text-sm font-medium text-gray-900 dark:text-white">${_esc(m.name)}${scoreBadge}</div>
|
|
${metaLine}
|
|
</button>`;
|
|
}
|
|
return `<button type="button"
|
|
onmousedown="event.preventDefault()"
|
|
onclick="onTypeaheadPick(event, '${_esc(input.dataset.clusterId)}', '${fieldKind}', '', ${JSON.stringify(it.name)})"
|
|
class="w-full text-left px-3 py-2 hover:bg-orange-50 dark:hover:bg-orange-900/20 border-t border-gray-200 dark:border-gray-700 text-seismo-orange font-medium text-sm">
|
|
+ ${_esc(it.label)}
|
|
</button>`;
|
|
}).join('');
|
|
dropdown.classList.remove('hidden');
|
|
}
|
|
|
|
function onTypeaheadPick(event, clusterId, fieldKind, entityId, name) {
|
|
// entityId is empty string for "create new", or a UUID for matched existing.
|
|
const inputs = document.querySelectorAll(`input[data-cluster-id="${clusterId}"]`);
|
|
let textInput = null;
|
|
let idInput = null;
|
|
inputs.forEach(i => {
|
|
if (i.dataset.field === fieldKind) textInput = i;
|
|
if (i.dataset.field === fieldKind + '_id') idInput = i;
|
|
});
|
|
if (textInput) textInput.value = name;
|
|
if (idInput) idInput.value = entityId;
|
|
// Hide this dropdown.
|
|
const dropdown = textInput.parentElement.querySelector('.typeahead-dropdown');
|
|
if (dropdown) dropdown.classList.add('hidden');
|
|
|
|
// If operator just picked a NEW project, clear the location id (forces
|
|
// operator to pick a location under the new project rather than leaving
|
|
// a stale id from another project).
|
|
if (fieldKind === 'project') {
|
|
const locId = document.querySelector(`input[type="hidden"][data-cluster-id="${clusterId}"][data-field="location_id"]`);
|
|
if (locId) locId.value = '';
|
|
}
|
|
}
|
|
|
|
async function runScan(force) {
|
|
const initial = document.getElementById('summary-initial');
|
|
const results = document.getElementById('summary-results');
|
|
const list = document.getElementById('cluster-list');
|
|
|
|
initial.classList.add('hidden');
|
|
results.classList.remove('hidden');
|
|
list.innerHTML = '<div class="text-center py-12 text-gray-500 dark:text-gray-400"><div class="animate-spin rounded-full h-8 w-8 border-b-2 border-seismo-orange mx-auto mb-3"></div>Scanning events…</div>';
|
|
|
|
document.getElementById('kpi-scanned').textContent = '…';
|
|
document.getElementById('kpi-already').textContent = '…';
|
|
document.getElementById('kpi-pending').textContent = '…';
|
|
document.getElementById('kpi-conflicts').textContent = '…';
|
|
document.getElementById('bulk-apply-card').classList.add('hidden');
|
|
|
|
try {
|
|
const r = await fetch('/api/admin/metadata_backfill/scan' + (force ? '?force=true' : ''));
|
|
if (!r.ok) throw new Error('HTTP ' + r.status);
|
|
_scanData = await r.json();
|
|
} catch (e) {
|
|
list.innerHTML = `<div class="bg-white dark:bg-slate-800 rounded-xl shadow-lg p-6 text-center text-red-500">Scan failed: ${_esc(e.message)}</div>`;
|
|
return;
|
|
}
|
|
|
|
document.getElementById('kpi-scanned').textContent = _scanData.scanned_event_count.toLocaleString();
|
|
document.getElementById('kpi-already').textContent = _scanData.already_attributed.toLocaleString();
|
|
document.getElementById('kpi-pending').textContent = _scanData.pending_count.toLocaleString();
|
|
document.getElementById('kpi-conflicts').textContent = _scanData.blocking_conflict_count.toLocaleString();
|
|
document.getElementById('summary-scanned-at').textContent =
|
|
'Scanned ' + new Date(_scanData.scanned_at * 1000).toLocaleString();
|
|
|
|
// Configure bulk-apply card.
|
|
const highApplicable = _scanData.by_confidence.high.filter(s => !s.blocking_conflict);
|
|
const newProjects = new Set(), newLocations = new Set();
|
|
for (const s of highApplicable) {
|
|
if (s.project_match === 'create_new') newProjects.add(s.project_suggested_name.toLowerCase());
|
|
if (s.location_match === 'create_new') newLocations.add(s.location_suggested_name.toLowerCase());
|
|
}
|
|
if (highApplicable.length > 0) {
|
|
document.getElementById('bulk-apply-card').classList.remove('hidden');
|
|
document.getElementById('bulk-applicable-count').textContent = highApplicable.length;
|
|
const parts = [];
|
|
if (newProjects.size > 0) parts.push(`${newProjects.size} project${newProjects.size === 1 ? '' : 's'}`);
|
|
if (newLocations.size > 0) parts.push(`${newLocations.size} location${newLocations.size === 1 ? '' : 's'}`);
|
|
parts.push(`${highApplicable.length} assignment${highApplicable.length === 1 ? '' : 's'}`);
|
|
document.getElementById('bulk-stats').textContent = parts.join(' · ');
|
|
}
|
|
|
|
renderClusterList();
|
|
}
|
|
|
|
function renderClusterList() {
|
|
const list = document.getElementById('cluster-list');
|
|
const all = [
|
|
..._scanData.by_confidence.high,
|
|
..._scanData.by_confidence.medium,
|
|
..._scanData.by_confidence.low,
|
|
];
|
|
|
|
if (all.length === 0) {
|
|
list.innerHTML = `<div class="bg-white dark:bg-slate-800 rounded-xl shadow-lg p-8 text-center">
|
|
<svg class="w-16 h-16 mx-auto mb-4 text-green-500" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
|
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 12l2 2 4-4m6 2a9 9 0 11-18 0 9 9 0 0118 0z"></path>
|
|
</svg>
|
|
<h3 class="text-lg font-semibold text-gray-900 dark:text-white mb-1">✅ All caught up</h3>
|
|
<p class="text-gray-500 dark:text-gray-400">Every event in SFM is either attributed to an existing assignment or has been skipped.</p>
|
|
</div>`;
|
|
return;
|
|
}
|
|
|
|
list.innerHTML = all.map(_renderCluster).join('');
|
|
}
|
|
|
|
function _renderCluster(s) {
|
|
const spanDays = (new Date(s.last_event_ts) - new Date(s.first_event_ts)) / 86400000;
|
|
const consistencyNote = s.metadata_consistency < 1.0
|
|
? `<span class="ml-2 text-xs text-amber-600 dark:text-amber-400" title="Some events in this cluster have slightly different metadata — possibly a typo or mid-stream change.">⚠ ${(s.metadata_consistency*100).toFixed(0)}% consistent</span>`
|
|
: '';
|
|
|
|
const blockingBanner = s.blocking_conflict
|
|
? `<div class="bg-red-50 dark:bg-red-900/20 border border-red-200 dark:border-red-800 rounded-lg p-3 mt-3 text-sm text-red-800 dark:text-red-300">
|
|
<strong>⚠ Blocking conflict.</strong>
|
|
${s.conflicts.map(c => `Unit ${_esc(s.serial)} is already assigned to <em>${_esc(c.other_project_name)} / ${_esc(c.other_location_name)}</em> during this window.`).join(' ')}
|
|
Resolve manually before this cluster can be applied.
|
|
</div>`
|
|
: '';
|
|
|
|
const orphanInputs = s.is_blank_meta
|
|
? `<div class="bg-amber-50 dark:bg-amber-900/20 border border-amber-200 dark:border-amber-800 rounded-lg p-3 mt-3">
|
|
<p class="text-sm text-amber-800 dark:text-amber-300 mb-2"><strong>⚠ Blank metadata.</strong> Operator didn't type project / location for these events. Fill in manually:</p>
|
|
<div class="grid grid-cols-2 gap-2">
|
|
<input type="text" placeholder="Project name" data-cluster-id="${_esc(s.cluster_id)}" data-field="project_name"
|
|
class="px-2 py-1 text-sm border border-amber-300 dark:border-amber-700 rounded bg-white dark:bg-slate-700 text-gray-900 dark:text-white">
|
|
<input type="text" placeholder="Location name" data-cluster-id="${_esc(s.cluster_id)}" data-field="location_name"
|
|
class="px-2 py-1 text-sm border border-amber-300 dark:border-amber-700 rounded bg-white dark:bg-slate-700 text-gray-900 dark:text-white">
|
|
</div>
|
|
</div>`
|
|
: '';
|
|
|
|
return `<div class="bg-white dark:bg-slate-800 rounded-xl shadow-lg p-4" data-cluster-id="${_esc(s.cluster_id)}">
|
|
<div class="flex items-start justify-between gap-3 mb-3 flex-wrap">
|
|
<div class="flex-1 min-w-0">
|
|
<div class="flex items-center gap-2 mb-1 flex-wrap">
|
|
${_confidenceBadge(s.confidence)}
|
|
<a href="/unit/${_esc(s.serial)}" class="font-mono font-semibold text-seismo-orange hover:text-seismo-navy">${_esc(s.serial)}</a>
|
|
<span class="text-sm text-gray-600 dark:text-gray-400">${_fmtDate(s.first_event_ts)} → ${_fmtDate(s.last_event_ts)}</span>
|
|
<span class="text-xs text-gray-500 dark:text-gray-400">(${s.event_count} event${s.event_count === 1 ? '' : 's'}, ${spanDays.toFixed(0)}d span)</span>
|
|
${consistencyNote}
|
|
</div>
|
|
<div class="text-sm text-gray-700 dark:text-gray-300 mt-2 space-y-2">
|
|
<!-- Project typeahead -->
|
|
<div class="flex items-start gap-2">
|
|
<span class="text-gray-500 dark:text-gray-400 w-24 shrink-0 pt-1.5">Project:</span>
|
|
<div class="flex-1 relative">
|
|
<input type="text"
|
|
class="w-full px-2 py-1 text-sm border border-gray-300 dark:border-gray-600 rounded bg-white dark:bg-slate-700 text-gray-900 dark:text-white"
|
|
value="${_esc(s.project_existing_name || s.project_suggested_name)}"
|
|
data-cluster-id="${_esc(s.cluster_id)}"
|
|
data-field="project"
|
|
data-initial-project-id="${_esc(s.project_existing_id || '')}"
|
|
placeholder="Type to search or create…"
|
|
oninput="onTypeaheadInput(event, 'project')"
|
|
onfocus="onTypeaheadFocus(event, 'project')"
|
|
onblur="onTypeaheadBlur(event)"
|
|
autocomplete="off">
|
|
<input type="hidden" data-cluster-id="${_esc(s.cluster_id)}" data-field="project_id" value="${_esc(s.project_existing_id || '')}">
|
|
<div class="typeahead-dropdown hidden absolute z-20 left-0 right-0 mt-1 bg-white dark:bg-slate-800 border border-gray-200 dark:border-gray-700 rounded-lg shadow-lg max-h-64 overflow-y-auto"></div>
|
|
<div class="text-xs mt-0.5">${_matchHint(s.project_match, s.project_match_score, s.project_existing_name, s.project_suggested_name)}</div>
|
|
${(s.project_root && s.project_raw && s.project_root !== s.project_raw)
|
|
? `<div class="text-xs text-gray-500 dark:text-gray-400 mt-0.5">↳ stripped trailing "Loc N" suffix; operator typed: <em>"${_esc(s.project_raw)}"</em></div>`
|
|
: ''}
|
|
</div>
|
|
</div>
|
|
|
|
<!-- Location typeahead -->
|
|
<div class="flex items-start gap-2">
|
|
<span class="text-gray-500 dark:text-gray-400 w-24 shrink-0 pt-1.5">Location:</span>
|
|
<div class="flex-1 relative">
|
|
<input type="text"
|
|
class="w-full px-2 py-1 text-sm border border-gray-300 dark:border-gray-600 rounded bg-white dark:bg-slate-700 text-gray-900 dark:text-white"
|
|
value="${_esc(s.location_existing_name || s.location_suggested_name)}"
|
|
data-cluster-id="${_esc(s.cluster_id)}"
|
|
data-field="location"
|
|
data-initial-location-id="${_esc(s.location_existing_id || '')}"
|
|
placeholder="Type to search or create…"
|
|
oninput="onTypeaheadInput(event, 'location')"
|
|
onfocus="onTypeaheadFocus(event, 'location')"
|
|
onblur="onTypeaheadBlur(event)"
|
|
autocomplete="off">
|
|
<input type="hidden" data-cluster-id="${_esc(s.cluster_id)}" data-field="location_id" value="${_esc(s.location_existing_id || '')}">
|
|
<div class="typeahead-dropdown hidden absolute z-20 left-0 right-0 mt-1 bg-white dark:bg-slate-800 border border-gray-200 dark:border-gray-700 rounded-lg shadow-lg max-h-64 overflow-y-auto"></div>
|
|
<div class="text-xs mt-0.5">${_matchHint(s.location_match, s.location_match_score, s.location_existing_name, s.location_suggested_name)}</div>
|
|
</div>
|
|
</div>
|
|
|
|
<div><span class="text-gray-500 dark:text-gray-400 w-24 inline-block">Assignment:</span> ${_fmtDateTime(s.proposed_assigned_at)} → ${s.proposed_assigned_until ? _fmtDateTime(s.proposed_assigned_until) : '<span class="text-green-700 dark:text-green-400 font-medium">present (active)</span>'}</div>
|
|
${s.client_raw ? `<div><span class="text-gray-500 dark:text-gray-400 w-24 inline-block">Client:</span> <em>${_esc(s.client_raw)}</em></div>` : ''}
|
|
</div>
|
|
${blockingBanner}
|
|
${orphanInputs}
|
|
</div>
|
|
<div class="flex flex-col gap-2 shrink-0">
|
|
<button onclick="showEventDetail('${_esc(s.sample_event_id)}')"
|
|
class="px-3 py-1.5 text-xs border border-gray-300 dark:border-gray-600 hover:bg-gray-50 dark:hover:bg-gray-700 text-gray-700 dark:text-gray-300 rounded">
|
|
Preview event
|
|
</button>
|
|
${s.blocking_conflict
|
|
? `<button disabled class="px-3 py-1.5 text-xs bg-gray-100 dark:bg-gray-800 text-gray-400 rounded cursor-not-allowed">Apply</button>`
|
|
: `<button onclick="applyOne('${_esc(s.cluster_id)}')"
|
|
class="px-3 py-1.5 text-xs bg-seismo-orange hover:bg-seismo-navy text-white rounded font-medium">
|
|
Apply
|
|
</button>`}
|
|
<button onclick="skipOne('${_esc(s.cluster_id)}')"
|
|
class="px-3 py-1.5 text-xs border border-gray-300 dark:border-gray-600 hover:bg-gray-50 dark:hover:bg-gray-700 text-gray-700 dark:text-gray-300 rounded">
|
|
Skip
|
|
</button>
|
|
</div>
|
|
</div>
|
|
</div>`;
|
|
}
|
|
|
|
function _gatherOverrides(clusterIds) {
|
|
// Per-cluster overrides sent to /apply. The backend understands four
|
|
// keys per cluster: project_id, project_name, location_id, location_name.
|
|
// We emit project_id+location_id when the operator picked from the
|
|
// typeahead dropdown; we emit project_name+location_name when they
|
|
// typed a free-form value (no id selected) that differs from the
|
|
// parser's original suggestion.
|
|
const overrides = {};
|
|
for (const cid of clusterIds) {
|
|
const inputs = document.querySelectorAll(`input[data-cluster-id="${cid}"]`);
|
|
if (inputs.length === 0) continue;
|
|
const o = {};
|
|
let projectText = null, projectId = null;
|
|
let locationText = null, locationId = null;
|
|
// Old-style flat fields (kept for blank-meta cluster inputs which
|
|
// use data-field="project_name" / "location_name").
|
|
let projectNameRaw = null, locationNameRaw = null;
|
|
inputs.forEach(i => {
|
|
const v = (i.value || '').trim();
|
|
const f = i.dataset.field;
|
|
if (f === 'project') projectText = v;
|
|
else if (f === 'project_id') projectId = v;
|
|
else if (f === 'location') locationText = v;
|
|
else if (f === 'location_id') locationId = v;
|
|
else if (f === 'project_name') projectNameRaw = v;
|
|
else if (f === 'location_name') locationNameRaw = v;
|
|
});
|
|
|
|
if (projectId) {
|
|
o.project_id = projectId;
|
|
} else if (projectText) {
|
|
o.project_name = projectText;
|
|
} else if (projectNameRaw) {
|
|
o.project_name = projectNameRaw;
|
|
}
|
|
|
|
if (locationId) {
|
|
o.location_id = locationId;
|
|
} else if (locationText) {
|
|
o.location_name = locationText;
|
|
} else if (locationNameRaw) {
|
|
o.location_name = locationNameRaw;
|
|
}
|
|
|
|
if (Object.keys(o).length > 0) overrides[cid] = o;
|
|
}
|
|
return overrides;
|
|
}
|
|
|
|
function _showToast(message, sub, kind) {
|
|
const toast = document.getElementById('apply-toast');
|
|
const icon = document.getElementById('toast-icon');
|
|
document.getElementById('toast-message').textContent = message;
|
|
document.getElementById('toast-sub').textContent = sub || '';
|
|
if (kind === 'success') {
|
|
icon.innerHTML = '<svg class="w-6 h-6 text-green-500" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M5 13l4 4L19 7"></path></svg>';
|
|
} else if (kind === 'error') {
|
|
icon.innerHTML = '<svg class="w-6 h-6 text-red-500" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 9v2m0 4h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z"></path></svg>';
|
|
} else {
|
|
icon.innerHTML = '<div class="animate-spin rounded-full h-6 w-6 border-b-2 border-seismo-orange"></div>';
|
|
}
|
|
toast.classList.remove('hidden');
|
|
}
|
|
|
|
function _hideToast(after) {
|
|
setTimeout(() => document.getElementById('apply-toast').classList.add('hidden'), after || 3000);
|
|
}
|
|
|
|
async function _apply(clusterIds) {
|
|
if (clusterIds.length === 0) return;
|
|
_showToast(`Applying ${clusterIds.length} cluster${clusterIds.length === 1 ? '' : 's'}…`);
|
|
try {
|
|
const r = await fetch('/api/admin/metadata_backfill/apply', {
|
|
method: 'POST',
|
|
headers: {'Content-Type': 'application/json'},
|
|
body: JSON.stringify({
|
|
cluster_ids: clusterIds,
|
|
overrides: _gatherOverrides(clusterIds),
|
|
}),
|
|
});
|
|
if (!r.ok) throw new Error('HTTP ' + r.status);
|
|
const d = await r.json();
|
|
const failedCount = (d.failed || []).length;
|
|
|
|
// Three states:
|
|
// total success — applied N, no failures → green toast, 4s
|
|
// partial — applied N, M failures → red toast + modal listing reasons
|
|
// total failure — applied 0, failures → red toast + modal
|
|
if (failedCount === 0) {
|
|
const sub = `${d.applied} applied · ${d.project_ids_created.length} new project(s) · ${d.location_ids_created.length} new location(s)`;
|
|
_showToast(`${d.applied} cluster${d.applied === 1 ? '' : 's'} applied`, sub, 'success');
|
|
_hideToast(4000);
|
|
} else {
|
|
const title = d.applied > 0
|
|
? `${d.applied} applied, ${failedCount} failed`
|
|
: `Apply failed — ${failedCount} cluster${failedCount === 1 ? '' : 's'} could not be applied`;
|
|
_showToast(title, 'See the details panel.', 'error');
|
|
_hideToast(6000);
|
|
_showFailureDetails(d.failed);
|
|
}
|
|
await runScan(true); // refresh
|
|
} catch (e) {
|
|
_showToast('Apply failed', e.message, 'error');
|
|
_hideToast(5000);
|
|
}
|
|
}
|
|
|
|
// Modal-ish panel that lists each failed cluster with its server-side
|
|
// reason. Common failure modes seen in dev: missing DB tables after a
|
|
// stale schema, blocking conflicts that slipped past the front-end guard,
|
|
// rapidfuzz/SQLAlchemy edge cases. Operator can dismiss and either
|
|
// retry the cluster, skip it, or fix the underlying issue.
|
|
function _showFailureDetails(failed) {
|
|
let panel = document.getElementById('apply-failure-panel');
|
|
if (!panel) {
|
|
panel = document.createElement('div');
|
|
panel.id = 'apply-failure-panel';
|
|
panel.className = 'fixed bottom-6 left-6 right-6 sm:right-auto sm:max-w-xl bg-white dark:bg-slate-800 rounded-xl shadow-2xl border border-red-200 dark:border-red-800 p-4 z-40';
|
|
document.body.appendChild(panel);
|
|
}
|
|
const rows = failed.map(f => `
|
|
<li class="flex items-start gap-2 text-sm border-l-2 border-red-300 dark:border-red-700 pl-3 py-1">
|
|
<code class="font-mono text-xs text-gray-500 dark:text-gray-400">${(f.cluster_id || '').slice(0, 8)}…</code>
|
|
<span class="text-gray-800 dark:text-gray-200 flex-1">${_esc(f.reason || '(no reason)')}</span>
|
|
</li>
|
|
`).join('');
|
|
panel.innerHTML = `
|
|
<div class="flex items-start justify-between gap-3 mb-2">
|
|
<h4 class="font-semibold text-gray-900 dark:text-white">
|
|
<svg class="w-5 h-5 inline text-red-500 -mt-0.5 mr-1" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 9v2m0 4h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z"/></svg>
|
|
${failed.length} cluster${failed.length === 1 ? '' : 's'} failed to apply
|
|
</h4>
|
|
<button onclick="document.getElementById('apply-failure-panel').remove()"
|
|
class="text-gray-400 hover:text-gray-600 dark:hover:text-gray-200">
|
|
<svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M6 18L18 6M6 6l12 12"/></svg>
|
|
</button>
|
|
</div>
|
|
<ul class="space-y-1 max-h-64 overflow-y-auto">${rows}</ul>
|
|
<p class="mt-3 text-xs text-gray-500 dark:text-gray-400">
|
|
Common causes: missing DB schema (run the migration sweep), blocking conflict
|
|
with an existing UnitAssignment, or a UNIQUE constraint collision on the
|
|
project name. Re-scan and the failed clusters reappear as pending — fix the
|
|
underlying issue and retry.
|
|
</p>
|
|
`;
|
|
}
|
|
|
|
async function applyOne(clusterId) { return _apply([clusterId]); }
|
|
|
|
async function applyBulkHighConfidence() {
|
|
const high = _scanData.by_confidence.high.filter(s => !s.blocking_conflict);
|
|
const ids = high.map(s => s.cluster_id);
|
|
if (ids.length === 0) return;
|
|
if (!confirm(`Apply ${ids.length} high-confidence cluster${ids.length === 1 ? '' : 's'}? This will create projects, locations, and assignments without further prompting.`)) return;
|
|
return _apply(ids);
|
|
}
|
|
|
|
async function skipOne(clusterId) {
|
|
if (!confirm('Skip this cluster? It will not reappear in future scans.')) return;
|
|
_showToast('Skipping…');
|
|
try {
|
|
const r = await fetch('/api/admin/metadata_backfill/skip', {
|
|
method: 'POST',
|
|
headers: {'Content-Type': 'application/json'},
|
|
body: JSON.stringify({ cluster_ids: [clusterId] }),
|
|
});
|
|
if (!r.ok) throw new Error('HTTP ' + r.status);
|
|
_showToast('Skipped', '', 'success');
|
|
_hideToast(2000);
|
|
await runScan(true);
|
|
} catch (e) {
|
|
_showToast('Skip failed', e.message, 'error');
|
|
_hideToast(4000);
|
|
}
|
|
}
|
|
</script>
|
|
{% endblock %}
|