Commit Graph
402 Commits
Author SHA1 Message Date
serversdown f73c8eec91 feat(db): update_event_review mirrors reviewed_real + enforces 3-state exclusivity 2026-08-25 00:34:25 +00:00
serversdown 23e4f585a8 fix(db): keep reviewed_real out of the Migration-1 rebuild table (positional SELECT *); regression test 2026-08-25 00:31:39 +00:00
serversdown d9cc5f1780 feat(db): reviewed_real column on events (+ auto-migrate) 2026-08-25 00:26:22 +00:00
serversdownandClaude Opus 4.8 5247e78669 docs(plan): B2-A — reviewed_real + 3-state mirror + twin review-propagation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
2026-08-25 00:25:02 +00:00
serversdown ac67e83bcf chore(release): v0.24.0 — waveform-shape metrics on events 2026-08-22 06:12:01 +00:00
serversdown c982512e17 feat(scripts): backfill events.shape_* from .h5 samples 2026-08-22 06:06:46 +00:00
serversdown e64e3bcd3e feat(ingest): compute shape from the written .h5 in all save paths 2026-08-22 06:01:43 +00:00
serversdown 54c4182023 feat(db): insert_events persists shape_* from waveform record 2026-08-22 05:55:25 +00:00
serversdown a894b001b1 feat(db): shape_* columns on events (+ auto-migrate) 2026-08-22 05:50:45 +00:00
serversdown cec82038ea test(shape): shape_from_h5 round-trips a real .h5 2026-08-22 05:46:58 +00:00
serversdown 2539f903de feat(shape): crest-factor + points-near-peak waveform metrics 2026-08-22 05:42:36 +00:00
serversdownandClaude Opus 4.8 eb92b13aac docs(plan): waveform-shape FT detection — Phase A (seismo-relay)
7-task TDD plan: shape DSP module (crest factor + points-near-peak), shape_*
columns + auto-migrate, insert_events persistence, ingest population in the
save paths, backfill script, /db/events exposure + v0.24.0 bump. Phase B
(terra-view scoring/UI/review) gets its own plan once this feed is live.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
2026-08-21 21:32:59 +00:00
serversdownandClaude Opus 4.8 aebb5644bd chore(release): v0.23.0 — per-channel ZC frequency in events store
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
2026-08-04 21:42:41 +00:00
serversdown ddf6a73292 feat(scripts): backfill event ZC freq columns from sidecars 2026-07-29 17:32:54 +00:00
serversdownandClaude Opus 4.8 1744eb1803 feat(db): persist per-channel ZC freq on insert/upsert
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
2026-07-29 17:27:36 +00:00
serversdown 9a71d5b914 feat(parse): carry per-channel ZC freq onto PeakValues (report + dict paths) 2026-07-29 17:20:17 +00:00
serversdown 248276d141 feat(db): add per-channel ZC-freq columns to events (+ migration) 2026-07-29 17:13:59 +00:00
serversdownandClaude Opus 4.8 12401b917c chore(release): v0.22.0
Full-snapshot bundle (SFM side): document the /db/snapshot,
/db/waveforms/recent.zip and gated /db/restore endpoints. Bump package
version 0.21.1 -> 0.22.0 (pyproject), README version badge + history, and
CHANGELOG. TOOL_VERSION (codec/output provenance) left at 0.21.1 on purpose
— the event codec is unchanged, so existing sidecars must not be flagged
stale for re-backfill.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WyT5m9xppYmoVGmCCVVV6X
2026-07-03 19:32:15 +00:00
serversdown 6e2994acdd fix(api): threadpool the /db/restore handler (drop async) 2026-07-02 17:50:28 +00:00
serversdown b56eb8c692 feat(api): /db/snapshot, /db/waveforms/recent.zip, gated /db/restore 2026-07-02 06:53:24 +00:00
serversdown 280b3d25ec fix(db): WAL-safe safety backup + tests for validate-first and zip-traversal guard 2026-07-02 06:45:17 +00:00
serversdown 7edd0a1265 feat(db): gated WAL-safe restore of seismo_relay.db + waveforms 2026-07-02 06:35:24 +00:00
serversdown 916e6fcabb feat(db): zip recent events' waveform files 2026-07-02 06:30:11 +00:00
serversdown 31660d24e9 feat(db): WAL-safe seismo_relay.db snapshot helper 2026-07-02 06:22:46 +00:00
serversdown 25386cab8b fix(backfill): regenerate IDFH .h5 + merge binary mic_pspl_psi onto bridge
Two gaps in backfill_thor_events.py that left old Thor events showing
stale charts after a v0.21.1 backfill pass:
1. IDFH events were skipped from .h5 regeneration (the "have decoded
   samples" gate was IDFW-only).  Histograms kept their pre-v0.21.1
   .h5 — written from raw_samples = None, which the renderer turned
   into a near-empty bar chart, or for older events the dB(L)-as-pseudo-
   psi mic scale that produced "107.7 psi" peaks (atomic-bomb level
   instead of footstep level).  Fix: synthesise the same 1-sample-per-
   interval array save_imported_idf v0.21.1 uses (peak ADC count per
   channel per interval) so the renderer's bar-chart grouping has
   data to work with.
2. The IDFW h5 path didn't merge binary_peaks.mic_pspl_psi onto the
   IdfEvent before to_minimateplus_event().  The live save_imported_idf
   does this merge — without it, IdfEvent.from_report() only sees the
   .txt's dB(L) value, the bridge falls back to the dBL→psi formula
   (instead of the binary-accurate 2.14e-6 psi/count value), and the
   h5 writer's per-count mic factor lands on a less-correct value.
   Fix: same merge the live ingest does (lift res.event.peaks.mic_pspl_psi
   onto idf_event.peaks before the bridge call).
Verified against UM6047_20250804190047.IDFH (250-interval prod
histogram): 250 intervals decode, mic_pspl_psi = 2.78e-5 (was being
treated as dB(L)=107.7 in the old h5).
Operator: re-run after deploy.  `docker compose exec sfm python
scripts/backfill_thor_events.py` is idempotent — the existing version
check still skips events already at the new TOOL_VERSION, and review
state + captured_at are preserved on the second pass.
2026-06-01 20:02:54 +00:00
serversdown 6cb619ecc4 version bump - 0.21.1 2026-06-01 19:33:44 +00:00
serversdown 1ed86244d0 fix(thor-events): add parallel field for mic psi. Now shows mic in dbl and psi. (psi for charts) 2026-06-01 18:27:24 +00:00
serversdown b2c565f217 fix(idf_waveforms): _find_waveform_body_offset() — scans every 00 02 00 magic past offset 0x0E00, runs decode_waveform_v2 on each candidate, picks the one that returns the most samples. Validated on 483 prod IDFW files: 0 preamble-only events (was ~50%), 355/483 fully decode, 126/483 partial (BW codec walker-stops-early on loud events — known issue).
IDFH now synthesises a 1-sample-per-interval array from the binary intervals and writes an .h5 so the existing renderer works unchanged. Each "sample" is the per-interval peak ADC count → h5_value = count × geo_fs/32768 yields the right bar height.
2026-05-31 20:51:09 +00:00
serversdownandClaude Opus 4.7 43f440812a scripts: add backfill_thor_events.py
Refreshes the bw_report sidecar block + .h5 waveform files for Thor
events ingested before the v0.21.0 adapter wiring + the bee1185 codec
fix.  Those events landed with extensions.idf_report only (no
bw_report, no .h5 for IDFW) — symptom on the UI side: the modal chart
404'd on /waveform.json and the PDF rendered from DB-only fields
without sensor self-check, full per-channel breakdown, or mic dB(L).

Walks <store>/<serial>/<filename>:
  - Reads the existing sidecar (preserves review state + captured_at)
  - Re-runs read_idf_file() on the binary bytes (passes data=
    kwarg so codec doesn't try the broken bare-path Path.read_bytes)
  - Reads extensions.idf_report from the existing sidecar
  - Runs build_bw_report_from_idf adapter
  - Writes refreshed sidecar with bw_report + bumped tool_version,
    preserving review block and original captured_at
  - For IDFW: regenerates .h5 by bridging IdfEvent.from_report ->
    to_minimateplus_event -> write_event_hdf5 (mirrors save_imported_idf
    steps 4-7)
  - IDFH events skip .h5 (histograms have no per-sample data)

Skips events already at current TOOL_VERSION with bw_report present.
--force overrides.  --skip-hdf5 limits to sidecar-only refresh.
--dry-run for preview.

Validated against the prod-snap waveform store: 3,815 Thor sidecars
refreshed cleanly with 0 errors, 462 IDFW .h5 files written, 2 skipped
(binaries with no sidecar — backfill doesn't conjure events from
nothing).  Verified one originally-broken IDFW event now serves
waveform.json (200, 168KB) and a fully populated PDF (119KB vs the
previous 56KB sparse output).

Operator workflow on prod:
  docker exec <sfm-container> python3 /app/scripts/backfill_thor_events.py --dry-run
  # Inspect counts, then for real:
  docker exec <sfm-container> python3 /app/scripts/backfill_thor_events.py

Idempotent — re-running it is a no-op once everything's at the current
TOOL_VERSION.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-30 04:37:43 +00:00
serversdownandClaude Opus 4.7 23e83908c2 report_pdf: fix PVS overlapping stats table, drop NA caption
Two related fixes to the per-channel stats block:

1. Pin the stats table's position via an explicit bbox= on
   ax.table() so the bottom edge is at a known axes-fraction Y.
   The previous loc="upper left" + tbl.scale(1, 1.4) combo let
   matplotlib choose row heights based on text size, which made the
   table extend further below the axes than the hard-coded PVS line
   at y=-0.08 expected.  Result was the "Peak Vector Sum X in/s"
   string landing horizontally inside the Peak Displacement row.

   With bbox=[0, 1-N*0.12, 0.80, N*0.12] the table is pinned to a
   precise rectangle (12% axes-fraction per row × N rows tall).
   _draw_stats_table now stashes the bottom Y on the axes for the
   PVS helper to reference, so the geometry stays in sync.

2. Center PVS horizontally (ha="center" at x=0.5 instead of ha="left"
   at x=0).  The previous left-edge alignment put PVS at the same
   X as the label column, which read as "off-center" once the rest
   of the stats data was column-aligned further right.

3. Drop the "NA: Not Applicable" caption.  It existed to explain
   "—" placeholder cells, but "—" is universally understood and the
   caption was always visually squished against the PVS line below.
   Less cruft on the page; one fewer position to manage.

Verified against a real BE12599 histogram event (5 data rows) and
a real UM12947 IDFW waveform event (6 data rows) — both layouts
clear the table cleanly with no overlap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 22:17:43 +00:00
serversdownandClaude Opus 4.7 bee118506b fix(idf): decode from in-memory bytes during ingest
Bug shipped in v0.21.0: save_imported_idf called read_idf_file()
with `source_path` (a bare filename like "UM12947_….IDFW") BEFORE
writing the binary to disk.  The codec did Path(path).read_bytes()
which resolved relative to /app and hit FileNotFoundError.  The
error was caught + logged as a warning, and ingest fell back to
.txt-only — events still landed in the DB but lost the bw_report
block + .h5 waveform that the codec was supposed to produce.

Observed during a full re-forward from thor-watcher on 2026-05-29:
every Thor event logged "binary codec failed for X: [Errno 2] No
such file or directory" and got binary_decoded=False.

Fix:
- read_idf_file() gains a `data: Optional[bytes]` kwarg.  When
  supplied, skips the disk read and decodes the provided bytes
  directly.  `path` stays required (used for filename in error
  messages + .IDFH vs .IDFW suffix detection); only the read is
  conditional.  Backward compatible — existing positional callers
  (CLI scripts, tests) continue to work unchanged.
- save_imported_idf passes `data=idf_bytes` since the bytes are
  already in memory from the multipart upload.  Filesystem write
  still happens at step 5 of the existing flow; codec just no
  longer depends on it.

Verified end-to-end against UM11719_20231219162723.IDFW from the
example-data corpus: ingest endpoint returns inserted=1, log line
shows binary_decoded=True + h5=...IDFW.h5, no warnings.

Re-forward existing Thor events from thor-watcher after deploy to
backfill the bw_report block — UPSERT preserves review state.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 20:09:54 +00:00
serversdownandClaude Opus 4.7 defd17d9c2 sfm_webapp: harmonize "Received by server at" → "Time received"
Matches Terra-View's event-modal relabel from the same iteration.
Wording was already clearer here than in Terra-View's "Captured at",
but using identical text across both surfaces means operators see the
same label whether they're in the native modal or the standalone
webapp.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 19:51:58 +00:00
serversdownandClaude Opus 4.7 e42956a20b release: v0.21.0 — Thor / Series IV codec + Thor→BW adapter
Documents two commits that landed on dev since v0.20.0:

  9b71ead  series 4 codec work, initial decode success
           micromate/idf_file.read_idf_file() decodes both IDFW
           (waveform; 87-99% sample fidelity reusing
           decode_waveform_v2 at offset 0x0f1f) and IDFH (histogram;
           dedicated segment-based decoder, all 859 corpus files
           decode, 181,071 intervals total).

  9fd52dd  feat: add thor report generation, pdf generation
           micromate/idf_to_bw_report.py adapter projects parsed
           Thor data into the bw_report sidecar shape so Thor
           events flow through sfm/report_pdf.py without a
           separate renderer.  Wired into save_imported_idf.

Net effect: a Thor event ingested via /db/import/idf_file now
lands with the same fidelity as a BW event, gets a per-event PDF
on demand, and renders in Terra-View's modal chart using the same
plotting code as a BW event.

Roadmap items closed:
- Binary .IDFW / .IDFH codec (was pending)
- Series IV (Thor IDF) binary codec reverse-engineering

Companion: Terra-View v0.13.0 ships in parallel and closes Phase 1
of the SFM integration.  No API changes in seismo-relay for that
piece — Terra-View just consumes existing endpoints better.

Bumps:
- pyproject.toml 0.20.0 → 0.21.0
- minimateplus.event_file_io.TOOL_VERSION 0.20.0 → 0.21.0
  (any subsequent backfill_sidecars.py --force will re-stamp
  existing sidecars; expected + harmless)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 19:25:44 +00:00
serversdown 9fd52ddabb feat: add thor report generation, pdf generation. 2026-05-29 19:03:06 +00:00
serversdown 9b71ead44b series 4 codec work, inital decode success 2026-05-29 06:33:13 +00:00
serversdownandClaude Opus 4.7 1bccc44b88 release: v0.20.0 — PDF + parser polish
Closes out the Event-Report PDF iteration started in v0.17.x and ships
the parser fixes the real-world events were tripping over.

Today's additions on top of the pre-v0.20 unreleased body:

- Server-wide display TZ via the TZ env var (default America/New_York
  on prod).  Affects server logs, the PDF report's "Created" footer,
  matplotlib datetime axes.  DB columns stay UTC.  Dockerfile now
  installs tzdata.
- ZC Freq "above-range" handling — parser stores 100.0 +
  zc_freq_above_range flag for BW's ">100 Hz" marker.  Renders as
  >100 in the PDF stats table, both modals (inline on webapp Peaks,
  new column on event-browser table).
- scripts/backfill_sidecars.py --reparse-txt — re-runs the current
  parser against the preserved _ASCII.TXT and overwrites the
  sidecar's bw_report block.  Lets parser fixes reach old events
  without re-forwarding.  Validated end-to-end against ~10k prod
  events.

Fixes shipped today:
- histogram_interval_size_s missing from ReportData → every
  histogram PDF render 500'd.
- Histogram PDF geo channels now share a nice-quantized y-axis
  (0.005-LSB-aware 1-2-5 step sequence) instead of auto-scaling
  per channel + inventing sub-LSB "0.003 in/s/div" footer labels.

Roadmap delta: closes the BW ASCII parser "PPV-miss on some TXT
formats", "histogram-specific structural fields", and ">100 Hz value
parsing" items.  Adds a new entry for the byte[5]==0 histogram body
sub-format observed on S353 events.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 21:17:53 +00:00
serversdownandClaude Opus 4.7 a3cc44d30a feat(backfill): --reparse-txt flag to refresh bw_report from preserved .TXT
The existing backfill_sidecars.py PRESERVES the bw_report block across
regenerations — it's treated as the source of truth from the original
ingest pass (the .TXT isn't reachable from the script's normal data
path, so it can't be re-derived).

That means parser-side fixes (like the 2026-05-28 ">100 Hz" ZC Freq
addition) won't reach old events even with --force.  The new
--reparse-txt flag fixes that: when the sidecar's source.txt_filename
points at a preserved <serial>/<filename>_ASCII.TXT, the script re-runs
the current parser against it and overwrites the bw_report block.

Implies sidecar regeneration on every event (bypasses the
sha-up-to-date / version-up-to-date skip), so that the .h5 cascade-
regenerates alongside.  No-op for events without a preserved .TXT
(legacy ingests pre-2026-05-27).  Idempotent — re-running it produces
the same sidecar bytes when the parser hasn't changed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 18:56:23 +00:00
serversdownandClaude Opus 4.7 6a73523e4d ui: surface per-channel ZC Freq (and ">100") in event modals
The PDF report shows per-channel ZC Freq alongside PPV in the stats
block, but neither modal exposed it.  Now that the sidecar projection
carries zc_freq_hz + zc_freq_above_range, plumb them through:

- sfm_webapp.html: inline suffix on existing Peaks cells, e.g.
  "Tran  0.04500 in/s · >100 Hz".  Empty suffix when no ZC is
  available (legacy events without a preserved .TXT).

- event_browser.html: new ZC Freq column on the per-channel stats
  table.  Required adding a parallel sidecar fetch in loadEvent()
  (waveform.json alone doesn't carry bw_report).  Fetch failure is
  non-fatal — falls back to "—" in the new column.

Above-range ZC peaks (BW ">100 Hz") render with a literal ">"
prefix mirroring the PDF, so operators don't have to generate the
PDF to see when a channel hit the zero-crossing ceiling.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 18:47:37 +00:00
serversdownandClaude Opus 4.7 780b45a371 feat: render ">100" for above-range ZC Freq instead of "—"
BW writes ">100 Hz" for ZC Freq when the zero-crossing algorithm sees a
peak too fast to count — the device's reporting ceiling is 100 Hz on
V10.72.  Our parser fell back to None via _parse_number (which requires
a leading digit), so the PDF rendered "—" where BW shows ">100".

Mirrors the OORANGE/saturated pattern already used for PPV and PSPL:
parser stores the threshold (100.0) on zc_freq_hz + sets a new
zc_freq_above_range flag.  Projection carries the flag through to the
sidecar; PDF renderer prepends ">" when set.

Affects both per-channel stats tables (waveform + histogram variants)
and the mic block's ZC Freq row.

Verified on the real T190LD5Q.LK0W fixture: Tran zc_freq_hz=100.0
above_range=True; Vert/Long (normal values) above_range=False; "N/A"
still produces zc_freq_hz=None which renders as "—" (unchanged).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 18:38:49 +00:00
serversdownandClaude Opus 4.7 f6abe3caa0 fix(report_pdf): histogram geo channels share nice-quantized y-axis
Two related visual bugs on histogram PDFs:

1. Per-channel auto-scale meant Tran/Vert/Long had different y-axes
   (e.g. 0-0.015, 0-0.025, 0-0.020) — bars looked taller on the
   channel that happened to be quietest.  Not directly comparable.

2. Footer "Amplitude Geo: X in/s/div" was just amax/5 of the FIRST
   geo channel with data, with no LSB quantization — producing
   nonsense like 0.003 in/s/div when the geophone LSB is 0.005.

Fix: compute a single shared geo y-axis range from max(Tran,Vert,Long),
quantize the per-division step to BW's 1-2-5 sequence rounded to the
0.005 LSB (0.005, 0.01, 0.025, 0.05, 0.1, 0.25, ...), apply the same
ylim + ticks to all three geo subplots, and use that same step for the
footer label.  MicL stays on its own auto-scale (different units).

Verified across edge cases including the reported event
(geo max 0.025 → 0.005/div, top 0.025), small PVS events, and large
blast amplitudes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 18:22:20 +00:00
serversdownandClaude Opus 4.7 ad2702d4bf fix(report_pdf): add missing histogram_interval_size_s field
The histogram-interval-times derivation block at line 314 references
rd.histogram_interval_size_s, but the field wasn't declared on the
ReportData dataclass — only the string form histogram_interval_size
was.  Result: every PDF render of a histogram event raised
AttributeError → 500 from /db/events/{id}/report.pdf.

Cause: when the histogram aggregation block was inlined into
gather_report_data, the seconds-numeric counterpart that the
projection already carries (bw_report.histogram.interval_size_s) was
never wired into the dataclass.  Waveform PDFs weren't affected
because the offending line is gated on is_histogram.

Fix: add the field, read it from the projection alongside the other
histogram keys.  No-op for waveform events (the field stays None and
the gate skips it).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 18:07:41 +00:00
serversdownandClaude Opus 4.7 86325b9bab docs: roadmap entry for a SECOND undecoded histogram sub-format (S353)
Observed in fresh ingest logs on 2026-05-28: BE17353 events
(S353L4H2.FZ0H, S353L4H2.P00H, etc.) cause "body codec failed to
decode" warnings.  Different from the byte[5]!=0 case already tracked
(T190 / O121) — these have byte[5]==0x00 with what looks like a
valid block header, but the walker finds zero data blocks anyway.

Operational impact identical to the existing case: ingestion
succeeds, DB peaks come from bw_report overlay, only the chart is
empty.  No data loss.

Pinning so it doesn't get lost — needs a hex dump of one body to
work out what's different about these.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 05:42:18 +00:00
serversdownandClaude Opus 4.7 6381dcb312 tz: server-wide display timezone via TZ env var (default EST/EDT)
User-reported issue: server logs were timestamped in UTC ("05:36:20"
when local was ~01:36 EDT), and the PDF report's "Created" footer
similarly showed raw UTC.  Inconsistent with the modal which already
converts to browser local via toLocaleString.

Solution: standard Linux TZ env var.  Set once in the container, and:
  - Python's datetime.now() uses local
  - Logging module's timestamps use local
  - matplotlib renderers + report_pdf formatters use local
  - astimezone() conversions resolve to the configured TZ

DB columns stay UTC (created_at uses SQLite's strftime('%Y-...Z', 'now')
which is always UTC, regardless of TZ env var — proper "store UTC,
display local" pattern).

Changes:
  - Dockerfile: install tzdata (python:3.11-slim omits the timezone
    database), set default TZ=America/New_York
  - sfm/report_pdf.py: _fmt_iso_to_bw and _split_iso_to_date_time now
    convert UTC inputs (Z-suffixed) to local via astimezone(); naïve
    inputs (BW recorded-at, already unit-local) returned as-is.
    New _to_display_local helper centralizes the logic.
  - "Created" line in the PDF page footer now uses the converted
    timestamp.

Override per-deployment via the TZ env var in docker-compose
(separate commit on terra-view side).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 05:41:10 +00:00
serversdownandClaude Opus 4.7 53c05d93e2 delete: also clean up preserved _ASCII.TXT file
_cleanup_event_files() removes the on-disk artifacts when an event is
hard-deleted (binary, a5_pickle, sidecar, h5).  Today's .TXT
preservation feature added a new on-disk file (_ASCII.TXT next to the
binary) but the cleanup didn't know about it — so any event deleted
via /db/events/{id} (single) or /db/events/delete_bulk (or the
Terra-View "SFM Event DB Manager" UI which proxies through to those
endpoints) was leaving orphan .TXT files in the store.

Added "txt" to the cleanup list using the new
WaveformStore.txt_path_for().  Safe for old events without a .TXT —
the exists() check skips the unlink.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 05:31:08 +00:00
serversdownandClaude Opus 4.7 a5888e1b5c report_pdf: PDF histogram aggregation + fix footer/x-axis overlap
Two issues spotted on a histogram event PDF:

1. Footer scale ("Time — /div  Amplitude Geo: X in/s/div  Mic: Y
   psi(L)/div") was overlapping horizontally with the x-axis tick
   labels (0, 20, 40, 60...).  Both rendered on the same Y row.
   Fix: bumped gridspec bottom margin from 0.06 → 0.12, moved the
   footer text from y=0.045 → y=0.030 (below the tick labels), moved
   the page-bottom Created/Event line from y=0.015 → y=0.005.
   Trigger legend on waveforms moved 0.030 → 0.018.  Everything
   stacks cleanly now without collision.

2. PDF was showing the raw codec output (~150+ bars per histogram)
   instead of BW's per-interval aggregation.  Why: the aggregation
   I'd added to /db/events/{id}/waveform.json wasn't replicated in
   the PDF gather path.  Now: gather_report_data does the same
   max-per-group aggregation when bw_report.histogram.n_intervals is
   populated, AND derives per-interval HH:MM:SS labels from the
   start time + interval_size_s.  Result: histogram PDFs now match
   BW's display (one bar per BW interval, x-axis labeled with actual
   times) — same fix as the modal chart, applied to the PDF.

For events ingested BEFORE the parser extension (no histogram block
in their sidecar), aggregation is a no-op — they still render with
per-block bars + interval-index x-axis (but the overlap fix applies
to them too).  Re-forwarding repopulates the histogram block.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 04:33:53 +00:00
serversdownandClaude Opus 4.7 b9f8bbb220 viewers: enforce minimum Y-range on histogram channels
Quiet histogram events were filling the chart panel even though the
peak was tiny (0.005 in/s rendered as 90% of chart height because
Chart.js auto-scaled to peak * 1.1).  Made everything look uniformly
loud regardless of actual amplitude.

BW's solution: a near-fixed scale per channel ("Geo: 0.002 in/s/div"
from the footer).  Quiet events render small, loud events render
proportionally tall.

Match the intent without copying BW's "no Y-axis labels at all"
convention.  For histogram channels:

  Geo (in/s):       min Y range 0.05 in/s
  Mic in psi:       min Y range 0.001 psi
  Mic in dBL:       unchanged (the 60 dBL floor + peak+5 top already
                    gives quiet events a sensible baseline)

So a 0.005 in/s geo event renders as ~10% of chart height; a 0.05
event fills it; a 5.0 event still fills it (max(peak*1.1, 0.05) ==
peak*1.1 for any peak > 0.045).

Waveform charts unchanged — they should zoom for shape detail.
Applied to both the modal in sfm_webapp.html and the standalone
/events page in event_browser.html.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 04:23:01 +00:00
serversdownandClaude Opus 4.7 b59f886cb7 docs: roadmap entry for sensor-check waveform extraction
BW's Event Report PDFs include a per-channel sensor-check response
waveform on the right side of the bottom plot (damped sinusoid for
geo channels, sawtooth-at-test-freq for mic).  Looks like real
per-sample data extracted from the binary, not synthesized.

Our parser captures the test results (freq, ratio, amplitude,
pass/fail) but not the waveform samples — so the report shows text
only for sensor check.  Pinning a roadmap entry to investigate the
binary for the sample data (path a) or fall back to synthesized
visualization (path b).

Current text-only display is operationally sufficient.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 04:17:50 +00:00
serversdownandClaude Opus 4.7 87aec3f4d1 viewers: smoother mic dBL chart + restore binary/TXT download links
Two issues spotted in the modal:

1. Mic dBL chart looked spikey/discontinuous — isolated bars at 80-95
   with gaps in between.  Cause: _psiToDbl() returns null for zero or
   negative samples, and most mic samples on a quiet event sit at the
   digitization noise floor where they're effectively zero.  Result:
   the chart only renders the moments when instantaneous SPL exceeded
   the Y-axis bottom — looks like a sound trigger gate.

   Fix: new _psiToDblForChart() rectifies the AC waveform (abs), then
   converts to dBL, then floors at MIC_DBL_FLOOR=60 dBL.  Chart now
   has a continuous 60 dBL baseline with peaks above it — matches how
   acoustic engineers expect SPL-vs-time.  Y-axis bottom pinned to
   MIC_DBL_FLOOR, top to peak + 5 dB headroom.  Peak label still uses
   the unrectified _psiToDbl so the displayed peak value is exact.

2. Filename in Source/Files block was unlinked.  Endpoint exists
   (/db/events/{id}/blastware_file) — just wasn't wired to the modal.
   Made it a clickable download link.  Same treatment for the
   preserved .TXT — added "(download .TXT)" link next to source kind
   when source.txt_filename is populated (events ingested after the
   .TXT preservation feature landed; older events show no link).

Applied to both the inline modal in sfm_webapp.html and the
standalone /events page in event_browser.html.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 23:08:21 +00:00
serversdownandClaude Opus 4.7 ace542cba5 report_pdf: wire histogram peak date/time + PVS-when + Finish field
Spotted comparing our PDF to BW's reference for T003LLUB.CE0H:
  - Finish blank
  - Per-channel Date / Time rows all dashes
  - MicL PSPL line missing "on May 27, 2026 at 06:19:14"
  - Peak Vector Sum missing "on May 27, 2026 At 06:06:14"

Root cause: I'd added these fields to the projection (write side) in
_bw_report_to_dict but never wired them into gather_report_data
(read side).  Plus the projection used keys "start"/"stop" while
gather was reading "start_str"/"stop_str" — typo'd lookup.

Fixes:
  - gather_report_data now reads bw_report.histogram.start /
    .stop / .channel_peak_when (correct keys, matching the projection)
  - Per-channel "peak_date" / "peak_time" populated from
    channel_peak_when[<channel>] for the histogram stats table
  - MicL PSPL line formats as "PSPL  125.7 dB(L) on May 27, 2026
    at 06:19:14" (BW style) when channel_peak_when["MicL"] is present;
    falls back to the waveform-relative "at 0.012 sec" otherwise
  - PVS line formats as "Peak Vector Sum  0.091 in/s on May 27, 2026
    At 06:06:14" (BW style) when bw_report.peaks.vector_sum.when is
    populated; falls back to the relative time_s for waveforms
  - New _split_iso_to_date_time() helper splits ISO timestamps into
    BW-formatted ("May 27 /26", "06:06:14") date+time pairs for the
    stats table's separate Date and Time rows

Events ingested BEFORE the parser extension landed (most of the
existing prod corpus) still show dashes — their sidecars lack the
histogram block.  Re-forwarding repopulates.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 22:47:53 +00:00
serversdownandClaude Opus 4.7 8cbda09917 viewers: render timestamps in browser-local time
Spotted on the SFM webapp event modal — "Received by server at" was
showing the raw ISO string "2026-05-27T21:59:57.213043Z" because we
were assigning ev.timestamp / src.captured_at directly to the
textContent of the modal fields, bypassing the existing _fmtTs()
helper that wraps them in toLocaleString().

Net effect for operators: confusing "21:59 vs it's 6 PM" mismatch
when the displayed UTC timestamp didn't match wall-clock time.  The
values were always correct; the display was just ambiguous.

After this fix:
  - "Recorded at" (naive ISO from BW = unit local time) renders
    cleanly as the unit wrote it: "5/27/2026, 6:00:13 AM"
  - "Received by server at" (UTC with Z suffix) converts to browser
    local: "5/27/2026, 5:59:57 PM"
  - Timestamp column in the history table already used _fmtTs —
    unchanged
  - Same fix applied to the standalone /events page (sidebar event
    list + meta header) via a new _fmtTsLocal helper

Note: did NOT add file-mtime-on-watcher-PC tracking as a separate
"Called in at" column — discussed and decided created_at is close
enough for schedule-compliance monitoring (worst case lag = watcher
poll interval ~60s, indistinguishable from BW write time at the
operationally-relevant resolution).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-27 22:30:43 +00:00