2 Commits

Author SHA1 Message Date
serversdown 86325b9bab docs: roadmap entry for a SECOND undecoded histogram sub-format (S353)
Observed in fresh ingest logs on 2026-05-28: BE17353 events
(S353L4H2.FZ0H, S353L4H2.P00H, etc.) cause "body codec failed to
decode" warnings.  Different from the byte[5]!=0 case already tracked
(T190 / O121) — these have byte[5]==0x00 with what looks like a
valid block header, but the walker finds zero data blocks anyway.

Operational impact identical to the existing case: ingestion
succeeds, DB peaks come from bw_report overlay, only the chart is
empty.  No data loss.

Pinning so it doesn't get lost — needs a hex dump of one body to
work out what's different about these.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 05:42:18 +00:00
serversdown 6381dcb312 tz: server-wide display timezone via TZ env var (default EST/EDT)
User-reported issue: server logs were timestamped in UTC ("05:36:20"
when local was ~01:36 EDT), and the PDF report's "Created" footer
similarly showed raw UTC.  Inconsistent with the modal which already
converts to browser local via toLocaleString.

Solution: standard Linux TZ env var.  Set once in the container, and:
  - Python's datetime.now() uses local
  - Logging module's timestamps use local
  - matplotlib renderers + report_pdf formatters use local
  - astimezone() conversions resolve to the configured TZ

DB columns stay UTC (created_at uses SQLite's strftime('%Y-...Z', 'now')
which is always UTC, regardless of TZ env var — proper "store UTC,
display local" pattern).

Changes:
  - Dockerfile: install tzdata (python:3.11-slim omits the timezone
    database), set default TZ=America/New_York
  - sfm/report_pdf.py: _fmt_iso_to_bw and _split_iso_to_date_time now
    convert UTC inputs (Z-suffixed) to local via astimezone(); naïve
    inputs (BW recorded-at, already unit-local) returned as-is.
    New _to_display_local helper centralizes the logic.
  - "Created" line in the PDF page footer now uses the converted
    timestamp.

Override per-deployment via the TZ env var in docker-compose
(separate commit on terra-view side).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 05:41:10 +00:00
3 changed files with 47 additions and 14 deletions
+12 -1
View File
@@ -2,10 +2,21 @@ FROM python:3.11-slim
WORKDIR /app
# tzdata is required for the TZ env var to take effect (python:slim
# omits the timezone database). Without it, datetime.now() / logging
# / matplotlib all stay in UTC regardless of TZ. Default zone gets
# set further down via ENV; users override per-deployment via the
# `TZ` env var in docker-compose.
RUN apt-get update && \
apt-get install -y --no-install-recommends curl && \
apt-get install -y --no-install-recommends curl tzdata && \
rm -rf /var/lib/apt/lists/*
# Default display timezone — applied to server logs, datetime.now(),
# matplotlib rendered timestamps, and any naïve-vs-aware datetime
# conversions in the PDF renderer. Override via TZ env var in
# docker-compose; storage in the DB is always UTC regardless.
ENV TZ=America/New_York
COPY pyproject.toml requirements.txt ./
COPY minimateplus ./minimateplus
COPY micromate ./micromate
+1
View File
@@ -567,4 +567,5 @@ Implementation steps (concrete):
- [ ] RV55 DCD/DTR — newer RV55 firmware doesn't assert DCD by default; units don't resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred).
- [ ] **NULL-timestamp duplicate-row dedup.** A small handful of events (2 known on prod as of 2026-05-22) have `events.timestamp IS NULL` because the codec couldn't extract a timestamp from the binary footer. The `UNIQUE(serial, timestamp)` constraint doesn't fire on `NULL` (SQL semantics: `NULL ≠ NULL`), so every `--force` backfill INSERTs a new row instead of UPSERTing the existing one. Cleanup: a one-shot SQL query that keeps only the newest row per `(serial, blastware_filename)` and deletes the rest. Longer-term: extend the unique key to `(serial, COALESCE(timestamp, blastware_filename))` or reject inserts with NULL timestamp.
- [ ] **Histogram body sub-format with `byte[5] != 0`.** ~3 events on prod (`T190LD5Q.LD0H`, `O121L4L1.GU0H`) use a histogram body my walker doesn't recognize — the first block has `byte[5] = 0x01` or `0x07` instead of `0x00`, and the entire body lacks the `1e 0a 00 00` tail signature. Codec returns 0 valid blocks; their DB PVS comes from the bw_report ASCII overlay (which BW computed from the same binary, so the DB columns are correct). Only the `.h5` waveform plot is empty. Cracking the sub-format would unlock the plot. Needs binary+ASCII pairs from a few `byte[5]!=0` events; same RE approach as the K558 case.
- [ ] **Histogram body sub-format with `byte[5] == 0x00` but undecodable.** Observed 2026-05-28 on BE17353 (S353) events: `S353L4H2.FZ0H`, `S353L4H2.P00H`, `S353L4H3.7O0H`, `S353L4H3.E10H`. Body starts `00 00 00 01 0a 00 XX 00 ...` which LOOKS like a valid histogram block header (marker 0x000a at byte[4:6] ✓, byte[5]=0x00 normal-format ✓), but the walker finds zero data blocks across the whole body. Likely an extra header before the block stream OR a different tail signature than `1e 0a 00 00`. Smaller body lengths (1900-2100 bytes) suggest these may be short-recording histogram variants. Same operational impact as the byte[5]!=0 case: event ingests cleanly, DB peaks correct via bw_report overlay, only the chart is empty. Worth dumping a hex view of one body to diagnose.
- [ ] **Sensor-check waveform extraction from the BW binary.** BW's Event Report PDFs include a narrow panel on the right side of the waveform plot showing each channel's response to the sensor self-check signal (a damped sinusoid for geo, sawtooth-at-test-freq for mic). Our parser captures the test RESULTS (`test_freq_hz`, `test_ratio`, `test_amplitude_mv`, `test_results` pass/fail) and the PDF + modal display them as text — but BW's per-sample sensor-check waveform isn't accessible to us today. Two paths to add it: (a) RE the binary to find where the sensor-check samples are stored — could be a section before STRT, after the footer, or in a separate sub-record; protocol reference doesn't currently mention it. (b) If samples aren't in the binary, synthesize a representative waveform from the test parameters (damped sinusoid at `test_freq_hz` with damping from `test_ratio`). Path (a) is the honest answer; path (b) is decorative. Until either lands, the text-only sensor-check display in the report is fine.
+34 -13
View File
@@ -348,9 +348,11 @@ def render_event_report_pdf(rd: ReportData) -> bytes:
# Page footer (common to both layouts) — Created date + event id.
# Pushed to the very page bottom so it doesn't collide with the
# waveform footer scale / trigger legend lines just above.
# Convert UTC server_received_at to local for display.
created_local = _fmt_iso_to_bw(rd.server_received_at) if rd.server_received_at else ""
fig.text(
0.07, 0.005,
f"Created: {rd.server_received_at or ''} • seismo-relay",
f"Created: {created_local} • seismo-relay",
fontsize=6, color="#888", ha="left",
)
fig.text(
@@ -419,31 +421,50 @@ def _render_histogram_layout(fig, rd: ReportData) -> None:
_draw_histogram_subplot(fig, gs[3], rd)
def _to_display_local(iso: str):
"""Parse an ISO timestamp and return a datetime in the system's local
timezone (set by the TZ env var, default America/New_York via the
Dockerfile).
Behaviour:
- "...Z" or "...+HH:MM" suffix → tz-aware UTC → converted to local
- Naïve "YYYY-MM-DDTHH:MM:SS" (no tz) → returned as-is. This
matches the convention used elsewhere in seismo-relay: BW's
recorded-at timestamps are naïve and ALREADY in the unit's
local clock; we don't second-guess them.
"""
import datetime as _dt
dt = _dt.datetime.fromisoformat(iso.replace("Z", "+00:00"))
if dt.tzinfo is not None:
# Convert from UTC (or other tz) → local per the TZ env var.
# astimezone() without arg uses the system timezone.
dt = dt.astimezone()
return dt
def _fmt_iso_to_bw(iso: Optional[str]) -> Optional[str]:
"""Convert a ISO-8601 timestamp like '2026-05-16T22:30:37' to BW's
display format '22:30:37 May 16, 2026'. Returns input unchanged if
it doesn't look like ISO."""
"""Convert an ISO-8601 timestamp to BW's display format
'22:30:37 May 16, 2026'. UTC inputs (with Z suffix) are
converted to the system's local timezone first; naïve inputs
are formatted as-is. Returns input unchanged on parse failure."""
if not iso or "T" not in iso:
return iso
try:
import datetime as _dt
dt = _dt.datetime.fromisoformat(iso.replace("Z", "+00:00"))
return dt.strftime("%H:%M:%S %B %d, %Y").replace(" 0", " ")
return _to_display_local(iso).strftime("%H:%M:%S %B %d, %Y").replace(" 0", " ")
except Exception:
return iso
def _split_iso_to_date_time(iso: Optional[str]) -> tuple[Optional[str], Optional[str]]:
"""Split an ISO timestamp into BW-formatted ("May 27 /26", "06:06:14")
"""Split an ISO timestamp into BW-formatted ('May 27 /26', '06:06:14')
date+time strings. Used for the histogram stats table where the
Date and Time rows are presented separately. Returns (None, None)
if the input isn't a valid ISO datetime."""
Date and Time rows are presented separately. UTC inputs are
converted to local time first. Returns (None, None) on parse failure."""
if not iso:
return (None, None)
try:
import datetime as _dt
dt = _dt.datetime.fromisoformat(iso.replace("Z", "+00:00"))
# BW format: "May 27 /26" (3-letter month + 2-digit year)
dt = _to_display_local(iso)
# BW format: 'May 27 /26' (3-letter month + 2-digit year)
date_str = dt.strftime("%b %d /%y").replace(" 0", " ")
time_str = dt.strftime("%H:%M:%S")
return (date_str, time_str)