2 Commits

Author SHA1 Message Date
serversdown 86325b9bab docs: roadmap entry for a SECOND undecoded histogram sub-format (S353)
Observed in fresh ingest logs on 2026-05-28: BE17353 events
(S353L4H2.FZ0H, S353L4H2.P00H, etc.) cause "body codec failed to
decode" warnings.  Different from the byte[5]!=0 case already tracked
(T190 / O121) — these have byte[5]==0x00 with what looks like a
valid block header, but the walker finds zero data blocks anyway.

Operational impact identical to the existing case: ingestion
succeeds, DB peaks come from bw_report overlay, only the chart is
empty.  No data loss.

Pinning so it doesn't get lost — needs a hex dump of one body to
work out what's different about these.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 05:42:18 +00:00
serversdown 6381dcb312 tz: server-wide display timezone via TZ env var (default EST/EDT)
User-reported issue: server logs were timestamped in UTC ("05:36:20"
when local was ~01:36 EDT), and the PDF report's "Created" footer
similarly showed raw UTC.  Inconsistent with the modal which already
converts to browser local via toLocaleString.

Solution: standard Linux TZ env var.  Set once in the container, and:
  - Python's datetime.now() uses local
  - Logging module's timestamps use local
  - matplotlib renderers + report_pdf formatters use local
  - astimezone() conversions resolve to the configured TZ

DB columns stay UTC (created_at uses SQLite's strftime('%Y-...Z', 'now')
which is always UTC, regardless of TZ env var — proper "store UTC,
display local" pattern).

Changes:
  - Dockerfile: install tzdata (python:3.11-slim omits the timezone
    database), set default TZ=America/New_York
  - sfm/report_pdf.py: _fmt_iso_to_bw and _split_iso_to_date_time now
    convert UTC inputs (Z-suffixed) to local via astimezone(); naïve
    inputs (BW recorded-at, already unit-local) returned as-is.
    New _to_display_local helper centralizes the logic.
  - "Created" line in the PDF page footer now uses the converted
    timestamp.

Override per-deployment via the TZ env var in docker-compose
(separate commit on terra-view side).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 05:41:10 +00:00
3 changed files with 47 additions and 14 deletions
+12 -1
View File
@@ -2,10 +2,21 @@ FROM python:3.11-slim
WORKDIR /app WORKDIR /app
# tzdata is required for the TZ env var to take effect (python:slim
# omits the timezone database). Without it, datetime.now() / logging
# / matplotlib all stay in UTC regardless of TZ. Default zone gets
# set further down via ENV; users override per-deployment via the
# `TZ` env var in docker-compose.
RUN apt-get update && \ RUN apt-get update && \
apt-get install -y --no-install-recommends curl && \ apt-get install -y --no-install-recommends curl tzdata && \
rm -rf /var/lib/apt/lists/* rm -rf /var/lib/apt/lists/*
# Default display timezone — applied to server logs, datetime.now(),
# matplotlib rendered timestamps, and any naïve-vs-aware datetime
# conversions in the PDF renderer. Override via TZ env var in
# docker-compose; storage in the DB is always UTC regardless.
ENV TZ=America/New_York
COPY pyproject.toml requirements.txt ./ COPY pyproject.toml requirements.txt ./
COPY minimateplus ./minimateplus COPY minimateplus ./minimateplus
COPY micromate ./micromate COPY micromate ./micromate
+1
View File
@@ -567,4 +567,5 @@ Implementation steps (concrete):
- [ ] RV55 DCD/DTR — newer RV55 firmware doesn't assert DCD by default; units don't resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred). - [ ] RV55 DCD/DTR — newer RV55 firmware doesn't assert DCD by default; units don't resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred).
- [ ] **NULL-timestamp duplicate-row dedup.** A small handful of events (2 known on prod as of 2026-05-22) have `events.timestamp IS NULL` because the codec couldn't extract a timestamp from the binary footer. The `UNIQUE(serial, timestamp)` constraint doesn't fire on `NULL` (SQL semantics: `NULL ≠ NULL`), so every `--force` backfill INSERTs a new row instead of UPSERTing the existing one. Cleanup: a one-shot SQL query that keeps only the newest row per `(serial, blastware_filename)` and deletes the rest. Longer-term: extend the unique key to `(serial, COALESCE(timestamp, blastware_filename))` or reject inserts with NULL timestamp. - [ ] **NULL-timestamp duplicate-row dedup.** A small handful of events (2 known on prod as of 2026-05-22) have `events.timestamp IS NULL` because the codec couldn't extract a timestamp from the binary footer. The `UNIQUE(serial, timestamp)` constraint doesn't fire on `NULL` (SQL semantics: `NULL ≠ NULL`), so every `--force` backfill INSERTs a new row instead of UPSERTing the existing one. Cleanup: a one-shot SQL query that keeps only the newest row per `(serial, blastware_filename)` and deletes the rest. Longer-term: extend the unique key to `(serial, COALESCE(timestamp, blastware_filename))` or reject inserts with NULL timestamp.
- [ ] **Histogram body sub-format with `byte[5] != 0`.** ~3 events on prod (`T190LD5Q.LD0H`, `O121L4L1.GU0H`) use a histogram body my walker doesn't recognize — the first block has `byte[5] = 0x01` or `0x07` instead of `0x00`, and the entire body lacks the `1e 0a 00 00` tail signature. Codec returns 0 valid blocks; their DB PVS comes from the bw_report ASCII overlay (which BW computed from the same binary, so the DB columns are correct). Only the `.h5` waveform plot is empty. Cracking the sub-format would unlock the plot. Needs binary+ASCII pairs from a few `byte[5]!=0` events; same RE approach as the K558 case. - [ ] **Histogram body sub-format with `byte[5] != 0`.** ~3 events on prod (`T190LD5Q.LD0H`, `O121L4L1.GU0H`) use a histogram body my walker doesn't recognize — the first block has `byte[5] = 0x01` or `0x07` instead of `0x00`, and the entire body lacks the `1e 0a 00 00` tail signature. Codec returns 0 valid blocks; their DB PVS comes from the bw_report ASCII overlay (which BW computed from the same binary, so the DB columns are correct). Only the `.h5` waveform plot is empty. Cracking the sub-format would unlock the plot. Needs binary+ASCII pairs from a few `byte[5]!=0` events; same RE approach as the K558 case.
- [ ] **Histogram body sub-format with `byte[5] == 0x00` but undecodable.** Observed 2026-05-28 on BE17353 (S353) events: `S353L4H2.FZ0H`, `S353L4H2.P00H`, `S353L4H3.7O0H`, `S353L4H3.E10H`. Body starts `00 00 00 01 0a 00 XX 00 ...` which LOOKS like a valid histogram block header (marker 0x000a at byte[4:6] ✓, byte[5]=0x00 normal-format ✓), but the walker finds zero data blocks across the whole body. Likely an extra header before the block stream OR a different tail signature than `1e 0a 00 00`. Smaller body lengths (1900-2100 bytes) suggest these may be short-recording histogram variants. Same operational impact as the byte[5]!=0 case: event ingests cleanly, DB peaks correct via bw_report overlay, only the chart is empty. Worth dumping a hex view of one body to diagnose.
- [ ] **Sensor-check waveform extraction from the BW binary.** BW's Event Report PDFs include a narrow panel on the right side of the waveform plot showing each channel's response to the sensor self-check signal (a damped sinusoid for geo, sawtooth-at-test-freq for mic). Our parser captures the test RESULTS (`test_freq_hz`, `test_ratio`, `test_amplitude_mv`, `test_results` pass/fail) and the PDF + modal display them as text — but BW's per-sample sensor-check waveform isn't accessible to us today. Two paths to add it: (a) RE the binary to find where the sensor-check samples are stored — could be a section before STRT, after the footer, or in a separate sub-record; protocol reference doesn't currently mention it. (b) If samples aren't in the binary, synthesize a representative waveform from the test parameters (damped sinusoid at `test_freq_hz` with damping from `test_ratio`). Path (a) is the honest answer; path (b) is decorative. Until either lands, the text-only sensor-check display in the report is fine. - [ ] **Sensor-check waveform extraction from the BW binary.** BW's Event Report PDFs include a narrow panel on the right side of the waveform plot showing each channel's response to the sensor self-check signal (a damped sinusoid for geo, sawtooth-at-test-freq for mic). Our parser captures the test RESULTS (`test_freq_hz`, `test_ratio`, `test_amplitude_mv`, `test_results` pass/fail) and the PDF + modal display them as text — but BW's per-sample sensor-check waveform isn't accessible to us today. Two paths to add it: (a) RE the binary to find where the sensor-check samples are stored — could be a section before STRT, after the footer, or in a separate sub-record; protocol reference doesn't currently mention it. (b) If samples aren't in the binary, synthesize a representative waveform from the test parameters (damped sinusoid at `test_freq_hz` with damping from `test_ratio`). Path (a) is the honest answer; path (b) is decorative. Until either lands, the text-only sensor-check display in the report is fine.
+34 -13
View File
@@ -348,9 +348,11 @@ def render_event_report_pdf(rd: ReportData) -> bytes:
# Page footer (common to both layouts) — Created date + event id. # Page footer (common to both layouts) — Created date + event id.
# Pushed to the very page bottom so it doesn't collide with the # Pushed to the very page bottom so it doesn't collide with the
# waveform footer scale / trigger legend lines just above. # waveform footer scale / trigger legend lines just above.
# Convert UTC server_received_at to local for display.
created_local = _fmt_iso_to_bw(rd.server_received_at) if rd.server_received_at else ""
fig.text( fig.text(
0.07, 0.005, 0.07, 0.005,
f"Created: {rd.server_received_at or ''} • seismo-relay", f"Created: {created_local} • seismo-relay",
fontsize=6, color="#888", ha="left", fontsize=6, color="#888", ha="left",
) )
fig.text( fig.text(
@@ -419,31 +421,50 @@ def _render_histogram_layout(fig, rd: ReportData) -> None:
_draw_histogram_subplot(fig, gs[3], rd) _draw_histogram_subplot(fig, gs[3], rd)
def _to_display_local(iso: str):
"""Parse an ISO timestamp and return a datetime in the system's local
timezone (set by the TZ env var, default America/New_York via the
Dockerfile).
Behaviour:
- "...Z" or "...+HH:MM" suffix → tz-aware UTC → converted to local
- Naïve "YYYY-MM-DDTHH:MM:SS" (no tz) → returned as-is. This
matches the convention used elsewhere in seismo-relay: BW's
recorded-at timestamps are naïve and ALREADY in the unit's
local clock; we don't second-guess them.
"""
import datetime as _dt
dt = _dt.datetime.fromisoformat(iso.replace("Z", "+00:00"))
if dt.tzinfo is not None:
# Convert from UTC (or other tz) → local per the TZ env var.
# astimezone() without arg uses the system timezone.
dt = dt.astimezone()
return dt
def _fmt_iso_to_bw(iso: Optional[str]) -> Optional[str]: def _fmt_iso_to_bw(iso: Optional[str]) -> Optional[str]:
"""Convert a ISO-8601 timestamp like '2026-05-16T22:30:37' to BW's """Convert an ISO-8601 timestamp to BW's display format
display format '22:30:37 May 16, 2026'. Returns input unchanged if '22:30:37 May 16, 2026'. UTC inputs (with Z suffix) are
it doesn't look like ISO.""" converted to the system's local timezone first; naïve inputs
are formatted as-is. Returns input unchanged on parse failure."""
if not iso or "T" not in iso: if not iso or "T" not in iso:
return iso return iso
try: try:
import datetime as _dt return _to_display_local(iso).strftime("%H:%M:%S %B %d, %Y").replace(" 0", " ")
dt = _dt.datetime.fromisoformat(iso.replace("Z", "+00:00"))
return dt.strftime("%H:%M:%S %B %d, %Y").replace(" 0", " ")
except Exception: except Exception:
return iso return iso
def _split_iso_to_date_time(iso: Optional[str]) -> tuple[Optional[str], Optional[str]]: def _split_iso_to_date_time(iso: Optional[str]) -> tuple[Optional[str], Optional[str]]:
"""Split an ISO timestamp into BW-formatted ("May 27 /26", "06:06:14") """Split an ISO timestamp into BW-formatted ('May 27 /26', '06:06:14')
date+time strings. Used for the histogram stats table where the date+time strings. Used for the histogram stats table where the
Date and Time rows are presented separately. Returns (None, None) Date and Time rows are presented separately. UTC inputs are
if the input isn't a valid ISO datetime.""" converted to local time first. Returns (None, None) on parse failure."""
if not iso: if not iso:
return (None, None) return (None, None)
try: try:
import datetime as _dt dt = _to_display_local(iso)
dt = _dt.datetime.fromisoformat(iso.replace("Z", "+00:00")) # BW format: 'May 27 /26' (3-letter month + 2-digit year)
# BW format: "May 27 /26" (3-letter month + 2-digit year)
date_str = dt.strftime("%b %d /%y").replace(" 0", " ") date_str = dt.strftime("%b %d /%y").replace(" 0", " ")
time_str = dt.strftime("%H:%M:%S") time_str = dt.strftime("%H:%M:%S")
return (date_str, time_str) return (date_str, time_str)