34 Commits

Author SHA1 Message Date
serversdown ec7be4d784 Merge branch 'feature/intelligent-caching' 2026-04-17 03:46:22 +00:00
claude b8ed237363 docs: update to 0.12.1 2026-04-16 18:31:20 -04:00
claude 5866ecdb3e docs: update protocol doc to reflect unkown status of max_range_geo. 2026-04-16 18:17:16 -04:00
serversdown ea9c69b7c9 chore: add sqlalchemy to pyproject 2026-04-16 21:22:04 +00:00
claude 71bcf71cf7 fix: convert raw psi 32 float into db(L). 2026-04-16 21:22:04 +00:00
claude 3e7de848bc fix: update unique constraints in events and monitor_log tables to use timestamp and serial number. Can't use event keys because minimates resuse them after clearing memory. 2026-04-16 21:22:04 +00:00
claude 72a4209cfd fix: sfm_webapp.html remove display: flex from base class, now shows active tab 2026-04-16 21:22:04 +00:00
claude 2b5574511e feat: add waveform viewer endpoint and enhance UI with new tabs for history, units, monitor log, and sessions 2026-04-16 21:22:04 +00:00
claude ce2c859f11 fix: update event count retrieval logic in AchSession and MiniMateClient 2026-04-16 21:22:04 +00:00
claude 7f322f9ff9 feat: add option to restart monitoring after event download in AchSession 2026-04-16 21:22:04 +00:00
serversdown 42b7a88c3d chore: add python build artifacts to gitignore 2026-04-16 21:22:04 +00:00
claude c474db4f69 build: update build backend to setuptools.build_meta 2026-04-16 21:22:04 +00:00
claude 2765ee6ea7 build: add pyproject.toml for editable install 2026-04-16 21:22:04 +00:00
claude ef88240796 docs: update README to v0.12.0
Rewrites the v0.6.0 README to reflect current project state:
ACH server, SQLite DB, SFM REST API with caching, monitor/erase, updated roadmap.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:22:04 +00:00
claude 5591d345d9 feat: v0.12.0 — live device cache (_LiveCache) in sfm/server.py
Ports the intelligent-caching branch concept to a plain Python in-memory
implementation — no SQLAlchemy, no extra DB table, no new dependencies.

_LiveCache (threading.Lock + dicts) caches:
  - device info: indefinite, invalidated by POST /device/config
  - events: keyed by (conn_key, device_event_count); count-probe fast path
    (~2s poll+count_events) avoids full downloads when nothing is new
  - monitor status: 30-second TTL, invalidated by monitor start/stop
  - waveforms: permanent per (conn_key, event_index)

All four cached endpoints accept ?force=true to bypass the cache.
Removes sfm/cache.py (SQLAlchemy experiment, now superseded).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:22:04 +00:00
claude 7883a31aa7 v0.11.0 — SQLite persistence layer (SeismoDb)
sfm/database.py (new)
- SeismoDb class: three tables keyed by unit serial number
  - ach_sessions: one row per ACH call-home
  - events: one row per triggered event, deduped by (serial, waveform_key)
  - monitor_log: one row per monitoring interval, deduped by (serial, waveform_key)
- WAL mode, per-request connections, silent dedup via UNIQUE constraint
- Query helpers: query_events(), query_monitor_log(), get_sessions(), query_units()
- false_trigger flag on events for future review UI / report filtering

bridges/ach_server.py
- Import SeismoDb; create shared instance at startup pointed at
  bridges/captures/seismo_relay.db
- After each call-home: insert_events() + insert_monitor_log() + insert_ach_session()
- DB failures logged as warnings, never abort the session

sfm/server.py
- Import SeismoDb; lazy singleton via _get_db()
- New DB read endpoints: GET /db/units, /db/events, /db/monitor_log, /db/sessions
- PATCH /db/events/{id}/false_trigger for manual review flagging

CLAUDE.md / CHANGELOG.md
- Document DB schema, SFM DB endpoints, architecture decision (unit-keyed only)
- Version bump to v0.11.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:19:47 +00:00
claude b241da970d v0.10.0 — monitor log entry support (SUB 0x0A partial records)
Add full decode pipeline for 0x2C partial records from the device's event
list, representing continuous monitoring intervals where no threshold was
crossed.  These records appear interleaved with full triggered events in the
browse walk and were previously ignored.

minimateplus/models.py
- Add MonitorLogEntry dataclass: key, start_time, stop_time, serial,
  geo_threshold_ips, raw_header, duration_seconds property

minimateplus/protocol.py
- read_waveform_header() now returns (data_rsp.data, length) — full payload
  including the record-type byte at position 0 — instead of the sliced header.
  Callers that need the old slice use raw_data[11:11+length] as before.

minimateplus/client.py
- Add _decode_0a_partial_header(): auto-detects 9-byte (sub_code=0x10) vs
  10-byte (sub_code=0x03) timestamp format, handles 1-byte inter-timestamp
  gap, extracts serial via BE anchor and geo threshold via Geo: anchor.
- Add get_monitor_log_entries(skip_keys=None): browse walk (1E → 0A → 1F),
  decodes partial records, skips full records and already-seen keys.

minimateplus/__init__.py
- Export MonitorLogEntry

bridges/ach_server.py
- After get_events(), call get_monitor_log_entries(skip_keys=seen_keys) and
  save new entries to monitor_log.json in the session directory.
- Add _monitor_log_entry_to_dict() helper.
- Include monitor log keys in downloaded_keys for state persistence.

CLAUDE.md / CHANGELOG.md
- Document 0x2C partial record layout (timestamp format, ASCII metadata
  region, 1-byte gap edge case) confirmed from 4-11-26 MITM capture.
- Version bump to v0.10.0; update What's next.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 6acb419ebd docs: update protocol reference with v0.9.0 erase-all protocol
Changelog section:
- 5 new entries (2026-04-11): erase-all confirmation, SUB 0x06 purpose
  resolved, §7.11 added, §14.6 ACH session lifecycle marked IMPLEMENTED

§5.1 Request Commands:
- SUB 0x06 description updated: "EVENT STORAGE RANGE READ" (not "CHANNEL
  CONFIG READ"), token=0xFE, last 8 bytes = first/last stored event keys
- SUB 0xA3 added: ERASE ALL BEGIN — standard build_bw_frame, token=0xFE, ack 0x5C
- SUB 0xA2 added: ERASE ALL CONFIRM — standard build_bw_frame, token=0xFE, ack 0x5D

§5.2 Response SUBs:
- 0x06→0xF9 marked CONFIRMED 2026-04-11
- 0xA3→0x5C and 0xA2→0x5D added with CONFIRMED status

§7.11 (new section): Erase-All Protocol
- Full wire sequence (6 request/response pairs)
- SUB 0x06 storage range payload layout (36 bytes, last 8 = first/last key)
- Post-erase key counter reset: device restarts from 0x01110000
- Implementation notes pointing to client.py and ach_server.py

§14.6 ACH Session Lifecycle:
- Removed "Future" label — fully implemented in bridges/ach_server.py
- Added step 6 (optional erase), step 8 (DCD/DTR auto-resume)
- Documents ach_server.py flags and ach_state.json schema
- Notes RV55 DCD/DTR issue as known open problem

Open Questions table:
- SUB 0x06 purpose RESOLVED
- Erase-all sequence RESOLVED
- ACH server RESOLVED
- Sensor Check byte: still open, added as formal question
- RV55 DCD/DTR: added as new open question

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude f6a0846bab docs: update CHANGELOG and CLAUDE.md for v0.9.0
CHANGELOG.md:
- New v0.9.0 section covering erase-all protocol, browse helpers,
  delete_all_events(), ach_mitm.py, and ACH server overhaul
- Back-filled v0.8.0 section (write pipeline, monitoring, ACH server)
  that was missing from the previous release notes

CLAUDE.md:
- Bump version to v0.9.0
- Add erase-all protocol section with full wire sequence, SUB 0x06
  storage range response layout, and post-erase key counter reset notes
- Document ACH server state format (ach_state.json v0.9.0 schema with
  downloaded_keys + max_downloaded_key)
- Add RV55 DCD/DTR issue to What's next

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 3d9db8b662 feat: add ach_mitm.py — transparent TCP MITM proxy for ACH session capture
Listens for inbound unit connections, connects upstream to a real Blastware
ACH server, and forwards bytes bidirectionally while saving both directions to
raw_bw_<ts>.bin and raw_s3_<ts>.bin in the existing capture format.

Used to capture the 4-11-26 Blastware ACH session that confirmed the erase-all
protocol (SUBs 0xA3/0x1C/0x06/0xA2) and the event deletion wire sequence.

Usage:
  python bridges/ach_mitm.py --bw-host 127.0.0.1 --bw-port 9999 --listen-port 9998
  Point the unit's call-home destination at this machine:9998.
  Point this proxy's --bw-host/port at the upstream Blastware ACH server.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude c7e7d177e6 feat: overhaul ACH server with key-based state, erase support, and reset detection
State format (ach_state.json):
- Replace event_count with downloaded_keys (set of hex strings) + max_downloaded_key
- Key-based tracking correctly handles delete-then-re-record: after device erase the
  count drops to 0, but new events have new (or recycled) keys

Browse pre-check:
- list_event_keys() walk before get_events() to bail early when nothing is new
- get_events() called with skip_waveform_for_keys= for already-seen keys, so repeat
  call-homes only download waveforms for genuinely new events

--clear-after-download flag:
- After saving new events, calls client.delete_all_events() (0xA3→0x1C→0x06→0xA2)
- On success: resets downloaded_keys=[] and max_downloaded_key="00000000" so the
  next session starts fresh (device counter resets to 0x01110000 after erase)

Post-erase key-reuse detection:
- Device counter resets to 0x01110000 after any erase; new events reuse old keys
- If max(device_keys) < max_downloaded_key, the device was wiped externally
  (Blastware, manual) — seen_keys is discarded and all device keys treated as new

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude a3b8d10fa8 feat: add erase-all protocol and browse helpers to protocol/client layer
protocol.py:
- SUB_ERASE_ALL_BEGIN = 0xA3, SUB_ERASE_ALL_CONFIRM = 0xA2 (confirmed 4-11-26 MITM)
- SUB_CHANNEL_CONFIG (0x06) data length = 0x24 (36 bytes) in DATA_LENGTHS
- begin_erase_all()              — single frame, token=0xFE, response 0x5C
- confirm_erase_all()            — single frame, token=0xFE, response 0x5D
- read_event_storage_range()     — two-step read (probe+data), token=0xFE
  Response last 8 bytes = first/last stored event key; both 0x01110000 after erase

client.py:
- list_event_keys()              — browse-mode 1E→0A→1F walk, no waveform download;
  returns list of hex key strings; used as fast pre-check before get_events()
- get_events(skip_waveform_for_keys=set())
  — for already-seen keys: only 0A+1F(browse), skips 1E-arm/0C/POLL×3/5A entirely
- delete_all_events()            — orchestrates the confirmed erase sequence:
  0xA3 → 0x1C → 0x06 → 0xA2; logs first/last key from storage range response

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 4921b0489a fix: correct Event and PeakValues field names in ach_server serialization
Event model uses peak_values (not peaks) and project_info (not direct fields).
PeakValues fields are tran/vert/long/micl/peak_vector_sum (not transverse etc).
ProjectInfo fields accessed via ev.project_info.project etc.

Also fix ev.timestamp serialization: use str() instead of .isoformat() since
Timestamp is a custom dataclass, not datetime.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 8688d815a0 fix: remove non-existent DeviceInfo fields from ach_server log and dict
calibration_date, aux_trigger, setup_name etc. don't exist directly on
DeviceInfo — they live in DeviceInfo.compliance_config (ComplianceConfig).
_device_info_to_dict now accesses them via cc = d.compliance_config.
Log line updated to show serial/firmware/model/event_count instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 9b50ec9133 fix: make Ctrl-C work on Windows by setting accept() timeout
socket.accept() on Windows blocks indefinitely and ignores KeyboardInterrupt.
Setting a 1-second timeout on the server socket causes the accept loop to wake
up every second and re-check, so Ctrl-C is handled within ~1 second.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude cba8b1b401 feat: defer session dir creation and add --allow-ip allowlist
- Session directory and log file are now created ONLY after startup() succeeds.
  Internet scanners and dropped connections no longer litter the output folder.
  Raw bytes are buffered in memory until startup succeeds, then flushed to disk.

- Add --allow-ip IP flag (repeatable) to allowlist specific source IPs.
  Connections from un-listed IPs are rejected immediately (socket closed, no log).
  If no --allow-ip flags are given, all IPs are still accepted (original behavior).
  Usage: --allow-ip 63.43.212.232 --allow-ip 152.1.2.3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 41a14ca468 fix: correct event count field offset and eliminate count_events() walk
_decode_event_count: read uint16 BE at offset 10 (confirmed 2026-04-10 from
live BE11529 event index — data[10:12]=0x0006=6, matches device LCD).
Previous uint32 at offset 3 always returned 1 regardless of event count.

ach_server.py: use device_info.event_count (already fetched during connect())
instead of calling count_events() separately. This saves 2*N round-trips and
avoids the 1F linked-list walk which was overcounting on some devices.
count_events() kept as fallback when connect() is skipped (--events-only).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 1bfc6e4258 fix: replace Unicode chars in log messages, fix DeviceInfo.serial, UTF-8 file log
- Replace all Unicode arrows/checkmarks (->  [OK]  [FAIL]) in ach_server.py
  and client.py log calls — Windows cp1252 console can't encode them
- Fix DeviceInfo attribute: serial_number -> serial
- Fix _device_info_to_dict key: serial_number -> serial
- Demote count_events 1E/1F per-key log lines from WARNING to DEBUG
  (they were flooding the console on devices with many stored events)
- FileHandler now opens with encoding='utf-8' so session log files
  can hold any characters without codec errors

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 574d40027f feat: enhance logging messages in ach_server.py and add experiments.py for protocol minimization 2026-04-16 21:14:58 +00:00
claude 0358acb51d feat: add high-water mark state tracking to ach_server + fix monitoring flag
ach_server.py:
- Add ach_state.json per-unit state tracking (keyed by serial number)
- count_events() before any download; skip session if no new events since last call-home
- Download only events beyond the previous high-water mark (all_events[last_count:])
- --max-events N safety cap for first-run units with many stored events
- state_path and max_events wired through AchSession constructor and serve()

client.py (_decode_monitor_status):
- Revert monitoring flag to section[1] == 0x10 (was incorrectly changed to section[6])
- Fix battery/memory offsets to section[-10:-8], [-8:-4], [-4:] (no trailing checksum byte)
- Both confirmed by full byte diff of all 144 0xE3 data frames in 4-8-26/2ndtry capture

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude cf7d838bf4 feat: add SocketTransport and ach_server.py inbound ACH server
minimateplus/transport.py:
- Add SocketTransport(TcpTransport) — wraps an already-accepted inbound
  socket; connect() is a no-op; everything else inherited from TcpTransport.
  Enables the ACH server to reuse all existing protocol/client code without
  any changes.

bridges/ach_server.py:
- Minimal inbound ACH server — listens on port 12345, accepts call-home
  connections from MiniMate Plus units, runs the full BW protocol:
  startup handshake → get_device_info → get_events(full_waveform=True)
- Saves device_info.json + events.json + raw_rx_<ts>.bin + session log
  per connection to bridges/captures/ach_inbound_<ts>/
- raw_rx.bin is byte-compatible with existing Analyzer tooling
- Taps transport.read() to capture raw S3 bytes alongside parsed output
- Each connection runs in its own daemon thread
- Clearly distinguishes push vs pull protocol in the startup log

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 5e44cdc668 feat: add splitter mode to ach_bridge.py (--mirror HOST:PORT)
Adds a production-safe headphone-splitter mode:
- Device bytes tee'd to both --upstream (primary/prod) and --mirror (new server)
- Only primary server responses are returned to the device
- Mirror connect/write failures are non-fatal and logged; prod is unaffected
- New raw_mirror_<ts>.bin capture file alongside raw_client/raw_server

Three modes: standalone (capture only), bridge (one upstream), splitter (two).
Default listen port changed to 12345 to match project ACH setup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 37d32077a4 feat: add ACH TCP bridge, serial tap tool, and Serial Watch tab
- bridges/ach_bridge.py: transparent TCP bridge that MITMs the MiniMate Plus
  call-home connection — forwards to real ACH server while logging all frames
  to raw_client/raw_server .bin files compatible with parse_capture.py;
  standalone capture mode for lab use without a real server

- bridges/serial_watch.py: RS-232 serial monitor with live S3 frame parsing;
  taps the line between MiniMate and modem (RV50/RV55); captures raw bytes,
  .log and .jsonl; --ack-ok mode auto-replies to AT commands; fixed fatal
  indentation bug in the original that silently prevented any data capture

- seismo_lab.py: new "Serial Watch" fourth tab (SerialWatchPanel) wrapping
  serial_watch.py functionality; COM port picker with refresh, baud config,
  ack-ok toggle, colour-coded live frame log (teal frames / yellow ctrl /
  blue AT), raw .bin capture auto-fed into Analyzer tab on stop

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
serversdown 2db565ff9c Add intelligent caching layer for SFM device data
Introduces sfm/cache.py — a SQLite-backed cache (via SQLAlchemy) that
sits between the SFM REST endpoints and the device, eliminating redundant
cellular downloads for data that doesn't change.

Cache behaviour by data type:
- Device info / compliance config: cached until a config write occurs;
  POST /device/config now calls mark_config_dirty() to force a fresh read
  on the next /device/info call.
- Event headers + peak values: cached permanently (append-only). On
  subsequent calls to /device/events, the server does a fast count_events()
  (~2s) instead of a full download (~10-30s); only new events are fetched
  from the device and merged into the cache.
- Full waveforms (raw ADC samples): cached permanently — immutable once
  recorded. Repeated requests for the same waveform return instantly with
  zero device contact.
- Monitor status (battery, memory, is_monitoring): 30-second TTL; auto-
  invalidated on start/stop monitoring commands.

All endpoints gain a ?force=true param to bypass the cache when needed.
New endpoints: GET /cache/stats, DELETE /cache/device.
Adds requirements.txt listing fastapi, uvicorn, sqlalchemy, pyserial.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 07:14:51 +00:00
7 changed files with 658 additions and 92 deletions
+26
View File
@@ -4,6 +4,32 @@ All notable changes to seismo-relay are documented here.
--- ---
## v0.12.1 — 2026-04-16
### Added
- **`sfm/server.py``_LiveCache`** — in-memory live device cache that eliminates
redundant TCP round-trips between web requests. Plain Python dict +
`threading.Lock`, no extra dependencies.
Cache strategy per endpoint:
| Endpoint | Strategy |
|---|---|
| `GET /device/info` | Indefinite; invalidated by `POST /device/config` |
| `GET /device/events` | Count-probe fast path — `poll()+count_events()` (~2 s); returns cached data if event count is unchanged; full download only when new events are detected |
| `GET /device/monitor/status` | 30-second TTL; invalidated immediately on monitor start/stop |
| `GET /device/event/{idx}/waveform` | Permanent per-index (waveforms are immutable once recorded) |
- **`?force=true`** query param on all cached endpoints — bypasses cache and forces
a fresh read from the device.
- **Cache invalidation hooks** — `POST /device/config` marks device info and events
stale; `POST /device/monitor/start` and `/stop` evict the monitor status entry
immediately so the next status poll reflects the actual device state.
---
## v0.12.0 — 2026-04-13 ## v0.12.0 — 2026-04-13
### Added ### Added
+4 -3
View File
File diff suppressed because one or more lines are too long
+10 -10
View File
@@ -36,7 +36,7 @@
| 2026-03-02 | §7.4 Event Index Block | **NEW:** `Monitoring LCD Cycle` identified at offsets +84/+85 as uint16 BE. Default value = 65500 (0xFFDC) = effectively disabled / maximum. Confirmed from operator manual §3.13.1g. | | 2026-03-02 | §7.4 Event Index Block | **NEW:** `Monitoring LCD Cycle` identified at offsets +84/+85 as uint16 BE. Default value = 65500 (0xFFDC) = effectively disabled / maximum. Confirmed from operator manual §3.13.1g. |
| 2026-03-02 | §7.4 Event Index Block | **UPDATED:** Backlight confirmed as uint8 range 0255 seconds per operator manual §3.13.1e ("adjustable timer, 0 to 255 seconds"). Power save unit confirmed as minutes per operator manual §3.13.1f. | | 2026-03-02 | §7.4 Event Index Block | **UPDATED:** Backlight confirmed as uint8 range 0255 seconds per operator manual §3.13.1e ("adjustable timer, 0 to 255 seconds"). Power save unit confirmed as minutes per operator manual §3.13.1f. |
| 2026-03-02 | Global | **NEW SOURCE:** Operator manual (716U0101 Rev 15) added as reference. Cross-referencing settings definitions, ranges, and units. Header updated. | | 2026-03-02 | Global | **NEW SOURCE:** Operator manual (716U0101 Rev 15) added as reference. Cross-referencing settings definitions, ranges, and units. Header updated. |
| 2026-03-02 | §14 Open Questions | Float 6.2061 in/s mystery: manual confirms only two geo ranges (1.25 in/s and 10.0 in/s). 6.2061 is NOT a user-selectable range → likely internal ADC full-scale calibration constant or hardware range ceiling. Downgraded to LOW priority. | | 2026-03-02 | §14 Open Questions | Float 6.2061 in/s mystery: manual confirms only two geo ranges (1.25 in/s and 10.0 in/s). 6.2061 is NOT a user-selectable range → originally speculated as internal ADC full-scale constant, but this is NOT confirmed. Using it as ADC full-scale produces ~9× PPV overread. Meaning unknown. Downgraded to LOW 2026-03-02, re-escalated to HIGH 2026-04-16. |
| 2026-03-02 | §14 Open Questions | `0x082A` hypothesis refined: 2090 decimal. At 1024 sps, 2 sec record = 2048 samples. Possible that 0x082A = total samples including 0.25s pre-trigger (256 samples) at some adjusted rate. Needs capture with different record time. | | 2026-03-02 | §14 Open Questions | `0x082A` hypothesis refined: 2090 decimal. At 1024 sps, 2 sec record = 2048 samples. Possible that 0x082A = total samples including 0.25s pre-trigger (256 samples) at some adjusted rate. Needs capture with different record time. |
| 2026-03-02 | §14 Open Questions | **NEW items added:** Trigger sample width (default=2), Auto Window (1-9 sec), Aux Trigger (enabled/disabled) — all confirmed settings from operator manual not yet mapped in protocol. | | 2026-03-02 | §14 Open Questions | **NEW items added:** Trigger sample width (default=2), Auto Window (1-9 sec), Aux Trigger (enabled/disabled) — all confirmed settings from operator manual not yet mapped in protocol. |
| 2026-03-02 | §14 Open Questions | Monitoring LCD Cycle resolved — removed from open questions. | | 2026-03-02 | §14 Open Questions | Monitoring LCD Cycle resolved — removed from open questions. |
@@ -92,7 +92,7 @@
| 2026-04-06 | §7.8.4 | **NEW — 5A end-of-stream signalling confirmed.** After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to the next chunk request, then goes silent for the full recv timeout. This byte is NOT a complete DLE-framed A5 response — the frame parser accumulates it as `bytes_fed=1` and never assembles a frame. This is the device's natural end-of-stream signal. Handling: on TimeoutError, if `bytes_fed > 0` AND prior chunks were received, treat as graceful end and proceed to the termination frame. A `bytes_fed=0` timeout with no prior chunks is a genuine transport failure and must still raise. | | 2026-04-06 | §7.8.4 | **NEW — 5A end-of-stream signalling confirmed.** After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to the next chunk request, then goes silent for the full recv timeout. This byte is NOT a complete DLE-framed A5 response — the frame parser accumulates it as `bytes_fed=1` and never assembles a frame. This is the device's natural end-of-stream signal. Handling: on TimeoutError, if `bytes_fed > 0` AND prior chunks were received, treat as graceful end and proceed to the termination frame. A `bytes_fed=0` timeout with no prior chunks is a genuine transport failure and must still raise. |
| 2026-04-06 | §7.8.4 | **NEW — 5A chunk timing and count (empirical, BE11529 at 1024 sps).** Each chunk response arrives within ~1 second over TCP/cellular. A 9,306-sample event (≈9.1 s at 1024 sps) produces **35 chunks** before end-of-stream. Chunks 116 have varying data lengths (10361123 bytes); chunks 1735 are uniformly 1036 bytes each (post-event silence, all-zero ADC samples). Safe recv timeout for chunk loop: **10 s** (10× typical response time). Default transport timeout (120 s) results in a ~2-minute stall per event at end-of-stream. | | 2026-04-06 | §7.8.4 | **NEW — 5A chunk timing and count (empirical, BE11529 at 1024 sps).** Each chunk response arrives within ~1 second over TCP/cellular. A 9,306-sample event (≈9.1 s at 1024 sps) produces **35 chunks** before end-of-stream. Chunks 116 have varying data lengths (10361123 bytes); chunks 1735 are uniformly 1036 bytes each (post-event silence, all-zero ADC samples). Safe recv timeout for chunk loop: **10 s** (10× typical response time). Default transport timeout (120 s) results in a ~2-minute stall per event at end-of-stream. |
| 2026-04-06 | §7.8.3 | **KNOWN ISSUE — `_decode_a5_waveform` hardcoded fi==9 skip.** The decoder contains `elif fi == 9: continue` which was written for the 9-frame original blast capture where frame 9 was a device terminator. For streams with >9 frames (current device produces 35+), frame index 9 is live waveform data — this skip discards ~1,070 bytes (~133 sample-sets) per event. The terminator is now detected via `page_key == 0x0000`, not by frame index. The fi==9 skip should be removed. | | 2026-04-06 | §7.8.3 | **KNOWN ISSUE — `_decode_a5_waveform` hardcoded fi==9 skip.** The decoder contains `elif fi == 9: continue` which was written for the 9-frame original blast capture where frame 9 was a device terminator. For streams with >9 frames (current device produces 35+), frame index 9 is live waveform data — this skip discards ~1,070 bytes (~133 sample-sets) per event. The terminator is now detected via `page_key == 0x0000`, not by frame index. The fi==9 skip should be removed. |
| 2026-04-06 | §7.8 | **CONFIRMED — ADC count-to-physical-unit conversion.** Raw waveform samples are signed 16-bit integers (counts). Conversion: `value = counts × (range / 32767)`. For geo channels: range = 10.000 in/s (from the device's compliance config geo range field). For the mic channel: range is in psi (device-specific). Near-full-scale counts (≈32,700) on all four channels simultaneously indicate ADC saturation (clipping) from a high-amplitude event. | | 2026-04-06 | §7.8 | **⚠ PARTIALLY INVALIDATED — ADC count-to-physical-unit conversion.** Raw waveform samples are signed 16-bit integers (counts). Conversion formula `value = counts × (range / 32767)` is believed correct, but the `range` value is UNKNOWN. The compliance config field labeled `max_range_geo` reads 6.206053 (bytes `40 C6 97 FD`), which does NOT match either user-selectable range shown in Blastware UI (1.25 or 10.000 in/s). The meaning and units of the 6.206053 value are unresolved — it may not be the ADC full-scale at all. See open question in §14. |
| 2026-04-08 | §5.1, §7.10, §12 | **NEW — Monitoring commands confirmed.** SUB 0x1C (monitor status), 0x96 (start monitoring), 0x97 (stop monitoring) all confirmed from 4-8-26/2ndtry capture. SESSION_RESET (`41 03`) required before POLL to wake a monitoring unit. | | 2026-04-08 | §5.1, §7.10, §12 | **NEW — Monitoring commands confirmed.** SUB 0x1C (monitor status), 0x96 (start monitoring), 0x97 (stop monitoring) all confirmed from 4-8-26/2ndtry capture. SESSION_RESET (`41 03`) required before POLL to wake a monitoring unit. |
| 2026-04-09 | §7.10 | **CORRECTED — monitoring flag and battery/memory offsets.** `section[1] == 0x10` is the monitoring flag (100% accurate across 144 data frames in 2ndtry capture). Previous note claiming `section[6]` was wrong — section[6] has device-specific non-binary values (0xea/0x07). Battery/memory offsets corrected: `section[-10:-8]` (battery×100), `section[-8:-4]` (memory_total), `section[-4:]` (memory_free). NOTE: `frame.data` has checksum stripped by parser — earlier offsets of `[-11:-9]`/`[-9:-5]`/`[-5:-1]` were wrong because they assumed a trailing checksum byte that isn't there. | | 2026-04-09 | §7.10 | **CORRECTED — monitoring flag and battery/memory offsets.** `section[1] == 0x10` is the monitoring flag (100% accurate across 144 data frames in 2ndtry capture). Previous note claiming `section[6]` was wrong — section[6] has device-specific non-binary values (0xea/0x07). Battery/memory offsets corrected: `section[-10:-8]` (battery×100), `section[-8:-4]` (memory_total), `section[-4:]` (memory_free). NOTE: `frame.data` has checksum stripped by parser — earlier offsets of `[-11:-9]`/`[-9:-5]`/`[-5:-1]` were wrong because they assumed a trailing checksum byte that isn't there. |
| 2026-04-08 | §7.10 | **NEW — SUBs 0x0E (channel sensor data) and 0x98 (trigger test) observed** in 4-8-26/sensor-check capture (Blastware "Unit Channel Test" comms check). SUB 0x0E: 2-step read with channel selector in `params[6:8]`, data length 0x0A per channel, RSP SUB = 0xF1. SUB 0x98: single probe frame with `params[0] = 0xFF`, RSP SUB = 0x67; sent twice per test cycle. Not yet implemented in SFM. | | 2026-04-08 | §7.10 | **NEW — SUBs 0x0E (channel sensor data) and 0x98 (trigger test) observed** in 4-8-26/sensor-check capture (Blastware "Unit Channel Test" comms check). SUB 0x0E: 2-step read with channel selector in `params[6:8]`, data length 0x0A per channel, RSP SUB = 0xF1. SUB 0x98: single probe frame with `params[0] = 0xFF`, RSP SUB = 0x67; sent twice per test cycle. Not yet implemented in SFM. |
@@ -528,7 +528,7 @@ The SUB `1A` read response (`E5`) and SUB `71` write block contain per-channel t
| Field | Example bytes | Decoded | Certainty | | Field | Example bytes | Decoded | Certainty |
|---|---|---|---| |---|---|---|---|
| `[00 00]` | `00 00` | Separator / padding | 🔶 INFERRED | | `[00 00]` | `00 00` | Separator / padding | 🔶 INFERRED |
| Max range float | `40 C6 97 FD` | 6.206 — full-scale range in in/s | 🔶 INFERRED | | Max range float | `40 C6 97 FD` | 6.206 — **value confirmed, meaning and units UNKNOWN** (does NOT match UI range options 1.25/10.000 in/s; not confirmed as ADC full-scale) | ❓ UNKNOWN |
| `[00 00]` | `00 00` | Separator / padding | 🔶 INFERRED | | `[00 00]` | `00 00` | Separator / padding | 🔶 INFERRED |
| **Trigger level** | `3F 19 99 9A` | **0.600 in/s** — IEEE 754 BE float | ✅ CONFIRMED | | **Trigger level** | `3F 19 99 9A` | **0.600 in/s** — IEEE 754 BE float | ✅ CONFIRMED |
| Unit string | `69 6E 2E 00` | `"in.\0"` | ✅ CONFIRMED | | Unit string | `69 6E 2E 00` | `"in.\0"` | ✅ CONFIRMED |
@@ -655,7 +655,7 @@ offset size type value (Tran example) meaning
+10 2 uint16 0x0015 = 21 unknown +10 2 uint16 0x0015 = 21 unknown
+12 4 bytes 03 02 04 01 flags (recording mode etc.) +12 4 bytes 03 02 04 01 flags (recording mode etc.)
+16 4 uint32 0x00000003 record time in seconds ✅ CONFIRMED +16 4 uint32 0x00000003 record time in seconds ✅ CONFIRMED
+1A 4 float32 6.2061 max range (in/s for geo, psi for mic) +1A 4 float32 6.2061 ❓ UNKNOWN field — value 6.2061 confirmed; meaning/units unresolved (NOT confirmed as max range or ADC full-scale)
+1E 2 00 00 padding +1E 2 00 00 padding
+20 4 float32 0.6000 trigger level ✅ CONFIRMED +20 4 float32 0.6000 trigger level ✅ CONFIRMED
+24 4 char[4] "in.\0" / "psi\0" unit string (geo vs mic) +24 4 char[4] "in.\0" / "psi\0" unit string (geo vs mic)
@@ -1235,13 +1235,13 @@ TimeoutError caught:
Chunks with uniform 1,036-byte payload (chunks 1735 in the observed event) contain all-zero ADC samples — the device continues recording silence until the configured record time expires before terminating the stream. Chunks with uniform 1,036-byte payload (chunks 1735 in the observed event) contain all-zero ADC samples — the device continues recording silence until the configured record time expires before terminating the stream.
**ADC count-to-physical conversion:** **ADC count-to-physical conversion — ⚠ SCALING UNKNOWN:**
Raw samples are signed 16-bit integers (32,768 to +32,767). To convert to physical units: Raw samples are signed 16-bit integers (32,768 to +32,767). The conversion formula is believed to be:
``` ```
value_in_s (in/s) = counts × (geo_range / 32767) value_in_s (in/s) = counts × (geo_range / 32767)
``` ```
where `geo_range` is from the compliance config (typically 10.000 in/s). Mic channel uses psi units with its own range. Near-full-scale values on all channels simultaneously indicate ADC saturation (clipping). However, the correct value of `geo_range` is **unknown**. The compliance config field `max_range_geo` reads 6.206053 (`40 C6 97 FD`) which does NOT match either user-selectable range (1.25 or 10.000 in/s) and produces ~9× too large PPV values compared to the on-device 0C record. Do not use 6.206053 or 10.000 as the scale factor until this is resolved. See §14 open question. Mic channel uses psi units with its own range (also unresolved).
**Known decoder issue — fi==9 hardcoded skip:** **Known decoder issue — fi==9 hardcoded skip:**
@@ -1267,7 +1267,7 @@ Fields visible in the Blastware "Compliance Setup" dialog. ✅ = byte offset co
| Geophone — Enable all | bool | ❓ | | Geophone — Enable all | bool | ❓ |
| Geophone — Trigger Source | bool | ❓ | | Geophone — Trigger Source | bool | ❓ |
| Chan 1-3 Trigger Level | float, in/s | ✅ `trigger_level_geo` | | Chan 1-3 Trigger Level | float, in/s | ✅ `trigger_level_geo` |
| Chan 1-3 Maximum Range | Normal 10.000 / 1.25 in/s | `max_range_geo` | | Chan 1-3 Maximum Range | Normal 10.000 / 1.25 in/s | `max_range_geo` offset found, value=6.206053 — does NOT match UI values; meaning unknown |
| Microphone — Enable all | bool | ❓ | | Microphone — Enable all | bool | ❓ |
| Microphone — Trigger Source | bool | ❓ | | Microphone — Trigger Source | bool | ❓ |
| Chan 4 Trigger Level | float, dB or psi | ❓ | | Chan 4 Trigger Level | float, dB or psi | ❓ |
@@ -1933,7 +1933,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
| **Auxiliary Trigger read location****RESOLVED:** SUB `FE` offset `0x0109`, uint8, `0x00`=disabled, `0x01`=enabled. Confirmed 2026-03-11 via controlled toggle capture. | RESOLVED | 2026-03-02 | Resolved 2026-03-11 | | **Auxiliary Trigger read location****RESOLVED:** SUB `FE` offset `0x0109`, uint8, `0x00`=disabled, `0x01`=enabled. Confirmed 2026-03-11 via controlled toggle capture. | RESOLVED | 2026-03-02 | Resolved 2026-03-11 |
| **Auxiliary Trigger write path** — Write command not yet captured in a clean session. Inner frame handshake visible in A4 (multiple WRITE_CONFIRM_RESPONSE SUBs appear, TRIGGER_CONFIG_RESPONSE removed), but the BW→S3 write command itself was in a partial session. Likely SUB `15` or similar. Deferred for clean capture. | LOW | 2026-03-11 | NEW | | **Auxiliary Trigger write path** — Write command not yet captured in a clean session. Inner frame handshake visible in A4 (multiple WRITE_CONFIRM_RESPONSE SUBs appear, TRIGGER_CONFIG_RESPONSE removed), but the BW→S3 write command itself was in a partial session. Likely SUB `15` or similar. Deferred for clean capture. | LOW | 2026-03-11 | NEW |
| ~~**SUB `6E` response to SUB `1C`**~~~~RESOLVED 2026-04-08: This was a misidentification.~~ The `1C → 6E` "exception" was misread — likely an inner A4 sub-frame. Confirmed from 4-8-26 capture (338 frames): SUB 0x1C always → 0xE3. No exceptions to the `0xFF SUB` rule are known. | RESOLVED | 2026-04-08 | CLOSED | | ~~**SUB `6E` response to SUB `1C`**~~~~RESOLVED 2026-04-08: This was a misidentification.~~ The `1C → 6E` "exception" was misread — likely an inner A4 sub-frame. Confirmed from 4-8-26 capture (338 frames): SUB 0x1C always → 0xE3. No exceptions to the `0xFF SUB` rule are known. | RESOLVED | 2026-04-08 | CLOSED |
| **Max Geo Range float 6.2061 in/s**NOT a user-selectable range (manual only shows 1.25 and 10.0 in/s). Likely internal ADC full-scale constant or hardware range ceiling. Not worth capturing. | LOW | 2026-02-26 | Downgraded 2026-03-02 | | **Max Geo Range float 6.2061**offset confirmed in channel block (`+1A`, `40 C6 97 FD`). Meaning and units are UNKNOWN. Value does NOT match either user-selectable range (1.25 / 10.0 in/s). Using it as ADC full-scale produces ~9× PPV overread vs on-device 0C values. Not simply metric vs imperial (25.4 factor doesn't reconcile). Needs investigation: examine surrounding channel block bytes, compare with a Blastware waveform CSV export to back-calculate the correct scale. Upgraded to HIGH priority. | HIGH | 2026-02-26 | Upgraded 2026-04-16 |
| MicL channel units — **RESOLVED: psi**, confirmed from `.set` file unit string `"psi\0"` | RESOLVED | 2026-03-01 | | | MicL channel units — **RESOLVED: psi**, confirmed from `.set` file unit string `"psi\0"` | RESOLVED | 2026-03-01 | |
| Backlight offset — **RESOLVED: +4B in event index data**, uint8, seconds | RESOLVED | 2026-03-02 | | | Backlight offset — **RESOLVED: +4B in event index data**, uint8, seconds | RESOLVED | 2026-03-02 | |
| Power save offset — **RESOLVED: +53 in event index data**, uint8, minutes | RESOLVED | 2026-03-02 | | | Power save offset — **RESOLVED: +53 in event index data**, uint8, minutes | RESOLVED | 2026-03-02 | |
@@ -1962,7 +1962,7 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
| Trigger Level (Mic) | §3.8.6 | Channel block, float | float32 BE | 100148 dB in 1 dB steps | | Trigger Level (Mic) | §3.8.6 | Channel block, float | float32 BE | 100148 dB in 1 dB steps |
| Alarm Level (Mic) | §3.9.10 | Channel block, float | float32 BE | higher than mic trigger | | Alarm Level (Mic) | §3.9.10 | Channel block, float | float32 BE | higher than mic trigger |
| Record Time | §3.8.9 | cfg anchor+10, float32 BE (wire); `.set` +16, uint32 LE (file) | float32 BE (wire) | 1105 s; confirmed 3→`40400000`, 5→`40A00000`, 8→`41000000`, 13→`41500000`. Use anchor §7.6.1/§7.6.3 — NOT fixed offset. | | Record Time | §3.8.9 | cfg anchor+10, float32 BE (wire); `.set` +16, uint32 LE (file) | float32 BE (wire) | 1105 s; confirmed 3→`40400000`, 5→`40A00000`, 8→`41000000`, 13→`41500000`. Use anchor §7.6.1/§7.6.3 — NOT fixed offset. |
| Max Geo Range | §3.8.4 | Channel block, float | float32 BE | 1.25 or 10.0 in/s (user); 6.2061 in protocol = internal constant | | Max Geo Range | §3.8.4 | Channel block, float | float32 BE | ❓ UNKNOWN — value 6.2061 confirmed at offset, but meaning/units unresolved. Does NOT equal 1.25 or 10.0 in/s. Do NOT use as ADC full-scale. |
| Microphone Units | §3.9.7 | Inline unit string | char[4] | `"psi\0"`, `"pa.\0"`, `"dB\0\0"` | | Microphone Units | §3.9.7 | Inline unit string | char[4] | `"psi\0"`, `"pa.\0"`, `"dB\0\0"` |
| Sample Rate | §3.8.2 | cfg anchor2, uint16 BE — anchor=`\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00` in cfg[40:100] | uint16 BE | Normal=1024, Fast=2048, Faster=4096 ✅ CONFIRMED 2026-04-01 (BE11529 S338.17). Anchor required — see §7.6.3 DLE jitter. | | Sample Rate | §3.8.2 | cfg anchor2, uint16 BE — anchor=`\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00` in cfg[40:100] | uint16 BE | Normal=1024, Fast=2048, Faster=4096 ✅ CONFIRMED 2026-04-01 (BE11529 S338.17). Anchor required — see §7.6.3 DLE jitter. |
| Record Mode | §3.8.1 | Unknown | — | Single Shot, Continuous, Manual, Histogram, Histogram Combo | | Record Mode | §3.8.1 | Unknown | — | Single Shot, Continuous, Manual, Histogram, Histogram Combo |
+1
View File
@@ -11,6 +11,7 @@ dependencies = [
"fastapi>=0.104", "fastapi>=0.104",
"uvicorn[standard]>=0.24", "uvicorn[standard]>=0.24",
"pyserial>=3.5", "pyserial>=3.5",
"sqlalchemy>=2.0",
] ]
[tool.setuptools.packages.find] [tool.setuptools.packages.find]
+4
View File
@@ -0,0 +1,4 @@
fastapi
uvicorn
sqlalchemy
pyserial
+376
View File
@@ -0,0 +1,376 @@
"""
sfm/cache.py Persistent SQLite cache for SFM device data.
Caching strategy
----------------
+------------------+----------------------------------+-------------------------+
| Data | Mutability | Invalidation |
+------------------+----------------------------------+-------------------------+
| Device info | Effectively immutable (firmware, | Manual clear / force |
| (serial, model, | serial never change) | refresh query param |
| compliance cfg) | | |
+------------------+----------------------------------+-------------------------+
| Event headers | Append-only (new events added, | Fetch new ones when |
| (peaks, ts, | old never modified) | device event count > |
| project info) | | cached count |
+------------------+----------------------------------+-------------------------+
| Full waveforms | Immutable once recorded | Never (permanent cache) |
| (raw ADC samples)| | |
+------------------+----------------------------------+-------------------------+
| Monitor status | Frequently changing | TTL = 30 seconds |
| (battery, memory)| | |
+------------------+----------------------------------+-------------------------+
Keys
----
All cached rows are keyed by (host, tcp_port) for TCP connections, or (port, baud)
for serial connections. Within a device, events are keyed by index (0-based).
The device serial number is stored once we learn it, and used for display / debugging
only the network address is the primary routing key (same as how the rest of the SFM
code operates).
"""
from __future__ import annotations
import json
import logging
import time
from pathlib import Path
from typing import Optional
try:
import sqlalchemy as sa
from sqlalchemy import orm
except ImportError:
raise ImportError(
"sqlalchemy is required for the SFM cache.\n"
"Install it with: pip install sqlalchemy"
)
log = logging.getLogger("sfm.cache")
# ── Schema ────────────────────────────────────────────────────────────────────
Base = orm.declarative_base()
_MONITOR_STATUS_TTL = 30 # seconds
class CachedDevice(Base):
"""
Device identity + compliance config, keyed by connection address.
Stores the full serialised JSON blob returned by /device/info so the
endpoint can return it verbatim on a cache hit without re-connecting.
"""
__tablename__ = "cached_devices"
# Connection key — either TCP (host+port) or serial (port+baud)
conn_key = sa.Column(sa.String, primary_key=True) # e.g. "tcp:1.2.3.4:12345"
serial = sa.Column(sa.String, nullable=True) # e.g. "BE11529"
info_json = sa.Column(sa.Text, nullable=False) # full /device/info response JSON
updated_at = sa.Column(sa.Float, nullable=False) # Unix timestamp of last write
# When a config write happens we set this flag so the next /device/info call
# fetches fresh data instead of serving stale compliance config.
config_dirty = sa.Column(sa.Boolean, default=False, nullable=False)
class CachedEvent(Base):
"""
Per-event header + peak values + project info, keyed by (conn_key, index).
Events are immutable once recorded on the device; once we have an event in
the cache it never needs to be re-downloaded unless explicitly requested.
"""
__tablename__ = "cached_events"
conn_key = sa.Column(sa.String, primary_key=True)
index = sa.Column(sa.Integer, primary_key=True)
event_json = sa.Column(sa.Text, nullable=False) # serialised Event dict
cached_at = sa.Column(sa.Float, nullable=False) # Unix timestamp
class CachedWaveform(Base):
"""
Full raw ADC waveform for a single event (SUB 5A full download).
These are large (up to several MB) and expensive to fetch over cellular.
Once downloaded they are immutable and cached permanently.
"""
__tablename__ = "cached_waveforms"
conn_key = sa.Column(sa.String, primary_key=True)
index = sa.Column(sa.Integer, primary_key=True)
waveform_json = sa.Column(sa.Text, nullable=False) # full /device/event/{idx}/waveform response JSON
cached_at = sa.Column(sa.Float, nullable=False)
class CachedMonitorStatus(Base):
"""
Monitor status (battery, memory, is_monitoring) with a short TTL.
These change frequently during field operations so we keep them only for
MONITOR_STATUS_TTL seconds before re-fetching from the device.
"""
__tablename__ = "cached_monitor_status"
conn_key = sa.Column(sa.String, primary_key=True)
status_json = sa.Column(sa.Text, nullable=False)
cached_at = sa.Column(sa.Float, nullable=False)
# ── Cache store ───────────────────────────────────────────────────────────────
class SFMCache:
"""
SQLite-backed cache for SFM device data.
Usage
-----
cache = SFMCache() # stores in sfm/data/sfm_cache.db by default
cache = SFMCache(":memory:") # in-memory (tests / ephemeral mode)
All public methods accept a *conn_key* string use make_conn_key() to
build a consistent key from the transport parameters.
"""
def __init__(self, db_path: str | Path | None = None) -> None:
in_memory = (db_path == ":memory:")
if db_path is None:
# Default: alongside this file in sfm/data/
db_path = Path(__file__).parent / "data" / "sfm_cache.db"
if not in_memory:
db_path = Path(db_path)
db_path.parent.mkdir(parents=True, exist_ok=True)
url = "sqlite:///:memory:" if in_memory else f"sqlite:///{db_path}"
engine = sa.create_engine(url, connect_args={"check_same_thread": False})
Base.metadata.create_all(engine)
self._Session = orm.sessionmaker(bind=engine)
log.info("SFM cache opened: %s", db_path)
# ── Connection key ────────────────────────────────────────────────────────
@staticmethod
def make_conn_key(
host: Optional[str],
tcp_port: int,
port: Optional[str],
baud: int,
) -> str:
"""Return a stable string key for this transport configuration."""
if host:
return f"tcp:{host}:{tcp_port}"
return f"serial:{port}:{baud}"
# ── Device info ───────────────────────────────────────────────────────────
def get_device_info(self, conn_key: str) -> Optional[dict]:
"""
Return cached device info dict, or None if not cached / config_dirty.
"""
with self._Session() as s:
row = s.get(CachedDevice, conn_key)
if row is None or row.config_dirty:
return None
return json.loads(row.info_json)
def set_device_info(self, conn_key: str, info: dict) -> None:
"""Store device info and clear any dirty flag."""
with self._Session() as s:
row = s.get(CachedDevice, conn_key)
serial = info.get("serial")
if row is None:
row = CachedDevice(
conn_key=conn_key,
serial=serial,
info_json=json.dumps(info),
updated_at=time.time(),
config_dirty=False,
)
s.add(row)
else:
row.serial = serial
row.info_json = json.dumps(info)
row.updated_at = time.time()
row.config_dirty = False
s.commit()
log.debug("cached device info for %s (serial=%s)", conn_key, serial)
def mark_config_dirty(self, conn_key: str) -> None:
"""
Called after a successful POST /device/config write.
Forces the next /device/info call to re-read compliance config from the
device instead of serving the now-stale cached version.
"""
with self._Session() as s:
row = s.get(CachedDevice, conn_key)
if row:
row.config_dirty = True
s.commit()
log.debug("marked config dirty for %s", conn_key)
# ── Events ────────────────────────────────────────────────────────────────
def get_cached_event_count(self, conn_key: str) -> int:
"""Return the number of events we have cached for this device."""
with self._Session() as s:
return s.query(CachedEvent).filter_by(conn_key=conn_key).count()
def get_all_events(self, conn_key: str) -> Optional[list[dict]]:
"""
Return all cached events as a list of dicts, sorted by index.
Returns None if nothing is cached yet.
"""
with self._Session() as s:
rows = (
s.query(CachedEvent)
.filter_by(conn_key=conn_key)
.order_by(CachedEvent.index)
.all()
)
if not rows:
return None
return [json.loads(r.event_json) for r in rows]
def get_event(self, conn_key: str, index: int) -> Optional[dict]:
"""Return a single cached event by index, or None if not cached."""
with self._Session() as s:
row = s.get(CachedEvent, (conn_key, index))
return json.loads(row.event_json) if row else None
def set_events(self, conn_key: str, events: list[dict]) -> None:
"""
Upsert a list of event dicts. Existing rows are updated; new rows are
inserted. This is used to add newly-discovered events to the cache.
"""
now = time.time()
with self._Session() as s:
for ev in events:
idx = ev["index"]
row = s.get(CachedEvent, (conn_key, idx))
if row is None:
row = CachedEvent(
conn_key=conn_key,
index=idx,
event_json=json.dumps(ev),
cached_at=now,
)
s.add(row)
log.debug("cached new event %d for %s", idx, conn_key)
else:
# Refresh in case project_info was backfilled after initial store
row.event_json = json.dumps(ev)
s.commit()
# ── Waveforms ─────────────────────────────────────────────────────────────
def get_waveform(self, conn_key: str, index: int) -> Optional[dict]:
"""Return a cached full waveform response dict, or None if not cached."""
with self._Session() as s:
row = s.get(CachedWaveform, (conn_key, index))
if row is None:
return None
log.debug("waveform cache hit: %s event %d", conn_key, index)
return json.loads(row.waveform_json)
def set_waveform(self, conn_key: str, index: int, waveform: dict) -> None:
"""Store a full waveform response dict permanently."""
with self._Session() as s:
row = s.get(CachedWaveform, (conn_key, index))
if row is None:
row = CachedWaveform(
conn_key=conn_key,
index=index,
waveform_json=json.dumps(waveform),
cached_at=time.time(),
)
s.add(row)
else:
row.waveform_json = json.dumps(waveform)
row.cached_at = time.time()
s.commit()
log.debug("cached waveform for %s event %d", conn_key, index)
# ── Monitor status ────────────────────────────────────────────────────────
def get_monitor_status(self, conn_key: str) -> Optional[dict]:
"""Return cached monitor status if it's within TTL, else None."""
with self._Session() as s:
row = s.get(CachedMonitorStatus, conn_key)
if row is None:
return None
age = time.time() - row.cached_at
if age > _MONITOR_STATUS_TTL:
log.debug("monitor status expired (age=%.1fs) for %s", age, conn_key)
return None
return json.loads(row.status_json)
def set_monitor_status(self, conn_key: str, status: dict) -> None:
"""Store monitor status."""
with self._Session() as s:
row = s.get(CachedMonitorStatus, conn_key)
if row is None:
row = CachedMonitorStatus(
conn_key=conn_key,
status_json=json.dumps(status),
cached_at=time.time(),
)
s.add(row)
else:
row.status_json = json.dumps(status)
row.cached_at = time.time()
s.commit()
def invalidate_monitor_status(self, conn_key: str) -> None:
"""
Called after start/stop monitoring so the next status poll re-reads from device.
"""
with self._Session() as s:
row = s.get(CachedMonitorStatus, conn_key)
if row:
s.delete(row)
s.commit()
# ── Cache management ──────────────────────────────────────────────────────
def clear_device(self, conn_key: str) -> dict:
"""
Remove all cached data for a device. Returns counts of deleted rows.
"""
counts = {}
with self._Session() as s:
counts["device_info"] = s.query(CachedDevice).filter_by(conn_key=conn_key).delete()
counts["events"] = s.query(CachedEvent).filter_by(conn_key=conn_key).delete()
counts["waveforms"] = s.query(CachedWaveform).filter_by(conn_key=conn_key).delete()
counts["monitor_status"] = s.query(CachedMonitorStatus).filter_by(conn_key=conn_key).delete()
s.commit()
log.info("cleared cache for %s: %s", conn_key, counts)
return counts
def stats(self) -> dict:
"""Return row counts for all cache tables (for /cache/stats endpoint)."""
with self._Session() as s:
return {
"devices": s.query(CachedDevice).count(),
"events": s.query(CachedEvent).count(),
"waveforms": s.query(CachedWaveform).count(),
"monitor_status": s.query(CachedMonitorStatus).count(),
}
# ── Module-level singleton ────────────────────────────────────────────────────
# Instantiated once when the module is imported; shared across all requests.
_cache: Optional[SFMCache] = None
def get_cache() -> SFMCache:
"""Return the module-level cache singleton, initialising it on first call."""
global _cache
if _cache is None:
_cache = SFMCache()
return _cache
+237 -79
View File
@@ -61,6 +61,7 @@ from minimateplus import MiniMateClient
from minimateplus.protocol import ProtocolError from minimateplus.protocol import ProtocolError
from minimateplus.models import ComplianceConfig, DeviceInfo, Event, PeakValues, ProjectInfo, Timestamp from minimateplus.models import ComplianceConfig, DeviceInfo, Event, PeakValues, ProjectInfo, Timestamp
from minimateplus.transport import TcpTransport, DEFAULT_TCP_PORT from minimateplus.transport import TcpTransport, DEFAULT_TCP_PORT
from sfm.cache import SFMCache, get_cache
from sfm.database import SeismoDb from sfm.database import SeismoDb
logging.basicConfig( logging.basicConfig(
@@ -388,6 +389,33 @@ def _run_with_retry(fn, *, is_tcp: bool):
return fn() # let any second failure propagate normally return fn() # let any second failure propagate normally
# ── Helpers ────────────────────────────────────────────────────────────────────
def _backfill_events(events: list, info: "DeviceInfo") -> None:
"""
Fill in sample_rate and project_info fields that the per-event waveform
record doesn't carry — sourced from the device's compliance config.
Extracted from device_events() so it can be called from both the full
download path and the partial (new-events-only) path.
"""
if info.compliance_config and info.compliance_config.sample_rate:
for ev in events:
if ev.sample_rate is None:
ev.sample_rate = info.compliance_config.sample_rate
if info.compliance_config:
cc = info.compliance_config
for ev in events:
if ev.project_info is None:
ev.project_info = ProjectInfo()
pi = ev.project_info
if pi.client is None: pi.client = cc.client
if pi.operator is None: pi.operator = cc.operator
if pi.sensor_location is None: pi.sensor_location = cc.sensor_location
if pi.notes is None: pi.notes = cc.notes
# ── Endpoints ────────────────────────────────────────────────────────────────── # ── Endpoints ──────────────────────────────────────────────────────────────────
@app.get("/health") @app.get("/health")
@@ -414,7 +442,7 @@ def device_info(
baud: int = Query(38400, description="Serial baud rate (default 38400)"), baud: int = Query(38400, description="Serial baud rate (default 38400)"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay (e.g. 203.0.113.5)"), host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay (e.g. 203.0.113.5)"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"), tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
force: bool = Query(False, description="Bypass cache and re-read from device"), force: bool = Query(False, description="Bypass cache and re-read from device"),
) -> dict: ) -> dict:
""" """
Connect to the device, perform the POLL startup handshake, and return Connect to the device, perform the POLL startup handshake, and return
@@ -423,16 +451,21 @@ def device_info(
Supply either *port* (serial) or *host* (TCP/modem). Supply either *port* (serial) or *host* (TCP/modem).
Equivalent to POST /device/connect provided as GET for convenience. Equivalent to POST /device/connect provided as GET for convenience.
Response is cached until a POST /device/config write invalidates it. **Caching**: device identity and compliance config are cached after the first
Pass ?force=true to bypass the cache. successful read (they rarely change). Pass *force=true* to bypass the cache
and re-read directly from the device (e.g. after a config push).
The cache is also automatically invalidated after POST /device/config.
""" """
conn_key = _live_cache.make_conn_key(host, tcp_port, port, baud)
log.info("GET /device/info port=%s host=%s tcp_port=%d force=%s", port, host, tcp_port, force) log.info("GET /device/info port=%s host=%s tcp_port=%d force=%s", port, host, tcp_port, force)
cache = get_cache()
conn_key = SFMCache.make_conn_key(host, tcp_port, port, baud)
if not force: if not force:
cached = _live_cache.get_device_info(conn_key) cached = cache.get_device_info(conn_key)
if cached is not None: if cached is not None:
log.debug("device_info cache hit for %s", conn_key) log.info("device info cache hit for %s", conn_key)
cached["_cached"] = True
return cached return cached
try: try:
@@ -454,7 +487,7 @@ def device_info(
raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc
result = _serialise_device_info(info) result = _serialise_device_info(info)
_live_cache.set_device_info(conn_key, result) cache.set_device_info(conn_key, result)
return result return result
@@ -478,8 +511,8 @@ def device_events(
baud: int = Query(38400, description="Serial baud rate"), baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"), host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"), tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
debug: bool = Query(False, description="Include raw record hex for field-layout inspection"), debug: bool = Query(False, description="Include raw record hex for field-layout inspection"),
force: bool = Query(False, description="Bypass cache and re-download from device"), force: bool = Query(False, description="Bypass cache and re-download all events from device"),
) -> dict: ) -> dict:
""" """
Connect to the device, read the event index, and download all stored Connect to the device, read the event index, and download all stored
@@ -497,38 +530,107 @@ def device_events(
This does NOT download raw ADC waveform samples those are large and This does NOT download raw ADC waveform samples those are large and
fetched separately via GET /device/event/{idx}/waveform. fetched separately via GET /device/event/{idx}/waveform.
**Caching**: event headers are cached after the first download. On subsequent
calls, the device is contacted only to check the event count (fast: ~2s).
If the count matches the cache, all events are returned from cache instantly.
If new events exist on the device, only the new ones are downloaded and merged.
Pass *force=true* to bypass the cache entirely and re-download everything.
""" """
conn_key = _live_cache.make_conn_key(host, tcp_port, port, baud)
log.info("GET /device/events port=%s host=%s debug=%s force=%s", port, host, debug, force) log.info("GET /device/events port=%s host=%s debug=%s force=%s", port, host, debug, force)
# ── Cache fast path ─────────────────────────────────────────────────────── cache = get_cache()
# Do a quick poll + count_events() probe (~2s over cellular) to check if the conn_key = SFMCache.make_conn_key(host, tcp_port, port, baud)
# device has any new events. If the count matches the cache, return early.
if not force and not debug:
try:
def _count():
with _build_client(port, baud, host, tcp_port) as client:
try:
client.poll()
except Exception:
client.poll()
return client.count_events()
device_count = _run_with_retry(_count, is_tcp=_is_tcp(host))
cached_events = _live_cache.get_events(conn_key, device_count)
if cached_events is not None:
log.info(" events cache hit (%d events, count=%d)", len(cached_events), device_count)
# Also serve cached device info if available
cached_info = _live_cache.get_device_info(conn_key)
return {
"device": cached_info or {},
"event_count": len(cached_events),
"events": cached_events,
"cached": True,
}
except Exception as exc:
log.warning(" count probe failed (%s) — falling through to full download", exc)
# ── Full download ───────────────────────────────────────────────────────── # ── Smart cache path (skip when debug=True or force=True) ────────────────
# debug mode uses raw_record_hex which isn't stored in the cache, so we
# must always go to the device when debug is requested.
if not force and not debug:
cached_events = cache.get_all_events(conn_key)
cached_count = len(cached_events) if cached_events else 0
if cached_count > 0:
# Quick device contact: just count events via the fast 1E/1F chain.
# This takes ~2s instead of the full event download (~10-30s).
try:
def _count():
with _build_client(port, baud, host, tcp_port) as client:
client.connect()
return client.count_events()
device_count = _run_with_retry(_count, is_tcp=_is_tcp(host))
except HTTPException:
raise
except (ProtocolError, OSError, Exception) as exc:
# If we can't reach the device at all, serve stale cache rather
# than returning an error — field units go offline regularly.
log.warning("count_events failed (%s) — serving stale cache for %s", exc, conn_key)
cached_info = cache.get_device_info(conn_key) or {}
return {
"device": cached_info,
"event_count": cached_count,
"events": cached_events,
"_cached": True,
"_stale": True,
}
if device_count == cached_count:
# Nothing new — return cache immediately, no event download needed.
log.info(
"event cache hit for %s: %d events, device count matches",
conn_key, cached_count,
)
cached_info = cache.get_device_info(conn_key) or {}
return {
"device": cached_info,
"event_count": cached_count,
"events": cached_events,
"_cached": True,
}
if device_count > cached_count:
# New events on the device — download all events but only store/return
# the new ones. Events are append-only; indices 0..(cached_count-1)
# are already in the cache and don't need to be re-downloaded logically,
# but the protocol requires iterating from event 0 to reach later ones.
# The device download time is dominated by the number of events requested,
# so we stop at the last known event index to avoid re-downloading everything.
log.info(
"new events on device %s: have %d, device has %d — fetching all up to %d",
conn_key, cached_count, device_count, device_count - 1,
)
try:
def _fetch_new():
with _build_client(port, baud, host, tcp_port) as client:
info = client.connect()
all_evs = client.get_events(stop_after_index=device_count - 1)
return info, all_evs
info, all_events = _run_with_retry(_fetch_new, is_tcp=_is_tcp(host))
except HTTPException:
raise
except ProtocolError as exc:
raise HTTPException(status_code=502, detail=f"Protocol error: {exc}") from exc
except OSError as exc:
raise HTTPException(status_code=502, detail=f"Connection error: {exc}") from exc
except Exception as exc:
raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc
_backfill_events(all_events, info)
# Only the new events (indices >= cached_count) are truly new.
new_events = [ev for ev in all_events if ev.index >= cached_count]
new_serialised = [_serialise_event(ev) for ev in new_events]
cache.set_events(conn_key, new_serialised)
cache.set_device_info(conn_key, _serialise_device_info(info))
merged_events = cache.get_all_events(conn_key)
return {
"device": _serialise_device_info(info),
"event_count": len(merged_events),
"events": merged_events,
"_cached": True,
"_new_events": len(new_events),
}
# ── Full download path (first call, force=True, or debug=True) ───────────
try: try:
def _do(): def _do():
with _build_client(port, baud, host, tcp_port) as client: with _build_client(port, baud, host, tcp_port) as client:
@@ -543,23 +645,14 @@ def device_events(
except Exception as exc: except Exception as exc:
raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc
# Fill sample_rate from compliance config where the event record doesn't supply it. _backfill_events(events, info)
if info.compliance_config and info.compliance_config.sample_rate: serialised = [_serialise_event(ev, debug=debug) for ev in events]
for ev in events:
if ev.sample_rate is None:
ev.sample_rate = info.compliance_config.sample_rate
# Backfill event.project_info fields that the 210-byte waveform record doesn't carry. if not debug:
if info.compliance_config: # Only cache when not in debug mode (debug adds raw_record_hex which
cc = info.compliance_config # we don't want polluting the normal cache entries).
for ev in events: cache.set_events(conn_key, serialised)
if ev.project_info is None: cache.set_device_info(conn_key, _serialise_device_info(info))
ev.project_info = ProjectInfo()
pi = ev.project_info
if pi.client is None: pi.client = cc.client
if pi.operator is None: pi.operator = cc.operator
if pi.sensor_location is None: pi.sensor_location = cc.sensor_location
if pi.notes is None: pi.notes = cc.notes
serialised_info = _serialise_device_info(info) serialised_info = _serialise_device_info(info)
serialised_events = [_serialise_event(ev, debug=debug) for ev in events] serialised_events = [_serialise_event(ev, debug=debug) for ev in events]
@@ -572,8 +665,7 @@ def device_events(
return { return {
"device": serialised_info, "device": serialised_info,
"event_count": len(events), "event_count": len(events),
"events": serialised_events, "events": serialised,
"cached": False,
} }
@@ -584,21 +676,36 @@ def device_event(
baud: int = Query(38400, description="Serial baud rate"), baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"), host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"), tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
force: bool = Query(False, description="Bypass cache and re-download from device"),
) -> dict: ) -> dict:
""" """
Download a single event by index (0-based). Download a single event by index (0-based).
Supply either *port* (serial) or *host* (TCP/modem). Supply either *port* (serial) or *host* (TCP/modem).
Performs: POLL startup event index event header waveform record. Performs: POLL startup event index event header waveform record.
**Caching**: if this event was already downloaded (e.g. via GET /device/events),
it is returned instantly from cache with no device contact.
""" """
log.info("GET /device/event/%d port=%s host=%s", index, port, host) log.info("GET /device/event/%d port=%s host=%s force=%s", index, port, host, force)
cache = get_cache()
conn_key = SFMCache.make_conn_key(host, tcp_port, port, baud)
if not force:
cached = cache.get_event(conn_key, index)
if cached is not None:
log.info("event cache hit for %s index %d", conn_key, index)
cached["_cached"] = True
return cached
try: try:
def _do(): def _do():
with _build_client(port, baud, host, tcp_port) as client: with _build_client(port, baud, host, tcp_port) as client:
client.connect() info = client.connect()
return client.get_events(stop_after_index=index) events = client.get_events(stop_after_index=index)
events = _run_with_retry(_do, is_tcp=_is_tcp(host)) return info, events
info, events = _run_with_retry(_do, is_tcp=_is_tcp(host))
except HTTPException: except HTTPException:
raise raise
except ProtocolError as exc: except ProtocolError as exc:
@@ -615,7 +722,14 @@ def device_event(
detail=f"Event index {index} not found on device", detail=f"Event index {index} not found on device",
) )
return _serialise_event(matching[0]) _backfill_events(matching, info)
result = _serialise_event(matching[0])
# Store all downloaded events (we paid for them anyway — indices 0..index)
all_serialised = [_serialise_event(ev) for ev in events]
cache.set_events(conn_key, all_serialised)
return result
@app.get("/device/event/{index}/waveform") @app.get("/device/event/{index}/waveform")
@@ -625,7 +739,7 @@ def device_event_waveform(
baud: int = Query(38400, description="Serial baud rate"), baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"), host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"), tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
force: bool = Query(False, description="Bypass cache and re-download waveform"), force: bool = Query(False, description="Bypass cache and re-download from device"),
) -> dict: ) -> dict:
""" """
Download the full raw ADC waveform for a single event (0-based index). Download the full raw ADC waveform for a single event (0-based index).
@@ -645,17 +759,22 @@ def device_event_waveform(
- **channels**: dict of channel name list of signed int16 ADC counts - **channels**: dict of channel name list of signed int16 ADC counts
(keys: "Tran", "Vert", "Long", "Mic") (keys: "Tran", "Vert", "Long", "Mic")
Waveforms are immutable once recorded and are cached permanently per **Caching**: full waveforms are cached permanently after the first download
(connection, event index). Pass ?force=true to re-download. they are immutable once recorded on the device. Subsequent requests for the
same event return instantly from cache without any device contact.
Pass *force=true* to force a fresh download (rarely needed).
""" """
conn_key = _live_cache.make_conn_key(host, tcp_port, port, baud)
log.info("GET /device/event/%d/waveform port=%s host=%s force=%s", index, port, host, force) log.info("GET /device/event/%d/waveform port=%s host=%s force=%s", index, port, host, force)
cache = get_cache()
conn_key = SFMCache.make_conn_key(host, tcp_port, port, baud)
if not force: if not force:
cached_waveform = _live_cache.get_waveform(conn_key, index) cached = cache.get_waveform(conn_key, index)
if cached_waveform is not None: if cached is not None:
log.debug("waveform cache hit: %s event %d", conn_key, index) log.info("waveform cache hit for %s event %d", conn_key, index)
return cached_waveform cached["_cached"] = True
return cached
try: try:
def _do(): def _do():
@@ -701,7 +820,7 @@ def device_event_waveform(
"peak_values": _serialise_peak_values(ev.peak_values), "peak_values": _serialise_peak_values(ev.peak_values),
"channels": raw, "channels": raw,
} }
_live_cache.set_waveform(conn_key, index, result) cache.set_waveform(conn_key, index, result)
return result return result
@@ -818,9 +937,9 @@ def device_config(
except Exception as exc: except Exception as exc:
raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc
# Config was written — invalidate cached device info and events so the next # Config was written to the device — the cached compliance config is now stale.
# /device/info or /device/events call re-reads fresh data from the device. conn_key = SFMCache.make_conn_key(host, tcp_port, port, baud)
_live_cache.mark_config_dirty(conn_key) get_cache().mark_config_dirty(conn_key)
return { return {
"status": "ok", "status": "ok",
@@ -862,14 +981,17 @@ def device_monitor_status(
Returns is_monitoring bool, battery voltage, and memory usage (total + free Returns is_monitoring bool, battery voltage, and memory usage (total + free
bytes). Battery and memory are only present when the unit is idle. bytes). Battery and memory are only present when the unit is idle.
**Caching:** response is cached for 30 seconds. Pass ?force=true to bypass. **Caching**: status is cached for 30 seconds to reduce cellular polling overhead.
Pass *force=true* to bypass the cache for an immediate fresh read.
""" """
conn_key = _live_cache.make_conn_key(host, tcp_port, port, baud) cache = get_cache()
conn_key = SFMCache.make_conn_key(host, tcp_port, port, baud)
if not force: if not force:
cached = _live_cache.get_monitor_status(conn_key) cached = cache.get_monitor_status(conn_key)
if cached is not None: if cached is not None:
log.debug("monitor_status cache hit for %s", conn_key) log.debug("monitor status cache hit for %s", conn_key)
cached["_cached"] = True
return cached return cached
with _build_client(port=port, baud=baud, host=host, tcp_port=tcp_port) as client: with _build_client(port=port, baud=baud, host=host, tcp_port=tcp_port) as client:
@@ -890,7 +1012,7 @@ def device_monitor_status(
result["memory_free_bytes"] = status.memory_free result["memory_free_bytes"] = status.memory_free
result["memory_free_kb"] = round(status.memory_free / 1024, 1) result["memory_free_kb"] = round(status.memory_free / 1024, 1)
_live_cache.set_monitor_status(conn_key, result) cache.set_monitor_status(conn_key, result)
return result return result
@@ -914,7 +1036,9 @@ def device_monitor_start(
log.warning("start monitoring poll retry: %s", exc) log.warning("start monitoring poll retry: %s", exc)
client.poll() client.poll()
client.start_monitoring() client.start_monitoring()
_live_cache.invalidate_monitor_status(conn_key)
conn_key = SFMCache.make_conn_key(host, tcp_port, port, baud)
get_cache().invalidate_monitor_status(conn_key)
return {"status": "started"} return {"status": "started"}
@@ -938,10 +1062,44 @@ def device_monitor_stop(
log.warning("stop monitoring poll retry: %s", exc) log.warning("stop monitoring poll retry: %s", exc)
client.poll() client.poll()
client.stop_monitoring() client.stop_monitoring()
_live_cache.invalidate_monitor_status(conn_key)
conn_key = SFMCache.make_conn_key(host, tcp_port, port, baud)
get_cache().invalidate_monitor_status(conn_key)
return {"status": "stopped"} return {"status": "stopped"}
# ── Cache management endpoints ────────────────────────────────────────────────
@app.get("/cache/stats")
def cache_stats() -> dict:
"""
Return row counts for all cache tables.
Useful for debugging and verifying that caching is working as expected.
"""
return get_cache().stats()
@app.delete("/cache/device")
def cache_clear_device(
port: Optional[str] = Query(None, description="Serial port (e.g. COM5)"),
baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
) -> dict:
"""
Clear all cached data for a specific device (identified by its connection address).
Clears: device info, all event headers, all waveforms, monitor status.
The next request to any endpoint for this device will re-fetch from the device.
Supply either *port* (serial) or *host* (TCP/modem) to identify the device.
"""
conn_key = SFMCache.make_conn_key(host, tcp_port, port, baud)
counts = get_cache().clear_device(conn_key)
return {"status": "cleared", "conn_key": conn_key, "deleted": counts}
# ── DB read endpoints ───────────────────────────────────────────────────────── # ── DB read endpoints ─────────────────────────────────────────────────────────
# #
# These endpoints expose the seismo-relay SQLite DB written by ach_server.py. # These endpoints expose the seismo-relay SQLite DB written by ach_server.py.