110 Commits

Author SHA1 Message Date
claude e1a6fd5386 fix(protocol): remove auto-detection of frame mode and ensure extra chunks are always used for valid waveform footer 2026-04-28 00:05:51 -04:00
claude 6b875e161b fix(protocol): implement auto-detection of frame mode based on probe response size for accurate chunk handling 2026-04-27 23:29:53 -04:00
claude f5c81f2cab fix: add new helper (_recv_5a_batch()) that helps with assembling chunks over TCP 2026-04-27 18:39:34 -04:00
claude a7585cb5e0 fix(blastware_file, server): implement logic to skip extra chunks after metadata for accurate file writing 2026-04-26 16:32:32 -04:00
claude ae30a02898 fix(blastware_file, server): enhance logging and correct chunk handling for accurate data processing 2026-04-26 16:03:07 -04:00
claude 2f084ed105 fix(protocol): update chunk counter formula to use max(key4[2:4], 0x0400) for accurate data streaming 2026-04-26 01:28:47 -04:00
claude 7976b544ed fix(blastware_file): never skip A5 frames based on classification at fi>0
Frame 0 is always the probe; frames 1+ are always data (waveform ADC
chunks, compliance config, compliance continuation).  Gating on
classify_frame() at fi>0 produces false positives: ADC binary data
can coincidentally contain b"STRT\xff\xfe", causing frames 1 and 5
to be silently dropped from the body (confirmed from live capture on
event key=01110000).  Remove all type-based filtering; include every
frame unconditionally with the standard index-based skip amounts.
2026-04-26 00:59:36 -04:00
claude 0415af19b4 fix(blastware_file): remove seen_metadata flag and adjust frame processing logic 2026-04-24 20:21:03 -04:00
claude 35c3f4f945 fix(protocol): correct A5 frame classification and chunk counter formula 2026-04-24 17:25:29 -04:00
claude 43c8158493 feat(blastware_file): classify A5 frames, only write waveform frames to body
Add classify_frame() which categorises each A5 frame by content:
  terminator    — page_key == 0x0000
  probe_or_strt — contains b"STRT"
  metadata      — contains compliance-config ASCII markers
                  (Project:, Client:, Standard Recording Setup, …)
  waveform      — binary-heavy (< 20% printable ASCII), i.e. raw ADC data
  unknown       — fallback

Update write_blastware_file() body loop: frame 0 (probe) is still
always processed; frames 1+ are only included when classify_frame
returns "waveform".  Metadata frames (compliance config block with
Project:/Client:/etc.) and any stray STRT-bearing frames are skipped
with a warning/debug log.  Terminator frame handling is unchanged.

Adds temporary print() diagnostics so each frame's classification is
visible in the server log to aid debugging.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 15:48:37 -04:00
claude 242666f358 fix(protocol): correct chunk counter formula for accurate data streaming 2026-04-24 12:52:02 -04:00
claude 03540fdc00 fix: raise max_chunks to 128 for metadata-only 5A download
For 2-second events at 1024 sps the "Project:" metadata frame appears
beyond chunk 32 (the old default cap), causing the safety limit to be
hit and ~34 KB of waveform data to be downloaded instead of stopping
at the metadata frame.  Raising max_chunks to 128 ensures
stop_after_metadata=True can locate the metadata frame for record
times up to ~4 seconds.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 02:19:27 -04:00
claude f83fd880c0 fix(protocol): update device_event_blastware_file to include extra chunk for accurate data retrieval 2026-04-24 00:35:34 -04:00
claude ab2c11e9a9 fix(protocol): refine extra chunk fetching logic for accurate termination response 2026-04-23 20:30:07 -04:00
claude fa887b85d9 fix(protocol): update extra chunk fetching logic to stop at silence detection 2026-04-23 18:28:14 -04:00
claude ecd980d345 fix(protocol): enhance extra chunk fetching logic to ensure footer detection 2026-04-23 18:22:27 -04:00
claude bc9f16e503 fix(protocol): adjust extra_chunks calculation to use integer conversion of record_time 2026-04-23 17:39:28 -04:00
claude aa2b02535b fix(protocol): add record_time based chunk scaling for longer event record times 2026-04-23 17:33:16 -04:00
claude 2a2031c3a9 fix(protocol): fetch additional chunk after metadata to ensure valid termination response 2026-04-23 17:08:36 -04:00
claude 9e7e0bce2a fix(protocol): adjust full_waveform setting for event downloads to end when it should. 2026-04-23 16:43:59 -04:00
claude 5e2f3bf2a1 fix(protocol): enable full_waveform for continuous mode. 2026-04-23 16:24:39 -04:00
claude 39ebd4bdaa fix(protocol): revert endpoint back to stop_after_metadata=True 2026-04-23 15:11:56 -04:00
claude 84c87d0b57 fix(protocol): adjust waveform download to use full_waveform for accurate event streaming 2026-04-23 13:02:55 -04:00
claude ec6362cb8e fix(protocol): include terminator in waveform stream downloads 2026-04-23 12:45:59 -04:00
claude 3eeafd24aa fix(protocol): improve terminator frame detection in write_blastware_file.
fix: rename .n00 to just blastware file (.n00 was false positive)
2026-04-23 01:33:44 -04:00
claude 8cb8b86192 fix(server): add error logging for device event handling 2026-04-22 23:48:59 -04:00
claude 6dcca4da79 feat(protocol): fully decode Blastware filename encoding and update related documentation 2026-04-22 23:43:31 -04:00
claude c47e3a3af0 feat(protocol): update Blastware file format documentation and encoding details 2026-04-22 19:16:05 -04:00
claude dfbc9f29c5 feat: first try at building waveform binary files. 2026-04-21 22:57:53 -04:00
claude 4331215e23 feat(protocol): enhance raw capture functionality and documentation updates
- Update `s3_bridge.py` to default raw capture file paths to "auto" for timestamped naming.
- Modify `gui_bridge.py` to pre-check raw capture options and streamline path handling.
- Extend `ach_server.py` to save both incoming and outgoing raw bytes for analysis.
- Revise `CHANGELOG.md` and `instantel_protocol_reference.md` to reflect changes in recording mode handling and compliance data encoding.
2026-04-21 16:07:24 -04:00
claude b3dcfe7239 fix(client): correct recording_mode anchor position in compliance config encoding 2026-04-21 01:17:45 -04:00
claude 9b5cdfd857 feat(logging): add detailed logging for anchor position in compliance config encoding/decoding 2026-04-21 00:23:15 -04:00
claude 7129aae279 fix(client): update compliance data size handling (less strict now) 2026-04-21 00:09:30 -04:00
claude 2186bc238b fix: call home settings tab display 2026-04-20 21:15:16 -04:00
claude 3fb24e1895 feat(call-home): Implement Auto Call Home configuration management
- Added `CallHomeConfig` model to represent the Auto Call Home settings.
- Introduced methods in `MiniMateClient` for reading (`get_call_home_config`) and writing (`set_call_home_config`) the call home configuration.
- Updated `MiniMateProtocol` with new commands for call home operations (SUB 0x2C for read, SUB 0x7E for write, and SUB 0x7F for confirm).
- Created API endpoints for retrieving and updating call home settings in the server.
- Enhanced the web interface with a new "Call Home" tab for user interaction with call home settings.
- Implemented JavaScript functions for reading and writing call home configurations from the web app.
2026-04-20 18:23:48 -04:00
claude 7bdd7c92f2 Merge branch 'protocol-exp' of https://gitea.serversdown.net/serversdown/seismo-relay into protocol-exp 2026-04-20 17:04:00 -04:00
claude b6ffdcfa87 feat: implement geophone sensitivity and recording mode settings in compliance config 2026-04-20 17:03:58 -04:00
serversdown a7aec31915 Merge pull request 'fix(parser): resolve BAD CHK for BW frames caused by SESSION_RESET bytes' (#4) from seismo-lab into protocol-exp
Reviewed-on: #4
2026-04-20 17:01:34 -04:00
Claude 34df9ec5fa fix(parser): resolve BAD CHK for BW frames caused by SESSION_RESET bytes
SESSION_RESET (41 03) is sent before each POLL frame to wake monitoring
units. The ETX lookahead in parse_bw only checked for ACK+STX directly
after ETX, so when 41 03 followed a frame's ETX, the check failed and the
ETX was swallowed into the body as a payload byte — giving a 19-byte body
instead of 17 for POLL frames and failing checksum validation.

Fix: scan past any SESSION_RESET (41 03) sequences when looking for the
next frame start, so the real ACK+STX boundary is found correctly.

Also adds SUM8 checksum validation to parse_s3, which previously left
checksum_valid=None for all S3 frames.

https://claude.ai/code/session_014NczSHUz9uTzCAf4cVASTJ
2026-04-20 20:47:35 +00:00
claude eec6c3dc6a feat: add histogram_interval setting and update UI with new field. 2026-04-20 16:25:56 -04:00
claude 702e06873e fix: add recording_mode option in html 2026-04-20 15:56:52 -04:00
claude 94767f5a9d feat: add recording_mode to config editor in sfm webapp 2026-04-20 15:54:08 -04:00
claude e04114fd6c feat: mapped record_mode protocol 2026-04-20 15:49:31 -04:00
claude f10c5c1b86 feat: add persistent bridge and streamlined capture pipeline to seismo_lab.py 2026-04-20 15:09:55 -04:00
claude aa28495a43 fix: rename max_geo_range to ADC scale, and make it so its not user configurable.
fix: change max_geo_range_enum to geo_range with two options (normal and sensitive)
2026-04-19 18:15:23 -04:00
claude b23cf4bb50 fix: max_geo_range correctly identified as ADC Scale factor number. 2026-04-17 19:43:45 -04:00
serversdown 969010b983 chore: cleanup claude.md mess 2026-04-17 03:58:50 +00:00
serversdown 5fba9bcff8 doc: version bump to 0.12.1 2026-04-17 03:56:33 +00:00
serversdown ec7be4d784 Merge branch 'feature/intelligent-caching' 2026-04-17 03:46:22 +00:00
claude b8ed237363 docs: update to 0.12.1 2026-04-16 18:31:20 -04:00
claude 5866ecdb3e docs: update protocol doc to reflect unkown status of max_range_geo. 2026-04-16 18:17:16 -04:00
serversdown ea9c69b7c9 chore: add sqlalchemy to pyproject 2026-04-16 21:22:04 +00:00
claude 71bcf71cf7 fix: convert raw psi 32 float into db(L). 2026-04-16 21:22:04 +00:00
claude 3e7de848bc fix: update unique constraints in events and monitor_log tables to use timestamp and serial number. Can't use event keys because minimates resuse them after clearing memory. 2026-04-16 21:22:04 +00:00
claude 72a4209cfd fix: sfm_webapp.html remove display: flex from base class, now shows active tab 2026-04-16 21:22:04 +00:00
claude 2b5574511e feat: add waveform viewer endpoint and enhance UI with new tabs for history, units, monitor log, and sessions 2026-04-16 21:22:04 +00:00
claude ce2c859f11 fix: update event count retrieval logic in AchSession and MiniMateClient 2026-04-16 21:22:04 +00:00
claude 7f322f9ff9 feat: add option to restart monitoring after event download in AchSession 2026-04-16 21:22:04 +00:00
serversdown 42b7a88c3d chore: add python build artifacts to gitignore 2026-04-16 21:22:04 +00:00
claude c474db4f69 build: update build backend to setuptools.build_meta 2026-04-16 21:22:04 +00:00
claude 2765ee6ea7 build: add pyproject.toml for editable install 2026-04-16 21:22:04 +00:00
claude ef88240796 docs: update README to v0.12.0
Rewrites the v0.6.0 README to reflect current project state:
ACH server, SQLite DB, SFM REST API with caching, monitor/erase, updated roadmap.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:22:04 +00:00
claude 5591d345d9 feat: v0.12.0 — live device cache (_LiveCache) in sfm/server.py
Ports the intelligent-caching branch concept to a plain Python in-memory
implementation — no SQLAlchemy, no extra DB table, no new dependencies.

_LiveCache (threading.Lock + dicts) caches:
  - device info: indefinite, invalidated by POST /device/config
  - events: keyed by (conn_key, device_event_count); count-probe fast path
    (~2s poll+count_events) avoids full downloads when nothing is new
  - monitor status: 30-second TTL, invalidated by monitor start/stop
  - waveforms: permanent per (conn_key, event_index)

All four cached endpoints accept ?force=true to bypass the cache.
Removes sfm/cache.py (SQLAlchemy experiment, now superseded).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:22:04 +00:00
claude 7883a31aa7 v0.11.0 — SQLite persistence layer (SeismoDb)
sfm/database.py (new)
- SeismoDb class: three tables keyed by unit serial number
  - ach_sessions: one row per ACH call-home
  - events: one row per triggered event, deduped by (serial, waveform_key)
  - monitor_log: one row per monitoring interval, deduped by (serial, waveform_key)
- WAL mode, per-request connections, silent dedup via UNIQUE constraint
- Query helpers: query_events(), query_monitor_log(), get_sessions(), query_units()
- false_trigger flag on events for future review UI / report filtering

bridges/ach_server.py
- Import SeismoDb; create shared instance at startup pointed at
  bridges/captures/seismo_relay.db
- After each call-home: insert_events() + insert_monitor_log() + insert_ach_session()
- DB failures logged as warnings, never abort the session

sfm/server.py
- Import SeismoDb; lazy singleton via _get_db()
- New DB read endpoints: GET /db/units, /db/events, /db/monitor_log, /db/sessions
- PATCH /db/events/{id}/false_trigger for manual review flagging

CLAUDE.md / CHANGELOG.md
- Document DB schema, SFM DB endpoints, architecture decision (unit-keyed only)
- Version bump to v0.11.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:19:47 +00:00
claude b241da970d v0.10.0 — monitor log entry support (SUB 0x0A partial records)
Add full decode pipeline for 0x2C partial records from the device's event
list, representing continuous monitoring intervals where no threshold was
crossed.  These records appear interleaved with full triggered events in the
browse walk and were previously ignored.

minimateplus/models.py
- Add MonitorLogEntry dataclass: key, start_time, stop_time, serial,
  geo_threshold_ips, raw_header, duration_seconds property

minimateplus/protocol.py
- read_waveform_header() now returns (data_rsp.data, length) — full payload
  including the record-type byte at position 0 — instead of the sliced header.
  Callers that need the old slice use raw_data[11:11+length] as before.

minimateplus/client.py
- Add _decode_0a_partial_header(): auto-detects 9-byte (sub_code=0x10) vs
  10-byte (sub_code=0x03) timestamp format, handles 1-byte inter-timestamp
  gap, extracts serial via BE anchor and geo threshold via Geo: anchor.
- Add get_monitor_log_entries(skip_keys=None): browse walk (1E → 0A → 1F),
  decodes partial records, skips full records and already-seen keys.

minimateplus/__init__.py
- Export MonitorLogEntry

bridges/ach_server.py
- After get_events(), call get_monitor_log_entries(skip_keys=seen_keys) and
  save new entries to monitor_log.json in the session directory.
- Add _monitor_log_entry_to_dict() helper.
- Include monitor log keys in downloaded_keys for state persistence.

CLAUDE.md / CHANGELOG.md
- Document 0x2C partial record layout (timestamp format, ASCII metadata
  region, 1-byte gap edge case) confirmed from 4-11-26 MITM capture.
- Version bump to v0.10.0; update What's next.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 6acb419ebd docs: update protocol reference with v0.9.0 erase-all protocol
Changelog section:
- 5 new entries (2026-04-11): erase-all confirmation, SUB 0x06 purpose
  resolved, §7.11 added, §14.6 ACH session lifecycle marked IMPLEMENTED

§5.1 Request Commands:
- SUB 0x06 description updated: "EVENT STORAGE RANGE READ" (not "CHANNEL
  CONFIG READ"), token=0xFE, last 8 bytes = first/last stored event keys
- SUB 0xA3 added: ERASE ALL BEGIN — standard build_bw_frame, token=0xFE, ack 0x5C
- SUB 0xA2 added: ERASE ALL CONFIRM — standard build_bw_frame, token=0xFE, ack 0x5D

§5.2 Response SUBs:
- 0x06→0xF9 marked CONFIRMED 2026-04-11
- 0xA3→0x5C and 0xA2→0x5D added with CONFIRMED status

§7.11 (new section): Erase-All Protocol
- Full wire sequence (6 request/response pairs)
- SUB 0x06 storage range payload layout (36 bytes, last 8 = first/last key)
- Post-erase key counter reset: device restarts from 0x01110000
- Implementation notes pointing to client.py and ach_server.py

§14.6 ACH Session Lifecycle:
- Removed "Future" label — fully implemented in bridges/ach_server.py
- Added step 6 (optional erase), step 8 (DCD/DTR auto-resume)
- Documents ach_server.py flags and ach_state.json schema
- Notes RV55 DCD/DTR issue as known open problem

Open Questions table:
- SUB 0x06 purpose RESOLVED
- Erase-all sequence RESOLVED
- ACH server RESOLVED
- Sensor Check byte: still open, added as formal question
- RV55 DCD/DTR: added as new open question

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude f6a0846bab docs: update CHANGELOG and CLAUDE.md for v0.9.0
CHANGELOG.md:
- New v0.9.0 section covering erase-all protocol, browse helpers,
  delete_all_events(), ach_mitm.py, and ACH server overhaul
- Back-filled v0.8.0 section (write pipeline, monitoring, ACH server)
  that was missing from the previous release notes

CLAUDE.md:
- Bump version to v0.9.0
- Add erase-all protocol section with full wire sequence, SUB 0x06
  storage range response layout, and post-erase key counter reset notes
- Document ACH server state format (ach_state.json v0.9.0 schema with
  downloaded_keys + max_downloaded_key)
- Add RV55 DCD/DTR issue to What's next

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 3d9db8b662 feat: add ach_mitm.py — transparent TCP MITM proxy for ACH session capture
Listens for inbound unit connections, connects upstream to a real Blastware
ACH server, and forwards bytes bidirectionally while saving both directions to
raw_bw_<ts>.bin and raw_s3_<ts>.bin in the existing capture format.

Used to capture the 4-11-26 Blastware ACH session that confirmed the erase-all
protocol (SUBs 0xA3/0x1C/0x06/0xA2) and the event deletion wire sequence.

Usage:
  python bridges/ach_mitm.py --bw-host 127.0.0.1 --bw-port 9999 --listen-port 9998
  Point the unit's call-home destination at this machine:9998.
  Point this proxy's --bw-host/port at the upstream Blastware ACH server.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude c7e7d177e6 feat: overhaul ACH server with key-based state, erase support, and reset detection
State format (ach_state.json):
- Replace event_count with downloaded_keys (set of hex strings) + max_downloaded_key
- Key-based tracking correctly handles delete-then-re-record: after device erase the
  count drops to 0, but new events have new (or recycled) keys

Browse pre-check:
- list_event_keys() walk before get_events() to bail early when nothing is new
- get_events() called with skip_waveform_for_keys= for already-seen keys, so repeat
  call-homes only download waveforms for genuinely new events

--clear-after-download flag:
- After saving new events, calls client.delete_all_events() (0xA3→0x1C→0x06→0xA2)
- On success: resets downloaded_keys=[] and max_downloaded_key="00000000" so the
  next session starts fresh (device counter resets to 0x01110000 after erase)

Post-erase key-reuse detection:
- Device counter resets to 0x01110000 after any erase; new events reuse old keys
- If max(device_keys) < max_downloaded_key, the device was wiped externally
  (Blastware, manual) — seen_keys is discarded and all device keys treated as new

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude a3b8d10fa8 feat: add erase-all protocol and browse helpers to protocol/client layer
protocol.py:
- SUB_ERASE_ALL_BEGIN = 0xA3, SUB_ERASE_ALL_CONFIRM = 0xA2 (confirmed 4-11-26 MITM)
- SUB_CHANNEL_CONFIG (0x06) data length = 0x24 (36 bytes) in DATA_LENGTHS
- begin_erase_all()              — single frame, token=0xFE, response 0x5C
- confirm_erase_all()            — single frame, token=0xFE, response 0x5D
- read_event_storage_range()     — two-step read (probe+data), token=0xFE
  Response last 8 bytes = first/last stored event key; both 0x01110000 after erase

client.py:
- list_event_keys()              — browse-mode 1E→0A→1F walk, no waveform download;
  returns list of hex key strings; used as fast pre-check before get_events()
- get_events(skip_waveform_for_keys=set())
  — for already-seen keys: only 0A+1F(browse), skips 1E-arm/0C/POLL×3/5A entirely
- delete_all_events()            — orchestrates the confirmed erase sequence:
  0xA3 → 0x1C → 0x06 → 0xA2; logs first/last key from storage range response

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 4921b0489a fix: correct Event and PeakValues field names in ach_server serialization
Event model uses peak_values (not peaks) and project_info (not direct fields).
PeakValues fields are tran/vert/long/micl/peak_vector_sum (not transverse etc).
ProjectInfo fields accessed via ev.project_info.project etc.

Also fix ev.timestamp serialization: use str() instead of .isoformat() since
Timestamp is a custom dataclass, not datetime.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 8688d815a0 fix: remove non-existent DeviceInfo fields from ach_server log and dict
calibration_date, aux_trigger, setup_name etc. don't exist directly on
DeviceInfo — they live in DeviceInfo.compliance_config (ComplianceConfig).
_device_info_to_dict now accesses them via cc = d.compliance_config.
Log line updated to show serial/firmware/model/event_count instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 9b50ec9133 fix: make Ctrl-C work on Windows by setting accept() timeout
socket.accept() on Windows blocks indefinitely and ignores KeyboardInterrupt.
Setting a 1-second timeout on the server socket causes the accept loop to wake
up every second and re-check, so Ctrl-C is handled within ~1 second.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude cba8b1b401 feat: defer session dir creation and add --allow-ip allowlist
- Session directory and log file are now created ONLY after startup() succeeds.
  Internet scanners and dropped connections no longer litter the output folder.
  Raw bytes are buffered in memory until startup succeeds, then flushed to disk.

- Add --allow-ip IP flag (repeatable) to allowlist specific source IPs.
  Connections from un-listed IPs are rejected immediately (socket closed, no log).
  If no --allow-ip flags are given, all IPs are still accepted (original behavior).
  Usage: --allow-ip 63.43.212.232 --allow-ip 152.1.2.3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 41a14ca468 fix: correct event count field offset and eliminate count_events() walk
_decode_event_count: read uint16 BE at offset 10 (confirmed 2026-04-10 from
live BE11529 event index — data[10:12]=0x0006=6, matches device LCD).
Previous uint32 at offset 3 always returned 1 regardless of event count.

ach_server.py: use device_info.event_count (already fetched during connect())
instead of calling count_events() separately. This saves 2*N round-trips and
avoids the 1F linked-list walk which was overcounting on some devices.
count_events() kept as fallback when connect() is skipped (--events-only).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 1bfc6e4258 fix: replace Unicode chars in log messages, fix DeviceInfo.serial, UTF-8 file log
- Replace all Unicode arrows/checkmarks (->  [OK]  [FAIL]) in ach_server.py
  and client.py log calls — Windows cp1252 console can't encode them
- Fix DeviceInfo attribute: serial_number -> serial
- Fix _device_info_to_dict key: serial_number -> serial
- Demote count_events 1E/1F per-key log lines from WARNING to DEBUG
  (they were flooding the console on devices with many stored events)
- FileHandler now opens with encoding='utf-8' so session log files
  can hold any characters without codec errors

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 574d40027f feat: enhance logging messages in ach_server.py and add experiments.py for protocol minimization 2026-04-16 21:14:58 +00:00
claude 0358acb51d feat: add high-water mark state tracking to ach_server + fix monitoring flag
ach_server.py:
- Add ach_state.json per-unit state tracking (keyed by serial number)
- count_events() before any download; skip session if no new events since last call-home
- Download only events beyond the previous high-water mark (all_events[last_count:])
- --max-events N safety cap for first-run units with many stored events
- state_path and max_events wired through AchSession constructor and serve()

client.py (_decode_monitor_status):
- Revert monitoring flag to section[1] == 0x10 (was incorrectly changed to section[6])
- Fix battery/memory offsets to section[-10:-8], [-8:-4], [-4:] (no trailing checksum byte)
- Both confirmed by full byte diff of all 144 0xE3 data frames in 4-8-26/2ndtry capture

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude cf7d838bf4 feat: add SocketTransport and ach_server.py inbound ACH server
minimateplus/transport.py:
- Add SocketTransport(TcpTransport) — wraps an already-accepted inbound
  socket; connect() is a no-op; everything else inherited from TcpTransport.
  Enables the ACH server to reuse all existing protocol/client code without
  any changes.

bridges/ach_server.py:
- Minimal inbound ACH server — listens on port 12345, accepts call-home
  connections from MiniMate Plus units, runs the full BW protocol:
  startup handshake → get_device_info → get_events(full_waveform=True)
- Saves device_info.json + events.json + raw_rx_<ts>.bin + session log
  per connection to bridges/captures/ach_inbound_<ts>/
- raw_rx.bin is byte-compatible with existing Analyzer tooling
- Taps transport.read() to capture raw S3 bytes alongside parsed output
- Each connection runs in its own daemon thread
- Clearly distinguishes push vs pull protocol in the startup log

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 5e44cdc668 feat: add splitter mode to ach_bridge.py (--mirror HOST:PORT)
Adds a production-safe headphone-splitter mode:
- Device bytes tee'd to both --upstream (primary/prod) and --mirror (new server)
- Only primary server responses are returned to the device
- Mirror connect/write failures are non-fatal and logged; prod is unaffected
- New raw_mirror_<ts>.bin capture file alongside raw_client/raw_server

Three modes: standalone (capture only), bridge (one upstream), splitter (two).
Default listen port changed to 12345 to match project ACH setup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 37d32077a4 feat: add ACH TCP bridge, serial tap tool, and Serial Watch tab
- bridges/ach_bridge.py: transparent TCP bridge that MITMs the MiniMate Plus
  call-home connection — forwards to real ACH server while logging all frames
  to raw_client/raw_server .bin files compatible with parse_capture.py;
  standalone capture mode for lab use without a real server

- bridges/serial_watch.py: RS-232 serial monitor with live S3 frame parsing;
  taps the line between MiniMate and modem (RV50/RV55); captures raw bytes,
  .log and .jsonl; --ack-ok mode auto-replies to AT commands; fixed fatal
  indentation bug in the original that silently prevented any data capture

- seismo_lab.py: new "Serial Watch" fourth tab (SerialWatchPanel) wrapping
  serial_watch.py functionality; COM port picker with refresh, baud config,
  ack-ok toggle, colour-coded live frame log (teal frames / yellow ctrl /
  blue AT), raw .bin capture auto-fed into Analyzer tab on stop

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude b384ba66d1 fix: convert raw psi 32 float into db(L). 2026-04-14 01:13:21 -04:00
claude 27d9823cc1 fix: update unique constraints in events and monitor_log tables to use timestamp and serial number. Can't use event keys because minimates resuse them after clearing memory. 2026-04-13 22:45:58 -04:00
claude 70c9528611 fix: sfm_webapp.html remove display: flex from base class, now shows active tab 2026-04-13 22:40:40 -04:00
claude e8bef1ac7c feat: add waveform viewer endpoint and enhance UI with new tabs for history, units, monitor log, and sessions 2026-04-13 22:34:28 -04:00
claude 27db663579 fix: update event count retrieval logic in AchSession and MiniMateClient 2026-04-13 18:46:23 -04:00
claude e5ea17388a feat: add option to restart monitoring after event download in AchSession 2026-04-13 18:23:27 -04:00
serversdown c0a5131c7d chore: add python build artifacts to gitignore 2026-04-13 21:59:52 +00:00
claude 4ec2f33308 build: update build backend to setuptools.build_meta 2026-04-13 17:56:15 -04:00
claude 6282eacf8b build: add pyproject.toml for editable install 2026-04-13 17:34:58 -04:00
claude 034b3f044d docs: update README to v0.12.0
Rewrites the v0.6.0 README to reflect current project state:
ACH server, SQLite DB, SFM REST API with caching, monitor/erase, updated roadmap.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 16:12:07 -04:00
claude 48d7e94c02 feat: v0.12.0 — live device cache (_LiveCache) in sfm/server.py
Ports the intelligent-caching branch concept to a plain Python in-memory
implementation — no SQLAlchemy, no extra DB table, no new dependencies.

_LiveCache (threading.Lock + dicts) caches:
  - device info: indefinite, invalidated by POST /device/config
  - events: keyed by (conn_key, device_event_count); count-probe fast path
    (~2s poll+count_events) avoids full downloads when nothing is new
  - monitor status: 30-second TTL, invalidated by monitor start/stop
  - waveforms: permanent per (conn_key, event_index)

All four cached endpoints accept ?force=true to bypass the cache.
Removes sfm/cache.py (SQLAlchemy experiment, now superseded).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 15:57:02 -04:00
claude 03d224ccc3 v0.11.0 — SQLite persistence layer (SeismoDb)
sfm/database.py (new)
- SeismoDb class: three tables keyed by unit serial number
  - ach_sessions: one row per ACH call-home
  - events: one row per triggered event, deduped by (serial, waveform_key)
  - monitor_log: one row per monitoring interval, deduped by (serial, waveform_key)
- WAL mode, per-request connections, silent dedup via UNIQUE constraint
- Query helpers: query_events(), query_monitor_log(), get_sessions(), query_units()
- false_trigger flag on events for future review UI / report filtering

bridges/ach_server.py
- Import SeismoDb; create shared instance at startup pointed at
  bridges/captures/seismo_relay.db
- After each call-home: insert_events() + insert_monitor_log() + insert_ach_session()
- DB failures logged as warnings, never abort the session

sfm/server.py
- Import SeismoDb; lazy singleton via _get_db()
- New DB read endpoints: GET /db/units, /db/events, /db/monitor_log, /db/sessions
- PATCH /db/events/{id}/false_trigger for manual review flagging

CLAUDE.md / CHANGELOG.md
- Document DB schema, SFM DB endpoints, architecture decision (unit-keyed only)
- Version bump to v0.11.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 00:45:38 -04:00
claude ef2c38e7db v0.10.0 — monitor log entry support (SUB 0x0A partial records)
Add full decode pipeline for 0x2C partial records from the device's event
list, representing continuous monitoring intervals where no threshold was
crossed.  These records appear interleaved with full triggered events in the
browse walk and were previously ignored.

minimateplus/models.py
- Add MonitorLogEntry dataclass: key, start_time, stop_time, serial,
  geo_threshold_ips, raw_header, duration_seconds property

minimateplus/protocol.py
- read_waveform_header() now returns (data_rsp.data, length) — full payload
  including the record-type byte at position 0 — instead of the sliced header.
  Callers that need the old slice use raw_data[11:11+length] as before.

minimateplus/client.py
- Add _decode_0a_partial_header(): auto-detects 9-byte (sub_code=0x10) vs
  10-byte (sub_code=0x03) timestamp format, handles 1-byte inter-timestamp
  gap, extracts serial via BE anchor and geo threshold via Geo: anchor.
- Add get_monitor_log_entries(skip_keys=None): browse walk (1E → 0A → 1F),
  decodes partial records, skips full records and already-seen keys.

minimateplus/__init__.py
- Export MonitorLogEntry

bridges/ach_server.py
- After get_events(), call get_monitor_log_entries(skip_keys=seen_keys) and
  save new entries to monitor_log.json in the session directory.
- Add _monitor_log_entry_to_dict() helper.
- Include monitor log keys in downloaded_keys for state persistence.

CLAUDE.md / CHANGELOG.md
- Document 0x2C partial record layout (timestamp format, ASCII metadata
  region, 1-byte gap edge case) confirmed from 4-11-26 MITM capture.
- Version bump to v0.10.0; update What's next.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 02:59:40 -04:00
claude b9a8e50b3c docs: update protocol reference with v0.9.0 erase-all protocol
Changelog section:
- 5 new entries (2026-04-11): erase-all confirmation, SUB 0x06 purpose
  resolved, §7.11 added, §14.6 ACH session lifecycle marked IMPLEMENTED

§5.1 Request Commands:
- SUB 0x06 description updated: "EVENT STORAGE RANGE READ" (not "CHANNEL
  CONFIG READ"), token=0xFE, last 8 bytes = first/last stored event keys
- SUB 0xA3 added: ERASE ALL BEGIN — standard build_bw_frame, token=0xFE, ack 0x5C
- SUB 0xA2 added: ERASE ALL CONFIRM — standard build_bw_frame, token=0xFE, ack 0x5D

§5.2 Response SUBs:
- 0x06→0xF9 marked CONFIRMED 2026-04-11
- 0xA3→0x5C and 0xA2→0x5D added with CONFIRMED status

§7.11 (new section): Erase-All Protocol
- Full wire sequence (6 request/response pairs)
- SUB 0x06 storage range payload layout (36 bytes, last 8 = first/last key)
- Post-erase key counter reset: device restarts from 0x01110000
- Implementation notes pointing to client.py and ach_server.py

§14.6 ACH Session Lifecycle:
- Removed "Future" label — fully implemented in bridges/ach_server.py
- Added step 6 (optional erase), step 8 (DCD/DTR auto-resume)
- Documents ach_server.py flags and ach_state.json schema
- Notes RV55 DCD/DTR issue as known open problem

Open Questions table:
- SUB 0x06 purpose RESOLVED
- Erase-all sequence RESOLVED
- ACH server RESOLVED
- Sensor Check byte: still open, added as formal question
- RV55 DCD/DTR: added as new open question

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 01:20:43 -04:00
claude 77d9c17680 docs: update CHANGELOG and CLAUDE.md for v0.9.0
CHANGELOG.md:
- New v0.9.0 section covering erase-all protocol, browse helpers,
  delete_all_events(), ach_mitm.py, and ACH server overhaul
- Back-filled v0.8.0 section (write pipeline, monitoring, ACH server)
  that was missing from the previous release notes

CLAUDE.md:
- Bump version to v0.9.0
- Add erase-all protocol section with full wire sequence, SUB 0x06
  storage range response layout, and post-erase key counter reset notes
- Document ACH server state format (ach_state.json v0.9.0 schema with
  downloaded_keys + max_downloaded_key)
- Add RV55 DCD/DTR issue to What's next

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 01:15:11 -04:00
claude 8a1bd34551 feat: add ach_mitm.py — transparent TCP MITM proxy for ACH session capture
Listens for inbound unit connections, connects upstream to a real Blastware
ACH server, and forwards bytes bidirectionally while saving both directions to
raw_bw_<ts>.bin and raw_s3_<ts>.bin in the existing capture format.

Used to capture the 4-11-26 Blastware ACH session that confirmed the erase-all
protocol (SUBs 0xA3/0x1C/0x06/0xA2) and the event deletion wire sequence.

Usage:
  python bridges/ach_mitm.py --bw-host 127.0.0.1 --bw-port 9999 --listen-port 9998
  Point the unit's call-home destination at this machine:9998.
  Point this proxy's --bw-host/port at the upstream Blastware ACH server.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 01:15:02 -04:00
claude 09788b931a feat: overhaul ACH server with key-based state, erase support, and reset detection
State format (ach_state.json):
- Replace event_count with downloaded_keys (set of hex strings) + max_downloaded_key
- Key-based tracking correctly handles delete-then-re-record: after device erase the
  count drops to 0, but new events have new (or recycled) keys

Browse pre-check:
- list_event_keys() walk before get_events() to bail early when nothing is new
- get_events() called with skip_waveform_for_keys= for already-seen keys, so repeat
  call-homes only download waveforms for genuinely new events

--clear-after-download flag:
- After saving new events, calls client.delete_all_events() (0xA3→0x1C→0x06→0xA2)
- On success: resets downloaded_keys=[] and max_downloaded_key="00000000" so the
  next session starts fresh (device counter resets to 0x01110000 after erase)

Post-erase key-reuse detection:
- Device counter resets to 0x01110000 after any erase; new events reuse old keys
- If max(device_keys) < max_downloaded_key, the device was wiped externally
  (Blastware, manual) — seen_keys is discarded and all device keys treated as new

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 01:14:50 -04:00
claude e712d68505 feat: add erase-all protocol and browse helpers to protocol/client layer
protocol.py:
- SUB_ERASE_ALL_BEGIN = 0xA3, SUB_ERASE_ALL_CONFIRM = 0xA2 (confirmed 4-11-26 MITM)
- SUB_CHANNEL_CONFIG (0x06) data length = 0x24 (36 bytes) in DATA_LENGTHS
- begin_erase_all()              — single frame, token=0xFE, response 0x5C
- confirm_erase_all()            — single frame, token=0xFE, response 0x5D
- read_event_storage_range()     — two-step read (probe+data), token=0xFE
  Response last 8 bytes = first/last stored event key; both 0x01110000 after erase

client.py:
- list_event_keys()              — browse-mode 1E→0A→1F walk, no waveform download;
  returns list of hex key strings; used as fast pre-check before get_events()
- get_events(skip_waveform_for_keys=set())
  — for already-seen keys: only 0A+1F(browse), skips 1E-arm/0C/POLL×3/5A entirely
- delete_all_events()            — orchestrates the confirmed erase sequence:
  0xA3 → 0x1C → 0x06 → 0xA2; logs first/last key from storage range response

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-11 01:14:37 -04:00
claude 8f5da918b5 fix: correct Event and PeakValues field names in ach_server serialization
Event model uses peak_values (not peaks) and project_info (not direct fields).
PeakValues fields are tran/vert/long/micl/peak_vector_sum (not transverse etc).
ProjectInfo fields accessed via ev.project_info.project etc.

Also fix ev.timestamp serialization: use str() instead of .isoformat() since
Timestamp is a custom dataclass, not datetime.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 02:09:57 -04:00
claude a03c77af09 fix: remove non-existent DeviceInfo fields from ach_server log and dict
calibration_date, aux_trigger, setup_name etc. don't exist directly on
DeviceInfo — they live in DeviceInfo.compliance_config (ComplianceConfig).
_device_info_to_dict now accesses them via cc = d.compliance_config.
Log line updated to show serial/firmware/model/event_count instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 01:43:02 -04:00
claude 87fa9c954f fix: make Ctrl-C work on Windows by setting accept() timeout
socket.accept() on Windows blocks indefinitely and ignores KeyboardInterrupt.
Setting a 1-second timeout on the server socket causes the accept loop to wake
up every second and re-check, so Ctrl-C is handled within ~1 second.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 01:19:36 -04:00
claude 3f7b5c07b5 feat: defer session dir creation and add --allow-ip allowlist
- Session directory and log file are now created ONLY after startup() succeeds.
  Internet scanners and dropped connections no longer litter the output folder.
  Raw bytes are buffered in memory until startup succeeds, then flushed to disk.

- Add --allow-ip IP flag (repeatable) to allowlist specific source IPs.
  Connections from un-listed IPs are rejected immediately (socket closed, no log).
  If no --allow-ip flags are given, all IPs are still accepted (original behavior).
  Usage: --allow-ip 63.43.212.232 --allow-ip 152.1.2.3

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 01:17:30 -04:00
claude 3d2ebfc057 fix: correct event count field offset and eliminate count_events() walk
_decode_event_count: read uint16 BE at offset 10 (confirmed 2026-04-10 from
live BE11529 event index — data[10:12]=0x0006=6, matches device LCD).
Previous uint32 at offset 3 always returned 1 regardless of event count.

ach_server.py: use device_info.event_count (already fetched during connect())
instead of calling count_events() separately. This saves 2*N round-trips and
avoids the 1F linked-list walk which was overcounting on some devices.
count_events() kept as fallback when connect() is skipped (--events-only).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 01:10:49 -04:00
claude 9d9c14af79 fix: replace Unicode chars in log messages, fix DeviceInfo.serial, UTF-8 file log
- Replace all Unicode arrows/checkmarks (->  [OK]  [FAIL]) in ach_server.py
  and client.py log calls — Windows cp1252 console can't encode them
- Fix DeviceInfo attribute: serial_number -> serial
- Fix _device_info_to_dict key: serial_number -> serial
- Demote count_events 1E/1F per-key log lines from WARNING to DEBUG
  (they were flooding the console on devices with many stored events)
- FileHandler now opens with encoding='utf-8' so session log files
  can hold any characters without codec errors

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-10 01:06:27 -04:00
claude ab14328c8b feat: enhance logging messages in ach_server.py and add experiments.py for protocol minimization 2026-04-10 00:58:54 -04:00
claude 0baf343bf5 feat: add high-water mark state tracking to ach_server + fix monitoring flag
ach_server.py:
- Add ach_state.json per-unit state tracking (keyed by serial number)
- count_events() before any download; skip session if no new events since last call-home
- Download only events beyond the previous high-water mark (all_events[last_count:])
- --max-events N safety cap for first-run units with many stored events
- state_path and max_events wired through AchSession constructor and serve()

client.py (_decode_monitor_status):
- Revert monitoring flag to section[1] == 0x10 (was incorrectly changed to section[6])
- Fix battery/memory offsets to section[-10:-8], [-8:-4], [-4:] (no trailing checksum byte)
- Both confirmed by full byte diff of all 144 0xE3 data frames in 4-8-26/2ndtry capture

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 14:38:44 -04:00
claude 05421764a5 feat: add SocketTransport and ach_server.py inbound ACH server
minimateplus/transport.py:
- Add SocketTransport(TcpTransport) — wraps an already-accepted inbound
  socket; connect() is a no-op; everything else inherited from TcpTransport.
  Enables the ACH server to reuse all existing protocol/client code without
  any changes.

bridges/ach_server.py:
- Minimal inbound ACH server — listens on port 12345, accepts call-home
  connections from MiniMate Plus units, runs the full BW protocol:
  startup handshake → get_device_info → get_events(full_waveform=True)
- Saves device_info.json + events.json + raw_rx_<ts>.bin + session log
  per connection to bridges/captures/ach_inbound_<ts>/
- raw_rx.bin is byte-compatible with existing Analyzer tooling
- Taps transport.read() to capture raw S3 bytes alongside parsed output
- Each connection runs in its own daemon thread
- Clearly distinguishes push vs pull protocol in the startup log

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 12:34:27 -04:00
claude 74233d7e31 feat: add splitter mode to ach_bridge.py (--mirror HOST:PORT)
Adds a production-safe headphone-splitter mode:
- Device bytes tee'd to both --upstream (primary/prod) and --mirror (new server)
- Only primary server responses are returned to the device
- Mirror connect/write failures are non-fatal and logged; prod is unaffected
- New raw_mirror_<ts>.bin capture file alongside raw_client/raw_server

Three modes: standalone (capture only), bridge (one upstream), splitter (two).
Default listen port changed to 12345 to match project ACH setup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 12:17:57 -04:00
claude 46a86939b7 feat: add ACH TCP bridge, serial tap tool, and Serial Watch tab
- bridges/ach_bridge.py: transparent TCP bridge that MITMs the MiniMate Plus
  call-home connection — forwards to real ACH server while logging all frames
  to raw_client/raw_server .bin files compatible with parse_capture.py;
  standalone capture mode for lab use without a real server

- bridges/serial_watch.py: RS-232 serial monitor with live S3 frame parsing;
  taps the line between MiniMate and modem (RV50/RV55); captures raw bytes,
  .log and .jsonl; --ack-ok mode auto-replies to AT commands; fixed fatal
  indentation bug in the original that silently prevented any data capture

- seismo_lab.py: new "Serial Watch" fourth tab (SerialWatchPanel) wrapping
  serial_watch.py functionality; COM port picker with refresh, baud config,
  ack-ok toggle, colour-coded live frame log (teal frames / yellow ctrl /
  blue AT), raw .bin capture auto-fed into Analyzer tab on stop

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-09 12:10:52 -04:00
26 changed files with 9606 additions and 604 deletions
+33 -28
View File
@@ -1,28 +1,33 @@
/bridges/captures/
/example-events/
/manuals/
# Python bytecode
__pycache__/
*.py[cod]
# Virtual environments
.venv/
venv/
env/
# Editor / OS
.vscode/
*.swp
.DS_Store
Thumbs.db
# Analyzer outputs
*.report
claude_export_*.md
# Frame database
*.db
*.db-wal
*.db-shm
/bridges/captures/
/example-events/
/manuals/
# Python build artifacts
*.egg-info/
dist/
build/
# Python bytecode
__pycache__/
*.py[cod]
# Virtual environments
.venv/
venv/
env/
# Editor / OS
.vscode/
*.swp
.DS_Store
Thumbs.db
# Analyzer outputs
*.report
claude_export_*.md
# Frame database
*.db
*.db-wal
*.db-shm
+383
View File
@@ -4,6 +4,389 @@ All notable changes to seismo-relay are documented here.
---
## v0.12.5 — 2026-04-21
### Changed
- **`s3_bridge.py` — raw captures always-on by default** — `--raw-bw` and `--raw-s3` now
default to `"auto"` instead of `None`. Every bridge session automatically generates
timestamped `raw_bw_<ts>.bin` and `raw_s3_<ts>.bin` files alongside the `.bin`/`.log`
session files. Pass `--raw-bw ""` (explicit empty string) to disable if needed.
- **`gui_bridge.py` — raw capture checkboxes pre-checked** — Both "BW→S3 raw" and
"S3→BW raw" checkboxes start checked. Path fields are empty by default (bridge auto-names
the files). Unchecking a box passes `--raw-bw ""` to explicitly disable capture.
- **`ach_server.py` — TX capture added (`raw_tx_<ts>.bin`)** — Every ACH inbound session
now saves both directions: `raw_rx_<ts>.bin` (device → us, S3 side, as before) and
`raw_tx_<ts>.bin` (us → device, BW side). Both files are usable in the Analyzer.
TX bytes are buffered in memory until startup handshake succeeds (same as RX), preventing
scanner probes from creating empty files.
---
## v0.12.4 — 2026-04-21 (protocol analysis / docs only — no code changes)
### Discovered
- **compliance_raw is wire-encoded, not logical bytes** — `read_compliance_config()` returns
bytes that include DLE prefix bytes (`0x10`) before any `0x03` values (because S3FrameParser
preserves DLE+ETX inner-frame pairs as two literal bytes). The previous CLAUDE.md claim that
"S3FrameParser handles this transparently so compliance_raw contains logical bytes" was wrong.
- **anchor-9 behavior per recording mode** (confirmed from 4-20-26 BW write captures):
- Single Shot (0x00) / Continuous (0x01): anchor-9 = `0x00`
- Histogram (0x03): anchor-9 = `0x10` — the E5 DLE prefix for the `0x03` recording_mode byte
- Histogram+Continuous (0x04): anchor-9 = `0x10` — an actual stored config byte for this mode
Anchor position shifts by ±1 when recording_mode = `0x03` due to the extra DLE byte; the
dynamic anchor search (`buf.find(ANCHOR, 0, 150)`) handles this correctly without code changes.
- **Write frame ETX escaping** — BW escapes `0x03` bytes in write frame data as `0x10 0x03`
on the wire. Our `build_bw_write_frame` sends data bytes raw without ETX escaping. Device
accepts our raw writes for all tested modes. Hypothesis: device write parser uses the
offset/length field for frame boundaries, not ETX scanning, making ETX escaping optional.
Histogram mode (recording_mode = 0x03) write via SFM from a non-Histogram starting state
not yet tested.
- **BW write payload vs E5 read payload are byte-identical** around the anchor region (confirmed
by comparing 3-11-26 BW TX and S3 captures). BW does NOT strip DLE prefix bytes before writing;
it round-trips the wire-encoded bytes verbatim with only the modified fields changed.
- **Capture folder content catalogued** — see CLAUDE.md "BW capture reference" table for a
summary of all available protocol captures and their contents.
---
## v0.12.3 — 2026-04-20
### Added
- **Auto Call Home config protocol** — Full read/write/decode/encode pipeline for the
device's Remote Access → Setup Unit ACH settings, confirmed from 4-20-26 call home
settings captures.
**Protocol (new):**
- `SUB 0x2C` — Call Home Config READ (response `0xD3`); two-step read; data offset
`0x7C` = 124; raw payload 125 bytes (1-byte longer than DATA_LENGTH due to DLE-escaped
`\x10\x03` at raw[117:119] representing num_retries = 3)
- `SUB 0x7E` — Call Home Config WRITE (response `0x81`); 127-byte payload (125-byte read
payload + `\x00\x00`); offset = `data[1]+2 = 0x7E`; write format (DLE-aware checksum)
- `SUB 0x7F` — Call Home WRITE CONFIRM (response `0x80`); no data
**Field map (confirmed from 10-frame BW TX diff):**
- `raw[5]` — auto_call_home_enabled (bool)
- `raw[6:46]` — dial_string (40-byte null-padded ASCII)
- `raw[87]` — after_event_recorded (bool)
- `raw[91]` — at_specified_times (bool)
- `raw[93]` — time1_enabled / `raw[101]` — time1_hour / `raw[102]` — time1_min
- `raw[95]` — time2_enabled / `raw[105]` — time2_hour / `raw[106]` — time2_min
- `raw[117:119]``\x10\x03` (DLE-escaped 0x03 = num_retries value 3)
- `raw[120]` — time_between_retries_sec / `raw[122]` — wait_for_connection_sec / `raw[124]` — warm_up_time_sec
**Library (`minimateplus/`):**
- `models.py``CallHomeConfig` dataclass (14 fields; `raw` bytes preserved for
round-trip writes)
- `protocol.py``SUB_CALL_HOME = 0x2C`, `SUB_CALL_HOME_WRITE = 0x7E`,
`SUB_CALL_HOME_CONFIRM = 0x7F`; `read_call_home_config()`, `write_call_home_config()`
- `client.py``get_call_home_config()`, `set_call_home_config()`,
`_decode_call_home_config()` (handles DLE prefix at raw[117]),
`_encode_call_home_config()` (patches in-place; raises `ValueError` if hour/min = 3)
**REST API (`sfm/server.py`):**
- `GET /device/call_home` — reads and decodes call home config from device
- `POST /device/call_home` — reads, patches specified fields, writes back to device
- `CallHomeConfigBody` Pydantic model with 9 optional writable fields
**Web UI (`sfm/sfm_webapp.html`):**
- New "Call Home" tab with enable flag, dial string (read-only), after-event trigger,
at-specified-times flag, two time slots (enable + HH:MM each), and read-only retry
settings (num_retries, time_between_retries_sec, wait_for_connection_sec,
warm_up_time_sec)
- "Read from Device", "Write to Device", "Clear Form" action buttons
- Client-side guard: rejects hour or minute value equal to 3 with a clear message
explaining the DLE-encoding limitation
---
## v0.12.2 — 2026-04-20
### Added / Fixed
- **Geophone sensitivity / maximum range field confirmed** — 4-20-26 geo sensitivity
captures (1.25 in/s vs 10 in/s) diffed across all three SUB 71 write chunks and both
E5 read payloads. The `geo_range` uint8 field per channel is now fully confirmed:
- E5 read offset: `channel_label + 33`; SUB 71 write offset: `channel_label + 29`
- `0x00` = Normal 10.000 in/s (standard gain); `0x01` = Sensitive 1.250 in/s (high gain)
- **Correction:** previous hypothesis (`channel_label+20`, `0x01`=Normal) was wrong.
`channel_label+20` reads `0x01` on ALL captures regardless of range — not this field.
- `_decode_compliance_config_into`: read offset corrected from `tran_pos+20``tran_pos+33`
- `_encode_compliance_config`: added `geo_range` parameter; writes to Tran/Vert/Long at `+29`
- `apply_config`: added `geo_range` parameter
- `POST /device/config`: added `geo_range` to `DeviceConfigBody`
- Web UI Config tab: added "Maximum Range — Geo" select (Normal / Sensitive)
- Web UI Device tab: added "Max Range (geo)" row to compliance table
- **`recording_mode` + `histogram_interval_sec` confirmed and implemented** (4-20-26 captures)
- `recording_mode`: uint8 at anchor8 (E5 read) / anchor7 (write); enum: 0x00=Single Shot,
0x01=Continuous, 0x03=Histogram, 0x04=Histogram+Continuous
- `histogram_interval_sec`: uint16 BE seconds at anchor4; same offset in read & write;
valid: 2, 5, 15, 60, 300, 900 (matching Blastware dropdown: 2s, 5s, 15s, 1m, 5m, 15m)
- Both fields added to `ComplianceConfig`, `_decode_compliance_config_into`,
`_encode_compliance_config`, `apply_config`, REST API body, and web UI
---
## v0.12.1 — 2026-04-16
### Added
- **`sfm/server.py``_LiveCache`** — in-memory live device cache that eliminates
redundant TCP round-trips between web requests. Plain Python dict +
`threading.Lock`, no extra dependencies.
Cache strategy per endpoint:
| Endpoint | Strategy |
|---|---|
| `GET /device/info` | Indefinite; invalidated by `POST /device/config` |
| `GET /device/events` | Count-probe fast path — `poll()+count_events()` (~2 s); returns cached data if event count is unchanged; full download only when new events are detected |
| `GET /device/monitor/status` | 30-second TTL; invalidated immediately on monitor start/stop |
| `GET /device/event/{idx}/waveform` | Permanent per-index (waveforms are immutable once recorded) |
- **`?force=true`** query param on all cached endpoints — bypasses cache and forces
a fresh read from the device.
- **Cache invalidation hooks** — `POST /device/config` marks device info and events
stale; `POST /device/monitor/start` and `/stop` evict the monitor status entry
immediately so the next status poll reflects the actual device state.
---
## v0.12.0 — 2026-04-13
### Added
- **`sfm/server.py``_LiveCache`** — in-memory live device cache, eliminating
redundant TCP round-trips between requests. No extra dependencies (plain Python
dict + threading.Lock). Replaces the SQLAlchemy-based `sfm/cache.py` experiment
from the `feature/intelligent-caching` branch.
Cache behaviour by endpoint:
| Endpoint | Cache strategy |
|---|---|
| `GET /device/info` | Indefinite; invalidated by `POST /device/config` |
| `GET /device/events` | Count-probe fast path: quick `poll()+count_events()` (~2s); return cache if count matches; full download only when new events detected |
| `GET /device/monitor/status` | 30-second TTL; invalidated by monitor start/stop |
| `GET /device/event/{idx}/waveform` | Permanent per-index (waveforms are immutable) |
- **`?force=true` param** on all four cached endpoints — bypasses cache and re-reads
from device.
- **`POST /device/config` cache invalidation** — marks device info + events dirty so
the next read reflects the new compliance config.
- **`POST /device/monitor/start` / `stop` cache invalidation** — evicts the monitor
status cache entry immediately so the next poll returns the updated state.
### Removed
- `sfm/cache.py` — SQLAlchemy-based cache from the experimental caching branch.
Its logic has been ported to the sqlite3-native `_LiveCache` class above.
`sqlalchemy` is no longer a dependency.
---
## v0.11.0 — 2026-04-13
### Added
- **`sfm/database.py` — SeismoDb** — SQLite persistence layer for all ACH data.
Three tables, all unit-keyed by serial number:
- `ach_sessions` — one row per inbound call-home: serial, timestamp, peer IP,
events_downloaded, monitor_entries, duration_seconds
- `events` — one row per triggered waveform event: serial, waveform_key (dedup),
timestamp, Tran/Vert/Long/VectorSum/Mic PPV, project/client/operator/sensor_location
strings, sample_rate, record_type, false_trigger flag
- `monitor_log` — one row per monitoring interval: serial, waveform_key (dedup),
start_time, stop_time, duration_seconds, geo_threshold_ips
- WAL mode, per-request connections — safe for the single-writer / occasional-reader
ACH server pattern
- Deduplication by `(serial, waveform_key)` UNIQUE constraint — re-runs and repeat
call-homes never produce duplicate rows
- **`ach_server.py` — DB integration** — after each successful call-home, writes new
events and monitor log entries to `seismo_relay.db` then records the session in
`ach_sessions`. DB write failures are logged as warnings and do not abort the session.
- **`sfm/server.py` — DB read endpoints**:
- `GET /db/units` — distinct serials with last_seen, total_events, total_monitor_entries
- `GET /db/events` — query events with serial / date range / false_trigger filters
- `GET /db/monitor_log` — query monitoring intervals
- `GET /db/sessions` — query ACH call-home sessions
- `PATCH /db/events/{id}/false_trigger` — flag/unflag false triggers (for review UI)
### Architecture
- seismo-relay DB is unit-keyed only — no project concepts. Project aggregation is
terra-view's responsibility via `UnitAssignment` / `DeploymentRecord` + date range
queries against the SFM DB endpoints.
- DB file lives at `bridges/captures/seismo_relay.db` by default.
---
## v0.10.0 — 2026-04-11
### Added
- **`MiniMateClient.get_monitor_log_entries(skip_keys=None)`** — browse-mode walk
(`1E → 0A → 1F`) that collects partial records (`0x2C` record type) from the device's
event list without triggering a full waveform download (no 0C or 5A). Returns
`list[MonitorLogEntry]`. Each entry represents one continuous monitoring interval where
no threshold was exceeded.
- **`_decode_0a_partial_header(raw_data, index, key4)`** in `client.py` — decodes a SUB
0x0A response payload whose record type is `0x2C`. Extracts:
- `start_time` / `stop_time` — two consecutive timestamps; auto-detects 9-byte
(sub_code=0x10, single-shot) vs 10-byte (sub_code=0x03, continuous) format from
`raw_data[11]`. Handles a 1-byte gap between the two timestamps that occurs when
ts1 and ts2 share the same minute:second.
- `serial` — device serial string found via `b"BE"` anchor scan.
- `geo_threshold_ips` — trigger level found via `b"Geo: "` anchor scan.
- **`MonitorLogEntry` dataclass** in `models.py` — new model for partial records:
`index`, `key`, `start_time`, `stop_time`, `serial`, `geo_threshold_ips`,
`raw_header`, and a `duration_seconds` property.
- **`read_waveform_header()` return value extended** — now returns `(data_rsp.data, length)`
(full payload) instead of `(data_rsp.data[11:11+length], length)`. Callers get the
complete payload including the record-type byte at position 0. Full records use
`raw_data[11:11+length]` as before; partial records are detected by `raw_data[0] == 0x2C`.
- **ACH server: monitor log collection** — after `get_events()`, calls
`get_monitor_log_entries(skip_keys=seen_keys)` and saves new entries to
`monitor_log.json` in the session directory. Monitor log keys are included in
`downloaded_keys` for state persistence (no re-processing on next call-home).
- **`_monitor_log_entry_to_dict()`** in `ach_server.py` — serialises a `MonitorLogEntry`
to a JSON-compatible dict with ISO-format timestamps.
### Protocol / Documentation
- **SUB 0x0A partial record (0x2C) format confirmed** (✅ 4-11-26 MITM capture, 12 frames):
- Record type `0x2C` at `raw_data[0]`; length < 64 bytes.
- Two timestamps at `raw_data[11:]` — start and stop of the monitoring interval.
- ASCII metadata region after timestamps: `BE<serial>\x00Geo: <float> in/s`.
- Edge case: 1-byte separator between timestamps when ts1 and ts2 share minute:second.
- 10-byte timestamp format (sub_code=0x03) signalled by `raw_data[11] == 0x10`.
- **Key reuse detection for monitor log entries** — monitor log keys are tracked alongside
event keys in `ach_state.json` so the ACH server does not re-process them after a
call-home cycle.
---
## v0.9.0 — 2026-04-11
### Added
- **`MiniMateClient.list_event_keys()`** — fast browse-mode walk (1E → 0A → 1F, no waveform
download) that returns the list of event key hex strings currently stored on the device.
Used by the ACH server as a cheap pre-check before deciding whether to call `get_events()`.
- **`get_events(skip_waveform_for_keys=set(...))`** — new optional parameter. For any key in
the set the function performs only 0A + 1F(browse) instead of the full
1E-arm → 0C → POLL×3 → 5A sequence. Eliminates redundant waveform downloads on repeat
call-homes when the device still holds previously downloaded events.
- **`MiniMateClient.delete_all_events()`** — erases all events from device memory using the
confirmed 4-step sequence:
- SUB 0xA3 `begin_erase_all` — initiate erase (token=0xFE) → ack 0x5C
- SUB 0x1C `read_monitor_status` — intermediate status read (Blastware-required)
- SUB 0x06 `read_event_storage_range` — verify storage state (token=0xFE) → 36-byte response
- SUB 0xA2 `confirm_erase_all` — commit erase (token=0xFE) → ack 0x5D
All four steps confirmed from 4-11-26 MITM capture of a live Blastware ACH session.
After a successful call, the device's event counter resets to `0x01110000`.
- **`MiniMateProtocol` erase methods**: `begin_erase_all()`, `confirm_erase_all()`,
`read_event_storage_range()` added to `protocol.py` with documented SUB constants
`SUB_ERASE_ALL_BEGIN = 0xA3` and `SUB_ERASE_ALL_CONFIRM = 0xA2`.
- **`bridges/ach_mitm.py`** — transparent TCP-to-TCP MITM proxy. Listens for inbound unit
connections, connects upstream to a real Blastware ACH server, and saves both directions
to `raw_bw_<ts>.bin` / `raw_s3_<ts>.bin` files matching the existing capture format.
Used to capture the 4-11-26 Blastware ACH session including event deletion.
Usage: `python bridges/ach_mitm.py --bw-host 127.0.0.1 --bw-port 9999 --listen-port 9998`
- **ACH server: key-based state tracking** — `ach_state.json` now stores
`downloaded_keys: [hex_strings]` and `max_downloaded_key: hex_string` per unit instead of
`event_count: N`. This correctly handles the standard workflow where events are deleted
from the device after upload — a count-based approach would see `count=0` on the next
call-home and silently skip new events.
- **ACH server: `--clear-after-download` flag** — after a successful download (at least one
new event saved), erases all events from the device using `delete_all_events()`. Mirrors
the standard Blastware ACH workflow. On success, `downloaded_keys` and
`max_downloaded_key` are reset to empty so the next session starts fresh.
- **ACH server: post-erase key-reuse detection** — after an external erase (Blastware or
manual), device keys restart from `0x01110000`, colliding with previously downloaded keys.
On each browse walk, if `max(device_keys) < max_downloaded_key` (device counter rolled
back), all device keys are treated as new regardless of `seen_keys`. This also catches
erases performed by Blastware between our sessions.
### Protocol / Documentation
- **SUB 0xA3 / SUB 0xA2 — erase-all sequence confirmed** (✅ 4-11-26 MITM capture):
Both frames use `token=0xFE` at `params[7]` and are standard `build_bw_frame` requests
(not write-format). Response SUBs follow the standard formula: 0x5C and 0x5D.
The intermediate 0x1C + 0x06 reads between them are required by Blastware.
- **SUB 0x06 — event storage range read confirmed** (✅ 4-11-26 MITM capture):
Two-step read, data offset = 0x24 (36 bytes). The last 8 bytes of the response contain
the first and last stored event keys (4 bytes each). After a successful erase, both keys
read as `01110000` (device-empty state).
- **Event key counter resets to `0x01110000` after erase** — confirmed by observing key
`01110000` on the device immediately after the MITM erase session.
---
## v0.8.0 — 2026-04-07
### Added
- **Write pipeline end-to-end** — `push_config_raw(event_index_data, compliance_data,
trigger_data, waveform_data)` on `MiniMateClient` orchestrates the full
`68→73 | 71×3→72 | 82→83 | 69→74→72` write sequence.
- **`build_bw_write_frame(sub, data, *, offset, params)`** in `framing.py` — dedicated frame
builder for write commands (SUBs 0x680x83). Doubles only the BW_CMD byte; all other
bytes including offset, params, data, and checksum are written raw. Uses the large-frame
DLE-aware checksum (`sum(b for b in payload[2:] if b != 0x10) + 0x10) & 0xFF`).
- **`MiniMateProtocol` write methods** — `write_event_index()`, `write_compliance()`,
`write_trigger_config()`, `write_waveform_data()`, `write_confirm()`,
`start_monitoring()`, `stop_monitoring()`.
- **`AchSession` inbound server** (`bridges/ach_server.py`) — accepts call-home TCP
connections, runs the full handshake + device-info + event-download sequence, saves
`device_info.json` + `events.json` per session.
### Protocol / Documentation
- **Write frame format confirmed** (✅ 3-11-26 BW TX capture, all 11 frames): only BW_CMD
byte `0x10` is doubled; all other bytes sent raw. Standard `build_bw_frame` DLE-stuffing
is incorrect for write commands.
- **Write ack responses** confirmed as 17-byte zero-data S3 frames.
- **Monitoring SUBs 0x96/0x97** confirmed from 4-8-26 capture.
- **SESSION_RESET signal** (`41 03`) required before POLL for monitoring units.
- **SUB 0x1C monitoring flag** at `section[1]`: `0x00` = idle, `0x10` = monitoring.
Confirmed by byte-diff of all 144 data frames in 4-8-26/2ndtry capture.
---
## v0.7.0 — 2026-04-03
### Added
+503 -41
View File
@@ -2,7 +2,9 @@
Ground-up Python replacement for **Blastware**, Instantel's Windows-only software for
managing MiniMate Plus seismographs. Connects over direct RS-232 or cellular modem
(Sierra Wireless RV50 / RV55). Current version: **v0.8.0**.
(Sierra Wireless RV50 / RV55). Current version: **v0.12.3**.
When new information about the protocol is discovered, please update the instantel_protocol_reference.md with the findings in addition to this document
---
@@ -25,9 +27,9 @@ CHANGELOG.md ← version history
---
## Current implementation state (v0.8.0)
## Current implementation state (v0.12.3)
Full read pipeline + write pipeline working end-to-end over TCP/cellular:
Full read pipeline + write pipeline + erase pipeline + monitor log + call home config working end-to-end over TCP/cellular:
| Step | SUB | Status |
|---|---|---|
@@ -41,12 +43,17 @@ Full read pipeline + write pipeline working end-to-end over TCP/cellular:
| Waveform record (peaks, timestamp, project) | 0C | ✅ |
| **Bulk waveform stream (event-time metadata)** | **5A** | ✅ new v0.6.0 |
| Event advance / next key | 1F | ✅ |
| **Write commands (push config to device)** | **6883** | ✅ **new v0.8.0** |
| **Write commands (push config to device)** | **6883** | ✅ new v0.8.0 |
| **Erase all events** | **0xA3 → 0x1C → 0x06 → 0xA2** | ✅ new v0.9.0 |
| **Monitor log entries (partial 0x2C records)** | **0A browse** | ✅ new v0.10.0 |
| **Auto Call Home config (read + write)** | **2C → 7E → 7F** | ✅ **new v0.12.3** |
`get_events()` sequence per event: `1E → 0A → 0C → 5A → 1F`
`push_config_raw()` write sequence: `68→73 | 71×3→72 | 82→83 | 69→74→72`
`delete_all_events()` erase sequence: `0xA3 → 0x1C → 0x06 → 0xA2`
---
## Protocol fundamentals
@@ -111,21 +118,29 @@ S3→BW (response):
Both differences confirmed by reproducing Blastware's exact wire bytes from the 1-2-26
BW TX capture. All 10 frames verified.
### SUB 5A — chunk counter is monotonic (CORRECTED 2026-04-06)
### SUB 5A — chunk counter formula (FINAL CORRECTION 2026-04-26)
**Chunk counters are `chunk_num * 0x0400` for ALL chunks including chunk 1.**
**Chunk counter = `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400` for ALL chunks.**
The 4-2-26 BW TX capture showed `counter=0x1004` for chunk 1 of event key `01110000`, which
led to `_CHUNK1_COUNTER = 0x1004` being hardcoded as a special case. This was a Blastware
artifact, not a protocol requirement. Empirical test 2026-04-06: with `counter=0x1004` for
chunk 1 the device times out (120 s); with `counter=0x0400` (= `1 * 0x0400`) it responds
immediately and streams all frames correctly.
where `key4[2:4] = (key4[2] << 8) | key4[3]` is the event's circular-buffer base offset.
The 4-3-26 capture confirms the pattern for a second event (key `0111245a`):
chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400). Blastware's
true formula is `key4[2:4] + n * 0x0400` — but since `key4[2:4]` of the first event is
`0x0000`, `n * 0x0400` produces the right result. The device does not strictly validate the
counter and streams data for any valid 5A request; using `chunk_num * 0x0400` is correct.
The `max(..., 0x0400)` guard is critical for events at the start of the circular buffer
(key4[2:4] == 0x0000, e.g. key `01110000`). Without it, chunk 1 gets counter=0x0000, which
is the same address as the probe frame — the device re-returns the STRT record data instead
of waveform payload. With the guard, chunk 1 gets counter=0x0400, which is confirmed correct
from the empirical live-device test 2026-04-06 (`counter=0x0400 → responds immediately and
streams all frames correctly`).
The 4-3-26 capture confirms the pattern for a second event (key `0111245a`, key4[2:4]=0x245a):
chunk 1 = `0x245A`, chunk 2 = `0x285A`, chunk 3 = `0x2C5A` (each +0x0400).
`max(0x245a, 0x0400) = 0x245a` → formula works correctly for non-zero base offset too.
**History:**
- Original: `_CHUNK1_COUNTER = 0x1004` hardcoded (Blastware capture artifact — WRONG).
- 2026-04-06: Corrected to `chunk_num * 0x0400` (worked for key 01110000 only).
- 2026-04-24: Corrected to `key4[2:4] + (chunk_num-1) * 0x0400` (fixed non-zero offsets,
but accidentally broke key 01110000 — counter=0x0000 sends probe address again).
- 2026-04-26: Final formula: `max(key4[2:4], 0x0400) + (chunk_num-1) * 0x0400`.
### SUB 5A — params are 11 bytes for chunk frames, 10 for termination
@@ -301,10 +316,16 @@ producing only ~1071 bytes instead of ~2126.
### SUB 1A — anchor search range
`_decode_compliance_config_into()` locates sample_rate and record_time via the anchor
`b'\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00'`. Search range is `cfg[0:150]`.
`_decode_compliance_config_into()` locates fields via the **6-byte stable anchor**
`b'\xbe\x80\x00\x00\x00\x00'`. Search range is `cfg[0:150]`.
Do not narrow this to `cfg[40:100]` — the old range was only accidentally correct because
**IMPORTANT — the "10-byte anchor" `\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00` is NOT fully constant.**
The first 2 bytes (`\x01\x2c` = 300) are the `histogram_interval_sec` field (uint16 BE, seconds) —
the value 300 is just the 5-minute default. When histogram interval is set to a different value
(e.g. 15min = 0x0384 = `\x03\x84`), those bytes change. Only the 6-byte suffix
`\xbe\x80\x00\x00\x00\x00` is truly constant. The code already uses the 6-byte anchor.
Do not narrow the search range to `cfg[40:100]` — the old range was only accidentally correct because
the orphaned-send bug was prepending a 44-byte spurious header, pushing the anchor from
its real position (cfg[11]) into the 40100 window.
@@ -326,6 +347,36 @@ Do NOT use fixed absolute offsets for sample_rate or record_time.
Quiet Mode enabled. Parser handles this — do not strip it manually before feeding to
`S3FrameParser`.
**SUB 5A (bulk waveform) TCP frame splitting — confirmed 2026-04-27:**
Over TCP via cellular modem, each 5A chunk request that produces a single ~1100-byte
A5 response over direct RS-232 may arrive as **two separate, complete S3 frames** of
~550 bytes each ("2-frame mode"). The modem's Data Forwarding Timeout (~100-150 ms)
can split the RS-232 response into two TCP segments, each parsed as a complete S3 frame.
Under different modem/timing conditions the full ~1100-byte response arrives as **one
S3 frame** ("1-frame mode").
**Both modes require `extra_chunks_after_metadata=1`** (the extra chunk at metadata_counter
+ 0x0400). The device's waveform footer data lives at circular-buffer address 0x1C00 for
this event; the terminator frame must be sent at 0x1C00 (not 0x1800) to receive it.
Example for a 2-second Continuous event (BE11529, key=01110000) via TCP:
- **2-frame mode:** 1 probe frame (554 B) + 5 chunks × 2 frames (556-573 B) + 1 extra chunk × 2 frames + 1 terminator (208 B) = **14 A5 frames** → 6864-byte file
- **1-frame mode:** 1 probe frame (~1097 B) + 5 chunks × 1 frame (~1079-1113 B) + 1 extra chunk × 1 frame (smaller, tail of event) + 1 terminator → **8 A5 frames** → 6864-byte file
- All frames contribute body data; using all of them gives the correct file.
**Fix (confirmed 2026-04-27):** `_recv_5a_batch()` in `protocol.py` collects ALL
A5 frames per chunk request before the next request is sent, using a 0.5 s batch
timeout after the first frame to catch the ~150 ms delayed second frame. `write_blastware_file()`
includes ALL body frames without skipping — the extra chunk's frames are part of the
body data, NOT padding to be discarded.
**WRONG earlier hypothesis (do not re-introduce):** An attempt was made to auto-detect
1-frame vs 2-frame mode from the probe frame size and skip the extra chunk when
`probe_data_len >= 700`. This was wrong — the extra chunk is always needed to advance
the device's internal state to the footer address. The `_probe_is_large` branch was
removed 2026-04-27.
### Required ACEmanager settings (Sierra Wireless RV50/RV55)
| Setting | Value | Why |
@@ -354,15 +405,72 @@ Do NOT use fixed absolute offsets for sample_rate or record_time.
| Field | How to find it |
|---|---|
| **recording_mode** | **uint8 at anchor 3 (write) / anchor 4 (read)** ✅ confirmed 2026-04-20 |
| sample_rate | uint16 BE at anchor 2 |
| **histogram_interval_sec** | **uint16 BE at anchor 4 (seconds); same offset in read & write** ✅ confirmed 2026-04-20 |
| record_time | float32 BE at anchor + 10 |
| trigger_level_geo | float32 BE, located in channel block |
| alarm_level_geo | float32 BE, adjacent to trigger_level_geo |
| max_range_geo | float32 BE, adjacent to alarm_level_geo |
| geo_hardware_constant (adc_scale_factor) | float32 BE at **channel_label+28** in both read (E5) and write (SUB 71) payloads — reads **6.206053** on BOTH tested units (BE11529 and BE18189); identical across all geo channels (Tran/Vert/Long) and all captures. **Confirmed 2026-04-17 from Interface Handbook §4.5**: this is the **ADC-to-velocity scale factor** = 1/sensitivity = (in/s per V). Firmware uses it as: `PPV (in/s) = ADC_voltage × 6.206053`. Cross-check: `1.61133 V (ADC full-scale) × 6.206053 = 10.000 in/s` (Normal range ✅). Do NOT write this field — it is a hardware/firmware constant. |
| geo_range (sensitivity selector) | **uint8 at channel_label+33** in both read (E5) and write (SUB 71) payloads — **CONFIRMED 2026-04-20** from 4-20-26 geo sensitivity captures: `0x00` = Normal 10.000 in/s (standard gain), `0x01` = Sensitive 1.250 in/s (high gain). Present in all three geo channel blocks (Tran, Vert, Long). **NOTE: `channel_label+20` reads `0x01` on ALL captures regardless of range setting — it is NOT this field.** Note: the "SUB 71 write offset = +29" that appears in earlier analysis was an artifact of incorrect BW-style destuffing applied to write frame data — write frame data is RAW, so the literal `0x10` bytes in the channel block header are preserved, and the offset is the same as in the E5 read payload. |
| setup_name | ASCII, null-padded, in cfg body |
| project / client / operator / sensor_location | ASCII, label-value pairs |
Anchor: `b'\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00'`, search `cfg[0:150]`
**True stable anchor: `b'\xbe\x80\x00\x00\x00\x00'` (6-byte suffix), search `cfg[0:150]`.**
The old "10-byte anchor" `b'\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00'` is partially variable:
bytes `\x01\x2c` = 300 (5-minute default histogram interval); changes when interval changes.
**Field layout relative to the 6-byte anchor (write payload / E5 read — noted where different):**
| Offset | Field | Format | Notes |
|---|---|---|---|
| anchor 9 | mode_prefix | uint8 | `0x00` for Single Shot / Continuous; `0x10` for Histogram (DLE prefix in E5 encoding) and Histogram+Continuous (actual config byte). See "compliance_raw DLE encoding" note below. |
| anchor 8 | recording_mode | uint8 | **Same offset for both read and write** — confirmed 2026-04-21. `_encode_compliance_config` writes `buf[anc-8]`. NOTE: for Histogram (0x03), E5 encodes the value as `0x10 0x03` so compliance_raw[anc-9]=0x10, compliance_raw[anc-8]=0x03. |
| anchor 7 | constant | `0x10` | Always `0x10` in both E5 read and BW write payloads (not a DLE marker — it is part of the sample_rate field area). Do NOT overwrite. |
| anchor 6 | sample_rate | uint16 BE | same in read & write |
| anchor 4 | histogram_interval_sec | uint16 BE | seconds; same in read & write ✅ 2026-04-20 |
| anchor 2 | `0x00 0x00` | padding | |
| anchor | `\xbe\x80\x00\x00\x00\x00` | anchor | |
| anchor + 6 | record_time | float32 BE | same in read & write |
**recording_mode enum** (confirmed 2026-04-20 from 4-20-26 captures):
| Value | Mode | anchor-9 in compliance_raw |
|---|---|---|
| `0x00` | Single Shot | `0x00` |
| `0x01` | Continuous | `0x00` |
| `0x02` | ❓ not observed | ❓ |
| `0x03` | Histogram | `0x10` (DLE prefix from E5 wire encoding of 0x03) |
| `0x04` | Histogram + Continuous | `0x10` (actual config byte for this mode) |
**compliance_raw DLE encoding — IMPORTANT (confirmed 2026-04-21 from 4-20-26 captures):**
`compliance_raw` (returned by `read_compliance_config()`) is NOT purely logical bytes — it is
the wire-encoded representation where `0x03` bytes in the config are preceded by a `0x10` DLE
prefix (because S3FrameParser preserves DLE+ETX inner-frame pairs as two literal bytes).
Consequences:
- When recording_mode = `0x03` (Histogram), `compliance_raw[anc-9] = 0x10` (DLE prefix) and
`compliance_raw[anc-8] = 0x03` (the value). The anchor position is +1 compared to modes
without `0x03` bytes before the anchor.
- For Histogram+Continuous (`0x04`), `compliance_raw[anc-9] = 0x10` for a different reason:
it is an actual stored config byte, not a DLE prefix.
- The anchor search (`buf.find(b'\xbe\x80\x00\x00\x00\x00', 0, 150)`) correctly locates
the anchor regardless of these mode-dependent shifts.
- When SFM writes recording_mode and round-trips the rest verbatim, the byte at `anc-9` is
preserved from the previous read. This means transitioning Histogram→other modes via SFM
leaves a `0x10` at `anc-9`. The device stores it as a literal byte; it does not affect
recording mode operation (which is at `anc-8`), but differs from what BW writes. This is a
known minor discrepancy that does not impact device behavior.
- **Histogram recording mode (0x03) write via SFM**: untested. When starting from a mode with
`anc-9 = 0x00`, SFM writes bare `0x03` at anc-8. BW would write `0x10 0x03`. Device likely
accepts both (write frames probably use offset/length for framing, not ETX scanning).
**DLE escaping in write frames — confirmed 2026-04-20:** Blastware escapes `0x03` bytes in
write frame data as `0x10 0x03` on the wire (defensive ETX escaping). Our `build_bw_write_frame`
does NOT do this escaping — it sends data bytes raw. Device acceptance of bare `0x03` bytes
in write frame data is confirmed for the tested modes (Single Shot, Continuous, Histogram+Continuous
where `0x10 0x03` already appears from round-tripping). Histogram mode (bare `0x03` write from
non-Histogram starting state) has not been directly tested.
### SUB 0C — Waveform Record (210 bytes = data[11:11+0xD2])
@@ -412,6 +520,8 @@ for 0x10 records).
## SFM REST API (sfm/server.py)
### Live device endpoints (connect to device per-request)
```
GET /device/info?port=COM5 ← serial
GET /device/info?host=1.2.3.4&tcp_port=9034 ← cellular
@@ -424,6 +534,19 @@ POST /device/monitor/stop?host=1.2.3.4&tcp_port=9034 ← stop recording
Server retries once on `ProtocolError` for TCP connections (handles cold-boot timing).
### DB read endpoints (query seismo_relay.db written by ach_server.py)
```
GET /db/units ← all known serials + summary stats
GET /db/events?serial=BE11529&from_dt=&to_dt=&limit= ← triggered events, newest first
GET /db/monitor_log?serial=BE11529&from_dt=&to_dt= ← monitoring intervals, newest first
GET /db/sessions?serial=BE11529&limit=50 ← ACH call-home sessions, newest first
PATCH /db/events/{id}/false_trigger?value=true ← flag/unflag false triggers
```
DB file: `bridges/captures/seismo_relay.db` (default; override with `--db-path` at startup).
All DB endpoints are read-only except `PATCH /db/events/{id}/false_trigger`.
---
## Key wire captures (reference material)
@@ -582,28 +705,32 @@ All confirmed from 4-8-26/2ndtry BW TX/S3 capture (clean start → 30s monitor
Standard two-step read (probe at offset 0x00, data at offset 0x2C).
Response SUB = 0xFF 0x1C = **0xE3** (standard formula — no exception).
**Payload length is ~4649 bytes in BOTH idle and monitoring states** — length alone
is NOT a reliable mode indicator. Earlier note claiming "12 bytes when monitoring"
was wrong (confirmed 2026-04-08 from 4-8-26/mid-monitor captures).
**Payload length is 4647 bytes IDLE, 4849 bytes MONITORING** — not a reliable sole
indicator due to 1-byte jitter overlap at the boundary.
**Monitoring flag (CORRECTED 2026-04-08 full byte diff of 2ndtry capture):**
- `section[6] == 0x00` → unit is **idle**
- `section[6] == 0x10` → unit is **monitoring**
**Monitoring flag (CONFIRMED 2026-04-09 — byte diff of all 144 data frames, 2ndtry capture):**
- `section[1] == 0x00` → unit is **idle**
- `section[1] == 0x10` → unit is **monitoring**
Earlier note claiming `section[1]` was the flag was WRONG — section[1] is always 0x00 in both states. The correction was found by diffing all 0xE3 data frames across the start/stop transitions: `section[6]` is the only byte that flips cleanly at frame #36 (start) and #132 (stop) within the 2ndtry 0xE3 frame sequence.
This is `data[12]` (= `frame.data[12]`). The flag is 0x00 in all 36 IDLE_BEFORE frames,
0x10 in all 98 MONITORING frames, and 0x00 in all 10 IDLE_AFTER frames — 100% accurate.
Battery and memory fields are present in **both** states, but the payload grows by **3 bytes** when monitoring is active (section goes from ~52 to ~55 bytes), shifting subsequent fields by +3.
**HISTORY OF THIS FIELD (do not re-derive):** The original implementation used `section[1]`.
A re-analysis in the prior session incorrectly concluded `section[1]` is always 0x00 and
"corrected" the flag to `section[6]`, which has non-binary values (0xea idle, 0x07 monitoring)
and is device-specific. The 2026-04-09 re-analysis confirms `section[1]` was right.
**Field offsets (relative to `data[11:]` = section):**
**IMPORTANT — `frame.data` has checksum already stripped** by `S3FrameParser._finalise()`
(`raw_payload = body[:-1]`; `data = raw_payload[5:]`). There is NO trailing checksum byte in
`section`. All relative-from-end offsets must account for this.
Battery and memory are at **relative offsets from the end** — the payload can vary by ±13 bytes due to counter jitter and monitoring-mode expansion, but these 10 bytes are always anchored at the end:
Battery and memory fields are present in **both** states:
| Offset (relative to end) | Field | Type | Notes |
|---|---|---|---|
| `section[-11:-9]` | battery voltage × 100 | uint16 BE | `0x02A8` = 680 → 6.80 V |
| `section[-9:-5]` | memory total (bytes) | uint32 BE | e.g. 983026 ≈ 960 KB |
| `section[-5:-1]` | memory free (bytes) | uint32 BE | decreases as events are stored |
| `section[-1]` | frame checksum | — | last byte, skip |
| `section[-10:-8]` | battery voltage × 100 | uint16 BE | `0x02A8` = 680 → 6.80 V |
| `section[-8:-4]` | memory total (bytes) | uint32 BE | e.g. 983026 ≈ 960 KB |
| `section[-4:]` | memory free (bytes) | uint32 BE | decreases as events are stored |
### SESSION_RESET signal (`41 03`) — required for monitoring units
@@ -657,7 +784,7 @@ Key findings:
**SFM behavior after `POST /device/monitor/start`:** `_pollMonitorConfirm()` polls
`/device/monitor/status` every 5 s for up to 60 s, updating the badge on each poll.
Status will show MONITORING once `section[6]` flips to `0x10`.
Status will show MONITORING once `section[1]` flips to `0x10`.
### SUBs known from sensor-check capture (4-8-26) — NOT YET IMPLEMENTED
@@ -678,16 +805,16 @@ Fields visible in the Blastware Compliance Setup dialog — most are NOT YET dec
offsets in the raw 1A/E5 payload. Only fields with `✅` have confirmed offsets in the code.
**Recording Setup tab:**
- Recording Mode: Continuous / Single Shot / Histogram (enum)
- Record Stop Mode: Fixed Record Time / Auto / Manual Stop (enum)
- Recording Mode: Continuous / Single Shot / Histogram / Histogram+Continuous ✅ (uint8 at anchor3 in write, anchor4 in read; 0x00=Single Shot, 0x01=Continuous, 0x03=Histogram, 0x04=Histogram+Continuous) — confirmed 2026-04-20
- Record Stop Mode: Fixed Record Time / Auto / Manual Stop (enum) ❓ (byte near recording_mode; data[40] in E5 sf1 changed 0x01→0x00 alongside Continuous→Single Shot — may be this field)
- Sample Rate: Standard 1024 / Fast 2048 / Faster 4096 sps ✅ (anchor2)
- Record Time: float, seconds ✅ (anchor+10)
- Histogram Interval: 5 / 15 / 30 / 60 minutes (enum, mode-gated)
- Histogram Interval: 2s / 5s / 15s / 1m / 5m / 15m ✅ (uint16 BE seconds at anchor4, same in read & write; mode-gated to Histogram/Histogram+Continuous) — confirmed 2026-04-20
- Storage Mode: Save All Data / Save Triggered (enum)
- Geophone Type: Standard Triaxial / 4.5 Hz (bool/enum)
- Geophone Channels: Enable all geophones (bool), Trigger Source (bool)
- Chan 1-3 Trigger Level (float, in/s) ✅ (`trigger_level_geo`)
- Chan 1-3 Maximum Range: Normal 10.000 / 1.25 in/s (enum) ✅ (`max_range_geo`)
- Chan 1-3 Maximum Range: Normal 10.000 / 1.25 in/s (enum) ✅ (`geo_range` uint8; **CONFIRMED 2026-04-20** from 4-20-26 geo sensitivity captures: offset = `channel_label+33` in both E5 read and SUB 71 write payloads (same bytes, round-tripped verbatim); `0x00` = Normal 10.000 in/s, `0x01` = Sensitive 1.250 in/s; applied to Tran/Vert/Long channel blocks). **IMPORTANT: `channel_label+20` reads `0x01` on ALL captures and is NOT this field** — it is a constant flag. The float32 at `channel_label+28` = 6.206053 is the ADC-to-velocity scale factor (hardware constant, do NOT write).
- Microphone Channels: Enable all microphones (bool), Trigger Source (bool)
- Chan 4 Trigger Level (dB or psi depending on units)
@@ -716,9 +843,344 @@ Full compliance config encoder is a future task.
---
---
## Erase-all protocol (SUBs 0xA3/0xA2/0x06) — confirmed 2026-04-11
Full sequence confirmed from 4-11-26 MITM capture of a live Blastware ACH session
(`bridges/captures/mitm/ach_mitm_20260411_001912/`).
### Wire sequence
```
BW → device: SUB 0xA3 params=00 00 00 00 00 00 00 FE 00 00 (begin erase)
device → BW: SUB 0x5C (ack)
BW → device: SUB 0x1C probe (offset=0x00)
device → BW: SUB 0xE3 (probe ack)
BW → device: SUB 0x1C data (offset=0x2C)
device → BW: SUB 0xE3 (monitor status response)
BW → device: SUB 0x06 probe (offset=0x00, params same)
device → BW: SUB 0xF9 (probe ack)
BW → device: SUB 0x06 data (offset=0x24)
device → BW: SUB 0xF9 (36-byte storage range response)
BW → device: SUB 0xA2 params=00 00 00 00 00 00 00 FE 00 00 (confirm erase)
device → BW: SUB 0x5D (ack — device memory is now cleared)
```
All frames use standard `build_bw_frame` (not write-format). Response SUBs follow the
standard `0xFF - SUB` formula; no exceptions.
### SUB 0x06 — event storage range response (36 bytes)
The 36-byte response body ends with two 4-byte event keys:
| Offset (from end) | Field | Notes |
|---|---|---|
| `[-8:-4]` | first stored event key | `01110000` when empty |
| `[-4:]` | last stored event key | `01110000` when empty |
Before erase: ends with `<first_key> <last_key>` (e.g. `0111ea60 0111eaa6`).
After erase: both bytes read `01110000` — device's empty/reset sentinel.
### Post-erase key counter reset
After a successful erase, the device resets its event counter. New events start from
key `0x01110000` again — the same key as the very first event ever recorded. This means
key-based deduplication in the ACH server must account for key reuse:
- After our own erase: `ach_state.json` `downloaded_keys` and `max_downloaded_key` are
cleared so the next session starts fresh.
- After an external erase: the ACH server detects it by comparing `max(device_keys)` to
`max_downloaded_key` from state. If the device max has rolled back below the historical
max, all current device keys are treated as new regardless of `seen_keys`.
### ACH server state format (v0.9.0)
`bridges/captures/ach_state.json`:
```json
{
"BE11529": {
"downloaded_keys": ["01110000", "0111245a"],
"max_downloaded_key": "0111245a",
"last_seen": "2026-04-11T01:04:36",
"serial": "BE11529",
"peer": "63.43.212.232:51920"
}
}
```
`max_downloaded_key` is the high-water mark — the largest key ever downloaded from the
unit. It is NOT reset when events are erased from the device (only when our server does
the erase). Used for post-erase detection.
---
## Monitor log entries — SUB 0x0A partial records (confirmed 2026-04-11)
Confirmed from 4-11-26 MITM capture: 12 partial records (record type `0x2C`) and 7 full
event records (record type `0x46`) across 19 total 0x0A responses.
### Record type detection
`read_waveform_header()` returns `(raw_data, length)` where `raw_data = data_rsp.data`
(the full payload including prefix bytes). The record type is at `raw_data[0]`:
| Value | Type | How to process |
|---|---|---|
| `0x46` | Full triggered event | Normal download: 0C → 5A → 1F |
| `0x2C` | Monitor log entry (partial) | No 0C/5A; decode inline from 0A payload |
Length heuristic: `length < 0x40` (64) reliably identifies partial records across all
observed captures. Both checks (`raw_data[0] == 0x2C` and `length < 0x40`) are used.
### SUB 0x0A partial record (0x2C) payload layout
All offsets are from `raw_data` (the full `data_rsp.data` array including the 11-byte
prefix before the actual header bytes start).
```
raw_data[0] = 0x2C ← record type (partial / monitor log)
raw_data[1:11] = prefix bytes (vary; contain key4 copy, flags, length)
raw_data[11:] = timestamp and ASCII metadata payload
```
**Timestamp auto-detection** (confirmed from 4-11-26 capture):
```
raw_data[11] == 0x10 → 10-byte sub_code=0x03 format (continuous mode)
raw_data[11] != 0x10 → 9-byte sub_code=0x10 format (single-shot mode)
```
**9-byte timestamp format (sub_code=0x10):**
| Byte | Field |
|---|---|
| 0 | day |
| 1 | `0x10` (sub_code marker) |
| 2 | month |
| 34 | year (uint16 BE) |
| 5 | unknown (0x00) |
| 6 | hour |
| 7 | minute |
| 8 | second |
**10-byte timestamp format (sub_code=0x03):**
| Byte | Field |
|---|---|
| 0 | `0x10` (marker) |
| 1 | day |
| 2 | `0x10` (marker) |
| 3 | month |
| 45 | year (uint16 BE) |
| 6 | unknown (0x00) |
| 7 | hour |
| 8 | minute |
| 9 | second |
**Two timestamps:** Each partial record contains two timestamps — `start_time` and
`stop_time` — stored consecutively:
- `ts1` (start) at `raw_data[ts_offset : ts_offset + ts_size]` where `ts_offset = 11`
- `ts2` (stop) at `raw_data[ts1_end : ts1_end + ts_size]`
**Edge case — 1-byte gap between timestamps:** Occurs when ts1 and ts2 share the same
minute:second. If `try_ts(raw_data[ts1_end:])` fails, try `try_ts(raw_data[ts1_end+1:])`.
Confirmed in frames 121, 161, 165 of the 4-11-26 MITM capture. Frame 121 still shows 0s
duration (both decode to 16:02:00) — the extra byte appears in all same-second cases.
**ASCII metadata after timestamps:**
```
<separator bytes> BE<serial>\x00Geo: <float> in/s ...
```
- Serial: scan for `b"BE"`, read until `b"\x00"` (e.g. `"BE11529"`)
- Geo threshold: scan for `b"Geo: "`, read float until next space (e.g. `0.254` in/s)
A separator of variable length (45 bytes of `\x00` + flags) sits between the two
timestamps and the ASCII region. The `b"BE"` anchor scan is robust to separator length
variation.
### `_decode_0a_partial_header(raw_data, index, key4)` — client.py
Returns a `MonitorLogEntry` or `None`. Called by `get_monitor_log_entries()` for each
event key whose 0x0A response has `raw_data[0] == 0x2C` or `length < 0x40`.
### `MiniMateClient.get_monitor_log_entries(skip_keys=None)` — client.py
Browse-mode walk: `1E → 0A → check type → decode if partial → 1F`. No 0x0C or 5A reads
performed. Full (0x46) records are skipped without decoding. Returns `list[MonitorLogEntry]`.
`skip_keys` (optional `set[str]`): keys in this set are still advanced through the walk
(to avoid disrupting the iteration sequence), but no `MonitorLogEntry` is created for them.
### `MonitorLogEntry` model — models.py
```python
@dataclass
class MonitorLogEntry:
index: int # 0-based position
key: str # 8-hex event key
start_time: Optional[datetime.datetime] = None
stop_time: Optional[datetime.datetime] = None
serial: Optional[str] = None
geo_threshold_ips: Optional[float] = None
raw_header: Optional[bytes] = field(default=None, repr=False)
@property
def duration_seconds(self) -> Optional[float]: ...
```
### ACH server integration (v0.10.0)
After `get_events()`, the ACH server calls `get_monitor_log_entries(skip_keys=seen_keys)`.
New entries are saved to `monitor_log.json` in the session directory. Monitor log keys are
included in `current_keys` for state persistence so they are not re-processed on the next
call-home.
---
## Auto Call Home config (SUBs 0x2C / 0x7E / 0x7F) — confirmed 2026-04-20
Full read/write pipeline confirmed from `bridges/captures/4-20-26/call home settings/`
(10 BW TX write frames diffed against the S3 read response).
Accessible in Blastware: **Remote Access → Setup Unit**.
### Protocol
**SUB 0x2C — Call Home Config READ (response 0xD3)**
Standard two-step read: probe offset `0x0000`, data offset `0x007C` (124).
Returns 125 raw bytes (one more than DATA_LENGTH) because the device encodes
num_retries value `3` as `\x10\x03` on the wire — S3FrameParser preserves both
bytes literally, shifting all subsequent field positions by +1.
**SUB 0x7E — Call Home Config WRITE (response 0x81)**
Write format (only BW_CMD `0x10` doubled on wire; DLE-aware checksum).
Payload = 125-byte read payload + `\x00\x00` = 127 bytes.
Offset = `data[1] + 2 = 0x7C + 2 = 0x7E`.
**SUB 0x7F — Call Home WRITE CONFIRM (response 0x80)**
Confirm frame, no data payload. Required after SUB 0x7E.
### Field map (raw 125-byte array from `data_rsp.data[11:]`)
| Raw Offset | Field | Notes |
|---|---|---|
| `[5]` | `auto_call_home_enabled` | `0x00`=off, `0x01`=on |
| `[6:46]` | `dial_string` | 40-byte null-padded ASCII |
| `[87]` | `after_event_recorded` | bool |
| `[91]` | `at_specified_times` | bool |
| `[93]` | `time1_enabled` | bool |
| `[101]` | `time1_hour` | 023 |
| `[102]` | `time1_min` | 059 |
| `[95]` | `time2_enabled` | bool |
| `[105]` | `time2_hour` | 023 |
| `[106]` | `time2_min` | 059 |
| `[117]` | DLE prefix `0x10` | Part of `\x10\x03` (DLE-escaped ETX encoding value 3) |
| `[118]` | `num_retries` | Value = 3; detect via `raw[117] == 0x10` |
| `[120]` | `time_between_retries_sec` | Shifted +1 from logical 119 |
| `[122]` | `wait_for_connection_sec` | Shifted +1 from logical 121 |
| `[124]` | `warm_up_time_sec` | Shifted +1 from logical 123 |
**DLE-escaped 0x03 at raw[117:119]:** The byte value `0x03` is indistinguishable from the
frame ETX terminator, so the device encodes it as `\x10\x03` (DLE + ETX inner-terminator).
S3FrameParser in `STATE_AFTER_DLE` on ETX appends both bytes as literal payload. The write
frame sends them verbatim — device accepts `\x10\x03` and interprets it as value 3.
**Unconfirmed fields:** time slots 3 and 4 (offsets unknown), `modem_power_relay_enabled`.
### `CallHomeConfig` model — models.py
```python
@dataclass
class CallHomeConfig:
raw: Optional[bytes] = None # 125-byte raw read payload
auto_call_home_enabled: Optional[bool] = None # raw[5]
dial_string: Optional[str] = None # raw[6:46]
after_event_recorded: Optional[bool] = None # raw[87]
at_specified_times: Optional[bool] = None # raw[91]
time1_enabled: Optional[bool] = None # raw[93]
time1_hour: Optional[int] = None # raw[101]
time1_min: Optional[int] = None # raw[102]
time2_enabled: Optional[bool] = None # raw[95]
time2_hour: Optional[int] = None # raw[105]
time2_min: Optional[int] = None # raw[106]
num_retries: Optional[int] = None # raw[118] (DLE-prefixed)
time_between_retries_sec: Optional[int] = None # raw[120] (shifted +1)
wait_for_connection_sec: Optional[int] = None # raw[122] (shifted +1)
warm_up_time_sec: Optional[int] = None # raw[124] (shifted +1)
```
### SFM REST API — sfm/server.py
```
GET /device/call_home?host=1.2.3.4&tcp_port=9034 ← read call home config
POST /device/call_home?host=1.2.3.4&tcp_port=9034 ← write call home config
```
POST body fields (all optional): `auto_call_home_enabled`, `after_event_recorded`,
`at_specified_times`, `time1_enabled`, `time1_hour`, `time1_min`, `time2_enabled`,
`time2_hour`, `time2_min`.
**Note:** `dial_string` is read-only in the current implementation (omitted from POST
body) because writing a dial string may require DLE escaping for embedded control characters.
---
## What's next
- **Database** — SQLite store for events + monitor log entries; dedup by key; queryable
- **Histograms** — decode histogram-mode A5 data (noise floor tracking)
- **Blastware-compatible file output** — `write_blastware_file()` and `write_mlg()` implemented. `blastware_filename()` generates correct Blastware filenames (AB0 for direct, AB0W/AB0H for ACH). **Confirmed working for Continuous mode events (2026-04-23):** SFM-generated file opens in Blastware, shows correct PPV/waveform/timestamp. File is ~200 bytes shorter than BW (missing last ADC tail slice) — all measurements correct. Histogram+Continuous mode deferred (5A stream for those events embeds histogram interval records that create spurious STRT markers in the body). Extension mapping: **CONFIRMED FALSE 2026-04-21** — extensions encode timestamp (AB0T for ACH, AB0 for direct), NOT recording mode. Filename format: `<prefix_letter><serial3><4-char-base36-stem><ext>`
**Serial encoding (CONFIRMED 2026-04-22):** `prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))`, `serial3 = f"{serial_numeric % 1000:03d}"`. Examples: BE6907→H907, BE11529→M529, BE14036→P036, BE17353→S353, BE18003→T003. The prefix letter encodes the production generation (batch of 1000 units).
**Stem encoding (FULLY CONFIRMED 2026-04-22):** stem = 4-char base-36 of `floor(total_seconds / 1296)` where `total_seconds = (event_local_time 1985-01-01T00:00:00_local)` in seconds. Epoch = `1985-01-01 00:00:00` device local time — confirmed against 3,248 files from 10-year production archive with zero errors. Decode: `event_time = datetime(1985,1,1) + timedelta(seconds=stem_int*1296 + ab_int)`. Example: P036L318.C80H → BE14036, 2025-05-26 15:00:08, Full Histogram.
- **Blastware filename extension — NEW FIRMWARE FULLY DECODED (confirmed 2026-04-21, further confirmed 2026-04-22 from 10-year production archive frequency analysis):**
Extension format = `AB0T` (4 chars):
- `AB` = 2-char base-36 encoding of `total_seconds % 1296` (seconds within the 21.6-min window, 01295); `A = value // 36`, `B = value % 36`
- `0` = always literal digit zero (third character, invariant)
- `T` = event type: `W` = Full Waveform, `H` = Full Histogram
Combined with the 4-char stem, the full filename encodes a complete second-resolution timestamp. Verified against three S353L4H0.{3M0W,8S0H,9X0W} events (all match to the second) plus large-scale frequency analysis of a 10-year archive.
**3-day cycle property (confirmed 2026-04-22):** A unit recording at a fixed daily time cycles through exactly **3 extensions** with a 3-day period. Each calendar day shifts `total_seconds % 1296` by 864 (since `86400 % 1296 = 864`). The cycle repeats every 3 days because `gcd(1296, 864) = 432` and `1296 / 432 = 3`. The three extension values are spaced 432 seconds apart. Confirmed from 10-year archive: the top 3 extensions overall were `CE0H` (95 files), `0E0H` (93), `OE0H` (91) — all three are the 3-day cycle of a 06:00:14 daily call-in time (seconds-in-window = 14, 446, 878; all three have `E` as second character because `14 = E` in base-36 and adding 864 never changes `value % 36` since `864 = 24 × 36`).
**B character invariance:** For a unit recording at a fixed time of day, the second character `B` of the extension (`value % 36`) **never changes** — only the first character `A` cycles through 3 values. This means same-time-of-day files from different dates all share the same `B` character.
**Old firmware (S338, 3-char extensions ending in `0`):** encoding unknown. Extension is NOT recording mode. `blastware_filename()` returns `.N00` as a placeholder for old-firmware units.
**Micromate Series 4** uses a different extension format entirely (observed: `IDFH`, `IDFW`). The `AB0T` formula applies only to MiniMate Plus / V10.72 firmware.
- Compliance config encoder — build raw write payloads from a `ComplianceConfig` object
- **Test Histogram recording mode (0x03) write via SFM** — confirmed working for Single Shot / Continuous / Histogram+Continuous; Histogram (0x03) needs a live test from a non-Histogram starting state (bare 0x03 in write vs BW's DLE-escaped `10 03`)
- **Compliance write anchor-9 cleanup** — when changing recording_mode via SFM, the byte at anchor-9 is not explicitly managed. A spurious `0x10` may persist after Histogram→other mode transitions. Does not affect device operation but differs from BW's byte-perfect output.
- Locate "Sensor Check" byte in compliance config (need capture with Disabled vs Before-monitoring)
- ACH inbound server — accept call-home connections from field units
- Call Home — map time slots 3/4 offsets; add dial_string write support; confirm `modem_power_relay_enabled`
- Modem manager — push RV50/RV55 configs via Sierra Wireless API
- RV55 DCD/DTR issue — newer RV55 firmware doesn't assert DCD by default; units don't
resume monitoring after call-home disconnect (`--restart-monitoring` flag deferred)
## BW capture reference
`bridges/captures/` contains the following BW TX + S3 response captures for protocol analysis:
| Folder / File | Contents |
|---|---|
| `3-11-26/raw_bw_20260311_170151.bin` | Full compliance write + event download (SUBs 68→83 confirmed, frames 102112) |
| `4-20-26/raw_bw_*_recording_mode_*.bin` | Recording mode changes: Continuous→Single Shot, →Histogram, →Histogram+Continuous |
| `4-20-26/histogram interval/` | Histogram interval changes: 1min, 5min, 15min, 15sec |
| `4-20-26/geo sensitivity/` | Geo sensitivity changes: 1.25 in/s (Sensitive), 10 in/s (Normal) |
| `4-20-26/call home settings/` | Call home config read/write captures |
| `4-8-26/` | Monitor status read, start/stop monitoring, SESSION_RESET signal, sensor check |
| `4-3-26-multi_event/` | Browse-mode S3 capture with 2+ events (1E/0A/1F iteration confirmed) |
| `4-2-26/` | Download-mode BW TX capture (5A bulk stream, POLL×3 requirement confirmed) |
| `3-31-26/` | Single-event download (148 BW / 147 S3 frames) |
| `mitm/ach_mitm_20260411_001912/` | Full ACH call-home MITM (erase protocol, 0xA3/0x06/0xA2 confirmed) |
To parse BW TX captures: use `bridges/captures/` scripts or adapt the `find_write_frames()` pattern
in `/tmp/analyze_write_payload.py` — it correctly handles `0x10 0x03` DLE-escaped ETX bytes
inside write frame data (the naive parser terminates early at the escaped `0x03`).
+170 -161
View File
@@ -1,16 +1,16 @@
# seismo-relay `v0.6.0`
# seismo-relay `v0.12.1`
A ground-up replacement for **Blastware** — Instantel's aging Windows-only
software for managing MiniMate Plus seismographs.
Built in Python. Runs on Windows. Connects to instruments over direct RS-232
or cellular modem (Sierra Wireless RV50 / RV55).
Built in Python. Runs on Windows, Linux, or macOS. Connects to instruments
over direct RS-232 or cellular modem (Sierra Wireless RV50 / RV55).
> **Status:** Active development. Full read pipeline working end-to-end:
> device info, compliance config (with geo thresholds), event download with
> true event-time metadata (project / client / operator / sensor location
> sourced from the device at record-time via SUB 5A). Write commands in progress.
> See [CHANGELOG.md](CHANGELOG.md) for version history.
> **Status:** Active development. Full read + write + erase + monitoring
> pipeline working end-to-end over TCP/cellular. ACH Auto Call Home server
> handles inbound unit connections, downloads events, and persists everything
> to a SQLite database. SFM REST API exposes device control and DB queries.
> See [CHANGELOG.md](CHANGELOG.md) for full version history.
---
@@ -21,26 +21,28 @@ seismo-relay/
├── seismo_lab.py ← Main GUI (Bridge + Analyzer + Console tabs)
├── minimateplus/ ← MiniMate Plus client library
│ ├── transport.py ← SerialTransport and TcpTransport
│ ├── protocol.py ← DLE frame layer (read/write/parse)
│ ├── client.py ← High-level client (connect, get_config, etc.)
│ ├── framing.py ← Frame builder/parser primitives
│ └── models.py ← DeviceInfo, EventRecord, etc.
│ ├── transport.py ← SerialTransport, TcpTransport, SocketTransport
│ ├── protocol.py ← DLE frame layer, SUB command dispatch
│ ├── client.py ← High-level client (connect, get_events, push_config, )
│ ├── framing.py ← Frame builders, DLE codec, S3FrameParser
│ └── models.py ← DeviceInfo, Event, ComplianceConfig, MonitorLogEntry, …
├── sfm/ ← SFM REST API server (FastAPI)
── server.py ← /device/info, /device/events, /device/event
├── sfm/ ← SFM REST API server (FastAPI, port 8200)
── server.py ← All device + DB endpoints
│ ├── database.py ← SeismoDb — SQLite persistence layer
│ └── sfm_webapp.html ← Embedded web UI (served at /)
├── bridges/
│ ├── s3-bridge/
│ └── s3_bridge.py RS-232 serial bridge (capture tool)
│ ├── ach_server.py ← Inbound ACH call-home server (main production server)
├── ach_mitm.py ← Transparent MITM proxy for capturing BW sessions
│ ├── s3-bridge/ ← RS-232 serial bridge (capture tool)
│ ├── tcp_serial_bridge.py ← Local TCP↔serial bridge (bench testing)
│ ├── gui_bridge.py ← Standalone bridge GUI (legacy)
│ ├── gui_bridge.py ← Standalone bridge GUI
│ └── raw_capture.py ← Simple raw capture tool
├── parsers/
│ ├── s3_parser.py ← DLE frame extractor
│ ├── s3_analyzer.py ← Session parser, differ, Claude export
│ ├── gui_analyzer.py ← Standalone analyzer GUI (legacy)
│ ├── gui_analyzer.py ← Standalone analyzer GUI
│ └── frame_db.py ← SQLite frame database
└── docs/
@@ -51,123 +53,88 @@ seismo-relay/
## Quick start
### Seismo Lab (main GUI)
### ACH inbound server (production)
The all-in-one tool. Three tabs: **Bridge**, **Analyzer**, **Console**.
Listens for inbound unit call-homes, downloads all new events and monitor log
entries, and writes everything to `bridges/captures/seismo_relay.db`.
```bash
python bridges/ach_server.py --port 12345 --output bridges/captures/
```
python seismo_lab.py
Point the unit's ACEmanager **Remote Host** to this machine's IP and **Remote Port** to `12345`.
Options:
```
--port N Listen port (default 12345)
--output DIR Capture directory (default bridges/captures/)
--allow-ip IP Allowlist an IP (repeat for multiple; default: accept all)
--max-events N Safety cap for first run (default: unlimited)
--clear-after-download Erase device memory after successful download
--verbose Debug logging
```
### SFM REST server
Exposes MiniMate Plus commands as a REST API for integration with other systems.
Exposes device control and DB queries as a REST API. Proxied by terra-view.
```
cd sfm
uvicorn server:app --reload
```bash
python sfm/server.py # default: 0.0.0.0:8200
python -m uvicorn sfm.server:app --host 0.0.0.0 --port 8200 --reload
```
**Endpoints:**
Open `http://localhost:8200` for the embedded web UI, or `http://localhost:8200/docs`
for the interactive API docs.
### Seismo Lab GUI
```bash
python seismo_lab.py
```
---
## SFM REST API
### Live device endpoints
Each call dials the device, does its work, and closes the connection. TCP
connections are retried once on `ProtocolError` to handle cold-boot timing.
**Caching** — frequently-polled endpoints are cached in-process to avoid
redundant TCP round-trips:
| Method | URL | Cache |
|--------|-----|-------|
| `GET` | `/device/info` | Indefinite; invalidated by `POST /device/config` |
| `GET` | `/device/events` | Count-probe fast path (~2s); full download only when new events detected |
| `GET` | `/device/event/{idx}/waveform` | Permanent per event index |
| `GET` | `/device/monitor/status` | 30-second TTL |
| `POST` | `/device/connect` | — |
| `POST` | `/device/config` | Writes compliance config; invalidates cache |
| `POST` | `/device/monitor/start` | Sends SUB 0x96 |
| `POST` | `/device/monitor/stop` | Sends SUB 0x97 |
All cached endpoints accept `?force=true` to bypass the cache.
Transport query params (supply one set):
```
Serial: ?port=COM5&baud=38400
TCP: ?host=1.2.3.4&tcp_port=12345
```
### DB read endpoints
Query the SQLite database written by `ach_server.py`. All read-only except
`PATCH /db/events/{id}/false_trigger`.
| Method | URL | Description |
|--------|-----|-------------|
| `GET` | `/device/info?port=COM5` | Device info via serial |
| `GET` | `/device/info?host=1.2.3.4&tcp_port=9034` | Device info via cellular modem |
| `GET` | `/device/events?port=COM5` | Event index |
| `GET` | `/device/event?port=COM5&index=0` | Single event record |
---
## Seismo Lab tabs
### Bridge tab
Captures live RS-232 traffic between Blastware and the seismograph. Sits in
the middle as a transparent pass-through while logging everything to disk.
```
Blastware → COM4 (virtual) ↔ s3_bridge ↔ COM5 (physical) → MiniMate Plus
```
Set your COM ports and log directory, then hit **Start Bridge**. Use
**Add Mark** to annotate the capture at specific moments (e.g. "changed
trigger level"). When the bridge starts, the Analyzer tab automatically wires
up to the live files and starts updating in real time.
### Analyzer tab
Parses raw captures into DLE-framed protocol sessions, diffs consecutive
sessions to show exactly which bytes changed, and lets you query across all
historical captures via the built-in SQLite database.
- **Inventory** — all frames in a session, click to drill in
- **Hex Dump** — full payload hex dump with changed-byte annotations
- **Diff** — byte-level before/after diff between sessions
- **Full Report** — plain text session report
- **Query DB** — search across all captures by SUB, direction, or byte value
Use **Export for Claude** to generate a self-contained `.md` report for
AI-assisted field mapping.
### Console tab
Direct connection to a MiniMate Plus — no bridge, no Blastware. Useful for
diagnosing field units over cellular without a full capture session.
**Connection:** choose Serial (COM port + baud) or TCP (IP + port for
cellular modem).
**Commands:**
| Button | What it does |
|--------|-------------|
| POLL | Startup handshake — confirms unit is alive and identifies model |
| Serial # | Reads unit serial number |
| Full Config | Reads full 166-byte config block (firmware version, channel scales, etc.) |
| Event Index | Reads stored event list |
Output is colour-coded: TX in blue, raw RX bytes in teal, decoded fields in
green, errors in red. **Save Log** writes a timestamped `.log` file to
`bridges/captures/`. **Send to Analyzer** injects the captured bytes into the
Analyzer tab for deeper inspection.
---
## Connecting over cellular (RV50 / RV55 modems)
Field units connect via Sierra Wireless RV50 or RV55 cellular modems. Use
TCP mode in the Console or SFM:
```
# Console tab
Transport: TCP
Host: <modem public IP>
Port: 9034 ← Device Port in ACEmanager (call-up mode)
```
```python
# In code
from minimateplus.transport import TcpTransport
from minimateplus.client import MiniMateClient
client = MiniMateClient(transport=TcpTransport("1.2.3.4", 9034), timeout=30.0)
info = client.connect()
```
### Required ACEmanager settings (Serial tab)
These must match exactly — a single wrong setting causes the unit to beep
on connect but never respond:
| Setting | Value | Why |
|---------|-------|-----|
| Configure Serial Port | `38400,8N1` | Must match MiniMate baud rate |
| Flow Control | `None` | Hardware flow control blocks unit TX if pins unconnected |
| **Quiet Mode** | **Enable** | **Critical.** Disabled → modem injects `RING`/`CONNECT` onto serial line, corrupting the S3 handshake |
| Data Forwarding Timeout | `1` (= 0.1 s) | Lower latency; `5` works but is sluggish |
| TCP Connect Response Delay | `0` | Non-zero silently drops the first POLL frame |
| TCP Idle Timeout | `2` (minutes) | Prevents premature disconnect |
| DB9 Serial Echo | `Disable` | Echo corrupts the data stream |
| `GET` | `/db/units` | All known serials with summary stats |
| `GET` | `/db/events` | Triggered events (filter by serial, date range, false_trigger) |
| `GET` | `/db/monitor_log` | Monitoring intervals |
| `GET` | `/db/sessions` | ACH call-home session history |
| `PATCH` | `/db/events/{id}/false_trigger?value=true` | Flag / unflag false triggers |
---
@@ -175,25 +142,76 @@ on connect but never respond:
```python
from minimateplus import MiniMateClient
from minimateplus.transport import SerialTransport, TcpTransport
from minimateplus.transport import TcpTransport
# Serial
client = MiniMateClient(port="COM5")
# TCP (cellular modem)
client = MiniMateClient(transport=TcpTransport("1.2.3.4", 9034), timeout=30.0)
client = MiniMateClient(transport=TcpTransport("1.2.3.4", 12345), timeout=30.0)
with client:
info = client.connect() # DeviceInfo — model, serial, firmware, compliance config
serial = client.get_serial() # Serial number string
config = client.get_config() # Full config block (bytes)
events = client.get_events() # List[EventRecord] with true event-time metadata
# Read
info = client.connect() # DeviceInfo — serial, firmware, compliance config
count = client.count_events() # Number of stored events
keys = client.list_event_keys() # Fast browse walk — event keys only, no download
events = client.get_events() # Full download: headers + peaks + metadata
monitor = client.get_monitor_status() # Battery, memory, is_monitoring flag
log = client.get_monitor_log_entries() # Monitoring intervals (partial 0x2C records)
# Write
client.apply_config(
sample_rate=1024,
trigger_level_geo=0.5,
project="Bridge Inspection 2026",
client_name="City of Portland",
operator="B. Harrison",
)
# Control
client.start_monitoring() # SUB 0x96
client.stop_monitoring() # SUB 0x97
client.delete_all_events() # Erase all (SUB 0xA3 → 0x1C → 0x06 → 0xA2)
```
`get_events()` runs the full download sequence per event: `1E → 0A → 0C → 5A → 1F`.
The SUB 5A bulk waveform stream is used to retrieve `client`, `operator`, and
`sensor_location` as they existed at record time — not backfilled from the current
compliance config.
`get_events()` runs the full per-event sequence: `1E → 0A → 0C → 5A → 1F`.
SUB 5A bulk stream provides `client`, `operator`, and `sensor_location` as they
existed at record time — not backfilled from the current compliance config.
---
## Database
`ach_server.py` writes to `bridges/captures/seismo_relay.db` (SQLite, WAL mode).
Three tables, all unit-keyed by serial number:
| Table | Key | Contents |
|-------|-----|----------|
| `ach_sessions` | UUID | Per-call-home audit record: serial, peer IP, events_downloaded, duration |
| `events` | UUID, UNIQUE(serial, waveform_key) | Triggered events: timestamp, PPV per channel, project/client/operator strings, false_trigger flag |
| `monitor_log` | UUID, UNIQUE(serial, waveform_key) | Monitoring intervals: start/stop time, duration, geo threshold |
Deduplication is by `(serial, waveform_key)` — repeat call-homes or re-runs
never produce duplicate rows. Post-erase key reuse is handled automatically
via the high-water mark in `ach_state.json`.
---
## Connecting over cellular (RV50 / RV55)
Field units connect via Sierra Wireless RV50 or RV55 cellular modems.
### Required ACEmanager settings
| Setting | Value | Why |
|---------|-------|-----|
| Configure Serial Port | `38400,8N1` | Must match MiniMate baud rate |
| Flow Control | `None` | Hardware FC blocks TX if pins unconnected |
| **Quiet Mode** | **Enable** | **Critical** — disabled injects `RING`/`CONNECT` onto serial, corrupting the S3 handshake |
| Data Forwarding Timeout | `1` (= 0.1 s) | Lower latency |
| TCP Connect Response Delay | `0` | Non-zero silently drops the first POLL frame |
| TCP Idle Timeout | `2` (minutes) | Prevents premature disconnect |
| DB9 Serial Echo | `Disable` | Echo corrupts the data stream |
---
@@ -204,23 +222,10 @@ compliance config.
| DLE | `0x10` | Data Link Escape |
| STX | `0x02` | Start of frame |
| ETX | `0x03` | End of frame |
| ACK | `0x41` (`'A'`) | Frame-start marker sent before every frame |
| ACK | `0x41` | Frame-start marker sent before every BW frame |
| DLE stuffing | `10 10` on wire | Literal `0x10` in payload |
**S3-side frame** (seismograph → Blastware): `ACK DLE+STX [payload] CHK DLE+ETX`
**De-stuffed payload header:**
```
[0] CMD 0x10 = BW request, 0x00 = S3 response
[1] ? unknown (0x00 BW / 0x10 S3)
[2] SUB Command/response identifier ← the key field
[3] PAGE_HI Page address high byte
[4] PAGE_LO Page address low byte
[5+] DATA Payload content
```
**Response SUB rule:** `response_SUB = 0xFF - request_SUB`
Example: request SUB `0x08` (Event Index) → response SUB `0xF7`
**Response SUB rule:** `response_SUB = 0xFF - request_SUB` (no exceptions)
Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/instantel_protocol_reference.md)
@@ -228,32 +233,36 @@ Full protocol documentation: [`docs/instantel_protocol_reference.md`](docs/insta
## Requirements
```
```bash
pip install pyserial fastapi uvicorn
```
Python 3.10+. Tkinter is included with the standard Python installer on
Windows (make sure "tcl/tk and IDLE" is checked during install).
Windows (check "tcl/tk and IDLE" during install).
---
## Virtual COM ports (bridge capture)
The bridge needs two COM ports on the same PC — one that Blastware connects
to, and one wired to the seismograph. Use a virtual COM port pair
(**com0com** or **VSPD**) to give Blastware a port to talk to.
```
Blastware → COM4 (virtual) ↔ s3_bridge.py ↔ COM5 (physical) → MiniMate Plus
```
Use **com0com** or **VSPD** to create the virtual COM pair on Windows.
---
## Roadmap
- [x] Event download — pull waveform records from the unit (`1E → 0A → 0C → 5A → 1F`)
- [x] True event-time metadata — project / client / operator / sensor location from SUB 5A
- [ ] Write commands — push config changes to the unit (compliance setup, channel config, trigger settings)
- [ ] ACH inbound server — accept call-home connections from field units
- [ ] Modem manager — push standard configs to RV50/RV55 fleet via Sierra Wireless API
- [ ] Full Blastware parity — complete read/write/download cycle without Blastware
- [x] Full read pipeline — device info, compliance config, event download with true event-time metadata
- [x] Write commands — push compliance config, trigger thresholds, project strings to device
- [x] Erase all events — confirmed erase sequence from live MITM capture
- [x] Monitor control — start/stop monitoring, read battery/memory/status
- [x] Monitor log entries — decode partial 0x2C records (continuous monitoring intervals)
- [x] ACH inbound server — accept call-home connections, download events, dedup by key
- [x] SQLite persistence — events, monitor log, and session history in `seismo_relay.db`
- [x] SFM REST API — device control + DB query endpoints, live device cache
- [ ] Terra-view integration — seismo-relay router, unit detail page, VISON-style event listing
- [ ] Vibration summary reports — highest legit PPV per project → Word doc (false trigger filtering first)
- [ ] Compliance config encoder — build raw write payloads from a `ComplianceConfig` object
- [ ] Modem manager — push RV50/RV55 configs via Sierra Wireless API
+627
View File
@@ -0,0 +1,627 @@
#!/usr/bin/env python3
"""
ach_bridge.py — Transparent TCP bridge / splitter for Instantel MiniMate Plus
call-home (ACH) traffic.
Modes
-----
standalone Accept connection, capture frames, do NOT forward anywhere.
Good for initial discovery with a test unit.
bridge Forward to one upstream server while capturing.
Use this for the initial discovery phase with your test server.
splitter Forward to the PRIMARY upstream (production ACH server) AND
mirror a copy to a SECONDARY server simultaneously.
The device never knows — it talks to the primary the whole time.
If the mirror fails, the primary connection is unaffected.
Think of it like a headphone splitter: one input, two outputs.
Primary → authoritative responses back to device.
Mirror → gets all device bytes, its responses are discarded.
Usage
-----
# Standalone capture (test/discovery — no forwarding)
python bridges/ach_bridge.py --standalone [--port 12345]
# Bridge mode (forward to one server, e.g. your test server)
python bridges/ach_bridge.py --upstream HOST:PORT [--port 12345]
# Splitter mode (production: forward to prod + mirror to your server)
python bridges/ach_bridge.py --upstream PROD_HOST:PORT --mirror MY_HOST:PORT [--port 12345]
Setup for discovery (test server, don't touch prod)
----------------------------------------------------
1. Stand up your test ACH server, note its IP and port (e.g. 192.168.1.50:12345).
2. Take ONE test unit. In ACEmanager → Call Home, point it at:
<this machine's LAN IP> : <--port>
3. Run: python bridges/ach_bridge.py --upstream TEST_SERVER:12345 --port 12345
4. Trigger the unit. Raw frames are saved to bridges/captures/ach_<ts>/.
5. Revert the unit's ACEmanager setting when done.
Setup for production splitter (when you're ready)
-------------------------------------------------
This does NOT touch the units. Instead you re-route traffic at the network
layer so that call-home packets arrive at a machine running this script first.
Typical approach: update the DNS entry / host record your prod ACH server is
registered under to point at this machine. The units keep their existing
ACEmanager settings.
python bridges/ach_bridge.py \\
--upstream PROD_ACH_HOST:12345 \\
--mirror MY_NEW_SERVER:12345 \\
--port 12345
Output (each connection gets its own timestamped sub-directory)
------
bridges/captures/ach_<ts>/
raw_client_<ts>.bin — raw bytes from the device (S3 side)
raw_server_<ts>.bin — raw bytes from the primary upstream (BW side)
raw_mirror_<ts>.bin — raw bytes from the mirror upstream (splitter mode only)
session_<ts>.log — human-readable frame parse log
session_<ts>.jsonl — JSON-lines frame log
raw_client / raw_server are byte-for-byte compatible with parse_capture.py.
"""
from __future__ import annotations
import argparse
import asyncio
import datetime
import json
import logging
import os
import sys
from pathlib import Path
from typing import List, Optional
# Add project root to path
sys.path.insert(0, str(Path(__file__).parent.parent))
from minimateplus.framing import S3FrameParser, S3Frame
log = logging.getLogger("ach_bridge")
# ── Frame label helpers ──────────────────────────────────────────────────────
_KNOWN_RSP_SUBS = {
0xA4: "POLL_RSP",
0xA5: "BULK_WAVEFORM_RSP",
0xE0: "ADVANCE_EVENT_RSP",
0xE1: "EVENT_INDEX_FIRST_RSP",
0xE3: "MONITOR_STATUS_RSP",
0xEA: "SERIAL_NUM_RSP",
0xF3: "WAVEFORM_RECORD_RSP",
0xF5: "WAVEFORM_HEADER_RSP",
0xF7: "EVENT_INDEX_RSP",
0xF9: "UNK_06_RSP",
0xFE: "DEVICE_INFO_RSP",
# Write acks
0x97: "EVT_IDX_WRITE_ACK",
0x8C: "CONFIRM_B_ACK",
0x8E: "COMPLIANCE_WRITE_ACK",
0x8D: "CONFIRM_A_ACK",
0x7D: "TRIGGER_WRITE_ACK",
0x7C: "TRIGGER_CONFIRM_ACK",
0x96: "WAVEFORM_WRITE_ACK",
0x8B: "CONFIRM_C_ACK",
0x69: "START_MONITOR_ACK",
0x68: "STOP_MONITOR_ACK",
}
_KNOWN_REQ_SUBS = {
0x5B: "POLL",
0x5A: "BULK_WAVEFORM",
0x1F: "ADVANCE_EVENT",
0x1E: "EVENT_INDEX_FIRST",
0x1C: "MONITOR_STATUS",
0x15: "SERIAL_NUM",
0x0C: "WAVEFORM_RECORD",
0x0A: "WAVEFORM_HEADER",
0x08: "EVENT_INDEX",
0x06: "UNK_06",
0x01: "DEVICE_INFO",
# Write commands
0x68: "EVT_IDX_WRITE",
0x73: "CONFIRM_B",
0x71: "COMPLIANCE_WRITE",
0x72: "CONFIRM_A",
0x82: "TRIGGER_WRITE",
0x83: "TRIGGER_CONFIRM",
0x69: "WAVEFORM_WRITE",
0x74: "CONFIRM_C",
0x96: "START_MONITOR",
0x97: "STOP_MONITOR",
}
def _label_s3_frame(frame: S3Frame) -> str:
name = _KNOWN_RSP_SUBS.get(frame.sub, f"UNK_0x{frame.sub:02X}")
chk = "" if frame.checksum_valid else "✗CHK"
return (
f"S3→ SUB=0x{frame.sub:02X} ({name}) "
f"page=0x{frame.page_key:04X} data={len(frame.data)}B {chk}"
)
def _label_bw_frame(data: bytes, prefix: str = " →BW") -> str:
"""Best-effort label for a raw BW request frame (wire bytes)."""
# Wire layout: 41 02 10 10 00 sub ...
if len(data) < 6:
return f"{prefix} (short {len(data)}B)"
sub = data[5]
name = _KNOWN_REQ_SUBS.get(sub, f"UNK_0x{sub:02X}")
return f"{prefix} SUB=0x{sub:02X} ({name}) {len(data)}B"
# ── Per-session capture writer ─────────────────────────────────────────────────
class CaptureSession:
"""Writes raw bytes + parsed log for one TCP connection."""
def __init__(self, capture_dir: Path, peer: str, *, has_mirror: bool = False):
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
self.dir = capture_dir / f"ach_{ts}"
self.dir.mkdir(parents=True, exist_ok=True)
self.peer = peer
self._raw_client = open(self.dir / f"raw_client_{ts}.bin", "wb")
self._raw_server = open(self.dir / f"raw_server_{ts}.bin", "wb")
self._raw_mirror = (
open(self.dir / f"raw_mirror_{ts}.bin", "wb") if has_mirror else None
)
self._log_fh = open(self.dir / f"session_{ts}.log", "w")
self._jsonl_fh = open(self.dir / f"session_{ts}.jsonl", "w")
self._s3_parser = S3FrameParser()
self._frame_count = 0
self._byte_count_client = 0
self._byte_count_server = 0
self._byte_count_mirror = 0
self._log(
f"# ACH capture — peer={peer} "
f"mirror={'yes' if has_mirror else 'no'} "
f"started={datetime.datetime.now().isoformat()}"
)
self._log(f"# Output dir: {self.dir}")
log.info("Capture session opened: %s (peer=%s)", self.dir, peer)
# ── public API ────────────────────────────────────────────────────────────
def feed_client(self, data: bytes) -> None:
"""Bytes FROM the device (S3 response frames)."""
self._raw_client.write(data)
self._raw_client.flush()
self._byte_count_client += len(data)
for byte in data:
frame = self._s3_parser.feed(bytes([byte]))
if frame:
frames = frame if isinstance(frame, list) else [frame]
for f in frames:
self._frame_count += 1
label = _label_s3_frame(f)
self._log(f"[{self._frame_count:04d}] {label}")
self._log(
f" hex: {f.data[:64].hex()}"
+ (" ..." if len(f.data) > 64 else "")
)
self._emit_json("s3", f)
def feed_server(self, data: bytes) -> None:
"""Bytes FROM the primary upstream server (BW request frames)."""
self._raw_server.write(data)
self._raw_server.flush()
self._byte_count_server += len(data)
label = _label_bw_frame(data, prefix=" →BW[primary]")
self._log(f" {label}")
def feed_mirror(self, data: bytes) -> None:
"""Bytes FROM the mirror server (logged, not forwarded to device)."""
if self._raw_mirror:
self._raw_mirror.write(data)
self._raw_mirror.flush()
self._byte_count_mirror += len(data)
label = _label_bw_frame(data, prefix=" →BW[mirror] ")
self._log(f" {label} [MIRROR — not sent to device]")
def close(self, reason: str = "connection closed") -> None:
self._log(f"# Session ended: {reason}")
self._log(
f"# Totals — client={self._byte_count_client}B "
f"server={self._byte_count_server}B "
f"mirror={self._byte_count_mirror}B "
f"s3_frames={self._frame_count}"
)
handles = [self._raw_client, self._raw_server, self._log_fh, self._jsonl_fh]
if self._raw_mirror:
handles.append(self._raw_mirror)
for fh in handles:
try:
fh.close()
except Exception:
pass
log.info(
"Session closed (%s): %dB client, %dB server, %dB mirror, %d S3 frames → %s",
reason,
self._byte_count_client, self._byte_count_server,
self._byte_count_mirror, self._frame_count,
self.dir,
)
# ── internals ─────────────────────────────────────────────────────────────
def _log(self, msg: str) -> None:
print(msg, file=self._log_fh, flush=True)
print(msg)
def _emit_json(self, direction: str, frame: S3Frame) -> None:
record = {
"dir": direction,
"sub": frame.sub,
"page_key": frame.page_key,
"data_len": len(frame.data),
"data_hex": frame.data.hex(),
"checksum_valid": frame.checksum_valid,
}
print(json.dumps(record), file=self._jsonl_fh, flush=True)
# ── Bridge / splitter connection handler ──────────────────────────────────────
class BridgeHandler:
"""
Handles inbound device connections.
Modes (determined by which upstreams are configured):
standalone — no upstream_host / no mirror_host
bridge — upstream_host set, no mirror_host
splitter — upstream_host AND mirror_host both set
"""
def __init__(
self,
capture_dir: Path,
upstream_host: Optional[str],
upstream_port: Optional[int],
mirror_host: Optional[str] = None,
mirror_port: Optional[int] = None,
):
self.capture_dir = capture_dir
self.upstream_host = upstream_host
self.upstream_port = upstream_port
self.mirror_host = mirror_host
self.mirror_port = mirror_port
async def handle(
self,
client_reader: asyncio.StreamReader,
client_writer: asyncio.StreamWriter,
) -> None:
peer = client_writer.get_extra_info("peername", ("?", 0))
peer_str = f"{peer[0]}:{peer[1]}"
log.info("Inbound connection from %s", peer_str)
has_mirror = bool(self.mirror_host)
session = CaptureSession(self.capture_dir, peer_str, has_mirror=has_mirror)
if not self.upstream_host:
# ── Standalone mode ──────────────────────────────────────────────
log.info("Standalone mode — recording inbound traffic only")
try:
while True:
data = await client_reader.read(4096)
if not data:
break
session.feed_client(data)
except asyncio.CancelledError:
pass
except Exception as exc:
log.warning("Standalone read error: %s", exc)
finally:
session.close("standalone capture ended")
try:
client_writer.close()
await client_writer.wait_closed()
except Exception:
pass
return
# ── Bridge / splitter mode ───────────────────────────────────────────
# Connect to primary upstream (required)
try:
up_reader, up_writer = await asyncio.open_connection(
self.upstream_host, self.upstream_port
)
log.info("Connected to primary %s:%s", self.upstream_host, self.upstream_port)
except Exception as exc:
log.error("Failed to connect to primary upstream: %s", exc)
session.close(f"primary connect failed: {exc}")
client_writer.close()
return
# Connect to mirror upstream (optional — failure is non-fatal)
mir_reader: Optional[asyncio.StreamReader] = None
mir_writer: Optional[asyncio.StreamWriter] = None
if self.mirror_host:
try:
mir_reader, mir_writer = await asyncio.open_connection(
self.mirror_host, self.mirror_port
)
log.info("Connected to mirror %s:%s", self.mirror_host, self.mirror_port)
except Exception as exc:
log.warning(
"Mirror connect failed — continuing without mirror: %s", exc
)
session._log(f"# WARNING: mirror connect failed: {exc}")
# Build relay tasks
#
# ┌──────────┐ device bytes ┌─────────────┐
# │ Device │ ─────────────► │ PRIMARY │ responses ──► device
# └──────────┘ └─────────────┘
# │
# │ device bytes (copy)
# ▼
# ┌─────────────┐
# │ MIRROR │ responses discarded (logged only)
# └─────────────┘
#
tasks = [
asyncio.create_task(
self._relay_device(client_reader, up_writer, mir_writer, session),
name="device→upstreams",
),
asyncio.create_task(
self._relay_simple(up_reader, client_writer, session, "server"),
name="primary→device",
),
]
if mir_reader is not None:
tasks.append(asyncio.create_task(
self._relay_drain(mir_reader, session),
name="mirror→drain",
))
try:
# Wait for the device-to-upstreams relay to exit first (device
# disconnected or primary dropped). Then cancel the rest.
done, pending = await asyncio.wait(
tasks,
return_when=asyncio.FIRST_COMPLETED,
)
for t in pending:
t.cancel()
try:
await t
except (asyncio.CancelledError, Exception):
pass
except Exception as exc:
log.warning("Bridge relay error: %s", exc)
finally:
session.close("relay ended")
for writer in filter(None, [client_writer, up_writer, mir_writer]):
try:
writer.close()
await writer.wait_closed()
except Exception:
pass
# ── Relay helpers ─────────────────────────────────────────────────────────
async def _relay_device(
self,
reader: asyncio.StreamReader,
primary_writer: asyncio.StreamWriter,
mirror_writer: Optional[asyncio.StreamWriter],
session: CaptureSession,
) -> None:
"""
Read bytes from the device, write to the primary server, and also
write a copy to the mirror server (if connected). Mirror write
failures are non-fatal — we log and continue.
"""
try:
while True:
data = await reader.read(4096)
if not data:
break
session.feed_client(data)
# Primary write — failure IS fatal (lose primary = lose prod)
primary_writer.write(data)
await primary_writer.drain()
# Mirror write — failure is non-fatal
if mirror_writer is not None:
try:
mirror_writer.write(data)
await mirror_writer.drain()
except Exception as exc:
log.warning("Mirror write failed (non-fatal): %s", exc)
session._log(f"# WARNING: mirror write failed: {exc}")
mirror_writer = None # stop trying
except (asyncio.IncompleteReadError, ConnectionResetError, BrokenPipeError):
pass
async def _relay_simple(
self,
reader: asyncio.StreamReader,
writer: asyncio.StreamWriter,
session: CaptureSession,
direction: str,
) -> None:
"""Standard single-pipe relay (primary→device or vice-versa)."""
try:
while True:
data = await reader.read(4096)
if not data:
break
if direction == "server":
session.feed_server(data)
else:
session.feed_client(data)
writer.write(data)
await writer.drain()
except (asyncio.IncompleteReadError, ConnectionResetError, BrokenPipeError):
pass
async def _relay_drain(
self,
reader: asyncio.StreamReader,
session: CaptureSession,
) -> None:
"""
Read mirror server responses, log them to session, do NOT forward to
device. The device only ever sees primary server responses.
"""
try:
while True:
data = await reader.read(4096)
if not data:
break
session.feed_mirror(data)
except (asyncio.IncompleteReadError, ConnectionResetError, BrokenPipeError):
pass
# ── Main ───────────────────────────────────────────────────────────────────────
async def main(args: argparse.Namespace) -> None:
capture_dir = Path(__file__).parent / "captures"
capture_dir.mkdir(parents=True, exist_ok=True)
upstream_host: Optional[str] = None
upstream_port: Optional[int] = None
mirror_host: Optional[str] = None
mirror_port: Optional[int] = None
if not args.standalone:
if not args.upstream:
print("ERROR: --upstream HOST:PORT is required unless --standalone is set.")
sys.exit(1)
parts = args.upstream.rsplit(":", 1)
if len(parts) != 2:
print("ERROR: --upstream must be HOST:PORT (e.g. 203.0.113.5:12345)")
sys.exit(1)
upstream_host = parts[0]
upstream_port = int(parts[1])
if args.mirror:
parts = args.mirror.rsplit(":", 1)
if len(parts) != 2:
print("ERROR: --mirror must be HOST:PORT (e.g. 192.168.1.50:12345)")
sys.exit(1)
mirror_host = parts[0]
mirror_port = int(parts[1])
handler = BridgeHandler(
capture_dir,
upstream_host, upstream_port,
mirror_host, mirror_port,
)
server = await asyncio.start_server(
handler.handle,
host="0.0.0.0",
port=args.port,
)
# ── Startup banner ────────────────────────────────────────────────────────
if args.standalone:
mode = "STANDALONE capture (no forwarding)"
elif mirror_host:
mode = f"SPLITTER primary={upstream_host}:{upstream_port} mirror={mirror_host}:{mirror_port}"
else:
mode = f"BRIDGE → {upstream_host}:{upstream_port}"
addrs = ", ".join(str(s.getsockname()) for s in server.sockets)
print(f"\n{'='*70}")
print(f" ACH bridge/splitter listening on {addrs}")
print(f" Mode: {mode}")
print(f" Captures: {capture_dir}/ach_<timestamp>/")
print(f"{'='*70}")
if upstream_host and not mirror_host:
print(f"\n DISCOVERY PHASE")
print(f" Point your TEST unit's ACEmanager call-home destination to:")
print(f" <this machine's LAN IP> : {args.port}")
print(f" All traffic will be forwarded to {upstream_host}:{upstream_port}")
elif mirror_host:
print(f"\n SPLITTER MODE — PRODUCTION SAFE")
print(f" Units connect as normal. Every byte is forwarded to:")
print(f" PRIMARY (authoritative): {upstream_host}:{upstream_port}")
print(f" MIRROR (your server): {mirror_host}:{mirror_port}")
print(f" Only PRIMARY responses reach the device.")
print(f" Mirror failures are logged and do not affect the device.")
else:
print(f"\n STANDALONE MODE — capture only, nothing forwarded")
print(f" Point a unit at <this machine's LAN IP> : {args.port}")
print(f"\n Waiting for inbound connections... (Ctrl-C to stop)\n")
async with server:
await server.serve_forever()
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(
description=(
"Transparent TCP bridge / splitter for Instantel MiniMate Plus "
"call-home (ACH) traffic."
),
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
p.add_argument(
"--upstream", "-u",
metavar="HOST:PORT",
help=(
"Primary upstream ACH server to forward to "
"(e.g. 203.0.113.5:12345). "
"Omit with --standalone for capture-only mode."
),
)
p.add_argument(
"--mirror", "-m",
metavar="HOST:PORT",
help=(
"Mirror / secondary server to receive a copy of all device bytes "
"(splitter mode). Mirror responses are logged but NOT forwarded "
"to the device. Mirror failures are non-fatal."
),
)
p.add_argument(
"--port", "-p",
type=int,
default=12345,
help="Local port to listen on (default: 12345).",
)
p.add_argument(
"--standalone", "-s",
action="store_true",
help="Capture-only mode: accept connection, do not forward anywhere.",
)
p.add_argument(
"--verbose", "-v",
action="store_true",
help="Enable debug logging.",
)
return p.parse_args()
if __name__ == "__main__":
args = parse_args()
logging.basicConfig(
level=logging.DEBUG if args.verbose else logging.INFO,
format="%(asctime)s %(levelname)-7s %(name)s %(message)s",
)
try:
asyncio.run(main(args))
except KeyboardInterrupt:
print("\nStopped.")
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env python3
"""
ach_mitm.py — TCP man-in-the-middle proxy for capturing Blastware ACH sessions.
The unit calls home to THIS proxy instead of directly to Blastware. The proxy
forwards every byte in both directions to the real Blastware ACH server and saves
the traffic to separate raw capture files that the Analyzer can load directly.
Setup
-----
1. Start Blastware's ACH server on the BW PC as normal (it listens on its port).
2. Run this proxy on any machine the unit can reach:
python bridges/ach_mitm.py --bw-host 192.168.1.50 --bw-port 9999
3. Point the unit's ACEmanager call-home destination to THIS machine's IP and
the --listen-port (default 9999).
4. Trigger a call-home (or wait for the unit to call in).
5. The proxy transparently forwards everything and saves two files per session:
ach_mitm_<ts>/raw_bw_<ts>.bin -- bytes Blastware sent to unit (BW TX)
ach_mitm_<ts>/raw_s3_<ts>.bin -- bytes unit sent to Blastware (S3 TX)
Both files load directly in the Analyzer (File > Open Capture).
The proxy exits cleanly when either side drops the connection.
Use case: capturing Blastware operations we haven't reverse-engineered yet,
e.g. event deletion, factory reset, firmware update.
"""
from __future__ import annotations
import argparse
import datetime
import logging
import socket
import sys
import threading
from pathlib import Path
log = logging.getLogger("ach_mitm")
def _pipe(src: socket.socket, dst: socket.socket, label: str, outfile) -> None:
"""Forward bytes from src to dst, writing everything to outfile."""
try:
while True:
data = src.recv(4096)
if not data:
break
dst.sendall(data)
outfile.write(data)
outfile.flush()
log.debug("%s %d bytes", label, len(data))
except OSError:
pass
finally:
log.info("%s pipe closed", label)
# Signal the other direction to stop by shutting down our end.
try:
dst.shutdown(socket.SHUT_WR)
except OSError:
pass
def handle(unit_sock: socket.socket, peer: str, bw_host: str, bw_port: int,
output_dir: Path) -> None:
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
session_dir = output_dir / f"ach_mitm_{ts}"
session_dir.mkdir(parents=True, exist_ok=True)
log.info("Session %s unit=%s forwarding to %s:%d", ts, peer, bw_host, bw_port)
# Connect upstream to Blastware.
bw_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
bw_sock.connect((bw_host, bw_port))
except OSError as exc:
log.error("Cannot reach Blastware at %s:%d: %s", bw_host, bw_port, exc)
unit_sock.close()
return
log.info("Connected to Blastware at %s:%d", bw_host, bw_port)
bw_path = session_dir / f"raw_bw_{ts}.bin" # Blastware → unit (BW TX)
s3_path = session_dir / f"raw_s3_{ts}.bin" # unit → Blastware (S3 TX)
with open(bw_path, "wb") as bw_fh, open(s3_path, "wb") as s3_fh:
# Two threads: one per direction.
t_bw = threading.Thread(
target=_pipe, args=(bw_sock, unit_sock, "BW->unit", bw_fh), daemon=True
)
t_s3 = threading.Thread(
target=_pipe, args=(unit_sock, bw_sock, "unit->BW", s3_fh), daemon=True
)
t_bw.start()
t_s3.start()
t_bw.join()
t_s3.join()
bw_bytes = bw_path.stat().st_size
s3_bytes = s3_path.stat().st_size
log.info(
"Session %s done BW->unit: %d bytes unit->BW: %d bytes -> %s",
ts, bw_bytes, s3_bytes, session_dir,
)
unit_sock.close()
bw_sock.close()
def serve(args: argparse.Namespace) -> None:
output_dir = Path(args.output)
output_dir.mkdir(parents=True, exist_ok=True)
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.bind(("0.0.0.0", args.listen_port))
server.listen(5)
server.settimeout(1.0)
print(f"\n{'='*60}")
print(f" ACH MITM proxy")
print(f" Listening on 0.0.0.0:{args.listen_port}")
print(f" Forwarding to {args.bw_host}:{args.bw_port}")
print(f" Captures in {output_dir.resolve()}/ach_mitm_<ts>/")
print(f"{'='*60}")
print(f"\n Point the unit's ACEmanager call-home to this machine on port {args.listen_port}")
print(f" Ctrl-C to stop\n")
try:
while True:
try:
client_sock, addr = server.accept()
except socket.timeout:
continue
peer = f"{addr[0]}:{addr[1]}"
log.info("Accepted connection from %s", peer)
t = threading.Thread(
target=handle,
args=(client_sock, peer, args.bw_host, args.bw_port, output_dir),
daemon=True,
)
t.start()
except KeyboardInterrupt:
print("\nStopping.")
finally:
server.close()
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--bw-host", required=True,
help="IP or hostname of the Blastware ACH server")
ap.add_argument("--bw-port", type=int, default=9999,
help="Port Blastware is listening on (default: 9999)")
ap.add_argument("--listen-port", type=int, default=9999,
help="Port this proxy listens on (default: 9999)")
ap.add_argument("--output", default="bridges/captures/mitm",
help="Directory for capture files")
ap.add_argument("--log-level", default="INFO",
choices=["DEBUG", "INFO", "WARNING", "ERROR"])
args = ap.parse_args()
logging.basicConfig(
level=getattr(logging, args.log_level),
format="%(asctime)s %(levelname)-7s %(name)s %(message)s",
stream=sys.stdout,
)
serve(args)
if __name__ == "__main__":
main()
+800
View File
@@ -0,0 +1,800 @@
#!/usr/bin/env python3
"""
ach_server.py — Minimal inbound ACH (Auto Call Home) server for MiniMate Plus.
This IS your test server. Run it on any machine on the same network, point a
unit's ACEmanager call-home destination at it, and it will speak the full BW
protocol to the device: handshake, pull device info, download all events, save
everything as JSON.
The key thing this script tells you that no amount of packet sniffing can:
- Does the device speak first (push) or wait for us to send POLL (pull)?
If startup() completes normally → it's pull protocol, same as Blastware.
If startup() times out → the device sent something first; check raw_rx.bin.
Usage
-----
python bridges/ach_server.py [--port 12345] [--output bridges/captures/]
Setup
-----
1. Run this script on a machine on your local network.
2. In ACEmanager → Application → ALEOS Application Framework (or equivalent)
find the Call Home / ACH settings. Set:
Remote Host: <this machine's LAN IP>
Remote Port: 12345
3. Trigger the unit (wait for a vibration event, or use the manual call-home
button if your firmware version has one).
4. The unit connects. This script handshakes, downloads all events,
and saves a timestamped session directory.
Output per session
------------------
bridges/captures/ach_inbound_<ts>/
device_info.json — serial number, firmware version, calibration date, etc.
events.json — all events: timestamp, PPV per channel, peaks, metadata
raw_rx_<ts>.bin — raw bytes from the device (S3 side) for Analyzer
raw_tx_<ts>.bin — raw bytes we sent to the device (BW side) for Analyzer
session_<ts>.log — detailed protocol log
What to look for
----------------
Push vs pull: Check session_<ts>.log. If the first line after "Connected"
shows bytes arriving BEFORE the POLL probe was sent, it's push. If POLL
gets a clean response, it's pull.
Frequency: Look at raw_rx.bin in the Analyzer. SUB 5A (0xA5 responses) carry
bulk waveform data — if frequency is sent pre-computed there will be float32
values before the ADC sample blocks.
ACH-specific framing: Does the unit send anything extra before the DLE+STX
framing starts? raw_rx.bin will show raw bytes including any preamble.
"""
from __future__ import annotations
import argparse
import datetime
import json
import logging
import socket
import sys
import threading
from pathlib import Path
from typing import Optional
sys.path.insert(0, str(Path(__file__).parent.parent))
from minimateplus.transport import SocketTransport
from minimateplus.client import MiniMateClient
from minimateplus.models import DeviceInfo, Event, MonitorLogEntry
from sfm.database import SeismoDb
log = logging.getLogger("ach_server")
# ── Per-unit state (downloaded-key set) ───────────────────────────────────────
# Persisted as <output_dir>/ach_state.json
# Format:
# {
# "BE11529": {
# "downloaded_keys": ["01110000", "0111245a"], # hex keys already on disk
# "max_downloaded_key": "0111245a", # highest key ever seen
# "last_seen": "2026-04-11T01:04:36"
# }
# }
#
# Key-based deduplication works well within a single "key generation" (between
# erases). After the device memory is erased the event counter resets to
# 0x01110000, so the first new event has the SAME key as the very first event
# we ever downloaded. We detect this situation with max_downloaded_key:
#
# if max(current_device_keys) < max_downloaded_key
# → device was wiped and keys have restarted → treat all device keys as new
#
# After our own erase (--clear-after-download) we also explicitly clear
# downloaded_keys and max_downloaded_key so the next session starts fresh.
_state_lock = threading.Lock()
def _load_state(state_path: Path) -> dict:
if state_path.exists():
try:
with open(state_path) as f:
return json.load(f)
except Exception:
pass
return {}
def _save_state(state_path: Path, state: dict) -> None:
with _state_lock:
with open(state_path, "w") as f:
json.dump(state, f, indent=2)
# ── Per-session handler ────────────────────────────────────────────────────────
class AchSession:
"""
Handles one inbound unit connection in its own thread.
Wraps the socket in a SocketTransport → MiniMateClient, then runs the
standard connect → get_device_info → get_events sequence.
State tracking (ach_state.json in output_dir):
On each successful download we record the SET of event keys downloaded.
On the next call-home we compare: if all device keys are already in the
set, there's nothing new. If any key is new (including after the device
was wiped and re-recorded), we download and save only those events.
"""
def __init__(
self,
sock: socket.socket,
peer: str,
output_dir: Path,
timeout: float,
events_only: bool,
max_events: Optional[int],
state_path: Path,
db: "SeismoDb",
clear_after_download: bool = False,
restart_monitoring: bool = False,
) -> None:
self.sock = sock
self.peer = peer
self.output_dir = output_dir
self.timeout = timeout
self.events_only = events_only
self.max_events = max_events
self.state_path = state_path
self.db = db
self.clear_after_download = clear_after_download
self.restart_monitoring = restart_monitoring
def run(self) -> None:
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
# Session dir and file handler are created lazily — only after startup
# succeeds. This prevents internet scanners and dropped connections from
# littering the output directory with empty session folders.
try:
self._run_inner(ts)
except Exception as exc:
log.error("Session failed (%s): %s", self.peer, exc, exc_info=True)
finally:
try:
self.sock.close()
except Exception:
pass
def _run_inner(self, ts: str) -> None:
transport = SocketTransport(self.sock, peer=self.peer)
# Collect raw bytes in memory until startup succeeds, then flush to disk.
raw_rx_buf: list[bytes] = [] # device → us (S3 side)
raw_tx_buf: list[bytes] = [] # us → device (BW side)
_orig_read = transport.read
_orig_write = transport.write
def tapped_read(n: int) -> bytes:
data = _orig_read(n)
if data:
raw_rx_buf.append(data)
return data
def tapped_write(data: bytes) -> None:
_orig_write(data)
if data:
raw_tx_buf.append(data)
transport.read = tapped_read # type: ignore[method-assign]
transport.write = tapped_write # type: ignore[method-assign]
serial: Optional[str] = None
# ── Step 1: startup handshake ─────────────────────────────────────────
# Do this BEFORE creating the session directory so that scanner probes
# and dropped connections leave no trace on disk.
try:
from minimateplus.protocol import MiniMateProtocol
client = MiniMateClient(transport=transport, timeout=self.timeout)
client.open()
proto = MiniMateProtocol(transport, recv_timeout=self.timeout)
proto.startup()
except Exception as exc:
log.warning("Startup failed from %s: %s -- ignoring", self.peer, exc)
return # no session dir created
# Startup succeeded — this is a real unit. Create session dir now.
session_dir = self.output_dir / f"ach_inbound_{ts}"
session_dir.mkdir(parents=True, exist_ok=True)
log_path = session_dir / f"session_{ts}.log"
raw_rx_path = session_dir / f"raw_rx_{ts}.bin" # device → us (S3 side)
raw_tx_path = session_dir / f"raw_tx_{ts}.bin" # us → device (BW side)
# Flush buffered bytes to files and switch to direct file writes.
raw_rx_fh = open(raw_rx_path, "wb")
raw_tx_fh = open(raw_tx_path, "wb")
for chunk in raw_rx_buf:
raw_rx_fh.write(chunk)
for chunk in raw_tx_buf:
raw_tx_fh.write(chunk)
raw_rx_buf.clear()
raw_tx_buf.clear()
def tapped_read_file(n: int) -> bytes:
data = _orig_read(n)
if data:
raw_rx_fh.write(data)
raw_rx_fh.flush()
return data
def tapped_write_file(data: bytes) -> None:
_orig_write(data)
if data:
raw_tx_fh.write(data)
raw_tx_fh.flush()
transport.read = tapped_read_file # type: ignore[method-assign]
transport.write = tapped_write_file # type: ignore[method-assign]
# Wire up file handler now that the session dir exists.
fh = logging.FileHandler(log_path, encoding="utf-8")
fh.setFormatter(logging.Formatter("%(asctime)s %(levelname)-7s %(name)s %(message)s"))
root_logger = logging.getLogger()
root_logger.addHandler(fh)
try:
# ── Step 2: device info ───────────────────────────────────────────
device_info = None
if not self.events_only:
log.info("Step 2/3: reading device info")
try:
device_info = client.connect()
serial = device_info.serial
_save_json(session_dir / "device_info.json", _device_info_to_dict(device_info))
log.info(
" [OK] Device: serial=%s firmware=%s model=%s events=%d",
serial,
device_info.firmware_version,
device_info.model,
device_info.event_count or 0,
)
except Exception as exc:
log.error(" [FAIL] Device info failed: %s", exc)
else:
log.info("Step 2/3: skipping device info (--events-only)")
# ── Step 3: check for new events by comparing key sets ────────────
log.info("Step 3/3: checking for new events")
state = _load_state(self.state_path)
unit_key = serial or self.peer # fall back to IP if no serial
unit_state = state.get(unit_key, {})
seen_keys: set[str] = set(unit_state.get("downloaded_keys", []))
# Highest event key ever downloaded from this unit (hex string, 8 chars).
# Used to detect post-erase key reuse — see comment block above.
max_seen_key: str = unit_state.get("max_downloaded_key", "00000000")
# Walk the event index (browse-mode, no 5A) to get the actual current
# key list. The SUB 08 event_count field is a lifetime "total events
# ever recorded" counter that does NOT decrement on erase — confirmed
# 2026-04-13. list_event_keys() via the 1E/1F chain is the only
# reliable way to know what is actually stored on the device right now.
log.info(" Checking device key list (browse walk, no waveform download)...")
try:
device_keys = client.list_event_keys()
except Exception as exc:
log.warning(" list_event_keys failed: %s -- falling back to full download", exc)
device_keys = None
# Use the walk result as our authoritative current count.
current_count = len(device_keys) if device_keys is not None else 0
log.info(" Unit has %d stored event(s); %d key(s) previously downloaded",
current_count, len(seen_keys))
if device_keys is not None and current_count == 0:
log.info(" [OK] No events on device -- nothing to download")
log.info("Session complete (no events) -> %s", session_dir)
return
if device_keys is not None:
# ── Post-erase detection ──────────────────────────────────────
# After the device memory is erased, new events start from key
# 01110000 again — the same keys we already downloaded. Detect
# this by comparing the device's current highest key against the
# historical maximum. If the device has rolled back below our
# high-water mark, its counter was reset and we must treat all
# its keys as new, regardless of what seen_keys contains.
if device_keys and max_seen_key != "00000000":
max_device_key = max(device_keys) # lexicographic; safe because
# keys share the same 4-char prefix
if max_device_key < max_seen_key:
log.info(
" Post-erase reset detected: "
"device max key %s < historical max %s "
"-- treating all device keys as new",
max_device_key, max_seen_key,
)
seen_keys = set() # discard stale dedup info for this session
new_key_set = set(device_keys) - seen_keys
log.info(" Device has %d key(s): %d new, %d already seen",
len(device_keys), len(new_key_set), len(device_keys) - len(new_key_set))
if not new_key_set:
log.info(" [OK] All events already downloaded -- nothing to do")
# Refresh state timestamp; preserve max_seen_key unchanged.
state[unit_key] = {
"downloaded_keys": sorted(seen_keys | set(device_keys)),
"max_downloaded_key": max_seen_key,
"last_seen": datetime.datetime.now().isoformat(),
"serial": serial,
"peer": self.peer,
}
_save_state(self.state_path, state)
# ── Erase even when no new events (if requested) ──────────
# Blastware ACH always erases after every session — even when
# nothing new was downloaded. Without the erase the device
# still sees stored events in its memory and immediately
# retries the call-home, causing the looping we observed.
# Only erase when device actually has events stored; skip
# the erase if device_keys is empty (nothing to erase).
if self.clear_after_download and device_keys:
log.info(
" Clearing device memory (--clear-after-download, "
"no new events but device has %d stored)...",
len(device_keys),
)
try:
client.delete_all_events()
log.info(" [OK] Device memory cleared")
# Reset state so the next session starts fresh.
state[unit_key] = {
"downloaded_keys": [],
"max_downloaded_key": "00000000",
"last_seen": datetime.datetime.now().isoformat(),
"serial": serial,
"peer": self.peer,
}
_save_state(self.state_path, state)
except Exception as exc:
log.error(
" [WARN] Event deletion failed: %s -- events NOT cleared",
exc,
)
log.info("Session complete (no new events) -> %s", session_dir)
return
else:
new_key_set = None # unknown; proceed with full download
# Apply max_events cap
# stop_idx: when we know the count from list_event_keys, use it as
# an upper bound. When list_event_keys failed (device_keys is None),
# pass None — get_events will run until the null sentinel naturally.
stop_idx: Optional[int] = (current_count - 1) if device_keys is not None else None
if self.max_events is not None:
cap = self.max_events - 1
stop_idx = cap if stop_idx is None else min(stop_idx, cap)
if device_keys is not None and self.max_events < current_count:
log.warning(
" max_events=%d cap: will download events 0-%d only "
"(unit has %d total)",
self.max_events, stop_idx, current_count,
)
try:
all_events = client.get_events(
full_waveform=True,
stop_after_index=stop_idx,
skip_waveform_for_keys=seen_keys if seen_keys else None,
)
# Filter to events whose keys we haven't saved before.
new_events = [
e for e in all_events
if e._waveform_key is None
or e._waveform_key.hex() not in seen_keys
]
skipped = len(all_events) - len(new_events)
log.info(" [OK] Downloaded %d event(s): %d new, %d skipped (already seen)",
len(all_events), len(new_events), skipped)
if skipped:
log.info(" (skipped %d already-downloaded event(s))", skipped)
if new_events:
_save_json(session_dir / "events.json", [_event_to_dict(e) for e in new_events])
for ev in new_events:
pv = ev.peak_values
pi = ev.project_info
key_hex = ev._waveform_key.hex() if ev._waveform_key else "????????"
log.info(
" NEW [%s] %s Tran=%.4f Vert=%.4f Long=%.4f VS=%.4f project=%r",
key_hex,
str(ev.timestamp) if ev.timestamp else "?",
pv.tran if pv else 0,
pv.vert if pv else 0,
pv.long if pv else 0,
pv.peak_vector_sum if pv else 0,
pi.project if pi else "",
)
else:
log.info(" [OK] No new events since last call-home -- nothing to save")
# ── Monitor log entries (partial records / continuous monitoring) ──
# Browse walk (0A + 1F only) to collect monitor log entries for
# recording intervals where no threshold was crossed. This is a
# second 1E-based pass over the device's record list, separate from
# the get_events() download loop above.
log.info(" Collecting monitor log entries (browse walk)...")
new_monitor_entries: list[MonitorLogEntry] = []
try:
new_monitor_entries = client.get_monitor_log_entries(
skip_keys=seen_keys if seen_keys else None,
)
if new_monitor_entries:
_save_json(
session_dir / "monitor_log.json",
[_monitor_log_entry_to_dict(e) for e in new_monitor_entries],
)
log.info(
" [OK] %d new monitor log entry(s) saved",
len(new_monitor_entries),
)
for ml in new_monitor_entries:
log.info(
" MONLOG [%s] %s%s (%s)",
ml.key,
ml.start_time.isoformat() if ml.start_time else "?",
ml.stop_time.isoformat() if ml.stop_time else "?",
f"{ml.duration_seconds:.0f}s" if ml.duration_seconds is not None else "?s",
)
else:
log.info(" [OK] No new monitor log entries")
except Exception as exc:
log.warning(
" [WARN] Monitor log collection failed: %s -- continuing",
exc,
)
# ── Persist to SQLite DB ─────────────────────────────────────
_session_start = datetime.datetime.now()
try:
_ev_ins, _ev_skip = self.db.insert_events(
new_events, serial=serial or self.peer, session_id=None
)
_ml_ins, _ml_skip = self.db.insert_monitor_log(
new_monitor_entries, session_id=None
)
_session_id = self.db.insert_ach_session(
serial=serial or self.peer,
peer=self.peer,
events_downloaded=_ev_ins,
monitor_entries=_ml_ins,
duration_seconds=(datetime.datetime.now() - _session_start).total_seconds(),
session_time=_session_start,
)
log.info(
" [DB] session=%s events +%d (skip %d) monitor +%d (skip %d)",
_session_id[:8], _ev_ins, _ev_skip, _ml_ins, _ml_skip,
)
except Exception as exc:
log.warning(" [WARN] DB write failed: %s -- continuing", exc)
# ── Optional: erase device memory after successful download ────
erased_successfully = False
if self.clear_after_download and new_events:
log.info(" Clearing device memory (--clear-after-download)...")
try:
client.delete_all_events()
log.info(" [OK] Device memory cleared")
erased_successfully = True
except Exception as exc:
log.error(
" [WARN] Event deletion failed: %s -- events NOT cleared",
exc,
)
# ── Update persistent state ───────────────────────────────────
# Include both triggered-event keys and monitor-log keys in the
# downloaded set so they are not re-processed on the next call-home.
current_event_keys = [
e._waveform_key.hex()
for e in all_events
if e._waveform_key is not None
]
current_monitor_keys = [e.key for e in new_monitor_entries]
current_keys = current_event_keys + current_monitor_keys
if erased_successfully:
# Device memory is clear. Reset downloaded_keys and the
# high-water mark so the next call-home starts fresh and
# doesn't mis-identify the recycled key 01110000 as "seen".
updated_keys = []
new_max_key = "00000000"
log.info(
" State reset after erase -- next session will download "
"from key 0 (device counter resets after erase)"
)
else:
# Normal (no erase): union of previously-seen + all keys on
# device now. Includes already-seen survivors so we never
# re-download them if the device somehow keeps old records.
updated_keys = sorted(set(seen_keys) | set(current_keys))
new_max_key = updated_keys[-1] if updated_keys else max_seen_key
state[unit_key] = {
"downloaded_keys": updated_keys,
"max_downloaded_key": new_max_key,
"last_seen": datetime.datetime.now().isoformat(),
"serial": serial,
"peer": self.peer,
}
_save_state(self.state_path, state)
except Exception as exc:
log.error(" [FAIL] Event download failed: %s", exc, exc_info=True)
# ── Optional: restart monitoring after successful download ─────────
if self.restart_monitoring:
log.info(" Restarting monitoring on device (--restart-monitoring)...")
try:
client.start_monitoring()
log.info(" [OK] Monitoring restarted")
except Exception as exc:
log.warning(" [WARN] Failed to restart monitoring: %s", exc)
finally:
raw_rx_fh.close()
raw_tx_fh.close()
client.close() # closes transport / socket cleanly
root_logger.removeHandler(fh)
fh.close()
log.info("Session complete -> %s", session_dir)
log.info("="*60)
# ── JSON helpers ───────────────────────────────────────────────────────────────
def _save_json(path: Path, obj: object) -> None:
with open(path, "w") as f:
json.dump(obj, f, indent=2, default=str)
log.debug("Saved %s", path)
def _device_info_to_dict(d: DeviceInfo) -> dict:
cc = d.compliance_config
return {
"serial": d.serial,
"firmware_version": d.firmware_version,
"dsp_version": d.dsp_version,
"model": d.model,
"event_count": d.event_count,
# compliance config fields (None if 1A read failed)
"setup_name": cc.setup_name if cc else None,
"sample_rate": cc.sample_rate if cc else None,
"record_time": cc.record_time if cc else None,
"trigger_level_geo": cc.trigger_level_geo if cc else None,
"alarm_level_geo": cc.alarm_level_geo if cc else None,
"geo_adc_scale": cc.geo_adc_scale if cc else None, # hw scale factor (in/s)/V
"geo_range": cc.geo_range if cc else None, # 0x01=Normal 10in/s, 0x00=Sensitive 1.25in/s (unconfirmed)
"project": cc.project if cc else None,
"client": cc.client if cc else None,
"operator": cc.operator if cc else None,
"sensor_location": cc.sensor_location if cc else None,
}
def _event_to_dict(e: Event) -> dict:
pv = e.peak_values
pi = e.project_info
peaks = {}
if pv:
peaks = {
"transverse": pv.tran,
"vertical": pv.vert,
"longitudinal": pv.long,
"vector_sum": pv.peak_vector_sum,
"mic": pv.micl,
}
samples = {}
if e.raw_samples:
samples = {
ch: vals[:20] # first 20 sample-sets to keep the file sane
for ch, vals in e.raw_samples.items()
}
samples["__note__"] = "first 20 sample-sets only; see raw_rx.bin for full waveform"
return {
"timestamp": str(e.timestamp) if e.timestamp else None,
"project": pi.project if pi else None,
"client": pi.client if pi else None,
"operator": pi.operator if pi else None,
"sensor_location": pi.sensor_location if pi else None,
"peaks": peaks,
"raw_samples_preview": samples,
}
def _monitor_log_entry_to_dict(e: MonitorLogEntry) -> dict:
return {
"key": e.key,
"start_time": e.start_time.isoformat() if e.start_time else None,
"stop_time": e.stop_time.isoformat() if e.stop_time else None,
"duration_seconds": e.duration_seconds,
"serial": e.serial,
"geo_threshold_ips": e.geo_threshold_ips,
}
# ── Main server loop ───────────────────────────────────────────────────────────
def serve(args: argparse.Namespace) -> None:
output_dir = Path(args.output)
output_dir.mkdir(parents=True, exist_ok=True)
state_path = output_dir / "ach_state.json"
db = SeismoDb(output_dir / "seismo_relay.db")
server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_sock.bind(("0.0.0.0", args.port))
server_sock.listen(5)
# Wake up every second so Ctrl-C is handled promptly on Windows.
# Without this, accept() blocks indefinitely and ignores KeyboardInterrupt.
server_sock.settimeout(1.0)
max_ev = args.max_events
print(f"\n{'='*60}")
print(f" ACH inbound server listening on 0.0.0.0:{args.port}")
print(f" Output: {output_dir.resolve()}/ach_inbound_<timestamp>/")
print(f" State file: {state_path}")
print(f" Max events per session: {max_ev if max_ev else 'unlimited'}")
print(f" Clear device after download: {'YES' if args.clear_after_download else 'no'}")
print(f" Restart monitoring after download: {'YES' if args.restart_monitoring else 'no'}")
print(f"{'='*60}")
print(f"\n Point your test unit's ACEmanager call-home settings to:")
print(f" Remote Host: <this machine's LAN IP>")
print(f" Remote Port: {args.port}")
print(f"\n Waiting for inbound connections... (Ctrl-C to stop)\n")
allow_ips = set(args.allow_ips)
if allow_ips:
print(f" Allowlist: {', '.join(sorted(allow_ips))}")
else:
print(" Allowlist: NONE -- accepting all IPs (add --allow-ip to restrict)")
try:
while True:
try:
client_sock, addr = server_sock.accept()
except socket.timeout:
continue # no connection this second; loop back and check for Ctrl-C
try:
peer_ip = addr[0]
peer = f"{addr[0]}:{addr[1]}"
if allow_ips and peer_ip not in allow_ips:
log.info("Rejected connection from %s (not in allowlist)", peer)
client_sock.close()
continue
log.info("Accepted connection from %s", peer)
session = AchSession(
sock=client_sock,
peer=peer,
output_dir=output_dir,
timeout=args.timeout,
events_only=args.events_only,
max_events=max_ev,
state_path=state_path,
db=db,
clear_after_download=args.clear_after_download,
restart_monitoring=args.restart_monitoring,
)
t = threading.Thread(target=session.run, daemon=True, name=f"ach-{peer}")
t.start()
except KeyboardInterrupt:
raise
except Exception as exc:
log.error("Accept error: %s", exc)
finally:
server_sock.close()
print("\nServer stopped.")
def parse_args() -> argparse.Namespace:
p = argparse.ArgumentParser(
description="Minimal inbound ACH server — speak BW protocol to calling MiniMate Plus units.",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
p.add_argument(
"--port", "-p",
type=int,
default=12345,
help="Port to listen on (default: 12345).",
)
p.add_argument(
"--output", "-o",
default=str(Path(__file__).parent / "captures"),
metavar="DIR",
help="Directory to write session captures (default: bridges/captures/).",
)
p.add_argument(
"--timeout", "-t",
type=float,
default=30.0,
help="Protocol receive timeout in seconds (default: 30.0).",
)
p.add_argument(
"--events-only",
action="store_true",
help="Skip the device-info step and go straight to event download.",
)
p.add_argument(
"--max-events",
type=int,
default=None,
metavar="N",
help=(
"Safety cap: download at most N events per session (default: unlimited). "
"Useful if a unit has many old events stored — prevents a very long first run."
),
)
p.add_argument(
"--allow-ip",
metavar="IP",
action="append",
dest="allow_ips",
default=[],
help=(
"Only accept connections from this IP address (repeat for multiple). "
"Example: --allow-ip 63.43.212.232 "
"If not specified, all IPs are accepted (not recommended for public servers)."
),
)
p.add_argument(
"--restart-monitoring",
action="store_true",
default=False,
help=(
"After downloading events, send SUB 0x96 (start monitoring) before "
"disconnecting. Required for RV55 units whose firmware does not assert "
"DCD on disconnect — without this the unit stays idle after a call-home."
),
)
p.add_argument(
"--clear-after-download",
action="store_true",
default=False,
help=(
"After successfully downloading new events, erase all events from the "
"device memory (SUB 0xA3 → 0x1C → 0x06 → 0xA2 sequence, confirmed from "
"4-11-26 MITM capture). Only fires when at least one new event was saved. "
"This mirrors the standard Blastware ACH workflow."
),
)
p.add_argument(
"--verbose", "-v",
action="store_true",
help="Enable debug logging.",
)
return p.parse_args()
if __name__ == "__main__":
args = parse_args()
logging.basicConfig(
level=logging.DEBUG if args.verbose else logging.INFO,
format="%(asctime)s %(levelname)-7s %(name)s %(message)s",
)
try:
serve(args)
except KeyboardInterrupt:
print("\nStopped.")
+34 -29
View File
@@ -58,16 +58,24 @@ class BridgeGUI(tk.Tk):
tk.Entry(self, textvariable=self.logdir_var, width=24).grid(row=1, column=3, sticky="we", **pad)
tk.Button(self, text="Browse", command=self._choose_dir).grid(row=1, column=4, sticky="w", **pad)
# Row 2: Raw taps
self.raw_bw_var = tk.StringVar(value="")
self.raw_s3_var = tk.StringVar(value="")
tk.Checkbutton(self, text="Save BW->S3 raw", command=self._toggle_raw_bw, onvalue="1", offvalue="").grid(row=2, column=0, sticky="w", **pad)
tk.Entry(self, textvariable=self.raw_bw_var, width=28).grid(row=2, column=1, columnspan=3, sticky="we", **pad)
tk.Button(self, text="...", command=lambda: self._choose_file(self.raw_bw_var, "bw")).grid(row=2, column=4, **pad)
# Row 2: Raw taps — ON by default; "auto" = timestamped name; blank checkbox = disabled
self.raw_bw_enabled = tk.IntVar(value=1)
self.raw_s3_enabled = tk.IntVar(value=1)
# Path fields: empty means "auto" (bridge picks a timestamped name)
self.raw_bw_path_var = tk.StringVar(value="")
self.raw_s3_path_var = tk.StringVar(value="")
tk.Checkbutton(self, text="Save S3->BW raw", command=self._toggle_raw_s3, onvalue="1", offvalue="").grid(row=3, column=0, sticky="w", **pad)
tk.Entry(self, textvariable=self.raw_s3_var, width=28).grid(row=3, column=1, columnspan=3, sticky="we", **pad)
tk.Button(self, text="...", command=lambda: self._choose_file(self.raw_s3_var, "s3")).grid(row=3, column=4, **pad)
tk.Checkbutton(self, text="BW→S3 raw (auto)", variable=self.raw_bw_enabled,
command=self._toggle_raw_bw).grid(row=2, column=0, sticky="w", **pad)
tk.Entry(self, textvariable=self.raw_bw_path_var, width=28,
fg="grey").grid(row=2, column=1, columnspan=3, sticky="we", **pad)
tk.Button(self, text="...", command=lambda: self._choose_file(self.raw_bw_path_var, "bw")).grid(row=2, column=4, **pad)
tk.Checkbutton(self, text="S3→BW raw (auto)", variable=self.raw_s3_enabled,
command=self._toggle_raw_s3).grid(row=3, column=0, sticky="w", **pad)
tk.Entry(self, textvariable=self.raw_s3_path_var, width=28,
fg="grey").grid(row=3, column=1, columnspan=3, sticky="we", **pad)
tk.Button(self, text="...", command=lambda: self._choose_file(self.raw_s3_path_var, "s3")).grid(row=3, column=4, **pad)
# Row 4: Status + buttons
self.status_var = tk.StringVar(value="Idle")
@@ -102,13 +110,11 @@ class BridgeGUI(tk.Tk):
var.set(filename)
def _toggle_raw_bw(self) -> None:
if not self.raw_bw_var.get():
# default name
self.raw_bw_var.set(os.path.join(self.logdir_var.get(), "raw_bw.bin"))
# Checkbox toggled — no path action needed; enabled state drives the flag.
pass
def _toggle_raw_s3(self) -> None:
if not self.raw_s3_var.get():
self.raw_s3_var.set(os.path.join(self.logdir_var.get(), "raw_s3.bin"))
pass
def start_bridge(self) -> None:
if self.process and self.process.poll() is None:
@@ -126,23 +132,22 @@ class BridgeGUI(tk.Tk):
args = [sys.executable, BRIDGE_PATH, "--bw", bw, "--s3", s3, "--baud", baud, "--logdir", logdir]
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
# Raw tap flags.
# Checkbox on + empty path → pass "auto" (bridge generates timestamped name).
# Checkbox on + explicit path → pass that path.
# Checkbox off → pass "" to disable (overrides bridge's auto default).
raw_bw_explicit = self.raw_bw_path_var.get().strip()
raw_s3_explicit = self.raw_s3_path_var.get().strip()
raw_bw = self.raw_bw_var.get().strip()
raw_s3 = self.raw_s3_var.get().strip()
if self.raw_bw_enabled.get():
args += ["--raw-bw", raw_bw_explicit if raw_bw_explicit else "auto"]
else:
args += ["--raw-bw", ""] # explicit disable
# If the user left the default generic name, replace with a timestamped one
# so each session gets its own file.
if raw_bw:
if os.path.basename(raw_bw) in ("raw_bw.bin", "raw_bw"):
raw_bw = os.path.join(os.path.dirname(raw_bw) or logdir, f"raw_bw_{ts}.bin")
self.raw_bw_var.set(raw_bw)
args += ["--raw-bw", raw_bw]
if raw_s3:
if os.path.basename(raw_s3) in ("raw_s3.bin", "raw_s3"):
raw_s3 = os.path.join(os.path.dirname(raw_s3) or logdir, f"raw_s3_{ts}.bin")
self.raw_s3_var.set(raw_s3)
args += ["--raw-s3", raw_s3]
if self.raw_s3_enabled.get():
args += ["--raw-s3", raw_s3_explicit if raw_s3_explicit else "auto"]
else:
args += ["--raw-s3", ""] # explicit disable
try:
self.process = subprocess.Popen(
+88 -13
View File
@@ -93,8 +93,11 @@ class SessionLogger:
self._bin_fh = open(bin_path, "ab", buffering=0)
self._lock = threading.Lock()
# Optional pure-byte taps (no headers). BW=Blastware tx, S3=device tx.
# These can be opened/closed on demand via start_raw_capture/stop_raw_capture.
self._raw_bw = open(raw_bw_path, "ab", buffering=0) if raw_bw_path else None
self._raw_s3 = open(raw_s3_path, "ab", buffering=0) if raw_s3_path else None
self._cap_bw_path: Optional[str] = raw_bw_path
self._cap_s3_path: Optional[str] = raw_s3_path
def log_line(self, line: str) -> None:
with self._lock:
@@ -124,6 +127,43 @@ class SessionLogger:
self.log_line(f"[{ts}] [INFO] {msg}")
self.bin_write_record(REC_INFO, msg.encode("utf-8", errors="replace"))
def start_raw_capture(self, label: str, logdir: str) -> tuple:
"""Open new raw tap files for a named capture. Returns (bw_path, s3_path)."""
ts = _dt.datetime.now().strftime("%Y%m%d_%H%M%S")
safe = "".join(c if c.isalnum() or c in "-_" else "_" for c in label)[:40] if label else ""
suffix = f"_{safe}" if safe else ""
bw_path = os.path.join(logdir, f"raw_bw_{ts}{suffix}.bin")
s3_path = os.path.join(logdir, f"raw_s3_{ts}{suffix}.bin")
with self._lock:
# Close any previously open taps first
if self._raw_bw:
self._raw_bw.close()
if self._raw_s3:
self._raw_s3.close()
self._raw_bw = open(bw_path, "ab", buffering=0)
self._raw_s3 = open(s3_path, "ab", buffering=0)
self._cap_bw_path = bw_path
self._cap_s3_path = s3_path
self.log_info(f"raw capture started: label={label!r} bw={bw_path} s3={s3_path}")
return bw_path, s3_path
def stop_raw_capture(self) -> tuple:
"""Close raw tap files. Returns (bw_path, s3_path) for the capture just closed."""
with self._lock:
bw = self._cap_bw_path
s3 = self._cap_s3_path
if self._raw_bw:
self._raw_bw.close()
self._raw_bw = None
if self._raw_s3:
self._raw_s3.close()
self._raw_s3 = None
self._cap_bw_path = None
self._cap_s3_path = None
if bw:
self.log_info(f"raw capture stopped: bw={bw} s3={s3}")
return bw, s3
def close(self) -> None:
with self._lock:
try:
@@ -291,8 +331,18 @@ def forward_loop(
time.sleep(0.002)
def annotation_loop(logger: SessionLogger, stop: threading.Event) -> None:
print("[MARK] Type 'm' + Enter to annotate the capture. Ctrl+C to stop.")
def annotation_loop(logger: SessionLogger, logdir: str, stop: threading.Event) -> None:
"""
Reads stdin commands while the bridge runs.
Commands:
m — prompt for a mark label (interactive)
CAP_START:<label> — begin a raw tap capture with the given label
CAP_STOP — stop the current raw tap capture
Responses (printed to stdout, parsed by the GUI):
[CAP_START] <bw_path>\\t<s3_path>
[CAP_STOP] <bw_path>\\t<s3_path>
"""
while not stop.is_set():
try:
line = input()
@@ -303,7 +353,21 @@ def annotation_loop(logger: SessionLogger, stop: threading.Event) -> None:
if not line:
continue
if line.lower() == "m":
if line.startswith("CAP_START:"):
label = line[10:].strip()
bw_path, s3_path = logger.start_raw_capture(label, logdir)
print(f"[CAP_START] {bw_path}\t{s3_path}")
sys.stdout.flush()
elif line == "CAP_STOP":
bw_path, s3_path = logger.stop_raw_capture()
if bw_path:
print(f"[CAP_STOP] {bw_path}\t{s3_path}")
else:
print("[CAP_STOP] no active capture")
sys.stdout.flush()
elif line.lower() == "m":
try:
sys.stdout.write(" Label: ")
sys.stdout.flush()
@@ -315,8 +379,9 @@ def annotation_loop(logger: SessionLogger, stop: threading.Event) -> None:
print(f" [MARK written] {label}")
else:
print(" (empty label — mark cancelled)")
else:
print(" (type 'm' + Enter to annotate)")
print(f" (unknown command: {line!r})")
def main() -> int:
@@ -325,8 +390,14 @@ def main() -> int:
ap.add_argument("--s3", default="COM5", help="S3-side COM port (default: COM5)")
ap.add_argument("--baud", type=int, default=38400, help="Baud rate (default: 38400)")
ap.add_argument("--logdir", default=".", help="Directory to write session logs into (default: .)")
ap.add_argument("--raw-bw", default=None, help="Optional file to append raw bytes sent from BW->S3 (no headers)")
ap.add_argument("--raw-s3", default=None, help="Optional file to append raw bytes sent from S3->BW (no headers)")
ap.add_argument("--raw-bw", default="auto",
help="File to append raw bytes sent from BW->S3 (no headers). "
"Default 'auto' generates a timestamped name in --logdir. "
"Pass an empty string to disable.")
ap.add_argument("--raw-s3", default="auto",
help="File to append raw bytes sent from S3->BW (no headers). "
"Default 'auto' generates a timestamped name in --logdir. "
"Pass an empty string to disable.")
ap.add_argument("--quiet", action="store_true", help="No console heartbeat output")
ap.add_argument("--status-every", type=float, default=0.0, help="Seconds between console heartbeat lines (default: 0 = off)")
args = ap.parse_args()
@@ -349,12 +420,16 @@ def main() -> int:
# If raw tap flags were passed without a path (bare --raw-bw / --raw-s3),
# or if the sentinel value "auto" is used, generate a timestamped name.
# If a specific path was provided, use it as-is (caller's responsibility).
raw_bw_path = args.raw_bw
raw_s3_path = args.raw_s3
if raw_bw_path in (None, "", "auto"):
raw_bw_path = os.path.join(args.logdir, f"raw_bw_{ts}.bin") if args.raw_bw is not None else None
if raw_s3_path in (None, "", "auto"):
raw_s3_path = os.path.join(args.logdir, f"raw_s3_{ts}.bin") if args.raw_s3 is not None else None
# Resolve raw tap paths.
# "auto" (default) → timestamped file in logdir (always captured).
# Explicit path → use verbatim.
# None or "" → disabled (pass --raw-bw "" to suppress capture).
raw_bw_path: Optional[str] = args.raw_bw if args.raw_bw else None
raw_s3_path: Optional[str] = args.raw_s3 if args.raw_s3 else None
if raw_bw_path == "auto":
raw_bw_path = os.path.join(args.logdir, f"raw_bw_{ts}.bin")
if raw_s3_path == "auto":
raw_s3_path = os.path.join(args.logdir, f"raw_s3_{ts}.bin")
logger = SessionLogger(log_path, bin_path, raw_bw_path=raw_bw_path, raw_s3_path=raw_s3_path)
@@ -391,7 +466,7 @@ def main() -> int:
t_ann = threading.Thread(
target=annotation_loop,
name="Annotator",
args=(logger, stop),
args=(logger, args.logdir, stop),
daemon=True,
)
+435
View File
@@ -0,0 +1,435 @@
#!/usr/bin/env python3
"""
serial_watch.py — Instantel Series-3 serial monitor with S3 frame parsing.
Taps the RS-232 line between the MiniMate Plus and its modem (RV50/RV55).
Saves raw binary captures compatible with the rest of the analysis toolchain,
plus a human-readable frame log.
Usage
-----
python bridges/serial_watch.py # interactive COM picker
python bridges/serial_watch.py --port COM3 # specify port
python bridges/serial_watch.py --port COM3 --ack-ok # reply OK to AT commands
# (useful if modem is absent
# and you want the device to
# proceed past AT negotiation)
python bridges/serial_watch.py --list # list available ports
Output
------
bridges/captures/serial_<ISO-timestamp>/
raw_s3_<ts>.bin — raw bytes from device (feeds directly into S3FrameParser)
session_<ts>.log — human-readable frame + control-line log
session_<ts>.jsonl — JSON-lines frame log
The raw_s3_*.bin file is byte-for-byte compatible with the existing capture
format used by bridges/parse_capture.py and all analysis scripts.
What to look for in a call-home capture
----------------------------------------
1. Does the device talk first after CONNECT, or does it wait?
- If raw_s3_*.bin has bytes before any AT/POLL exchange → PUSH protocol
- If it stays silent → PULL protocol (same as Blastware manual download)
2. Look for "Operating System" ASCII at the start — the device sends this 16-byte
boot string on cold start before entering DLE-framed mode.
3. RING/CONNECT from the modem appear as ASCII before the DLE frames — the parser
handles these automatically (scans forward to DLE+STX).
"""
from __future__ import annotations
import argparse
import sys
import threading
import time
from datetime import datetime
from pathlib import Path
try:
import serial
from serial.tools import list_ports
except ModuleNotFoundError:
print(
"pyserial not found. Install with:\n python -m pip install pyserial",
file=sys.stderr,
)
sys.exit(1)
# Add project root so we can import the frame parser
sys.path.insert(0, str(Path(__file__).parent.parent))
from minimateplus.framing import S3FrameParser, S3Frame
import json
# ── Helpers ───────────────────────────────────────────────────────────────────
def _ts() -> str:
return datetime.now().strftime("%Y-%m-%d %H:%M:%S.%f")[:-3]
def _hexdump(b: bytes) -> str:
return " ".join(f"{x:02X}" for x in b)
def _printable(b: bytes) -> str:
return b.decode("latin1", errors="replace")
_KNOWN_SUBS = {
0xA4: "POLL_RSP", 0xA5: "BULK_WAVEFORM_RSP", 0xE0: "ADVANCE_EVENT_RSP",
0xE1: "EVENT_IDX_FIRST_RSP", 0xE3: "MONITOR_STATUS_RSP", 0xEA: "SERIAL_NUM_RSP",
0xF3: "WAVEFORM_RECORD_RSP", 0xF5: "WAVEFORM_HEADER_RSP", 0xF7: "EVENT_INDEX_RSP",
0xF9: "UNK_06_RSP", 0xFE: "DEVICE_INFO_RSP",
0x69: "START_MONITOR_ACK", 0x68: "STOP_MONITOR_ACK",
0x97: "EVT_IDX_WRITE_ACK", 0x8C: "CONFIRM_B_ACK", 0x8E: "COMPLIANCE_WRITE_ACK",
0x8D: "CONFIRM_A_ACK", 0x7D: "TRIGGER_WRITE_ACK", 0x7C: "TRIGGER_CONFIRM_ACK",
0x96: "WAVEFORM_WRITE_ACK", 0x8B: "CONFIRM_C_ACK",
}
def _label_frame(frame: S3Frame) -> str:
name = _KNOWN_SUBS.get(frame.sub, f"UNK_0x{frame.sub:02X}")
chk = "" if frame.checksum_valid else "✗ BAD_CHK"
peek = frame.data[:24].hex() + ("" if len(frame.data) > 24 else "")
return (
f"S3 SUB=0x{frame.sub:02X} ({name:<22}) "
f"page=0x{frame.page_key:04X} data={len(frame.data):4d}B {chk} {peek}"
)
# ── Logger ────────────────────────────────────────────────────────────────────
class Logger:
def __init__(self, log_path: Path, jsonl_path: Path, raw_path: Path) -> None:
self._log = log_path.open("a", encoding="utf-8", newline="")
self._jl = jsonl_path.open("a", encoding="utf-8", newline="")
self._raw = raw_path.open("ab")
self._lock = threading.Lock()
self._frame_count = 0
def info(self, msg: str) -> None:
line = f"[{_ts()}] INFO | {msg}"
with self._lock:
print(line)
print(line, file=self._log, flush=True)
def ctrl(self, msg: str) -> None:
line = f"[{_ts()}] CTRL | {msg}"
with self._lock:
print(line)
print(line, file=self._log, flush=True)
def data_hex(self, msg: str) -> None:
line = f"[{_ts()}] HEX | {msg}"
with self._lock:
print(line)
print(line, file=self._log, flush=True)
def data_ascii(self, msg: str) -> None:
line = f"[{_ts()}] DATA | {msg}"
with self._lock:
print(line)
print(line, file=self._log, flush=True)
def frame(self, f: S3Frame) -> None:
with self._lock:
self._frame_count += 1
label = f"[{_ts()}] FRAME | #{self._frame_count:04d} {_label_frame(f)}"
print(label)
print(label, file=self._log, flush=True)
record = {
"frame": self._frame_count,
"sub": f.sub,
"page_key": f.page_key,
"data_len": len(f.data),
"data_hex": f.data.hex(),
"checksum_valid": f.checksum_valid,
}
print(json.dumps(record), file=self._jl, flush=True)
def write_raw(self, data: bytes) -> None:
with self._lock:
self._raw.write(data)
self._raw.flush()
def close(self) -> None:
with self._lock:
for fh in (self._log, self._jl, self._raw):
try:
fh.flush()
fh.close()
except Exception:
pass
# ── Control-line monitor thread ───────────────────────────────────────────────
def _monitor_control_lines(
ser: serial.Serial,
logger: Logger,
stop: threading.Event,
interval: float,
) -> None:
prev = dict(CTS=None, DSR=None, DCD=None, RI=None)
try:
prev.update(CTS=ser.cts, DSR=ser.dsr, DCD=ser.cd)
try:
prev["RI"] = ser.ri
except Exception:
pass
except Exception as exc:
logger.ctrl(f"Init error: {exc}")
return
logger.ctrl(
f"Initial: CTS={prev['CTS']} DSR={prev['DSR']} DCD={prev['DCD']} RI={prev['RI']}"
)
while not stop.is_set():
try:
cur = dict(CTS=ser.cts, DSR=ser.dsr, DCD=ser.cd, RI=None)
try:
cur["RI"] = ser.ri
except Exception:
pass
for name, val in cur.items():
if val != prev[name]:
logger.ctrl(f"{name}{val}")
prev[name] = val
except serial.SerialException as exc:
logger.ctrl(f"Poll error: {exc}")
break
stop.wait(interval)
# ── Serial open ───────────────────────────────────────────────────────────────
_PARITY = {
"N": serial.PARITY_NONE, "E": serial.PARITY_EVEN, "O": serial.PARITY_ODD,
"M": serial.PARITY_MARK, "S": serial.PARITY_SPACE,
}
_STOPBITS = {
1: serial.STOPBITS_ONE, 1.5: serial.STOPBITS_ONE_POINT_FIVE, 2: serial.STOPBITS_TWO,
}
def _open_serial(args: argparse.Namespace, logger: Logger) -> serial.Serial | None:
for attempt in range(1, args.open_retries + 2):
logger.info(
f"Opening {args.port} @ {args.baud},{args.bytesize}{args.parity}{args.stopbits} "
f"rtscts={args.rtscts} xonxoff={args.xonxoff} dsrdtr={args.dsrdtr} "
f"(attempt {attempt})"
)
try:
ser = serial.Serial(
port=args.port,
baudrate=args.baud,
bytesize=args.bytesize,
parity=_PARITY[args.parity],
stopbits=_STOPBITS[args.stopbits],
timeout=args.timeout,
xonxoff=args.xonxoff,
rtscts=args.rtscts,
dsrdtr=args.dsrdtr,
write_timeout=0,
)
try:
ser.setDTR(args.dtr == "on")
ser.setRTS(args.rts == "on")
logger.ctrl(f"Set DTR={args.dtr} RTS={args.rts}")
except Exception as exc:
logger.ctrl(f"DTR/RTS set failed: {exc}")
if args.send_break > 0:
try:
ser.break_condition = True
time.sleep(args.send_break / 1000.0)
ser.break_condition = False
logger.ctrl(f"BREAK held {args.send_break} ms")
except Exception as exc:
logger.ctrl(f"BREAK failed: {exc}")
return ser
except serial.SerialException as exc:
logger.info(f"Open failed: {exc}")
if attempt <= args.open_retries:
time.sleep(args.open_retry_delay)
return None
# ── Port picker ───────────────────────────────────────────────────────────────
def _list_ports() -> list:
ports = list(list_ports.comports())
if not ports:
print("No serial ports found.")
return []
print("Available serial ports:")
for i, p in enumerate(ports, 1):
print(f" {i:2d}) {p.device:<12} {p.description or ''}")
return ports
def _pick_port() -> str:
ports = _list_ports()
if not ports:
sys.exit(1)
if len(ports) == 1:
print(f"Auto-selecting: {ports[0].device}")
return ports[0].device
while True:
sel = input("Select port (number or name, e.g. COM3): ").strip()
if sel.isdigit() and 1 <= int(sel) <= len(ports):
return ports[int(sel) - 1].device
for p in ports:
if p.device.upper() == sel.upper():
return p.device
print("Not recognised. Enter list number or exact port name.")
# ── Main loop ─────────────────────────────────────────────────────────────────
def main() -> None:
ap = argparse.ArgumentParser(
description="Monitor Instantel Series-3 serial traffic with S3 frame parsing."
)
ap.add_argument("--port", "-p",
help="COM port (e.g. COM3). Omit to be prompted.")
ap.add_argument("--baud", "-b", type=int, default=38400)
ap.add_argument("--bytesize", type=int, choices=[5, 6, 7, 8], default=8)
ap.add_argument("--parity", choices=["N", "E", "O", "M", "S"], default="N")
ap.add_argument("--stopbits", type=float, choices=[1, 1.5, 2], default=1)
ap.add_argument("--rtscts", action="store_true")
ap.add_argument("--xonxoff", action="store_true")
ap.add_argument("--dsrdtr", action="store_true")
ap.add_argument("--dtr", choices=["on", "off"], default="on")
ap.add_argument("--rts", choices=["on", "off"], default="on")
ap.add_argument("--send-break", type=int, default=0,
help="Hold BREAK for N ms after open.")
ap.add_argument("--show", choices=["ascii", "hex", "both", "frames"],
default="frames",
help="'frames' (default) shows only parsed S3 frames. "
"'ascii'/'hex'/'both' also show raw bytes.")
ap.add_argument("--encoding", default="latin1")
ap.add_argument("--read-chunk", type=int, default=4096)
ap.add_argument("--timeout", type=float, default=0.05)
ap.add_argument("--poll-lines-interval", type=float, default=0.2)
ap.add_argument("--open-retries", type=int, default=0)
ap.add_argument("--open-retry-delay", type=float, default=0.8)
ap.add_argument("--ack-ok", action="store_true",
help="Auto-reply OK to AT* commands (except ATDT). "
"Useful for testing without a real modem.")
ap.add_argument("--list", action="store_true",
help="List available serial ports and exit.")
args = ap.parse_args()
if args.list:
_list_ports()
return
args.port = args.port or _pick_port()
# Build output paths
ts_str = datetime.now().strftime("%Y%m%d_%H%M%S")
out_dir = Path(__file__).parent / "captures" / f"serial_{ts_str}"
out_dir.mkdir(parents=True, exist_ok=True)
log_path = out_dir / f"session_{ts_str}.log"
jsonl_path = out_dir / f"session_{ts_str}.jsonl"
raw_path = out_dir / f"raw_s3_{ts_str}.bin"
logger = Logger(log_path, jsonl_path, raw_path)
logger.info(f"Output directory: {out_dir}")
logger.info(f"raw_s3 → {raw_path.name} (compatible with parse_capture.py)")
ser = _open_serial(args, logger)
if ser is None:
logger.info("Could not open serial port. Exiting.")
logger.close()
sys.exit(1)
s3_parser = S3FrameParser()
rx_buf = bytearray()
stop_evt = threading.Event()
ctrl_thread = threading.Thread(
target=_monitor_control_lines,
args=(ser, logger, stop_evt, args.poll_lines_interval),
daemon=True,
)
ctrl_thread.start()
logger.info("Monitoring started. Waiting for call-home. Press Ctrl+C to stop.")
try:
while True:
try:
data = ser.read(args.read_chunk)
except serial.SerialException as exc:
logger.info(f"Read error: {exc}")
break
if not data:
continue
# 1. Save raw bytes
logger.write_raw(data)
# 2. Optional raw display
if args.show in ("ascii", "both"):
txt = _printable(data)
for line in txt.splitlines():
logger.data_ascii(line)
if args.show in ("hex", "both"):
logger.data_hex(_hexdump(data))
# 3. Parse S3 frames
for byte in data:
result = s3_parser.feed(bytes([byte]))
if result:
frames = result if isinstance(result, list) else [result]
for f in frames:
logger.frame(f)
# 4. AT command handling for --ack-ok
if args.ack_ok:
rx_buf.extend(data)
while b"\r" in rx_buf or b"\n" in rx_buf:
for sep in (b"\r", b"\n"):
idx = rx_buf.find(sep)
if idx != -1:
line_bytes = bytes(rx_buf[:idx])
del rx_buf[:idx + 1]
break
else:
break
line_str = line_bytes.decode("latin1", errors="ignore").strip().upper()
if line_str.startswith("AT") and not line_str.startswith("ATDT"):
try:
ser.write(b"\r\nOK\r\n")
ser.flush()
logger.info(f"AT ack: {line_str!r} → OK")
except Exception as exc:
logger.info(f"AT ack write failed: {exc}")
except KeyboardInterrupt:
logger.info("Ctrl+C — stopping.")
finally:
stop_evt.set()
try:
ser.close()
except Exception:
pass
ctrl_thread.join(timeout=1.0)
logger.info(f"Capture saved to: {out_dir}")
logger.close()
if __name__ == "__main__":
main()
+618 -37
View File
@@ -36,7 +36,7 @@
| 2026-03-02 | §7.4 Event Index Block | **NEW:** `Monitoring LCD Cycle` identified at offsets +84/+85 as uint16 BE. Default value = 65500 (0xFFDC) = effectively disabled / maximum. Confirmed from operator manual §3.13.1g. |
| 2026-03-02 | §7.4 Event Index Block | **UPDATED:** Backlight confirmed as uint8 range 0255 seconds per operator manual §3.13.1e ("adjustable timer, 0 to 255 seconds"). Power save unit confirmed as minutes per operator manual §3.13.1f. |
| 2026-03-02 | Global | **NEW SOURCE:** Operator manual (716U0101 Rev 15) added as reference. Cross-referencing settings definitions, ranges, and units. Header updated. |
| 2026-03-02 | §14 Open Questions | Float 6.2061 in/s mystery: manual confirms only two geo ranges (1.25 in/s and 10.0 in/s). 6.2061 is NOT a user-selectable range → likely internal ADC full-scale calibration constant or hardware range ceiling. Downgraded to LOW priority. |
| 2026-03-02 | §14 Open Questions | Float 6.2061 in/s mystery: manual confirms only two geo ranges (1.25 in/s and 10.0 in/s). 6.2061 is NOT a user-selectable range → originally speculated as internal ADC full-scale constant, but was NOT confirmed at this time. Using it directly as the range produces ~9× PPV overread. Meaning unknown. Downgraded to LOW 2026-03-02, re-escalated to HIGH 2026-04-16. **RESOLVED 2026-04-17 — see §7.6.2 and changelog entry.** |
| 2026-03-02 | §14 Open Questions | `0x082A` hypothesis refined: 2090 decimal. At 1024 sps, 2 sec record = 2048 samples. Possible that 0x082A = total samples including 0.25s pre-trigger (256 samples) at some adjusted rate. Needs capture with different record time. |
| 2026-03-02 | §14 Open Questions | **NEW items added:** Trigger sample width (default=2), Auto Window (1-9 sec), Aux Trigger (enabled/disabled) — all confirmed settings from operator manual not yet mapped in protocol. |
| 2026-03-02 | §14 Open Questions | Monitoring LCD Cycle resolved — removed from open questions. |
@@ -92,12 +92,24 @@
| 2026-04-06 | §7.8.4 | **NEW — 5A end-of-stream signalling confirmed.** After streaming all waveform chunks, the device sends exactly **1 raw byte** in response to the next chunk request, then goes silent for the full recv timeout. This byte is NOT a complete DLE-framed A5 response — the frame parser accumulates it as `bytes_fed=1` and never assembles a frame. This is the device's natural end-of-stream signal. Handling: on TimeoutError, if `bytes_fed > 0` AND prior chunks were received, treat as graceful end and proceed to the termination frame. A `bytes_fed=0` timeout with no prior chunks is a genuine transport failure and must still raise. |
| 2026-04-06 | §7.8.4 | **NEW — 5A chunk timing and count (empirical, BE11529 at 1024 sps).** Each chunk response arrives within ~1 second over TCP/cellular. A 9,306-sample event (≈9.1 s at 1024 sps) produces **35 chunks** before end-of-stream. Chunks 116 have varying data lengths (10361123 bytes); chunks 1735 are uniformly 1036 bytes each (post-event silence, all-zero ADC samples). Safe recv timeout for chunk loop: **10 s** (10× typical response time). Default transport timeout (120 s) results in a ~2-minute stall per event at end-of-stream. |
| 2026-04-06 | §7.8.3 | **KNOWN ISSUE — `_decode_a5_waveform` hardcoded fi==9 skip.** The decoder contains `elif fi == 9: continue` which was written for the 9-frame original blast capture where frame 9 was a device terminator. For streams with >9 frames (current device produces 35+), frame index 9 is live waveform data — this skip discards ~1,070 bytes (~133 sample-sets) per event. The terminator is now detected via `page_key == 0x0000`, not by frame index. The fi==9 skip should be removed. |
| 2026-04-06 | §7.8 | **CONFIRMED — ADC count-to-physical-unit conversion.** Raw waveform samples are signed 16-bit integers (counts). Conversion: `value = counts × (range / 32767)`. For geo channels: range = 10.000 in/s (from the device's compliance config geo range field). For the mic channel: range is in psi (device-specific). Near-full-scale counts (≈32,700) on all four channels simultaneously indicate ADC saturation (clipping) from a high-amplitude event. |
| 2026-04-08 | §5.1, §7.10, §12 | **NEW — Monitoring commands confirmed.** SUB 0x1C (monitor status), 0x96 (start monitoring), 0x97 (stop monitoring) all confirmed from 4-8-26/2ndtry capture. SESSION_RESET (`41 03`) required before POLL to wake a monitoring unit. `section[6] == 0x10` is the monitoring flag (CORRECTED 2026-04-08 — was wrongly `section[1]`). Battery/memory at relative-from-end offsets: `section[-11:-9]` (battery×100), `section[-9:-5]` (memory_total), `section[-5:-1]` (memory_free) — stable across all payload size variants (5255 bytes). |
| 2026-04-06 | §7.8 | **⚠ PARTIALLY INVALIDATED — ADC count-to-physical-unit conversion.** Raw waveform samples are signed 16-bit integers (counts). Conversion formula `value = counts × (range / 32767)` is believed correct, but the `range` value was UNKNOWN at time of writing. **UPDATED 2026-04-17:** `max_range_geo` = 6.206053 is confirmed as the ADC-to-velocity scale factor (inverse sensitivity, (in/s)/V). The correct conversion is therefore: `PPV (in/s) = counts × (1.61133 / 32767) × 6.206053` = `counts × 4.982e-5` in/s per count. The earlier ~9× overread from using 6.206053 directly as the range was because the range IS 1.61133 × 6.206053 = 10.000 in/s, not 6.206053. See §7.6.2 for the confirmed field layout. |
| 2026-04-08 | §5.1, §7.10, §12 | **NEW — Monitoring commands confirmed.** SUB 0x1C (monitor status), 0x96 (start monitoring), 0x97 (stop monitoring) all confirmed from 4-8-26/2ndtry capture. SESSION_RESET (`41 03`) required before POLL to wake a monitoring unit. |
| 2026-04-09 | §7.10 | **CORRECTED — monitoring flag and battery/memory offsets.** `section[1] == 0x10` is the monitoring flag (100% accurate across 144 data frames in 2ndtry capture). Previous note claiming `section[6]` was wrong — section[6] has device-specific non-binary values (0xea/0x07). Battery/memory offsets corrected: `section[-10:-8]` (battery×100), `section[-8:-4]` (memory_total), `section[-4:]` (memory_free). NOTE: `frame.data` has checksum stripped by parser — earlier offsets of `[-11:-9]`/`[-9:-5]`/`[-5:-1]` were wrong because they assumed a trailing checksum byte that isn't there. |
| 2026-04-08 | §7.10 | **NEW — SUBs 0x0E (channel sensor data) and 0x98 (trigger test) observed** in 4-8-26/sensor-check capture (Blastware "Unit Channel Test" comms check). SUB 0x0E: 2-step read with channel selector in `params[6:8]`, data length 0x0A per channel, RSP SUB = 0xF1. SUB 0x98: single probe frame with `params[0] = 0xFF`, RSP SUB = 0x67; sent twice per test cycle. Not yet implemented in SFM. |
| 2026-04-08 | §7.10 | **NEW — SUBs 0x15 and 0x01 observed in sensor-check capture.** SUB 0x15 (serial number short form, data length 0x0A, RSP 0xEA) and SUB 0x01 (device info block, data length 0x98 = 152 bytes, RSP 0xFE) seen in Blastware's "Unit Channel Test" init sequence. Note: SUB 0x01 response SUB 0xFE collides with the existing SUB 0xFE → RSP 0x01 naming convention — they are inverse commands. |
| 2026-04-08 | §12 | **CONFIRMED — Unit partially reachable during on-device sensor check.** 4-8-26/sensor-check capture shows: POLL responds normally throughout; SUB 0x0E channel reads partially served (channels 04 responded), then ~40s silent gap while sensor check ran, then channels 57 responded. On-device sensor check duration ≈ 40 s. SFM `_pollMonitorConfirm()` polls status every 5 s for up to 60 s after start_monitoring. |
| 2026-04-08 | §7.9 (NEW) | **NEW — Compliance config field inventory captured from Blastware UI.** See §7.9 for full field list (Recording Setup, Notes, Special Setups tabs). Most fields NOT yet mapped to raw byte offsets. Confirmed decoded: sample_rate, record_time, trigger_level_geo, alarm_level_geo, max_range_geo, backlight_on_time, power_saving_timeout, monitoring_lcd_cycle, project/client/operator/sensor_location/notes. Sensor Check dropdown (Before monitoring / After each event / Disabled) NOT YET LOCATED in raw config bytes. |
| 2026-04-11 | §5.1, §5.2 | **NEW — Erase-all command sequence confirmed from MITM capture.** SUB 0xA3 (begin erase, token=0xFE → ack 0x5C) + SUB 0xA2 (confirm erase, token=0xFE → ack 0x5D). Standard `build_bw_frame` format (not write-format). Required intermediate steps: 0x1C probe+data (monitor status read) + 0x06 probe+data (event storage range). All response SUBs follow the standard 0xFFSUB formula with no exceptions. |
| 2026-04-11 | §5.1 | **CONFIRMED — SUB 0x06 (CHANNEL CONFIG READ) now confirmed as event storage range.** Two-step read, data offset = 0x24 (36 bytes). Token=0xFE at params[7]. Last 8 bytes of response: first stored event key (bytes 8:4) and last stored event key (bytes 4:). Both equal `01110000` when device memory is empty. Used by Blastware to verify erase completion. |
| 2026-04-11 | §7.11 (NEW) | **NEW — §7.11 Erase-All Protocol added.** Full wire sequence, SUB 0x06 storage range payload layout, post-erase key counter reset (resets to `0x01110000`). Confirmed from 4-11-26 MITM capture of live Blastware ACH session. |
| 2026-04-11 | §14.6 | **RESOLVED — ACH Session Lifecycle is no longer "Future".** `bridges/ach_server.py` fully implements inbound ACH: POLL handshake, device info, event download. State tracked via `ach_state.json` (key-based, with `max_downloaded_key` for post-erase detection). `--clear-after-download` flag added for the standard delete-after-upload workflow. |
| 2026-04-17 | §7.6.2, §14 | **RESOLVED — Float 6.206053 at channel_label+28 is the ADC-to-velocity scale factor.** Confirmed from Series III Interface Handbook §4.5 formula: `Range (×1) = 1.61133 V / Sensitivity (V/unit)`. For the standard Instantel geophone at Normal range (10.000 in/s): Sensitivity = 1.61133 / 10 = 0.161133 V/(in/s). The stored value is the **inverse sensitivity** = 1/0.161133 = **6.206053 (in/s)/V**. Cross-check: 1.61133 V × 6.206053 = 10.000 in/s ✅. The firmware uses it as: `PPV (in/s) = ADC_voltage (V) × 6.206053`. Value is identical on all Instantel standard geophones — it is a hardware/firmware constant, NOT a user-configurable setting. Do NOT write this field. Open question §14 item "Max Geo Range float 6.2061" is now **RESOLVED**. |
| 2026-04-20 | §7.6.4 (NEW), §7.9, Appendix B | **CONFIRMED — Recording Mode byte location.** Three targeted captures (4-20-26) confirmed `recording_mode` at anchor8 in both the E5 read payload and the BW write payload (6-byte anchor `\xbe\x80\x00\x00\x00\x00`). BW write payload and E5 read payload are **byte-identical** around the anchor region — Blastware round-trips the wire-encoded E5 bytes verbatim with only the target field modified. Anchor position varies by ±1 depending on whether recording_mode = 0x03 (Histogram), because E5 wire-encodes `0x03` as the inner DLE+ETX pair `\x10\x03` (2 bytes), which S3FrameParser preserves as two literal bytes in `compliance_raw`. Enum: `0x00`=Single Shot, `0x01`=Continuous, `0x03`=Histogram, `0x04`=Histogram+Continuous. `0x02` value not yet observed. The byte at anchor9 is `0x00` for Single Shot / Continuous, and `0x10` for Histogram (DLE prefix from E5 encoding) and Histogram+Continuous (actual config byte). See §7.6.4 for full details. |
| 2026-04-21 | Appendix D (NEW) | **NEW — Blastware .N00 and .MLG file formats fully decoded.** `minimateplus/blastware_file.py` implements `write_n00()` and `write_mlg()`. N00 file format confirmed: 22B header + 21B STRT record + variable body + 26B footer. Body reconstructed from A5 bulk waveform stream frames with per-frame skip amounts (probe=7+strt_pos+21, A5[1]=13, A5[2+]=12, terminator=11) and DLE strip rule (strip `0x10` before `{0x02,0x03,0x04}`, keep following byte). Footer extracted verbatim from terminator frame's last 26 bytes. Split-pair edge case: when `frame.data[-1]==0x10` and `chk_byte∈{0x02,0x03,0x04}`, reunite both bytes before stripping and always remove trailing chk_byte (`stripped[:-1]`) — chk_byte is checksum, not payload. STRT record must be copied verbatim from A5[0]; bytes [10:20] are device-specific and cannot be reconstructed from Event fields. `write_n00` verified byte-perfect against `M529LIY6.N00` from 4-3-26-multi_event capture. MLG format: 308B header + N×292B records; CRC algorithm unknown (write as 0x0000). |
| 2026-04-21 | Appendix D §D.5 (NEW) | **NEW — Blastware filename encoding fully decoded.** Serial prefix: `chr(ord('B') + floor(serial/1000))` + last 3 digits zero-padded. Stem: 4-char base-36 of `floor(total_seconds/1296)`. Extension: `AB0` for manual/direct downloads (3 chars), `AB0W` or `AB0H` for ACH/call-home downloads (4 chars), where `AB` = 2-char base-36 of `total_seconds % 1296` and W/H = waveform/histogram. Epoch = 1985-01-01 00:00:00 device local time. Confirmed against 3,248 files from 10-year production archive with zero errors. 3-day cycle property: same daily recording time cycles through 3 extensions (864s/day shift, period=3 days). `blastware_filename(event, serial, ach=False)` implements full formula. |
| 2026-04-21 | §7.6.2, §5.3 | **CORRECTED — compliance_raw contains wire-encoded bytes, NOT logical bytes.** S3FrameParser appends DLE+ETX inner-frame pairs as two literal bytes to the frame body. Any `0x03` values in the compliance config appear in `compliance_raw` as `\x10\x03` (two bytes), not as a single `0x03`. The previous claim "S3FrameParser handles this transparently so compliance_raw contains logical (destuffed) bytes" was wrong. Consequence: `compliance_raw` is the wire-encoded E5 payload; anchor-relative reads work correctly because the anchor position automatically accounts for any DLE-encoded bytes before it. For write-back, round-tripping `compliance_raw` verbatim sends the correct wire bytes to the device. **DLE ETX escaping in write frames:** Blastware escapes `0x03` bytes in write frame data as `\x10\x03` on wire; our `build_bw_write_frame` does not (writes data raw). Device is confirmed to accept raw writes for all tested modes — likely uses the offset/length field for write frame framing, not ETX scanning. |
| 2026-04-20 | §7.6.2, §7.9, Appendix B | **CONFIRMED — Geophone maximum range / sensitivity selector byte location.** Two targeted captures (4-20-26, geo sensitivity folder): one at Normal 10.000 in/s, one at Sensitive 1.250 in/s. E5 read payload diff: exactly 3 bytes differ at channel_label+33 for Tran/Vert/Long. Values: `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. Same offset applies to the SUB 71 write payload (which is the same 2126-byte E5-format buffer round-tripped verbatim). **`channel_label+20` reads `0x01` in ALL captures regardless of range setting — it is NOT this field.** Previous hypothesis (uint8 at Tran+20, 0x01=Normal) was WRONG. Stored as `geo_range` in `ComplianceConfig`. Encoded to all three geo channel blocks (Tran/Vert/Long) at label+33. |
| 2026-04-20 | §5.1, §5.3, §7.12 (NEW) | **NEW — Auto Call Home config protocol confirmed from 4-20-26 call home settings captures.** SUB 0x2C (Call Home Config READ, response 0xD3, data offset 0x7C=124) and SUB 0x7E/0x7F (WRITE + CONFIRM, response 0x81/0x80) confirmed. Write payload = read payload (125 bytes) + `\x00\x00` (127 bytes total). **DLE-escaped ETX at raw[117:119]:** the device returns logical value 0x03 (num_retries=3) as `\x10\x03` on the wire — S3FrameParser preserves both bytes as two literals, causing a +1 byte shift for all subsequent fields. Write frame sends these bytes verbatim (device interprets `\x10\x03` as literal value 3). Field map confirmed from 10-frame BW TX diff. See §7.12 for full layout. |
---
@@ -242,7 +254,7 @@ Step 4 — Device sends actual data payload:
| `15` | **SERIAL NUMBER REQUEST** | Requests device serial number. | ✅ CONFIRMED |
| `01` | **FULL CONFIG READ** | Requests complete device configuration block (~0x98 bytes). Firmware, model, serial, channel config, scaling factors. | ✅ CONFIRMED |
| `08` | **EVENT INDEX READ** | Requests the event record index (0x58 bytes). Event count and record pointers. | ✅ CONFIRMED |
| `06` | **CHANNEL CONFIG READ** | Requests channel configuration block (0x24 bytes). | ✅ CONFIRMED |
| `06` | **EVENT STORAGE RANGE READ** | Requests event storage range block (0x24 = 36 bytes). Token=0xFE at params[7]. Last 8 bytes of response: first stored event key (`[-8:-4]`) and last stored event key (`[-4:]`). Both equal `01110000` when device is empty. Used by Blastware as part of the erase-all verification step. Previously labelled "CHANNEL CONFIG READ" — function now confirmed from 4-11-26 MITM capture. | ✅ CONFIRMED 2026-04-11 |
| `1C` | **TRIGGER CONFIG READ** | Requests trigger settings block (0x2C bytes). | ✅ CONFIRMED |
| `1E` | **EVENT HEADER READ** | Gets first waveform key. Token byte at params[7] (0x00=browse, 0xFE=download-arm). Key at data[11:15]; trailing offset at data[15:19] (0 = only one event). Two uses: (1) all-zero to get key0; (2) token=0xFE after 0A, before 0C — REQUIRED to arm device for SUB 5A. | ✅ CONFIRMED 2026-04-06 |
| `0A` | **WAVEFORM HEADER READ** | Checks record type for a given waveform key. Variable DATA_LENGTH: 0x30=full bin, 0x26=partial bin. Key at params[4..7]. Required before every 1F call to establish device waveform context. | ✅ CONFIRMED 2026-03-31 |
@@ -252,13 +264,16 @@ Step 4 — Device sends actual data payload:
| `24` | **WAVEFORM PAGE A?** | Paged waveform read, possibly channel group A. | 🔶 INFERRED |
| `25` | **WAVEFORM PAGE B?** | Paged waveform read, possibly channel group B. | 🔶 INFERRED |
| `09` | **UNKNOWN READ A** | Read command, response (`F6`) returns 0xCA (202) bytes. Purpose unknown. | 🔶 INFERRED |
| `1A` | **COMPLIANCE CONFIG READ** | Multi-step sequence (A+B+C+D frames). Response (E5) carries sample_rate (uint16 BE at anchor2), record_time (float32 BE at anchor+10), trigger/alarm/max_range floats, and project strings. Anchor: `\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00`, search cfg[0:150]. Total ~2126 cfg bytes. | ✅ CONFIRMED 2026-04-02 |
| `1A` | **COMPLIANCE CONFIG READ** | Multi-step sequence (A+B+C+D frames). Response (E5) carries recording_mode (uint8 at anchor4 in E5 sf1), sample_rate (uint16 BE at anchor2), record_time (float32 BE at anchor+10), trigger/alarm/max_range floats, and project strings. Anchor: `\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00`, search cfg[0:150]. Total ~2126 cfg bytes. See §7.6.4 for recording_mode enum. | ✅ CONFIRMED 2026-04-02; recording_mode added 2026-04-20 |
| `2E` | **UNKNOWN READ B** | Read command, response (`D1`) returns 0x1A (26) bytes. Purpose unknown. | 🔶 INFERRED |
| `0E` | **CHANNEL SENSOR DATA** | Real-time sensor reading for one channel. Two-step read, data length 0x0A (10 bytes). Channel selector in params[6:8] (0x00000x0007 for 8 channels). Response (F1) carries amplitude, frequency, overswing data for that channel. Used by Blastware "Unit Channel Test" comms check. | ✅ CONFIRMED 2026-04-08 |
| `98` | **TRIGGER TEST** | Trigger-test command. Single probe frame; `params[0] = 0xFF`. Response (0x67) is all-zero data. Sent twice per Blastware comms-check cycle. Not a full POLL, no monitor state change. | ✅ CONFIRMED 2026-04-08 |
| `1C` | **MONITOR STATUS READ** | Two-step read, data offset 0x2C (44 bytes). `section[6] == 0x10` → monitoring; `0x00` → idle (CORRECTED 2026-04-08 — was wrongly documented as section[1]). Payload length varies (5255 bytes) but battery/memory block is always the last 10 bytes before checksum: `section[-11:-9]` = battery×100 (uint16 BE), `section[-9:-5]` = memory_total (uint32 BE), `section[-5:-1]` = memory_free (uint32 BE). Confirmed from 2ndtry 4-8-26 full byte diff across 3 payload size variants. | ✅ CONFIRMED 2026-04-08 |
| `1C` | **MONITOR STATUS READ** | Two-step read, data offset 0x2C (44 bytes). `section[1] == 0x10` → monitoring; `0x00` → idle (CONFIRMED 2026-04-09, 100% accuracy on 144 frames). Payload length: 4647 bytes IDLE, 4849 bytes MONITORING. `frame.data` has checksum stripped — no trailing byte to skip. Battery/memory at end: `section[-10:-8]` = battery×100 (uint16 BE), `section[-8:-4]` = memory_total (uint32 BE), `section[-4:]` = memory_free (uint32 BE). | ✅ CONFIRMED 2026-04-09 |
| `96` | **START MONITORING** | Single write frame, no data payload. Transitions unit from idle to monitoring mode (after optional on-device sensor check ~40 s). | ✅ CONFIRMED 2026-04-08 |
| `97` | **STOP MONITORING** | Single write frame, no data payload. Stops monitoring, unit returns to idle. | ✅ CONFIRMED 2026-04-08 |
| `2C` | **CALL HOME CONFIG READ** | Two-step read, data offset 0x7C (124 bytes + 1-byte DLE artefact = 125 raw bytes). Returns Auto Call Home configuration: enable flag, dial string, scheduled call times, retry settings, modem timing. Response SUB = 0xD3. **DLE note:** logical value 0x03 (num_retries) is returned as `\x10\x03` on the wire, which S3FrameParser preserves as two literal bytes — this shifts all subsequent field positions by +1. See §7.12 for full field map. | ✅ CONFIRMED 2026-04-20 |
| `A3` | **ERASE ALL BEGIN** | Single frame, token=0xFE at params[7]. Initiates device memory erase. Must be followed by 0x1C probe+data + 0x06 probe+data + 0xA2 to complete. Standard `build_bw_frame` (not write-format). Response ack SUB = 0x5C. | ✅ CONFIRMED 2026-04-11 |
| `A2` | **ERASE ALL CONFIRM** | Single frame, token=0xFE at params[7]. Commits the erase initiated by 0xA3. After this ack (SUB 0x5D), device memory is cleared and the event counter resets to `0x01110000`. | ✅ CONFIRMED 2026-04-11 |
All requests use CMD byte `0x02`. All responses use CMD byte `0x10 0x02` (which, after de-stuffing, is just the DLE+CMD combination — see §3).
@@ -272,7 +287,7 @@ All requests use CMD byte `0x02`. All responses use CMD byte `0x10 0x02` (which,
| `15` | `EA` | ✅ CONFIRMED |
| `01` | `FE` | ✅ CONFIRMED |
| `08` | `F7` | ✅ CONFIRMED |
| `06` | `F9` | ✅ CONFIRMED |
| `06` | `F9` | ✅ CONFIRMED 2026-04-11 |
| `1C` | `E3` | ✅ CONFIRMED 2026-04-08 |
| `1E` | `E1` | ✅ CONFIRMED |
| `0A` | `F5` | ✅ CONFIRMED |
@@ -286,6 +301,9 @@ All requests use CMD byte `0x02`. All responses use CMD byte `0x10 0x02` (which,
| `98` | `67` | ✅ CONFIRMED 2026-04-08 |
| `96` | `69` | ✅ CONFIRMED 2026-04-08 |
| `97` | `68` | ✅ CONFIRMED 2026-04-08 |
| `2C` | `D3` | ✅ CONFIRMED 2026-04-20 |
| `A3` | `5C` | ✅ CONFIRMED 2026-04-11 |
| `A2` | `5D` | ✅ CONFIRMED 2026-04-11 |
---
@@ -305,6 +323,8 @@ Write commands are initiated by Blastware (`BW->S3`) and use SUB bytes in the `0
| `72` | **WRITE CONFIRM A** | Short frame, no data. Likely commit/confirm step after `71`. | `8D` | ✅ CONFIRMED |
| `73` | **WRITE CONFIRM B** | Short frame, no data. | `8C` | ✅ CONFIRMED |
| `74` | **WRITE CONFIRM C** | Short frame, no data. | `8B` | ✅ CONFIRMED |
| `7E` | **CALL HOME CONFIG WRITE** | Writes Auto Call Home configuration (127 bytes: 125-byte read payload + `\x00\x00`). Offset = data[1]+2 = 0x7E. Write format (DLE-aware checksum, only BW_CMD `0x10` doubled on wire). Response SUB = 0x81. Must be followed by SUB 0x7F confirm. | `81` | ✅ CONFIRMED 2026-04-20 |
| `7F` | **CALL HOME WRITE CONFIRM** | Short frame, no data. Commits call home config write from SUB 0x7E. Response SUB = 0x80. | `80` | ✅ CONFIRMED 2026-04-20 |
| `82` | **TRIGGER CONFIG WRITE** | Writes trigger config block (0x1C bytes, mirrors SUB `1C` read). | `7D` | ✅ CONFIRMED |
| `83` | **TRIGGER WRITE CONFIRM** | Short frame, no data. Likely commit step after `82`. | `7C` | ✅ CONFIRMED |
@@ -318,6 +338,8 @@ Write commands are initiated by Blastware (`BW->S3`) and use SUB bytes in the `0
| `72` | `8D` |
| `73` | `8C` |
| `74` | `8B` |
| `7E` | `81` |
| `7F` | `80` |
| `82` | `7D` |
| `83` | `7C` |
@@ -519,7 +541,7 @@ The SUB `1A` read response (`E5`) and SUB `71` write block contain per-channel t
| Field | Example bytes | Decoded | Certainty |
|---|---|---|---|
| `[00 00]` | `00 00` | Separator / padding | 🔶 INFERRED |
| Max range float | `40 C6 97 FD` | 6.206 — full-scale range in in/s | 🔶 INFERRED |
| ADC scale factor | `40 C6 97 FD` | **6.206053 (in/s)/V — CONFIRMED 2026-04-17.** This is the inverse sensitivity of the standard Instantel geophone = 1/0.161133. Interface Handbook §4.5: `Range = 1.61133 V × 6.206053 = 10.000 in/s`. Used by firmware: `PPV (in/s) = ADC_voltage × 6.206053`. Hardware constant — do NOT write. | ✅ CONFIRMED |
| `[00 00]` | `00 00` | Separator / padding | 🔶 INFERRED |
| **Trigger level** | `3F 19 99 9A` | **0.600 in/s** — IEEE 754 BE float | ✅ CONFIRMED |
| Unit string | `69 6E 2E 00` | `"in.\0"` | ✅ CONFIRMED |
@@ -611,6 +633,53 @@ The sample rate bytes sit immediately before a `0x10` (DLE) prefix byte in the r
---
### 7.6.4 Recording Mode
> ✅ **CONFIRMED — 2026-04-20** (BE11529 / firmware S338.17). Three targeted captures in a single Blastware session (4-20-26 directory), changing Recording Mode only between each write.
Recording mode is stored as a **uint8** with different anchor-relative positions depending on whether you are reading from a device response or constructing a write payload.
**In the SUB 71 write payload (3-chunk compliance write, `cfg[5]`):**
| Enum | Mode |
|---|---|
| `0x00` | Single Shot |
| `0x01` | Continuous |
| `0x02` | Unknown (not yet observed) |
| `0x03` | Histogram |
| `0x04` | Histogram + Continuous (combined mode) |
Anchor-relative position: **anchor 3** (3 bytes before the 10-byte anchor in the write payload). The write payload layout in the region around the anchor:
```
cfg[anchor - 3] = recording_mode (uint8)
cfg[anchor - 2] = sample_rate_hi (uint8, MSB of uint16 BE)
cfg[anchor - 1] = sample_rate_lo (uint8, LSB of uint16 BE)
cfg[anchor:anchor+10] = \x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00 ← anchor
cfg[anchor + 10:anchor + 14] = record_time (float32 BE)
```
**In the E5 read response (sub-frame 1, page=`0x0010`, `data[17]`):**
The anchor appears at `data[21]` in this sub-frame. Recording mode is at `data[17]` = **anchor 4** (one position earlier than in the write payload). This is because an extra `0x10` byte is present at `data[18]` in the read format (between recording_mode and sample_rate), which is NOT present in the write payload. The read-format layout:
```
data[17] = recording_mode (uint8)
data[18] = 0x10 ← extra byte present in E5 read only; absent in SUB 71 write
data[19] = sample_rate_hi (uint8, MSB of uint16 BE)
data[20] = sample_rate_lo (uint8, LSB of uint16 BE)
data[21:31] = anchor (\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00)
data[31:35] = record_time (float32 BE)
```
**Chunk checksum at `cfg[1024]`:** The first of the three SUB 71 write chunks (1027 bytes) contains a running checksum byte at `cfg[1024]` whose delta exactly equals the delta of `cfg[5]` (recording_mode). This byte reflects the cumulative change from `recording_mode` through to its position and should not be mistaken for a second copy of the recording_mode field.
**Decode path (`_decode_compliance_config_into`):** use `data[anchor_pos - 4]` where `anchor_pos` is the index of the first byte of the anchor in the assembled E5 cfg bytes.
**Encode path (`_encode_compliance_config`):** use `cfg[anchor_pos - 3]` = recording_mode value (write-payload offset; no extra `0x10` byte).
---
### 7.7 Blastware `.set` File Format
> 🔶 **INFERRED — 2026-03-01** from `Standard_Recording_Setup.set` cross-referenced against known wire payloads.
@@ -646,7 +715,7 @@ offset size type value (Tran example) meaning
+10 2 uint16 0x0015 = 21 unknown
+12 4 bytes 03 02 04 01 flags (recording mode etc.)
+16 4 uint32 0x00000003 record time in seconds ✅ CONFIRMED
+1A 4 float32 6.2061 max range (in/s for geo, psi for mic)
+1A 4 float32 6.206053 ✅ CONFIRMED 2026-04-17 — ADC-to-velocity scale factor (= 1/sensitivity = (in/s)/V). Interface Handbook §4.5: Range = 1.61133 V × 6.206053 = 10.000 in/s (Normal range). Firmware uses: PPV (in/s) = ADC_voltage × 6.206053. Hardware constant — identical on all tested units. Do NOT write.
+1E 2 00 00 padding
+20 4 float32 0.6000 trigger level ✅ CONFIRMED
+24 4 char[4] "in.\0" / "psi\0" unit string (geo vs mic)
@@ -1162,26 +1231,52 @@ Two critical differences from `build_bw_frame`:
| Frame | offset_word | counter | params | Purpose |
|---|---|---|---|---|
| Probe | `0x1004` | `0x0000` | 10 bytes (`bulk_waveform_params(0)`) | Initiate transfer |
| Chunk 1 | `0x1004` | `0x0400` | 11 bytes | First data chunk |
| Chunk 2 | `0x1004` | `0x0800` | 11 bytes | Second chunk |
| Chunk N | `0x1004` | `N * 0x0400` | 11 bytes | Nth chunk |
| Chunk 1 | `0x1004` | `max(key4[2:4], 0x0400)` | 11 bytes | First data chunk |
| Chunk 2 | `0x1004` | `max(key4[2:4], 0x0400) + 0x0400` | 11 bytes | Second chunk |
| Chunk N | `0x1004` | `max(key4[2:4], 0x0400) + (N-1) * 0x0400` | 11 bytes | Nth chunk |
| … | … | … | … | … |
| Termination | `0x005A` | `last + 0x0400` | 10 bytes | End transfer |
| Termination | `0x005A` | `max(key4[2:4], 0x0400) + N * 0x0400` | 10 bytes | End transfer |
> ⚠️ **2026-04-06 CORRECTED — chunk counter is monotonic for ALL chunks.**
> The 4-2-26 BW TX capture showed counter=0x1004 for chunk 1, which was hardcoded as a
> special case. This was a Blastware artifact. Empirically confirmed: counter=0x0400 for
> chunk 1 works correctly; counter=0x1004 causes the device to time out. The device does
> NOT strictly validate the counter value — it streams data for any valid 5A request for
> the given key. Use `chunk_num * 0x0400` (monotonic) for all chunks.
> BW's true internal formula is `key4[2:4] + n * 0x0400`. For event 1 (key `01110000`)
> this equals `n * 0x0400` since `key4[2:4] = 0x0000`. The monotonic formula is correct
> for all keys encountered on this device.
> ⚠️ **2026-04-06 CORRECTED — chunk counter is `key4[2:4] + (N-1) * 0x0400`.**
> The 4-2-26 BW TX capture showed counter=0x1004 for chunk 1 of key `01110000`, leading to
> an interim "monotonic n * 0x0400" formula. This was accidentally correct because
> `key4[2:4] == 0x0000` for that event.
>
> **2026-04-24 CORRECTION:** The counter is an absolute circular-buffer address.
> BW's true formula is `key4[2:4] + (chunk_num - 1) * 0x0400` where `key4[2:4]` is the
> event's storage base offset (`(key4[2]<<8) | key4[3]`). For keys where
> `key4[2:4] != 0x0000` (e.g. key `01111884`), using `n * 0x0400` sends requests into the
> wrong buffer region — the device returns data from a completely different event.
>
> **2026-04-26 FINAL CORRECTION:** The formula `key4[2:4] + (N-1) * 0x0400` is wrong when
> `key4[2:4] == 0x0000` (e.g. event key `01110000`, the very first event after a device erase).
> Counter=0x0000 for chunk 1 is the same address as the probe frame — the device re-returns
> the STRT record data instead of waveform payload (frame 1 has len=1097, same as probe, and
> contains `b"STRT\xff\xfe"`, contributing zero waveform bytes).
> Final formula: `max(key4[2:4], 0x0400) + (chunk_num - 1) * 0x0400`.
> For key `01110000`: chunk 1 = 0x0400 (confirmed working, empirical test 2026-04-06).
> For key `0111245a`: chunk 1 = 0x245a (unchanged, confirmed from 4-3-26 capture).
The `stop_after_metadata=True` flag causes the loop to stop as soon as `b"Project:"` is
found in the accumulated A5 frame data, typically after 79 chunks. A termination frame
found in the accumulated A5 frame data, typically after 49 chunks. A termination frame
is always sent before returning.
**IMPORTANT — one extra chunk required after "Project:" for valid file footer (confirmed 2026-04-23):**
When writing a Blastware-compatible waveform file, stopping immediately at "Project:" and
sending termination produces an empty termination response with no footer bytes (`0e 08`
marker missing). Blastware downloads exactly **one more chunk** after finding "Project:"
before sending termination — that extra chunk primes the device to return valid footer
bytes (monitoring start/stop timestamps) in the termination response.
`read_bulk_waveform_stream(stop_after_metadata=True)` implements this: after the "Project:"
chunk is received, one additional chunk is requested before breaking. The termination
response (`include_terminator=True`) then contains the correct `0e 08` footer.
**do NOT use `full_waveform=True` for Blastware file writing** — for events with long
post-event silence (35 chunks), the silence chunks contain embedded device-internal
pointer structures that produce spurious STRT markers in the file body. Blastware only
downloads 45 chunks (metadata + one signal chunk) regardless of event length.
#### 7.8.3 A5 Frame Layout
Each A5 response frame contains a chunk of raw bulk data. Frame 7 of the stream carries the
@@ -1226,13 +1321,19 @@ TimeoutError caught:
Chunks with uniform 1,036-byte payload (chunks 1735 in the observed event) contain all-zero ADC samples — the device continues recording silence until the configured record time expires before terminating the stream.
**ADC count-to-physical conversion:**
**ADC count-to-physical conversion — ✅ CONFIRMED 2026-04-17:**
Raw samples are signed 16-bit integers (32,768 to +32,767). Source: Interface Handbook §4.5.
**CONFIRMED 2026-04-17** — The `max_range_geo` field (float32 = 6.206053, bytes `40 C6 97 FD`) is the **ADC-to-velocity scale factor** (inverse sensitivity, (in/s)/V) for the standard Instantel geophone, confirmed from Interface Handbook §4.5. The correct conversion formula is:
Raw samples are signed 16-bit integers (32,768 to +32,767). To convert to physical units:
```
value_in_s (in/s) = counts × (geo_range / 32767)
PPV (in/s) = ADC_voltage (V) × 6.206053
= counts × (1.61133 / 32767) × 6.206053
= counts × 4.982e-5 (in/s per count at full scale)
```
where `geo_range` is from the compliance config (typically 10.000 in/s). Mic channel uses psi units with its own range. Near-full-scale values on all channels simultaneously indicate ADC saturation (clipping).
where `geo_range = 1.61133 V × 6.206053 = 10.000 in/s` is the Normal (Gain=1) full-scale range. The earlier ~9× overread was caused by mistakenly using 6.206053 as the range directly — it is actually the scale factor, and the range itself is `ADC_fullscale × scale_factor = 1.61133 × 6.206053 = 10.000 in/s`. Mic channel uses psi units with its own range (still unresolved).
**Known decoder issue — fi==9 hardcoded skip:**
@@ -1248,8 +1349,8 @@ Fields visible in the Blastware "Compliance Setup" dialog. ✅ = byte offset co
| Field | Values / Type | Status |
|---|---|---|
| Recording Mode | Continuous / Single Shot / Histogram | ❓ |
| Record Stop Mode | Fixed Record Time / Auto / Manual Stop | ❓ |
| Recording Mode | Single Shot (`0x00`) / Continuous (`0x01`) / Histogram (`0x03`) / Histogram+Continuous (`0x04`) | ✅ `recording_mode` — write: `cfg[anchor3]`; read E5 sf1: `data[anchor4]` — confirmed 2026-04-20 |
| Record Stop Mode | Fixed Record Time / Auto / Manual Stop | ❓ Hint: `data[40]` in E5 sf1 changed `01 7F``00 00` alongside Continuous → Single Shot; may be related but unconfirmed independently |
| Sample Rate | Standard 1024 / Fast 2048 / Faster 4096 sps | ✅ `sample_rate` (anchor2) |
| Record Time | float, seconds (3, 5, 8, 10, 13…) | ✅ `record_time` (anchor+10) |
| Histogram Interval | 5 / 15 / 30 / 60 min (mode-gated behind Histogram mode) | ❓ |
@@ -1258,7 +1359,8 @@ Fields visible in the Blastware "Compliance Setup" dialog. ✅ = byte offset co
| Geophone — Enable all | bool | ❓ |
| Geophone — Trigger Source | bool | ❓ |
| Chan 1-3 Trigger Level | float, in/s | ✅ `trigger_level_geo` |
| Chan 1-3 Maximum Range | Normal 10.000 / 1.25 in/s | ✅ `max_range_geo` |
| Chan 1-3 Maximum Range (range selector) | Normal 10.000 / 1.25 in/s | ✅ `geo_range` uint8 — **CONFIRMED 2026-04-20.** Offset = Tran+33 (same in E5 read and SUB 71 write — 2126-byte buffer is round-tripped verbatim). `0x00`=Normal 10 in/s, `0x01`=Sensitive 1.25 in/s. Applied to Tran/Vert/Long. **`Tran+20` is NOT this field** (constant 0x01 on all captures). |
| Chan 1-3 ADC Scale Factor | 6.206053 (in/s)/V | ✅ `geo_adc_scale` float32 — **CONFIRMED 2026-04-17.** Offset = Tran+28 (same in E5 read and SUB 71 write). Inverse sensitivity = 1/0.161133. Interface Handbook §4.5: 1.61133 V × 6.206053 = 10.000 in/s. Hardware constant — do NOT write. |
| Microphone — Enable all | bool | ❓ |
| Microphone — Trigger Source | bool | ❓ |
| Chan 4 Trigger Level | float, dB or psi | ❓ |
@@ -1385,6 +1487,188 @@ Contains serial number, firmware bytes, and floating-point calibration fields. F
---
## 7.11 Erase-All Protocol (SUBs 0xA3 / 0xA2 / 0x06) ✅ 2026-04-11
> ✅ **Confirmed 2026-04-11** from MITM capture of a live Blastware ACH session
> (`bridges/captures/mitm/ach_mitm_20260411_001912/`).
Blastware uses a 4-step sequence to erase all stored events from device memory.
All frames use standard `build_bw_frame` format (NOT write-format).
### 7.11.1 Wire Sequence
```
BW → device: SUB 0xA3 offset=0x0000 params=00 00 00 00 00 00 00 FE 00 00
device → BW: SUB 0x5C (begin-erase ack)
BW → device: SUB 0x1C offset=0x0000 params=00 00 00 00 00 00 00 FE 00 00 (probe)
device → BW: SUB 0xE3 (probe ack)
BW → device: SUB 0x1C offset=0x002C params=(same) (data)
device → BW: SUB 0xE3 (44-byte monitor status response)
BW → device: SUB 0x06 offset=0x0000 params=00 00 00 00 00 00 00 FE 00 00 (probe)
device → BW: SUB 0xF9 (probe ack)
BW → device: SUB 0x06 offset=0x0024 params=(same) (data)
device → BW: SUB 0xF9 (36-byte storage range response)
BW → device: SUB 0xA2 offset=0x0000 params=00 00 00 00 00 00 00 FE 00 00
device → BW: SUB 0x5D (confirm-erase ack — device memory is now cleared)
```
All response SUBs follow the standard formula `0xFF request_SUB`. No exceptions.
The `token=0xFE` at `params[7]` is required for 0xA3, 0x06, and 0xA2.
### 7.11.2 SUB 0x06 Storage Range Response (36 bytes)
The 36-byte response from the data step ends with two 4-byte event keys:
| Offset (from response end) | Field | Notes |
|---|---|---|
| `[-8:-4]` | First stored event key | e.g. `0111ea60` before erase |
| `[-4:]` | Last stored event key | e.g. `0111eaa6` before erase |
After a successful erase:
- Both keys read `01110000` (device-empty sentinel)
- The device's internal event counter has reset
Example pre-erase: `... 0111ea60 0111eaa6`
Example post-erase: `... 01110000 01110000`
### 7.11.3 Post-Erase Key Counter Reset
After a successful erase the device resets its event counter. New events start
from key `0x01110000` — the same key as the very first event ever recorded on
the device. This means:
- Any system using event keys for deduplication must clear its "seen keys" state
after an erase, or risk treating fresh events as already downloaded.
- Detection heuristic: if `max(device_keys) < historical_max_key`, the counter
was reset. All device keys should be treated as new regardless of prior state.
The `ach_server.py` implementation stores `max_downloaded_key` in `ach_state.json`
and applies this heuristic on every call-home.
### 7.11.4 Implementation Notes
- `MiniMateClient.delete_all_events()` in `client.py` orchestrates the full sequence.
- `MiniMateProtocol` exposes `begin_erase_all()`, `confirm_erase_all()`, and
`read_event_storage_range()` as separate methods.
- The ACH server `--clear-after-download` flag calls `delete_all_events()` after a
successful event download and resets `ach_state.json` state for the unit.
---
### 7.12 Auto Call Home Config (SUB 0x2C / 0x7E / 0x7F) — ✅ CONFIRMED 2026-04-20
> Confirmed from `bridges/captures/4-20-26/call home settings/` — 10 BW TX write frames
> diffed against the S3 read payload. Accessible in Blastware via Remote Access → Setup Unit.
#### 7.12.1 Read Protocol — SUB 0x2C → Response 0xD3
Standard two-step read:
| Step | Offset | Purpose |
|---|---|---|
| Probe | `0x0000` | Get ack (no data returned) |
| Data | `0x007C` (124) | Receive 125-byte raw payload |
`DATA_LENGTHS[SUB_CALL_HOME] = 0x7C`
The raw payload is accessed as `data_rsp.data[11:]` — this is 125 bytes (not 124) because
the device returns logical value 0x03 (num_retries=3) as the two-byte wire sequence
`\x10\x03`. S3FrameParser is in `STATE_IN_FRAME` when it sees `0x10`, transitions to
`STATE_AFTER_DLE`, and then on `0x03` (ETX qualifier) it would normally end the frame —
but in the `_IN_FRAME_DLE` state it instead appends **both** the `0x10` and the `0x03`
literally to the payload. The result: `raw[117] = 0x10`, `raw[118] = 0x03`, and all
subsequent fields are shifted +1 from their logical positions.
#### 7.12.2 Raw Payload Field Map (125 bytes, from `data_rsp.data[11:]`)
> All offsets are into the 125-byte raw array. Offsets ≥ 119 are shifted +1 from logical
> due to the DLE-escaped 0x03 at raw[117:119].
| Raw Offset | Field | Type | Notes |
|---|---|---|---|
| `[5]` | `auto_call_home_enabled` | uint8 | `0x00` = disabled, `0x01` = enabled |
| `[6:46]` | `dial_string` | ASCII | 40-byte null-padded, e.g. `"12345"` or phone number |
| `[87]` | `after_event_recorded` | uint8 | `0x00` = off, `0x01` = on |
| `[91]` | `at_specified_times` | uint8 | `0x00` = off, `0x01` = on |
| `[93]` | `time1_enabled` | uint8 | `0x00` = off, `0x01` = on |
| `[101]` | `time1_hour` | uint8 | 023 |
| `[102]` | `time1_min` | uint8 | 059 |
| `[95]` | `time2_enabled` | uint8 | `0x00` = off, `0x01` = on |
| `[105]` | `time2_hour` | uint8 | 023 |
| `[106]` | `time2_min` | uint8 | 059 |
| `[117]` | DLE prefix `0x10` | — | Part of `\x10\x03` wire encoding for num_retries value 3 |
| `[118]` | `num_retries` (value = 3) | uint8 | Logical value 0x03; check `raw[117] == 0x10` to detect DLE prefix |
| `[120]` | `time_between_retries_sec` | uint8 | Shift +1 from logical 119 |
| `[122]` | `wait_for_connection_sec` | uint8 | Shift +1 from logical 121 |
| `[124]` | `warm_up_time_sec` | uint8 | Shift +1 from logical 123 |
**Unconfirmed fields** (offsets not yet mapped from captures):
- Time slots 3 and 4 (if they exist — Blastware UI only shows 2 time slots in observed sessions)
- `modem_power_relay_enabled` (bool)
- `storage_mode` (call home trigger on all events vs. triggered only?)
#### 7.12.3 DLE-Escaped 0x03 — Critical Detail
The `\x10\x03` sequence at raw[117:119] is **not** a DLE stuffing artifact in the usual
sense. Standard DLE stuffing escapes `\x10``\x10\x10`. But here the device is encoding
the integer value `3` in a position where the byte `\x03` would be indistinguishable from
the frame ETX terminator. The device therefore sends `\x10\x03` (DLE + ETX = "inner-frame
terminator" in S3 inner-frame syntax). S3FrameParser correctly handles this: in
`STATE_AFTER_DLE`, seeing `\x03` (ETX) while **inside** an outer frame causes it to
append both `\x10` and `\x03` as literal bytes rather than ending the frame. The outer
frame only terminates on a **bare** `\x03` (without the DLE prefix).
The write frame sends these bytes verbatim — the device accepts `\x10\x03` in the write
payload and interprets it as the value 3. No transformation is needed in
`_encode_call_home_config()`.
**Limitation:** Any field that needs to encode the value `3` (0x03) requires this DLE
prefix. The current encoder raises `ValueError` if any hour or minute field equals 3,
since the encoder does not yet implement DLE-prefixed writes for arbitrary field positions.
In practice, 3:00 AM / 3 minutes past are unlikely scheduled call times.
#### 7.12.4 Write Protocol — SUB 0x7E → 0x7F
Write format (same as other write commands — only BW_CMD `0x10` doubled on wire;
all other bytes written raw; DLE-aware checksum):
| Step | SUB | Payload | Offset | Response |
|---|---|---|---|---|
| Data write | `0x7E` | 127 bytes (125-byte read payload + `\x00\x00`) | `data[1]+2 = 0x7E` (126) | `0x81` |
| Confirm | `0x7F` | empty | `0x00` | `0x80` |
**Write payload construction:**
```python
write_payload = bytearray(raw_125_bytes)
write_payload.append(0x00)
write_payload.append(0x00)
# patch fields in-place, then pass bytes(write_payload) to build_bw_write_frame
```
**Offset formula:** `write_payload[1] = 0x7C` (same as DATA_LENGTH).
`offset = write_payload[1] + 2 = 0x7C + 2 = 0x7E = 126`.
This follows the identical pattern as SUB 0x68 (event index write) and SUB 0x69 (waveform write).
**No preceding 0x2C read required** — Blastware sends SUB 0x7E directly using cached
state. The `seismo-relay` implementation always reads first (`get_call_home_config()`)
before writing for safety.
#### 7.12.5 Implementation Notes
- `MiniMateProtocol.read_call_home_config()` — standard two-step read; returns `data_rsp.data[11:]` (125 bytes raw)
- `MiniMateProtocol.write_call_home_config(data)` — sends SUB 0x7E (127-byte payload) then SUB 0x7F confirm
- `MiniMateClient.get_call_home_config()``CallHomeConfig` dataclass
- `MiniMateClient.set_call_home_config(...)` — reads current config, patches via `_encode_call_home_config()`, writes back
- `_decode_call_home_config(raw)` — handles DLE prefix detection at raw[117]
- `_encode_call_home_config(raw, ...)` — patches in-place, appends 2 trailing zeros; raises `ValueError` if any hour/min == 3
- REST API: `GET /device/call_home` and `POST /device/call_home` in `sfm/server.py`
- Web UI: "Call Home" tab in `sfm/sfm_webapp.html`
---
## 8. Timestamp Format
Two timestamp wire formats are used:
@@ -1775,7 +2059,7 @@ The TCP port is **user-configurable** in both Blastware and the modem. There is
---
### 14.6 ACH Session Lifecycle (Call Home Mode — Future)
### 14.6 ACH Session Lifecycle (Call Home Mode) ✅ IMPLEMENTED 2026-04-11
When the unit calls home under ACH, the session lifecycle from the unit's perspective is:
@@ -1784,10 +2068,28 @@ When the unit calls home under ACH, the session lifecycle from the unit's perspe
3. Unit waits for "Wait for Connection" window for first BW frame from server
4. Server sends POLL_PROBE → unit responds with POLL_RESPONSE (same as serial)
5. Server reads serial number, full config, events as needed
6. Server disconnects (or unit disconnects on Serial Idle Time expiry)
7. Unit powers modem down, returns to monitor mode
6. (Optional) Server erases device memory: SUB 0xA3 → 0x1C → 0x06 → 0xA2
7. Server disconnects (or unit disconnects on Serial Idle Time expiry)
8. Unit detects DCD/DTR going low (modem signals line drop), returns to monitor mode automatically
Step 4 onward is **identical to the serial/call-up protocol**. The only difference from our perspective is that we are the **listener** rather than the **connector**. A future `AchServer` class will accept the incoming TCP connection and hand the socket to `TcpTransport` for processing.
Step 4 onward is **identical to the serial/call-up protocol**. The only difference
from our perspective is that we are the **listener** rather than the **connector**.
**Implementation: `bridges/ach_server.py`** — run with `python bridges/ach_server.py`.
Key flags:
- `--clear-after-download` — erase device memory after a successful event download
- `--allow-ip IP` — restrict to specific unit IPs
- `--max-events N` — cap events per session for safety
**State persistence: `ach_state.json`** — tracks `downloaded_keys` (set of event key
hex strings) and `max_downloaded_key` (high-water mark) per unit serial number.
Post-erase key reuse (`0x01110000` recycled) is detected via the high-water mark.
**Note on DCD/DTR:** The MiniMate Plus monitors the RS-232 DCD line. When the TCP
connection closes, the Sierra Wireless modem drops DCD, which the unit interprets as
"serial connection ended" and automatically resumes monitoring. No `start_monitoring()`
(SUB 0x96) command is needed from the server. ⚠️ Newer RV55 firmware may not assert DCD
by default — known issue, not yet resolved.
---
@@ -1835,11 +2137,16 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
| **Auxiliary Trigger read location****RESOLVED:** SUB `FE` offset `0x0109`, uint8, `0x00`=disabled, `0x01`=enabled. Confirmed 2026-03-11 via controlled toggle capture. | RESOLVED | 2026-03-02 | Resolved 2026-03-11 |
| **Auxiliary Trigger write path** — Write command not yet captured in a clean session. Inner frame handshake visible in A4 (multiple WRITE_CONFIRM_RESPONSE SUBs appear, TRIGGER_CONFIG_RESPONSE removed), but the BW→S3 write command itself was in a partial session. Likely SUB `15` or similar. Deferred for clean capture. | LOW | 2026-03-11 | NEW |
| ~~**SUB `6E` response to SUB `1C`**~~~~RESOLVED 2026-04-08: This was a misidentification.~~ The `1C → 6E` "exception" was misread — likely an inner A4 sub-frame. Confirmed from 4-8-26 capture (338 frames): SUB 0x1C always → 0xE3. No exceptions to the `0xFF SUB` rule are known. | RESOLVED | 2026-04-08 | CLOSED |
| **Max Geo Range float 6.2061 in/s** — NOT a user-selectable range (manual only shows 1.25 and 10.0 in/s). Likely internal ADC full-scale constant or hardware range ceiling. Not worth capturing. | LOW | 2026-02-26 | Downgraded 2026-03-02 |
| ~~**Max Geo Range float 6.2061**~~**RESOLVED 2026-04-17.** Confirmed as the **ADC-to-velocity scale factor** = inverse sensitivity = 1/0.161133 = **6.206053 (in/s)/V**. Source: Interface Handbook §4.5 formula `Range = 1.61133 V / Sensitivity`. For standard Instantel geo at Normal (Gain=1) range: Sensitivity = 1.61133/10 = 0.161133 V/(in/s), scale = 6.206053. Firmware: `PPV (in/s) = ADC_voltage × 6.206053`. The earlier ~9× overread was from using 6.206053 directly as range instead of as scale factor (range = 1.61133 V × 6.206053 = 10.000 in/s). Hardware constant — do NOT write. | RESOLVED | 2026-02-26 | Resolved 2026-04-17 |
| MicL channel units — **RESOLVED: psi**, confirmed from `.set` file unit string `"psi\0"` | RESOLVED | 2026-03-01 | |
| Backlight offset — **RESOLVED: +4B in event index data**, uint8, seconds | RESOLVED | 2026-03-02 | |
| Power save offset — **RESOLVED: +53 in event index data**, uint8, minutes | RESOLVED | 2026-03-02 | |
| Monitoring LCD Cycle — **RESOLVED: +54/+55 in event index data**, uint16 BE, seconds (65500 = disabled) | RESOLVED | 2026-03-02 | |
| **SUB 0x06 purpose — RESOLVED: event storage range.** Previously labeled "CHANNEL CONFIG READ". 4-11-26 MITM capture confirms it returns first/last stored event keys in the final 8 bytes of the 36-byte response. Used by Blastware as part of the erase-all verification step. | RESOLVED | 2026-04-11 | |
| **Erase-all command sequence — RESOLVED.** SUB 0xA3 (begin) + 0x1C (monitor status) + 0x06 (storage range) + 0xA2 (confirm). Confirmed from 4-11-26 MITM capture. All frames standard `build_bw_frame`, token=0xFE. | RESOLVED | 2026-04-11 | |
| **ACH inbound server — RESOLVED.** `bridges/ach_server.py` implements full inbound ACH pipeline. `--clear-after-download` flag for delete-after-upload workflow. Post-erase key-reuse detection via `max_downloaded_key` high-water mark. | RESOLVED | 2026-04-11 | |
| **Sensor Check dropdown byte location** — byte offset in 1A compliance config payload for the "Sensor Check: Before monitoring / After each event / Disabled" setting is NOT YET LOCATED. Confirmed: unit always runs with "Before monitoring" set. Need a capture with "Disabled" to diff. | MEDIUM | 2026-04-08 | Still open |
| **RV55 DCD/DTR default** — newer Sierra Wireless RV55 firmware does not assert DCD/DTR by default, so the MiniMate Plus never detects TCP disconnect and stays idle instead of resuming monitoring. Root cause: RV55 ACEmanager `DCD Control` setting. Workaround not yet found. | MEDIUM | 2026-04-11 | Still open |
---
@@ -1859,10 +2166,11 @@ The `.bin` files produced by `s3_bridge` are **not raw wire bytes**. The logger
| Trigger Level (Mic) | §3.8.6 | Channel block, float | float32 BE | 100148 dB in 1 dB steps |
| Alarm Level (Mic) | §3.9.10 | Channel block, float | float32 BE | higher than mic trigger |
| Record Time | §3.8.9 | cfg anchor+10, float32 BE (wire); `.set` +16, uint32 LE (file) | float32 BE (wire) | 1105 s; confirmed 3→`40400000`, 5→`40A00000`, 8→`41000000`, 13→`41500000`. Use anchor §7.6.1/§7.6.3 — NOT fixed offset. |
| Max Geo Range | §3.8.4 | Channel block, float | float32 BE | 1.25 or 10.0 in/s (user); 6.2061 in protocol = internal constant |
| ADC Scale Factor (geo_adc_scale) | §3.8.4 / Interface Handbook §4.5 | Channel block, Tran+28 (same in E5 read and SUB 71 write), float32 BE | float32 BE = 6.206053 | ✅ CONFIRMED 2026-04-17 — inverse sensitivity (in/s)/V. `Range = 1.61133 V × 6.206053 = 10.000 in/s`. Firmware: `PPV (in/s) = ADC_voltage × 6.206053`. Hardware constant, identical on all units. Do NOT write. |
| Max Geo Range (geo_range) | §3.8.4 | Channel block, Tran+33 (same in E5 read and SUB 71 write), uint8; applied to Tran/Vert/Long | uint8 | ✅ CONFIRMED 2026-04-20 — `0x00`=Normal 10.000 in/s, `0x01`=Sensitive 1.250 in/s. **NOTE: `Tran+20` reads `0x01` on ALL captures regardless of range — it is NOT this field.** |
| Microphone Units | §3.9.7 | Inline unit string | char[4] | `"psi\0"`, `"pa.\0"`, `"dB\0\0"` |
| Sample Rate | §3.8.2 | cfg anchor2, uint16 BE — anchor=`\x01\x2c\x00\x00\xbe\x80\x00\x00\x00\x00` in cfg[40:100] | uint16 BE | Normal=1024, Fast=2048, Faster=4096 ✅ CONFIRMED 2026-04-01 (BE11529 S338.17). Anchor required — see §7.6.3 DLE jitter. |
| Record Mode | §3.8.1 | Unknown | — | Single Shot, Continuous, Manual, Histogram, Histogram Combo |
| Record Mode | §3.8.1 | Write: `cfg[anchor3]`, uint8. Read (E5 sf1): `data[anchor4]`, uint8. Note: extra `0x10` byte at read `data[anchor3]` shifts offset by 1 vs write. | uint8 | `0x00`=Single Shot, `0x01`=Continuous, `0x02`=unknown, `0x03`=Histogram, `0x04`=Histogram+Continuous. ✅ CONFIRMED 2026-04-20 |
| Trigger Sample Width | §3.13.1h | BW→S3 SUB `0x82` write frame, destuffed `[22]`, uint8 | uint8 | Default=2; confirmed 4=`0x04`, 3=`0x03`. **BW-side write only** — not visible in S3 compliance reads. Mode-gated: only sent in Compliance/Single-Shot/Fixed mode. |
| Auto Window | §3.13.1b | **Mode-gated — NOT YET MAPPED** | uint8? | 19 seconds; only active when Record Stop Mode = Auto. Capture in Fixed mode produced no wire change. |
| Auxiliary Trigger | §3.13.1d | SUB `FE` (FULL_CONFIG_RESPONSE) offset `0x0109` (read); write path not yet isolated | uint8 (bool) | `0x00`=disabled, `0x01`=enabled; confirmed 2026-03-11 |
@@ -1965,6 +2273,279 @@ Semantic Interpretation <- settings, events, responses
---
---
## Appendix D — Blastware Binary File Formats (.N00 / .MLG / others)
> ✅ CONFIRMED 2026-04-21 — all fields verified by binary diff of reconstructed vs reference
> files from the 4-3-26-multi_event capture (M529LIY6.N00, BE11529.MLG).
>
> ⚠️ EXTENSION MAPPING REFUTED 2026-04-21 — earlier assumption that extension encodes
> recording mode is **FALSE**. A continuous-mode event produced `.EI0`, not `.9T0`.
> Extension encoding algorithm is unknown. Do not use extension to infer recording mode.
### D.1 Common File Header (22 bytes)
All Blastware files (regardless of type) share an 18-byte prefix followed by a 4-byte type tag.
| Offset | Length | Value | Description |
|---|---|---|---|
| 0x00 | 6 | `10 00 01 80 00 00` | Fixed prefix |
| 0x06 | 10 | `Instantel\x00` | ASCII string |
| 0x10 | 2 | `07 2c` | Fixed suffix |
| 0x12 | 4 | varies | File type tag (see below) |
**Total header: 22 bytes.**
**Type tags:**
| Extension | Type tag | Description |
|---|---|---|
| `.N00` | `00 12 03 00` | Waveform event (confirmed) |
| `.9T0` | `00 12 03 00` | Waveform event — same type tag as .N00 (assumed; not independently confirmed) |
| `.EI0` | `00 12 03 00` | Waveform event — same type tag (assumed; continuous-mode event observed 2026-04-21) |
| `.MLG` | `22 01 0e a0` | Monitor log |
**Extension encoding — new firmware (V10.72+) FULLY DECODED (confirmed 2026-04-22):**
The extension differs depending on how the file was saved:
| Download method | Extension format | Example |
|---|---|---|
| Manual / direct (Blastware connected to unit) | `AB0` (3 chars) | `.CE0` |
| Call-home / ACH | `AB0W` or `AB0H` (4 chars) | `.CE0H` |
Where:
- `AB` = 2-char base-36 of `total_seconds % 1296`; `A = value // 36`, `B = value % 36`
- `total_seconds = (event_local_time 1985-01-01T00:00:00_local)` in seconds
- `0` = always literal digit zero
- `W` = Full Waveform, `H` = Full Histogram (ACH only)
Base-36 alphabet: `09` = 09, `AZ` = 1035.
The 10-year production archive contains only ACH files (all end in W or H). Manual Blastware downloads produce the same `AB0` prefix but without the trailing type character.
**3-day cycle property (confirmed 2026-04-22):** A unit recording at a fixed daily time cycles through exactly **3 different extensions** with a 3-day period. Each calendar day shifts `total_seconds % 1296` by 864 (since `86400 % 1296 = 864`). The cycle repeats every 3 days because `gcd(1296, 864) = 432`. Confirmed from archive: top 3 extensions `CE0H` (95), `0E0H` (93), `OE0H` (91) are the 3-day cycle of a 06:00:14 daily call-in (seconds-in-window = 446, 14, 878).
**B character invariance:** `864 = 24 × 36`, so adding one day never changes `value % 36` — the second extension character is invariant for a fixed daily recording time. Only the first character cycles through 3 values.
**Old firmware (S338):** 3-char extensions observed (`.N00`, `.EI0`, etc.) — may simply be manual downloads under the same AB0 scheme, or a different encoding. Not yet confirmed.
**Micromate Series 4** uses a different extension format (observed: `IDFH`, `IDFW`). This formula does NOT apply to Micromate units.
All waveform files share the same `00 12 03 00` type tag regardless of extension. Blastware identifies file type by extension, not by type tag alone.
### D.2 Timestamp Encoding (Blastware files)
All timestamps in N00 and MLG files use an **8-byte big-endian format**:
| Byte | Field |
|---|---|
| 0 | day (uint8) |
| 1 | month (uint8) |
| 23 | year (uint16 BE) |
| 4 | `0x00` (reserved) |
| 5 | hour (uint8) |
| 6 | minute (uint8) |
| 7 | second (uint8) |
Example: `01 04 07 ea 00 00 1c 08` → April 1, 2026, 00:28:08.
Note: this differs from the 8-byte protocol timestamp (`[day][sub_code][month][year_HI][year_LO][0x00][hour][min][sec]` = 9 bytes) used in the device's on-wire 0C waveform records. The file format uses a compact 8-byte layout without the `sub_code` byte.
### D.3 N00 File Format — Single-Shot Waveform Event
**File layout:** `[22B header] [21B STRT record] [body bytes] [26B footer]`
#### D.3.1 STRT Record (21 bytes)
The STRT record immediately follows the 22-byte header.
| Offset | Length | Field | Notes |
|---|---|---|---|
| 0 | 4 | `STRT` | ASCII literal |
| 4 | 2 | `ff fe` | Fixed |
| 6 | 4 | event key (key4) | 4-byte waveform key |
| 10 | 4 | device-specific | NOT a repeat of key4 — device-internal field |
| 14 | 6 | device-specific | NOT zero-padded — device-internal fields |
| 20 | 1 | rectime | uint8 seconds |
**Critical:** The STRT record must be copied verbatim from A5[0].data[7+strt_pos:] — bytes [10:20] contain device-specific values that cannot be reconstructed from protocol-level Event fields alone.
#### D.3.2 Body Bytes (variable)
The body is reconstructed from the raw A5 bulk waveform stream frames by stripping DLE framing markers and taking the appropriate slice of each frame's data section.
**Per-frame contribution (from `frame.data`):**
| Frame | Skip amount | Notes |
|---|---|---|
| A5[0] (probe) | `7 + strt_pos_in_w0 + 21` | Skip frame.data prefix + STRT record |
| A5[1] | 13 | 7-byte prefix + 6-byte first-chunk header |
| A5[2..N] | 12 | 7-byte prefix + 5-byte chunk header |
| Terminator (page_key=0x0000) | 11 | 7-byte prefix + 4-byte terminator header |
**DLE strip rule:** For each frame's contribution (`frame.data[skip:]`), strip any `0x10` byte immediately followed by `0x02`, `0x03`, or `0x04`. Only the `0x10` is stripped; the following byte is kept as payload.
**Split-pair edge case:** When `frame.data[-1] == 0x10` AND `frame.chk_byte ∈ {0x02, 0x03, 0x04}`, the S3FrameParser split a DLE+XX pair at the payload/checksum boundary. Reunite the bytes before stripping (`relevant + bytes([chk_byte])`), then always remove the trailing chk_byte from the result (`stripped[:-1]`) — chk_byte is the wire checksum, never payload.
**Body/footer split:** Accumulate all frame contributions (data frames + terminator) into `all_bytes`. Then:
- `body = all_bytes[:-26]` (variable length)
- `footer = all_bytes[-26:]` (always 26 bytes — extracted from terminator content)
#### D.3.3 Footer (26 bytes)
The footer terminates the N00 file. Its bytes come directly from the terminator A5 frame's inner content — do NOT reconstruct from event metadata.
| Offset | Length | Field | Notes |
|---|---|---|---|
| 0 | 2 | `0e 08` | Fixed marker |
| 2 | 8 | ts1 | Start timestamp (8B big-endian) |
| 10 | 8 | ts2 | Stop timestamp (8B big-endian) |
| 18 | 6 | `00 01 00 02 00 00` | Fixed |
| 24 | 2 | CRC | 2-byte CRC — algorithm unconfirmed |
**CRC:** The 2-byte CRC at footer[24:26] has an unconfirmed algorithm. In M529LIY6.N00 it reads `fe da`. Attempts to match CRC-16/CCITT, CRC-16/IBM, CRC-32 (truncated), and 40+ polynomial/init combinations all failed. The writer copies it verbatim from the terminator frame.
### D.4 MLG File Format — Monitor Log
**File layout:** `[308B header] [N × 292B records]`
#### D.4.1 MLG Header (308 bytes)
| Offset | Length | Field | Notes |
|---|---|---|---|
| 0x00 | 22 | common header | prefix + `22 01 0e a0` type tag |
| 0x16 | 16 | unknown | observed as zeros in BE11529.MLG |
| 0x2A | 8 | serial number | null-padded ASCII (e.g. `"BE11529"`) |
| 0x32 | remainder | zero pad | pads to 308 bytes total |
#### D.4.2 MLG Record (292 bytes each)
| Offset | Length | Field | Notes |
|---|---|---|---|
| 0 | 2 | CRC | 2-byte CRC — algorithm unconfirmed; write as `00 00` |
| 2 | 4 | `22 01 0e 80` | Record marker |
| 6 | 8 | ts1 | Start timestamp (8B big-endian) |
| 14 | 8 | ts2 | Stop timestamp (8B big-endian); zeros if no stop |
| 22 | 4 | flags | Record type flags (see below) |
| 26 | 10 | serial | Null-padded ASCII serial number |
| 36 | variable | text | Type-dependent content |
| — | remainder | zero pad | pads to 292 bytes total |
**Record flags:**
| Value | Meaning |
|---|---|
| `ff ff 00 00` | Monitoring start with no stop recorded |
| `01 00 02 00` | Triggered event (has ts1 + ts2) |
| `02 00 00 00` | Monitoring interval (has ts1 + ts2) |
**Text content for triggered events (`flags = 01 00 02 00`):**
| Byte | Field |
|---|---|
| 0 | `0x08` |
| 18 | ts1 copy (8B big-endian) |
| 9+ | `"Geo: X.XXX in/s\x00"` ASCII geo threshold |
#### D.4.3 MLG CRC
The 2-byte CRC at record[0:2] uses an unconfirmed algorithm. Tested against CRC-16/CCITT, CRC-16/IBM, CRC-32 (truncated), word sums, XOR variants, and 40+ polynomial/init combinations — none matched. The writer emits `00 00`. Blastware may reject files with incorrect CRCs (impact on import unknown — TODO: test).
### D.5 Filename Encoding ✅ PARTIALLY CONFIRMED 2026-04-22
Blastware assigns waveform filenames of the form `<prefix_letter><serial3><stem><ext>`, where:
#### D.5.1 Serial Prefix ✅ CONFIRMED 2026-04-22
The first 4 characters of the filename encode the full device serial number:
```
prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))
serial3 = f"{serial_numeric % 1000:03d}" (last 3 digits, zero-padded)
```
Where `serial_numeric` is the integer after the "BE" device-type prefix.
Examples (all confirmed from archive):
| Serial | serial_numeric / 1000 | prefix_letter | serial3 | Filename prefix |
|--------|----------------------|---------------|---------|-----------------|
| BE6907 | 6 | H | 907 | H907 |
| BE7145 | 7 | I | 145 | I145 |
| BE11529 | 11 | M | 529 | M529 |
| BE14036 | 14 | P | 036 | P036 |
| BE17353 | 17 | S | 353 | S353 |
| BE18003 | 18 | T | 003 | T003 |
| BE18191 | 18 | T | 191 | T191 |
| BE18676 | 18 | T | 676 | T676 |
**Interpretation:** The prefix letter encodes the production generation (batch of 1000 units). B=generation 0 (serials 0999), C=generation 1 (10001999), etc. No units with prefix A have been observed — the earliest known units start around serial 2000+ (prefix D).
**Note:** The "BE" device-type prefix is implicit. The filename only encodes the numeric part of the serial. Other Instantel device types (Micromate, Blastmate) may use a different scheme.
#### D.5.2 Stem + Extension — full timestamp encoding ✅ FULLY CONFIRMED 2026-04-22
The stem (4 chars) and AB extension (2 chars) together form a 6-digit base-36 number encoding a complete second-resolution timestamp:
```python
total_seconds = stem_int * 1296 + ab_int
event_local_time = datetime(1985, 1, 1) + timedelta(seconds=total_seconds)
```
- **Epoch:** `1985-01-01 00:00:00` **device local time** ✅ CONFIRMED — verified against 3,248 files from a 10-year production archive; zero errors (only 2 mismatches were Micromate `IDFH`/`IDFW` files which use a completely different naming scheme)
- **Unit:** 1296 seconds = 36² ≈ 21.6 minutes per stem increment
- **Alphabet:** `"0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"` (digits then uppercase letters)
- **Collision:** Events within the same 21.6-minute window share a stem; extension distinguishes them
**Decoding example — `P036L318.C80H` (BE14036, Full Histogram):**
```
stem L318 = 21×36³ + 3×36² + 1×36 + 8 = 983,708
AB C8 = 12×36 + 8 = 440
total_sec = 983,708 × 1296 + 440 = 1,274,886,008
event_time = 1985-01-01 + 1,274,886,008s = 2025-05-26 15:00:08 local
```
**Note on local time:** The device's onboard clock is set to the local timezone of the deployment site. The epoch and all timestamps are in that same local time — there is no UTC conversion. Files moved between timezones will decode to the original deployment timezone.
#### D.5.3 Extension taxonomy
Third character of extension is always `'0'`. File type is identified by extension, not by the type tag in the header (all waveform extensions share type tag `00 12 03 00`).
| Extension | Recording mode | Sample rate | Status |
|---|---|---|---|
| `.N00` | Single Shot (0x00) | 1024 sps | ✅ CONFIRMED |
| `.9T0` | Continuous (0x01) | 1024 sps | ✅ CONFIRMED |
| `.490` | ? | ? | ❓ observed from M529LJ8V.490 |
| `.5K0` | ? | ? | ❓ observed from M529LJDY.5K0 |
| `.980` | ? | ? | ❓ observed from M529LJDY.980 |
| `.ML0` | ? | ? | ❓ observed from M529LJDY.ML0 (167s duration; possibly Histogram) |
**Why 5 extensions for "Continuous"?** Binary analysis of all 6 example files shows that `.9T0`, `.490`, `.5K0`, `.980`, `.ML0` are byte-for-byte identical in all metadata regions (compliance anchor block, channel descriptor blocks `Tran/Vert/Long/MicL`). The A5 frame 7 body reflects the **session-start** compliance config, not the per-event capture config. All 5 files show recording_mode=0x01 and sample_rate=1024 in the body. The extension must therefore encode the **capture-time** compliance state — likely a combination of recording mode, sample rate, and possibly mic units or other options. This cannot be determined from file body alone without capture-time compliance data from the 0C record sub_code and the actual waveform sample count.
**DLE-shift offset note for reading recording_mode from N00/9T0 body:**
The compliance block in the file body has been through `_strip_inner_frame_dles`. The 0x10 constant at logical `anchor7` (between recording_mode and sample_rate_HI) gets stripped when sample_rate_HI = `0x04` (1024 sps), because `0x10` precedes `0x04 ∈ {0x02,0x03,0x04}`. After stripping, the anchor shifts left by 1, so:
| 1024 sps (strip occurs) | 2048 or 4096 sps (no strip) |
|---|---|
| `file[anc7]` = recording_mode | `file[anc8]` = recording_mode |
| `file[anc6:anc4]` = sample_rate | `file[anc6:anc4]` = sample_rate |
For 1024 sps files, the expected file bytes around the anchor are:
```
file[anc9]: mode_prefix (0x00 for Single Shot/Continuous; 0x10 for Histogram)
file[anc8]: 0x00 (was recording_mode, but shifted away — now reads 0x00 for mode_prefix)
file[anc7]: recording_mode (0x00=Single Shot, 0x01=Continuous, etc.)
file[anc6]: 0x04 (sample_rate_HI for 1024 sps)
file[anc5]: 0x00 (sample_rate_LO)
file[anc4]: histogram_interval_HI
file[anc3]: histogram_interval_LO
```
---
*All findings reverse-engineered from live RS-232 bridge captures.*
*Cross-referenced from 2026-03-02 with Instantel MiniMate Plus Operator Manual (716U0101 Rev 15).*
*This is a living document — append changelog entries and timestamps as new findings are confirmed or corrected.*
+634
View File
@@ -0,0 +1,634 @@
#!/usr/bin/env python3
"""
experiments.py Protocol minimization experiments for MiniMate Plus.
Goal: figure out which steps in Blastware's sequences are truly required vs.
cargo-culted, so we can build a faster, smarter client.
Each experiment is self-contained (opens its own TCP connection) and reports
PASS / FAIL / INCONCLUSIVE with timing and notes.
Usage:
python experiments.py [--host IP] [--port PORT] [exp1 exp2 ...]
Run all: python experiments.py
Run specific: python experiments.py cold_status fast_event_count no_5a
Available experiments
---------------------
cold_status EXP1 Monitor status (1C) with NO prior POLL
fast_event_count EXP2 Event count via POLL+08 only skip identity reads
no_5a EXP3 Event record (0C) without bulk waveform stream (5A)
skip_1e EXP4 0A/0C directly with cached key skip initial 1E
fewer_polls EXP5 Only 1 POLL before 5A instead of Blastware's 3
compliance_only EXP6 Write compliance ONLY (71x372), skip event index+trigger+waveform
"""
from __future__ import annotations
import argparse
import logging
import struct
import sys
import time
from dataclasses import dataclass, field
from typing import Optional
logging.basicConfig(
level=logging.WARNING, # experiment output is via print(); set DEBUG for wire trace
format="%(asctime)s %(levelname)-7s %(name)-20s %(message)s",
datefmt="%H:%M:%S",
)
log = logging.getLogger("experiments")
# ── Imports ───────────────────────────────────────────────────────────────────
from minimateplus.transport import TcpTransport
from minimateplus.protocol import (
MiniMateProtocol,
ProtocolError,
TimeoutError as ProtoTimeout,
SUB_MONITOR_STATUS,
SUB_SERIAL_NUMBER,
SUB_FULL_CONFIG,
SUB_EVENT_INDEX,
SUB_COMPLIANCE,
SUB_WRITE_CONFIRM_A,
SUB_WRITE_CONFIRM_B,
)
from minimateplus.framing import build_bw_frame, SESSION_RESET
from minimateplus.client import (
MiniMateClient,
_decode_compliance_config_into,
_encode_compliance_config,
)
from minimateplus.models import DeviceInfo
DEFAULT_HOST = "63.43.212.232"
DEFAULT_PORT = 9034
# ── Result container ──────────────────────────────────────────────────────────
@dataclass
class Result:
name: str
outcome: str # "PASS" | "FAIL" | "INCONCLUSIVE"
elapsed: float = 0.0
notes: str = ""
details: dict = field(default_factory=dict)
def __str__(self) -> str:
sym = {"PASS": "", "FAIL": "", "INCONCLUSIVE": "⚠️ "}.get(self.outcome, "?")
lines = [f" {sym} {self.outcome:13s} {self.name} ({self.elapsed:.1f}s)"]
if self.notes:
lines.append(f" {self.notes}")
for k, v in self.details.items():
lines.append(f" {k}: {v}")
return "\n".join(lines)
# ── Connection helpers ────────────────────────────────────────────────────────
def connect_proto(host: str, port: int, timeout: float = 15.0) -> tuple[TcpTransport, MiniMateProtocol]:
"""Open a raw TCP connection and return (transport, proto) without any handshake."""
t = TcpTransport(host, port)
t.connect()
proto = MiniMateProtocol(t, recv_timeout=timeout)
return t, proto
def connect_client(host: str, port: int, timeout: float = 30.0) -> tuple[MiniMateClient, DeviceInfo]:
"""Open a MiniMateClient and run the full connect() handshake."""
transport = TcpTransport(host, port)
client = MiniMateClient(transport=transport, timeout=timeout)
client.open()
info = client.connect()
return client, info
# ── Experiment runner ─────────────────────────────────────────────────────────
def run(name: str, fn, *args, **kwargs) -> Result:
print(f"\n{''*60}")
print(f" Running: {name}")
print(f"{''*60}")
t0 = time.time()
try:
outcome, notes, details = fn(*args, **kwargs)
except Exception as exc:
outcome = "FAIL"
notes = f"Uncaught exception: {exc}"
details = {}
log.exception("Experiment %s raised:", name)
elapsed = time.time() - t0
r = Result(name=name, outcome=outcome, elapsed=elapsed, notes=notes, details=details)
print(str(r))
return r
# ══════════════════════════════════════════════════════════════════════════════
# EXP1 — Monitor status (1C) with NO prior POLL
# ══════════════════════════════════════════════════════════════════════════════
#
# Blastware always does a full POLL handshake before any other command.
# We want to know: can we query SUB 1C (battery, memory, monitoring state)
# cold, with only a SESSION_RESET signal and no POLL at all?
#
# If PASS: status checks become near-instant (no ~1s POLL round-trip).
# If FAIL: we need POLL first, but maybe we can cache it.
def exp_cold_status(host: str, port: int) -> tuple[str, str, dict]:
"""SUB 1C without any POLL — just SESSION_RESET + 1C probe + 1C data."""
t, proto = connect_proto(host, port)
try:
print(" Sending SESSION_RESET only (no POLL)")
t.write(SESSION_RESET)
time.sleep(0.1)
print(" Sending SUB 1C probe (no POLL first)…")
rsp_sub = (0xFF - SUB_MONITOR_STATUS) & 0xFF # 0xE3
t.write(build_bw_frame(SUB_MONITOR_STATUS, 0x00))
probe = proto._recv_one(expected_sub=rsp_sub, timeout=8.0)
print(f" 1C probe OK page_key=0x{probe.page_key:04X} data={probe.data.hex()}")
t.write(build_bw_frame(SUB_MONITOR_STATUS, 0x2C))
data_rsp = proto._recv_one(expected_sub=rsp_sub, timeout=8.0)
section = data_rsp.data
print(f" 1C data OK {len(section)} bytes hex: {section.hex()}")
# Decode battery + memory from the end of the section
details = {"raw_bytes": len(section)}
if len(section) >= 10:
batt_raw = struct.unpack_from(">H", section, len(section) - 10)[0]
mem_total = struct.unpack_from(">I", section, len(section) - 8)[0]
mem_free = struct.unpack_from(">I", section, len(section) - 4)[0]
is_monitoring = (section[1] == 0x10)
details["battery_v"] = f"{batt_raw / 100:.2f} V"
details["memory_total"] = f"{mem_total:,} bytes"
details["memory_free"] = f"{mem_free:,} bytes"
details["monitoring"] = is_monitoring
print(f" battery={batt_raw/100:.2f}V mem_free={mem_free:,} monitoring={is_monitoring}")
return "PASS", "SUB 1C responded without any POLL — cold status read works!", details
except ProtoTimeout:
return "FAIL", "Device did not respond to 1C without POLL (timeout)", {}
except ProtocolError as exc:
return "FAIL", f"Protocol error: {exc}", {}
finally:
t.disconnect()
# ══════════════════════════════════════════════════════════════════════════════
# EXP2 — Fast event count: POLL + SUB 08 only (skip identity reads)
# ══════════════════════════════════════════════════════════════════════════════
#
# Blastware's connect() does: POLL → 15 → 01 → 1A → 08
# We want to know: can we skip 15/01/1A and go straight from POLL to 08?
#
# Reading identity (15, 01) and full compliance (1A, ~2126 bytes over TCP)
# takes several seconds each connect. If we only need event count, skipping
# them would be a huge win.
#
# If PASS: fast status poll = POLL + 08 only (~2 round trips vs ~8+).
def exp_fast_event_count(host: str, port: int) -> tuple[str, str, dict]:
"""POLL startup → SUB 08 only, skip serial/config/compliance reads."""
t, proto = connect_proto(host, port)
try:
print(" Running startup (POLL only)…")
proto.startup()
print(" POLL OK — now reading SUB 08 (event index) directly…")
idx_raw = proto.read_event_index()
print(f" SUB 08 OK {len(idx_raw)} bytes")
# Try to decode event count from SUB 08 payload
# The raw block is 88 bytes; bytes [3:7] may be a count (uint32 BE)
details = {"idx_raw_len": len(idx_raw)}
if len(idx_raw) >= 7:
count_candidate = struct.unpack_from(">I", idx_raw, 3)[0]
details["count_candidate"] = count_candidate
print(f" idx[3:7] as uint32 BE = {count_candidate} (may or may not be event count)")
# Also verify we can read 1E without the identity reads having been done
print(" Reading 1E (event header) to confirm event access works…")
key4, data8 = proto.read_event_first()
is_empty = data8[4:8] == b"\x00\x00\x00\x00"
details["first_key"] = key4.hex()
details["is_empty"] = is_empty
print(f" 1E OK key={key4.hex()} empty={is_empty}")
return "PASS", "POLL+08+1E all work without identity reads (15/01/1A skipped)", details
except ProtocolError as exc:
return "FAIL", f"Protocol error: {exc}", {}
finally:
t.disconnect()
# ══════════════════════════════════════════════════════════════════════════════
# EXP3 — Get event record (0C) without bulk waveform stream (5A)
# ══════════════════════════════════════════════════════════════════════════════
#
# Blastware's event download = 1E → 0A → 1E-arm → 0C → 1F(dl) → POLL×3 → 5A → 1F(browse)
#
# The 5A bulk stream is the slow part (several large frames, ~1s+ per event).
# We only need 5A for: client, operator, seis_loc, notes (not in 0C).
# If you don't need those fields, can we do: 1E → 0A → 0C → 1F(browse) ?
#
# Two variants tested:
# 3a: Skip 1E-arm AND 5A — just 0A → 0C → 1F(browse)
# 3b: Include 1E-arm but skip 5A+POLL — 0A → 1E-arm → 0C → 1F(browse)
#
# If PASS: event peak values available without the slow bulk stream.
# If FAIL on 3a but PASS on 3b: 1E-arm required even without 5A.
def exp_no_5a(host: str, port: int) -> tuple[str, str, dict]:
"""Event record via 0A→0C without 5A or POLL×3. Tests both with and without 1E-arm."""
t, proto = connect_proto(host, port)
try:
print(" Startup (POLL)…")
proto.startup()
# Get the first event key via 1E
key4, data8 = proto.read_event_first()
if data8[4:8] == b"\x00\x00\x00\x00":
return "INCONCLUSIVE", "Device has no stored events — cannot test", {}
print(f" First event key: {key4.hex()}")
details: dict = {"key": key4.hex()}
# ── Variant 3a: 0A → 0C → 1F(browse), no 1E-arm ─────────────────────
print("\n [3a] 0A → 0C → 1F(browse) (NO 1E-arm, NO 5A)")
try:
_hdr, rec_len = proto.read_waveform_header(key4)
print(f" 0A OK rec_len=0x{rec_len:02X}")
record_3a = proto.read_waveform_record(key4)
print(f" 0C OK {len(record_3a)} bytes")
# Check for recognizable content
has_tran = b"Tran" in record_3a
has_vert = b"Vert" in record_3a
has_long = b"Long" in record_3a
print(f" 0C content check: Tran={has_tran} Vert={has_vert} Long={has_long}")
details["3a_0c_bytes"] = len(record_3a)
details["3a_has_peaks"] = has_tran and has_vert and has_long
# Now try browse 1F without any 5A
key4_next, data8_next = proto.advance_event(browse=True)
null_sentinel = data8_next[4:8] == b"\x00\x00\x00\x00"
print(f" 1F(browse) → key={key4_next.hex()} null={null_sentinel}")
details["3a_1f_ok"] = True
details["3a_outcome"] = "PASS"
except ProtocolError as exc:
print(f" 3a FAILED: {exc}")
details["3a_outcome"] = f"FAIL: {exc}"
# Try to recover by reconnecting for 3b
t.disconnect()
t2, proto2 = connect_proto(host, port)
proto2.startup()
key4, data8 = proto2.read_event_first()
if data8[4:8] == b"\x00\x00\x00\x00":
return "FAIL", f"3a failed and device empty on retry: {exc}", details
t, proto = t2, proto2
# ── Variant 3b: 0A → 1E-arm → 0C → 1F(browse), no 5A ───────────────
print("\n [3b] 0A → 1E-arm(0xFE) → 0C → 1F(browse) (NO POLL×3, NO 5A)")
try:
_hdr, rec_len = proto.read_waveform_header(key4)
print(f" 0A OK rec_len=0x{rec_len:02X}")
# 1E download-arm (token=0xFE) between 0A and 0C
proto.read_event_first(token=0xFE)
print(" 1E-arm OK")
record_3b = proto.read_waveform_record(key4)
print(f" 0C OK {len(record_3b)} bytes")
has_tran = b"Tran" in record_3b
print(f" 0C content check: Tran={has_tran} Vert={b'Vert' in record_3b}")
details["3b_0c_bytes"] = len(record_3b)
details["3b_has_peaks"] = has_tran
# Browse 1F without 5A / POLL×3
key4_next2, data8_next2 = proto.advance_event(browse=True)
null_sentinel2 = data8_next2[4:8] == b"\x00\x00\x00\x00"
print(f" 1F(browse) → key={key4_next2.hex()} null={null_sentinel2}")
details["3b_1f_ok"] = True
details["3b_outcome"] = "PASS"
except ProtocolError as exc:
print(f" 3b FAILED: {exc}")
details["3b_outcome"] = f"FAIL: {exc}"
# Summarize
a_ok = details.get("3a_outcome") == "PASS"
b_ok = details.get("3b_outcome") == "PASS"
if a_ok:
return "PASS", "3a: 0A→0C works with NO 1E-arm and NO 5A. Huge speedup possible!", details
elif b_ok:
return "PASS", "3b: 0A→1E-arm→0C works without 5A (1E-arm still needed before 0C)", details
else:
return "FAIL", "Both 3a and 3b failed — 5A may be required for device state", details
except ProtocolError as exc:
return "FAIL", f"Protocol error during setup: {exc}", {}
finally:
try:
t.disconnect()
except Exception:
pass
# ══════════════════════════════════════════════════════════════════════════════
# EXP4 — Skip initial 1E if we already know the event key
# ══════════════════════════════════════════════════════════════════════════════
#
# In Blastware, every session starts with 1E to discover the first key.
# But if we already fetched and cached the event keys from a previous session,
# can we skip 1E entirely and go straight to 0A(cached_key)?
#
# Practical use case: we poll the device every N minutes. We already know
# all the event keys from last time. On re-connect, can we go direct to 0A?
#
# If PASS: subsequent polls that don't add new events can skip 1E discovery.
def exp_skip_1e(host: str, port: int) -> tuple[str, str, dict]:
"""Get the first event key, disconnect, reconnect, go straight to 0A (skip 1E)."""
# Phase 1: get the key
t, proto = connect_proto(host, port)
try:
proto.startup()
key4, data8 = proto.read_event_first()
if data8[4:8] == b"\x00\x00\x00\x00":
return "INCONCLUSIVE", "No events stored — cannot test", {}
print(f" Phase 1: got event key = {key4.hex()}")
finally:
t.disconnect()
time.sleep(0.5)
# Phase 2: fresh connection, skip 1E, go straight to 0A with cached key
t2, proto2 = connect_proto(host, port)
try:
print(" Phase 2: fresh connection — startup + 0A directly (no 1E)")
proto2.startup()
_hdr, rec_len = proto2.read_waveform_header(key4)
print(f" 0A OK rec_len=0x{rec_len:02X}")
record = proto2.read_waveform_record(key4)
has_peaks = b"Tran" in record
print(f" 0C OK {len(record)} bytes has_peaks={has_peaks}")
details = {
"cached_key": key4.hex(),
"0c_bytes": len(record),
"has_peaks": has_peaks,
}
return "PASS", "0A works with cached key — 1E discovery can be skipped on known sessions", details
except ProtocolError as exc:
return "FAIL", f"0A failed with cached key (device needs 1E first?): {exc}", {"key": key4.hex()}
finally:
t2.disconnect()
# ══════════════════════════════════════════════════════════════════════════════
# EXP5 — Fewer POLLs before 5A (try POLL×1 instead of Blastware's POLL×3)
# ══════════════════════════════════════════════════════════════════════════════
#
# Blastware always sends 3 full POLL probe+data cycles between 1F and 5A.
# Each POLL is a round trip. Can we get away with just 1?
#
# WARNING: If POLL×1 fails, the device may be in a bad state. We try to
# recover with an extra POLL×2 and a fresh 5A attempt. Even on failure we
# try to leave the device in a usable state.
#
# Strategy: run the full event sequence up to 1F(download), then try 5A
# with only 1 POLL. If 5A responds → PASS. If timeout → try 2 more POLLs
# and check if the device recovers.
def exp_fewer_polls(host: str, port: int) -> tuple[str, str, dict]:
"""Full sequence to 1F, then only 1 POLL before 5A (Blastware does 3)."""
t, proto = connect_proto(host, port)
try:
proto.startup()
key4, data8 = proto.read_event_first()
if data8[4:8] == b"\x00\x00\x00\x00":
return "INCONCLUSIVE", "No events stored — cannot test", {}
print(f" Event key: {key4.hex()}")
# Full setup: 0A → 1E-arm → 0C → 1F(download)
_hdr, rec_len = proto.read_waveform_header(key4)
print(f" 0A OK rec_len=0x{rec_len:02X}")
proto.read_event_first(token=0xFE) # 1E-arm
print(" 1E-arm OK")
proto.read_waveform_record(key4)
print(" 0C OK")
arm_key4, _ = proto.advance_event(browse=False) # 1F(download) — arms 5A
print(f" 1F(download) OK arm_key={arm_key4.hex()}")
# Only 1 POLL (Blastware does 3)
print(" Sending 1 POLL (instead of 3)…")
proto.poll()
print(" POLL ok — now probing 5A…")
try:
frames = proto.read_bulk_waveform_stream(key4, stop_after_metadata=True, max_chunks=12)
print(f" 5A OK after 1 POLL — {len(frames)} frames received")
details = {"poll_count": 1, "frames": len(frames)}
return "PASS", "5A works with only 1 POLL (saved 2 round-trips per event)!", details
except ProtoTimeout:
print(" 5A timed out after 1 POLL — device needs more POLLs")
# Attempt recovery: send 2 more POLLs and see if 5A then works
print(" Attempting recovery: 2 more POLLs…")
try:
proto.poll()
proto.poll()
frames2 = proto.read_bulk_waveform_stream(key4, stop_after_metadata=True, max_chunks=12)
print(f" 5A worked after total 3 POLLs ({len(frames2)} frames)")
return "FAIL", "5A needs 3 POLLs — 1 is not enough (recovery confirmed 3 still works)", {
"poll_count_tried": 1, "recovery_polls": 3, "recovery_frames": len(frames2)
}
except ProtocolError as exc2:
return "FAIL", f"5A failed even after 3 total POLLs — device may need reconnect: {exc2}", {}
except ProtocolError as exc:
return "FAIL", f"Setup failed: {exc}", {}
finally:
t.disconnect()
# ══════════════════════════════════════════════════════════════════════════════
# EXP6 — Compliance-only write (71×3→72), skip event index + trigger + waveform
# ══════════════════════════════════════════════════════════════════════════════
#
# Blastware's full write sequence: 68→73 | 71×3→72 | 82→83 | 69→74→72
# We want to know: can we write ONLY the compliance block (71×3→72)?
#
# Test procedure:
# 1. Read current compliance config (SUB 1A)
# 2. Patch the "notes" field to a test marker
# 3. Write ONLY 71×3→72 (skip 68, 73, 82, 83, 69, 74, final 72)
# 4. Read back (SUB 1A) and verify the change was written
# 5. Restore original value
#
# If PASS: we can push individual config fields without touching event index,
# trigger config, or waveform data — huge simplification.
# If FAIL: the device needs the full write sequence (may reject partial write).
#
# SAFETY: We restore original data in a finally block. If the restore write
# fails, the device will have the test marker in "notes" — harmless but visible.
_EXP6_MARKER = "[exp6-test]"
def exp_compliance_only(host: str, port: int) -> tuple[str, str, dict]:
"""Write compliance block alone (71×3→72), verify, and restore."""
client, info = connect_client(host, port)
original_raw: Optional[bytes] = None
try:
proto = client._proto
if proto is None:
return "FAIL", "Could not get protocol handle from client", {}
# 1. Read current compliance
print(" Reading current compliance config (SUB 1A)…")
original_raw = proto.read_compliance_config()
print(f" Got {len(original_raw)} bytes of compliance config")
# Find current notes value for display
info_obj = DeviceInfo()
_decode_compliance_config_into(original_raw, info_obj)
cc = info_obj.compliance_config
orig_notes = cc.notes if cc else "(unknown)"
print(f" Current notes field: {orig_notes!r}")
# 2. Build modified payload with test marker in notes
test_notes = _EXP6_MARKER
modified_raw = _encode_compliance_config(
original_raw,
notes=test_notes,
)
print(f" Encoded modified compliance payload ({len(modified_raw)} bytes)")
print(f" Patching notes: {orig_notes!r}{test_notes!r}")
# 3. Write ONLY the compliance block: 71×3 → 72
print(" Writing compliance ONLY (71×3→72) — skipping 68/73/82/83/69/74…")
proto.write_compliance_config_raw(modified_raw)
print(" Write complete — device acked 71×3→72")
# 4. Read back and verify
print(" Reading back compliance config to verify…")
readback_raw = proto.read_compliance_config()
readback_info = DeviceInfo()
_decode_compliance_config_into(readback_raw, readback_info)
rb_cc = readback_info.compliance_config
readback_notes = rb_cc.notes if rb_cc else "(decode failed)"
print(f" Read-back notes: {readback_notes!r}")
write_worked = (readback_notes == test_notes)
print(f" Write verified: {write_worked}")
details = {
"original_notes": orig_notes,
"written_notes": test_notes,
"readback_notes": readback_notes,
"write_verified": write_worked,
}
if write_worked:
return "PASS", "Compliance-only write works! No event index or trigger writes needed.", details
else:
return "FAIL", f"Write was not reflected in read-back (got {readback_notes!r})", details
except ProtocolError as exc:
return "FAIL", f"Protocol error: {exc}", {}
finally:
# Restore original compliance data regardless of outcome
if original_raw is not None:
print(" Restoring original compliance config…")
try:
proto2 = client._proto
if proto2:
proto2.write_compliance_config_raw(
_encode_compliance_config(original_raw) # no-op patch = verbatim
)
print(" Restore complete")
else:
print(" WARNING: protocol handle gone — could not restore")
except Exception as exc_r:
print(f" WARNING: restore failed: {exc_r}")
client.close()
# ══════════════════════════════════════════════════════════════════════════════
# Registry + main
# ══════════════════════════════════════════════════════════════════════════════
EXPERIMENTS = {
"cold_status": ("EXP1", exp_cold_status, "Monitor status (1C) with no POLL"),
"fast_event_count": ("EXP2", exp_fast_event_count, "Event count via POLL+08, skip identity reads"),
"no_5a": ("EXP3", exp_no_5a, "Event record (0C) without bulk waveform (5A)"),
"skip_1e": ("EXP4", exp_skip_1e, "0A/0C with cached key — skip initial 1E"),
"fewer_polls": ("EXP5", exp_fewer_polls, "1 POLL before 5A instead of Blastware's 3"),
"compliance_only": ("EXP6", exp_compliance_only, "Compliance-only write (71×3→72), no other blocks"),
}
def main() -> None:
ap = argparse.ArgumentParser(description="MiniMate Plus protocol minimization experiments")
ap.add_argument("--host", default=DEFAULT_HOST)
ap.add_argument("--port", type=int, default=DEFAULT_PORT)
ap.add_argument("--debug", action="store_true", help="Enable DEBUG wire logging")
ap.add_argument("experiments", nargs="*",
help=f"Which to run (default: all). Choices: {', '.join(EXPERIMENTS)}")
args = ap.parse_args()
if args.debug:
logging.getLogger().setLevel(logging.DEBUG)
which = args.experiments or list(EXPERIMENTS.keys())
unknown = [e for e in which if e not in EXPERIMENTS]
if unknown:
print(f"Unknown experiments: {unknown}")
print(f"Available: {', '.join(EXPERIMENTS)}")
sys.exit(1)
print(f"\n{''*60}")
print(f" MiniMate Plus Protocol Minimization Experiments")
print(f" Target: {args.host}:{args.port}")
print(f" Running: {', '.join(which)}")
print(f"{''*60}")
results: list[Result] = []
for key in which:
tag, fn, desc = EXPERIMENTS[key]
label = f"{tag}: {desc}"
r = run(label, fn, args.host, args.port)
results.append(r)
time.sleep(1.5) # brief pause between experiments — let device settle
print(f"\n\n{''*60}")
print(" SUMMARY")
print(f"{''*60}")
for r in results:
sym = {"PASS": "", "FAIL": "", "INCONCLUSIVE": "⚠️ "}.get(r.outcome, "?")
print(f" {sym} {r.outcome:13s} {r.name}")
print(f"{''*60}")
passed = sum(1 for r in results if r.outcome == "PASS")
failed = sum(1 for r in results if r.outcome == "FAIL")
skipped = sum(1 for r in results if r.outcome == "INCONCLUSIVE")
print(f" {passed} passed {failed} failed {skipped} inconclusive")
if __name__ == "__main__":
try:
main()
except KeyboardInterrupt:
print("\nInterrupted.")
sys.exit(0)
+2 -2
View File
@@ -20,8 +20,8 @@ Typical usage (TCP / modem):
"""
from .client import MiniMateClient
from .models import DeviceInfo, Event
from .models import DeviceInfo, Event, MonitorLogEntry
from .transport import SerialTransport, TcpTransport
__version__ = "0.1.0"
__all__ = ["MiniMateClient", "DeviceInfo", "Event", "SerialTransport", "TcpTransport"]
__all__ = ["MiniMateClient", "DeviceInfo", "Event", "MonitorLogEntry", "SerialTransport", "TcpTransport"]
+949
View File
@@ -0,0 +1,949 @@
"""
blastware_file.py Blastware binary file codec for bidirectional interoperability.
Reads and writes the proprietary Instantel/Blastware file formats:
Waveform events (.CE0W, .VM0H, .440, .7M0, etc.) (extension encoding UNKNOWN see below)
.MLG Monitor log (monitoring session history)
All waveform formats share a common 22-byte file header prefix and identical
internal binary structure (same type tag 00 12 03 00, same STRT record layout).
Blastware identifies the file type by extension, not by a magic marker.
EXTENSION ENCODING V10.72 firmware FULLY CONFIRMED 2026-04-22:
Direct / manual download: AB0 (3-char, no type character)
Call-home (ACH) download: AB0W or AB0H (4-char, W=waveform H=histogram)
AB = 2-char base-36 of (total_seconds % 1296), where
total_seconds = (event_local_time 1985-01-01T00:00:00_local).
0 = always literal digit zero.
Verified against 3,248 call-home files from a 10-year production archive.
The 10-year archive contains only ACH files (all end in W or H).
Manual Blastware downloads produce 3-char AB0 extensions same encoding
but without the trailing type character.
Old firmware (S338, 3-char extensions): encoding unknown / same as manual?
Micromate Series 4 uses a different scheme (literal datetime in filename).
File structure overview
Waveform file structure (confirmed from example-events/4-3-26-multi/M529LIY6 (example event)):
[22B header] [21B STRT record] [body bytes] [26B footer]
Header (22 bytes):
10 00 01 80 00 00 fixed prefix
49 6e 73 74 61 6e 74 65 6c 00 b'Instantel\x00'
07 2c fixed
00 12 03 00 waveform file type tag (shared by all waveform extensions)
STRT record (21 bytes, immediately follows header):
53 54 52 54 b'STRT'
ff fe fixed (2 bytes)
[key4] 4-byte waveform event key
[key4] 4-byte waveform event key (repeated)
[zeros] 7 bytes padding
[rectime] uint8 record time in seconds
Body (variable reconstructed from A5 frame data):
The body bytes are derived from the raw A5 frame wire content, specifically
from the DLE-decoded representation of each frame's contribution. See the
_frame_body_bytes() helper for the exact algorithm.
Footer (26 bytes):
0e 08
[ts1: 8B big-endian timestamp] start timestamp
[ts2: 8B big-endian timestamp] stop timestamp
00 01 00 02 00 00
[crc: 2B] CRC (algorithm unconfirmed; written as 0x00 0x00 placeholder)
Timestamp format (big-endian, 8 bytes):
[day] [month] [year_HI] [year_LO] [0x00] [hour] [min] [sec]
MLG (monitor log, confirmed from example-events/4-3-26-multi/BE11529.MLG):
[308B header] [N × 292B records]
Header (308 bytes):
Offset 0x00: 10 00 01 80 00 00 Instantel\x00 07 2c 22 01 0e a0 fixed (16B)
Offset 0x10: ... (unknown structure, written as zeros + serial)
Offset 0x2A: serial number (8 bytes, null-padded ASCII, e.g. "BE11529")
... zero-padded to 308 bytes total
Record (292 bytes each):
[2B CRC] unknown algorithm; written as 0x00 0x00
22 01 0e 80 record marker
[ts1: 8B big-endian timestamp] start time
[ts2: 8B big-endian timestamp] stop time (zeros if no stop)
[4B flags] see MLG_FLAGS_* constants below
[10B serial] null-padded serial number ASCII
[text] for trigger records: [0x08][8B ts1_copy] then ASCII "Geo: X.XXX in/s"
for monitoring records: b'' (or minimal separator)
[zero-padded to 292 bytes]
Critical implementation notes
Waveform body reconstruction algorithm (confirmed 2026-04-21 from verification against
M529LIY6 (example event) using raw_s3_20260403_153508.bin capture):
The waveform body bytes come from the A5 frame content, stripped of DLE-framing
artifacts. Each A5 frame contributes a different slice of its data section,
with DLE+{0x02,0x03,0x04} byte pairs stripped.
Skip amounts per frame index (offsets into frame.data):
A5[0] (probe): data[strt_pos + 21 + 7] (skip header + STRT record)
strt_pos found by searching frame.data[7:] for b'STRT';
the contribution starts at strt_pos + 21 within data[7:]
which equals strt_pos + 21 + 7 within frame.data.
A5[1]: data[13] (skip 7-byte frame.data prefix + 6 header bytes)
A5[2..N]: data[12] (skip 7-byte frame.data prefix + 5 header bytes)
Terminator A5: data[11] (1 byte less than chunk frames; terminator inner header
is 4 bytes instead of 5 confirmed 2026-04-21)
DLE strip rule (applied AFTER slicing):
Strip any 0x10 byte that is immediately followed by 0x02, 0x03, or 0x04.
This undoes the DLE-escape that S3FrameParser preserves as literal pairs.
Applied to frame.data[skip:] + bytes([frame.chk_byte]) together, then
conditionally exclude the trailing chk_byte from the output.
chk_byte absorption:
When frame.data[-1] == 0x10 AND frame.chk_byte {0x02, 0x03, 0x04},
the last byte of frame.data is the DLE prefix of a split DLE+chk pair.
Including chk_byte in the strip buffer allows the pair to be stripped as
a unit. After stripping, the trailing chk_byte is ALWAYS removed because
_strip_inner_frame_dles keeps the byte after the DLE (the chk_byte value),
and that value is the checksum, never payload. This applies to all three
cases (chk {0x02, 0x03, 0x04}) identically.
MLG CRC:
The algorithm that produces the 2-byte CRC at the start of each MLG record
is unknown. All examined records use non-zero values that do not match
CRC-16/CCITT, CRC-16/IBM, CRC-32 (truncated), word sums, XOR variants, or
any of the 40+ polynomial/init combinations tested. The writer emits 0x0000.
This produces files that Blastware may reject or display without the CRC check
the exact impact on BW import is unknown (TODO: test).
Public API
blastware_filename(event, serial)
Return the correct Blastware filename for an event (e.g. "M529LIY6.CE0W").
Full AB0T extension encoding confirmed 2026-04-22 against 3,248 archive files.
Extension matches what Blastware itself would generate for the same event.
write_blastware_file(event, a5_frames, path)
Create a Blastware waveform file from an Event and the full A5 frame list.
All waveform extensions share the same binary format the extension is set
by blastware_filename() based on the event timestamp and type.
read_blastware_file(path) Event
Parse a Blastware waveform file into an Event object with waveform data populated.
(Not yet implemented placeholder raises NotImplementedError.)
write_mlg(entries, serial, path)
Create a .MLG file from a list of MonitorLogEntry objects.
read_mlg(path) list[MonitorLogEntry]
Parse a .MLG file into MonitorLogEntry objects.
(Not yet implemented placeholder raises NotImplementedError.)
"""
from __future__ import annotations
import datetime
import logging
import struct
from pathlib import Path
from typing import Optional, Union
from .framing import S3Frame
from .models import Event, MonitorLogEntry, Timestamp
log = logging.getLogger(__name__)
# ── File header constants ─────────────────────────────────────────────────────
# Common 16-byte prefix shared by waveform files and MLG (confirmed from binary inspection).
_FILE_HEADER_PREFIX = bytes.fromhex("1000018000004973") + b"tantel\x00\x07\x2c"
# = 10 00 01 80 00 00 49 73 74 61 6e 74 65 6c 00 07 2c (17 bytes)
# Confirmed breakdown: 10 00 01 80 00 00 = fixed; "Instantel\x00" = 10B; 07 2c = fixed
# Simpler construction:
_FILE_HEADER_PREFIX = b"\x10\x00\x01\x80\x00\x00Instantel\x00\x07\x2c" # 17 bytes
# Waveform file type tag (4 bytes after common prefix) — shared by ALL waveform extensions
_WAVEFORM_TYPE_TAG = b"\x00\x12\x03\x00" # confirmed from M529LIY6 (example event) — same tag for .CE0W, .VM0H, etc.
# MLG type tag (4 bytes after common prefix)
_MLG_TYPE_TAG = b"\x22\x01\x0e\xa0" # confirmed from BE11529.MLG offset 0x11..0x14
# Total header sizes
_WAVEFORM_HEADER_SIZE = 22 # 17 + 4 = 21... wait. Let me recalculate.
# From binary: first 22 bytes = header, then STRT at byte 22.
# 17-byte common prefix + 4-byte type tag = 21 bytes. But observed header is 22B.
# Checking: 6 fixed + 10 "Instantel\x00" + 2 "07 2c" = 18B prefix, then 4B type tag = 22B.
# Re-count: b"\x10\x00\x01\x80\x00\x00" = 6B + b"Instantel\x00" = 10B + b"\x07\x2c" = 2B = 18B prefix.
_FILE_HEADER_PREFIX = b"\x10\x00\x01\x80\x00\x00Instantel\x00\x07\x2c" # 18 bytes
_WAVEFORM_HEADER_SIZE = 22 # 18 + 4 = 22 bytes ✅
_MLG_HEADER_SIZE = 308 # confirmed from BE11529.MLG
# MLG record marker (4 bytes after 2-byte CRC at start of each record)
_MLG_RECORD_MARKER = b"\x22\x01\x0e\x80"
_MLG_RECORD_SIZE = 292 # bytes per record (confirmed from BE11529.MLG)
# MLG record flags (4 bytes at record[22:26])
# Confirmed from BE11529.MLG binary inspection:
MLG_FLAGS_START_ONLY = b"\xff\xff\x00\x00" # monitoring start with no stop
MLG_FLAGS_TRIGGER = b"\x01\x00\x02\x00" # triggered event (has ts1 + ts2)
MLG_FLAGS_INTERVAL = b"\x02\x00\x00\x00" # monitoring interval (has ts1 + ts2)
# ── Timestamp helpers ─────────────────────────────────────────────────────────
def _encode_ts_be(ts: Optional[datetime.datetime]) -> bytes:
"""
Encode a datetime as an 8-byte big-endian Blastware timestamp.
Format (waveform file and MLG record timestamps):
[day][month][year_HI][year_LO][0x00][hour][min][sec]
Big-endian year confirmed from M529LIY6 (example event) footer:
footer bytes [2..9] = 01 04 07 ea 00 00 1c 08
day=1 month=4 year=0x07ea=2026 hour=0 min=28 sec=8
Returns 8 zero bytes if ts is None.
"""
if ts is None:
return bytes(8)
return bytes([
ts.day,
ts.month,
(ts.year >> 8) & 0xFF,
ts.year & 0xFF,
0x00,
ts.hour,
ts.minute,
ts.second,
])
def _decode_ts_be(raw: bytes) -> Optional[datetime.datetime]:
"""
Decode an 8-byte big-endian Blastware timestamp.
Returns None if the bytes are all zero or structurally invalid.
"""
if len(raw) < 8 or raw == bytes(8):
return None
day = raw[0]
month = raw[1]
year = (raw[2] << 8) | raw[3]
hour = raw[5]
minute = raw[6]
sec = raw[7]
try:
return datetime.datetime(year, month, day, hour, minute, sec)
except ValueError:
return None
def _ts_from_model(ts: Optional[Timestamp]) -> Optional[datetime.datetime]:
"""Convert a models.Timestamp to datetime.datetime, or None."""
if ts is None:
return None
try:
return datetime.datetime(ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second)
except (ValueError, TypeError):
return None
# ── DLE strip helper ──────────────────────────────────────────────────────────
def _strip_inner_frame_dles(data: bytes) -> bytes:
"""
Strip DLE (0x10) framing markers from A5 inner-frame content.
The A5 (bulk waveform stream) response body contains DLE-encoded sub-frame
structure. S3FrameParser preserves DLE+XX pairs as two literal bytes in
frame.data. Only the DLE marker byte needs to be removed; the following
byte is actual payload content.
Rule: when 0x10 is immediately followed by {0x02, 0x03, 0x04}, strip the
0x10 (DLE marker) and keep the following byte as payload.
Lone 0x10 bytes not followed by {0x02, 0x03, 0x04} are kept as-is.
Confirmed correct by verifying reconstructed waveform body against M529LIY6 (example event):
- 0x10 0x02 in terminator 0x02 kept
- 0x10 0x04 in terminator (month byte) 0x04 kept
"""
out = bytearray()
i = 0
while i < len(data):
b = data[i]
if b == 0x10 and i + 1 < len(data) and data[i + 1] in {0x02, 0x03, 0x04}:
# Strip the DLE marker; the next byte is payload and will be appended
# in the next loop iteration.
i += 1
continue
out.append(b)
i += 1
return bytes(out)
def _frame_body_bytes(frame: S3Frame, skip: int) -> bytes:
"""
Extract the waveform body contribution from one A5 S3Frame.
The contribution is frame.data[skip:] with inner-frame DLE pairs stripped
per _strip_inner_frame_dles(). The chk_byte is temporarily appended before
stripping to handle the split-pair edge case where a DLE at the end of
frame.data is paired with chk_byte.
Split-pair edge case (confirmed for A5[8] of M529LIY6 (example event), 2026-04-21):
S3FrameParser appends DLE+XX pairs as two literal bytes when XX {DLE, ETX}.
When the LAST occurrence of such a pair straddles the payload/checksum boundary
(i.e., DLE is the last byte of raw_payload and XX is the checksum), the parser
splits them:
- DLE ends up as the last byte of frame.data (frame.data[-1] == 0x10)
- XX is stored as frame.chk_byte
To strip the pair correctly, we reunite the bytes before calling the strip
function. Since chk_byte is the checksum (not payload data), it is excluded
from the final output regardless of whether it was part of a pair.
Post-strip chk_byte removal (ALL cases):
_strip_inner_frame_dles strips the 0x10 and KEEPS chk_byte in all cases.
Chk_byte is always the checksum (not payload), so always strip it off.
Args:
frame: S3Frame with frame.data and frame.chk_byte populated.
skip: Number of leading bytes in frame.data to exclude (frame header).
Returns:
bytes the waveform body contribution for this frame.
"""
if skip >= len(frame.data):
return b""
relevant = frame.data[skip:]
# Detect split DLE+chk pair at the frame boundary.
has_split_pair = (
len(relevant) > 0
and relevant[-1] == 0x10
and frame.chk_byte in {0x02, 0x03, 0x04}
)
if has_split_pair:
# Reunite the split pair so the strip function sees both bytes together.
buf = relevant + bytes([frame.chk_byte])
stripped = _strip_inner_frame_dles(buf)
# _strip_inner_frame_dles strips the DLE (0x10) and KEEPS chk_byte.
# chk_byte is the received checksum — never payload — so remove it.
# This is correct for all values in {0x02, 0x03, 0x04}.
if stripped:
stripped = stripped[:-1]
return stripped
else:
return _strip_inner_frame_dles(relevant)
# ── Filename helper ───────────────────────────────────────────────────────────
_INSTANTEL_EPOCH = datetime.datetime(1985, 1, 1, 0, 0, 0)
"""
Instantel timestamp epoch January 1, 1985, 00:00:00 local time.
Confirmed 2026-04-21: stem values for 6 independent events (April 19, 2026)
all converge to this epoch when decoded as floor(seconds_since_epoch / 1296).
1985 is the year Instantel was founded.
"""
_STEM_UNIT_SEC = 1296 # = 36^2 seconds ≈ 21.6 minutes per stem unit
_STEM_CHARS = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
# ── Waveform file extension encoding ─────────────────────────────────────────
#
# NEW FIRMWARE (V10.72+) — FULLY DECODED (confirmed 2026-04-21, 10-year archive):
#
# Extension format: AB0T (4 characters)
# AB = 2-char base-36 encoding of (seconds_since_epoch % 1296)
# i.e. the number of seconds into the current 21.6-minute stem window
# Range: 0 ("00") to 1295 ("ZZ")
# 0 = always literal '0'
# T = event type: 'W' = Full Waveform, 'H' = Full Histogram
#
# Combined with the 4-char stem (which encodes seconds_since_epoch // 1296),
# the FULL filename gives a second-resolution timestamp:
# total_seconds = stem_val * 1296 + ab_val
# timestamp = EPOCH + timedelta(seconds=total_seconds)
#
# Verified against three S353L4H0 events (all three match to the second):
# S353L4H0.3M0W Full Waveform 2025-06-23 13:57:22 AB=3M=130 ✓
# S353L4H0.8S0H Full Histogram 2025-06-23 14:00:28 AB=8S=316 ✓
# S353L4H0.9X0W Full Waveform 2025-06-23 14:01:09 AB=9X=357 ✓
#
# OLD FIRMWARE (S338, 3-char extensions ending in '0') — UNKNOWN:
# Observed (old firmware / manual downloads): .440, .470, .7M0, .9T0, .EI0, etc.
# The V10.72 formula does NOT apply to these.
# Extension is NOT recording mode (refuted 2026-04-21: continuous → .EI0, not .9T0).
# blastware_filename() computes the correct AB0 extension for V10.72 firmware.
#
# WRONG earlier assumption (do not re-introduce):
# Extension was believed to encode recording mode × sample rate.
# Refuted by continuous-mode event producing .EI0 instead of .9T0.
def _make_stem(ts_local: datetime.datetime) -> str:
"""
Encode a local timestamp as a 4-character uppercase base-36 stem.
Algorithm (confirmed 2026-04-21 from 6 known file/timestamp pairs):
stem_int = floor((ts_local - Jan_1_1985_midnight_local) / 1296_seconds)
stem = 4-char uppercase base-36 encoding of stem_int
Unit = 36² = 1296 seconds 21.6 minutes. Events within the same 1296-second
window receive the same stem; their extension distinguishes them.
"""
delta_sec = int((ts_local - _INSTANTEL_EPOCH).total_seconds())
n = delta_sec // _STEM_UNIT_SEC
s = ""
for _ in range(4):
s = _STEM_CHARS[n % 36] + s
n //= 36
return s
def blastware_filename(event: Event, serial: str, ach: bool = False) -> str:
"""
Return the correct Blastware filename for an event.
CONFIRMED 2026-04-22 verified against 3,248 files from a 10-year archive.
Filename format: <prefix_letter><serial3><stem><AB>0[T]
where:
prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))
encodes the production generation (batch of 1000 units)
e.g. BE6907H, BE11529M, BE14036P, BE18003T
serial3 = f"{serial_numeric % 1000:03d}"
last 3 digits of numeric serial, zero-padded
stem = 4-char base-36 of floor(total_seconds / 1296)
encodes which 21.6-minute window the event fell in
AB = 2-char base-36 of (total_seconds % 1296)
encodes seconds within the window (01295)
0 = always literal digit zero
T = 'W' or 'H' ONLY appended for call-home (ACH) downloads (ach=True).
Manual / direct downloads produce a 3-char extension (AB0) with no type char.
Call-home downloads produce a 4-char extension (AB0W or AB0H).
total_seconds = (event_local_time 1985-01-01T00:00:00_local) in seconds
The 10-year production archive contains only call-home files (all end in W or H).
Manual Blastware downloads produce 3-char extensions the same AB0 prefix but
without the trailing type character.
Micromate Series 4 uses a completely different naming scheme (literal datetime
in filename); this function does not apply to Micromate units.
Args:
event: Event object with timestamp set.
serial: Device serial number string (e.g. "BE11529").
ach: If True, append W/H type character (call-home style).
If False (default), omit type character (direct download style).
Returns:
Filename string, e.g. "M529LIY6.CE0" (direct) or "M529LIY6.CE0H" (ACH).
"""
# ── Serial prefix ──────────────────────────────────────────────────────────
serial_digits = "".join(c for c in serial if c.isdigit())
if len(serial_digits) >= 1:
serial_numeric = int(serial_digits)
generation = serial_numeric // 1000
prefix_letter = chr(ord('B') + generation)
serial3 = f"{serial_numeric % 1000:03d}"
else:
prefix_letter = "M" # fallback
serial3 = "000"
prefix = prefix_letter + serial3
# ── Stem + AB extension from timestamp ────────────────────────────────────
if event.timestamp is not None:
try:
ts_local = datetime.datetime(
event.timestamp.year, event.timestamp.month, event.timestamp.day,
event.timestamp.hour, event.timestamp.minute, event.timestamp.second,
)
delta_sec = int((ts_local - _INSTANTEL_EPOCH).total_seconds())
stem = _make_stem(ts_local)
ab_val = delta_sec % _STEM_UNIT_SEC
ab_str = _STEM_CHARS[ab_val // 36] + _STEM_CHARS[ab_val % 36]
except (ValueError, TypeError, AttributeError):
stem = "0000"
ab_str = "00"
else:
stem = "0000"
ab_str = "00"
# ── Type character (ACH only) ─────────────────────────────────────────────
if ach:
if getattr(event, 'recording_mode', None) in (3, 4): # Histogram / Hist+Cont
type_char = 'H'
else:
type_char = 'W'
ext = f".{ab_str}0{type_char}"
else:
ext = f".{ab_str}0"
return prefix + stem + ext
# ── A5 frame classifier ───────────────────────────────────────────────────────────
# ASCII markers that identify a compliance-config / metadata frame.
# These strings appear in the A5 bulk stream as part of the device's
# compliance setup payload. They should NEVER appear in raw ADC waveform
# frames (which are binary-heavy, < 20 % printable ASCII).
_METADATA_FRAME_MARKERS = (
b"Project:",
b"Client:",
b"Standard Recording Setup",
b"Extended Notes",
b"User Name:",
b"Seis Loc:",
)
def classify_frame(frame: S3Frame) -> str:
"""
Classify an A5 bulk waveform stream frame by its content.
Returns one of:
"terminator" page_key == 0x0000
"probe_or_strt" data contains b"STRT\xff\xfe" (the initial probe response)
"metadata" data contains ASCII compliance-config markers
"waveform" predominantly binary (< 20 % printable ASCII)
"unknown" none of the above criteria matched
Used by write_blastware_file() to filter non-waveform frames out of
the reconstructed body so that metadata blocks (Project:, Client:, )
and spurious STRT records do not corrupt the output file.
"""
if frame.page_key == 0x0000:
return "terminator"
data = bytes(frame.data)
if b"STRT\xff\xfe" in data:
return "probe_or_strt"
if any(m in data for m in _METADATA_FRAME_MARKERS):
return "metadata"
if len(data) > 0:
printable = sum(1 for b in data if 32 <= b < 127)
if printable / len(data) < 0.20:
return "waveform"
return "unknown"
# ── Waveform file writer ───────────────────────────────────────────────────────────
def write_blastware_file(
event: Event,
a5_frames: list[S3Frame],
path: Union[str, Path],
) -> None:
"""
Write a Blastware waveform file from a downloaded event.
Args:
event: Event object (populated by get_events() or download_waveform()).
Used for the STRT record (key, rectime) and footer timestamps.
a5_frames: Complete A5 frame list INCLUDING the terminator frame
(page_key=0x0000). Pass include_terminator=True to
read_bulk_waveform_stream() when collecting frames.
Must have at least 2 frames (probe + terminator).
path: Destination file path. Parent directory must exist.
Extension should be set via blastware_filename().
File layout:
[22B header] [21B STRT] [body bytes] [26B footer]
Raises:
ValueError: if a5_frames is empty or has no terminator (page_key=0).
OSError: if the file cannot be written.
Confirmed correct waveform body reconstruction against M529LIY6 (example event) (2026-04-21).
"""
if not a5_frames:
raise ValueError("a5_frames must not be empty")
path = Path(path)
# ── Extract STRT record from probe frame ────────────────────────────────
# The STRT record (21 bytes) lives verbatim inside A5[0].data[7:].
# It is stored as-is in the waveform file — do NOT reconstruct it from Event
# fields, as bytes [10:14] and [14:20] contain device-specific values
# (not simply key4 repeated or zero-padded). Confirmed 2026-04-21.
#
# STRT layout (21 bytes, observed in M529LIY6 files):
# [0:4] b'STRT'
# [4:6] 0xff 0xfe (fixed)
# [6:10] key4 (event key)
# [10:14] device-specific field (NOT a key4 repeat)
# [14:20] device-specific fields (NOT zeros)
# [20] rectime uint8 seconds
# Extract STRT from the DLE-stripped probe frame.
#
# frame.data[7:] is the raw wire representation; it may contain DLE+{02,03,04}
# inner-frame pairs that S3FrameParser preserves as two literal bytes. The
# Blastware file stores the stripped form, so we must strip before extracting.
#
# Example (M529LK0Y, 2026-04-21): STRT contains value 0x02 encoded as [10 02]
# on the wire. Without stripping, STRT is 22 raw bytes → write_blastware_file writes the
# DLE prefix into the file AND begins the body 1 byte too early (probe_skip off
# by 1). Stripping fixes both.
#
# probe_skip must be computed in the RAW frame.data domain (it is used as the
# `skip` argument to _frame_body_bytes which operates on raw frame.data).
# We walk the raw bytes counting stripped bytes until we have passed
# strt_pos + 21 stripped bytes, giving the raw offset of the first body byte.
w0_raw = bytes(a5_frames[0].data[7:])
w0_stripped = _strip_inner_frame_dles(w0_raw)
strt_pos_stripped = w0_stripped.find(b"STRT")
if strt_pos_stripped >= 0:
strt = bytes(w0_stripped[strt_pos_stripped : strt_pos_stripped + 21])
# Walk raw bytes to find the raw-domain end of the STRT (= body start).
target_stripped = strt_pos_stripped + 21
stripped_so_far = 0
raw_i = 0
while stripped_so_far < target_stripped and raw_i < len(w0_raw):
if (w0_raw[raw_i] == 0x10
and raw_i + 1 < len(w0_raw)
and w0_raw[raw_i + 1] in {0x02, 0x03, 0x04}):
raw_i += 2 # DLE pair → 1 stripped byte, 2 raw bytes
else:
raw_i += 1 # normal byte → 1 stripped byte, 1 raw byte
stripped_so_far += 1
probe_skip = 7 + raw_i # raw bytes to skip: 7 header + raw STRT length
else:
# Fallback: construct a minimal STRT if probe frame lacks it
key4 = event._waveform_key if hasattr(event, '_waveform_key') and event._waveform_key else bytes(4)
rectime = event.rectime_seconds if event.rectime_seconds is not None else 0
strt = b"STRT" + b"\xff\xfe" + key4 + bytes(14) + bytes([rectime & 0xFF])
probe_skip = 7 + 21
log.warning(
"write_blastware_file: strt_pos_stripped=%d probe_skip=%d "
"probe_data_len=%d strt_hex=%s",
strt_pos_stripped if strt_pos_stripped >= 0 else -1,
probe_skip,
len(a5_frames[0].data),
strt.hex() if len(strt) >= 4 else "(short)",
)
if len(strt) != 21:
raise ValueError(f"STRT record must be 21 bytes, got {len(strt)}")
# ── Build waveform file header ─────────────────────────────────────────────────────
header = _FILE_HEADER_PREFIX + _WAVEFORM_TYPE_TAG
assert len(header) == _WAVEFORM_HEADER_SIZE, f"Waveform header must be {_WAVEFORM_HEADER_SIZE} bytes"
# ── Build body from A5 frames ────────────────────────────────────────────
# The waveform body is reconstructed from ALL A5 frames (data + terminator).
# The terminator frame's contribution includes the 26-byte footer at its end.
#
# Reconstruction layout (confirmed from M529LIY6 captures, 2026-04-21):
# all_bytes = contributions from A5[0..N] + terminator_contribution
# body = all_bytes[:-26] (everything except the last 26 bytes)
# footer = all_bytes[-26:] (last 26 bytes = the waveform file footer)
#
# The footer bytes come directly from the terminator frame's inner content —
# using them verbatim ensures timestamps match the device's recorded values.
# Separate terminator from data frames.
# Search from the FRONT for the first terminator (page_key == 0x0000).
# Do NOT use a5_frames[-1] — if _a5_frames contains stray frames from a
# subsequent event (a known get_events side-effect), the last frame will
# not be the terminator and the footer will be mis-identified.
term_idx: Optional[int] = None
for _i, _f in enumerate(a5_frames):
if _f.page_key == 0x0000:
term_idx = _i
break
if term_idx is not None:
body_frames = a5_frames[:term_idx]
term_frame = a5_frames[term_idx]
else:
body_frames = a5_frames
term_frame = None
log.warning(
"write_blastware_file: %d body_frames term_idx=%s",
len(body_frames),
str(term_idx) if term_idx is not None else "None",
)
all_bytes = bytearray()
for fi, frame in enumerate(body_frames):
# All body frames contribute to the waveform body — no frames are skipped.
#
# Over TCP via cellular modem, _recv_5a_batch() correctly collects all
# A5 frames per chunk request (the device's ~1100-byte RS-232 response
# is forwarded as ~2 TCP segments of ~550 bytes each, each parsed as a
# separate S3 frame). ALL of these frames contain ADC body data and
# must be included in the file — confirmed from 4-27-26 TCP capture
# analysis: contributions from all 14 frames → 6821 bytes → file 6864 bytes.
#
# Skip amounts (offsets into frame.data):
# fi=0 (probe): probe_skip — skips the type_tag header + STRT record
# fi=1: 13 — 7-byte frame.data prefix + 6 inner header bytes
# fi>=2: 12 — 7-byte frame.data prefix + 5 inner header bytes
if fi == 0:
skip = probe_skip
elif fi == 1:
skip = 13
else:
skip = 12
contribution = _frame_body_bytes(frame, skip)
log.warning("write_blastware_file: fi=%d skip=%d raw_data=%d contribution=%d",
fi, skip, len(frame.data), len(contribution))
all_bytes.extend(contribution)
# Terminator contributes its content, which ends with the 26-byte footer.
# skip=11 (not 12) because the terminator's inner frame header is 4 bytes,
# one shorter than chunk frames' 5-byte inner header. Confirmed 2026-04-21.
if term_frame is not None:
term_contribution = _frame_body_bytes(term_frame, 11)
log.warning(
"write_blastware_file: term_frame data_len=%d skip=11 "
"contribution_len=%d first8=%s",
len(term_frame.data),
len(term_contribution),
term_contribution[:8].hex() if len(term_contribution) >= 8 else term_contribution.hex(),
)
all_bytes.extend(term_contribution)
log.warning(
"write_blastware_file: all_bytes total=%d last28=%s",
len(all_bytes),
bytes(all_bytes[-28:]).hex() if len(all_bytes) >= 28 else bytes(all_bytes).hex(),
)
if len(all_bytes) >= 26:
body = bytes(all_bytes[:-26])
footer = bytes(all_bytes[-26:])
else:
# Fallback: no terminator or very short stream → build footer from event metadata
body = bytes(all_bytes)
start_dt = _ts_from_model(event.timestamp)
stop_dt: Optional[datetime.datetime] = None
if start_dt is not None and event.rectime_seconds:
stop_dt = start_dt + datetime.timedelta(seconds=event.rectime_seconds)
footer = (
b"\x0e\x08"
+ _encode_ts_be(start_dt)
+ _encode_ts_be(stop_dt)
+ b"\x00\x01\x00\x02\x00\x00"
+ b"\x00\x00" # CRC placeholder
)
# ── Write file ───────────────────────────────────────────────────────────
with open(path, "wb") as f:
f.write(header)
f.write(strt)
f.write(body)
f.write(footer)
def read_blastware_file(path: Union[str, Path]) -> Event:
"""
Parse a Blastware waveform file into an Event object.
NOT YET IMPLEMENTED.
Args:
path: Path to the waveform file.
Returns:
Event object with waveform data populated.
Raises:
NotImplementedError: always (pending implementation).
"""
raise NotImplementedError("read_blastware_file() is not yet implemented")
# ── MLG file writer ───────────────────────────────────────────────────────────
def _build_mlg_header(serial: str) -> bytes:
"""
Build the 308-byte MLG file header.
Header structure (confirmed from BE11529.MLG binary inspection):
Offset 0x00: 10 00 01 80 00 00 Instantel\x00 07 2c 22 01 0e a0 (22B)
Offset 0x16: ... (16B unknown observed as zeros in BE11529.MLG)
Offset 0x2A: serial number (8 bytes, null-padded ASCII)
... rest zero-padded to 308 bytes
The serial string "BE11529" appears at offset 0x2A (42 decimal).
"""
buf = bytearray(_MLG_HEADER_SIZE)
# Common prefix + MLG type tag
prefix = _FILE_HEADER_PREFIX + _MLG_TYPE_TAG # 22 bytes
buf[0:len(prefix)] = prefix
# Serial number at offset 0x2A
serial_bytes = serial.encode("ascii", errors="replace")[:8]
serial_padded = serial_bytes.ljust(8, b"\x00")
buf[0x2A : 0x2A + 8] = serial_padded
return bytes(buf)
def _build_mlg_record(
entry: MonitorLogEntry,
serial: str,
) -> bytes:
"""
Build one 292-byte MLG record from a MonitorLogEntry.
Record layout (confirmed from BE11529.MLG binary inspection):
[0:2] CRC 2-byte CRC (algorithm unknown; written as 0x0000)
[2:6] marker 22 01 0e 80
[6:14] ts1 8B big-endian start timestamp
[14:22] ts2 8B big-endian stop timestamp
[22:26] flags 4B record flags (see MLG_FLAGS_* constants)
[26:36] serial 10B null-padded serial number
[36:] text for triggered events: [0x08][8B ts1_copy]["Geo: X.XXX in/s"]
for monitoring intervals: b"" or minimal separator
[... zero-padded to 292 bytes]
Flags based on entry type:
- MonitorLogEntry with start_time only (no stop_time): MLG_FLAGS_START_ONLY
- MonitorLogEntry with both times and geo_threshold_ips set: MLG_FLAGS_TRIGGER
- MonitorLogEntry with both times (monitoring interval): MLG_FLAGS_INTERVAL
The triggered-event text block (flags = MLG_FLAGS_TRIGGER):
[0x08] [ts1: 8B] [ASCII "Geo: X.XXX in/s\x00"]
Confirmed from BE11529.MLG records at offset 0x0134 and 0x0258.
"""
buf = bytearray(_MLG_RECORD_SIZE)
start_dt = (
datetime.datetime(
entry.start_time.year, entry.start_time.month, entry.start_time.day,
entry.start_time.hour, entry.start_time.minute, entry.start_time.second,
)
if entry.start_time else None
)
stop_dt = (
datetime.datetime(
entry.stop_time.year, entry.stop_time.month, entry.stop_time.day,
entry.stop_time.hour, entry.stop_time.minute, entry.stop_time.second,
)
if entry.stop_time else None
)
# [0:2] CRC placeholder
buf[0:2] = b"\x00\x00"
# [2:6] Record marker
buf[2:6] = _MLG_RECORD_MARKER
# [6:14] ts1
buf[6:14] = _encode_ts_be(start_dt)
# [14:22] ts2
buf[14:22] = _encode_ts_be(stop_dt)
# [22:26] flags
if stop_dt is None:
flags = MLG_FLAGS_START_ONLY
elif entry.geo_threshold_ips is not None:
flags = MLG_FLAGS_TRIGGER
else:
flags = MLG_FLAGS_INTERVAL
buf[22:26] = flags
# [26:36] serial (10B null-padded)
serial_bytes = serial.encode("ascii", errors="replace")[:10]
buf[26 : 26 + len(serial_bytes)] = serial_bytes
# [36:] text content
pos = 36
if flags == MLG_FLAGS_TRIGGER:
# Extra ts1 copy: [0x08][ts1: 8B]
buf[pos] = 0x08
pos += 1
buf[pos : pos + 8] = _encode_ts_be(start_dt)
pos += 8
if entry.geo_threshold_ips is not None:
geo_text = f"Geo: {entry.geo_threshold_ips:.3f} in/s\x00".encode("ascii")
buf[pos : pos + len(geo_text)] = geo_text
pos += len(geo_text)
return bytes(buf)
def write_mlg(
entries: list[MonitorLogEntry],
serial: str,
path: Union[str, Path],
) -> None:
"""
Write a Blastware .MLG monitor log file.
Args:
entries: List of MonitorLogEntry objects (from get_monitor_log_entries()).
Each entry produces one 292-byte record in the file.
serial: Device serial number string (e.g. "BE11529").
Written to the file header and each record.
path: Destination file path. Extension is not enforced use ".MLG".
File layout:
[308B header] [N × 292B records]
Note: The 2-byte CRC at the start of each record is written as 0x0000.
The CRC algorithm is unknown (see module docstring).
Raises:
OSError: if the file cannot be written.
"""
path = Path(path)
header = _build_mlg_header(serial)
with open(path, "wb") as f:
f.write(header)
for entry in entries:
record = _build_mlg_record(entry, serial)
f.write(record)
def read_mlg(path: Union[str, Path]) -> list[MonitorLogEntry]:
"""
Parse a Blastware .MLG file into a list of MonitorLogEntry objects.
NOT YET IMPLEMENTED.
Args:
path: Path to the .MLG file.
Returns:
List of MonitorLogEntry objects.
Raises:
NotImplementedError: always (pending implementation).
"""
raise NotImplementedError("read_mlg() is not yet implemented")
+898 -128
View File
File diff suppressed because it is too large Load Diff
+10 -4
View File
@@ -457,6 +457,11 @@ class S3Frame:
page_lo: int # PAGE_LO from header
data: bytes # payload data section (payload[5:], checksum already stripped)
checksum_valid: bool
chk_byte: int = 0 # actual checksum byte received from wire (body[-1])
# needed for waveform file reconstruction: when the last data byte
# is 0x10 and chk_byte ∈ {0x02, 0x03, 0x04}, the DLE+chk pair
# must be included in the DLE-strip operation to correctly
# reconstruct the Blastware binary body.
@property
def page_key(self) -> int:
@@ -592,9 +597,10 @@ class S3FrameParser:
return None
return S3Frame(
sub = raw_payload[2],
page_hi = raw_payload[3],
page_lo = raw_payload[4],
data = raw_payload[5:],
sub = raw_payload[2],
page_hi = raw_payload[3],
page_lo = raw_payload[4],
data = raw_payload[5:],
checksum_valid = (chk_received == chk_computed),
chk_byte = chk_received,
)
+161 -6
View File
@@ -14,6 +14,7 @@ Notes on certainty:
from __future__ import annotations
import datetime
import struct
from dataclasses import dataclass, field
from typing import Optional
@@ -268,7 +269,7 @@ class ChannelConfig:
label: str # e.g. "Tran", "Vert", "Long", "MicL" ✅
trigger_level: float # in/s (geo) or psi (MicL) ✅
alarm_level: float # in/s (geo) or psi (MicL) ✅
max_range: float # full-scale calibration constant (e.g. 6.206) 🔶
max_range: float # hardware/firmware sensitivity constant (e.g. 6.206053) ✅ confirmed same on all units
unit_label: str # e.g. "in./s" or "psi" ✅
@@ -337,15 +338,34 @@ class ComplianceConfig:
raw: Optional[bytes] = None # full 2090-byte payload (for debugging)
# Recording parameters (✅ CONFIRMED from §7.6)
record_time: Optional[float] = None # seconds (7.0, 10.0, 13.0, etc.)
sample_rate: Optional[int] = None # sps (1024, 2048, 4096, etc.) — NOT YET FOUND ❓
recording_mode: Optional[int] = None # uint8: 0x00=Single Shot, 0x01=Continuous,
# 0x03=Histogram, 0x04=Histogram+Continuous ✅ confirmed 2026-04-20
# Read (E5): data[anchor_pos - 8] (6-byte anchor)
# Write (SUB 71): data[anchor_pos - 7]
sample_rate: Optional[int] = None # sps (1024, 2048, 4096)
histogram_interval_sec: Optional[int] = None # uint16 BE, seconds ✅ confirmed 2026-04-20
# anchor_pos - 4 (same offset in read & write)
# Valid values: 2, 5, 15, 60, 300, 900
# Mode-gated: only active in Histogram/Histogram+Continuous
record_time: Optional[float] = None # seconds (e.g. 3.0, 5.0, 8.0, 10.0)
# Trigger/alarm levels (✅ CONFIRMED per-channel at §7.6)
# For now we store the first geo channel (Transverse) as representatives;
# full per-channel data would require structured Channel objects.
trigger_level_geo: Optional[float] = None # in/s (first geo channel)
alarm_level_geo: Optional[float] = None # in/s (first geo channel)
max_range_geo: Optional[float] = None # in/s full-scale range
trigger_level_geo: Optional[float] = None # in/s (first geo channel)
alarm_level_geo: Optional[float] = None # in/s (first geo channel)
geo_adc_scale: Optional[float] = None # ADC-to-velocity scale factor (float32 at Tran+28) ✅
# = inverse sensitivity = 1/sensitivity (in/s per V)
# Formula (Interface Handbook §4.5): Range = 1.61133 V × scale_factor
# → 1.61133 × 6.206053 = 10.000 in/s (Normal range) ✅
# Firmware uses: PPV (in/s) = ADC_voltage (V) × 6.206053
# Identical on BE11529 and BE18189 — same Instantel geophone hardware.
# NOT a user-configurable setting. Must NOT be written.
geo_range: Optional[int] = None # range/sensitivity selector — CONFIRMED 2026-04-20
# 0x00 = Normal 10.000 in/s (standard gain)
# 0x01 = Sensitive 1.250 in/s (high gain)
# Offset: Tran+33 in both E5 read and SUB 71 write payloads
# (same 2126-byte buffer is round-tripped; applied to Tran/Vert/Long)
# Project/setup strings (sourced from E5 / SUB 71 write payload)
# These are the FULL project metadata from compliance config,
@@ -358,6 +378,78 @@ class ComplianceConfig:
notes: Optional[str] = None # extended notes / additional info
# ── Call Home Config ──────────────────────────────────────────────────────────
@dataclass
class CallHomeConfig:
"""
Auto Call Home (ACH) configuration from SUB 0x2C (response 0xD3).
Read with a standard two-step protocol (probe offset=0x00, data offset=0x7C).
Written via SUB 0x7E (write, 127-byte payload) + SUB 0x7F (confirm).
Confirmed from 4-20-26 call home settings captures (11 BW + S3 capture pairs).
Raw payload layout (data[11:] from S3 response, 125 bytes):
[0] 0x00 header byte
[1] 0x7C = 124 inner length (= offset for SUB 0x7E write - 2)
[2] 0xDC constant
[3:5] 0x00 0x00 padding
[5] auto_call_home_enabled (0x00=off, 0x01=on)
[6:46] dial_string 40-byte null-padded ASCII
[46:87] auto_answer_raw AT command strings (not decoded) present
[87] after_event_recorded (0x01=on, 0x00=off)
[91] at_specified_times (0x01=on, 0x00=off)
[93] time1_enabled (0x01=on, 0x00=off)
[95] time2_enabled (0x01=on, 0x00=off)
[101] time1_hour uint8 decimal 0-23
[102] time1_min uint8 decimal 0-59
[105] time2_hour uint8 decimal 0-23
[106] time2_min uint8 decimal 0-59
[117] DLE prefix (0x10) DLE-escaped num_retries=3 (0x03)
[118] 0x03 device stores/returns 0x03 DLE-escaped
[120] time_between_retries_sec uint8 (= 0x0F = 15 s default)
[122] wait_for_connection_sec uint8 (= 0x3C = 60 s default)
[124] warm_up_time_sec uint8 (= 0x3C = 60 s default)
Write payload = raw 125 bytes + b'\\x00\\x00' (2 trailing zeros) = 127 bytes.
Offset for SUB 0x7E: data[1] + 2 = 0x7C + 2 = 0x7E = 126.
Note on DLE-escaped 0x03: The device's S3 response DLE-escapes ETX (0x03)
bytes as \\x10\\x03. The S3FrameParser preserves both bytes in frame.data.
Subsequent fields after offset 117 are therefore at raw_offset = logical+1.
The raw payload must be round-tripped verbatim in write; do NOT reapply DLE
destuffing or stripping.
"""
raw: Optional[bytes] = None # raw 125-byte read payload (for round-trip write)
# ── Main enable ──────────────────────────────────────────────────────────
auto_call_home_enabled: Optional[bool] = None # raw[5] ✅
# ── Dial string ──────────────────────────────────────────────────────────
dial_string: Optional[str] = None # raw[6:46] 40-byte null-padded ASCII ✅
# ── When to call ─────────────────────────────────────────────────────────
after_event_recorded: Optional[bool] = None # raw[87] ✅
at_specified_times: Optional[bool] = None # raw[91] ✅
# ── Time slot 1 ──────────────────────────────────────────────────────────
time1_enabled: Optional[bool] = None # raw[93] ✅
time1_hour: Optional[int] = None # raw[101] 0-23 ✅
time1_min: Optional[int] = None # raw[102] 0-59 ✅
# ── Time slot 2 ──────────────────────────────────────────────────────────
time2_enabled: Optional[bool] = None # raw[95] ✅
time2_hour: Optional[int] = None # raw[105] 0-23 ✅
time2_min: Optional[int] = None # raw[106] 0-59 ✅
# ── Retry / timeout settings (read-only; not writable via set_call_home_config) ──
num_retries: Optional[int] = None # raw[117:119]=10 03 → value 3 ✅
time_between_retries_sec: Optional[int] = None # raw[120] (shifted +1 by DLE) ✅
wait_for_connection_sec: Optional[int] = None # raw[122] ✅
warm_up_time_sec: Optional[int] = None # raw[124] ✅
# ── Event ─────────────────────────────────────────────────────────────────────
@dataclass
@@ -401,6 +493,10 @@ class Event:
# Set by get_events(); required by download_waveform().
_waveform_key: Optional[bytes] = field(default=None, repr=False)
# Raw A5 frames from the full bulk waveform download (full_waveform=True).
# Populated by get_events() when full_waveform=True; used by write_blastware_file().
_a5_frames: Optional[list] = field(default=None, repr=False)
def __str__(self) -> str:
ts = str(self.timestamp) if self.timestamp else "no timestamp"
ppv = ""
@@ -419,6 +515,65 @@ class Event:
return f"Event#{self.index} {ts}{ppv}"
# ── MonitorLogEntry ───────────────────────────────────────────────────────────
@dataclass
class MonitorLogEntry:
"""
A monitor log entry decoded from a SUB 0x0A (WAVEFORM_HEADER) response
whose first byte is 0x2C (partial record, recording mode = continuous
monitoring without a triggered event).
These are the "partial bins" that Blastware stores between triggered events.
Each entry represents one monitoring interval the span of time during
which the unit was actively monitoring but no threshold crossing occurred.
Confirmed from 4-11-26 MITM capture analysis (2026-04-11):
Header layout (full response data[0:]):
data[0] = 0x2C (partial record type / data length in probe response)
data[1:5] = 0x00 × 4
data[5:9] = event key (4 bytes, big-endian hex)
data[9:11] = 0x00 × 2
data[11:] = timestamp_start (9 or 10 bytes depending on recording mode)
+ timestamp_stop (same format)
+ separator (45 bytes, variable)
+ serial null-terminated (e.g. "BE11529\\0")
+ "Geo: X.XXX in/s\\0" (trigger threshold string)
Timestamp format detection:
data[11] == 0x10 10-byte sub_code=0x03 (continuous) format
data[12] == 0x10 9-byte sub_code=0x10 (single-shot) format
In contrast to Event (triggered records, type 0x46), MonitorLogEntry
records do NOT have a waveform record (SUB 0x0C) or bulk waveform stream
(SUB 5A). All available metadata is in the 0x0A header alone.
"""
index: int # 0-based position in device record list
key: str # 8-hex event key (e.g. "01114290") ✅
start_time: Optional[datetime.datetime] = None # monitoring session start ✅
stop_time: Optional[datetime.datetime] = None # monitoring session stop ✅
serial: Optional[str] = None # device serial (e.g. "BE11529") ✅
geo_threshold_ips: Optional[float] = None # trigger level from "Geo: X.XXX in/s" ✅
# Raw bytes for debugging / future decoding
raw_header: Optional[bytes] = field(default=None, repr=False)
@property
def duration_seconds(self) -> Optional[float]:
"""Duration of monitoring interval in seconds, or None if times unavailable."""
if self.start_time and self.stop_time:
return (self.stop_time - self.start_time).total_seconds()
return None
def __str__(self) -> str:
start = self.start_time.isoformat() if self.start_time else "?"
stop = self.stop_time.isoformat() if self.stop_time else "?"
dur = f" ({self.duration_seconds:.0f}s)" if self.duration_seconds is not None else ""
return f"MonitorLog#{self.index} key={self.key} {start}{stop}{dur}"
# ── MonitorStatus ─────────────────────────────────────────────────────────────
@dataclass
+362 -46
View File
@@ -57,7 +57,7 @@ SUB_POLL = 0x5B
SUB_SERIAL_NUMBER = 0x15
SUB_FULL_CONFIG = 0x01
SUB_EVENT_INDEX = 0x08
SUB_CHANNEL_CONFIG = 0x06
SUB_CHANNEL_CONFIG = 0x06 # Event storage range read (first/last key) ✅
SUB_MONITOR_STATUS = 0x1C # Monitoring status read (battery, memory, mode) ✅
SUB_EVENT_HEADER = 0x1E
SUB_EVENT_ADVANCE = 0x1F
@@ -65,6 +65,7 @@ SUB_WAVEFORM_HEADER = 0x0A
SUB_WAVEFORM_RECORD = 0x0C
SUB_BULK_WAVEFORM = 0x5A
SUB_COMPLIANCE = 0x1A
SUB_CALL_HOME = 0x2C # Call home config read → response 0xD3 ✅
SUB_UNKNOWN_2E = 0x2E
# Write command SUBs (= Read SUB + 0x60, confirmed from BW captures 3-11-26)
@@ -78,10 +79,20 @@ SUB_WRITE_CONFIRM_C = 0x74 # Confirm C — sent after 69 ✅
SUB_TRIGGER_CONFIG_WRITE = 0x82 # Write trigger config (0x22 + 0x60) ✅
SUB_TRIGGER_CONFIRM = 0x83 # Confirm trigger write ✅
# Call home write SUBs (confirmed from 4-20-26 call home settings captures)
SUB_CALL_HOME_WRITE = 0x7E # Write call home config → response 0x81 ✅
SUB_CALL_HOME_CONFIRM = 0x7F # Confirm call home write → response 0x80 ✅
# Monitoring control SUBs (confirmed from 4-8-26/2ndtry BW TX capture)
SUB_START_MONITORING = 0x96 # Start monitoring → response 0x69 ✅
SUB_STOP_MONITORING = 0x97 # Stop monitoring → response 0x68 ✅
# Erase-all SUBs (confirmed from 4-11-26 MITM capture)
# Both use token=0xFE at params[7] and return minimal 11-byte acks.
# Standard response formula applies: 0xFF - SUB.
SUB_ERASE_ALL_BEGIN = 0xA3 # Begin erase all events → response 0x5C ✅
SUB_ERASE_ALL_CONFIRM = 0xA2 # Confirm erase all events → response 0x5D ✅
# Hardcoded data lengths for the two-step read protocol.
#
# The S3 probe response page_key is always 0x0000 — it does NOT carry the
@@ -96,12 +107,14 @@ DATA_LENGTHS: dict[int, int] = {
SUB_SERIAL_NUMBER: 0x0A, # 10-byte serial number block ✅
SUB_FULL_CONFIG: 0x98, # 152-byte full config block ✅
SUB_EVENT_INDEX: 0x58, # 88-byte event index ✅
SUB_CHANNEL_CONFIG: 0x24, # 36-byte event storage range (first/last key) ✅
SUB_MONITOR_STATUS: 0x2C, # 44-byte monitor status block (idle) ✅
SUB_EVENT_HEADER: 0x08, # 8-byte event header (waveform key + event data) ✅
SUB_EVENT_ADVANCE: 0x08, # 8-byte next-key response ✅
# SUB_WAVEFORM_HEADER (0x0A) is VARIABLE — length read from probe response
# data[4]. Do NOT add it here; use read_waveform_header() instead. ✅
SUB_WAVEFORM_RECORD: 0xD2, # 210-byte waveform/histogram record ✅
SUB_CALL_HOME: 0x7C, # 124-byte call home config ✅ (confirmed 4-20-26)
SUB_UNKNOWN_2E: 0x1A, # 26 bytes, purpose TBD 🔶
0x09: 0xCA, # 202 bytes, purpose TBD 🔶
# SUB_COMPLIANCE (0x1A) uses a multi-step sequence with a 2090-byte total;
@@ -113,10 +126,12 @@ DATA_LENGTHS: dict[int, int] = {
_BULK_CHUNK_OFFSET = 0x1004 # offset field for probe + all regular chunk requests ✅
_BULK_TERM_OFFSET = 0x005A # offset field for termination request ✅
_BULK_COUNTER_STEP = 0x0400 # chunk counter increment per chunk ✅
# Chunk counter formula: chunk_num * 0x0400 for ALL chunks including chunk 1.
# Earlier captures showed 0x1004 for chunk 1 — that was a Blastware artifact, not a
# protocol requirement. Confirmed 2026-04-06: 0x0400 for chunk 1 works; 0x1004
# causes a 120-second device timeout. Formula n * 0x0400 is used for all chunks.
# Chunk counter formula: key4[2:4] + (chunk_num - 1) * 0x0400
# where key4[2:4] is the event's circular-buffer base offset ((key4[2]<<8)|key4[3]).
# Earlier captures showed 0x1004 for chunk 1 of key 01110000 — that was a Blastware
# artifact. For keys where key4[2:4] != 0x0000 (e.g. key 01111884) the old
# "n * 0x0400" formula sends counters from the wrong buffer region and the device
# returns data from a different event. Confirmed correct 2026-04-24.
# Default timeout values (seconds).
# MiniMate Plus is a slow device — keep these generous.
@@ -387,23 +402,32 @@ class MiniMateProtocol:
Send the SUB 0A (WAVEFORM_HEADER) two-step read for *key4*.
The data length for 0A is VARIABLE and must be read from the probe
response at data[4]. Two known values:
0x30 full histogram bin (has a waveform record to follow)
0x26 partial histogram bin (no waveform record)
response at data[4]. Two confirmed values:
0x46 (70) full triggered event (has 0C waveform record to follow)
0x2C (44) partial / monitor-log entry (no 0C record; 0A header only)
Args:
key4: 4-byte waveform record address from 1E or 1F.
Returns:
(header_bytes, record_length) where:
header_bytes raw data section starting at data[11]
record_length DATA_LENGTH read from probe (0x30 or 0x26)
(raw_data, record_length) where:
raw_data complete data_rsp.data bytes (full response payload)
record_length DATA_LENGTH read from probe (0x46 for full, 0x2C for partial)
The raw_data layout:
raw_data[0] = record type (0x46 = full triggered event, 0x2C = partial/monitor)
raw_data[1:5] = 0x00 × 4
raw_data[5:9] = event key (4 bytes)
raw_data[9:11] = 0x00 × 2
raw_data[11:] = timestamps + separator + serial + channel strings
(see MonitorLogEntry in models.py for full layout)
Raises:
ProtocolError: on timeout, bad checksum, or wrong response SUB.
Confirmed from 3-31-26 capture: 0A probe response data[4] carries
Confirmed from 4-11-26 MITM capture: 0A probe response data[4] carries
the variable length; data-request uses that length as the offset byte.
record_length == data[0] in virtually all cases (confirmed empirically).
"""
rsp_sub = _expected_rsp_sub(SUB_WAVEFORM_HEADER)
params = waveform_key_params(key4)
@@ -413,7 +437,7 @@ class MiniMateProtocol:
probe_rsp = self._recv_one(expected_sub=rsp_sub)
# Variable length — read from probe response data[4]
length = probe_rsp.data[4] if len(probe_rsp.data) > 4 else 0x30
length = probe_rsp.data[4] if len(probe_rsp.data) > 4 else 0x46
log.debug("read_waveform_header: 0A data request offset=0x%02X", length)
if length == 0:
@@ -422,12 +446,11 @@ class MiniMateProtocol:
self._send(build_bw_frame(SUB_WAVEFORM_HEADER, length, params))
data_rsp = self._recv_one(expected_sub=rsp_sub)
header_bytes = data_rsp.data[11:11 + length]
log.debug(
"read_waveform_header: key=%s length=0x%02X is_full=%s",
key4.hex(), length, length == 0x30,
key4.hex(), length, length >= 0x40,
)
return header_bytes, length
return data_rsp.data, length
def read_waveform_data_raw(self) -> bytes:
"""
@@ -505,7 +528,9 @@ class MiniMateProtocol:
*,
stop_after_metadata: bool = True,
max_chunks: int = 32,
) -> list[bytes]:
include_terminator: bool = False,
extra_chunks_after_metadata: int = 1,
) -> list[S3Frame]:
"""
Download the SUB 5A (BULK_WAVEFORM_STREAM) A5 frames for one event.
@@ -521,7 +546,9 @@ class MiniMateProtocol:
4. Termination (offset=_BULK_TERM_OFFSET, counter=last+_BULK_COUNTER_STEP)
Device responds with a final A5 frame (page_key=0x0000).
The termination frame (page_key=0x0000) is NOT included in the returned list.
By default the termination frame (page_key=0x0000) is NOT included in the
returned list. Pass include_terminator=True to append it; the blastware_file
writer needs the terminator frame's body to reconstruct the waveform file footer.
Args:
key4: 4-byte waveform key from EVENT_HEADER (1E).
@@ -531,11 +558,16 @@ class MiniMateProtocol:
hundred KB). Set False to download everything.
max_chunks: Safety cap on the number of chunk requests sent
(default 32; a typical event uses 9 large frames).
include_terminator: If True, append the terminator A5 frame
(page_key=0x0000) to the returned list. The
terminator carries the waveform file footer bytes.
Default False preserves existing caller behaviour.
Returns:
List of raw data bytes from each A5 response frame (not including
the terminator frame). Frame indices match the request sequence:
index 0 = probe response, index 1 = first chunk, etc.
List of S3Frame objects from each A5 response frame. Frame indices
match the request sequence: index 0 = probe response, index 1 = first
chunk, etc. If include_terminator=True, the last element is the
terminator frame (page_key=0x0000).
Raises:
ProtocolError: on timeout, bad checksum, or unexpected SUB.
@@ -550,16 +582,24 @@ class MiniMateProtocol:
raise ValueError(f"waveform key must be 4 bytes, got {len(key4)}")
rsp_sub = _expected_rsp_sub(SUB_BULK_WAVEFORM) # 0xFF - 0x5A = 0xA5
frames_data: list[bytes] = []
frames_data: list[S3Frame] = []
counter = 0
# BW counter formula (confirmed from 4-3-26 capture for key 0111245a,
# and empirical live-device test 2026-04-06 for key 01110000):
# counter for chunk n = max(key4[2:4], 0x0400) + (n - 1) * 0x0400
# key4[2:4] is the event's circular-buffer base offset. The max() guard
# ensures chunk 1 never uses counter=0x0000 (which equals the probe address
# and causes the device to re-return STRT record data for the first chunk).
_key4_offset = (key4[2] << 8) | key4[3]
# ── Step 1: probe ────────────────────────────────────────────────────
log.debug("5A probe key=%s", key4.hex())
log.debug("5A probe key=%s key4_offset=0x%04X", key4.hex(), _key4_offset)
params = bulk_waveform_params(key4, 0, is_probe=True)
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
self._parser.reset() # reset bytes_fed counter before probe recv
try:
rsp = self._recv_one(expected_sub=rsp_sub, reset_parser=False)
probe_batch = self._recv_5a_batch(rsp_sub)
except TimeoutError:
log.warning(
"5A probe TIMED OUT for key=%s"
@@ -567,23 +607,54 @@ class MiniMateProtocol:
key4.hex(), self._parser.bytes_fed,
)
raise
frames_data.append(rsp.data)
log.debug("5A A5[0] page_key=0x%04X %d bytes", rsp.page_key, len(rsp.data))
frames_data.extend(probe_batch)
log.debug(
"5A probe: %d frame(s) page_keys=%s",
len(probe_batch),
[f"0x{f.page_key:04X}" for f in probe_batch],
)
# Log probe frame size for diagnostics.
# The device always needs extra_chunks_after_metadata chunks after the
# metadata frame before termination to prime the valid waveform footer.
# This holds regardless of TCP frame size (1-frame vs 2-frame mode).
_effective_extra_chunks = extra_chunks_after_metadata
log.warning(
"5A probe data_len=%d effective_extra_chunks=%d",
len(probe_batch[0].data),
_effective_extra_chunks,
)
# ── Step 2: chunk loop ───────────────────────────────────────────────
# Chunk counters are monotonic: chunk_num * 0x0400 for all chunks.
# The 4-2-26 BW TX capture showed 0x1004 for chunk 1, but this is a
# Blastware artifact — the device accepts any counter value and streams
# data regardless. Empirically confirmed 2026-04-06: 0x0400 for chunk 1
# works; 0x1004 causes the device to ignore the frame (timeout).
# Counter formula: _chunk_base + (chunk_num - 1) * 0x0400
# where _chunk_base = max(key4[2:4], 0x0400).
#
# For events with key4[2:4] != 0 (e.g. key 0111245a, offset 0x245a):
# _chunk_base = 0x245a → chunk 1=0x245a, chunk 2=0x285a, ...
# Confirmed from 4-3-26 capture.
#
# For events with key4[2:4] == 0 (e.g. key 01110000):
# _chunk_base = max(0, 0x0400) = 0x0400
# → chunk 1=0x0400, chunk 2=0x0800, ... (= old chunk_num*0x0400)
# CRITICAL: counter=0x0000 (same as the probe) causes the device to
# re-return the STRT record data for chunk 1, making frame 1 look like
# a second probe response (confirmed from server log: frame 1 len=1097,
# contains STRT\xff\xfe, contributes zero body bytes after DLE-strip).
# counter=0x0400 for chunk 1 confirmed working (empirical test 2026-04-06).
_chunk_base = max(_key4_offset, _BULK_COUNTER_STEP)
for chunk_num in range(1, max_chunks + 1):
counter = chunk_num * _BULK_COUNTER_STEP
counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP
params = bulk_waveform_params(key4, counter)
log.debug("5A chunk %d counter=0x%04X", chunk_num, counter)
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
self._parser.reset() # reset bytes_fed for accurate per-chunk count
try:
rsp = self._recv_one(expected_sub=rsp_sub, reset_parser=False, timeout=10.0)
# Collect ALL frames from this chunk response.
# Over TCP via modem, a single large A5 device response (~1100 bytes
# RS-232) is split across ~2 TCP segments, each parsed as its own
# complete S3 frame. _recv_5a_batch gathers all of them so that
# every subsequent chunk request is paired with the correct response.
batch = self._recv_5a_batch(rsp_sub, first_timeout=10.0)
except TimeoutError:
raw = self._parser.bytes_fed
log.warning(
@@ -602,20 +673,51 @@ class MiniMateProtocol:
break
raise
log.warning(
"5A RX chunk=%d page_key=0x%04X data_len=%d contains_Project=%s",
chunk_num, rsp.page_key, len(rsp.data), b"Project:" in rsp.data,
)
# Process all frames from this batch.
metadata_found = False
for rsp in batch:
log.warning(
"5A RX chunk=%d page_key=0x%04X data_len=%d contains_Project=%s",
chunk_num, rsp.page_key, len(rsp.data), b"Project:" in rsp.data,
)
if rsp.page_key == 0x0000:
# Device unexpectedly terminated mid-stream.
log.debug("5A page_key=0x0000 — device terminated early")
if include_terminator:
frames_data.append(rsp)
return frames_data
frames_data.append(rsp)
if stop_after_metadata and b"Project:" in rsp.data:
metadata_found = True
if rsp.page_key == 0x0000:
# Device unexpectedly terminated mid-stream (no termination needed).
log.debug("5A A5[%d] page_key=0x0000 — device terminated early", chunk_num)
return frames_data
frames_data.append(rsp.data)
if stop_after_metadata and b"Project:" in rsp.data:
log.debug("5A A5[%d] metadata found — stopping early", chunk_num)
if metadata_found:
# Download extra_chunks_after_metadata more chunks after metadata.
# This primes the device to return the valid waveform footer in the
# termination response — without it the terminator carries too few bytes
# (confirmed 2026-04-23). The extra chunk data also belongs in the
# file body (confirmed from TCP capture analysis 2026-04-27).
log.debug("5A metadata found — fetching %d more chunk(s)",
_effective_extra_chunks)
for _extra_n in range(_effective_extra_chunks):
chunk_num += 1
counter = _chunk_base + (chunk_num - 1) * _BULK_COUNTER_STEP
params = bulk_waveform_params(key4, counter)
self._send(build_5a_frame(_BULK_CHUNK_OFFSET, params))
try:
extra_batch = self._recv_5a_batch(rsp_sub, first_timeout=10.0)
for ef in extra_batch:
log.debug(
"5A extra chunk page_key=0x%04X data_len=%d",
ef.page_key, len(ef.data),
)
if ef.page_key == 0x0000:
if include_terminator:
frames_data.append(ef)
return frames_data
frames_data.append(ef)
except TimeoutError:
log.debug("5A extra chunk %d timed out — end of stream", _extra_n + 1)
break
break
else:
log.warning(
@@ -637,6 +739,8 @@ class MiniMateProtocol:
"5A termination response page_key=0x%04X %d bytes",
term_rsp.page_key, len(term_rsp.data),
)
if include_terminator:
frames_data.append(term_rsp)
except TimeoutError:
log.debug("5A no termination response — device may have already closed")
@@ -1072,6 +1176,89 @@ class MiniMateProtocol:
self._send(frame)
return self.recv_write_ack(expected_sub=rsp_sub)
# ── Call home config (SUBs 0x2C / 0x7E / 0x7F) ──────────────────────────
def read_call_home_config(self) -> bytes:
"""
Read the auto call home configuration (SUB 0x2C response 0xD3).
Standard two-step read: probe (offset=0x00) then data (offset=0x7C=124).
Returns the raw 125-byte payload (data[11:] of the data response).
Confirmed from 4-20-26 call home settings capture:
- Probe response: data[4]=0x7C (confirms data length = 124)
- Data response: 136 bytes total (11-byte echo header + 125 bytes payload)
- Payload[0:3] = 0x00 0x7C 0xDC (header: zero, inner-length, constant)
- Payload[5] = auto_call_home_enabled
- Payload[6:46] = dial_string (40-byte null-padded ASCII "RADIO RING")
Returns:
Raw 125-byte call home config payload (data[11:]).
Suitable for round-trip write (append \\x00\\x00 127-byte write payload).
Raises:
ProtocolError: on timeout or wrong response SUB.
"""
rsp_sub = _expected_rsp_sub(SUB_CALL_HOME) # 0xFF - 0x2C = 0xD3
length = DATA_LENGTHS[SUB_CALL_HOME] # 0x7C = 124
log.debug("read_call_home_config: 0x2C probe")
self._send(build_bw_frame(SUB_CALL_HOME, 0))
self._recv_one(expected_sub=rsp_sub)
log.debug("read_call_home_config: 0x2C data request offset=0x%02X", length)
self._send(build_bw_frame(SUB_CALL_HOME, length))
data_rsp = self._recv_one(expected_sub=rsp_sub)
payload = data_rsp.data[11:]
log.debug("read_call_home_config: received %d payload bytes", len(payload))
return payload
def write_call_home_config(self, data: bytes) -> None:
"""
Write the auto call home configuration (SUB 0x7E 0x7F confirm).
Write sequence (confirmed from 4-20-26 call home settings captures):
SUB 0x7E write 127-byte payload device acks SUB 0x81
SUB 0x7F confirm (no data) device acks SUB 0x80
The 127-byte write payload = 125-byte read payload + b'\\x00\\x00'.
The offset field = data[1] + 2 = 0x7C + 2 = 0x7E = 126.
Write frame format: build_bw_write_frame (minimal DLE stuffing only
BW_CMD is doubled; all other bytes are RAW). The \\x10\\x03 sequence
within the payload is preserved as-is (device interprets DLE+ETX as the
literal value 0x03 per the inner-frame terminator convention).
Args:
data: 127-byte write payload (read payload + \\x00\\x00 footer).
Must start with [0x00][0x7C][...] (standard header).
Raises:
ValueError: if data is not exactly 127 bytes or lacks expected header.
ProtocolError: on timeout or wrong response SUB.
"""
if len(data) < 2:
raise ValueError(f"call home write payload must be at least 2 bytes, got {len(data)}")
rsp_sub_write = _expected_rsp_sub(SUB_CALL_HOME_WRITE) # 0xFF - 0x7E = 0x81
rsp_sub_confirm = _expected_rsp_sub(SUB_CALL_HOME_CONFIRM) # 0xFF - 0x7F = 0x80
# Offset formula: data[1] + 2 (same pattern as other single-chunk writes)
offset = data[1] + 2 # 0x7C + 2 = 0x7E = 126
frame = build_bw_write_frame(SUB_CALL_HOME_WRITE, data, offset=offset)
log.debug(
"write_call_home_config: %d bytes data[1]=0x%02X offset=0x%04X",
len(data), data[1], offset,
)
self._send(frame)
self.recv_write_ack(expected_sub=rsp_sub_write)
log.debug("write_call_home_config: write acked; sending confirm 0x7F")
confirm_frame = build_bw_write_frame(SUB_CALL_HOME_CONFIRM, b"")
self._send(confirm_frame)
self.recv_write_ack(expected_sub=rsp_sub_confirm)
log.debug("write_call_home_config: confirm acked — done")
# ── Monitoring ────────────────────────────────────────────────────────────
def read_monitor_status(self) -> S3Frame:
@@ -1137,6 +1324,78 @@ class MiniMateProtocol:
self._send(frame)
return self.recv_write_ack(expected_sub=rsp_sub)
def read_event_storage_range(self) -> S3Frame:
"""
Read event storage range (SUB 0x06 response 0xF9).
Two-step read: probe (offset=0x00) then data (offset=0x24 = 36 bytes).
Uses token=0xFE at params[7] same as the erase sequence.
The 36-byte response ends with two 4-byte event keys (first and last
stored event key). After a successful erase, both keys are 0x01110000
(device-empty sentinel). Confirmed from 4-11-26 MITM capture.
Returns:
S3Frame with 36 bytes of storage range data.
Raises:
ProtocolError: on timeout or wrong response SUB.
"""
rsp_sub = _expected_rsp_sub(SUB_CHANNEL_CONFIG) # 0xFF - 0x06 = 0xF9
params = token_params(0xFE)
log.debug("read_event_storage_range: probe step rsp_sub=0x%02X", rsp_sub)
self._send(build_bw_frame(SUB_CHANNEL_CONFIG, offset=0x00, params=params))
self._recv_one(expected_sub=rsp_sub)
log.debug(
"read_event_storage_range: data step offset=0x%02X",
DATA_LENGTHS[SUB_CHANNEL_CONFIG],
)
self._send(build_bw_frame(SUB_CHANNEL_CONFIG,
offset=DATA_LENGTHS[SUB_CHANNEL_CONFIG],
params=params))
return self._recv_one(expected_sub=rsp_sub)
def begin_erase_all(self) -> S3Frame:
"""
Send Begin-Erase-All command (SUB 0xA3 response 0x5C).
Single frame with token=0xFE at params[7]. The device acknowledges with
a minimal ack and begins the erase process. Follow up with
read_monitor_status() + read_event_storage_range() + confirm_erase_all()
to complete the sequence. Confirmed from 4-11-26 MITM capture.
Returns:
S3Frame ack from device (SUB 0x5C).
Raises:
ProtocolError: on timeout or wrong response SUB.
"""
rsp_sub = _expected_rsp_sub(SUB_ERASE_ALL_BEGIN) # 0xFF - 0xA3 = 0x5C
log.debug("begin_erase_all: rsp_sub=0x%02X", rsp_sub)
self._send(build_bw_frame(SUB_ERASE_ALL_BEGIN, params=token_params(0xFE)))
return self._recv_one(expected_sub=rsp_sub)
def confirm_erase_all(self) -> S3Frame:
"""
Send Confirm-Erase-All command (SUB 0xA2 response 0x5D).
Single frame with token=0xFE at params[7]. Must be preceded by
begin_erase_all() + read_monitor_status() + read_event_storage_range().
After this call the device memory is cleared. Confirmed from 4-11-26
MITM capture.
Returns:
S3Frame ack from device (SUB 0x5D).
Raises:
ProtocolError: on timeout or wrong response SUB.
"""
rsp_sub = _expected_rsp_sub(SUB_ERASE_ALL_CONFIRM) # 0xFF - 0xA2 = 0x5D
log.debug("confirm_erase_all: rsp_sub=0x%02X", rsp_sub)
self._send(build_bw_frame(SUB_ERASE_ALL_CONFIRM, params=token_params(0xFE)))
return self._recv_one(expected_sub=rsp_sub)
# ── Internal helpers ──────────────────────────────────────────────────────
def _send(self, frame: bytes) -> None:
@@ -1144,6 +1403,63 @@ class MiniMateProtocol:
log.debug("TX %d bytes: %s", len(frame), frame.hex())
self._transport.write(frame)
def _recv_5a_batch(
self,
expected_sub: int,
first_timeout: float = 10.0,
batch_timeout: float = 0.5,
) -> list[S3Frame]:
"""
Collect all S3 frames that arrive as part of one device response.
Over TCP via cellular modem, a single device A5 response (~1100 bytes of
RS-232 data) is forwarded in multiple TCP segments due to the modem's
data-forwarding timeout (~100-150 ms per segment). Each TCP segment
contains a complete, valid S3 frame (~550 bytes). Calling _recv_one()
once returns only the first segment's frame and misses the rest, causing
the chunk request/response pairing to cascade out of alignment.
This helper collects ALL frames before returning, by trying additional
short-timeout receives after the first frame arrives.
The caller must call self._parser.reset() before this method to ensure
bytes_fed is accurate; this method always uses reset_parser=False.
Args:
expected_sub: Expected SUB byte for validation.
first_timeout: Timeout for the mandatory first frame. Should be
generous (default 10 s) since the device may be slow.
batch_timeout: Short timeout for subsequent frames. Default 0.5 s
comfortably longer than the modem forwarding gap
(~150 ms) but short enough to avoid stalling when
only one frame is expected (probe, terminator).
Returns:
List of S3Frame objects in arrival order (at least one).
Raises:
TimeoutError: If no frame arrives within first_timeout.
UnexpectedResponse: If any frame has the wrong SUB byte.
"""
frames: list[S3Frame] = []
first = self._recv_one(
expected_sub=expected_sub,
reset_parser=False,
timeout=first_timeout,
)
frames.append(first)
while True:
try:
extra = self._recv_one(
expected_sub=expected_sub,
reset_parser=False,
timeout=batch_timeout,
)
frames.append(extra)
except TimeoutError:
break
return frames
def _recv_one(
self,
expected_sub: Optional[int] = None,
+36
View File
@@ -418,3 +418,39 @@ class TcpTransport(BaseTransport):
def __repr__(self) -> str:
state = "connected" if self.is_connected else "disconnected"
return f"TcpTransport({self.host!r}, port={self.port}, {state})"
# ── Inbound / accepted-socket transport ───────────────────────────────────────
class SocketTransport(TcpTransport):
"""
Like TcpTransport but wraps an already-accepted inbound socket.
Used by the ACH inbound server (bridges/ach_server.py) the device dials
IN to us, so by the time we create this transport the socket is already live.
connect() is a no-op; everything else (read, write, read_until_idle, ) is
inherited unchanged from TcpTransport.
Args:
sock: An already-connected socket.socket returned by server_socket.accept().
peer: Human-readable peer label for repr / logging (e.g. "203.0.113.5:54321").
"""
def __init__(self, sock: socket.socket, peer: str = "inbound") -> None:
# Bypass TcpTransport.__init__ — we already have a live socket.
self.host = peer
self.port = 0
self.connect_timeout = 0.0
self._sock = sock
sock.settimeout(self._RECV_TIMEOUT)
def connect(self) -> None:
"""No-op — socket was already accepted inbound."""
pass # Already have a live socket; nothing to open.
@property
def is_connected(self) -> bool:
return self._sock is not None
def __repr__(self) -> str:
return f"SocketTransport(peer={self.host!r})"
+28 -10
View File
@@ -33,7 +33,7 @@ STX = 0x02
ETX = 0x03
ACK = 0x41
__version__ = "0.2.2"
__version__ = "0.2.3"
@dataclass
@@ -227,17 +227,32 @@ def parse_s3(blob: bytes, trailer_len: int) -> List[Frame]:
trailer_end = trailer_start + trailer_len
trailer = blob[trailer_start:trailer_end]
# For S3 mode we don't assume checksum type here yet.
chk_valid = None
chk_type = None
chk_hex = None
payload = bytes(body)
if len(body) >= 1:
received_chk = body[-1]
computed_chk = checksum8_sum(bytes(body[:-1]))
if computed_chk == received_chk:
chk_valid = True
chk_type = "SUM8"
chk_hex = f"{received_chk:02x}"
payload = bytes(body[:-1])
else:
chk_valid = False
frames.append(Frame(
index=idx,
start_offset=start_offset,
end_offset=end_offset,
payload_raw=bytes(body),
payload=bytes(body),
payload=payload,
trailer=trailer,
checksum_valid=None,
checksum_type=None,
checksum_hex=None
checksum_valid=chk_valid,
checksum_type=chk_type,
checksum_hex=chk_hex
))
idx += 1
@@ -298,10 +313,13 @@ def parse_bw(blob: bytes, trailer_len: int, validate_checksum: bool) -> List[Fra
if b == ETX:
# Candidate end-of-frame.
# Accept ETX if the next bytes look like a real next-frame start (ACK+STX),
# or we're at EOF. This prevents chopping on in-payload 0x03.
next_is_start = (i + 2 < n and blob[i + 1] == ACK and blob[i + 2] == STX)
at_eof = (i == n - 1)
# Skip any SESSION_RESET (41 03) sequences — sent before POLL to wake
# monitoring units — to find the real next frame start (ACK+STX).
j = i + 1
while j + 1 < n and blob[j] == ACK and blob[j + 1] == ETX:
j += 2
next_is_start = (j + 1 < n and blob[j] == ACK and blob[j + 1] == STX)
at_eof = (i == n - 1) or (j >= n)
if not (next_is_start or at_eof):
# Not a real boundary -> payload byte
+20
View File
@@ -0,0 +1,20 @@
[build-system]
requires = ["setuptools>=68", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "seismo-relay"
version = "0.12.0"
description = "Python client and REST server for MiniMate Plus seismographs"
requires-python = ">=3.10"
dependencies = [
"fastapi>=0.104",
"uvicorn[standard]>=0.24",
"pyserial>=3.5",
"sqlalchemy>=2.0",
]
[tool.setuptools.packages.find]
# Auto-discovers minimateplus/, sfm/, bridges/ as packages
where = ["."]
include = ["minimateplus*", "sfm*", "bridges*"]
+593 -34
View File
@@ -97,16 +97,24 @@ class AnalyzerState:
class BridgePanel(tk.Frame):
"""
All bridge controls and live log output.
Calls on_bridge_started(raw_bw_path, raw_s3_path) when the bridge starts
so the parent can wire up the Analyzer.
Calls on_bridge_started(struct_bin_path) when the bridge starts.
Calls on_capture_started(bw_path, s3_path, label) when a capture begins.
Calls on_capture_complete(bw_path, s3_path, label) when a capture ends.
"""
def __init__(self, parent: tk.Widget, on_bridge_started, on_bridge_stopped, **kw):
def __init__(self, parent: tk.Widget, on_bridge_started, on_bridge_stopped,
on_capture_started=None, on_capture_complete=None, **kw):
super().__init__(parent, bg=BG2, **kw)
self._on_started = on_bridge_started # signature: (raw_bw, raw_s3, struct_bin)
self._on_stopped = on_bridge_stopped
self._on_started = on_bridge_started # signature: (struct_bin)
self._on_stopped = on_bridge_stopped
self._on_cap_started = on_capture_started # (bw, s3, label)
self._on_cap_complete = on_capture_complete # (bw, s3, label)
self.process: Optional[subprocess.Popen] = None
self._stdout_q: queue.Queue[str] = queue.Queue()
# Capture state
self._capturing = False
self._cap_label: Optional[str] = None
self._cap_history: list[dict] = [] # {label, status, bw, s3}
self._build()
self._poll_stdout()
@@ -146,17 +154,7 @@ class BridgePanel(tk.Frame):
tk.Button(cfg, text="Browse", bg=BG3, fg=FG, relief="flat", cursor="hand2",
font=MONO, command=self._choose_dir).grid(row=1, column=5, **pad)
# Row 2: raw taps (always enabled — timestamped names generated at start)
self._raw_bw_on = tk.BooleanVar(value=True)
self._raw_s3_on = tk.BooleanVar(value=True)
tk.Checkbutton(cfg, text="Capture BW->S3 raw", variable=self._raw_bw_on,
bg=BG2, fg=FG, selectcolor=BG3, activebackground=BG2,
font=MONO).grid(row=2, column=0, columnspan=2, sticky="w", **pad)
tk.Checkbutton(cfg, text="Capture S3->BW raw", variable=self._raw_s3_on,
bg=BG2, fg=FG, selectcolor=BG3, activebackground=BG2,
font=MONO).grid(row=2, column=2, columnspan=2, sticky="w", **pad)
# Row 3: buttons + status
# Row 2: buttons + status
btn_row = tk.Frame(self, bg=BG2)
btn_row.pack(side=tk.TOP, fill=tk.X, padx=4, pady=2)
@@ -170,6 +168,18 @@ class BridgePanel(tk.Frame):
command=self.stop_bridge, state="disabled")
self.stop_btn.pack(side=tk.LEFT, padx=4)
tk.Frame(btn_row, bg=BG2, width=16).pack(side=tk.LEFT) # spacer
self.cap_btn = tk.Button(btn_row, text="⬤ New Capture", bg=ORANGE, fg="#000000",
relief="flat", padx=10, cursor="hand2", font=MONO_B,
command=self._start_capture, state="disabled")
self.cap_btn.pack(side=tk.LEFT, padx=4)
self.stop_cap_btn = tk.Button(btn_row, text="■ Stop Capture", bg=BG3, fg=RED,
relief="flat", padx=10, cursor="hand2", font=MONO_B,
command=self._stop_capture, state="disabled")
self.stop_cap_btn.pack(side=tk.LEFT, padx=4)
self.mark_btn = tk.Button(btn_row, text="Add Mark", bg=BG3, fg=FG,
relief="flat", padx=10, cursor="hand2", font=MONO,
command=self.add_mark, state="disabled")
@@ -179,9 +189,34 @@ class BridgePanel(tk.Frame):
tk.Label(btn_row, textvariable=self.status_var,
bg=BG2, fg=FG_DIM, font=MONO).pack(side=tk.LEFT, padx=10)
# Capture history panel
hist_outer = tk.Frame(self, bg=BG2)
hist_outer.pack(side=tk.TOP, fill=tk.X, padx=4, pady=(2, 0))
tk.Label(hist_outer, text="Captures:", bg=BG2, fg=FG_DIM,
font=MONO_SM, anchor="w").pack(side=tk.LEFT, padx=(4, 6))
hist_inner = tk.Frame(hist_outer, bg=BG2)
hist_inner.pack(side=tk.LEFT, fill=tk.X, expand=True)
self._hist_lb = tk.Listbox(
hist_inner, bg=BG3, fg=FG, font=MONO_SM,
height=3, relief="flat", selectbackground=BG,
selectforeground=ACCENT, activestyle="none",
highlightthickness=0,
)
hist_vsb = ttk.Scrollbar(hist_inner, orient="vertical", command=self._hist_lb.yview)
self._hist_lb.configure(yscrollcommand=hist_vsb.set)
hist_vsb.pack(side=tk.RIGHT, fill=tk.Y)
self._hist_lb.pack(side=tk.LEFT, fill=tk.X, expand=True)
self._hist_lb.bind("<Double-Button-1>", self._on_hist_dblclick)
tk.Label(hist_outer, text="dbl-click to reload", bg=BG2, fg=FG_DIM,
font=MONO_SM, anchor="e").pack(side=tk.RIGHT, padx=6)
# Log output
self.log_view = scrolledtext.ScrolledText(
self, height=18, font=MONO_SM,
self, height=14, font=MONO_SM,
bg=BG, fg=FG, insertbackground=FG,
relief="flat", state="disabled",
)
@@ -221,14 +256,8 @@ class BridgePanel(tk.Frame):
args = [sys.executable, str(BRIDGE_PATH),
"--bw", bw, "--s3", s3, "--baud", baud, "--logdir", logdir]
raw_bw_path = raw_s3_path = None
if self._raw_bw_on.get():
raw_bw_path = os.path.join(logdir, f"raw_bw_{ts}.bin")
args += ["--raw-bw", raw_bw_path]
if self._raw_s3_on.get():
raw_s3_path = os.path.join(logdir, f"raw_s3_{ts}.bin")
args += ["--raw-s3", raw_s3_path]
# Raw BW/S3 taps are NOT opened at bridge start.
# Use "New Capture" to start a labeled tap on demand.
# Structured bin path — written by bridge automatically, named by ts
struct_bin_path = os.path.join(logdir, f"s3_session_{ts}.bin")
@@ -250,11 +279,12 @@ class BridgePanel(tk.Frame):
self.status_var.set(f"Running — {bw} <-> {s3}")
self.start_btn.configure(state="disabled")
self.stop_btn.configure(state="normal", bg=RED)
self.mark_btn.configure(state="normal")
self.cap_btn.configure(state="normal")
self._append_log(f"== Bridge started [{ts}] ==\n")
self._append_log(" Click 'New Capture' when ready to record a setting change.\n")
# Notify parent so Analyzer can wire up live mode
self._on_started(raw_bw_path, raw_s3_path, struct_bin_path)
# Notify parent — no raw files yet, just the structured bin path
self._on_started(struct_bin_path)
def stop_bridge(self) -> None:
if self.process and self.process.poll() is None:
@@ -270,7 +300,11 @@ class BridgePanel(tk.Frame):
self.status_var.set("Stopped")
self.start_btn.configure(state="normal")
self.stop_btn.configure(state="disabled", bg=BG3)
self.cap_btn.configure(state="disabled")
self.stop_cap_btn.configure(state="disabled", bg=BG3)
self.mark_btn.configure(state="disabled")
self._capturing = False
self._cap_label = None
self._append_log("== Bridge stopped ==\n")
def _reader_thread(self) -> None:
@@ -288,12 +322,120 @@ class BridgePanel(tk.Frame):
self._bridge_ended()
self._on_stopped()
break
stripped = line.strip()
# Handle capture lifecycle events from bridge
if stripped.startswith("[CAP_START] ") and "\t" in stripped:
parts = stripped[12:].split("\t", 1)
if len(parts) == 2:
bw_path, s3_path = parts[0].strip(), parts[1].strip()
self._on_cap_started_msg(bw_path, s3_path)
elif stripped.startswith("[CAP_STOP] ") and "\t" in stripped:
parts = stripped[11:].split("\t", 1)
if len(parts) == 2:
bw_path, s3_path = parts[0].strip(), parts[1].strip()
self._on_cap_stopped_msg(bw_path, s3_path)
self._append_log(line)
except queue.Empty:
pass
finally:
self.after(100, self._poll_stdout)
# ── capture control ───────────────────────────────────────────────────
def _start_capture(self) -> None:
"""Ask for a label and tell the bridge to start writing raw tap files."""
if not self.process or self.process.poll() is not None:
return
label = simpledialog.askstring(
"New Capture",
"Label for this capture\n(e.g. 'recording_mode_continuous').\nLeave blank for timestamp only:",
parent=self,
)
if label is None:
return # user hit Cancel
label = label.strip()
try:
self.process.stdin.write(f"CAP_START:{label}\n")
self.process.stdin.flush()
except Exception as e:
messagebox.showerror("Error", f"Failed to start capture:\n{e}")
return
self._capturing = True
self._cap_label = label or datetime.datetime.now().strftime("%H%M%S")
self.cap_btn.configure(state="disabled")
self.stop_cap_btn.configure(state="normal", bg=RED)
self.mark_btn.configure(state="normal")
self._append_log(f"[CAPTURE] Starting: {self._cap_label!r}...\n")
# Add to history as recording (paths filled in when [CAP_START] arrives)
self._cap_history.append({"label": self._cap_label, "status": "recording",
"bw": None, "s3": None})
self._refresh_hist()
def _stop_capture(self) -> None:
"""Tell the bridge to flush and close the current raw tap files."""
if not self.process or self.process.poll() is not None:
return
try:
self.process.stdin.write("CAP_STOP\n")
self.process.stdin.flush()
except Exception as e:
messagebox.showerror("Error", f"Failed to stop capture:\n{e}")
# UI is updated when [CAP_STOP] arrives in stdout
def _on_cap_started_msg(self, bw_path: str, s3_path: str) -> None:
"""Called when bridge confirms capture has started (files are open)."""
# Fill in paths for the last 'recording' history entry
for entry in reversed(self._cap_history):
if entry["status"] == "recording" and entry["bw"] is None:
entry["bw"] = bw_path
entry["s3"] = s3_path
break
if self._on_cap_started:
self._on_cap_started(bw_path, s3_path, self._cap_label or "")
def _on_cap_stopped_msg(self, bw_path: str, s3_path: str) -> None:
"""Called when bridge confirms capture has stopped (files are closed)."""
label = self._cap_label or "capture"
# Mark history entry as done
for entry in reversed(self._cap_history):
if entry["status"] == "recording":
entry["status"] = "done"
entry["bw"] = bw_path
entry["s3"] = s3_path
break
self._refresh_hist()
self._capturing = False
self._cap_label = None
self.cap_btn.configure(state="normal")
self.stop_cap_btn.configure(state="disabled", bg=BG3)
self._append_log(f"[CAPTURE] Done: {label!r} — ready in Analyzer\n")
if self._on_cap_complete:
self._on_cap_complete(bw_path, s3_path, label)
def _refresh_hist(self) -> None:
self._hist_lb.delete(0, tk.END)
for entry in self._cap_history:
icon = "🔴" if entry["status"] == "recording" else ""
label = entry["label"] or "(unlabeled)"
self._hist_lb.insert(tk.END, f" {icon} {label}")
if self._cap_history:
self._hist_lb.see(tk.END)
def _on_hist_dblclick(self, _e=None) -> None:
sel = self._hist_lb.curselection()
if not sel:
return
entry = self._cap_history[sel[0]]
if entry["status"] == "done" and entry["bw"] and entry["s3"]:
if self._on_cap_complete:
self._on_cap_complete(entry["bw"], entry["s3"], entry["label"])
# ── mark ──────────────────────────────────────────────────────────────
def add_mark(self) -> None:
if not self.process or not self.process.stdin or self.process.poll() is not None:
return
@@ -1071,6 +1213,398 @@ class AnalyzerPanel(tk.Frame):
# ─────────────────────────────────────────────────────────────────────────────
# ─────────────────────────────────────────────────────────────────────────────
# Serial Watch panel — tap the RS-232 line between device and modem
# ─────────────────────────────────────────────────────────────────────────────
try:
import serial as _serial
from serial.tools import list_ports as _list_ports
_SERIAL_OK = True
except ImportError:
_SERIAL_OK = False
from minimateplus.framing import S3FrameParser as _S3FrameParser # noqa: E402
_SW_KNOWN_SUBS = {
0xA4: "POLL_RSP", 0xA5: "BULK_WAVEFORM_RSP", 0xE0: "ADV_EVENT_RSP",
0xE1: "EVT_IDX_FIRST_RSP", 0xE3: "MONITOR_STATUS_RSP", 0xEA: "SERIAL_NUM_RSP",
0xF3: "WAVEFORM_REC_RSP", 0xF5: "WAVEFORM_HDR_RSP", 0xF7: "EVENT_INDEX_RSP",
0xF9: "UNK_06_RSP", 0xFE: "DEVICE_INFO_RSP",
0x69: "START_MON_ACK", 0x68: "STOP_MON_ACK",
}
class SerialWatchPanel(tk.Frame):
"""
Tap the RS-232 line between the MiniMate Plus and its modem (RV50/RV55).
Runs the serial reader in a background thread; surfaces parsed S3 frames
live in the log view. Writes raw_s3_<ts>.bin compatible with Analyzer.
Typical use for call-home capture:
1. Connect a USB-to-serial tap to the RS-232 line.
2. Pick that COM port here, click Start.
3. Wait for the unit to trigger / call home.
4. Click Stop, then 'Open in Analyzer' to inspect the frames.
"""
_COL_FRAME = "#4ec9b0" # teal — parsed S3 frame
_COL_CTRL = "#dcdcaa" # yellow — control-line change
_COL_AT = "#9cdcfe" # blue — AT command / ASCII noise
_COL_ERR = "#f44747" # red — error
def __init__(self, parent: tk.Widget, on_capture_ready=None, **kw):
"""
on_capture_ready(raw_s3_path: str) called when capture stops,
so the parent can inject the file into the Analyzer.
"""
super().__init__(parent, bg=BG2, **kw)
self._on_capture_ready = on_capture_ready
self._serial: Optional[object] = None # serial.Serial instance
self._reader_thread: Optional[threading.Thread] = None
self._stop_evt = threading.Event()
self._log_q: queue.Queue[tuple[str, str]] = queue.Queue() # (text, colour)
self._raw_fh = None # open binary file handle
self._raw_path: Optional[str] = None
self._frame_count = 0
self._build()
self._poll_log_queue()
# ── build ─────────────────────────────────────────────────────────────
def _build(self) -> None:
pad = {"padx": 6, "pady": 4}
cfg = tk.Frame(self, bg=BG2)
cfg.pack(side=tk.TOP, fill=tk.X, padx=4, pady=4)
# Row 0 — port picker
tk.Label(cfg, text="COM port:", bg=BG2, fg=FG, font=MONO
).grid(row=0, column=0, sticky="e", **pad)
self._port_var = tk.StringVar()
self._port_cb = ttk.Combobox(cfg, textvariable=self._port_var,
width=12, font=MONO, state="normal")
self._port_cb.grid(row=0, column=1, sticky="w", **pad)
tk.Button(cfg, text="", bg=BG3, fg=FG, relief="flat", cursor="hand2",
font=MONO, command=self._refresh_ports
).grid(row=0, column=2, **pad)
tk.Label(cfg, text=" Baud:", bg=BG2, fg=FG, font=MONO
).grid(row=0, column=3, sticky="e", **pad)
self._baud_var = tk.StringVar(value="38400")
tk.Entry(cfg, textvariable=self._baud_var, width=8,
bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO
).grid(row=0, column=4, sticky="w", **pad)
self._ack_ok_var = tk.BooleanVar(value=False)
tk.Checkbutton(cfg, text="Ack OK to AT commands",
variable=self._ack_ok_var,
bg=BG2, fg=FG, selectcolor=BG3, activebackground=BG2,
font=MONO).grid(row=0, column=5, sticky="w", **pad)
# Row 1 — capture dir
tk.Label(cfg, text="Save to:", bg=BG2, fg=FG, font=MONO
).grid(row=1, column=0, sticky="e", **pad)
self._dir_var = tk.StringVar(
value=str(SCRIPT_DIR / "bridges" / "captures"))
tk.Entry(cfg, textvariable=self._dir_var, width=40,
bg=BG3, fg=FG, insertbackground=FG, relief="flat", font=MONO
).grid(row=1, column=1, columnspan=4, sticky="we", **pad)
tk.Button(cfg, text="Browse", bg=BG3, fg=FG, relief="flat",
cursor="hand2", font=MONO, command=self._choose_dir
).grid(row=1, column=5, **pad)
# Button row
btn_row = tk.Frame(self, bg=BG2)
btn_row.pack(side=tk.TOP, fill=tk.X, padx=4, pady=2)
self._start_btn = tk.Button(
btn_row, text="Start Watch", bg=GREEN, fg="#000000",
relief="flat", padx=12, cursor="hand2", font=MONO_B,
command=self._start)
self._start_btn.pack(side=tk.LEFT, padx=6)
self._stop_btn = tk.Button(
btn_row, text="Stop", bg=BG3, fg=FG,
relief="flat", padx=12, cursor="hand2", font=MONO,
command=self._stop, state="disabled")
self._stop_btn.pack(side=tk.LEFT, padx=4)
self._analyzer_btn = tk.Button(
btn_row, text="Open in Analyzer", bg=BG3, fg=FG,
relief="flat", padx=10, cursor="hand2", font=MONO,
command=self._send_to_analyzer, state="disabled")
self._analyzer_btn.pack(side=tk.LEFT, padx=4)
tk.Button(btn_row, text="Clear", bg=BG3, fg=FG,
relief="flat", padx=8, cursor="hand2", font=MONO,
command=self._clear_log).pack(side=tk.LEFT, padx=4)
self._status_var = tk.StringVar(value="Idle")
tk.Label(btn_row, textvariable=self._status_var,
bg=BG2, fg=FG_DIM, font=MONO).pack(side=tk.LEFT, padx=10)
# Log view
self._log = scrolledtext.ScrolledText(
self, height=24, font=MONO_SM,
bg=BG, fg=FG, insertbackground=FG,
relief="flat", state="disabled",
)
self._log.pack(fill=tk.BOTH, expand=True, padx=4, pady=4)
self._log.tag_config("frame", foreground=self._COL_FRAME)
self._log.tag_config("ctrl", foreground=self._COL_CTRL)
self._log.tag_config("at", foreground=self._COL_AT)
self._log.tag_config("err", foreground=self._COL_ERR)
self._log.tag_config("dim", foreground=FG_DIM)
# Populate ports on first load
self._refresh_ports()
# ── port helpers ──────────────────────────────────────────────────────
def _refresh_ports(self) -> None:
if not _SERIAL_OK:
self._port_cb["values"] = ["(pyserial not installed)"]
return
ports = [p.device for p in _list_ports.comports()]
self._port_cb["values"] = ports
if ports and not self._port_var.get():
self._port_var.set(ports[0])
def _choose_dir(self) -> None:
d = filedialog.askdirectory(initialdir=self._dir_var.get())
if d:
self._dir_var.set(d)
# ── start / stop ──────────────────────────────────────────────────────
def _start(self) -> None:
if not _SERIAL_OK:
messagebox.showerror(
"pyserial missing",
"Install pyserial first:\n pip install pyserial")
return
port = self._port_var.get().strip()
if not port or "not installed" in port:
messagebox.showerror("Error", "Select a valid COM port first.")
return
try:
baud = int(self._baud_var.get().strip())
except ValueError:
messagebox.showerror("Error", "Invalid baud rate.")
return
# Open output files
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
out_dir = Path(self._dir_var.get()) / f"serial_{ts}"
out_dir.mkdir(parents=True, exist_ok=True)
self._raw_path = str(out_dir / f"raw_s3_{ts}.bin")
try:
self._raw_fh = open(self._raw_path, "wb")
except OSError as exc:
messagebox.showerror("Error", f"Cannot open capture file:\n{exc}")
return
# Open serial port
try:
ser = _serial.Serial(
port=port, baudrate=baud,
bytesize=8, parity=_serial.PARITY_NONE,
stopbits=_serial.STOPBITS_ONE,
timeout=0.05, write_timeout=0,
)
ser.setDTR(True)
ser.setRTS(True)
except Exception as exc:
self._raw_fh.close()
self._raw_fh = None
messagebox.showerror("Error", f"Cannot open {port}:\n{exc}")
return
self._serial = ser
self._stop_evt.clear()
self._frame_count = 0
self._analyzer_btn.configure(state="disabled")
self._reader_thread = threading.Thread(
target=self._reader_loop,
args=(ser, baud),
daemon=True,
)
self._reader_thread.start()
self._status_var.set(f"Watching {port} @ {baud}")
self._start_btn.configure(state="disabled")
self._stop_btn.configure(state="normal", bg=RED)
self._append(f"── Serial watch started {port} @ {baud} [{ts}] ──\n", "dim")
self._append(f" Capture: {self._raw_path}\n", "dim")
self._append(" Waiting for data…\n\n", "dim")
def _stop(self) -> None:
self._stop_evt.set()
if self._serial:
try:
self._serial.close()
except Exception:
pass
self._serial = None
if self._raw_fh:
self._raw_fh.close()
self._raw_fh = None
self._status_var.set("Stopped")
self._start_btn.configure(state="normal")
self._stop_btn.configure(state="disabled", bg=BG3)
if self._raw_path and Path(self._raw_path).exists():
self._analyzer_btn.configure(state="normal")
self._append("\n── Watch stopped ──\n", "dim")
# ── reader thread ─────────────────────────────────────────────────────
def _reader_loop(self, ser, baud: int) -> None:
parser = _S3FrameParser()
rx_buf = bytearray()
ack_ok = self._ack_ok_var.get()
# Monitor control lines in a sub-thread
ctrl_stop = threading.Event()
ctrl_thread = threading.Thread(
target=self._ctrl_loop, args=(ser, ctrl_stop), daemon=True)
ctrl_thread.start()
try:
while not self._stop_evt.is_set():
try:
data = ser.read(4096)
except Exception as exc:
self._log_q.put((f"Read error: {exc}\n", "err"))
break
if not data:
continue
# Save raw bytes
if self._raw_fh:
try:
self._raw_fh.write(data)
self._raw_fh.flush()
except Exception:
pass
# Parse S3 frames
for byte in data:
result = parser.feed(bytes([byte]))
if result:
frames = result if isinstance(result, list) else [result]
for f in frames:
self._frame_count += 1
name = _SW_KNOWN_SUBS.get(f.sub, f"UNK_0x{f.sub:02X}")
chk = "" if f.checksum_valid else "✗ BAD_CHK"
peek = f.data[:32].hex() + ("" if len(f.data) > 32 else "")
msg = (
f"[{self._frame_count:04d}] "
f"SUB=0x{f.sub:02X} ({name:<22}) "
f"page=0x{f.page_key:04X} "
f"data={len(f.data):4d}B {chk}\n"
f" {peek}\n"
)
self._log_q.put((msg, "frame"))
# AT command handling for --ack-ok mode
if ack_ok:
rx_buf.extend(data)
while b"\r" in rx_buf or b"\n" in rx_buf:
for sep in (b"\r", b"\n"):
idx = rx_buf.find(sep)
if idx != -1:
line_bytes = bytes(rx_buf[:idx])
del rx_buf[:idx + 1]
break
else:
break
line_str = line_bytes.decode("latin1", errors="ignore").strip()
if line_str.upper().startswith("AT"):
self._log_q.put((f"AT: {line_str!r}\n", "at"))
if not line_str.upper().startswith("ATDT"):
try:
ser.write(b"\r\nOK\r\n")
ser.flush()
self._log_q.put((f" → OK\n", "at"))
except Exception:
pass
finally:
ctrl_stop.set()
ctrl_thread.join(timeout=0.5)
# Signal the main thread that the reader ended naturally
if not self._stop_evt.is_set():
self._log_q.put(("<<done>>", ""))
def _ctrl_loop(self, ser, stop: threading.Event) -> None:
prev = {}
try:
prev = dict(CTS=ser.cts, DSR=ser.dsr, DCD=ser.cd)
try:
prev["RI"] = ser.ri
except Exception:
prev["RI"] = None
except Exception:
return
while not stop.is_set():
try:
cur = dict(CTS=ser.cts, DSR=ser.dsr, DCD=ser.cd, RI=None)
try:
cur["RI"] = ser.ri
except Exception:
pass
for name, val in cur.items():
if val != prev.get(name):
self._log_q.put((f"CTRL {name}{val}\n", "ctrl"))
prev[name] = val
except Exception:
break
stop.wait(0.2)
# ── log view ──────────────────────────────────────────────────────────
def _poll_log_queue(self) -> None:
try:
while True:
text, tag = self._log_q.get_nowait()
if text == "<<done>>":
self._stop()
break
self._append(text, tag)
except queue.Empty:
pass
finally:
self.after(80, self._poll_log_queue)
def _append(self, text: str, tag: str = "") -> None:
self._log.configure(state="normal")
if tag:
self._log.insert(tk.END, text, tag)
else:
self._log.insert(tk.END, text)
self._log.see(tk.END)
self._log.configure(state="disabled")
def _clear_log(self) -> None:
self._log.configure(state="normal")
self._log.delete("1.0", tk.END)
self._log.configure(state="disabled")
# ── send to analyzer ──────────────────────────────────────────────────
def _send_to_analyzer(self) -> None:
if self._raw_path and self._on_capture_ready:
self._on_capture_ready(self._raw_path)
# Console panel (tk.Frame — lives inside a notebook tab)
# ─────────────────────────────────────────────────────────────────────────────
@@ -1492,6 +2026,8 @@ class SeismoLab(tk.Tk):
nb,
on_bridge_started=self._on_bridge_started,
on_bridge_stopped=self._on_bridge_stopped,
on_capture_started=self._on_capture_started,
on_capture_complete=self._on_capture_complete,
)
nb.add(self._bridge_panel, text=" Bridge ")
@@ -1504,26 +2040,49 @@ class SeismoLab(tk.Tk):
)
nb.add(self._console_panel, text=" Console ")
self._serial_watch_panel = SerialWatchPanel(
nb,
on_capture_ready=self._on_serial_capture_ready,
)
nb.add(self._serial_watch_panel, text=" Serial Watch ")
self._nb = nb
self.protocol("WM_DELETE_WINDOW", self._on_close)
def _on_bridge_started(self, raw_bw: Optional[str], raw_s3: Optional[str],
struct_bin: Optional[str] = None) -> None:
"""Bridge started — inject paths into analyzer and start live mode."""
self._analyzer_panel.set_live_files(raw_bw, raw_s3, struct_bin)
# Switch to Analyzer tab so the user can watch it update
self._nb.select(1)
def _on_bridge_started(self, struct_bin: Optional[str] = None) -> None:
"""Bridge started — stash the structured bin path; stay on Bridge tab."""
if struct_bin:
self._analyzer_panel.bin_var.set(struct_bin)
def _on_bridge_stopped(self) -> None:
self._analyzer_panel.stop_live()
def _on_capture_started(self, bw_path: str, s3_path: str, label: str) -> None:
"""A capture began — wire up live mode in the Analyzer and switch tabs."""
self._analyzer_panel.set_live_files(bw_path, s3_path)
self._nb.select(1)
def _on_capture_complete(self, bw_path: str, s3_path: str, label: str) -> None:
"""A capture stopped — stop live mode, run full analysis, switch to Analyzer."""
self._analyzer_panel.stop_live()
self._analyzer_panel.s3_var.set(s3_path)
self._analyzer_panel.bw_var.set(bw_path)
self._analyzer_panel._run_analyze()
self._nb.select(1)
def _on_console_send_to_analyzer(self, raw_s3_path: str) -> None:
"""Console captured bytes → inject into Analyzer S3 field and switch tab."""
self._analyzer_panel.s3_var.set(raw_s3_path)
self._nb.select(1)
def _on_serial_capture_ready(self, raw_s3_path: str) -> None:
"""Serial Watch capture finished → inject into Analyzer and switch tab."""
self._analyzer_panel.s3_var.set(raw_s3_path)
self._nb.select(1)
def _on_close(self) -> None:
self._bridge_panel.stop_bridge()
self._serial_watch_panel._stop()
self.destroy()
+486
View File
@@ -0,0 +1,486 @@
"""
sfm/database.py SQLite persistence layer for seismo-relay.
Three tables, all keyed by unit serial number:
ach_sessions one row per inbound ACH call-home
events one row per triggered waveform event (deduped by serial+timestamp)
monitor_log one row per monitoring interval (deduped by serial+start_time)
The DB file lives at:
<output_dir>/seismo_relay.db (default: bridges/captures/seismo_relay.db)
Usage
-----
from sfm.database import SeismoDb
db = SeismoDb("bridges/captures/seismo_relay.db")
# Write a call-home session
session_id = db.insert_ach_session(serial="BE11529", peer="1.2.3.4:51920",
events_downloaded=3, monitor_entries=2,
duration_seconds=47.3)
# Write events (silently skips duplicates)
db.insert_events(events, serial="BE11529", session_id=session_id)
# Write monitor log entries
db.insert_monitor_log(entries, session_id=session_id)
# Query
rows = db.query_events(serial="BE11529", from_dt=datetime(...), to_dt=datetime(...))
"""
from __future__ import annotations
import datetime
import logging
import sqlite3
import uuid
from pathlib import Path
from typing import Optional
from minimateplus.models import Event, MonitorLogEntry
log = logging.getLogger("sfm.database")
# ── Schema ─────────────────────────────────────────────────────────────────────
_SCHEMA = """
PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
CREATE TABLE IF NOT EXISTS ach_sessions (
id TEXT PRIMARY KEY, -- UUID
serial TEXT NOT NULL,
session_time TEXT NOT NULL, -- ISO-8601 UTC
peer TEXT, -- "ip:port"
events_downloaded INTEGER NOT NULL DEFAULT 0,
monitor_entries INTEGER NOT NULL DEFAULT 0,
duration_seconds REAL,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);
CREATE INDEX IF NOT EXISTS idx_ach_sessions_serial ON ach_sessions(serial);
CREATE INDEX IF NOT EXISTS idx_ach_sessions_time ON ach_sessions(session_time);
CREATE TABLE IF NOT EXISTS events (
id TEXT PRIMARY KEY, -- UUID
serial TEXT NOT NULL,
waveform_key TEXT NOT NULL, -- 8-hex device key (dedup field)
session_id TEXT, -- FK ach_sessions.id
timestamp TEXT, -- ISO-8601 local time from device
tran_ppv REAL, -- in/s
vert_ppv REAL, -- in/s
long_ppv REAL, -- in/s
peak_vector_sum REAL, -- in/s
mic_ppv REAL, -- psi or dB depending on setup
project TEXT,
client TEXT,
operator TEXT,
sensor_location TEXT,
sample_rate INTEGER,
record_type TEXT, -- "single_shot" | "continuous"
false_trigger INTEGER NOT NULL DEFAULT 0, -- 0=no, 1=yes (manual flag)
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
UNIQUE(serial, timestamp)
);
CREATE INDEX IF NOT EXISTS idx_events_serial ON events(serial);
CREATE INDEX IF NOT EXISTS idx_events_timestamp ON events(timestamp);
CREATE INDEX IF NOT EXISTS idx_events_session ON events(session_id);
CREATE TABLE IF NOT EXISTS monitor_log (
id TEXT PRIMARY KEY, -- UUID
serial TEXT NOT NULL,
waveform_key TEXT NOT NULL, -- 8-hex device key (dedup field)
session_id TEXT, -- FK ach_sessions.id
start_time TEXT, -- ISO-8601
stop_time TEXT, -- ISO-8601
duration_seconds REAL,
geo_threshold_ips REAL, -- in/s
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
UNIQUE(serial, start_time)
);
CREATE INDEX IF NOT EXISTS idx_monitor_log_serial ON monitor_log(serial);
CREATE INDEX IF NOT EXISTS idx_monitor_log_start ON monitor_log(start_time);
CREATE INDEX IF NOT EXISTS idx_monitor_log_session ON monitor_log(session_id);
"""
# ── SeismoDb class ─────────────────────────────────────────────────────────────
class SeismoDb:
"""
Thin SQLite wrapper for seismo-relay persistence.
Thread-safe: each call opens, uses, and closes a connection with
check_same_thread=False and WAL mode enabled. For the ACH server's
single-writer / occasional-reader pattern this is more than sufficient.
"""
def __init__(self, db_path: str | Path) -> None:
self.db_path = Path(db_path)
self.db_path.parent.mkdir(parents=True, exist_ok=True)
self._init_schema()
log.info("SeismoDb initialised at %s", self.db_path)
# ── Internal helpers ───────────────────────────────────────────────────────
def _connect(self) -> sqlite3.Connection:
conn = sqlite3.connect(str(self.db_path), check_same_thread=False)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA journal_mode = WAL")
conn.execute("PRAGMA foreign_keys = ON")
return conn
def _init_schema(self) -> None:
with self._connect() as conn:
conn.executescript(_SCHEMA)
self._migrate(conn)
def _migrate(self, conn: sqlite3.Connection) -> None:
"""Apply in-place schema migrations for existing databases."""
# Migration 1: change events UNIQUE from (serial, waveform_key) [or any
# waveform_key-based variant] to (serial, timestamp).
# Rationale: device key counter resets to 01110000 after every erase, so
# waveform_key is not a stable dedup field across erase cycles. The event
# timestamp (from the device clock) is the correct natural key.
row = conn.execute(
"SELECT sql FROM sqlite_master WHERE type='table' AND name='events'"
).fetchone()
if row and "UNIQUE(serial, timestamp)" not in row[0]:
log.info("_migrate: rebuilding events table — UNIQUE(serial, timestamp)")
conn.executescript("""
ALTER TABLE events RENAME TO events_old;
CREATE TABLE events (
id TEXT PRIMARY KEY,
serial TEXT NOT NULL,
waveform_key TEXT NOT NULL,
session_id TEXT,
timestamp TEXT,
tran_ppv REAL,
vert_ppv REAL,
long_ppv REAL,
peak_vector_sum REAL,
mic_ppv REAL,
project TEXT,
client TEXT,
operator TEXT,
sensor_location TEXT,
sample_rate INTEGER,
record_type TEXT,
false_trigger INTEGER NOT NULL DEFAULT 0,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
UNIQUE(serial, timestamp)
);
INSERT OR IGNORE INTO events SELECT * FROM events_old;
DROP TABLE events_old;
CREATE INDEX IF NOT EXISTS idx_events_serial ON events(serial);
CREATE INDEX IF NOT EXISTS idx_events_timestamp ON events(timestamp);
CREATE INDEX IF NOT EXISTS idx_events_session ON events(session_id);
""")
log.info("_migrate: events table rebuilt OK")
# Migration 2: change monitor_log UNIQUE from (serial, waveform_key) to
# (serial, start_time) — same reasoning as events.
row = conn.execute(
"SELECT sql FROM sqlite_master WHERE type='table' AND name='monitor_log'"
).fetchone()
if row and "UNIQUE(serial, start_time)" not in row[0]:
log.info("_migrate: rebuilding monitor_log table — UNIQUE(serial, start_time)")
conn.executescript("""
ALTER TABLE monitor_log RENAME TO monitor_log_old;
CREATE TABLE monitor_log (
id TEXT PRIMARY KEY,
serial TEXT NOT NULL,
waveform_key TEXT NOT NULL,
session_id TEXT,
start_time TEXT,
stop_time TEXT,
duration_seconds REAL,
geo_threshold_ips REAL,
created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now')),
UNIQUE(serial, start_time)
);
INSERT OR IGNORE INTO monitor_log SELECT * FROM monitor_log_old;
DROP TABLE monitor_log_old;
CREATE INDEX IF NOT EXISTS idx_monitor_log_serial ON monitor_log(serial);
CREATE INDEX IF NOT EXISTS idx_monitor_log_start ON monitor_log(start_time);
CREATE INDEX IF NOT EXISTS idx_monitor_log_session ON monitor_log(session_id);
""")
log.info("_migrate: monitor_log table rebuilt OK")
@staticmethod
def _iso(dt: Optional[datetime.datetime]) -> Optional[str]:
return dt.isoformat() if dt is not None else None
@staticmethod
def _new_id() -> str:
return str(uuid.uuid4())
# ── ACH sessions ──────────────────────────────────────────────────────────
def insert_ach_session(
self,
*,
serial: str,
peer: Optional[str] = None,
events_downloaded: int = 0,
monitor_entries: int = 0,
duration_seconds: Optional[float] = None,
session_time: Optional[datetime.datetime] = None,
) -> str:
"""Insert a new ACH session row. Returns the new session UUID."""
sid = self._new_id()
ts = self._iso(session_time or datetime.datetime.utcnow())
with self._connect() as conn:
conn.execute(
"""
INSERT INTO ach_sessions
(id, serial, session_time, peer,
events_downloaded, monitor_entries, duration_seconds)
VALUES (?, ?, ?, ?, ?, ?, ?)
""",
(sid, serial, ts, peer,
events_downloaded, monitor_entries, duration_seconds),
)
log.debug("ach_session inserted: %s serial=%s events=%d monitor=%d",
sid, serial, events_downloaded, monitor_entries)
return sid
def get_sessions(
self,
serial: Optional[str] = None,
limit: int = 50,
) -> list[dict]:
"""Return recent ACH sessions, newest first."""
with self._connect() as conn:
if serial:
rows = conn.execute(
"SELECT * FROM ach_sessions WHERE serial=? "
"ORDER BY session_time DESC LIMIT ?",
(serial, limit),
).fetchall()
else:
rows = conn.execute(
"SELECT * FROM ach_sessions ORDER BY session_time DESC LIMIT ?",
(limit,),
).fetchall()
return [dict(r) for r in rows]
# ── Events ────────────────────────────────────────────────────────────────
def insert_events(
self,
events: list[Event],
*,
serial: str,
session_id: Optional[str] = None,
) -> tuple[int, int]:
"""
Insert triggered events. Silently skips duplicates (serial+timestamp).
Returns (inserted, skipped).
"""
inserted = skipped = 0
with self._connect() as conn:
for ev in events:
key = ev._waveform_key.hex() if ev._waveform_key else None
if key is None:
skipped += 1
continue
ts = None
if ev.timestamp:
try:
ts = datetime.datetime(
ev.timestamp.year, ev.timestamp.month, ev.timestamp.day,
ev.timestamp.hour, ev.timestamp.minute, ev.timestamp.second,
).isoformat()
except Exception:
ts = str(ev.timestamp)
pv = ev.peak_values
pi = ev.project_info
try:
conn.execute(
"""
INSERT INTO events
(id, serial, waveform_key, session_id, timestamp,
tran_ppv, vert_ppv, long_ppv, peak_vector_sum, mic_ppv,
project, client, operator, sensor_location,
sample_rate, record_type)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
""",
(
self._new_id(), serial, key, session_id, ts,
pv.tran if pv else None,
pv.vert if pv else None,
pv.long if pv else None,
pv.peak_vector_sum if pv else None,
pv.micl if pv else None,
pi.project if pi else None,
pi.client if pi else None,
pi.operator if pi else None,
pi.sensor_location if pi else None,
ev.sample_rate,
ev.record_type,
),
)
inserted += 1
except sqlite3.IntegrityError:
skipped += 1
log.debug("insert_events serial=%s inserted=%d skipped=%d",
serial, inserted, skipped)
return inserted, skipped
def query_events(
self,
serial: Optional[str] = None,
from_dt: Optional[datetime.datetime] = None,
to_dt: Optional[datetime.datetime] = None,
false_trigger: Optional[bool] = None,
limit: int = 500,
offset: int = 0,
) -> list[dict]:
"""Query events with optional filters. Returns newest first."""
clauses: list[str] = []
params: list = []
if serial:
clauses.append("serial = ?")
params.append(serial)
if from_dt:
clauses.append("timestamp >= ?")
params.append(from_dt.isoformat())
if to_dt:
clauses.append("timestamp <= ?")
params.append(to_dt.isoformat())
if false_trigger is not None:
clauses.append("false_trigger = ?")
params.append(1 if false_trigger else 0)
where = ("WHERE " + " AND ".join(clauses)) if clauses else ""
params += [limit, offset]
with self._connect() as conn:
rows = conn.execute(
f"SELECT * FROM events {where} "
f"ORDER BY timestamp DESC LIMIT ? OFFSET ?",
params,
).fetchall()
return [dict(r) for r in rows]
def set_false_trigger(self, event_id: str, value: bool) -> bool:
"""Set or clear the false_trigger flag on an event. Returns True if found."""
with self._connect() as conn:
cur = conn.execute(
"UPDATE events SET false_trigger=? WHERE id=?",
(1 if value else 0, event_id),
)
return cur.rowcount > 0
# ── Monitor log ───────────────────────────────────────────────────────────
def insert_monitor_log(
self,
entries: list[MonitorLogEntry],
*,
session_id: Optional[str] = None,
) -> tuple[int, int]:
"""
Insert monitor log entries. Silently skips duplicates (serial+start_time).
Returns (inserted, skipped).
"""
inserted = skipped = 0
with self._connect() as conn:
for e in entries:
try:
conn.execute(
"""
INSERT INTO monitor_log
(id, serial, waveform_key, session_id,
start_time, stop_time, duration_seconds,
geo_threshold_ips)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
""",
(
self._new_id(),
e.serial or "",
e.key,
session_id,
self._iso(e.start_time),
self._iso(e.stop_time),
e.duration_seconds,
e.geo_threshold_ips,
),
)
inserted += 1
except sqlite3.IntegrityError:
skipped += 1
log.debug("insert_monitor_log inserted=%d skipped=%d", inserted, skipped)
return inserted, skipped
def query_monitor_log(
self,
serial: Optional[str] = None,
from_dt: Optional[datetime.datetime] = None,
to_dt: Optional[datetime.datetime] = None,
limit: int = 500,
offset: int = 0,
) -> list[dict]:
"""Query monitor log entries with optional filters. Returns newest first."""
clauses: list[str] = []
params: list = []
if serial:
clauses.append("serial = ?")
params.append(serial)
if from_dt:
clauses.append("start_time >= ?")
params.append(from_dt.isoformat())
if to_dt:
clauses.append("start_time <= ?")
params.append(to_dt.isoformat())
where = ("WHERE " + " AND ".join(clauses)) if clauses else ""
params += [limit, offset]
with self._connect() as conn:
rows = conn.execute(
f"SELECT * FROM monitor_log {where} "
f"ORDER BY start_time DESC LIMIT ? OFFSET ?",
params,
).fetchall()
return [dict(r) for r in rows]
# ── Fleet overview ────────────────────────────────────────────────────────
def query_units(self) -> list[dict]:
"""
Return one row per known serial with summary stats:
last_seen, total_events, total_monitor_entries.
"""
with self._connect() as conn:
rows = conn.execute(
"""
SELECT
s.serial,
MAX(s.session_time) AS last_seen,
SUM(s.events_downloaded) AS total_events,
SUM(s.monitor_entries) AS total_monitor_entries,
COUNT(*) AS total_sessions
FROM ach_sessions s
GROUP BY s.serial
ORDER BY last_seen DESC
"""
).fetchall()
return [dict(r) for r in rows]
+569 -26
View File
@@ -34,8 +34,11 @@ or:
from __future__ import annotations
import datetime
import logging
import sys
import threading
import time
from pathlib import Path
from typing import Optional
@@ -56,9 +59,11 @@ except ImportError:
from minimateplus import MiniMateClient
from minimateplus.protocol import ProtocolError
from minimateplus.models import ComplianceConfig, DeviceInfo, Event, PeakValues, ProjectInfo, Timestamp
from minimateplus.models import CallHomeConfig, ComplianceConfig, DeviceInfo, Event, PeakValues, ProjectInfo, Timestamp
from minimateplus.transport import TcpTransport, DEFAULT_TCP_PORT
from minimateplus.blastware_file import write_blastware_file, blastware_filename
from sfm.cache import SFMCache, get_cache
from sfm.database import SeismoDb
logging.basicConfig(
level=logging.INFO,
@@ -89,6 +94,151 @@ app.add_middleware(
)
# ── DB ────────────────────────────────────────────────────────────────────────
# Shared SeismoDb instance. Path can be overridden by --db-path at startup,
# or defaults to bridges/captures/seismo_relay.db relative to the repo root.
_DEFAULT_DB_PATH = Path(__file__).parent.parent / "bridges" / "captures" / "seismo_relay.db"
_db: Optional[SeismoDb] = None
def _get_db() -> SeismoDb:
global _db
if _db is None:
_db = SeismoDb(_DEFAULT_DB_PATH)
return _db
# ── Live device cache ─────────────────────────────────────────────────────────
# In-memory cache for live device data. Avoids re-dialing the device on every
# request when the data hasn't changed.
#
# Keyed by conn_key ("tcp:host:port" or "serial:port:baud").
# Does NOT persist across server restarts — this is purely an in-process cache
# to reduce TCP round-trips and cellular data usage.
#
# Invalidation rules:
# device_info — cached until POST /device/config marks it dirty
# events — cached by (conn_key, device_event_count); re-fetched when
# a quick count_events() probe shows new events on the device
# monitor_status — 30-second TTL (changes frequently during monitoring)
# waveforms — permanent (immutable once recorded; indexed by conn_key+idx)
#
# All endpoints accept ?force=true to bypass the cache and re-read from device.
_MONITOR_STATUS_TTL = 30.0 # seconds
class _LiveCache:
"""
Thread-safe in-memory cache for live SFM device data.
One singleton per server process.
"""
def __init__(self) -> None:
self._lock = threading.Lock()
# conn_key → serialised device info dict
self._device_info: dict[str, dict] = {}
# conn_key → (device_event_count_when_cached, [event dicts])
self._events: dict[str, tuple[int, list]] = {}
# conn_key → (fetched_at_unix, status_dict)
self._monitor_status: dict[str, tuple[float, dict]] = {}
# conn_key → bool (True = re-read device on next /device/info)
self._config_dirty: dict[str, bool] = {}
# (conn_key, event_index) → waveform dict (permanent)
self._waveforms: dict[tuple, dict] = {}
# ── Connection key ────────────────────────────────────────────────────────
@staticmethod
def make_conn_key(
host: Optional[str],
tcp_port: int,
port: Optional[str],
baud: int,
) -> str:
if host:
return f"tcp:{host}:{tcp_port}"
return f"serial:{port}:{baud}"
# ── Device info ───────────────────────────────────────────────────────────
def get_device_info(self, conn_key: str) -> Optional[dict]:
with self._lock:
if self._config_dirty.get(conn_key):
return None
return self._device_info.get(conn_key)
def set_device_info(self, conn_key: str, info: dict) -> None:
with self._lock:
self._device_info[conn_key] = info
self._config_dirty[conn_key] = False
# ── Events ────────────────────────────────────────────────────────────────
def get_events(self, conn_key: str, device_count: int) -> Optional[list]:
"""
Return cached events if the device's current event count matches what
we had when we last fetched. Returns None (cache miss) otherwise.
"""
with self._lock:
if self._config_dirty.get(conn_key):
return None
entry = self._events.get(conn_key)
if entry is None:
return None
cached_count, events = entry
return events if cached_count == device_count else None
def set_events(self, conn_key: str, device_count: int, events: list) -> None:
with self._lock:
self._events[conn_key] = (device_count, events)
# ── Monitor status ────────────────────────────────────────────────────────
def get_monitor_status(self, conn_key: str) -> Optional[dict]:
with self._lock:
entry = self._monitor_status.get(conn_key)
if entry is None:
return None
fetched_at, status = entry
if time.time() - fetched_at > _MONITOR_STATUS_TTL:
return None
return status
def set_monitor_status(self, conn_key: str, status: dict) -> None:
with self._lock:
self._monitor_status[conn_key] = (time.time(), status)
def invalidate_monitor_status(self, conn_key: str) -> None:
with self._lock:
self._monitor_status.pop(conn_key, None)
# ── Config dirty flag ─────────────────────────────────────────────────────
def mark_config_dirty(self, conn_key: str) -> None:
"""
Called after a successful POST /device/config write.
Forces next /device/info and /device/events to re-read from the device.
"""
with self._lock:
self._config_dirty[conn_key] = True
self._events.pop(conn_key, None)
# ── Waveforms (permanent cache) ───────────────────────────────────────────
def get_waveform(self, conn_key: str, index: int) -> Optional[dict]:
with self._lock:
return self._waveforms.get((conn_key, index))
def set_waveform(self, conn_key: str, index: int, waveform: dict) -> None:
with self._lock:
self._waveforms[(conn_key, index)] = waveform
_live_cache = _LiveCache()
# ── Serialisers ────────────────────────────────────────────────────────────────
# Plain dict helpers — avoids a Pydantic dependency in the library layer.
@@ -136,11 +286,14 @@ def _serialise_compliance_config(cc: Optional["ComplianceConfig"]) -> Optional[d
if cc is None:
return None
return {
"record_time": cc.record_time,
"sample_rate": cc.sample_rate,
"recording_mode": cc.recording_mode, # 0x00=Single Shot, 0x01=Continuous, 0x03=Histogram, 0x04=Histogram+Continuous
"sample_rate": cc.sample_rate,
"histogram_interval_sec": cc.histogram_interval_sec, # seconds; None if not Histogram mode
"record_time": cc.record_time,
"trigger_level_geo": cc.trigger_level_geo,
"alarm_level_geo": cc.alarm_level_geo,
"max_range_geo": cc.max_range_geo,
"geo_adc_scale": cc.geo_adc_scale, # hw scale factor (in/s)/V — informational only, do not write
"geo_range": cc.geo_range, # CONFIRMED 2026-04-20: 0x00=Normal 10in/s, 0x01=Sensitive 1.25in/s
"setup_name": cc.setup_name,
"project": cc.project,
"client": cc.client,
@@ -150,6 +303,27 @@ def _serialise_compliance_config(cc: Optional["ComplianceConfig"]) -> Optional[d
}
def _serialise_call_home_config(ch: Optional["CallHomeConfig"]) -> Optional[dict]:
if ch is None:
return None
return {
"auto_call_home_enabled": ch.auto_call_home_enabled,
"dial_string": ch.dial_string,
"after_event_recorded": ch.after_event_recorded,
"at_specified_times": ch.at_specified_times,
"time1_enabled": ch.time1_enabled,
"time1_hour": ch.time1_hour,
"time1_min": ch.time1_min,
"time2_enabled": ch.time2_enabled,
"time2_hour": ch.time2_hour,
"time2_min": ch.time2_min,
"num_retries": ch.num_retries,
"time_between_retries_sec": ch.time_between_retries_sec,
"wait_for_connection_sec": ch.wait_for_connection_sec,
"warm_up_time_sec": ch.warm_up_time_sec,
}
def _serialise_device_info(info: DeviceInfo) -> dict:
return {
"serial": info.serial,
@@ -281,11 +455,17 @@ def webapp():
return str(Path(__file__).parent / "sfm_webapp.html")
@app.get("/waveform", response_class=FileResponse)
def waveform_viewer():
"""Serve the standalone waveform viewer."""
return str(Path(__file__).parent / "waveform_viewer.html")
@app.get("/device/info")
def device_info(
port: Optional[str] = Query(None, description="Serial port (e.g. COM5, /dev/ttyUSB0)"),
baud: int = Query(38400, description="Serial baud rate (default 38400)"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay (e.g. 203.0.113.5)"),
port: Optional[str] = Query(None, description="Serial port (e.g. COM5, /dev/ttyUSB0)"),
baud: int = Query(38400, description="Serial baud rate (default 38400)"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay (e.g. 203.0.113.5)"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
force: bool = Query(False, description="Bypass cache and re-read from device"),
) -> dict:
@@ -352,9 +532,9 @@ def device_connect(
@app.get("/device/events")
def device_events(
port: Optional[str] = Query(None, description="Serial port (e.g. COM5)"),
baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
port: Optional[str] = Query(None, description="Serial port (e.g. COM5)"),
baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
debug: bool = Query(False, description="Include raw record hex for field-layout inspection"),
force: bool = Query(False, description="Bypass cache and re-download all events from device"),
@@ -365,6 +545,11 @@ def device_events(
Supply either *port* (serial) or *host* (TCP/modem).
**Caching:** a quick count_events() probe (~2s) is performed first. If the
device's event count matches the cached count, the cached response is returned
immediately without a full download. Pass ?force=true to skip this and always
re-download.
Pass debug=true to include raw_record_hex in each event useful for
verifying field offsets against the protocol reference.
@@ -494,8 +679,16 @@ def device_events(
cache.set_events(conn_key, serialised)
cache.set_device_info(conn_key, _serialise_device_info(info))
serialised_info = _serialise_device_info(info)
serialised_events = [_serialise_event(ev, debug=debug) for ev in events]
# Update cache (skip if debug=True — raw hex blobs shouldn't pollute the cache)
if not debug:
_live_cache.set_device_info(conn_key, serialised_info)
_live_cache.set_events(conn_key, len(events), serialised_events)
return {
"device": _serialise_device_info(info),
"device": serialised_info,
"event_count": len(events),
"events": serialised,
}
@@ -567,9 +760,9 @@ def device_event(
@app.get("/device/event/{index}/waveform")
def device_event_waveform(
index: int,
port: Optional[str] = Query(None, description="Serial port (e.g. COM5)"),
baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
port: Optional[str] = Query(None, description="Serial port (e.g. COM5)"),
baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
force: bool = Query(False, description="Bypass cache and re-download from device"),
) -> dict:
@@ -656,6 +849,109 @@ def device_event_waveform(
return result
@app.get("/device/event/{index}/blastware_file")
def device_event_blastware_file(
index: int,
port: Optional[str] = Query(None, description="Serial port (e.g. COM5)"),
baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
) -> FileResponse:
"""
Download the waveform for a single event (0-based index) and return it
as a Blastware-compatible binary file with a correct Blastware filename.
Supply either *port* (serial) or *host* (TCP/modem).
The file is written to /tmp and streamed back as a binary download.
Blastware can open it directly filename encodes serial + timestamp.
Filename format: <prefix><serial3><stem><AB>0<W|H>
- prefix letter = chr(ord('B') + floor(serial_numeric / 1000))
- stem + AB = second-resolution timestamp since 1985-01-01 local
- W / H = Full Waveform / Full Histogram (defaults to W for
triggered events; histogram requires recording_mode
to be populated from compliance config)
Performs: POLL startup get_events(full_waveform=False, extra_chunks=1,
stop_after_index=index) write_blastware_file() FileResponse.
"""
log.info(
"GET /device/event/%d/blastware_file port=%s host=%s",
index, port, host,
)
try:
def _do():
with _build_client(port, baud, host, tcp_port, timeout=120.0) as client:
info = client.connect()
# Use stop_after_metadata=True (full_waveform=False) with 1 extra
# chunk after "Project:". The extra chunk primes the device so that
# the termination response carries the full waveform footer bytes.
# Without it the terminator returns only ~90 bytes (no useful footer).
#
# The extra chunk's ADC data IS part of the Blastware file body —
# confirmed from 4-27-26 TCP capture: all 14 A5 frames (including the
# extra chunk's 2 TCP sub-frames) contribute to the correct 6864-byte
# output. write_blastware_file() includes all frames unconditionally.
#
# full_waveform=True (natural end-of-stream) downloads ALL chunks
# including post-event silence (35+ chunks for a 9-sec event at
# 1024 sps) — this produces 24KB+ files that Blastware rejects.
events = client.get_events(
full_waveform=False,
stop_after_index=index,
extra_chunks_after_metadata=1,
)
matching = [ev for ev in events if ev.index == index]
return matching[0] if matching else None, info
ev, info = _run_with_retry(_do, is_tcp=_is_tcp(host))
except HTTPException:
raise
except ProtocolError as exc:
log.error("blastware_file: protocol error: %s", exc, exc_info=True)
raise HTTPException(status_code=502, detail=f"Protocol error: {exc}") from exc
except OSError as exc:
log.error("blastware_file: connection error: %s", exc, exc_info=True)
raise HTTPException(status_code=502, detail=f"Connection error: {exc}") from exc
except Exception as exc:
log.error("blastware_file: unexpected error: %s", exc, exc_info=True)
raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc
if ev is None:
raise HTTPException(
status_code=404,
detail=f"Event index {index} not found on device",
)
a5_frames = getattr(ev, "_a5_frames", None)
if not a5_frames:
raise HTTPException(
status_code=502,
detail=f"No waveform data received for event index {index} — 5A download failed",
)
# Determine serial number from device info
serial = getattr(info, "serial", None) or "UNKNOWN"
# Build filename using the same algorithm Blastware uses
filename = blastware_filename(ev, serial)
# Write to /tmp so FastAPI can stream it back
out_path = Path("/tmp") / filename
write_blastware_file(ev, a5_frames, out_path)
log.info(
"blastware_file: wrote %s (%d A5 frames, serial=%s)",
out_path, len(a5_frames), serial,
)
return FileResponse(
path=str(out_path),
filename=filename,
media_type="application/octet-stream",
)
# ── Write endpoints ───────────────────────────────────────────────────────────
class DeviceConfigBody(BaseModel):
@@ -667,16 +963,15 @@ class DeviceConfigBody(BaseModel):
Recording parameters
--------------------
sample_rate : Samples per second. Valid values: 1024, 2048, 4096.
record_time : Record duration in seconds (e.g. 1.0, 2.0, 3.0).
recording_mode : Recording mode enum. Values: 0=Single Shot, 1=Continuous, 3=Histogram, 4=Histogram+Continuous.
sample_rate : Samples per second. Valid values: 1024, 2048, 4096.
record_time : Record duration in seconds (e.g. 1.0, 2.0, 3.0).
Trigger / alarm thresholds (geo channels, in/s)
------------------------------------------------
Trigger / alarm thresholds and range (geo channels)
----------------------------------------------------
trigger_level_geo : Trigger threshold in in/s (e.g. 0.5).
alarm_level_geo : Alarm threshold in in/s (e.g. 1.0).
max_range_geo : Full-scale calibration constant (e.g. 6.206).
Rarely changed only set if you know what you're doing.
geo_range : Geophone range/sensitivity. 0=Normal 10.000 in/s, 1=Sensitive 1.250 in/s.
Project / operator strings (max 41 ASCII characters each)
----------------------------
project : Project description.
@@ -686,12 +981,14 @@ class DeviceConfigBody(BaseModel):
notes : Extended notes.
"""
# Recording parameters
sample_rate: Optional[int] = None
record_time: Optional[float] = None
# Threshold parameters
recording_mode: Optional[int] = None
sample_rate: Optional[int] = None
record_time: Optional[float] = None
histogram_interval_sec: Optional[int] = None # seconds: 2, 5, 15, 60, 300, 900 (mode-gated)
# Threshold parameters / geo range
trigger_level_geo: Optional[float] = None
alarm_level_geo: Optional[float] = None
max_range_geo: Optional[float] = None
geo_range: Optional[int] = None # 0=Normal 10.000 in/s, 1=Sensitive 1.250 in/s
# Project / operator strings
project: Optional[str] = None
client_name: Optional[str] = None
@@ -719,6 +1016,7 @@ def device_config(
Example body (all fields optional include only what you want to change):
{
"recording_mode": 1,
"sample_rate": 1024,
"record_time": 3.0,
"trigger_level_geo": 0.5,
@@ -738,6 +1036,7 @@ def device_config(
422 if neither port nor host is provided.
"""
changed = body.model_dump(exclude_none=True)
conn_key = _live_cache.make_conn_key(host, tcp_port, port, baud)
log.info("POST /device/config port=%s host=%s fields=%s", port, host, list(changed.keys()))
try:
@@ -745,11 +1044,13 @@ def device_config(
with _build_client(port, baud, host, tcp_port) as client:
client.connect()
client.apply_config(
recording_mode=body.recording_mode,
sample_rate=body.sample_rate,
record_time=body.record_time,
histogram_interval_sec=body.histogram_interval_sec,
trigger_level_geo=body.trigger_level_geo,
alarm_level_geo=body.alarm_level_geo,
max_range_geo=body.max_range_geo,
geo_range=body.geo_range,
project=body.project,
client_name=body.client_name,
operator=body.operator,
@@ -859,6 +1160,7 @@ def device_monitor_start(
Sends SUB 0x96 and waits for ack SUB 0x69.
"""
conn_key = _live_cache.make_conn_key(host, tcp_port, port, baud)
with _build_client(port=port, baud=baud, host=host, tcp_port=tcp_port) as client:
try:
client.poll()
@@ -884,6 +1186,7 @@ def device_monitor_stop(
Sends SUB 0x97 and waits for ack SUB 0x68.
"""
conn_key = _live_cache.make_conn_key(host, tcp_port, port, baud)
with _build_client(port=port, baud=baud, host=host, tcp_port=tcp_port) as client:
try:
client.poll()
@@ -897,6 +1200,144 @@ def device_monitor_stop(
return {"status": "stopped"}
# ── Call home config endpoints ───────────────────────────────────────────────
@app.get("/device/call_home")
def device_call_home_get(
port: Optional[str] = Query(None, description="Serial port (e.g. COM5)"),
baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
) -> dict:
"""
Read the Auto Call Home (ACH) configuration from the device.
Sends SUB 0x2C (two-step read) and returns the decoded call home config.
Confirmed from 4-20-26 call home settings captures (BE11529).
Returns:
{
"auto_call_home_enabled": true/false,
"dial_string": "RADIO RING",
"after_event_recorded": true/false,
"at_specified_times": true/false,
"time1_enabled": true/false, "time1_hour": 19, "time1_min": 55,
"time2_enabled": false, "time2_hour": 0, "time2_min": 0,
"num_retries": 3,
"time_between_retries_sec": 15,
"wait_for_connection_sec": 60,
"warm_up_time_sec": 60
}
"""
try:
def _do():
with _build_client(port, baud, host, tcp_port) as client:
client.poll()
return client.get_call_home_config()
ch_config = _run_with_retry(_do, is_tcp=_is_tcp(host))
except HTTPException:
raise
except ProtocolError as exc:
raise HTTPException(status_code=502, detail=f"Protocol error: {exc}") from exc
except OSError as exc:
raise HTTPException(status_code=502, detail=f"Connection error: {exc}") from exc
except Exception as exc:
raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc
return _serialise_call_home_config(ch_config) or {}
class CallHomeConfigBody(BaseModel):
"""
Request body for POST /device/call_home.
All fields are optional only supplied (non-null) fields are modified.
All other call home config bytes are round-tripped verbatim from the device.
Confirmed writable fields (4-20-26 captures):
auto_call_home_enabled : bool master enable for auto call home
after_event_recorded : bool call home after each triggered event
at_specified_times : bool enable time-based scheduled calls
time1_enabled : bool enable time slot 1
time1_hour : int hour for slot 1 (0-23; avoid 3 DLE escape limitation)
time1_min : int minute for slot 1 (0-59; avoid 3)
time2_enabled : bool enable time slot 2
time2_hour : int hour for slot 2 (0-23; avoid 3)
time2_min : int minute for slot 2 (0-59; avoid 3)
Read-only fields (not writable via this endpoint):
dial_string, num_retries, time_between_retries_sec,
wait_for_connection_sec, warm_up_time_sec
"""
auto_call_home_enabled: Optional[bool] = None
after_event_recorded: Optional[bool] = None
at_specified_times: Optional[bool] = None
time1_enabled: Optional[bool] = None
time1_hour: Optional[int] = None
time1_min: Optional[int] = None
time2_enabled: Optional[bool] = None
time2_hour: Optional[int] = None
time2_min: Optional[int] = None
@app.post("/device/call_home")
def device_call_home_set(
body: CallHomeConfigBody,
port: Optional[str] = Query(None, description="Serial port (e.g. COM5)"),
baud: int = Query(38400, description="Serial baud rate"),
host: Optional[str] = Query(None, description="TCP host — modem IP or ACH relay"),
tcp_port: int = Query(DEFAULT_TCP_PORT, description=f"TCP port (default {DEFAULT_TCP_PORT})"),
) -> dict:
"""
Read the current call home config, apply supplied changes, and write back.
Only non-null fields are modified. All other bytes round-trip verbatim.
Write sequence (confirmed from 4-20-26 call home settings captures):
SUB 0x2C (read 2-step) 125-byte raw payload
patch fields
SUB 0x7E (write 127-byte payload) ack 0x81
SUB 0x7F (confirm) ack 0x80
Example body:
{ "auto_call_home_enabled": true, "after_event_recorded": true,
"time1_enabled": true, "time1_hour": 20, "time1_min": 0 }
"""
changed = body.model_dump(exclude_none=True)
log.info("POST /device/call_home port=%s host=%s fields=%s", port, host, list(changed.keys()))
try:
def _do():
with _build_client(port, baud, host, tcp_port) as client:
client.poll()
client.set_call_home_config(
auto_call_home_enabled=body.auto_call_home_enabled,
after_event_recorded=body.after_event_recorded,
at_specified_times=body.at_specified_times,
time1_enabled=body.time1_enabled,
time1_hour=body.time1_hour,
time1_min=body.time1_min,
time2_enabled=body.time2_enabled,
time2_hour=body.time2_hour,
time2_min=body.time2_min,
)
_run_with_retry(_do, is_tcp=_is_tcp(host))
except HTTPException:
raise
except ProtocolError as exc:
raise HTTPException(status_code=502, detail=f"Protocol error: {exc}") from exc
except OSError as exc:
raise HTTPException(status_code=502, detail=f"Connection error: {exc}") from exc
except ValueError as exc:
raise HTTPException(status_code=422, detail=str(exc)) from exc
except Exception as exc:
raise HTTPException(status_code=500, detail=f"Device error: {exc}") from exc
return {"status": "ok", "updated_fields": changed}
# ── Cache management endpoints ────────────────────────────────────────────────
@app.get("/cache/stats")
@@ -929,6 +1370,108 @@ def cache_clear_device(
return {"status": "cleared", "conn_key": conn_key, "deleted": counts}
# ── DB read endpoints ─────────────────────────────────────────────────────────
#
# These endpoints expose the seismo-relay SQLite DB written by ach_server.py.
# All queries are read-only. Terra-view calls these to build project event
# views, unit history panels, and (eventually) vibration summary reports.
@app.get("/db/units")
def db_units() -> list[dict]:
"""
Return one row per known serial with summary stats:
last_seen, total_events, total_monitor_entries, total_sessions.
"""
return _get_db().query_units()
@app.get("/db/events")
def db_events(
serial: Optional[str] = Query(None, description="Filter by unit serial (e.g. BE11529)"),
from_dt: Optional[str] = Query(None, description="ISO-8601 start datetime (inclusive)"),
to_dt: Optional[str] = Query(None, description="ISO-8601 end datetime (inclusive)"),
false_trigger: Optional[bool] = Query(None, description="Filter by false_trigger flag"),
limit: int = Query(500, description="Max rows to return (default 500)"),
offset: int = Query(0, description="Pagination offset"),
) -> dict:
"""
Query triggered events from the DB.
Returns events newest-first. All filter params are optional.
Example:
GET /db/events?serial=BE11529&from_dt=2026-04-01&limit=100
"""
from_parsed = datetime.datetime.fromisoformat(from_dt) if from_dt else None
to_parsed = datetime.datetime.fromisoformat(to_dt) if to_dt else None
rows = _get_db().query_events(
serial=serial,
from_dt=from_parsed,
to_dt=to_parsed,
false_trigger=false_trigger,
limit=limit,
offset=offset,
)
return {"count": len(rows), "events": rows}
@app.patch("/db/events/{event_id}/false_trigger")
def db_set_false_trigger(
event_id: str,
value: bool = Query(..., description="True to flag as false trigger, False to clear"),
) -> dict:
"""
Set or clear the false_trigger flag on a single event.
Used by the terra-view event review UI.
Returns 404 if the event_id is not found.
"""
found = _get_db().set_false_trigger(event_id, value)
if not found:
raise HTTPException(status_code=404, detail=f"Event {event_id} not found")
return {"status": "ok", "event_id": event_id, "false_trigger": value}
@app.get("/db/monitor_log")
def db_monitor_log(
serial: Optional[str] = Query(None, description="Filter by unit serial"),
from_dt: Optional[str] = Query(None, description="ISO-8601 start datetime (inclusive)"),
to_dt: Optional[str] = Query(None, description="ISO-8601 end datetime (inclusive)"),
limit: int = Query(500, description="Max rows to return"),
offset: int = Query(0, description="Pagination offset"),
) -> dict:
"""
Query monitor log entries (continuous monitoring intervals) from the DB.
Returns entries newest-first.
"""
from_parsed = datetime.datetime.fromisoformat(from_dt) if from_dt else None
to_parsed = datetime.datetime.fromisoformat(to_dt) if to_dt else None
rows = _get_db().query_monitor_log(
serial=serial,
from_dt=from_parsed,
to_dt=to_parsed,
limit=limit,
offset=offset,
)
return {"count": len(rows), "entries": rows}
@app.get("/db/sessions")
def db_sessions(
serial: Optional[str] = Query(None, description="Filter by unit serial"),
limit: int = Query(50, description="Max rows to return"),
) -> dict:
"""
Query ACH call-home sessions from the DB, newest first.
"""
rows = _get_db().get_sessions(serial=serial, limit=limit)
return {"count": len(rows), "sessions": rows}
# ── Entry point ────────────────────────────────────────────────────────────────
if __name__ == "__main__":
+983 -35
View File
File diff suppressed because it is too large Load Diff
+7 -4
View File
@@ -183,7 +183,7 @@
<h1>SFM Waveform Viewer</h1>
<div class="conn-group">
<label>API</label>
<input type="text" id="api-base" value="http://localhost:8200" style="width:180px" />
<input type="text" id="api-base" style="width:180px" />
</div>
<div class="conn-group">
<label>Device host</label>
@@ -240,7 +240,7 @@
let charts = {};
let lastData = null;
let unitInfo = null;
let geoRange = 10.0; // in/s full-scale for geo channels; updated on connect
let geoAdcScale = 10.0; // in/s full-scale for geo channels; updated on connect
let eventList = []; // populated from /device/events after connect
let currentEventIndex = 0;
@@ -278,7 +278,7 @@
throw new Error(err.detail || resp.statusText);
}
unitInfo = await resp.json();
geoRange = unitInfo.compliance_config?.max_range_geo ?? 10.0;
geoAdcScale = unitInfo.compliance_config?.geo_adc_scale ?? 10.0;
} catch (e) {
setStatus(`Error: ${e.message}`, 'error');
btn.disabled = false;
@@ -457,7 +457,7 @@
if (isGeo) {
// Geo channels: counts × (range / 32767) → in/s
const scale = geoRange / 32767;
const scale = geoAdcScale / 32767;
plotSamples = samples.map(c => c * scale);
const peakIns = Math.max(...plotSamples.map(Math.abs));
peakLabel = `${peakIns.toFixed(5)} in/s`;
@@ -588,6 +588,9 @@
}
}
// Auto-detect API base from wherever this page was served from
document.getElementById('api-base').value = window.location.origin;
// Allow Enter key on connection inputs to trigger connect
['api-base', 'dev-host', 'dev-tcp-port'].forEach(id => {
document.getElementById(id).addEventListener('keydown', e => {