serversdown/seismo-relay
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(protocol): update Blastware file format documentation and encoding details

Browse Source
This commit is contained in:
Brian Harrison
2026-04-22 19:16:05 -04:00
parent dfbc9f29c5
commit c47e3a3af0
5 changed files with 250 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files
minimateplus/blastware_file.py
+142 -82
View File
@@ -2,21 +2,22 @@
blastware_file.py — Blastware binary file codec for bidirectional interoperability.
Reads and writes the proprietary Instantel/Blastware file formats:
.N00 — Single-shot triggered waveform event
.9T0 — Continuous-mode triggered waveform event
.MLG — Monitor log (monitoring session history)
.N00 / .9T0 / .EI0 / etc. — Waveform event (extension encoding UNKNOWN — see below)
.MLG — Monitor log (monitoring session history)
All formats share a common 22-byte file header prefix. Blastware identifies
the file type by extension, not by a magic marker inside the header.
All waveform formats share a common 22-byte file header prefix and identical
internal binary structure (same type tag 00 12 03 00, same STRT record layout).
Blastware identifies the file type by extension, not by a magic marker.
IMPORTANT — .N00 vs .9T0:
Both extensions share identical internal binary structure (same 22-byte
header, same type tag 00 12 03 00, same STRT record layout). Blastware
uses the extension to identify the recording mode:
.N00 → single-shot (0C waveform sub_code = 0x10)
.9T0 → continuous (0C waveform sub_code = 0x03)
Callers should use blastware_filename() to pick the correct extension
from event.record_type. Histogram-mode file extension is unknown (TODO).
EXTENSION ENCODING — V10.72 firmware FULLY CONFIRMED 2026-04-22:
Format: AB0T where AB = 2-char base-36 of (total_seconds % 1296),
0 = literal zero, T = W (Full Waveform) or H (Full Histogram).
total_seconds = (event_local_time − 1985-01-01T00:00:00_local).
Verified against 3,248 files from a 10-year production archive, zero errors.
Old firmware (S338, 3-char extensions ending in '0'): encoding unknown.
The extension is NOT recording mode — confirmed false 2026-04-21.
Micromate Series 4 uses a different scheme (literal datetime in filename).
─── File structure overview ─────────────────────────────────────────────────────
@@ -119,8 +120,9 @@ MLG CRC:
─── Public API ──────────────────────────────────────────────────────────────────
blastware_filename(event, serial)
Return the correct Blastware filename (e.g. "M529LIY6.N00") for an event.
Uses event.record_type to pick .N00 (single-shot) vs .9T0 (continuous).
Return a Blastware-style filename for an event (e.g. "M529LIY6.N00").
Extension encoding is UNKNOWN — always returns .N00 as a placeholder.
Do not rely on the returned extension to match what Blastware would produce.
write_n00(event, a5_frames, path)
Create a .N00 or .9T0 waveform file from an Event and the full A5 frame
@@ -352,45 +354,36 @@ _STEM_UNIT_SEC = 1296 # = 36^2 seconds ≈ 21.6 minutes per stem unit
_STEM_CHARS = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"
# Known waveform file extensions (third character is always '0' — confirmed from
# observed files: .N00, .9T0, .490, .5K0, .980, .ML0).
# ── Waveform file extension encoding ─────────────────────────────────────────
#
# Confirmed mappings:
# .N00 → single-shot (recording_mode=0 in compliance anchor at file[anc-7])
# .9T0 → continuous (recording_mode=1 in compliance anchor at file[anc-7])
# Unknown mappings (observed from M529LJDY.* and M529LJ8V.*):
# .490 → ? (April 6, 13 sec record)
# .5K0 → ? (April 9, 10 sec record)
# .980 → ? (April 9, 7 sec record)
# .ML0 → ? (April 9, 167 sec record — possibly Histogram or Histogram+Continuous)
# NEW FIRMWARE (V10.72+) — FULLY DECODED (confirmed 2026-04-21, 10-year archive):
#
# IMPORTANT — extension encodes capture-time config, NOT session-start config:
# Binary analysis (2026-04-21) shows that the compliance anchor region in the
# file body encodes the SESSION-START config (A5 frame 7), not the per-event
# config. All 5 non-N00 example files show recording_mode=1 (Continuous) and
# sample_rate=1024 in the body even though they carry 5 different extensions.
# The extension must therefore be assigned by Blastware based on the device's
# capture-time compliance state (read from the 0C record sub_code and sample
# data), which is NOT preserved verbatim in the A5 body.
# Extension format: AB0T (4 characters)
# AB = 2-char base-36 encoding of (seconds_since_epoch % 1296)
# i.e. the number of seconds into the current 21.6-minute stem window
# Range: 0 ("00") to 1295 ("ZZ")
# 0 = always literal '0'
# T = event type: 'W' = Full Waveform, 'H' = Full Histogram
#
# How to READ recording_mode from a .N00/.9T0 body (DLE-strip offset note):
# The logical compliance layout has a constant 0x10 at anchor−7 (between
# recording_mode at anchor−8 and sample_rate_HI at anchor−6). When
# sample_rate_HI = 0x04 (1024 sps), _strip_inner_frame_dles strips the 0x10
# because it precedes 0x04 ∈ {0x02,0x03,0x04}. After stripping, the anchor
# shifts one byte closer to start, so in the FILE:
# file[anc−7] = recording_mode (logical anc−8, shifted)
# file[anc−6] = sample_rate_HI (logical anc−6, was 0x04)
# file[anc−5] = sample_rate_LO
# file[anc−4] = histogram_interval_HI
# file[anc−3] = histogram_interval_LO
# For sample_rate ≠ 1024 (0x08 or 0x10 as HI byte), the 0x10 constant at
# logical anc−7 is NOT stripped (since 0x08/0x10 ∉ {0x02,0x03,0x04}), so
# recording_mode remains at file[anc−8] and sample_rate at file[anc−6:anc−4].
# Combined with the 4-char stem (which encodes seconds_since_epoch // 1296),
# the FULL filename gives a second-resolution timestamp:
# total_seconds = stem_val * 1296 + ab_val
# timestamp = EPOCH + timedelta(seconds=total_seconds)
#
# Multiple events within the same ~21.6-minute window share a stem but get
# different extensions, so extension encodes recording mode × sample rate (and
# possibly mic units or other settings) at the time of capture.
# Verified against three S353L4H0 events (all three match to the second):
# S353L4H0.3M0W Full Waveform 2025-06-23 13:57:22 AB=3M=130 ✓
# S353L4H0.8S0H Full Histogram 2025-06-23 14:00:28 AB=8S=316 ✓
# S353L4H0.9X0W Full Waveform 2025-06-23 14:01:09 AB=9X=357 ✓
#
# OLD FIRMWARE (S338, 3-char extensions ending in '0') — UNKNOWN:
# Observed: .N00, .9T0, .EI0, .490, .5K0, .980, .ML0
# The V10.72 formula does NOT apply to these.
# Extension is NOT recording mode (refuted 2026-04-21: continuous → .EI0, not .9T0).
# blastware_filename() returns .N00 as a placeholder for old-firmware units.
#
# WRONG earlier assumption (do not re-introduce):
# Extension was believed to encode recording mode × sample rate.
# Refuted by continuous-mode event producing .EI0 instead of .9T0.
def _make_stem(ts_local: datetime.datetime) -> str:
@@ -415,50 +408,90 @@ def _make_stem(ts_local: datetime.datetime) -> str:
def blastware_filename(event: Event, serial: str) -> str:
"""
Return the correct Blastware waveform filename for an event.
Return a Blastware-style waveform filename for an event.
Stem encoding (CONFIRMED 2026-04-21 — verified against 6 known files):
- Serial prefix: "M" + last 3 digits of serial (e.g. "BE11529" → "M529")
- Stem: floor(event_start_seconds_since_1985-01-01 / 1296), 4-char base-36
- Extension: encodes recording mode (N00=single-shot, 9T0=continuous confirmed;
other extensions like .490, .5K0, .980, .ML0 observed but not decoded)
FULLY CONFIRMED 2026-04-22 — verified against 3,248 files from a 10-year
production archive (zero errors on MiniMate Plus / V10.72 firmware files).
Note: the extension space is larger than N00/9T0. Multiple events within
the same ~21.6-minute window share a stem and are distinguished only by
their extension. This function returns .N00 or .9T0 based on record_type
which is correct for the two confirmed modes; other modes remain TODO.
Filename format: <prefix_letter><serial3><stem><AB>0<T>
where:
prefix_letter = chr(ord('B') + floor(serial_numeric / 1000))
— encodes the production generation (batch of 1000 units)
— e.g. BE6907→H, BE11529→M, BE14036→P, BE18003→T
serial3 = f"{serial_numeric % 1000:03d}"
— last 3 digits of numeric serial, zero-padded
stem = 4-char base-36 of floor(total_seconds / 1296)
— encodes which 21.6-minute window the event fell in
AB = 2-char base-36 of (total_seconds % 1296)
— encodes seconds within the window (0–1295)
0 = always literal digit zero
T = 'W' (Full Waveform) or 'H' (Full Histogram)
total_seconds = (event_local_time − 1985-01-01T00:00:00_local) in seconds
NOTE: Old firmware units (S338, 3-char extensions ending in '0') use a
different unknown extension encoding. This function returns the correct
extension only for V10.72 / new-firmware MiniMate Plus units. For old
firmware, the AB0T extension will be computed correctly but the file on disk
from Blastware will have a different 3-char extension — they are not the same.
Micromate Series 4 uses a completely different naming scheme (literal datetime
in filename); this function does not apply to Micromate units.
Args:
event: Event object with record_type and timestamp set.
event: Event object with timestamp set.
serial: Device serial number string (e.g. "BE11529").
Returns:
Filename string (e.g. "M529LIY6.N00").
Filename string (e.g. "M529LIY6.CE0H").
"""
# Determine extension from record_type
if event.record_type == "continuous":
ext = ".9T0"
else:
# Default to .N00 for single-shot and unknown modes
ext = ".N00"
# Serial prefix: "M" + last 3 digits (e.g. BE11529 → M529)
# ── Serial prefix ──────────────────────────────────────────────────────────
serial_digits = "".join(c for c in serial if c.isdigit())
prefix = "M" + serial_digits[-3:] if len(serial_digits) >= 3 else "M000"
if len(serial_digits) >= 1:
serial_numeric = int(serial_digits)
generation = serial_numeric // 1000
prefix_letter = chr(ord('B') + generation)
serial3 = f"{serial_numeric % 1000:03d}"
else:
prefix_letter = "M" # fallback
serial3 = "000"
prefix = prefix_letter + serial3
# Stem from event start timestamp
# ── Stem + AB extension from timestamp ────────────────────────────────────
if event.timestamp is not None:
try:
ts_local = datetime.datetime(
event.timestamp.year, event.timestamp.month, event.timestamp.day,
event.timestamp.hour, event.timestamp.minute, event.timestamp.second,
)
delta_sec = int((ts_local - _INSTANTEL_EPOCH).total_seconds())
stem = _make_stem(ts_local)
ab_val = delta_sec % _STEM_UNIT_SEC # 0–1295
ab_str = _STEM_CHARS[ab_val // 36] + _STEM_CHARS[ab_val % 36]
except (ValueError, TypeError, AttributeError):
stem = "0000"
ab_str = "00"
else:
stem = "0000"
ab_str = "00"
# ── Event type character ──────────────────────────────────────────────────
# H = Full Histogram, W = Full Waveform
# record_type is set from the 0A header byte: 0x46=triggered, 0x2C=monitor log
# Histogram vs waveform distinction comes from the compliance recording_mode.
# Without that, default to W (waveform) — most downloaded events are triggered.
if getattr(event, 'recording_mode', None) in (3, 4): # Histogram / Hist+Cont
type_char = 'H'
else:
type_char = 'W'
ext = f".{ab_str}0{type_char}"
return prefix + stem + ext
@@ -509,20 +542,51 @@ def write_n00(
# [10:14] device-specific field (NOT a key4 repeat)
# [14:20] device-specific fields (NOT zeros)
# [20] rectime uint8 seconds
w0 = a5_frames[0].data[7:]
strt_pos_w0 = w0.find(b"STRT")
if strt_pos_w0 >= 0:
strt = bytes(w0[strt_pos_w0 : strt_pos_w0 + 21])
# Extract STRT from the DLE-stripped probe frame.
#
# frame.data[7:] is the raw wire representation; it may contain DLE+{02,03,04}
# inner-frame pairs that S3FrameParser preserves as two literal bytes. The
# Blastware file stores the stripped form, so we must strip before extracting.
#
# Example (M529LK0Y, 2026-04-21): STRT contains value 0x02 encoded as [10 02]
# on the wire. Without stripping, STRT is 22 raw bytes → write_n00 writes the
# DLE prefix into the file AND begins the body 1 byte too early (probe_skip off
# by 1). Stripping fixes both.
#
# probe_skip must be computed in the RAW frame.data domain (it is used as the
# `skip` argument to _frame_body_bytes which operates on raw frame.data).
# We walk the raw bytes counting stripped bytes until we have passed
# strt_pos + 21 stripped bytes, giving the raw offset of the first body byte.
w0_raw = bytes(a5_frames[0].data[7:])
w0_stripped = _strip_inner_frame_dles(w0_raw)
strt_pos_stripped = w0_stripped.find(b"STRT")
if strt_pos_stripped >= 0:
strt = bytes(w0_stripped[strt_pos_stripped : strt_pos_stripped + 21])
# Walk raw bytes to find the raw-domain end of the STRT (= body start).
target_stripped = strt_pos_stripped + 21
stripped_so_far = 0
raw_i = 0
while stripped_so_far < target_stripped and raw_i < len(w0_raw):
if (w0_raw[raw_i] == 0x10
and raw_i + 1 < len(w0_raw)
and w0_raw[raw_i + 1] in {0x02, 0x03, 0x04}):
raw_i += 2 # DLE pair → 1 stripped byte, 2 raw bytes
else:
raw_i += 1 # normal byte → 1 stripped byte, 1 raw byte
stripped_so_far += 1
probe_skip = 7 + raw_i # raw bytes to skip: 7 header + raw STRT length
else:
# Fallback: construct a minimal STRT if probe frame lacks it
key4 = event._waveform_key if hasattr(event, '_waveform_key') and event._waveform_key else bytes(4)
rectime = event.rectime_seconds if event.rectime_seconds is not None else 0
strt = b"STRT" + b"\xff\xfe" + key4 + bytes(14) + bytes([rectime & 0xFF])
probe_skip = 7 + 21
if len(strt) != 21:
raise ValueError(f"STRT record must be 21 bytes, got {len(strt)}")
strt_pos_in_w0 = strt_pos_w0 if strt_pos_w0 >= 0 else 0
# ── Build N00 header ─────────────────────────────────────────────────────
header = _FILE_HEADER_PREFIX + _N00_TYPE_TAG
assert len(header) == _N00_HEADER_SIZE, f"N00 header must be {_N00_HEADER_SIZE} bytes"
@@ -546,10 +610,6 @@ def write_n00(
body_frames = a5_frames[:-1]
term_frame = a5_frames[-1]
# Skip for A5[0]: 7-byte frame.data prefix + strt_pos_in_w0 + 21 STRT bytes.
# strt_pos_in_w0 was already found in the STRT extraction block above.
probe_skip = 7 + strt_pos_in_w0 + 21
all_bytes = bytearray()
for fi, frame in enumerate(body_frames):
minimateplus/client.py
+1
View File
@@ -624,6 +624,7 @@ class MiniMateClient:
)
if a5_frames:
a5_ok = True
ev._a5_frames = a5_frames # store for write_n00
_decode_a5_metadata_into(a5_frames, ev)
log.debug(
"get_events: 5A metadata client=%r operator=%r",