Compare commits
15
Commits
@@ -4,6 +4,68 @@ All notable changes to seismo-relay are documented here.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Unreleased
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **Waveform event times were the monitoring-session start, not the trigger
|
||||||
|
(~hours off).** `read_blastware_file` stamped events with footer `ts1`, which
|
||||||
|
for a waveform is the session start a unit shares across every event that day
|
||||||
|
(a unit arming at 06:00 stamped 06:00 on all of them — the modal and PDF both
|
||||||
|
showed it, since it's the stored value). The event time is footer `ts2` (the
|
||||||
|
recording stop), and Blastware's trigger = `ts2 - record time`. The record
|
||||||
|
time is a big-endian float32 in the recording-setup config block (30 bytes
|
||||||
|
before the `Standard Recording Setup` marker), so the **exact trigger is now
|
||||||
|
recovered from the binary alone** — all 7 BE12844 oracle events decode to
|
||||||
|
their exact Blastware time (e.g. N844LQHB 10:33:29), no paired `.TXT` needed.
|
||||||
|
Histograms keep `ts1` (the ~24 h window start). A paired report's
|
||||||
|
`event_datetime` stays authoritative (unit-clock drift).
|
||||||
|
⚠ **Needs a re-decode backfill** to correct existing stored events' timestamps.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Diagnostics tab in the SFM standalone webapp.** Surfaces the device
|
||||||
|
endpoints that previously existed only as `curl`: `events/storage_range` and
|
||||||
|
`events/index` alongside `monitor/status`, then stop monitoring, disable ACH
|
||||||
|
(`rescue?erase=false`, so stored events survive), and erase. The wedged-unit
|
||||||
|
ladder — slow drip and blind stop — sits under its own heading pointing at
|
||||||
|
`docs/runbooks/wedged_unit_recovery.md`, with the reminder that `slow_drip`'s
|
||||||
|
success signal is `bytes_received > 0` and not a clean duration. Erase is
|
||||||
|
guarded by typing the unit's serial: auth answers *who*, not *did you mean
|
||||||
|
it*, and Swagger's try-it-out button on `/device/events/erase` is live on
|
||||||
|
`:8200/docs`.
|
||||||
|
|
||||||
|
- **`docs/sfm_tool_status.md`** — an honest per-capability maturity assessment:
|
||||||
|
what is production-grade (the codec library, the data side), what is
|
||||||
|
emergency-grade (the device side), what is a research artifact, the
|
||||||
|
known-issues table, and the gap to a real tool. Also records the **5A
|
||||||
|
page-boundary bug** as known: `parse_strt_end_offset()` discards the key's
|
||||||
|
page byte, so once a unit has recorded more than 64 KB since its last erase,
|
||||||
|
an event spanning the boundary reads an `end_offset` *behind* its own start —
|
||||||
|
the chunk loop fetches nothing and TERM packs a negative `offset_word`, which
|
||||||
|
500s. Reproduced on BE12599. Production is unaffected: it ingests complete
|
||||||
|
files via the watcher path and never runs this walk.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **Connecting to a unit no longer walks its event chain.** `/device/events`
|
||||||
|
reads every event header over the cellular link; on a unit with a large or
|
||||||
|
wrapped chain that takes minutes or fails outright, and it fired
|
||||||
|
automatically on every connect. Connect now uses only ~2 s probes —
|
||||||
|
`/device/info` (which already carried the compliance config the walk was
|
||||||
|
re-reading) plus `events/storage_range` — and the Device tab gains an Event
|
||||||
|
Chain card. The walk moved behind a **Load events** button in the Events
|
||||||
|
toolbar. Knowing whether a unit's ACH is on no longer requires reading every
|
||||||
|
event it has stored.
|
||||||
|
|
||||||
|
### Migration
|
||||||
|
|
||||||
|
**None.** Frontend and documentation only — no codec, waveform-store or DB
|
||||||
|
change, no schema change, and no `TOOL_VERSION` bump. The webapp is served
|
||||||
|
from the image, so the change appears after the next `sfm` rebuild.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## v0.31.0 — 2026-09-18
|
## v0.31.0 — 2026-09-18
|
||||||
|
|
||||||
**Report parity, and a second way to rescue a runaway unit.** Two threads.
|
**Report parity, and a second way to rescue a runaway unit.** Two threads.
|
||||||
|
|||||||
@@ -0,0 +1,825 @@
|
|||||||
|
# Micromate Protocol Reference — Thor / Micromate Series IV, live wire protocol
|
||||||
|
|
||||||
|
Sibling to [instantel_protocol_reference.md](instantel_protocol_reference.md)
|
||||||
|
(Series III, "the Rosetta Stone") and
|
||||||
|
[idf_protocol_reference.md](idf_protocol_reference.md) (Series IV *file*
|
||||||
|
format). This document covers the Series IV **live device protocol** — what
|
||||||
|
the unit says over the wire, as opposed to what it writes into a `.IDFW`.
|
||||||
|
|
||||||
|
**Status (2026-09-23): opening session.** Everything below was established in
|
||||||
|
a single bench session against one unit. Treat it as a strong start, not a
|
||||||
|
settled spec — in particular, everything here comes from **one unit, over USB,
|
||||||
|
with no events stored**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The headline
|
||||||
|
|
||||||
|
**A Micromate running the *Blastware* firmware answers Series III command
|
||||||
|
frames.**
|
||||||
|
|
||||||
|
An unmodified Series III `POLL` (`SUB 0x5B`), built by
|
||||||
|
`minimateplus.framing.build_bw_frame` with no changes at all, produced a
|
||||||
|
complete two-step probe/data cycle. Ten Series III read commands were then
|
||||||
|
tried and **all ten answered**, every one obeying the Series III response-SUB
|
||||||
|
rule.
|
||||||
|
|
||||||
|
⚠ **That qualifier is load-bearing, and it was discovered after the fact.**
|
||||||
|
Instantel ships the Micromate in two firmware lines:
|
||||||
|
|
||||||
|
| firmware | Instantel's own description |
|
||||||
|
|---|---|
|
||||||
|
| `11.0CB` | "Utilize with **Blastware**" |
|
||||||
|
| `11.0BD` | "Utilize with **THOR, Vision, Vision II**" |
|
||||||
|
|
||||||
|
The bench unit reports **`11.0CB`** — the Blastware build. So the clean
|
||||||
|
Series III behaviour above is very likely *because the unit is in Blastware
|
||||||
|
mode*, not because the Micromate natively speaks Series III. **Nothing here
|
||||||
|
should be assumed to hold on a `11.0BD` unit until tested.**
|
||||||
|
|
||||||
|
This reframes the project. The question is no longer only "what is the
|
||||||
|
Series IV protocol" but **"which firmware line do we target, and does one of
|
||||||
|
them let the existing Series III stack drive the whole fleet?"**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Firmware — the variable nobody knew was a variable
|
||||||
|
|
||||||
|
This explains a production problem TMI has lived with: two ACH servers, two
|
||||||
|
machines, and units that will not cross over. It is not a misconfiguration.
|
||||||
|
**Instantel ships different firmware for different host software**, and the
|
||||||
|
wire protocol differs with it.
|
||||||
|
|
||||||
|
### Fleet audit (physical, 2026-09-22/23)
|
||||||
|
|
||||||
|
| unit | firmware | line | location |
|
||||||
|
|---|---|---|---|
|
||||||
|
| UM11719 | `11.0CB` | Blastware | Ped Bridge Loc 2 |
|
||||||
|
| UM6047 | `11.0CB` | Blastware | Brookville Loc 9 |
|
||||||
|
| UM12947 | `11.0CB` | Blastware | **bench** |
|
||||||
|
| UM14133 | `11.0CB` | Blastware | Pitt-Music Bldg Loc 1 |
|
||||||
|
| UM11402 | `11.0BD` | Thor | Ped Bridge Loc 1 |
|
||||||
|
| UM20147 | `11.0BD` | Thor | **bench** |
|
||||||
|
| UM13981 | `11.0AK` | pre-split | RKM Loc 1 |
|
||||||
|
| UM20146 | `11.0AK` | pre-split | Karns Loc 2 |
|
||||||
|
| UM12420 | `10.90GC` | pre-split | RKM Loc 2 |
|
||||||
|
|
||||||
|
**4 Blastware / 2 Thor / 3 pre-split.** The Blastware line is already the
|
||||||
|
plurality, which makes "standardise on Blastware" less disruptive than it
|
||||||
|
first appeared.
|
||||||
|
|
||||||
|
A store-derived audit (firmware is recorded in every `.sfm.json` sidecar as
|
||||||
|
`extensions.idf_report.version`) agreed with the physical audit on **7 of 9**.
|
||||||
|
The two that differed — UM6047 and UM14133 — are the most recently deployed
|
||||||
|
units, reflashed after their last stored event. Useful technique: the fleet's
|
||||||
|
firmware history is reconstructable from the store without touching a unit,
|
||||||
|
but it lags reality by one deployment.
|
||||||
|
|
||||||
|
Firmware is **not stable per-unit over time** — five of nine have been
|
||||||
|
reflashed at least once. Any fleet-wide claim needs a fresh audit.
|
||||||
|
|
||||||
|
### ⚠ Retraction: firmware line does NOT determine Thor compatibility
|
||||||
|
|
||||||
|
An earlier draft of this document suggested that UM12947's trouble with Thor
|
||||||
|
was explained by its being on the Blastware build. **That is not supported.**
|
||||||
|
|
||||||
|
Ped Bridge runs UM11402 (`11.0BD`) and UM11719 (`11.0CB`) side by side, both
|
||||||
|
deployed 2026-04-20, and **both call Thor successfully** — UM11719 has 331
|
||||||
|
Thor-collected events in the store while on `11.0CB`, through 2026-08-23.
|
||||||
|
|
||||||
|
So a Blastware-line unit does feed Thor. Whatever the CB/BD split changes, it
|
||||||
|
is not "which host software can collect from it", and UM12947's specific
|
||||||
|
problem remains unexplained.
|
||||||
|
|
||||||
|
**What is actually established:** a `11.0CB` unit answers Series III command
|
||||||
|
frames. Whether a `11.0BD` unit does is **untested** — and UM20147 (`11.0BD`)
|
||||||
|
is on the bench, which makes that a direct A/B away.
|
||||||
|
|
||||||
|
The version is readable over the wire from `SUB 0x01` as two separate ASCII
|
||||||
|
runs — `"0CB"` then `"11"` — with no single concatenated string.
|
||||||
|
|
||||||
|
### The strategic fork
|
||||||
|
|
||||||
|
- **Option A — standardise the fleet on the Blastware line.** Every unit,
|
||||||
|
MiniMate and Micromate alike, then speaks Series III, and the existing
|
||||||
|
`minimateplus/` stack drives all of it. One protocol, one call-home
|
||||||
|
receiver. Dramatically cheaper *if* it holds up.
|
||||||
|
- **Option B — reverse-engineer the Thor line (`11.0BD`) and support both.**
|
||||||
|
|
||||||
|
Option A is the shortcut, but it is unproven and carries real unknowns, all
|
||||||
|
of which are cheap to answer on the bench and expensive to discover later:
|
||||||
|
|
||||||
|
1. **What file format does a `11.0CB` unit produce?** If it emits Blastware
|
||||||
|
binaries rather than `.IDFW`/`.IDFH`, the (now exact) Series III decoder
|
||||||
|
applies and the IDF codec becomes a legacy path. Not a loss — we have
|
||||||
|
both — but it changes what the ingest pipeline sees.
|
||||||
|
2. **Does Thor still work with a `11.0CB` unit?** If not, flashing a
|
||||||
|
production unit breaks data collection until the replacement path exists.
|
||||||
|
3. **Are any Micromate-specific capabilities lost** on the Blastware line?
|
||||||
|
4. **Is the flash reversible in the field**, and what does it cost in downtime?
|
||||||
|
|
||||||
|
## How this was obtained
|
||||||
|
|
||||||
|
No Thor, no modem, no Windows machine. The Micromate exposes its protocol on
|
||||||
|
a **USB CDC-ACM virtual serial port**:
|
||||||
|
|
||||||
|
```
|
||||||
|
ID 2504:0300 Instantel Inc. MICROMATE COM PORT
|
||||||
|
driver: cdc_acm ATTRS{serial}=="V1.00"
|
||||||
|
→ /dev/ttyACM0
|
||||||
|
```
|
||||||
|
|
||||||
|
Plain CDC ACM, so no vendor driver and no proprietary USB layer — any host
|
||||||
|
that can open a serial port can talk to the unit.
|
||||||
|
|
||||||
|
**Baud is irrelevant over USB.** Identical byte-for-byte responses at 38400
|
||||||
|
and 115200; CDC-ACM ignores the line rate. TMI provisions Micromate *modem*
|
||||||
|
links at **115200** (Series III uses 38400) — that matters for the cellular
|
||||||
|
path, not for USB.
|
||||||
|
|
||||||
|
**The device never speaks first.** 20 s of passive listening on an idle open
|
||||||
|
port produced zero bytes. It is strictly request/response.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Physical + framing layer
|
||||||
|
|
||||||
|
### Requests — Series III format, unmodified
|
||||||
|
|
||||||
|
Every frame in this session was produced by `build_bw_frame(sub, offset)` with
|
||||||
|
no Series IV changes, and the device accepted all of them:
|
||||||
|
|
||||||
|
```
|
||||||
|
[ACK 0x41] [STX 0x02] [10 10] [flags 00] [SUB] [00] [00] [offset] [params×10] [chk] [ETX 0x03]
|
||||||
|
```
|
||||||
|
|
||||||
|
⚠ Only the doubled `BW_CMD` (`10 10`) form has been exercised. Whether other
|
||||||
|
literal `0x10` bytes inside params require stuffing is **untested** — none of
|
||||||
|
the probes sent carried one.
|
||||||
|
|
||||||
|
### Responses — Series III *minus the DLE prefix*
|
||||||
|
|
||||||
|
```
|
||||||
|
Series III: [DLE 0x10] [STX 0x02] … [chk] [ETX 0x03]
|
||||||
|
Micromate: [STX 0x02] … [chk] [ETX 0x03] ← no leading DLE
|
||||||
|
```
|
||||||
|
|
||||||
|
This single byte matters operationally: Blastware's parser locates frames by
|
||||||
|
scanning for `DLE+STX`, so it will **never find a frame boundary** in Micromate
|
||||||
|
traffic no matter what else is correct. That is a structural reason a
|
||||||
|
Micromate cannot call into a Blastware ACH server, independent of any baud
|
||||||
|
mismatch.
|
||||||
|
|
||||||
|
### Response payload header
|
||||||
|
|
||||||
|
```
|
||||||
|
[0] CMD 0x00 same as Series III
|
||||||
|
[1] flags 0xC5 ← Series III uses 0x10. Constant across all 10 SUBs.
|
||||||
|
[2] SUB 0xFF − request_SUB
|
||||||
|
[3] PAGE_HI
|
||||||
|
[4] PAGE_LO
|
||||||
|
[5+] data
|
||||||
|
```
|
||||||
|
|
||||||
|
### Checksum — the DLE-aware variant
|
||||||
|
|
||||||
|
```python
|
||||||
|
chk = sum(b for b in payload if b != 0x10) & 0xFF
|
||||||
|
```
|
||||||
|
|
||||||
|
Confirmed on every frame captured. The `POLL` probe response contains no
|
||||||
|
`0x10` and so cannot distinguish plain SUM8 from the DLE-aware form; the
|
||||||
|
`POLL` **data** response contains a `0x10` at payload offset 42, and only the
|
||||||
|
DLE-aware rule matches there. This is the same checksum Series III uses for
|
||||||
|
its `5A` bulk-stream and write frames — not the plain SUM8 of ordinary
|
||||||
|
Series III reads.
|
||||||
|
|
||||||
|
### The probe response carries the data length
|
||||||
|
|
||||||
|
Series III hardcodes `DATA_LENGTHS` per SUB. On the Micromate the **probe
|
||||||
|
response tells you**, as a **uint16 BE at `payload[8:10]`**:
|
||||||
|
|
||||||
|
⚠ **Corrected 2026-09-23.** An earlier draft read this as a single byte at
|
||||||
|
`payload[9]`. That is right only while the high byte is zero, and it is
|
||||||
|
catastrophically wrong for `SUB 0x1A`, whose real length is `0x082C` = **2092**
|
||||||
|
— read as a byte it gives **44**, a 47× under-read. Always read the pair.
|
||||||
|
|
||||||
|
| SUB | command | `payload[9]` | Series III constant |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `0x15` | serial number | `0x0A` | `0x0A` ✓ |
|
||||||
|
| `0x01` | device info | `0x98` | `0x98` ✓ |
|
||||||
|
| `0x1C` | monitor status | `0x2C` | `0x2C` ✓ |
|
||||||
|
| `0x06` | storage range | `0x24` | `0x24` ✓ |
|
||||||
|
| `0x2C` | call-home config | `0x7E` | `0x7C` ✗ **differs by 2** |
|
||||||
|
| `0x08` | event index | `0x5A` | — |
|
||||||
|
| `0x1E` | event header | `0x08` | — |
|
||||||
|
| `0x1A` | compliance config | `0x082C` (2092) | — |
|
||||||
|
| `0x0A` | waveform header | `0x00` | — (no event context) |
|
||||||
|
| `0xFE` | full config | `0x00` | — (see note) |
|
||||||
|
|
||||||
|
Four of four known Series III lengths match exactly. **Read the length from
|
||||||
|
the probe rather than hardcoding it** — it is free, and it already caught the
|
||||||
|
call-home divergence.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Confirmed commands (read-only)
|
||||||
|
|
||||||
|
All ten below answered with a correct `0xFF − SUB` response. Nothing that
|
||||||
|
writes, erases, or changes monitoring state has been sent to a unit.
|
||||||
|
|
||||||
|
| SUB | RSP | Command | Data proven |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `0x5B` | `0xA4` | POLL / handshake | yes — ID block |
|
||||||
|
| `0x15` | `0xEA` | Serial number | yes — `UM12947` |
|
||||||
|
| `0x01` | `0xFE` | Device info | yes — 152 B |
|
||||||
|
| `0x2C` | `0xD3` | Call-home config | yes — 126 B |
|
||||||
|
| `0x1C` | `0xE3` | Monitor status | yes — 44 B |
|
||||||
|
| `0x06` | `0xF9` | Event storage range | yes — 36 B |
|
||||||
|
| `0x08` | `0xF7` | Event index | probe only |
|
||||||
|
| `0x1E` | `0xE1` | Event header / first key | probe only |
|
||||||
|
| `0x0A` | `0xF5` | Waveform header | probe only |
|
||||||
|
| `0x1A` | `0xE5` | Compliance config | probe only |
|
||||||
|
|
||||||
|
### Decoded so far
|
||||||
|
|
||||||
|
**`SUB 0x15` — serial.** ASCII, null-terminated: `UM12947`.
|
||||||
|
|
||||||
|
**`SUB 0x5B` / `0x01` — identification strings.**
|
||||||
|
`Instantel\0` and `MM/ISEE/S/IO` (MicroMate / ISEE standard). `0x01` also
|
||||||
|
carries eight consecutive `3f 80 00 00` float32 values (= `1.0f`) — almost
|
||||||
|
certainly per-channel calibration/scale factors, by analogy with Series III's
|
||||||
|
`geo_hardware_constant`. **Unverified.**
|
||||||
|
|
||||||
|
**`SUB 0x1C` — monitor status. Series III field offsets apply unchanged:**
|
||||||
|
|
||||||
|
| field | offset | read |
|
||||||
|
|---|---|---|
|
||||||
|
| battery × 100 | `payload[-10:-8]` uint16 BE | `0x017D` → **3.81 V** |
|
||||||
|
| memory total | `payload[-8:-4]` uint32 BE | 15,000,000 |
|
||||||
|
| memory free | `payload[-4:]` uint32 BE | 15,000,000 (empty) |
|
||||||
|
| date | `payload[18:22]` | day 23, month 9, year `0x07EA` = 2026 |
|
||||||
|
|
||||||
|
The battery reading independently corroborates: Thor's own event reports for
|
||||||
|
these units print `BatteryLevel : 3.8 volts`.
|
||||||
|
|
||||||
|
**`SUB 0x2C` — call-home config.** Contains the ASCII string `RADIO RING`.
|
||||||
|
Worth flagging: that is the *exact* string seen in the RV50 `ALEOS_SERIAL`
|
||||||
|
debug during the BE12599 incident —
|
||||||
|
`'ATQ1^MATE0^MATS0=2^M^MRADIO RING^M'`. So this block holds the modem dial /
|
||||||
|
answer strings, and it is the most directly relevant command to the
|
||||||
|
call-home-receiver goal. Field layout **not yet mapped**; Series III's map
|
||||||
|
(`raw[5]` enabled, `raw[6:46]` dial string) is a starting hypothesis only, and
|
||||||
|
the length already differs (`0x7E` vs `0x7C`).
|
||||||
|
|
||||||
|
**`SUB 0x06` — storage range.** All zeros on this unit, consistent with
|
||||||
|
`memory free == memory total`. Series III reads first/last event keys from
|
||||||
|
the final 8 bytes; untestable until the unit holds events.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The event chain — walked end to end (2026-09-23, 5 events)
|
||||||
|
|
||||||
|
With 5 events on the bench unit (4 waveform + 1 histogram), the Series III
|
||||||
|
browse walk works unmodified:
|
||||||
|
|
||||||
|
```
|
||||||
|
1E (all-zero params) -> first key + size
|
||||||
|
0A (key) -> partial record, histogram only
|
||||||
|
0C (key) -> 210-byte waveform record
|
||||||
|
1F (all-zero params/browse) -> next key + size
|
||||||
|
... repeat ...
|
||||||
|
1F -> all-zero key = NULL SENTINEL, chain ends
|
||||||
|
```
|
||||||
|
|
||||||
|
The sentinel terminated correctly after exactly 5 events.
|
||||||
|
|
||||||
|
### Event keys are sequential, not addresses
|
||||||
|
|
||||||
|
```
|
||||||
|
055d4a81 055d4a82 055d4a83 055d4a84 055d4a85
|
||||||
|
```
|
||||||
|
|
||||||
|
**This is a real divergence.** Series III keys are flash-buffer *addresses*
|
||||||
|
(`01110000`, `011121F2`, …) that advance by the event's byte length, which is
|
||||||
|
why its 5A chunk walk is address-arithmetic. Micromate keys are a plain
|
||||||
|
incrementing counter. Any port of the Series III download walk must not
|
||||||
|
assume key arithmetic means anything.
|
||||||
|
|
||||||
|
### The 4 bytes after the key are the event's size
|
||||||
|
|
||||||
|
`1E`/`1F` return `[key 4B][size 4B]`. Series III uses that slot as an offset
|
||||||
|
to the next key; here it is a byte count:
|
||||||
|
|
||||||
|
| key | size | kind |
|
||||||
|
|---|---|---|
|
||||||
|
| `055d4a81` | 4,076 | histogram |
|
||||||
|
| `055d4a82` | 11,032 | waveform |
|
||||||
|
| `055d4a83` | 11,502 | waveform |
|
||||||
|
| `055d4a84` | 13,424 | waveform |
|
||||||
|
| `055d4a85` | 8,746 | waveform |
|
||||||
|
|
||||||
|
Consistent with real file sizes (corpus `.IDFH` ≈ 3.7–25 KB, `.IDFW` ≈
|
||||||
|
8.6–15.8 KB), and the histogram is unmistakably the small one. ⚠ Inferred,
|
||||||
|
not proven: the sizes sum to 48,780 while monitor status reports 57,344 bytes
|
||||||
|
used, so ~8.5 KB of overhead is unaccounted for.
|
||||||
|
|
||||||
|
### `SUB 0x0C` — waveform record, and it carries the job metadata
|
||||||
|
|
||||||
|
**Length `0xD2` = 210 bytes — identical to Series III.** Contents confirmed
|
||||||
|
across all 5 events:
|
||||||
|
|
||||||
|
- the event key, echoed
|
||||||
|
- date + time (`17 09 07 ea` → 23 Sep 2026, then `10 21` → 16:33 — matching
|
||||||
|
the actual bench recording time)
|
||||||
|
- title note `"Location"`
|
||||||
|
- **the project string** — `"Univ of Pitt-1st Yr Housing-Loc1 Ruskin"`
|
||||||
|
- serial `"UM12947"`
|
||||||
|
- channel labels `Tran` / `Vert` / `Long` / `Mic` — the same labels Series III
|
||||||
|
uses, and the same label-relative float32 layout
|
||||||
|
- per-event float32 peaks: 3.5152, 1.3720, 2.3542, 3.5152, 0.4227 in/s
|
||||||
|
across the five events (varied deliberately during recording)
|
||||||
|
|
||||||
|
**This closes the biggest open question for the call-home-receiver goal.**
|
||||||
|
The job identity strings (`project` / `client` / `operator` / `setup`) that
|
||||||
|
today arrive only via Thor's `.txt` sidecar — and which no amount of sample
|
||||||
|
decoding can reconstruct — are **available over the wire from `0x0C`**. A
|
||||||
|
direct-to-SFM event need not arrive with blank metadata.
|
||||||
|
|
||||||
|
### `SUB 0x0A` — partial record, histogram only
|
||||||
|
|
||||||
|
`0x0A` returned `len = 0x1E` (30 B) for the histogram and `len = 0x00` for all
|
||||||
|
four waveforms. The histogram payload carries **two timestamps** and the
|
||||||
|
ASCII string `"\r Vert: 0.300 in/s"` — structurally the Series III
|
||||||
|
**monitor-log partial record** (`0x2C` type), which likewise holds a start/stop
|
||||||
|
pair and a `"Geo: <float> in/s"` trigger string.
|
||||||
|
|
||||||
|
So on the Micromate the division of labour is: `0x0A` describes interval-style
|
||||||
|
records, `0x0C` describes triggered events. Series III uses `0x0A`'s
|
||||||
|
*response length* (`0x46` vs `0x2C`) to tell real events from boundaries;
|
||||||
|
that discriminator does not apply here.
|
||||||
|
|
||||||
|
### DLE stuffing in responses — confirmed present
|
||||||
|
|
||||||
|
Earlier marked untested. The `0x0C` timestamp field contains `10 10`, which
|
||||||
|
destuffs to a single `0x10` and yields a sensible clock reading. **Responses
|
||||||
|
are DLE-stuffed**, so a parser must destuff before applying field offsets.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## A/B: Blastware build vs Thor build (2026-09-23)
|
||||||
|
|
||||||
|
UM12947 (`11.0CB`) and UM20147 (`11.0BD`) were each put on the bench and given
|
||||||
|
the identical read-only sweep. **Both answer Series III command frames.**
|
||||||
|
|
||||||
|
| | UM12947 `11.0CB` | UM20147 `11.0BD` |
|
||||||
|
|---|---|---|
|
||||||
|
| POLL answers | ✓ | ✓ |
|
||||||
|
| All 10 read SUBs answer | ✓ | ✓ |
|
||||||
|
| `response_SUB = 0xFF − req` | ✓ | ✓ |
|
||||||
|
| DLE-aware checksum valid | ✓ | ✓ |
|
||||||
|
| Two-step probe/data read | ✓ | ✓ |
|
||||||
|
| ID string | `MM/ISEE/S/IO` | `MM/ISEE/S` |
|
||||||
|
| **flags byte** | **`0xC5`** | **`0x03`** |
|
||||||
|
| **`0x1C` length** | **`0x2C`** | **`0x30`** |
|
||||||
|
|
||||||
|
**The firmware line does not change the wire protocol.** One protocol stack
|
||||||
|
can drive the whole fleet regardless of which build a unit is on. This is the
|
||||||
|
single most consequential finding so far: "standardise the fleet on one
|
||||||
|
firmware" becomes an *optional* convenience rather than a prerequisite for
|
||||||
|
building a call-home receiver.
|
||||||
|
|
||||||
|
### The two differences that do exist
|
||||||
|
|
||||||
|
**1. The flags byte identifies the build.** Response `payload[1]` is `0xC5`
|
||||||
|
on the Blastware line and `0x03` on the Thor line, constant across all ten
|
||||||
|
SUBs on both units. That makes firmware line detectable from *any* response,
|
||||||
|
without reading device info. ⚠ Two units, one each — treat as a strong
|
||||||
|
hypothesis, not a proven encoding.
|
||||||
|
|
||||||
|
Note `0x03` is ETX, so on Thor-line units it arrives DLE-escaped as `10 03`.
|
||||||
|
A parser that fails to destuff will mis-locate every field by one byte on
|
||||||
|
exactly half your fleet.
|
||||||
|
|
||||||
|
**2. `SUB 0x1C` (monitor status) is 4 bytes longer on the Thor line** —
|
||||||
|
`0x30` vs `0x2C` — with four extra trailing bytes (`0f a0 00 00`, purpose
|
||||||
|
unknown).
|
||||||
|
|
||||||
|
⚠ **This breaks relative-to-end parsing.** Series III reads battery and
|
||||||
|
memory from the *end* of the `0x1C` block (`[-10:-8]`, `[-8:-4]`, `[-4:]`).
|
||||||
|
Those offsets are correct on `11.0CB` and wrong on `11.0BD` — applying them
|
||||||
|
blindly to UM20147 yields a battery reading of **577.92 V**. Parse forward
|
||||||
|
from the declared length instead of backward from the end.
|
||||||
|
|
||||||
|
With the offsets shifted by 4, UM20147 reads correctly: battery **3.81 V**,
|
||||||
|
memory 15,000,000 total and free (no events stored).
|
||||||
|
|
||||||
|
## Divergences from Series III (running list)
|
||||||
|
|
||||||
|
1. **No `DLE` prefix on responses** — bare `STX`.
|
||||||
|
2. **Response flags byte is `0xC5`**, not `0x10`.
|
||||||
|
3. **Call-home config is 126 bytes**, not 124.
|
||||||
|
4. **Data lengths are discoverable** from the probe response at `payload[9]`.
|
||||||
|
4b. **Event keys are a sequential counter**, not flash addresses.
|
||||||
|
4c. **`1E`/`1F` return the event size**, where Series III returns an offset.
|
||||||
|
4d. **`0x0A` vs `0x0C` split by record type**, not by the `0x46`/`0x2C`
|
||||||
|
length discriminator Series III uses.
|
||||||
|
4e. **Response `payload[1]` (flags) encodes the firmware line** — `0xC5`
|
||||||
|
Blastware, `0x03` Thor — where Series III has a constant `0x10`.
|
||||||
|
5. **Modem serial rate is 115200**, not 38400 (per TMI provisioning practice;
|
||||||
|
not independently verified here).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The firmware images are unencrypted — and they document themselves
|
||||||
|
|
||||||
|
`ref-stuff/micromate-firmware/MICROMATE(CB).BIN` and `MICROMATE(BD).BIN`,
|
||||||
|
~2.77 MB each and within 192 bytes of one another.
|
||||||
|
|
||||||
|
- **Entropy 6.08 bits/byte** — neither encrypted nor compressed. Plain code
|
||||||
|
and data.
|
||||||
|
- Header is a **big-endian vector table**, handlers at `0x4010_30xx`.
|
||||||
|
- **~16,700 extractable strings**, including the developers' own debug
|
||||||
|
`printf` format strings with function names intact.
|
||||||
|
|
||||||
|
This is a legitimate interoperability reference for hardware TMI owns, and it
|
||||||
|
short-circuits work I had scoped as "only answerable from a live modem
|
||||||
|
capture".
|
||||||
|
|
||||||
|
### The call-home state machine, verbatim
|
||||||
|
|
||||||
|
```
|
||||||
|
ACH_NOT_STARTED → ACH_IDLE → ACH_INITIALIZING → ACH_CONNECTING
|
||||||
|
→ ACH_CONNECTED → ACH_TRANSFER_DATA
|
||||||
|
→ ACH_RETRY / ACH_QUITTING (also ACH_STARTED)
|
||||||
|
```
|
||||||
|
|
||||||
|
Supporting strings:
|
||||||
|
|
||||||
|
```
|
||||||
|
ACH: Entry StartCallHome()
|
||||||
|
ACH: CallHome_task ; CheckAliveTime CANCEL ; TimeBetweenRetries = %d
|
||||||
|
ACH: CallHome_task ; !ExpectedCommunicationsDetected() CANCEL ; TimeBetweenRetries = %d
|
||||||
|
ACH: CallHomeCommectionCompleteProcessing() ACH=%s EAMWC=%s
|
||||||
|
ACH: %s() three attempts and it's over
|
||||||
|
ACH: %s() Send CMD_START_MONITOR
|
||||||
|
ACH: %s() Send CMD_STOP_MONITOR
|
||||||
|
ACH: Start Ignore request, Call Home is in progress
|
||||||
|
```
|
||||||
|
|
||||||
|
What this tells us without a single captured packet:
|
||||||
|
|
||||||
|
1. **Retry limit is three** — "three attempts and it's over".
|
||||||
|
2. **`ExpectedCommunicationsDetected()` gates the session.** If the host does
|
||||||
|
not say something the unit recognises, the call is *cancelled* and
|
||||||
|
rescheduled after `TimeBetweenRetries`. A homebrew receiver must satisfy
|
||||||
|
this check or units will retry forever — which is exactly the failure mode
|
||||||
|
seen on BE12599.
|
||||||
|
3. **The unit stops monitoring to call home and restarts afterwards**
|
||||||
|
(`Send CMD_STOP_MONITOR` / `CMD_START_MONITOR`). Relevant to any
|
||||||
|
wedged-unit rescue: the monitoring state around a call is the device's own
|
||||||
|
doing, not ours.
|
||||||
|
4. **Calls are not re-entrant** — "Call Home is in progress" is ignored.
|
||||||
|
|
||||||
|
### Internal command table
|
||||||
|
|
||||||
|
`CMD_CALLHOME`, `CMD_CALLHOME_CANCEL`, **`CMD_CALLHOME_CONNECTION_CONFIRMED`**,
|
||||||
|
`CMD_CALLHOME_CONNECTION_COMPLETE`, `CMD_CALLHOME_SET_SCHEDULE`,
|
||||||
|
`CMD_CALLHOME_CLEAR_SCHEDULE`, `CMD_STOP_CALLHOME_FILETRANSFER`,
|
||||||
|
`CMD_DUTYCYCLE_AUTOCALLHOME`, `CMD_PURGE_EVENT_FLASH`.
|
||||||
|
|
||||||
|
`CONNECTION_CONFIRMED` as a distinct state from `CONNECTION_COMPLETE` implies
|
||||||
|
a **handshake the host must complete before data flows** — the concrete shape
|
||||||
|
of `ExpectedCommunicationsDetected()`.
|
||||||
|
|
||||||
|
### Event delivery — the mechanism, probably
|
||||||
|
|
||||||
|
```
|
||||||
|
All Events Uploaded
|
||||||
|
Mark/Unmark File Delete Marked Events Marked Events were Deleted
|
||||||
|
MONITOR::MESG PURGE_EVENT_FLASH BEGIN / END
|
||||||
|
```
|
||||||
|
|
||||||
|
A **marking** mechanism exists, alongside a distinct "all uploaded" terminal
|
||||||
|
state. 🔶 **Inferred:** events are *marked* as transferred rather than
|
||||||
|
deleted on send, and purging is a separate explicit act. If so, a receiver
|
||||||
|
that fails to mark would see the same events re-offered every call — the
|
||||||
|
question that gates a safe homebrew receiver. **Not yet confirmed**; needs
|
||||||
|
either a live call-home capture or disassembly around these strings.
|
||||||
|
|
||||||
|
### A full user manual is embedded
|
||||||
|
|
||||||
|
The firmware carries its own HTML help, which documents configuration we would
|
||||||
|
otherwise have to infer:
|
||||||
|
|
||||||
|
- **Modem mode**: `Generic` (through a modem) vs `USB to PC`.
|
||||||
|
- **Modem baud**: 9600 / 19200 / 38400 / 57600 / 115200 / 230400 — "must match
|
||||||
|
the expected rate of the PC or modem". Confirms 115200 is a *setting*, not
|
||||||
|
a fixed rate.
|
||||||
|
- **Modem relay + warmup** (0–300 s), auxiliary mode, warning/alarm hold.
|
||||||
|
- **Record modes**: Waveform, Waveform Manual, Histogram, Histogram-Combo;
|
||||||
|
sample rates 1024 / 2048 / 4096.
|
||||||
|
- **A scheduler downloaded from THOR** that can start/stop monitoring, change
|
||||||
|
record mode, trigger a call home, or run a self check on a daily/weekly
|
||||||
|
schedule. Pairs with `CMD_CALLHOME_SET_SCHEDULE`.
|
||||||
|
|
||||||
|
### Still worth doing
|
||||||
|
|
||||||
|
Diffing the two images should isolate exactly what the CB/BD split changes —
|
||||||
|
we know the wire protocol is not it, and the flags byte (`0xC5` vs `0x03`)
|
||||||
|
gives a concrete anchor to search for.
|
||||||
|
|
||||||
|
## `SUB 0x5A` — bulk download. It streams the `.IDFW` file verbatim.
|
||||||
|
|
||||||
|
**The complete read path works with no Instantel software in the loop.**
|
||||||
|
|
||||||
|
### It needs no arming sequence
|
||||||
|
|
||||||
|
Series III ignores a `5A` probe unless preceded by
|
||||||
|
`1E → 0A → 1E(token 0xFE) → 0C → 1F(token 0xFE) → POLL × 3`. The Micromate
|
||||||
|
answers a **bare `5A` request** with nothing before it. That whole ritual is
|
||||||
|
gone.
|
||||||
|
|
||||||
|
### The offset word is a LENGTH, not a position
|
||||||
|
|
||||||
|
This is the key divergence. Series III walks chunks by absolute flash
|
||||||
|
address, stepping `0x0200` per request. On the Micromate the offset word
|
||||||
|
requests *how much to send*:
|
||||||
|
|
||||||
|
```
|
||||||
|
offset_word = 0x1000 + 2 × pages pages = ceil(event_size / 512)
|
||||||
|
```
|
||||||
|
|
||||||
|
| `offset_word` | pages | destuffed file bytes |
|
||||||
|
|---|---|---|
|
||||||
|
| `0x1002` | 1 | 518 |
|
||||||
|
| `0x1004` | 2 | 1,030 |
|
||||||
|
| `0x1006` | 3 | 1,541 |
|
||||||
|
| `0x102C` | 22 | **11,033 — the whole event** |
|
||||||
|
|
||||||
|
`event_size` comes from the chain walk (the 4 bytes after the key in
|
||||||
|
`1E`/`1F`). **One request returns the entire event**; there is no chunk loop,
|
||||||
|
no `STRT` end-offset parsing, and no `TERM` frame. Over-requesting is safe —
|
||||||
|
`0x1030` (24 pages) returned exactly the same bytes as `0x102C`, so the device
|
||||||
|
caps at the real size.
|
||||||
|
|
||||||
|
Params are the Series III *probe* form: `[0x00][key4][6 × 0x00]`.
|
||||||
|
|
||||||
|
### The payload is the `.IDFW` file, byte for byte
|
||||||
|
|
||||||
|
```
|
||||||
|
[18-byte frame header] [ .IDFW file ] [chk] [ETX] ← raw wire
|
||||||
|
^ destuffed offset 16
|
||||||
|
```
|
||||||
|
|
||||||
|
The file begins `00 12 01 00 00 00 "Instantel\0"` — `_THOR_PREFIX` +
|
||||||
|
`_INSTANTEL_TAG` from `micromate/idf_file.py`. The first 32 bytes are
|
||||||
|
**identical to a production `.IDFW`** pulled from the store.
|
||||||
|
|
||||||
|
⚠ Responses are DLE-stuffed. Destuff before locating the file, or the raw
|
||||||
|
byte count overshoots (11,781 raw → 11,049 destuffed for an 11,032-byte event).
|
||||||
|
|
||||||
|
### End-to-end proof
|
||||||
|
|
||||||
|
Event `055d4a82` downloaded over USB and fed straight to `read_idf_file()`:
|
||||||
|
|
||||||
|
```
|
||||||
|
serial UM12947
|
||||||
|
timestamp 2026-09-23 16:33:19
|
||||||
|
samples Tran 3072 Vert 3072 Long 3072 MicL 3072
|
||||||
|
peaks Tran 0.2433 Vert 1.3706 Long 0.2672 in/s
|
||||||
|
```
|
||||||
|
|
||||||
|
All four channels equal length, and the timestamp matches the `0x0C` record
|
||||||
|
for the same key. **Independent cross-check:** `0x0C` reports a stored peak
|
||||||
|
of **1.3720** for this event; the decoded samples give **1.3706** — two
|
||||||
|
unrelated paths agreeing to 0.1%.
|
||||||
|
|
||||||
|
**Consequence:** no new codec work is needed. The bytes off the wire are the
|
||||||
|
same bytes `thor-watcher` forwards today, so `/db/import/idf_file` ingests a
|
||||||
|
directly-downloaded event unchanged. Everything the IDF decoder already does
|
||||||
|
per-sample-exact applies.
|
||||||
|
|
||||||
|
### What a full read now looks like
|
||||||
|
|
||||||
|
```
|
||||||
|
1E → first key + size
|
||||||
|
0C(key) → project/client/operator, timestamp, peaks
|
||||||
|
5A(key, 0x1000+2×ceil(size/512)) → the whole .IDFW
|
||||||
|
1F → next key + size (until null sentinel)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Setups are FILES, not a config block
|
||||||
|
|
||||||
|
Series III has one compliance config you overwrite. Series IV keeps **named
|
||||||
|
setup files on an on-device filesystem**, with a pointer to the current one.
|
||||||
|
From the firmware:
|
||||||
|
|
||||||
|
```
|
||||||
|
csetup.MMB the current setup
|
||||||
|
factory.MMB Factory Default Setup File
|
||||||
|
callhome.MMB call-home config is a file too
|
||||||
|
"Current Setup File: " "Can Not Delete Active Setup File"
|
||||||
|
GetSelectedSetupFilePathName() CSelectSetupFiles CSaveSetupFile
|
||||||
|
```
|
||||||
|
|
||||||
|
Names are up to 20 characters and may contain spaces, hyphens, underscores.
|
||||||
|
The unit's help text describes selecting, renaming and deleting them, and the
|
||||||
|
event list records which setup file produced each event.
|
||||||
|
|
||||||
|
Filesystem primitives exist internally (`NS_ReadFile_internal`,
|
||||||
|
`NS_WriteFile_internal`, `NS_SeekFile_internal`), but **no generic
|
||||||
|
file-transfer command is exposed on the wire** — the only file-transfer string
|
||||||
|
is `CMD_STOP_CALLHOME_FILETRANSFER`. So setups are unlikely to be pushed as
|
||||||
|
raw `.MMB` blobs over the protocol.
|
||||||
|
|
||||||
|
### `SUB 0x1A` reads the whole active setup — 2,092 bytes
|
||||||
|
|
||||||
|
Structurally close to Series III's ~2,126-byte compliance block, and it
|
||||||
|
carries everything a setup consists of:
|
||||||
|
|
||||||
|
- **the setup file name** — `Univ of Pitt-1st Yr. Housing-Loc1 Ruskin.MMB`
|
||||||
|
- all four title note/value pairs — `Location`, `Client`, `Company`,
|
||||||
|
`General Notes`, with their strings
|
||||||
|
- the sensor location string (`Loc 1`)
|
||||||
|
- per-channel labels *and units*: `Tran in./s.`, `Vert in./s.`,
|
||||||
|
`Long in./s.`, `Mic psi (L)`, `LMic psi (L)`, `SMic (A)`
|
||||||
|
|
||||||
|
Note `LMic` / `SMic` — linear and sound-level microphone variants that
|
||||||
|
Series III does not have.
|
||||||
|
|
||||||
|
This is the **read half of setup management**, and it means a setup can be
|
||||||
|
round-tripped: read the active config, modify, write it back. The write half
|
||||||
|
is not yet attempted.
|
||||||
|
|
||||||
|
## Static analysis of the firmware (2026-09-23, solo session)
|
||||||
|
|
||||||
|
### Architecture
|
||||||
|
|
||||||
|
**ColdFire / 68K, big-endian, Freescale MQX RTOS** — not ARM as the vector
|
||||||
|
table first suggested. The giveaway is the function epilogue/prologue
|
||||||
|
`4E 5E 4E 75 4E 56` = `UNLK A6` / `RTS` / `LINK A6`, littered through both
|
||||||
|
images, plus an `MQX_OK` assertion string.
|
||||||
|
|
||||||
|
### CB vs BD: the same source, ~10 lines apart
|
||||||
|
|
||||||
|
A byte diff is useless — **68% of bytes differ** because the two are separately
|
||||||
|
linked builds with everything relocated. A *string-set* diff is
|
||||||
|
position-independent and tells the real story: **17,128 strings shared**, and
|
||||||
|
almost every "unique" string is the same message with a different source line
|
||||||
|
number:
|
||||||
|
|
||||||
|
```
|
||||||
|
CB: MONITOR[3268]: STATUS_BATTERY_LOW
|
||||||
|
BD: MONITOR[3258]: STATUS_BATTERY_LOW ← consistently 10 lines apart
|
||||||
|
```
|
||||||
|
|
||||||
|
The offset is exactly 10 across `STATUS_BATTERY_LOW`, `STATUS_BATTERY_CRITICAL`,
|
||||||
|
`Battery Critical Exit Monitor`, `histogram interval size of 0` and
|
||||||
|
`Offsets by channel` — so one ~10-line block differs in the monitor module and
|
||||||
|
essentially nothing else. The only functional string unique to either build is
|
||||||
|
`CITIZEN` (a receipt-printer brand) in BD.
|
||||||
|
|
||||||
|
**This corroborates the bench A/B from the other direction:** the CB/BD split
|
||||||
|
is a tiny code delta, not two protocol stacks. Whatever drives Instantel to
|
||||||
|
ship two downloads, it is not a different wire protocol.
|
||||||
|
|
||||||
|
⚠ The `SUB` dispatch is a 68K switch jump table (`CMPI.L` bounds check →
|
||||||
|
`MOVE.W (table,PC,Dn)` → `JMP (d8,PC,Xn)`). Byte-pattern hunting will not
|
||||||
|
isolate the write opcodes — that needs a real disassembler.
|
||||||
|
|
||||||
|
### Call-home config field names, from the firmware's own debug dump
|
||||||
|
|
||||||
|
```
|
||||||
|
CallHome.Enable = %s
|
||||||
|
CallHome.DialString = "%s"
|
||||||
|
CallHome.Retries = %d
|
||||||
|
CallHome.SessionTimeout = %d
|
||||||
|
CallHome.WaitForConnection = %d
|
||||||
|
CallHome.WarmupTime = %d
|
||||||
|
CallHome.PowerSave = %s
|
||||||
|
```
|
||||||
|
|
||||||
|
Seven fields, which is what the 126-byte `SUB 0x2C` block has to encode. Note
|
||||||
|
`SessionTimeout` and `PowerSave` have no Series III equivalent, and Series III's
|
||||||
|
scheduled-time fields (`time1/time2 hour/min`) are absent here — consistent
|
||||||
|
with Series IV moving scheduling into the THOR-downloaded scheduler instead.
|
||||||
|
|
||||||
|
Also present: `AT+CSQ` (signal quality), so the firmware talks AT to the modem
|
||||||
|
directly.
|
||||||
|
|
||||||
|
## All five bench events, downloaded and decoded
|
||||||
|
|
||||||
|
Read-only, over USB, no Instantel software:
|
||||||
|
|
||||||
|
| key | declared size | got | decoded |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `055d4a81` | 4,076 | 4,076 | histogram, 1 interval, 16:33:16 |
|
||||||
|
| `055d4a82` | 11,032 | 11,032 | waveform, 3072 × 4 ch, 16:33:19 |
|
||||||
|
| `055d4a83` | 11,502 | 11,502 | waveform, 3072 × 4 ch, 16:33:27 |
|
||||||
|
| `055d4a84` | 13,424 | 13,424 | waveform, 3072 × 4 ch, 16:33:34 |
|
||||||
|
| `055d4a85` | 8,746 | 8,746 | waveform, 2048 × 4 ch, 16:33:36 |
|
||||||
|
|
||||||
|
Every event arrived at exactly its declared size, every channel came out equal
|
||||||
|
length, and the timestamps are sequential across the recording session.
|
||||||
|
|
||||||
|
### Record type + filename: generate it, don't detect it
|
||||||
|
|
||||||
|
`read_idf_file()` decides waveform vs histogram from the **filename suffix** —
|
||||||
|
and there is no filename when downloading over the wire.
|
||||||
|
|
||||||
|
⚠ Worth correcting a natural assumption: **Series III does not detect this from
|
||||||
|
content either.** `event_file_io.derive_record_type_from_filename()` reads the
|
||||||
|
last character of the extension (`M529LKIQ.G10H` → `H` → Histogram). Nothing
|
||||||
|
in the codebase infers record type from file content, for either family.
|
||||||
|
|
||||||
|
And there is no obvious type field to find. The first 64 bytes of a histogram
|
||||||
|
and a waveform are byte-identical; they diverge at ~`0x0947` into wholly
|
||||||
|
different structures rather than differing by a flag.
|
||||||
|
|
||||||
|
**The answer is the Series III pattern — generate the name.** Series III has
|
||||||
|
`blastware_filename()`, which builds a name from serial + timestamp + type.
|
||||||
|
Series IV needs the same thing, and its convention is far simpler:
|
||||||
|
|
||||||
|
```
|
||||||
|
<serial>_<YYYYMMDDHHMMSS>.IDF{W,H} e.g. UM12947_20260923163319.IDFW
|
||||||
|
```
|
||||||
|
|
||||||
|
versus Series III's `<letter><serial3><4-char base-36 stem><AB0T ext>`, where
|
||||||
|
the stem is base-36 of seconds-since-1985 ÷ 1296.
|
||||||
|
|
||||||
|
All three inputs are already available on a direct download:
|
||||||
|
|
||||||
|
| input | source |
|
||||||
|
|---|---|
|
||||||
|
| serial | `extract_binary_metadata()` — decoded from the IDF header |
|
||||||
|
| timestamp | `extract_binary_metadata()` — same |
|
||||||
|
| **type** | **the chain walk** — `SUB 0x0A` length `0x1E` = histogram, `0x00` = waveform |
|
||||||
|
|
||||||
|
Verified against all five bench events: the generated names match the
|
||||||
|
convention of real files in the production store byte for byte. A directly
|
||||||
|
downloaded event can therefore be filed under exactly the name Thor would have
|
||||||
|
given it, and `/db/import/idf_file` needs no change at all.
|
||||||
|
|
||||||
|
⚠ The type still comes from the *protocol*, not the payload — so a downloader
|
||||||
|
must carry it out of the chain walk. Losing it means losing the ability to
|
||||||
|
name the file correctly.
|
||||||
|
|
||||||
|
### ⚠ Unresolved: the `0x0C` peak float
|
||||||
|
|
||||||
|
The float32 extracted from `0x0C` runs 2–5% above `max(Tran, Vert, Long)` from
|
||||||
|
the decoded samples:
|
||||||
|
|
||||||
|
| key | `0x0C` float | max channel |
|
||||||
|
|---|---|---|
|
||||||
|
| `…81` | 3.5152 | 3.4419 |
|
||||||
|
| `…82` | 1.3720 | 1.3706 |
|
||||||
|
| `…83` | 2.3542 | 2.2227 |
|
||||||
|
| `…84` | 3.5152 | 3.4419 |
|
||||||
|
| `…85` | 0.4227 | 0.4198 |
|
||||||
|
|
||||||
|
It is not peak vector sum either (computed PVS runs *higher* than both). The
|
||||||
|
field may not be the peak at all — its offset was inferred from a byte marker,
|
||||||
|
not established. **Do not rely on it** until it is pinned properly.
|
||||||
|
|
||||||
|
Worth noting the histogram (`…81`) and the loudest waveform (`…84`) report
|
||||||
|
*identical* peaks to four decimals, in both measures. That is self-consistent:
|
||||||
|
the histogram's single 1-minute interval spans the whole thumping session, so
|
||||||
|
its maximum should equal the loudest event in it.
|
||||||
|
|
||||||
|
## ⚠ Untested and unsafe-until-agreed
|
||||||
|
|
||||||
|
Nothing below has been sent to a unit, and nothing should be without an
|
||||||
|
explicit decision:
|
||||||
|
|
||||||
|
- **Writes** (`0x68`–`0x83`), **call-home write** (`0x7E`/`0x7F`)
|
||||||
|
- **Erase** (`0xA3` / `0xA2`)
|
||||||
|
- **Start / stop monitoring** (`0x96` / `0x97`)
|
||||||
|
- `0x1F` (advance event pointer) — non-destructive on Series III but it does
|
||||||
|
move device state, so it is parked with the rest
|
||||||
|
|
||||||
|
Also unknown:
|
||||||
|
|
||||||
|
- Whether `0x10` bytes inside request params need stuffing
|
||||||
|
- Everything about the **call-home session** — the device-initiated direction
|
||||||
|
has not been observed at all. Specifically: how a unit announces itself,
|
||||||
|
and **how it learns an event was accepted so it stops re-sending it.**
|
||||||
|
That last question gates any homebrew receiver and cannot be answered over
|
||||||
|
USB.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Session provenance
|
||||||
|
|
||||||
|
Unit **UM12947**, firmware **`11.0CB`** (Blastware line), on the bench via
|
||||||
|
USB, **zero events stored** (memory free == total). Read commands only.
|
||||||
|
Every response in this document was checksum-validated.
|
||||||
|
|
||||||
|
⚠ Firmware is the single biggest caveat on this document. Every finding here
|
||||||
|
is from one unit on the Blastware build. A `11.0BD` unit has not been
|
||||||
|
touched.
|
||||||
|
|
||||||
|
An empty unit is a real limitation: `0x08`, `0x1E`, `0x0A` and `0x06` all have
|
||||||
|
event-dependent payloads that could not be exercised. Recording a couple of
|
||||||
|
events on the bench unit would unlock the entire event-walk half of the
|
||||||
|
protocol.
|
||||||
@@ -296,6 +296,16 @@ def apply_report_to_event(event: Event, report: BwAsciiReport) -> None:
|
|||||||
event.sample_rate = report.sample_rate_sps
|
event.sample_rate = report.sample_rate_sps
|
||||||
if report.record_time_s is not None:
|
if report.record_time_s is not None:
|
||||||
event.rectime_seconds = report.record_time_s
|
event.rectime_seconds = report.record_time_s
|
||||||
|
# The report's event_datetime is Blastware's exact trigger time (parsed
|
||||||
|
# from Event Time + Event Date). Prefer it over the binary footer's stop
|
||||||
|
# time so a report-paired import matches BW to the second.
|
||||||
|
edt = report.event_datetime
|
||||||
|
if edt is not None:
|
||||||
|
event.timestamp = Timestamp(
|
||||||
|
raw=b"", flag=0x10,
|
||||||
|
year=edt.year, unknown_byte=0, month=edt.month, day=edt.day,
|
||||||
|
hour=edt.hour, minute=edt.minute, second=edt.second,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def apply_bw_report_dict_to_event(event: Event, bw_report: dict) -> None:
|
def apply_bw_report_dict_to_event(event: Event, bw_report: dict) -> None:
|
||||||
@@ -808,6 +818,30 @@ def derive_record_type_from_filename(filename, default: str = "Waveform") -> str
|
|||||||
return _RECORD_TYPE_BY_EXT_SUFFIX.get(ext[-1].upper(), default)
|
return _RECORD_TYPE_BY_EXT_SUFFIX.get(ext[-1].upper(), default)
|
||||||
|
|
||||||
|
|
||||||
|
# Marker for the recording-setup config block, and the offset of the record-time
|
||||||
|
# float32 within it. The configured post-trigger record time (seconds) is a
|
||||||
|
# big-endian float32 exactly 30 bytes before the "Standard Recording Setup"
|
||||||
|
# label. Verified across the corpus reading 1.0 / 2.0 / 3.0 s on different
|
||||||
|
# setups — and ts2 - record_time reproduces Blastware's trigger to the second
|
||||||
|
# (N844LQHB: stop 10:33:32 - 3.0 = 10:33:29).
|
||||||
|
_RECSETUP_MARKER = b"Standard Recording Setup"
|
||||||
|
_RECTIME_OFFSET_BEFORE_MARKER = 30
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_record_time_seconds(raw: bytes) -> Optional[float]:
|
||||||
|
"""The configured post-trigger record time in seconds, from the recording-
|
||||||
|
setup config block, or None when absent / implausible."""
|
||||||
|
a = raw.find(_RECSETUP_MARKER)
|
||||||
|
if a < _RECTIME_OFFSET_BEFORE_MARKER:
|
||||||
|
return None
|
||||||
|
off = a - _RECTIME_OFFSET_BEFORE_MARKER
|
||||||
|
try:
|
||||||
|
rt = struct.unpack(">f", raw[off:off + 4])[0]
|
||||||
|
except struct.error:
|
||||||
|
return None
|
||||||
|
return rt if 0.05 <= rt <= 600.0 else None
|
||||||
|
|
||||||
|
|
||||||
def read_blastware_file(path: Union[str, Path]) -> Event:
|
def read_blastware_file(path: Union[str, Path]) -> Event:
|
||||||
"""
|
"""
|
||||||
Parse a Blastware waveform file into an Event.
|
Parse a Blastware waveform file into an Event.
|
||||||
@@ -917,6 +951,10 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
|
|||||||
# rest of the event (timestamp, waveform_key, project strings) is
|
# rest of the event (timestamp, waveform_key, project strings) is
|
||||||
# still recoverable and useful.
|
# still recoverable and useful.
|
||||||
decoded = decode_waveform_v2(body)
|
decoded = decode_waveform_v2(body)
|
||||||
|
# Discriminator for the timestamp logic below: a waveform (trigger) event
|
||||||
|
# vs a histogram window. Keyed on the codec, not the filename — the
|
||||||
|
# save_imported_bw path passes a tmp ".bw" name whose extension lies.
|
||||||
|
is_waveform_body = decoded is not None
|
||||||
if decoded is None:
|
if decoded is None:
|
||||||
decoded = decode_histogram_body(body)
|
decoded = decode_histogram_body(body)
|
||||||
if decoded is None:
|
if decoded is None:
|
||||||
@@ -948,7 +986,31 @@ def read_blastware_file(path: Union[str, Path]) -> Event:
|
|||||||
ev.total_samples = strt_fields.get("total_samples")
|
ev.total_samples = strt_fields.get("total_samples")
|
||||||
ev.pretrig_samples = strt_fields.get("pretrig_samples")
|
ev.pretrig_samples = strt_fields.get("pretrig_samples")
|
||||||
|
|
||||||
if ts1 is not None:
|
# Event timestamp. The footer's two timestamps mean different things by
|
||||||
|
# record type:
|
||||||
|
# * Waveform: ts1 = the monitoring-SESSION start (shared across every
|
||||||
|
# event that day — a unit arming at 06:00 stamps 06:00 on all of them),
|
||||||
|
# ts2 = THIS event's recording STOP. Blastware's Date/Time is the
|
||||||
|
# TRIGGER = ts2 - record time, and the record time is a float32 in the
|
||||||
|
# recording-setup config block (see _parse_record_time_seconds), so the
|
||||||
|
# exact trigger is recoverable from the binary alone. Falls back to ts2
|
||||||
|
# (the stop, within the record duration) if the config block is absent.
|
||||||
|
# (Stamping ts1 showed the session start, hours off.)
|
||||||
|
# * Histogram / undecodable: ts1 = the window start, which IS the event
|
||||||
|
# time — keep it.
|
||||||
|
# Discriminate by ``is_waveform_body`` (the codec), not the filename.
|
||||||
|
if is_waveform_body and ts2 is not None:
|
||||||
|
_stop = datetime.datetime(ts2.year, ts2.month, ts2.day,
|
||||||
|
ts2.hour, ts2.minute, ts2.second)
|
||||||
|
_rt = _parse_record_time_seconds(raw)
|
||||||
|
_trig = _stop - datetime.timedelta(seconds=_rt) if _rt is not None else _stop
|
||||||
|
ev.timestamp = Timestamp(
|
||||||
|
raw=footer[10:18],
|
||||||
|
flag=0x10,
|
||||||
|
year=_trig.year, unknown_byte=0, month=_trig.month, day=_trig.day,
|
||||||
|
hour=_trig.hour, minute=_trig.minute, second=_trig.second,
|
||||||
|
)
|
||||||
|
elif ts1 is not None:
|
||||||
ev.timestamp = Timestamp(
|
ev.timestamp = Timestamp(
|
||||||
raw=footer[2:10],
|
raw=footer[2:10],
|
||||||
flag=0x10,
|
flag=0x10,
|
||||||
|
|||||||
Vendored
BIN
Binary file not shown.
@@ -0,0 +1,54 @@
|
|||||||
|
"""Event timestamp decode — waveform trigger/stop vs histogram window start.
|
||||||
|
|
||||||
|
The Blastware footer holds two timestamps: ts1 = footer[2:10], ts2 = footer[10:18].
|
||||||
|
Their meaning depends on record type:
|
||||||
|
|
||||||
|
* Waveform: ts1 is the monitoring-SESSION start (e.g. 06:00 for a unit that
|
||||||
|
arms at 06:00 daily — shared across every event that day), and ts2 is THIS
|
||||||
|
event's recording STOP. read_blastware_file used to stamp events with ts1 →
|
||||||
|
every waveform showed the session start (~4.5 h off). Binary-only, the best
|
||||||
|
estimate is ts2 (the stop); the exact trigger BW displays (= ts2 - record
|
||||||
|
duration) comes from the paired report's event_datetime, since the binary
|
||||||
|
STRT record-time byte is a misparsed record-type marker.
|
||||||
|
* Histogram: ts1/ts2 are the ~24 h window [start, stop]; the event time is the
|
||||||
|
window start = ts1 (unchanged).
|
||||||
|
"""
|
||||||
|
import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from minimateplus.event_file_io import read_blastware_file, apply_report_to_event
|
||||||
|
from minimateplus.bw_ascii_report import BwAsciiReport
|
||||||
|
from minimateplus.models import Event
|
||||||
|
|
||||||
|
FIX = Path(__file__).parent / "fixtures"
|
||||||
|
WAVEFORM = FIX / "fft-oracle-2026-09-14" / "N844LQHB.ZT0W" # footer ts2 = 2026-08-25 10:33:32
|
||||||
|
HISTOGRAM = FIX / "ts-fix" / "K441LKZU.C30H" # window start 2026-05-10 19:04:50
|
||||||
|
|
||||||
|
|
||||||
|
def _tuple(ts):
|
||||||
|
return (ts.year, ts.month, ts.day, ts.hour, ts.minute, ts.second)
|
||||||
|
|
||||||
|
|
||||||
|
def test_waveform_timestamp_is_exact_trigger_from_binary():
|
||||||
|
ev = read_blastware_file(WAVEFORM)
|
||||||
|
# The EXACT Blastware trigger, from the binary alone: ts2 (stop 10:33:32)
|
||||||
|
# minus the config record time (3.0 s) = 10:33:29 — NOT the 06:00:13
|
||||||
|
# monitoring-session start the old decode used.
|
||||||
|
assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 33, 29), _tuple(ev.timestamp)
|
||||||
|
|
||||||
|
|
||||||
|
def test_histogram_timestamp_is_window_start_unchanged():
|
||||||
|
ev = read_blastware_file(HISTOGRAM)
|
||||||
|
# Histogram event time = the window start (ts1); must NOT get the waveform
|
||||||
|
# ts2 treatment (that would land ~24 h off).
|
||||||
|
assert _tuple(ev.timestamp) == (2026, 5, 10, 19, 4, 50), _tuple(ev.timestamp)
|
||||||
|
|
||||||
|
|
||||||
|
def test_report_event_datetime_is_authoritative_over_binary():
|
||||||
|
# The binary already yields the exact trigger, but a paired report stays
|
||||||
|
# authoritative (e.g. if the unit clock had drifted) — applying it wins.
|
||||||
|
ev = read_blastware_file(WAVEFORM)
|
||||||
|
assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 33, 29) # exact, from binary
|
||||||
|
apply_report_to_event(ev, BwAsciiReport(
|
||||||
|
event_datetime=datetime.datetime(2026, 8, 25, 10, 35, 0)))
|
||||||
|
assert _tuple(ev.timestamp) == (2026, 8, 25, 10, 35, 0) # report wins
|
||||||
Reference in New Issue
Block a user