d522d31d631ddf27815d568d80744aa4ca50f093
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  serversdownandClaude Opus 5
|
d522d31d63 |
docs(series4): what THOR's "status check" actually is -- 11 commands, 563 MB/month
First capture through bridges/mm_link.py, with THOR polling a unit over the bench
link at "check connection every 5 s / check status every 5 s".
A status check is ELEVEN commands, not one:
POLL -> DEVICE_INFO -> 0x49 -> 0x5C -> MONITOR_STATUS -> SETUP_NAME_READ
-> STORAGE_RANGE -> 0x02 -> OPERATOR -> 0x47 -> CALLHOME_CFG
Each check opens a NEW TCP connection, runs all eleven exchanges in ~300 ms and
closes it. Measured over 21 consecutive checks: 236 B out, 936 B back, plus a
full handshake each time -- about 2.2 KB per check.
At the observed cadence that is 18.8 MB/day, 563 MB/month, per unit. On a
metered cellular plan that is real money, and most of it is waste: the check
re-reads the call-home config, operator name, active setup name and full device
info every ten seconds, none of which changes. SETUP_NAME_READ alone returns 274
bytes a time. POLL + MONITOR_STATUS answers "alive?" and "monitoring?" in two
commands and 131 bytes.
Both intervals set to 5 s yields one combined pass every 10.1 s, steady across
eight measured connections. So the two settings are not independent 5-second
timers, which is a plausible reason changing them appears to do nothing.
REVISES an earlier hypothesis. Because idle polling reconnects every cycle, a
silently-dead link is LESS dangerous while idle than I assumed -- a dead socket
fails at connect and the next cycle retries. The exposure is during OPERATIONS:
THOR held one connection from 00:30 to 00:47 last night while downloading events
and pushing setups. A link dying mid-operation leaves it waiting on a socket the
OS will not fail for ~2 h. The blackhole test should therefore be run during a
download, not while idle.
Also fixes a mislabel in mm_link.py: the SUB byte is DLE-escaped when its value is
0x02/0x03/0x04/0x10, so reading it raw reported SUB 0x02 as "SUB_10".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
45f2997a5b |
feat(bridges): mm_link -- a bench "modem" with a readable log and fault injection
THOR gives almost no visibility into a connection: a refresh button, two poll
intervals, and no way to see whether a check succeeded, timed out, or was never
attempted. When a unit "won't stay connected" there is nothing to look at. This
sits where the cellular modem would and answers that directly.
Over socat -x it adds the two things that were missing:
* A READABLE LOG. Frames are decoded and timestamped as they pass --
"THOR->unit POLL (21 B)" rather than hex -- so THOR's polling cadence, and
its silences, are visible. Raw .bin pairs are still written alongside and
load straight into scratch/mm_frame_parse.py.
* FAULT INJECTION, via a control file read on the fly:
pass normal relay
blackhole TCP stays up, bytes are swallowed
drop close the connection abruptly
delay:N forward N seconds late, both directions
onewaydev THOR->unit passes, unit->THOR is swallowed
`blackhole` is the point of the exercise. It reproduces the classic cellular
failure -- socket open at both ends, nothing crossing -- which a real cell link
will not do on cue. THOR was observed last night holding one TCP connection for
17 minutes (00:30 to 00:47), so if the link dies silently the OS will not tell it
for roughly the default keepalive, ~2 hours. That is a candidate explanation for
"refresh does nothing and only a restart helps", and this makes it testable
rather than speculative.
No pyserial: the port is driven through stdlib termios. The bench hosts are
whatever is to hand and requiring a pip install on someone else's machine is a
poor trade for ~30 lines. Deployed and verified on mint-mac (Python 3.12, no
third-party modules) against UM12947 -- a POLL round-trips and decodes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
96a8831472 |
docs(series4): pin down what the ACH config's volatile field is NOT
Followed up the write[118:120] field flagged in the previous commit. A third
sample plus a brute-force sweep rules out most of the obvious explanations.
Samples: 43 23 at 00:50 (idle), still 43 23 at 01:16 twenty-six minutes later,
then 2e 5e after a config write. Thor echoes back whatever it last read --
including a value that no longer matches the config it is sending -- and the
write is accepted regardless.
Ruled out:
* a clock or timer -- identical across 26 minutes of idle; only a write moved it
* a counter -- it decreased, 17187 -> 11870
* computed by Thor -- Thor demonstrably sends a stale value
* a standard CRC16 -- swept all 65,536 polynomials x init {0x0000,0xFFFF} x all
four reflection combinations over four candidate regions.
No match. Recorded so the sweep is not repeated.
It behaves like a unit-computed hash: a one-byte input change scattered the output
(XOR 0x6D7D) where a sum would move by 1. But two samples cannot separate that
from a nonce regenerated per write.
Operationally it does not matter, which is the point: the unit does not validate
the field on input, so read-modify-write with the rest of the block echoed
verbatim is provably safe. Never synthesise or zero it.
Notes what would resolve it -- several config writes with times recorded, cheap to
collect during any future ACH capture.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
8e37803d40 |
docs(series4): event download, PER-EVENT delete, and the ACH config write
Three captures with operator ground truth (Thor screenshots of the event list and
the weekly schedule). All Thor-originated.
DAY AND SCHEDULE TYPE, both settled by a weekly schedule. Thor's screen showed
"Start Monitoring 8:00 AM every day, alternating TEST1/test2, Repeat Weekly
disabled". The file is 7 x 260 + 4 = 1824 bytes and every record matches row for
row:
Day = 0 Sunday .. 6 Saturday
and [0] on record 0 read 4, against 2 and 3 in the daily schedules, so it carries
schedule TYPE as well as Repeat: 2 daily, 3 daily+repeat, 4 weekly,
5 weekly+repeat (predicted, unobserved). Bit 0 is Repeat; 2 and 4 are the bases.
Same bitmask style as [6].
In daily schedules Day reads 3 everywhere and is presumably ignored -- inference,
and the value 3 is unexplained.
EVENT DOWNLOAD. SUB 0x93 -> 0x6C arms each event before 1E/1F, with empty params
and an all-zero ack -- the Series IV analogue of Series III's 1E(token=0xFE), and
simpler. Event keys are a plain sequential counter (055D4A81..86 for six events)
at data[11:15], with the event size at data[17:19]. SUB 0x0A walks the list as
30-byte timestamped records; the dates match Thor's event list exactly.
DELETE IS PER-EVENT -- and this is the last piece a homebrew ACH receiver was
missing:
0xA8 params[0:4] = <event key> -> ack 0x57
0xAA params = zeros -> ack 0x55
The operator deleted the top row of Thor's list (the newest event) and 0xA8
carried 055D4A86, the highest key from the walk. Confirmed end to end.
Strictly safer than Series III, which can only erase everything: a receiver can
delete exactly what it has confirmed it stored. Different opcodes -- do not reach
for 0xA3/0xA2. Noted that SUB 0x06 read identically before and after, so it is
not a way to confirm a deletion landed.
ACH CONFIG. 0x2C / 0x7E / 0x7F with acks 0xD3 / 0x81 / 0x80 -- identical to
Series III. 126-byte write payload, offset 0x007E; the 0x2C read returns the same
bytes behind an 11-byte prefix. The enable flag is write[5]: 0x05 enabled, 0x04
disabled. Bit 0 is the flag, bit 2 set in both states -- do NOT test for
0x01/0x00 as Series III does. Dial string at write[6:] ("RADIO RING").
Flagged: write[118:120] changed on its own between the two sessions (43 23 ->
2e 5e) with nothing touched, and Thor writes back whatever it read. Round-trip
that field, never synthesise it. Beyond the enable byte and dial string the field
map is NOT established -- only one setting was varied, and Series III's offsets
are a hypothesis, not a transfer.
Every command on the unsafe list is now observed. None has been originated by
us, which is the line that still matters.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
f839541d08 |
docs(series4): CORRECT the schedule record -- [6] is the Action bitmask, not [0]
A five-entry schedule (start/stop/self-check/start/ACH) overturns the two
previous readings of this record, and the operator supplied a Thor screenshot of
the schedule as ground truth.
[6] is the Action, and the values are powers of two:
2 = Start Monitoring 4 = Stop Monitoring
8 = Self Check 16 = Auto Call Home
Bits 1-4 of a bitmask; bit 0 (value 1) is unobserved -- a natural home for the
setup-less DUTYCYCLE_START_MONITOR, but that is a guess.
Two retractions:
* [6] was recorded as "the one unidentified field" and predicted to be the
Repeat flag. It is the Action.
* [0] was labelled Action, then "Action with repeat folded in". Both wrong.
[0] is non-zero only on record 0 -- records 0 and 3 here are the SAME action
with different [0] values. It is a schedule-level field carried in the first
record, holding Repeat: 3 on, 2 off, matching "Repeat Daily: Disabled" on the
Thor screen.
The earlier repeat capture was consistent with both readings because it had one
start entry and moved one byte. A single-variable test is not always enough; it
took four distinct actions to separate the fields.
SUB 0x47 is confirmed as the scheduler enable. Previously recorded as "genuinely
undetermined" whether it sets or reads -- Thor's notification pane timestamps it:
schedule write completes 00:51:29, "successfully SENT" 00:51:31, the 0x47 pair at
00:51:31 and 00:51:33, "successfully ENABLED" 00:51:35. Nothing else sits between
the two notifications. params[7] in {1,3} is still open and is more likely a
selector than a value, since a lone params[7]=3 also appears at session start.
It must be DLE-escaped -- a bare 0x03 truncates the frame.
SECOND RETRACTION: setups ARE written as raw .MMB files. This document twice
said they are not. Thor used both paths in one session, choosing by whether the
setup is active:
TEST1.mmb (active) 0xDA -> 0x68/0x73 -> 0x82/0x83 -> 0x71/0x72
test2.mmb (not active) 0x8D \system\setups\test2.mmb -> 0x8E (2192 B)
The .MMB file is nearly the compliance block -- 1968/2086 bytes equal (94.3%) at
a 4-byte shift, 102 bytes longer, name at offset 38 vs 42. Same structure,
different framing. Writing a setup as a file is the cleaner path for SFM: two
frames, no 0xDA/0x68/0x82 ritual, and it does not disturb the active setup.
Also: file writes are chunked. The schedule's 1,304 bytes went as 1,024 + 280
with each offset = that chunk's length, but the 2,192-byte setup went in one
frame, so 1,024 is not a hard ceiling. Rule unexplained, recorded as observed.
Capture provenance: the seismo_lab bins were empty (capture not stopped), and the
session was recovered from the socat relay log again -- 38 frames each way, 0 bad
checksums. That fallback has now saved two captures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
cc1388df65 |
feat(scratch): recover captures from the socat relay log -- validated byte-exact
The bench relay runs socat with -x, which hex-dumps every forwarded byte in both directions. That makes its log a complete second copy of every capture taken through it, independent of whether seismo_lab was recording. On 2026-09-25 a capture's .bin files never left the Windows machine and the session was rebuilt from the relay log instead. When the real bins turned up afterwards, the reconstruction was byte-for-byte IDENTICAL in both directions (3,595 and 4,004 bytes) -- verified again through the committed script, not just the ad-hoc version used at the time. So this is a validated fallback, not a lossy approximation. Adds scratch/socat_log_split.py, with --from-line/--to-line for picking one session out of a log that spans several (split on the "accepting connection" markers, or the frame walk runs sessions together). Also documents the -x flag and the fallback in the session-provenance section, so the next person runs the relay in a way that keeps the safety net. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 5
|
98cbdad489 |
docs(series4): Repeat is folded into Action -- and [6] is not the repeat flag
A capture that changed ONLY the Repeat Daily checkbox (pull schedule, disable
repeat, push schedule) moved exactly one byte in the entire session:
record @0 Action 0x03 -> 0x02
Record 2 unchanged, [6] unchanged, and 0xDA / 0x68 / 0x82 / 0x71 / 0x94 / 0x8D all
byte-identical.
Two corrections to the previous commit:
* [6] is NOT the repeat flag. That was the field I predicted this capture would
isolate; it stayed 2 and 16. Still unidentified.
* The label "Action" on [0] was too simple -- it carries repeat behaviour too.
Leading hypothesis: 2 and 3 are the two setup-bearing start actions the firmware
names, with repeat selecting between them --
0 = DUTYCYCLE_CALLHOME
2 = DUTYCYCLE_START_MONITOR_WITH_SETUP (repeat off)
3 = DUTYCYCLE_START_MONITOR_WITH_SETUP_STOP_COMPLETE (repeat on)
Supported independently by the THOR manual, which says a repeating schedule
hitting a Start Monitoring event while already monitoring will "stop the current
monitoring session, run any Auto Call Home actions, load the compliance setup and
continue monitoring" -- exactly what _STOP_COMPLETE should mean. A repeating
start must terminate the in-progress session; a one-shot start need not. The
firmware name, the manual's behaviour and the single moved byte all agree.
Explicitly NOT claiming the enum ordering: the action names were recovered with
`strings | sort`, so source order is lost. Do not infer 1 = START_MONITOR just
because it falls between the two known values. Three of six codes observed.
Also noted: Thor re-pushed the whole 2,090-byte config for a one-byte schedule
change -- a third instance of the schedule<->config coupling.
Capture provenance: the seismo_lab bins did not reach the dev box, but the socat
relay on mint-mac keeps its own timestamped -x log, and the session was
reconstructed from it byte-for-byte (25 frames each way, 0 bad checksums). That
backup log is worth keeping in the loop -- it has now saved a capture once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
4a1cccf3f5 |
docs(series4): the schedule file is DECODED -- two records, five confirmations
The operator supplied the ground truth: entry 1 is "start monitoring TEST1" at
07:30, entry 2 is "Auto Call Home" at 19:30. That decodes the file completely.
The body is two 260-byte records plus four zero bytes = 524 exactly, and every
non-zero byte falls inside them:
[0] Action (0 = Auto Call Home, 3 = start monitoring with setup)
[1:4] padding
[4] 1/2h half-hour slot, 0-47
[5] Day
[6] ?? the one unidentified field
[7] name length
[8:260] setup name, null-padded
record @ 0: Action=3 1/2h=15 -> 07:30 Day=3 [6]=2 namelen=9 "TEST1.mmb"
record @260: Action=0 1/2h=39 -> 19:30 Day=3 [6]=16 namelen=0 (no setup)
Five independent confirmations, no fitting:
1. Slots 15 and 39 match the stated 07:30 and 19:30 on a 30-minute grid, and
39-15 = 24 slots = exactly 12 hours.
2. The length byte reads 9 for TEST1.mmb, 40 for the long name in the read
capture, and 0 for the Auto Call Home entry.
3. Auto Call Home carries NO setup name -- direct proof that a schedule entry
can exist with no setup attached, which is what the earlier
DUTYCYCLE_START_MONITOR finding predicted from the firmware side.
4. The 260-byte stride lands record 2's 1/2h exactly at [264].
5. 2 x 260 + 4 = 524, the whole body, nothing left over.
CORRECTION: `27 03 10` at [264] was recorded in the previous commit as a possible
trailer. It is record 2's 1/2h, Day and [6] fields. I had assumed the file held
one record and read the second one as padding -- the non-zero bytes were sitting
there the whole time.
Still open: [6] (2 on the start entry, 16 on the ACH entry -- a Repeat or Day/Week
capture would isolate it), the four remaining action codes, and whether the file
can hold unused record slots.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
508448e2dd |
docs(series4): the schedule record format, named by the firmware itself
A debug printf in the scheduler names the record's fields outright: SCHEDULER : _ReadRecord(%d) -> %s (Action=%u, 1/2h=%u, Day=%u, Setup="%s") [WDAY=%d] They fit the captured record, and the name-length byte anchors the alignment -- it reads 40 for the 40-character name pulled off the unit and 9 for TEST1.mmb written back, same position, both directions: [0] Action = 3 [1:4] zero (padding, or Action is a uint32) [4] 1/2h = 15 -> 30-minute resolution, 48 slots/day [5] Day = 3 [6] unidentified (WDAY?) [7] name length -- 0x28=40 read, 0x09=9 written <-- confirms the layout [8:] setup name [264] trailer 27 03 10 Slot 15 would be 07:30 counted from midnight; flagged unconfirmed because the schedule's actual time was not recorded with the capture. Six duty-cycle actions, not the five previously recorded -- there is also DUTYCYCLE_START_MONITOR_WITH_SETUP_STOP_COMPLETE. THOR exposes four. Two start variants it never offers, one needing no setup file. Strengthened the setup-less-action finding and ruled out an alternative explanation I had not considered: the Micromate has a separate Timer Mode (MODE_TIMER, Monitor Once Only, under Special Setup), so START_MONITOR could have belonged to that path. It does not -- it is a case in _PSA(), the scheduler's own dispatcher for _ReadRecord's Action field, and `_PSA() send ->> CMD_DUTYCYCLE_ START_MONITOR` shows it is live code sending a real message, not a dead case. Also: SysPref.bMonitorScheduler places the scheduler enable in system preferences, which confirms from the other side why the 0x71 block was byte-identical when the scheduler was switched on -- the flag was never going to be in the compliance config. It also suggests 0x47 is a SysPref get/set rather than anything scheduler-specific, which would explain its params[7] selector and its response shape matching 0x48's page-0 descriptor. Still a hypothesis. Names the one capture that would settle the rest: a schedule with TWO entries at different times with different actions. That yields the record stride, the action code values, and the time encoding at once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 5
|
3f58402085 |
docs(series4): the ACH session, from the THOR manual -- and I was asking the wrong question
Brian uploaded the Instantel manuals (gitignored, manuals/). The THOR Operator
Manual Rev 08 settles the thing this document called the blocker for a homebrew
receiver.
I had written that what gates a receiver is "how it learns an event was accepted
so it stops re-sending it." There is no such mechanism to find, because the unit
does not track it. Per THOR manual 6.2.2.2 an ACH session is a list of
SERVER-chosen actions -- Copy events, Copy monitor log, Delete events and Logs
from Unit, Set Date/Time -- and "Only applies to events not previously
downloaded" is computer-side bookkeeping. The manual's own warning proves copy
and delete are decoupled: enable delete but disable copy and "the events and logs
will be deleted without being uploaded."
That is exactly the model our Series III ACH server already implements
(ach_state.json high-water mark, erase as a deliberate separate step). No new
mechanism is needed for Series IV. A receiver needs: accept, identify, walk the
events (already solved), keep our own high-water mark, optionally erase. The
ERASE OPCODES are now the only genuinely missing piece and stay on the unsafe
list.
Other things the manual settles:
* The session is server-driven, matching the firmware state machine. Scheduled
and event-triggered ACH differ: with Monitoring While Calling Home enabled, an
event-triggered session will NOT delete events or sync time. So a receiver
that relies on erase to avoid re-reading would silently never erase on those
units -- the high-water mark has to be primary, erase an optimisation.
* Session Time Out is unit-side only, which places it in callhome.MMB -- another
reason to read that file with 0x94.
* Units are routed by serial number with wildcards, so the serial is presented
early enough for a server to dispatch on it.
* THOR requires Idle for ACH setup too, confirming the greyed-out send is
deliberate policy rather than a device refusal.
* THOR exposes four schedule actions; the firmware has five. 6.3.2 step 8
("A Unit Setup must exist") is THOR's own requirement, while the same section
says the unit "will execute any actions in a schedule using its current
settings" -- the two pull opposite ways, consistent with START_MONITOR
existing and THOR never emitting it.
* Schedule fields to look for when the entry is decoded: action, setup name,
time, day-or-week, day selection, repeat. The captured entry has seven bytes
before the name, the right order of magnitude for that list.
Flagged: the manual's filter example contradicts its own table (it has UM* and MP*
backwards). The table is right.
Marked throughout as vendor documentation rather than observed bytes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
8c2dacf035 |
docs(series4): the schedule has a setup-less start action -- Thor never uses it
The schedule<->config coupling looked like it might be a workaround for the unit
crashing on a missing setup. The firmware says otherwise: there are two distinct
start-monitoring actions in the scheduler's duty-cycle dispatch, and only one
involves a setup file.
_PSA() case DUTYCYCLE_START_MONITOR
_PSA() case DUTYCYCLE_START_MONITOR_WITH_SETUP
Full action set: START_MONITOR, START_MONITOR_WITH_SETUP, STOP_MONITOR,
CALLHOME, SELF_CHECK, plus ON/OFF/NEXT for scheduler state.
So the coupling is not a crash workaround -- Thor picks the more demanding of two
available actions every time. SFM can emit START_MONITOR and skip the config.
On whether a missing setup would actually break the unit: the firmware suggests
graceful degradation (`Setup File Not Found`, and `Invalid parameters reset to
factory default - please review setup`, a deliberate fallback). Untested, and
recorded as untested.
Hypothesis, flagged as such: the schedule entry's leading byte may be the action
code -- the one captured entry reads `03 00 00 00 0f 03 02 [namelen][name]` and
03 would fit START_MONITOR_WITH_SETUP. One entry, nothing to diff, unverified.
Names the capture that would settle it, and which is worth more than the 0x47
disable/enable test: a schedule entry with a setup-less action (stop monitoring,
or call home). It would confirm or kill the action-code hypothesis and prove
from the other direction that a schedule needs no config push.
Also noted: DUTYCYCLE_CALLHOME -> CMD_SCHEDULE_CALL_HOME means a scheduled
call-home can make a unit dial out on demand -- the one remaining lever on the
unsolved call-home direction.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
ebcc55e2ec |
docs(series4): the schedule<->config coupling is Thor's, not the protocol's
In Thor, a schedule entry that starts monitoring forces you to attach a setup,
and sending the schedule pushes that setup too, overwriting anything with the
same name. The protocol requires none of it.
Evidence from the scheduler capture:
* The two writes are separate operations, not one transaction. The config
write ends at frame 18, Thor sends a fresh POLL preamble, and only then opens
the schedule at frame 20. Different commands, different paths:
config 0xDA -> 0x68/0x73 -> 0x82/0x83 -> 0x71/0x72
schedule 0x8D -> 0x8E
* The schedule stores a length-prefixed NAME, not a config blob. It is a
reference, and a reference does not require rewriting its referent.
* The config Thor pushed was already on the unit unchanged -- its 2,090-byte
0x71 payload is byte-identical to the previous capture's, zero differences.
Thor spent a whole block write re-sending a setup the device already had.
So SFM can, with today's protocol: enumerate setups with 0x3F/0x40, write ONLY
the schedule when the referenced setup already exists, and push a config only
when it is genuinely missing or deliberately edited. Common case drops from
524 + 2090 bytes to 524, and the write disappears entirely.
It also removes a real hazard. Because the reference is by name, and a same-name
write overwrites silently with an indistinguishable ack, Thor's pattern means
scheduling something can quietly rewrite a setup that other schedules or the
operator's own work depend on. Validating the reference instead of rewriting the
referent avoids the class of problem.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
bd876b6752 |
docs(series4): separate Thor's conventions from the protocol's requirements
SFM is not meant to reimplement Thor. Thor is the only available teacher of the
wire protocol, but almost nothing about how it sequences its work has been shown
to be required by the device, and this document was starting to blur the two --
it said "a client should mirror it" where the honest claim is "Thor does this and
we have not checked whether the unit cares."
Adds a table separating the two, with required / not-required / unknown marked
honestly, and corrects the two places that gave Thor-copying advice.
The consequential unknowns, all testable:
* Thor's POLL -> 0x15 -> 0x49 -> POLL preamble before EVERY operation.
Plausibly required (Series III needed POLL x3 before 5A) but Thor sends it
before trivial reads too.
* 0x68 and 0x82 appear in every setup push carrying near-zero payloads that
changed nothing in either capture. If optional, our setup write is 3 frames
instead of 7 with less to get wrong. Worth settling BEFORE building the
writer.
* Whether a narrower write than the full 2,090-byte block is accepted.
One place Thor's shortcut is probably worse than the alternative: it reads with
offset=0xFFFF and skips the probe, but the probe works and reports the length
rather than making us trust a fixed one.
Two reliability problems to design against, both observed rather than assumed:
a zero ack does not mean a write applied (no failing write has ever been seen),
and nothing warns before clobbering a monitoring unit's active setup.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
c5eed6fa46 |
docs(series4): RETRACT "no file transfer" -- 0x94/0x48/0x8D/0x8E read and write by path
The scheduler capture caught a generic file transfer in the open, and it
invalidates a claim made earlier today.
RETRACTION. The "Setups are FILES" section concluded "no generic file-transfer
command is exposed on the wire", reasoning from the absence of firmware strings.
Wrong. The commands exist, they carry a full filesystem path in plain ASCII,
and they were the first thing Thor did when asked for the schedule:
0x94 <path> -> 0x6B open for read
0x48 -> 0xB7 read next page, until an all-zero response = EOF
0x8D <path> -> 0x72 open for write
0x8E <body> -> 0x71 write the body
Path is unpadded with offset = its exact length; 0x94 and 0x8D sent byte-
identical payloads for "\system\schedule\schedule.dat". The 0x48 read is paged
with the page number in the response header at payload[3:5].
The lesson: absence of a firmware string is not absence of a command. Dispatch
is a 68K jump table and these carry no strings. The setups half of the original
claim survives -- setups go via 0xDA plus the config block, not via this.
Why it matters beyond the scheduler: callhome.MMB is a file too, and call-home
is the last unsolved goal. Reading it may be a matter of pointing 0x94 at the
right path. Recorded as a lead -- no path but schedule.dat has been tried.
The schedule file: an entry carries a length-prefixed SETUP FILE NAME (0x28=40
for the name read off the unit, 0x09=9 for TEST1.mmb written back -- confirmed
both directions). So a schedule entry says "at this time, load this setup",
which is how the help text's "change the record mode" works, and it couples the
scheduler to the setup list. Entry internals are NOT decoded and are recorded
as observed bytes only -- one entry, no variation to diff.
SUB 0x47 is the scheduler enable, probably: two bare frames differing only in
params[7] (0x01, 0x03). Whether it sets or reads is genuinely undetermined --
both returned the same value and there is no disabled reading to compare.
Flagged do-not-implement until a disable-then-enable capture settles it.
A prediction made before the capture -- that the Scheduler On/Off switch would
show up as a byte in the 0x71 block, since the unit's help text lists it beside
Record Mode -- did not hold. 0x71, 0x68 and 0x82 are byte-identical to the
previous capture. Noted that the test is weak (the operator re-sent the same
config), but enabling the scheduler required no config write either way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
5400f1bef7 |
docs(series4): monitoring control, the setup-list walk, and the device clock
Thor started monitoring, listed the unit's setups and stopped monitoring while
seismo_lab recorded. 40 requests, 40 responses, every checksum valid. As
before, Thor did all of it -- we have still never originated any of these.
Confirmed identical to Series III:
* SUB 0x96 start monitoring -> ack 0x69
* SUB 0x97 stop monitoring -> ack 0x68
Both bare frames, no params, no data. These were on the unsafe-until-agreed
list as entirely unobserved; they are now observed but still never sent by us.
Erase (0xA3/0xA2) is now the only genuinely untouched destructive path.
NOT identical to Series III, and worth not reusing constants for:
* The monitoring flag is SUB 0x1C data[12] = 0x0E monitoring / 0x00 idle.
Series III uses 0x10.
* SUB 0x49 -> 0xB6 is a second, cheaper monitoring indicator at data[11]
(0x02 monitoring / 0x00 idle) in a 21-byte response rather than 60. Thor
puts it in its preamble before every operation, so it is the routine check.
New this capture:
* SUB 0x1C carries the DEVICE CLOCK at data[13:21] -- day, month, year (u16
BE), hour, minute, second. Verified against the capture's own wall time.
Nothing else read so far reports the unit's time. data[17] remains
unidentified (32 monitoring, 100 idle) -- not claimed as anything.
* Memory total is exactly 15,000,000 bytes; free dropped 4,096 bytes across a
~70s monitoring session, so free memory is not stable to compare against.
* SUB 0x3F/0x40 walk the setup-file list, the same first/next shape as
Series III's 1E/1F event walk. 0x3F -> 0xC0 first, 0x40 -> 0xBF next,
terminating on an empty name. 23 setups on this unit.
* Setup records carry ONLY the name -- 11-byte header, null-terminated name,
zero padding. There is no active-setup flag; the header is byte-identical
on every record including the terminator. The active setup is identified
solely by SUB 0x41, which uses the same record format. The asterisk on the
unit's screen is UI decoration, not a field.
* TEST1.mmb, created over the wire earlier today, appears in the list and is
what 0x41 reports as active -- a written setup becomes a real enumerable
file.
Also: Thor greys out send-to-unit while a unit is monitoring, and transmits
nothing (this capture contains no 0xDA or 0x71). That is Thor policy, not a
device refusal -- nothing suggests the Micromate would reject it, and a push to
the active setup overwrites silently. Thor is guarding the footgun the protocol
leaves open, and any client we write should do the same. Checking 0x49 data[11]
first makes that cheap.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
b5e34ce8ae |
docs(series4): overwrite is protocol-identical to create -- no handshake
The firmware carries `Overwrite File`, `MFS FILE EXISTS` and `Cannot be
Overwritten`, which suggested the wire path might negotiate an overwrite. It
does not. Those strings belong to the on-device Save screen (CSaveSetupFile),
not the protocol.
A second Thor push to TEST1.mmb -- a name that now existed, and which SUB 0x41
confirmed was the ACTIVE setup -- produced an identical sequence:
* same 12 SUBs in the same order, same offset fields
* 0xDA / 0x68 / 0x82 data byte-identical
* 0x71 differs in exactly 18 bytes = the one edited note string
* all seven write acks identical and still all-zero
* no dialog on Thor
Verified on the unit: the edited General Notes string is present in the setup on
the device. The write applied silently and in place, and being the active setup
bought it no protection.
Two consequences recorded:
* A writer needs no exists-check and no overwrite negotiation.
* We have never seen this protocol report a FAILED write -- acks are all-zero
across create and overwrite alike. Do not treat a zero ack as proof a write
applied; read back with 0x41 + 0x1A and compare. And a remote push to a
monitoring unit's active setup changes what it is recording with, unprompted
-- gating that belongs in SFM, because the device will not do it.
Still untested: overwriting a non-active setup, and factory.MMB. Neither
blocks a writer.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
d33e2d85be |
docs(series4): 0xDA creates setup files -- confirmed on the device
TEST1.mmb did not exist on UM12947 before the push. After it, the setup is present in the unit's own setup list and selected as active -- verified on the Micromate's screen, not inferred from the ack. This was the last open question about whether Series IV setup management is reachable without Thor. It is: 0x41 read name, 0x1A read block, 0xDA name the target, 0x71 -> 0x72 write it back. No file-transfer primitive is needed and the target file does not have to exist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 5
|
c8d972f685 |
docs(series4): the setup-write path, observed end to end
Thor pushed a setup named TEST1.mmb to UM12947 while seismo_lab's TCP bridge
recorded both directions. We still have not originated a write frame -- the
wire format is now known, our encoder is not written.
Topology worth reusing: socat shares /dev/ttyACM0 on TCP from mint-mac,
seismo_lab relays Thor to it. Thor is pointed at 127.0.0.1 as if the unit were
a field modem. No modem, no SIM, production Thor box untouched.
The sequence is Series III's, plus one command:
Thor: 5B | 41 | 08 | 2E | 1A | DA | 68->73 | 82->83 | 71->72
unit: A4 | BE | F7 | D1 | E5 | 25 | 97 8C | 7D 7C | 8E 8D
All 12 device responses checksum-validate and every write is acked. Every
write response SUB matches the Series III table exactly.
New:
* SUB 0xDA names the target .MMB file -- 256 bytes, filename null-padded,
nothing else. This is why no generic file-transfer command exists: Thor
names the file, then writes the ordinary config block into it.
* SUB 0x41 reads the active setup's filename; SUB 0x2E reads trigger config.
* Reads are single-step -- Thor asks offset=0xFFFF and skips the probe.
* 0x71 writes the whole 2090-byte block in ONE frame, not Series III's three
chunks. 0x69/0x74 are absent.
Write-frame destuffing is `10 XX` -> `XX` uniformly, including `10 03`. Chosen
by checksum, not assumption: of four candidate rules, only this one makes all
four data-carrying write frames validate. 0x71's data holds 4 literal 0x03
bytes escaped as `10 03`, so escaping is mandatory for any writer.
The write body IS the read body -- 0x71 and the 0xE5 response align at a fixed
11-byte shift with 1902/2090 bytes equal (91.0%). Setups are read-modify-write.
The 12 differing regions are fully mapped: setup name, four 64-byte
[label:22][value:42] note entries, sensor location, and the three geo trigger
levels (0.3 -> 0.5 in/s) on a 48-byte channel stride.
Independent confirmation of the geo LSB: each channel block carries float32BE
3.10308 at label+24. 3.10308/10000 = 0.000310308 = _GEO_LSB_IPS to 8 figures,
and 10.0/3.10308*10000 = 32226.046 = the 32226.05 full scale. That value was
derived statistically from 991,415 rounding constraints in v0.30.0; the unit
reports it directly. It is exactly half Series III's 6.206053, so the ADC runs
10,000 counts per volt. Do NOT retune _GEO_LSB_IPS -- this corroborates it.
The `offset` field is NOT a single length formula: two frames are len, two are
len+2, and Series III's data[1]+2 reproduces neither. Recorded as observed
constants the device accepted; pinning the rule needs a capture with
differently-sized payloads. This doc has been wrong once by inferring a length
field -- not inferring this one.
Also adds scratch/mm_frame_parse.py, because S3FrameParser cannot see Micromate
responses at all (it scans for DLE+STX; Micromate responses start at a bare
STX). That is why the first pass at this capture looked like 12 unanswered
requests. 24/24 frames parse with 0 bad checksums.
Stale claims corrected: the "write half is not yet attempted" note, the
"empty unit" limitation (5 events since 2026-09-23), and the unsafe-until-agreed
list, which now distinguishes observed from exercised.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
701af47170 |
docs(series4): generate IDF filenames rather than detecting record type
Closes the record-type gap flagged earlier, and corrects the premise behind it.
Series III does NOT detect record type from file content --
event_file_io.derive_record_type_from_filename() reads the last character of
the extension (M529LKIQ.G10H -> H -> Histogram). Nothing in the codebase infers
record type from content, for either family.
Nor is there an obvious type field to find in an IDF: the first 64 bytes of a
histogram and a waveform are byte-identical, and they diverge at ~0x0947 into
wholly different structures rather than differing by a flag.
The answer is the Series III pattern -- generate the name. Series III has
blastware_filename(); Series IV needs the same, and its convention is far
simpler:
<serial>_<YYYYMMDDHHMMSS>.IDF{W,H} e.g. UM12947_20260923163319.IDFW
against Series III's <letter><serial3><base-36 stem><AB0T ext>.
All three inputs are already available on a direct download: serial and
timestamp from extract_binary_metadata(), and type from the chain walk (SUB
0x0A returns 0x1E for a histogram, 0x00 for a waveform). Verified on all five
bench events -- generated names match real production-store filenames byte for
byte, so a directly downloaded event can be filed under exactly the name Thor
would have given it and /db/import/idf_file needs no change.
The type still comes from the protocol rather than the payload, so a
downloader must carry it out of the chain walk; losing it means losing the
ability to name the file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
02ed22f561 |
docs(series4): firmware static analysis, and all five bench events decoded
Solo session while the bench was unattended. Read-only throughout.
Architecture: ColdFire/68K, big-endian, Freescale MQX RTOS -- not ARM as the
vector table first suggested. The tell is 4E 5E 4E 75 4E 56 (UNLK A6 / RTS /
LINK A6) throughout both images, plus an MQX_OK assertion.
CB vs BD: a byte diff is useless (68% of bytes differ -- separately linked
builds, everything relocated). A string-set diff is position-independent and
shows 17,128 strings shared, with almost every "unique" string being the same
message at a different source line:
CB: MONITOR[3268]: STATUS_BATTERY_LOW
BD: MONITOR[3258]: STATUS_BATTERY_LOW
Consistently 10 lines apart across five different MONITOR messages, so one
~10-line block differs in the monitor module and essentially nothing else. The
only functional string unique to either build is CITIZEN (a printer brand) in
BD. This corroborates the bench A/B from the other direction: the split is a
tiny code delta, not two protocol stacks.
The SUB dispatch is a 68K switch jump table, so byte-pattern hunting will not
isolate the write opcodes -- that needs a disassembler.
Call-home config field names recovered from the firmware's own debug dump:
Enable, DialString, Retries, SessionTimeout, WaitForConnection, WarmupTime,
PowerSave -- seven fields for the 126-byte SUB 0x2C block. SessionTimeout and
PowerSave have no Series III equivalent, and Series III's scheduled-time fields
are absent, consistent with scheduling moving into the THOR-downloaded
scheduler. AT+CSQ is present, so the firmware speaks AT to the modem directly.
All five bench events downloaded and decoded over USB: each arrived at exactly
its declared size, every channel equal length, timestamps sequential.
Two gaps recorded:
- No content-based record-type discriminator. read_idf_file() dispatches on the
.IDFH/.IDFW filename suffix, which does not exist over the wire, and the
first 64 bytes of a histogram and a waveform are byte-identical. The protocol
supplies one instead: SUB 0x0A returns 0x1E for a histogram and 0x00 for a
waveform, so the type must be carried from the chain walk.
- The 0x0C peak float runs 2-5% above max(Tran,Vert,Long) and is not the vector
sum either. Its offset was inferred from a byte marker rather than
established, so it may not be the peak at all. Marked do-not-rely-on.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
23cdbef737 |
docs(series4): setups are files; and the length field is a uint16
Two findings and one correction. CORRECTION: the probe response's data length is a uint16 BE at payload[8:10], not a single byte at payload[9] as an earlier draft claimed. That reading is right only while the high byte is zero. For SUB 0x1A the real length is 0x082C = 2092; read as a byte it gives 44, a 47x under-read. Setups are FILES, not a config block. Series III has one compliance config you overwrite; Series IV keeps named .MMB setup files on an on-device filesystem with a current-selection pointer -- csetup.MMB, factory.MMB, and callhome.MMB for the call-home config. Names up to 20 chars. Filesystem primitives exist internally (NS_ReadFile_internal / NS_WriteFile_internal / NS_SeekFile_internal) but no generic file-transfer command is exposed on the wire, so setups are unlikely to be pushed as raw .MMB blobs over the protocol. SUB 0x1A reads the whole active setup in 2,092 bytes -- structurally close to Series III's ~2,126-byte compliance block -- carrying the setup FILE NAME, all four title note/value pairs (Location, Client, Company, General Notes), the sensor location, and per-channel labels with units. Note LMic and SMic (linear and sound-level microphone variants) which Series III does not have. That is the read half of setup management, so a setup can in principle be round-tripped. The write half has not been attempted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 5
|
71f19c90d1 |
docs(series4): SUB 5A streams the .IDFW file verbatim -- read path complete
The complete read path now works with no Instantel software in the loop. Three divergences from Series III, all simplifications: - No arming sequence. Series III ignores a 5A probe unless preceded by 1E / 0A / 1E(0xFE) / 0C / 1F(0xFE) / POLL x3. The Micromate answers a bare 5A request with nothing before it. - The offset word is a LENGTH, not a position: 0x1000 + 2*pages, where pages = ceil(event_size / 512), and event_size comes from the chain walk. ONE request returns the entire event -- no chunk loop, no STRT end-offset parsing, no TERM frame. Over-requesting is safe; the device caps at the real size. - Params are the Series III probe form: [0x00][key4][6 x 0x00]. The payload is the .IDFW file byte for byte. It begins 00 12 01 00 00 00 "Instantel\0" -- _THOR_PREFIX + _INSTANTEL_TAG from micromate/idf_file.py -- and the first 32 bytes are identical to a production .IDFW from the store. Responses are DLE-stuffed, so destuff before locating the file (11,781 raw -> 11,049 destuffed for an 11,032-byte event). End-to-end: event 055d4a82 downloaded over USB and fed straight to read_idf_file() yields serial UM12947, timestamp 2026-09-23 16:33:19, and 3072 samples on all four channels. Cross-check: the 0C record reports a stored Vert peak of 1.3720 for this event; the decoded samples give 1.3706 -- two unrelated paths agreeing to 0.1%. Consequence: no new codec work is needed. The bytes off the wire are the same bytes thor-watcher forwards today, so /db/import/idf_file ingests a directly downloaded event unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 5
|
45007e12d8 |
docs(series4): the firmware images are unencrypted and self-documenting
Both MICROMATE(CB).BIN and MICROMATE(BD).BIN are plain code and data --
entropy 6.08 bits/byte, big-endian vector table at 0x4010_30xx, ~16,700
extractable strings including the developers' own debug printf formats with
function names intact.
This answers, from strings alone, questions I had scoped as needing a live
modem capture.
The call-home state machine, verbatim:
ACH_NOT_STARTED -> ACH_IDLE -> ACH_INITIALIZING -> ACH_CONNECTING
-> ACH_CONNECTED -> ACH_TRANSFER_DATA -> ACH_RETRY / ACH_QUITTING
And with it:
- Retry limit is three ("three attempts and it's over").
- ExpectedCommunicationsDetected() gates the session: if the host does not say
something the unit recognises, the call is cancelled and rescheduled after
TimeBetweenRetries. A homebrew receiver must satisfy this check or units
retry forever -- exactly the BE12599 failure mode.
- The unit stops monitoring to call home and restarts after
(Send CMD_STOP_MONITOR / CMD_START_MONITOR), so monitoring state around a
call is the device's own doing.
- Calls are not re-entrant.
- CMD_CALLHOME_CONNECTION_CONFIRMED exists as a state distinct from
CONNECTION_COMPLETE, implying a handshake the host must complete before data
flows.
Event delivery, inferred not confirmed: "All Events Uploaded" plus
"Mark/Unmark File" / "Delete Marked Events" / CMD_PURGE_EVENT_FLASH suggest
events are marked as transferred rather than deleted on send, with purging a
separate explicit act. If so, a receiver that fails to mark would see the same
events re-offered every call. Needs a live capture or disassembly to confirm.
The firmware also embeds its own HTML user manual, documenting modem mode
(Generic vs USB to PC), the modem baud options (9600-230400, confirming 115200
is a setting not a fixed rate), modem relay/warmup, record modes, and a
scheduler downloaded from THOR that pairs with CMD_CALLHOME_SET_SCHEDULE.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
38ad58d4a4 |
docs(series4): A/B the two firmware lines -- the protocol is the same
UM12947 (11.0CB, Blastware line) and UM20147 (11.0BD, Thor line) each given the identical read-only sweep on the bench. Both answer Series III command frames: all ten read SUBs, correct response-SUB rule, valid DLE-aware checksums, working two-step probe/data reads. The firmware line does not change the wire protocol. One protocol stack can drive the whole fleet regardless of build, which downgrades "standardise the fleet on one firmware" from a prerequisite to an optional convenience. Two differences do exist: 1. Response payload[1] (flags) is 0xC5 on the Blastware line and 0x03 on the Thor line, constant across all ten SUBs on both units -- so the build is detectable from any response without reading device info. Two units, one each, so this is a strong hypothesis rather than a proven encoding. Note 0x03 is ETX, so it arrives DLE-escaped as 10 03 on Thor-line units. A parser that does not destuff will mis-locate every field by one byte on half the fleet. 2. SUB 0x1C (monitor status) is 4 bytes longer on the Thor line, 0x30 vs 0x2C, with four extra trailing bytes (0f a0 00 00, purpose unknown). That second one breaks relative-to-end parsing: Series III reads battery and memory from the end of the 0x1C block, and those offsets yield a battery reading of 577.92 V on UM20147. Parse forward from the declared length, not backward from the end. With the shift applied, UM20147 reads 3.81 V and 15,000,000 bytes total/free. Also noted: ID string is MM/ISEE/S/IO on the Blastware unit and MM/ISEE/S on the Thor one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 5
|
492b6683a4 |
docs(series4): fleet firmware audit, and retract the Thor-compatibility claim
Physical audit of all nine Micromates: 4 on the Blastware line (11.0CB), 2 on the Thor line (11.0BD), 3 pre-split (11.0AK x2, 10.90GC). The Blastware line is already the plurality, which makes "standardise on Blastware" less disruptive than it first looked. Cross-checked against a store-derived audit (firmware is recorded in every .sfm.json as extensions.idf_report.version): 7 of 9 agree. The two that differ, UM6047 and UM14133, are the most recently deployed and were reflashed after their last stored event -- so the store reconstructs firmware history without touching a unit, but lags reality by one deployment. RETRACTION: an earlier draft suggested UM12947's trouble with Thor was explained by its Blastware firmware. Not supported. Ped Bridge runs UM11402 (11.0BD) and UM11719 (11.0CB) side by side from the same deploy date and both call Thor fine -- UM11719 has 331 Thor-collected events while on 11.0CB. A Blastware-line unit does feed Thor, so the CB/BD split is not "which host can collect from it", and UM12947's problem remains unexplained. What is actually established is narrower: a 11.0CB unit answers Series III command frames. Whether a 11.0BD unit does is untested -- and UM20147 (11.0BD) is on the bench, so that is one A/B away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 5
|
73eaa0a6ac |
docs(series4): the event chain, walked end to end
Five events on the bench unit (4 waveform + 1 histogram). The Series III browse walk -- 1E, then 0A/0C per key, then 1F to advance -- works unmodified, and the null sentinel terminated correctly after exactly 5. Findings: - Event keys are a sequential counter (055d4a81..85), NOT flash-buffer addresses. Series III key arithmetic does not carry over; its 5A chunk walk assumes addresses and must not be ported blindly. - The 4 bytes after the key in 1E/1F are the event's SIZE in bytes, where Series III puts an offset to the next key. 4,076 for the histogram and 8.7-13.4 KB for the waveforms, matching real .IDFH/.IDFW file sizes. - SUB 0x0C returns a 210-byte (0xD2) waveform record -- the same length as Series III -- carrying the event key, date/time, the title note "Location", the PROJECT STRING, the serial, channel labels Tran/Vert/Long/Mic and float32 peaks. That last point closes the biggest open question for the call-home receiver: the job identity strings that today arrive only via Thor's .txt sidecar, and which no amount of sample decoding can reconstruct, are readable over the wire. Direct-to-SFM events need not arrive with blank metadata. - SUB 0x0A returns len 0x1E for the histogram and 0x00 for every waveform. The histogram payload holds two timestamps plus a "Vert: 0.300 in/s" trigger string -- structurally the Series III monitor-log partial record. So 0A describes interval records and 0C describes triggered events; Series III's 0x46-vs-0x2C length discriminator does not apply. - DLE stuffing in responses is now confirmed (previously marked untested): the 0C timestamp contains 10 10, which destuffs to one 0x10 and yields a clock reading of 16:33 on 23 Sep 2026 -- matching when the events were recorded. Read-only throughout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 5
|
b9c52442a7 |
docs(series4): the Series III behaviour is firmware-conditional
The bench unit reports 11.0CB -- Instantel's *Blastware* firmware line. It almost certainly answers Series III commands because it is in Blastware mode, not because the Micromate natively speaks Series III. Instantel ships two lines: 11.0CB (Blastware) and 11.0BD (THOR, Vision, Vision II). That also explains the two-ACH-server problem as designed behaviour rather than misconfiguration. Corpus firmware audit: 932 event files from 11.0AK, 83 from 10.90GC. UM12947 itself produced 10.90GC files in production last year and reports 11.0CB now, so units get reflashed and firmware is not stable per-unit over time. Records the resulting strategic fork (standardise on the Blastware line vs reverse-engineer the Thor line) with the four unknowns that decide it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 5
|
095834183e |
docs(series4): open the Micromate live-protocol reference
First bench session against a Micromate over USB. The headline: the unit answers Series III command frames unmodified. An untouched Series III POLL (SUB 0x5B), built by build_bw_frame with no changes, completed a full two-step probe/data cycle. Ten Series III read commands were then tried and all ten answered, every one obeying the response_SUB = 0xFF - request_SUB rule. Confirmed this session: - Transport is a plain USB CDC-ACM port (2504:0300, "MICROMATE COM PORT"). No vendor driver, no Thor, no Windows box needed. Baud is ignored over USB (identical responses at 38400 and 115200). - Device never speaks first -- 20 s idle listen produced nothing. - Responses are Series III framing MINUS the leading DLE: bare [STX][payload][chk][ETX]. This alone means Blastware can never find a frame boundary in Micromate traffic, since its parser scans for DLE+STX. - Response flags byte is 0xC5, not Series III's 0x10. - Checksum is the DLE-aware variant (SUM8 excluding 0x10 bytes) -- the same one Series III uses for 5A and write frames, not the plain SUM8 of its ordinary reads. Disambiguated by the POLL data frame, which contains a 0x10. - The probe response carries the data length at payload[9]. Four of four known Series III lengths match; call-home config differs (0x7E vs 0x7C). - Series III monitor-status field offsets apply unchanged: battery 3.81 V (Thor's own reports say 3.8), memory 15,000,000 total and free, date 23 Sep 2026. - SUB 0x2C carries the string "RADIO RING" -- the same string seen in the RV50 ALEOS debug during the BE12599 incident. That block holds the modem dial/answer strings and is the most relevant command to the call-home goal. Read commands only. Nothing that writes, erases, or changes monitoring state has been sent to a unit; those are listed as unsafe-until-agreed. Caveat recorded in the doc: one unit, over USB, with zero events stored, so the event-walk commands (0x08, 0x1E, 0x0A, 0x06) could only be probed, not exercised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 4.8
|
154186a6cd |
Merge feat/event-timestamp-fix: exact waveform trigger time from the binary
read_blastware_file stamped waveforms with footer ts1 (the monitoring-session start, hours off — vomit-list #3). The event time is ts2 (recording stop) and the trigger = ts2 - record time, a float32 in the recording-setup config block, so the exact Blastware trigger is recovered from the binary alone (no .TXT). Histograms keep ts1; a paired report's event_datetime stays authoritative. Needs a re-decode backfill to correct existing stored events' timestamps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
1765b3300d |
docs(changelog): waveform event-time fix (exact trigger from binary)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
1e76d08b37 |
fix(decode): recover the exact waveform trigger from the binary (no .TXT)
Follow-up to the ts1→ts2 fix: get the trigger to the second from the binary alone, instead of falling back to the stop time (~record-duration late) for no-report events. The configured post-trigger record time is a big-endian float32 in the recording-setup config block, exactly 30 bytes before the "Standard Recording Setup" marker. _parse_record_time_seconds reads it; the waveform branch now stamps trigger = ts2 - record_time. Verified: the field reads 1.0 / 2.0 / 3.0 s across different setups in the corpus, and all 7 BE12844 oracle events now decode to their exact Blastware trigger (N844LQHB 10:33:29) from the binary, no paired .TXT needed. Falls back to ts2 (the stop) if the config block is absent. A paired report's event_datetime stays authoritative (clock drift). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
a84a46e9d4 |
fix(decode): stamp waveform events with the event time, not the session start
read_blastware_file built ev.timestamp from footer ts1, which for a WAVEFORM is the monitoring-session start (a unit arming at 06:00 stamps 06:00 on every event that day) — so every waveform's time was hours off (vomit-list #3, "~4.5 h off"). The event time is footer ts2 (the recording stop); BW's displayed Date/Time is the trigger = ts2 - record duration. Root cause proven against the BE12844 oracle set: 5 of 7 events decoded to the identical 06:00:13 (the shared session start); ts2 gives distinct plausible event times (N844LQHB ts2 = 10:33:32, BW trigger 10:33:29 = ts2 - 3.0 s rectime). * read_blastware_file now uses ts2 for waveforms (discriminated by which codec decoded the body, not the filename — save_imported_bw passes a tmp name). Histograms keep ts1 (the ~24 h window start, which IS the event time). * Binary-only decode can't get the exact trigger: the STRT record-time byte is a misparsed record-type marker (0x46=70), so ts2 (the stop, ~record duration after the trigger) is the best estimate. A paired BW report carries the exact trigger — apply_report_to_event now overlays event.timestamp from report.event_datetime, matching the existing build-path override (line ~441). Tests: waveform → ts2, histogram → ts1 unchanged, report → exact trigger. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 5
|
ada5bc2a82 |
docs(changelog): Unreleased — cheap connect, Diagnostics tab, tool status
Written on dev as part of finishing the merge, per the convention adopted 2026-09-18: feature branches do not touch CHANGELOG.md, and the entry describes what actually landed rather than what a branch intended. First time through the new way rather than discovering the conflict afterward — the merge was clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN |
||
|
serversdownandClaude Opus 5
|
f1ab5b1e9d |
docs: record the 5A page-boundary bug, and assess SFM as a tool
Two things Brian asked for after the BE12599 work. The known bug: the 5A walk discards the key's page byte, so once a unit has recorded more than 64 KB since its last erase, an event spanning the boundary reads an end_offset behind its own start. The chunk loop then fetches nothing and TERM packs a negative offset_word, which is the 500. Reproduced on BE12599. It hid this long because every capture the walk was verified against came from a freshly-erased BE11529 — all three confirmed TERM examples sit inside page 0x11. Prod is unaffected; it ingests complete files and never runs this walk. The status doc exists because "is SFM reliable?" has three different answers depending on which tier is meant. The codec library and the data side are production — verified per-sample at scale, carrying Terra-View daily. The device side is emergency-grade: it works, but it is synchronous, unauthenticated, and thinly tested. The lab is research artifacts. Most confusion comes from answering for the wrong tier. It covers all three of what Brian asked for: maturity per capability, an operator-facing "what to use when" (the cheap probes are cheap and the event walk is not), the known-issues table, and the gap analysis. That gap is mostly auth, async and guardrails — not protocol work. The protocol is the finished part. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN |
||
|
serversdownandClaude Opus 5
|
6589da445b |
feat(webapp): cheap connect, opt-in event walk, and a Diagnostics tab
Connecting to a unit fired /device/events automatically, which walks the whole event chain — every event header over a cellular link. On BE12599 that took minutes and then 500'd outright, because its buffer has wrapped past 0xFFFF and the uint16 offset arithmetic goes negative. Wanting to know whether ACH was on should not require reading every event the unit has stored. Connect now uses only cheap probes: /device/info (which already carries the compliance config the event walk was re-reading) plus /device/events/storage_ range. The chain walk moves behind a "Load events" button in the Events toolbar, and the Device tab gains an Event Chain card showing the first/last keys. Adds a Diagnostics tab for the endpoints that previously existed only as curl: storage_range and events/index alongside monitor/status, then stop monitoring, disable ACH (rescue?erase=false, so events survive), and erase. The wedged-unit ladder — slow drip and blind stop — sits under its own heading pointing at the runbook, with the reminder that slow_drip's success signal is bytes_received>0 and not a clean duration. Erase is guarded by typing the unit's serial. Auth answers who, not whether you meant it, and Swagger's try-it-out button on /device/events/erase is live on :8200/docs — the realistic risk here is an accident. Lifetime events is displayed but labelled unreliable: SUB 0x08 reports 0 on units with years of history, which is a decode bug we have not chased yet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN |
||
|
serversdownandClaude Opus 5
|
0b58415fe2 |
chore(release): v0.31.0 — report parity + the inverted rescue
Cuts Unreleased to v0.31.0 and writes the theme now that the whole release is visible, per the convention adopted today. Two threads landed. Blastware Event/FFT-Report parity — the FFT, the USBM RI8507 compliance chart, and the sensor self-check decoded for both series and standardized into the .h5 (schema v2, /sensor_check). And the ach_server rescue flags out of the BE12599 field emergency, which invert the wedged-unit recovery: answer the unit's call instead of racing a Stop into the gaps between its dial-outs. Version stamped in pyproject.toml, CLAUDE.md and README.md. TOOL_VERSION was already at 0.31.0 — it came in with the sensor-check work, and it is what makes the backfill pick up the new /sensor_check group without --force. ⚠ This release owes prod a backfill: .h5 schema v1 -> v2, ~2 h on the NAS. Stated in the Migration block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN |
||
|
serversdownandClaude Opus 5
|
fa22bb9f59 |
Merge feat/sensor-check-h5 into dev
Sensor self-check standardized into the .h5 (schema v2, /sensor_check group), decoded for both series-3 and series-4, plus the Thor backfill script. CHANGELOG resolved per the convention adopted today: the incoming Unreleased preamble was dropped rather than reconciled — no preamble under Unreleased, the theme gets written at release time — and its load-bearing half was folded into ### Migration, which said "None" and is now false. That block now states the real cost: .h5 schema v1 -> v2, TOOL_VERSION 0.31.0 so the standard backfill picks the traces up with no --force, and ~2 h on the NAS. The FFT, the compliance chart and the ach_server rescue flags still owe nothing. The branch's rewritten "Sensor self-check — both series" entry merged cleanly and is kept. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN |
||
|
serversdown
|
27e9c56393 |
Merge pull request 'Feat/ach rescue on connect' (#38) from feat/ach-rescue-on-connect into dev
Reviewed-on: #38 |
||
|
serversdownandClaude Opus 5
|
0408c37866 |
docs: write the changelog on dev, not on feature branches
Reverses the "entry goes in with the work" rule from two commits ago. That was wrong on the evidence: of the docs(changelog) commits in history, 3 of 4 in seismo-relay and 2 of 4 in Terra-View were made directly on dev. The rule was generalized from one unrepresentative commit rather than from the pattern. It also caused the exact problem it was supposed to avoid. With four worktrees in flight, every branch edits the same few lines at the top of CHANGELOG.md; feat/ach-rescue-on-connect and feat/sensor-check-h5 collide on that file and nothing else. Writing the entry once, on dev, after the merge removes the whole conflict class. The second benefit is accuracy: an entry written after the merge describes what actually landed, including anything that changed during conflict resolution. The sensor-check branch is a live example — its Unreleased preamble describes a release that no longer looks like that. The failure mode of writing it later is forgetting, so the merge is explicitly not finished until Unreleased is updated — same sitting, reconstructed from the branch commit messages. Unchanged: no preamble under Unreleased, the mandatory operational consequence, and cutting the version on dev when ready to ship to main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN |
||
|
serversdownandClaude Opus 5
|
a42e8d3651 |
docs: make the release cadence explicit
Brian described the practice: Unreleased is the staging area for what is going
into the next release, and the version bump happens when enough has
accumulated to be worth shipping — not per commit, not per merge. The
convention already implied it ("never touch the changelog at a merge
boundary") but never said it outright.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN
|
||
|
serversdownandClaude Opus 5
|
2fabf84d4d |
docs: adopt a changelog convention, and make Unreleased follow it
Brian asked what the standard is; there wasn't a written one, only a de facto pattern in the history. This writes it down in CLAUDE.md and fixes the one place the repo already diverged from it. The rule: write the entry in the same commit as the work, under ## Unreleased; cut the version on dev in a dedicated chore(release) commit; never touch the changelog at a merge boundary. The entry goes in with the change because that is the only moment you still know why. Two additions beyond what the history already did: No preamble under ## Unreleased. The themed opening paragraph gets written at release time, when the whole release is visible and can be named honestly. The current one proved the point — "Blastware Event/FFT-Report parity: the FFT, the USBM compliance chart, and the sensor self-check" was accurate when the first item landed and stopped being accurate once rescue-on-connect landed under the same heading. Removed here; the release commit writes a new one covering everything actually in the release. And the operational consequence is now mandatory on any entry touching the codec, the waveform store, or the DB — including when it is "none". This repo's changelog is how future-you learns whether a deploy costs two hours on the NAS, so silence is ambiguous and "none" is information. The old preamble's load-bearing half is preserved as an explicit ### Migration block rather than dropped with the prose around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN |
||
|
serversdownandClaude Opus 5
|
402bf30e37 |
docs(runbook): reframe as one disease with two cures, intercept first
The previous commit called BE12599 a second failure mode and claimed the device "never enters S3 mode at all" and that no inbound work could reach it. That was an overclaim built on a single slow_drip attempt, and Brian was right to push back. It is the same disease. Method B's step 1 worked fine on BE12599 — clearing the Destination did stop the dial-outs. It was step 2 that did not land, on one attempt, run ~90 s after a modem reboot with a dead session visible in the log in that same window; BE9558H needed hours of attempts before one landed. And the AT-init loop the ALEOS log revealed is almost certainly what BE9558H was doing too — we just never turned on serial debug in May to look. The device speaks S3 fine; it handshook cleanly the moment it had a session. What is genuinely new is the cure, and it deserves to be the default rather than a footnote. Racing a Stop into the gaps between dial-outs is a coin flip. Intercepting is deterministic: the unit dials every ~75 s, so give it somewhere to dial and answer it. It will not answer us because it is on the phone — so be the one it calls. Restructures accordingly: a "two cures" table up top, the intercept promoted to Method A with its own procedure (listener before modem, stop at step 1.5, drain before disabling ACH, restore the Destination and confirm it), and the original inbound procedure kept intact as Method B for when there is no listener the modem can reach. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN |
||
|
serversdownandClaude Opus 5
|
c6fc3d0241 |
docs: BE12599 incident — the inverted rescue, plus a rescue-listener plan
The wedged_unit_recovery runbook covered exactly one failure mode. BE12599 turned out to be a second one wearing the same symptoms, and the existing procedure did not work on it. Adds a "TWO failure modes" table up front so the next incident branches correctly, and a full second-incident section covering what the ALEOS serial debug log revealed: the device repeating a 29-byte AT modem-init string (ATQ1/ATE0/ATS0=2, no ATD) every 75 s, never getting an OK because the modem is in TCP data mode, and therefore never entering S3 mode at all. Inbound cannot win against that, no matter how well framed. Also records the two red herrings, since together they cost ~90 minutes: the RV50 trusted-IP whitelist drops non-listed sources silently (presents as a connect timeout, and Brian's dynamic dev IP had rotated off the list), and sfm/server.py returns 502 for BOTH "Protocol error:" and "Connection error:", so a 502 was misread as "TCP connected, device mute" and a theory built on it. And the gotchas worth never re-deriving: slow_drip's send_error=null plus a full duration is not success (only bytes_received > 0 is); stopping monitoring removes the call-in trigger, so it costs you the channel; --events-only skips the device-info step, so the serial is never read and ach_state keys on peer:ephemeral_port, silently breaking dedup and re-downloading the same event every session. The plan doc captures the tool Brian wants built out of this — a rescue listener with a real lifecycle and, critically, a confirmation gate before shutdown, because leaving the modem's Destination pointed at a dead listener is worse than never having started. Open questions are listed rather than guessed at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN |
||
|
serversdownandClaude Opus 5
|
9f1050b5e7 |
feat(ach): rescue-on-connect — stop monitoring / disable ACH from the server side
A unit whose geophone offset has grown past its trigger level records back-to-back and, with ACH set to "after event recorded", re-dials every time. The wedged_unit_recovery runbook handles that by reaching the unit inbound and clearing the modem's Destination Address so it stops dialing. That fails when the device is wedged mid-modem-init. BE12599 (2026-09-16) sat repeating a 29-byte AT setup string — ATQ1/ATE0/ATS0=2, no ATD — every 75 s. The modem is in TCP data mode, never interprets it, never answers OK, so the device never progresses into S3 mode and ignores every frame we send. Worse, each attempt makes ALEOS log "tcpmode trying to send to invalid socket" and re-run "Initialize Auto answer on port 9034", which orphans any held inbound session — slow_drip reports a clean 120 s hold with bytes_received=0 because the modem stopped bridging after the first re-init. Inbound cannot win that race. But the modem auto-dials its Destination whenever serial data arrives while closed, so pointing Destination at an ach_server turns those 75 s attempts into a device-initiated session that the modem bridges correctly. Adds --stop-monitoring, --disable-ach and --rescue. They run as step 1.5, after the handshake and before the event walk, each independently guarded so a failure does not abort the download. Outcome is written to rescue.json. Startup banner reports both, and warns when --restart-monitoring would undo --stop-monitoring. Prefer --stop-monitoring alone on first contact: --disable-ach stops the unit calling, which is the only channel to a unit in this state, and halting the recording ends the loop on its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qcu9ByJfuKBQxmrWb8rSrN |
||
|
serversdownandClaude Opus 4.8
|
8265e32ad5 |
feat(backfill): regenerate .h5 with /sensor_check; TOOL_VERSION 0.31.0
Complete the sensor-check standardization: existing events need their .h5
regenerated to gain the v2 /sensor_check group.
* backfill_thor_events.py attaches the decoded series-4 traces
(micromate.sensor_check) on its own IDF decode path, mirroring
save_imported_idf, so regenerated Thor .h5 files get the group. Series-3
backfill needs no change — it re-decodes via read_blastware_file, which now
attaches the traces itself.
* TOOL_VERSION 0.30.0 → 0.31.0 so the standard backfill regenerates every
event (no --force): the tool now produces the /sensor_check group. Purely
additive — no decoded value changes.
* CHANGELOG (Unreleased): sensor-check now series-3 + series-4, standardized
into the .h5 (schema v2), with the ⚠ backfill note; FFT + compliance stay
no-backfill (they read existing .h5 samples).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
|
||
|
serversdownandClaude Opus 4.8
|
8d3cdba1b5 |
feat(h5): standardize sensor-check into the .h5 (schema v2); SFM reads it
Make the sensor self-check a first-class part of the standardized decoded event
so SFM stops decoding it at report time — device-agnostic, per the store's
decoder→standardized-.h5→SFM model.
* Event gains a `sensor_check` field; both decoders attach the traces where
they set raw_samples — series-3 in event_file_io.read_blastware_file
(minimateplus.sensor_check), series-4 in waveform_store's IDF path
(micromate.sensor_check). Covers ingest and backfill (both re-decode).
* event_hdf5 bumps schema_version 1→2 and writes an optional /sensor_check
group (raw counts, int32, per channel present). read_event_hdf5 returns
it; plot_json_from_hdf5 carries it as a top-level key. Old v1 files still
read cleanly (no group → None), so nothing breaks before the backfill.
* gather_report_data reads sensor_check_waveforms from the .h5 and drops the
report-time series-3 decode — the report no longer reaches into a decoder,
and a series-4 event now lights up the same strip automatically.
Stored as raw counts (a shape diagnostic, rendered fit-to-box): the per-series
count scale differs and a physical mic unit is ill-defined, so conversion would
add complexity for no display benefit — easy to add later if a numeric use
appears.
Tests: .h5 roundtrip + backward-compat + plot_json + real series-3 decode
attaches to the Event.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
|
||
|
serversdownandClaude Opus 4.8
|
685a17d180 |
feat(series4): decode sensor self-check waveforms from the IDFW binary
The Thor/Micromate (series-4) IDFW binary carries the sensor self-check in its
fixed-header region (before the waveform body), as up to four records tagged
01 0e 3c/3d/3e/3f — the SAME channel ids as series-3 (Tran/Vert/Long/MicL).
Unlike series-3's delta-coded trailing block, series-4 stores each trace as a
raw int16-BE array after an 18-byte record header (2-byte sample count at
offset +8). Three-channel (mic-disabled) units carry only 3c/3d/3e.
New micromate/sensor_check.py: decode_idf_sensor_check(raw) locates the record
chain (id-ordered marker run, so a stray body match can't chain) and reads each
trace's int16 samples → {Tran,Vert,Long[,MicL]: [counts]}, or {} when absent.
Reverse-engineered + validated against 4 UM oracle events (added as fixtures):
clean geophone ring-downs on all, mic pulse trains on the 4-channel units,
correctly no MicL on the two 3-channel units. Validated by shape + cross-event
consistency (no Thor report strip to exact-match, unlike series-3's BW reports).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
|
||
|
serversdownandClaude Opus 4.8
|
dcd9ad6f48 |
Merge feat/fft-series3: Blastware FFT, USBM compliance chart, sensor self-check
Reverse-engineered Blastware Event/FFT-Report parity, all additive (reads the existing .h5 samples + retained raw binary, no DB/.h5 change or backfill): - Blastware-compatible channel FFT (waveform_fft) - USBM RI8507/OSMRE compliance chart on the event-report PDF (sfm/compliance) - sensor self-check strip decoded from the series-3 binary trailing block (minimateplus/sensor_check) + Frequency/Overswing sub-rows - seismo_lab Inspector hex reader (minimateplus/binary_annotate) - report fixes: stacked-lane y-tick collision, header serial fit Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
4f73e919a0 |
docs(changelog): Unreleased — FFT, USBM compliance chart, sensor self-check
Document the feat/fft-series3 work under Unreleased: Blastware-compatible channel FFT, the USBM RI8507/OSMRE compliance chart on the event-report PDF, the decoded sensor self-check strip + Frequency/Overswing sub-rows, and the seismo_lab Inspector hex reader — plus the two report-panel fixes (tick collision, header serial fit). Additive, no .h5/DB change or backfill. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
2a747f6893 |
fix(report): attach sensor-check strip to the waveform panel; move "0.0"
Match Blastware's layout, measured off the reference PDF: the sensor-check strip shares a border with the main waveform panel (no gap between them), and the per-lane "0.0" baseline labels sit to the RIGHT of the strip. Previously the strip floated with a gap and the "0.0" label overprinted the strip's left edge. Purely layout — the traces and decode are unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
2c5c20cfd7 |
fix(report): fit sensor-check mini-plots to their boxes
The sensor-check strip used a symmetric ±max scale, so the one-sided geophone ring-downs (a dip to ~-990 with the baseline at 0) sat in the bottom half of each mini-box with the top half blank — visibly off next to Blastware. Scale each mini-plot to its actual data range with a small pad instead, and draw a faint zero baseline, so the ring-downs and the mic pulse train fill their boxes the way BW draws them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
ab9d84fde6 |
feat(report): render the sensor-check strip + report polish
Wire the decoded sensor self-check waveforms (previous commit) onto the event
report PDF, and fold in two related waveform-panel cleanups.
Sensor-check strip (matches Blastware):
* ReportData gains sensor_check_waveforms; gather_report_data decodes it from
the retained raw BW binary (store.paths_for) at report time — no ingest or
.h5 change, waveform events only.
* _draw_waveform_subplot now draws a narrow right-hand strip of per-channel
mini-plots (MicL pulse train + Long/Vert/Tran ring-downs) aligned to the
lanes, captioned "Sensor Check".
* stats table gains the "Frequency" / "Overswing Ratio" sub-rows under Sensor
Check (7.5/7.7/7.3 Hz, 3.6/3.3/3.7), formatted to 1 decimal like BW; values
come from the already-parsed sensor_check scalars.
Cleanups (pre-existing, in the same panel):
* fix the stacked-lane y-tick collision — adjacent lanes' -1.0 / 1.0 labels
overprinted at the shared boundary; prune the extreme ticks (MaxNLocator
prune="both") so each lane shows clean interior ticks only.
* fix the header serial+firmware line running off the right page edge —
tighter right-column indent + BW's slightly smaller 7.5pt header.
Tests: sensor-check + compliance + geo-scale + fft all green (15). The
test_bw_ascii_report failures are pre-existing (gitignored decode-re fixtures
absent in this worktree), unrelated to this change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
|
||
|
serversdownandClaude Opus 4.8
|
6341432524 |
feat(series3): decode sensor self-check waveforms from the binary
The Blastware Event Report draws a "Sensor Check" strip on the right of the
waveform panel — the little traces the unit records when it pulses each sensor
before monitoring. Those live in the series-3 binary's trailing block, after
the main waveform record-chain and the per-channel calibration records, as four
length-prefixed records tagged 0x3c-0x3f (Tran/Vert/Long geophone ring-downs +
MicL pulse train). Reverse-engineered against 7 BE12844 oracle events.
New minimateplus/sensor_check.py: decode_sensor_check(raw) locates the record
chain (validated by walking the ids 0x3c->0x3f via their length prefixes) and
decodes each record's delta stream (payload[20:len-8]) with the same 10/20/30/00
delta-block tags as the main waveform codec, from an anchor of 0. Returns
{Tran,Vert,Long,MicL: [samples]} in raw 16-count units, or {} when absent.
Validated: mic pulse-train zero-crossing frequency = 20.1 Hz (exact match to
BW's mic Channel Test freq); geophone ring-downs are consistent ~-990 raw
deflections that damp to a ~-310 settle across all 7 events (a fixed
calibration pulse, so near-identical every run).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
|
||
|
serversdownandClaude Opus 4.8
|
dc74c97ade |
feat(report): size the USBM compliance chart to match Blastware
The compliance chart on the event-report PDF was correctly drawn but far
too small ("it's tiny") — a ~2.6in square dropped between the mic and
stats rows. Resize + reposition it to match Blastware's Event Report,
measured directly off a BW reference PDF (n844lqhbzt0w) rasterized with
fitz: the chart data box now spans figure fractions x[0.489,0.951]
y[0.502,0.867] — a ~3.9in square running from just under the header down
through the stats band, hard against the right page margin, exactly as BW
draws it. Title updated to BW's "USBM RI8507 And OSMRE".
To clear room for the BW-sized chart (waveform layout only):
* _draw_stats_table gains bbox_width/col_widths/fontsize params; the
waveform layout packs the Tran/Vert/Long table into the left ~0.42 so
its columns no longer sit under the chart. Histogram layout keeps the
wider defaults (byte-identical output; it has no compliance chart).
* the mic block's long "Channel Test Passed (Freq … Amp … mv)" line gets
a tighter indent + one-point-smaller font so it ends before the chart's
left edge instead of running behind it (_kv gains a fontsize param).
* the Peak Vector Sum line left-aligns under the compacted table (one pt
smaller) so it clears the chart's bottom-left tick labels.
Chart placement centralized in the _COMPLIANCE_BOX constant. No change to
the compliance math, the scatter, or the histogram report.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf
|
||
|
serversdownandClaude Opus 4.8
|
95f926c318 |
feat(report): enlarge the compliance chart to a full upper-right panel
The chart was cramped into the short mic band (~2in) and rendered tiny. Move it to its own large square panel (_draw_compliance_panel) spanning the mic + stats rows on the right, clear of the stats columns — matching Blastware's Event Report proportions. _draw_mic_and_usbm now draws only the mic block. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
6ad4fd73dd |
fix(compliance): square plot box (set_box_aspect) so the chart isn't squashed
The compliance chart sits in the short, wide mic-and-USBM band on the event report; without a fixed aspect matplotlib stretched it wide-and-short. Force a square plot box, which is how log-log compliance charts are conventionally drawn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
db818f716c |
feat(report): draw the USBM RI8507 compliance chart on the event report
Replace the "[compliance chart coming soon]" placeholder in _draw_mic_and_usbm with a real inset axes calling sfm.compliance.draw_compliance_chart on rd.channels / rd.sample_rate_sps (the full-rate in/s waveform samples). Title updated "USBM RI8507 And OSMRE" → "USBM RI8507" — we draw only the RI8507 lines (Drywall 0.75 + plaster 0.50); the OSMRE overlay is dropped by choice. Waveform events only (the histogram layout has no USBM chart). Falls back to a "(no waveform data)" note when samples are unavailable. Closes the 1.0 compliance-chart blocker. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
dad35e47fe |
feat(compliance): USBM RI8507/OSMRE compliance chart + reference doc
sfm/compliance.py renders the velocity-vs-frequency blasting compliance chart Blastware draws on its Event Report: - limit_at()/limit_curve() — the RI8507 Fig B-1 / 30 CFR 816.67 curve as data (Drywall 0.75 + plaster 0.50 lines): 0.030in low-freq bound, plateau, 0.008in rising diagonal to a 2.0 in/s cap at ~40 Hz, drawn continuous. - channel_compliance_points() — the per-cycle (freq, peak-velocity) scatter by the zero-crossing method (matches Blastware; cloud ceiling = channel PPV). - draw_compliance_chart() — matplotlib rendering (both lines + scatter, BW tick scales + channel markers). Verified against 7 BE12844 Blastware reports. docs/ri8507_compliance_curve.md captures the curve construction, the SHM basis, and the scatter method. Not yet wired into report_pdf.py — that placeholder is the next step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
2902ab373e |
feat(fft): Blastware-compatible channel FFT (waveform_fft)
channel_spectrum(samples, sps) → the single-sided amplitude spectrum Blastware's FFT Report draws, and dominant_frequency() picks its peak in the 2–250 Hz band. Reverse-engineered against 7 BE12844 (MiniMate Plus) events with Blastware FFT reports as ground truth. Recipe: DC-remove, NO window (a window smears the peak and worsens the match), zero-pad to 4096 (→ 0.25 Hz bins at 1024 sps — the resolution every reported dominant frequency lands on), single-sided 2/N amplitude. Reproduces Blastware's dominant frequency to the exact bin on all 28 channels and the amplitude to report precision. This is the missing piece for both the USBM RI8507 compliance chart (its scatter is these (freq, amp) points vs the limit curve) and the FFT view. Pure numpy, series-agnostic (feed it in/s samples from either decoder). The 7 events land in tests/fixtures as the oracle (force-added past the fixtures gitignore, matching 5-11-26 / decode-re-5-8-26). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
11e3e515f3 |
feat(seismo_lab): Inspector tab — annotated hex reader for Series-3 binaries
New top-level "Inspector" tab: open any Series-3 waveform binary and read it as a colour-coded hex dump driven by binary_annotate. Each region is labelled with its offset range and size (header / STRT / per-channel sample records / footer), and everything the decoder can't account for is painted UNKNOWN (red) so gaps stand out — the point being to comb for undecoded data (e.g. a stored FFT/ spectral block). A summary shows total size, region count, and % unknown. Read-only reader/translator; Series-3 only for now (Series-4 later). The GUI needs tkinter + a display (not available in the dev venv); the annotator core it calls is unit-tested headless. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
845ec38f96 |
feat(inspector): Series-3 binary structural annotator (binary_annotate)
annotate_blastware_binary(raw) → a gap-free tiling of labelled Spans (header / STRT / per-channel sample records / footer / unknown) for a hex viewer to paint. Every byte is covered; anything the decoder can't account for is a first-class `unknown` span, so undecoded regions stand out. Composes the existing waveform_codec.walk_records over the body between the STRT record and the 26-byte footer. On the cracking fixtures this already surfaces a ~1700-byte undecoded trailing region (stream-end marker + serial + …) per file — a candidate home for stored spectral/FFT data. TDD: tests assert the spans tile the whole file, STRT is located, the geo sample records are labelled, and the footer is last. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 5
|
88c0e2b765 |
chore(release): v0.30.0 — series-4 correctness
Bumps package version, README banner, CLAUDE.md header and TOOL_VERSION to
0.30.0, and cuts the CHANGELOG entry for the Thor / Micromate decoder work.
Also documents the previously-unreleased event-report PDF fix (
|
||
|
serversdownandClaude Opus 5
|
904522a9c5 |
fix(codec): 40 NN int16 blocks are not capped at NN=8
data_block_len() rejected any `40 NN` block with NN > 0x08. That guard had
no evidence behind it: every corpus available when it was written used only
NN in {1,2,3,4,8}, so it was never exercised. Loud UM12947 events use NN of
12, 16, 20 ... up to 196.
Because walk_body/run stop at the first unrecognised tag rather than
raising, rejecting those blocks surfaced as silently short channels -- e.g.
Tran 1812 / Vert 2132 / Long 2324 on a file whose export carries 2324 for
all three. The real bound is the buffer; the caller additionally clamps to
the record end.
Verified against Thor's own CSV exports for UM12947 (2025-07-14 .. 09-25,
167 waveforms, supplied as CSV.zip):
length mismatches 22 -> 0
per-sample exact 1,476,242 / 1,476,249
These are NOT truncated recordings, which was the competing hypothesis --
the exports carry the full sample count.
tests/test_waveform_codec.py asserted the cap as intended behaviour. That
assertion encoded an assumption, not a verified fact, and is replaced with
one pinning the opposite plus the evidence.
Across all three ground-truth corpora: 459 waveform files,
3,807,158 / 3,807,165 samples exact. Production IDFW is now 575/575 with
zero truncations and zero decode failures (median PPV error -0.0007% across
8 units). Series-3 re-verified unchanged at 14,338/14,338.
The 7 residual samples each differ by one 4th-decimal tick and are Thor's
own rounding: intersecting the per-sample rounding constraints over that
corpus is infeasible (binding pair contradict by 2.3e-11, 7e-5 relative),
so no single linear LSB reproduces every printed value. _GEO_LSB_IPS is
already pinned to ~1e-11; do not retune it to chase these.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
c07aaa552c |
fix(series4): support mic-disabled (3-channel) Thor units
Verified against a second Thor corpus (9-10-26-csv-req: UM11402, UM12947,
UM20147) with per-sample CSV exports: 139/139 waveforms exact
(1,273,380/1,273,380 samples) and 877/877 histograms within 2% of Thor's
reported PPV -- up from 66.9% and 56.6%.
Some units run with the microphone disabled, which changes two structural
things that were both hardcoded to the 4-channel shape:
- Waveform body head sat below the scan floor. A 3-channel unit has a
shorter fixed header and puts its record chain head at 0x0dba, under the
old _BODY_SCAN_FLOOR of 0x0E00. The scan could not see it and fell
through to the Vert segment-0 record, decoding a body shifted one
position around the channel rotation -- Vert came up exactly 512 samples
short. Floor lowered to 0x0C00. The body-offset scoring also had to stop
requiring four channels, or `equal` is permanently False for these events
and the pick falls back to raw sample count.
- Histogram interval record is 56 bytes, not 72. It is
16 * n_channels + 8, and is not inferable from the segment length alone.
The interval count now comes from the segment's cumulative counter
(n = counter - prev_counter) and the stride is derived from it. Assuming
72 read 7 intervals out of every 10-interval segment, then walked off
alignment into garbage that decoded as ~10 in/s peaks -- inflating some
files' PPV by up to 191,000%. Also recovers 4 files that previously
decoded no intervals at all.
Combined across both corpora: 292/292 waveform files,
2,330,916/2,330,916 samples exact. Production IDFW truncations 41 -> 22.
Series-3 unaffected (no shared-codec change in this commit; last full run
14,338/14,338).
Known open, diagnosed but NOT verified: the remaining 22 unequal + 1 failing
production IDFW files (all UM12947, 2025-07-14..09-23) stop the block walker
on tag 40 0c. data_block_len() caps the 40 NN int16 block at NN > 0x08 while
those files use NN up to 196. Both verified corpora only ever use
NN in {1,2,3,4,8}, so the cap is untested there and lifting it leaves both at
100.000% -- which is not evidence it decodes these correctly. Deliberately
not shipped; needs Thor CSV exports for UM12947 in that date range.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
726c2ce1b5 |
fix(series4): Thor/Micromate decoder is now per-sample exact
Verified against Thor's own CSV exports, which carry a per-sample four-column block beside every binary (CSV/<name>.IDFW.csv). Those 1,012 paired files were in the corpus all along; the decoder had been pinned to a superseded walker on the stated grounds that "Thor has no ASCII ground truth in the corpus and its geo scaling is separately suspect". Both premises were false. IDFW per-sample exact 39.1% -> 100.000% (1,057,536/1,057,536) IDFW files fully exact 0/153 -> 153/153 IDFW PPV median error -3.32% -> -0.002% IDFH within 2% of Thor PPV 51.1% -> 100.0% (858/858) prod IDFW, 8 units -3.3% -> -0.001% Four independent root causes: - Geo LSB was 0.0003, the 4-dp *display rounding* of the real 0.000310308 mistaken for the LSB, so every series-4 geophone sample read 3.3% low. Pinned to +-6e-11 by intersecting 991,415 rounding constraints; corroborated by the +-full-scale seed (+-32226) left in unwritten IDFH slots. IDFH had a separate, also wrong, 10.0/32768. - IDFH histograms were capped at 250 intervals: the segment validator required the interval counter's high byte to be zero, but the counter is a uint16 cumulative index, so every segment past interval 255 was rejected. Runs over ~4 hours lost their tail, often the peak. 540/858 corpus files affected. - Record mode 00 00 (raw int16, 10-byte header) was unhandled and fell through the dispatch, silently dropping each channel's first 512 samples -- the long-standing "loud events truncate" symptom. MODE_ABSOLUTE is now also accepted as a segment-0 preamble. - The body-offset search matched 00 02 00 *inside* record headers, selecting a candidate part-way down the chain and decoding a rotation-shifted body. It now anchors on record headers and takes the chain head (6 ms/file). Also fixes the separately tracked "UM-series decodes ~1000x low" bug. Series-3 re-verified unchanged at 14,338/14,338 exact after the shared waveform_codec change. Known open: 41/575 prod IDFW files (7%, mostly UM12947/UM20147) decode with unequal channel lengths and also fail metadata extraction -- a different header variant with no Thor export in the store. NOTE: this is a codec change; the Thor store owes a regeneration via scripts/backfill_thor_events.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 4.8
|
91b9b4578c |
fix(pdf): shared geo Y scale across Long/Vert/Tran (was per-trace)
The event-report waveform plot scaled each geo lane to its own peak, so a small channel filled its lane looking as big as a large one — and the "Geo: X in/s/div" footer only reflected whichever channel was checked first, so its div value was wrong for the other two. Now all three geo lanes share ONE symmetric scale = max |sample| across them (padded, 0.05 in/s floor), matching the event modal and BW's single amp/div; the footer reflects that shared scale. Mic keeps its own psi scale. Big events are unchanged (e.g. BE12844 stays 0.185 in/s/div). Test-first: tests/test_report_pdf_geo_scale.py (shared scale + floor), 2 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 5
|
f600fee965 |
feat(offset): the non-motion test, and BE12599 diagnosed as a connector
Brian noticed BE12599's 2026-08-09 event reports no ZC frequency because the trace never crosses zero. That is the best detector in this investigation. A geophone has no DC response, so its output must integrate to ~zero over a record. |mean|/peak is therefore ~0 for real motion and ~1 for anything electrical. Across 12,068 channel-events with peak >= 0.05 in/s the statistic is bimodal with a 1.09% dead zone, and at mp >= 0.8 it returns exactly the five confirmed units -- from physics rather than a tuned threshold. Two detectors on different principles agreeing is the strongest corroboration the list has had. It also settles BE11007 as NOT an offset: mp 0.75-0.89 but frac_neg 0.99 at peaks of 7.4-9.4 in/s, i.e. a one-sided near-full-scale blast. Journal 8e diagnoses BE12599 specifically. Its August waveforms are unipolar impulses with an RC tail (26 ms -> 118 ms -> never recovers over 14 days), and the fault MOVES between Long and Tran while the sensor self-check passes on every event. A failing element cannot hop channels; a connector can -- which also explains why the swing test never fails and why an autozero rarely helps. Corrects 8c's claim that the spread gate is blind to onsets: of 87 BE18438|Vert events it rejected one, the transitional record. Narrower than stated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
58c1fe8a96 |
docs(offset): mechanism campaign — five hypotheses dead, onset is a ramp
Records the mechanism investigation in journal 8c. The headline: still unknown, but the shape is now constrained and a long list of dead ends is closed. Onset is a ramp of minutes-to-hours, not a step — BE18438 Vert resolved to one-minute cadence via the histogram corpus, 50% of the excursion in 7 minutes, >=25 intermediates, validated 75/75 against Blastware's own ASCII. That kills both poles of the original dichotomy: not a latched digital step, not slow component wear. What survives is a reversible two-time-constant settling process, which is a shape constraint and not a mechanism. Thermal, ground-motion shock, handling/redeployment, accumulated duty, age, firmware and a mechanical element fault are each refuted or explicitly bounded, with the power behind every negative stated. Retracts two claims this journal carried: polarity consistency was a tautology of offset_scan3's spread gate, and the fleet is 8-9 units rather than 5 once that gate is dropped. Also notes the gate is blind to onsets by construction -- it rejects a moving floor, and it rejected the one record where the ramp shows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
a27310c3c6 |
docs(changelog): record the BlastMate serial fix under v0.29.0
The release is bumped but not tagged, and the fix is now in dev — which is what gets built — so the notes would otherwise understate the build. No TOOL_VERSION change: the fix alters which serial an import is filed under, not any decoded value, so no backfill is owed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdown
|
233bfcefd0 |
Merge feat/offset-histogram-scan: BlastMate serials + the histogram scan
Two things, both starting from the same root cause. The BW filename encodes the serial NUMBER only; the two-letter family prefix is not in it. Every offset scanner synthesised "BE", which mislabels the four BlastMates in the archive (BA9229, BA10060, BA10895, BA15957) and — in the store's import path — would have filed a BlastMate under a unit that does not exist. BlastMates are Series III and byte-identical to MiniMate Plus, so the serial string was the only thing blocking SFM support; reading it from the file body is the whole fix. Separately, the archive's 63,535 histograms were scanned for offsets for the first time. The result is largely a documented dead end — the detector finds 2 of the 5 confirmed units and a clean histogram is not evidence of health — but it produced the BA10895 reclassification and a labelling caveat on offset_scan3's spread gate. Journal §8b. |
||
|
serversdownandClaude Opus 5
|
84bb53e185 |
docs(offset): relabel the four BlastMate units BA, not BE
BA9229, BA10060, BA10895 and BA15957 were reported throughout as BE — the scanners synthesised the family prefix, which the BW filename does not carry. Corrected across the journal with a note recording why, so the mistake is legible rather than silently patched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
9ceff65bfb |
fix(sfm): read the serial family prefix from the file, enabling BlastMates
The BW filename encodes only the serial NUMBER — `<letter><3 digits>`, so
`L895…` is 10895 and nothing more. The two-letter family prefix is not in it:
"BE" is a MiniMate Plus, "BA" a BlastMate. Both are Series III and their
files are byte-identical in every way that matters — all 1,493 BlastMate
binaries in the DL2 archive decode through the existing codec at 100%, same
four channels — so the serial string was the only thing standing between SFM
and BlastMate support.
Two sites synthesised the prefix and got it wrong:
- waveform_store `_serial_from_bw_filename` returned f"BE{num}" on import, so
a BlastMate event was filed under a unit that does not exist, silently, and
Terra-View read it straight through. Split into
`_serial_number_from_bw_filename` (the number, which the filename really
does carry) and a new `_serial_from_bw_bytes` that reads the serial out of
the body and accepts it only when its numeric part agrees with the
filename. save_imported_bw now prefers hint -> body -> filename guess.
Verified against real archive bytes for BA9229, BA10060, BA10895, BA15957
and BE9558/BE11529/BE18003.
- client `_decode_0a_partial_header` searched for a literal b"BE" in the
monitor-log partial record. On a BlastMate that returns -1 and skips the
whole block, so the geo threshold went missing along with the serial. Now
matches any two-letter prefix, and requires the NUL terminator — stricter
than the bare two-byte search it replaces.
Nothing to migrate: no BlastMate events are in prod. The archive's BA units
last recorded 2018-10 (BA9229, BA15957), 2023-08 (BA10895) and 2023-11
(BA10060), and the prod backfill only reaches back to ~May 2025.
21 tests. Suite: 309 passed, same 16 pre-existing failures as at HEAD
(15 missing ASCII fixtures + one peak_values assertion, all untouched here).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
|
||
|
serversdownandClaude Opus 5
|
9982938b0b |
fix(offset): read the real serial from the file body, not "BE" + the number
The BW filename encodes only the serial NUMBER — `<letter><3 digits>` where
letter = chr(ord('B') + serial // 1000), so `L895…` decodes to 10895. The
two-letter family prefix is not in the filename at all, and every offset
scanner synthesized it as f"BE{num}".
Four of the 43 archive units are BA, not BE. Their binaries say so plainly:
BA9229, BA10060, BA10895, BA15957. Brian caught BA10895 by recognising that
no such unit as BE10895 exists.
serial_of() now reads the serial string out of the file body and falls back
to the old synthesis only when no matching string is found. No analysis
changes: grouping was by the numeric part, which was always correct, and no
unit number maps to more than one serial (checked across all 43).
The same assumption is live in two production sites and is NOT touched here,
because fixing ingest renames rows a running store and Terra-View already
reads them:
- sfm/waveform_store.py:870 `return f"BE{serial_num}"` on import
- minimateplus/client.py:2538 `raw_data.find(b"BE")` in the monitor-log
partial-record decode, which yields serial=None on a BA unit
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
|
||
|
serversdownandClaude Opus 5
|
1daf693b32 |
feat(offset): scan the histogram corpus — the other 90% of the archive
offset_scan3.py covers only waveforms (6,577 unique binaries). The archive also holds 63,535 unique histograms, which the pre-trigger method cannot touch: a histogram carries no samples, only a per-interval per-channel peak. scratch/offset_hist_scan.py scans them — 63,505/63,535 decoded (99.95%), 43 units, 77.9M intervals. It emits every candidate floor statistic per (file, channel) rather than deciding anything, so thresholds get calibrated against the waveform ground truth instead of guessed. Journal §8b records the outcome. What survives is a site-quiet-gated cross-channel differential that independently confirms BE18438|Vert and BE9558|Tran+Long with a clean 2.5x separation gap and 0.037% day-level false alarm, threshold-insensitive across a 2.3x span — the first operating point in this investigation to pass that test cleanly. What it does not do, recorded just as plainly: it finds 2 of the 5 confirmed units, not 5. DC leakage into the interval peak is bimodal (0.9 on BE18438, 0.02 on BE12599), so a negative histogram result is not evidence of health. Per-channel attribution is not established (channel-scramble p = 0.769) and timing resolves to ~a month, not a day. Two dead ends buried for good: the absolute floor is retired (66% of its discrimination is a day/site confound), and zero-fraction is structurally impossible — the device clamps every interval peak at >= 1 A/D count. Two findings independent of the histograms: - offset_scan3's spread<=0.02 gate discards 18.8% of rows with |pre|>=0.025, concentrated on 41 unit-channels currently labelled clean; 4 would be sustained positives without it. The fleet label is three-state, not two. - The waveform corpus observes ~7% of the days a unit was deployed. BE10895 is reclassified from transient to a genuine Vert fault of a different subtype: 49.4% single-axis-dominant events, the highest in the fleet, all on Vert. The other six marginal units are clean. Not done: the 11 thin-coverage units were not screened, and no completeness audit was run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 4.8
|
89ad7cf49d |
chore(release): v0.29.0 — offset detector + false_trigger_reason (first prod-bound build since 0.27.0)
Bumps TOOL_VERSION 0.28.0 -> 0.29.0 and pyproject/CLAUDE/README 0.27.0 -> 0.29.0, and dates the CHANGELOG section. v0.28.0 (offset DC-baseline detector) was version-bumped in-tree but never tagged or deployed, so 0.29.0 is the first build to carry both it and the false_trigger_reason column to prod. Pairs with Terra-View >= 0.24.0. false_trigger_reason auto-migrates on startup; the offset detector needs the shape backfill (scripts/backfill_event_shape.py) on the prod store to populate shape_offset* on existing rows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
523f22c96b |
Merge feat/ft-reason: optional false_trigger_reason (offset) FT subtype
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
cfdd153b5a |
docs(changelog): false_trigger_reason column under [Unreleased]
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
c0cf6547d9 |
feat(ft): optional false_trigger_reason ("offset" etc.) as an FT subtype
A reason records *why* an event is a false trigger. It is optional (plain FT flags still record no reason) and is a subtype of the FT flag: setting a reason implies false_trigger=1, and the reason is cleared whenever FT ends up 0 (confirm-real, clear-FT, set_false_trigger(false)). Twin propagation carries the reason to the histogram/waveform twin alongside the FT flag. New nullable `false_trigger_reason TEXT` column (schema + _migrate ADD COLUMN only — not the Migration-1 rebuild). 7 tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
4cf0fda804 |
chore(release): v0.28.0 — offset (DC-baseline) false-trigger detector
Bumps TOOL_VERSION 0.27.0 -> 0.28.0 (drives the SFM /health + OpenAPI version too). Rolls CHANGELOG [Unreleased] -> v0.28.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 4.8
|
3554d00583 |
feat(offset): DC-offset detector productionized into the shape pipeline
Productionizes the validated scratch/offset_scan3.py: a DC offset (baseline shifted off zero — sensor bumped/settled/drifted) is |median(pre-trigger)| >= 5 counts (0.025 in/s) AND flat across pre/mid/end thirds (spread <= 0.02); a transient moves one third and is rejected by the spread test. - shape_metrics: offset_from_samples / offset_from_h5 (reads .h5 samples + pretrig_samples attr; range-aware via the .h5's in/s float samples) - events schema: shape_offset / _axis / _pre / _spread (via _SCHEMA + the _migrate ADD COLUMN loop only; NOT the Migration-1 rebuild), threaded through insert + upsert mirroring shape_* - ingest: computed at all three waveform_store save paths alongside shape - backfill_event_shape: also computes + stores (and stale-clears) offset - exposed via /db/events automatically (SELECT *) Gating to waveforms is done downstream in terra-view ft_suspicion (mirrors how shape is ignored for histograms), not at the SFM call sites. 13 new tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 5
|
b29ca50b35 |
docs: point CLAUDE.md at the shared stack context doc
The stack-level context (version pairing across seismo-relay / Terra-View / SLMM, and which repo a change belongs in) now lives version-controlled at terra-view/docs/tmi-stack.md, symlinked as ~/CLAUDE.md. Reference it here so the three project docs are symmetric. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
e07f76dd31 |
docs: correct the v0.27.0 backfill claim — prod needs no backfill
The v0.27.0 notes said prod held 4 histograms that would stay empty until a backfill. That was wrong, and asserted without checking. Verified: all four recovered files (K440HJCN.3C0H, K557IF1U.8K0H, T191HVNP.0S0H, T193L0XM.CI0H) are archive-only — none appears in the production store or the events DB. Re-running stride detection over the prod store's 10,215 histogram binaries under both the old and new code shows 0 files whose decode changes. So the partial-final-block fix is forward-looking: it matters for future ingests of sub-minute histograms with a partial final block, not for anything already stored. TOOL_VERSION still moves with the release, so a future backfill run will regenerate the whole store instead of skipping. Harmless — byte-identical output for every stored file — but it costs the full ~2 hours on the NAS, so it should not be started casually. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
8e808b09d4 |
chore(release): v0.27.0 — decoder verified at scale; offset investigation
Bumps pyproject, TOOL_VERSION, README and CLAUDE.md to 0.27.0. sfm/server.py
now derives its version from TOOL_VERSION (
|
||
|
serversdownandClaude Opus 5
|
ad84a04404 |
feat(offset): detector v3 — pre-trigger floor with a constant-floor test
Brian's method, and better than v2's whole-record median: the pre-trigger
window is definitionally quiet (the buffer captured before the trigger fired),
whereas a median is merely robust to the event. Requiring the floor to hold
across pre-trigger / middle / end rejects transients that a median cannot.
per channel: pre/mid/end medians, spread = max - min
offset when |pre| >= floor AND spread <= 0.02 in/s
real fault >= 3 consecutive flagged events on that channel
The empirical noise floor justifies the threshold and validates the decoder:
across 19,244 non-flagged channel-events the pre-trigger floor is 62.7% exactly
0.000, 94.5% within +/-1 quantisation unit, median +0.0000, mean -0.0008. There
is no systematic zero-point bias — an independent confirmation of the
32000-count geo scale.
The result is threshold-insensitive across a 2x range (0.020 to 0.040 in/s),
which is what separates a real signal from a tuned one:
FINAL: 5 of 45 units (11%) — BE9558, BE11529, BE12599, BE13117, BE18438
BE11007 and BE10895 drop out; the spread test identifies them as transients
rather than pedestals. v1's 11% headline was right by luck — it included
BE11007 and named the wrong channel on most units.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
|
||
|
serversdownandClaude Opus 5
|
1fdc665675 |
fix(offset): retract the v1 detector — per-channel median, not dominant-axis mean
Brian challenged the v1 finding that offsets "come and go", against field experience that a unit which develops one stays broken until the geophone is replaced. He was right; v1 had two flaws, both of which manufactured false recoveries: 1. It scored only the axis with the largest peak, so a real event on one axis hid a persistent pedestal on another. BE12599 on 2026-08-21 read "clean" because Long had a 1.065 in/s event, while Tran sat at +0.4732 in/s and was never examined. 2. It used the mean, which a real transient perturbs. The median is the resting baseline and a blast does not move it. Same event, Long channel: mean +0.0783 vs median -0.0050. offset_scan2.py flags a CHANNEL when |median| >= 0.025 in/s (5 A/D counts, Instantel's own criterion) and treats >=3 consecutive flagged events as the real signal. No m/p ratio guard is needed — that existed only to compensate for the mean. Corrected results: units with any flagged event 6 -> 19 of 45 units with a sustained pedestal 8 of 45 (18%) runs >=3 consecutive 29; 1-2 event runs (noise) 69 Also corrected: the affected channel is most often Vert, not Tran (v1 named whichever axis had the largest peak, so it was frequently wrong). BE10895 and BE18003 were invisible to v1. BE12599's fault began 2026-08-14, not 08-17. The decode itself was never in question and is confirmed against Blastware's own ASCII export: on K558LJN3.BK0W, BW shows Tran parked at +0.265..+0.375 in/s for the entire record while Vert and Long sit at ~0.005 — Instantel's "parallel lines above or below the zero line". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 4.8
|
c8c4ec2b9f |
fix(sfm): /health reports the real service version, not a stale 0.1.0
terra-view's SFM Admin page (/admin/sfm) displays whatever /health returns for `version`. That was hardcoded to "0.1.0" and never bumped, so the page showed 0.1.0 while the service was actually 0.26.0. Point both /health and the FastAPI OpenAPI version at the release-bumped TOOL_VERSION (single source of truth), so they can't drift again. Adds httpx-free regression tests (call health() directly). Note: minimateplus.__version__ is separately stale at 0.1.0 — left as-is here (nothing user-facing reads it; touching the package __init__ risks import order). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 5
|
5f1ee5ba91 |
docs: offset investigation journal; strip NUL corruption from CLAUDE.md
Adds docs/offset_investigation.md — a dated journal of the "offset" hardware
fault, in the style of the codec status docs: findings with provenance, dead
ends kept with the reason they died, and per-unit case files.
Contents:
- base rate 5-6 of 45 units (11-13%) over the DL2 archive, 2018-2026,
confirming rather than overturning the earlier 2-of-21 estimate
- the detector, with the rationale for each term and its known blind spot
(event traces carry real motion, so only trace-dominating offsets show)
- the bimodality result: relaxing the amplitude floor 11x adds no new units
- Instantel's own procedure and thresholds from their FAQs 13-0-21 / 12-0-10:
A/D-mode ">5 counts", the autozero key sequence, and the 2027-2069 X1/X8
acceptance window that explains the ~10% field success rate of a re-zero
- four ruled-out hypotheses, each with the evidence that killed it:
condensation, clipping, the sensor check as a predictor (102 offset events,
zero failures — a grossly offset unit passes its own self-check), and the
calibration-timing correlation (confounded, one unit per time bucket)
- open questions, chiefly whether SUB 0x0E carries the autozero numbers
Cross-referenced from CLAUDE.md and Appendix E of the protocol reference
(whose CRLF line endings are preserved).
Separately: CLAUDE.md had 793 NUL bytes appended after its last line. They
predate this work (present at least as far back as
|
||
|
serversdownandClaude Opus 5
|
4839ddfa0e |
fix(scratch): dedupe the DL2 Sent/ mirror; correct the recovered-file count
The DL2 export keeps a byte-identical `Sent/` copy of its root, so walking it counts every binary twice: 127,035 histogram paths are 63,535 distinct files, and 13,077 waveform paths are 6,577. offset_scan.py now keeps the first occurrence of each basename. Corrects the previous commit's changelog claim of 8 recovered files — it is 4: K440HJCN.3C0H and K557IF1U.8K0H (stride 252), T191HVNP.0S0H (92), T193L0XM.CI0H (612). Still zero regressions. The per-unit breakdown reading exactly 2-2-2-2 should have given the doubling away. The 14,338-exact verification result is unaffected: ASCII exports are not mirrored (14,340 paths, 14,340 distinct names), and the harness enumerates those rather than the binaries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
14e997b20c |
fix(histogram): partial final block no longer discards the correct stride
detect_multi_interval_stride() confirmed a candidate stride on a third block header whenever the body was long enough to contain one. But a body can exceed two strides and still hold only two real blocks: a partial final block leaves trailing padding. BE18193 T193L0XM.CI0H — 51 intervals at 2 s, i.e. one full 30-interval block plus a 21-interval remainder in a 2787-byte body — had every decisive check pass at stride 612 (header at 0, header at 612, block counter 256 -> 257) and was then rejected for the absent third header at 1224. It decoded to nothing. A missing third header now means end-of-stream rather than disqualification. The block-counter check is untouched — that is the test that prevents the false positives which once handed 9,082 standard-block files to the multi-interval walker. Found by running the full DL2 archive against its preserved Blastware ASCII exports (14,340 paired files, 11x the previous ground-truth corpus). Measured over 127,035 archive histogram binaries: recovered 8 files (strides 92, 252, 612; BE18193, BE18191, BE9557, BE9440) regressed 0 files Full-corpus verification: 14,337 -> 14,338 exact of 14,338 decodable pairs (the 2 excluded are series-4 IDF, a different codec). Also adds scratch/verify_against_ascii.py (per-sample decoder verification against BW exports, with a saturation carve-out — BW clamps clipped events to the range max while the decoder reports true counts) and scratch/offset_scan.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 4.8
|
75ac610c61 |
fix(twins): interval-based histogram/waveform matching in find_twins (#102 sub-task 2)
A real trigger is recorded twice — as a triggered waveform (stamped at the trigger instant) and inside the scheduled histogram whose interval contains it (stamped at the 7am/7pm interval start). The two twins routinely differ by HOURS, so the old ±5-minute window in find_twins silently missed them — which broke review propagation (flagging one twin left its twin unflagged). Twins are now matched by: same serial + identical peak_vector_sum + OPPOSITE record type + the waveform's timestamp falling within the histogram's interval (bounded by the next same-serial histogram). Matching keys off record timestamps (not call-in/received times, which drift with field connectivity). window_seconds is retained but ignored. Rewrote test_find_twins + test_twin_propagation for the new contract (incl. the 75-min-apart UM12947 case, cross-type exclusion, containing-interval selection, open-ended latest interval). Full suite: 264 passed; the 16 failures are pre-existing (missing gitignored fixtures + a v0.26.0 codec case), unchanged from baseline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YDXjZCr4RqT2U3QvMDhgzf |
||
|
serversdownandClaude Opus 5
|
dedf1f02c9 |
fix(release): bump TOOL_VERSION to 0.26.0 — sidecar staleness was inert
TOOL_VERSION had been frozen at 0.21.1 for four releases despite its own
comment saying "Bump this constant and CHANGELOG.md together at release
time". It is not cosmetic: backfill_sidecars.py decides whether to
regenerate with
ver_ok = sidecar.source.tool_version >= event_file_io.TOOL_VERSION
so with the constant stuck at 0.21.1 and every sidecar stamped 0.21.1,
a backfill WITHOUT --force skipped the entire store. That is precisely
the failure the check exists to prevent, and it means every sidecar
regenerated during the 0.26.0 decode work is stamped 0.21.1 while having
been produced by 0.26.0 code.
Verified: a non-force dry-run over the snapshot now reports
written=11603 skipped(uptodate)=0, where before it would have skipped
all 11,603. Prod therefore does not need --force to pick up the decode
corrections — the version difference alone is enough.
Note the installed dist metadata reads 0.12.0, older than the constant,
so the best-effort "prefer installed metadata when newer" path correctly
defers to TOOL_VERSION.
Also bumps the README header, which still read v0.22.0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
|
||
|
serversdownandClaude Opus 5
|
b2ef02ebcc |
chore(release): v0.26.0 — series-3 decode correctness
Two body-model rewrites, a systematic scale error affecting every geophone reading the system ever produced, a recovered file format, and two artifact-hygiene bugs where stale files outlived the decodes that made them. - geo full scale is 32000 ADC counts, not 32768 (every reading 2.34% low) - the waveform body is a record chain, not a tag stream - the histogram block is big-endian, with a terminal tail - sub-minute intervals pack several per block (415 files recovered) - three more defects found by a full-corpus sweep, each masking the next - stale .h5 files and stale shape_* columns are now cleared, not left All 11,603 series-3 binaries in the production snapshot pass every check. Ground truth: 1,211/1,211 histograms exact per-interval, 75/75 waveform sample counts exact, multi-interval fixture exact on all 45,680 values. Also corrects a changelog note that went stale within the same day: the "3 of 75 events still truncate" item was resolved by the record-chain rewrite, and the remaining open items are now listed explicitly. CLAUDE.md gains a "Where things stand" block at the top — the header had been reading v0.21.0, four releases behind, which is the first thing you see when picking the project back up. Tests: 259 passed; the 16 failures are pre-existing (gitignored fixtures) and unchanged from baseline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
a3b69a62a6 |
fix(histogram): three defects found by a full series-3 sweep — 11603/11603 clean
Swept every series-3 binary with the live decoder against five independent checks: decode exceptions, zero samples, unequal geo channel lengths, peaks above range full scale, decoded peak vs device-reported PPV, and waveform length vs declared record time. 1. block[22] is NOT a constant and must not be tested. Documented as always 0x00, it carries data on loud blocks, and rejecting those threw away the interval holding the event peak. BE18350/T350L7HR.NL0H block 92 has block[22]=0x26 and a Tran peak of 0x0563 = 1379 counts = 6.895 in/s — exactly the device-reported PPV — while the file decoded to 0.015 in/s. block[0]==0, block[4]==0x0A and the 4-byte tail are six bytes of constraint, which is what keeps trailer content out. 2. Block-model dispatch now goes on signature strength rather than on whichever decoder returns first. A multi-interval body also yields scattered standard-tail blocks by coincidence, so "first non-empty" handed 193 BE18193 files to the standard walker and produced peaks of 149 in/s against a 10 in/s full scale. 3. Multi-interval stride detection requires the block counter to increment by exactly 1. Without it the detector false-positives on ordinary standard-block bodies: they carry a header every 32 bytes, and 192 = 12 + 20*9 and 512 = 12 + 20*25 are both multiples of 32, so a stride "fits" while skipping 6 or 16 real blocks. That misrouted 9,082 files. Partial-block garbage is trimmed within the final block only, stopping at the first slot with a non-zero tail word or a geo peak above full scale (2000 counts in 16-count units). Trimming purely from the end left garbage stranded behind a slot that happened to have a zero tail word; trimming on the tail word alone truncated four BE9440 files by up to 2,800 intervals. Result: 11,603 / 11,603 series-3 binaries clean on every check. Ground truth unchanged: 1211/1211 histograms exact per-interval, 75/75 waveform sample counts exact (73/75 fully exact, the 2 differ by 1 LSB on rail samples), and the multi-interval fixture still matches its BW ASCII export on all 45,680 values. Tests: 259 passed, failure list unchanged from baseline. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
306104354b |
feat(histogram): decode multi-interval blocks — recovers 415 files
Sub-minute histogram intervals are packed several to a block so that
every block still covers exactly one minute of data:
interval intervals/block stride
1 minute 1 32 <- the standard big-endian block
15 s 4 92
2 s 30 612
stride = 12 + n * 20
Block = [00][segment][ctr uint16 LE][0a][00], then n x 20-byte records of
8 x uint16 LITTLE-endian values (T_peak, T_halfp, V_peak, V_halfp,
L_peak, L_halfp, M_peak, M_halfp) plus a 2-word tail whose first word is
0000 on every real interval, then a 6-byte block trailer.
The standard 32-byte block is BIG-endian; this variant is LITTLE-endian.
The tail-word check matters: a session ending mid-block leaves buffer
garbage in the remaining interval slots, which decoded as peaks
thousands of times the real value. Stride detection also requires at
least 2 records, since a 1-record block would have stride 32 and
collide with the standard block.
Recovers 415 files that decoded to nothing: 216 on BE18193 (2 s
intervals) and 199 on BE9440 (15 s). Before decoding to nothing they
were being accepted by the WAVEFORM codec, which returned garbage
peaking up to 400x the device-reported PPV.
Ground truth BE9440/K440L3AQ.T70H (5,710 intervals) matches its
Blastware ASCII export exactly: 17,130/17,130 geo peaks, 22,840/22,840
frequencies, 5,710/5,710 mic dB(L). Across all 455 affected files,
1,354/1,365 channel peaks (99.2%) match the device-reported PPV; the 11
that don't are under-reads on BE9440 where the walk stops early.
Fixture (binary + ASCII) saved under tests/fixtures/, which is
gitignored per repo practice — the ground-truth test skips when absent.
Tests: 258 passed, failure list unchanged from baseline.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
|
||
|
serversdownandClaude Opus 5
|
4c58a532de |
fix(backfill): remove stale .h5 when nothing decodes; log the 415-file histogram variant
backfill_sidecars.py skipped the .h5 write when a file produced no samples, with the stated intent of not replacing it with an empty placeholder. That silently preserved output from a superseded decoder. After the record-chain fix, 415 histogram files stopped decoding (216 on BE18193, 199 on BE9440) but kept .h5 files whose peaks ran up to 400x the device-reported PPV. Those were feeding charts and the false-trigger detector with nothing marking them. The .h5 is now removed in that case and the run reports stale_h5_removed. Store-wide effect, series-3, decoded peak vs device-reported PPV: waveform 1307/1307 (100%), mean abs ratio error 0.00000 histogram 4434/4435 (100%) Both were 99% with a tail of 18 and 25 wrong files respectively. The 415 files are a genuine unmapped format variant, not a regression: their bodies open `00 00 00 01 0a 00` (valid block header, marker 0a at [4], block_ctr 256) but block[28:32] matches neither known tail, and no stride from 8 to 64 bytes places a marker at [4] consistently. Bodies are very large (one is 360,573 bytes). They were previously being decoded by the WAVEFORM codec, which accepted them and returned garbage - so the gap pre-dates today's work; the fix only exposed it. Logged as an open question in the protocol reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
9bb95003e9 |
fix(codec): the waveform body is a record chain, not a tag stream
Supersedes the segment-header model entirely, including the fixes made earlier today. Found via multi-agent structural analysis of the 25 files that stalled the walker, then verified independently. Records are self-delimiting: off+2 is a uint16 BE length, next_record = off + 2 + len, and the chain ends on a record whose chan_id is 0x06. off+8 carries a 3-valued mode enum: 02 00 14-byte header, 2 anchors, then CUMULATIVE delta blocks 01 00 10-byte header, no anchors, blocks are ABSOLUTE values 00 03 10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute `40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header. Reading it as a 2*NN + 16 header is what made walks drift — the "variable-prefix segment descriptors" reported earlier today were not a format feature, just walker drift of exactly 4 - (old_stop - true_record_start), on all 25 affected files. Measured on the production snapshot: all four channels equal length 156/1388 -> 1388/1388 ASCII sample-count exact 72/75 -> 75/75 ASCII fully exact 70/75 -> 73/75 device PPV waveform (live) 1288/1306 -> 1306/1306 (mean err 0.00000) device PPV histogram (live) 4434/4459 -> 4458/4459 Also eliminates the walker-over-read class: 24 of those 35 files were histograms that read_blastware_file fed to the waveform codec first; the old walker accepted them and returned garbage (one yielded 98,923 "intervals"), while the record-chain decoder returns None so they fall through to histogram_codec. 00 03 records are DECODED, not skipped. Skipping them silently shifts the time base of everything after them on that channel — BE9558/ K558LOF2.820W had MicL displaced by exactly 512 samples with nothing marking the gap. Footer detection now prefers the 0e 08 candidate whose body yields a chain terminating on 0x06; the signature can occur inside a sample stream. Blast radius 1 file of 1388. The superseded model survives as decode_waveform_legacy, pinned by micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes candidates and keeps whichever yields the most samples, so the new decoder returning None where the old returned garbage changes that heuristic's winner. Deferred until that search uses the record chain. Tests: 253 passed (+11), failure list unchanged from baseline. The 9 tests pinning the superseded model are retargeted at decode_waveform_legacy, which still implements it. NOTE: stored .h5 files need regenerating — nearly all get longer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
260bf0bc67 |
fix(backfill): clear stale shape_* when the .h5 can no longer yield a shape
backfill_event_shape.py skipped rows whose .h5 produced no shape and left the previously stored value in place. A stale shape outlives the decode it came from and silently feeds the false-trigger detector. Found while re-running the backfill after the histogram codec fix: 493 rows in the prod snapshot were carrying shape metrics that no longer matched their .h5 — e.g. BE17353/S353LDOK.XZ0H held crest_factor from a 223-sample decode while its .h5 holds a single interval. These predate today's work (present in the pre-32000 snapshot), so this is pre-existing behaviour rather than fallout from the codec fixes. Now NULLs shape_crest_factor / shape_near_peak_count / shape_sample_count / shape_axis in that case and reports a `cleared_stale` count. Verified on the snapshot: 493 cleared, 0 stale rows remaining, 11570 rows matching their .h5 exactly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
ef1e99b0a0 |
fix(histogram): block is big-endian + terminal block tail — 1/1196 to 1211/1211
Two errors in the series-3 histogram block model, both found by diffing against the per-interval data table in the preserved Blastware ASCII exports (1211 files in the prod snapshot — far stronger ground truth than the header PPV used previously). 1. The block is uniformly BIG-ENDIAN. Peaks and half-periods are uint16 BE (T_peak [5:7], T_halfperiod [7:9], V_peak [9:11], V_halfperiod [11:13], L_peak [13:15], L_halfperiod [15:17], M_peak [17:19], M_halfperiod [19:21]); only block_ctr [2:4] is little-endian. The old uint8-peak model silently CLIPPED any peak above 1.275 in/s: the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590. The byte documented as a per-channel "annotation" was never an annotation — it is the half-period's high byte, which is exactly why it was non-zero on the sub-Hz intervals BW renders as "<1.0". The marker is block[4] alone. Testing [4:6] as a uint16 LE marker forced block[5] == 0, which is what capped the peak at one byte. 2. The final block of each stream carries tail 9c 06 00 42 instead of 1e 0a 00 00, and holds arbitrary bytes at [21:23]. Rejecting it dropped the last interval of nearly every histogram — frequently the interval holding the event peak, so the file's PPV read low. Verified end to end through the production path: 1211/1211 histograms decode exactly (interval count + every per-interval peak), plus 842,442 per-interval frequency comparisons with zero mismatches. Previously 1 of 1196 files was fully correct. decode_histogram_body_full records expose `is_terminal` in place of the removed `annotations` tuple. +6 tests. No regressions: full-suite failure list unchanged from baseline. NOTE: stored histogram .h5 files need regenerating to pick this up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
e449ac04af |
docs: sharpen the series-3 histogram open item — dropped intervals, not wrong values
Re-measured properly. The first pass compared h5 max against the ASCII
header PPV and reported "26% of channels miss the peak". The histogram
ASCII actually carries a full per-interval data table (Tran/Vert/Long
peak + freq + PVS per interval), which is real ground truth, so the
comparison should have been per-interval from the start.
Per-interval result, n=1196 series-3 histograms:
- decoded VALUES are right: 1031/1196 (86%) match within 1 LSB across
the overlapping prefix
- the interval COUNT is short in 1195 of 1196 files: median 1 missing,
1088 short by 1-2, 65 by 3-10, 39 by 11-100, 3 by >100 (max 205)
- decoded max falls below the device PPV in 169/1196 files (14%), not
26% — that happens when a dropped interval held the peak
So it is a termination bug in histogram_codec.decode_histogram_body,
the same family as the waveform-walker truncation fixed earlier today,
rather than mis-decoded interval values. Series-3 only; there is no
preserved series-4 ASCII in the snapshot to compare against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
|
||
|
serversdownandClaude Opus 5
|
4f8224a751 |
docs(appendix-e): offset fault is geophone-side — operator swap test + MicL evidence
Operator report: attaching a different geophone to an affected unit makes the offset go away. That rules out the unit's analog front-end and any stored per-channel zero constant (a constant lives in the unit and would survive a sensor swap). The stored data agrees — MicL, a separate transducer on its own cable, shows no offset during either episode (|mean|/peak 0.17 and 0.02) while the geo channels on the same unit at the same moment are pinned. Two distinct sensor-side patterns recorded: BE18438 Vert 0.97, Tran 0.16, Long 0.18 -> one conductor pair BE9558 Long 0.99, Tran 0.90, Vert 0.81 -> shared return / ground Candidate mechanisms narrowed to three, since a geophone coil is passive and cannot generate sustained DC: galvanic corrosion at a connector or splice (matches the ~46 mV referred to the ADC input), a leakage path to shield, or changed coil DC resistance interacting with the amplifier's input bias current. Also records the confound: swapping a sensor requires a monitoring restart, and these units run Sensor Check "Before monitoring", so the restart re-zeros too. The swap does not cleanly separate "new sensor" from "the restart re-zeroed it". Controls and the single best measurement (open-circuit DC across the suspect connector) documented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |