3f58402085054517fe49d959356f04779f868dc8
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  serversdownandClaude Opus 5
|
c8d972f685 |
docs(series4): the setup-write path, observed end to end
Thor pushed a setup named TEST1.mmb to UM12947 while seismo_lab's TCP bridge
recorded both directions. We still have not originated a write frame -- the
wire format is now known, our encoder is not written.
Topology worth reusing: socat shares /dev/ttyACM0 on TCP from mint-mac,
seismo_lab relays Thor to it. Thor is pointed at 127.0.0.1 as if the unit were
a field modem. No modem, no SIM, production Thor box untouched.
The sequence is Series III's, plus one command:
Thor: 5B | 41 | 08 | 2E | 1A | DA | 68->73 | 82->83 | 71->72
unit: A4 | BE | F7 | D1 | E5 | 25 | 97 8C | 7D 7C | 8E 8D
All 12 device responses checksum-validate and every write is acked. Every
write response SUB matches the Series III table exactly.
New:
* SUB 0xDA names the target .MMB file -- 256 bytes, filename null-padded,
nothing else. This is why no generic file-transfer command exists: Thor
names the file, then writes the ordinary config block into it.
* SUB 0x41 reads the active setup's filename; SUB 0x2E reads trigger config.
* Reads are single-step -- Thor asks offset=0xFFFF and skips the probe.
* 0x71 writes the whole 2090-byte block in ONE frame, not Series III's three
chunks. 0x69/0x74 are absent.
Write-frame destuffing is `10 XX` -> `XX` uniformly, including `10 03`. Chosen
by checksum, not assumption: of four candidate rules, only this one makes all
four data-carrying write frames validate. 0x71's data holds 4 literal 0x03
bytes escaped as `10 03`, so escaping is mandatory for any writer.
The write body IS the read body -- 0x71 and the 0xE5 response align at a fixed
11-byte shift with 1902/2090 bytes equal (91.0%). Setups are read-modify-write.
The 12 differing regions are fully mapped: setup name, four 64-byte
[label:22][value:42] note entries, sensor location, and the three geo trigger
levels (0.3 -> 0.5 in/s) on a 48-byte channel stride.
Independent confirmation of the geo LSB: each channel block carries float32BE
3.10308 at label+24. 3.10308/10000 = 0.000310308 = _GEO_LSB_IPS to 8 figures,
and 10.0/3.10308*10000 = 32226.046 = the 32226.05 full scale. That value was
derived statistically from 991,415 rounding constraints in v0.30.0; the unit
reports it directly. It is exactly half Series III's 6.206053, so the ADC runs
10,000 counts per volt. Do NOT retune _GEO_LSB_IPS -- this corroborates it.
The `offset` field is NOT a single length formula: two frames are len, two are
len+2, and Series III's data[1]+2 reproduces neither. Recorded as observed
constants the device accepted; pinning the rule needs a capture with
differently-sized payloads. This doc has been wrong once by inferring a length
field -- not inferring this one.
Also adds scratch/mm_frame_parse.py, because S3FrameParser cannot see Micromate
responses at all (it scans for DLE+STX; Micromate responses start at a bare
STX). That is why the first pass at this capture looked like 12 unanswered
requests. 24/24 frames parse with 0 bad checksums.
Stale claims corrected: the "write half is not yet attempted" note, the
"empty unit" limitation (5 events since 2026-09-23), and the unsafe-until-agreed
list, which now distinguishes observed from exercised.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL
|
||
|
serversdownandClaude Opus 5
|
726c2ce1b5 |
fix(series4): Thor/Micromate decoder is now per-sample exact
Verified against Thor's own CSV exports, which carry a per-sample four-column block beside every binary (CSV/<name>.IDFW.csv). Those 1,012 paired files were in the corpus all along; the decoder had been pinned to a superseded walker on the stated grounds that "Thor has no ASCII ground truth in the corpus and its geo scaling is separately suspect". Both premises were false. IDFW per-sample exact 39.1% -> 100.000% (1,057,536/1,057,536) IDFW files fully exact 0/153 -> 153/153 IDFW PPV median error -3.32% -> -0.002% IDFH within 2% of Thor PPV 51.1% -> 100.0% (858/858) prod IDFW, 8 units -3.3% -> -0.001% Four independent root causes: - Geo LSB was 0.0003, the 4-dp *display rounding* of the real 0.000310308 mistaken for the LSB, so every series-4 geophone sample read 3.3% low. Pinned to +-6e-11 by intersecting 991,415 rounding constraints; corroborated by the +-full-scale seed (+-32226) left in unwritten IDFH slots. IDFH had a separate, also wrong, 10.0/32768. - IDFH histograms were capped at 250 intervals: the segment validator required the interval counter's high byte to be zero, but the counter is a uint16 cumulative index, so every segment past interval 255 was rejected. Runs over ~4 hours lost their tail, often the peak. 540/858 corpus files affected. - Record mode 00 00 (raw int16, 10-byte header) was unhandled and fell through the dispatch, silently dropping each channel's first 512 samples -- the long-standing "loud events truncate" symptom. MODE_ABSOLUTE is now also accepted as a segment-0 preamble. - The body-offset search matched 00 02 00 *inside* record headers, selecting a candidate part-way down the chain and decoding a rotation-shifted body. It now anchors on record headers and takes the chain head (6 ms/file). Also fixes the separately tracked "UM-series decodes ~1000x low" bug. Series-3 re-verified unchanged at 14,338/14,338 exact after the shared waveform_codec change. Known open: 41/575 prod IDFW files (7%, mostly UM12947/UM20147) decode with unequal channel lengths and also fail metadata extraction -- a different header variant with no Thor export in the store. NOTE: this is a codec change; the Thor store owes a regeneration via scripts/backfill_thor_events.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ru8Lg9HkkYvX9VWWo65SmL |
||
|
serversdownandClaude Opus 5
|
f600fee965 |
feat(offset): the non-motion test, and BE12599 diagnosed as a connector
Brian noticed BE12599's 2026-08-09 event reports no ZC frequency because the trace never crosses zero. That is the best detector in this investigation. A geophone has no DC response, so its output must integrate to ~zero over a record. |mean|/peak is therefore ~0 for real motion and ~1 for anything electrical. Across 12,068 channel-events with peak >= 0.05 in/s the statistic is bimodal with a 1.09% dead zone, and at mp >= 0.8 it returns exactly the five confirmed units -- from physics rather than a tuned threshold. Two detectors on different principles agreeing is the strongest corroboration the list has had. It also settles BE11007 as NOT an offset: mp 0.75-0.89 but frac_neg 0.99 at peaks of 7.4-9.4 in/s, i.e. a one-sided near-full-scale blast. Journal 8e diagnoses BE12599 specifically. Its August waveforms are unipolar impulses with an RC tail (26 ms -> 118 ms -> never recovers over 14 days), and the fault MOVES between Long and Tran while the sensor self-check passes on every event. A failing element cannot hop channels; a connector can -- which also explains why the swing test never fails and why an autozero rarely helps. Corrects 8c's claim that the spread gate is blind to onsets: of 87 BE18438|Vert events it rejected one, the transitional record. Narrower than stated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
9982938b0b |
fix(offset): read the real serial from the file body, not "BE" + the number
The BW filename encodes only the serial NUMBER — `<letter><3 digits>` where
letter = chr(ord('B') + serial // 1000), so `L895…` decodes to 10895. The
two-letter family prefix is not in the filename at all, and every offset
scanner synthesized it as f"BE{num}".
Four of the 43 archive units are BA, not BE. Their binaries say so plainly:
BA9229, BA10060, BA10895, BA15957. Brian caught BA10895 by recognising that
no such unit as BE10895 exists.
serial_of() now reads the serial string out of the file body and falls back
to the old synthesis only when no matching string is found. No analysis
changes: grouping was by the numeric part, which was always correct, and no
unit number maps to more than one serial (checked across all 43).
The same assumption is live in two production sites and is NOT touched here,
because fixing ingest renames rows a running store and Terra-View already
reads them:
- sfm/waveform_store.py:870 `return f"BE{serial_num}"` on import
- minimateplus/client.py:2538 `raw_data.find(b"BE")` in the monitor-log
partial-record decode, which yields serial=None on a BA unit
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
|
||
|
serversdownandClaude Opus 5
|
1daf693b32 |
feat(offset): scan the histogram corpus — the other 90% of the archive
offset_scan3.py covers only waveforms (6,577 unique binaries). The archive also holds 63,535 unique histograms, which the pre-trigger method cannot touch: a histogram carries no samples, only a per-interval per-channel peak. scratch/offset_hist_scan.py scans them — 63,505/63,535 decoded (99.95%), 43 units, 77.9M intervals. It emits every candidate floor statistic per (file, channel) rather than deciding anything, so thresholds get calibrated against the waveform ground truth instead of guessed. Journal §8b records the outcome. What survives is a site-quiet-gated cross-channel differential that independently confirms BE18438|Vert and BE9558|Tran+Long with a clean 2.5x separation gap and 0.037% day-level false alarm, threshold-insensitive across a 2.3x span — the first operating point in this investigation to pass that test cleanly. What it does not do, recorded just as plainly: it finds 2 of the 5 confirmed units, not 5. DC leakage into the interval peak is bimodal (0.9 on BE18438, 0.02 on BE12599), so a negative histogram result is not evidence of health. Per-channel attribution is not established (channel-scramble p = 0.769) and timing resolves to ~a month, not a day. Two dead ends buried for good: the absolute floor is retired (66% of its discrimination is a day/site confound), and zero-fraction is structurally impossible — the device clamps every interval peak at >= 1 A/D count. Two findings independent of the histograms: - offset_scan3's spread<=0.02 gate discards 18.8% of rows with |pre|>=0.025, concentrated on 41 unit-channels currently labelled clean; 4 would be sustained positives without it. The fleet label is three-state, not two. - The waveform corpus observes ~7% of the days a unit was deployed. BE10895 is reclassified from transient to a genuine Vert fault of a different subtype: 49.4% single-axis-dominant events, the highest in the fleet, all on Vert. The other six marginal units are clean. Not done: the 11 thin-coverage units were not screened, and no completeness audit was run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
ad84a04404 |
feat(offset): detector v3 — pre-trigger floor with a constant-floor test
Brian's method, and better than v2's whole-record median: the pre-trigger
window is definitionally quiet (the buffer captured before the trigger fired),
whereas a median is merely robust to the event. Requiring the floor to hold
across pre-trigger / middle / end rejects transients that a median cannot.
per channel: pre/mid/end medians, spread = max - min
offset when |pre| >= floor AND spread <= 0.02 in/s
real fault >= 3 consecutive flagged events on that channel
The empirical noise floor justifies the threshold and validates the decoder:
across 19,244 non-flagged channel-events the pre-trigger floor is 62.7% exactly
0.000, 94.5% within +/-1 quantisation unit, median +0.0000, mean -0.0008. There
is no systematic zero-point bias — an independent confirmation of the
32000-count geo scale.
The result is threshold-insensitive across a 2x range (0.020 to 0.040 in/s),
which is what separates a real signal from a tuned one:
FINAL: 5 of 45 units (11%) — BE9558, BE11529, BE12599, BE13117, BE18438
BE11007 and BE10895 drop out; the spread test identifies them as transients
rather than pedestals. v1's 11% headline was right by luck — it included
BE11007 and named the wrong channel on most units.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
|
||
|
serversdownandClaude Opus 5
|
1fdc665675 |
fix(offset): retract the v1 detector — per-channel median, not dominant-axis mean
Brian challenged the v1 finding that offsets "come and go", against field experience that a unit which develops one stays broken until the geophone is replaced. He was right; v1 had two flaws, both of which manufactured false recoveries: 1. It scored only the axis with the largest peak, so a real event on one axis hid a persistent pedestal on another. BE12599 on 2026-08-21 read "clean" because Long had a 1.065 in/s event, while Tran sat at +0.4732 in/s and was never examined. 2. It used the mean, which a real transient perturbs. The median is the resting baseline and a blast does not move it. Same event, Long channel: mean +0.0783 vs median -0.0050. offset_scan2.py flags a CHANNEL when |median| >= 0.025 in/s (5 A/D counts, Instantel's own criterion) and treats >=3 consecutive flagged events as the real signal. No m/p ratio guard is needed — that existed only to compensate for the mean. Corrected results: units with any flagged event 6 -> 19 of 45 units with a sustained pedestal 8 of 45 (18%) runs >=3 consecutive 29; 1-2 event runs (noise) 69 Also corrected: the affected channel is most often Vert, not Tran (v1 named whichever axis had the largest peak, so it was frequently wrong). BE10895 and BE18003 were invisible to v1. BE12599's fault began 2026-08-14, not 08-17. The decode itself was never in question and is confirmed against Blastware's own ASCII export: on K558LJN3.BK0W, BW shows Tran parked at +0.265..+0.375 in/s for the entire record while Vert and Long sit at ~0.005 — Instantel's "parallel lines above or below the zero line". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
4839ddfa0e |
fix(scratch): dedupe the DL2 Sent/ mirror; correct the recovered-file count
The DL2 export keeps a byte-identical `Sent/` copy of its root, so walking it counts every binary twice: 127,035 histogram paths are 63,535 distinct files, and 13,077 waveform paths are 6,577. offset_scan.py now keeps the first occurrence of each basename. Corrects the previous commit's changelog claim of 8 recovered files — it is 4: K440HJCN.3C0H and K557IF1U.8K0H (stride 252), T191HVNP.0S0H (92), T193L0XM.CI0H (612). Still zero regressions. The per-unit breakdown reading exactly 2-2-2-2 should have given the doubling away. The 14,338-exact verification result is unaffected: ASCII exports are not mirrored (14,340 paths, 14,340 distinct names), and the harness enumerates those rather than the binaries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
14e997b20c |
fix(histogram): partial final block no longer discards the correct stride
detect_multi_interval_stride() confirmed a candidate stride on a third block header whenever the body was long enough to contain one. But a body can exceed two strides and still hold only two real blocks: a partial final block leaves trailing padding. BE18193 T193L0XM.CI0H — 51 intervals at 2 s, i.e. one full 30-interval block plus a 21-interval remainder in a 2787-byte body — had every decisive check pass at stride 612 (header at 0, header at 612, block counter 256 -> 257) and was then rejected for the absent third header at 1224. It decoded to nothing. A missing third header now means end-of-stream rather than disqualification. The block-counter check is untouched — that is the test that prevents the false positives which once handed 9,082 standard-block files to the multi-interval walker. Found by running the full DL2 archive against its preserved Blastware ASCII exports (14,340 paired files, 11x the previous ground-truth corpus). Measured over 127,035 archive histogram binaries: recovered 8 files (strides 92, 252, 612; BE18193, BE18191, BE9557, BE9440) regressed 0 files Full-corpus verification: 14,337 -> 14,338 exact of 14,338 decodable pairs (the 2 excluded are series-4 IDF, a different codec). Also adds scratch/verify_against_ascii.py (per-sample decoder verification against BW exports, with a saturation carve-out — BW clamps clipped events to the range max while the decoder reports true counts) and scratch/offset_scan.py. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 5
|
0f6c9d930f |
data: offset-candidate event list from the 2026-08-25 survey
274 series-3 waveform events across 6 episodes on 2 units (BE9558, BE18438) whose dominant geo axis sits pinned at a DC offset above that unit's own geo trigger level — the "offset" hardware fault that makes a unit retrigger continuously and flood the ACH queue. Detection rule: dominant-axis |mean|/peak > 0.7 AND |mean| >= 0.9 x the unit's geo trigger level. Bare |mean|/peak is useless on quiet events — a trace at the 0.010 in/s noise floor clears any ratio threshold. Not a decode artifact: these reproduce exactly in Blastware's own ASCII export. Kept as the starting point for the archive-wide analysis. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
Claude
|
07675626dc |
codec-re: channel rotation CONFIRMED — full multi-channel decoder works
The segment-channel scoring analyzer (from scratch/next_experiment_skeleton.py) ran and immediately confirmed the rotation hypothesis: SP0 seg 0: best fit Vert 508/508 ✓ SP0 seg 1: best fit Long 508/508 ✓ SP0 seg 3: best fit Tran 508/508 ✓ (Tran continuation) SP0 seg 5: best fit Long 508/508 ✓ SP0 seg 9: best fit Long 508/508 ✓ V70 seg 0: best fit Vert 508/508 ✓ V70 seg 1: best fit Long 508/508 ✓ Channels rotate Tran → Vert → Long → MicL per 40 02 segment header. Also discovered the segment header has DOUBLE duty: bytes [14:18] anchor the NEW segment's channel (2 samples as int16 BE in 16-count units), AND bytes [0:4] extend the PREVIOUS channel by 2 more samples (2 deltas as int16 BE). This is the same "2 anchors + delta stream" structure as the body preamble for Tran. decode_waveform_v2 now returns full per-channel sample dicts. Byte-exact verified ranges: V70: Tran 512, Vert 512, Long 512 (all first segments) JQ0: Tran 512, Vert 258 SP0: Long 1536 (all 3 L segments) Still open: the 30 NN block format (high-amplitude packed deltas) — appears mid-segment when single-byte deltas can't carry the magnitude. 6 new tests bring the count to 46. All passing. |
||
|
Claude
|
ae0e17b5dc |
codec-re: handoff polish — readmes, skeleton, remove decode-re/ duplicate
Three things to make pickup smoother: 1. analysis/README.md (NEW): catalogues the ~25 scratch scripts. Categorizes them as "still useful" / "superseded — keep for archaeology" / "pure exploration". Tells a fresh engineer which files to read first and which to ignore. 2. scratch/next_experiment_skeleton.py (NEW): stub + spec for the segment-channel scoring analyzer. Includes the fixture loader, block walker, and decode-segment-as-channel helper — just enough scaffolding that the next pass starts from "fill in score_segment_against_all_channels()" rather than from scratch. Already runs and confirms 13 segments per 3-sec event with sample starts going to 6590 (way past the 3328 actual samples) — strong evidence that not all segments carry Tran. 3. Removed decode-re/ duplicate. It was a mirror of tests/fixtures/. Analysis scripts that hardcoded decode-re/ paths updated to point at tests/fixtures/. CLAUDE.md note updated: future event uploads go directly into a dated subdirectory under tests/fixtures/. All 40 tests still pass. Skeleton runs. |