Commit Graph
56 Commits
Author SHA1 Message Date
serversdownandClaude Opus 5 4c58a532de fix(backfill): remove stale .h5 when nothing decodes; log the 415-file histogram variant
backfill_sidecars.py skipped the .h5 write when a file produced no
samples, with the stated intent of not replacing it with an empty
placeholder.  That silently preserved output from a superseded decoder.

After the record-chain fix, 415 histogram files stopped decoding (216 on
BE18193, 199 on BE9440) but kept .h5 files whose peaks ran up to 400x
the device-reported PPV.  Those were feeding charts and the
false-trigger detector with nothing marking them.  The .h5 is now
removed in that case and the run reports stale_h5_removed.

Store-wide effect, series-3, decoded peak vs device-reported PPV:
  waveform   1307/1307 (100%), mean abs ratio error 0.00000
  histogram  4434/4435 (100%)
Both were 99% with a tail of 18 and 25 wrong files respectively.

The 415 files are a genuine unmapped format variant, not a regression:
their bodies open `00 00 00 01 0a 00` (valid block header, marker 0a at
[4], block_ctr 256) but block[28:32] matches neither known tail, and no
stride from 8 to 64 bytes places a marker at [4] consistently.  Bodies
are very large (one is 360,573 bytes).  They were previously being
decoded by the WAVEFORM codec, which accepted them and returned garbage
- so the gap pre-dates today's work; the fix only exposed it.  Logged as
an open question in the protocol reference.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
2026-08-25 22:23:10 +00:00
serversdownandClaude Opus 5 9bb95003e9 fix(codec): the waveform body is a record chain, not a tag stream
Supersedes the segment-header model entirely, including the fixes made
earlier today.  Found via multi-agent structural analysis of the 25 files
that stalled the walker, then verified independently.

Records are self-delimiting: off+2 is a uint16 BE length, next_record =
off + 2 + len, and the chain ends on a record whose chan_id is 0x06.
off+8 carries a 3-valued mode enum:
  02 00  14-byte header, 2 anchors, then CUMULATIVE delta blocks
  01 00  10-byte header, no anchors, blocks are ABSOLUTE values
  00 03  10-byte header, NO TAGS AT ALL - raw 12-bit packed absolute

`40 NN` is an ordinary int16 BE data block (2*NN + 2), never a header.
Reading it as a 2*NN + 16 header is what made walks drift — the
"variable-prefix segment descriptors" reported earlier today were not a
format feature, just walker drift of exactly
4 - (old_stop - true_record_start), on all 25 affected files.

Measured on the production snapshot:
  all four channels equal length   156/1388 -> 1388/1388
  ASCII sample-count exact           72/75  ->   75/75
  ASCII fully exact                  70/75  ->   73/75
  device PPV waveform (live)       1288/1306 -> 1306/1306  (mean err 0.00000)
  device PPV histogram (live)      4434/4459 -> 4458/4459

Also eliminates the walker-over-read class: 24 of those 35 files were
histograms that read_blastware_file fed to the waveform codec first; the
old walker accepted them and returned garbage (one yielded 98,923
"intervals"), while the record-chain decoder returns None so they fall
through to histogram_codec.

00 03 records are DECODED, not skipped.  Skipping them silently shifts
the time base of everything after them on that channel — BE9558/
K558LOF2.820W had MicL displaced by exactly 512 samples with nothing
marking the gap.

Footer detection now prefers the 0e 08 candidate whose body yields a
chain terminating on 0x06; the signature can occur inside a sample
stream.  Blast radius 1 file of 1388.

The superseded model survives as decode_waveform_legacy, pinned by
micromate/idf_file.py: its Thor IDFW body-offset search trial-decodes
candidates and keeps whichever yields the most samples, so the new
decoder returning None where the old returned garbage changes that
heuristic's winner.  Deferred until that search uses the record chain.

Tests: 253 passed (+11), failure list unchanged from baseline.  The 9
tests pinning the superseded model are retargeted at
decode_waveform_legacy, which still implements it.

NOTE: stored .h5 files need regenerating — nearly all get longer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
2026-08-25 22:13:29 +00:00
serversdownandClaude Opus 5 ef1e99b0a0 fix(histogram): block is big-endian + terminal block tail — 1/1196 to 1211/1211
Two errors in the series-3 histogram block model, both found by diffing
against the per-interval data table in the preserved Blastware ASCII
exports (1211 files in the prod snapshot — far stronger ground truth
than the header PPV used previously).

1. The block is uniformly BIG-ENDIAN.  Peaks and half-periods are uint16
   BE (T_peak [5:7], T_halfperiod [7:9], V_peak [9:11], V_halfperiod
   [11:13], L_peak [13:15], L_halfperiod [15:17], M_peak [17:19],
   M_halfperiod [19:21]); only block_ctr [2:4] is little-endian.

   The old uint8-peak model silently CLIPPED any peak above 1.275 in/s:
   the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's
   export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590.

   The byte documented as a per-channel "annotation" was never an
   annotation — it is the half-period's high byte, which is exactly why
   it was non-zero on the sub-Hz intervals BW renders as "<1.0".

   The marker is block[4] alone.  Testing [4:6] as a uint16 LE marker
   forced block[5] == 0, which is what capped the peak at one byte.

2. The final block of each stream carries tail 9c 06 00 42 instead of
   1e 0a 00 00, and holds arbitrary bytes at [21:23].  Rejecting it
   dropped the last interval of nearly every histogram — frequently the
   interval holding the event peak, so the file's PPV read low.

Verified end to end through the production path: 1211/1211 histograms
decode exactly (interval count + every per-interval peak), plus 842,442
per-interval frequency comparisons with zero mismatches.  Previously
1 of 1196 files was fully correct.

decode_histogram_body_full records expose `is_terminal` in place of the
removed `annotations` tuple.  +6 tests.  No regressions: full-suite
failure list unchanged from baseline.

NOTE: stored histogram .h5 files need regenerating to pick this up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
2026-08-25 18:59:25 +00:00
serversdownandClaude Opus 5 e449ac04af docs: sharpen the series-3 histogram open item — dropped intervals, not wrong values
Re-measured properly.  The first pass compared h5 max against the ASCII
header PPV and reported "26% of channels miss the peak".  The histogram
ASCII actually carries a full per-interval data table (Tran/Vert/Long
peak + freq + PVS per interval), which is real ground truth, so the
comparison should have been per-interval from the start.

Per-interval result, n=1196 series-3 histograms:
  - decoded VALUES are right: 1031/1196 (86%) match within 1 LSB across
    the overlapping prefix
  - the interval COUNT is short in 1195 of 1196 files: median 1 missing,
    1088 short by 1-2, 65 by 3-10, 39 by 11-100, 3 by >100 (max 205)
  - decoded max falls below the device PPV in 169/1196 files (14%), not
    26% — that happens when a dropped interval held the peak

So it is a termination bug in histogram_codec.decode_histogram_body,
the same family as the waveform-walker truncation fixed earlier today,
rather than mis-decoded interval values.  Series-3 only; there is no
preserved series-4 ASCII in the snapshot to compare against.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
2026-08-25 16:23:51 +00:00
serversdownandClaude Opus 5 4f8224a751 docs(appendix-e): offset fault is geophone-side — operator swap test + MicL evidence
Operator report: attaching a different geophone to an affected unit
makes the offset go away.  That rules out the unit's analog front-end
and any stored per-channel zero constant (a constant lives in the unit
and would survive a sensor swap).

The stored data agrees — MicL, a separate transducer on its own cable,
shows no offset during either episode (|mean|/peak 0.17 and 0.02) while
the geo channels on the same unit at the same moment are pinned.

Two distinct sensor-side patterns recorded:
  BE18438  Vert 0.97, Tran 0.16, Long 0.18  -> one conductor pair
  BE9558   Long 0.99, Tran 0.90, Vert 0.81  -> shared return / ground

Candidate mechanisms narrowed to three, since a geophone coil is passive
and cannot generate sustained DC: galvanic corrosion at a connector or
splice (matches the ~46 mV referred to the ADC input), a leakage path to
shield, or changed coil DC resistance interacting with the amplifier's
input bias current.

Also records the confound: swapping a sensor requires a monitoring
restart, and these units run Sensor Check "Before monitoring", so the
restart re-zeros too.  The swap does not cleanly separate "new sensor"
from "the restart re-zeroed it".  Controls and the single best
measurement (open-circuit DC across the suspect connector) documented.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
2026-08-25 15:06:34 +00:00
serversdownandClaude Opus 5 5d3963b545 docs: record the 2026-08-25 body-codec and geo-scale findings
Brings the protocol reference, CLAUDE.md and the codec RE status doc up
to date with everything confirmed in this pass.

instantel_protocol_reference.md
  - Changelog row for the five findings.
  - S7.6.1: scope table showing the 32000 scale correction applies to
    series-3 waveform, series-3 histogram and series-4 Thor alike, with
    the measured before/after ratios for each.
  - S15: closed "Full channel ID mapping in SUB 5A stream" — resolved by
    the segment-header channel id ([channel][00][00][segment], 0x46=Tran
    0x47=Vert 0x48=Long 0x49=MicL, 1697/1697 verified).  Four new open
    questions: variable-prefix segment descriptors, the histogram codec
    missing peak intervals (26% of channels), UM-series IDF decoding
    ~1000x low, and the Thor per-count LSB residual.
  - NEW Appendix E — Known Device Faults.  Documents the field-observed
    "offset" fault: symptom, why it floods the ACH queue (pedestal
    exceeds the unit's own geo trigger level), the episode table, the
    detection rule that works, what the data rules out (not the
    geophone, not the battery, not environmental, not condensation),
    and the two remaining candidate mechanisms with the test that
    separates them.  Explicitly flags that it is NOT a decode artifact,
    since that mistake has already been made once.

CLAUDE.md
  - Body-codec section: the four framing cases and the channel-id
    finding, with the corpus result.
  - "What's NOT solved": replaced the stale walker-edge-cases bullet
    with the four genuinely open items.

waveform_codec_re_status.md
  - Scale scope table matching the protocol reference.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
2026-08-25 14:25:26 +00:00
serversdownandClaude Opus 5 686ab6e7a6 fix(codec): geo full scale is 32000 counts; 4 walker framing cases; channel-id from header
Two independent bugs, both found by diffing 75 production events against
their preserved Blastware ASCII exports (<store>/<serial>/<file>_ASCII.TXT).

1. Geo full scale was wrong — every geophone reading was 2.34% low.
   The codec emits geo samples in 16-count units with a documented LSB of
   exactly 0.005 in/s, and decoded_to_adc_counts multiplies by 16, so one
   ADC count is 0.005/16 in/s and 10.000 in/s is 10.0/(0.005/16) = 32000
   counts.  sfm/event_hdf5.py and minimateplus/event_file_io.py both
   divided by 32768 (2^15), scaling every sample and derived peak down by
   1 - 32000/32768.  The error scales with amplitude, so it was invisible
   on quiet events and worst on the loud ones that matter for compliance.
   Mic is unaffected (it back-solves its scale from the device peak).

   216 per-channel comparisons: 32768 -> 151/216 exact, worst error 0.238
   in/s on a 10 in/s event; 32000 -> 216/216 exact, worst 0.005 = 1 LSB.

2. walk_body silently truncated channels on four unhandled framing cases.
   An unrecognised tag ends the walk and decode_waveform_v2 returns
   whatever it got, so this surfaced as short channels, never an error:
     - wide-NN RLE `0X NN` (runs longer than 252 samples)
     - `30 NN` with NN > 0x10 (the old cap was arbitrary)
     - variable-width `40 NN` headers: NN counts previous-channel
       continuation deltas, so the header is 2*NN + 16 bytes; `40 01`
       and `40 03` occur alongside `40 02`
     - tagless segment headers: no `40 NN` tag at all, just the 14-byte
       tail [field2:2][len:2][channel_id:4][marker:2][anchors:4]

Also: the header field documented as a "monotonic uint32 LE counter" is
really [channel_id][00][00][segment_index], with 0x46=Tran 0x47=Vert
0x48=Long 0x49=MicL — verified on 1697/1697 segment headers, zero
disagreements.  decode_waveform_v2 now takes the channel from that field
instead of rotation position, which was fragile: one missed header
desynced every channel after it.

parse_segment_header now returns n_prev_deltas/prev_deltas/marker/
anchors/channel/segment_index; the old fixed_pattern (02 00 00 01)
conflated the 2-byte marker with the first anchor.

Ground-truth corpus, end to end through the production path:
  exact 37 -> 72, truncated 23 -> 3, full-length value errors 15 -> 0.
Store-wide, 729 of 1388 series-3 waveform events decode differently and
728 gain samples; the scale fix changes float values on all of them, so
stored .h5 files need regenerating.

Still open: 3 events truncate at a header variant with a variable-width
prefix (2/4/6 bytes) before the channel id and an `01 00` marker.
Documented in docs/instantel_protocol_reference.md with byte offsets.

+20 tests.  No regressions: the byte-exact fixture suite still passes and
the full-suite failure list is unchanged from baseline (16 pre-existing
failures from gitignored fixtures).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog
2026-08-25 08:11:11 +00:00
serversdown beca5de06e docs: clean up and verify s3 protocol docs 2026-05-20 17:55:02 +00:00
Claude f68ee9f0f9 docs: clean up waveform-codec doc layers per review
Three "truth layers" had drifted apart between commits.  Fixed:

1. waveform_codec.py docstring rewritten from the 2026-05-08
   "structural framing only" state to the 2026-05-11 "Tran segment 0
   solved + segment-header partially decoded" state.  Killed stale
   "~80 sample-sets per segment" language (real segments are
   flash-page-byte-sized, not sample-count-sized; observed first-segment
   sizes are 42-510 samples depending on signal).  Killed stale
   "preamble is 7 or 9 bytes" language (always 7).

2. docs/instantel_protocol_reference.md §7.6.1: added a clear
   "CURRENT STATUS" box at the top with a status table.  Replaced the
   stale "~80 sample-sets" line with the verified per-event segment
   sizes.  Merged two redundant segment-header field-table sections.

3. docs/waveform_codec_re_status.md (NEW): clean working-status doc.
   Solved / not solved / hypothesis / next experiment / fixtures /
   tests.  The protocol reference remains the historical Rosetta
   Stone; this new file is the current-truth working note that
   shouldn't accumulate fossil layers.

4. CLAUDE.md §"Waveform body codec": prominent warning box at top —
   "DO NOT TRUST decoded sample arrays yet."  BW binary passthrough
   is the only sample-bearing output to trust until the decoder
   lands.  Added a "Next experiment" subsection pointing the next
   pass at the segment-channel scoring analyzer.

40 tests still pass.
2026-05-20 17:28:54 +00:00
Claude a0c9a482c7 codec-re: 00 NN is RLE; full Tran segment-0 decode (4 of 5 events)
User uploaded a Vert-heavy event (JQ0) and a Mic-heavy event (V70).
Those two were exactly what was needed to crack the next piece:

- 00 NN block = run-length-encoded zero deltas in the current channel.
  Append NN copies of the current cumulative value (no change).
- find_data_start now recognizes 00 NN as a valid first tag (some events
  begin with a leading 00 NN RLE block).
- decode_tran_initial now decodes the FULL segment 0 (not just the first
  data block).

Results across 5 fixture events:
  - M529LL1A.SP0 (loud-all-channels)  : 510 / 510  ✓
  - M529LL1L.JQ0 (Vert-heavy)         : 510 / 510  ✓
  - M529LL1L.V70 (Mic-heavy)          : 510 / 510  ✓
  - M529LL1A.SV0 (loud-from-start)    :  58 /  58  ✓
  - M529LL1A.SS0 (loud-from-start)    :  42 / 502  (stops at first 30 04)

The 30 04 block (only seen in loud-from-start events) hasn't been
decoded yet — likely a channel-switch marker for the high-amplitude
regime.

Also discovered: segment header (40 02) payload bytes [0:2] = T_delta
at first sample of new segment, [6:8] = byte length to next segment.
Multi-segment Tran decoding still diverges after sample 512 because
the per-segment channel ordering after the header is unknown.

Tests: 40 pass (up from 36).

Files:
- minimateplus/waveform_codec.py: find_data_start fix, RLE handling,
  full segment-0 decode in decode_tran_initial
- tests/test_waveform_codec.py: synthetic RLE test, full segment 0
  tests for JQ0 and V70
- tests/fixtures/5-11-26/: M529LL1L.JQ0, M529LL1L.V70 + TXT exports
- docs/instantel_protocol_reference.md §7.6.1: RLE + segment-header docs
2026-05-20 17:28:54 +00:00
Claude 6ac126e05c codec-re: crack Tran channel codec with high-amplitude May 11 bundle
User uploaded 3 high-amplitude events (PPV 6-7 in/s — shook the geophone
hard) to decode-re/5-11-26/.  These cracked the Tran codec:

- Preamble bytes [3:5] and [5:7] = Tran[0] and Tran[1] as int16 BE
  in 16-count units (LSB = 0.005 in/s).  Confirmed across all 7
  fixtures.
- First data block carries Tran deltas from sample 2 onward:
  * 10 NN block: NN/2 bytes of payload, each byte = two 4-bit signed
    nibble deltas (high nibble first)
  * 20 NN block: NN int8 signed deltas

Verified 22+42+46 = 110 Tran samples across SP0/SS0/SV0 with 0 errors
against BW's ASCII export.

Why the earlier 96-combination brute force failed: the quiet 5-8
events all had T[0] = T[1] ≈ 0 so the preamble's per-channel encoding
was undetectable.  Loud events made the encoding obvious.

What's solved:
- minimateplus.waveform_codec.decode_tran_initial: returns first
  N Tran samples in 16-count units for any body.
- Walker length formula for in-data 30 NN blocks (NN*2 instead of NN*4).
- Walker now handles bodies that start with 20 NN (in addition to 10 NN).

What's still open:
- Tran past the first data block (multi-block channel switching).
- Vert / Long / MicL channel encodings.
- Walker correctness past offset ~427 in event-b.

Tests: 36 pass.  decode_waveform_v2 still returns None — the full
multi-channel decoder is not wired up.  decode_tran_initial is the
new verified entry point.

Files: minimateplus/waveform_codec.py, tests/test_waveform_codec.py
(adds 5-11-26 fixtures + decode_tran_initial tests), and
docs/instantel_protocol_reference.md §7.6.1 (Tran codec spec).
2026-05-20 17:28:54 +00:00
Claude d3f77d1d96 codec-re: solve waveform body block framing; per-byte sample mapping still open
Decoded the structural framing of the Blastware waveform body — the bytes
between the 21-byte STRT record and the 26-byte file footer.  The body is
a sequence of tagged variable-length blocks, NOT raw int16 LE.  Five tag
types (10/20/00/30/40 NN) and their lengths are now confirmed against the
4-event May 2026 fixture bundle.  Body splits cleanly into ~16 segments
(for a 1280-sample event) separated by 40 02 segment headers carrying a
monotonically incrementing uint32 LE counter at bytes [8:12].

What's done:
- minimateplus/waveform_codec.py — block walker, segment splitter, segment
  header parser.  decode_waveform_v2 is a stub returning None until the
  byte-to-sample mapping is solved; client.py is unchanged.
- tests/test_waveform_codec.py — 31 tests covering block detection, lengths,
  contiguous-walk, segment splitting, segment-header parsing, and counter
  monotonicity.  All pass.
- tests/fixtures/decode-re-5-8-26/ — bundled fixtures (4 events, BW binary
  + Blastware ASCII export each).
- docs/instantel_protocol_reference.md §7.6.1 — replaced retraction box
  with the verified structural decoding plus an explicit list of what's
  still open.

What's still open: the per-byte mapping inside 10 NN / 20 NN blocks.  96
channel-permutation × nibble-order × sign-convention combinations were
brute-force tested; none match BW's ASCII export to within ±1 ADC count.
The codec is more elaborate than uniform 4-bit deltas — likely a hybrid
variable-bit-width scheme with segment-anchor resync points.  Next
recommended step: capture an event with a known calibration tone to pin
down magnitude scaling.

Walker also bails out partway through event-b (open issue documented in
both the module and the protocol reference).
2026-05-20 17:28:54 +00:00
Claude 8316a1bbd8 docs(protocol): accuracy sweep across the protocol reference
Three-pass audit of docs/instantel_protocol_reference.md against
CLAUDE.md and the minimateplus/ implementation. Closes long-standing
discrepancies that had accumulated as the protocol understanding
evolved month over month.

Major corrections:
- §2/§3: S3 frames terminate on bare ETX, not DLE+ETX; payload
  byte[1] is flags / byte[2] is SUB (was wrongly DLE/ADDR).
- §4.2: probe responses do not carry data length; DATA_LENGTH
  is a per-SUB hardcoded constant.
- §5.1: dropped stale duplicate "SUB 1C = TRIGGER CONFIG READ"
  row; SUB 0A lengths corrected from 0x30/0x26 to 0x46/0x2C.
- §5.3: added the missing write-frame mechanics (BW_CMD-only
  doubling, DLE-aware checksum, offset = data[1]+2, ack format,
  SUB 71 chunk parameters).
- §7.6.x: switched compliance-anchor convention from the unstable
  10-byte form to the canonical 6-byte `\xbe\x80\x00\x00\x00\x00`;
  recording_mode confirmed at anchor−8 in both read and write
  (the prior anchor−3/−4 split caused anchor drift on write).
  Sample_rate at anchor−6, histogram_interval at anchor−4 (now ✅),
  record_time at anchor+6. Geo_range row added at channel_label+33.
- §7.5b/§8: added the 10-byte sub_code=0x03 continuous-mode
  timestamp variant; peak vector sum location corrected from
  fixed offset 87 to label-relative tran_pos−12.
- §7.7.2: SUB 1E/1F token byte at params[7], not params[6].
- §7.7.3: SUB 0A length disambiguation rewritten.
- §7.8.4/§7.8.7: fi==9 skip marked FIXED; metadata-page TODO
  replaced with current decoder state.
- §11: POLL example wire bytes corrected; SUB 5A row added to
  checksum table.
- §13/§14: device-under-test updated to BE11529/S338.17; TCP
  Idle Timeout consistency fix (0→2 min); Data Forwarding
  Timeout units clarified.
- §15 (renumbered from second §14): open-question entries
  already resolved in CLAUDE.md closed out.
- Appendix D: extension taxonomy rewritten — extensions encode
  a timestamp (AB0T scheme), not recording mode.

Navigation note added to §7 acknowledging the organic-growth
duplicate section numbers (§7.5/§7.5b, §7.6, §7.7, §7.8, §7.9)
and pointing readers to the canonical sections for each topic.

https://claude.ai/code/session_019tWZybD94YUsBaEGhnM5A2
2026-05-20 15:41:42 +00:00
serversdown 8aea46b8a0 doc(fix): retracts raw int16 LE sample set assumptions. 2026-05-08 19:26:25 +00:00
claude c914a15e12 docs: update for v0.14.3 - Full continuous waveform download successful! 2026-05-05 20:37:52 -04:00
claude 7444738883 debug(protocol): event-N probe is now at counter = start_offset instead of start_offset + 0x46 2026-05-05 16:46:35 -04:00
claude 1ef55521b1 Fix: Removed duplicates from merge botch. Stable version of seismo_lab.py 2026-05-01 17:34:41 -04:00
claude 2f084ed105 fix(protocol): update chunk counter formula to use max(key4[2:4], 0x0400) for accurate data streaming 2026-04-26 01:28:47 -04:00
claude 35c3f4f945 fix(protocol): correct A5 frame classification and chunk counter formula 2026-04-24 17:25:29 -04:00
claude aa2b02535b fix(protocol): add record_time based chunk scaling for longer event record times 2026-04-23 17:33:16 -04:00
claude 6dcca4da79 feat(protocol): fully decode Blastware filename encoding and update related documentation 2026-04-22 23:43:31 -04:00
claude c47e3a3af0 feat(protocol): update Blastware file format documentation and encoding details 2026-04-22 19:16:05 -04:00
claude dfbc9f29c5 feat: first try at building waveform binary files. 2026-04-21 22:57:53 -04:00
claude 4331215e23 feat(protocol): enhance raw capture functionality and documentation updates
- Update `s3_bridge.py` to default raw capture file paths to "auto" for timestamped naming.
- Modify `gui_bridge.py` to pre-check raw capture options and streamline path handling.
- Extend `ach_server.py` to save both incoming and outgoing raw bytes for analysis.
- Revise `CHANGELOG.md` and `instantel_protocol_reference.md` to reflect changes in recording mode handling and compliance data encoding.
2026-04-21 16:07:24 -04:00
claude 3fb24e1895 feat(call-home): Implement Auto Call Home configuration management
- Added `CallHomeConfig` model to represent the Auto Call Home settings.
- Introduced methods in `MiniMateClient` for reading (`get_call_home_config`) and writing (`set_call_home_config`) the call home configuration.
- Updated `MiniMateProtocol` with new commands for call home operations (SUB 0x2C for read, SUB 0x7E for write, and SUB 0x7F for confirm).
- Created API endpoints for retrieving and updating call home settings in the server.
- Enhanced the web interface with a new "Call Home" tab for user interaction with call home settings.
- Implemented JavaScript functions for reading and writing call home configurations from the web app.
2026-04-20 18:23:48 -04:00
claude b6ffdcfa87 feat: implement geophone sensitivity and recording mode settings in compliance config 2026-04-20 17:03:58 -04:00
claude e04114fd6c feat: mapped record_mode protocol 2026-04-20 15:49:31 -04:00
claude b23cf4bb50 fix: max_geo_range correctly identified as ADC Scale factor number. 2026-04-17 19:43:45 -04:00
claude 5866ecdb3e docs: update protocol doc to reflect unkown status of max_range_geo. 2026-04-16 18:17:16 -04:00
claudeandClaude Sonnet 4.6 6acb419ebd docs: update protocol reference with v0.9.0 erase-all protocol
Changelog section:
- 5 new entries (2026-04-11): erase-all confirmation, SUB 0x06 purpose
  resolved, §7.11 added, §14.6 ACH session lifecycle marked IMPLEMENTED

§5.1 Request Commands:
- SUB 0x06 description updated: "EVENT STORAGE RANGE READ" (not "CHANNEL
  CONFIG READ"), token=0xFE, last 8 bytes = first/last stored event keys
- SUB 0xA3 added: ERASE ALL BEGIN — standard build_bw_frame, token=0xFE, ack 0x5C
- SUB 0xA2 added: ERASE ALL CONFIRM — standard build_bw_frame, token=0xFE, ack 0x5D

§5.2 Response SUBs:
- 0x06→0xF9 marked CONFIRMED 2026-04-11
- 0xA3→0x5C and 0xA2→0x5D added with CONFIRMED status

§7.11 (new section): Erase-All Protocol
- Full wire sequence (6 request/response pairs)
- SUB 0x06 storage range payload layout (36 bytes, last 8 = first/last key)
- Post-erase key counter reset: device restarts from 0x01110000
- Implementation notes pointing to client.py and ach_server.py

§14.6 ACH Session Lifecycle:
- Removed "Future" label — fully implemented in bridges/ach_server.py
- Added step 6 (optional erase), step 8 (DCD/DTR auto-resume)
- Documents ach_server.py flags and ach_state.json schema
- Notes RV55 DCD/DTR issue as known open problem

Open Questions table:
- SUB 0x06 purpose RESOLVED
- Erase-all sequence RESOLVED
- ACH server RESOLVED
- Sensor Check byte: still open, added as formal question
- RV55 DCD/DTR: added as new open question

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16 21:14:58 +00:00
claude 574d40027f feat: enhance logging messages in ach_server.py and add experiments.py for protocol minimization 2026-04-16 21:14:58 +00:00
claude 990cb8850e fix: correct monitoring flag and battery/memory offsets in _decode_monitor_status
section[6] is the monitoring flag (was wrongly section[1] — section[1] is always
0x00 in both states). Battery and memory fields use relative-from-end offsets
(section[-11:-9], section[-9:-5], section[-5:-1]) instead of absolute positions,
which broke when the payload grew by 3 bytes in monitoring mode.

Confirmed from full byte diff of 142 0xE3 frames in 4-8-26/2ndtry capture.
SFM start_monitoring now polls /device/monitor/status every 5s for up to 60s
instead of a fixed 25s delay (unit runs ~40s on-device sensor check before
confirming monitoring state).

Also corrects stale 1C→6E response anomaly claim in protocol reference — no
exceptions to the 0xFF−SUB rule are known.
2026-04-08 23:41:11 -04:00
claudeandClaude Sonnet 4.6 9bef430451 docs: document 5A end-of-stream signal, chunk timing, fi==9 bug, ADC conversion
Adds §7.8.4 to protocol reference and corresponding CLAUDE.md sections:

- End-of-stream: device sends exactly 1 raw byte after last chunk; handled
  via TimeoutError + bytes_fed>0 check → graceful break to termination
- Chunk timing: ~1s per chunk, 35 chunks for a 9,306-sample event, safe
  timeout is 10s (not default 120s)
- fi==9 decoder bug: hardcoded skip drops ~133 sample-sets per event;
  noted as known issue pending fix
- ADC conversion: counts × (range/32767) → physical units (in/s for geo)

Changelog entries added for all four items (2026-04-06).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 21:28:26 -04:00
claude 5b3e8af1e3 fix: remove special case chunk counter, all chunks use chunk_num * 0x0400 2026-04-06 17:03:09 -04:00
claude ad1c9e48b0 fix: update MiniMateClient and protocol to ensure correct handling of 1F calls and improve event download sequence 2026-04-06 15:58:03 -04:00
claude 790e442a7a feat: implement raw ADC waveform decoding and download functionality
- Added `_decode_a5_waveform()` to parse SUB 5A frames into per-channel time-series data.
- Introduced `download_waveform(event)` method in `MiniMateClient` to fetch full waveform data.
- Updated `Event` model to include new fields: `total_samples`, `pretrig_samples`, `rectime_seconds`, and `_waveform_key`.
- Enhanced documentation in `CHANGELOG.md` and `instantel_protocol_reference.md` to reflect new features and confirmed protocol details.
2026-04-03 13:53:09 -04:00
claude 5d0f0855f2 doc: update to .0.6.0 with full working event read loop 2026-04-02 17:30:33 -04:00
claude 6a42facf02 docs: update protocol reference for SUB 1A compliance config read
New/corrected sections:

§7.6.1 Record Time — corrected: anchor+10 supersedes the +0x28 absolute
offset.  Added 3 s, 5 s, 8 s confirmations alongside existing 7/10/13.
Warning added: do NOT use fixed offset for BE11529.

§7.6.2 (NEW) SUB 1A Multi-Frame Read Protocol — 4-frame A/B/C/D
sequence documented (reverse-engineered from raw_bw capture).  E5
page_key field explained.  BE11529 duplicate-page behaviour and
(page_key, chunk_len) dedup strategy documented.

§7.6.3 (NEW) Sample Rate and DLE Jitter — Normal/Fast/Faster = 1024/
2048/4096 Sa/s confirmed.  Root cause of ±1 byte cfg jitter explained:
4096 = 0x1000 → `10 10 00` in raw frame → `10 00` after DLE unstuffing
= 1 byte shorter than 04 00/08 00.  Anchor search requirement explained.

Changelog — 5 new entries covering the 4-frame sequence, duplicate-page
detection, record_time anchor correction, sample_rate confirmation, and
_pending_frames / reset_parser=False implementation notes.

Quick Reference — Record Time and Sample Rate rows updated with correct
locations, types, and confirmed values.

Open Questions — SUB 1A item updated to "substantially resolved".
Record time item updated. Sample rate added as resolved.
2026-04-01 16:54:51 -04:00
claudeandClaude Sonnet 4.6 4944974f6e feat: decode waveform record timestamp, record type, and Peak Vector Sum
Confirmed 2026-04-01 against Blastware event report for BE11529 thump
event ("00:28:12 April 1, 2026", PVS 3.906 in/s).

models.py:
- Timestamp.from_waveform_record(): decode 9-byte format from 0C record
  bytes[0-8]: [day][sub_code][month][year:2BE][?][hour][min][sec]
- Timestamp: add hour/minute/second optional fields; __str__ includes
  time when available
- PeakValues: add peak_vector_sum field (confirmed fixed offset 87)

client.py:
- _decode_waveform_record_into: add timestamp decode from bytes[0:9]
- _extract_record_type: decode byte[1] (sub_code), not ASCII string
  search; 0x10 → "Waveform", histogram TBD
- _extract_peak_floats: add PVS from offset 87 (IEEE 754 BE float32)
  = √(T²+V²+L²) at max instantaneous vector moment

sfm/server.py:
- _serialise_timestamp: add hour/minute/second/day fields to JSON
- _serialise_peak_values: add peak_vector_sum to JSON

docs: update §7.7.5 and §8 with confirmed 9-byte timestamp layout,
PVS field, and byte[1] record type encoding; update command table;
close resolved open questions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-01 00:53:34 -04:00
claude 9f52745bb4 feat: add full event download pipeline 2026-03-31 20:48:03 -04:00
claudeandClaude Sonnet 4.6 1078576023 Add Console tab to seismo_lab + document RV50/RV55 modem config
seismo_lab.py:
- Add ConsolePanel — third tab for direct device connections over serial
  or TCP without the bridge subprocess
- Commands: POLL, Serial #, Full Config, Event Index (open/close per cmd)
- Colour-coded output: TX blue, RX raw teal, parsed green, errors red
- Save Log and Send to Analyzer buttons; auto-saves to bridges/captures/
- Queue/after(100) pattern — no performance impact
- Add SCRIPT_DIR to sys.path so minimateplus imports work from GUI

docs/instantel_protocol_reference.md:
- Confirm calibration year field at SUB FE payload offset 0x56–0x57
  (uint16 BE): 0x07E7=2023 (BE18189), 0x07E9=2025 (BE11529)
- Document full Sierra Wireless RV50/RV55 required ACEmanager settings
  (Quiet Mode, Data Forwarding Timeout, TCP Connect Response Delay, etc.)
- Correct §14.2: RV50/RV55 sends RING/CONNECT over TCP to caller even
  with Quiet Mode on; parser handles by scanning for DLE+STX
- Confirm "Operating System" boot string capture via cold-start Console
- Resolve open question: 0x07E7 field = calibration year

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-31 17:19:31 -04:00
claude 51d1aa917a Add TCP/modem transport (Sierra Wireless RV55/RX55 field units)
- minimateplus/transport.py: add TcpTransport — stdlib socket-based transport
  with same interface as SerialTransport. Overrides read_until_idle() with
  idle_gap=1.5s to absorb the modem's 1-second serial data forwarding buffer.
- minimateplus/client.py: make `port` param optional (default "") so
  MiniMateClient works cleanly when a pre-built transport is injected.
- minimateplus/__init__.py: export SerialTransport and TcpTransport.
- sfm/server.py: add `host` / `tcp_port` query params to all device endpoints.
  New _build_client() helper selects TCP or serial transport automatically.
  OSError (connection refused, timeout) now returns HTTP 502.
- docs/instantel_protocol_reference.md: add changelog entry and full §14
  (TCP/Modem Transport) documenting confirmed transparent passthrough, no ENQ
  on connect, modem forwarding delay, call-up vs ACH modes, and hardware note
  deprecating Raven X in favour of RV55/RX55.

Usage: GET /device/info?host=<modem_ip>&tcp_port=12345
2026-03-31 00:44:50 -04:00
claude f8f590b19b sfm first build 2026-03-30 23:23:29 -04:00
claude 58a35a3afd feat: add large BW→S3 write frame checksum validation in s3_parser 2026-03-13 17:53:11 -04:00
claude 99d66453fe feat: enhance session completeness tracking in s3_analyzer and seismo_lab 2026-03-11 18:48:42 -04:00
claude faa869d03b doc: protocol ref updated to v0.20 2026-03-09 19:02:53 -04:00
claude fa9873cf4a doc: §2, §10, Appendix C | **MILESTONE — Link-layer grammar formally confirmed.** 2026-03-04 17:42:15 -05:00
claude 22d4023ea0 chore: update version to v0.5.1 in s3_bridge.py
docs: update Instantel protocol reference with framing corrections and clarifications
2026-03-03 16:30:09 -05:00
claude feceb7b482 docs: updated protocol reference with findings from 3-2-26 (v0.19) 2026-03-03 13:56:22 -05:00
claude 43c9c8b3a3 feat: added raw binary data tracking for accurate format parser. 2026-03-02 15:47:52 -05:00