3f58402085054517fe49d959356f04779f868dc8
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  serversdownandClaude Opus 5
|
ef1e99b0a0 |
fix(histogram): block is big-endian + terminal block tail — 1/1196 to 1211/1211
Two errors in the series-3 histogram block model, both found by diffing against the per-interval data table in the preserved Blastware ASCII exports (1211 files in the prod snapshot — far stronger ground truth than the header PPV used previously). 1. The block is uniformly BIG-ENDIAN. Peaks and half-periods are uint16 BE (T_peak [5:7], T_halfperiod [7:9], V_peak [9:11], V_halfperiod [11:13], L_peak [13:15], L_halfperiod [15:17], M_peak [17:19], M_halfperiod [19:21]); only block_ctr [2:4] is little-endian. The old uint8-peak model silently CLIPPED any peak above 1.275 in/s: the final interval of BE18193/T193LQ9K.OE0H reads 8.270 in/s in BW's export (1654 counts = 0x0676) and decoded as 0x76 = 118 = 0.590. The byte documented as a per-channel "annotation" was never an annotation — it is the half-period's high byte, which is exactly why it was non-zero on the sub-Hz intervals BW renders as "<1.0". The marker is block[4] alone. Testing [4:6] as a uint16 LE marker forced block[5] == 0, which is what capped the peak at one byte. 2. The final block of each stream carries tail 9c 06 00 42 instead of 1e 0a 00 00, and holds arbitrary bytes at [21:23]. Rejecting it dropped the last interval of nearly every histogram — frequently the interval holding the event peak, so the file's PPV read low. Verified end to end through the production path: 1211/1211 histograms decode exactly (interval count + every per-interval peak), plus 842,442 per-interval frequency comparisons with zero mismatches. Previously 1 of 1196 files was fully correct. decode_histogram_body_full records expose `is_terminal` in place of the removed `annotations` tuple. +6 tests. No regressions: full-suite failure list unchanged from baseline. NOTE: stored histogram .h5 files need regenerating to pick this up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HgTe8CamXAHcAmaQ6QNcog |
||
|
serversdownandClaude Opus 4.7
|
d506ebc103 |
histogram_codec: peak count is uint8 (not uint16 LE) — properly cracks
the BE9558 / BE18003 extension-byte case The bytes at [7]/[11]/[15]/[19] are an annotation field (purpose still unclear — empirically non-zero on intervals with sub-Hz or unmeasurable freq), NOT the high byte of the peak count. The N844 fixture corpus the original RE was done against had zero values in those bytes for every block, so uint8 and uint16 LE were equivalent there — but on real BE9558 Tran-drift events and BE18003 Histogram+Continuous events the uint16 LE interpretation produced peaks up to 268 in/s and 35× inflated PVS sums. Cross-correlated against BW's per-interval ASCII export on: - K558LKZU/LL1P/LL3K → 100% T/V/L/M peak match (1435 blocks each) - T003LKZR/LL0O/LL1M → 100% T/V/L, 99.3% M (0.05 dB rounding only) - N599LKZS/LL0L → 100% all channels - N844 fixture corpus → 100% all channels (unchanged) Annotations preserved on every record for future RE; the defensive _MAX_PEAK_COUNT bound is no longer needed (uint8 maxes at 1.275 in/s, well below any physical limit). Synthetic regression test added using the verbatim K558LKZU.RE0H interval-12 block. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
serversdownandClaude Opus 4.7
|
7183b953e4 |
minimateplus: histogram body codec — FULLY DECODED
The histogram-mode event body is now byte-exact decodable.
Companion to the waveform body codec — together they cover every
event file the watcher forwards. Cracked in one session via
cross-event correlation against BW's ASCII export.
The §7.6.2 spec in instantel_protocol_reference.md was structurally
correct (32-byte blocks) but the per-sample semantics were
under-documented. Cross-checking block 130 of N844L6Z8.ZR0H
against its TXT row revealed the layout perfectly:
slot[0] = 10 (constant marker)
slot[1] = T_peak_count (× 0.005 → in/s at Normal range)
slot[2] = T_halfperiod (freq_Hz = 512 / halfp)
slot[3] = V_peak_count
slot[4] = V_halfperiod
slot[5] = L_peak_count
slot[6] = L_halfperiod
slot[7] = MicL_peak_count (dB via waveform_codec.mic_count_to_db)
slot[8] = MicL_halfperiod
The `>100 Hz` sentinel is halfperiod ≤ 5 (since 512/5 = 100 Hz).
Mic dB uses the SAME formula as the waveform codec (sign × (81.94
+ 20·log10(|count|))) — they share the mic ADC calibration constant.
Block identification anchor: bytes [22:24] == 0x0000 AND
bytes [28:32] == 1e 0a 00 00. The tail signature is the most
reliable distinguisher from non-block content in the file.
Files:
minimateplus/histogram_codec.py (new) — decoder + public API
matching the waveform codec's shape:
walk_body(body) -> records
decode_histogram_body(body) -> {Tran, Vert, Long, MicL}
decode_histogram_body_full(body) -> [per-interval dicts]
half_period_to_hz, geo_count_to_ins helpers
minimateplus/event_file_io.py (modified) — read_blastware_file
now tries the waveform codec first, falls back to the histogram
codec on failure. Same output shape, same downstream pipeline.
tests/test_histogram_codec.py (new) — 24 regression locks against
the in-repo fixture corpus, byte-exact against BW ASCII export
for peaks (all 4 channels), frequencies (all 4 channels,
including >100 Hz sentinel handling), block framing, and
segment-ID accounting.
scripts/backfill_sidecars.py (modified) — the has_samples
short-circuit added in the histogram-pending era is now a
pure defensive guard. Histograms in prod will regen .h5 files
correctly on the next backfill run.
docs/histogram_codec_re_status.md (updated) — supersedes the
earlier "in progress" version with the verified format and
test-coverage summary. Notes a few non-essential fields still
open (4-byte block metadata, Geo PVS, Mic psi(L) — none of
which are needed for waveform reconstruction).
Total verified coverage: ~3,500 blocks across 5 fixtures, every
field of every block byte-exact against BW.
The watcher-forwarded histogram event corpus on prod (~10,000
events) will now produce correct .h5 sidecars on the next backfill
run. No additional changes needed to the backfill flow — the
existing tool_version-bump cascade picks them up automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
serversdownandClaude Opus 4.7
|
c3c7fe559c |
docs: histogram body codec RE — starting-point status doc
Captures everything learned in the 2026-05-20 session before scope
forced a pause:
- Block framing is solved: 32-byte blocks, one per histogram
interval, signature byte pattern `[22:24]=0x0000` +
`[28:32]=0x1e 0x0a 0x00 0x00` reliably identifies data blocks.
- Block count = interval count (791 blocks in N844L20G.630H for
a TXT-reported 792 intervals).
- Sample[0] = Tran peak in 0.0005 in/s/count units (verified on
one event — needs cross-event confirmation).
- Samples 1-8 → channel/metric mapping is still open. None of
the obvious layouts (peak-then-freq alternating, all-peaks-
then-all-freqs, per-channel 3-tuples) match the TXT values
across multiple blocks. Likely needs a higher-activity
fixture (current N844 corpus is all noise-floor data) to
disambiguate.
- `>100 Hz` sentinel encoding in the binary is unknown.
- 4-byte variable metadata field at block[24:28] needs
correlation work against TXT columns.
Doc mirrors the structure of docs/waveform_codec_re_status.md so
a future RE session has a familiar entry point. Includes the
suggested attack plan + the code seam where the eventual decoder
will land (minimateplus/histogram_codec.py).
The §7.6.2 spec in instantel_protocol_reference.md is structurally
correct but doesn't pin down per-sample semantics — this doc
supersedes it where they conflict on confidence level.
No code shipped on this branch. When the codec is cracked, the
plan is to land minimateplus/histogram_codec.py + wire into
event_file_io.read_blastware_file() + remove the has_samples
short-circuit from scripts/backfill_sidecars.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|