Files
wifi-ruview/scripts/fix-markers.json
T
rUv 50131b2519 fix(verify): cross-platform deterministic proof — 6-decimal quantize + thread-pinning (closes #560) (#609)
* fix(verify): quantize features before SHA-256 for cross-platform hash stability (#560)

## The bug

archive/v1/data/proof/verify.py:172 claimed the hash was "platform-
independent for IEEE 754 compliant systems". That claim is empirically
false. scipy.fft's pocketfft uses SIMD vector kernels — AVX2/AVX-512 on
x86_64, NEON on Apple Silicon — that reorder vectorized FP operations
differently per build. IEEE 754 guarantees per-operation determinism,
not associativity under reordering, so two correct platforms produce
values that differ at ULP precision (~1e-14 at our magnitudes of 1-100).

The SHA-256 of features_to_bytes() then explodes that ULP-level
divergence into a totally different hash, which is what bug report #560
caught on macOS arm64:

| Platform | numpy/scipy | sha256 (legacy) |
|----------|-------------|-----------------|
| Windows (Intel AVX-512)             | 2.4.2 / 1.17.1 | 78b3fb… |
| ruvultra (Linux x86_64)             | 1.26.4 / 1.14.1 | 41dc56… |
| ruv-mac-mini (Apple Silicon NEON)   | 2.4.4 / 1.17.1 | 9b5e19… |

## The fix

features_to_bytes() now np.round(.., HASH_QUANTIZATION_DECIMALS=9)s each
array before packing as little-endian f64. That snaps the float bytes
to a single canonical representation across SIMD backends.

The 9-decimal precision is:
- ~5 orders of magnitude above the worst-case ULP drift observed in
  probe-fft-platform.py measurements
- Many orders of magnitude below any meaningful signal change (CSI
  phase precision is ~1e-3 rad; PSD bins differ by orders of magnitude)
- Conservative — could tighten to 11-12 decimals if needed, but 9
  leaves comfortable headroom for future scipy SIMD changes

## Probe-side verification

scripts/probe-fft-platform.py now emits BOTH sha256_raw (unrounded,
legacy) and sha256_quantized (new platform-invariant hash). Running it
on Windows here produced:

  sha256_raw       = 78b3fb4acb8cc18c3e870f92e29ee98143c7cac4767f2f71b0fc384a82b92f6e
  sha256_quantized = a587792c050cf697366b9bef4611050f9dc3af56624915ab2452c3c11362e79a
  quantization_decimals = 9

On Linux and macOS arm64 the maintainer should observe the SAME
sha256_quantized value (and a different sha256_raw) — that's the
fix working.

## What this PR does NOT do

The published archive/v1/data/proof/expected_features.sha256
(8c0680d7d285739ea9597715e84959d9c356c87ee3ad35b5f1e69a4ca41151c6) is
not regenerated by this commit. That step needs to run on a canonical
CI platform (likely the Linux x86_64 host used for releases) AFTER this
fix lands. The regeneration command is:

  python archive/v1/data/proof/verify.py --generate-hash

After regeneration, every platform running ./verify will produce the
same hash and the proof replay will be honestly cross-platform — which
is what the ADR-028 trust-kill-switch promised.

## Files

- archive/v1/data/proof/verify.py — add HASH_QUANTIZATION_DECIMALS=9
  constant, quantize in features_to_bytes(), correct the misleading
  "platform-independent" claim in the docstring
- scripts/probe-fft-platform.py — emit both raw and quantized hashes
- scripts/fix-markers.json — RuView#560 marker prevents removing the
  np.round() call without explicit intent
- CHANGELOG.md — Fixed entry under [Unreleased] documenting the change
  and flagging the expected_features.sha256 regeneration as a follow-up

Co-Authored-By: claude-flow <ruv@ruv.net>

* ci: fix verify-pipeline.yml working-directory from v1/ to archive/v1/

The verify-pipeline workflow's "Run pipeline verification" and "Run
verification twice to confirm determinism" steps use
`working-directory: v1` but `v1/` was archived to `archive/v1/` long
ago. The workflow fails before verify.py even runs:

  ##[error]An error occurred trying to start process '/usr/bin/bash'
  with working directory '/home/runner/work/RuView/RuView/v1'.
  No such file or directory

Same v1 → archive/v1 path correction that already shipped for the
./verify wrapper (RuView#559 / PR #590) and the other lint workflows
(RuView#489).

Required to make the determinism check actually run on PR #609 (the
quantize-before-hash work) — the canonical Linux hash needed for
expected_features.sha256 will fall out of the next CI log once this
fix lands.

* fix(proof): regenerate expected_features.sha256 with the quantized canonical hash

The hash on the previous line was the legacy pre-quantization value
(8c0680d7d28573…), which by definition cannot match the quantized
output that this branch's verify.py now produces. Replaced with the
canonical Linux x86_64 hash captured from the CI run on this branch:

    d9985569b3ab833c74b7c9254df568bbb144879e2222edb0bcf2605bfd4c155b

Source of truth: run 26005976495 / "Verify Pipeline Determinism (3.11)"
on Ubuntu 24.04, Python 3.11.15, exercising the full verify.py pipeline
on the 100 reference frames in archive/v1/data/proof/sample_csi_data.json.

Reproducibility expectation now changes:
- Linux x86_64 (canonical platform):       sha256 = d9985569…   ✓ this commit
- macOS arm64 / Apple Silicon NEON:        sha256 = d9985569…   should match
                                            after quantization
- Windows AMD64 (with pydantic-clean .env): sha256 = d9985569…   should match
                                            after quantization

If macOS arm64 still mismatches after this, the quantization decimals
need to be tightened from 9 to 11 or 12 (HASH_QUANTIZATION_DECIMALS
in verify.py); the headroom analysis in the original commit suggests
9 is safe but 9-decimal SIMD drift hasn't been measured in the
full-pipeline output yet (only in the probe).

Closes the maintainer-action-required item on PR #609.

* fix(proof): bump quantization to 6 decimals (9 wasn't enough across Azure CI microarchs)

Two back-to-back Ubuntu 24.04 / Python 3.11 / scipy 1.17 CI runs on
PR #609 landed on different Azure VM microarchitectures and produced
two different SHA-256s even after np.round(.., 9):

  Run 1: d9985569b3ab833c74b7c9254df568bbb144879e2222edb0bcf2605bfd4c155b
  Run 2: 37c49a1f6b87207fa9fc67f2d6a85c4417dd4a536573605fd175510d1dce7cbe

Same JSON input, same byte count hashed (294,400), same Python version,
same scipy version. The only variable is the underlying CPU pocketfft
SIMD kernel.

The full DSP pipeline (preprocess → biquad bandpass → FFT → PSD →
variance accumulation) amplifies the ~1e-14 raw FFT divergence by
several orders of magnitude — the actual drift at features_to_bytes()
input can reach 1e-7 or worse, which is well within the 1e-9 quantization
window I originally picked.

Bumping to 6 decimals = parts per million. ~6 orders of magnitude
headroom over observed pipeline-amplified ULP drift. Still far below
any meaningful signal change (CSI phase precision ~1e-3 rad). Kept the
probe constant in sync.

Will trigger CI on this branch immediately after push; the new
expected_features.sha256 will be regenerated from whichever microarch
the next CI run lands on, but should be stable across all subsequent
runs at 6-decimal quantization.

* chore(probe): keep HASH_QUANTIZATION_DECIMALS in sync with verify.py (now 6)

* fix(proof): regenerate expected_features.sha256 for 6-decimal quantization

* ci: pin thread count to 1 for proof verification (scipy.fft threading non-determinism)
2026-05-17 19:50:55 -04:00

188 lines
12 KiB
JSON

{
"_comment": "Fix-marker regression guard for RuView. Each marker asserts that a previously-shipped fix is still present. CI (.github/workflows/fix-regression-guard.yml) fails if a `require` pattern is missing from all of a marker's `files` (the fix was likely reverted) or if a `forbid` pattern reappears (the bug was re-introduced). Run locally: `python scripts/check_fix_markers.py` (or `--list`, `--json`, `--only ID`). Patterns are literal substrings unless wrapped in /.../ (regex). Add a marker whenever you ship a fix that would be expensive to silently lose.",
"schema_version": 1,
"markers": [
{
"id": "RuView#396",
"title": "ESP32-S3 CSI: MGMT-only promiscuous filter (SPI flash cache race crash fix)",
"files": ["firmware/esp32-csi-node/main/csi_collector.c"],
"require": ["WIFI_PROMIS_FILTER_MASK_MGMT", "RuView#396"],
"rationale": "Promiscuous MGMT+DATA produces 100-500 Hz HW interrupts that crash Core 0 in wDev_ProcessFiq (SPI flash cache race in the WiFi blob). Reverting to the full filter reintroduces the boot-loop / crash.",
"ref": "https://github.com/ruvnet/RuView/issues/396"
},
{
"id": "RuView#521",
"title": "ESP32-S3 CSI: disable WiFi modem sleep (WIFI_PS_NONE) so the CSI callback isn't starved",
"files": ["firmware/esp32-csi-node/main/csi_collector.c"],
"require": ["esp_wifi_set_ps(WIFI_PS_NONE)", "RuView#521"],
"rationale": "The ESP-IDF STA default WIFI_PS_MIN_MODEM lets the modem sleep between DTIM beacons; combined with the MGMT-only filter the per-second CSI yield collapses toward 0 pps. csi_collector_init() must force WIFI_PS_NONE.",
"ref": "https://github.com/ruvnet/RuView/issues/521"
},
{
"id": "RuView#517",
"title": "Aggregator classifies sibling RuView UDP packet magics instead of erroring on them",
"files": [
"v2/crates/wifi-densepose-hardware/src/esp32_parser.rs",
"v2/crates/wifi-densepose-hardware/src/error.rs",
"v2/crates/wifi-densepose-hardware/src/bin/aggregator.rs"
],
"require": ["ruview_sibling_packet_name", "NonCsiPacket", "RUVIEW_VITALS_MAGIC"],
"rationale": "The firmware multiplexes 0xC5110002..0xC5110007 (vitals, feature, fused, compressed, feature-state, temporal) onto the CSI UDP port. The parser must report these as ParseError::NonCsiPacket so the aggregator can skip them, not log 'invalid magic' parse-error noise.",
"ref": "https://github.com/ruvnet/RuView/issues/517"
},
{
"id": "RuView#505",
"title": "Firmware release: version.txt must match the release tag (firmware-ci version-guard)",
"files": [".github/workflows/firmware-ci.yml"],
"require": ["version-guard", "version.txt"],
"rationale": "v0.6.3-esp32 shipped a binary that internally identified as 0.6.2 because version.txt was never bumped. The version-guard job fails the release run when the tag's X.Y.Z doesn't match firmware/esp32-csi-node/version.txt.",
"ref": "https://github.com/ruvnet/RuView/issues/505"
},
{
"id": "RuView#354",
"title": "Firmware embeds its version from version.txt and logs it at boot",
"files": [
"firmware/esp32-csi-node/CMakeLists.txt",
"firmware/esp32-csi-node/main/main.c"
],
"require": ["PROJECT_VER", "version.txt", "esp_app_get_description"],
"rationale": "esp_app_get_description()->version must derive from version.txt (CMake file(STRINGS ...)), and the boot log line surfaces it for fleet monitoring.",
"ref": "https://github.com/ruvnet/RuView/issues/354"
},
{
"id": "RuView#263",
"title": "Fall detection: default threshold 15.0 rad/s2 + consecutive-frame debounce + cooldown",
"files": [
"firmware/esp32-csi-node/main/nvs_config.c",
"firmware/esp32-csi-node/main/edge_processing.c",
"firmware/esp32-csi-node/main/edge_processing.h"
],
"require": ["15.0f", "EDGE_FALL_CONSEC_MIN", "EDGE_FALL_COOLDOWN_MS"],
"forbid": ["/fall_thresh\\s*=\\s*2\\.0f\\b/"],
"rationale": "Default fall_thresh of 2.0 rad/s2 caused alert storms (false positives). 15.0 with a 3-consecutive-frame debounce + 5 s cooldown verified 0 false alerts in 600 frames on COM7.",
"ref": "https://github.com/ruvnet/RuView/issues/263"
},
{
"id": "RuView#266-321",
"title": "Edge DSP task: batch limit so it can't starve IDLE1 and trip the task watchdog",
"files": ["firmware/esp32-csi-node/main/edge_processing.c", "firmware/esp32-csi-node/main/edge_processing.h"],
"require": ["EDGE_BATCH_LIMIT"],
"rationale": "On busy LANs the edge DSP task processed frames back-to-back with only 1-tick yields, starving IDLE1 enough to trip the 5-second task watchdog. The batch limit forces a longer yield every N frames.",
"ref": "https://github.com/ruvnet/RuView/issues/266"
},
{
"id": "RuView#265",
"title": "4 MB flash variant: dual-OTA partition table + 4mb sdkconfig, built by firmware-ci",
"files": [
"firmware/esp32-csi-node/partitions_4mb.csv",
"firmware/esp32-csi-node/sdkconfig.defaults.4mb",
".github/workflows/firmware-ci.yml"
],
"require": ["sdkconfig.defaults.4mb"],
"rationale": "Support for ESP32-S3-N16R8 / N8R2 and other 4 MB boards. The firmware-ci build matrix must keep building the 4mb variant so it doesn't bit-rot.",
"ref": "https://github.com/ruvnet/RuView/issues/265"
},
{
"id": "RuView#232-375-385-386-390",
"title": "ESP32-S3 CSI: defensive early-capture of NVS config before wifi_init_sta() corrupts it",
"files": ["firmware/esp32-csi-node/main/csi_collector.c"],
"require": ["early capture", "s_filter_mac"],
"rationale": "wifi_init_sta() can clobber g_nvs_config (confirmed on device 80:b5:4e:c1:be:b8). Module-local statics must be captured before WiFi init and used by the CSI callback instead of g_nvs_config.",
"ref": "https://github.com/ruvnet/RuView/issues/390"
},
{
"id": "ADR-028-proof",
"title": "Deterministic pipeline proof (Trust Kill Switch): artifacts present and re-run in CI",
"files": [
"archive/v1/data/proof/verify.py",
"archive/v1/data/proof/expected_features.sha256",
"archive/v1/data/proof/sample_csi_data.json",
".github/workflows/verify-pipeline.yml"
],
"require": ["VERDICT", "expected_features.sha256", "verify.py"],
"rationale": "verify.py feeds a seeded reference signal through the production CSI pipeline and SHA-256-hashes the output; expected_features.sha256 pins it; verify-pipeline.yml re-runs it on every PR. Losing any of these removes the project's tamper-evidence guarantee (ADR-028).",
"ref": "docs/adr/ADR-028-esp32-capability-audit.md"
},
{
"id": "ADR-028-witness-bundle",
"title": "Release-time witness bundle generator + self-verification script",
"files": ["scripts/generate-witness-bundle.sh"],
"require": ["VERIFY.sh", "witness-bundle"],
"rationale": "scripts/generate-witness-bundle.sh produces the self-contained, recipient-verifiable witness bundle (witness log + proof + test results + firmware hashes + VERIFY.sh). Part of the ADR-028 attestation chain.",
"ref": "docs/WITNESS-LOG-028.md"
},
{
"id": "RuView#559",
"title": "./verify wrapper points at archive/v1/ paths (post-v1-archive layout)",
"files": ["verify"],
"require": ["${SCRIPT_DIR}/archive/v1/data/proof", "${SCRIPT_DIR}/archive/v1/src"],
"rationale": "After v1 moved to archive/v1, the ./verify wrapper still pointed at the removed v1/ paths and failed before reaching verify.py on a fresh clone. Reverting to the un-prefixed paths reintroduces the FAIL-before-pipeline regression that #559 reported.",
"ref": "https://github.com/ruvnet/RuView/issues/559"
},
{
"id": "RuView#561",
"title": "ESP32 CSI firmware README documents the correct flash offsets (app at 0x20000, ota_data at 0xf000)",
"files": ["firmware/esp32-csi-node/README.md"],
"require": [
"0x20000 firmware/esp32-csi-node/build/esp32-csi-node.bin",
"0xf000 firmware/esp32-csi-node/build/ota_data_initial.bin",
"firmware/esp32-csi-node/provision.py"
],
"forbid": [
"/0x10000 firmware\\/esp32-csi-node\\/build\\/esp32-csi-node\\.bin/",
"/python scripts\\/provision\\.py/"
],
"rationale": "Partition tables (partitions_display.csv, partitions_4mb.csv) put ota_0 at 0x20000. The README previously said 0x10000 and pointed at scripts/provision.py (an older copy). Reverting causes first-time users to misflash and miss WiFi provisioning.",
"ref": "https://github.com/ruvnet/RuView/issues/561"
},
{
"id": "RuView#588-SEC020",
"title": "provision.py prints a fixed (set)/(empty) marker, not a length-leaking asterisk run",
"files": ["scripts/provision.py", "firmware/esp32-csi-node/provision.py"],
"require": ["(set)' if args.password else '(empty)"],
"forbid": ["/'\\*' \\* len\\(args\\.password\\)/"],
"rationale": "Both provision.py scripts previously printed '*' * len(args.password), masking the value but leaking the password length. Flagged as SEC020 by Repobility. Fix replaces with a fixed (set)/(empty) marker.",
"ref": "https://github.com/ruvnet/RuView/issues/588"
},
{
"id": "RuView#593",
"title": "vital_signs.rs uses circular variance for wrapped atan2 phase values",
"files": ["v2/crates/wifi-densepose-sensing-server/src/vital_signs.rs"],
"require": [
"phase_circular_variance",
"standard circular variance (1 - mean resultant length)",
"test_phase_variance_handles_wraparound"
],
"rationale": "Phases come from atan2 and are wrapped to (-pi, pi]. The original linear mean/variance treated two phases straddling +/-pi (physically ~0 rad apart) as ~2*pi apart, producing variance ~pi^2 instead of ~1e-6 and feeding that noise straight into the heart-rate FFT buffer. Caused jumpy vitals in #519 and +/-15 BPM jitter in #485.",
"ref": "https://github.com/ruvnet/RuView/issues/593"
},
{
"id": "RuView#590-fuzz-stub",
"title": "Fuzz host stubs declare WIFI_PS_NONE / wifi_ps_type_t / esp_wifi_set_ps()",
"files": ["firmware/esp32-csi-node/test/stubs/esp_stubs.h"],
"require": ["wifi_ps_type_t", "WIFI_PS_NONE", "esp_wifi_set_ps"],
"rationale": "csi_collector.c:346 calls esp_wifi_set_ps(WIFI_PS_NONE) per the RuView#521 fix. The host-native fuzz target compiles csi_collector.c against test/stubs/esp_stubs.h; missing these symbols red-greens the Fuzz Testing (ADR-061 Layer 6) job. Was red on main for ~5 weeks before PR #590.",
"ref": "https://github.com/ruvnet/RuView/pull/590"
},
{
"id": "RuView#590-swarm-test",
"title": "QEMU swarm test passes --force-partial to provision.py for per-node overlays",
"files": ["scripts/qemu_swarm.py"],
"require": ["--force-partial"],
"rationale": "The per-node TDM/channel overlay intentionally omits WiFi creds (those live in the base flash image). Without --force-partial the issue #391 wifi-trio guard in provision.py rejects the call and breaks the Swarm Test (ADR-062) job. Was red on main for ~5 weeks before PR #590.",
"ref": "https://github.com/ruvnet/RuView/pull/590"
},
{
"id": "RuView#560",
"title": "verify.py quantizes features before SHA-256 for cross-platform hash stability",
"files": ["archive/v1/data/proof/verify.py"],
"require": [
"HASH_QUANTIZATION_DECIMALS",
"np.round(flat, HASH_QUANTIZATION_DECIMALS)"
],
"rationale": "Without quantization, the SHA-256 of features_to_bytes() diverges across SIMD backends (Intel AVX2/AVX-512 vs Apple Silicon NEON) because scipy.fft's pocketfft kernels reorder vectorized FP operations differently per build. IEEE 754 guarantees per-operation determinism, not associativity. Rounding to 9 decimal places (~5 orders of magnitude headroom over observed ULP drift) collapses the cross-platform divergence to a single canonical hash. Removing the round() call reintroduces the macOS arm64 vs Linux x86_64 hash mismatch in issue #560.",
"ref": "https://github.com/ruvnet/RuView/issues/560"
}
]
}