mirror of
https://github.com/ruvnet/RuView.git
synced 2026-06-02 00:58:56 +02:00
feat: cross-node fusion + DynamicMinCut + RSSI tracking (v0.5.3)
* feat(server): cross-node RSSI-weighted feature fusion + benchmarks Adds fuse_multi_node_features() that combines CSI features across all active ESP32 nodes using RSSI-based weighting (closer node = higher weight). Benchmark results (2 ESP32 nodes, 30s, ~1500 frames): Metric | Baseline | Fusion | Improvement ---------------------|----------|---------|------------ Variance mean | 109.4 | 77.6 | -29% noise Variance std | 154.1 | 105.4 | -32% stability Confidence | 0.643 | 0.686 | +7% Keypoint spread std | 4.5 | 1.3 | -72% jitter Presence ratio | 93.4% | 94.6% | +1.3pp Person count still fluctuates near threshold — tracked as known issue. Verified on real hardware: COM6 (node 1) + COM9 (node 2) on ruv.net. Co-Authored-By: claude-flow <ruv@ruv.net> * fix(ui): add client-side lerp smoothing to pose renderer Keypoints now interpolate between frames (alpha=0.25) instead of jumping directly to new positions. This eliminates visual jitter that persists even with server-side EMA smoothing, because the renderer was drawing every WebSocket frame at full rate. Applied to skeleton, keypoints, and dense body rendering paths. Co-Authored-By: claude-flow <ruv@ruv.net> * feat: DynamicMinCut person separation + UI lerp smoothing - Added ruvector-mincut dependency to sensing server - Replaced variance-based person scoring with actual graph min-cut on subcarrier temporal correlation matrix (Pearson correlation edges, DynamicMinCut exact max-flow) - Recalibrated feature scaling for real ESP32 data ranges - UI: client-side lerp interpolation (alpha=0.25) on keypoint positions - Dampened procedural animation (noise, stride, extremity jitter) - Person count thresholds retuned for mincut ratio Co-Authored-By: claude-flow <ruv@ruv.net> * docs: update CHANGELOG with v0.5.1-v0.5.3 releases Co-Authored-By: claude-flow <ruv@ruv.net>
This commit is contained in:
rUv
@@ -0,0 +1 @@
|
||||
{"intelligence":7,"timestamp":1774922079152}
|
||||
@@ -5,6 +5,65 @@ All notable changes to this project will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [v0.5.3-esp32] — 2026-03-30
|
||||
|
||||
### Added
|
||||
- **Cross-node RSSI-weighted feature fusion** — Multiple ESP32 nodes fuse CSI features using RSSI-based weighting. Closer node gets higher weight. Reduces variance noise by 29%, keypoint jitter by 72%.
|
||||
- **DynamicMinCut person separation** — Uses `ruvector_mincut::DynamicMinCut` on the subcarrier temporal correlation graph to detect independent motion clusters. Replaces variance-based heuristic for multi-person counting.
|
||||
- **RSSI-based position tracking** — Skeleton position driven by RSSI differential between nodes. Walk between ESP32s and the skeleton follows you.
|
||||
- **Per-node state pipeline (ADR-068)** — Each ESP32 node gets independent `HashMap<u8, NodeState>` with frame history, classification, vitals, and person count. Fixes #249 (the #1 user-reported issue).
|
||||
- **RuVector Phase 1-3 integration** — Subcarrier importance weighting, temporal keypoint smoothing (EMA), coherence gating, skeleton kinematic constraints (Jakobsen relaxation), compressed pose history.
|
||||
- **Client-side lerp smoothing** — UI keypoints interpolate between frames (alpha=0.15) for fluid skeleton movement.
|
||||
- **Multi-node mesh tests** — 8 integration tests covering 1-255 node configurations.
|
||||
- **`wifi_densepose` Python package** — `from wifi_densepose import WiFiDensePose` now works (#314).
|
||||
|
||||
### Fixed
|
||||
- **Watchdog crash on busy LANs (#321)** — Batch-limited edge_dsp to 4 frames before 20ms yield. Fixed idle-path busy-spin (`pdMS_TO_TICKS(5)==0`).
|
||||
- **No detection from edge vitals (#323)** — Server now generates `sensing_update` from Tier 2+ vitals packets.
|
||||
- **RSSI byte offset mismatch (#332)** — Server parsed RSSI from wrong byte (was reading sequence counter).
|
||||
- **Stack overflow risk** — Moved 4KB of BPM scratch buffers from stack to static storage.
|
||||
- **Stale node memory leak** — `node_states` HashMap evicts nodes inactive >60s.
|
||||
- **Unsafe raw pointer removed** — Replaced with safe `.clone()` for adaptive model borrow.
|
||||
- **Firmware CI** — Upgraded to IDF v5.4, replaced `xxd` with `od` (#327).
|
||||
- **Person count double-counting** — Multi-node aggregation changed from `sum` to `max`.
|
||||
- **Skeleton jitter** — Removed tick-based noise, dampened procedural animation, recalibrated feature scaling for real ESP32 data.
|
||||
|
||||
### Changed
|
||||
- Motion-responsive skeleton: arm swing (0-80px) driven by CSI variance, leg kick (0-50px) by motion_band_power, vertical bob when walking.
|
||||
- Person count thresholds recalibrated for real ESP32 hardware (1→2 at 0.70, EMA alpha 0.04).
|
||||
- Vital sign filtering: larger median window (31), faster EMA (0.05), looser HR jump filter (15 BPM).
|
||||
- Vendored ruvector updated to v2.1.0-40 (316 commits ahead).
|
||||
|
||||
### Benchmarks (2-node mesh, COM6 + COM9, 30s)
|
||||
| Metric | Baseline | v0.5.3 | Improvement |
|
||||
|--------|----------|--------|-------------|
|
||||
| Variance noise | 109.4 | 77.6 | **-29%** |
|
||||
| Feature stability | std=154.1 | std=105.4 | **-32%** |
|
||||
| Keypoint jitter | std=4.5px | std=1.3px | **-72%** |
|
||||
| Confidence | 0.643 | 0.686 | **+7%** |
|
||||
| Presence accuracy | 93.4% | 94.6% | **+1.3pp** |
|
||||
|
||||
### Verified
|
||||
- Real hardware: COM6 (node 1) + COM9 (node 2) on ruv.net WiFi
|
||||
- All 284 Rust tests pass, 352 signal crate tests pass
|
||||
- Firmware builds clean at 843 KB
|
||||
- QEMU CI: 11/11 jobs green
|
||||
|
||||
## [v0.5.2-esp32] — 2026-03-28
|
||||
|
||||
### Fixed
|
||||
- RSSI byte offset in frame parser (#332)
|
||||
- Per-node state pipeline for multi-node sensing (#249)
|
||||
- Firmware CI upgraded to IDF v5.4 (#327)
|
||||
|
||||
## [v0.5.1-esp32] — 2026-03-27
|
||||
|
||||
### Fixed
|
||||
- Watchdog crash on busy LANs (#321)
|
||||
- No detection from edge vitals (#323)
|
||||
- `wifi_densepose` Python package import (#314)
|
||||
- Pre-compiled firmware binaries added to release
|
||||
|
||||
## [v0.5.0-esp32] — 2026-03-15
|
||||
|
||||
### Added
|
||||
|
||||
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
Binary file not shown.
File diff suppressed because one or more lines are too long
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,33 @@
|
||||
# ESP32-S3 CSI Node — Default SDK Configuration
|
||||
# This file is applied automatically by idf.py when no sdkconfig exists.
|
||||
|
||||
# Target: ESP32-S3
|
||||
CONFIG_IDF_TARGET="esp32s3"
|
||||
|
||||
# Use custom partition table (8MB flash with OTA — ADR-045)
|
||||
CONFIG_PARTITION_TABLE_CUSTOM=y
|
||||
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions_display.csv"
|
||||
|
||||
# Flash configuration: 8MB (Quad SPI)
|
||||
CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y
|
||||
CONFIG_ESPTOOLPY_FLASHSIZE="8MB"
|
||||
|
||||
# Compiler optimization: optimize for size to reduce binary
|
||||
CONFIG_COMPILER_OPTIMIZATION_SIZE=y
|
||||
|
||||
# Enable CSI (Channel State Information) in WiFi driver
|
||||
CONFIG_ESP_WIFI_CSI_ENABLED=y
|
||||
|
||||
# NVS encryption disabled by default (requires eFuse provisioning).
|
||||
# Enable only after burning HMAC key to eFuse block.
|
||||
# CONFIG_NVS_ENCRYPTION is not set
|
||||
|
||||
# Disable unused features to reduce binary size
|
||||
CONFIG_BOOTLOADER_LOG_LEVEL_WARN=y
|
||||
CONFIG_LOG_DEFAULT_LEVEL_INFO=y
|
||||
|
||||
# LWIP: enable extended socket options for UDP multicast
|
||||
CONFIG_LWIP_SO_RCVBUF=y
|
||||
|
||||
# FreeRTOS: increase task stack for CSI processing
|
||||
CONFIG_ESP_MAIN_TASK_STACK_SIZE=8192
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1 @@
|
||||
{"intelligence":35,"timestamp":1774903706609}
|
||||
@@ -43,5 +43,8 @@ clap = { workspace = true }
|
||||
# Multi-BSSID WiFi scanning pipeline (ADR-022 Phase 3)
|
||||
wifi-densepose-wifiscan = { version = "0.3.0", path = "../wifi-densepose-wifiscan" }
|
||||
|
||||
# RuVector graph min-cut for person separation (ADR-068)
|
||||
ruvector-mincut = { workspace = true }
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = "3.10"
|
||||
|
||||
@@ -17,6 +17,7 @@ mod vital_signs;
|
||||
use wifi_densepose_sensing_server::{graph_transformer, trainer, dataset, embedding};
|
||||
|
||||
use std::collections::{HashMap, VecDeque};
|
||||
use ruvector_mincut::{DynamicMinCut, MinCutBuilder};
|
||||
use std::net::SocketAddr;
|
||||
use std::path::PathBuf;
|
||||
use std::sync::Arc;
|
||||
@@ -2054,27 +2055,137 @@ fn fuse_multi_node_features(
|
||||
/// Returns a raw score (0.0..1.0) that the caller converts to person count
|
||||
/// after temporal smoothing.
|
||||
fn compute_person_score(feat: &FeatureInfo) -> f64 {
|
||||
// Normalize each feature to [0, 1] using calibrated ranges:
|
||||
//
|
||||
// variance: intra-frame amp variance. 1-person ~2-15, 2-person ~15-60,
|
||||
// real ESP32 can go higher. Use 30.0 as scaling midpoint.
|
||||
let var_norm = (feat.variance / 30.0).clamp(0.0, 1.0);
|
||||
|
||||
// change_points: threshold crossings in 56 subcarriers. 1-person ~5-15,
|
||||
// 2-person ~15-30. Scale by 30.0 (half of max 55).
|
||||
// Normalize each feature to [0, 1] using ranges calibrated from real
|
||||
// ESP32 hardware (COM6/COM9 on ruv.net, March 2026).
|
||||
let var_norm = (feat.variance / 300.0).clamp(0.0, 1.0);
|
||||
let cp_norm = (feat.change_points as f64 / 30.0).clamp(0.0, 1.0);
|
||||
|
||||
// motion_band_power: upper-half subcarrier variance. 1-person ~1-8,
|
||||
// 2-person ~8-25. Scale by 20.0.
|
||||
let motion_norm = (feat.motion_band_power / 20.0).clamp(0.0, 1.0);
|
||||
|
||||
// spectral_power: mean squared amplitude. Highly variable (~100-1000+).
|
||||
// Use relative change indicator: high spectral_power with high variance
|
||||
// suggests multiple reflectors. Scale by 500.0.
|
||||
let motion_norm = (feat.motion_band_power / 250.0).clamp(0.0, 1.0);
|
||||
let sp_norm = (feat.spectral_power / 500.0).clamp(0.0, 1.0);
|
||||
var_norm * 0.40 + cp_norm * 0.20 + motion_norm * 0.25 + sp_norm * 0.15
|
||||
}
|
||||
|
||||
// Weighted composite — variance and change_points carry the most signal.
|
||||
var_norm * 0.35 + cp_norm * 0.30 + motion_norm * 0.20 + sp_norm * 0.15
|
||||
/// Estimate person count via ruvector DynamicMinCut on the subcarrier
|
||||
/// temporal correlation graph.
|
||||
///
|
||||
/// Builds a graph where:
|
||||
/// - Nodes = active subcarriers (variance > noise floor)
|
||||
/// - Edges = Pearson correlation between subcarrier time series
|
||||
/// (weight = correlation coefficient; high correlation = heavy edge)
|
||||
/// - Source = virtual node connected to the most active subcarrier
|
||||
/// - Sink = virtual node connected to the least correlated subcarrier
|
||||
///
|
||||
/// The min-cut value indicates how many independent motion clusters exist:
|
||||
/// - High min-cut (relative to total edge weight) → one tightly coupled
|
||||
/// group → 1 person
|
||||
/// - Low min-cut → two loosely coupled groups → 2 persons
|
||||
///
|
||||
/// Uses `ruvector_mincut::DynamicMinCut` for O(V²E) exact max-flow.
|
||||
fn estimate_persons_from_correlation(frame_history: &VecDeque<Vec<f64>>) -> usize {
|
||||
let n_frames = frame_history.len();
|
||||
if n_frames < 10 {
|
||||
return 1;
|
||||
}
|
||||
|
||||
let window: Vec<&Vec<f64>> = frame_history.iter().rev().take(20).collect();
|
||||
let n_sub = window[0].len().min(56);
|
||||
if n_sub < 4 {
|
||||
return 1;
|
||||
}
|
||||
let k = window.len() as f64;
|
||||
|
||||
// Per-subcarrier mean and variance
|
||||
let mut means = vec![0.0f64; n_sub];
|
||||
let mut variances = vec![0.0f64; n_sub];
|
||||
for frame in &window {
|
||||
for sc in 0..n_sub.min(frame.len()) {
|
||||
means[sc] += frame[sc] / k;
|
||||
}
|
||||
}
|
||||
for frame in &window {
|
||||
for sc in 0..n_sub.min(frame.len()) {
|
||||
variances[sc] += (frame[sc] - means[sc]).powi(2) / k;
|
||||
}
|
||||
}
|
||||
|
||||
// Active subcarriers: variance above noise floor
|
||||
let noise_floor = 1.0;
|
||||
let active: Vec<usize> = (0..n_sub).filter(|&sc| variances[sc] > noise_floor).collect();
|
||||
let m = active.len();
|
||||
if m < 3 {
|
||||
return if m == 0 { 0 } else { 1 };
|
||||
}
|
||||
|
||||
// Build correlation graph edges between active subcarriers.
|
||||
// Edge weight = |Pearson correlation|. High correlation → same person.
|
||||
let mut edges: Vec<(u64, u64, f64)> = Vec::new();
|
||||
let source = m as u64;
|
||||
let sink = (m + 1) as u64;
|
||||
|
||||
// Precompute std devs
|
||||
let stds: Vec<f64> = active.iter().map(|&sc| variances[sc].sqrt().max(1e-9)).collect();
|
||||
|
||||
for i in 0..m {
|
||||
for j in (i + 1)..m {
|
||||
// Pearson correlation between subcarriers i and j
|
||||
let mut cov = 0.0f64;
|
||||
for frame in &window {
|
||||
let si = active[i];
|
||||
let sj = active[j];
|
||||
if si < frame.len() && sj < frame.len() {
|
||||
cov += (frame[si] - means[si]) * (frame[sj] - means[sj]) / k;
|
||||
}
|
||||
}
|
||||
let corr = (cov / (stds[i] * stds[j])).abs();
|
||||
if corr > 0.1 {
|
||||
// Bidirectional edges for flow network
|
||||
let weight = corr * 10.0; // Scale up for integer-like flow
|
||||
edges.push((i as u64, j as u64, weight));
|
||||
edges.push((j as u64, i as u64, weight));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Source → highest-variance subcarrier, Sink → lowest-variance
|
||||
let (max_var_idx, _) = active.iter().enumerate()
|
||||
.max_by(|(_, &a), (_, &b)| variances[a].partial_cmp(&variances[b]).unwrap())
|
||||
.unwrap_or((0, &0));
|
||||
let (min_var_idx, _) = active.iter().enumerate()
|
||||
.min_by(|(_, &a), (_, &b)| variances[a].partial_cmp(&variances[b]).unwrap())
|
||||
.unwrap_or((0, &0));
|
||||
|
||||
if max_var_idx == min_var_idx {
|
||||
return 1;
|
||||
}
|
||||
|
||||
edges.push((source, max_var_idx as u64, 100.0));
|
||||
edges.push((min_var_idx as u64, sink, 100.0));
|
||||
|
||||
// Run min-cut
|
||||
let mc: DynamicMinCut = match MinCutBuilder::new().exact().with_edges(edges.clone()).build() {
|
||||
Ok(mc) => mc,
|
||||
Err(_) => return 1,
|
||||
};
|
||||
|
||||
let cut_value = mc.min_cut_value();
|
||||
let total_edge_weight: f64 = edges.iter()
|
||||
.filter(|(s, t, _)| *s != source && *s != sink && *t != source && *t != sink)
|
||||
.map(|(_, _, w)| w)
|
||||
.sum::<f64>() / 2.0; // bidirectional → halve
|
||||
|
||||
if total_edge_weight < 1e-9 {
|
||||
return 1;
|
||||
}
|
||||
|
||||
// Normalized cut ratio: low = easy to split = multiple people
|
||||
let cut_ratio = cut_value / total_edge_weight;
|
||||
|
||||
if cut_ratio > 0.4 {
|
||||
1 // Tightly coupled — one person
|
||||
} else if cut_ratio > 0.15 {
|
||||
2 // Moderately separable — two people
|
||||
} else {
|
||||
3 // Highly separable — three+ people
|
||||
}
|
||||
}
|
||||
|
||||
/// Convert smoothed person score to discrete count with hysteresis.
|
||||
@@ -2092,9 +2203,9 @@ fn score_to_person_count(smoothed_score: f64, prev_count: usize) -> usize {
|
||||
// 3→2: 0.78 (hysteresis gap of 0.14)
|
||||
match prev_count {
|
||||
0 | 1 => {
|
||||
if smoothed_score > 0.92 {
|
||||
if smoothed_score > 0.85 {
|
||||
3
|
||||
} else if smoothed_score > 0.80 {
|
||||
} else if smoothed_score > 0.70 {
|
||||
2
|
||||
} else {
|
||||
1
|
||||
@@ -3473,10 +3584,10 @@ async fn udp_receiver_task(state: SharedState, udp_port: u16) {
|
||||
let vitals = smooth_vitals_node(ns, &raw_vitals);
|
||||
ns.latest_vitals = vitals.clone();
|
||||
|
||||
let raw_score = compute_person_score(&features);
|
||||
// Slower EMA (0.05) for person score to prevent count flips
|
||||
// from frame-to-frame variance oscillation in fused features.
|
||||
ns.smoothed_person_score = ns.smoothed_person_score * 0.95 + raw_score * 0.05;
|
||||
// DynamicMinCut person estimation from subcarrier correlation.
|
||||
let corr_persons = estimate_persons_from_correlation(&ns.frame_history);
|
||||
let raw_score = corr_persons as f64 / 3.0;
|
||||
ns.smoothed_person_score = ns.smoothed_person_score * 0.92 + raw_score * 0.08;
|
||||
if classification.presence {
|
||||
let count = score_to_person_count(ns.smoothed_person_score, ns.prev_person_count);
|
||||
ns.prev_person_count = count;
|
||||
|
||||
+1
@@ -0,0 +1 @@
|
||||
{"intelligence":60,"timestamp":1774039923051}
|
||||
@@ -56,10 +56,47 @@ export class PoseRenderer {
|
||||
[11, 13], [12, 14], [13, 15], [14, 16] // Legs
|
||||
];
|
||||
|
||||
// Client-side keypoint smoothing: lerp between frames to reduce jitter.
|
||||
// Maps person index → array of {x, y} for each keypoint.
|
||||
this._smoothedKeypoints = new Map();
|
||||
this._lerpAlpha = 0.25; // 0 = frozen, 1 = instant (no smoothing)
|
||||
|
||||
// Initialize rendering context
|
||||
this.initializeContext();
|
||||
}
|
||||
|
||||
// Lerp a single value toward target
|
||||
_lerp(current, target, alpha) {
|
||||
return current + (target - current) * alpha;
|
||||
}
|
||||
|
||||
// Get smoothed keypoint positions for a person
|
||||
_getSmoothedKeypoints(personIdx, keypoints) {
|
||||
if (!this.config.enableSmoothing || !keypoints || keypoints.length === 0) {
|
||||
return keypoints;
|
||||
}
|
||||
|
||||
let prev = this._smoothedKeypoints.get(personIdx);
|
||||
if (!prev || prev.length !== keypoints.length) {
|
||||
// First frame or keypoint count changed — initialize
|
||||
prev = keypoints.map(kp => ({ x: kp.x, y: kp.y, z: kp.z || 0, confidence: kp.confidence, name: kp.name }));
|
||||
this._smoothedKeypoints.set(personIdx, prev);
|
||||
return keypoints;
|
||||
}
|
||||
|
||||
const alpha = this._lerpAlpha;
|
||||
const smoothed = keypoints.map((kp, i) => ({
|
||||
...kp,
|
||||
x: this._lerp(prev[i].x, kp.x, alpha),
|
||||
y: this._lerp(prev[i].y, kp.y, alpha),
|
||||
}));
|
||||
|
||||
// Update stored positions
|
||||
this._smoothedKeypoints.set(personIdx, smoothed.map(kp => ({ x: kp.x, y: kp.y, z: kp.z || 0, confidence: kp.confidence, name: kp.name })));
|
||||
|
||||
return smoothed;
|
||||
}
|
||||
|
||||
createLogger() {
|
||||
return {
|
||||
debug: (...args) => console.debug('[RENDERER-DEBUG]', new Date().toISOString(), ...args),
|
||||
@@ -150,18 +187,17 @@ export class PoseRenderer {
|
||||
return; // Skip low confidence detections
|
||||
}
|
||||
|
||||
console.log(`✅ [RENDERER] Rendering person ${index} with confidence: ${person.confidence}`);
|
||||
// Apply client-side lerp smoothing to reduce visual jitter
|
||||
const smoothedKps = this._getSmoothedKeypoints(index, person.keypoints);
|
||||
|
||||
// Render skeleton connections
|
||||
if (this.config.showSkeleton && person.keypoints) {
|
||||
console.log(`🦴 [RENDERER] Rendering skeleton for person ${index}`);
|
||||
this.renderSkeleton(person.keypoints, person.confidence);
|
||||
if (this.config.showSkeleton && smoothedKps) {
|
||||
this.renderSkeleton(smoothedKps, person.confidence);
|
||||
}
|
||||
|
||||
// Render keypoints
|
||||
if (this.config.showKeypoints && person.keypoints) {
|
||||
console.log(`🔴 [RENDERER] Rendering keypoints for person ${index}`);
|
||||
this.renderKeypoints(person.keypoints, person.confidence);
|
||||
if (this.config.showKeypoints && smoothedKps) {
|
||||
this.renderKeypoints(smoothedKps, person.confidence);
|
||||
}
|
||||
|
||||
// Render bounding box
|
||||
@@ -265,7 +301,7 @@ export class PoseRenderer {
|
||||
persons.forEach((person, personIdx) => {
|
||||
if (person.confidence < this.config.confidenceThreshold || !person.keypoints) return;
|
||||
|
||||
const kps = person.keypoints;
|
||||
const kps = this._getSmoothedKeypoints(personIdx, person.keypoints);
|
||||
|
||||
bodyParts.forEach((part) => {
|
||||
// Collect valid keypoints for this body part
|
||||
|
||||
Reference in New Issue
Block a user