mirror of
https://github.com/ruvnet/RuView.git
synced 2026-06-02 00:58:56 +02:00
3314c8db8d
* feat(cog-pose-estimation): scaffold first Cog from this repo (ADR-100 + ADR-101) Adds the foundation for the pose-estimation Cog that ships from this repo into Cognitum V0 appliances. Companion ADR-225 + crate land in cognitum-one/v0-appliance. ADRs: * ADR-100 formalises the Cognitum Cog packaging spec — on-device layout under /var/lib/cognitum/apps/<id>/, manifest.json schema (incl. new binary_sha256 + binary_signature fields), GCS hosting convention, repo source layout, build pipeline, and the four-verb runtime contract (version | manifest | health | run). Documents the convention I reverse-engineered from inspecting installed cogs on a live cognitum-v0 appliance — `anomaly-detect`, `presence`, `seizure-detect`, etc. * ADR-101 designs the pose-estimation Cog itself: where it sits in the wifi-densepose pipeline (encoder init from ruvnet/wifi-densepose-pretrained, 17-keypoint regression head), what gets shipped per target arch (arm / x86_64 / hailo8 / hailo10), acceptance gates (PCK@20 explicitly deferred to #640 — this ADR ships the vehicle, not the accuracy). Crate v2/crates/cog-pose-estimation/: * Cargo.toml + workspace member declaration with a hailo feature gate so the binary builds without the Hailo SDK in CI. * main.rs implements the four-verb CLI exactly per ADR-100. * config.rs / manifest.rs / publisher.rs / inference.rs / runtime.rs — small modules, each <100 lines. * publisher.rs emits ADR-100 structured JSON events. * inference.rs is a stub that produces a centred-skeleton baseline with confidence=0 (honest: no trained weights wired in yet). * runtime.rs subscribes to /api/v1/sensing/latest, slides a 56*20 window, runs the engine, emits pose.frame events. * cog/manifest.template.json + cog/config.schema.json define the release artifact + runtime config schemas. * cog/Makefile holds build / sign / upload targets. * tests/smoke.rs covers manifest roundtrip + engine I/O surface. Verified locally: * cargo check -p cog-pose-estimation: clean. * cargo test -p cog-pose-estimation: 4/4 pass. * ./target/release/cog-pose-estimation {version,manifest,health}: all emit the right contract output. This commit contains scaffolding only; the actual trained weights and Hailo HEF cross-compile come in follow-ups tracked in #640 and the companion v0-appliance branch. * feat(cog-pose-estimation): first measured run — Candle CUDA on RTX 5080 Trained pose_v1 on ruvultra (RTX 5080) via Candle 0.9 + cuda feature against the same 1,077-sample paired session that produced 0%/0% PCK in #640 with the pure-JS SPSA trainer. First real numbers: PCK@20 = 3.0% (up from 0.0%) PCK@50 = 18.5% (up from 0.0%) MPJPE = 0.093 (down from 0.66, ~7x improvement) 400 epochs in 2.1 s wall time, full-batch, ~5 ms/epoch. Loss curve 0.181 -> 0.014 over the run, eval 0.010. Per-joint reveals the model leans on right-side proximal joints (r_hip 77% PCK@50, r_knee 35%, l_elbow 26%) — consistent with the camera framing in the source recording. Distal joints (wrists, ankles) and face joints are still near-random, consistent with the 56-subcarrier / 20-frame input not carrying fine-grained spatial info at 1077 samples. This commit: * Adds v2/crates/cog-pose-estimation/cog/artifacts/{pose_v1.safetensors, train_results.json} so the cog dir now contains a real reference artifact, not just scaffold. * Updates cog/README.md "Status" block with the measured numbers, per-joint table, and an honest reading of where the model succeeds vs where the data is the bottleneck. * Adds docs/benchmarks/pose-estimation-cog.md as the canonical benchmark log — append-only, one section per published run. * Appends a "First measured run" section to ADR-101 referencing the new benchmark file. Still pending in the follow-up: * Wire pose_v1.safetensors into src/inference.rs (replace stub). * ONNX export (Candle lacks a writer — needs external conversion). * Hailo HEF cross-compile + cluster deploy. The data-bound gap to PCK@20 >= 35% is tracked in #640. * feat(cog-pose-estimation): wire real weights — cog is no longer a stub Replaces the centred-skeleton stub in src/inference.rs with a real Candle-based loader that reads cog/artifacts/pose_v1.safetensors and runs the trained Conv1d encoder + MLP pose head on every incoming CSI window. What changes: * src/inference.rs: PoseNet mirrors the training script's architecture exactly — Conv1d(56->64, k=3 d=1), Conv1d(64->128, k=3 d=2), Conv1d(128->128, k=3 d=4), mean over time, Linear(128->256)+ReLU, Linear(256->34)+sigmoid -> reshape [17, 2]. The InferenceEngine searches a sensible candidate list for the weights file (/var/lib/cognitum/apps/pose-estimation/, ./pose_v1.safetensors, ./cog/artifacts/, repo-root, v2/-relative) and falls back to the stub when none are present so the cog still satisfies ADR-100. * Cargo.toml: adds candle-core 0.9 + candle-nn 0.9 (no-default-features, CPU build by default) + safetensors 0.4. New `cuda` feature opt-in for GPU inference on hosts that have it. Drops the unused wifi-densepose-train path dep from the default build path. * src/main.rs + src/publisher.rs: health.ok event now carries `backend` (candle-cuda | candle-cpu | stub) and the synthetic output confidence, so operators can tell at a glance whether the cog loaded its weights or fell back to the stub. * tests/smoke.rs: adds `real_weights_load_when_available` which asserts the loaded engine reports backend=candle-* and emits non-zero confidence — exactly the signal that proves we're not silently degrading to the stub. Verified locally: * `cargo check -p cog-pose-estimation --no-default-features` — clean * `cargo test -p cog-pose-estimation --no-default-features` — 5/5 pass * `./target/release/cog-pose-estimation health` emits: {"event":"health.ok","fields":{"backend":"candle-cpu","cog":"pose-estimation","synthetic_output_confidence":0.185}} — 0.185 is the published PCK@50 from cog/artifacts/train_results.json, emitted by the real Candle inference path (would be 0.0 if it had fallen back to the stub). The cog now runs the trained pose_v1 model end-to-end. Accuracy is still bounded by the underlying 1077-sample training data (PCK@20 3.0%, PCK@50 18.5% per docs/benchmarks/pose-estimation-cog.md) — that gap is data-bound and tracked in #640. ONNX export + Hailo HEF cross-compile remain follow-ups. * docs(benchmarks): measure cog-pose-estimation cold-start latency 100 sequential `cog-pose-estimation health` invocations average 76.2 ms each on a Windows x86_64 host using the `candle-cpu` backend. Each invocation re-loads pose_v1.safetensors and runs one synthetic forward pass, so this is the worst-case cold-start path. Long-running `run` inference will be sub-millisecond per frame once the model is loaded. Updates the benchmarks doc accordingly. * feat(cog-pose-estimation): ONNX export — pose_v1.onnx + scripts/export-onnx.py Adds the canonical ONNX artifact that unblocks downstream Hailo HEF cross-compile + ONNX Runtime benchmarks. Generated on ruvultra (torch 2.12.0 + CUDA), 12,059 bytes, opset 18, dynamic batch axis. * scripts/export-onnx.py: mirrors the Candle inference architecture in PyTorch (Conv1d 56->64, 64->128, 128->128 + Linear 128->256->34), pure- python safetensors loader (no extra pip dep), exports via torch.onnx.export, then verifies via onnx.checker.check_model and numerical parity against the torch reference. * Verified parity vs torch: max |torch - onnx| = 8.94e-8 (1e-5 threshold). Effectively bit-perfect. * v2/crates/cog-pose-estimation/cog/artifacts/pose_v1.onnx — the artifact itself, 12 KB. * docs/benchmarks/pose-estimation-cog.md — adds an ONNX export section with the verification numbers. Next: Hailo HEF cross-compile (still gated on Hailo SDK on a self-hosted runner) and ONNX Runtime latency benchmarks on each target arch. * feat(cog-pose-estimation): release v0.0.1 — signed aarch64 binary on GCS End-to-end deploy: cross-compiled to aarch64-unknown-linux-gnu on ruvultra, ran via qemu-aarch64-static, then smoke-tested on a real cognitum-v0 Pi 5. Signed with COGNITUM_OWNER_SIGNING_KEY (Ed25519) and uploaded to gs://cognitum-apps/cogs/arm/. Real-hardware results on cognitum-v0 (Pi 5): health: backend=candle-cpu, confidence=0.185, real weights loaded 30x sequential `health`: 0.251 s total -> 8.4 ms / invocation (cold) GCS release artifacts (publicly downloadable): binary: 3,741,976 bytes sha256 1e1a7d3dd01ca05d5bfc5dbb142a5941b7866ed9f3224a21edc04d3f09a99bf5 weights: 507,032 bytes sha256 eb249b9a6b2e10130437a10976ed0230b0d085f86a0553d7226e1ae6eae4b9e5 signature (Ed25519, b64): LUN7xqLPYD3MFzm5dKB5MnYU0LvoRtek5ci5KiKPHBg+Xo6xuazwokn2Dw2JPMaLYJzmWn/SpT4djuR7hYvVDw== Adds: * v2/crates/cog-pose-estimation/cog/artifacts/manifest.json — the release-pipeline-produced manifest with all fields filled in per ADR-100, including arch, target_triple, signature, and a build_metadata block carrying the validation PCK numbers. * docs/benchmarks/pose-estimation-cog.md — new sections covering the real Pi 5 smoke (8.4 ms cold-start) and the signed GCS release artifacts. Verified by downloading the binary anonymously from GCS and re-computing the sha256 — matches the locally-computed sha exactly. Signature decoded to the expected 64-byte Ed25519 length. Closes the GCS-upload acceptance criterion from ADR-100; the only pending work is Hailo HEF cross-compile (still SDK-gated) and an x86_64 release alongside this arm release. * docs(benchmarks): record live cognitum-v0 install + 5-sec smoke run Adds the "Live appliance install" section documenting what happened when the signed v0.0.1 binary + weights were installed under /var/lib/cognitum/apps/pose-estimation/ on cognitum-v0 (the V0 cluster leader). * Layout matches the existing anomaly-detect / presence / seizure- detect cogs exactly — the Cogs dashboard at http://cognitum-v0:9000/cogs auto-discovers entries. * `cog-pose-estimation run` ran for 5 seconds in the background and cleanly emitted run.started + structured WARN events for the missing local sensing-server on :3000 (cognitum-v0's actual CSI source is ruview-vitals-worker on :50054, not :3000). No crashes, no NaN, no leaks. * Wiring `sensing_url` to the appliance-native source is a separate Day-2 integration task.
144 lines
4.7 KiB
Python
144 lines
4.7 KiB
Python
#!/usr/bin/env python3
|
|
"""Export pose_v1.safetensors -> pose_v1.onnx.
|
|
|
|
Builds the same architecture as v2/crates/cog-pose-estimation/src/inference.rs
|
|
in PyTorch, loads the trained weights from safetensors, and runs a torch.onnx
|
|
export with a fixed [1, 56, 20] input. Then verifies the ONNX loads and
|
|
matches the torch output to within 1e-5.
|
|
"""
|
|
|
|
import json
|
|
import struct
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import numpy as np
|
|
import torch
|
|
import torch.nn as nn
|
|
|
|
|
|
N_SUB = 56
|
|
N_FRAMES = 20
|
|
N_KP = 17
|
|
|
|
|
|
class PoseNet(nn.Module):
|
|
"""Mirrors inference.rs::PoseNet exactly."""
|
|
|
|
def __init__(self) -> None:
|
|
super().__init__()
|
|
self.c1 = nn.Conv1d(N_SUB, 64, kernel_size=3, padding=1, dilation=1)
|
|
self.c2 = nn.Conv1d(64, 128, kernel_size=3, padding=2, dilation=2)
|
|
self.c3 = nn.Conv1d(128, 128, kernel_size=3, padding=4, dilation=4)
|
|
self.fc1 = nn.Linear(128, 256)
|
|
self.fc2 = nn.Linear(256, N_KP * 2)
|
|
|
|
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
|
# x: [B, 56, 20]
|
|
h = torch.relu(self.c1(x))
|
|
h = torch.relu(self.c2(h))
|
|
h = torch.relu(self.c3(h))
|
|
h = h.mean(dim=2) # [B, 128]
|
|
h = torch.relu(self.fc1(h))
|
|
h = torch.sigmoid(self.fc2(h))
|
|
return h
|
|
|
|
|
|
def load_safetensors(path: Path) -> dict[str, torch.Tensor]:
|
|
"""Pure-python safetensors reader. Avoids the safetensors pip dep."""
|
|
with path.open("rb") as f:
|
|
header_len = struct.unpack("<Q", f.read(8))[0]
|
|
header = json.loads(f.read(header_len).decode("utf-8"))
|
|
out: dict[str, torch.Tensor] = {}
|
|
for name, meta in header.items():
|
|
if name == "__metadata__":
|
|
continue
|
|
start, end = meta["data_offsets"]
|
|
shape = meta["shape"]
|
|
dtype = meta["dtype"]
|
|
assert dtype == "F32", f"unsupported dtype {dtype} for {name}"
|
|
f.seek(8 + header_len + start)
|
|
buf = f.read(end - start)
|
|
arr = np.frombuffer(buf, dtype=np.float32).copy().reshape(shape)
|
|
out[name] = torch.from_numpy(arr)
|
|
return out
|
|
|
|
|
|
def main() -> None:
|
|
weights_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("pose_v1.safetensors")
|
|
out_path = Path(sys.argv[2]) if len(sys.argv) > 2 else Path("pose_v1.onnx")
|
|
|
|
if not weights_path.exists():
|
|
raise SystemExit(f"weights file not found: {weights_path}")
|
|
|
|
print(f"reading {weights_path}")
|
|
tensors = load_safetensors(weights_path)
|
|
print(f" found {len(tensors)} tensors: {sorted(tensors.keys())}")
|
|
|
|
model = PoseNet()
|
|
# Map safetensors names (enc.c1.weight, head.fc1.weight, ...) to module params
|
|
mapping = {
|
|
"enc.c1.weight": "c1.weight",
|
|
"enc.c1.bias": "c1.bias",
|
|
"enc.c2.weight": "c2.weight",
|
|
"enc.c2.bias": "c2.bias",
|
|
"enc.c3.weight": "c3.weight",
|
|
"enc.c3.bias": "c3.bias",
|
|
"head.fc1.weight": "fc1.weight",
|
|
"head.fc1.bias": "fc1.bias",
|
|
"head.fc2.weight": "fc2.weight",
|
|
"head.fc2.bias": "fc2.bias",
|
|
}
|
|
state = {dst: tensors[src] for src, dst in mapping.items()}
|
|
model.load_state_dict(state)
|
|
model.eval()
|
|
print(" weights loaded into PyTorch model")
|
|
|
|
# Sanity check forward
|
|
x = torch.zeros(1, N_SUB, N_FRAMES)
|
|
with torch.no_grad():
|
|
y = model(x)
|
|
print(f" zero-input forward: shape={tuple(y.shape)} sample={y[0, :4].tolist()}")
|
|
|
|
# Export to ONNX
|
|
torch.onnx.export(
|
|
model,
|
|
x,
|
|
out_path,
|
|
export_params=True,
|
|
opset_version=18,
|
|
do_constant_folding=True,
|
|
input_names=["csi_window"],
|
|
output_names=["keypoints"],
|
|
dynamic_axes={"csi_window": {0: "batch"}, "keypoints": {0: "batch"}},
|
|
)
|
|
print(f" wrote {out_path} ({out_path.stat().st_size} bytes)")
|
|
|
|
# Verify the ONNX file loads + matches torch output
|
|
try:
|
|
import onnx
|
|
import onnxruntime as ort
|
|
|
|
onnx_model = onnx.load(str(out_path))
|
|
onnx.checker.check_model(onnx_model)
|
|
print(" ONNX model checker: ok")
|
|
|
|
sess = ort.InferenceSession(str(out_path), providers=["CPUExecutionProvider"])
|
|
rng = np.random.default_rng(42)
|
|
x_np = rng.standard_normal((1, N_SUB, N_FRAMES), dtype=np.float32)
|
|
with torch.no_grad():
|
|
y_torch = model(torch.from_numpy(x_np)).numpy()
|
|
y_onnx = sess.run(["keypoints"], {"csi_window": x_np})[0]
|
|
max_abs = float(np.max(np.abs(y_torch - y_onnx)))
|
|
print(f" parity vs torch: max |torch - onnx| = {max_abs:.2e}")
|
|
assert max_abs < 1e-5, "ONNX output diverges from torch output"
|
|
print(" parity ok (<1e-5)")
|
|
except ImportError as e:
|
|
print(f" WARN: onnx/onnxruntime not installed, skipping verification: {e}")
|
|
|
|
print("\nDone.")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|