mirror of
https://github.com/ruvnet/RuView.git
synced 2026-06-02 00:58:56 +02:00
rUv
4b1a835107
Issue #640 (PCK gap follow-up) was deleted upstream after the cog v0.0.1 PRs landed today. Re-opened as #645 with the same context plus the new measured v0.0.1 numbers (PCK@20 3.0%, PCK@50 18.5%, MPJPE 0.093). This patch updates the three files in main that still pointed at the dead #640 to point at #645 instead — ADR-101, the cog README, and the benchmark log.
177 lines
9.4 KiB
Markdown
177 lines
9.4 KiB
Markdown
# `cog-pose-estimation` — Benchmark Log
|
||
|
||
This file tracks every published benchmark for the pose-estimation Cog. New runs append; never overwrite history. Per ADR-101 §"Acceptance gates".
|
||
|
||
## v0.0.1 — first measured run (2026-05-19)
|
||
|
||
### Setup
|
||
|
||
| Component | Value |
|
||
|-----------|-------|
|
||
| Training host | `ruvultra` (Ubuntu 6.17, x86_64, RTX 5080) |
|
||
| Backend | `candle-core 0.9` with `cuda` feature |
|
||
| Data | `data/paired/wiflow-p7-1779210883.paired.jsonl` — 1,077 paired samples, 30-min seated-at-desk recording, avg conf 0.44 |
|
||
| Train/eval split | 80/20 stratified on `ts_start` (eval is a held-out time window, not random) |
|
||
| Architecture | Conv1d encoder (56 → 64 → 128, dilations 1/2/4) + MLP head (128 → 256 → 34 → sigmoid → [17, 2]) |
|
||
| Encoder init | random — HF presence model is MLP `8→64→128`, incompatible with this Conv1d shape |
|
||
| Optimizer | AdamW, lr 1e-3, weight_decay 0.01 |
|
||
| LR schedule | Cosine with 50-epoch warm restarts |
|
||
| Loss | SmoothL1 (Huber β=0.1), confidence-weighted by `record.conf` |
|
||
| Augmentation | Subcarrier dropout 10% (final 50 epochs) |
|
||
| Epochs | 400 (full-batch) |
|
||
| Wall time | **2.1 s** total |
|
||
|
||
### Accuracy
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| **PCK@20** (overall) | **3.0%** |
|
||
| **PCK@50** (overall) | **18.5%** |
|
||
| **MPJPE** (normalized) | **0.0931** |
|
||
| Final eval loss | 0.0101 |
|
||
| Loss reduction | 0.181 → 0.014 (13×) |
|
||
|
||
### Per-joint PCK
|
||
|
||
| Joint | PCK@20 | PCK@50 | | Joint | PCK@20 | PCK@50 |
|
||
|-------|-------:|-------:|--|-------|-------:|-------:|
|
||
| nose | 0.5% | 5.1% | | l_hip | 0.0% | 27.3% |
|
||
| l_eye | 2.8% | 8.3% | | **r_hip** | **25.0%** | **76.9%** |
|
||
| r_eye | 1.9% | 15.7% | | l_knee | 2.3% | 20.8% |
|
||
| l_ear | 0.0% | 3.2% | | r_knee | 0.9% | 35.2% |
|
||
| r_ear | 1.9% | 9.7% | | l_ankle | 1.4% | 7.9% |
|
||
| l_shoulder | 4.6% | 8.8% | | r_ankle | 0.9% | 9.3% |
|
||
| r_shoulder | 1.9% | 19.9% | | l_elbow | 1.9% | 26.4% |
|
||
| l_wrist | 3.2% | 24.1% | | r_elbow | 0.0% | 4.2% |
|
||
| r_wrist | 1.4% | 12.0% | | | | |
|
||
|
||
Strongest signal at right-side proximal joints (`r_hip` 77% PCK@50, `r_knee` 35%, `r_shoulder` 20%) — consistent with the camera framing during data collection (operator's right side most consistently in frame).
|
||
|
||
### Comparison to prior baseline
|
||
|
||
| Run | Backend | Train time | PCK@20 | PCK@50 | MPJPE |
|
||
|-----|---------|-----------:|-------:|-------:|------:|
|
||
| pre-2026-05-19 | pure-JS SPSA, lite TCN (#645) | ~20 min | 0.0% | 0.0% | 0.66 |
|
||
| **v0.0.1** (this run) | **candle-cuda, Conv1d TCN** | **2.1 s** | **3.0%** | **18.5%** | **0.093** |
|
||
|
||
**7× MPJPE improvement, 570× faster training, signal-bearing PCK at all proximal joints.** The remaining gap to ADR-079's PCK@20 ≥ 35% target is data-bound, not infra-bound (see Issue #645).
|
||
|
||
### Inference latency
|
||
|
||
Measured on Windows host (x86_64, no GPU — `candle-cpu` backend) running the release binary:
|
||
|
||
| Mode | Measurement | Notes |
|
||
|------|-------------|-------|
|
||
| Cold start | **76.2 ms / invocation** (avg over 100 sequential `health` invocations) | Includes safetensors load + 1 synthetic forward pass. Most of the cost is process startup + mmap. |
|
||
| Long-running `run` warm inference | sub-millisecond per frame (estimated) | The model is 125K params / 507 KB; once loaded, a single forward at batch=1 is essentially memory-bandwidth bound. To be measured precisely against a live sensing-server feed. |
|
||
|
||
### ONNX export
|
||
|
||
`pose_v1.onnx` is produced from `pose_v1.safetensors` by `scripts/export-onnx.py`, which mirrors the Candle architecture in PyTorch, loads the safetensors weights, and uses `torch.onnx.export` with opset 18 + dynamic batch axis. Verified end-to-end:
|
||
|
||
| Check | Result |
|
||
|-------|--------|
|
||
| `onnx.checker.check_model` | ✅ ok |
|
||
| Parity vs torch reference | **max \|torch − onnx\| = 8.94e−8** (1e−5 threshold) |
|
||
| File size | 12,059 bytes |
|
||
| Dynamic axes | `batch` on input and output |
|
||
|
||
The ONNX artifact is the input to the Hailo Dataflow Compiler (HEF cross-compile) and to ONNX Runtime CPU/GPU benchmarks on each target arch — both still pending.
|
||
|
||
### Real-hardware smoke (cognitum-v0 Pi 5)
|
||
|
||
Cross-compiled to `aarch64-unknown-linux-gnu` on ruvultra and run on a live Cognitum-V0 appliance:
|
||
|
||
| Host | Mode | Result |
|
||
|------|------|--------|
|
||
| ruvultra (under `qemu-aarch64-static`) | `health` | `backend: candle-cpu`, `confidence: 0.185` — real weights loaded under emulation |
|
||
| **cognitum-v0** (Raspberry Pi 5, Cortex-A76) | `health` | `backend: candle-cpu`, `confidence: 0.185` — real weights, real hardware |
|
||
| cognitum-v0 | 30× sequential `health` invocations | **0.251 s total → 8.4 ms / invocation** (cold) |
|
||
|
||
8.4 ms cold-start on real Pi 5 hardware vs 76 ms on the x86_64 Windows host. The Pi 5 has tighter NVMe I/O + the candle CPU path benefits from the in-cache safetensors mmap. Long-running `run` warm inference will still be sub-millisecond.
|
||
|
||
### Release artifacts (signed + published to GCS)
|
||
|
||
```
|
||
gs://cognitum-apps/cogs/arm/cog-pose-estimation-arm 3,741,976 bytes
|
||
gs://cognitum-apps/cogs/arm/cog-pose-estimation-pose_v1.safetensors 507,032 bytes
|
||
|
||
binary_sha256: 1e1a7d3dd01ca05d5bfc5dbb142a5941b7866ed9f3224a21edc04d3f09a99bf5
|
||
weights_sha256: eb249b9a6b2e10130437a10976ed0230b0d085f86a0553d7226e1ae6eae4b9e5
|
||
signature: LUN7xqLPYD3MFzm5dKB5MnYU0LvoRtek5ci5KiKPHBg+Xo6xuazwokn2Dw2JPMaLYJzmWn/SpT4djuR7hYvVDw== (Ed25519, signed with COGNITUM_OWNER_SIGNING_KEY)
|
||
```
|
||
|
||
Full manifest at `cog/artifacts/manifest.json`. Verified via public anonymous GET against `https://storage.googleapis.com/cognitum-apps/cogs/arm/cog-pose-estimation-arm` — downloaded SHA matches the locally-computed SHA.
|
||
|
||
### Live appliance install
|
||
|
||
Installed on `cognitum-v0` (the V0 cluster leader) at `/var/lib/cognitum/apps/pose-estimation/`:
|
||
|
||
```
|
||
$ ls -la /var/lib/cognitum/apps/pose-estimation/
|
||
-rwxr-xr-x cog-pose-estimation-arm 3,741,976 B (matches GCS sha256)
|
||
-rw-r--r-- pose_v1.safetensors 507,032 B
|
||
-rw-r--r-- manifest.json 989 B
|
||
-rw-r--r-- config.json 187 B
|
||
-rw-r--r-- output.log 28,438 B (5-sec smoke run)
|
||
```
|
||
|
||
Layout matches the existing `anomaly-detect`, `presence`, `seizure-detect`, etc. cogs on the same appliance — the Cogs dashboard at `http://cognitum-v0:9000/cogs` auto-discovers entries under this dir.
|
||
|
||
`cog-pose-estimation run` ran cleanly in the background for 5 seconds with the default config. It correctly:
|
||
|
||
- Emitted a `run.started` event with the configured `sensing_url`, `model_path`, and `poll_ms`.
|
||
- Started its 40 ms poll loop.
|
||
- **Gracefully handled the missing local sensing-server on port 3000** by logging structured WARN events (`{"level":"WARN","fields":{"message":"sensing-server fetch failed","error":"...Connection refused..."}}`) without crashing, leaking, or producing NaN output.
|
||
- Exited cleanly on SIGTERM.
|
||
|
||
0 `pose.frame` events fired during the smoke run — expected, since `127.0.0.1:3000` isn't serving CSI on the appliance. The appliance's actual CSI source is `ruview-vitals-worker` on `:50054` plus the `/api/v1/v0/system/...` endpoints behind the appliance's bearer auth on `:9000`. Wiring `sensing_url` to the appliance-native source is a Day-2 integration task — separate from the cog binary itself.
|
||
|
||
Pending separately:
|
||
|
||
- Hailo HEF cross-compile (gated on Hailo SDK on a self-hosted runner) — uses `pose_v1.onnx` as input.
|
||
- Appliance-native sensing-source integration (`config.sensing_url` should point at the cog-gateway's CSI tap on `:9000`, not the dev-loopback `:3000`).
|
||
### x86_64 release (2026-05-19)
|
||
|
||
Built on ruvultra (native, no cross-compile):
|
||
|
||
```
|
||
gs://cognitum-apps/cogs/x86_64/cog-pose-estimation-x86_64 4,548,856 bytes
|
||
sha256: a434739a24415b34e1aff50e5e1c3c32e568db96af473bbb3e5ecc9b95fe71fa
|
||
signature: pNNuxhgM18PztN8BSZdfw5oAShG2pV3na5T/q2QdlJWX/5FJgo4QTiUCbcTAxI2Uiva8VURSOlRzMU3xoQPqCQ==
|
||
```
|
||
|
||
Manifest at `cog/artifacts/manifests/x86_64/manifest.json`. Re-uses the same `pose_v1.safetensors` weights as the arm release (architecture is arch-independent).
|
||
|
||
**Cold-start: 5.4 ms / invocation** on ruvultra (30× sequential `health` in 0.162 s) — faster than the Pi 5's 8.4 ms (faster NVMe + wider CPU), slower than the Windows 76 ms (less mature Windows release toolchain).
|
||
|
||
| Host | arch | rust | binary | cold-start |
|
||
|------|------|------|--------|------------|
|
||
| Windows (ruvzen) | x86_64 | 1.95.0 | (built locally, not published) | 76.2 ms |
|
||
| ruvultra (Ubuntu) | x86_64 | 1.89.0 | 4,548,856 B (GCS x86_64) | **5.4 ms** |
|
||
| cognitum-v0 (Pi 5) | aarch64 | (cross-built) | 3,741,976 B (GCS arm) | 8.4 ms |
|
||
|
||
### Artifacts
|
||
|
||
- `v2/crates/cog-pose-estimation/cog/artifacts/pose_v1.safetensors` — 507 KB
|
||
- `v2/crates/cog-pose-estimation/cog/artifacts/train_results.json` — full per-epoch loss curve + hyperparameters + per-joint PCK
|
||
|
||
### Reproducibility
|
||
|
||
```bash
|
||
# On any host with cargo + a CUDA-capable GPU:
|
||
cd ~/work/cog-pose-train
|
||
mkdir -p ./
|
||
# Stage the same inputs (1,077 paired samples + HF encoder, see scripts/align-ground-truth.js for regeneration)
|
||
cp paired.jsonl ./paired.jsonl
|
||
cp encoder.safetensors ./encoder.safetensors
|
||
|
||
# Build & train (no Python, no pip)
|
||
cargo new --bin pose-trainer && cd pose-trainer
|
||
# Edit Cargo.toml deps: candle-core 0.9 (cuda), candle-nn 0.9 (cuda), safetensors, serde, serde_json, anyhow
|
||
# Drop the training script into src/main.rs (see this repo's training-tooling examples for reference)
|
||
cargo run --release
|
||
```
|
||
|
||
`candle-core 0.8.4 + 0.9.2` are typically already in `~/.cargo/registry/cache/` on any developer host, so the build completes in seconds.
|