diff --git a/FINDINGS.md b/FINDINGS.md index d254a0a..c7ceecb 100644 --- a/FINDINGS.md +++ b/FINDINGS.md @@ -82,6 +82,16 @@ documented failure mode, a single-glyph silent misread (`125f9e6e1c77` → `125f9e6a1c77`); one passing trial also misread an adjacent field (`cc33ae67` → `cc33a867`). The verbatim-risk guard remains required. +*Expansion (2026-06-11, n=15 to match the Opus haystack count):* 5 fresh +dense-JSON pages at the production 5×8 cell (40 lines/page, ~4.7k chars), +3 needles per page, one-shot `claude -p` per trial, exact-match: **13/15** +(vs Opus **0/15**). Both misses are single-glyph confabulations on visually +adjacent hex digits — `5a7373d4187f` → `5a7973d4107f` (3→9, 8→0) and +`b8fce698f971` → `b0fce698f971` (8→0) — plausible-looking, no error signal. +Same conclusion at 4× the sample: Fable reads dense renders near-verbatim, +but ~13% silent-misread on exact strings means the rule stands — anything +that must round-trip byte-exact stays text. + **4. Economics.** Fable is $10/$50 per MTok (2× Opus 4.8). Token-for-token savings are identical (same tokenizer), so every saved token is worth 2× the dollars — pxpipe is more valuable on Fable in absolute terms. diff --git a/README.md b/README.md index 5318131..215f654 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ text; only older bulk history is imaged. **It is lossy.** pxpipe is a *gist* tier, not a lossless store. In a needle-in-haystack eval, exact 12-char hex strings inside dense imaged content -came back **0/15** on Opus and 3/4 on Fable 5, and the failure mode is +came back **0/15** on Opus and 13/15 on Fable 5, and the failure mode is *silent confabulation*: a plausible wrong value, not an error. Anything you need back byte-exact (IDs, hashes, secrets, exact numbers) must stay text. Recent turns do; a dedicated verbatim-risk guard is not built yet. @@ -79,7 +79,7 @@ Measured with novel random-number problems the model cannot have memorized: | state tracking (value mutated 3x, final/first/count), Fable 5 | 18/arm | 18/18 | **18/18** | - | | confabulation on never-stated facts (lower is better), Fable 5 | 16/arm | 0/16 | **0/16** | - | | verbatim 12-char hex recall, dense render, Opus | 15 | 15/15 | **0/15** | - | -| verbatim 12-char hex recall, dense render, Fable 5 | 4 | - | 3/4 | - | +| verbatim 12-char hex recall, dense render, Fable 5 | 15 | - | **13/15** | - | ### SWE-bench Lite pilot (end-to-end task quality) diff --git a/eval/verbatim-15/golds.json b/eval/verbatim-15/golds.json new file mode 100644 index 0000000..00ea126 --- /dev/null +++ b/eval/verbatim-15/golds.json @@ -0,0 +1,77 @@ +[ + { + "page": 0, + "dur": 4439, + "gold": "c9c947f680ec" + }, + { + "page": 0, + "dur": 812, + "gold": "851eb3af1bd1" + }, + { + "page": 0, + "dur": 6150, + "gold": "ade34f70fd73" + }, + { + "page": 1, + "dur": 7978, + "gold": "c5d68855f46d" + }, + { + "page": 1, + "dur": 8071, + "gold": "92abade01aad" + }, + { + "page": 1, + "dur": 3309, + "gold": "ffe21785b09d" + }, + { + "page": 2, + "dur": 7215, + "gold": "87cb51eb0e99" + }, + { + "page": 2, + "dur": 4397, + "gold": "93c3ced96dac" + }, + { + "page": 2, + "dur": 4495, + "gold": "f152ae9bfb8f" + }, + { + "page": 3, + "dur": 1622, + "gold": "5a7373d4187f" + }, + { + "page": 3, + "dur": 2025, + "gold": "44ea8c7aeedd" + }, + { + "page": 3, + "dur": 6533, + "gold": "8145b5a0fd46" + }, + { + "page": 4, + "dur": 2921, + "gold": "b8fce698f971" + }, + { + "page": 4, + "dur": 8475, + "gold": "4a8164556b99" + }, + { + "page": 4, + "dur": 8799, + "gold": "e53112c4b5a4" + } +] \ No newline at end of file diff --git a/eval/verbatim-15/page0.png b/eval/verbatim-15/page0.png new file mode 100644 index 0000000..1adba5d Binary files /dev/null and b/eval/verbatim-15/page0.png differ diff --git a/eval/verbatim-15/page0_big.png b/eval/verbatim-15/page0_big.png new file mode 100644 index 0000000..4c39da4 Binary files /dev/null and b/eval/verbatim-15/page0_big.png differ diff --git a/eval/verbatim-15/page1.png b/eval/verbatim-15/page1.png new file mode 100644 index 0000000..ae18975 Binary files /dev/null and b/eval/verbatim-15/page1.png differ diff --git a/eval/verbatim-15/page2.png b/eval/verbatim-15/page2.png new file mode 100644 index 0000000..6201cef Binary files /dev/null and b/eval/verbatim-15/page2.png differ diff --git a/eval/verbatim-15/page3.png b/eval/verbatim-15/page3.png new file mode 100644 index 0000000..7d80723 Binary files /dev/null and b/eval/verbatim-15/page3.png differ diff --git a/eval/verbatim-15/page4.png b/eval/verbatim-15/page4.png new file mode 100644 index 0000000..21dd408 Binary files /dev/null and b/eval/verbatim-15/page4.png differ diff --git a/eval/verbatim-15/run.sh b/eval/verbatim-15/run.sh new file mode 100755 index 0000000..2394db1 --- /dev/null +++ b/eval/verbatim-15/run.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# 25 verbatim hex-recall trials, parallelism 3 +cd /tmp/verb25 +python3 -c " +import json +for i, g in enumerate(json.load(open('golds.json'))): + print(i, g['page'], g['dur'], g['gold']) +" > trials.txt +run_one() { + local i=$1 page=$2 dur=$3 gold=$4 + local ans + ans=$(~/.claude/local/claude -p --model claude-fable-5 "Read the image at /tmp/verb25/page${page}.png. Find the JSON line whose dur_ms is exactly ${dur} and report ONLY its 'id' field value (12 hex chars), nothing else. Read it visually from the image; do not use code." 2>/dev/null | tr -d '[:space:]' | grep -oE '[0-9a-f]{12}' | head -1) + if [ "$ans" = "$gold" ]; then + echo "HIT trial=$i page=$page dur=$dur gold=$gold" + else + echo "MISS trial=$i page=$page dur=$dur gold=$gold got=${ans:-EMPTY}" + fi +} +export -f run_one +cat trials.txt | xargs -P 3 -L 1 bash -c 'run_one "$@"' _ +echo "=== done ==="