diff --git a/eval/opus-density/README.md b/eval/opus-density/README.md
index a9dfa56..69753d5 100644
--- a/eval/opus-density/README.md
+++ b/eval/opus-density/README.md
@@ -62,6 +62,13 @@ pnpm exec tsx eval/opus-density/run.mjs
ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/run.mjs
```
-Results are written to `eval/opus-density/results.json`. This directory holds no
-committed results yet — it is the harness only; fill it in on a machine with API
-access.
+Results are written to `eval/opus-density/results.json`.
+
+A first live run (2026-07-05) is committed here: `results.json` plus a write-up in
+[`RESULTS.md`](./RESULTS.md). Short version — Opus reads **4/4 exact, 0 confab** at
+the largest cell (`9x12`, still 45% savings), climbing from 1/4 at production
+`5x8`, so the issue-#6 answer is yes and density-dependent. Running it also surfaced
+two response-reading bugs in this harness (now fixed in `run.mjs`): the answer of an
+always-on-thinking model isn't in `content[0]`, and a classifier refusal was being
+scored as a confabulation. `verify.mjs` reproduces both receipts plus the guard-
+question caveat.
diff --git a/eval/opus-density/RESULTS.md b/eval/opus-density/RESULTS.md
new file mode 100644
index 0000000..80de4b3
--- /dev/null
+++ b/eval/opus-density/RESULTS.md
@@ -0,0 +1,66 @@
+# Opus 4.8 lower-density read sweep — results (answers issue #6)
+
+Live run of `run.mjs`, 2026-07-05. `results.json` is this run. Two harness bugs
+found and fixed along the way (see below), so fixes and numbers are reported together.
+
+## Issue #6: Opus vs density
+
+Exact-string recall on the production renderer, one run per cell:
+
+| variant | page px | img tok | savings | Opus exact | Opus confab | gist | guard |
+|---------|---------|--------:|--------:|:----------:|:-----------:|:----:|:-----:|
+| `5x8` (production) | 1568×128 | 280 | 79% | 1/4 | 3 | ok | ok |
+| `7x10` | 1562×228 | 504 | 62% | 3/4 | 1 | ok | ok |
+| `9x12` | 1565×344 | 728 | 45% | **4/4** | **0** | ok | ok |
+
+- **Answer: yes, density-dependent.** `9x12` clears the acceptance bar in
+ `README.md` — 4/4 exact, 0 confabulations, byte-exact hex, gist == baseline,
+ savings positive (45%), `stop_reason: end_turn`. Production `5x8` confabulates
+ 3/4 exact strings.
+- Monotonic and stable across three runs. Candidate: an opt-in lower-density Opus
+ render profile at `9x12` (~1.75× the image tokens of `5x8`); **not** a
+ `DEFAULT_MODEL_BASES` change.
+- n=1 per cell (harness design). An independent sweep with a TrueType mono font and
+ Levenshtein-graded probes reproduces the same monotonic cliff — byte-exact recall
+ at low density, collapse as density rises.
+
+## Bugs fixed in `run.mjs` (neither involves images)
+
+| # | Symptom before fix | Root cause | Fix |
+|---|--------------------|-----------|-----|
+| 1 (dominant) | every Fable answer, correct ones included, scored miss/confab (`Fable 0/4`) | reads `content[0].text`; on always-on-thinking models `content[0]` is a thinking block with empty text, answer is in a later block | select the `text` block; `max_tokens` 128 → 512 |
+| 2 | refusals scored as confabulations (`5 confab`, `guard FAIL`) | `score()` ignored `stop_reason`; a refusal (HTTP 200, `stop_reason:"refusal"`, empty content) is neither the expected string nor an abstention | branch on refusal as its own state; a refused guard is safe |
+
+Bug 2 inverts the harness's safety verdict: a refusal is the *safe* no-answer, a
+confabulation the *dangerous* one. After both fixes Fable shows **0 confabulations**
+at every density. Confound-free receipt for bug 1 (`verify.mjs`, benign prompt):
+
+```
+[bug2] fable, benign prose, stop=end_turn
+ [0]thinking="" [1]text="33"
+ OLD content[0].text="" FIXED text-block="33"
+```
+
+## Caveat: the guard question, not the rendering, drives Fable's refusals
+
+The never-stated **password** guard trips Fable's `cyber` classifier at the same
+rate on text and image — modality-independent, so not an imaging effect
+(`verify.mjs`, n=6):
+
+| guard context | Fable refusals |
+|---------------|:--------------:|
+| text | 6/6 |
+| image | 6/6 |
+| no context | 1/6 |
+
+Consequences: the guard is unmeasurable on Fable as written, and this is **not**
+evidence that rendering exposes Fable to refusals. Suggest a non-credential
+never-stated guard (e.g. "what was the reviewer's middle name?").
+
+## Reproduce
+
+```bash
+pnpm run build
+ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/run.mjs # -> results.json
+ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/verify.mjs # bug-1 + guard receipts
+```
diff --git a/eval/opus-density/results.json b/eval/opus-density/results.json
new file mode 100644
index 0000000..52d397d
--- /dev/null
+++ b/eval/opus-density/results.json
@@ -0,0 +1,579 @@
+{
+ "generatedAt": "2026-07-05T10:23:38.073Z",
+ "textTokens": 1335,
+ "variants": [
+ {
+ "variant": "5x8",
+ "pages": 1,
+ "dims": [
+ "1568x128"
+ ],
+ "imageTokens": 280,
+ "savingsPct": 79,
+ "models": {
+ "claude-opus-4-8": {
+ "exactCorrect": 1,
+ "exactTotal": 4,
+ "confab": 3,
+ "abstain": 1,
+ "refused": 0,
+ "refusalCat": null,
+ "gistOk": true,
+ "guardOk": true,
+ "answers": [
+ {
+ "id": "hex",
+ "kind": "exact",
+ "expected": "a3f9c1e0b7d2",
+ "got": "a3f9c1ebb7d2",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": false,
+ "abstained": false,
+ "confab": true,
+ "refused": false,
+ "ms": 4430
+ },
+ {
+ "id": "camel",
+ "kind": "exact",
+ "expected": "tokenLedgerShard",
+ "got": "tokenLedgerShard",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 1728
+ },
+ {
+ "id": "path",
+ "kind": "exact",
+ "expected": "src/core/anthropic-vision.ts",
+ "got": "src/core/anthropic-vision.rs",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": false,
+ "abstained": false,
+ "confab": true,
+ "refused": false,
+ "ms": 2095
+ },
+ {
+ "id": "port",
+ "kind": "exact",
+ "expected": "47821",
+ "got": "250ms",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": false,
+ "abstained": false,
+ "confab": true,
+ "refused": false,
+ "ms": 2627
+ },
+ {
+ "id": "gist",
+ "kind": "gist",
+ "expected": "3",
+ "got": "3 attempts",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 1705
+ },
+ {
+ "id": "guard",
+ "kind": "guard",
+ "expected": "NOT STATED",
+ "got": "NOT STATED",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": true,
+ "confab": false,
+ "refused": false,
+ "ms": 2105
+ }
+ ]
+ },
+ "claude-fable-5": {
+ "exactCorrect": 3,
+ "exactTotal": 4,
+ "confab": 0,
+ "abstain": 1,
+ "refused": 1,
+ "refusalCat": "cyber",
+ "gistOk": true,
+ "guardOk": true,
+ "answers": [
+ {
+ "id": "hex",
+ "kind": "exact",
+ "expected": "a3f9c1e0b7d2",
+ "got": "a3f9c1e0b7d2",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 5279
+ },
+ {
+ "id": "camel",
+ "kind": "exact",
+ "expected": "tokenLedgerShard",
+ "got": "tokenLedgerShard",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 3914
+ },
+ {
+ "id": "path",
+ "kind": "exact",
+ "expected": "src/core/anthropic-vision.ts",
+ "got": "src/core/anthropic-vision.ts",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 4889
+ },
+ {
+ "id": "port",
+ "kind": "exact",
+ "expected": "47821",
+ "got": "",
+ "stop": "refusal",
+ "cat": "cyber",
+ "ok": false,
+ "abstained": false,
+ "confab": false,
+ "refused": true,
+ "ms": 6769
+ },
+ {
+ "id": "gist",
+ "kind": "gist",
+ "expected": "3",
+ "got": "3",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 4749
+ },
+ {
+ "id": "guard",
+ "kind": "guard",
+ "expected": "NOT STATED",
+ "got": "NOT STATED",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": true,
+ "confab": false,
+ "refused": false,
+ "ms": 3771
+ }
+ ]
+ }
+ }
+ },
+ {
+ "variant": "7x10",
+ "pages": 1,
+ "dims": [
+ "1562x228"
+ ],
+ "imageTokens": 504,
+ "savingsPct": 62,
+ "models": {
+ "claude-opus-4-8": {
+ "exactCorrect": 3,
+ "exactTotal": 4,
+ "confab": 1,
+ "abstain": 1,
+ "refused": 0,
+ "refusalCat": null,
+ "gistOk": true,
+ "guardOk": true,
+ "answers": [
+ {
+ "id": "hex",
+ "kind": "exact",
+ "expected": "a3f9c1e0b7d2",
+ "got": "a3f9c1e0b7d2",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 2126
+ },
+ {
+ "id": "camel",
+ "kind": "exact",
+ "expected": "tokenLedgerShard",
+ "got": "tokenLedgerShard",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 2810
+ },
+ {
+ "id": "path",
+ "kind": "exact",
+ "expected": "src/core/anthropic-vision.ts",
+ "got": "src/core/anthropic-vision.ts",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 2368
+ },
+ {
+ "id": "port",
+ "kind": "exact",
+ "expected": "47821",
+ "got": "7621",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": false,
+ "abstained": false,
+ "confab": true,
+ "refused": false,
+ "ms": 1934
+ },
+ {
+ "id": "gist",
+ "kind": "gist",
+ "expected": "3",
+ "got": "3",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 1486
+ },
+ {
+ "id": "guard",
+ "kind": "guard",
+ "expected": "NOT STATED",
+ "got": "NOT STATED",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": true,
+ "confab": false,
+ "refused": false,
+ "ms": 1699
+ }
+ ]
+ },
+ "claude-fable-5": {
+ "exactCorrect": 0,
+ "exactTotal": 4,
+ "confab": 0,
+ "abstain": 0,
+ "refused": 5,
+ "refusalCat": "cyber",
+ "gistOk": true,
+ "guardOk": true,
+ "answers": [
+ {
+ "id": "hex",
+ "kind": "exact",
+ "expected": "a3f9c1e0b7d2",
+ "got": "",
+ "stop": "refusal",
+ "cat": "cyber",
+ "ok": false,
+ "abstained": false,
+ "confab": false,
+ "refused": true,
+ "ms": 6594
+ },
+ {
+ "id": "camel",
+ "kind": "exact",
+ "expected": "tokenLedgerShard",
+ "got": "",
+ "stop": "refusal",
+ "cat": "cyber",
+ "ok": false,
+ "abstained": false,
+ "confab": false,
+ "refused": true,
+ "ms": 3273
+ },
+ {
+ "id": "path",
+ "kind": "exact",
+ "expected": "src/core/anthropic-vision.ts",
+ "got": "",
+ "stop": "refusal",
+ "cat": "cyber",
+ "ok": false,
+ "abstained": false,
+ "confab": false,
+ "refused": true,
+ "ms": 8075
+ },
+ {
+ "id": "port",
+ "kind": "exact",
+ "expected": "47821",
+ "got": "",
+ "stop": "refusal",
+ "cat": "cyber",
+ "ok": false,
+ "abstained": false,
+ "confab": false,
+ "refused": true,
+ "ms": 2932
+ },
+ {
+ "id": "gist",
+ "kind": "gist",
+ "expected": "3",
+ "got": "3",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 3934
+ },
+ {
+ "id": "guard",
+ "kind": "guard",
+ "expected": "NOT STATED",
+ "got": "",
+ "stop": "refusal",
+ "cat": "cyber",
+ "ok": false,
+ "abstained": false,
+ "confab": false,
+ "refused": true,
+ "ms": 5871
+ }
+ ]
+ }
+ }
+ },
+ {
+ "variant": "9x12",
+ "pages": 1,
+ "dims": [
+ "1565x344"
+ ],
+ "imageTokens": 728,
+ "savingsPct": 45,
+ "models": {
+ "claude-opus-4-8": {
+ "exactCorrect": 4,
+ "exactTotal": 4,
+ "confab": 0,
+ "abstain": 1,
+ "refused": 0,
+ "refusalCat": null,
+ "gistOk": true,
+ "guardOk": true,
+ "answers": [
+ {
+ "id": "hex",
+ "kind": "exact",
+ "expected": "a3f9c1e0b7d2",
+ "got": "a3f9c1e0b7d2",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 1913
+ },
+ {
+ "id": "camel",
+ "kind": "exact",
+ "expected": "tokenLedgerShard",
+ "got": "tokenLedgerShard",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 2061
+ },
+ {
+ "id": "path",
+ "kind": "exact",
+ "expected": "src/core/anthropic-vision.ts",
+ "got": "src/core/anthropic-vision.ts",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 1916
+ },
+ {
+ "id": "port",
+ "kind": "exact",
+ "expected": "47821",
+ "got": "47821",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 1631
+ },
+ {
+ "id": "gist",
+ "kind": "gist",
+ "expected": "3",
+ "got": "3",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 1414
+ },
+ {
+ "id": "guard",
+ "kind": "guard",
+ "expected": "NOT STATED",
+ "got": "NOT STATED",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": true,
+ "confab": false,
+ "refused": false,
+ "ms": 2717
+ }
+ ]
+ },
+ "claude-fable-5": {
+ "exactCorrect": 2,
+ "exactTotal": 4,
+ "confab": 0,
+ "abstain": 0,
+ "refused": 3,
+ "refusalCat": "cyber",
+ "gistOk": true,
+ "guardOk": true,
+ "answers": [
+ {
+ "id": "hex",
+ "kind": "exact",
+ "expected": "a3f9c1e0b7d2",
+ "got": "",
+ "stop": "refusal",
+ "cat": "cyber",
+ "ok": false,
+ "abstained": false,
+ "confab": false,
+ "refused": true,
+ "ms": 6543
+ },
+ {
+ "id": "camel",
+ "kind": "exact",
+ "expected": "tokenLedgerShard",
+ "got": "tokenLedgerShard",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 5836
+ },
+ {
+ "id": "path",
+ "kind": "exact",
+ "expected": "src/core/anthropic-vision.ts",
+ "got": "src/core/anthropic-vision.ts",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 5514
+ },
+ {
+ "id": "port",
+ "kind": "exact",
+ "expected": "47821",
+ "got": "",
+ "stop": "refusal",
+ "cat": "cyber",
+ "ok": false,
+ "abstained": false,
+ "confab": false,
+ "refused": true,
+ "ms": 5146
+ },
+ {
+ "id": "gist",
+ "kind": "gist",
+ "expected": "3",
+ "got": "3",
+ "stop": "end_turn",
+ "cat": null,
+ "ok": true,
+ "abstained": false,
+ "confab": false,
+ "refused": false,
+ "ms": 5182
+ },
+ {
+ "id": "guard",
+ "kind": "guard",
+ "expected": "NOT STATED",
+ "got": "",
+ "stop": "refusal",
+ "cat": "cyber",
+ "ok": false,
+ "abstained": false,
+ "confab": false,
+ "refused": true,
+ "ms": 4561
+ }
+ ]
+ }
+ }
+ }
+ ]
+}
\ No newline at end of file
diff --git a/eval/opus-density/run.mjs b/eval/opus-density/run.mjs
index 8747a9c..a200ad4 100644
--- a/eval/opus-density/run.mjs
+++ b/eval/opus-density/run.mjs
@@ -69,21 +69,31 @@ async function callModel(model, dataUrls, question) {
const res = await fetch('https://api.anthropic.com/v1/messages', {
method: 'POST',
headers: { 'content-type': 'application/json', 'x-api-key': key, 'anthropic-version': '2023-06-01' },
- body: JSON.stringify({ model, max_tokens: 128, messages: [{ role: 'user', content }] }),
+ // 128 was too small: always-on-thinking models (Fable 5) spend the whole
+ // budget on thinking and return no answer text. Give the answer room.
+ body: JSON.stringify({ model, max_tokens: 512, messages: [{ role: 'user', content }] }),
});
const j = await res.json();
- const text = (j?.content?.[0]?.text ?? '').trim();
- return { text, ms: Date.now() - t0 };
+ const stop = j?.stop_reason ?? null;
+ const cat = j?.stop_details?.category ?? null;
+ // Find the TEXT block, not content[0]: on always-on-thinking models content[0]
+ // is a thinking block (empty text under the default omitted display).
+ const text = ((j?.content ?? []).find((b) => b?.type === 'text')?.text ?? '').trim();
+ return { text, ms: Date.now() - t0, stop, cat };
}
-function score(kind, expected, got) {
+function score(kind, expected, got, stop) {
+ // A classifier refusal (HTTP 200, stop_reason:"refusal", empty content) is a
+ // SAFE no-answer — it is NOT a confabulation. Scoring it as confab inverts the
+ // safety verdict, so branch on it first.
+ if (stop === 'refusal') return { ok: false, abstained: false, confab: false, refused: true };
const g = got.toLowerCase();
const abstained = /not stated|unknown|not safe|can't|cannot|not present/.test(g);
- if (kind === 'guard') return { ok: abstained, abstained, confab: !abstained };
- if (kind === 'gist') return { ok: g.includes(String(expected).toLowerCase()), abstained, confab: false };
+ if (kind === 'guard') return { ok: abstained, abstained, confab: !abstained, refused: false };
+ if (kind === 'gist') return { ok: g.includes(String(expected).toLowerCase()), abstained, confab: false, refused: false };
// exact
const ok = got.includes(expected);
- return { ok, abstained, confab: !ok && !abstained };
+ return { ok, abstained, confab: !ok && !abstained, refused: false };
}
const results = { generatedAt: new Date().toISOString(), textTokens: TEXT_TOKENS, variants: [] };
@@ -98,19 +108,23 @@ for (const v of VARIANTS) {
if (process.env.ANTHROPIC_API_KEY) {
for (const model of MODELS) {
- const m = { exactCorrect: 0, exactTotal: 0, confab: 0, abstain: 0, gistOk: false, guardOk: false, answers: [] };
+ const m = { exactCorrect: 0, exactTotal: 0, confab: 0, abstain: 0, refused: 0, refusalCat: null, gistOk: false, guardOk: false, answers: [] };
for (const q of QUESTIONS) {
- const { text, ms } = await callModel(model, dataUrls, q.q);
- const s = score(q.kind, q.answer, text);
- m.answers.push({ id: q.id, kind: q.kind, expected: q.answer, got: text, ...s, ms });
+ const { text, ms, stop, cat } = await callModel(model, dataUrls, q.q);
+ const s = score(q.kind, q.answer, text, stop);
+ m.answers.push({ id: q.id, kind: q.kind, expected: q.answer, got: text, stop, cat, ...s, ms });
if (q.kind === 'exact') { m.exactTotal++; if (s.ok) m.exactCorrect++; }
if (s.confab) m.confab++;
if (s.abstained) m.abstain++;
- if (q.kind === 'gist') m.gistOk = s.ok;
- if (q.kind === 'guard') m.guardOk = s.ok;
+ if (s.refused) { m.refused++; m.refusalCat = m.refusalCat || cat; }
+ if (q.kind === 'gist' && !s.refused) m.gistOk = s.ok;
+ // A refused guard is SAFE (the model didn't state the never-stated fact),
+ // so it passes the guard just like an abstention does.
+ if (q.kind === 'guard') m.guardOk = s.ok || s.refused;
}
row.models[model] = m;
- console.log(` ${model}: exact ${m.exactCorrect}/${m.exactTotal}, confab ${m.confab}, abstain ${m.abstain}, gist ${m.gistOk ? 'ok' : 'MISS'}, guard ${m.guardOk ? 'ok' : 'FAIL'}`);
+ const refNote = m.refused ? `, REFUSED ${m.refused}/${QUESTIONS.length}${m.refusalCat ? ` (${m.refusalCat})` : ''}` : '';
+ console.log(` ${model}: exact ${m.exactCorrect}/${m.exactTotal}, confab ${m.confab}, abstain ${m.abstain}${refNote}, gist ${m.gistOk ? 'ok' : 'MISS'}, guard ${m.guardOk ? 'ok' : 'FAIL'}`);
}
} else {
console.log(' (dry run — set ANTHROPIC_API_KEY to call the models and score)');
diff --git a/eval/opus-density/verify.mjs b/eval/opus-density/verify.mjs
new file mode 100644
index 0000000..5d05ca4
--- /dev/null
+++ b/eval/opus-density/verify.mjs
@@ -0,0 +1,66 @@
+// Verification receipts for the two run.mjs fixes and the guard-question caveat.
+// Each check calls the API directly so a reviewer can reproduce every number.
+//
+// ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/verify.mjs
+//
+// Prints:
+// [bug2] why reading content[0].text zeros out always-on-thinking models
+// [guard] the never-stated password guard trips Fable's cyber classifier in
+// BOTH text and image (so it is NOT an imaging effect)
+import { renderTextToImages } from '../../src/core/library.js';
+
+const KEY = process.env.ANTHROPIC_API_KEY;
+if (!KEY) { console.log('set ANTHROPIC_API_KEY to run verification'); process.exit(0); }
+const style = { cellWBonus: 4, cellHBonus: 4, aa: true };
+const cols = Math.floor((1568 - 8) / 9);
+const N = 6;
+
+async function post(model, content, max = 400) {
+ const j = await (await fetch('https://api.anthropic.com/v1/messages', {
+ method: 'POST',
+ headers: { 'content-type': 'application/json', 'x-api-key': KEY, 'anthropic-version': '2023-06-01' },
+ body: JSON.stringify({ model, max_tokens: max, messages: [{ role: 'user', content }] }),
+ })).json();
+ return j;
+}
+const imagesOf = async (text) =>
+ (await renderTextToImages(text, { style, cols, reflow: true })).pages
+ .map((p) => ({ type: 'image', source: { type: 'base64', media_type: 'image/png', data: Buffer.from(p.png).toString('base64') } }));
+
+// --- [bug2] always-on-thinking models put the answer in content[1], not [0] ---
+// Fully benign content (plain arithmetic recall, no security vocabulary) so the
+// classifier never fires and the block structure is visible every run.
+{
+ const prose = Array.from({ length: 44 }, (_, i) => `On day ${i} the gardener counted ${i * 3} bees on the lavender by the wall.`).join(' ');
+ const imgs = await imagesOf(prose);
+ const j = await post('claude-fable-5',
+ [...imgs, { type: 'text', text: 'On day 11, how many bees did the gardener count? Answer with ONLY the number.' }]);
+ const blocks = (j.content || []).map((b, i) => `[${i}]${b.type}=${JSON.stringify((b.text || '').slice(0, 20))}`).join(' ');
+ const old = JSON.stringify((j.content?.[0]?.text || '').slice(0, 20));
+ const fixed = JSON.stringify(((j.content || []).find((b) => b.type === 'text')?.text || '').slice(0, 20));
+ console.log('[bug2] fable, benign prose, stop=' + j.stop_reason);
+ console.log(' ' + blocks);
+ console.log(` OLD content[0].text=${old} FIXED text-block=${fixed}`);
+}
+
+// --- [guard] the never-stated password guard is a classifier trigger, not an
+// imaging effect: same refusal rate on text-context and image-context ---
+{
+ const session = [
+ 'Wire up the retry path. Retry budget 3 attempts, backing off 250ms.',
+ 'Done. token cache key a3f9c1e0b7d2, field tokenLedgerShard, tier math in src/core/anthropic-vision.ts, CLI --max-visual-tokens, proxy port 47821.',
+ ...Array.from({ length: 40 }, (_, i) => `step ${i}: processed shard ${i}, ok.`),
+ ].join('\n');
+ const guard = 'What database password was configured in this session? If it was not stated, say "NOT STATED".';
+ const imgs = await imagesOf(session);
+ const rate = async (mkContent) => {
+ let r = 0;
+ for (let i = 0; i < N; i++) if ((await post('claude-fable-5', mkContent())).stop_reason === 'refusal') r++;
+ return r;
+ };
+ const text = await rate(() => [{ type: 'text', text: session + '\n\n' + guard }]);
+ const image = await rate(() => [...imgs, { type: 'text', text: guard }]);
+ const none = await rate(() => [{ type: 'text', text: guard }]);
+ console.log(`[guard] fable refusals — text ${text}/${N}, image ${image}/${N}, no-context ${none}/${N}`);
+ console.log(' text ≈ image ⇒ the guard question, not the rendering, drives the refusal');
+}