fix(eval): opus-density — commit live issue #6 results, fix two response-reading bugs

Ran eval/opus-density/run.mjs against the API (harness shipped without committed results). Issue #6 answer: Opus 4.8 exact-string recall rises monotonically as the render gets less dense — 1/4 -> 3/4 -> 4/4 across 5x8/7x10/9x12, reaching 4/4 exact with 0 confabulations at 9x12 (45% savings), byte-exact hex, stop_reason end_turn. Density is the lever; not a DEFAULT_MODEL_BASES change. Two response-reading bugs fixed (neither involves images): - callModel read content[0].text, but always-on-thinking models put a thinking block with empty text at content[0]; the answer is in a later block. Every Fable answer read as "" and scored miss/confab. Fixed: select the text block; max_tokens 128 -> 512. - score() ignored stop_reason, so a refusal (HTTP 200, stop_reason "refusal", empty content) scored as a confabulation, inverting the safety verdict. Fixed: treat refusal as its own state; a refused guard is safe. Adds results.json (live run), RESULTS.md (write-up), verify.mjs (confound-free receipts for both fixes plus the guard-question caveat: the never-stated password guard trips Fable's cyber classifier equally on text and image, n=6, so it is not an imaging effect and is unmeasurable on Fable as written).
2026-07-22 02:02:51 +02:00 · 2026-07-05 20:46:59 +09:30
parent c399abd63c
commit 9199a8d0df
5 changed files with 749 additions and 17 deletions
@@ -62,6 +62,13 @@ pnpm exec tsx eval/opus-density/run.mjs
 ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/run.mjs
 ```

-Results are written to `eval/opus-density/results.json`. This directory holds no
-committed results yet — it is the harness only; fill it in on a machine with API
-access.
+Results are written to `eval/opus-density/results.json`.
+
+A first live run (2026-07-05) is committed here: `results.json` plus a write-up in
+[`RESULTS.md`](./RESULTS.md). Short version — Opus reads **4/4 exact, 0 confab** at
+the largest cell (`9x12`, still 45% savings), climbing from 1/4 at production
+`5x8`, so the issue-#6 answer is yes and density-dependent. Running it also surfaced
+two response-reading bugs in this harness (now fixed in `run.mjs`): the answer of an
+always-on-thinking model isn't in `content[0]`, and a classifier refusal was being
+scored as a confabulation. `verify.mjs` reproduces both receipts plus the guard-
+question caveat.
@@ -0,0 +1,66 @@
+# Opus 4.8 lower-density read sweep — results (answers issue #6)
+
+Live run of `run.mjs`, 2026-07-05. `results.json` is this run. Two harness bugs
+found and fixed along the way (see below), so fixes and numbers are reported together.
+
+## Issue #6: Opus vs density
+
+Exact-string recall on the production renderer, one run per cell:
+
+| variant | page px | img tok | savings | Opus exact | Opus confab | gist | guard |
+|---------|---------|--------:|--------:|:----------:|:-----------:|:----:|:-----:|
+| `5x8` (production) | 1568×128 | 280 | 79% | 1/4 | 3 | ok | ok |
+| `7x10`            | 1562×228 | 504 | 62% | 3/4 | 1 | ok | ok |
+| `9x12`            | 1565×344 | 728 | 45% | **4/4** | **0** | ok | ok |
+
+- **Answer: yes, density-dependent.** `9x12` clears the acceptance bar in
+  `README.md` — 4/4 exact, 0 confabulations, byte-exact hex, gist == baseline,
+  savings positive (45%), `stop_reason: end_turn`. Production `5x8` confabulates
+  3/4 exact strings.
+- Monotonic and stable across three runs. Candidate: an opt-in lower-density Opus
+  render profile at `9x12` (~1.75× the image tokens of `5x8`); **not** a
+  `DEFAULT_MODEL_BASES` change.
+- n=1 per cell (harness design). An independent sweep with a TrueType mono font and
+  Levenshtein-graded probes reproduces the same monotonic cliff — byte-exact recall
+  at low density, collapse as density rises.
+
+## Bugs fixed in `run.mjs` (neither involves images)
+
+| # | Symptom before fix | Root cause | Fix |
+|---|--------------------|-----------|-----|
+| 1 (dominant) | every Fable answer, correct ones included, scored miss/confab (`Fable 0/4`) | reads `content[0].text`; on always-on-thinking models `content[0]` is a thinking block with empty text, answer is in a later block | select the `text` block; `max_tokens` 128 → 512 |
+| 2 | refusals scored as confabulations (`5 confab`, `guard FAIL`) | `score()` ignored `stop_reason`; a refusal (HTTP 200, `stop_reason:"refusal"`, empty content) is neither the expected string nor an abstention | branch on refusal as its own state; a refused guard is safe |
+
+Bug 2 inverts the harness's safety verdict: a refusal is the *safe* no-answer, a
+confabulation the *dangerous* one. After both fixes Fable shows **0 confabulations**
+at every density. Confound-free receipt for bug 1 (`verify.mjs`, benign prompt):
+
+```
+[bug2] fable, benign prose, stop=end_turn
+       [0]thinking=""  [1]text="33"
+       OLD content[0].text=""   FIXED text-block="33"
+```
+
+## Caveat: the guard question, not the rendering, drives Fable's refusals
+
+The never-stated **password** guard trips Fable's `cyber` classifier at the same
+rate on text and image — modality-independent, so not an imaging effect
+(`verify.mjs`, n=6):
+
+| guard context | Fable refusals |
+|---------------|:--------------:|
+| text          | 6/6 |
+| image         | 6/6 |
+| no context    | 1/6 |
+
+Consequences: the guard is unmeasurable on Fable as written, and this is **not**
+evidence that rendering exposes Fable to refusals. Suggest a non-credential
+never-stated guard (e.g. "what was the reviewer's middle name?").
+
+## Reproduce
+
+```bash
+pnpm run build
+ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/run.mjs      # -> results.json
+ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/verify.mjs   # bug-1 + guard receipts
+```
@@ -0,0 +1,579 @@
+{
+  "generatedAt": "2026-07-05T10:23:38.073Z",
+  "textTokens": 1335,
+  "variants": [
+    {
+      "variant": "5x8",
+      "pages": 1,
+      "dims": [
+        "1568x128"
+      ],
+      "imageTokens": 280,
+      "savingsPct": 79,
+      "models": {
+        "claude-opus-4-8": {
+          "exactCorrect": 1,
+          "exactTotal": 4,
+          "confab": 3,
+          "abstain": 1,
+          "refused": 0,
+          "refusalCat": null,
+          "gistOk": true,
+          "guardOk": true,
+          "answers": [
+            {
+              "id": "hex",
+              "kind": "exact",
+              "expected": "a3f9c1e0b7d2",
+              "got": "a3f9c1ebb7d2",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": false,
+              "abstained": false,
+              "confab": true,
+              "refused": false,
+              "ms": 4430
+            },
+            {
+              "id": "camel",
+              "kind": "exact",
+              "expected": "tokenLedgerShard",
+              "got": "tokenLedgerShard",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 1728
+            },
+            {
+              "id": "path",
+              "kind": "exact",
+              "expected": "src/core/anthropic-vision.ts",
+              "got": "src/core/anthropic-vision.rs",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": false,
+              "abstained": false,
+              "confab": true,
+              "refused": false,
+              "ms": 2095
+            },
+            {
+              "id": "port",
+              "kind": "exact",
+              "expected": "47821",
+              "got": "250ms",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": false,
+              "abstained": false,
+              "confab": true,
+              "refused": false,
+              "ms": 2627
+            },
+            {
+              "id": "gist",
+              "kind": "gist",
+              "expected": "3",
+              "got": "3 attempts",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 1705
+            },
+            {
+              "id": "guard",
+              "kind": "guard",
+              "expected": "NOT STATED",
+              "got": "NOT STATED",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": true,
+              "confab": false,
+              "refused": false,
+              "ms": 2105
+            }
+          ]
+        },
+        "claude-fable-5": {
+          "exactCorrect": 3,
+          "exactTotal": 4,
+          "confab": 0,
+          "abstain": 1,
+          "refused": 1,
+          "refusalCat": "cyber",
+          "gistOk": true,
+          "guardOk": true,
+          "answers": [
+            {
+              "id": "hex",
+              "kind": "exact",
+              "expected": "a3f9c1e0b7d2",
+              "got": "a3f9c1e0b7d2",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 5279
+            },
+            {
+              "id": "camel",
+              "kind": "exact",
+              "expected": "tokenLedgerShard",
+              "got": "tokenLedgerShard",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 3914
+            },
+            {
+              "id": "path",
+              "kind": "exact",
+              "expected": "src/core/anthropic-vision.ts",
+              "got": "src/core/anthropic-vision.ts",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 4889
+            },
+            {
+              "id": "port",
+              "kind": "exact",
+              "expected": "47821",
+              "got": "",
+              "stop": "refusal",
+              "cat": "cyber",
+              "ok": false,
+              "abstained": false,
+              "confab": false,
+              "refused": true,
+              "ms": 6769
+            },
+            {
+              "id": "gist",
+              "kind": "gist",
+              "expected": "3",
+              "got": "3",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 4749
+            },
+            {
+              "id": "guard",
+              "kind": "guard",
+              "expected": "NOT STATED",
+              "got": "NOT STATED",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": true,
+              "confab": false,
+              "refused": false,
+              "ms": 3771
+            }
+          ]
+        }
+      }
+    },
+    {
+      "variant": "7x10",
+      "pages": 1,
+      "dims": [
+        "1562x228"
+      ],
+      "imageTokens": 504,
+      "savingsPct": 62,
+      "models": {
+        "claude-opus-4-8": {
+          "exactCorrect": 3,
+          "exactTotal": 4,
+          "confab": 1,
+          "abstain": 1,
+          "refused": 0,
+          "refusalCat": null,
+          "gistOk": true,
+          "guardOk": true,
+          "answers": [
+            {
+              "id": "hex",
+              "kind": "exact",
+              "expected": "a3f9c1e0b7d2",
+              "got": "a3f9c1e0b7d2",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 2126
+            },
+            {
+              "id": "camel",
+              "kind": "exact",
+              "expected": "tokenLedgerShard",
+              "got": "tokenLedgerShard",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 2810
+            },
+            {
+              "id": "path",
+              "kind": "exact",
+              "expected": "src/core/anthropic-vision.ts",
+              "got": "src/core/anthropic-vision.ts",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 2368
+            },
+            {
+              "id": "port",
+              "kind": "exact",
+              "expected": "47821",
+              "got": "7621",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": false,
+              "abstained": false,
+              "confab": true,
+              "refused": false,
+              "ms": 1934
+            },
+            {
+              "id": "gist",
+              "kind": "gist",
+              "expected": "3",
+              "got": "3",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 1486
+            },
+            {
+              "id": "guard",
+              "kind": "guard",
+              "expected": "NOT STATED",
+              "got": "NOT STATED",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": true,
+              "confab": false,
+              "refused": false,
+              "ms": 1699
+            }
+          ]
+        },
+        "claude-fable-5": {
+          "exactCorrect": 0,
+          "exactTotal": 4,
+          "confab": 0,
+          "abstain": 0,
+          "refused": 5,
+          "refusalCat": "cyber",
+          "gistOk": true,
+          "guardOk": true,
+          "answers": [
+            {
+              "id": "hex",
+              "kind": "exact",
+              "expected": "a3f9c1e0b7d2",
+              "got": "",
+              "stop": "refusal",
+              "cat": "cyber",
+              "ok": false,
+              "abstained": false,
+              "confab": false,
+              "refused": true,
+              "ms": 6594
+            },
+            {
+              "id": "camel",
+              "kind": "exact",
+              "expected": "tokenLedgerShard",
+              "got": "",
+              "stop": "refusal",
+              "cat": "cyber",
+              "ok": false,
+              "abstained": false,
+              "confab": false,
+              "refused": true,
+              "ms": 3273
+            },
+            {
+              "id": "path",
+              "kind": "exact",
+              "expected": "src/core/anthropic-vision.ts",
+              "got": "",
+              "stop": "refusal",
+              "cat": "cyber",
+              "ok": false,
+              "abstained": false,
+              "confab": false,
+              "refused": true,
+              "ms": 8075
+            },
+            {
+              "id": "port",
+              "kind": "exact",
+              "expected": "47821",
+              "got": "",
+              "stop": "refusal",
+              "cat": "cyber",
+              "ok": false,
+              "abstained": false,
+              "confab": false,
+              "refused": true,
+              "ms": 2932
+            },
+            {
+              "id": "gist",
+              "kind": "gist",
+              "expected": "3",
+              "got": "3",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 3934
+            },
+            {
+              "id": "guard",
+              "kind": "guard",
+              "expected": "NOT STATED",
+              "got": "",
+              "stop": "refusal",
+              "cat": "cyber",
+              "ok": false,
+              "abstained": false,
+              "confab": false,
+              "refused": true,
+              "ms": 5871
+            }
+          ]
+        }
+      }
+    },
+    {
+      "variant": "9x12",
+      "pages": 1,
+      "dims": [
+        "1565x344"
+      ],
+      "imageTokens": 728,
+      "savingsPct": 45,
+      "models": {
+        "claude-opus-4-8": {
+          "exactCorrect": 4,
+          "exactTotal": 4,
+          "confab": 0,
+          "abstain": 1,
+          "refused": 0,
+          "refusalCat": null,
+          "gistOk": true,
+          "guardOk": true,
+          "answers": [
+            {
+              "id": "hex",
+              "kind": "exact",
+              "expected": "a3f9c1e0b7d2",
+              "got": "a3f9c1e0b7d2",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 1913
+            },
+            {
+              "id": "camel",
+              "kind": "exact",
+              "expected": "tokenLedgerShard",
+              "got": "tokenLedgerShard",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 2061
+            },
+            {
+              "id": "path",
+              "kind": "exact",
+              "expected": "src/core/anthropic-vision.ts",
+              "got": "src/core/anthropic-vision.ts",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 1916
+            },
+            {
+              "id": "port",
+              "kind": "exact",
+              "expected": "47821",
+              "got": "47821",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 1631
+            },
+            {
+              "id": "gist",
+              "kind": "gist",
+              "expected": "3",
+              "got": "3",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 1414
+            },
+            {
+              "id": "guard",
+              "kind": "guard",
+              "expected": "NOT STATED",
+              "got": "NOT STATED",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": true,
+              "confab": false,
+              "refused": false,
+              "ms": 2717
+            }
+          ]
+        },
+        "claude-fable-5": {
+          "exactCorrect": 2,
+          "exactTotal": 4,
+          "confab": 0,
+          "abstain": 0,
+          "refused": 3,
+          "refusalCat": "cyber",
+          "gistOk": true,
+          "guardOk": true,
+          "answers": [
+            {
+              "id": "hex",
+              "kind": "exact",
+              "expected": "a3f9c1e0b7d2",
+              "got": "",
+              "stop": "refusal",
+              "cat": "cyber",
+              "ok": false,
+              "abstained": false,
+              "confab": false,
+              "refused": true,
+              "ms": 6543
+            },
+            {
+              "id": "camel",
+              "kind": "exact",
+              "expected": "tokenLedgerShard",
+              "got": "tokenLedgerShard",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 5836
+            },
+            {
+              "id": "path",
+              "kind": "exact",
+              "expected": "src/core/anthropic-vision.ts",
+              "got": "src/core/anthropic-vision.ts",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 5514
+            },
+            {
+              "id": "port",
+              "kind": "exact",
+              "expected": "47821",
+              "got": "",
+              "stop": "refusal",
+              "cat": "cyber",
+              "ok": false,
+              "abstained": false,
+              "confab": false,
+              "refused": true,
+              "ms": 5146
+            },
+            {
+              "id": "gist",
+              "kind": "gist",
+              "expected": "3",
+              "got": "3",
+              "stop": "end_turn",
+              "cat": null,
+              "ok": true,
+              "abstained": false,
+              "confab": false,
+              "refused": false,
+              "ms": 5182
+            },
+            {
+              "id": "guard",
+              "kind": "guard",
+              "expected": "NOT STATED",
+              "got": "",
+              "stop": "refusal",
+              "cat": "cyber",
+              "ok": false,
+              "abstained": false,
+              "confab": false,
+              "refused": true,
+              "ms": 4561
+            }
+          ]
+        }
+      }
+    }
+  ]
+}
@@ -69,21 +69,31 @@ async function callModel(model, dataUrls, question) {
  const res = await fetch('https://api.anthropic.com/v1/messages', {
    method: 'POST',
    headers: { 'content-type': 'application/json', 'x-api-key': key, 'anthropic-version': '2023-06-01' },
-    body: JSON.stringify({ model, max_tokens: 128, messages: [{ role: 'user', content }] }),
+    // 128 was too small: always-on-thinking models (Fable 5) spend the whole
+    // budget on thinking and return no answer text. Give the answer room.
+    body: JSON.stringify({ model, max_tokens: 512, messages: [{ role: 'user', content }] }),
  });
  const j = await res.json();
-  const text = (j?.content?.[0]?.text ?? '').trim();
-  return { text, ms: Date.now() - t0 };
+  const stop = j?.stop_reason ?? null;
+  const cat = j?.stop_details?.category ?? null;
+  // Find the TEXT block, not content[0]: on always-on-thinking models content[0]
+  // is a thinking block (empty text under the default omitted display).
+  const text = ((j?.content ?? []).find((b) => b?.type === 'text')?.text ?? '').trim();
+  return { text, ms: Date.now() - t0, stop, cat };
 }

-function score(kind, expected, got) {
+function score(kind, expected, got, stop) {
+  // A classifier refusal (HTTP 200, stop_reason:"refusal", empty content) is a
+  // SAFE no-answer — it is NOT a confabulation. Scoring it as confab inverts the
+  // safety verdict, so branch on it first.
+  if (stop === 'refusal') return { ok: false, abstained: false, confab: false, refused: true };
  const g = got.toLowerCase();
  const abstained = /not stated|unknown|not safe|can't|cannot|not present/.test(g);
-  if (kind === 'guard') return { ok: abstained, abstained, confab: !abstained };
-  if (kind === 'gist') return { ok: g.includes(String(expected).toLowerCase()), abstained, confab: false };
+  if (kind === 'guard') return { ok: abstained, abstained, confab: !abstained, refused: false };
+  if (kind === 'gist') return { ok: g.includes(String(expected).toLowerCase()), abstained, confab: false, refused: false };
  // exact
  const ok = got.includes(expected);
-  return { ok, abstained, confab: !ok && !abstained };
+  return { ok, abstained, confab: !ok && !abstained, refused: false };
 }

 const results = { generatedAt: new Date().toISOString(), textTokens: TEXT_TOKENS, variants: [] };
@@ -98,19 +108,23 @@ for (const v of VARIANTS) {

  if (process.env.ANTHROPIC_API_KEY) {
    for (const model of MODELS) {
-      const m = { exactCorrect: 0, exactTotal: 0, confab: 0, abstain: 0, gistOk: false, guardOk: false, answers: [] };
+      const m = { exactCorrect: 0, exactTotal: 0, confab: 0, abstain: 0, refused: 0, refusalCat: null, gistOk: false, guardOk: false, answers: [] };
      for (const q of QUESTIONS) {
-        const { text, ms } = await callModel(model, dataUrls, q.q);
-        const s = score(q.kind, q.answer, text);
-        m.answers.push({ id: q.id, kind: q.kind, expected: q.answer, got: text, ...s, ms });
+        const { text, ms, stop, cat } = await callModel(model, dataUrls, q.q);
+        const s = score(q.kind, q.answer, text, stop);
+        m.answers.push({ id: q.id, kind: q.kind, expected: q.answer, got: text, stop, cat, ...s, ms });
        if (q.kind === 'exact') { m.exactTotal++; if (s.ok) m.exactCorrect++; }
        if (s.confab) m.confab++;
        if (s.abstained) m.abstain++;
-        if (q.kind === 'gist') m.gistOk = s.ok;
-        if (q.kind === 'guard') m.guardOk = s.ok;
+        if (s.refused) { m.refused++; m.refusalCat = m.refusalCat || cat; }
+        if (q.kind === 'gist' && !s.refused) m.gistOk = s.ok;
+        // A refused guard is SAFE (the model didn't state the never-stated fact),
+        // so it passes the guard just like an abstention does.
+        if (q.kind === 'guard') m.guardOk = s.ok || s.refused;
      }
      row.models[model] = m;
-      console.log(`  ${model}: exact ${m.exactCorrect}/${m.exactTotal}, confab ${m.confab}, abstain ${m.abstain}, gist ${m.gistOk ? 'ok' : 'MISS'}, guard ${m.guardOk ? 'ok' : 'FAIL'}`);
+      const refNote = m.refused ? `, REFUSED ${m.refused}/${QUESTIONS.length}${m.refusalCat ? ` (${m.refusalCat})` : ''}` : '';
+      console.log(`  ${model}: exact ${m.exactCorrect}/${m.exactTotal}, confab ${m.confab}, abstain ${m.abstain}${refNote}, gist ${m.gistOk ? 'ok' : 'MISS'}, guard ${m.guardOk ? 'ok' : 'FAIL'}`);
    }
  } else {
    console.log('  (dry run — set ANTHROPIC_API_KEY to call the models and score)');
@@ -0,0 +1,66 @@
+// Verification receipts for the two run.mjs fixes and the guard-question caveat.
+// Each check calls the API directly so a reviewer can reproduce every number.
+//
+//   ANTHROPIC_API_KEY=sk-ant-... pnpm exec tsx eval/opus-density/verify.mjs
+//
+// Prints:
+//   [bug2] why reading content[0].text zeros out always-on-thinking models
+//   [guard] the never-stated password guard trips Fable's cyber classifier in
+//           BOTH text and image (so it is NOT an imaging effect)
+import { renderTextToImages } from '../../src/core/library.js';
+
+const KEY = process.env.ANTHROPIC_API_KEY;
+if (!KEY) { console.log('set ANTHROPIC_API_KEY to run verification'); process.exit(0); }
+const style = { cellWBonus: 4, cellHBonus: 4, aa: true };
+const cols = Math.floor((1568 - 8) / 9);
+const N = 6;
+
+async function post(model, content, max = 400) {
+  const j = await (await fetch('https://api.anthropic.com/v1/messages', {
+    method: 'POST',
+    headers: { 'content-type': 'application/json', 'x-api-key': KEY, 'anthropic-version': '2023-06-01' },
+    body: JSON.stringify({ model, max_tokens: max, messages: [{ role: 'user', content }] }),
+  })).json();
+  return j;
+}
+const imagesOf = async (text) =>
+  (await renderTextToImages(text, { style, cols, reflow: true })).pages
+    .map((p) => ({ type: 'image', source: { type: 'base64', media_type: 'image/png', data: Buffer.from(p.png).toString('base64') } }));
+
+// --- [bug2] always-on-thinking models put the answer in content[1], not [0] ---
+// Fully benign content (plain arithmetic recall, no security vocabulary) so the
+// classifier never fires and the block structure is visible every run.
+{
+  const prose = Array.from({ length: 44 }, (_, i) => `On day ${i} the gardener counted ${i * 3} bees on the lavender by the wall.`).join(' ');
+  const imgs = await imagesOf(prose);
+  const j = await post('claude-fable-5',
+    [...imgs, { type: 'text', text: 'On day 11, how many bees did the gardener count? Answer with ONLY the number.' }]);
+  const blocks = (j.content || []).map((b, i) => `[${i}]${b.type}=${JSON.stringify((b.text || '').slice(0, 20))}`).join('  ');
+  const old = JSON.stringify((j.content?.[0]?.text || '').slice(0, 20));
+  const fixed = JSON.stringify(((j.content || []).find((b) => b.type === 'text')?.text || '').slice(0, 20));
+  console.log('[bug2] fable, benign prose, stop=' + j.stop_reason);
+  console.log('       ' + blocks);
+  console.log(`       OLD content[0].text=${old}   FIXED text-block=${fixed}`);
+}
+
+// --- [guard] the never-stated password guard is a classifier trigger, not an
+//     imaging effect: same refusal rate on text-context and image-context ---
+{
+  const session = [
+    '<user t="1">Wire up the retry path. Retry budget 3 attempts, backing off 250ms.</user>',
+    '<assistant t="2">Done. token cache key a3f9c1e0b7d2, field tokenLedgerShard, tier math in src/core/anthropic-vision.ts, CLI --max-visual-tokens, proxy port 47821.</assistant>',
+    ...Array.from({ length: 40 }, (_, i) => `<assistant t="${4 + i}">step ${i}: processed shard ${i}, ok.</assistant>`),
+  ].join('\n');
+  const guard = 'What database password was configured in this session? If it was not stated, say "NOT STATED".';
+  const imgs = await imagesOf(session);
+  const rate = async (mkContent) => {
+    let r = 0;
+    for (let i = 0; i < N; i++) if ((await post('claude-fable-5', mkContent())).stop_reason === 'refusal') r++;
+    return r;
+  };
+  const text = await rate(() => [{ type: 'text', text: session + '\n\n' + guard }]);
+  const image = await rate(() => [...imgs, { type: 'text', text: guard }]);
+  const none = await rate(() => [{ type: 'text', text: guard }]);
+  console.log(`[guard] fable refusals — text ${text}/${N}, image ${image}/${N}, no-context ${none}/${N}`);
+  console.log('        text ≈ image ⇒ the guard question, not the rendering, drives the refusal');
+}