mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
feat(render): use 5x8 code-font atlas
This commit is contained in:
Steven Chong
@@ -97,11 +97,12 @@ Claude Code ──► pixelpipe ──► │ (system + tools as text)
|
||||
|
||||
The proxy intercepts `POST /v1/messages`, pulls the system prompt + tool
|
||||
documentation out of the JSON body, renders it into one or more grayscale
|
||||
PNGs using a build-time-generated GNU Unifont glyph atlas (~35k BMP
|
||||
codepoints by default — Latin, Cyrillic, Greek, CJK, Hiragana, Katakana,
|
||||
Hangul, Hebrew, Arabic, math symbols, box drawing, decorative symbols),
|
||||
and substitutes those PNGs back in as `image` content blocks with an
|
||||
`ephemeral` `cache_control` breakpoint.
|
||||
PNGs using a build-time-generated hybrid glyph atlas: Spleen 5×8 for
|
||||
printable ASCII/code glyphs, with GNU Unifont 8px fallback for ~35k BMP
|
||||
codepoints by default — Latin extended, Cyrillic, Greek, CJK, Hiragana,
|
||||
Katakana, Hangul, Hebrew, Arabic, math symbols, box drawing, decorative
|
||||
symbols. It substitutes those PNGs back in as `image` content blocks with
|
||||
an `ephemeral` `cache_control` breakpoint.
|
||||
|
||||
Three independent derivations, each anchored on a number you can verify
|
||||
against the source.
|
||||
@@ -131,21 +132,20 @@ billed.
|
||||
|
||||
The "English prose ≈ 4 chars/token" rule from Anthropic's
|
||||
[pricing docs](https://docs.anthropic.com/en/docs/about-claude/pricing)
|
||||
does not survive contact with real Claude Code traffic. Across N=354
|
||||
production `count_tokens` probes on real `/v1/messages` bodies:
|
||||
does not survive contact with real Claude Code traffic. Across N=391
|
||||
production `count_tokens` probes on Opus 4.7 `/v1/messages` bodies:
|
||||
|
||||
```
|
||||
median 1.17 chars/token
|
||||
p75 1.19 chars/token
|
||||
p95 2.50 chars/token
|
||||
max 2.62 chars/token
|
||||
avg outgoing text chars 231,925
|
||||
avg real input tokens 115,893
|
||||
observed mean 1.91 chars/token
|
||||
```
|
||||
|
||||
Real bodies are JSON-dense — tool definitions, schemas, structured
|
||||
`CLAUDE.md` slabs, `tool_result` blocks — which tokenize 2-4× denser than
|
||||
prose. The gate `isCompressionProfitable()` uses
|
||||
`SLAB_CHARS_PER_TOKEN = 2.5` at the slab call site (the 95th percentile of
|
||||
observed real cpt with a small safety margin), so it only compresses when
|
||||
`SLAB_CHARS_PER_TOKEN = 2.0` at the slab call site (slightly conservative
|
||||
versus the observed 1.91 cpt), so it only compresses when
|
||||
the text actually costs more tokens than the image will. At the textbook
|
||||
4 ch/tok the gate silently rejects every realistic slab as
|
||||
`not_profitable` — that bug is what motivates the constant.
|
||||
@@ -307,8 +307,9 @@ recognizing font family/style. For pixelpipe this means:
|
||||
- There is no separate letter-spacing knob — density is mostly the atlas
|
||||
cell (`ATLAS_CELL_W × ATLAS_CELL_H`).
|
||||
- The promising foundational experiment is a denser-but-readable bitmap
|
||||
atlas (e.g. current Unifont 5×11 → candidate 4×8 or 4×7), with exact
|
||||
retrieval tests on code/tool-doc slabs before making it default. Avoid
|
||||
atlas. We now ship a conservative version of that idea: Spleen 5×8 for
|
||||
ASCII/code plus Unifont 8px fallback, after 4×8 proved too brittle in
|
||||
exact code-reading tests. Avoid
|
||||
jumping to ~6 px effective text height without quality evidence.
|
||||
- Gutter/padding changes are secondary; the gutter is an OCR-ordering cue
|
||||
for multi-column layouts, so removing it can save pixels while silently
|
||||
@@ -448,9 +449,8 @@ prose. The N=10 rejected history events in `events.jsonl` had real cpt
|
||||
1.08–1.10 — every one of them was a profitable compression the gate
|
||||
dropped on the floor.
|
||||
|
||||
Fixed by wiring `HISTORY_CHARS_PER_TOKEN = 2.5` at the call site (same
|
||||
shape as the slab's `SLAB_CHARS_PER_TOKEN = 2.5` fix from `e8545a9`
|
||||
earlier today). Live data after restart shows it firing: the 12:30:01
|
||||
Fixed by wiring `HISTORY_CHARS_PER_TOKEN = 2.0` at the call site (same
|
||||
shape as the slab's `SLAB_CHARS_PER_TOKEN = 2.0` Opus-4.7 calibration). Live data after restart shows it firing: the 12:30:01
|
||||
event in `events.jsonl` has `historyReason: 'collapsed'`,
|
||||
`collapsed_turns: 175`, `collapsed_chars: 180,684`.
|
||||
|
||||
@@ -738,38 +738,40 @@ src/
|
||||
└── worker.ts export default { fetch }
|
||||
|
||||
scripts/
|
||||
├── gen-atlas.ts build-time: OTF → atlas.ts (uses @napi-rs/canvas)
|
||||
├── gen-atlas.ts build-time: font files → atlas.ts (uses @napi-rs/canvas)
|
||||
└── build.mjs esbuild bundler for Node target
|
||||
|
||||
assets/
|
||||
├── Unifont-16.0.04.otf primary font (~35k BMP codepoints w/ full-bmp profile)
|
||||
├── Spleen-5x8.otb primary ASCII/code bitmap font (BSD-2-Clause)
|
||||
├── SPLEEN_LICENSE.txt Spleen license
|
||||
├── Unifont-16.0.04.otf Unicode fallback (~35k BMP codepoints w/ full-bmp profile)
|
||||
├── UNIFONT_LICENSE.txt OFL + GPL-with-font-exception
|
||||
└── JetBrainsMono-Regular.ttf legacy / ASCII-only fallback (kept on disk)
|
||||
```
|
||||
|
||||
The atlas is generated **at build time** from `Unifont-16.0.04.otf`,
|
||||
base64-inlined into a `.ts` file with sparse codepoint + offset tables
|
||||
(binary-packed), and shipped with the bundle. At runtime there are zero
|
||||
external files to read and zero non-Web-Standard imports — that's the
|
||||
only way this works in Workers without per-request asset fetches.
|
||||
The atlas is generated **at build time** from `Spleen-5x8.otb` (printable
|
||||
ASCII/code) plus `Unifont-16.0.04.otf` (Unicode fallback), base64-inlined
|
||||
into a `.ts` file with sparse codepoint + offset tables (binary-packed),
|
||||
and shipped with the bundle. At runtime there are zero external files to
|
||||
read and zero non-Web-Standard imports — that's the only way this works
|
||||
in Workers without per-request asset fetches.
|
||||
|
||||
Regenerate the atlas (after swapping fonts, sizes, or codepoint profile):
|
||||
|
||||
```bash
|
||||
pnpm run build:atlas # default: full-bmp (~35k cp, all BMP Unifont covers)
|
||||
pnpm run build:atlas # default: Spleen 5×8 ASCII + full-bmp Unifont fallback
|
||||
ATLAS_PROFILE=practical pnpm run build:atlas # drops Hangul (~24k cp; for Workers free-tier)
|
||||
FONT_PX=12 pnpm run build:atlas # nondefault size; verify cells
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
- The bundled GNU Unifont at 10px (cell 5×11 px Latin, 10×11 CJK) is
|
||||
Anthropic-OCR-clean for ~35k BMP codepoints by default (`full-bmp`
|
||||
profile): Latin, Cyrillic, Greek, CJK Unified Ideographs, Hiragana,
|
||||
Katakana, Hangul, Hebrew, Arabic, math symbols, box-drawing, arrows,
|
||||
Dingbats, Letterlike Symbols, Enclosed Alphanumerics, etc. Drops for
|
||||
- The bundled hybrid atlas uses Spleen 5×8 for printable ASCII/code and
|
||||
Unifont 8px fallback for ~35k BMP codepoints by default (`full-bmp`
|
||||
profile): Latin extended, Cyrillic, Greek, CJK Unified Ideographs,
|
||||
Hiragana, Katakana, Hangul, Hebrew, Arabic, math symbols, box-drawing,
|
||||
arrows, Dingbats, Letterlike Symbols, Enclosed Alphanumerics, etc. Drops for
|
||||
codepoints outside the profile (e.g. emoji 😀 — supplementary plane)
|
||||
get counted in `events.jsonl#dropped_chars` (with the top-20 broken
|
||||
out as `dropped_codepoints_top`) so you can spot patterns. For
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
Copyright (c) 2018-2026, Frederic Cambus
|
||||
All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions are met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
|
||||
* Redistributions in binary form must reproduce the above copyright
|
||||
notice, this list of conditions and the following disclaimer in the
|
||||
documentation and/or other materials provided with the distribution.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
||||
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
||||
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS
|
||||
BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
|
||||
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
|
||||
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
||||
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
|
||||
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
|
||||
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
|
||||
POSSIBILITY OF SUCH DAMAGE.
|
||||
Binary file not shown.
+142
-170
@@ -1,34 +1,23 @@
|
||||
/**
|
||||
* Build-time glyph atlas generator — Unicode-capable variant.
|
||||
* Build-time glyph atlas generator — Unicode-capable hybrid code-font atlas.
|
||||
*
|
||||
* Reads assets/Unifont-16.0.04.otf, rasterizes a configurable subset of
|
||||
* Unicode BMP into a sparse-indexed grayscale atlas, and emits
|
||||
* src/core/atlas.ts with binary-packed lookup tables inlined as base64.
|
||||
* Default atlas (2026-05): Spleen 5×8 for printable ASCII/code glyphs,
|
||||
* Unifont 16.0.04 at 8px for Unicode fallback (CJK, arrows, symbols,
|
||||
* math, Hangul, etc.). The runtime renderer still sees one sparse atlas:
|
||||
* codepoint → bit offset + wide flag. There is no runtime font dependency.
|
||||
*
|
||||
* Why Unifont:
|
||||
* - Single font covers ~57k BMP codepoints (Latin + Cyrillic + CJK + Hira +
|
||||
* Kata + Greek + Hebrew + Arabic + box-drawing + math symbols + …).
|
||||
* - Pixel-perfect at 10px (it's a bitmap font designed for that grid).
|
||||
* - East Asian Width property respected by @napi-rs/canvas — CJK measures as
|
||||
* 2× Latin width. The renderer assumes this; gen-atlas asserts it.
|
||||
* - License: OFL + GPL-with-font-exception. Ship-friendly.
|
||||
*
|
||||
* Why sparse:
|
||||
* - The contiguous-range approach (FIRST..LAST) doesn't work when we want
|
||||
* widely-separated blocks (e.g. ASCII + Cyrillic + CJK). A sparse codepoint
|
||||
* list + binary search at render time is the natural shape.
|
||||
* Why hybrid instead of replacing Unifont outright:
|
||||
* - Spleen 5×8 is a real bitmap/code font and is materially denser than
|
||||
* Unifont 10px: 5×8 cells vs 5×11, ~38% more rows per 1568px image.
|
||||
* - Spleen intentionally targets small code/terminal glyphs, but does not
|
||||
* cover CJK/symbol blocks. Unifont remains the broad fallback so existing
|
||||
* dropped-glyph behavior does not regress for non-ASCII text.
|
||||
* - The generator bakes both into one 1-bit atlas, preserving the Workers-safe
|
||||
* zero-runtime-dependency contract.
|
||||
*
|
||||
* Profiles (selected via ATLAS_PROFILE env, default 'full-bmp'):
|
||||
* - 'full-bmp' (~35k codepoints): everything Unifont covers in BMP,
|
||||
* including Hangul. Default. Bundle exceeds the Cloudflare Workers
|
||||
* free-tier 1 MB compressed-bundle cap; suitable for Node and paid
|
||||
* Workers tier deployments.
|
||||
* - 'practical' (~24k codepoints): drops Hangul Syllables to fit Workers
|
||||
* free-tier. Use via `ATLAS_PROFILE=practical pnpm run build:atlas`
|
||||
* when deploying under the 1 MB compressed-bundle cap.
|
||||
*
|
||||
* Runs only at build time. Has zero runtime deps — the generated atlas.ts
|
||||
* works identically in Node and Cloudflare Workers.
|
||||
* - 'full-bmp' (~35k codepoints): practical Unicode blocks + Hangul.
|
||||
* - 'practical' (~24k codepoints): drops Hangul Syllables for smaller bundles.
|
||||
*/
|
||||
|
||||
import { GlobalFonts, createCanvas } from '@napi-rs/canvas';
|
||||
@@ -36,19 +25,18 @@ import { readFileSync, writeFileSync } from 'node:fs';
|
||||
import { resolve } from 'node:path';
|
||||
|
||||
const ROOT = resolve(import.meta.dirname, '..');
|
||||
const OTF_PATH = resolve(ROOT, 'assets/Unifont-16.0.04.otf');
|
||||
const PRIMARY_FONT_PATH = resolve(ROOT, 'assets/Spleen-5x8.otb');
|
||||
const FALLBACK_FONT_PATH = resolve(ROOT, 'assets/Unifont-16.0.04.otf');
|
||||
const OUT_PATH = resolve(ROOT, 'src/core/atlas.ts');
|
||||
|
||||
const FONT_FAMILY = 'Unifont';
|
||||
const FONT_PX = Number(process.env.FONT_PX ?? 10);
|
||||
// Default to `full-bmp` so Korean / decorative blocks / math symbols all
|
||||
// ship by default. `ATLAS_PROFILE=practical` is the Workers-free-tier
|
||||
// escape hatch for deployments under the 1 MB compressed-bundle cap.
|
||||
const PRIMARY_FONT_FAMILY = 'Spleen';
|
||||
const FALLBACK_FONT_FAMILY = 'Unifont';
|
||||
const PRIMARY_FONT_PX = 8;
|
||||
const FALLBACK_FONT_PX = 8;
|
||||
const FONT_FAMILY_LABEL = 'Spleen 5x8 ASCII + Unifont 8px fallback';
|
||||
const PROFILE = (process.env.ATLAS_PROFILE ?? 'full-bmp') as 'practical' | 'full-bmp';
|
||||
|
||||
/** Codepoint blocks included in each profile. The order doesn't affect
|
||||
* correctness (we sort by codepoint before emitting); it's just a
|
||||
* readable build log. */
|
||||
/** Codepoint blocks included in each profile. */
|
||||
const PRACTICAL_RANGES: ReadonlyArray<readonly [number, number, string]> = [
|
||||
[0x0020, 0x007e, 'ASCII printable'],
|
||||
[0x00a0, 0x024f, 'Latin-1 Supp + Latin Extended-A + Latin Extended-B'],
|
||||
@@ -57,9 +45,6 @@ const PRACTICAL_RANGES: ReadonlyArray<readonly [number, number, string]> = [
|
||||
[0x0590, 0x05ff, 'Hebrew'],
|
||||
[0x0600, 0x06ff, 'Arabic'],
|
||||
[0x2000, 0x206f, 'General Punctuation'],
|
||||
// Added based on production drop histogram (#27 + #28): 95% of live drops
|
||||
// fall in these seven blocks — mostly decorative symbols, math notation,
|
||||
// and info bullets Claude Code's tool output uses heavily (✓ ✗ ⚠ ℝ ℕ ℤ ⓘ).
|
||||
[0x2100, 0x214f, 'Letterlike Symbols'],
|
||||
[0x2190, 0x21ff, 'Arrows'],
|
||||
[0x2200, 0x22ff, 'Mathematical Operators'],
|
||||
@@ -83,176 +68,173 @@ const HANGUL: ReadonlyArray<readonly [number, number, string]> = [
|
||||
|
||||
const RANGES = PROFILE === 'full-bmp' ? [...PRACTICAL_RANGES, ...HANGUL] : PRACTICAL_RANGES;
|
||||
|
||||
// --- Register the font -----------------------------------------------------
|
||||
const otfBytes = readFileSync(OTF_PATH);
|
||||
GlobalFonts.register(otfBytes, FONT_FAMILY);
|
||||
// --- Register fonts --------------------------------------------------------
|
||||
GlobalFonts.register(readFileSync(PRIMARY_FONT_PATH), PRIMARY_FONT_FAMILY);
|
||||
GlobalFonts.register(readFileSync(FALLBACK_FONT_PATH), FALLBACK_FONT_FAMILY);
|
||||
|
||||
// --- Probe cell dimensions -------------------------------------------------
|
||||
// Use a measurement canvas large enough to hold a wide glyph + descenders.
|
||||
// Spleen 5x8 defines the global Latin cell. Unifont fallback at 8px is
|
||||
// narrower (4px Latin / 8px CJK) and is painted into the same 5/10px cells,
|
||||
// leaving a little right-side blank space rather than changing renderer math.
|
||||
const probe = createCanvas(64, 64);
|
||||
const pctx = probe.getContext('2d');
|
||||
pctx.font = `${FONT_PX}px ${FONT_FAMILY}`;
|
||||
pctx.textBaseline = 'alphabetic';
|
||||
|
||||
// --- cell dimensions: derive from probe data, not hardcoded ---------------
|
||||
//
|
||||
// Latin advance ('M') and CJK advance ('中') give the visual cell widths.
|
||||
// East Asian Width = Wide requires CJK = 2 × Latin EXACTLY — verified on the
|
||||
// UNROUNDED floats (Unifont at 11px reports Latin=5.5, CJK=11.0; that's a
|
||||
// clean 2× ratio that integer rounding would falsely reject as 6 ≠ 11). Use
|
||||
// Math.ceil for the cell width so half-pixel sizes round UP and CJK glyphs
|
||||
// don't clip on the right edge.
|
||||
//
|
||||
// Cell height: probe a representative set of glyphs covering Latin caps,
|
||||
// Latin descenders, CJK extremes (tallest + lowest), and box-drawing /
|
||||
// math symbols. Take the max ascent and max descent across the set; cellH =
|
||||
// ceil(maxAscent + maxDescent). At 10px this works out to 7 + 2 = 9,
|
||||
// matching the previous hardcoded value byte-for-byte; at 12px it gives
|
||||
// 8 + 2 = 10; at 16px it gives 10 + 6 = 16 (CJK descenders kick in).
|
||||
const mLatin = pctx.measureText('M');
|
||||
const mCjk = pctx.measureText('中');
|
||||
const ratio = mCjk.width / mLatin.width;
|
||||
// Allow tiny float drift (e.g. 1.99999 vs 2.0) but reject anything that
|
||||
// clearly isn't East Asian Wide. 0.01 absolute tolerance is generous.
|
||||
if (Math.abs(ratio - 2) > 0.01) {
|
||||
throw new Error(
|
||||
`[gen-atlas] expected CJK advance = 2×Latin, got ratio=${ratio.toFixed(4)} ` +
|
||||
`(latin=${mLatin.width}, cjk=${mCjk.width}). ` +
|
||||
`Renderer assumes East Asian Width = Wide for CJK Unified Ideographs.`,
|
||||
);
|
||||
}
|
||||
const cellW = Math.ceil(mLatin.width);
|
||||
|
||||
// Probe glyphs covering the extremes we need to cover:
|
||||
// - 'M' : Latin caps (mid ascent, no descent)
|
||||
// - 'gpy' : Latin descenders (deepest in Latin)
|
||||
// - '中漢國' : CJK ascent ceiling
|
||||
// - '⌊∫' : math + box-drawing descenders
|
||||
// We measure each, take max(ascent), max(descent), and round up.
|
||||
const heightProbes = ['M', 'g', 'p', 'y', 'j', '中', '漢', '國', '⌊', '∫', '日', 'カ'];
|
||||
pctx.font = `${PRIMARY_FONT_PX}px ${PRIMARY_FONT_FAMILY}`;
|
||||
const primaryLatin = pctx.measureText('M');
|
||||
let maxAscent = 0;
|
||||
let maxDescent = 0;
|
||||
for (const ch of heightProbes) {
|
||||
for (const ch of ['M', 'g', 'p', 'y', 'j', '0', 'O', 'l', 'I', '{', '}', '[', ']']) {
|
||||
const m = pctx.measureText(ch);
|
||||
// actualBoundingBoxAscent / Descent give the inked extent of the glyph
|
||||
// (in pixels above / below the baseline). Some glyphs (box-drawing) have
|
||||
// NEGATIVE descent (they sit above baseline); clamp to 0 for those.
|
||||
const asc = m.actualBoundingBoxAscent;
|
||||
const desc = m.actualBoundingBoxDescent;
|
||||
if (Number.isFinite(asc) && asc > maxAscent) maxAscent = asc;
|
||||
if (Number.isFinite(desc) && desc > maxDescent) maxDescent = desc;
|
||||
if (Number.isFinite(m.actualBoundingBoxAscent) && m.actualBoundingBoxAscent > maxAscent) {
|
||||
maxAscent = m.actualBoundingBoxAscent;
|
||||
}
|
||||
if (Number.isFinite(m.actualBoundingBoxDescent) && m.actualBoundingBoxDescent > maxDescent) {
|
||||
maxDescent = m.actualBoundingBoxDescent;
|
||||
}
|
||||
}
|
||||
// Also verify fallback fits in the same height budget at its own baseline.
|
||||
pctx.font = `${FALLBACK_FONT_PX}px ${FALLBACK_FONT_FAMILY}`;
|
||||
for (const ch of ['M', 'g', 'p', 'y', 'j', '中', '漢', '國', '⌊', '∫', '日', 'カ', '한']) {
|
||||
const m = pctx.measureText(ch);
|
||||
if (Number.isFinite(m.actualBoundingBoxAscent) && m.actualBoundingBoxAscent > maxAscent) {
|
||||
maxAscent = m.actualBoundingBoxAscent;
|
||||
}
|
||||
if (Number.isFinite(m.actualBoundingBoxDescent) && m.actualBoundingBoxDescent > maxDescent) {
|
||||
maxDescent = m.actualBoundingBoxDescent;
|
||||
}
|
||||
}
|
||||
|
||||
const cellW = Math.ceil(primaryLatin.width); // 5 at Spleen 8px
|
||||
const ascent = Math.ceil(maxAscent); // 7 for Spleen/Unifont 8px
|
||||
const descent = Math.ceil(maxDescent); // 1 for Spleen/Unifont 8px
|
||||
const cellH = ascent + descent; // 8
|
||||
|
||||
if (cellW !== 5 || cellH !== 8) {
|
||||
throw new Error(
|
||||
`[gen-atlas] Spleen 5x8 invariant drifted: got cell=${cellW}×${cellH} ` +
|
||||
`(asc=${ascent} desc=${descent}). Refusing to silently change density.`,
|
||||
);
|
||||
}
|
||||
|
||||
// Probe fallback advance against fallback's own Latin baseline. We classify
|
||||
// fallback glyphs as one or two visual cells, then paint them into Spleen's
|
||||
// 5px or 10px cell width. This keeps CJK wrapping at 2 cells even though
|
||||
// Unifont 8px ink itself is 8px wide.
|
||||
pctx.font = `${FALLBACK_FONT_PX}px ${FALLBACK_FONT_FAMILY}`;
|
||||
const fallbackLatinW = pctx.measureText('M').width;
|
||||
if (!Number.isFinite(fallbackLatinW) || fallbackLatinW <= 0) {
|
||||
throw new Error('[gen-atlas] could not measure fallback Unifont Latin width');
|
||||
}
|
||||
const ascent = Math.ceil(maxAscent);
|
||||
const descent = Math.ceil(maxDescent);
|
||||
const cellH = ascent + descent;
|
||||
|
||||
console.log(
|
||||
`[gen-atlas] font=${FONT_FAMILY} px=${FONT_PX} profile=${PROFILE} ` +
|
||||
`[gen-atlas] font=${FONT_FAMILY_LABEL} profile=${PROFILE} ` +
|
||||
`cell=${cellW}×${cellH} (asc=${ascent} desc=${descent}, wide=${2 * cellW}×${cellH})`,
|
||||
);
|
||||
|
||||
// --- Enumerate the codepoint set ------------------------------------------
|
||||
// For each range, walk every codepoint and keep only those Unifont actually
|
||||
// has a glyph for. @napi-rs/canvas returns 0 for codepoints not in the cmap;
|
||||
// we use that as the absence test. Then categorize by advance width: narrow
|
||||
// (== cellW) or wide (== 2*cellW). Anything else means a font version drift
|
||||
// — fail loudly rather than silently corrupt the atlas.
|
||||
|
||||
interface Found {
|
||||
cp: number;
|
||||
wide: boolean;
|
||||
source: 'primary' | 'fallback';
|
||||
}
|
||||
|
||||
function sourceForCodepoint(cp: number): 'primary' | 'fallback' {
|
||||
// Code-font-first means the exact ASCII/code glyphs that dominate Claude
|
||||
// Code prompts use Spleen. Unicode punctuation/symbols/CJK keep Unifont.
|
||||
if (cp >= 0x20 && cp <= 0x7e) return 'primary';
|
||||
return 'fallback';
|
||||
}
|
||||
|
||||
function classifyFallbackWidth(cp: number): boolean | null {
|
||||
const ch = String.fromCodePoint(cp);
|
||||
const w = pctx.measureText(ch).width;
|
||||
if (!Number.isFinite(w) || w <= 0) return null;
|
||||
const ratio = w / fallbackLatinW;
|
||||
if (Math.abs(ratio - 1) < 0.05) return false;
|
||||
if (Math.abs(ratio - 2) < 0.05) return true;
|
||||
throw new Error(
|
||||
`[gen-atlas] fallback codepoint U+${cp.toString(16).toUpperCase()} has advance ` +
|
||||
`${w}px (Unifont Latin=${fallbackLatinW}px, ratio=${ratio.toFixed(3)}; expected 1× or 2×).`,
|
||||
);
|
||||
}
|
||||
|
||||
const found: Found[] = [];
|
||||
for (const [lo, hi, label] of RANGES) {
|
||||
let kept = 0;
|
||||
let primary = 0;
|
||||
let fallback = 0;
|
||||
for (let cp = lo; cp <= hi; cp++) {
|
||||
const w = pctx.measureText(String.fromCodePoint(cp)).width;
|
||||
if (w === 0) continue; // not in cmap
|
||||
// Classify by raw advance ratio against the Latin baseline. Tolerant of
|
||||
// half-pixel drift (Unifont at odd sizes reports e.g. 5.5 / 11.0). Round
|
||||
// up to integer cells so glyphs never get clipped on the right edge.
|
||||
const ratioToLatin = w / mLatin.width;
|
||||
if (Math.abs(ratioToLatin - 1) < 0.01) found.push({ cp, wide: false });
|
||||
else if (Math.abs(ratioToLatin - 2) < 0.01) found.push({ cp, wide: true });
|
||||
else {
|
||||
throw new Error(
|
||||
`[gen-atlas] codepoint U+${cp.toString(16).toUpperCase()} has advance ` +
|
||||
`${w}px (Latin=${mLatin.width}px, ratio=${ratioToLatin.toFixed(3)}; ` +
|
||||
`expected 1× or 2×). Font version drift?`,
|
||||
);
|
||||
const source = sourceForCodepoint(cp);
|
||||
if (source === 'primary') {
|
||||
found.push({ cp, wide: false, source });
|
||||
primary++;
|
||||
kept++;
|
||||
continue;
|
||||
}
|
||||
const wide = classifyFallbackWidth(cp);
|
||||
if (wide == null) continue;
|
||||
found.push({ cp, wide, source });
|
||||
fallback++;
|
||||
kept++;
|
||||
}
|
||||
console.log(
|
||||
`[gen-atlas] ${label.padEnd(48)} ` +
|
||||
`U+${lo.toString(16).padStart(4, '0').toUpperCase()}..` +
|
||||
`U+${hi.toString(16).padStart(4, '0').toUpperCase()} ` +
|
||||
`kept ${kept}/${hi - lo + 1}`,
|
||||
`kept ${kept}/${hi - lo + 1} (spleen=${primary}, unifont=${fallback})`,
|
||||
);
|
||||
}
|
||||
// Sort by codepoint so the runtime can binary-search.
|
||||
|
||||
found.sort((a, b) => a.cp - b.cp);
|
||||
const wideCount = found.filter((f) => f.wide).length;
|
||||
console.log(`[gen-atlas] total glyphs: ${found.length} (${wideCount} wide)`);
|
||||
const primaryCount = found.filter((f) => f.source === 'primary').length;
|
||||
console.log(`[gen-atlas] total glyphs: ${found.length} (${wideCount} wide, ${primaryCount} Spleen primary)`);
|
||||
|
||||
// --- Rasterize each glyph --------------------------------------------------
|
||||
// All glyphs go into a single flat Uint8Array; OFFSETS[] points into it.
|
||||
// Width depends on the glyph (cellW or 2*cellW); height is always cellH.
|
||||
// --- Rasterize glyphs ------------------------------------------------------
|
||||
const contexts = {
|
||||
primary: {
|
||||
narrow: createCanvas(cellW, cellH).getContext('2d'),
|
||||
wide: createCanvas(2 * cellW, cellH).getContext('2d'),
|
||||
font: `${PRIMARY_FONT_PX}px ${PRIMARY_FONT_FAMILY}`,
|
||||
},
|
||||
fallback: {
|
||||
narrow: createCanvas(cellW, cellH).getContext('2d'),
|
||||
wide: createCanvas(2 * cellW, cellH).getContext('2d'),
|
||||
font: `${FALLBACK_FONT_PX}px ${FALLBACK_FONT_FAMILY}`,
|
||||
},
|
||||
} as const;
|
||||
for (const src of [contexts.primary, contexts.fallback]) {
|
||||
for (const ctx of [src.narrow, src.wide]) {
|
||||
ctx.font = src.font;
|
||||
ctx.textBaseline = 'alphabetic';
|
||||
}
|
||||
}
|
||||
|
||||
const wideCanvas = createCanvas(2 * cellW, cellH);
|
||||
const wideCtx = wideCanvas.getContext('2d');
|
||||
wideCtx.font = `${FONT_PX}px ${FONT_FAMILY}`;
|
||||
wideCtx.textBaseline = 'alphabetic';
|
||||
|
||||
const narrowCanvas = createCanvas(cellW, cellH);
|
||||
const narrowCtx = narrowCanvas.getContext('2d');
|
||||
narrowCtx.font = `${FONT_PX}px ${FONT_FAMILY}`;
|
||||
narrowCtx.textBaseline = 'alphabetic';
|
||||
|
||||
// Pixel data is BIT-PACKED, MSB-first. Unifont glyphs are inherently 1-bit
|
||||
// (every pixel is exactly 0 or 255 — no antialiasing), so storing as bytes
|
||||
// wastes 7 bits/pixel. The runtime decoder extracts the bit at index
|
||||
// bitIdx = OFFSETS[rank] + row * srcW + col
|
||||
// where srcW is `cellW` for narrow glyphs or `2*cellW` for wide (CJK) ones —
|
||||
// driven by WIDE_FLAGS[rank]. OFFSETS is now a BIT offset (was byte offset
|
||||
// in the prior 8-bit format). Bit indices fit comfortably in Uint32:
|
||||
// 35k glyphs × 10×11 px max = ~3.9M bits, well under 2^32.
|
||||
const codepoints = new Uint32Array(found.length);
|
||||
const offsets = new Uint32Array(found.length);
|
||||
const wideFlags = new Uint8Array(found.length);
|
||||
// Collect per-glyph bit slices first; we'll pack them into a single
|
||||
// MSB-first bitstream at the end so glyph N starts exactly where glyph N-1
|
||||
// ended (no byte alignment between glyphs).
|
||||
const cellBitSlices: Uint8Array[] = []; // one entry per glyph; values are 0 or 1
|
||||
const cellBitSlices: Uint8Array[] = [];
|
||||
let totalBits = 0;
|
||||
|
||||
for (let i = 0; i < found.length; i++) {
|
||||
const { cp, wide } = found[i]!;
|
||||
const { cp, wide, source } = found[i]!;
|
||||
codepoints[i] = cp;
|
||||
wideFlags[i] = wide ? 1 : 0;
|
||||
offsets[i] = totalBits;
|
||||
|
||||
const w = wide ? 2 * cellW : cellW;
|
||||
const ctx = wide ? wideCtx : narrowCtx;
|
||||
const ctx = wide ? contexts[source].wide : contexts[source].narrow;
|
||||
|
||||
// Paint glyph: black canvas, white text. R-channel coverage is what we keep.
|
||||
ctx.fillStyle = '#000';
|
||||
ctx.fillRect(0, 0, w, cellH);
|
||||
ctx.fillStyle = '#fff';
|
||||
ctx.fillText(String.fromCodePoint(cp), 0, ascent);
|
||||
|
||||
const img = ctx.getImageData(0, 0, w, cellH);
|
||||
// Threshold the R channel to 1-bit. Unifont is already 0/255 black-and-
|
||||
// white so the threshold is academic, but use ≥128 for safety against
|
||||
// any subpixel-rendering surprises.
|
||||
const bits = new Uint8Array(w * cellH);
|
||||
for (let p = 0; p < bits.length; p++) bits[p] = img.data[p * 4]! >= 128 ? 1 : 0;
|
||||
cellBitSlices.push(bits);
|
||||
totalBits += bits.length;
|
||||
}
|
||||
|
||||
// Pack into MSB-first bitstream: bit at index B lives in byte B>>3 at
|
||||
// position 7-(B&7). Total bytes = ceil(totalBits / 8).
|
||||
const totalBytes = (totalBits + 7) >>> 3;
|
||||
const pixels = new Uint8Array(totalBytes);
|
||||
{
|
||||
@@ -273,11 +255,6 @@ console.log(
|
||||
`(was ${totalBits} bytes at 8-bit; ${(totalBits / totalBytes).toFixed(1)}× pre-deflate shrink)`,
|
||||
);
|
||||
|
||||
// --- Encode binary blobs as base64 ----------------------------------------
|
||||
// JSON array literals of 41k numbers would blow up atlas.ts to several MB
|
||||
// of TS source. base64'd typed-array bytes are ~6× tighter and the decoder
|
||||
// is 10 lines that runs once at module load.
|
||||
|
||||
function bytesB64(bytes: Uint8Array): string {
|
||||
return Buffer.from(bytes).toString('base64');
|
||||
}
|
||||
@@ -291,13 +268,12 @@ console.log(
|
||||
`offsets=${offsetsB64.length} wide=${wideFlagsB64.length} pixels=${pixelsB64.length}`,
|
||||
);
|
||||
|
||||
// --- Emit src/core/atlas.ts -----------------------------------------------
|
||||
const banner = `// AUTO-GENERATED by scripts/gen-atlas.ts — DO NOT EDIT.
|
||||
// Regenerate with: pnpm run build:atlas
|
||||
// (or ATLAS_PROFILE=practical pnpm run build:atlas to drop Hangul for
|
||||
// Workers free-tier deployments under the 1 MB compressed-bundle cap)
|
||||
// Source font: assets/Unifont-16.0.04.otf @ ${FONT_PX}px (profile: ${PROFILE})
|
||||
// Glyphs: ${found.length} codepoints (${wideCount} wide)
|
||||
// Source fonts: assets/Spleen-5x8.otb @ ${PRIMARY_FONT_PX}px for ASCII/code; assets/Unifont-16.0.04.otf @ ${FALLBACK_FONT_PX}px fallback (profile: ${PROFILE})
|
||||
// Glyphs: ${found.length} codepoints (${wideCount} wide, ${primaryCount} Spleen primary)
|
||||
`;
|
||||
|
||||
const body = `
|
||||
@@ -309,10 +285,10 @@ export const ATLAS_CELL_H = ${cellH};
|
||||
export const ATLAS_ASCENT = ${ascent};
|
||||
/** Distance from baseline to cell bottom. */
|
||||
export const ATLAS_DESCENT = ${descent};
|
||||
/** Font size used when rasterizing. */
|
||||
export const ATLAS_FONT_PX = ${FONT_PX};
|
||||
/** Font family name used at build time. Renderer never re-loads the font. */
|
||||
export const ATLAS_FONT_FAMILY = ${JSON.stringify(FONT_FAMILY)};
|
||||
/** Primary font size used when rasterizing ASCII/code glyphs. */
|
||||
export const ATLAS_FONT_PX = ${PRIMARY_FONT_PX};
|
||||
/** Font family label used at build time. Renderer never re-loads the font. */
|
||||
export const ATLAS_FONT_FAMILY = ${JSON.stringify(FONT_FAMILY_LABEL)};
|
||||
/** Profile used to build this atlas. */
|
||||
export const ATLAS_PROFILE = ${JSON.stringify(PROFILE)};
|
||||
|
||||
@@ -340,22 +316,18 @@ function decodeU32(b64: string): Uint32Array {
|
||||
* at \`rank\` in OFFSETS / WIDE_FLAGS / PIXELS. */
|
||||
export const ATLAS_CODEPOINTS: Uint32Array = /* @__PURE__ */ decodeU32(CODEPOINTS_B64);
|
||||
|
||||
/** BIT offset into ATLAS_PIXELS for the glyph at each rank. (Was byte offset
|
||||
* in the prior 8-bit format.) */
|
||||
/** BIT offset into ATLAS_PIXELS for the glyph at each rank. */
|
||||
export const ATLAS_OFFSETS: Uint32Array = /* @__PURE__ */ decodeU32(OFFSETS_B64);
|
||||
|
||||
/** 1 if the glyph at this rank is double-wide (East Asian Wide), 0 otherwise. */
|
||||
export const ATLAS_WIDE_FLAGS: Uint8Array = /* @__PURE__ */ decodeB64(WIDE_FLAGS_B64);
|
||||
|
||||
/** Bit-packed 1-bit pixel data, MSB-first. Unifont glyphs are inherently
|
||||
* 1-bit (every pixel is 0 or 255 — no antialiasing), so we store 8 px per
|
||||
* byte. The runtime decoder (blitGlyph) extracts:
|
||||
/** Bit-packed 1-bit pixel data, MSB-first. Runtime extraction:
|
||||
* bitIdx = OFFSETS[rank] + row * srcW + col
|
||||
* byteIdx = bitIdx >>> 3
|
||||
* bitOff = 7 - (bitIdx & 7)
|
||||
* pixel = (ATLAS_PIXELS[byteIdx] >>> bitOff) & 1 // 0 or 1
|
||||
* where srcW is CELL_W (narrow) or 2*CELL_W (wide, per WIDE_FLAGS[rank]).
|
||||
* Output framebuffer maps 0 → 0 (background) and 1 → 255 (full ink). */
|
||||
* pixel = (ATLAS_PIXELS[byteIdx] >>> bitOff) & 1
|
||||
* where srcW is CELL_W (narrow) or 2*CELL_W (wide, per WIDE_FLAGS[rank]). */
|
||||
export const ATLAS_PIXELS: Uint8Array = /* @__PURE__ */ decodeB64(PIXELS_B64);
|
||||
|
||||
/** Number of glyphs in the atlas. */
|
||||
|
||||
+15
-19
File diff suppressed because one or more lines are too long
+1
-1
@@ -309,7 +309,7 @@ export async function renderTextToPngs(
|
||||
// --- R2 multi-column rendering --------------------------------------------
|
||||
//
|
||||
// Single-column packing leaves Anthropic's 1568×1568 image area badly
|
||||
// under-used: at cell 5×11 and cols=100, our render canvas is only 508 px
|
||||
// under-used: at cell 5×8 and cols=100, our render canvas is only 508 px
|
||||
// wide — ~32% of the horizontal budget. Most real Claude Code tool docs +
|
||||
// CLAUDE.md content wraps at well under 100 chars/row, so we end up paying
|
||||
// the per-image cost (~2,500 tokens) for an image that's mostly whitespace.
|
||||
|
||||
@@ -280,14 +280,13 @@ function effectiveTokensPerImage(numCols: number): number {
|
||||
*
|
||||
* Formula: `cols × floor((MAX_HEIGHT_PX − 2·PAD_Y) / ATLAS_CELL_H)`
|
||||
*
|
||||
* At the shipping config (Unifont, cell 5×11, cols=100):
|
||||
* 100 × floor((1568 − 8) / 11) = 100 × 141 = 14,100
|
||||
* At the shipping config (Spleen/Unifont hybrid, cell 5×8, cols=100):
|
||||
* 100 × floor((1568 − 8) / 8) = 100 × 195 = 19,500
|
||||
*
|
||||
* When the atlas swaps (e.g. Cozette 4×7, cell H=7), this auto-updates:
|
||||
* 100 × floor(1560 / 7) = 100 × 222 = 22,200
|
||||
* …and the break-even threshold drops accordingly. Without this, the
|
||||
* hardcoded 14,100 would silently let net-loss compressions through on
|
||||
* every smaller-cell atlas. */
|
||||
* When the atlas swaps again, this auto-updates and the break-even
|
||||
* threshold moves with the actual renderer geometry. Without this, a
|
||||
* stale hardcoded chars/image constant would silently mis-price image
|
||||
* compressions. */
|
||||
/** Visual rows per image at the current atlas cell. Derived once at module
|
||||
* load. Auto-updates when gen-atlas regenerates with a different font/size. */
|
||||
export const LINES_PER_IMAGE = Math.max(1, Math.floor((MAX_HEIGHT_PX - 2 * PAD_Y) / ATLAS_CELL_H));
|
||||
|
||||
+1
-1
@@ -50,7 +50,7 @@ export default {
|
||||
compressToolResults: truthy(env.COMPRESS_TOOL_RESULTS, true),
|
||||
minCompressChars: env.MIN_COMPRESS_CHARS ? Number(env.MIN_COMPRESS_CHARS) : 2000,
|
||||
// Raised to 10,000 — per-block break-even point at current renderer
|
||||
// config (Unifont 10px, cell 5×11, 100 cols). Real gate is
|
||||
// config (Spleen/Unifont 5×8 hybrid, 100 cols). Real gate is
|
||||
// `isCompressionProfitable()` in transform.ts; this is a fast-path
|
||||
// skip for obvious-no cases. Keep in sync with DEFAULTS.
|
||||
minReminderChars: env.MIN_REMINDER_CHARS ? Number(env.MIN_REMINDER_CHARS) : 10000,
|
||||
|
||||
Vendored
+15
-19
@@ -9,8 +9,8 @@
|
||||
*
|
||||
* ## Why these matter
|
||||
*
|
||||
* `SLAB_CHARS_PER_TOKEN = 2.5` and `HISTORY_CHARS_PER_TOKEN = 2.5` are frozen
|
||||
* empirical fits from H=354 samples. The synthetic `'A'.repeat(N)` shapes
|
||||
* `SLAB_CHARS_PER_TOKEN = 2.0` and `HISTORY_CHARS_PER_TOKEN = 2.0` are frozen
|
||||
* empirical fits from Opus 4.7 production samples. The synthetic `'A'.repeat(N)` shapes
|
||||
* elsewhere in the suite prove the *math* is wired correctly; these prove the
|
||||
* *constants* still match real Claude Code traffic. If a future model variant
|
||||
* (Sonnet 4.6 vs Opus 4.7) tokenizes differently and the textbook 4 chars/token
|
||||
@@ -45,21 +45,18 @@ export interface RealShape {
|
||||
/**
|
||||
* The production-shape slab that motivated the `e8545a9` fix.
|
||||
*
|
||||
* Before the fix: gate estimated `text_tokens = 161101/4 = 40275` and rejected
|
||||
* because `image_cost (8 × 5500 = 44000) > text_tokens`. After the fix with
|
||||
* `SLAB_CHARS_PER_TOKEN = 2.5`: `text_tokens = 161101/2.5 = 64440`, which is
|
||||
* 20k tokens above the conservative image cost — clear ACCEPT with safety
|
||||
* margin.
|
||||
*
|
||||
* Real `count_tokens` probe measured ~99k tokens for this body (cpt ≈ 1.62),
|
||||
* so the gate's 2.5 estimate (64k tok) is still a *lower bound* on the real
|
||||
* cost. We compress when we know we'll win; we don't risk net-losers.
|
||||
* Before the fix: gate used the textbook `chars/token=4` and rejected dense
|
||||
* Claude Code slabs because it under-counted text cost. The current Opus-4.7
|
||||
* gate uses `SLAB_CHARS_PER_TOKEN = 2.0`, which remains a lower-bound
|
||||
* estimate versus real `count_tokens` (~99k tokens for this body, cpt≈1.62)
|
||||
* while making this shape a clear ACCEPT. We compress when we know we'll win;
|
||||
* we don't risk net-losers.
|
||||
*/
|
||||
export const PRODUCTION_SLAB_161K: RealShape = {
|
||||
name: 'production slab (161k chars, multi-col)',
|
||||
origChars: 161101,
|
||||
numCols: 2,
|
||||
approxCharsPerRow: 52, // 11 images × 141 lines/image × 2 cols ≈ 3102 rows
|
||||
approxCharsPerRow: 52, // 8 images × 195 lines/image × 2 cols ≈ 3120 rows
|
||||
decision: 'accept',
|
||||
gate: 'slab',
|
||||
baselineTokens: 99478,
|
||||
@@ -77,9 +74,8 @@ export const PRODUCTION_SLAB_161K: RealShape = {
|
||||
*
|
||||
* The production event for this shape was compressed (gate accepted), but
|
||||
* the synthetic `'A'.repeat(19)` lines we generate from the shape don't
|
||||
* capture the real density. With ~6533 lines / 141 rows-per-image / 2
|
||||
* multiCol ≈ 24 images × 5500 = 132k img-tokens vs 130665/2.5 = 52k
|
||||
* text-tokens → the gate REJECTS the synthetic form even at cpt=2.5.
|
||||
* capture the real density. The synthetic form still overruns the text-token
|
||||
* budget under the conservative cpt=2.0 gate, so it REJECTS.
|
||||
*
|
||||
* The fixture pins the gate's decision on the *synthetic* shape, not the
|
||||
* production outcome. Real text at this density (mixed line lengths, dense
|
||||
@@ -101,9 +97,9 @@ export const PRODUCTION_SLAB_135K_DENSE: RealShape = {
|
||||
* The largest production slab we have data for. ~16 chars/row — almost all
|
||||
* newlines (deeply nested JSON or tabular tool output).
|
||||
*
|
||||
* At cpt=4 textbook estimate: `169632/4 = 42408` tok vs `image_cost = 37 ×
|
||||
* 5500 / 2 ≈ 101750` tok → REJECT. At cpt=2.5: `169632/2.5 = 67852` tok —
|
||||
* still REJECT under the gate's conservative math, and production confirms
|
||||
* At cpt=4 textbook estimate the text is badly undercounted; even at the
|
||||
* current conservative cpt=2.0, this newline-heavy synthetic shape still
|
||||
* REJECTS under the gate's image-cost math, and production confirms
|
||||
* this: the event has `compressed=true` because by the time the slab grew
|
||||
* that large the *real* token count (count_tokens ≈ image_cost) had crossed
|
||||
* over. The gate is conservative; the regression test pins that the gate
|
||||
@@ -161,7 +157,7 @@ export const BELOW_MIN_CHARS_BORDERLINE: RealShape = {
|
||||
* (537k chars) into one synthetic prepended user message + image block.
|
||||
*
|
||||
* Pinned here so the regression test confirms the `historyReason:
|
||||
* 'collapsed'` path stays healthy under `HISTORY_CHARS_PER_TOKEN = 2.5`.
|
||||
* 'collapsed'` path stays healthy under `HISTORY_CHARS_PER_TOKEN = 2.0`.
|
||||
* Same workload as the slab fix; this exercises the *different* call site.
|
||||
*/
|
||||
export const HISTORY_COLLAPSED_LONG_SESSION: RealShape = {
|
||||
|
||||
+17
-22
@@ -448,29 +448,24 @@ describe('transformRequest history compression (always-on)', () => {
|
||||
expect(info.historyReason).toBe('no_closed_prefix');
|
||||
});
|
||||
|
||||
it('borderline fixture: collapses with built-in HISTORY_CHARS_PER_TOKEN=2.5 where cpt=4 would reject', async () => {
|
||||
it('borderline fixture: collapses with built-in HISTORY_CHARS_PER_TOKEN=2.0 where cpt=4 would reject', async () => {
|
||||
// Empirical 2026-05-20: N=10 production "history rejected as
|
||||
// not_profitable" events have body cpt 1.08-1.10. The gate using
|
||||
// cpt=4 was estimating text as 3.7× cheaper than reality and
|
||||
// rejecting compressions that real billing would have approved.
|
||||
//
|
||||
// This fixture pins the wiring of HISTORY_CHARS_PER_TOKEN=2.5 into the
|
||||
// transformRequest → collapseHistory gate. It sits in the band where
|
||||
// image cost (5000 tok ≈ 2 images) is:
|
||||
// • > text-tokens at cpt=4 → REJECT under stale default
|
||||
// • < text-tokens at cpt=2.5 → ACCEPT under the empirical fix
|
||||
//
|
||||
// 10 collapsed turns × 1500 chars/turn ≈ 15-17k chars after framing.
|
||||
// Single-col packs ≈ 14k chars/image → 2 images. At cpt=4: text=4k tok
|
||||
// vs image=5k tok → reject. At cpt=2.5: text=6.4k tok > image=5k tok
|
||||
// → accept with ≈1.4k tok headroom.
|
||||
// This fixture pins the wiring of HISTORY_CHARS_PER_TOKEN=2.0 into the
|
||||
// transformRequest → collapseHistory gate after the 5×8 atlas change.
|
||||
// It sits in the band where image cost is:
|
||||
// • > text-tokens at cpt=4 → REJECT under stale/default prose cpt
|
||||
// • < text-tokens at cpt=2.0 → ACCEPT under Opus 4.7 telemetry
|
||||
const msgs: Message[] = [];
|
||||
for (let i = 0; i < 14; i++) {
|
||||
const body = `turn ${i}: ` + bigPlain(1500);
|
||||
const body = `turn ${i}: ` + bigPlain(900);
|
||||
msgs.push(i % 2 === 0 ? usr(body) : asst(body));
|
||||
}
|
||||
const { info } = await transformRequest(mkBody(msgs, bigPlain(80_000)));
|
||||
// Under HISTORY_CHARS_PER_TOKEN=2.5 this fixture collapses cleanly.
|
||||
// Under HISTORY_CHARS_PER_TOKEN=2.0 this fixture collapses cleanly.
|
||||
expect(info.historyReason).toBe('collapsed');
|
||||
expect(info.collapsedTurns).toBe(10);
|
||||
|
||||
@@ -496,13 +491,13 @@ describe('transformRequest history compression (always-on)', () => {
|
||||
// gate now uses `!== undefined` so a literal 4 stays a literal 4.
|
||||
//
|
||||
// Observable proof: a borderline-density fixture (1200-char bodies × 14
|
||||
// turns) that is rejected at cpt=4 but accepted at cpt=2.5. If the gate
|
||||
// silently swapped explicit 4 → 2.5, this fixture would collapse — but
|
||||
// turns) that is rejected at cpt=4 but accepted at cpt=2.0. If the gate
|
||||
// silently ignored explicit 4, this fixture would collapse — but
|
||||
// with the fix it stays rejected, confirming the gate honored the literal
|
||||
// 4. The companion test below pins cpt=2.5 collapse on the same shape.
|
||||
// 4. The companion test below pins cpt=2.0 collapse on the same shape.
|
||||
const msgs: Message[] = [];
|
||||
for (let i = 0; i < 14; i++) {
|
||||
const body = `turn ${i}: ` + bigPlain(1200);
|
||||
const body = `turn ${i}: ` + bigPlain(900);
|
||||
msgs.push(i % 2 === 0 ? usr(body) : asst(body));
|
||||
}
|
||||
const explicit4 = await transformRequest(mkBody(msgs, bigPlain(80_000)), {
|
||||
@@ -511,13 +506,13 @@ describe('transformRequest history compression (always-on)', () => {
|
||||
expect(explicit4.info.historyReason).toBe('not_profitable');
|
||||
expect(explicit4.info.collapsedTurns).toBeUndefined();
|
||||
|
||||
// Same shape at cpt=2.5 collapses — proves the fixture actually straddles
|
||||
// Same shape at cpt=2.0 collapses — proves the fixture actually straddles
|
||||
// the threshold and isn't a tautology.
|
||||
const explicit25 = await transformRequest(mkBody(msgs, bigPlain(80_000)), {
|
||||
charsPerToken: 2.5,
|
||||
const explicit20 = await transformRequest(mkBody(msgs, bigPlain(80_000)), {
|
||||
charsPerToken: 2.0,
|
||||
});
|
||||
expect(explicit25.info.historyReason).toBe('collapsed');
|
||||
expect(explicit25.info.collapsedTurns).toBe(10);
|
||||
expect(explicit20.info.historyReason).toBe('collapsed');
|
||||
expect(explicit20.info.collapsedTurns).toBe(10);
|
||||
});
|
||||
|
||||
it('history-image blocks carry NO cache_control (conservative first-cut)', async () => {
|
||||
|
||||
+17
-17
@@ -25,11 +25,11 @@ import {
|
||||
import { toTrackEvent } from '../src/core/tracker.js';
|
||||
import type { ProxyEvent } from '../src/core/proxy.js';
|
||||
|
||||
// Default render config: cols=100, ~141 lines/img → ~14,100 chars/img if
|
||||
// Default render config: cols=100, 195 lines/img → ~19,500 chars/img if
|
||||
// lines fully fill the width. For shorter lines, the budget is dominated
|
||||
// by row count (each line takes ≥1 row regardless of length).
|
||||
const COLS = 100;
|
||||
const ROWS_PER_IMG = 141; // floor((1568 - 8) / 11)
|
||||
const ROWS_PER_IMG = 195; // floor((1568 - 8) / 8), Spleen 5×8 hybrid atlas
|
||||
|
||||
describe('estimateImageCount', () => {
|
||||
it('returns 1 for empty / tiny text', () => {
|
||||
@@ -38,14 +38,14 @@ describe('estimateImageCount', () => {
|
||||
});
|
||||
|
||||
it('scales linearly with row count for short-line content', () => {
|
||||
// 141 lines of "x" (1 char) = 141 rows = 1 image.
|
||||
const oneImage = Array.from({ length: 141 }, () => 'x').join('\n');
|
||||
// 195 lines of "x" (1 char) = 195 rows = 1 image.
|
||||
const oneImage = Array.from({ length: ROWS_PER_IMG }, () => 'x').join('\n');
|
||||
expect(estimateImageCount(oneImage, COLS)).toBe(1);
|
||||
// 142 lines = 2 images (just over the line).
|
||||
const justOver = Array.from({ length: 142 }, () => 'x').join('\n');
|
||||
// 196 lines = 2 images (just over the line).
|
||||
const justOver = Array.from({ length: ROWS_PER_IMG + 1 }, () => 'x').join('\n');
|
||||
expect(estimateImageCount(justOver, COLS)).toBe(2);
|
||||
// 10 × 141 = 1410 lines → 10 images.
|
||||
const tenImages = Array.from({ length: 1410 }, () => 'x').join('\n');
|
||||
// 10 × 195 = 1950 lines → 10 images.
|
||||
const tenImages = Array.from({ length: ROWS_PER_IMG * 10 }, () => 'x').join('\n');
|
||||
expect(estimateImageCount(tenImages, COLS)).toBe(10);
|
||||
});
|
||||
|
||||
@@ -53,18 +53,18 @@ describe('estimateImageCount', () => {
|
||||
// A single 1000-char line wraps to ceil(1000/100) = 10 rows.
|
||||
const wrapped = 'x'.repeat(1000);
|
||||
expect(estimateImageCount(wrapped, COLS)).toBe(1); // 10 rows, fits in 1 img
|
||||
// 14,100 chars on one line → 141 rows → 1 image.
|
||||
const oneImg = 'x'.repeat(14_100);
|
||||
// 19,500 chars on one line → 195 rows → 1 image.
|
||||
const oneImg = 'x'.repeat(19_500);
|
||||
expect(estimateImageCount(oneImg, COLS)).toBe(1);
|
||||
// 14,101 chars → 142 rows → 2 images.
|
||||
const twoImgs = 'x'.repeat(14_101);
|
||||
// 19,501 chars → 196 rows → 2 images.
|
||||
const twoImgs = 'x'.repeat(19_501);
|
||||
expect(estimateImageCount(twoImgs, COLS)).toBe(2);
|
||||
});
|
||||
|
||||
it('also accepts a numeric length (legacy chars-based estimate)', () => {
|
||||
expect(estimateImageCount(0, COLS)).toBe(1);
|
||||
expect(estimateImageCount(14_100, COLS)).toBe(1);
|
||||
expect(estimateImageCount(14_101, COLS)).toBe(2);
|
||||
expect(estimateImageCount(19_500, COLS)).toBe(1);
|
||||
expect(estimateImageCount(19_501, COLS)).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -152,7 +152,7 @@ describe('truncateForBudget', () => {
|
||||
|
||||
it('truncates head+tail for log-shaped content over the budget', () => {
|
||||
// 10k log lines, each ~32 chars → ~320k chars total. With short lines
|
||||
// the row budget dominates: 10k rows >> 10 × 141 = 1410 row budget.
|
||||
// the row budget dominates: 10k rows >> 10 × 195 = 1950 row budget.
|
||||
const lines: string[] = [];
|
||||
for (let i = 0; i < 10_000; i++) {
|
||||
lines.push(`2026-05-18T12:00:${String(i % 60).padStart(2, '0')}Z entry ${i}`);
|
||||
@@ -361,9 +361,9 @@ describe('paging end-to-end (transformRequest)', () => {
|
||||
expect(info.truncatedToolResults).toBe(2);
|
||||
// Both should have been truncated → omittedChars roughly doubled. The
|
||||
// exact bound depends on renderer config: at multiCol=1 each image
|
||||
// packs ~14k chars worst-case, multiCol=2 packs ~28k → less omitted
|
||||
// packs ~19.5k chars worst-case, multiCol=2 packs ~39k → less omitted
|
||||
// at the same maxImagesPerToolResult cap. Threshold below covers both.
|
||||
expect(info.omittedChars).toBeGreaterThan(700_000);
|
||||
expect(info.omittedChars).toBeGreaterThan(600_000);
|
||||
});
|
||||
|
||||
it('handles array-shaped tool_result content', async () => {
|
||||
|
||||
+54
-59
@@ -154,7 +154,7 @@ describe('renderer', () => {
|
||||
});
|
||||
|
||||
it('multi-col halves image count on row-heavy input', async () => {
|
||||
// ~500 lines of narrow content. Single-col packs 141 lines/image →
|
||||
// ~500 lines of narrow content. Single-col packs 195 lines/image →
|
||||
// ~4 images. Two columns should drop that to ~2.
|
||||
const text = ('lorem ipsum dolor sit amet\n'.repeat(500));
|
||||
const single = await renderTextToPngs(text, 100);
|
||||
@@ -299,7 +299,7 @@ describe('renderer', () => {
|
||||
}
|
||||
});
|
||||
|
||||
// ---- Unicode coverage tests (Unifont atlas) -------------------------------
|
||||
// ---- Unicode coverage tests (hybrid atlas fallback) -------------------------------
|
||||
// These confirm the sparse-codepoint + wide-glyph machinery works end-to-end.
|
||||
// None of them assert specific PNG bytes (the byte-deterministic guarantee
|
||||
// is covered by the 'renders identical input...' test below); they assert
|
||||
@@ -1739,11 +1739,11 @@ describe('transform', () => {
|
||||
|
||||
// --- Per-block break-even gate (URGENT slice, supersedes prior threshold tests) ---
|
||||
// history-researcher's round-3 analysis measured Anthropic's real per-image
|
||||
// cost at ~2,500 tokens. At the current renderer config (14,100 chars/image)
|
||||
// cost at ~2,500 tokens. At the current renderer config (19,500 chars/image)
|
||||
// the break-even point is 10,000 chars per image. Blocks shorter than that
|
||||
// cost MORE as images than as text. The fix: gate every per-block image
|
||||
// encoding on `isCompressionProfitable()` which checks
|
||||
// ceil(textLen / 14100) * 2500 < textLen / 4
|
||||
// ceil(textLen / 19500) * 2500 < textLen / 4
|
||||
// Tests below confirm the function math AND that the gates correctly skip
|
||||
// net-loss compressions in the full pipeline.
|
||||
|
||||
@@ -1897,10 +1897,10 @@ describe('transform', () => {
|
||||
expect(out.info.imageCount ?? 0).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
it('isCompressionProfitable: slab cpt=2.5 flips a 161k production-shape slab profitable at multi-col=2', () => {
|
||||
// Pin the math directly. Image cost at multi-col=2: 8 imgs × 5500 =
|
||||
// 44,000 tok. At cpt=4, text=40,275 → REJECT. At cpt=2.5, text=64,440
|
||||
// → ACCEPT with 20k headroom over the conservative slab cpt.
|
||||
it('isCompressionProfitable: 5x8 atlas makes a 161k production-shape slab profitable even at cpt=4', () => {
|
||||
// Pin the math directly. The 5×8 atlas drops this shape to ~6 two-column
|
||||
// images: 6 × 5500 = 33,000 image tokens. At cpt=4, text≈40,275, so
|
||||
// it now ACCEPTS even under a conservative prose/token override.
|
||||
const parts: string[] = [];
|
||||
let acc = 0;
|
||||
while (acc < 161_101) {
|
||||
@@ -1909,26 +1909,26 @@ describe('transform', () => {
|
||||
acc += len + 1;
|
||||
}
|
||||
const slab = parts.join('\n').slice(0, 161_101);
|
||||
expect(isCompressionProfitable(slab, 100, undefined, 2, 4)).toBe(false);
|
||||
expect(isCompressionProfitable(slab, 100, undefined, 2, 4)).toBe(true);
|
||||
expect(isCompressionProfitable(slab, 100, undefined, 2, 2.5)).toBe(true);
|
||||
});
|
||||
|
||||
it('TransformOptions.charsPerToken: explicit 4 is honored (no silent swap to SLAB_CHARS_PER_TOKEN)', async () => {
|
||||
// Fragility #2 regression: the override-gate previously used a `!==
|
||||
// CHARS_PER_TOKEN` check that silently swapped 4 → 2.5 because the static
|
||||
// default *also* happens to be 4. After the fix it uses `!== undefined`,
|
||||
// so passing exactly 4 is honored as an explicit override.
|
||||
// CHARS_PER_TOKEN` check that silently swapped 4 → the built-in slab cpt
|
||||
// because the static default *also* happened to be 4. After the fix it
|
||||
// uses `!== undefined`, so passing exactly 4 is honored as an explicit
|
||||
// host override.
|
||||
//
|
||||
// Observable proof: the 161k production-shape slab is REJECTED at cpt=4
|
||||
// (text=40,275 tok < image=44,000 tok) and ACCEPTED at the built-in
|
||||
// SLAB_CHARS_PER_TOKEN=2.5 (text=64,440 tok). If the collision bypass
|
||||
// breaks, the slab will compress under an explicit `4` — which would
|
||||
// mean the host can't ever pin to the conservative English-prose value.
|
||||
// Observable proof after the 5×8 atlas: a row-heavier 50k slab is
|
||||
// REJECTED at cpt=4 but ACCEPTED at the built-in Opus-4.7 cpt=2. If the
|
||||
// collision bypass breaks, the slab will compress under explicit `4` —
|
||||
// meaning the host cannot pin to conservative English-prose density.
|
||||
const parts: string[] = [];
|
||||
let acc = 0;
|
||||
const target = 161_101;
|
||||
const target = 50_000;
|
||||
while (acc < target) {
|
||||
const len = 60 + (acc % 40);
|
||||
const len = 50;
|
||||
parts.push('A'.repeat(len) + (acc % 200 === 0 ? ' ' : ''));
|
||||
acc += len + 1;
|
||||
}
|
||||
@@ -1940,40 +1940,40 @@ describe('transform', () => {
|
||||
});
|
||||
const bytes = new TextEncoder().encode(req);
|
||||
|
||||
// Built-in cpt (no override): slab compresses via SLAB_CHARS_PER_TOKEN=2.5.
|
||||
// Built-in cpt (no override): slab compresses via SLAB_CHARS_PER_TOKEN=2.0.
|
||||
const builtin = await transformRequest(bytes, { multiCol: 2 });
|
||||
expect(builtin.info.compressed).toBe(true);
|
||||
|
||||
// Explicit cpt=4: host pinned to the English-prose value. The slab gate
|
||||
// must honor it and reject the slab — not silently fall back to 2.5.
|
||||
// must honor it and reject the slab — not silently fall back to 2.0.
|
||||
const overridden = await transformRequest(bytes, { multiCol: 2, charsPerToken: 4 });
|
||||
expect(overridden.info.compressed).toBe(false);
|
||||
expect(overridden.info.reason).toMatch(/^not_profitable/);
|
||||
});
|
||||
|
||||
// --- Adaptive break-even: CHARS_PER_IMAGE derived from atlas cell, not hardcoded ---
|
||||
// Brief: when font-rater swaps to a smaller cell (e.g. Cozette 4×7), more chars
|
||||
// pack into one image, so the N-image break-even thresholds shift. Tests below
|
||||
// verify both the regression case (current Unifont 5×11) AND that the formula
|
||||
// Brief: when font-rater swaps the atlas cell height, more/fewer chars pack
|
||||
// into one image, so the N-image break-even thresholds shift. Tests below
|
||||
// verify both the regression case (current Spleen/Unifont 5×8 hybrid) AND that the formula
|
||||
// responds to `cols` (which scales chars/image linearly the same way a smaller
|
||||
// cell-H would).
|
||||
|
||||
it('maxCharsPerImage: matches the historic 14,100 constant at the shipping config', () => {
|
||||
// Unifont 5×11, cols=100 → floor((1568−8)/11) × 100 = 141 × 100 = 14,100.
|
||||
it('maxCharsPerImage: matches the 19,500 constant at the 5x8 shipping config', () => {
|
||||
// Spleen/Unifont 5×8, cols=100 → floor((1568−8)/8) × 100 = 195 × 100 = 19,500.
|
||||
// If this ever drifts, every break-even test downstream needs re-pinning.
|
||||
expect(maxCharsPerImage(100)).toBe(14_100);
|
||||
expect(maxCharsPerImage(100)).toBe(19_500);
|
||||
});
|
||||
|
||||
it('maxCharsPerImage: scales linearly with cols (same atlas)', () => {
|
||||
expect(maxCharsPerImage(50)).toBe(7_050);
|
||||
expect(maxCharsPerImage(200)).toBe(28_200);
|
||||
expect(maxCharsPerImage(50)).toBe(9_750);
|
||||
expect(maxCharsPerImage(200)).toBe(39_000);
|
||||
});
|
||||
|
||||
it('isCompressionProfitable: doubling cols halves the 2-image break-even threshold', () => {
|
||||
// At cols=100, CHARS_PER_IMAGE=14,100. 20,000 chars needs 2 images (cost
|
||||
// At cols=100, CHARS_PER_IMAGE=19,500. 20,000 chars needs 2 images (cost
|
||||
// 5000 tokens) vs 5000 text-tokens → tied, strict `<` returns false.
|
||||
expect(isCompressionProfitable(20_000, 100)).toBe(false);
|
||||
// At cols=200, CHARS_PER_IMAGE=28,200. 20,000 chars fits in 1 image
|
||||
// At cols=200, CHARS_PER_IMAGE=39,000. 20,000 chars fits in 1 image
|
||||
// (cost 2500 tokens) vs 5000 text-tokens → clear win.
|
||||
expect(isCompressionProfitable(20_000, 200)).toBe(true);
|
||||
});
|
||||
@@ -1989,21 +1989,21 @@ describe('transform', () => {
|
||||
it('isCompressionProfitable(string): row-aware → newline-heavy sparse content (~5500 chars/img) rejected as net-loss', () => {
|
||||
// Regression for the -69% dashboard bug: the number-arg form estimates
|
||||
// by chars/charsPerImage which assumes uniform line-fill. That assumes
|
||||
// 14100 chars/image but renderTextToPngs actually packs ~141 visual
|
||||
// 19500 chars/image but renderTextToPngs actually packs ~195 visual
|
||||
// rows/image — newline-heavy code/logs hit row cap WAY before char cap.
|
||||
//
|
||||
// 50000 chars of `x.md\n` is 5000 short lines → 5000 rows / 141 = 36
|
||||
// images. 36 * 2500 = 90000 image tokens vs 50000/4 = 12500 text tokens.
|
||||
// 50000 chars of `x.md\n` is 5000 short lines → 5000 rows / 195 = 26
|
||||
// images. 26 * 2500 = 65000 image tokens vs 50000/4 = 12500 text tokens.
|
||||
// Massive net loss. Number-arg form would incorrectly accept (50000 chars
|
||||
// / 14100 chars-per-img = 4 imgs → 10000 < 12500 → "profitable").
|
||||
// / 19500 chars-per-img = 3 imgs → 7500 < 12500 → "profitable").
|
||||
const sparse = 'x.md\n'.repeat(10_000);
|
||||
expect(isCompressionProfitable(sparse, 100)).toBe(false);
|
||||
expect(isCompressionProfitable(sparse.length, 100)).toBe(true); // back-compat: looser estimate
|
||||
});
|
||||
|
||||
it('isCompressionProfitable(string): row-aware → dense single-line content packs full-width and profits', () => {
|
||||
// Same 50000 chars but as ONE line wraps to 100-char rows → 500 rows / 141
|
||||
// = 4 images. 4 * 2500 = 10000 image tokens vs 50000/4 = 12500 text →
|
||||
// Same 50000 chars but as ONE line wraps to 100-char rows → 500 rows / 195
|
||||
// = 3 images. 3 * 2500 = 7500 image tokens vs 50000/4 = 12500 text →
|
||||
// profitable. Both forms agree on dense content.
|
||||
const dense = 'x'.repeat(50_000);
|
||||
expect(isCompressionProfitable(dense, 100)).toBe(true);
|
||||
@@ -2013,7 +2013,7 @@ describe('transform', () => {
|
||||
it('isCompressionProfitable(string, cols, cap): truncation cap lets 500KB log become profitable', () => {
|
||||
// For tool_result paging — actual image cost is bounded by maxImagesPerToolResult
|
||||
// while the SAVED text is the full pre-truncation length. Without cap we'd
|
||||
// reject (50k rows = 355 images), with cap=10 we accept (10*2500=25000 vs
|
||||
// reject (50k rows = 257 images), with cap=10 we accept (10*2500=25000 vs
|
||||
// 500000/4=125000 text → win by 100k).
|
||||
const lines: string[] = [];
|
||||
for (let i = 0; i < 10_000; i++) lines.push(`log entry ${i} payload`);
|
||||
@@ -2022,16 +2022,11 @@ describe('transform', () => {
|
||||
expect(isCompressionProfitable(log, 100, 10)).toBe(true); // capped, profits
|
||||
});
|
||||
|
||||
it('isCompressionProfitable: smaller-cell atlas (Cozette-shape) would let 16k blocks become 1-image wins (cols proxy)', () => {
|
||||
// True smaller-cell test would need to mock ATLAS_CELL_H. We use cols as
|
||||
// a proxy since CHARS_PER_IMAGE = cols × floor((1568−8)/cell_H) — doubling
|
||||
// cols at fixed cell_H is mathematically the same as halving cell_H at
|
||||
// fixed cols. A Cozette 4×7 cell at cols=100 yields floor(1560/7)×100 =
|
||||
// 22,200 chars/image, ~57% more than today. Equivalent: cols=157 at the
|
||||
// current cell. A 16,000-char block needs 2 images today (2-image break-
|
||||
// even fails); at the equivalent Cozette-shape config it fits in 1.
|
||||
expect(isCompressionProfitable(16_000, 100)).toBe(false); // 2 imgs @ 5000 vs 4000 text
|
||||
expect(isCompressionProfitable(16_000, 157)).toBe(true); // 1 img @ 2500 vs 4000 text
|
||||
it('isCompressionProfitable: 5x8 atlas lets 16k blocks become 1-image wins', () => {
|
||||
// The old Unifont 5×11 atlas packed 14,100 chars/image, so 16k chars
|
||||
// needed 2 images and failed break-even. The 5×8 hybrid atlas packs
|
||||
// 19,500 chars/image, so the same block fits in one image and wins.
|
||||
expect(isCompressionProfitable(16_000, 100)).toBe(true);
|
||||
});
|
||||
|
||||
it('break-even gate: 7000-char tool_result stays as text (below break-even)', async () => {
|
||||
@@ -2342,20 +2337,20 @@ describe('transform', () => {
|
||||
describe('real-shape regression (anonymized production events.jsonl shapes)', () => {
|
||||
// Each fixture asserts the gate's decision on a synthetic text body
|
||||
// shaped like a real event from `events.jsonl` (2026-05-19 → 2026-05-20).
|
||||
// The constants `SLAB_CHARS_PER_TOKEN = 2.5` and `HISTORY_CHARS_PER_TOKEN = 2.5`
|
||||
// are empirical fits to N=354 production samples. If a future model
|
||||
// The constants `SLAB_CHARS_PER_TOKEN = 2.0` and `HISTORY_CHARS_PER_TOKEN = 2.0`
|
||||
// are empirical fits to Opus 4.7 production samples. If a future model
|
||||
// (Sonnet 4.6 vs Opus 4.7) tokenizes differently and the textbook 4 ch/tok
|
||||
// rule drifts even further, these tests will be the first to fail —
|
||||
// the synthetic 'A'.repeat(N) shapes elsewhere prove the math but not
|
||||
// the *constants*. Refresh the shape constants from a fresh events.jsonl
|
||||
// when that happens; see tests/fixtures/real-shapes.ts.
|
||||
|
||||
it('production slab (161k chars, multi-col): ACCEPTED at slab cpt=2.5', () => {
|
||||
it('production slab (161k chars, multi-col): ACCEPTED at slab cpt=2.0', () => {
|
||||
const shape = PRODUCTION_SLAB_161K;
|
||||
const text = synthesizeText(shape);
|
||||
// The body that motivated e8545a9. Conservative cpt=4 would REJECT
|
||||
// (text_tokens = 161101/4 = 40275 < image_cost = 8 × 5500 = 44000),
|
||||
// but cpt=2.5 lifts text_tokens to 64440 → ACCEPT with ~20k headroom.
|
||||
// The body that motivated the cpt calibration. Conservative cpt=4 would
|
||||
// reject many dense slabs under the older geometry; cpt=2.0 reflects
|
||||
// Opus 4.7 telemetry and keeps this shape accepted with margin.
|
||||
expect(
|
||||
isCompressionProfitable(text, 100, undefined, shape.numCols, SLAB_CHARS_PER_TOKEN),
|
||||
).toBe(true);
|
||||
@@ -2363,13 +2358,13 @@ describe('transform', () => {
|
||||
expect(isCompressionProfitable(text, 100, undefined, shape.numCols)).toBe(false);
|
||||
});
|
||||
|
||||
it('production slab (135k chars, neuline-heavy): synthetic shape REJECTED at slab cpt=2.5', () => {
|
||||
it('production slab (135k chars, newline-heavy): synthetic shape REJECTED at slab cpt=2.0', () => {
|
||||
const shape = PRODUCTION_SLAB_135K_DENSE;
|
||||
const text = synthesizeText(shape);
|
||||
// Note: the real production event for this shape was ACCEPTED (compressed),
|
||||
// but uniform `'A'.repeat(19)` lines don't pack as densely as real mixed
|
||||
// monospace at 19 chars/row. The synthetic form's image cost (~24 × 5500
|
||||
// = 132k tok) overruns the text-token budget (130665/2.5 = 52k). The
|
||||
// = 132k tok) overruns the text-token budget (130665/2.0 = 65k). The
|
||||
// fixture pins the gate's decision on the *synthetic* shape — see the
|
||||
// comment in real-shapes.ts for why this divergence is expected.
|
||||
expect(
|
||||
@@ -2377,11 +2372,11 @@ describe('transform', () => {
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it('production slab (169k chars, very dense): REJECTED even at slab cpt=2.5', () => {
|
||||
it('production slab (169k chars, very dense): REJECTED even at slab cpt=2.0', () => {
|
||||
const shape = PRODUCTION_SLAB_169K_HEAVY;
|
||||
const text = synthesizeText(shape);
|
||||
// The largest real-event shape we logged. Even at cpt=2.5 the body
|
||||
// (169632/2.5 = 67852 tok) doesn't clear the image cost (37 imgs × 5500
|
||||
// The largest real-event shape we logged. Even at cpt=2.0 the body
|
||||
// (169632/2.0 = 84816 tok) doesn't clear the image cost (37 imgs × 5500
|
||||
// × 2 = 407k tok at multiCol=2). Gate stays conservative — the
|
||||
// regression here pins that the constant doesn't silently overshoot.
|
||||
expect(
|
||||
|
||||
Reference in New Issue
Block a user