mirror of
https://github.com/teamchong/pxpipe.git
synced 2026-07-22 02:02:51 +02:00
fix(savings): use observed cache state for text baseline
This commit is contained in:
Steven Chong
+140
-515
@@ -1,588 +1,213 @@
|
||||
# Prompt-caching alignment & honest savings math
|
||||
# Prompt-Caching Alignment And Honest Savings Math
|
||||
|
||||
This doc answers two questions that keep coming up:
|
||||
This doc answers two questions:
|
||||
|
||||
1. **How does pxpipe stay aligned with Anthropic's prompt cache** when it rewrites
|
||||
the bulky parts of a request into images? (Rewriting the prefix normally
|
||||
*destroys* a warm cache — why doesn't that sink the whole idea?)
|
||||
2. **How do we compute "savings" without counting the prompt-caching discount as
|
||||
if pxpipe earned it?** Caching is a discount Anthropic gives *both* the
|
||||
pxpipe and the no-pxpipe path. If we let it land on only our side of the
|
||||
ledger we'd be inflating the number. This explains the accounting that
|
||||
prevents that.
|
||||
1. How pxpipe stays aligned with Anthropic prompt caching when it rewrites bulky context into images.
|
||||
2. How pxpipe reports savings without counting the provider cache discount as a pxpipe win.
|
||||
|
||||
Source of truth in code: `src/core/baseline.ts` (the math) and
|
||||
`src/core/transform.ts` (the cache-aligned splice). This doc is the prose
|
||||
version of the comments there — if they ever disagree, the code wins.
|
||||
Source of truth in code: `src/core/transform.ts` for the cache-aligned rewrite and `src/core/baseline.ts` for the accounting.
|
||||
|
||||
---
|
||||
|
||||
## Part 0 — Background: how Anthropic prompt caching actually works
|
||||
## Anthropic Prompt Cache Basics
|
||||
|
||||
Everything below follows from one invariant:
|
||||
Anthropic prompt caching is prefix-based:
|
||||
|
||||
> **Prompt caching is a prefix match. Any byte change anywhere in the prefix
|
||||
> invalidates the cache for everything after it.**
|
||||
- The cache key is derived from the exact rendered prompt bytes up to each `cache_control` breakpoint.
|
||||
- Render order is `tools` -> `system` -> `messages`.
|
||||
- A breakpoint is a `"cache_control": {"type": "ephemeral"}` marker on a content block.
|
||||
- The response usage block reports `input_tokens`, `cache_creation_input_tokens`, and `cache_read_input_tokens`.
|
||||
- Total prompt tokens for the actual request are `input_tokens + cache_creation_input_tokens + cache_read_input_tokens`.
|
||||
|
||||
Concretely:
|
||||
Pricing relative to base input rate:
|
||||
|
||||
- The cache key is derived from the **exact bytes** of the rendered prompt up to
|
||||
each `cache_control` breakpoint.
|
||||
- Render order is **`tools` → `system` → `messages`**. A breakpoint caches
|
||||
everything rendered before it.
|
||||
- A breakpoint is a `"cache_control": {"type": "ephemeral"}` marker on a content
|
||||
block. **Max 4 per request.**
|
||||
- The cached prefix has to clear a **minimum size** or it silently won't cache
|
||||
(no error, just `cache_creation_input_tokens: 0`). On **Fable 5 that floor is
|
||||
2048 tokens** (pxpipe is Fable-5-only, so that's our number).
|
||||
- **Pricing**, relative to the base input rate:
|
||||
| bucket | meaning | rate |
|
||||
|---|---|---:|
|
||||
| `input_tokens` | uncached input | `1.0x` |
|
||||
| `cache_creation_input_tokens` (`cc`) | cache write | `1.25x` |
|
||||
| `cache_read_input_tokens` (`cr`) | cache read | `0.1x` |
|
||||
| output | model reply | `5x` input on Fable 5 |
|
||||
|
||||
| bucket | what it is | rate |
|
||||
|---|---|---:|
|
||||
| `input_tokens` | uncached input, paid in full | **1.0×** |
|
||||
| `cache_creation_input_tokens` (`cc`) | tokens written to cache this turn (5-min TTL) | **1.25×** |
|
||||
| `cache_read_input_tokens` (`cr`) | tokens served from a warm cache | **0.1×** |
|
||||
| output | the model's reply (never cached, never compressed) | 5× input on Fable 5 |
|
||||
|
||||
- The response `usage` block reports `input_tokens`, `cache_creation_input_tokens`,
|
||||
and `cache_read_input_tokens`. **Total prompt size = the sum of all three.**
|
||||
|
||||
The economics that make caching matter: a warm read is **~10×** cheaper than
|
||||
paying full freight, but the *first* turn that establishes a cache entry pays a
|
||||
**1.25× write premium**. So a stable prefix that's reused across turns is very
|
||||
cheap after turn 1; a prefix that changes every turn is *more* expensive than
|
||||
not caching at all (you pay the 1.25× write every time and never read).
|
||||
|
||||
That last sentence is the whole problem pxpipe has to solve.
|
||||
A stable prefix is cheap after it is cached, but a prefix that changes every turn repeatedly pays the write premium. pxpipe's cache-alignment work exists to keep the image prefix stable.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — Cache alignment: why rewriting the prefix doesn't break caching
|
||||
## Cache-Aligned Rewrite
|
||||
|
||||
### 1.1 The hazard
|
||||
Claude Code sends a large stable prefix: system prompt, tool docs, reminders, and older history. It also sends a small per-turn tail: the current user message and dynamic runtime context.
|
||||
|
||||
Claude Code's request is mostly a big, **stable prefix** — system prompt, tool
|
||||
docs, `<system-reminder>` blocks, older history — followed by a small,
|
||||
**per-turn tail** (your new message). Claude Code marks the end of that stable
|
||||
prefix with a `cache_control` breakpoint, so from turn 2 on the prefix is served
|
||||
from cache at 0.1×.
|
||||
pxpipe rewrites only the bulky, cacheable parts into images. The dynamic parts stay as text so they do not pollute the image cache key.
|
||||
|
||||
pxpipe rewrites parts of that stable prefix into PNGs. The naive way to do that
|
||||
would **change the cache key**: a prefix that used to be 25k text tokens is now
|
||||
~2.7k image tokens with different bytes. Anthropic sees a brand-new prefix,
|
||||
can't match the old cache entry, and charges `cache_create` (1.25×) on the new
|
||||
content. If pxpipe re-decided every turn — text one turn, image the next — it
|
||||
would pay that write premium *repeatedly* and never settle into a warm read.
|
||||
That's "gate flapping," and it's a money-loser.
|
||||
The key invariant:
|
||||
|
||||
### 1.2 The rule: relocate the marker, never add one
|
||||
> pxpipe does not add new cache-control markers. It relocates the caller's existing marker onto the last image block produced from the same logical content.
|
||||
|
||||
pxpipe's invariant (the code calls it **Task #21**, in `transform.ts`):
|
||||
That keeps the breakpoint at the end of the rewritten stable content. The provider then caches the image prefix exactly as it would have cached the text prefix, just with fewer input tokens under that cache entry.
|
||||
|
||||
> **pxpipe NEVER adds its own `cache_control` marker. It only *relocates* a
|
||||
> marker the caller already set, moving it onto the LAST image block produced
|
||||
> from that content.**
|
||||
The transformed request shape is:
|
||||
|
||||
Why this matters:
|
||||
```text
|
||||
system:
|
||||
billing line / dynamic context / other text-only system content
|
||||
|
||||
- It **doesn't spend any of the 4-breakpoint budget.** The number and rough
|
||||
position of breakpoints is whatever Claude Code chose; pxpipe just follows the
|
||||
text→image flip with the marker so the breakpoint still sits at the *end of
|
||||
the same logical content*.
|
||||
- The cache **anchors at the end of the rewritten static block**, so the
|
||||
per-turn *user* content that follows it stays *outside* the cached region and
|
||||
doesn't pollute the key. (The per-turn *system* blocks — `<env>` etc. — are
|
||||
handled separately, by keeping them out of the image entirely; see 1.3.)
|
||||
|
||||
### 1.3 Split static from dynamic *before* imaging
|
||||
|
||||
The first move is the one that makes the whole thing cache-safe. Claude Code
|
||||
mixes a large **static** slab (the system prompt, agent defs, tool docs) with a
|
||||
handful of **per-turn dynamic** blocks injected into the system text —
|
||||
`<env>`, `<context>`, `<git_status>`, `<directoryStructure>`,
|
||||
`<system-reminder>` (the `DYNAMIC_BLOCK_TAGS` list in `transform.ts`). Those
|
||||
dynamic blocks carry cwd, git branch, today's date, etc., so **their bytes drift
|
||||
turn-to-turn.**
|
||||
|
||||
`splitStaticDynamic` pulls them apart:
|
||||
|
||||
- the **static slab** → rendered into the image (this is the cache anchor);
|
||||
- the **dynamic slab** → forwarded as cheap **text** in the `system` field, never
|
||||
imaged.
|
||||
|
||||
The reason is exactly the prefix invariant: if a drifting `<env>` block were
|
||||
baked *into* the image, the image's bytes would change every turn and its cache
|
||||
entry would die every turn. Keeping the volatile blocks out of the image is what
|
||||
lets the imaged slab stay byte-identical across turns. There's even a canary —
|
||||
any *unrecognized* tag-shaped block left in the static slab is surfaced as
|
||||
telemetry (`unknownTags`), so a future Claude Code release that ships a new
|
||||
per-turn tag can't silently get baked into the cache.
|
||||
|
||||
### 1.4 The cache-friendly splice
|
||||
|
||||
After the split, the request is laid out like this (verified against the splice
|
||||
at the end of `transformRequest` in `transform.ts`):
|
||||
|
||||
```
|
||||
system: [ billing line ] ┐ cheap text, NO cache_control
|
||||
[ dynamic blocks: <env>… ] │ (the drifting per-turn slab)
|
||||
[ sysRemainder ] ┘
|
||||
|
||||
messages[0] (user):
|
||||
[ image block ] ┐ static, rendered slab
|
||||
[ image block ] │
|
||||
[ image block ] ← cache_control (caller's relocated marker = breakpoint)
|
||||
[End of rendered context.] ┐ static text closer for the image
|
||||
[ processed existing content ] ← per-turn user content (incl. reminder
|
||||
images), NO cache_control
|
||||
messages[0] user:
|
||||
image block
|
||||
image block
|
||||
image block + cache_control
|
||||
[End of rendered context.]
|
||||
original user content / live tail
|
||||
```
|
||||
|
||||
Two mechanical constraints drive this shape:
|
||||
|
||||
- **Images can only live in a `user` message.** Anthropic's `system` field
|
||||
accepts text blocks only — an image there returns
|
||||
`400 system.N.type: Input should be 'text'`. So pxpipe moves the imaged slab
|
||||
into the first user message; the `system` field is left holding only cheap
|
||||
text (billing line + the dynamic slab + any non-text `sysRemainder`).
|
||||
- **The marker rides the last image.** Whatever block the caller had marked
|
||||
(the last static system block, a `<system-reminder>`, etc.), its `cache_control`
|
||||
is re-attached to the final image produced from that content, so the breakpoint
|
||||
lands right where the static content ends. The per-turn user content that
|
||||
follows the closer sits *after* that breakpoint, so it never pollutes the image
|
||||
cache key.
|
||||
|
||||
The net effect: the imaged slab is *itself* a stable, cacheable prefix. Once it's
|
||||
written once, every later turn reads it back at 0.1× — exactly like the text
|
||||
prefix did, but over ~9× fewer tokens.
|
||||
|
||||
### 1.5 The one-time "burn" and the anti-flapping gate
|
||||
|
||||
There's no free lunch on the **turn pxpipe first flips text→image** (or flips
|
||||
back). The new image prefix has a different cache key from whatever was warm
|
||||
before, so that turn pays `cache_create` (1.25×) on the image prefix instead of
|
||||
`cache_read` (0.1×) on the old text prefix. The profitability gate accounts for
|
||||
this with a **symmetric burn term** (`isCompressionProfitable` in
|
||||
`transform.ts`):
|
||||
|
||||
```
|
||||
burnImageSide = priorWarmTokens × (CACHE_CREATE_RATE − CACHE_READ_RATE) // = ×1.15
|
||||
burnTextSide = priorWarmImageTokens × (CACHE_CREATE_RATE − CACHE_READ_RATE)
|
||||
|
||||
compress iff imageTokens + burnImageSide < textTokens + burnTextSide
|
||||
```
|
||||
|
||||
> ⚠️ **Implementation note:** the burn term is applied **undivided** — it is *not*
|
||||
> divided by the horizon. (A JSDoc line on `PxpipeOptions.priorWarmTokens` writes
|
||||
> `… / N`, but every call site — `evalCompressionProfitability`,
|
||||
> `isCompressionProfitable`, `isCompressionProfitableAmortized` — computes
|
||||
> `priorWarmTokens × (CACHE_CREATE_RATE − CACHE_READ_RATE)` with no division. The
|
||||
> code, not that comment, is authoritative.)
|
||||
|
||||
The two knobs are what keep the gate from flapping:
|
||||
|
||||
- `priorWarmTokens` — tokens the *un-rewritten* (text) prefix would have read
|
||||
warm. Charged to the **image** side (flipping to image burns the warm text
|
||||
cache, so it discourages compressing while text is warm).
|
||||
- `priorWarmImageTokens` — tokens the *image* prefix is holding warm. Charged to
|
||||
the **text** side (flipping back to text burns the warm image cache).
|
||||
|
||||
Without the symmetric term the gate ping-pongs: per-turn cost favors flipping,
|
||||
the flip forces a fresh `cache_create`, and the next turn flips back — paying the
|
||||
write premium twice. The burn pins the session in its current mode unless the
|
||||
per-turn delta genuinely exceeds the flip cost. Cold-start safe: both default to
|
||||
0, which disables the burn entirely (correct for turn 1 of a fresh conversation).
|
||||
|
||||
### 1.6 Where the horizon *does* divide: the history-collapse gate
|
||||
|
||||
Separately, the **history-collapse** call site uses
|
||||
`isCompressionProfitableAmortized`, which is where `historyAmortizationHorizon`
|
||||
(`N`) earns its keep. It compares *expected lifetime cost* over `N` turns —
|
||||
worst-case-warm for the image (one `cache_create`, then `cache_read` for turns
|
||||
2..N) against best-case-warm for the text (`cache_read` every turn):
|
||||
|
||||
```
|
||||
accept the collapse iff I × (CC + CR×(N−1)) < T × CR × N
|
||||
where CC = 1.25, CR = 0.10
|
||||
```
|
||||
|
||||
So `N` scales the *main* image-vs-text comparison (e.g. `N=1` ⇒ collapse almost
|
||||
never wins, `N=10` ⇒ collapse wins when `I < 0.47·T`), while the burn term above
|
||||
is added on top, undivided. The framing is "assume this prefix gets reused `N`
|
||||
more times; decide once; eat the loss if the session ends early" — the same logic
|
||||
as JIT tiered compilation deciding whether to optimize a hot path. Falls back to
|
||||
the cold per-turn gate when `N ≤ 1`.
|
||||
|
||||
> **Takeaway for Part 1:** pxpipe doesn't fight the cache — it rebuilds an
|
||||
> *equivalent, smaller* cacheable prefix and moves the existing breakpoint to the
|
||||
> end of it. The cache keeps working; it just covers fewer tokens.
|
||||
Images must be placed in a user message because Anthropic does not accept images in the `system` field.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — Savings math: how the cache discount is kept *out* of "savings"
|
||||
## Gate And One-Time Burn
|
||||
|
||||
### 2.1 The trap we're avoiding
|
||||
The raw compression gate compares image token cost with text token cost. For Anthropic, image cost is estimated from pixel area and text cost comes from the configured chars/token estimate for the relevant bucket.
|
||||
|
||||
The cache discount is something Anthropic would give **either path**. If you had
|
||||
*not* run pxpipe, your text prefix would still cache and still read at 0.1× from
|
||||
turn 2 on. So if we measured pxpipe's savings as "full-price text vs.
|
||||
cache-discounted image," we'd be crediting pxpipe with a discount the no-pxpipe
|
||||
path also gets. That double-counts caching as if pxpipe earned it.
|
||||
Switching modes can also burn a warm cache. If a text prefix is warm and pxpipe flips to images, the first image turn may pay a cache write. If an image prefix is warm and pxpipe flips back to text, the text path may pay the write. The symmetric burn terms model that cost:
|
||||
|
||||
The fix is to **apply identical cache pricing to both sides of the same
|
||||
request**, so the discount cancels in the subtraction and only the *token
|
||||
reduction* survives as savings.
|
||||
```text
|
||||
burnImageSide = priorWarmTokens * (1.25 - 0.10)
|
||||
burnTextSide = priorWarmImageTokens * (1.25 - 0.10)
|
||||
|
||||
### 2.2 The measurement (both sides, same request, same moment)
|
||||
|
||||
For every `/v1/messages` POST, the proxy does three things in parallel
|
||||
(`proxy.ts`):
|
||||
|
||||
1. **Forward the real (compressed) request** and read its actual `usage` block:
|
||||
`input_tokens`, `cc`, `cr`. This is what pxpipe *actually cost*.
|
||||
2. **`count_tokens` probe on the ORIGINAL, pre-compression body** → `baseline`.
|
||||
This is the counterfactual: "what would the request have been if pxpipe were
|
||||
off?" `count_tokens` is free and runs concurrently, so it adds no billed cost
|
||||
and ~no latency.
|
||||
3. **`count_tokens` probe on the original body truncated at the last
|
||||
`cache_control` marker** → `baselineCacheable`. This is the size of the prefix
|
||||
that *would have cached* on the unproxied path.
|
||||
|
||||
All three land in the same row of `~/.pxpipe/events.jsonl`, so there's **no
|
||||
turn-count or run-to-run confound** — both arms are the same request at the same
|
||||
instant.
|
||||
|
||||
### 2.3 The proxied (actual) side — `computeActualInputEff`
|
||||
|
||||
```ts
|
||||
actual_eff = input_tokens
|
||||
+ cc × 1.25 // CACHE_CREATE_RATE
|
||||
+ cr × 0.10 // CACHE_READ_RATE
|
||||
compress iff imageTokens + burnImageSide < textTokens + burnTextSide
|
||||
```
|
||||
|
||||
Straight from the billed usage block. No modeling — these are the numbers
|
||||
Anthropic actually charged.
|
||||
This is separate from dashboard savings. The gate decides whether to transform. The dashboard reports what the transformed request actually cost against the measured text counterfactual.
|
||||
|
||||
### 2.4 The counterfactual side — `computeBaselineInputEff`
|
||||
---
|
||||
|
||||
This is the subtle part. We reconstruct what the **unproxied** (text) request
|
||||
would have been billed *against a text cache built up turn-by-turn the same way*.
|
||||
The realization that drives the whole function: **a text prefix is only a cheap
|
||||
cache-read when a warm cache actually existed this turn.** Pricing the cacheable
|
||||
prefix at the read rate unconditionally fabricates a "free read" on genuinely
|
||||
cold turns — turns where the text path would in fact pay the 1.25× *create*,
|
||||
exactly as the imaged path does. That phantom read lands as a fake pxpipe *loss*
|
||||
on cold/growth turns.
|
||||
## Savings Accounting
|
||||
|
||||
So the baseline is **warmth-aware**, and warmth is the honest **union of two
|
||||
independent witnesses** that the *text* prefix was cached this turn — warm iff
|
||||
EITHER fires:
|
||||
The cache discount is provider behavior, not pxpipe savings. To avoid counting cache as savings, both sides use the same observed cache state:
|
||||
|
||||
1. **Wall clock + same static prefix** — the same session had a usage-bearing
|
||||
turn **less than `CACHE_TTL_SEC` (300s) ago**, and the static-prefix hash
|
||||
(`system_sha8`) matches. The text counterfactual is a *separate* hypothetical
|
||||
client whose prefix is **append-only**, so a recent matching prior turn proves
|
||||
that prefix is still cached on Anthropic's side **even when pxpipe busted its
|
||||
OWN image cache** (`cr = 0`) by re-rendering the prefix in place this turn.
|
||||
The text's cache fate is **not** tied to the image's: pricing it warm here
|
||||
correctly *surfaces* the re-imaging loss (cheap warm text < the imaged actual)
|
||||
instead of hiding it behind a matching cold baseline. If `system_sha8`
|
||||
changed, this is a different provider cache key, so the prior does not prove
|
||||
warmth.
|
||||
2. **Observed read** — `cache_read > 0` directly witnesses Anthropic serving a
|
||||
cached prefix this turn. This rescues the **first turn after a restart / TTL /
|
||||
`SESSION_CAP` eviction**, where this process has no in-memory prior yet but the
|
||||
cache is provably warm.
|
||||
- If the actual request has `cr > 0`, the imagined text baseline is priced warm too.
|
||||
- If the actual request has `cr === 0`, the imagined text baseline is priced cold too.
|
||||
|
||||
`cr = 0` does **not** force cold when the static-prefix hash is unchanged — leg 1
|
||||
carries cache-busted re-renders within the window. `cr` is only an *additional*
|
||||
sufficient witness, never a necessary one. A fresh prior with a different
|
||||
`system_sha8` is cold unless `cr > 0` independently proves a read. That is the
|
||||
entire difference from the old `cr > 0`-only rule, which treated image and text
|
||||
as one shared cache slot and so mispriced cache-busted re-renders **cold**, hiding
|
||||
a real loss (see the audit history below).
|
||||
pxpipe does not infer a warm text baseline from wall-clock TTL alone. The text request does not actually exist, so cache warmth for that counterfactual is based only on the real request's server-reported cache read.
|
||||
|
||||
```
|
||||
cacheable = min(baselineCacheable, baseline) // the would-have-cached prefix
|
||||
coldTail = baseline − cacheable // always-cold tail (both paths)
|
||||
For each `/v1/messages` request, pxpipe records three measurements:
|
||||
|
||||
1. Actual upstream usage from the transformed request: `input_tokens`, `cc`, `cr`, and output.
|
||||
2. `baseline_tokens`: `/count_tokens` on the original body before compression.
|
||||
3. `baseline_cacheable_tokens`: `/count_tokens` on the original body truncated at the last `cache_control` marker.
|
||||
|
||||
The actual input cost is:
|
||||
|
||||
```text
|
||||
actual_eff = input_tokens + cc * 1.25 + cr * 0.10
|
||||
```
|
||||
|
||||
Then price `cacheable` by warmth:
|
||||
The text baseline first splits the measured text tokens:
|
||||
|
||||
| case | condition | how the text path is billed |
|
||||
|---|---|---|
|
||||
| **cold turn** | no matching fresh prior **and** `cr = 0` (first turn, a `system_sha8` change, or a >300s gap that let the entry expire with no observed read) | `cacheable × 1.25 + coldTail × 1.0` |
|
||||
| **warm turn** | fresh same-session prior <300s ago with matching `system_sha8` **OR** `cr > 0` (the union) | `reused × 0.10 + grown × 1.25 + coldTail × 1.0` |
|
||||
|
||||
where, on a warm turn:
|
||||
|
||||
```
|
||||
reused = min(prevCacheable, cacheable) // prefix carried from last turn → read at 0.10×
|
||||
grown = cacheable − reused // net-new cacheable bytes this turn → created at 1.25×
|
||||
```text
|
||||
cacheable = min(baseline_cacheable_tokens, baseline_tokens)
|
||||
coldTail = baseline_tokens - cacheable
|
||||
```
|
||||
|
||||
`prevCacheable` is the cacheable-prefix size on the **same session's previous
|
||||
turn** when leg 1 fired (a fresh prior). A growth turn (the conversation got
|
||||
longer) reads the part it already had and creates only the delta; a shrink turn
|
||||
caps `reused` at the current `cacheable` so `grown = 0`. When the turn is warm
|
||||
**only** via leg 2 (`cr > 0` but no usable prior — e.g. just after a restart),
|
||||
there is no measured prior size, so `prevCacheable = cacheable`: `cr` proves a
|
||||
read happened but not the split, so we credit full reuse. Either way this is what
|
||||
the **text** path pays *regardless of what pxpipe's image cache did* — if the
|
||||
image prefix grew and busted its own entry this turn, that real loss stays on the
|
||||
actual side; the baseline doesn't hide it by also charging the text path a create
|
||||
it never would have paid.
|
||||
If the actual request is cold (`cr === 0`):
|
||||
|
||||
> **Signature note.**
|
||||
> `computeBaselineInputEff(baseline, baselineCacheable, inputTokens, cc, cr, warm, prevCacheable)`.
|
||||
> The `warm` flag and `prevCacheable` are produced by **`deriveBaselineWarmth`**
|
||||
> (the union: `warm = freshPrior(<300s, matching system_sha8) || cr > 0`) at every
|
||||
> call site — including the dashboard/sessions replay path, which passes the
|
||||
> **persisted** timestamp and `system_sha8` so it reproduces the live decision
|
||||
> exactly. Once warm, the magnitude of the read is driven by `prevCacheable` (how
|
||||
> much prefix carried over), not by the raw `cr` count.
|
||||
|
||||
Two guard rails short-circuit before the warmth split:
|
||||
|
||||
- `baseline ≤ 0` → return `0` (nothing to compare).
|
||||
- `baselineCacheable ≤ 0` (no markers in the body, or the cacheable probe failed)
|
||||
→ we can't split prefix from tail, so we **credit nothing**: return
|
||||
`computeActualInputEff(...)`, making savings for that turn exactly `0` rather
|
||||
than a guess.
|
||||
|
||||
> **Audit history — this has been wrong three times; don't make it four.**
|
||||
> 1. The *first* baseline collapsed the whole counterfactual into one cache
|
||||
> weight (`cr>0 ? 0.1 : cc>0 ? 1.25 : 1.0`). On a warm turn with mixed
|
||||
> `cc`/`cr` it priced 100% of the unproxied prefix at `0.1×`, making the
|
||||
> unproxied path look 12.5× too cheap and pxpipe look like it *lost money*
|
||||
> (a 7-event May-2026 sample swung from −9,786 "saved" to +19,452 once the
|
||||
> prefix was split from the tail).
|
||||
> 2. The split version was still **warmth-free** — it always read the cacheable
|
||||
> prefix at `0.1×`. That fabricated the "free read" above, resurfacing a
|
||||
> phantom loss on cold/TTL-expiry turns where text really pays the create.
|
||||
> 3. The warmth fix in (2) gated `warm` too narrowly, in two ways that both
|
||||
> priced a **genuinely warm** text prefix COLD — each producing the opposite
|
||||
> lie:
|
||||
> - **Requiring an in-memory prior** missed the first turn after a restart /
|
||||
> TTL / `SESSION_CAP` eviction (cache warm on Anthropic's side, `cr > 0`, but
|
||||
> no prior in this process). Priced cold, its baseline was billed the 1.25×
|
||||
> create against a cheap warm actual — **fabricating an inflated "99% saved"
|
||||
> row** (the operator's originally reported bug).
|
||||
> - **Requiring an observed read (`cr > 0`)** missed the **cache-busted
|
||||
> re-render within the TTL**: pxpipe re-imaged the append-only prefix in
|
||||
> place, so the image missed (`cr = 0`) and paid the create, but the text
|
||||
> prefix was still cached. Pricing it cold gave the baseline a matching cold
|
||||
> create, so the row showed ≈0 — **hiding a real re-imaging loss** (the
|
||||
> inverse of the (2) phantom-saving bug).
|
||||
> 4. Both of (3) are fixed by the **union**: `warm = freshPrior(<300s, matching
|
||||
> system_sha8) || cr > 0`. Leg 1 (wall clock plus same static-prefix hash)
|
||||
> prices cache-busted re-renders warm so the loss surfaces; leg 2 (`cr`) rescues
|
||||
> the no-prior post-restart turn. The text counterfactual is a *separate*
|
||||
> append-only client — its cache fate is decoupled from whether pxpipe busted
|
||||
> its own image cache. If `system_sha8` changed, that is not an append-only
|
||||
> continuation, so the text counterfactual is cold too unless `cr > 0` proves a
|
||||
> read.
|
||||
>
|
||||
> The current model does all of it: split the prefix, gate the read rate on real
|
||||
> warmth, and take the **union** of the matching-hash wall-clock prior and the
|
||||
> observed read so neither a phantom saving (2) nor a hidden loss (3) can return.
|
||||
> `tests/baseline.test.ts` locks `cold(prefix) > warm(prefix)` and the union truth
|
||||
> table; `tests/dashboard-api.test.ts` and the sessions replay tests lock both the
|
||||
> cache-busted-re-render-within-TTL case (warm text, cold image, **loss surfaced**)
|
||||
> and the post-restart `cr`-only case, so none of these regressions can creep back.
|
||||
|
||||
### 2.5 Savings = the difference (caching already cancelled)
|
||||
|
||||
```
|
||||
savings_eff (input) = baseline_eff − actual_eff
|
||||
```text
|
||||
baseline_eff = cacheable * 1.25 + coldTail
|
||||
```
|
||||
|
||||
Output tokens are **excluded from both sides** — they're identical on the two
|
||||
paths (pxpipe never touches the response) and accumulate in their own dashboard
|
||||
bucket. For the **dollar** headline, both sides are converted with the same
|
||||
Fable 5 list ratios — `input ×1.0, cache_write ×1.25, cache_read ×0.1,
|
||||
output ×5` — and since those weights are applied identically on both arms, the
|
||||
caching discount and the output cost both cancel out of the *difference*. What's
|
||||
left is purely the text→image token reduction.
|
||||
If the actual request is warm (`cr > 0`):
|
||||
|
||||
### 2.6 Worked example — same body, warm vs cold
|
||||
|
||||
Take one body two ways. **30,000 tokens**, of which **28,000** are the cacheable
|
||||
prefix (`baseline = 30000`, `baselineCacheable = 28000`), so `coldTail = 2000`.
|
||||
pxpipe images that prefix down to ~3,000 image tokens.
|
||||
|
||||
**(a) Mid-session warm turn.** The previous turn cached a 27,000-token prefix
|
||||
(`prevCacheable = 27000`), so the prefix grew 1,000 this turn. The real response
|
||||
bills `input_tokens = 2000`, `cc = 1000`, `cr = 3000`.
|
||||
```text
|
||||
reused = min(prevCacheable, cacheable)
|
||||
grown = cacheable - reused
|
||||
|
||||
baseline_eff = reused * 0.10 + grown * 1.25 + coldTail
|
||||
```
|
||||
Counterfactual (text, warm):
|
||||
|
||||
`prevCacheable` is used only after `cr > 0` proves a warm read. It refines how much of the text baseline was reused vs newly grown. If there is no completed same-session prior with the same static-prefix hash, pxpipe assumes full reuse for the text baseline: `prevCacheable = cacheable`. That is conservative for savings because it makes the text baseline cheaper.
|
||||
|
||||
Replay uses request start time (`ts - duration_ms`) to avoid overlapping requests refining each other's `prevCacheable` split before the earlier request had completed.
|
||||
|
||||
The savings number is then:
|
||||
|
||||
```text
|
||||
savings_eff = baseline_eff - actual_eff
|
||||
```
|
||||
|
||||
This can be negative when imaging actually costs more under the same cache state. Negative rows are not hidden or floored.
|
||||
|
||||
Rows without a trustworthy cacheable-prefix probe contribute zero savings rather than guessing. Uncompressed rows also contribute zero savings.
|
||||
|
||||
---
|
||||
|
||||
## Worked Examples
|
||||
|
||||
Assume a request whose original text baseline is `30,000` input tokens. Of those, `28,000` are before the cache-control marker, so `coldTail = 2,000`. pxpipe renders that prefix to `3,000` image tokens.
|
||||
|
||||
### Warm Request
|
||||
|
||||
The actual request reports `input_tokens = 2,000`, `cc = 1,000`, `cr = 3,000`. A prior completed row shows `prevCacheable = 27,000`.
|
||||
|
||||
```text
|
||||
Text baseline:
|
||||
reused = min(27000, 28000) = 27000
|
||||
grown = 28000 − 27000 = 1000
|
||||
baseline_eff = 27000×0.10 + 1000×1.25 + 2000×1.0
|
||||
= 2700 + 1250 + 2000 = 5,950
|
||||
grown = 28000 - 27000 = 1000
|
||||
baseline_eff = 27000*0.10 + 1000*1.25 + 2000
|
||||
= 2700 + 1250 + 2000 = 5950
|
||||
|
||||
Proxied (actual):
|
||||
actual_eff = 2000 + 1000×1.25 + 3000×0.10
|
||||
= 2000 + 1250 + 300 = 3,550
|
||||
Actual image request:
|
||||
actual_eff = 2000 + 1000*1.25 + 3000*0.10
|
||||
= 2000 + 1250 + 300 = 3550
|
||||
|
||||
Savings = 5950 − 3550 = 2,400 token-equivalents (~40%)
|
||||
Savings = 5950 - 3550 = 2400
|
||||
```
|
||||
|
||||
The `grown × 1.25` term (1,250) is the same net-new content both paths must
|
||||
create — it cancels against the actual `cc` term. The win is that the proxied arm
|
||||
reads **3,000 image tokens** at 0.1× where the text arm reads **27,000** — 9×
|
||||
fewer tokens sitting under the same discount. That reduction, not the cache
|
||||
discount, is what pxpipe is credited with.
|
||||
The cache read discount applies to both sides. The win is that the warm image prefix is `3,000` tokens while the warm text prefix would have been `27,000` reused text tokens plus `1,000` grown text tokens.
|
||||
|
||||
**(b) Cold turn (first turn, changed static prefix, or a >5-min idle).** Same
|
||||
body, but *genuinely* no warm cache for either path — no fresh same-session prior
|
||||
with matching `system_sha8` **and** `cr = 0`. The imaged request creates its
|
||||
~3,000-token image prefix cold: `input_tokens = 2000`, `cc = 3000`, `cr = 0`.
|
||||
Both legs of the union fail, so the text counterfactual is priced cold too: it
|
||||
would equally have re-created its prefix. (`cr = 0` *alone* is not the tell — a
|
||||
`cr = 0` turn that sits <300s after a matching prior is warm via leg 1; that's
|
||||
case (c).)
|
||||
### Cold Request
|
||||
|
||||
```
|
||||
Counterfactual (text, cold):
|
||||
baseline_eff = 28000×1.25 + 2000×1.0
|
||||
= 35000 + 2000 = 37,000
|
||||
The actual request reports `input_tokens = 2,000`, `cc = 3,000`, `cr = 0`.
|
||||
|
||||
Proxied (actual):
|
||||
actual_eff = 2000 + 3000×1.25 + 0
|
||||
= 2000 + 3750 = 5,750
|
||||
```text
|
||||
Text baseline:
|
||||
baseline_eff = 28000*1.25 + 2000
|
||||
= 35000 + 2000 = 37000
|
||||
|
||||
Savings = 37000 − 5750 = 31,250 token-equivalents
|
||||
Actual image request:
|
||||
actual_eff = 2000 + 3000*1.25
|
||||
= 2000 + 3750 = 5750
|
||||
|
||||
Savings = 37000 - 5750 = 31250
|
||||
```
|
||||
|
||||
The cold turn is pxpipe's biggest win: the text path eats a full `28000×1.25`
|
||||
create; the image path eats only `3000×1.25`.
|
||||
|
||||
> This genuinely-cold turn is what the **warmth-free** version got wrong: it
|
||||
> always read the prefix at `28000×0.10 = 2,800` → `baseline_eff = 4,800`, then
|
||||
> subtracted the real `actual_eff = 5,750` for a **−950 "loss"** on a turn that
|
||||
> actually saved 31,250. The union prices it cold the *honest* way — both legs
|
||||
> fail (no matching fresh prior **and** `cr = 0`), so the text path re-creates its
|
||||
> prefix at 1.25× exactly as the imaged path does, and the phantom loss is gone.
|
||||
> A `cr = 0` turn that *does* sit <300s after a matching prior is the opposite
|
||||
> animal: leg 1 fires, the text is warm, and pricing it cold would hide a loss —
|
||||
> that's (c).
|
||||
|
||||
**(c) Cache-busted re-render inside the window (warm text, cold image).** The
|
||||
*identical* actual request to (b) — `input_tokens = 2000`, `cc = 3000`, `cr = 0`
|
||||
(pxpipe re-rendered the image prefix in place, so its own image cache missed) —
|
||||
but this turn lands <300s after a prior with the same `system_sha8` that cached a
|
||||
27,000-token prefix (`prevCacheable = 27000`, grown 1,000). Leg 1 fires, so the
|
||||
text counterfactual is **warm** even though `cr = 0`:
|
||||
|
||||
```
|
||||
Counterfactual (text, warm — leg 1, prefix was cached regardless of the image):
|
||||
reused = min(27000, 28000) = 27000
|
||||
grown = 28000 − 27000 = 1000
|
||||
baseline_eff = 27000×0.10 + 1000×1.25 + 2000×1.0
|
||||
= 2700 + 1250 + 2000 = 5,950
|
||||
|
||||
Proxied (actual, image re-created cold):
|
||||
actual_eff = 2000 + 3000×1.25 + 0
|
||||
= 2000 + 3750 = 5,750
|
||||
|
||||
Savings = 5950 − 5750 = 200 token-equivalents (≈ break-even)
|
||||
```
|
||||
|
||||
The text prefix was cached no matter what pxpipe did to its image, so it is priced
|
||||
warm and the re-render's full `3000×1.25` create lands on the **actual** side
|
||||
where it belongs — here a near wash.
|
||||
|
||||
> **Why the union, not `cr`, decides this.** (b) and (c) send the byte-identical
|
||||
> actual request (`cr = 0`); only the wall-clock prior differs. The old
|
||||
> `cr > 0`-only rule can't tell them apart — it prices the text **cold** in both
|
||||
> and reports (b)'s `37000 − 5750 = +31,250` for (c) too, a **+31,050 phantom
|
||||
> saving** on a turn pxpipe barely won. Push the re-imaged prefix up relative to
|
||||
> the stable text and (c)'s real number turns **negative** — a genuine re-imaging
|
||||
> loss the cr-only rule buries under that same fake cold create. Leg 1 is what
|
||||
> keeps (c) honest. `tests/dashboard-api.test.ts` locks exactly this contrast.
|
||||
Both sides are cold. pxpipe is not credited for cache; it is credited because the cold image write is much smaller than the cold text write.
|
||||
|
||||
---
|
||||
|
||||
## Part 3 — Reproduce it yourself
|
||||
## Reproducing The Dashboard Math
|
||||
|
||||
Every row in `~/.pxpipe/events.jsonl` carries both arms of the same request.
|
||||
**The JSONL uses shortened key names** (mapped from the Anthropic usage block in
|
||||
`tracker.ts` → `toTrackEvent`):
|
||||
Every row in `~/.pxpipe/events.jsonl` carries the fields needed to reproduce the input-side savings:
|
||||
|
||||
- `baseline_tokens` — `count_tokens` on the original body (full counterfactual)
|
||||
- `baseline_cacheable_tokens` — `count_tokens` truncated at the last
|
||||
`cache_control` marker (omitted/`0` if the body had no markers)
|
||||
- `first_user_sha8` — the **session key**. Rows sharing this value are the same
|
||||
conversation; warmth comes from `deriveBaselineWarmth` over consecutive rows
|
||||
that share it — a fresh prior within the 300s TTL with matching `system_sha8`,
|
||||
**or** an observed `cr > 0`.
|
||||
- the billed `input_tokens`, `cache_create_tokens` (← Anthropic's
|
||||
`cache_creation_input_tokens`), and `cache_read_tokens` (← Anthropic's
|
||||
`cache_read_input_tokens`) from the real response. (A 1-hour cache tier, if
|
||||
ever used, splits out as `cache_create_5m_tokens` / `cache_create_1h_tokens`.)
|
||||
- `baseline_tokens`
|
||||
- `baseline_cacheable_tokens`
|
||||
- `input_tokens`
|
||||
- `cache_create_tokens`
|
||||
- `cache_read_tokens`
|
||||
- `first_user_sha8`
|
||||
- `system_sha8`
|
||||
- `ts`
|
||||
- `duration_ms`
|
||||
|
||||
Because warmth is a **cross-turn** property, replay isn't purely per-row: group
|
||||
rows by `first_user_sha8`, sort each group by `ts`, then walk each session in
|
||||
time order. For each row call
|
||||
`deriveBaselineWarmth(prev, ts, baseline_cacheable_tokens, cache_read_tokens,
|
||||
CACHE_TTL_SEC, system_sha8)` — the exact function the live path uses — where
|
||||
`prev` is the previous in-session row (or `undefined` for the first). It returns
|
||||
`{ warm, prevCacheable }` via the **union**: warm iff a fresh same-session prior
|
||||
exists within the 300s TTL with matching `system_sha8` **or**
|
||||
`cache_read_tokens > 0`. So a post-restart row with no in-session prior but
|
||||
`cr > 0` still prices warm, and a `cr = 0` re-render <300s after a matching prior
|
||||
still prices warm — neither falls through to cold. A `cr = 0` row after a
|
||||
`system_sha8` change is cold because the text counterfactual has a different
|
||||
provider cache key too. (`prevCacheable` follows: the prior row's
|
||||
`baseline_cacheable_tokens` when the fresh-prior leg fired, else this row's own
|
||||
`cacheable` when warm via `cr` alone, else `0`.) Feed `(baseline_tokens,
|
||||
baseline_cacheable_tokens, input_tokens, cache_create_tokens, cache_read_tokens,
|
||||
warm, prevCacheable)` through `computeBaselineInputEff` and the billed triple
|
||||
through `computeActualInputEff` (both exported from `src/core/baseline.ts`), sum
|
||||
the per-row differences, convert with the list ratios above, and you've re-derived
|
||||
the headline. The live dashboard (`DashboardState`) and the JSONL replay both call
|
||||
these **same functions** with the persisted `ts` and `system_sha8`, so the views
|
||||
can't drift.
|
||||
Walk rows in completion order. For each session (`first_user_sha8`), keep the latest completed row's `baseline_cacheable_tokens`, `system_sha8`, and completion timestamp. For the current row:
|
||||
|
||||
### Edge cases worth knowing
|
||||
1. Set `warm = cache_read_tokens > 0`.
|
||||
2. If `warm` and the previous row completed before this request started and has the same `system_sha8`, use its `baseline_cacheable_tokens` as `prevCacheable`.
|
||||
3. If `warm` but no usable prior exists, use this row's `cacheable` as `prevCacheable`.
|
||||
4. If not `warm`, use `prevCacheable = 0`.
|
||||
5. Compute `baseline_eff`, `actual_eff`, and `baseline_eff - actual_eff` with the formulas above.
|
||||
|
||||
- **No `cache_control` markers in the body** (or the cacheable probe failed) →
|
||||
`baselineCacheable ≤ 0`, so `computeBaselineInputEff` returns
|
||||
`computeActualInputEff(...)` and the row contributes **exactly 0 savings** — we
|
||||
refuse to invent a prefix we couldn't measure. The `partial`/`unmeasured`
|
||||
status flags in `transform.ts` record a failed probe so it reads as a visible
|
||||
zero rather than silently biasing the rollup.
|
||||
- **Uncompressed turns are credited 0.** A row pxpipe didn't compress (e.g. a
|
||||
body below the min-chars gate) carries the cost-side gate `creditSaving =
|
||||
haveBaseline && haveUsage && compressed`; when `compressed` is false the
|
||||
baseline is forced to equal the actual, so passthrough turns can't manufacture
|
||||
phantom savings.
|
||||
- **Cold turns cost more on the text side, by design.** The cold branch prices
|
||||
the cacheable prefix at the 1.25× create rate, never the read rate — this is
|
||||
the warmth fix, not a bug. `tests/baseline.test.ts` asserts
|
||||
`cold(prefix) > warm(prefix)` for the same prefix.
|
||||
- **Output is never in this math.** It's identical on both arms and lives in its
|
||||
own accumulator.
|
||||
The live dashboard and replay path both use `deriveBaselineWarmth`, `computeBaselineInputEff`, and `computeActualInputEff` from `src/core/baseline.ts`, so the UI and session summaries use the same math.
|
||||
|
||||
---
|
||||
|
||||
## One-paragraph summary
|
||||
## Summary
|
||||
|
||||
pxpipe stays cache-aligned by rebuilding an *equivalent but smaller* cacheable
|
||||
prefix out of images and **relocating** the caller's existing `cache_control`
|
||||
marker onto the last image — it never adds a marker of its own, so the cache
|
||||
breakpoint still sits at the end of the same logical content and the per-turn
|
||||
tail stays outside the cached region. The one-time `cache_create` "burn" on the
|
||||
flip turn is charged to whichever side would force it, which pins the gate
|
||||
against mode-flapping; the history-collapse gate separately weighs image-vs-text
|
||||
cost over an expected reuse horizon. Savings are then measured by pricing
|
||||
**both** the real request and a `count_tokens` counterfactual of the original
|
||||
body with the **same** cache rates (create 1.25×, read 0.1×) and the **same
|
||||
warmth** — the text counterfactual only reads its prefix cheaply on a turn where
|
||||
the cache genuinely existed for it (a fresh same-session prior within the 300s
|
||||
TTL with matching `system_sha8`, **or** an observed read `cr > 0`), and pays the
|
||||
create otherwise, exactly as the imaged path does. Because both arms face the
|
||||
same discount under the same warmth, it cancels in the difference and what
|
||||
remains as "savings" is only the token reduction from turning dense text into
|
||||
images, never the prompt-caching discount itself.
|
||||
pxpipe stays cache-aligned by replacing stable text context with stable image context and relocating the caller's existing cache marker to the end of the rewritten content. Savings are measured by comparing the real transformed request with a `/count_tokens` text counterfactual under the same observed cache state. If the actual request read cache, both sides are warm. If it did not, both sides are cold. Therefore the provider cache discount is not counted as pxpipe savings; the reported savings are only the token reduction from text to images.
|
||||
|
||||
+30
-43
@@ -8,58 +8,48 @@
|
||||
export const CACHE_CREATE_RATE = 1.25;
|
||||
export const CACHE_READ_RATE = 0.1;
|
||||
|
||||
/** Anthropic prompt-cache TTL (seconds). A turn within this window of the same
|
||||
* session's previous turn still finds its append-only text prefix cached. */
|
||||
/** Anthropic prompt-cache TTL (seconds). Kept for callers that display provider
|
||||
* docs, but savings math does not use TTL to infer a hypothetical text-cache
|
||||
* hit: text is considered warm only when the actual request reports cr > 0. */
|
||||
export const CACHE_TTL_SEC = 300;
|
||||
|
||||
/** This session's previous usage-bearing turn, for wall-clock warmth. */
|
||||
/** This session's previous usage-bearing turn, used only for warm split sizing. */
|
||||
export interface BaselineWarmthPrev {
|
||||
/** Wall-clock seconds of that turn. */
|
||||
/** Completion time of that turn, in wall-clock seconds. */
|
||||
ts: number;
|
||||
/** Cacheable-prefix tokens measured that turn (0 if the probe missed). */
|
||||
cacheable: number;
|
||||
/** Hash of the image-bound/static text prefix. If it changes, the text prefix
|
||||
* was not the same cache entry even inside the TTL. */
|
||||
/** Hash of the image-bound/static text prefix. If it changes, do not reuse the
|
||||
* prior prefix size for this row's text reused/grown split. */
|
||||
prefixSha?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide whether the TEXT counterfactual's prefix was warm this turn, and what
|
||||
* prior prefix size to credit as reused.
|
||||
* Decide whether the TEXT counterfactual's prefix was warm this turn.
|
||||
*
|
||||
* Warmth is the honest UNION of two independent witnesses that the text prefix
|
||||
* was cached this turn — warm iff EITHER fires:
|
||||
* Strict accounting rule: the imagined text path gets the same observed cache
|
||||
* state as the real image path. `cr > 0` is server proof that the request read a
|
||||
* warm prefix, so the text baseline is warm too. `cr === 0` means the actual
|
||||
* request did not read cache, so the text baseline is priced cold too. We do not
|
||||
* use wall-clock TTL to claim that text would have been warm while images were
|
||||
* cold; that would be an unobservable counterfactual and can create negative
|
||||
* rows from cache assumptions rather than token savings.
|
||||
*
|
||||
* 1. WALL CLOCK — a fresh same-session prior within `ttlSec`. The text prefix
|
||||
* is append-only and shares the session's prompt-cache TTL, so a recent
|
||||
* prior turn proves it is still cached even when pxpipe busted its OWN image
|
||||
* cache (cr === 0) by re-rendering the prefix in place this turn. This leg
|
||||
* is decoupled from the image's cache state; the old `cr > 0`-only rule
|
||||
* lacked it and mispriced cache-busted re-renders COLD, turning a real
|
||||
* re-imaging LOSS into a fabricated "saving".
|
||||
*
|
||||
* 2. OBSERVED READ — cr > 0 directly witnesses Anthropic serving a cached
|
||||
* prefix this turn. This rescues the first turn after a pxpipe restart /
|
||||
* SESSION_CAP eviction while the cache is still warm (no in-memory prior,
|
||||
* yet cr proves warmth). Without it that turn is priced COLD and fabricates
|
||||
* an inflated "saved" row — the operator's original reported bug.
|
||||
*
|
||||
* Crucially, cr === 0 does NOT force cold when the prefix hash is unchanged
|
||||
* (leg 1 carries those turns); cr is only an ADDITIONAL sufficient witness,
|
||||
* never a necessary one. But a fresh prior with a different prefix hash is cold:
|
||||
* it is a different provider cache key, not an append-only continuation.
|
||||
* See docs/CACHING_AND_SAVINGS.md.
|
||||
* When cr proves warmth, a completed same-prefix prior is used only to estimate
|
||||
* how much of the text prefix was reused vs grown. If none is available, assume
|
||||
* full reuse of this turn's cacheable prefix; this is conservative for savings.
|
||||
*
|
||||
* @param prev this session's previous usage-bearing turn, or undefined.
|
||||
* @param nowSec wall-clock seconds of the current turn (replay passes the
|
||||
* persisted ts so it reproduces the live decision exactly).
|
||||
* @param nowSec request-start wall-clock seconds, used only to reject prior
|
||||
* rows that had not completed before this request was sent.
|
||||
* @param cacheable this turn's cacheable-prefix tokens (the full-reuse credit
|
||||
* when warm only via cr, since cr proves a read but not the split).
|
||||
* @param cr observed cache-read tokens this turn (the leg-2 witness).
|
||||
* @param ttlSec cache TTL window (defaults to CACHE_TTL_SEC).
|
||||
* @param cr observed cache-read tokens this turn; the only warm/cold signal.
|
||||
* @param ttlSec legacy parameter; no longer decides warm/cold. It only
|
||||
* bounds whether a prior prefix size is used for reused/grown
|
||||
* splitting after cr > 0 has already proved warmth.
|
||||
* @param prefixSha stable-prefix fingerprint for the text counterfactual. A
|
||||
* fresh wall-clock prior only proves warmth when this matches
|
||||
* the prior turn; otherwise the provider would see a new key.
|
||||
* prior prefix size is reused only when this matches.
|
||||
*/
|
||||
export function deriveBaselineWarmth(
|
||||
prev: BaselineWarmthPrev | undefined,
|
||||
@@ -74,15 +64,12 @@ export function deriveBaselineWarmth(
|
||||
|| prev.prefixSha === undefined
|
||||
|| prefixSha === undefined
|
||||
|| prev.prefixSha === prefixSha;
|
||||
// Leg 1: a fresh same-session prior within the TTL (wall-clock warmth).
|
||||
// cr is the only warm/cold signal. A prior only refines the warm split.
|
||||
if (!(cr > 0)) return { warm: false, prevCacheable: 0 };
|
||||
// Fresh prior: use its real prefix size for the reused/grown split. Without
|
||||
// one, cr proves warmth but not the split, so assume full reuse.
|
||||
const freshPrior = prev !== undefined && age >= 0 && age < ttlSec && samePrefix;
|
||||
// Leg 2: an observed read directly witnesses a warm cache. Union of the two.
|
||||
const warm = freshPrior || cr > 0;
|
||||
// Fresh prior → credit its real measured prefix as reused (the reused/grown
|
||||
// split). Warm only via cr (no usable prior) → cr proves a read happened but
|
||||
// not the split, so assume full reuse of this turn's cacheable prefix.
|
||||
const prevCacheable = freshPrior ? prev!.cacheable : warm ? cacheable : 0;
|
||||
return { warm, prevCacheable };
|
||||
return { warm: true, prevCacheable: freshPrior ? prev!.cacheable : cacheable };
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
+35
-47
@@ -423,14 +423,13 @@ export class DashboardState {
|
||||
* ever carried a `firstUserSha8` (e.g. a cold start with only passthrough
|
||||
* hits that the upstream probe never tagged). */
|
||||
private currentSessionId: string | null = null;
|
||||
/** Per-session warmth for the cache-aware TEXT baseline: the wall-clock
|
||||
* second + cacheable-prefix size of each session's previous turn. A turn is
|
||||
* "warm" when <5min since the same session's last turn (Anthropic's cache
|
||||
* TTL), which decides whether the text counterfactual reads (0.10×) or
|
||||
* re-creates (1.25×) its prefix. Reconstructed identically in replay() from
|
||||
* persisted `ts`, so live and restored numbers agree. Capped with sessions. */
|
||||
/** Per-session prior prefix size for the cache-aware TEXT baseline. Warm/cold
|
||||
* comes only from the server-observed cache_read on the actual request; this
|
||||
* map is used only after cr>0 to split the text counterfactual into reused vs
|
||||
* grown prefix tokens. Reconstructed identically in replay() from persisted
|
||||
* timestamps, so live and restored numbers agree. Capped with sessions. */
|
||||
private baselineWarmth: Map<string, { ts: number; cacheable: number; prefixSha?: string }> = new Map();
|
||||
/** Anthropic prompt-cache TTL in seconds; a gap beyond this is a cold turn. */
|
||||
/** Max age for reusing a prior prefix size after cr>0 has proved warmth. */
|
||||
private static readonly CACHE_TTL_SEC = 300;
|
||||
/** Hard cap on `sessions` Map entries. Keeps memory bounded in
|
||||
* long-running deployments. 50 sessions × ~13 numeric fields each is
|
||||
@@ -583,9 +582,8 @@ export class DashboardState {
|
||||
let rawBaseline: number; // raw text-only counterfactual tokens
|
||||
let baselineForRow: number; // baseline token count for contextHistory/recent
|
||||
let cacheReadForRow: number; // tokens to surface in the "Cache hits" column
|
||||
let warmForRow: boolean; // did the TEXT baseline read warm? (wall-clock for
|
||||
// Anthropic; cached_tokens>0 for GPT) — drives the Context Map narration so
|
||||
// the sub-line can't contradict baselineInputEff on a cache-busted turn.
|
||||
let warmForRow: boolean; // did the TEXT baseline read warm? Server-observed:
|
||||
// Anthropic cr>0 or GPT cached_tokens>0. Drives Context Map narration.
|
||||
|
||||
if (gpt) {
|
||||
// GPT cost model: no count_tokens probe, no cache-create premium, no
|
||||
@@ -641,38 +639,29 @@ export class DashboardState {
|
||||
// actually compressed AND we have a usable probe.
|
||||
creditSaving = haveBaseline && haveUsage && compressed;
|
||||
|
||||
// Cache-aware, warmth-aware baseline. INVARIANT: pxpipe is credited ONLY for
|
||||
// the text it imaged away — NEVER for caching. The shared cached prefix is
|
||||
// priced at the SAME warmth on both sides (deriveBaselineWarmth forces the
|
||||
// text counterfactual warm whenever the actual got a read, cr>0), so the
|
||||
// 0.10×/1.25× cache rates cancel and only the token-count reduction survives
|
||||
// into the saving. Warmth can therefore only LOWER the credited saving, never
|
||||
// raise it (pinned: savings-honesty.test.ts "OVERCLAIM GUARD"). No >=0 floor —
|
||||
// a net-losing cc-heavy / cache-busted re-render lowers it honestly. Per-session
|
||||
// warmth is keyed by firstUserSha8 and fed the persisted ts on replay, so live
|
||||
// update() and replay() agree exactly. Uncompressed rows fall back to
|
||||
// actualInputEff → zero savings. See src/core/baseline.ts + docs/CACHING_AND_SAVINGS.md.
|
||||
// Cache-aware, server-observed baseline. INVARIANT: pxpipe is credited ONLY
|
||||
// for the text it imaged away — NEVER for caching. The imagined text path
|
||||
// gets the same observed cache state as the actual request: cr>0 means warm
|
||||
// for both, cr===0 means cold for both. No wall-clock-only inference.
|
||||
// Uncompressed rows fall back to actualInputEff → zero savings.
|
||||
const cacheable = info?.baselineCacheableTokens ?? 0;
|
||||
// Cache-aware warmth: was this session's prefix warm (<TTL since its last
|
||||
// turn)? Decides whether the text counterfactual reads or re-creates. Keyed
|
||||
// by firstUserSha8; live uses the wall clock, replay() the persisted ts.
|
||||
// If cr>0 proved warmth, a completed prior with the same prefix refines the
|
||||
// reused/grown split for the text baseline. Use request start for that
|
||||
// lookup; an overlapping request that had not completed could not provide a
|
||||
// prior prefix size for this in-flight request.
|
||||
const sidNow = info?.firstUserSha8;
|
||||
const prefixShaNow = info?.systemSha8;
|
||||
const nowSec = Date.now() / 1000;
|
||||
const completionSec = Date.now() / 1000;
|
||||
const requestStartSec = completionSec - Math.max(0, ev.durationMs || 0) / 1000;
|
||||
const warmthPrev =
|
||||
typeof sidNow === 'string' && sidNow.length > 0
|
||||
? this.baselineWarmth.get(sidNow)
|
||||
: undefined;
|
||||
// Warmth = honest UNION of two witnesses the TEXT prefix was cached: a
|
||||
// fresh same-session prior within the TTL AND the same static-prefix hash,
|
||||
// OR cr>0 (an observed read). The hash guard matters when opencode changes
|
||||
// system/tool guidance mid-session: same wall clock, different cache key.
|
||||
// cr>0 still rescues the first post-restart turn that has no in-memory
|
||||
// prior yet. Centralised in deriveBaselineWarmth so update()/replay()/
|
||||
// sessions can't drift apart. See docs.
|
||||
// Warmth itself is cr-only; prior state only estimates the warm split.
|
||||
// Centralised in deriveBaselineWarmth so update()/replay()/sessions can't drift.
|
||||
const { warm, prevCacheable } = deriveBaselineWarmth(
|
||||
warmthPrev,
|
||||
nowSec,
|
||||
requestStartSec,
|
||||
cacheable,
|
||||
cr,
|
||||
DashboardState.CACHE_TTL_SEC,
|
||||
@@ -689,12 +678,11 @@ export class DashboardState {
|
||||
prevCacheable,
|
||||
)
|
||||
: actualInputEff;
|
||||
// Record this turn's warmth footprint for the next turn in this session.
|
||||
// Touch on every usage-bearing row (the cache is warmed regardless of
|
||||
// compression); carry the prior cacheable when this row has no probe.
|
||||
// Record this completed turn's prefix size for future cr>0 split estimates.
|
||||
// Carry the prior cacheable when this row has no probe.
|
||||
if (typeof sidNow === 'string' && sidNow.length > 0 && haveUsage) {
|
||||
this.baselineWarmth.set(sidNow, {
|
||||
ts: nowSec,
|
||||
ts: completionSec,
|
||||
cacheable: cacheable > 0 ? cacheable : (warmthPrev?.cacheable ?? 0),
|
||||
prefixSha: prefixShaNow ?? warmthPrev?.prefixSha,
|
||||
});
|
||||
@@ -716,7 +704,7 @@ export class DashboardState {
|
||||
rawBaseline = baseline ?? 0;
|
||||
baselineForRow = baseline ?? 0;
|
||||
cacheReadForRow = cr;
|
||||
warmForRow = warm; // union: fresh wall-clock prior OR cr>0 (see deriveBaselineWarmth)
|
||||
warmForRow = warm; // server-observed cache read (cr>0)
|
||||
}
|
||||
|
||||
// Record the request's transform breakdown for the Context Map panel. This
|
||||
@@ -970,19 +958,19 @@ export class DashboardState {
|
||||
// compressed rows. Uncompressed/passthrough rows fall back to the
|
||||
// actual cost so they show zero saved (no fabricated savings).
|
||||
creditSaving = haveBaseline && haveUsage && compressed;
|
||||
// Cache-aware warmth, reconstructed from persisted ts so replay matches
|
||||
// the live update() path (see baselineWarmth field + update()).
|
||||
// Warm/cold is reconstructed from server-observed cr only. Persisted
|
||||
// completion ts + duration_ms are used only to find a prior prefix size
|
||||
// for the reused/grown split after cr>0 has proved warmth.
|
||||
const sidR = (t as { first_user_sha8?: string }).first_user_sha8;
|
||||
const prefixShaR = (t as { system_sha8?: string }).system_sha8;
|
||||
const tsSec = Date.parse(t.ts) / 1000;
|
||||
const completionSecR = Date.parse(t.ts) / 1000;
|
||||
const requestStartSecR = completionSecR - Math.max(0, t.duration_ms || 0) / 1000;
|
||||
const warmthPrevR =
|
||||
typeof sidR === 'string' && sidR.length > 0 ? this.baselineWarmth.get(sidR) : undefined;
|
||||
// Same union warmth as update() (persisted ts instead of the live clock
|
||||
// so replay reproduces live numbers exactly): matching-hash fresh prior
|
||||
// OR cr>0. See deriveBaselineWarmth.
|
||||
// Same cr-only warmth as update(); prior state only refines the split.
|
||||
const { warm: warmR, prevCacheable: prevCacheableR } = deriveBaselineWarmth(
|
||||
warmthPrevR,
|
||||
tsSec,
|
||||
requestStartSecR,
|
||||
cacheable,
|
||||
cr,
|
||||
DashboardState.CACHE_TTL_SEC,
|
||||
@@ -1001,7 +989,7 @@ export class DashboardState {
|
||||
: actualInputEff;
|
||||
if (typeof sidR === 'string' && sidR.length > 0 && haveUsage) {
|
||||
this.baselineWarmth.set(sidR, {
|
||||
ts: tsSec,
|
||||
ts: completionSecR,
|
||||
cacheable: cacheable > 0 ? cacheable : (warmthPrevR?.cacheable ?? 0),
|
||||
prefixSha: prefixShaR ?? warmthPrevR?.prefixSha,
|
||||
});
|
||||
@@ -1014,7 +1002,7 @@ export class DashboardState {
|
||||
rawBaseline = baseline ?? 0;
|
||||
baselineForRow = baseline ?? 0;
|
||||
cacheReadForRow = cr;
|
||||
warmForRow = warmR; // wall-clock text warmth, NOT cr
|
||||
warmForRow = warmR; // server-observed cache read
|
||||
}
|
||||
// Rebuild the Context Map breakdown so old rows keep their "Saved" value
|
||||
// and "Details" link after a restart. The PNG ring is in-memory and gone,
|
||||
|
||||
+11
-27
@@ -348,14 +348,10 @@ export interface ContextMapData {
|
||||
baselineInputEff: number; // cache-WEIGHTED baseline — what text would actually be billed
|
||||
actualInputEff: number; // cache-WEIGHTED actual — what the images were actually billed
|
||||
haveBaseline: boolean; // weighted pair is trustworthy (baseline probe resolved)
|
||||
cacheRead: number; // the IMAGE request's cache_read tokens this turn. >0 ⇒ the
|
||||
// image hit cache; ===0 ⇒ a cold or cache-busted image render. Used ONLY to
|
||||
// explain the image side of the gap — it does NOT decide text warmth.
|
||||
warm: boolean; // did the TEXT baseline's prefix read warm? A wall-clock decision
|
||||
// (same session within the cache TTL), decoupled from the image cache state —
|
||||
// see deriveBaselineWarmth / src/core/baseline.ts. Drives the narration so the
|
||||
// sub-line can't contradict baselineInputEff on a cache-busted turn (text warm
|
||||
// at 0.1× while the image was re-created cold).
|
||||
cacheRead: number; // cache_read tokens this turn. >0 ⇒ the actual request hit cache.
|
||||
warm: boolean; // did the TEXT baseline's prefix read warm? Server-observed only:
|
||||
// true iff the actual request had cache_read > 0. This keeps the text baseline
|
||||
// on the same cache state as the image path; no wall-clock-only inference.
|
||||
output: number;
|
||||
imageCount: number;
|
||||
buckets: Partial<Record<string, number>>; // bucket → chars rendered to PNG
|
||||
@@ -423,18 +419,10 @@ export function renderContextMapFragment(
|
||||
? `<div class="pages-title">${c.imageCount} image page${c.imageCount === 1 ? '' : 's'} were sent — thumbnails expired when the proxy restarted. The breakdown above is reconstructed from the saved log.</div>`
|
||||
: '';
|
||||
|
||||
// Did the TEXT baseline's prefix read warm this turn? This is a WALL-CLOCK
|
||||
// decision (same session within the cache TTL), decoupled from the image
|
||||
// request's cache state — see deriveBaselineWarmth / src/core/baseline.ts.
|
||||
// Keying the narration on c.warm (NOT raw cache_read) is what keeps this
|
||||
// sub-line consistent with baselineInputEff on a cache-busted turn, where the
|
||||
// text prefix is warm (0.1× read) yet the image was re-created cold.
|
||||
// Did the TEXT baseline's prefix read warm this turn? This follows the actual
|
||||
// request's observed cache state: cache_read > 0 means warm, cache_read === 0
|
||||
// means cold. No wall-clock-only counterfactual is credited.
|
||||
const warm = showCompare && c.warm;
|
||||
// Cache-busted re-render: the text counterfactual stayed warm but pxpipe
|
||||
// re-imaged the prefix and missed the image cache (cache_read===0), so the
|
||||
// image paid the 1.25× create rate. This is why a physically-smaller prefix
|
||||
// can still cost MORE as images than it would have as cached text.
|
||||
const imageBusted = warm && c.cacheRead === 0;
|
||||
const textNoun = warm ? 'cached text' : 'text';
|
||||
// Raw count_tokens can grow (imaging bloated a short prompt), so say so rather
|
||||
// than rendering a nonsensical "shrank -36%".
|
||||
@@ -445,18 +433,14 @@ export function renderContextMapFragment(
|
||||
: pct >= 0
|
||||
? `<span class="ctx-big">${pct}%</span> smaller — ${textNoun} would bill as <strong>${kFmt(base)}</strong> input tokens; images billed as <strong>${kFmt(real)}</strong>`
|
||||
: `<span class="ctx-big">${-pct}%</span> bigger — images billed as <strong>${kFmt(real)}</strong> input tokens vs <strong>${kFmt(base)}</strong> for ${textNoun}`;
|
||||
// Clarifying sub-line. It must match the turn's real cache state: claiming a
|
||||
// 0.1× read discount on a cold turn (where the prefix actually paid the 1.25×
|
||||
// create rate) — or, conversely, denying the text its warm read on a turn
|
||||
// where pxpipe merely busted its OWN image cache — is exactly the kind of
|
||||
// contradiction this panel exists to kill.
|
||||
// Clarifying sub-line. It must match the actual request's cache state: claiming
|
||||
// a 0.1× read discount when cache_read===0 would count hypothetical cache as a
|
||||
// pxpipe effect, so cold rows price both paths cold.
|
||||
const subnote = !showCompare
|
||||
? 'Billed tokens count cache discounts (reads at 0.1×) — no trustworthy text baseline for this request yet.'
|
||||
: !warm
|
||||
? `No warm text cache this turn — the text counterfactual's prefix is priced at the 1.25× create rate (the same event the imaged path pays), identical basis to the Saved column. The gap is purely token count. ${rawPhrase}`
|
||||
: imageBusted
|
||||
? `The text counterfactual stays cached — its append-only prefix reads at 0.1× (same session, within the cache TTL) — but pxpipe re-imaged the prefix and missed the image cache this turn (cache_read = 0), paying the 1.25× create rate. So imaging cost more even though the text would have read warm. Same basis as the Saved column. ${rawPhrase}`
|
||||
: pct < 0 && rawShrink > 0
|
||||
: pct < 0 && rawShrink > 0
|
||||
? `Billed = after cache discounts (reads at 0.1×), same basis as the Saved column. The raw text is ${rawShrink}% smaller, but most of it would have been a cheap cache-read — so imaging it cost more.`
|
||||
: `Billed = after cache discounts (reads at 0.1×), same basis as the Saved column. ${rawPhrase}`;
|
||||
const title = isLatest ? 'Latest request' : 'Selected request';
|
||||
|
||||
+16
-27
@@ -136,9 +136,10 @@ export async function aggregateSessions(
|
||||
): Promise<AggregateResult> {
|
||||
const sessions = new Map<string, SessionSummary>();
|
||||
const sidecarsBySession = new Map<string, Set<string>>();
|
||||
// Per-session warmth for the cache-aware text counterfactual, reconstructed
|
||||
// from event ts as we scan in time order (mirrors DashboardState). Kept out
|
||||
// of SessionSummary so it never leaks into the /api/sessions.json shape.
|
||||
// Per-session prior prefix sizes for the cache-aware text counterfactual.
|
||||
// Warm/cold comes only from server-observed cache_read; this map only refines
|
||||
// reused/grown splitting after cr>0 has proved warmth. Kept out of
|
||||
// SessionSummary so it never leaks into the /api/sessions.json shape.
|
||||
const warmth = new Map<string, { ts: number; cacheable: number; prefixSha?: string }>();
|
||||
const CACHE_TTL_SEC = 300;
|
||||
|
||||
@@ -172,18 +173,9 @@ export async function aggregateSessions(
|
||||
// rare and the first cwd is the most stable identifier.
|
||||
if (s.project === undefined && ev.cwd) s.project = ev.cwd;
|
||||
// Real per-session savings, cache-aware. See src/core/baseline.ts for the
|
||||
// full derivation: the cached prefix cancels (paid identically on both
|
||||
// paths), so we credit only the net-new uncached text pxpipe compressed
|
||||
// away — honest `baselineEff − actualEff`, NO >=0 floor, so a net-losing
|
||||
// turn (cc-heavy rewrite) lowers the total. Warmth decides whether the text
|
||||
// counterfactual reads (0.10×) or re-creates (1.25×) the cacheable prefix:
|
||||
// the text prefix's warmth tracks its OWN wall-clock stability (a fresh
|
||||
// same-session prior within the TTL), decoupled from whether pxpipe busted
|
||||
// its IMAGE cache this turn — so a cache-busted re-render (`cr === 0`) under
|
||||
// a warm clock still reads warm for text, and we don't fabricate a 1.25×
|
||||
// create it never would have paid. An observed read (`cr > 0`) is the other
|
||||
// leg, rescuing post-restart turns with no in-memory prior. See
|
||||
// deriveBaselineWarmth for the exact union.
|
||||
// full derivation: pxpipe is credited only for token reduction, never for
|
||||
// caching. The imagined text baseline gets the SAME observed cache state as
|
||||
// the actual request: cr>0 means warm for both, cr===0 means cold for both.
|
||||
// Events missing either probe stay out of the rollup — no estimation.
|
||||
const inp = ev.input_tokens ?? 0;
|
||||
const cc = ev.cache_create_tokens ?? 0;
|
||||
@@ -197,16 +189,14 @@ export async function aggregateSessions(
|
||||
) {
|
||||
const cacheable = ev.baseline_cacheable_tokens ?? 0;
|
||||
const prefixSha = ev.system_sha8;
|
||||
const tsSec = Date.parse(ev.ts) / 1000;
|
||||
const completionSec = Date.parse(ev.ts) / 1000;
|
||||
const requestStartSec = completionSec - Math.max(0, ev.duration_ms || 0) / 1000;
|
||||
const prev = warmth.get(id);
|
||||
// Warmth = honest union (mirrors dashboard.ts, centralised in
|
||||
// deriveBaselineWarmth): warm iff a fresh same-session prior is within the
|
||||
// TTL AND has the same static-prefix hash, OR cr>0 witnesses a read. cr=0
|
||||
// no longer forces cold when the hash matches (cache-busted image re-render),
|
||||
// and cr>0 rescues the first post-restart turn with no in-memory prior.
|
||||
// Warmth is cr-only; a completed same-prefix prior only refines the
|
||||
// reused/grown split after cr>0 has proved warmth.
|
||||
const { warm, prevCacheable } = deriveBaselineWarmth(
|
||||
prev,
|
||||
tsSec,
|
||||
requestStartSec,
|
||||
cacheable,
|
||||
cr,
|
||||
CACHE_TTL_SEC,
|
||||
@@ -226,16 +216,15 @@ export async function aggregateSessions(
|
||||
s.tokensSavedEst += Math.round(tokensSaved);
|
||||
s.charsSaved += Math.round(tokensSaved * 4);
|
||||
}
|
||||
// Warm the cache for the next turn in this session (every usage-bearing
|
||||
// row warms it, regardless of compression). Carry prior cacheable when this
|
||||
// row had no probe so a probe gap doesn't reset warmth.
|
||||
// Record this completed row's prefix size for future cr>0 split estimates.
|
||||
// Carry prior cacheable when this row had no probe.
|
||||
if (haveUsage) {
|
||||
const tsSec = Date.parse(ev.ts) / 1000;
|
||||
const completionSec = Date.parse(ev.ts) / 1000;
|
||||
const cacheable = ev.baseline_cacheable_tokens ?? 0;
|
||||
const prefixSha = ev.system_sha8;
|
||||
const prev = warmth.get(id);
|
||||
warmth.set(id, {
|
||||
ts: tsSec,
|
||||
ts: completionSec,
|
||||
cacheable: cacheable > 0 ? cacheable : (prev?.cacheable ?? 0),
|
||||
prefixSha: prefixSha ?? prev?.prefixSha,
|
||||
});
|
||||
|
||||
+20
-25
@@ -70,15 +70,12 @@ describe('computeBaselineInputEff (warmth-aware)', () => {
|
||||
});
|
||||
|
||||
/**
|
||||
* deriveBaselineWarmth decides WHEN the text counterfactual was warm — the
|
||||
* honest UNION of two independent witnesses that the text prefix was cached:
|
||||
* 1. a fresh same-session prior within the TTL (wall clock), and
|
||||
* 2. an observed read (cr>0).
|
||||
* cr === 0 does NOT force cold — the wall-clock leg carries a cache-busted
|
||||
* re-render — and the cr leg rescues the post-restart turn that has no in-memory
|
||||
* prior yet. The old rule used cr ALONE (cr necessary), which mispriced both.
|
||||
* deriveBaselineWarmth decides WHEN the text counterfactual was warm. The rule
|
||||
* is server-observed: text is warm iff the actual request reported cr>0. A prior
|
||||
* completed turn can refine reused-vs-grown prefix size after cr has proved a
|
||||
* warm read, but it never makes a cr===0 row warm by itself.
|
||||
*/
|
||||
describe('deriveBaselineWarmth (honest union: fresh prior in TTL OR cr>0)', () => {
|
||||
describe('deriveBaselineWarmth (server-observed: cr>0 only)', () => {
|
||||
const prev = (ts: number, cacheable: number, prefixSha?: string) => ({
|
||||
ts,
|
||||
cacheable,
|
||||
@@ -96,26 +93,24 @@ describe('deriveBaselineWarmth (honest union: fresh prior in TTL OR cr>0)', () =
|
||||
expect(deriveBaselineWarmth(undefined, 1000, 5000, 100)).toEqual({ warm: true, prevCacheable: 5000 });
|
||||
});
|
||||
|
||||
it('THE FIX: warm via a fresh prior even when cr===0 (pxpipe busted its own image cache)', () => {
|
||||
// 60s since the last turn, image cache missed (cr=0) — text prefix is still
|
||||
// cached. The old cr-alone rule returned cold here and fabricated savings.
|
||||
it('keeps text cold when cr===0 even with a fresh prior', () => {
|
||||
// The text path is hypothetical. Without a server-observed read on the real
|
||||
// request, do not claim the text counterfactual would have read cache.
|
||||
expect(deriveBaselineWarmth(prev(1000, 8000), 1060, 5000, 0)).toEqual({
|
||||
warm: true,
|
||||
prevCacheable: 8000,
|
||||
});
|
||||
});
|
||||
|
||||
it('cold when the fresh prior has a different static-prefix hash', () => {
|
||||
// Same session + inside TTL is not enough: if the system/tool prefix changed,
|
||||
// the text-only request would use a different prompt-cache key too.
|
||||
expect(deriveBaselineWarmth(prev(1000, 8000, 'old'), 1060, 5000, 0, CACHE_TTL_SEC, 'new')).toEqual({
|
||||
warm: false,
|
||||
prevCacheable: 0,
|
||||
});
|
||||
});
|
||||
|
||||
it('keeps the fresh-prior warmth when the static-prefix hash matches', () => {
|
||||
expect(deriveBaselineWarmth(prev(1000, 8000, 'same'), 1060, 5000, 0, CACHE_TTL_SEC, 'same')).toEqual({
|
||||
it('observed read is warm even when the prior hash cannot refine the split', () => {
|
||||
expect(deriveBaselineWarmth(prev(1000, 8000, 'old'), 1060, 5000, 10, CACHE_TTL_SEC, 'new')).toEqual({
|
||||
warm: true,
|
||||
prevCacheable: 5000,
|
||||
});
|
||||
});
|
||||
|
||||
it('uses a matching fresh prior only to refine the warm reused/grown split', () => {
|
||||
expect(deriveBaselineWarmth(prev(1000, 8000, 'same'), 1060, 5000, 10, CACHE_TTL_SEC, 'same')).toEqual({
|
||||
warm: true,
|
||||
prevCacheable: 8000,
|
||||
});
|
||||
@@ -129,8 +124,8 @@ describe('deriveBaselineWarmth (honest union: fresh prior in TTL OR cr>0)', () =
|
||||
});
|
||||
|
||||
it('a fresh prior gives the real prior prefix size; cr-only warmth assumes full reuse', () => {
|
||||
// fresh prior → prevCacheable = prev.cacheable (real reused/grown split)
|
||||
expect(deriveBaselineWarmth(prev(1000, 3000), 1100, 5000, 0).prevCacheable).toBe(3000);
|
||||
// observed read + fresh prior → prevCacheable = prev.cacheable (real reused/grown split)
|
||||
expect(deriveBaselineWarmth(prev(1000, 3000), 1100, 5000, 50).prevCacheable).toBe(3000);
|
||||
// warm via cr but stale prior → full-reuse assumption (prevCacheable = cacheable)
|
||||
expect(deriveBaselineWarmth(prev(0, 3000), 1_000_000, 5000, 50).prevCacheable).toBe(5000);
|
||||
});
|
||||
@@ -144,7 +139,7 @@ describe('deriveBaselineWarmth (honest union: fresh prior in TTL OR cr>0)', () =
|
||||
// Whichever way warmth resolves, pricing warm must not claim MORE savings
|
||||
// than cold for the same prefix (warm lowers, or holds, the baseline).
|
||||
const cacheable = 4000;
|
||||
const { warm, prevCacheable } = deriveBaselineWarmth(prev(1000, 4000), 1100, cacheable, 0);
|
||||
const { warm, prevCacheable } = deriveBaselineWarmth(prev(1000, 4000), 1100, cacheable, 50);
|
||||
// inp/cc/cr only feed the probe-miss branch (cacheable>0 here ⇒ unused).
|
||||
const warmEff = computeBaselineInputEff(5000, cacheable, 0, 0, 0, warm, prevCacheable);
|
||||
const coldEff = computeBaselineInputEff(5000, cacheable, 0, 0, 0, false, 0);
|
||||
|
||||
+16
-30
@@ -6,12 +6,9 @@
|
||||
* the exact contradiction that made the number untrustworthy. These tests pin
|
||||
* the two panels together.
|
||||
*
|
||||
* Warmth is the UNION decision from deriveBaselineWarmth (a fresh same-session
|
||||
* prior within the cache TTL OR an observed read), carried on
|
||||
* ContextMapData.warm and decoupled from the image request's own cache_read.
|
||||
* The narration is keyed on c.warm (text warmth), NOT raw cache_read, so it can
|
||||
* never contradict baselineInputEff on a cache-busted re-render (text warm at
|
||||
* 0.1× while the image was re-created cold).
|
||||
* Warmth is server-observed: ContextMapData.warm is true only when the actual
|
||||
* request reported cache_read > 0. The narration must never claim a hypothetical
|
||||
* text cache read on a cache_read=0 row.
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import {
|
||||
@@ -101,10 +98,9 @@ describe('renderContextMapFragment — cache-aware headline', () => {
|
||||
|
||||
describe('renderContextMapFragment — cold vs warm honesty', () => {
|
||||
// The headline/sub-line must not claim a 0.1× read discount on a turn whose
|
||||
// text prefix was NOT warm. Warmth is c.warm (the union decision), not the
|
||||
// image's raw cache_read. On a cold turn (no fresh prior and no read — e.g.
|
||||
// the first turn of a big document) the text baseline's prefix is priced at
|
||||
// the 1.25× create rate, so "cached text" / "reads at 0.1×" would be a lie.
|
||||
// actual request had no cache read. On a cold turn the text baseline's prefix
|
||||
// is priced at the 1.25× create rate too, so "cached text" / "reads at 0.1×"
|
||||
// would be counting unobserved cache as savings.
|
||||
it('COLD turn (no warmth): no read discount claimed, text is not called "cached"', () => {
|
||||
const html = renderContextMapFragment(
|
||||
ctx({
|
||||
@@ -166,33 +162,23 @@ describe('renderContextMapFragment — cold vs warm honesty', () => {
|
||||
expect(html).not.toContain('cheap cache-read');
|
||||
});
|
||||
|
||||
it('cache-busted re-render within the TTL: text stays warm, image missed → loss surfaced without contradiction', () => {
|
||||
// The union regression case (point 1). pxpipe re-imaged the append-only
|
||||
// prefix in place: the IMAGE missed its cache (cacheRead === 0) and paid the
|
||||
// 1.25× create, but the TEXT counterfactual is still warm (fresh same-session
|
||||
// prior within the TTL — c.warm === true). So the cache-weighted baseline is
|
||||
// the cheap warm read, the row is an honest LOSS (image actual > warm text),
|
||||
// and the narration must NOT fall into the cold branch and price the text at
|
||||
// the create rate — that would hide the loss behind a matching cold baseline.
|
||||
it('cache_read=0: text is cold too, no cache-busted warm-text narration', () => {
|
||||
const html = renderContextMapFragment(
|
||||
ctx({
|
||||
warm: true,
|
||||
warm: false,
|
||||
cacheRead: 0,
|
||||
baselineInputEff: 1000,
|
||||
actualInputEff: 2000,
|
||||
baselineTokens: 4000,
|
||||
baselineInputEff: 3500,
|
||||
actualInputEff: 2500,
|
||||
baselineTokens: 3000,
|
||||
realInput: 2000,
|
||||
}),
|
||||
[],
|
||||
);
|
||||
// Honest loss in the headline, against the WARM ("cached text") basis.
|
||||
expect(html).toContain('bigger');
|
||||
expect(html).toContain('for cached text');
|
||||
// The image-busted explanation — text warm, image cold, loss surfaced.
|
||||
expect(html).toContain('re-imaged the prefix and missed the image cache');
|
||||
expect(html).toContain('the text would have read warm');
|
||||
// It is NOT the cold branch: the text really was warm this turn.
|
||||
expect(html).not.toContain('No warm text cache this turn');
|
||||
expect(html).toContain('smaller');
|
||||
expect(html).toContain('text would bill as');
|
||||
expect(html).toContain('No warm text cache this turn');
|
||||
expect(html).not.toContain('cached text');
|
||||
expect(html).not.toContain('re-imaged the prefix and missed the image cache');
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
+63
-37
@@ -350,23 +350,10 @@ describe('GPT savings split', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('union warmth: fresh wall-clock prior OR cr>0 (no phantom savings, no hidden losses)', () => {
|
||||
// The text counterfactual's warmth is the honest UNION of two independent
|
||||
// witnesses that the TEXT prefix was cached this turn: a fresh same-session
|
||||
// prior within the TTL (wall clock), OR an observed read (cr>0). The text
|
||||
// prefix is append-only, so a recent prior keeps it warm even when pxpipe
|
||||
// busts its OWN image cache (cr=0) by re-rendering the prefix in place. On
|
||||
// such a turn pxpipe really did pay the 1.25× create, so pricing the text
|
||||
// baseline warm correctly SURFACES that re-imaging loss instead of hiding it
|
||||
// behind a matching cold baseline — gating warmth on cr alone fabricated a
|
||||
// win out of a real loss. (The cr>0 leg separately rescues the post-restart
|
||||
// turn that has no in-memory prior yet but is provably cached on Anthropic's
|
||||
// side.)
|
||||
|
||||
// A warm-priming turn followed by a cache-busted re-render (cr=0) in the SAME
|
||||
// session within the TTL. The fresh prior makes the text baseline warm, so
|
||||
// the row reads the still-cached text prefix cheaply and honestly shows the
|
||||
// re-imaging loss rather than a phantom saving.
|
||||
describe('server-observed warmth: text follows actual cache_read', () => {
|
||||
// The text counterfactual is hypothetical, so its cache state follows the only
|
||||
// server-observed signal we have: cr>0 means warm for both paths, cr===0 means
|
||||
// cold for both paths. A prior row only refines reused/grown split after cr>0.
|
||||
function antEvt(
|
||||
usage: {
|
||||
input_tokens: number;
|
||||
@@ -397,8 +384,9 @@ describe('union warmth: fresh wall-clock prior OR cr>0 (no phantom savings, no h
|
||||
};
|
||||
}
|
||||
|
||||
it('surfaces the re-imaging loss on a cache-busted re-render within the TTL', async () => {
|
||||
// Turn 1: genuine warm read — primes the per-session warmth map.
|
||||
it('prices text cold when the actual image request has cache_read=0', async () => {
|
||||
// Turn 1 records a prior prefix size, but it must not make a later cr=0 row
|
||||
// warm by wall-clock inference alone.
|
||||
dash.update(
|
||||
antEvt(
|
||||
{
|
||||
@@ -411,10 +399,8 @@ describe('union warmth: fresh wall-clock prior OR cr>0 (no phantom savings, no h
|
||||
) as never,
|
||||
);
|
||||
|
||||
// Turn 2: pxpipe busted its OWN image cache and re-rendered the prefix in
|
||||
// place — cache_read === 0, full re-create. But the TEXT prefix is
|
||||
// append-only and still cached (fresh same-session prior within the TTL),
|
||||
// so the text counterfactual is warm.
|
||||
// Turn 2: actual request has cache_read === 0 and pays a full re-create.
|
||||
// The imagined text path gets the same cold cache state.
|
||||
dash.update(
|
||||
antEvt(
|
||||
{
|
||||
@@ -436,19 +422,59 @@ describe('union warmth: fresh wall-clock prior OR cr>0 (no phantom savings, no h
|
||||
// actual = 100 + 20000×1.25 = 25100 (what pxpipe actually paid this turn).
|
||||
expect(miss.actual_input).toBe(25100);
|
||||
|
||||
// WARM text baseline: a text-only client would have READ its still-cached
|
||||
// append-only prefix at 0.1× (20000×0.1) + 10000 cold tail = 12000 — NOT
|
||||
// the cold 20000×1.25 + 10000 = 35000 the old cr-alone rule produced.
|
||||
expect(miss.baseline_input).toBe(12000);
|
||||
|
||||
// So this turn is an HONEST LOSS: pxpipe's re-imaging cost 13100 tokens more
|
||||
// than a text-only client would have paid (12000 − 25100). The dashboard
|
||||
// surfaces it — not floored, not hidden behind a matching cold baseline.
|
||||
expect(miss.session_saved_so_far_delta).toBe(-13100);
|
||||
expect(miss.session_saved_so_far_delta!).toBeLessThan(0);
|
||||
// Cold text baseline: 20000×1.25 + 10000 tail = 35000.
|
||||
expect(miss.baseline_input).toBe(35000);
|
||||
expect(miss.session_saved_so_far_delta).toBe(9900);
|
||||
});
|
||||
|
||||
it('prices text cold when the static prefix hash changed inside the TTL', async () => {
|
||||
it('does not let an overlapping request warm the text counterfactual before it completed', async () => {
|
||||
writeEvents(tmp, [
|
||||
ev({
|
||||
ts: '2026-05-19T00:00:20.000Z',
|
||||
duration_ms: 20_000,
|
||||
compressed: true,
|
||||
first_user_sha8: 'overlap',
|
||||
system_sha8: 'stable-system',
|
||||
baseline_probe_status: 'ok',
|
||||
baseline_tokens: 30_000,
|
||||
baseline_cacheable_tokens: 20_000,
|
||||
input_tokens: 100,
|
||||
output_tokens: 50,
|
||||
cache_create_tokens: 20_000,
|
||||
cache_read_tokens: 0,
|
||||
}),
|
||||
ev({
|
||||
// Starts at 00:00:15, five seconds BEFORE the prior request completed.
|
||||
// cr>0 proves warmth, but that prior could not refine the text baseline's
|
||||
// reused/grown split for this in-flight request.
|
||||
ts: '2026-05-19T00:00:25.000Z',
|
||||
duration_ms: 10_000,
|
||||
compressed: true,
|
||||
first_user_sha8: 'overlap',
|
||||
system_sha8: 'stable-system',
|
||||
baseline_probe_status: 'ok',
|
||||
baseline_tokens: 32_000,
|
||||
baseline_cacheable_tokens: 22_000,
|
||||
input_tokens: 100,
|
||||
output_tokens: 50,
|
||||
cache_create_tokens: 2_000,
|
||||
cache_read_tokens: 20_000,
|
||||
}),
|
||||
]);
|
||||
await dash.replay(tmp.eventsFile);
|
||||
|
||||
const recent = (await dash.serveRecent().json()) as RecentPayload;
|
||||
const overlap = recent.recent.at(-1)!;
|
||||
expect(overlap.cache_read).toBe(20000);
|
||||
expect(overlap.actual_input).toBe(4600);
|
||||
// Warm via cr>0, but no completed prior was available at send time, so the
|
||||
// text baseline assumes full reuse instead of using the overlapping prior:
|
||||
// 22000×0.1 + 10000 tail = 12200.
|
||||
expect(overlap.baseline_input).toBe(12200);
|
||||
expect(overlap.session_saved_so_far_delta).toBe(7600);
|
||||
});
|
||||
|
||||
it('prices text cold when cache_read=0 even if the static prefix hash changed inside the old TTL window', async () => {
|
||||
dash.update(
|
||||
antEvt(
|
||||
{
|
||||
@@ -479,7 +505,7 @@ describe('union warmth: fresh wall-clock prior OR cr>0 (no phantom savings, no h
|
||||
|
||||
const recent = (await dash.serveRecent().json()) as RecentPayload;
|
||||
const changed = recent.recent.at(-1)!;
|
||||
// Static prefix changed, so the text-only path would create too:
|
||||
// cache_read=0, so the text path is cold too:
|
||||
// baseline = 20000*1.25 + 10000 tail = 35000, not warm 12000.
|
||||
expect(changed.baseline_input).toBe(35000);
|
||||
expect(changed.session_saved_so_far_delta).toBe(9900);
|
||||
@@ -522,8 +548,8 @@ describe('union warmth: fresh wall-clock prior OR cr>0 (no phantom savings, no h
|
||||
|
||||
it('prices a warm read warm even with NO prior warmth state (post-restart)', async () => {
|
||||
// The cache is already warm on Anthropic's side (cr>0), but this process has
|
||||
// never seen the session — exactly the first turn after a pxpipe restart, a
|
||||
// >5min idle (TTL eviction), or a SESSION_CAP eviction. The OLD code required
|
||||
// never seen the session — exactly the first turn after a pxpipe restart or a
|
||||
// SESSION_CAP eviction. The OLD code required
|
||||
// an in-memory warmthPrev entry, so it fell through to the COLD branch and
|
||||
// billed the known-cached prefix the 1.25× CREATE rate — fabricating the
|
||||
// inflated "99% saved" row the operator reported. cr>0 is direct proof the
|
||||
|
||||
+55
-28
@@ -147,8 +147,8 @@ describe('aggregateSessions', () => {
|
||||
cache_create_tokens: 800,
|
||||
cache_read_tokens: 100,
|
||||
}),
|
||||
// WARM turn (same session, same ts < TTL): prior cacheable 18000 >= 9000,
|
||||
// so the whole prefix is reused @0.1 = 900. actual = 5 + 8000*0.1 = 805.
|
||||
// WARM turn because cr>0. Prior cacheable 18000 >= 9000, so the whole
|
||||
// text prefix is reused @0.1 = 900. actual = 5 + 8000*0.1 = 805.
|
||||
// saved = 95.
|
||||
ev({
|
||||
first_user_sha8: 'aaaaaaaa',
|
||||
@@ -194,10 +194,9 @@ describe('aggregateSessions', () => {
|
||||
input_tokens: 5,
|
||||
cache_read_tokens: 90_000,
|
||||
}),
|
||||
// Genuine loss turn: tiny body (2000) but pp wrote 5000 cache_create. The
|
||||
// prior turn was a probe miss (no cacheable recorded), so warmth carries
|
||||
// prevCacheable=0 -> no reuse credited: text re-creates 1900 prefix at
|
||||
// 1.25x. baseline = 1900*1.25 + 100 = 2475 ; actual = 3000 + 5000*1.25
|
||||
// Genuine loss turn: tiny body (2000) but pxpipe wrote 5000 cache_create.
|
||||
// cr=0, so text is cold too and re-creates 1900 prefix at 1.25x:
|
||||
// baseline = 1900*1.25 + 100 = 2475 ; actual = 3000 + 5000*1.25
|
||||
// = 9250. saved = 2475 - 9250 = -6775. Honest formula, no clamp.
|
||||
ev({
|
||||
first_user_sha8: 'bbbbbbbb',
|
||||
@@ -215,20 +214,14 @@ describe('aggregateSessions', () => {
|
||||
expect(s.charsSaved).toBe(-6_775 * 4);
|
||||
});
|
||||
|
||||
it('prices a busted image re-render within the TTL as a real loss (text stays warm)', async () => {
|
||||
// Two turns 60s apart — well inside the 300s TTL. Turn 2's cache_read_tokens
|
||||
// === 0: pxpipe's IMAGE cache missed and it re-created the prefix (cc>0).
|
||||
// But the text counterfactual's prefix is APPEND-ONLY — turn 1 cached it and
|
||||
// it is still warm 60s later regardless of what pxpipe's images did. So the
|
||||
// honest price is WARM for text: this turn pxpipe genuinely LOST tokens by
|
||||
// busting an image cache that text would have read for free. cr ALONE (the
|
||||
// old rule) called turn 2 cold and fabricated a 31250 "win" it never earned.
|
||||
// Empirically 12.8% of warm turns hit this cr=0 / text-warm shape — see
|
||||
// docs/CACHING_AND_SAVINGS.md.
|
||||
it('prices text cold when the actual request has cache_read=0', async () => {
|
||||
// Turn 2 is 60s after turn 1, but cr=0 means the server did not report a
|
||||
// cache read for the actual request. The imagined text path gets the same
|
||||
// cold cache state; no wall-clock-only cache warmth is credited.
|
||||
writeEvents(tmp, [
|
||||
// Turn 1 (genuine cold first turn) — establishes a 28k cacheable prefix.
|
||||
// Turn 1 (genuine cold first turn).
|
||||
// cold baseline = 28000*1.25 + 2000 tail = 37000 ; actual = 2000 + 3000*1.25 = 5750
|
||||
// saved = 31250. (Also seeds warmth: ts + prevCacheable=28000.)
|
||||
// saved = 31250. (Also records prevCacheable=28000 for future cr>0 rows.)
|
||||
ev({
|
||||
first_user_sha8: 'cccccccc',
|
||||
ts: '2026-05-19T00:00:00.000Z',
|
||||
@@ -239,11 +232,8 @@ describe('aggregateSessions', () => {
|
||||
cache_create_tokens: 3_000,
|
||||
cache_read_tokens: 0,
|
||||
}),
|
||||
// Turn 2, +60s (inside TTL), cache_read_tokens=0 — pxpipe re-rendered and
|
||||
// its image cache missed. Text was still warm (append-only prefix, 60s old):
|
||||
// WARM (correct): reused 28000@0.1 = 2800 + 2000 tail = 4800 ; actual = 5750 ;
|
||||
// saved = 4800 - 5750 = -950 — the real cost of busting a warm image cache.
|
||||
// COLD (old cr-alone bug): baseline = 28000*1.25 + 2000 = 37000 ; saved = 31250 fabricated.
|
||||
// Turn 2, +60s, cache_read_tokens=0 — actual request is cold, so the text
|
||||
// counterfactual is cold too: baseline = 28000*1.25 + 2000 = 37000.
|
||||
ev({
|
||||
first_user_sha8: 'cccccccc',
|
||||
ts: '2026-05-19T00:01:00.000Z',
|
||||
@@ -257,9 +247,46 @@ describe('aggregateSessions', () => {
|
||||
]);
|
||||
const { sessions } = await aggregateSessions(tmp);
|
||||
const s = sessions.get('cccccccc')!;
|
||||
// 31250 (honest cold win) + (-950) (honest busted-rerender loss) = 30300.
|
||||
// The old cr-alone code booked the second turn as +31250 → a 62500 overclaim.
|
||||
expect(s.tokensSavedEst).toBe(30_300);
|
||||
// 31250 + 31250. No hypothetical text-cache read is credited on cr=0.
|
||||
expect(s.tokensSavedEst).toBe(62_500);
|
||||
});
|
||||
|
||||
it('does not treat an overlapping in-flight request as a completed warm prior', async () => {
|
||||
writeEvents(tmp, [
|
||||
ev({
|
||||
first_user_sha8: 'eeeeeeee',
|
||||
system_sha8: 'stable-system',
|
||||
ts: '2026-05-19T00:00:20.000Z',
|
||||
duration_ms: 20_000,
|
||||
compressed: true,
|
||||
baseline_tokens: 30_000,
|
||||
baseline_cacheable_tokens: 20_000,
|
||||
input_tokens: 100,
|
||||
cache_create_tokens: 20_000,
|
||||
cache_read_tokens: 0,
|
||||
}),
|
||||
ev({
|
||||
// Starts at 00:00:15, before the previous request completed at 00:00:20.
|
||||
// cr>0 proves warmth, but the prior row was not available to split the
|
||||
// text baseline into reused/grown tokens at this request's send time.
|
||||
first_user_sha8: 'eeeeeeee',
|
||||
system_sha8: 'stable-system',
|
||||
ts: '2026-05-19T00:00:25.000Z',
|
||||
duration_ms: 10_000,
|
||||
compressed: true,
|
||||
baseline_tokens: 32_000,
|
||||
baseline_cacheable_tokens: 22_000,
|
||||
input_tokens: 100,
|
||||
cache_create_tokens: 2_000,
|
||||
cache_read_tokens: 20_000,
|
||||
}),
|
||||
]);
|
||||
const { sessions } = await aggregateSessions(tmp);
|
||||
const s = sessions.get('eeeeeeee')!;
|
||||
// Turn 1 saved 9900. Turn 2 is warm via cr>0, but with no completed prior at
|
||||
// send time it assumes full reuse: baseline = 22000*0.1 + 10000 = 12200;
|
||||
// actual = 100 + 2000*1.25 + 20000*0.1 = 4600; saved = 7600.
|
||||
expect(s.tokensSavedEst).toBe(17_500);
|
||||
});
|
||||
|
||||
it('does not treat a fresh prior as warm when the static prefix hash changed', async () => {
|
||||
@@ -293,8 +320,8 @@ describe('aggregateSessions', () => {
|
||||
const s = sessions.get('dddddddd')!;
|
||||
// Turn 1: baseline full-reuse via cr = 20000*0.1 + 10000 = 12000;
|
||||
// actual = 100 + 20000*0.1 = 2100; saved = 9900.
|
||||
// Turn 2: static hash changed, so text is cold too: baseline = 35000;
|
||||
// actual = 25100; saved = 9900. Old wall-clock-only warmth booked -13100.
|
||||
// Turn 2: cr=0, so text is cold too: baseline = 35000;
|
||||
// actual = 25100; saved = 9900. No wall-clock-only warmth is credited.
|
||||
expect(s.tokensSavedEst).toBe(19_800);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user