17 September 2026 · Explainer · 3 min read

Your inference bill is mostly prefill, and you are paying it twice

Most teams price an LLM feature by output tokens and are surprised by the invoice. The prompt is the expensive half, it is usually identical every call, and there is a way to stop paying for it twice.

You wrote a classifier. The system prompt is 4,000 tokens of rules and examples, carefully tuned. The user sends a support ticket — about 40 tokens. The model answers with one word.

You budgeted for the one word.

The arithmetic nobody does first

A model call has two phases and they cost different things. Prefill reads the prompt: every token, through every layer, in one pass. Decode writes the answer, one token at a time, each one reading everything before it.

For the classifier above, one call is 4,040 tokens in and 1 token out. The output is 0.02% of the work. Price that call by its output and you have priced a rounding error.

Only the last row matches the mental model most people are using.

And you send it again

Here is the part that turns an annoyance into a bill. That 4,000-token system prompt does not change. It is identical on ticket one and ticket ten thousand.

A hosted API charges you for it every time. Ten thousand tickets is forty million prompt tokens, of which 39,996,000 are the same four thousand tokens re-read.

The chat case is worse, because it grows. Turn ten re-sends turns one through nine — the whole transcript, every time — so a conversation’s cost is quadratic in its length while it feels linear to the person having it.

What to do about it

Get the prompt computed once. Both hosted providers and local engines offer some version of this. Anthropic and OpenAI have prompt caching you opt into; the discount is real and the cache is theirs, with its own expiry rules.

Locally the equivalent is a prefix cache, and llamay makes it an object you control rather than a behaviour you hope for: compute the prompt once as a context, then fork it per request.

# The instruction, computed once and kept.
ctx = post("/v1/contexts", {"messages": [{"role": "system", "content": RULES}]})

for ticket in tickets:
    fid = post(f"/v1/contexts/{ctx['id']}/fork?n=1", {})["data"][0]["id"]
    out = post("/v1/chat/completions", {
        "model": "", "llamay_session": fid, "max_tokens": 4, "temperature": 0,
        "messages": [{"role": "user", "content": ticket}],
    })

A fork is a page-table copy, not a memory copy. Forking a 4,000-token instruction a thousand ways costs a thousand small page tables — not four million tokens of key-value cache, and not four million tokens of prefill.

Stop breaking your own cache. The commonest way to lose a prefix cache is to put something that changes at the top of the prompt. A timestamp. A request id. A shuffled example order. Anything that varies invalidates everything after it, so the cache never hits and nobody notices, because the only symptom is a bill.

Put the stable material first and the variable material last. Always.

Check whether you are hitting it. llamay reports it:

curl -s localhost:11435/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"","messages":[{"role":"user","content":"hello"}]}' \
  | jq '.usage.prompt_tokens_details'

{"cached_tokens": 2437}

If that number is zero on the second identical request, something above it is changing.

The number that decides it

High volume and low value per item is the shape where this stops being an optimisation and becomes the whole economics. Parsing a bill of lading is worth a fraction of a cent. Classifying a ticket is worth less. Per-token pricing is designed for the opposite shape — a small number of valuable calls — and that mismatch is where most disappointing API invoices come from.

Run the same workload on hardware you already own and the marginal cost of the next ten thousand tickets is electricity.