How LLM API pricing actually works, and why your bill does not match your estimate

You are billed twice, and the second time costs more

Every request to a hosted language model bills in tokens, and almost every provider charges input and output at different rates. Input is what you send — instructions, context, retrieved documents, conversation history. Output is what the model generates back.

Output is normally the more expensive side, by a substantial multiple, because generating text is the part that occupies the GPU for the longest. So a request that is tiny going out and verbose coming back is not a cheap request.

• A rough English ratio: about 4 characters per token, so roughly 1.3 tokens per word.
• A 1,000-word document is therefore around 1,300 tokens before any formatting or markup tax.
• Numbers, code and punctuation tokenise worse than prose — budget more for them, not less.

Estimate in tokens, never in words. Words feel like the unit because we read in them, but they are not the unit you are charged for, and the error compounds with every request.

Why a chat feels cheap and an agent does not

This is the single largest source of surprise in AI budgets, and it is structural rather than a vendor decision.

Language model APIs are stateless. The model does not remember your last message. To hold a conversation, your application re-sends the entire history on every turn: the system prompt, every earlier user message, every earlier reply, and the new one. All of it is billed as input, every single time.

So the cost of a conversation is not linear in its length. It is closer to triangular — each turn costs more than the last, and the total grows with the square of the number of turns.

Twenty turns in the table above bill roughly 130,000 input tokens in total. The same twenty exchanges sent independently, with no history, would bill about 10,000. The conversation itself — not the questions, not the answers — is the cost.

An agent makes this worse in two ways: it adds tool descriptions and tool results to the context, and it loops. A five-step agent task can re-send the same growing context five times before it finishes, and bill for the retries too.

Three costs that never appear in the estimate

• Reasoning tokens. Models that think before answering bill their internal working as output. You pay for text you never see, and it can exceed the visible answer.
• Retries and failures. A timed-out request, a rate-limited call, a malformed tool response — all of them consumed input tokens that were still billed.
• Retrieved context. Retrieval-augmented systems insert documents into every request. Retrieve a 5,000-token passage into a 20-turn conversation and you have added 100,000 tokens to the bill.

Prompt caching is the biggest lever you are probably not using

Most providers offer cached input pricing: if the beginning of your request is byte-identical to a recent one, that prefix may be billed at a much lower rate than fresh input.

The mechanism has one design consequence. Caching works on a prefix, so the stable part must come first. Put the unchanging system instructions at the top, then the static reference material, then the variable user input at the end. Interleave them and you break the cache on every request — and pay full price while believing you are caching.

Reordering a prompt so the static prefix sits first can cut input cost several-fold with no change in behaviour. It is the highest-return hour of work in most AI cost reductions.

A support bot, costed

Illustrative figures, chosen to show the shape of the arithmetic rather than to quote any provider's current rates. Rates change constantly — use your own numbers in the estimator.

The two lines worth attention are the cached share and the retry rate. Doubling the cached share cuts input cost materially. Removing the retries removes cost that delivered no answer at all — and a 4% retry rate means about 1,200 requests a month are billed twice.

Controls that measurably reduce the bill

1. Route by difficulty. Send classification and simple extraction to the smallest model that is accurate enough, and reserve the expensive model for reasoning. Most traffic does not need the largest model.
2. Cap the context and summarise. Replacing turns 1 to 10 with a 200-token summary stops the triangular growth without losing the thread.
3. Stabilise the prefix. Static content first, variable content last, so caching actually engages.
4. Log tokens per request. A monthly total tells you the bill; a per-request distribution tells you which feature to fix. The expensive calls are usually a small, findable minority.
5. Retry deliberately. Bound the retries, and back off rather than hammering — a retry storm costs money and worsens the rate limiting that caused it.

Estimate before you build, not after

The LLM API Cost Estimator models input, output, cached and retry volumes together, with per-model prices held in a single file that carries its own verification date and source links — and an override so you can drop in today's rates without waiting for an update.

What an estimate cannot tell you

Token counts vary with the tokeniser, so any estimate is a model of a model. Treat the output as a planning figure with a range, not a quote. And prices move: a provider reducing rates by half is a normal event, not an exception, which is why the estimator keeps its sources visible rather than burying them.