You priced the feature at $0.003 per call. The invoice came in three times higher and nobody on the team can explain the gap. You’re not alone — most teams shipping AI features can quote the provider’s per-million-token pricing from memory but cannot itemize their own bill.
That gap is not a billing error. It is a measurement problem dressed up as a cost problem, and the fix isn’t another optimization article. It’s an honest accounting of where the money actually goes. By the end of this, you’ll know the four buckets your LLM token counting api costs live in — and which one is usually the only one worth touching.
The pricing page is not your unit cost
Same prompt, same response, three different providers — and three different token counts. Independent benchmarks have measured up to 2.65x variance in tokenization across major models for identical inputs. The number on the pricing page is multiplied by a hidden coefficient nobody publishes.
The variance gets uglier on tool-heavy workloads. Claude can land roughly 5x more expensive than GPT on the same agent trajectory, even when the per-token rate looks competitive, because the way it tokenizes tool calls inflates the count more aggressively. If you compared the two by their pricing pages and picked the cheaper-looking one, you may have picked the more expensive one.
The honest version of model comparison: run your actual prompts through each model in a one-day spike, log the input and output tokens each one bills you for, then compare. A serious multi-LLM routing strategy starts there, not on a pricing page.
Tokenization explains a chunk of the gap between what you planned and what you paid. It is not the biggest part.
Where the tokens actually go in production
Your bill lives in four buckets. Most teams only see one of them.
Bucket one — visible completion tokens. The part you actually planned for in the spreadsheet. In a typical production system this is somewhere between 30% and 40% of the bill. It is also the bucket every “8 ways to cut your AI costs” article is implicitly talking about.
Bucket two — system prompt overhead. A 2,000-token system prompt sent on every call costs roughly $0.004 per call on a mid-tier model. Multiply by a million calls a month and that’s $4,000 you didn’t put on the napkin math. The system prompt does not appear in the response. It does appear on the invoice.
Bucket three — context growth in multi-turn conversations. Every turn re-sends the entire conversation history. By turn 20 of a chat session, the user is paying to re-process the system prompt twenty times and the conversation history a growing number of times. Per-session token cost rises closer to quadratically than linearly.
Bucket four — retries and tool-call overhead. Failed JSON parses, timeouts, agent loops that re-prompt themselves when a tool returns garbage. Every retry pays full input cost again. Retry rates of 5–15% are common in production, and most teams have never measured theirs.
These four buckets are not equal. Which one dominates depends entirely on what you’re building.
Your workload decides which bucket dominates
A long-running chat or assistant has context growth as the dominant bucket. By the twentieth turn, the same system prompt and conversation history are being re-billed over and over. The optimization target is summarization, sliding-window history, and prompt caching for Claude or its equivalent on other providers.
A stateless API or single-shot completion has system prompt overhead as the dominant bucket. The instructions are large relative to a small response, so you’re paying for the prompt on every call. The optimization target is trimming the system prompt, caching it, or moving boilerplate behavior into a fine-tune.
An agent or tool-calling workflow has the retry-and-tool bucket as the dominant one. Every reasoning step pays full input cost, and one bad tool response can trigger three to five extra calls. The optimization target is using a smaller model for routing decisions, hard step caps on agent loops, and structured output to cut JSON-parse retries.
A batch or async pipeline has visible completion tokens as the dominant bucket because there’s no conversation state. The optimization target is batch API discounts, model downsizing, and prompt compression — the levers everyone writes about actually apply here.
The strategies you read about online ARE real. Each one targets a different bucket. Pick the wrong one and you’ll burn the only resource more expensive than tokens — engineering time — without moving the bill.
That’s only true if you actually know which bucket is yours.
Measure first, optimize second
The instrumentation that lets you see your buckets fits in an afternoon. Log five things on every call: input tokens, output tokens, system-prompt tokens (separately, not bundled into input), retry count, and the feature or endpoint that triggered the call. Without that last field, you cannot allocate cost back to product surfaces — the entire exercise stops being useful.
Roll those logs into a daily report grouped by feature, not by model. The question “which feature is expensive?” leads to action. “Which model is expensive?” leads to a discussion.
Then compute the one number that matters: cost per unit of value. Cost per conversation. Cost per support ticket resolved. Cost per active user per day. If you cannot produce that number, you cannot tell whether the bill is reasonable or runaway. Token accounting for ai applications is just product accounting with extra steps.
The infrastructure for this is cheap. Every major provider returns token counts in the response body — log them next to your existing product analytics. A dedicated LLM observability tool helps once you scale, but it is not a prerequisite for getting started.
You’ll see the buckets. And then you’ll see something most articles never mention.
When optimizing is the wrong move
Engineering time is more expensive than your API bill, almost always. A senior engineer-week runs $5,000 to $10,000 fully loaded. A project that saves $200 a month pays itself back in three to four years — assuming nothing breaks and no one ever has to maintain it.
The 1% rule: if your LLM bill is under 1% of revenue, optimization is almost certainly the wrong project. Ship features. Grow the top line. Revisit when the bill becomes a real line item, not a rounding error.
Quality regressions are the silent cost nobody invoices you for. Prompt compression, smaller models, and aggressive caching all carry quality risk. That risk is expensive to detect, embarrassing in production, and rarely shows up in the savings calculation.
Optimization is worth it when three things are true at once: the bill is a meaningful percent of COGS, you have measurement in place, and one bucket is clearly dominant. Fix that one bucket. Then stop.
The Monday-morning version
Step one: instrument. Log input, output, system-prompt tokens, and retry count per feature. One afternoon.
Step two: compute cost per unit of value and compare it to revenue per unit.
Step three: identify which of the four buckets dominates yours.
Step four: only optimize if the bill matters AND one bucket is clearly the problem. Otherwise go ship something.
The gap between the pricing page and the invoice was never about price-per-token. It was about not knowing what you were buying. You know now.