A team builds a weather agent. They estimate 500 input tokens plus 200 output tokens per query, run the math against their provider's rate card, and budget $3.60 a month. In production, the same agent costs $25–40 a month — not because the model got more expensive, but because the spreadsheet never modeled the loop.
This is the pattern I keep seeing: teams treat token cost as a pricing problem and go looking for cheaper models, when the actual problem is architectural. The loop is billing you for every retry, every accumulated tool result, every framework-injected system prompt that never shows up in your cost estimate. You can swap to a cheaper model and still watch the bill climb, because you haven't touched the thing that's actually compounding.
Your Cost Estimate Is Missing Four Multipliers
The spreadsheet model assumes one prompt in, one response out. Production agentic workflows don't work that way. Here's what the estimate misses:
Context accumulation. Each tool result gets appended to the conversation context. An agent with five tools can accumulate 2,000+ tokens of context before generating a final answer — and every subsequent LLM call in that run pays for all of it again.
Retry amplification. Most agent frameworks implement automatic retry logic. If your model returns a malformed tool call 20% of the time, you're paying 1.2x expected token cost — and each retry includes the full accumulated context, not just the failed call.
Multi-step reasoning overhead. A ReAct-style agent that loops three times doesn't cost 3x the base. It costs roughly 5–7x, because each iteration sends the entire conversation history plus new reasoning.
Framework overhead. LangChain, LlamaIndex, and similar frameworks inject system prompts, format instructions, and parsing prompts that are invisible in your application code. These add 200–500 tokens per call that never appear in your estimate.
The fix for all four of these is the same: measure per-task cost, not per-month cost. You can't route intelligently or set meaningful budgets until you know which task types are actually expensive. One practitioner running an autonomous coding agent on Claude found that code review passes — the verification step — were his single largest cost line, at 34% of monthly spend. The work cost less than the checking. That's the kind of thing that only surfaces when you're logging token usage per task type, not staring at a monthly total.
Two Levers That Actually Move the Number
Once you have per-task visibility, two interventions tend to produce the most immediate results.
Model routing. Not every subtask needs your flagship model. Mechanical work — import reordering, variable renames, formatting — requires no reasoning. Routing those to a smaller model while reserving the top tier for architecture decisions and verification is the single biggest lever in most agentic systems. The same practitioner cut his overall bill by roughly 70% through a combination of routing and the fixes below — with no measurable drop in output quality.
The routing logic doesn't need to be sophisticated. A rule table keyed on task category and a couple of signals (estimated diff size, number of files touched) is enough to get most of the savings. Save the classifier complexity for later.
Prompt caching. Most production apps open every request with the same system prompt, tool definitions, and documentation context — often thousands of tokens — before they get to the user's actual input. Without caching, the model reprocesses that static prefix from scratch on every call, and you pay full price each time. With caching enabled, the provider stores the processed prefix and charges a steep discount on cache hits — currently 10% of the input rate on both Anthropic and OpenAI's current models, per DoiT's August 2026 cost analysis. For agents with long system prompts running at any real volume, this is free money you're leaving on the table if you haven't structured your prompts to put stable content first.
Budget Caps Are a Fail-Safe, Not a Strategy
The instinct when costs spike is to set a monthly budget cap on the API key. That's not cost control — it's a fail-safe that trips after the damage is done. A cap acts on a total that's only known after the fact. The run that blew the budget has already finished by the time the alarm fires.
Real control is run-scoped: a shared token budget enforced in real time across all the model calls a single agent run makes, with the ability to stop or redirect the run while it's still live. This is harder to instrument than a monthly cap, but it's the only mechanism that actually prevents a retry storm from becoming a billing incident rather than just explaining it afterward.
The broader principle: treat a cost spike as an incident with a root cause, not a line item to accept. The same discipline you'd apply to a latency regression — attribution, root cause, fix — applies here. Token spend is observable if you instrument it. Most teams just haven't.
The model isn't the problem. The loop is. Fix the loop first.
