Most teams treat rising LLM costs as a vendor negotiation waiting to happen. They watch the per-token price, wait for a cheaper model to drop, and assume the bill will eventually come down on its own. It won't — at least not for the reason they think.
Per-token prices have fallen dramatically. Enterprise generative AI spending still tripled to $37 billion in 2025, because usage grows faster than price drops. The bill is bigger than ever, and cheaper tokens are part of why: lower unit costs invite more usage, more features, more context stuffed into every call. The pricing problem solved itself. The architecture problem didn't.
When you pull apart where the money actually goes, the same three culprits show up in almost every production stack.
The Waste Is Already in Your Prompts
System prompts grow by accretion. A few extra instructions here, a couple of edge-case examples there, and six months later every single API call is carrying a paragraph of context that stopped earning its keep in March. Most development teams waste 40–60% of their token budgets on suboptimal implementations — and bloated system prompts are the single most common culprit.
The fix isn't to manually trim a few sentences. Automated prompt compression tools like Microsoft Research's LLMLingua use a small model to score and drop low-information tokens, reporting up to 20x compression with roughly 1.5% quality loss on standard benchmarks. For RAG workloads specifically, where you're routinely stuffing 4,000–8,000 tokens of retrieved context into every call, compression can cut that to 800–2,000 tokens while keeping answers accurate. That's not a marginal improvement — it's a structural change to your cost curve.
The second culprit is context management. Sending full chat histories and complete documents on every call when the model only needs a slice is the equivalent of emailing someone a 200-page manual every time you want to ask them one question. Output tokens are typically priced 5–10x higher than input tokens, which means unnecessary verbosity in responses compounds the problem on both ends of the call.
Caching Is the Highest-Leverage Fix Most Teams Skip
If your app resends the same system prompt, tool definitions, or reference documents on every call, you're paying full price to reprocess text the model has already seen. Anthropic discounts cached input tokens by 90%; OpenAI by 50%. A support bot with a 2,000-token policy document in every prompt, called 50,000 times a day, saves the equivalent of processing that document from scratch 45,000 times once caching is active.
The mechanics matter, though, and this is where teams leave performance on the table. Caching is prefix-based — the cached portion must appear at the start of your prompt and match byte-for-byte. Any change invalidates the cache for everything after the change point. If user-specific context appears before your stable instructions, the cache breaks on every session. The fix is to sort prompt content from most-stable to least-stable: provider docs, legal disclaimers, large tool definitions, and few-shot examples lead; user-specific and message-specific content trails.
There's a subtler trap when you combine caching with compression. Recent research on Anthropic's Sonnet API found that query-aware compression — which produces a different compressed prefix for every query — mechanically invalidates the prefix cache on every call. The paper proposes Cache-Aware Prompt Compression (CAPC), which pairs query-agnostic compression with explicit cache control, and reports mean savings of 49% over cache-only and 90% over vanilla prompting, at quality within 0.05 of the uncompressed baseline. The lesson: compression and caching interact, and optimizing them independently can make each one worse.
Route by Complexity Before You Do Anything Else
The fastest single win, if you haven't done it yet, is model routing. Not every request deserves your most expensive model. Intent classification, short summaries, and routine support replies don't need a frontier model — and the cheaper tiers have gotten dramatically more capable. Classify incoming tasks into complexity buckets, send the easy majority to a smaller model, and reserve the expensive one for work that actually requires it.
This pairs naturally with the infrastructure side of the problem. Running Llama 3.1 70B on a single H100 in single-stream mode costs roughly $0.60–$0.80 per million tokens; continuous batching at batch size 8 drops that to $0.15–$0.25 per million tokens — a 3–4x reduction with no additional hardware. The GPU didn't get faster. You stopped leaving it idle between requests.
The pattern across all of these levers is the same: the bill is a systems problem wearing a pricing costume. Prompt architecture, cache structure, routing logic, and batching strategy are engineering decisions that compound. None of them require a cheaper model. They require treating token spend as a first-class design constraint from the start — not something you optimize after the invoice arrives.
The teams getting this right aren't negotiating better API rates. They're auditing their system prompts monthly, structuring cache breakpoints deliberately, and routing traffic by complexity before a single token hits the expensive model. That's the playbook. The invoice is just the symptom.
