The support assistant analytics are damning in the best way. Out of roughly 400,000 distinct user queries, the top 500 phrasings account for 30% of volume, and another 25% clusters into near-duplicate queries differing only in wording — each one burning ~2,500 input tokens and ~300 output tokens on a response functionally identical to what the last thousand askers received. The case for caching writes itself.
So teams add a semantic cache, watch hit rates climb, and declare victory. Then, three months later, someone notices the cache is confidently returning wrong answers to questions it almost-but-not-quite understood. The hit rate was real. The correctness wasn't.
I've written before about cache key design and why TTL is the wrong abstraction for LLM response freshness. What I haven't addressed directly is the three-way tension that makes LLM caching categorically different from caching a REST endpoint: freshness, cost, and consistency don't optimize in the same direction, and the layer you choose determines which one you sacrifice.
The Four Layers Aren't Interchangeable
There are four distinct caching layers in an AI application: exact response, semantic, provider prefix, and derived artifacts. Most teams treat them as a single dial labeled "cache more." They're not. Each has a different key, a different correctness hazard, and a different payoff profile.
Exact-match caching is the safe one. The key is a hash of everything that affects the answer — model ID (not alias), prompt template version, temperature, tool schema hash, and rendered user content. The two omissions that cause real bugs are task version and a concrete model ID: without the first, editing a prompt has no effect on anything already cached; without the second, a model alias that silently points at a new revision serves answers generated by the old one indefinitely. Get the key right and invalidation mostly handles itself — a change to any input produces a different key. Where this layer shines is machine-driven workloads: batch jobs, page renders shared across many users, CI test runs. User chat, where exact repeats are rare, barely benefits.
Semantic caching is where teams get into trouble. It's the only caching layer that can return a wrong answer — a cached response matched to a similar-but-not-identical query. The similarity threshold is a correctness dial masquerading as a performance dial. Tighten it and you're back to near-exact matching. Loosen it and you're serving confident wrong answers to users whose queries were close but not equivalent. A cache hit on the wrong query returns a confidently-stated wrong answer, which is worse than the slow-but-correct baseline. That's the trade-off in plain language.
Provider prefix caching is the underused one. You don't store anything — you shape your prompt so the provider can reuse computed KV state for the stable prefix. The invariant across Anthropic, OpenAI, Gemini, vLLM, and SGLang is the same: change one early token and everything after it stops matching. The six recurring breakers are timestamps, tool-list changes, nondeterministic JSON, memory or RAG injected into the cached region, invisible prefix mutations, and compaction. One cross-provider evaluation reports 41–80% API cost cuts from prefix caching under stated test conditions — but only if you treat the stable prefix as an engineering contract, not a hint.
Freshness Is an Invalidation Strategy, Not a TTL
The freshness problem is where most production caches quietly rot. Every cached response needs an expiry rule, and time-based TTL alone isn't sufficient. The right model combines TTL with event-based invalidation: when the knowledge base is re-ingested, invalidate anything that cited changed chunks. TTL bounds staleness in time; event-based invalidation reacts to content changes. Running only one of these means either serving stale answers until a timer fires or never expiring entries that are still technically fresh but factually wrong.
Braintrust's gateway caching defaults to one week TTL with per-request override, which is a reasonable starting point for deterministic prompts and CI runs. For user-facing features touching live data, a week is an eternity. The TTL should be sized to how long a stale answer is acceptable for that specific request type — and that answer varies by feature, not by system.
One more thing: cache failures for a much shorter period than successes, or don't cache them at all. A cached 429 that lives for an hour is a self-inflicted outage. This sounds obvious until you're debugging why a provider rate limit is cascading into user-facing errors six hours after the original incident.
The Monitoring Gap
Cache hit rate is a vanity metric without quality signal attached to it. A high cache-hit rate can still include responses that are incorrect, stale, or matched to the wrong request — and you won't know which without evaluation and tracing connected to cache performance. The x-bt-cached: HIT header tells you a response came from cache. It tells you nothing about whether that response was still correct.
The operational discipline here is the same as any other probabilistic system: instrument the cache layer with quality checks, not just hit/miss counters. Sample cached responses through your evaluation pipeline. Track staleness by feature area, not globally. Flag semantic cache hits that fall below a confidence threshold for human review or fallback to live generation.
The teams who've made caching work in production aren't the ones who found the right similarity threshold. They're the ones who stopped treating cache hit rate as the goal and started treating answer quality as the constraint.
