The incident is always the same. A customer asks "where's my delivery?" and gets a confident, detailed answer — about someone else's order. Two users asked semantically similar questions minutes apart; the cache scored a hit and served the first customer's response to the second. TrueFoundry's writeup on semantic caching documents exactly this failure mode: the cache was working as designed. It just had no concept that "where's my delivery?" is a question whose answer is never reusable across users.
This is the core trap with LLM response caching. The latency wins are real — cache hits can drop response times from 900ms to under 40ms. The cost wins are real — you skip inference entirely. But the failure modes are silent. No exception is thrown. Hit rate goes up. The user just gets the wrong answer and leaves.
Two Mechanisms, Two Risk Profiles
Before you can cache safely, you need to be clear about which kind of caching you're actually doing, because the operational risks are completely different.
Provider-side prefix caching — offered by Anthropic, OpenAI, and Google — lets you mark a stable prefix of your prompt (system instructions, retrieved context) so the provider keeps the computed attention state warm and bills repeated use of that prefix at a steep discount. You still make an API call. The model still runs. The output is freshly generated. As wolf-tech.io's production guide puts it, this is "almost pure upside" — you're not reusing an answer, only the work of reading the same preamble. The one real constraint: volatile content (timestamps, user messages, request IDs) must come after the cache breakpoint, never inside the prefix.
Application-side response caching — storing the model's full output and returning it directly on a later request — is where the latency and cost wins are largest, and where every correctness and security failure lives. The rest of this post is about doing that second kind without shooting yourself.
The Cache Key Is Not a Detail
Most stale hits and cross-tenant data leaks trace to the same root cause: a weak cache key. If you hash only the prompt body, every other factor that actually changes the correct answer — a model upgrade, a system prompt edit, a change in retrieved context, a different user's session — never reaches the key. The old answer survives and gets served confidently.
Claude Lab's field notes on cache failures articulate the rule clearly: everything that can affect the answer goes into the key; nothing that can't (request IDs, timestamps) ever does. In practice that means your cache fingerprint should include the model version, the full system prompt, the retrieved context, the user's tenant or session scope, and any feature flags that affect output. Miss any of these and you have a bug with a hit rate.
Cloudflare's AI Gateway documentation shows what exact-match caching keys on by default: provider, endpoint, model, auth header, and full request body. Any difference in the body produces a separate cache entry. That's conservative and safe for exact-match caching, but it also means your hit rate will be low for anything conversational. The moment you move toward semantic caching — embedding requests and matching on vector similarity — you've traded the safety of exact matching for higher hit rates and a new class of false-hit failures.
Where Response Caching Is Actually Safe
The honest test, per wolf-tech.io: if two requests produce the same key, is it always correct to give them the same answer? If you can't say yes without caveats, you don't have a cache — you have a bug with a hit rate.
Three categories pass the test cleanly:
- Deterministic transformations: classification, extraction, translation, and reformatting of a fixed input at temperature zero. The output is stable; caching it is safe.
- Static reference content: FAQ answers, documentation lookups, product descriptions that change on a known schedule. Cache these aggressively, but wire your invalidation to your content deployment pipeline — not to a TTL you set once and forgot.
- High-volume identical queries: if your analytics show that 30% of requests are literally the same question, exact-match caching on that query is free money with no correctness risk.
What fails badly: anything user-specific (order status, account data, personalized recommendations), anything time-sensitive (prices, availability, news), and anything where the intent behind similar-sounding questions differs. "Can I get a refund?" and "I was told no refund — why?" are semantically close. The correct answers are opposite.
The Metric You're Probably Not Watching
Amazon's Builders' Library piece on caching has a warning that applies directly here: services become "addicted" to their caches. Dependencies get scaled down assuming the cache absorbs load. Then traffic patterns shift, or the cache goes cold, and the downstream system can't handle the surge. For LLM caching specifically, the addiction risk is compounded by the silent failure modes — you won't see wrong answers in your error rate, because the cache isn't erroring.
The metric to instrument from day one: stale hit rate and false hit rate, not just overall hit rate. Redis's token optimization guide frames this as a token economics problem, but the operational framing matters more — a cache that's confidently wrong at 5% of hits is worse than one with a 20% lower hit rate that's always right.
Build the gauges before you build the cache. Otherwise you'll find out about the failures from your users, not your dashboards.
