A team ships semantic caching. Cache hit rates climb to 30%. Costs drop. Everyone's happy — until a user asks "what's our refund policy?" and gets an answer that was accurate six months ago, before the policy changed. The cache TTL was set to 24 hours. The policy changed at 9am. The stale response served until 9am the next day.
That's not a caching failure. That's a TTL strategy that was designed for database query results being applied to LLM responses, which have completely different invalidation semantics.
Back in June, I wrote about how cache key design determines whether your LLM caching actually works. The failure mode there was structural — bad keys producing wrong hits. This is the next layer down: even with good keys, your TTL and invalidation logic is probably borrowed from a context where it doesn't belong.
Two Caching Layers, Two Different Problems
Before getting into TTL strategy, it's worth being precise about what you're actually caching, because the two main approaches have fundamentally different failure modes.
Prompt caching operates at the provider level — it discounts stable prefix tokens (system prompts, tool definitions, long context) when those tokens appear unchanged at the start of a request. OpenAI offers roughly 50% cost reduction on cache hits; Anthropic's pricing drops from $3.00/M to $0.30/M on cache hits for Claude Sonnet 4.6, a 90% reduction. The TTL question here is mostly managed for you: the provider controls when the cached KV state expires. Your job is keeping the prefix stable so hits actually occur. Switching models, changing effort levels, or connecting a new MCP server all invalidate the cache — and that first post-invalidation turn pays full price.
Semantic caching is where TTL strategy gets genuinely hard. This operates at the application layer: you embed incoming queries, find semantically similar past queries above a similarity threshold, and return the stored response without hitting the model at all. Production deployments hit 20-45% of traffic without touching the model, which sounds great until you think carefully about what "semantically similar" means for time-sensitive content.
The TTL Mismatch Problem
Time-to-live made sense when you were caching a database row. The row changes when someone writes to the database. You can set a TTL that's shorter than your typical write frequency, accept some staleness, and move on. The invalidation model is simple: data changes, cache expires, fresh data loads.
LLM response caching doesn't work this way. The "data" being cached is a generated response to a query — and that response can become wrong not because the underlying record changed, but because the world changed. Your refund policy changed. Your pricing changed. A product was discontinued. None of those events trigger a cache write that you can hook invalidation logic onto.
This is the core mismatch: TTL-based invalidation assumes you know when data changes. For LLM responses, you often don't.
The practical consequence is that teams end up choosing between two bad options. Short TTLs (minutes to hours) keep responses fresh but destroy your hit rate — you're paying for the cache infrastructure without capturing the cost savings. Long TTLs (days to weeks) maximize hit rates but create a system that confidently serves outdated information.
What Actually Works in Production
The teams getting this right are treating cache invalidation as a first-class operational concern, not an afterthought to TTL configuration.
Content-type-aware TTL tiers. Not all LLM responses have the same staleness risk. A response explaining how OAuth2 works can safely cache for days. A response about current pricing should probably never cache at all, or cache for minutes. Cache key design should encode content type so you can apply different TTL policies by namespace — pricing:* keys get a 5-minute TTL, docs:* keys get 24 hours.
Event-driven invalidation alongside TTL. TTL is a fallback, not the primary mechanism. When your policy document updates, your pricing table changes, or your product catalog shifts, those events should trigger explicit cache invalidation for the affected key namespaces. This requires your caching layer to be aware of your data model — more engineering work upfront, but it's the only way to avoid the "stale answer served confidently" failure mode.
Similarity threshold as a staleness proxy. One underused lever: tighten your semantic similarity threshold for queries that touch volatile content. A 0.95 cosine similarity requirement for pricing questions means you're only returning cached responses for nearly identical queries — reducing the blast radius when a cached response goes stale.
Monitor your hit rate by content type, not just overall. An aggregate 40% cache hit rate can hide a 90% hit rate on safe content and a 5% hit rate on volatile content. If you're not segmenting hit rate metrics by the TTL tier or content namespace, you don't actually know whether your caching strategy is working.
One more thing worth flagging: research published this year formalizes what practitioners have been discovering operationally — semantic caching's fuzzy-match approach creates an inherent tension between hit rate and collision resistance. The same property that makes semantic caching effective (returning responses for similar queries, not just identical ones) is what makes it possible to get a wrong cached response for a query that's close but not equivalent. That's not just a security concern; it's an accuracy concern that your TTL strategy can't fix.
The right mental model: TTL is your last line of defense against stale responses, not your first. Build the invalidation logic that should catch changes before they age out naturally. Then set TTLs as a backstop for the changes you didn't anticipate.
