The April incidents should have been a wake-up call. A ten-hour Claude outage on April 6 and a major OpenAI platform outage on April 20 — both multi-hour, both affecting production applications with no warning. If your response to either was "we added a retry with exponential backoff," you didn't build a fallback chain. You built a delay before the same failure.
There's a meaningful difference between a retry policy and a fallback architecture, and most teams conflate them until the bill becomes the incident.
Retries and Fallbacks Are Not the Same Thing
A retry answers: "Can I try again?" A fallback architecture answers: when to retry, when to switch models, when to degrade the feature, and when to stop. That's a control plane problem, not a catch block problem.
The failure modes that actually hurt production systems aren't the obvious ones. Provider downtime is easy to understand and easy to route around. The harder failures look normal from the outside: a request returns 200 but the JSON is invalid, the model answers but ignores the required tool call, a fallback model works but can't follow the same structured output contract, a region returns intermittent 429s, a background worker retries a long prompt until the bill becomes the incident.
That last one is the one that pages finance before it pages engineering.
One team running chaos testing on their CI fleet discovered exactly this: their "retry on 5xx" logic was happily burning $80/hr re-sending the same 200k-token context to Anthropic during a brownout. The retry loop was working as designed. The design was wrong. The gateway they put in front of the problem fixed the routing. It didn't fix the application-level retry behavior — that required a separate policy layer they had to write themselves.
This is the honest version of what gateway tooling solves and what it doesn't.
Where the Gateway Layer Actually Helps
An LLM gateway centralizes request handling — authentication, routing, fallback, observability, and cost tracking — through a unified API. The moment you're managing multiple providers, failover logic, or team-level cost controls, you're already building gateway infrastructure whether you call it that or not.
The operational win is real. When you push fallback logic into a gateway tier rather than application code, three failure modes stop recurring: each application reimplementing retry logic inconsistently, provider-specific error handling scattered across codebases, and no single place to observe cross-provider behavior. The CI team above moved to a gateway sidecar on each agent host and found that when toxiproxy killed OpenAI, builds kept moving by routing to Anthropic, with build time bumping maybe 20% — nobody paged. That's the right outcome.
But the gateway is infrastructure, not policy. It routes around provider failures cleanly. It does not know whether your fallback model can honor your structured output schema. It does not know that your summarization step costs ten times more than your classification step. It does not know when degrading gracefully is better than retrying at all.
The Policy Layer Nobody Builds Until It's Too Late
Production AI applications are rarely a single model call. A production app may call a fast model for classification, a stronger model for reasoning, an embedding model, a reranker, a tool-calling agent, a vision model, and a safety classifier — each call can fail differently, and a retry that helps one layer can make another layer more expensive or less reliable.
The policy questions you need answers to before an incident, not during one:
- Schema compatibility. Does your fallback model produce the same structured output your downstream code expects? A fallback that returns valid JSON in a different shape is a silent failure, not a recovery.
- Cost asymmetry. If your primary is a cheap fast model and your fallback is a capable expensive one, unlimited retries against the fallback will show up in your bill before they show up in your error logs.
- Degradation thresholds. Some features should stop rather than degrade. A code review bot returning a lower-quality response is fine. A compliance check returning a lower-quality response is not.
Agents aren't entities — they're systems with interacting pieces that have different responsibilities, failure modes, and levels of observability. Fallback chains fail for the same reason agents fail: the architecture was decided by the goal, not by the engineering requirements.
What to Actually Build
The practical starting point: treat your fallback chain as a decision tree with explicit exit conditions, not a list of providers to try in sequence. Each node in the chain should answer three questions — what error class triggers this transition, what constraints does the next model need to satisfy, and what's the cost ceiling before we stop retrying entirely.
Run a game day against it before you need it. The CI team found their retry-cost problem during a scheduled failure injection, not during a real incident. That's the correct order of discovery.
The gateway handles the routing. You still have to own the policy.
