Thursday, September 3rd: ChatGPT, Claude, and Grok all degraded within the same morning window. Pagerly's incident guide documented the pattern most teams experienced — a trickle of user complaints, then a spike in 5xx rates, then several minutes later someone finds a news article and the picture snaps into focus. That gap between "our thing is broken" and "the thing our thing depends on is broken" is where the real cost lives.
Most teams responded by adding fallback chains. That's the right instinct. But the way most fallback chains get built creates a different class of problem — one that's harder to detect because it looks like it's working.
Retry Is Not Fallback, and Stacking Both Is Expensive
The first mistake is treating retry and fallback as the same layer. They're not, and conflating them multiplies your problem.
BackendBytes makes this concrete: both the official Go SDKs for OpenAI and Anthropic already retry twice internally on connection errors, 408s, 409s, 429s, and 500s. If you add your own retry loop on top, a single failed request can generate six or more actual provider attempts before your fallback logic even fires. You've burned error budget, added latency, and potentially hit rate limits on a provider that was fine — all before the backup model got a chance.
The fix is simple in principle and easy to skip in practice: retry at exactly one layer. Let the circuit breaker see post-retry failures, not pre-retry noise.
The second mistake is building a circuit breaker that only watches HTTP status codes. BackendBytes documents three failure modes that arrive as HTTP 200: Anthropic's mid-stream overloaded_error, OpenAI's silent Fast-mode downgrade, and mid-stream refusals. A status-code-only circuit breaker misses all three. Your fallback chain never fires. Your users get degraded output and your dashboard stays green.
The Ordering Problem Nobody Talks About
Assuming your circuit breaker actually trips, the order of your fallback chain matters more than which models are in it.
The instinct is to fall back to whatever's cheapest or fastest. The operationally correct answer is to fall back to whatever preserves behavior. BackendBytes frames this as: same model via Bedrock or Vertex before a different model, because a different model changes your product. If your primary is Claude Sonnet and your fallback is GPT-4o, you haven't just switched providers — you've changed the reasoning patterns, the refusal thresholds, and the output format your downstream code expects. That's a product decision, not an infrastructure one, and making it at 2am while the status page is red is how you get data corruption.
Verel Systems puts numbers on what happens when this goes wrong: in complex agent environments without semantic retries, tool-use failure rates hover between 8–15%. A naive fallback that feeds raw errors back to the LLM without constraints often produces the same malformed request on retry — a runaway loop that silently drains API budget while appearing to function.
The DEV Community post on multi-model fallback patterns identifies the other sharp edge: retry won't fix quota exhaustion or model deprecation. If your key hits its monthly limit, exponential backoff just burns your error budget faster. If the provider sunsets the model version you pinned, every call fails with model_not_found and retry spins uselessly. These aren't transient failures — they require fallback, not patience.
Observability Is Where Fallback Chains Actually Break Down
Here's the part that doesn't get enough attention: fallback chains that work correctly are still invisible unless you instrument them properly.
OneUptime's tracing guide describes the accounting problem precisely. A request tries one provider twice, then succeeds with a fallback. The user sees one answer. The provider clients made three attempts. If every layer adds its own totals, one answer becomes six apparent model calls in your cost dashboard. You can't optimize what you can't count correctly.
The fix is structural: one root span for the user-visible request, with routing and each provider attempt represented underneath it. A fallback is another attempt with a different provider; a retry is another attempt under the same logical operation. Don't overwrite failed attempt status just because the fallback succeeded — those earlier failures are the signal you need for capacity planning and provider evaluation.
Ken Ashe's post on AI API postmortems frames the broader problem: when a vendor gives no useful postmortem, your team has to guess whether the issue was your prompt, their model, your retry logic, or a quota problem. The next sprint gets spent adding duct tape. Good fallback observability is what lets you answer that question yourself, without waiting for a status page update that may never come.
The September 3rd correlated outage was unusual in scale. The failure modes it exposed — retry stacking, behavior-blind fallback ordering, HTTP-200 errors that bypass circuit breakers, and cost double-counting — are not unusual at all. They're in most fallback chains right now, waiting for the next Thursday morning.
