Every gateway vendor's homepage says the same thing: automatic failover. Anthropic goes down, OpenAI picks up the slack, nobody notices. That's the pitch. It's also, according to a documentation audit of five managed gateways, not remotely what "automatic" means in practice — and the gaps are exactly the kind that show up as a 2am page instead of a line item in a postmortem.
I've written before about the local-vs-API routing decision and about fallback chains that behave like retry loops in a trench coat. Multi-provider load balancing across OpenAI, Anthropic, and the rest is the next layer up: you've accepted that no single provider is reliable enough on its own, so now you're routing across vendors. The problem is that "routing across vendors" is not one behavior. It's a pile of vendor-specific defaults that only reveal themselves when a provider actually goes down.
"Automatic" Has Five Different Definitions
A documentation comparison of Concentrate, OpenRouter, Vercel AI Gateway, Portkey, and Cloudflare AI Gateway, checked against each vendor's public docs at the end of October, found that only two of the five reroute a failed request with zero configuration — Concentrate and OpenRouter. Vercel reroutes across providers by default but needs an explicit model list to try alternatives. Portkey and Cloudflare do nothing until you hand-build a fallback policy — a strategy.mode: fallback config for one, a dynamic route for the other.
That split matters more than it looks. If your team picked a gateway because the marketing page said "automatic failover" and never read past it, you may have shipped a system where a provider outage just... fails. No retry, no reroute, nothing — because the gateway was waiting for a config block nobody wrote. The same audit notes that only two of the five vendors publish their retry counts and backoff curves, and none of them publish a default timeout for managed inference. You are, in effect, trusting a black box to decide when "slow" becomes "dead," and most of them won't tell you the threshold.
Retries Are a Tracing Problem, Not a Config Problem
This is where the self-hosted alternative to a managed gateway earns its complexity. MLflow's guide to LLM API error handling makes the point that getting retries right requires four separate decisions — which errors you retry, how long you wait, when you give up and switch models, and how you log every attempt so you can debug it afterward. Their framing that "a retry you cannot see is a retry you cannot tune" is the whole argument against trusting a managed gateway's opaque failover logic. If your observability stack can't tell you that request 4,812 silently retried three times against OpenAI before falling over to Anthropic, you don't have reliability — you have a slower failure with extra steps.
The practitioner response to this is predictable: people are building their own thin routing layers instead of adopting a managed one. A latency-based load balancer shared on Hacker News routes across OpenAI, Azure OpenAI, Vertex AI, and Anthropic by continuously sending a share of traffic to slower providers as a live probe — so a recovered provider wins back volume without a human flipping a switch. It's a single static binary, explicitly positioned against the "large surface area and security issues" of heavier gateways. That's not a universal fix, but it's a sane reaction to a market where the big managed options disagree on whether failover needs config at all.
The Routing Decision That Actually Saves Money Is a Different One
It's worth separating two problems that get conflated under "multi-provider routing." Failover routing — what happens when a provider errors or times out — is a reliability problem. Model routing — sending easy questions to cheap models and hard ones to frontier models — is a cost problem, and a good one. Elastic's instrumentation of a LangChain shopping assistant found that three model calls consumed 92% of a ten-second request, and swapping the comparison step to a cheaper model cut spend 76–83% per question — while making latency worse on four of five test questions. That's the trade nobody puts on the gateway homepage: optimizing provider cost and optimizing provider reliability pull in opposite directions, and a gateway that only advertises "automatic failover" is silent on which one it's actually tuned for.
What to Actually Check Before You Trust the Failover Button
Don't take a vendor's "automatic failover" claim at face value. Pull up the actual docs for whichever gateway you're running — or evaluating — and find the three numbers that matter: what triggers a failover, how many retries happen before it gives up, and what the default timeout is before a request gets marked dead. If any of those three is unpublished, assume the worst case and set your own policy explicitly rather than inheriting the vendor's defaults. The next provider outage will find that gap for you if you don't find it first.
