Most teams add a fallback model and call it done. Then they hit a 429 at 11am on a Tuesday, watch their retry logic turn one failed request into forty, and discover that "just add a backup model" was never actually the plan — it was the sketch of a plan.
The real problem isn't which model you fall back to. It's everything that happens in the 200 milliseconds before the fallback fires.
The 429 You're Getting Isn't the One You Think
Here's the thing that trips up most bulk pipelines: you can have RPM headroom and still get rate-limited. As a recent Medium breakdown explains, OpenAI's TPM limit counts input and output tokens combined inside a rolling 60-second window. One oversized prompt can consume most of a minute's token budget before you've made enough calls to worry about request rate. You're not making too many requests — you're sending too many tokens per request, which is a different fix entirely.
The multi-agent case is worse. OpenLegion's rate limiting breakdown puts the math plainly: OpenAI Tier 1 caps gpt-4o at 500 RPM across all callers on the same API key. A fleet of 10 agents making 1 call per second each generates 600 RPM before any of them finishes its first task. No individual agent is misbehaving. The aggregate behavior exceeds the limit, and application-layer self-policing can't fix it because each agent has no visibility into what the others are consuming. Only infrastructure-layer enforcement — a shared rate limiter sitting above all agents — can hold the aggregate.
Anthropic's Tier 1 limits are tighter still: 50 RPM and 1,000 RPD. Five agents making one call per minute will exhaust the daily request budget in hours. If you're doing agent development at Tier 1, you're already in this regime.
Naive Retries Make This Worse, Not Better
The instinct when you see a 429 is to retry. The problem is that failed requests still count against your limit — a rejected call still consumed the request slot it tried to use. Retrying immediately adds another failed attempt on top of an already-full window.
The thundering herd version of this is what actually kills production systems. TrueFoundry's failover guide describes the pattern: when 10 agents hit the rate limit simultaneously, they all receive 429s simultaneously. If each retries after a fixed 1-second delay, they all retry simultaneously — generating another burst, producing another wave of 429s, and so on indefinitely. A fixed-interval retry absorbs an occasional blip and collapses under a real backlog.
The fix is exponential backoff with jitter, and it's not optional. ML Journey's production guide covers the baseline: catch 429 errors, read the Retry-After header before deciding how long to wait, then add randomness to the wait time so concurrent clients don't all retry at the same moment. The Anthropic and OpenAI SDKs include built-in retry logic with exponential backoff — max_retries=5 on the client constructor gets you most of the way there for simple cases.
For bulk pipelines, the Medium piece is direct: firing requests concurrently without a limiter guarantees a 429 at any tier. asyncio.gather() across a few hundred items sends them effectively at once. No tier is high enough that a fully unthrottled burst won't outrun it.
The Fallback Chain Has Its Own Failure Mode
Once you've got backoff right, the fallback logic introduces a different class of bug. DeepInspect's fallback routing piece identifies the one that shows up in audit records rather than dashboards: a fallback that reaches an endpoint the caller was not authorized to reach. The request succeeds from the caller's perspective. The operational metrics look fine. The compliance record shows a request handled by a model the policy didn't permit.
The other failure mode is subtler: model context doesn't survive a provider switch cleanly. A DEV Community breakdown of failover patterns notes that OpenAI-style and Anthropic-style APIs use genuinely different request shapes. An "OpenAI-compatible" gateway smooths over the request/response format — it does not guarantee the fallback model behaves the same way given the same input. System prompts, conversation history, tool schemas, output format rules: none of these are guaranteed to survive a provider switch intact. You can validate the response format and still ship a response that's semantically wrong for the task.
The practical implication: fallback chains need output validation, not just error handling. Check cache first, check model health, validate the output, log which model actually handled the request. DeepInspect distinguishes four trigger types — provider outage, rate limit, model degradation, and policy denial — because the retry semantics differ per trigger. A rate-limit fallback is safe to retry identically. A model-degradation fallback (where the primary produced output the caller may have partially consumed) is not.
What to Actually Build
The queue is the feature. Before you add a third fallback model, make sure you have: a shared rate limiter above your agent fleet enforcing aggregate limits, exponential backoff with jitter reading Retry-After headers, output validation before responses reach downstream tool calls, and audit logging that records which model handled each request.
Watch your TPM consumption per request, not just your RPM. If you're running agents at Tier 1 limits, check your RPD math — daily request budgets run out faster than minute-level limits suggest. And read the x-ratelimit-remaining-* headers proactively rather than waiting for the 429 to tell you you're already over.
The backup model is the last line of defense. The queue is what keeps you from needing it every five minutes.
