The first time an LLM feature hits real traffic, the logs fill with 429s, and someone ships a retry loop that looks exactly like the one they wrote for the last REST API: catch the error, sleep, try again, maybe double the delay. It works in the demo. Then the bill shows up higher than expected, or a batch job runs four times slower than planned, or the app keeps hammering a dead account for an hour after the spending cap hit — because the retry loop was never the hard part. The hard part is knowing when not to retry at all.
Not every 429 means "slow down"
Treat HTTP 429 as a single error type and you'll build the wrong fix. One provider's error documentation splits the status into three distinct buckets: too many requests per minute, too many concurrent requests running at once, and a generic overload condition that ships with an explicit Retry-After header telling you exactly how long to wait — each bucket needs a different fix, and a fix for one does nothing for another. Halving your request rate accomplishes nothing if the real bottleneck is twenty workers firing in parallel; you have to read the error code before you touch a dial.
The split gets sharper on some APIs, where a 429 can also mean an account hit an enforced spending limit rather than a temporary rate ceiling — a condition no amount of backoff will ever clear. The same distinction shows up across providers more broadly: billing and quota errors require administrative intervention, while temporary throttling can be resolved by backing off and retrying. A naive loop treats both cases identically and burns through attempts, log lines, and queue time on a request that was never going to succeed. Classify first. Retry second.
Stop stacking retry layers on top of each other
Here's the failure mode I see most in postmortems: a team wraps their own retry loop around an SDK that already retries internally. Most official SDKs ship with default retry behavior covering connection errors, 408s, 409s, 429s, and 5xx responses with built-in exponential backoff. Add your own three-attempt wrapper on top of that and you've turned one call into nine requests under load — which is exactly the moment a brief slowdown turns into a self-inflicted stampede. Pick one layer to own retries. Let the SDK handle transient blips for interactive traffic, and raise the retry count per-call for background jobs instead of writing a parallel wrapper that fights it.
This matters because rate limiting and retrying are two separate mechanisms that only work when paired correctly: retries without rate limiting amplify traffic during an outage, while rate limiting without retries just exposes users to failures that were otherwise recoverable. You need backpressure on the way out — controlling how much traffic you send — and selective retry on the way back, not a single loop pretending to do both jobs.
Streaming turns a rate-limit problem into a slot-leak problem
Backpressure gets harder once you add streaming. A normal request holds a concurrency slot for a second or two; a streamed response holds that slot until the connection closes, which can mean tens of seconds for a long answer. In a chat app with even a handful of active conversations, a small concurrency ceiling disappears fast once every slot is tied up in an open stream. The two mistakes that make this worse are almost always client-side: abandoning a stream without closing the connection, which leaves the slot occupied indefinitely, and retrying by opening a second stream while the first is still alive, doubling slot usage at the exact moment you're short on capacity. If your traffic needs more concurrent long streams than your key allows, queue on your own side and show users a waiting state — don't let raw 429s leak through to the UI.
This is also the kind of regression that's easy to miss until it's already paging someone — the same reason teams are starting to lean on agents that watch release-to-release changes in retry behavior and latency automatically, rather than waiting for a human to notice the pattern in a dashboard after the fact, an idea Honeycomb has been pushing under the banner of proactive, not just reactive, production monitoring.
What to actually instrument
The practical takeaway isn't "add more retries." It's: read the response headers that tell you your real ceiling, classify every 429 before deciding what to do with it, pick exactly one layer in your stack that owns retry logic, and treat streamed concurrency as a resource you manage explicitly rather than assume away. None of this requires new infrastructure — it requires reading the error body instead of the status code, and resisting the urge to paper over a capacity problem with a bigger exponent on your backoff timer.
The next provider outage won't announce itself as "no capacity." It'll show up as a wave of 429s that look identical in your logs until you've built the classification to tell them apart. Build that before the wave hits, not during it.
