The message that ruins a Tuesday
"The chatbot just stopped responding." That's the whole Slack message. No stack trace, no context, sent by a client mid-demo to a room of people who were supposed to be impressed. We pulled up the logs and found a wall of 429s and 500s, all pointed at the same provider, all within the same eleven minutes. Nothing we wrote had broken. The model just wasn't answering.
We'd shipped that feature three months earlier under a deadline, on a single provider, with a try/catch around the API call that logged the error and returned a generic "something went wrong" message to the user. At the time, "add a fallback provider" was a ticket we punted because the happy path worked and the client wanted the feature live. It sat in the backlog labeled "nice to have." It was not, it turns out, a nice to have.
Why a try/catch doesn't save you
The instinct when this happens is to wrap the call in a retry loop and call it resilience. It isn't. Retrying against the same provider during an outage just means you fail slower and burn more of your rate limit doing it. The real problem shows up the moment you try to add a second provider properly: every one of them speaks a slightly different dialect. Anthropic, OpenAI, and Google format function-calling schemas differently, count tokens differently, and return errors under different shapes with different meanings for the same HTTP status code. A 429 from one provider means "you're over your quota for the month." From another it can mean "slow down, you're bursting." Code written for one doesn't degrade gracefully into the other — it just throws a new, different exception in production, usually at the worst possible moment, because outages and traffic spikes tend to arrive together.
What we actually had to build
Real failover meant three things we didn't have: a normalization layer so a prompt and its response look the same regardless of which provider answers it, a router that tracks provider health from live latency and error data instead of a static config, and a way to switch providers without a deploy, because the ten minutes it takes to ship a hotfix is exactly the ten minutes your client is staring at a broken demo. Once we'd built that internally for the third client in a row, it stopped being a one-off and became infrastructure. That's the honest origin story of ElseLane — it's the failover layer we kept rebuilding from scratch, now built once, with credits that work across providers so switching isn't a renegotiation, it's a routing decision.
The takeaway
Single-provider AI integrations work fine right up until the moment they matter most — and that's not a coincidence. Providers get overloaded during the same traffic spikes and news cycles that make your feature matter to a customer in the first place. If your AI feature has exactly one path to an answer, you don't have a feature, you have a demo that hasn't failed yet. Treat provider diversity as plumbing, not a roadmap item, and put it in before the Tuesday afternoon that makes you wish you had.