A tiered strategy so a model outage, rate limit, or bad output doesn't take the whole feature down — you fall back to a cheaper model, a cached answer, or a deterministic path.
How it works
The fallback and graceful degradation strategy operates through a tiered system of models and responses. You start with a primary model, like Claude Opus, which handles most requests. If this model encounters an issue—say, it goes down or returns an invalid output—the system automatically routes the request to a secondary model, such as Sonnet or GPT-4o-mini. These secondary models are designed to be less resource-intensive, allowing for quicker responses even if they sacrifice some quality. If both models fail, the system resorts to a last-resort method, which could be a deterministic template response or a cached answer that has been pre-approved for use. This tiered approach ensures that the user experience remains smooth and uninterrupted.
To implement this, gateways like LiteLLM, OpenRouter, or Portkey manage the routing of requests between these tiers. Each output from the models must be validated against a predefined schema or set of guardrails to ensure it meets quality standards. If the output fails this validation, the system drops down to the next tier. For example, in a classification task, if the model fails to classify correctly, the system might revert to a simple keyword rule. In a chat scenario, if the model cannot provide a suitable answer, it might respond with a generic message like, 'I can't answer that right now,' and offer the option for a human to take over the conversation. This cascading fallback mechanism keeps the service functional even in adverse conditions.
Consider a practical example: imagine a user queries a chatbot for medical advice. The primary model, Claude Opus, might provide a detailed response. If it fails, Sonnet could offer a simpler, yet still relevant answer. If both fail, the user receives a standard response that indicates a human will assist them. Each step down the tier ensures that the user is not left without any assistance, maintaining engagement and trust in the system.
- Start with primary model output.
- Validate output against schema.
- Fallback to secondary model if primary fails.
- Use cached answer or template as last resort.
- Decide based on user context and stakes.
The software-engineering analogue
This concept parallels traditional software engineering practices, such as serving a stale cache when the primary data source is unavailable. For instance, if your database goes down, you might return the last known good data to keep the application running. However, the key difference here is that your fallback tier often has Lower quality outputs, not just older data. This means you need to carefully define what constitutes a degraded but still acceptable response. For instance, serving stale data might be fine for a news site, but providing a low-quality medical diagnosis is unacceptable.
In this context, you must decide how much degradation is tolerable. Unlike a simple cache that might serve outdated information, your fallback models could potentially misinform users if they are not carefully vetted. Therefore, the challenge lies in balancing responsiveness with the integrity of the information being provided.
Why not the obvious alternative
A newcomer might think the simplest solution is to surface any errors directly to the user. However, this approach can severely damage user trust. Every hiccup in the provider's service becomes an immediate failure in your application, leading to frustration and a poor user experience. When a user sees an error message, it can lead them to question the reliability of your service, which can be detrimental in competitive markets.
Instead, a multi-tier fallback system allows you to maintain a responsive feature even when one provider fails. By routing requests through different models, you ensure that users receive some form of response rather than an outright error. This not only keeps the feature operational but also builds trust, as users feel they can rely on your service even during outages or issues with specific models.
When to use which
In user-facing applications where latency is critical, you should implement aggressive fallbacks to smaller models. This ensures that users receive timely responses, even if the quality is slightly lower. However, in high-stakes situations, such as medical or legal applications, you should avoid falling back to a weaker model that might generate incorrect information. Instead, it's better to defer to a human expert, ensuring that the response is accurate and trustworthy, even if it takes longer.
The decision on which fallback to use hinges on the specific context of the application. If the primary goal is to maintain user engagement and responsiveness, lean towards smaller models. If the priority is accuracy and correctness, particularly in sensitive domains, always opt for a human intervention rather than risking misinformation from a less capable model.
In production: when it breaks
Why it happens. The feature may be hitting a rate limit or experiencing model unavailability during peak hours, causing it to fail without a proper fallback mechanism in place. This can occur if the primary model is overloaded and the fallback strategies are not correctly configured or activated.