Timeouts bound how long a single call can block; circuit breakers trip open after repeated failures so you stop sending doomed requests and give the provider (and your queue) room to recover.
How it works
Timeouts set limits on how long a single call can block your application. In the context of large language models (LLMs), latency can be unpredictable and long-tailed; a single generation might take anywhere from 30 to 60 seconds. To manage this, you should implement Streaming with a time-to-first-token timeout. This approach allows you to receive initial tokens quickly, rather than waiting for the entire response, which keeps the user experience responsive. By using this method, you can ensure that if the model takes too long to respond, you can handle it gracefully without locking up resources.
Circuit breakers play a crucial role in managing failures. Tools like resilience4j or Polly can be used to implement a breaker that counts failures over a specific time window. Once the number of failures exceeds a predefined threshold, the breaker opens, which means it stops sending requests to the failing service and instead routes to a fallback mechanism. This cooldown period allows the upstream service to recover. After some time, the breaker enters a half-open state, where it allows a limited number of requests to test if the service is back up. This mechanism prevents overwhelming the service while it is still recovering.
You should cap the total token budget per request. This limits both the cost and the time spent on a single call. By doing this, you ensure that you don’t run into excessive charges or long wait times that could affect your application's performance. The combination of timeouts and circuit breakers creates a resilient system that can handle failures gracefully.
- Set time-to-first-token timeout
- Count failures over a defined window
- Open circuit on failure threshold
- Use fallback during cooldown period
- Probe for recovery with half-open state
The software-engineering analogue
In traditional software engineering, you might implement a circuit breaker pattern using tools like Hystrix or resilience4j to manage flaky microservices. These systems are designed to handle quick, synchronous calls that usually complete in milliseconds. However, with LLMs, the situation is different; their calls can take several seconds to complete. This means that the timeout thresholds need to be adjusted accordingly. For instance, setting a 2-second timeout would likely result in failing every healthy call, which is not acceptable. This shift in design requires you to rethink how you implement these patterns for longer, asynchronous operations.
Why not the obvious alternative
If you skip implementing a circuit breaker, you risk creating a retry storm during a provider brownout. This happens when your application continuously tries to send requests to a failing service, which can lead to a backlog of in-flight requests. This backlog can exhaust your worker pool and cause cascading failures throughout your system. Without a mechanism to control this behavior, your service could become unresponsive and unable to handle legitimate requests.
The fail-fast approach combined with a cooldown period protects your service from being overwhelmed. By stopping requests to the failing provider, you give your own system a chance to recover, rather than compounding the problem. This proactive strategy ensures that you maintain a level of service and responsiveness, even when external dependencies are struggling.
When to use which
You should implement circuit breakers on every external model dependency to protect your application from potential failures. For interactive user experiences, tighten the time-to-first-token timeout to ensure users receive quick feedback. In contrast, for batch or offline jobs, you can afford to relax the total time limit since responsiveness is less critical. This tailored approach allows you to optimize performance based on the specific use case.
In production: when it breaks
Why it happens. The timeouts are likely occurring because the external API is experiencing high latency or is down, leading to repeated failures. As requests stack up, the circuit breaker may not be tripping properly due to misconfigured thresholds, exacerbating the issue.