Making it safe to re-issue an LLM request after a timeout or 5xx without double-charging, double-sending, or duplicating a tool action the model already triggered.
How it works
To ensure safety when re-issuing requests to a large language model (LLM) after a timeout or a server error, we wrap our calls in an exponential backoff strategy with jitter. This means that if we encounter a 429 status code, indicating too many requests, we respect the Retry-After header to determine when to try again. Each request should also include an Idempotency key, which is a unique identifier for that specific request. This key helps us track any downstream effects that the LLM might trigger, such as database updates or external API calls, preventing duplicate actions from occurring.
For tool calls, we need to ensure that the tools themselves are designed to be idempotent. This means using operations like upsert based on a natural key rather than performing a simple insert. When the model re-emits the same call due to a partial failure, we want the operation to either create a new entry or update an existing one without causing duplicates. We should cache the results of completed requests based on a hash of the request. This way, if a generation is retried, it can return the previous answer without incurring extra costs.
As an example, consider a scenario where a user requests a summary of a document, and the LLM encounters a temporary error. By implementing exponential backoff, we wait a bit before retrying, and the idempotency key ensures that if the request is repeated, it won't trigger another summary generation or other side effects. If the request is identical, the cached result can be returned immediately, saving both time and resources.
- Attach idempotency key to requests.
- Implement exponential backoff with jitter.
- Cache results by request hash.
- Use upsert for side-effecting tools.
- Avoid naive retries on irreversible actions.
The software-engineering analogue
This concept parallels the discipline found in idempotent HTTP PUT requests and at-least-once message queues. In both cases, the goal is to ensure that repeating an operation does not lead to unintended consequences, like duplicate entries or actions. However, the key difference in the context of LLMs is that the 'message' being processed is nondeterministic. This means that when we retry a generation, we might receive different text outputs each time, depending on the model's state and randomness.
Because of this nondeterminism, the deduplication key we use must be based on the input rather than the output. If we were to use the output as the key, we could end up with multiple entries for the same input, which defeats the purpose of ensuring idempotency. Thus, we need to carefully design our system to account for this unique challenge.
Why not the obvious alternative
A newcomer might think that implementing naive retry loops is a sufficient solution for handling errors when making requests to the LLM. However, this approach can lead to significant issues, especially when the retried request inadvertently re-executes actions like send_email or charge_card. Without proper safeguards in place, these actions could be duplicated, leading to unintended consequences like multiple charges or spam emails.
The root cause of this problem lies in the assumption that a simple retry on error is safe. In reality, the only way to ensure that we don't perform the same action multiple times is to use idempotency keys. These keys provide a mechanism to track requests and their effects, ensuring that even if a request is retried, only one action is taken, thus maintaining data integrity and user trust.
When to use which
When dealing with pure text generation tasks, you can retry requests freely, as the output is not tied to any side effects. In these cases, caching results by input hash is sufficient to optimize performance and reduce costs. However, when working with agents that perform side-effecting actions, such as making API calls or modifying databases, it becomes crucial to implement idempotency keys. In such scenarios, you should also consider gating irreversible actions behind a confirmation step rather than allowing them to auto-retry, which could lead to unintended consequences.
In production: when it breaks
Why it happens. The system may not be properly tracking the state of requests or responses, leading to multiple executions of the same action when a retry occurs. If the request is not idempotent, the same input can trigger multiple outputs, resulting in duplicate actions.
- Check the request logging to see if the initial request was recorded before the error.
- Verify if the retry mechanism is using the same unique identifier for the original request.
- Inspect the application logic to ensure that duplicate processing is prevented for retries.