Classifying and routing the distinct failure modes of an LLM call: transport errors, rate limits, content filters, context-length overflows, refusals, and malformed output that returns HTTP 200 but is semantically wrong.
How it works
When a model call fails, the system needs to branch based on the type of error encountered. For example, if the API returns a 429 status code indicating too many requests, the system should implement a backoff strategy before retrying the call. This prevents overwhelming the service and allows it to recover. If the error is 400 with a message like context_length_exceeded, the system should truncate or summarize the input to fit within the limits and then re-issue the request. Similarly, if a content filter blocks the output, the system must either fallback to a human operator or handle the situation differently rather than simply retrying the same request, which would likely lead to the same failure.
In cases where the output is malformed but the response returns a 200 OK status, the system must actively validate the content. If the output does not conform to the expected schema, the system can re-ask the model with the validation error appended to provide context for the correction. Tools like Pydantic or Instructor can help validate the structure automatically and can be configured to retry the request if a parse failure occurs. It's crucial to log the raw response and the finish reason for every failure, so that a truncated response is not misinterpreted as a complete answer.
- Check HTTP status code first.
- Use backoff strategy for 429 errors.
- Fallback for content filter issues.
- Re-issue requests for schema errors.
- Log all failures for analysis.
The software-engineering analogue
In software engineering, you often deal with typed exception hierarchies where you can catch distinct types of exceptions and handle them accordingly. This allows for clean and precise error management. However, the key difference in the context of LLM calls is that the most problematic error is a 200 OK response that contains incorrect content. Unlike traditional exceptions that throw errors, this scenario requires you to actively validate the output instead of relying on the status code, which can lead to misleading assumptions about the success of the operation.
Why not the obvious alternative
A naive approach might involve a single try/except: retry block that simply retries the request whenever an error occurs. This method fails to distinguish between errors that can be retried and those that cannot, such as a moderation block or a schema mismatch. As a result, it risks entering a loop where the same unretriable error keeps occurring, and it may also silently ship truncated answers without addressing the underlying issues.
The root cause of this failure is the lack of per-class error handling. Without structured error management, the system cannot differentiate between recoverable and unrecoverable errors. This leads to a situation where the system either keeps retrying the same failed request or returns incomplete information, which can severely degrade user experience and trust in the system.
When to use which
For structured-output pipelines, it's essential to implement schema validation along with a repair loop that includes a retry cap to avoid infinite retries. This approach ensures that the system can handle errors effectively while maintaining the integrity of the output. In contrast, for free-form chat applications, the focus should be on managing finish reasons and handling refusals appropriately, as these scenarios often involve more variability and less predictable outputs.
In production: when it breaks
Why it happens. This issue often arises from the model's internal handling of ambiguous or unexpected input, leading to malformed outputs. The model may return a response that technically adheres to the API contract but fails to meet the semantic expectations due to misinterpretation of the context or input data.