Systematic measurement of LLM-system quality. Offline evals run against a fixed dataset in CI/dev; online evals measure real production traffic with live signals.
How it works
Offline evaluations start with assembling a labeled and curated test set that represents the capabilities we want to measure in the LLM system. Once we have this dataset, we run the system against it, generating outputs for each input. We then score these outputs using various methods, such as exact-match comparisons, regular expressions, or even employing another LLM as a judge to assess the quality. After scoring, we aggregate the results to determine pass rates for each capability, giving us a clear picture of how well the model performs on the defined tasks. This structured approach helps us identify strengths and weaknesses in a controlled environment.
Online evaluations operate differently by sampling live traffic from real users interacting with the system. We score these interactions using automated judges that can analyze the responses based on user feedback, such as thumbs up/down, edits, or retries. We track product metrics like task completion rates and deflection rates, often in conjunction with A/B tests to measure the impact of changes. This method allows us to capture the model's performance in the wild, providing insights into how it behaves under actual usage conditions and identifying areas for improvement based on real-world data.
Both offline and online evaluations are crucial for a comprehensive understanding of an LLM's performance. Offline evaluations provide a baseline and ensure that changes do not regress the model's capabilities, while online evaluations help us understand how the model adapts to real-world scenarios and user needs. By combining these two approaches, we can create a feedback loop that continuously improves the model's performance.
- Assemble curated test set for offline evals.
- Run system and score outputs with graders.
- Sample live traffic for online evals.
- Collect user feedback and product metrics.
- Feed online failures back to offline tests.
The software-engineering analogue
In software engineering, offline evaluations are similar to unit and integration tests, where we verify that individual components and their interactions work as expected. These tests run against a fixed set of inputs and produce deterministic outputs, allowing us to easily identify failures. Online evaluations parallel production monitoring and analytics, where we track how the system performs in real-time with actual user data. However, the key difference here is that LLM outputs are non-deterministic and open-ended. This means that instead of a simple pass/fail result, we get a distribution of scores that reflect the variability in outputs, making it more complex to assess quality.
Why not the obvious alternative
Many newcomers might think that simply 'eyeballing a few examples' or doing vibe checks on the outputs is sufficient for evaluation. However, this approach quickly becomes unscalable as the number of examples grows, and it often misses subtle regressions that could affect performance. Without systematic evaluation, it’s challenging to determine whether a change in prompts or model parameters has genuinely improved the output or inadvertently degraded it.
Evals transform these subjective impressions into quantifiable metrics. By establishing a structured evaluation process, we can track performance over time, identify specific areas for improvement, and make informed decisions about model changes. This systematic approach allows us to move from guesswork to deliberate adjustments, ensuring that we maintain and enhance the quality of our LLM systems.
When to use which
When deciding between offline and online evaluations, consider the stage of development and the specific goals of your testing. Use offline evaluations to guard against regressions during development and continuous integration. This method is ideal for validating changes against a fixed dataset that represents known capabilities. In contrast, online evaluations are essential for catching real-world drift and identifying edge cases that the offline dataset might miss. Mature teams typically employ both strategies, using insights from online evaluations to refine and expand their offline test sets.
In production: when it breaks
Why it happens. This drop may occur due to a misalignment between the model's performance during offline evaluations and real-world user interactions. The model could have overfitted to the offline dataset, failing to generalize well to the diverse language and context found in live user queries.
- Check user satisfaction scores over time to identify when the decline began.
- Analyze the model's performance on recent offline evals to see if there is a significant discrepancy.
- Review logs for common themes in negative feedback to pinpoint specific failure modes.
What it delivers
The primary benefit of implementing both offline and online evaluations is that they provide the confidence to iterate on the model. With a systematic evaluation framework, you can substantiate that a change is genuinely an improvement rather than relying on intuition or hope. This evidence-based approach allows teams to make informed decisions, ensuring that each iteration of the model enhances its performance and user satisfaction.