Attaching human-provided labels, rankings, or corrections to data — the source of truth for supervised fine-tuning, preference data, and eval sets.
How it works
Labeling and annotation start with defining a clear rubric and guidelines that outline what each label means and how it should be applied. This clarity helps ensure consistency across different annotators. Once the guidelines are in place, items are routed to annotators, which can include in-house subject matter experts, external vendors like Scale or Surge, or even crowdsourced workers. As the annotators work, it’s crucial to measure inter-annotator agreement to assess how consistently different annotators apply the labels. When disagreements arise, a process for adjudication helps resolve these conflicts and refine the labeling process. This ensures that the labels reflect a consensus view.
Modern labeling practices often incorporate Model-assisted labeling, where a machine learning model pre-labels the data. This approach allows human annotators to focus their efforts on low-confidence cases or instances where there is disagreement among the labels. This method is part of an active learning strategy, where the model learns from the feedback provided by humans. Tools like Label Studio, Argilla, and Prodigy streamline this process, enabling efficient collaboration between humans and models. Using a large language model (LLM) as a judge can help label data at scale, but it’s essential to include human-calibrated spot checks to ensure accuracy and reliability.
For example, consider a dataset of customer reviews that need sentiment labeling. The team defines guidelines that specify what constitutes positive, negative, and neutral sentiments. Annotators then label the reviews based on these guidelines. A model might pre-label the reviews, identifying those with low confidence scores for human review. The combination of human insight and model efficiency helps create a high-quality labeled dataset that can be used for supervised fine-tuning of sentiment analysis models.
- Define clear rubric for labeling.
- Route items to appropriate annotators.
- Measure agreement to ensure quality.
- Adjudicate disagreements for consistency.
The software-engineering analogue
Think of labeling and annotation like writing assertions in your test suite. Just as you define what constitutes a passing test, you establish criteria for what a correct label is. However, the key difference lies in the nature of correctness. In software testing, the expected output is often binary—pass or fail. In labeling, the 'correct answer' can be subjective and varies among annotators. This means that quality becomes a distribution across annotators rather than a single expected value, leading to a more nuanced understanding of correctness.
Why not the obvious alternative
The idea of fully automated labeling might seem appealing because it promises speed and efficiency. However, relying solely on a model for labeling can be problematic. A model trained on existing data may simply replicate its own errors, effectively laundering inaccuracies into what is considered ground truth. This creates a feedback loop where mistakes propagate, undermining the integrity of the data used for training or evaluation.
Human anchoring is crucial in this process. By incorporating human judgment, you maintain a check on the model’s outputs, ensuring that the labels reflect a more accurate understanding of the data. This human oversight helps to correct biases and errors that the model might introduce, keeping the labeling process honest and reliable.
When to use which
When deciding on the approach to labeling, consider the stakes involved. For evaluation sets, safety-critical applications, and cases where the rubric is ambiguous, a human-heavy approach is essential. These scenarios demand the highest level of trust in the labels, as they directly impact model performance and safety. On the other hand, for high-volume, lower-stakes labeling tasks, model-assisted or active learning can be effective. This approach allows you to scale your labeling efforts while still maintaining some level of human oversight. Regardless of the method chosen, it’s vital to ensure that the golden evaluation set is always human-verified to uphold its integrity.
In production: when it breaks
Why it happens. Labeling errors can occur due to inconsistencies in the annotation guidelines or fatigue among annotators. When multiple annotators are involved, discrepancies in their understanding of the criteria can lead to significant variations in labeling quality.