Training a smaller "student" model to imitate a larger "teacher" — transferring capability into a model that's faster and cheaper to serve.
How it works
Distillation works by training a smaller 'student' model to mimic the output of a larger 'teacher' model. The classic approach involves matching the teacher's full output distribution, known as soft logits. This method captures a richer signal than just using hard labels, as it retains nuanced information about the teacher's predictions. In practice, especially with large language models (LLMs), this often involves Data distillation. For example, you might run a model like GPT-4 over a set of prompts, collecting its responses along with the reasoning behind those answers. This data becomes the synthetic dataset used to train the student model through a process called supervised fine-tuning (SFT). Models like DeepSeek-R1's distilled Qwen and Llama variants, as well as many others trained on GPT-4 outputs, follow this approach. However, it's crucial to watch the provider's terms of service regarding training on outputs.
The process of distillation essentially transfers knowledge from the teacher to the student. The student learns to approximate the teacher's behavior by analyzing the outputs generated from the prompts. This means that the student model doesn't just learn the correct answers; it also picks up on the patterns and reasoning that led to those answers. As a result, the student can perform well on similar tasks, even if it has fewer parameters. The trade-off here is that while the student gains efficiency and speed, it can also inherit the teacher's biases and blind spots, which can affect its performance in certain contexts.
In the context of LLMs, the distilled model is typically smaller and faster, making it more practical for deployment in real-time applications. For instance, a distilled model might only have 7 billion parameters compared to the teacher's 175 billion. This size reduction leads to significant improvements in latency and cost, allowing organizations to serve more requests without the delays or expenses associated with querying the larger model. However, the effectiveness of the distilled model heavily relies on the quality of the teacher's outputs and the diversity of the training data used during the distillation process.
- Run teacher model on prompts.
- Capture outputs and reasoning traces.
- Create synthetic dataset from outputs.
- Fine-tune student model on dataset.
- Evaluate student for performance.
The software-engineering analogue
Think of distillation like an optimizing compiler that takes a high-level programming language and produces a lean binary executable. The compiler analyzes the original code, optimizes it, and removes unnecessary parts while preserving the intended functionality. Similarly, the student model aims to replicate the behavior of the teacher model but in a more efficient form. However, the key difference here is that the student model approximates the teacher's behavior statistically rather than through direct code translation. This means the student can still carry over the teacher's blind spots, biases, and errors, which can lead to unexpected outcomes in certain situations.
Why not the obvious alternative
A newcomer might think the simplest solution is to just use the larger model directly in production. While this approach might seem straightforward, it comes with significant drawbacks. Running a large model like GPT-4 directly can be slow, costly, and subject to rate limits imposed by the service provider. Each query can take time to process, leading to delays in response times that are unacceptable for many applications.
In contrast, a distilled model, such as a 7B parameter version, can deliver 80-95% of the quality of the larger model on specific tasks while operating at a fraction of the cost and latency. This efficiency makes a distilled model a more viable option for high-volume applications where quick responses are crucial. The trade-off is that while you gain speed and cost savings, you may sacrifice some level of performance on more complex or nuanced tasks.
When to use which
Use distillation when you have a task that requires high volume and low latency, especially if you are working in a scoped domain where the model's performance can be optimized. For example, if you need to serve many requests quickly, a distilled model can handle that efficiently. However, if your application involves open-ended queries, low volume, or rapidly changing information, sticking with the larger frontier model is advisable. The distilled model may lag behind in such scenarios, as it may not adapt as quickly to new information or complex queries.
In production: when it breaks
Why it happens. The student model may not have learned the essential features from the teacher model due to insufficient training data or inadequate distillation techniques. This can occur if the temperature during distillation is set too low, causing the student to focus too narrowly on the teacher's predictions without capturing the underlying distribution of the data.