Adapting an embedding model so vectors for your queries and documents land close when they're semantically relevant in your domain — the quality lever most RAG systems ignore.
How it works
To fine-tune an embedding model, you start by training on triplets of data consisting of a query, a positive example, and a negative example. The goal is to adjust the model so that the vectors representing semantically relevant items are closer together in the vector space. You use a contrastive loss function, like InfoNCE, which pulls the positive examples closer to the query vector while pushing the negative examples further away. This process is crucial because it ensures that the model learns the specific relationships and nuances of your domain, rather than relying on generic patterns.
Positives can come from real user interactions, like click data, or from generating query-passage pairs with a large language model (LLM). However, the negative examples are particularly important. These should be hard negatives—passages that seem plausible but are incorrect. They help the model understand what not to retrieve. Tools like sentence-transformers or services from providers like Cohere and Voyage can assist in this fine-tuning process, allowing you to create embeddings that are tailored to your specific needs.
The result of this fine-tuning is a set of embeddings that reflect the unique vocabulary and context of your domain. For instance, if your domain involves medical terminology, the model will learn to differentiate between terms like 'MI' as myocardial infarction and other meanings. Without this fine-tuning, the embeddings might not capture the relevant semantics, leading to poor retrieval performance.
- Collect (query, positive, negative) triplets.
- Apply contrastive loss to adjust embeddings.
- Use domain-specific vocabulary for training.
- Consider reranking for initial improvements.
- Evaluate retrieval precision and recall.
The software-engineering analogue
Think of this process like tuning a search relevance function based on your own query logs. You analyze which results users find most useful and adjust your ranking algorithms accordingly. However, the key difference here is that you are not just tweaking rules; you are reshaping a geometric vector space. This means that the relationships between vectors are altered in a way that directly impacts how similar or different they are perceived to be, which is a more complex and nuanced task.
Why not the obvious alternative
Many newcomers might first consider using off-the-shelf embeddings, which are trained on generic datasets like web text. While this approach seems straightforward, it often fails in specialized domains. For instance, if your domain frequently uses terms like 'MI', the model might confuse it with Michigan or other meanings, leading to irrelevant retrievals. This mismatch occurs because the model lacks the contextual understanding needed to differentiate between these terms.
Even if you try to prompt a large language model to generate better results, it won't fix the underlying issue of retrieval quality. If the model retrieves the wrong chunk based on the initial embedding, any subsequent generation will be based on flawed information. This garbage-in, garbage-out problem highlights why relying solely on generic embeddings can be detrimental to your application.
When to use which
You should consider fine-tuning embeddings when you notice that retrieval recall is the bottleneck in your retrieval-augmented generation (RAG) system. If your domain has specific vocabulary or relevance criteria that off-the-shelf models fail to meet, fine-tuning becomes essential. However, a more cost-effective first step might be to implement a fine-tuned or off-the-shelf reranker on top of stock embeddings. This approach can often yield about 80% of the performance improvement for significantly less effort and resource investment.
In production: when it breaks
Why it happens. The drop in performance often occurs due to domain misalignment, where the embeddings are not adequately adapted to the specific terminology and context of the domain. This can lead to vectors for relevant queries and documents not being positioned close enough in the embedding space, resulting in poor retrieval accuracy.