Combining a keyword index (BM25) with vector similarity so you catch both exact-term matches and paraphrased meaning in one result set.
How it works
Hybrid search combines two powerful indexing techniques: keyword-based search using BM25 and vector similarity search. When a user submits a query, the system first queries both indexes simultaneously. The BM25 index excels at retrieving documents that contain exact terms, particularly rare tokens like error codes, product SKUs, and specific acronyms. On the other hand, the vector index captures semantic meaning, allowing it to find documents that may not contain the exact words but convey similar concepts, such as 'how do I cancel' being similar to 'terminate my subscription.'
Once both searches return their respective results, the next step is to merge these lists. This is typically done using techniques like Reciprocal Rank Fusion or a weighted scoring system. These methods take into account the relevance scores from both indexes and combine them in a way that reflects the strengths of each. For instance, if a document is highly relevant in the BM25 results but less so in the vector results, the fusion method can still elevate its rank based on its exact match quality, ensuring that both types of relevance are considered.
This combination allows the system to maintain high recall across various query types. For example, in a technical support context, a user might enter a specific error code, and the BM25 index will quickly retrieve the relevant documentation. Simultaneously, if a user phrases their query in a more natural language, the vector index can still provide useful results. The fusion of these two approaches ensures that the final result set is comprehensive, catering to both precise and paraphrased queries.
- Submit user query to both indexes.
- Retrieve results from BM25 for exact matches.
- Retrieve results from vector index for semantic matches.
- Merge results using Reciprocal Rank Fusion.
- Return final ranked results to user.
The software-engineering analogue
In software engineering, this approach is akin to querying two different databases and merging the results based on a ranking algorithm. You might think of it as performing a UNION operation on two sets of results. However, the key difference here is that the two indexes measure fundamentally different aspects of relevance. The keyword index focuses on surface-level matches, while the vector index captures deeper semantic meaning. This means that instead of simply deduplicating results, the fusion process must intelligently combine these different relevance signals.
Why not the obvious alternative
A newcomer might first consider using pure vector search because it seems modern and sophisticated. However, this approach often fails to retrieve exact identifiers. For example, embeddings can confuse similar error codes, like 'ERR_2043' and 'ERR_2049,' leading to missed results. Conversely, relying solely on BM25 can overlook important synonyms and variations in phrasing that users might employ, resulting in incomplete search outcomes.
The hybrid approach, while requiring an additional query and a merge step, consistently outperforms either method alone. This is especially true in real-world scenarios where queries often mix technical jargon with natural language. By combining both strategies, the hybrid search effectively addresses the shortcomings of each, ensuring comprehensive coverage of both exact matches and semantic meanings.
When to use which
When deciding which method to use, consider the nature of your data and user queries. If your corpus contains a lot of specific codes, names, or technical terms, leaning on keyword search with BM25 is wise. On the other hand, if your application involves conversational Q&A or user-generated content, vector search will likely yield better results. However, the hybrid search serves as a safe default, especially when user queries are unpredictable, which is common in most production systems.
In production: when it breaks
Why it happens. The hybrid search may be failing due to an imbalance in the weighting of the keyword and vector components. If the keyword index (BM25) is overly dominant, it may prioritize exact matches over semantically relevant results, leading to a poor user experience.
- Check the weights assigned to keyword and vector scores in the hybrid search algorithm.
- Log the top N results for various queries and analyze their relevance scores from both BM25 and vector models.
- A/B test different weight configurations to see how they impact result relevance.
What it delivers
The hybrid search approach delivers a balanced recall that doesn’t falter on either exact-match queries or paraphrased inquiries. It provides a single retriever that effectively covers the blind spots of both keyword and vector searches, ensuring that users can find the information they need regardless of how they phrase their queries.