A cache keyed on query embeddings: if a new query is semantically close enough to a past one, return the stored response instead of calling the model again.
How it works
Semantic caching starts by embedding the incoming query into a vector space using a model like Sentence-BERT or OpenAI's embedding API. This transforms the query into a numerical representation that captures its semantic meaning. Next, the system performs a vector search against the cache, looking for previously stored queries that are semantically similar. If the nearest hit's similarity score exceeds a predefined threshold, the system retrieves and serves the cached answer instead of querying the model again. If no suitable match is found, the model processes the query, generates a response, and stores this new query-response pair in the cache for future use.
This approach hinges on the effectiveness of the embedding model and the similarity threshold. For instance, if the threshold is set too high, the system may miss valid cached responses, leading to unnecessary model calls. Conversely, if the threshold is too low, the system might serve an answer that is not quite relevant, which could confuse users. Tools like GPTCache implement this mechanism in front of large language models (LLMs), optimizing query handling and reducing costs associated with repeated model invocations.
As an example, consider a user asking, 'What are the store hours?' If a similar past query, 'When does the store open?' exists in the cache and meets the similarity threshold, the cached response will be returned. This process not only speeds up response times but also minimizes the load on the LLM, allowing for more efficient resource management.
- 1. Embed the incoming query.
- 2. Perform a vector search in the cache.
- 3. Check similarity against a threshold.
- 4. Return cached response if similar.
- 5. Call model and store new response if not.
The software-engineering analogue
You can think of semantic caching as similar to using Memcache or Redis in front of an expensive API endpoint. In traditional caching, you store exact string matches to responses, which work well when users phrase their requests identically. However, with semantic caching, the key is fuzzy. For example, the queries 'reset my password' and 'how do I change my password' are different in wording but similar in intent, and they will collide in the cache.
This fuzzy matching introduces a unique challenge: cache correctness becomes probabilistic. Unlike traditional caching where you can be certain of a match, here you have to tune your similarity threshold carefully to balance between relevance and accuracy. A poorly set threshold can lead to serving incorrect or misleading answers, which is a risk you don't face with exact-string caching.
Why not the obvious alternative
The obvious alternative to semantic caching is exact-string caching, where you store responses based on precise query matches. However, this approach typically results in a very low hit rate, often around 0%, especially with free-text queries. Users phrase their questions in countless ways, and without semantic understanding, the system fails to recognize that different queries can seek the same information.
In contrast, semantic matching significantly improves hit rates by allowing for variations in wording and phrasing. This not only cuts down on latency but also reduces token usage when querying the model. The downside is that if the similarity threshold is set too loosely, the system might return a cached answer that is subtly wrong, which can mislead users. Therefore, careful tuning of the threshold is essential to ensure that the cached responses remain accurate and relevant.
When to use which
Semantic caching is particularly useful for high-volume, repetitive queries, such as those found in FAQ bots or customer support systems. These scenarios often involve users asking similar questions, making semantic caching a great fit. However, it can be risky for personalized, time-sensitive, or user-context-specific queries. In such cases, the cache should be scoped narrowly or avoided altogether to ensure that users receive the most accurate and relevant responses.
When deciding whether to implement semantic caching, consider the nature of your queries. If they are generally consistent and factual, go ahead and use semantic caching. If they vary significantly based on user context or require real-time updates, it’s better to query the model directly to avoid serving outdated or incorrect information.
In production: when it breaks
Why it happens. The low cache hit rate may be due to an inadequate similarity threshold for determining when to use cached responses. If the threshold is set too strict, semantically similar queries may not be recognized, leading to unnecessary model calls.