Storing and reusing prior LLM outputs. Exact caching keys on the literal prompt; semantic caching returns a stored answer when a new prompt is close enough in meaning to a past one.
How it works
Response caching operates on two levels: exact and semantic. For exact caching, you start by hashing the normalized prompt along with any parameters. This hash serves as a unique key to look up in a key-value store, like Redis. If there's a match, you simply serve the stored output, which is incredibly fast and efficient. This method works well for prompts that are identical or nearly identical, allowing you to bypass the overhead of generating a new response.
On the other hand, semantic caching takes a more nuanced approach. Here, you embed the incoming query into a vector space using models like OpenAI's embeddings or Sentence Transformers. You then perform a vector similarity search against previously stored embeddings to find responses that are meaningfully similar. If the similarity score exceeds a defined threshold, you return the cached response. This method allows you to serve answers even when the new prompt isn't an exact match, expanding the range of queries that can benefit from caching.
there’s a related concept called prompt caching, which focuses on optimizing the cost and time to first token (TTFT) when dealing with long shared prefixes in prompts. Instead of caching entire responses, you cache the key-value state of the model after processing a shared prefix. This way, if the next prompt is a continuation, you can skip generating from scratch and instead build on the cached state, significantly reducing costs and response times.
- Hash the prompt for exact caching.
- Embed the prompt for semantic caching.
- Use Redis for key-value storage.
- Set a similarity threshold for semantic hits.
- Tune TTL based on usage patterns.
The software-engineering analogue
You can think of response caching as similar to HTTP or CDN response caching, where frequently requested data is stored for quick retrieval. In software engineering, memoization serves a similar purpose by storing the results of expensive function calls and returning the cached result when the same inputs occur again. However, the key difference with semantic caching is that it matches on approximate meaning rather than exact inputs. This means a 'hit' in semantic caching is based on a similarity judgment, which introduces a risk of false positives that a traditional hash cache does not have.
Why not the obvious alternative
The naive approach many newcomers might take is to simply regenerate answers for identical or similar prompts each time. This leads to unnecessary costs and increased latency, especially for frequently asked questions or repeated queries. Each generation incurs computational costs, and the delay can frustrate users who expect quick responses.
By using caching, you can serve these common queries in milliseconds at a fraction of the cost. Without caching, you’re stuck in a cycle of wasteful computation, which not only drains resources but also diminishes the user experience due to longer wait times for responses.
When to use which
Use exact caching for deterministic and repeated prompts, such as document Q&A or classification tasks. In these cases, both exact and semantic caching can yield significant performance improvements. However, for personalized, stateful, or high-stakes queries, you need to be cautious. A loose semantic threshold may return subtly incorrect responses, which can be problematic. It's crucial to tune the similarity threshold and time-to-live (TTL) settings carefully, and in cases where user-specific context is involved, consider skipping caching altogether.
In production: when it breaks
Why it happens. This increase in response time can occur due to cache misses, where the system fails to retrieve stored responses for incoming requests. If the cache is not effectively storing or retrieving responses due to an inadequate keying strategy or stale data, the system may revert to generating responses from scratch, which is significantly slower.