Two ways to feed a model the data it needs. A feature store serves precomputed, versioned structured features (classic ML). Runtime context is the text — retrieved chunks, tool results, user history — assembled into the prompt per request (LLM apps).
How it works
Feature stores like Feast, Tecton, and Databricks maintain two key components: an offline store for training data and a low-latency online store for inference. This setup ensures that the features used during model training are the same as those used during inference, which is crucial for maintaining Train/serve consistency. When a model needs to make a prediction, it retrieves these precomputed, versioned structured features quickly, allowing for efficient and reliable decision-making.
In contrast, Runtime context is created dynamically at the moment of inference. This involves a retrieval and orchestration layer that gathers various data sources such as RAG (retrieval-augmented generation) chunks, API results, and user history. Unlike feature stores, runtime context does not have an offline training counterpart; it is assembled on the fly and is inherently limited by the model's token window. This means that the context must be carefully curated to fit within the constraints of the model's input size, which can affect the richness of the information provided.
For example, imagine a fraud detection model that uses a feature store to access a precomputed fraud score based on user behavior. When a new transaction occurs, the model retrieves the score quickly from the feature store. In contrast, if the model needs to reference recent user interactions or external API data, it will rely on runtime context to compile this information into the prompt for the model, ensuring that the most relevant and timely data is used for the prediction.
- Use feature store for structured data.
- Use runtime context for unstructured data.
- Combine both for optimal performance.
- Feature store ensures low-latency access.
- Runtime context adapts to request needs.
The software-engineering analogue
You can think of a feature store as similar to a materialized view or cache in software engineering, where data is precomputed and stored with strict versioning to ensure consistency. This allows for quick access to structured data that can be reused across different models. On the other hand, runtime context resembles the process of building a request payload for an API call, where the data is assembled at the moment of the request based on the current needs.
The key difference here is that while a feature store provides structured, consistent data, runtime context deals with unstructured prose and competes for a finite token budget. This means that the context must be optimized for relevance and clarity, as it can only include a limited amount of information due to the constraints of the model's input.
Why not the obvious alternative
A common misconception is that RAG can completely replace feature stores. This is not the case. For numeric signals like fraud scores, lifetime value (LTV), or embeddings that need to be reused, having these precomputed and stored in a feature store is essential. If you were to recompute these values on-the-fly within the prompt, you would introduce latency and potential inconsistencies.
relying solely on runtime context can lead to training-serving skew. This occurs when the model is trained on one set of inputs but serves predictions based on another, potentially leading to degraded performance. Precomputed features in a feature store help to mitigate this risk by ensuring that the same data is used during both training and inference.
When to use which
Use a feature store when you need structured, reusable, and latency-critical signals that are shared across multiple models. This is particularly important for maintaining train/serve parity. In contrast, opt for runtime context when you require unstructured knowledge, per-request facts, or outputs from external tools. Many real-world systems effectively combine both approaches, incorporating features from the feature store into the prompt as context to enrich the model's inputs.
In production: when it breaks
Why it happens. This issue may arise from the model relying on runtime context, which can lead to inconsistent and incomplete data being fed into the model. If the retrieval system is slow or if the context is not properly assembled, it can degrade the model's performance and response time.
- Step 1: Monitor the time taken for retrieval of context data and compare it against the expected benchmarks.
- Step 2: Log the consistency and completeness of the retrieved context data for a sample of requests.
- Step 3: A/B test the predictions using both feature store data and runtime context to evaluate performance differences.
What it delivers
The combination of these two approaches delivers consistency and reuse for structured data while providing freshness and flexibility for unstructured knowledge. By strategically using both feature stores and runtime context, practitioners can optimize their models to perform effectively across various scenarios.