The layers of state an agent carries: short-term (the current context window), long-term (durable facts/preferences retrieved later), and episodic (records of past interactions and their outcomes).
How it works
The agent's memory system consists of three layers: short-term, long-term, and episodic memory. Short-term memory holds the current context of the conversation, which is limited by the model's context window. As the conversation progresses and the window fills, the agent summarizes previous exchanges to make space for new information. This ensures that the most relevant details remain accessible while keeping the conversation coherent. For example, in a customer service scenario, the agent might summarize user queries and responses to maintain a smooth dialogue.
Long-term memory stores durable facts and preferences in a persistent database, such as a vector database or a key-value store. When the agent needs to recall information in subsequent interactions, it retrieves relevant data based on the current context. This retrieval process is crucial because it allows the agent to provide personalized responses. For instance, if a user frequently asks about a specific product, the agent can remember this preference and tailor its responses accordingly, enhancing the user experience.
Episodic memory records past interactions and their outcomes, allowing the agent to learn from previous experiences. This is particularly useful when the agent needs to adapt its behavior based on past failures or successes. For example, if an agent recalls that a particular approach failed in a previous interaction, it can adjust its strategy in the current conversation. Libraries like Mem0 or Zep are often used to manage the extraction and retrieval of this memory, ensuring that the agent can access the right information at the right time.
- Use short-term for live context.
- Store facts in long-term memory.
- Log interactions in episodic memory.
- Retrieve only relevant memories.
- Summarize context to fit window.
The software-engineering analogue
In software engineering, you can think of short-term memory as akin to CPU registers, which store temporary data for quick access during processing. Long-term memory resembles a database that holds persistent data, while episodic memory is similar to an event log that records system activities over time. However, the key difference lies in how memory is managed in AI systems. The writes and reads in an AI context are lossy and model-mediated, meaning that what gets remembered is often a summarized or embedded version of the original data, rather than an exact record. This can lead to instances where important details are lost or misrepresented.
Why not the obvious alternative
A common initial approach might be to simply append all previous interactions to the prompt, creating a running history of the conversation. However, this method quickly fills the context window, leading to degraded attention and performance as the model struggles to process an overwhelming amount of information. It incurs higher costs, as every token in the prompt must be paid for with each call. This approach fails to maintain a coherent context and can result in the agent losing track of the conversation.
By externalizing memory and retrieving only the relevant pieces of information, the agent can keep the context lean and efficient. This allows knowledge to persist beyond a single session, enabling a more personalized experience for returning users. However, this method introduces its own challenges, such as the risk of retrieving incorrect memories or failing to surface relevant ones, which can impact the agent's performance and reliability.
When to use which
Short-term memory is sufficient for single-shot tasks where context is not critical. When dealing with returning users or tasks that benefit from personalization, long-term memory becomes essential, as it allows the agent to recall user preferences and past interactions. Episodic memory should be incorporated when the agent needs to learn from its previous experiences, enabling it to adapt and improve over time. It’s important to avoid building complex memory infrastructure unless you plan to query it, as unnecessary complexity can lead to maintenance challenges and inefficiencies.
In production: when it breaks
Why it happens. This issue often arises when the agent's episodic memory is not properly updated or accessed. If the memory storage is limited or if there’s a bug in the retrieval logic, the agent may fail to use critical past interactions, resulting in a lack of context and poor user experience.
- Check the memory update logs for recent interactions to ensure they are being recorded correctly.
- Measure the size of the episodic memory storage to confirm it is not exceeding its limits.
- Test the retrieval function by simulating queries for past interactions and verifying if the expected results are returned.
What it delivers
The memory system provides continuity and personalization that extends beyond a single context window. This approach avoids the pitfalls of stuffing history into every prompt, allowing the agent to maintain a coherent and relevant dialogue while enhancing the user experience through tailored responses.