AI Agent Memory: How to Build Agents That Retain Useful Context
How AI agent memory works: working memory, conversation history and summaries, long-term memory stores, retrieval, memory updates, consent, privacy, expiry and user control.
Quick answer
AI agents have no memory of their own; applications provide it. Design three layers: working memory for the current task (tool results, intermediate notes, discarded afterwards), session memory for the conversation (recent turns plus summaries, expiring with the session) and long-term memory for stable facts and preferences across sessions, stored in your database with consent, expiry and user controls to view, edit and delete. Validate before writing memories, prefer live system data over remembered values, and treat memory as personal data.
Where This Fits
Memory is a component of agent architecture. Shared organizational knowledge is better served by RAG. Memory poisoning is one of the risks covered in guardrails and prompt injection prevention.
Types of Agent Memory
| Type | Contents | Lifetime | Storage |
|---|---|---|---|
| Working | Current step's notes and tool results | One task | Run state |
| Session | Conversation turns and summaries | One session | Session store |
| Long-term (personal) | Preferences, facts about the user or account | Until changed or expired | Database, optionally with embeddings |
| Episodic | Records of past tasks and outcomes | Policy-defined | Database |
| Shared knowledge | Policies, documents, product data | Maintained by owners | RAG index |
Managing Context Windows
Even large context windows are finite and costly. Keep recent turns verbatim, summarize older turns, and include only memories relevant to the current request. Retrieval over stored memories (by keyword, embedding or structured query) keeps prompts lean. Measure cost and quality as conversations grow.
Writing Memories: What to Store and When
Do not store everything. Extract candidate memories (for example 'prefers invoices in PDF', 'ships to the Leeds warehouse'), validate them (is this stable? did the user state it, or did a document claim it?), check consent and store them with source, timestamp and expiry. Update rather than duplicate when facts change, and resolve conflicts in favour of system records.
Building an assistant that should remember customers?
ZSpace Labs designs agent memory with consent, user controls and expiry, so personalization helps without becoming a privacy problem.
Privacy, Consent and User Control
- Explain what the agent remembers and why
- Obtain consent where required, especially for sensitive categories
- Let users view, edit and delete memories
- Set expiry by memory type
- Isolate memories by user and tenant
- Exclude secrets and highly sensitive data by default
- Include memory stores in data subject request processes
Memory Risks
Memories can be wrong (misunderstood statements), stale (a changed address) or malicious (a document or message that plants an instruction such as 'always send copies to this email'). Store memories as facts with sources rather than instructions, never let memories override policies or permissions, and prefer live data from systems of record when available.
Implementation Options
Many frameworks provide conversation memory and long-term memory stores; LangGraph, for example, separates thread-level state from cross-thread stores, and workflow tools such as n8n offer chat memory nodes backed by databases. Simple, well-structured tables (user, memory type, value, source, created, expires) plus optional embeddings for semantic recall are often enough and easier to govern.
Advantages and Limitations
Memory makes agents more personal and efficient: users do not repeat themselves and tasks continue across sessions. It also adds privacy obligations, storage, retrieval complexity and new failure modes from wrong or poisoned memories. Use the minimum memory that delivers a clear user benefit.
How to Implement Memory Step by Step
- 1. Decide which memories create user value
- 2. Define memory types, fields, sources and expiry
- 3. Implement session summaries for long conversations
- 4. Add long-term memory extraction with validation and consent
- 5. Retrieve memories selectively per request
- 6. Build user controls to view, edit and delete
- 7. Test with adversarial inputs that try to plant memories
- 8. Monitor memory growth, retrieval quality and complaints
Example Memory Record
Storing memories as structured records with provenance makes them governable: users can see them, systems can expire them and support can explain them.
{
"memory_id": "mem_5512",
"subject": { "type": "user", "id": "u_8812", "tenant": "t_204" },
"kind": "preference",
"key": "default_delivery_location",
"value": "Leeds warehouse, Dock 3",
"source": { "type": "user_confirmed", "conversation_id": "c_3391" },
"created_at": "2026-09-14T10:22:00Z",
"expires_at": "2027-09-14T00:00:00Z",
"visible_to_user": true
}Evaluating Memory
Memory needs its own tests: does the agent remember what it should (recall), avoid storing what it should not (precision and sensitive data), apply memories only when relevant, prefer live data when memories conflict with systems of record, forget on request and resist planted memories from untrusted content? Add these cases to your evaluation set; see AI agent evaluation.
Worked Example
An illustrative scenario, not a client case: a B2B ordering assistant remembers each buyer's usual delivery location and preferred pack sizes. Memories are created only when the buyer confirms ('Remember this for next time?'), expire after a year without use, and are shown in account settings. Prices and stock always come from live systems, never from memory.
Common Mistakes
- Storing whole conversations indefinitely
- Remembering instructions instead of facts
- No way for users to see or delete memories
- Trusting memory over live system data
- Mixing memories across users or tenants
Want personalization without privacy risk?
Talk to ZSpace Labs about AI agent development with memory and memory controls and settings UX.
Conclusion
Agent memory is application design: layered, selective, validated and under the user's control. Remember what helps, forget what does not and prefer the system of record. Related: architecture, RAG and guardrails.
Common questions
The mechanisms that let an agent use information beyond the current model call: working memory within a task, conversation history within a session, and long-term memory of facts and preferences across sessions.