Say “agent memory” and you will get a dozen frameworks thrown back at you: working and episodic memory, MemGPT, Mem0, vector stores, knowledge graphs, context engineering. They are usually presented as if you have to pick one. You do not. They answer different questions, and most of the confusion comes from comparing across layers that were never in competition.
There are four layers, and everything gets easier once you keep them separate.
| Layer | The question it answers |
|---|---|
| Kinds | What kinds of memory exist |
| Management | How to manage a finite context budget |
| Storage | Where the memory physically lives |
| Product | Who already built it |
What kinds: the cognitive taxonomy
This taxonomy is borrowed from how psychologists divide human memory, and it transfers cleanly.
- Working memory is the scratchpad for the current task. In practice it is whatever sits in the context window right now.
- Episodic memory is what happened, and when. This is the layer that lets an agent say “we already tried that.”
- Semantic memory holds durable facts about the user and the world. This layer is your knowledge base.
- Procedural memory is how to do things: learned skills and routines.
Most complaints that “my agent has no memory” are really about a missing episodic layer. I have written before about why a bigger context window does not fix that. Notice the third entry as well: a knowledge base is the semantic layer of memory. That is why the two topics keep collapsing into each other.
How to manage it: the budget problem
Context is a finite attention budget, so the real job is curating what goes into it. The framework here is context engineering, and it consists of four moves. Offload pushes state out to files or a store. Retrieve pulls back only what is relevant. Compress summarizes the old material. Isolate gives each sub-task only what that sub-task needs.
MemGPT, now called Letta, made this concrete with an operating-system metaphor: treat the context window as RAM and external storage as disk, page memory in and out, and let the agent edit its own memory.
Where it lives: storage
The same memory can sit in three kinds of store, and serious systems mix them.
- Vector stores use embeddings and similarity search. This is the default choice.
- Graph stores hold a temporal knowledge graph, recording who relates to whom and when. They are strong on relationships and on how those relationships change.
- Files mean Markdown in git: human-readable, auditable, and free of lock-in. Anthropic’s own Claude memory is a folder of Markdown files.
Who built it: products
If you would rather not assemble this yourself, the options include Mem0 (multi-level, auto-extracted), Letta/MemGPT (OS-style tiered memory), Zep (temporal knowledge graph), Anthropic’s Memory Files (file-based), and LangMem and Cognee in the LangChain orbit.
The honest version
None of these is a choice you make once. A real agent uses several kinds of memory at once, managed by context engineering, sitting in some mix of stores, and reached through a product or through your own glue code.
The trap is treating “memory” as a single switch. The sign that you have fallen into it is comparing a kind (episodic) against a store (vectors) against a product (Mem0), as though those were the same decision.
And here is the conclusion I keep arriving at: a knowledge base is just the semantic layer with writing turned off. Let the agent write back to it, and it stops being a knowledge base and starts being a memory.