When teams first build an agent, "memory" usually means one thing: a vector store of past messages. That's fine for a demo. It falls apart the moment the agent has to do anything serious over a long stretch of time. Memory isn't one thing. It's at least three.
The three layers we actually ship
Borrowing terms from cognitive science doesn't always pay off, but here it maps directly onto the failure modes we keep running into in production. Real agents need separate stores for what's happening now, what happened recently, and what's generally true.
Why a single vector store breaks
The temptation is to dump everything into one collection and let cosine similarity sort it out. Three failure modes show up almost immediately:
- Recent crowds out relevant. A flurry of recent tool calls outranks the one foundational fact the agent actually needs.
- Stale crowds out fresh. A six-month-old conversation surfaces in front of yesterday's decision because the embedding happened to score higher.
- Identity blurs. The agent can't tell its own past actions apart from external knowledge, so it cites tool output as if it were doctrine.
Memory is a routing problem, not a storage problem. The hard part is deciding what kind of memory a question needs, not where to put the bytes.
How we route in practice
Before any retrieval, a small classifier inspects the agent's intent and picks a strategy:
- "What did we just do?" → working memory only. No retrieval.
- "Have we seen this user before?" → episodic, filtered by user id and recency window.
- "What's the policy on X?" → semantic, filtered by document type.
- "Why did you do that last week?" → episodic, with a re-ranker that prefers reflection summaries over raw turns.
The summariser between layers
Episodic memory grows fast. We compress it nightly: a model reads a day's worth of events and writes a short summary keyed by themes, things like "user fought with the refund flow twice" or "agent escalated three tickets to a human." Summaries get stored alongside the raw events with a higher retrieval weight, so the agent reaches for the digest before the firehose.
What we'd tell a team starting today
- Split your stores on day one. Migrating later is painful.
- Tag every memory write with a source, a timestamp, and a TTL. Cheap to add up front, impossible to backfill.
- Make retrieval observable. Log every query, every result rank, and every miss. Most "the model is dumb" complaints are actually retrieval bugs.
- Write your evals against memory operations directly, not just end-to-end conversations. You'll catch regressions faster.
Done well, memory becomes the part of the system users notice most, not because it's flashy but because the agent stops feeling like it just met them.