A model has no memory. Each request is independent, and everything it appears to remember was re-sent in the prompt.
"Agent memory" is therefore not a model capability. It is a storage system you build, plus a policy for what enters the context on each turn. Treating it as a prompt problem is why assistants forget things users told them ten minutes ago.
Four kinds, with different homes
Diagram, as text
- Working memory: context window leads to Session memory: transcript plus summary
- Session memory: transcript plus summary leads to Long-term facts: structured, with provenance and expiry
- Session memory: transcript plus summary leads to Episodic archive: embedded, retrieved on reference
- Long-term facts: structured, with provenance and expiry leads to User can list, edit, delete
| Kind | Holds | Lives in | Lifetime |
|---|---|---|---|
| Working | Current trajectory: steps, tool results | Context window | One task |
| Session | This conversation | Cache or database | Hours to days |
| Long-term | Durable facts about the user | Database | Indefinite |
| Semantic | Knowledge to retrieve | Vector store | Indefinite |
Conflating them is the usual mistake. A user's plan tier is long-term structured data; a tool result from three steps ago is working memory; product documentation is semantic memory. They need different storage, different retrieval and different lifetimes.
Working memory
Everything the agent has done in the current task — the goal, actions taken, observations returned.
It grows fast. Ten steps with substantial tool results can exhaust a context window on their own, which is the practical limit on trajectory length.
Three ways to control it:
Summarise old steps. Compact the early trajectory into a few sentences once it exceeds a threshold.
Truncate tool results. A tool returning 500 rows should return the 10 that matter, or a summary plus a handle for fetching more. This is usually the single largest saving.
Drop irrelevant branches. If the agent explored a path and abandoned it, the details rarely need to stay.
Session memory
The conversation. Handled with the layered approach from the context engineering lesson — durable facts as structured state, older turns summarised, recent turns verbatim.
The design questions are ordinary systems questions, and treating them that way avoids most problems:
Where does it live? Redis for active sessions with a TTL; a database if it must survive longer. It is state, so the statelessness constraint from the scalability lesson applies — never in a single server's memory, or a user's conversation vanishes when a deploy rolls.
When does it expire? A support conversation is stale after a day. A long-running project assistant should persist.
Who can read it? Conversations contain personal data. Scope by user, encrypt at rest, and honour deletion requests — a point the privacy lesson develops.
Long-term memory
The interesting one, and the one that makes an assistant feel like it knows you.
Two approaches, and the difference matters more than it first appears.
Extracted structured facts. After a conversation, a model pulls durable facts into a schema:
{
"preferences": { "communication": "concise", "timezone": "Asia/Kolkata" },
"context": { "plan": "enterprise", "role": "engineering manager" },
"history": { "recurring_issue": "SSO timeouts on mobile" }
}Small, queryable, auditable, correctable, and cheap to include in every prompt. This should be your default.
Embedded memories. Store past interactions as vectors and retrieve relevant ones per turn. More flexible for open-ended recall, but you inherit every retrieval problem — irrelevant memories surfacing, relevant ones missed — and it is far harder to inspect or correct.
The practical design uses both: structured facts always present, embedded memories retrieved when a question is about something historical.
Memory that is wrong
Unlike a cache, incorrect memory does not expire into correctness. It persists and shapes every future interaction.
A user mentions they work at one company, later changes jobs. The stored fact is now wrong, and the assistant keeps referring to the old employer with total confidence.
Three defences:
Timestamp everything, and prefer recent facts on conflict.
Make memory visible and editable. Let users see what is stored about them and correct it. This is a product feature, a debugging tool and, in several jurisdictions, a legal requirement.
Re-verify durable claims occasionally rather than treating an extraction from six months ago as permanent truth.
State for long-running trajectories
An agent working through a multi-step task is a stateful process, and the systems questions are the ones from distributed coordination and sagas:
Persist after each step, so a crash at step seven resumes rather than restarts.
Make steps replayable. Replaying a read is free; replaying a write is a duplicate unless it is idempotent.
Handle concurrency. Two requests for the same session must not both mutate memory unchecked — the same lost-update race as anywhere else, and the fix is the same atomic conditional write.
Compensate on failure. If step three sent an email and step five fails, the email is sent. Decide in advance what unwinding means, exactly as in the saga pattern.
Check your understanding
Common mistakes
Treating memory as a prompt problem. It is storage with a retrieval policy.
One store for everything. Working, session, long-term and semantic memory have different shapes and lifetimes.
No write policy. Remembering everything degrades retrieval and costs money.
Append-only facts. Without supersession, corrections never take effect.
Sessions in server memory. Lost on deploy, and broken behind a load balancer.
No user visibility. Users cannot correct what they cannot see, and you cannot debug it either.
Unbounded tool results in working memory. The fastest way to exhaust a context window.
Practice
Design memory for a personal assistant used daily over years: what is stored, where, how it is retrieved, and how it is corrected.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Four kinds of memory with four different lifetimes and four different stores.
Show a strong answer
Working memory — the context window. The current turn, recent messages, active tool results. Ephemeral by design, reconstructed each turn, never persisted as such.
Session memory — a durable record keyed by conversation. Full transcript plus a running summary. Retrieved by loading the summary and the last few turns. Retention months rather than forever, and it is what makes "carry on from yesterday" work.
Long-term facts — a structured store, not a vector index. "Prefers evening meetings", "partner's name is X", "allergic to shellfish". Each record carries the value, when it was learned, the turn it came from, and a confidence. Structured rather than embedded because these need to be listed, edited and deleted by the user — operations a vector store does badly.
Episodic memory — the searchable archive. Past conversations, embedded, retrieved by similarity when a question refers to something discussed previously.
How it is retrieved: long-term facts are loaded on every turn, because they are small and always relevant. Episodic memory is retrieved only when the query suggests a reference to the past. Loading everything every turn is the mistake that dilutes context and raises cost for no benefit.
How it is corrected — the part usually missing:
- The user can see and edit their memories. A memory the user cannot inspect is one they cannot fix, and they will instead conclude the assistant is wrong about them.
- New information supersedes rather than accumulates. If the user states a new preference, mark the old record superseded with a timestamp rather than storing both — otherwise retrieval returns contradictory facts and the model picks arbitrarily.
- Facts decay by class. A dietary restriction is stable; "currently working on project Y" should expire in weeks. Give each class a default lifetime rather than treating all memory as permanent.
- Provenance on every record, so the assistant can say why it believes something. "You mentioned this on 3 March" is what lets a user correct it with confidence.
An agent is halfway through a multi-step task — it has charged a customer and updated a record — when the process crashes. Design the state handling so the restart is safe.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The question is not how to resume, but how to know what already happened.
Show a strong answer
The requirement: a restart must not repeat any side effect. That rules out keeping progress only in the context, because the context dies with the process.
Persist the trajectory as it happens, not at the end:
- Before each step with a side effect, write an intent record — step ID, action, arguments, idempotency key, status
pending— committed durably before the action is attempted. - Perform the action, passing the idempotency key so the downstream system can recognise a repeat.
- Record the outcome — status
completedwith the result, orfailedwith the error.
On restart, load the trajectory and inspect the last record:
completed— resume from the next step. Safe and unambiguous.failed— the action did not take effect; retry or handle it.pending— the dangerous case, and the whole reason for the design. The process died between writing the intent and recording the outcome, so you do not know whether the charge happened. Do not guess. Query the downstream system using the idempotency key to find out, and only then decide.
What makes this work: the idempotency key is generated with the intent record, not at call time, so a retry presents the same key. That is what lets the payment provider answer "you already did this, here is the original result" rather than charging twice.
What I would add: a reconciliation job for trajectories stuck in pending beyond a threshold, because a crash between the two writes leaves a record nobody will otherwise look at. Silent pending records are how money goes missing.
Key takeaways
- Models are stateless — memory is a storage system you build, plus a policy for what enters the prompt.
- Distinguish working, session, long-term and semantic memory; they need different stores and lifetimes.
- Working memory grows fast; truncating tool results is usually the biggest saving, and it is where context poisoning happens.
- Prefer extracted structured facts for long-term memory — small, queryable, correctable — with embedded memories for open-ended recall.
- The hard part is the write policy: deciding what deserves remembering, not how to retrieve it.
- Wrong memory does not expire. Timestamp facts, resolve conflicts by recency, and let users inspect and edit them.
- A long trajectory is a stateful workflow — persist each step, make steps replayable and idempotent, and plan compensation.
Further reading
- Charles Packer et al., "MemGPT: Towards LLMs as Operating Systems" (arXiv, 2023).
- Joon Sung Park et al., "Generative Agents: Interactive Simulacra of Human Behavior" (UIST 2023) — memory retrieval, reflection and decay.
- Pat Helland, "Idempotence Is Not a Medical Condition" (ACM Queue, 2012).
- Hector Garcia-Molina and Kenneth Salem, "Sagas" (SIGMOD 1987) — durable multi-step processes with compensation.
- Gunnar Morling, "The Outbox Pattern" (2019) — writing intent and effect without losing either.