Agent memory design: what to remember across sessions, where to keep it, and how to stop it rotting

Language models do not remember anything between calls. The CoALA paper from Princeton puts it bluntly: "Language models are stateless: they do not persist information across calls." Everything an agent appears to remember is something your code chose to store and chose to send back. That makes memory a design problem rather than a model feature, and it is one where the defaults are easy to get wrong in both directions. Remember too little and the agent asks the same questions every session. Remember too much, or the wrong things, and you get a growing pile of stale notes, leaked personal data and an attack surface that outlives the conversation that created it.

This article looks at how three tools people actually build with handle memory today: Anthropic's client-side memory tool, sessions in the OpenAI Agents SDK, and the short-term and long-term memory split in LangGraph. Then it covers the decisions none of them make for you.

Two kinds of memory, and why the split matters

Every framework here draws the same line, even if the names differ. LangGraph's docs describe short-term memory as "thread-scoped memory" that "tracks the ongoing conversation by maintaining message history within a session," persisted with a checkpointer so a thread can be resumed. Long-term memory "stores user-specific or application-level data across sessions and is shared across conversational threads," and lives in a store organized by custom namespaces rather than by thread ID.

The OpenAI Agents SDK's sessions sit on the short-term side. The docs describe the behavior plainly: before each run, the runner retrieves the session's conversation history and prepends it to the input; after each run, all new items, including user input, assistant responses and tool calls, are stored back in the session. You pick a backend (SQLiteSession for local work, RedisSession for shared workers, SQLAlchemySession for an existing database, and several more) and the SDK does the bookkeeping.

from agents import Agent, Runner, SQLiteSession

agent = Agent(name="Assistant", instructions="Reply very concisely.")
session = SQLiteSession("support_ticket_456", "conversations.db")

await Runner.run(agent, "My order 1182 arrived damaged.", session=session)
await Runner.run(agent, "Can you check its status?", session=session)  # sees turn one

Anthropic's memory tool sits on the long-term side. It gives Claude a directory of files under /memories that it can view, create, edit, rename and delete, and the docs say Claude "automatically checks its memory directory before starting a task." The important detail is that the tool is client-side: Claude only requests file operations, and your application runs them against storage you control, whether that is a per-user directory or rows in a database.

Keeping these two separate matters because they fail differently. Short-term history grows with every turn and needs trimming, which is the problem we covered in context compaction for agent loops. Long-term memory grows with every session and needs curating, which is a slower and sneakier problem. If you let one mechanism do both jobs, for example by replaying a user's entire history from six months ago into every new run, you pay for both problems at once.

Decide what kind of thing you are storing

The LangGraph memory guide borrows a useful vocabulary from cognitive science, the same one CoALA uses. Semantic memory holds facts, such as "this user deploys to Cloudflare and writes Python." Episodic memory holds experiences, such as the sequence of steps that worked last time a migration failed. Procedural memory holds rules for doing the task, which for an agent means its prompt, its code and the model weights.

The split is practical, not academic, because each type wants a different storage shape. LangGraph's guide describes two ways to hold semantic memory. A profile is a single JSON document per user that you update each time, which is easy to inject but becomes error-prone as it grows, since the model has to reconcile new information with the whole existing document. A collection is many small documents, which the guide says tends to give higher recall because "it's easier for an LLM to generate new objects for new information than reconcile new information with an existing profile." The cost is that the model now has to decide when to update or delete existing items, and the guide notes some models over-insert while others over-update.

Episodic memory is usually best stored as examples. The same guide says episodic memories are often implemented as few-shot examples, where the hard part is picking the most relevant ones for the current input. Procedural memory is the one to treat with the most caution. CoALA notes that writing to procedural memory "is significantly riskier than writing to episodic or semantic memory, as it can easily introduce bugs or allow an agent to subvert its designers' intentions." An agent that can rewrite its own instructions based on what it read in a web page is an agent whose instructions belong to whoever wrote that page.

A good first exercise is to write down, for your agent, which of these three it actually needs. A coding agent working across sessions mostly needs episodic progress notes. A support agent mostly needs a small semantic profile per customer. Very few production agents need to rewrite their own procedures, and if yours does, put that write behind the same kind of human check we described in approval gates for agent tool calls.

Decide who gets to write, and when

There are two ways for memory to get written, and LangGraph's docs lay out the tradeoff clearly. Writing "in the hot path" means the agent decides to save something during the conversation, usually through a tool. New memories are available immediately and users can be told when something was saved, but deciding what to remember adds latency and competes with the agent's actual task. Writing "in the background" means a separate job reviews conversations later and extracts memories. That removes latency from the main loop and separates concerns, but you have to decide how often to run it, and other threads will not see new facts until it does.

The Claude memory tool is a hot-path design. When the tool is enabled, the API adds a memory protocol to the system prompt that tells Claude to view its memory directory before doing anything else and to "ASSUME INTERRUPTION: Your context window might be reset at any moment, so you risk losing any progress that is not recorded in your memory directory." That framing is a good fit for long-running work, where losing progress is the main risk. It is a less natural fit for remembering user preferences, where a background pass that sees the whole conversation can make a more considered decision about what is worth keeping.

CoALA's worked example is a good template for access rules. For a retail assistant, the authors suggest read and write access to episodic memory so the agent can store new interactions, but read-only access to semantic and procedural memory, "since it should not update the inventory or its own code." Write that down for your own agent as an explicit table of memory types and permissions, then enforce it in the tool handler rather than in the prompt.

The storage layer is your security boundary

Because the Claude memory tool and most Agents SDK session backends run against storage you supply, the security of the memory is the security of your handler. The Anthropic docs list the safeguards that become your job. Claude usually refuses to write sensitive information, but for stronger guarantees you should strip it in your handler before writing. You should cap how large memory files can grow and how much a single view returns. You should periodically delete files that have not been accessed in a long time. And you must validate every path, because a request like /memories/../../secrets.env can otherwise reach files outside the memory directory.

The docs recommend resolving paths to canonical form and checking they stay inside the root, and suggest Python's pathlib.Path.resolve() and relative_to(). A minimal version of that check looks like this:

from pathlib import Path

MEMORY_ROOT = Path("/srv/agent-memory").resolve()

def resolve_memory_path(user_id: str, requested: str) -> Path:
    if not requested.startswith("/memories"):
        raise ValueError("path must start with /memories")
    user_root = (MEMORY_ROOT / user_id).resolve()
    relative = requested.removeprefix("/memories").lstrip("/")
    target = (user_root / relative).resolve()
    target.relative_to(user_root)  # raises ValueError if it escaped the user's root
    return target

Note the per-user root. The /memories prefix the model sees is a virtual path, and the docs describe mapping it onto real storage "such as a per-user directory or keys in a database." Scoping by user in the handler is what stops one customer's agent from reading another customer's notes, and it is not something the model can do for you.

The OpenAI Agents SDK docs are equally direct about sessions. In the section on the SQLite storage trust boundary, they state that a session ID "selects conversation history; the session ID does not authenticate a user or authorize access to that history." The backend does not detect external edits, deletions, reordering or replay, so anything changed directly in the database will show up in the next run's history. The application must authorize access to each session and protect the database file and its backups. The SDK's EncryptedSession wrapper adds encryption and a TTL, but the docs say it does not verify the completeness or order of history and is not a substitute for integrity controls.

There is also a quieter risk. Memory is the one place where untrusted text can survive past the conversation that brought it in. If your agent reads a web page or an email and writes a "fact" from it into memory, every future session will load that fact as if the agent had learned it from you. Treat memory writes that originate from tool output with the same suspicion we recommended in our post on prompt injection in tool results: record where each memory came from, and do not let content from untrusted sources write into the memory that steers future behavior.

Give every memory a shape and a source

The simplest way to make all of the above enforceable is to store memories as records rather than free text. Something like this is enough for most agents:

{
  "id": "mem_0192",
  "type": "semantic",
  "subject": "user:4411",
  "text": "Prefers email follow-ups over phone calls.",
  "source": "user_message",
  "created": "2026-10-31",
  "last_used": "2026-10-31"
}

The type field lets your handler apply the permission table. The source field lets you refuse or quarantine writes that came from tool output. The last_used field gives you the expiration policy Anthropic recommends without guesswork: anything not read in, say, ninety days gets reviewed or dropped. LangGraph's store supports this style directly, saving JSON documents under a namespace such as (user_id, "preferences") with a key, and letting you search within a namespace by content filter and, when an index is configured, by semantic similarity.

CoALA points out that most agent memory work has focused on adding memories, while "modifying and deleting (a case of 'unlearning') are understudied." In practice that is where memory systems rot. A user who changed jobs, a project that was renamed, a preference that reversed: each leaves a stale record that the agent will keep trusting. Building delete and update paths on day one, and testing them, is cheaper than cleaning up later.

Memory and compaction do different jobs

It is tempting to treat compaction as memory, since both keep an agent going past the limits of its context window. They solve different problems. Anthropic's context engineering guide describes compaction as summarizing a conversation nearing the context limit and starting a new window with the summary, and structured note-taking as the agent regularly writing notes "persisted to memory outside of the context window" that are pulled back in later. The memory tool docs suggest using both for long-running agents: compaction keeps the active context small, and memory preserves the information that must survive summarization.

The Agents SDK draws the same line from the other side. SessionSettings(limit=N) caps how many history items are retrieved per run, a session_input_callback lets you filter or reorder history before the model call without changing what is stored, and OpenAIResponsesCompactionSession wraps another session to compact stored history with the Responses API. None of those decide what deserves to outlive the conversation. That still needs an explicit memory write.

A pattern for multi-session work

For agents that work on the same project across many sessions, the memory tool docs describe a pattern worth copying even if you do not use Claude. The first session is an initializer that sets up the memory files before any real work starts: a progress log of what has been done and what comes next, a feature checklist that defines the scope, and a pointer to any startup script. Every later session begins by reading those files, which restores the project state without re-exploring the codebase. Before a session ends, it updates the progress log. The docs add one rule that keeps the log honest: work on one feature at a time, and mark a feature complete only after end-to-end verification confirms it works, not when the code is written.

That last rule generalizes. A memory is only useful if it is true, so write memories at the moment you have evidence, not at the moment you have an intention. "Tests pass on the payment refactor" is a memory. "Working on the payment refactor" is a status, and it should be overwritten, not appended.

The short version

Decide which memory types your agent needs, and keep short-term history and long-term memory in separate mechanisms. Store long-term memories as small typed records with a source and a last-used date. Enforce read and write permissions, per-user scoping, size caps and path validation in your handler, because the model cannot enforce them for you. Do not let untrusted tool output write into memory that steers future runs. Build update and delete paths from the start, and expire what nobody reads. The frameworks give you the storage plumbing; the judgment about what is worth remembering is still yours.

For a broader look at how memory, retrieval and tool use fit together in production LLM applications, AI Engineering by Chip Huyen has a chapter on RAG and agents that pairs well with the patterns above.

If you run models on your own machine, we keep a short list of local LLM gear — GPUs, Macs, RAM and storage — with plain-text Amazon links.