How do you give an LLM long-term memory?

Short answer

A language model forgets everything between calls, so long-term memory always lives outside the model. The application saves what matters to a store (memory files, a database, a vector index or a graph) and puts the relevant part back into the prompt next time. You can build that with a memory tool such as Anthropic's, a framework store such as LangGraph's, or a memory service such as Mem0.

Why the model itself cannot remember

A large language model is stateless. Its weights are fixed after training, and each request only sees what is in its context window. So “long-term memory” for an LLM is always an engineering choice around the model: something writes information to a store, and something reads the right part back into the prompt later. The design questions are where you store it, who decides what gets written, and how the right memory comes back at the right moment.

Four ways to build it

Approach How it works Good for Watch out for
1. Memory files The agent reads a folder of notes at the start and edits them as it works One agent, project context, progress logs Files grow messy without rules for cleanup
2. Framework store A key-value or document store with namespaces, as in LangGraph Developers who already use a framework You still decide what to write and when
3. Memory service A service extracts facts from conversations and decides to add, update or delete Chat assistants with many users Extraction is an LLM judgment that can be wrong
4. Self-editing memory The agent keeps memory blocks in its prompt and edits them with tools, as in Letta Long-running agents with a stable persona or task The agent can overwrite something important

Memory files. Anthropic’s memory tool (memory_20250818) gives Claude commands to view, create, edit, insert, delete and rename files under /memories. It is client-side: your application runs the operations against storage you choose, such as a per-user folder or a database. The docs recommend capping file sizes, deleting files that have not been used for a long time, and guarding against path traversal.

Framework store. LangGraph separates short-term memory (the current thread, kept in state) from long-term memory, which it stores “as JSON documents in a store” under custom namespaces, so any thread can recall it. It describes semantic memory (facts), episodic memory (past events, often as few-shot examples) and procedural memory (instructions). You can keep one profile document per user or a collection of many small memories.

Memory service. The Mem0 paper describes a two-phase pipeline. For each new message pair, an LLM extracts candidate facts using a conversation summary and the last 10 messages. Each fact is then compared with the 10 most similar existing memories, and the LLM chooses ADD, UPDATE, DELETE or NOOP.

Self-editing memory. In Letta, memory is organized into blocks that are pinned to the system prompt and that “the agent can modify” through tools. All messages are persisted in a database, so they remain retrievable after they leave the context window.

The step most people skip: deciding when to write

LangGraph names the trade-off clearly. Writing memory “in the hot path”, during the conversation, makes it available at once but adds latency and gives the agent one more job. Writing in the background, as a separate task, removes the delay but means you must decide how often and on what trigger. Whatever you choose, plan how memories get updated and removed, not only added. A memory that only grows becomes a pile of contradictions. More on that in How should an AI agent manage its memory?

The misunderstanding: long-term memory is just RAG over old chats

Retrieving old conversation chunks is one ingredient, but it is not memory in the useful sense. Raw transcripts repeat themselves, contain things that were later corrected, and rarely say which version is current. Good long-term memory distills: it stores the fact, the decision or the lesson, and replaces it when it changes. See Is RAG the same as AI memory? for the difference in detail, and What are the types of memory in AI agents? for the vocabulary.

What this means for a team

Each approach above is usually scoped to one agent, one user or one application. In a team, people use several tools, and each tool keeps its own memory, if any. What one colleague taught Claude does not reach another colleague’s Copilot or Codex. If the goal is that the team’s AI learns once and remembers everywhere, the store has to sit outside any single tool, with rules for who may change shared memories and a history of what changed. Before you trust a memory system, test it on your own questions: How do you evaluate LLM agent memory?

Where Bombos fits

Bombos is long-term memory that sits outside any one model or tool: a plugin and shared memory layer underneath Claude, Claude Code, ChatGPT, Copilot, Grok, Codex, Cursor and your own agents. It learns how your team works, draws on what already sits in mail, documents and repos, and shows with every answer what it rests on. It runs in the cloud in Europe or standalone on your own server, and one person can try it first with their own agents.

Related questions

Can I give an LLM long-term memory by fine-tuning it?

Fine-tuning changes how a model behaves, but it is a slow and costly way to store facts that change. Most long-term memory for agents is stored outside the model and retrieved at run time, so it can be read, corrected and deleted.

Is a bigger context window the same as long-term memory?

No. A context window holds one conversation or task. When the session ends, nothing carries over unless the application saved it somewhere and loads it again.

What is the simplest way to start?

A small set of memory files the agent reads at the start of every session and updates at the end. Anthropic's memory tool works this way: Claude reads and writes files under /memories, and your application decides where those files are stored.

Where do the memories live with the Claude memory tool?

In your own application. The tool is client-side: Claude requests file operations, your code runs them against storage you control.

Sources

  1. Memory tool (Claude Platform docs)
  2. Memory overview (LangChain / LangGraph docs)
  3. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (arXiv 2504.19413)
  4. Letta docs: agent memory
The first half hour · free

What does your AI forget?

  1. 01You tell us which AI you use and where it keeps starting over.
  2. 02We show what a shared memory changes there, on your own examples.
  3. 03You keep the plan, with or without Bombos.
Piet Baudoin
Piet BaudoinYou talk to Piet, from the Bombos teamDutch team from Utrecht and Nijmegen · data in Europe
We reply the same working day.