Three benchmarks you will see cited
When a memory product claims it is “state of the art”, it almost always refers to one of three public benchmarks. They test different things.
| Benchmark | Paper | What it contains | What it tests |
|---|---|---|---|
| LoCoMo | Maharana et al., February 2024 | Very long conversations, about 300 turns and 9,000 tokens on average, over up to 35 sessions | Question answering, event summarization, multi-modal dialogue generation |
| LongMemEval | Wu et al., ICLR 2025 | 500 curated questions embedded in long user-assistant chat histories | Information extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention |
| MemoryAgentBench | Hu, Wang and McAuley, first posted July 2025 | Long-context datasets turned into incremental multi-turn input, plus new datasets | Accurate retrieval, test-time learning, long-range understanding, selective forgetting |
LoCoMo answers the question in its title: how well do agents remember very long conversations? Earlier dialogue work rarely went beyond five sessions. The authors found that models struggle with long-range temporal and causal links, and that long-context models and retrieval help but stay well below human performance. Many memory papers, including A-Mem and MemoryOS, report LoCoMo results.
LongMemEval is built around the abilities a chat assistant needs. Two of them are easy to forget: knowledge updates (the user changed jobs, so the old answer is wrong now) and abstention (saying “I don’t know” when the history never mentioned it). The authors report about a 30% drop in accuracy for commercial assistants and long-context models on information kept over sustained interactions. They also split memory design into three stages, indexing, retrieval and reading, which is a useful way to locate where a system fails.
MemoryAgentBench changes the format. Instead of handing over one long history, it gives the agent information piece by piece, as an agent meets it in real work. It adds test-time learning (picking up a new skill or rule from the conversation) and selective forgetting (dropping facts that were overwritten). Its conclusion is sober: from plain long context to RAG to agents with external memory modules, no current method masters all four.
The misunderstanding: one score says which memory is best
A single LoCoMo number hides a lot. Results move with the model behind the memory, the prompt, the scoring metric (F1, BLEU or an LLM as judge) and which question categories are included. Some categories are tricky to score at all. When vendors publish their own runs, set-ups differ, so two numbers from two blogs are rarely comparable. Use public benchmarks to shortlist, not to decide.
Also note what none of them covers: several people writing to one memory, knowledge that changes because a colleague changed it, and permissions. Their results do not establish how a shared team memory handles permissions and knowledge contributed by multiple users.
How to test memory on your own work
- Collect 30 to 50 real questions your team asks AI tools, each with a known correct answer and the source that proves it.
- Include updates. Add questions whose answer changed recently, such as a new price or a replaced process. Old answers should lose.
- Include questions with no answer. The right response is “I don’t know”, not a guess. This is LongMemEval’s abstention test in miniature.
- Spread the facts over time. Feed the knowledge in over several sessions or days, the MemoryAgentBench way, rather than in one paste.
- Score answer and source. A correct answer from the wrong source is luck. Check which memory the answer used.
- Rerun after every change to the memory system, the model or the prompts, and keep the scores.
What this means for a team
Whether memory works shows up as fewer re-explanations and fewer repeated mistakes, not as a benchmark score. A small evaluation set of your own makes that visible, and it keeps working when you switch model or tool. Background on the memory types these tests probe: What are the types of memory in AI agents?, and an example of a system scored on LoCoMo: What is A-Mem?
Where Bombos fits
Bombos gives every AI tool your team uses one shared memory with team knowledge, learned rules and what already sits in mail, documents and repos. Because each answer shows which knowledge it rests on, you can run exactly the kind of test above: check the answer and its source, question by question. One person can try the plugin first for their own agents.
Related questions
What is LoCoMo?
LoCoMo is the dataset from the paper "Evaluating Very Long-Term Conversational Memory of LLM Agents" (Maharana et al., February 2024). Each conversation has about 300 turns and 9,000 tokens on average, over up to 35 sessions, with tasks for question answering, event summarization and multi-modal dialogue generation.
Can I trust a vendor's LoCoMo score?
Treat it as a hint, not proof. Scores depend on the model, the prompt, the metric and the question categories included, and vendors run the benchmark themselves. Compare only results run the same way, and check them on your own data.
Is a long context window enough to pass these benchmarks?
No. The LoCoMo paper found that long-context models and RAG help but still lag well behind humans, and LongMemEval reports about a 30% accuracy drop for commercial assistants and long-context models on information kept across sustained interactions.
What does MemoryAgentBench add?
It feeds information in step by step, the way an agent receives it over time, instead of all at once. It tests accurate retrieval, test-time learning, long-range understanding and selective forgetting, and found that current methods do not master all four.
