Summary Memory Gets Worse the More You Use It. We Benchmarked 9 Agent Memory Architectures

The simplest way to give an LLM assistant long-term memory is a rolling summary. Every few turns, a model rewrites a bounded note from the old note plus the new messages, and the note goes into every prompt. It's what we shipped in aicoo, the assistant that lives in a team's workspace, and last month it took aicoo from answering about half of our questions about a month of chat to nine in ten.

A rolling summary has a structural flaw. The note has a fixed budget and the history doesn't. Once the note is full, every rewrite is also a compaction: the model has to decide, at write time, what to throw away, without knowing what anyone will ask later. A light user rarely fills the note. A heavy user fills it within weeks, and from then on almost every write loses something.

We measured how bad that gets. For the heaviest of four users (1,188 messages over 90 days), 110 of the note's 147 rewrites were compactions, and it scored 48 out of 100: exactly what no memory scores with the last 14 messages. Every design whose store grows with the history beat it on that user by 15 to 29 points, except one of ours.

We benchmarked nine architectures: the rolling summary we ran, a full-context baseline, tiered compaction, a Mem0-style fact store, a MEMORY.md-style index of notes, and three designs of our own.

Key takeaways

• A fixed-size summary has a fixed capacity. At 1,188 messages, a 6,000-character note was worth as much as the last 14 messages (48 and 48). Three in four of its rewrites were compactions. For our lightest user it was compacted once.
• Capacity, not cleverness, decided the heavy user. Every design whose store grows with the history beat the rolling summary on that user by 15 to 29 points (all significant), except our append-only ledger.
• Deciding what matters at read time is cheap. Keeping the summary and adding a log of dated facts, retrieved per message with BM25, gained 6.0 points overall (95% interval +1.0 to +11.9) and 15 on the heavy user, at the same input tokens per answer (11.2k). We shipped it.
• Full context wins only while the history fits. It scored highest (77.1) at 6.3 times the tokens. At our heaviest real user's pace, the same design passes a 1M-token window within a year, and even at 90 days it made up answers to questions about things never said more often than no memory did.
• Memory makes promise tracking worse. "Did I ever send Priya the doc?" No memory scored 58. Every memory design scored lower.
• Your memory benchmark may be measuring your infrastructure. Provider routing silently broke JSON mode, the grader lacked context, and malformed tool calls were scored as answers. Each moved results by about as much as the differences between designs.

How aicoo's memory works, and what's wrong with it

Before each reply in its private chat with a person, aicoo reads two notes. SOUL is a default persona plus whatever the person asked for ("call me Kai", "no bullet points"). MEMORY has sections for who they are, what they've asked for, what to keep, and what's been happening lately, plus a list of their saved preferences. Every 8 new turns, at least 3 hours since the last write, a small model rewrites MEMORY in full, up to 6,000 characters. Past that, it compacts.

That design has four problems:

1. A full rewrite is lossy. The model has to copy the old note back out, and it paraphrases, merges and drops as it goes. The same mechanism once rewrote aicoo's own persona: the model compressed the default SOUL along with the user's notes. We've since changed SOUL to store only what the person asked for.
2. A cap means compaction, and compaction means loss. A light user never hits the cap. A heavy user hits it on nearly every rewrite.
3. No provenance. If a line in the note is wrong, there's no way back to what was actually said.
4. It only learns from one place. The private chat feeds it; team chats, documents and meetings don't.

The mainstream designs

design	examples	how it stores and retrieves	strong at	weak at	here
rolling summary	MemoryBank; recursive summarization; what we ran	one note, rewritten every batch; always in the prompt	simple; profile and current state	capped, so it compacts and drops facts; rewrites drift	61.6
raw recent + compressed past	context compaction in coding agents; LangChain's summary buffer	recent days verbatim, older days in coarser summaries; all in the prompt	facts that changed; recent detail	summaries eat old detail; more tokens	67.6
index + topic notes, read on demand	Claude Code's MEMORY.md; Anthropic's memory tool	a one-page index always loaded; the model decides which notes to open	dates; how to address them	only works if the model chooses to read; its two runs differed by 9 points	61.0 (66.7 with recall)
atomic facts + add/update/delete	Mem0	one fact per line, reconciled with ADD / UPDATE / DELETE; top-k by the query	cheapest (5.3k tokens per answer); applying preferences	no standing profile, so a nickname or tone rule isn't retrieved	62.6
OS-style tiers, agent-managed	MemGPT / Letta	core blocks in context, archive and history searchable, edited by the agent with tools	flexible	depends on the agent's discipline; extra calls	≈ index + notes
temporal knowledge graph	Zep / Graphiti	entities and relations with validity intervals; hybrid search plus graph traversal	time; versions	heavy infrastructure	not tested
memory stream + reflection	Generative Agents	a timestamped stream with periodic reflections; retrieved by recency × importance × relevance	abstract understanding	costly; unclear for factual questions	not tested
full context	long-context models	store nothing; send everything	most accurate at 90 days	6× the tokens and growing; makes up more answers	77.1

"Here" is the score on this benchmark, out of 100.

They differ on three axes: whether the store is bounded, whether it's rewritten or appended to, and whether what reaches the prompt is fixed or chosen per question. A rolling summary is bounded, rewritten and fixed, which is the combination that fails a heavy user. We tested five of the eight families directly, plus three designs of our own. MemGPT-style self-managed tiers behave much like an index of notes the model opens by itself, so we tested that instead. Temporal knowledge graphs (Zep, Graphiti) and reflection-based memory streams (Generative Agents) need more infrastructure than this round allowed.

The benchmark

We wrote four synthetic people and generated 90 days of their private chat with aicoo, from August 3 to October 31. On the morning of November 1, each asked 45 questions.

person	who	messages in 90 days
mo	founder of a six-person health startup, Chinese	711
dan	backend engineer, English, terse	837
yuan	third-year PhD student, Chinese with English terms	921
kai	head of product, mixes Chinese and English in every sentence	1,188

The questions come in eight kinds, partly modelled on LongMemEval:

kind	per person	example
an old fact	10	"What was Q3's actual revenue? Grace wants the final number."
a fact that changed	6	"What's the billing cutover date?" (set, then moved twice)
when something happened	5	"Ravi's integration was due Sep 15. What day did it actually go live?"
counting across sessions	5	"How many data engineering candidates did we interview?"
a promise's status	5	"Did I ever send Priya the timeline doc?"
how to treat them	4	"morning" (weeks after asking to be called by a nickname)
a work preference	4	"Draft a PR description" (PR descriptions end with a Risk section)
something never said	6	"When is Tom's vacation?" (Lena's vacation was mentioned, Tom's never was)

A stronger model designed 86 to 118 events per person and the questions about them; a smaller one wrote the chat a few days at a time. Every planted fact had to appear at its day and time with its numbers intact. Each chat also carries a dozen or so decoys with nearby values, such as another budget or another deadline, so a memory that blurs numbers gets confused. Any message that would have answered a never-said question was removed.

The stronger model then audited every question against the chat as actually generated. It dropped 5 as ambiguous and corrected the expected answer on 16. Scoring only the 159 questions it left untouched gives the same ranking.

Nine architectures

Every design writes on aicoo's schedule: after 8 new turns, 4 of them from the person, at least 3 hours since the last write. Every design can also search the raw chat, as aicoo can today. The rolling summary uses aicoo's production rewrite prompt word for word. What differs is what's stored, whether the store is bounded, and what reaches the prompt.

design	after	store	in the prompt	retrieval
no memory		none	the last 14 messages	chat search
full context	long-context models	every message	every message
rolling summary (what we ran)		one note, capped at 6,000 characters	the note	chat search
tiered compaction	context compaction in coding agents	the last 7 days raw; older days as daily, weekly, then monthly summaries	all of it	chat search
index + topic notes	Claude Code's MEMORY.md	an index plus topic notes	the index and an "about them" note	open or search notes
fact store	Mem0	atomic dated facts	the 12 facts closest to the message	search facts
append-only ledger (ours)	event sourcing	an event log	computed views: about them, their asks, open promises, the last 10 days	search the log, open the source message
summary + fact recall (ours)	rolling summary + fact store	the capped note plus a fact log	the note plus the 12 facts closest to the message	search facts
index + notes + recall (ours)	index + topic notes + recall	an index plus topic notes	the index plus the 10 note lines closest to the message	open or search notes

DeepSeek-V4-Flash wrote every memory and answered every question, with up to 5 tool calls per answer. DeepSeek-V4-Pro graded each answer against the expected one. We ran the whole benchmark twice, independently, pooled the two runs per question, and computed 95% intervals with a paired bootstrap over questions.

Results

Any memory is worth at least 10 points over none. Beyond that, only two practical designs beat the rolling summary overall with an interval clear of zero: summary + fact recall and tiered compaction, each by 6 points. The rest sit within noise of it. On the heavy user, the picture changes completely (right panel): nearly everything beats it.

The two winners win on different questions. Tiered compaction wins on facts that changed, because the latest value is usually still inside the raw 7-day window. Fact recall wins on old facts, dates and counting, because each fact is a dated atom that is never re-summarized, so it survives no matter how long the history gets. The fact store and the ledger apply work preferences as well as the summary does. Counting and promises defeat everyone.

Capacity is the whole story on the heavy user

For mo, the lightest user, the rolling summary was the second-best design. For kai, the heaviest, it scored 48, exactly what no memory scores.

The two designs with a fixed-size store converged. No memory keeps the last 14 messages. The rolling summary keeps 6,000 characters. On kai they scored the same, because once the note is full, it holds about as much useful history as a short window does.

Kai's note reached the cap within weeks and never came back down. Of its 147 rewrites, 110 also had to compact it. Mo's note was compacted once.

Why does compaction lose exactly the facts people ask about? A compaction ranks what to keep by how important it looks at write time. Two months later, Q3's final revenue looks like stale detail: the conversation has moved on, so it's the first thing cut. The question that needs it hasn't been asked yet. Designs that retrieve defer that choice to read time, when the question is known. The rule we take from this: don't decide what's important until you know the question.

Our heaviest real users exchange nearly as many messages with aicoo in two weeks as kai did in three months.

What it costs

Fact recall is free per answer. The twelve facts cost about as many tokens as they save, because the model searches the raw chat less often. Writing them costs two extra small-model calls per rewrite (extract, then reconcile).

Writing the memory, for four people over 90 days on Flash: the rolling summary $0.17, the index of notes $0.14, the ledger $0.05, the fact store $0.04, tiered compaction $0.02. The summary is the most expensive writer because it regenerates all 6,000 characters on every write.

Tiered compaction matched fact recall overall at 1.7 times the input tokens per answer.

Full context wins only while the history fits

Full context scored 77.1, nearly ten points above anything else. That's a result about 90 days, not about memory.

At 90 days the whole history is about 71,000 tokens per answer: roughly 73 tokens per message, plus the prompt. That fits comfortably in a modern context window, and at that length the model still finds what it needs (96 on old facts). It's the regime where full context is supposed to win. Project it forward:

At kai's pace, a year is about 4,800 messages and 357k tokens per answer, past a 128k window. At our heaviest real user's pace (about 1,000 messages in two weeks), a year is about 28,000 messages and 2.1M tokens, past even a 1M window. Three things break on the way:

1. It stops fitting. Past the window you have to cut something, and cutting the oldest messages turns full context into a long recent window: a fixed-capacity design again, the kind we just watched collapse.
2. It degrades before it stops fitting. Models use long inputs unevenly. Facts in the middle of a long prompt get missed more often (Liu et al., "Lost in the Middle"), and accuracy falls as inputs grow even on simple tasks (Chroma, "Context Rot"). We already saw an early sign at 90 days: on questions about things never said, full context answered correctly 75% of the time against 81% for no memory. Every message is in the prompt, nothing marks which ones matter, and a plausible date is always nearby.
3. Every answer pays for the whole history. Input tokens grow linearly with every message the person sends. Prompt caching cuts the price of the repeated prefix, but it doesn't raise the window or stop the dilution.

Full context doesn't lose information until it overflows. It drowns in it. The way out is to keep what full context gets right and add what retrieval gets right: store every message, so the store grows with the history, and choose what enters the prompt per question. That's the direction of the design we shipped and of what we test next.

Memory makes promise tracking worse

Asked whether a promise was kept ("Did I ever send Priya the timeline doc?"), no memory scored 58, about the same as full context (59), and higher than every memory design. With no memory, the model searched the chat, found the promise, and found nothing after it. Memories record that a promise was mentioned, and a later rewrite describes it as handled. The summary turns an open loop into a closed one.

Our append-only ledger was built for exactly this, with a computed view of every promise not yet marked done. It scored lowest of all, 38. The cause was mundane: the extractor almost never labelled "I'll send it" or "remind me" as a promise, finding 3 to 21 per person in 90 days, so the open-promises view was nearly empty. The idea hasn't been tested yet; its extractor has.

Counting failed everywhere too. No design scored above 56. Memories are built to keep the latest state of something. Counting needs every event.

Your memory benchmark may be measuring your infrastructure

Building nine memories took less time than getting numbers we trust. Three bugs in our measurement each moved results by several points, about the size of the effects we were trying to measure:

1. Provider routing broke the protocol. Our gateway routes each call to whichever upstream provider is available. Two providers ignore JSON mode on long prompts (8k to 60k tokens) and answer in plain text, and a plain-text reply can't call a tool. Short prompts work fine on both, so a smoke test passes. Two designs we tested in the evening answered in plain text 67 times out of 175; designs tested earlier, between 1 and 34 times. One design, rerun later the same evening, went from 5% plain text to 46%. Pinned to a single provider: 1 plain-text answer in 1,620.
2. The grader couldn't see enough. The first grader saw only the planted facts, so when an answer cited a true detail from elsewhere in the chat, it marked it as made up. Every negative verdict got a second look with the surrounding chat, which flipped 8 to 19 verdicts per design.
3. Malformed tool calls were scored as answers. Between 2% and 10% of answers were a tool call in a slightly different JSON shape, which our harness took as the final reply. That penalized the designs with more tools most.

None of these is about memory, and each one alone could have reordered the middle of the table. Before you compare designs, report each design's infrastructure failure rate: unparseable replies, which provider served each call, and how often the grader lacked context. That check is what found all three.

Six designs we think are worth building

aicoo differs from a general chat assistant in two ways. The person's to-dos, documents and meetings sit right next to it, and its job is to run operations, so the memory that matters most is who promised what. That suggests designs the mainstream ones don't cover:

1. Summary + per-message fact recall. Tested, shipped. Keep the profile; add dated atomic facts; recall the 12 closest per message. +6.0 overall, +15 on the heavy user, same tokens.
2. Append-only ledger with computed views. Tested, didn't win. Never rewrite; every entry has a type and a link to the message it came from; a changed value appends a new entry pointing at the old one; the profile, asks and open promises are computed by code. It tied with the summary because promise extraction failed. Fix the extractor, then test again.
3. Don't memorize what you can look up. Not tested. To-dos, documents and meetings are already sources of truth. Memory should hold pointers to them, not copies. In one real memory we inspected, most of the kept lines were copies of to-dos the person had created, taking up space under the cap.
4. Repeated events as a series. Not tested. "Moved the dentist again", "interviewed another candidate": store these as one series with a count and a list of dates, so counting has something to count.
5. A promise list that closes only on evidence. Not tested. Promises get their own list, and an item closes only on an explicit "sent" or "done", never because a summary says it was handled.
6. Consolidation while the user is away. Not tested. Recompute the profile and views offline, off the reply path, without needing the person's page to be open. Letta calls this sleep-time compute.

What we shipped

The summary stays. After each rewrite, a second pass runs over the chat since the last fact pass:

1. Extract. A small model pulls out atomic facts, each starting with the date it was said: "Oct 7: the launch moved to Nov 10." Who, what, numbers, decisions, plans, changes, promises. A preference about how aicoo should treat them counts only if they asked for it themselves.
2. Reconcile. A second call compares the candidates with the closest facts already kept and returns ADD, UPDATE, DELETE or NONE for each, the way Mem0 does. Candidates are numbered, so an ADD that omits its text falls back to the candidate it points at, and a reply with no decisions saves nothing.
3. Recall. Before each reply, the 12 facts closest to the person's message are added to the prompt. Retrieval is BM25 over Latin words plus Chinese unigrams and bigrams, with numbers normalized so "$5,000" matches "5000". A search_memory tool lets the model look further.

A few engineering details mattered more than they look:

• At-least-once processing. The pass keeps its own cursor and advances it only after a durable write that goes straight to the database, not through the page's write queue, which swallows failures. A failed save leaves the cursor where it was, and the next pass rereads those messages.
• No migration. The fact log sits beside the summary, not inside it, so the server function that stores the summary never touches it.
• Bounded. The log keeps at most 500 facts; past that, the least recently touched go first.
• Private. search_memory is refused in team chats, so a teammate can't get aicoo to read someone's facts into a shared thread.

Limits

• The people are synthetic. Real chats are messier, and our heaviest real users are heavier than kai.
• Four people. The volume effect rests on the mechanism (1 compaction for mo, 110 for kai) and on kai's score, not on a curve fitted to four points.
• 90 days. Full context hasn't hit its limits here; the scaling section is a projection from measured tokens per message, not a measurement at a year.
• One model. DeepSeek-V4-Flash wrote and answered; aicoo runs on a different model in production. We expect the ranking to hold better than the absolute numbers.
• A simplified prompt, not the live product's.
• A model grader, not calibrated against human grading for this benchmark.
• One implementation per design, with BM25, not embeddings. A better-prompted ledger or index might do better.
• About 44 questions per person, so single-person scores carry roughly ±7 points of noise. The differences on kai are significant; finer differences among the middle designs are not.

Next

1. Stop compacting. When the summary is full, move the overflow into the fact log instead of discarding it.
2. Series, not facts, for counting. Track repeated events with a count and a list of dates.
3. Promises as their own list that closes only on an explicit "sent" or "done", then rerun the ledger with promises actually extracted.
4. Raw recent days plus recalled facts. One wins on facts that changed, the other on old ones.
5. A year, not 90 days. Rerun the leaders, and full context, on longer histories to find where full context falls behind.
6. Rerun the leaders on aicoo's production model.

The benchmark cost $4.80 in model calls, 19,017 of them across both runs. The harness, the four chats, every answer and every verdict are kept. If you want to check our numbers or run your own memory design against the same benchmark, get in touch.