Memory for AI agents: layers, dreaming and evals that don't fool you

Equipe MEYJE
6 min read
Isometric illustration: four stacked plates, from large dark ones at the bottom to a small bright one on top with a green outline, with thin arrows circling between the layers
On this page

An AI agent that talks to the same customer every day should get better over time. In practice, every session starts from scratch, and the usual workaround is to push everything into the context, which works for a few weeks and then starts to break down.

In this post we explain the layered memory that Anthropic described on November 6, 2026, the process it calls dreaming, and what the tests show. Then, drawing on an Anthropic post from November 13, we show how to evaluate a change like this without fooling yourself.

#Why dumping everything into the context doesn't scale

For Anthropic, agent memory is the knowledge a model gains from experience and keeps from one session to the next. Loading all of it into every session feels safe, but it has three problems:

  • Cost. Everything in the context is paid for in tokens and competes for space with the task.
  • Adherence. According to the post, longer instruction files are followed less faithfully.
  • Repeated mistakes. A wrong note that goes into every session contaminates all of them.

In a test with Pokémon Red, Sonnet 3.7 wrote 413 memory files across 225 folders and earned one badge. Fable 5, with nine files in two folders plus one top-level file, earned four. More is not better.

#Memory in four layers

The proposal is a stack, from the rawest layer to the most distilled:

  1. Transcripts. The full record of every session, accessed through search.
  2. Notes. Observations the model writes itself, such as a customer preference.
  3. Representations. Structures built from notes and logs, such as a wiki with linked pages.
  4. Always-loaded context. The small slice that goes into every session, and that needs to stay small.

Claude Code follows the same logic: at the start of each conversation, auto memory loads only the first 200 lines or the first 25 KB of MEMORY.md, whichever comes first. The file works as an index, and the details live in topic files that are opened when needed.

A generic example, for a customer service agent:

text
memory/
├── INDEX.md                # always loaded: short, one line per topic
├── notes/
│   ├── preferences.md      # "prefers short answers, no jargon"
│   └── corrections.md      # past mistakes and how they were fixed
├── wiki/
│   ├── products.md
│   └── processes.md        # pages linked to each other
└── transcripts/            # full history, for search only
    └── 2026-11-14.jsonl

The rule of thumb: the index holds only what the agent uses in almost every conversation, plus the path to everything else.

#What dreaming is

Anthropic takes its cue from the brain: the hippocampus records experiences quickly and, during rest, replays them to consolidate them in the neocortex. In an agent, dreaming is that consolidation step: the model reflects on its logs to fix mistakes and produce new notes and representations. It checks notes against the transcript, expands a wiki, promotes a note to the top, or moves details down and leaves just a link.

Pushing this work to the background keeps in-the-moment note-taking from distracting the model. The trade-off is that the results may not be available right away.

In Pokémon, the model mistakenly noted that the Pokémon Center was on the east side and, following that note, dropped back to an earlier stretch of the game. During dreaming, it compared the note with the transcript and fixed the error. With the corrected note, it reached Cerulean City. With the wrong note, or with no memory, it never did.

#What the numbers show

The numbers come from Anthropic, in its own tests:

  • CL-Bench. On a continual learning benchmark (Continual Learning Bench) built on database lookups, Opus 5.5 went from a mean reward of 0.655 without memory to 0.838 with memory.
  • Rigid rules backfire. With the rule "record only failures," Opus 5 and Opus 4.7 lost much of the gain. For stronger models, letting the model judge what to keep worked better: tell it what memory is for, not how to use it.
  • A large document base. On a corpus of about 350,000 files (roughly 420 million tokens), Opus 5 agents spent 128 sessions building a wiki through dreaming. Using the wiki alone, accuracy on 25 questions rose from 13% to 48% of graded criteria. With the wiki plus search it reached 61%, versus 53% with search alone, and saved about 20,000 tokens per question.

There are limits: dreaming costs compute, wikis need maintenance and don't always help, and transcript search may be the foundation of it all.

#How to evaluate a change like this without fooling yourself

The document base test used 25 questions: a good sign, not a guarantee for your agent. To know, you have to measure.

The November 13 post is about safety evaluations (evals, tests that measure how a model behaves). In our reading, several of its recommendations apply to any change to an agent:

  1. Decide on the analysis up front. This is preregistration: write down the main metric, the comparisons, the data slices and the threshold that counts as an improvement or a regression before you see the results.
  2. Use a large enough sample. Anthropic treats 100 items as the floor for a complex, multi-turn eval and 1,000 for a simpler, single-turn one. Run each case several times and report the spread.
  3. Compare in pairs. Run both versions on the same items and compute the confidence interval on the per-item differences.
  4. Calibrate the grader. If a model does the grading, experts label a sample without seeing its verdicts. Agreement should be corrected for chance, with statistics such as Cohen's kappa, and come close to how well the experts agree with each other.
  5. Show the uncertainty on every number. In multi-turn evals, confidence intervals can exceed ±10 percentage points. In the post, 98% versus 94% between two models was not a statistically significant difference.
  6. Change one thing at a time. For Fable 5.1, the appropriate response rate differed by up to 34 percentage points between the API and claude.ai, because claude.ai applies a system prompt.

For memory, the playbook looks like this: a fixed set of tasks that resemble the real ones, a metric and a minimum gain decided up front, versions with and without memory run on the same items with everything else equal, and a gain accepted only if the confidence interval of the difference sits entirely above zero.

#What this changes for people building agents

At MEYJE we build AI agents for clients, such as the RuralZap financial assistant on WhatsApp. Three lessons are now part of how we design them:

  1. Small, curated memory. What goes into every conversation fits in a short index. The rest lives in notes and pages, and the full history stays searchable.
  2. Say what it's for, not how to do it. Explain the goal, let the model judge what to keep, and set aside a periodic review, outside the conversation, to fix and reorganize.
  3. Measure before you believe. Without a metric decided up front, a large enough sample, a calibrated grader and confidence intervals, a nice-looking gain may be nothing but noise.

Good memory isn't about storing more. It's about helping the agent make fewer mistakes tomorrow. If you want to put an AI agent to work with your company's systems, talk to us.

Sources

  1. The components of agent memory (Anthropic, Nov 6, 2026)
  2. Building strong safety evaluations (Anthropic, Nov 13, 2026)
  3. How Claude remembers your project (Claude Code documentation)