MemoryTax: The Bitter Lesson of Agent Memory

Key points

• We ask whether designed memory still helps once the model is strong enough to search for itself.1
• On one LoCoMo conversation (419 turns, 152 questions), we run four memory methods (full context, BM25, Mem0, Zep) and three coding agents (Pi, Codex, Claude Code), each with GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5, and record what every rollout costs.1
• Agent memory is a costly patch for a weak model: Mem0 beats full context only with GPT-4o-mini, and its build alone costs more than a whole full-context rollout.1
• Part of that lead comes from photo captions that only the agents could read: on the 88 questions whose evidence has no photo, Pi and full context differ by one answer.1

Sources (1)

• [1]MemoryTax: The Bitter Lesson of Agent Memory
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 02:46 PM

  Agent memory systems such as Mem0 and Zep decide what to keep before any question arrives; coding agents keep the raw history and let the model search it with its own tools.

  We ask whether designed memory still helps once the model is strong enough to search for itself.

Extractive summary: sentences quoted from the sources.