MemDerive: A Diagnostic Benchmark for Agent Memory over Evolving Data
We present MemDerive, a diagnostic benchmark for agent memory over evolving data.
ProofPaper ↗
Key points
- Long-running agents rely on memory of their past interactions for task state, such as what has been completed and which conditions still hold.
- Such state is usually a derived state.
- It specifies each scenario symbolically, derives exact gold answers with Datalog, records the gold derivation DAG and the supporting events of every query, and renders the history in natural language with fictional entities and distractors.
- Our evaluation of four memory systems reveals failures that final-answer benchmarks hide.
Sources (1)
- [1]MemDerive: A Diagnostic Benchmark for Agent Memory over Evolving DataarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 04:24 PM
We present MemDerive, a diagnostic benchmark for agent memory over evolving data.
Long-running agents rely on memory of their past interactions for task state, such as what has been completed and which conditions still hold.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 18, 2026CoEvo-RSI: When Harnesses Evolve, Which Models Should Follow?
- Oct 18, 2026StopLoss: Teaching Agents to Predict Failure and Stop Early
- Oct 17, 2026HybridResearcher: Bridging the Structured-Unstructured Boundary for Deep Research Agents
- Oct 17, 2026HSPR: Recovering Software-Agent Benchmark Scores via Hierarchical Posterior Inference
- Oct 17, 2026Design Anything, Design at Scale
- Oct 16, 2026OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners