VitaBench: Memory Lifecycles Beyond Retrieval
Long-term memory benchmarks for language agents primarily evaluate whether information from earlier interactions can be retrieved and used when relevant.
ProofPaper ↗
Key points
- Persistent memory systems face an additional problem before future use is known: deciding whether a memory should remain exact, be compressed, consolidated, or deleted.
- Existing benchmarks capture long-range retrieval, temporal reasoning, evidence reuse, and incremental interaction, but do not directly evaluate these earlier representation decisions.
- We formalize memory lifecycle evaluation and introduce VitaBench, a benchmark that makes lifecycle decisions measurable through repeated memory reviews and full-horizon counterfactual evaluation.
- We characterize lifecycle decision structure with three diagnostics: lifecycle headroom, measuring the advantage of adaptive choices over the best fixed strategy; horizon reversal, measuring when short-horizon decisions change under delayed consequences; and optimal-action diversity, measuring whether different memory states favor different lifecycle operations.
Sources (1)
- [1]VitaBench: Memory Lifecycles Beyond RetrievalarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 08:51 AM
Long-term memory benchmarks for language agents primarily evaluate whether information from earlier interactions can be retrieved and used when relevant.
Persistent memory systems face an additional problem before future use is known: deciding whether a memory should remain exact, be compressed, consolidated, or deleted.
Extractive summary: sentences quoted from the sources.