Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems
We propose an Interventional Memory Evaluation Protocol (IMEP) for vision-language-action (VLA) systems to address four questions: (1) Does memory provide a net benefit?
ProofPaper ↗
Key points
- When we say that a robot uses memory, does that memory actually play a role?
- (3) Do changes to memory alter task outcomes?
- At the retrieval interface, we compare target-fact substitutions with fact-preserving replacements, minimal numerical perturbations, and exact replicas, while assessing decision correctness separately from motor execution success.
- To examine when memory helps, we introduce ChainBench, where correct decisions require historical information unavailable in current observations.
Sources (1)
- [1]Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action SystemsarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 12:23 PM
We propose an Interventional Memory Evaluation Protocol (IMEP) for vision-language-action (VLA) systems to address four questions: (1) Does memory provide a net benefit?
When we say that a robot uses memory, does that memory actually play a role?
Extractive summary: sentences quoted from the sources.
Before this
- Oct 16, 2026NestedVLA: Learning to Consolidate and Generate Skills for Vision-Language-Action Models
- Oct 15, 2026TS-WAM: Temporal Latent Actions and World-Action Coupling for Vision-Language Robot Control
- Oct 15, 2026What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning
- Oct 14, 2026A Phenomenology of Neural Geometry in Vision-Language-Action Models
- Oct 14, 2026Causal Analysis of Robot Foundation Models Enables Efficiency Gains
- Oct 14, 2026AdaRSI: On-Demand Recursive Self-Improvement for Embodied Agents