VLA-GAUGE: Systematically Evaluating Generalization and Robustness in Vision-Language-Action Models

Key points

• We introduce VLA-GAUGE, a diagnostic framework that unifies both evaluations under a shared executable task representation.1
• We evaluate 12 VLAs spanning different model sizes and action-modeling paradigms under a unified protocol.1
• We find that (1) The mean Zero-Shot Score on VLA-GAUGE is only 22.3%, with SimpleVLA-RL scoring 8.0% and ranking second to last among the evaluated models.1
• (3) Diagnosis-driven fine-tuning improves $π{0.5}$'s zero-shot generalization and robustness under a selected perturbation combination on held-out scenarios by 72.4 and 31.8 scoring points, respectively; the latter exceeds the 10.9-point gain from LIBERO-Plus single-factor fine-tuning despite fewer successful training trajectories.1

Sources (1)

• [1]VLA-GAUGE: Systematically Evaluating Generalization and Robustness in Vision-Language-Action Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 10:48 AM

  Vision-Language-Action models (VLAs) have achieved high success rates on familiar manipulation tasks, yet it remains unclear whether such scores reflect genuine zero-shot generalization and perturbation robustness.

  We introduce VLA-GAUGE, a diagnostic framework that unifies both evaluations under a shared executable task representation.

Extractive summary: sentences quoted from the sources.