LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models

Key points

• Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, calibrations drift, controllers run at different rates, and end-effector pads wear down.1
• We evaluate nine recent vision-language-action models using 500 episodes per condition and substantiate each finding with rigorous statistical tests.1
• Performance degradation varies substantially across perturbation axes and action interfaces, and training on visual perturbations does not mitigate faults that leave camera images unchanged.1
• We release the benchmark together with manifests specifying the fixed perturbation parameters for each episode.1

Sources (1)

• [1]LIBERO-RIG: A Hardware-Setup Robustness Benchmark for Vision-Language-Action Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:57 PM

  We present LIBERO-RIG, a benchmark that perturbs only the robot's hardware and setup while holding every LIBERO task fixed.

  Robot policies are expected to generalize across deployment environments, including variations in the robot itself: cameras are bumped and remounted, calibrations drift, controllers run at different rates, and end-effector pads wear down.

Extractive summary: sentences quoted from the sources.