GazeAct: A Human-Gaze-Annotated Manipulation Dataset for Attention-Regularized Policies

Key points

• Vision-language-action (VLA) policies must ground language instructions in the right objects, contact points, and target locations for reliable manipulation.1
• Using screen-based teleoperation, we record gaze directly on the robot camera feed viewed by the operator, yielding 12000 demonstrations across 40 tasks and five domains with 300 Hz gaze synchronized with images, proprioception, and actions.1
• We then use these gaze-derived targets to regularize language-conditioned visual attention during training while keeping inference gaze-free.1
• The gains transfer to LIBERO-Spatial (57.9% to 69.6%) and different VLA backbones.These results show that gaze collected in the robot observation space can provide effective training-time targets for improving visual grounding in VLA manipulation.1

Sources (1)

• [1]GazeAct: A Human-Gaze-Annotated Manipulation Dataset for Attention-Regularized Policies
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 07:07 AM

  We introduce GazeAct, a gaze-equipped real-robot manipulation dataset and training framework for multi-view VLA policies.

  Vision-language-action (VLA) policies must ground language instructions in the right objects, contact points, and target locations for reliable manipulation.

Extractive summary: sentences quoted from the sources.