ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction

Key points

• Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force.1
• Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects.1
• We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution.1
• On held-out human data, ACTS improves scene-view prediction by 0.7-2.0 dB PSNR over the stronger of two visuo-tactile world models trained from scratch on React.1

Sources (1)

• [1]ACTS: Learning Visuo-Tactile World Models from Human Physical Interaction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 10:33 PM

  We present ACTS, a visuo-tactile world model trained only on human interaction data.

  Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force.

Extractive summary: sentences quoted from the sources.