FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity

Key points

• Human demonstrations offer a scalable alternative to costly robot data collection for dexterous manipulation, but most human datasets lack force measurements.1
• Dexterous manipulation requires not only visual and proprioceptive understanding but also the ability to anticipate and regulate interaction forces.1
• We present FACT, a robot foundation model that cascades from prospective visual planning to force-aware refinement for dexterous manipulation.1
• Ablations show that force supervision in human pretraining data is crucial for downstream robot performance after fine-tuning, while visual track prediction and training with future-force prediction improve dexterous execution.1

Sources (1)

• [1]FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 08:21 PM

  To scale force-aware human-robot co-training, we introduce EgoForce, a large-scale human tactile dataset capturing force-rich manipulation experience, and combine it with robot demonstrations for pretraining.

  Human demonstrations offer a scalable alternative to costly robot data collection for dexterous manipulation, but most human datasets lack force measurements.

Extractive summary: sentences quoted from the sources.