From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation

Key points

• Effective manipulation requires a policy to capture not only about the current scene geometry, but also about how that geometry will evolve under interaction.1
• Therefore, we propose a two-stage framework, which learns complementary predictive representations from multi-view video clips.1
• During pretraining, a frozen 3D foundation model provides dense geometric features that are compressed into predictive geometric representations and optimized to preserve scene structure and its temporal evolution.1
• Action supervision is then applied to geometric transitions to extract a complementary motor representation that emphasizes action-relevant transitions.1

Sources (1)

• [1]From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 01:08 PM

  Existing approaches either augment policies with geometric information from current observations or introduce future prediction as auxiliary supervision, leaving open how predictive geometry should be represented for action generation.

  Effective manipulation requires a policy to capture not only about the current scene geometry, but also about how that geometry will evolve under interaction.

Extractive summary: sentences quoted from the sources.