PINT: A Unified Model with Shared Predictive Intent for Robot Manipulation

Key points

• Generalist robot manipulation can benefit from pretrained vision-language models (VLMs) for semantic reasoning and video generation models (VGMs) for world modeling.1
• PINT inserts learnable Intent Tokens into the VLM sequence to form this representation, conditioning both the VGM and an Action Expert (AE).1
• Training uses two stages: Stage 1 freezes the VGM so that future video prediction shapes Predictive Intent with task-relevant future dynamics, while Stage 2 fixes the learned Intent and adapts the VGM and AE.1
• Without embodied pretraining, PINT achieves strong in-distribution (ID) performance and substantially improves out-of-distribution (OOD) generalization, reaching 15.5% on RoboTwin 2.0-Hard compared with 1.8% for Fast-WAM and 3.2% for StarVLA, and 79.5% on LIBERO-Plus compared with 51.5% and 75.0%, respectively.1

Sources (2)

• [1]PINT: A Unified Model with Shared Predictive Intent for Robot Manipulation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 08:12 AM

  We introduce PINT, a unified model that coordinates semantic reasoning, world modeling, and action generation through a shared chunk-level Predictive Intent while preserving expert-specific pathways.

  Generalist robot manipulation can benefit from pretrained vision-language models (VLMs) for semantic reasoning and video generation models (VGMs) for world modeling.

• [2]PINT: A Unified Model with Shared Predictive Intent for Robot Manipulation
  Hugging Face Daily Papers · Oct 12, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.