FlowVLA: Enabling World Dynamics Understanding through Pluggable 3D Flow Learning

Key points

• Recent advances in 3D reconstruction enable the extraction of temporally consistent 3D motion from RGB observations; however, current Vision-Language-Action (VLA) models still predominantly rely on 2D image-text priors.1
• This over-reliance on static 2D visual cues limits the capacity of VLA models in achieving robust robot manipulation.1
• Instead of requiring explicit 3D sensor inputs or altering the inference-time architecture, our method is designed to utilize temporally aligned 3D flow only during training.1
• To bridge scene-level dynamics with robotic manipulation, we propose a modular 3D flow learning framework with dual-level flow supervision.1

Sources (1)

• [1]FlowVLA: Enabling World Dynamics Understanding through Pluggable 3D Flow Learning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 04:34 AM

  To address these limitations, we introduce FlowVLA, a reconstruction-guided, pluggable 3D flow learning framework aimed at empowering VLA models with an inherent understanding of action-relevant dynamics.

  Recent advances in 3D reconstruction enable the extraction of temporally consistent 3D motion from RGB observations; however, current Vision-Language-Action (VLA) models still predominantly rely on 2D image-text priors.

Extractive summary: sentences quoted from the sources.