Learning the 4D World through Point Trajectory Forecasting

Key points

• Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene.1
• We extend a pretrained world model with camera-conditioned multi-view processing that integrates complementary observations across cameras, and add a trajectory head that predicts how physical points move through time.1
• To support training and evaluation, we build VPM-Data, a simulation corpus pairing synchronized multi-view videos with persistent point trajectories, and VP-WorldBench, a forecasting benchmark spanning simulated and real environments.1
• On VP-WorldBench and existing 4D generation benchmarks, point supervision improves both video fidelity and 3D motion accuracy over a video-only counterpart, and in simulated and real-robot experiments, it also raises the manipulation success of the action-enabled model.1

Sources (2)

• [1]Learning the 4D World through Point Trajectory Forecasting
  Hugging Face Daily Papers · Oct 20, 12:00 AM

  To this end, we introduce Video-Point Model (VPM), a world model that conditions on multi-view video histories and a language instruction to jointly forecast future videos from arbitrary viewpoints and the 3D trajectories of persistent scene points.

  Reliable multi-view forecasting requires videos from different cameras to describe the same evolving 3D scene.

• [2]Learning the 4D World through Point Trajectory Forecasting
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 05:45 PM · same content

Extractive summary: sentences quoted from the sources.