Persistent Planar Memory for Video World Models

Key points

• Video world models that generate and explore environments at scale need a persistent 3D memory of what they have produced.1
• Plane primitives offer a natural candidate: a compact set of oriented planes captures the dominant structural elements (e.g., walls, floors, facades) of indoor and outdoor scenes, while requiring only 7 parameters per primitive.1
• PPMem augments a video diffusion transformer with a planar prediction head and a Depth Token Fusion (DTF) module: each autoregressive chunk jointly produces RGB video and planar geometry, which merges into a persistent plane memory that conditions subsequent chunks via rendered depth.1
• On DL3DV and RealEstate10K, PPMem outperforms prior 3D-grounded world models in video quality while jointly producing metric-scale depth, surface normals, and exportable planar geometry, sustaining coherence across hundreds of frames under large viewpoint changes.1

Sources (2)

• [1]Persistent Planar Memory for Video World Models
  Hugging Face Daily Papers · Oct 16, 12:00 AM

  We present PPMem, a video world model that adopts plane primitives as its persistent 3D memory.

  Video world models that generate and explore environments at scale need a persistent 3D memory of what they have produced.

• [2]Persistent Planar Memory for Video World Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 05:21 PM · same content

Extractive summary: sentences quoted from the sources.