SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

Key points

• This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes?1
• Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (SceneAE).1
• reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory.1
• For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and1

Sources (1)

• [1]SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 14, 07:37 AM

  Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation.

  This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes?

Extractive summary: sentences quoted from the sources.