Drive2Gauss: FeedForward 4D Gaussian Reconstruction from Multimodal Generative Driving Videos

Key points

• Generating reusable driving environments requires both controllable future observations and an explicit representation that supports rendering from different viewpoints, yet existing methods still recover explicit scenes from generated RGB videos through optimization for each sequence.1
• A Multimodal Video Generator predicts the three modalities from reference multiview images and future control signals.1
• Furthermore, we introduce a Gaussian Query Decoder that uses depth to initialize sparse 3D queries and flow to associate the queries across time, and integrates appearance information through spatiotemporal feature aggregation and iterative query refinement.1
• Extensive results on nuScenes demonstrate that our approach attains an FID of 6.38 and an FVD of 12.43 for future RGB generation, a PSNR of 29.91 dB when reconstructing its generated observations, and supports both direct rendering under lateral camera shifts and downstream video fixing.1

Sources (1)

• [1]Drive2Gauss: FeedForward 4D Gaussian Reconstruction from Multimodal Generative Driving Videos
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:34 PM

  To overcome these limitations, we present Drive2Gauss, a feedforward framework that converts jointly generated RGB, metric depth, and dynamic flow into dynamic 4D Gaussian scenes.

  Generating reusable driving environments requires both controllable future observations and an explicit representation that supports rendering from different viewpoints, yet existing methods still recover explicit scenes from generated RGB videos through optimization for each sequence.

Extractive summary: sentences quoted from the sources.