NeuralSRender: G-Buffer Guided Video Diffusion for Sim-to-Real Rendering

Key points

• The scarcity of high-quality annotated visual data remains a critical bottleneck for autonomous-driving perception.1
• Built on a pretrained video diffusion transformer (DiT), NeuralSRender uses a lightweight projector to inject geometric and material priors from G-buffers, and LoRA enables parameter-efficient adaptation.1
• To support training and evaluation, we introduce GenReal, a large-scale driving video dataset comprising synthetic and real-world videos with corresponding G-buffers.1
• Training perception models on our translated videos instead of raw simulator renders improves detection $AP{50}$ on nuImages from 35.1 to 40.6 and segmentation mIoU on Cityscapes from 32.4 to 40.8.1

Sources (1)

• [1]NeuralSRender: G-Buffer Guided Video Diffusion for Sim-to-Real Rendering
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 04:45 AM

  To address this limitation, we propose NeuralSRender, a sim-to-real video translation framework that enhances the photorealism of synthetic videos while maintaining geometric consistency with the source scenes.

  The scarcity of high-quality annotated visual data remains a critical bottleneck for autonomous-driving perception.

Extractive summary: sentences quoted from the sources.