Motion- and Gravity-Aware Feedforward 3D Reconstruction

Key points

• Feed-forward transformer models have recently achieved strong performance in multi-view 3D reconstruction, but they still rely primarily on appearance-based attention and often struggle in dynamic scenes with weak geometric cues.1
• During training, residual optical flow and gravity direction are provided as privileged signals to guide geometric representation learning; during inference, the model uses standard visual inputs only.1
• We integrate these priors into transformer backbones with lightweight architectural changes, preserving feed-forward efficiency.1
• These results suggest that LUPI offers a practical way to inject motion and physical priors into feed-forward 3D reconstruction without increasing test-time complexity.1

Sources (1)

• [1]Motion- and Gravity-Aware Feedforward 3D Reconstruction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 04:34 PM

  In this work, we introduce a motion- and gravity-aware framework under the Learning Using Privileged Information (LUPI) paradigm.

  Feed-forward transformer models have recently achieved strong performance in multi-view 3D reconstruction, but they still rely primarily on appearance-based attention and often struggle in dynamic scenes with weak geometric cues.

Extractive summary: sentences quoted from the sources.