Differentiable Task-Driven Coded Aperture Camera Design with 3D Gaussian Scene Representations

Key points

• Camera design heavily influences perception task performance, yet cameras are typically designed by domain experts through costly experimentation, in isolation from the tasks they serve.1
• We use it to design two coded-aperture depth-from-defocus cameras, a monocular and a stereo configuration, by jointly optimising the coded aperture mask, camera parameters, and a depth estimation network end-to-end.1
• Our framework represents scenes with 3D Gaussian Splatting reconstructed from real-world captures, providing a differentiable image formation process for gradient-based optimisation over design spaces intractable for prior differentiable approaches.1
• Results demonstrate that our differentiable method improves over existing coded aperture design and non-differentiable camera design baselines, offering a step forward in automated and principled camera design for autonomous systems.1

Sources (1)

• [1]Differentiable Task-Driven Coded Aperture Camera Design with 3D Gaussian Scene Representations
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 10:48 AM

  We present a differentiable framework for jointly designing complex camera systems and downstream perception tasks.

  Camera design heavily influences perception task performance, yet cameras are typically designed by domain experts through costly experimentation, in isolation from the tasks they serve.

Extractive summary: sentences quoted from the sources.