Noise-Invariant Spatial Encoding for 3D Manipulation Policies under Depth Corruption

Key points

• 3D manipulation policies derive actions from explicit scene geometry rather than color images alone, enabling them to reason over metric distances, object relations, and robot-scene reachability.1
• We study this failure under realistic depth corruption and observe a sharp depth-noise cliff across modern 3D policies, even when they are trained with matched noisy depth.1
• This residual gap motivates a spatial-encoding approach: corrupted depth becomes entangled with the token coordinates and geometry features consumed by the action expert.1
• Across simulated benchmarks with sensor-realistic corruption and real-robot tasks involving transparent and reflective materials, NISE improves depth-noise robustness while preserving clean-depth performance, supporting 3D policies designed for commodity RGB-D deployment rather than clean-depth benchmarks alone.1

Sources (1)

• [1]Noise-Invariant Spatial Encoding for 3D Manipulation Policies under Depth Corruption
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 17, 12:58 PM

  We propose Noise-Invariant Spatial Encoding (NISE), a 3D policy architecture that learns trustworthy geometry features even facing imperfect depth.

  3D manipulation policies derive actions from explicit scene geometry rather than color images alone, enabling them to reason over metric distances, object relations, and robot-scene reachability.

Extractive summary: sentences quoted from the sources.