Localization and Neural Splatting on the Edge: An FPGA Scene Memory for Robots

Key points

• Resource-constrained robots and autonomous vehicles need scene memories that localize a camera and render nearby views without a desktop GPU.1
• Scene-coordinate regression (SCR) and 3D Gaussian splatting (3DGS) fit because they avoid an SfM stage, but ACEsplat-style pipelines are GPU-centered, and moving them to an edge FPGA is hard because SCR is pose-sensitive and 3DGS rendering is irregular and capacity-sensitive.1
• Rather than proposing a new scene model, LENS co-designs the two dominant kernels with the board data path: an INT16 SCR accelerator that preserves pose accuracy, and a bit-exact fixed-point ordered 3DGS renderer that rewrites alpha transmittance as a log-sum-exp recurrence and removes provably useless streamed Gaussians.1
• The SCR engine preserves DSAC localization accuracy at 25.42 ms, the on-board renderer reaches 34.11 dB on Cambridge against ACEsplat's 33.20, novel-view synthesis on RealEstate10K reaches 27.84 dB against 26.11, and all ten Wayspots scenes reach the ACEsplat rendering target.1

Sources (1)

• [1]Localization and Neural Splatting on the Edge: An FPGA Scene Memory for Robots
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 18, 07:18 AM

  We present LENS, an edge scene-memory acceleration system on the AMD KR260 that brings an ACEsplat-style workload to a low-power FPGA/ARM platform.

  Resource-constrained robots and autonomous vehicles need scene memories that localize a camera and render nearby views without a desktop GPU.

Extractive summary: sentences quoted from the sources.