Align-VAE: Aligning Attention and Geometry in VecSet VAEs for 3D Reconstruction

Key points

• Although each token originates from a query at a sampled surface location, whether these latents truly align with geometry remains unclear.1
• We investigate this alignment and identify four issues: (i) attention-sink-like tokens that attract queries from across the shape; (ii) spatially periodic repetitions in decoder attention responses; (iii) mismatched normal information in encoder and decoder query features; and (iv) missing supervision of the intrinsic relation between signed distance fields and surface normals.1
• These changes align latent retrieval and field prediction with geometry without enlarging the per-shape code.1
• On the ABO dataset, Align-VAE reduces Chamfer distance by 26.6% compared to the standard VecSet VAE.1

Sources (1)

• [1]Align-VAE: Aligning Attention and Geometry in VecSet VAEs for 3D Reconstruction
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 10, 08:53 AM

  Set-based variational autoencoders (VecSet VAEs) represent 3D shapes as compact sets of latent tokens.

  Although each token originates from a query at a sampled surface location, whether these latents truly align with geometry remains unclear.

Extractive summary: sentences quoted from the sources.