DewHand: Direct Metric World-Space Hand Motion Reconstruction from Egocentric Videos

Key points

• Recovering metric world-space hand motion from egocentric video supports understanding human interactions and transferring them to embodied agents.1
• Three challenges complicate this task: multi-stage pipelines propagate errors from independently estimated hand and camera motion; monocular observations leave physical scale ambiguous; and occlusion and out-of-view hands admit multiple plausible motion trajectories.1
• To avoid compounding errors from multi-stage estimation, DewHand jointly predicts hand and camera motion in a shared world coordinate system within a single-stage pipeline.1
• To handle occlusion and out-of-view hands, we formulate reconstruction as conditional motion generation with flow matching.1

Sources (1)

• [1]DewHand: Direct Metric World-Space Hand Motion Reconstruction from Egocentric Videos
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 04:08 PM

  We present DewHand, a unified model for direct joint reconstruction of world-space hand and camera motion.

  Recovering metric world-space hand motion from egocentric video supports understanding human interactions and transferring them to embodied agents.

Extractive summary: sentences quoted from the sources.