V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos

Key points

• Egocentric video supports diverse applications, including robot learning, augmented reality, and human activity understanding.1
• Recovering metric 3D hand trajectories from such video requires estimating both hand motion and camera egomotion to express the hands in a consistent reference frame.1
• Many existing pipelines recover hand and camera motion in separate stages, combining hand detection, hand reconstruction, camera tracking, and trajectory refinement.1
• It uses a self-supervised V-JEPA 2.1 encoder to extract spatio-temporal features from full-frame video and a query-based predictor to decode these features into per-frame hand poses, hand presence, and camera motion.1

Sources (1)

• [1]V-JEPA Hand: Simple, Accurate and Fast 3D Hand and Camera Trajectory Recovery from Egocentric Videos
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 05:34 PM

  We present V-JEPA Hand, a simple feed-forward video model that jointly reconstructs bilateral hand motion and camera motion from egocentric clips.

  Egocentric video supports diverse applications, including robot learning, augmented reality, and human activity understanding.

Extractive summary: sentences quoted from the sources.