EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience

Key points

• We present EgoXperience, a full-stack paradigm that addresses these needs through data collection, automatic annotation, unified modeling, and benchmarking.1
• Central to EgoXperience is an automatic annotation engine, EgoX-Engine, that couples geometrically verified object tracking with SLAM-based world alignment, yielding consistent supervision for the camera, scene, hands, and objects.1
• To translate this supervision into reconstruction, we introduce EgoX, trained with EgoX-Data-6M and complementary annotations from additional public datasets.1
• Given monocular egocentric video and first-frame object masks, our model achieves joint metric reconstruction of the camera, scene, hands, and objects in a shared world coordinate frame with a single forward pass.1

Sources (2)

• [1]EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience
  Hugging Face Daily Papers · Oct 19, 12:00 AM

  Reconstructing human activity from egocentric video requires more than recovering individual 3D components: the observer's motion, surrounding geometry, and hand-object interactions must fit together in a consistent world.

  We present EgoXperience, a full-stack paradigm that addresses these needs through data collection, automatic annotation, unified modeling, and benchmarking.

• [2]EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 19, 03:56 PM · same content

Extractive summary: sentences quoted from the sources.