What Egocentric Video Embeddings Encode and How to Combine Them with Inertial Sensors for Human Activity Recognition

Key points

• What these embeddings represent when the camera sits on the head, and how they should be combined with inertial sensing, has received little attention.1
• Across VideoMAEv2, CLIP, DINOv2 and the egocentric VC-1, the leading principal directions of the embedding are dominated by where the camera looks.1
• Measured head orientation explains them on Ego-Exo4D for egocentric (R^2 0.49-0.70) but not exocentric cameras, and the scene layout explains them on WEAR.1
• On WEAR, with temporal order removed and one random limb per one-second clip, we propagate inertial predictions over a within-participant video graph whose leading components are removed.1

Sources (1)

• [1]What Egocentric Video Embeddings Encode and How to Combine Them with Inertial Sensors for Human Activity Recognition
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 03:15 AM

  Egocentric video and body-worn inertial sensors are increasingly recorded together, and the video is usually encoded by pretrained models.

  What these embeddings represent when the camera sits on the head, and how they should be combined with inertial sensing, has received little attention.

Extractive summary: sentences quoted from the sources.