When Left Is Right? Camera-Motion Understanding beyond Screen

Key points

• Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, most do no better than a coin flip.1
• We argue that camera-motion understanding needs two conditions at once: the representation must couple off-screen camera motion with on-screen flow dynamics, and it must be interpreted by the language model rather than trained as a standalone pose or depth estimator.1
• We propose CameraVLM, which pairs a motion-state representation with state-language alignment.1
• Camera tokens capture how the camera moves off screen, flow tokens capture how the subject and background move on screen, and a two-stage training strategy keeps language in the loop so that both are learned together with camera-motion semantics.1

Sources (2)

• [1]When Left Is Right? Camera-Motion Understanding beyond Screen
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 06:09 PM

  Can a vision-language model tell left from right?

  Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, most do no better than a coin flip.

• [2]When Left Is Right? Camera-Motion Understanding beyond Screen
  Hugging Face Daily Papers · Oct 12, 12:00 AM · same content

Extractive summary: sentences quoted from the sources.