VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings

Key points

• Fine-grained alignment between text and video object trajectories is a core challenge in multimodal understanding.1
• Existing embedding models excel at global retrieval, yet their single global representation entangles multiple objects and their spatiotemporal dynamics, limiting direct object-level grounding.1
• Experimental results show that VTrajEmbed-5B improves text-to-trajectory and trajectory-to-text R@1 over Qwen3-VL-Embedding-8B by 15.48 and 18.05 percentage points, respectively, and achieves 49.48% average R@1 across five global retrieval benchmarks, compared with 42.86% for Qwen3-VL-Embedding-8B.1

Sources (1)

• [1]VTrajEmbed: Aligning Text with Video Object Trajectories via Spatiotemporal Embeddings
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 16, 02:34 AM

  To address this issue, we propose VTrajEmbed, an embedding model that decomposes a video into object trajectories and aligns each trajectory embedding with text for object-level grounding through retrieval.

  Fine-grained alignment between text and video object trajectories is a core challenge in multimodal understanding.

Extractive summary: sentences quoted from the sources.