Where, How, and When: Query-Guided Grounding for Video Understanding
For object-level grounding, we present QDETRv, a query-guided extension of the Detection Transformer for one-shot localization of unseen objects in videos, which combines query-context fusion, a memory module for temporal consistency, and unsupervised pretraining on synthetic object trajectories.
ProofPaper ↗
Key points
- Understanding video content requires reasoning at multiple levels of granularity, from identifying objects to interpreting their interactions and localizing when specific actions occur.
- This thesis studies three query-guided grounding tasks that answer where key objects are (object-level grounding), how they interact (relationship-level grounding), and when specific moments occur (moment-level grounding).
- For relationship-level grounding, we introduce few-shot referring relationships in videos: given a (subject, predicate, object) query and a few support videos sharing the predicate, the entities are localized by optimizing a T-partite random field using a meta-learned relation network with query-conditioned translational embeddings, Global Semantic Aggregation, Local Localization Aggregation, and belief-propagation-based message passing.
- For moment-level grounding, we propose the Moment Alignment Transformer (MATR) for video-to-video moment retrieval, which uses dual-stage Dynamic Time Warping for pre- and post-fusion alignment, learnable queries for boundary prediction, and self-supervised pretraining with temporal augmentations; MATR also extends to text-to-video moment retrieval with performance comparable to, or better than, the state of the art.
Sources (1)
- [1]Where, How, and When: Query-Guided Grounding for Video UnderstandingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 15, 07:32 AM
For object-level grounding, we present QDETRv, a query-guided extension of the Detection Transformer for one-shot localization of unseen objects in videos, which combines query-context fusion, a memory module for temporal consistency, and unsupervised pretraining on synthetic object trajectories.
Understanding video content requires reasoning at multiple levels of granularity, from identifying objects to interpreting their interactions and localizing when specific actions occur.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 15, 2026SpaceCache: Intermediate Feature Reuse for Efficient Semantic Segmentation on Point Cloud Streams
- Oct 15, 2026Learning to Think and Explore in 3D World
- Oct 15, 2026DuET: Note-Frame Dual Encoding for Piano Performance-to-Score Transcription
- Oct 14, 2026RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generation
- Oct 14, 2026mistralai/Timestral-209M
- Oct 14, 2026Your Transformer Is Secretly Elastic