A Principled Optimal Transport Framework for Frame Selection in Long Video Understanding
Long-video understanding with multimodal LLMs is fundamentally constrained by the mismatch between video length and the model's limited visual context budget, making frame selection essential for practical reasoning.
ProofPaper ↗
Key points
- In this paper, we revisit long-video frame selection from a principled optimization perspective.
- We formulate it as a structured evidence-allocation problem and propose OTFS, an optimal transport framework that maps multiple evidence sources onto video frames.
- We further show that, in the single-source case, our formulation induces a Gibbs-type distribution over frames, making Q-Frame's temperature-scaled distribution a special case of our framework while also providing a broader perspective for understanding related prior methods.
- Experiments on three long-video understanding benchmarks show that OTFS consistently outperforms strong training-free baselines.
Sources (1)
- [1]A Principled Optimal Transport Framework for Frame Selection in Long Video UnderstandingarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 06:12 AM
Long-video understanding with multimodal LLMs is fundamentally constrained by the mismatch between video length and the model's limited visual context budget, making frame selection essential for practical reasoning.
In this paper, we revisit long-video frame selection from a principled optimization perspective.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide
- Oct 19, 2026Introducing Clairic, a new System One class of extraction models
- Oct 19, 2026TokenGate: Adapting LLMs for Fast and Memory-Efficient Prefill with Learned Token Pruning
- Oct 18, 2026Toward Agentic Retrieval-Augmented Generation for Enterprise Retail Workflows over Heterogeneous Data Sources
- Oct 18, 2026Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation
- Oct 18, 2026BiDex-100: A Comprehensive Benchmark for Bimanual Dexterous Manipulation Across 100+ Simulation Tasks