Meta-Video: Efficient Standardized Evaluation for Video Understanding

Key points

• The reported benchmark scores in these works, however, are almost always produced by frameworks (e.g., lmms eval) used as black boxes: certain evaluation choices like frame sampling, preprocessing, and input formatting are inherited from preset defaults and seldom inspected carefully.1
• We show these unexamined choices are a major source of variance.1
• First, we curate existing suites for discriminability and efficiency, producing Meta-Video, a balanced 1,000-question benchmark over 791 videos spanning diverse tasks and durations, which largely preserves model rankings, at 5 times inference speedup for a full-frame count sweep experiment (4-128 frames).1
• We hope that the proposed evaluation framework and the Meta-Video benchmark will support video understanding research by providing a reproducible and efficient foundation to measure progress in a reliable way.1

Sources (1)

• [1]Meta-Video: Efficient Standardized Evaluation for Video Understanding
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 03:18 PM

  Video understanding benchmarks are the primary yardstick for measuring progress in video-language modeling, and a growing body of work relies on them to claim improvements.

  The reported benchmark scores in these works, however, are almost always produced by frameworks (e.g., lmms eval) used as black boxes: certain evaluation choices like frame sampling, preprocessing, and input formatting are inherited from preset defaults and seldom inspected carefully.

Extractive summary: sentences quoted from the sources.