CountVideoBench: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models

Key points

• Counting questions are included in many existing VideoQA benchmarks, but the ground-truth count distributions in these benchmarks are heavily skewed toward small values.1
• Across the nine benchmarks we survey, the median ground-truth count is at most five, and the three most frequent values together account for between 45% and 93% of the videos.1
• We introduce CountVideoBench, a benchmark of 2,600 synthetic videos covering 26 cardinalities in the interval [5, 50] (100 videos each), grouped into six clusters of consecutive counts that span the range while enabling near-neighbor comparisons.1
• We evaluate sixteen video-language models from nine architecture families in a zero-shot setting.1

Sources (1)

• [1]CountVideoBench: A Benchmark to Evaluate Prediction Biases on the Object Counting task by Video-Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 03:20 PM

  Object counting in videos remains a poorly characterized capability of multimodal large language models.

  Counting questions are included in many existing VideoQA benchmarks, but the ground-truth count distributions in these benchmarks are heavily skewed toward small values.

Extractive summary: sentences quoted from the sources.