EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning

Key points

• Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning.1
• Existing audio-visual reasoning benchmarks that mainly rely on multiple-choice questions usually score models solely based on answer correctness, making it difficult to assess whether a model truly forms a reasoning chain from evidence to the correct answer.1
• We design a series of rigorous annotation procedures in which experts annotate audio and visual evidence and construct video-based distractors; humans review whether questions meet strict inspection criteria, and finally models filter textual and unimodal shortcuts.1
• We propose the Coverage Grounded Score (C.G.) to measure how much annotated evidence a model identifies, and the Bottleneck Grounded Score (B.G.) to penalize poor temporal localization in either modality; both are incorporated into answer correctness.1

Sources (2)

• [1]EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning
  Hugging Face Daily Papers · Oct 20, 12:00 AM

  To address these issues, we introduce EvaOmni, an evidence-grounded benchmark containing 392 videos and 1,027 purely human-annotated question-answer pairs, covering 15 capabilities and three levels: retrieval and localization, basic understanding, and complex reasoning.

  Omni LLMs can naturally locate key audio and visual evidence and integrate information across modalities for audiovisual reasoning.

• [2]EvaOmni: An Evidence-Grounded Benchmark for Audio-Visual Reasoning
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 20, 04:03 PM · same content

Extractive summary: sentences quoted from the sources.