ScreenASR-Bench: A Tiered Benchmark for Speech Recognition with Visual Context
To address this gap, we introduce ScreenASR-Bench , a tiered benchmark with three progressively challenging levels: Explicit Textual Evidence (ETE), based on direct on-screen text; Explicit Visual Evidence (EVE), using visible content rather than on-screen text; and Implicit Semantic Evidence (ISE), requiring scene-level inference.
ProofPaper ↗
Key points
- Current ASR models perform well on general benchmarks but still struggle with acoustically ambiguous words.
- Visual context can aid recognition in screen-based scenarios, yet existing benchmarks provide limited insight into the benefits of different types of visual evidence.
- Extensive experiments on ASR models, Large Audio-Language Models (LALMs) and Omni models show that incorporating visual context substantially improves recall over the audio-only setting across all three model families.
- To facilitate future research on visually grounded speech recognition, we publicly release ScreenASR-Bench.
Sources (1)
- [1]ScreenASR-Bench: A Tiered Benchmark for Speech Recognition with Visual ContextarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 12, 06:36 AM
To address this gap, we introduce ScreenASR-Bench , a tiered benchmark with three progressively challenging levels: Explicit Textual Evidence (ETE), based on direct on-screen text; Explicit Visual Evidence (EVE), using visible content rather than on-screen text; and Implicit Semantic Evidence (ISE), requiring scene-level inference.
Current ASR models perform well on general benchmarks but still struggle with acoustically ambiguous words.
Extractive summary: sentences quoted from the sources.