Omnilingual-MsASR: Benchmarking Multimodal, Multilingual, Long-Form Multi-Speaker ASR of Omni Large Language Models

Key points

• Omni-modal large language models (LLMs) have achieved remarkable progress in audio and video understanding.1
• However, existing models struggle with long-form, multi-speaker videos, especially in understanding the entire video and recognizing who says what and when.1
• Omnilingual-MsASR provides parallel audio and video inputs to evaluate models' cross-modal consistency and their ability to exploit complementary cues across modalities.1
• These findings highlight fine-grained temporal localization, speaker disambiguation, and cross-modal synergy as key challenges for future omni-modal foundation models.1

Sources (1)

• [1]Omnilingual-MsASR: Benchmarking Multimodal, Multilingual, Long-Form Multi-Speaker ASR of Omni Large Language Models
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 21, 09:11 AM

  To characterize the capability limit of current omni LLMs and guide future model development, we introduce Omnilingual-MsASR, a multilingual, multimodal benchmark for long-form Multi-speaker Automatic Speech Recognition (MsASR) covering six languages, recordings up to 30 minutes, and scenarios with up to 20 speakers.

  Omni-modal large language models (LLMs) have achieved remarkable progress in audio and video understanding.

Extractive summary: sentences quoted from the sources.