Connecting the Dots: Enduring Performance Gaps in ASR Systems Between Standard and Minority English Varieties
Automatic Speech Recognition (ASR) system evaluations have consistently revealed performance disparities along lines of race, gender, socioeconomic status, and language variety.
ProofPaper ↗
Key points
- We challenge this assumption of steady progress in reducing accent-related performance disparities, conducting a longitudinal evaluation of 16 prominent systems released between 2021-2025 across 8 datasets representing diverse English accent varieties.
- We find that the performance disparity between Mainstream US English (MUSE) and minority accents is stubborn, not reliably declining as a function of time.
- Specifically, while systems are reliably performant on MUSE, minority accents are subject to risks of large performance degradations between generations of systems, sometimes worsening by 65% absolute Word Error Rate.
- We argue that standard ASR benchmarks enact an implicit language policy privileging Mainstream US English varieties.
Sources (1)
- [1]Connecting the Dots: Enduring Performance Gaps in ASR Systems Between Standard and Minority English VarietiesarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 13, 05:42 AM
Automatic Speech Recognition (ASR) system evaluations have consistently revealed performance disparities along lines of race, gender, socioeconomic status, and language variety.
We challenge this assumption of steady progress in reducing accent-related performance disparities, conducting a longitudinal evaluation of 16 prominent systems released between 2021-2025 across 8 datasets representing diverse English accent varieties.
Extractive summary: sentences quoted from the sources.
